WorksComputer Science
Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel
Abstract
Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control. A prominent example is LeWorldModel (LeWM), which trains end-to-end from pixels using an isotropic Gaussian anti-collapse regularizer (SIGReg) and plans by minimizing the squared Euclidean distance between predicted and goal embeddings. In this work, we investigate the test-time robustness of this latent planning interface. Analyzing the learned metric on Push-T, we first find that task-relevant object state heavily dominates latent distance: shifting the block by a single pixel displaces the embedding further than an equivalent agent shift, and swapping the block pose alone accounts for of the distance between scene pairs. However, because this pose information is distributed across a curved, high-dimensional manifold rather than a fixed linear subspace, the encoder cannot be trivially edited post-hoc to isolate or redirect it. We next demonstrate that this geometry is highly vulnerable at test time. A white-box perturbation of just applied to a single input frame causes the encoder to register the block at an arbitrary, attacker-chosen pose in – of scenes without disturbing the encoded agent state. Perturbing either the goal image or the live camera observation drops Push-T planning success from to , whereas uniform random noise at twice the magnitude has negligible impact. This vulnerability persists across all four standard LeWM benchmarks (Push-T, Two-Room, Reacher, and OGBench-Cube). At , the attack bridges – of the squared latent distance to the target embedding in every environment, inducing planning degradation proportional to each task's reliance on the spoofed entity.
Introduction
World models allow agents to plan by predicting how an environment evolves across candidate future trajectories [20, 21]. An increasingly popular approach instantiates these models as Joint-Embedding Predictive Architectures (JEPAs) [2, 26], where an encoder maps observations into a compact latent space and a predictor forecasts subsequent latents conditioned on actions without pixel reconstruction. Because pretrained JEPAs are expensive to train and are now shared as off-the-shelf dynamics backbones [7, 43], their test-time reliability is a practical security concern as much as a benchmark accuracy problem.
LeWorldModel (LeWM) [29] is the first JEPA trained stably end-to-end from raw pixels using only a next-embedding prediction loss and the Sketched-Isotropic-Gaussian Regularizer (SIGReg) [5], which prevents collapse by constraining the latent distribution toward an isotropic Gaussian. At test time, LeWM plans via model-predictive control (MPC) by encoding the current observation and a goal image, rolling the predictor forward under candidate action sequences, and selecting the sequence whose terminal predicted embedding is closest in squared Euclidean distance to the goal. Every control decision thus depends entirely on two encoder passes and a single latent distance metric.
This design exposes a natural vulnerability. Because the planner observes only latent embeddings rather than true physical state, an adversary who subtly perturbs the input pixels to alter the encoded representation can cause the planner to optimize toward an incorrect outcome while leaving the model weights, solver, and goal specification intact [30]. While adversarial examples are well studied for classifiers and generative models [17, 28, 31, 38] and policy networks [24, 27, 42], JEPA planners differ in two key respects. First, the perturbation must act through an intervening optimization loop, reshaping the planner’s cost landscape rather than directly perturbing action logits. Second, LeWM is equipped with an isotropic Gaussian regularizer intended to regularize latent geometry and a runtime “surprise” score designed to flag out-of-distribution events [29], inviting the question of whether these components provide built-in adversarial robustness.
We show that neither component protects the planner at test time. We first examine how task-relevant state is represented in LeWM’s latent metric on Push-T § 4. By rendering counterfactual scenes that vary one state factor at a time, we find that block pose dominates latent distance: shifting the block by one pixel displaces the embedding roughly further than an equivalent agent shift, and swapping the block pose alone accounts for 99% of the squared distance between scene pairs. This clarifies the recent finding of Hu et al. [23] that physical state “barely influences” LeWM’s latent distance (Spearman 0.13). We show that this low global correlation stems from a ceiling effect, where latent distance correlates strongly with state among nearby scenes (Spearman 0.41 on the closest 5% of pairs) but saturates across distant pairs. Moreover, because block pose is encoded along a curved, distributed manifold whose local tangent rotates with object position, no single linear weight can strip or redirect this information, yet an image-specific perturbation easily can.
We leverage this structure to construct a test-time state-spoofing attack (§ 5 and § 7; Figure 2). Optimizing a white-box perturbation of just against the public encoder weights on a single image causes an independent state probe to decode the block at an attacker-chosen pose in 73–95% of scenes, while the decoded agent position shifts by only a few pixels. Applying this perturbation offline to the goal image alone drops Push-T planning success from 87% to 32% (Figure 1). Perturbing the live camera stream is equally damaging, whereas uniform noise at twice the magnitude has no effect. The attack transfers across all four released LeWM environments (Push-T, Two-Room, Reacher, and OGBench-Cube), where the magnitude of planning failure reflects each task’s dependence on the spoofed object, and where on-manifold counterfactual targets prove essential for accurate evaluation (§ 7.3).
Figure 1. A perturbation of the goal image redirects an unchanged world-model planner. The same Push-T episode is planned twice with identical LeWM weights, solver and seed: with the clean goal image (top) and with one perturbed by at most per pixel (bottom), visually indistinguishable from it. Dashed outlines mark the true goal pose (blue) and the fake pose the attacker steers the encoder toward (orange): the clean run ends on the former, the attacked run next to the latter (an episode in which the attack succeeds, Sec. G; scenes cropped to the region where the block moves). Right: planning success over 300 episodes; noise of twice the magnitude is harmless.
Figure 2. Overview. Top: LeWM encodes the camera frame and the goal image with a frozen ViT-Tiny encoder , forecasts the latent under candidate actions with an action-conditioned transformer predictor , and a CEM planner searches for the actions that minimize the latent cost . Bottom: our attacker (orange) leaves all of this untouched. It (1) renders a counterfactual scene in which the task object is displaced, (2) encodes it with the public encoder to a target , (3) optimizes an -bounded perturbation with by gradient steps through the encoder, and (4) adds to the goal image (i) or to the camera frames (ii). The planner then pursues the fake goal.
Because the attack leaves model weights untouched and operates purely on input frames, standard model-side checks including SIGReg statistics, clean prediction loss, and nominal task accuracy pass identically to the unperturbed baseline. While LeWM’s surprise monitor can detect isolated perturbed frames, it struggles against temporally consistent spoofing and fails completely against goal-image perturbations, as the forward dynamics model never predicts the goal. To our knowledge, this is the first demonstration that a Gaussian-regularized JEPA planner can be reliably broken by imperceptible, checkpoint-preserving test-time perturbations.
Contributions.
We analyze the latent geometry of LeWM on Push-T, showing that block pose dominates latent distance along a curved, high-dimensional manifold and explaining why prior reports of weak state–latent alignment arise from a distance-saturation ceiling effect (§ 4).
We propose a white-box, single-image state-spoofing attack () that forces the encoder to report an arbitrary target object pose while preserving non-targeted state variables (§ 5 and § 7.1).
We evaluate end-to-end MPC planning under the standard
LeWM protocol, demonstrating that perturbing either the goal image or live camera observations drops success from 87% to 21–35% at =4/255–8/255, whereas 2× larger random noise is harmless (§ 7.2).
We demonstrate transfer across all four LeWM benchmarks, showing that planning degradation scales with task sensitivity to the spoofed entity and establishing why on-manifold counterfactuals are required for faithful measurement (§ 7.3).
We show that standard diagnostics like SIGReg, prediction loss, and runtime surprise fail to safeguard the planner, with goal-image attacks bypassing surprise checks by construction (§ 7.4).
Related Work
JEPA world models and planning. Visual world models learn environment dynamics to plan over imagined future trajectories [20, 21, 22]. Joint-Embedding Predictive Architectures (JEPAs) achieve this by predicting target embeddings rather than reconstructing pixels [2, 7, 26]. In action-conditioned settings, DINO-WM [43] plans over frozen DINOv2 features [33], PLDM [37] trains end-to-end with a VICReg-inspired objective [6], and LeWM [29] trains end-to-end from raw pixels using a single Gaussian regularizer, SIGReg [5]. At test time, these frameworks all optimize a latent goal-matching cost using sampling-based solvers such as CEM [13, 34]. We center our empirical study on LeWM because its checkpoints and planning code are publicly available, though the core attack surface, control mediated solely through an image encoder and Euclidean latent distance, applies broadly across this family.
Representation geometry for planning. Preventing latent collapse does not guarantee a representation geometry conducive to planning, since task-relevant state may be linearly decodable while contributing little to the overall latent distance. Hu et al. [23] termed this property metric leverage and proposed SCALE to calibrate latent distances against task-state distances during training. Building on their geometric diagnostics, we show that block pose in Push-T actually dominates local latent distances, and we exploit the non-linear structure of this representation to mount a test-time spoofing attack.
Adversarial examples and feature-space attacks. Small -bounded perturbations reliably mislead classifiers [17, 38], typically via projected gradient descent (PGD) [10, 28]. Instead of targeting class logits, we build on feature-space attacks that match internal representations to a guide image [36] or steer generative models at runtime [4]. Here, we attack a world-model encoder by matching the embedding of an on-manifold counterfactual scene to misdirect a downstream planner. Following standard evaluation practices [3, 11, 39], we measure spoofing via an external state probe and benchmark against matched-budget noise rather than reporting only the optimized latent loss.
Attacks on control and world models. Prior adversarial attacks in control largely target policies that map observations directly to actions, including deep RL agents [16, 24, 27, 42] and vision-language-action models [40]. Most relevant to our work, Riaño et al. [35] recently introduced a supply-chain backdoor for latent world models (including LeWM), where a compromised checkpoint embeds a trigger-gated action hijack that survives downstream fine-tuning, echoing backdoors in pretrained vision encoders [18, 25]. Our threat model is complementary to theirs, as we consider an attacker who cannot modify the checkpoint, training data, or controller, and instead only perturb an input image at test time, leaving the model weights completely clean. While various input transformations and auxiliary detectors have been explored for classifiers [12, 19, 32, 41], they remain brittle under adaptive attacks [9, 39]. Because LeWM includes no explicit input defense, we test whether its SIGReg objective and runtime surprise metric provide implicit protection.
Background and Threat Model
LeWM and Latent Planning
LeWM pairs a ViT-Tiny encoder , which maps a image observation to a latent vector , with an action-conditioned transformer predictor that forecasts [14, 29]. Both modules are trained jointly on offline trajectories to minimize the one-step prediction loss plus SIGReg (). SIGReg prevents latent collapse by driving the embedding distribution toward an isotropic Gaussian, projecting each batch onto resampled unit directions and penalizing the mean 1D Epps–Pulley normality statistic [5, 15]. Note that SIGReg defines no hard pass/fail threshold, operating purely as a batch-averaged training regularizer.
At test time, the frozen model plans via receding-horizon model-predictive control (MPC). Given the current frame and a goal image , LeWM encodes and and runs the Cross-Entropy Method (CEM) [13] to find an action sequence minimizing the terminal latent cost
replanning as new observations arrive [29]. Importantly, Eq. (1) routes the entire control objective through the Euclidean geometry of the latent space, anchored by just two encoder passes on and .
Experimental Setup
We evaluate the official quentinll/lewm-pusht checkpoint on Push-T [29, 43], a 2D task where a circular blue agent pushes a grey T-shaped block toward a goal pose. The arena spans 512 × 512 units (224 × 224 pixels, or 2.28 units/pixel), with success defined as placing the block within 20 units and 20° of the target. Following the published protocol (CEM with 300 samples, 30 iterations, horizon 5, frame-skip 5, 50-step budget, and goals sampled 25 steps ahead), we reproduce the reported 96% success rate on the original episode subset. Because the baseline success rate across independent sets of episodes vary between 85% and 87%, every attack is benchmarked against a paired clean run using identical episodes and CEM seeds. All experiments run the encoder in evaluation mode (Sec. B), and the other three environments follow the same protocol (§ 7.3).
Threat Model
We study a test-time, input-only adversary (Tab. A1, appendix) who perturbs images entering the encoder, the goal image, the live camera frame, or both, within an bound of while maintaining valid 8-bit pixel values. The attacker has white-box access to the public encoder weights and can render a counterfactual target scene (an assumption we revisit in § 8), but cannot modify the checkpoint, predictor, planner, solver randomness, or task evaluation criteria. This threat model is orthogonal to the supply-chain backdoor of Riaño et al. [35], which assumes a poisoned checkpoint evaluated on clean inputs. Here, the checkpoint remains completely unmodified while test-time inputs are perturbed, guaranteeing that any weight-level audit including SIGReg statistics, clean prediction loss, and nominal task accuracy passes by construction.
Where Task State Lives in LeWM
Before attacking the planner, we characterize how LeWM’s latent metric represents task state on Push-T. This analysis explains why a small pixel perturbation can selectively shift the planner’s estimate of the block and why this vulnerability cannot be eliminated by a single linear edit of the encoder. We probe the encoder using counterfactual renders across 4,000 real scenes, re-rendering each in the simulator with a single state variable altered (agent position, block position, or block angle) and measuring the resulting latent shift relative to the median distance between unrelated scenes (19.7).
Block Pose Dominates the Latent Metric
Sensitivity. Shifting the block by one pixel displaces the embedding by 0.82 on average, compared to 0.28 for a one-pixel agent shift, roughly 3× less (Figure 3a). Relative to the task’s success tolerances, block position is approximately 3–4× more influential than agent position.
Figure 3. Latent geometry of task state in LeWM on Push-T. (a) Embedding displacement for a single-variable shift as a percentage of the median distance between unrelated scenes (19.7), with interquartile bands across scenes. Block position displaces the embedding far more than agent position and saturates near 16–32 pixels. (b) Cosine similarity between the actual finite displacement and the local tangent predicted from an infinitesimal shift. Alignment decays rapidly for the block, indicating a curved manifold, whereas the agent trajectory remains nearly linear.
Swap test. Transplanting only the block pose from one scene into another reproduces 99% of the mean squared latent distance between them (90% from position alone, 61% from angle alone), whereas transplanting the agent position reproduces 44%. Because the planning objective (Eq. (1)) directly minimizes squared Euclidean distance, block pose dominates the cost landscape optimized by the planner, consistent with recent analyses of metric calibration [23]. A ceiling, not an absence. Hu et al. [23] report a Spearman rank correlation of 0.13 between squared latent distance and squared state distance on Push-T, interpreting this low value as evidence that task state “barely” influences latent distance. We reproduce this global figure but find that it stems primarily from saturation. Latent distance grows rapidly with block displacement at short range before flattening, reaching 59% of the unrelated-scene distance at 16 pixels and 88% at 32 pixels (Figure 3a). Because the latent distance between randomly sampled scene pairs rapidly saturate near this ceiling, distant pairs carry minimal ordering signal and depress the global correlation. Restricting the evaluation to local neighborhoods yields a correlation of 0.41 on the closest 5% of pairs and 0.57 on the closest 0.1%, versus 0.11 on the remainder (Table A3, appendix). Task state is thus strongly represented at short range, the exact regime where the planner refines trajectories near the goal.
| Scene pair subset | block sep. | latent dist. | Spearman |
| All pairs | 51 px | 19.7 | 0.16 |
| Closest 0.1% | 4 px | 8.0 | 0.57 |
| Closest 5% | 14 px | 18.5 | 0.41 |
| Farthest 95% | 53 px | 19.7 | 0.11 |
Table A3. Distance saturation in LeWM’s latent metric. We report the Spearman rank correlation () between squared latent distance and squared ground-truth state distance across 6.3M clean Push-T scene pairs, stratified by physical state separation (alongside median block separation and median Euclidean latent distance). Results are consistent across random seeds.
The Representation Is Curved and Distributed
Dominating the metric does not imply that block pose occupies a low-dimensional linear subspace.
No fixed low-dimensional subspace. Capturing 90% of the block-position response across scenes requires 52 of the ∼101 active latent dimensions (39 for block angle, 55 for agent position). These subspaces also overlap heavily, with the 52 block dimensions capturing 60% of the agent response, comparable to the 57% they capture of generic inter-scene variation. Within any single scene, however, the local displacement directions of the block and agent are nearly orthogonal (median of 0.12–0.21). Thus, while the two objects are locally separable, no global “block axis” exists.
The block tangent rotates. Comparing the actual latent displacement under a finite shift to the local tangent direction estimated from an infinitesimal shift (Figure 3b), cosine similarity for block translation drops from 0.90–0.93 at 2–4 pixels to 0.82 at 8, 0.65 at 16, and 0.28 at 32 pixels, and from 0.93 at 5° to 0.30 at 45° for rotation. In contrast, the agent stays near 0.88 out to 16 pixels. The block encoding is therefore strongly curved, with its local sensitivity direction rotating as the block moves.
Implication for attacks. A curved, distributed manifold cannot be stripped or redirected by a single fixed linear edit of the encoder weights, since a projection that cancels block sensitivity in one scene is misaligned in another. In auxiliary experiments, global linear edits were either undone by brief predictor re-fitting or flagged by spikes in prediction loss, and even optimal linear concept erasure [8] leaves block pose nonlinearly decodable. Because the block and agent are locally orthogonal and the block tangent is well-defined at each state, however, an input perturbation optimized per image can selectively spoof the block pose while leaving the agent representation intact.
Test-Time State-Spoofing Attack
Objective
The attacker aims to make the encoder register a target object (here, the block) at a chosen pose while leaving the remaining scene state intact, subject to an imperceptible bound and valid 8-bit quantization. Given a clean image with embedding , the attacker renders a counterfactual scene identical to except with the block shifted or rotated by a specified offset, yielding the target embedding . The attack then searches for a perturbed image within the -ball around whose embedding matches :
Because is the embedding of a valid on-manifold scene differing from only in block pose, matching it forces the encoder to represent the block at the counterfeit pose while preserving the agent and background state. Although the same formulation can spoof the agent, we focus on the block as the primary task-relevant object (Figure 2).
Optimization
We solve Eq. (2) using projected gradient descent (PGD) [28] against the frozen public encoder weights, running 100 sign-gradient steps of size from a uniform random initialization in the -ball, followed by 20 discrete unit steps on the integer lattice so that is a valid uint8 image with no quantization loss (Sec. B). Optimization takes roughly 0.5 s per frame on a single GPU (0.06 s amortized over a batch, Table A2) and requires no access to the predictor or planner.
| Steps | passes | GPU (s) | batch (ms) | CPU (s) | spoof succ. |
| 100 + 20 | 121 | 0.46 | 57 | 3.39 | 73% |
| 50 + 20 | 71 | 0.27 | 35 | 1.83 | 73% |
| 20 + 20 | 41 | 0.19 | 20 | 1.21 | 71% |
| 10 + 20 | 31 | 0.12 | 15 | 0.91 | 70% |
| 20 + 5 | 26 | 0.10 | 13 | 0.76 | 70% |
| 10 + 5 | 16 | 0.07 | 8 | 0.44 | 62% |
| 5 + 5 | 11 | 0.04 | 5 | 0.29 | 57% |
Table A2. Computational cost versus PGD iteration budget (Push-T, +17 px block target, /255, 599 scenes). “Steps” denotes continuous sign-gradient steps in the -ball plus discrete unit steps on the integer lattice (top row corresponds to our default configuration). Runtime is reported for a single image on one GPU (NVIDIA RTX PRO 5000), across 8 CPU threads, and amortized per image over a batch of 250 on the GPU. “Spoof succ.” matches the definition in Table 1.
Two deployment points. Perturbations can be applied wherever observations enter the encoder (Figure 2). In the goal-image attack, only the goal image is perturbed once offline before the episode starts, leaving live camera frames untouched so the planner tracks the true state but optimizes toward a corrupted goal pose. In the camera-frame attack, the goal image remains clean while each live observation encoded at replanning steps are perturbed so that the block appears displaced, causing the planner to mislocalize the block relative to the true goal. The goal-image variant requires a weaker threat assumption (a single offline edit) and proves harder to detect.
Measuring Success
Rather than evaluating the attack by the latent distance loss it directly optimizes, we use an independent state probe [1]: an MLP trained solely on clean renders to predict agent position, block position, and block angle (median test error of 1.2 pixels on block position, 1.3° on angle, and 2.3 pixels on agent position). Because the probe is never accessed during attack optimization, it serves as an external readout of the state encoded in . We define spoof success as placing the decoded block closer to the target counterfactual pose than the true pose, and we report drift in the decoded agent position to measure selectivity. For closed-loop planning, we use the simulator’s ground-truth success criterion (§ 3.2). To distinguish selective spoofing from generic corruption [11, 39], we compare against matched-budget uniform integer noise and an untargeted PGD baseline that maximizes .
Experiments
We evaluate our attack on Push-T under the standard protocol (§ 3.2), testing state-spoofing selectivity (§ 7.1), closed-loop planning degradation (§ 7.2), transfer across LeWM’s other three benchmarks (§ 7.3), and evasion of built-in runtime checks (§ 7.4).
The Attack Spoofs the Block Reading
Table 1 summarizes spoofing accuracy across 520–600 scenes per setting. At , the independent probe decodes the block closer to a +17-pixel counterfactual target
| Counterfactual scene | spoof succ. | non-target drift | |
| Block +9 px | 2/4 | 64% / 95% | agent 3 px |
| Block +17 px | 4/8 | 73% / 98% | agent 6 px |
| Block +35 px | 4/8 | 24% / 90% | agent 16/12 px |
| Block rot. 45° | 4/8 | 81% / 95% | agent 9/7 px |
| Agent +9 px | 1/2/4 | 54% / 79% / 90% | block < 1 px |
| Uniform noise | 8 | < 1% | all < 1 px |
Table 1. Spoof success on Push-T. (520–600 scenes per row). Share of scenes where the independent probe places the target object closer to the counterfactual pose than the true pose, and median drift of the non-targeted object (selectivity).
than its true pose in 73% of scenes, rising to 90–98% at =8/255 for shifts up to 35 pixels and rotations up to 90°. For moderate shifts, the non-targeted agent position drifts by only a few pixels, on par with the probe’s 2.3-pixel clean test error, confirming that the attack exploits the local separability of block and agent representations (§ 4.2). Spoofing the agent instead is even easier (90% at =4/255 for a 9-pixel shift) while leaving the decoded block pose intact. Both controls behave as expected: matched-budget uniform noise shifts the embedding by only 0.06–0.42 (versus 7–18 required for a block spoof) and decoded states by under one pixel, whereas untargeted PGD moves the embedding substantially (6.4 at =1/255) but corrupts the agent state rather than selectively spoofing the block.
Planning Collapses Under the Attack
We next inject the perturbation into the MPC loop using the +17-pixel block target across 300 episodes (6 seeds 50), paired with clean runs on the exact same episodes (Table 2; Figure A1).
Table 2. Planning under attack on Push-T (success %, 300 episodes, +17 px block target). “Perfect” feeds the planner a clean render of the counterfactual target; matched-budget noise is harmless.
| Attacked input | clean | 2 | 4 | 8 | perfect | |
| Goal image | 87.0 | 79.0 | 62.0 | 32.3 | 21.3 | 18.3 |
| Camera frame | 87.3 | – | 49.3 | 35.3 | 25.3 | 27.0 |
| Noise | goal 87.7 | camera 88.3 |
Table 2.
Figure A1. Closed-loop planning success versus perturbation budget on Push-T (300 episodes; error bars denote 95% Wilson confidence intervals). Both the goal-image and camera-frame attacks degrade performance monotonically toward the unconstrained counterfactual floor, while uniform random noise of equal magnitude has no effect.
Goal-image attack. Perturbing only the goal image reduces planning success monotonically from 87.0% to 32.3% at =4/255 (−54.7 points, paired 95% CI [−60.7, −48.7]) and 21.3% at =8/255. Feeding the planner a clean render of the counterfactual goal (“perfect”) yields 18.3%, showing that =4/255 already achieves most of the possible degradation. The failure is directed rather than random: across evaluation trials, the fraction of episodes in which the block terminates closer to the spoofed target than the ground-truth goal increase from 6% at =1/255 to 65% at =8/255, while the agent still reaches its target region. Other counterfactual targets behave similarly (Table 3), with a 35-pixel shift dropping success to 7.7% at =8/255 (matching the perfect-fake floor) and a 45° rotation dropping it to 25.3%.
Table 3. Other counterfactual targets on Push-T (goal-image attack, success %, 300 episodes). “Nearer” is the share of episodes at where the block ends closer to the spoofed pose than the true goal.
| Counterfactual scene | 2 | 4 | 8 | perfect | nearer | |
| Block +17 px | 79.0 | 62.0 | 32.3 | 21.3 | 18.3 | 65% |
| Block +35 px | 76.3 | 57.7 | 25.0 | 7.7 | 7.7 | 64% |
| Block rot. | 82.7 | 67.3 | 32.0 | 25.3 | 28.0 | 47% |
Table 3.
Camera-frame attack and controls. Perturbing live camera observations during replanning is equally damaging (35.3% at =4/255, 25.3% at =8/255; idealized floor 27.0%). Uniform noise at =8/255 leaves success unchanged at both injection points, and a clean re-render of the true goal matches baseline performance (85.3%), confirminging that degradation arises from the optimized perturbation rather than noise or rendering artifacts.
The Attack Transfers Across Environments
We evaluate the goal-image attack on LeWM’s three remaining benchmarks, Two-Room, Reacher, and OGBench-Cube, under the same protocol (5 seeds 50 episodes), shifting the target entity by roughly twice its goal tolerance (0.4 rad on Reacher, as twice its tolerance is a barely visible ; Sec. F). Figure 4 shows that the perturbations remain visually imperceptible while driving the embedding nearly all the way to the counterfactual target. At , the attack closes 82–97% of the squared clean-to-counterfactual latent distance (90–99% at ; Table A7, appendix), and by planning success falls to within 3 points of (or below) the idealized floor across all tasks while uniform noise remains harmless (Table 4).
Table 4. Goal-image attack across all four LeWM environments (success %; Push-T: 6 seeds, others 5, episodes). “Perfect” uses a dataset-faithful counterfactual render; “Noise” is uniform noise.
| Environment | clean | 8 | 16 | perfect | noise | |
| Push-T | 87.0 | 32.3 | 21.3 | 19.0 | 18.3 | 87.7 |
| Two-Room | 86.0 | 38.4 | 30.0 | 32.4 | 38.4 | 87.2 |
| Reacher | 61.2 | 29.6 | 26.0 | 20.4 | 17.6 | 60.4 |
| Cube | 70.0 | 53.6 | 53.2 | 50.0 | 55.6 | 70.8 |
Table 4.

Figure 4. Clean versus attacked goal frames across the four LeWM environments (; selected by a fixed rule, Sec. G). Residuals are amplified 30×. Circles mark the true object pose (block, agent, fingertip, cube) and crosses mark the spoofed target pose; the attack closes 86%, 92%, 82%, and 95% of the squared distance to the target embedding (top to bottom).
| Environment | 4 | 8 | 16 | |
| Push-T | .52 | .86 | .97 | .99 |
| Two-Room | .86 | .91 | .86 | .90 |
| Reacher | .79 | .82 | .82 | .95 |
| Cube | .93 | .96 | .97 | .99 |
Table A7. Fraction of squared latent distance closed by the goal-image attack, defined as (median across episodes per seed, averaged across seeds; 6 seeds on Push-T, 5 on others).
Planning collapse tracks task sensitivity. In tasks where the planner directly controls the spoofed variable, agent position in Two-Room and arm configuration in Reacher, planning drops sharply at (86.0% 30.0% and 61.2% 26.0%), matching Push-T. On Cube, the encoder is spoofed equally well (97% distance closed), yet overall success drops only from 70.0% to 53.2% (paired 95% CI [−23, −1] points vs. [−71, −60] on Push-T). Table 5 explains why: in 73.2% of Cube episodes, the goal cube already lies within the 4 cm tolerance at the start (vs. < 9% elsewhere), and on the remaining episodes the clean planner succeeds only 38.8% of the time, leaving little margin for a fake goal to degrade (an unperturbed fake goal yields 26.9%). Even when the cube starts inside the goal tolerance,
Table 5. Task sensitivity. Share of episodes already satisfying the goal at initialization, and goal-image success (%) on the remaining episodes (). Last row: the 183 Cube episodes already met at start.
| Environment | met at start | clean | perfect | |
| Push-T | 0.3% | 87.0 | 21.1 | 18.1 |
| Two-Room | 8.4% | 84.7 | 23.6 | 32.8 |
| Reacher | 0.4% | 61.0 | 25.7 | 17.3 |
| Cube | 73.2% | 38.8 | 28.4 | 26.9 |
| already met | – | 81.4 | 62.3 | 66.1 |
| Attacked input (=2/4/8) | Surprise () | AUROC | TPR@5% | |
| Single frame | 23/70/116 | 1.00 | 100% | |
| All frames | 3.4/5.4/5.6 | .84/.89/.91 | 44/57/59% | |
| Goal image (any ) | not predicted (no surprise) |
Table 5. Surprise as a runtime detector (Push-T, 498 clips, +17 px block target; TPR reported at 5% FPR). An isolated spoofed frame is reliably flagged, a temporally consistent camera attack is missed roughly half the time, and goal-image attacks trigger no surprise by construction.
a spoofed goal still costs 15–19 points (81.4% → 62.3%). Latent spoofing is thus necessary but not sufficient for catastrophic failure, as downstream impact is bounded by the task’s reliance on the spoofed variable.
On-manifold counterfactuals are required. For the MuJoCo benchmarks, naively re-rendering a state can push the target image off the training manifold. Our standalone Reacher renderer draws the DM-Control checker floor in a shifted phase relative to the dataset, causing the encoder to map an identical state 223 squared units away from the dataset frame, more than a real 0.4 rad arm move (193). Using this off-manifold render fabricates an artificial collapse (replacing a goal frame with its zero-shift re-render drops clean success from 56% to 27–30%) and misleads the attack (Sec. F). We therefore draw MuJoCo counterfactuals on-manifold: a real same-episode frame with the arm displaced for Reacher, and a re-rendered simulator state with only the cube moved for Cube (whose renderer matches the dataset). An independent probe trained on real frames then decodes the attacked Reacher arm 20° from its true angle at =8/255 (tolerance 2.9°).
Built-In Diagnostics Do Not Catch the Attack
Because the checkpoint is unmodified, SIGReg, clean prediction loss, and nominal task success match the official release. At runtime, LeWM’s only built-in monitor is surprise, the discrepancy between the predicted next latent and the encoded observation (Tab. 6). While surprise flags a single spoofed frame (100% of clips), it catches only 42–59% of clips at a 5% false-positive rate when frames are perturbed consistently across the rollout (AUROC 0.76–0.91). Crucially, the goal-image attack escapes this check entirely because the forward model never predicts the goal image. Evaluating SIGReg on test batches is similarly ineffective, flagging 49–100% of clean, low-motion trajectories (Sec. E).
Discussion and Limitations
Assumptions. Our attack operates white-box against the public encoder weights and perturbs digital images entering the encoder rather than executing a physical-world attack. Generating the counterfactual target also relies on rendering a modified simulator state, which an external attacker would have to estimate from observations. Given that an MLP probe accurately decodes physical state from clean embeddings (§ 6.2), we expect such estimation to be viable, though fully black-box and transfer attacks remain open for future work.
Scope. While we evaluate all four LeWM benchmarks, our cross-environment study focuses on the goal-image attack (with camera-frame experiments on Push-T and Two-Room in Sec. F), tests a single target offset per environment (Reacher at 0.4 rad rather than twice its tolerance), and restrict the detailed geometry analysis to Push-T. The variation across benchmarks also nuances our central finding (§ 7.3), demonstrating that a complete latent spoof only collapses planning when the task actively depends on the spoofed variable. Lastly, we evaluate LeWM’s existing diagnostics as detectors but do not propose a new defense.
Defenses and the regularizer. Candidate input-level defenses include input smoothing or quantization [12, 19, 41], checking latent consistency between the goal image and initial camera frame to catch goal-image spoofing, and adversarial training of the encoder. Because auxiliary detectors are routinely broken by adaptive attacks [9, 39], any defense must be benchmarked against a defense-aware adversary, and whether geometry-calibrated encoders such as SCALE [23] are more or less vulnerable remains an open question. Fundamentally, SIGReg regularizes the global batch distribution of embeddings rather than the encoder’s local sensitivity to individual inputs, confirming that anti-collapse regularization provides no robustness at the pixel-to-latent interface. This complements the supply-chain backdoor of Riaño et al. [35], showing that LeWM can be compromised either by a poisoned checkpoint on clean inputs or by a clean checkpoint on a single perturbed input.
Conclusion
We demonstrated that a Gaussian-regularized JEPA planner can be driven from 87% to 32% success on Push-T by an imperceptible =4/255 perturbation of a single input image, while leaving the official checkpoint, and every model-side health check, completely untouched. The attack transfers across LeWM’s other three benchmarks, where planning degradation scales with task dependence on the spoofed object. This vulnerability stems directly from the model’s latent geometry: block pose dominates the planning metric but is encoded along a curved, distributed manifold that prevents global linear erasure while permitting local per-image redirection. Our findings highlight the image-to-latent interface as a critical vulnerability in reusable JEPA world models and motivate input-level defenses that are robust by design.
Threat Model Comparison
Table Table A1 contrasts our test-time, input-only threat model (§ 3.3) against the supply-chain checkpoint backdoor of Riaño et al. [35], summarizing how the assumptions on attacker capability differs between the two settings.
| Attacker controls | Backdoor [35] | Ours |
| Released checkpoint (weights) | ✓ | ✗ |
| Training data / pipeline | ✗ | ✗ |
| Downstream controller / planner | ✗ | ✗ |
| Single input image at test time | ✗ | ✓ |
| Passes clean checkpoint audit | partial | by constr. |
Table A1. Comparison of threat models. We contrast our test-time, input-only adversary against the supply-chain checkpoint backdoor of Riaño et al. [35]. A “✓” indicates components modified or controlled by the adversary.
Implementation Details
Evaluation protocol. All planning experiments execute the vision encoder in standard evaluation mode. Because LeWM’s projector head incorporates BatchNorm layers, its latent embeddings differ between training and evaluation modes; we use evaluation mode throughout to match the deployed inference configuration. Evaluation episodes are sampled identically to the official LeWM benchmark code (using the same dataset starting indices with goal observations sampled 25 steps ahead). Every attack condition replays the exact same episodes using identical CEM random seeds, ensuring strictly paired comparisons. Push-T evaluations use seeds 42–47 (6 × 50 = 300 episodes), while the remaining three benchmarks use seeds 42–46 (5 × 50 = 250 episodes).
Attack optimization. We optimize our objective using 100 PGD sign-gradient steps of size from a uniform random initialization within the -ball, projecting back onto both the -ball and the valid [0, 255] pixel range after each iteration. The resulting continuous iterate is then rounded to the nearest integer and refined with 20 discrete unit-sign steps directly on the integer lattice, retaining the candidate perturbation across all iterations that achieve the lowest latent matching loss so that the final output is a valid 8-bit image. Table A2 ablates the optimization budget without retuning the step size. Under our default 100 + 20 schedule, attacking a single frame takes 0.46 s on one GPU (121 forward-backward encoder passes at ∼4 ms per pass), 3.39 s across 8 CPU threads, and 57 ms per frame when amortized across a batch of 250 images. Reducing the schedule to 20 + 5 steps (26 passes, 0.10 s per image) preserves nearly all of the spoofing effectiveness.
State probe architecture (Push-T). Our external state probe is a two-hidden-layer MLP (256 hidden units per layer with GELU activations) that maps the 192-dimensional latent embedding to agent position, block position, and block orientation (parameterized as ). We train the probe on approximately 20,000 clean simulator renders (90/10 train/validation split) using AdamW with learning rate , weight decay , and a cosine annealing schedule over 3,000 steps at a batch size of 2,048. On held-out clean data, the probe achieves median errors of 1.2 pixels on block position, 1.3° on block angle, and 2.3 pixels on agent position. The probe weights are kept strictly isolated from the attack optimization loop. The Reacher joint-angle probe in Sec. F uses the exact same architecture and training hyperparameters on 9,000 real dataset frames.
Geometry Details
Table A3 provides the quantitative breakdown showing that the weak global correlation between latent distance and state distance discussed in § 4.1 is driven by distance saturation. When isolating block translation by restricting the evaluation to scene pairs with similar block orientations, the rank correlation between latent and physical distances remain strong up to roughly 16 pixels of separation (Spearman ) before collapsing to near zero beyond that threshold ().
Planning Success versus Budget
Figure A1 visualizes the dose-response curves corresponding to Table 2. Perturbing either the static goal image or the incoming camera frames at replanning steps steadily drive Push-T task completion down toward the idealized “perfect counterfactual goal” floor as increases, whereas matched-budget uniform random noise leaves planning performance untouched.
Detection by Surprise
LeWM’s primary runtime monitoring signal is surprise, the squared Euclidean error between the forward predictor’s one-step forecast and the actually observed embedding , which the original authors propose as a violation-of-expectation detector. We evaluate surprise across 498 trajectory clips under a counterfactual attack shifting the block by 17 pixels (Tab. 6, main text). Surprise reliably flags an isolated spoofed frame injected into an otherwise clean rollout, as a single-step shift mimics an instantaneous object teleportation and elevates prediction error by 23–132× across 100% of clips. However, when an adversary perturbs every incoming frame toward a temporally consistent counterfactual trajectory, the resulting latent sequence remains dynamically coherent: surprise increases by only 2.8–5.6× and flags just 42–59% of clips at a 5% false-positive rate (AUROC 0.76–0.91). More fundamentally, the goal-image attack bypasses this mechanism by construction, because the forward dynamics model never predicts the static goal embedding and thus computes no prediction residual for it.
We also evaluated whether test-time SIGReg statistics can serve as an input-level anomaly monitor, but found that the normality score across batches of perturbed observations provide no discriminative signal beyond the natural variance of the counterfactual scenes themselves. In practice, SIGReg flags 49–100% of clean trajectory batches that exhibit low motion diversity, rendering it unable to distinguish adversarial perturbations from benign distribution shifts. Consequently, the goal-image attack evades all built-in diagnostics offered by the model.
Cross-Environment Details
Evaluation protocol. Table A4 summarizes the manipulated target entity, nominal goal tolerance, counterfactual shift magnitude, and counterfactual generation method for each benchmark. We set the counterfactual displacement to twice the task’s success tolerance across all environments except Reacher, where twice the nominal tolerance (0.1 rad, or 5.7°) produces a barely perceptible arm movement; we therefore use a 0.4 rad (23°) shift on Reacher. Closed-loop planning uses each benchmark’s official pretrained LeWM checkpoint and hyperparameter configuration. Across all tables, “perfect” feeds the unperturbed counterfactual target image directly to the planner, while “noise” applies uniform integer perturbations in at to the goal image.
| Environment | entity | tolerance | target shift | target source |
| Push-T | block | 8.8 px | 17.5 px | render |
| Two-Room | agent | 16 px | 32 px | render |
| Reacher | arm | 0.05 rad | 0.4 rad | real frame |
| Cube | cube | 4 cm | 8 cm | goal state |
Table A4.
Table A4. Cross-environment attack protocol. Counterfactual target sources: render uses our standalone simulator renderer; real frame selects an on-manifold dataset frame from the same episode with a displaced arm configuration; goal state re-renders the goal frame’s exact simulator state after translating only the cube.
Renderer fidelity. To verify whether standalone simulator renders lie on the training manifold, we measure the squared latent distance between real dataset frames and our simulator’s re-render of the exact same physical state, comparing this gap against the latent distance to the displaced counterfactual target (see Table A5; 50 goal frames per benchmark, seed 42). For Push-T, Two-Room, and Cube, simulator re-renders match the dataset manifold closely, with a baseline rendering gap of squared units that is negligible compared to the -unit distance induced by the counterfactual state shift. On Reacher, however, the standalone DeepMind Control Suite renderer draws the checkerboard floor with a shifted texture phase relative to the released dataset, causing the encoder to map an identical arm configuration 223 squared units away from the dataset frame, exceeding the 193-unit shift produced by a true 0.4 rad arm displacement.
| Environment | re-render vs. dataset | clean goal → target |
| Push-T | 0.34 | 278 |
| Two-Room | 390 | |
| Cube | 0.005 | 222 |
| Reacher | 223 | 193 |
Table A5. Simulator rendering fidelity. Median squared Euclidean distance in latent space between a dataset frame and our standalone re-render of the identical state, compared against the median squared distance between the clean goal frame and the displaced counterfactual target (50 goal frames per benchmark).
Why Reacher requires on-manifold dataset targets. Using an off-manifold re-render as the counterfactual target distorts evaluation in two ways. First, it fabricates artificial planning failures: simply replacing the dataset goal image with our standalone re-render at zero arm displacement drops Reacher planning success from 56.0% to 27.3% (under the goal-image protocol) and 30.0% (under the camera-frame protocol) across 3 seeds × 50 episodes, meaning that the vast majority of the apparent performance collapse stems from the rendering discrepancy rather than arm mislocalization. Second, evaluating the attack exclusively on simulator re-renders, where both the clean observation and the counterfactual target is generated by the standalone renderer, understates attack effectiveness: at , PGD closes only 60% of the squared latent gap on re-rendered pairs (and 56% in a preliminary 400-scene probe study where the probe decoded the arm closer to the target in 55% of trials), compared to closing 82% of the gap when attacking real dataset frames (81% when targeting a re-rendered goal from a real frame). We therefore source all Reacher counterfactual targets from real dataset frames throughout our experiments.
Joint-angle probe on real Reacher frames. To verify that our attack genuinely shifts the encoded arm state on Reacher, we train an MLP joint-angle probe on 9,000 clean dataset frames (achieving a median held-out error of 0.26°) and attack 80 real goal frames toward displaced dataset frames (22.9° median displacement, vs. a 2.9° task tolerance). As shown in Table A6, the attack shifts the decoded joint angles by 20.1° away from the true configuration at , placing the arm closer to the counterfactual target in 72.5% of scenes, and brings the decoded pose to within 4.9° of the target configuration at .
| nearer target | err. to target | err. to true | gap closed | |
| 2 | 56.3% | 15.6° | 17.6° | 0.75 |
| 4 | 58.8% | 13.4° | 16.0° | 0.82 |
| 8 | 72.5% | 11.9° | 20.1° | 0.80 |
| 16 | 98.8% | 4.9° | 20.8° | 0.95 |
Table A6. Reacher joint-angle probe evaluation on real dataset frames (80 goal frames across budgets ). Angular errors denote the maximum per-joint deviation in degrees (baseline clean probe error is 0.26°).
Task sensitivity analysis on Cube. Table 5 evaluates each benchmark’s ground-truth goal criterion directly on the initial start and goal states of the evaluation episodes (Push-T: agent and block position within 20 units and angle within ; Two-Room: agent within 16 px; Reacher: both joints within 0.05 rad; Cube: cube position within 4 cm) and partitions per-episode planning outcomes according to whether the goal tolerance is already satisfied at (which occurs in only a single episode each on Push-T and Reacher). On Cube, the attack reduces success by −10.4 percentage points across the 67 active episodes that require moving the cube (paired 95% CI [−25.4, +4.5]) and by −19.1 points across the 183 episodes initialized inside the goal region ([−25.7, −12.6]); an unperturbed counterfactual goal reduces success by −11.9 and −15.3 points on these respective splits. Extending the episode horizon does not amplify the degradation: when sampling goals 75 steps ahead (2 seeds × 30 episodes), Cube planning success moves from 71.7% (clean) to 65.0% under both the unconstrained counterfactual goal and the attack.
Latent distance gap closed across budgets. Table A7 reports the fraction of squared clean-to-counterfactual latent distance closed by the goal-image attack across budgets, corresponding to the summary ranges cited in the main text (0.82–0.97 at and 0.90–0.99 at $\epsilon=16/255). Using the identical metric across all four benchmarks, the fraction of distance closed increases monotonically with everywhere except Two-Room, where it plateaus between 0.86 and 0.91.
Additional budgets and Two-Room camera-frame attack. At a smaller budget of , the goal-image attack reduces planning success to 62.0% on Push-T, 43.6% on Two-Room, 30.0% on Reacher, and 57.2% on Cube. Note that Two-Room success is non-monotonic between (30.0%) and (32.4%), a minor fluctuation well within seed-to-seed variance. Applying the live camera-frame attack to Two-Room (5 seeds 50 episodes) reduces planning success from 86.0% (clean) to 55.2%, 46.4%, and 48.0% at , , and , respectively (compared to 44.8% under an unperturbed counterfactual observation stream and 86.4% under uniform noise).
Example Selection Protocol
Qualitative attack examples (Figure 4). To avoid cherry-picking, each environment in Figure 4 is illustrated using a single representative frame chosen via a deterministic rule without visual inspection: among the candidate evaluation frames, we select the sample whose fraction of squared latent distances closed at lie closest to the candidate median. For Two-Room, Reacher, and Cube, the candidate pool comprises the first 16 planning-evaluation goal frames from seed 42 (with Cube restricted to the 10 frames in which the block rests on the tabletop so the counterfactual shift does not leave the cube suspended mid-air). For Push-T, the candidate pool comprises the 6 stored scenes from the state-probe evaluation in § 7.1 (rendered scenes with a +17.5-pixel block shift). The median fraction of latent distance closed across the candidate pools is 0.87, 0.91, 0.80, and 0.96 for Push-T, Two-Room, Reacher, and Cube, respectively, and the selected frames close 0.86, 0.92, 0.82, and 0.95. Across all four visualizations, the maximum pixel modification is bounded by .
Teaser episode selection (Figure 1). To select the rollout shown in Figure 1, we execute the 50 Push-T evaluation episodes from seed 42 under both the clean goal image and the perturbed goal image (achieving 94% clean success vs. 30% attacked success on this seed, whereas the bar chart in the figure reports the 6-seed mean). We filter for the 15 episodes in which the clean rollout succeeds, the attacked rollout fails, and the terminal block position under attack lands within the task’s 20-unit success threshold of the spoofed target pose. We then select the median episode ranked by the terminal distance margin (distance to the true goal minus distance to the spoofed target), in which the block finishes 6.8 pixels from the spoofed pose and 12.8 pixels from the true goal.
References
- [1]Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations (ICLR) Workshop, 2017.arxiv.org/abs/1610.01644
- [2]Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15619–15629, 2023.arxiv.org/abs/2301.08243
- [3]Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning (ICML), 2018.arxiv.org/abs/1802.00420
- [4]Luke Bailey, Evan Ong, Stuart Russell, and Scott Emmons. Image hijacks: Adversarial images can control generative models at runtime. In International Conference on Machine Learning (ICML), 2024.arxiv.org/abs/2309.00236
- [5]Randall Balestriero and Yann LeCun. LeJEPA: Provable and scalable self-supervised learning without the heuristics. arXiv preprint arXiv:2511.08544, 2025.arxiv.org/abs/2511.08544
- [6]Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations (ICLR), 2022.arxiv.org/abs/2105.04906
- [7]Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video. Transactions on Machine Learning Research (TMLR), 2024.arxiv.org/abs/2404.08471
- [8]Nora Belrose, David Schneider-Joseph, Shauli Ravfogel, Ryan Cotterell, Edward Raff, and Stella Biderman. LEACE: Perfect linear concept erasure in closed form. In Advances in Neural Information Processing Systems (NeurIPS), 2023.arxiv.org/abs/2306.03819
- [9]Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In ACM Workshop on Artificial Intelligence and Security (AISec), 2017.
- [10]Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (S&P), pages 39–57, 2017.arxiv.org/abs/1608.04644
- [11]Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705, 2019.arxiv.org/abs/1902.06705
- [12]Jeremy M. Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning (ICML), 2019.arxiv.org/abs/1902.02918
- [13]Pieter-Tjerk De Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. A tutorial on the cross-entropy method. Annals of Operations Research, 134(1):19–67, 2005.DOI
- [14]Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
- [15]Thomas W. Epps and Lawrence B. Pulley. A test for normality based on the empirical characteristic function. Biometrika, 70(3):723–726, 1983.DOI
- [16]Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial policies: Attacking deep reinforcement learning. In International Conference on Learning Representations (ICLR), 2020.arxiv.org/abs/1905.10615
- [17]Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.DOI
- [18]Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230–47244, 2019.DOI
- [19]Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1711.00117
- [20]David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems (NeurIPS), 2018.arxiv.org/abs/1809.01999
- [21]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640(8059):647–653, 2025.DOI
- [22]Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for continuous control. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.16828
- [23]Jiaming Hu, Yan Zheng, Shi Bo, Tian Wang, Florian Dubost, Alejandro Mottini, Junze Liu, Arvind Srinivasan, Kai Zhong, Kun Qian, Sharon Gao, and Qingjun Cui. Representation geometry matters for planning with JEPA world models. arXiv preprint arXiv:2608.16287, 2026.arxiv.org/abs/2608.16287
- [24]Sandy Huang, Nicolas Papernot, Ian Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks on neural network policies. In International Conference on Learning Representations (ICLR) Workshop, 2017.arxiv.org/abs/1702.02284
- [25]Jinyuan Jia, Yupei Liu, and Neil Zhenqiang Gong. BadEncoder: Backdoor attacks to pre-trained encoders in self-supervised learning. In IEEE Symposium on Security and Privacy (S&P), pages 2043–2059, 2022.arxiv.org/abs/2108.00352
- [26]Yann LeCun. A path towards autonomous machine intelligence. Open Review, 62(1), 2022.
- [27]Yen-Chen Lin, Zhang-Wei Hong, Yuan-Hong Liao, Meng-Li Shih, Ming-Yu Liu, and Min Sun. Tactics of adversarial attack on deep reinforcement learning agents. In International Joint Conference on Artificial Intelligence (IJCAI), 2017.
- [28]Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.arxiv.org/abs/1706.06083
- [29]Lucas Maes, Quentin Le Lidec, Damien Scieur, Yann LeCun, and Randall Balestriero. LeWorldModel: Stable end-to-end joint-embedding predictive architecture from pixels. arXiv preprint arXiv:2603.19312, 2026.arxiv.org/abs/2603.19312
- [30]Nicole Meng, Ronak Sahu, Miao Yin, Faysal Hossain Shezan, and Yingjie Lao. Whisperplat: Lossless steganography in 3d gaussian splatting. In Forty-third International Conference on Machine Learning.
- [31]Nicole Meng, Caleb Manicke, Ronak Sahu, Caiwen Ding, and Yingjie Lao. Advancing adversarial robustness in gnerfs: The il2-nerf attack. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16388–16397. IEEE, 2025.DOI
- [32]Jan Hendrik Metzen, Tim Genewein, Volker Fischer, and Bastian Bischoff. On detecting adversarial perturbations. In International Conference on Learning Representations (ICLR), 2017.arxiv.org/abs/1702.04267
- [33]Maxime Ouqab, Timothée Darcet, Théo Moutakanni, et al. DINOv2: Learning robust visual features without supervision. In Transactions on Machine Learning Research (TMLR), 2024.
- [34]Cristina Pinneri, Shambhuraj Sawant, Sebastian Blaes, Jan Achterhold, Joerg Stueckler, Michal Rolinek, and Georg Martius. Sample-efficient cross-entropy method for real-time planning. In Conference on Robot Learning (CoRL), pages 1049–1065, 2021.arxiv.org/abs/2008.06389
- [35]Roberto Riaño, Gorka Abad, Stepan Picek, and Aitor Urbieta. When the world lies: Backdoor attacks on latent world models for downstream control. arXiv preprint arXiv:2609.15781, 2026.arxiv.org/abs/2609.15781
- [36]Sara Sabour, Yanshuai Cao, Fartash Faghri, and David J. Fleet. Adversarial manipulation of deep representations. In International Conference on Learning Representations (ICLR), 2016.arxiv.org/abs/1511.05122
- [37]Vlad Sobal, Wancong Zhang, Kyunghyun Cho, Randall Balesterio, Tim G. J. Rudner, and Yann LeCun. Learning from reward-free offline data: A case for planning with latent dynamics models. arXiv preprint arXiv:2502.14819, 2025.arxiv.org/abs/2502.14819
- [38]Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations (ICLR), 2014.
- [39]Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. In Advances in Neural Information Processing Systems (NeurIPS), 2020.arxiv.org/abs/2002.08347
- [40]Taowen Wang, Cheng Liu, Xiaohui Xu, et al. Exploring the adversarial vulnerabilities of vision-language-action models in robotics. arXiv preprint arXiv:2411.13587, 2024.arxiv.org/abs/2411.13587
- [41]Weilin Xu, David Evans, and Yanjun Qi. Feature squeezing: Detecting adversarial examples in deep neural networks. In Network and Distributed Systems Security Symposium (NDSS), 2018.
- [42]Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane Boning, and Cho-Jui Hsieh. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems (NeurIPS), 33:21024–21037, 2020.arxiv.org/abs/2003.08938
- [43]Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. In International Conference on Machine Learning (ICML), 2025.arxiv.org/abs/2411.04983



