Rethinking Video Autoencoders for Interactive World Models

Introduction

A world model is a learned simulator of an environment (Ha & Schmidhuber, 2018). Given past frames and an agent’s actions, a video world model generates the next frames, and can be used to train agents [4] (Hafner et al. 2025), evaluate robot policies (Yang et al., 2024), plan the outcome of actions (Bar et al., 2025), or simply play a game (Bruce et al., 2024; Valevski et al., 2025; Hu et al., 2026). To enable all these applications, two different aspects have to be optimized: the throughput that measures the number of frames generated per second, and the latency that measures the delay between the action and its effect on the output. Throughput limits how fast a simulation can happen, whereas latency limits how reactive and faithful a simulation can be.

Most video world models operate in the latent space of a video autoencoder (Bruce et al., 2024; Valevski et al., 2025). To avoid designing an ad hoc autoencoder, many methods (He et al., 2025; Li et al., 2025a; Zhu et al., 2026a) reuse the autoencoder of a text-to-video generator such as LTX-2 (HaCohen et al., 2026), Wan 2.2 (Wan et al., 2025) or HunyuanVideo (Kong et al., 2024). They thus inherit the throughput and latency that come with it. Yet, such autoencoders were not designed with the throughput-latency trade-off of world models in mind, and often perform massive temporal compression since it has low impact on visual quality. A typical $8 \times$ temporal compression delays every action by at least 400 ms at 20 frames per second. Humans notice when their own movement appears more than about 230 ms late (Shimada et al., 2009), and interaction feels immediate only below 100 ms (Miller, 1968; Claypool & Claypool, 2006). This raises the question of how other throughput-latency trade-offs affect rollout quality and action following.

To address this question, we propose CADRE, an open family of causal diffusion representation autoencoders built for interactive world models. Its members differ only in a small bottleneck on frozen features, which sets the compression, and share a causal diffusion decoder that needs one or a few steps. To study compression across settings, we train the same world model, a causal diffusion transformer following MIRA (Hu et al., 2026), on each autoencoder, in two domains: manipulation (by a robot and by human hands) and navigation (driving and indoor). We compare CADRE with the autoencoders of Wan 2.2, Cosmos (NVIDIA et al., 2025) and LTX-2, the image autoencoder DC-AE (Chen et al., 2025b) and the MIRA autoencoder retrained on our data. CADRE forms most of the latency-quality frontier, especially with a moving camera Figure 1.

Interestingly, our study points to four findings. (1) At strong compression, a diffusion decoder is sharper than perceptual regression, and a few-step flow model trained by progressive self-distillation is a good fit. (2) In robot manipulation, temporal compression is more effective in the autoencoder than in the world model: patching inside the world model loses detail, and at eight frames also action following. (3) Manipulation and navigation call for opposite compressions. In manipulation the camera is still, so temporal compression costs little while spatial compression is costly. In navigation the camera moves and the balance reverses, so it pays to keep frames and compress spatially, the low-latency choice. (4) Surprisingly, with a still camera, coarser spatial tokens hurt motion far more than image quality, and, in robot manipulation, action following too.

Contributions.

• What compression controls in an interactive world model. We make explicit how an autoencoder’s spatial and temporal compression set a world model’s throughput and action latency, and we separate how much to compress from where to compress (Section 2).

• CADRE, causal diffusion representation autoencoders for interactive world models. Open models at seven spatio-temporal compressions, whose decoder learns one- or few-step decoding by progressive self-distillation, in a single stage and without an adversarial loss (Section 3).

• Where, and how much, to compress video for a world model. A controlled study in four environments, judged by rollouts and action following (Sections 4 and 5).

What Compression Controls in an Interactive World Model

Before studying how to compress, we make explicit what compression controls. An interactive world model generates in the latent space of an autoencoder that compresses a clip by $\tau$ temporally and $s$ spatially, written $\tau\times s \times s$: $2 \times32 \times32$ compresses $2 \times$ temporally and $32 \times$ spatially. These two factors set the world model’s cost, action latency and throughput. The compression itself can happen in the autoencoder or in the world model (Section 5.1).

Compression sets the world model’s cost. The world model we study is a causal, space-time-factorized diffusion transformer that predicts the next latent frame from the previous ones and the current action or action chunk (§ 4, Appendix § 14). Its cost grows with the number of tokens it processes, which only the spatial and temporal compression set: a clip of $T$ frames of size $H \times W$ becomes $N = (T/\tau)(H/s)(W/s)$ latent tokens for the world model.

A latent frame waits for all its actions. Every frame registers one action. A latent frame spans $\tau$ frames, so the world model can predict it only once all $\tau$ conditioning actions are known. It therefore consumes actions in chunks, and the actions of one chunk shape the next (Figure 2). The agent thus commits to $\tau$ actions before seeing their effect, as action-chunking policies do ((Zhao et al., 2023; Chi et al., 2024; Black et al., 2025)). Causality holds between chunks. Within a chunk, the $\tau$ frames decode from the same latent, and their order is learned from real video.

Temporal compression delays every action. The first action of a chunk waits $\tau/f$ for the chunk to close, where $f$ is the frame rate, then $t_{\mathrm{gen}}$ for the world model to generate the latent frame and $t_{\mathrm{dec}}$ for the autoencoder to decode it into $\tau$ frames, which sets the throughput:

No speed-up shortens the first term, which at $\tau= 8$ and 20 frames per second is 400 ms, beyond the 230 ms lag humans notice in their own movement ((Shimada et al., 2009)). In exchange, a larger $\tau$ raises the throughput and, at a fixed spatial factor, cuts the token count $N$.

CADRE: Causal Diffusion Representation Autoencoders

Studying compression means training the autoencoder at many compression points. The autoencoders of text-to-video generators release their weights but not their training data or code, so they cannot be retrained. We instead design an open family of autoencoders for world models, one per compression point (Figure 3). Its members share one recipe, differ only in their compression and are each trained from scratch, so the world models trained on them differ by compression alone.

Architecture

Frozen features and one trained layer encode the clip. As in MIRA ((Hu et al., 2026)), the encoder is that of a representation autoencoder ((Zheng et al., 2026)) and a frozen DiT-like extractor ((Siméoni et al., 2026)) encodes each frame into one feature per $p \times p$ patch, aggregating several layers as in RAЕv2 ((Singh et al., 2026)). One trained 3D convolution with kernel $(\tau,k,k)$ and stride $(\tau,s',s')$ compresses $\tau$ frames of $s'^2$ patches into a deterministic latent, with $s = ps'$ (Appendix § 12).

A causal diffusion decoder keeps textures sharp. A strongly compressed latent keeps a scene’s layout and semantics but little texture, so a perceptual regression decoder averages the textures consistent with it into a blur (Xu et al., 2026). We therefore use a diffusion decoder (Preechakul et al., 2022; Pernías et al., 2024; Chen et al., 2025d), which samples the clip conditioned on the latent. It is a space-time-factorized transformer [6] (Bertasius et al. 2021) on a $q \times q$ patch grid, whose blocks attend over space, then causally over time. Most blocks process one token per latent frame, that is, per $\tau$ video frames. A linear map then splits each token into $\tau$ tokens, one per frame, so the last blocks resolve motion within a latent frame. The latent conditions the diffusion decoder in three places: (i) lifted by a transposed convolution and summed with the patchified noisy clip, (ii) through a spatial AdaLN (Park et al., 2019; Zhu et al., 2020; Dufour et al., 2022; Yu et al., 2026) in every block, which also receives the flow time $t$ and the jump length $d$, the size of the denoising step the decoder takes (§ 3.2, Appendix § 12), and (iii) through a zero-initialized skip into a light convolutional head. The decoder is time-causal, so it streams chunk by chunk.

Training

The decoder learns by flow matching and perceptual losses. The decoder $g_\theta$ uses the $x$-prediction, $x$-loss form of flow matching (Li & He, 2026). It predicts the clean clip $x$ from the latent $z$ and a noisy clip $x_t = t x + (1-t)\epsilon$, with $\epsilon\sim\mathcal{N}(0,I)$ and $t=0$ pure noise. As in diffusion forcing (Chen et al., 2024a), each chunk of $\tau$ frames draws its own $t \sim U(0,1)$. The loss is

where the last argument, the jump length $d$, is zero here. The term $\ell_{\mathrm{rec}}$ sums a mean absolute error, two perceptual distances (LPIPS (Zhang et al., 2018) and its analog in DINO feature space) and a 3D wavelet term as in LTX-Video (HaCohen et al., 2024), weighted by gradient balancing. CADRE trains on a 76k-hour mixture of web, robot, driving and camera video (Appendix § 13).

Self-distillation makes one or two decoding steps enough. Flow-matching trajectories curve, so following them takes many steps, while an interactive world model needs single- or few-step decoding. Shortcut models (Frans et al., 2025) and self-distilled flow maps (Boffi et al., 2025) condition the network on its step size and train one large step to agree with two smaller ones. Following them, $g_\theta(z,x_r,r,d)$ predicts where a jump from $r$ to $r+d$ lands, through $x_{r+d}=x_r+\frac{d}{1-r}(\hat{x}-x_r)$ with implied velocity $v_\theta=(\hat{x}-x_r)/(1-r)$. One jump from $r$ to $t$ is trained to land where two half jumps through $r=(r+t)/2$ (dropping $z$ from the notation),

where sg stops the gradient and $\hat{x}_u$ is the point reached by the first half jump. Unlike the consistency decoder (OpenAI, 2023) and SSDD (Vallely et al., 2026), which distill a decoder in a second stage, the CADRE decoder is its own teacher and learns one- or few-jump decoding in one stage.

Judging an Autoencoder by Its World Model

An autoencoder is trained to reconstruct its input, yet in latent generation, reconstruction and generation trade off (Yao et al., 2025; Hansen-Estruch et al., 2025), and a line of work shapes the latent for the generator instead ((Leng et al., 2025; Kouzelis et al., 2025; Zheng et al., 2026; Xie et al., 2026; Ramanana Rahary et al., 2026b)). We therefore score rollouts instead of reconstructions (Appendix § 16). As in MIRA ((Hu et al., 2026)), we judge each autoencoder by the rollouts of a world model trained on its latents; the world model, clips and actions are shared, so only the latent varies.

Four environments span manipulation and navigation. We train world models in four environments, two of manipulation and two of navigation (Table 1), so that no conclusion rests on a single environment. In manipulation the view is static for the robot and nearly static for the head-mounted camera of the human, so mainly the arms or hands and the objects they handle move. In navigation the view itself moves, so the whole frame changes; we separate driving, a dynamic scene where other road users move on their own, from indoor navigation in static scenes. Action complexity ranges from 2 dimensions for driving to 102 for two hands. All videos are resampled to 20 frames per second at $288 \times512$. Evaluation uses held-out episodes, and the official split for driving.

	Environment	Dataset	Camera	Action	DoF
Manipulation	Robot	ABC ([3])	static	joint deltas	14
Manipulation	Human	EgoDex ((Hoque et al., 2026))	nearly static	hand-pose deltas	102
Navigation	Driving	Physical AI AV ((NVIDIA, 2025))	moving	speed, curvature	2
Navigation	Indoor	RealEstate10K ((Zhou et al., 2018))	moving	camera twist	6

One world model recipe serves every autoencoder. The world model is a 1.2B-parameter causal diffusion transformer conditioned on the $\tau$ actions of each latent frame and trained with diffusion forcing ((Chen et al., 2024a)) (Appendix § 14). Every autoencoder, CADRE included, uses the same recipe from MIRA ((Hu et al., 2026)), with its latents standardized by their own statistics. Following SD3 ((Esser et al., 2024)) and RAE ((Zheng et al., 2026)), our tuning includes shifting the noise schedule with each autoencoder’s latent size and token count.

Rollouts are scored on image and motion quality, and on action following. Each world model receives 40 real frames (2 s) with their actions and generates the next 96 (4.8 s) under the recorded actions, on 2048 held-out samples per environment shared across autoencoders. We score the last 16 decoded frames. FID ((Heusel et al., 2017)) measures image quality and FVD ((Unterthiner et al., 2019)) also penalizes implausible motion and temporal seams. PSNR and LPIPS ((Zhang et al., 2018)) measure closeness to the recorded future; PSNR should always be accompanied by distributional metrics, as it rewards blur. Action following is measured by the action recoverability ratio (ARR), which extends the inverse dynamics evaluation of MIRA ((Hu et al., 2026)): a frozen probe reads the actions back from the rollout, and their coefficient of determination ($R^2$) is normalized by the probe’s $R^2$ on real video, so an ARR of one means real-video recoverability (Appendix § 15).

We compare seven CADRE autoencoders with five recent ones. The seven differ only in their compression $\tau\times s \times s$, and we name each CADRE-$\tau\times s \times s$ (Figure 12, Appendix § 17). Of the others, four are off the shelf and used at their native compression: Wan 2.2 ((Wan et al., 2025)) ($4 \times16 \times16$), Cosmos ((NVIDIA et al., 2025)) ($8 \times8 \times8$), LTX-2 ((HaCohen et al., 2026)) ($8 \times32 \times32$) and the image autoencoder DC-AE ((Chen et al., 2025b)) ($1 \times32 \times32$). The fifth retrains the MIRA recipe ((Hu et al., 2026)) on our data at $2 \times32 \times32$. The LTX-2 decoder reads future latents, so we decode each prefix and keep its newest frames, as a streaming system would. The generators they come from train on far more video, such as 20M curated hours for Cosmos, likely including many datasets used here.

Should the decoder use perceptual regression or diffusion?

A decoder can regress one video from the latent or sample one by diffusion. MIRA ((Hu et al., 2026)) takes the first route: it replaces the adversarial losses used since VQGAN ((Esser et al., 2021)) with perceptual losses (LPIPS and DINO) and works well on Rocket League. It is also the only autoencoder besides CADRE that runs at 20 frames per second within the 230 ms lag humans notice (Figure 1). To test this route beyond games, we retrain the MIRA recipe on our mixture (Appendix § 13) at $2 \times32 \times32$ with our extractor and budget (Figure 4a). Against the CADRE decoder, it gives higher PSNR but worse FID and FVD in every environment. In one step, the CADRE decoder lowers FVD by 1.4 to $2.5 \times$ with similar action following. A second step lowers FID by up to 23%.

Should the decoder use perceptual regression or diffusion?

Decode a compressed latent by diffusion. Such a latent fits many videos. Perceptual regression outputs their blurry average, while diffusion outputs one sharp video. Self-distilled, it decodes in one step as fast as regression with $1.4$ to $2.5\times$ lower FVD and similar action following; a second step improves it further.

How do the autoencoders trade quality for latency?

To compare the autoencoders on both axes at once, we plot rollout quality against action latency (Figure 4b for robot manipulation, Figure 1 for driving, Figure 14 for all); an autoencoder lies on the Pareto frontier when no other is both faster and better. CADRE autoencoders form most of this frontier in every environment. In 42 of the 48 comparisons with MIRA, Wan 2.2, Cosmos and LTX-2 (four environments; FID, FVD and ARR), a CADRE autoencoder is at least as fast and better by more than 2%; DC-AE, faster than all, is compared below. With two decoding steps, the best FID of every environment belongs to CADRE, most often to CADRE-$4 \times16 \times16$, and so does the best FVD, except in robot manipulation, where CADRE and Cosmos tie. The gap widens with a moving camera: in navigation, all CADRE autoencoders but one beat every other on FID and FVD, while Wan 2.2’s FVD grows from 53 in robot manipulation to 600 or more in navigation and LTX-2’s ARR falls to 0.04 indoors. CADRE also drifts least in 64 s robot manipulation rollouts (Appendix § 18).

At the fast end, CADRE-$1 \times48 \times48$ reacts about as fast as DC-AE and beats it in navigation; DC-AE wins in manipulation. Few autoencoders are both fluid and responsive: streaming on one H100 with eight denoising steps, only CADRE-$2 \times32 \times32$, CADRE-$2 \times16 \times16$ and MIRA run at 20 frames per second under the 230 ms lag humans notice Shimada et al. (2009) (Figure 1). The world models of Figure 5 run fluidly, at 29 to 58 frames per second (Appendix § 14), and, from the same context, follow actions never recorded with it, such as holding still or turning the other way (Figure 22).

How to Compress for Interactive World Models

We now ask three questions: which component should compress temporally, how many frames one latent should span, and what coarser spatial tokens trade for their speed. Comparisons follow the protocol of § 4, so arms differ only in how the video is compressed.

Should the autoencoder or the world model compress temporally?

Temporal compression can happen in the autoencoder or in the world model. The alternative to CADRE is an image autoencoder that leaves frames temporally uncompressed, with the world model patching its latent temporally. Both routes give the same token grid, but a different model learns the compression. In robot manipulation, we reach each grid a second time by patching the latent of CADRE-$1 \times16 \times16$ inside the world model (Figure 6). Only the compression changes. Actions and noise levels stay per frame, and a patched token receives the actions of every frame it covers.

Temporal compression in the autoencoder gives better frames than in the world model. Compressing in the autoencoder gives better FID, FVD, PSNR and LPIPS at every temporal grid, except FVD at $2\times32\times32$, where space is compressed too. The FVD gap grows with the factor, from 9 points when compressing 2 frames to 28 when compressing 8. Decoder capacity does not explain it, since the patched arm decodes with CADRE-$1\times16\times16$, the best reconstruction among them (Table 6). We attribute this to the training objectives. The autoencoder is trained to reconstruct pixels, so its compression keeps appearance. The world model is trained to predict latents, so its compression keeps only what helps that prediction. This matches image and video generation, where compressing in the autoencoder beats patching (Chen et al., 2025b; Gupta et al., 2024).

Table 6: Reconstruction of 3072 held-out OpenVid-HD (Nan et al., 2025) clips (16 frames at 288$\times$512), with the one-step decode for our autoencoders. No world model is involved.

Autoencoder ($\tau\times s \times s$)	FID$\downarrow$	FVD$\downarrow$	PSNR$\uparrow$	LPIPS$\downarrow$
CADRE 1$\times\$48$\times$48$ | 9.71 | 43.3 | 21.21 | 0.127
2$\times\$32$\times$32$ | 6.87 | 21.7 | 22.86 | 0.093
4$\times\$24$\times$24$ | 7.06 | 17.7 | 23.32 | 0.087
8$\times\$16$\times$16$ | 5.93 | 14.7 | 24.18 | 0.077
1$\times\$16$\times$16$ | 4.27 | 10.9 | 25.05 | 0.057
2$\times\$16$\times$16$ | 4.77 | 11.5 | 25.12 | 0.059
4$\times\$16$\times$16$ | 4.91 | 11.2 | 24.76 | 0.065
Others MIRA recipe (2$\times$32$\times$32)	15.73	127.1	24.44	0.146
Wan 2.2 (4$\times\$16$\times$16$) | 0.78 | 0.8 | 35.02 | 0.012
Cosmos (8$\times\$8$\times$8$) | 6.93 | 6.1 | 31.39 | 0.064
LTX-2 (8$\times\$32$\times$32$) | 4.76 | 6.5 | 28.24 | 0.039
DC-AE (1$\times\$32$\times$32$) | 3.59 | 22.7 | 26.83 | 0.060

Where temporal compression happens barely changes action following. The exception is 8 frames, where patching drops ARR from 0.72 to 0.56. Action following depends on the motion the actions cause, which the world model’s compression is trained to keep.

How many frames should one latent span?

To separate temporal compression from the cost of coarser tokens, we vary $\tau$ in two sweeps (Figure 7). The first holds $s = 16$ and isolates temporal compression. The second holds about 1100 tokens per 16-frame clip, so at a given $\tau$ its gap to the first measures the cost of coarser spatial tokens. All world models are matched in training steps and total training frames.

Latents should span many redundant frames in manipulation and few in navigation. In manipulation, the camera is mostly still, so consecutive frames are redundant and one latent can span many of them at no cost (green curve in Figure 7). In navigation, the camera moves and the view shifts by more than a token within one long latent (Appendix § 19), so long latents degrade motion, raising FVD by up to 64% over two-frame latents. Action following changes less (Appendix § 17). Camera motion is our leading explanation, though the domains also differ in actions and scenes.

In 64 s robot manipulation rollouts, long latents do not make errors accumulate: CADRE-8×16×16 is best from about 13 s on and ends with the lowest FVD (Appendix § 18). Long latents do leave seams, visible jumps between chunks. A boundary excess compares the frame-to-frame change across chunk boundaries with the change within chunks, relative to real video (Appendix § 21). It grows with $\tau$ in all four environments, to 1.6–1.8 at $\tau= 8$ (Table 10). With a moving view, the seams also show in FVD, which grows 1.6 to 2.1× more at $\tau= 8$ than at $\tau= 2$.

Environment	$\tau= 2$	$\tau= 4$	$\tau= 8$
Manipulation (robot)	1.17	1.34	1.81
Manipulation (human)	1.47	1.56	1.75
Navigation (driving)	1.22	1.34	1.83
Navigation (indoor)	1.08	1.18	1.58

At a fixed token budget, navigation favors few frames per latent and manipulation many. Along the orange curve of Figure 7, autoencoders give about 1100 tokens per 16-frame clip. In navigation, the low-latency CADRE-2×32×32 is best, with two frames and coarser tokens. In manipulation, CADRE-8×16×16 gives the most plausible motion, with eight frames and fine tokens.

What do coarser spatial tokens trade for their speed?

At a given $\tau$, coarser spatial tokens cut the world model’s cost, and the gap between the two sweeps isolates what they lose in quality and action following.

Fine spatial tokens matter most when the action is small and stays in view. In robot manipulation, the action shows in a small region that stays in view. At $\tau= 2$, coarser spatial tokens multiply FVD by 2.4 and lower action following, with almost no change in FID (orange against green in Figure 7). In navigation, the action moves the whole frame, which coarse tokens already capture, and the gap nearly vanishes (Appendix § 17). FID may miss errors confined to small regions, but ARR, which reads the motion itself, drops too (Appendix § 17).

What do coarser spatial tokens trade for their speed?

Surprisingly, coarser tokens hurt motion and action following far more than image quality, above all in manipulation. In manipulation, per-frame quality barely changes, but motion degrades, and so does action following in robot manipulation. In navigation, where the whole view shifts, they cost little.

Related Work

World models and their autoencoders. Video world models serve driving (Russell et al., 2025), robotics (Yang et al., 2024) and open scenes (Bruce et al., 2024). Interactive ones stream in real time [4] (Valevski et al. 2025), often by distilling a text-to-video generator whose autoencoder they inherit (He et al., 2025; Li et al., 2025a). Few study that autoencoder. SANA-WM (Zhu et al., 2026a) swaps two of them offline, and MIRA (Hu et al., 2026) judges its own through the world model, but only on a single game. We compare twelve autoencoders of varied compression through world models in four environments.

Latent structure and compression. Generation quality depends on the structure of the latent (Rombach et al., 2022; Yao et al., 2025). Representation autoencoders generate in the space of a frozen self-supervised encoder (Zheng et al., 2026; Xie et al., 2026), at one compression rate. Video autoencoders mostly compress 8× in space and 4× in time (Wan et al., 2025; Kong et al., 2024), some far more (HaCohen et al., 2024; 2026; Chen et al., 2025b). A generator can also compress through its patch size (Peebles & Xie, 2023; Gupta et al., 2024). For images, DC-AE (Chen et al., 2025b) finds that compressing in the autoencoder beats patching. We test this for video world models and build representation autoencoders that span seven spatio-temporal compressions.

Few-step diffusion decoders. Diffusion decoders avoid adversarial losses (Preechakul et al., 2022; Chen et al., 2025d) but need many steps, or a second stage to reach one (OpenAI, 2023; Vallaeys et al., 2026). Shortcut models and flow maps learn few-step generation in one stage from their own predictions (Frans et al., 2025; Boffi et al., 2025). Our causal video decoder learns one- or two-step decoding this way, with no adversarial loss or second stage. Appendix L extends this section.

Conclusion

An interactive world model inherits its cost per frame, its action latency and much of its quality from its autoencoder, so its compression is a design choice. To study this choice, we build CADRE, an open family spanning seven compressions, and train the same world model on each in manipulation and navigation. CADRE forms most of the latency-quality frontier, especially with a moving camera, and our study points to four findings. At strong compression, a diffusion decoder is sharper than perceptual regression, even in one or two steps. In robot manipulation, temporal compression is more effective in the autoencoder than in the world model. Manipulation and navigation call for opposite compressions: temporal compression costs little when the camera is still, and spatial compression when it moves. Surprisingly, with a still camera, coarser spatial tokens hurt motion far more than image quality, and, in robot manipulation, action following too.

Our world models each cover one environment; whether these findings carry over to a generalist one, and to policy training and planning, remains open. Our manipulation data use still or nearly still cameras; wrist-mounted cameras move, so our account predicts they favor the compression of navigation. We hope open autoencoders will let practitioners fit a world model’s compression to its scenes and to the latency and throughput its users need.

Reproducibility Statement

§ 3 and Appendix § 12 describe the autoencoder architecture, losses and training recipe, and Table 3 lists the hyperparameters of both the autoencoders and the world models. Appendix § 13 lists every source of the autoencoder training mixture with its sampling weight, size and preprocessing. Appendix § 14 gives the world model architecture, conditioning, objective, training recipe and inference protocol, and Appendix § 15 specifies the inverse dynamics probes behind action recoverability. § 4 fixes the evaluation protocol shared by every autoencoder, and Table 7 reports every number behind the figures. We release the code and the checkpoints.

	CADRE autoencoder	World model
Architecture	frozen DINOv3 ViT-B/16, conv bottleneck, ViT decoder 24 blocks, width 1024, 16 heads	causal DiT, 16 blocks, width 2048, 16 heads (4 key/value), 1.2B parameters
Latent	64 channels	autoencoder latent, no further patching
Objective	flow matching, MAE, LPIPS, DINO, 3D Haar, PSD	flow matching, diffusion forcing, PSD-U
Optimizer	AdamW, $\beta= (0.9, 0.95)$, no weight decay	AdamW, $\beta= (0.9, 0.99)$, weight decay 0.1
Learning rate	$2 \times10^{-4}$, 2k warm-up, then constant	$10^{-4}$, 1k warm-up, then constant
Gradient clipping	none	1.0
Steps, batch	150k, 96 clips (2 H100 nodes, 16 GPUs)	100k, 64 clips (2 H100 nodes, 16 GPUs)
Clips	16 frames, $288 \times512$, 20 fps	80 frames (40 context), $288 \times512$, 20 fps
Precision, EMA	bf16, 0.9999	bf16, 0.9999
Inference	1 or 2 decoding steps	10 denoising steps, context noise 0.2

Autoencoder ($\tau\times s \times s$)	Tokens	FID$\downarrow$	FVD$\downarrow$	PSNR$\uparrow$	LPIPS$\downarrow$	ARR$\uparrow$
Manipulation (robot, ABC)
CADRE $1 \times48 \times48$	1056	12.15 (14.69)	180.3 (186.7)	19.41 (19.54)	0.179 (0.184)	0.48 (0.48)
$2 \times32 \times32$	1152	8.92 (11.63)	95.2 (105.0)	20.78 (20.79)	0.144 (0.149)	0.45 (0.45)
$4 \times24 \times24$	1056	11.46 (13.95)	83.9 (93.8)	20.66 (20.67)	0.148 (0.152)	0.49 (0.48)
$8 \times16 \times16$	1152	9.34 (11.41)	40.1 (50.6)	21.33 (21.42)	0.127 (0.131)	0.72 (0.72)
$1 \times16 \times16$	9216	9.11 (10.55)	46.3 (56.2)	19.76 (19.90)	0.148 (0.150)	0.74 (0.74)
$2 \times16 \times16$	4608	8.53 (10.48)	39.4 (47.6)	20.80 (20.80)	0.132 (0.135)	0.73 (0.73)
$4 \times16 \times16$	2304	8.45 (10.86)	41.5 (51.9)	21.07 (21.09)	0.125 (0.129)	0.79 (0.79)
Others	MIRA recipe ($2 \times32 \times32$)	1152	29.52	267.1	21.12	0.172	0.39
	Wan 2.2 ($4 \times16 \times16$)	2304	9.29	52.9	20.85	0.118	0.85
	Cosmos ($8 \times8 \times8$)	4608	13.10	39.5	20.04	0.156	0.78
	LTX-2 ($8 \times32 \times32$)	288	22.24	69.6	19.47	0.164	0.25
	DC-AE ($1 \times32 \times32$)	2304	10.07	147.3	19.77	0.162	0.65
Manipulation (human, EgoDex)
CADRE $1 \times48 \times48$	1056	16.38 (19.32)	110.1 (109.2)	19.33 (19.59)	0.316 (0.318)	0.64 (0.63)
$2 \times32 \times32$	1152	12.66 (15.62)	57.2 (63.8)	20.08 (20.15)	0.288 (0.289)	0.75 (0.75)
$4 \times24 \times24$	1056	15.10 (17.29)	67.7 (69.2)	20.22 (20.21)	0.247 (0.249)	0.65 (0.65)
$8 \times16 \times16$	1152	11.84 (13.58)	42.6 (40.7)	20.71 (20.81)	0.241 (0.242)	0.63 (0.63)
$2 \times16 \times16$	4608	12.53 (13.60)	37.3 (42.4)	19.20 (19.25)	0.306 (0.307)	0.73 (0.73)
$4 \times16 \times16$	2304	10.93 (12.15)	50.8 (49.5)	20.32 (20.29)	0.248 (0.248)	0.78 (0.78)
Others	MIRA recipe ($2 \times32 \times32$)	1152	20.65	89.7	21.06	0.248	0.75
	Wan 2.2 ($4 \times16 \times16$)	2304	14.60	55.0	19.55	0.259	0.72
	Cosmos ($8 \times8 \times8$)	4608	15.33	83.8	20.41	0.266	−0.27
	LTX-2 ($8 \times32 \times32$)	288	24.78	64.9	19.91	0.274	−0.63
	DC-AE ($1 \times32 \times32$)	2304	11.82	61.0	19.65	0.307	0.39
Navigation (driving, Physical AI AV)
CADRE $1 \times48 \times48$	1056	15.58 (17.89)	253.1 (227.2)	17.02 (17.23)	0.385 (0.384)	0.90 (0.88)
$2 \times32 \times32$	1152	13.84 (15.99)	168.7 (171.3)	17.58 (17.64)	0.353 (0.354)	0.89 (0.89)
$4 \times24 \times24$	1056	20.79 (22.39)	403.0 (378.4)	17.39 (17.38)	0.370 (0.372)	0.67 (0.68)
$8 \times16 \times16$	1152	15.84 (17.26)	196.6 (196.5)	17.37 (17.41)	0.353 (0.353)	0.86 (0.84)
$2 \times16 \times16$	4608	16.45 (17.51)	150.5 (162.6)	17.83 (17.86)	0.351 (0.352)	0.97 (0.97)
$4 \times16 \times16$	2304	19.34 (20.45)	195.3 (198.5)	17.11 (17.10)	0.352 (0.353)	0.88 (0.89)
Others	MIRA recipe ($2 \times32 \times32$)	1152	21.42	311.6	18.05	0.376	0.86
	Wan 2.2 ($4 \times16 \times16$)	2304	37.36	599.7	16.69	0.407	0.66
	Cosmos ($8 \times8 \times8$)	4608	30.12	452.5	18.89	0.383	0.62
	LTX-2 ($8 \times32 \times32$)	288	98.36	913.3	19.21	0.429	0.57
	DC-AE ($1 \times32 \times32$)	2304	30.67	389.6	16.57	0.408	0.81
Navigation (indoor, RealEstate10K); PSNR at frame 48
CADRE $1 \times48 \times48$	1056	38.01 (40.13)	240.3 (238.4)	14.87 (15.07)	0.532 (0.531)	0.87 (0.88)
$2 \times32 \times32$	1152	33.69 (35.27)	182.9 (191.9)	15.35 (15.37)	0.524 (0.525)	0.91 (0.91)
$4 \times24 \times24$	1056	28.43 (29.36)	435.0 (426.2)	12.93 (12.96)	0.645 (0.647)	0.45 (0.45)
$8 \times16 \times16$	1152	36.42 (36.55)	271.5 (259.8)	15.80 (15.85)	0.543 (0.546)	0.79 (0.79)
$2 \times16 \times16$	4608	33.50 (34.13)	165.4 (167.3)	14.88 (14.90)	0.533 (0.533)	0.90 (0.90)
$4 \times16 \times16$	2304	30.68 (31.69)	161.7 (165.5)	15.74 (15.77)	0.473 (0.475)	0.93 (0.93)
Others	MIRA recipe ($2 \times32 \times32$)	1152	41.97	298.5	15.73	0.526	0.92
	Wan 2.2 ($4 \times16 \times16$)	2304	128.35	771.9	13.94	0.696	0.34
	Cosmos ($8 \times8 \times8$)	4608	60.53	377.7	14.74	0.624	0.39
	LTX-2 ($8 \times32 \times32$)	288	123.61	606.2	13.04	0.681	0.04
	DC-AE ($1 \times32 \times32$)	2304	49.13	339.3	14.24	0.586	0.86

Ethics Statement

All training and evaluation data come from published datasets, used under their licenses. Some show people, such as the hands in human manipulation and the faces in talking-head and action videos; we use them only to learn video representations and do not identify anyone. Generated video can be misused to fabricate footage. Our world models are specialized to manipulation and navigation, run at $288 \times512$ and follow low-level actions, which limits such use.

AI Use Statement

We used generative AI tools to implement and debug code, including the training, evaluation and plotting scripts; to draft and edit parts of the paper for readability; to create and edit figures; and to search for and format references. We reviewed all AI-assisted work and take responsibility for the final content of this work, including text, claims or artifacts produced with the aid of generative AI.

Overview

The supplementary material details how CADRE and its world models are built and scored, reports every autoencoder in every environment and metric, and adds the analyses behind the explanations of § 5: long rollouts, camera motion, counterfactual actions and seams. It also compares rollouts of every autoencoder.

Supplementary findings at a glance

• CADRE world models run fluidly on one H100. With 10 denoising steps, every CADRE autoencoder with $\tau\ge4$ sustains 20 frames per second, as does CADRE-2×32×32 with one decoding step, at 185 ms of action latency. At $\tau= 1$, this needs one decoding step and at most four denoising steps (Appendix § 14).

• Rollouts follow the actions, not only the usual motion of the environment. Scored against commands shifted by 48 frames, which the rollout never received, every CADRE autoencoder keeps a clear gap in every environment. The gap of Wan 2.2, Cosmos and LTX-2 shrinks to 0.04 to 0.14 in driving, where their rollouts reproduce forward travel (Appendix § 15).

• Reconstruction does not predict rollout quality. The rank correlation between reconstruction and rollout FVD is +0.76 in robot manipulation but −0.39 in driving and −0.35 indoors. Wan 2.2 reconstructs best by far, yet ranks fifth of twelve on rollout FVD in robot manipulation and last indoors (Appendix § 16).

• With two decoding steps, CADRE holds the best FID and FVD of every environment. Only Cosmos ties it, on robot-manipulation FVD. At 2×32×32, the diffusion decoder beats perceptual regression on FID and FVD in all four environments, and regression wins only PSNR, a sign of blur (Appendix § 17).

• Manipulation tolerates temporal compression, navigation spatial compression. At $s = 16$, robot-manipulation FVD stays between 39 and 46 from $\tau= 1$ to 8, while from $\tau= 2$ to 8 it rises by 31% in driving and 64% indoors. At $\tau= 2$, 32×32 tokens multiply robot-manipulation FVD by 2.4 and lower ARR from 0.73 to 0.45 at nearly the same FID, but raise navigation FVD by only 11 to 12% (Appendix § 17).

• Over 64 s, no metric collapses and CADRE drifts least. CADRE-8×16×16 is best from about 13 s on. FVD grows by 0.8 to 1.8 per second for CADRE, 2.0 to 2.7 for the others and 14.5 for Wan 2.2, which starts best. Action recoverability barely decays, though single rollouts can still fail late (Appendix § 18).

• The view moves far more in navigation than in manipulation. In manipulation, content moves by about one pixel per frame or less, under one token over eight frames. In navigation, it moves by more than two, and half to three quarters of the patches change per frame instead of a third (Appendix § 19).

• World models follow actions never recorded with their context. From the same real context, they hold still or replay another clip’s actions in manipulation, and turn left or right in indoor navigation (Appendix § 20).

• Long latents leave seams. At $\tau= 8$, rollouts put 58 to 83% more change at chunk boundaries than real video, against 8 to 47% at $\tau= 2$ (Appendix § 21).

Supplementary map.

• B Autoencoder implementation: encoder layers and bottleneck strides, decoder blocks, spatial AdaLN, gradient balancing, losses and progressive self-distillation (§ 3).

• C Autoencoder training data: the 23 video sources, about 76k hours, shared by every CADRE autoencoder and the MIRA recipe (Figure 8, Table 2).

• D World model: architecture, action conditioning and training recipe (Table 3), the streaming inference loop (Figure 11), and the timing behind every latency and frame rate of the paper (Table 4, Figure 10).

• E Inverse dynamics probes: probe architecture and training, the definition of ARR, and a control against shifted commands (Table 5).

• F Reconstruction: held-out reconstruction of every autoencoder (Table 6), and why we do not rank autoencoders by it (§ 4).

• G Full results: every autoencoder in every environment and metric, with one and two decoding steps (Table 7); the compression plane (Figure 12); and Figure 4 and Figure 7 in full (Figure 13, Figure 14 and Figure 15), the last with CADRE-4×24×24.

Category	Source	Share (%)	Hours
Web video	Panda-70M ((Chen et al., 2024b))	14.3	10,330
	LiveCC ((Chen et al., 2025a))	7.1	15,388
	LLaVA-Video-178K ((Zhang et al., 2025b))	7.1	2,744
	WebVid ((Bain et al., 2021))	7.1	1,506
Robot manipulation	RoboCOIN, AgiBot World, Open X-Embodiment, Robo-MIND, DreamDojo GR-1 and RoboCasa GR-1 ([1] (Wu et al. 2025b; O’Neill et al. 2024; Wu et al. 2025a; Gao et al. 2026a; Nasiriany et al. 2024))	8.9	5,387
	ABC, three cameras ([3])	5.4	10,790
	Galaxea Open-World ((Jiang et al., 2025))	5.4	1,390
	HIW-500 humanoid, head camera ((BitRobot et al., 2026))	1.8	268
Scene walkthroughs	DL3DV-10K ((Ling et al., 2024))	7.1	206
	RealEstate10K ((Zhou et al., 2018))	3.6	83
People	Kinetics-700 ((Carreira et al., 2019))	3.6	1,583
	CelebV-HQ and Hallo3 portraits ((Zhu et al., 2022; Cui et al., 2025))	3.6	240
	LiveSports-3K ((Chen et al., 2025a))	3.6	32
Egocentric, drones, sidewalk robots	Sekai walking, real and game ((Li et al., 2025b))	4.6	7,546
	Sekai drones, real and game ((Li et al., 2025b))	2.9	109
	FrodoBots-2K ((FrodoBots Lab, 2023))	1.8	2,415
Navigation (driving)	Physical AI AV, seven cameras ((NVIDIA, 2025))	5.4	6,019
Games	Rocket League ((Hu et al., 2026))	3.6	4,840
	OpenCS2 ((Blanchon, 2026))	1.4	5,084
Synthetic	Hallo3 generated portraits ((Cui et al., 2025))	1.8	6

Autoencoder	$\tau$	$t_{\mathrm{gen}}$ (ms)	$t_{\mathrm{dec}}$ (ms)	Latency (ms)	Max. fps
CADRE-1$\times$48$\times$48	1	56	22 / 44	128 / 150	12.8 / 10.0
CADRE-2$\times$32$\times$32	2	61	25 / 50	185 / 211	23.4 / 18.1
CADRE-4$\times$24$\times$24	4	65	26 / 51	291 / 317	44.0 / 34.3
CADRE-8$\times$16$\times$16	8	83	28 / 56	510 / 538	72.5 / 57.8
CADRE-1$\times$16$\times$16	1	106	22 / 45	178 / 200	7.8 / 6.7
CADRE-2$\times$16$\times$16	2	91	25 / 49	216 / 240	17.3 / 14.3
CADRE-4$\times$16$\times$16	4	86	26 / 51	311 / 337	36.0 / 29.2
MIRA recipe	2	61	14	175	26.6
Wan 2.2	4	85	32	317	34.2
Cosmos	8	198	7	605	39.0
LTX-2†	8	62	29	492	87.4
DC-AE	1	64	5	118	14.6

Table 5: Action recoverability and its control at frame 96. Gap is the raw $R^2$ against the commanded actions minus the control, against commands shifted by 48 frames. Compare gaps within an environment. CADRE rows use the two-step decode; the CADRE-1$\times$16$\times$16 world model is trained on robot manipulation only.

Autoencoder	Manipulation (robot) ARR	control	gap	Manipulation (human) ARR	control	gap	Navigation (driving) ARR	control	gap	Navigation (indoor) ARR	control	gap
1$\times\$48$\times$48$ | 0.48 | −0.99 | 1.46 | 0.64 | −0.54 | 1.07 | 0.90 | 0.48 | 0.35 | 0.87 | 0.10 | 0.65
2$\times\$32$\times$32$ | 0.45 | −0.97 | 1.40 | 0.75 | −0.58 | 1.19 | 0.89 | 0.47 | 0.37 | 0.91 | 0.08 | 0.70
4$\times\$24$\times$24$ | 0.49 | −0.84 | 1.31 | 0.65 | −0.49 | 1.03 | 0.67 | 0.40 | 0.24 | 0.45 | 0.01 | 0.38
8$\times\$16$\times$16$ | 0.72 | −0.73 | 1.40 | 0.63 | −0.50 | 1.03 | 0.86 | 0.51 | 0.29 | 0.79 | −0.01 | 0.69
1$\times\$16$\times$16$ | 0.74 | −0.89 | 1.58 | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A | N/A
2$\times\$16$\times$16$ | 0.73 | −0.82 | 1.49 | 0.73 | −0.53 | 1.13 | 0.97 | 0.49 | 0.41 | 0.90 | 0.05 | 0.73
4$\times\$16$\times$16$ | 0.79 | −0.77 | 1.50 | 0.78 | −0.53 | 1.18 | 0.88 | 0.44 | 0.38 | 0.93 | 0.08 | 0.72
MIRA recipe	0.39	−1.00	1.37	0.75	−0.58	1.19	0.86	0.43	0.38	0.92	0.03	0.76
Wan 2.2	0.85	−0.86	1.64	0.72	−0.53	1.12	0.66	0.49	0.13	0.34	0.03	0.26
Cosmos	0.78	−0.78	1.50	−0.27	−0.44	0.24	0.62	0.45	0.14	0.39	0.04	0.31
LTX-2	0.25	−0.80	1.05	−0.63	−0.62	0.12	0.57	0.52	0.04	0.04	−0.09	0.15
DC-AE	0.65	−0.94	1.55	0.39	−0.53	0.85	0.81	0.33	0.43	0.86	0.12	0.62

• H Long-horizon rollouts: 64 s rollouts on robot manipulation (Figure 17), their drift rates (Table 8) and examples (Figure 16).

• I Motion in the evaluation sets: how much each environment moves (Table 9), behind the camera-motion explanation of Section 5.2.

• J Qualitative results: counterfactual rollouts (Figure 22), and rollouts of every autoencoder on the same clip in each environment (Figure 18 to Figure 21).

• K Boundary excess: the seam metric and its values at $\tau= 2$, 4 and 8 (Table 10).

• L Extended related work.

Autoencoder	FID/s	FVD/s	PSNR loss (dB/s)	LPIPS/s ($\times10^{-3}$)
1$\times$48$\times$48	0.15	0.8	0.046	1.4
2$\times$32$\times$32	0.19	1.2	0.060	1.6
4$\times$24$\times$24	0.29	1.1	0.058	1.8
8$\times$16$\times$16	0.25	0.9	0.059	1.6
1$\times$16$\times$16	0.35	1.8	0.062	2.1
2$\times$16$\times$16	0.24	1.5	0.065	1.8
4$\times$16$\times$16	0.19	1.3	0.068	1.8
MIRA recipe	0.22	2.4	0.077	1.8
Wan 2.2	1.75	14.5	0.140	7.4
Cosmos	0.35	2.0	0.071	2.1
LTX-2	0.47	2.7	0.066	2.2
DC-AE	0.41	2.1	0.073	3.0

Environment	Pixel difference per frame	Pixels that change per frame	Patches that change per frame	Optical flow (pixels per frame)
Manipulation (robot)	0.010	2.6%	32%	0.24
Manipulation (human)	0.009	3.6%	33%	1.27
Navigation (driving)	0.017	8.2%	48%	2.13
Navigation (indoor)	0.032	15.7%	76%	2.69

Autoencoder implementation details

Encoder. DINOv3 ViT-B/16 (Simeoni et al., 2026) ($p = 16$) runs at $288 \times512$, and its features average layers 2 and 4 to 11 plus the last layer. The bottleneck convolution maps its 768 channels to 64. In time, its kernel equals its stride $\tau$ and it has no padding, so it is causal. In space, the $18 \times32$ grid is zero-padded to a multiple of the stride and cropped back in the decoder. Kernel and stride are 3 for $1 \times48 \times48$ ($6 \times11$ tokens), 2 for $2 \times32 \times32$ and 1 for $s = 16$. CADRE-$4 \times24 \times24$ uses kernel 5, stride 3 and padding 1, and reshapes $4 \times64$ output channels into a $2 \times2$ block of latents ($12 \times22$ tokens).

Decoder. It has 24 space-time blocks of width 1024 with 16 heads on a $16 \times16$ patch grid; the last 4 form the temporal tail at the native frame rate. Blocks use QK-normalization, gated attention, rotary embeddings in space and time, SwiGLU and LayerScale, with 4 spatial and 1 temporal register tokens. After the unpatchify, a refine head applies two residual blocks of dilated depthwise and causal temporal convolutions. The decoder has about 480M parameters.

Spatial AdaLN. AdaLN in DiT (Peebles & Xie, 2023) modulates all tokens of a sample with one vector. Like SPADE (Park et al., 2019), SEAN (Zhu et al., 2020), SCAM (Dufour et al., 2022) and PixelDiT (Yu et al., 2026), spatial AdaLN modulates each token $h_i$ from the latent $z_i$ at its position. With $c$ the embedding of the flow time $t$ and jump $d$,

It is applied before space attention, time attention and the MLP, with separate scales and shifts. One head is shared by all blocks, with a learned per-block offset, and is zero-initialized.

Gradient balancing. Each weight in Equation (6) is set at every step from gradient norms, as in the adaptive weight of VQGAN (Esser et al., 2021):

where $W$ is the weight of the head’s last layer, $\ell_1$ the pixel term and $\ell_i$ an auxiliary term, with $\delta= 10^{-6}$ and weights clipped at $10^4$.

Losses. The reconstruction term of Equation (2) is

where $\ell_{\mathrm{dino}}$ is the squared distance between normalized DINO features and $\ell_{\mathrm{dwt}}$ the $L_1$ distance between two-level 3D Haar coefficients, which weights the high frequencies that pixel losses underweight (Degeorge et al., 2026). LPIPS and DINO use a random quarter of the frames. The wavelet term has an extra factor of 0.25 and weights its low-pass band by 0.25.

Progressive self-distillation. $\mathcal{L}_{\mathrm{psd}}$ has weight 1 on a random quarter of steps. For each latent frame, $r$ and $t$ are two sorted uniform draws, and both half jumps use the current weights without gradients. The reconstruction and perceptual losses supervise only $d = 0$. Table 3 gives the training recipe.

Released autoencoders. We also release CADRE-$2 \times32 \times32$, $2 \times16 \times16$, $4 \times16 \times16$ and $8 \times16 \times16$ trained beyond our experiments: to 360k steps, with the learning rate decayed from $2 \times10^{-4}$ to $2 \times10^{-5}$ over the last 70k, then for 40k steps on clips of five resolutions from $256 \times448$ to $384 \times640$. Our results and world models use the 150k-step versions.

Autoencoder training data

Every CADRE autoencoder and the MIRA-recipe baseline train on the same 23 video sources, about 76k hours (Figure 8, Table 2). A sample is a random 16-frame window from a video of a randomly drawn source, resized to cover $288 \times512$, center-cropped and resampled to 20 frames per second. Videos shorter than 16 frames or below 256 pixels on the short side are dropped.

World model architecture and training

The world model follows MIRA (Hu et al., 2026), and Figure 9 is redrawn from theirs. Table 3 lists its hyperparameters.

Blocks. Each block runs bidirectional space attention within a latent frame, then a SwiGLU MLP; blocks 0, 4, 8, 12 and 15 add causal time attention in between. Attention uses 16 query and 4 key/value heads, QK-normalization, a gate and rotary embeddings ($2\mathrm{D}$ in space, $1\mathrm{D}$ in time). Blocks are pre-norm, without LayerScale or registers.

Conditioning. Each latent frame is conditioned on its flow time and on the $\tau$ actions that produced it, action $a_j$ producing frame $j + 1$. Actions are normalized per environment; their per-frame embeddings are concatenated, projected and passed through an MLP. The flow time gets a sinusoidal embedding and an MLP, and the sum of both embeddings drives AdaLN (Peebles & Xie, 2023) before space attention, time attention and the MLP of every block. In human manipulation, each hand contributes the frame-to-frame change in wrist position (3), wrist orientation (3) and 15 finger-joint rotations (45).

Objective and recipe. Under diffusion forcing (Chen et al., 2024a), each latent frame draws its own flow time $t_i$, and the model regresses the velocity $z_1 - z_0$ from $z_{t_i} = t_i z_1 + (1 - t_i)z_0$ (Figure 9). PSD-U, the progressive self-distillation of Boffi et al. (2025) with a uniformly sampled intermediate time, is added on a fixed quarter of every batch. The recipe (Table 3) is the same for every autoencoder; only the action encoder follows the environment. The world model does not patch the latent further, except in Section 5.1.

Inference. The context is encoded once. At each chunk boundary, the world model reads the previous latent frame and the $\tau$ actions entered during the chunk and predicts the next latent frame in 10 denoising steps. The frame is decoded for display, re-noised to level 0.2 and cached (Figure 11). Since $a_t$ produces frame $t + 1$, an action first shows $\tau$ frames after it is entered. This timing comes from supervision: the architecture only keeps an action from changing frames already shown, and within a latent frame each decoded frame can depend on all $\tau$ actions, so frame 5 reflects $a_4$ and earlier actions such as $a_3$.

Latency. Figure 4b and Figure 14 plot the action latency of Equation (1) for the first action of a chunk, the worst case, at $f = 20$ frames per second. We time $t_{\mathrm{gen}}$, one world model call with its 10 denoising steps and cache pass, and $t_{\mathrm{dec}}$, the decode of one new latent frame, on one H100 at batch 1 in bf16 with torch.compile, averaged over 32 latent frames after warm-up on the robot-manipulation world models (Table 4). Each decoder runs its most efficient streaming decode: causal decoders decode only the new latent and reuse the context (exactly for Wan 2.2; for CADRE, MIRA and Cosmos we time the decode of the new latent, the cost of a decoder cache), DC-AE decodes each frame alone, and the non-causal LTX-2 re-decodes an 8-latent window. The rollouts of Table 7 are decoded over each decoder’s trained window. The resulting maximum frame rate, $\tau/(t_{\mathrm{gen}} + t_{\mathrm{dec}})$, differs from the displayed rate, which is always 20 frames per second; it only tells whether the world model runs fluidly. Every CADRE autoencoder with $\tau\ge4$ does, as does CADRE-2×32×32 with one step; those with $\tau= 1$ reach only 7 to 13 frames per second. Raising $\tau$ raises both the action latency and the maximum frame rate (Figure 1). With fewer denoising steps, every CADRE autoencoder with $\tau\ge2$ runs fluidly, while at $\tau= 1$ this needs one decoding step and at most four denoising steps (Figure 10).

Inverse dynamics probes for action recoverability

A frozen probe, one per environment, reads the actions back from decoded frames, so it scores every autoencoder in the same space.

Architecture. The probe maps a window of 8 frames to one action per frame. A frozen DINOv3 ViT-B/16 (Siméoni et al., 2026) gives patch features from layers 2, 5, 8 and 11, combined by a learned softmax. Six factorized space-time blocks with bidirectional attention follow, then a per-frame attention pool and a two-layer head (27M trained parameters).

Training. The probe trains on real clips from the world models’ training split with a smooth $L_1$ loss on standardized actions and photometric augmentation only. It uses AdamW (learning rate $3\times10^{-4}$, batch 384) with cosine decay over 60k steps; we keep the 55k-step checkpoint (60k for human manipulation).

Score. For each action dimension $d$, $R_d^2$ compares the recovered actions with the commanded ones over the last 16 generated frames of all clips, and ARR averages it normalized by the probe’s $R^2$ on the real continuation,

On real video, the probe reaches $R^2$ of 0.91 in robot manipulation, 0.93 in driving and 0.83 in indoor navigation. Human manipulation has 102 action dimensions, and the probe barely reads the 90 finger-joint dimensions at $288\times512$, so $D$ holds only the 12 wrist dimensions there (real-video $R^2$ 0.82).

Control. We also score the recovered actions against the clip’s own commands shifted by 48 frames, which the rollout never received. A world model that ignores its actions scores alike on both, so the gap between the two raw $R^2$ values measures action following (Table 5). Every CADRE autoencoder keeps a clear gap in every environment. In driving, the gap of Wan 2.2, Cosmos and LTX-2 is small (0.04 to 0.14), and so is that of Cosmos and LTX-2 in human manipulation (0.24 and 0.12): their rollouts reproduce motion common to the environment, such as forward travel in driving.

Reconstruction

Table 6 reports how well each autoencoder reconstructs held-out general video. Reconstruction does not predict rollout quality. Across autoencoders, the rank correlation between reconstruction and one-step rollout FVD is +0.76 in robot manipulation but −0.39 in driving and −0.35 in indoor navigation. Wan 2.2 reconstructs best by far, yet its world model ranks fifth of twelve on rollout FVD in robot manipulation, second to last in driving and last in indoor navigation (Table 7).

Full results

Table 7 lists every autoencoder in every environment with both decoding step counts. Figure 12 places the autoencoders in the compression plane, and Figure 14 extends Figure 1 and Figure 4b (driving and robot manipulation) to every environment and metric.

Figure 15 extends Figure 7 to all environments and adds CADRE-4×24×24, whose overlapping kernel confounds it; it is an outlier in navigation (FVD 403 in driving, 435 indoors). We hypothesize that its channel-to-space reshape, which turns each position into a correlated 2$\times$2 block of latents, is harder to predict when the whole view moves.

Along the green curves, which vary $\tau$ at $s = 16$, FVD stays between 39 and 46 in robot manipulation and ARR between 0.72 and 0.79. In navigation, FVD rises from $\tau= 2$ to $\tau= 8$ by 31% in driving, where the rise already appears at $\tau= 4$, and by 64% in indoor navigation, where it appears only at $\tau= 8$; ARR drops by 0.11 in driving and by up to 0.15 in indoor navigation. At the same $\tau$, coarser spatial tokens cost far more in robot manipulation: at $\tau= 2$, going from $s = 16$ to $s = 32$ multiplies FVD by 2.4 and lowers ARR from 0.73 to 0.45, with almost no change in FID, whereas in navigation FVD rises by only 11 to 12% and ARR changes by at most 0.08. At $\tau= 1$, FVD grows more between frames 32 and 96 than at $\tau= 2$ in four of five comparisons, by 11 to 113%, a confounded sign of error accumulation.

Long-horizon rollouts

To test whether the rankings hold beyond 4.8 s, we roll out the world models of all twelve autoencoders on robot manipulation for 64 s (1280 frames, 256 clips, one-step decode; Figure 17), the only environment with many episodes long enough for such rollouts. No metric collapses. CADRE-$8 \times16 \times16$ is best from about 13 s on and ends with the lowest LPIPS and FVD and the highest PSNR, while the $s = 16$ autoencoders lose their early lead in LPIPS and PSNR by then. Action recoverability barely decays. The other autoencoders drift faster (Table 8): FVD grows by 0.8 to 1.8 per second for the CADRE autoencoders, 2.0 to 2.7 for the others and 14.5 for Wan 2.2, which starts best but whose PSNR falls below every CADRE autoencoder by about 9 s. Drift rates are least-squares slopes between 4.8 and 64 s; the upward bias of Fréchet distances at 256 clips is roughly constant over time, so slopes stay comparable. The non-causal LTX-2 decoder uses a 57-frame (8-latent) history window here. Figure 16 shows one rollout per autoencoder; there, Wan 2.2 corrupts its colors by 48 s.

Motion in the evaluation sets

Table 9 measures motion on the real continuation of every evaluation clip: the mean frame-to-frame pixel difference, the share of pixels and of $32 \times32$ patches that change by more than 0.05, and the RAFT ((Teed & Deng, 2020)) optical flow in pixels per frame.

In manipulation, a third of the image changes per frame and content moves by about one pixel per frame or less, less than one 16-pixel token over eight frames. In navigation, half to three quarters of the image changes and content moves by more than two pixels per frame, more than a token over eight frames. The head-mounted camera of human manipulation shifts the whole image slightly, which optical flow captures and pixel differences miss.

Qualitative results

Figure 22 shows counterfactual rollouts: from the same real context, the world model follows the recorded actions and two action sequences never recorded with it. Rollouts under recorded actions are in Figure 1 and Figure 5. Figure 18 to Figure 21 compare every autoencoder on the same clip in each environment, with its recorded actions.

Boundary excess

Let $d_t$ be the mean absolute difference between generated frames $t$ and $t-1$, with pixels in $[0,1]$. $E$ is the ratio of the mean $d_t$ over boundary pairs ($t$ a multiple of $\tau$) to that over the other pairs, and the boundary excess is $E/E_{\mathrm{real}}$, with $E_{\mathrm{real}}$ from the real continuation at the same indices. One means no seam. We compute it on 32 clips per autoencoder with the one-step decode; repeated draws differ by about 0.15. At $\tau=8$, rollouts put 58 to 83% more change at chunk boundaries than real video (Table 10).

Extended related work

We study how much the autoencoder of an interactive world model compresses space and time, and whether that compression belongs in the autoencoder or in the world model.

World models, from offline to streaming. A world model predicts how an environment evolves under actions. Early ones predict compact latents for control (Ha & Schmidhuber, 2018; Hafner et al., 2020; Micheli et al., 2023); most recent ones generate the video itself (Bai et al., 2024), for driving (Gao et al., 2024; Russell et al., 2025; Carrión et al., 2025), robotics (Yang et al., 2024), egocentric motion (Bai et al., 2025; Wang et al., 2026a) and open scenes (Bruce et al., 2024; Mao et al., 2025; PAN Team et al., 2025; Zhu et al., 2026b). Most are offline, taking seconds to minutes per clip. They compare autoencoders only in that regime: SANA-WM (Zhu et al., 2026a) swaps the Wan 2.1 autoencoder for LTX-2’s. An interactive world model streams instead and must render each frame before the next action arrives. Some reach real time by rendering directly from controls (Kim et al., 2020; Menapace et al., 2021) or with dedicated models [4] (Valevski et al. 2025; Bar et al. 2025; Ramanana Rahary et al. 2026a). Others adapt a large video generator by streaming distillation (Yin et al., 2025; Huang et al., 2025), as Matrix-Game 2.0 (He et al., 2025) and Hunyuan-GameCraft (Li et al., 2025a) do. They inherit its autoencoder unchanged, at $8 \times$ in space and $4 \times$ in time. Their blockwise generation also adds serial computation (Chao et al., 2026). MineWorld (Guo et al., 2025) uses discrete tokens instead. Once generation is fast, the decoder becomes the bottleneck, which Matrix-Game 3.0 (Wang et al., 2026b) and WorldPlay (Sun et al., 2026) prune. MIRA (Hu et al., 2026) instead trains its own autoencoder and judges it by the world model; we extend that principle along both compression axes.

Diffusion and flow-based generation. Both the world model and the decoder of CADRE are diffusion models trained with flow matching. Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song & Ermon, 2019; Song et al., 2021b) and their refinements of sampler, schedule, loss weighting and preconditioning (Song et al., 2021a; Lu et al., 2022; Nichol & Dhariwal, 2021; Kingma et al., 2021; Kingma & Gao, 2023; Hang et al., 2023; Karras et al., 2022; 2024b) converge in flow matching [2] (Lipman et al. 2023; Liu et al. 2023), which regresses a velocity along an interpolant between noise and data and scales with transformers (Bao et al., 2023; Peebles & Xie, 2023; Ma et al., 2024; Esser et al., 2024). Three of its design choices interact with compression. The first is the prediction target. Noise and velocity prediction (Salimans & Ho, 2022) degrade when each token is high-dimensional, and clean-sample prediction does not (Li & He, 2026). Our decoder outputs large pixel patches, so it predicts the clean sample, and its wavelet loss, related to the frequency balancing of Degeorge et al. (2026), emphasizes the fine texture it must restore. The world model’s tokens keep 64 channels at every compression, so it keeps velocity prediction. The second is the noise schedule, whose optimum shifts with the number of tokens (Chen, 2023; Hoogeboom et al., 2025): RAE (Zheng et al., 2026) generalizes the shift of Esser et al. (2024) to the number of tokens times their channels. Our compressions change the number of tokens by more than an order of magnitude, so our tuning shifts the schedule with each autoencoder’s latent size and token count. The third is the representation. Aligning a generator’s features to a self-supervised encoder speeds up its training (Yu et al., 2025), as masking does (Gao et al., 2023); we place that prior in the autoencoder instead, and the world model generates in the features of a frozen DINOv3.

Conditioning. The world model is conditioned on actions, and how faithfully it follows them is one of our three axes. Image generators usually strengthen adherence to their condition with guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2022; Kynkäänniemi et al., 2024; Karras et al., 2024a); we use none, so action following reflects the latent and the model alone. A condition can also be weak, as with noisy web captions (Dai et al., 2023; Podell et al., 2024), and some action dimensions barely change the image, such as the finger joints in human manipulation, which our probe cannot read even on real video. We therefore measure action following directly, by how well an inverse dynamics probe recovers the actions from the generated frames Appendix § 15.

Evaluation. We judge each autoencoder by the world model it yields, with FID (Heusel et al., 2017) and FVD (Unterthiner et al., 2019). These metrics are fragile. FID mixes fidelity and diversity (Kynkäänniemi et al., 2019), depends on resizing and ImageNet features (Parmar et al., 2022; Kynkäänniemi et al., 2023), can disagree with human judgment (Stein et al., 2023; Jayasumana et al., 2024), and moves more when the model is retrained with another seed than when it is resampled (Dufour et al., 2026).

Video and causal generation. Video diffusion moved from pixel space (Ho et al., 2022) to latents, first by inflating image networks (Blattmann et al., 2023) and then with spatio-temporal transformers (Yang et al., 2025b; Kong et al., 2024; Polyak et al., 2024; Wan et al., 2025), which denoise a whole clip at one noise level. Causal generation breaks that symmetry. Rolling diffusion (Ruhe et al., 2024) gives later frames more noise, and diffusion forcing (Chen et al., 2024a) draws one noise level per frame, so the next frame is denoised from a clean or partly noisy past. History guidance (Song et al., 2025) builds on it, and MAGI-1 (Sand.ai et al., 2025) scales chunk-wise autoregression to 24B parameters. Noise on the context (Valevski et al., 2025) and training on the model’s own rollouts (Huang et al., 2025) curb drift over long horizons, and FramePack (Zhang et al., 2025a) compresses the history to bound its cost. Discrete alternatives predict quantized tokens autoregressively (Yu et al., 2024; Bruce et al., 2024; Wu et al., 2024) or by masked prediction (Chang et al., 2022), and MAR (Li et al., 2024) keeps autoregression but replaces the quantizer with a per-token diffusion loss. Our world model combines flow matching, diffusion forcing and context noise, and stays fixed while the autoencoder varies.

The autoencoder as an object of study. In latent diffusion (Rombach et al., 2022; Esser et al., 2021), generation quality depends on the structure of the latent. A line of work improves it at a fixed generator (§ 4). Representation autoencoders go further and generate in the space of a frozen self-supervised encoder (Zheng et al., 2026; Singh et al., 2026), for text-to-image (Tong et al., 2026), low-dimensional latents (Gao et al., 2026b; Shi et al., 2026; Bi et al., 2026) and video (Xie et al., 2026; Guo et al., 2026; Hu et al., 2026), on encoders such as DINO (Caron et al., 2021; Oquab et al., 2024; Simeoni et al., 2026) and V-JEPA 2 (Assran et al., 2025; Mur-Labadia et al., 2026). Distilling such an encoder into the latent, as in GAIA (Hu et al., 2023; Russell et al., 2025) and Mimi (Défossez et al., 2024), follows the same idea and makes latents easier to diffuse. These works use one compression rate; our representation autoencoders vary it in space and time, one trained autoencoder per compression point.

Video autoencoders and strong compression. Video autoencoders use causal 3D convolutions (Yu et al., 2024). Most settle at 8× in space and 4× in time (Yang et al., 2025b; Wan et al., 2025; Kong et al., 2024). Others compress harder: LTX-Video (HaCohen et al., 2024), DC-AE (Chen et al., 2025b), DC-VideoGen (Chen et al., 2025c), H3AE (Wu et al., 2025c), ProMAG (Mahapatra et al., 2025) and DeltaTok (Kerssies et al., 2026), down to one token per frame. Comparisons across rates use reconstruction (Tang et al., 2024; NVIDIA et al., 2025) or text-to-video generation (Hansen-Estruch et al., 2025). The two rates usually move together. We vary them independently and judge each by an action-conditioned world model.

Diffusion decoders and single-step generation. Pixel and perceptual losses alone blur, so autoencoders add an adversarial loss (Esser et al., 2021), at the price of a sensitive discriminator. Diffusion decoders are the alternative (Preechakul et al., 2022; Pandey et al., 2022; Pernías et al., 2024), and scale as tokenizers with a single flow-matching loss (Chen et al., 2025d), but decode in many steps. Single-step diffusion decoders so far need a second stage or an adversarial term (Vallayers et al., 2026; Sargent et al., 2025; Zhao et al., 2025); video diffusion decoders keep the adversarial loss (HaCohen et al., 2024) or several steps (Yang et al., 2025a; Teng et al., 2026). Few-step generation itself comes from distillation (Salimans & Ho, 2022; Luo et al., 2023; Sauer et al., 2024) and consistency models (Song et al., 2023; Lu & Song, 2025); shortcut models (Frans et al., 2025), flow maps (Boffi et al., 2025) and MeanFlow (Geng et al., 2025) learn it in one stage from their own predictions. We bring that one-stage self-distillation to a causal decoder with no discriminator.

Where compression lives. A generator can also compress through its patch size. DiT (Peebles & Xie, 2023) improves as the patch shrinks, pixel-space transformers (Li & Kwon, 2026; Hoogeboom et al., 2025) replace the autoencoder with a large patch, W.A.L.T. (Gupta et al., 2024) and Latte (Ma et al., 2025) find temporal patching costliest, and FlexiDiT [5] varies the patch per step. For images, DC-AE (Chen et al., 2025b) finds that compressing in the autoencoder beats patching at matched tokens. We make that comparison for a video world model.

Acknowledgments

We thank Hedi Ben-Younes, Moritz Böhle, Andreas Geiger, Alyx Liao, Renaud Marlet, Juliette Marrie, Manu Orsini, Amélie Royer, Nicolas Sereyol-Garros and Václav Volhej n for interesting discussions.