WorksComputer Science
Self-Improving Video-to-Action Models
Abstract
Video generative models are an emerging tool for embodied agents: they can serve as scalable visual planners, generating videos that depict how to complete previously unseen tasks. However, executing these plans typically requires a video-to-action or "inverse dynamics" model trained on large, human-collected video-action datasets, severely limiting scalability. We introduce Mirror, a self-supervised approach that learns video-to-action mappings through environment interactions, without prior action data or hand-crafted rewards. Mirror learns by attempting to follow a visual plan by executing its best guess of the underlying actions in the environment. Mirror then trains on the resulting video-action pairs, incrementally self-improving and following the visual planner more closely. Across Atari, OGBench, and DeepMind Control, Mirror outperforms leading learning-from-observation methods with learned video plans, and achieves near-expert performance with oracle plans. On Minecraft BASALT, it outperforms methods with access to expert actions or hand-engineered task knowledge. In Minecraft survival, it is the first method to acquire an iron pickaxe through self-improvement alone, without rewards or expert action labels. In the offline setting, with expert action labels but no environment interaction, it sets a new state of the art in obtaining diamonds. Our results suggest that advances in video models can translate into increasingly capable agents without a corresponding need for hand-crafted rewards or large collections of expert action data.
Introduction
Video models are emerging as scalable learners of the physical world. Trained on vast collections of unstructured web video, they absorb knowledge about objects, environments, human behavior, and how tasks unfold over time. Recent work has begun to harness this knowledge for robot control by using video models as visual planners (e.g., [11, 8, 67, 48]). Given past frames and optional task conditioning, these models generate a video depicting how the task could be completed. An embodiment-specific video-to-action (V2A) or “inverse dynamics” model then translates the generated visual trajectory into robot actions. This has yielded impressive generalization across unseen tasks, objects, environments, and embodiments.
However, training the V2A model still requires significant amounts of expert-collected, paired video-action data from oracle motion planners or human teleoperation. Moreover, agents cannot easily generate such video-action data by rolling out random policies in the environment, as critical environment transitions—such as executing a robot grasp or crafting a diamond pickaxe in Minecraft—are unreachable by a random policy. Today, video planners thus inherit the same scalability bottleneck as video-language-action models: a reliance on expensive expert teleoperation data.
In this paper, we introduce Mirror, a fully self-supervised method for learning to execute the policy encoded by a video model only by interacting with the world, without any expert video-action data. We assume access to an expert video planner that, given the current agent observation, can generate an expert-level visual plan. Under this assumption, we show that no further expert action supervision or hand-crafted rewards are necessary to achieve expert-level performance. Mirror thus represents a bet on the continued scaling of video models: as visual planners improve, their capabilities can be translated into actions without a corresponding increase in expert action data.
Our key idea is to let a randomly initialized video-to-action model self-improve against the expert video planner. Given the current agent state, we query the planner to generate a video plan. The current video-to-action model then predicts the corresponding actions and the agent executes them in the environment. Interactions produce new paired video-action data, which is used to improve the video-to-action model. Critically, this data is not random but concentrated at the frontier of the agent’s current capability to follow the video plan. Repeating this process allows the agent to follow progressively more of the plan until it faithfully reproduces the planner’s behavior. Mirror does not require rewards, privileged state information, verifiers, or trajectory filtering. Furthermore, we show that our self-improving approach can outperform training on expert actions.
We evaluate Mirror in two settings: (i) the Minecraft survival challenge [17] and Minecraft BASALT competition tasks [52]; and (ii) standard decision-making benchmarks, including Atari [4], OGBench [46], and DeepMind Control [54]. To the best of our knowledge, no prior method without either hand-crafted rewards or expert action labels achieves non-zero success on these Minecraft tasks. Mirror outperforms baselines, including those with privileged information, and further exploits the emergent in-context learning of large video models, improving faster than a baseline that cannot leverage context across attempts.
Our contributions are summarized as follows:
Mirror, a scalable agent that learns long-horizon tasks by self-improving against a pre-trained video planner, without task rewards or expert video-action data.
A heavily adapted 14B-parameter NVIDIA Cosmos3 video planner with efficient long-context planning via causal KV-caching. Code and weights will be available soon on our website.
Mirror is the first method to achieve non-zero success rates on the BASALT Minecraft suite without expert video-action data or hard-coded actions.
In Minecraft survival, Mirror is the first method to obtain an iron pickaxe through self-improvement alone, using neither rewards nor expert action labels. Trained offline with action labels, it sets a new state of the art, obtaining diamonds nearly eight times as often as Dreamer V4.
On a suite of 24 common RL tasks, Mirror outperforms baselines using a generative video planner, and, when paired with an oracle video planner, nearly matches expert returns.
Related Work
Video planners. A nascent line of work leverages video generative models to generate a visual plan of how a given task should be solved, and then translates it into actions [11, 25, 29, 36, 5]. As video models can be pre-trained on web-scale data, robots can inherit impressive generalization capabilities [8, 28]. However, translating videos into robot actions generally requires training on paired video-action data [11, 56, 44]. Our agent learns to act from video and self-play in an interactive environment alone, reaching imagined futures without a single action label.
Learning from demonstrations. We focus on imitation from observation, where an agent learns from expert observations without action labels or task rewards [58]. Adversarial approaches match expert observation distributions through learned discriminator rewards [59, 33, 16], but can be unstable and sensitive to hyperparameters [22]. Iterative behavioral cloning alternates between training a V2A model on the agent’s own experience and imitating V2A-labeled demonstrations [58], but collecting informative transitions remains an exploration challenge on long-horizon tasks [50]. ASH [51] applied a similar recipe to long-horizon games such as Pokémon and Zelda, training an inverse dynamics model on the agent’s own trajectories to pseudo-label internet gameplay videos. Latent-action approaches such as LAPO and LAPA still ground their latent actions with action labels or rewards [50, 66]. Mirror instead uses a video planner to guide exploration, collecting V2A training data without action labels, rewards, or adversarial optimization.
Minecraft agents. Minecraft has become a standard benchmark for long-horizon decision making, ranging from the fuzzy objectives of BASALT [52] to the canonical MineRL challenges [17]. Existing agents rely on some combination of expert action demonstrations and hand-designed rewards. For example, VPT [3] trains a behavioral cloning (BC) policy using about 2,000 hours of action-labeled gameplay recordings from paid contractors, then fine-tunes the policy with about 230,000 hours of online RL. While Dreamer V3 [18] trains a Minecraft RL agent with only 1,400 hours of online RL, they simplify parts of the Minecraft action space and still require hand-designed reward. Dreamer V4 [19] likewise uses reward, and also requires expert actions to train a video world model. Pan-4B [45] trains a goal-conditioned RL policy on 500k hours of internet video, yet still post-trains on 2,000 hours of expert action labels. To make long-horizon tasks more tractable, many works improve upon Minecraft policies with language models. MineDojo [14] trains PPO agents on shaped rewards and a learned video-language critic, while STEVE-1 [31], DEPS [62], and JARVIS-1 [63] use language-based planning.
Prior agents that solve long-horizon tasks thus depend on hand-crafted rewards, expert video-action data, or both; Mirror instead only assumes access to an expert-level video planner.
Method
We seek a task-conditioned policy , which predicts actions based on past observations and a task . We model indirectly with two components: a video planner (VP) and a video-to-action (V2A) model2. The VP predicts future observations based on historical observations and a task , while the V2A model predicts actions given the combined video sequence . Sequentially sampling from the two models yields the agent’s policy:
Crucially, every plan is conditioned on the frames the agent actually observed, never on its previous plans. The VP therefore always plans from wherever the agent ended up, so deviations from earlier plans can be corrected. For the same reason, errors do not compound over time, which removes the need for anti-drift techniques against exposure bias in autoregressive video generation [7, 23, 71].
This factorization differs from the world-model (WM) and policy decomposition used by VLAs and world-model agents [19], although both model the same joint distribution:
Algorithm 1 Learning without action labels: VP-guided self-training of the V2A model
Require: video planner , environment , exploration distribution , horizon .
Initialize V2A model with parameters from scratch; replay buffer .
Repeat until convergence:
Data collection episode
Reset ; observe ; initialize and
while do
and
Execute in and observe
and
end while
V2A model learning
Sample trajectories and timesteps from each trajectory
The two decompositions differ in where the difficult reasoning is learned. Long-horizon tasks require both procedural knowledge of what to do and physical knowledge of how the world responds. In the WM + policy decomposition, the policy must acquire the procedural knowledge and the WM the action-conditioned physical knowledge, both from scarce, embodiment-specific action data. In the VP + V2A decomposition, the VP learns both from abundant action-free video, which enables web-scale pretraining. This leaves the V2A model a much easier problem: a task-agnostic mapping from observed transitions to actions. It therefore needs far less embodiment-specific data, such as action labels. The VP + V2A decomposition is thus considerably more scalable, as our experiments (§ 6.2) and recent results in robot control (Li et al., [28]) show.
Our method, Mirror, reveals a far more striking consequence of this decomposition: given a capable VP and the ability to interact with the environment, the V2A model needs no action labels at all. Because it is task-agnostic, it can learn from any transition the agent itself produces, while the VP steers the agent toward the transitions the task requires. § 3.1 shows how the V2A model can be trained from scratch on the data the agent collects while following these plans, and § 4 describes how we make a generative video model VP practical for an online agent.
Self-Improving Video-to-Action Models
Our key insight is to use a pretrained video planner to guide data collection for training the V2A model. The model learns from the agent’s own rollouts, without expert action labels. As summarized in Algorithm 1, training alternates between data collection and model updates. During data collection, we sample an action from the agent’s two-stage policy and add exploration noise . We execute the resulting action in the environment and store the transition in a replay buffer. To update the model, we sample trajectories from the buffer and train the V2A model to predict from . Because the targets are conditioned on the achieved observation rather than the planned one , every transition is a valid training example, even when execution deviates from the plan. Further details can be found in Algorithm 1.
Exploration via stochastic V2A. Figure 1 illustrates how this process guides exploration. Early in training, the V2A model often fails to follow the plan, so the realized next observation may differ substantially from the video plan . However, even these failed attempts provide useful training examples: the V2A model still learns to predict the action that produced the observed transition. As the model improves, the agent can follow the planner more closely and reach later stages of the task. In turn, these rollouts provide data to improve the V2A model at the newly reached stages. Repeating this cycle allows the agent to progress ever further through the task. In contrast, purely random exploration often struggles to reach these stages because they require long sequences of consistently coordinated actions.
Figure 1. Enabling long-horizon exploration with a video planner. Left: Random exploration struggles to reach later stages of long-horizon tasks that require specific sequences of actions. Right: The video planner guides exploration. As the V2A model learns from achieved transitions, the agent follows the plan more accurately and reaches progressively later stages of the task. These new transitions further improve the model, eventually enabling the agent to execute the full plan.
If the V2A model were trained only on data from its own rollouts, it would collapse to predicting the same action regardless of the observations. We hence make the V2A model stochastic during data collection by adding noise to its predictions. Stochasticity also plays a second, central role: carrying the agent past the stages it cannot yet execute (Figure 2). Where the V2A model is already competent, action noise is harmless: while the agent may drift off the currently active video plan, the video planner re-plans from the state actually reached, preventing error accumulation and allowing the agent to progress. At a milestone the V2A model has not yet learned, such as the first time a block has to be mined, the V2A model predicts the wrong action, and the agent thus fails to produce the planned transition. The planner, however, continues re-planning, repeatedly proposing the same transition. A deterministic agent would then be blocked at this milestone, as it would have never seen the action that would lead to this correct transition. The stochastic V2A model is what overcomes the roadblock: eventually, the correct random action is sampled, and the resulting transition enters the replay buffer (where it is then learned by the V2A model), enabling the agent to complete the milestone in future episodes. Random exploration, in contrast, would first have to stumble upon the entire action sequence leading to the milestone before it could even attempt the critical action, see Figure 1.
Figure 2. Re-planning holds the agent at the milestone until noise supplies the critical action.
A Video Planner for Online Learning and Closed-Loop Control
Following plans in closed-loop places three demands on the VP. First, it must re-plan often and cheaply. Second, it must run over long episodes, often well over a minute in our Minecraft benchmarks. Third, it must stay on task even when the agent’s failures push it off the distribution of expert videos. To avoid re-encoding the history at every re-plan, we use a block-causal architecture: each observed frame is encoded once to cache its keys and values while the plan is denoised bidirectionally against this cache. Sliding-window attention over a fixed-size cache keeps the cost per plan constant, so a planner trained on 32 s clips can run over 5 minutes of context without degradation. Finally, we propose task-aware guidance, which leverages diffusion guidance ([21, 24]) to guide the task-conditional video planner with a task-unconditional one, significantly improving the planner’s task adherence. Figure 3 and Appendix § 9 detail the architecture, training, and inference.
Figure 3. The causal video planner. (a) Tokenized frames and text are fed into the 14B-parameter Cosmos3-Nano ([43]) model, forming a causal stream of contexts whose keys and values are cached and reused. The denoiser generates future frames while cross-attending to the context KV cache, so each observation and instruction is encoded only once. (b) Attention mask. Video history is encoded block-causally. The future plan attends to this history and bidirectionally to itself. Sliding-window attention lets the planner generalize beyond the window while keeping the KV cache within a constant budget.
Experiments
We empirically validate Mirror’s ability to solve tasks across Minecraft (§ 6) and traditional RL environments (§ 7.1). In § 7, we study what emergent abilities generative video planners can provide with scale and how to leverage them. In § 7.1, we investigate how our method generalizes to various environments and control settings. Please find video results on our website.
Solving Long-Horizon Challenges in Minecraft
Mirror solves long-horizon challenges in Minecraft without reward supervision or expert actions. On Minecraft, we pre-train a video planner on 2,946 hours of videos from the VPT dataset (Baker et al., [3]), equipping our video planner with general physical and procedural knowledge of Minecraft. The VP is finetuned further for BASALT and Minecraft survival mode. With this VP, we are able to train our self-improving V2A model from scratch with self-play. Our agent operates in the MineRL 1.0 (Guss et al., [17]) action space: at every step (20 Hz), it chooses among over 4,000 distinct combinations of 20 keyboard buttons while moving the mouse continuously in two dimensions. We do not simplify the action space or game mechanics.
Learning to Solve Fuzzy Tasks from Human Demonstration
We first evaluate Mirror on the Minecraft BASALT (Benchmark for Agents that Solve Almost-Lifelike Tasks) (Shah et al., [52]) challenge. Unlike traditional RL environments, its task objectives are intentionally fuzzy and hard-to-verify. For instance, consider the MakeWaterfall task:
Place a water bucket to make a beautiful waterfall. Then take an aesthetic picture of it by moving to a good location, positioning the player to have a nice view of the waterfall.
Our method is particularly suited to this kind of task where reward specification is difficult. It learns directly from fuzzy human demonstration videos without any reward supervision. For each task, we finetune our video planner on the corresponding BASALT human video demonstrations. We then train our V2A model from scratch only via self-play. After only 5 hours (360,000 environment steps) of in-game play, our method is already able to outperform the baselines. Figure 4 demonstrates how Mirror solves these tasks. Using the same VP, we also evaluate a V2A model pretrained on expert actions from the VPT dataset.
Figure 4. Learning from human demonstration. Mirror can complete a variety of open-ended BASALT tasks from self-play without any expert action labels or reward supervision. (a) The VP inherits long-horizon planning capabilities learned from video pre-training, while the V2A model learns to ground the video plans to action via online interaction. (b) The constructed waterfalls are diverse and similar to human creations. (c) Mirror achieves near-human performance on FindCave.
In Table 1, we compare our method against an action free baseline, VIPER (Escontrela et al., [12]), a behavioral cloning (BC) policy fine-tuned from VPT, action-supervised methods and heuristic methods from the BASALT challenge (Milani et al., [38]), and vision-language-action (VLA) Minecraft models: STEVE-1 (Lifshitz et al., [31]), GROOT (Cai et al., [6]), JARVIS-VLA (Li et al., [27]), OpenHA (Wang et al., [64]). Of the BASALT challenge submissions, KAIROS is an inverse RL method based on IQ-Learn (Garg et al., [15]), UniTeam augments the VPT BC policy with search, and GoUp executes pre-programmed scripts (Milani et al., [38]).
| No action labels | With action labels | Vision-language-action (VLA) models | Heuristic | Human | ||||||||
| Task | Mirror (Ours) | VIPER | Mirror (Ours) | KAIROS | BC (VPT-SFT) | UniTeam | STEVE-1 | GROOT | JARVIS-VLA | OpenHA | GoUp | |
| FindCave | 82% | 0% | 91% | 27% | 27% | 64% | 45% | 45% | 36% | 45% | 36% | 95% |
| MakeWaterfall | 36% | 0% | 45% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 36%{sup}†` | 100% |
| CreateAnimalPen | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 18%{sup}†` | 95% |
| BuildVillageHouse | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 0% | 91%{sup}†` | 100% |
Table 1. Solving fuzzy tasks from BASALT. Success rates for methods without action labels (our self-improving method and VIPER [12]), action-supervised and heuristic BASALT baselines [38], vision-language-action (VLA) models, and humans.
Even without expert actions, Mirror outperforms all baselines that do not rely on hand-crafted heuristics. Only 5 hours of self-play were required to achieve the highest success rate on FindCave and MakeWaterfall tasks. To the best of our knowledge, Mirror is the first method to achieve non-zero success on MakeWaterfall from learned behavior alone (i.e., without classifiers or hard-coded actions). The baselines fail in different ways (Figure 18): the action-supervised methods (KAIROS, BC and UniTeam) drift out of distribution on multi-step tasks such as MakeWaterfall, the VLA models struggle to follow the task descriptions, and GoUp’s hard-coded scripts are brittle, placing, for instance, the water bucket but freezing once underwater, whereas Mirror escapes the water using knowledge learned from human gameplay. CreateAnimalPen and BuildVillageHouse remain unsolved by all learning-based methods. Lastly, while Mirror does not require action labels, it can still leverage them for a further small performance gain.

Figure 18. MakeWaterfall comparisons. Mirror is capable of the long horizon planning necessary for MakeWaterfall. The agent must make a waterfall, remember its location, and move to another location to take a picture of it. Even though BC, UniTeam and KAIROS are trained with action labels, they achieve 0% success on this long-horizon task. They often go out of distribution. GoUp, which relies on scripted actions, cannot adapt to unseen scenarios such as when underwater. While it is able to place the water bucket with a pre-programmed instruction, it does not know what to do when underwater. The VLA approaches, STEVE-1, GROOT, JARVIS-VLA, and OpenHA, struggle at following the task instructions. VIPER, our action-free baseline, struggles to learn the Minecraft action space.
Self-Improvement in Minecraft Survival Mode
We next evaluate Mirror on the Minecraft Diamond Challenge (Guss et al., [17]), in which the agent must obtain a diamond in survival mode, starting from a new world and an empty inventory.
In the online setting, Mirror is the first method to acquire an iron pickaxe through self-improvement alone, without reward supervision or expert action labels. In the offline setting, where expert action labels are available but environment interaction is not, Mirror obtains diamonds at nearly eight times the success rate of Dreamer V4 [19], the state-of-the-art world-model agent on Minecraft (Figure 5).
Figure 5. Minecraft Diamond Challenge. Success rates across twelve acquisition milestones. Offline methods learn from VPT human data without environment interaction. Online methods learn by interacting with the environment for about 10M steps (140 in-game hours) without expert action labels. In the offline setting, Mirror (supervised) sets a new state of the art. In the online setting, Mirror (self-improving) is the only method to reach the iron pickaxe without action labels or rewards. Mirror is evaluated on 128 episodes of 60 minutes with respawning, following VPT and Dreamer V4. Offline baselines are from Dreamer V4 ([19]). Online baselines at 10M environment steps are from Dreamer V3 ([18]).
Unlike the BASALT tasks, the Diamond Challenge has an unambiguous and easily verifiable objective, but reaching it requires an extremely long horizon of up to an hour of gameplay. The agent must complete a long chain of prerequisite subgoals, from chopping trees and crafting tools to smelting iron and mining deep underground. While our video planner alone can solve the 3–5-minute BASALT tasks without any task decomposition (§ 6.1), an hour-long horizon remains beyond it. Such a horizon is hard even for humans: players given 20 minutes to obtain a diamond pickaxe collect diamonds in only 15% of episodes [3].
To solve such an extremely long-horizon task, we follow Dreamer V4 and condition the agent on its current subgoal. Whereas Dreamer V4 conditions its policy on one-hot task embeddings, we instead condition only the video planner: we finetune it on subgoal intervals of VPT human gameplay with text prompts that describe the subgoal. During rollouts, a heuristic subgoal task planner chooses the prompt from the player’s own state, namely the inventory, the health, and whether a GUI screen is open. The V2A model therefore never observes the subgoal. Since Dreamer V4 publishes neither how it extracts subgoal intervals from the VPT recordings nor how it decides when to advance to the next subgoal during evaluation, we implement our own data curation (Appendix § 13.1.4) and heuristic subgoal task planner (Appendix § 13.1.5).
Video planning with a pretrained V2A model. To study the video planner independently of the self-improvement algorithm, we first evaluate it with a V2A model pretrained on 3k hours of paired human action-demonstration data from the VPT dataset [3]. In this offline setting, our agent sets a new state of the art, obtaining a diamond in 5.5% of episodes, compared with 0.7% for Dreamer 4 [19], the best world-model agent. This supports the VP + V2A decomposition in equation 1 over WM + policy decompositions such as Dreamer 4’s. A WM + policy agent must learn procedural and physical knowledge from action-labeled data, whereas the VP learns both from internet-scale video. Vision-language pretraining does not supply this knowledge either: the Gemma 3-based VLA ([55]) evaluated by Hafner et al. [19] never obtains a diamond.
Self-improvement with a generative VP. Without action labels or rewards, our self-improving V2A model matches its supervised counterpart on early-game items and remains competitive with offline methods trained on 2.5k hours of action-labeled demonstrations. We begin self-improvement with a V2A model trained from scratch and a goal to acquire wood. While the V2A model may output random actions early in training, re-planning allows the agent to recover from its mistakes so it remains in distribution. Once the agent succeeds at a goal, the text prompt is updated with the next goal. The agent thus collects progressively more challenging items through online interaction. Appendix § 13.1.6 details the exploration strategy and the training protocol. After 140 hours of online interaction (about 10M environment steps), our agent crafts a stone pickaxe in 72.7% of episodes and an iron pickaxe in 15.6%. At the same budget, Dreamer V3 (Hafner et al., [18] crafts an iron pickaxe in only 1.7% of episodes, despite using task rewards, simplified crafting, and accelerated mining (Figure 5). Mirror retains a large margin even when matching Dreamer V3’s evaluation protocol (Appendix § 15.1.3), in which episodes last 30 minutes and end at the first death, rather than the 60-minute episodes with respawning used by VPT ([3]) and Dreamer V4 ([19]).
Self-improvement with an oracle VP. Generative video planners are already capable, and we expect them to keep improving. To examine how Mirror scales with better planners, we decouple self-improvement from the generative VP using an oracle video planner (oracle VP), which we construct from an expert policy and a copy of the environment. At each planning step, we synchronize a separate instance of the game to the agent’s current state and let the expert policy, conditioned on the agent’s past observations , play forward in it. The frames rendered by this copy serve as the plan , so the oracle has the same input-output behavior as a video planner, , while the expert’s actions are never shown to the V2A model.
Using VPT’s early-game RL policy ([3]) as the oracle, we train a V2A model from scratch with self-play. After only about 80 hours of online interaction, the agent acquires diamonds for the first time (Figure 5), an order of magnitude fewer environment interactions than Dreamer V3 (Hafner et al., [18]) requires, despite Dreamer V3’s use of simplified crafting and accelerated mining. RL agents take far longer because they must explore the full action space on their own. This experiment suggests that the remaining bottleneck is not Mirror’s ability to learn from interaction, but the capability of the video planner itself.
| World | Instruction | Mirror (random) | Mirror (self-improv.) | VIPER | BC | Mirror (supervised) |
| Woodland | Swim | 70% | 80% | 0% | 10% | 80% |
| Woodland | Chop tree | 0% | 100% | 0% | 0% | 80% |
| Woodland | Ascend hill | 100% | 90% | 0% | 0% | 90% |
| Woodland | Dig ground | 0% | 90% | 0% | 0% | 100% |
| Camp | Ascend hill | 70% | 90% | 0% | 0% | 90% |
| Camp | Enter cave | 90% | 50% | 0% | 0% | 60% |
| Camp | Light cave | 0% | 0% | 0% | 0% | 20% |
| Camp | Mine stone | 0% | 10% | 0% | 0% | 40% |
| Camp | Place blocks | 70% | 30% | 20% | 20% | 70% |
| Camp | Dig ground | 20% | 100% | 0% | 0% | 80% |
| Total | 42% | 64% | 2% | 3% | 71% |
Table 7.
Emergent Capabilities of the Video Planner
Training the video planner on internet-scale video data unlocks novel capabilities without requiring additional fine-tuning for each task. We explore such emergent behaviors in two settings.
Emergent in-context learning from failure. We examine if Mirror can utilize its history, , for in-context learning. In Figure 6, we challenge our agent to escape a lava lake. If the agent falls in the lava, it respawns at the origin. We compare a case where the agent’s history is preserved between deaths, and a case when the history is reset after every death. The task is difficult: with fresh histories every life, the escape rate is only 3%. When video context is retained between lives, however, the agent leverages its past failures to improve its escape rate. Figure 6 (b) shows the resulting agent’s trajectories. With shared history, the agent exhibits a clear directional bias, demonstrating that prior attempts influence the agent’s behavior. With independent history, the agent moves in relatively uniform directions around the starting point. Retaining history does not provide a comparable benefit to the VPT BC policy, perhaps even hindering performance. This experiment shows that our video planning agent exhibits some emergent in-context adaptation.
Figure 6. Emergent in-context learning from failure. (a) We challenge our agent to escape a lava lake without fine-tuning. (b) Retaining context between deaths causes a directional bias in the agent’s trajectories, indicating in-context adaptation. (c) Shared history allows the agent to learn from its past failures in-context, leading to higher escape rates. Retaining recurrent state does not provide a comparable benefit for VPT BC.
Text-conditioned video planning. As a proof of concept, we test whether our pretrained video planner can be steered by text zero-shot. Since our planner was trained only on captions generated automatically by a small VLM (Appendix § 9.§ 5), we evaluate simple goals such as “chop down the tree” or “dig into the ground,” rather than complex instructions. Accordingly, we do not aim to compete in language understanding with Minecraft-specialized vision-language-action models such as JARVIS-VLA (Li et al. [27]) and OpenHA (Wang et al. [64]). Each task nevertheless lasts about 30 seconds and requires consistent, prolonged behavior. We pair the text-conditioned video planner with a V2A model pre-trained on random walks, a V2A model pre-trained on the VPT expert actions, and our self-improving V2A model. We also compare against VIPER (Escontrela et al. [12]), which similarly does not use action labels, and a text-conditioned BC policy (Appendix § 15.1.4). As shown in Figure 7, Mirror reliably harvests trees, climbs hills, and digs underground, although some tasks remain difficult across all V2A variants. The self-improving V2A model generally outperforms the one trained on random walks and remains competitive with the one supervised on human actions. These results show that the V2A model can execute plans without ever knowing the task. Since the V2A model is task-agnostic, a sufficiently capable video planner would let the agent solve new tasks from a text prompt alone, without task-specific action data or retraining.

Figure 7. Text-conditioned video planning. We evaluate text-conditioned control on 10 tasks across two unseen environments (see Appendix § 15.1.4). Each task is specified only through a text prompt, with no task-specific fine-tuning. VIPER (Escontrela et al., [12]) is included as a reward-free video-based baseline. Combined with our self-improving V2A approach, the agent achieves high success rates although it has never seen expert demonstrations on these tasks. Left: Success rates of each method. Right: Text guidance leads the agent to complete different tasks.
Solving Robotics Tasks and Video Games
Setup. We evaluate Mirror on standard subsets of key synthetic decision-making benchmarks: OGBench (Park et al. [46]), DeepMindControl (Tassa et al. [54]), and Atari (Bellemare et al. [4]; Escontrela et al. [12]). We benchmark with BCO (Torabi et al. [58]), which alternates between learning a V2A policy rollouts and cloning its inferred actions on expert demonstrations, VIPER (Escontrela et al. [12]), which uses video prediction likelihoods as rewards, LAIfO (Giammarino et al. [16]), which learns a policy by adversarially matching expert and agent transitions, and the method proposed by Luo & Du (Luo & Du [36]) (which we abbreviate as “Grounding”) which learns a goal-conditioned policy to reach video-predicted subgoals. We evaluate Mirror with both a learned video planner and an oracle planner that supplies future frames from an RL expert. Unlike in our Minecraft setting, large-scale human play datasets are not available, and we thus found it essential to augment the generative planner’s training data with recovery examples collected using DAgger (Ross et al. [49]). Please find an in-depth discussion of dataset construction, evaluated tasks, planner training, and baseline implementations in Appendix § 13.
Generative VP results. As shown in Figure 8, among methods evaluated without oracle planners, Mirror surpasses all baselines on Atari and OGBench, and learns faster than LAIfO on DMC. LAIfO performs less consistently across domains, learning less efficiently on OGBench and achieving limited returns on Atari. VIPER also achieves low returns on Atari, suggesting that using video model predictions directly provides better supervision than likelihood-based rewards.

Figure 8. Imitation learning performance across three benchmark suites. Left: learning curves on 13 DeepMind Control (DMC) tasks, four OGBench tasks, and seven Atari games. Colors identify methods; dashed method curves use an oracle VP and solid method curves use a generative VP. The horizontal dashed line marks expert performance. Curves average tasks equally, with shaded regions showing ±1 standard error across three seeds per task. The Atari VIPER curve is digitized from Escontrela et al. [12] over its first 1M decisions and rescaled to % of expert. Right: final performance at 1M environment transitions (mean ± SE, equal task weights), separated by planner type.
The comparisons with BCO and Grounding help explain these gains. Although BCO also learns a V2A model from self-collected experience, it performs substantially worse, indicating the importance of using the video planner to guide data collection. Grounding improves on BCO, consistent with the benefit of video-guided exploration, but still remains substantially below Mirror. The remaining gap suggests an advantage to learning a V2A model that translates frame-to-frame transitions into actions over learning a goal-conditioned policy to reach predicted subgoals.
Oracle VP results. To evaluate our self-improvement algorithm independently of video-planner quality, we also use an Oracle VP that supplies frames from an RL expert. In this setting, our method matches the expert on Atari, suggesting that our Atari performance is limited by the video planner, most likely by the compute needed to train a model that accurately predicts fine details such as the motion of small projectiles. This result demonstrates that as video models continue to scale and improve, our method provides a simple framework that is able to solve a wide variety of tasks. In Appendix § 15.§ 2, we also demonstrate that Mirror learns a V2A model that generalizes better to unseen goals.
Mirror also exceeds the expert’s return in Assault and Zaxxon by 10% and 25%, respectively. We observe that the expert issues redundant fire commands that do not produce additional shots. The V2A model can thus follow the plan without copying them. As a result, Mirror makes fewer ineffective firing attempts in Assault, reducing overheating, and releases FIRE more frequently in Zaxxon, enabling subsequent presses to trigger new shots.
Conclusion
We have introduced Mirror, a method that learns to translate behaviors encoded in large video planners to policies executable by an agent. Unlike prior work, our method assumes neither paired video-action data nor task-specific rewards, but instead learns to translate video into actions simply by self-play in an interactive environment.
Limitations. Our approach has several limitations. First, without a separate high-level task planner that conditions the video planner on subgoals, our agent could not cover the hour-long horizon of the diamond task. We expect capable VLM-based task planners to fill this role in the future: VLMs excel at abstract, high-level task planning but struggle with dense, low-level control, whereas video planners show the opposite strengths, making the two naturally complementary. Second, with self-improvement alone, our agent reaches the iron pickaxe but does not yet mine diamonds. An imperfect V2A model often leads the agent into suboptimal states that human players rarely visit, and recovering from them demands very strong generalization from the video planner. Our video planner falls short of this at later milestones, each of which is covered by only a few hours of human gameplay in our training data. Third, some actions leave no visible trace, so inferring them from vision alone and from scratch is ill-posed; in practice, we expect a small amount of action data to resolve this. Lastly, training a capable generative video planner can be data-intensive. Unlike human gameplay, which naturally contains mistakes and recoveries, the data generated by expert policies in our robotics and control benchmarks (§ 7.1) lacked diversity, so we needed to augment it with recovery data collected via DAgger [49] to train a sufficiently robust video planner.
We believe Mirror offers a promising path toward general-purpose agents that acquire diverse behaviors from video pre-training and learn to self-improve their execution through interaction.
Acknowledgements. This work was supported by the Toyota Research Institute (TRI) University 3.0 (URP) program, the National Science Foundation under Grant No. 2211260 and Cooperative Agreement EEC-2330040 (NSF Engineering Research Center for Human Augmentation via Dexterity, HAND), AMD via the MIT AI Hardware Program, and the Amazon Science Hub. We further gratefully acknowledge significant compute and technical support by AMD. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of any other entity. We thank Chonghyuk Song for helpful discussions.
AI Use Statement. We used generative AI tools to assist with the implementation and editing of experimental code, literature retrieval and discovery, the generation of synthetic data, and language polishing of the manuscript for clarity and readability. Generative AI tools were not used to formulate or prove mathematical claims. All experimental results and conclusions were independently evaluated by the authors. The authors take full responsibility for the final content of this work.
Video Planner
Design principle. The central goal behind the design of our video planner is long context. Many of our tasks require memory over long horizons. Long context may also help error recovery in closed-loop planning. Early in training, the imperfect V2A model often deviates from expert behavior, forcing the planner to operate on trajectories outside its training distribution. With a short context, we often saw the planner repeat the same failed behavior. With long context, we instead observed emergent recovery behavior, as the planner can use its previous failures to change strategy (see § 7) and replan. This section describes how we make such long context affordable.
Architecture
Videos are encoded with the Wan2.2 VAE (Wan Team, 2025), which compresses 4 frames in time and pixels in space into 48 latent channels. The zeroth pixel frame is encoded on its own rather than grouped with other frames, so latent frames correspond to pixel frames. The MineRL environment [3] renders frames at 20 fps. We resize each frame to , so that every four-frame chunk (0.2 s) is encoded into a latent array (height width channels). A patchifier then turns each latent frame into 60 tokens of 192 dimensions, which form the input of the video planner.
Backbone. The planner is initialized from Cosmos3-Nano [43], a mixture of transformers [30] with a text tower and a video tower. The two towers share the same Qwen3-VL [2] block design but have separate weights. Each has 7B parameters, 36 layers of width 4096, 32q / 8kv GQA heads of dimension 128, SwiGLU MLPs of width 12,288. For positional embedding, 3D-MROPE [53, 2, 43] is used in which the 128-dim attention head is split into [48, 24, 24]-dims, each encoding the time, height and width of the video. We freeze the text tower and finetune only the video tower. The video tower plays two roles: as a context encoder, it encodes the clean history of observed frames, and as a denoiser, it denoises the frames of the plan. We refer to the two roles separately for clarity, although they share the same architecture and weights.
Block-sparse attention with FlexAttention. The original Cosmos3 implementation uses NAT-TEN [20], which cannot efficiently express our block-causal sliding-window attention pattern. We instead use FlexAttention [10], which supports efficient attention with arbitrary block-sparse masks. On NVIDIA Hopper and Blackwell GPUs, we use FlexAttention’s experimental FlashAttention-4 [68] backend, which is 1.35–1.55 faster than the default Triton [57] backend on our workload. On AMD GPUs, we use the Triton backend.
Mixture-of-transformers attention. In the original Cosmos3, all video tokens attend to each other bidirectionally under a single mask. In our implementation, however, the video tower’s role is split into a context encoder and a denoiser, which require distinct attention patterns. The video history is block-causally encoded frame by frame, whereas the future plans are jointly denoised and thus require bidirectional attention. Together with the text, this yields three token streams. At every layer, each stream self-attends to its own tokens and cross-attends to the streams it is conditioned on:
Text (text tower): self-attends causally, token by token. It carries the video metadata (resolution, aspect ratio, duration and frame rate), which also serves as an attention sink, and optional task instructions.
History (context encoder): self-attends block-causally, i.e., bidirectionally within a frame and causally across frames. Cross-attends to the text.
Plan (denoiser): self-attends bidirectionally within the plan. Cross-attends to the history and the text.
Handling all these attention patterns in a single joint attention would require an overly complex attention mask. This is not only cumbersome to implement but also substantially slows down the FlexAttention kernel. We instead compute attention with a simpler mask for each source and merge the results as
where the attention output and the log-sum-exp are computed separately for each source . This merge is mathematically equivalent to standard scaled dot-product attention (SDPA) computed jointly over all sources. Keeping the sources separate makes the implementation of variable-length training and cached inference straightforward, at negligible overhead.
Attention sink. In video models such as Wan (Wan Team, 2025), the zeroth latent frame serves as the attention sink. In Cosmos3, whose mixture-of-transformers structure lets every video token attend to the text, the text tokens appear to play this role instead. Removing the text tokens greatly slowed down the finetuning. Provided that every video query can properly attend to the text metadata tokens, adapting the pretrained bidirectional model to block-causal attention is fast, taking fewer than 1,000 finetuning steps. On the other hand, removing the zeroth latent frame from the attention context of the denoiser caused only a small drop in performance if the history is long enough. The sliding-window attention described below therefore does not keep the zeroth latent frame; instead, every query always attends to the text.
Text conditioning format. Our video planner is conditioned on two kinds of captions, written in Cosmos3’s structured JSON format and wrapped in a chat template. Each caption contains the video metadata (resolution, aspect ratio, duration and frame rate) and, optionally, temporal information and language instructions.
The context encoder takes a task-level caption, which describes the long-horizon task sequence. It can optionally state how much time has elapsed since the current task sequence began. We find that providing this resume timestamp helps when the model context must be reset in the middle of a task. The caption may also describe the preceding clip, although we do not use this in our experiments. With all fields present, it reads:
{"temporal_caption": "<observations of the preceding 10 s> The clip resumes at 1:23.4. Task
sequence: collect wood; then craft a crafting table", "resolution": {"H": 192, "W": 320}, "
aspect_ratio": "5,3", "duration": "32.0s", "fps": 20}
The denoiser takes a control-level caption, which describes the desired short-horizon behavior of the plan. The duration field is included to help the denoiser handle variable-length prediction. It can optionally state the time range of the plan to generate. An unconditional plan uses the metadata-only caption.
{"actions": [{"time": "1:25.0–1:26.6", "description": "Place the crafting table"}],
"resolution": {"H": 192, "W": 320}, "aspect_ratio": "5,3", "duration": "1.6s", "fps": 20}
Training
Objective. We train with the linear interpolant and velocity prediction [32, 34], with each latent frame averaged over averaged over latent dimensions. Every target latent frame carries equal weight in the loss. Following Esser et al. [13] and Wan (Wan Team, 2025), is drawn from a mixture of a logit-normal distribution shifted towards higher noise scale. As our plan resolution and length are smaller than those of typical video models, we use relatively low shift. We use shift for VPT pretraining and BASALT finetuning, and use 3 for the diamond finetuning. We also sample from a uniform distribution 10% of the time to ensure better GUI interaction details and likelihood estimation.
Variable plan length. The right length for a plan is unknown in advance. Using too short plans or planning autoregressively with the shortest chunk the VAE allows (one latent frame, 0.2 s) produced poor plans, while 4 s is too long for open-loop execution. We therefore train a single model on a mixture of plan lengths from 0.2 s to 4 s. All frames of a prediction window share one diffusion time, and different windows draw diffusion time and noise independently. We also stochastically drop the text conditions of a subset of the prediction windows for unconditional generation and text-conditional classifier-free guidance [21].
Each training sample is a 32 s video clip, which is block-causally encoded as the clean history. Because this history is shared, we predict many prediction windows from a single video clip at once. This reuse of the context encoding is our key scaling knob. The learning signal comes only from the denoised tokens, so predicting many windows per video clip maximizes the learning signal per token. In our training setup, the prediction windows of each video clip span twice its 32 s length, totaling 64 s. Since the windows differ in position, length, diffusion time and noise, predicting the same video clip twice still provides new learning signal. In our gradient noise scale (McCandlish et al., [37]) measurement at 30k steps, this double pass over a video clip reduced the gradient noise as much as an additional fresh video clip would. As the cost of encoding 32 s of history is amortized over all windows of the video clip, long-context training becomes affordable. Combined with sliding-window attention, this long-context training lets the planner roll out far beyond the training length, to about five minutes in our experiments.
Static shape training for variable-length plans. Compiling every transformer block is essential for training speed, and it requires static tensor shapes. FlexAttention makes this even more important: its block mask is built for a specific sequence shape, so any change of shape forces the mask and the attention kernel to be rebuilt and recompiled. While FlexAttention also supports dynamic-shape compilation, it is far less efficient than static compilation in both throughput and memory. We therefore keep every tensor shape fixed for training and express the variable prediction windows purely through the attention mask. We illustrate this in Figure 9.
Figure 9. Variable prediction length training. Prediction windows of variable length over the same 32 s clip. The clean history is encoded once by the context encoder and shared by all prediction windows. With sliding-window attention, the clean history tokens and the noisy plan windows attend to the preceding 12 s and 8 s clean frames, respectively.
In our setup, each GPU processes 6,144 text tokens for the frozen text tower and one 32 s video clip as a fixed-length sequence of 9,600 clean video tokens (160 latent frames × 60 tokens), followed by 19,200 noisy plan tokens. All three numbers are multiples of FlexAttention’s 128-token block size, so the sequences align exactly with the 128 × 128 attention blocks for best efficiency. The number, lengths and positions of the prediction windows change at every step, but they only change the values of the boolean and integer window/frame id tensors that the mask function reads. The mask function for the video self-attention is:
where SWA() is the set of clean tokens in the sliding window of . For a clean query, this is the preceding 12 s up to and including its own frame. For a noisy query, this corresponds to the 8 s of clean frames immediately before its prediction window. is the index of the prediction window containing . We use for clean tokens.
Parallelism and memory. We shard the video tower with Hybrid Sharded Data Parallel (HSDP) strategy on multiple nodes using Pytorch FSDP2 library (Zhao et al., [70]). Parameters are sharded within each 8-GPU node and replicated across nodes. We use selective activation checkpointing (Chen et al., [9]) for each transformer block. On AMD MI350X GPUs (288 GB of memory) we do not use activation checkpointing.
Precision. We use BF16 precision for the computation. FSDP2 automatically downcasts the FP32 weights to BF16. The gradients are reduced across ranks in FP32. Numerically sensitive parts stay in FP32: the input patch projection, the timestep embedder, the velocity head, RMS/LayerNorm, the rotary embeddings, and the LSE attention merge. Training losses are computed in double precision. The frozen text tower and the VAE run in BF16. For pretraining, we additionally use FP8 mixed precision training with dynamic tensorwise scaling for the linear layers. Because we use sliding window attention (SWA), which is computed by an efficient block-sparse kernel, the training step is dominated by these matrix multiplications: FP8 speeds them up by about and a full training step by about . We found no evidence that FP8 training degrades the planning quality. FP8 can be disabled for cooldown or finetuning to reach a lower loss floor.
Optimization and pretraining schedule. We use AdamW [35] with , and weight decay 0.01, a 1,000-step linear warm-up followed by a constant learning rate, gradient clipping at global norm 3.0, and an EMA of the weights with decay 0.9999. We chose AdamW because Cosmos3 was trained with it. We pretrained on VPT contractor dataset [3] for 220k steps. As GPUs became available we grew the global batch from 8 to 32, raising the learning rate roughly with the square root of the batch size (Table 2), and ended with a 5k-step cooldown at a constant . The run took roughly 4,700 H200 GPU-hours of training steps. The final pretrain checkpoint is the step-220k EMA.
| Steps | GPUs | Global batch | Learning rate | Wall-clock |
| 0–30k | 8 H200 | 8 | 25 h | |
| 30k–80k | 16 H200 (2 nodes) | 16 | 42 h | |
| 80k–215k | 32 H200 (4 nodes) | 32 | 115 h | |
| 215k–220k | 8 H200 | 32 | (cooldown) | 10 h |
Table 2. Pretraining schedule of the video planner on VPT dataset.
Pretraining data. The VPT contractor dataset contains 2,945.7 hours of gameplay. We excluded one game-version subset (6.9; 271 recordings, 21.8 h) whose logged cursor coordinates are malformed; the coordinates are needed because the recordings do not show the cursor inside menus, so we draw it back into those frames. We also excluded low-quality data (7.9 h), leaving 55,919 recordings (2,916.0 h) for training. Training samples a recording uniformly and then a random 32 s clip from it. Frames are resized from 640 × 360 to 320 × 192 with bilinear interpolation and antialiasing. Over 220k steps, the planner saw about 98,000 hours of noisy video clips. The captions are generated from the videos with the Cosmos3-Nano VLM, as described in Appendix § 9.§ 5.
Finetuning. The pretrained video planner is finetuned for each downstream task with global batch 8 (16 to 32 for the Diamond planners) and an EMA decay of 0.9995. The number of steps varies from 10k to 50k depending on the task difficulty and the amount of available video data. The BASALT planners are trained only on human gameplay videos, without text instructions, with a learning rate of , a 1k-step warm-up and cosine decay. BASALT episodes end when the player presses the Esc button, which is not visible in the video. We therefore finetune the video planner to predict a gray screen once the player presses the Esc button. The Diamond planners were finetuned with 16 MI350X GPUs on our curated split of human VPT gameplay, divided into subgoal clips (Appendix § 13.1.4). Text conditioning informs the planner of the current stage of the diamond tech tree.
Inference
Rolling cache with a fixed budget. At inference, the planner keeps the keys and values of only the most recent 12 s of observations in a fixed-size ring-buffer cache. Each newly observed frame is encoded once by the context encoder using this cache. This new frame replaces the oldest frame in the cache, which no query can attend to anymore because of the sliding window. To plan, the denoiser generates the next 1.6 s future by attending to the last 8 s of the cache and optionally to the control-level caption. Because the cache size never grows, every plan costs the same and the model can be compiled with static shape. We illustrate this in Figure 10.
Figure 10. Rolling cache at inference. (1) Each new observation is encoded once by the context encoder, which attends to the 12 s cache. (2) The resulting keys and values are appended and the oldest context is evicted, keeping the cache size fixed. (3) The denoiser generates the next 1.6 s of video while attending to the last 8 s of cached context and, optionally, to a control-level caption. Cache cells schematically group frames.
Sampling. Plans are sampled with the second-order UniPC solver [69] for 20 steps with a time shift of 3. We quantize the linear layers of the video tower to FP8 with per-tensor dynamic scaling, which makes planning faster than BF16 on an H200. With 20 solver steps and autoguidance, the planner generates about 4.2 s of video per second of computation on a single H200.
Task-aware guidance. We use two forms of task-aware guidance. For the BASALT tasks, we use a variant of autoguidance [24], where the positive model is the finetuned planner and the negative model is a planner trained on broader, less task-specific data, such as its parent before task finetuning; we use a guidance scale of 1.5. For § 7 and § 6.2, we use text classifier-free guidance [21] with a scale of 7. For § 6.2, the text consists of one of 16 subgoal captions from log to diamond, chosen by the subgoal task planner (Appendix § 13.1.5). The conditional and unconditional branches share the cached history, so guidance doubles only the denoiser’s computation. We find that task-aware guidance is crucial for goal persistence. Without it, the planner often abandons the current goal after repeated failures. As a result, it may skip necessary steps and move on to the next subgoal before completing the current one. Task-aware classifier-free guidance effectively suppresses repeated mistakes and strengthens adherence to the intended goal.
Caption Generation
Pipeline. We captioned the VPT corpus with the vision-language model of Cosmos3-Nano [43], served offline by vLLM (Kwon et al., 2023) with JSON-schema-constrained decoding (temperature 0.7, top- 0.8, top- 20). We aim for text that is both descriptive, grounded in what is visible, and instructional, phrased the way a user would command an agent. We therefore generate captions in three stages. A strictly factual stage comes first, and later stages only rephrase or group its output.
Descriptions. The model watches overlapping windows of every video at four temporal scales. These are 8 s windows every 4 s at 4 fps, 24 s windows every 12 s at 1 fps, and 64 s windows every 32 s at 0.5 fps, all at . The fourth scale is short windows at full resolution centered on salient moments (attacks, opening a menu, jumps, sharp turns, climbs and descents of at least four blocks). The model is not told why a window was chosen. The model lists up to six atomic, intent-free descriptions per window, each with a start and end time.
Instructions. Batches of 40 descriptions, together with the video covering their time span, are rewritten one-to-one into imperative instructions. Each instruction inherits the description’s time span.
Tasks. From the video duration and up to 110 descriptions evenly distributed across time, the model partitions the whole recording into 1–8 consecutive task segments and infers the intent that makes each segment’s behavior coherent. We run this stage five times per video with different seeds, which yields five alternative task partitions.
The description stage took 4.6 hours on 32 H200 GPUs, plus 1.2 hours for the coarsest scale.
Prompts. The instructions of the three stages, verbatim:
Stage 1: descriptions (system: “You are a helpful assistant.”; input: video)
This is a clip from a Minecraft gameplay recording; each frame is preceded by its <t seconds>
timestamp within the full episode. List the distinct things the player OBSERVABLY does, as
separate facts. For each fact give start_s and end_s in those same episode seconds, read from
the frame timestamps. Rules: describe only what is visible on screen --- never goals,
purposes or attempts (no ’trying to’, ’in order to’, ’prepares to’, ’searching for’); one
atomic action or event per fact; repeated actions separated in time are separate facts; name
blocks/items/mobs/GUI screens concretely when identifiable (hotbar, inventory, crafting table
, crosshair target); 6-20 words per fact. If nothing notable happens, return one fact
describing the ongoing state (movement direction, terrain).
Stage 2: instructions (system: “You are a precise Minecraft gameplay annotator.”; input: video + indexed descriptions)
Below are OBSERVATIONS of what a Minecraft player did, each on its own line with an index.
Rewrite EVERY observation as a CONTROL: the instruction a coach could have given just before
it, which that behaviour then obeys.
Rules for each control:
- Imperative, 3-14 words, concrete Minecraft vocabulary. Command the ACTION, not the scene: ’
Mine the coal seam in the left wall’, not ’The player mines coal’.
- Say only what the observation says. Add no goals, no reasons, no purpose clauses (’to get
wood’, ’so that...’), no outcomes the player could not know in advance.
- A coach speaks before the fact, so drop discoveries and results from the wording: ’Open the
furnace’ rather than ’Open the furnace, revealing an iron ingot’.
- Keep the affirmative form; never write ’Don’t’, ’Avoid’ or ’Stop’.
- Return one control per index, for every index given, in the same order.
Stage 3: tasks (system: “You are a precise annotator of MINECRAFT gameplay. Every session, task and control below comes from a recording of a human playing Minecraft, and every word you write must be Minecraft vocabulary.”; input: text only)
You are splitting one Minecraft recording into its TASKS.
A TASK answers exactly one question:
What ongoing purpose makes this stretch of the trajectory coherent?
It is NOT a summary of what happened in order.
BAD (activity summary): "Walks around, gathers wood, opens the inventory, and mines."
GOOD (intent hypothesis): "Prepare equipment for underground exploration."
The segments run back to back and together they cover the WHOLE recording, from the
first frame to the last. You give only the END of each segment: the first ends at
0 s, every later one starts where the previous ended, and the last one ends at the
video duration. Give the ends in increasing order.
Rules:
- A segment should normally cover at least 30 seconds.
- Split where the player’s PURPOSE changes, not where the scenery changes.
- Name the intent that makes each whole segment coherent, in 6-20 words of concrete
Minecraft vocabulary, grounded in the facts.
- Do not add narrative detail the evidence does not support.
- Cite the fact ids that support each segment.
VIDEO DURATION: <duration> s
FACTS (intent-neutral observations, video-relative seconds):
[f12] 3.0-5.5s <observation>
...
Return the JSON object with your segments.
Filtering. We clean the generated captions with simple rules:
decoding is constrained to a JSON schema;
description times are clamped to their window;
empty or zero-length items are dropped, 1.5% of descriptions in the rewrite;
task boundaries are closed into an exact partition on the frame clock.
Statistics. Table 3 summarizes the corpus: 58,121 videos (2,943 h), about 28M captions and 489M tokens. Descriptions and instructions overlap heavily by design. Their total span is about the video duration, so every moment is described several times at several scales. Each task partition tiles its video exactly. 94.6% of task intents and 84.9% of instructions are unique strings.
| Count | Per video (median) | Span (s) | Tokens / caption | |
| Videos | 58,121 | – | 241.0 (duration) | – |
| Descriptions | 13,352,359 | 244 | 3.5 [1.0, 8.0] | 19.8 |
| Instructions | 13,155,310 | 237 | 3.5 [1.0, 8.0] | 15.2 |
| Task segments | 1,506,066 | 26 (5 partitions) | 31.5 [12.0, 62.6] | 16.1 |
| Total tokens | 489.1M |
Table 3. Caption corpus statistics for the VPT dataset. Spans are median [10th, 90th percentile]. Tokens are counted with the Cosmos3-Nano tokenizer.
Implementation Details and Baseline Selection
Minecraft
V2A Implementation Details
Architecture. Unless noted otherwise, the V2A model for Minecraft uses an 80-million parameter V-JEPA 2.1 ViT-B model ([1, 41]) as a backbone. The model takes up to 8 history frames and 16 future frames as input, and predicts a chunk of up to 16 actions. The V2A predicts keyboard and mouse actions from the MineRL 1.0 action space ([17]). The button actions are parameterized as categorical distributions, following the factorized button representation of VPT. The and offsets for mouse movement are parameterized as a 2D vector. For the diamond task, the final self-improving model is retrained with the larger V-JEPA 2.1 ViT-L encoder (Appendix § 13.1.6).
Exploration Noise. Exploration noise helps the agent learn novel interactions with the environment. However, if the noise is too high frequency, it may harm performance. Real-world actions are typically low frequency—a player will often repeat the same action, such as moving forward, over several time steps. We generate such low frequency exploration noise with random walks. We inject Brownian motion with standard deviation 1.0 for mouse exploration, and add noise corresponding to random walks of button presses for discrete actions.
BASALT Evaluation
Our method and the BASALT baselines were evaluated using the 11 world seeds from the BASALT 2022 challenge ([38]). Specific task prompts and environment settings can be found on the MineRL website. The BASALT 2022 retrospective report ([38]) contains comprehensive details and code describing the winning solutions to the challenge: GoUp, UniTeam, KAIROS, and others.
While the BASALT criteria are intentionally fuzzy, we evaluate each method on objective criteria for fairness. For FindCave, success is granted when the agent enters a cave or ravine. On MakeWaterfall, we report success as when the player successfully places a water bucket, and ends the episode facing the waterfall to “take a picture”. The waterfall must be in view during the episode’s final frame for success to be granted. Because Mirror does not rely on action labels and its video planner makes the decisions, the planner also decides when an episode ends: it predicts a gray screen where human players pressed Esc. On MakeWaterfall, we therefore treat a gray screen predicted by the video planner, or reaching the time limit, as the agent pressing Esc. As the only exception, we ignore gray screens predicted before the agent has given one second of movement input (about four blocks of walking), to filter out false positives at the very start of an episode.
Diamond Challenge
The following subsections describe the diamond experiments of § 6.2: the video planner’s data (Appendix § 13.1.4), the subgoal task planner (Appendix § 13.1.5), the training of the self-improving V2A model (Appendix § 13.1.6), and the post-processing of the videos (Appendix § 13.1.7). Here we summarize what supervision enters the pipeline.
Neither the VP nor the self-improving V2A model is trained on expert actions, rewards, or privileged game state; the only actions the self-improving V2A model learns from are the ones the agent executed itself. The VPT action and event logs come in at two places. First, as the mouse cursor is missing inside menus in the VPT recordings, we draw it using the logged mouse position, as VPT’s own data loader does [3]. This only restores what the player saw on screen. Second, the logs help curate the VP’s data. They direct the captioning VLM to salient moments (Appendix § 9.§ 5) and mark where each subgoal of the diamond task happens (Appendix § 13.1.4). This makes up for the limited capacity of our relatively small captioning model, the roughly 8B-parameter VLM of Cosmos3-Nano [43], which lacks Minecraft-specific domain knowledge. We believe that stronger VLMs with broader domain knowledge will make these log-based aids unnecessary. During rollouts, the subgoal task planner reads the agent’s inventory, health, and whether a GUI is open. These state inputs are used only to adjust the VP’s behavior by modulating its text prompt and guidance scale.
Diamond Video Planner Data
Our VP was finetuned only on human gameplay from the VPT contractor recordings [3]. Starting from the video planner pretrained on the VPT dataset (Appendix § 9), we first finetune it for 30k steps on gameplay rendered with the post-processing of Appendix § 13.1.7, so that the planner gets used to the new cursor and hotbar. We then post-train it for about 50k steps on a curated VPT subset split into subgoals.
We split the task into 16 subgoals, from log to diamond (Figure 11). In the 2,212 hours of VPT data versions 6.10 to 10, an inventory gain in the recorder log marks when a subgoal is completed, and a game statistic tells whether the item was mined, crafted, placed, or taken from a furnace. Each subgoal interval covers the activity leading up to the gain, such as the last run of attacks before a mined block or the time since the crafting screen opened. Training clips of 32 s lie at least half inside an interval of their subgoal, and subgoals are sampled in proportion to the fourth root of their footage, which roughly balances them. The resulting clips span 665 hours, 403 of which fall inside subgoal intervals. Footage is plentiful for gathering (153 hours for cobblestone, 126 for log, and 68 for iron ore) but scarce for everything else: 9.5 hours for diamond and 2.7–12.3 hours for each crafting, placing, or smelting subgoal.
Figure 11. Subgoal task planner. (1) Once per plan, the planner reads the agent’s inventory, health, and whether a GUI is open, and sends a subgoal caption and a guidance scale to the video planner; the V2A model never sees them. (2) Guidance-scale rules; the dashed one is used only during training. (3) Subgoals in the order the planner picks them from an empty inventory. Each craft happens at a crafting table placed for it and picked up afterwards.
Each subgoal has a single caption of the form “Current subgoal: {item}. {instruction}”, for example “Current subgoal: stone pickaxe. Craft a stone pickaxe using cobblestone and sticks in the crafting grid.” Captions carry no quantities, since the task planner decides when to move on. Prediction windows inside a subgoal interval get its caption, and some windows are left without text to train the unconditional branch for classifier-free guidance.
Subgoal Task Planner
Dreamer V4 [19] also conditions its agent on subgoals, prompting a one-hot task input through a fixed sequence of 25 tasks during evaluation. As the code is not public, we wrote our own heuristic subgoal planner (Figure 11). Once per plan, it picks the first unmet subgoal chaining backward from the diamond through the recipes, given the current inventory. It collects spare wood before going underground and enough cobblestone for both the stone pickaxe and the furnace, and it places a crafting table before every craft and picks it up afterwards, since it cannot tell whether an earlier table is still within reach. Its only inputs are the inventory counts, the health, and whether a GUI is open. Dreamer V4 does not say how it decides that a subgoal is done, but its task set includes open_crafting_table and open_furnace, and its evaluation prompts include use_stone_pickaxe and use_iron_pickaxe (Table 4 and Table 6 of [19]). None of these can be read from the inventory or a GUI flag. Our planner uses no such game events and no learned classifier. The task planner’s only outputs are the subgoal caption and the text CFG scale of the video planner. The V2A model never sees either.
| Method | Inputs | Outputs | Minecraft Data (hours) | ||
| Offline | Web | Online | |||
| Dreamer V3 | Video, inventory state | Keyboard, camera, abstract crafting | – | – | 1.4K |
| Dreamer V4 | Video | Keyboard, mouse | 2.5K | – | – |
| VPT (BC) | Video | Keyboard, mouse | 2K | 70K | – |
| VPT (RL) | Video | Keyboard, mouse | 2K | 70K | 230K |
| Video Planner (Ours) | Video, text prompt (optional) | Video | 3K | – | – |
| V2A Model (Ours) | Video | Keyboard, mouse | – | – | 140 |
Table 4. Data scale of different Minecraft agents
| Environment | Ours | Random-V2A | BCO-V2A | Grounding |
| Cube Single | 78.3 2.8 | 25.3 4.2 | 8.5 2.5 | 5.0 0.8 |
| Cube Double | 27.0 4.6 | 20.7 2.8 | 2.0 1.6 | 1.0 0.7 |
| Scene | 32.5 3.5 | 50.0 5.4 | 1.5 1.5 | 3.0 0.5 |
| AntMaze Large | 25.8 1.5 | 0.0 0.0 | 0.0 0.0 | 0.0 0.0 |
Table 6. Transfer to unseen OGBench goals. Success rate (%, mean SE across five seeds), averaged over target goals. All methods use target-task oracle planners; BCO-V2A uses BCO’s saved V2A model.
The text CFG scale is 7 by default. After the agent takes damage, it drops to 0.01 until the lost health is repaired at 1 HP per 1.5 s, so the video planner falls back on what players typically do when hurt, e.g., fight or flee. In a fraction of the training episodes, it also oscillates between 7 and 1/7 on the subgoals where the agent stalls most (cobblestone, iron ore, loading the furnace, iron ingot, and diamond). Evaluation never uses this rule.
Diamond V2A training
Online training. The self-improving V2A model starts from a pretrained V-JEPA 2.1 ViT-B encoder ([1, 41]) with freshly initialized action heads and learns online in 56 parallel environments. Training batches are sampled uniformly from a replay buffer that keeps the latest four in-game minutes of experience per environment. We keep the update-to-data ratio at about 1, lower than in typical sample-efficient RL methods that update many times per environment step. Our V2A objective has much lower variance than RL objectives, so the model learns efficiently from each transition, and compute is better spent on exploring new parts of the task than on replaying old experience. The V2A model’s observations pass through a Wan2.2 VAE round trip, so that they look like the generated plans. Online training ran for 45k updates at batch 224, over 140 in-game hours.
Exploration. Exploration comes from two sources. The first is behavioral: lowering the text CFG scale pulls the plans toward the unconditional VP, which reproduces the exploratory and recovery behavior of human players. The second is action noise (Appendix § 13.1.1), whose main role is to keep a minimum probability on every action. Actions with little or no visible effect, such as mouse clicks in an empty space, easily collapse to a single mode. The V2A model trains on its own executed actions and this results in a self-reinforcing collapse. The action exploration noise keeps a minimal floor probability, preventing any action from being starved. Note that both kinds of exploration are double-edged: they diversify the data, but they also slow the agent down and lower its chance of reaching deep subgoals. We therefore explore mostly at the frontier, where the agent is believed to be stuck. In the last four minutes before each milestone deadline, the guidance scale oscillates between 7 and 0.05 on a log scale and the action noise turns on and off, each with a random period of 0.5 to 3 minutes. In a fraction of the training episodes, the task planner also oscillates the guidance scale on stall-prone subgoals (Figure 11). Action noise is applied only during these exploration periods and the staggered entry below, and the V2A model trains on the noisy actions that were actually executed, rather than the clean actions it predicted. Even a small noise at this frontier is enough to prevent self-reinforcing collapse. Finally, if all environments started their episodes at once, the replay would be dominated by whichever subgoal they were all working on. To balance the subgoals, we stagger the start of training: environment first plays an unconditional world with action noise for s (16 s to 29.6 min) before its first task episode, which spreads the environments across the task. This staggered entry is applied only once, when training starts.
Figure 12. Video post-processing for the diamond task. (a) The original VPT video recordings do not include the cursor inside GUI menus. (b) As in VPT’s data loader [3], we draw the cursor from the mouse position log. However, VPT’s original cursor is too small for the Wan2.2 VAE, which blurs or drops it (Figure 13). Thus, for the diamond task we draw a high-visibility cursor: VPT’s cursor enlarged , tinted coral, and outlined in black and white. (c) The VAE also blurs the small item counts or turns them into other digits, so we crop the -pixel hotbar from the video, enlarge it with nearest-neighbor upsampling, and paste it back at the bottom center, moving the health and food bars up. (d) Finally, we resize the frames to with bilinear interpolation and antialiasing. Of the filters we tried, it is the blurriest but keeps all task-relevant detail and causes the fewest VAE artifacts (reconstruction PSNR 32.7 dB, versus 24.5–30.4 dB for bilinear without antialiasing, bicubic, area, and Lanczos).
Figure 13. The Wan2.2 VAE loses the VPT cursor. The same frame with (a) VPT’s cursor and (b) the high-visibility cursor, before and after a round trip through the Wan2.2 VAE. VPT’s small cursor nearly vanishes in the reconstruction, which is why we draw a larger, outlined cursor for GUI-heavy tasks.
Episodes and savepoints. To save compute, a training episode is split into segments that end at four milestones read from the inventory (a crafting table, a wooden pickaxe, a stone pickaxe, and three iron ore) and a final segment that ends at the diamond. The segments have limits of 10, 10, 10, 15, and 15 minutes plus the four-minute exploration period; a missed deadline or a third death ends the episode. We also save the world when an episode reaches the stone pickaxe, keep the 14 latest save points, and start a new episode from a random save with probability 0.25 times the filled fraction of this pool. These rules are not used at evaluation. We note that these rules could tilt the training distribution through selection bias. The limits, however, are loose. During training, each limit was 1.9 to 2.8 times the mean time the agent took to reach the milestone, and at least 92% of the reached milestones came within the limit. The deadlines thus rarely cut episodes that were still making progress and are unlikely to accelerate learning the way a reward would. If anything, the tilt works against speed, as an episode nearing its deadline enters the exploration period, which slows the agent down. Even if such a positive tilt exists, the four milestones are far coarser than the 15 to 20 rewarded subgoals of RL agents [3, 19]. For the oracle VP, we do not use the milestone deadlines and keep only a global episode limit of up to 10 minutes. The oracle is an RL policy that progresses much faster than human players, so deadlines set for our VP, which follows the pace of human gameplay, are rarely reached.
Final model. After online training, we retrain a fresh V2A model with the larger V-JEPA 2.1 ViT-L encoder [1, 41] on the last 110 of the 140 in-game hours of online experience, excluding the first 30 hours, which are dominated by exploration. It trains on the agent’s own executed actions for 20k steps at batch 128 and is the self-improving model in Figure 5.
GUI Video Post-Processing
These steps change only the video, not the game. As with the VPT recordings, the environment renders frames without a cursor and we draw ours on top, and we apply the same post-processing to the VP’s training data and to the agent’s observations during training and evaluation. The steps are needed because of the Wan2.2 VAE (Wan Team, 2025), whose latent space Cosmos3 is trained in. The VAE does not preserve small, high-precision details such as text and the cursor, which matter for computer-use tasks. With the oracle VP, which does not go through the VAE, the agent learns to mine diamonds without any of this post-processing. We therefore see the post-processing as a limitation of the Wan2.2 VAE rather than of our method, and we expect it to become unnecessary once video VAEs are trained on computer-use data.
Robotics and Atari
Environments and Evaluation
We evaluate on 13 DeepMind Control (DMC) tasks, four single-task OGBench environments, and seven Atari games. Each online method receives environment actions and is evaluated every 25,000 actions over 20 episodes, with five training seeds per task. Random-V2A collects its interaction budget before offline fitting on different-sized subsets. We average tasks equally, reporting native return on DMC, success rate on OGBench, and on Atari.
OGBench. We use four visual single-task environments: cube-single and cube-double require rearranging one and two cubes; scene requires manipulating a drawer, window, and locking buttons; and antmaze-large requires navigating a quadruped robot to a goal. Observations are RGB images. We use task 2 for Cube Single, Cube Double, and Scene, and task 1 for AntMaze Large.
DeepMind Control. Following Giammarino et al. [16], we use 13 tasks from DMC (Tassa et al. [54]): Ball-in-Cup Catch, Finger Spin, Cartpole Balance and Swingup, Pendulum Swingup, Walker Stand/Walk/Run, Hopper Stand/Hop, Quadruped Walk/Run, and Cheetah Run. Agents receive RGB observations with three-frame histories.
Atari. Following Escontrela et al. [12], we evaluate on Assault, Atlantis, Boxing, Freeway, Kangaroo, Pong, and Zaxxon from the Arcade Learning Environment (Bellemare et al. [4]). We use grayscale observations, four-frame histories, the minimal action set, sticky-action probability 0.25, and action repeat four. One million Atari decisions correspond to approximately four million emulator frames.
Shared Implementation
Action models. Mirror, BCO, Grounding, and Random-V2A use flow-based action models with method-specific conditioning. Continuous-control models use an IMPALA-style visual encoder and four 1024-unit flow layers. Atari uses a smaller encoder, one 512-unit flow layer, and factorized one-hot action codes projected to legal actions. Training uses Adam with learning rate , batch size 256, gradient clipping at 10, and random-shift augmentation. Action sampling uses ten Euler integration steps, from zero initialization at evaluation. LAIfO uses its separate single-action learner.
The action horizon is five on OGBench; three on DMC except Cartpole Balance (seven), Cartpole Swingup (ten), and Pendulum Swingup (twenty); and two on Atari except Atlantis and Kangaroo (four).
Expert information. Experts are BRO [42] on DMC, FQL [47] on OGBench, and DQN [40] on Atari. BCO and LAIfO receive observation-only demonstrations: 50 episodes per Atari game and 1,000/500/500/50 episodes for Cube Single/Cube Double/Scene/AntMaze Large. On OGBench, LAIfO uses full-horizon trajectories that continue after success, whereas BCO uses the original shorter demonstrations. DMC uses 50 episodes for BCO and 100 for LAIfO. No method uses expert action labels, task rewards, or proprioceptive state.
Learned video planners. Mirror and Grounding use task-specific latent flow predictors with four residual blocks of width 512. A spatial autoencoder with encoder widths 64, 128, 128, 128 maps frames at their native resolution to latents on OGBench and DMC and latents on Atari. The predictor models normalized latent differences from the current frame. Autoencoders train for 250,000 updates and predictors for updates; flow training uses Adam with peak learning rate , cosine decay, batch size 256, gradient clipping at 1, and exponential-moving-average weights with decay 0.999, and we keep the checkpoints with the lowest validation loss. Prediction uses ten Euler steps and averages eight latent samples per predicted frame. Autoencoders are frozen during flow training, and both models remain frozen during downstream learning. Each task shares one planner checkpoint across downstream seeds. Atari planners use three context frames, while action models use four.
Methods and Hyperparameter Selection
Mirror. After 5,000 random actions, Mirror repeatedly obtains future frames, predicts a -action sequence from the current visual history and these frames, and executes it before replanning. Achieved observations and executed actions enter a 100,000-transition replay buffer, with 0.2 V2A model updates per environment action. Training actions are sampled stochastically from the flow policy on all benchmarks. On OGBench and DMC, each step also adds Gaussian noise to one randomly chosen action coordinate ( on OGBench, task-specific on DMC). Atari relies on the stochastic flow sampling alone. Evaluation uses deterministic V2A model readout without collection noise.
For learned-planner experiments, we build the planner’s training set by iterative dataset aggregation: starting from 200,000 expert transitions, each round executes the current planner–V2A controller and branches 20-step expert recoveries (30 steps in later rounds for three DMC tasks) from its states with probability 0.1 per step, with later rounds adding action noise and finally a mix of action corruptions (Gaussian noise, random, scaled, sticky, or delayed actions) to the executed controller actions. The final planner is trained from scratch on the aggregated corpus (3.5–13 million transitions per task, 5% held out for validation). It conditions on three frames and predicts one future frame at a time, generating -frame plans autoregressively.
Behavioral Cloning from Observation. BCO [58] alternates learner interaction, V2A model fitting, and policy fitting. The V2A model infers action sequences from expert observations, and an observation-conditioned policy learns these pseudo-labels and executes action chunks open loop. Both networks retain their parameters and optimizer state across rounds. Episode-level validation selects each fit, and pending fits finish before final evaluation.
We tune the initial collection size and collection ratio , giving approximately actions between refits. Nine pairs are evaluated with three seeds on Cube Single, Finger Spin, and Pong, ranked by the final three evaluations. Selected values are for OGBench, for DMC, and for Atari.
LAIfO. LAIfO [16] learns an imitation reward by discriminating expert and learner observation transitions, then trains a single-action off-policy learner. Transition distance controls discriminator inputs and critic bootstrapping. DMC uses released task-specific settings with our BRO experts; Atari uses a dueling Double-DQN [65, 60] adaptation with greedy evaluation.
We tune exploration decay, transition distance, batch size, and a paired learning-rate/feature-dimension setting on Cube Single and Pong using 24- and 36-configuration grids, respectively, with three seeds per configuration. Selection uses performance at one million actions. Both suites select a 100k exploration decay, batch size 512, learning rate , and feature dimension 100. Transition distance is three for OGBench and one for Atari; discriminator learning rate is .
Grounding. Grounding [36] trains a goal-conditioned action model from random and plan-guided rollouts. Each training segment uses its final achieved image as the goal and executed actions as targets. Following an initial random-only fitting phase, replay sampling uses 30% random and 70% guided data. On OGBench, we follow the released LIBERO implementation, conditioning the model on a single current frame, with each guided collection following a single video plan without replanning. During evaluation, we periodically replan from the latest observation, with the number of plans set by each environment’s time limit. For DMC and Atari, we use three and four frame observation histories, respectively. To accommodate their longer task horizon, we replan during both collection and evaluation after every two subgoals in oracle planner experiments and after every subgoal in generative planner experiments.
We tune random-action hold duration and optimizer-update intervals between guided and random collection using 27-configuration, three-seed sweeps on Cube Single, Finger Spin, and Pong, ranked by the final five evaluations. Selected (hold, guided interval, random interval) values are (5, 100, 500), (10, 100, 500), and (5, 200, 1000) for OGBench, DMC, and Atari, respectively. OGBench collection intervals scale with task length.
Grounding’s learned planners use the shared architecture and training recipe with task-specific subgoal targets. DMC and Atari reuse the aggregated expert/recovery corpus, conditioning on three frames to predict an expert-continuation subgoal actions ahead. OGBench uses 1,000 expert episodes per task with a 950/50 training/validation split, conditions on one frame, and jointly predicts 7/14/20/70 subgoals for Cube Single/Cube Double/Scene/AntMaze Large. Following the original implementation, targets are spaced evenly from a sampled starting frame to shortly after expert success, with the final frame used for unsuccessful episodes.
Random-V2A. This baseline fits the same V2A model as Mirror independently on cumulative subsets of 100k to 1M transitions from a frozen uniformly random corpus. We hold out 5% of episodes and select checkpoints by validation flow loss. Evaluation translates visual plans into action chunks using deterministic V2A readout; generative-VP results re-evaluate the same fitted models with the learned planner in place of the oracle.
VIPER. VIPER [12] learns a generative video model from expert observations and uses observation likelihoods as an RL reward. We include it as an Atari-only likelihood-reward reference, using estimates from its published results.
Additional Results
Minecraft
Data Scale
We compare the data scale of our method with prior Minecraft agents in Table 4. The VPT BC and RL policies [3] were trained on significantly greater amounts of data taken from internet videos.
Basalt Challenge
Figure 18 contains an additional comparison to the BASALT challenge methods [38]. Episodes from each BASALT baseline were taken directly from [39].
Diamond Challenge under the Dreamer V3 Protocol
Figure 5 uses 60-minute episodes in which the player respawns after dying, as in VPT [3] and Dreamer V4 [19]. Dreamer V3 [18], and the PPO, Rainbow, and IMPALA results it reports, instead end each episode after 30 minutes or at the first death. Table 5 scores our self-improving agent under this protocol by cutting each of its 128 evaluation episodes at 30 minutes or at its first death, whichever comes first, which is equivalent to running them under Dreamer V3’s protocol. Among Mirror’s 128 episodes, 48 end early by death. Mirror still leads at every milestone except cobblestone. Mirror crafts an iron pickaxe in 8.6% of episodes, against 1.7% for Dreamer V3, even though the baselines use task rewards, abstract crafting actions, and faster block breaking.
| Milestone | PPO | Rainbow | IMPALA | Dreamer V3 | Mirror (self-improving) |
| Log | 75.2 | 74.4 | 71.0 | 95.0 | 99.2 |
| Planks | 74.6 | 73.3 | 69.7 | 95.0 | 99.2 |
| Crafting table | 69.8 | 65.4 | 61.8 | 91.3 | 99.2 |
| Stick | 74.2 | 72.3 | 69.3 | 94.6 | 99.2 |
| Wooden pickaxe | 51.2 | 48.4 | 45.5 | 76.9 | 94.5 |
| Cobblestone | 26.2 | 23.9 | 31.0 | 67.4 | 57.0 |
| Stone pickaxe | 5.7 | 4.7 | 10.1 | 39.5 | 53.1 |
| Iron ore | 0.4 | 0.7 | 1.4 | 12.7 | 22.7 |
| Furnace | 14.1 | 11.0 | 18.9 | 51.7 | 53.1 |
| Iron ingot | 0.0 | 0.3 | 0.9 | 7.0 | 18.8 |
| Iron pickaxe | 0.0 | 0.0 | 0.0 | 1.7 | 8.6 |
| Diamond | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 |
Table 5. Diamond Challenge under the Dreamer V3 protocol. Success rate (%) per milestone after about 10M environment steps, with 30-minute episodes that end at the first death. Baseline values are read from Figure 9 of [18]; Mirror is evaluated on 128 episodes.
Text-Conditioned Minecraft Evaluation
We evaluate text control in two worlds: Woodland and Camp. The worlds are designed to contain various locations of interest, such as trees, a pool, a hill, and a cave. In each world, the player starts with an iron sword, iron tools, torches, wood planks, bread, and a water bucket. We evaluate 10 episodes per text prompt. In each episode, we first make the player pan the camera left and right to observe their environment. Then it must complete each task given by the text prompt within 30 seconds. The video planner uses a classifier-free guidance scale of 7.0. We include more results on text-conditioned task guidance in Figure 14.

Figure 14. Text-conditioned planning on Camp. We demonstrate text-conditioned planning on the camp environment. While our self-improving approach is able to solve many tasks, it struggles on tasks with fine-grained environment interactions such as placing torches in the cave. Meanwhile, VIPER, our action-free baseline, struggles to learn the action space of Minecraft. BC solves some tasks, but remains limited in its generalization ability.
OGBench Generalization Analysis
Table 6 evaluates transfer to unseen OGBench goals, with all methods using oracle planners for the target tasks. Our V2A model outperforms Random-V2A in three of four environments and
BCO-V2A, the V2A model learned by BCO, in all four. These results suggest that video-guided exploration helps the model acquire behaviors that remain useful beyond the tasks used for data collection. Grounding achieves at most 3.8% success, suggesting that learning to translate frame-level transitions into actions supports stronger task generalization than learning a goal-conditioned policy.
Individual Task Learning Curves for Robotics and Atari
We report per-task results for DeepMind Control, OGBench, and Atari in Figure 15, Figure 16, and Figure 17, respectively.
Figure 15. Per-task performance on DeepMind Control. Episode return on 13 tasks. Curves show the mean ± one standard error over three seeds.
Figure 16. Per-task performance on OGBench. Success rate on four single-task environments. Curves show the mean ± one standard error over three seeds.
Figure 17. Per-game performance on Atari. Scores on seven games, expressed as a percentage of each game’s expert score. Curves show the mean ± one standard error over three seeds.
References
- [1]Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, et al. V-Jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint arXiv:2506.09985, 2025. 23, 25, 26arxiv.org/abs/2506.09985
- [2]Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunj iang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. 17arxiv.org/abs/2511.21631
- [3]Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video PreTraining (VPT): Learning to act by watching unlabeled online videos. In Advances in Neural Information Processing Systems (NeurIPS), 2022. 3, 6, 7, 8, 17, 20, 24, 26, 29, 30arxiv.org/abs/2206.11795
- [4]M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013. ISSN 1076-9757. doi: 10.1613/jair.3912. URL http://dx.doi.org/10.1613/jair.3912. 2, 10, 27
- [5]Homanga Bharadhwaj, Debidatta Dwibedi, Abhinav Gupta, Shubham Tulsiani, Carl Doersch, Ted Xiao, Dhruv Shah, Fei Xia, Dorsa Sadigh, and Sean Kirmiani. Gen2Act: Human video generation in novel scenarios enables generalizable robot manipulation. arXiv preprint arXiv:2409.16283, 2024. 2
- [6]Shaofei Cai, Bowei Zhang, Zihao Wang, Xiaojian Ma, Anji Liu, and Yitao Liang. Groot: Learning to follow instructions by watching gameplay videos. In International Conference on Learning Representations, volume 2024, pp. 5523–5554, 2024. 6arxiv.org/abs/2310.08235
- [7]Boyuan Chen, Diego Martí Monsó, Yilun Du, Max Simchowitz, Russ Tedrake, and Vincent Sitzmann. Diffusion forcing: Next-token prediction meets full-sequence diffusion. Advances in Neural Information Processing Systems, 37:24081–24125, 2024. 3arxiv.org/abs/2407.01392
- [8]Boyuan Chen, Tianyuan Zhang, Haoran Geng, Caiyi Zhang, Peihao Li, Kiwhan Song, William T. Freeman, Jitendra Malik, Pieter Abbeel, Russ Tedrake, Vincent Sitzmann, and Yilun Du. Large video planner enables generalizable robot control. arXiv preprint arXiv:2512.15840, 2025. 1, 2arxiv.org/abs/2512.15840
- [9]Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174, 2016. 19arxiv.org/abs/1604.06174
- [10]Juechu Dong, Boyuan Feng, Driss Guessous, Yanbo Liang, and Horace He. Flex attention: A programming model for generating optimized attention kernels. arXiv preprint arXiv:2412.05496, 2024. 17arxiv.org/abs/2412.05496
- [11]Yilun Du, Sherry Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 1, 2arxiv.org/abs/2302.00111
- [12]Alejandro Escontrela, Ademi Adeniji, Wilson Yan, Ajay Jain, Xue Bin Peng, Ken Goldberg, Youngwoon Lee, Danijar Hafner, and Pieter Abbeel. Video prediction models as rewards for reinforcement learning. In Advances in Neural Information Processing Systems, 2023. 6, 7, 9, 10, 27, 29arxiv.org/abs/2305.14343
- [13]Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024. 18 Preprint. Under review.arxiv.org/abs/2403.03206
- [14]Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. MineDojo: Building open-ended embodied agents with internet-scale knowledge. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2022.arxiv.org/abs/2206.08853
- [15]Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34: 4028–4039, 2021.arxiv.org/abs/2106.12142
- [16]Vittorio Giammarino, James Queeney, and Ioannis Ch. Paschalidis. Adversarial imitation learning from visual observations using latent information, 2024. URL https://arxiv.org/abs/2309.17371.
- [17]William H. Guss, Brandon Houghton, Nicholay Topin, Phillip Wang, Cayden Codel, Manuela Veloso, and Ruslan Salakhutdinov. MineRL: A large-scale dataset of Minecraft demonstrations. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), 2019.arxiv.org/abs/1907.13440
- [18]Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, 640:647 – 653, 2025a.DOI
- [19]Danijar Hafner, Wilson Yan, and Timothy Lillicrap. Training agents inside of scalable world models. arXiv preprint arXiv:2509.24527, 2025b.arxiv.org/abs/2509.24527
- [20]Ali Hassani, Steven Walton, Jiachen Li, Shen Li, and Humphrey Shi. Neighborhood attention transformer. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6185–6194. IEEE, 2023.arxiv.org/abs/2204.07143
- [21]Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.arxiv.org/abs/2207.12598
- [22]Bo-Ruei Huang, Chun-Kai Yang, Chun-Mao Lai, Dai-Jie Wu, and Shao-Hua Sun. Diffusion imitation from observation, 2024. URL https://arxiv.org/abs/2410.05429.
- [23]Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion. In Advances in Neural Information Processing Systems, 2025.arxiv.org/abs/2506.08009
- [24]Tero Karras, Miika Aittala, Tuomas Kynkäänniemi, Jaakko Lehtinen, Timo Aila, and Samuli Laine. Guiding a diffusion model with a bad version of itself. Advances in Neural Information Processing Systems, 37:52996–53021, 2024.arxiv.org/abs/2406.02507
- [25]Po-Chen Ko, Jiayuan Mao, Yilun Du, Shao-Hua Sun, and Joshua B. Tenenbaum. Learning to act from actionless videos through dense correspondences. In International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.08576
- [27]Muyao Li, Zihao Wang, Kaichen He, Xiaojian Ma, and Yitao Liang. Jarvis-vla: Post-training large-scale vision language models to play visual games with keyboards and mouse. In Findings of the Association for Computational Linguistics: ACL 2025, pp. 17878–17899, 2025.arxiv.org/abs/2503.16365
- [28]Sizhe Lester Li, Evan Kim, Xingjian Bai, Tong Zhao, Tao Pang, Max Simchowitz, and Vincent Sitzmann. Turning video models into generalist robot policies. arXiv preprint arXiv:2605.27817, 2026.arxiv.org/abs/2605.27817
- [29]Junbang Liang, Ruoshi Liu, Ege Ozguroglu, Sruthi Sudhakar, Achal Dave, Pavel Tokmakov, Shuran Song, and Carl Vondrick. Dreamitate: Real-world visuomotor policy learning via video generation. In Conference on Robot Learning (CoRL), 2024a. Preprint. Under review.arxiv.org/abs/2406.16862
- [30]Weixin Liang, Lili Yu, Liang Luo, Srinivasan Iyer, Ning Dong, Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen-tau Yih, Luke Zettlemoyer, et al. Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models. arXiv preprint arXiv:2411.04996, 2024b. 17arxiv.org/abs/2411.04996
- [31]Shalev Lifshitz, Keiran Paster, Harris Chan, Jimmy Ba, and Sheila McIlraith. STEVE-1: A generative model for text-to-behavior in Minecraft. In Advances in Neural Information Processing Systems (NeurIPS), 2023. 3, 6arxiv.org/abs/2306.00937
- [32]Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022. 18arxiv.org/abs/2210.02747
- [33]Minghuan Liu, Tairan He, Weinan Zhang, Shuicheng Yan, and Zhongwen Xu. Visual imitation learning with patch rewards, 2023. URL https://arxiv.org/abs/2302.00965. 2
- [34]Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003, 2022. 18arxiv.org/abs/2209.03003
- [35]Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 20arxiv.org/abs/1711.05101
- [36]Yunhao Luo and Yilun Du. Grounding video models to actions through goal conditioned exploration. ArXiv, abs/2411.07223, 2024. 2, 10, 29arxiv.org/abs/2411.07223
- [37]Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162, 2018. 19arxiv.org/abs/1812.06162
- [38]Stephanie Milani, Anssi Kanervisto, Karolis Ramanauskas, Sander Schulhoff, Brandon Houghton, Sharada Mohanty, Byron Galbraith, Ke Chen, Yan Song, Tianze Zhou, Bingquan Yu, He Liu, Kai Guan, Yujing Hu, Tangjie Lv, Federico Malato, Florian Leopold, Amogh Raut, Ville Hautamäki, Andrew Melnik, Shu Ishida, João Henriques, Robert Klassert, Walter Laurito, Lucas Cazzonelli, Cedric Kulbach, Nicholas Popovic, Marvin Schweizer, Ellen Novoseller, Vinicius Goecks, Nicholas Waytowich, David Watkins, Josh Miller, and Rohin Shah. Towards solving fuzzy tasks with human feedback: A retrospective of the minerl basalt 2022 competition. In Marco Ciccone, Gustavo Stolovitzky, and Jacob Albrecht (eds.), Proceedings of the NeurIPS 2022 Competitions Track, volume 20 of Proceedings of Machine Learning Research, pp. 171–188. PMLR, 28 Nov–09 Dec 2022. URL https://proceedings.mlr.press/v220/milani23a.html. 6, 7, 23, 29arxiv.org/abs/2303.13512
- [39]Stephanie Milani, Anssi Kanervisto, Karolis Ramanauskas, Sander Schulhoff, Brandon Houghton, and Rohin Shah. BEDD: The MineRL BASALT evaluation and demonstrations dataset for training and benchmarking agents that solve fuzzy tasks. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2023. 29arxiv.org/abs/2312.02405
- [40]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning, 2013. URL https://arxiv.org/abs/1312.5602. 28
- [41]Lorenzo Mur-Labadia, Matthew Muckley, Amir Bar, Mido Assran, Koustuv Sinha, Michael Rabbat, Yann LeCun, Nicolas Ballas, and Adrien Bardes. V-jepa 2.1: Unlocking dense features in video self-supervised learning. ArXiv, abs/2603.14482, 2026. URL https://api.semanticscholar.org/CorpusID:286527317. 23, 25, 26
- [42]Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłos, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample-efficient continuous control, 2024. URL https://arxiv.org/abs/2405.16158. 28
- [43]NVIDIA. Cosmos 3: Omnimodal world models for physical ai. ArXiv, abs/2606.02800, 2026. 5, 17, 21, 24
- [44]Jonas Pai, Liam Achenbach, Victoriano Montesinos, Benedek Forrai, Oier Mees, and Elvis Nava. mimic-video: Video-action models for generalizable robot control beyond VLAs. arXiv preprint arXiv:2512.15692, 2025. 2 Preprint. Under review.
- [45]Pantograph. A general goal-conditioned minecraft model. Pantograph Research, July 2026. URL https://pantograph.com/journal/pan-1. Accessed: 2026-09-04.
- [46]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. Ogbench: Benchmarking offline goal-conditioned rl, 2025a. URL https://arxiv.org/abs/2410.20092.
- [47]Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning, 2025b. URL https://arxiv.org/abs/2502.02538.
- [48]Rhoda. Causal video models are data-efficient robot policy learners. Rhoda AI Research blog, 2026. URL https://www.rhoda.ai/research/direct-video-action.
- [49]Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning, 2011. URL https://arxiv.org/abs/1011.0686.
- [50]Dominik Schmidt and Minqi Jiang. Learning to act without actions, 2024. URL https://arxiv.org/abs/2312.10812.
- [51]Benjamin Schneider, Xavier Schneider, Victor Zhong, and Sun Sun. Ash: Agents that self-hone via embodied learning. arXiv preprint arXiv:2605.14211, 2026.
- [52]Rohin Shah, Cody Wild, Steven H. Wang, Neel Alex, Brandon Houghton, William Guss, Sharada Mohanty, Anssi Kanervisto, Stephanie Milani, Nicholay Topin, Pieter Abbeel, Stuart Russell, and Anca Dragan. The mineral basalt competition on learning from human feedback. 2021. URL https://arxiv.org/abs/2107.01969.
- [53]Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.arxiv.org/abs/2104.09864
- [54]Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. Deepmind control suite, 2018. URL https://arxiv.org/abs/1801.00690.
- [55]Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.19786.
- [56]Yang Tian, Sizhe Yang, Jia Zeng, Ping Wang, Dahua Lin, Hao Dong, and Jiangmiao Pang. Pre-dictivie inverse dynamics models are scalable learners for robotic manipulation. In International Conference on Learning Representations (ICLR), 2025.DOI
- [57]Philippe Tillet, Hsiang-Tsung Kung, and David Cox. Triton: an intermediate language and compiler for tiled neural network computations. In Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19, 2019.DOI
- [58]Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation, 2018. URL https://arxiv.org/abs/1805.01954.
- [59]Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation, 2019. URL https://arxiv.org/abs/1807.06158.
- [60]Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning, 2015. URL https://arxiv.org/abs/1509.06461.
- [62]Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents. In Advances in Neural Information Processing Systems (NeurIPS), 2023. Preprint. Under review.
- [63]Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang. JARVIS-1: Open-world multi-task agents with memory-augmented multimodal language models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. arXiv:2311.05997. 3arxiv.org/abs/2311.05997
- [64]Zihao Wang, Muyao Li, Kaichen He, Xiangyu Wang, Zhancun Mu, Anji Liu, and Yitao Liang. Openha: A series of open-source hierarchical agentic models in minecraft. arXiv preprint arXiv:2509.13347, 2025. 6, 10arxiv.org/abs/2509.13347
- [65]Ziyu Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, and Nando de Freitas. Dueling network architectures for deep reinforcement learning, 2016. URL https://arxiv.org/abs/1511.06581. 28
- [66]Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, and Minjoon Seo. Latent action pretraining from videos, 2025. URL https://arxiv.org/abs/2410.11758. 2
- [67]Seonghyeon Ye, Yunhao Ge, Kaiyuan Zheng, Shenyuan Gao, Shiyun Yu, George Kurian, Suneel Indupuru, You Liang Tan, Chuning Zhu, Jiannan Xiang, et al. World action models are zero-shot policies. arXiv preprint arXiv:2602.15922, 2026. 1arxiv.org/abs/2602.15922
- [68]Ted Zadouri, Markus Hoehnerbach, Jay Shah, Vijay Thakkar, and Tri Dao. Flashattention-4: Algorithm and kernel pipelining co-design for asymmetric hardware scaling. Proceedings of Machine Learning and Systems, 8:912–926, 2026. 17arxiv.org/abs/2603.05451
- [69]Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipc: A unified predictor-corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 36:49842–49869, 2023a. 20arxiv.org/abs/2302.04867
- [70]Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Schleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.11277, 2023b. 19arxiv.org/abs/2304.11277
- [71]Hongzhou Zhu, Min Zhao, Guande He, Hang Su, Chongxuan Li, and Jun Zhu. Causal forcing: Autoregressive diffusion distillation done right for high-quality real-time interactive video generation. arXiv preprint arXiv:2602.02214, 2026. 3arxiv.org/abs/2602.02214













