WorksComputer Science
ReLAP: Latent Action Pretraining through Recurrent Video Prediction in JEPA Space
Abstract
Videos provide a rich source of physical interaction for learning robot behavior without action annotations, yet effective latent actions must capture not only local transitions but also how those transitions evolve over time. We introduce ReLAP, a latent-action pretraining framework that couples local transition learning with recurrent multi-step prediction in a frozen JEPA representation space. A shared inverse dynamics model infers compact continuous latent actions from adjacent observations, while a latent world model recurrently predicts the future JEPA features. Because each predicted feature becomes the context for the next transition, errors at later steps backpropagate through the rollout to earlier latent actions, encouraging temporally coherent dynamics beyond pairwise reconstruction. We transfer this representation to a vision-language model using two complementary objectives: latent distillation from the frozen inverse teacher and recurrent future-feature prediction through the frozen latent world model. The resulting policy is then adapted to robot demonstrations with action flow matching. At deployment, the explicit latent-action and world-model branches are removed, and the fine-tuned VLM predicts physical actions directly, reducing inference complexity. With only 11M pretraining samples and 1.06B trainable parameters, ReLAP achieves 97.9% on standard LIBERO and 70.0% over zero-shot LIBERO-Plus perturbations. On the LARY latent-action benchmark, ReLAP ranks second in both mean regression error and mean classification accuracy among the compared methods. These results demonstrate that recurrent action-free latent action pretraining can learn compact, temporally structured, and transferable representations for downstream robot control.
Introduction
Videos of human and robot interaction provide abundant supervision for robot learning without requiring physical action annotations. Latent action models exploit such action-free data by inferring compact pseudo-actions from visual transitions. A typical latent action model combines an inverse dynamics model, which infers a latent action from two consecutive observations, with a forward dynamics model that reconstructs the resulting transition. These inferred actions can then serve as supervision for pretraining vision-language-action policies before adaptation to action-labeled robot demonstrations. While effective, pairwise transition reconstruction does not explicitly require consecutive latent actions to remain compatible over time. Each transition is typically reconstructed from its observed starting state, so accurate one-step predictions do not necessarily yield an accurate multi-step trajectory when predicted states become the context for later transitions. This distinction is particularly important for manipulation: a reach changes the state from which a grasp begins, and the grasp changes the state from which a lift is executed. Recent work addresses this temporal

Figure 1. (a) Conventional latent action models infer a latent action from an observed transition and reconstruct its immediate consequence. (b) ReLAP infers a sequence of continuous latent actions and recurrently rolls them out in JEPA feature space, allowing later prediction errors to supervise earlier latent actions and encouraging temporal coherence across the sequence.
structure in different ways. RotVLA introduces structured continuous latent actions, while ALAM and related compositional latent-action models impose explicit consistency constraints on how latent transitions combine. Other approaches use future prediction or world modeling to enrich latent-action supervision. In contrast, we encourage temporal coherence through the predicted consequences of the latent sequence itself, without prescribing an explicit algebraic composition rule in latent space. Specifically, we recurrently roll out the inferred latent actions in a frozen JEPA representation space, where each predicted feature becomes the input to the next transition. This allows errors at later horizons to propagate through the predicted trajectory and supervise earlier latent actions. Combined with local transition reconstruction, this objective preserves the grounding of each latent to its observed transition while encouraging the full sequence to remain temporally coherent under multi-step prediction.
We introduce ReLAP, a framework for latent-action pretraining through recurrent video prediction in JEPA space (Fig. 1). A shared inverse dynamics model infers a continuous latent action from each adjacent observation pair encoded by a frozen V-JEPA 2.1 encoder [1]. Starting from the first observed feature map, a shared forward dynamics model recurrently applies the inferred latent sequence, using each predicted feature as the input to the next transition. We supervise every rollout step and backpropagate through the predicted intermediate states, allowing later prediction errors to constrain earlier latent actions. A local objective anchors each latent to its observed transition, while recurrent prediction encourages the sequence to remain temporally coherent under its own predicted dynamics. In Stage 2, a vision-language model predicts the complete latent sequence from the current image and language instruction in a single forward pass. The inverse and forward models remain frozen: latent distillation aligns the VLM predictions with the video-inferred actions, while recurrent future-feature supervision evaluates their cumulative effects under the same learned dynamics. In Stage 3, robot demonstrations introduce physical-action supervision through flow matching, transferring the pretrained transition knowledge to executable control. The latent rollout spans 1.6 s, while the policy executes a 0.4 s action chunk before replanning. Future observations and the inverse teacher are used only during training; at deployment, the explicit latent-action and JEPA dynamics branches are discarded, and the fine-tuned VLM directly predicts physical actions. We evaluate the downstream control and rigorously analyze the learned latent action representations. ReLAP achieves 97.9% success over 2,000 episodes on standard LIBERO [2] and 70.0% pooled success over 10,030 zero-shot LIBERO-Plus perturbations [3]. On the LARY benchmark [4], ReLAP obtains 0.24 mean regression MSE and 65.76% mean classification accuracy, ranking second among the SOTA latent-action models. Targeted representation analyses further show that the learned latents are sample-specific, temporally structured, and aligned with corresponding physical actions. Our contributions are threefold:
Recurrent latent-action learning in JEPA space. We couple local transition reconstruction with recurrent rollout prediction in frozen V-JEPA 2.1 features, allowing later rollout errors to supervise earlier video-inferred latent actions through predicted intermediate states.
Dynamics-aware transfer to a VLM. We combine latent distillation with recurrent future-feature supervision to transfer the discovered latent actions in stage 1 to VLM pretraining and finetuning, thereby leveraging the benefits of both inverse dynamics model and forward model of stage 1.
Comprehensive representation and control evaluation. We evaluate ReLAP on LIBERO, zero-shot LIBERO-Plus, and LARY, together with targeted probes of sample specificity, temporal structure, and physical-action alignment.
Related Work
Latent Action Learning from Action-Free Video
Latent action models (LAMs) provide a scalable way to learn action-like representations from videos without physical action annotations. Early approaches such as LAPA learn discrete latent actions through inverse–forward reconstruction and use them as pseudo-action targets for VLA pretraining [5, 6, 7]. Subsequent work explored richer representations by exploring task-centric discrete actions in DINO feature space [8], while CoMo [9] and Garrido et al. [10] show the benefits of continuous latent actions for capturing fine-grained motion and diverse in-the-wild dynamics. JALA [11] instead avoids explicit observation reconstruction and aligns predictive action embeddings with inverse dynamics and physical actions. Most of these approaches primarily optimize the quality of individual transitions or the predictability of latent actions. ReLAP instead focuses on whether a sequence of video-inferred actions remains predictive when its own earlier consequences become the context for later transitions. This introduces recurrent supervision directly into latent-action discovery rather than only into the downstream policy.
Temporal Structure and Future Prediction
Recent works highlight that independently reconstructed latent actions need not form a coherent sequence. Moto [12] models latent motion tokens autoregressively, while ViPRA jointly predicts future visual observations and fine-grained latent-action sequences and regularizes the underlying motion representation with perceptual and optical-flow objectives [13]. Other approaches impose explicit structure directly in the latent space. AC-LAM [14] enforces additive composition and inverse consistency; RotVLA [15] represents actions on a rotational manifold and uses frame triplets to capture temporal relations; and ALAM [16] and DLAM [17] impose composition and reversal constraints on deterministic and distributional latent transitions, respectively. These approaches establish the importance of temporal structure, but largely encode it either through explicit algebraic constraints. ReLAP takes a complementary view: temporal compatibility is defined by the predicted consequences of the latent sequence. Each action is applied to the feature state produced by the preceding action, and losses at later horizons backpropagate through these predicted states to earlier inferred actions. Thus, temporal structure emerges through learned state-dependent dynamics without adding additional complexities of additive, rotational, or other fixed composition rule in latent space.
Future prediction provides another route for strengthening latent-action learning. LAW-M [18] uses world modeling as supervision for latent-action pretraining, while CoLA-World [19] and CLAW [20] jointly optimize latent actions with expressive world models, allowing predictive losses to shape the learned action interface. ViPRA [13] similarly combines latent-action prediction with future visual prediction. ReLAP is closest in spirit to this line of work, but differs in where the temporal constraint is applied: rather than evaluating future prediction only from observed context, it recurrently feeds predicted intermediate states back into the forward model, explicitly coupling earlier latent actions to the accuracy of later transitions.
Feature-Space Dynamics for Robot Learning
Predicting in pretrained visual feature spaces avoids the cost and ambiguity of reconstructing every pixel. DINO-WM [21] predicts future DINO features conditioned on physical actions for test-time planning, while V-JEPA 2-AC [22] learns action-conditioned dynamics in V-JEPA representations from robot interaction data. In latent-action learning, UniVLA [8] moves reconstruction from pixels to DINO features. Most closely related to ReLAP, Garrido et al. [10] jointly learn a continuous inverse model and world model in frozen V-JEPA 2-L features from action-free video. Their forward model, however, is trained with teacher forcing, so each next-state prediction is conditioned on observed history rather than the model’s recursively predicted intermediate states. JEPA representations are also increasingly used to provide predictive supervision directly to robot policies. VLA-JEPA [23] predicts future latent states without pixel generation, while JEPA-WAM [24] couples transition prediction and continuous action generation in pretrained V-JEPA space. FLARE [25] similarly aligns policy representations with future latent observations. These methods demonstrate the value of predictive feature-space supervision, but do not address the same problem as ReLAP: learning a sequence of latent actions from action-free video by allowing future rollout errors to shape the inverse model that inferred the earlier actions.
ReLAP lies at the intersection of three emerging directions: label-free latent-action discovery, temporally structured latent actions, and predictive world modeling in pretrained feature space. Prior work has explored each component separately—including future prediction, explicit latent composition, and JEPA-space dynamics—but their intersection remains less studied. ReLAP combines them through a simple recurrent principle: local transition reconstruction grounds each latent action, while multi-step rollout through predicted JEPA states couples the sequence through its decoded visual effects. This provides temporal supervision without physical action labels, pixel reconstruction, or an explicit algebraic composition rule.
Proposed Approach
ReLAP learns latent actions from action-free video by requiring their predicted latent actions to remain coherent over a sequence. The central mechanism is a shared latent world model that consumes its own previous feature prediction at each step. Figure 2 summarizes the three stages: learning this representation from video, predicting it from the current observation and instruction, and adapting it to robot control. Physical action labels are used only in the third stage.

Figure 2. Overview of ReLAP. Stage 1 learns temporally coherent latent actions from unlabeled human and robot video through local and recurrent prediction in frozen JEPA features. Stage 2 trains a vision-language model to predict the latent sequence from the current image and instruction under the frozen Stage-1 models. Stage 3 adds a flow-matching action expert of alternating DiT blocks conditioned on the VLM context, the predicted latent actions, and the current and predicted future JEPA features; the forward model stays trainable under the terminal feature loss.
Stage 1: Learning Latent Actions through Recurrent Prediction in JEPA Space
Stage 1 learns compact latent actions that explain local visual transitions while remaining consistent over a multi-step rollout. Given a sequence of observations , we extract patch features using a frozen V-JEPA 2.1 encoder [1]. We use transitions sampled at 0.4 s intervals and perform all prediction directly in the JEPA representation space. For each adjacent feature pair, the inverse model infers a -dimensional continuous latent action. Each latent is inferred independently from its corresponding transition. A low-dimensional bottleneck together with KL regularization toward encourages a compact representation of the observed visual change.
The forward latent world model predicts how a latent action transforms the current feature map. We first optimize a local transition loss, which reconstructs each observed transition independently by minimizing the Smooth-L1 distance between and , with sampled from the posterior. This anchors each latent action to its corresponding local transition. Local reconstruction alone, however, does not require consecutive latent actions to remain compatible when applied sequentially. We therefore introduce recurrent prediction. Starting from the first observed feature, the model recursively applies the inferred latent sequence,
The rollout loss measures the Smooth-L1 distance between each predicted feature and its corresponding target . Gradients are propagated through the entire rollout, allowing later prediction errors to supervise earlier latent actions through the intermediate predicted states. Consequently, the latent sequence is encouraged to capture transitions whose effects remain predictive as they become the context for subsequent transitions, without imposing an explicit composition rule in latent space. The complete Stage 1 objective is
where denotes the average KL divergence between the latent posterior and . We gradually introduce after an initial period of local transition learning and ramp early in training. The visual encoder remains frozen throughout Stage 1. Because the inverse model observes both sides of each transition, Stage 1 learns a representation of observed visual dynamics rather than predicting future actions from the current observation.
Stage 2: Latent Action Pretraining with VLM
Stage 2 trains a vision-language model (VLM) to predict the latent sequence from the current main-camera image and instruction . We denote the resulting student predictor by . A Q-Former head uses learned latent queries to attend to the VLM hidden states and predicts all latent actions jointly in a single forward pass. Unlike autoregressive decoding, the latent queries use bidirectional self-attention, allowing each slot to interact with the others and encouraging the predicted latent actions to form a temporally coherent sequence. Future observations are used only to construct training targets and are not available to the student input. Stage 2 is supervised by two complementary objectives: latent distillation and future-feature prediction. We freeze the visual encoder , inverse model , and forward latent world model . The posterior means produced by serve as latent-action targets, while the student-predicted sequence is recursively rolled out through the frozen forward model:
The Stage 2 objective is
where the first term aligns the VLM predictions with the Stage 1 latent-action space, while the second evaluates whether those predictions reproduce the corresponding future V-JEPA feature trajectory. These two objectives play complementary roles. Latent distillation preserves the semantic coordinates learned by the inverse model, whereas future-feature prediction constrains the predicted latent sequence through its cumulative visual effects. Although remains frozen, gradients propagate through the full recursive rollout to the Q-Former and VLM, encouraging latent actions that are individually consistent with the teacher and jointly predictive of future visual dynamics. The rollout loss is introduced gradually after initial latent distillation to stabilize optimization. Since the inverse teacher observes future frames, its posterior latents provide training references rather than unique action targets: multiple latent trajectories may be compatible with the same current observation and instruction. Stage 2 therefore learns a future-free latent prior that predicts a coherent transition sequence from the current visual-language context alone, without using physical-action supervision.
Stage 3: Adapting Latent Representations to Robot Control
Stage 3 adapts the pretrained representation to physical robot control using action-labeled demonstrations. The policy receives the current main-camera image, wrist image, and language instruction. The latent queries are positioned before the wrist-view tokens in the causal VLM sequence, so latent prediction depends only on the main view and instruction, while the action pathway can additionally exploit wrist-view information. The Stage 2 Q-Former is retained, with bidirectional attention among the four latent queries to preserve sequence-level consistency. During training, the predicted latent sequence is recurrently decoded by the Stage 1 forward model to obtain its anticipated future JEPA representation.
A flow-matching action head then predicts an eight-step, seven-dimensional action chunk from the VLM context. We additionally expose the action head to the predicted latent sequence and current/future JEPA features during training, providing complementary semantic and visual-dynamics context. For a demonstration action chunk , we train the action head with standard conditional flow matching. Using a noisy interpolation between Gaussian noise and , the model predicts the corresponding velocity . The complete Stage 3 objective is
where preserves alignment with the frozen Stage 1 inverse teacher and matches the predicted terminal JEPA feature to the observed future feature. We use . Thus, Stage 3 jointly optimizes executable action prediction while retaining the transition structure acquired during action-free pretraining. The visual encoder and inverse teacher remain frozen, while the VLM, latent prediction head, forward model, and action head are adapted to robot demonstrations. Importantly, the latent-action and JEPA world-model branches serve as training-time supervision rather than mandatory deployment modules. At inference, we retain only the fine-tuned VLM direct-action pathway and discard the latent decoder and visual-dynamics attention branches. Future observations and the inverse teacher are therefore never required at deployment, yielding a simpler and lower-latency policy.
Experimental Setup
Datasets. Stages 1 and 2 train on a mixture of seven large-scale video datasets: Bridge-v2 [26], DROID [27], AgiBot [28], EgoDex [29], Airoa-MOMA [30], Intern-Data [31], and Ego4D [32]. These datasets provide diverse demonstrations of physical interactions in both human and robot environments. In stage 1, we use only the raw video streams without accessing any robot action annotations, proprioceptive signals, or language instructions, ensuring that the latent actions learned are derived exclusively from visual dynamics. We follow the preprocessing strategy of OSCAR [33] to process raw videos, standardizing frame rates and resolutions while preserving the temporal integrity of the interactions. Specifically, videos are sampled at 20 Hz, and each training clip consists of 5 frames spaced 0.4 seconds apart, yielding 11,127,635 training windows as shown in Figure 3. In stage 3, we use the LIBERO benchmark suite [2], which includes 40 tasks across four suites: LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, and LIBERO-10, resulting in 2K demonstrations and perform downstream evaluation of LIBERO and LIBERO plus benchmarks.
Figure 3. Stage-1 video mixture. Outer ring: each source’s share of the 11.1M available 1.6-second training windows. Inner ring: its share of sampled training examples under the mixture weights.
Implementation Details. ReLAP learns latent actions in the V-JEPA representation space. We use a frozen V-JEPA 2.1 ViT-B [1] encoder on RGB frames, producing 256 visual tokens per frame. In Stage 1, the inverse dynamics model consists of 24 transformer layers and the forward latent world model uses six transformer layers, both with hidden width 1024. Each transition is represented by a 16-dimensional continuous latent action, and four consecutive transitions span a 1.6 s clip. Stage 1 is trained for 100k steps using AdamW with a learning rate of and an effective batch size of 512. The recurrent prediction loss weight is gradually increased to 2, while the KL coefficient is ramped to early in training for stable optimization. In Stage 2, we initialize from Qwen3-VL-2B and retain its first 16 text Transformer blocks. Eight learned latent-query tokens interact with the backbone through a single-layer Q-Former. Stage 2 is trained with an effective batch size of 256, with the recurrent prediction objective gradually introduced from 5k steps. For Stage 3, we initialize the latent-action model from the final Stage 1 checkpoint and the vision-language backbone from the 80k-step Stage 2 checkpoint, while initializing the action expert from scratch. The action expert is a 16-block flow-matching Transformer with hidden width 1024. The policy receives the current main-camera image, wrist-camera image, and language instruction; no proprioceptive state is used. In the causal input sequence, latent queries are placed before the wrist-view tokens, such that latent actions are predicted only from the main view and language instruction, while the action expert can additionally access wrist-view information through the full backbone hidden sequence. Stage 3 is optimized using AdamW with a learning rate of , batch size 256, 1,500 warmup steps, and cosine decay to . Gradients are clipped to a maximum norm of 1. The loss weights are 1.0 for action flow matching, 0.1 for terminal future-feature prediction, and 0.1 for latent-action distillation. Throughout Stage 3, the visual encoder and inverse dynamics model remain frozen, while the forward latent world model, latent prior, action expert, and VLM backbone are optimized.
Evaluation Benchmarks We evaluate on LIBERO [10] and LIBERO-Plus [11]. LIBERO has four suites, Spatial, Object, Goal, and Long (the libero 10 suite), each with ten tasks. We run 50 rollouts per task, giving 2,000 episodes. LIBERO-Plus perturbs the same tasks along seven axes: camera viewpoint, robot initial state, language instruction, lighting, background texture, sensor noise, and object layout.
| Method | Spatial | Object | Goal | Long | Average |
| Pretraining with physical action supervision | |||||
| VLA-JEPA [23] | 96.2 | 99.9 | 97.2 | 95.8 | 97.2 |
| JALA [11] | 96.0 | 98.2 | 97.4 | 96.0 | 96.9 |
| Action-free latent pretraining | |||||
| LAPA [5] | 73.8 | 74.6 | 58.8 | 55.4 | 65.7 |
| UniVLA [8] | 96.5 | 96.8 | 95.6 | 92.0 | 95.2 |
| ViPRA [13] | — | — | — | — | |
| RotVLA [15] | 98.2 | 99.6 | 98.4 | 96.4 | 98.2 |
| ALAM [16] | 99.2 | 99.6 | 99.0 | 94.4 | 98.1 |
| CoMo [9] | 80.3 | 97.0 | 81.0 | 62.0 | 80.1 |
| MoLA [34] | 93.0 | 99.5 | 99.5 | 96.0 | 97.0 |
| ReLAP (ours) | 99.6 | 99.0 | 97.8 | 95.2 | 97.9 |
Table 1. Performance comparison on LIBERO.
It contains 10,030 perturbed task instances, and following the benchmark convention we run each once. The same Stage-3 checkpoint is used on both benchmarks with no fine-tuning on perturbed tasks, evaluating the zero-shot generalization to LIBERO plus. Because the Stage-3 demonstrations cover the same 40 tasks that LIBERO evaluates, standard LIBERO measures down-stream task performance, while LIBERO-Plus measures how far that performance survives distribution shift.
Results and Discussion
To evaluate the proposed approach, we conducted extensive experiments to understand the robustness of the latent actions of ReLAP and whether it is translated to downstream policy performance.
Comparison with State-of-the-Art Methods
Tables I–III compare ReLAP with recent latent-action models on standard LIBERO and LIBERO-Plus. Furthermore, we also compare the analysis of latent action representations on the LARY benchmark [4]. LIBERO. ReLAP achieves 97.9% average success on LIBERO (1,958/2,000 episodes) which is within 0.3 percentage points of RotVLA (98.2%) and 0.2 points of ALAM (98.1%), while exceeding VLA-JEPA (97.2%), MoLA (97.0%), JALA (96.9%), and UniVLA (95.2%). ReLAP obtains the highest spatial score among the compared methods and remains competitive across the other suites. The remaining failures are concentrated in LIBERO-Long, indicating that long-horizon execution remains the primary limitation under the standard benchmark. It is worth mentioning that ReLAP uses 11M pretraining samples which is substantially less than other competitive methods such as RotVLA, and ALAM. Furthermore, we use a 2B-parameter model, of which only 1.06B parameters are optimized compared to other large size models such as 7B UniVLA.
Table 2. Performance comparison on LIBERO Plus.
| Probe | Bridge (ID) | LIBERO-40 (OOD) |
| Dependence on the inferred latents (prediction error, %) | ||
| Global mean latent | +34.80 | +28.65 |
| Zero latent | +77.70 | +99.67 |
| Swapping different/same visual transition | 0.439/0.074 | 0.471/0.211 |
| Temporal structure (prediction error, %) | ||
| Sequence reversal | +4.03 | +2.50 |
| Single-shot corruption | +6.03–13.52 | +5.73–6.70 |
| Physical action Correspondence | ||
| Matched / mismatched action | 0.360/0.225 | 0.530/0.310 |
Table 4. Latent action probe analysis of ReLAP.
LIBERO-Plus. Without any additional adaptation, we evaluate ReLAP zero-shot on LIBERO-Plus and obtain a 70.0% macro-average success rate across the seven perturbation categories. ReLAP is particularly robust to robot initial-state (74.9%) and object-layout (90.1%) perturbations, achieving the highest reported performance in both categories, while remaining competitive under language (82.8%), lighting (95.5%), and background (90.8%) variations. The primary robustness gap is concentrated in camera viewpoint and sensor noise, where success drops to 20.5% and 52.8%, respectively. Consequently, ReLAP remains below the reported aggregate performance of VLA-JEPA (79.5%), and LAWA (74.4%), despite being competitive across most other perturbation types. This limitation suggests that ReLAP is substantially more robust to changes in task semantics and scene configuration than to perturbations that directly alter the visual observation distribution. We hypothesize that ReLAP’s sensitivity to viewpoint and sensor perturbations can be attributed by its Stage 2 pretraining setup, where the VLM observes only the main camera view. Thus, although Stage 3 introduces wrist-view information, the pretrained vision-language representation has not been explicitly optimized for cross-view consistency. The same qualitative weakness is visible in UniVLA’s LIBERO-Plus results, which also degrade sharply under camera and sensor perturbations. In contrast, VLA-JEPA explicitly supports multi-view visual
conditioning during training, including two-view world-model inputs, which may contribute to its stronger robustness under camera variation.
LARYBench. Table 3 evaluates latent-action quality independent of closed-loop policy performance. Under the LARY linear-probing protocol, ReLAP achieves an average regression MSE of 0.24 and classification accuracy of 65.76%, second only to RotVLA (0.20 and 70.98%, respectively). ReLAP achieves better regression error on CALVIN (0.28) and VLABench (0.07). Its lower human-action classification accuracy relative to RotVLA may partly reflect ReLAP’s pretraining mixture, which is dominated by robot-manipulation data. The regression gap on RoboCOIN and AgiBotWorld should also be interpreted in light of pretraining exposure: RotVLA includes both datasets in its pretraining mixture, whereas ReLAP uses neither. These domains therefore constitute genuine cross-dataset transfer tests for ReLAP. Despite this distribution shift, ReLAP achieves MSEs of 0.51 on RoboCOIN and 0.10 on AgiBotWorld, while remaining close to RotVLA in average regression performance (0.24 vs. 0.20). Together, the LARY results indicate that ReLAP learns compact, action-relevant transition representations that generalize beyond its pretraining distribution.
| Method | CALVIN | VLAB. | RoboC. | AgiBot | Avg. | Robot | Human | Avg. |
| Regression MSE ↓ | Classification acc. (%) ↑ | |||||||
| LAPA | 0.96 | 0.95 | 0.96 | 1.00 | 0.97 | 23.64 | 14.61 | 19.13 |
| UniVLA | 0.82 | 0.74 | 0.94 | 0.97 | 0.87 | 18.56 | 19.08 | 18.82 |
| villa-X | 0.86 | 0.72 | 0.94 | 0.97 | 0.87 | 29.90 | 17.80 | 23.85 |
| LAPA-DINOv3 | 0.50 | 0.25 | 0.82 | 0.84 | 0.63 | 27.04 | 64.19 | 45.62 |
| RotVLA | 0.30 | 0.08 | 0.35 | 0.07 | 0.20 | 67.62 | 74.33 | 70.98 |
| ReLAP (ours) | 0.28 | 0.07 | 0.51 | 0.10 | 0.24 | 65.69 | 65.83 | 65.76 |
Table 3. Comparision of latent action probes of different latent action methods on LARY benchmark [4].
Latent Action Representation Analysis
A key challenge in latent action learning is to capture controllable, semantically meaningful transition dynamics from visual observations without merely encoding future-state information or action-irrelevant visual changes [36]. We therefore analyze whether ReLAP latents are necessary for future prediction, sample-specific, temporally structured, and aligned with physical actions as shown in Table IV. We evaluate on 1,536 held-out Bridge windows as an in-distribution (ID) test set and 2,936 LIBERO-40 windows as out-of-distribution (OOD) transfer set unseen during Stage-1 training.
Latent dependency and sample specificity. We first test whether the recurrent world model genuinely depends on transition-specific latent information. We replace the inferred sequence with three controls: global mean, which uses the dataset-average latent; and zero latent, which removes latent conditioning entirely. These interventions increase latent world model prediction error by 34.80%, and 77.70% on Bridge, and by 28.65%, and 99.67% on LIBERO-40, respectively. Thus, accurate future prediction requires not merely a latent input, but the transition-specific information carried by the inferred sequence. We further test sample specificity by substituting a latent action from a different transition, and found that the latent world predictions degrade with latents from different transitions (0.439/0.074) on Bridge and (0.471/0.211) on LIBERO-40., highlighting the sample specific latent actions of ReLAP.
Temporal Coherence We first test whether the four latent slots form a temporally coherent sequence. Reversing the latent order while preserving the same four latent vectors increases rollout error by 4.03% on Bridge and 2.50% on LIBERO-40. Corrupting a single latent slot causes a larger degradation of 6.03–13.52% and 5.73–6.70%, respectively. These results indicate that the representation is temporally aligned and slot-specific: each latent captures transition information associated with a particular temporal interval, while global ordering has a smaller but measurable effect.
Physical-action correspondence We next evaluate whether the temporal structure of the latent actions aligns with robot actions. For each latent slot , we linearly predict each of the four future action segments. Temporally matched latent–action pairs explain 36.0% of the action variance on Bridge, compared with 22.5% for mismatched pairs, and 53.0% versus 31.0% on LIBERO-40. Thus, each latent is preferentially predictive of the physical action occurring during the same temporal interval, providing direct evidence that the learned latent sequence is aligned with the temporal structure of robot behavior.
Qualitative Analysis of Results
Across the examples, the policy successfully executes diverse multi-stage manipulation behaviors. In Figure 4(a), the robot first interacts with the stove and subsequently transports the moka pot to the target location, requiring sequential interaction with two distinct objects. Figure 4(b) demonstrates object-specific grasping and spatial placement of the book into the designated compartment of the caddy. Figure 4(c) further illustrates a longer compositional task in which two different objects must be sequentially manipulated and placed into the basket. These rollouts qualitatively demonstrate that the final VLM policy can ground language instructions into temporally coordinated, object-centric manipulation behaviors, including grasping, transport, placement, and multi-object sequencing. Notably, these behaviors are produced through direct VLM-to-action prediction at deployment, suggesting that the transition structure learned during latent-action pretraining can be transferred into the downstream policy without requiring the explicit latent-action pathway at inference.

Figure 4. Representative ReLAP rollouts on three LIBERO-Long tasks, viewed from the main camera. Each row shows six frames of one successful episode in temporal order.
Discussion and Limitations
Our results also reveal two important limitations of the current ReLAP design. First, recurrent latent rollout introduces an inherent temporal asymmetry across the latent sequence. Earlier latent actions influence not only their immediate transition, but also all subsequent predictions through the recurrent rollout, and therefore receive stronger effective supervision than later latent actions. This bias can favor accurate short-horizon dynamics while making errors increasingly difficult to correct toward the end of the 1.6 s prediction horizon. Second, the current pretraining pipeline does not explicitly enforce multi-view invariance. In Stage 2, latent prediction is learned primarily from the main camera view, and no objective aligns representations across different camera viewpoints. Although the Stage 3 action policy can access the wrist image, the pretrained visual representation is therefore only weakly constrained to remain consistent under viewpoint changes. Extending the recurrent latent representation to substantially longer horizons and incorporating robustness to multiple views remain important future directions.
Conclusion
ReLAP achieves superior performance on standard LIBERO with only 11M pretraining samples and 1.06B trainable parameters, remains competitive under zero-shot LIBERO-Plus perturbations, and learns latent representations that transfer strongly on the LARY benchmark. Our analyses show that these latents are sample-specific, temporally segmented, and aligned with future physical actions, including on datasets unseen during pretraining. These results suggest that action-free visual pretraining can provide compact and transferable supervision for downstream robot control without requiring an explicit latent-action bottleneck at deployment. The remaining gaps are concentrated in long-horizon manipulation and robustness to camera and sensor shifts, motivating more balanced long-horizon supervision and multi-view pretraining.
References
- [1]L. Mu-Labadia, M. Muckley, A. Bar, M. Assran, K. Sinha, M. Rabbat, Y. LeCun, N. Ballas, and A. Bardes, “V-JEPA 2.1: Unlocking dense features in video self-supervised learning,” arXiv:2603.14482, 2026.
- [2]B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone, “LIBERO: Benchmarking knowledge transfer for lifelong robot learning,” arXiv:2306.03310, 2023.arxiv.org/abs/2306.03310
- [3]S. Fei, S. Wang, J. Shi, Z. Dai, J. Cai, P. Qian, L. Ji, X. He, S. Zhang, Z. Fei, J. Fu, J. Gong, and X. Qiu, “LIBERO-Plus: A progressive robustness benchmark for visual-language-action models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026, pp. 38 574–38 583.
- [4]D. Nie, F. Chen, Q. Lv, J. Kuyang, X. Li, X. Cao, and X. Cai, “LARY: A latent action representation yielding benchmark for generalizable vision-to-action alignment,” arXiv:2604.11689, 2026.arxiv.org/abs/2604.11689
- [5]S. Ye, J. Jang, B. Jeon, S. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y.-W. Chao, B. Y. Lin, L. Liden, K. Lee, J. Gao, L. Zettlemoyer, D. Fox, and M. Seo, “Latent action pretraining from videos,” arXiv:2410.11758, 2025.arxiv.org/abs/2410.11758
- [6]D. Schmidt and M. Jiang, “Learning to act without actions,” arXiv:2312.10812, 2024, ICLR 2024.arxiv.org/abs/2312.10812
- [7]J. Bruce, M. Dennis, A. Edwards, J. Parker-Holder, Y. Shi, E. Hughes, M. Lai, A. Mavalanakar, R. Steigerwald, C. Apps, Y. Ay tar, S. Bechtle, F. Behbahani, S. Chan, N. Hess, L. Gonzalez, S. Osindero, S. Ozair, S. Reed, J. Zhang, K. Zolna, J. Clune, N. de Freitas, S. Singh, and T. Rocktäschel, “Genie: Generative interactive environments,” arXiv:2402.15391, 2024.arxiv.org/abs/2402.15391
- [8]Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li, “UniVLA: Learning to act anywhere with task-centric latent actions,” arXiv:2505.06111, 2025.arxiv.org/abs/2505.06111
- [9]J. Yang, Y. Shi, H. Zhu, M. Liu, K. Ma, Y. Wang, G. Wu, T. He, and L. Wang, “CoMo: Learning continuous latent motion from internet videos for scalable robot learning,” arXiv:2505.17006, 2025, CVPR 2026.arxiv.org/abs/2505.17006
- [10]Q. Garrido, T. Nagarajan, B. Terv er, N. Ballas, Y. LeCun, and M. Rabbat, “Learning latent action world models in the wild,” arXiv:2601.05230, 2026.arxiv.org/abs/2601.05230
- [11]H. Luo, Y. Wang, W. Zhang, H. Yuan, Y. Feng, H. Xu, S. Zheng, and Z. Lu, “Joint-aligned latent action: Towards scalable VLA pretraining in the wild,” arXiv:2602.21736, 2026, CVPR 2026.arxiv.org/abs/2602.21736
- [12]Y. Chen, Y. Ge, Y. Li, Y. Ge, M. Ding, Y. Shan, and X. Liu, “Moto: Latent motion token as the bridging language for learning robot manipulation from videos,” arXiv:2412.04445, 2024, ICCV 2025.arxiv.org/abs/2412.04445
- [13]S. Routray, H. Pan, U. Jain, S. Bahl, and D. Pathak, “VIPra: Video prediction for robot actions,” arXiv:2511.07732, 2026.arxiv.org/abs/2511.07732
- [14]H. Wei, X. Chen, C. Zhang, T. Pearce, J. Chen, A. Lamb, L. Zhao, and J. Bian, “Learning additively compositional latent actions for embodied ai,” 2026. [Online]. Available: https://arxiv.org/abs/2604.03340
- [15]Q. Li, X. Gong, X. Li, P. Li, Q. Zhou, H. Ye, J. Zhou, and Y. Mu, “RotVLA: Rotational latent action for vision-language-action model,” arXiv:2605.13403, 2026.arxiv.org/abs/2605.13403
- [16]Z. Tang, H. Liu, X. Chang, C. Wu, D. Huo, Y. Yang, B. Liu, Z. Cai, F. Xiong, M. Xu, J. Luo, D. Ma, Z. Ma, and G. Pan, “ALAM: Algebraically consistent latent action model for vision-language-action models,” arXiv:2605.10819, 2026.arxiv.org/abs/2605.10819
- [17]Z. Tang, F. Luo, H. Liu, B. Yuan, D. Qi, R. Chen, Y. Yang, T. Lin, X. Chang, M. Xu, B. Liu, D. Ma, and Z. Ma, “DLAM: Distributional latent actions with temporal constraints,” arXiv:2607.27138, 2026.arxiv.org/abs/2607.27138
- [18]B. Tharwat, Y. Nasser, A. Abouzeid, and I. Reid, “Latent action pretraining through world modeling,” arXiv:2509.18428, 2026.arxiv.org/abs/2509.18428
- [19]Y. Wang, F. Zhang, D.-C. Zhan, L. Zhao, K. Wang, and J. Bian, “Co-evolving latent action world models,” 2026. [Online]. Available: https://openreview.net/forum?id=WwwXi3rkUWarxiv.org/abs/2510.26433
- [20]T. Ayalew, M. Jeung, S. Wheeler, et al., “CLAW: Learning continuous latent action world models via adversarial latent regularization,” arXiv:2606.04130, 2026.arxiv.org/abs/2606.04130
- [21]G. Zhou, H. Pan, Y. LeCun, and L. Pinto, “DINO-WM: World models on pre-trained visual features enable zero-shot planning,” arXiv:2411.04983, 2024.arxiv.org/abs/2411.04983
- [22]M. Assran, A. Bardes, D. Fan, Q. Garrido, R. Howes, et al., “V-JEPA 2: Self-supervised video models enable understanding, prediction and planning,” arXiv:2506.09985, 2025.arxiv.org/abs/2506.09985
- [23]J. Sun, W. Zhang, Z. Qi, S. Ren, Z. Liu, H. Zhu, G. Sun, X. Jin, and Z. Chen, “VLA-JEPA: Enhancing vision-language-action model with latent world model,” arXiv:2602.10098, 2026.arxiv.org/abs/2602.10098
- [24]Y. Lin, J. He, S. Bao, C. Zhao, Y. Li, X. Wang, Y. Wang, C. Chi, and J. Zhang, “JEPA-WAM: Learning vision-language-action policies with joint-embedding world modeling,” arXiv:2608.09381, 2026.arxiv.org/abs/2608.09381
- [25]R. Zheng, J. Wang, S. Reed, et al., “FLARE: Robot learning with implicit world modeling,” arXiv:2505.15659, 2025.arxiv.org/abs/2505.15659
- [26]H. Walke, K. Black, A. Lee, M. J. Kim, M. Du, C. Zheng, T. Zhao, P. Hansen-Estruch, Q. Young, A. He, V. Myers, K. Fang, C. Finn, and S. Levine, “BridgeData V2: A dataset for robot learning at scale,” arXiv:2308.12952, 2023. [Online]. Available: https://arxiv.org/abs/2308.12952
- [27]A. Khazatsky, K. Pertsch, S. Nair, et al., “DROID: A large-scale in-the-wild robot manipulation dataset,” arXiv:2403.12945, 2024. [Online]. Available: https://arxiv.org/abs/2403.12945
- [28]AgiBot-World-Contributors, Q. Bu, J. Cai, et al., “AgiBot World Colosseo: A large-scale manipulation platform for scalable and intelligent embodied systems,” arXiv:2503.06669, 2025. [Online]. Available: https://arxiv.org/abs/2503.06669
- [29]R. Hoque, P. Huang, D. J. Yoon, M. Sivapavapu, and J. Zhang, “EgoDex: Learning dexterous manipulation from large-scale egocentric video,” arXiv:2505.11709, 2025, ICLR 2026. [Online]. Available: https://arxiv.org/abs/2505.11709
- [30]R. Takanami, P. Khrapchenkov, S. Morikuni, et al., “AIRoA MoMa dataset: A large-scale hierarchical dataset for mobile manipulation,” arXiv:2509.25032, 2025. [Online]. Available: https://arxiv.org/abs/2509.25032
- [31]Y. Tian, Y. Yang, Y. Xie, Z. Cai, X. Shi, N. Gao, H. Liu, X. Jiang, Z. Qiu, F. Yuan, Y. Li, P. Wang, J. Cai, J. Zeng, H. Dong, and J. Pang, “InternData-A1: Pioneering high-fidelity synthetic data for pre-training generalist policy,” arXiv:2511.16651, 2025. [Online]. Available: https://arxiv.org/abs/2511.16651
- [32]K. Grauman, A. Westbury, E. Byrne, et al., “Ego4D: Around the world in 3,000 hours of egocentric video,” arXiv:2110.07058, 2022, CVPR 2022. [Online]. Available: https://arxiv.org/abs/2110.07058
- [33]Z. Wu and J. Gao, “OSCAR: Omni-embodiment action-conditioned world model for robotics,” arXiv:2606.04463, 2026. [Online]. Available: https://arxiv.org/abs/2606.04463
- [34]Y. Li, B. Zhang, C. Gu, Z. Ma, J. Zhang, J. Deng, X. Zhu, and L. Zhang, “From imagined futures to executable actions: Mixture of latent actions for robot manipulation,” arXiv:2605.12167, 2026, ICML 2026.arxiv.org/abs/2605.12167
- [35]X. Li, Y. Zheng, S. Gu, H. Ma, F. Yu, Y. Zheng, X. Nie, S. Yuan, Y. Zang, W. Li, S. Tian, M. Liu, Y.-Q. Zhang, and W. Ding, “Latent action as intention enables efficient future imagination for world action models,” arXiv:2608.24882, 2026.arxiv.org/abs/2608.24882
- [36]C. Zhang, T. Pearce, P. Zhang, K. Wang, X. Chen, W. Shen, L. Zhao, and J. Bian, “What do latent action models actually learn?” in The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. [Online]. Available: https://openreview.net/forum?id=DQmjemrVhearxiv.org/abs/2506.15691
