betaFeedback

WorksComputer Science

ELA: End-to-End Latent Action Policy Learning from Video Demonstrations

Abstract

Robot imitation learning is limited by the scarcity of action-labeled data, while action-free demonstrations such as videos are abundant. Recent latent-action methods learn action-relevant representations from observation transitions, but they do not optimize latent actions for downstream control, leading to latent representations that are difficult for policies to model and difficult to accurately decode into continuous robot actions. This paper introduces ELA, an end-to-end framework for learning from video demonstrations via latent actions jointly optimized for policy learning and action decoding. ELA jointly trains a latent action model, latent action policy, and action decoder, enabling policies to leverage abundant video data with only a handful of action-labeled robot demonstrations. Experiments on simulated and real-world manipulation tasks show that ELA consistently outperforms prior latent action baselines. Moreover, it integrates seamlessly with existing imitation learning methods through lightweight 4M-parameter action decoders, requiring no changes to policy architectures or training objectives. Additional materials can be found on: https://end2end-latent-action.github.io/

Introduction

Data scarcity remains a fundamental bottleneck for robot imitation learning, because collecting large-scale datasets [1, 2] with robot action labels is expensive and often infeasible. In contrast, video demonstrations, including human videos and unlabeled robot trajectories, are significantly easier to obtain at scale. This contrast motivates imitation learning approaches that learn directly from video data without requiring action labels.

Latent action modeling [3, 4] has recently emerged as a promising direction for leveraging such data. These methods infer action-relevant structure from video demonstrations and use it as the supervision target for policy learning, removing the need for explicit action supervision. This property makes latent action learning particularly promising for leveraging large-scale video data for imitation learning. However, while effective as an abstract representation, existing approaches do not explicitly optimize the latent action space for downstream control.

We identify two key properties that a latent action space should satisfy for effective policy learning: modelability, meaning latent actions can be reliably inferred from observations for policy learning, and decodability, meaning they can be faithfully mapped into executable continuous robot actions. Existing methods [4, 5, 6, 7] typically adopt a two-stage pipeline, where latent actions are first learned from image pairs and then used as fixed targets for a downstream policy. This decoupling leads to a latent space that is not explicitly shaped for either reliable inference from observations or accurate execution, potentially resulting in latent actions that are difficult to predict or decode.

In this work, we study whether latent actions can instead serve as a unified interface for imitation learning that is jointly optimized for both modelability and decodability. To that end, this paper introduces the ELA framework shown in Figure 1, in which latent actions act as both the supervision signal for policy learning and the interface for action execution. By jointly optimizing the latent space with respect to both policy learning and action decoding, our method learns latent actions that can effectively leverage action-free video data while requiring relatively little action-labeled data for decoding into continuous robot actions.

ELA overview showing the standard behavior cloning approach and ELA

Figure 1. ELA Overview. Our framework extends existing behavior cloning methods, e.g. diffusion policies, with latent actions jointly optimized for policy learning and action decoding. The approach enables training on mixtures of action-labeled and video demonstrations with little overhead, requiring only a lightweight action decoder.

ELA learns from a mixture of action-labeled and action-free demonstrations by jointly training three components: a Latent Action Model, a Latent Action Policy, and an Action Decoder. The Latent Action Model infers a latent action responsible for each state transition, which are then used as supervision for the Latent Action Policy. This approach enables policy learning directly from video data without modifying the underlying policy architecture or objective, while ensuring modelability by shaping a latent space such that latent actions corresponding to demonstrations are predictable. The Action Decoder maps latent actions into continuous robot control commands, ensuring decodability. To prevent latent action collapse during joint training, we further introduce a representation alignment objective that stabilizes the latent space.

Overall, ELA enables imitation learning from video demonstrations by learning a latent action space that is explicitly optimized for both modelability and decodability. It effectively leverages video data and outperforms other latent action baselines across simulated and real-world benchmarks. Moreover, it can be integrated into existing imitation learning approaches without modifying policy architectures or training objectives, using a lightweight Action Decoder that adds a fixed 4M-parameter overhead independent of policy size.

Related Work

Learning from video demonstrations [8] enables robots to acquire skills without requiring action labels, alleviating the data scarcity bottleneck in robot imitation learning. Prior works extracts action-relevant information from videos through intermediate representations such as hand poses [9, 10], object poses [11, 12], and point tracks [13, 14]. These methods typically rely on external vision systems, including hand pose predictors [15] and object pose estimators [16], and often require carefully designed architectures tailored to specific representations. Another line of work, world action models (WAM) [17, 18, 19], leverages video data through world modeling objectives such as next-frame prediction. While effective for learning visual dynamics, these approaches generally do not recover action representations that can directly supervise policy learning.

Latent action modeling [3, 4] instead learns inverse dynamics models (IDMs) that infer latent actions from video, enabling training on diverse data sources including human videos [20] and robot datasets [2]. However, latent actions are often too abstract for direct low-level control and are mainly used for policy pretraining [4, 5, 6, 7]. Recent methods [21, 22, 23] incorporate proprioception and action supervision, but the separate training of the IDM and policy often leads to sub-optimal performance. In contrast, ELA jointly learns the inverse dynamics model and control policy, enabling direct end-to-end policy learning from action-labeled and action-free demonstrations.

Approach

In this section, we introduce ELA, a latent-action approach for learning from a mixture of action-labeled and action-free demonstrations. ELA jointly trains three components, a Latent Action Model, a Latent Action Policy, and an Action Decoder. A high-level overview of the proposed architecture is shown in Figure 2.

ELA training and deployment architecture

Figure 2. ELA Training and Deployment. During training, image pairs are used with an inverse dynamics model (IDM) to infer latent actions, which supervise a forward dynamics model (FDM) for future frame reconstruction and an action decoder for robot action prediction. A latent action policy is then trained to output the latent actions. During deployment, the latent action policy maps image observations to latent actions, which are decoded into robot actions by the action decoder.

Problem Formulation

We assume access to two types of expert demonstration data: an action-free dataset of RGB observation trajectories V={ot}\mathcal{V} = \{o_t\} and a smaller action-labeled dataset R={(ot,at)}\mathcal{R} = \{(o_t, a_t)\}, where ot∈Oo_t \in\mathcal{O} denotes RGB image observations at time tt and at∈Aa_t \in\mathcal{A} denotes expert robot actions at time tt. The action-free dataset may come from the same embodiment as the target robot or from a different source, e.g., human demonstrations, whereas the action-labeled dataset is collected on the target embodiment and provides ground-truth control commands. Given these datasets, our objective is to learn a vision-based imitation learning policy Π(at∣ot)\Pi(a_t \mid o_t) that executes robot actions ata_t from observations oto_t to achieve task success, measured by a binary success signal.

Method Overview

To learn from both action-free and action-labeled data, we first introduce a latent action model consisting of an inverse dynamics model (IDM) and a forward dynamics model (FDM). For policy learning, we use the IDM, which maps a pair of observations separated by kk time steps to a latent action: zt=IDM⁡(ot,ot+k)z_t = \operatorname{IDM}(o_t, o_{t+k}). Next, we decompose the policy into a latent action policy and an action decoder: Π(at:t+k∣ot)=ψ(at:t+k∣zt) π(zt∣ot)\Pi(a_{t:t+k} \mid o_t) = \psi(a_{t:t+k} \mid z_t)\,\pi(z_t \mid o_t), where the latent action policy π:O→Z\pi: \mathcal{O} \to\mathcal{Z} outputs a latent action ztz_t from a single observation oto_t, and the action decoder ψ:Z→Ak\psi: \mathcal{Z} \to\mathcal{A}^k maps each latent action to a kk-step sequence of robot actions. This decoupling allows the latent action policy π\pi to be trained using both action-free and action-labeled data from datasets V\mathcal{V} and R\mathcal{R}, since latent actions are inferred by the IDM. In contrast, the action decoder ψ\psi is trained exclusively on action-labeled data from R\mathcal{R}.

The latent action serves as the key intermediate representation connecting the policy and the action decoder. It acts both as the prediction target for the behavior cloning policy π\pi and as the sole input to the action decoder ψ\psi. To learn this representation effectively, we introduce ELA, which jointly trains three components: the inverse dynamics model IDM, the latent action policy π\pi, and the action decoder ψ\psi, in order to shape the latent action zz.

Latent Action Learning

We instantiate latent actions by modeling state transitions over kk time steps. Given observation pairs (ot,ot+k)(o_t, o_{t+k}), we learn a latent action ztz_t that explains the transition dynamics induced by human or robot. To learn this latent action, we parameterize a latent action model consisting of an inverse dynamics model (IDM), pIDM(zt∣ot,ot+k)p_{\mathrm{IDM}}(z_t \mid o_t, o_{t+k}), which infers latent actions from observed transitions, and a forward dynamics model (FDM), pFDM(ot+k∣ot,zt)p_{\mathrm{FDM}}(o_{t+k} \mid o_t, z_t), which enforces that the latent action is sufficient to predict future observations.

The learned latent actions serve as the interface between policy and action decoder, enabling both modelability and decodability. Modelability is realized by learning a latent policy π(zt∣ot)\pi(z_t \mid o_t) that infers latent actions from observations, while decodability is ensured by an action decoder ψ(at:t+k∣zt)\psi(a_{t:t+k} \mid z_t) that maps latent actions to continuous robot actions.

Inverse and Forward Dynamics Model. We implement the IDM by tokenizing (ot,ot+k)(o_t, o_{t+k}) and processing them with a transformer encoder and learnable register tokens [24], which aggregate inter-frame dynamics into a fixed-length latent action. A possible failure mode is that the IDM captures action-irrelevant details, or even memorizes raw image content due to the reconstruction objective. To prevent these possibilities, prior work [4] applies quantization as an information bottleneck, discretizing latent actions into tokens so that the learned representations are encouraged to retain only action-relevant information. Following this idea, we apply an information regularization scheme based on iFSQ [25] quantization. The IDM and FDM are jointly trained using an image reconstruction loss Lfdm=∥o^t+k−ot+k∥22\mathcal{L}_{\mathrm{fdm}} = \lVert\hat{o}_{t+k} - o_{t+k} \rVert_2^2, where o^t+k=pFDM(ot,zt)\hat{o}_{t+k} = p_{\mathrm{FDM}}(o_t, z_t).

Latent Action Policy. We build our latent action policy by adapting existing behavior cloning approaches [26, 17] to operate in the latent action space. Instead of predicting low-level robot actions directly, the policy π\pi takes an observation oto_t as input and outputs a latent action ztz_t. This latent action prediction is then passed to the action decoder ψ\psi to obtain executable robot actions. This modularity allows standard policy architectures to learn from latent action supervision without requiring changes to their underlying structure or training objectives. The policy is trained by the policy loss Lpolicy=∥z^t−zt∥22\mathcal{L}_{\mathrm{policy}} = \lVert\hat{z}_t - z_t \rVert_2^2, where z^t=π(ot)\hat{z}_t = \pi(o_t).

Action Decoder. We implement the action decoder as a single-pass transformer decoder similar to prior work [27, 28]. The decoder is trained only on data with ground-truth action labels using an action reconstruction objective. Given a latent action ztz_t encoded from frames (ot,ot+k)(o_t, o_{t+k}), the action loss Lact=∥a^t:t+k−at:t+k∥22\mathcal{L}_{\mathrm{act}} = \lVert\hat{a}_{t:t+k} - a_{t:t+k} \rVert_2^2, where a^t:t+k=ψ(zt)\hat{a}_{t:t+k} = \psi(z_t).

Joint Training with Representation Alignment

In our formulation of latent action policy learning, we jointly train the latent action model (LAM), policy, and action decoder so that the latent space is encouraged to exhibit both modelability and decodability. However, a naive end-to-end objective that combines all losses into a single optimization step leads to latent action collapse in the IDM, a failure mode observed in prior work [21, 29]. In this setting, policy gradients bias the latent space toward representations that are easy to predict, even when they conflict with the objectives required for accurate decoding and LAM learning. To address this issue, instead of directly propagating policy gradients into the LAM, we align both models to a shared reference representation space. Given visual representations produced by a frozen perceptual encoder, e.g. DINO [30], we adopt a representation alignment objective [31, 32] that regularizes the latent space around semantically meaningful visual structure while preserving action decodability. The policy is trained to align with the same reference representation.

Specifically, we extract features from the intermediate layer of the policy encoder and align both the IDM and policy to a shared representation produced by the frozen perceptual encoder. To avoid shortcut problem arising from direct joint optimization, we alternate between two update passes, as illustrated in Figure 3. In the LAM-update pass, the policy network is frozen, and the alignment loss is backpropagated only into the LAM. Because the policy remains fixed, the IDM is encouraged to produce latent actions whose induced policy features align with the reference representation, reshaping the latent space toward targets that are easier for the policy to model. In the Policy-update pass, the LAM is frozen while the same alignment objective is used to optimize the policy. This objective pushes the policy features toward the shared reference space, improving representation quality and latent action prediction. Together, the two passes improve modelability from complementary directions: the LAM-update pass encourages latent actions that are easier to predict, while the Policy-update pass improves the policy’s ability to predict them. Both are coordinated through a shared observation-derived representation space.

End-to-End Joint Training

Figure 3. End-to-End Joint Training. ELA jointly trains the latent action model (LAM), latent action policy, and action decoder using two alternating forward passes. In the LAM-update pass, the policy is frozen and the alignment, action, and FDM losses are backpropagated to update the LAM. In the Policy-update pass, the LAM is frozen and the alignment and policy losses are used to update the policy network.

Representation Alignment Loss. The alignment objective is defined as

Lalign(ϕ,θ)=−E(ot,ot+k)∼D[Sim⁡(y,hθ(h~))],(1)\mathcal{L}_{\mathrm{align}}(\phi,\theta)=-\mathbb{E}_{(o_t,o_{t+k})\sim\mathcal{D}}\left[\operatorname{Sim}\left(y,h_\theta(\tilde{h})\right)\right], \tag*{(1)}

where y=F(ot,ot+k)y=\mathcal{F}(o_t,o_{t+k}) denotes the reference representation extracted from a frozen perceptual encoder F\mathcal{F}. The feature h~\tilde{h} is taken from an intermediate layer of the policy encoder. Specifically, let gθg_\theta denote the mapping defined by the policy input embedding followed by the first LL transformer blocks, where LL corresponds to the midpoint of the transformer stack. Given latent action zz and observation oto_t, we extract the intermediate policy feature h~=gθ(z,ot)\tilde{h}=g_\theta(z,o_t), which is then projected through a trainable MLP head hθh_\theta before being aligned with the reference representation yy. Here, Sim⁡(⋅,⋅)\operatorname{Sim}(\cdot,\cdot) denotes cosine similarity between the reference representation yy and the projected policy feature hθ(h~)h_\theta(\tilde{h}).

Overall Training. We optimize the following objective:

L(θ,ϕ,ω)=Lfdm(θ)+Lact(θ)+Lalign(θ,ϕ,ω)+Lpolicy(ϕ),\mathcal{L}(\theta,\phi,\omega)=\mathcal{L}_{\mathrm{fdm}}(\theta)+\mathcal{L}_{\mathrm{act}}(\theta)+\mathcal{L}_{\mathrm{align}}(\theta,\phi,\omega)+\mathcal{L}_{\mathrm{policy}}(\phi),

where θ\theta, ϕ\phi, and ω\omega denote the parameters of the LAM, policy, and MLP projection head, respectively. All losses are optimized over a mixture of action-free and action-labeled video data, except for Lact\mathcal{L}_{\mathrm{act}}, which is applied only to action-labeled samples.

Implementation Details. We use a frozen DINOv3 [30] model as the perceptual encoder for computing the reference representations in the alignment objective. We employ a warm-up stage of 5k iterations in which only Lfdm\mathcal{L}_{\mathrm{fdm}} and Lact\mathcal{L}_{\mathrm{act}} are optimized. The warm-up allows the IDM to first learn a stable latent action space before introducing alignment and policy learning.

Experiments

Our experiments are designed to investigate the following research questions.

  1. Few-shot Learning from Action-Free Video in Simulation (§ 4.1): Can ELA improve task performance in simulation by leveraging additional robot video alongside a small set of action-labeled robot demonstrations, beyond methods trained only on action-labeled data?

  2. Cross-Embodiment Learning in the Real World (§ 4.2): Can ELA effectively learn real-world policies from a combination of human video demonstrations and a limited set of action-labeled robot demonstrations?

  3. Analysis of Decodability and Modelability (§ 4.3 and § 4.4): How does ELA compare to IDM baselines trained with significantly larger data budgets in terms of decodability and modelability under few-shot settings?

Few-shot Policy Learning from Action-Free Data

Benchmark Setup. We evaluate all baselines on LIBERO-10, the most challenging subset of the LIBERO benchmark [33], which consists of 10 long-horizon tabletop manipulation tasks. For training of each task, we provide 50 robot demonstrations, of which NN samples are action-labeled, where N∈{5,10,20}N \in\{5, 10, 20\}. Task success is measured in simulation, and results are averaged over 3 runs with different random seeds.

Baselines. To evaluate the effectiveness of our approach in a data-constrained setting with limited action-labeled data and abundant action-free data, we apply ELA to two representative imitation learning baselines: Diffusion Policy (DP) and Unified World Model (UWM) [17], which represent standard imitation learning and world-action modeling approaches, respectively. We extend both baselines with our IDM, action decoder, and end-to-end training recipe, resulting in ELADP\mathrm{ELA}_{\mathrm{DP}} and ELAUWM\mathrm{ELA}_{\mathrm{UWM}}, respectively. We further compare against state-of-the-art methods designed specifically for learning from video demonstrations, including ATM [13] and AMPLIFY [14]. Notably, both ATM and AMPLIFY require specialized data preprocessing pipelines that depend on external point-tracking models, whereas our approach does not rely on such additional components or preprocessing steps.

Training Details. DP cannot leverage action-free data and is therefore trained only on the NN action-labeled demonstrations available for each task. In contrast, all the other methods can utilize action-free data and are trained on a mixture of action-free and action-labeled data. We consider the single-task policy setting, in which all methods are first pretrained on all multi-task data (500 demonstrations) and then fine-tuned using 50 demonstrations from each target task. During ELA fine-tuning, the IDM and Action Decoder remain frozen, allowing the policy to adapt only within the latent action manifold learned during pretraining.

Results. As shown in Figure 4, UWM outperforms DP when trained with the full set of 50 action-labeled demonstrations, corresponding to the upper-bound setting. However, in low-data regimes with only N∈{5,10}N \in\{5, 10\} action-labeled demonstrations, UWM underperforms DP, highlighting the greater difficulty of training world action models compared to standard imitation learning methods under limited supervision. Despite this challenge, ELA consistently improves the performance of both DP and UWM by enabling supervision through latent actions inferred from all available demonstrations, including both action-free and action-labeled samples. For instance, when N=5N=5, ELA improves UWM from 19% to 59% success rate, and DP from 31% to 54%. In the low-data setting N∈{5,10}N \in\{5, 10\}, both ELADP_{\mathrm{DP}} and ELAUWM_{\mathrm{UWM}} outperform prior state-of-the-art methods such as ATM and AMPLIFY, which rely on external models and specialized policy architectures.

Figure 4. Few-shot Policy Learning. Comparative analysis of success rates on the LIBERO-10 benchmark. All evaluated methods were trained using NN action-labeled samples selected from 50 robot demonstrations per task. The two dashed lines at the top indicate the upper-bound performance of DP and UWM trained on the full set of 50 action-labeled demonstrations.

Cross-Embodiment Learning from Human Data

Benchmark Setup. To evaluate the effectiveness of our approach in the cross-embodiment learning setting, we consider scenarios with limited action-labeled robot data and more abundant human video data. For each task, we use 10 action-labeled robot demonstrations and 40 human demonstrations. Our real-world task suite covers a range of everyday activities, from standard pick-and-place tasks to manipulation of articulated objects. Task details are provided in the supplementary material. For evaluation, object locations are randomly initialized within the region shared between the robot camera’s field of view and the robot arm’s reachable workspace. For each task, we run 20 trials and measure success.

Baselines. We evaluate the effectiveness of learned latent actions in our policy learning setup by comparing against several baselines, including LAPA [4], Villa-X [22], and UniVLA [6]. These methods are trained on large-scale cross-embodiment datasets containing both human and robot demonstrations. Their model sizes and training data budgets are summarized in Table 1. Note that the reported parameter counts include only the IDM component and exclude additional modules such as the FDM. We further include LAPA-DINO, a variant of LAPA that incorporates DINOv3 [30] features and achieves state-of-the-art performance on the LARY benchmark [34]. We denote our full method as ELADP_{\mathrm{DP}}, which augments DP with an IDM and an action decoder. We also include an ablation variant, ELADP_{\mathrm{DP}} (w/o align), which removes the representation alignment module.

BaselineModel Param.Training Data (# trajectories)
LAPA [4]206M1M+
Villa-X [22]124M1M+
UniVLA [6]34M†1M+
LAPA-DINO [34]102M†250
ELADP_{\mathrm{DP}} (Ours)6M250

Table 1. IDM model sizes and training budgets for baselines. † DINO encoder parameters are excluded from the count.

Training Details. All baselines except ELA follow a two-stage training pipeline, where the IDM is trained independently from the policy. We use publicly available IDM weights as frozen encoders and train a downstream policy and action decoder on top of them. For LAPA-DINO, we additionally train the model on our dataset to serve as an in-domain baseline, while other methods are used directly without further adaptation.

Results. As shown in Figure 5(a), ELADP_{\mathrm{DP}} achieves the best performance across all evaluated methods. Although LAPA, Villa-X, and UniVLA have substantially larger model capacities and are pre-trained on large-scale datasets, their gains do not translate into improved downstream policy performance in this setting. Meanwhile, despite being trained on in-domain demonstrations, LAPA-DINO still exhibits a substantial performance gap compared to ELADP and even the baseline DP model. These results suggest that pretraining scale and data diversity alone are insufficient to ensure effective downstream policy learning in the few-shot regime.

Figure 5. Real-world Results. (a) Comparative analysis of success rates on the real-world benchmark. (b) Ablation study of representation alignment across a subset of task suites. (c) Modelability versus Decodability for each latent action spaces.

To control for differences in data access, we additionally evaluate Dynamo [35], a self-supervised policy-pretraining method based on world-modeling objectives. Unlike standard DP, which cannot directly leverage human data for policy learning, Dynamo can exploit both human and robot videos. We pretrain Dynamo on videos collected across all tasks and then fine-tune the resulting representation for diffusion policy learning. As shown in Table 2, Dynamo underperforms both standard DP and ELADP, suggesting that access to human and robot data alone is insufficient to achieve strong downstream policy performance. One possible explanation is that image-representation pretraining is more susceptible to cross-embodiment appearance shifts, whereas ELADP learns modelable and decodable latent actions that provide more direct supervision for cross-embodiment policy learning.

MethodSuccess
Villa-XFS_{\mathrm{FS}}0.06
Villa-XFT_{\mathrm{FT}}0.21
Dynamo0.38
DP0.48
ELADP0.70

Table 2. Real-world success under matched data access.

We next investigate whether the performance gap of existing latent-action approaches can be attributed to a lack of in-domain training. To this end, we evaluate two additional Villa-X variants: Villa-XFT_{\mathrm{FT}}, fine-tuned on our in-domain data, and Villa-XFS_{\mathrm{FS}}, trained from scratch using the same data as ELADP. As shown in Table 2, both variants underperform the original Villa-X, with success rates of 0.21 and 0.06, respectively, compared with 0.27 for the original model. This result suggests that simply adapting existing latent-action models to in-domain data is insufficient to achieve strong performance in the few-shot regime.

Finally, to assess the contribution of our representation alignment approach, we conduct an ablation study. As shown in Figure 5(b), removing the alignment component reduces the success rate from 0.83 to 0.45, demonstrating the importance of representation alignment for downstream policy learning.

Analysis of Decodability and Modelability

Study Setup. To further analyze the gap in real-world experiments, we evaluate each IDM latent action space along two dimensions in Figure 5(c) using mean squared error (MSE) against ground-truth actions on 20 held-out real-world robot demonstrations from § 4.2. Decodability (MSEidm_{\mathrm{idm}}) measures how well actions can be reconstructed from IDM latents extracted from frame transitions, indicating how much action-relevant information is preserved. Modelability (MSEdp_{\mathrm{dp}}) measures reconstruction error from latents predicted by a diffusion policy conditioned on observations, indicating how predictable the latents are for the policy. Since policy-predicted latents approximate IDM latents, MSEidm\mathrm{MSE}_{\mathrm{idm}} is a lower bound on MSEdp\mathrm{MSE}_{\mathrm{dp}}. A useful latent space must be both decodable and modelable, corresponding to the lower-left region of Figure 5(c).

Results. IDM methods trained without an action decoder, i.e., UniVLA, LAPA, and LAPA-DINO, exhibit poor decodability and modelability. Villa-X slightly improves both metrics by aligning latent actions with underlying robot states and actions through its FDM objective. ELADP\mathrm{ELA}_{\mathrm{DP}} (w/o align) achieves strong decodability but remains sub-optimal in modelability. Our full approach ELADP\mathrm{ELA}_{\mathrm{DP}} achieves the best performance on both axes, highlighting the importance of joint training and representation alignment in learning effective latent action spaces.

Linear Probing for Action Prediction

Study Setup. Although Figure 5(c) shows that ELA achieves lower action reconstruction errors in both decodability and modelability, these metrics may partly reflect differences in the learned action decoders. To better disentangle representation quality from decoder performance, we conduct an additional linear probing experiment following a standard protocol in representation learning [34]. Specifically, we freeze each learned latent representation and train an identical linear probe to predict target actions for 10 epochs, using a learning rate of 10−310^{-3} and a cosine learning-rate schedule. We evaluate the probes on 20 held-out samples across all tasks. By keeping the probe architecture and training protocol fixed across methods, this experiment provides a more direct comparison of the action-relevant information accessible in each representation while reducing the influence of decoder capacity.

Results. As shown in Table 3, ELADP\mathrm{ELA}_{\mathrm{DP}} achieves the lowest action prediction MSE of 0.015, outperforming all evaluated baselines. Among the original pretrained baselines, Villa-X achieves the second-lowest MSE of 0.024, followed by LAPA-DINO and the fine-tuned Villa-X variant, both with an MSE of 0.029. Notably, fine-tuning Villa-X on in-domain data does not improve its linear probing performance, while training from scratch yields a substantially higher MSE of 0.059. These results complement our decodability and modelability analysis, providing additional evidence that ELADP\mathrm{ELA}_{\mathrm{DP}} learns representations that make action-relevant information more accessible to a simple predictor, rather than relying solely on the capacity of its learned action decoder to recover the underlying actions.

MethodMSE
UniVLA0.043
LAPA0.042
Villa-X0.024
Villa-XFS_{\mathrm{FS}}0.059
Villa-XFT_{\mathrm{FT}}0.029
LAPA-DINO0.029
ELADP\mathbf{ELA}_{\mathbf{DP}}0.015

Table 3. Linear probing for action regression.

Limitations

ELA effectively learns from video data, but has not yet been validated at large scale. Scaling to cross-embodiment datasets [20, 2] and evaluating across diverse scenarios are natural directions for future work. Second, ELA operates directly on raw RGB pixels and does not explicitly leverage depth or 3D priors. Incorporating visual foundation models, such as monocular depth estimation [36], or training with 3D supervision could help mitigate this limitation. Finally, our evaluation excludes highly dexterous or precision-intensive tasks, leaving fine-grained manipulation as future work.

Conslusion

In this paper, we introduce ELA, an end-to-end framework for learning from video demonstrations via latent actions, jointly optimized for policy learning and action decoding. ELA consists of a latent action model, a latent-action policy, and a lightweight decoder, enabling effective use of large-scale action-free video with only a small set of robot demonstrations. Simulated and real-world experiments show consistent improvements over prior methods in data-constrained settings. Our analysis further indicates that the learned latent actions enhance both modelability and decodability. Overall, ELA highlights the promise of leveraging abundant video data to improve imitation learning.

Acknowledgements

This work has taken place in the Learning Agents Research Group (LARG) at the Artificial Intelligence Laboratory, The University of Texas at Austin. LARG research is supported in part by the National Science Foundation (FAIN-2019844, NRT-2125858, OIC-2535195), the Office of Naval Research (N00014-24-1-2550), Army Research Office (W911NF-17-2-0181, W911NF-23-2-0004, W911NF-25-1-0065), Lockheed Martin, Nuro, Lyda Hill Philanthropies, and Good Systems, a research grand challenge at the University of Texas at Austin. The views and conclusions contained in this document are those of the authors alone. Peter Stone serves as the Chief Scientist of Sony Corporate Technology Center and receives financial compensation for that role. The terms of this arrangement have been reviewed and approved by the University of Texas at Austin in accordance with its policy on objectivity in research.

Latent Action Ablations

To evaluate the impact of latent-action design choices, we ablate our approach ELADP on the soup-cheese and book-caddy tasks from the LIBERO-10 benchmark under the N=10N = 10 setting. The result is shown in Table 4.

(a) Num. of latent-action tokens kk(b) FSQ codebook
kkSuccessLevelsCodebook sizeSuccess
40.59[5,5,5,5][5,5,5,5]6250.47
80.76[7,7,7,7][7,7,7,7]24010.78
16†16^{\dagger}0.83[9,9,9,9]†[9,9,9,9]^{\dagger}65610.83
320.78[7,7,7,7,7][7,7,7,7,7]168070.57

Table 4. Ablations on the latent action. We vary the number of latent-action tokens (k) (left) and the FSQ codebook configuration (right), one factor at a time while keeping all other settings fixed. The default configuration is marked with (†).

Number of Latent-Action Tokens. We vary the latent-action length k∈{4,8,16,32}k \in\{4, 8, 16, 32\} while keeping all other hyperparameters fixed. Performance improves from 0.59 at k=4k = 4 to a peak of 0.83 at k=16k = 16, and then decreases slightly to 0.78 at k=32k = 32. These results indicate that increasing the number of latent-action tokens improves performance up to a moderate value, while further increasing the sequence length provides no additional benefit. We therefore use k=16k = 16 in all experiments.

Codebook Size. We evaluate four FSQ level configurations: [5,5,5,5][5, 5, 5, 5] (625 codes), [7,7,7,7][7, 7, 7, 7] (2,401 codes), [9,9,9,9][9, 9, 9, 9] (6,561 codes), and [7,7,7,7,7][7, 7, 7, 7, 7] (16,807 codes). The [9,9,9,9][9, 9, 9, 9] configuration achieves the highest success rate (0.83). Performance is substantially lower with the smallest codebook (0.47), while increasing the codebook size beyond 6,561 codes yields little to no improvement. Based on these results, we adopt [9,9,9,9][9, 9, 9, 9] as the default configuration.

Real-world Experiment Details

Hardware Details

We conduct real-world experiments using a Franka Panda robot, as shown in Figure 6. The robot is mounted on a wheeled desk, allowing it to be easily deployed across a variety of environments and task scenarios. The observation space consists of a single scene camera, while an additional external camera is used solely for evaluation to monitor task initialization and execution.

Robot setup

Figure 6. Robot Setup.

The robot is controlled at 10 Hz using an Operational Space Control (OSC) controller. The action space is parameterized as a delta end-effector (EE) pose, consisting of incremental translational and rotational commands relative to the current EE pose. Gripper control is represented by a single continuous action dimension, where 0 denotes a fully open gripper and 1 denotes a fully closed gripper. This setup enables responsive and smooth execution of manipulation tasks.

Baseline Details

We compare against several latent action models. LAPA uses the publicly available checkpoint2, which is pretrained on the full Open X-Embodiment dataset [1]. Villa-X employs the publicly released model3. Since Villa-X is trained on a diverse collection of robot and egocentric human datasets, we refer readers to the original paper for a complete description of its training data. For UniVLA, we use the latent action model4 trained on Open X-Embodiment [1] and Ego4D [?] dataset. The implementation of LAPA-DINO is obtained from the LARYBench [34] repository5. For a controlled comparison, we train LAPA-DINO using the same training dataset as our method, ELA.

TaskUniVLALAPAVilla-XLAPA-DINODPELADP\mathrm{ELA}_{\mathrm{DP}}
place-food0.250.550.500.500.600.95
place-food-cabinet0.000.250.200.500.550.80
stack-bowl0.150.350.450.350.400.70
pour-bowl0.000.100.100.200.500.75
open-oven0.000.050.100.100.350.30
Average0.080.260.270.330.480.70

Table 5. Success rates on the real-world benchmark (20 trials per task).

Task Details

We provide detailed descriptions of the tasks used in the real-world benchmark.

  • place-food: Pick up a bag of food from the counter and place it onto a plate.

  • place-food-cabinet: Pick up a bag of food from the top cabinet and place it onto a plate on the counter. The task requires handling a larger vertical displacement and precise placement onto a constrained target region.

  • stack-bowl: Grasp a bowl and stack it on top of another bowl. Successful execution requires precise alignment and stable object placement.

  • pour-bowl: Pick up a container and perform a pouring motion into a target bowl. The task requires coordinated manipulation and accurate control of the object’s orientation throughout the pouring trajectory.

  • open-oven: Grasp the oven handle and open the oven door while respecting the articulation constraints of the door. Successful execution requires accurate handle grasping and coordinated motion along the door’s opening trajectory.

Detailed Results

Real-world Benchmark

Table 5 reports the per-task success rates of all baselines on the real-world benchmark. Across all tasks, most baselines underperform the DP policy, primarily due to their reliance on learned inverse dynamics models (IDMs) that generate latent actions with limited decodability and modelability for downstream control. In contrast, ELADP\mathrm{ELA}_{\mathrm{DP}} consistently improves over DP by incorporating additional human demonstrations and more effectively shaping the latent action space through our joint training recipe. The only exception is the open-oven task, where ELADP\mathrm{ELA}_{\mathrm{DP}} does not provide further gains. This task demands highly precise continuous control under strong articulation constraints, which exceeds the modeling capacity of the IDM when learning from typical human demonstration videos, thereby limiting the benefit of additional data.

TaskDPELADP\mathrm{ELA}_{\mathrm{DP}}UWMELAUWM\mathrm{ELA}_{\mathrm{UWM}}
N=20N = 20
bowl-drawer0.80 ±\pm 0.040.89 ±\pm 0.080.91 ±\pm 0.060.93 ±\pm 0.03
cheese-butter0.69 ±\pm 0.080.77 ±\pm 0.100.75 ±\pm 0.020.74 ±\pm 0.00
moka-moka0.43 ±\pm 0.090.26 ±\pm 0.070.38 ±\pm 0.030.49 ±\pm 0.03
mug-microwave0.58 ±\pm 0.060.85 ±\pm 0.030.66 ±\pm 0.000.91 ±\pm 0.01
mug-mug0.36 ±\pm 0.040.46 ±\pm 0.030.55 ±\pm 0.060.59 ±\pm 0.03
mug-pudding0.25 ±\pm 0.150.17 ±\pm 0.040.32 ±\pm 0.070.45 ±\pm 0.03
soup-cheese0.63 ±\pm 0.110.88 ±\pm 0.020.71 ±\pm 0.140.83 ±\pm 0.01
soup-sauce0.40 ±\pm 0.040.71 ±\pm 0.040.64 ±\pm 0.090.65 ±\pm 0.04
stove-moka0.42 ±\pm 0.080.57 ±\pm 0.080.81 ±\pm 0.060.95 ±\pm 0.01
book-caddy0.81 ±\pm 0.050.82 ±\pm 0.050.80 ±\pm 0.000.95 ±\pm 0.02
Average (N=20N=20)0.550.620.650.75
N=10N = 10
bowl-drawer0.71 ±\pm 0.090.93 ±\pm 0.020.57 ±\pm 0.130.92 ±\pm 0.04
cheese-butter0.27 ±\pm 0.010.43 ±\pm 0.050.47 ±\pm 0.080.70 ±\pm 0.03
moka-moka0.23 ±\pm 0.050.25 ±\pm 0.080.03 ±\pm 0.020.70 ±\pm 0.07
mug-microwave0.54 ±\pm 0.090.71 ±\pm 0.060.32 ±\pm 0.070.77 ±\pm 0.04
mug-mug0.28 ±\pm 0.030.47 ±\pm 0.020.33 ±\pm 0.010.45 ±\pm 0.03
mug-pudding0.14 ±\pm 0.040.23 ±\pm 0.010.22 ±\pm 0.070.33 ±\pm 0.06
soup-cheese0.39 ±\pm 0.110.65 ±\pm 0.170.39 ±\pm 0.010.79 ±\pm 0.02
soup-sauce0.31 ±\pm 0.050.54 ±\pm 0.060.42 ±\pm 0.050.83 ±\pm 0.07
stove-moka0.51 ±\pm 0.020.83 ±\pm 0.110.58 ±\pm 0.140.93 ±\pm 0.03
book-caddy0.80 ±\pm 0.000.85 ±\pm 0.040.49 ±\pm 0.050.95 ±\pm 0.02
Average (N=10N=10)0.420.590.380.74
N=5N = 5
bowl-drawer0.60 ±\pm 0.090.79 ±\pm 0.010.36 ±\pm 0.030.81 ±\pm 0.01
cheese-butter0.25 ±\pm 0.030.58 ±\pm 0.140.11 ±\pm 0.010.69 ±\pm 0.06
moka-moka0.07 ±\pm 0.050.27 ±\pm 0.110.03 ±\pm 0.020.16 ±\pm 0.02
mug-microwave0.45 ±\pm 0.010.39 ±\pm 0.020.14 ±\pm 0.030.51 ±\pm 0.01
mug-mug0.23 ±\pm 0.030.47 ±\pm 0.150.14 ±\pm 0.030.53 ±\pm 0.06
mug-pudding0.02 ±\pm 0.030.06 ±\pm 0.020.07 ±\pm 0.010.19 ±\pm 0.06
soup-cheese0.41 ±\pm 0.020.37 ±\pm 0.110.05 ±\pm 0.080.62 ±\pm 0.04
soup-sauce0.17 ±\pm 0.060.47 ±\pm 0.220.14 ±\pm 0.040.63 ±\pm 0.03
stove-moka0.49 ±\pm 0.080.82 ±\pm 0.210.28 ±\pm 0.100.91 ±\pm 0.07
book-caddy0.41 ±\pm 0.020.63 ±\pm 0.020.56 ±\pm 0.070.84 ±\pm 0.00
Average (N=5N=5)0.310.540.190.59

Table 6. LIBERO-10 results across different numbers of action-labeled demonstrations (N) out of 50 demonstrations per task. Each entry reports mean ±\pm standard deviation over 3 runs. Best results per task are highlighted in bold.

Simulation Benchmark

Table 6 reports per-task success rates of all baselines on the LIBERO-10 benchmark. ELADP\mathrm{ELA}_{\mathrm{DP}} and ELAUWM\mathrm{ELA}_{\mathrm{UWM}} consistently improves both DP and UWM across all evaluated settings, demonstrating stable gains in the low-data regime. These results suggest that the benefits of ELA are not confined to a specific data scale or model capacity, but instead generalize robustly across varying levels of data availability and task difficulty.

Model and Training Details

Inverse Dynamics Model. Our inverse dynamics model (IDM) is a register-based vision transformer that infers a compact latent action from an observation pair. Each frame is tokenized by a lightweight convolutional patch embedder. The transformer encoder then processes the frame pair with 16 learnable register tokens summarize the visual context under a causal-last attention mask, inducing a coarse-to-fine ordering in which each register conditions only on its predecessors. The register tokens are projected to a 4-dimensional latent and discretized by iFSQ.

Forward Dynamics Model. To ground the latent space in physical dynamics, a forward dynamics model (FDM) provides auxiliary self-supervision: a 2-layer transformer decoder with learnable patch queries cross-attends to the first-frame tokens and the inferred registers, and a U-Net upsampling head reconstructs the target frame. First-frame feeding the FDM are detached so that the predictive signal shapes the latent action rather than leaking appearance information. The IDM and FDM are trained jointly end-to-end, with the action-prediction loss applied only to action-labeled samples and the FDM reconstruction loss applied to all pairs.

Gripper Action Decoding. The action decoder maps latent actions to continuous robot control commands. Gripper control, however, is treated separately. This design is motivated by the nature of the latent action representation, which is learned through an inverse dynamics model trained on consecutive image pairs and therefore primarily captures visually observable state changes. While arm motions generate pronounced visual differences that are readily encoded in the latent action space, gripper open–close actions often result in only subtle, localized changes that are difficult to recover reliably from image observations alone.

To address this limitation, we exclude gripper control from the latent action space and instead predict it using a dedicated policy head supervised by the recorded binary gripper signal. Although this approach requires action labels for gripper prediction and therefore cannot directly leverage action-free demonstrations, our experiments show that separating gripper control from latent action decoding improves action reconstruction accuracy without compromising task performance in the low-data regime.

Diffusion Policy. The downstream policy is formulated as a latent diffusion transformer. An ImageNet-pretrained ResNet-18 encodes the two-frame observation into a conditioning vector, while a 12-layer AdaLN Transformer denoises the k×(lz+1)k \times(l_z+1) target, formed by fusing the k×lzk \times l_z latent action with the gripper channel. Here, kk denotes the latent action token length (the number of register tokens) and lzl_z the per-token latent dimension. The model is trained using a DDPM schedule, employing 50 diffusion steps during training and 10 steps during inference. Unless otherwise specified, we set k=16k = 16 and lz=4l_z = 4 across all experiments.

Representation Alignment. Following REPA-E, we introduce a perceptual representation alignment objective that encourages the latent diffusion policy to match features from a frozen DINOv3 ViT-B/16 encoder. DINOv3 patch-token features are precomputed offline for all training clips. During training, hidden states from a fixed intermediate block of the diffusion transformer are projected through an MLP and aligned with the corresponding DINOv3 features using a negative cosine similarity loss.

Training Schedule. ELA training consists of three stages: a LAM warmup, a pretraining phase, and a fine-tuning phase. During pretraining, we jointly optimize the Latent Action Model (LAM), the latent action policy, and the action decoder to learn and organize the latent action space. In the fine-tuning stage, the IDM and Action Decoder are kept frozen, allowing the policy to adapt solely within the latent action manifold learned during pretraining. The complete training schedule and optimizer configurations for both stages are provided in Table 7.

Phase 1Phase 2Phase 3
LAM WarmupPretrainingPer-task Finetuning
Trained module(s)LAM + Action DecoderLAM + Action Decoder + PolicyPolicy
Frozen module(s)––LAM + Action Decoder
Steps5k200k10k
Batch size646464
OptimizerAdamWAdamWAdamW
Learning rate5×10−55 \times10^{-5}5×10−55 \times10^{-5}1×10−41 \times10^{-4}
Weight decay001×10−61 \times10^{-6}
(β1,β2)(\beta_1, \beta_2)(0.9,0.95)(0.9, 0.95)(0.9,0.95)(0.9, 0.95)(0.9,0.95)(0.9, 0.95)
Grad-norm clip1.01.01.0
LR scheduleconstantconstantcosine (1k warmup)
EMA––✓
LossesLact,Lfdm\mathcal{L}_{\mathrm{act}}, \mathcal{L}_{\mathrm{fdm}}Lact,Lfdm,Lpolicy,Lalign\mathcal{L}_{\mathrm{act}}, \mathcal{L}_{\mathrm{fdm}}, \mathcal{L}_{\mathrm{policy}}, \mathcal{L}_{\mathrm{align}}Lpolicy\mathcal{L}_{\mathrm{policy}}

Table 7. ELA Training schedule

References

  1. [1]A. O’Neill, A. Rehman, A. Maddukuri, A. Gupta, A. Padalkar, A. Lee, A. Pooley, A. Gupta, A. Mandlekar, A. Jain, et al. Open X-Embodiment: Robotic learning datasets and RT-X models. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892–6903. IEEE, 2024.arxiv.org/abs/2310.08864
  2. [2]A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, et al. Droid: A large-scale in-the-wild robot manipulation dataset. arXiv preprint arXiv:2403.12945, 2024.arxiv.org/abs/2403.12945
  3. [3]D. Schmidt and M. Jiang. Learning to act without actions. In International Conference on Learning Representations, volume 2024, pages 9379–9395, 2024.arxiv.org/abs/2312.10812
  4. [4]S. Ye, J. Jang, B. Jeon, S. J. Joo, J. Yang, B. Peng, A. Mandlekar, R. Tan, Y.-W. Chao, B. Y. Lin, et al. Latent action pretraining from videos. In International Conference on Learning Representations, volume 2025, pages 28213–28239, 2025.arxiv.org/abs/2410.11758
  5. [5]H. Kim, J. Kang, H. Kang, M. Cho, S. J. Kim, and Y. Lee. Uniskill: Imitating human videos via cross-embodiment skill representations. arXiv preprint arXiv:2505.08787, 2025.
  6. [6]Q. Bu, Y. Yang, J. Cai, S. Gao, G. Ren, M. Yao, P. Luo, and H. Li. UniVLA: Learning to act anywhere with task-centric latent actions. arXiv preprint arXiv:2505.06111, 2025.
  7. [7]S. Routray, H. Pan, U. Jain, S. Bahl, and D. Pathak. ViPRA: Video prediction for robot actions. arXiv preprint arXiv:2511.07732, 2025.arxiv.org/abs/2511.07732
  8. [8]F. Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. arXiv preprint arXiv:1805.01954, 2018.arxiv.org/abs/1805.01954
  9. [9]S. Haldar and L. Pinto. Point policy: Unifying observations and actions with key points for robot manipulation. arXiv preprint arXiv:2502.20391, 2025.
  10. [10]M. Hong, A. Liang, K. Kim, H. Rajaprakash, J. Thomason, E. Byink, and J. Zhang. Hand me the data: Fast robot adaptation via hand path retrieval. arXiv preprint arXiv:2505.20455, 2025.arxiv.org/abs/2505.20455
  11. [11]C.-C. Hsu, B. Wen, J. Xu, Y. Narang, X. Wang, Y. Zhu, J. Biswas, and S. Birchfield. SPOT: SE (3) pose trajectory diffusion for object-centric manipulation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 4853–4860. IEEE, 2025.
  12. [12]Y. Su, X. Zhan, H. Fang, Y.-L. Li, C. Lu, and L. Yang. Motion before action: Diffusing object motion as manipulation condition. IEEE Robotics and Automation Letters, 2025.
  13. [13]C. Wen, X. Lin, J. So, K. Chen, Q. Dou, Y. Gao, and P. Abbeel. Any-point trajectory modeling for policy learning. arXiv preprint arXiv:2401.00025, 2023.
  14. [14]J. A. Collins, L. Cheng, K. Aneja, A. Wilcox, B. Joffe, and A. Garg. Amplify: Actionless motion priors for robot learning from videos. arXiv preprint arXiv:2506.14198, 2025.arxiv.org/abs/2506.14198
  15. [15]B. Chen, T. Zhang, H. Geng, K. Song, C. Zhang, P. Li, W. T. Freeman, J. Malik, P. Abbeel, R. Tedrake, et al. Large video planner enables generalizable robot control. arXiv preprint arXiv:2512.15840, 2025.arxiv.org/abs/2512.15840
  16. [16]S. Patel, S. Mohan, H. Mai, U. Jain, S. Lazebnik, and Y. Li. Robotic manipulation by imitating generated videos without physical demonstrations. arXiv preprint arXiv:2507.00990, 2025.arxiv.org/abs/2507.00990
  17. [17]C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. arXiv preprint arXiv:2504.02792, 2025.arxiv.org/abs/2504.02792
  18. [18]S. Li, Y. Gao, D. Sadigh, and S. Song. Unified video action model. arXiv preprint arXiv:2503.00200, 2025.arxiv.org/abs/2503.00200
  19. [19]J. Pai, L. Achenbach, V. Montesinos, B. Forrai, O. Mees, and E. Nava. mimic-video: Video-action models for generalizable robot control beyond vlas. arXiv preprint arXiv:2512.15692, 2025.arxiv.org/abs/2512.15692
  20. [20]R. Yang, Q. Yu, Y. Wu, R. Yan, B. Li, A.-C. Cheng, X. Zou, Y. Fang, X. Cheng, R.-Z. Qiu, et al. EgoVLA: Learning vision-language-action models from egocentric human videos. arXiv preprint arXiv:2507.12440, 2025.arxiv.org/abs/2507.12440
  21. [21]A. Liang, P. Czempin, M. Hong, Y. Zhou, E. Biyik, and S. Tu. CLAM: Continuous latent action models for robot learning from unlabeled demonstrations. arXiv preprint arXiv:2505.04999, 2025.arxiv.org/abs/2505.04999
  22. [22]X. Chen, H. Wei, P. Zhang, C. Zhang, K. Wang, Y. Guo, R. Yang, Y. Wang, X. Xiao, L. Zhao, et al. Villa-X: enhancing latent action modeling in vision-language-action models. arXiv preprint arXiv:2507.23682, 2025.arxiv.org/abs/2507.23682
  23. [23]C. Zhang, J. Wang, Z. Gao, Y. Su, T. Dai, C. Zhou, J. Lu, and Y. Tang. CLAP: Contrastive latent action pretraining for learning vision-language-action models from human videos. arXiv preprint arXiv:2601.04061, 2026.arxiv.org/abs/2601.04061
  24. [24]T. Darcet, M. Oquab, J. Mairal, and P. Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588, 2023.arxiv.org/abs/2309.16588
  25. [25]B. Lin, Z. Li, Y. Niu, K. Gong, Y. Ge, Y. Lin, M. Zheng, J. Zhang, M. Yang, Z. Zhong, et al. iFSQ: Improving fsq for image generation with 1 line of code. arXiv preprint arXiv:2601.17124, 2026.arxiv.org/abs/2601.17124
  26. [26]C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 44(10-11):1684–1704, 2025.
  27. [27]T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. arXiv preprint arXiv:2304.13705, 2023.arxiv.org/abs/2304.13705
  28. [28]C. Liu, X. Han, J. Gao, Y. Zhao, H. Chen, and Y. Du. OAT: Ordered action tokenization. arXiv preprint arXiv:2602.04215, 2026.arxiv.org/abs/2602.04215
  29. [29]A. Vahdat, K. Kreis, and J. Kautz. Score-based generative modeling in latent space. Advances in neural information processing systems, 34:11287–11302, 2021.arxiv.org/abs/2106.05931
  30. [30]O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, and M. Oquab. DINOv3. arXiv preprint arXiv:2508.10104, 2(4):5, 2025.arxiv.org/abs/2508.10104
  31. [31]S. Yu, S. Kwak, H. Jang, J. Jeong, J. Huang, J. Shin, and S. Xie. Representation alignment for generation: Training diffusion transformers is easier than you think. arXiv preprint arXiv:2410.06940, 2024.arxiv.org/abs/2410.06940
  32. [32]X. Leng, J. Singh, Y. Hou, Z. Xing, S. Xie, and L. Zheng. REPA-E: Unlocking vae for end-to-end tuning of latent diffusion transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 18262–18272, 2025.arxiv.org/abs/2504.10483
  33. [33]B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. Advances in Neural Information Processing Systems, 36:44776–44791, 2023.arxiv.org/abs/2306.03310
  34. [34]D. Nie, F. Chen, Q. Lv, J. Kuang, X. Li, X. Cao, and X. Cai. LARY: A latent action representation yielding benchmark for generalizable vision-to-action alignment. arXiv preprint arXiv:2604.11689, 2026.arxiv.org/abs/2604.11689
  35. [35]Z. J. Cui, H. Pan, A. Iyer, S. Haldar, and L. Pinto. Dynamo: In-domain dynamics pretraining for visuo-motor control. Advances in Neural Information Processing Systems, 2024.arxiv.org/abs/2409.12192
  36. [36]A. Bochkovskiy, A. Delaunoy, H. Germain, M. Santos, Y. Zhou, S. Richter, and V. Koltun. Depth pro: Sharp monocular metric depth in less than a second. In International Conference on Learning Representations, volume 2025, pages 75602–75637, 2025.

Paper details

vlnc:W_fa4kx8fm
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents