Multi-View Latent Action Pretraining for Cross-Embodiment Robot Learning from Videos

Introduction

Data availability remains a limiting factor for learning robot policies. Recent works propose leveraging action-free video data that contains semantically rich information, which can aid robots in long-term task planning. One approach infers latent actions from videos using an inverse dynamics model (IDM), and conditions a downstream policy on those latents [1, 2, 3]. The latent actions are an abstract representation describing the transition dynamics in the video. Downstream policy success highly depends on how well the latent actions predict real robot actions. A number of approaches have been proposed to improve this transferability, including grounding latent actions in real robot actions or states [4, 5] and visual pre-processing approaches [6, 7, 8]. However, we show that existing single-view approaches cannot sufficiently recover the full 6D poses that define robot actions. Single views

capture actions only ambiguously: out-of-plane motion is underdetermined and parts of the scene may be occluded.

We therefore propose a Latent Action Model (LAM) that ingests synchronized multi-camera views and attends to image patches across viewpoints—recovering depth and occlusion a single camera leaves ambiguous—before compressing them into a latent action. We demonstrate that this spatial awareness leads to better recoverability of 6D robot actions compared to single-view approaches. Crucially, multiple views are only required during LAM pretraining to improve latent action quality, but not during the policy learning stage where only a single external camera and wrist view are used.

A key advantage of learning from video is the ability to leverage cross-embodiment demonstrations, with the ultimate goal of learning from human demonstrations. Most existing works use only videos from the target robot embodiment, or leverage large-scale public datasets that confound different embodiments, tasks, and visual diversity [3, 4, 8], making performance attribution to cross-embodiment transfer difficult. We specifically evaluate our policy under cross-embodiment demonstrations, where task-specific demonstrations come from other robots in simulation or humans on hardware. We then add a small number of demonstrations on the target embodiment and investigate performance as we scale. We find that a single cross-embodiment video source already enables transfer learning both in simulation and on hardware. Adding a second viewpoint consistently outperforms adding cross-embodiments, establishing multiple viewpoints as an affordable lever for policy performance.

We propose to combine our LAM with a lightweight policy training jointly on the small action-labeled dataset and the much larger video-only set using sequential flow-matching experts. The sequential design provides a strong inductive prior, enabling the action expert to effectively leverage the latent plan. We benchmark our method on LIBERO [9], LIBERO-Plus [10], and MimicGen [11], and a real robot using 850 rollouts. Our main contributions are:

• We propose an approach for multi-view IDM-style video pretraining and a downstream policy improving action-only learning by +13.5% in simulation and +20% on hardware.

• Training LAMs on in-domain multi-view videos raises the decodability of ground-truth actions by 17% and policy success rate by +7.3% over single-view LAMs, and outperforms large-scale single-view pretraining, which does not fully recover out-of-plane motions.

• We show that a single cross-embodiment already enables skill transfer, and that videos from a second viewpoint outperform adding embodiments.

• On hardware, dual-view pretraining improves policy success rate by +20%, and we demonstrate skill transfer from human-only demonstrations for novel tasks.

Related Work

Latent Action Pretraining. Inverse-dynamics models (IDMs) for latent action learning started with LAPO [1] and Genie [2]. The LAPO design of coupling an IDM and FDM (forward dynamics model) forms the template of modern LAM approaches. LAPA [3] adopted this for robotics by finetuning a VLM on latent actions, and subsequently adding a small MLP head to predict actions. villa-X [4] uses robots’ proprioceptive states and CLAM [5] and LAOM [6] robot actions during IDM training for physical grounding of the latent actions. However, these methods ground latent actions solely in robot data from the target platform and leave human videos entirely ungrounded. LAOF [7] introduces optical flow as an intermediate representation to learn latent actions from videos, and UniVLA [8] proposes using DINO features instead of pixels. Those methods still rely on video preprocessing from a single viewpoint, whereas we propose using multiple viewpoints to improve action recoverability.

Multi-View Learning. Most robot policies operate on a single external viewpoint [12, 13], although recent work shows that 3D spatial perception yields stronger manipulation capabilities [14, 15]. One approach injects explicit geometric reconstructions using point clouds [14], depth maps [16], or voxels. These approaches rely on auxiliary depth sensors or complex reconstruction pipelines that introduce noise. Geometric structure can be learned implicitly using multi-view supervision: RoboUniView [16] pretrains a unified view representation from multi-view images, 3D-MVP [15] pretrains the visual encoder via masked autoencoding over multi-view RGB-D renderings, and CLAMP [17] learns visual representations through a contrastive objective. MV-MWM [18] and ReViWo [19] train multi-view encoder-decoders to build viewpoint-robust world models. Concurrent work MVP-LAM [20] proposes latent action alignment between multiple viewpoints.

As latent action models aim to encode robot actions in 3D space, they stand to benefit substantially from 3D-aware pretraining. We propose to leverage multi-view pretraining of LAPA-style models to improve latent action learning.

Cross-Embodiment Learning. Videos are a natural source for cross-embodiment learning. Demonstrations from different robots can be architecturally absorbed through per-embodiment input tokenizers [13], separate action heads [12], or embodiment identity tokens [21] around a shared trunk. The visual gap can be closed directly by editing pixels through inpainting at test time [22] or train time [23]. The same method can bridge the human–robot gap by rendering a robot at the estimated hand pose [24]. These methods make the mechanism of transfer interpretable, but depend on URDFs, renderers, or hand-pose estimators. XSkill [25] discovers embodiment-agnostic skills from unlabeled videos. UniAct [26] decodes a vector-quantized codebook of universal behaviors, and [27] contrastively aligns end-effector actions across hands and grippers.

LAMs can achieve embodiment transfer from video alone without explicit reconstruction. The transfer question, however, remains confounded. Work that leverages aggregated datasets [4] such as OXE [28] varies embodiment, task, scene, and visual diversity all at once. Those settings make it difficult to quantify cross-embodiment skill transfer. We adopt a controlled protocol in which task-specific video demonstrations stem from other robots in simulation and from human demonstrations on hardware.

Method

We consider language-conditioned visual manipulation. Given a visual observation $o_t \in\mathcal{O}$ and a task instruction $\ell$, the policy acts in the robot’s action space $\mathcal{A}$. We seek a policy $\pi_\theta$ that predicts an action chunk $a_{t:t+K}$ of $K$ future actions $a_t \in\mathcal{A}$ from the current context $x_t = (o_t,\ell)$. We learn this policy by imitation. The imitation objective for learning from a dataset $\mathcal{D}_a = \{(\ell^{(j)}, o_{0:T_j}^{(j)}, a_{0:T_j}^{(j)})\}_j$ of action-labeled trajectories minimizes the behavior cloning loss

The action labels are scarce and $\mathcal{D}_a$ holds only a small number of demonstrations. The second action-free dataset $\mathcal{D}_v = \{(\ell^{(k)},o_{0:T_k}^{(k)})\}_k$ holds a much larger set of video-only demonstrations of cross-embodiments. The problem we address is learning a policy $\pi_\theta$ that leverages both datasets.

Our method has two stages. A latent action model (LAM) is pretrained on videos to infer a latent action for each observed transition (see Sec. III-A), using multiple viewpoints to provide spatial grounding (Sec. III-B). Then, a policy is trained on both sources jointly in a sequential manner (Sec. III-C).

Latent Action Model (LAM)

Following [1, 2, 3], our LAM couples an inverse dynamics model (IDM) that infers an abstract latent action from observed transitions with a forward dynamics model (FDM) that reconstructs the future from the present and that latent.

Unlike the common formulation, which infers the latent from a current and a future frame of a single viewpoint [1, 3], our IDM ingests a synchronized multi-view frame pair $O_t = (o_t,o_{t+H})$ at a fixed frame offset $H$ — the horizon the policy plans over — and encodes it jointly with a multi-view spatiotemporal encoder $\Phi$, producing a fused token map for each of the two frames, $f_t = \Phi_0(O_t)$ and $f_{t+H} = \Phi_H(O_t)$. A small bottleneck $B$ compresses each map. The latent action is the difference of the bottleneck codes of the future and the current frame:

The FDM reconstructs the future frame $o_{t+H}$ from the current-frame tokens $e_t = E(o_t)$ — the patch embedding $E$ shared with the encoder $\Phi$ — and the latent $z_t$, trained with a robust reconstruction loss $\rho$ (Huber loss):

The stop-gradient $\operatorname{sg}[\cdot]$ on the current-frame tokens forces the only trainable path that lowers future-reconstruction error to run through the latent (and not through $E$), so $z_t$ captures the controllable change rather than static scene content.

Multi-Viewpoint Fusion

An action manifests as 6D poses of the end-effector, which a single RGB view captures only ambiguously — from any single viewpoint out-of-plane motion is underdetermined, and parts of the scene may be occluded. We hypothesize that multiple synchronized viewpoints give the LAM a better 3D understanding of the scene and hence more spatially-grounded latents than a single view.

Multi-view spatiotemporal encoder. Each RGB frame $I_t^{(v)}$ (view $v \in\{1,\ldots,V\}$, window offset $t \in\{0,H\}$) is split into $N$ patches; patch $p$ of view $v$ is embedded by the shared patch embedding $E$ and tagged with a learned camera-identity embedding $c_v$, giving tokens $x_{t,v,p}$. These pass through a stack of spatial attention blocks followed by a stack of temporal attention blocks. We borrow the factorized space-time attention that has proven effective for video understanding in computer vision [29] and adapt it to the multi-view setting, similar to recent world models [2, 18]. Every block uses cosine multi-head attention [30], where queries and keys are $\ell_2$-normalized, rescaled by learned per-dimension scales, and multiplied by a fixed temperature before the softmax. Rotary position encodings (RoPE) are applied to the normalized queries and keys.

Joint cross-view attention. We let the spatial block attend over the union of all views’ patches — so a patch in one view attends to patches in every other view for a given timestep $t$. This way, attention can resolve geometry and occlusion across viewpoints. The temporal block then attends across the two frames at each fixed location. Views are pooled into the fused map that defines the latent:

Encoding the views jointly — rather than with independent per-view encoders and late fusion — lets the latent exploit cross-view correspondence at every layer. This factorization suits our setting: because the viewpoints are synchronized and static, patches across views correspond to shared 3D structure at each instant, so cross-view attention in the spatial stage resolves per-frame geometry (depth, occlusion) while the temporal stage isolates how that geometry changes via the action. The resulting maps $f_t=\Phi_0(O_t)$ and $f_{t+H}=\Phi_H(O_t)$ feed the difference latent of Eq. (1).

Latent-Augmented Policy

The latent plan, as well as robot actions, are jointly consumed by the policy. We design a hierarchical policy that first predicts a latent plan from context, then an action chunk conditioned on that plan, factorizing the joint distribution as

where $x_t=(O_t,\ell)$ is the context (Figure 1). Mirroring the action chunk of $K$ actions, the latent plan $z_{[t]}=(z_{t+jH})_{j=0}^{M-1}$ is a chunk of $M=K/H$ latent steps spanning the same interval, but with fewer entries since each latent covers the $H$ frames of the LAM’s frame offset (Eq. (1)). This differs from joint latent–action models [4] where the action depends on plan through a joint denoising target instead of an explicit conditioning interface.

The interaction between the backbone and action experts alternates cross-attention to the backbone’s keys and values with self-attention over the expert’s own tokens, similar to SmolVLA [31]. Each expert is trained by conditional flow matching: for a target $y$ (a latent plan or action chunk) with conditioning $c$, noise $\varepsilon\sim\mathcal{N}(0,I)$ and time $\tau$,

with denoised estimate $\hat{y}=x^\tau-\tau v_\theta(x^\tau,\tau,c)$. The two experts differ only in their conditioning: $c_z=x_t$ for the latent expert and $c_a=(x_t,z_{[t]})$ for the action expert. Every sample of $\mathcal{D}_v$ and $\mathcal{D}_a$ carries latent labels, while only $\mathcal{D}_a$ carries action labels. We formulate a loss that jointly trains a policy on both targets:

where $\ell_i^a,\ell_i^z$ are the per-sample flow-matching losses of Eq. (5) averaged over chunk steps and dimensions, and $\lambda_a,\lambda_z$ are weights. We train the action expert using scheduled sampling [32] to enable conditioning on both ground-truth plans $z_t$ and its own noisy predictions $\hat{z}_t$ during training.

Implementation Details

Implementation Details. Our LAM trains from scratch only on in-domain videos on four NVIDIA H100 GPUs for 12 h, and the flow-matching experts for 13 h on one. Table I lists the model and training parameters.

Latent Action Probing. We use probing [4, 6]—fitting a lightweight model to predict ground-truth robot actions from frozen latent embeddings—as a metric for evaluating the quality of the learned latent actions. We report the coefficient of determination,

where $f_\theta$ is a small MLP probe fit on the frozen latents $z_i$, $i$ indexes heldout transitions in the dataset, and $D=7$ is the action dimension. The probe is small and the latents are frozen, so it cannot recover information that is absent in the latent. It reflects how well the latent predicts robot actions.

Oracle Latents. To estimate an upper bound of policy performance assuming perfectly reconstructed latent actions, we introduce oracle latents. The oracle latents are a compression of the actions over the chosen time horizon $H$. The oracle latents are computed directly from the available ground-truth 7D actions $a_t$:

and reflect the robot’s actions over the latent horizon. We train the downstream policies identically to using the real latents. It serves as an estimation of the upper performance boundary we expect when adding video data to the policy.

Model and training hyperparameters
Latent action model (344M params)
Hidden dim $d$	1024
Spatial / temporal / FDM layers	8 / 8 / 8 (FDM cross-attends $z$)
Bottleneck $B$, latent	$1024 \rightarrow8$ strided conv; $1 \times8$ cont.
Input res. / offset $H$	$256 \times256$ / 5 (0.25 s @ 20 Hz)
Reconstruction loss $\rho$	Huber, $\beta=0.1$
Policy (202M trainable / 552M total)
Backbone	SmolVLM2-500M (frozen)
Hidden dim (backbone / expert)	960 / 720
Expert layers	16 (each expert)
Chunk $K$ / plan $M$	50 / 10 (stride $H=5$)
Loss weights / teacher forcing $p$	$\lambda_a=\lambda_z=1$ / $1.0\rightarrow0.2$
Training	LAM	Policy
Steps	70k	100k
Batch (per GPU $\times$ GPUs)	$32 \times4$	$80 \times1$
LR peak $\rightarrow$ final (cosine decay)	$10^{-4}\rightarrow10^{-6}$	$10^{-4}\rightarrow2.5\times10^{-6}$
Optimizer	AdamW, bf16	AdamW, bf16

Remaining gaps are attributable to how well latent action learning actually helps task-solving, or how the policy can leverage latent actions.

Simulation Benchmarks. We benchmark on LIBERO [9], LIBERO-Plus [10], and MimicGen [11]. In all suites, we generate the dataset with a second side-view camera for LAM training. The deployed embodiment is Franka Panda for all suites, and cross-embodiments are UR5e, KUKA LBR iiwa, Sawyer, and Kinova Gen3 (those only contribute videos). LIBERO is a benchmark with four task suites: Spatial, Object, Goal, and LIBERO-10. Each task suite contains 10 tasks, and each task contains 50 demonstrations. We replay the source demonstrations on the different embodiments, yielding 160 action-annotated Panda and 4,293 video cross-embodiment demonstrations across 32 tasks. In addition, we define eight tasks (two per suite) as heldout tasks that never see any robot action from Franka Panda to test generalization under video-only settings. LIBERO-90 adds a further 13,338 episodes (3,936 Panda) of off-task play data. LIBERO-Plus [10] adds visual disturbances on top of LIBERO, and we follow the same generation recipe. MimicGen [11] is a data-generation scheme including a suite of 28 challenging, high-precision tasks. Retaining only task-embodiment combinations with $>20\%$ generation success rate yields 107 of 140 task$\times$embodiment combinations in 42.3k episodes, of which 12.7k are Panda. We use 22 tasks as play data and evaluate on 6 eval tasks.

Evaluation

Policy Performance under Video Supervision (Table II)

We evaluate policies under settings with scarce action demonstrations on the deployed embodiment, and the in-domain cross-embodiment demonstrations always only contribute videos. For comparison, we train policies on the scarce action budget only without using video data achieving 42.7% SR, and on the full action budget of the entire dataset achieving 75.1%. Those values provide upper and lower reference points. To verify that the lower reference is a ends_mid=1 data gap and not a model gap, we finetune a model with significantly more parameters than ours ($\pi_{0.5}$ 4.1B vs. our 202M) and 4× more compute. $\pi_{0.5}$ does not recover the full action budget SR. These results show that the little action availability limits policy performance, and leaves room for improvement when adding video data.

Method	MimicGen	LIBERO	LIB.+	Mean
Full action budget
Ours	65.3	75.0	85.1	75.1
Low action budget
Ours (202M params)	25.7	52.6	49.8	42.7
$\pi_{0.5}$ (4.1B params)	32.0	53.0	54.0	45.0
Low action budget (our policy) + LAM
villa-X (state grnd., VQ latent)†	18.8	42.8	45.1	35.6
villa-X (state grnd., cont. latent)†	22.0	43.8	51.5	39.1
Ours (single-view)	31.0	53.5	62.1	48.9
MVP-LAM‡	24.7	57.6	65.4	49.1
UniVLA (DINOv3 features)‡	28.7	59.3	60.9	49.6
CLAM (action grounding)‡	32.3	57.2	63.1	50.9
LAOF (flow constraint)‡	33.4	58.8	60.5	50.9
LAPA-pretrained	30.4	59.9	67.7	52.7
Ours (multi-view)	38.7	62.3	67.6	56.2
Oracle*	39.2	71.0	72.4	60.9

Cross-embodiment video supervision in the single-view setting improves SR by an absolute +6.2% for our method. Multi-view pretraining improves policy SR by another +7.3%, resulting in +13.5% over action-only. This demonstrates that multiple viewpoints add significant information to the latent that the policy can leverage to determine robot actions. The gains from adding multi-view are concentrated in better 6D information contained in the latent actions as we show in Figure 3. The improvements from adding a second viewpoint are more significant than baseline methods that propose visual preprocessing of single-view inputs (LAOF), action grounding (CLAM), or use large-scale pretraining (LAPA).

MVP-LAM also pretrains on both views, yet significantly

trails our approach. MVP-LAM couples viewpoints through a cross-view reconstruction loss but encodes them separately, whereas our LAM jointly encodes both viewpoints into a single latent that resolves depth. CLAM-style action-grounding improves policy success only by +2.0% compared to our +7.3%. This is because action-grounding can only use a subset of the available dataset for grounding instead of the entire dataset—only the fraction of data carrying actions.

Training on a small dual-view dataset outperforms a baseline pretrained on orders of magnitude more single-view data. The closest comparison to ours is initializing our single-view variant pretrained weights from LAPA trained on 970K robot trajectories [3], and then finetuning on our data. Despite being pretrained on two orders of magnitude more data, it does not outperform our dual-view LAM. The advantage of LAPA-pretraining over no pretraining originates from general visual robustness, but not from fully recovered latent action information in the depth dimension (Figure 3). For instance, LAPA performs best on LIBERO+, which introduces visual disturbances. Villa-X was pretrained on 5.2M human and robot trajectories, and we re-use their checkpoints without finetuning, as no code is available. The best version does not improve on baseline. More single-view data cannot fully resolve the depth ambiguity of a 2D projection. Thus, pretraining on orders of magnitude more single-view data does not necessarily substitute for a second, calibrated viewpoint.

Scaling With Sources of Demonstrations (Fig. 2)

Figure 2 (a) scales the available action-annotated demonstrations on the deployed embodiment, while keeping the rest fixed. For all budgets, video pretraining consistently improves on action only. The results are separated by in-distribution and heldout tasks. On in-distribution tasks, where the target embodiment receives a varying number of action-annotated demonstrations, the advantage of video data decreases as more demonstrations are added. The heldout tasks only receive cross-embodiment videos but no action-annotated demonstrations, and the SR remains near zero without adding video demonstrations (action-only). Adding cross-embodiment video demonstrations achieves 48.6% SR on heldout-tasks. Thus, the video behavior of cross-embodiment demonstrations successfully transfers to

the unseen tasks. We observe that increasing the demonstrations for other tasks also increases performance on heldout tasks. This is because LAM and policy have more data to ground latent actions in real actions.

In Figure 2 (b) we vary the amount of play data, i.e., data from non-eval tasks. On in-distribution tasks, the size of the play data has marginal impact, with 0 play data performing as well as the full play dataset of $\approx10$ k cross-embodiment and $3$ k panda trajectories (66.6% vs 66.8%). For heldout-tasks that never receive action demonstrations, a larger off-task play data corpus improves SR, as the policy has been exposed to more diverse data and scenes, helping generalization. For generalization to novel tasks a large play dataset is desirable.

In Figure 2 (c), we scale the number of cross-embodiments that contribute videos for both single- and dual-view training. In general, adding a second viewpoint is a cheaper way to improve policy performance compared to adding additional embodiments. A single cross-embodiment with a dual-view setting outperforms four additional embodiments in single-view settings. For both single-view and dual-view, a single embodiment recovers most of the gains for the heldout tasks. For instance, the dual-view LAM achieves 58.3% with one cross-embodiment compared to 58.9% with four (not shown in the figure). We also show behavioral transfer across embodiments in Figure 3 (right) and Figure 6. After convergence, the latents of different embodiments merge into a single manifold, i.e., different robots reuse the same latents, and the latent encodes what the action is and not which robot executes it. Similarly, on hardware, we show that the predicted latents transfer between human and robot embodiments (Figure 6).

Latent Quality for Dual-View LAMs and Predictiveness for Policy Success Rate (Fig. 3)

The second view improves action decodability $R^2$ from 0.54 for single-view to 0.63 for dual view (+17%). The gains concentrate in the dimensions $\Delta x$ (0.62 to 0.83 avg.), and $\Delta y$ (0.29 to 0.41 avg.), which are the out-of-plane motions for the first camera view. Thus, the second view effectively recovers action-decodability for 6D motions and does not merely overfit to existing dimensions. Cross-view attention resolves movements across views, so latents represent 3D motion rather than its ambiguous 2D projections into a single view.

In contrast, large-scale pretraining (LAPA) primarily improves reconstruction of existing in-plane motions and partially gripper actions, but does not recover motions in the out-of-plane direction. This shows that simply scaling single-view data collection is not an effective way to recover 3D motions. Instead, collecting more high-quality calibrated multi-view datasets for robot learning is desirable. Similarly, baselines that use visual preprocessing methods on single-view, such as optical flow constraints (LAOF) or DINO encoding (UniVLA) primarily improve on in-plane motions and gripper actions, but not in out-of-plane directions.

For CLAM-style action grounding, we find that action decodability is separated by embodiments: $R^2 = 0.74$ on Panda vs. 0.57 on the others. This is due to an asymmetric training scheme that leverages action grounding only on the target embodiment for which actions are available. This leads to a separated latent space and hinders cross-embodiment learning. For instance, action grounding improves policy SR by +5.0% on in-distribution tasks for which Franka Panda demonstrations are available, but not on heldout tasks that only see cross-embodiment videos (−1.4% SR). In comparison, for our dual-view pretraining action decodability stays flat across embodiments (0.58 Panda vs. 0.61 others), and we observe significant gains even on heldout tasks (+15.1%), proving effective skill transfer.

Reconstructing gripper actions from video is particularly difficult, as they are often low-displacement events that are weakly reflected in pixel changes. CLAM action grounding and LAOF flow constraint help here, as ground-truth gripper actions or informative flow loss help to capture those fine movements. Flow constraints and action grounding are both complementary to multi-view learning. We show that failed gripper actions are a major failure case when learning from video data (§ 5-E).

The probing metric reliably identifies degenerate LAMs that do not capture the relative movement between frames. For instance, villa-X’s quantized latents probe at only 0.26 and yield the lowest downstream policy SR (not shown in the figure). Across the 18 LAM-suite combinations in Figure 3, probe $R^2$ and SR correlate with $r = 0.83$. Within a benchmark, LAMs whose $R^2$ lies within 0.05 of each other are ordered correctly in 59% of cases.

Gap of Latent-Augmented to Action-Supervised Policies

The oracle latents replace the learned latents with targets computed from the ground-truth actions (Sec. IV), estimating perfect latent reconstruction. The ours-vs-oracle gap therefore isolates teacher quality, i.e., how much does policy performance improve with perfect latent actions.

Our multi-view latents recover 99% of the oracle baseline for MimicGen, 88% for LIBERO, and 93% for LIBERO+. Thus, our LAM is close to the performance expected under estimated perfect latent actions, highlighting its reconstruction quality.

The remaining gap between oracle and the full action policies is larger on some benchmarks. Oracle recovers 95% of the full action budget policies for LIBERO, 85% for LIBERO+, and 60% for MimicGen. The oracle latents reflect our LAM training settings; specifically, they compress the ground-truth actions over the latent’s horizon. Tasks that require high precision have limited benefit from sparsely annotated latent actions and require dense action chunks instead. The performance bottleneck is thus not necessarily latent reconstruction, but settings of the LAM and latent usefulness for the particular tasks. Analysis further shows that oracle latents lower overall failures, but do not recover grasping performance. Full action supervision cuts grasping failure to 42.5%. Grasping is therefore limited by action supervision, not by latent quality.

Real World Experiments (Fig. 4, Fig. 5 Fig. 6)

On hardware, we use human demonstrations as video data. The policies run on a TRLC-DK1 ⁵ robot.

Adding human videos on average improves policy by +20% absolute across all tasks. Multi-view LAM training is a significant driver, improving on average by +25% over single-view (46% vs. 21%), explained by better action recoverability through the second view (Figure 7) similar to our results in simulation. In Figure 6, we demonstrate latent transfer between human hands and robot rollouts.

’Novel placement’ tasks achieve the highest SR (avg. 51%), followed by ’novel object’ (avg. 34%). Placement semantics do not require dexterous object interaction and can be carried over from human videos, leading to high SRs. In contrast, handling novel objects (34%) and novel movements (16%) requires object interaction that is difficult to recover without robot demonstrations.

While we observe generalization under human-only settings without any robot data, the SRs for these cases remain relatively low ($\approx20\%$), limiting the practical applicability of zero-shot transfer.

Scaling-wise, adding five robot demonstrations on the target embodiment lifts SR from 26% to 42% (+16%) with video, and 6% to 23% (+17%) for action-only. Even for high numbers of robot demonstrations for a task (50), adding human data improves policy success rate in real-world settings. This indicates that our video and robot data are complementary.

Failure cases. Analysis of failure cases shows that human³ videos mainly improve high-level planning. Adding videos reduces wrong-plan failures (e.g., gripping far off the object) from 52% to 23% of rollouts, while missed grips still account for 54% of the remaining failures (see Figure 8). Simulation shows the same pattern, where grasping accounts for 51% of failures on LIBERO. Grasping and object interaction remain a major failure case.

Conclusion

Our results show that spatial grounding through multi-view training is an effective and affordable lever for robot policy training. Calibrated dual-view datasets can surpass far larger single-view corpora, showing that simply scaling single-view data collection is not an effective way to recover 3D motions. Scaling embodiments is a secondary factor, and a single cross-embodiment such as a human hand is sufficient to achieve skill transfer. Video and robot demonstrations remain complementary; gains from video data are task-dependent and diminish on precision bottlenecks and tasks relying heavily on gripper actions, where more robot data supervision is required.