betaFeedback

WorksComputer Science

UMI-Live for In-The-Wild Bimanual Robotics Data Collection

Abstract

Universal Manipulation Interface (UMI) devices allow human experts to demonstrate tasks for robot learning without teleoperating physical robots. Using these demonstrations as training data still requires a consistent hardware setup, calibration, and data processing. We present UMI-Live, a full-stack open-source bimanual UMI system for in-the-wild data collection. We design our custom UMI-Live hardware to integrate camera and LiDAR–inertial sensing, onboard compute, and power, without external tracking equipment or calibration within the collection environment. We implement a simple user interface for recording data while an asynchronous pipeline performs parallel SLAM reconstruction and publishes training-ready datasets. We also release software that supports dual-YAM teleoperation and VLA policy training and inference. We validate UMI-Live data quality by fine-tuning π0.5π_{0.5} and MolmoAct2, achieving success rates of 10/10 and 8/10, respectively on a bimanual pick-and-place task. Furthermore, we release 40 hours of UMI-Live data across three tasks, and complete hardware designs, software, and reproduction guides at our \href\href{https://umi-live.github.io/}{\text{project website}}.

Introduction

Much of the recent progress in robot manipulation has come from imitation learning, in which policies learn manipulation behaviors from human demonstrations, increasingly drawn from large and diverse datasets. Diffusion policies [1] and vision-language-action (VLA) models such as RT2, OpenVLA, π0\pi_0, π0.5\pi_{0.5}, and GR00T N1 [2, 3, 4, 5, 6] are all trained this way. More recently, world-action models (WAMs) jointly predict future visual observations and robot actions [7, 8, 9], which allows them to build on large-scale video pretraining. Video alone, however, carries no robot action labels, so WAMs still learn to act from action-labeled demonstrations. These methods allow robots to perform tasks ranging from pick-and-place to dexterous manipulation, including towel and shirt folding [4]. Learning these skills in new environments still depends on expert demonstrations covering the relevant objects and interactions. Collecting high-quality demonstrations across diverse environments therefore continues to be a key challenge in robot learning.

Action-labeled demonstrations are commonly collected by teleoperating physical robots. Large collection efforts such as DROID and ABC-130K show the value of collecting demonstrations across operators, tasks, and environments [10, 11]. Because each collection site must host the robot along with its control, perception, and safety infrastructure, expanding to new environments requires transporting this equipment or maintaining additional robot stations. The Universal Manipulation Interface (UMI) removes this requirement by letting experts demonstrate tasks with handheld grippers instead of robots [12]. UMI showed that policies trained on wrist observations and reconstructed gripper trajectories from in-the-wild demonstrations can perform challenging tasks, including bimanual ones.

Subsequent UMI systems have explored different sensing, tracking, and gripper designs. However, reproducing these systems requires clear instructions for building and calibrating the hardware, together with software for processing the recordings. Some UMI configurations use stationary tracking rigs or backpack-mounted computers, which add equipment to set up or carry [13, 14]. Bulky gripper designs can also make dexterous manipulation of small objects difficult [13]. Furthermore, each recording must be reconstructed and converted into a training-ready dataset. Manually managing these steps becomes tedious, and we believe that operators should be able to start the next demonstration while the software prepares completed recordings for training and physical robot evaluation.

We present UMI-Live, a complete hardware and software system for in-the-wild bimanual collection and robot learning. Each UMI-Live gripper integrates camera and LiDAR–inertial sensing, onboard compute and storage, and removable power. An operator can continue recording data with the press of a single button, while an asynchronous pipeline transfers completed recordings and performs parallel reconstruction, gripper position and jaw opening estimation, validation, and dataset preparation. We also provide software for dual-YAM teleoperation, policy training, and inference. We release approximately 40 hours of demonstrations across three tasks and evaluate policies fine-tuned on mixtures containing mostly UMI demonstrations. The handheld demonstrations are collected at a different location and under different lighting conditions from those used for robot evaluation, while the smaller teleoperation component provides demonstrations from the target robot and scene.

We make the following contributions.

  1. Reproducible bimanual hardware for in-the-wild collection. We present self-contained handheld grippers with integrated sensing, compute, storage, power, and operator controls, together with hardware designs and construction and calibration guides for reproducing the system.

  1. Complete software integration from recording to robot deployment. We provide software for coordinated recording, asynchronous transfer, parallel and batched processing, LeRobot dataset preparation, dual-YAM teleoperation, and policy training and inference. Operators can continue collecting while completed demonstrations are processed.

  1. A public bimanual dataset and physical robot validation. We release approximately 40 hours of demonstrations across three manipulation tasks and evaluate two VLA policies through physical rollouts. Following mixed-data fine-tuning, π0.5\pi_{0.5} succeeds in up to 10/10 trials and MolmoAct2 succeeds in 8/10 trials compared with 0/10 for its released checkpoint.

Related Work

Robot demonstration collection. Teleoperation records observations and actions directly on the target embodiment. BridgeData V2, DROID, RoboMIND, ALOHA Unleashed, and ABC-130K illustrate the breadth of manipulation data that can be collected through physical robot operation [15, 10, 16, 17, 11]. These efforts cover different tasks, environments, and embodiments, but require a robot for demonstration capture. UMI-Live supports handheld collection independently of robot operation and provides a separate teleoperation configuration for obtaining complementary demonstrations on the target embodiment.

Handheld UMI systems. The original UMI combines a handheld gripper, wrist video, gripper-state sensing, and visual–inertial trajectory reconstruction to support robot learning from portable demonstrations [12]. FastUMI develops the hardware, tracking, and collection tools for broader deployment [18]. UMI-3D introduces LiDAR-centered reconstruction and an integrated acquisition, calibration, training, and deployment pipeline, with a reported single-arm configuration [19]. UMI-Live uses two independently instrumented LiDAR–inertial devices as one coordinated bimanual collection system.

Other systems make different choices about sensing and interaction. YUBI emphasizes finger-aligned operation and bimanual dexterity, with stationary and portable collection configurations [13]. ActiveUMI combines a VR headset with a backpack-mounted computer to record active egocentric perception [14]. HiFi-UMI uses head-mounted stereo–inertial sensing, hardware synchronization, and direct relative-pose estimation to improve bimanual trajectory fidelity [20]. UMI-Live places sensing, compute, storage, power, and recording controls on the handheld devices and connects their outputs to an automated processing system and a dual-YAM learning workflow.

Collection and robot-learning software. LeRobot provides shared dataset representations and tools for policy training, evaluation, and inference [21]. RoboPocket connects portable collection to remote policy feedback and asynchronous fine-tuning [22], and ABC provides open infrastructure spanning collection, training, and evaluation [11]. UMI-Live combines bimanual LiDAR–inertial recording, asynchronous reconstruction and LeRobot dataset publication, and dual-YAM teleoperation, policy training, and inference in a reproducible setup.

System Design

Figure 1 shows how UMI-Live connects in-the-wild bimanual collection to automated processing, LeRobot datasets, and physical robot learning. We first describe the handheld hardware, including the gripper mechanism, sensing, and power design (§ 3.1), followed by recording, trajectory reconstruction, dataset export, and parallel processing (§ 3.3). We then evaluate policies trained on the collected demonstrations (§ 4).

UMI-Live connects in-the-wild bimanual collection and automated processing to LeRobot datasets, policy training, and physical evaluation on dual i2RT YAM arms

Figure 1. UMI-Live connects in-the-wild bimanual collection and automated processing to LeRobot datasets, policy training, and physical evaluation on dual i2RT YAM arms.

Hardware Design

Design objectives. UMI-Live hardware consists of two handheld grippers with onboard sensing, compute, and storage for bimanual data collection. Our design focuses on modularity, usability, and user-friendliness. Figure 2 shows the assembled unit from the front, side, and rear, and Figure 3 shows its main components. Appendix § 8 provides additional hardware details.

Assembled UMI-Live hardware: (a) front view, (b) side view, (c) rear view

Figure 2. Assembled UMI-Live hardware. (a) Front view showing the wrist camera and ArUco tags. (b) Side view showing the operator-facing buttons and battery module. (c) Rear view showing the LCD and cable routing.

Exploded CAD view of one UMI-Live unit with labels: Camera; ArUco tag locations; Redesigned gripper fingers; Removable battery module; LiDAR; LCD screen; Raspberry Pi 5; Operator-facing buttons

Figure 3. Exploded CAD view of one UMI-Live unit with its main components labeled.

Gripper mechanism. We improve the original UMI gripper mechanism [12] by using bearing-supported trigger and gear pivots for smoother actuation and reduced backlash. Squeezing the trigger rotates two sector gears that engage racks on the finger mounts (Figure 9a,Figure 9b). The mounts travel along a linear guide to produce opposing parallel-jaw motion. To achieve smoother gripper motion, we apply adhesive tape as shown in Figure 9. We also place ArUco tags [23] on the moving finger mounts to measure the jaw opening.

Gripper bearing mechanism

(a) Gripper bearing mechanism

Trigger and spring

(b) Trigger and spring

Power enclosure

(c) Power enclosure

Onboard electronics: LCD, Raspberry Pi, buttons, camera, and LiDAR

(d) Onboard electronics: LCD, Raspberry Pi, buttons, camera, and LiDAR

Figure 9. Mechanical and electronic components of UMI-Live. (a) Gripper mechanism with sector gears, bearing-supported pivots, and a linear guide. (b) Trigger and return spring within the handle. (c) Removable power module with the battery, DC-DC converter, power switch, and wiring. (d) Onboard electronics, showing the Raspberry Pi and LCD before LiDAR installation (left), operator-facing buttons and the rear-mounted LCD (center), and the front camera beneath the LiDAR (right).

Gripper finger design. We redesign the UMI gripper finger to adapt to more tasks that involve dexterous manipulation with smaller objects. We print the gripper fingers with a rigid PLA+ base and a thin TPU contact layer (Figure 4). The rigid base resists bending under grasping loads and it supports force application and precision contact with small objects such as screws. The TPU layer provides compliance at the contact surface, which helps the fingers conform to different object shapes. We demonstrate the capabilities of the redesigned grippers through human-operated tasks involving small-object grasping and insertion, match lighting, and shirt folding in Appendix § 9.

Redesigned gripper fingers and their installation on a YAM arm

Figure 4. Redesigned gripper fingers and their installation on a YAM arm.

Sensing and electronics. Each gripper carries a RoboSense Airy LiDAR with an integrated IMU4, a global-shutter wide-angle USB camera5, and a Raspberry Pi 5 (Figure 3 and Figure 9d). The camera records wrist views at 800 × 600 pixels and 30 fps. We choose to use an ultrawide camera to capture the workspace and relevant global context as we do not use an overhead top camera.

Power. We use a removable battery module that clips to the base of each handle (Figure 9c). The module houses the battery, power switch, and DC-DC converter, which supplies 5 V USB-C power to the Raspberry Pi and 12 V to the RoboSense Airy LiDAR.

In-the-wild operation. Operators start and stop recording on both grippers with a single button, and the LCD shows recording and transfer status. Operators can collect in the wild without a robot, tripod-mounted cameras, external tracking equipment, or environment specific calibration. Table 1 lists the main specifications.

SpecificationValue
Mass per unit with battery1.42 kg
Gripper width80 mm
Runtime per charge1.5 h
Hardware cost per pair$1,930

Table 1. Hardware specifications of the evaluated revision.

Operator Interface and Recording

The operator starts and stops recording on both grippers with a single button. The LCD shows recording status and transfer progress (Figure 5).

LCD interface with example values

Figure 5. LCD interface with example values.

For episode ee, each gripper i∈{L,R}i \in\{L, R\} stores timestamped wrist video and raw LiDAR and IMU packets locally on its Raspberry Pi as Ri(e)\mathcal{R}^{(e)}_i. Together, the two recordings form R(e)=(RL(e),RR(e))\mathcal{R}^{(e)} = (\mathcal{R}^{(e)}_L, \mathcal{R}^{(e)}_R). Completed recordings are transferred to the server and processed asynchronously, which allows the operator to continue collecting demonstrations.

Automated Data Processing

Once the completed recordings reach the server, they enter the processing pipeline shown in Figure 6. The server validates synchronization, estimates TCP trajectories and gripper openings, and combines these measurements with wrist images into episodes ready for training.

Data collection and processing in UMI-Live

Figure 6. Data collection and processing in UMI-Live. (a) Each gripper i∈{L,R}i \in\{L, R\} records wrist images IiI^{i}, LiDAR data Pi\mathcal{P}^{i}, and IMU data Ui\mathcal{U}^{i}. (b) Raw episodes R(e)\mathcal{R}^{(e)} are processed in parallel to estimate jaw widths wtiw^{i}_{t}, reconstruct poses TtiT^{i}_{t}, and align the streams. (c) We assemble demonstration trajectories T(e)\mathcal{T}^{(e)} with wrist images, 14-dimensional bimanual states, and task instructions into dataset D\mathcal{D}. Training samples use observations ot=(ItL,ItR,gtL,gtR,ℓ)o_{t} = (I^{L}_{t}, I^{R}_{t}, g^{L}_{t}, g^{R}_{t}, \ell) and query-relative action chunks At∈RH×20A_{t} \in\mathbb{R}^{H \times20}, as defined in Appendices § 12 and § 13.

Synchronization and time correction. The server first checks the synchronization evidence saved during recording and excludes episodes with unacceptable sensor synchronization or an estimated cross-device offset above 5 ms. During recording, Chrony synchronizes the Raspberry Pi clocks, and gPTP synchronizes each Airy sensor with its connected Pi. Camera exposure timestamps are mapped from the Pi’s monotonic clock to host time. The server then derives an affine mapping between embedded LiDAR timestamps and Pi-side packet-arrival timestamps. The corrected timestamps are used when reconstructed poses are aligned with camera observations.

Pose and gripper estimation. The server decodes each device’s LiDAR and IMU packets and runs GLIM [24] independently for the two grippers. The tool center point (TCP) is defined at the center of the gripper tips. A fixed sensor-to-TCP offset converts the reconstructed LiDAR body motion into a TCP trajectory. Appendix § 14.§ 2 specifies the corresponding TCP frame and flange-to-TCP transform used on the YAM arms.

Jaw opening is estimated from the two ArUco tags attached to each gripper’s moving finger mounts. The server unprojects tag centers through the calibrated fisheye model and combines their angular separation with the fixed camera-to-tag geometry to estimate physical width. Missing detections are interpolated between valid measurements. Calibrated closed and open endpoints map these widths to normalized gripper values.

Temporal alignment and dataset assembly. The leader camera defines the episode timeline. We interpolate reconstructed positions linearly and rotations through quaternion spherical interpolation at these frame times. The follower video contributes the latest available frame at or before each corresponding time, and gripper openings are interpolated onto the same timeline.

Each hand’s TCP pose is expressed relative to its own fixed episode-start pose. At timestep tt, its state is

sti=[pti;ϕti;gti]∈R7,s^{i}_{t} = [p^{i}_{t}; \phi^{i}_{t}; g^{i}_{t}] \in\mathbb{R}^{7},

@page 6/22 starts_mid=1 ends_mid=0

where ptip_t^i is position in meters, ϕti\phi_t^i is a rotation vector in radians, and gti∈[0,1]g_t^i \in[0,1] is normalized gripper opening. The bimanual state is

stbi=[stL,stR]∈R14.s_t^{\mathrm{bi}} = [s_t^L, s_t^R] \in\mathbb{R}^{14}.

An episode with NN frames and task instruction ℓ\ell becomes

T=[(ItL,ItR,stbi,ℓ)]t=0N−1.\mathcal{T} = \left[(I_t^L, I_t^R, s_t^{\mathrm{bi}}, \ell)\right]_{t=0}^{N-1}.

We collect these trajectories into D={T(e)}e=1M\mathcal{D} = \{\mathcal{T}^{(e)}\}_{e=1}^{M}, where MM is the number of episodes.

The software checks stream integrity, synchronization evidence, required pose and gripper measurements, frame counts, and dataset consistency before accepting an episode. Finalized episodes are stored in LeRobot format. For cloud publication, the software synchronizes the dataset and raw-episode archive with Hugging Face. The software retains the local dataset if publication fails.

Parallel execution. Processing follows a dependency graph across hands and episodes. LiDAR decoding, SLAM, and gripper estimation execute concurrently when their inputs are available. Dataset assembly waits for the required outputs and uses a serialized writer. The software tracks completed stages so interrupted processing can resume from available outputs.

Policy Training and Deployment

We fine-tune π0.5\pi_{0.5} and MolmoAct2 on UMI and robot teleoperation demonstrations using the same observation and action representation. The policies take two wrist images, gripper openings, and a task instruction as input. They predict action chunks containing query-relative TCP poses and absolute gripper openings. Appendix § 13 describes the representation, training objective, and settings.

During deployment, a remote policy server returns action chunks to the local dual-YAM client, which converts them into joint and gripper commands and coordinates the two arms. Appendix § 14 describes inference and the kinematic model.

Experimental Results

We evaluate whether demonstrations collected with UMI-Live support bimanual policy learning and physical robot execution. Our evaluation uses an orange-to-bowl task on dual YAM arms. We fine-tune π0.5\pi_{0.5} [5] and MolmoAct2 [25] using mixtures containing mostly UMI demonstrations and compare the fine-tuned MolmoAct2 policy with its released checkpoint.

Experimental Setup

In our task, the robot must pick up six oranges from a table and place them in a bowl (Figure 7b). Each trial starts from a fixed robot configuration, with randomized orange and bowl positions. A trial succeeds if all six oranges are placed in the bowl and fails if any remain outside the bowl when the trial ends. We allow grasp retries and recovery of dropped oranges. We conduct ten trials for each evaluated condition.

Figure 7. Handheld collection and physical robot evaluation. (a) Wrist-camera views at two moments in a UMI demonstration collected at a different location and under different lighting from the robot evaluation, brightened for display. (b) Dual i2RT YAM arms with UMI-Live gripper fingers, six oranges, and a bowl. (c) Selected frames from one autonomous π0.5\pi_{0.5} rollout that ends with all six oranges in the bowl. Frames are ordered from left to right. Appendix § 10, Figure 13, shows the full sequence.

The UMI demonstrations were collected at a different location and under different lighting from the robot evaluation (Figure 7a). Robot teleoperation demonstrations were collected in the evaluation scene using the setup described in Appendix § 11. The experiments therefore evaluate learning from handheld demonstrations supplemented with robot data from the target environment.

During policy execution, the robot supplies two ultrawide wrist-camera images, the current gripper openings, and a task instruction. TCP poses and joint angles are omitted from the policy inputs. The policies predict bimanual action chunks containing relative TCP poses and absolute gripper openings. The execution controller maps these predictions to robot joint and gripper commands, as described in Appendix § 14.

Training Conditions

The training sources contain 296 UMI demonstrations and 75 robot teleoperation demonstrations of the orange-to-bowl task. These counts describe the training pool for this task, while the full approximately 40-hour dataset release spans three manipulation tasks.

We evaluate π0.5\pi_{0.5} with UMI/teleoperation data splits of 95%/5% and 90%/10%, and MolmoAct2 with a split of 75%/25%. Each split specifies the UMI and robot teleoperation shares of the fine-tuning data, which sum to 100%. Both sources are converted to the common observation and query-relative action representation described in Appendix § 13.§ 2.

We initialize each π0.5\pi_{0.5} run independently from its released base checkpoint and MolmoAct2 from the released MolmoAct2-BimanualYAM checkpoint. We update all π0.5\pi_{0.5} parameters. For MolmoAct2, we train LoRA [26] adapters in the vision-language backbone and the full action expert and keep the remaining backbone parameters frozen. We evaluate the final checkpoint from each run and use one training seed per condition. Appendix § 13.§ 3 and Table 3 and Table 4 provide the remaining hyperparameters.

HyperparameterValue
Global batch size32
Parameters updatedFull model
Peak learning rate2.5×10−52.5 \times10^{-5}
AdamW β\beta(0.9,0.95)(0.9, 0.95)
AdamW ϵ\epsilon10−810^{-8}
Weight decay0.01
Model precisionFP32
Flow samples per training query1

Table 3. π0.5\pi_{0.5} fine-tuning hyperparameters for the fresh-base runs.

HyperparameterValue
Global batch size32
Parameters updatedVLM LoRA adapters and full action expert
Peak adapter learning rate5×10−55 \times10^{-5}
Peak action expert learning rate10−410^{-4}
AdamW β\beta(0.9,0.95)(0.9, 0.95)
AdamW ϵ\epsilon10−610^{-6}
Weight decay0
LoRA rank / α\alpha / dropout64 / 16 / 0.05
Language-model residual dropout0.1
Model precisionBF16
Flow samples per training query8

Table 4. MolmoAct2 fine-tuning hyperparameters for the fresh-base run.

As a baseline, we evaluate the released MolmoAct2-BimanualYAM checkpoint without additional training on our demonstrations.

Results

Table 2 reports the results. π0.5\pi_{0.5} succeeds in nine of ten trials with the 95/5 mixture and all ten trials with the 90/10 mixture. MolmoAct2 succeeds in eight of ten trials after fine-tuning with the 75/25 mixture, compared with zero successful trials for its released checkpoint. Figure 7c shows selected frames from one autonomous π0.5\pi_{0.5} execution. The full sequence appears in Appendix § 10, Figure 13.

PolicyTraining conditionUMI / teleoperationSuccessful trials
MolmoAct2Released checkpoint—0/10
π0.5\pi_{0.5}Fine-tuned95% / 5%9/10
π0.5\pi_{0.5}Fine-tuned90% / 10%10/10
MolmoAct2Fine-tuned75% / 25%8/10

Table 2. Physical orange-to-bowl evaluation on dual i2RT YAM arms. Every fine-tuned condition starts from its released checkpoint and receives 12,000 updates with a global batch size of 32. Success requires placing all six oranges in the bowl.

One autonomous rollout on dual i2RT YAM arms, transferring six oranges from the table into a bowl

Figure 13. One autonomous π0.5\pi_{0.5} rollout on dual i2RT YAM arms, transferring six oranges from the table into a bowl. Frames are shown in chronological order from left to right and top to bottom.

These results demonstrate successful physical execution after fine-tuning on mixtures containing mostly UMI-Live demonstrations. The handheld demonstrations were collected outside the evaluation environment, while the smaller teleoperation component supplies observations and motions from the target robot and scene. The evaluation covers one task and one training seed per condition.

Released Dataset

We release 3,255 bimanual episodes comprising approximately 39.7 hours at 30 Hz. The release covers orange placement, water pouring, and towel folding, with synchronized wrist videos, bimanual pose and gripper states, and task instructions. The complete dataset is available on Hugging Face.

Each hand uses a fixed episode-start reference, and we provide source provenance in the release. The standardized action field stores the same-frame state. Training loaders select future frames and derive the query-relative targets described in Appendix § 13.

Discussion and Limitations

UMI-Live spans handheld collection, automatic data preparation, and robot execution, and enables operators to continue collecting while completed recordings are simultaneously processed. However, UMI-Live devices still require one-time camera and gripper calibration. Additionally, the quality of LiDAR–inertial reconstruction depends on scene geometry and motion. Integrating sensing, compute, storage, and power increases mass to 1.42 kg per unit. Our robot evaluation covers one task, ten trials per condition, and one training seed per condition, using mixtures of UMI and teleoperation data. Without UMI-only and teleoperation-only comparisons, we do not isolate the contribution of either source. Further evaluation should measure trajectory accuracy, processing latency and throughput, operator efficiency during extended collection, and policy performance across more tasks, training seeds, and data mixtures.

Conclusion

We present UMI-Live, a bimanual collection and robot-learning system that combines self-contained handheld hardware with automated data preparation and dual-YAM software integration. Operators control recording through a single interface while completed episodes undergo parallel processing. The resulting demonstrations connect to policy training and physical deployment through a common task-space representation.

We release approximately 40 hours of demonstrations across three tasks. Physical experiments show successful orange-to-bowl execution after fine-tuning π0.5\pi_{0.5} and MolmoAct2 on mixtures containing mostly UMI-Live demonstrations. Altogether, our hardware, software, dataset, and robot evaluation compose a reproducible system spanning from handheld demonstration to physical robot learning.

Use of Generative AI

The authors used LLMs to help with coding implementation and as a writing and editing aid for this manuscript. The authors take full responsibility for its content.

Acknowledgments

This material is based upon work supported by the U.S. National Science Foundation Graduate Research Fellowship Program under Grant No. DGE-2630152. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. This work received no other third-party funding. The authors declare no competing interests.

Hardware Details

Figure 2 shows the assembled device. Figure 8 shows the components used to assemble one unit. Figure 9 shows the gripper mechanism, power module, and onboard electronics.

Components used to assemble a UMI-Live gripper

Figure 8. Components used to assemble a UMI-Live gripper, including the LiDAR, camera, Raspberry Pi, LCD, printed parts, linear guide, gears, fasteners, and battery and power electronics.

UMI-Live Manipulation Demonstrations

We demonstrate the manipulation capabilities of UMI-Live through three human-operated tasks involving small-object grasping and insertion, match lighting, and shirt folding.

Small-Object Grasping and Insertion

Figure 10 shows three manually operated sequences using UMI-Live to grasp small screwdriver bits and insert them into the tool case.

Small-object grasping and insertion sequences

Figure 10. Small-object grasping and insertion with UMI-Live. Each row begins at tool pickup and follows the handoff between the grippers, insertion into the tool case, and release. Six uncropped frames per row are selected at key moments and shown in chronological order from left to right.

Match Lighting

Figure 11 shows a human operator using both UMI-Live grippers to open a matchbox, retrieve a match, and light it.

Bimanual match lighting sequence

Figure 11. Bimanual match lighting with UMI-Live. A human operator coordinates the two grippers to open the matchbox, retrieve a match, and strike it. The final frame shows the lit match. Frames are shown in chronological order from left to right and top to bottom.

Shirt Folding

Figure 12 shows a human operator using both UMI-Live grippers to spread and fold a shirt.

Bimanual shirt folding with UMI-Live

Figure 12. Bimanual shirt folding with UMI-Live. A human operator coordinates both grippers to spread and align the shirt, fold it lengthwise, and complete the final folds. Frames are shown in chronological order from left to right and top to bottom.

Autonomous Rollout

Figure 13 shows a complete autonomous π0.5\pi_{0.5} rollout on the orange collection task.

Teleoperation Setup

For teleoperation, we replace the LiDAR on each UMI-Live gripper with a VIVE tracker6 and use Lighthouse base stations7 for external tracking (Figure 14). We align the tracking frame with the robot base axes and capture reference poses when control is engaged. Subsequent changes in tracker position and orientation are mapped to Cartesian targets for the corresponding YAM arm and converted into joint commands through IK. Jaw opening is measured from ArUco tags in the wrist-camera images and streamed from the Raspberry Pi to control the corresponding robot gripper, with opening changes scaled by the calibrated gripper span. We release the teleoperation software, setup and calibration instructions in UMI-Teleop.

Tracker-equipped gripper and teleoperation setup

Figure 14. Teleoperation setup. (a) A VIVE tracker replaces the LiDAR on each handheld gripper. (b) External base-station tracking supports teleoperation of the dual YAM arms.

Dataset Representation

Each UMI-Live demonstration is recorded as an episode with a task instruction ℓ\ell. For each gripper i∈{L,R}i \in\{L, R\}, we define its initial TCP pose as zero and express all subsequent TCP positions and orientations relative to this fixed reference. At frame tt, we store a wrist image ItiI^{i}_{t} and the state

sti=[xti,yti,zti,ϕx,ti,ϕy,ti,ϕz,ti,gti]∈R7,(1)s^{i}_{t} = [x^{i}_{t}, y^{i}_{t}, z^{i}_{t}, \phi^{i}_{x,t}, \phi^{i}_{y,t}, \phi^{i}_{z,t}, g^{i}_{t}] \in\mathbb{R}^{7}, \tag*{(1)}

where pti=(xti,yti,zti)⊤∈R3p^{i}_{t} = (x^{i}_{t}, y^{i}_{t}, z^{i}_{t})^{\top} \in\mathbb{R}^{3} is position in meters, ϕti=(ϕx,ti,ϕy,ti,ϕz,ti)⊤∈R3\phi^{i}_{t} = (\phi^{i}_{x,t}, \phi^{i}_{y,t}, \phi^{i}_{z,t})^{\top} \in\mathbb{R}^{3} is a rotation vector in radians, and gti∈[0,1]g^{i}_{t} \in[0, 1] is the normalized gripper opening, such that

gti=clip⁡(wti−wclosediwopeni−wclosedi,0,1),(2)g^{i}_{t} = \operatorname{clip}\left(\frac{w^{i}_{t} - w^{i}_{\mathrm{closed}}}{w^{i}_{\mathrm{open}} - w^{i}_{\mathrm{closed}}}, 0, 1\right), \tag*{(2)}

where wtiw^{i}_{t} are the physical jaw widths and wopen,wclosedw_{\mathrm{open}}, w_{\mathrm{closed}} are the calibrated open and closed endpoints.

The bimanual state concatenates both hands,

stbi=[stL;stR]∈R14.(3)s^{\mathrm{bi}}_{t} = [s^{L}_{t}; s^{R}_{t}] \in\mathbb{R}^{14}. \tag*{(3)}

An episode with NN frames is represented as the demonstration trajectory:

T=[(ItL,ItR,stbi,ℓ)]t=0N−1.(4)\mathcal{T} = \left[(I^{L}_{t}, I^{R}_{t}, s^{\mathrm{bi}}_{t}, \ell)\right]_{t=0}^{N-1}. \tag*{(4)}

We denote D={T(e)}e=1M\mathcal{D} = \{\mathcal{T}^{(e)}\}_{e=1}^{M} as a dataset of MM demonstration trajectories, and T=(T0,…,TN−1)\mathcal{T} = (\mathcal{T}_{0}, \ldots, \mathcal{T}_{N-1}).

Figure 15 shows the left and right wrist views and corresponding TCP position trajectories during an episode.

Wrist images and reconstructed TCP position trajectories during bimanual folding

Figure 15. Wrist images and reconstructed TCP position trajectories during bimanual folding. Each column shows the left wrist view (top), right wrist view (middle), and the two TCP position traces in a common plotting frame (bottom).

Policy Training

We fine-tune MolmoAct2 [25] and π0.5\pi_{0.5} [5] to predict bimanual TCP trajectories. We first define training on UMI demonstrations, then extend the same representation and losses to mixtures with robot teleoperation data.

Training on UMI Demonstrations

We uniformly shuffle training frames across D\mathcal{D} and retrieve the corresponding trajectory T\mathcal{T} for each frame tt. The current entry Tt\mathcal{T}_{t} provides the policy observation, while the next HH entries provide the future states used to construct supervision,

ot=(ItL,ItR,gtL,gtR,ℓ),Yt=(st+1bi,…,st+Hbi).o_t = \left(I_t^{L}, I_t^{R}, g_t^{L}, g_t^{R}, \ell\right), \qquad Y_t = \left(s_{t+1}^{\mathrm{bi}}, \ldots, s_{t+H}^{\mathrm{bi}}\right).

where HH is the action horizon. We omit TCP pose and joint information from the policy inputs. Both π0.5\pi_{0.5} and MolmoAct2 inherit visual representations from large-scale vision-language pretraining. Together with the broad workspace coverage of our ultra-wide wrist cameras, these representations provide visual context for learning bimanual coordination.

For each gripper i∈{L,R}i \in\{L, R\}, the stored state is

sti=[pti;ϕti;gti]∈R7.s_t^i = \left[p_t^i; \phi_t^i; g_t^i\right] \in\mathbb{R}^{7}.

We reconstruct the TCP pose as

Rti=exp⁡([ϕti]×)∈R3×3,Tti=[Rtipti0T1]∈SE(3),R_t^i = \exp\left([\phi_t^i]_{\times}\right) \in\mathbb{R}^{3 \times3}, \qquad T_t^i = \begin{bmatrix} R_t^i & p_t^i \\ 0^{\mathsf{T}} & 1 \end{bmatrix} \in SE(3),

where [⋅]×[\cdot]_{\times} denotes the skew-symmetric matrix associated with a vector.

We express each future pose relative to the current pose,

ΔTt,ki=(Tti)−1Tt+ki,k=1,…,H.\Delta T_{t,k}^i = \left(T_t^i\right)^{-1}T_{t+k}^i, \qquad k = 1,\ldots,H.

Its translation and rotation are

Δpt,ki=(Rti)T(pt+ki−pti),ΔRt,ki=(Rti)TRt+ki.\Delta p_{t,k}^i = \left(R_t^i\right)^{\mathsf{T}}\left(p_{t+k}^i - p_t^i\right), \qquad\Delta R_{t,k}^i = \left(R_t^i\right)^{\mathsf{T}}R_{t+k}^i.

We encode rotation using the first two rows of the rotation matrix to construct the 6D rotation vector [27],

r6(R)=[R11,R12,R13,R21,R22,R23]T.r_6(R) = \left[R_{11}, R_{12}, R_{13}, R_{21}, R_{22}, R_{23}\right]^{\mathsf{T}}.

The resulting action target for each gripper is

at,ki=[Δpt,ki;r6(ΔRt,ki);gt+ki]∈R10.a^{i}_{t,k} = \left[\Delta p^{i}_{t,k}; r_{6}(\Delta R^{i}_{t,k}); g^{i}_{t+k}\right] \in\mathbb{R}^{10}.

Concatenating the two grippers produces the bimanual action chunk

At=[[at,1L;at,1R]T⋮[at,HL;at,HR]T]∈RH×20.A_t = \begin{bmatrix} \left[a^{L}_{t,1};a^{R}_{t,1}\right]^{\mathsf{T}}\\ \vdots\\ \left[a^{L}_{t,H};a^{R}_{t,H}\right]^{\mathsf{T}} \end{bmatrix} \in\mathbb{R}^{H \times20}.

All future poses share the current frame tt as their reference, while gripper openings remain absolute targets.

We train the conditional action model using flow matching [28]. Given Gaussian noise ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0,I) and flow time τ\tau, we construct

Xt,τ=(1−τ)At+τϵ.X_{t,\tau} = (1-\tau)A_t + \tau\epsilon.

The model fθ(ot,Xt,τ,τ)f_{\theta}(o_t,X_{t,\tau},\tau) predicts a velocity field with the same dimensions as the action chunk. We minimize

Lflow=ET,t,ϵ,τ[12H∥fθ(ot,Xt,τ,τ)−(ϵ−At)∥F2].\mathcal{L}_{\mathrm{flow}} = \mathbb{E}_{T,t,\epsilon,\tau} \left[ \frac{1}{2H} \left\lVert f_{\theta}(o_t,X_{t,\tau},\tau)-(\epsilon-A_t)\right\rVert_{F}^{2} \right].

Training with Teleoperation Data

We supplement UMI demonstrations with robot teleoperation data to expose the policies to observations and motions collected on the target robot. Each teleoperation episode contains the two wrist images, measured arm joint angles qtiq^{i}_{t}, gripper openings, and a task instruction ℓ\ell. For each arm i∈{L,R}i \in\{L,R\}, we recover the TCP pose through forward kinematics,

Tti,robot=FKi(qti).T^{i,\mathrm{robot}}_{t} = \mathrm{FK}_{i}(q^{i}_{t}).

We normalize gripper openings using the calibrated closed and open endpoints and construct future pose targets relative to the current robot pose,

ΔTt,ki=(Tti,robot)−1Tt+ki,robot,k=1,…,H.\Delta T^{i}_{t,k} = \left(T^{i,\mathrm{robot}}_{t}\right)^{-1}T^{i,\mathrm{robot}}_{t+k}, \qquad k=1,\ldots,H.

The translation, six-dimensional rotation encoding, and future gripper opening are assembled into AtA_t using the same construction as for UMI demonstrations. Both sources therefore provide the same observation format oto_t and action representation AtA_t. Joint angles are used to reconstruct robot trajectories and are omitted from the policy inputs. Let Dteleop\mathcal{D}_{\mathrm{teleop}} denote the teleoperation trajectories. We fine-tune on the combined dataset

Dmix=D∪Dteleop.\mathcal{D}_{\mathrm{mix}} = \mathcal{D} \cup\mathcal{D}_{\mathrm{teleop}}.

using the training objective defined above. The UMI and robot teleoperation shares sum to 100% of the fine-tuning mixture. § 4 reports the percentage splits used for each policy.

Training Setup

We initialize π0.5\pi_{0.5} from the released base model and MolmoAct2 from MolmoAct2-BimanualYAM. Both models are fine-tuned for 12,000 optimizer updates using AdamW [29], 600 warmup updates, and cosine learning-rate decay to 0.1 of each parameter group’s peak rate. We use an action horizon of H=24H=24 at 30 Hz, gradient clipping at global norm 1. We update all π0.5\pi_{0.5} parameters. For MolmoAct2, we train LoRA adapters in the vision-language backbone and the full action expert, and we keep the remaining backbone weights and input embeddings fixed. MolmoAct2 combines flow matching with discrete action-token supervision. Table 3 and Table 4 summarize the settings for these fresh-base runs.

Policy Inference

At each policy query tt, the robot provides the observation

ot=(ItL,ItR,gtL,gtR,ℓ).o_t = (I_t^L, I_t^R, g_t^L, g_t^R, \ell).

Conditioned on oto_t, we generate a bimanual action chunk by integrating the learned flow field. We express both policies using the flow-time convention defined during training, with Gaussian noise at τ=1\tau= 1 and actions at τ=0\tau= 0. We initialize Xt,1=ϵX_{t,1} = \epsilon, where ϵ∼N(0,I)\epsilon\sim\mathcal{N}(0, I) has shape H×20H \times20, and solve

dXt,τdτ=fθ(ot,Xt,τ,τ)\frac{\mathrm{d}X_{t,\tau}}{\mathrm{d}\tau} = f_{\theta}(o_t, X_{t,\tau}, \tau)

from τ=1\tau= 1 to τ=0\tau= 0. Here, τ\tau denotes flow time, distinct from the robot timestep tt. The observation remains fixed while the entire action chunk is refined. Using KK Euler steps with τj=1−j/K\tau_j = 1 - j/K, we compute

Xt,τj+1=Xt,τj−1Kfθ(ot,Xt,τj,τj),j=0,…,K−1.X_{t,\tau_{j+1}} = X_{t,\tau_j} - \frac{1}{K} f_{\theta}(o_t, X_{t,\tau_j}, \tau_j), \qquad j = 0, \ldots, K - 1.

The final estimate gives the predicted action chunk

A^t=Xt,0∈RH×20.\widehat{A}_t = X_{t,0} \in\mathbb{R}^{H \times20}.

Each predicted action contains a relative translation, a six-dimensional rotation representation, and an absolute gripper opening for each arm. We recover the relative rotation through Gram-Schmidt orthonormalization and combine it with the predicted translation to form ΔT^t,ki\widehat{\Delta T}_{t,k}^i. At the query timestep, the robot computer records the measured joint angles qtiq_t^i and computes the TCP pose

Tti,robot=FK⁡i(qti).T_t^{i,\mathrm{robot}} = \operatorname{FK}_i(q_t^i).

These joint measurements are used by the execution controller and are omitted from the policy inputs. Each relative prediction is mapped to a robot target pose,

T^t,ki,target=Tti,robotΔT^t,ki,k=1,…,H.\widehat{T}_{t,k}^{i,\mathrm{target}} = T_t^{i,\mathrm{robot}}\widehat{\Delta T}_{t,k}^i, \qquad k = 1, \ldots, H.

Inverse kinematics converts this target into joint commands,

q^t,ki=IK⁡i(T^t,ki,target;qti),\hat{\mathbf{q}}^{i}_{t,k} = \operatorname{IK}_{i}\left(\hat{\mathbf{T}}^{i,\mathrm{target}}_{t,k}; \mathbf{q}^{i}_{t}\right),

using the measured query joints as the initial configuration. Predicted gripper openings are mapped to calibrated robot gripper commands. All targets within a chunk share the pose measured at its query. The robot executes actions while the server processes subsequent observations. Returned predictions are aligned with the robot’s execution timeline, and elapsed predictions are discarded. Each new query supplies updated observations and a fresh pose reference.

Dual-YAM Hardware Integration

We extend LeRobot’s existing client–server inference architecture to support our dual-YAM setup. We run policy inference on a remote server and execute the returned action chunks through a local robot client. We run each arm in a separate worker process that reads joint and gripper feedback and dispatches commands through I2RT. We assign left and right commands a common scheduled execution time to coordinate bimanual motion. We use I2RT for forward and inverse kinematics, and we verify each inverse-kinematics solution against pose-error tolerances, joint limits, and a bound on joint displacement. We map predicted gripper openings to hardware commands using saved gripper calibrations. We also provide an action-probe mode that logs predicted commands without dispatching policy actions.

Kinematic Model

We retain the six-joint kinematic structure and arm geometry of the default YAM configuration. We adapt the model to our replacement gripper by updating the TCP position and orientation through a fixed flange-to-TCP transform.

We denote a transformation from frame aa to frame bb by TabT^{b}_{a}. We describe the arm using the classical Denavit–Hartenberg convention

Tjj−1(qj)=Rz(qj+θjoffset)Tz(dj)Tx(aj)Rx(αj),T^{j-1}_{j}(q_j) = R_z(q_j + \theta^{\mathrm{offset}}_j) T_z(d_j) T_x(a_j) R_x(\alpha_j),

where qjq_j is the angle of joint jj. We use the parameters in Table 5 from the I2RT YAM model8, with distances in meters and angles in radians.

Joint jjθjoffset\theta^{\mathrm{offset}}_jdjd_jaja_jαj\alpha_j
100.11350.02−π/2-\pi/2
2−π-\pi00.264−π-\pi
3−2.901418-2.90141800.2522390
4−0.240173-0.24017300.073999π/2\pi/2
5π/2\pi/200π/2\pi/2
6π/2\pi/20.03560−π-\pi

Table 5. Classical Denavit–Hartenberg parameters for the YAM arm.

We denote the base, flange, and TCP frames by BB, FF, and GG, with joint frames 0=B0 = B and 6=F6 = F. We place the TCP frame origin at the center of the gripper tips, 220 mm along the negative zz-axis of the flange frame, and orient its axes to match the UMI convention. We use the fixed transform

TGF=[−1000010000−1−0.2200001].T^{F}_{G} = \begin{bmatrix} -1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & -1 & -0.220 \\ 0 & 0 & 0 & 1 \end{bmatrix}.

For either arm, we obtain the TCP pose as

TGB(q)=(∏j=16Tjj−1(qj))TGE.T_{G}^{B}(\mathbf{q}) = \left(\prod_{j=1}^{6} T_{j}^{j-1}(q_{j})\right) T_{G}^{E}.

We incorporate this TCP transform directly into the I2RT robot model and use its MJCF-based implementation for forward and inverse kinematics during execution.

References

  1. [1]C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song. Diffusion policy: Visuomotor policy learning via action diffusion. In Robotics: Science and Systems (RSS), 2023. doi:10.15607/RSS.2023.XIX.026.DOI
  2. [2]A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, P. Florence, C. Fu, M. G. Arenas, K. Gopalakrishnan, K. Han, K. Hausman, A. Herzog, J. Hsu, B. Ichter, A. Irpan, N. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, L. Leal, L. Lee, T.-W. E. Lee, S. Levine, Y. Lu, H. Michalewski, I. Mordatch, K. Pertsch, K. Rao, K. Reymann, M. Ryoo, G. Salazar, P. Sanketi, P. Sermanet, J. Singh, A. Singh, R. Soricut, H. Tran, V. Vanhoucke, Q. Vuong, A. Wahid, S. Welker, P. Wohlhart, J. Wu, F. Xia, T. Xiao, P. Xu, S. Xu, T. Yu, and B. Zitkovich. RT-2: Vision-language-action models transfer web knowledge to robotic control, 2023. URL https://arxiv.org/abs/2307.15818.
  3. [3]M. J. Kim, K. Pertsch, S. Karamcheti, T. Xiao, A. Balakrishna, S. Nair, R. Rafailov, E. Foster, G. Lam, P. Sanketi, Q. Vuong, T. Kohler, B. Burchfiel, R. Tedrake, D. Sadigh, S. Levine, P. Liang, and C. Finn. OpenVLA: An open-source vision-language-action model, 2024. URL https://arxiv.org/abs/2406.09246.
  4. [4]K. Black, N. Brown, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, S. Jakubczak, T. Jones, L. Ke, S. Levine, A. Li-Bell, M. Mothukuri, S. Nair, K. Pertsch, L. X. Shi, L. Smith, J. Tanner, Q. Vuong, A. Walling, H. Wang, and U. Zhilinsky. π₀: A vision-language-action flow model for general robot control. In Robotics: Science and Systems (RSS), 2025. doi:10.15607/RSS.2025.XXI.010.DOI
  5. [5]K. Black, N. Brown, J. Darpinian, K. Dhabalia, D. Driess, A. Esmail, M. R. Equi, C. Finn, N. Fusai, M. Y. Galliker, et al. π₀.₅: A vision-language-action model with open-world generalization. In Conference on Robot Learning (CoRL), 2025. URL https://proceedings.mlr.press/v305/black25a.html.arxiv.org/abs/2504.16054
  6. [6]NVIDIA, J. Bjorck, F. Castañeda, N. Cherniadev, X. Da, R. Ding, L. J. Fan, Y. Fang, D. Fox, F. Hu, S. Huang, J. Jang, Z. Jiang, J. Kautz, K. Kundalia, L. Lao, Z. Li, Z. Lin, K. Lin, G. Liu, E. Llontop, L. Magne, A. Mandlekar, A. Narayan, S. Nasiriany, S. Reed, Y. L. Tan, G. Wang, Z. Wang, J. Wang, Q. Wang, X. Xiang, Y. Xie, Y. Xu, Z. Xu, S. Ye, Z. Yu, A. Zhang, H. Zhang, Y. Zhao, R. Zheng, and Y. Zhu. GROOT N1: An open foundation model for generalist humanoid robots, 2025. URL https://arxiv.org/abs/2503.14734.
  7. [7]C. Zhu, R. Yu, S. Feng, B. Burchfiel, P. Shah, and A. Gupta. Unified world models: Coupling video and action diffusion for pretraining on large robotic datasets. In Robotics: Science and Systems (RSS), 2025. doi:10.15607/RSS.2025.XXI.015.DOI
  8. [8]M. J. Kim, Y. Gao, T.-Y. Lin, Y.-C. Lin, Y. Ge, G. Lam, P. Liang, S. Song, M.-Y. Liu, C. Finn, and J. Gu. Cosmos policy: Fine-tuning video models for visuomotor control and planning, 2026. URL https://arxiv.org/abs/2601.16163.
  9. [9]S. Ye, Y. Ge, K. Zheng, S. Gao, S. Yu, G. Kurian, S. Indupuru, Y. L. Tan, C. Zhu, J. Xiang, A. Malik, K. Lee, W. Liang, N. Ranawaka, J. Gu, Y. Xu, G. Wang, F. Hu, A. Narayan, J. Bjorck, J. Wang, G. Kim, D. Niu, R. Zheng, Y. Xie, J. Wu, Q. Wang, R. Julian, D. Xu, Y. Du, Y. Chebotar, S. Reed, J. Kautz, Y. Zhu, L. J. Fan, and J. Jang. World action models are zero-shot policies, 2026. URL https://arxiv.org/abs/2602.15922.
  10. [10]A. Khazatsky, K. Pertsch, S. Nair, A. Balakrishna, S. Dasari, S. Karamcheti, S. Nasiriany, M. K. Srirama, L. Y. Chen, K. Ellis, P. D. Fagan, J. Hejna, M. Itkina, M. Lepert, Y. J. Ma, P. T. Miller, J. Wu, S. Belkhale, S. Dass, H. Ha, A. Jain, A. Lee, Y. Lee, M. Memmel, S. Park, I. Radosavovic, K. Wang, A. Zhan, K. Black, C. Chi, K. B. Hatch, S. Lin, J. Lu, J. Mercat, A. Rehman, P. R. Sanketi, A. Sharma, C. Simpson, Q. Vuong, H. R. Walke, B. Wulfe, T. Xiao, J. H. Yang, A. Yavary, T. Z. Zhao, C. Agia, R. Baijal, M. G. Castro, D. Chen, Q. Chen, T. Chung, J. Drake, E. P. Foster, J. Gao, V. Guizilini, D. A. Herrera, M. Heo, K. Hsu, J. Hu, M. Z. Irshad, D. Jackson, C. Le, Y. Li, K. Lin, R. Lin, Z. Ma, A. Maddukuri, S. Mirchandani, D. Morton, T. Nguyen, A. O’Neill, R. Scalise, D. Seale, V. Son, S. Tian, E. Tran, A. E. Wang, Y. Wu, A. Xie, J. Yang, P. Yin, Y. Zhang, O. Bastani, G. Berseth, J. Bohg, K. Goldberg, A. Gupta, A. Gupta, D. Jayaraman, J. J. Lim, J. Malik, R. Martín-Martín, S. Ramamoorthy, D. Sadigh, S. Song, J. Wu, M. C. Yip, Y. Zhu, T. Kollar, S. Levine, and C. Finn. DROID: A large-scale in-the-wild robot manipulation dataset, 2025. URL https://arxiv.org/abs/2403.12945.
  11. [11]A. Allshire, H. G. Singh, R. Singh, A. Rashid, H. Choi, D. McAllister, J. Yu, Y. Chen, H. Huang, P. Abbeel, X. Chen, R. Duan, P. Isola, J. Malik, F. Shentu, G. Shi, P. Wu, and A. Kanazawa. Scalable behavior cloning with open data, training, and evaluation, 2026. URL https://arxiv.org/abs/2606.27375.
  12. [12]C. Chi, Z. Xu, C. Pan, E. Cousineau, B. Burchfiel, S. Feng, R. Tedrake, and S. Song. Universal manipulation interface: In-the-wild robot teaching without in-the-wild robots. In Robotics: Science and Systems (RSS), 2024. doi:10.15607/RSS.2024.XX.045.DOI
  13. [13]T. Ohkawa, J. Arima, Y. Noguchi, M. Tateno, M. Sugiura, T. Okubo, K. Ikeuchi, Y. Shin, H. Nishizawa, N. Kanazawa, Y. Wakayama, D. Fukunaga, K. Makihara, T. Motoda, F. Erich, Y. Domae, T. Matsushima, Y. Okumatsu, and K. Ota. YUBI: Yielding universal bimanual interface for bimanual dexterous manipulation at scale, 2026. URL https://arxiv.org/abs/2606.10244.
  14. [14]Q. Zeng, C. Li, J. St. John, Z. Zhou, J. Wen, G. Feng, Y. Zhu, and Y. Xu. ActiveUMI: Robotic manipulation with active perception from robot-free human demonstrations, 2025. URL https://arxiv.org/abs/2510.01607.
  15. [15]H. R. Walke, K. Black, T. Z. Zhao, Q. Vuong, C. Zheng, P. Hansen-Estruch, A. W. He, V. Myers, M. J. Kim, M. Du, A. Lee, K. Fang, C. Finn, and S. Levine. BridgeData V2: A dataset for robot learning at scale. In Conference on Robot Learning (CoRL), 2023. URL https://proceedings.mlr.press/v229/walke23a.html.arxiv.org/abs/2308.12952
  16. [16]K. Wu, C. Hou, J. Liu, Z. Che, X. Ju, Z. Yang, M. Li, Y. Zhao, Z. Xu, G. Yang, S. Fan, X. Wang, F. Liao, Z. Zhao, G. Li, Z. Jin, L. Wang, J. Mao, N. Liu, P. Ren, Q. Zhang, Y. Lyu, M. Liu, J. He, Y. Luo, Z. Gao, C. Li, C. Gu, Y. Fu, D. Wu, X. Wang, S. Chen, Z. Wang, P. An, S. Qian, S. Zhang, and J. Tang. RoboMIND: Benchmark on multi-embodiment intelligence normative data for robot manipulation. In Robotics: Science and Systems (RSS), 2025. doi:10.15607/RSS.2025.XXI.152.DOI
  17. [17]T. Z. Zhao, J. Tompson, D. Driess, P. Florence, S. K. S. Ghasemipour, C. Finn, and A. Wahid. ALOHA unleashed: A simple recipe for robot dexterity. In Conference on Robot Learning (CoRL), 2024. URL https://proceedings.mlr.press/v270/zhao25b.html.arxiv.org/abs/2410.13126
  18. [18]Zhaxizhuoma, K. Liu, C. Guan, Z. Jia, Z. Wu, X. Liu, T. Wang, S. Liang, P. Chen, P. Zhang, H. Song, D. Qu, D. Wang, Z. Wang, N. Cao, Y. Ding, B. Zhao, and X. Li. FastUMI: A scalable and hardware-independent universal manipulation interface with dataset. In Conference on Robot Learning (CoRL), 2025. URL https://proceedings.mlr.press/v305/zhaxizhuoma25a.html.arxiv.org/abs/2409.19499
  19. [19]Z. Wang. UMI-3D: Extending universal manipulation interface from vision-limited to 3D spatial perception, 2026. URL https://arxiv.org/abs/2604.14089.
  20. [20]Simple AI, Y. Wei, J. Ma, J. Wang, W. Zhou, Y. Zuo, K. Rui, M. Li, J. Zhang, Z. Pan, X. Wang, H. Jia, H. Du, Z. Zeng, J. Ma, G. Qin, D. Zhang, and X. Li. HiFi-UMI: Learning deployable manipulation policies from high-fidelity UMI data alone, 2026. URL https://arxiv.org/abs/2607.25895.
  21. [21]R. Cadene, S. Alibert, F. Capuano, M. Aractingi, A. Zouitine, P. Kooijmans, J. Choghari, M. Russi, C. Pascal, S. Palma, D. Aubakirova, M. Shukor, J. Moss, A. Soare, Q. Lhoest, Q. Gallouédec, and T. Wolf. LeRobot: An open-source library for end-to-end robot learning. In International Conference on Learning Representations (ICLR), 2026. URL https://arxiv.org/abs/2602.22818.
  22. [22]J. Fang, W. Chen, H. Xue, F. Zhou, T. Le, Y. Wang, Y. Zhang, J. Lv, C. Wen, and C. Lu. RoboPocket: Improve robot policies instantly with your phone, 2026. URL https://arxiv.org/abs/2603.05504.
  23. [23]S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez. Automatic generation and detection of highly reliable fiducial markers under occlusion. Pattern Recognition, 47(6):2280–2292, 2014. doi:10.1016/j.patcog.2014.01.005.
  24. [24]K. Koide, M. Yokozuka, S. Oishi, and A. Banno. GLIM: 3D range-inertial localization and mapping with GPU-accelerated scan matching factors. Robotics and Autonomous Systems, 179:104750, 2024. doi:10.1016/j.robot.2024.104750.DOI
  25. [25]H. Fang, J. Duan, D. Clay, S. Wang, S. Liu, W. Huang, X. Fan, W.-C. Tsai, S. Chen, Y. R. Wang, et al. MolmoAct2: Action reasoning models for real-world deployment, 2026. URL https://arxiv.org/abs/2605.02881.
  26. [26]E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL https://arxiv.org/abs/2106.09685.
  27. [27]Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity of rotation representations in neural networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. doi:10.1109/CVPR.2019.000589.arxiv.org/abs/1812.07035
  28. [28]Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le. Flow matching for generative modeling. In International Conference on Learning Representations (ICLR), 2023. URL https://arxiv.org/abs/2210.02747.
  29. [29]I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. URL https://arxiv.org/abs/1711.05101.

Paper details

vlnc:W_tnbx8b6n
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

LLMs

View statement
Looking for related papers…
Contents