betaFeedback

WorksComputer Science

ReDOA: Real-World Distillation via Observation Alignment for Image-Based Manipulation Policy Learning

Abstract

Data-driven learning for general-purpose robotic manipulation presents a fundamental trade-off between data-collection cost and adaptability to real-world environments. Imitation learning (IL) requires costly human demonstrations, whereas simulation-based learning is vulnerable to sim-to-real gaps in dynamics and sensor observations. We propose ReDOA (Real-World Distillation via Observation Alignment), a framework that decouples control learning from perception learning to learn image-based policies in the real world. A state-based teacher policy trained in simulation is executed on a real robot, and its control knowledge is distilled into an image-based student policy using the collected trajectories. Observation Alignment resolves observation mismatches between data collection and deployment. This framework collects successful, efficiently executed trajectories paired with real-world visual observations, without requiring demonstrations from human experts. In real-robot experiments on three non-prehensile manipulation tasks, including bimanual manipulation, our method achieves better overall performance when jointly considering task success and execution efficiency than both IL based on human demonstrations and policy distillation in simulation. https://isri-aist.github.io/redoa-project-page/

Introduction

Although demand for automating manipulation tasks in unstructured environments, such as homes and industrial workplaces, continues to grow, successful real-world deployments remain limited. Data-driven learning has therefore attracted considerable attention as a promising solution. However, learning both robust control under complex dynamics and reliable perception across diverse real-world environments remains challenging while keeping the costs of collecting demonstrations and setting up environments low.

Existing data-driven approaches face an inherent difficulty in balancing data collection cost and real-world adaptability. Imitation learning (IL) based on human demonstrations can naturally adapt to real-world visual observations and contact dynamics, but teaching complex dual-arm manipulation through teleoperation requires operator skill and teleoperation interface setup. Moreover, because vision-based IL is sensitive even to minor appearance variations such as lighting or background shifts, adapting to new environments requires collecting demonstrations again, repeatedly imposing substantial labor and time on human demonstrators. In contrast, simulation-based methods (sim-to-real approaches)

enable training on large datasets at low cost, but gaps in dynamics and sensor observations tend to degrade real-world performance.

However, recent work has shown that separating control learning in simulation from perception learning in the real world can bypass the sim-to-real gap in observations. For example, PTLD [1] uses real-world policy distillation to bypass the sim-to-real gap of tactile sensors.

We propose ReDOA (Real-World Distillation via Observation Alignment), which applies distillation across observation spaces in the real world to image-based manipulation policy learning. Figure 1 illustrates the concept of our method. The pipeline proceeds as follows. First, we roll out the teacher policy while estimating the object state using temporarily installed trackers, collecting demonstrations for distillation. The trackers appear in the collected images; using these images directly to train the student policy creates an observation mismatch when the trackers are removed at deployment. We therefore perform Observation Alignment to resolve this mismatch and train the student policy using the aligned observations and collected actions. These stages form a sim-to-real-to-real pipeline.

Conceptual overview of ReDOA (Real-World Distillation via Observation Alignment)

Figure 1. Conceptual overview of ReDOA (Real-World Distillation via Observation Alignment).

The framework reduces the burden of human demonstration and bypasses the visual sim-to-real gap by learning perception from images collected in the real world.

We evaluate the proposed method on three non-prehensile manipulation tasks using a real robot. Non-prehensile manipulation requires continuous visual perception to maintain precise robot–object–environment contact, making it a suitable benchmark.

Prior sim-to-real studies have attempted to limit the performance degradation of vision-based policies by using visual domain randomization [2], green-screening, or texture projection [3]. However, these methods suffer from training instability caused by randomization [4] and require substantial effort to set up the environment. Our method uses images collected in the real environment to train the student policy, eliminating the need to render its training images in simulation. Observation Alignment then mitigates the appearance mismatch caused by trackers introduced for real-world data collection.

Unlike conventional IL based on human demonstrations, our method collects data using a consistent policy without relying on human experts or being affected by human fatigue or individual operating habits. Because the state-based teacher does not depend on visual appearance, it can also autonomously collect new real-world demonstrations without retraining when lighting or backgrounds change.

In summary, our contributions are as follows:

  • A real-world distillation framework for bypassing the visual sim-to-real gap through Observation Alignment that resolves observation mismatches between demonstration collection and deployment.

  • Automated generation of high-quality demonstration data without expert teleoperation, reducing human labor costs.

  • Real-world validation on three non-prehensile tasks, demonstrating the simultaneous achievement of high success rates and high motion efficiency.

Related Work

Challenges in IL and Demonstration Data Collection

Recent advances in IL, including ACT [5] and Diffusion Policy [6], have led to significant success in approaches that generate robot actions directly from visual observations. However, the performance of these data-driven methods depends strongly on the quality and quantity of demonstration data collected from human experts [7, 8].

In particular, the non-prehensile manipulation tasks considered in this study require controlling continuous contact dynamics among the object, the environment, and the robot [9]. Prior work has proposed RL-based approaches focused primarily on control, including maintaining contact modes during whole-arm manipulation [10], integrating tactile sensors into the end effectors of a dual-arm robot [11], and combining optimization-based demonstration generation with RL [12]. These studies show that non-prehensile manipulation requires continuous state observation and sequential action correction to handle delicate physical interactions. Consequently, collecting a large-scale, high-quality, and consistent real-world dataset for IL through teleoperation is costly.

To address the cost of collecting human demonstrations, RLDG uses reinforcement learning directly on real robots to collect demonstrations containing real-world RGB images and fine-tunes a generalist model on these data [13]. By autonomously generating high-quality trajectories through reward maximization, this approach has been shown to outperform fine-tuning on human demonstrations.

In contrast, this study focuses on training task-specific image-based policies from scratch.

Policy Distillation across Observation Spaces

Deploying a state-based policy in the real world requires continuously providing the policy with task-relevant state information, such as object poses. Such state information can be obtained using planar fiducial markers attached to objects [14] or marker-based motion capture [15]. However, in settings where many objects of the same type are handled in sequence, attaching markers to each object in advance is impractical.

Image-based policies avoid explicit object-state measurement [16, 17]. However, end-to-end policy learning from high-dimensional visual observations requires learning of visual representations and control simultaneously, which tends to reduce sample efficiency and training stability. In addition, repeatedly rendering visual observations in simulation also increases computational resource requirements and training time.

Policy distillation across observation spaces is used to address these sample-efficiency and computational-cost issues. A teacher policy is first trained in simulation using privileged observations, and its control knowledge is then distilled, also in simulation, into a student policy that uses observations available at deployment, such as RGB images [18, 19, 20], depth images [21], or point clouds [22, 23].

PTLD [1] also performs policy distillation in the real world. It bypasses the sim-to-real gap of tactile sensors by using real-world data to learn to estimate latent representations based on privileged information from tactile and proprioceptive observations.

Our method likewise performs distillation in the real world, using Observation Alignment to resolve the image-observation mismatch caused by the presence of trackers during data collection and their absence at deployment, thereby enabling real-world distillation into an image-based policy.

Visual Sim-to-Real Gap

In sim-to-real transfer, the visual gap associated with RGB images is a serious problem and degrades real-robot deployment performance [24, 25]. Real-to-sim-to-real methods such as Re3Sim\mathrm{Re}^{3}\mathrm{Sim} [26] reduce the visual sim-to-real gap by reconstructing real-world 3D geometry and appearance in photorealistic simulation environments. Whereas their training data are generated entirely in simulation, our method executes the teacher policy on a real robot to generate demonstrations that directly reflect real-world visual observations and contact dynamics. Moreover, Re3Sim\mathrm{Re}^{3}\mathrm{Sim} considers only rigid objects, and its applicability to articulated-object manipulation, as studied here, remains unverified.

Overview of the ReDOA learning pipeline

Figure 2. Overview of the ReDOA (Real-World Distillation via Observation Alignment) learning pipeline.

By training the student policy using real-world images, our method eliminates the need to transfer from simulated images to real images. Observation Alignment further mitigates the observation mismatch caused by tracker artifacts in the real data.

Method

We propose a sim-to-real-to-real pipeline in which a state-based teacher policy trained in simulation is executed on a real robot, and the resulting demonstrations are used to train an image-based student policy.

The underlying problem is formulated as a partially observable Markov decision process (POMDP), denoted by (S,O,A,T,R,γ)(\mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma). Here, S\mathcal{S} denotes the state space, O\mathcal{O} the observation space, A\mathcal{A} the action space, T(st+1∣st,at)\mathcal{T}(s_{t+1} \mid s_t, a_t) the state transition function, R(st,at)\mathcal{R}(s_t, a_t) the reward function, and γ∈[0,1)\gamma\in[0, 1) the discount factor.

The proposed learning pipeline is illustrated in Fig. 2. The procedure is described in the following four stages.

Phase 1: State-based RL in Simulation

In the first phase, the teacher policy πteacher\pi_{\mathrm{teacher}} is trained in simulation using state-based RL. Here, the observation space of πteacher\pi_{\mathrm{teacher}} is denoted by Ostate\mathcal{O}_{\mathrm{state}}. In addition to robot proprioception, Ostate\mathcal{O}_{\mathrm{state}} includes pose information about the object.

The objective of the teacher policy is to find an optimal policy πteacher∗\pi_{\mathrm{teacher}}^* that maximizes the expected discounted return under the transition function Tsim\mathcal{T}_{\mathrm{sim}} of the simulation environment:

πteacher∗=arg⁡max⁡π Eat∼π[∑t=0TγtR(st,at)](1)\pi_{\mathrm{teacher}}^* = \underset{\pi}{\arg\max}\ \mathbb{E}_{a_t \sim\pi}\left[\sum_{t=0}^{T} \gamma^t \mathcal{R}(s_t, a_t)\right] \tag*{(1)}

Phase 2: Real-world Data Generation with Object Tracking

In the second phase, the teacher policy πteacher\pi_{\mathrm{teacher}}, trained in simulation, is executed on the real robot to autonomously collect demonstration data for training the student policy.

During real-world rollouts, trackers whose poses can be easily estimated are placed on the object and tracked online. The estimation function ϕ\phi reconstructs the state from the image observation oimgrawo_{\mathrm{img}}^{\mathrm{raw}}, in which the trackers are visible, as follows:

ostate=ϕ(oimgraw)(2)o_{\mathrm{state}} = \phi(o_{\mathrm{img}}^{\mathrm{raw}}) \tag*{(2)}

Throughout this paper, oimgo_{\mathrm{img}} denotes an observation comprising an RGB image and robot proprioception, such as joint angles.

Actions aa are generated from this reliable state observation ostate∈Ostateo_{\mathrm{state}} \in\mathcal{O}_{\mathrm{state}}, and demonstration data are collected accordingly. Formally, we write

a∼πteacher(⋅∣ostate)(3)a \sim\pi_{\mathrm{teacher}}(\cdot\mid o_{\mathrm{state}}) \tag*{(3)}
Draw:={(oimgraw,a)}.(4)\mathcal{D}_{\mathrm{raw}} := \{(o_{\mathrm{img}}^{\mathrm{raw}}, a)\}. \tag*{(4)}

Phase 3: Observation Alignment for Distillation

In the third phase, we perform tracker artifact removal on the observations in Draw\mathcal{D}_{\mathrm{raw}} to construct a deployment-aligned demonstration dataset for offline IL.

The raw observations oimgrawo_{\mathrm{img}}^{\mathrm{raw}} contain tracker artifacts that are absent at deployment.

We define Observation Alignment as a transformation ff that maps data-collection images to the deployment observation domain. In this study, we implement this transformation through image compositing, which fills the regions occluded by the trackers. The resulting deployment-aligned image is given by

oimgalign=f(oimgraw)o_{\mathrm{img}}^{\mathrm{align}} = f(o_{\mathrm{img}}^{\mathrm{raw}})

We use planar fiducial markers as trackers. We chose them because they provide stable estimates of the object’s six-degree-of-freedom pose. In addition, a marker board with multiple markers printed at equal intervals allows pose estimation to continue from the remaining markers even when some are occluded by the robot.

The overall workflow is illustrated in Figure 3. The specific procedure is as follows.

Workflow for tracker artifact removal from raw demonstration images

Figure 3. Workflow for tracker artifact removal from raw demonstration images. We used planar fiducial markers as trackers for this whole study.

  1. The marker region is detected in the image using an existing marker detector.

  2. The target object is segmented using Segment Anything Model 2 (SAM 2) [27] to distinguish its pixels within the detected marker region from pixels belonging to the robot or other objects. In this process, the

@page 3/9 starts_mid=0 ends_mid=1

Overview of the ReDOA learning pipeline

Figure 2 (PDF p. 3). Overview of the ReDOA (Real-World Distillation via Observation Alignment) learning pipeline.

By training the student policy using real-world images, our method eliminates the need to transfer from simulated images to real images. Observation Alignment further mitigates the observation mismatch caused by tracker artifacts in the real data.

Method

We propose a sim-to-real-to-real pipeline in which a state-based teacher policy trained in simulation is executed on a real robot, and the resulting demonstrations are used to train an image-based student policy.

The underlying problem is formulated as a partially observable Markov decision process (POMDP), denoted by (S,O,A,T,R,γ)(\mathcal{S}, \mathcal{O}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma). Here, S\mathcal{S} denotes the state space, O\mathcal{O} the observation space, A\mathcal{A} the action space, T(st+1∣st,at)\mathcal{T}(s_{t+1} \mid s_t, a_t) the state transition function, R(st,at)\mathcal{R}(s_t, a_t) the reward function, and γ∈[0,1)\gamma\in[0, 1) the discount factor.

The proposed learning pipeline is illustrated in Fig. 2. The procedure is described in the following four stages.

Phase 1: State-based RL in Simulation

In the first phase, the teacher policy πteacher\pi_{\mathrm{teacher}} is trained in simulation using state-based RL. Here, the observation space of πteacher\pi_{\mathrm{teacher}} is denoted by Ostate\mathcal{O}_{\mathrm{state}}. In addition to robot proprioception, Ostate\mathcal{O}_{\mathrm{state}} includes pose information about the object.

The objective of the teacher policy is to find an optimal policy πteacher∗\pi_{\mathrm{teacher}}^* that maximizes the expected discounted return under the transition function Tsim\mathcal{T}_{\mathrm{sim}} of the simulation environment:

πteacher∗=arg⁡max⁡π Eat∼π[∑t=0TγtR(st,at)](5)\pi_{\mathrm{teacher}}^* = \underset{\pi}{\arg\max}\ \mathbb{E}_{a_t \sim\pi}\left[\sum_{t=0}^{T} \gamma^t \mathcal{R}(s_t, a_t)\right] \tag*{(5)}

Phase 2: Real-world Data Generation with Object Tracking

In the second phase, the teacher policy πteacher\pi_{\mathrm{teacher}}, trained in simulation, is executed on the real robot to autonomously collect demonstration data for training the student policy.

During real-world rollouts, trackers whose poses can be easily estimated are placed on the object and tracked online. The estimation function ϕ\phi reconstructs the state from the image observation oimgrawo_{\mathrm{img}}^{\mathrm{raw}}, in which the trackers are visible, as follows:

ostate=ϕ(oimgraw)(6)o_{\mathrm{state}} = \phi(o_{\mathrm{img}}^{\mathrm{raw}}) \tag*{(6)}

Throughout this paper, oimgo_{\mathrm{img}} denotes an observation comprising an RGB image and robot proprioception, such as joint angles.

Actions aa are generated from this reliable state observation ostate∈Ostateo_{\mathrm{state}} \in\mathcal{O}_{\mathrm{state}}, and demonstration data are collected accordingly. Formally, we write

a∼πteacher(⋅∣ostate)(7)a \sim\pi_{\mathrm{teacher}}(\cdot\mid o_{\mathrm{state}}) \tag*{(7)}
Draw:={(oimgraw,a)}.(8)\mathcal{D}_{\mathrm{raw}} := \{(o_{\mathrm{img}}^{\mathrm{raw}}, a)\}. \tag*{(8)}

Phase 3: Observation Alignment for Distillation

In the third phase, we perform tracker artifact removal on the observations in Draw\mathcal{D}_{\mathrm{raw}} to construct a deployment-aligned demonstration dataset for offline IL.

The raw observations oimgrawo_{\mathrm{img}}^{\mathrm{raw}} contain tracker artifacts that are absent at deployment.

We define Observation Alignment as a transformation ff that maps data-collection images to the deployment observation domain. In this study, we implement this transformation through image compositing, which fills the regions occluded by the trackers. The resulting deployment-aligned image is given by

oimgalign=f(oimgraw)o_{\mathrm{img}}^{\mathrm{align}} = f(o_{\mathrm{img}}^{\mathrm{raw}})

We use planar fiducial markers as trackers. We chose them because they provide stable estimates of the object’s six-degree-of-freedom pose. In addition, a marker board with multiple markers printed at equal intervals allows pose estimation to continue from the remaining markers even when some are occluded by the robot.

The overall workflow is illustrated in Figure 3. The specific procedure is as follows.

  1. The marker region is detected in the image using an existing marker detector.

  2. The target object is segmented using Segment Anything Model 2 (SAM 2) [27] to distinguish its pixels within the detected marker region from pixels belonging to the robot or other objects. In this process, the

detected marker region is used to provide SAM 2 with a positive prompt for the target object.

  1. The intersection between the marker region and the segmented object region is computed, and this region is identified as the mask.

  1. The identified masked region is inpainted using the original color values of the object.

This marker-removal process makes it possible to inpaint only the marker regions while preserving image regions containing the robot, even when some markers are occluded by the robot or other objects. Applying the tracker artifact removal process ff to the raw dataset Draw\mathcal{D}_{\mathrm{raw}} yields the deployment-aligned demonstration dataset

Dalign={(f(oimgraw),a)}={(oimgalign,a)}.\mathcal{D}_{\mathrm{align}} = \{(f(o_{\mathrm{img}}^{\mathrm{raw}}), a)\} = \{(o_{\mathrm{img}}^{\mathrm{align}}, a)\}.

Offline IL

Offline IL is performed using the deployment-aligned demonstration dataset Dalign\mathcal{D}_{\mathrm{align}}. The objective of the student policy πstudent\pi_{\mathrm{student}} is to approximate the expert action distribution, and the training objective is defined as

πstudent∗=arg⁡min⁡πE(oimgalign,a)∼Dalign[−log⁡π(a∣oimgalign)].(9)\pi_{\mathrm{student}}^{*} = \arg\min_{\pi} \mathbb{E}_{(o_{\mathrm{img}}^{\mathrm{align}},a)\sim\mathcal{D}_{\mathrm{align}}}\left[-\log\pi(a\mid o_{\mathrm{img}}^{\mathrm{align}})\right]. \tag*{(9)}

Our method uses image compositing to align images with deployment observation conditions, then trains the student policy using the aligned observations and teacher actions.

Experimental Setup

Hardware and Task Details

We used a dual-arm platform comprising two UFACTORY xArm7 manipulators. For each task, we placed one fixed

external camera (RealSense D435) at a position that provided a clear view of the workspace.

The experimental tasks consisted of the Box Rotation Task, the Drawer Opening Task, and the Cylinder Rolling Task, as shown in Figure 4.

Figure 4. Real-world manipulation tasks.

In the Box Rotation Task, the robot rotates a box from a randomly assigned initial pose to 90 degrees, where the box is perpendicular to the silver pedestal. A trial is successful if the box comes to rest at an orientation within 5.0 degrees of 90.0 degrees. For evaluation, we defined 10 initial box poses relative to the center of the worktable. The poses were selected from within ranges of ±80\pm80 mm in lateral displacement, −30-30 to +50+50 mm in longitudinal displacement, and ±25\pm25 degrees in orientation. Each trial had a time limit of 30.0 s.

In the Drawer Opening Task, the robot opens a closed drawer from a randomly assigned initial position by at least 10 cm. Because the proposed method obtains state information from object-mounted trackers, it can be readily applied to tasks involving articulated objects by mounting trackers on their individual links. For evaluation, we similarly defined 10 initial drawer positions relative to the reference point. The positions were selected from within ranges of ±45\pm45 mm in lateral displacement and ±30\pm30 mm in longitudinal displacement. Each trial had a time limit of 7.5 s.

In the Cylinder Rolling Task, the robot rolls a tapered cylindrical object until it comes to rest with its label facing upward, corresponding to an orientation of 0 degrees. Because the object can quickly reach high speeds while rolling, we expected demonstration collection via teleoperation to be difficult and selected this task to evaluate the effectiveness of the proposed method. For evaluation, we similarly defined 10 initial cylinder poses relative to the reference point. The poses were selected from within ranges of ±120\pm120 mm in lateral displacement, −30-30 to +60+60 mm in longitudinal displacement, and +90+90 to +270+270 degrees in orientation. Each trial had a time limit of 15 s.

For each task, the 10 initial configurations used for evaluation intentionally included two or three configurations outside the initial-pose distribution used during demonstration collection. We treated these as out-of-distribution (OOD) initial configurations and used them to evaluate policy generalization beyond the training distribution.

Across all tasks, the robot was configured to stop automatically when the contact force exceeded a predefined threshold, and any such contact was considered a failure.

Accordingly, we used the following three metrics to evaluate the proposed method in terms of both task-completion capability and motion efficiency. Here, ↑ indicates higher is better, and ↓ indicates lower is better.

  • Success Rate (SR↑)

  • Average Time for Successful Rollouts (ATR↓)

  • Throughput-with-Regret (TPR↑)

TPR↑ is a metric proposed in [28] that combines success rate and execution time for successful rollouts. It is defined as follows:

TPR=1N∑i=1N((1ti⋅Si)−(1tmax⁡⋅(1−Si))),(10)\mathrm{TPR} = \frac{1}{N}\sum_{i=1}^{N}\left(\left(\frac{1}{t_i}\cdot S_i\right)-\left(\frac{1}{t^{\max}}\cdot(1-S_i)\right)\right), \tag*{(10)}

where tit_i denotes the execution time of trial ii, tmax⁡t^{\max} is the maximum time limit allowed for each trial, and Si∈{0,1}S_i \in\{0,1\} indicates the success of trial ii. It assigns the same penalty to every failure while rewarding faster successes more highly, enabling a joint assessment of success rate and execution time.

In the evaluation experiments, each policy trained with a given random seed was tested on the 10 initial configurations defined for evaluation.

Implementation Details of the Proposed Method

State-Based RL in Simulation In all tasks, each action specified incremental joint position commands. The state observation included a task-specific subset of the following variables: robot joint positions and velocities, current and target tool-center-point (TCP) poses, and current and previous object poses. We used both arms for Box Rotation and Cylinder Rolling and a single arm equipped with a gripper for Drawer Opening. During training, we added noise to the object-pose observations to reduce the sim-to-real gap and stabilize real-world execution.

For each task, the dense reward comprised four components: (1) progress toward the target object state, (2) end-effector proximity to the desired contact points, (3) end-effector retreat after task completion, and (4) penalties for undesirable contacts such as self-collision. This reward design was informed by prior work, including ManiSkill [29] and Bi-Touch [11]. We used Soft Actor-Critic (SAC) [30] for Box Rotation and Proximal Policy Optimization (PPO) [31] for the other two tasks. We trained all RL policies in ManiSkill at a control frequency of 20 Hz.

Demonstration Collection and Tracker Artifact Removal We used the RoboManipBaselines software framework [32] for demonstration collection and subsequent IL. We used an ArUco Marker Board [33] to track the manipulated object.

The number of collected episodes was determined by the initial configurations and data-collection constraints of each task. During data collection for all tasks, we repeated rollouts following teacher-policy failures. For Box Rotation, the grid of initial configurations yielded 45 episodes. For this task, we used two teacher policies because we did not obtain a single policy that consistently succeeded across the collection region. For Drawer Opening, the real-world teacher policy failed to complete the task from some initial positions, preventing the collection of successful trajectories. Possible causes include the sim-to-real gap in the controller. We therefore reduced the real-world collection area and used 40 episodes collected within the region where the teacher succeeded. For Cylinder Rolling, the markers left the camera’s field of view during execution from some initial configurations; we excluded these configurations and used 36 episodes. The student policy input for IL was an observation oimgo_{\mathrm{img}} comprising a single marker-powered RGB image from the fixed external camera and robot joint angles.

Imitation Learning We used Diffusion Policy (DP) [6], a strong and widely adopted imitation learning algorithm. Visual observations had a resolution of 320×240320 \times240 pixels and were cropped to 288×216288 \times216 pixels before being input to the network. We used a CNN backbone and trained the policy for 1000 epochs. During deployment of DP, the control frequency was approximately 20 Hz on average.

Experimental Results

Overview of Compared Methods

Figure 5 summarizes the workflows and characteristics of the learning pipelines for the three baseline methods and the proposed method.

Comparison of the training pipelines of the proposed method and the three baseline methods

Figure 5. Comparison of the training pipelines of the proposed method and the three baseline methods. The key characteristics of each method are also summarized on the right.

Distillation in Simulation is a method that collects demonstration data from rollouts of state-based RL in simulation and then trains an image-based policy on the collected data via offline IL. Because the entire data collection process is completed in simulation, this method enables efficient large-scale data generation. On the other hand, the method is more susceptible to visual and physical sim-to-real gaps when deployed in the real world. For this baseline, we randomized the lighting conditions in simulation and sampled the initial object configurations more densely, resulting in a training dataset with twice as many demonstration episodes as those used for the other methods. To examine the effect of the environment in which demonstrations are generated, we used the same offline IL procedure (Diffusion Policy) for this baseline and the proposed method.

IL with Human Demonstrations is a representative approach that directly uses real-world data. A skilled teleoperator collected demonstrations using two 3Dconnexion SpaceMouse devices. We trained a Diffusion Policy using a dataset containing the same number of episodes as the proposed method and evaluated its real-robot performance.

  • Real-world data are used; however, the training images undergo tracker artifact removal.

This baseline provides a reference for IL performance when demonstrations from a skilled human operator are available.

Image-based RL is an approach that directly trains a policy with RL using image observations as input. Although this approach can be conducted entirely in simulation, it has the disadvantage of being challenging due to training instability and high computational cost.

ReDOA (Ours) is described in detail in Section III.

Real-World Comparison with Baselines

We compare the proposed method with Distillation in Simulation and IL with Human Demonstrations through real-world experiments. The former evaluates whether performing policy distillation in the real world alleviates the visual sim-to-real gap, while the latter assesses how effectively the proposed framework can replace expert human demonstrations. Since Image-based RL was evaluated only in simulation, we discuss it separately in § 6-D, focusing on its training behavior rather than real-world performance. As the primary evaluation metric, we use Throughput-with-Regret (TPR↑\uparrow), which jointly captures task success and execution efficiency.

The quantitative results are summarized in Table I. The proposed method achieved the highest TPR↑\uparrow on all three tasks.

The comparison with Distillation in Simulation highlights the importance of performing policy distillation in the real world. The baseline achieved only 14.0% SR↑\uparrow on Box Rotation, and therefore was not evaluated on the remaining tasks. Despite using randomized lighting conditions and twice as many demonstrations as the other methods, the policy did not achieve sufficient real-world transfer performance under the dataset size and training conditions used in this experiment. Furthermore, successful trials were limited to relatively easy initial configurations in which the box was already rotated by approximately 15 degrees. Consequently, its Average Time for Successful Rollouts (ATR↓\downarrow), computed only over successful trials, was the shortest (7.56 s).

The proposed method also outperformed IL with Human Demonstrations in execution efficiency. Averaged over all three tasks, it reduced ATR↓\downarrow by 5.33 s, with the largest improvement (11.45 s) observed on Box Rotation. This difference primarily reflects the execution efficiency of the demonstration trajectories themselves: the average ATR↓\downarrow of the RL-generated and human demonstrations was 6.88 s and 21.98 s, respectively.

The efficiency gain was particularly pronounced in Box Rotation, where collecting demonstrations through teleoperation is inherently challenging. During manipulation, the box frequently occluded one of the end-effectors, preventing the operator from continuously monitoring the contact relationships between the end-effectors, the object, and the tabletop. As a result, the operator tended to execute conservative motions with larger safety margins. In contrast, the RL teacher policy had already learned efficient manipulation strategies in simulation based on the spatial relationship between the robot and the object. Consequently, it could execute the task efficiently using observations such as the object pose and robot joint states. These findings suggest that the proposed framework can generate efficient behaviors that are difficult to demonstrate through vision-based teleoperation.

The proposed method achieved an average SR↑\uparrow that was 8.0 percentage points lower than IL with Human Demonstrations. We attribute this gap to two factors. First, unlike the human demonstrations, which maintained relatively large safety margins around the manipulated object, the RL-generated demonstrations occasionally contained trajectories passing closer to the object. As a result, for some out-of-distribution initial configurations, unintended collisions reduced the success rates in Box Rotation and Drawer Opening. This observation suggests a trade-off between execution efficiency and robustness to out-of-distribution initial states.

Second, the marker-removal preprocessing may have slightly degraded the visual observations. Filling the tracker regions with a uniform color removed local shadow cues and introduced subtle appearance discontinuities, which may have negatively affected the learned policy. We investigate this effect in the following ablation study.

MethodBox RotationDrawer OpeningCylinder Rolling
SR (%)↑\uparrowATR (s)↓\downarrowTPR↑\uparrowSR (%)↑\uparrowATR (s)↓\downarrowTPR↑\uparrowSR (%)↑\uparrowATR (s)↓\downarrowTPR↑\uparrow
Distill. in Sim14.07.56-0.010------
IL with Human Demo76.020.260.03778.05.490.11972.09.840.056
ReDOA (Ours)70.08.810.07564.02.750.18768.08.030.068

Table 1. Performance evaluation results. Results are averaged over 5 random seeds (10 trials per seed, 50 trials in total).

Overall, although the proposed method achieved a lower SR↑SR \uparrow than IL with Human Demonstrations, it achieved the highest TPR↑TPR \uparrow across all evaluated tasks. These results indicate that the proposed framework attains strong task performance and efficient execution while significantly reducing the need for costly human data collection.

Ablation Study on Marker Removal

The proposed method removes the visible trackers from the demonstration images by filling their regions with a uniform color. This process removes the original shadows and creates slight color discontinuities between the filled and surrounding regions. If the IL policy learns to rely on these artificial appearance cues, its performance may degrade at deployment.

To evaluate this effect, we conducted an ablation study on the Box Rotation Task by varying whether image compositing was applied and whether the deployment environment contained markers, as shown in Table II.

The bottom row reports the results of deploying the policy trained without image compositing in the Marker-Based environment. This setting requires markers during deployment but does not require their removal from the training images, thereby matching the visual conditions between training and deployment. We therefore use this result as an empirical upper-bound reference for image-based policy performance in our problem setting.

When the same policy was deployed directly in the Marker-Free environment, as shown in the second row, SRSR decreased by 20.0 percentage points, from 73.3% to 53.3%, and TPRTPR decreased from 0.086 to 0.047. This result shows that mismatched marker conditions between training and deployment substantially degrade policy performance.

With image compositing, the proposed method achieved an SRSR of 70.0% and a TPRTPR of 0.079 in the same Marker-Free environment, differing from the upper-bound reference by 3.3 percentage points and 0.007, respectively. Thus, image compositing enabled near-upper-bound performance in a marker-free deployment environment.

Image CompositingDeploy EnvSR (%)↑\uparrowATR (s)↓\downarrowTPR↑\uparrow
ProposedMarker-Free70.08.620.079
w/o Image CompositingMarker-Free53.38.940.047
w/o Image CompositingMarker-Based73.37.950.086

Table 2. Ablation study evaluating the effect of marker removal. Results are averaged over 3 random seeds (10 trials per seed, 30 trials in total).

Training curves of state-based SAC and image-based SAC in simulation

Figure 6. Training curves of state-based SAC and image-based SAC in simulation. Curves are smoothed for visualization. State-based SAC learns substantially faster and converges to a higher return than image-based SAC.

Training Behavior of Image-Based RL

Figure 6 shows the simulated training curves of state-based SAC and image-based SAC for Box Rotation. While state-based SAC converged to a high return at an early stage, image-based SAC also made learning progress but remained inferior to state-based SAC in both convergence speed and final performance. In addition, because image-based SAC involves image rendering in simulation, its computational cost was higher than that of state-based SAC. In practice, training for 15M timesteps required 12 hours for state-based SAC and 26 hours for image-based SAC.

Thus, although image-based RL can serve as an end-to-end solution for this task, it still faces challenges in both training efficiency and achievable performance.

Limitations

The first and most fundamental limitation of our method is its assumption that an RL teacher policy capable of completing the target task on the real robot can be obtained in advance. Consequently, the method is difficult to apply to objects whose dynamics cannot be modeled with sufficient fidelity in simulation, such as cardboard with folds. It is also difficult to apply the method to objects such as cloth, for which the state information cannot be measured reliably in the real world. Extending the method to such objects will require advances in simulation fidelity and real-world state-estimation techniques.

A second limitation is that the current marker-removal process fills marker regions with a uniform color and therefore cannot reproduce shadows, brightness variations, or printed patterns, leaving appearance inconsistencies with the surrounding regions. Aligning the filled regions with their surroundings using scanned texture projection or image harmonization [34] could improve IL performance and broaden the method’s applicability to objects with text or graphical patterns.

A third limitation is that using planar fiducial markers as trackers restricts the method to objects with rigid planar surfaces and prevents it from handling occlusions that hide all markers. Embedding a magnetic tracker inside an object entails substantial setup costs, but this approach can overcome these limitations, thereby extending the method to a wider range of objects and tasks. Because the tracker does not appear in the images, this approach can also eliminate the need for tracker removal.

Our evaluation is limited to offline IL using fixed datasets. Introducing DAgger-style data aggregation into ReDOA could mitigate state-distribution mismatch and further improve performance by learning additional teacher actions for states visited by the student policy. Evaluating the benefit of this extension is left for future work. The simulation-distillation baseline could also benefit from DAgger-style learning. Since we have not evaluated this effect, the present comparison does not establish ReDOA’s superiority over DAgger-style distillation in simulation.

Conclusion

In this paper, we introduced ReDOA, a real-world distillation framework for learning image-based manipulation policies while bypassing the visual sim-to-real gap. We roll out a state-based teacher trained in simulation on a real robot, fill tracker regions in the collected images through image compositing, and train a student policy. The key idea is to mitigate the mismatch between data-collection and deployment images caused by the trackers used to measure the teacher’s state observations, transferring control knowledge to a policy that requires no trackers during deployment.

Experiments on three non-prehensile manipulation tasks, including bimanual tasks, showed success rates close to those of IL based on human demonstrations and higher motion efficiency, demonstrating a balance between task success and execution efficiency. The ablation study further confirmed that tracker removal, our implementation of Observation Alignment, reduces the observation mismatch between data collection and deployment and is an important component supporting performance. These results demonstrate that ReDOA is a practical pipeline for learning robust, motion-efficient image-based policies.

References

  1. [1]R. Chen, M. Mukadam, M. Kaess, T. Wu, F. R. Hogan, J. Malik, and A. Sharma, “PTLD: Sim-to-real privileged tactile latent distillation for dexterous manipulation,” arXiv preprint arXiv:2603.04531, 2026.arxiv.org/abs/2603.04531
  2. [2]J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017, pp. 23–30.arxiv.org/abs/1703.06907
  3. [3]X. Li, K. Hsu, J. Gu, O. Mees, K. Pertsch, H. R. Walke, C. Fu, I. Lunawat, I. Sieh, S. Kirmani, S. Levine, J. Wu, C. Finn, H. Su, Q. Vuong, and T. Xiao, “Evaluating real-world robot manipulation policies in simulation,” in Conference on Robot Learning (CoRL), vol. 270. PMLR, 2025, pp. 3705–3728.arxiv.org/abs/2405.05941
  4. [4]I. Josifovski, M. Malmir, N. Klarmann, B. L. Žagar, N. Navarro-Guerrero, and A. Knoll, “Analysis of randomization effects on sim2real transfer in reinforcement learning for robotic manipulation tasks,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022, pp. 10 193–10 200.
  5. [5]T. Z. Zhao, A. Jain, B. Ichter, and S. Levine, “Learning fine-grained bimanual manipulation with low-cost hardware,” in Proceedings of Robotics: Science and Systems (RSS), 2023.
  6. [6]C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” in Proceedings of Robotics: Science and Systems (RSS), 2023.arxiv.org/abs/2303.04137
  7. [7]A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang, R. Kulkarni, L. Fei-Fei, S. Savarese, Y. Zhu, and R. Martín-Martín, “What matters in learning from offline human demonstrations for robot manipulation,” arXiv preprint arXiv:2108.03298, 2021.
  8. [8]K. Kase, N. Matsumoto, and T. Ogata, “Leveraging motor babbling for efficient robot learning,” J. Robot. Mechatron., vol. 33, no. 5, pp. 1063–1074, 2021.
  9. [9]F. Ruggiero, V. Lippiello, and B. Siciliano, “Nonprehensile dynamic manipulation: A survey,” IEEE Robotics and Automation Letters, vol. 3, no. 3, pp. 1711–1718, 2018.DOI
  10. [10]N. Kawarai and Y. Kobayashi, “Learning of whole arm manipulation with constraint of contact mode maintaining,” J. Robot. Mechatron., vol. 22, no. 4, pp. 542–550, 2010.
  11. [11]Y. Lin, A. Church, M. Yang, H. Li, J. Lloyd, D. Zhang, and N. F. Lepora, “Bi-Touch: Bimanual tactile manipulation with sim-to-real deep reinforcement learning,” IEEE Robotics and Automation Letters, vol. 8, no. 9, pp. 5472–5479, 2023.
  12. [12]Y. Shirai, K. Ota, D. K. Jha, and D. Romeres, “Sim-to-Real Contact-Rich Pivoting via Optimization-Guided RL with Vision and Touch,” in NeurIPS 2025 Workshop on Embodied World Models for Decision Making, 2025.
  13. [13]C. Xu, Q. Li, J. Luo, and S. Levine, “RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning,” in Proceedings of Robotics: Science and Systems (RSS), 2025.
  14. [14]L. Cong, M. Görner, P. Ruppel, H. Liang, N. Hendrich, and J. Zhang, “Self-adapting recurrent models for object pushing from learning in simulation,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 5304–5310.
  15. [15]Y. Sueoka, S. Ozaki, Y. Kato, and T. Yoshida, “Real-world validation of a learned policy for cooperative object transport with the transformer encoder,” J. Robot. Mechatron., vol. 37, no. 5, pp. 1034–1041, 2025.
  16. [16]L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel, “Asymmetric actor critic for image-based robot learning,” in Proceedings of Robotics: Science and Systems (RSS), 2018.
  17. [17]D. Yarats, R. Fergus, A. Lazaric, and L. Pinto, “Mastering visual continuous control: Improved data-augmented reinforcement learning,” in International Conference on Learning Representations, 2022.arxiv.org/abs/2107.09645
  18. [18]R. Singh, A. Allshire, A. Handa, N. Ratliff, and K. V. Wyk, “DextrAH-RGB: Visuomotor policies to grasp anything with dexterous hands,” arXiv preprint arXiv:2412.01791, 2024.
  19. [19]T. He, Z. Wang, H. Xue, Q. Ben, Z. Luo, W. Xiao, Y. Yuan, X. Da, F. Castaneda, S. Sastry, C. Liu, G. Shi, L. Fan, and Y. Zhu, “VIRAL: Visual sim-to-real at scale for humanoid loco-manipulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.
  20. [20]M. Sorokin, J. Tan, C. K. Liu, and S. Ha, “Learning to navigate sidewalks in indoor environments,” IEEE Robotics and Automation Letters, vol. 7, no. 2, pp. 3906–3913, 2022.arxiv.org/abs/2109.05603
  21. [21]T. G. W. Lum, M. Matak, V. Makoviychuk, A. Handa, A. Allshire, T. Hermans, N. D. Ratliff, and K. V. Wyk, “DextrAH-G: Pixels-to-action dexterous arm-hand grasping with geometric fabrics,” in Conference on Robot Learning (CoRL), 2024.
  22. [22]J. Yamada, A. L. Mitchell, J. Collins, and I. Posner, “COMBO-Grasp: Learning constraint-based manipulation for bimanual occluded grasping,” in Conference on Robot Learning (CoRL), vol. 305. PMLR, 2025, pp. 1102–1119.
  23. [23]Z. Chen, Q. Yan, Y. Chen, T. Wu, J. Zhang, Z. Ding, J. Li, Y. Yang, and H. Dong, “ClutterDexGrasp: A sim-to-real system for general dexterous grasping in cluttered scenes,” in Conference on Robot Learning (CoRL), 2025.
  24. [24]C. Acar, K. Binici, A. Tekirdag, and Y. Wu, “Visual-policy learning through multi-camera view to single-camera view knowledge distil- lation for robot manipulation tasks,” IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 691–698, 2024.
  25. [25]W. Liu, Y. Wang, J. Wang, Y. Kuang, W. Cui, X. Shi, H. Li, D. Zhao, Z. Zhang, and H. Wang, “Fetchbot: Learning generalizable object fetching in cluttered scenes via zero-shot Sim2Real,” arXiv preprint arXiv:2502.17894, 2025.arxiv.org/abs/2502.17894
  26. [26]X. Han, J. Yu, M. Liu, Y. Chen, X. Lyu, Y. Tian, B. Wang, W. Zhang, and J. Pang, “Re3Sim: Generating high-fidelity simulation data via 3d-photorealistic real-to-sim for robotic manipulation,” arXiv preprint arXiv:2502.08645, 2025.arxiv.org/abs/2502.08645
  27. [27]N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Grschik, P. Dollár, and C. Feichtenhofer, “SAM 2: Segment anything in images and videos,” arXiv preprint arXiv:2408.00714, 2024.
  28. [28]N. R. Arachchige, Z. Chen, W. Jung, W. C. Shin, R. Bansal, P. Barroso, Y. H. He, Y. C. Lin, B. Joffe, S. Kousik, and D. Xu, “SAIL: Faster-than-demonstration execution of imitation learning policies,” in Conference on Robot Learning (CoRL), vol. 305. PMLR, 2025, pp. 721–749.arxiv.org/abs/2506.11948
  29. [29]S. Tao, F. Xiang, A. Shukla, Y. Qin, X. Hinrichsen, C. Yuan, C. Bao, X. Lin, Y. Liu, T.-K. Chan, Y. Gao, X. Li, T. Mu, N. Xiao, A. Gurha, V. N. Rajesh, Y. W. Choi, Y.-R. Chen, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su, “ManiSkill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI,” in Proceedings of Robotics: Science and Systems (RSS), 2025.arxiv.org/abs/2410.00425
  30. [30]T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proceedings of the 35th International Conference on Machine Learning (ICML), vol. 80. PMLR, 2018, pp. 1856–1865.
  31. [31]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  32. [32]M. Murooka, T. Motoda, R. Nakajo, H. Oh, K. Makihara, K. Shirai, T. Ogata, and Y. Domae, “RoboManipBaselines: A unified framework for imitation learning in robotic manipulation across real and simulation environments,” IEEE Access, vol. 14, pp. 97896–97909, 2026.DOI
  33. [33]S. Garrido-Jurado, R. Muñoz-Salinas, F. J. Madrid-Cuevas, and M. J. Marín-Jiménez, “Automatic generation and detection of highly reliable fiducial markers under occlusion,” Pattern Recognition, vol. 47, no. 6, pp. 2280–2292, 2014.DOI
  34. [34]Y.-H. Tsai, X. Shen, Z. Lin, K. Sunkavalli, X. Lu, and M.-H. Yang, “Deep image harmonization,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2799–2807.

Paper details

vlnc:W_y9r6mgqf
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents