betaFeedback

WorksComputer Science

Online Adaptation to Latent Physical Properties for Learning from Demonstration

Abstract

Manipulating visually identical objects with different physical properties requires adapting actions based on information acquired through interaction. We address the problem of learning from demonstrations how to adapt manipulation actions, such as regrasping and changing contact locations, based on physical-property information acquired during interaction. To this end, we propose a learning-from-demonstration method that learns latent representations of physical properties needed to predict object responses and conditions a policy on online estimates of these representations. A dynamics model and object-specific latent representations are jointly learned from demonstrations that include behaviors adapting to the objects' physical properties, and the learned representations are then used to condition an imitation policy. During execution, the dynamics model and policy weights remain fixed, while the latent representation is updated based on the discrepancy between predicted and actual observations, allowing interaction with the object to inform action generation. Real-world and simulation experiments with objects whose centers of mass differ suggest that online adaptation through latent representations of physical properties can improve task success rates. However, in simulation, success rates on center-of-mass locations absent from the demonstrations remained around 30%, indicating limited adaptation to unseen physical properties. Project page: https://isri-aist.github.io/latent-dynamics-project-page/

Introduction

Learning from demonstration (LfD) has been applied to a wide range of robotic manipulation tasks, including manipulation of deformable objects and long-horizon tasks. These methods have shown strong performance in settings where visual observations provide sufficient information for action selection [1], while adding force measurements supports delicate tasks such as wiping [2], [3]. Humans infer physical properties such as mass and friction through interaction and use this information to guide subsequent actions, for example, adjusting their grasp and posture when lifting a heavy object.

Regrasping or changing contact locations based on inferred properties requires retaining information from earlier contacts and using it to select subsequent actions. Policies conditioned on a short observation window may struggle when action selection requires information outside that window [4]. In this work, we focus specifically on physical properties inferred from earlier interactions.

Beyond LfD, online system identification has been extensively studied through analytical approaches, dynamics-based property estimation [5], and adaptation using simulation-trained dynamics models [6]. Connecting this body of work with LfD could substantially broaden the range of manipulation problems that can be learned from demonstrations.

In this work, we collect demonstrations of property-dependent manipulation using visually identical objects with different centers of mass (CoM) or centers of friction (CoF). The demonstrations include corrective actions in response to unexpected object motion, such as regrasping or changing contact locations. Within this setting, we propose and evaluate an imitation learning method that adapts online to differences in object physical properties.

Figure 1 provides an overview of the proposed approach. To adapt to these latent physical properties, we introduce an implicit dynamics parameter di\mathbf{d}_i as a latent representation of object ii. We first learn the dynamics model and di\mathbf{d}_i from demonstrations, then train a policy conditioned on the learned representation. During execution, we update the estimate d^\hat{\mathbf{d}} online using discrepancies between model predictions and actual observations, and generate actions conditioned on the inferred properties.

Overview of the proposed approach

Figure 1. Overview of the proposed approach. ① Teleoperated demonstrations include adaptive actions based on observed object responses, such as regrasping or changing contact locations. ② Using the same demonstrations, we first jointly learn a dynamics model and object-specific physical representations, then train an imitation policy conditioned on the learned representations. This learns both dynamics and adaptive actions that depend on physical properties. ③ Starting from a reference physical representation, the robot updates it from prediction errors and conditions subsequent actions on the estimate. The figure illustrates uniform density as the initial assumption.

The method has two advantages. First, it initially learns a

This work was partially supported by JST CREST, Japan, under Grant Number JPMJCR2553, and JST Moonshot R&D Program, Japan, under Grant Number JPMJMS263D. dynamics model and implicit physical representations solely from demonstrations, without prior measurements of the objects’ physical properties. Second, its continuous representation, rather than a discrete object category, is intended to support adaptation to unseen objects.

Our contributions are as follows:

  • We show that interaction data in manipulation demonstrations can be used to learn implicit representations of differences in object physical properties.

  • We demonstrate that combining LfD with dynamics-based property estimation improves performance on tasks involving visually identical objects with different physical properties, including deformable objects.

Related Work

Physical Property Estimation for Manipulation

Physical properties such as mass and friction affect object responses and therefore matter for prediction and control. Analytical models support approaches such as hybrid model predictive control for planar manipulation [7], but require specifying physical parameters and suitable assumptions about contact and friction. The cost of characterizing each object in advance motivates estimating its properties through interaction.

For predicting object responses, latent representations can capture relevant physical differences without explicitly estimating physical parameters such as mass or friction coefficients. Longhini et al. [8] learn latent representations of elasticity from pulling interactions to condition graph-based predictions of cloth dynamics. Evans et al. [9] infer latent context from state transitions, including real-robot sliding interactions with different objects and surfaces, to adapt dynamics predictions. Simulation-trained models also enable adaptation to real objects: EXI-Net [6] updates explicit and implicit dynamics parameters from prediction errors during planar pushing, while Poke and Strike [10] learns an exploration policy and a property estimator, then uses the estimated properties to condition task execution.

Physical representations have also been incorporated into LfD. Aoyama et al. [11] pretrain haptic representations from simulated force sequences and use a few real demonstrations to learn wiping motions adapted to sponge stiffness and surface friction. Saito et al. [12] infer ingredient characteristics from multimodal observations during stirring with a wooden spatula, then use the inferred representation to select and manipulate tools. Both methods obtain these representations through a dedicated exploration phase before task execution. Our demonstrations instead integrate task execution with responses to unexpected object behavior, including regrasping and changes in contact location. We update the physical representation throughout execution to support these adaptive behaviors without explicitly separating exploration from the task.

History-Dependent Imitation Learning

Adaptive manipulation may depend on information that is no longer directly observable: an off-center mass detected during lifting, for example, can inform a later grasp after the object has been put down. A policy conditioned only on a recent observation window loses access to such information once it falls outside the window. Simply extending the window does not guarantee better performance. Sobol Mark et al. [13] show that limited coverage of observation histories can lead policies to rely on spurious correlations that fail under rollout distribution shift. Their BPP method uses a vision-language model to identify keyframes corresponding to task-relevant events, reducing these correlations and improving task performance.

Recurrent approaches, including the spatial-attention model of Ichihara et al. [14] and Mamba-based MTIL [4], instead encode observation history in an internal state used for action generation. Our approach focuses this retained information on physical properties relevant to predicting object responses, using a latent representation updated from dynamics prediction errors during execution.

World Models for Policy Learning and Adaptation

World models predict the consequences of actions and can inform policy learning and execution. An et al. [15] propose a Feedback World Model that predicts future visual features and corrects its predictions online using observation feedback to guide Diffusion Policy denoising. For nonprehensile manipulation, Lyu et al. [16] introduce DyWA, which distills a simulation-trained reinforcement learning teacher into a student that jointly predicts actions and subsequent task states. Its history-based adaptation representation is supervised by teacher features encoding object geometry and physical parameters.

Yuan et al. [17] integrate a pretrained Cosmos world model [18], an action model, and a force predictor in Ada-WorldPolicy. Prediction errors drive online updates to low-rank network parameters, enabling adaptation to both visual and physical domain shifts. Our focus is on learning object-specific physical representations by minimizing prediction errors on demonstration data, without physical-parameter supervision. Our method instead updates only the latent physical representation online, keeping the dynamics model and policy weights fixed.

Proposed Method

Overview

Figure 2 summarizes the proposed method. We first collect demonstrations of manipulation adapted to the differing physical properties of visually identical objects (step ① in Figure 1). From these demonstrations, we jointly learn a dynamics model that predicts future observations and an implicit dynamics parameter representing the latent physical properties of each object (step ②-1 in Figure 2). In step ②-2, we annotate the demonstrations with the learned physical representations and train a Diffusion Policy conditioned on observations and these representations. During deployment (step ③), we keep the model and policy weights fixed, update the implicit dynamics parameter using discrepancies between predicted and actual observations, and generate actions conditioned on the current estimate.

Training and deployment pipeline of the proposed method

Figure 2. Training and deployment pipeline of the proposed method. tt denotes the discrete time-step index, and HH is the prediction horizon in steps.

Data Collection, Training, and Online Adaptation

① Demonstration Collection: We construct a dataset D\mathcal{D} by collecting demonstrations of manipulation adapted to the differing physical properties of MM types of visually identical objects (step ① in Figure 1). The operator commands adaptive actions based on information acquired through interaction during manipulation. For each demonstration, we record observations and robot action commands and assign an object-type label ii (i=0,…,M−1i = 0,\ldots,M-1). We use the same dataset to train the dynamics model and the policy.

②-1 Learning Dynamics and Physical Representations: We first train a variational autoencoder (VAE) [19] from scratch on the demonstration images (step ②-1 in Figure 2). After training, we freeze the encoder and denote the low-dimensional features extracted from the image at time tt by ztz_t.

Let did_i denote the implicit dynamics parameter for object type ii. It is a learnable embedding vector shared across all timesteps and demonstrations of that object type. We initialize each component of did_i to 0.2i0.2i.

Let ptp_t denote the robot’s proprioceptive information at time tt. For the pushing task, the observations also include forces and torques wtw_t measured by a six-axis wrist force/torque sensor.

Let ata_t be the action command issued to the robot at time tt and HH the prediction horizon. The dynamics model predicts future observations from the image features and proprioceptive information at the previous and current timesteps, the action sequence over the next HH steps, and the implicit dynamics parameter. The prediction target is yt=zty_t = z_t, or yt=(zt,wt)y_t = (z_t,w_t) when forces and torques are included for the pushing task. We implement the dynamics model as a multilayer perceptron (MLP):

y^t+1:t+H=fθ(zt−1:t,pt−1:t,at:t+H−1,di).(1)\hat{y}_{t+1:t+H}=f_\theta\left(z_{t-1:t},p_{t-1:t},a_{t:t+H-1},d_i\right). \tag*{(1)}

Here, y^\hat{y} denotes the predictions and θ\theta the model parameters. Training minimizes the mean squared error between the predictions and the corresponding actual observations. Omitting the prediction-horizon indices t+1:t+Ht+1:t+H, the prediction loss is

Ldyn(θ,di)=MSE⁡(z^,z)+λwMSE⁡(w^,w).(2)\mathcal{L}_{\mathrm{dyn}}(\theta,d_i)=\operatorname{MSE}(\hat{z},z)+\lambda_w\operatorname{MSE}(\hat{w},w). \tag*{(2)}

Here, MSE denotes the mean squared error over all timesteps and components in the prediction horizon, and λw\lambda_w weights the force/torque prediction error. We omit the second term for tasks without force/torque prediction. For each demonstration, we select did_i using its object label ii and jointly optimize θ\theta and {di}\{d_i\} to minimize the prediction loss:

(θ∗,{di∗})=arg⁡min⁡θ,{di} ED[Ldyn(θ,di)].(3)(\theta^*,\{d_i^*\})=\underset{\theta,\{d_i\}}{\arg\min}\ \mathbb{E}_{\mathcal{D}}\left[\mathcal{L}_{\mathrm{dyn}}(\theta,d_i)\right]. \tag*{(3)}

Thus, the learned physical representations capture object differences relevant to observation prediction, without requiring explicit physical parameter values.

②-2 Policy Learning with Physical Representations: We annotate the demonstrations of object ii with its learned physical representation di∗d_i^* from (3), yielding a dataset Dd\mathcal{D}_d (step ②-2 in Figure 2). Let o\mathbf{o} denote the observation history supplied to the policy and a\mathbf{a} the demonstrated action sequence. We train a Diffusion Policy [1] π(a∣o,di∗)\pi(\mathbf{a}\mid\mathbf{o},\mathbf{d}_i^*) conditioned on the observation history and di∗\mathbf{d}_i^*. During this stage, di∗\mathbf{d}_i^* remains fixed and only the policy is trained.

The imitation learning objective can be expressed in terms of the conditional action distribution as

π∗=arg⁡min⁡π E(o,a,di∗)∼Dd[−log⁡π(a∣o,di∗)].(4)\pi^* = \underset{\pi}{\arg\min}\ \mathbb{E}_{(\mathbf{o},\mathbf{a},\mathbf{d}_i^*)\sim\mathcal{D}_d}\left[-\log\pi(\mathbf{a}\mid\mathbf{o},\mathbf{d}_i^*)\right]. \tag*{(4)}

③ Deployment and Online Adaptation: Finally, we deploy the learned policy to control the robot and update the physical representation online using observations acquired through interaction (step ③ in Figure 2). Let d^\hat{\mathbf{d}} denote the estimated physical representation of the object being manipulated. We initialize it with a learned di∗\mathbf{d}_i^* corresponding to a plausible assumption about the object’s physical properties based on its appearance, such as uniform density or a center of friction at the center of the contact area. Given the observation history and the current estimate, the learned policy π∗(a∣o,d^)\pi^*(\mathbf{a}\mid\mathbf{o},\hat{\mathbf{d}}) generates an action sequence for the robot to execute.

We store executed action commands in a buffer and input them to the dynamics model fθ∗f_{\theta^*} together with past image features, proprioceptive information, and d^\hat{\mathbf{d}}. The model predicts observations from a past timestep up to the present, and we compute the loss in (2) from the discrepancies between these predictions and the actual observations. We update only d^\hat{\mathbf{d}} while keeping the model parameters θ∗\theta^* and the weights of π∗\pi^* fixed. In gradient descent form, the update is

d^←d^−ηd∇d^Ldyn(θ∗,d^).(5)\hat{\mathbf{d}} \leftarrow\hat{\mathbf{d}}-\eta_d\nabla_{\hat{\mathbf{d}}}\mathcal{L}_{\mathrm{dyn}}(\theta^*,\hat{\mathbf{d}}). \tag*{(5)}

Here, ηd\eta_d is the step size for updating d^\hat{\mathbf{d}}. We use the updated estimate for the next policy inference and repeat action generation and online adaptation.

Retraining with an Augmented Dataset

If the dynamics model does not adequately capture interactions encountered during execution, prediction-error-based updates may fail to estimate the physical representation corresponding to the target object. To improve online estimation, we can retrain both the dynamics model and the VAE that provides its visual representation using action-observation data collected during rollouts, without retraining the policy. Training the dynamics model on a broader data distribution is expected to improve the accuracy of online adaptation.

First, we add rollout data to the demonstrations collected in step ① of § 3-B, regardless of task success, to form an augmented dataset.

Next, we retrain the VAE from scratch on images from the augmented dataset, freeze the resulting encoder, and retrain the dynamics model. During dynamics model retraining, we keep {di∗}\{\mathbf{d}_i^*\} fixed and optimize only θ\theta. Keeping the weights of π∗\pi^* and the learned representations {di∗}\{\mathbf{d}_i^*\} fixed allows us to use additional, autonomously collected interaction data without retraining the policy. Although the rollouts are autonomous, their collection still requires operational effort, such as resetting the environment.

Experimental Setup

Manipulation Tasks

We evaluate the proposed method on two real-world tasks and use a simulation task to compare dynamics model input representations and observation history lengths (Figure 3). Throughout this paper, unseen objects refer to objects with CoM locations not included in the demonstrations. We evaluate adaptation to these unseen CoM locations in Box Pushing and Horizontal Lifting.

Manipulation tasks with objects whose centers of mass differ: (a) Pouch Placement, (b) Box Pushing, and (c) Horizontal Lifting

Figure 3. Manipulation tasks with objects whose centers of mass differ: (a) Pouch Placement and (b) Box Pushing on a real robot, and (c) Horizontal Lifting in simulation.

(a) Pouch Placement: The robot must pick up a deformable pouch and place it inside a designated frame within a time limit (Figure 3(a)). We vary the CoM by changing the position of a weight inside the pouch and collect demonstrations for three configurations, with the CoM near the robot, at the center, or far from the robot. Inserting the pouch into the frame is difficult unless it is level, so the demonstrator gently sets it down and regrasp it only if it tilts. We evaluate the policies on the same three CoM configurations used for demonstration collection.

(b) Box Pushing: The robot must push a rectangular box to the left until its two left corners lie on the yellow tape (Figure 3(b)). In the demonstrations, the box is initially placed at a random orientation between two yellow lines; evaluation uses two fixed initial poses. In the demonstrations, the operator first aligns the box with the tape, then pushes at a location aligned with its CoM to translate it without rotation. Task performance therefore depends directly on how quickly the CoM can be inferred during execution. A slider provides 11 equally spaced weight positions; we collect demonstrations at five positions and evaluate at all 11, including the six unseen positions.

(c) Horizontal Lifting: The robot must lift a bar and hold it level above a specified height at the end of a 17-s episode (Figure 3(c)). The bar has red and blue halves of equal length. We collect demonstrations for three CoM locations: the center of the red half, the center of the bar, and the center of the blue half. In every demonstration, a teleoperator who does not know the object’s physical properties first grasps the bar at its geometric center. If the bar remains level, the operator maintains that posture; if it tilts, the operator gently sets it down, regrasps near the inferred CoM, and lifts it again. Thus, the demonstrations explicitly include deliberate adaptation, rather than recovery after an accidental drop. Evaluation covers these three seen configurations and four additional, unseen CoM locations.

Experimental Protocol

All experiments are conducted using RoboManiPelines [20]. The real-world experiments use a UFACTORY xArm7 and one fixed external RealSense D435 camera positioned to view the workspace for each task. The simulation uses a Universal Robots UR5e manipulator with a parallel gripper. The control period is approximately 0.129 s for the real robot and 0.096 s for Horizontal Lifting; online updates of the physical representation use the same period.

Table 1 summarizes the demonstration and evaluation counts. Within each task, we use the same initial-pose distribution for all object configurations to avoid bias associated with object type. We train each method with three random seeds and aggregate success rates across evaluation rollouts.

TaskDemonstrationsEvaluation rollouts
Pouch Placement12 (36)3 (9)
Box Pushing10 (50)2 (22)
Horizontal Lifting25 (75)10 (70)

Table 1. Demonstration and evaluation counts. Each entry gives the count per object configuration, with the total in parentheses. Evaluation counts are per training seed.

Compared Methods and Policy Inputs

Our baseline is a standard Diffusion Policy (DP) trained on pooled demonstrations without distinguishing object types. Both the baseline and our method use an observation history of two timesteps (approximately 0.2 s). For the two real-world tasks, we compare this baseline with our method using an image-based dynamics model. For Horizontal Lifting, we compare four methods: standard DP, DP with a 16-step observation history (approximately 1.5 s), denoted DP (long history), and our method with either a state-based or an image-based dynamics model. The state-based dynamics model replaces the image features ztz_t with the object’s position and orientation from the simulator in both the input history and the prediction targets. The image-based model uses VAE features as described in Section III. These variants differ in the observations used as dynamics model inputs and prediction targets; their policy inputs are identical.

For both real-world tasks, the policies receive RGB images and end-effector position and orientation. Additional inputs are gripper joint angles for Pouch Placement and measured forces and torques for Box Pushing. The policies output commands specifying target end-effector position and orientation, with an additional gripper joint-angle command for Pouch Placement.

For Horizontal Lifting, all four methods use a state-based DP conditioned on end-effector position and orientation, gripper joint angles, and object position and orientation from the simulator. The actions specify target end-effector position and orientation and a gripper joint-angle command. These observation and action definitions are shared by all methods within each task.

Training and Online Adaptation

Across all tasks, we restrict changes in weight position and the resulting changes in CoM and CoF locations to one dimension. Accordingly, we use a scalar implicit dynamics parameter. We set the dynamics model’s prediction horizon to H=14H = 14 steps, corresponding to approximately 1.8 s in the real-world experiments and 1.3 s in simulation.

The image-based dynamics model uses nine-dimensional VAE features, with measured and commanded joint positions, including the gripper, as proprioceptive inputs and action inputs, respectively. A given end-effector pose can correspond to multiple joint configurations, which appear differently in the fixed external camera (Fig. 2). Using end-effector pose as input can make image-feature prediction unstable; joint positions avoid this ambiguity. For Box Pushing, we normalize image features and force/torque measurements separately and weight their prediction errors equally (λw=1\lambda_w = 1 in (2)).

We update the physical representation online using Adam [21], with a step size of ηd=6×10−3\eta_d = 6 \times10^{-3} for the state-based dynamics model and ηd=8×10−3\eta_d = 8 \times10^{-3} for the image-based model.

Experimental Results

Physical Representation Learning and Online Estimation

Figure 4 evaluates whether the learned physical representations di∗d_i^* capture differences in object responses and whether d^\hat{d} can be estimated online during rollouts.

Dynamics model evaluation

Figure 4. Dynamics model evaluation. Left: prediction errors on validation demonstrations excluded from all training. Solid lines use the representation of the actual object; dashed lines use those of other objects. Errors are measured in image features, except in (c-1), which shows position error. Panel (b) also compares predicted and measured force FyF_y along the pushing direction. Panels (c-1) and (c-2) use the same episode. Center: prediction-error differences averaged over all validation data for each object, relative to using its corresponding di∗d_i^*. Rows indicate the actual objects and columns the input representations. Positive values indicate larger errors when using another object’s representation. Values in the heatmaps are reported in units of 10−310^{-3}. Right: online estimates d^\hat{d} for all evaluation rollouts on seen objects only, with one curve per rollout. Horizontal dashed lines mark the learned, fixed representations di∗d_i^*; curves and reference lines for the same object share a color. For unseen objects with weight positions between those of two seen objects, convergence to values between the corresponding learned representations is desirable. Box Pushing trajectories are shown in Figure 6. The objects with centered weights have indices 1, 2, and 0 in Pouch Placement, Box Pushing, and Horizontal Lifting, respectively.

The examples in the left column of Figure 4 illustrate lower prediction errors when the model is conditioned on the corresponding di∗d_i^*, although each example represents only a single episode.

The heatmaps in the center column show that, for Pouch Placement and Horizontal Lifting, the mean prediction error is lower with the representation corresponding to the actual object than with those of other objects. This suggests that the learned di∗d_i^* capture differences in object responses and that the model uses these differences for prediction. For Box Pushing, however, the corresponding di∗d_i^* does not yield the lowest error for some objects, indicating that the representations do not always adequately capture differences between objects.

Real-World Experiments

Figure 5 summarizes the real-world results.

Task success rates for Pouch Placement and Box Pushing

Figure 5. Task success rates for Pouch Placement and Box Pushing on a real robot. “Proposed + retraining” denotes retraining the VAE and dynamics model on the augmented dataset while keeping the policy fixed. Error bars denote approximate 95% Wilson confidence intervals computed from rollouts pooled across three training seeds, treating rollouts as independent. All seeds were evaluated on the same fixed test configurations. Successes per seed: Pouch Placement (of 9), DP 4, 4, 3; Proposed 6, 6, 6. Box Pushing (of 22), DP 9, 11, 12; Proposed 10, 10, 13; Proposed + retraining 13, 13, 17.

For Pouch Placement, the proposed method achieves a success rate of 66.7%, compared with 40.7% for standard DP, an improvement of approximately 26 percentage points. In failed DP rollouts, the policy sometimes regrasped the pouch at a location that did not account for its CoM, even after lifting it and setting it down once.

Failures of the proposed method exhibited a different pattern: when the pouch tilted after lifting, d^\hat{d} sometimes updated too slowly and remained near its initial value corresponding to a centered CoM. The policy then attempted to insert the tilted pouch into the frame without first setting it down to change the grasp location, resulting in failure. Such attempts to insert a tilted pouch were not observed in standard DP rollouts. This suggests that, when visual observations and the estimated physical representation favor different actions, the policy may prioritize the behavior associated with d^\hat{d}.

For Box Pushing, standard DP and the proposed method trained only on demonstrations achieved similar success rates. DP’s successes were unevenly distributed across objects, favoring those with lower indices, and were more common when little change in pushing location was needed. after the initial contact. Before retraining, the proposed method showed less variation in success across objects than DP. However, it sometimes changed the contact location unnecessarily even when the initial push was already at an appropriate location. This behavior was observed when d^\hat{d} took an incorrect value.

We next augmented the original demonstrations with autonomous rollouts of the proposed method on the five training objects, regardless of task success. Keeping the policy weights and the learned physical representations {di∗}\{d_i^*\} fixed, we reinitialized the VAE and dynamics model and retrained both from scratch on the augmented dataset.

After retraining, the success rate increased by approximately 15 percentage points. Pooled across seeds, success rates on seen and unseen CoM locations were similar for each method: 50.0% and 47.2% for DP, 53.3% and 47.2% for the proposed method, and 66.7% and 63.9% after retraining. Figure 6 compares the trajectories of d^\hat{d} before and after retraining.

Figure 6. Online estimates d^\hat{d} for Box Pushing before (left) and after (right) reinitializing and retraining the VAE and dynamics model on the augmented dataset. Colors indicate object types. Dashed horizontal lines denote the learned di∗d_i^* for objects 0–4 (blue, orange, green, red, and purple, respectively).

As shown in Figure 6, the estimates for some objects exhibit smaller deviations and less variability after retraining. The initial-position settings for the box were unchanged between the two evaluations, and the added rollouts were collected under the same initial-position conditions as the evaluation. Improved prediction and estimation under these conditions

are therefore expected from the additional data. This comparison examines how improved physical-property estimation affects behavior and task success while the policy remains fixed. Because retraining uses observation–action sequences regardless of task outcome, failed rollouts can also contribute to model learning.

After retraining, rollouts exhibited fewer unnecessary changes in contact location and more successful episodes. These results suggest that more stable physical-property estimation reduced unnecessary contact changes and contributed to the increase in success rate.

Across both real-world tasks, episodes in which d^\hat{d} took an incorrect value tended to fail. This suggests that d^\hat{d} influences action selection as a policy conditioning input and that accurate estimation is important for generating appropriate actions. The results therefore suggest that task success depends on the predictive accuracy of the dynamics model used for online physical-property estimation.

Simulation Experiments

Table 2 reports the results for Horizontal Lifting. For each training seed, we evaluate 30 rollouts on seen objects and 40 on unseen objects; the total success rate is computed over all 70 rollouts.

MethodSeen ObjectsUnseen ObjectsTotal
DP85.6±5.185.6 \pm5.121.7±5.221.7 \pm5.249.0±0.849.0 \pm0.8
DP (long history)85.6±5.185.6 \pm5.16.7±2.96.7 \pm2.940.5±3.040.5 \pm3.0
Proposed (state-based)96.7±3.396.7 \pm3.330.0±6.630.0 \pm6.658.6±2.958.6 \pm2.9
Proposed (image-based)92.2±3.892.2 \pm3.832.5±2.532.5 \pm2.558.1±0.858.1 \pm0.8

Table 2. Success rates (%) for Horizontal Lifting, reported as mean ±\pm sample standard deviation across three training seeds.

The proposed method achieves a higher mean overall success rate than both standard DP and DP (long history), with either the state-based or the image-based dynamics model. Compared with standard DP, DP (long history) achieves the same success rate on seen objects but a lower rate on unseen objects, indicating that simply extending the observation history does not improve performance in this experiment.

As shown in Figure 4(c-1) and (c-2), both dynamics models achieve the lowest mean prediction error when conditioned on the representation di∗d_i^* corresponding to the actual object.

The online estimates tend to vary more with the image-based dynamics model than with the state-based model, yet the two variants achieve overall success rates.

On unseen objects, both variants outperform the baselines in mean success rate, but their rates remain around 30%. Adapting to CoM locations absent from the demonstrations therefore remains challenging.

Limitations

Our method does not always reliably estimate the physical representation d^\hat{d} from brief interactions. In the real-world experiments, delayed or inaccurate estimates were associated with inappropriate insertion attempts and unnecessary changes in contact location. In simulation, success rates on CoM locations absent from the demonstrations remained around 30%, indicating limited adaptation to unseen physical properties. Effective policy conditioning therefore requires rapid and accurate estimation of d^\hat{d} from observations during task execution. Future work should investigate whether pretrained visual representations and dynamics models can improve prediction accuracy and the stability of online estimation, and whether these improvements lead to better adaptation to unseen CoM locations.

Our experiments primarily varied weight positions in one dimension and used a scalar physical representation di\mathbf{d}_i. We have not evaluated conditions in which the CoM varies in three dimensions or tasks that require switching behaviors according to object mass. Further evaluation with multidimensional physical representations is needed to determine whether the method can learn adaptive behaviors for these variations in physical properties.

Conclusion

We presented an imitation learning method that learns latent representations of physical properties from demonstrations to predict object responses and updates these representations online using prediction errors during task execution. The same demonstrations provide supervision for predicting object responses and learning adaptive actions, including regrasping and changes in contact location. Conditioning the fixed Diffusion Policy on the online estimate allows information acquired through interaction to guide these behaviors. Real-world and simulation experiments showed that the learned representations capture differences in object responses and that conditioning on their online estimates strongly influences action selection and affects adaptation performance. Retraining the VAE and dynamics model on demonstrations augmented with autonomous rollout data further improved task success while keeping the policy fixed. Our evaluation was limited to one-dimensional variation in mass distribution, with Pouch Placement evaluated only on seen configurations, and adaptation to unseen configurations was not strong in either the real-world or the simulation experiments. Nevertheless, these results suggest that online adaptation through latent physical representations can improve imitation-based manipulation of objects whose physical properties cannot be distinguished visually.

References

  1. [1]C. Chi, Z. Xu, S. Feng, E. Cousineau, Y. Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” The International Journal of Robotics Research, vol. 44, no. 10-11, pp. 1684–1704, 2025.
  2. [2]Y. Hou, Z. Liu, C. Chi, E. Cousineau, N. Kuppuswamy, S. Feng, B. Burchfiel, and S. Song, “Adaptive compliance policy: Learning approximate compliance for diffusion guided control,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 4829–4836.arxiv.org/abs/2410.09309
  3. [3]H. Choi, Y. Hou, C. Pan, S. Hong, A. Patel, X. Xu, M. R. Cutkosky, and S. Song, “In-the-wild compliant manipulation with UMI-FT,” in 2026 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2026.arxiv.org/abs/2601.09988
  4. [4]Y. Zhou, Y. Lin, F. Peng, J. Chen, K. Huang, H. Yang, and Z. Yin, “MTIL: Encoding full history with mamba for temporal imitation learning,” IEEE Robotics and Automation Letters, vol. 10, no. 11, pp. 11 761–11 767, 2025.DOI
  5. [5]M. Murooka, S. Nozawa, Y. Kakiuchi, K. Okada, and M. Inaba, “Feasibility evaluation of object manipulation by a humanoid robot based on recursive estimation of the object’s physical properties,” in 2017 IEEE International Conference on Robotics and Automation (ICRA), 2017, pp. 4082–4089.DOI
  6. [6]T. Murooka, M. Hamaya, F. v. Drigalski, K. Tanaka, and Y. Ijiri, “EXI-Net: Explicitly/implicitly conditioned network for multiple environment sim-to-real transfer,” in Proceedings of the 2020 Conference on Robot Learning, vol. 155. PMLR, 2021, pp. 1221–1230.
  7. [7]F. R. Hogan and A. Rodriguez, “Reactive planar non-prehensile manipulation with hybrid model predictive control,” The International Journal of Robotics Research, vol. 39, no. 7, pp. 755–773, 2020.DOI
  8. [8]A. Longhini, M. Moletta, A. Reichlin, M. C. Welle, D. Held, Z. Erickson, and D. Kragic, “EDO-Net: Learning elastic properties of deformable objects from graph dynamics,” in 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 3875–3881.arxiv.org/abs/2209.08996
  9. [9]B. Evans, A. Thankaraj, and L. Pinto, “Context is everything: Implicit identification for dynamics adaptation,” in 2022 International Conference on Robotics and Automation (ICRA), 2022, pp. 2642–2648.arxiv.org/abs/2203.05549
  10. [10]M. Y. Aoyama, J. Moura, J. D. A. Ferrandis, and S. Vijayakumar, “Poke and strike: Learning task-informed exploration policies,” in 9th Annual Conference on Robot Learning, 2025.arxiv.org/abs/2509.00178
  11. [11]M. Y. Aoyama, J. Moura, N. Saito, and S. Vijayakumar, “Few-shot learning of force-based motions from demonstration through pre-training of haptic representation,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024, pp. 12 839–12 845.arxiv.org/abs/2309.04640
  12. [12]N. Saito, T. Ogata, S. Funabashi, H. Mori, and S. Sugano, “How to select and use tools? : Active perception of target objects using multimodal deep learning,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 2517–2524, 2021.DOI
  13. [13]M. S. Mark, M. Attarian, C. Fu, D. Dwibedi, J. Liang, D. Shah, and A. Kumar, “Long-Context Robot Imitation Learning by Focusing on Key History Frames,” in Proceedings of Robotics: Science and Systems (RSS), Sydney, Australia, 2026.
  14. [14]H. Ichiwara, H. Ito, K. Yamamoto, H. Mori, and T. Ogata, “Spatial attention point network for deep-learning-based robust autonomous robot motion generation,” arXiv preprint arXiv:2103.01598, 2021.arxiv.org/abs/2103.01598
  15. [15]T. An, J. Jia, G. Li, J. Li, C. Zhou, P. Liu, B. Lyu, J. Bai, X. Guo, G. Li, and J. Yang, “Feedback world model enables precise guidance of diffusion policy,” arXiv preprint arXiv:2605.15705, 2026.arxiv.org/abs/2605.15705
  16. [16]J. Lyu, Z. Li, X. Shi, C. Xu, Y. Wang, and H. Wang, “DyWA: Dynamics-adaptive world action model for generalizable non-prehensile manipulation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 11 058–11 068.arxiv.org/abs/2503.16806
  17. [17]G. Yuan, Q. Qiao, J. Zhang, and D. Xu, “AdaWorldPolicy: World-model-driven diffusion policy with online adaptive learning for robotic manipulation,” arXiv preprint arXiv:2602.0057, 2026.arxiv.org/abs/2602.20057
  18. [18]A. Ali, J. Bai, M. Bala, Y. Balaji, A. Blakeman, T. Cai, et al., “World simulation with video foundation models for physical AI,” arXiv preprint arXiv:2511.00062, 2025.arxiv.org/abs/2511.00062
  19. [19]D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” in International Conference on Learning Representations (ICLR), 2014.arxiv.org/abs/1312.6114
  20. [20]M. Murooka, T. Motoda, R. Nakajo, H. Oh, K. Makihara, K. Shirai, T. Ogata, and Y. Domae, “RoboManipBaselines: A unified framework for imitation learning in robotic manipulation across real and simulation environments,” IEEE Access, vol. 14, pp. 97 896–97 909, 2026.DOI
  21. [21]D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in International Conference on Learning Representations (ICLR), 2015.arxiv.org/abs/1412.6980

Paper details

vlnc:W_sep34pyh
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents