WorksComputer Science
Learning Object-Centric Motion Priors from Human for Robotic Dexterous Manipulation
Abstract
Manipulating diverse objects with multi-fingered dexterous hands is challenging due to the high dimensionality and complex dynamics. Human-Object Interaction (HOI) datasets provide rich knowledge about task information and embodied interactions. Instead of solely imitating the human demonstrations, our method learns to holistically predict future hand-object states by leveraging these datasets. The predicted future states of the object can serve as a general-purpose reward term for reinforcement learning, reducing reliance on task-specific reward engineering and enhancing generalization across tasks. We conduct extensive experiments on three manipulation tasks in simulation and the real world. Our approach outperforms existing SOTA methods in both success rate and generalizability on novel objects. Furthermore, we validate the cross-embodiment compatibility of our methods by successfully deploying the skills on different robot hands.
Introduction
Achieving human-level dexterous manipulation is a critical but challenging goal for robotics, primarily due to the high degrees of freedom in robot hands and complex contact dynamics. Previous methods, such as model-based trajectory optimization [24, 14], struggle with these dynamic complexities. Meanwhile, model-free reinforcement learning (RL) [31, 12, 1, 7, 43] is often inefficient, demanding extensive exploration and task-specific reward engineering. While imitation learning from robot data [10] presents a viable solution, the required large-scale data collection is frequently impractical for research labs [42, 3]. In contrast, the vast amount of available human data, which implicitly encodes rich manipulation experience, offers a promising and scalable alternative for learning dexterous manipulation.
What should we learn from the human demonstration? To answer this, let’s look at how humans learn. Children learn to interact with the physical world using the following two-stage learning paradigm. The first stage involves forming an abstract task concept through watching others play. In the second stage, this conceptual knowledge is grounded and refined through physical experimentation by themselves, facilitating the development of operational competence. Similarly, prior literature [28, 9, 36, 8] (Li et al. 2025) investigates a similar two-stage pipeline for learning dexterous manipulation skills. Most of them either focus solely on imitating human motions or designing task-specific rewards, then testing on only one task, which makes it hard to draw a general conclusion from that. Some of these works overlook an important fact: while humans are manipulating objects, they are also subconsciously predicting the future state changes of the objects. This ability to predict the future state of objects not only helps humans understand tasks but also enables them to perform more general operations on different tasks. Inspired by that, our observation is that considering human-object as a whole and learning human manipulation priors from hand-object interactions provides more general and efficient task-level guidance for the second-stage low-level policy learning. Instead of using human-designed task-specific rewards like lifting rewards in previous work [9, 36], we believe that human-object interaction priors encourage robots to manipulate the object in the way described in the dataset, which is task-agnostic to different tasks.
Specifically, we propose a pipeline to auto-regressively predict future hand-object states holistically. And we use the predicted hand and object states to guide the low-level policy learning in different tasks without task-specific reward design in low-level reinforcement learning in a physical simulation. Then we perform zero-shot sim-to-real skill transfer to the real-world robots. Our contributions are as follows:
We propose an efficient learning framework to utilize human-object interaction priors for robot skill learning, without the need for real-robot data.
Predicting hand-object priors holistically and properly utilizing the predicted object states can not only facilitate reinforcement learning by alleviating the heavy reliance on task-specific reward design, but also enable our pipeline to be more general to various task settings.
Copyright © 2026, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
We conducted extensive real-robot experiments on various tasks, embodiments, and object settings, outperforming existing SOTA methods in all three challenging tasks.
Related Work
Dexterous Manipulation
Dexterous manipulation is complicated by the high dimensionality of the end effector. This complexity poses a problem for traditional trajectory optimization techniques [34, 21] (Wu, Guo, and Liu 2022), as they require precise dynamic models that are impractical for real-world applications. Learning-based methods address this by learning an observation-to-action policy directly from data, but they introduce new challenges. Reinforcement learning approaches [48, 43] (Qi et al. 2023) can be sample-inefficient and depend on meticulous reward engineering. On the other hand, imitation learning methods [2, 10], which use data from teleoperation [29, 42], are often constrained by the expense of data collection and the quality of the expert demonstrations. Vision-Language-Action methods [23, 50] aim at utilizing pretrained language models to boost the network performance, while only demonstrating tasks not harder than pick-and-place. Grasp synthesis methods [41] (Ye et al. 2025) are mainly tested in simulation, and the potential in more challenging tasks beyond pick-and-place is not fully investigated. Our methods combine the merits of using human data and scaled-up simulation, enabling real-robot dexterous manipulation with high success rates.
Learning Dexterous Manipulation from Human
Internet human video data contains abundant information about human experiences. Previous research [20, 22] (Ma et al. 2022) seeks to learn general visual representations from internet-scale human video data to guide robot learning. However, those methods only showcase results on a 1-DoF robot gripper since the features trained on 2D human videos lack the 3D information necessary for learning high-dimensional skills like dexterous manipulation. Different from internet video human data, human-object interaction (HOI) data [6, 11, 17] are usually collected by multi-view RGBD cameras, with extracted and labeled 3D human pose estimations and 6-DoF object pose estimations, which contain abundant 3D information necessary for learning dexterous manipulation. Given the 3D motions of the human hand, an intuitive idea is to directly transfer human motions to robot actions. Qin et al. [29] proposed a retargeting system based on inverse kinematics to map the human hand motions to robot hand states and actions. Qin et al. [28]; ); Liu et al. [16] further prove that the retargeted trajectories can be used to train imitation learning policies in different tasks.
Following them, Chen et al. [9]; ); Zhou et al. [51]; ); Shi et al. [36] apply trajectory augmentation and train a state-based policy based on that. Different from direct mapping, Chen et al. [8] proposes to train a human motion prediction network given a target object trajectory, then augment and refine it in physical simulation using Reinforcement Learning (RL). However, defining target object trajectories in open world requires extra efforts. Li et al. (2025) train a 2-stage process imitating human, but their input is fixed object initial poses with object pose randomization as 0, constraining open-world applications.
To address this, Zhao et al. [49] introduces human-in-the-loop adjustments in generating reference trajectories, but it requires human labor. Singh et al. [37] requires real-world robot demonstrations or RL to finetune the policy. Most of them focus on a single task and can hardly generalize to novel object configurations. We demonstrate our applicability in various challenging tasks under different object configurations.
Methods
Our proposed pipeline is illustrated in Figure 1. The process begins by extracting observations from the initial state of a real-world robot and object scene. These observations are then fed into a motion predictor to generate a reference trajectory. Using this trajectory as guidance, we employ a reinforcement learning framework to learn the desired skill policy. Finally, this learned policy is transferred directly to the physical robot using a zero-shot sim-to-real approach.

Figure 1. Our method begins by extracting observations from the initial robot-object configuration. These observations are input into a motion predictor to generate a reference trajectory. Guided by this trajectory, we employ reinforcement learning to train the target policy. Finally, the learned policy is deployed directly to the physical robot via zero-shot Sim-to-Real transfer. Notably, since the learned priors are general to hand morphologies, our framework is adaptable to various dexterous hands.
Hand-Object Poses Prediction
Figure 2 is our hand-object-interaction (HOI) motion predictor. We use Dex-YCB [6] and ARCTIC [11] as our datasets. Both of them have well-labeled 6DoF hand-object pose pairs for each timestamp, noted as which is represented by MANO [32] hand poses and which are the quaternion and the translation of a object pose. We use a coordinate system with the geometric center of the object at as the origin. For articulated objects like a laptop and a box, we record the relative pose of the upper part to the lower part (base). For each motion sequence, we first use farthest point sampling (FPS) (Qi et al. 2017b) to extract a 3D point cloud consisting of 100 points from the object mesh, which is later processed by a PointNet (Qi et al. 2017a) to get a feature about the object shape information. We use a 6-layer GPT-2 transformer [30] to predict future hand-object pose pairs using the history observations in an auto-regressive manner. The history observations are concatenated with the PointNet feature to feed into the transformer, and the training is performed as supervised learning:

Figure 2. Illustration of components in our HOI motion predictor.
We use in our experiments. The predicted observations are taken as input again for the next prediction until the length of the predicted sequence reach the sequence length in datasets. We utilize a single, unified model for all datasets, adapting the input as necessary. For the bimanual ARCTIC dataset [11], both hands are directly fed into the network. To accommodate data from Dex-YCB [6], we apply zero-padding to align input dimensions. In practice, we use , . is larger than since the dimensions that belong to the object in the input have a lower ratio than the hand.
During inference on real robots, we regard robot hands as subcollections of human hands and directly apply MANO estimation on them. We extract 3D coordinates of robot joints (semantically corresponding to a subset of 21 joints () in MANO), then modify FrankMocap [33] by replacing
the original optimization goal
with
using robot proprioception and forward kinematics to get 3d joint positions , instead of projecting () the 3d joints to 2D and optimize in 2D. During optimization, shape parameters will be adaptively adjusted for different limb lengths, making it compatible with different types of robot hands with distinct sizes. During inference time, the object point cloud is attained by applying farthest point sampling (FPS) (Qi et al. 2017b) to the object mesh, which is placed under 6DoF object pose estimation by model-based tracking in FoundationPose [44].
Robot-Object Trajectories Extraction
The predicted hand motions are further converted to motions of a robotic hand using optimization-based hand motion retargeting. Following AnyTeleop [29], the loss function minimizes the distance between the robot and human fingertip positions:
subject to:
where is the robot’s joint configuration at time , and is the corresponding robot fingertip position calculated via forward kinematics to match the human’s fingertip position, . The robot’s joints are constrained by lower () and upper () bounds. To ensure temporal smoothness, an additional penalty term is weighted by . We solve this optimization problem using the NLopt solver [29]. In practice, we set , with and .
Trajectory-Guided Reinforcement Learning
Due to the embodiment gap, the retargeted robot trajectories are imperfect and require further refinement. However, these predicted trajectories provide crucial guidance for learning robotic skills via reinforcement learning (RL). We represent the task using the Markov Decision Process (MDP) framework , where the components correspond to the state space (), action space (), transition model (), reward function (), and discount factor (), respectively.
We train a state-based skill policy, . The policy’s input state is composed of robot joint positions and 6D object poses. The output action consists of delta end-effector poses and delta hand joint positions.
At each time step of the interaction, the environment transitions to based on and provides a reward . During control, the target end-effector pose is converted to arm joint positions via inverse kinematics (IK) using Pinocchio [5]. Each policy from a set is optimized to maximize the expected cumulative reward, conditioned on a reference trajectory from a human demonstration with task horizon :
Our reward function consists of four items:
The first term, robot following reward , encourages the robot to follow reference hand motions.
The first component encourage the robot hand to follow reference robot hand joint positions, where and are robot hand joint positions at timestep and reference robot hand joint positions at corresponding timestep . The second one and the third one constrains robot End-Effector (EE) pose where are robot EE position and computes the angular distance between robot EE orientation with reference .
The second term is an object following reward. Instead of using human-designed task-specific reward like lifting reward in previous work [9, 36], we utilize the predicted object states as guiding signals to encourage the robots to manipulate the object in the way that is in the dataset, which is more generalizable to different tasks:
where is the Logarithmic Map which converts a quaternion to its axis-angle representation, and denotes the Hamilton product.
We encourage finger exploration through , which gives a reward only when the thumb finger and at least one other finger contact with the object. The contact information is derived from the collision check in SAPIEN simulation [46]. In obstacle avoidance tasks, we further add a large penalty when any part of the hand or the arm contact the obstacle, forcing the policy to explore a plausible route to avoid obstacles.
We give a large reward when task is finished with success. We use PPO [35] for state-based policy learning with delta end-effector pose control, where we randomize initial object position within cm in -plane and rotation within around the -axis.
We carefully tuned the reward weights. In our experiments, we use where upon task success and otherwise. In YCB grasping with obstacle task, the weights of the contact reward are modified to be when there is any contact between the robot hand links or the robot arm links with the obstacle.
Sim-to-Real Transfer
We have several designs to overcome the control gap and perception gap between the simulation and the real world, enabling zero-shot Sim-to-Real transfer. Before training the reinforcement learning policy, we perform system identification (Ma et al. 2025) to mitigate the control gap. Specifically, we control the simulated robot and the real robot to move in the same direction in the same time interval and minimize the error between the two by adjusting the parameters of the simulated robot, including PID parameters and force limits for each joint motors. For different robot hand embodiments, we tune the control parameters for each to minimize the control gap when switched to another embodiment. Besides, we adopt domain randomization [39] by artificially adding random noise to the observations as well as adding simulation parameter randomizations to friction, object scale, and object weights in the simulation to improve the robustness of the trained policies to the real-world noise.
Experiments
Experiment Setting
As shown in Figure 3, our experiments cover the following 4 settings:

Figure 3. Qualitative results from real-world experiments.
YCB object grasping: the success is defined when the robot hand successfully grasps a target YCB object placed on the platform and lifts it up to 10 cm above its initial height.
Articulated object manipulation: the task success is defined as the hinged object being rotated more than 30 degrees along the axis.
Grasping with obstacles: the distinction between this task and a standard grasping task is the deliberate introduction of an obstacle along the robot’s path towards the target object. An additional constraint is that any part of the robotic arm and dexterous hand must not collide with the obstacle at any point during the entire process of grasping the target object.
Cross-embodiment test: we replace the right-side robot hand with a different model. We then reuse the motion predictor to generate human-object interaction trajectories and apply retargeting adapted to the specific robot configuration. Finally, we perform trajectory-guided reinforcement learning to learn the skills, using identical reward terms and parameters for different robot hands.
Abbreviation:
In the following paragraphs, the terms seen and unseen (as applied to objects or settings) refer to whether that configuration was used to train our HOI motion predictor and the corresponding motion extractors for the baseline methods.
Simulation Configuration
For simulation, we employ SAPIEN3 [46] and OpenAI-Gymnasium [40] to enable GPU-parallel training and rendering, we have 1024 environments running in parallel on single NVIDIA RTX 4090 GPU (24GB) and i7-14700K CPU, with average steps per second at 10000, and training for 1M steps for the policies to converge. Parameters used in training the PPO reinforcement learning policies are shown in Table 1.
| Parameter | Value |
| Maximum steps | |
| Bootstrap at done | True |
| Discount factor () | 0.95 |
| GAE parameter () | 0.9 |
| Learning rate | |
| Parallel environments | 1024 |
Table 1. Training parameters of PPO policies.
Real-World Robot Setup
Our bimanual robot platform consists of two xArm-7 arms, each mounted with a PSYONIC Ability hand, a ROBOTERA XHand1 or an Inspire robot hand. Object pose estimation is performed using model-based tracking with FoundationPose [44] based on RGB-D data from a RealSense D435i camera, which was calibrated using EasyHec++ [13].
Baselines
For YCB object grasping tasks, we compare our results with existing SOTA methods. They are ViViDex [9], AdaDexGrasp [36], ManipTrans [15], HOP [37] using RL for finetuning, and PPO w/o following, which substitutes following reward and using a reaching reward:
where is the translation of the robot TCP pose, which refers to the pose of the palm of the robot hand, and is the global translation of the object. When there is a novel object configuration, ViViDex [9] and AdaDexGrasp [36] use existing reference trajectories in the dataset. For ARCTIC articulated object manipulation, we compare with Obj-Dex [8], ManipTrans [15], and PPO w/o following. For grasp with obstacle tasks, we also compare ours with HOP [37]. We use the PSYONIC Ability hand, ROBOTERA XHand1 or Inspire robot hand for the cross-embodiment test and compare the results. All the simulation experiments are tested with 5 seeds, and we report the average success rates in policy evaluation using 100 trials each after the convergence of RL training. All the real-world robot experiments are repeated 20 times to get success rates.
YCB Object Grasping
We test the methods on different YCB objects (5 objects in Dex-YCB datasets [6], 5 objects from YCB datasets [4] which are not included in Dex-YCB datasets) and 5 different initial object poses with randomized position and rotation inside the size of an A4 paper. The task success is defined as the object being grasped and lifted up over 10 cm. Success rates in simulation and real-world robots are reported in Table 2. ViViDex and AdaDexGrasp struggle to handle non-Dex-YCB objects. ManipTrans has difficulty dealing with novel initial object poses. Our method demonstrates consistent performance across different task configurations.
| Method | Simulation | Real world | ||||||
| DexYCB | Non-DexYCB | Novel poses | Avg. | DexYCB | Non-DexYCB | Novel poses | Avg. | |
| ViViDex [9] | 0.87 | 0.37 | 0.75 | 0.66 | 0.35 | 0.40 | 0.25 | 0.33 |
| AdaDexGrasp [36] | 0.82 | 0.21 | 0.64 | 0.56 | 0.80 | 0.20 | 0.65 | 0.55 |
| ManipTrans (Li et al. 2025) | 0.61 | 0.28 | 0.03 | 0.31 | 0.60 | 0.30 | 0.05 | 0.32 |
| HOP [37] | 0.63 | 0.59 | 0.54 | 0.59 | 0.60 | 0.45 | 0.40 | 0.48 |
| PPO w/o following [35] | 0.37 | 0.42 | 0.27 | 0.35 | 0.30 | 0.40 | 0.20 | 0.30 |
| Ours | 0.90 | 0.86 | 0.75 | 0.84 | 0.85 | 0.80 | 0.65 | 0.77 |
Table 2. Success rates for YCB object grasping in simulation and on real robots. The novel-pose condition uses DexYCB objects.
Articulated Object Manipulation
We test the performance of the methods on 3 objects in ARCTIC [11] dataset, which are wooden laptop, wooden box, mixer, and 3 objects in our daily life: real laptop, larger box, toy coffee machine. The task success is defined as the hinged object rotating more than 30 degrees along the axis. We report success rates in simulation and real-world robots in Table 3, where (Li et al. 2025), [8] fail to handle the novel daily objects, while we still observe success in ours in simulation and real robots.
Table 3. Success rates for articulated object manipulation in simulation and on real robots. ARCTIC and Daily denote objects from the ARCTIC dataset and daily-life objects, respectively.
Grasping with Obstacle
The distinction between this task and a standard grasping task is the deliberate introduction of an obstacle along the robot’s path towards the target object. While the obstacle’s mesh is known, its type and position are variable. An additional constraint is that the robotic arm and dexterous hand must not collide with the obstacle at any point during the process of grasping the target object; otherwise, the task is considered a failure. We select 5 objects in the DexYCB datasets and 5 3D-printed object to make up our obstacle library and choose one obstacle from it each time. In this experiment, a large penalty item is introduced to each method when any part of the hand or the arm contacts the obstacle during RL training, forcing the policy to explore a plausible route to avoid obstacles. Besides, we selectively filter out all time frames where the dexterous hand meshes from reference trajectories collide with obstacles using cuRobo [38] motion planning, disabling the following reward in these frames for ours and baselines. We report success rates in simulation and real-world robots in Table 4.
| Method | Simulation | Real world | ||||
| DexYCB obstacle | 3D-printed obstacle | Avg. | DexYCB obstacle | 3D-printed obstacle | Avg. | |
| ViViDex [9] | 0.17 | 0.38 | 0.28 | 0.15 | 0.40 | 0.28 |
| AdaDexGrasp [36] | 0.55 | 0.67 | 0.61 | 0.55 | 0.60 | 0.58 |
| ManipTrans [15] | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| HOP [37] | 0.32 | 0.41 | 0.37 | 0.25 | 0.40 | 0.33 |
| PPO w/o following [35] | 0.58 | 0.63 | 0.61 | 0.50 | 0.55 | 0.53 |
| Ours | 0.57 | 0.74 | 0.66 | 0.55 | 0.70 | 0.63 |
Table 4. Success rates for grasping with obstacles in simulation and on real robots. Real-world collisions are detected using xArm’s built-in collision detector.
Cross-Embodiment Test
To validate the generalizability of our method across different robotic platforms, we replace the right-side robot hand in the robotic setting with another robot hand. We then reexecute the retargeting and reinforcement learning algorithms based on the trajectories generated by our motion predictor and evaluated the performance on various tasks in the real-world robots. See Figure 4, our methods perform well across different embodiments of dexterous hands, showing consistent performance in different tasks.
Figure 4. Cross-embodiment test on 3 robot hands.
Ablation Study on the Reward Design
To validate the effectiveness of our reward design, we conduct ablation studies by: 1) removing the object following term , and 2) removing the contact reward . We evaluate these on the YCB object grasping task using 5 objects (5 seeds each) from the DexYCB dataset and report the average success rates in Figure 5. We observe that when is removed, sample efficiency decreases dramatically. When is removed, training becomes unstable, resulting in lower success rates. These results demonstrate that the object following reward is critical for ensuring stable and efficient policy learning.

Figure 5. Learning curves for reward ablation (Solid lines: the mean over all seeds. Shaded areas: standard deviation).
Conclusion
We propose a hierarchical framework for learning dexterous manipulation from object-centric human motion priors to achieve generalizable manipulation, enabling zero-shot Sim-to-Real transfer. Our method outperforms existing baselines in YCB object grasping, articulated object manipulation, and grasping with obstacles in terms of success rates and generalizations to unseen objects. We demonstrate the importance of object following reward through ablation studies. We also validate the cross-embodiment potentials of our methods.
References
- [1]Akkaya, I.; Andrychowicz, M.; Chociej, M.; Litwin, M.; McGrew, B.; Petron, A.; Paino, A.; Plappert, M.; Powell, G.; Ribas, R.; et al. 2019. Solving rubik’s cube with a robot hand. arXiv preprint arXiv:1910.07113.arxiv.org/abs/1910.07113
- [2]Bain, M.; and Sammut, C. 1995. A Framework for Behavioural Cloning. In Machine intelligence 15, 103–129.
- [3]Black, K.; Brown, N.; Driess, D.; Esmail, A.; Equi, M.; Finn, C.; Fusai, N.; Groom, L.; Hausman, K.; Ichter, B.; et al. 2024. π0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164.arxiv.org/abs/2410.24164
- [4]Calli, B.; Walsman, A.; Singh, A.; Srinivasa, S.; Abbeel, P.; and Dollar, A. M. 2015. Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols. arXiv preprint arXiv:1502.03143.DOI
- [5]Carpentier, J.; Saurel, G.; Buondonno, G.; Mirabel, J.; Lamiraux, F.; Stasse, O.; and Mansard, N. 2019. The Pinocchio C++ library – A fast and flexible implementation of rigid body dynamics algorithms and their analytical derivatives. In IEEE International Symposium on System Integrations (SII).
- [6]Chao, Y.-W.; Yang, W.; Xiang, Y.; Molchanov, P.; Handa, A.; Tremblay, J.; Narang, Y. S.; Van Wyk, K.; Iqbal, U.; Birchfield, S.; et al. 2021. DexYCB: A benchmark for capturing hand grasping of objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9044–9053.arxiv.org/abs/2104.04631
- [7]Chen, T.; Tippur, M.; Wu, S.; Kumar, V.; Adelson, E.; and Agrawal, P. 2023. Visual dexterity: In-hand reorientation of novel and complex object shapes. Science Robotics, 8(84): eadc9244.DOI
- [8]Chen, Y.; Wang, C.; Yang, Y.; and Liu, C. K. 2025a. Object-Centric Dexterous Manipulation from Human Motion Data. In Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, 3828–3846. PMLR.arxiv.org/abs/2411.04005
- [9]Chen, Z.; Chen, S.; Arlaud, E.; Laptev, I.; and Schmid, C. 2025b. ViViDex: Learning Vision-based Dexterous Manipulation from Human Videos. In 2025 IEEE International Conference on Robotics and Automation (ICRA).arxiv.org/abs/2404.15709
- [10]Chi, C.; Xu, Z.; Feng, S.; Cousineau, E.; Du, Y.; Burchfiel, B.; Tedrake, R.; and Song, S. 2023. Diffusion policy: Visuomotor policy learning via action diffusion. The International Journal of Robotics Research, 02783649241273668.
- [11]Fan, Z.; Taheri, O.; Tzionas, D.; Kocabas, M.; Kaufmann, M.; Black, M. J.; and Hilliges, O. 2023. ARCTIC: A dataset for dexterous bimanual hand-object manipulation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 12943–12954.arxiv.org/abs/2204.13662
- [12]Gupta, A.; Eppner, C.; Levine, S.; and Abbeel, P. 2016. Learning dexterous manipulation for a soft robotic hand from human demonstrations. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 3786–3793. IEEE.DOI
- [13]Hong, Z.; Zheng, K.; and Chen, L. 2024. EasyHeC++: Fully Automatic Hand-Eye Calibration with Pretrained Image Models. International Conference on Intelligent Robots and Systems (IROS).arxiv.org/abs/2410.09293
- [14]Jin, W. 2024. Complementarity-free multi-contact modeling and optimization for dexterous manipulation. arXiv preprint arXiv:2408.07855.arxiv.org/abs/2408.07855
- [15]Li, K.; Li, P.; Liu, T.; Li, Y.; and Huang, S. 2025. Manip-Trans: Efficient Dexterous Bimanual Manipulation Transfer via Residual Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 6991–7003.DOI
- [16]Liu, X.; Adaliabieke, J.; Han, Q.; Qin, Y.; and Yi, L. 2025. DexTrack: Towards Generalizable Neural Tracking Control for Dexterous Manipulation from Human References. arXiv preprint arXiv:2502.09614.arxiv.org/abs/2502.09614
- [17]Liu, Y.; Liu, Y.; Jiang, C.; Lyu, K.; Wan, W.; Shen, H.; Liang, B.; Fu, Z.; Wang, H.; and Yi, L. 2022. HOI4D: A 4d egocentric dataset for category-level human-object interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 21013–21022.arxiv.org/abs/2203.01577
- [20]Mendonca, R.; Bahl, S.; and Pathak, D. 2023. Structured World Models from Human Videos. In Robotics: Science and Systems XIX.arxiv.org/abs/2308.10901
- [21]Mordatch, I.; Popović, Z.; and Todorov, E. 2012. Contact-invariant optimization for hand manipulation. In Proceedings of the ACM SIGGRAPH/Eurographics symposium on computer animation, 137–144.DOI
- [22]Nair, S.; Rajeswaran, A.; Kumar, V.; Finn, C.; and Gupta, A. 2022. R3M: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601.arxiv.org/abs/2203.12601
- [23]Pan, C.; Junge, K.; and Hughes, J. 2024. Vision-language-action model and diffusion policy switching enables dexterous control of an anthropomorphic hand. arXiv preprint arXiv:2410.14022.
- [24]Pang, T.; Suh, H. T.; Yang, L.; and Tedrake, R. 2023. Global planning for contact-rich manipulation via local smoothing of quasi-dynamic contact models. IEEE Transactions on robotics, 39(6): 4691–4711.arxiv.org/abs/2206.10787
- [28]Qin, Y.; Wu, Y.-H.; Liu, S.; Jiang, H.; Yang, R.; Fu, Y.; and Wang, X. 2022. DexMV: Imitation learning for dexterous manipulation from human videos. In European Conference on Computer Vision, 570–587. Springer.arxiv.org/abs/2108.05877
- [29]Qin, Y.; Yang, W.; Huang, B.; Van Wyk, K.; Su, H.; Wang, X.; Chao, Y.-W.; and Fox, D. 2023. AnyTeleop: A general vision-based dexterous robot arm-hand teleoperation system. arXiv preprint arXiv:2307.04577.arxiv.org/abs/2307.04577
- [30]Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; and Sutskever, I. 2019. Language Models are Unsupervised Multitask Learners. OpenAI technical report.
- [31]Rajeswaran, A.; Kumar, V.; Gupta, A.; Vezzani, G.; Schulman, J.; Todorov, E.; and Levine, S. 2017. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087.arxiv.org/abs/1709.10087
- [32]Romero, J.; Tzionas, D.; and Black, M. J. 2017. Embodied Hands: Modeling and Capturing Hands and Bodies Together. ACM Transactions on Graphics, 36(6): 245:1–245:17.DOI
- [33]Rong, Y.; Shiratori, T.; and Joo, H. 2020. FrankMocap: Fast monocular 3d hand and body motion capture by regression and integration. arXiv preprint arXiv:2008.08324.arxiv.org/abs/2008.08324
- [34]Rus, D. 1999. In-hand dexterous manipulation of piecewise-smooth 3-d objects. The International Journal of Robotics Research, 18(4): 355–381.DOI
- [35]Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.arxiv.org/abs/1707.06347
- [36]Shi, L.; Liu, Y.; Zeng, L.; Ai, B.; Hong, Z.; and Su, H. 2025. Learning Adaptive Dexterous Grasping from Single Demonstrations. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).arxiv.org/abs/2503.20208
- [37]Singh, H. G.; Loquercio, A.; Sferrazza, C.; Wu, J.; Qi, H.; Abbeel, P.; and Malik, J. 2025. Hand-Object Interaction Pre-training from Videos. In 2025 IEEE International Conference on Robotics and Automation (ICRA).
- [38]Sundaralingam, B.; Hari, S. K. S.; Fishman, A.; Garrett, C.; Van Wyk, K.; Blukis, V.; Millane, A.; Oleynikova, H.; Handa, A.; Ramos, F.; Ratliff, N.; and Fox, D. 2023. cuRobo: Parallelized Collision-Free Minimum-Jerk Robot Motion Generation. arXiv preprint arXiv:2310.17274.arxiv.org/abs/2310.17274
- [39]Tobin, J.; Fong, R.; Ray, A.; Schneider, J.; Zaremba, W.; and Abbeel, P. 2017. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), 23–30. IEEE.arxiv.org/abs/1703.06907
- [40]Towers, M.; Kwiatkowski, A.; Terry, J.; Balis, J. U.; De Cola, G.; Deleu, T.; Goulao, M.; Kallinteris, A.; Krim-mel, M.; KG, A.; et al. 2024. Gymnasium: A Standard Interface for Reinforcement Learning Environments. arXiv preprint arXiv:2407.17032.arxiv.org/abs/2407.17032
- [41]Wan, W.; Geng, H.; Liu, Y.; Shan, Z.; Yang, Y.; Yi, L.; and Wang, H. 2023. UniDexGrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalist-specialist learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3891–3902.arxiv.org/abs/2304.00464
- [42]Wang, C.; Shi, H.; Wang, W.; Zhang, R.; Fei-Fei, L.; and Liu, C. K. 2024a. DexCap: Scalable and portable mocap data collection system for dexterous manipulation. arXiv preprint arXiv:2403.07788.arxiv.org/abs/2403.07788
- [43]Wang, J.; Yuan, Y.; Che, H.; Qi, H.; Ma, Y.; Malik, J.; and Wang, X. 2024b. Lessons from Learning to Spin “Pens”. arXiv preprint arXiv:2407.18902.
- [44]Wen, B.; Yang, W.; Kautz, J.; and Birchfield, S. 2024. FoundationPose: Unified 6d pose estimation and tracking of novel objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 17868–17879.arxiv.org/abs/2312.08344
- [46]Xiang, F.; Qin, Y.; Mo, K.; Xia, Y.; Zhu, H.; Liu, F.; Liu, M.; Jiang, H.; Yuan, Y.; Wang, H.; et al. 2020. SAPIEN: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 11097–11107.arxiv.org/abs/2003.08515
- [48]Yuan, Y.; Che, H.; Qin, Y.; Huang, B.; Yin, Z.-H.; Lee, K.-W.; Wu, Y.; Lim, S.-C.; and Wang, X. 2024. Robot synesthesia: In-hand manipulation with visuotactile sensing. In 2024 IEEE International Conference on Robotics and Automation (ICRA), 6558–6565. IEEE.arxiv.org/abs/2312.01853
- [49]Zhao, S.; Zhu, X.; Chen, Y.; Li, C.; Zhang, X.; Ding, M.; and Tomizuka, M. 2024. DexH2R: Task-oriented Dexterous Manipulation from Human to Robots. arXiv preprint arXiv:2411.04428.arxiv.org/abs/2411.04428
- [50]Zhong, Y.; Huang, X.; Li, R.; Zhang, C.; Liang, Y.; Yang, Y.; and Chen, Y. 2025. DexGraspVLA: A vision-language-action framework towards general dexterous grasping. arXiv preprint arXiv:2502.20900.arxiv.org/abs/2502.20900
- [51]Zhou, B.; Yuan, H.; Fu, Y.; and Lu, Z. 2024. Learning diverse bimanual dexterous manipulation skills from human demonstrations. arXiv preprint arXiv:2410.02477.arxiv.org/abs/2410.02477
