DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations

Introduction

Dexterous manipulation supports household assistance, industrial assembly, and assistive care. Unlike gripper-based manipulation, it requires precise coordination of finger motions and changing hand–object contacts to grasp, reorient, and reposition objects [28, 2, 8, 33]. Generalizing across objects and behaviors is challenging, especially when human demonstrations must be retargeted to robot hands with different kinematics [26, 19, 38]. Simulation-ready objects [4, 5, 6] and synthetic trajectories [12, 40] enable large-scale interaction data, but may not fully capture natural human manipulation. Recent trackers therefore learn from human hand–object motion [19, 38, 1], yet remain largely limited to single-hand tasks. Many everyday tasks instead require asymmetric bimanual coordination: one hand stabilizes an object while the other opens, rotates, or repositions it [9, 41, 43]. This motivates a generalist bimanual reference tracker that learns from diverse human motion to control different objects and tasks with one policy.

Developing such a tracker poses two challenges. First, public datasets include bimanual manipulation [9, 41, 20, 10], but fully tracked hand–object sequences with sustained two-hand contact remain scarce, limiting the available coordination patterns. Second, a shared policy must learn trajectories with changing hand roles and contact timing. Hand-pose accuracy alone does not ensure successful object manipulation, while requiring both hands to satisfy contact conditions can mask one hand’s progress. Unequal clip difficulty also creates advantage offsets that can weaken updates for useful actions in harder sequences. Shared training therefore needs feedback grounded in object motion and each hand’s reference contacts, alongside advantages that preserve differences among transitions within each clip.

We address these challenges with DONIC, a generalist bimanual reference tracker, and BiHOI, a multi-view hand–object interaction dataset. BiHOI contains 1,575 sequences spanning 229 objects and 14.77 hours, captured from 32 calibrated RGB views. It covers symmetric manipulation and asymmetric coordination. We reconstruct both hands and the object, then retarget hand motion to robot references paired with the object trajectory. Combined with public references, BiHOI broadens the training objects and coordination patterns. DONIC learns one policy for both hands through proximal policy optimization (PPO) [31] and behavioral cloning (BC) on verified demonstrations. Trajectory-specific policies optimize screened references, while successful shared-policy executions are verified and reused. At inference, one checkpoint tracks new references without further training. Our object-centered per-hand reward maintains object-motion feedback and supervises each hand’s reference contacts, including asynchronous engagement. Our Clip-Centered Advantages (CCA) subtract each clip’s mean estimated advantage before PPO updates, removing clip-dependent offsets while preserving within-clip differences. On held-out trajectories and objects from public datasets, DONIC exceeds the stronger bimanual baseline by 77.36 and 45.73 percentage points in success rate, respectively.

Our contributions are:

• To our knowledge, DONIC is the first bimanual reference tracker jointly trained across operation categories and human motion datasets; a single checkpoint is evaluated on unseen trajectories and objects without fine-tuning.

• We introduce BiHOI, a multi-view hand-object interaction dataset emphasizing coordinated bimanual manipulation, and reconstruct its recordings into robot motion references.

• We propose an object-centered per-hand reward and CCA to support joint PPO training across bimanual references with different contact roles and difficulty.

• On our two public-data evaluation splits, DONIC outperforms bimanual adaptations of DexTrack and Dexplore in success rate and completion, reaching 86.79% and 57.32% success on unseen trajectories and objects, respectively.

Related Work

From human references to executable manipulation. Human trajectories provide motion targets, but retargeting alone cannot ensure successful contact. Sequence-specific policies commonly use reinforcement learning to correct retargeted motion. ManipTrans [15] reuses a general hand-motion imitator but learns a task-specific residual policy for interaction. DexMachina [23] learns from individual demonstrations by gradually removing virtual object assistance. CHORD [44] guides sequence-specific learning through contact-wrench matching, while ConTrack [17] trains one policy per clip with an adaptive balance between object tracking and motion fidelity. We instead learn one generalist bimanual policy that tracks diverse references without per-reference training or fine-tuning at test time.

Generalist manipulation policies. Generalist policies share knowledge across objects, motions, or skills, predominantly in single-hand settings. UniDexGrasp++ [33] generalizes grasping across object geometries, UniCross [42] unifies multiple manipulation skills, and ArtManip [39] learns category-level articulated in-hand manipulation. For reference tracking, DexTrack [19] combines reinforcement learning with demonstrations mined by trajectory-specific controllers. Dexplore [38] expands exploration around reference motions, while AdaDexTrack [1] combines specialist distillation with online reference modulation. Bimanual tracking must coordinate complementary hand roles and contact timing. Existing work explores narrower generalization: BiDexHD [43] distills teachers into a separate trajectory-conditioned policy for each operation category and evaluates new object–tool combinations and instances within categories; PhysGraph [16] transfers task-trained policies to new tool and object geometries. We jointly train one bimanual reference tracker across operation categories and human motion datasets, and evaluate it on unseen trajectories and objects without fine-tuning.

Bimanual hand–object interaction datasets. GRAB [32] records whole-body grasping with detailed contact information, while ARCTIC [9] captures the coupled motion of two hands and articulated objects. OakInk2 [41] organizes bimanual manipulation into primitive and complex tasks, and TACO [20] covers diverse tool–action–object compositions. GigaHands [10] further expands the scale and diversity of bimanual activities with language annotations. Complementing these motion collections, H2O [13] and HOI4D [21] support egocentric interaction understanding, while HOT3D [3] and HO-Cap [34] provide recordings and annotations for hand–object reconstruction and tracking. Capture and annotation conventions differ, and human kinematics do not ensure executable robot actions. We harmonize, retarget, and screen zero-residual feasibility to form a common bimanual reference pool for joint training.

Bimanual Hand–Object Dataset

MV: multi-view; ego: egocentric. †14.77 hours at 30 fps, counting synchronized time steps once. Public frame counts follow source conventions. Contact is the total number of both-hand-contact frames divided by frames with at least one hand in contact, pooled across analyzed clips; –: unavailable.

To expand limited bimanual training data, we introduce BiHOI, a multi-view hand–object interaction dataset. We describe its capture system and composition, then the annotation and retargeting pipeline that converts recordings into robot references.

Multi-View Capture and Dataset

Multi-view capture system. We record hand–object interactions with 32 calibrated RGB cameras surrounding a tabletop workspace, at a resolution of $1920 \times1232$ pixels and a frame rate of 30 fps. Complementary views support reconstruction under occlusion; calibration supplies a common metric coordinate system.

Dataset composition. BiHOI contains 1,575 interaction sequences covering 229 objects and 14.77 hours of unique interaction. The objects span household items, containers, utensils, tools, and electronics. Participants perform natural object-centric interactions rather than following fixed motion templates. The recordings include symmetric bimanual manipulation, asymmetric bimanual coordination, and coordinated multi-object manipulation. Table 1 compares BiHOI with commonly used hand–object interaction datasets. Our collection emphasizes sustained two-hand interaction. Among frames with at least one hand in contact, 65.91% have both hands in contact, exceeding H2O, the highest analyzed public subset at 38.57%, by 27.34 percentage points.

Source	Capture	Sequences	Frames	Objects	Bimanual contact (%)
BiHOI (ours)	MV RGB	1,575	≈1.60M†	229	65.91
GRAB [2020]	Marker MoCap	1,334	1.622M	51	32.21
OakInk2 [2024]	MV RGB + MoCap	627	4.01M	75	24.97
TACO (V1) [2024]	MV RGB + ego RGB-D	2.5K	5.2M	206	15.69
GigaHands [2025]	MV RGB	–	183M	417	34.74
H2O [2021]	MV RGB-D	–	571.6K	8	38.57
HOI4D [2022]	Ego RGB-D	4,000	2.4M	800	–
HO-Cap (rev.) [2025]	MV RGB-D	64	656K	64	–

Annotation and Reference Construction

Reconstruction and annotation. We reconstruct metric hand–object trajectories from synchronized RGB recordings. SAM3 provides object masks [7], while multi-view Depth Anything 3 provides spatially consistent depth that we align to metric scale using the calibrated camera extrinsics [18]. MV-SAM3D reconstructs a canonical object mesh from first-frame views and masks [14]; back-projected depth determines its physical scale. FoundationPose then recovers 6-DoF object motion [37], using multi-view pose consensus to handle occlusion and re-registration when consensus fails.

For each participant, triangulated calibration keypoints are used to estimate metric hand scale and calibrate a participant-specific MANO shape. WiLoR provides per-view hand observations [24]; these observations are triangulated, temporally smoothed, and fitted with the MANO hand model [29], with shape held fixed across the sequence. Joint refinement against multi-view metric depth combines geometric alignment with reconstruction and temporal regularization, producing bimanual MANO parameters, a metric object mesh, and its 6-DoF trajectory in a common world frame. Reconstruction and refinement details appear in Appendix § 8.§ 2.

Retargeting. We map the reconstructed human motion to the Wuji robot hands through framewise kinematic fitting, following standard retargeting practice [11, 27, 19]. After wrist-frame alignment, we fit hand keypoints to robot links under joint limits, initializing and regularizing each frame against the preceding solution. Smoothing occurs only before retargeting. The resulting wrist and finger references are paired with the object trajectory.

Methodology

DONIC learns a generalist bimanual tracker across operation categories and human-data sources (Figure 2). Its object-centered per-hand reward supports complementary roles and asynchronous contacts while maintaining object-motion feedback. CCA removes reference-dependent offsets after generalized advantage estimation (GAE) [30], preserving within-reference advantage differences.

Task Definition

Given a reference for two hands and an object, one policy controls both robot hands to reproduce the object motion through physical interaction. Following DexTrack [19], training combines PPO with BC from executable demonstrations; the policy tracks unseen trajectories and objects without fine-tuning.

For a clip spanning $T$ control steps, we write the reference as

Here $t$ is the frame index, $h \in\{L, R\}$ identifies a hand, and a hat denotes a reference quantity. The configuration $\hat{q}^{h}_{t} \in\mathbb{R}^{d}$ contains the wrist coordinates and finger-joint angles, with $d$ controlled coordinates per hand. The object position and orientation are $\hat{p}^{o}_{t} \in\mathbb{R}^{3}$ and $\hat{R}^{o}_{t} \in SO(3)$, where $SO(3)$ is the space of three-dimensional rotations. We concatenate the two hand configurations into $\hat{q}_{t} \in\mathbb{R}^{2d}$. Corresponding quantities without hats denote simulated states. Object linear velocity is denoted by $v^{o}_{t}$; its reference $\hat{v}^{o}_{t}$ is obtained by finite differences. Positions, rotations, and linear velocities use metres, radians, and metres per second, respectively.

Reference-Conditioned Bimanual Control

A single stochastic policy $\pi_{\theta}(a_{t} \mid o_{t})$, parameterized by $\theta$, predicts joint actions $a_{t}$ for both hands from observation $o_{t}$. Following reference-based trackers [19, 38], $o_{t}$ concatenates two per-hand blocks containing proprioception, object pose and velocity, upcoming reference targets, reference-state errors, and fingertip-to-object distance vectors. Spatial features use each wrist’s heading frame. We append a basis-point-set shape encoding [25], object scale and centroid, hand identities, and the fixed left-hand reward weight $w_L \in(0,1]$. Contact-force channels are zeroed; simulated forces supervise rewards only.

Given $o_t$, the policy predicts bounded corrections $a_t \in[-1,1]^{2d}$ to the next reference, producing a proportional–derivative (PD) control target as in prior trackers [19, 23]:

where $s \in\mathbb{R}^{2d}$ sets the correction range, $\odot$ denotes element-wise multiplication, and clamp enforces the coordinate limits $q_{\min}$ and $q_{\max}$. Corrections are anchored to the next reference at every step and do not accumulate. The PD controller executes the target, and the resulting state supplies the next observation. We build on this control interface with the following reward and training procedure.

Object-Centered Per-Hand Reward

Hand-pose accuracy does not ensure contact or object motion, and the hands may engage asynchronously with different roles. We therefore reward object motion throughout the clip while supervising each hand’s contact only when required by the reference.

Tracking the object. We reward agreement in object position, orientation, and linear velocity:

Positive coefficients $\kappa_p,\kappa_R,\kappa_v$ control sensitivity to position, rotation, and velocity errors. Here $\lVert\cdot\rVert_2$ is the Euclidean norm and $\angle(\cdot,\cdot)$ the rotation geodesic distance. The product rewards joint agreement, with velocity penalizing stationary solutions to moving references. This kinematic term [35, 36] is independent of the contact gate.

Supervising each hand’s contact. Object tracking alone does not specify how the hands should support the motion. Let $C_t^h$ be the set of fingertips on hand $h$ required to contact the object by the reference at frame $t$. For fingertip $i$, let $\delta_{i,t}^h$ denote its distance to the object surface and $f_{i,t}^h$ its simulated net contact-force magnitude. We estimate contact using

where $\mathbb{1}[\cdot]$ is an indicator and $f_{\min},\delta_{\max}>0$ are force and distance thresholds. Proximity reduces false object-contact attribution, but net force does not identify the collision partner.

For a hand with at least one required fingertip, we define

Here $\lvert C_t^h\rvert$ is the number of required fingertips, $\kappa_c>0$ controls distance sensitivity, and $\zeta\in[0,1]$ balances proximity and contact. Proximity guides contact acquisition; the hold term rewards instantaneous required-contact coverage, allowing valid sliding or rolling.

Combining the learning signals. We aggregate contact rewards over the engaged-hand set $E_t=\{h\in\{L,R\}:C_t^h\ne\varnothing\}$. For $u\in\{\mathrm{cont},\mathrm{hold}\}$, let

where $w_R=1$ and $w_L\in(0,1]$ are fixed hand weights. Averaging preserves either engaged hand’s feedback when the other loses contact; excluding inactive hands avoids rewarding attachment during reference release.

Finally, we combine these signals with the inherited hand-tracking reward $r_t^{\mathrm{hand}}\in[0,1]$, which measures wrist, finger-joint, and keypoint agreement [38]:

The nonnegative $\beta$ coefficients sum to one; $(1+w_L)$ sets the reward scale. These terms jointly supervise hand motion, object outcomes, and reference contact roles (settings in Appendix § 9.§ 2).

Learning a Generalist from Verified Demonstrations

Following DexTrack [19], we alternate PPO+BC training with demonstration collection. PPO learns through simulated interaction, while BC supplies successful actions that shared exploration may struggle to discover. Trajectory-specific policies optimize references retained by replay screening; their verified rollouts form a demonstration library $\mathcal{D}$ of observation–action pairs. As the generalist improves, its verified successes are added to $\mathcal{D}$ for subsequent updates. All demonstrations come from training references.

Demonstrations supply successful behavior, while CCA addresses unequal reference difficulty in PPO updates.

Clip-Centered Advantages. Shared PPO batches mix contact patterns and coordination demands. GAE combines temporally weighted value-prediction residuals, but unequal critic accuracy across references can leave clip-dependent offsets. Difficult clips may then receive consistently negative advantages despite relatively successful transitions. CCA removes this common offset while preserving within-clip differences.

Formally, let $\mathcal{B}$ denote an on-policy rollout batch, $j$ index a transition, and $m(j)$ identify its reference clip. After computing the standard GAE advantage $A_j^{\mathrm{GAE}}$ from rewards and the learned value function, we group transitions by reference as $\mathcal{G}_m = \{j \in\mathcal{B} : m(j) = m\}$ and use

Subtracting the clip mean preserves pairwise advantage differences among its transitions while removing their common empirical offset. The adjustment is applied after GAE; the learned critic and temporal advantage estimation remain unchanged.

Groups are defined by reference identity rather than by environment instance or temporal phase, allowing parallel rollouts of the same reference to contribute to a common baseline. Unlike batch-wide centering, CCA removes an offset specific to each reference instead of applying one shared shift to all clips. We subtract only the group mean, without dividing by a per-clip standard deviation, preserving the scale of within-clip advantage differences. This adjustment acts on the policy’s learning signal; it does not modify the reward or the critic’s regression targets.

Joint PPO and BC updates. We use $\widetilde{A}_j$ in the standard PPO actor loss $\mathcal{L}_{\mathrm{PPO}}$, retaining its on-policy probability ratio and clipping rule. For a demonstration observation–action pair $(o^D, a^D) \in\mathcal{D}$, let $\mu_\theta(o^D)$ denote the shared Gaussian policy’s mean action. The combined actor objective is

where $\lambda> 0$ weights BC and the expectation averages over sampled demonstration pairs. Demonstrations use a separate sampling budget and do not enter the PPO ratio or GAE. The critic retains its standard value-regression objective. Sampling, verification, and training details are provided in Appendices § 9.§ 4–§ 9.§ 5.

Experiments

Setup

Datasets. We curate bimanual manipulation references from GRAB [32], GigaHands [10], OakInk2 [41], TACO [20], and H2O [13]. We retain clips in which both hands and the manipulated object are tracked, and each hand contacts the object at some point during the clip; simultaneous contact is not required. Motions are harmonized, retargeted to the Wuji Hand V1, and replay-screened (Appendix § 8.§ 1). This yields 1,845 public-data training clips, separate from the held-out evaluation references. Ours+BiHOI additionally uses all 1,575 BiHOI sequences, bringing its training pool to 3,420 clips.

For evaluation, we reserve two splits from the public datasets: UNSEEN-TRAJ contains 53 unseen trajectories of training objects, and UNSEEN-OBJ contains 164 clips involving held-out objects. Neither split is used for policy training or demonstration collection. All methods are evaluated on the same splits without fine-tuning.

Metrics. We extend AdaDexTrack’s evaluation protocol [1] to bimanual tracking. We report object errors ($T_o$, $R_o$), wrist errors ($T_h$, $R_h$), and finger-joint error ($E_{\mathrm{finger}}$), with translation in centimetres and angles in radians. Wrist errors use robot forward kinematics against retargeted references; finger error averages over 20 finger joints. Reported hand errors take the worse of the two trajectory-averaged hand errors, then average across trajectories. SR requires both hands independently to satisfy $\frac{1}{2}(T_o + T_h) \le10\ \mathrm{cm}$, $\frac{1}{2}(R_o + R_h) \le\pi/3$, and $E_{\mathrm{finger}} \le2\pi/9$ on trajectory-averaged errors. Comp is the fraction of frames satisfying all three conditions for both hands, averaged over trajectories.

Baselines. We compare against DexTrack-bi and Dexplore-bi, using their released code with observation and control interfaces adapted to two Wuji hands. Both baselines share DONIC’s public-data pool, Isaac Gym simulator, robot model, and held-out splits. DexTrack-bi follows the released specialist–distillation pipeline [19]. Dexplore-bi retains Dexplore’s original reward and reference-scoped exploration [38]. Additional implementation details are provided in

Appendix D.3.

Main Results

One DONIC checkpoint improves both held-out splits without fine-tuning (Table 2). On UNSEEN-TRAJ, DONIC achieves 86.79% SR and 67.62% Comp, exceeding the stronger baseline, Dexplore-bi, by 77.36 and 39.75 percentage points, respectively. On UNSEEN-OBJ, it reaches 57.32% SR and 54.69% Comp, compared with 11.59% and 26.37% for Dexplore-bi. The gains extend to hand tracking: wrist translation error decreases from 21.88 to 3.96 cm on unseen trajectories and from 19.46 to 4.58 cm on unseen objects. DONIC reduces all reported errors relative to both baselines except object rotation on UNSEEN-TRAJ. Because SR and Comp require both hands to meet the tracking criteria, these improvements reflect successful bimanual execution rather than accuracy from only one hand.

Table 2 | Reference tracking without fine-tuning, using one checkpoint per method across both splits. Black and blue bold mark the best and second-best results within each split; ties share highlighting.

Dataset	Method	$T_o \downarrow$ (cm)	$R_o \downarrow$ (rad)	$T_h \downarrow$ (cm)	$R_h \downarrow$ (rad)	$E_{\mathrm{finger}} \downarrow$ (rad)	SR$\uparrow$ (%)	Comp$\uparrow$ (%)
Unseen trajectory	DexTrack-bi	41.86	2.3340	23.03	0.8296	0.6301	0.00	1.30
	Dexplore-bi	14.73	0.8597	21.88	0.8412	0.3166	9.43	27.87
	w/o CCA	11.27	0.9585	4.34	0.2980	0.0433	81.13	62.72
	w/o contact gate	10.26	0.9589	4.00	0.3013	0.0338	81.13	65.03
	Ours	10.27	0.9277	3.96	0.2905	0.0372	86.79	67.62
	Ours+BiHOI	9.74	0.8637	3.32	0.2818	0.0338	88.68	68.72
Unseen object	DexTrack-bi	127.87	2.3532	72.83	1.0629	0.6455	0.00	0.88
	Dexplore-bi	17.92	1.1431	19.46	0.9509	0.3685	11.59	26.37
	w/o CCA	16.83	1.1464	5.52	0.3235	0.0378	48.17	43.89
	w/o contact gate	17.61	1.2041	4.97	0.3386	0.0371	48.78	49.52
	Ours	15.74	1.1310	4.58	0.3226	0.0373	57.32	54.69
	Ours+BiHOI	13.96	1.1157	4.54	0.3206	0.0361	59.76	55.35

On the two held-out object trajectories in Figure 3, DONIC coordinates both hands and more closely reproduces reference object motion. Dexplore-bi moves its hands but produces limited object motion, whereas DexTrack-bi loses the hand–object alignment needed to execute the reference.

Ablation Studies

Learning signals. Removing CCA reduces SR from 86.79% to 81.13% on unseen trajectories and from 57.32% to 48.17% on unseen objects, while lowering Comp and increasing all five tracking errors (Table 2). Removing the contact-ownership gate reduces SR to 81.13% and 48.78%, respectively. Despite lower individual errors in some cases, this variant also has lower Comp on both splits. Both learning signals thus improve joint bimanual success and completion.

Additional bimanual data. Adding all 1,575 BiHOI sequences expands training from 1,845 to 3,420 clips and improves all seven metrics on both splits (Ours+BiHOI, Table 2). SR reaches 88.68% and 59.76% on unseen trajectories and objects, with Comp of 68.72% and 55.35%. Unseen-object translation error falls by 11.3%, from 15.74 to 13.96 cm. Ranking first on all unseen-object metrics, Ours+BiHOI demonstrates the benefit of additional bimanual data.

Conclusion, Limitations and Future Work

We presented DONIC, a generalist bimanual tracker, and BiHOI, a multi-view bimanual interaction dataset. Object-centered per-hand rewards and CCA support iterative PPO+BC training, yielding one checkpoint that generalizes to unseen trajectories and objects without fine-tuning and substantially outperforms adapted baselines. Ablations support both learning signals and the benefit of additional BiHOI data. However, our quantitative evaluation focuses on simulated single-object interactions and relies on reconstructed, retargeted motion references. We aim to extend DONIC to longer-horizon and multi-object manipulation, evaluate on an independent benchmark, and investigate transfer to physical bimanual systems.

This appendix provides additional related work (Appendix § 7), dataset construction and retargeting details (Appendix § 8), controller and training specifications (Appendix § 9), and experimental protocols and results (Appendix § 10).

Additional Related Work

Executable human references and shared control. Human demonstrations provide motion targets but do not directly specify robot actions: differences in hand kinematics and simulated contact can make a kinematically plausible reference physically difficult to execute. Reference-based trackers address this gap through closed-loop control. DexTrack [19] combines trajectory-specific demonstration generation with generalist learning; Dexplore [38] expands exploration around reference motions; and AdaDexTrack [1] combines specialist distillation with reference modulation. Our reference-conditioned action interface and iterative reinforcement–imitation learning loop build on this line of work. Our contributions within that loop are the object-centered per-hand reward and Clip-Centered Advantages (CCA), designed for shared bimanual tracking across heterogeneous references.

What generalization means here. Generalization across grasped objects, manipulation skills, and complete reference trajectories imposes different requirements. UniDexGrasp++ [33] studies generalizable dexterous grasping, while BiDexHD [43] learns trajectory-conditioned bimanual policies within operation categories. Our controller is jointly trained across operation categories and human-motion sources. We evaluate new trajectories of training objects and trajectories involving held-out objects with the same checkpoint. These two splits test reference and object generalization; they do not, by themselves, establish generalization to held-out operation categories or arbitrary manipulation goals.

Interaction capture and usable control data. GRAB, GigaHands, OakInk2, TACO, and H2O provide complementary sources of hand–object interaction [32, 10, 41, 20, 13]. HOI4D and HO-Cap further broaden capture settings and annotation tools [21, 34]. Our comparison distinguishes the inventory of a source dataset from the subset retained for robot training. A clip enters our bimanual pool only if the required hand and object tracks are available and both hands contact the object at some point. This allows sequential as well as simultaneous hand participation. BiHOI complements these sources with a higher measured proportion of simultaneous bimanual contact in the analyzed recordings.

Dataset and Reference Construction

Capture, Composition, and Training Inventory

BiHOI contains 1,575 interaction sequences covering 229 objects and 14.77 hours of unique interaction. The objects span household items, containers, utensils, tools, and electronics. A calibrated array of 32 RGB cameras records a tabletop workspace at 1920 × 1232 pixels and 30 frames per second. Duration counts unique interaction time, not the sum over camera views. Recordings include symmetric manipulation, asymmetric coordination, and coordinated multi-object interactions. Participants perform natural object-centric interactions rather than fixed motion templates. These describe the capture inventory; the control formulation tracks two robot hands and one reference object.

We characterize contact as simultaneous or sequential and measure the duration of simultaneous contact. In the analyzed BiHOI recordings, simultaneous bimanual contact accounts for 65.91% of contact-active time (at least one hand in contact). The highest proportion among the analyzed public subsets is 38.57% for H2O, a difference of 27.34 percentage points. Table S1 reports the selected public training inventory and the measured contact proportions. These subset measurements should not be interpreted as statistics over every recording released by each source.

Source	Training sequences	Simultaneous contact (%)
GRAB	505	32.21
GigaHands	592	34.74
OakInk2	449	24.97
H2O	127	38.57
TACO	172	15.69
Public-data pool	1,845	—
BiHOI	1,575	65.91
Public-data pool + BiHOI	3,420	—

For the public pool, both hands and the manipulated object must be tracked, and each hand must contact the object during the clip; continuous or simultaneous contact is not an admission requirement. Motions are converted to a common representation consisting of per-hand MANO parameters, an object mesh, and object position and orientation in a gravity-aligned world frame. Retargeting and replay screening then produce robot references. The 53 unseen-trajectory and 164 unseen-object evaluation clips are separate from the training counts.

Contact measurement. For clip $m$, frame $t$, and hand $h$, let $c^{h}_{m,t,i}$ denote the reference label at fingertip slot $i \in\{3, 6, 9, 12, 15\}$. Define activity $a^{h}_{m,t} = 1[\exists i : c^{h}_{m,t,i} > 0.01]$, using the same reference-hand activity rule as training. The reported percentage is

Thus the denominator is contact-active time, and aggregation pools frames across clips. It is neither the fraction of all sequence frames nor a mean of per-clip percentages. On the analyzed public pool, this measure is 31.40%; using all frames as the denominator instead gives 22.56%.

Single-object representation. For multi-object recordings, conversion retains one primary object: TACO keeps the tool rather than the target, and OakInk2 selects the main object from its hand-role program annotations. The controller represents the selected object by a fixed mesh and a rigid pose. Articulation and deformation are not modeled explicitly, even when the captured object belongs to those categories.

Multi-View Reconstruction and Annotation

The reconstruction pipeline uses learned image predictions to initialize geometry, then calibrated multi-view observations to recover physical scale and temporal consistency. Its five stages are mask/depth extraction, object reconstruction and tracking, participant-specific hand calibration, hand-motion reconstruction, and joint refinement. Figure S1 summarizes the complete calibrated multi-view reconstruction and annotation pipeline.

Masks and metric depth. We use SAM3 [7] in two stages. The first frames from all cameras form an ordered cross-camera sequence: a single object prompt initializes consistent masks across views. Each camera video is then tracked temporally using its first-frame mask. At each synchronized time, Depth Anything 3 [18] processes the views with calibrated intrinsics and world-to-camera extrinsics. Extrinsic-scale alignment anchors depth to metric camera translations. The resulting depth supports object-scale recovery and subsequent pose refinement.

Metric object reconstruction. MV-SAM3D [14] reconstructs a canonical mesh from distributed first-frame views and masks. We back-project masked depth from valid views into the common world frame, reject inconsistent per-camera point-cloud centers, and fuse the remaining observations. To estimate physical scale without aligning an ambiguous canonical orientation, we compare robust principal-component extents of the depth cloud and mesh samples. After removing radial outliers, let $e_k^{\mathrm{depth}}$ and $e_k^{\mathrm{mesh}}$ be their sorted 1–99 percentile extents along the three principal axes. The isotropic scale is

Only scale is transferred; the canonical mesh orientation is retained.

Multi-view object tracking. We triangulate centers of minimum-area rectangles fitted to object masks to initialize translation, and initialize orientation from the preceding accepted pose. The ten views with the largest visible masks are refined independently with Foundation-Pose [37]. Candidate poses are transformed into the world frame and grouped by geometric agreement. Two candidates $(R_i,t_i)$ and $(R_j,t_j)$ are compatible when their rotation geodesic distance $d_{\mathrm{SO}(3)}(R_i,R_j) < 30^\circ$ and $\lVert t_i-t_j\rVert_2 < 5\ \mathrm{cm}$. At least four mutually compatible views are required. Translation is averaged with mask-area weights, while orientation is taken from the largest-mask view in the accepted cluster. Failed consensus triggers re-registration.

The object trajectory undergoes translation spike repair and Kalman filtering, followed by rotation spike repair and filtering on $SO(3)$, the space of three-dimensional rotations. Interpolation is restricted to short spike bursts. We subsequently refine the full trajectory against fused metric depth using alternating iterative closest point correspondence updates. Correspondences farther than 3 cm are rejected. For point-to-plane and point-to-point residuals $r_{\mathrm{p2pl}}$ and $r_{\mathrm{p2p}}$, respectively, the geometric contribution is

where $\rho$ is the robust residual penalty. Reconstruction priors and first- and second-order temporal regularization constrain the update. Translation and rotation corrections remain within 3 cm and $20^\circ$ of the input trajectory.

Participant-specific hand calibration. For each hand, a flat-hand calibration sequence provides metric triangulated positions for a 21-joint skeleton. An initial MANO shape $\beta_0$ [29] is calibrated against these observations. Let $\mathcal{B}$ contain the 20 skeleton bones, $\mathbf{X}^{\mathrm{tri}}_a$ be triangulated joint $a$, and $\mathbf{J}_a(\beta_0)$ the corresponding model joint. Physical scale is

The explicit scale is retained because MANO shape coefficients are not a global scale parameter. Components requiring a MANO-only shape vector receive an approximate scale-adjusted shape from ridge fitting in centered canonical vertex space. The participant-specific calibration is reused across sequences.

Hand-motion reconstruction. WiLoR [24] provides 21 image-space keypoints per hand and view. For each joint, valid two-view pairs generate triangulation hypotheses. Reprojection residuals below 50 pixels define inliers, with at least three views required. The selected hypothesis is re-triangulated using all inliers. At the reconstruction stage’s 10 fps sampling rate, each joint is filtered with a constant-velocity state $[x,y,z,v_x,v_y,v_z]^{\mathsf{T}}$, where $x,y,z$ are coordinates and $v_x,v_y,v_z$ their velocities. Measurement and acceleration-process standard deviations are 0.01 m and 0.12 m/s$^2$, respectively; missing observations use prediction only. This reconstruction sampling rate is distinct from the 30 fps capture rate. MANO fitting combines filtered metric joints, multi-view image keypoints, and calibrated projections, keeping hand shape fixed while estimating pose and global motion.

Joint hand–object refinement. The hand and object estimates are refined against shared metric depth. Variables are per-frame hand pose and global transforms, object pose, and one object-scale variable shared by the sequence. The objective is

Here $\mathcal{L}_{\mathrm{SDF}}$ aligns reconstructed geometry to a signed distance field (SDF) built from depth; $\mathcal{L}_{\mathrm{reg}}^h$ and $\mathcal{L}_{\mathrm{reg}}^o$ preserve hand and object initialization, and the two smoothness terms regularize their temporal evolution. The MeshSDF formulation follows HO-Cap [34], using 32 depth views in the interaction region and 3 mm voxels. Optimization first uses priors and smoothness, then activates SDF alignment, and finally refines object scale with poses fixed. It does not include explicit contact, penetration, collision, or snapping losses.

Export representation. The standardized export uses a right-handed, $+Z$-up world frame with its recorded $4 \times4$ transform from the reconstruction frame. The tabletop defines $z = 0$, estimated from the fifth percentile of the object bottom. MANO uses 45 axis–angle pose coordinates without PCA, with a flat-hand mean and no added mean-pose offset. Object vertices retain their canonical coordinates without recentering, axis flips, or mirroring.

Kinematic Retargeting and Replay Screening

Retargeting follows position-based kinematic fitting [11, 27, 19]. For each frame and hand, fifteen target entries are fitted to corresponding Wuji-hand links: five MANO fingertips (indices 4, 8, 12, 16, 20), five additional finger keypoints (3, 7, 11, 15, 19), and the wrist (index 0) repeated five times. The fit uses SmoothL1 position loss with transition scale 0.02 m and NLopt optimization with robot joint limits as hard bounds. The implementation uses a joint-limit tolerance of $10^{-3}$, initializes from the preceding frame’s solution, and uses inter-frame regularization parameter norm_delta = $4 \times10^{-3}$.

At conversion, source motion is interpreted at 10 fps and converted to 30 fps control references (SOURCE_FPS=10, FPS_OUT=30, RESAMPLE_STRIDE=3). The source visualization videos and per-clip provenance determine this timing; the export metadata’s nominal 30 fps field is not used as the source rate. Pre-retargeting cleanup uses a 0.10 m per-frame displacement threshold at 30 fps, at most three despiking passes, and nearest-neighbor filling. These settings belong to conversion cleanup, not reconstruction filtering.

Trajectory cleanup occurs before retargeting. The retargeter’s low-pass coefficient is 1, so it performs no additional low-pass smoothing. Wrist translation is computed as the MANO global translation plus its shape-dependent root joint, $\mathbf{t} + J_0(\boldsymbol{\beta})$; a fixed frame correction aligns global orientation with the robot convention. The output pairs robot wrist/finger configurations with the reconstructed object trajectory.

Open-loop replay with zero policy correction classifies each candidate reference into six outcomes: immediate object loss (A), object ejection (B), contact without lifting (C), partial execution (D), tracking with an incorrect release or ending (E), and complete execution (F). Categories A and B are excluded, whereas C–F remain available for closed-loop training. Category C is retained because failure to lift under reference-only replay does not rule out recovery by feedback control. Screening determines reference eligibility; it is distinct from the verification used to admit executable demonstrations.

Method and Training Details

Controller Interface

A reference clip $m$ contains robot configurations $\hat{\mathbf{q}}^h_t \in\mathbb{R}^{26}$ for hand $h \in\{L, R\}$, object position $\hat{\mathbf{p}}^o_t \in\mathbb{R}^3$, and object rotation $\hat{\mathbf{R}}^o_t \in SO(3)$ at control step $t$. Hats denote references; quantities without hats denote simulated states. Each hand has six wrist coordinates and twenty finger joints. The shared policy $\pi_\theta(\mathbf{a}_t \mid\mathbf{o}_t)$, with parameters $\theta$, receives observation $\mathbf{o}_t$ and produces a 52-dimensional action $\mathbf{a}_t$ for both hands.

The 3,016-dimensional observation combines 2,884 base features with a 132-dimensional object descriptor: 128 basis-point-set features [25], scale, and centroid. Per-hand blocks contain proprioception, object pose and velocity, upcoming reference targets, state errors, and fingertip-to-object distance vectors in each wrist heading frame. Hand identity and left-hand reward weight are included. Forty contact-force channels are set to zero; contact forces contribute to reward supervision, not policy input.

Concatenating both reference configurations as $\hat{\mathbf{q}}_{t+1}$, the proportional–derivative (PD) target is

The vector $\mathbf{s}$ sets coordinate-specific correction bounds, $\odot$ denotes element-wise multiplication, and $\mathbf{q}_{\min},\mathbf{q}_{\max}$ are coordinate limits. Bounds are 0.1 m for wrist translation, 1.0 rad for wrist rotation, and 0.15 rad for finger joints. Corrections are applied around the next reference at every step and do not accumulate. This interface is inherited from reference-tracking approaches [19, 38].

Reward Definition and Coefficients

We combine hand tracking, object motion, per-hand contact, and contact maintenance:

The fixed hand weights are $w_L = 0.5$ and $w_R = 1$; adaptive hand weighting is disabled. The $\beta$ coefficients sum to one. The inherited hand-tracking term uses Dexplore’s per-hand multiplicative chain for wrist, finger-joint, and keypoint agreement [38], with the released subterm coefficients unchanged. Its two per-hand values are combined as $(r_t^{\mathrm{hand,R}} + w_L r_t^{\mathrm{hand,L}})/(1+w_L)$. Object-position and object-rotation factors within this hand chain have zero weights (reward.op=reward.or=0), avoiding double counting with the separate object reward.

For object linear velocity $\mathbf{v}^{o}_{t}$ and reference velocity $\hat{\mathbf{v}}^{o}_{t}$ from finite differences, the object reward is

The implemented object reward multiplies the three factors and has no $\eta$ mixture weights. Joint position, orientation, and velocity agreement is rewarded; the velocity factor penalizes holding a moving reference stationary. This term is independent of contact gating. The positive coefficients $(\kappa_p,\kappa_R,\kappa_v) = (9, 2, 14)$ control the three error sensitivities. The full reward still adds the hand, object, contact, and hold terms.

The reference-required set $C_t^h$ uses the five fingertip slots $[3, 6, 9, 12, 15]$ within reference contact columns $[206:222]$ for the left hand and $[222:238]$ for the right. A fingertip is required when its channel exceeds 0.01; hand activity uses an element-wise logical OR over these tests, not a sum of signed labels. This reference mask is distinct from simulated contact. For fingertip $i$, surface distance $\delta^h_{i,t}$ and simulated net force magnitude $f^h_{i,t}$ define a contact proxy

For nonempty $C_t^h$, proximity and instantaneous required-contact coverage give

Parameter	Setting
$(\beta_{\mathrm{hand}},\beta_{\mathrm{obj}},\beta_{\mathrm{cont}},\beta_{\mathrm{hold}})$	$(0.25,0.45,0.15,0.15)$
$(\kappa_p,\kappa_R,\kappa_v)$	$(9,2,14)$
$(f_{\min},\delta_{\max})$	$(0.5\ \mathrm{N},0.03\ \mathrm{m})$
$(\kappa_c,\zeta)$	$(30\ \mathrm{m}^{-1},0.5)$
Parallel simulation environments	8,192
BC coefficient / pairs per minibatch	50 / 4,096
Recovery / perturbation curriculum	Disabled
Fixed hand weights $(w_L,w_R)$	$(0.5,1)$

The set of engaged hands is $\mathcal{E}_t=\{h:C_t^h\ne\varnothing\}$. Each contact-related term $u\in\{\mathrm{cont},\mathrm{hold}\}$ is aggregated as

Thus an inactive hand contributes no contact reward, and an unsuccessful engaged hand does not multiply away the other hand’s signal. The force–distance rule remains a proxy because net force does not identify the collision partner. Contact maintenance measures instantaneous coverage, not a no-slip condition, allowing valid sliding and rolling. Proximity guides fingertips before contact; the hold term evaluates required-contact coverage throughout the reference contact interval.

Clip-Centered Advantages and PPO+BC

We use proximal policy optimization (PPO) [31] with generalized advantage estimation (GAE) [30], and add behavioral cloning (BC) from verified demonstrations. Let $V_\phi(o_t)$ denote the learned value estimate, $\gamma$ the discount factor, and $\lambda_{\mathrm{GAE}}$ the temporal trace parameter. Writing $b_t$ for the bootstrap-validity mask, the temporal-difference residual is

GAE accumulates these residuals with weights $(\gamma\lambda_{\mathrm{GAE}})^\ell$, stopping the trace at episode boundaries. Bootstrap and timeout handling follow the base implementation. This temporal estimate is computed before any clip-level adjustment.

For an on-policy batch $\mathcal{B}$, transition index $j$, and reference identity $m(j)$, define $G_m=\{j\in\mathcal{B}:m(j)=m\}$. CCA uses

Each nonempty group has zero empirical mean after centering. For two transitions $j,k$ from the same clip, $\widetilde{A}_j-\widetilde{A}_k=A_j^{\mathrm{GAE}}-A_k^{\mathrm{GAE}}$. These identities explain the intended behavior: remove a common reference-dependent offset while preserving within-reference differences. CCA is a mean shift, not division by a clip standard deviation, and it retains the learned critic and temporal GAE calculation. Its empirical centering changes the actor update; the algebraic identities alone do not establish an unbiased policy-gradient estimator.

Let $\theta_{\mathrm{old}}$ denote the policy that collected $\mathcal{B}$, $\epsilon$ the PPO clipping radius, and $\rho_j(\theta) = \frac{\pi_\theta(a_j \mid o_j)}{\pi_{\theta_{\mathrm{old}}}(a_j \mid o_j)}$. The actor loss is

For demonstration library $\mathcal{D}$ and policy mean $\mu_\theta$, the combined objective is

The superscript $D$ denotes a stored observation–action pair. Demonstrations use a separate sampling budget and do not enter the PPO probability ratio or GAE. The critic retains its value-regression objective.

Demonstration Collection, Sampling, and Compatibility

Following DexTrack’s reinforcement–imitation loop [19], specialist training covers all references retained by replay screening. Each trajectory-specific controller seeks an executable realization of its reference; successful rollouts are exported and replay-verified before admission to the demonstration library. The shared controller is periodically evaluated on the training pool, and its verified successes supplement the library for subsequent PPO+BC updates. Held-out references are excluded throughout.

Verification checks object displacement and, when required by the reference, lifting. An exported rollout must succeed again on replay. This admission procedure selects executable supervision and is distinct from the trajectory-fidelity metrics in Section D.2. Each BC minibatch samples 4,096 observation–action pairs independently of the PPO minibatch.

A versioned action/reference contract records the action parameterization, reference version, and retargeting fingerprint. Changes to this contract invalidate incompatible stored actions.

Optimization and Simulation Settings

The generalist recipe uses Isaac Gym [22] with 8,192 environments. Actor and critic use separate multilayer perceptrons with widths [1024, 1024, 1024, 512] and ReLU activations. The Gaussian policy uses a fixed, non-learned scale parameter configured by sigma_init=-2.9. Input, value, and advantage normalization and value bootstrapping are enabled.

Demonstration supervision uses $\lambda_{\mathrm{BC}}=50$ with a separate minibatch budget. Adaptive recovery sampling and object-position perturbation curricula are disabled. Checkpoints are saved every 25 training iterations.

Experimental Protocols and Additional Details

Evaluation Splits

The public-only model uses 1,845 references and Ours+BiHOI uses 3,420, including all 1,575 BiHOI sequences. Evaluation comprises 53 new trajectories of training objects (UNSEEN-TRAJ) and 164 clips involving held-out objects (UNSEEN-OBJ). Neither split contributes to policy optimization, specialist training, or demonstration collection. Each method uses one checkpoint across both splits without fine-tuning.

Setting	Value
Optimizer / learning rate / schedule	Adam / $2 \times10^{-5}$ / constant
Rollout horizon / PPO minibatch	64 / 16,384
PPO passes per rollout / clipping radius	6 / 0.2
Discount / GAE trace parameter	0.99 / 0.95
Value / entropy / action-bound loss coefficients	5 / 0 / 10
Value clipping / gradient truncation	Disabled / disabled
Configured gradient norm / seed	1.0 / 42
Maximum episode steps	2,000
Control-frequency inverse / simulation substeps	2 / 4
Object density / hand friction	400 / 2.0
Linear / angular object-velocity caps	20 / 50
Reset angular-velocity cap	20
Pre-grasp initialization	0.9
Reference feed-forward / deployment mask	Enabled / enabled
Reference-correction bounds (wrist trans./rot.; fingers)	0.1 m / 1.0 rad / 0.15 rad
BC coefficient / demonstration batch	50 / 4,096
Recovery / object-perturbation curriculum	Disabled
Environment transitions per rollout iteration	$8,192 \times64 = 524,288$

The main DONIC results use a single frozen checkpoint. Evaluation uses the policy mean and starts clips from their first frame. Neither evaluation split supplies training transitions or demonstrations.

Bimanual Tracking Metrics

We extend AdaDexTrack’s trajectory-fidelity evaluation [1] to require both hands to satisfy the tracking criteria. Let $\mathcal{M}$ be the evaluated clips, $T_m$ the number of evaluated frames in clip $m$, and $h \in\{L,R\}$ a hand. The following definitions apply to each frame; clip indices are omitted when unambiguous.

Object, wrist, and finger errors. Object translation error $e^o_{p,t}$ is the Euclidean distance between simulated and reference positions. For unit object-orientation quaternions $u^o_t$ and $\hat{u}^o_t$, rotation error is

The absolute inner product accounts for the quaternion double cover, giving an angle in $[0,\pi]$.

For each hand, the same robot Unified Robot Description Format (URDF) forward kinematics maps simulated and retargeted reference configurations to wrist poses $(p^h_t,R^h_t)$ and $(\hat{p}^h_t,\hat{R}^h_t)$. The wrist errors are

The source human wrist is not used in this comparison: the metric measures tracking after retargeting, rather than adding human-to-robot retargeting mismatch. With finger-joint angles $q^h_{t,k}$ and $\hat{q}^h_{t,k}$, finger error is

The six virtual wrist coordinates are excluded. All angular errors and thresholds are computed in radians; translation errors are expressed in centimetres for reporting and thresholding.

Aggregation of reported errors. For any frame error $e$, write $\bar{e}_m = T_m^{-1} \sum_t e_{m,t}$ for its trajectory mean. The reported object metrics are

Hand metrics first take the worse of the two per-hand trajectory means, then average over clips:

The worse hand can differ by metric. In particular, this is not an average across hands or a framewise maximum before temporal averaging.

Success rate and completion. For object errors $x^o_p,x^o_R$ and hand errors $x^h_p,x^h_R,x^h_f$, define

The angular thresholds $\pi/3$ and $2\pi/9$ correspond to $60^\circ$ and $40^\circ$. Let $\bar{e}^o_m$ and $\bar{e}^h_m$ collect the relevant trajectory means. Mean-error success rate is

The product implements a logical AND: both hands must independently pass using their own trajectory means. Completion instead applies the same tests separately at every frame:

A later valid frame contributes even after an earlier violation. Comp therefore measures per-frame satisfaction, not the length of the prefix before the first failure. Both metrics weight trajectories equally rather than weighting longer clips more heavily.

Baseline Implementations and Comparison Scope

DexTrack-bi. We use DexTrack’s released specialist–distillation pipeline [19], adapting its observation and action interfaces to two Wuji hands. Trajectory-specific controllers produce demonstrations for generalist distillation, and evaluation follows the released per-trajectory protocol.

Dexplore-bi. We run the released Dexplore code [38] with two-hand observation and action interfaces, retaining its original reward and reference-scoped exploration.

Evaluation settings. All methods use Isaac Gym, two Wuji hands, the same public training clips, and the same held-out split identities. Tracking errors are computed using the bimanual metrics defined in Section D.2. All 164 unseen-object and 53 unseen-trajectory clips are included.

Ablation Definitions and Additional Data

The learning-signal ablations in the main table examine CCA and object-centered supervision. The w/o CCA variant removes the clip-wise mean subtraction after GAE. The w/o contact gate variant disables DEXPLORE_CG_REWARD, replacing the simulated force-and-distance contact proxy by the force-only test $1[f_{i,t}^{h} > 0.5]$. It removes the 3 cm contact-ownership condition, not the object-motion reward. Reward coefficients $\beta$ and fixed hand weights are unchanged; they are not renormalized. Ours+BiHOI tests the effect of adding all 1,575 captured sequences to the public pool while retaining the same public evaluation splits.

Ablation outcomes. Removing either learning signal lowers both SR and Comp on both splits. CCA removal also increases all five tracking errors; removing the contact gate improves some individual errors but reduces joint bimanual success. Adding BiHOI improves all seven metrics on both splits, reaching SR of 88.68% and 59.76% on unseen trajectories and objects, respectively. Unseen-object translation error falls from 15.74 to 13.96 cm.

The learning-signal ablations retain the full model’s training data and optimization settings, changing only the indicated component. Ours+BiHOI uses the same training recipe with the expanded reference pool.

Qualitative Comparisons

Figure S2 compares five bimanual manipulation sequences. Across these examples, DONIC more closely reproduces reference hand–object coordination and object motion, whereas the baselines exhibit limited object motion or loss of hand–object alignment.

Limitations

The quantitative experiments evaluate reference-conditioned, single-rigid-object tracking in simulation. Unconstrained goal-conditioned manipulation and held-out operation-category generalization are outside this evaluation. The capture inventory includes broader object types than the rigid-object representation described in Section B.1. Contact supervision uses a net-force/proximity proxy, and reconstruction has no explicit contact or collision loss.