WorksComputer Science
Embodiment-Aware Planning for Heterogeneous Bimanual Manipulation
Abstract
Many manipulation tasks that are effortless for dexterous human hands remain challenging for robotic systems equipped with simple, task-specialized grippers. Industrial platforms commonly rely on end-effectors such as suction cups or parallel-jaw grippers, yet no single mode can reliably satisfy the diverse constraints encountered across different phases of complex manipulation tasks, including pickup, reorientation, insertion, and constrained placement. In this work, we investigate whether complementary simple grippers within a heterogeneous bimanual setup can collectively realize tasks that exceed the capability of any individual gripper alone. Given a single human demonstration specifying the desired object motion, we formulate embodiment-aware execution as a finite-horizon optimization over arm-trajectory assignments. At each segment, the planner determines whether single-arm manipulation or a coordinated dual arm handover is required, jointly reasoning over heterogeneous grasp feasibility, motion constraints, manipulability, and cross-gripper compatibility. Exploiting this formulation, we compute globally consistent assignments with explicit switching regularization. Experiments on a dual Franka FR3 platform equipped with suction and parallel-jaw grippers demonstrate improved feasibility and reliability in tasks that are infeasible under single-gripper execution.
Introduction
Humans perform complex manipulation tasks using highly dexterous hands capable of continuously varying contact modes, regrasping strategies, and in-hand adjustments. In contrast, most robotic systems rely on simple, task-specialized end-effectors such as suction cups or parallel-jaw grippers. While these grippers are robust, affordable, and widely deployed in industrial environments, each offers limited affordances. A suction gripper excels at lifting flat surfaces but struggles in confined insertions; a parallel-jaw gripper provides stable force-closure grasps yet often fails on thin or flush objects. Consequently, many tasks that are effortless for human hands cannot be reliably executed using a single simple robotic gripper.
In modern industrial settings such as warehouse automation, packaging lines, and retail fulfillment, objects vary widely in geometry, thickness, surface properties, and accessibility. During a single transport task, different phases may demand different grasp modalities. Rather than relying on complex anthropomorphic hands, a practical alternative is to combine complementary simple grippers within a heterogeneous bimanual system. Such a configuration has the potential to approximate the functional diversity of human manipulation while retaining the robustness and simplicity of industrial hardware. However, enabling two specialized grippers to collectively realize tasks that exceed the capability of either alone requires systematic coordination and planning.
Figure 1 illustrates this principle. A thin plate must be transferred from a narrow shelf to a table. The suction gripper cannot safely execute the pickup due to geometric interference during approach, whereas the parallel jaw gripper successfully performs the initial pick and transport. However, it cannot complete the placement due to collision constraints with table. The task becomes feasible only when the system performs a coordinated dual-arm handover, allowing the suction gripper to execute the final placement. Importantly, neither gripper alone can solve the task, yet their coordinated use enables seamless execution. This example captures the central premise of our work: heterogeneous coordination can elevate simple industrial grippers’ performance, provided that switching and compatibility are planned systematically.

Figure 1. Heterogeneous Gripper coordination for Bimanual Manipulation A plate transfer task that is infeasible for either gripper alone. The parallel-jaw gripper cannot safely perform the placement due to collisions with table, while the suction gripper cannot complete the constrained shelf pickup. By coordinating complementary simple grippers, the system extends its manipulation capability: parallel jaw performs the initial pick and transport (left), a dual-arm handover enables modality transition (center), and the suction gripper completes precise placement (right). Active arm(s) are shown in green, inactive in red. This example illustrates how heterogeneous coordination can approximate the versatility of human dexterity using simple industrial hardware.
In this work, we study embodiment-aware realization of demonstrated manipulation tasks using a heterogeneous dual-arm robotic system. A single human demonstration provides the desired object motion as a task-level specification. Rather than learning a policy, our objective is to determine how the demonstrated task can be executed under the constraints of heterogeneous robotic hardware. Given a discretized object trajectory, we determine at each segment which arm should manipulate the object and when coordinated handovers are necessary to maintain feasibility under grasp, kinematic, and collision constraints.
Heterogeneous execution introduces unique planning challenges. At different phases of a task, only specific gripper modalities may satisfy feasibility requirements. Coordinated handovers must respect cross-gripper grasp compatibility, and excessive switching increases execution time and failure risk creating the need for a global reasoning. We formulate this problem as a finite-horizon optimization over arm–trajectory assignments. Each state encodes both the active robot and its associated grasp-induced motion. Segment costs capture feasibility and manipulability quality, while transition costs encode heterogeneous grasp-pair compatibility and switching penalties. Exploiting the optimal substructure of this formulation, we compute a globally consistent execution plan via dynamic programming, equivalent to a structured shortest-path computation over a time-expanded graph.
Our contributions are threefold:
Heterogeneous gripper-level coordination: We formalize how a complementary simple gripper system can collectively realize manipulation tasks that are infeasible for any single modality alone, extending beyond simple pick-only tasks.
Globally consistent embodiment-aware assignment: We introduce a structured planning formulation that integrates heterogeneous grasp affordances and handover constraints into a globally consistent assignment.
Sim-to-real transfer: We demonstrate the framework on a dual Franka FR3 platform equipped with suction and parallel-jaw grippers on a task requiring modality transitions and coordinated handovers.
Related Work
Dexterous Manipulation vs. Simple Grippers
Robotic dexterous hands aim to replicate the versatility of human manipulation through multi-fingered contact, in-hand regrasping, and rich contact reasoning [1, 2]. While highly expressive, they introduce substantial mechanical and control complexity. Classical multi-fingered manipulation established many of the analytical foundations for contact kinematics and grasp planning [3, 4, 5]. Industrial systems instead rely on simple, task-specific end-effectors such as suction cups or parallel-jaw grippers for their robustness and cost efficiency. However, each supports only limited contact modalities. This leaves the challenge of expanding manipulation capability without complex anthropomorphic hardware largely open.
Recent reinforcement learning has enabled impressive dexterous manipulation on anthropomorphic hands, including large-scale sim-to-real policies and reorientation tasks [6, 7, 8]. We instead explore extending capability by coordinating complementary simple grippers in a heterogeneous bimanual system. Rather than increasing single-hand dexterity, we study whether structured coordination between specialized modalities can approximate the functional diversity.
Bimanual Manipulation and Grasp Planning
Bimanual manipulation has been widely studied for coordinated transport, reorientation, and in-hand manipulation [9]. Early work largely focused on symmetric dual-arm systems with identical end-effectors, emphasizing synchronized motion planning and closed-chain control [10, 11]. Several works also address grasp generation and object handover between two arms. GraspGen [12] generates grasps for parallel-jaw grippers, while sampling-based and graph-based methods search over candidate grasp pairs and intermediate transfer poses [13, 14].
More recently, learning-based approaches have demonstrated coordinated dual-arm behaviors, including end-to-end Vision–Language–Action models [15], learned bimanual coordination [16], and reactive self-handover in confined environments [17]. Manipulability-aware dual-arm grasp planning has also been explored in redundant systems [18], along with physics-aware saliency learning for bimanual grasp point selection [19]. However, most existing methods assume identical end-effectors or treat grasp feasibility separately for each arm. Coordination with heterogeneous grippers and explicit reasoning over grasp compatibility and switching across task phases still remains a challenge. Our approach instead performs global planning that jointly models grasp compatibility, motion feasibility, and modality transitions.
Heterogeneous Gripper Systems
There is limited prior work on planning with heterogeneous grippers to extend the manipulation capability of bimanual robots. HeterBot [20] introduces a mobile platform with two different grippers but focuses mainly on teleoperation rather than autonomous coordination. VIKI-R [21] assigns task primitives across heterogeneous platforms on high-level, but does not explicitly reason about cross-gripper low-level manipulation. [22],[23] and [24] employs heterogeneous grippers; however, its applicability is restricted to object picking tasks. Overall, existing work focuses on gripper selection for individual pick actions, while holistic task planning and feasibility across heterogeneous embodiments remain understudied.
Positioning: Most existing methods assume identical end-effectors, treat grasp selection separately from task planning, or rely on heuristic or manual switching between arms. In contrast, our method jointly reasons about grasp feasibility, grasp-pair compatibility, and arm switching in a single global planning framework.
Problem Formulation
We consider a heterogeneous bimanual manipulation system with two robot arms operating in a shared workspace. Each arm is equipped with a possibly different end-effector , leading to asymmetric grasp affordances.
Task Setup: An object must be moved from a start pose to a goal pose . A single human demonstration provides an object-centric trajectory describing the desired object motion. The trajectory is discretized into ordered segments . Due to heterogeneous gripper affordances and kinematic limits, different segments may only be executable by specific arms.
For each segment , we construct a finite set of feasible arm–trajectory pairs , where is the active robot and is a grasp-conditioned, collision-free trajectory that executes . Let denote the set of all feasible states. A plan is a sequence such that exactly one arm manipulates the object at each segment. The objective is to select a sequence that remains feasible while minimizing motion and switching effort over the full task.
Embodiment-Aware Planning Framework
Given the feasible state set for each segment, task execution is formulated as a finite-horizon discrete optimization problem. Candidate grasps on the object generate multiple trajectory instances for each arm. These candidates are evaluated for grasp validity, kinematic reachability, and collision-free motion. Each feasible state is assigned a segment cost, while transitions between states encode switching penalties and grasp compatibility. A dynamic programming procedure then computes the minimum-cost sequence of states over the task horizon, producing a feasible and smooth bimanual execution plan such as . The individual components of the framework are described in the following subsections.
Trajectory Extraction from Human Demonstration
We extract the object trajectory via model-based 6D pose tracking using FoundationPose [25], leveraging the known CAD mesh to enforce geometric consistency during manipulation (See Figure 2). While the mesh is required for planning, recent work enables reliable mesh reconstruction of novel objects from multiple RGB-D images [26]. The pipeline initializes an pose from RGB-D observations and then tracks the object by conditioning each estimate on the previous pose to maintain temporal coherence. At each timestep, pose alignment is optimized between observed depth and the rendered mesh. The resulting poses are temporally filtered to suppress high-frequency jitter while preserving rigid-body consistency, yielding smooth trajectories robust to partial occlusion. From these trajectory points, we define motion segments between consecutive poses.

Figure 2. Overview of the planning framework. For each trajectory segment, feasible robot–trajectory pairs are generated from grasp and motion feasibility checks. Segment costs capture execution quality, while transition costs encode switching penalties and grasp compatibility. A dynamic program then computes the globally optimal sequence of arm assignments over the full task horizon. Red denotes infeasible regions, blue denotes high-cost regions, and green indicates the optimal path for the task.
While Foundation Pose and trajectory extraction from Human Demonstration serve as a general approach, our method works with any method that can provide an object trajectory; thus, we also tested our approach with the RRT* algorithm for object trajectory generation.
Grasp Generation, Ranking and End-Effector Trajectory Formation
A feasible plan requires at least one robot to have a valid grasp pose for gripper at object pose . Let denote the candidate grasp set; feasibility requires . This captures gripper-dependent affordances (e.g., suction on flat surfaces and parallel-jaw grasping on accessible edges), modeled over the coupled state of both grippers and the object rather than any single gripper in isolation. We use a geometry-based grasp generator [12]. Scene-conditioned generation is avoided because a grasp infeasible in one segment may become feasible later due to changes in arm assignment or handover configuration, complicating consistent reasoning across segments.
To encourage grasp diversity, we bypass the generator’s internal ranking and instead perform entropy-based subset selection. Given candidate grasps with positive quality scores , we construct a Determinantal Point Process (DPP) using an -ensemble. The -ensemble matrix is defined as
The grasp distance metric is given by
where and denote the position and orientation (quaternion) of grasp , respectively. We select a subset of size by greedily maximizing , which jointly promotes grasp quality (via ) and spatial-rotational diversity (via ). Each selected grasp is used to generate the corresponding end-effector trajectory. These trajectories share the same time parameterization as the original object trajectory.
Segment-wise Feasibility and Cost Formulation
For each transport segment , we determine which arm can execute the motion. An action is defined as , representing transport of object from to using arm . Its feasibility depends on grasp and motion feasibility.
Grasp Feasibility
A valid grasp must exist for gripper at pose . Let denote the candidate grasp set; feasibility requires (already enforced by the filtering step in Section IV-B). We additionally compute mutually feasible grasp pairs across robots, i.e., collision-free grasp pairs that can simultaneously hold the object, which are required for handovers.
Motion Planning Feasibility
A collision-free trajectory must exist for arm to transport the object from to . Feasible motions must also exist for pick at , release at , and handover execution. If any motion primitive is infeasible, the action is rejected.
An analytical Inverse Kinematics (EAIK) [27] solver evaluates exact endpoint reachability in microseconds, acting as a rapid feasibility filter. If analytical reachability is confirmed, we then evaluate the required motion primitives.
Segment Cost
In order to avoid awkward or near-singular configurations, we define a task cost based on the Yoshikawa manipulability measure and smoothness of the transitions between points. For a configuration corresponding to action , we define two cost components.
a) Smoothness Cost.: The smoothness term penalizes large joint variations between consecutive configurations:
b) Manipulability Cost.: Given the Jacobian , the Yoshikawa manipulability index [28] for a joint configuration is
To penalize near-singular configurations along the transport segment, we evaluate the index at both the start and end configurations of the action. We define the cost
where is a small constant for numerical stability.
c) Combined Task Cost.: Let denote the feasibility of the action. The overall task cost is
where balance smoothness and manipulability. Through a grid search for and , these weights were set to and to yield the best performance. For kinematically redundant manipulators, we include the manipulability term directly in the inverse kinematics (IK) optimization too. By doing so, the solver naturally prefers well-conditioned configurations, resulting in smooth and highly manipulable robot motions.
Transition Design for Heterogeneous Handover
We define the handover primitive at a point as a dual-arm exchange between a transfer and a receiver arm using predefined grasp transforms. Each end-effector pose is computed by combining the object pose with the inverse of its grasp frame. The primitive consists of three stages: Pre-alignment, where the object is slightly offset while the receiver aligns to this intermediate pose; Receiver approach, where the receiver interpolates smoothly to the final grasp along the grasp normal; and Transfer retreat, where the original arm retreats along a fixed axis after handoff. This sequence ensures smooth, coordinated, and geometrically consistent dual-arm handovers.
Transitions between identical robot states are disallowed by assigning infinite cost to such edges. Although counter-intuitive, this is necessary for feasibility: switching planners mid-trajectory requires an instantaneous release and regrasp, which may cause the object to fall. If a grasp change is required, it can be achieved through two handovers, but handover costs discourage unnecessary frequent switching unless required for feasibility.
Following Section IV-C, we also enforce paired heterogeneous grasp feasibility by assigning infinite cost to infeasible simultaneous grasp configurations, thereby eliminating transitions corresponding to infeasible dual-arm trajectories. For feasible handover primitives, the cost is defined analogously to the segment cost in Section IV-C. Specifically, the total cost of a handover primitive is given by
where is the combined smoothness–manipulability cost defined previously.
The transition cost between states and is defined as
Here, denotes the active robot, the associated trajectory index, and is a switching penalty that regularizes inter-robot handovers.
Dynamic Programming and Optimal Sequence Recovery
The problem admits optimal substructure and is solved via dynamic programming. The value function is defined as,
which represents the minimum cumulative cost of reaching segment and terminating in state .
a) Initialization.:
b) Bellman Recursion.: For , recursion is
Infeasible transitions are excluded by setting . The recursion is equivalent to a shortest-path computation on a layered directed acyclic graph. For policy recovery, we store the predecessor mapping
Each optimal state encodes both the selected robot and its associated trajectory, i.e., . The resulting assignment sequence is , from which the robot and trajectory sequences can be extracted as and . Switching events are defined as .
Complexity Reduction via Beam Search and Stride
To mitigate the worst-case time complexity of the dynamic programming pass and ensure scalability to longer horizons, we introduce a beam search with a stride parameter. We evaluate potential dual-arm handovers only every steps (stride) rather than at every discretized trajectory waypoint. Furthermore, during the forward pass, we retain only the top lowest-cost states (beam width) for evaluation at the next step. This significantly prunes the state space and branching factor, preserving performance while drastically reducing planning times.
Experiments and Results
Our experiments are designed to systematically evaluate the proposed system framework. Specifically, we aim to answer the following questions:
Q1: To what extent does the proposed dynamic programming formulation improve task completion rates in scenarios involving heterogeneous grippers?
Q2: What is the impact of incorporating diversity-aware grasp selection and manipulability-based costs on overall system performance?
Q3: Which components are most crucial to the method’s performance, and what are the dominant failure modes observed in practice?
Experimental Setup
All experiments are conducted on a heterogeneous dual-arm platform with two 7-DoF Franka FR3 manipulators: one equipped with a parallel-jaw gripper and the other with a suction gripper. The arms are arranged with overlapping workspaces to enable coordinated manipulation and object handovers, and their relative configuration is varied to induce different reachability and feasibility constraints. This setup allows evaluation of grasp-dependent affordances and asymmetric reachability. All experiments are conducted on a workstation equipped with an AMD Ryzen Threadripper PRO 7975WX processor and an NVIDIA RTX 6000 Ada Generation GPU.
Grasp Generation, Motion Planning and Execution:
Inverse Kinematics: An EAI IK based IK solver with multi-joint enumeration (using the redundancy of 7 DOF) to check for the most continuous and manipulable joint pose.
Grasp Generation: 6-DoF grasp candidates are sampled via GraspGen[12] for both parallel-jaw and suction grippers, followed by collision-based feasibility filtering. 40 grasps are sampled per gripper type, except in Task A: Plate from Table to Shelf (as described below) due to its restrictive environment, thus requiring more diverse sampling (60 grasps).
Collision modeling: Full robot self-collision, object and environment meshes, and dual-arm interactions checked using MuJoCo’s default collision engine.
Simultaneous dual-arm motions use an asymmetric leader-follower setup: Arm A follows the planned, while Arm B tracks a reference via a reactive mink QP with collision-avoidance constraints. We have used mink here due to its natural incorporation with mujoco and being a QP solver it can incorporate posture tasks (for warm starts), collision handling and velocity/joint limits in its optimization while finding optimal control steps to control the arm (during planning we wanted speed in order to iterate fast upon a lot of possible configuration and solutions, thus used EAIK). Arm A’s joints are pinned, and transport segments are executed as constrained joint tracking to faithfully follow the object trajectory.
Task Design
The tasks are designed with two objectives: (1) to enforce heterogeneous gripper use by making at least one trajectory segment infeasible for a single arm, requiring a handover; and (2) to introduce reachability-based infeasibility, demonstrating that the framework generalizes beyond collision constraints. We design both single-switch and two-switch tasks, described below (see Figure 3 for task examples).

Figure 3. The figure illustrates the four tasks evaluated in our framework. The tasks are specifically designed to highlight the advantages and coordination challenges of the proposed heterogeneous bimanual setup during real-world experiments. The execution sequence of the tasks is from left to right.
Task A: Plate from Table to Shelf A flat plate is positioned flush on the table to mandate a suction pick, while the destination shelf’s narrow clearance prohibits
suction entry, deliberately forcing a handover for final placement.
Task B: YCB Plate from Shelf to Table The recessed YCB plate blocks suction access, requiring a parallel-jaw grasp; however, tight boundaries at the target cause gripper collisions, necessitating a handover to suction for top-down placement on the table.
Task C: Mug Inversion An inverted mug prevents parallel-jaw rim grasping, requiring suction pickup. A handover to the parallel jaw then enables flipping and upright placement in a container.
Task D: Plate from box to box A flat plate is transferred between two containers. Only the suction gripper can pick and place at the containers due to collision constraints for the parallel jaw. However, intermediate regions are unreachable by suction due to a wall, along the constraint that the plate can’t go over it, thereby requiring two handovers.
Baselines
Exisiting object manipulation research with heterogeneous grippers has mainly focused on grasp generation and picking. We therefore compare our method against the following configurations:
| Method | Assign (%) | Task Success (%) | Handover (%) | Avg. Switches | Plan Time (s) | Exec Time (s) |
| Single-Arm Baseline | ||||||
| Single Robot | 0.0 | 0.0 | - | 0 | - | - |
| Bimanual Method | ||||||
| Greedy (beam = 1) | 30.0 | 20.0 | 75.0 | 1.5 | 24.03 | 95.8 |
| No-Diversity | 75.0 | 67.5 | 100.0 | 1.3 | 1015.87 | 84.8 |
| No-Manipulability () | 80.0 | 52.5 | 89.1 | 2.7 | 502.43 | 138.7 |
| No-Switching Penalty () | 80.0 | 25.0 | 81.6 | 10.7 | 154.60 | 286.5 |
| Proposed (Exact DP) | 80.0 | 72.5 | 100.0 | 1.3 | 532.96 | 92.0 |
| Proposed (Beam 10, Stride 3) | 72.5 | 65.0 | 100.0 | 1.3 | 91.05 | 87.9 |
Table 1. TABLE I: OVERALL PERFORMANCE COMPARISON ACROSS METHODS. Task Success denotes the end-to-end completion rate (assignment success valid execution success). Handover (%) measures the success of valid transitions, excluding drops that occurred prior to the handover. Plan Time is the assignment computation duration over sampled grasp trajectories, and Exec Time is the task execution duration. Avg. Switches reflects the mean number of arm transitions.
| Task | Assign (%) | Task Success (%) | Handover (%) | Avg. Switches | Plan Time (s) | Exec Time (s) |
| Task A: Plate from table to shelf | 70.0 | 50.0 | 100.0 | 1.0 | 129.87 | 75.9 |
| Task B: YCB Plate from shelf to table | 60.0 | 60.0 | 100.0 | 1.0 | 99.37 | 81.6 |
| Task C: Mug Inversion | 80.0 | 80.0 | 100.0 | 1.0 | 84.67 | 63.8 |
| Task D: Plate from box to box | 80.0 | 70.0 | 100.0 | 2.0 | 50.30 | 127.1 |
Table 2. TABLE II: PER-EXPERIMENT EVALUATION (PROPOSED: BEAM 10, STRIDE 3). A task-specific breakdown of the performance metrics defined in Table I.
Greedy Reachability
Greedy Reachability: At each segment, the robot with a feasible grasp and higher instantaneous manipulability is greedily selected (equivalent to a beam width of 1).
No-Diversity Grasp Selection
No-Diversity Grasp Selection: The top- grasps are selected purely based on grasp quality scores, without applying DPP-based diversity filtering.
We also evaluated single-robot execution, which achieved 0% success across all scenarios, confirming that the tasks require heterogeneous bimanual coordination.
Quantitative Results
Table I summarizes performance across 520 simulated runs to address Q1, evaluating success rates, switches, and timing. Our exact DP method achieves 72.5% task success compared to the greedy baseline’s 20.0%, demonstrating the critical need for global look-ahead over myopic reachability decisions in complex transport tasks. Furthermore, excluding DPP-based grasp diversity (67.5% success) heavily penalizes planning time (1015.87 s), as concentrating grasps in similar poses redundantly expands the dynamic programming search space without improving workspace reachability.
While global optimization is vital, the Beam 10, Stride 3 configuration offers a practical computational trade-off. It drastically cuts average planning time from 532.96 s (Exact DP) to 91.05 s, while still maintaining a 65.0% task success rate and a 100.0% handover success rate.
Per-Experiment Details and Evaluation
Table II details performance across the four tasks using the practical Beam 10, Stride 3 configuration, maintaining a robust 100.0% handover success globally. Task C: Mug Inversion achieved the highest success (80.0%) and rapid execution (63.8 s).
Conversely, Task A: Plate from Table to Shelf proved most challenging (50.0% success, 129.87 s planning time) due to environmental complexity and increased grasp sampling (60 vs. 40). Failures primarily stemmed from the inability to find valid robot assignments or confined insertion tolerances that forced suboptimal grasps, leading to transport slippage. Tasks B and D maintained balanced rates of 60.0% and 70.0%.
Ablation Studies
We evaluate the contribution of key components via ablation studies, maintaining the rest of the pipeline unchanged (Table I).
Switching Penalty and Frequency ()
Switching Penalty and Frequency (): Removing the switching penalty leads to excessive, unsystematic handovers, increasing the average switches from 1.3 to 10.7. Consequently, task success plummets to 25.0%, handover success drops to 81.6%, and execution time increases by 211.4%, demonstrating that unregularized handovers compound failure points. Conversely, systematic switching remains highly effective: valid executed plans requiring two switches maintain a robust 87.0% task success rate, compared to 90.3% for one-switch plans. This confirms that deliberate handovers safely navigate restricted workspaces without the severe performance drops of unregularized switching.
Manipulability Cost ()
Manipulability Cost (): Disabling the manipulability term drops overall task success to 52.5% and handover success to 89.1%. Without this objective, the solver frequently selects poorly conditioned kinematic states, resulting in jerky trajectories and unstable grasps. Prioritizing manipulability is therefore vital to smoothly and safely execute the complex, contact-rich motions required during physical dual-arm handovers.
Real-World Validation via Sim and Real Digital Twin
To validate our simulated results, we deployed the framework on a physical dual Franka FR3 setup using Task B: YCB Plate from Shelf to Table and Task C: Mug Inversion. We bridged the sim-to-real gap via a ROS 2 and MuJoCo-based digital twin, computing dynamically verified trajectories with the Beam=10, Stride=3 configuration (which demonstrated a 60% and 80.0% task success rate). During physical deployment, impedance control safely absorbed minor calibration offsets during contact-rich handovers, successfully replicating the digital twin’s precise and safe bimanual behavior.
Limitations and Conclusion
Limitations We rely on pre-computed CAD meshes for tracking and DPP-based grasp generation, which limits zero-shot deployment on novel objects. Recent advances such as FoundationPose [25] have enabled on-the-fly mesh reconstruction from RGB-D observations and reduce this dependency. The framework also assumes a fixed, human-provided object trajectory. Future work could allow embodiment-aware trajectory refinement, where the planner slightly adjusts the demonstrated trajectory to achieve more stable handovers.
In conclusion, we present an embodiment-aware planning framework for heterogeneous bimanual manipulation. By jointly optimizing segment reachability, asymmetric grasp compatibility, and transition penalties, our method translates a single human demonstration into a globally optimal execution plan. Supported by extensive simulation sweeps and validated via a real-world digital twin, this approach systematically elevates simple industrial grippers toward the versatile execution of complex manipulation tasks.
References
- [1]W. Lin, S. Yuan, C. Gong, and J. Wang, “A comprehensive review of dexterous manipulation and learning methods for robotic hands,” Biomimetics, vol. 10, no. 6, p. 400, 2025.
- [2]A. Bicchi and V. Kumar, “Robotic grasping and contact: A review,” Proceedings of the IEEE International Conference on Robotics and Automation, pp. 348–353, 2000.
- [3]J. Kerr and B. Roth, “Analysis of multifingered hands,” The International Journal of Robotics Research, vol. 4, no. 4, pp. 3–17, 1986.
- [4]D. J. Montana, “The kinematics of contact and grasp,” The International Journal of Robotics Research, vol. 7, no. 7, pp. 17–32, 1988.DOI
- [5]M. R. Cutkosky, “On grasp choice, grasp models, and the design of hands for manufacturing tasks,” IEEE Transactions on Robotics and Automation, vol. 5, no. 3, pp. 269–279, 1989.DOI
- [6]OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba, “Learning dexterous in-hand manipulation,” The International Journal of Robotics Research, vol. 39, no. 1, pp. 3–20, 2020.arxiv.org/abs/1808.00177
- [7]A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learning complex dexterous manipulation with deep reinforcement learning and demonstrations,” in Robotics: Science and Systems (RSS), 2018.arxiv.org/abs/1709.10087
- [8]I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, and W. Zaremba, “Solving rubik’s cube with a robot hand,” arXiv preprint arXiv:1910.007113, 2019.
- [9]M. S. Islam, Z. Najdovski, M. A. Hosen, and V. T. Huynh, “An in-depth review of dual-arm robot learning methods with advances and challenges in variable compliance-aware control for uncertain tasks,” Annual Reviews in Control, vol. 61, p. 101050, 2026.DOI
- [10]R. Shome, K. Solovey, J. Yu, K. Bekris, and D. Halperin, “Fast, high-quality dual-arm rearrangement in synchronous, monotone tabletop setups,” in International Workshop on the Algorithmic Foundations of Robotics. Springer, 2018, pp. 778–795.arxiv.org/abs/1810.12202
- [11]F. J. Martínez-Peral, J. B. Méndez, D. Mronga, J. V. Segura-Heras, and C. Perez-Vidal, “Trajectory planning system for bimanual robots: Achieving efficient collision-free manipulation,” Robotics and Autonomous Systems, vol. 194, p. 105118, 2025.
- [12]A. Murali, S. Fidler, A. Garg, S. James, A. Khazatsky, A. Kurenkov, Y. Li, T. Zhang, Y. Zhang, R. Zhong, et al., “Graspgen: A diffusion-based approach for 6-dof grasp generation on unseen objects,” arXiv preprint arXiv:2507.13097, 2025.
- [13]M. Djeha, A. Dallard, A. Zermane, P. Gergondet, and A. Kheddar, “Human-robot handovers using task-space quadratic programming,” in 2022 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 2022, pp. 518–523.
- [14]B. Cohen, M. Phillips, and M. Likhachev, “Planning single-arm manipulations with n-arm robots,” Proceedings of the International Symposium on Combinatorial Search, vol. 6, pp. 226–227, 09 2021.DOI
- [15]K. Black, N. Brown, D. Driess, A. Esmail, M. Equi, C. Finn, N. Fusai, L. Groom, K. Hausman, B. Ichter, et al., “pi₀: A vision-language-action flow model for general robot control,” arXiv preprint arXiv:2410.24164, 2024.arxiv.org/abs/2410.24164
- [16]H. Chen, J. Xu, L. Sheng, T. Ji, S. Liu, Y. Li, and K. Driggs-Campbell, “Learning coordinated bimanual manipulation policies using state diffusion and inverse dynamics models,” in 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025.arxiv.org/abs/2503.23271
- [17]Z. Geng, Z. Yang, W. Xu, W. Guo, and X. Sheng, “A globally guided dual-arm reactive motion controller for coordinated self-handover in a confined domestic environment,” Biomimetics, vol. 9, no. 10, p. 629, 2024.DOI
- [18]C.-C. Wong, C.-Y. Tsai, Y.-C. Lai, and S.-W. Wong, “Manipulability-aware task-oriented grasp planning and motion control with application in a seven-dof redundant dual-arm robot,” Electronics, vol. 13, no. 24, p. 5025, 2024.DOI
- [19]S. Wang, X. Liu, C. C. L. Wang, and J. Liu, “Physics-aware iterative learning and prediction of saliency map for bimanual grasp planning,” Computer Aided Geometric Design, vol. 111, p. 102298, 2024.arxiv.org/abs/2404.08944
- [20]Y. Wang, X. Wang, C. Cao, X. Wang, M. Ding, F. Li, J. Duan, J. Liu, and J. Wang, “Heterobot: A heterogeneous mobile manipulation robot for versatile operation,” IET Cyber-Systems and Robotics, vol. 5, no. 1, p. e12068, 2023.DOI
- [21]Y. Kang, J. Frey, G. Yang, and J. Peters, “Viki-r: Coordinating embodiment and task for cross-platform generalization in robot manipulation,” arXiv preprint arXiv:2506.09049, 2025.
- [22]N. Nechyporenko, A. Morales, E. Cervera, and A. P. del Pobil, “A practical approach for picking items in an online shopping warehouse,” Applied Sciences, vol. 11, no. 13, 2021. [Online]. Available: https://www.mdpi.com/2076-3417/11/13/5805DOI
- [23]A. Wu and D. Kruse, “Graspable object picking with bimanual nonprehensile manipulation,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 5669–5676.
- [24]J. Mahler, M. Matl, V. Satish, M. Danielczuk, B. DeRose, S. McKinley, and K. Goldberg, “Learning ambidextrous robot grasping policies,” Science Robotics, vol. 4, no. 26, p. eaau4984, 2019. [Online]. Available: https://www.science.org/doi/abs/10.1126/scirobotics.aau4984DOI
- [25]B. Wen, W. Yang, J. Kautz, and S. Birchfield, “Foundationpose: Unified 6d pose estimation and tracking of novel objects,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 17 868–17 879.arxiv.org/abs/2312.08344
- [26]B. Wen, J. Tremblay, V. Blukis, S. Tyree, T. Müller, A. Evans, D. Fox, J. Kautz, and S. Birchfield, “BundleSdf: Neural 6-dof tracking and 3d reconstruction of unknown objects,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 606–617.arxiv.org/abs/2303.14158
- [27]D. Ostermeier, J. Külz, and M. Althoff, “Automatic geometric decomposition for analytical inverse kinematics,” IEEE Robotics and Automation Letters, 2025.DOI
- [28]T. Yoshikawa, “Manipulability of robotic mechanisms,” The international journal of Robotics Research, vol. 4, no. 2, pp. 3–9, 1985.DOI