betaFeedback

WorksComputer Science

HuG: Learning Humanoid Whole-Body Affordance-Aware Grasping with Active Perception

Abstract

Acquiring objects in human environments is hard for humanoids: targets are partially occluded under egocentric viewpoints, and reaching them demands coordinated whole-body motion that keeps the target visible and reachable while the base stays stable. We present HuG, a framework for whole-body affordance-aware grasping with active perception that couples perception and action through a Perceive-Align-Grasp pipeline: belief-driven active perception resolves occlusion by treating viewpoint as an explicit decision variable; a point-cloud-conditioned affordance policy proposes whole-body grasps; and an affordance-aware alignment stage repositions the body to the most executable configuration. A key insight is that, on a floating base, kinematic validity is not enough: we curate training data with controller-in-the-loop dynamics simulation, rejecting 73% of kinematically valid grasps that would destabilize the deployment controller, so the policy learns kinodynamically feasible affordances. Trained entirely in simulation with zero real demonstrations, HuG attains 87.5% real-world success on a multi-level shelf task, generalizes to novel objects, and holds at least 73% success at every tested height. Beyond generalizing within its training distribution, the policy extrapolates spatially: at scale in simulation it succeeds at table heights well below any seen during training, indicating that kinodynamically filtered synthetic data grounds affordances in object geometry and whole-body feasibility rather than in memorized configurations.

xuanbinpeng.com/hug

Conference on Robot Learning (CoRL) 2026, Austin, TX, USA

Introduction

Humanoid robots promise to bring general-purpose object acquisition into human environments. Yet in realistic homes and labs, targets are rarely presented in a clean tabletop configuration: they are frequently partially occluded under egocentric viewpoints and spread across diverse spatial configurations, and reaching them demands coordinated whole-body motion—turning the head, bending the torso, squatting—to maintain visibility, reachability, and stability at once. Consider retrieving an item from a cluttered multi-level shelf: the target may be visible only from a specific angle, sit at a height requiring a deep squat or a full extension, and afford a stable grasp only from particular approach directions. Such tasks demand tightly coupling perception with whole-body action: not as independent modules, but as stages in which viewpoint, posture, and grasp decisions jointly shape one another.

Three threads of prior work have each made striking progress on this coupling, yet each leaves a critical gap. (i) Imitation from real demonstrations scales humanoid manipulation through teleoperation [1, 2, 3, 4], with ACT [5]-style architectures and diffusion policies such as Via [6] supporting long-horizon tasks; but these policies are bounded by their demonstration coverage, generalizing poorly to objects and layouts outside it; worse, on a floating base every action error also shifts the body, so errors compound over long-horizon rollouts into end-effector drift that centimeter-level grasping cannot absorb. (ii) Large-scale synthetic learning enables zero-shot sim-to-real via end-to-end policies [7, 8, 9, 10] and modular grasp synthesis [13, 14, 15, 16, 17] [11, 12]; but end-to-end humanoid policies reason about dynamics only implicitly, while grasp planners are built for fixed-base tabletops, so a kinematically valid grasp can be dynamically infeasible once a floating-base humanoid must execute it under whole-body control. Neither line explicitly enforces kinodynamic whole-body feasibility during data generation. (iii) Active perception regulates viewpoints, from classical active perception and active SLAM [18, 19] to learned perceptual strategies [20, 21, 22] [6]; but on humanoids, viewpoint regulation either emerges implicitly from demonstrations, bounded by their coverage [6], or is confined to the head with the body left passive [23]. Yet a body-attached camera binds viewpoint to whole-body posture [24, 25, 26], so resolving occlusion can require lowering the torso or leaning, not merely looking. Humans, by contrast, couple perception and whole-body action instinctively: they lean past occlusions, squat to low shelves, and turn to approach from stable angles. Current humanoid grasping systems lack this coupling.

We present HuG, a hierarchical framework for whole-body affordance-aware grasping with active perception ( Figure 1) that realizes this coupling through a Perceive–Align–Grasp pipeline: (1) **belief-driven active perception, a Bayes-style belief filter over an egocentric viewpoint grid that resolves occlusion by systematic search, treating viewpoint as an explicit decision variable rather than emergent behavior; (2) kinodynamically filtered affordance learning, where an affordance is a kinodynamically feasible whole-body grasp configuration for the observed object: a point-cloud-conditioned CVAE is trained on 600k synthetic trajectories that each pass controller-in-the-loop dynamics simulation, which rejects 73% of kinematically valid grasps that would destabilize the controller; and (3) whole-body affordance-aware alignment, which samples KK diverse candidates and repositions the whole body to the configuration of highest execution quality rather than highest grasp confidence. Trained entirely from synthetic data with zero real demonstrations, our system attains 87.5% real-world success on a multi-level shelf task, generalizes across novel objects (90/85% ID/OOD), holds ≥73%\geq73\% at every tested height, and transfers zero-shot to diverse everyday scenes with cluttered backgrounds and unseen objects. At scale in simulation, the same policy extrapolates to table heights well below its training range, where every single-component ablation loses 25–81 points: the learned affordances are grounded in object geometry and whole-body feasibility, not in memorized configurations.

Humanoid whole-body affordance-aware grasping

Figure 1. Humanoid whole-body affordance-aware grasping. Top: our system runs zero-shot on a Unitree G1, coupling active perception with whole-body control to discover and grasp objects across diverse everyday scenes, from deep squats at low tables to full extension at high shelves. Bottom: the policy is trained entirely on large-scale synthetic data generated and filtered with kinodynamic feasibility.

Our contributions are:

  • Kinodynamic-aware whole-body affordance learning. A large-scale synthetic data engine that co-optimizes whole-body posture with the hand and enforces controller-in-the-loop dynamic feasibility, so the learned affordances are kinodynamically feasible by construction.

  • A zero-demonstration Perceive–Align–Grasp system. A complete humanoid pipeline that discovers occluded targets, repositions the whole body to an executable grasp configuration, and executes it under whole-body control, without any real-world demonstration.

  • Extensive real-world validation. The complete pipeline runs zero-shot on a Unitree G1 humanoid, outperforming imitation and grasp-planning baselines and generalizing to unseen objects, heights, and everyday scenes.

Related Work

Humanoid Whole-Body Loco-Manipulation

Coordinating locomotion and manipulation on humanoids couples floating-base stability with end-effector precision. Learned whole-body controllers now track expressive motions and coordinate legs and arms under contact and force disturbances [27, 28, 29, 30, 31, 32, 33, 34, 35, 36], and teleoperation systems built on them have made whole-body demonstration collection practical enough to train capable manipulation policies [1, 2, 3, 4, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52]. These policies, however, remain bounded by their demonstration coverage, and on a floating base, pose drift compounds over long-horizon rollouts. Sim-to-real visual policies [7, 8, 9, 10, 17, 22, 25, 53, 54, 55, 56] remove the demonstration bottleneck, most directly VIRAL [7], but leave perception to data scaling. In contrast, we formulate perception as an active optimization problem, coordinating whole-body motion to search for and maximize target observability.

Learning from Synthetic Data

The cost of real interaction data has driven a shift toward simulation-based learning, from domain randomization [57, 58, 59, 60, 61, 62, 63, 64] and synthetic grasp datasets [13, 14, 15, 65, 66, 67, 68] [11, 12] to generative, automated pipelines [69, 70, 71, 72, 73]. Most directly comparable to ours, billion-scale grasping foundation models [16] backed by GPU-accelerated simulators [74, 75] and large asset libraries [76] achieve zero-shot generalization in cluttered scenes. These efforts, however, predominantly target fixed-base settings or omit the dynamical plausibility of generated samples, admitting candidates that are kinematically valid but dynamically infeasible under closed-loop whole-body control. In contrast, we co-optimize whole-body posture with the hand for each grasp and enforce kinodynamic feasibility under whole-body execution on a floating base, with the deployment controller in the loop.

Active Perception for Manipulation

Active perception spans reconstruction, recognition, and manipulation [18, 19, 77, 78, 79, 80, 81], and has been coupled with grasping on fixed arms and mobile manipulators [20, 21, 82, 83], where the camera moves largely independently of balance. On humanoids the camera is body-attached, so a viewpoint change is a whole-body motion [24, 25, 26], and reaching a viewpoint is itself a whole-body planning problem [84]. Recent humanoid systems address this only partially: ViA [6] learns coordinated head–bimanual control from demonstrations but is bounded by their coverage, and Choice Policy [23] regulates an active head while leaving the body passive, so the camera cannot reach the configurations occlusion demands.

Method

Our framework comprises two complementary pipelines (Figure 2). The system pipeline follows a Perceive–Align–Grasp structure: active perception (§ 3.2) discovers the target under egocentric occlusion; the affordance model proposes whole-body grasp configurations, which alignment ranks and repositions the body toward; and a collision-aware planner executes the selected grasp through the whole-body controller (§ 3.4). The training pipeline builds large-scale synthetic data filtered for kinodynamic feasibility, yielding the affordance model that drives both alignment and grasping (§ 3.3). Both share one hierarchical whole-body controller coupling upper-body IK with an RL lower-body policy (§ 3.1).

System and training pipelines

Figure 2. System and training pipelines. The system pipeline (top) follows a Perceive–Align–Grasp structure, combining active perception, whole-body alignment, and robust grasp execution. The training pipeline (bottom) constructs large-scale synthetic data with kinodynamic filtering to learn affordance-aware whole-body grasping.

Hierarchical Whole-Body Controller

Whole-body grasping places two competing demands on the controller: task-space precision for end-effector placement, and floating-base stability across the diverse postures that different object heights and reaches require. End-to-end policies struggle to meet both under the tight tolerances of zero-shot sim-to-real transfer.

A hierarchical two-layer design decouples these concerns. The high-level whole-body IK [85] maps desired end-effector poses (wrist, hand, and head, in the world frame) to upper-body joint targets qupper\mathbf{q}_{\mathrm{upper}} and lower-body commands ulower=(vx,vy,vyaw,h)\mathbf{u}_{\mathrm{lower}} = (v_x, v_y, v_{\mathrm{yaw}}, h), with hh the pelvis height. For the low-level RL locomotion policy, we use the decoupled whole-body controller of SONIC (GR00T) [38], which tracks ulower\mathbf{u}_{\mathrm{lower}} from proprioception, outputting lower-body targets that realize the motion while maintaining balance.

This coupling poses a challenge: as the RL policy shifts pelvis height or base position, the root pose drifts in the world frame, and an IK solver blind to this drift renders world-frame targets unreachable—fatal for centimeter-level grasping. A closed-loop pelvis compensator closes the gap, pre-warping every task-space target by the low-pass-filtered mismatch between the IK-assumed and FK-realized pelvis pose (App. § 8 gives the SE(3)SE(3) update and filter). Crucially, the same controller drives synthetic-data filtering (§ 3.3), so training data is supported on actions the deployment system can actually execute.

Active Perception

We cast object discovery under egocentric occlusion as sequential viewpoint selection that maximizes the probability of detecting the target. The module is a finite-state machine with a Search mode (global exploration) and an Inspect mode (local refinement after detection); the target is named by a text query to Grounding DINO [86] (e.g., “red cup”), making acquisition language-conditioned without a separate language module.

Search. We maintain a normalized belief p(i,j)p(i,j), the probability that the target is visible from camera pose (yawi,zj)(\mathrm{yaw}_i,z_j), over a discretized grid of camera yaw and height. Each camera pose is a head-pose target for the whole-body controller (§ 3.1), so reaching a viewpoint is a whole-body motion: the robot squats or rises to look, not only turns its head. For a candidate pose TT, the view frustum projects to a mask M(T)\mathcal{M}(T) with probability of detection PoD⁡(T)=∑(i,j)∈M(T)p(i,j)\operatorname{PoD}(T) = \sum_{(i,j)\in\mathcal{M}(T)} p(i,j). We execute the viewpoint maximizing score⁡(T)=PoD⁡(T)−wc(dyaw2+dz2)−woroverlap\operatorname{score}(T) = \operatorname{PoD}(T) - w_c(d_{\mathrm{yaw}}^2 + d_z^2) - w_o r_{\mathrm{overlap}}, with (dyaw,dz)(d_{\mathrm{yaw}},d_z) the displacement from the current camera pose, trading detection likelihood against camera motion and revisiting.

Belief update. After observing Grounding DINO confidence sts_t at the executed viewpoint TtT_t, cells in the frustum Mt\mathcal{M}_t are updated by a Bayes-style multiplicative rule,

pt+1(i,j)∝λt(i,j) pt(i,j),∑i,jpt+1=1,(1)p_{t+1}(i,j) \propto\lambda_t(i,j)\,p_t(i,j), \qquad\sum_{i,j} p_{t+1}=1, \tag*{(1)}

with a detection-dependent likelihood ratio λt\lambda_t that amplifies frustum cells on a hit (λt=1+αst−τ1−τ\lambda_t = 1 + \alpha\frac{s_t-\tau}{1-\tau} for st≥τs_t \ge\tau, rising with confidence above threshold τ\tau), decays them on a miss (λt=1−γ\lambda_t = 1-\gamma for st<τs_t < \tau), and leaves cells outside the frustum unchanged. Hits concentrate belief; misses redistribute mass to unvisited viewpoints, driving systematic exploration.

Inspect and termination. Search hands off to Inspect once max⁡i,jp(i,j)\max_{i,j} p(i,j) stays above a convergence threshold for kk steps; Inspect then refines the viewpoint by point-cloud visibility quality (visible-point count, depth continuity, normal stability) and terminates when observation quality saturates, passing a stable observation to alignment.

Learning Kinodynamic-Aware Whole-Body Affordance

Given a stable observation, the robot must decide how to approach and grasp the object with its whole body. We learn a whole-body affordance policy that maps a single-view segmented point cloud P\mathbf{P} to candidate keyframe actions, each a (pre-grasp, grasp, post-grasp) sequence of end-effector poses and hand commands executed by the whole-body controller. Every training sample has survived controller-in-the-loop execution, so the learned affordances are kinodynamically feasible by construction.

Large-scale World-frame Randomization. Unlike fixed-base grasping, where object variation is largely in-plane, a floating-base humanoid must adapt its entire body to different object heights and reaches. We randomize table geometry and object placement across a wide reachable workspace in the world frame, with objects drawn from a large-scale mesh dataset under randomized scale and mass (App. § 9). This forces diverse whole-body strategies: squatting for low objects, extending for far ones, and adapting the whole-body configuration to everything in between.

Kinodynamic-aware Data Engine. We build the dataset through a three-stage pipeline: (i) whole-body–hand grasp co-optimization jointly synthesizes the whole-body configuration and hand joints for each of NcandN_{\mathrm{cand}} candidates by energy-based optimization in joint space [85], coupling hand contact with a reachable whole-body posture, unlike gripper-based or fixed-upper-body generators that ignore humanoid morphology (energy terms in App. § 9); (ii) motion planning builds collision-free pre-grasp→\rightarrowgrasp→\rightarrowpost-grasp trajectories; (iii) controller-in-the-loop dynamics simulation rolls out each trajectory under the same WB-IK + RL stack used at deployment. A trajectory is kept only if it satisfies three criteria: task success (object lifted ≥5\ge5 cm and held), stable execution (no fall, bounded base attitude), and trackability (end-effector error within deployment tolerances). This last stage is the key to kinodynamic awareness: in our profiling it rejects 73% of kinematically valid grasps (§ 5.1), because the whole-body controller cannot stay stable through the reach or IK tracking diverges under interaction forces. We further filter by camera visibility to avoid degenerate observations (App. § 9).

Observation and Action Spaces. The observation is a segmented single-view partial point cloud P∈RN×3\mathbf{P} \in\mathbb{R}^{N \times3} of the target, transformed from camera to robot task frame to match the egocentric deployment view. The action is a three-stage keyframe trajectory a=(apre,agrasp,apost)\mathbf{a} = (\mathbf{a}^{\mathrm{pre}}, \mathbf{a}^{\mathrm{grasp}}, \mathbf{a}^{\mathrm{post}}), where each stage as=[ps,rs,hs]\mathbf{a}^{s} = [\mathbf{p}^{s}, \mathbf{r}^{s}, \mathbf{h}^{s}] gives end-effector translation ps∈R3\mathbf{p}^{s} \in\mathbb{R}^{3}, orientation rs∈S3\mathbf{r}^{s} \in\mathbb{S}^{3} (unit quaternion), and hand joint targets hs∈R6\mathbf{h}^{s} \in\mathbb{R}^{6}.

Whole-body Affordance Policy. Since a point cloud admits multiple feasible grasps, we model the affordance as a multimodal distribution with a point-cloud conditioned CVAE: PointNet [87] encodes P\mathbf{P}, conditioning a CVAE that decodes the keyframe action from a sampled latent. At inference we draw several latents for diverse proposals, ranked and executed by the alignment stage (§ 3.4; architecture and training in App. § 10).

Whole-Body Alignment and Execution

Once active perception settles on a stable viewpoint, the target is segmented with SAM [88] into an object-only point cloud and passed to the affordance model (§ 3.3) for grasp proposal and whole-body alignment.

Candidate Generation and Ranking. The CVAE decoder draws KK latents into KK diverse candidate keyframe sequences {ak}k=1K\{a_k\}_{k=1}^{K}, trained only on kinodynamically verified data, its proposals are strongly biased toward configurations the whole-body controller can execute. For each, we solve whole-body IK for the pre-grasp target and rank by rank⁡(ak)=wikQIK(ak)−wprox∥qk−qcurr∥\operatorname{rank}(a_k) = w_{\mathrm{ik}}Q_{\mathrm{IK}}(a_k) - w_{\mathrm{prox}}\lVert q_k - q_{\mathrm{curr}}\rVert, where QIKQ_{\mathrm{IK}} is IK quality (inverse residual norm) and ∥qk−qcurr∥\lVert q_k - q_{\mathrm{curr}}\rVert the joint-space reconfiguration cost. This favors kinematically achievable proposals that need minimal whole-body motion, which matters for floating-base stability. IK evaluation is batched and lightweight (∼1\sim1 ms for all KK candidates).

Execution. The top-ranked candidate is executed via collision-aware Cartesian motion planning through the whole-body controller (§ 3.1); if planning fails, e.g., due to scene collision, the system falls back to the next candidate in ranked order. The robot approaches the pre-grasp pose, closes the hand, and performs the post-grasp lift. Unlike methods that select from a predefined library of stable poses, our alignment configurations are learned from kinodynamically filtered data, object-specific via point-cloud conditioning, and dynamics-aware by construction.

Experiments

We evaluate our system through three complementary evaluations: Task 1 tests full pipeline integration under realistic shelf conditions; Task 2 isolates manipulation from perception with oracle observations on real hardware; Task 3 measures spatial and object generalization at scale in simulation, including extrapolation beyond the training height range. Beyond aggregate comparison, we investigate whether large-scale kinodynamically filtered synthetic training generalizes across the full humanoid workspace and novel objects, which pipeline component fails in which regime, and how performance scales with synthetic data volume.

Experimental Setup

Hardware. We evaluate on the Unitree G1 humanoid with a 3-DOF active head [39], ZED Mini stereo camera, and Inspire dexterous hands. All inference runs on an external desktop (RTX 5090); commands are transmitted via Ethernet to the onboard computer.

Task 1: Full Pipeline (Perceive →\to Align →\to Grasp). Objects are placed on a multi-level shelf with partial occlusions and randomized heights. The robot starts from a nominal pose with no prior knowledge of the target and must autonomously discover, align to, and grasp the object. We run N=40N=40 trials over 10 YCB [89] objects (5 ID, 5 OOD); OOD objects are held out from the training set. Target objects are specified via natural language, conditioning the DINO text prompt for detection.

Task 2: Align & Grasp (Oracle Perception). To isolate manipulation from perception confounds, we provide oracle observations on a height-adjustable table at five controlled levels: H1 (0.50 m), H2 (0.60 m), H3 (0.70 m), H4 (0.80 m), H5 (0.85 m); 10 objects ×\times 5 heights ×\times 3 trials = 150 (75 ID, 75 OOD). Task 2 is the hardware anchor for the height profile measured at scale in Task 3. ACT and ViA receive demonstrations at H2 and H4 only, so the remaining heights measure their sensitivity to demonstration coverage; our method trains on continuously randomized heights in simulation.

Task 3: Large-Scale Simulation Evaluation. To evaluate spatial generalization at a scale impractical on hardware, we sweep 17 table heights (0.10–0.90 m in 5 cm steps) with randomized planar placement and randomized object yaw, over two object catalogs: YCB (12 everyday objects) and RoboCasa [90] (11 objects spanning a broader shape distribution: smaller, thinner, and more elongated geometries, e.g., peeler, whisk, chicken drumstick). Six system variants (Ours and five ablations/baselines, below) are evaluated on identical scene sets, so every delta is paired; ∼18,800 trials in total. Success is scored from telemetry under a criterion stricter than the real-world one: the object must be lifted ≥5 cm and held for 2 s with bounded in-hand slip, and the robot must remain upright (App. E.5). Crucially, training tables span only 0.40–0.90 m: all evaluations at 0.10–0.35 m probe extrapolation below the training range.

Metrics. Success rate (SR): object lifted ≥5 cm and stably held. For Task 1 we additionally report Det. (%): object successfully localized, and Gr∣DetGr|Det (%): grasp success given detection, which decomposes failures into perception vs. manipulation stages.

Baselines. We compare against end-to-end imitation methods, modular grasp planners, and ablations of our pipeline. Vanilla ACT [5]: end-to-end visuomotor policy with a fixed head and no active viewpoint control. ACT + Active Head: ACT with the active head enabled during both demonstration collection and policy execution, isolating the contribution of active perception. ViA [6]: diffusion policy with learned active perception, adapted to floating-base control. GraspNet + WB-IK [11]: parallel-jaw grasp poses from the pretrained checkpoint, mapped to dexterous-hand pre-grasps and executed via our WB-IK controller (+ AP: adding our active perception). Dex1B (adapted) + WB-IK + AP [12]: large-scale dexterous grasp framework adapted to our humanoid. Data is generated across the full workspace with upper-body optimization (arm IK only, no whole-body–hand co-optimization), without dynamics filtering or alignment. Dex1B shares our CVAE + PointNet architecture and hand type, directly isolating the contribution of kinodynamic filtering and whole-body alignment. ACT and ViA are trained on teleoperated demonstrations of ID objects (Task 1: 70 full-pipeline demonstrations per object at every evaluated shelf level, with randomized horizontal placement, so their height coverage matches the test conditions; Task 2: grasp-only at H2/H4). Our method uses zero real demonstrations: 600k synthetic trajectories only.

MethodSR (%)Stage (%)
AllID/OODDet.Gr|Det
Baselines
Vanilla ACT32.555/1052.561.9
ACT + Active Head40.065/1557.569.6
ViA (adapted)50.085/1570.071.4
GraspNet + WB-IK22.5–50.045.0
+ Our Perc.40.0–90.044.4
Dex1B + WB-IK + AP52.555/5092.556.8
Ours (Full)87.590/8592.594.6
Abl.
–Perceive50.055/4557.587.0
–Whole-Body20.020/2087.522.9
–Align62.565/6085.073.5
–Dyn. Filter62.565/6090.069.4

Table 3a.

Figure 3b.

StageFilterOutputSurvival
1Grasp opt. + IK + coll.12.4 M2.4%
2Collision-free MP298 K2.4%
3Dynamics sim. (WBC)80.7 K27.0%
Final (+ view sel.)18 K—

Table 3c.

Ablations. –Perceive: fixed camera pose, bypassing active viewpoint search. –Whole-Body: active perception is enabled for object discovery, but the base is fixed during grasping, with no squatting or torso adaptation (arm-only manipulation). –Align: skip the whole-body alignment stage; instead of ranking KK candidates and repositioning the body, sample a single grasp candidate and execute directly from the current configuration. –Dyn. Filter: remove dynamics simulation from the data pipeline, retaining only kinematic and collision filtering. Compensator: disable the closed-loop pelvis compensator (§ 3.1), so task-space targets are not corrected for pelvis drift.

Main Results

Figure 3a summarizes Task 1. Our full system reaches 87.5% SR (90/85 ID/OOD) with 94.6% Gr|Det (once the object is found, grasping almost always succeeds), using zero real demonstrations, while the strong imitation baseline (ViA) needs teleoperated demos to reach 50%.

Quantitative results

Figure 3. Quantitative results. (a) Task 1 full-pipeline evaluation (N=40N=40, 10 objects, 5 ID/5 OOD); Det. = object localized, Gr|Det = grasp success given detection. (b) Task 2 spatial generalization across five heights (oracle perception, 10 objects; solid = ID, hatched = OOD, green = ACT/ViA demo heights): ours ≥73%\ge73\% everywhere, while ACT/ViA succeed only at their demonstrated heights (a demonstration-coverage effect; § 5.2) and Dex1B/–Dyn. Filter degrade at the extremes. (c) Kinodynamic data filtering: 73% of kinematically valid grasps fail under the deployment whole-body controller (508 M SDF candidates →\to 18 K training grasps).

Baselines fail along distinct axes. End-to-end imitation memorizes object-specific behavior: ViA holds 85% ID but collapses to 15% OOD, as cumulative errors over the long-horizon floating-base action space stall it on unseen objects. Gripper-based planners fail at execution, not discovery: adding our perception to GraspNet lifts Det. 50→90%50 \to90\%, yet Gr|Det is unchanged (45.0→44.4%45.0 \to44.4\%), because parallel-jaw pre-grasps give postures unnatural for the humanoid hand (Figure 7c) and hand–environment collisions (Figure 7d). Large-scale dexterous data alone is insufficient: Dex1B (adapted), which shares our CVAE + PointNet and hand type, matches our Det. (92.5%) yet reaches only 56.8% Gr|Det, below –Dyn. Filter (69.4%): built for tabletop settings, its data lacks both whole-body–hand co-optimization and dynamic feasibility (§ 5.2.1 dissects this by height).

Four failure-mode examples: unstable jittering, twisted infeasible poses, unnatural gripper-designed grasps, and hand–environment collisions

Figure 7. Failure modes eliminated by our pipeline. (a) No dynamics filter: unstable jittering. (b) No WB co-optimization/alignment: twisted infeasible poses. (c) GraspNet: unnatural gripper-designed grasps. (d) No WB–hand co-optimization: hand–environment collisions.

Each module addresses a distinct failure stage. Stage-wise metrics separate perception (Det.) from manipulation (Gr|Det): –Perceive cuts Det. by 35 pts (perception is the discovery bottleneck, § 5.2.2), while the three manipulation ablations each collapse Gr|Det along a different axis. –Whole-Body falls to 22.9% (cannot reach without squatting), –Align to 73.5% (sub-optimal approach from the current pose), and –Dyn. Filter to 69.4% (dynamically infeasible training data). The full 94.6% confirms all four are necessary; we dissect them next and resolve each ablation’s failure regime across the workspace in § 5.2.1.

Analysis

Kinodynamic filtering rejects 73% of valid grasps

Kinematic feasibility alone is insufficient for a floating base. Figure 3c traces the pipeline on an example profiling run: 508M candidates yield 18K training grasps, and 73% of kinematically valid trajectories are dynamically infeasible: they satisfy IK, collision, and planning constraints yet jitter (Figure 7a), twist (Figure 7b), or destabilize under the deployment controller. Removing the filter (–Dyn. Filter) drops Gr|Det 94.6→69.4%94.6 \to69.4\% at near-identical Det. (90.0 vs. 92.5%); Dex1B, which also lacks it, drops further (56.8%); controller-in-the-loop verification is essential for training-data quality. The consequence is sharpest at scale (Figure 4): trained on the same scenes without the filter, the policy is merely weaker inside the training range (64 vs. 87/80% on YCB/RoboCasa) but collapses to 3–6% below it—the filter is what turns in-distribution competence into extrapolation (§ 5.2).

Success rate versus table height for YCB and RoboCasa

Figure 4. Large-scale simulation evaluation (∼\sim18,800 paired trials; Task 3 protocol in § 4.1). (a) SR vs. table height on YCB (left) and RoboCasa (right). Training tables span 0.40–0.90 m (dashed line); the shaded band is extrapolation below the training range. (b) SR (%) aggregated over the extrapolation and in-range bands (95% Wilson CIs for the Full rows are within ±\pm4 pts, e.g. YCB extrapolation 81.1 [77.7, 84.1]). The full system loses only 6–8 pts below the training floor; removing any single component collapses extrapolation to 0–6% (–Align to ∼\sim40–46%).

Extrapolation beyond the training height range

At scale in simulation (Task 3, Figure 4, ∼18.8K paired trials), the full system holds 87.4/79.8% (YCB/RoboCasa) within the trained 0.40–0.90 m range and 81.1/71.4% at 0.10–0.35 m, below the training floor and reached from deep squats at shin height. It loses only 6–8 pts under extrapolation, while the other variants collapse there to 0–6% and –Align drops to 40–46%. Trained on large-scale, kinodynamically filtered data, the affordance policy learns rich local geometric features under a strong kinodynamic prior: its proposals remain object-relative and controller-executable even when the body configuration leaves the training distribution. Below the training floor (0.10–0.35 m), whole-body control and dynamics filtering become indispensable (removing either collapses success to near zero), and alignment’s contribution roughly doubles (13–15 → 25–41 pts), as maintaining stable, feasible whole-body execution becomes the dominant factor (§ 5.2.1). Object difficulty and the spatial envelope are analyzed in App. § 12.§ 6.

The full system’s profile carries over to hardware. Across five real heights (Task 2, Figure 3b, 150 trials), ours holds 73–97% with near-identical ID/OOD (85.3 vs. 84.0%), comparable to the simulated 80–93% (YCB) at the same heights; Dex1B and –Dyn. Filter degrade at the extremes, qualitatively consistent with the simulation results. ACT and ViA, given demonstrations at two heights, collapse at the unseen ones, as expected: teleoperated policies are bounded by their demonstration coverage, and collecting demonstrations across the full workspace is prohibitively expensive—precisely where sim-to-real training on large-scale synthetic data shines.

MethodYCB Extrap. 0.10–0.35 mYCB In-range 0.40–0.90 mRoboCasa Extrap. 0.10–0.35 mRoboCasa In-range 0.40–0.90 m
Ours (Full)81.187.471.479.8
–Align40.172.446.067.2
–Dyn. Filter2.864.45.763.6
Dex1B + WB-IK0.747.70.439.2
–Compensator2.143.61.944.0
–Whole-Body0.06.10.011.4

Table 4.

Each component has a distinct failure signature

The per-height curves (Figure 4a) resolve where each component matters, which a single ablation number cannot. –Compensator rises with height, from ∼\sim0% below 0.40 m to 59–81% at 0.70–0.90 m: pelvis drift grows with squat depth, so uncompensated targets miss most exactly where posture deviates most from nominal; the curve is a direct measurement of drift versus depth. –Dyn. Filter shows a cliff at the training floor (0% at ≤\le0.20 m, ≥\ge60% from 0.45 m): unfiltered data yields grasps executable only in nominal postures. –Whole-Body is near zero below 0.65 m and peaks at 21–27%: this is the arm-only reach envelope, and it quantifies how much of the workspace exists only through the torso and legs. –Align costs a uniform 13–15 pts inside the training range but 25–41 pts below it: candidate selection matters most where feasible proposals are scarce. Dex1B is 0% at ≤\le0.30 m and falls off again above 0.80 m (34–23% on YCB): fixed-upper-body generation contains neither low postures nor extended high reaches, a coverage limit of the generator. Coverage is not the whole gap: even in-range it stays ∼\sim40 pts below Full (47.7/39.2 vs. 87.4/79.8%), the compound cost of generating without whole-body–hand co-optimization or dynamic-feasibility filtering. Together these signatures explain the Task 1 stage-wise ablations (§ 4.2) mechanistically: each module removes a different failure regime, and only their union covers the workspace.

Whole-body spatial coverage

Figure 5. Whole-body spatial coverage. Adapting its entire body, our policy grasps in simulation across a continuous table-height range (0.10–0.90 m), from deep squats for low targets to full extension for high ones. Training tables spanned 0.40–0.90 m; the leftmost frames lie below the training range.

Active perception resolves the discovery bottleneck

Is the bottleneck finding the object or grasping it? –Perceive cuts Det. by 35 pts (92.5→57.5%) while Gr|Det stays high (87.0 vs. 94.6%): grasping is reliable given a good view; discovery is the bottleneck. Learned alternatives confirm the gap (ViA 70.0%, ACT+Active Head 57.5% Det. vs. our 92.5%). Our systematic viewpoint search takes 10–15 updates (∼8 s), a long search that end-to-end policies would have to learn jointly with manipulation; this motivates the modular separation of discovery from execution.

Synthetic data scaling drives generalization

In a dedicated real-world scaling study (App. § 12.§ 4), SR rises from 9.2% (5k trajectories) to ∼89% at 400k and plateaus thereafter (88.3% at 600k), with OOD benefiting most (5.0→90.0%) and the relative OOD degradation narrowing from 62% to under 2% above 300k; data diversity progressively closes the generalization gap to novel objects. Scale and filtering are complementary: scale closes the object gap, while kinodynamic filtering is what enables extrapolation beyond the training range (§ 5.2).

Conclusion and Limitations

We presented a hierarchical Perceive–Align–Grasp framework for humanoid whole-body affordance-aware grasping with active perception. By coordinating belief-driven viewpoint regulation, whole-body alignment, and a kinodynamically filtered affordance model, our system reaches 87.5% real-world success with zero real demonstrations, outperforming end-to-end imitation and modular grasp-planning baselines by large margins. Three findings anchor the design: 73% of kinematically valid grasps are dynamically infeasible under the deployment controller, so controller-in-the-loop data curation is essential for floating-base manipulation; perception is the dominant bottleneck under egocentric occlusion (without active perception, detection falls to 57.5% while Gr|Det remains 87%); and at scale in simulation the policy extrapolates to table heights well below its training range, evidence that kinodynamically filtered affordances are grounded in object geometry and whole-body feasibility rather than memorized configurations.

Limitations. Our system is predominantly open-loop at the high-level decision stage, prioritizing interpretability and stability over reactive replanning. The remaining real-world failures (∼12.5%) stem mainly from open-vocabulary detection, where DINO + SAM mis-identify targets under severe occlusion or on reflective surfaces, and from occasional collisions at workspace extremes near joint limits. Deployment currently uses double support without stepping, so targets beyond arm-plus-lean reach require repositioning; the controller interface already exposes velocity commands, and extending alignment to stepping is a natural next step. Synthetic-data generation is also costly (∼700 GPU-hours per 600k trajectories): fully parallelizable and amortized across deployments, but re-run when the controller or policy architecture changes. Future work may close the loop across perception and execution, co-train the stages end-to-end, and adapt data synthesis to the deployed controller.

Acknowledgments

This work was supported, in part, by NSF CAREER Award IIS-2240014 and NSF CCF-2112665 (TILOS).

Hierarchical Whole-Body Loco-Manipulation Controller

Frames and Notation

We use the standard rigid-body transform notation ATB∈SE(3){}^{A}T_{B} \in SE(3), which maps a 3D point expressed in frame BB into frame AA. The corresponding rotation and translation are ARB∈SO(3){}^{A}R_{B} \in SO(3) and ApB∈R3{}^{A}p_{B} \in\mathbb{R}^{3}.

Key frames..

  • W0(t)\mathcal{W}_{0}(t): the instantaneous support frame used by the upper-body controller. In double support, its origin is defined at the mid-point of the two contact feet and its yaw is aligned with the average heading of the feet. We set roll/pitch to zero to maintain a gravity-aligned frame and to avoid injecting foot tilt noise into the upper-body task frame.

  • O\mathcal{O}: the kinematic reference frame of the full-body URDF used for forward kinematics (FK) from joint encoders.

  • T\mathcal{T}: pelvis frame, treated as the floating base in the reduced upper-body IK model.

  • FL,FR\mathcal{F}_{L}, \mathcal{F}_{R}: left/right foot frames used as contacts.

  • EL,ER\mathcal{E}_{L}, \mathcal{E}_{R}: left/right end-effector frames used by the manipulation tasks.

  • EH\mathcal{E}_{H}: head camera frame used for active perception. We treat head gaze as an additional task-space target in the same support frame W0(t)\mathcal{W}_{0}(t), i.e., W0TEHdes(t){}^{\mathcal{W}_{0}}T_{\mathcal{E}_{H}}^{\mathrm{des}}(t) (or equivalently a desired head orientation / optical-axis direction), and include HH in the task index set K\mathcal{K} when head tracking is enabled.

Robot configuration.. We split the robot state into a lower-body part and an upper-body part:

q=[qlowerqupper],(2)\mathbf{q} = \begin{bmatrix} \mathbf{q}_{\mathrm{lower}} \\ \mathbf{q}_{\mathrm{upper}} \end{bmatrix}, \tag*{(2)}

where qlower\mathbf{q}_{\mathrm{lower}} contains leg joints (and any joints actuated by the low-level locomotion controller), and qupper\mathbf{q}_{\mathrm{upper}} contains waist/torso joints, arms, and optionally head joints (depending on the IK model).

Hierarchical Control Decomposition

Our control stack follows a standard outer/inner-loop organization: (i) an outer-loop whole-body IK module produces task-consistent upper-body commands together with a compact lower-body command interface, and (ii) an inner-loop RL controller (the decoupled whole-body controller of SONIC (GR00T) [38]) realizes the commanded pelvis motion while enforcing contact consistency and stability. A key design requirement is to maintain consistency between the pelvis pose assumed by the reduced IK model and the pelvis motion realized by the low-level controller.

High-level IK (upper-body whole-body IK).. At time tt, given desired task-space targets expressed in the support frame W0(t)\mathcal{W}_{0}(t),

{W0TEkdes(t)}k∈K,K⊆{L,R,H},(3)\left\{ {}^{\mathcal{W}_{0}}T_{\mathcal{E}_{k}}^{\mathrm{des}}(t) \right\}_{k \in\mathcal{K}}, \qquad\mathcal{K} \subseteq\{L,R,H\}, \tag*{(3)}

the IK module solves

(qupper⋆(t),ulower(t))=πIK({W0TEkdes(t)}k∈K,W0T^T(t)),(4)\left(\mathbf{q}_{\mathrm{upper}}^{\star}(t), \mathbf{u}_{\mathrm{lower}}(t)\right) = \pi_{\mathrm{IK}}\left(\left\{ {}^{\mathcal{W}_{0}}T_{\mathcal{E}_{k}}^{\mathrm{des}}(t) \right\}_{k \in\mathcal{K}}, {}^{\mathcal{W}_{0}}\widehat{T}_{\mathcal{T}}(t)\right), \tag*{(4)}

where W0T^T(t){}^{\mathcal{W}_{0}}\widehat{T}_{\mathcal{T}}(t) is a contact-consistent reconstruction of the pelvis pose in the same support frame (Sec. A.3). This term closes the inter-layer loop by aligning the IK solver’s floating-base state with the physically realized pelvis motion.

Low-level RL controller (lower-body tracking). The low-level controller executes the command ulower(t)\mathbf{u}_{\mathrm{lower}}(t) and outputs lower-body joint targets:

qlower⋆(t)=πlower(slower(t),ulower(t)),(5)\mathbf{q}^{\star}_{\mathrm{lower}}(t)=\pi_{\mathrm{lower}}\left(\mathbf{s}_{\mathrm{lower}}(t),\mathbf{u}_{\mathrm{lower}}(t)\right), \tag*{(5)}

where slower(t)\mathbf{s}_{\mathrm{lower}}(t) is the proprioceptive observation (joint encoders, IMU, contact flags, etc.).

Lower-body command interface. In our standing manipulation experiments (double support, no stepping), we instantiate the general interface ulower=(vx,vy,vyaw,h)\mathbf{u}_{\mathrm{lower}}=(v_x,v_y,v_{\mathrm{yaw}},h) of § 3.1 as a low-dimensional pelvis command, with planar velocities set to zero and yaw commanded as a heading ψ\psi rather than a rate, to reduce inter-layer coupling and improve robustness under rate mismatch and model error:

ulower(t)=[h(t), ψ(t)].(6)\mathbf{u}_{\mathrm{lower}}(t)=\left[h(t),\ \psi(t)\right]. \tag*{(6)}

The same hierarchical formulation directly supports loco-manipulation by extending ulower\mathbf{u}_{\mathrm{lower}} to a more general locomotion interface (e.g., planar velocity and yaw-rate commands) while keeping Eqs. (4)–(5) unchanged.

Commanded output. The final motor command is the concatenation of the lower- and upper-body references:

q⋆(t)=[qlower⋆(t)qupper⋆(t)].(7)\mathbf{q}^{\star}(t)= \begin{bmatrix} \mathbf{q}^{\star}_{\mathrm{lower}}(t)\\ \mathbf{q}^{\star}_{\mathrm{upper}}(t) \end{bmatrix}. \tag*{(7)}

In our deployed system, the low-level controller and the upper-body IK run synchronously at the same control rate (50 Hz), and the robot manager dispatches the fused command q⋆(t)\mathbf{q}^{\star}(t) at each cycle.

Contact-Consistent Kinematic State Reconstruction

A central challenge in hierarchical control is that the pelvis pose assumed by the reduced IK model can deviate from the pelvis pose realized by the low-level controller due to tracking error, compliance, and modeling mismatch. To close this gap, we compute a contact-consistent reconstruction of the pelvis pose in the support frame W0(t)\mathcal{W}_0(t) using joint encoders under double-foot contact. Importantly, we use an instantaneous support frame definition (no integration): all quantities are expressed relative to the current mid-feet frame computed from FK.

Foot/pelvis FK in the URDF reference frame. Given measured joint positions q(t)\mathbf{q}(t), we compute the forward kinematics in the full-body URDF:

OTFL(t),OTFR(t),OTT(t)=FK⁡(q(t)).(8){}^{\mathcal{O}}\mathbf{T}_{\mathcal{F}_L}(t),\quad{}^{\mathcal{O}}\mathbf{T}_{\mathcal{F}_R}(t),\quad{}^{\mathcal{O}}\mathbf{T}_{\mathcal{T}}(t)=\operatorname{FK}(\mathbf{q}(t)). \tag*{(8)}

Instantaneous mid-feet support frame. We define W0(t)\mathcal{W}_0(t) from the two contact feet:

OpW0(t)=12(OpFL(t)+OpFR(t)),(9){}^{\mathcal{O}}\mathbf{p}_{\mathcal{W}_0}(t)=\frac{1}{2}\left({}^{\mathcal{O}}\mathbf{p}_{\mathcal{F}_L}(t)+{}^{\mathcal{O}}\mathbf{p}_{\mathcal{F}_R}(t)\right), \tag*{(9)}
ψˉ(t)=avgYaw⁡(ORFL(t),ORFR(t)),(10)\bar{\psi}(t)=\operatorname{avgYaw}\left({}^{\mathcal{O}}\mathbf{R}_{\mathcal{F}_L}(t),{}^{\mathcal{O}}\mathbf{R}_{\mathcal{F}_R}(t)\right), \tag*{(10)}
ORW0(t)=Rz(ψˉ(t)),(11){}^{\mathcal{O}}\mathbf{R}_{\mathcal{W}_0}(t)=\mathbf{R}_z\left(\bar{\psi}(t)\right), \tag*{(11)}
OTW0(t)=[ORW0(t)OpW0(t)0T1].(12){}^{\mathcal{O}}\mathbf{T}_{\mathcal{W}_0}(t)= \begin{bmatrix} {}^{\mathcal{O}}\mathbf{R}_{\mathcal{W}_0}(t) & {}^{\mathcal{O}}\mathbf{p}_{\mathcal{W}_0}(t)\\ \mathbf{0}^{\mathsf{T}} & 1 \end{bmatrix}. \tag*{(12)}

This matches the controller’s task frame: centered between the feet and aligned with the support yaw, while remaining gravity-aligned.

Pelvis pose in the support frame.. The reconstructed pelvis pose used by IK is

W^0TT(t)=(TTW0(t))−1TTT(t).(13)\widehat{\mathcal{W}}_0\mathbf{T}_{\mathcal{T}}(t)=\left({}^{\mathcal{T}}\mathbf{T}_{\mathcal{W}_0}(t)\right)^{-1}{}^{\mathcal{T}}\mathbf{T}_{\mathcal{T}}(t). \tag*{(13)}

Remark (estimator modularity).. While we use a purely kinematic, contact-consistent reconstruction in double support for simplicity and low latency, the proposed interface only requires an estimate of the pelvis pose in the task frame, W^0TT(t)\widehat{\mathcal{W}}_0\mathbf{T}_{\mathcal{T}}(t). In general, W^0TT(t)\widehat{\mathcal{W}}_0\mathbf{T}_{\mathcal{T}}(t) can be provided by any state estimator that is consistent with contact constraints, e.g., IMU–encoder fusion, visual–inertial odometry with contact updates, or factor-graph filtering, without changing the downstream compensation in Sec. A.4.

Closed-Loop Pelvis Consistency for Upper-Body IK

Let W0TTIK(t)\mathcal{W}_0\mathbf{T}_{\mathcal{T}}^{\mathrm{IK}}(t) denote the pelvis pose assumed by the IK model (i.e., the floating-base state inside the reduced IK solver), and let W^0TT(t)\widehat{\mathcal{W}}_0\mathbf{T}_{\mathcal{T}}(t) be the contact-consistent reconstruction from Eq. (13). We compute the mismatch transform

ΔTpelvis(t)=W^0TT(t)(W0TTIK(t))−1.(14)\Delta\mathbf{T}_{\mathrm{pelvis}}(t)=\widehat{\mathcal{W}}_0\mathbf{T}_{\mathcal{T}}(t)\left(\mathcal{W}_0\mathbf{T}_{\mathcal{T}}^{\mathrm{IK}}(t)\right)^{-1}. \tag*{(14)}

Compensating task targets (equivalently, updating the tracking frame).. For any desired end-effector target expressed in W0(t)\mathcal{W}_0(t), we compensate by the inverse mismatch:

W0TEkcorr(t)=(ΔTpelvis(t))−1W0TEkdes(t),(15)\mathcal{W}_0\mathbf{T}_{\mathcal{E}_k}^{\mathrm{corr}}(t)=\left(\Delta\mathbf{T}_{\mathrm{pelvis}}(t)\right)^{-1}\mathcal{W}_0\mathbf{T}_{\mathcal{E}_k}^{\mathrm{des}}(t), \tag*{(15)}

and we pass W0TEkcorr(t)\mathcal{W}_0\mathbf{T}_{\mathcal{E}_k}^{\mathrm{corr}}(t) to the IK solver. This can be interpreted either as correcting task targets or as updating an internal IK tracking frame; both are equivalent up to where the correction is applied.

Filtering on $SE(3).. To reduce jitter from encoder noise and small contact deformation, we apply a low-pass filter on SE(3)SE(3) (in the Lie algebra) before compensation:

ΔT‾pelvis(t)=LPF⁡(ΔTpelvis(t)),(16)\overline{\Delta\mathbf{T}}_{\mathrm{pelvis}}(t)=\operatorname{LPF}\left(\Delta\mathbf{T}_{\mathrm{pelvis}}(t)\right), \tag*{(16)}
W0TEkcorr(t)=(ΔT‾pelvis(t))−1W0TEkdes(t).(17)\mathcal{W}_0\mathbf{T}_{\mathcal{E}_k}^{\mathrm{corr}}(t)=\left(\overline{\Delta\mathbf{T}}_{\mathrm{pelvis}}(t)\right)^{-1}\mathcal{W}_0\mathbf{T}_{\mathcal{E}_k}^{\mathrm{des}}(t). \tag*{(17)}

Synthetic Data Generation: Kinodynamically Validated Trajectories

Our goal is to generate large-scale grasp demonstrations that are not only kinematically valid (reachable and collision-free), but also dynamically executable under the same hierarchical whole-body control stack used at deployment. This controller-in-the-loop filtering is key to learning kinodynamic-aware affordances for floating-base humanoids, where many geometrically plausible grasps fail due to stability and tracking limitations.

Domain Randomization

Object assets.. We draw object meshes from Objaverse [76] and YCB [89]; OOD evaluation objects are held out from the training set. To promote shape- and scale-level generalization, each object is isotropically re-scaled by a random factor within a bounded interval.

Scene parameterization.. Each scene contains a single axis-aligned box table and one target object:

S={(Ttable,stable),(Tobj,mobj)}.(18)\mathcal{S} = \{(\mathbf{T}_{\mathrm{table}}, \mathbf{s}_{\mathrm{table}}),(\mathbf{T}_{\mathrm{obj}},m_{\mathrm{obj}})\}. \tag*{(18)}

Table dimensions are sampled uniformly within predefined bounds. The table-top height is randomized relative to the workspace origin to induce diverse reach regimes:

ztable∼U(zmin⁡,zmax⁡)+zws.(19)z_{\mathrm{table}} \sim\mathcal{U}(z_{\min},z_{\max}) + z_{\mathrm{ws}}. \tag*{(19)}

Tabletop heights span 0.40–0.90 m, the training range referred to in § 4. The object is placed uniformly over the table surface with an edge margin. Its vertical position is set such that the lowest point of the scaled mesh rests on the tabletop, its mass is sampled as mobj∼U(0.1,0.5) kgm_{\mathrm{obj}} \sim\mathcal{U}(0.1,0.5)\,\mathrm{kg}, and its yaw is randomized:

ψ∼U(ψmin⁡,ψmax⁡).(20)\psi\sim\mathcal{U}(\psi_{\min},\psi_{\max}). \tag*{(20)}

World-frame perturbations.. To avoid a single canonical base-to-object transform, we additionally perturb the table center in the world frame using a run-specific random seed, and apply the corresponding rigid transform to the object pose. This produces distinct scene files across parallel collection runs while preserving valid on-table placement.

Grasp Optimization and Motion Planning

Given a randomized scene S\mathcal{S}, we generate grasp demonstrations in batches. For each scene, we sample up to NcandN_{\mathrm{cand}} grasp candidates and retain only those that are (i) kinematically feasible (reachable and collision-free) and (ii) dynamically executable under simulation evaluation (Sec. B.3).

Energy-Based Grasp Optimization

We optimize grasp configurations in joint space. Let q∈RD\mathbf{q} \in\mathbb{R}^{D} denote the robot configuration, comprising the pelvis pose, waist, arm, and hand joints, so that the hand and the posture carrying it are optimized jointly, and let {pic(q)}i=1Nc\{\mathbf{p}^{c}_{i}(\mathbf{q})\}_{i=1}^{N_c} be the fingertip contact points computed by forward kinematics. We minimize an energy function

E(q)=wdisEdis(q)+wpenEpen(q)+wspenEspen(q)+wjntEjnt(q).(21)E(\mathbf{q}) = w_{\mathrm{dis}}E_{\mathrm{dis}}(\mathbf{q}) + w_{\mathrm{pen}}E_{\mathrm{pen}}(\mathbf{q}) + w_{\mathrm{spen}}E_{\mathrm{spen}}(\mathbf{q}) + w_{\mathrm{jnt}}E_{\mathrm{jnt}}(\mathbf{q}). \tag*{(21)}

Contact distance.. We encourage designated fingertip points to lie near the object surface via the object signed distance field (SDF):

Edis(q)=∑i=1Nc∣SDF⁡obj(pic(q))∣.(22)E_{\mathrm{dis}}(\mathbf{q}) = \sum_{i=1}^{N_c} \left|\operatorname{SDF}_{\mathrm{obj}}(\mathbf{p}^{c}_{i}(\mathbf{q}))\right|. \tag*{(22)}

Scene collision.. We penalize penetration of robot collision spheres into the scene SDF:

Epen(q)=∑j=1Nsmax⁡(0,dpad−SDF⁡scene(sj(q))).(23)E_{\mathrm{pen}}(\mathbf{q}) = \sum_{j=1}^{N_s} \max(0,d_{\mathrm{pad}}-\operatorname{SDF}_{\mathrm{scene}}(\mathbf{s}_{j}(\mathbf{q}))). \tag*{(23)}

where sj(q)\mathbf{s}_{j}(\mathbf{q}) is the center of collision sphere jj and dpadd_{\mathrm{pad}} is a safety padding.

Self-collision.. To avoid inter-finger collisions, we penalize close pairs:

Espen(q)=∑(j,k)∈Pmax⁡(0,dself−∥sj(q)−sk(q)∥)2.(24)E_{\mathrm{spen}}(\mathbf{q}) = \sum_{(j,k)\in\mathcal{P}} \max(0,d_{\mathrm{self}}-\lVert\mathbf{s}_{j}(\mathbf{q})-\mathbf{s}_{k}(\mathbf{q})\rVert)^{2}. \tag*{(24)}

Joint limits. We apply a soft penalty for joint limit violation:

Ejnt(q)=∑i=1D[max⁡(0,qimin⁡−qi)2+max⁡(0,qi−qimax⁡)2].(25)E_{\mathrm{jnt}}(\mathbf{q}) = \sum_{i=1}^{D} \left[\max(0,q_i^{\min}-q_i)^2+\max(0,q_i-q_i^{\max})^2\right]. \tag*{(25)}

Reachability and balance. The joint limits include the pelvis range the locomotion controller can realize, so every optimized posture can be commanded. Balance is not an energy term: a static criterion does not predict the closed-loop RL policy under a reaching load. Dynamic feasibility is verified downstream (Sec. B.3).

Optimization. We solve Eq. (21) with a simulated-annealing-style optimizer that combines autodiff gradients with stochastic acceptance [85]. At iteration tt, a candidate q′\mathbf{q}' is proposed by an adaptive step,

q′=q−η(t)⋅∇qE(q)vt+ϵ,vt=μ(∇qE)2+(1−μ)vt−1,(26)\mathbf{q}' = \mathbf{q} - \eta(t) \cdot\frac{\nabla_{\mathbf{q}}E(\mathbf{q})}{\sqrt{\mathbf{v}_t}+\epsilon}, \qquad\mathbf{v}_t = \mu(\nabla_{\mathbf{q}}E)^2 + (1-\mu)\mathbf{v}_{t-1}, \tag*{(26)}

and accepted with probability

P(accept)=min⁡(1,exp⁡(E(q)−E(q′)T(t))).(27)P(\mathrm{accept}) = \min\left(1,\exp\left(\frac{E(\mathbf{q})-E(\mathbf{q}')}{T(t)}\right)\right). \tag*{(27)}

The step size η(t)\eta(t) and temperature T(t)T(t) follow exponential decay schedules (Table 1).

StageParameterSymbolValue
Grasp Optimization (Simulated Annealing)
IterationsNumber of iterationsNiterN_{\mathrm{iter}}5,000
BatchingGrasp samples per sceneNcandN_{\mathrm{cand}}8,192
ContactsFingertip contact pointsNcN_{\mathrm{c}}4
EnergyContact-distance weightwdisw_{\mathrm{dis}}100
EnergyScene-penetration weightwpenw_{\mathrm{pen}}100
EnergySelf-penetration weightwspenw_{\mathrm{spen}}50
EnergyJoint-limit weightwjntw_{\mathrm{jnt}}1
StepsInitial step sizeη0\eta_{0}0.005
StepsStep decay periodτη\tau_{\eta}50
Temp.Initial temperatureT0T_{0}18
Temp.Temperature decay rateγT\gamma_{T}0.95
Temp.Annealing periodτT\tau_{T}30
Init.Front approach angle rangeθfront\theta_{\mathrm{front}}[−π/2,−π/10][-\pi/2,-\pi/10]
Init.Approach distance rangedd[0.02, 0.15] m
Init.Hand template (6-DoF)μhand\boldsymbol{\mu}_{\mathrm{hand}}(0.8, 0.05, 0.6, 0.6, 0.6, 0.6)
Motion Planning (Pre/Grasp/Post + Smoothing)
PhasesInit→Pre-grasp stepsTinitT_{\mathrm{init}}150
PhasesPre-grasp→Grasp stepsTpgT_{\mathrm{pg}}80
PhasesHolding stepsTholdT_{\mathrm{hold}}15
PhasesGrasp→Post-grasp stepsTgpT_{\mathrm{gp}}50
PhasesLift heightδz\delta_{z}0.05 m
CollisionSafety paddingdpadd_{\mathrm{pad}}0.025 m
HeuristicRetreat step factor (arm)warmw_{\mathrm{arm}}0.001
HeuristicClosure factor (hand)whandw_{\mathrm{hand}}0.8
SmoothingArm smoothness weightwsmarmw_{\mathrm{sm}}^{\mathrm{arm}}100
SmoothingHand smoothness weightwsmhandw_{\mathrm{sm}}^{\mathrm{hand}}10
BatchingTrajectories per batchBtrajB_{\mathrm{traj}}1,024

Table 1. Key hyperparameters for synthetic grasp generation (optimization and motion planning).

Motion Planning and Kinematic Filtering

For each optimized grasp configuration qgrasp\mathbf{q}_{\mathrm{grasp}}, we construct a multi-phase joint-space trajectory {q(t)}t=0Ttot\{\mathbf{q}(t)\}_{t=0}^{T_{\mathrm{tot}}} comprising: (i) retreat to a pre-grasp, (ii) approach to grasp, (iii) hold, (iv) post-grasp closure, and (v) lift. The pre-grasp and post-grasp configurations are generated by SDF-gradient updates that move the arm (retreat) or hand joints (closure) along ∇SDFscene\nabla\mathrm{SDF}_{\mathrm{scene}}. The lift keyframe is obtained by raising the end-effector by δz\delta_z and solving IK; a keyframe is reachable if this IK converges within joint limits, so kinematic validity is a property of the whole approach, grasp, and lift motion rather than of a static grasp pose. Intermediate segments are initialized by interpolation and refined by trajectory smoothing.

Trajectory smoothing. We optimize the full trajectory by minimizing a collision penalty and a smoothness term:

Ltraj=Lpen+λsmoothLsmooth,(28)\mathcal{L}_{\mathrm{traj}} = \mathcal{L}_{\mathrm{pen}} + \lambda_{\mathrm{smooth}}\mathcal{L}_{\mathrm{smooth}}, \tag*{(28)}

where

Lpen=1TtotNs∑t,j[max⁡(0,dpad−SDFscene(sj(t)))]2,(29)\mathcal{L}_{\mathrm{pen}} = \frac{1}{T_{\mathrm{tot}}N_s}\sum_{t,j}\left[\max\left(0,d_{\mathrm{pad}}-\mathrm{SDF}_{\mathrm{scene}}(\mathbf{s}_j(t))\right)\right]^2, \tag*{(29)}

and

Lsmooth=∑i=1Dwi(Std⁡(Δqi)+αaccStd⁡(Δ2qi)).(30)\mathcal{L}_{\mathrm{smooth}} = \sum_{i=1}^{D} w_i\left(\operatorname{Std}(\Delta q_i)+\alpha_{\mathrm{acc}}\operatorname{Std}(\Delta^2 q_i)\right). \tag*{(30)}

Kinematic feasibility. We treat a sample as kinematically feasible only if it passes both the grasp optimization stage and the motion planning stage. For a configuration q\mathbf{q}, define the scene penetration of collision sphere jj as

pen⁡j(q)≜max⁡(0,dpad−SDFscene(sj(q))),(31)\operatorname{pen}_j(\mathbf{q}) \triangleq\max\left(0,d_{\mathrm{pad}}-\mathrm{SDF}_{\mathrm{scene}}(\mathbf{s}_j(\mathbf{q}))\right), \tag*{(31)}

and the fingertip-to-object surface distance for contact point ii as

dis⁡i(q)≔∣SDF⁡obj(pic(q))∣.(32)\operatorname{dis}_{i}(\mathbf{q}) \coloneqq\left|\operatorname{SDF}_{\mathrm{obj}}\left(\mathbf{p}^{c}_{i}(\mathbf{q})\right)\right|. \tag*{(32)}

Grasp optimization succeeds if every keyframe is reachable and the optimized grasp configuration qgrasp\mathbf{q}_{\mathrm{grasp}} satisfies

opt_success=[max⁡jpen⁡j(qgrasp)≤ϵcol]∧[max⁡idis⁡i(qgrasp)≤ϵdis].(33)\mathtt{opt\_success} = \left[ \max_{j}\operatorname{pen}_{j}(\mathbf{q}_{\mathrm{grasp}}) \le\epsilon_{\mathrm{col}} \right] \land \left[ \max_{i}\operatorname{dis}_{i}(\mathbf{q}_{\mathrm{grasp}}) \le\epsilon_{\mathrm{dis}} \right]. \tag*{(33)}

Motion planning succeeds if the planned trajectory {q(t)}t=0Ttot\{\mathbf{q}(t)\}_{t=0}^{T_{\mathrm{tot}}} remains collision-free (with padding) along the entire execution:

mp_success=[max⁡t,jpen⁡j(q(t))≤ϵtraj].(34)\mathtt{mp\_success} = \left[ \max_{t,j}\operatorname{pen}_{j}(\mathbf{q}(t)) \le\epsilon_{\mathrm{traj}} \right]. \tag*{(34)}

A sample is kinematically valid iff

kin_success=opt_success∧mp_success.(35)\mathtt{kin\_success}=\mathtt{opt\_success}\land\mathtt{mp\_success}. \tag*{(35)}

Only trajectories with kin_success=1\mathtt{kin\_success}=1 are passed to the dynamics-aware feasibility filtering stage. Optimization and planning schedules are listed in Table 1.

Dynamics-Aware Feasibility Filtering

Why dynamics filtering? For floating-base humanoids, kinematic feasibility does not guarantee executability: trajectories can violate stability constraints, exceed controller tracking capability, or induce unstable whole-body transients. We therefore apply a controller-in-the-loop dynamics check to retain only trajectories that are feasible under the deployment control stack.

Massively parallel controller replay. We replay each kinematically feasible trajectory in ManiSkill3 [75], a GPU-parallel physics simulator built on SAPIEN [91], at the deployment control rate (50 Hz). Crucially, the same hierarchical whole-body control structure as real deployment is used during replay (whole-body IK producing upper-body targets and low-level locomotion control executing the commanded motion).

Dynamic feasibility criteria. A trajectory is marked sim_success\mathtt{sim\_success} if it satisfies:

  1. Task success: the object is lifted by at least δlift=0.05\delta_{\mathrm{lift}}=0.05 m and remains in hand.

  2. Stable execution: no fall/termination, and base attitude remains bounded, ∣roll∣,∣pitch∣≤θmax⁡\lvert\mathrm{roll}\rvert,\lvert\mathrm{pitch}\rvert\le\theta_{\max}.

  3. Trackability: tracking error of commanded end-effector keyframes stays below a threshold, max⁡t∥pee(t)−pee∗(t)∥≤emax⁡\max_{t}\left\|\mathbf{p}_{\mathrm{ee}}(t)-\mathbf{p}^{*}_{\mathrm{ee}}(t)\right\|\le e_{\max}, with no abrupt spikes in commanded motion.

We treat (θmax⁡,emax⁡)(\theta_{\max},e_{\max}) as fixed thresholds chosen to match the tolerances of the deployment controller.

Final acceptance mask. We retain a sample only if it passes both kinematic and dynamics feasibility:

success[i]=kin_success[i]∧sim_success[i].(36)\mathtt{success}[i]=\mathtt{kin\_success}[i]\land\mathtt{sim\_success}[i]. \tag*{(36)}

By construction, the resulting dataset is supported on actions that are executable under whole-body kinodynamic constraints, enabling the learned policy to internalize kinodynamic-aware affordances rather than purely geometric ones. Figure 3c profiles an example collection run (508M candidates to 18K accepted grasps); its survival rates are representative of every run, and the full training set aggregates all parallel runs with distinct seeds into 600k trajectories.

Rendering and View Selection

We render object-centric point cloud observations for each accepted sample using a GPU-parallel renderer (ManiSkill3 [75]). For each scene, we render V=16V = 16 camera views with small RPY perturbations around a calibrated nominal extrinsic, and extract an object-only segmented point cloud of Npts=4096N_{\mathrm{pts}} = 4096 points, randomly downsampled to N=2048N = 2048 for the policy.

View selection.. Among candidate views whose frustums contain the object 3D bounding box, we select the view maximizing the number of valid (nonzero) 3D points:

v∗=arg⁡max⁡v∈Vvisible∑i=1Npts[∥pi(v)∥>ϵ].(37)v^* = \arg\max_{v \in\mathcal{V}_{\mathrm{visible}}} \sum_{i=1}^{N_{\mathrm{pts}}} \left[\left\lVert\mathbf{p}_i^{(v)} \right\rVert> \epsilon\right]. \tag*{(37)}

If no view passes the visibility check, the scene is discarded to avoid degenerate observations. At deployment, this offline oracle view selection is replaced by the online active perception module in the main paper.

Policy Architecture

Overview.. We learn a conditional generative grasp policy that maps a segmented object point cloud P∈RN×3\mathcal{P} \in\mathbb{R}^{N \times3} to a distribution over multi-stage end-effector (EE) actions. Let f=PointNet⁡(P)∈Rdf\mathbf{f} = \operatorname{PointNet}(\mathcal{P}) \in\mathbb{R}^{d_f} be a global shape feature. Our policy is a conditional VAE (CVAE) with latent z∈Rdz\mathbf{z} \in\mathbb{R}^{d_z}, defining

pθ(a∣P)=∫pθ(a∣z,f) p(z) dz,p(z)=N(0,I),(38)p_\theta(\mathbf{a} \mid\mathcal{P}) = \int p_\theta(\mathbf{a} \mid\mathbf{z}, \mathbf{f})\,p(\mathbf{z})\,d\mathbf{z}, \qquad p(\mathbf{z}) = \mathcal{N}(\mathbf{0}, \mathbf{I}), \tag*{(38)}

where a∈RT×Da\mathbf{a} \in\mathbb{R}^{T \times D_a} is a TT-stage grasp action (pre-grasp, grasp, post-grasp). All architectural and training hyperparameters are summarized in Sec. D.

Action parameterization.. Each stage action at∈RDa\mathbf{a}_t \in\mathbb{R}^{D_a} consists of EE translation, EE orientation as a unit quaternion, and hand joint targets:

at≜[pt;rt;ht],a≜[a1;…;aT]∈RT×Da.(39)\mathbf{a}_t \triangleq[\mathbf{p}_t; \mathbf{r}_t; \mathbf{h}_t], \qquad\mathbf{a} \triangleq[\mathbf{a}_1; \ldots; \mathbf{a}_T] \in\mathbb{R}^{T \times D_a}. \tag*{(39)}

Vision Encoder: PointNet

We encode P\mathcal{P} using a PointNet-style [87] encoder:

f=ϕ(max⁡nh(P)),(40)\mathbf{f} = \phi\left(\max_n h(\mathcal{P})\right), \tag*{(40)}

where h(⋅)h(\cdot) is a shared-point MLP (implemented by stacked Conv1D+ReLU blocks), max⁡n\max_n is channel-wise max-pooling over points, and ϕ(⋅)\phi(\cdot) projects to dimension dfd_f. We omit BatchNorm in the encoder to avoid train/test mismatch under small-batch deployment.

Conditional VAE (CVAE)

Let aflat∈RT⋅Da\mathbf{a}^{\mathrm{flat}} \in\mathbb{R}^{T \cdot D_a} be the flattened action. During training, the encoder models an approximate posterior

qφ(z∣aflat,f)=N(μ,diag⁡(σ2)).(41)q_\varphi(\mathbf{z} \mid\mathbf{a}^{\mathrm{flat}}, \mathbf{f}) = \mathcal{N}\left(\boldsymbol{\mu}, \operatorname{diag}(\boldsymbol{\sigma}^2)\right). \tag*{(41)}
[μ,log⁡σ2]=MLP⁡enc([aflat;f]),z=μ+σ⊙ϵ,ϵ∼N(0,I),[\boldsymbol{\mu}, \log\boldsymbol{\sigma}^2] = \operatorname{MLP}_{\mathrm{enc}}([\mathbf{a}^{\mathrm{flat}}; \mathbf{f}]), \qquad\mathbf{z} = \boldsymbol{\mu} + \boldsymbol{\sigma} \odot\boldsymbol{\epsilon}, \quad\boldsymbol{\epsilon} \sim\mathcal{N}(\mathbf{0}, \mathbf{I}),

where log⁡σ2\log\boldsymbol{\sigma}^2 is clamped to a fixed range (Sec. D) for numerical stability. The decoder predicts the action:

a^flat=MLP⁡dec([z;f]),a^=reshape⁡(a^flat)∈RT×Da.(42)\hat{\mathbf{a}}^{\mathrm{flat}} = \operatorname{MLP}_{\mathrm{dec}}([\mathbf{z}; \mathbf{f}]), \qquad\hat{\mathbf{a}} = \operatorname{reshape}(\hat{\mathbf{a}}^{\mathrm{flat}}) \in\mathbb{R}^{T \times D_a}. \tag*{(42)}

Training Objective and Inference

Training Objective and Inference

We minimize the negative evidence lower bound (ELBO):

L=Eqφ(z∣aflat,f)[Lrec(a^,a)]+wKLDKL(qφ(z∣aflat,f) ∥ N(0,I)).(43)\mathcal{L} = \mathbb{E}_{q_{\varphi}(\mathbf{z}\mid\mathbf{a}^{\mathrm{flat}},\mathbf{f})}\left[\mathcal{L}_{\mathrm{rec}}(\hat{\mathbf{a}},\mathbf{a})\right] + w_{\mathrm{KL}}D_{\mathrm{KL}}\left(q_{\varphi}(\mathbf{z}\mid\mathbf{a}^{\mathrm{flat}},\mathbf{f}) \,\middle\|\, \mathcal{N}(\mathbf{0},\mathbf{I})\right). \tag*{(43)}

We use an ℓ1\ell_{1} reconstruction loss over translation, quaternion, and hand joints across stages. Quaternions are normalized and sign-canonicalized to resolve r≡−r\mathbf{r}\equiv-\mathbf{r}:

rˉ←r/∥r∥,rˉ^←r^/∥r^∥,rˉ^←sign⁡(⟨rˉ^,rˉ⟩)⋅rˉ^.(44)\bar{\mathbf{r}} \leftarrow\mathbf{r}/\lVert\mathbf{r}\rVert,\qquad\hat{\bar{\mathbf{r}}} \leftarrow\hat{\mathbf{r}}/\lVert\hat{\mathbf{r}}\rVert,\qquad\hat{\bar{\mathbf{r}}} \leftarrow\operatorname{sign}\left(\left\langle\hat{\bar{\mathbf{r}}},\bar{\mathbf{r}}\right\rangle\right)\cdot\hat{\bar{\mathbf{r}}}. \tag*{(44)}

Inference. At test time, we discard the encoder and draw KK latents z∼N(0,I)\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) for the alignment stage (§ 3.4), decoding each as a^=MLP⁡dec([z;PointNet⁡(P)])\hat{\mathbf{a}}=\operatorname{MLP}_{\mathrm{dec}}([\mathbf{z};\operatorname{PointNet}(\mathcal{P})]). The predicted EE targets are executed by the downstream whole-body stack (IK, collision-aware planning, and controller) as in the main paper.

Hyperparameter Summary

ModuleParameterSymbolValue
Policy (PointNet + CVAE)
InputPolicy input point cloud sizeNN2,048
ActionStages (pre/grasp/post)TT3
ActionAction dimensionDaD_a13
ActionHand DoF in actionDhD_h6
EncoderPointNet feature dimensiondfd_f512
LatentCVAE latent dimensionDzD_z256
MLPEncoder/decoder hidden dims—[256, 256, 256]
MLPActivation / normalization—SiLU / LayerNorm
Stabilitylog σ2\sigma^2 clamp range[ℓmin⁡,ℓmax⁡][\ell_{\min}, \ell_{\max}][−10,10][-10, 10]
LossKL weightwKLw_{\mathrm{KL}}0.1
Optim.Optimizer—AdamW
Optim.Learning rateηlr\eta_{\mathrm{lr}}10−410^{-4}
Optim.Weight decayλwd\lambda_{\mathrm{wd}}10−410^{-4}
BatchBatch size (scenes)BB128
SamplingActions per sceneNaN_a12
ScheduleLR decay step (iter)tdecayt_{\mathrm{decay}}66,667
ScheduleLR decay factorγlr\gamma_{\mathrm{lr}}0.1
TrainingTotal iterationsTiterT_{\mathrm{iter}}100,000
SplitTrain/test split (scenes)—97% / 3%
Eval.Evaluation frequency—every 10,000 iters
Simulation Evaluation
EnginePhysics simulator—ManiSkill3 (SAPIEN)
ParallelismParallel environmentsNenvN_{\mathrm{env}}512
ControlControl frequencyfctrlf_{\mathrm{ctrl}}50 Hz
ObjectObject massmobjm_{\mathrm{obj}}U(0.1,0.5)\mathcal{U}(0.1, 0.5) kg
MetricLift thresholdδlift\delta_{\mathrm{lift}}0.05 m
ContactFriction (dyn/static)—2.0 / 2.0
ContactRestitution—0.0
Rendering / View Selection
ViewsViews per sceneVV16
PointsRendered point cloud size (before downsample)NrawN_{\mathrm{raw}}4,096
PointsPolicy input points (after downsample)NN2,048
JitterRoll jitterΔϕ\Delta\phi±5∘\pm5^\circ
JitterPitch jitterΔθ\Delta\theta±8∘\pm8^\circ
JitterYaw jitterΔψ\Delta\psi±12∘\pm12^\circ
Seg.Segment extraction—object-only

Table 2. Hyperparameters for policy learning, evaluation, and rendering.

Extended Experimental Analysis

This section collects supplementary figures and analyses referenced from § 4 of the main paper: real-world qualitative rollouts, visual failure modes, scene generalization, the training-data scaling curve, and the full results of the large-scale simulation evaluation (spatial success-rate fields and per-object success rates).

Real-World Qualitative Rollouts

Figure 6 shows representative successes across diverse objects, reaches, and scenes: tabletop manipulation (a), long-reach grasping (b), and zero-shot deployment in unseen environments at varied heights (c). Across all settings the policy coordinates active perception, whole-body motion, and dexterous manipulation with no scene-specific tuning.

Real-world rollouts across tabletop, long-reach, and varied environments

Figure 6. Real-world rollouts, zero-shot from simulation. (a) Tabletop grasping of diverse objects; (b) long-reach grasping across a table; (c) deployment in varied unseen environments and heights (shelves, benches, tables).

Visual Failure Modes

Figure 7 visualizes the four representative failure modes that the Full pipeline eliminates relative to its ablations and to the gripper-based baseline. (a) and (b) are the kinematically valid but dynamically infeasible configurations that the dynamics filter rejects; (c) and (d) are the morphology-mismatched and collision-prone grasps produced when whole-body and hand DoFs are not jointly optimized.

Scene Generalization across Real-World Environments

Beyond the per-height generalization reported in the main Analysis (§ 5.2), Figure 8 shows the same policy operating across different real-world environments: kitchens, hallways with railings, multi-tier shelves, dining tables, and open lab benches at varying heights. The policy is trained purely on randomized tabletop scenes in simulation and uses only the egocentric RGB-D and active perception stack at deployment; no scene-specific fine-tuning, per-scene demonstrations, or environment annotation is required. Robustness to surface geometry and clutter follows from the diversity of the training distribution rather than any scene-aware module; lighting and background variation are absorbed by the open-vocabulary detection and segmentation front end, since the policy consumes only the segmented object point cloud.

The full pipeline operating in fourteen real-world environments

Figure 8. Our full pipeline in fourteen real-world environments.

Synthetic Data Scaling

Figure 9 reports real-world SR as a function of the synthetic training-data volume. Each data scale is evaluated with oracle perception on 120 trials (60 ID, 60 OOD), for 1,080 real-world trials over nine data scales; this protocol is separate from Task 2’s 150-trial height sweep, so absolute values are not directly comparable to Figure 3b. SR rises monotonically from 9.2% at 5k trajectories to 89.2% at 400k and plateaus thereafter. OOD objects benefit most (5.0%→90.0%5.0\% \to90.0\%) while ID saturates earlier; the relative OOD degradation narrows from 62% at 5k to under 2% above 300k, indicating that data diversity progressively closes the generalization gap to novel objects.

Figure 9. Data scaling (real world, oracle perception, 5 ID + 5 OOD objects; 120 trials per data scale, a separate protocol from Task 2). SR scales from 9.2% at 5k trajectories to ∼89% at 400k and plateaus thereafter, with OOD benefiting most.

Computational cost. Offline generation: ∼700\sim700 GPU-hours for 600k trajectories (fully parallelizable); CVAE training: ∼1\sim1 h on a single GPU. At deployment (RTX 5090): perception (DINO + SAM) 148 ms, CVAE inference 0.4 ms, motion planning ∼200\sim200 ms, WB-IK ∼1\sim1 ms per batched solve over all KK candidates.

End-to-end grasp attempts take 15–40 s, dominated by physical robot motion.

Large-Scale Simulation Evaluation

This section reports the full results of the Task 3 simulation evaluation (protocol in § 4.1; band aggregates and SR-vs-height curves in Figure 4): the spatial success-rate fields (side and top view) and per-object success rates.

Protocol details.. The evaluation runs in ManiSkill3 [75] with the deployment whole-body controller in the loop. Each condition sweeps 17 table heights (0.10–0.90 m in 5 cm steps) with randomized planar object placement (x∈[0.25,0.55]x \in[0.25, 0.55] m, y∈[−0.175,0.125]y \in[-0.175, 0.125] m) and randomized yaw, over the YCB (12 objects) and RoboCasa [90] (11 objects) catalogs. Scene sets are identical across the six variants, so all deltas are paired. Training tables span 0.40–0.90 m; heights 0.10–0.35 m evaluate extrapolation below the training range.

Success criterion.. A trial succeeds only if it is held and upright: the object is lifted ≥0.05\ge0.05 m and remains lifted after a 2 s hold with ≤0.03\le0.03 m of in-hand slip, while the robot stays upright (pelvis height ≥0.20\ge0.20 m, base tilt ≤30∘\le30^\circ, and foot drift ≤0.05\le0.05 m). Scoring is a pure function of the recorded telemetry, with no manual judgment. This criterion is stricter than the real-world one (lift ≥5\ge5 cm and stably held), since it additionally penalizes post-grasp slip and any loss of balance during the hold; simulation success rates are therefore conservative relative to the hardware tasks.

Spatial envelope and object difficulty.. The success field over height × reach (Figure 10) is dense throughout the 0.10–0.90 × 0.25–0.55 m envelope and thins mainly along the far-reach column (x≥0.50x \ge0.50 m), most severely when far reach is combined with deep squat, where extension and balance compete. Across objects (Figure 12), the broader RoboCasa shape distribution costs ∼8 pts in-range (87.4 → 79.8%) and widens the per-object spread (51.5–93.4% vs. 67.7–94.1% for YCB); the hardest objects are thin or elongated (whisk, chicken drumstick, dish brush) and small near-spherical fruit (peach, apple), which demand centimeter-precise closure. The two degradations are close to additive: YCB in-range (87.4%) to RoboCasa extrapolation (71.4%) costs 16 pts, within ∼2 pts of the sum of the single-axis costs (6.3 + 7.6), so there is no catastrophic interaction.

Figure 10. Side-view success-rate field (Full pipeline): table height vs. forward reach xx, per catalog. The dashed line marks the 0.40 m training floor. Success stays high across the whole 0.10–0.90 × 0.25–0.55 m envelope and degrades only at the far-reach column and the deepest squat row; per-cell nn varies with random placement (8–42), so the field should be read as a whole rather than cell-by-cell.

Figure 12. Per-object success rate (Full pipeline, n=136n=136 trials/object, 95% Wilson CIs). The RoboCasa catalog’s broader shape distribution widens the outcome spread (51.5–93.4% vs. 67.7–94.1% for YCB); the hardest objects are thin and elongated (whisk, chicken drumstick, dish brush), whose narrow graspable surfaces demand centimeter-precise hand placement.

Figure 11. Top-view success-rate fields (plan view; robot at the bottom, grasping-hand side to the right): Full vs. −Dyn. Filter vs. −Compensator on both catalogs, aggregated over all heights. The Full envelope is broad and dense; removing the dynamics filter keeps coverage but at uniformly lower success density; removing the compensator degrades the whole field, worst at far cells.

References

  1. [1]Q. Ben, F. Jia, J. Zeng, J. Dong, D. Lin, and J. Pang. HOMIE: Humanoid loco-manipulation with isomorphic exoskeleton cockpit, 2025. URL https://arxiv.org/abs/2502.13013.
  2. [2]Y. Ze, S. Zhao, W. Wang, A. Kanazawa, R. Duan, P. Abbeel, G. Shi, J. Wu, and C. K. Liu. TWIST2: Scalable, portable, and holistic humanoid data collection system, 2025. URL https://arxiv.org/abs/2511.02832.
  3. [3]T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to-humanoid real-time whole-body teleoperation. arXiv preprint arXiv:2403.04436, 2024.arXiv
  4. [4]Y. Li, Y. Lin, J. Cui, T. Liu, W. Liang, Y. Zhu, and S. Huang. CLONE: Closed-loop whole-body humanoid teleoperation for long-horizon tasks, 2025. URL https://arxiv.org/abs/2506.08931.
  5. [5]T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. RSS, 2023.arxiv.org/abs/2304.13705
  6. [7]T. He, Z. Wang, H. Xue, Q. Ben, Z. Luo, W. Xiao, Y. Yuan, X. Da, F. Castañeda, S. Sastry, C. Liu, G. Shi, L. Fan, and Y. Zhu. VIRAL: Visual sim-to-real at scale for humanoid loco-manipulation. arXiv preprint arXiv:2511.15200, 2025.arXiv
  7. [8]H. Xue, T. He, Z. Wang, Q. Ben, W. Xiao, Z. Luo, X. Da, F. Castañeda, G. Shi, S. Sastry, L. J. Fan, and Y. Zhu. Opening the sim-to-real door for humanoid pixel-to-action policy transfer. arXiv preprint arXiv:2512.01061, 2025. URL https://arxiv.org/abs/2512.01061.
  8. [9]T. Lin, K. Sachdev, L. Fan, J. Malik, and Y. Zhu. Sim-to-real reinforcement learning for vision-based dexterous manipulation on humanoids. arXiv preprint arXiv:2502.20396, 2025.arXiv
  9. [10]S. Wei, H. Jing, B. Li, Z. Zhao, J. Mao, Z. Ni, S. He, J. Liu, X. Liu, K. Kang, S. Zang, W. Yuan, M. Pavone, D. Huang, and Y. Wang. ψ₀: An open foundation model towards universal humanoid loco-manipulation, 2026. URL https://arxiv.org/abs/2603.12263.
  10. [13]R. Wang, J. Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang. Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simulation. In ICRA, 2023.arxiv.org/abs/2210.02697
  11. [14]J. Zhang, H. Liu, D. Li, X. Yu, H. Geng, Y. Ding, J. Chen, and H. Wang. DexGraspNet 2.0: Learning generative dexterous grasping in large-scale synthetic cluttered scenes, 2024. URL https://arxiv.org/abs/2410.23004.
  12. [15]Xu, Y., Wan, W., Zhang, J., Liu, H., Shan, Z., Shen, H., Wang, R., Geng, H., Weng, Y., Chen, J., & al., . et . (2023). Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. CVPR.
  13. [16]S. Deng, M. Yan, S. Wei, H. Ma, Y. Yang, J. Chen, Z. Zhang, T. Yang, X. Zhang, W. Zhang, H. Cui, Z. Zhang, and H. Wang. GraspVLA: a grasping foundation model pre-trained on billion-scale synthetic action data, 2025. URL https://arxiv.org/abs/2505.03233.
  14. [17]R. Dong, Z. Li, A. Gupta, X. He, and S. Gupta. HERO: Learning humanoid end-effector control for visual whole-body open-vocabulary object grasping, 2026. URL https://arxiv.org/abs/2602.16705.
  15. [18]R. Bajcsy. Active perception. Proceedings of the IEEE, 76(8):966–1005, 1988. doi: 10.1109/5.5968.DOI
  16. [19]J. A. Placed, J. Strader, H. Carrillo, N. Atanasov, V. Indelman, L. Carlone, and J. A. Castellanos. A survey on active simultaneous localization and mapping: State of the art and new frontiers, 2023. URL https://arxiv.org/abs/2207.00254.
  17. [20]R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 7(2):2146–2153, 2022. doi: 10.1109/LRA.2022.3144512.DOI
  18. [21]E. S. Hu, J. Wang, X. Yuan, F. Luo, M. Li, G. Lambrechts, O. Rybkin, and D. Jayaraman. Real-world reinforcement learning of active perception behaviors, 2025. URL https://arxiv.org/abs/2512.01188.
  19. [22]Luo, Z., Tessler, C., Lin, T., Yuan, Y., He, T., Xiao, W., Guo, Y., Chechik, G., Kitani, K., Fan, L., & Zhu, Y. (2025). Emergent active perception and dexterity of simulated humanoids from visual reinforcement learning. In arXiv preprint arXiv:2505.12278. https://arxiv.org/abs/2505.12278
  20. [23]H. Qi, Y.-J. Wang, T. Lin, Y. Brent, Y. Ma, K. Sreenath, and J. Malik. Coordinated humanoid manipulation with choice policies. arXiv:2512.25072, 2025.arXiv
  21. [24]D. Kim, M. Srouji, C. Chen, and J. Zhang. ARMOR: Egocentric perception for humanoid robot collision avoidance and motion planning, 2024. URL https://arxiv.org/abs/2412.00396.
  22. [25]M. Liu, Z. Chen, X. Cheng, Y. Ji, R. Qiu, R. Yang, and X. Wang. Visual whole-body control for legged loco-manipulation. The 8th Conference on Robot Learning, 2024.arxiv.org/abs/2403.16967
  23. [26]T. Yin, Z. Mei, T. Sun, L. Zha, E. Zhou, J. Bao, M. Yamane, O. Shorinwa, and A. Majumdar. WoMAP: World models for embodied open-vocabulary object localization, 2025. URL https://arxiv.org/abs/2506.01600.
  24. [27]X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-body control for humanoid robots. In Robotics: Science and Systems (RSS), 2024. URL https://arxiv.org/abs/2402.16796.
  25. [28]C. Lu, X. Cheng, J. Li, S. Yang, M. Ji, C. Yuan, G. Yang, S. Yi, and X. Wang. Mobile-TeleVision: Predictive motion priors for humanoid whole-body control. In IEEE International Conference on Robotics and Automation (ICRA), 2025. URL https://arxiv.org/abs/2412.07773.
  26. [29]Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu. BeyondMimic: From motion tracking to versatile humanoid control via guided diffusion, 2025. URL https://arxiv.org/abs/2508.08241.
  27. [30]L. Yang, X. Huang, Z. Wu, A. Kanazawa, P. Abbeel, C. Sferrazza, C. K. Liu, R. Duan, and G. Shi. OmniRetarget: Interaction-preserving data generation for humanoid whole-body loco-manipulation and scene interaction, 2025. URL https://arxiv.org/abs/2509.26633.
  28. [31]Y. Zhang, Y. Yuan, P. Gurunath, I. Gupta, S. Omidshafiei, A. akbar Agha-mohammadi, M. Vazquez-Chanlatte, L. Pedersen, T. He, and G. Shi. FALCON: Learning force-adaptive humanoid loco-manipulation, 2025. URL https://arxiv.org/abs/2505.06776.
  29. [32]W. Sun, L. Feng, B. Cao, Y. Liu, Y. Jin, and Z. Xie. ULC: A unified and fine-grained controller for humanoid loco-manipulation, 2025. URL https://arxiv.org/abs/2507.06905.
  30. [33]L. Wei, X. Peng, R.-Z. Qiu, T. Huang, X. Cheng, and X. Wang. HMC: Learning heterogeneous metacontrol for contact-rich loco-manipulation, 2025. URL https://arxiv.org/abs/2511.14756.
  31. [34]S. Chen, Z. Cao, Z. Luo, F. Castañeda, C. Li, T. Wang, Y. Yuan, L. Fan, C. K. Liu, and Y. Zhu. CHIP: Adaptive compliance for humanoid control through hindsight perturbation, 2025. URL https://arxiv.org/abs/2512.14689.
  32. [35]T. Zhang, X. Peng, F. Lin, X. Xiong, and Y. Lou. Whole-body compliance control for quadruped manipulator with actuation saturation of joint torque and ground friction. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11124–11131, 2024. doi: 10.1109/IROS58592.2024.10802754.DOI
  33. [36]Y. Niu, Z. Fang, B. Chen, S. Zhou, R. K. Senthilkumaran, H. Zhang, B. Chen, C. Qiu, H. E. Tseng, J. Francis, and D. Zhao. Learning versatile humanoid manipulation with touch dreaming, 2026. URL https://arxiv.org/abs/2604.13015.
  34. [37]M. Ji, X. Peng, F. Liu, J. Li, G. Yang, X. Cheng, and X. Wang. ExBody2: Advanced expressive humanoid whole-body control. arXiv preprint arXiv:2412.13196, 2024.arXiv
  35. [38]Z. Luo, Y. Yuan, T. Wang, C. Li, S. Chen, F. Castañeda, Z.-A. Cao, J. Li, D. Minor, Q. Ben, X. Da, R. Ding, C. Hogg, L. Song, E. Lim, E. Jeong, T. He, H. Xue, W. Xiao, Z. Wang, S. Yuen, J. Kautz, Y. Chang, U. Iqbal, L. Fan, and Y. Zhu. SONIC: Supersizing motion tracking for natural humanoid whole-body control. arXiv preprint arXiv:2511.07820, 2025.DOI
  36. [39]J. Li, X. Cheng, T. Huang, S. Yang, R.-Z. Qiu, and X. Wang. AMO: Adaptive motion optimization for hyper-dexterous humanoid whole-body control, 2025. URL https://arxiv.org/abs/2505.03738.
  37. [40]T. He, Z. Luo, X. He, W. Xiao, C. Zhang, W. Zhang, K. Kitani, C. Liu, and G. Shi. OmniH2O: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. In Conference on Robot Learning (CoRL), 2024. URL https://arxiv.org/abs/2406.08858.
  38. [41]Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, and C. Finn. HumanPlus: Humanoid shadowing and imitation from humans. In Conference on Robot Learning (CoRL), 2024. URL https://arxiv.org/abs/2406.10454.
  39. [42]Ze, Y., Chen, Z., Araújo, J. P., Cao, Z., Peng, X. B., Wu, J., & Liu, C. K. (2025). TWIST: Teleoperated whole-body imitation system. https://arxiv.org/abs/2505.02833
  40. [43]X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-TeleVision: Teleoperation with immersive active visual feedback. In Conference on Robot Learning (CoRL), 2024. URL https://arxiv.org/abs/2407.01512.
  41. [44]Y. Li, L. Ma, Y. Lin, Y. Du, M. Liu, K. Hu, J. Cui, Y. Zhu, W. Liang, B. Jia, and S. Huang. OmniClone: Engineering a robust, all-rounder whole-body humanoid teleoperation system, 2026. URL https://arxiv.org/abs/2603.14327.
  42. [45]T. Zhu, G. Cai, Z. Yang, G. Ren, H. Xie, Z. Wang, J. Wu, J. Wang, X. Yang, Y. Mu, and Y. Yan. CLOT: Closed-loop global motion tracking for whole-body humanoid teleoperation, 2026. URL https://arxiv.org/abs/2602.15060.
  43. [46]R. Nai, B. Zheng, J. Zhao, H. Zhu, S. Dai, Z. Chen, Y. Hu, Y. Hu, T. Zhang, C. Wen, and Y. Gao. Humanoid manipulation interface: Humanoid whole-body manipulation from robot-free demonstrations, 2026. URL https://arxiv.org/abs/2602.06643.
  44. [47]L. Heng, Y. Tang, J. Xu, H. Bao, D. Huang, and Y. Wang. HumDex: Humanoid dexterous manipulation made easy, 2026. URL https://arxiv.org/abs/2603.12260.
  45. [48]J. Li, Y. Zhu, Y. Xie, Z. Jiang, M. Seo, G. Pavlakos, and Y. Zhu. OKAMI: Teaching humanoid robots manipulation skills through single video imitation. In Conference on Robot Learning (CoRL), 2024. URL https://arxiv.org/abs/2410.11792.
  46. [49]R.-Z. Qiu, S. Yang, X. Cheng, C. Chawla, J. Li, T. He, G. Yan, D. J. Yoon, R. Hoque, L. Paulsen, G. Yang, J. Zhang, S. Yi, G. Shi, and X. Wang. Humanoid Policy ∼ human policy, 2025. URL https://arxiv.org/abs/2503.13441.
  47. [50]Ze, Y., Chen, Z., Wang, W., Chen, T., He, X., Yuan, Y., Peng, X. B., & Wu, J. (2025). Generalizable humanoid manipulation with 3D diffusion policies. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). https://arxiv.org/abs/2410.10803
  48. [51]G. Jiang, Y. Liang, J. Ye, J.-Y. Huang, C. Jing, R. Duan, P. Abbeel, X. Wang, and X. Zou. Cross-hand latent representation for vision-language-action models, 2026. URL https://arxiv.org/abs/2603.10158.
  49. [52]R.-Z. Qiu, Y. Song, X. Peng, S. A. Suryadevara, G. Yang, M. Liu, M. Ji, C. Jia, R. Yang, X. Zou, and X. Wang. WildLMa: Long horizon loco-manipulation in the wild. arXiv preprint arXiv:2411.15131, 2024.arxiv.org/abs/2411.15131
  50. [53]X. Xu, Y. Zhang, Y.-L. Li, L. Li, L. Han, and C. Lu. HumanVla: Towards vision-language directed object rearrangement by physical humanoid. Advances in Neural Information Processing Systems, 37:18633–18659, 2024.arxiv.org/abs/2406.19972
  51. [54]Z. Luo, J. Cao, S. Christen, A. Winkler, K. Kitani, and W. Xu. Omnigrasp: Grasping diverse objects with simulated humanoids. Advances in Neural Information Processing Systems, 37:2161–2184, 2024.arxiv.org/abs/2407.11385
  52. [55]W. Liu, Y. Wan, J. Wang, Y. Kuang, W. Cui, X. Shi, H. Li, D. Zhao, Z. Zhang, and H. Wang. FetchBot: Learning generalizable object fetching in cluttered scenes via zero-shot Sim2Real, 2025. URL https://arxiv.org/abs/2502.17894.
  53. [56]N. Sun, H. Shen, Y. Zhang, L. Feng, H. Wang, Y. Xiangli, Y. Jin, and P. Liu. DexWeave: Learning dexterous humanoid loco-manipulation from human demonstrations. arXiv, 2026.arxiv.org/abs/2609.34724
  54. [57]J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017. URL https://arxiv.org/abs/1703.06907.
  55. [58]X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), page 3803–3810. IEEE, May 2018. doi: 10.1109/icra.2018.8460528. URL http://dx.doi.org/10.1109/ICRA.2018.8460528.
  56. [59]OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation, 2019. URL https://arxiv.org/abs/1808.00177.
  57. [60]OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving Rubik’s cube with a robot hand, 2019. URL https://arxiv.org/abs/1910.07113.
  58. [61]J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots, 2018. URL https://arxiv.org/abs/1804.10332.
  59. [62]A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid motor adaptation for legged robots, 2021. URL https://arxiv.org/abs/2107.04034.
  60. [63]Z. Xie, G. Berseth, P. Clary, J. Hurst, and M. van de Panne. Feedback control for Cassie with deep reinforcement learning, 2018. URL https://arxiv.org/abs/1803.05580.
  61. [64]L. Wei, J. Ma, Y. Hu, and R. Zhang. Ensuring force safety in vision-guided robotic manipulation via implicit tactile calibration, 2025. URL https://arxiv.org/abs/2412.10349.
  62. [65]Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Ojea, J. A., & Goldberg, K. (2017). Dex-Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. https://arxiv.org/abs/1703.09312
  63. [66]Zhang, G., Xu, Q., Zhang, H., Ma, J., He, L., Bao, Y., Ping, Z., Yuan, Z., Lu, C., Yuan, C., & al., . et . (2026). UniDex: A robot foundation suite for universal dexterous hand control from egocentric human videos. In arXiv preprint arXiv:2603.22264. 2603.22264
  64. [67]Chao, Y.-W., Yang, W., Xiang, Y., Molchanov, P., Handa, A., Tremblay, J., Narang, Y. S., Van Wyk, K., Iqbal, U., Birchfield, S., & al., . et . (2021). DexYCB: A benchmark for capturing hand grasping of objects. CVPR.
  65. [68]J. Ye, L. Wei, G. Jiang, C. Jing, X. Zou, and X. Wang. From power to precision: Learning fine-grained dexterity for multi-fingered robotic hands, 2025. URL https://arxiv.org/abs/2511.13710.
  66. [69]Y. Wang, Z. Xian, F. Chen, T.-H. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. RoboGen: Towards unleashing infinite data for automated robot learning via generative simulation, 2024. URL https://arxiv.org/abs/2311.01455.
  67. [70]A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. MimicGen: A data generation system for scalable robot learning using human demonstrations, 2023. URL https://arxiv.org/abs/2310.17596.
  68. [71]Z. Xue, S. Deng, Z. Chen, Y. Wang, Z. Yuan, and H. Xu. DemoGen: Synthetic demonstration generation for data-efficient visuomotor policy learning, 2025. URL https://arxiv.org/abs/2502.16932.
  69. [72]L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang. GenSim: Generating robotic simulation tasks via large language models, 2024. URL https://arxiv.org/abs/2310.01361.
  70. [73]Yu, T., Xiao, T., Stone, A., Tompson, J., Brohan, A., Wang, S., Singh, J., Tan, C., M, D., Peralta, J., Ichter, B., Hausman, K., & Xia, F. (2023). Scaling robot learning with semantically imagined experience. https://arxiv.org/abs/2302.11550
  71. [74]NVIDIA, Mittal, M., Roth, P., Tigue, J., Richard, A., Zhang, O., & al., . et . (2025). Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning. https://arxiv.org/abs/2511.04831
  72. [75]S. Tao, F. Xiang, A. Shukla, Y. Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y. Liu, T. kai Chan, Y. Gao, X. Li, T. Mu, N. Xiao, A. Gurha, V. N. Rajesh, Y. W. Choi, Y.-R. Chen, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su. ManiSkill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI, 2025. URL https://arxiv.org/abs/2410.00425.
  73. [76]M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3D objects, 2022. URL https://arxiv.org/abs/2212.08051.
  74. [77]R. Eidenberger and J. Scharinger. Active perception and scene modeling by planning with probabilistic 6D object poses. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1036–1043, 2010. doi: 10.1109/IROS.2010.5651927.DOI
  75. [78]S. Isler, R. Sabzevari, J. Delmerico, and D. Scaramuzza. An information gain formulation for active volumetric 3D reconstruction. In IEEE International Conference on Robotics and Automation (ICRA), pages 3477–3484, 2016. doi: 10.1109/ICRA.2016.7487527.DOI
  76. [79]J. Delmerico, S. Isler, R. Sabzevari, and D. Scaramuzza. A comparison of volumetric information gain metrics for active 3D object reconstruction. Autonomous Robots, 42(2):197–208, 2018. doi: 10.1007/s10514-017-9634-0.DOI
  77. [80]N. Dengler, S. Pan, V. Kalagaturu, R. Menon, M. Dawood, and M. Bennewitz. Viewpoint push planning for mapping of unknown confined spaces. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023. URL https://arxiv.org/abs/2303.03126.
  78. [81]X. Zou, Y. Song, R.-Z. Qiu, X. Peng, J. Ye, S. Liu, and X. Wang. M3: 3D-spatial MultiModal memory, 2025. URL https://arxiv.org/abs/2503.16413.
  79. [82]M. Breyer, L. Ott, R. Siegwart, and J. J. Chung. Closed-loop next-best-view planning for target-driven grasping. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022. URL https://arxiv.org/abs/2207.10543.
  80. [83]S. Jauhri, S. Lueth, and G. Chalvatzaki. Active-perceptive motion generation for mobile manipulation. In IEEE International Conference on Robotics and Automation (ICRA), 2024. URL https://arxiv.org/abs/2310.00433.
  81. [84]P. Kaiser, C. Mandery, A. Boltres, and T. Asfour. Affordance-based multi-contact whole-body pose sequence planning for humanoid robots in unknown environments. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3114–3121, 2018. doi: 10.1109/ICRA.2018.8461087.DOI
  82. [85]J. Ye, X. Peng, C. Mc Gartoll, C. Jing, L. Wei, J. Chen, X. Zou, and X. Wang. RoboKit: Accelerating parallel differentiable optimization for robot learning. In Conference on Robot Learning (CoRL), 2026.arxiv.org/abs/2611.20185
  83. [86]S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In European Conference on Computer Vision (ECCV), 2024. URL https://arxiv.org/abs/2303.05499.
  84. [87]C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 652–660, 2017. URL https://arxiv.org/abs/1612.00593.
  85. [88]A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026, 2023. URL https://arxiv.org/abs/2304.02643.
  86. [89]B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar. Benchmarking in manipulation research: Using the Yale-CMU-Berkeley object and model set. IEEE Robotics & Automation Magazine, 22(3):36–52, 2015. doi: 10.1109/MRA.2015.2448951.DOI
  87. [90]S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024. URL https://arxiv.org/abs/2406.02523.
  88. [91]F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11097–11107, 2020.arxiv.org/abs/2003.08515

Paper details

vlnc:W_w45pn9mg
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents