WorksComputer Science
HuG: Learning Humanoid Whole-Body Affordance-Aware Grasping with Active Perception
Abstract
Acquiring objects in human environments is hard for humanoids: targets are partially occluded under egocentric viewpoints, and reaching them demands coordinated whole-body motion that keeps the target visible and reachable while the base stays stable. We present HuG, a framework for whole-body affordance-aware grasping with active perception that couples perception and action through a Perceive-Align-Grasp pipeline: belief-driven active perception resolves occlusion by treating viewpoint as an explicit decision variable; a point-cloud-conditioned affordance policy proposes whole-body grasps; and an affordance-aware alignment stage repositions the body to the most executable configuration. A key insight is that, on a floating base, kinematic validity is not enough: we curate training data with controller-in-the-loop dynamics simulation, rejecting 73% of kinematically valid grasps that would destabilize the deployment controller, so the policy learns kinodynamically feasible affordances. Trained entirely in simulation with zero real demonstrations, HuG attains 87.5% real-world success on a multi-level shelf task, generalizes to novel objects, and holds at least 73% success at every tested height. Beyond generalizing within its training distribution, the policy extrapolates spatially: at scale in simulation it succeeds at table heights well below any seen during training, indicating that kinodynamically filtered synthetic data grounds affordances in object geometry and whole-body feasibility rather than in memorized configurations.
xuanbinpeng.com/hug
Conference on Robot Learning (CoRL) 2026, Austin, TX, USA
Introduction
Humanoid robots promise to bring general-purpose object acquisition into human environments. Yet in realistic homes and labs, targets are rarely presented in a clean tabletop configuration: they are frequently partially occluded under egocentric viewpoints and spread across diverse spatial configurations, and reaching them demands coordinated whole-body motion—turning the head, bending the torso, squatting—to maintain visibility, reachability, and stability at once. Consider retrieving an item from a cluttered multi-level shelf: the target may be visible only from a specific angle, sit at a height requiring a deep squat or a full extension, and afford a stable grasp only from particular approach directions. Such tasks demand tightly coupling perception with whole-body action: not as independent modules, but as stages in which viewpoint, posture, and grasp decisions jointly shape one another.
Three threads of prior work have each made striking progress on this coupling, yet each leaves a critical gap. (i) Imitation from real demonstrations scales humanoid manipulation through teleoperation [1, 2, 3, 4], with ACT [5]-style architectures and diffusion policies such as Via [6] supporting long-horizon tasks; but these policies are bounded by their demonstration coverage, generalizing poorly to objects and layouts outside it; worse, on a floating base every action error also shifts the body, so errors compound over long-horizon rollouts into end-effector drift that centimeter-level grasping cannot absorb. (ii) Large-scale synthetic learning enables zero-shot sim-to-real via end-to-end policies [7, 8, 9, 10] and modular grasp synthesis [13, 14, 15, 16, 17] [11, 12]; but end-to-end humanoid policies reason about dynamics only implicitly, while grasp planners are built for fixed-base tabletops, so a kinematically valid grasp can be dynamically infeasible once a floating-base humanoid must execute it under whole-body control. Neither line explicitly enforces kinodynamic whole-body feasibility during data generation. (iii) Active perception regulates viewpoints, from classical active perception and active SLAM [18, 19] to learned perceptual strategies [20, 21, 22] [6]; but on humanoids, viewpoint regulation either emerges implicitly from demonstrations, bounded by their coverage [6], or is confined to the head with the body left passive [23]. Yet a body-attached camera binds viewpoint to whole-body posture [24, 25, 26], so resolving occlusion can require lowering the torso or leaning, not merely looking. Humans, by contrast, couple perception and whole-body action instinctively: they lean past occlusions, squat to low shelves, and turn to approach from stable angles. Current humanoid grasping systems lack this coupling.
We present HuG, a hierarchical framework for whole-body affordance-aware grasping with active perception ( Figure 1) that realizes this coupling through a Perceive–Align–Grasp pipeline: (1) **belief-driven active perception, a Bayes-style belief filter over an egocentric viewpoint grid that resolves occlusion by systematic search, treating viewpoint as an explicit decision variable rather than emergent behavior; (2) kinodynamically filtered affordance learning, where an affordance is a kinodynamically feasible whole-body grasp configuration for the observed object: a point-cloud-conditioned CVAE is trained on 600k synthetic trajectories that each pass controller-in-the-loop dynamics simulation, which rejects 73% of kinematically valid grasps that would destabilize the controller; and (3) whole-body affordance-aware alignment, which samples diverse candidates and repositions the whole body to the configuration of highest execution quality rather than highest grasp confidence. Trained entirely from synthetic data with zero real demonstrations, our system attains 87.5% real-world success on a multi-level shelf task, generalizes across novel objects (90/85% ID/OOD), holds at every tested height, and transfers zero-shot to diverse everyday scenes with cluttered backgrounds and unseen objects. At scale in simulation, the same policy extrapolates to table heights well below its training range, where every single-component ablation loses 25–81 points: the learned affordances are grounded in object geometry and whole-body feasibility, not in memorized configurations.

Figure 1. Humanoid whole-body affordance-aware grasping. Top: our system runs zero-shot on a Unitree G1, coupling active perception with whole-body control to discover and grasp objects across diverse everyday scenes, from deep squats at low tables to full extension at high shelves. Bottom: the policy is trained entirely on large-scale synthetic data generated and filtered with kinodynamic feasibility.
Our contributions are:
Kinodynamic-aware whole-body affordance learning. A large-scale synthetic data engine that co-optimizes whole-body posture with the hand and enforces controller-in-the-loop dynamic feasibility, so the learned affordances are kinodynamically feasible by construction.
A zero-demonstration Perceive–Align–Grasp system. A complete humanoid pipeline that discovers occluded targets, repositions the whole body to an executable grasp configuration, and executes it under whole-body control, without any real-world demonstration.
Extensive real-world validation. The complete pipeline runs zero-shot on a Unitree G1 humanoid, outperforming imitation and grasp-planning baselines and generalizing to unseen objects, heights, and everyday scenes.
Related Work
Humanoid Whole-Body Loco-Manipulation
Coordinating locomotion and manipulation on humanoids couples floating-base stability with end-effector precision. Learned whole-body controllers now track expressive motions and coordinate legs and arms under contact and force disturbances [27, 28, 29, 30, 31, 32, 33, 34, 35, 36], and teleoperation systems built on them have made whole-body demonstration collection practical enough to train capable manipulation policies [1, 2, 3, 4, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52]. These policies, however, remain bounded by their demonstration coverage, and on a floating base, pose drift compounds over long-horizon rollouts. Sim-to-real visual policies [7, 8, 9, 10, 17, 22, 25, 53, 54, 55, 56] remove the demonstration bottleneck, most directly VIRAL [7], but leave perception to data scaling. In contrast, we formulate perception as an active optimization problem, coordinating whole-body motion to search for and maximize target observability.
Learning from Synthetic Data
The cost of real interaction data has driven a shift toward simulation-based learning, from domain randomization [57, 58, 59, 60, 61, 62, 63, 64] and synthetic grasp datasets [13, 14, 15, 65, 66, 67, 68] [11, 12] to generative, automated pipelines [69, 70, 71, 72, 73]. Most directly comparable to ours, billion-scale grasping foundation models [16] backed by GPU-accelerated simulators [74, 75] and large asset libraries [76] achieve zero-shot generalization in cluttered scenes. These efforts, however, predominantly target fixed-base settings or omit the dynamical plausibility of generated samples, admitting candidates that are kinematically valid but dynamically infeasible under closed-loop whole-body control. In contrast, we co-optimize whole-body posture with the hand for each grasp and enforce kinodynamic feasibility under whole-body execution on a floating base, with the deployment controller in the loop.
Active Perception for Manipulation
Active perception spans reconstruction, recognition, and manipulation [18, 19, 77, 78, 79, 80, 81], and has been coupled with grasping on fixed arms and mobile manipulators [20, 21, 82, 83], where the camera moves largely independently of balance. On humanoids the camera is body-attached, so a viewpoint change is a whole-body motion [24, 25, 26], and reaching a viewpoint is itself a whole-body planning problem [84]. Recent humanoid systems address this only partially: ViA [6] learns coordinated head–bimanual control from demonstrations but is bounded by their coverage, and Choice Policy [23] regulates an active head while leaving the body passive, so the camera cannot reach the configurations occlusion demands.
Method
Our framework comprises two complementary pipelines (Figure 2). The system pipeline follows a Perceive–Align–Grasp structure: active perception (§ 3.2) discovers the target under egocentric occlusion; the affordance model proposes whole-body grasp configurations, which alignment ranks and repositions the body toward; and a collision-aware planner executes the selected grasp through the whole-body controller (§ 3.4). The training pipeline builds large-scale synthetic data filtered for kinodynamic feasibility, yielding the affordance model that drives both alignment and grasping (§ 3.3). Both share one hierarchical whole-body controller coupling upper-body IK with an RL lower-body policy (§ 3.1).

Figure 2. System and training pipelines. The system pipeline (top) follows a Perceive–Align–Grasp structure, combining active perception, whole-body alignment, and robust grasp execution. The training pipeline (bottom) constructs large-scale synthetic data with kinodynamic filtering to learn affordance-aware whole-body grasping.
Hierarchical Whole-Body Controller
Whole-body grasping places two competing demands on the controller: task-space precision for end-effector placement, and floating-base stability across the diverse postures that different object heights and reaches require. End-to-end policies struggle to meet both under the tight tolerances of zero-shot sim-to-real transfer.
A hierarchical two-layer design decouples these concerns. The high-level whole-body IK [85] maps desired end-effector poses (wrist, hand, and head, in the world frame) to upper-body joint targets and lower-body commands , with the pelvis height. For the low-level RL locomotion policy, we use the decoupled whole-body controller of SONIC (GR00T) [38], which tracks from proprioception, outputting lower-body targets that realize the motion while maintaining balance.
This coupling poses a challenge: as the RL policy shifts pelvis height or base position, the root pose drifts in the world frame, and an IK solver blind to this drift renders world-frame targets unreachable—fatal for centimeter-level grasping. A closed-loop pelvis compensator closes the gap, pre-warping every task-space target by the low-pass-filtered mismatch between the IK-assumed and FK-realized pelvis pose (App. § 8 gives the update and filter). Crucially, the same controller drives synthetic-data filtering (§ 3.3), so training data is supported on actions the deployment system can actually execute.
Active Perception
We cast object discovery under egocentric occlusion as sequential viewpoint selection that maximizes the probability of detecting the target. The module is a finite-state machine with a Search mode (global exploration) and an Inspect mode (local refinement after detection); the target is named by a text query to Grounding DINO [86] (e.g., “red cup”), making acquisition language-conditioned without a separate language module.
Search. We maintain a normalized belief , the probability that the target is visible from camera pose , over a discretized grid of camera yaw and height. Each camera pose is a head-pose target for the whole-body controller (§ 3.1), so reaching a viewpoint is a whole-body motion: the robot squats or rises to look, not only turns its head. For a candidate pose , the view frustum projects to a mask with probability of detection . We execute the viewpoint maximizing , with the displacement from the current camera pose, trading detection likelihood against camera motion and revisiting.
Belief update. After observing Grounding DINO confidence at the executed viewpoint , cells in the frustum are updated by a Bayes-style multiplicative rule,
with a detection-dependent likelihood ratio that amplifies frustum cells on a hit ( for , rising with confidence above threshold ), decays them on a miss ( for ), and leaves cells outside the frustum unchanged. Hits concentrate belief; misses redistribute mass to unvisited viewpoints, driving systematic exploration.
Inspect and termination. Search hands off to Inspect once stays above a convergence threshold for steps; Inspect then refines the viewpoint by point-cloud visibility quality (visible-point count, depth continuity, normal stability) and terminates when observation quality saturates, passing a stable observation to alignment.
Learning Kinodynamic-Aware Whole-Body Affordance
Given a stable observation, the robot must decide how to approach and grasp the object with its whole body. We learn a whole-body affordance policy that maps a single-view segmented point cloud to candidate keyframe actions, each a (pre-grasp, grasp, post-grasp) sequence of end-effector poses and hand commands executed by the whole-body controller. Every training sample has survived controller-in-the-loop execution, so the learned affordances are kinodynamically feasible by construction.
Large-scale World-frame Randomization. Unlike fixed-base grasping, where object variation is largely in-plane, a floating-base humanoid must adapt its entire body to different object heights and reaches. We randomize table geometry and object placement across a wide reachable workspace in the world frame, with objects drawn from a large-scale mesh dataset under randomized scale and mass (App. § 9). This forces diverse whole-body strategies: squatting for low objects, extending for far ones, and adapting the whole-body configuration to everything in between.
Kinodynamic-aware Data Engine. We build the dataset through a three-stage pipeline: (i) whole-body–hand grasp co-optimization jointly synthesizes the whole-body configuration and hand joints for each of candidates by energy-based optimization in joint space [85], coupling hand contact with a reachable whole-body posture, unlike gripper-based or fixed-upper-body generators that ignore humanoid morphology (energy terms in App. § 9); (ii) motion planning builds collision-free pre-graspgrasppost-grasp trajectories; (iii) controller-in-the-loop dynamics simulation rolls out each trajectory under the same WB-IK + RL stack used at deployment. A trajectory is kept only if it satisfies three criteria: task success (object lifted cm and held), stable execution (no fall, bounded base attitude), and trackability (end-effector error within deployment tolerances). This last stage is the key to kinodynamic awareness: in our profiling it rejects 73% of kinematically valid grasps (§ 5.1), because the whole-body controller cannot stay stable through the reach or IK tracking diverges under interaction forces. We further filter by camera visibility to avoid degenerate observations (App. § 9).
Observation and Action Spaces. The observation is a segmented single-view partial point cloud of the target, transformed from camera to robot task frame to match the egocentric deployment view. The action is a three-stage keyframe trajectory , where each stage gives end-effector translation , orientation (unit quaternion), and hand joint targets .
Whole-body Affordance Policy. Since a point cloud admits multiple feasible grasps, we model the affordance as a multimodal distribution with a point-cloud conditioned CVAE: PointNet [87] encodes , conditioning a CVAE that decodes the keyframe action from a sampled latent. At inference we draw several latents for diverse proposals, ranked and executed by the alignment stage (§ 3.4; architecture and training in App. § 10).
Whole-Body Alignment and Execution
Once active perception settles on a stable viewpoint, the target is segmented with SAM [88] into an object-only point cloud and passed to the affordance model (§ 3.3) for grasp proposal and whole-body alignment.
Candidate Generation and Ranking. The CVAE decoder draws latents into diverse candidate keyframe sequences , trained only on kinodynamically verified data, its proposals are strongly biased toward configurations the whole-body controller can execute. For each, we solve whole-body IK for the pre-grasp target and rank by , where is IK quality (inverse residual norm) and the joint-space reconfiguration cost. This favors kinematically achievable proposals that need minimal whole-body motion, which matters for floating-base stability. IK evaluation is batched and lightweight ( ms for all candidates).
Execution. The top-ranked candidate is executed via collision-aware Cartesian motion planning through the whole-body controller (§ 3.1); if planning fails, e.g., due to scene collision, the system falls back to the next candidate in ranked order. The robot approaches the pre-grasp pose, closes the hand, and performs the post-grasp lift. Unlike methods that select from a predefined library of stable poses, our alignment configurations are learned from kinodynamically filtered data, object-specific via point-cloud conditioning, and dynamics-aware by construction.
Experiments
We evaluate our system through three complementary evaluations: Task 1 tests full pipeline integration under realistic shelf conditions; Task 2 isolates manipulation from perception with oracle observations on real hardware; Task 3 measures spatial and object generalization at scale in simulation, including extrapolation beyond the training height range. Beyond aggregate comparison, we investigate whether large-scale kinodynamically filtered synthetic training generalizes across the full humanoid workspace and novel objects, which pipeline component fails in which regime, and how performance scales with synthetic data volume.
Experimental Setup
Hardware. We evaluate on the Unitree G1 humanoid with a 3-DOF active head [39], ZED Mini stereo camera, and Inspire dexterous hands. All inference runs on an external desktop (RTX 5090); commands are transmitted via Ethernet to the onboard computer.
Task 1: Full Pipeline (Perceive Align Grasp). Objects are placed on a multi-level shelf with partial occlusions and randomized heights. The robot starts from a nominal pose with no prior knowledge of the target and must autonomously discover, align to, and grasp the object. We run trials over 10 YCB [89] objects (5 ID, 5 OOD); OOD objects are held out from the training set. Target objects are specified via natural language, conditioning the DINO text prompt for detection.
Task 2: Align & Grasp (Oracle Perception). To isolate manipulation from perception confounds, we provide oracle observations on a height-adjustable table at five controlled levels: H1 (0.50 m), H2 (0.60 m), H3 (0.70 m), H4 (0.80 m), H5 (0.85 m); 10 objects 5 heights 3 trials = 150 (75 ID, 75 OOD). Task 2 is the hardware anchor for the height profile measured at scale in Task 3. ACT and ViA receive demonstrations at H2 and H4 only, so the remaining heights measure their sensitivity to demonstration coverage; our method trains on continuously randomized heights in simulation.
Task 3: Large-Scale Simulation Evaluation. To evaluate spatial generalization at a scale impractical on hardware, we sweep 17 table heights (0.10–0.90 m in 5 cm steps) with randomized planar placement and randomized object yaw, over two object catalogs: YCB (12 everyday objects) and RoboCasa [90] (11 objects spanning a broader shape distribution: smaller, thinner, and more elongated geometries, e.g., peeler, whisk, chicken drumstick). Six system variants (Ours and five ablations/baselines, below) are evaluated on identical scene sets, so every delta is paired; ∼18,800 trials in total. Success is scored from telemetry under a criterion stricter than the real-world one: the object must be lifted ≥5 cm and held for 2 s with bounded in-hand slip, and the robot must remain upright (App. E.5). Crucially, training tables span only 0.40–0.90 m: all evaluations at 0.10–0.35 m probe extrapolation below the training range.
Metrics. Success rate (SR): object lifted ≥5 cm and stably held. For Task 1 we additionally report Det. (%): object successfully localized, and (%): grasp success given detection, which decomposes failures into perception vs. manipulation stages.
Baselines. We compare against end-to-end imitation methods, modular grasp planners, and ablations of our pipeline. Vanilla ACT [5]: end-to-end visuomotor policy with a fixed head and no active viewpoint control. ACT + Active Head: ACT with the active head enabled during both demonstration collection and policy execution, isolating the contribution of active perception. ViA [6]: diffusion policy with learned active perception, adapted to floating-base control. GraspNet + WB-IK [11]: parallel-jaw grasp poses from the pretrained checkpoint, mapped to dexterous-hand pre-grasps and executed via our WB-IK controller (+ AP: adding our active perception). Dex1B (adapted) + WB-IK + AP [12]: large-scale dexterous grasp framework adapted to our humanoid. Data is generated across the full workspace with upper-body optimization (arm IK only, no whole-body–hand co-optimization), without dynamics filtering or alignment. Dex1B shares our CVAE + PointNet architecture and hand type, directly isolating the contribution of kinodynamic filtering and whole-body alignment. ACT and ViA are trained on teleoperated demonstrations of ID objects (Task 1: 70 full-pipeline demonstrations per object at every evaluated shelf level, with randomized horizontal placement, so their height coverage matches the test conditions; Task 2: grasp-only at H2/H4). Our method uses zero real demonstrations: 600k synthetic trajectories only.
| Method | SR (%) | Stage (%) | ||
| All | ID/OOD | Det. | Gr|Det | |
| Baselines | ||||
| Vanilla ACT | 32.5 | 55/10 | 52.5 | 61.9 |
| ACT + Active Head | 40.0 | 65/15 | 57.5 | 69.6 |
| ViA (adapted) | 50.0 | 85/15 | 70.0 | 71.4 |
| GraspNet + WB-IK | 22.5 | – | 50.0 | 45.0 |
| + Our Perc. | 40.0 | – | 90.0 | 44.4 |
| Dex1B + WB-IK + AP | 52.5 | 55/50 | 92.5 | 56.8 |
| Ours (Full) | 87.5 | 90/85 | 92.5 | 94.6 |
| Abl. | ||||
| –Perceive | 50.0 | 55/45 | 57.5 | 87.0 |
| –Whole-Body | 20.0 | 20/20 | 87.5 | 22.9 |
| –Align | 62.5 | 65/60 | 85.0 | 73.5 |
| –Dyn. Filter | 62.5 | 65/60 | 90.0 | 69.4 |
Table 3a.
Figure 3b.
| Stage | Filter | Output | Survival |
| 1 | Grasp opt. + IK + coll. | 12.4 M | 2.4% |
| 2 | Collision-free MP | 298 K | 2.4% |
| 3 | Dynamics sim. (WBC) | 80.7 K | 27.0% |
| Final (+ view sel.) | 18 K | — |
Table 3c.
Ablations. –Perceive: fixed camera pose, bypassing active viewpoint search. –Whole-Body: active perception is enabled for object discovery, but the base is fixed during grasping, with no squatting or torso adaptation (arm-only manipulation). –Align: skip the whole-body alignment stage; instead of ranking candidates and repositioning the body, sample a single grasp candidate and execute directly from the current configuration. –Dyn. Filter: remove dynamics simulation from the data pipeline, retaining only kinematic and collision filtering. Compensator: disable the closed-loop pelvis compensator (§ 3.1), so task-space targets are not corrected for pelvis drift.
Main Results
Figure 3a summarizes Task 1. Our full system reaches 87.5% SR (90/85 ID/OOD) with 94.6% Gr|Det (once the object is found, grasping almost always succeeds), using zero real demonstrations, while the strong imitation baseline (ViA) needs teleoperated demos to reach 50%.

Figure 3. Quantitative results. (a) Task 1 full-pipeline evaluation (, 10 objects, 5 ID/5 OOD); Det. = object localized, Gr|Det = grasp success given detection. (b) Task 2 spatial generalization across five heights (oracle perception, 10 objects; solid = ID, hatched = OOD, green = ACT/ViA demo heights): ours everywhere, while ACT/ViA succeed only at their demonstrated heights (a demonstration-coverage effect; § 5.2) and Dex1B/–Dyn. Filter degrade at the extremes. (c) Kinodynamic data filtering: 73% of kinematically valid grasps fail under the deployment whole-body controller (508 M SDF candidates 18 K training grasps).
Baselines fail along distinct axes. End-to-end imitation memorizes object-specific behavior: ViA holds 85% ID but collapses to 15% OOD, as cumulative errors over the long-horizon floating-base action space stall it on unseen objects. Gripper-based planners fail at execution, not discovery: adding our perception to GraspNet lifts Det. , yet Gr|Det is unchanged (), because parallel-jaw pre-grasps give postures unnatural for the humanoid hand (Figure 7c) and hand–environment collisions (Figure 7d). Large-scale dexterous data alone is insufficient: Dex1B (adapted), which shares our CVAE + PointNet and hand type, matches our Det. (92.5%) yet reaches only 56.8% Gr|Det, below –Dyn. Filter (69.4%): built for tabletop settings, its data lacks both whole-body–hand co-optimization and dynamic feasibility (§ 5.2.1 dissects this by height).

Figure 7. Failure modes eliminated by our pipeline. (a) No dynamics filter: unstable jittering. (b) No WB co-optimization/alignment: twisted infeasible poses. (c) GraspNet: unnatural gripper-designed grasps. (d) No WB–hand co-optimization: hand–environment collisions.
Each module addresses a distinct failure stage. Stage-wise metrics separate perception (Det.) from manipulation (Gr|Det): –Perceive cuts Det. by 35 pts (perception is the discovery bottleneck, § 5.2.2), while the three manipulation ablations each collapse Gr|Det along a different axis. –Whole-Body falls to 22.9% (cannot reach without squatting), –Align to 73.5% (sub-optimal approach from the current pose), and –Dyn. Filter to 69.4% (dynamically infeasible training data). The full 94.6% confirms all four are necessary; we dissect them next and resolve each ablation’s failure regime across the workspace in § 5.2.1.
Analysis
Kinodynamic filtering rejects 73% of valid grasps
Kinematic feasibility alone is insufficient for a floating base. Figure 3c traces the pipeline on an example profiling run: 508M candidates yield 18K training grasps, and 73% of kinematically valid trajectories are dynamically infeasible: they satisfy IK, collision, and planning constraints yet jitter (Figure 7a), twist (Figure 7b), or destabilize under the deployment controller. Removing the filter (–Dyn. Filter) drops Gr|Det at near-identical Det. (90.0 vs. 92.5%); Dex1B, which also lacks it, drops further (56.8%); controller-in-the-loop verification is essential for training-data quality. The consequence is sharpest at scale (Figure 4): trained on the same scenes without the filter, the policy is merely weaker inside the training range (64 vs. 87/80% on YCB/RoboCasa) but collapses to 3–6% below it—the filter is what turns in-distribution competence into extrapolation (§ 5.2).

Figure 4. Large-scale simulation evaluation (18,800 paired trials; Task 3 protocol in § 4.1). (a) SR vs. table height on YCB (left) and RoboCasa (right). Training tables span 0.40–0.90 m (dashed line); the shaded band is extrapolation below the training range. (b) SR (%) aggregated over the extrapolation and in-range bands (95% Wilson CIs for the Full rows are within 4 pts, e.g. YCB extrapolation 81.1 [77.7, 84.1]). The full system loses only 6–8 pts below the training floor; removing any single component collapses extrapolation to 0–6% (–Align to 40–46%).
Extrapolation beyond the training height range
At scale in simulation (Task 3, Figure 4, ∼18.8K paired trials), the full system holds 87.4/79.8% (YCB/RoboCasa) within the trained 0.40–0.90 m range and 81.1/71.4% at 0.10–0.35 m, below the training floor and reached from deep squats at shin height. It loses only 6–8 pts under extrapolation, while the other variants collapse there to 0–6% and –Align drops to 40–46%. Trained on large-scale, kinodynamically filtered data, the affordance policy learns rich local geometric features under a strong kinodynamic prior: its proposals remain object-relative and controller-executable even when the body configuration leaves the training distribution. Below the training floor (0.10–0.35 m), whole-body control and dynamics filtering become indispensable (removing either collapses success to near zero), and alignment’s contribution roughly doubles (13–15 → 25–41 pts), as maintaining stable, feasible whole-body execution becomes the dominant factor (§ 5.2.1). Object difficulty and the spatial envelope are analyzed in App. § 12.§ 6.
The full system’s profile carries over to hardware. Across five real heights (Task 2, Figure 3b, 150 trials), ours holds 73–97% with near-identical ID/OOD (85.3 vs. 84.0%), comparable to the simulated 80–93% (YCB) at the same heights; Dex1B and –Dyn. Filter degrade at the extremes, qualitatively consistent with the simulation results. ACT and ViA, given demonstrations at two heights, collapse at the unseen ones, as expected: teleoperated policies are bounded by their demonstration coverage, and collecting demonstrations across the full workspace is prohibitively expensive—precisely where sim-to-real training on large-scale synthetic data shines.
| Method | YCB Extrap. 0.10–0.35 m | YCB In-range 0.40–0.90 m | RoboCasa Extrap. 0.10–0.35 m | RoboCasa In-range 0.40–0.90 m |
| Ours (Full) | 81.1 | 87.4 | 71.4 | 79.8 |
| –Align | 40.1 | 72.4 | 46.0 | 67.2 |
| –Dyn. Filter | 2.8 | 64.4 | 5.7 | 63.6 |
| Dex1B + WB-IK | 0.7 | 47.7 | 0.4 | 39.2 |
| –Compensator | 2.1 | 43.6 | 1.9 | 44.0 |
| –Whole-Body | 0.0 | 6.1 | 0.0 | 11.4 |
Table 4.
Each component has a distinct failure signature
The per-height curves (Figure 4a) resolve where each component matters, which a single ablation number cannot. –Compensator rises with height, from 0% below 0.40 m to 59–81% at 0.70–0.90 m: pelvis drift grows with squat depth, so uncompensated targets miss most exactly where posture deviates most from nominal; the curve is a direct measurement of drift versus depth. –Dyn. Filter shows a cliff at the training floor (0% at 0.20 m, 60% from 0.45 m): unfiltered data yields grasps executable only in nominal postures. –Whole-Body is near zero below 0.65 m and peaks at 21–27%: this is the arm-only reach envelope, and it quantifies how much of the workspace exists only through the torso and legs. –Align costs a uniform 13–15 pts inside the training range but 25–41 pts below it: candidate selection matters most where feasible proposals are scarce. Dex1B is 0% at 0.30 m and falls off again above 0.80 m (34–23% on YCB): fixed-upper-body generation contains neither low postures nor extended high reaches, a coverage limit of the generator. Coverage is not the whole gap: even in-range it stays 40 pts below Full (47.7/39.2 vs. 87.4/79.8%), the compound cost of generating without whole-body–hand co-optimization or dynamic-feasibility filtering. Together these signatures explain the Task 1 stage-wise ablations (§ 4.2) mechanistically: each module removes a different failure regime, and only their union covers the workspace.

Figure 5. Whole-body spatial coverage. Adapting its entire body, our policy grasps in simulation across a continuous table-height range (0.10–0.90 m), from deep squats for low targets to full extension for high ones. Training tables spanned 0.40–0.90 m; the leftmost frames lie below the training range.
Active perception resolves the discovery bottleneck
Is the bottleneck finding the object or grasping it? –Perceive cuts Det. by 35 pts (92.5→57.5%) while Gr|Det stays high (87.0 vs. 94.6%): grasping is reliable given a good view; discovery is the bottleneck. Learned alternatives confirm the gap (ViA 70.0%, ACT+Active Head 57.5% Det. vs. our 92.5%). Our systematic viewpoint search takes 10–15 updates (∼8 s), a long search that end-to-end policies would have to learn jointly with manipulation; this motivates the modular separation of discovery from execution.
Synthetic data scaling drives generalization
In a dedicated real-world scaling study (App. § 12.§ 4), SR rises from 9.2% (5k trajectories) to ∼89% at 400k and plateaus thereafter (88.3% at 600k), with OOD benefiting most (5.0→90.0%) and the relative OOD degradation narrowing from 62% to under 2% above 300k; data diversity progressively closes the generalization gap to novel objects. Scale and filtering are complementary: scale closes the object gap, while kinodynamic filtering is what enables extrapolation beyond the training range (§ 5.2).
Conclusion and Limitations
We presented a hierarchical Perceive–Align–Grasp framework for humanoid whole-body affordance-aware grasping with active perception. By coordinating belief-driven viewpoint regulation, whole-body alignment, and a kinodynamically filtered affordance model, our system reaches 87.5% real-world success with zero real demonstrations, outperforming end-to-end imitation and modular grasp-planning baselines by large margins. Three findings anchor the design: 73% of kinematically valid grasps are dynamically infeasible under the deployment controller, so controller-in-the-loop data curation is essential for floating-base manipulation; perception is the dominant bottleneck under egocentric occlusion (without active perception, detection falls to 57.5% while Gr|Det remains 87%); and at scale in simulation the policy extrapolates to table heights well below its training range, evidence that kinodynamically filtered affordances are grounded in object geometry and whole-body feasibility rather than memorized configurations.
Limitations. Our system is predominantly open-loop at the high-level decision stage, prioritizing interpretability and stability over reactive replanning. The remaining real-world failures (∼12.5%) stem mainly from open-vocabulary detection, where DINO + SAM mis-identify targets under severe occlusion or on reflective surfaces, and from occasional collisions at workspace extremes near joint limits. Deployment currently uses double support without stepping, so targets beyond arm-plus-lean reach require repositioning; the controller interface already exposes velocity commands, and extending alignment to stepping is a natural next step. Synthetic-data generation is also costly (∼700 GPU-hours per 600k trajectories): fully parallelizable and amortized across deployments, but re-run when the controller or policy architecture changes. Future work may close the loop across perception and execution, co-train the stages end-to-end, and adapt data synthesis to the deployed controller.
Acknowledgments
This work was supported, in part, by NSF CAREER Award IIS-2240014 and NSF CCF-2112665 (TILOS).
Hierarchical Whole-Body Loco-Manipulation Controller
Frames and Notation
We use the standard rigid-body transform notation , which maps a 3D point expressed in frame into frame . The corresponding rotation and translation are and .
Key frames..
: the instantaneous support frame used by the upper-body controller. In double support, its origin is defined at the mid-point of the two contact feet and its yaw is aligned with the average heading of the feet. We set roll/pitch to zero to maintain a gravity-aligned frame and to avoid injecting foot tilt noise into the upper-body task frame.
: the kinematic reference frame of the full-body URDF used for forward kinematics (FK) from joint encoders.
: pelvis frame, treated as the floating base in the reduced upper-body IK model.
: left/right foot frames used as contacts.
: left/right end-effector frames used by the manipulation tasks.
: head camera frame used for active perception. We treat head gaze as an additional task-space target in the same support frame , i.e., (or equivalently a desired head orientation / optical-axis direction), and include in the task index set when head tracking is enabled.
Robot configuration.. We split the robot state into a lower-body part and an upper-body part:
where contains leg joints (and any joints actuated by the low-level locomotion controller), and contains waist/torso joints, arms, and optionally head joints (depending on the IK model).
Hierarchical Control Decomposition
Our control stack follows a standard outer/inner-loop organization: (i) an outer-loop whole-body IK module produces task-consistent upper-body commands together with a compact lower-body command interface, and (ii) an inner-loop RL controller (the decoupled whole-body controller of SONIC (GR00T) [38]) realizes the commanded pelvis motion while enforcing contact consistency and stability. A key design requirement is to maintain consistency between the pelvis pose assumed by the reduced IK model and the pelvis motion realized by the low-level controller.
High-level IK (upper-body whole-body IK).. At time , given desired task-space targets expressed in the support frame ,
the IK module solves
where is a contact-consistent reconstruction of the pelvis pose in the same support frame (Sec. A.3). This term closes the inter-layer loop by aligning the IK solver’s floating-base state with the physically realized pelvis motion.
Low-level RL controller (lower-body tracking). The low-level controller executes the command and outputs lower-body joint targets:
where is the proprioceptive observation (joint encoders, IMU, contact flags, etc.).
Lower-body command interface. In our standing manipulation experiments (double support, no stepping), we instantiate the general interface of § 3.1 as a low-dimensional pelvis command, with planar velocities set to zero and yaw commanded as a heading rather than a rate, to reduce inter-layer coupling and improve robustness under rate mismatch and model error:
The same hierarchical formulation directly supports loco-manipulation by extending to a more general locomotion interface (e.g., planar velocity and yaw-rate commands) while keeping Eqs. (4)–(5) unchanged.
Commanded output. The final motor command is the concatenation of the lower- and upper-body references:
In our deployed system, the low-level controller and the upper-body IK run synchronously at the same control rate (50 Hz), and the robot manager dispatches the fused command at each cycle.
Contact-Consistent Kinematic State Reconstruction
A central challenge in hierarchical control is that the pelvis pose assumed by the reduced IK model can deviate from the pelvis pose realized by the low-level controller due to tracking error, compliance, and modeling mismatch. To close this gap, we compute a contact-consistent reconstruction of the pelvis pose in the support frame using joint encoders under double-foot contact. Importantly, we use an instantaneous support frame definition (no integration): all quantities are expressed relative to the current mid-feet frame computed from FK.
Foot/pelvis FK in the URDF reference frame. Given measured joint positions , we compute the forward kinematics in the full-body URDF:
Instantaneous mid-feet support frame. We define from the two contact feet:
This matches the controller’s task frame: centered between the feet and aligned with the support yaw, while remaining gravity-aligned.
Pelvis pose in the support frame.. The reconstructed pelvis pose used by IK is
Remark (estimator modularity).. While we use a purely kinematic, contact-consistent reconstruction in double support for simplicity and low latency, the proposed interface only requires an estimate of the pelvis pose in the task frame, . In general, can be provided by any state estimator that is consistent with contact constraints, e.g., IMU–encoder fusion, visual–inertial odometry with contact updates, or factor-graph filtering, without changing the downstream compensation in Sec. A.4.
Closed-Loop Pelvis Consistency for Upper-Body IK
Let denote the pelvis pose assumed by the IK model (i.e., the floating-base state inside the reduced IK solver), and let be the contact-consistent reconstruction from Eq. (13). We compute the mismatch transform
Compensating task targets (equivalently, updating the tracking frame).. For any desired end-effector target expressed in , we compensate by the inverse mismatch:
and we pass to the IK solver. This can be interpreted either as correcting task targets or as updating an internal IK tracking frame; both are equivalent up to where the correction is applied.
Filtering on $SE(3).. To reduce jitter from encoder noise and small contact deformation, we apply a low-pass filter on (in the Lie algebra) before compensation:
Synthetic Data Generation: Kinodynamically Validated Trajectories
Our goal is to generate large-scale grasp demonstrations that are not only kinematically valid (reachable and collision-free), but also dynamically executable under the same hierarchical whole-body control stack used at deployment. This controller-in-the-loop filtering is key to learning kinodynamic-aware affordances for floating-base humanoids, where many geometrically plausible grasps fail due to stability and tracking limitations.
Domain Randomization
Object assets.. We draw object meshes from Objaverse [76] and YCB [89]; OOD evaluation objects are held out from the training set. To promote shape- and scale-level generalization, each object is isotropically re-scaled by a random factor within a bounded interval.
Scene parameterization.. Each scene contains a single axis-aligned box table and one target object:
Table dimensions are sampled uniformly within predefined bounds. The table-top height is randomized relative to the workspace origin to induce diverse reach regimes:
Tabletop heights span 0.40–0.90 m, the training range referred to in § 4. The object is placed uniformly over the table surface with an edge margin. Its vertical position is set such that the lowest point of the scaled mesh rests on the tabletop, its mass is sampled as , and its yaw is randomized:
World-frame perturbations.. To avoid a single canonical base-to-object transform, we additionally perturb the table center in the world frame using a run-specific random seed, and apply the corresponding rigid transform to the object pose. This produces distinct scene files across parallel collection runs while preserving valid on-table placement.
Grasp Optimization and Motion Planning
Given a randomized scene , we generate grasp demonstrations in batches. For each scene, we sample up to grasp candidates and retain only those that are (i) kinematically feasible (reachable and collision-free) and (ii) dynamically executable under simulation evaluation (Sec. B.3).
Energy-Based Grasp Optimization
We optimize grasp configurations in joint space. Let denote the robot configuration, comprising the pelvis pose, waist, arm, and hand joints, so that the hand and the posture carrying it are optimized jointly, and let be the fingertip contact points computed by forward kinematics. We minimize an energy function
Contact distance.. We encourage designated fingertip points to lie near the object surface via the object signed distance field (SDF):
Scene collision.. We penalize penetration of robot collision spheres into the scene SDF:
where is the center of collision sphere and is a safety padding.
Self-collision.. To avoid inter-finger collisions, we penalize close pairs:
Joint limits. We apply a soft penalty for joint limit violation:
Reachability and balance. The joint limits include the pelvis range the locomotion controller can realize, so every optimized posture can be commanded. Balance is not an energy term: a static criterion does not predict the closed-loop RL policy under a reaching load. Dynamic feasibility is verified downstream (Sec. B.3).
Optimization. We solve Eq. (21) with a simulated-annealing-style optimizer that combines autodiff gradients with stochastic acceptance [85]. At iteration , a candidate is proposed by an adaptive step,
and accepted with probability
The step size and temperature follow exponential decay schedules (Table 1).
| Stage | Parameter | Symbol | Value |
| Grasp Optimization (Simulated Annealing) | |||
| Iterations | Number of iterations | 5,000 | |
| Batching | Grasp samples per scene | 8,192 | |
| Contacts | Fingertip contact points | 4 | |
| Energy | Contact-distance weight | 100 | |
| Energy | Scene-penetration weight | 100 | |
| Energy | Self-penetration weight | 50 | |
| Energy | Joint-limit weight | 1 | |
| Steps | Initial step size | 0.005 | |
| Steps | Step decay period | 50 | |
| Temp. | Initial temperature | 18 | |
| Temp. | Temperature decay rate | 0.95 | |
| Temp. | Annealing period | 30 | |
| Init. | Front approach angle range | ||
| Init. | Approach distance range | [0.02, 0.15] m | |
| Init. | Hand template (6-DoF) | (0.8, 0.05, 0.6, 0.6, 0.6, 0.6) | |
| Motion Planning (Pre/Grasp/Post + Smoothing) | |||
| Phases | Init→Pre-grasp steps | 150 | |
| Phases | Pre-grasp→Grasp steps | 80 | |
| Phases | Holding steps | 15 | |
| Phases | Grasp→Post-grasp steps | 50 | |
| Phases | Lift height | 0.05 m | |
| Collision | Safety padding | 0.025 m | |
| Heuristic | Retreat step factor (arm) | 0.001 | |
| Heuristic | Closure factor (hand) | 0.8 | |
| Smoothing | Arm smoothness weight | 100 | |
| Smoothing | Hand smoothness weight | 10 | |
| Batching | Trajectories per batch | 1,024 |
Table 1. Key hyperparameters for synthetic grasp generation (optimization and motion planning).
Motion Planning and Kinematic Filtering
For each optimized grasp configuration , we construct a multi-phase joint-space trajectory comprising: (i) retreat to a pre-grasp, (ii) approach to grasp, (iii) hold, (iv) post-grasp closure, and (v) lift. The pre-grasp and post-grasp configurations are generated by SDF-gradient updates that move the arm (retreat) or hand joints (closure) along . The lift keyframe is obtained by raising the end-effector by and solving IK; a keyframe is reachable if this IK converges within joint limits, so kinematic validity is a property of the whole approach, grasp, and lift motion rather than of a static grasp pose. Intermediate segments are initialized by interpolation and refined by trajectory smoothing.
Trajectory smoothing. We optimize the full trajectory by minimizing a collision penalty and a smoothness term:
where
and
Kinematic feasibility. We treat a sample as kinematically feasible only if it passes both the grasp optimization stage and the motion planning stage. For a configuration , define the scene penetration of collision sphere as
and the fingertip-to-object surface distance for contact point as
Grasp optimization succeeds if every keyframe is reachable and the optimized grasp configuration satisfies
Motion planning succeeds if the planned trajectory remains collision-free (with padding) along the entire execution:
A sample is kinematically valid iff
Only trajectories with are passed to the dynamics-aware feasibility filtering stage. Optimization and planning schedules are listed in Table 1.
Dynamics-Aware Feasibility Filtering
Why dynamics filtering? For floating-base humanoids, kinematic feasibility does not guarantee executability: trajectories can violate stability constraints, exceed controller tracking capability, or induce unstable whole-body transients. We therefore apply a controller-in-the-loop dynamics check to retain only trajectories that are feasible under the deployment control stack.
Massively parallel controller replay. We replay each kinematically feasible trajectory in ManiSkill3 [75], a GPU-parallel physics simulator built on SAPIEN [91], at the deployment control rate (50 Hz). Crucially, the same hierarchical whole-body control structure as real deployment is used during replay (whole-body IK producing upper-body targets and low-level locomotion control executing the commanded motion).
Dynamic feasibility criteria. A trajectory is marked if it satisfies:
Task success: the object is lifted by at least m and remains in hand.
Stable execution: no fall/termination, and base attitude remains bounded, .
Trackability: tracking error of commanded end-effector keyframes stays below a threshold, , with no abrupt spikes in commanded motion.
We treat as fixed thresholds chosen to match the tolerances of the deployment controller.
Final acceptance mask. We retain a sample only if it passes both kinematic and dynamics feasibility:
By construction, the resulting dataset is supported on actions that are executable under whole-body kinodynamic constraints, enabling the learned policy to internalize kinodynamic-aware affordances rather than purely geometric ones. Figure 3c profiles an example collection run (508M candidates to 18K accepted grasps); its survival rates are representative of every run, and the full training set aggregates all parallel runs with distinct seeds into 600k trajectories.
Rendering and View Selection
We render object-centric point cloud observations for each accepted sample using a GPU-parallel renderer (ManiSkill3 [75]). For each scene, we render camera views with small RPY perturbations around a calibrated nominal extrinsic, and extract an object-only segmented point cloud of points, randomly downsampled to for the policy.
View selection.. Among candidate views whose frustums contain the object 3D bounding box, we select the view maximizing the number of valid (nonzero) 3D points:
If no view passes the visibility check, the scene is discarded to avoid degenerate observations. At deployment, this offline oracle view selection is replaced by the online active perception module in the main paper.
Policy Architecture
Overview.. We learn a conditional generative grasp policy that maps a segmented object point cloud to a distribution over multi-stage end-effector (EE) actions. Let be a global shape feature. Our policy is a conditional VAE (CVAE) with latent , defining
where is a -stage grasp action (pre-grasp, grasp, post-grasp). All architectural and training hyperparameters are summarized in Sec. D.
Action parameterization.. Each stage action consists of EE translation, EE orientation as a unit quaternion, and hand joint targets:
Vision Encoder: PointNet
We encode using a PointNet-style [87] encoder:
where is a shared-point MLP (implemented by stacked Conv1D+ReLU blocks), is channel-wise max-pooling over points, and projects to dimension . We omit BatchNorm in the encoder to avoid train/test mismatch under small-batch deployment.
Conditional VAE (CVAE)
Let be the flattened action. During training, the encoder models an approximate posterior
where is clamped to a fixed range (Sec. D) for numerical stability. The decoder predicts the action:
Training Objective and Inference
Training Objective and Inference
We minimize the negative evidence lower bound (ELBO):
We use an reconstruction loss over translation, quaternion, and hand joints across stages. Quaternions are normalized and sign-canonicalized to resolve :
Inference. At test time, we discard the encoder and draw latents for the alignment stage (§ 3.4), decoding each as . The predicted EE targets are executed by the downstream whole-body stack (IK, collision-aware planning, and controller) as in the main paper.
Hyperparameter Summary
| Module | Parameter | Symbol | Value |
| Policy (PointNet + CVAE) | |||
| Input | Policy input point cloud size | 2,048 | |
| Action | Stages (pre/grasp/post) | 3 | |
| Action | Action dimension | 13 | |
| Action | Hand DoF in action | 6 | |
| Encoder | PointNet feature dimension | 512 | |
| Latent | CVAE latent dimension | 256 | |
| MLP | Encoder/decoder hidden dims | — | [256, 256, 256] |
| MLP | Activation / normalization | — | SiLU / LayerNorm |
| Stability | log clamp range | ||
| Loss | KL weight | 0.1 | |
| Optim. | Optimizer | — | AdamW |
| Optim. | Learning rate | ||
| Optim. | Weight decay | ||
| Batch | Batch size (scenes) | 128 | |
| Sampling | Actions per scene | 12 | |
| Schedule | LR decay step (iter) | 66,667 | |
| Schedule | LR decay factor | 0.1 | |
| Training | Total iterations | 100,000 | |
| Split | Train/test split (scenes) | — | 97% / 3% |
| Eval. | Evaluation frequency | — | every 10,000 iters |
| Simulation Evaluation | |||
| Engine | Physics simulator | — | ManiSkill3 (SAPIEN) |
| Parallelism | Parallel environments | 512 | |
| Control | Control frequency | 50 Hz | |
| Object | Object mass | kg | |
| Metric | Lift threshold | 0.05 m | |
| Contact | Friction (dyn/static) | — | 2.0 / 2.0 |
| Contact | Restitution | — | 0.0 |
| Rendering / View Selection | |||
| Views | Views per scene | 16 | |
| Points | Rendered point cloud size (before downsample) | 4,096 | |
| Points | Policy input points (after downsample) | 2,048 | |
| Jitter | Roll jitter | ||
| Jitter | Pitch jitter | ||
| Jitter | Yaw jitter | ||
| Seg. | Segment extraction | — | object-only |
Table 2. Hyperparameters for policy learning, evaluation, and rendering.
Extended Experimental Analysis
This section collects supplementary figures and analyses referenced from § 4 of the main paper: real-world qualitative rollouts, visual failure modes, scene generalization, the training-data scaling curve, and the full results of the large-scale simulation evaluation (spatial success-rate fields and per-object success rates).
Real-World Qualitative Rollouts
Figure 6 shows representative successes across diverse objects, reaches, and scenes: tabletop manipulation (a), long-reach grasping (b), and zero-shot deployment in unseen environments at varied heights (c). Across all settings the policy coordinates active perception, whole-body motion, and dexterous manipulation with no scene-specific tuning.

Figure 6. Real-world rollouts, zero-shot from simulation. (a) Tabletop grasping of diverse objects; (b) long-reach grasping across a table; (c) deployment in varied unseen environments and heights (shelves, benches, tables).
Visual Failure Modes
Figure 7 visualizes the four representative failure modes that the Full pipeline eliminates relative to its ablations and to the gripper-based baseline. (a) and (b) are the kinematically valid but dynamically infeasible configurations that the dynamics filter rejects; (c) and (d) are the morphology-mismatched and collision-prone grasps produced when whole-body and hand DoFs are not jointly optimized.
Scene Generalization across Real-World Environments
Beyond the per-height generalization reported in the main Analysis (§ 5.2), Figure 8 shows the same policy operating across different real-world environments: kitchens, hallways with railings, multi-tier shelves, dining tables, and open lab benches at varying heights. The policy is trained purely on randomized tabletop scenes in simulation and uses only the egocentric RGB-D and active perception stack at deployment; no scene-specific fine-tuning, per-scene demonstrations, or environment annotation is required. Robustness to surface geometry and clutter follows from the diversity of the training distribution rather than any scene-aware module; lighting and background variation are absorbed by the open-vocabulary detection and segmentation front end, since the policy consumes only the segmented object point cloud.

Figure 8. Our full pipeline in fourteen real-world environments.
Synthetic Data Scaling
Figure 9 reports real-world SR as a function of the synthetic training-data volume. Each data scale is evaluated with oracle perception on 120 trials (60 ID, 60 OOD), for 1,080 real-world trials over nine data scales; this protocol is separate from Task 2’s 150-trial height sweep, so absolute values are not directly comparable to Figure 3b. SR rises monotonically from 9.2% at 5k trajectories to 89.2% at 400k and plateaus thereafter. OOD objects benefit most () while ID saturates earlier; the relative OOD degradation narrows from 62% at 5k to under 2% above 300k, indicating that data diversity progressively closes the generalization gap to novel objects.
Figure 9. Data scaling (real world, oracle perception, 5 ID + 5 OOD objects; 120 trials per data scale, a separate protocol from Task 2). SR scales from 9.2% at 5k trajectories to ∼89% at 400k and plateaus thereafter, with OOD benefiting most.
Computational cost. Offline generation: GPU-hours for 600k trajectories (fully parallelizable); CVAE training: h on a single GPU. At deployment (RTX 5090): perception (DINO + SAM) 148 ms, CVAE inference 0.4 ms, motion planning ms, WB-IK ms per batched solve over all candidates.
End-to-end grasp attempts take 15–40 s, dominated by physical robot motion.
Large-Scale Simulation Evaluation
This section reports the full results of the Task 3 simulation evaluation (protocol in § 4.1; band aggregates and SR-vs-height curves in Figure 4): the spatial success-rate fields (side and top view) and per-object success rates.
Protocol details.. The evaluation runs in ManiSkill3 [75] with the deployment whole-body controller in the loop. Each condition sweeps 17 table heights (0.10–0.90 m in 5 cm steps) with randomized planar object placement ( m, m) and randomized yaw, over the YCB (12 objects) and RoboCasa [90] (11 objects) catalogs. Scene sets are identical across the six variants, so all deltas are paired. Training tables span 0.40–0.90 m; heights 0.10–0.35 m evaluate extrapolation below the training range.
Success criterion.. A trial succeeds only if it is held and upright: the object is lifted m and remains lifted after a 2 s hold with m of in-hand slip, while the robot stays upright (pelvis height m, base tilt , and foot drift m). Scoring is a pure function of the recorded telemetry, with no manual judgment. This criterion is stricter than the real-world one (lift cm and stably held), since it additionally penalizes post-grasp slip and any loss of balance during the hold; simulation success rates are therefore conservative relative to the hardware tasks.
Spatial envelope and object difficulty.. The success field over height × reach (Figure 10) is dense throughout the 0.10–0.90 × 0.25–0.55 m envelope and thins mainly along the far-reach column ( m), most severely when far reach is combined with deep squat, where extension and balance compete. Across objects (Figure 12), the broader RoboCasa shape distribution costs ∼8 pts in-range (87.4 → 79.8%) and widens the per-object spread (51.5–93.4% vs. 67.7–94.1% for YCB); the hardest objects are thin or elongated (whisk, chicken drumstick, dish brush) and small near-spherical fruit (peach, apple), which demand centimeter-precise closure. The two degradations are close to additive: YCB in-range (87.4%) to RoboCasa extrapolation (71.4%) costs 16 pts, within ∼2 pts of the sum of the single-axis costs (6.3 + 7.6), so there is no catastrophic interaction.
Figure 10. Side-view success-rate field (Full pipeline): table height vs. forward reach , per catalog. The dashed line marks the 0.40 m training floor. Success stays high across the whole 0.10–0.90 × 0.25–0.55 m envelope and degrades only at the far-reach column and the deepest squat row; per-cell varies with random placement (8–42), so the field should be read as a whole rather than cell-by-cell.
Figure 12. Per-object success rate (Full pipeline, trials/object, 95% Wilson CIs). The RoboCasa catalog’s broader shape distribution widens the outcome spread (51.5–93.4% vs. 67.7–94.1% for YCB); the hardest objects are thin and elongated (whisk, chicken drumstick, dish brush), whose narrow graspable surfaces demand centimeter-precise hand placement.
Figure 11. Top-view success-rate fields (plan view; robot at the bottom, grasping-hand side to the right): Full vs. −Dyn. Filter vs. −Compensator on both catalogs, aggregated over all heights. The Full envelope is broad and dense; removing the dynamics filter keeps coverage but at uniformly lower success density; removing the compensator degrades the whole field, worst at far cells.
References
- [1]Q. Ben, F. Jia, J. Zeng, J. Dong, D. Lin, and J. Pang. HOMIE: Humanoid loco-manipulation with isomorphic exoskeleton cockpit, 2025. URL https://arxiv.org/abs/2502.13013.
- [2]Y. Ze, S. Zhao, W. Wang, A. Kanazawa, R. Duan, P. Abbeel, G. Shi, J. Wu, and C. K. Liu. TWIST2: Scalable, portable, and holistic humanoid data collection system, 2025. URL https://arxiv.org/abs/2511.02832.
- [3]T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to-humanoid real-time whole-body teleoperation. arXiv preprint arXiv:2403.04436, 2024.arXiv
- [4]Y. Li, Y. Lin, J. Cui, T. Liu, W. Liang, Y. Zhu, and S. Huang. CLONE: Closed-loop whole-body humanoid teleoperation for long-horizon tasks, 2025. URL https://arxiv.org/abs/2506.08931.
- [5]T. Z. Zhao, V. Kumar, S. Levine, and C. Finn. Learning fine-grained bimanual manipulation with low-cost hardware. RSS, 2023.arxiv.org/abs/2304.13705
- [7]T. He, Z. Wang, H. Xue, Q. Ben, Z. Luo, W. Xiao, Y. Yuan, X. Da, F. Castañeda, S. Sastry, C. Liu, G. Shi, L. Fan, and Y. Zhu. VIRAL: Visual sim-to-real at scale for humanoid loco-manipulation. arXiv preprint arXiv:2511.15200, 2025.arXiv
- [8]H. Xue, T. He, Z. Wang, Q. Ben, W. Xiao, Z. Luo, X. Da, F. Castañeda, G. Shi, S. Sastry, L. J. Fan, and Y. Zhu. Opening the sim-to-real door for humanoid pixel-to-action policy transfer. arXiv preprint arXiv:2512.01061, 2025. URL https://arxiv.org/abs/2512.01061.
- [9]T. Lin, K. Sachdev, L. Fan, J. Malik, and Y. Zhu. Sim-to-real reinforcement learning for vision-based dexterous manipulation on humanoids. arXiv preprint arXiv:2502.20396, 2025.arXiv
- [10]S. Wei, H. Jing, B. Li, Z. Zhao, J. Mao, Z. Ni, S. He, J. Liu, X. Liu, K. Kang, S. Zang, W. Yuan, M. Pavone, D. Huang, and Y. Wang. ψ₀: An open foundation model towards universal humanoid loco-manipulation, 2026. URL https://arxiv.org/abs/2603.12263.
- [13]R. Wang, J. Zhang, J. Chen, Y. Xu, P. Li, T. Liu, and H. Wang. Dexgraspnet: A large-scale robotic dexterous grasp dataset for general objects based on simulation. In ICRA, 2023.arxiv.org/abs/2210.02697
- [14]J. Zhang, H. Liu, D. Li, X. Yu, H. Geng, Y. Ding, J. Chen, and H. Wang. DexGraspNet 2.0: Learning generative dexterous grasping in large-scale synthetic cluttered scenes, 2024. URL https://arxiv.org/abs/2410.23004.
- [15]Xu, Y., Wan, W., Zhang, J., Liu, H., Shan, Z., Shen, H., Wang, R., Geng, H., Weng, Y., Chen, J., & al., . et . (2023). Unidexgrasp: Universal robotic dexterous grasping via learning diverse proposal generation and goal-conditioned policy. CVPR.
- [16]S. Deng, M. Yan, S. Wei, H. Ma, Y. Yang, J. Chen, Z. Zhang, T. Yang, X. Zhang, W. Zhang, H. Cui, Z. Zhang, and H. Wang. GraspVLA: a grasping foundation model pre-trained on billion-scale synthetic action data, 2025. URL https://arxiv.org/abs/2505.03233.
- [17]R. Dong, Z. Li, A. Gupta, X. He, and S. Gupta. HERO: Learning humanoid end-effector control for visual whole-body open-vocabulary object grasping, 2026. URL https://arxiv.org/abs/2602.16705.
- [18]R. Bajcsy. Active perception. Proceedings of the IEEE, 76(8):966–1005, 1988. doi: 10.1109/5.5968.DOI
- [19]J. A. Placed, J. Strader, H. Carrillo, N. Atanasov, V. Indelman, L. Carlone, and J. A. Castellanos. A survey on active simultaneous localization and mapping: State of the art and new frontiers, 2023. URL https://arxiv.org/abs/2207.00254.
- [20]R. Jangir, N. Hansen, S. Ghosal, M. Jain, and X. Wang. Look closer: Bridging egocentric and third-person views with transformers for robotic manipulation. IEEE Robotics and Automation Letters, 7(2):2146–2153, 2022. doi: 10.1109/LRA.2022.3144512.DOI
- [21]E. S. Hu, J. Wang, X. Yuan, F. Luo, M. Li, G. Lambrechts, O. Rybkin, and D. Jayaraman. Real-world reinforcement learning of active perception behaviors, 2025. URL https://arxiv.org/abs/2512.01188.
- [22]Luo, Z., Tessler, C., Lin, T., Yuan, Y., He, T., Xiao, W., Guo, Y., Chechik, G., Kitani, K., Fan, L., & Zhu, Y. (2025). Emergent active perception and dexterity of simulated humanoids from visual reinforcement learning. In arXiv preprint arXiv:2505.12278. https://arxiv.org/abs/2505.12278
- [23]H. Qi, Y.-J. Wang, T. Lin, Y. Brent, Y. Ma, K. Sreenath, and J. Malik. Coordinated humanoid manipulation with choice policies. arXiv:2512.25072, 2025.arXiv
- [24]D. Kim, M. Srouji, C. Chen, and J. Zhang. ARMOR: Egocentric perception for humanoid robot collision avoidance and motion planning, 2024. URL https://arxiv.org/abs/2412.00396.
- [25]M. Liu, Z. Chen, X. Cheng, Y. Ji, R. Qiu, R. Yang, and X. Wang. Visual whole-body control for legged loco-manipulation. The 8th Conference on Robot Learning, 2024.arxiv.org/abs/2403.16967
- [26]T. Yin, Z. Mei, T. Sun, L. Zha, E. Zhou, J. Bao, M. Yamane, O. Shorinwa, and A. Majumdar. WoMAP: World models for embodied open-vocabulary object localization, 2025. URL https://arxiv.org/abs/2506.01600.
- [27]X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-body control for humanoid robots. In Robotics: Science and Systems (RSS), 2024. URL https://arxiv.org/abs/2402.16796.
- [28]C. Lu, X. Cheng, J. Li, S. Yang, M. Ji, C. Yuan, G. Yang, S. Yi, and X. Wang. Mobile-TeleVision: Predictive motion priors for humanoid whole-body control. In IEEE International Conference on Robotics and Automation (ICRA), 2025. URL https://arxiv.org/abs/2412.07773.
- [29]Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu. BeyondMimic: From motion tracking to versatile humanoid control via guided diffusion, 2025. URL https://arxiv.org/abs/2508.08241.
- [30]L. Yang, X. Huang, Z. Wu, A. Kanazawa, P. Abbeel, C. Sferrazza, C. K. Liu, R. Duan, and G. Shi. OmniRetarget: Interaction-preserving data generation for humanoid whole-body loco-manipulation and scene interaction, 2025. URL https://arxiv.org/abs/2509.26633.
- [31]Y. Zhang, Y. Yuan, P. Gurunath, I. Gupta, S. Omidshafiei, A. akbar Agha-mohammadi, M. Vazquez-Chanlatte, L. Pedersen, T. He, and G. Shi. FALCON: Learning force-adaptive humanoid loco-manipulation, 2025. URL https://arxiv.org/abs/2505.06776.
- [32]W. Sun, L. Feng, B. Cao, Y. Liu, Y. Jin, and Z. Xie. ULC: A unified and fine-grained controller for humanoid loco-manipulation, 2025. URL https://arxiv.org/abs/2507.06905.
- [33]L. Wei, X. Peng, R.-Z. Qiu, T. Huang, X. Cheng, and X. Wang. HMC: Learning heterogeneous metacontrol for contact-rich loco-manipulation, 2025. URL https://arxiv.org/abs/2511.14756.
- [34]S. Chen, Z. Cao, Z. Luo, F. Castañeda, C. Li, T. Wang, Y. Yuan, L. Fan, C. K. Liu, and Y. Zhu. CHIP: Adaptive compliance for humanoid control through hindsight perturbation, 2025. URL https://arxiv.org/abs/2512.14689.
- [35]T. Zhang, X. Peng, F. Lin, X. Xiong, and Y. Lou. Whole-body compliance control for quadruped manipulator with actuation saturation of joint torque and ground friction. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11124–11131, 2024. doi: 10.1109/IROS58592.2024.10802754.DOI
- [36]Y. Niu, Z. Fang, B. Chen, S. Zhou, R. K. Senthilkumaran, H. Zhang, B. Chen, C. Qiu, H. E. Tseng, J. Francis, and D. Zhao. Learning versatile humanoid manipulation with touch dreaming, 2026. URL https://arxiv.org/abs/2604.13015.
- [37]M. Ji, X. Peng, F. Liu, J. Li, G. Yang, X. Cheng, and X. Wang. ExBody2: Advanced expressive humanoid whole-body control. arXiv preprint arXiv:2412.13196, 2024.arXiv
- [38]Z. Luo, Y. Yuan, T. Wang, C. Li, S. Chen, F. Castañeda, Z.-A. Cao, J. Li, D. Minor, Q. Ben, X. Da, R. Ding, C. Hogg, L. Song, E. Lim, E. Jeong, T. He, H. Xue, W. Xiao, Z. Wang, S. Yuen, J. Kautz, Y. Chang, U. Iqbal, L. Fan, and Y. Zhu. SONIC: Supersizing motion tracking for natural humanoid whole-body control. arXiv preprint arXiv:2511.07820, 2025.DOI
- [39]J. Li, X. Cheng, T. Huang, S. Yang, R.-Z. Qiu, and X. Wang. AMO: Adaptive motion optimization for hyper-dexterous humanoid whole-body control, 2025. URL https://arxiv.org/abs/2505.03738.
- [40]T. He, Z. Luo, X. He, W. Xiao, C. Zhang, W. Zhang, K. Kitani, C. Liu, and G. Shi. OmniH2O: Universal and dexterous human-to-humanoid whole-body teleoperation and learning. In Conference on Robot Learning (CoRL), 2024. URL https://arxiv.org/abs/2406.08858.
- [41]Z. Fu, Q. Zhao, Q. Wu, G. Wetzstein, and C. Finn. HumanPlus: Humanoid shadowing and imitation from humans. In Conference on Robot Learning (CoRL), 2024. URL https://arxiv.org/abs/2406.10454.
- [42]Ze, Y., Chen, Z., Araújo, J. P., Cao, Z., Peng, X. B., Wu, J., & Liu, C. K. (2025). TWIST: Teleoperated whole-body imitation system. https://arxiv.org/abs/2505.02833
- [43]X. Cheng, J. Li, S. Yang, G. Yang, and X. Wang. Open-TeleVision: Teleoperation with immersive active visual feedback. In Conference on Robot Learning (CoRL), 2024. URL https://arxiv.org/abs/2407.01512.
- [44]Y. Li, L. Ma, Y. Lin, Y. Du, M. Liu, K. Hu, J. Cui, Y. Zhu, W. Liang, B. Jia, and S. Huang. OmniClone: Engineering a robust, all-rounder whole-body humanoid teleoperation system, 2026. URL https://arxiv.org/abs/2603.14327.
- [45]T. Zhu, G. Cai, Z. Yang, G. Ren, H. Xie, Z. Wang, J. Wu, J. Wang, X. Yang, Y. Mu, and Y. Yan. CLOT: Closed-loop global motion tracking for whole-body humanoid teleoperation, 2026. URL https://arxiv.org/abs/2602.15060.
- [46]R. Nai, B. Zheng, J. Zhao, H. Zhu, S. Dai, Z. Chen, Y. Hu, Y. Hu, T. Zhang, C. Wen, and Y. Gao. Humanoid manipulation interface: Humanoid whole-body manipulation from robot-free demonstrations, 2026. URL https://arxiv.org/abs/2602.06643.
- [47]L. Heng, Y. Tang, J. Xu, H. Bao, D. Huang, and Y. Wang. HumDex: Humanoid dexterous manipulation made easy, 2026. URL https://arxiv.org/abs/2603.12260.
- [48]J. Li, Y. Zhu, Y. Xie, Z. Jiang, M. Seo, G. Pavlakos, and Y. Zhu. OKAMI: Teaching humanoid robots manipulation skills through single video imitation. In Conference on Robot Learning (CoRL), 2024. URL https://arxiv.org/abs/2410.11792.
- [49]R.-Z. Qiu, S. Yang, X. Cheng, C. Chawla, J. Li, T. He, G. Yan, D. J. Yoon, R. Hoque, L. Paulsen, G. Yang, J. Zhang, S. Yi, G. Shi, and X. Wang. Humanoid Policy ∼ human policy, 2025. URL https://arxiv.org/abs/2503.13441.
- [50]Ze, Y., Chen, Z., Wang, W., Chen, T., He, X., Yuan, Y., Peng, X. B., & Wu, J. (2025). Generalizable humanoid manipulation with 3D diffusion policies. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). https://arxiv.org/abs/2410.10803
- [51]G. Jiang, Y. Liang, J. Ye, J.-Y. Huang, C. Jing, R. Duan, P. Abbeel, X. Wang, and X. Zou. Cross-hand latent representation for vision-language-action models, 2026. URL https://arxiv.org/abs/2603.10158.
- [52]R.-Z. Qiu, Y. Song, X. Peng, S. A. Suryadevara, G. Yang, M. Liu, M. Ji, C. Jia, R. Yang, X. Zou, and X. Wang. WildLMa: Long horizon loco-manipulation in the wild. arXiv preprint arXiv:2411.15131, 2024.arxiv.org/abs/2411.15131
- [53]X. Xu, Y. Zhang, Y.-L. Li, L. Li, L. Han, and C. Lu. HumanVla: Towards vision-language directed object rearrangement by physical humanoid. Advances in Neural Information Processing Systems, 37:18633–18659, 2024.arxiv.org/abs/2406.19972
- [54]Z. Luo, J. Cao, S. Christen, A. Winkler, K. Kitani, and W. Xu. Omnigrasp: Grasping diverse objects with simulated humanoids. Advances in Neural Information Processing Systems, 37:2161–2184, 2024.arxiv.org/abs/2407.11385
- [55]W. Liu, Y. Wan, J. Wang, Y. Kuang, W. Cui, X. Shi, H. Li, D. Zhao, Z. Zhang, and H. Wang. FetchBot: Learning generalizable object fetching in cluttered scenes via zero-shot Sim2Real, 2025. URL https://arxiv.org/abs/2502.17894.
- [56]N. Sun, H. Shen, Y. Zhang, L. Feng, H. Wang, Y. Xiangli, Y. Jin, and P. Liu. DexWeave: Learning dexterous humanoid loco-manipulation from human demonstrations. arXiv, 2026.arxiv.org/abs/2609.34724
- [57]J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017. URL https://arxiv.org/abs/1703.06907.
- [58]X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE International Conference on Robotics and Automation (ICRA), page 3803–3810. IEEE, May 2018. doi: 10.1109/icra.2018.8460528. URL http://dx.doi.org/10.1109/ICRA.2018.8460528.
- [59]OpenAI, M. Andrychowicz, B. Baker, M. Chociej, R. Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation, 2019. URL https://arxiv.org/abs/1808.00177.
- [60]OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, J. Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving Rubik’s cube with a robot hand, 2019. URL https://arxiv.org/abs/1910.07113.
- [61]J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots, 2018. URL https://arxiv.org/abs/1804.10332.
- [62]A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid motor adaptation for legged robots, 2021. URL https://arxiv.org/abs/2107.04034.
- [63]Z. Xie, G. Berseth, P. Clary, J. Hurst, and M. van de Panne. Feedback control for Cassie with deep reinforcement learning, 2018. URL https://arxiv.org/abs/1803.05580.
- [64]L. Wei, J. Ma, Y. Hu, and R. Zhang. Ensuring force safety in vision-guided robotic manipulation via implicit tactile calibration, 2025. URL https://arxiv.org/abs/2412.10349.
- [65]Mahler, J., Liang, J., Niyaz, S., Laskey, M., Doan, R., Liu, X., Ojea, J. A., & Goldberg, K. (2017). Dex-Net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. https://arxiv.org/abs/1703.09312
- [66]Zhang, G., Xu, Q., Zhang, H., Ma, J., He, L., Bao, Y., Ping, Z., Yuan, Z., Lu, C., Yuan, C., & al., . et . (2026). UniDex: A robot foundation suite for universal dexterous hand control from egocentric human videos. In arXiv preprint arXiv:2603.22264. 2603.22264
- [67]Chao, Y.-W., Yang, W., Xiang, Y., Molchanov, P., Handa, A., Tremblay, J., Narang, Y. S., Van Wyk, K., Iqbal, U., Birchfield, S., & al., . et . (2021). DexYCB: A benchmark for capturing hand grasping of objects. CVPR.
- [68]J. Ye, L. Wei, G. Jiang, C. Jing, X. Zou, and X. Wang. From power to precision: Learning fine-grained dexterity for multi-fingered robotic hands, 2025. URL https://arxiv.org/abs/2511.13710.
- [69]Y. Wang, Z. Xian, F. Chen, T.-H. Wang, Y. Wang, K. Fragkiadaki, Z. Erickson, D. Held, and C. Gan. RoboGen: Towards unleashing infinite data for automated robot learning via generative simulation, 2024. URL https://arxiv.org/abs/2311.01455.
- [70]A. Mandlekar, S. Nasiriany, B. Wen, I. Akinola, Y. Narang, L. Fan, Y. Zhu, and D. Fox. MimicGen: A data generation system for scalable robot learning using human demonstrations, 2023. URL https://arxiv.org/abs/2310.17596.
- [71]Z. Xue, S. Deng, Z. Chen, Y. Wang, Z. Yuan, and H. Xu. DemoGen: Synthetic demonstration generation for data-efficient visuomotor policy learning, 2025. URL https://arxiv.org/abs/2502.16932.
- [72]L. Wang, Y. Ling, Z. Yuan, M. Shridhar, C. Bao, Y. Qin, B. Wang, H. Xu, and X. Wang. GenSim: Generating robotic simulation tasks via large language models, 2024. URL https://arxiv.org/abs/2310.01361.
- [73]Yu, T., Xiao, T., Stone, A., Tompson, J., Brohan, A., Wang, S., Singh, J., Tan, C., M, D., Peralta, J., Ichter, B., Hausman, K., & Xia, F. (2023). Scaling robot learning with semantically imagined experience. https://arxiv.org/abs/2302.11550
- [74]NVIDIA, Mittal, M., Roth, P., Tigue, J., Richard, A., Zhang, O., & al., . et . (2025). Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning. https://arxiv.org/abs/2511.04831
- [75]S. Tao, F. Xiang, A. Shukla, Y. Qin, X. Hinrichsen, X. Yuan, C. Bao, X. Lin, Y. Liu, T. kai Chan, Y. Gao, X. Li, T. Mu, N. Xiao, A. Gurha, V. N. Rajesh, Y. W. Choi, Y.-R. Chen, Z. Huang, R. Calandra, R. Chen, S. Luo, and H. Su. ManiSkill3: GPU parallelized robotics simulation and rendering for generalizable embodied AI, 2025. URL https://arxiv.org/abs/2410.00425.
- [76]M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3D objects, 2022. URL https://arxiv.org/abs/2212.08051.
- [77]R. Eidenberger and J. Scharinger. Active perception and scene modeling by planning with probabilistic 6D object poses. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1036–1043, 2010. doi: 10.1109/IROS.2010.5651927.DOI
- [78]S. Isler, R. Sabzevari, J. Delmerico, and D. Scaramuzza. An information gain formulation for active volumetric 3D reconstruction. In IEEE International Conference on Robotics and Automation (ICRA), pages 3477–3484, 2016. doi: 10.1109/ICRA.2016.7487527.DOI
- [79]J. Delmerico, S. Isler, R. Sabzevari, and D. Scaramuzza. A comparison of volumetric information gain metrics for active 3D object reconstruction. Autonomous Robots, 42(2):197–208, 2018. doi: 10.1007/s10514-017-9634-0.DOI
- [80]N. Dengler, S. Pan, V. Kalagaturu, R. Menon, M. Dawood, and M. Bennewitz. Viewpoint push planning for mapping of unknown confined spaces. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2023. URL https://arxiv.org/abs/2303.03126.
- [81]X. Zou, Y. Song, R.-Z. Qiu, X. Peng, J. Ye, S. Liu, and X. Wang. M3: 3D-spatial MultiModal memory, 2025. URL https://arxiv.org/abs/2503.16413.
- [82]M. Breyer, L. Ott, R. Siegwart, and J. J. Chung. Closed-loop next-best-view planning for target-driven grasping. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022. URL https://arxiv.org/abs/2207.10543.
- [83]S. Jauhri, S. Lueth, and G. Chalvatzaki. Active-perceptive motion generation for mobile manipulation. In IEEE International Conference on Robotics and Automation (ICRA), 2024. URL https://arxiv.org/abs/2310.00433.
- [84]P. Kaiser, C. Mandery, A. Boltres, and T. Asfour. Affordance-based multi-contact whole-body pose sequence planning for humanoid robots in unknown environments. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3114–3121, 2018. doi: 10.1109/ICRA.2018.8461087.DOI
- [85]J. Ye, X. Peng, C. Mc Gartoll, C. Jing, L. Wei, J. Chen, X. Zou, and X. Wang. RoboKit: Accelerating parallel differentiable optimization for robot learning. In Conference on Robot Learning (CoRL), 2026.arxiv.org/abs/2611.20185
- [86]S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, Q. Jiang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang. Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection. In European Conference on Computer Vision (ECCV), 2024. URL https://arxiv.org/abs/2303.05499.
- [87]C. R. Qi, H. Su, K. Mo, and L. J. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 652–660, 2017. URL https://arxiv.org/abs/1612.00593.
- [88]A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything. In IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026, 2023. URL https://arxiv.org/abs/2304.02643.
- [89]B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M. Dollar. Benchmarking in manipulation research: Using the Yale-CMU-Berkeley object and model set. IEEE Robotics & Automation Magazine, 22(3):36–52, 2015. doi: 10.1109/MRA.2015.2448951.DOI
- [90]S. Nasiriany, A. Maddukuri, L. Zhang, A. Parikh, A. Lo, A. Joshi, A. Mandlekar, and Y. Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. In Robotics: Science and Systems (RSS), 2024. URL https://arxiv.org/abs/2406.02523.
- [91]F. Xiang, Y. Qin, K. Mo, Y. Xia, H. Zhu, F. Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated part-based interactive environment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11097–11107, 2020.arxiv.org/abs/2003.08515




