WorksComputer Science
HINT-VLM: Humanoid Interactive Navigation over Rough Terrain with Embodiment-Grounded Vision-Language Model
Abstract
Humanoid robots have achieved significant progress in locomotion and manipulation, enabling traversal over rough terrain and richer interaction with the environment, such as moving obstacles. However, most traditional navigation methods assume non-interactive environments and focus only on traversal. While Navigation Among Movable Obstacles (NAMO) approaches relax this assumption, prior efforts have largely focused on flat environments and reactive interaction, rather than predicting object movability and planning ahead. These limitations become critical in outdoor rough-terrain scenarios, where robots must jointly reason about terrain traversability and object movability. In this work, we address NAMO for humanoid robots in cluttered outdoor rough terrain by proposing a hybrid planning framework that combines semantic priors from a Vision-Language Model (VLM) with embodied robot interaction data through a Gaussian process (GP) residual model. The GP corrects the VLM-derived movability prior to produce embodiment-grounded probabilistic movability estimates, which are then incorporated into hierarchical uncertainty-aware planning. The proposed approach allows humanoids to jointly leverage locomotion and manipulation for efficient and safe navigation, representing the first solution to NAMO in rough terrain through predictive movability reasoning and probabilistic VLM integration.
Introduction
Humanoid robots have drawn significant interest due to their ability to traverse diverse rough terrains and actively interact with the environment through whole-body manipulation using their hands or limbs [1]. In parallel with this growing interest, both locomotion and manipulation capabilities have seen substantial progress [2, 3, 4]. However, comparatively little attention has been given to navigation problems that require these two capabilities to be used jointly in cluttered rough environments.
Most existing navigation approaches treat obstacles as entities to avoid rather than objects that the robot can physically interact with to facilitate navigation [5], [6]. More recently, a separate line of work studies Navigation Among Movable Obstacles (NAMO), or Interactive Navigation, where robots proactively interact with and manipulate objects to reach a goal [7], [8]. Existing NAMO approaches [7] have primarily focused on wheeled or quadrupedal platforms, where interaction is often limited to simple forward pushing with the body
in relatively flat indoor environments. This setting presents a natural opportunity for humanoid robots, which combine bipedal mobility over rough terrain with object manipulation capability. While recent work has begun to explore wheeled humanoids with bimanual pushing, the range of object geometries remains limited and the evaluation is still restricted to flat terrain [9]. Consequently, handling a wide variety of objects within NAMO, particularly in rough terrain settings, remains largely unexplored.
Another limitation of existing NAMO research lies in its
This work was in part supported by VinMotion.
predictive planning capability. Most prior works [7, 9] adopt reactive strategies that focus on improving manipulation performance or detecting failures after the physical interaction has already been initiated, often resulting in trial-and-error object interactions. In contrast, relatively few studies [8, 10] address the problem of identifying movable obstacles at a distance and selecting interaction targets before physical contact, which is critical for efficient long-horizon planning.
The need for predictive planning becomes more pronounced in rough-terrain settings, where the robot must reason jointly about path length, terrain traversability, and object movability. In such scenarios, successful displacement depends not only on the object itself but also on the terrain conditions beneath, which influence the outcome of interaction. As a result, a humanoid robot must jointly plan rough-terrain traversal and object interactions based on its locomotion and manipulation capabilities to determine an optimal navigation path. This setting motivates the study of NAMO in outdoor, unstructured environments, which remains largely unexplored.
Fortunately, recent advances in Vision-Language Models (VLMs) offer a promising approach for inferring object movability before physical contact. Through large-scale pretraining, VLMs acquire strong language-conditioned semantic understanding and generalization capabilities that are difficult to achieve with conventional neural networks trained from scratch for language-conditioned navigation tasks [11, 12, 13, 14]. Motivated by these capabilities, we propose to leverage VLMs to infer object movability without requiring large-scale, language-annotated training data. This enables predictive planning by allowing the robot to determine which obstacles are preferable to manipulate before committing to physical interaction. However, directly treating VLM predictions as reliable estimates can be problematic because their generic semantic priors do not reflect the physical capabilities of a specific robot embodiment and controller. We therefore correct the VLM prior using a residual Gaussian process (GP) learned from robot interaction data, producing embodiment-grounded movability estimates. Movability is represented probabilistically and incorporated into uncertainty-aware planning, allowing the planner to explicitly account for the potential cost of failed object interactions.
In this work, we address humanoid interactive navigation on rough terrain (HINT) by integrating embodiment-grounded VLM-GP-based object movability reasoning with an uncertainty-aware planner. As illustrated in Figure 1, the proposed framework integrates vision-based object movability reasoning, object and terrain mapping, uncertainty-aware global planning, and interaction execution within a unified navigation system. The key contributions of this paper are summarized as follows:
Figure 1. Overview of HINT-VLM. The proposed framework uses an embodiment-grounded VLM-GP to predict object movability from visual observations, maps objects and reconstructs the underlying terrain, and incorporates the resulting information into an uncertainty-aware global planner. A finite-state machine switches among path tracking, interaction, and failure recovery.
NAMO in cluttered rough terrain with humanoids: To the best of our knowledge, this is the first work to address the NAMO problem in cluttered rough terrain for humanoids, jointly considering terrain traversability and object interaction, while enabling dynamic humanoid navigation with object manipulation.
Predictive object movability reasoning using an embodiment-grounded VLM-GP: We use a VLM-GP to estimate object movability as the probability of successful object interaction from visual observations, enabling the robot to reason about obstacle movability before physical interaction. The VLM prior is refined by a residual GP learned from robot interaction outcomes to produce robot-specific movability estimates.
Uncertainty-aware navigation planning with probability output from VLM: We propose an uncertainty-aware planning framework that leverages the probabilistic movability and explicitly accounts for both successful and failed interaction outcomes.
Related Works
Navigation Among Movable Obstacles
Predictive NAMO methods often rely on restrictive assumptions about object geometry or movability supervision. For example, the work in [15] considers only box-shaped obstacles, while the authors in [10] approximate objects using geometric primitives such as cylinders and cuboids and estimate movability through hand-designed geometric heuristics. More recently, the study of [8] predicts movability from DINO-based visual features but relies on game-derived data with limited object diversity and manually assigned scores that are not conditioned on robot or controller capabilities. Consequently, estimating the movability of diverse unseen objects from visual observations without privileged physical information or hand-designed heuristics remains underexplored. We address these limitations by combining VLM semantic priors with robot interaction outcomes rather than relying on manually assigned movability scores.
Rough Terrain Navigation for Humanoids
Many NAMO approaches focus on flat indoor environments. Extending NAMO to cluttered rough terrain presents an interesting opportunity for applications such as construction sites and disaster response [16], where humanoids can leverage both dynamic locomotion and manipulation capabilities. For such rough-terrain navigation, a large body of work has studied traversability estimation for legged robots. Early approaches [17, 18] rely on manually designed rules, while more recent methods [5] employ neural networks to learn richer traversability representations. However, these works primarily assume static environments and do not consider movable objects, which are common in cluttered rough terrain. This limitation makes the integration of NAMO and rough-terrain navigation a particularly compelling direction.
VLM-Based Navigation
Recently, VLMs have demonstrated strong language-conditioned semantic understanding for navigation tasks. Various forms of semantic guidance, including social norms [19], path preferences [12], and potential hazards [14], have been incorporated into VLM-based navigation frameworks. These approaches generally follow one of two paradigms: end-to-end Vision-Language Navigation (VLN), or modular approaches in which VLM outputs are integrated into a classical planner. While end-to-end VLN offers language steerability, approaches without explicit geometric representations can struggle to account for terrain traversability and object interactions in rough terrain [12]. Modular approaches retain classical planning while leveraging VLM semantic reasoning [14], [19], but typically treat VLM outputs as heuristic scores without grounding them in robot capabilities or incorporating them into probabilistic planning. We instead ground VLM movability predictions in robot interaction outcomes through a residual GP, yielding embodiment-grounded probabilistic estimates for uncertainty-aware planning.
Object Movability Estimation with Embodiment-Grounded VLM-GP
For each target object, our goal is to estimate its movability as the probability that the deployed robot and controller can successfully move the object. While a pretrained VLM provides a broad semantic prior for movability, this prior does not account for the capabilities of a specific robot. We therefore develop an embodiment-grounded VLM-GP, in which the VLM provides the semantic movability prior and a residual GP, learned from robot interaction outcomes, corrects the prior toward robot-specific movability.
Two-Stage Object Identification and Movability Inference
Inspired by the work of [14], we decompose the inference process into two stages. First, a pretrained VLM identifies potentially movable objects from the input image and outputs their semantic labels. Given each identified object label together with the image, the embodiment-grounded VLM-GP then estimates its movability as a probability of successful interaction. This probabilistic formulation is advantageous in two respects: it provides a natural VLM prior that can be residual-corrected using robot interaction data, and it enables uncertainty-aware reasoning in the downstream planner. These aspects are described in detail in § 3-B and § 5-A.
Embodiment-Grounded VLM-GP for Movability Estimate
The movability estimator takes the target object and its corresponding image as input and predicts the probability that the robot can sufficiently displace the object while maintaining stability. In principle, the mismatch between generic VLM movability priors and robot- and controller-specific capabilities could be addressed by fully adapting the model using diverse robot interaction data. However, large-scale real-world collection is impractical, and simulation still lacks sufficiently diverse, visually realistic, and physically faithful assets [20], particularly for objects with wheels or articulation. We therefore retain the broad semantic prior of the pretrained VLM and use the available robot interaction data to learn robot-specific corrections. Specifically, we use the VLM prediction as the prior mean and learn a residual correction with a GP. This formulation preserves the VLM prior in regimes weakly
covered by robot data while introducing robot- and controller-specific corrections in regions supported by observed interaction outcomes. Accordingly, we formulate the log-odds of a successful push as:
where denotes the logistic function and is the estimated probability of successful interaction. The VLM semantic prior is computed as the difference between the logits for “Yes” and “No” in response to a movability query on image . The residual correction is learned from robot interaction outcomes and conditioned on the object’s physical attributes , including mass, friction, geometry, and material class.
To learn the residual GP correction, we simulate the robot pushing a diverse set of objects in IsaacSim [21] using the same interaction controller employed during deployment. As shown in Figure 2, the controller maintains a forward-stretched arm configuration using the whole-body controller in [22], while adaptively adjusting the commanded center-of-mass (CoM) velocity to track a target pushing speed of 0.5 m/s. An interaction is labeled successful if the robot maintains at least 10% of the target pushing speed while displacing the object without becoming unstable, and unsuccessful otherwise. For each trial, we record the binary interaction outcome, the egocentric camera observation, the semantic name and the physical properties of the target object. We construct the simulation dataset by spawning approximately 400 furniture objects and everyday object assets derived from PhysX-3D [20] on various ground surfaces. To improve robustness to visual and geometric variations, we randomize camera intrinsics and extrinsics, object class names, and simulation backgrounds, and additionally apply image augmentation during training. This procedure yields approximately 20,000 interaction samples.
Figure 2. Overview of the proposed VLM-GP training and inference. The residual GP is trained on simulated robot interaction outcomes and ground-truth physical attributes, while perceived attributes are used at inference for embodiment-grounded movability estimation.
Using the collected interaction trials, we construct the GP training observations as follows. Trials of the same asset share mass, size, and material, so we group those whose friction falls in the same within-asset quartile into a cell . Each cell has representative attributes and successes out of trials. Mass, friction, and size are simulator ground truth. The simulator does not label material, so material class is the VLM’s majority reading over the asset’s renders. We then fit
where is a Matérn-5/2 kernel with one length scale per standardized attribute [23]. By placing the VLM prediction in the GP mean function, the GP learns only the residual correction needed to align the generic semantic prior with the observed robot interaction outcomes. We approximate the posterior with the Laplace approximation [23] and take its mean as . The kernel hyperparameters are estimated once by maximizing the marginal likelihood of a GP regression on the empirical log-odds of the cells and then kept fixed, which generalizes better to unseen object categories than maximizing the Laplace-approximated marginal likelihood of the Binomial model [23]. As the query moves away from regions supported by the robot interaction data, the residual correction diminishes and the estimate naturally reverts toward the VLM prior.
While the GP is fitted on true simulator attributes, at test time must be perceived from the image. For both the VLM prior and attribute estimation, we use the same pretrained Qwen3.5-9B NF4 VLM without providing the object name. Instead, we draw a red contour along the object mask obtained from Grounded-SAM 2 (GSAM) [24] to visually specify the target object while reducing bias from its semantic label. To account for uncertainty in the VLM-estimated physical attributes, we query the VLM times for the object’s mass range, friction coefficient, and material class. Object size is estimated geometrically rather than by the VLM: we lift the masked depth pixels into 3D and compute the dimensions of their bounding box. Each VLM reading is represented as a Gaussian in log space centered at its estimated value, with a standard deviation equal to twice the reported half-range width, such that the readings form a Gaussian mixture. From each reading, we draw samples of mass and friction, retain the corresponding material class, and combine them with the measured object size, to obtain attribute vectors . The final movability estimate is computed as
Thus, uncertainty in the perceived object attributes is marginalized rather than relying on a single VLM estimate.
Spatial Grounding of Objects and Its Underlying Terrain Reconstruction
Section III describes how the VLM-GP estimates the movability of an object from its image. To use this estimate for path planning, it must be associated with the spatial extent of the corresponding object in the bird’s-eye-view (BEV) map used by the planner. We reuse the GSAM object mask obtained during movability estimation to establish this spatial correspondence. As illustrated in Figure 3, the masked depth pixels are lifted into a 3D point cloud and projected onto the BEV map, yielding the object footprint for planning.

Figure 3. Spatial grounding and terrain reconstruction for a candidate object. GSAM uses the semantic label to obtain the object mask, which is applied to the aligned depth image. The masked region is then inpainted with LaMa, and both the object mask and reconstructed region are lifted into 3D to obtain the object footprint and reconstructed terrain for planning.
While such object grounding is generally sufficient for interactive navigation on flat terrain, rough-terrain settings introduce an additional requirement. Even if an object is movable, the terrain beneath it may still be unsafe to traverse. We therefore reconstruct both the terrain geometry and its traversability beneath each candidate object. Specifically, we remove the depth pixels within the object mask and use LaMa [25] to inpaint the occluded region, providing an estimate of the terrain that would be observed in the absence of the object. The reconstructed region is then lifted into 3D and integrated into the terrain map. We subsequently apply STATE-NAV [5] to estimate traversability, represented as the maximum speed at which the robot can safely traverse each region. If the reconstructed terrain is deemed untraversable at the target pushing speed defined in § 3-B, the planner excludes the corresponding passage even when the object itself is predicted to be movable.
Navigation System for NAMO
Uncertainty-Aware Path Planning with VLM Output as Probability
Given the estimated movability of each candidate object, the planner jointly considers travel distance, terrain traversability, and object interaction costs. Prior works incorporate VLM-derived semantic scores into classical planning [14, 19] and learned object movability scores into interactive navigation [8], typically through additive costs of the form , where is the path cost, is the object-related cost, and is a tunable weight. However, such formulations do not explicitly account for uncertainty in interaction outcomes or the downstream cost of failure. We instead evaluate the expected cost of interacting with object as
Here, is the cost of reaching the object, is the remaining cost after successful interaction, and is the cost of retreating and reaching the goal via an alternative route after failure. As illustrated in Figure 4, the success and failure costs are weighted by their corresponding probabilities, while non-interactive detours are evaluated directly by their path costs. The planner selects the option with minimum expected cost. For example, the planner may prefer a non-interactive detour even when an object has high predicted movability, if approaching the object incurs substantial cost or a failed interaction requires significant backtracking. All path-dependent costs are computed using terrain-aware traversal costs and A* planning [5], consistently accounting for traversability across successful interactions, failed interactions, and detours.

Figure 4. Illustration of uncertainty-aware path planning. The solid green path leads to the object, while the dotted and dashed green paths represent the routes following successful and failed interactions, respectively. The orange path denotes a non-interactive detour.
Finite State Machine and System Integration
When the global planner selects a route requiring object interaction, a finite state machine (FSM) switches the robot from normal path following to the pushing behavior introduced in § 3-B. As the robot approaches the target, it adopts a forward-reaching posture and slowly pushes the object along the planned obstacle-passing route. During interaction, VideoSAM [26] tracks the object and updates its position online. Once sufficient clearance is created, the FSM terminates the interaction and resumes path following toward the goal. For local path tracking, we employ the terrain-aware controller from [5].
Experimental Results
We organize our evaluation around five research questions (RQs). First, we evaluate the movability estimator to investigate (RQ1) how effective a pretrained VLM is for movability estimation of arbitrary objects, and how much residual GP correction using robot interaction data improves prediction and sim-to-real transfer. We then introduce the navigation benchmarks, experimental setup, and evaluation metrics used to investigate (RQ2) what new navigation capabilities the proposed framework enables, (RQ3) how residual correction using robot interaction data affects navigation performance, (RQ4) whether uncertainty-aware planning with probabilistic movability estimates improves NAMO performance on rough terrain, and (RQ5) whether the proposed navigation stack transfers to a physical humanoid in the real world.
VLM-GP Movability Estimation and Sim-to-Real Transfer
1) Movability Estimation Setup: To address RQ1, we first analyze the accuracy and sim-to-real transfer of the proposed
movability estimator. For the simulation data collected in Section III-B, we split the interaction samples into training and validation sets. To evaluate sim-to-real generalization, we additionally collect a real-world dataset following the same interaction procedure. We place diverse objects on multiple floor surfaces and evaluate their movability using the same pushing behavior as in simulation. In total, we physically test 73 objects, including five wheeled objects absent from the simulation training data. We additionally collect images of 10 clearly immovable objects, such as a public bench and a heavy refrigerator, without conducting physical interactions. We define three evaluation sets: Real-Tried, comprising the 68 physically tested non-wheeled objects; Real-OOD, comprising the five wheeled objects and 10 additional immovable objects; and Real-All, containing all 83 objects. Thus, Real-Tried and Real-OOD are disjoint subsets of Real-All.
We compare the proposed VLM-GP against three alternative movability estimators. The first is the pretrained Qwen3.5-9B NF4 VLM without residual correction. The second is a low-rank adaptation (LoRA) fine-tuned version of the same VLM, trained on the simulated interaction data using binary cross-entropy over the “Yes” and “No” outputs. The third is a DINO-based predictor inspired by [8], using DINOv3 [27] to extract visual features from the masked object region, followed by a multilayer perceptron (MLP) trained on the same simulated interaction data. We evaluate all methods on the simulation validation set and the real-world test set using the area under the receiver operating characteristic curve (AUROC), Brier score, and the resolution component of the Brier score decomposition. Higher AUROC indicates better discrimination, while lower Brier score indicates better probabilistic accuracy. However, the Brier score can favor conservative predictions near the empirical success rate even when they are weakly discriminative. We therefore also report resolution, where higher values indicate greater separation among samples with different empirical success rates and thus more informative probabilities for downstream planning. To account for the domain shift in probability estimates, we reserve 20% of the real-world data for post-hoc calibration [28] and report the real-world probability metrics on the remaining held-out data.
2) Performance Analysis: As shown in Table I, the DINO-based predictor and the LoRA fine-tuned VLM slightly outperform the VLM-GP and pretrained VLM on the simulation validation set. However, their performance degrades substantially on the real-world data. Although the LoRA fine-tuned VLM and DINO-based predictor achieve lower Brier scores than VLM-GP on Real-Tried, both learned baselines exhibit substantially lower resolution, indicating that their favorable Brier scores partly arise from conservative predictions with limited discrimination. This suggests that directly adapting high-dimensional visual representations to simulated interaction data can lead to poorer transfer to real-world observations. By comparison, VLM-GP retains the pretrained VLM prior and learns only a residual correction in a low-dimensional physical attribute space, which is less sensitive to visual domain shift.
| Model | Real | Sim | ||
| All | Tried | OOD | ||
| AUROC | ||||
| VLM-GP | 0.868 | 0.770 | 1.000 | 0.702 |
| LoRA fine-tuned | 0.817 | 0.740 | 0.960 | 0.712 |
| Vanilla VLM | 0.827 | 0.681 | 1.000 | 0.662 |
| DINOv3 | 0.736 | 0.695 | 0.880 | 0.726 |
| Brier | ||||
| VLM-GP | 0.140 | 0.157 | 0.064 | 0.219 |
| LoRA fine-tuned | 0.148 | 0.139 | 0.187 | 0.203 |
| Vanilla VLM | 0.168 | 0.109 | 0.109 | 0.276 |
| DINOv3 | 0.175 | 0.152 | 0.277 | 0.195 |
| Resolution | ||||
| VLM-GP | 0.101 | 0.045 | 0.222 | 0.035 |
| LoRA fine-tuned | 0.078 | 0.033 | 0.178 | 0.038 |
| Vanilla VLM | 0.083 | 0.021 | 0.222 | 0.024 |
| DINOv3 | 0.043 | 0.021 | 0.165 | 0.045 |
Table I. Object movability prediction performance on the simulation validation set (Sim) and real-world test set (Real), where Real is further divided into the Tried and Out-of-Domain (OOD) subsets. Brier scores are computed from raw predictions for Sim and from calibrated predictions for Real.
The effect of residual correction is particularly evident on Real-Tried, where VLM-GP shows a larger improvement over the pretrained VLM than on Real-All or Real-OOD. This indicates that the GP provides useful robot-specific corrections when the test objects are supported by the interaction data. The LoRA fine-tuned and DINO-based predictors degrade more strongly on Real-OOD, whereas VLM-GP is less affected because its residual correction diminishes away from the training distribution and the prediction reverts toward the pretrained VLM prior.
Finally, the raw pretrained VLM exhibits particularly poor Brier performance on the uncalibrated simulation set despite providing useful semantic movability predictions. Unlike the raw VLM, the learned estimators are trained directly on binary robot interaction outcomes, providing supervision on the probability scale of interaction success. Representing movability as an event probability rather than an unscaled semantic score also allows the prediction to be refined or calibrated using interaction outcomes directly obtainable from robot execution. Such embodiment-grounded probabilistic processing is important for uncertainty-aware planning, since directly treating raw VLM outputs as interaction-success probabilities can produce poorly scaled estimates and distort downstream planning decisions.
Experimental Setup for Planner Testing
We deploy various planners on a Unitree G1 robot equipped with an RGB-D camera. Planners must generate a path and motion plan that guide the robot safely through obstacle-cluttered rough terrain while minimizing instability, clearing objects when necessary, and reaching the goal efficiently.
1) Planner Benchmarks: We evaluate six planners to isolate the contributions of the proposed framework:
Proposed: Uses VLM-GP movability estimates for uncertainty-aware planning while reconstructing terrain traversability beneath objects.
Pro-NoRecon: Assumes the terrain beneath objects is traversable without reconstruction. All other components remain identical to Proposed.
STATE-NAV [5]: A non-interactive rough-terrain planner that treats objects as terrain obstacles while sharing the traversability estimator and underlying planner with Proposed.
Pro-VanillaVLM: Replaces VLM-GP with the uncalibrated pretrained VLM evaluated in Table I, leaving all other components unchanged.
VAMOS [12]: An end-to-end VLN method that generates candidate paths from language instructions, from which we select the one with the lowest traversability cost. Among the tested prompts, the best-performing is “{object} is {probability} likely to be clearable. Clear it and continue, or go around it.”, where both variables are provided by the same VLM-GP used in Proposed for a fair comparison.
Pro-NoUnc (similar to [8, 14]): Treats VLM-GP movability estimates as scalar uncertainties in an additive planning objective, , where is a tunable weight, following the score-based planning structure of prior work.
2) Simulation Benchmark: Existing navigation benchmarks typically represent objects as static meshes and lack outdoor rough terrain, making them unsuitable for NVMO evaluation. We therefore construct three IsaacSim environments, shown in Figure 5. To prevent asset overlap with the simulation training data, we reserve 11 randomly selected objects exclusively for evaluation. Each planner is evaluated over 44 trials per environment using four random seeds. In each trial, the robot starts on flat terrain and is given 11 s to rotate, scan the environment, and initialize its plan toward a predefined goal. All experiments run on a desktop equipped with an Intel Core Ultra 9 285 CPU, 64 GB RAM, and an NVIDIA GeForce RTX 5090 GPU with 32 GB VRAM.
Figure 5. Overview of the IsaacSim benchmark environments. (a) Garden: The robot starts on the white tiles and reaches the goal through the left opening or right dirt path, both blocked by movable obstacles. (b) Construction: The wooden plank bridge offers a shorter route across uneven terrain than the detour along the concrete boundary. (c) Park: The shorter route is blocked by an object on highly uneven terrain, where pushing it may cause the robot to trip.
3) Real-World Validation: For real-world validation, we deploy the proposed framework on a Unitree G1 equipped with an NVIDIA Jetson AGX Orin (64 GB) and a ZED X camera, supported by an external laptop with an RTX 5090 Laptop GPU. The robot’s Livox LiDAR provides point clouds, while the ZED X supplies RGB and depth images. FAST-LIO [29] performs localization on the G1’s built-in Jetson Orin NX, while all other modules run on the AGX, except for VLM and GSAM inference, which are executed on the external laptop and transmitted to the AGX over the network. We retain the simulation settings except for the computational budgets of STATE-NAV’s A* and model predictive path integral (MPPI) modules.
| Planner | Garden | Construction | Park | ||||||||||
| N-SR | (Mean, s) | (CVaR, s) | I-SR | N-SR | (Mean, s) | (CVaR, s) | I-SR | N-SR | (Mean, s) | (CVaR, s) | I-SR | ||
| Proposed | 95.5% | 14.6 (+16%) | 20.0 (+13%) | 91.1% | 97.7% | 20.3 (+5.7%) | 24.9 (+0%) | 100% | 100% | 17.8 (+0%) | 19.0 (+0%) | - | |
| Pro-NoRecon | 95.5% | 14.4 (+14%) | 18.7 (+5.6%) | 93.5% | 93.2% | 21.2 (+10%) | 33.0 (+33%) | 96.0% | 86.4% | 18.9 (+6.5%) | 28.6 (+51%) | 0.0% | |
| Pro-VanillaVLM | 93.2% | 16.5 (+31%) | 30.4 (+72%) | 84.0% | 97.7% | 22.4 (+17%) | 25.2 (+1.4%) | 90.0% | 93.2% | 17.8 (+0.3%) | 19.7 (+3.7%) | - | |
| Pro-NoUnc (Best ) | 95.5% | 14.4 (+14%) | 21.0 (+19%) | 100% | 97.7% | 20.0 (+4.0%) | 32.6 (+31%) | 84.6% | 100% | 18.0 (+1.1%) | 20.9 (+10%) | 0.0% | |
| STATE-NAV | 0%† | - | - | - | 100% | 24.6 (+28%) | 25.0 (+0.5%) | - | 97.7% | 17.8 (+0%) | 19.3 (+1.7%) | - | |
| VAMOS | 52.3% | 12.6 (+0%) | 17.7 (+0%) | 75.6% | 63.6% | 19.2 (+0%) | 42.2 (+70%) | 51.9% | 11.4% | 21.3 (+20%) | 52.8 (+180%) | 42.9% |
Table 2. Navigation performance in Garden, Construction, and Park. N-SR denotes goal-reaching success with body tilt at most 45°. Mean and -CVaR are computed over successful runs, with the latter averaging the slowest 10%. I-SR denotes the success rate of attempted interactions. For Pro-NoUnc, is tuned independently for each environment based on N-SR. Bold indicates the best value in each column, and parentheses show relative degradation computed from unrounded values. † STATE-NAV cannot find a path because obstacles block all routes. A dash indicates unavailable metrics, including I-SR when no interaction is attempted.
As illustrated in Figure 6, we evaluate the framework in an indoor environment where obstacles are added to or removed from a previously mapped scene, requiring the navigation system to adapt to the changes. We test four objects available at the site, including a wagon used to transport the robot, tables, and a chair in the lobby. Each planner is evaluated in two trials per object, totaling eight trials. Given the limited sample size, we replace -CVaR with mean navigation distance, as reliable tail-risk estimation would require more trials. To evaluate the contribution of terrain reconstruction through the Pro-NoRecon ablation, we additionally construct rough terrain at the test site, as shown in Figure 8. Finally, we demonstrate the feasibility of the proposed framework in a natural outdoor environment, as shown in Figure 7.

Figure 6. Illustration of the real-world testing environment with path plan candidates. The green dashed line represents a route to the goal that involves interaction with a black wagon, while the orange line represents an alternative detouring route.
Figure 7. Outdoor testing environment (22 m × 41 m) with a shortcut path blocked by a chair. The green path indicates the route selected by the robot, which clears the chair to take the shortcut. The inset on the lower right shows a top-down view of the entire test site, including the selected shortcut path and the alternative orange detour path.

Figure 8. Real-world terrain reconstruction on rough terrain. (a) Egocentric view of a chair resting on an artificial rough-terrain setup constructed in the indoor environment shown in Figure 6. (b) BEV elevation map measured with the chair present. (c) Terrain map reconstructed using the method in § 4. (d) BEV elevation map measured after the chair is removed.
Performance Metrics
We evaluate navigation success rate (N-SR), mean navigation time (), conditional value at risk (-CVaR), and interaction success rate (I-SR). A run succeeds if the robot reaches the goal without its body tilt exceeding 45°. N-SR is computed over all runs, while mean and -CVaR use successful runs only. The latter measures tail performance as the average navigation time among the slowest 10% of successful runs. I-SR is the fraction of successful obstacle-pushing interactions among all attempts. Since all planners share the same interaction controller, a higher I-SR indicates more reliable selection of objects for interaction.
Planner Testing Results
RQ2: What new navigation capability does the proposed framework enable? We examine the navigation capabilities enabled by the proposed framework across the planner benchmarks in Table II. In Garden, the non-interactive STATE-NAV planner cannot generate a path because movable objects block every route to the goal. In Construction, STATE-NAV achieves a 100% N-SR but a higher mean because it cannot exploit the shorter interactive route, whereas successful interactive runs complete the task in approximately 15 s. In Park, Pro-NoRecon exhibits a notably lower N-SR and higher -CVaR because it more frequently selects obstacle-passing routes without accounting for the traversability of the terrain beneath the object. This leads to falls or backtracking when the revealed terrain cannot be safely traversed, demonstrating the importance of reasoning about the terrain beneath movable objects. By jointly reasoning about object movability and terrain traversability, the proposed framework enables reliable navigation through cluttered rough terrain.
RQ3: Does embodiment-grounded VLM-GP improve navigation performance? To assess how movability estimation affects navigation, we compare the proposed VLM-GP with the vanilla VLM beyond the prediction-level results in Table I. The vanilla VLM exhibits a substantially higher Brier score, and Pro-VanillaVLM correspondingly achieves a lower N-SR in Table II. More importantly, I-SR degrades notably in Garden and Construction, indicating that inaccurate movability estimates can lead the planner to select difficult-to-move objects or avoid favorable interactions. These errors also worsen tail performance, with Pro-VanillaVLM exhibiting a higher -CVaR in all environments, particularly in Garden. Overall, more accurate embodiment-grounded movability estimates enable better interaction decisions and improve both navigation reliability and tail performance.
RQ4: Is uncertainty-aware planning with probabilistic VLM outputs effective for NAMO on rough terrain? Finally, we compare the proposed planner with VAMOS and Pro-NoUnc to examine planning formulations that jointly reason about terrain traversal, object interaction, and alternative routes. As shown in Table II, VAMOS performs notably worse
| Planner | N-SR | (Mean, s) | Distance (Mean, m) |
| Proposed | 100% | 23.2 (+0%) | 10.3 (+0%) |
| Pro-VanillaVLM | 100% | 29.2 (+26%) | 16.7 (+62%) |
| STATE-NAV | 100% | 27.6 (+19%) | 16.8 (+61%) |
| VAMOS | 0%† | - | - |
| Pro-NoRecon | 0%† | - | - |
Table 3. Real-world navigation performance. N-SR denotes goal-reaching success without safety intervention by a human operator. Mean navigation time () and distance are computed over successful runs. Bold indicates the best value in each column, and parentheses show relative degradation from the best value. VAMOS and Pro-NoRecon are evaluated only on the chair and wagon. All attempted trials require early termination due to imminent instability, yielding 0% N-SR on this limited subset. Further testing is discontinued for chair safety. A dash indicates unavailable metrics or no interaction attempts.
than the other methods and exhibits substantially higher -CVaR in Construction and Park. This degradation appears to reflect limitations in the global spatial reasoning of its end-to-end VLN formulation. Without an explicit spatial map over its observation history, VAMOS often fails to identify globally preferable detours and frequently selects object interaction when an object is mentioned in the prompt. Moreover, in Garden, when both candidate objects are visible in the same field of view, VAMOS consistently moves toward the object closer to the image center despite differences in their estimated movability. These observations indicate difficulty in comparing competing interaction and detour options and highlight the benefit of explicit spatial representations with persistent memory for long-horizon navigation.
Pro-NoUnc exhibits a different limitation. For Table II, we report its best-performing configuration from an environment-specific sweep over the weight . Although this tuning yields N-SR and mean comparable to the proposed method, its I-SR and -CVaR are notably worse in Construction despite using the same movability estimator, indicating that the difference arises from the planning formulation. Pro-NoUnc combines path-traversal and object-related costs through an additive weighted sum without explicitly accounting for the consequences of successful and failed interactions, so a favorable weighted score can still lead to a costly fallback after interaction failure. Moreover, the best weights differ substantially across environments, with , 20, and 10 for Garden, Construction, and Park, respectively, and Figure 9 shows strong performance sensitivity to within each environment. This sensitivity arises from directly combining quantities with different meanings and scales, making the appropriate weight dependent on the environment and route geometry. By contrast, the proposed formulation treats movability as the probability of interaction success and uses it to combine the navigation costs of the corresponding success and failure outcomes within a common cost domain. Together, these results indicate that explicit spatial memory and uncertainty-aware probabilistic planning provide a more robust formulation for humanoid NAMO on rough terrain.
Figure 9. Sensitivity to across three environments. The top and bottom rows show navigation failure rate () and -CVaR, respectively. Lower values indicate better performance for both metrics. Dashed lines represent Pro-NoUnc with varying , while solid lines represent the proposed method without weight tuning.
RQ5: Does the proposed navigation stack transfer to the real world? The real-world results in Table III evaluate the feasibility of deploying the proposed framework on a physical humanoid. Given the limited number of trials, we focus on Proposed, Pro-VanillaVLM, and STATE-NAV to illustrate how movability estimation affects navigation decisions. For the three movable objects, the vanilla VLM predicts an average movability of 0.33, whereas the proposed embodiment-grounded VLM-GP estimates 0.90. Consequently, Proposed favors interaction routes, while Pro-VanillaVLM and the non-interactive STATE-NAV favor detours. Although all three planners achieve 100% N-SR, Proposed reduces both navigation time and distance. We also test VAMOS in two trials, both of which require manual termination due to imminent instability, prompting us to discontinue further testing for hardware safety.
To assess terrain reconstruction, we additionally introduce artificial rough terrain consisting of wooden boards and bricks beneath the chair and the wagon. As shown in Figure 8, the proposed method reconstructs the underlying terrain with an average height RMSE of 5.0 cm across the chair and wagon tests, although some sharp terrain features are smoothed. This reconstruction enables Proposed to identify the underlying terrain as hazardous and select a detour, whereas Pro-NoRecon attempts to traverse the same region and requires early termination due to imminent instability. A video of the Pro-NoRecon failure is available on our project website.
Finally, we demonstrate the framework in a natural outdoor environment, as shown in Figure 7, where the robot selects a shortcut by clearing a chair blocking the path. Together, these preliminary experiments demonstrate the feasibility of the integrated framework on a physical humanoid and suggest that embodiment-grounded movability estimation and terrain-aware planning improve navigation efficiency and robustness.
Conclusion and Future Works
In this paper, we present a framework that extends humanoid navigation beyond obstacle avoidance to physical interaction on rough terrain. Our embodiment-grounded VLM-GP combines pretrained semantic priors with robot interaction outcomes to produce embodiment-specific probabilistic movability estimates. By integrating these estimates with object grounding, underlying terrain reconstruction, and uncertainty-aware planning, the framework evaluates interaction opportunities while accounting for potential failure costs. Extensive simulation experiments and real-world validation demonstrate the effectiveness and feasibility of the proposed framework.
Nevertheless, the current framework relies on a relatively simple pushing strategy without explicitly optimizing object displacement for subsequent navigation. Scalability to densely cluttered environments also remains challenging, particularly when distinguishing multiple object instances and repeatedly evaluating their movability using VLMs. Future work will investigate more sophisticated interaction strategies and scalable perception and planning to further advance long-horizon humanoid NAMO in complex environments.
Acknowledgment
LLMs, including ChatGPT and Claude Code, are used in polishing the writing of this paper. In particular, spelling and grammar structure are aided by LLMs. Additionally, Claude Code is used to clean and polish our code base. This aids in formatting and removing deprecated code.
References
- [1]Z. Gu, J. Li, W. Shen, W. Yu, Z. Xie, S. McCrory, X. Cheng, A. Shanshah, R. Griffin, C. K. Liu et al., “Humanoid locomotion and manipulation: Current progress and challenges in control, planning, and learning,” IEEE/ASME Transactions on Mechatronics, vol. 31, no. 2, pp. 2300–2330, 2026.
- [2]J. He, C. Zhang, F. Jenelten, R. Grandia, M. Bächer, and M. Hutter, “Attention-based map encoding for learning generalized legged locomotion,” Science Robotics, vol. 10, no. 105, p. eadv3604, 2025.arxiv.org/abs/2506.09588
- [3]S. Wei, H. Jing, B. Li, Z. Zhao, J. Mao, Z. Ni, S. He, J. Liu, X. Liu, K. Kang et al., “Psi-0: An open foundation model towards universal humanoid loco-manipulation,” arXiv preprint arXiv:2603.12263, 2026.arxiv.org/abs/2603.12263
- [4]F. Liu, Z. Gu, Y. Cai, Z. Zhou, H. Jung, J. Jang, S. Zhao, S. Ha, Y. Chen, D. Xu et al., “Gpt2Skill: Imitating dynamically-feasible whole-body trajectories for versatile humanoid loco-manipulation,” IEEE Robotics and Automation Letters, 2025.arxiv.org/abs/2409.20514
- [5]Z. Yoon, L. Y. Zhu, J. Lu, L. Gan, and Y. Zhao, “STATE-NAV: Stability-aware traversability estimation for bipedal navigation on rough terrain,” IEEE Robotics and Automation Letters, vol. 11, no. 2, pp. 2338–2345, 2026.arxiv.org/abs/2506.01046
- [6]M. G. Castro, S. Triest, W. Wang, J. M. Gregory, F. Sanchez, J. G. Rogers, and S. Scherer, “How does it feel? Self-supervised costmap learning for off-road vehicle traversability,” in 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023, pp. 931–938.
- [7]D. M. Ramesh, A. Sivaramakrishnan, S. Keskar, K. E. Bekris, J. Yu, and A. Boularis, “PROBE: Proprioceptive obstacle detection and estimation while navigating in clutter,” in 2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 698–704.arxiv.org/abs/2505.11848
- [8]P. Schoch, F. Yang, Y. Ma, S. Leutenegger, M. Hutter, and Q. Leboute, “In-Sight: Interactive navigation through sight,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 7794–7800.arxiv.org/abs/2408.00343
- [9]C. Dai, X. Liu, K. Sreenath, Z. Li, and R. Hollis, “Interactive navigation with adaptive non-prehensile mobile manipulation,” IEEE Robotics and Automation Letters, vol. 11, no. 2, pp. 1154–1161, 2026.arxiv.org/abs/2410.13418
- [10]M. Wang, R. Luo, A. Ö. Oñol, and T. Padir, “Affordance-based mobile robot navigation among movable obstacles,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2020, pp. 2734–2740.arxiv.org/abs/2102.04918
- [11]A.-C. Cheng, Y. Ji, Z. Yang, Z. Gongye, X. Zhou, J. Kautz, E. Biyik, H. Yin, S. Liu, and X. Wang, “NaVILA: Legged robot vision-language-action model for navigation,” arXiv preprint arXiv:2412.04453, 2024.arxiv.org/abs/2412.04453
- [12]M. G. Castro, S. Rajagopal, D. Gorbatov, M. Schmitt, R. Baijal, O. Zhang, R. Scalise, S. Talia, E. Romig, C. de Melo et al., “VAMOS: A hierarchical vision-language-action model for capability-modulated and steerable navigation,” arXiv preprint arXiv:2510.20818, 2025.
- [13]D. Song, J. Liang, X. Xiao, and D. Manocha, “VL-TGS: Trajectory generation and selection using vision language models in mapless outdoor environments,” IEEE Robotics and Automation Letters, 2025.arxiv.org/abs/2408.02454
- [14]M. Oh, C. Kim, S.-W. Seo, and S.-W. Kim, “Language as cost: Proactive hazard mapping using VLM for robot navigation,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025, pp. 21 543–21 550.arxiv.org/abs/2508.03138
- [15]K. Ellis, H. Zhang, D. Stoyanov, and D. Kanoulas, “Navigation among movable obstacles with object localization using photorealistic simulation,” in 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2022, pp. 1711–1716.DOI
- [16]A. Shamsham, J. Jiang, Z. Yoon, S. Coogan, and Y. Zhao, “Terrain-aware model predictive control of heterogeneous bipedal and aerial robot coordination for search and rescue tasks,” in IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 12 352–12 358.
- [17]M. Wermelinger, P. Fankhauser, R. Diethelm, P. Krüsi, R. Siegwart, and M. Hutter, “Navigation planning for legged robots in challenging terrain,” in 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016, pp. 1184–1189.
- [18]S. McCrory, M. Mishra, R. Griffin, J. Pratt, and H. E. Sevil, “Bipedal navigation planning over rough terrain using traversability models,” in SoutheastCon 2023. IEEE, 2023, pp. 89–95.DOI
- [19]D. Song, J. Liang, A. Payandeh, A. H. Raj, X. Xiao, and D. Manocha, “VLM-Social-Nav: Socially aware robot navigation through scoring using vision-language models,” IEEE Robotics and Automation Letters, vol. 10, no. 1, pp. 508–515, 2025.arxiv.org/abs/2404.00210
- [20]Z. Cao, Z. Chen, L. Pan, and Z. Liu, “Physyx-3D: Physical-grounded 3D asset generation,” Advances in Neural Information Processing Systems, vol. 38, pp. 93 771–93 784, 2025.DOI
- [21]M. Mittal, P. Roth, J. Tigue, A. Richard, O. Zhang, P. Du, A. Serrano-Munoz, X. Yao, R. Zurbrügg, N. Rudin et al., “Isaac Lab: A GPU-accelerated simulation framework for multi-modal robot learning,” arXiv preprint arXiv:2511.04831, 2025.arxiv.org/abs/2511.04831
- [22]H. Zhao, R. Cathemon, L. Gulich, W. Liu, E. A. Ongań, M. Lin, S. Jain, S. Pouya, and Y. Chang, “AGILE: A comprehensive workflow for humanoid loco-manipulation learning,” arXiv preprint arXiv:2603.20147, 2026.arxiv.org/abs/2603.20147
- [23]C. E. Rasmussen and C. K. I. Williams, Gaussian Processes for Machine Learning. Cambridge, MA: MIT Press, 2006.DOI
- [24]T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan et al., “Grounded SAM: Assembling open-world models for diverse visual tasks,” arXiv preprint arXiv:2401.14159, 2024.arxiv.org/abs/2401.14159
- [25]R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukhа, A. Silvestrov, N. Kong, H. Goka, K. Park, and V. Lempitsky, “Resolution-robust large mask inpainting with Fourier convolutions,” in 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2022, pp. 3172–3182.arxiv.org/abs/2109.07161
- [26]N. Ravi, V. Gabuer, Y.-T. Hu, R. Hu, C. Rualdi, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V. Alwala, N. Carion, C.-Y. Wu, R. Girshick, P. Dollár, and C. Feichtenhofer, “SAM 2: Segment anything in images and videos,” arXiv preprint arXiv:2408.00714, 2024. [Online]. Available: https://arxiv.org/abs/2408.00714
- [27]O. Siméoni, H. V. Vo, M. Seitzer, F. Baldassarre, M. Oquab, C. Jose, V. Khalidoy, M. Szafraniec, S. Yi, M. Ramamonjisoa et al., “DINOv3,” arXiv preprint arXiv:2508.10104, 2025.arxiv.org/abs/2508.10104
- [28]C. Guo, P. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in International Conference on Machine Learning (ICML). PMLR, 2017, pp. 1321–1330.arxiv.org/abs/1706.04599
- [29]W. Xu, Y. Cai, D. He, J. Lin, and F. Zhang, “FAST-LIO2: Fast direct LiDAR-inertial odometry,” IEEE Transactions on Robotics, vol. 38, no. 4, pp. 2053–2073, 2022.arxiv.org/abs/2107.06829



