TravSUITE: Traversability via Self-Supervised, Uncertainty-Aware IRL and Terrain Estimation

Introduction

Many important application domains such as construction [3, 8], disaster response [54, 62, 4], forestry [11, 9], and defense [66, 6, 73], require robots to be able to navigate in off-road environments reliably and autonomously. Most practical approaches see a mobile robot 1) use its on-board sensors to build a local map of its surroundings, 2) perform traversability analysis on the local map to determine where it can and cannot traverse, and 3) use the results of its traversability analysis to plan safe paths to a desired goal. This approach is preferred in safety-critical applications due to its modularity, interpretability, and ease of incorporating additional safety constraints, and has thus been the dominant framework for performing navigation across legged [24, 26, 62, 10] and wheeled [61, 6, 40, 38], for both learning [15, 27, 63, 74, 71, 62] and non-learning based [18, 24, 16, 36] methods.

While this approach may seem simple at first glance, both the local mapping and traversability problems become very challenging in complex environments such as off-road settings. The quality of the robot’s local map can be significantly degraded due to factors such as observation noise and occlusion. Cost function design is a complex function of robot capability, operator preference and the environment’s geometry and semantics. Effective robot navigation systems must be able to produce reliable and nuanced traversability estimates even in complex environments where the presence of observational uncertainty may render the local map erroneous or incomplete.

There exist several recent works that aim to improve the resilience of off-road mapping pipelines to observational noise and uncertainty [57, 40, 64, 27]. These approaches typically rely on using a large neural network to predict some form of ground-truth map representation (derived from privileged data such as hindsight, hand-labeling, simulation, etc.) from the current observations. These approaches have been demonstrated to produce reliable local maps that are robust to occlusions and measurement sparsity at high speeds and long ranges [40, 27]. However, these methods often rely on robot-specific traversability data [27], or environment-specific semantics and hand-annotations [57, 40], making it challenging to generalize these approaches to novel robots or environments.

To address the challenge and tedium of hand-designing traversability rules, there has been significant prior work in learning cost functions directly from robot traversal data [6, 26, 63, 15, 33, 65, 74]. While these approaches have demonstrated promising results across many environments and robot morphologies, there lacks a common consensus on the choice of input representation (e.g. geometric, semantic, visual foundation model (VFM) features, etc.) [15, 33, 26, 65], and its effect on the downstream traversability are not well-explored. Additionally, these approaches may not account for perceptual uncertainty [51, 15, 33], and often lack large-scale datasets and dense supervision signals [26, 64, 59, 15].

TravSUITE (Figure 1) synthesizes insights from both families of work to produce reliable and expressive traversability estimates in the presence of observational uncertainty. TravSUITE consists of three major components, 1) a flexible and efficient mapping module that produces real-time geometric-semantic map representations, 2) a framework for generating large, self-supervised, annotation-free perception and behavior datasets from robot experience, and 3) a unified neural network trained to jointly predict inpainted map representations and traversability estimates from this dataset with uncertainty. We leverage the flexibility of our mapping module and learning framework to compare against state-of-the-art approaches [40, 65, 74] and systematically evaluate what design decisions (such as mapping representation and training tasks) are important for improving path planning performance. We find that across all of our experiments, the existence of an inpainting task significantly improves traversability estimation performance, regardless of mapping representation. We also demonstrate that learning distributions of cost is important to enabling safer traversal at deploy time.

Image not extracted

Related Work

The vast majority of fieldable off-road systems make use of some local map of the robot’s immediate surroundings [61, 4, 40, 62, 54]. Typically, this local map is typically represented in one of two ways, as bird’s eye view (BEV/2.5D), or volumetric (3D). BEV-based approaches elect to partition the robot’s surroundings into “cells” of finite horizontal ($xy$) extent and either infinite (or hand-designed) vertical extent. Volumetric approaches partition the local environment into cells of finite $xyz$ extent (i.e. voxels) [25, 44, 5]. In either case, cells store navigation-relevant quantities derived from sensor data, such as the elevation of the support surface [23, 22], semantic probabilities [38, 40], appearance data [15, 33] or deep features [20, 65, 32]. While BEV-based approaches sacrifice representation fidelity (e.g. overhangs, multiple support surfaces), they are often more computationally efficient and closer represent the manifold that terrestrial robots must reside on. Volumetric approaches, on the other hand, more closely represent the local environment. However, they typically are more challenging to maintain and often require additional support surface estimation techniques for practical use by terrestrial robots [36].

Similarly, various methods for encoding semantics have been proposed in the literature. The dominant approach for off-road driving is to perform semantic segmentation over a fixed set of traversability-relevant classes (e.g. trails, rocks, etc.) [39, 30, 40, 42], or traversability classes themselves [28, 57]. There has also been recent interest in encoding semantic information via VFMs such as Segment Anything [35] or Dino [43]. These approaches have the potential benefit of additional semantic granularity [65], or open-set segmentation capabilities [67]. In either case, both semantic probabilities and VFM features can be aggregated in both BEV and voxel-based approaches [39, 44, 20, 57, 40, 65, 74, 32].

There also exists a significant body of work on filling in occlusions in the local map. Early approaches rely on leveraging classical techniques such as Markov random fields [38, 69] to inpaint the terrain surface. Advances in computer vision have enabled large neural networks to learn BEV-space inpainting on a number of quantities including geometry [64, 32], semantics [57, 40], costs [27, 46] or VFM features [74]. These approaches typically rely on predicting the map representation at a fixed number of steps in the future. To the authors’ knowledge, inpainting is always performed in BEV space and no work currently exists that directly inpaints volumetric representations for off-road environments.

There also exist many approaches to performing off-road traversability. A popular and straightforward approach is to hand design a cost function that relies on geometric properties of the local environment [24, 16, 36, 22, 64] such as the local slope and roughness of the terrain surface. While such an approach has been demonstrated to be effective for certain use cases and austere environments [61, 41, 47, 4, 54], a well-known failure case of these methods is false positives due to tall but compliant obstacles such as vegetation. Thus, approaches that hand-craft a traversability function based on geometry and semantics have been used to navigate in more challenging environments such as forests and snow [39, 57, 40, 27]. Another popular family of approaches relies on regressing some sort of traversability signal to the perception representation using deep learning. Many possible quantities have been proposed and demonstrated as a proxy for traversability, including experienced slip or roughness [68, 15, 26, 13, 33], simulated rollouts [25], and probability of expert demonstrations [51, 75, 63, 74, 31]. Oftentimes, it is valuable to produce a distribution of costs to account for uncertainty in the perception representation or traversability function. Uncertainty in the perception can be expressed as a distribution over geometric features [22] or semantic classes [40, 57]. This uncertainty is propagated through, or incorporated in, simpler traversability rules [22, 40] to produce a cost distribution. Similarly, uncertainty in the cost function evaluation can also be expressed via reconstruction error [60, 65, 55], ensembling [51, 63], or evidential learning [13]. In all cases, this results in a distribution of costs per state that must be compressed into a single value for downstream path planning. It is common practice to use either expected value [51, 40], or conditional value-at-risk (CVaR) [52] to capture long-tail, worst-case costs [22, 13, 63].

TravSUITE represents an advance in state-of-the-art for autonomous local navigation in that our framework is flexible with respect to choices of both input representation and semantics. We also design our system to jointly perform map inpainting and traversability learning without any human annotations while maintaining an environment-generic representation. This enables us to perform a systematic evaluation of many design decisions, including input representation, traversability function structure, and use of inpainting tasks. Lastly, our choice of network architecture allows us to produce uncertainty estimates for test-time deployment, and is extensible to other off-road perception tasks, such as roughness/slip prediction [15, 33, 26], which we do not focus on in this work.

Method

Problem Setup

We will consider a robot tasked with navigating to a distant goal point $x_G$. At every discrete timestep $t$, the robot will have access to a state estimate $x_t$ and observation $O_t$. We will assume we have access to a dataset of expert traversals that are assumed to be locally optimal. A datapoint in this dataset thus consists of three components $[O_{0:t}, x_G, \tau^E]$, which are: the set of observations $O_{0:t}$ that the robot has accrued to this point, a desired goal state $x_G$, and the trajectory that the expert took to reach the goal $\tau^E$. The traversability estimation problem can be thought of as finding a cost function $f_\theta(\tau\mid O_{0:t}, x_G)$ under which the cost-minimizing trajectory $\hat{\tau}$ closely matches $\tau^E$ under some objective function $\mathcal{L}$ (1).

In the scope of this work, we will assume that the cost function $f_\theta(\tau)$ is a weighted combination of distance to the goal $x_G$, and cost incurred from a “behavior map” $\Pi_t$. For the purposes of this paper, we will define our behavior map to be in BEV, and contain a cost and maximum speed for every cell (i.e. $\Pi_t = [C_t,S_t], \Pi_t \in\mathbb{R}^{2\times H\times W}$). By a slight abuse of notation, let $\Pi_t(x)$ represent a function that extracts the relevant values at state $x$ from a map. Given this, we can perform planning by scoring trajectories, conditioned on $\Pi_t$ via Equation (2) ($K_1 \approx\infty$ is used to ensure no state in the trajectory exceeds its corresponding speed limit, and $K_2$ is a coefficient used to balance the tradeoff between minimizing costmap cost and final-state distance to goal).

Decomposing Mapping and Traversability

Given the form of the cost function in Eq. (2), the goal of the traversability estimation problem is to find the best form and parameters of $f_\theta$ (and thus $\Pi_t$) that yield trajectories $\hat{\tau}$ that minimize the objective $\mathcal{L}(\hat{\tau},\tau^E)$. It is helpful to decompose the traversability function $f_\theta$ into two components, a mapping function $M_t=h_\psi(O_{0:t})$ that compresses sensor observations into a map $M_t$, and a traversability function $g_\phi(M_t)$ that converts the map into the behavior map $\Pi_t$ (Eq. (3)).

It is important to note that the behavior map $\Pi_t$ is produced via a composition of the traversability function $g_\phi$ and mapping function $h_\psi$, which each have their own parameter vector. This is important as many prior works rely on directly improving the mapping function $h_\psi$ through auxiliary tasks such as inpainting [40, 64, 74]. In this context, these approaches can be thought of as introducing an auxiliary objective $\mathcal{L}_{aux}$ and dataset $\mathcal{D}_{aux}$ to add additional structure to $M_t = h_\psi(O_{0:t})$. Since the trajectory objective in Eq. (1) depends on the mapping representation $M$ via Eqs. (3) and (2), it stands to reason that this auxiliary task may be able to improve the downstream traversability estimation. Ultimately, this results in a unified objective (Eq. (4)) in which both the trajectory $\hat{\tau}$ and map $M_t$ are both being optimized.

Many state-of-the-art traversability approaches can be viewed under this framework [40, 57, 18, 74, 63, 33] through different forms of $g,h$ and whether to supervise $M,\tau$. An overview of our method’s supervision can be found in Figure 2. In the following sections, we will describe our choice of supervision for the behavior learning and map learning components of our method.

Image not extracted

Optimizing Behavior

In order to perform the behavior learning component of Eq. (4) (and to generate the behavior labels in Figure 2), we elect to use the MaxEntIRL [77] framework, in which the likelihood of a trajectory is inversely proportional to the exponent of its cost, and the objective is to maximize the likelihood of $\tau_E$ (Eq. (5)). We follow a LEARCH-like procedure [51], where we sample an optimal trajectory $\hat{\tau}$ under the current costmap and goal, and then compute the gradient of the IRL objective w.r.t. the costmap to be the difference in state visitations between the expert and learner trajectories ($SV$ is a state visitation extraction function defined in the appendix). This gradient can be used to update $g_\phi$ and $h_\psi$.

We also use the expert trajectory $\tau_E$ to learn a speedmap $S_t$, which contains the maximum safe traversal speed for a given cell. It is important to note that we must produce an upper bound, making it necessary to predict distributions of speed rather than just the mean value. As such, we follow the example used by Cai et al. [13] by parameterizing our speedmap $S_t$ as a categorical distribution over fixed bins, per cell. We train our model to minimize squared earth mover’s distance to a delta distribution centered on the expert speed for every cell that the expert traversed. In order to encourage safe behavior, we also add a small amount of negative samples (where the target is 0) for untraversed cells (Eq. (6)).

In practice, we obtain labels for this behavior learning by extracting fixed-length trajectory segments from our dataset of expert traversals. For each datapoint, we sample a trajectory segment 50m in length and assign $x_G$ to be the final state in the trajectory. This approach is preferred to time-based sampling as it ensures that demonstrations in more challenging scenarios (with slower expert speeds) do not have reduced path complexity. In order to sample learner trajectories $\hat{\tau}$, we use MPPI [72] under a kinematic bicycle model and the cost function from Eq. (2). Note that for training stability, we set the speed penalty ($K_1$) to 0 at train time.

Optimizing the Perception Representation

In order to instantiate the perception learning component of Eq. (4) (and the perception labels in Figure 2), we must define $M_t^*$. Similar to prior work [40, 74], we use hindsight supervision in order to generate labels $M_t^*$, and train our mapping network $M_t=h_\psi(O_{0:t})$ to predict $M_t^*$ (Eq. (7)). Note that this objective encourages inpainting as $M_t^*$ is a function of both past and future timesteps (e.g. $M_t^* \sim O_{0:T}$).

Implementation Details

Aggregating Observations with VFM Voxel Mapping

In practice, it is not feasible to maintain and process an ever-growing list of observations $O_{0:t}$ when $t$ can span thousands of timesteps or more. To address this, we develop a voxel mapper that aggregates visual and geometric features in a fixed volume around the robot, given state estimates $x_t$. We chose to use a voxel grid as our core mapping representation instead of a BEV-based approach in order to enable our system to be more generically used across environments and tasks (e.g. indoor environments, manipulation), which may require 3D information. Furthermore, we are able to generate an accurate BEV representation from a voxel grid, while the opposite is not true. We also find that our voxel mapper’s processing times are competitive with an elevation mapping-based baseline and is sufficient for real-time operation. Further details are provided in the appendix. For every timestep in our expert dataset, we are able to represent $O_{0:t}$ with either a voxel map $V_t$ or BEV map $B_t$ containing much of the traversability-relevant information from all prior observations. These maps $V_t,B_t$ will serve as input to the learned components of TravSUITE. For all experiments, our map size is $100m\times100m\times20m$ at $0.4m\times0.4m\times0.1m$ per cell, centered on the robot. The list of map features can be found in Table I.

Feature	Modality	Rep.	Description
Terrain	Geom.	BEV	Elevation of the terrain surface
Slope	Geom.	BEV	Slope of the terrain surface
Diff	Geom.	BEV	Max elevation over the terrain
min/max z	Geom.	Voxel	The min/max height in each voxel
Seg Logit x16	Seg.	Both	Class score from Talk2Dino [7]
VFM x32	VFM	Both	VFM feature from R dichaioV3 [50]

Generating Hindsight Supervision from Voxel Maps

In order to perform inpainting, we construct an oracular voxel map ($M_t^*$) and BEV map ($B_t^*$) for every timestep in our dataset. We do this in a similar fashion to TerrainNet [40] by building a voxel map $V_t^*$ from both past and future sensor measurements. Instead of setting a fixed time window to aggregate (e.g. $V^{*}_{T} \sim[V_{t-k} \ldots V_{t+k}]$), we instead keep track of the mapping volumes for each timestep and take as $V^{*}_{t}$ the aggregate of all voxels from every voxel grid that intersects with the volume defined by $V_{t}$ (detailed algorithm in appendix). This has an advantage in that the quality of aggregation does not depend on robot speed. Additionally, since we directly predict the future map (instead of semantic annotations), this process is fully annotation-free. For both voxel and BEV-based approaches, we choose to only use $B^{*}$ as a prediction target. Since the supervision is derived from local perception, cells in $B^{*}$ may still be unreliable at longer sensing distances. To address this, we mask out BEV cells that were observed at a distance no closer than $10m$ from the robot. Note that this distance includes all future poses $x_{0:T}$, meaning $B^{*}_{t}$ can contain labels at any distance from the current state $x_{t}$ and thus requires the network to inpaint.

Network Architecture

We instantiate the perception and traversability function from Equation (3) as a multi-headed neural network, similar to ParaDrive [70]. The network takes as input $M_{t}$ (which may be either $V_{t}$ or $B_{t}$), and uses an encoder and backbone to produce a latent BEV representation $Z_{t}$. Multiple different heads are then used to produce navigation-relevant quantities (e.g. costmaps, speedmaps, semantic maps) from $Z_{t}$. A high-level architecture diagram is presented in Figure 3.

Image not extracted

Encoder and Backbone

1) Encoder and Backbone: The role of the encoder and backbone is analogous to the mapping function $h_{\psi}$ in the previous section, where the goal is to convert the input into a BEV-space feature tensor of a given channel dimension (e.g. $h_{\psi}(O_{0:t}) \in\mathbb{R}^{F \times W \times H}$). If the input is $B_{t}$, this process is simple and can be implemented as a simple ResNet-style [29] network. If the input is $V_{t}$, we use a 3D sparse convolution-based analogue using spconv [17] and then splat the 3D features into BEV using a channel-wise max pool.

The backbone of our network is a UNet [53]. This UNet comprises the bulk of the learnable parameters in our network and enables the network to attend to longer-range dependencies in the data. The output of the UNet is a feature tensor $Z \in\mathbb{R}^{F \times W \times H}$ and is used as a shared input across all prediction heads $g_{\phi}^{i}$.

Prediction Heads

2) Prediction Heads: In order to convert $Z$ into the same semantic space as $B^{*}$, another lightweight decoder FCN [37] $g_{\phi}(Z_{t})$ is used. Following TerrainNet’s example, we use a separate head for each feature modality (e.g. geometry, semantics). A similar decoder architecture is used to predict cost and speed. However, as we are interested in estimating distributions of cost and speed, we parameterize the outputs of these heads as a categorical distribution over fixed bins. This approach has been demonstrated to be useful when the support of the target variable is known (e.g. traction, speed) [13, 65]. This also enables risk-aware behavior via sampling different risk measures of the resulting distributions (Figure 4).

Image not extracted

Leveraging Uncertainty

Since both predicted cost and speed are parameterized as categorical distributions, we are able to evaluate various risk measures such as VaR (quantiles) and CVaR (Figure 4). At test time, we are able to compute a risk-aware cost by evaluating the CVaR of every cell in the costmap. Similarly, we are able to tune the aggressiveness of our speedmap by choosing different quantiles of the resulting speed distributions.

We also implemented a very simple test-time CVaR modulation technique for costs in which a running risk value $\nu_{t}$ was maintained and updated based on the current speed. If the current vehicle speed exceeded a set value, or fell below another set value, the risk tolerance was slightly decremented or incremented, respectively (Equation (8)). While simple, this method has the desirable behavior of encouraging safe behavior if the robot is moving quickly, but allowing riskier behavior if little navigation progress is being made.

Training Details

For all experiments, we leverage a dataset of roughly four hours of expert driving at a primary and secondary test site, across all four seasons using an full-scale, autonomous off-road vehicle equipped with the sensor suite from TartanDrive 2.0 [58] (of which we use the LiDAR, IMU and RGB camera). In order to generate state estimates $x_t$, we leverage an off-the-shelf Lidar-inertial odometry algorithm [76]. Given these state estimates, we are able to run our VFM voxel mapper to generate inertial-frame VFM voxel maps at $10hz$. The dataset contains roughly 100,000 individual frames with a timestep of $0.1s$, split roughly 80/20 between the two locations. The dataset was divided into a train set and two separate test sets. The Train and Test 1 sets were partitioned (roughly 60/40 by GPS region) from the primary test site, while Test 2 contains the entirety of the data from the secondary site. Experiments were run for five epochs with a batch size of 3. Since the BEV inpainting targets are multi-modal (e.g. geometric/VFM/semantic features), we follow the procedure used in TerrainNet [40], where each modality is weighted separately. Smooth-L1 [2] ($\beta= 0.2$) is used for continuous features, while cross-entropy is used for segmentation features (Eq. (9)).

Experiments

In order to evaluate what design decisions matter for downstream path planning, we performed a series of rigorous online and offline evaluations using our dataset and real-world navigation trials.

Offline Experiments

In our offline experiments, we answer these questions:

1. RQ1: Do auxiliary tasks such as inpainting improve downstream path planning and if so which?

2. RQ2: How much does the choice of input representation affect downstream performance?

3. RQ3: How do we compare to relevant baselines?

Across our offline experiments, we use a common set of behavior and inpainting metrics. In order to measure the efficacy of the behavioral learning, we report two metrics. Modified Hausdorff distance [19] (MHD) between the expert trajectory and optimal learner trajectory is used to evaluate the cost prediction component. The average probability of the expert speed under the learned speedmap is used to evaluate the speedmap prediction component. To measure inpainting performance, we can directly compute the error between $h_\psi(O_{0:t})$ and $B^*$. We compute error separately for each feature modality as:

1. Geometry: $L_1$ error.

2. Semantics: $L_1$ distance between the semantic distributions (note that neither distribution is one-hot and this essentially measures total variation distance).

3. VFM: $L_2$ error.

RQ1: The Effect of Inpainting on Behavior Learning

In order to answer RQ1, we trained several variations of our voxel-based network using different permutations of inpainting modalities. In particular, we compare using no inpainting task (None in Table II), inpainting only geometric/VFM/segmentation features, and cotraining all inpainting modalities together. Overall we find that across both datasets, inpainting improves both the costmap and speedmap components of our network. Notably, all ablations in this experiment had the same number of behavioral examples, confirming the hypothesis that learning a better map representation $M$ results in better planning behavior. Perhaps unsurprisingly, using more modalities together seems to result in the best performance, as traversability is a function of both geometry and semantics.

RQ2: Choice of Input Representation

In order to answer RQ2, we explore several choices of input representations to our method. In particular, we consider three types of inputs:

1. Voxel: $V_t$ from Section IV-A.

2. BEV: $B_t$ from Section IV-A.

3. Voxel 1-frame: $V_t$, but only containing voxels from the current timestep (to ablate the importance of aggregating observations, as some approaches that inpaint do not aggregate measurements [48, 74], or find performance gains by not doing so [57, 40]).

Similar to the previous section, we report the planning metrics on the held-out dataset in Table III. Additionally, we report the inpainting accuracy in Table IV. Note that the cells used for With respect to path-planning metrics, we observe that the voxel method performs better on the costmap metric while the BEV method is better with respect to the speedmap metric. This indicates that (at least for our environments), both BEV and 3D representations perform reasonably well for traversability. Across both datasets, the single-frame voxel input method performs the worst, suggesting that significant performance gains can be achieved by aggregating sensor measurements.

We find again that aggregation of observations results in better inpainting performance for both observed and unobserved cells. This corroborates the findings of Shaban et al. [57] in that temporal aggregation is helpful in simplifying the map prediction task when odometry is reliable. We observe that with aggregation, both BEV and voxel-based inputs perform similarly, with the BEV-based model performing marginally better. This is likely due to the labels being generated using the same process as the inputs, making the BEV to BEV learning task somewhat simpler.

Input	Dataset	MHD ↓	Speed Prob. ↑
Voxel	Test 1	1.3936	0.1159
BEV	Test 1	1.4440	0.1184
Voxel no agg.	Test 1	1.5061	0.1109
Voxel	Test 2	1.3988	0.1258
BEV	Test 2	1.4373	0.1311
Voxel no agg.	Test 2	1.4452	0.1031

Input	Dataset	Geom ↓	VFM ↓	Seg ↓
Voxel	Test 1	0.7253	1.2324	0.3087
BEV	Test 1	0.7137	1.1450	0.3019
Voxel No Agg.	Test 1	1.5952	1.9533	0.5064
Voxel	Test 2	0.5221	1.2389	0.4279
BEV	Test 2	0.4704	1.1313	0.4019
Voxel No Agg.	Test 2	1.6326	2.0331	0.6033

RQ3: Comparison to Other Methods

We also compare our method to several approaches in recent literature. In order to do so, many of the ablations made use of an inpainting-only backbone. This backbone shared the architecture described in the previous section, but was trained only on the inpainting task. TerrainNet Cfn.: In order to produce a TerrainNet-style [40] baseline, we used the predicted geometric and semantic maps to implement the hand-designed cost function prescribed by TerrainNet. In order to improve the performance, we fine-tuned the coefficients of the cost function with IRL. As TerrainNet does not produce a speedmap, we do not report a speed metric. Cascaded Trav.: We produced a Creste-style [74] baseline by training a FCN with IRL, using the predicted geometric and VFM maps as input. FCN no Inpaint: We also trained a Velociraptor-style [65] baseline by training a lightweight FCN on $B_t$, using only the behavioral objective. Parallel Trav. is equivalent to inpainting with all modalities from Table II. Results can be found in Table V.

Inpainting	Dataset	MHD ↓	Speed Prob. ↑
All	Test 1	1.3936	0.1159
Geometry	Test 1	1.4138	0.1054
VFM	Test 1	1.5400	0.1162
Segmentation	Test 1	1.4877	0.1120
None	Test 1	1.7744	0.1026
All	Test 2	1.3988	0.1258
Geometry	Test 2	1.4410	0.0982
VFM	Test 2	1.4619	0.1158
Segmentation	Test 2	1.4228	0.1004
None	Test 2	1.5400	0.1086

Approach	Dataset	MHD ↓	Speed Prob. ↑
Parallel Trav.	Test 1	1.3936	0.1159
Cascaded Trav.	Test 1	1.3043	0.1076
TerrainNet Cfn.	Test 1	1.4553	N/A
FCN no Inpaint	Test 1	1.5246	0.0967
Parallel Trav.	Test 2	1.3988	0.1258
Cascaded Trav.	Test 2	1.3845	0.1156
TerrainNet Cfn.	Test 2	1.3965	N/A
FCN no Inpaint	Test 2	1.4961	0.1017

Overall, we find that both the parallel and cascaded traversability methods outperform the TerrainNet baseline, suggesting that a more complex cost function can lead to improved navigation performance. Interestingly, we find that the FCN baseline (the only method that does not use inpainting) performs the worst across both metrics, suggesting that inpainting with larger architectures is important to learning effective navigation maps. Additionally, we note that the FCN baseline outperforms the larger architecture (no inpainting from Table II), suggesting that it is difficult to train a large-scale architecture with traversability supervision alone. The performance of the parallel and cascaded architecture is relatively comparable with the former producing better speedmaps, and the latter producing better costmaps. Overall, this suggests that learning traversability from demonstrations can outperform traditional baselines, but it is important to add an inpainting task. Some qualitative prediction examples are presented in the hardware experiments section.

Sample qualitative results are presented in Figure 5 in which we compare the voxel network with inpainting (Voxel+All in Table II) to two networks without inpainting (Voxel+None in Table II and FCN no Inpaint from Table V). We observe significant improvements in the stability and accuracy of both the cost and speed maps as a result of inpainting. In practice, this prediction stability is important for real-world navigation.

Image not extracted

Hardware Experiments

We also perform several real-world hardware experiments to answer these questions:

1. Does our approach enable off-road navigation?

2. Does risk-aware planning enable safer navigation?

3. Do our offline findings translate to hardware?

In order to evaluate our algorithm on physical hardware, we set up a navigation trial using our full-scale ATV at our primary test site. The ATV was tasked with navigating as quickly as possible to a series of waypoints designed to force the vehicle to interact with a mixture of on-trail and off-trail scenarios, as well as vegetation and man-made obstacles. The experiment took place in the winter with somewhat significant snow cover. A full traversal of the course was roughly 1.4km.

A more complex version of MPPI was used on hardware. This version of MPPI used a kinematic bicycle model with additional throttle and steering dynamics to model the transient behavior of the throttle and steering actuators on the platform. In addition to the cost function from Equation (2), an additional lateral acceleration limit was added. Given the challenging conditions, the mean predicted speed from the speedmap prediction was used. All mapping and traversability computation was run at roughly $5\mathrm{hz}$ using an on-board laptop with an 13th gen Intel i9 CPU and Nvidia 4090 laptop GPU.

Several ablations from the previous section were compared:

1. Parallel Trav.: from Section V-D.

2. TerrainNet Cfn.: from Section V-D. Note that we also trained a speed prediction head from the inpainted latent features in a similar fashion to the previous sections.

3. FCN no Inpaint: from Section V-D.

4. Parallel Trav. + CVaR: Parallel Trav., but also using the online CVaR selection method from the Section IV-C

We use the following metrics for evaluation:

1. Number of Interventions (Int.): Number of times a safety operator was required to correct the robot. The

safety operator was instructed to intervene if and only if 1) a collision with an obstacle appeared imminent, or 2) the robot stopped making progress towards the next waypoint. If an intervention was called, the safety operator was instructed to teleoperate the vehicle past the offending obstacle and then resume autonomy.

2. Number of Collisions (Col.): Of the interventions above, the amount that were due to an obstacle collision. We report this number separately as many interventions due to lack of progress can be addressed with better backtracking and planning, while collisions cannot.

3. Average/Top Auto. Speed: The average/top speed while autonomous.

Analysis of Hardware Results

High-level metrics from each of the autonomy runs are presented in Table VI. A GPS plot of the interventions with FPV views is presented in Figure 6. Overall, we observed that the best-performing method in terms of intervention metrics is Parallel Trav. + CVaR. Notably, this was the only method that was able to successfully avoid all of the man-made obstacles, as Parallel Trav. and TerrainNet Cfn. each had a collision, suggesting that adding risk-awareness results in safer navigation. A comparison of the various methods navigating around some man-made cones can be seen in Figure 8. This scenario required the vehicle to divert from the trail to avoid the cones. Since cones were out-of-distribution from the train set, Parallel Trav. was overly optimistic. Similarly, the semantic inpainting struggled to inpaint the cones in the BEV map, leading TerrainNet Cfn. to understimate the cost. However, using CVaR to capture the semantic uncertainty enabled the robot to avoid the cones. Additionally, we can see that the FCN no Inpaint method was both the slowest and had the largest number of interventions, suggesting that the inpainting had a positive impact on both traversal speed and number of required backtracks. A qualitative example of the value of inpainting is presented in Figure 7 in which the FCN no Inpaint method assigned low cost to an area containing vegetation that was out of direct view. Since the goal point was directly beyond this region, the robot mistakenly started planning into this vegetation, resulting in it getting stuck in a cul-de-sac. However, with inpainting, all other models were

Method	Col.	Int.	Avg. Speed	Top Speed
Parallel Trav.	2	2	4.79m/s	7.66m/s
Terrainnet Cfn.	1	1	5.23m/s	7.72m/s
FCN no Inpaint	1	4	4.33m/s	6.84m/s
Parallel Trav. + CVaR	0	1	4.49m/s	7.16m/s

Image not extracted

able to pick up on the surrounding geometric and semantic cues and inferred that the unobserved area was likely high cost, leading the planner to take a longer but safer path on the trail. We also observed that all methods which inpainting had higher average autonomous speed. This is likely because the inpainting objective resulted in more temporally stable speed predictions. Fluctuations in the speedmap, like those seen in Figure 5 force the ATV to reduce its speed not just for the offending cell, but also for prior cells as braking cannot occur instantaneously. Unsurprisingly, adding CVaR (thus reducing risk-tolerance) decreased autonomous speed as the robot had to avoid more objects, but reduced the number of interventions.

Conclusion

In this work, we presented TravSUITE, a fully self-supervised traversability and map inpainting learning framework. We demonstrated that inpainting and risk estimation play an important role in learned traversability estimation methods in both offline and hardware results. Furthermore, we have demonstrated a straightforward technique for generating large-scale geometric-semantic inpainting supervision that requires no additional engineering overhead. We believe that this is a significant step forward for self-supervised traversability estimation methods in unstructured terrain.

Image not extracted

Image not extracted

While TravSUITE represents step forward for learning-based local traversability, there are some limitations of TravSUITE worth highlighting. First, while this method is environment and robot-generic in principle, additional experiments in additional biomes and platforms are needed to fully substantiate this claim. Second, while we provide a rudimentary uncertainty estimation technique that seems to work well in practice, a more principled treatment of uncertainty (such as [32, 14]) may improve performance in OOD. Lastly, TravSUITE only performs local traversability estimation and cannot reason about information beyond the mapping horizon.

Future work will include exploring additional sources of high-quality self-supervised data and demonstrating our system in more environments such as multi-floor buildings and on other platforms such as legged robots. We are also interested in co-training additional tasks such as full 3d path planning and segmentation that necessitate a volumetric representation. We are also interested in exploring non-metric representations to enable long-range traversability estimation and planning [21, 56].

Acknowledgments

The authors would like to thank Matthew Sivaprakasam, Micah Nye and Parv Maheshwari for their help with field testing and feedback in the paper-writing stage. This work was supported by DSO National Laboratories (DSO) Contract #DSOC025020. This work used Bridges-2 at PSC through allocation cis220039p from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program which is supported by NSF grants #2138259, #2138286, #2138307, #2137603, and #213296. This work was supported in part by Field AI, Inc.

Appendix

In Figure 9, we provide a GPS plot of the online CVaR value during the Parallel Trav. + CVaR experiment. We can see that maximum risk aversion ($\nu= 0.8$) is sufficient for the majority of the course, with a few more aggressive areas ($\nu= 0.2 - 0.5$) being required for a few areas with denser vegetation.

Image not extracted

We also present an additional qualitative example of the robot navigating under a significant overhang (Test 2) in Figure 10 in which we compare two of our networks trained on the full traversability and inpainting objective, where the only difference is whether the input representation was in 3D or BEV (Voxel and BEV from Table 4 in the main paper). Here, we can see that the voxel-based network correctly discards the information from the overhanging canopy and predicts low cost, while the BEV-based network mis-classifies the trail under the canopy as an obstacle.

Image not extracted

We will describe in greater detail the procedure we use for voxel mapping. We implement our voxel mapper entirely from torch [45] and torch_scatter [1] operations, enabling efficient parallel computation on GPUs. For the purposes of this section, we will assume that the vehicle receives a state estimate $x$, point cloud $P$ and image $I$ at every timestep. We will assume that all sensor streams are synchronized, though in practice we perform motion compensation to temporally align the image and point cloud.

Semantic Feature Extraction and Lifting

First, in order to aggregate our measurements in 3D space, we first extract meaningful semantic content from our image stream $I_t$, and lift it using the pointcloud $P_t$. We use two choices of semantic features which are common in prior work.

1. VFM semantics: A common approach to extracting semantics is to perform open-set segmentation on VFM features. In line with this, we leverage Talk2Dino [7] with text prompts from the Rellis3d semantic ontology [30] (with an additional snow class), and map the similarity between each pixel feature $z_I$ and text embedding $z_{c_i}$. Empirically, we found that “sharpening” the cosine similarities (by multiplying by a fixed $\beta$ before softmax) produced better segmentation results (Eq. (10)).

2. Raw VFM: Another popular approach to feature extraction is to simply map the VFM features directly [65, 5]. In this case, we choose to use the visual encoder used in RayFronts [5] (Radiov3 [50] with Siglip2 adaptor and NAClip attention). We then perform a principal component analysis to reduce the feature dimension further to the top 32 components.

In either case, the end result is a “feature image” $I^f \in\mathbb{R}^{W//k \times H//k \times n_f}$, where $k$ is the reduction in resolution from the original image.

Given the feature image, we must then lift our FPV image features into 3D. Fortunately, this process is straightforward via projection of points $[x,y,z] \in P_t$ into the image frame using Eq. (11), given camera intrinsics $\mathbf{K}$ and camera-lidar extrinsics $\mathbf{R}, \mathbf{t}$. (Note that $\mathbf{S} \triangleq\operatorname{diag}([\frac{1}{16},\frac{1}{16},1])$ is an additional scale matrix used to account for lower resolution of the feature image).

Voxel Local Mapping with Features

We apply several known tricks to improve the sharpness and accuracy of our feature pointcloud, such as bilinear interpolation and per-pixel depth binning [49, 5]. While this reduces the overall number of points that receive a VFM feature, it is essential to maintain sufficient sharpness of features, especially near edges of objects.

Given our streams of feature point clouds $P_t$, we can now accumulate them into a voxel grid. This voxel grid $V = \langle\mathcal{M}, \{i,f_i\} \rangle$ can be expressed as a local spatial extent (containing a region, length and resolution $\mathcal{M}_o,\mathcal{M}_l,\mathcal{M}_r$, each in $\mathbb{R}^3$), a set of voxels which can each be represented as a unique integer index $\{i \in[0,N_xN_yN_z]\}$ and feature vector $\{f_i \in\mathbb{R}^{n_f}\}$ (where $N_{x,y,z} = \lfloor\frac{\mathcal{M}_{x,y,z}}{\mathcal{M}_{r,x,y,z}} \rfloor$). In practice the voxels are stacked into one and two dimensional torch [45] arrays, respectively. The sparsity of this representation enables the voxel grid to be sufficiently efficient and compact to be suitable for real-time, online computation. Furthermore, this representation is analogous to that used by common sparse convolution libraries such as spconv [17]. The remainder of this section will detail the implementation of our voxel mapper, given our voxel representation.

The goal of the voxel mapper is to maintain and aggregate streams of feature pointclouds (transformed into an inertial frame via odometry) over time. Given the pose and feature pointcloud at the current timestep, and voxel grid at the previous timestep (and a fixed local extent $\mathcal{M}_{local}$), we can perform voxel mapping of features by performing the following operations:

1. EXTENT($x_t,\mathcal{M}_{local}$): Compute the current extent $\mathcal{M}_t$ from the current pose and local extent.

2. DISCRETIZE($x_t,P_t,\mathcal{M}_t$): Compute a voxel grid $\dot{V}$ from the current sensor measurements.

3. SHIFT($V_{t-1}, M_t$): Shift the previous voxel grid into the current frame ($V'_{t-1}$).

4. MERGE($V'_{t-1}, \dot{V}$): Update the shifted voxel grid with the instantaneous voxel grid.

Fortunately, efficient instantiations of the above methods can be implemented using a combination of torch [45] and torch_scatter [1] operations.

EXTENT

1. EXTENT: The EXTENT operation can be implemented via Equation 12. Note that we choose to have the extent origin be a multiple of the resolution to simplify the implementation of the downstream operations.

DISCRETIZE

2. DISCRETIZE: The core of the DISCRETIZE operation relies on a mapping between points in the mapping extent and unique integer indices defined in Equation 13. The instantaneous voxel grid can then be computed by computing the set of unique indices $i$, and defining how to compute a feature from the set of points with the same index $i$. We choose to have the voxel take the feature of the point closest to the robot pose, though other aggregation functions such as average are reasonable as well.

SHIFT

3. SHIFT: Given our choice of index assignment in Equation 13, we can efficiently perform transforms of integer multiples of the voxel map resolution via Equation 14.

Indices (and corresponding features) that fall outside of the mapping volume are discarded.

MERGE

4. MERGE: Finally, in order to merge two voxel grids together, the resulting set of indices can be computed as the set intersection of the indices in $V'_{t-1}$ and $\dot{V}$. If the index is in only one set, the corresponding feature is copied over. Otherwise, the features are combined via an exponential moving average.

We perform an additional, optional raycasting operation using $P_t$ to help with noisy observations and dynamic objects. For every voxel, we maintain an additional hit and miss feature. Voxels are initialized with $hits=1$, $misses=0$, and the counts are summed in the MERGE operation. A voxel’s miss counter is incremented if a ray from $x_t$ to any point in $P_t$ passes through the voxel. At every timestep, voxels with a passthrough ratio $\left(\frac{misses}{hits+misses}\right)$ over 0.75 are discarded.

Runtime Experiments

We ran an experiment to compare the speed of our voxel mapper to an elevation mapping-based alternative. In both cases, the local mapper was tasked to map the environment as well as the VFM features from Radiov3 [50]. In Table VII, we report the mean, minimum and maximum times (in $ms$) over a run of roughly 2000 frames for each subcomponent of the elevation and voxel pipelines. We observe that the total time per frame between each mapping pipeline is very similar, totaling a speed difference of fewer than $2ms$ on average per timestep. We note that while the voxel mapping is slower than the elevation mapping, the fraction of time spent per cycle on the mapping step is a very small fraction of the total mapping speed. Visual and BEV feature extraction, which includes computation of image features and terrain estimation, accounts for 70–80% of the total mapping pipeline runtime for both approaches. In either case, both methods are capable of running at around $8-10hz$ (notwithstanding the ROS2 interface). We note that the voxel-based network, however, is considerably slower, and may not scale as effectively to larger map sizes.

	Image Feature Extraction	Feature Lifting	Mapping	BEV Feature	Inference
Voxel	26.3/25.6/27.0	2.8/2.6/3.9	8.5/8.1/11.2	12.3/12.0/13.6*	31.1/74.7/103.5
BEV	26.4/25.7/27.9	2.8/2.7/4.9	3.5/3.2/7.3	15.8/14.6/19.4	25.9/27.2/32.0

We provide additional implementation details that were omitted from the main paper.

Computing Behavior Labels

Computation of the behavior labels for $C_t, S_t$ require computation of BEV-space state visitation distributions and speed supervision. These must both be of the same width and weight of the behavior map $\Pi_t$. In order to generate these labels, we must have access to 1) the current robot state $x$, which must have a planar position $x,y=x_p$ and orientation $x_\theta$, 2) the mapping extent $\mathcal{M}$, and 3) optionally a footprint, defined as a set of positional offsets $\Delta$ to account for the robot occupying multiple BEV cells. Given this, we are able to generate labels by iterating through every state $x$ in the trajectory $\tau$, optionally expanding it into multiple states $x'$ via the footprint, computing the map index of every cell, and scattering the appropriate quantity into a $N_x \times N_y$ buffer. For state visitations, we scatter-sum ones for every state and normalize the buffer to sum to one. For speeds, we scatter-average the corresponding speed for every point $x'$.

Estimating the Terrain Surface

In order to properly compute BEV features from voxel grids, it is important to have an understanding of the height of the terrain surface for every cell in the BEV map. However, due to occlusions and noise, it is unreliable to simply assume that the minimum-height voxel in a column belongs to the terrain surface. As such, we follow the approach used in prior work [69, 38] and use a Markov random field to estimate the underlying terrain surface for our BEV features. We then use the estimated terrain surface to determine the diff feature and overhang threshold for every cell.

Computing Hindsight Labels

Since are able to derive BEV labels $B^*$ from any voxel map $V^*$, our hindsight labeling algorithm focuses on $V_t^*$, as $B_t^*$ can be obtained by running BEV feature extraction on $V_t^*$.

Prior work typically relies on producing $V_t^*$ by aggregating observations in a fixed time window, e.g. $[t-k,t+k]$. While this approach is straightforward, it can lead to suboptimal supervision if the speed of demonstrations varies significantly. For instance, if the robot is traveling very quickly, only a small number of frames in the window $[t-k,t+k]$ will actually fall within the mapping volume at time $t$. Conversely, if the robot is stationary, then all observations in $[t-k,t+k]$ will be the same and $V_t^*$ will degenerate to $V_t$.

To address this, we propose a simple two-pass algorithm for computing voxel supervision that is invariant to robot speed. For ease of presentation, we define two functions $V_t = \operatorname{voxelmap}(V_{t-1}, x_t, O_t, \mathcal{M}_{\mathrm{local}})$, which corresponds to running the voxel mapper on the current sensing data with a given local extent, and $\operatorname{crop}(V_t,\mathcal{M})$, which extracts the voxels from $V_t$ that lie within the mapping extent $\mathcal{M}$ (note that the cropping extent will differ from the extent of $V_t$).

At a high level, this algorithm works by computing a target time $t'$ for all times $t$ in the trajectory, where $t' \geq t$ is the last time at which the future mapping extent $\mathcal{M}_{t'}$ intersects the current mapping extent $\mathcal{M}_t$. This timestep is significant because the voxel grid $V_{t'+1}$ will share no voxels with $V_t$. Thus, $V_t^*$ only contains contributions from voxel grids $V_0$ to $V_{t'}$ (note that this discounts backtracks, however accounting for backtracks of arbitrary length is challenging because it requires perfect state estimation). The intuition for this is presented in Figure 11. Since this time interval is contiguous and monotonically increasing, we can re-purpose the original voxel mapper to compute $V_t^*$ by deriving it from $V_{t'}$. However, since $x_t$ and $x_{t'}$ are different states, we must use a larger mapping extent such that the new extent $\mathcal{M}'_{t'}$ at the target time $t'$ fully contains the original extent $\mathcal{M}_t$ at the current time $t$. Fortunately, since we know that $\mathcal{M}_{t'}$ intersects $\mathcal{M}_t$, it must be the case that a volume with triple the side length, centered at $x_{t'}$, fully contains $\mathcal{M}_t$. Given this, we can get $V_t^*$ by cropping $V_{t'}$ with extent $\mathcal{M}_t$. The detailed algorithm is presented in Algorithm 2. It is important to note that while this algorithm has $O(T^2)$ volume intersection operations, it maintains $O(T)$ voxel processing operations. This is the same asymptotic complexity as the methods used by prior work [57, 40] with respect to voxel mapping operations, which comprise nearly all of the total computation time. Once we obtain $V_t^*$ for every timestep in a trajectory, we then generate $B_t^*$ by running the BEV feature extractor on $V_t^*$.

Hyperparameters

Provided in this section are the various hyperparameters for our method (Tables VIII IX X XI). For all experiments, the voxel and BEV metadatas match in the $x$ and $y$ dimensions.

Algorithm 2: Voxel Hindsight Label Generation

Input:
    Trajectory τ = x₀:T,
    Observations O₀:T,
    Map Extent 𝓜local
Output:
    V*ₜ, ∀t

// First Pass
Tᵉⁿᵈ = [t, ∀t ∈ [0...T]]                         ◁ initialize end time for all timesteps
for t₁ ∈ [0...T] do
    for t₂ ∈ [t₁...T] do
        𝓜₁ = xₜ₁ + 𝓜local                              ◁ compute mapping volume at t₁
        𝓜₂ = xₜ₂ + 𝓜local                              ◁ compute mapping volume at t₂
        if 𝓜₁ ∩ 𝓜₂ ≠ ∅ then
            Tᵉⁿᵈₜ₁ := min(T, t₂ + 1)                  ◁ if the mapping volumes intersect, increase the end time for t₁
        end
    end
end

// Second Pass
V* = [∅, ∀t ∈ [0...T]]                              ◁ initialize output maps for all timesteps
𝓜′local = (𝓜ˡlocal − 𝓜ˡlocal, 𝓜ʳlocal, 3𝓜ˡlocal)     ◁ initialize new mapping extent with 3x size
V₋₁ = ∅                                               ◁ initialize empty voxel map
for t ∈ [0...T] do
    Vₜ = voxelmap(Vₜ₋₁, xₜ, Oₜ, 𝓜′local)              ◁ voxel map the current data with larger extent
    for [t′ s.t. t′ = Tₜ] do
        𝓜ₜ′ = xₜ′ + 𝓜local                             ◁ compute target mapping volume for t′
        V*ₜ′ = crop(Vₜ, 𝓜ₜ′)                            ◁ crop out voxels in the target volume
    end
end

Image not extracted

Parameter	Value
Mapping Origin ($\mathcal{M}_o$)	$-50\mathrm{m} \times-50\mathrm{m} \times-10\mathrm{m}$
Mapping Length ($\mathcal{M}_l$)	$100\mathrm{m} \times100\mathrm{m} \times20\mathrm{m}$
Mapping Resolution ($\mathcal{M}_r$)	$0.4\mathrm{m} \times0.4\mathrm{m} \times0.1\mathrm{m}$
Mapping Cell Counts ($\mathcal{M}_n$)	$250 \times250 \times200$
Input Image Shape	$848 \times448 \times3$

Parameter	Value
Voxel Encoder	3D ResNet Block $\times4$
BEV Encoder	ResNet Block $\times4$
Backbone	UNet Block $\times3$
VFM Decoder	ResNet Block $\times2$
Seg. Decoder	ResNet Block $\times2$
Geom. Decoder	ResNet Block $\times2$
Cost Decoder	ResNet Block $\times2$
Speed Decoder	ResNet Block $\times2$
Cost Range	$[0, 5]$
Cost N Bins	32
Speed Range	$[0, 15]\mathrm{m/s}$
Speed N Bins	32

Parameter	Value
Goal Weight ($K_2$, Eq. (3))	5.0
Vehicle Footprint	$4m \times2m$
Behavior Weight ($\lambda_1$, Eq. (4))	1.0
Inpaint Weight ($\lambda_2$, Eq. (4))	1.0
Geom Inpaint Weight ($\lambda_{\mathrm{geom}}$, Eq. (9))	1.0
VFM Inpaint Weight ($\lambda_{\mathrm{vfm}}$, Eq. (9))	1.0
Seg Inpaint Weight ($\lambda_{\mathrm{seg}}$, Eq. (9))	1.0
Speed Contrastive Weight ($\lambda_s$, Eq. (6))	0.05
Optimizer	Adam [34]

Parameter	Value
CVaR min/max	[0.0, 0.8]
CVaR vel min/max ($v_{\mathrm{low}}, v_{\mathrm{high}}$, Eq. (8))	[2, 5]$\mathrm{m}/\mathrm{s}$
CVaR step ($\epsilon$, Eq. (8))	0.02
Lat Acc. Max	$5\mathrm{m}/\mathrm{s}^{2}$