betaFeedback

WorksComputer Science

TACOS: Terrain-Aware COntact Supervision for Visual Footstep Planning

Abstract

Humans navigate their environment based on their vision system, anticipating how the ground will respond before committing to a step. Self-supervised learning could provide legged robots the same ability, but the supervisory signal is typically chosen on intuition, and its effect on what the model learns is rarely examined. We present Terrain-Aware COntact Supervision (TACOS), which proposes a meaningful and consistent footstep instability score, based on the residual of a factor graph in legged state estimation. It projects the continuous score into preceding images to supervise a dense visual predictor. We propose two variants, a hard score that exploits hindsight and a causal soft score available immediately. Both metrics are compared against state-of-the-art alternatives in two independent evaluations that share no data or labels. On 1716 manually annotated contacts, both variants outperform every evaluated alternative at detecting instability and ranking its severity, improving pooled severity ranking by 28 % over the strongest baseline. On three outdoor test runs, our metric also trains the most accurate visual predictor, improving rank correlation by 58 % over the strongest alternative, and by a further 25 % when supplemented with onboard context. In a real-world experiment over a 50 m forest trajectory, the trained predictor handles non-geometric terrain properties such as mud and vegetation, while stepping around geometric obstacles like fallen branches. Code is available for reproducibility at https://github.com/norlab-ulaval/TACOS.

Introduction

Quadruped robots demonstrate remarkable agility in complex environments [1, 2], enabling applications in search and rescue, forest monitoring, and planetary exploration. Exploiting this mobility in unstructured natural terrain requires selecting reliable footholds, whose behavior during contact cannot be determined from geometry alone [3]. A seemingly suitable surface may offer insufficient friction or deform under load, while vegetation can conceal exposed roots that trap a foot and cause a fall, as illustrated in Figure 1. Avoiding such failures requires anticipating how the foot will interact with the terrain before committing to a step [4]. Humans use visual cues to plan future steps, but such anticipation remains an open challenge for legged robots [5].

Quadruped robot traversing unstructured forest terrain, with examples of a stuck foot and fallen robot

Figure 1. Quadruped robot traversing unstructured forest terrain, where roots, branches, and leaves create candidate footholds whose supportiveness cannot be inferred from geometry alone. Close-ups show, on the right, the GoPro view of the foot getting stuck, wedged among exposed roots, and on the left, the resulting robot fall.

Most footstep planners rely on geometric representations [6], while reinforcement learning (RL), effective for proprioceptive locomotion, remains sample-inefficient on high-dimensional visual inputs [5]. Self-supervised learning offers an alternative by leveraging the robot’s own interactions with the terrain as supervision. However, the resulting visual representation depends directly on the quality of the supervisory signal. Obtaining scalable, physically meaningful labels for foothold instability is particularly challenging, since manual annotation is subjective [7] and terrain traversability is inherently robot- and task-dependent [5].

Existing self-supervised traversability methods commonly use body-level quantities, such as tracking error [8] or current consumption [9], which describe the robot’s overall response rather than individual footholds. Others instead used footstep-level supervision [3], gaining spatial precision but relying on fragile foot force-torque sensors rarely available on commercial platforms. Legged state estimation literature proposed methods to detect unreliable contacts from proprioception [10, 11], but uses them to correct odometry reactively rather than to anticipate instability from exteroception.

In this work, we propose Terrain-Aware COntact Supervision (TACOS), a self-supervised framework for predicting foothold instability from vision in unstructured terrain. Our key insight is that the rigid-contact assumptions underlying legged state estimation provide a natural measure of foothold instability. When a foot slips, sinks, or loses support, the resulting inconsistency appears in the residual of the corresponding contact constraint. We aggregate these residuals over each stance into a continuous per-footstep metric, in a hard variant that exploits the full estimation window and a soft variant available online. Projecting the observed footholds into preceding images yields sparse labels that supervise a dense visual predictor. We evaluate the metric against manually annotated contact instability, assess its suitability for visual learning on unseen trajectories, and demonstrate its use in closed-loop footstep planning.

In short, our contributions are :

  • A novel continuous per-footstep instability score derived from factor graph residuals, in a hard variant used as our reference label and a soft variant suitable for online use, both validated against manually annotated unstable contacts;

  • An analysis of the influence of the supervisory metric and of context conditioning on visual instability learning from sparse footstep labels;

  • A closed-loop demonstration of the learned visual instability predictor for footstep planning in unstructured forest terrain.

Legged State Estimation

Legged state estimation commonly exploits the assumption that a stance foot remains stationary relative to the terrain. Early approaches combined inertial measurements with leg kinematics and contact information to constrain the robot state and reduce inertial drift [12]. This principle was subsequently incorporated into invariant filtering [13] and smoothing formulations [10, 14], where individual stance contacts constrain the estimated robot motion relative to foothold locations. The reliability of these constraints depends on the validity of the assumed contact model. Slip, compliance, and terrain deformation can introduce disagreement between observed kinematics and the stationary-foot assumption. Contact-aware estimators therefore identify unreliable measurements and adjust their contribution to the state estimate [10, 11, 15]. Beyond their effect on localization, these inconsistencies provide observations of the underlying foot-terrain interaction, on which our instability score is based.

Foothold Instability Estimation

Contact instability is not directly observable and must be inferred from the robot’s response during stance. Although much of the literature focuses on slip detection, natural terrain also introduces sinkage and loss of support, motivating a broader assessment of foothold reliability. Force-based approaches characterize contact through measured or estimated Ground Reaction Forces (GRFs). Foothold robustness has been defined as the force a contact can sustain and predicted through haptic exploration [16]. Similar work has used internal force sensing to estimate reliable-contact probabilities for leg odometry [15]. For terrain characterization, Wellhausen et al. [3] derived a Ground Reaction Score (GRS) through dimensionality reduction of force-torque measurements from sensorized feet. Estimating GRFs from actuator torques removes the need for foot sensors, but introduces sensitivity to joint friction, backlash, and model errors [17]. Jenelten et al. [18] instead modeled the uncertainty of contact and slip with probabilistic state machines, whose slip probability expresses confidence in a slipping state, not its severity. More recently, OCELOT [11] combined force-based contact detection with a kinematic stationarity test. Their continuous metric indicates whether a foot is both loaded and stationary, connecting force sensing with motion consistency to quantify contact reliability for leg odometry.

Foothold instability can also be evaluated without force sensing. Early approaches compared each stance foot’s velocity with the median across stance feet, requiring at least three simultaneous contacts [19]. Comparing foot motion with its commanded trajectory removes this requirement and detects concurrent slips [20], but assumes planned foot trajectories, which modern reinforcement learning controllers do not provide. Other methods examine stance-foot motion through a robot state estimate. Using a proprioceptive Kalman filter, Yan et al. [21] derived a slip-likelihood score from each stance foot’s displacement relative to its touchdown location and its lateral velocity. By retaining a history of robot states, the invariant smoother of Yoon et al. [10] re-evaluates estimated foot velocities as new measurements become available, improving the identification of dynamic contacts. These approaches use deviations from expected contact behavior to identify unreliable support. Our method build on this principle by aggregating contact-model residuals over each stance into a continuous per-footstep instability score, providing localized supervision for visual instability prediction.

Self-Supervised Terrain Learning

Manual terrain annotation is expensive and difficult to scale across diverse natural environments [7]. Self-supervised approaches instead obtain supervision from the robot’s physical experience. Wild Visual Navigation (WVN) derives a traction metric from the discrepancy between commanded and measured body velocity [8], P-STEER uses the instantaneous current as an indicator of traversal effort [9], and ProNav combines actuator and battery signals into a low-dimensional representation [22]. Others instead learn a continuous embedding of the interaction, scoring terrain from operator preferences [23] or by proximity to a reference terrain [24] rather than from a physical quantity.

Self-supervision from robot experience is inherently sparse, since the robot probes the terrain only where it steps. All the aforementioned methods expand these sparse labels to larger image regions through learned segmentation or visual-feature grouping [8, 9, 24], or replace the egocentric image by footprint-sized patches [23]. Wellhausen et al. [3] instead associates each per-footstep GRS with its foothold location and projects it into preceding images, supervising only where feet physically touched the terrain. These association strategies trade supervision coverage for spatial fidelity. Extending labels to visually similar regions assumes comparable physical support, which may be unreliable in heterogeneous natural terrain [1, 8]. Sparse annotations have nonetheless been shown to support dense traversability prediction [7], motivating supervision without densification. We therefore retain supervision only at physically observed footholds [3], and examine how the choice of instability score affects the visual predictor’s ability to anticipate contact instability.

Methodology

We formulate foothold instability estimation as a self-supervised learning problem in which the robot’s interactions with the terrain provide supervision for visual perception. While walking, each leg alternates between stance, when its foot supports the robot, and swing, when it travels to the next foothold. Each stance is treated as a local probe of the terrain, i.e., measurements collected during the stance characterize the quality of the foot-terrain interaction. Meanwhile, the touchdown location identifies the corresponding terrain region.

Our methodology consists of three main steps. First, we derive a per-footstep instability score from factor graph contact residuals. Second, each measurement is associated with its foothold location and projected into synchronized camera images to generate sparse visual supervision. These two stages are performed offline, and together form the label-generation pipeline summarized in Figure 2. Finally, these sparse observations are used to train a dense visual predictor of foothold instability, as described in Section III-D.

Figure 2. Offline label-generation pipeline. (Left) Stances are extracted from the onboard contact detector, where F and H denote front and hind, and L and R left and right. (Center) Contact residuals di,kd_{i,k} from a factor graph leg odometry are aggregated over each stance into the instability score mi,hardm_{i,\mathrm{hard}}. (Right) Footstep placements are computed in the world frame using exteroceptive odometry and forward kinematics (FK), then projected into an image acquired before contact and rasterized into a sparse label mask.

Throughout this section, ii indexes stance events and kk indexes proprioceptive robot-state measurements, with SiS_i denoting the set of sample indices belonging to stance ii. Frames are denoted by calligraphic symbols, with W\mathcal{W}, B\mathcal{B}, and C\mathcal{C} representing the world, robot-base, and camera frames, respectively. We denote by BAT∈SE(3){}^{\mathcal{A}}_{\mathcal{B}}T \in SE(3) the rigid transformation from frame B\mathcal{B} to frame A\mathcal{A}, and by Ap{}^{\mathcal{A}}p a point expressed in frame A\mathcal{A}.

Contact Detection

Since foothold instability is computed only over samples assigned to a stance, contact-detection errors directly degrade label quality. We initialize stance intervals from the contact states provided on the ANYmal robot, which are estimated from foot kinematics and GRFs [17]. Each detected contact interval defines a stance ii, with samples SiS_i over which the instability score is computed, shown on the left of Figure 2. The raw contact signal occasionally contains short spurious transitions, which we remove with a morphological debounce filter. An opening step discards contacts shorter than 100 ms, and a subsequent closing step bridges gaps shorter than 25 ms. We also observed that some detected contacts extend beyond the physical end of stance and into foot liftoff. Since these samples no longer characterize the foot-terrain interaction, we refine the end of each stance offline. Within a 150 ms window preceding the detected liftoff, we search backward for the last sample whose GRF exceeds 50 N and truncate the stance there, discarding the unloaded tail. These four thresholds were determined empirically by inspecting stable and unstable stance windows in the recorded data. The resulting stance intervals are used for all subsequent computations.

Residual-based Instability Score

Given the detected stance intervals, we assess foothold instability through violations of the stationary-foot assumption underlying leg odometry. Following Dellaert et al. [25], we formulate proprioceptive odometry as a fixed-lag factor graph combining Inertial Measurement Unit (IMU) preintegration [26], bias estimation, and per-leg contact factors, solved incrementally using GTSAM [27]. Each keyframe kk contains the floating-base state Xk=(Rk,pk,vk)\mathbf{X}_k = (\mathbf{R}_k, \mathbf{p}_k, \mathbf{v}_k) and IMU bias bk\mathbf{b}_k, where Rk∈SO(3)\mathbf{R}_k \in SO(3) maps base-frame vectors to the world frame, and pk,vk∈R3\mathbf{p}_k, \mathbf{v}_k \in\mathbb{R}^3 denote the world-frame base position and velocity. For stance ii, the measured base-frame foot position Bsi,k{}^{\mathcal{B}}\mathbf{s}_{i,k} is obtained from joint positions through forward kinematics (FK).

We consider two variants of the instability score, hard and soft, both using the residual

ei,k=RkT(fi,k−pk)−Bsi,k,(1)\mathbf{e}_{i,k} = \mathbf{R}_k^{\mathsf{T}}(\mathbf{f}_{i,k} - \mathbf{p}_k) - {}^{\mathcal{B}}\mathbf{s}_{i,k}, \tag*{(1)}

where fi,k∈R3\mathbf{f}_{i,k} \in\mathbb{R}^3 denotes the world-frame foothold position. At touchdown, the foothold is initialized from the base pose and FK, with a weak prior of covariance σf2I3\sigma_f^2 \mathbf{I}_3, where σf=0.5\sigma_f = 0.5 m. A Huber kernel with standard δ=1.345\delta= 1.345 limits the influence of an unstable foothold on the estimated trajectory. We normalize the residual using its Mahalanobis norm

di,k=ei,kTΣct−1ei,k,Σct=σct2I3,(2)d_{i,k} = \sqrt{\mathbf{e}_{i,k}^{\mathsf{T}}\mathbf{\Sigma}_{\mathrm{ct}}^{-1}\mathbf{e}_{i,k}}, \qquad\mathbf{\Sigma}_{\mathrm{ct}} = \sigma_{\mathrm{ct}}^{2}\mathbf{I}_{3}, \tag*{(2)}

with σct=8\sigma_{\mathrm{ct}} = 8 cm selected empirically. The isotropic normalization makes the score rotation-invariant.

In the hard formulation, a single foothold state is shared across the stance, so fi,k=fi\mathbf{f}_{i,k} = \mathbf{f}_{i}. The residual therefore measures disagreement with a stationary foothold. We read each residual only as its stance leaves the fixed-lag window, once later measurements have refined the states it spans, as shown in the center of Figure 2. We aggregate these residuals using their maximum over the stance interval, yielding

mi,hard=max⁡{di,k∣k∈Si}.(3)m_{i,\mathrm{hard}} = \max\{d_{i,k} \mid k \in S_i\}. \tag*{(3)}

In the soft formulation, the shared foothold state is replaced by per-sample states linked by a random walk:

fi,k+1=fi,k+wi,k,wi,k∼N(0,QfΔtk),(4)\mathbf{f}_{i,k+1} = \mathbf{f}_{i,k} + \mathbf{w}_{i,k}, \qquad\mathbf{w}_{i,k} \sim\mathcal{N}\left(\mathbf{0}, Q_f \Delta t_k\right), \tag*{(4)}

where Δtk\Delta t_k is the sampling interval and Qf=σrw2I3Q_f = \sigma_{\mathrm{rw}}^{2}\mathbf{I}_3, with σrw2=0.04 m2/s\sigma_{\mathrm{rw}}^{2} = 0.04\ \mathrm{m}^{2}/\mathrm{s}, also selected empirically. Only the initial foothold state receives the touchdown prior. Allowing gradual foothold motion absorbs part of the discrepancy into the foothold state rather than the residual.

This formulation also supports causal evaluation by computing each residual immediately after the smoother update incorporating its sample, enabling online feedback on foothold stability. We reduce sensitivity to isolated peaks by aggregating the residuals over the completed stance using its 95th percentile, yielding

mi,soft=P95{di,k∣k∈Si}.(5)m_{i,\mathrm{soft}} = P_{95}\{d_{i,k} \mid k \in S_i\}. \tag*{(5)}

The hard metric is therefore the reference formulation, and the one we use to supervise the visual predictor, while the soft one is an instantaneous approximation of it, trading accuracy for online availability.

Image Association and Projection

Each instability measurement is associated with the terrain location probed during the corresponding stance, and projected into camera images acquired before touchdown. For this spatial association, we rely on the exteroceptive poses computed using FAST-LIVO2 [28], which remain unaffected by foot slippage and thus prevent instability events from shifting the projected footholds. The world-to-camera transformation for an image acquired at time tt is WCT(t)=BCT WBT(t){}^{\mathrm{C}}_{\mathrm{W}}\mathbf{T}(t) = {}^{\mathrm{C}}_{\mathrm{B}}\mathbf{T}\,{}^{\mathrm{B}}_{\mathrm{W}}\mathbf{T}(t), where BCT{}^{\mathrm{C}}_{\mathrm{B}}\mathbf{T} is the camera extrinsic calibration, obtained with the Kalibr toolbox. The touchdown location of stance ii is computed from the synchronized robot pose and FK foot position at touchdown, expressed in the world frame, and subsequently transformed into the camera frame. Corresponding pixel coordinates are then computed with the calibrated pinhole-camera model.

We retain projections that correspond to a touchdown occurring at most 10 s after the image acquisition, at a distance between 0.5 m and 3 m in front of the camera. Each foothold is rasterized as the image projection of a 6 cm circle, the diameter of the robot’s foot, lying flat on the ground at the contact location. When multiple projected footholds overlap, their instability scores are averaged.

Visual Instability Predictor

The projected footsteps provide sparse image-space supervision, while the objective is to predict foothold instability densely over the visible terrain. We use a Dense Prediction Transformer (DPT) [29] with a frozen DINOv3-Base encoder [30], as illustrated in Figure 3. The DPT architecture suits our task because it regresses a continuous value per pixel while preserving sharp boundaries, as in monocular depth estimation. Its decoder progressively fuses features from four intermediate layers into a single-channel, full-resolution prediction.

Visual instability predictor

Figure 3. Visual instability predictor. A frozen DINOv3-B encoder feeds a trainable DPT decoder that outputs a dense instability map. Camera roll and pitch (ϕ,θ)(\phi, \theta), and recent instability history (mˉf,mˉh)(\bar{m}_{f}, \bar{m}_{h}), enter as context tokens through GCA at each decoder stage. The loss is applied only at projected footstep locations.

We supplement the prediction head with the camera roll and pitch (ϕ,θ)(\phi, \theta) and a short instability history, averaged over preceding footsteps separately for the front (mˉf\bar{m}_{f}) and hind feet (mˉh\bar{m}_{h}). Each context value is normalized and embedded into its own learned token. These tokens provide the keys and values for Gated Cross-Attention (GCA) at each of the four DPT decoder stages, where image features act as queries, allowing each image location to weight the contextual inputs differently. The attention output is projected to match the decoder feature dimension, then added to the decoder features through a learned scalar gate. The gate is initialized to zero at each stage, so conditioning initially has no effect. We scale the learning rate of the GCA parameters by a factor of ten, so that the gates open within the training budget. During training, the encoder remains frozen, while the decoder and context conditioning modules are optimized.

The residual-based instability score exhibits a long-tailed distribution. We therefore apply a logarithmic transformation to the projected labels before training. Supervision loss is applied only to pixels covered by projected footsteps, while being ignored for all remaining pixels. We optimize the predictor using a masked Huber loss, which reduces the influence of large prediction errors compared with a squared-error loss while retaining a quadratic penalty near zero.

Footstep Planning

We integrate the visual predictor into a simple foothold adaptation planner running onboard. The low-level controller of ANYmal is a learned policy driven by body velocities, which offers no control over where the feet land. We therefore replace it with the contact-tracking policy of Atanassov et al. [31], which accepts desired contact positions as input and runs at 50 Hz.

Predictions are accumulated in a 5 m square costmap at 4 cm resolution, centered on the robot. Since the camera provides no depth, the terrain is approximated by a plane fitted to the recent touchdowns of the four feet. Each pixel of the predicted instability map is projected onto this plane, and then fused into the cell through an exponential moving average (EMA) weighted by the inverse distance to the robot.

At each swing phase, the gait generator produces a nominal foothold x0x_0. The candidate set C\mathcal{C} gathers every map cell within 25 cm of x0x_0 that lies in the leg workspace and covers only observed cells. We then command the foothold

x∗=arg⁡min⁡x∈C[cˉ(x)+w∥x−x0∥2],(6)x^{\ast} = \underset{x \in\mathcal{C}}{\arg\min}\left[\bar{c}(x) + w\lVert x-x_0\rVert^2\right], \tag*{(6)}

where cˉ(x)\bar{c}(x) is the worst cost of the map over the foot contact patch centered on xx and ww weights the deviation from the nominal foothold.

Experiments and Results

We evaluate the approach in three stages. We first validate the instability score against manually annotated contacts, then assess whether it can be predicted from images alone on unseen trajectories, and finally deploy the predictor on a robot for visual footstep planning.

Instability Score Evaluation

We collected a controlled dataset in three scenarios chosen to induce different forms of foothold instability, namely an ice rink partially covered with carpet, a pile of wood logs, and a slippery 45∘45^\circ slope. Our ANYmal-D robot carried two GoPro cameras directed toward its feet, highlighted in Figure 1, continuously recording the foot-terrain interactions. To limit labeling errors, two annotators inspected these recordings frame by frame and labeled 1716 footholds, of which 275 were considered unstable. Every method, including ours, was tuned on this dataset, with the PCA-based scores fitted on the same data, so that each is compared at its best achievable performance.

Both TACOS variants achieve higher overall instability-detection Average Precision (AP) than all evaluated baselines. Since unstable contacts account for only 16 % of the dataset, we report precision-recall (PR) curves, which remain informative under such class imbalance [32], and this proportion sets the random baseline. As shown in Figure 4, the hard and soft scores reach 0.96 and 0.93, respectively, compared with 0.90 for the method of Jenelten et al. [18]. The hard variant maintains near-perfect precision up to approximately 75 % recall and stays above every baseline over most of the recall range. The slight advantage of Yan et al. [21] at very high recall may reflect sensitivity to brief foot motion with little accumulated displacement, but comes after a larger loss of precision at lower recall. Since these scores serve as self-supervision labels, every mislabeled foothold

Figure 4. Precision-Recall curves for unstable-foothold classification on the human-labeled dataset, with each metric’s Average Precision (AP) in the legend. Our hard and soft residuals are compared against state-of-the-art alternatives. The dashed line marks the random baseline, set by the proportion of unstable footholds in the dataset.

propagates noise into training, making precision at high recall the decisive factor.

Beyond detection, we examine whether the residual magnitude also reflects the severity of an unstable contact. Annotators assigned each unstable foothold a severity of low, medium or high, according to the total foot displacement observed during stance. Figure 5 shows the distribution of the hard TACOS metric for stable footholds and for the three severity levels, together with a representative event for each severity level. Stable footholds concentrate tightly at low values, whereas the three unstable groups shift progressively upward as the annotated severity increases. Adjacent levels still overlap, which is expected given the subjectivity of the annotations. Table I quantifies this ordering through the Value Under the Surface (VUS), i.e., the probability that one contact drawn from each severity level is ranked in the correct order. Both TACOS variants again lead on every test run and on the pooled data, hard reaching 0.893 against 0.700 for Yan et al. [21] and 0.167 at random. The two variants are also complementary, as hard dominates on ice (0.942), where sliding is sustained, while soft leads on logs (0.880), where

Distribution of the proposed hard TACOS metric for stable footholds grouped by annotated instability severity

Figure 5. Distribution of the proposed hard TACOS metric for stable footholds grouped by annotated instability severity. The insets show representative events recorded by the feet-facing cameras for each severity. Stable footholds concentrate at low values, and the distributions shift upward at each severity level.

MethodLogsIceSlopePooled
ProNav [22]0.3410.3230.3120.342
GRS [3]0.4790.3390.2450.333
OCELOT [11]0.3130.4590.4400.400
Yan et al. [21]0.6930.7760.7730.700
Yoon et al. [10]0.7970.5820.7770.653
Jenelten et al. [18]0.7490.5260.7000.566
Soft TACOS0.8800.7090.9300.777
Hard TACOS0.8380.9420.9390.893

Table I. Value Under the Surface (VUS) per method, per test run and pooled over all runs. Higher is better; best per column in bold, second best underlined.

motion is abrupt. The baseline ordering differs from the detection results. While Jenelten et al. [18] was the strongest baseline at detection, it falls behind both Yan et al. [21] and Yoon et al. [10] at grading severity, indicating that detecting instability does not imply being able to grade it. The two lowest scores, GRS [3] and ProNav [22], both reduce proprioceptive features to a principal component, whose leading direction captures the dominant source of variance rather than instability itself, whereas the three strongest baselines are kinematic. This is consistent with our evaluation, which defines instability severity through observed foot displacement during stance. The TACOS metrics are kinematic as well, but read the contact residual directly from the factor graph rather than a velocity smoothed from it [10]. The residual accumulates constraint violation over the stance, providing a continuous measure that captures both the occurrence and the severity of instability.

Predicting Instability from Vision

Foothold instability can be inferred from visual appearance, and the choice of supervisory signal largely determines how well. We demonstrate this using the GrandTour dataset [33], collected with an ANYmal-D quadruped, from which we selected 23 outdoor runs in unstructured environments. We use the TierIV HDR camera feed, sampled every second, and discard the upper half of each image, as footholds lie below the horizon, for a total of 7543 images. To limit data leakage between training and validation samples, each run is divided into contiguous temporal segments, with 10 % kept for validation. Every quantitative result is reported over five cross-validation folds, which limits the influence of training stochasticity on the comparison. Three runs are withheld entirely as the test set, namely SBB-2, KAB-2 and ALB-3, providing evaluation on trajectories not observed during training or validation. The prediction head is trained with AdamW at a learning rate of 3×10−53 \times10^{-5} and weight decay of 0.01, under a cosine schedule with 100 warm-up steps, using bfloat16 precision, gradient clipping at 1.0 and an effective batch size of 252, for up to 100 epochs with early stopping on validation loss. Images are resized and randomly cropped to 512×704512 \times704 pixels and horizontally flipped for data augmentation.

Qualitative Evaluation – Figure 6 visually compares predictions from WVN [8], GRS [3], and our hard residual on two unseen images from GrandTour. We reproduce WVN by projecting its traction metric onto the robot’s traversed trajectory, without expanding the labels, and train it on our network. Each costmap is normalized between the 5th and 95th percentiles of its own training labels, so that methods with different label scales remain visually comparable. WVN produces a coarse, high-level costmap, as expected from body-level supervision, which spans the traversed region rather than a single foot contact and labels roughly 15 times more pixels per image. The two bottom predictors share the same per-footstep masks and differ only in the metric producing the labels, yet GRS yields a blurry and indecisive map, whereas hard TACOS resolves finer terrain structure, separating the wooden railway supports from the surrounding gravel and the forest path from the adjacent vegetation.

Network predictions on two unseen outdoor scenes for three supervisory signals

Figure 6. Network predictions on two unseen outdoor scenes for three supervisory signals, with the input image at the top. WVN is supervised at body level, whereas GRS and ours share the same per-footstep labels, while their metrics differ. Darker colors indicate higher predicted cost, each map being normalized between the 5th and 95th percentiles of its own training labels.

Quantitative Evaluation – Foothold selection is a ranking problem, so we compare the predicted and true instability values at every projected footstep of the test set through the Spearman rank correlation. Each label source carries its own units and range, so an absolute error such as the Mean

Absolute Error (MAE) would be dominated by label scale rather than by how well the predictor recovers the ordering. The Spearman rank correlation instead depends only on the monotone ordering of the predictions, not on the units or range of the labels. All predictors are trained identically on the same footstep masks, and differ only in the value assigned to each footstep. Hard TACOS trains the most accurate visual predictor, as presented in Figure 7, reaching a median Spearman correlation of ρ=0.554\rho= 0.554 over five-fold cross-validation. Soft TACOS is close second at ρ=0.449\rho= 0.449, still ahead of every metric from Section IV-A, showing that the causal approximation remains a strong supervisory signal. GRS [3] stays close to no correlation, as labels dominated by noise offer nothing to tie to appearance and drive the regressor toward predicting their average. Interestingly, this ordering closely follows the one measured against human annotations in Table I, although the two evaluations share no labels, data or procedure. The consistency of TACOS labels therefore translates into a sharper mapping between appearance and instability, whereas noisier metrics collapse it toward the mean.

Figure 7. Footstep-level Spearman correlation between predicted and true cost labels, per label source, over five-fold cross-validation.

Ablation Study – Visual appearance therefore carries real signal about instability, but it cannot convey the state of the robot at the moment of image acquisition, which the network must otherwise infer from the image. We quantify this through the context conditioning of Section III-D, ablating the camera attitude and the recent instability history in Table II. Attitude alone raises the pooled correlation by 4%4\%, while the instability history alone gains 25%25\%, indicating that instability is temporally consistent, as consecutive footholds tend to probe terrain of similar nature. Combining both adds only a marginal gain, consistent with the history already carrying most of that information. Attitude is therefore a cheap substitute when the residual cannot be computed onboard, since it comes directly from the IMU and still recovers a sizable part of the improvement.

ϕ/θ\phi/\thetamˉf,h\bar{m}_{f,h}SBB-2KAB-2ALB-3Pooled
––0.510±.0230.510_{\pm.023}0.501±.0200.501_{\pm.020}0.373±.0260.373_{\pm.026}0.557±.0070.557_{\pm.007}
✓–0.509±.0270.509_{\pm.027}0.522±.0290.522_{\pm.029}0.377±.0410.377_{\pm.041}0.580±.0150.580_{\pm.015}
–✓0.584±.0180.584_{\pm.018}0.633±.0160.633_{\pm.016}0.612±.0180.612_{\pm.018}0.694±.0070.694_{\pm.007}
✓✓0.589±.0220.589_{\pm.022}0.635±.0090.635_{\pm.009}0.613±.0170.613_{\pm.017}0.694±.0080.694_{\pm.008}

Table 2. Ablation of context conditioning with GCA. All configurations use images, and check marks indicate the added roll/pitch (ϕ,θ)(\phi, \theta) and instability history (mˉf,h)(\bar{m}_{f,h}). Spearman correlation between predictions and hard residual labels is reported per test run and pooled, as mean ±\pm standard deviation over 5 folds. Higher is better, with best values in bold and second best underlined.

Real-World Experiment

We finally deployed the trained predictor in the footstep planner of Section III-E, through a real-world experiment in moderate forest terrain. The same 50 m trajectory was executed twice, once in geometric mode and once with TACOS. Images were captured by an onboard Basler ace2 Pro camera, and model inference ran entirely onboard on an NVIDIA Jetson AGX Thor at 10 Hz. Figure 8 shows the environment together with the built-in geometric elevation map and the TACOS costmap built along the way. Geometry alone perceives the mudhole at Ⓐ as a smooth surface, which leads to foot sinkage, and treats the traversable vegetation at Ⓑ as an impassable barrier, marked by the red cross. The TACOS costmap instead captures both, and at Ⓒ it leads the robot to step around the fallen branches rather than onto them. While this demonstration remains limited in scope, it exemplifies the benefit of bringing terrain-aware contact supervision into footstep planning.

Real-world experiment in forest terrain, geometric costmap, and TACOS costmap

Figure 8. Real-world experiment in forest terrain. (Left) the geometric costmap (Center) and the TACOS costmap (Right). Corresponding locations mark Ⓐ, terrain associated with foot sinkage; Ⓑ, a vegetation barrier; and Ⓒ, fallen branches. The smaller photographs illustrate the robot’s interaction with these obstacles. Red and white curves show trajectories. The red cross marks where the geometric costmap incorrectly treats the traversable vegetation as an impassable barrier.

Conclusion

We presented TACOS, a self-supervised framework for predicting foothold instability from vision. It turns factor graph contact residuals into continuous per-footstep scores that supervise a dense visual predictor. The hard variant exploits the full estimation window, while the soft variant eval-

uates residuals causally as measurements arrive. Both outperform every evaluated alternative at detecting unstable contacts and at ranking their severity, improving pooled VUS by 28% over the strongest baseline. Hard TACOS also trained the most accurate visual predictor, leading rank correlation by 58%. These two evaluations share no data, labels or procedure, yet place the metrics in the same order. A closed-loop demonstration in forest terrain shows the value of bringing terrain-aware contact supervision into footstep planning where geometry alone does not capture the complexity of natural environments.

Limitations include the dependency on accurate contact detection, state estimation, and timing synchronization, which are challenging to achieve in practice. Future works will explore incorporating depth and geometry into the system, adopting probabilistic learning approaches such as adaptive binning, and enabling continual learning over successive deployments to improve the predictor’s performance over time.

Acknowledgment

This work was supported by the Canada Research Chair on Field Robotics (CRC-2023-00396), the NSERC project CRYOTIC (ALLRP 585289-23). The first author was further supported by NSERC CGS D under grant 600814. Special thanks to Vassil Atanassov for sharing the contact tracking policy used for the experiments in Section IV-C.

Generative AI (Claude, Anthropic) was used to assist in generating the plotting code for the data-driven result figures in Section IV; the underlying data, analysis, and figure content were produced and verified by the authors.

References

  1. [1]T. Miki, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science Robotics, vol. 7, no. 62, eabk2822, 2022.DOI
  2. [2]C. Qin et al., “A comprehensive review of quadruped robots: Vision perception, motion control, applications and challenges,” J. Autom. Intell., 2026.DOI
  3. [3]L. Wellhausen, A. Dosovitskiy, R. Ranftl, K. Walas, C. Caderna, and M. Hutter, “Where Should I Walk? Predicting Terrain Properties From Images Via Self-Supervised Learning,” IEEE RA-L, vol. 4, no. 2, pp. 1509–1516, 2019.DOI
  4. [4]S. Fahmi, V. Barasuol, D. Esteban, O. Villarreal, and C. Semini, “ViTAL: Vision-Based Terrain-Aware Locomotion for Legged Robots,” IEEE T-RO, vol. 39, no. 2, pp. 885–904, 2022.DOI
  5. [5]J. Frey et al., “Advances, challenges, and opportunities for legged robots,” Science Robotics, vol. 11, no. 116, eaee0787, 2026.DOI
  6. [6]F. Jenelten, T. Miki, A. E. Vijayan, M. Bjelonic, and M. Hutter, “Perceptive Locomotion in Rough Terrain – Online Foothold Optimization,” IEEE RA-L, vol. 5, no. 4, pp. 5370–5376, 2020.DOI
  7. [7]A. Schreiber, A. N. Sivakumar, P. Du, M. W. Gasparino, G. Chowdhary, and K. Driggs-Campbell, “W-RIZZ: A Weakly-Supervised Framework for Relative Traversability Estimation in Mobile Robotics,” IEEE RA-L, vol. 9, no. 6, pp. 5623–5630, 2024.arxiv.org/abs/2406.02822
  8. [8]M. Mattamala et al., “Wild Visual Navigation: Fast Traversability Learning via Pre-Trained Models and Online Self-Supervision,” Autonomous Robots, vol. 49, no. 3, p. 19, 2025.
  9. [9]A. E. Carvalho, D. Portugal, and P. Peixoto, “Self-Supervised Learning for Energy-Efficient Traversability Prediction in Natural Terrain Using Proprioception,” in IEEE ICARSC, 2026, pp. 236–241.DOI
  10. [10]Z. Yoon, J.-H. Kim, and H.-W. Park, “Invariant Smoother for Legged Robot State Estimation With Dynamic Contact Event Information,” IEEE T-RO, vol. 40, pp. 193–212, 2023.
  11. [11]E. Girgin and C. Kilic, “OCELOT: Odometry and Contact Estimation for Legged Robots,” arXiv preprint arXiv:2605.21863, 2026.
  12. [12]M. Bloesch et al., “State Estimation for Legged Robots - Consistent Fusion of Leg Kinematics and IMU,” Robotics, vol. 17, pp. 17–24, 2013.
  13. [13]R. Hartley, M. Ghaffari, R. M. Eustice, and J. W. Grizzle, “Contact-aided invariant extended Kalman filtering for robot state estimation,” IJRR, vol. 39, no. 4, pp. 402–430, 2020.
  14. [14]D. Wisth, M. Camurri, and M. Fallon, “Robust Legged Robot State Estimation Using Factor Graph Optimization,” IEEE RA-L, vol. 4, no. 4, pp. 4507–4514, 2019.DOI
  15. [15]M. Camurri et al., “Probabilistic Contact Estimation and Impact Detection for State Estimation of Quadruped Robots,” IEEE RA-L, vol. 2, no. 2, pp. 1023–1030, 2017.DOI
  16. [16]M. A. Hoepflinger, M. Hutter, C. Gehring, M. Blösch, and R. Siegwart, “Unsupervised identification and prediction of foothold robustness,” in IEEE ICRA, 2013, pp. 3293–3298.DOI
  17. [17]J. Hwangbo, C. D. Bellicoso, P. Fankhauser, and M. Hutter, “Probabilistic Foot Contact Estimation by Fusing Information from Dynamics and Differential/Forward Kinematics,” in IEEE/RSJ IROS, 2016, pp. 3872–3878.DOI
  18. [18]F. Jenelten, J. Hwangbo, F. Tresoldi, C. D. Bellicoso, and M. Hutter, “Dynamic locomotion on slippery ground,” IEEE RA-L, vol. 4, no. 4, pp. 4170–4176, 2019.DOI
  19. [19]M. Focchi, V. Barasuol, M. Frigerio, D. G. Caldwell, and C. Semini, “Slip Detection and Recovery for Quadruped Robots,” in Robotics Research: Volume 2, Springer, 2017, pp. 185–199.
  20. [20]Y. Nisticò, S. Fahmi, L. Pallottino, C. Semini, and G. Fink, “On Slip Detection for Quadruped Robots,” Sensors, vol. 22, no. 8, p. 2967, 2022.DOI
  21. [21]C. Yan, J. Qin, Q. Liu, and Q. Ma, “Slip Detection and Recovery for Quadruped Robots via Orthogonal Decomposition,” IEEE T-I E, vol. 72, no. 6, pp. 6166–6174, 2024.DOI
  22. [22]M. Elnoor, A. J. Sathyamoorthy, K. Weerakoon, and D. Manocha, “ProNav: Proprioceptive Traversability Estimation for Legged Robot Navigation in Outdoor Environments,” IEEE RA-L, vol. 9, no. 8, pp. 7190–7197, 2024.arxiv.org/abs/2307.09754
  23. [23]H. Karnan, Y. Yang, D. Farkash, G. Warnell, J. Biswas, and P. Stone, “STERLING: Self-Supervised Terrain Representation Learning from Unconstrained Robot Experience,” in CoRL, 2023.arxiv.org/abs/2309.15302
  24. [24]J. Hindel, S. Bultmann, H. Masnavi, D. Cattaneo, and A. Valada, “Self-Supervised Online Robot-Agnostic Traversability Estimation for Open-World Environments,” arXiv preprint arXiv:2605.28442, 2026.arxiv.org/abs/2605.28442
  25. [25]F. Dellaert, C. Noh, V. Agrawal, and A. Kim, “Four Simple Proprioceptive Estimators for Legged Robots,” arXiv preprint arXiv:2605.23100, 2026.arxiv.org/abs/2605.23100
  26. [26]C. Forster, L. Carlone, F. Dellaert, and D. Scaramuzza, “IMU preintegration on manifold for efficient visual-inertial maximum-a-posteriori estimation,” in RSS XI, 2015.DOI
  27. [27]F. Dellaert and G. Contributors, Borglab/gtsam, version 4.2a8, May 2022.
  28. [28]C. Zheng et al., “FAST-LIVO2: Fast, Direct Li-DAR–Inertial–Visual Odometry,” IEEE T-RO, vol. 41, pp. 326–346, 2024.arxiv.org/abs/2408.14035
  29. [29]R. Ranftl, A. Bochkovskiy, and V. Koltun, “Vision Transformers for Dense Prediction,” in IEEE/CVF ICCV, 2021, pp. 12 159–12 168.arxiv.org/abs/2103.13413
  30. [30]O. Simeoni et al., “DINOv3,” arXiv preprint arXiv:2508.10104, 2025.arxiv.org/abs/2508.10104
  31. [31]V. Atanassov, W. Yu, S. Gangapurwala, J. Wilson, and I. Havoutis, “GeCCo–a Generalist Contact-Conditioned Policy for Loco-Manipulation Skills on Legged Robots,” arXiv preprint arXiv:2509.17582, 2025.arxiv.org/abs/2509.17582
  32. [32]T. Saito and M. Rehmsmeier, “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets,” PLoS one, vol. 10, no. 3, e0118432, 2015.DOI
  33. [33]T. Tuna et al., “GrandTour: A Legged Robotics Dataset in the Wild for Multi-Modal Perception and State Estimation,” arXiv preprint arXiv:2602.18164, 2026.arxiv.org/abs/2602.18164

Paper details

vlnc:W_j2w84f7w
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Claude

View statement
Looking for related papers…
Contents