betaFeedback

WorksComputer Science

PROOF: Perceptive Robustness via Out-of-Distribution Fallback for Legged Locomotion

Abstract

Vision-based perceptive locomotion has enabled legged robots to achieve agile and terrain-aware locomotion across complex environments. However, the reliability of such systems heavily depends on visual observations, which are inherently susceptible to occlusion, reflective surfaces, and illumination variations, often resulting in catastrophic locomotion failures under degraded perception conditions. In this paper, we present PROOF (Perceptive Robustness via Out-of-Distribution Fallback for Legged Locomotion), a perceptive locomotion framework that enables robust locomotion under severe visual disturbances. Our method adopts a two-stage teacher-student learning paradigm that explicitly models perception reliability: the teacher learns both perception-driven and proprioceptive fallback behaviors, while the student predicts exteroceptive features with uncertainty and suppresses them under out-of-distribution inputs. We evaluate our method on a Unitree Go2 quadruped equipped with a RealSense D435 depth camera in real-world settings. Under normal visual conditions, our method enables the robot to successfully perform high-dynamic parkour maneuvers, including traversing large gaps, elevated platforms, and narrow beam terrains. Under severe visual degradation, such as heavy occlusion and strong illumination changes, the robot continues to maintain stable and robust locomotion performance.

PROOF enables agile perceptive robot parkour under challenging real-world conditions

Figure 1. PROOF enables agile perceptive robot parkour under challenging real-world conditions. The robot traverses large platforms and discontinuous obstacles while remaining reliable under severe visual degradation. When exteroceptive observations become unreliable, PROOF suppresses corrupted visual features and falls back to proprioceptive locomotion behaviours.

Introduction

Perceptive locomotion has significantly advanced the mobility of legged robots in complex environments [1, 2, 3]. By incorporating exteroceptive sensors such as depth cameras and LiDAR, legged robots can proactively perceive surrounding terrain geometry, enabling highly dynamic and agile behaviors including stair climbing, gap traversal, and parkour maneuvers [4, 5, 6, 7, 8, 9, 10]. Compared to traditional proprioceptive locomotion policies, perceptive locomotion provides substantially improved terrain awareness and maneuverability. However, the performance of such systems heavily depends on the reliability and accuracy of exteroceptive observations, making perception both the key enabler of agile locomotion and a major source of system vulnerability.

In real-world environments, visual perception is highly susceptible to severe environmental disturbances like sensor occlusion, illumination overexposure, reflective surfaces, motion blur and so on. Such disturbances introduce substantial out-of-distribution (OOD) shifts in visual inputs. For perceptive end-to-end locomotion policies that fully rely on exteroceptive information, corrupted visual observation directly misleads the policy’s action, ultimately resulting in unstable locomotion, collisions, or catastrophic failures.

Existing works have attempted to improve the robustness of perceptive locomotion through domain randomization, sensor dropout, adversarial perturbation training, and privileged learning mechanisms[2, 11, 12]. While these approaches improve robustness against perceptual noise and mild distribution shifts, they largely assume that exteroceptive observations remain at least partially informative. Under severe perception degradation, however, visual inputs may become fundamentally unreliable, and continuously relying on corrupted exteroceptive features can further amplify system failure. Therefore, the core challenge is not merely improving robustness to noisy perception, but enabling graceful degradation under catastrophic perception failures.

To address this challenge, we propose PROOF (Perceptive Robustness via Out-Of-distribution Fallback), a novel two-stage perceptive locomotion framework that efficiently learns agile parkour skills while maintaining resilience against severe visual perception disturbances. To improve robustness against unreliable visual observations, we introduce a two-stage teacher–student framework with uncertainty-aware exteroceptive gating. The privileged teacher is trained with terrain elevation map under random exteroceptive masking, enabling it to learn both perception-driven agile behaviors and proprioception-based fallback locomotion. The resulting behavior is distilled into a depth-vision student, where a recurrent estimator predicts latent exteroceptive features and their uncertainty. This uncertainty serves as an input of an adaptive gate that suppresses corrupted visual inputs and falls back to proprioceptive control under OOD perception. Finally, we validate our method on a real-world Unitree Go2 equipped with a single front-facing RealSense D435 camera, demonstrating agile parkour maneuvers under clean vision and stable locomotion under severe visual degradation.

The contributions of this work are summarized as follows:

  • We present PROOF, a perceptive locomotion framework that learns agile quadruped parkour under reliable vision and falls back to proprioception when depth becomes unreliable.

  • We introduce a dual-guard fallback mechanism that detects OOD visual disturbances via latent uncertainty estimation and switches to proprioceptive control when vision becomes unreliable.

  • We validate PROOF on a real-world Unitree Go2 hardware with a single RealSense D435, showing robust obstacle traversal and failure avoidance under extreme visual corruptions.

Related Work

Learning-Based Legged Locomotion

Deep reinforcement learning has emerged as the leading paradigm for legged locomotion, allowing quadruped robots to learn motor skills in simulation and then deploy the trained policies onto physical robotic hardware [13, 14, 15]. Compared to traditional model-based control pipelines, reinforcement learning demonstrates remarkable flexibility and robustness across diverse locomotion scenarios, enabling legged system works across various scenarios like agile locomotion, navigation, force adaptive interaction, motion tracking, etc.[16, 17, 18, 19, 20, 21, 22, 23, 10, 9].

Perceptive Legged Locomotion

To enable legged robots to traverse unknown and dynamic environments, a large body of work focuses on proprioceptive locomotion, where policies infer terrain properties and robot dynamics from onboard available proprioception without explicit perception input [24, 16, 25, 26, 17, 27]. These methods achieve impressive robustness on most uneven terrains, but they only react after terrain changes are reflected in contact or body motion, which limits their ability to prepare for obstacles that require anticipatory foothold or body adjustment.

Perceptive locomotion addresses this limitation by incorporating exteroceptive observations such as depth images, reconstructed elevation map and so on. [2] learns an attention-based recurrent controller that fuses proprioception and exteroception for robust outdoor locomotion. Subsequent works improve visual policy learning through end-to-end cross-modal architectures [1, 3, 28, 4, 5, 29, 8, 6, 7]. These methods demonstrate that visual perception can substantially expand traversability with lightweight hardware integration, but typically rely on the quality of alignment for visual features between sim and real. This visual sim-to-real gap may be caused by differences in scene richness and imaging mechanisms and unmodeled environmental noise. As a result, corrupted or OOD visual observations directly influence robot’s action. Hence, our work focuses on making perceptive locomotion more robust to visual perception, especially for visual degeneration which leads to catastrophic system failure, which is crucial for a safe working robot system.

Method

As shown in Figure 2, the overall pipeline consists of an elevation-map teacher policy, which is subsequently distilled into a deployable depth-vision student policy. Importantly, we make the teacher and distillation procedure aware of out-of-distribution situations and therefore use a gating mechanism to let the student fallback to safe perception.

Figure 2. Our framework consists of two parts: a teacher policy, trained using elevation maps in simulation; and a student policy, trained using depth vision and teacher-student distillation.

Parkour as Velocity Tracking

We formulate agile parkour locomotion as a velocity-tracking problem, where the robot follows commanded planar velocities while interacting with diverse terrain structures. This formulation provides a unified objective across locomotion behaviors, while allowing terrain geometry to implicitly induce complex skills such as gap traversal, beam walking, and platform climbing.

Reward Design The teacher is trained with a set of velocity-tracking locomotion rewards consisting of (i) velocity tracking terms, (ii) regularization terms on motion smoothness and energy, and (iii) additional parkour-specific terms. More details are provided in Table 3 of the Appendix.

Reward termsWeightFunction
Task rewards
Linear velocity xy tracking1.51.5exp⁡(−((vxcmd−vˉx)2+(vycmd−vy)2)/0.25)\exp\left(-\left((v_x^{\mathrm{cmd}}-\bar{v}_x)^2+(v_y^{\mathrm{cmd}}-v_y)^2\right)/0.25\right), where vˉx=min⁡(vx,vxcmd+0.1)\bar{v}_x=\min(v_x,v_x^{\mathrm{cmd}}+0.1) on parkour terrains and vˉx=vx\bar{v}_x=v_x otherwise.
Angular velocity z tracking1.01.0exp⁡(−(ωzcmd−ωz)2/0.25)\exp\left(-(\omega_z^{\mathrm{cmd}}-\omega_z)^2/0.25\right).
Regularization rewards
Vertical velocity−1.0-1.0vz2v_z^2.
Roll/pitch angular velocity−0.05-0.05∥ωxy∥22\lVert\omega_{xy}\rVert_2^2.
Hip-joint deviation−0.2-0.2∥qhip−qhipo∥22\lVert q_{\mathrm{hip}}-q_{\mathrm{hip}}^o\rVert_2^2, where qhipoq_{\mathrm{hip}}^o is nominal joint position.
Joint deviation−2.5×10−3-2.5\times10^{-3}∥q−qo∥22\lVert q-q^o\rVert_2^2, where qoq^o is nominal joint position.
Joint acceleration−2.5×10−7-2.5\times10^{-7}∥q¨∥22\lVert\ddot{q}\rVert_2^2.
Joint torque−1.0×10−4-1.0\times10^{-4}∥τ∥22\lVert\tau\rVert_2^2.
Joint velocity−2.0×10−4-2.0\times10^{-4}∥q˙∥22\lVert\dot{q}\rVert_2^2.
Joint power−2.0×10−5-2.0\times10^{-5}∥τ⊙q˙∥1\lVert\tau\odot\dot{q}\rVert_1.
Joint-power variance−2.0×10−6-2.0\times10^{-6}Var⁡(τ⊙q˙)\operatorname{Var}(\tau\odot\dot{q}), this is to discourage overusing certain joint motors.
Action rate−0.01-0.01∥at−at−1∥22\lVert a_t-a_{t-1}\rVert_2^2.
Foot air-time0.50.5∑f(tfair−0.5)1f,first\sum_f(t_f^{\mathrm{air}}-0.5)1_{f,\mathrm{first}} under nonzero commands.
Safety and feasibility rewards
Alive bonus1.01.011 if the episode is not terminated.
Joint power limit−10.0-10.01{max⁡j∣τjq˙j∣>300 W}1\{\max_j\lvert\tau_j\dot{q}_j\rvert>300\ \mathrm{W}\}.
Joint position limit−10.0-10.0Soft joint-limit violation distance.
Foot stumble−0.5-0.51{∥Fxyf∥>4∣Fzf∣}1\{\lVert F_{xy}^f\rVert>4\lvert F_z^f\rvert\} for any foot, which is to prevent foot tips from hitting vertical surfaces.
Undesired contacts−1.0-1.0Count of non-foot bodies with contact force above 1 N1\ \mathrm{N}.
Stuck penalty−1.0-1.01{∣vx∣<0.1,∣vxcmd∣>0.1}1\{\lvert v_x\rvert<0.1,\lvert v_x^{\mathrm{cmd}}\rvert>0.1\} plus backward motion under forward command.
Terrain Edge penalty−1.0-1.0∑i∈P∥di∥∥vi∥\sum_{i\in\mathcal{P}}\lVert d_i\rVert\lVert v_i\rVert for foot monitoring-point penetration into virtual edge obstacles.

Table 3. Reward terms used for teacher training.

Conventional velocity-tracking task reward requires the robot to follow the exactly commanded speed and penalizes deviations from it. During parkour tasks, however, the robot may briefly accelerate beyond the command to build momentum for takeoff, which conflicts with this strict tracking objective. To enable dynamic maneuvers such as jumping over obstacles, we relax the linear velocity tracking objective along the forward (xx) direction on parkour terrains, and allow the robot to temporarily exceed the target speed to build sufficient momentum for takeoff. The relaxed velocity tracking reward is defined as:

rv=exp⁡(−(vxcmd−vˉx)2+(vycmd−vy)2σ2),vˉx={min⁡(vx,vxcmd+δ),parkour terrains,vx,otherwise.(1)r_v = \exp\left(-\frac{(v_x^{\mathrm{cmd}}-\bar{v}_x)^2+(v_y^{\mathrm{cmd}}-v_y)^2}{\sigma^2}\right), \qquad \bar{v}_x = \begin{cases} \min(v_x,v_x^{\mathrm{cmd}}+\delta), & \text{parkour terrains},\\ v_x, & \text{otherwise}. \end{cases} \tag*{(1)}

This modification encourages aggressive yet controlled acceleration before takeoff, while retaining standard tracking behavior on normal terrains.

Teacher Policy with Elevation Map-based Perception

Teacher Training Design The teacher policy uses elevation maps as exteroceptive observations, which provide accurate local terrain description while remaining low-cost to simulate. We adopt an asymmetric actor-critic framework [30] to train the teacher policy. The actor operates at 50 Hz and adopts a two-branch architecture. The proprioceptive branch encodes a 10-frame history otpo_t^p into ztp=Ep(otp)z_t^p = E_p(o_t^p), where otpo_t^p consists of base angular velocity, projected gravity, velocity command, joint positions and velocities, and previous actions. The exteroceptive branch encodes the current local elevation map hth_t into zte=Ee(ht)z_t^e = E_e(h_t). The resulting latent features are used by the policy to generate locomotion actions. While the critic is provided with task-related privileged observations, including base linear and angular velocities, projected gravity, velocity command, joint positions, velocities, efforts, and accelerations, previous actions, foot contact flags, foot heights, and the elevation map. The actions are treated as target joint angles, which are then converted into torque signals by a high-frequency low-level PD controller. In simulation, we adopt a delayed implicit actuator model in Isaac Lab.

Since base linear velocity is critical for velocity tracking but unavailable on the real robot, the first three dimensions of ztpz_t^p are supervised to estimate it [25]. The action is atT=πT(ztp,mtztv)a_t^T = \pi_T(z_t^p, m_t z_t^v), where mtm_t masks the exteroceptive latent. The teacher policy is updated by Proximal Policy Optimization (PPO) [31]:

LT=LPPO+λvMSE⁡(v^,v)(2)\mathcal{L}_T = \mathcal{L}_{\mathrm{PPO}} + \lambda_v \operatorname{MSE}(\hat{v}, v) \tag*{(2)}

To discourage unstable foot-ground interactions near terrain edges, we detect sharp edges on gap and beam terrains and instantiate virtual cylindrical obstacles around them, similar to the virtual-obstacle design in [32]. Four monitoring points are attached to each foot (Figure 4). For each point i∈Pi \in\mathcal{P}, let di\mathbf{d}_i be its penetration offset into an edge obstacle and vi\mathbf{v}_i be its linear velocity in the world frame. Following [28], we define the edge penalty as

Virtual edge penalty illustration

Figure 4. Virtual edge penalty illustration. Red cylinders denote auto-generated virtual obstacles, and green points attached to each foot tip denote virtual monitoring points.

redge=−∑i∈P∥di∥⋅∥vi∥,(3)r_{\mathrm{edge}} = -\sum_{i \in\mathcal{P}} \lVert\mathbf{d}_i \rVert\cdot\lVert\mathbf{v}_i \rVert, \tag*{(3)}

This term assigns larger penalties to high-speed impacts and scraping motions near obstacle edges, encouraging the policy to select safer footholds.

Student Policy with Depth Camera and Uncertainty Fallback

Student Training with Out-of-Distribution Awareness. We distill the elevation-map teacher into a deployable depth-vision student using online DAgger-style supervision [33], as shown in Figure 2. The student keeps the teacher’s frozen proprioceptive encoder and learns to replace the teacher’s elevation-map latent with a depth-based estimate. Concretely, a CNN encodes two delayed 10 Hz depth frames into z~tv\tilde{z}_t^v, and a GRU predicts the exteroceptive latent mean z^te\hat{z}_t^e from [ztp,ztv][z_t^p, z_t^v].

Uncertainty as OOD signal. To assess the reliability of the predicted exteroceptive features, the student models the latent ztez_t^e as a Gaussian distribution with mean z^te\hat{z}_t^e and diagonal log-variance ℓt\ell_t. A separate head predicts log-variance ℓt\ell_t from sg⁡([z^te,ztv,ztp])\operatorname{sg}([\hat{z}_t^e, z_t^v, z_t^p]), where sg⁡[⋅]\operatorname{sg}[\cdot] denotes stop-gradient. The model is trained with a negative log-likelihood (NLL) objective:

LNLL=12D∑i=1D((zt,ie−z^t,ie)2exp⁡(−ℓt,i)+ℓt,i).(4)\mathcal{L}_{\mathrm{NLL}} = \frac{1}{2D}\sum_{i=1}^{D}\left((z_{t,i}^e-\hat{z}_{t,i}^e)^2\exp(-\ell_{t,i})+\ell_{t,i}\right). \tag*{(4)}

This objective encourages input-dependent uncertainty: when the depth observation is informative and consistent with the teacher latent, the model minimizes error with small variance; when the observation is ambiguous or corrupted (e.g., under OOD visual disturbances), the model can reduce penalty by increasing ℓt\ell_t. As a result, the predicted log-variance serves as a learned proxy for epistemic unreliability of exteroception, capturing both explicit corruption and intrinsic perceptual ambiguity.

The student is trained jointly with action imitation and latent regression:

LS=λaMSE⁡(atS,atT)+λnLNLL(z^te,ℓt,zte,T)+λgBCE⁡(g~t,mt),(5)\mathcal{L}_S = \lambda_a \operatorname{MSE}(a_t^S, a_t^T) + \lambda_n \mathcal{L}_{\mathrm{NLL}}(\hat{z}_t^e, \ell_t, z_t^{e,T}) + \lambda_g \operatorname{BCE}(\tilde{g}_t, m_t), \tag*{(5)}

where the gate term g~t\tilde{g}_t will be introduced in the next paragraph.

Fallback Mechanism. Our framework constructs a robust fallback behavior through two complementary supervision pathways: (1) Teacher-induced action-level fallback. During training, we inject corrupted depth bursts and align them with an exteroceptive mask mtm_t. When mt=0m_t = 0, the teacher is queried with its exteroceptive latent masked, producing a proprioceptive-only action. This provides the student with explicit supervision of how to act without vision, ensuring that a valid fallback policy is learned at the action level. Simultaneously, the corresponding teacher latent zte,Tz_t^{e,T} is set to zero, encouraging the student’s vision branch to represent the absence of reliable exteroception.

(2) Uncertainty-aware gating for fallback activation. Fitting the exteroceptive latent to zero is not by itself a clean fallback signal: non-zero residual values of z^te\hat{z}_t^e may still affect the policy. We therefore introduce an extra gate that explicitly suppresses exteroception. A gate head maps sg⁡([ztp,ℓt,z^te])\operatorname{sg}([z_t^p,\ell_t,\hat{z}_t^e]) to a raw score g~t∈(0,1)\tilde{g}_t \in(0,1), trained by binary cross-entropy BCE⁡(g~t,mt)\operatorname{BCE}(\tilde{g}_t,m_t) so that the target is 1 on clean observations and 0 under labeled corruption. Simply smoothing this score and multiplying it onto the latent would expose the actor to intermediate latent values. During teacher training, the exteroceptive latent is either fully present or exactly zero, the actor never sees values in between. While, a hard mask in {0,1}\{0,1\} stays consistent, but the binary gate value might flip between the two states and disturb the policy. Appendix Table 12 therefore tests the gate as a binary classifier at a threshold of 0.5, and finds that the gate behaviour is acceptably stable. Hence we adopt the hard mask with a hysteresis: a closed gate opens only when g~t≥0.8\tilde{g}_t \ge0.8, and an open gate closes when g~t<0.5\tilde{g}_t < 0.5. The resulting binary mask gt∈{0,1}g_t \in\{0,1\} enters the policy as

atS=πS([ztp,gtz^te]).(6)a_t^S = \pi_S([z_t^p,g_t\hat{z}_t^e]). \tag*{(6)}

Additional Design Details

Terrain Settings As shown in Figure 3, the training terrains include normal uneven terrains, high platforms, gaps, beams, and other parkour obstacles. We organize each terrain type into 10 difficulty levels and apply a game-inspired terrain curriculum [15]. The key terrain ranges are: climb height 0.20–0.70 m, gap width 0.05–0.80 m, stair height 0.02–0.22 m, and slope magnitude 0.00–0.45. At the end of each episode, the robot is moved to a harder level if it traverses the current terrain, moved to an easier level if it fails to make sufficient progress, and randomly reassigned after solving the highest level. We found that if we spawn robots on both challenging parkour and non-parkour terrains, the policy will struggle to learn proper trot gait. Hence we ”warm up” the policy learning on non-parkour terrains first for 1500 iterations, then learns on all terrains.

Training terrain library used in our experiments

Figure 3. Training terrain library used in our experiments. From left to right and top to bottom, the subterrains correspond to: (a) beams, (b) gap, (c) gap with narrow stages, (d) high platform, (e) uphill slope, (f) downhill slope, (g) rough plane, (h) ascending stairs, and (i) descending stairs.

Depth Image Processing The student observes the terrain through a forward-facing RealSense D435 depth camera, simulated in Isaac Lab with NVIDIA Warp ray casting and a pinhole camera model. As shown in Figure 5, depth images are clipped to [0.1, 2.0] m and preprocessed before being fed to the policy. To reduce the sim-to-real gap, we randomize camera extrinsics, perturb ray origins and directions, and inject minimum- or maximum-depth artifact patches that approximate blind spots and reflective failures. To simulate the depth stream latency, we apply 0.05 s depth frame latency considering the communication cost on real robot hardware. Additional simulation details are provided in the Appendix.

Depth preprocessing used to align simulated ray-caster depth with RealSense observations

Figure 5. Depth preprocessing used to align simulated ray-caster depth with RealSense observations.

Experiments

In this section, we evaluate the effectiveness of the proposed framework under corrupted visual observations and real-world parkour agility. Both the teacher and student policies are trained utilizing NVIDIA Isaac Lab [34] for high physical fidelity. The teacher and student training take approximately 4.5 hours on a single NVIDIA RTX 4090 GPU to guarantee a robust deployable depth-vision-based policy (teacher for 8000 iterations and student for 4000 iterations). Specifically, we aim to answer the following questions: 1) Does the predicted log-variance reflect visual uncertainty? 2) Is PROOF robust to OOD visual disturbances? 3) Does PROOF achieve strong real-world parkour performance?

Behaviour of the learned uncertainty

Figure 6 separates reconstruction uncertainty from the fallback decision. In the gap example, the depth stream is not externally corrupted, yet log-variance rises during the flight phase because the relevant edge geometry rapidly leaves the camera’s view and the exteroceptive latent becomes harder to infer. In the stair example, severe image degradation causes the gate to close, preventing the corrupted visual features from entering the policy. These results indicate that the uncertainty head is not merely an auxiliary loss, but provides a reliability signal that can modulate the student policy.

Visualization of the learned uncertainty signal

Figure 6. (a) Log-variance during gap traversal. (b) Gate response to depth corruption.

Figure 6: Visualization of the learned uncertainty signal. (a) On gap terrains, the predicted mean log-variance remains low before takeoff, rises sharply while the robot is airborne over the gap, and decreases again after landing. (b) On stair terrain (as an example of non-parkour terrain), the gate remains open for reliable depth observations and quickly closes when the input is corrupted by strong random noise, large depth artifacts, or full-frame masking.

Robustness against OOD visual disturbances

We compare PROOF with two baselines under randomized visual disturbances injected during evaluation. Baseline 1 removes exteroceptive masking in teacher training, so the teacher does not learn a proprioceptive fallback behavior. Baseline 2 keeps the same teacher masking and corruption schedule, but removes the variance head, the NLL term, and the gate. It regresses the exteroceptive latent with MSE and feeds that latent directly to the actor. Since severe visual disturbances make terrain geometry required for anticipatory parkour unrecoverable, this evaluation measures locomotion on the non-parkour terrains in Figure 3. All policies are evaluated at terrain level 5, i.e., the middle difficulty level of the curriculum. During rollouts, the injected corruptions are stronger forms of the three identified patterns in training: full-frame depth blocking, large rectangular artifacts, and global random noise (Figure 6(b)).

MethodRoughFlatUphillDownhillUpstairsDownstairs
Baseline 1: w/o teacher mask0.28 / 00.11 / 00.35 / 00.15 / 00.22 / 0
Baseline 2: MSE latent distillation0.83 / 960.76 / 790.75 / 840.69 / 730.71 / 91
PROOF0.86 / 1000.79 / 990.78 / 1000.72 / 950.93 / 100

Table 1. Simulation robustness under randomized strong visual corruption on non-parkour terrains. Each terrain cell reports Rvx/SR_{vx}/S, where RvxR_{vx} is the mean x-direction velocity tracking reward and SS is the traversal success rate. A trial is counted as successful if the robot traverses the current terrain segment, following the same terrain evaluation logic as the training curriculum. For each method and terrain type, evaluation uses 100 parallel environments and 100 test episodes.

Without teacher masking, the student has no reliable fallback target when depth observation becomes out-of-distribution, so Baseline 1 fails under strong visual corruption. Baseline 2 already inherits the masked teacher’s fallback behaviour and therefore remains close to PROOF.

Beyond the three corruption patterns seen in training, we further test the gate behaviour on two held-out patterns: randomly oriented stripe masks and reflection-like elliptical artifacts (Appendix 12). The gate rejects these seen and unseen corruptions while remaining open on clean vision during parkour.

Additionally, the fallback robustness test on hardware under controlled visual corruptions is performed. The designed visual corruptions are physical occlusion, flashlight overexposure, reflection from a waving mirror, coded EMA motion blur, depth dropout, and depth noise, applied on flat ground and on stairs at commanded forward speeds of 0.0, 0.5, and 1.0 m/s (Appendix Table 13). Each velocity-terrain pair is tested for five trials. PROOF maintains normal locomotion in nearly all of these conditions without any failure. The remaining failures are strong coded EMA motion blur on stairs at 0.5 and 1.0 m/s (success 0.6 and 0.4).

Real-world parkour evaluation

Finally, we deploy the policy on a Unitree Go2 with RealSense D435 camera. The ONNX-exported policy runs onboard on an NVIDIA Jetson Nano. We evaluate two representative parkour skills in the real world: gap traversal and high-platform climbing. Although the Go2 reports a maximum joint torque of 45 Nm, using this value in simulation led to a large sim-to-real gap during aggressive motions; reducing the simulated torque limit to 30 Nm substantially improved transfer. Since prior parkour systems use lighter platforms such as Unitree A1, Unitree Go1, and Deep Robotics Lite3 (around 12 kg, while go2 weighs 15 kg) [4, 28, 29], we do not make direct numerical comparisons with their hardware results.

As shown in Figure 7, all methods perform reliably at low difficulty, but differs as difficulty increases. PROOF remains competitive on moderate gaps, achieves the highest success rate on the largest tested gap, and maintains strong platform-climbing performance in the mid-to-high difficulty range.

Real-world parkour deployment showing success rates for huge gap leaps and high platform climbs

Figure 7. Real-world parkour deployment on a Unitree Go2. We report success rates for two representative high-dynamic skills: leaping increasingly width gaps and climbing increasingly height platforms.

These results validate the complete perception-control pipeline under real camera artifacts, latency, calibration errors, and unmodeled contact dynamics.

Conclusion

We presented PROOF, a two-stage perceptive locomotion framework that enables agile quadruped parkour maneuvers and remains robust towards severe visual degradation. The central idea is to treat severe perception failure not as ordinary observation noise, but as a condition under which the controller should stop using exteroception. By training a teacher policy with randomized exteroceptive masking, PROOF learns both perception-guided parkour skills and proprioceptive fallback behaviors. The distilled depth-vision student further models the predicted exteroceptive latent as a Gaussian distribution, allowing the learned log-variance to serve as an interpretable reliability signal for uncertainty-aware gating. Experiments show expressive real-world parkour under clean observations, and stable locomotion when depth perception becomes corrupted or OOD. These results suggest that explicit perception reliability estimation, coupled with a learned fallback policy, is a practical route toward safer deployment of perceptive legged locomotion systems in visually unreliable environments.

Limitations

PROOF is designed to improve robustness under representative severe depth degradation, including occlusion, artifact patches, and strong visual corruption. While these disturbances cover common real-world failure modes, they cannot exhaustively model all possible sensor and environmental conditions. In addition, the predicted log-variance is a relative reconstruction-uncertainty feature for fallback activation, rather than a calibrated measure of terrain-estimation error or a corruption probability. Future work should reduce the dependence on hand-designed OOD injection and develop more principled uncertainty evaluation protocols.

Acknowledgments

We thank our co-authors for their valuable contributions. Special thanks go to Botian Xu for his fruitful and constructive discussions, which substantially benefited this work. We also acknowledge Hao Cheng for the camera mount design.

This work was supported in part by the National Natural Science Foundation of China (Grant No. 62506183), the Guangdong S&T Program (Grant Nos. 2025A0505020002 and 2024B0101010003), the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2025A1515012998), the Guangdong Province Talent Program, the PCL Major Key Project (PCL2025A17-2), the Shenzhen Key Laboratory of Robotics Perception and Intelligence (ZDSYS20220810171800001), and the High-Level Special Funds (G03034K003) from Southern University of Science and Technology, Shenzhen, China.

Simulation Setup and MDP Details

This section specifies the simulation interfaces and perturbations used by the two-stage training pipeline. Both teacher training and student distillation use 4096 parallel Isaac Lab environments, 200 Hz physics, and a 50 Hz policy step.

Observation Space Specification

We use separate observation groups for the teacher actor, teacher critic, and deployable student. The teacher actor receives proprioception, a local elevation map, and an exteroceptive latent mask. The critic additionally receives privileged simulation states. During distillation, the student replaces the elevation map with depth images, while the elevation map and mask remain teacher-side supervision.

Obs GroupDim. / ShapeUsed byContents
Proprioception10×4510 \times45Teacher actor, studentBase angular velocity, projected gravity, velocity command, joint positions, joint velocities, and previous actions.
Elevation map162Teacher actor/criticYaw-aligned terrain scan over 1.7 m×0.8 m1.7\,\mathrm{m} \times0.8\,\mathrm{m} with 0.1 m0.1\,\mathrm{m} spacing; ray origin is shifted 0.6 m0.6\,\mathrm{m} forward and heights use a 0.25 m0.25\,\mathrm{m} vertical offset.
Depth image12×23×442 \times23 \times44StudentTwo delayed forward-camera depth frames after ray clipping, crop, artifact noise, blur, and normalization.
Exteroceptive mask1Teacher actor, distillation lossBinary gate target; 0 masks the teacher’s exteroceptive latent.
Privileged state80Teacher criticBase linear/angular velocity, gravity, command, joint positions, velocities, efforts, and accelerations, actions, foot contact flags, and foot heights.

Table 2. Observation groups used in teacher training and student distillation.

The teacher actor encodes the proprioceptive history and the elevation map into two 64-dimensional latents. The first three dimensions of the proprioceptive latent are supervised to estimate base linear velocity. The student keeps the same proprioceptive interface and predicts the 64-dimensional exteroceptive latent from depth through a CNN–GRU estimator. Noise is applied only to deployable sensing channels: proprioception uses uniform sensor noise, the actor elevation map uses small uniform height noise, and depth images use the vision randomization described below. The privileged state and binary masks are not corrupted.

Action Space Specification

The policy outputs a 12-D target joint-position offset from the default Go2 posture. Hip joints use scale 0.125; thigh and calf joints use scale 0.25.

Targets are executed by a delayed implicit PD actuator solved inside PhysX. We use stiffness 25.0, damping 0.5, effort limit 30 Nm, and velocity limit 30 rad/s. At each reset, actuator delay is sampled from 0 to 3 physics steps, i.e., 0–15 ms.

Reward Specification

The teacher is trained with a weighted sum of velocity-tracking, regularization, and terrain-safety terms:

rt=∑iwiri.(7)r_t=\sum_i w_i r_i. \tag*{(7)}

The main tracking terms use exponential kernels,

rvxy=exp⁡(−∥evxy∥22/0.25),rωz=exp⁡(−eωz2/0.25),(8)r_{v_{xy}}=\exp\left(-\lVert e_{v_{xy}}\rVert_2^2/0.25\right),\qquad r_{\omega_z}=\exp\left(-e_{\omega_z}^2/0.25\right), \tag*{(8)}

where evxye_{v_{xy}} and eωze_{\omega_z} are command-tracking errors in the base frame. On parkour terrains, forward overspeed is tolerated by evaluating the tracking error with vˉx=min⁡(vx,vxcmd+0.1)\bar{v}_x=\min(v_x,v_x^{\mathrm{cmd}}+0.1); otherwise vˉx=vx\bar{v}_x=v_x.

Episode Termination Specification

Episodes are reset either by time truncation or by failure termination:

dt=dttimeout∨dtori∨dtcontact.(9)d_t=d_t^{\mathrm{timeout}}\lor d_t^{\mathrm{ori}}\lor d_t^{\mathrm{contact}}. \tag*{(9)}

The timeout term is treated as truncation; the other two terms mark failed unfinished rollouts.

TermTypeCondition
Episode length limitTruncationEpisode length reaches 20 s, i.e., 1000 policy steps at 50 Hz.
Bad base orientationFailurearccos⁡(−gb,z)>1.5\arccos(-g_{b,z}) > 1.5 rad, where gb,zg_{b,z} is the vertical component of projected gravity in the base frame.
Base contactFailureThe net contact force on robot base link exceeds 5 N.

Table 4. Episode termination terms.

Velocity Command Sampling

Velocity commands are resampled every 4–8 s. The nominal ranges are

vx∈[0,1.2] m/s,vy∈[−0.5,0.5] m/s,ωz∈[−1.5,1.5] rad/s.v_x \in[0, 1.2]\ \mathrm{m/s},\quad v_y \in[-0.5, 0.5]\ \mathrm{m/s},\quad\omega_z \in[-1.5, 1.5]\ \mathrm{rad/s}.

Small planar commands below 0.2 m/s are zeroed. For parkour terrains, lateral velocity and heading targets are forced to zero so that the robot approaches obstacles head-on. For beam, crawl, tilt, and stair terrains, the forward command is additionally overridden by vx∈[0.8,1.2]v_x \in[0.8, 1.2] m/s. Other non-parkour terrains use the nominal ranges.

Domain Randomization

Regular Randomization. We randomize dynamics properties at startup and inject external perturbations during rollouts. Details are in Table 5.

D.R. ItemsRange / setting
Foot frictionStatic and dynamic friction in [0.2,1.5][0.2, 1.5]; restitution in [0,0.5][0, 0.5].
Base massAdditive mass offset in [−0.5,1.0][-0.5, 1.0] kg.
Other body massesScale by [0.8,1.2][0.8, 1.2].
Base center of massOffset each axis by [−0.05,0.05][-0.05, 0.05] m.
Actuator gainsScale stiffness and damping by [0.8,1.2][0.8, 1.2].
Joint frictionAdditive friction in [0,0.1][0, 0.1].
Reset stateJoint positions scaled by [0.75,1.25][0.75, 1.25]; base xx reset in [−0.5,0.5][-0.5, 0.5] m.
External pushEvery 10 s, set base planar velocity in [−0.5,0.5][-0.5, 0.5] m/s per axis.

Table 5. Regular domain randomization.

Depth Vision Randomization. The student depth branch is trained with randomized camera geometry, image corruption, and image frame delay. Details are in Table 6.

ItemRange / setting
Camera translationUniform perturbation in [−0.01,0.01][-0.01, 0.01] m per axis.
Camera rotationRoll/yaw in [−2∘,2∘][-2^\circ, 2^\circ]; pitch in [−8∘,8∘][-8^\circ, 8^\circ].
Ray start point offsetsPositional in [−0.005,0.005][-0.005, 0.005] m; Rotational in [−1∘,1∘][-1^\circ, 1^\circ] per ray.
Sparse artifactsRandom 0/1 depth artifacts with a probability of 0.0002.
Frame delay20 Hz sampling in simulation with 0–1 frame delay, i.e., 0–0.05 s.

Table 6. Depth-vision randomization.

Teacher Exteroception Masking

For teacher training, the exteroceptive latent is randomly masked only on non-parkour terrains: flat rough, slope, inverse slope, and 30° stairs in both directions. At reset, eligible environments sample a binary mask with probability 0.4 of closing the exteroceptive branch. The actor therefore sees πT(zp,mze)\pi_T(z^p,m^{z^e}) and must learn both perceptive and proprioceptive fallback behaviors.

Student Vision Disturbance

For student distillation, depth disturbances are enabled on non-parkour terrain envs. A disturbance burst starts with probability 0.01 per 20 Hz depth update and lasts 2–100 frames, i.e., 0.1–5 s. Each active burst samples one of three modes: (1). full-frame blocking with probability 0.6 using value 0 or 1; (2). Gaussian noise with probability 0.2 and standard deviation in [0.15,0.45][0.15, 0.45]; (3). **large rectangular artifacts with probability 0.2, at most three blocks, 8±\pm4 pixels per side, and value 0 or 1.

When a depth burst is active, the aligned teacher mask is set to zero, so that student is supervised by both teacher’s proprioceptive fallback action and closing visual gate target.

Training Pipeline and Implementation Details

This section summarizes the execution order of the teacher and student stages. The teacher is first trained with privileged elevation-map perception and randomized exteroceptive masking. The student then distills the frozen teacher into a depth-based recurrent policy with uncertainty-aware gating.

Algorithm Overview

The detailed teacher and student training procedures are summarized in Table 7 and Table 8, respectively.

Algorithm 1: Teacher Policy Training
Obs proprioception otpo_t^p, elevation map hth_t, privileged state otprivo_t^{\mathrm{priv}}, mask mtm_t
/* Rollout collection */
1. for each rollout step tt do
2.Sample terrain, command, and domain-randomized dynamics;
3.if terrain is non-parkour then sample the exteroceptive mask; else keep perception unmasked;
4.Compute ztp←Ep(otp)z_t^p \leftarrow E_p(o_t^p), zte←Ee(ht)z_t^e \leftarrow E_e(h_t), and at←πT(ztp,zte,mtzte)a_t \leftarrow\pi_T(z_t^p, z_t^e, m_t z_t^e);
5.Feed the action into the simulator for physics step;
6.Compute rewards;
7.Store rollout data;
/* Return and advantage estimation */
8.Compute GAE returns and advantages;
9.Normalize advantages over the rollout batch;
/* PPO update */
10. for each epoch and mini-batch do
11.Compute PPO surrogate loss Lclip\mathcal{L}_{\mathrm{clip}}, value loss LV\mathcal{L}_V, and entropy H\mathcal{H};
12.Compute velocity-estimation supervision Lv←MSE⁡(v^t,vt)\mathcal{L}_v \leftarrow\operatorname{MSE}(\hat{v}_t, v_t);
13.LT←Lclip+λVLV−λHH+λvLv\mathcal{L}_T \leftarrow\mathcal{L}_{\mathrm{clip}} + \lambda_V\mathcal{L}_V - \lambda_H\mathcal{H} + \lambda_v\mathcal{L}_v;
14.Update teacher actor, critic, and encoders with AdamW optimizer;

Table 7. Teacher policy training pipeline.

Algorithm 2: Student Distillation
Obs proprioception otpo_t^p, depth dtd_t, elevation map hth_t, mask mtm_t; frozen teacher πT\pi_T
/* Initialization */
1. Copy the teacher proprioceptive encoder and actor network;
2. Freeze the student proprioceptive encoder;
/* Rollout and online teacher supervision */
3. for each rollout step tt do
4. Apply depth randomization and burst disturbance; set the teacher’s mask per env;
5. Compute z^te,ℓt,gt\hat{z}_t^e,\ell_t,g_t from (otp,dt)(o_t^p,d_t) then student action atS←πS(ztp,gtz^te)a_t^S \leftarrow\pi_S(z_t^p,g_t\hat{z}_t^e);
6. Compute the teacher target ztp←Ee(ht)z_t^p \leftarrow E_e(h_t) and atT←πT(ztp,mtzte)a_t^T \leftarrow\pi_T(z_t^p,m_t z_t^e);
7. Feed the student action into the simulator for physics step;
8. Store student outputs, teacher targets, uncertainty, and gate labels;
/* Distillation update */
9. for each epoch over rollout sequences do
10. Recompute student predictions over rollout sequences with recurrent state reset at episode boundaries;
11. Compute action-imitation loss La\mathcal{L}_a, latent MSE Lz\mathcal{L}_z, variance NLL LNLL\mathcal{L}_{\mathrm{NLL}} on a detached mean, and gate BCE Lg\mathcal{L}_g;
12. LS←λaLa+λzLz+λnLNLL+λgLg\mathcal{L}_S \leftarrow\lambda_a\mathcal{L}_a+\lambda_z\mathcal{L}_z+\lambda_n\mathcal{L}_{\mathrm{NLL}}+\lambda_g\mathcal{L}_g;
13. Update πS\pi_S with AdamW optimizer and gradient clipping;

Table 8. Student distillation pipeline.

Network Architecture and Hyperparameters

The teacher and student network architectures are summarized in Table 9 and Table 10, respectively. The teacher actor uses separate proprioceptive and elevation-map encoders, while the teacher critic is a pure MLP that takes privileged state and elevation-map observations. Unless otherwise specified, MLP layers use ELU activations; the depth CNN in the student uses LayerNorm followed by ReLU.

ModuleSpecification
Proprioceptive encoderMLP [256, 128] →\rightarrow 64; outputs ztpz_t^p from otpo_t^p. The first three latent dimensions are supervised to estimate base linear velocity.
Elevation-map encoderMLP [128] →\rightarrow 64; outputs exteroceptive latent ztez_t^e from elevation map hth_t.
Mask interfaceActor input is (ztp,mtzte)(z_t^p,m_tz_t^e); mt=0m_t=0 removes elevation-map information and mt=1m_t=1 keeps it.
Actor NetworkMLP [512, 256, 128] →\rightarrow 12; Gaussian policy with initial std. 1.5.
CriticPure MLP [512, 256, 128] →\rightarrow 1; uses privileged state and elevation map.

Table 9. Teacher network architecture.

ModuleSpecification
Proprioceptive branchCopied from the teacher and frozen during distillation; outputs ztpz_t^p.
Depth encoderCNN channels [4, 1], 3×33 \times3 kernels, stride 1, LayerNorm, ReLU; projection MLP [128] →\to 64 gives visual feature ztvz_t^v.
GRU estimatorOne-layer GRU with hidden size 64 predicts z^te\hat{z}_t^e from (ztp,ztv)(z_t^p, z_t^v).
Uncertainty headMLP [64] predicts log-variance ℓt\ell_t for the latent NLL loss.
Gate headMLP [64, 64] →\to 1 predicts a sigmoid score g~t\tilde{g}_t. The actor receives a binary hysteresis mask: a closed gate opens at 0.8 and an open gate closes below 0.5.
Actor networkInitialized from the teacher; input is (ztp,gtz^te)(z_t^p, g_t\hat{z}_t^e) and output is a 12-D Gaussian action.

Table 10. Student network architecture.

Optimization settings are listed in Table 11. Both stages use 4096 parallel Isaac Lab environments with AdamW optimization. The full two-stage training takes approximately 4.5 hours on a single NVIDIA RTX 4090 GPU, with the teacher trained for 8000 iterations (first 1500 iters are all on non-parkour-terrains) and the student for 4000 iterations.

ItemTeacher PPOStudent distillation
Rollout
Parallel envs40964096
Horizon24 steps/env24 steps/env
Optimization
OptimizerAdamWAdamW
Learning rate10−310^{-3} with adaptive KL schedule10−410^{-4}
Update epochs5 epochs, 4 mini-batches5 epochs, sequence length 24
Gradient clippingMax norm 1.0Max norm 1.0
Loss and initialization
PPO constantsγ=0.99\gamma= 0.99, λ=0.95\lambda= 0.95, clip 0.2, desired KL 0.01—
Loss weightsλV=1.0\lambda_V = 1.0, λH=0.005\lambda_H = 0.005, λv=1.0\lambda_v = 1.0action λa=2.0\lambda_a = 2.0, latent MSE λz=2.0\lambda_z = 2.0, NLL λn=1.0\lambda_n = 1.0, gate λg=1.0\lambda_g = 1.0
InitializationPyTorch Default InitCopy teacher proprioceptive encoder and actor network

Table 11. Training hyperparameters.

Evaluation Details

Simulation Evaluation In simulation, each policy is evaluated independently on each terrain type with 100 parallel environments. We collect 100 completed episodes for every policy–terrain pair. Robots are initialized on the easiest terrain level, and the terrain-level update follows the same curriculum logic as training. Velocity-command sampling and domain randomization are also kept consistent with the training setup. We report the traversal success rate and the mean velocity-tracking reward over the collected episodes.

Real-World Parkour Evaluation Real-world parkour evaluations are conducted indoors on the Unitree Go2 with the RealSense D435. Visual corruptions are not injected during these trials. For each policy and terrain setting, we conduct 10 trials. After each trial, the robot is reset before the next rollout. A trial is successful only if the robot completes the obstacle without falling, triggering an emergency stop, or requiring human intervention.

Gate Reliability

A hard binary gate may switch at high frequency and disturb the policy’s action. A continuous gate value in (0,1)(0, 1) is also a unreasonable interface to the actor: the teacher latent is only ever fully present or exactly zero, so intermediate values would have to be absorbed by finetuning the actor during distillation. If the binary gate is reliable, the hard mask is therefore preferred. To verify this, we train with a hard gate and evaluate it as a binary classifier on parkour and non-parkour terrain under seen and unseen corruptions, with gate closure (g=0g = 0) as the positive class. AUROC and AUPR use the continuous score. Precision, recall, false-positive rate, and false-negative rate binarize that score at 0.5. Each condition collects 10,000 policy steps on 100 parallel environments and is repeated over five random seeds.

On non-parkour terrain, a corruption begins with probability 0.02 at each camera refresh and lasts 2–100 frames. Seen patterns are those used in training: full-frame masking, global random depth noise, and rectangular depth artifacts. Unseen patterns are randomly oriented stripe masks and reflection-like elliptical artifacts. Clean parkour rollouts report only the false-positive rate, since severe vision corruption leads to inability to parkour. Table 12 shows near-perfect ranking on seen corruptions and strong ranking on held-out patterns. Therefore, it is reasonable to use a final binary gate as a hard mask for fallback behaviour activation.

ConditionAUROC ↑\uparrowAUPR ↑\uparrowPrec. ↑\uparrowRec. ↑\uparrowFPR ↓\downarrowFNR ↓\downarrow
Seen, non-parkour0.9990.9990.9960.9690.0040.031
Unseen, non-parkour0.9950.9950.9970.7600.0030.240
Clean parkour––––0.004–

Table 12. Gate reliability under seen/unseen non-parkour corruptions and clean parkour vision (mean ±\pm std)

Hardware Visual Corruption Tests

Table 13 reports locomotion stability under controlled visual corruption types. Each corruption is tested for five trials at commanded forward speeds of 0.0, 0.5, and 1.0 m/s, on flat ground and on stairs with 30 cm treads and 15 cm risers. A trial succeeds if the robot maintains a normal locomotion without failure for the commanded motion. Physical occlusion, flashlight overexposure, and a waving mirror are applied by the operator in the scene. Motion blur is a coded temporal EMA smooth of the depth stream with mixing weight 0.8. Depth dropout replaces the image by zeros, and depth noise adds Gaussian noise. Depth latency and large calibration shifts are not included: the policy already consumes 10 Hz depth, which is already tolerant, and small camera-pose error is covered by mild domain randomization.

CorruptionFlat 0.0Flat 0.5Flat 1.0Stairs 0.0Stairs 0.5Stairs 1.0
Physical occlusion1.01.01.01.01.01.0
Overexposure (flashlight)1.01.01.01.01.00.8
Reflection (waving mirror)1.01.01.01.00.81.0
Motion blur1.01.01.01.00.60.4
Depth dropout1.01.01.01.01.01.0
Depth noise1.01.01.01.01.01.0

Table 13. Hardware success rate under controlled visual corruptions.

References

  1. [1]R. Yang, M. Zhang, N. Hansen, H. Xu, and X. Wang. Learning vision-guided quadrupedal locomotion end-to-end with cross-modal transformers. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nmHJ3oo6AB.arxiv.org/abs/2107.03996
  2. [2]T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62): eabk2822, Jan. 2022. ISSN 2470-9476. doi:10.1126/scirobotics.abk2822. URL http://arxiv.org/abs/2201.08117. arXiv:2201.08117 [cs].
  3. [3]A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. In K. Liu, D. Kulic, and J. Ichinowski, editors, Proceedings of the 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 403–415. PMLR, 2023. URL https://proceedings.mlr.press/v205/agarwal23a.html.arxiv.org/abs/2211.07638
  4. [4]X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. In Proceedings of the IEEE International Conference on Robotics and Automation, pages 11443–11450, 2024. doi:10.1109/ICRA57147.2024.10610200. URL https://doi.org/10.1109/ICRA57147.2024.10610200.
  5. [5]Z. Zhuang, S. Yao, and H. Zhao. Humanoid parkour learning. In P. Agrawal, O. Kroemer, and W. Burgard, editors, Proceedings of The 8th Conference on Robot Learning, volume 270 of Proceedings of Machine Learning Research, pages 1975–1991. PMLR, 06–09 Nov 2025. URL https://proceedings.mlr.press/v270/zhuang25a.html.arxiv.org/abs/2406.10759
  6. [6]D. Hoeller, N. Rudin, D. V. Sako, and M. Hutter. ANYmal parkour: Learning agile navigation for quadrupedal robots. Science Robotics, 9(88):eadi7566, 2024. doi:10.1126/scirobotics.adi7566. URL https://www.science.org/doi/10.1126/scirobotics.adi7566.DOI
  7. [7]N. Rudin, J. He, J. Aurand, and M. Hutter. Parkour in the Wild: Learning a General and Extensible Agile Locomotion Policy Using Multi-expert Distillation and RL Fine-tuning, May 2025. URL http://arxiv.org/abs/2505.11164. arXiv:2505.11164 [cs] version: 1.
  8. [8]H. Lai, J. Cao, J. Xu, H. Wu, Y. Lin, T. Kong, Y. Yu, and W. Zhang. World model-based perception for visual legged locomotion. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 11531–11537, 2025. doi:10.1109/ICRA55743.2025.11128762.DOI
  9. [9]Z. Wu, X. Huang, L. Yang, Y. Zhang, X. Chen, P. Abbeel, R. Duan, A. Kanazawa, C. Sferrazza, G. Shi, and C. K. Liu. Perceptive humanoid parkour: Chaining dynamic human skills via motion matching, 2026. URL https://arxiv.org/abs/2602.15827.
  10. [10]Z. Zhuang, S. Zhu, M. Zhao, and H. Zhao. Deep Whole-body Parkour, Jan. 2026. URL http://arxiv.org/abs/2601.07701. arXiv:2601.07701 [cs].
  11. [11]Q. Ben, B. Xu, K. Li, F. Jia, W. Zhang, J. Wang, J. Wang, D. Lin, and J. Pang. Gallant: Voxel grid-based humanoid locomotion and local-navigation across 3d constrained terrains, 2025. URL https://arxiv.org/abs/2511.14625.
  12. [12]W. Sun, B. Cao, L. Chen, Y. Su, Y. Liu, Z. Xie, and H. Liu. Learning perceptive humanoid locomotion over challenging terrain. In 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6571–6578. IEEE, 2025.arxiv.org/abs/2503.00692
  13. [13]J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner, S. Bohez, and V. Vanhoucke. Sim-to-real: Learning agile locomotion for quadruped robots. In Proceedings of Robotics: Science and Systems, Pittsburgh, Pennsylvania, June 2018. doi:10.15607/RSS.2018.XIV.010.DOI
  14. [14]J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsounis, V. Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26):eaau5872, Jan. 2019. ISSN 2470-9476. doi:10.1126/scirobotics.aau5872. URL https://www.science.org/doi/10.1126/scirobotics.aau5872.DOI
  15. [15]N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 91–100. PMLR, 08–11 Nov 2022. URL https://proceedings.mlr.press/v164/rudin22a.html.arxiv.org/abs/2109.11978
  16. [16]A. Kumar, Z. Fu, D. Pathak, and J. Malik. RMA: Rapid Motor Adaptation for Legged Robots. In Proceedings of Robotics: Science and Systems, Virtual, July 2021. doi:10.15607/RSS.2021.XVII.011.DOI
  17. [17]S. Choi, G. Ji, J. Park, H. Kim, J. Mun, J. H. Lee, and J. Hwangbo. Learning quadrupedal locomotion on deformable terrain. Science Robotics, 8(74):eade2256, 2023. doi:10.1126/scirobotics.ade2256. URL https://www.science.org/doi/10.1126/scirobotics.ade2256.DOI
  18. [18]E. Chane-Sane, J. Amigo, T. Flayols, L. Righetti, and N. Mansard. Soloparkour: Constrained reinforcement learning for visual locomotion from privileged experience. In Conference on Robot Learning (CoRL), 2024.
  19. [19]G. B. Margolis, G. Yang, K. Paigwar, T. Chen, and P. Agrawal. Rapid locomotion via reinforcement learning. The International Journal of Robotics Research, 43(4):572–587, 2024.
  20. [20]T. He, C. Zhang, W. Xiao, G. He, C. Liu, and G. Shi. Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion. In Proceedings of Robotics: Science and Systems, Delft, Netherlands, July 2024. doi:10.15607/RSS.2024.XX.059.DOI
  21. [21]Q. Liao, T. E. Truong, X. Huang, Y. Gao, G. Tevet, K. Sreenath, and C. K. Liu. Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion, 2025. URL https://arxiv.org/abs/2508.08241.
  22. [22]B. Xu, H. Weng, Q. Lu, Y. Gao, and H. Xu. Facet: Force-adaptive control via impedance reference tracking for legged robots. In J. Lim, S. Song, and H.-W. Park, editors, Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 705–720. PMLR, 27–30 Sep 2025. URL https://proceedings.mlr.press/v305/xu25c.html.
  23. [23]P. Zhi, P. Li, J. Yin, B. Jia, and S. Huang. Learning a unified policy for position and force control in legged loco-manipulation. In J. Lim, S. Song, and H.-W. Park, editors, Proceedings of The 9th Conference on Robot Learning, volume 305 of Proceedings of Machine Learning Research, pages 652–669. PMLR, 27–30 Sep 2025. URL https://proceedings.mlr.press/v305/zhi25a.html.arxiv.org/abs/2505.20829
  24. [24]J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning Quadrupedal Locomotion over Challenging Terrain. Science Robotics, 5(47):eabc5986, Oct. 2020. ISSN 2470-9476. doi:10.1126/scirobotics.abc5986. URL http://arxiv.org/abs/2010.11251. arXiv:2010.11251 [cs].
  25. [25]G. Ji, J. Mun, H. Kim, and J. Hwangbo. Concurrent Training of a Control Policy and a State Estimator for Dynamic and Robust Legged Locomotion. IEEE Robotics and Automation Letters, 7(2):4630–4637, Apr. 2022. ISSN 2377-3766. doi:10.1109/LRA.2022.3151396. URL https://ieeexplore.ieee.org/document/9714001/?arnumber=9714001. Conference Name: IEEE Robotics and Automation Letters.DOI
  26. [26]I. M. Aswin Nahrendra, B. Yu, and H. Myung. Dreamwaq: Learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5078–5084, 2023. doi: 10.1109/ICRA48891.2023.10161144.
  27. [27]H. Wang, H. Luo, W. Zhang, and H. Chen. Cts: Concurrent teacher-student reinforcement learning for legged locomotion. IEEE Robotics and Automation Letters, 9(11):9191–9198, 2024. doi:10.1109/LRA.2024.3457379.
  28. [28]Z. Zhuang, Z. Fu, J. Wang, C. G. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. In J. Tan, M. Toussaint, and K. Darvish, editors, Proceedings of The 7th Conference on Robot Learning, volume 229 of Proceedings of Machine Learning Research, pages 73–92. PMLR, 06–09 Nov 2023. URL https://proceedings.mlr.press/v229/zhuang23a.html.arxiv.org/abs/2309.05665
  29. [29]S. Luo, S. Li, R. Yu, Z. Wang, J. Wu, and Q. Zhu. Pie: Parkour with implicit-explicit learning framework for legged robots. IEEE Robotics and Automation Letters, 9(11):9986–9993, 2024. doi:10.1109/LRA.2024.3459797.DOI
  30. [30]L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel. Asymmetric actor critic for image-based robot learning. Robotics: Science and Systems XIV, 2018.arxiv.org/abs/1710.06542
  31. [31]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017. URL https://arxiv.org/abs/1707.06347.
  32. [32]S. Zhu, Z. Zhuang, M. Zhao, K.-Y. Lee, and H. Zhao. Hiking in the wild: A scalable perceptive parkour framework for humanoids, 2026. URL https://arxiv.org/abs/2601.07718.
  33. [33]S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In G. Gordon, D. Dunson, and M. Dudík, editors, Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learning Research, pages 627–635. Fort Lauderdale, FL, USA, 11–13 Apr 2011. PMLR. URL https://proceedings.mlr.press/v15/ross11a.html.arxiv.org/abs/1011.0686
  34. [34]NVIDIA, ; M. Mittal, P. Roth, J. Tigue, A. Richard, O. Zhang, P. Du, A. Serrano-Muñoz, X. Yao, R. Zurbrügg, N. Rudin, L. Wawrzyniak, M. Rakhsha, A. Denzler, E. Heiden, A. Borovicka, O. Ahmed, I. Akinola, A. Anwar, M. T. Carlson, J. Y. Feng, A. Garg, R. Gasato, L. Gulich, Y. Guo, M. Gussert, A. Hansen, M. Kulkarni, C. Li, W. Liu, V. Makoviychuk, G. Malczyk, H. Mazhar, M. Moghani, A. Murali, M. Noseworthy, A. Poddubny, N. Ratliff, W. Rehberg, C. Schwerke, R. Singh, J. L. Smith, B. Tang, R. Thaker, M. Trepte, K. V. Wyk, F. Yu, A. Millane, V. Ramasamy, R. Steiner, S. Subramanian, C. Volk, C. Chen, N. Jawale, A. V. Kuruttukulam, M. A. Lin, A. Mandlekar, K. Patzwaldt, J. Welsh, H. Zhao, F. Anes, J.-F. Lafleche, N. Moënne-Loccoz, S. Park, R. Stepinski, D. V. Gelder, C. Amevor, J. Carius, J. Chang, A. H. Chen, P. de Heras Ciechomski, G. Daviet, M. Mohajerani, J. von Muralt, V. Reutsky, M. Sauter, S. Schirm, E. L. Shi, P. Terdiman, K. Vilella, T. Widmer, G. Yeoman, T. Chen, S. Grizan, C. Li, L. Li, C. Smith, R. Wiltz, K. Alexis, Y. Chang, D. Chu, L. J. Fan, F. Farshidian, A. Handa, S. Huang, M. Hutter, Y. Narang, S. Pouya, S. Sheng, Y. Zhu, M. Macklin, A. Moravanszky, P. Reist, Y. Guo, D. Hoeller, and G. State. Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning, 2025. URL https://arxiv.org/abs/2511.04831.

Paper details

vlnc:W_xj5ax86e
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 10

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents