WorksComputer Science
Active Reachability Learning for Humanoid Recoverability Monitoring
Abstract
Deploying humanoid task policies requires reliable runtime monitors that anticipate unrecoverable falls in time to trigger a fallback. We formulate this task as learning the stochastic, finite-horizon recoverability envelope of a given fallback policy. Synthesizing this monitor is challenging: existing methods assume training data naturally covers the recoverability envelope, yet acquiring such data remains a major bottleneck for contact-rich systems. Consequently, they recover only a fraction of the true envelope, leaving them prone to optimistic hallucinations for out-of-distribution (OOD) inputs. We resolve this via a unified framework that iterates between value learning and active exploration. Data exploration is reframed as a reach-avoid problem, where an uncertainty-guided explorer systematically uncovers unseen states while avoiding confidently unrecoverable regions. This loop is further supported by a combined Temporal Difference and Monte Carlo objective for accurate in-distribution value prediction and reliable uncertainty estimation. By explicitly treating the remaining high-uncertainty regions as unrecoverable, the monitor mitigates optimistic OOD hallucinations. Evaluated on stochastic benchmarks and a 29-DoF Unitree G1 humanoid, our approach drastically improves monitor coverage and accuracy and successfully transfers zero-shot to real-world hardware.
Correspondence: [email protected], [email protected]
Introduction
The deployment of contact-rich robotic systems, such as humanoids, holds immense potential for automating tasks in unstructured environments. However, inherent instability and underactuation make humanoids prone to catastrophic failures, such as unrecoverable falls. Hence, safely deploying versatile task policies necessitates reliable runtime monitors that detect impending failures in advance, allowing the system to trigger a robust fallback policy before a fall becomes unavoidable. In this paper, we formulate the monitor synthesis as learning an empirical, finite-horizon recoverability (or liveness) envelope for a fixed fallback balancing policy. Such reliable monitors enable rapid development and safe deployment of arbitrary humanoid task policies while drastically reducing the risk of hardware damage.
The synthesis of this recoverability monitor exhibits extreme numerical stiffness, as the underlying liveness value may change violently during exact rigid-body interactions. For example, a slipping foot and a firm step may share nearly identical states at the moment of contact, yet lead to completely different outcomes. More fundamentally, training requires a comprehensive mapping of the underlying liveness set, encompassing states that are recoverable and reachable under realistic physical disturbances. To cover these states, we cannot uniformly sample the state space or artificially “teleport” the robot to expand coverage, as minor kinematic offsets often cause rigid-body penetration, triggering severe numerical artifacts in the simulator that corrupt the training data. Therefore, we are restricted to acquiring data via episodic rollouts driven by dynamic controls or physical perturbations. However, rolling out from a manually designed initial-state distribution yields insufficient liveness set coverage, leaving the monitor prone to making overly optimistic predictions for out-of-distribution (OOD) inputs.
Existing safety frameworks struggle with the compounded dimensionality and data coverage challenges. Analytical Control Barrier Functions (CBFs) [1, 2, 3, 4] are effective for task-space constraints but scale poorly to whole-body recoverability monitoring. Similarly, existing stochastic reachability solvers [5], [6] are computationally restricted to low-dimensional systems. Physics-Informed Neural Network (PINN) reachability methods [7, 8, 9] handle high dimensionality with deep learning, but rely on uniform state-space sampling. This renders them incompatible with contact-rich domains where uniform sampling inevitably yields mostly irrelevant or non-physical, corrupted samples.
On the other hand, Reinforcement Learning (RL)-based methods learn from simulator rollouts, which respect physical constraints. RL-based CBFs [10, 11, 12] and RL-based reachability approaches [13, 14, 15, 16, 17] adopt this strategy. However, because these rollouts originate from nominal initial-state distributions, they only recover a narrow fraction of the true recoverability envelope. This limited coverage not only reduces the allowable operation space but also causes overly optimistic predictions for OOD states. While recent work attempts to mitigate this by treating high-uncertainty OOD states as unrecoverable [18], this assumes the training dataset naturally captured enough “OOD” states (determined by a frozen world model) prior to learning. Standard RL exploration techniques [19, 20, 21] also cannot fill this gap, as they are designed to maximize task reward rather than systematically chart the boundaries of recovery failure. Consequently, despite its critical importance for real-world deployment, the problem of actively exploring the recoverability envelope under non-teleporting physical constraints remains under-explored by the community.
We tackle this challenge through a unified bi-level synthesis framework. In the Inner Level (Value Learning), we estimate the finite-horizon liveness value of the fallback policy using a combined Temporal Difference (TD) and Monte Carlo (MC) objective, providing stable gradients for accurate in-distribution (in-D) value learning and reliable epistemic uncertainty estimation. In the Outer Level (Exploration), we reframe data augmentation as a reach-avoid problem, where an active explorer targets OOD states to expand coverage without appending confidently unrecoverable samples. By systematically uncovering the recoverability envelope and treating remaining high-uncertainty regions as failures, the monitor effectively mitigates optimistic hallucinations without sacrificing allowable operation space. Finally, we validate this framework on stochastic benchmarks and demonstrate zero-shot transfer on a 29-DoF Unitree G1 humanoid.
Problem Formulation
System Dynamics
We consider a system operating within a discrete-time Markov Decision Process under a given fallback controller . The dynamics form a stochastic system , where is the continuous state at step , is the control action, and is an optional artificial disturbance. Crucially, we model the system as inherently stochastic rather than relying on traditional worst-case bounded disturbances for liveness synthesis, because compounding worst-case disturbances across multiple sources can easily overpower a humanoid’s control authority, yielding impractically small recoverability envelopes.
During data collection (episodic rollouts), the system is closed-loop under , with and . During data exploration (novel initial state discovery), an active explorer instead freely selects the control and may additionally apply to steer the system toward more informative states. We assume a narrow, manually defined distribution of initial states to seed the first episodic rollouts. As training progresses, subsequent rollouts can optionally be initialized from states accumulated in the replay buffer to continuously expand data coverage.
Liveness Value Function and Safe-Set Coverage
The controller aims to stabilize the system toward a target set (e.g., set of upright stationary states). We quantify the liveness of given through a value function , defined as the expected minimum target function over all possible future trajectories:
A smaller liveness value indicates a more recoverable current state. In the special case of a binary indicator — inside and otherwise— reduces to the exact probability that the system fails to reach .
For effective real-world monitoring, we evaluate the learned value function on two critical axes: predictive accuracy and spatial coverage. Specifically, our goal is to learn a neural value function as the recoverability monitor by minimizing the approximation error over the -liveness set, defined as . Mathematically, we aim to minimize:
Intuitively, coverage is unimportant in the deeply unrecoverable region where , because by mapping accurately, the monitor can reliably alert before the system state drifts OOD. However, for underactuated systems, fully covering using only episodic rollouts for exploration is often infeasible. Consequently, the monitor must empirically remain conservative on OOD inputs, treating unexplored regions as inherently unrecoverable to prevent hallucinations.
Humanoid Balancing Instantiation
We instantiate the framework on humanoid applications, where is a balancing policy and the stochasticity in arises from observation noise and domain randomization (e.g., ground friction, body mass, and PD-gain perturbations). denotes the set of stationary states, in which the robot’s torso and joint velocity magnitudes stay within designated bounds and the torso height remains close to its canonical value. We assume that if can recover balance from , the running minimum of along the trajectory is attained within a horizon of steps. Under this assumption, the infinite-horizon value coincides with the fixed-horizon one , where
Although the monitor only queries , learning provides a stronger signal from boundary anchoring () and a temporal curriculum. Concretely, we use steps with s, i.e., we certify whether the robot returns to a stationary balanced pose within 3 s under . Assuming the fallback policy is empirically more robust than arbitrary task policies, the learned monitor can function as a universal safety layer during deployment.
Remark (Practical Consideration: Reach-Avoid Extension). The liveness formulation assumes that reaching means has recovered without collapsing to the ground and allows the system to stay stationary forever. However, highly agile recovery controllers may allow the system to temporarily collapse then recover, and deployment may impose strict collision-avoidance constraints to prevent damage to the surrounding environment and humans. For these scenarios, our framework naturally extends to reach-avoid formulations.
Method
Overview and Active Learning Loop
Because manually defining the edge of the recoverability envelope for a high-dimensional, contact-rich humanoid is intractable, we cast the monitor synthesis as an iterative active-learning loop. The framework alternates between accurately learning the liveness value of known states and actively discovering novel states at the fringe of the policy’s capabilities. We begin by rolling out the given policy from to collect an initial dataset . Using this dataset, we learn an accurate in-D value function alongside an uncertainty estimator. Leveraging this uncertainty as a reward signal, and initializing from states already in , we then train a disturbance explorer to actively push the system toward novel OOD states that are at most marginally unrecoverable. If the explorer discovers a sufficient number of high-uncertainty states, we roll out the policy from these newly discovered states to augment and return to the value-learning phase. Once the explorer fails to find sufficient meaningful new states, the loop terminates, and the framework outputs the learned value function and uncertainty estimator as the recoverability monitor. For our primary humanoid application, during the deployment of an arbitrary task policy, the monitor triggers the balancing policy until a stationary state is reached whenever either the liveness value or uncertainty exceeds a predefined threshold (). The full active-learning loop is shown in Alg. 1.
Algorithm 1 Uncertainty-Guided Monitor Synthesis Loop
Require: Policy , initial-state distribution
Initialize dataset by rolling out from .
for round do
Fit via (7) and uncertainty on .
Form and from , .
Fine-tune and over a medium horizon (warm-started when possible).
Roll out from to find novel seed states.
Execute from seeds to collect data; append to .
if fewer than novel seeds found then
break
return Recoverability monitor and
In-Distribution Value Function Learning
We learn a discrete-time value that evaluates the expected recoverability of states given policy over a finite horizon . The value is the expected minimum target function achieved along the trajectory, as defined in (3). The induced liveness set is . Crucially, this induced set declares which states the monitor treats as recoverable at deployment (), distinct from the broader mapped during synthesis.
Rather than a single-step TD backup, we train using multi-step (TD-) targets that mix an observed running minimum with a bootstrap. Specifically, for a window of steps we form the running minimum and the TD- target
where is drawn from a truncated geometric distribution (a TD- schedule with decay , capped at steps or the boundary ), and is the stop-gradient operator. The TD loss is the mean-squared residual,
Remark (Conservative bootstrap bias in stochastic systems). Let , so that and . As is concave, Jensen’s inequality gives
where denotes the target (4) evaluated at the exact bootstrap . The -step target therefore overestimates the value even under zero learning error, which shrinks the liveness set. This bias arises only from the bootstrapped tail and vanishes at , where reduces to the unbiased but high-variance MC estimate.
To ground the network in actual liveness outcomes, we further include long-horizon MC supervision. The empirical liveness target for a sampled state over rollouts is
and the MC loss is likewise the prediction residual, . MC targets are far more expensive to generate, since each demands many long-horizon rollouts. In practice we therefore generate a small set of MC targets as unbiased anchors that stabilize training, while collecting abundant TD targets to enforce local temporal consistency of the value function. Together, each iteration minimizes a weighted combination,
Initial dataset and time curriculum. We construct the initial dataset in two stages. We first roll out from samples of the initial-state distribution to form the TD dataset. We then draw a second set of initial states—from both and the states visited in the TD dataset—and generate long-horizon rollouts from them to form the MC dataset; the initial dataset is the union of the two. Importantly, accurately learning the in- value function is crucial for meaningful epistemic uncertainty estimation. To stabilize temporal value learning, we adopt a standard time-curriculum strategy [7], in which the sampled time step for the TD and MC targets is expanded gradually from to the full horizon , with denser sampling near the boundary . Once the exploration loop begins and grows round by round, we remove the curriculum and fine-tune directly over the full horizon on the expanded dataset.
Uncertainty- and Reachability-Based Explorer
To efficiently augment the existing dataset with novel but potentially recoverable states, we cast exploration as a Backward Reach-Avoid Tube (BRAT) problem. Starting from states in the replay buffer , we train an exploration policy that drives the system into a target set of high epistemic uncertainty while staying out of an avoid set of confidently unrecoverable states.
Specifically, we first train an off-the-shelf epistemic uncertainty estimator (e.g., a value-function ensemble or Random Network Distillation (RND)) on the current dataset to determine the high-uncertainty target set:
The avoid set consists of states with high predicted failure and low uncertainty:
that is, requires both (likely unrecoverable) and (low uncertainty). Importantly, the monitor triggering value needs to be substantially smaller than the exploration stopping value for robust deployment. Both and are continuous, providing a dense gradient signal even far from their boundaries. Consequently, even when is effectively empty, the learned still moves toward relatively more novel states, and the system’s momentum helps uncover genuinely new data.
We solve for the exploration value function and its induced policy using an existing learning-based reachability toolbox [22]. While the toolbox traditionally approximates an optimal time-conditioned policy , we instead learn a time-independent policy to simplify the data exploration loop at a slight cost to optimality. The training horizon for the explorer is set as a hyperparameter . A moderate is crucial: it prevents the ineffective exploration from a myopic horizon, while ensuring seeds do not collapse onto a few distant high-uncertainty peaks, as happens with overly long horizons.
Data augmentation. Each round, we roll out from states sampled from . Notably, the rollout horizon for these exploration trajectories can extend beyond the explorer’s training horizon to encourage exploration. Along each trajectory, we extract states that successfully enter the high-uncertainty target set to serve as novel seeds. Because multiple valid seeds may be encountered within a single continuous rollout, we cap the extraction to at most 5 seeds per trajectory to prevent deep penetration into unrecoverable OOD regions. From each extracted seed, we then execute the fallback policy to generate fresh TD and MC data, which are subsequently appended to . Value learning and exploration alternate until a round yields fewer than novel seeds or a predefined round budget is reached.
Results
We evaluate the framework across three settings. First, we conduct case studies on two underactuated benchmarks—a stochastic 4D planar double integrator (DI) and a 3D Dubins car. Both systems admit uniform grid sampling strictly for offline evaluation purposes, allowing us to benchmark our pipeline against near-exact ground truths and isolate individual design choices. During monitor synthesis, only an extremely narrow initial distribution is provided for initial data collection, as illustrated in Figure 2, and new states can only be explored by rolling out from buffered states. Hence, the Dubins car presents severe underactuation for exploration because rollouts are constrained by nonholonomic dynamics with a fixed velocity and bounded turn rate. The DI, conversely, serves as a benchmark with higher stochasticity but lower underactuation. These case studies are structured around three questions: (Q1) Coverage—can our framework recover most of the recoverable states, and which design choice performs best? (Q2) Accuracy—does the TD+MC loss outperform TD() alone or a time-agnostic TD loss [23]? (Q3) Sensitivity—how do the MC sample budget and the explorer’s training horizon affect performance? Second, we scale to a Unitree G1 humanoid, synthesizing a liveness value and failure probability for a balancing policy on complex high-dimensional dynamics. Third, we close the loop by deploying the learned monitor on a walking task policy in simulation and on hardware.

Figure 2. Value heatmaps and level-set contours in the - plane (Left: DI; Right: Dubins). Black dashed boxes denote the initial distribution projected onto the - plane (unshown dimensions are sampled uniformly). Solid circles mark the target set, defined via , where SDF denotes the signed distance function to the target set.
a) Metrics: We evaluate against “ground-truth” empirical liveness values, where each value is labeled via 1000 MC rollouts. We report six metrics: recall—the fraction of truly recoverable states within the -liveness set correctly identified (); precision—the fraction of predicted-recoverable states that are genuinely recoverable, penalizing overoptimism; —their harmonic mean; MSE—the mean squared error; cumulative seeds—the total exploration seeds collected across all rounds, indicating data efficiency; and confident coverage—the fraction of the true recoverable set covered with low epistemic uncertainty (recall restricted to in- predictions). We also report wall time on an NVIDIA RTX 6000 GPU.
b) Baselines: (Q1) We compare our reach-avoid explorer against unbiased random-action exploration and biased random exploration that samples a constant per-episode bias before each exploration rollout. Each explorer is paired with either an ensemble or RND uncertainty estimator. All configurations share identical termination criteria: halting if a round yields < 20 novel seeds or hits a 20-round budget. (Q2) We compare our TD+MC loss against a TD()-only variant, similar to [16]. (Q3) We sweep the MC anchor count and explorer training horizon .
Case Studies: DI and 3D Dubins
Systems. The 4D DI state comprises planar position and velocity with additive Gaussian noise. The 3D Dubins car state is , controlled by a bounded turning rate at a fixed forward speed. Detailed system descriptions are deferred to the appendix. For the Dubins car, we study two variants: the standard model, and Dubins3D-Explore, which augments the explorer’s action space with lateral velocity to isolate the effect of control authority on active exploration. All systems operate under a stochastic PD controller whose output is deliberately flipped at , which mimics humanoid falling—where gravity overpowers stabilizing joint torques once the robot tilts beyond a critical angle—making recoverability synthesis more challenging. Both systems use a 2 s horizon with s. See Figure 2 for representative -liveness set slices.
Metrics are aggregated over 5 random seeds. The ensemble uncertainty is the standard deviation across 3 member networks: . The RND uncertainty is the discrepancy between a distillation network (supervised by on the same dataset) and the learned value at . To ensure both estimators remain informative in OOD regions, they incorporate a strong, untrainable prior , such that . For novel seed exploration, we roll out at most 20K trajectories within each training loop, with .
a) Q1: Liveness-set coverage and design choices: Figure 3 details the per-round learning curves and final metrics. Evidently, the RND estimator provides a highly efficient alternative to ensemble methods. When paired with the reach-avoid explorer, RND matches or surpasses ensemble performance while requiring a lower computational cost as it eliminates the need to train multiple member networks. It additionally provides sharper OOD detection, which effectively bounds the augmented dataset size (cumulative seeds).

Figure 3. Ablation across toy systems (plot rows: DI, Dubins3D, Dubins3D-Exp.). Curves (left) show per-round mean over seeds; the table (right) reports the final-round means for the same metrics. RND uses a single value net, ensemble uses nets. Our default is RND + reach-avoid.
Furthermore, reach-avoid exploration demonstrates a significant advantage over unbiased and biased random exploration, particularly in highly underactuated environments. On the Dubins car, unbiased exploration completely collapses (0.267 coverage) and biased exploration manages only a partial recovery (0.408). By contrast, the reach-avoid explorer achieves a coverage of 0.656. On the DI system, it similarly elevates coverage to 0.941, compared to 0.86 for both random baselines. More importantly, these superior results are obtained in fewer active-learning rounds, demonstrating the efficiency of the reach-avoid explorer. Driven by these margins, we adopt RND + reach-avoid as our default configuration for the remaining experiments.
Despite these positive outcomes, residual coverage gaps remain. Beyond minor learning errors, we attribute these gaps to the inherent underactuation of the systems rather than explorer inefficiency. On the DI system, the recoverable states missed by our default configuration are primarily those moving rapidly toward the target within the control-flip zone (analogous to a humanoid bouncing upward from a highly tilted configuration). Reaching these novel states from existing data requires simultaneously increasing position while driving velocity more negative—a maneuver the closed-loop dynamics simply cannot execute. This limitation is more pronounced in the standard Dubins car, where non-holonomic constraints render a large part of the recoverable set unreachable from without first traversing unrecoverable regions. Granting the explorer additional control authority—as in the Dubins3D-Explore variant, which explores by controlling both turning rate and lateral velocity—closes much of this gap (Figure 4). This augmented authority raises the RND + reach-avoid coverage from 0.656 to 0.921 and to 0.970, confirming that achievable coverage is fundamentally bounded by the explorer’s action space. While the ground-truth recoverable set is not exhaustively covered, the missed states remain kinematically unreachable from without traversing unrecoverable regions or experiencing severe unmodeled deployment-time perturbations.

Figure 4. 3D Dubins: Data coverage evolution over rounds on an originally OOD slice . Dashed black: GT recoverable set; Green: learned set; Blue: all datapoints with .
b) Q2: Accuracy—TD+MC vs. TD-only: Since liveness values do not converge within the 2-s horizon in the toy examples, we defer comparison against the time-agnostic TD baseline to the humanoid experiments (Sec. IV-B). Instead, we focus on validating the necessity of the MC anchoring loss by comparing against the TD-only variant. We conduct this ablation on the DI system due to its higher stochasticity.
| Method | Precision | Recall | MSE | Cum. seeds | Wall (min) | |
| RND + reach-avoid (TD+MC) | 0.942 | 0.955 | 0.949 | 0.006 | 53524 | 71 |
| RND + reach-avoid (TD only) | 0.855 | 0.940 | 0.912 | 0.015 | 72710 | 57 |
| Ensemble + unbiased (TD+MC) | 0.955 | 0.909 | 0.932 | 0.010 | 86464 | 48 |
| Ensemble + unbiased (TD only) | 0.904 | 0.615 | 0.731 | 0.031 | 180467 | 35 |
Table 1. Final-round mean results: TD-only vs TD+MC on DI.
As anticipated, the TD+MC objective outperforms the TD-only baseline across all metrics (Table I). Surprisingly, the TD-only loss yields an overall underestimation of the liveness value (by being overly optimistic), seemingly contradicting the theoretical conservatism established by Jensen’s inequality (see Remark). We attribute this to incomplete data coverage: the network, trained solely on local backups, extrapolates overly optimistic predictions into OOD states. Notably, under a weaker exploration configuration, ensemble + unbiased, the TD-only training completely collapses into an extremely noisy and conservative approximation. In contrast, adding the MC loss provides crucial grounding to stabilize training dynamics.
c) Q3: Sensitivity to and explorer horizon: Table II evaluates the sensitivity of our method to , the number of rollouts used to generate each MC target. Core metrics remain largely robust for , but degrade at where the target becomes a high-variance binary label (see Figure 5 for qualitative heatmaps). Here, the training efficiency scales clearly with : a larger budget yields a cleaner grounding signal that suppresses redundant exploration. This variance reduction shrinks the augmented dataset (cumulative seeds), reduces the number of active learning rounds, and lowers total training time. Combined with the findings from Q2, this confirms that MC anchoring consistently improves stability and accuracy while introducing minimal computational overhead.

Figure 5. DI value heatmaps across different .
Table 2 also examines the effect of the reach-avoid explorer’s training horizon. The ablation is conducted on the heavily underactuated Dubins system, where the horizon is expected to play a critical role. A moderate 10-step horizon (0.2 s) optimally balances the training time against the ability to actively discover a broad range of novel states. Conversely, an overly myopic horizon leads to poor exploration capacity, while an overly long horizon spikes the wall time and causes the exploration to collapse to a few deeply OOD states.
| Coverage | Precision | MSE | Cum. seeds | Wall (min) | ||
| 1 | 48666 | 80 | ||||
| 10 | 405432 | 80 | ||||
| 30 | 164003 | 78 | ||||
| 100 | 60884 | 79 | ||||
| 300 | 69518 | 79 | ||||
| 1000 | 53524 | 71 | ||||
| Horizon | Coverage | Precision | MSE | Cum. seeds | Wall (min) | |
| 1 | 29442 | 10 | ||||
| 5 | 25965 | 12 | ||||
| 10 | 27868 | 21 | ||||
| 25 | 27449 | 28 | ||||
| 50 | 30254 | 33 | ||||
| 100 | 27788 | 69 |
Table 2. Hyperparameter sensitivity (mean std over seeds). Top: DI sweep (RND + reach-avoid). Bottom: 3D Dubins reach-avoid horizon sweep ().
Unitree G1 Humanoid: Open-Loop Metrics
We now validate the framework’s scalability by certifying a Unitree G1 balancing policy in a 69D state space, simulated via mjlab [24] at 50 Hz. To specify the balancing task, we define the target function as , where bounds the base velocities, joint velocities , and torso height to a stationary upright posture. We evaluate over a 3-second liveness horizon, which extends well beyond the time typically required for to successfully recover balance. The state representation encompasses the torso’s height, roll and pitch angles, linear and angular velocities, as well as full-body joint configurations and velocities. Although some inputs are unnecessary for training a robust RL policy, the recoverability monitor requires them to ensure the mapping between the physical state and the predicted value remains injective. In hardware deployment, the body-frame velocities and height are estimated using a light-weight LSTM with proprioceptive inputs. The initial-state distribution comprises: (1) static canonical upright postures with base linear velocities m/s, and (2) nominal walking states with forward velocity m/s, and lateral/angular velocities .
Framework Design Choices. Guided by the toy case studies, we configure the active learning loop with the RND + reach-avoid explorer operating over a moderate 0.4 s training horizon. We set to balance the variance-reduction benefits of MC grounding against the significantly higher computational expense of humanoid rollouts. Crucially, inspired by the Dubins3D-Explore ablation, we carefully design the explorer’s control authority. Granting the explorer full whole-body joint control in this extremely high-dimensional space is prohibitively inefficient. Instead, we define its action space as learned hip-joint offsets (left and right; roll, pitch, and yaw) added to the nominal walking policy, alongside a longitudinal external torso push bounded by N, resulting in a compact 7D trainable action space for the explorer. We note that this formulation scales naturally: the explorer can be easily trained to output offsets conditioned on multiple distinct nominal policies (e.g., walking, running, whole-body manipulation, etc.) simultaneously by simply augmenting its input state with a policy indicator variable. We train both a continuous liveness value (where is labeled recoverable iff for metrics computation) and a failure probability using binary (labeled recoverable iff ).
Validation and OOD Test Set Generation
To evaluate the learned solutions, we generate a validation set by rolling out trajectories from using held-out seeds and labeling intermediate states, alongside a challenging OOD test set. We construct OOD states by warming up the environment with a periodic walking policy, then injecting a constant 100 N external force to a random link (pelvis, torso, or feet). This force is applied in a random horizontal direction (a body-frame forward cone for torso/pelvis, backward for feet) for exactly 1 s. States captured immediately upon release are binned and equalized by estimated post-disturbance difficulty . Finally, we assign liveness value labels to these states via 100 MC rollouts of per state.
Main Results. After 30 active-learning rounds (Table III, 6h wall time), attains an excellent score of on the validation set and maintains a highly robust of on the aggressive OOD push dataset, yielding confident coverages of 0.944 and 0.815, respectively. The estimator trades a marginal amount of classification accuracy ( Val / OOD) for better interpretability. Notably, both and coverage improve almost monotonically across rounds. The most substantial gains occur in OOD recall (, Figure 6, left), surging from 0.672 to 0.893 overall, and from 0.425 to 0.871 in the most difficult [3.0, 4.0) bin, as the explorer actively unearths boundary states that are unreachable by simply rolling out from . A similar trend appears in the estimator, though coverage expansion is less pronounced due to the inherently sparser gradient signal provided by probabilistic learning. While exhibiting lower overall recall, maintains high precision—meaning it rarely hallucinates false recoverability—which is critical for deploying a real-world monitor.
| Head | Set | Prec. | Recall | Coverage | |
| Value | Val | ||||
| Value | OOD | ||||
| Val | |||||
| OOD |
Table III. Humanoid results: mean std over 3 seeds.

Figure 6. (Left) Per-bin performance on the OOD push test (seed 0). Bins labeled by . (Right) Closed-loop failure/trigger rates across various maximum push impulse durations. The monitor with our TD+MC objective triggers least often while achieving the smallest failure rates.
Runtime Monitor and Hardware Deployment
After validating the open-loop metrics, we evaluate the recoverability monitors by simulating continuous walking under domain randomization and varying velocity commands
) m/s, rad/s). After a brief warm-up to establish a periodic gait, a multidirectional impulse is applied to a random body link (pelvis, torso, or foot, with tiered magnitudes of 150 N, 250 N, and 50 N, respectively). The impulse duration is sampled uniformly from , where sweeping from 0 to 0.30 s scales the disturbance from benign to severe across 4,096 8-second trials per sweep. The learned monitor switches the system to the balancing policy if the state is predicted to be unrecoverable () or highly uncertain (). We evaluate the monitor both on a nominal “healthy” walking policy (mimicking standard deployment phases) and a “degraded” policy subjected to severe observation noise and sensor blackouts (mimicking rapid development phases).
Learning Objective Comparison. We first evaluate the monitors trained under different learning objectives on the healthy walking policy across varying push durations (Figure 6, right). Our TD+MC monitor avoids excessive conservatism, raising zero false alarms during undisturbed walking or gentle pushes ( s). In contrast, the time-agnostic TD baseline [23] degenerates completely (triggering on of benign states), and the TD-only variant also over-triggers on normal walking, both of which would constantly interrupt task execution in practice. Despite this improved precision, our TD+MC monitor achieves the lowest failure rates across all disturbance bands, and its trigger rate strictly bounds the actual failure rate while predicting 100% of eventual falls in advance. Finally, even under the most severe disturbance band, our monitor provides an average lead time of 0.58 s between a high-confidence failure prediction () and torso-ground contact, securing sufficient margin to potentially activate a harm-minimizing fallback.
Performance Across Policies. Table IV details the closed-loop failure rates across the two policies. The monitor consistently reduces failure rates across all disturbance tiers, especially on the degraded walker. This confirms both the accuracy of the liveness value and the improved robustness of the fallback balancing policy. Aside from minor learning errors, the non-zero residual monitored failure rates stem from the highly stochastic nature of our simulation environment (aggressive domain randomization and observation noise). However, a physical robot represents only a single instantiation of these randomized parameters. Assuming the actual hardware dynamics fall well within the domain randomization bounds and physical sensor noise is more lenient, the empirical failure probability during real-world deployment is expected to be substantially lower. Finally, we qualitatively demonstrate the zero-shot performance on the G1 hardware in Figure 1 and the supplementary video.

Figure 1. Runtime recoverability monitoring using a liveness value function and uncertainty estimator on a G1 humanoid.
| (s) | Healthy Policy (%) | Degraded Policy (%) | ||||||||
| Unmon. Fall | Mon. Fall | Rel. Red. | Trigger | Unmon. Fall | Mon. Fall | Rel. Red. | Trigger | |||
| 0.00 | 0.00 | 0.00 | — | 0.0 | 35.01 | 10.55 | 69.9 | 93.3 | ||
| 0.05 | 0.00 | 0.00 | — | 0.0 | 34.23 | 10.89 | 68.2 | 93.1 | ||
| 0.10 | 0.63 | 0.44 | 30.8 | 2.8 | 37.65 | 11.67 | 69.0 | 93.8 | ||
| 0.15 | 2.98 | 2.59 | 13.1 | 9.2 | 41.21 | 13.28 | 67.8 | 94.4 | ||
| 0.20 | 5.86 | 4.59 | 21.7 | 13.0 | 43.60 | 16.02 | 63.3 | 94.1 | ||
| 0.25 | 8.59 | 7.13 | 17.0 | 16.2 | 44.04 | 18.31 | 58.4 | 94.3 | ||
| 0.30 | 12.21 | 8.94 | 26.8 | 20.2 | 47.75 | 20.75 | 56.5 | 94.8 |
Table IV. Closed-loop failure (fall) and trigger rates across maximum push durations for the healthy and degraded policies. Unmon.: Unmonitored; Mon.: Monitored; Rel. Red.: relative reduction in failure rate.
Conclusion and Limitations
In this work, we presented a framework combining a TD+MC learning objective with uncertainty-guided reach-avoid exploration to learn policy recoverability under stochastic dynamics. Our approach successfully scales to a 29-DoF humanoid, yielding an accurate and efficient runtime monitor. To our knowledge, this is the first effort to comprehensively map the liveness-set envelope for high-dimensional, contact-rich systems. Despite achieving promising outcomes, the framework currently acts reactively, rather than proactively, to OOD inputs. Furthermore, while empirically accurate, the learned solutions remain uncalibrated. Lastly, the maximum achievable coverage is fundamentally bounded by the choice of the explorer’s action space.
Looking forward, although our framework is theoretically amenable to any contact-rich system, its efficacy on complex dexterous manipulation, long-horizon or extremely dynamic tasks requires further investigation. In addition, extending it to general reach-avoid settings, where the robot must achieve goals while obeying safety specifications, will significantly broaden its practical impact. Ultimately, leveraging these learned reachability values to improve the quality and efficiency of policy synthesis stands as a natural next step.
Acknowledgments
We thank Nicholas Morozovsky, Yinai Fan, and Shravan Tangudu for proposing the humanoid test cases.
Appendix
Planar Double Integrator Let and . The state is , with initial states drawn uniformly from . The dynamics are and , with a target set . The nominal control is a noisy P-controller () with a saturation bound along each axis: , where and . As mentioned, the controller is flipped to steer away from the target for states where . The disturbance is , featuring an outward radial bias (, ) and distance-scaled noise .
Dubins Car A unicycle benchmark with state moving at a constant speed m/s, with initial states drawn uniformly from . The dynamics are , , and , sharing the same target as the double integrator. The nominal policy is a P-controller () steering the system to head toward the origin: , with , , and . Disturbances are biased: and . The Dubins3D-Explore variant augments the exploration control to with m/s, adding a lateral push perpendicular to the heading: and .
References
- [1]A. D. Ames, S. Coogan, M. Egstedt, G. Notomista, K. Sreenath, and P. Tabuada, “Control barrier functions: Theory and applications,” in European Control Conference, 2019, pp. 3420–3431.arxiv.org/abs/1903.11199
- [2]L. Yang, B. Werner, M. de Sa, and A. D. Ames, “Cbf-rl: Safety filtering reinforcement learning in training with control barrier functions,” 2026. [Online]. Available: https://arxiv.org/abs/2510.14959
- [3]R. M. Bena, G. Bahati, B. Werner, R. K. Cosner, L. Yang, and A. D. Ames, “Geometry-aware predictive safety filters on humanoids: From poisson safety functions to cbf constrained mpc,” in 2025 IEEE-RAS 24th International Conference on Humanoid Robots (Humanoids). IEEE, 2025, pp. 1–8.arxiv.org/abs/2508.11129
- [4]L. Yang, B. Werner, R. K. Cosner, D. Fridovich-Keil, P. Culbertson, and A. D. Ames, “Shield: Safety on humanoids via cbfs in expectation on learned dynamics,” in 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2025, pp. 203–210.DOI
- [5]P. M. Esfahani, D. Chatterjee, and J. Lygeros, “The stochastic reach-avoid problem and set characterization for diffusions,” Automatica, vol. 70, pp. 43–56, 2016.DOI
- [6]H. Hoshino and Y. Nakahira, “Physics-informed reinforcement learning for stochastic reach-avoid analysis,” arXiv preprint arXiv:2608.17117, 2026.arxiv.org/abs/2608.17117
- [7]S. Bansal and C. J. Tomlin, “Deepreach: A deep learning approach to high-dimensional reachability,” in 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1817–1824.
- [8]Z. Feng, L. Qiu, and S. Bansal, “Bridging model predictive control and deep learning for scalable reachability analysis,” in Robotics: Science and Systems, 2025.arxiv.org/abs/2505.03830
- [9]V. K. Chikamarri, Z. Feng, and S. Bansal, “Reachability analysis for black-box dynamical systems,” in 2025 IEEE International Conference on Robotics and Automation (ICRA), 2025, pp. 3552–3558.
- [10]O. So, C. Ge, and C. Fan, “Solving minimum-cost reach avoid using reinforcement learning,” in Advances in Neural Information Processing Systems, vol. 37. Curran Associates, Inc., 2024.arxiv.org/abs/2410.22600
- [11]O. So, Z. Serlin, M. Mann, J. Gonzales, K. Rutledge, N. Roy, and C. Fan, “How to train your neural control barrier function: Learning safety filters for complex input-constrained systems,” in 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024.arxiv.org/abs/2310.15478
- [12]D. Morton, P. Mohnot, and M. Pavone, “Constrained whole-body tracking for humanoid robots,” 2026. [Online]. Available: https://arxiv.org/abs/2606.00374
- [13]J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin, “Bridging hamilton-jacobi safety analysis and reinforcement learning,” in International Conference on Robotics and Automation (ICRA), 2019.DOI
- [14]K.-C. Hsu*, V. Rubies-Royo*, C. Tomlin, and J. Fisac, “Safety and liveness guarantees through reach-avoid reinforcement learning,” in Robotics: Science and Systems, 2021.DOI
- [15]H. Wang, J. Bowden, C. Crosby, and S. Bansal, “Offline policy evaluation for manipulation policies via discounted liveness formulation.” Robotics: Science and Systems, 2026.arxiv.org/abs/2605.11479
- [16]R. Chen, S. Li, Y. Sun, and C. Liu, “λ-reachability: Geometric-horizon safety bellman equations for humanoid safety,” 2026. [Online]. Available: https://arxiv.org/abs/2606.16022
- [17]J. Long, P. Abbeel, K. Sreenath, R. Horowitz, G. Shi, and C. K. Liu, “Humanoid safe stop via learned stoppability value,” 2026. [Online]. Available: https://arxiv.org/abs/2609.02358
- [18]J. Seo, K. Nakamura, and A. Bajcsy, “uncertainty-aware latent safety filters for avoiding out-of-distribution failures.” Conference on Robot Learning (CoRL), 2025.arxiv.org/abs/2505.00779
- [19]I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration via bootstrapped dqn,” Advances in neural information processing systems, vol. 29, 2016.arxiv.org/abs/1602.04621
- [20]I. Osband, B. Van Roy, D. J. Russo, and Z. Wen, “Deep exploration via randomized value functions,” Journal of machine learning research, vol. 20, no. 124, pp. 1–62, 2019.
- [21]H. Tang, R. Houthooft, D. Foote, A. Stooke, O. Xi Chen, Y. Duan, J. Schulman, F. DeTurck, and P. Abbeel, “# exploration: A study of count-based exploration for deep reinforcement learning,” Advances in neural information processing systems, vol. 30, 2017.arxiv.org/abs/1611.04717
- [22]Z. Feng, A. F. Sahin, S. Thorup, and S. Bansal, “Gradient-free neural hamilton-jacobi reachability for scalable safety-critical control,” 2026. [Online]. Available: https://arxiv.org/abs/2609.14087
- [23]J. F. Fisac, N. F. Lugovoy, V. Rubies-Royo, S. Ghosh, and C. J. Tomlin, “Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning,” International Conference on Robotics and Automation, 2019.DOI
- [24]K. Zakka, Q. Liao, B. Yi, L. L. Lay, K. Sreenath, and P. Abbeel, “mjlab: A lightweight framework for gpu-accelerated robot learning,” 2026. [Online]. Available: https://arxiv.org/abs/2601.22074