WorksComputer Science
Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing
Abstract
Vision-Language-Action models (VLAs) enable general-purpose embodied intelligence, but their performance is brittle under distribution shift, where unanticipated failures can damage equipment or endanger people. Deploying VLAs in real-world systems therefore requires a runtime monitor that is timely, calibrated, and informative while being efficient enough to run on the edge. Existing methods fail to fulfill these requirements. We present VLA-Tug, a lightweight probe over the latent layers of a VLA backbone that improves early failure detection and bounds the missed-detection and false-alarm rates, with minimal inference overhead. We show that supervising on the pre-failure segment of a trajectory increases early failure detection at matched false-positive rates by up to 30% over the strongest learned baseline. Under shift to unseen tasks, the bounds on missed-detection and false-alarm rates remain valid in up to 86% of runs, versus at most 21% for baselines. Unlike existing methods, VLA-Tug predicts the failure mode to enable corrective actions, raising top-3 accuracy on unseen tasks by up to 44% over a majority baseline. VLA-Tug enables real-time monitoring where the well-timed and informative probe supports lightweight steering that improves task success. We release a dataset of 3,180 annotated manipulation episodes to support future work on latent failure detection in VLAs at https://vla-tug.github.io.
Introduction
Vision-Language-Action models (VLAs) (Physical Intelligence et al., [21] (2026); NVIDIA et al. (2025)) are strong candidates for general-purpose embodied intelligence. By pairing a VLM backbone pre-trained on internet-scale data with an action expert fine-tuned on demonstration trajectories, they excel at complex dexterous tasks such as tightening screws or assembling cardboard boxes. Yet their performance is brittle and can degrade significantly under distribution shift, such as changes in lighting, novel textures, or rare object combinations (Wang et al., [27]; Fei et al. [7]). A VLA can therefore fail unexpectedly, especially as task and environment novelty increases, and such failures can be costly, damaging materials and robots, or outright dangerous to humans. This brittleness has slowed the adoption of VLAs in factories, homes, and public spaces.
Among multiple solutions, one solution that the robotics community is exploring is VLA steering. The idea of steering comes from research on LLMs, where activations are selectively modified to nudge a model towards a desired output (Turner et al., [26]). Steering VLAs is particularly interesting because it can unlock behaviors that cannot be reached through prompting alone (Buonamiger et al., (2026); Swann et al. [24]). Various steering methods have recently been proposed (Li et al., [14]; Liu et al. [16]; Tan et al. [25]; Swann et al. [24]; Cui et al. [6]; Jeong et al. [11]; Li [12]), but they rarely use a calibrated probe to decide when to steer. Some steer continuously (Li et al., [14]; Liu et al. [16]; Cui et al. [6]), some rely on human-specified offline triggers (Swann et al., [24]; Li [12]), and others are gated by a one-sided calibrated trigger (Tan et al., [25]; Jeong et al. [11]). Overall, they lack an efficient and principled trigger that would make them deployable in real-world systems.
Research on failure detection for VLAs is becoming increasingly mainstream (Gu et al., (2025); Agia et al. [1]; Ma et al. (2026); Huang et al. [10]). In a nutshell, these methods compute a failure score from signals available in the VLA, often its latent variables (Gu et al., 2025) or predicted actions (Agia et al., [1]; Huang et al. [10]; Ma et al. (2026)), that encode how likely the policy is to fail at the current task. A detection threshold is then calibrated to give probabilistic guarantees on the detector’s performance (Gu et al., 2025). However, these guarantees often cover
Preprint. only the false-positive rate (FPR), i.e. the rate of successes misclassified as failures, and not a target true-positive rate (TPR), i.e. the rate of correctly classified failures, which is arguably even more important in safety-critical settings where a single failure could harm a person or damage costly equipment. These methods are also seldom evaluated systematically against ground-truth failure times, which we argue is crucial to assess whether they can prevent failures before they happen and thus enable VLA steering. Finally, they cannot predict failure modes (grasping, pushing, opening), so steering cannot be targeted towards the root cause and would require human interventions.
We argue that steering would be best supported by a probe that is (T.) timely, detecting failures before they occur; (I.) informative, outputting semantic information beyond a numerical score; (C.) calibrated, achieving the desired error rates; and (E.) efficient to train, calibrate, and deploy.
We address this with VLA-TUG1, a well-behaved probe designed to support VLA steering in the real world. We train VLA-TUG on the latent layers of two VLAs (Physical Intelligence et al., [21]; NVIDIA et al., 2025) using 228 distinct tasks in two simulation environments and 8 real-world tasks, and evaluate it against ground-truth failure times. We show that supervision should focus on the pre-failure part of the trajectory to improve early failure detection, and that calibrating the detection threshold to guarantee both TPR and
FPR simultaneously yields guarantees that hold far more often, including on unseen tasks. Building on these findings, we propose a real-time architecture that incorporates VLA-TUG to autonomously support the detection and prevention of VLA failures. Our contributions are as follows:
VLA-TUG: A timely, informative, and efficient probe for early failure detection. On unseen tasks, our probe improves the area under the early-detection curve by up to 30% over the strongest learned baseline. VLA-TUG also predicts the failure mode, raising the top-3 accuracy on unseen tasks by up to 44% over a majority baseline, at under 1 ms per step.
Guarantees on false alarms and missed detections. We certify FPR and TPR targets jointly; both held in every certified run, compared to 57–71% of runs for baselines. Under distribution shift to unseen tasks, the false-alarm target still held in 40–86% of certified runs, compared to 0–21% for baselines.
A deployable failure-monitoring system. Through real-world experiments, we demonstrate that integrating the timely failure predictions and associated modes with steering strategies can lead to enhanced success rates.
An annotated VLA failure dataset. We release a dataset of ≈289k policy-step VLA backbone embeddings from annotated 3,180 episodes of diverse tasks.
Related Work
Existing methods can detect failures, but their alarms are not necessarily timely, informative, or sufficiently calibrated. Our method addresses these limitations through failure-aware representations, onset-based detection, and joint statistical guarantees.
Existing works and their limitations. Existing VLA failure detectors leverage different signals to identify potential failures, including external assessments from world models or LLM judges (Liu et al., 2026b; Li et al., 2026b), action consistency across sampled or consecutive action chunks [1, 10], distributional deviations in policy inputs, outputs, or embeddings (Xu et al., 2025; Gu et al., 2025), and internal policy representations [23, 4, 20, 8] (Gu et al. 2025; Ma et al. 2026).
These approaches have three key limitations. (1) Detection performance is commonly evaluated at fixed time points using metrics such as AUROC or balanced accuracy [10] (Gu et al. 2025), without explicitly measuring whether alarms precede the actual failure onset. Moreover, supervision ranges from episode-level failure labels assigned to all timesteps (Gu et al., 2025), to heuristic or human-refined timestep labels (Ma et al., 2026), to failure-free training (Xu et al., 2025), without explicitly accounting for the timing of failures. (2) Most detectors produce scalar risk scores without identifying the underlying failure modes. While semantic supervision and representation interpretability have been explored for embodied policies [24, 3] (Xu et al. 2024), they do not directly establish informative failure alarms. (3) Existing calibration methods typically control a single criterion. Conformal methods provide false-positive guarantees for sequential failure detection [23, 20] (Gu et al. 2025), calibrate anomaly or failure scores (Xu et al., 2025; Ma et al., 2026), or provide guarantees for one-shot policy compliance and embodied decision making [30, 5, 2, 31]. However, these guarantees do not jointly account for false alarms and timely failure detection.
VLA-TUG addresses the limitations. We address these limitations by developing a failure detector that is timely, informative, and jointly calibrated. Following the internal-representation probing direction, we reuse the VLA’s latent representations and align them with failure-mode semantics, enabling the detector to identify both potential failures and their types. To evaluate timeliness, we measure detection relative to annotated failure onsets and investigate supervision strategies that explicitly account for when failures occur. To provide statistical guarantees, we adopt Learn-
then-Test (Angelopoulos et al., 2022) to jointly calibrate multiple detection criteria, including the false-positive rate (FPR) and early true-positive rate (TPR). Existing failure detectors primarily raise alarms without steering, while steering methods rely on predefined schedules or calibrated triggers. In contrast, to the best of our knowledge, VLA-TUG is the first framework to combine joint FPR and early-TPR calibration, failure-onset-aware evaluation, and failure-mode prediction for calibrated failure-triggered steering. Table 1 compares VLA-TUG against the existing works.
| Method | Trigger | FPG | TPG | EE | FM |
| Failure detectors (no steering) | |||||
| SAFE (Gu et al., 2025) | one-sided conformal | ✓ | × | o† | × |
| ActProbe [10] | split conformal | ✓ | × | ׇ | × |
| STAC [1] | one-sided conformal | ✓ | × | ׇ | × |
| FailureSpot (Ma et al., 2026) | one-sided conformal | ✓ | × | o† | × |
| Steering methods | |||||
| VLA-Pilot [14] | always on | – | – | – | – |
| VLS [16] | always on§ | – | – | – | – |
| DreamSteer [6] | fixed period | – | – | – | – |
| SAE-VLA [24] | human-specified | – | – | – | – |
| TLI [12] | human-specified | – | – | – | – |
| R-VLA [25] | one-sided conformal | ✓ | × | × | × |
| LFP [11] | one-sided conformal | ✓‡ | × | × | × |
| VLA-TUG (Ours) | jointly calibrated | ✓ | ✓ | ✓ | ✓ |
Table 1. Comparison with related methods. FPG / TPG: formal calibrated guarantee on the false-positive / true-positive rate. EE: early evaluation, i.e. a per-episode criterion against annotated failure time; o denotes partial use. FM: predicts the failure mode. Trigger: what decides when steering happens. A dash means n/a.
Problem Statement
In this section, we start by giving several broad definitions used to describe how failure detectors for VLAs function. Next, we give a mathematical formulation of our problem and then introduce how VLA-TUG addresses it.
Definitions
To facilitate comparisons across methods, we give general definitions for the concepts of failure score, detection threshold, detection time, failure time, and failure mode.
Failure score. The score , defined at every timestep , encodes the probe’s belief that the VLA is failing. It need not be a probability in the strict sense.
Detection threshold. The threshold over separates failures from successes.
Detection time. The time is
the first time the score crosses the threshold.
Failure time. The time , which we also call the failure onset, is when the policy makes an obvious mistake or its progress stalls to the point where an intervention, human or otherwise, is needed.
Failure mode. The label is a high-level semantic category assigned to each failure, such as grasping failure, placement failure, or no progress.
These definitions are general, and their exact form varies slightly across methods. The interpretation of depends on the training strategy, but a higher score usually signals failure and a lower score success. The threshold may be fixed or vary with time, and it is often calibrated with probabilistic guarantees on a set of calibration rollouts . If for all , the detection time is undefined, and the rollout raises no alarm. The failure time definition follows (Gu et al., 2025).
Problem Formulation
Consider a VLA policy and a rollout of policy steps. At each policy step , takes as input RGB observations (agent and wrist cameras) and a natural language prompt , and produces a chunk of control actions . We denote the VLA’s internal embedding at step as , so that . Each rollout has an episode-level label , where marks a failure and a success. If , the rollout also has a failure time and a failure mode , where is a fixed taxonomy of failure modes.
Problem. Given a training set and a calibration set , and access to the internal embeddings of , predict an imminent task failure before it happens, i.e. where is the detection time, and predict the failure mode in a manner that holds up empirically on tasks unseen during calibration.
We propose VLA-TUG, a probe that is trained and calibrated to detect failures before they happen. It is:
(T.) Timely. We propose a training and evaluation strategy that accounts for the failure time and improves early failure detection.
(C.) Calibrated. We introduce a risk formulation that jointly bounds the false-positive rate and the early miss rate , and examine how well these targets hold on unseen tasks.
(I.) Informative. We design VLA-TUG to predict the failure mode, giving the semantic information needed to choose a corrective action.
(E.) Efficient. VLA-TUG adds minimal latency and parameters to the VLA, and is cheap to train and calibrate.
Methodology
In this section we introduce VLA-TUG. In subsection 4.1 we present the probe and how we train it, and in subsection 4.2 we show how we calibrate its alarm threshold and what guarantees this gives.
Failure Detection Through Latent Space Probing
Building on existing works (Gu et al., 2025), we instantiate the probe as a combination of LSTMs and MLPs, the goal being to capture a non-linear pattern in the backbone embeddings that generalizes beyond the seen episodes. We briefly introduce the model inputs, followed by a discussion of the model architecture and finally the training strategy.
We use VLA backbone embeddings as the inputs. We explore two distinct layers: backbone layer 18 (after normalization) in and layer 16 in GROOT-N1.7, focusing on the latter in particular. At every policy step we average the backbone tokens of each modality (agent camera, wrist camera, instructions) and concatenate the three 2048-dimensional vectors into a single embedding .
We leverage a combination of LSTMs and MLPs. A single-layer LSTM with 256 hidden units reads the sequence of embeddings and produces a latent state at every policy step. Three MLP heads read this shared latent state (see Figure 1). The failure score head predicts the failure score, . To make the probe informative, the mode head predicts a distribution over the 10 failure modes , . Following (Xu et al., 2024), an auxiliary head, , aligns the latent state with a CLIP embedding of the description of the current action. Each head is an MLP with one hidden layer of 64 ReLU units. The probe has 6.6M parameters.

Figure 1. We introduce VLA-TUG, a probe on the VLA backbone embeddings that predicts failures and their mode. (a) A recurrent encoder maps the embedding into a latent failure space; a score head and a calibrated trigger raise the alarm, and a mode head then predicts the failure mode. (b) VLA-TUG alarms before the failure, allowing early stopping or intervention by an expert model.
We train the probe to predict features that precede a failure. We hypothesize that training on post-failure embeddings teaches the probe what already failed trajectories look like, not what trajectories about to fail look like. If so, to predict failures the probe must be trained on the embeddings before the failure time (pre-onset supervision), rather than after it (Ma et al., 2026) or at all times of failed rollouts (Gu et al., 2025). To test this, we compare four ways of labeling the embeddings of failed rollouts (Figure 2; the mathematical formulation is in App. § 8.§ 2). Successful rollouts are labeled 0 in all four. The window in onset and post-onset is the time span before that is labeled as failure. Pre-onset and onset truncate the rollout at the failure time, while post-onset and trajectory keep all steps. We exclude the steps after in the first two strategies instead of labeling them 0: these embeddings already show a failed rollout, so labeling them as successes would give the probe contradicting training signals.

Figure 2. How the embeddings of a failed rollout are labelled for the four supervision strategies.
We optimize three objectives. With the labels above, the probe is trained with
with . is a binary cross-entropy over all steps that are not excluded from the loss, weighted by the ratio of negative to positive steps in the training set. is a cross-entropy against the true failure mode, applied to the positive steps of failed rollouts and weighted to balance the modes. is an auxiliary contrastive loss that aligns with the CLIP embedding of the step’s description. We train with Adam (learning rate , batch size 32 rollouts, at most 100 epochs) on standardized inputs, with early stopping on the validation (patience 15).
Uncertainty Quantification and Guarantees
Where previous works only provided guarantees of false-positive rates (i.e. the rate of successful rollouts that were flagged as failures) [23] (Gu et al. 2025), they did not provide guarantees on the false-negative rate (i.e. the rate of unsuccessful trajectories that were not flagged). We will first formulate the risks and show how to calibrate the detection threshold . In an effort to make these bounds interpretable to the practitioner, we compute the probability of failure given that a rollout was flagged.
VLA-TUG guarantees both FPR and TPR. In general, the FPR is an operational metric that we wish to minimize to prevent unnecessary steering, the TPR is a safety metric that we wish to maximize to prevent undetected failures. We can control these risks using the Learn-then-Test framework (Angelopoulos et al., 2022). Let be a constant detection threshold, and the failure score produced by our probe at each timestep for each rollout . Furthermore, let the detection time be the first time the failure score exceeds the threshold . Finally, the ground truth failure time remains the first time when failure becomes apparent or a human operator would choose to take over. We propose to formulate the above mentioned risks as follows:
where and are the sets of successful and failed rollouts , respectively, in the calibration set . We are interested in finding the set of values that satisfy:
where and are user-selected rates. This guarantee can be understood as saying that any guarantees that the false-positive rate (FPR) is at most and the early true-positive rate (TPR) is at least , with high-probability of no less than . We will see, in the experimental section, that this translates to guarantees that empirically behave better under distribution shift. Following (Angelopoulos et al., 2022) we select a set of candidate values ; from this set of values we select the value that maximizes .
Probability of failure given that an alarm is raised. Since our goal is a probe that can be deployed in real-world systems, we complement the formal TPR and FPR guarantees with an actionable interpretation: a bound on the positive predictive value (PPV). The PPV is the probability that failure occurs given score exceeds the threshold . Existing calibration methods (Gu et al., 2025) cannot give such a bound, since they lack guarantees on the TPR.
Let FAILURE denote rollout-level failure, and let EXCEED denote the event that the score exceeds the threshold at some time . The PPV is . Define the failure prevalence , the true positive rate , and the false positive rate . By Bayes’ rule,
Detections that occur before the failure onset are a subset of all detections, so . Our guarantees give and . Since the PPV is non-decreasing in TPR and non-increasing in FPR, substituting the worst-case values yields the conservative bound:
The bound increases with , so a lower bound on the prevalence suffices. Given an estimate of , this gives practitioners an intuitive reading of the guarantees.
Dataset
To evaluate VLA-TUG, we create a labeled dataset of rollouts of two VLA policies, (Physical Intelligence et al. [21]) and GR00T-N1.7 (NVIDIA et al. (2025)), with failure labels and semantic descriptions of the robot’s actions. We first describe the benchmarks and the splits, and then the labeling pipeline.
Benchmarks and splits. We use LIBERO (Liu et al. [15]), LIBERO-Occ (Li et al. (2026a)), and LIBERO-X (Wang et al. [27]), as well as real-robot tasks. Both policies succeed on most LIBERO episodes, so we combine it with LIBERO-Occ to obtain enough failures. We propose splits of increasing distribution shift to evaluate generalization in a principled way; Table 2 gives how the tasks are held out and their counts. In LIBERO-X, each task has a skill, the manipulation behavior (such as pushing a button or stacking objects), and a signature, the combination of its skill, target class, relation, landmark, and horizon. The splits are:
| Dataset | Split | Seen tasks | Unseen tasks |
| LIBERO | leave-three-out | 7 | 3 |
| LIBERO-X | NEW-TASK | 175 | 43 |
| LIBERO-X | NEW-SIGNATURE | 162 | 56 |
| LIBERO-X | NEW-SKILL | 159–209 | 9–59 |
| Real-world | leave-one-out | 7 | 1 |
Table 2. Task count by dataset and split. There are 8 skills, each with a different seen/unseen split.
NEW-TASK holds out individual tasks; their skill and signature can still occur in the training tasks, the shift is mild.
NEW-SIGNATURE holds out all tasks with a given signature: the components are familiar, but their combination is new.
NEW-SKILL holds out a complete skill, so the tested behavior is absent from training: this is the largest shift of the three.
Efficient labeling pipeline. We label each rollout from its video with Gemini (gemini-3.5-flash), as in Figure 3. Every rollout receives a timeline label, and failed rollouts also receive a failure mode label and a failure time label. The timeline is a short keyword phrase per 3-second window

Figure 3. A successful (top) and a failed (bottom) rollout with their timeline, failure time, and failure mode labels.
that we use for CLIP alignment. The failure mode comes from a 10-way taxonomy . The failure time is the earliest point at which the mistake responsible for the failure becomes apparent or, following Gu et al. (2025), the time a human operator would take over if none is apparent.
We open-source this dataset including over 288,000 embeddings from 3,180 labelled rollouts at vla-tug.github.io.
Experimental Results
In this section, we answer five questions based on our problem formulation (§ 3): (Q1. Timeliness) Does training on features that precede the failure onset improve early failure detection on unseen tasks (subsection 6.1)? (Q2. Calibration) Can VLA-TUG jointly bound the FPR and , with bounds that hold on seen tasks and degrade gracefully on unseen tasks (subsection 6.2)? (Q3. Informativeness) Can VLA-TUG predict the failure mode based on VLA backbone embeddings (subsection 6.3)? (Q4. Efficiency) Are the cost and time of labeling, training, and inference appropriate for deployment in real-world systems (subsection 6.4)? (Q5. Steering) Can VLA-TUG, as a timely, informative, calibrated, and efficient probe, support VLA steering (subsection 6.5)? We ablate our design choices in App. D.4.
Failure Detection and Timeliness Evaluation
We evaluate VLA-TUG against a series of baselines using metrics that depend on the ground-truth failure time. Firstly, we describe how we split the data. Secondly, we review the baselines. Thirdly, we formally define our evaluation metrics. Lastly, we discuss the results.
Data split. For every run, we split the episodes as in Figure 4. The training set fits the probe, the validation set selects the checkpoint, and the threshold is chosen on the calibration set. The calibration and held-out seen sets are exchangeable, and our guarantees apply to the held-out seen set. The unseen set, which the probe and the threshold never see, measures robustness to distribution shift. On LIBERO-X, the held-out tasks of the NEW-TASK and NEW-SIGNATURE splits are the same for every seed, because they are defined by the semantic axis described in § 5; the seed changes Table 3. Early failure detection on seen and unseen tasks: area under the TPR early vs. FPR curve, averaged over splits. Seen: held-out episodes of training tasks; unseen: held-out tasks. Cal.: calibrator (MC max-conformal, FC functional conformal). Cost on LIBERO-X (lower is better): fit time, memory to score a new episode, and per-step latency on a GPU and a CPU. Dashes: not available. Best in bold, second best in italics.

Figure 4. Task- and episode-level data splits. Episodes from seen tasks are divided into training (65%), validation (15%), calibration (10%), and held-out seen (10%) sets. Unseen tasks form a separate evaluation set. Task-level proportions are schematic.
| Method | Cal. | LIBERO-10 Seen | LIBERO-10 Unseen | LIBERO-10 GR00T Seen | LIBERO-10 GR00T Unseen | LIBERO-X task Seen | LIBERO-X task Unseen | LIBERO-X signature Seen | LIBERO-X signature Unseen | LIBERO-X skill Seen | LIBERO-X skill Unseen | Real robot Seen | Real robot Unseen | Fit (s) | State (MB) | GPU (ms) | CPU (ms) |
| SAFE-LSTM | FC | 75.5 | 34.9 | 74.1 | 51.2 | 64.6 | 40.4 | 64.7 | 41.2 | 62.5 | 30.1 | 44.5 | 25.1 | 43.5 | 26.2 | 0.066 | 0.86 |
| SAFE-MLP | FC | 74.1 | 39.9 | 62.9 | 39.2 | 59.3 | 42.3 | 53.7 | 46.2 | 56.8 | 23.5 | 40.1 | 20.2 | 114.5 | 6.3 | 0.034 | 0.032 |
| STAC | MC | 33.8 | 36.4 | 24.2 | 17.5 | 21.0 | 28.0 | 25.6 | 28.7 | 24.6 | 21.5 | – | – | 0.0 | 0.0 | 21.30 | – |
| ActProbe | MC | 39.3 | 26.7 | 32.4 | 30.1 | 12.6 | 22.7 | 11.9 | 13.9 | 20.3 | 13.6 | – | – | 7.1 | 0.09 | 0.15 | 0.11 |
| FailureSpot | FC | 61.9 | 36.2 | 36.9 | 44.7 | 26.3 | 20.4 | 28.0 | 26.2 | 30.4 | 19.6 | – | – | 9.2 | 6.4 | 0.033 | 0.029 |
| Mahalanobis | FC | 85.7 | 40.2 | 70.7 | 29.2 | 69.6 | 30.6 | 64.4 | 22.3 | 71.2 | 15.9 | 43.0 | 19.6 | 15.2 | 302.0 | 0.41 | 5.37 |
| Euclidean | FC | 73.1 | 24.2 | 56.0 | 23.1 | 70.5 | 28.5 | 65.4 | 19.7 | 68.5 | 19.0 | 45.1 | 8.3 | 0.4 | 3056 | 13.19 | 368.83 |
| PCA-KMeans | FC | 35.7 | 12.9 | 44.5 | 20.0 | 31.7 | 25.0 | 35.3 | 32.5 | 37.8 | 20.7 | 24.0 | 19.4 | 39.9 | 1.6 | 0.12 | 0.055 |
| VLA-TUG (Ours) | MC | 75.1 | 47.9 | 70.5 | 47.4 | 67.9 | 40.2 | 67.0 | 47.0 | 67.8 | 39.0 | 47.6 | 25.8 | 6.8 | 26.5 | 0.12 | 0.94 |
only the shuffle of the seen episodes and the initialization. The NEW-SKILL folds and the LIBERO-10 seeds hold out different tasks.
Baselines. We consider the following baselines:
Embedding distance (Mahalanobis, Euclidean, PCA-KMeans). A step is scored by how much closer its backbone embedding is to failed than to successful training steps.
Action consistency (STAC, ActProbe, FailureSpot). Steps are scored based on the discrepancy between predicted and executed steps.
Latent (SAFE). A step is scored by a learned projection of its backbone embedding.
Timeliness Metrics. We evaluate VLA-TUG against three key metrics: TPR, , and FPR. Existing works often evaluate their methods using AUROC [10] (Gu et al. 2025; Ma et al. 2026) without accounting for the true failure times, making it hard to distinguish timely detections. The TPR and FPR follow their standard definitions. The is the rate of failures detected before the failure time, i.e. . We additionally define as the area under the -FPR curve. We compute it on the support FPR , normalized by its width 0.4 (reported in %), since a detector that raises false alarms on more than 40% of successful rollouts is impractical to deploy. We formally define these metrics in App. § 10.§ 1.
Results and Discussion. We ask whether training on features that precede the failure onset improves early failure detection on unseen tasks. Our hypothesis is that it does: post-failure features only describe failures the probe has already seen, so training on them keeps it from learning a general notion of what leads to a failure, while features before the onset carry the cues that come before one. We compare VLA-TUG against the baselines in Table 3 and Figure 5. On unseen tasks, our domain of interest, VLA-TUG is the best method in four of six settings: LIBERO-10 with , NEW-SIGNATURE, NEW-SKILL, and the real robot. Against SAFE-LSTM, the strongest learned baseline, its margin grows from on NEW-TASK to on NEW-SIGNATURE and on NEW-SKILL. On the real robot, the lead is small, and VLA-TUG does not lead on LIBERO-10 with GR00T. On seen tasks, the results are mixed. We ablate the label in App. § 11.§ 4: replacing the pre-onset label with the other variants lowers early detection on LIBERO-X.

Figure 5. Early detection on seen (top) and unseen (bottom) tasks: the share of failures detected at or before the failure time () against the false-positive rate, for VLA-TUG and the learned and action-based baselines. Each curve is the mean over the splits of a setting, and the area under it up to 40% FPR is the metric of Table 3. Panel titles give the benchmark, the policy, and the held-out split.
Answer to Q1. Timeliness. Yes: for most unseen settings. Training on features that precede the failure onset lets VLA-TUG raise the alarm at or before the failure onset more often than the baselines, and on LIBERO-X the gap widens as the test tasks move away from the training tasks.
Calibration Evaluation
We validate the theoretical results from § 4 on seen tasks and examine how they hold under distribution shift on unseen tasks. We calibrate our probe with Learn-then-Test (LTT, ) at seven false-positive-rate (FPR) targets (10 to 50%), each certified together with a TPR early target, on the 14 LIBERO-X runs (3 NEW-TASK seeds, 3 NEW-SIGNATURE seeds, and 8 NEW-SKILL folds). We compare against max conformal (MC) [1] and functional conformal (FC) (Gu et al., 2025) calibration (the complete table is in App. § 11.§ 1). We draw three conclusions:
LTT bounds the together with the FPR, which hold in all certified runs on seen tasks and in 40–86% (FPR) and 57–90% (TPR early) of runs on unseen tasks; baselines give no guarantee on the TPR.
LTT states upfront whether the system can support the required guarantee whereas existing methods can give a false sense of safety. LTT abstains on 4 to 7 of the 14 runs, depending on the target (FPR targets of 15–50%), which tells us before deployment that the model cannot support the requested rates. MC always returns a threshold, but it meets the FPR target in only 57–71% of runs on seen tasks and in at most 21% on unseen tasks.
Our LTT targets give a lower bound on the positive predictive value (PPV), the probability that an alarm is a true failure. It is 58–71% across the FPR targets of the full table and holds in every certified run on seen tasks and in 70–100% of runs on unseen tasks. Methods without a TPR guarantee cannot give it, and it tells the practitioner what an alarm means.
On unseen tasks, the guarantees hold only empirically, since new samples are not exchangeable with the calibration set, which applies to all three methods. The shares above are trends over 14 runs, not precise rates.
Answer to Q2. Calibration. Yes: the rates we calibrate for are the rates we get on seen tasks. Under distribution shift, they hold in far more runs than with MC and FC, and when a target cannot be supported, LTT abstains instead of silently missing it. Together with the PPV bound, this makes VLA-TUG easier to integrate into real-world systems.
Evaluation of Informativeness
Here we focus on failure mode prediction, a crucial element to enable failure prevention through steering. Failure modes are strongly imbalanced, so we report balanced accuracy, the mean over classes of the share of their failures predicted correctly. Since existing baselines do not predict failure modes, we compare against a majority baseline and focus on LIBERO-X.
On LIBERO-X, the probe improves top-1 accuracy over the majority baseline by up to 149% on seen and 79% on unseen tasks, and top-3 accuracy by a smaller but consistent margin on both. On the real robot, it improves top-1 by 171% and 8% on seen and unseen tasks, respectively, and stays well above the baseline in top-3 on both.
The practical consequence is that the probe narrows a failure down to three candidate classes, which are more likely than the majority baseline to include the true one. Different failure modes carry different risks, so a practitioner can use these candidates to select the level of response, for example responding to the riskiest of the three predicted classes (a stop, a retry, or a handover to a human) instead of applying the same intervention to every failure.
Answer to Q3. Informativeness. Yes: backbone embeddings carry enough information to predict the failure class above a majority baseline, which lets a practitioner match the response to the predicted risk.
Practical Considerations
Any real-world system must balance performance (T.) Timeliness, (C.) Calibration, (I.) Informativeness) with the cost of building and running it ((E.) Efficiency). We first evaluate the cost of labeling the dataset, then of training the probe, calibrating it, and running it at inference time.
Dataset. Labeling costs about one cent per failed episode (under $50 for all annotations, including queries not used in this paper). Labels are therefore cheap, and practitioners will often already have unlabeled rollouts from system deployment.
Training. VLA-TuG trains in 6.8 s on one RTX 4090, against 43.5 s for SAFE-LSTM, and holds 6.6M parameters (26.5 MB), about 0.2% of the VLA. It is therefore cheap enough to retrain whenever the policy or the task set changes, or to train online as new rollouts are produced.
Calibration. Calibrating takes under 2 ms per target pair on a CPU, so VLA-TuG can be re-calibrated if system requirements change, or online along with the probe.
Inference. The VLA outputs a chunk of actions per call (8 at 20 Hz on LIBERO-X), so a policy step lasts 400 ms, and the probe reads the embedding of each call. It adds 0.12 ms per step on the GPU (0.2% of the 54 ms GR00T policy call) and 0.9 ms on four CPU threads, so its score is available before the chunk is executed and its latency is not observable: the step period is about the combined VLA and probe time in simulation and about on the real robot on our GPU.
Faster baselines offer diminishing benefit, since the policy frequency bounds how fast a detector needs to be, while slower ones such as k-NN and STAC add noticeable latency to every step. VLA-TUG costs about the same as the cheapest learned baselines and far less than k-NN and STAC, while providing improved informativeness and timeliness on most unseen settings, which makes it a practical option for real-world deployment.
Answer to Q4. Efficiency. Yes: labeling, training, calibrating, and running the probe introduce minimal cost, and on a GPU the probe adds under 1% to the policy call and fits within the step period.
Steering Demonstration
We demonstrate how the four properties of VLA-TUG: (T.) timeliness, (C.) calibration, (I.) informativeness, and (E.) efficiency, can jointly support failure prevention through VLA steering. Our goal is to illustrate how an effective failure predictor can provide the signals needed to guide interventions rather than develop a steering method itself. We consider a steering architecture in which an expert model (e.g. a VLM) can invoke failure-recovery tools in response to the probe’s predictions. For our experiments, we implement these tools using failure-mode-specific heuristics in simulation and demonstrate corresponding interventions on a real robot.
For example, in the simulated rollout shown in Figure 6a, the task is “place the largest bowl inside the left compartment of the bowl drainer.” The probe raises an alarm before the bowl is mistakenly placed in the compartment on the right. After steering the end-effector towards the left side, the policy is resumed and the rollout succeeds. Without intervention, the bowl would be placed on the right side and the rollout fails.

Figure 6. (a) Example steering for the simulated task “place the largest bowl inside the left compartment of the bowl drainer.” The gripper drops the bowl into the right compartment with no steering and successfully drops it in the left compartment when steered towards the target location.
(b) Example steering for the real-world task “put the mushroom in the plate.” The gripper misses the mushroom without steering and successfully grasps it when steered towards the mushroom.
Figure 6. Steering demonstrations showing failure scores and corrections for simulated and real-world tasks. When the probe’s predicted failure score rises above the threshold, the VLA policy is automatically paused. For the simulated bowl task, we find that nudging the end-effector toward the drop-off location can help lead to success. For the real-world mushroom task, we find that nudging the wrist camera towards the target grasp object can help. Future work could systematically explore steering strategies. Refer to Figure 17a for more steering strategies and Figure 17b for another real world steering example.
Our real-world experiments similarly demonstrate that the joint properties of VLA-TUG help in steering a VLA away from failure. Figure 6b illustrates this with the “put the mushroom on the plate” task. Without VLA-TUG, the gripper misses the mushroom and hits the table. With it turned on, however, the probe predicts the imminent grasp failure (which would likely happen), and corrective action is taken. In the example, the corrective action is to pause the VLA, let an expert model shift trajectory to a successful one, and resume the VLA.
Such a strategy is possible with a failure detector like VLA-TUG, which anticipates the failure before it happens, rather than after it is too late. Our approach is agnostic towards the steering architecture. We leave exploration of robust steering strategies for future work. Nevertheless, these examples demonstrate the applicability of VLA-TUG to be utilized in safe and principled robotic systems.
Answer to Q5. Steering. Yes: Our proof-of-concept demonstrations show how VLA-TUG can provide the timing and failure mode information needed for targeted interventions during VLA execution. Timeliness creates an opportunity to intervene, calibration controls when interventions are triggered, informativeness guides which intervention is applied, and efficiency allows the probe to operate alongside the VLA.
Conclusion
We introduced VLA-TUG, a lightweight probe on the latent layers of a VLA that is timely, calibrated, informative, and efficient, making it well suited for real-world deployment. It is timely: supervising on the pre-failure part of the trajectory drives early detection, and the probe’s performance compared to the strongest learned baseline increases as the shift from the training tasks increases. It is calibrated: calibrating the FPR and the TPR jointly provides guarantees that hold in far more runs than the baselines under shift. It is informative: beyond a failure score, it names the likely failure mode. It is efficient: the probe adds under 1% to the cost of a policy call. Our steering demonstrations, in simulation and on a real robot, show that VLA-TUG can inform targeted interventions, and we release 3,180 annotated episodes to support further work on latent failure detection.
Limitations. First, our calibration guarantees rely on exchangeability between the calibration and test episodes, which weakens as the distribution shift grows. Empirically, we observe that the degradation is gradual. Second, we do not analyze the choice of source backbone embedding systematically. Third, the steering component, in which an expert model intervenes on a predicted failure, is heuristic and not evaluated in the scope of this work.
Future Work. Looking forward, first, we will adapt the probe and its calibration guarantees online, which our preliminary experiments suggest is computationally feasible. Second, we will maintain and extend our embedding dataset with additional backbones and action-expert layers to determine which layers contain the most generalizable failure prediction signal. Third, we will explore agentic frameworks as the expert model in our setup, combining fast and local inference of a VLA with general-purpose intervention and steering from a cloud-based LLM, triggered by a principled detector such as VLA-TUG.
Method Details
This section gives the details behind § 4.
Probe and Training
Inputs. At every policy step we average the backbone tokens of the agent camera, of the wrist camera, and of the (unpadded) language prompt, and concatenate the three means into . No other feature of the backbone is used. All models in this paper read this vector, the baselines included.
Probe. A single-layer LSTM with 256 hidden units reads . Three heads read its state, each an MLP with one hidden layer of 64 ReLU units: the failure score head (one sigmoid output), the failure mode head (logits over the 10 modes), and the semantic head (a 64-dimensional vector).
Semantic (CLIP) head. The target space of the semantic head is a PCA of the CLIP text embeddings of the timeline phrases (clip-vit-base-patch32), reduced to 64 dimensions with a PCA fitted on the training split only. The loss is InfoNCE with temperature 0.07: the normalized output of a step must pick the target of the step’s phrase among the targets of all phrases of the training split. A step whose phrase is not among them is left out of this term. At inference the head is only a 64-dimensional MLP output; no CLIP encoder runs.
Loss and training. is a binary cross-entropy over the steps that are not excluded, weighted by the ratio of negative to positive steps of the training set. is a cross-entropy with class weights that balance the modes, on the positive steps of failed rollouts. Both auxiliary terms have weight 1. Early stopping follows on the validation episodes only, so the auxiliary heads never choose the checkpoint.
Labeling window. The window of the onset and post-onset labels, and the window of the mode read-out (subsection C.2), is 2.4 s: 6 policy steps for GR00T on LIBERO, 20 on the real robot, and 10 steps (2.5 s) for .
Supervision Strategies
Let be the backbone embedding at step and its target, where excludes the step from the loss. Let be the episode label (success, failure) and the width of the window around . For a failed rollout () the four strategies of Figure 2 are
Pre-onset.
Onset.
Post-onset.
Trajectory. for all .
In all four, for every step of a successful rollout. Excluding a step () is the same as giving it zero loss. Trajectory is the labeling of SAFE.
Calibration and Guarantees
Alarm rule and risks. For a constant threshold the alarm time of an episode is . The is the probability that a successful episode has an alarm, and the probability that a failed episode has its first alarm at or before .
Learn-then-Test. We follow LTT ((Angelopoulos et al., 2022)) with two risks, the FPR and the early miss rate (Section 2.4 there). A threshold is tested against the null that it violates either target, and its p-value is the larger of the two p-values (Proposition 6). Both losses are binary, so each p-value is an exact binomial tail (Section 3.2): the number of calibration failures without an alarm at or before is compared with , and the number of calibration successes with an alarm with . We combine the p-values by multi-start fixed-sequence testing (Section 2.3.1, Algorithm 1, Proposition 3) over a grid of 181 thresholds , , ordered from low to high, with starts and , i.e. a level of 0.0025 per walk. By Theorem of Angelopoulos et al. (2022), the set of accepted thresholds satisfies , where stands for each of our two risks with its target; that is, with probability at least every accepted threshold meets both targets, for episodes exchangeable with the calibration episodes. We select the accepted threshold with the highest calibration (ties: lowest calibration FPR, then lowest threshold), and abstain if none is accepted.
Max-conformal (MC). For STAC (whose accumulated scores are non-negative, so the maximum over steps is the final score), for ActProbe (as in its paper), and for our probe in the main table, we use max-conformal prediction. It takes the maximum score of each of the calibration successes and sets the constant threshold to the -th smallest of them, with , and to if . For a new success exchangeable with the calibration successes, its maximum has a uniformly distributed rank, so the false-alarm probability is at most , and conservative by less than . It says nothing about the TPR.
Functional conformal (FC). For SAFE and the methods that follow its calibration (FailureSpot and the embedding-distance baselines), we use the one-sided functional conformal band of SAFE (Gu et al., 2025). The band is a time-varying threshold built from the score trajectories of the calibration successes: the mean trajectory gives its shape, the local spread its width over time, and a conformal quantile of the successes’ deviations sets its level so that a success stays below it with probability about .
Which method gets which calibrator. SAFE-MLP, SAFE-LSTM, FailureSpot, and the three embedding-distance baselines use the SAFE band, as in their own work. STAC, ActProbe, and our probe use max-conformal.
Datasets and Labeling
Benchmarks and Splits
LIBERO-10 with occlusions. We use the 10-suite of LIBERO-Occ (Li et al., 2026a), an extension of LIBERO [15] that occludes the agent-view camera. Both policies fail rarely on the original benchmark, so we mix occluded and unoccluded episodes in equal parts to obtain enough failures (250 of each per policy).
LIBERO-X. LIBERO-X [27] introduces five levels of distribution shift. We use levels L1–L3 and select 218 tasks that cover diverse skills. Table 2 gives the task counts of the splits.
Real robot. We collected demonstrations for 8 tabletop manipulation tasks on a Kinova Gen3 arm (Table 4) and fine-tuned GR00T on them. The evaluation data are 399 rollouts of the fine-tuned policy (49 to 50 per task), 254 of them failures; Table 5 gives their failure modes.
| Task instruction | Third-person view | Demos |
| Close the drawer | !Third-person view | 50 |
| Flip the basket upright | !Third-person view | 50 |
| Push the pineapple off the stove | !Third-person view | 100 |
| Put the mushroom on the plate | !Third-person view | 50 |
| Put the tomato in the bin | !Third-person view | 100 |
| Stack the yellow block on the wooden block | !Third-person view | 60 |
| Take the eggplant off the plate | !Third-person view | 50 |
| Take the pineapple out of the drawer | !Third-person view | 50 |
Table 4. The eight real-world tasks and the number of demonstrations used to fine-tune the policy (510 in total).
| Mode | LIBERO-10-Occ % of failures | LIBERO-10-Occ Time (s) | GR00T LIBERO-10-Occ % of failures | GR00T LIBERO-10-Occ Time (s) | GR00T LIBERO-X % of failures | GR00T LIBERO-X Time (s) | GR00T real robot % of failures | GR00T real robot Time (s) |
| stuck_or_no_progress | 24.5 | 30.2 | 14.1 | 12.2 | ||||
| object_displacement | 18.2 | 17.8 | 10.8 | 9.4 | ||||
| placement_or_insertion | 13.6 | 16.0 | 9.9 | 11.4 | ||||
| grasp_failure | 17.3 | 13.0 | 9.0 | 56.7 | ||||
| open_close_failure | 7.3 | 9.5 | 4.7 | 7.5 | ||||
| wrong_target | 13.6 | 8.9 | 12.3 | 0.0 | – | |||
| wrong_object | 0.0 | – | 1.8 | 37.0 | 1.2 | |||
| press_failure | 2.7 | 1.8 | 1.5 | 0.0 | – | |||
| timeout | 2.7 | 1.2 | 0.2 | 1.6 | ||||
| other | 0.0 | – | 0.0 | – | 0.4 | 0.0 | – | |
| failed / total | 110/500 | 169/500 | 1258/2180 | 254/399 |
Table 5. Failure-mode distribution and mean std failure time (s) per mode, by policy and dataset.
Construction of the splits. The episodes of the held-out tasks form the unseen set. The episodes of the seen tasks are split 65/15/20% into train, validation, and test. The test part is halved, into the calibration set and the held-out seen set. The calibration sets of the LIBERO-X runs hold 158 to 209 episodes (66 to 90 successes and 92 to 120 failures). On the real robot each run holds one task out, so the calibration part has only 12 to 14 successes, a max-conformal threshold exists only for of about 0.07 and above, and the curves of all methods start at an FPR of about 8%. The area up to 40% FPR is still defined and treats every method the same, but it is noisy. The seed changes the shuffle of the seen episodes and the initialization of the learned models; for the new-task and new-signature splits the unseen episodes are identical in the three seeds, so their spread is not a spread over different unseen tasks.
Labeling Pipeline
We label each rollout with Gemini (gemini-3.5-flash) from its video, in three passes. A coarse pass over the whole video of a failed rollout returns the failure mode and a failure time; a second pass on a slowed clip around that time refines the failure time to a single frame; and a third pass on the same video session gives the timeline: a short keyword phrase (2–6 words, e.g. “reaching for mug”) for each 3-second window, which provides the targets of the semantic head. The failure time is the earliest point at which the mistake responsible for the failure becomes apparent. Following (Gu et al., 2025), if there is no single mistake but a human operator would take over (for example when the robot is stuck), we use that time, and if the robot behaves plausibly but is too slow, the failure time is the end of the rollout.
The failure mode is one of 10 classes: wrong_object (acts on an object or control other than the one the task specifies); wrong_target (right object, wrong destination); grasp_failure (never acquires the correct object securely); object_displacement (knocks over, pushes away, or drops an object); placement_or_insertion_failure (right object and target, wrong final pose); open_close_failure (right container, but not the required open or closed state); press_failure (right button or knob, but no effect); stuck_or_no_progress (looping, oscillating, hovering, or freezing); timeout_or_insufficient_progress (purposeful but too slow, no identifiable mistake); and other. The full definitions and the prompts are on the project page, vla-tug.github.io, and on the card of each dataset.
Dataset Statistics and Release
Table 5 gives, for each policy and dataset, the share of each failure mode among the failures and the mean and standard deviation of its failure time.
We release the embeddings, the labels, the prompts, and the splits of the simulated datasets, 3,180 episodes, at vla-tug.github.io.
Experimental Setup
Baselines
We re-implement all baselines on the same inputs and splits as our probe. Where a baseline has a configuration grid, we run the whole grid on every split and select, for every setting, the configuration with the highest mean area on the validation episodes, never looking at the unseen tasks; ties go to the earlier configuration of the grid. LIBERO-10 and the real robot pool their runs into one choice per method. On LIBERO-X, SAFE uses one configuration for the three kinds of splits, as in its own protocol, and every other method chooses per kind of held-out set. Our probe uses one architecture in every setting.
Embedding distance. Mahalanobis, Euclidean, and PCA-KMeans are SAFE’s embedding baselines (Gu et al., 2025). The score of a step is , where is the Mahalanobis distance to a Gaussian fitted to the steps of class , the mean Euclidean distance to the nearest stored steps of that class (Euclidean, also called k-NN), or the distance to the nearest of a few -means centroids in a PCA-reduced space (PCA-KMeans); it can be accumulated over the steps. We run a sweep of 26 configurations (Mahalanobis 2, Euclidean 6, PCA-KMeans 18).
Action consistency. STAC [1] compares the unexecuted tail of the previous plan with the first actions of the new plan, using the plans the policy samples at every call and the maximum mean discrepancy with an RBF kernel; nothing is learned. We run 8 variants (MMD over 3, 6, or 7 action dimensions, with the step score or accumulated; and STAC-Single, SAFE’s single-plan variant). ActProbe [10] is a small LSTM on two action signals and an embedding of the task instruction; we train it with the recipe of its repository and select its checkpoint with its own validation criterion. FailureSpot (Ma et al., 2026) has no public code, so we implemented it from the paper: a detector on the pooled features is trained on weak targets computed from action consistency and magnitude, and then fine-tuned on timestamp labels of the 15% most uncertain episodes.
Latent. SAFE-MLP and SAFE-LSTM (Gu et al., 2025) are written from SAFE’s code with its losses and its labels: every step of a failed rollout is positive, i.e. our trajectory label. We run SAFE’s learning-rate and regularization grid, 12 configurations per model. SAFE trains for a fixed number of epochs without validation; we stop early after 50 epochs without improvement of its validation loss (cap: 1,000 epochs) and keep the best epoch, so that all learned methods are treated alike.
Metrics
We define three events for a rollout: detection , early detection , and no alarm . We count as a true positive (TP) and as an early true positive (); the complements are the false negatives. The TPR and FPR follow their standard definitions, and
so that is reported in % and a detector with everywhere scores 100.
Full Results
Table 6 gives Table 3 with the standard deviation over the splits, and Figure 7 shows the curves of every setting and method. On the real robot the standard deviation is over the 40 splits (8 held-out tasks, 5 seeds).
| Method | Cal. | LIBERO-10 Seen | LIBERO-10 Unseen | LIBERO-10 GR00T Seen | LIBERO-10 GR00T Unseen | LIBERO-X task Seen | LIBERO-X task Unseen | LIBERO-X signature Seen | LIBERO-X signature Unseen | LIBERO-X skill Seen | LIBERO-X skill Unseen | Real robot Seen | Real robot Unseen | Fit (s) | State (MB) | GPU (ms) | CPU (ms) |
| SAFE-LSTM | FC | 75.5±13.4 | 34.9±3.5 | 74.1±13.0 | 51.2±8.4 | 64.6±3.5 | 40.4±2.3 | 64.7±7.4 | 41.2±7.1 | 62.5±2.4 | 30.1±8.5 | 44.5±13.5 | 25.1±18.8 | 43.5 | 26.2 | 0.066 | 0.86 |
| SAFE-MLP | FC | 74.1±2.0 | 39.9±15.1 | 62.9±18.2 | 39.2±9.4 | 59.3±2.0 | 42.3±2.9 | 53.7±2.4 | 46.2±2.3 | 56.8±6.0 | 23.5±12.4 | 40.1±15.9 | 20.2±16.6 | 114.5 | 6.3 | 0.034 | 0.032 |
| STAC | MC | 33.8±5.1 | 36.4±5.3 | 24.2±7.5 | 17.5±9.8 | 21.0±2.3 | 28.0±0.1 | 25.6±3.7 | 28.7±0.3 | 24.6±5.0 | 21.5±8.0 | – | – | 0.0 | 0.0 | 21.30 | – |
| ActProbe | MC | 39.3±11.3 | 26.7±2.6 | 32.4±16.8 | 30.1±9.3 | 12.6±4.2 | 22.7±4.6 | 11.9±3.1 | 13.9±1.5 | 20.3±9.4 | 13.6±7.1 | – | – | 7.1 | 0.09 | 0.15 | 0.11 |
| FailureSpot | FC | 61.9±3.0 | 36.2±21.0 | 36.9±19.8 | 44.7±18.6 | 26.3±2.1 | 20.4±1.2 | 28.0±2.5 | 26.2±5.6 | 30.4±3.4 | 19.6±7.9 | – | – | 9.2 | 6.4 | 0.033 | 0.029 |
| Mahalanobis | FC | 85.7±4.1 | 40.2±15.5 | 70.7±6.4 | 29.2±8.7 | 69.6±6.2 | 30.6±2.2 | 64.4±3.0 | 22.3±4.0 | 71.2±3.1 | 15.9±9.6 | 43.0±16.5 | 19.6±2.5 | 15.2 | 302.0 | 0.41 | 5.37 |
| Euclidean | FC | 73.1±4.4 | 24.2±10.6 | 56.0±6.8 | 23.1±11.4 | 70.5±4.2 | 28.5±3.0 | 65.4±2.1 | 19.7±0.8 | 68.5±4.8 | 19.0±6.5 | 45.1±14.2 | 8.3±12.8 | 0.4 | 3056 | 13.19 | 368.83 |
| PCA-KMeans | FC | 35.7±12.2 | 12.9±5.1 | 44.5±8.7 | 20.0±0.0 | 31.7±3.4 | 25.0±1.2 | 35.3±1.8 | 32.5±1.7 | 37.8±7.7 | 20.7±7.1 | 24.0±11.4 | 19.4±7.7 | 39.9 | 1.6 | 0.12 | 0.055 |
| VLA-TUG (Ours) | MC | 75.1±12.5 | 47.9±20.9 | 70.5±10.4 | 47.4±13.5 | 67.9±5.0 | 40.2±1.2 | 67.0±6.6 | 47.0±0.6 | 67.8±4.8 | 39.0±7.0 | 47.6±10.9 | 25.8±11.6 | 6.8 | 26.5 | 0.12 | 0.94 |
Table 6.

Figure 7. TPR against FPR for every setting and method, on seen (top) and unseen (bottom) tasks; mean over the splits of a setting.
Results in Detail
Calibration and Guarantees
This section gives the details behind subsection 6.2.
Setup. We run this analysis on LIBERO-X only. LIBERO-10 has only a few tens of failures per split, and the real robot has only 12 to 14 calibration successes per run, too few for LTT to certify any target together with an FPR target in at least half of the runs; its LTT rows would be empty. The 14 runs are the 3 NEW-TASK, 3 NEW-SIGNATURE, and 8 NEW-SKILL splits. We use the FPR targets 10, 15, 20, 25, 30, 40, and 50%. For each of them, we report the highest target that LTT certifies together with it in at least half of the runs, searched on a grid from 10% to 95% in steps of 5%. LTT uses and starting points for its fixed-sequence walks (subsection A.3). Table 7 gives for each FPR target the shares of runs in which the targets hold on the held-out seen and unseen episodes, the realized rates, and the PPV bound, for LTT and for the two calibrators of the baselines, max-conformal (MC) and functional conformal (FC).
| FPR target | TPR-early target | PPV bound | Method | Runs | Seen tasks: FPR holds | Seen tasks: TPR holds | Seen tasks: FPR | Seen tasks: TPR early | Seen tasks: PPV | Seen tasks: PPV holds (own ) | Unseen tasks: FPR holds | Unseen tasks: TPR holds | Unseen tasks: FPR | Unseen tasks: TPR early | Unseen tasks: PPV | Unseen tasks: PPV holds (own ) | ||
| 10% | – | – | LTT | 0% (0/14) | – | – | – | – | – | – | – | – | – | – | – | – | ||
| – | – | – | Max conformal (MC) | 100% (14/14) | 57% | – | 9.3±4.4 | 53.4±12.7 | 90.1 | – | – | 50% | – | 15.6±12.3 | 35.3±20.5 | 80.7 | – | |
| – | – | – | Functional conformal (FC) | 100% (14/14) | 71% | – | 9.8±5.8 | 33.9±17.2 | 90.7 | – | – | 36% | – | 14.9±9.4 | 26.3±16.2 | 84.3 | – | |
| 15% | 15% | 58.1 | LTT | 50% (7/14) | 100% | 100% | 5.3±3.1 | 39.2±8.0 | 92.5 | 100% | 100% | 86% | 71% | 9.8±11.5 | 27.1±14.7 | 85.8 | 100% | 100% |
| – | – | – | Max conformal (MC) | 100% (14/14) | 57% | – | 14.2±7.3 | 64.8±9.0 | 88.1 | – | – | 14% | – | 25.8±12.3 | 50.7±15.7 | 77.5 | – | |
| – | – | – | Functional conformal (FC) | 100% (14/14) | 64% | – | 13.2±6.1 | 40.0±13.9 | 88.9 | – | – | 29% | – | 21.9±11.9 | 32.8±18.5 | 81.4 | – | |
| 20% | 35% | 70.7 | LTT | 50% (7/14) | 100% | 100% | 10.4±3.5 | 58.1±5.4 | 89.7 | 100% | 100% | 71% | 57% | 15.5±10.4 | 37.0±16.4 | 81.1 | 100% | 100% |
| – | – | – | Max conformal (MC) | 100% (14/14) | 57% | – | 19.2±7.2 | 72.9±5.8 | 85.5 | – | – | 21% | – | 37.2±13.8 | 61.8±13.9 | 73.6 | – | |
| – | – | – | Functional conformal (FC) | 100% (14/14) | 79% | – | 18.0±6.1 | 54.4±11.2 | 86.6 | – | – | 21% | – | 30.7±11.1 | 47.0±16.7 | 77.9 | – | |
| 25% | 35% | 65.7 | LTT | 57% (8/14) | 100% | 100% | 12.4±3.7 | 64.5±5.2 | 88.8 | 100% | 100% | 75% | 75% | 22.7±11.6 | 49.2±18.9 | 79.9 | 100% | 100% |
| – | – | – | Max conformal (MC) | 100% (14/14) | 57% | – | 25.1±8.1 | 80.0±5.0 | 83.1 | – | – | 7% | – | 48.7±14.3 | 72.1±11.8 | 70.2 | – | |
| – | – | – | Functional conformal (FC) | 100% (14/14) | 79% | – | 22.7±7.0 | 63.8±12.0 | 84.7 | – | – | 7% | – | 42.5±12.1 | 57.5±16.1 | 74.3 | – | |
| 30% | 45% | 67.4 | LTT | 50% (7/14) | 100% | 100% | 15.2±3.5 | 68.2±4.6 | 87.4 | 100% | 100% | 86% | 57% | 24.5±6.6 | 50.6±12.6 | 77.9 | 100% | 100% |
| – | – | – | Max conformal (MC) | 100% (14/14) | 57% | – | 28.6±7.9 | 83.1±4.5 | 81.6 | – | – | 0% | – | 56.9±13.4 | 77.3±10.2 | 67.9 | – | |
| – | – | – | Functional conformal (FC) | 100% (14/14) | 71% | – | 27.8±7.4 | 71.2±11.1 | 82.5 | – | – | 7% | – | 54.1±15.3 | 67.3±15.0 | 70.7 | – | |
| 40% | 55% | 65.8 | LTT | 71% (10/14) | 100% | 100% | 24.1±7.3 | 78.4±5.7 | 83.5 | 100% | 100% | 60% | 80% | 43.3±14.6 | 66.6±13.8 | 70.5 | 80% | 100% |
| – | – | – | Max conformal (MC) | 100% (14/14) | 64% | – | 37.6±6.2 | 87.5±3.1 | 77.7 | – | – | 0% | – | 69.8±14.1 | 84.9±7.9 | 65.0 | – | |
| – | – | – | Functional conformal (FC) | 100% (14/14) | 64% | – | 38.7±7.3 | 82.0±8.1 | 77.8 | – | – | 0% | – | 74.0±15.0 | 80.2±8.1 | 65.0 | – | |
| 50% | 65% | 64.3 | LTT | 71% (10/14) | 100% | 100% | 31.1±5.4 | 84.5±3.3 | 80.2 | 100% | 100% | 40% | 90% | 59.2±18.3 | 77.6±10.7 | 67.7 | 70% | 60% |
| – | – | – | Max conformal (MC) | 100% (14/14) | 71% | – | 46.1±6.2 | 92.1±2.2 | 74.6 | – | – | 40% | – | 79.2±13.2 | 89.8±6.8 | 63.0 | – | |
| – | – | – | Functional conformal (FC) | 100% (14/14) | 50% | – | 49.0±7.7 | 90.6±3.3 | 73.8 | – | – | 0% | – | 83.1±16.2 | 87.2±5.9 | 62.4 | – |
Table 7. Learn-then-Test (LTT), max-conformal (MC), and functional conformal (FC) at the same FPR target, over the 14 LIBERO-X runs (). LTT certifies the TPR-early target of the second column together with the FPR target, and abstains otherwise; its percentages are over the certified runs. PPV bound: .
How to read Table 7. Runs is the share of runs for which LTT certifies the targets; the other columns of an LTT row are over these runs. Holds is the share of runs in which the target is met on the evaluated episodes. The PPV bound is with the prevalence of the calibration set, and PPV holds is the share of runs in which the realized PPV is at least the bound (computed with the calibration set’s prevalence, or with that of the evaluated set).
PPV bound. The bound follows from the two LTT targets, so for VLA-TUG it comes directly from the FPR and TPR targets we calibrate for. It depends on the prevalence , which we estimate on the calibration set. On seen tasks the estimate is stable. Under shift it can drift on individual NEW-SKILL folds, where the share of failures can differ a lot between the calibration set and the held-out tasks; this is what limits the bound on unseen tasks, and using the unseen set’s own removes most of the misses. A moving average of over the observations at deployment time could deal with more severe shift.
Unseen tasks. On the NEW-TASK and NEW-SIGNATURE splits, the three runs of each split share the same unseen tasks and differ only in the calibration data and the initialization. The held-out tasks define the shift each split is designed to test, so we cannot redraw them between runs. Only the eight NEW-SKILL folds hold out different tasks. The shares on unseen tasks therefore rest on fewer distinct unseen sets than runs, so they indicate trends and not precise rates.
Informativeness
This section gives the details behind subsection 6.3.
Classes. The failure modes are strongly imbalanced, so we report a balanced accuracy: the mean over classes of the share of their failures predicted correctly. We merge the two smallest classes repeatedly, once per dataset, until every class holds at least 5% of the failures. The rule uses label frequencies only. The mode head is not retrained: a merged class gets the sum of the probabilities of its modes. The accuracy is computed on the predictions pooled over the splits of a setting, since a split has few failures of a rare class. We exclude LIBERO-10: it has too few failures for a per-class accuracy, since even after merging its smallest classes hold only 6 to 16 failures, so one failure changes a class’s accuracy by several points.
Table 6. Table 3 with the standard deviation over the splits: area under TPR early vs. FPR up to 40% (in %). Dashes: the action chunks the method needs were not saved. Best in bold, second best in italics.
Prediction. A failed episode is predicted at an anchor step: the class with the highest mean probability over the 2.4 s window before the anchor and the anchor itself (the top three classes for top-3). The anchor is either the probe’s first alarm (max-conformal at ; episodes without an alarm are not scored) or the true failure time, which is an upper bound that a deployed system can reach. The baselines predict the most common training classes (always) or guess uniformly among the training classes.
Cost and Latency
This section gives the details behind subsection 6.4.
How the VLA runs. At each policy step the VLA receives the camera images and the prompt and produces a chunk of actions ( on LIBERO-X, 5 for on LIBERO-10, 6 on the real robot), which the robot executes at its control rate (20 Hz in simulation, 50 Hz on the real robot). A policy step therefore lasts divided by the control rate: 400 ms on LIBERO-X, 250 ms for , and 120 ms on the real robot. The probe reads the embedding of each call and must return its score well within one step.
Measurement. All times are measured on one RTX 4090 and 32 CPU threads. The fit time is the training time of the probe or the SAFE models, or the time to fit the statistics of the distance baselines. The state is what a fitted method must keep to score a new episode: the weights, the stored training steps of the k-NN, or the statistics of the distance baselines. The latency is the online latency: the time to turn one new policy step into a score, one embedding at a time and with the recurrent state of the previous step, after a warm-up and with the device synchronized, as the median over up to 2,000 steps. The CPU is timed with four threads. Online latency is the one that matters for deployment.
Baselines. The k-NN baseline has to keep every training step and compare each new step with all of them, so its cost grows with the training set; Mahalanobis keeps two 6144 × 6144 matrices. STAC needs the plans of the policy at every call, which no other method needs, so its latency is its scoring time plus the time to sample them.
Probe. On the GPU the LSTM takes most of the time; the mode head and the CLIP head add little, and on the CPU they add nothing measurable. The probe is not the fastest online method (SAFE-LSTM and the MLP detectors are faster), but it is far below the duration of a policy step. The average, and are lower on every kind of held-out split, seen and unseen; labeling the steps after the onset as positive is worst. On LIBERO-10 the differences are smaller ( to on average), within the spread across splits for , and clear on seen tasks with GROOT.
| Method | \multicolumn{8}{c}{Top-1 accuracy (%)} | \multicolumn{8}{c}{Top-3 accuracy (%)} | ||||||||||||||
| \multicolumn{2}{c}{task} | \multicolumn{2}{c}{signature} | \multicolumn{2}{c}{skill} | \multicolumn{2}{c}{Real robot} | \multicolumn{2}{c}{task} | \multicolumn{2}{c}{signature} | \multicolumn{2}{c}{skill} | \multicolumn{2}{c}{Real robot} | |||||||||
| Seen | Unseen | Seen | Unseen | Seen | Unseen | Seen | Unseen | Seen | Unseen | Seen | Unseen | Seen | Unseen | Seen | Unseen | |
| Most common | 14.3 | 14.3 | 14.3 | 14.3 | 14.3 | 14.3 | 20.0 | 20.0 | 42.9 | 42.9 | 42.9 | 42.9 | 43.9 | 39.0 | 60.4 | 32.6 |
| Random | 14.3 | 14.3 | 14.3 | 14.3 | 14.3 | 14.3 | 20.0 | 20.0 | 42.9 | 42.9 | 42.9 | 42.9 | 42.9 | 42.9 | 60.0 | 60.0 |
| VLA-Tug (Ours), own detection (first alarm) | ||||||||||||||||
| onset_pre | 34.2 | 25.5 | 35.6 | 22.6 | 33.6 | 20.3 | 54.3 | 21.7 | 65.1 | 59.9 | 69.8 | 56.3 | 69.4 | 56.3 | 88.2 | 70.4 |
| onset_traj | 39.3 | 25.9 | 37.1 | 29.9 | 39.7 | 27.0 | 57.2 | 15.8 | 71.8 | 58.8 | 71.4 | 61.1 | 69.8 | 59.7 | 87.7 | 58.6 |
| onset | 28.0 | 19.3 | 32.8 | 19.5 | 25.0 | 17.2 | 54.9 | 13.2 | 63.8 | 54.6 | 59.3 | 50.4 | 56.0 | 48.1 | 88.0 | 57.8 |
| onset_post | 31.1 | 19.8 | 32.1 | 22.1 | 32.5 | 25.2 | 56.6 | 18.8 | 65.3 | 55.7 | 69.6 | 58.0 | 65.2 | 57.2 | 91.4 | 54.4 |
| VLA-Tug (Ours), oracle onset (upper bound) | ||||||||||||||||
| onset_pre | 41.3 | 31.0 | 41.0 | 30.2 | 42.4 | 30.0 | 68.4 | 12.1 | 73.0 | 65.0 | 78.4 | 66.8 | 72.9 | 64.8 | 92.5 | 57.1 |
| onset_traj | 36.1 | 29.3 | 39.8 | 29.9 | 39.1 | 27.4 | 66.1 | 12.4 | 69.5 | 59.3 | 73.3 | 57.1 | 71.5 | 59.4 | 93.8 | 53.1 |
| onset | 37.5 | 30.5 | 39.7 | 29.2 | 37.1 | 27.1 | 70.2 | 11.4 | 74.2 | 67.5 | 75.0 | 66.9 | 70.1 | 62.7 | 92.6 | 60.0 |
| onset_post | 23.0 | 20.0 | 28.0 | 22.2 | 33.0 | 21.4 | 67.8 | 16.3 | 64.8 | 59.2 | 69.6 | 56.2 | 66.0 | 58.8 | 94.5 | 52.9 |
Table 8. Balanced top-1 and top-3 failure-mode accuracy (%), pooled over the splits of each kind. Shaded rows: our probe, by label variant; the oracle rows use the true failure time and are an upper bound. Best in bold, second best in italics, among the other rows.
Calibration. Per target pair, max-conformal takes 0.1 ms, functional conformal 0.7 ms, and LTT 1.3 ms on a CPU. The calibration sets hold 158 to 209 labeled episodes, of which 92 to 120 are failures.
Ablations
We ablate the supervision strategy, the auxiliary heads, and the failure time used for evaluation.
Supervision strategy. Table 9 compares our pre-onset label (every step before the failure time is positive) with the window before the onset only, every step of the failed rollout, and the window before the onset together with all steps after it. The pre-onset label is the best, or within the spread across splits, in nearly every setting. On LIBERO-X the alternatives lose 24% to 50% of the detection area on average, and are lower on every kind of held-out split, seen and unseen; labeling the steps after the onset as positive is worst. On LIBERO-10 the differences are smaller ( to on average), within the spread across splits for , and clear on seen tasks with GROOT.
Table 9. Ablations of our probe in the simulated settings: area under TPR early vs. FPR up to 40% (in %), mean ± standard deviation over the splits, and in brackets the change relative to our probe (shaded). Best in bold, second best in italics, within each block.
| LIBERO-10 Seen | LIBERO-10 Unseen | LIBERO-10 GR00T Seen | LIBERO-10 GR00T Unseen | LIBERO-X task Seen | LIBERO-X task Unseen | LIBERO-X signature Seen | LIBERO-X signature Unseen | LIBERO-X skill Seen | LIBERO-X skill Unseen | |
| Heads (LSTM, pre-onset label) | ||||||||||
| Modes + CLIP (ours) | ||||||||||
| No head | (+10%) | (+7%) | (−1%) | (+16%) | (−1%) | (+6%) | (+2%) | (−4%) | (+2%) | (+12%) |
| Modes only | (+7%) | (+1%) | (+0%) | (+12%) | (+2%) | (−1%) | (+2%) | (−1%) | (+1%) | (+12%) |
| CLIP only | (+4%) | (+22%) | (+3%) | (−6%) | (−3%) | (+3%) | (+1%) | (−2%) | (+0%) | (−1%) |
| Training strategy (modes + CLIP heads) | ||||||||||
| onset pre (ours) | ||||||||||
| onset traj | (+3%) | (+9%) | (−9%) | (−8%) | (−27%) | (−5%) | (−28%) | (−33%) | (−23%) | (−45%) |
| onset | (−2%) | (−5%) | (−15%) | (−17%) | (−28%) | (−1%) | (−29%) | (−22%) | (−32%) | (−35%) |
| onset post | (−2%) | (+5%) | (−47%) | (−12%) | (−65%) | (−22%) | (−58%) | (−47%) | (−55%) | (−56%) |
Auxiliary heads. We include the mode and CLIP heads because their labels already exist for every failure. We find no consistent benefit from either on detection (Table 9): removing the heads changes the area by less than the spread across splits in most settings. For mode prediction (Table 10), the CLIP head changes top-1 accuracy by less than one point on average, while training the mode head jointly with the failure score adds three to four points over a mode-only probe. The CLIP alignment might help some subpopulations and hurt others, which we leave to future work.
Table 10. What the CLIP head and the score term add to failure-mode prediction: top-1 and top-3 accuracy (in %), mean ± standard deviation over the splits. Our probe is shaded. Best in bold, second best in italics, per column and anchor.
| Top-1 accuracy (%) task Seen | Top-1 accuracy (%) task Unseen | Top-1 accuracy (%) signature Seen | Top-1 accuracy (%) signature Unseen | Top-1 accuracy (%) skill Seen | Top-1 accuracy (%) skill Unseen | Top-1 accuracy (%) Real robot Seen | Top-1 accuracy (%) Real robot Unseen | Top-3 accuracy (%) task Seen | Top-3 accuracy (%) task Unseen | Top-3 accuracy (%) signature Seen | Top-3 accuracy (%) signature Unseen | Top-3 accuracy (%) skill Seen | Top-3 accuracy (%) skill Unseen | Top-3 accuracy (%) Real robot Seen | Top-3 accuracy (%) Real robot Unseen | |
| At the first alarm of our probe | ||||||||||||||||
| Score + modes + CLIP (ours) | ||||||||||||||||
| Score + modes | ||||||||||||||||
| Modes + CLIP | ||||||||||||||||
| Modes | ||||||||||||||||
| At the true failure onset (upper bound) | ||||||||||||||||
| Score + modes + CLIP (ours) | ||||||||||||||||
| Score + modes | ||||||||||||||||
| Modes + CLIP | ||||||||||||||||
| Modes |
Sensitivity to the failure time. Moving the failure time later makes the metric easier, since more alarms then count as early, so we check how much our results depend on it.
We score the saved scores again with the failure time of every failed episode moved 1 and 2 s later, with the same thresholds and without retraining (Table 11). Every method gains, but late detectors gain more than VLA-TUG, which already alarms early: on unseen LIBERO-X VLA-TUG gains 2.1 to 2.4 points of area over 2 s and SAFE-LSTM 3.5 to 4.0. On LIBERO-X our rank among the methods does not change, and our margins hold, so these results do not depend on the exact failure time. On the real robot, where our lead is a single point, SAFE-LSTM overtakes VLA-TUG at 1 s for the same reason.
Table 11. Early-detection area (in %) on the unseen episodes when the failure time is moved later by 0, 1, and 2 s; nothing is retrained. Ours is shaded. Best in bold, second best in italics.
| Method | LIBERO-10 0 s | LIBERO-10 +1 s | LIBERO-10 +2 s | LIBERO-10 GR00T 0 s | LIBERO-10 GR00T +1 s | LIBERO-10 GR00T +2 s | LIBERO-X task 0 s | LIBERO-X task +1 s | LIBERO-X task +2 s | LIBERO-X signature 0 s | LIBERO-X signature +1 s | LIBERO-X signature +2 s | LIBERO-X skill 0 s | LIBERO-X skill +1 s | LIBERO-X skill +2 s | Real robot 0 s | Real robot +1 s | Real robot +2 s |
| SAFE-LSTM | 34.9 | 37.2 | 39.2 | 51.2 | 53.1 | 55.0 | 40.4 | 42.1 | 44.4 | 41.2 | 42.8 | 44.9 | 30.1 | 31.7 | 33.6 | 25.1 | 26.8 | 28.9 |
| SAFE-MLP | 39.9 | 42.1 | 43.9 | 39.2 | 40.6 | 43.9 | 42.3 | 43.3 | 44.7 | 46.2 | 47.2 | 47.9 | 23.5 | 24.3 | 25.7 | 20.2 | 21.4 | 22.8 |
| STAC | 36.4 | 43.0 | 53.2 | 17.5 | 22.0 | 27.6 | 28.0 | 29.5 | 32.2 | 28.7 | 30.6 | 34.0 | 21.5 | 22.6 | 25.1 | – | – | – |
| ActProbe | 26.6 | 31.0 | 37.2 | 30.1 | 33.3 | 38.6 | 22.7 | 24.7 | 27.5 | 13.9 | 16.1 | 18.8 | 13.6 | 14.3 | 15.7 | – | – | – |
| FailureSpot | 36.2 | 39.4 | 42.7 | 44.7 | 46.5 | 46.6 | 20.4 | 20.9 | 21.7 | 26.2 | 27.2 | 29.1 | 19.6 | 20.8 | 22.8 | – | – | – |
| Mahalanobis | 40.2 | 42.1 | 45.9 | 29.2 | 30.4 | 32.6 | 30.6 | 31.7 | 34.0 | 22.3 | 23.1 | 24.4 | 15.9 | 16.6 | 17.4 | 19.6 | 19.7 | 19.8 |
| Euclidean | 24.2 | 28.2 | 31.2 | 23.1 | 25.4 | 30.4 | 28.5 | 29.7 | 31.1 | 19.7 | 20.8 | 22.5 | 19.0 | 20.0 | 21.4 | 8.3 | 9.7 | 12.2 |
| PCA-KMeans | 12.9 | 12.9 | 13.8 | 20.0 | 20.0 | 20.0 | 25.0 | 25.3 | 25.9 | 32.5 | 34.0 | 36.4 | 20.7 | 21.5 | 23.1 | 19.4 | 19.9 | 20.3 |
| VLA-TuG (Ours) | 47.9 | 52.5 | 55.4 | 47.4 | 49.1 | 49.3 | 40.2 | 41.3 | 42.3 | 47.0 | 48.2 | 49.1 | 39.0 | 40.5 | 41.4 | 25.8 | 26.3 | 26.5 |
Qualitative Results
Latent Space
VLA-TUG encodes the VLA backbone embedding of each step with an LSTM into a latent state, which three MLP heads read: the failure score, the failure mode, and a CLIP embedding. The following figures show four of these spaces in two dimensions: the backbone embedding itself (the input), the LSTM state, and the hidden layers of the failure score head and of the failure mode head. Each is projected onto its first two principal components, fitted on the pre-failure steps of the training episodes only, and drawn for the training episodes and for the unseen tasks of the LIBERO-X NEW-TASK split. The points are the steps up to the failure of failed episodes and all steps of successful episodes; the ellipses are the 1 ellipses of a Gaussian fitted to each class, drawn over the points. The classes are the outcome (success in blue, failure in red) or the failure mode. The mode head is trained on failures only and has only a mode plot.

Figure 8. VLA backbone embedding, by outcome.

Figure 9. VLA backbone embedding, by failure mode.

Figure 10. Probe LSTM state, by outcome.

Figure 11. Probe LSTM state, by failure mode.

Figure 12. Failure score head layer, by outcome.

Figure 13. Failure score head layer, by failure mode.
Latent Trajectories
To see how an episode moves through these spaces, Figure 15 and Figure 16 follow one success and one failure step by step, for episodes of seen and of unseen tasks. Each panel shows one space in its first two principal components: (a) the VLA backbone embedding, (b) the probe’s LSTM state, and the hidden layers of (c) the failure score head and (d) the failure mode head. The points and ellipses are those of the episodes of the set shown (the training episodes, or the unseen tasks), grouped by outcome and failure mode, and the mode head is shown on failures only. The principal components are always those fitted on the training episodes. A trajectory starts at the ring; a failure is dashed and ends at the cross, at its failure step.

Figure 15. Seen tasks (training episodes). A success (green, “pick up the bowl on the plate and place it to the left of the bowl drainer”) and an object displacement failure (red, “place the striped book in the top region of the shelf”).

Figure 16. Unseen tasks. A success (green, “place the red bowl in the basket”) and a press failure (red, “turn on the flat stove”).
Steering Demonstrations
We use steering only to show how the signal of VLA-TUG could be used; the heuristics are simple and are not evaluated. When the score crosses the calibrated threshold, the policy is paused, an intervention chosen from the predicted failure mode is applied, and the policy resumes.
Simulation. We implemented three heuristics for LIBERO-X. For a wrong-target or placement/insertion failure, the end effector is displaced by a small distance towards the destination. For a grasp failure, it is displaced laterally, perpendicular to its recent motion. For every other predicted mode, the policy replans from the current state. Figure 17a shows the three on the task “place the white-yellow mug on top of the shelf”: after the alarm, the probe predicts the failure mode, the matching intervention is applied, and the policy resumes and succeeds, whereas without intervention the mug is placed on the table and the task fails.

Figure 17. Steering strategies, in simulation and on the real robot.
Real robot. In the task “put the mushroom on the plate”, the gripper misses the mushroom and hits the table without the probe; the probe predicts a grasp failure, and an expert would steer the wrist camera towards the grasp target and resume the policy. In the task “flip the basket upright”, we start with the basket standing on its tall end, outside the training demonstrations; an expert might move the gripper along the direction, so that the joint states are in the distribution of the flip motion, as the demonstrations are collected with the basket on its long side (Figure 17b).

Figure 14. Failure mode head layer, by failure mode (failures only).
References
- [1]Agia, C., Sinha, R., Yang, J., Cao, Z.-a., Antonova, R., Pavone, M., and Bohg, J. Unpacking failure modes of generative policies: Runtime monitoring of consistency and progress. arXiv preprint arXiv:2410.04640, 2024. Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., and Lei, L. Learn then Test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052, 2022.
- [2]Bhatt, N. P., Yang, Y., Siva, R., Milan, D., Topcu, U., and Wang, Z. Know where you’re uncertain when planning with multimodal foundation models: A formal framework. arXiv preprint arXiv:2411.01639, 2025.arxiv.org/abs/2411.01639
- [3]Buurmeijer, H., Alonso, C. A., Swann, A., and Pavone, M. Observing and controlling features in vision-language-action models. arXiv preprint arXiv:2603.05487, 2026.arxiv.org/abs/2603.05487
- [4]Chen, J. AEGIS: A backup reflex for physical AI. arXiv preprint arXiv:2606.06660, 2026.arxiv.org/abs/2606.06660
- [5]Chen, L., Lyu, Z., and Beksi, W. J. ReconVLA: An uncertainty-guided and failure-aware vision-language-action framework for robotic control. arXiv preprint arXiv:2604.16677, 2026.
- [6]Cui, H., Arnaud, S., Majumdar, A., Dugas, D., Aljalbout, E., Desingh, K., Jatavallabhula, K. M., and Meier, F. DREAMSTEER: Latent world models can steer VLA policies during deployment without any finetuning. arXiv preprint arXiv:2607.02865, 2026.arxiv.org/abs/2607.02865
- [7]Fei, S., Wang, S., Shi, J., Dai, Z., Cai, J., Qian, P., Ji, L., He, X., Zhang, S., Fei, Z., Fu, J., Gong, J., and Qiu, X. LIBERO-Plus: In-depth robustness analysis of vision-language-action models. arXiv preprint arXiv:2510.13626, 2025.arxiv.org/abs/2510.13626
- [8]Francis-Meretzki, S., Mutti, M., Romano, Y., and Tamar, A. Temporal difference calibration in sequential tasks: Application to vision-language-action models. arXiv preprint arXiv:2604.20472, 2026.
- [10]Huang, B., Li, X., Wang, X., Mi, L., Hao, Z., Wang, W., Wu, H., Li, K., Liu, Y., and Cao, T. ActProbe: Action-space probe for early failure detection of generative robot policies. arXiv preprint arXiv:2606.08508, 2026.arxiv.org/abs/2606.08508
- [11]Jeong, H. J., Swamy, G., and Bajcsy, A. Learning what to say to your VLA: Mostly harmless vision language action model steering. arXiv preprint arXiv:2606.12299, 2026.arxiv.org/abs/2606.12299
- [12]Li, Q. VLAs are confined yet capable of generalizing to novel instructions. arXiv preprint arXiv:2505.03500, 2025.arxiv.org/abs/2505.03500
- [14]Li, Z., Liu, J., Dong, Z., Teng, T., Rouxel, Q., Caldwell, D., and Chen, F. Towards deploying VLA without fine-tuning: Plug-and-play inference-time VLA policy steering via embodied evolutionary diffusion. IEEE Robotics and Automation Letters, 11(5):6234–6241, 2026b. doi: 10.1109/LRA.2026.3678455. URL https://ieeexplore.ieee.org/document/11457243/.DOI
- [15]Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., and Stone, P. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.arxiv.org/abs/2306.03310
- [16]Liu, S., Singh, I. S., Xu, Y., Duan, J., and Krishna, R. VLS: Steering pretrained robot policies via vision-language models. arXiv preprint arXiv:2602.03973, 2026a.arxiv.org/abs/2602.03973
- [20]Park, S., Li, W., Oh, C., Yeh, S., Kira, Z., Hagenow, M., and Li, S. Hide-and-seek in trajectories: Discovering failure signals for VLA runtime monitoring. arXiv preprint arXiv:2605.30834, 2026.arxiv.org/abs/2605.30834
- [21]Physical Intelligence, Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Galliker, M. Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Ren, A. Z., Shi, L. X., Smith, L., Springenberg, J. T., Stachowicz, K., Tanner, J., Vuong, Q., Walke, H., Walling, A., Wang, H., Yu, L., and Zhilinsky, U. π₀.₅: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.
- [23]Rajaprakash, H., Prajapati, A., Xue, R., Anwar, A., and Thomason, J. SAFECAST: Robust failure detection for VLA policies with contrast-set training and calibration. arXiv preprint arXiv:2608.04246, 2026.arxiv.org/abs/2608.04246
- [24]Swann, A., McGranahan, L., Buurmeijer, H., Kennedy, M., and Schwager, M. Sparse autoencoders reveal interpretable and steerable features in VLA models. arXiv preprint arXiv:2603.19183, 2026.arxiv.org/abs/2603.19183
- [25]Tan, D. M. S., Shailesh, S., Iyer, S., Teo, W. W. J., Ju, Y., Gu, Q., and Sartoretti, G. RL²-VLA: Adaptive RL latent compositional steering with test-time scaling for vision-language-action models. arXiv preprint arXiv:2607.26991, 2026.arxiv.org/abs/2607.26991
- [26]Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., and MacDiarmid, M. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2024.arxiv.org/abs/2308.10248
- [27]Wang, G., Zhang, C., Liu, Q., Zhang, J., Cai, J., Liu, J., and Liu, X. LIBERO-X: Robustness litmus for vision-language-action models. arXiv preprint arXiv:2602.06556, 2026.arxiv.org/abs/2602.06556
- [30]Yang, Y., Bhatt, N. P., Samineni, P., Siva, R., Wang, Z., and Topcu, U. RepV: Safety-separable latent spaces for scalable neurosymbolic plan verification. arXiv preprint arXiv:2510.26935, 2025a.arxiv.org/abs/2510.26935
- [31]Yang, Y., Hu, Y., Ye, M., Zhang, Z., Lu, Z., Xu, Y., Topcu, U., and Snyder, B. Uncertainty-guided enhancement on driving perception system via foundation models. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 8752–8758, 2025b. doi: 10.1109/ICRA55743.2025.11127651. URL https://ieeexplore.ieee.org/document/11127651/.DOI