betaFeedback

WorksComputer Science

Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing

Abstract

Vision-Language-Action models (VLAs) enable general-purpose embodied intelligence, but their performance is brittle under distribution shift, where unanticipated failures can damage equipment or endanger people. Deploying VLAs in real-world systems therefore requires a runtime monitor that is timely, calibrated, and informative while being efficient enough to run on the edge. Existing methods fail to fulfill these requirements. We present VLA-Tug, a lightweight probe over the latent layers of a VLA backbone that improves early failure detection and bounds the missed-detection and false-alarm rates, with minimal inference overhead. We show that supervising on the pre-failure segment of a trajectory increases early failure detection at matched false-positive rates by up to 30% over the strongest learned baseline. Under shift to unseen tasks, the bounds on missed-detection and false-alarm rates remain valid in up to 86% of runs, versus at most 21% for baselines. Unlike existing methods, VLA-Tug predicts the failure mode to enable corrective actions, raising top-3 accuracy on unseen tasks by up to 44% over a majority baseline. VLA-Tug enables real-time monitoring where the well-timed and informative probe supports lightweight steering that improves task success. We release a dataset of 3,180 annotated manipulation episodes to support future work on latent failure detection in VLAs at https://vla-tug.github.io.

Introduction

Vision-Language-Action models (VLAs) (Physical Intelligence et al., [21] (2026); NVIDIA et al. (2025)) are strong candidates for general-purpose embodied intelligence. By pairing a VLM backbone pre-trained on internet-scale data with an action expert fine-tuned on demonstration trajectories, they excel at complex dexterous tasks such as tightening screws or assembling cardboard boxes. Yet their performance is brittle and can degrade significantly under distribution shift, such as changes in lighting, novel textures, or rare object combinations (Wang et al., [27]; Fei et al. [7]). A VLA can therefore fail unexpectedly, especially as task and environment novelty increases, and such failures can be costly, damaging materials and robots, or outright dangerous to humans. This brittleness has slowed the adoption of VLAs in factories, homes, and public spaces.

Among multiple solutions, one solution that the robotics community is exploring is VLA steering. The idea of steering comes from research on LLMs, where activations are selectively modified to nudge a model towards a desired output (Turner et al., [26]). Steering VLAs is particularly interesting because it can unlock behaviors that cannot be reached through prompting alone (Buonamiger et al., (2026); Swann et al. [24]). Various steering methods have recently been proposed (Li et al., [14]; Liu et al. [16]; Tan et al. [25]; Swann et al. [24]; Cui et al. [6]; Jeong et al. [11]; Li [12]), but they rarely use a calibrated probe to decide when to steer. Some steer continuously (Li et al., [14]; Liu et al. [16]; Cui et al. [6]), some rely on human-specified offline triggers (Swann et al., [24]; Li [12]), and others are gated by a one-sided calibrated trigger (Tan et al., [25]; Jeong et al. [11]). Overall, they lack an efficient and principled trigger that would make them deployable in real-world systems.

Research on failure detection for VLAs is becoming increasingly mainstream (Gu et al., (2025); Agia et al. [1]; Ma et al. (2026); Huang et al. [10]). In a nutshell, these methods compute a failure score from signals available in the VLA, often its latent variables (Gu et al., 2025) or predicted actions (Agia et al., [1]; Huang et al. [10]; Ma et al. (2026)), that encode how likely the policy is to fail at the current task. A detection threshold is then calibrated to give probabilistic guarantees on the detector’s performance (Gu et al., 2025). However, these guarantees often cover

Preprint. only the false-positive rate (FPR), i.e. the rate of successes misclassified as failures, and not a target true-positive rate (TPR), i.e. the rate of correctly classified failures, which is arguably even more important in safety-critical settings where a single failure could harm a person or damage costly equipment. These methods are also seldom evaluated systematically against ground-truth failure times, which we argue is crucial to assess whether they can prevent failures before they happen and thus enable VLA steering. Finally, they cannot predict failure modes (grasping, pushing, opening), so steering cannot be targeted towards the root cause and would require human interventions.

We argue that steering would be best supported by a probe that is (T.) timely, detecting failures before they occur; (I.) informative, outputting semantic information beyond a numerical score; (C.) calibrated, achieving the desired error rates; and (E.) efficient to train, calibrate, and deploy.

We address this with VLA-TUG1, a well-behaved probe designed to support VLA steering in the real world. We train VLA-TUG on the latent layers of two VLAs (Physical Intelligence et al., [21]; NVIDIA et al., 2025) using 228 distinct tasks in two simulation environments and 8 real-world tasks, and evaluate it against ground-truth failure times. We show that supervision should focus on the pre-failure part of the trajectory to improve early failure detection, and that calibrating the detection threshold to guarantee both TPR and

FPR simultaneously yields guarantees that hold far more often, including on unseen tasks. Building on these findings, we propose a real-time architecture that incorporates VLA-TUG to autonomously support the detection and prevention of VLA failures. Our contributions are as follows:

  • VLA-TUG: A timely, informative, and efficient probe for early failure detection. On unseen tasks, our probe improves the area under the early-detection curve by up to 30% over the strongest learned baseline. VLA-TUG also predicts the failure mode, raising the top-3 accuracy on unseen tasks by up to 44% over a majority baseline, at under 1 ms per step.

  • Guarantees on false alarms and missed detections. We certify FPR and TPRearly_{\mathrm{early}} targets jointly; both held in every certified run, compared to 57–71% of runs for baselines. Under distribution shift to unseen tasks, the false-alarm target still held in 40–86% of certified runs, compared to 0–21% for baselines.

  • A deployable failure-monitoring system. Through real-world experiments, we demonstrate that integrating the timely failure predictions and associated modes with steering strategies can lead to enhanced success rates.

  • An annotated VLA failure dataset. We release a dataset of ≈289k policy-step VLA backbone embeddings from annotated 3,180 episodes of diverse tasks.

Related Work

Existing methods can detect failures, but their alarms are not necessarily timely, informative, or sufficiently calibrated. Our method addresses these limitations through failure-aware representations, onset-based detection, and joint statistical guarantees.

Existing works and their limitations. Existing VLA failure detectors leverage different signals to identify potential failures, including external assessments from world models or LLM judges (Liu et al., 2026b; Li et al., 2026b), action consistency across sampled or consecutive action chunks [1, 10], distributional deviations in policy inputs, outputs, or embeddings (Xu et al., 2025; Gu et al., 2025), and internal policy representations [23, 4, 20, 8] (Gu et al. 2025; Ma et al. 2026).

These approaches have three key limitations. (1) Detection performance is commonly evaluated at fixed time points using metrics such as AUROC or balanced accuracy [10] (Gu et al. 2025), without explicitly measuring whether alarms precede the actual failure onset. Moreover, supervision ranges from episode-level failure labels assigned to all timesteps (Gu et al., 2025), to heuristic or human-refined timestep labels (Ma et al., 2026), to failure-free training (Xu et al., 2025), without explicitly accounting for the timing of failures. (2) Most detectors produce scalar risk scores without identifying the underlying failure modes. While semantic supervision and representation interpretability have been explored for embodied policies [24, 3] (Xu et al. 2024), they do not directly establish informative failure alarms. (3) Existing calibration methods typically control a single criterion. Conformal methods provide false-positive guarantees for sequential failure detection [23, 20] (Gu et al. 2025), calibrate anomaly or failure scores (Xu et al., 2025; Ma et al., 2026), or provide guarantees for one-shot policy compliance and embodied decision making [30, 5, 2, 31]. However, these guarantees do not jointly account for false alarms and timely failure detection.

VLA-TUG addresses the limitations. We address these limitations by developing a failure detector that is timely, informative, and jointly calibrated. Following the internal-representation probing direction, we reuse the VLA’s latent representations and align them with failure-mode semantics, enabling the detector to identify both potential failures and their types. To evaluate timeliness, we measure detection relative to annotated failure onsets and investigate supervision strategies that explicitly account for when failures occur. To provide statistical guarantees, we adopt Learn-

then-Test (Angelopoulos et al., 2022) to jointly calibrate multiple detection criteria, including the false-positive rate (FPR) and early true-positive rate (TPRearly_{\mathrm{early}}). Existing failure detectors primarily raise alarms without steering, while steering methods rely on predefined schedules or calibrated triggers. In contrast, to the best of our knowledge, VLA-TUG is the first framework to combine joint FPR and early-TPR calibration, failure-onset-aware evaluation, and failure-mode prediction for calibrated failure-triggered steering. Table 1 compares VLA-TUG against the existing works.

MethodTriggerFPGTPGEEFM
Failure detectors (no steering)
SAFE (Gu et al., 2025)one-sided conformal✓×o†×
ActProbe [10]split conformal✓×ׇ×
STAC [1]one-sided conformal✓×ׇ×
FailureSpot (Ma et al., 2026)one-sided conformal✓×o†×
Steering methods
VLA-Pilot [14]always on––––
VLS [16]always on§––––
DreamSteer [6]fixed period––––
SAE-VLA [24]human-specified––––
TLI [12]human-specified––––
R2^{2}-VLA [25]one-sided conformal✓×××
LFP [11]one-sided conformal✓‡×××
VLA-TUG (Ours)jointly calibrated✓✓✓✓

Table 1. Comparison with related methods. FPG / TPG: formal calibrated guarantee on the false-positive / true-positive rate. EE: early evaluation, i.e. a per-episode criterion against annotated failure time; o denotes partial use. FM: predicts the failure mode. Trigger: what decides when steering happens. A dash means n/a.

Problem Statement

In this section, we start by giving several broad definitions used to describe how failure detectors for VLAs function. Next, we give a mathematical formulation of our problem and then introduce how VLA-TUG addresses it.

Definitions

To facilitate comparisons across methods, we give general definitions for the concepts of failure score, detection threshold, detection time, failure time, and failure mode.

  • Failure score. The score st∈Rs_t \in\mathbb{R}, defined at every timestep tt, encodes the probe’s belief that the VLA is failing. It need not be a probability in the strict sense.

  • Detection threshold. The threshold h⋆h^\star over sts_t separates failures from successes.

  • Detection time. The time tdet=min⁡{t:st≥h⋆}t_{\mathrm{det}} = \min\{t : s_t \ge h^\star\} is

the first time the score crosses the threshold.

  • Failure time. The time tfailt_{\mathrm{fail}}, which we also call the failure onset, is when the policy makes an obvious mistake or its progress stalls to the point where an intervention, human or otherwise, is needed.

  • Failure mode. The label fmodef_{\mathrm{mode}} is a high-level semantic category assigned to each failure, such as grasping failure, placement failure, or no progress.

These definitions are general, and their exact form varies slightly across methods. The interpretation of sts_t depends on the training strategy, but a higher score usually signals failure and a lower score success. The threshold h⋆h^{\star} may be fixed or vary with time, and it is often calibrated with probabilistic guarantees on a set of calibration rollouts Dcal\mathcal{D}_{\mathrm{cal}}. If st<h⋆s_t < h^{\star} for all tt, the detection time tdett_{\mathrm{det}} is undefined, and the rollout raises no alarm. The failure time definition follows (Gu et al., 2025).

Problem Formulation

Consider a VLA policy fVLA(⋅)f_{\mathrm{VLA}}(\cdot) and a rollout τi\tau_i of TT policy steps. At each policy step t∈{1,…,T}t \in\{1, \ldots, T\}, fVLA(⋅)f_{\mathrm{VLA}}(\cdot) takes as input RGB observations ot\mathbf{o}_t (agent and wrist cameras) and a natural language prompt p\mathbf{p}, and produces a chunk of mm control actions at;0:m−1\mathbf{a}_{t;0:m-1}. We denote the VLA’s internal embedding at step tt as et\mathbf{e}_t, so that τi={(p,ot,et,at;0:m−1)}t=1T\tau_i = \{(\mathbf{p}, \mathbf{o}_t, \mathbf{e}_t, \mathbf{a}_{t;0:m-1})\}_{t=1}^{T}. Each rollout has an episode-level label Yi∈{0,1}Y_i \in\{0,1\}, where Yi=1Y_i = 1 marks a failure and Yi=0Y_i = 0 a success. If Yi=1Y_i = 1, the rollout also has a failure time tfailt_{\mathrm{fail}} and a failure mode fmode∈Ff_{\mathrm{mode}} \in\mathcal{F}, where F\mathcal{F} is a fixed taxonomy of failure modes.

Problem. Given a training set and a calibration set Dcal={τi}\mathcal{D}_{\mathrm{cal}} = \{\tau_i\}, and access to the internal embeddings et\mathbf{e}_t of fVLA(⋅)f_{\mathrm{VLA}}(\cdot), predict an imminent task failure before it happens, i.e. tdet≤tfailt_{\mathrm{det}} \leq t_{\mathrm{fail}} where tdett_{\mathrm{det}} is the detection time, and predict the failure mode fmodef_{\mathrm{mode}} in a manner that holds up empirically on tasks unseen during calibration.

We propose VLA-TUG, a probe that is trained and calibrated to detect failures before they happen. It is:

(T.) Timely. We propose a training and evaluation strategy that accounts for the failure time and improves early failure detection.

(C.) Calibrated. We introduce a risk formulation that jointly bounds the false-positive rate and the early miss rate 1−TPRearly1 - \mathrm{TPR}_{\mathrm{early}}, and examine how well these targets hold on unseen tasks.

(I.) Informative. We design VLA-TUG to predict the failure mode, giving the semantic information needed to choose a corrective action.

(E.) Efficient. VLA-TUG adds minimal latency and parameters to the VLA, and is cheap to train and calibrate.

Methodology

In this section we introduce VLA-TUG. In subsection 4.1 we present the probe and how we train it, and in subsection 4.2 we show how we calibrate its alarm threshold and what guarantees this gives.

Failure Detection Through Latent Space Probing

Building on existing works (Gu et al., 2025), we instantiate the probe as a combination of LSTMs and MLPs, the goal being to capture a non-linear pattern in the backbone embeddings that generalizes beyond the seen episodes. We briefly introduce the model inputs, followed by a discussion of the model architecture and finally the training strategy.

We use VLA backbone embeddings as the inputs. We explore two distinct layers: backbone layer 18 (after normalization) in π0.5\pi_{0.5} and layer 16 in GROOT-N1.7, focusing on the latter in particular. At every policy step tt we average the backbone tokens of each modality (agent camera, wrist camera, instructions) and concatenate the three 2048-dimensional vectors into a single embedding et∈R6144\mathbf{e}_t \in\mathbb{R}^{6144}.

We leverage a combination of LSTMs and MLPs. A single-layer LSTM with 256 hidden units reads the sequence of embeddings and produces a latent state ut=LSTM⁡(e1:t)\mathbf{u}_t = \operatorname{LSTM}(\mathbf{e}_{1:t}) at every policy step. Three MLP heads read this shared latent state (see Figure 1). The failure score head predicts the failure score, st=σ(gs(ut))s_t = \sigma(g_s(\mathbf{u}_t)). To make the probe informative, the mode head predicts a distribution over the 10 failure modes fmodef_{\mathrm{mode}}, mt=softmax⁡(gm(ut))\mathbf{m}_t = \operatorname{softmax}(g_m(\mathbf{u}_t)). Following (Xu et al., 2024), an auxiliary head, ct=gc(ut)\mathbf{c}_t = g_c(\mathbf{u}_t), aligns the latent state with a CLIP embedding of the description of the current action. Each head gg is an MLP with one hidden layer of 64 ReLU units. The probe has 6.6M parameters.

Overview of VLA-TUG and runtime failure detection

Figure 1. We introduce VLA-TUG, a probe on the VLA backbone embeddings that predicts failures and their mode. (a) A recurrent encoder maps the embedding into a latent failure space; a score head and a calibrated trigger raise the alarm, and a mode head then predicts the failure mode. (b) VLA-TUG alarms before the failure, allowing early stopping or intervention by an expert model.

We train the probe to predict features that precede a failure. We hypothesize that training on post-failure embeddings teaches the probe what already failed trajectories look like, not what trajectories about to fail look like. If so, to predict failures the probe must be trained on the embeddings et\mathbf{e}_t before the failure time tfailt_{\mathrm{fail}} (pre-onset supervision), rather than after it (Ma et al., 2026) or at all times of failed rollouts (Gu et al., 2025). To test this, we compare four ways of labeling the embeddings of failed rollouts (Figure 2; the mathematical formulation is in App. § 8.§ 2). Successful rollouts are labeled 0 in all four. The window ww in onset and post-onset is the time span before tfailt_{\mathrm{fail}} that is labeled as failure. Pre-onset and onset truncate the rollout at the failure time, while post-onset and trajectory keep all steps. We exclude the steps after tfailt_{\mathrm{fail}} in the first two strategies instead of labeling them 0: these embeddings already show a failed rollout, so labeling them as successes would give the probe contradicting training signals.

Four supervision strategies showing pre-onset, onset, post-onset, and trajectory labels

Figure 2. How the embeddings ete_t of a failed rollout are labelled for the four supervision strategies.

We optimize three objectives. With the labels above, the probe is trained with

L=Lscore+λmLmode+λcLCLIP,(1)\mathcal{L}=\mathcal{L}_{\mathrm{score}}+\lambda_m\mathcal{L}_{\mathrm{mode}}+\lambda_c\mathcal{L}_{\mathrm{CLIP}}, \tag*{(1)}

with λm=λc=1\lambda_m=\lambda_c=1. Lscore\mathcal{L}_{\mathrm{score}} is a binary cross-entropy over all steps that are not excluded from the loss, weighted by the ratio of negative to positive steps in the training set. Lmode\mathcal{L}_{\mathrm{mode}} is a cross-entropy against the true failure mode, applied to the positive steps of failed rollouts and weighted to balance the modes. LCLIP\mathcal{L}_{\mathrm{CLIP}} is an auxiliary contrastive loss that aligns ctc_t with the CLIP embedding of the step’s description. We train with Adam (learning rate 3×10−43 \times10^{-4}, batch size 32 rollouts, at most 100 epochs) on standardized inputs, with early stopping on the validation Lscore\mathcal{L}_{\mathrm{score}} (patience 15).

Uncertainty Quantification and Guarantees

Where previous works only provided guarantees of false-positive rates (i.e. the rate of successful rollouts that were flagged as failures) [23] (Gu et al. 2025), they did not provide guarantees on the false-negative rate (i.e. the rate of unsuccessful trajectories that were not flagged). We will first formulate the risks and show how to calibrate the detection threshold h⋆h^\star. In an effort to make these bounds interpretable to the practitioner, we compute the probability of failure given that a rollout was flagged.

VLA-TUG guarantees both FPR and TPRearly_{\mathrm{early}}. In general, the FPR is an operational metric that we wish to minimize to prevent unnecessary steering, the TPR is a safety metric that we wish to maximize to prevent undetected failures. We can control these risks using the Learn-then-Test framework (Angelopoulos et al., 2022). Let h⋆h^\star be a constant detection threshold, and sts_t the failure score produced by our probe at each timestep tt for each rollout τ\tau. Furthermore, let the detection time tdett_{\mathrm{det}} be the first time the failure score sts_t exceeds the threshold h⋆h^\star. Finally, the ground truth failure time tfailt_{\mathrm{fail}} remains the first time tt when failure becomes apparent or a human operator would choose to take over. We propose to formulate the above mentioned risks as follows:

FPR⁡(h)=Eτ∼Dcalsucc[1{∃ tdet:st(τ)≥h}],(2)\operatorname{FPR}(h)=\mathbb{E}_{\tau\sim\mathcal{D}_{\mathrm{cal}}^{\mathrm{succ}}}\left[\mathbf{1}\{\exists\ t_{\mathrm{det}}:s_t(\tau)\ge h\}\right], \tag*{(2)}
TPR⁡early(h)=Eτ∼Dcalfail[1{∃ tdet≤tfail:st(τ)≥h}],(3)\operatorname{TPR}_{\mathrm{early}}(h)=\mathbb{E}_{\tau\sim\mathcal{D}_{\mathrm{cal}}^{\mathrm{fail}}}\left[\mathbf{1}\{\exists\ t_{\mathrm{det}}\le t_{\mathrm{fail}}:s_t(\tau)\ge h\}\right], \tag*{(3)}

where Dcalsucc\mathcal{D}_{\mathrm{cal}}^{\mathrm{succ}} and Dcalfail\mathcal{D}_{\mathrm{cal}}^{\mathrm{fail}} are the sets of successful and failed rollouts τ\tau, respectively, in the calibration set Dcal\mathcal{D}_{\mathrm{cal}}. We are interested in finding the set Λ^\widehat{\Lambda} of values h⋆h^\star that satisfy:

P(FPR⁡(h)≤α, TPR⁡early(h)≥β ∀h∈Λ^)≥1−δ,(4)P\left(\operatorname{FPR}(h)\le\alpha,\ \operatorname{TPR}_{\mathrm{early}}(h)\ge\beta\ \forall h\in\widehat{\Lambda}\right)\ge1-\delta, \tag*{(4)}

where α∈[0,1]\alpha\in[0,1] and β∈[0,1]\beta\in[0,1] are user-selected rates. This guarantee can be understood as saying that any h∈Λ^h\in\widehat{\Lambda} guarantees that the false-positive rate (FPR) is at most α\alpha and the early true-positive rate (TPRearly_{\mathrm{early}}) is at least β\beta, with high-probability of no less than 1−δ1-\delta. We will see, in the experimental section, that this translates to guarantees that empirically behave better under distribution shift. Following (Angelopoulos et al., 2022) we select a set of candidate values Λ^\widehat{\Lambda}; from this set Λ\Lambda of values hh we select the value h⋆h^\star that maximizes TPR⁡early\operatorname{TPR}_{\mathrm{early}}.

Probability of failure given that an alarm is raised. Since our goal is a probe that can be deployed in real-world systems, we complement the formal TPRearly_{\mathrm{early}} and FPR guarantees with an actionable interpretation: a bound on the positive predictive value (PPV). The PPV is the probability that failure occurs given score sts_t exceeds the threshold h⋆h^\star. Existing calibration methods (Gu et al., 2025) cannot give such a bound, since they lack guarantees on the TPR.

Let FAILURE denote rollout-level failure, and let EXCEED denote the event that the score sts_t exceeds the threshold h⋆h^\star at some time tt. The PPV is P(FAILURE∣EXCEED)P(\mathrm{FAILURE}\mid\mathrm{EXCEED}). Define the failure prevalence π=P(FAILURE)\pi=P(\mathrm{FAILURE}), the true positive rate TPR=P(EXCEED∣FAILURE)\mathrm{TPR}=P(\mathrm{EXCEED}\mid\mathrm{FAILURE}), and the false positive rate FPR=P(EXCEED∣NOT FAILURE)\mathrm{FPR}=P(\mathrm{EXCEED}\mid\mathrm{NOT\ FAILURE}). By Bayes’ rule,

P(FAILURE∣EXCEED)=π⋅TPRπ⋅TPR+(1−π)⋅FPR.(5)P(\mathrm{FAILURE}\mid\mathrm{EXCEED})=\frac{\pi\cdot\mathrm{TPR}}{\pi\cdot\mathrm{TPR}+(1-\pi)\cdot\mathrm{FPR}}. \tag*{(5)}

Detections that occur before the failure onset are a subset of all detections, so TPR≥TPRearly\mathrm{TPR}\ge\mathrm{TPR}_{\mathrm{early}}. Our guarantees give TPRearly≥β\mathrm{TPR}_{\mathrm{early}}\ge\beta and FPR≤α\mathrm{FPR}\le\alpha. Since the PPV is non-decreasing in TPR and non-increasing in FPR, substituting the worst-case values yields the conservative bound:

P(FAILURE∣EXCEED)≥π⋅βπ⋅β+(1−π)⋅α.(6)P(\mathrm{FAILURE}\mid\mathrm{EXCEED})\ge\frac{\pi\cdot\beta}{\pi\cdot\beta+(1-\pi)\cdot\alpha}. \tag*{(6)}

The bound increases with π\pi, so a lower bound on the prevalence suffices. Given an estimate of π\pi, this gives practitioners an intuitive reading of the guarantees.

Dataset

To evaluate VLA-TUG, we create a labeled dataset of rollouts of two VLA policies, π0.5\pi_{0.5} (Physical Intelligence et al. [21]) and GR00T-N1.7 (NVIDIA et al. (2025)), with failure labels and semantic descriptions of the robot’s actions. We first describe the benchmarks and the splits, and then the labeling pipeline.

Benchmarks and splits. We use LIBERO (Liu et al. [15]), LIBERO-Occ (Li et al. (2026a)), and LIBERO-X (Wang et al. [27]), as well as real-robot tasks. Both policies succeed on most LIBERO episodes, so we combine it with LIBERO-Occ to obtain enough failures. We propose splits of increasing distribution shift to evaluate generalization in a principled way; Table 2 gives how the tasks are held out and their counts. In LIBERO-X, each task has a skill, the manipulation behavior (such as pushing a button or stacking objects), and a signature, the combination of its skill, target class, relation, landmark, and horizon. The splits are:

DatasetSplitSeen tasksUnseen tasks
LIBEROleave-three-out73
LIBERO-XNEW-TASK17543
LIBERO-XNEW-SIGNATURE16256
LIBERO-XNEW-SKILL⋆^\star159–2099–59
Real-worldleave-one-out71

Table 2. Task count by dataset and split. ⋆^\star There are 8 skills, each with a different seen/unseen split.

  • NEW-TASK holds out individual tasks; their skill and signature can still occur in the training tasks, the shift is mild.

  • NEW-SIGNATURE holds out all tasks with a given signature: the components are familiar, but their combination is new.

  • NEW-SKILL holds out a complete skill, so the tested behavior is absent from training: this is the largest shift of the three.

Efficient labeling pipeline. We label each rollout from its video with Gemini (gemini-3.5-flash), as in Figure 3. Every rollout receives a timeline label, and failed rollouts also receive a failure mode label and a failure time label. The timeline is a short keyword phrase per 3-second window

A successful and a failed rollout with their timelines, failure time, and failure mode labels

Figure 3. A successful (top) and a failed (bottom) rollout with their timeline, failure time, and failure mode labels.

that we use for CLIP alignment. The failure mode fmodef_{\mathrm{mode}} comes from a 10-way taxonomy F\mathcal{F}. The failure time tfailt_{\mathrm{fail}} is the earliest point at which the mistake responsible for the failure becomes apparent or, following Gu et al. (2025), the time a human operator would take over if none is apparent.

We open-source this dataset including over 288,000 embeddings from 3,180 labelled rollouts at vla-tug.github.io.

Experimental Results

In this section, we answer five questions based on our problem formulation (§ 3): (Q1. Timeliness) Does training on features that precede the failure onset improve early failure detection on unseen tasks (subsection 6.1)? (Q2. Calibration) Can VLA-TUG jointly bound the FPR and TPRearlyTPR_{\mathrm{early}}, with bounds that hold on seen tasks and degrade gracefully on unseen tasks (subsection 6.2)? (Q3. Informativeness) Can VLA-TUG predict the failure mode based on VLA backbone embeddings (subsection 6.3)? (Q4. Efficiency) Are the cost and time of labeling, training, and inference appropriate for deployment in real-world systems (subsection 6.4)? (Q5. Steering) Can VLA-TUG, as a timely, informative, calibrated, and efficient probe, support VLA steering (subsection 6.5)? We ablate our design choices in App. D.4.

Failure Detection and Timeliness Evaluation

We evaluate VLA-TUG against a series of baselines using metrics that depend on the ground-truth failure time. Firstly, we describe how we split the data. Secondly, we review the baselines. Thirdly, we formally define our evaluation metrics. Lastly, we discuss the results.

Data split. For every run, we split the episodes as in Figure 4. The training set fits the probe, the validation set selects the checkpoint, and the threshold is chosen on the calibration set. The calibration and held-out seen sets are exchangeable, and our guarantees apply to the held-out seen set. The unseen set, which the probe and the threshold never see, measures robustness to distribution shift. On LIBERO-X, the held-out tasks of the NEW-TASK and NEW-SIGNATURE splits are the same for every seed, because they are defined by the semantic axis described in § 5; the seed changes Table 3. Early failure detection on seen and unseen tasks: area under the TPR early vs. FPR curve, averaged over splits. Seen: held-out episodes of training tasks; unseen: held-out tasks. Cal.: calibrator (MC max-conformal, FC functional conformal). Cost on LIBERO-X (lower is better): fit time, memory to score a new episode, and per-step latency on a GPU and a CPU. Dashes: not available. Best in bold, second best in italics.

Task- and episode-level data splits

Figure 4. Task- and episode-level data splits. Episodes from seen tasks are divided into training (65%), validation (15%), calibration (10%), and held-out seen (10%) sets. Unseen tasks form a separate evaluation set. Task-level proportions are schematic.

MethodCal.LIBERO-10 π0.5\pi_{0.5} SeenLIBERO-10 π0.5\pi_{0.5} UnseenLIBERO-10 GR00T SeenLIBERO-10 GR00T UnseenLIBERO-X task SeenLIBERO-X task UnseenLIBERO-X signature SeenLIBERO-X signature UnseenLIBERO-X skill SeenLIBERO-X skill UnseenReal robot SeenReal robot UnseenFit (s)State (MB)GPU (ms)CPU (ms)
SAFE-LSTMFC75.534.974.151.264.640.464.741.262.530.144.525.143.526.20.0660.86
SAFE-MLPFC74.139.962.939.259.342.353.746.256.823.540.120.2114.56.30.0340.032
STACMC33.836.424.217.521.028.025.628.724.621.5––0.00.021.30–
ActProbeMC39.326.732.430.112.622.711.913.920.313.6––7.10.090.150.11
FailureSpotFC61.936.236.944.726.320.428.026.230.419.6––9.26.40.0330.029
MahalanobisFC85.740.270.729.269.630.664.422.371.215.943.019.615.2302.00.415.37
EuclideanFC73.124.256.023.170.528.565.419.768.519.045.18.30.4305613.19368.83
PCA-KMeansFC35.712.944.520.031.725.035.332.537.820.724.019.439.91.60.120.055
VLA-TUG (Ours)MC75.147.970.547.467.940.267.047.067.839.047.625.86.826.50.120.94

only the shuffle of the seen episodes and the initialization. The NEW-SKILL folds and the LIBERO-10 seeds hold out different tasks.

Baselines. We consider the following baselines:

Embedding distance (Mahalanobis, Euclidean, PCA-KMeans). A step is scored by how much closer its backbone embedding is to failed than to successful training steps.

Action consistency (STAC, ActProbe, FailureSpot). Steps are scored based on the discrepancy between predicted and executed steps.

Latent (SAFE). A step is scored by a learned projection of its backbone embedding.

Timeliness Metrics. We evaluate VLA-TUG against three key metrics: TPR, TPRearly\mathrm{TPR}_{\mathrm{early}}, and FPR. Existing works often evaluate their methods using AUROC [10] (Gu et al. 2025; Ma et al. 2026) without accounting for the true failure times, making it hard to distinguish timely detections. The TPR and FPR follow their standard definitions. The TPRearly\mathrm{TPR}_{\mathrm{early}} is the rate of failures detected before the failure time, i.e. tdet≤tfailt_{\mathrm{det}} \le t_{\mathrm{fail}}. We additionally define AUROCearly\mathrm{AUROC}_{\mathrm{early}} as the area under the TPRearly\mathrm{TPR}_{\mathrm{early}}-FPR curve. We compute it on the support FPR ∈[0,0.4]\in[0, 0.4], normalized by its width 0.4 (reported in %), since a detector that raises false alarms on more than 40% of successful rollouts is impractical to deploy. We formally define these metrics in App. § 10.§ 1.

Results and Discussion. We ask whether training on features that precede the failure onset improves early failure detection on unseen tasks. Our hypothesis is that it does: post-failure features only describe failures the probe has already seen, so training on them keeps it from learning a general notion of what leads to a failure, while features before the onset carry the cues that come before one. We compare VLA-TUG against the baselines in Table 3 and Figure 5. On unseen tasks, our domain of interest, VLA-TUG is the best method in four of six settings: LIBERO-10 with π0.5\pi_{0.5}, NEW-SIGNATURE, NEW-SKILL, and the real robot. Against SAFE-LSTM, the strongest learned baseline, its margin grows from −1%-1\% on NEW-TASK to +14%+14\% on NEW-SIGNATURE and +30%+30\% on NEW-SKILL. On the real robot, the lead is small, and VLA-TUG does not lead on LIBERO-10 with GR00T. On seen tasks, the results are mixed. We ablate the label in App. § 11.§ 4: replacing the pre-onset label with the other variants lowers early detection on LIBERO-X.

Early detection on seen and unseen tasks across benchmarks, policies, and held-out splits

Figure 5. Early detection on seen (top) and unseen (bottom) tasks: the share of failures detected at or before the failure time (TPRearly\mathrm{TPR}_{\mathrm{early}}) against the false-positive rate, for VLA-TUG and the learned and action-based baselines. Each curve is the mean over the splits of a setting, and the area under it up to 40% FPR is the metric of Table 3. Panel titles give the benchmark, the policy, and the held-out split.

Answer to Q1. Timeliness. Yes: for most unseen settings. Training on features that precede the failure onset lets VLA-TUG raise the alarm at or before the failure onset more often than the baselines, and on LIBERO-X the gap widens as the test tasks move away from the training tasks.

Calibration Evaluation

We validate the theoretical results from § 4 on seen tasks and examine how they hold under distribution shift on unseen tasks. We calibrate our probe with Learn-then-Test (LTT, δ=0.1\delta= 0.1) at seven false-positive-rate (FPR) targets (10 to 50%), each certified together with a TPR early target, on the 14 LIBERO-X runs (3 NEW-TASK seeds, 3 NEW-SIGNATURE seeds, and 8 NEW-SKILL folds). We compare against max conformal (MC) [1] and functional conformal (FC) (Gu et al., 2025) calibration (the complete table is in App. § 11.§ 1). We draw three conclusions:

  • LTT bounds the TPRearly\mathrm{TPR}_{\mathrm{early}} together with the FPR, which hold in all certified runs on seen tasks and in 40–86% (FPR) and 57–90% (TPR early) of runs on unseen tasks; baselines give no guarantee on the TPR.

  • LTT states upfront whether the system can support the required guarantee whereas existing methods can give a false sense of safety. LTT abstains on 4 to 7 of the 14 runs, depending on the target (FPR targets of 15–50%), which tells us before deployment that the model cannot support the requested rates. MC always returns a threshold, but it meets the FPR target in only 57–71% of runs on seen tasks and in at most 21% on unseen tasks.

  • Our LTT targets give a lower bound on the positive predictive value (PPV), the probability that an alarm is a true failure. It is 58–71% across the FPR targets of the full table and holds in every certified run on seen tasks and in 70–100% of runs on unseen tasks. Methods without a TPR guarantee cannot give it, and it tells the practitioner what an alarm means.

On unseen tasks, the guarantees hold only empirically, since new samples are not exchangeable with the calibration set, which applies to all three methods. The shares above are trends over 14 runs, not precise rates.

Answer to Q2. Calibration. Yes: the rates we calibrate for are the rates we get on seen tasks. Under distribution shift, they hold in far more runs than with MC and FC, and when a target cannot be supported, LTT abstains instead of silently missing it. Together with the PPV bound, this makes VLA-TUG easier to integrate into real-world systems.

Evaluation of Informativeness

Here we focus on failure mode prediction, a crucial element to enable failure prevention through steering. Failure modes are strongly imbalanced, so we report balanced accuracy, the mean over classes of the share of their failures predicted correctly. Since existing baselines do not predict failure modes, we compare against a majority baseline and focus on LIBERO-X.

On LIBERO-X, the probe improves top-1 accuracy over the majority baseline by up to 149% on seen and 79% on unseen tasks, and top-3 accuracy by a smaller but consistent margin on both. On the real robot, it improves top-1 by 171% and 8% on seen and unseen tasks, respectively, and stays well above the baseline in top-3 on both.

The practical consequence is that the probe narrows a failure down to three candidate classes, which are more likely than the majority baseline to include the true one. Different failure modes carry different risks, so a practitioner can use these candidates to select the level of response, for example responding to the riskiest of the three predicted classes (a stop, a retry, or a handover to a human) instead of applying the same intervention to every failure.

Answer to Q3. Informativeness. Yes: backbone embeddings carry enough information to predict the failure class above a majority baseline, which lets a practitioner match the response to the predicted risk.

Practical Considerations

Any real-world system must balance performance (T.) Timeliness, (C.) Calibration, (I.) Informativeness) with the cost of building and running it ((E.) Efficiency). We first evaluate the cost of labeling the dataset, then of training the probe, calibrating it, and running it at inference time.

Dataset. Labeling costs about one cent per failed episode (under $50 for all annotations, including queries not used in this paper). Labels are therefore cheap, and practitioners will often already have unlabeled rollouts from system deployment.

Training. VLA-TuG trains in 6.8 s on one RTX 4090, against 43.5 s for SAFE-LSTM, and holds 6.6M parameters (26.5 MB), about 0.2% of the VLA. It is therefore cheap enough to retrain whenever the policy or the task set changes, or to train online as new rollouts are produced.

Calibration. Calibrating takes under 2 ms per target pair on a CPU, so VLA-TuG can be re-calibrated if system requirements change, or online along with the probe.

Inference. The VLA outputs a chunk of mm actions per call (8 at 20 Hz on LIBERO-X), so a policy step lasts 400 ms, and the probe reads the embedding of each call. It adds 0.12 ms per step on the GPU (0.2% of the 54 ms GR00T policy call) and 0.9 ms on four CPU threads, so its score is available before the chunk is executed and its latency is not observable: the step period is about 7×7\times the combined VLA and probe time in simulation and about 2×2\times on the real robot on our GPU.

Faster baselines offer diminishing benefit, since the policy frequency bounds how fast a detector needs to be, while slower ones such as k-NN and STAC add noticeable latency to every step. VLA-TUG costs about the same as the cheapest learned baselines and far less than k-NN and STAC, while providing improved informativeness and timeliness on most unseen settings, which makes it a practical option for real-world deployment.

Answer to Q4. Efficiency. Yes: labeling, training, calibrating, and running the probe introduce minimal cost, and on a GPU the probe adds under 1% to the policy call and fits within the step period.

Steering Demonstration

We demonstrate how the four properties of VLA-TUG: (T.) timeliness, (C.) calibration, (I.) informativeness, and (E.) efficiency, can jointly support failure prevention through VLA steering. Our goal is to illustrate how an effective failure predictor can provide the signals needed to guide interventions rather than develop a steering method itself. We consider a steering architecture in which an expert model (e.g. a VLM) can invoke failure-recovery tools in response to the probe’s predictions. For our experiments, we implement these tools using failure-mode-specific heuristics in simulation and demonstrate corresponding interventions on a real robot.

For example, in the simulated rollout shown in Figure 6a, the task is “place the largest bowl inside the left compartment of the bowl drainer.” The probe raises an alarm before the bowl is mistakenly placed in the compartment on the right. After steering the end-effector towards the left side, the policy is resumed and the rollout succeeds. Without intervention, the bowl would be placed on the right side and the rollout fails.

Steering demonstrations showing failure scores and corrections for simulated and real-world tasks

Figure 6. (a) Example steering for the simulated task “place the largest bowl inside the left compartment of the bowl drainer.” The gripper drops the bowl into the right compartment with no steering and successfully drops it in the left compartment when steered towards the target location.

(b) Example steering for the real-world task “put the mushroom in the plate.” The gripper misses the mushroom without steering and successfully grasps it when steered towards the mushroom.

Figure 6. Steering demonstrations showing failure scores and corrections for simulated and real-world tasks. When the probe’s predicted failure score rises above the threshold, the VLA policy is automatically paused. For the simulated bowl task, we find that nudging the end-effector toward the drop-off location can help lead to success. For the real-world mushroom task, we find that nudging the wrist camera towards the target grasp object can help. Future work could systematically explore steering strategies. Refer to Figure 17a for more steering strategies and Figure 17b for another real world steering example.

Our real-world experiments similarly demonstrate that the joint properties of VLA-TUG help in steering a VLA away from failure. Figure 6b illustrates this with the “put the mushroom on the plate” task. Without VLA-TUG, the gripper misses the mushroom and hits the table. With it turned on, however, the probe predicts the imminent grasp failure (which would likely happen), and corrective action is taken. In the example, the corrective action is to pause the VLA, let an expert model shift trajectory to a successful one, and resume the VLA.

Such a strategy is possible with a failure detector like VLA-TUG, which anticipates the failure before it happens, rather than after it is too late. Our approach is agnostic towards the steering architecture. We leave exploration of robust steering strategies for future work. Nevertheless, these examples demonstrate the applicability of VLA-TUG to be utilized in safe and principled robotic systems.

Answer to Q5. Steering. Yes: Our proof-of-concept demonstrations show how VLA-TUG can provide the timing and failure mode information needed for targeted interventions during VLA execution. Timeliness creates an opportunity to intervene, calibration controls when interventions are triggered, informativeness guides which intervention is applied, and efficiency allows the probe to operate alongside the VLA.

Conclusion

We introduced VLA-TUG, a lightweight probe on the latent layers of a VLA that is timely, calibrated, informative, and efficient, making it well suited for real-world deployment. It is timely: supervising on the pre-failure part of the trajectory drives early detection, and the probe’s performance compared to the strongest learned baseline increases as the shift from the training tasks increases. It is calibrated: calibrating the FPR and the TPRearly_{\text{early}} jointly provides guarantees that hold in far more runs than the baselines under shift. It is informative: beyond a failure score, it names the likely failure mode. It is efficient: the probe adds under 1% to the cost of a policy call. Our steering demonstrations, in simulation and on a real robot, show that VLA-TUG can inform targeted interventions, and we release 3,180 annotated episodes to support further work on latent failure detection.

Limitations. First, our calibration guarantees rely on exchangeability between the calibration and test episodes, which weakens as the distribution shift grows. Empirically, we observe that the degradation is gradual. Second, we do not analyze the choice of source backbone embedding systematically. Third, the steering component, in which an expert model intervenes on a predicted failure, is heuristic and not evaluated in the scope of this work.

Future Work. Looking forward, first, we will adapt the probe and its calibration guarantees online, which our preliminary experiments suggest is computationally feasible. Second, we will maintain and extend our embedding dataset with additional backbones and action-expert layers to determine which layers contain the most generalizable failure prediction signal. Third, we will explore agentic frameworks as the expert model in our setup, combining fast and local inference of a VLA with general-purpose intervention and steering from a cloud-based LLM, triggered by a principled detector such as VLA-TUG.

Method Details

This section gives the details behind § 4.

Probe and Training

Inputs. At every policy step we average the backbone tokens of the agent camera, of the wrist camera, and of the (unpadded) language prompt, and concatenate the three means into et∈R6144e_t \in\mathbb{R}^{6144}. No other feature of the backbone is used. All models in this paper read this vector, the baselines included.

Probe. A single-layer LSTM with 256 hidden units reads e1:te_{1:t}. Three heads read its state, each an MLP with one hidden layer of 64 ReLU units: the failure score head (one sigmoid output), the failure mode head (logits over the 10 modes), and the semantic head (a 64-dimensional vector).

Semantic (CLIP) head. The target space of the semantic head is a PCA of the CLIP text embeddings of the timeline phrases (clip-vit-base-patch32), reduced to 64 dimensions with a PCA fitted on the training split only. The loss is InfoNCE with temperature 0.07: the normalized output of a step must pick the target of the step’s phrase among the targets of all phrases of the training split. A step whose phrase is not among them is left out of this term. At inference the head is only a 64-dimensional MLP output; no CLIP encoder runs.

Loss and training. Lscore\mathcal{L}_{\mathrm{score}} is a binary cross-entropy over the steps that are not excluded, weighted by the ratio of negative to positive steps of the training set. Lmode\mathcal{L}_{\mathrm{mode}} is a cross-entropy with class weights that balance the modes, on the positive steps of failed rollouts. Both auxiliary terms have weight 1. Early stopping follows Lscore\mathcal{L}_{\mathrm{score}} on the validation episodes only, so the auxiliary heads never choose the checkpoint.

Labeling window. The window ww of the onset and post-onset labels, and the window of the mode read-out (subsection C.2), is 2.4 s: 6 policy steps for GR00T on LIBERO, 20 on the real robot, and 10 steps (2.5 s) for π0.5\pi_{0.5}.

Supervision Strategies

Let ete_t be the backbone embedding at step tt and yt∈{0,1,∅}y_t \in\{0,1,\varnothing\} its target, where yt=∅y_t = \varnothing excludes the step from the loss. Let Yi∈{0,1}Y_i \in\{0,1\} be the episode label (success, failure) and ww the width of the window around tfailt_{\mathrm{fail}}. For a failed rollout (Yi=1Y_i = 1) the four strategies of Figure 2 are

Pre-onset. yt={1t≤tfail,∅t>tfail,y_t = \begin{cases} 1 & t \le t_{\mathrm{fail}},\\ \varnothing& t > t_{\mathrm{fail}}, \end{cases}

Onset. yt={1t∈[tfail−w,tfail],0t<tfail−w,∅t>tfail,y_t = \begin{cases} 1 & t \in[t_{\mathrm{fail}}-w,t_{\mathrm{fail}}],\\ 0 & t < t_{\mathrm{fail}}-w,\\ \varnothing& t > t_{\mathrm{fail}}, \end{cases}

Post-onset. yt={1t≥tfail−w,0t<tfail−w,y_t = \begin{cases} 1 & t \ge t_{\mathrm{fail}}-w,\\ 0 & t < t_{\mathrm{fail}}-w, \end{cases}

Trajectory. yt=1y_t = 1 for all tt.

In all four, yt=0y_t = 0 for every step of a successful rollout. Excluding a step (yt=∅y_t = \varnothing) is the same as giving it zero loss. Trajectory is the labeling of SAFE.

Calibration and Guarantees

Alarm rule and risks. For a constant threshold hh the alarm time of an episode is tdet(h)=min⁡{t:st≥h}t_{\mathrm{det}}(h) = \min\{t : s_t \ge h\}. The FPR(h)\mathrm{FPR}(h) is the probability that a successful episode has an alarm, and TPRearly(h)\mathrm{TPR}_{\mathrm{early}}(h) the probability that a failed episode has its first alarm at or before tfailt_{\mathrm{fail}}.

Learn-then-Test. We follow LTT ((Angelopoulos et al., 2022)) with two risks, the FPR and the early miss rate 1−TPRearly1-\mathrm{TPR}_{\mathrm{early}} (Section 2.4 there). A threshold hh is tested against the null that it violates either target, and its p-value is the larger of the two p-values (Proposition 6). Both losses are binary, so each p-value is an exact binomial tail (Section 3.2): the number of calibration failures without an alarm at or before tfailt_{\mathrm{fail}} is compared with Bin⁡(nf,1−β)\operatorname{Bin}(n_f,1-\beta), and the number of calibration successes with an alarm with Bin⁡(ns,α)\operatorname{Bin}(n_s,\alpha). We combine the p-values by multi-start fixed-sequence testing (Section 2.3.1, Algorithm 1, Proposition 3) over a grid of 181 thresholds sigmoid⁡(z)\operatorname{sigmoid}(z), z∈[−9,9]z \in[-9,9], ordered from low to high, with S=40S=40 starts and δ=0.1\delta=0.1, i.e. a level of 0.0025 per walk. By Theorem of Angelopoulos et al. (2022), the set Λ^\widehat{\Lambda} of accepted thresholds satisfies P(sup⁡h∈Λ^{R(h)}≤α)≥1−δ\mathbb{P}\left(\sup_{h\in\widehat{\Lambda}}\{R(h)\}\le\alpha\right)\ge1-\delta, where RR stands for each of our two risks with its target; that is, with probability at least 1−δ1-\delta every accepted threshold meets both targets, for episodes exchangeable with the calibration episodes. We select the accepted threshold with the highest calibration TPRearly\mathrm{TPR}_{\mathrm{early}} (ties: lowest calibration FPR, then lowest threshold), and abstain if none is accepted.

Max-conformal (MC). For STAC (whose accumulated scores are non-negative, so the maximum over steps is the final score), for ActProbe (as in its paper), and for our probe in the main table, we use max-conformal prediction. It takes the maximum score MiM_i of each of the nn calibration successes and sets the constant threshold to the kk-th smallest of them, with k=⌈(n+1)(1−α)⌉k = \lceil(n + 1)(1 - \alpha) \rceil, and to +∞+\infty if k>nk > n. For a new success exchangeable with the calibration successes, its maximum has a uniformly distributed rank, so the false-alarm probability is at most α\alpha, and conservative by less than 1/(n+1)1/(n + 1). It says nothing about the TPR.

Functional conformal (FC). For SAFE and the methods that follow its calibration (FailureSpot and the embedding-distance baselines), we use the one-sided functional conformal band of SAFE (Gu et al., 2025). The band is a time-varying threshold built from the score trajectories of the calibration successes: the mean trajectory gives its shape, the local spread its width over time, and a conformal quantile of the successes’ deviations sets its level so that a success stays below it with probability about 1−α1 - \alpha.

Which method gets which calibrator. SAFE-MLP, SAFE-LSTM, FailureSpot, and the three embedding-distance baselines use the SAFE band, as in their own work. STAC, ActProbe, and our probe use max-conformal.

Datasets and Labeling

Benchmarks and Splits

LIBERO-10 with occlusions. We use the 10-suite of LIBERO-Occ (Li et al., 2026a), an extension of LIBERO [15] that occludes the agent-view camera. Both policies fail rarely on the original benchmark, so we mix occluded and unoccluded episodes in equal parts to obtain enough failures (250 of each per policy).

LIBERO-X. LIBERO-X [27] introduces five levels of distribution shift. We use levels L1–L3 and select 218 tasks that cover diverse skills. Table 2 gives the task counts of the splits.

Real robot. We collected demonstrations for 8 tabletop manipulation tasks on a Kinova Gen3 arm (Table 4) and fine-tuned GR00T on them. The evaluation data are 399 rollouts of the fine-tuned policy (49 to 50 per task), 254 of them failures; Table 5 gives their failure modes.

Task instructionThird-person viewDemos
Close the drawer!Third-person view50
Flip the basket upright!Third-person view50
Push the pineapple off the stove!Third-person view100
Put the mushroom on the plate!Third-person view50
Put the tomato in the bin!Third-person view100
Stack the yellow block on the wooden block!Third-person view60
Take the eggplant off the plate!Third-person view50
Take the pineapple out of the drawer!Third-person view50

Table 4. The eight real-world tasks and the number of demonstrations used to fine-tune the policy (510 in total).

Modeπ0.5\pi_{0.5} LIBERO-10-Occ % of failuresπ0.5\pi_{0.5} LIBERO-10-Occ Time (s)GR00T LIBERO-10-Occ % of failuresGR00T LIBERO-10-Occ Time (s)GR00T LIBERO-X % of failuresGR00T LIBERO-X Time (s)GR00T real robot % of failuresGR00T real robot Time (s)
stuck_or_no_progress24.510.3±3.510.3 \pm3.530.28.6±3.78.6 \pm3.714.115.5±10.715.5 \pm10.712.23.8±3.03.8 \pm3.0
object_displacement18.211.9±5.311.9 \pm5.317.811.8±6.711.8 \pm6.710.815.1±10.915.1 \pm10.99.44.9±2.34.9 \pm2.3
placement_or_insertion13.615.2±4.215.2 \pm4.216.012.9±7.012.9 \pm7.09.921.9±10.221.9 \pm10.211.48.3±4.58.3 \pm4.5
grasp_failure17.39.6±5.49.6 \pm5.413.08.6±4.38.6 \pm4.39.012.0±9.712.0 \pm9.756.74.7±2.34.7 \pm2.3
open_close_failure7.312.6±4.812.6 \pm4.89.513.9±3.113.9 \pm3.14.718.7±11.518.7 \pm11.57.53.8±1.73.8 \pm1.7
wrong_target13.66.8±4.96.8 \pm4.98.94.9±4.44.9 \pm4.412.314.3±9.714.3 \pm9.70.0–
wrong_object0.0–1.85.3±3.15.3 \pm3.137.05.8±7.25.8 \pm7.21.22.0±0.82.0 \pm0.8
press_failure2.74.4±1.44.4 \pm1.41.810.7±4.810.7 \pm4.81.526.8±18.526.8 \pm18.50.0–
timeout2.725.9±0.025.9 \pm0.01.225.9±0.025.9 \pm0.00.260.0±0.060.0 \pm0.01.66.1±1.16.1 \pm1.1
other0.0–0.0–0.48.6±7.48.6 \pm7.40.0–
nn failed / total110/500169/5001258/2180254/399

Table 5. Failure-mode distribution and mean ±\pm std failure time (s) per mode, by policy and dataset.

Construction of the splits. The episodes of the held-out tasks form the unseen set. The episodes of the seen tasks are split 65/15/20% into train, validation, and test. The test part is halved, into the calibration set and the held-out seen set. The calibration sets of the LIBERO-X runs hold 158 to 209 episodes (66 to 90 successes and 92 to 120 failures). On the real robot each run holds one task out, so the calibration part has only 12 to 14 successes, a max-conformal threshold exists only for α\alpha of about 0.07 and above, and the curves of all methods start at an FPR of about 8%. The area up to 40% FPR is still defined and treats every method the same, but it is noisy. The seed changes the shuffle of the seen episodes and the initialization of the learned models; for the new-task and new-signature splits the unseen episodes are identical in the three seeds, so their spread is not a spread over different unseen tasks.

Labeling Pipeline

We label each rollout with Gemini (gemini-3.5-flash) from its video, in three passes. A coarse pass over the whole video of a failed rollout returns the failure mode and a failure time; a second pass on a slowed clip around that time refines the failure time to a single frame; and a third pass on the same video session gives the timeline: a short keyword phrase (2–6 words, e.g. “reaching for mug”) for each 3-second window, which provides the targets of the semantic head. The failure time tfailt_{\mathrm{fail}} is the earliest point at which the mistake responsible for the failure becomes apparent. Following (Gu et al., 2025), if there is no single mistake but a human operator would take over (for example when the robot is stuck), we use that time, and if the robot behaves plausibly but is too slow, the failure time is the end of the rollout.

The failure mode is one of 10 classes: wrong_object (acts on an object or control other than the one the task specifies); wrong_target (right object, wrong destination); grasp_failure (never acquires the correct object securely); object_displacement (knocks over, pushes away, or drops an object); placement_or_insertion_failure (right object and target, wrong final pose); open_close_failure (right container, but not the required open or closed state); press_failure (right button or knob, but no effect); stuck_or_no_progress (looping, oscillating, hovering, or freezing); timeout_or_insufficient_progress (purposeful but too slow, no identifiable mistake); and other. The full definitions and the prompts are on the project page, vla-tug.github.io, and on the card of each dataset.

Dataset Statistics and Release

Table 5 gives, for each policy and dataset, the share of each failure mode among the failures and the mean and standard deviation of its failure time.

We release the embeddings, the labels, the prompts, and the splits of the simulated datasets, 3,180 episodes, at vla-tug.github.io.

Experimental Setup

Baselines

We re-implement all baselines on the same inputs and splits as our probe. Where a baseline has a configuration grid, we run the whole grid on every split and select, for every setting, the configuration with the highest mean area on the validation episodes, never looking at the unseen tasks; ties go to the earlier configuration of the grid. LIBERO-10 and the real robot pool their runs into one choice per method. On LIBERO-X, SAFE uses one configuration for the three kinds of splits, as in its own protocol, and every other method chooses per kind of held-out set. Our probe uses one architecture in every setting.

Embedding distance. Mahalanobis, Euclidean, and PCA-KMeans are SAFE’s embedding baselines (Gu et al., 2025). The score of a step is Dsucc(x)−Dfail(x)D_{\mathrm{succ}}(x)-D_{\mathrm{fail}}(x), where DcD_c is the Mahalanobis distance to a Gaussian fitted to the steps of class cc, the mean Euclidean distance to the kk nearest stored steps of that class (Euclidean, also called k-NN), or the distance to the nearest of a few kk-means centroids in a PCA-reduced space (PCA-KMeans); it can be accumulated over the steps. We run a sweep of 26 configurations (Mahalanobis 2, Euclidean 6, PCA-KMeans 18).

Action consistency. STAC [1] compares the unexecuted tail of the previous plan with the first actions of the new plan, using the K=10K = 10 plans the policy samples at every call and the maximum mean discrepancy with an RBF kernel; nothing is learned. We run 8 variants (MMD over 3, 6, or 7 action dimensions, with the step score or accumulated; and STAC-Single, SAFE’s single-plan variant). ActProbe [10] is a small LSTM on two action signals and an embedding of the task instruction; we train it with the recipe of its repository and select its checkpoint with its own validation criterion. FailureSpot (Ma et al., 2026) has no public code, so we implemented it from the paper: a detector on the pooled features is trained on weak targets computed from action consistency and magnitude, and then fine-tuned on timestamp labels of the 15% most uncertain episodes.

Latent. SAFE-MLP and SAFE-LSTM (Gu et al., 2025) are written from SAFE’s code with its losses and its labels: every step of a failed rollout is positive, i.e. our trajectory label. We run SAFE’s learning-rate and regularization grid, 12 configurations per model. SAFE trains for a fixed number of epochs without validation; we stop early after 50 epochs without improvement of its validation loss (cap: 1,000 epochs) and keep the best epoch, so that all learned methods are treated alike.

Metrics

We define three events for a rollout: detection A={tdet≤tmax∣Y=1}\mathcal{A} = \{t_{\mathrm{det}} \le t_{\mathrm{max}} \mid Y = 1\}, early detection Aearly={tdet≤tfail∣Y=1}\mathcal{A}_{\mathrm{early}} = \{t_{\mathrm{det}} \le t_{\mathrm{fail}} \mid Y = 1\}, and no alarm N={tdet=null∣Y=0}\mathcal{N} = \{t_{\mathrm{det}} = \mathrm{null} \mid Y = 0\}. We count A\mathcal{A} as a true positive (TP) and Aearly\mathcal{A}_{\mathrm{early}} as an early true positive (TPearly\mathrm{TP}_{\mathrm{early}}); the complements are the false negatives. The TPR and FPR follow their standard definitions, and

TPRearly=TPearlyTPearly+FNearly,(7)\mathrm{TPR}_{\mathrm{early}} = \frac{\mathrm{TP}_{\mathrm{early}}}{\mathrm{TP}_{\mathrm{early}} + \mathrm{FN}_{\mathrm{early}}}, \tag*{(7)}
AUROCearly=10.4∫00.4TPRearly(FPR) dFPR,(8)\mathrm{AUROC}_{\mathrm{early}} = \frac{1}{0.4} \int_{0}^{0.4} \mathrm{TPR}_{\mathrm{early}}(\mathrm{FPR})\,d\mathrm{FPR}, \tag*{(8)}

so that AUROC⁡early\operatorname{AUROC}_{\mathrm{early}} is reported in % and a detector with TPR⁡early=100%\operatorname{TPR}_{\mathrm{early}} = 100\% everywhere scores 100.

Full Results

Table 6 gives Table 3 with the standard deviation over the splits, and Figure 7 shows the curves of every setting and method. On the real robot the standard deviation is over the 40 splits (8 held-out tasks, 5 seeds).

MethodCal.LIBERO-10 π0.5\pi_{0.5} SeenLIBERO-10 π0.5\pi_{0.5} UnseenLIBERO-10 GR00T SeenLIBERO-10 GR00T UnseenLIBERO-X task SeenLIBERO-X task UnseenLIBERO-X signature SeenLIBERO-X signature UnseenLIBERO-X skill SeenLIBERO-X skill UnseenReal robot SeenReal robot UnseenFit (s)State (MB)GPU (ms)CPU (ms)
SAFE-LSTMFC75.5±13.434.9±3.574.1±13.051.2±8.464.6±3.540.4±2.364.7±7.441.2±7.162.5±2.430.1±8.544.5±13.525.1±18.843.526.20.0660.86
SAFE-MLPFC74.1±2.039.9±15.162.9±18.239.2±9.459.3±2.042.3±2.953.7±2.446.2±2.356.8±6.023.5±12.440.1±15.920.2±16.6114.56.30.0340.032
STACMC33.8±5.136.4±5.324.2±7.517.5±9.821.0±2.328.0±0.125.6±3.728.7±0.324.6±5.021.5±8.0––0.00.021.30–
ActProbeMC39.3±11.326.7±2.632.4±16.830.1±9.312.6±4.222.7±4.611.9±3.113.9±1.520.3±9.413.6±7.1––7.10.090.150.11
FailureSpotFC61.9±3.036.2±21.036.9±19.844.7±18.626.3±2.120.4±1.228.0±2.526.2±5.630.4±3.419.6±7.9––9.26.40.0330.029
MahalanobisFC85.7±4.140.2±15.570.7±6.429.2±8.769.6±6.230.6±2.264.4±3.022.3±4.071.2±3.115.9±9.643.0±16.519.6±2.515.2302.00.415.37
EuclideanFC73.1±4.424.2±10.656.0±6.823.1±11.470.5±4.228.5±3.065.4±2.119.7±0.868.5±4.819.0±6.545.1±14.28.3±12.80.4305613.19368.83
PCA-KMeansFC35.7±12.212.9±5.144.5±8.720.0±0.031.7±3.425.0±1.235.3±1.832.5±1.737.8±7.720.7±7.124.0±11.419.4±7.739.91.60.120.055
VLA-TUG (Ours)MC75.1±12.547.9±20.970.5±10.447.4±13.567.9±5.040.2±1.267.0±6.647.0±0.667.8±4.839.0±7.047.6±10.925.8±11.66.826.50.120.94

Table 6.

TPR early against FPR plots for every setting and method

Figure 7. TPRearly_{\mathrm{early}} against FPR for every setting and method, on seen (top) and unseen (bottom) tasks; mean over the splits of a setting.

Results in Detail

Calibration and Guarantees

This section gives the details behind subsection 6.2.

Setup. We run this analysis on LIBERO-X only. LIBERO-10 has only a few tens of failures per split, and the real robot has only 12 to 14 calibration successes per run, too few for LTT to certify any TPR⁡early\operatorname{TPR}_{\mathrm{early}} target together with an FPR target in at least half of the runs; its LTT rows would be empty. The 14 runs are the 3 NEW-TASK, 3 NEW-SIGNATURE, and 8 NEW-SKILL splits. We use the FPR targets 10, 15, 20, 25, 30, 40, and 50%. For each of them, we report the highest TPR⁡early\operatorname{TPR}_{\mathrm{early}} target that LTT certifies together with it in at least half of the runs, searched on a grid from 10% to 95% in steps of 5%. LTT uses δ=0.1\delta= 0.1 and S=40S = 40 starting points for its fixed-sequence walks (subsection A.3). Table 7 gives for each FPR target the shares of runs in which the targets hold on the held-out seen and unseen episodes, the realized rates, and the PPV bound, for LTT and for the two calibrators of the baselines, max-conformal (MC) and functional conformal (FC).

FPR targetTPR-early targetPPV boundMethodRunsSeen tasks: FPR holdsSeen tasks: TPR holdsSeen tasks: FPRSeen tasks: TPR earlySeen tasks: PPVSeen tasks: PPV holds (own π\pi)Unseen tasks: FPR holdsUnseen tasks: TPR holdsUnseen tasks: FPRUnseen tasks: TPR earlyUnseen tasks: PPVUnseen tasks: PPV holds (own π\pi)
10%––LTT0% (0/14)––––––––––––
–––Max conformal (MC)100% (14/14)57%–9.3±4.453.4±12.790.1––50%–15.6±12.335.3±20.580.7–
–––Functional conformal (FC)100% (14/14)71%–9.8±5.833.9±17.290.7––36%–14.9±9.426.3±16.284.3–
15%15%58.1LTT50% (7/14)100%100%5.3±3.139.2±8.092.5100%100%86%71%9.8±11.527.1±14.785.8100%100%
–––Max conformal (MC)100% (14/14)57%–14.2±7.364.8±9.088.1––14%–25.8±12.350.7±15.777.5–
–––Functional conformal (FC)100% (14/14)64%–13.2±6.140.0±13.988.9––29%–21.9±11.932.8±18.581.4–
20%35%70.7LTT50% (7/14)100%100%10.4±3.558.1±5.489.7100%100%71%57%15.5±10.437.0±16.481.1100%100%
–––Max conformal (MC)100% (14/14)57%–19.2±7.272.9±5.885.5––21%–37.2±13.861.8±13.973.6–
–––Functional conformal (FC)100% (14/14)79%–18.0±6.154.4±11.286.6––21%–30.7±11.147.0±16.777.9–
25%35%65.7LTT57% (8/14)100%100%12.4±3.764.5±5.288.8100%100%75%75%22.7±11.649.2±18.979.9100%100%
–––Max conformal (MC)100% (14/14)57%–25.1±8.180.0±5.083.1––7%–48.7±14.372.1±11.870.2–
–––Functional conformal (FC)100% (14/14)79%–22.7±7.063.8±12.084.7––7%–42.5±12.157.5±16.174.3–
30%45%67.4LTT50% (7/14)100%100%15.2±3.568.2±4.687.4100%100%86%57%24.5±6.650.6±12.677.9100%100%
–––Max conformal (MC)100% (14/14)57%–28.6±7.983.1±4.581.6––0%–56.9±13.477.3±10.267.9–
–––Functional conformal (FC)100% (14/14)71%–27.8±7.471.2±11.182.5––7%–54.1±15.367.3±15.070.7–
40%55%65.8LTT71% (10/14)100%100%24.1±7.378.4±5.783.5100%100%60%80%43.3±14.666.6±13.870.580%100%
–––Max conformal (MC)100% (14/14)64%–37.6±6.287.5±3.177.7––0%–69.8±14.184.9±7.965.0–
–––Functional conformal (FC)100% (14/14)64%–38.7±7.382.0±8.177.8––0%–74.0±15.080.2±8.165.0–
50%65%64.3LTT71% (10/14)100%100%31.1±5.484.5±3.380.2100%100%40%90%59.2±18.377.6±10.767.770%60%
–––Max conformal (MC)100% (14/14)71%–46.1±6.292.1±2.274.6––40%–79.2±13.289.8±6.863.0–
–––Functional conformal (FC)100% (14/14)50%–49.0±7.790.6±3.373.8––0%–83.1±16.287.2±5.962.4–

Table 7. Learn-then-Test (LTT), max-conformal (MC), and functional conformal (FC) at the same FPR target, over the 14 LIBERO-X runs (δ=0.1\delta= 0.1). LTT certifies the TPR-early target of the second column together with the FPR target, and abstains otherwise; its percentages are over the certified runs. PPV bound: βπ/(βπ+α(1−π))\beta\pi/(\beta\pi+ \alpha(1-\pi)).

How to read Table 7. Runs is the share of runs for which LTT certifies the targets; the other columns of an LTT row are over these runs. Holds is the share of runs in which the target is met on the evaluated episodes. The PPV bound is βπ/(βπ+α(1−π))\beta\pi/(\beta\pi+\alpha(1-\pi)) with π\pi the prevalence of the calibration set, and PPV holds is the share of runs in which the realized PPV is at least the bound (computed with the calibration set’s prevalence, or with that of the evaluated set).

PPV bound. The bound follows from the two LTT targets, so for VLA-TUG it comes directly from the FPR and TPRearly_{\mathrm{early}} targets we calibrate for. It depends on the prevalence π\pi, which we estimate on the calibration set. On seen tasks the estimate is stable. Under shift it can drift on individual NEW-SKILL folds, where the share of failures can differ a lot between the calibration set and the held-out tasks; this is what limits the bound on unseen tasks, and using the unseen set’s own π\pi removes most of the misses. A moving average of π\pi over the observations at deployment time could deal with more severe shift.

Unseen tasks. On the NEW-TASK and NEW-SIGNATURE splits, the three runs of each split share the same unseen tasks and differ only in the calibration data and the initialization. The held-out tasks define the shift each split is designed to test, so we cannot redraw them between runs. Only the eight NEW-SKILL folds hold out different tasks. The shares on unseen tasks therefore rest on fewer distinct unseen sets than runs, so they indicate trends and not precise rates.

Informativeness

This section gives the details behind subsection 6.3.

Classes. The failure modes are strongly imbalanced, so we report a balanced accuracy: the mean over classes of the share of their failures predicted correctly. We merge the two smallest classes repeatedly, once per dataset, until every class holds at least 5% of the failures. The rule uses label frequencies only. The mode head is not retrained: a merged class gets the sum of the probabilities of its modes. The accuracy is computed on the predictions pooled over the splits of a setting, since a split has few failures of a rare class. We exclude LIBERO-10: it has too few failures for a per-class accuracy, since even after merging its smallest classes hold only 6 to 16 failures, so one failure changes a class’s accuracy by several points.

Table 6. Table 3 with the standard deviation over the splits: area under TPR early vs. FPR up to 40% (in %). Dashes: the action chunks the method needs were not saved. Best in bold, second best in italics.

Prediction. A failed episode is predicted at an anchor step: the class with the highest mean probability over the 2.4 s window before the anchor and the anchor itself (the top three classes for top-3). The anchor is either the probe’s first alarm (max-conformal at α=0.2\alpha= 0.2; episodes without an alarm are not scored) or the true failure time, which is an upper bound that a deployed system can reach. The baselines predict the most common training classes (always) or guess uniformly among the training classes.

Cost and Latency

This section gives the details behind subsection 6.4.

How the VLA runs. At each policy step the VLA receives the camera images and the prompt and produces a chunk of mm actions (m=8m = 8 on LIBERO-X, 5 for π0.5\pi_{0.5} on LIBERO-10, 6 on the real robot), which the robot executes at its control rate (20 Hz in simulation, 50 Hz on the real robot). A policy step therefore lasts mm divided by the control rate: 400 ms on LIBERO-X, 250 ms for π0.5\pi_{0.5}, and 120 ms on the real robot. The probe reads the embedding of each call and must return its score well within one step.

Measurement. All times are measured on one RTX 4090 and 32 CPU threads. The fit time is the training time of the probe or the SAFE models, or the time to fit the statistics of the distance baselines. The state is what a fitted method must keep to score a new episode: the weights, the stored training steps of the k-NN, or the statistics of the distance baselines. The latency is the online latency: the time to turn one new policy step into a score, one embedding at a time and with the recurrent state of the previous step, after a warm-up and with the device synchronized, as the median over up to 2,000 steps. The CPU is timed with four threads. Online latency is the one that matters for deployment.

Baselines. The k-NN baseline has to keep every training step and compare each new step with all of them, so its cost grows with the training set; Mahalanobis keeps two 6144 × 6144 matrices. STAC needs the K=10K = 10 plans of the policy at every call, which no other method needs, so its latency is its scoring time plus the time to sample them.

Probe. On the GPU the LSTM takes most of the time; the mode head and the CLIP head add little, and on the CPU they add nothing measurable. The probe is not the fastest online method (SAFE-LSTM and the MLP detectors are faster), but it is far below the duration of a policy step. The average, and are lower on every kind of held-out split, seen and unseen; labeling the steps after the onset as positive is worst. On LIBERO-10 the differences are smaller (−1%-1\% to −14%-14\% on average), within the spread across splits for π0.5\pi_{0.5}, and clear on seen tasks with GROOT.

Method\multicolumn{8}{c}{Top-1 accuracy (%)}\multicolumn{8}{c}{Top-3 accuracy (%)}
\multicolumn{2}{c}{task}\multicolumn{2}{c}{signature}\multicolumn{2}{c}{skill}\multicolumn{2}{c}{Real robot}\multicolumn{2}{c}{task}\multicolumn{2}{c}{signature}\multicolumn{2}{c}{skill}\multicolumn{2}{c}{Real robot}
SeenUnseenSeenUnseenSeenUnseenSeenUnseenSeenUnseenSeenUnseenSeenUnseenSeenUnseen
Most common14.314.314.314.314.314.320.020.042.942.942.942.943.939.060.432.6
Random14.314.314.314.314.314.320.020.042.942.942.942.942.942.960.060.0
VLA-Tug (Ours), own detection (first alarm)
onset_pre34.225.535.622.633.620.354.321.765.159.969.856.369.456.388.270.4
onset_traj39.325.937.129.939.727.057.215.871.858.871.461.169.859.787.758.6
onset28.019.332.819.525.017.254.913.263.854.659.350.456.048.188.057.8
onset_post31.119.832.122.132.525.256.618.865.355.769.658.065.257.291.454.4
VLA-Tug (Ours), oracle onset (upper bound)
onset_pre41.331.041.030.242.430.068.412.173.065.078.466.872.964.892.557.1
onset_traj36.129.339.829.939.127.466.112.469.559.373.357.171.559.493.853.1
onset37.530.539.729.237.127.170.211.474.267.575.066.970.162.792.660.0
onset_post23.020.028.022.233.021.467.816.364.859.269.656.266.058.894.552.9

Table 8. Balanced top-1 and top-3 failure-mode accuracy (%), pooled over the splits of each kind. Shaded rows: our probe, by label variant; the oracle rows use the true failure time and are an upper bound. Best in bold, second best in italics, among the other rows.

Calibration. Per target pair, max-conformal takes 0.1 ms, functional conformal 0.7 ms, and LTT 1.3 ms on a CPU. The calibration sets hold 158 to 209 labeled episodes, of which 92 to 120 are failures.

Ablations

We ablate the supervision strategy, the auxiliary heads, and the failure time used for evaluation.

Supervision strategy. Table 9 compares our pre-onset label (every step before the failure time is positive) with the window before the onset only, every step of the failed rollout, and the window before the onset together with all steps after it. The pre-onset label is the best, or within the spread across splits, in nearly every setting. On LIBERO-X the alternatives lose 24% to 50% of the detection area on average, and are lower on every kind of held-out split, seen and unseen; labeling the steps after the onset as positive is worst. On LIBERO-10 the differences are smaller (−1%-1\% to −14%-14\% on average), within the spread across splits for π0.5\pi_{0.5}, and clear on seen tasks with GROOT.

Table 9. Ablations of our probe in the simulated settings: area under TPR early vs. FPR up to 40% (in %), mean ± standard deviation over the splits, and in brackets the change relative to our probe (shaded). Best in bold, second best in italics, within each block.

LIBERO-10 π0.5\pi_{0.5} SeenLIBERO-10 π0.5\pi_{0.5} UnseenLIBERO-10 GR00T SeenLIBERO-10 GR00T UnseenLIBERO-X task SeenLIBERO-X task UnseenLIBERO-X signature SeenLIBERO-X signature UnseenLIBERO-X skill SeenLIBERO-X skill Unseen
Heads (LSTM, pre-onset label)
Modes + CLIP (ours)75.1±12.575.1 \pm12.547.9±20.947.9 \pm20.970.5±10.470.5 \pm10.447.4±13.547.4 \pm13.567.9±5.067.9 \pm5.040.2±1.240.2 \pm1.267.0±6.667.0 \pm6.647.0±0.647.0 \pm0.667.8±4.867.8 \pm4.839.0±7.039.0 \pm7.0
No head82.3±6.482.3 \pm6.4 (+10%)51.1±4.051.1 \pm4.0 (+7%)70.1±11.070.1 \pm11.0 (−1%)55.1±5.555.1 \pm5.5 (+16%)67.1±1.867.1 \pm1.8 (−1%)42.7±1.542.7 \pm1.5 (+6%)68.3±5.768.3 \pm5.7 (+2%)45.0±1.345.0 \pm1.3 (−4%)69.2±4.469.2 \pm4.4 (+2%)43.6±11.543.6 \pm11.5 (+12%)
Modes only80.4±8.480.4 \pm8.4 (+7%)48.3±8.848.3 \pm8.8 (+1%)70.9±8.270.9 \pm8.2 (+0%)52.9±8.452.9 \pm8.4 (+12%)69.4±1.069.4 \pm1.0 (+2%)39.9±3.839.9 \pm3.8 (−1%)68.0±8.868.0 \pm8.8 (+2%)46.6±3.546.6 \pm3.5 (−1%)68.3±3.568.3 \pm3.5 (+1%)43.7±10.243.7 \pm10.2 (+12%)
CLIP only78.2±11.478.2 \pm11.4 (+4%)58.3±15.358.3 \pm15.3 (+22%)72.6±4.872.6 \pm4.8 (+3%)44.4±12.744.4 \pm12.7 (−6%)66.1±3.166.1 \pm3.1 (−3%)41.4±2.441.4 \pm2.4 (+3%)67.3±8.367.3 \pm8.3 (+1%)46.2±2.846.2 \pm2.8 (−2%)67.8±5.567.8 \pm5.5 (+0%)38.5±9.238.5 \pm9.2 (−1%)
Training strategy (modes + CLIP heads)
onset pre (ours)75.1±12.575.1 \pm12.547.9±20.947.9 \pm20.970.5±10.470.5 \pm10.447.4±13.547.4 \pm13.567.9±5.067.9 \pm5.040.2±1.240.2 \pm1.267.0±6.667.0 \pm6.647.0±0.647.0 \pm0.667.8±4.867.8 \pm4.839.0±7.039.0 \pm7.0
onset traj77.6±6.577.6 \pm6.5 (+3%)52.0±14.952.0 \pm14.9 (+9%)64.5±2.064.5 \pm2.0 (−9%)43.8±10.943.8 \pm10.9 (−8%)49.7±2.449.7 \pm2.4 (−27%)38.4±2.938.4 \pm2.9 (−5%)48.0±6.848.0 \pm6.8 (−28%)31.3±3.131.3 \pm3.1 (−33%)52.1±4.352.1 \pm4.3 (−23%)21.6±3.921.6 \pm3.9 (−45%)
onset73.3±3.573.3 \pm3.5 (−2%)45.4±14.645.4 \pm14.6 (−5%)60.3±7.760.3 \pm7.7 (−15%)39.5±10.739.5 \pm10.7 (−17%)48.8±1.448.8 \pm1.4 (−28%)39.9±6.639.9 \pm6.6 (−1%)47.6±11.847.6 \pm11.8 (−29%)36.6±5.736.6 \pm5.7 (−22%)46.4±5.446.4 \pm5.4 (−32%)25.5±9.025.5 \pm9.0 (−35%)
onset post73.5±4.373.5 \pm4.3 (−2%)50.2±11.050.2 \pm11.0 (+5%)37.4±3.037.4 \pm3.0 (−47%)41.8±11.841.8 \pm11.8 (−12%)24.1±2.424.1 \pm2.4 (−65%)31.3±0.931.3 \pm0.9 (−22%)28.3±3.628.3 \pm3.6 (−58%)25.0±0.325.0 \pm0.3 (−47%)30.6±2.530.6 \pm2.5 (−55%)17.0±3.417.0 \pm3.4 (−56%)

Auxiliary heads. We include the mode and CLIP heads because their labels already exist for every failure. We find no consistent benefit from either on detection (Table 9): removing the heads changes the area by less than the spread across splits in most settings. For mode prediction (Table 10), the CLIP head changes top-1 accuracy by less than one point on average, while training the mode head jointly with the failure score adds three to four points over a mode-only probe. The CLIP alignment might help some subpopulations and hurt others, which we leave to future work.

Table 10. What the CLIP head and the score term add to failure-mode prediction: top-1 and top-3 accuracy (in %), mean ± standard deviation over the splits. Our probe is shaded. Best in bold, second best in italics, per column and anchor.

Top-1 accuracy (%) task SeenTop-1 accuracy (%) task UnseenTop-1 accuracy (%) signature SeenTop-1 accuracy (%) signature UnseenTop-1 accuracy (%) skill SeenTop-1 accuracy (%) skill UnseenTop-1 accuracy (%) Real robot SeenTop-1 accuracy (%) Real robot UnseenTop-3 accuracy (%) task SeenTop-3 accuracy (%) task UnseenTop-3 accuracy (%) signature SeenTop-3 accuracy (%) signature UnseenTop-3 accuracy (%) skill SeenTop-3 accuracy (%) skill UnseenTop-3 accuracy (%) Real robot SeenTop-3 accuracy (%) Real robot Unseen
At the first alarm of our probe
Score + modes + CLIP (ours)41.0±1.441.0 \pm1.427.2±2.527.2 \pm2.537.3±7.237.3 \pm7.223.7±0.723.7 \pm0.742.3±5.342.3 \pm5.332.2±4.332.2 \pm4.357.9±10.657.9 \pm10.661.5±44.061.5 \pm44.076.0±2.676.0 \pm2.665.9±5.765.9 \pm5.774.2±1.074.2 \pm1.062.8±1.862.8 \pm1.874.6±3.474.6 \pm3.463.2±5.663.2 \pm5.693.3±7.393.3 \pm7.3100.0±0.0100.0 \pm0.0
Score + modes40.6±3.740.6 \pm3.725.8±2.525.8 \pm2.544.0±4.644.0 \pm4.625.6±0.825.6 \pm0.842.5±5.342.5 \pm5.328.0±7.028.0 \pm7.057.4±11.857.4 \pm11.864.1±45.464.1 \pm45.476.4±1.576.4 \pm1.564.0±2.864.0 \pm2.875.6±2.875.6 \pm2.863.3±1.063.3 \pm1.075.7±2.975.7 \pm2.961.9±7.961.9 \pm7.991.9±7.091.9 \pm7.0100.0100.0
Modes + CLIP37.3±5.837.3 \pm5.823.7±1.423.7 \pm1.429.3±1.129.3 \pm1.117.7±9.317.7 \pm9.344.0±6.544.0 \pm6.530.5±5.330.5 \pm5.355.0±14.255.0 \pm14.259.0±42.859.0 \pm42.872.3±1.772.3 \pm1.764.9±3.364.9 \pm3.362.6±1.562.6 \pm1.554.3±13.454.3 \pm13.475.3±2.275.3 \pm2.261.0±5.661.0 \pm5.690.4±7.990.4 \pm7.9100.0±0.0100.0 \pm0.0
Modes36.5±6.036.5 \pm6.023.7±3.223.7 \pm3.235.7±5.735.7 \pm5.720.6±4.820.6 \pm4.840.9±6.740.9 \pm6.730.1±4.730.1 \pm4.753.8±12.553.8 \pm12.556.4±41.856.4 \pm41.871.9±7.871.9 \pm7.863.6±1.463.6 \pm1.472.2±6.672.2 \pm6.657.3±7.357.3 \pm7.371.5±3.471.5 \pm3.461.6±8.761.6 \pm8.794.7±7.294.7 \pm7.2100.0±0.0100.0 \pm0.0
At the true failure onset (upper bound)
Score + modes + CLIP (ours)49.2±5.749.2 \pm5.740.6±4.540.6 \pm4.547.7±9.047.7 \pm9.040.9±3.140.9 \pm3.148.6±5.948.6 \pm5.939.6±8.039.6 \pm8.061.9±8.461.9 \pm8.423.7±22.023.7 \pm22.080.3±3.680.3 \pm3.673.6±2.673.6 \pm2.681.7±5.881.7 \pm5.877.5±1.577.5 \pm1.577.4±4.077.4 \pm4.074.7±6.774.7 \pm6.795.9±4.695.9 \pm4.662.4±35.162.4 \pm35.1
Score + modes54.7±3.254.7 \pm3.239.6±5.039.6 \pm5.049.1±4.549.1 \pm4.539.0±0.439.0 \pm0.448.4±8.048.4 \pm8.040.4±8.040.4 \pm8.059.4±5.059.4 \pm5.025.7±21.125.7 \pm21.182.5±3.582.5 \pm3.572.6±0.772.6 \pm0.779.6±3.879.6 \pm3.874.5±1.774.5 \pm1.777.9±3.077.9 \pm3.073.4±8.173.4 \pm8.196.4±3.296.4 \pm3.262.8±33.362.8 \pm33.3
Modes + CLIP42.4±5.342.4 \pm5.337.0±7.737.0 \pm7.733.7±5.933.7 \pm5.931.2±1.731.2 \pm1.749.4±5.849.4 \pm5.839.8±6.439.8 \pm6.462.1±8.862.1 \pm8.819.5±19.819.5 \pm19.877.3±10.277.3 \pm10.269.1±5.369.1 \pm5.369.2±15.669.2 \pm15.661.4±23.261.4 \pm23.277.5±4.777.5 \pm4.772.2±5.972.2 \pm5.996.4±3.996.4 \pm3.959.5±32.959.5 \pm32.9
Modes48.9±5.648.9 \pm5.638.1±8.138.1 \pm8.142.7±7.242.7 \pm7.235.0±12.035.0 \pm12.049.8±4.949.8 \pm4.940.8±5.540.8 \pm5.559.4±7.659.4 \pm7.632.4±26.332.4 \pm26.376.4±4.776.4 \pm4.768.8±5.468.8 \pm5.474.2±5.374.2 \pm5.367.4±13.267.4 \pm13.277.4±4.077.4 \pm4.073.2±7.373.2 \pm7.396.4±3.396.4 \pm3.366.1±34.866.1 \pm34.8

Sensitivity to the failure time. Moving the failure time later makes the metric easier, since more alarms then count as early, so we check how much our results depend on it.

We score the saved scores again with the failure time of every failed episode moved 1 and 2 s later, with the same thresholds and without retraining (Table 11). Every method gains, but late detectors gain more than VLA-TUG, which already alarms early: on unseen LIBERO-X VLA-TUG gains 2.1 to 2.4 points of area over 2 s and SAFE-LSTM 3.5 to 4.0. On LIBERO-X our rank among the methods does not change, and our margins hold, so these results do not depend on the exact failure time. On the real robot, where our lead is a single point, SAFE-LSTM overtakes VLA-TUG at 1 s for the same reason.

Table 11. Early-detection area (in %) on the unseen episodes when the failure time is moved later by 0, 1, and 2 s; nothing is retrained. Ours is shaded. Best in bold, second best in italics.

MethodLIBERO-10 π0.5\pi_{0.5} 0 sLIBERO-10 π0.5\pi_{0.5} +1 sLIBERO-10 π0.5\pi_{0.5} +2 sLIBERO-10 GR00T 0 sLIBERO-10 GR00T +1 sLIBERO-10 GR00T +2 sLIBERO-X task 0 sLIBERO-X task +1 sLIBERO-X task +2 sLIBERO-X signature 0 sLIBERO-X signature +1 sLIBERO-X signature +2 sLIBERO-X skill 0 sLIBERO-X skill +1 sLIBERO-X skill +2 sReal robot 0 sReal robot +1 sReal robot +2 s
SAFE-LSTM34.937.239.251.253.155.040.442.144.441.242.844.930.131.733.625.126.828.9
SAFE-MLP39.942.143.939.240.643.942.343.344.746.247.247.923.524.325.720.221.422.8
STAC36.443.053.217.522.027.628.029.532.228.730.634.021.522.625.1–––
ActProbe26.631.037.230.133.338.622.724.727.513.916.118.813.614.315.7–––
FailureSpot36.239.442.744.746.546.620.420.921.726.227.229.119.620.822.8–––
Mahalanobis40.242.145.929.230.432.630.631.734.022.323.124.415.916.617.419.619.719.8
Euclidean24.228.231.223.125.430.428.529.731.119.720.822.519.020.021.48.39.712.2
PCA-KMeans12.912.913.820.020.020.025.025.325.932.534.036.420.721.523.119.419.920.3
VLA-TuG (Ours)47.952.555.447.449.149.340.241.342.347.048.249.139.040.541.425.826.326.5

Qualitative Results

Latent Space

VLA-TUG encodes the VLA backbone embedding of each step with an LSTM into a latent state, which three MLP heads read: the failure score, the failure mode, and a CLIP embedding. The following figures show four of these spaces in two dimensions: the backbone embedding itself (the input), the LSTM state, and the hidden layers of the failure score head and of the failure mode head. Each is projected onto its first two principal components, fitted on the pre-failure steps of the training episodes only, and drawn for the training episodes and for the unseen tasks of the LIBERO-X NEW-TASK split. The points are the steps up to the failure of failed episodes and all steps of successful episodes; the ellipses are the 1σ\sigma ellipses of a Gaussian fitted to each class, drawn over the points. The classes are the outcome (success in blue, failure in red) or the failure mode. The mode head is trained on failures only and has only a mode plot.

VLA backbone embedding, by outcome, with Train and Unseen panels, PC1 and PC2 axes, and success and failure episode legends

Figure 8. VLA backbone embedding, by outcome.

VLA backbone embedding, by failure mode, with Train and Unseen panels, PC1 and PC2 axes, and failure-mode episode legends

Figure 9. VLA backbone embedding, by failure mode.

Probe LSTM state by outcome, showing Train and Unseen panels with PC1 (5.0%) and PC2 (4.3%) axes, and episodes (train / unseen) legend for success (469 / 200) and failure (669 / 230)

Figure 10. Probe LSTM state, by outcome.

Probe LSTM state by failure mode, showing Train and Unseen panels with PC1 (5.0%) and PC2 (4.3%) axes, and episodes (train / unseen) legend for success (469 / 200), wrong_object (256 / 67), stuck_or_no_progress (99 / 33), wrong_target (67 / 40), object_displacement (76 / 26), placement_or_insertion_failure (68 / 23), grasp_failure (61 / 16), open_close_failure (28 / 15), and press_failure (9 / 8)

Figure 11. Probe LSTM state, by failure mode.

Train and unseen scatter plots labelled “Train” and “Unseen”, with axes PC1 (52.5%) and PC2 (8.4%), episodes (train / unseen), and legend success (469 / 200) and failure (669 / 230)

Figure 12. Failure score head layer, by outcome.

Train and unseen scatter plots labelled “Train” and “Unseen”, with axes PC1 (52.5%) and PC2 (8.4%), episodes (train / unseen), and legend success (469 / 200), wrong_object (256 / 67), stuck_or_no_progs (99 / 33), wrong_target (67 / 40), object_displacement (76 / 26), placement_or_insertion_failure (68 / 23), grasp_failure (61 / 16), open_close_failure (28 / 15), and press_failure (9 / 8)

Figure 13. Failure score head layer, by failure mode.

Latent Trajectories

To see how an episode moves through these spaces, Figure 15 and Figure 16 follow one success and one failure step by step, for episodes of seen and of unseen tasks. Each panel shows one space in its first two principal components: (a) the VLA backbone embedding, (b) the probe’s LSTM state, and the hidden layers of (c) the failure score head and (d) the failure mode head. The points and ellipses are those of the episodes of the set shown (the training episodes, or the unseen tasks), grouped by outcome and failure mode, and the mode head is shown on failures only. The principal components are always those fitted on the training episodes. A trajectory starts at the ring; a failure is dashed and ends at the cross, at its failure step.

Seen tasks (training episodes)

Figure 15. Seen tasks (training episodes). A success (green, “pick up the bowl on the plate and place it to the left of the bowl drainer”) and an object displacement failure (red, “place the striped book in the top region of the shelf”).

Four-panel visualization of the VLA backbone embedding, LSTM state, failure score head, and failure mode head

Figure 16. Unseen tasks. A success (green, “place the red bowl in the basket”) and a press failure (red, “turn on the flat stove”).

Steering Demonstrations

We use steering only to show how the signal of VLA-TUG could be used; the heuristics are simple and are not evaluated. When the score crosses the calibrated threshold, the policy is paused, an intervention chosen from the predicted failure mode is applied, and the policy resumes.

Simulation. We implemented three heuristics for LIBERO-X. For a wrong-target or placement/insertion failure, the end effector is displaced by a small distance towards the destination. For a grasp failure, it is displaced laterally, perpendicular to its recent motion. For every other predicted mode, the policy replans from the current state. Figure 17a shows the three on the task “place the white-yellow mug on top of the shelf”: after the alarm, the probe predicts the failure mode, the matching intervention is applied, and the policy resumes and succeeds, whereas without intervention the mug is placed on the table and the task fails.

Steering strategies in simulation and on the real robot: (a) simulation, “place the white-yellow mug on top of the shelf.” Initial state; Failure predicted; if grasp failure, End-effector realignment, nudge perp. to recent motion; if wrong target, placement / insertion, End-effector displacement, nudge toward the target; if all other modes, Replanning, discard pending action chunk, re-query GR00T; SUCCESS; 5.2 s, 8.2 s, 11.1 s; 8.9 s, 10.1 s, 11.3 s; 8.8 s, 10.1 s, 11.3 s. (b) real robot, “flip the basket upright.” Initial state; Predicted failure; No steering, FAILURE; With steering, SUCCESS; Steering; Correction; Policy continues; Basket rotates; Upright; Failure threshold 0.9186; Resume; Failure predicted score 0.925; Top 3 predicted failure reasons; 1. Object displacement 71.3%; 2. Grasp failure 18.0%; 3. Placement / insertion failure 3.1%; VLA policy paused. The expert model: move gripper lower by 0.05m; if grasp failure, End-effector realignment, Move the wrist camera 0.05m towards the target grasp object; if object displacement, End-effector displacement, Move the gripper 0.05m towards the usual gripper distribution; if all other modes, Retraction, discard pending action chunk, retract and re-query Gr00t; 0.0 s, 2.2 s, 4.0 s, 7.0 s, 11.0 s, 15.0 s, 18.5 s, 21.5 s, 23.5 s, 28.5 s; Time since policy start (s); Failure score

Figure 17. Steering strategies, in simulation and on the real robot.

Real robot. In the task “put the mushroom on the plate”, the gripper misses the mushroom and hits the table without the probe; the probe predicts a grasp failure, and an expert would steer the wrist camera towards the grasp target and resume the policy. In the task “flip the basket upright”, we start with the basket standing on its tall end, outside the training demonstrations; an expert might move the gripper along the −z-z direction, so that the joint states are in the distribution of the flip motion, as the demonstrations are collected with the basket on its long side (Figure 17b).

Failure mode head layer, by failure mode (failures only)

Figure 14. Failure mode head layer, by failure mode (failures only).

References

  1. [1]Agia, C., Sinha, R., Yang, J., Cao, Z.-a., Antonova, R., Pavone, M., and Bohg, J. Unpacking failure modes of generative policies: Runtime monitoring of consistency and progress. arXiv preprint arXiv:2410.04640, 2024. Angelopoulos, A. N., Bates, S., Candès, E. J., Jordan, M. I., and Lei, L. Learn then Test: Calibrating predictive algorithms to achieve risk control. arXiv preprint arXiv:2110.01052, 2022.
  2. [2]Bhatt, N. P., Yang, Y., Siva, R., Milan, D., Topcu, U., and Wang, Z. Know where you’re uncertain when planning with multimodal foundation models: A formal framework. arXiv preprint arXiv:2411.01639, 2025.arxiv.org/abs/2411.01639
  3. [3]Buurmeijer, H., Alonso, C. A., Swann, A., and Pavone, M. Observing and controlling features in vision-language-action models. arXiv preprint arXiv:2603.05487, 2026.arxiv.org/abs/2603.05487
  4. [4]Chen, J. AEGIS: A backup reflex for physical AI. arXiv preprint arXiv:2606.06660, 2026.arxiv.org/abs/2606.06660
  5. [5]Chen, L., Lyu, Z., and Beksi, W. J. ReconVLA: An uncertainty-guided and failure-aware vision-language-action framework for robotic control. arXiv preprint arXiv:2604.16677, 2026.
  6. [6]Cui, H., Arnaud, S., Majumdar, A., Dugas, D., Aljalbout, E., Desingh, K., Jatavallabhula, K. M., and Meier, F. DREAMSTEER: Latent world models can steer VLA policies during deployment without any finetuning. arXiv preprint arXiv:2607.02865, 2026.arxiv.org/abs/2607.02865
  7. [7]Fei, S., Wang, S., Shi, J., Dai, Z., Cai, J., Qian, P., Ji, L., He, X., Zhang, S., Fei, Z., Fu, J., Gong, J., and Qiu, X. LIBERO-Plus: In-depth robustness analysis of vision-language-action models. arXiv preprint arXiv:2510.13626, 2025.arxiv.org/abs/2510.13626
  8. [8]Francis-Meretzki, S., Mutti, M., Romano, Y., and Tamar, A. Temporal difference calibration in sequential tasks: Application to vision-language-action models. arXiv preprint arXiv:2604.20472, 2026.
  9. [10]Huang, B., Li, X., Wang, X., Mi, L., Hao, Z., Wang, W., Wu, H., Li, K., Liu, Y., and Cao, T. ActProbe: Action-space probe for early failure detection of generative robot policies. arXiv preprint arXiv:2606.08508, 2026.arxiv.org/abs/2606.08508
  10. [11]Jeong, H. J., Swamy, G., and Bajcsy, A. Learning what to say to your VLA: Mostly harmless vision language action model steering. arXiv preprint arXiv:2606.12299, 2026.arxiv.org/abs/2606.12299
  11. [12]Li, Q. VLAs are confined yet capable of generalizing to novel instructions. arXiv preprint arXiv:2505.03500, 2025.arxiv.org/abs/2505.03500
  12. [14]Li, Z., Liu, J., Dong, Z., Teng, T., Rouxel, Q., Caldwell, D., and Chen, F. Towards deploying VLA without fine-tuning: Plug-and-play inference-time VLA policy steering via embodied evolutionary diffusion. IEEE Robotics and Automation Letters, 11(5):6234–6241, 2026b. doi: 10.1109/LRA.2026.3678455. URL https://ieeexplore.ieee.org/document/11457243/.DOI
  13. [15]Liu, B., Zhu, Y., Gao, C., Feng, Y., Liu, Q., Zhu, Y., and Stone, P. LIBERO: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.arxiv.org/abs/2306.03310
  14. [16]Liu, S., Singh, I. S., Xu, Y., Duan, J., and Krishna, R. VLS: Steering pretrained robot policies via vision-language models. arXiv preprint arXiv:2602.03973, 2026a.arxiv.org/abs/2602.03973
  15. [20]Park, S., Li, W., Oh, C., Yeh, S., Kira, Z., Hagenow, M., and Li, S. Hide-and-seek in trajectories: Discovering failure signals for VLA runtime monitoring. arXiv preprint arXiv:2605.30834, 2026.arxiv.org/abs/2605.30834
  16. [21]Physical Intelligence, Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Galliker, M. Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., Li-Bell, A., Mothukuri, M., Nair, S., Pertsch, K., Ren, A. Z., Shi, L. X., Smith, L., Springenberg, J. T., Stachowicz, K., Tanner, J., Vuong, Q., Walke, H., Walling, A., Wang, H., Yu, L., and Zhilinsky, U. π₀.₅: a vision-language-action model with open-world generalization. arXiv preprint arXiv:2504.16054, 2025.
  17. [23]Rajaprakash, H., Prajapati, A., Xue, R., Anwar, A., and Thomason, J. SAFECAST: Robust failure detection for VLA policies with contrast-set training and calibration. arXiv preprint arXiv:2608.04246, 2026.arxiv.org/abs/2608.04246
  18. [24]Swann, A., McGranahan, L., Buurmeijer, H., Kennedy, M., and Schwager, M. Sparse autoencoders reveal interpretable and steerable features in VLA models. arXiv preprint arXiv:2603.19183, 2026.arxiv.org/abs/2603.19183
  19. [25]Tan, D. M. S., Shailesh, S., Iyer, S., Teo, W. W. J., Ju, Y., Gu, Q., and Sartoretti, G. RL²-VLA: Adaptive RL latent compositional steering with test-time scaling for vision-language-action models. arXiv preprint arXiv:2607.26991, 2026.arxiv.org/abs/2607.26991
  20. [26]Turner, A. M., Thiergart, L., Leech, G., Udell, D., Vazquez, J. J., Mini, U., and MacDiarmid, M. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248, 2024.arxiv.org/abs/2308.10248
  21. [27]Wang, G., Zhang, C., Liu, Q., Zhang, J., Cai, J., Liu, J., and Liu, X. LIBERO-X: Robustness litmus for vision-language-action models. arXiv preprint arXiv:2602.06556, 2026.arxiv.org/abs/2602.06556
  22. [30]Yang, Y., Bhatt, N. P., Samineni, P., Siva, R., Wang, Z., and Topcu, U. RepV: Safety-separable latent spaces for scalable neurosymbolic plan verification. arXiv preprint arXiv:2510.26935, 2025a.arxiv.org/abs/2510.26935
  23. [31]Yang, Y., Hu, Y., Ye, M., Zhang, Z., Lu, Z., Xu, Y., Topcu, U., and Snyder, B. Uncertainty-guided enhancement on driving perception system via foundation models. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. 8752–8758, 2025b. doi: 10.1109/ICRA55743.2025.11127651. URL https://ieeexplore.ieee.org/document/11127651/.DOI

Paper details

vlnc:W_j6zc9e2x
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 8

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents