WorksComputer Science
VoxelSBFL: Voxel-Level Spectrum-Based Failure Prediction for Vision-Language-Action Models
Abstract
Vision-Language-Action (VLA) models integrate visual perception, language understanding, and action generation to control robots. Testing these systems produces trajectories and task outcomes, but existing approaches provide limited guidance on using these recorded trajectories to assess subsequent executions or inform policy repair. We conjecture that, for a given policy and task, traversing certain workspace regions is associated with a higher likelihood of failure. To investigate this conjecture, we propose VoxelSBFL, a black-box adaptation of Spectrum-Based Fault Localization (SBFL) in which workspace voxels replace program elements as units of analysis. Recorded tool-center-point (TCP) trajectories provide binary voxel coverage, which is combined with execution outcomes to construct suspiciousness maps using established SBFL formulas. These maps identify failure-associated regions and score unseen trajectories without requiring internal model access. We evaluate 34 spectrum-based techniques across four VLA models using 96,000 Pick-Up trajectories in simulation and compare the leading techniques against eight action- and trajectory-based metrics. VoxelSBFL obtains an overall macro ROC-AUC of 0.704, significantly outperforming state-of-the art baselines. Sensitivity analyses show that performance varies with spatial granularity and training-set size. These findings support spatial spectra for VLA execution assessment and motivate future research on runtime monitoring, self-healing and targeted policy improvement.
ACM Reference Format:
Arusa Kanwal, Pablo Valle, Paola Roggero, Miren Illarramendi, Shaukat Ali, and Aitor Arrieta. 2018. VoxelSBFL: Voxel-Level Spectrum-Based Failure Prediction for Vision-Language-Action Models. J. ACM 37, 4, Article 111 (August 2018), 36 pages. https://doi.org/XXXXXXXX.XXXXXXX
Authors’ Contact Information: Arusa Kanwal, [email protected], Mondragon University, Mondragon, Guipuzcoa, Spain; Pablo Valle, [email protected], Mondragon University, Mondragon, Guipuzcoa, Spain; Paola Roggero, [email protected], Mondragon University, Mondragon, Guipuzcoa, Spain; Miren Illarramendi, [email protected], Mondragon University, Mondragon, Guipuzcoa, Spain; Shaukat Ali, [email protected], Simula Research Laboratory, Oslo, Norway; Aitor Arrieta, aar [email protected], Mondragon University, Mondragon, Guipuzcoa, Spain.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-735X/2018/8-ART111 https://doi.org/XXXXXXX.XXXXXXX
Introduction
Vision-Language-Action (VLA) models [36] integrate visual perception, language understanding, and action generation to control robots that follow users’ instructions. Models such as the GR00T-N1 VLA family [5] and [6] translate visual observations and natural-language instructions into robotic actions. Within the complete robotic stack, these actions are executed by low-level controllers, producing trajectories through the physical workspace. Therefore, assessing the reliability of VLA-enabled robots requires understanding when and under which conditions their executions fail. To support this assessment, recent approaches have proposed different testing methods, including test generation approaches [14, 32] or methods that aim to deal with the test oracle problem [28, 30]. However, once these tests have been executed and their outcomes determined, existing testing approaches offer limited guidance on how to use the recorded trajectories to predict failures in subsequent executions or inform policy repair. Certain approaches aim at predicting failures during VLA executions. For instance, Valle et al. [29] propose both white-box and black-box approaches to distinguish successful and failed executions or high, medium or low quality task executions. SAFE [11] and VLA-Tug [24] rely on internal VLA representations, restricting their use to policies that expose those representations. VLA-Tug also uses annotated failure onsets during training. Nevertheless, these approaches have certain limitations. From the debugging perspective, the scores and alerts produced by these methods give developers limited guidance and explain-ability of under what circumstances the VLA fails. From the practical perspectives, some of such approaches require internal access to the VLA policy (e.g., white-box uncertainty metrics from [29], internal VLA activations at each policy step for obtaining training data [24]), which is not always easy to obtain.
Figure 1 shows the robot’s end-effector trajectories in three dimensions (left) and projected onto the 2D X-Y plane (right), with failed executions shown in red and successful ones in green for the model. The 2D projection makes the spatial pattern particularly clear. Failed trajectories predominate in the upper region outlined in red, while successful trajectories predominate in the right-hand region outlined in green. We therefore conjecture that, in VLA-enabled robots, traversing certain regions of the workspace is associated with a higher likelihood of task failure. This conjecture draws on an analogy with conventional software. A test that executes a faulty statement may be more likely to fail because certain inputs or program states can trigger the fault. Similarly, a robot execution may be more likely to fail when the end effector passes through a problematic region of the workspace. This analogy motivates treating the regions visited by a robot as a form of coverage, much like the statements executed by a software test.

Figure 1. Spatial Distribution of Successful and Failed Robot Trajectories
Building on this intuition, we propose VoxelSBFL, which recasts failure prediction for VLA-enabled robots as a spatial adaptation of spectrum-based fault localization (SBFL) [13]. We partition the robot’s three-dimensional workspace into voxels and use these voxels in place of program elements. Just as a software test covers a statement by executing it, a robot trajectory covers a voxel if at least one recorded position of the end effector’s tool center point falls within it. We combine this voxel coverage information with the success or failure outcome of each execution and apply standard SBFL formulas to quantify how strongly coverage of each voxel is associated with failure. The resulting suspiciousness map highlights workspace regions that warrant closer inspection. Developers can use it to identify segments of failed trajectories to examine, including the actions and observations that preceded entry into a highly suspicious voxel. The same map could guide the generation of new trajectories that exercise these regions and the collection of successful demonstrations for targeted repair (e.g., through policy fine-tuning). In this study, we evaluate how well the map distinguishes successful from failed executions, as a proof to determine whether spectrum-based techniques can be employed in this context.
We evaluate how well VoxelSBFL distinguishes successful from failed executions using complete, held-out trajectories. We compare 34 SBFL formulas on 96,000 simulated executions across four state-of-the-art VLA models and benchmark the leading formulas against eight other black-box metrics proposed by Valle et al. [29], including action- and trajectory-based metrics. Harmonic Mean and Rogot2 achieve the highest macro-averaged ROC-AUC among the SBFL formulas (0.704), outperforming the strongest motion-instability baseline (0.655). We also find that voxel size affects the relative ranking of the formulas and that Harmonic Mean improves as more labeled trajectories are used to construct the map. These results not only suggest that VoxelSBFL can predict failing executions, but that certain VLA failures may be associated with certain spatial regions and that certain spectrum-based techniques can help in identifying suspicious voxels.
The key contributions of this paper can be summarized as follows:
We introduce VoxelSBFL, a black-box adaptation of spectrum-based fault localization that uses end-effector visits to workspace voxels as coverage and combines them with episode outcomes.
Through a large-scale evaluation across four VLA models, we show that voxel coverage can distinguish successful from failed trajectories and that the leading SBFL formulas outperform generic action- and motion-instability metrics overall.
We characterize how voxel size and training-set size affect performance and document the drop in discrimination when transferring from Pick-Up to Move-Near.
We show how voxel suspiciousness maps can help developers prioritize trajectory segments for inspection and discuss their potential for targeted test generation, policy repair, and runtime monitoring.
We provide a replication package [15] to support replication of the experiments and extension of the evaluation to other VLA models and robotic tasks by other researchers.
The rest of the paper is structured as follows. § 2 provides background on Vision-Language-Action (VLA) models and Spectrum-Based Fault Localization (SBFL). § 3 presents the proposed VoxelSBFL approach, including workspace voxelization, trajectory coverage, and voxel suspiciousness computation. § 4 describes the empirical evaluation setup and research questions. § 5 presents and discusses the experimental results, including comparisons of SBFL techniques, voxel-size and coverage sensitivity, and cross-task generalizability. § 6 discusses threats to validity, while § 7 reviews related work. Finally, § 8 concludes the paper and outlines directions for future research.
Background
Vision-Language-Action Models
Vision-Language-Action models (VLAs) are multi-modal models that integrate visual perception, language understanding, and action generation to control robots [5, 6, 17, 25, 26]. Robotic systems that use these models for decision-making constitute an emerging class of Cyber-Physical Systems (CPSs), in which learned policies interconnect sensor inputs and user instructions (in the form of natural language) with physical actions. While conventional robotic architectures typically organize perception, planning, and control into separate components, VLA models learn a direct mapping from multi-modal observations into actions. Their architectures combine visual and language representations along with proprioceptive information to generate the next action(s) to be taken. This enables a robot to interpret a natural-language-based instruction such as “Pick up the apple” in the context of its environment, identify the intended object, and generate the movements required to interact with it and follow the instruction given by the human.
Figure 2 shows the interaction between the sensors, VLA-based decision-making, and low-level control in a VLA-enabled robot. Sensors capture the visual observations and the robot’s proprioceptive state, which are provided to the VLA model along with the user instruction. Within the model, modality-specific encoders and multi-modal processing components produce contextual representations used by an action decoder to generate the control commands. These commands are subsequently passed to the robot’s low-level control and actuation components, which translate them into platform-specific actuator commands employing traditional well-known controllers (e.g., Proportional-Integral-Derivative (PID) controllers).

Figure 2. Overview of the Architecture of a VLA-Enabled Robot
Formally, at a decision step , the model receives an observation, , where denotes an RGB image captured by camera , is the natural-language instruction, and represents the robot’s proprioceptive state, such as joint positions and velocities, or gripper forces. For models that use action chunking [6], the policy models a conditional distribution over action chunks, , where is the prediction horizon and each represents a control command with dimensions. The action representation depends on the robot and policy as it may specify joint targets, end-effector poses or displacements, and gripper commands. The prediction horizon also varies across models and configurations (e.g., employed chunks of size in their original paper [6]).
By iteratively generating action chunks, the VLA model produces control commands that, when executed by the robot, induce a trajectory. Let denote the action chunk generated at decision step , and let denote the sequence of commands actually executed before the next chunk is generated. Depending on the execution strategy, may contain all actions in or only a subset, for example when a new prediction replaces the current chunk before its execution is complete since the execution horizon is shorter than the generation horizon. The resulting sequence of executed commands is , where is the number of applied chunks. Executing these commands produces a trajectory , where is the total number of executed actions. For manipulation tasks, each state can be characterized by the configuration of the robot’s end-effector, i.e., Tool Center Point (TCP), including its position and orientation, represented by a unit quaternion . The robot’s low-level control system translates the generated actions into joint movements, allowing the VLA model to connect natural-language instructions and visual observations with physically grounded behavior.
VLA models differ in how they generate actions. While some architectures predict discretized action tokens autoregressively [17], models such as [6] and the GR00T models (i.e., Gr00t-N1.5, Gr00t-N1.6, and Gr00t-N1.7) [5] use conditional flow matching to generate continuous action chunks. At inference time, these models initialize an action chunk with noise and iteratively transform it into an action sample conditioned on visual, linguistic, and proprioceptive information. In GR00T models, this process is implemented using a diffusion transformer that combines state and action representations with features from a vision-language backbone. These iterative updates are internal steps of action generation and are distinct from the physical control steps at which the robot executes actions. VLA models are commonly trained on robot demonstration datasets that associate visual observations and natural-language instructions with the actions performed by a robot [5, 6, 17]. From these demonstrations, the model learns to relate instructions to the observed environment and generate actions that accomplish the requested task. Large-scale datasets, such as Open X-Embodiment [21], provide demonstrations spanning multiple tasks and robot embodiments, supporting the learning of transferable visuomotor behaviors. Subsequent fine-tuning and task- or robot-specific demonstrations adapt the policy to the target deployment setting, including its observation and action interfaces [17]. This adaptation can improve task performance, but does not by itself guarantee that generated actions satisfy the robot’s physical constraints or the system’s safety requirements.
Spectrum-based Fault Localization
Spectrum-based fault localization (SBFL) assists debugging by ranking program components according to their association with observed test failures [1, 13, 33]. It combines execution information, represented as program spectra, with test outcomes. For example, a binary block-hit spectrum records whether each code block is executed at least once by a test. Using these spectra and the corresponding pass or fail outcomes, SBFL assigns each component a suspiciousness score and ranks components in descending order of suspiciousness to guide inspection. Different SBFL techniques use different formulas to calculate suspiciousness [20, 33]. A representative technique is Tarantula [13]. For a covered block , assuming at least one failing and one passing test, its score is:
where and denote the numbers of failing and passing tests that execute , respectively, and and denote the total numbers of failing and passing tests. Tarantula therefore assigns higher scores to blocks executed by a larger proportion of failing tests and a smaller proportion of passing tests.
Table 1 illustrates this using a buggy implementation of a fee-calculation program and five tests, of which two fail and three pass (, ). The bug lies in block , where the multiplier is 0.20 instead of the intended 0.10. Both failing test cases execute , whereas no passing test case executes it. Consequently, Tarantula assigns this block the maximum score of 1.000, placing it first in the inspection ranking. In this example, a block’s rank is one plus the number of blocks with a strictly higher score; blocks with identical scores therefore share the same rank.
Table 1. SBFL example using Tarantula. A bullet (•) denotes execution and a dash (–) denotes non-execution; F and P denote failing and passing tests, respectively. The faulty block and its score are highlighted in bold.
| ID | Program block | Suspiciousness | Rank | |||||||||
| fee = 0 | ||||||||||||
| if priority: | • | • | • | • | • | 2 | 3 | 2 | 3 | 0.500 | 3 | |
| fee += 5 | • | – | • | • | – | 1 | 2 | 2 | 3 | 0.429 | 6 | |
| if amount_based: | • | • | • | • | • | 2 | 3 | 2 | 3 | 0.500 | 3 | |
| if amount > 200: | • | • | • | – | – | 2 | 1 | 2 | 3 | 0.750 | 2 | |
| fee += 0.20 * amount | • | • | – | – | – | 2 | 0 | 2 | 3 | 1.000 | 1 | |
| else: fee += 0.10 * amount | – | – | • | – | – | 0 | 1 | 2 | 3 | 0.000 | 7 | |
| else: fee += 10 | – | – | – | • | • | 0 | 2 | 2 | 3 | 0.000 | 7 | |
| return fee | • | • | • | • | • | 2 | 3 | 2 | 3 | 0.500 | 3 | |
| Test outcome | F | F | P | P | P |
Table 1.
Approach
We propose a spatial adaptation of Spectrum-Based Fault Localization (SBFL) for VLAs. Unlike in traditional SBFL [33], which uses program elements, such as statements or methods, our approach uses spatial regions of the robot workspace. Robot trajectories then play the role of test executions: a trajectory covers a spatial region when the robot visits it, and its task outcome determines whether the execution passes or fails. The approach requires only recorded robot positions and episode-level outcomes. A key advantage of our approach is that it does not require access to the internal parameters, gradients, activations, or architecture of the VLA model. Therefore, unlike other approaches, such as those aiming to measure uncertainty [8, 16, 29, 38], our method can be applied as a non-invasive, model-agnostic analysis procedure to determine at run-time when a failure is about to happen. The approach consists of four main stages: (1) voxelizing the robot workspace; (2) transforming labeled trajectories into binary voxel-coverage spectra; (3) computing voxel suspiciousness; and (4) applying the resulting suspiciousness map to unseen trajectories. Figure 3 depicts an intuitive representation of VoxelSBFL and an overview of the approach.

Figure 3. Overview of VoxelSBFL. Successful and failed tool-center-point trajectories are mapped onto a voxel grid. SBFL ranks visited voxels by their association with failed executions. The resulting suspiciousness map is used to score unseen trajectories and predict whether they are going to fail.
Workspace Voxelization
Let the robot workspace of a VLA-enabled robot be a three-dimensional region . We partition this workspace using a regular voxel grid defined by an origin , and a voxel size . Given a Cartesian robot position , its voxel index is obtained as
where subtraction, division, and the floor operation are applied element-wise. A voxel with index represents the half-open spatial region, formalized as follows:
This definition ensures that every position is assigned to exactly one voxel, including positions lying on grid boundaries. Although the grid is conceptually unbounded, the implementation stores only voxels visited by at least one training trajectory. The voxel size controls the spatial granularity of the analysis. Smaller voxels provide finer localization but produce a larger and sparser spectrum. Larger voxels combine nearby positions, increasing support but reducing spatial resolution.2
Trajectory and Outcome Representation
A robot episode produces a time-ordered trajectory, i.e., , where is the recorded Cartesian position at time step , and is the number of recorded steps. Each episode is accompanied by an externally provided binary outcome , where refers to episode failure, while refers to episode pass. The outcome is not inferred from the trajectory and is not produced by the SBFL approach. It is determined by the success criterion of the underlying manipulation task.
The voxel coverage set of trajectory is . Coverage is binary at the trajectory level. We define , where is the indicator function. Therefore, a trajectory contributes at most one coverage observation to each voxel, regardless of how many times it visits that voxel. This representation follows the semantics of conventional SBFL, in which a program element is either executed or not executed by a test case. It also prevents long trajectories or trajectories that remain in one spatial region for many time steps from dominating the spectrum. The current formulation consequently captures spatial visitation but does not use aspects such as visit frequency, time spent in a voxel, or temporal ordering.
Construction of the Voxel Spectrum
Let denote a collection of labeled training trajectories. The set of training-observed voxels is . For every voxel , we construct a four-element coverage spectrum:
where refers to failed trajectories that visit voxel ; refers to successful trajectories that visit voxel ; refers to failed trajectories that do not visit voxel ; and : successful trajectories that do not visit voxel . Every training trajectory contributes exactly once to one of these four categories for each voxel. Therefore, and , where and are the total numbers of failed and successful training trajectories, respectively.
A voxel with a large and a small is visited more frequently by failed trajectories than by successful ones and may consequently be associated with failure. However, the precise relationship between these four counts is determined by the selected SBFL technique. Voxels that are never observed during training are not assigned a spectrum or suspiciousness value. Assigning them a score based on would incorrectly imply that the training data contain evidence about their behavior.
Voxel visitation also provides a basis for assessing the spatial adequacy of a test suite, which we call the Workspace Voxel Coverage (WVC). Let be a suite of test trajectories, let denote the set of voxels visited by trajectory , and let be a fixed, nonempty set of reference voxels representing the workspace where the root operates. We define the WVC of as the proportion of reference voxels visited by at least one trajectory in the suite:
A test suite satisfies the workspace voxel coverage criterion at a prespecified threshold if . For example, a threshold of 0.8 requires the suite to visit at least 80% of the reference voxels. Comparisons between test suites shall use the same coordinate frame, voxel grid, and reference set.
Voxel Suspiciousness
Let denote the suspiciousness function associated with the SBFL technique (e.g., Trantula, Ochiai). The raw suspiciousness of voxel is . Our approach can use any SBFL techniques from the literature. These techniques combine the spectrum counts differently and may react differently to class imbalance, sparse voxel support, and increasing workspace coverage. Some SBFL techniques’ formulas may be undefined for particular spectra, e.g., when their denominator is zero. We handle such cases by using a predefined finite policy. An undefined 0/0 expression receives the technique’s neutral value, whereas mathematical positive or negative infinities are replaced by finite rank-preserving extreme values. These cases are recorded as diagnostics rather than silently removing the corresponding voxels.
Because the selected techniques produce scores with different numerical ranges, their raw suspiciousness values are transformed into empirical percentile ranks. Let be the number of training-observed voxels. For a voxel , let be the number of training voxels with a strictly smaller score than , and let be the number with the same score. Its tie-aware percentile is
when . If only one voxel is observed during training, it receives the neutral percentile 0.5.
The percentile expresses the relative position of the voxel within the suspiciousness map. A value close to one indicates that the voxel is ranked among the most failure-associated spatial regions according to technique . Voxels with identical raw suspiciousness receive the same average percentile. The output of the training stage is therefore a technique-specific suspiciousness map, i.e., . Sorting in descending order of produces a ranked list of failure-associated spatial regions. Because every voxel index can be transformed back into its physical bounds or center position, this ranking can be visualized in the robot workspace and used to identify candidate regions for further inspection or intervention.
Application to Unseen Trajectories
The suspiciousness map provides the voxel-level localization output. To evaluate whether this map captures spatial behavior associated with failures, we additionally use it to score unseen trajectories. For an unseen trajectory , the set of visited voxels for which the training process produced a suspiciousness value is . The trajectory-level risk assigned by technique is
The maximum operator reflects the hypothesis that visiting one highly suspicious spatial region may be sufficient to place an execution at risk. It also preserves a direct connection between trajectory-level predictions and individual suspicious voxels. Unknown voxels are ignored when at least one known voxel exists. When a trajectory contains no training-observed voxel, the available data provide no evidence for classifying it as either successful or failed. We therefore assign it the neutral score 0.5. The test trajectory outcome is used only after has been computed. It is never used to modify the voxel map or select its suspicious regions. This separation allows the failure-discrimination ability of the learned spatial map to be evaluated on previously unseen executions without label leakage.
Empirical Evaluation
Our empirical evaluation explores the following five Research Questions (RQs):
RQ1 – Overall SBFL Technique Performance: Which spectrum-based technique is the most effective at identifying failure-associated voxels in VLA models? With this RQ, we aim to determine which SBFL techniques produce suspiciousness maps that best distinguish failed trajectories from successful ones. To this end, we compare the scores derived from each technique under the same experimental conditions.
RQ2 – Comparison baseline: How do the best spectrum-based techniques compare with respect to state-of-the-art VLA black-box uncertainty detection methods? In this RQ, we aim to assess whether the best-performing SBFL techniques can discriminate failures as effectively
as existing black-box metrics from a prior study [29]. To this end, we compare their failure-discrimination performance on the same evaluation data.
RQ3 – Voxel-Size Sensitivity: How does voxel size affect localization effectiveness, spatial coverage, computational cost, and the relative performance of spectrum-based techniques? With this RQ, we aim to understand the sensitivity that voxel size has on the effectiveness of SBFL techniques.
RQ4 – Voxel-Coverage Sensitivity: How does the workspace voxel coverage achieved by the training trajectories affect the effectiveness and robustness of different SBFL techniques? In this RQ, we investigate whether the techniques remain effective when the available training trajectories cover only part of the workspace. To this end, we vary the number of training trajectories and examine the resulting voxel coverage and failure-discrimination performance.
RQ5 – Cross-Task Generalizability: To what extent do suspiciousness maps learned from one robotic task retain their failure-discrimination effectiveness when applied to a different task? With this RQ, we aim to determine whether failure-associated spatial patterns learned from Pick-Up trajectories remain useful for identifying failures in a different task. To this end, we evaluate the learned maps from Pick-Up on Move-Near trajectories without retraining or target-task calibration.
We evaluate the proposed approach through a sequence of controlled experiments designed to isolate the effects of the SBFL technique, voxel granularity, training coverage, and task distribution. RQ1 uses deterministic leave-one-run-out validation within each combination of VLA model, trajectory-generation condition, and scene budget. Nine complete runs are used for training and the remaining run is used for testing, until every run has served as the test set. RQ2 retains the same evaluation partitions for the baseline comparison, while RQ3 uses these partitions to examine the effect of voxel size. RQ4 introduces a leakage-safe three-way partition consisting of training-pool, calibration, and test runs, allowing training subsets with different spatial coverage to be selected without consulting the test data.
All comparisons between techniques are paired: techniques are evaluated using the same trajectories, voxel grids, and experimental partitions. Results are first averaged over repeated runs and subsequently macro-averaged over the experimental configurations, preventing configurations containing more trajectories from dominating the results.
Selected SBFL Techniques
We evaluated 34 spectrum-based techniques reported by Wong et al. [33]. Their corresponding formulas are described in the survey, while the complete set of techniques evaluated in our study is reported in Table 3. Each technique received the same voxel-level spectrum and is evaluated on the same held-out trajectories. No technique-specific parameter tuning or trajectory-aggregation rule was used. Undefined expressions, such as 0/0, were assigned a predefined neutral value. Mathematical infinities were replaced by finite, rank-preserving values. These cases were recorded separately as diagnostics, ensuring that every technique produces finite scores without silently discarding voxels or trajectories. The complete catalogue was retained in RQ1–RQ4. Consequently, changes in technique ranking can be attributed to the experimental factor under study rather than to differences in the set of techniques being compared.
| Technique | EO1 | GR00T | Pi0 | SpatialVLA | Overall |
| Harmonic Mean | 0.665 (1) | 0.673 (1) | 0.746 (1) | 0.732 (1) | 0.704 (1) |
| Rogot2 | 0.665 (1) | 0.673 (1) | 0.746 (1) | 0.732 (1) | 0.704 (1) |
| Dennis | 0.646 (5) | 0.660 (3) | 0.738 (3) | 0.708 (3) | 0.688 (3) |
| Phi (Geometric Mean) | 0.638 (6) | 0.653 (4) | 0.737 (4) | 0.699 (4) | 0.682 (4) |
| Tarantula | 0.659 (3) | 0.645 (6) | 0.671 (9) | 0.644 (8) | 0.655 (5) |
| Tarwid | 0.659 (4) | 0.645 (5) | 0.671 (10) | 0.644 (7) | 0.655 (6) |
| Arithmetic Mean | 0.628 (7) | 0.622 (7) | 0.711 (5) | 0.651 (6) | 0.653 (7) |
| Michael | 0.625 (8) | 0.618 (8) | 0.707 (6) | 0.631 (9) | 0.645 (8) |
| Cohen | 0.619 (9) | 0.618 (9) | 0.707 (7) | 0.628 (10) | 0.643 (9) |
| Zoltar | 0.579 (15) | 0.599 (10) | 0.703 (8) | 0.690 (5) | 0.643 (10) |
| Ochiai2 | 0.605 (10) | 0.563 (12) | 0.667 (11) | 0.521 (12) | 0.589 (11) |
| Mountford | 0.556 (21) | 0.574 (11) | 0.648 (12) | 0.566 (11) | 0.586 (12) |
| Rogot1 | 0.597 (11) | 0.521 (21) | 0.610 (13) | 0.495 (18) | 0.556 (13) |
| Scott | 0.597 (11) | 0.521 (21) | 0.610 (13) | 0.495 (18) | 0.556 (13) |
| Fleiss | 0.595 (13) | 0.521 (23) | 0.593 (15) | 0.495 (20) | 0.551 (15) |
| Ample | 0.581 (14) | 0.522 (19) | 0.533 (18) | 0.481 (32) | 0.529 (16) |
| Hamann | 0.557 (16) | 0.529 (13) | 0.474 (19) | 0.496 (13) | 0.514 (17) |
| Hamming | 0.557 (16) | 0.529 (13) | 0.474 (19) | 0.496 (13) | 0.514 (17) |
| Rogers & Tanimoto | 0.557 (16) | 0.529 (13) | 0.474 (19) | 0.496 (13) | 0.514 (17) |
| Simple Matching | 0.557 (16) | 0.529 (13) | 0.474 (19) | 0.496 (13) | 0.514 (17) |
| Sokal | 0.557 (16) | 0.529 (13) | 0.474 (19) | 0.496 (13) | 0.514 (17) |
| Pearson | 0.537 (22) | 0.521 (20) | 0.568 (16) | 0.399 (33) | 0.506 (22) |
| Baroni-Urbani & Buser | 0.525 (23) | 0.486 (24) | 0.414 (26) | 0.494 (23) | 0.480 (23) |
| Gower | 0.455 (34) | 0.526 (18) | 0.548 (17) | 0.384 (34) | 0.478 (24) |
| Braun-Banquet | 0.517 (24) | 0.472 (27) | 0.414 (33) | 0.494 (30) | 0.474 (25) |
| Ochiai | 0.504 (25) | 0.472 (25) | 0.414 (24) | 0.495 (21) | 0.471 (26) |
| Fossum | 0.504 (25) | 0.472 (26) | 0.414 (24) | 0.495 (21) | 0.471 (27) |
| Anderberg | 0.504 (27) | 0.472 (28) | 0.414 (27) | 0.494 (24) | 0.471 (28) |
| Dice | 0.504 (27) | 0.472 (28) | 0.414 (27) | 0.494 (24) | 0.471 (28) |
| Goodman | 0.504 (27) | 0.472 (28) | 0.414 (27) | 0.494 (24) | 0.471 (28) |
| Jaccard | 0.504 (27) | 0.472 (28) | 0.414 (27) | 0.494 (24) | 0.471 (28) |
| Kulczynski | 0.504 (27) | 0.472 (28) | 0.414 (27) | 0.494 (24) | 0.471 (28) |
| Sorensen-Dice | 0.504 (27) | 0.472 (28) | 0.414 (27) | 0.494 (24) | 0.471 (28) |
| Pierce | 0.504 (33) | 0.470 (34) | 0.413 (34) | 0.494 (31) | 0.470 (34) |
Table 3. RQ1 – Macro ROC-AUC of the 34 SBFL techniques for each VLA model. Parentheses report tie-aware ranks. Bold denotes rank 1 and underlining denotes the remaining top-five ranks. Harmonic Mean and Rogot2 are tied because they produce identical results in every reported column.
Selected VLA Models
The evaluation includes four VLA models: E01 [25], GR00T-N1.6 [5], Pi0 [6], and SpatialVLA [26]. These models were selected to cover different architectural designs and action generation mechanisms. Specifically, E01 integrates multimodal reasoning and action generation within a shared transformer backbone, whereas Pi0 and GR00T-N1.6 combine vision-language backbones with dedicated action generation components based on flow matching. In contrast, SpatialVLA incorporates explicit spatial representations and adaptive discretized action grids. Furthermore, these models have been used across multiple empirical studies on robotic manipulation, robustness assessment, and automated testing [9, 14, 23, 29, 30, 31, 32], supporting their relevance as subjects for our evaluation. Moreover, all of these models are compatible with the open-source simulation environment SimplerEnv [18], allowing our evaluation to be easily replicated by other researchers.
We conducted our experiments using the environment adopted from state-of-the-art works [14, 29, 30, 32], employing the fine-tuned versions of each model adapted to the fractal benchmark dataset [7] corresponding to the task categories used in our evaluation suite. By considering multiple VLA policies, we aim to investigate whether the behavior and relative effectiveness of the SBFL techniques remain consistent across models with different architectures and action representations.
Simulator, Robot Embodiment and Robotic Tasks
Following prior work [14, 29, 30, 32], we conducted our experiments using SimplerEnv [18], an open-source benchmark for evaluating robot manipulation policies in simulation. Built on SAPIEN and ManiSkill2, SimplerEnv reproduces real-world robotic manipulation setups, providing a controlled environment in which experiments can be repeated under consistent conditions. This facilitates systematic execution of test cases and the comparison of the SBFL techniques across the evaluated VLA models. Figure 4 shows example visual observations from the simulated manipulation environments. Within this benchmark, we used the Google Robot embodiment3, also known as the Everyday Robot, together with VLA checkpoints trained on the Fractal dataset [7]. At each decision step, the VLA policy processes visual observations, the natural-language task instruction, and the robot’s proprioceptive state to generate actions. The simulated robot executes these actions, producing changes in both its position and the surrounding environment that are reflected in subsequent observations. This interaction continues until the task is completed or execution reaches its predefined limit, i.e., 80 steps for the “Pick-up” task and 120 steps for the “Move-Near” task. Using the same embodiment across experiments provides a common robotic platform for assessing our approach, while allowing different VLA policies to control the robot.

Figure 4. Example visual observations from SimplerEnv: (a) a can on a tabletop for a “Pick-up” task; and (b) a sponge, a can, and a bottle on a tabletop for a “Move Near” task.
We organized the task selection around the objectives of the RQs. For RQ1–RQ4, we used the “Pick-up” task for both training and testing the SBFL techniques. This task requires the robot to grasp the target object and lift it by at least 0.02 m for five consecutive simulation frames. These experiments assess the techniques within a common task setting. Building on this evaluation, RQ5 investigates whether their effectiveness transfers to a different task. Both tasks used the same robotic embodiment, consistent with the Fractal dataset on which the VLA checkpoints were trained. This setup allows us to assess the cross-task generalizability of our approach while keeping the embodiment unchanged.
Selected Baselines
We compared VoxelSBFL against eight trajectory-level uncertainty and quality metrics proposed by Valle et al. [29]. We only used black-box techniques, as VoxelSBFL is also black-box. On the other hand, we opted not to include white-box uncertainty techniques from [29] because (1) they included some overhead, which prevented us from executing a large set of trajectories and performing a comprehensive evaluation, and (2) each VLA would require the adaptation of such metrics, which is non-trivial. The uncertainty baselines comprised three action-level metrics: Action Position Instability, Action Velocity Instability, and Action Acceleration Instability. The quality baselines comprised TCP Position, Velocity, and Acceleration Instability, Trajectory Instability, and Optimal Trajectory. Because Optimal Trajectory requires information about the object and goal, we considered it a task-aware baseline. Valle et al. [29] also proposed four token-level uncertainty metrics: Token Probability, Prediction Confidence Score (PCS), DeepGini, and Entropy. As mentioned above, we excluded these metrics because they require access to token-level prediction probabilities, which may depend on model-specific internal components and may not be available for every action-generation mechanism. Instead, we treat the VLA as a black box and consider information available at the interface between the policy and the robotic system, such as generated actions and executed trajectories. Therefore, for the rest of the paper, we call these black-box baseline metrics.
For each baseline, per-step and per-dimension values were reduced to a single trajectory-level score using their arithmetic mean. Score directions were fixed before evaluation, with larger instability, deviation, and distance values treated as stronger evidence of failure. A baseline was evaluated for a run only when its metric was available for every trajectory, thereby avoiding outcome-dependent episode removal. The primary cross-VLA comparison included only metrics available across every evaluated experimental cell.
Generation of traces and spectra
To evaluate VoxelSBFL, we sought successful and failed executions that varied in which parts of the workspace they visited. We adapted DeepHyperion [37] because its MAP-Elites search lets us pursue these two goals together. The search explores different initial object positions, which can elicit different reaching and grasping trajectories, and uses a failure-aware fitness function to prioritize scenes that expose failures within each region of the search grid. For each evaluated scene, we record the robot’s TCP trajectory and the binary task outcome (i.e., pass or fail). Each scene is represented as , where denotes the manipulated object, its initial position, and its orientation. We consider seven objects and five predefined orientations, with and meters. MAP-Elites partitions the initial object positions into a 25 × 25 grid. Starting from 20 random scenes, the search mutates object positions and orientations. The fitness function prioritizes failures while also accounting for proximity to the object and spatial coverage.
We used four search configurations to examine whether the comparison of SBFL techniques depended on how the traces were generated. The baseline (DH) uses the MAP-Elites archive to guide exploration. The other configurations share the same scene representation and fitness function but guide candidate selection using a decision tree (DH_HybridDT), a random forest (DH_HybridRF), or an adaptive weighting of random-forest failure predictions, diversity, and novelty (DH_WeightedRF). The predictors are trained on previously evaluated scenes and their observed outcomes. Within each experimental configuration, all SBFL techniques are evaluated on the same trajectories. For RQ1, we also recompute the aggregate ranking after excluding each generator in turn.
Evaluation Metrics
The primary evaluation metric was the area under the receiver operating characteristic curve (ROC-AUC), considering failed trajectories as the positive class. ROC-AUC measures whether the trajectory scores derived from the voxel suspiciousness map rank failed executions above successful ones. It is threshold-independent and is therefore suitable for comparing techniques, VLA models, and voxel sizes for which selecting a universal suspiciousness threshold would be arbitrary and potentially unfair. Conventional SBFL studies frequently use the EXAM score, defined as the percentage of ranked program elements that must be inspected before reaching the first known faulty element [33]. EXAM assumes the existence of a ground-truth faulty element. This assumption does not hold in our dataset: a voxel is a spatial region rather than an intrinsically faulty executable element, failures may result from interactions across multiple regions and time steps, and no causal failure voxel is annotated. Consequently, the rank of the “first faulty voxel” is undefined. Our ROC-AUC results therefore evaluate failure-associated trajectory discrimination, instead of causal voxel localization.
For RQ3, we also examine changes in technique rankings across voxel sizes, pairwise rank correlations, and the computational and sparsity costs associated with each resolution. For RQ4, we report the achieved calibration and test coverage and the matched difference in ROC-AUC between high- and low-coverage training subsets.
Statistical tests
Statistical comparisons were based on matched experimental observations. ROC-AUC values were first averaged across the ten leave-one-run-out folds within each configuration and then equally averaged across the two scene budgets. This yielded 16 matched blocks, corresponding to the combinations of four VLA models and four trajectory-generation strategies, which were used for the primary paired comparisons. When more than two conditions were compared, we used the Friedman test as an omnibus test, followed, where appropriate, by paired two-sided exact Wilcoxon signed-rank tests. We applied Holm correction to account for multiple comparisons within separately defined families of hypotheses. For RQ1, the correction covered all 561 pairwise comparisons among the 34 SBFL techniques. The remaining RQs used separate correction families corresponding to their respective statistical comparisons. Alongside the Holm-adjusted -values, we reported paired rank-biserial correlations () as effect sizes. For RQ1, RQ2 and RQ3, we estimated pointwise 95% confidence intervals using a hierarchical percentile bootstrap with 10,000 repetitions, resampling experimental blocks and validation folds while preserving paired comparisons. We considered absolute ROC-AUC differences below 0.01 practically negligible when they were not statistically significant after Holm correction. This criterion was used to describe practical similarity rather than establish formal statistical equivalence.
Dataset Construction
The dataset contains episode metadata and time-ordered robot trajectories. Each episode includes a globally unique identifier, the VLA model, trajectory-generation condition, scene budget, run identifier, manipulation target, number of recorded steps, and binary pass/fail outcome. Each trajectory consists of a sequence of tool-center-point positions represented in the robot-base coordinate frame. A fixed grid origin of was used. Unless voxel size was the factor under study, we used the reference size coordinate units.
Dataset for RQ1 and RQ3
The dataset used for RQ1 and RQ3 contains 96,000 trajectories and 7,680,000 recorded Cartesian positions. It comprises four VLA models, four trajectory-generation conditions, two scene budgets (200 and 400 scenes), and ten independent runs for every model–condition–budget combination. This results in 32 configurations and 320 leave-one-run-out folds.
RQ1 used the reference voxel size of 0.02. For every configuration, the suspiciousness map is trained using nine complete runs and evaluated on the remaining run. RQ3 reused the same trajectories, outcomes, and held-out-run identities at voxel sizes 0.01, 0.02, and 0.05. The resulting matched design contains 960 voxel-size–fold cells and 32,640 technique–size–fold measurements.
Dataset for RQ4
RQ4 used the same trajectory collection but varied the number of trajectories used to construct the spectra. For each outer fold, one complete run was reserved as the test set, the next run in cyclic order was used as a calibration set, and the remaining eight runs formed the candidate training pool. The test run was never used for training or subset construction. The calibration set was used only to quantify the voxel coverage achieved by each training subset; it did not influence subset selection.
We considered training-set sizes of 10, 20, 50, 100, and 200 trajectories. For every configuration and outer fold, we generated ten repeated sequences of nested subsets. Within each repetition, successful and failed trajectories were randomly ordered separately, and a subset of size contained the first trajectories from each class. Consequently, every subset contained exactly 50% failed and 50% successful trajectories, and each smaller subset was contained in the subsequent larger subset. Calibration coverage was measured as
All 34 SBFL techniques were fitted to each subset and evaluated on the untouched test run. The resulting design comprised 32 VLA–trajectory-generation–budget configurations, ten outer folds, ten sampling repetitions, and five training-set sizes. This produced 16,000 fitted experimental cells and 544,000 technique-level measurements. This design treated training-set size as the controlled experimental factor and voxel coverage as an observed consequence of increasing the number of available trajectories.
Dataset for RQ5
RQ5 evaluated the transfer of suspiciousness maps from the Pick-Up task to the Move-Near task. We used the maps reconstructed from the available Pick-Up training data, without retraining or calibrating them on Move-Near. For each VLA model, all 34 SBFL techniques were evaluated on the same selected Move-Near executions. No Move-Near execution was used to construct the suspiciousness maps.
The Move-Near working space comprised 72 voxels. Although our initial target was approximately 500 task instances per VLA model, we generated and executed 504 instances to ensure equal coverage of the working space: seven scenarios per voxel. This design provided balanced spatial coverage and increased the diversity of the evaluated scenarios. Task instances were generated using fixed random seeds, and, the same scene specifications were evaluated across VLA models. For each execution, we recorded the TCP trajectory and binary pass/fail outcome.
We then selected evaluation subsets from these executions using the proportions of successful and failed Pick-Up executions in RQ1 as a reference. For each model, these proportions were calculated by pooling the execution records from the 200- and 400-execution budgets.
Let denote the smaller of the pass and fail proportions for model . We calculated the reference sample size corresponding to 100 expected observations of the minority class:
For each VLA model, we then selected exactly 100 successful executions and the number of failed executions required to approximate the corresponding RQ1 failure proportion. Selection used a fixed random seed, with spatial constraints to maintain coverage across the 72 cells. The resulting evaluation subsets contained 298 executions for Pi0, 246 for GR00T, 223 for EO1, and 342 for SpatialVLA. Table 2 reports the reference proportions, calculated sample sizes, and selected evaluation subsets.
| VLA | RQ1 pass (%) | Selected pass | Selected fail | Selected | |
| 33.592 | 298 | 100 | 198 | 298 | |
| GR00T | 40.654 | 246 | 100 | 146 | 246 |
| EO1 | 44.875 | 223 | 100 | 123 | 223 |
| SpatialVLA | 29.246 | 342 | 100 | 242 | 342 |
Table 2. Pooled RQ1 success proportions and selected target-task evaluation samples. denotes the reference budget required for at least 100 expected observations of the minority class under the pooled RQ1 proportions.
Analysis of Results and Discussion
RQ1 – Overall SBFL Technique Performance
Table 3 presents the macro ROC-AUC obtained by the 34 SBFL techniques for each VLA model at the primary voxel size. Harmonic Mean and Rogot2 jointly achieved the highest performance overall, with a macro ROC-AUC of 0.704, and ranked first for every VLA model. Their performance, nevertheless, varied across VLAs: their ROC-AUC ranged from 0.665 for EO1 to 0.746 for Pi0. This variation indicated that the separability of successful and failed trajectories depended partly on the VLA model, even though the identity of the leading techniques remained stable. Dennis exhibited comparatively consistent behavior, ranking among the five best techniques for all four VLAs, while Phi ranked among the top five for GR00T, Pi0, and SpatialVLA but ranked sixth for EO1. In contrast, some techniques were more model-dependent. Tarantula ranked third for EO1 but sixth for GR00T, ninth for Pi0, and eighth for SpatialVLA. Similarly, Tarwid ranked fourth for EO1 and fifth for GR00T, but only tenth for Pi0 and seventh for SpatialVLA. Zoltar ranged from fifth for SpatialVLA to fifteenth for EO1. These differences suggested that the distribution of passing and failing voxel spectra varied across VLA models and affected the formulas differently.
The table also revealed substantial differences among the 34 techniques. The techniques occupying the top five overall ranks obtained macro ROC-AUC values between 0.655 and 0.704, whereas several techniques performed close to random discrimination, with some obtaining ROC-AUC values below 0.5 for at least one VLA. Tarantula and Tarwid also produced nearly identical overall results, both obtaining a macro ROC-AUC of 0.655 at the reported precision. Moreover, several groups of techniques produced identical or nearly identical results, including Hamann, Hamming, Rogers & Tanimoto, Simple Matching, and Sokal, as well as Anderberg, Dice, Goodman, Jaccard, Kulczynski, and Sorensen-Dice. This redundancy may have resulted from mathematical relationships among the formulas or from different formulas inducing the same voxel ordering under sparse and highly tied spectra. Similar redundancies among SBFL techniques have also been reported in previous comparative studies [3].
Table 4 presents the statistical comparison of the techniques that reached a top-five rank for at least one VLA. Rogot2 reproduced exactly the performance of Harmonic Mean: its ROC-AUC gap was zero across all 16 matched experimental blocks, its Holm-adjusted -value was 1.000, and its paired rank-biserial effect size was zero. Consequently, both techniques achieved identical failure-discrimination performance under the evaluated conditions. Dennis and Phi were the closest alternatives, with ROC-AUC gaps of 0.016 and 0.022, respectively. However, neither difference remained statistically significant after Holm correction ( and , respectively). In contrast, Tarantula, Tarwid, Arithmetic Mean, and Zoltar performed significantly worse than Harmonic Mean, with adjusted -values between 0.017 and 0.018. The 95% confidence intervals for Dennis and Phi excluded zero, although these pointwise intervals were not adjusted for multiple comparisons. Therefore, their observed performance differences should not be interpreted as statistically significant under the corrected tests.
| Technique | ROC-AUC [95% CI] | Gap [95% CI] | Holm | Equivalent | |
| Harmonic Mean | 0.704 [0.680, 0.728] | Ref. | – | – | Ref. |
| Rogot2 | 0.704 [0.680, 0.728] | 0.000 [0.000, 0.000] | 1.000 | 0.000 | Yes |
| Dennis | 0.688 [0.667, 0.710] | 0.016 [0.007, 0.025] | 0.279 | 0.838 | No |
| Phi (Geometric Mean) | 0.682 [0.659, 0.705] | 0.022 [0.012, 0.032] | 0.134 | 0.882 | No |
| Tarantula | 0.655 [0.636, 0.675] | 0.049 [0.031, 0.069] | 0.018 | 0.985 | No |
| Tarwid | 0.655 [0.636, 0.675] | 0.049 [0.031, 0.069] | 0.018 | 0.985 | No |
| Arithmetic Mean | 0.653 [0.632, 0.676] | 0.051 [0.036, 0.067] | 0.017 | 1.000 | No |
| Zoltar | 0.643 [0.614, 0.672] | 0.061 [0.049, 0.075] | 0.017 | 1.000 | No |
Table 4. RQ1 – Statistical comparison of the techniques reaching a top-five rank for at least one VLA. The AUC gap is the leader’s ROC-AUC minus the candidate’s ROC-AUC. Confidence intervals are pointwise 95%; -values are adjusted using Holm correction over all 561 pairwise comparisons among the 34 SBFL techniques, and is the paired rank-biserial effect size.
Overall, Harmonic Mean and Rogot2 consistently achieved the highest observed ROC-AUC across all four VLA models. Their joint leading position was also preserved when each VLA model or trajectory-generation strategy was excluded in turn, indicating that the overall ranking was not driven by a single experimental group. Across the complete family of pairwise comparisons, Harmonic Mean showed statistically significant differences against 30 of the remaining 33 techniques, whereas the comparisons against Rogot2, Dennis, and Phi were not significant after Holm correction. These results supported retaining Harmonic Mean as the reference technique for the subsequent research questions, while acknowledging Rogot2 as an equally effective choice under the primary experimental conditions. Dennis and Phi remained the strongest alternatives in terms of observed ROC-AUC, although the corrected statistical analysis did not establish a significant difference from the leading techniques. Overall, the first RQ can be answered as follows:
Answer to RQ1: Under the studied conditions, Harmonic Mean and Rogot2 achieved the highest observed failure-discrimination performance among the 34 evaluated SBFL techniques, jointly ranking first for every VLA model with an overall macro ROC-AUC of 0.704. The corrected statistical analysis showed significant differences between Harmonic Mean and 30 of the other 33 techniques, but not Rogot2, Dennis, or Phi.
RQ2 – Baseline Comparison
Table 5 compares Harmonic Mean, the VoxelSBFL reference technique selected in RQ1, with the black-box metrics proposed by Valle et al. [29]. These metrics capture different aspects of failure behavior, including changes in generated actions, instability in the executed trajectory, and progress towards task-relevant positions. The comparison allows us to assess whether the spatial information captured by VoxelSBFL provides effective discrimination between successful and failed executions. Paired Wilcoxon signed-rank tests with Holm correction showed that Harmonic Mean significantly outperformed all seven action- and motion-instability baselines (). Optimal Trajectory, however, significantly outperformed Harmonic Mean ().
| Method | EO1 | GR00T | Pi0 | SpatialVLA | Overall |
| Optimal Trajectory† | 0.6666 | 0.7131 | 0.8029 | 0.8408 | 0.7559 |
| Harmonic Mean★ | 0.6652 | 0.6730 | 0.7461 | 0.7315 | 0.7039 |
| Rogot2★ | 0.6652 | 0.6730 | 0.7461 | 0.7315 | 0.7039 |
| Dennis★ | 0.6462 | 0.6602 | 0.7383 | 0.7075 | 0.6880 |
| Phi (Geometric Mean)★ | 0.6375 | 0.6533 | 0.7370 | 0.6994 | 0.6818 |
| TCP Acceleration Instability | 0.6995 | 0.5598 | 0.6671 | 0.6934 | 0.6550 |
| Tarantula★ | 0.6590 | 0.6451 | 0.6715 | 0.6439 | 0.6549 |
| TCP Velocity Instability | 0.6782 | 0.5964 | 0.6136 | 0.6745 | 0.6407 |
| Trajectory Instability | 0.6891 | 0.5432 | 0.6195 | 0.6750 | 0.6317 |
| TCP Position Instability | 0.6124 | 0.6496 | 0.5624 | 0.5965 | 0.6052 |
| Action Position Instability | 0.5518 | 0.5796 | 0.5571 | 0.7038 | 0.5981 |
| Action Velocity Instability | 0.5614 | 0.5599 | 0.5606 | 0.6992 | 0.5953 |
| Action Acceleration Instability | 0.5569 | 0.5555 | 0.5569 | 0.6941 | 0.5908 |
Table 5. RQ2 – Trajectory-level failure-discrimination ROC-AUC. Harmonic Mean is the VoxelSBFL reference technique selected in RQ1. Bold denotes the best result and underlining denotes the second-best result in each column. † denotes a task-aware metric using object and goal information and ★ indicates an SBFL technique.
At the aggregate level, Optimal Trajectory achieved the highest macro ROC-AUC, followed jointly by Harmonic Mean and Rogot2, which produced identical reported scores for every VLA model. Dennis and Phi (Geometric Mean) followed in fourth and fifth overall ranking, respectively. All these four SBFL techniques outperformed every reported action- and motion-instability baseline overall. Among these baselines, TCP Acceleration Instability achieved the highest value, followed by TCP Velocity Instability and Trajectory Instability. However, this advantage did not extend to every SBFL technique. For instance, Tarantula achieved nearly matching scores to TCP Acceleration Instability at the reported precision. These results suggest that information about where the robot moves (VoxelSBFL) can complement information about how (our previous work [29]) it moves when distinguishing successful and failed executions. The instability metrics proposed in our previous study [29] characterize variations in actions or motion, whereas VoxelSBFL uses previously labeled executions to associate workspace regions with failures. Consequently, a trajectory may receive a high suspiciousness score based on VoxelSBFL because it visits regions associated with failures. However, its movements can be soft and not exhibit pronounced instability.
Relative performance nevertheless varied across VLA models. For GR00T, Pi0, and SpatialVLA, Optimal Trajectory achieved the highest ROC-AUC, while Harmonic Mean and Rogot2 shared second place. Both techniques outperformed every action- and motion-instability metric baseline for these three models. Dennis also exceeded all instability baselines for these models. Phi similarly exceeded all instability baselines for GR00T and Pi0, but its SpatialVLA result was below Action Position Instability. Tarantula showed a less consistent advantage, exceeding all instability baselines for Pi0 but not for the other models. Nevertheless, EO1 exhibited a different pattern. TCP Acceleration Instability achieved the highest ROC-AUC score, followed by Trajectory Instability and TCP Velocity Instability. Thus, the three strongest motion-instability metrics outperformed both the Optimal Trajectory metric and all reported SBFL techniques for EO1. This pattern suggests that motion characteristics were more informative for failure discrimination in this model under the evaluated conditions. A plausible explanation might be that passing and failing trajectories visited similar regions but differed in their movements within those regions.
Optimal Trajectory provided the strongest overall discrimination. However, its application requires task-specific information to estimate the optimal trajectory, including the position of the target object, and, where applicable, the goal. Consequently, its application in real-world operation depends on the availability of this information, which may need to be estimated rather than obtained directly from the environment. In contrast, VoxelSBFL does not require explicit object or goal positions, but only suspiciousness maps constructed from previously labeled executions. The comparison therefore reflects differences in both predictive performance and the information required by each method.
Answer to RQ2: Harmonic Mean and Rogot2 achieved higher overall ROC-AUC than all evaluated action- and motion-instability baselines. The advantage of Harmonic Mean and Rogot2 was statistically significant after Holm correction. Only Optimal Trajectory achieved significantly higher performance, but it requires task-specific object and goal information that may not be available in real-world operation. The leading SBFL techniques therefore offer a competitive alternative to the evaluated black-box metrics, although their relative performance varies across models.
RQ3 – Voxel-Size Sensitivity:
Table 6 reports the sensitivity of Harmonic Mean, the reference technique selected in RQ1, to the voxel size. At the aggregate level, the primary size of 0.02 achieved the highest macro ROC-AUC (0.7039), followed by 0.01 (0.6946) and 0.05 (0.6930). However, the descriptively best size varied across VLA models. EO1 and SpatialVLA performed best with 0.01, Pi0 performed best with 0.02, and GR00T performed best with 0.05. The magnitude and direction of the changes also differed: increasing the voxel size reduced the performance of EO1 and SpatialVLA, whereas it progressively improved the performance of GR00T. These results indicate that no voxel size was descriptively optimal for every VLA. This model-specific behavior could have resulted from differences in the spatial distributions explored by their trajectories: models producing more concentrated trajectories might have benefited from finer spatial distinctions, whereas models covering more variable regions might have benefited from the aggregation provided by larger voxels.
| VLA | 0.01 | 0.02 | 0.05 | Descriptive best size |
| EO1 | 0.6711 | 0.6652 | 0.6253 | 0.01 |
| GR00T | 0.6574 | 0.6730 | 0.6815 | 0.05 |
| Pi0 | 0.7101 | 0.7461 | 0.7416 | 0.02 |
| SpatialVLA | 0.7398 | 0.7315 | 0.7235 | 0.01 |
| Macro | 0.6946 | 0.7039 | 0.6930 | 0.02 |
Table 6. RQ3 – Voxel-size sensitivity of Harmonic Mean, the frozen RQ1 reference technique. Values are macro ROC-AUC over the four trajectory-generation conditions for each VLA. Bold denotes the descriptive best size within a row; inferential tests are reported separately in Table 9.
Table 7 shows how voxel size affected the relative performance of the techniques that reached a top-five rank for at least one size within each VLA. At the primary size of 0.02, Harmonic Mean and Rogot2 jointly ranked first for all four VLAs. This consistency was not always preserved when voxel size varied. At 0.01, Harmonic Mean and Rogot2 remained the leaders for EO1, but Dennis became the best technique for GR00T and Pi0, while Zoltar ranked first for SpatialVLA. The most pronounced change occurred at 0.05, where Tarantula ranked first for EO1, GR00T, and Pi0, whereas Tarwid ranked first for SpatialVLA. The two techniques produced nearly identical results at the reported precision, with ROC-AUC values between 0.6877 and 0.7649, and their different ranks resulted from small differences in the unrounded values. However, both techniques performed poorly at 0.01: Tarantula ranked between tenth and sixteenth, while Tarwid ranked between tenth and fifteenth. Zoltar exhibited the opposite pattern, ranking among the five best techniques for every VLA at 0.01 but falling between fourteenth and thirty-second at 0.05. Therefore, voxel size did not merely shift absolute ROC-AUC values; it also changed the relative suitability of the suspiciousness formulas. A plausible explanation is that coarse voxels combined visits from more trajectories and produced more stable class-conditional evidence, which may have favored techniques such as Tarantula and Tarwid. In contrast, finer partitions preserved localized distinctions that may have benefited Zoltar.
Table | Technique | 0.01 ROC-AUC (rank) | 0.02 ROC-AUC (rank) | 0.05 ROC-AUC (rank) |
| (a) | Harmonic Mean | 0.6711 (1) | 0.6652 (1) | 0.6253 (5) |
| Rogot2 | 0.6711 (1) | 0.6652 (1) | 0.6253 (4) | |
| Tarantula | 0.5460 (16) | 0.6590 (3) | 0.7089 (1) | |
| Tarwid | 0.5460 (15) | 0.6590 (4) | 0.7089 (2) | |
| Dennis | 0.6555 (4) | 0.6462 (5) | 0.6680 (3) | |
| Zoltar | 0.6705 (3) | 0.5794 (15) | 0.4999 (32) | |
| Phi (Geometric Mean) | 0.5977 (5) | 0.6375 (6) | 0.6204 (6) | |
| (b) | Harmonic Mean | 0.6574 (2) | 0.6730 (1) | 0.6815 (4) |
| Rogot2 | 0.6574 (2) | 0.6730 (1) | 0.6817 (3) | |
| Dennis | 0.6594 (1) | 0.6602 (3) | 0.6668 (5) | |
| Tarantula | 0.5289 (11) | 0.6451 (6) | 0.6877 (1) | |
| Tarwid | 0.5290 (10) | 0.6451 (5) | 0.6877 (2) | |
| Phi (Geometric Mean) | 0.6561 (4) | 0.6533 (4) | 0.6627 (6) | |
| Zoltar | 0.6556 (5) | 0.5985 (10) | 0.4999 (25) | |
| (c) | Harmonic Mean | 0.7101 (3) | 0.7461 (1) | 0.7416 (3) |
| Rogot2 | 0.7101 (3) | 0.7461 (1) | 0.7416 (3) | |
| Dennis | 0.7149 (1) | 0.7383 (3) | 0.7096 (5) | |
| Tarantula | 0.5152 (15) | 0.6715 (9) | 0.7649 (1) | |
| Phi (Geometric Mean) | 0.7147 (2) | 0.7370 (4) | 0.6705 (6) | |
| Tarwid | 0.5152 (14) | 0.6714 (10) | 0.7649 (2) | |
| Arithmetic Mean | 0.6989 (6) | 0.7106 (5) | 0.6517 (7) | |
| Zoltar | 0.7017 (5) | 0.7031 (8) | 0.5402 (15) | |
| (d) SpatialVLA | Harmonic Mean | 0.7398 (2) | 0.7315 (1) | 0.7235 (3) |
| Rogot2 | 0.7398 (2) | 0.7315 (1) | 0.7235 (3) | |
| Zoltar | 0.7421 (1) | 0.6902 (5) | 0.5452 (14) | |
| Tarwid | 0.5113 (11) | 0.6439 (7) | 0.7564 (1) | |
| Tarantula | 0.5113 (10) | 0.6439 (8) | 0.7564 (2) | |
| Dennis | 0.7129 (4) | 0.7075 (3) | 0.7141 (5) | |
| Phi (Geometric Mean) | 0.7115 (5) | 0.6994 (4) | 0.6960 (6) |
Table 7. RQ3 – Voxel-size sensitivity of techniques reaching a top-five rank for at least one size within each VLA. Each cell reports macro ROC-AUC and the rank in parentheses. Ranks use unrounded values. Bold denotes rank 1 and underlining denotes the remaining top-five ranks.
Table 8 analyzes the coverage and computational consequences of changing the voxel size. The fraction of known voxels increased consistently with voxel size. At 0.01, known coverage ranged from 84.2% for to 92.8% for ; at 0.02, it ranged from 97.8% to 98.8%; and at 0.05, it reached between 99.7% and 99.8%. This improvement was accompanied by a substantial reduction in the number of training voxels. Depending on the VLA, the mean number decreased from approximately 31,000–38,000 at 0.01 to 5,700–7,500 at 0.02 and only 570–740 at 0.05. The complete 34-technique artifacts decreased from 1.58–2.86 MB at 0.01 to 0.15–0.18 MB at 0.05. Execution times decreased accordingly: fitting required 0.137–0.180 seconds at 0.01 and 0.025–0.034 seconds at 0.05, while scoring decreased from 0.244–0.296 seconds to 0.014–0.016 seconds. Thus, while larger voxels produce considerably more compact and computationally efficient models, smaller voxels remain fast for practical adoption.
Table 8. RQ3 – Coverage and computational diagnostics by VLA and voxel size for Harmonic Mean. Values are averaged over the 80 held-out folds in each VLA-size cell. Artifact sizes use decimal megabytes.
| VLA | Size | Known | Ties | Train voxels | Artifact (MB) | Fit (s) | Score (s) |
| EO1 | 0.01 | 92.4% | 90.1% | 30,791 | 2.12 | 0.146 | 0.244 |
| 0.02 | 98.8% | 67.2% | 5,651 | 0.93 | 0.061 | 0.057 | |
| 0.05 | 99.8% | 84.6% | 572 | 0.15 | 0.028 | 0.014 | |
| GR00T | 0.01 | 92.8% | 90.7% | 37,545 | 2.86 | 0.180 | 0.296 |
| 0.02 | 98.8% | 59.1% | 7,231 | 1.18 | 0.080 | 0.071 | |
| 0.05 | 99.8% | 75.0% | 736 | 0.18 | 0.034 | 0.016 | |
| Pi0 | 0.01 | 84.2% | 93.3% | 37,656 | 1.58 | 0.149 | 0.287 |
| 0.02 | 97.9% | 69.2% | 7,532 | 1.03 | 0.061 | 0.068 | |
| 0.05 | 99.7% | 71.9% | 738 | 0.18 | 0.027 | 0.016 | |
| SpatialVLA | 0.01 | 86.9% | 91.1% | 33,879 | 1.69 | 0.137 | 0.264 |
| 0.02 | 97.8% | 66.3% | 6,773 | 0.88 | 0.057 | 0.064 | |
| 0.05 | 99.7% | 76.8% | 692 | 0.16 | 0.025 | 0.015 |
Table 8.
The score-tie results in Table 8 exhibited a non-monotonic pattern. The finest voxel size produced the largest tie fractions, ranging from 90.1% to 93.3%. These fractions decreased substantially at 0.02 before increasing again at 0.05. This pattern suggested two different sources of ambiguity. With 0.01 voxels, the large and sparsely covered voxel space may have caused many low-support voxels to share the same contingency counts and suspiciousness values, leading multiple trajectories to receive identical aggregated risk scores. Conversely, at 0.05, aggregating many positions into a small number of voxels may have caused more trajectories to traverse voxels with similar spectra, again reducing the diversity of their risk scores. The intermediate size of 0.02 produced the lowest tie fraction for every VLA while maintaining known coverage between 97.8% and 98.8%. Descriptively, it therefore offered the most favorable compromise between spatial resolution, coverage, and trajectory-score discrimination for Harmonic Mean. The exact paired comparisons identified ten significant technique-specific voxel-size effects after Holm correction, indicating that sensitivity to voxel size depended on the suspiciousness technique.
Table 9 showed that the differences observed for Harmonic Mean were not statistically significant at the global level. The Friedman test did not reject the null hypothesis of equal performance across voxel sizes (, ). Compared with 0.02, the ROC-AUC decreased by 0.0093 at 0.01 and by 0.0110 at 0.05, and both confidence intervals included zero. The 0.01 configuration was classified as practically equivalent to the reference size, whereas 0.05 fell outside the predefined practical-equivalence group. Therefore, the data did not provide sufficient evidence of a general voxel-size effect for Harmonic Mean, although 0.02 remained the descriptively strongest configuration.
| (a) Voxel-size effect for Harmonic Mean | |||||
| Size | ROC-AUC [95% CI] | Δ vs. 0.02 [95% CI] | Equivalent | ||
| 0.01 | 0.6946 [0.6727, 0.7178] | -0.0093 [-0.0219, +0.0030] | Yes | ||
| 0.02 | 0.7039 [0.6796, 0.7280] | +0.0000 [+0.0000, +0.0000] | Yes | ||
| 0.05 | 0.6930 [0.6652, 0.7188] | -0.0110 [-0.0271, +0.0033] | No | ||
| (b) Practically meaningful technique–size interactions | |||||
| Size vs. 0.02 | Technique | ΔAUC | ΔΔAUC [95% CI] | Holm | |
| 0.01 | Mountford | +0.0506 | +0.0600 [+0.0342, +0.0895] | 0.002 | +1.000 |
| 0.01 | Zoltar | +0.0496 | +0.0590 [+0.0467, +0.0721] | 0.002 | +1.000 |
| 0.01 | Ochiai2 | -0.0658 | -0.0565 [-0.0690, -0.0438] | 0.004 | -0.985 |
| 0.01 | Tarwid | -0.1295 | -0.1202 [-0.1338, -0.1068] | 0.002 | -1.000 |
| 0.01 | Tarantula | -0.1295 | -0.1202 [-0.1338, -0.1068] | 0.002 | -1.000 |
| 0.05 | Tarwid | +0.0746 | +0.0856 [+0.0615, +0.1123] | 0.002 | +1.000 |
| 0.05 | Tarantula | +0.0746 | +0.0856 [+0.0615, +0.1123] | 0.002 | +1.000 |
| 0.05 | Zoltar | -0.1215 | -0.1105 [-0.1340, -0.0848] | 0.004 | -0.985 |
| (c) Ranking stability | |||||
| Voxel-size pair | Spearman | ||||
| 0.01–0.02 | 0.921 | ||||
| 0.01–0.05 | 0.769 | ||||
| 0.02–0.05 | 0.899 |
Table 9. RQ3 – Global statistical analysis over the 16 matched VLA–trajectory-condition blocks. Panel (a) reports the frozen Harmonic Mean size comparison (Friedman , ). Panel (b) contains interactions satisfying Holm-corrected and . Panel (c) reports tie-aware Spearman rank correlations across all 34 techniques.
The interaction analysis nevertheless identified eight cases in which voxel size affected a technique differently from Harmonic Mean. At 0.01, Mountford and Zoltar improved relative to the primary size, with difference-in-differences values of 0.0600 and 0.0590, respectively. In contrast, Ochiai2 deteriorated at 0.01 (). Tarwid and Tarantula exhibited the strongest size-dependent responses: both deteriorated sharply at 0.01 () but improved at 0.05 (). Zoltar exhibited the reverse response at 0.05 (). All eight interactions remained significant after Holm correction and showed very large paired effect sizes, with absolute values between 0.985 and 1.000. The technique rankings were nevertheless relatively stable overall, particularly between 0.01 and 0.02 () and between 0.02 and 0.05 (). The lower correlation between the two extreme sizes () indicated that ranking changes accumulated as the difference in spatial granularity increased.
Overall, voxel size did not produce a statistically significant global change in the performance of Harmonic Mean, although its descriptive effect varied across VLAs and several S B F L techniques responded differently to changes in granularity. The size of 0.02 achieved the highest aggregate ROC-AUC for the reference technique and offered high known coverage with substantially fewer voxels and fewer score ties than 0.01. Although 0.05 was the most computationally efficient configuration and strongly favored Tarantula and Tarwid, it reduced spatial resolution and was not practically equivalent to 0.02 for Harmonic Mean. Consequently, 0.02 descriptively represented the most balanced default for trajectory-level evaluation in the considered setting. However, the strong interactions observed for Tarantula, Tarwid, Zoltar, Mountford, and Ochiai2 demonstrated that voxel size needed to be controlled when comparing SBFL techniques. This selection should not be interpreted as establishing 0.02 as the optimal localization resolution, which would require ground truth about the problematic spatial regions.
Answer to RQ3: No voxel size was uniformly optimal across all VLA models and SBFL techniques. For the reference technique obtained from RQ1, i.e., Harmonic Mean, voxel size did not produce a statistically significant global effect, although 0.02 achieved the highest macro ROC-AUC and provided the best balance between performance, coverage, score discrimination, and computational cost. Some techniques were substantially more sensitive to spatial granularity: Tarantula and Tarwid were strongly favored by coarse voxels, whereas Zoltar and Mountford benefited from finer voxels. Ochiai2 also deteriorated significantly at the finest resolution relative to Harmonic Mean. Thus, voxel size had to be fixed or explicitly controlled when comparing SBFL techniques.
RQ4 – Voxel Coverage Sensitivity
Table 10 reports the voxel coverage and held-out performance obtained as the number of spectrum-construction trajectories increased. Calibration coverage increased from 8.2% to 67.6% for EO1, from 9.1% to 68.6% for GR00T, from 5.1% to 55.6% for Pi0, and from 6.0% to 57.0% for SpatialVLA. Calibration and held-out coverage remained almost identical throughout the experiment, with a maximum difference of only 0.071 percentage points. This suggested that the calibration runs provided a representative estimate of the spatial coverage subsequently observed in the untouched test runs.
Table 10. RQ4 learning-curve coverage and held-out performance by VLA. Coverage values are percentages. The Harmonic Mean cell reports ROC-AUC and its within-VLA, within-size rank among the 34 techniques in parentheses; leaders were determined from unrounded ROC-AUC.
| VLA | Calibration cov. (%) | Held-out cov. (%) | Harmonic Mean AUC (rank) | Leading technique | Leading AUC | |
| EO1 | 10 | 8.2 | 8.2 | 0.5608 (10) | Ochiai2 | 0.5684 |
| 20 | 15.3 | 15.3 | 0.5688 (16) | Rogot1 / Scott | 0.5824 | |
| 50 | 31.7 | 31.7 | 0.5938 (9) | Dennis | 0.6029 | |
| 100 | 49.3 | 49.3 | 0.6174 (3) | Dennis | 0.6199 | |
| 200 | 67.6 | 67.6 | 0.6377 (2.5) | Zoltar | 0.6381 | |
| GR00T | 10 | 9.1 | 9.1 | 0.5213 (7) | Dennis | 0.5226 |
| 20 | 16.8 | 16.8 | 0.5350 (4) | Dennis | 0.5375 | |
| 50 | 34.1 | 34.1 | 0.5656 (2) | Dennis | 0.5667 | |
| 100 | 51.5 | 51.6 | 0.5902 (3) | Zoltar | 0.5906 | |
| 200 | 68.6 | 68.6 | 0.6217 (3.5) | Mountford | 0.6220 | |
| Pi0 | 10 | 5.1 | 5.1 | 0.5764 (5) | Rogot2 | 0.5799 |
| 20 | 9.6 | 9.7 | 0.6043 (6) | Arithmetic Mean | 0.6064 | |
| 50 | 21.5 | 21.6 | 0.6225 (16) | Ochiai2 | 0.6389 | |
| 100 | 36.5 | 36.6 | 0.6437 (15.5) | Arithmetic Mean | 0.6586 | |
| 200 | 55.6 | 55.7 | 0.6707 (11.5) | Phi (Geometric Mean) | 0.6826 | |
| SpatialVLA | 10 | 6.0 | 6.0 | 0.5403 (8) | Tarantula / Tarwid | 0.5482 |
| 20 | 11.2 | 11.2 | 0.5654 (5) | Zoltar | 0.5709 | |
| 50 | 23.9 | 23.9 | 0.6101 (3) | Zoltar | 0.6151 | |
| 100 | 39.0 | 39.0 | 0.6480 (3.5) | Zoltar | 0.6502 | |
| 200 | 57.0 | 57.0 | 0.6833 (3.5) | Mountford | 0.6840 |
Table 10.
Harmonic Mean’s ROC-AUC increased with every training-set enlargement for all four VLA models. Between 10 and 200 trajectories, its ROC-AUC increased from 0.5608 to 0.6377 for EO1, from 0.5213 to 0.6217 for GR00T, from 0.5764 to 0.6707 for Pi0, and from 0.5403 to 0.6833 for SpatialVLA. The corresponding improvements were 0.0769, 0.1004, 0.0942, and 0.1430, respectively. SpatialVLA therefore benefited most from the additional trajectories, whereas EO1 exhibited the smallest improvement. Nevertheless, the positive trend was observed consistently across all four models.
The descriptively leading technique also changed with both the VLA model and the number of trajectories. For EO1, Ochiai2 led at 10 trajectories, Rogot1 and Scott tied at 20, Dennis led at 50 and 100, and Zoltar led at 200. Dennis led the GR00T configurations with 10, 20, and 50 trajectories, whereas Zoltar led at 100 and Mountford at 200. For Pi0, Rogot2 led at 10 trajectories, Arithmetic Mean at 20, Ochiai2 at 50, Arithmetic Mean again at 100, and Phi at 200. For SpatialVLA, Tarantula and Tarwid tied at 10 trajectories, Zoltar led between 20 and 100, and Mountford led at 200. These changes indicated that the most effective suspiciousness formula depended not only on the VLA model but also on the amount of data available for constructing its spectra.
Table 11 summarizes performance over the complete learning curve using the normalized area under the learning curve (NAULC). Dennis obtained the highest macro NAULC, with 0.5994, followed closely by Rogot2 with 0.5972 and Harmonic Mean with 0.5971. Dennis’s advantage over Harmonic Mean was therefore minimal. Moreover, Dennis did not lead every VLA: it ranked first for EO1 and GR00T, Arithmetic Mean ranked first for Pi0, and Zoltar ranked first for SpatialVLA. Harmonic Mean remained competitive for EO1, GR00T, and SpatialVLA but ranked only 16th for Pi0. Ochiai2 also showed substantial model dependence: it ranked fifth for EO1 and fourth for Pi0, but only 15th for GR00T and 18th for SpatialVLA, resulting in a macro rank of 15. Consequently, no technique dominated the complete learning curve for every VLA model.
Table 11. RQ4 sample efficiency for techniques reaching a top-five NAULC rank for at least one VLA. VLA and macro cells report NAULC with rank among the 34 techniques in parentheses. Bold denotes rank 1 and underlining the remaining top-five ranks. Endpoint and gain are macro ROC-AUC.
| Technique | EO1 | GR00T | Pi0 | SpatialVLA | Macro | AUC@200 | AUC |
| Dennis | 0.6000 (1) | 0.5646 (1) | 0.6302 (3) | 0.6026 (5) | 0.5994 (1) | 0.6501 | +0.0958 |
| Rogot2 | 0.5941 (2) | 0.5644 (3) | 0.6232 (15) | 0.6072 (4) | 0.5972 (2) | 0.6534 | +0.1024 |
| Harmonic Mean | 0.5938 (3) | 0.5645 (2) | 0.6227 (16) | 0.6072 (3) | 0.5971 (3) | 0.6534 | +0.1036 |
| Phi (Geometric Mean) | 0.5929 (4) | 0.5632 (5) | 0.6310 (2) | 0.5956 (6) | 0.5957 (4) | 0.6466 | +0.0969 |
| Mountford | 0.5910 (9) | 0.5638 (4) | 0.6130 (17) | 0.6092 (2) | 0.5942 (5) | 0.6501 | +0.1032 |
| Arithmetic Mean | 0.5909 (10) | 0.5585 (7) | 0.6317 (1) | 0.5795 (7) | 0.5902 (6) | 0.6294 | +0.0796 |
| Zoltar | 0.5816 (18) | 0.5621 (6) | 0.6048 (18) | 0.6108 (1) | 0.5898 (7) | 0.6484 | +0.1150 |
| Ochiai2 | 0.5922 (5) | 0.5445 (15) | 0.6302 (4) | 0.5325 (18) | 0.5748 (15) | 0.5882 | +0.0381 |
Table 11.
The endpoint results differed slightly from the sample-efficiency ranking. At 200 trajectories, Harmonic Mean and Rogot2 jointly obtained the highest macro ROC-AUC of 0.6534, compared with 0.6501 for Dennis. Zoltar exhibited the largest improvement between 10 and 200 trajectories, increasing by 0.1150, but its lower performance with small training sets reduced its macro NAULC to 0.5898. These results illustrated the distinction between selecting a technique for strong performance with limited data and selecting one for its final performance after a larger spectrum-construction suite had become available.
Table 12 presents the matched statistical comparisons for Harmonic Mean. Every increase in training-set size produced a positive mean ROC-AUC change: 0.0187 from 10 to 20 trajectories, 0.0296 from 20 to 50, 0.0268 from 50 to 100, and 0.0285 from 100 to 200. Each improvement was positive in all 16 matched VLA–trajectory-generation blocks. All four comparisons remained statistically significant after Holm correction () and obtained a rank-biserial effect size of 1.0. The evidence therefore showed that the performance improvements were consistently observed across all evaluated VLA models and trajectory-generation conditions, rather than being driven by only a few particularly favorable experimental blocks.
Table 12. RQ4 matched adjacent-size effects for Harmonic Mean over the 16 VLA–trajectory-generation blocks. Tests used paired Wilcoxon signed-rank tests with one Holm correction.
| Transition | Mean AUC | Median AUC | Positive (%) | Holm | |
| 10–20 | +0.0187 | +0.0170 | 100.0 | 0.00012 | +1.000 |
| 20–50 | +0.0296 | +0.0288 | 100.0 | 0.00012 | +1.000 |
| 50–100 | +0.0268 | +0.0245 | 100.0 | 0.00012 | +1.000 |
| 100–200 | +0.0285 | +0.0299 | 100.0 | 0.00012 | +1.000 |
Table 12.
Finally, Table 13 examines whether coverage remained associated with ROC-AUC after controlling for training-set size, configuration, and outer fold. The estimated change per additional ten percentage points of coverage ranged from −0.008 for Baroni-Urbani & Buser to +0.0047 for Zoltar. None of the 34 techniques exhibited a statistically significant conditional relationship after Holm correction; all adjusted -values were 1.000. Thus, although coverage and ROC-AUC generally increased together as more trajectories were added, the results did not provide evidence that coverage independently explained the performance improvement. A plausible explanation was that additional trajectories improved the spectrum counts for already-observed voxels while also discovering new voxels, whereas the scalar coverage measure captured only the latter effect.
| Technique | Mean effect | Median effect | Positive blocks (%) | Holm | |
| Zoltar | +0.0047 | +0.0063 | 50.0 | 1.0000 | +0.206 |
| Gower | +0.0017 | +0.0034 | 56.2 | 1.0000 | +0.118 |
| Mountford | +0.0009 | +0.0046 | 56.2 | 1.0000 | -0.015 |
| Braun-Banquet | +0.0006 | +0.0035 | 62.5 | 1.0000 | +0.118 |
| Dennis | +0.0004 | -0.0014 | 43.8 | 1.0000 | +0.029 |
| Cohen | -0.0012 | -0.0053 | 37.5 | 1.0000 | -0.074 |
| Hamann | -0.0012 | -0.0053 | 37.5 | 1.0000 | -0.074 |
| Hamming | -0.0012 | -0.0053 | 37.5 | 1.0000 | -0.074 |
| Michael | -0.0012 | -0.0053 | 37.5 | 1.0000 | -0.074 |
| Rogers & Tanimoto | -0.0012 | -0.0053 | 37.5 | 1.0000 | -0.074 |
| Simple Matching | -0.0012 | -0.0053 | 37.5 | 1.0000 | -0.074 |
| Sokal | -0.0012 | -0.0053 | 37.5 | 1.0000 | -0.074 |
| Rogot2 | -0.0012 | +0.0031 | 50.0 | 1.0000 | -0.147 |
| Harmonic Mean | -0.0014 | +0.0025 | 50.0 | 1.0000 | -0.162 |
| Phi (Geometric Mean) | -0.0014 | +0.0000 | 50.0 | 1.0000 | -0.044 |
| Arithmetic Mean | -0.0015 | -0.0014 | 43.8 | 1.0000 | -0.088 |
| Rogot1 | -0.0021 | +0.0001 | 50.0 | 1.0000 | -0.147 |
| Scott | -0.0021 | +0.0000 | 50.0 | 1.0000 | -0.147 |
| Anderberg | -0.0022 | -0.0010 | 43.8 | 1.0000 | -0.162 |
| Dice | -0.0022 | -0.0010 | 43.8 | 1.0000 | -0.162 |
| Goodman | -0.0022 | -0.0010 | 43.8 | 1.0000 | -0.162 |
| Jaccard | -0.0022 | -0.0010 | 43.8 | 1.0000 | -0.162 |
| Kulczynski | -0.0022 | -0.0010 | 43.8 | 1.0000 | -0.162 |
| Sorensen-Dice | -0.0022 | -0.0010 | 43.8 | 1.0000 | -0.162 |
| Fleiss | -0.0028 | -0.0009 | 50.0 | 1.0000 | -0.176 |
| Pearson | -0.0030 | -0.0064 | 31.2 | 1.0000 | -0.176 |
| Fossum | -0.0032 | +0.0009 | 50.0 | 1.0000 | -0.162 |
| Ochiai2 | -0.0032 | -0.0043 | 43.8 | 1.0000 | -0.176 |
| Ample | -0.0034 | -0.0032 | 50.0 | 1.0000 | -0.147 |
| Pierce | -0.0041 | -0.0040 | 43.8 | 1.0000 | -0.338 |
| Ochiai | -0.0045 | -0.0003 | 50.0 | 1.0000 | -0.162 |
| Tarwid | -0.0068 | -0.0067 | 37.5 | 1.0000 | -0.441 |
| Tarantula | -0.0068 | -0.0067 | 37.5 | 1.0000 | -0.441 |
| Baroni-Urbani & Buser | -0.0088 | -0.0034 | 50.0 | 1.0000 | -0.294 |
Table 13. Complete conditional coverage analysis for RQ4. Effects are ROC-AUC changes per ten percentage points of calibration coverage after centering within configuration, outer fold, and suite size. Holm correction covered all 34 techniques.
Overall, increasing the number of trajectories consistently enlarged voxel coverage and improved the failure discrimination of Harmonic Mean across all evaluated VLA models and trajectory-generation conditions. However, the conditional analysis did not show that coverage alone explained these improvements, suggesting that the additional spectrum evidence provided by larger training sets also played an important role. Moreover, the technique that used this additional information most effectively varied across VLA models and training-set sizes.
Answer to RQ4: Increasing the spectrum-construction suite from 10 to 200 trajectories consistently increased voxel coverage and significantly improved the ROC-AUC of Harmonic Mean across all evaluated VLA models and trajectory-generation conditions. Nevertheless, after controlling for training-set size, the analysis found no statistically significant independent association between greater coverage and performance for any of the 34 techniques. Dennis achieved the highest observed macro NAULC, whereas Harmonic Mean and Rogot2 jointly achieved the highest macro ROC-AUC at 200 trajectories. The best-performing technique therefore depended on both the VLA model and the number of available trajectories.
RQ5: Cross-Task Generalizability
To assess cross-task generalizability, we constructed an evaluation subset for each VLA model from the 504 executions performed on the Move-Near task. This resulted in 298 selected executions for Pi0, 246 for GR00T, 223 for EO1, and 342 for SpatialVLA. Executions were randomly selected using a fixed seed, with outcome-stratified sampling and spatial constraints to approximately preserve the failure proportion observed in RQ1. The selected trajectories were evaluated using the suspiciousness maps reconstructed from the Pick-Up training data, without retraining or calibration on Move-Near. This setup examined whether failure-associated information captured from Pick-Up trajectories remained useful for distinguishing passing from failing executions on a different task.
Table 14 reports the ROC-AUC values of the 34 SBFL techniques for each VLA model and their overall macro-average across the four models. Zoltar achieved the highest overall value, at 0.589, followed by Mountford at 0.572. Harmonic Mean and Rogot2, the RQ1 reference techniques, tied for third place, also with an overall value of 0.572 at the reported precision. Gower completed the top five overall ranks with a value of 0.561. Therefore, although RQ1 reference techniques remained among the strongest overall performers, their observed discrimination performance was lower on the Move-Near task than on the Pick-Up.
| Technique | EO1 | GR00T | Pi0 | SpatialVLA | Overall |
| Zoltar | 0.604 (15) | 0.571 (4) | 0.677 (1) | 0.504 (3) | 0.589 (1) |
| Mountford | 0.593 (20) | <u>0.568 (5)</u> | <u>0.632 (2)</u> | 0.494 (7) | <u>0.572 (2)</u> |
| Harmonic Mean | 0.618 (6) | <u>0.574 (2)</u> | <u>0.585 (4)</u> | 0.511 (1) | <u>0.572 (3)</u> |
| Rogot2 | 0.618 (6) | <u>0.574 (2)</u> | <u>0.585 (4)</u> | 0.511 (1) | <u>0.572 (3)</u> |
| Gower | 0.535 (33) | 0.605 (1) | <u>0.606 (3)</u> | 0.497 (6) | <u>0.561 (5)</u> |
| Dennis | <u>0.650 (2)</u> | 0.537 (9) | 0.521 (9) | 0.502 (4) | 0.552 (6) |
| Phi (Geometric Mean) | 0.650 (1) | 0.538 (8) | 0.521 (8) | <u>0.499 (5)</u> | 0.552 (7) |
| Tarwid | 0.608 (13) | 0.551 (6) | 0.525 (7) | 0.493 (9) | 0.544 (8) |
| Tarantula | 0.608 (14) | 0.551 (7) | 0.525 (6) | 0.493 (8) | 0.544 (9) |
| Arithmetic Mean | 0.645 (3) | 0.533 (13) | 0.513 (12) | 0.480 (10) | 0.543 (10) |
| Michael | <u>0.643 (4)</u> | 0.533 (12) | 0.514 (11) | 0.473 (12) | 0.541 (11) |
| Cohen | <u>0.639 (5)</u> | 0.533 (11) | 0.514 (10) | 0.472 (13) | 0.540 (12) |
| Hamann | 0.612 (8) | 0.517 (15) | 0.493 (13) | 0.452 (29) | 0.519 (13) |
| Hamming | 0.612 (8) | 0.517 (15) | 0.493 (13) | 0.452 (29) | 0.519 (13) |
| Rogers & Tanimoto | 0.612 (8) | 0.517 (15) | 0.493 (13) | 0.452 (29) | 0.519 (13) |
| Simple Matching | 0.612 (8) | 0.517 (15) | 0.493 (13) | 0.452 (29) | 0.519 (13) |
| Sokal | 0.612 (8) | 0.517 (15) | 0.493 (13) | 0.452 (29) | 0.519 (13) |
| Ochiai2 | 0.604 (16) | 0.511 (20) | 0.492 (18) | 0.452 (28) | 0.515 (18) |
| Fleiss | 0.604 (17) | 0.501 (23) | 0.484 (19) | 0.452 (25) | 0.510 (19) |
| Ochiai | 0.592 (21) | 0.504 (21) | 0.482 (20) | 0.453 (15) | 0.508 (20) |
| Fossum | 0.591 (22) | 0.504 (22) | 0.481 (21) | 0.453 (16) | 0.507 (21) |
| Rogot1 | 0.599 (18) | 0.495 (24) | 0.479 (22) | 0.452 (26) | 0.506 (22) |
| Scott | 0.599 (18) | 0.495 (24) | 0.479 (22) | 0.452 (26) | 0.506 (22) |
| Baroni-Urbani & Buser | 0.580 (23) | 0.487 (26) | 0.473 (30) | 0.453 (24) | 0.498 (24) |
| Anderberg | 0.576 (25) | 0.486 (27) | 0.473 (24) | 0.453 (17) | 0.497 (25) |
| Dice | 0.576 (25) | 0.486 (27) | 0.473 (24) | 0.453 (17) | 0.497 (25) |
| Goodman | 0.576 (25) | 0.486 (27) | 0.473 (24) | 0.453 (17) | 0.497 (25) |
| Jaccard | 0.576 (25) | 0.486 (27) | 0.473 (24) | 0.453 (17) | 0.497 (25) |
| Kulczynski | 0.576 (25) | 0.486 (27) | 0.473 (24) | 0.453 (17) | 0.497 (25) |
| Sorensen-Dice | 0.576 (25) | 0.486 (27) | 0.473 (24) | 0.453 (17) | 0.497 (25) |
| Braun-Banquet | 0.574 (31) | 0.485 (33) | 0.473 (31) | 0.453 (23) | 0.496 (31) |
| Pierce | 0.546 (32) | 0.470 (34) | 0.453 (32) | 0.451 (34) | 0.480 (32) |
| Ample | 0.576 (24) | 0.521 (14) | 0.344 (33) | 0.477 (11) | 0.479 (33) |
| Pearson | 0.513 (34) | 0.535 (10) | 0.260 (34) | 0.456 (14) | 0.441 (34) |
Table 14. RQ5 – Macro ROC-AUC of the 34 SBFL techniques for each VLA model on the Move-Near task. Parentheses report tie-aware ranks. Bold denotes rank 1 and underlining denotes the remaining top-five ranks. Harmonic Mean and Rogot2 are tied because they produce identical results in every reported column.
Unlike in RQ1, where Harmonic Mean and Rogot2 ranked first for all four VLA models, no technique led across all models on the Move-Near task. Phi (Geometric Mean) ranked first EO1, followed by Dennis. Gower ranked first for GR00T, while Zoltar led for Pi0. Harmonic Mean and Rogot2 shared first place for SpatialVLA. These results indicate that the strongest transferred technique depended on the VLA model. The model-specific rankings also showed why the overall results should be interpreted alongside the individual columns. Although Zoltar ranked first overall, it was the fifteenth for EO1, fourth for GR00T, first for Pi0, and third for SpatialVLA. Mountford ranked second for Pi0 and fifth for GR00T, but twentieth for EO1 and seventh for SpatialVLA. Harmonic Mean and Rogot2 were more consistent, as they ranked second for GR00T, fourth for Pi0, and first for SpatialVLA, although they placed sixth for EO1. The degree of discrimination also varied across models. The best transferred techniques achieved ROC-AUC values of 0.650 for EO1, 0.605 for GR00T, and 0.677 for Pi0. In contrast, the highest value for SpatialVLA was 0.511, indicating discrimination close to chance level (i.e., random discrimination). Several techniques also produced values below 0.5 for individual models. For example, Pearson obtained 0.260 for Pi0, while Ample obtained 0.344. Therefore, transferring a suspiciousness map did not consistently preserve useful failure discrimination across techniques and models. Several groups of techniques shared the same reported values and ranks. Harmonic Mean and Rogot2 tied in every column. The same pattern occurred for Hamann, Hamming, Rogers & Tanimoto, Simple Matching, and Sokal; and for Anderberg, Dice, Goodman, Jaccard, Kulczynski, and Sorensen-Dice. These similarities are consistent with mathematical relationships among the formulas or equivalent trajectory rankings on the evaluated data.
Compared with RQ1, Harmonic Mean and Rogot2 decreased from an overall macro ROC-AUC of 0.704 to approximately 0.572, a reduction of 0.132. The largest decrease occurred for SpatialVLA, from 0.732 to 0.511, followed by Pi0, from 0.746 to 0.585. Their values also decreased for GR00T, from 0.673 to 0.574, and for EO1, from 0.665 to 0.618. Thus, the discrimination achieved within the Pick-Up task transferred only partially, with the extent of the decrease depending on the model. One plausible explanation is that regions associated with failures during Pick-Up are less informative about failures during the Move-Near task. Moreover, the ordering of the techniques also changed between tasks. Zoltar moved from tenth overall in RQ1 to first on Move-Near, despite its overall macro ROC-AUC decreasing from 0.643 to 0.589. Mountford moved from twelfth to second overall, while its value decreased from 0.586 to 0.572. These changes show that the improvement in relative rank does not necessarily imply an improvement in discrimination. They also indicate that the technique rankings observed in RQ1 did not remain stable after transfer.
When comparing against the baselines, as shown in Table 15, among the generic baselines, TCP Acceleration Instability achieved the highest overall macro ROC-AUC, at 0.6172, followed by Trajectory Instability at 0.6162 and TCP Velocity Instability at 0.5985. All three exceeded Zoltar, the strongest transferred SBFL technique overall, at 0.5892. Harmonic Mean and Rogot2 each achieved 0.5718. Consequently, the strongest generic baselines provided better overall discrimination than the transferred suspiciousness maps on the evaluated samples. Nevertheless, this comparison differed across models. For EO1, TCP Acceleration Instability achieved 0.7881, exceeding the best transferred SBFL result of 0.650 from Phi (Geometric Mean). For GR00T, Action Position Instability achieved 0.6568, compared with 0.6047 for Gower, the strongest transferred SBFL technique for that model. For Pi0, Zoltar achieved 0.6771, slightly exceeding the strongest generic baseline, TCP Acceleration Instability, at 0.6708. For SpatialVLA, Harmonic Mean and Rogot2 achieved 0.5105, exceeding the best generic baseline, TCP Acceleration Instability, at 0.4682. However, the SpatialVLA results remained close to chance level even for the leading transferred techniques. Thus, generic baselines performed better for EO1 and GR00T, whereas transferred SBFL achieved higher values for Pi0 and SpatialVLA. These results suggest that, although the baselines performed better overall, metric selection should account for both the VLA model and the target task.
| Method | EO1 | GR00T | Pi0 | SpatialVLA | Overall | Rank |
| Optimal Trajectory† | 0.8980 | 0.8742 | 0.9473 | 0.7644 | 0.8710 | 1 |
| TCP Acceleration Instability | 0.7881 | 0.5416 | 0.6708 | 0.4682 | 0.6172 | 2 |
| Trajectory Instability | 0.7858 | 0.5529 | 0.6615 | 0.4645 | 0.6162 | 3 |
| TCP Velocity Instability | 0.7752 | 0.5618 | 0.6285 | 0.4286 | 0.5985 | 4 |
| Zoltar★ | 0.6041 | 0.5714 | 0.6771 | 0.5044 | 0.5892 | 5 |
| Action Position Instability | 0.7724 | 0.6568 | 0.5065 | 0.4125 | 0.5871 | 6 |
| Action Velocity Instability | 0.7723 | 0.6441 | 0.5011 | 0.4131 | 0.5826 | 7 |
| Action Acceleration Instability | 0.7707 | 0.6381 | 0.4920 | 0.4027 | 0.5759 | 8 |
| Mountford★ | 0.5926 | 0.5682 | 0.6320 | 0.4945 | 0.5718 | 9 |
| Harmonic Mean★ | 0.6182 | 0.5738 | 0.5846 | 0.5105 | 0.5718 | 10 |
| Rogot2★ | 0.6182 | 0.5738 | 0.5846 | 0.5105 | 0.5718 | 10 |
| TCP Position Instability | 0.7285 | 0.6412 | 0.5816 | 0.3317 | 0.5708 | 12 |
| Gower★ | 0.5347 | 0.6047 | 0.6064 | 0.4973 | 0.5608 | 13 |
Table 15. RQ5 baseline trajectory-level failure-discrimination ROC-AUC on the selected RQ5 evaluation samples. Overall is the equal-weight mean across the four models. Bold denotes the best value and underlining the second-best value per column. † indicates a task-aware metric using object/goal state and ★ indicates an SBFL technique.
Optimal Trajectory achieved the highest ROC-AUC for every VLA model. However, it uses task-specific object and goal information, whereas the generic baselines use trajectory characteristics, and the transferred SBFL maps use failure-associated information from Pick-Up. These differences in the information required by each method should be considered alongside their discrimination performance when selecting a metric for a target task.
Answer to RQ5: Under the studied conditions, suspiciousness maps constructed from Pick-Up trajectories retained some ability to distinguish passing from failing Move-Near executions, but their effectiveness depended on the VLA model and SBFL technique. The strongest generic baselines outperformed transferred SBFL overall and for EO1 and GR00T, while transferred SBFL achieved higher values for Pi0 and SpatialVLA. Therefore, transferred suspiciousness maps should be validated on the intended target task before being relied upon in operation.
Concluding Remark and Discussion
The results show that the use of VoxelSBFL provides useful evidence for distinguishing successful from failed VLA executions. However, its effectiveness depends on the policy, task, and spectrum-construction setting. Although Harmonic Mean and Rogot2 led at the reference voxel size, changes in granularity, training-set size, and task affected technique rankings. These findings inform the potential applications of VoxelSBFL and the conditions under which its maps can be reused.
Key findings
Our study leverages four key findings that are worth discussing:
Finding 1 – SBFL can distinguish successful and failed VLA executions from physical workspace coverage: By treating workspace voxels as coverage elements, VoxelSBFL applies spectrum-based analysis to the observable behavior of a robot. It constructs spectra from recorded tool-center-point trajectories and episode outcomes, which are available independently of the policy’s architecture. At the reference voxel size, Harmonic Mean and Rogot2 ranked first for all four VLAs and achieved a macro ROC-AUC of 0.704. This result shows that physical workspace coverage can serve as a basis for applying SBFL to black-box VLA policies and that certain failures can be associated with some VLA workspace regions.
Finding 2 – Voxel visitation reveals spatial patterns associated with failure: The discrimination achieved on held-out executions indicates that where the robot moves carries information about whether it succeeds. VoxelSBFL makes this information inspectable by ranking workspace regions according to their association with failed executions. A highly ranked region may be where an unsuccessful grasp occurs, where the effects of an earlier error become visible, or where the robot attempts to recover. The map therefore helps narrow down which segments of a trajectory warrant inspection. Establishing the cause of an individual failure requires examining what the robot did in and around those segments.
Finding 3 – Spatial visitation and motion instability capture different aspects of failure: On the “Pick-Up” task, Harmonic Mean outperformed all seven action- and motion-instability baselines proposed by Valle et al. [29] overall, while several motion-based metrics performed better for EO1. One plausible explanation is that a smooth trajectory can still take the end effector to an unsuitable location, while successful and failed trajectories that visit similar regions can differ in how the robot moves through them. The task-aware Optimal Trajectory metric achieved the highest overall discrimination using additional information about the object and goal. Together, these results motivate studying when spatial visitation, motion dynamics, and task context provide complementary evidence.
Finding 4 – The learned spatial patterns depend on the policy and task: Figure 5 shows where the highest-ranked voxels occur for each VLA under the same Pick-Up task and robot embodiment. In the common viewing orientation, EO1’s higher-ranked voxels appear mainly along the left flank of the visited volume, whereas GR00T shows a prominent higher-ranked patch on the right flank. Pi0 shows higher-ranked voxels on both outer flanks around a lower-ranked central region. For SpatialVLA, higher-ranked voxels extend along an elongated band beside a lower-ranked strip. These differences illustrate how the same SBFL formula can direct an engineer toward different parts of the workspace for different policies. Colors encode relative ranks within each policy’s maps, and marker size and opacity indicate the amount of execution evidence supporting each voxel.
Figure 5. Three-dimensional distribution of failure-associated voxel suspiciousness for the four VLA models. Colors represent the median within-map percentile of the Harmonic Mean suspiciousness score at voxel size 0.02, computed across the 80 RQ1 maps available for each VLA. Warmer colors indicate higher suspiciousness, while marker size and opacity increase with episode support.
Applicability and Implications
VoxelSBFL may have two distinct yet complementary applications:
Runtime intervention and self-healing: Voxel suspiciousness could serve as a signal for an external guardrail that supervises the VLA. When a trajectory enters a highly suspicious region, or is predicted to enter one, the guardrail could suspend VLA control, bring the robot to a controlled stop, and request human assistance. For a robot with a mobile base, the guardrail could instead transfer the main control to a navigation policy that repositions the robot relative to the target object. From the new base pose, the VLA could retry the manipulation from a different viewpoint and approach the object through a different part of its reachable workspace. This is a form of non-invasive runtime recovery in which the guardrail changes the conditions under which the VLA acts. Because our evaluation scores complete trajectories, a runtime study would need to establish whether alerts arrive early enough to intervene, how often they trigger unnecessary interventions, and whether repositioning improves task completion.
Offline diagnosis and targeted VLA repair: Engineers could use suspicious voxels to select segments of failed executions for inspection and compare them with successful executions under similar scene conditions. Reviewing camera observations, gripper actions, and object motion before and after a visit can help identify where the behaviors began to diverge. Engineers could then vary the object’s initial pose to test whether the suspected behavior recurs and collect successful demonstrations under those conditions. For example, if inspection reveals that the gripper repeatedly closes before reaching the object, the new demonstrations could show the correct approach and closure timing. These demonstrations could support targeted fine-tuning, with follow-up tests assessing whether the revised policy succeeds in the scenarios that exposed the original failure.
Threats to Validity
One threat to construct validity concerns the use of binary execution outcomes and ROC-AUC to evaluate failure discrimination. Although these measures allow us to distinguish successful from failed trajectories, they do not establish whether highly suspicious voxels correspond to the actual causes of failure. Furthermore, the evaluation considers complete trajectories rather than measuring failure anticipation during execution. To mitigate this threat, we applied consistent outcome labels and scoring procedures across the compared techniques and evaluated their discrimination capability on held-out trajectories.
Regarding internal validity, the results may be influenced by the trajectory-generation strategies, spatial discretization, and selection of evaluation trajectories. In particular, the outcome-stratified sampling used in RQ5 may affect the observed transfer performance. To mitigate these threats, we evaluated the techniques on the same trajectories within each experimental configuration and separated training and evaluation through leave-one-run-out validation. Furthermore, we considered four trajectory-generation strategies and examined the influence of voxel size, training-set size, and execution coverage through sensitivity analyses in RQ3 and RQ4. For RQ5, spatial constraints were applied to ensure balanced coverage across the evaluation regions.
A potential threat to conclusion validity concerns the statistical interpretation of the experimental results. The experimental configurations share underlying factors, and the validation folds are not entirely independent, which may affect the interpretation of statistical significance. In addition, the reported confidence intervals are pointwise, and non-significant differences do not establish formal statistical equivalence. To mitigate these threats, we aggregated folds and scene budgets before performing paired statistical comparisons, applied exact Wilcoxon signed-rank tests with Holm correction, and reported effect sizes and confidence intervals where applicable. We also assessed the sensitivity of the bootstrap results and interpreted the cross-task differences in RQ5 descriptively.
Finally, external validity may be affected by the selection of VLA models, tasks, and environments. Our evaluation was limited to four VLA models and two robotic manipulation tasks in simulation. Consequently, the findings may not generalize to other VLA architectures, tasks, or environments. To mitigate this threat, we evaluated four different models from different architectural families across four trajectory-generation strategies and multiple experimental configurations, ensuring high diversity. We also examined cross-task transfer without retraining on the target task, providing an initial assessment of the generalizability beyond the original task. Another threat to external validity concerns the sim-to-real gap, as our experiments were conducted exclusively in simulation. Evaluating VoxelSBFL on a physical robot was not feasible because the required robotic platform was unavailable and reproducing thousands of trajectories would be impractical and costly. To partially mitigate this threat, we used SimplerEnv, whose authors reported a strong correlation between simulated and real-world performance of robotic manipulation policies [18].
Related Work
This section reviews prior work on testing and robustness evaluation of VLA models, failure prediction and uncertainty estimation in robotic policies, extensions of spectrum-based fault localization, and spatial representations for robotic manipulation. We position VoxelSBFL within this literature by highlighting its similarities to and differences from existing approaches.
Failure Prediction and Uncertainty Estimation in Robotic Policies
Failure prediction in learned robotic policies has been investigated using uncertainty metrics, execution-quality metrics, internal representations, and behavioral consistency. Valle et al. [29] studied eight uncertainty metrics and five quality metrics for VLA-enabled robots, evaluating their agreement with human assessments and their ability to distinguish successful from unsuccessful executions. Their metrics characterize token distributions, action instability, variability across repeated inferences, and trajectory quality. Their information requirements differ since token-based metrics require prediction probabilities, action and trajectory metrics use policy-robot interface signals, and the Optimal Trajectory metric additionally uses object and goal information. Similar to this approach, confidence metrics have also been used for intervention and calibration. Karli et al. [16] used token-level entropy and perplexity from a fine-tuned -FAST policy to request human corrections and investigate uncertainty-weighted shared control. Zollo and Zemel [38] examined confidence calibration over task execution and proposed prompt ensembles and action-wise Platt scaling. VoxelSBFL also assesses execution reliability, but learns spatial associations from TCP trajectories labeled with episode outcomes and scores unseen trajectories according to their visited regions. Its contribution is therefore a learned spatial representation of failure-associated behavior, whose percentiles express relative suspiciousness rather than calibrated failure probabilities. We directly compare this representation with three action-instability metrics and five quality metrics from Valle et al. [29] that are used as baseline metrics.
Other methods exploit policy representations or temporal behavior. For example, Gu et al. [11] proposed SAFE, learning a failure detector from internal VLA features and successful and failed rollouts, with conformal thresholds and evaluation on unseen tasks. Podolinsky et al. [24] introduced VLA-Tug, a recurrent probe trained on internal VLA representations with annotated failure onsets that predicts the likely failure mode and calibrates its alarm threshold to jointly control false alarms and missed early detections. Lyu et al. [19] introduced SentryVLA, combining conformalized quantile regression for scoring stochastic action candidates with Mahalanobis-distance monitoring of latent execution states. Bhardwaj et al. [4] introduced Progress Probes (), which train linear probes on internal VLA activations from successful demonstrations to estimate task progress. Their runtime monitor flags executions whose estimated progress falls behind the nominal progression expected from successful executions. Agia et al. [2] proposed Sentinel, combining Statistical Temporal Action Consistency with vision-language-model-based assessment of task progress. VoxelSBFL instead operates on externally recorded TCP positions, without requiring internal policy features, additional action sampling, or a vision-language monitor. Its map explicitly identifies physical regions associated with failed executions, although binary voxel coverage discards temporal ordering and visit frequency. These methods therefore provide methodological comparisons concerning representation and information requirements.
Testing and Robustness Evaluation of Vision-Language-Action Models
Existing studies investigate VLA reliability through simulation-based evaluation, systematic variation of execution conditions, and automated test generation. Li et al. [18] introduced SIMPLER, addressing visual and control discrepancies between real and simulated environments to support reproducible policy evaluation. Wang et al. [32] proposed VLATest, a fuzzing framework that varies target objects, confounding objects, lighting, and camera pose, while also examining unseen objects and instruction mutations. Peng et al. [23] introduced NEBULA, combining capability tests with stress tests to characterize policy capabilities and robustness beyond aggregate task success. Other approaches address test oracles and failure discovery. For example, Valle et al. [30] proposed metamorphic relations that compare robot trajectories under related inputs, alleviating the need for an explicit correctness oracle for each execution. For VLA testing, Kanwal et al. [14] proposed FATE-VLA, combining diversity-oriented exploration with surrogate predictions over scene configurations to discover diverse failures.
These approaches primarily support robustness evaluation, test input generation, and test-oracle design. Our work, in contrast, focuses on failure prediction by constructing spatial coverage spectra from robot trajectories and externally supplied episode outcomes. These spectra are used to compute suspiciousness scores for workspace voxels, identifying spatial regions associated with failed executions. The resulting suspiciousness map can then be used to anticipate failures and act on it by using specific guardrails (e.g., stopping the robot movement and asking for human intervention). Beyond failure prediction, this map may also support failure analysis and explanation by highlighting regions to prioritize for further investigation and debugging, which may help future VLA repair techniques (e.g., by generating valid trajectories that target highly suspicious voxels and afterward retraining it). However, highly suspicious voxels indicate statistical associations with failure rather than establishing its underlying causes.
Spectrum-Based Fault Localization and Its Extensions
Several empirical studies have investigated the effectiveness of SBFL techniques under different experimental conditions. Jones and Harrold [12] compared Tarantula with four fault-localization techniques and reported higher localization effectiveness on their subject programs. Abreu et al. [1] investigated similarity coefficients and the influence of the quantity and quality of execution observations, finding favorable results for Ochiai. Pearson et al. [22] revisited comparisons between spectrum-based and mutation-based techniques using artificial and real faults. Their results showed that conclusions obtained with artificial faults did not necessarily hold for real faults, highlighting the importance of the evaluation context when comparing localization techniques. In addition, theoretical studies provide complementary perspectives on suspiciousness formulas. Naish et al. [20] analyzed spectral ranking methods using a model of software diagnosis, including relationships between equivalent rankings. Xie et al. [34] investigated theoretical relationships between risk-evaluation formulas. Together with the broader survey by Wong et al. [33], these studies motivate examining established coefficients under different coverage representations rather than assuming that their relative performance transfers across contexts. These studies examine SBFL in the context of software fault localization. In contrast, our study adapts its coverage–outcome principle to the physical workspace of robotic manipulators. We recast the problem by treating workspace voxels as spectrum elements, TCP visitation as coverage, and episode success or failure as the execution outcome. This adaptation changes both the unit of analysis and the objective: instead of ranking potentially faulty code elements, we identify spatial regions associated with unsuccessful robotic executions and use their suspiciousness scores to predict failure in unseen trajectories.
SBFL has also been extended beyond conventional code-level bug localization. Arrieta et al. [3] ranked software-product-line features and constructed small valid products containing suspicious features to support fault isolation. Eniser et al. [10] introduced DeepFault, which applies suspiciousness measures to neuron activation spectra and uses gradients to generate inputs exercising suspicious neurons. Sun et al. [27] proposed DeepCover, a black-box approach that ranks image features through statistical fault localization to explain classifier decisions. Addressing the oracle problem, Xie et al. [35] introduced metamorphic slices, combining program slicing with violations of metamorphic relations instead of requiring pass/fail outcomes for individual test cases. These extensions demonstrate the applicability of spectrum-based reasoning beyond conventional program elements. To the best of our knowledge, however, our work is the first to investigate SBFL in the context of VLA models. This domain introduces a different analysis target: the physical behavior produced by a learned robotic policy, observed through trajectories and episode outcomes. VoxelSBFL introduces a spatial spectrum representation that connects this behavior to execution failures, enabling established suspiciousness techniques (e.g., Tarantula) to identify failure-associated workspace regions and score unseen trajectories. Our study investigates the effectiveness of this formulation across VLA models and examines its sensitivity to spatial resolution, training coverage, and transfer between tasks.
Conclusions and Future Work
In this paper, we introduced VoxelSBFL, a spatial adaptation of SBFL that allows for measuring suspiciousness levels of voxels in the robot’s workspace. By replacing code elements with workspace voxels, the approach connects recorded TCP trajectories and episode outcomes with suspiciousness scores. The resulting maps identify failure-associated regions and score unseen trajectories without requiring internal model access or explicit object and goal positions. We evaluated 34 spectrum-based techniques across four VLA models using 96,000 Pick-Up trajectories and compared the leading techniques against eight baseline metrics from [29]. Harmonic Mean and Rogot2 achieved the highest observed performance at the reference voxel size, with an overall macro ROC-AUC of 0.704, and significantly outperformed the evaluated action- and motion-instability baselines overall. Nevertheless, Optimal Trajectory achieved stronger discrimination using additional task-specific information, although this specific metric has several limitations for real-world applications. (i.e., it is task-specific and requires target object position information in operation, which is not available in most contexts). Our sensitivity analyses further showed that no voxel size was uniformly optimal across models and techniques. Increasing the training set size improved the reference technique’s discrimination, although the analysis did not establish an independent relationship between spatial coverage and performance. Moreover, transfer to Move-Near yielded weaker discrimination, highlighting the task dependence of the learned spatial associations. These findings support spatial spectra as a useful representation for VLA execution assessment while emphasizing that suspicious regions indicate associations with failures.
Our work opens several research avenues. An interesting direction that requires a deep adaptation of our approach is to extend VoxelSBFL to robots with multiple arms, including humanoids. We will compare a shared workspace suspiciousness map with separate maps for each arm. We hypothesize that arm-specific maps will better capture failure-associated regions because the two arms may differ in reachability and dexterity even where their workspaces overlap. A related question is how to define voxel coverage when a single pass/fail outcome applies to the entire execution. For example, if one arm misses a grasp while the other remains idle, counting visits from both arms equally could assign high suspiciousness to the idle arm’s position. We will investigate how to account for each arm’s role at different stages of the task, such as grasping, stabilizing an object, or remaining idle. We will also examine whether tracking which voxels the arms occupy simultaneously helps identify failures that arise during coordinated actions such as handovers. A second direction is to use suspiciousness maps to guide targeted test generation and demonstration collection for policy improvement. This will involve prioritizing execution contexts associated with highly suspicious regions and assessing whether the additional demonstrations support effective fine-tuning. Finally, we plan to quantify the sim-to-real gap by evaluating whether suspiciousness maps learned in simulation can anticipate failures on physical robots. We will apply the same maps to partial trajectories from matched simulated and physical executions and measure how often they flag failures before onset at a fixed false-alarm rate, as well as how much lead time they provide. Comparing these results with those obtained using maps learned from physical executions will help determine how much of the gap can be closed with real-world data.
Acknowledgments
Arusa Kanwal, Miren Ilarramendi, Shaukat Ali and Aitor Arrieta are supported by the InnoGuard Marie Skłodowska-Curie Doctoral Network (Grant Agreement No. 101169233). Pablo Valle is supported by the Pre-doctoral Program for the Formation of Non-Doctoral Research Staff of the Education Department of the Basque Government (Grant n. PRE_2025_2_0252). Aitor Arrieta and Miren Ilarramendi are partly supported by the Spanish Ministry of Science, Innovation and Universities (project PID2023-152979OA-I00), funded by MCIU /AEI /10.13039/501100011033/FEDER, UE. The authors are part of the Systems and Software Engineering group of Mondragon Unibertsitatea (IT1919-26), supported by the Department of Education, Universities and Research of the Basque Country.
References
References
- [1]Rui Abreu, Peter Zoeteweij, Rob Golstein, and Arjan JC Van Gemund. 2009. A practical evaluation of spectrum-based fault localization. Journal of Systems and Software 82, 11 (2009), 1780–1792.
- [2]Christopher Agia, Rohan Sinha, Jingyun Yang, Zi-ang Cao, Rika Antonova, Marco Pavone, and Jeannette Bohg. 2024. Unpacking failure modes of generative policies: Runtime monitoring of consistency and progress. arXiv preprint arXiv:2410.04640 (2024).arxiv.org/abs/2410.04640
- [3]Aitor Arrieta, Sergio Segura, Urtzi Markiegi, Goizuria Sagardui, and Leire Etxeberria. 2018. Spectrum-based fault localization in software product lines. Information and Software Technology 100 (2018), 18–31.DOI
- [4]Atiksh Bhardwaj, Edward W Duan, Junwon Seo, Pritwhish Dan, Wei-Chiu Ma, Andrea Bajscy, and Preston Culbertson. 2026. Progress Probes P²: Lightweight Runtime Monitoring for Vision-Language-Action Models. arXiv preprint arXiv:2610.31613 (2026).
- [5]Johan Björck, Fernando Castañeda, Nikita Cherniadev, Xingye Da, Runyu Ding, Linxi Fan, Yu Fang, Dieter Fox, Fengyuan Hu, Spencer Huang, et al. 2025. Gr00t n1: An open foundation model for generalist humanoid robots. arXiv preprint arXiv:2503.14734 (2025).arxiv.org/abs/2503.14734
- [6]Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. 2024. π0: A Vision-Language-Action Flow Model for General Robot Control. arXiv preprint arXiv:2410.24164 (2024).arxiv.org/abs/2410.24164
- [7]Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al. 2022. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817 (2022).arxiv.org/abs/2212.06817
- [8]Lingling Chen, Zongyao Lyu, and William J Beksi. 2026. ReconVLA: An Uncertainty-Guided and Failure-Aware Vision-Language-Action Framework for Robotic Control. arXiv preprint arXiv:2604.16677 (2026).
- [9]Shizhe Chen, Paul Pacaud, and Cordelia Schmid. 2026. Pointact: Vision-language-action models with multi-scale point-action interaction. arXiv preprint arXiv:2605.21414 (2026).arxiv.org/abs/2605.21414
- [10]Hasan Ferit Eniser, Simos Gerasimou, and Alper Sen. 2019. Deepfault: Fault localization for deep neural networks. In International conference on fundamental approaches to software engineering. Springer, 171–191.arxiv.org/abs/1902.05974
- [11]Qiao Gu, Yuanjiang Ju, Shengxiang Sun, Igor Gilchenstski, Haruki Nishimura, Masha Itkina, and Florian Shkurti. 2026. Safe: Multitask failure detection for vision-language-action models. Advances in Neural Information Processing Systems 38 (2026), 40041–40076.
- [12]James A Jones and Mary Jean Harrold. 2005. Empirical evaluation of the tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM international Conference on Automated software engineering. 273–282.DOI
- [13]James A Jones, Mary Jean Harrold, and John Stasko. 2002. Visualization of test information to assist fault localization. In Proceedings of the 24th international conference on Software engineering. 467–477.
- [14]Arusa Kanwal, Pablo Valle, Shaukat Ali, and Aitor Arrieta. 2026. FATE-VLA: Failure-aware test generation for vision-language-action models. arXiv preprint arXiv:2606.02307 (2026).arxiv.org/abs/2606.02307
- [15]Arusa Kanwal, Pablo Valle, Paola Narguero, Miren Illarramendi, Shaukat Ali, and Aitor Arrieta. 2026. Replication Package for VoxelSBFL. https://github.com/ArusaKanwal1225/VLA_SBFL_Coverage
- [16]Ulas Berk Karli, Tetsu Kurumisawa, and Tesca Fitzgerald. 2025. Ask before you act: Token-level uncertainty for intervention in vision-language-action models. In Second Workshop on Out-of-Distribution Generalization in Robotics at RSS 2025.
- [17]Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. 2024. Openvla: An open-source vision-language-action model. arXiv preprint arXiv:2406.09246 (2024).arxiv.org/abs/2406.09246
- [18]Xuanlin Li, Kyle Hsu, Jiayuan Gu, Karl Pertsch, Oier Mees, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. 2024. Evaluating Real-World Robot Manipulation Policies in Simulation. arXiv preprint arXiv:2405.05941 (2024).
- [19]Zongyao Lyu, Lingling Chen, and William J. Beksi. 2026. SentryVLA: Uncertainty-Guided Action Selection and State Monitoring for Vision-Language-Action Robotic Control. arXiv preprint arXiv:2604.16677 (2026).arxiv.org/abs/2604.16677
- [20]Lee Naish, Hua Jie Lee, and Kotagiri Ramamohanarao. 2011. A model for spectra-based software diagnosis. ACM Transactions on software engineering and methodology (TOSEM) 20, 3 (2011), 1–32.DOI
- [21]Abby O’Neill, Abdul Rehman, Abhiram Maddikuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. 2024. Open x-embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration 0. In 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 6892–6903.arxiv.org/abs/2310.08864
- [22]Spencer Pearson, José Campos, René Just, Gordon Fraser, Rui Abreu, Michael D Ernst, Deric Pang, and Benjamin Keller. 2017. Evaluating and improving fault localization. In 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 609–620.DOI
- [23]Jierui Peng, Yanyan Zhang, Yicheng Duan, Tuo Liang, Yipin Chaudhary, and Yu Yin. 2025. Nebula: Do we evaluate vision-language-action agents correctly? arXiv preprint arXiv:2510.16263 (2025).arxiv.org/abs/2510.16263
- [24]Michal P Podolinsky, Yunhao Yang, Daniel Little, Hong Lu, Rohan Siva, Olin Goudey, Neel P Bhatt, Matthias Scheutz, and Ufuk Topcu. 2026. Failure Prediction and Classification in Vision-Language-Action Models Via Calibrated Latent Probing. arXiv preprint arXiv:2611.08425 (2026).arxiv.org/abs/2611.08425
- [25]Delin Qu, Haoming Song, Qizhi Chen, Zhaoqing Chen, Xiangqiang Gao, Xinyi Ye, Qi Lv, Modi Shi, Guanghui Ren, Cheng Ruan, Maoging Yao, Haoran Yang, Jiacheng Bao, Bin Zhao, and Dong Wang. 2025. EO-1: Interleaved Vision-Text-Action Pretraining for General Robot Control. arXiv preprint arXiv:2508.21112 (2025).
- [26]Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, et al. 2025. SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model. arXiv preprint arXiv:2501.15830 (2025).
- [27]Youcheng Sun, Hana Chockler, Xiaowei Huang, and Daniel Kroening. 2020. Explaining image classifiers using statistical fault localization. In European conference on computer vision. Springer, 391–406.arxiv.org/abs/1908.02374
- [28]Pablo Valle, Shaukat Ali, Aitor Arrieta, and Lionel Briand. 2026. MANGO: Automated Multi-Agent Test Oracle Generation for Vision-Language-Action Models. arXiv preprint arXiv:2606.24815 (2026).arxiv.org/abs/2606.24815
- [29]Pablo Valle, Chenjie Lu, Shaukat Ali, and Aitor Arrieta. 2025. Evaluating uncertainty and quality of visual language action-enabled robots. arXiv preprint arXiv:2507.17049 (2025).arxiv.org/abs/2507.17049
- [30]Pablo Valle, Sergio Segura, Shaukat Ali, and Aitor Arrieta. 2026. Metamorphic Testing of Vision–Language Action–Enabled Robots. In 2026 IEEE International Conference on Software Testing, Verification and Validation (ICST). IEEE, 52–63.
- [31]An Dinh Vuong, Tuan Van Vo, Abdullah Sohail, Hoaran Ding, Liang Ma, Xiaodan Liang, Anqing Duan, Ivan Laptev, and Ian Reid. 2026. World2Act: Latent Action Post-Training from World Model Dynamics. arXiv preprint arXiv:2603.10422 (2026).arxiv.org/abs/2603.10422
- [32]Zhijie Wang, Zhehua Zhou, Jiayang Song, Yuheng Huang, Zhan Shu, and Lei Ma. 2025. VLATest: Testing and Evaluating Vision–Language-Action Models for Robotic Manipulation. Proceedings of ACM Software Engineering 2, Article FSE073 (July 2025). Issue FSE.DOI
- [33]W Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, and Franz Wotawa. 2016. A survey on software fault localization. IEEE Transactions on Software Engineering 42, 8 (2016), 707–740.
- [34]Xiaoyuan Xie, Tsong Yueh Chen, Fei-Ching Kuo, and Baowen Xu. 2013. A theoretical analysis of the risk evaluation formulas for spectrum-based fault localization. ACM Transactions on software engineering and methodology (TOSEM) 22, 4 (2013), 1–40.DOI
- [35]Xiaoyuan Xie, W Eric Wong, Tsong Yueh Chen, and Baowen Xu. 2013. Metamorphic slice: An application in spectrum-based fault localization. Information and Software Technology 55, 5 (2013), 866–879.DOI
- [36]Yifan Zhong, Fengshuo Bai, Shaofei Cai, Xuchuan Huang, Zhang Chen, Xiaowei Zhang, Yuanfei Wang, Shaoyang Guo, Tianrui Guan, Ka Nam Lui, et al. 2025. A survey on vision-language-action models: An action tokenization perspective. arXiv preprint arXiv:2507.01925 (2025).arxiv.org/abs/2507.01925
- [37]Tahereh Zohdinassab, Vincenzo Riccio, Alessio Gambi, and Paolo Tonella. 2021. Deephyperion: exploring the feature space of deep learning-based systems through illumination search. In Proceedings of the 30th ACM SIGSOFT international symposium on software testing and analysis. 79–90.
- [38]Thomas P Zollo and Richard Zemel. 2025. Confidence calibration in vision-language-action models. arXiv preprint arXiv:2507.17383 (2025).arxiv.org/abs/2507.17383
