betaFeedback

WorksComputer Science

Beyond Behaviour: A Stimulus-Response Safety Evaluation of Vision-Language-Action Models

Abstract

Humans exhibit biological sensitivity to safety-critical stimuli that can trigger defensive action. Across three models and two video benchmarks, we investigate whether similar responses emerge in vision-language-action models by examining their internal representations and action tendencies in response to visible hazards and harmful consequences. Our findings suggest that task execution may remain the default response even when danger is internally registered.

Introduction

Recent advances in vision-language-action (VLA) models have enabled robots to execute a wider range of manipulation tasks across different embodiments and environments [1, 2]. However, successful task completion does not necessarily imply safe execution. Existing safety evaluations examine whether VLAs violate safety constraints during execution [3, 4], refuse unsafe instructions [5], or detect unsafe actions in videos [6]. We here approach this problem from a fundamental perspective: why are humans sensitive to danger, and how does this sensitivity shape perception and action?

One biological foundation is the need to maintain physiological conditions compatible with survival. Homeostasis describes how organisms maintain internal physiological conditions within ranges necessary for survival, motivating responses to threats that could disrupt these conditions. Allostasis extends this view by emphasising anticipation of future demands and preparation for potential threats [7, 8]. From an evolutionary standpoint, the smoke detector principle suggests that defensive systems may favour false alarms when failing to detect a threat is substantially more costly than an unnecessary response [9]. Complementing these theories, human experiments show that unexpected events can recruit action-stopping mechanisms and transiently suppress motor output [10]. Together, these accounts connect safety to the processes that register potentially consequential stimuli and prepare an organism to respond.

Inspired by these viewpoints, we investigate a limited functional analogy in VLA policies: (1) do VLA models show internal sensitivity to safety-critical visual stimuli? and (2) do their predicted actions exhibit avoidance behaviour in the presence of danger? To the first question, we measure the level of atypicality in internal representations and their change relative to the baselines estimated from scenes in which harmful consequences are not yet visible. We examine these departures across the vision encoders, vision-language backbones, and action generation heads, comparing responses before and after the onset of harmful consequences. To the second question, we assess whether the initial predicted action moves the gripper of a robot arm towards or away from an estimated danger location. Our analysis therefore complements the existing safety evaluations by characterizing response signatures under safety-critical inputs, yet without assuming a biological equivalence or treating these signatures as evidence of danger recognition or safe behaviour.

Methodology

We address the two research questions through complementary analyses. The representational sensitivity study examines whether the internal representations of a VLA model depart from a calm baseline before and immediately after harmful consequences become visible. The avoidance behaviour study subsequently assesses whether the initial predicted action approaches or retreats from an estimated danger location.

Videos and temporal segments. The evaluation dataset consists of videos D={ve}e=1N\mathcal{D} = \{v_e\}_{e=1}^{N} depicting household scenarios in which a robot causes harm, for example by dropping a mug, starting a kitchen fire, or stabbing a person. We distinguish a hazard, a potential source of harm, from danger, defined here as the visible occurrence of its physical consequence. Each video contains sampled frames xe,tx_{e,t} and has an associated harmful-consequence onset time te⋆t_e^{\star}, marking when the consequence first becomes visible. To compare responses around the danger onset, we define τe,t=te⋆−t\tau_{e,t} = t_e^{\star} - t, which is positive before onset and negative afterwards, and suppress the clip index when unambiguous. We select three temporal groups of scenes relative to where the danger is present

calm: tmin⁡≤t<ce,near: 0<τ≤w−,just after: −w+<τ≤0.(1)\mathrm{calm}: \ t_{\min} \leq t < c_e,\qquad\mathrm{near}: \ 0 < \tau\leq w_-,\qquad\mathrm{just\ after}: \ -w_+ < \tau\leq0. \tag*{(1)}

Here, tmin⁡=0.5t_{\min} = 0.5 s is the earliest eligible timestamp, and the calm cutoff cec_e is taken from the dataset’s manual annotations. Calm frames may contain hazards but should not show their consequences. The window widths w−w_- and w+w_+ define the near group before onset and the just-after group at and immediately after onset, respectively.

Representational Sensitivity Analysis

A frozen VLA policy π\pi receives one frame xtx_t and a fixed instruction shared across all clips, without frame history or action feedback. At each selected layer ℓ\ell, we extract a hidden representation Htℓ∈RdℓH_t^\ell\in\mathbb{R}^{d_\ell}. We organise these extraction sites into three groups: vision, the encoder output before language projection; language, the fused vision–language backbone; and action generation, the action head at its final denoising step.

We quantify representational “surprise” with a transition surprise score QQ, which measures how atypical the change in a representation is between consecutive sampled frames, relative to the changes seen during calm frames. For a clip ee and sampled frame time tt (in seconds), let Htℓ∈RdℓH_t^\ell\in\mathbb{R}^{d_\ell} denote the representation at site ℓ\ell, i.e. a module output mapped to a vector. Each frame is processed independently, so HtℓH_t^\ell depends on frame tt alone. For a frame with a preceding sampled frame at t−t_- in the same clip, we define the time-normalized change δtℓ=(Htℓ−Ht−ℓ)/(t−t−)\delta_t^\ell= (H_t^\ell- H_{t_-}^\ell)/(t - t_-).

Let N\mathcal{N} denote the calm frames (e,t)(e,t) across all clips, i.e. frames before any harm is visible. To ensure that no clip is scored by a baseline fitted on its own frames, we use KK-fold cross-fitting, where clips are randomly partitioned into folds K1,…,KK\mathcal{K}_1,\ldots,\mathcal{K}_K. For fold kk, the baseline is fitted on

Fk={(e,t)∈N:(e,t−)∈N, e∉Kk},(2)\mathcal{F}_k = \{(e,t) \in\mathcal{N}: (e,t_-) \in\mathcal{N},\ e \notin\mathcal{K}_k\}, \tag*{(2)}

the changes between consecutive calm frames of clips outside the fold, and is then used to score every frame of the clips in Kk\mathcal{K}_k, including their calm frames. Let n=∣Fk∣n = |\mathcal{F}_k|. For each fold and site, we estimate the mean and coordinate variances of the calm changes:

μ^δ,jℓ=1n∑i∈Fkδi,jℓ,σ^δ,j2,ℓ=1n−1∑i∈Fk(δi,jℓ−μ^δ,jℓ)2,(3)\hat{\mu}_{\delta,j}^{\ell} = \frac{1}{n}\sum_{i \in\mathcal{F}_k}\delta_{i,j}^{\ell},\qquad \hat{\sigma}_{\delta,j}^{2,\ell} = \frac{1}{n-1}\sum_{i \in\mathcal{F}_k}\left(\delta_{i,j}^{\ell}-\hat{\mu}_{\delta,j}^{\ell}\right)^2, \tag*{(3)}

and use the regularized diagonal covariance Σ^δ,regℓ=diag⁡j(σ^δ,j2,ℓ+ϵ σ^δ2,ℓ)\widehat{\Sigma}_{\delta,\mathrm{reg}}^\ell= \operatorname{diag}_j\left(\hat{\sigma}_{\delta,j}^{2,\ell} + \epsilon\,\hat{\sigma}_{\delta}^{2,\ell}\right), where σ^δ2,ℓ=dℓ−1∑jσ^δ,j2,ℓ\hat{\sigma}_{\delta}^{2,\ell} = d_\ell^{-1}\sum_j\hat{\sigma}_{\delta,j}^{2,\ell} and ϵ\epsilon is a relative ridge coefficient. The diagonal form remains well defined when dℓd_\ell is comparable to or larger than nn, where a full sample covariance would be rank-deficient. Finally, we define QQ as the regularized Mahalanobis distance of a frame’s change from the calm changes:

Qtℓ=∥δtℓ−μ^δℓ∥(Σ^δ,regℓ)−1=(∑j=1dℓ(δt,jℓ−μ^δ,jℓ)2σ^δ,j2,ℓ+ϵ σ^δ2,ℓ)1/2,(4)Q_t^\ell= \left\|\delta_t^\ell-\hat{\mu}_{\delta}^\ell\right\|_{\left(\widehat{\Sigma}_{\delta,\mathrm{reg}}^\ell\right)^{-1}} = \left(\sum_{j=1}^{d_\ell}\frac{\left(\delta_{t,j}^\ell-\hat{\mu}_{\delta,j}^\ell\right)^2}{\hat{\sigma}_{\delta,j}^{2,\ell}+\epsilon\,\hat{\sigma}_{\delta}^{2,\ell}}\right)^{1/2}, \tag*{(4)}

where ∥v∥A=(vTAv)1/2\|v\|_A = (v^\mathsf{T}Av)^{1/2} and μ^δℓ\hat{\mu}_{\delta}^\ell, Σ^δ,regℓ\widehat{\Sigma}_{\delta,\mathrm{reg}}^\ell are those of the fold containing clip ee.

Avoidance Behaviour Analysis

In response to the second question, we examine whether the initial predicted action is directed towards or away from an estimated danger location. We evaluate the onset and final frames of each video to assess responses when the harmful consequence first becomes visible and after it has occurred. The video datasets do not directly provide the 3D geometry needed to measure gripper-danger distance. We therefore construct a digital cousin: a simplified simulated scene that reproduces the gripper-danger geometry estimated from image annotations and metric depth. This reconstruction provides a common spatial frame for evaluating how the initial predicted action changes that distance.

Let p0\mathbf{p}_0 be the initial gripper position, pend\mathbf{p}_{\mathrm{end}} the realized endpoint or predicted target, and q\mathbf{q} a fixed danger anchor at the location of the visible harmful consequence. For policies that predict delta actions, the relative changes in the gripper’s position or orientation, we execute the predicted command for a fixed interval of eight simulation steps (≈0.4\approx0.4 s) and use the resulting gripper position as pend\mathbf{p}_{\mathrm{end}}. For policies that predict absolute positions, we use the predicted position target directly.

Let Δd=∥pend−q∥2−∥p0−q∥2\Delta d = \lVert\mathbf{p}_{\mathrm{end}}-\mathbf{q}\rVert_2 - \lVert\mathbf{p}_0-\mathbf{q}\rVert_2. The metric of interest is the net reduction in gripper-danger distance RR, normalized by the gripper’s displacement magnitude: R=−Δd∥pend−p0∥2R = -\frac{\Delta d}{\lVert\mathbf{p}_{\mathrm{end}}-\mathbf{p}_0\rVert_2}. For non-zero displacement, R∈[−1,1]R \in[-1,1], in which positive values indicate approach and negative values indicate retreat. For zero displacement, we assign R=0R=0. To summarize behavioural tendencies, we group the actions into three descriptive bins: away (R<−0.5R < -0.5), sideways (∣R∣≤0.5|R| \le0.5), and straight at (R>0.5R > 0.5). Note that these labels summarise net point-to-point distance change rather than exact motion directions. Reconstruction and execution details are provided in Appendix § 5.§ 2.

Results and Key Findings

Experimental setup. We use 1,200 RoboShackles videos [5] and 306 HomeSafe-Bench videos [6] in which a hazard develops into a visible harmful consequence. We sample every second frame at 24 fps (about 0.1 s apart). VLA-JEPA [11], GR00T N1.7 [12], and X-VLA [13] are evaluated with LIBERO [14] and SimplerEnv [15] fine-tuned checkpoints. Each frame is paired with the instruction “complete the household task.” Avoidance behaviour evaluation only involves LIBERO checkpoints and the selected onset and final frames with visible grippers.

Internal sensitivity is most evident in vision and language. Compared with all calm frames (Figure 1, grey), the transition scores in the vision and language backbones rise as the consequence approaches and peaks at its onset in all six settings and both datasets, while action generation sites respond less (Figure 3 to Figure 6). This raw comparison is however ambiguous as calm frames come from the first seconds of a clip, hence later frames also reflect how far the generated video has drifted from its start, and the video’s own motion peaks at the same moment. Controlling for the same second into the clip removes the drift (Figure 1, blue). In HomeSafe-Bench, the pre-impact rise flattens to the calm level, while the peak at impact remains. In RoboShackles, the two curves nearly coincide and the rise before onset persists in all six settings. However, object motion persists in the same way, so timing alone does not separate a response to harm from a response to on-screen change.

Figure 1. Transition surprise at the vision encoder. Each curve measures the transition difference from calm frames: the QQ score of frames at time xx from the consequence minus the score of calm frames (Eq. (7)). Grey refers to the comparisons with all calm frames. Blue refers to the comparisons only with calm frames at the same second into the clip controlling for the progression of the video over time. Detailed metrics are presented in Appendix § 5.1.4.

Figure 3. HomeSafe-Bench: transition surprise QQ relative to calm frames along the time from the impact (marked zero on the x-axis), without matching: each time bin is compared with all calm frames. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the impact.

RoboShackles transition surprise plots at the same elapsed time for language, vision, and action-generation sites

Figure 6. RoboShackles: transition surprise QQ relative to calm frames along the time from the consequence onset (marked zero on the x-axis), at the same elapsed time: each time bin is compared only with calm frames at the same time into the clip; bins with too few matched frames are not drawn. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the consequence onset.

Sensitivity partly reflects the visual changes over time. Such potential confounding effect motivates the additional control for object motion and appearance change in Figure 7 and Figure 8, which reduces the vision and language QQ difference near the consequence to about a third. Thus, part of the measured sensitivity reflects the visual changes accompanying harmful events rather than necessarily their harmfulness. An exception is GR00T N1.7, whose language states remain significantly atypical after onset in both settings, given the same elapsed time, visual change and danger category.

Figure 7. HomeSafe-Bench: difference in QQ between groups of frames, per model and site group, without matching and under each control (same elapsed time; same image change). Columns: near vs calm, just after vs calm, just after vs near. Units: clip-to-clip spreads of the reference group (calm frames, or near frames in the last column); zero means no difference. Filled markers with thick lines are significant after Benjamini–Hochberg correction. HomeSafe-Bench has no joint elapsed-time and image-change control.

Figure 8. RoboShackles: difference in QQ between groups of frames, per model and site group, without matching and under each control (same elapsed time; same image change; same elapsed time and image change). Columns: near vs calm, just after vs calm, just after vs near. Units: clip-to-clip spreads of the reference group (calm frames, or near frames in the last column); zero means no difference. Filled markers with thick lines are significant after Benjamini–Hochberg correction.

Initial actions predominantly move towards danger. For the second question, the dominant initial response in the evaluated scenes is movement towards danger rather than withdrawal. This holds for all three models on RoboShackles and for GR00T N1.7 and VLA-JEPA on HomeSafe-Bench (Figure 2). X-VLA behaves differently on HomeSafe-Bench, moving towards, away from, or sideways relative to danger with similar frequency. GR00T N1.7 and VLA-JEPA tend to move towards danger across all categories, whereas the behaviour of X-VLA varies more with the type of danger (Figure 10 and Figure 11). Under the shared task instruction, these patterns suggest that task execution may remain the default response even when harmful consequences are visible.

Action direction relative to danger over the onset and final frames across categories

Figure 2. Action direction relative to danger over the onset and final frames across categories. (a) Median distance-reduction ratio RR with 95% frame-bootstrap confidence intervals; (b) proportion of frames across three types of behaviour.

HomeSafe-Bench action tendencies by danger level across four panels

Figure 10. HomeSafe-Bench: action tendencies by danger level (the higher the level, the more dangerous the event): L1 and L2 in the top row, and L3 and L4 in the bottom row.

Six-panel plots showing RoboShackles action tendencies by danger type

Figure 11. RoboShackles: action tendencies by danger type: hand and human in the top row; fire hazard and electrical hazard in the middle row; water safety and falling risk in the bottom row.

Conclusion

This paper examined VLA responses to safety-critical videos in terms of internal sensitivity to danger and avoidance behaviour in action generation. Internal responses were more pronounced in the vision and language backbones and were also influenced by visual changes and video progression. By contrast, safety-related signals remained comparatively weak in the action generation head, while initial actions predominantly moved towards danger under a fixed task instruction across most settings. Together, these findings suggest that heightened internal sensitivity can coexist with movement towards visible harm, while task execution may remain the default response even when safety-critical stimuli are internally registered. However, representational atypicality alone does not establish danger recognition, while action direction without calm-frame comparisons cannot establish intentional avoidance or danger-induced behavioural change.

Experimental Design

Tables 1 and 2 summarize the numerical settings for the representational and behavioural studies, respectively. Details on the experimental design are presented in the following sections.

SettingValue
Annotation image long side768 px
RoboShackles focal length526.7 px
HomeSafe-Bench focal lengthDepth Pro’s per-frame estimate
Replay steps JJ / frequency ν\nu8 / 20 Hz (0.4 s)
Controller position / rotation scales0.05 m / 0.5 rad per step
Marker extent cap0.2 m
Initial offset reach cap0.855 m
Placement tolerance / raised clearance0.02 m / 0.03 m
Scene centre in front of robot base0.55 m
Standing-layout centre height1.0 m
Assumed camera tilt for standing layouts29.9° downward
Directional cutoff β\beta0.5
Bootstrap resamples / interval level / seed2,000 / 95% / 0

Table 2. Settings for avoidance behaviour analysis. Controller scales apply to delta-action replay.

Representational Sensitivity Analysis

Videos, model inputs, and representation extraction

Video selection and sampling. We use 306 Veo-generated household videos from HomeSafe-Bench, covering four levels of harm severity: L1, L2, L3, L4. For RoboShackles, we use all 1,200 test videos: five-second robot-arm videos generated by Wan using initial images derived from DROID. The dataset contains 200 clips in each of six categories: hand, human, fire hazard, electrical hazard, water safety, and falling risk. The sampled frames for RoboShackles also include the final frame of each clip.

Model inputs. For each of the models VLA-JEPA, GR00T N1.7, and X-VLA, we evaluate one LIBERO checkpoint and one SimplerEnv checkpoint. Each frozen policy receives a single source frame and the instruction “complete the household task”, without frame history or action feedback. Robot state is omitted for VLA-JEPA and zero-filled for GR00T (with state dropout) and X-VLA. For LIBERO, the unavailable wrist view is omitted for VLA-JEPA, replaced with a blank view for GR00T, and masked out for X-VLA. We reset the random seed to zero before processing each frame to hold the flow-matching head’s initial noise fixed. These input conventions remain unchanged except in the X-VLA action evaluation described in Appendix § 5.§ 2.

Representation extraction. Each hidden representation HtℓH_t^\ell is extracted using a fixed token-selection and averaging rule. For vision, we use one site per model at the encoder’s final stage before the language projector, averaging over the image tokens. Language representations are extracted from every fourth backbone layer and the final layer, with averaging over all token positions. Action generation representations are extracted from the action heads at the final denoising step and averaged over action tokens. Checkpoints, preprocessing, and representation mappings are held fixed during baseline fitting and scoring. Predicted actions are used only in the avoidance behaviour study.

Temporal groups, onset annotation, and dataset partitions

Temporal groups. We use the temporal groups defined in Eq. (1), excluding calm frames from the near and just-after groups. These groups need not cover the entire clip. For RoboShackles, we also exclude frames with t<tmin⁡t < t_{\min} from the target groups. For HomeSafe-Bench, t∗t^* is the annotated impact time, while tintent,et_{\mathrm{intent},e} denotes the annotated intent onset. RoboShackles onset times are obtained using the procedure below.

RoboShackles onset annotation. All RoboShackles clips last five seconds, so treating the final frame as the event onset would confound proximity to the harmful consequence with elapsed clip time. We therefore estimate an onset time for each clip. Qwen3-VL-8B-Instruct [16] is prompted with a category-specific yes/no question for each frame. The estimated onset is the first time at or after 1.5 s at which three consecutive frames satisfy Pr⁡(harm)≥0.5\Pr(\mathrm{harm}) \ge0.5.

The 208 clips whose onsets remain undetermined, together with a random sample of 30 clips used to assess annotation accuracy, are annotated manually without access to the model outputs. Manual onset times are used for all 238 clips. In the accuracy sample, the estimated onsets are a median of 0.4 s later than the manual annotations and more than 1 s later in 10% of clips. Such timing discrepancies may blur the distinction between the near and just-after groups.

Dataset-preparation partitions. During dataset preparation, we reserve a fraction ρ\rho of eligible clips as donors, which supply calm frames for fixed fitting and calibration partitions. The remaining clips form the evaluation set E\mathcal{E}. For HomeSafe-Bench, donors are selected from clips with intent onset after 0 s and contribute only calm frames; near and just-after frames are drawn from E\mathcal{E}. For RoboShackles, donor clips are included when constructing all three temporal groups. Bootstrap resampling is performed separately within the fixed fitting, calibration, and evaluation partitions.

These preparation partitions are distinct from the cross-fitting used to obtain the reported scores. In cross-fitting, each baseline is fitted using calm frames from both donor and evaluation clips, while excluding every clip in the fold containing the clip being scored. Table 1 reports the partition sizes, the within-donor split, and the random seed used for preparation and fold assignment.

SettingSymbolHomeSafe-BenchRoboShackles
Frame groups and preparation
Calm starttmin⁡t_{\min}0.5 s0.5 s
Calm cutoffcec_etintent,et_{\mathrm{intent},e}te∗−1t_e^{*} - 1 s
Near / just-after window widthsw−,w+w_{-}, w_{+}2 s, 2 s1 s, 1 s
Donor fractionρ\rho0.200.20
Donor / evaluation clips—61/245240/960
Preparation and fold seedζ\zeta00
Baselines and controls
Cross-fitting foldsKK55
Relative covariance ridgeϵ\epsilon10−610^{-6}10−610^{-6}
Flow resolution / consistency tolerancerflow,ηflowr_{\mathrm{flow}}, \eta_{\mathrm{flow}}224 px, 1 px224 px, 1 px
Elapsed-time bin widthhth_t0.5 s0.5 s
Image-change binsbcb_c57
Time + image-change binsbcb_c—7
Category + time + image-change binsbcb_c—4
Minimum clips per group and stratummm55
Inference
Bootstrap resamples / interval levelB,1−αB, 1-\alpha2,000, 0.952,000, 0.95
Adjusted significance thresholdqq0.010.01
Coverage thresholdsκmin⁡,κlow\kappa_{\min}, \kappa_{\mathrm{low}}0.15, 0.250.15, 0.25
Bootstrap seedsRbootR_{\mathrm{boot}}{0,1,2}\{0, 1, 2\}{0,1,2}\{0, 1, 2\}

Table 1. Settings for the representational sensitivity analysis. Image-change bin counts apply separately to object motion and appearance change. A dash denotes an analysis not performed.

Control variables and matched comparisons

Higher surprise near a harmful consequence may also reflect elapsed clip time, image change, or danger category composition. We therefore compare near versus calm, just-after versus calm, and just-after versus near, both without matching and with separate or joint matching on these variables. In each contrast, the first group is the target TT and the second is the reference RR. Elapsed time and danger category are obtained from video timestamps and dataset labels, respectively. For image change, we distinguish camera-compensated object motion from residual appearance change that is not explained by motion.

Image change measurements. We analyze the combination of two aspects of visual change:

  • Object motion is optical-flow movement after removing the fitted camera motion. It is measured as the mean per-second magnitude of dense Farnebäck optical flow [17] on grayscale frames at resolution rflowr_{\mathrm{flow}}, after removing camera motion by fitting a 2D similarity transform to the flow correspondences using RANSAC [18];

  • Appearance change is pixel change that movement doesn’t explain, such as spills, fire or flashes. It is measured as the mean absolute per-second residual between the current frame and the preceding frame warped using backward flow, after removing the frame-wide brightness shift. Only pixels that pass a forward–backward consistency check with tolerance ηflow\eta_{\mathrm{flow}} contribute to the appearance-change measure.

Matching strata and configurations. Each control cc is defined by its strata. Elapsed-time strata are bins of width hth_t of time into the clip, and category strata are the dataset’s danger categories. Image-change strata cross bcb_c quantile bins of object motion with bcb_c quantile bins of appearance change, with bin boundaries fitted on T∪RT \cup R. Joint controls cross the corresponding strata, with the image-change bins then fitted within each elapsed-time bin (and each category).

For both datasets, we run separate controls for elapsed time, image change and danger category. Category matching uses the six RoboShackles categories and three HomeSafe-Bench danger types, with L1 and L2 combined because of their small sample sizes. For RoboShackles, we additionally match elapsed time and image change jointly, with and without the danger category. HomeSafe-Bench does not support these joint controls: its calm frames end at the hazard onset, about 1.8 s into the clip, too early to share later elapsed-time bins with the target frames. Bin widths and counts are listed in Table 1.

Detailed computation of metrics

Site-group summaries. We summarize QQ over groups of sites gg (i.e., vision encoder, vision–language backbone, action generation). The score of each site, denoted as ytgy_t^g, is first standardized over all frames of the run, so that no site dominates through its dimension, and the group’s sites are then averaged:

ytg=1∣g∣∑ℓ∈gQtℓ−mℓsℓ,(5)y_t^g = \frac{1}{|g|} \sum_{\ell\in g} \frac{Q_t^\ell- m_\ell}{s_\ell}, \tag*{(5)}

where mℓm_\ell and sℓs_\ell are the mean and standard deviation of QℓQ^\ell over all frames. The same comparisons below are applied to ygy^g and, as a model-free reference, to the video’s own object motion.

Groups of frames. Let te⋆t_e^\star be the time of the harmful consequence in clip ee and τ=te⋆−t\tau= t_e^\star- t the time to it (τ>0\tau> 0 before the consequence). We compare three groups of frames: near (0<τ<w0 < \tau< w, the last ww seconds before the consequence), just after (−w<τ≤0-w < \tau\le0), and the calm frames N\mathcal{N}, which are kept out of the other two. Here w=2w = 2 s for HomeSafe-Bench and w=1w = 1 s for RoboShackles.

Each comparison takes a target group TT and a reference group RR: the calm frames N\mathcal{N}, or the near frames when comparing just-after with near. Since frames of the same clip are not independent, each group is first averaged within each clip and then over clips:

yˉG=1∣EG∣∑e∈EG1∣Ge∣∑t∈Geyt,(6)\bar{y}_G = \frac{1}{|E_G|} \sum_{e \in E_G} \frac{1}{|G_e|} \sum_{t \in G_e} y_t, \tag*{(6)}

where EGE_G is the set of clips with frames in group GG and GeG_e those frames of clip ee. Frames near or after the consequence also differ from calm frames in nuisance properties, such as how far into the clip they are or how much the image changes.

A control cc holds one such property fixed by comparing the groups within strata ss of it, where yˉT,s\bar{y}_{T,s} and yˉR,s\bar{y}_{R,s} are the averages of Eq. (6) taken within stratum ss. We report

Dc(T,R)=Δc(T,R)σR,Δc(T,R)={yˉT−yˉR,c=raw,∑s∈ScnT,s(yˉT,s−yˉR,s)∑s∈ScnT,s,otherwise,(7)D_c(T,R)=\frac{\Delta_c(T,R)}{\sigma_R}, \qquad\Delta_c(T,R)= \begin{cases} \bar{y}_T-\bar{y}_R, & c=\mathrm{raw},\\ \frac{\sum_{s\in S_c} n_{T,s}\left(\bar{y}_{T,s}-\bar{y}_{R,s}\right)}{\sum_{s\in S_c}n_{T,s}}, & \text{otherwise}, \end{cases} \tag*{(7)}

where nT,sn_{T,s} is the number of target clips in stratum ss, ScS_c contains the strata of control cc with at least five clips in both groups, and σR\sigma_R is the standard deviation of the reference group’s per-clip means.

The matched difference thus standardizes the reference to the target’s distribution of the nuisance property, and the share of target frames falling in ScS_c (the coverage) is reported with it. Dividing by σR\sigma_R expresses DD in clip-to-clip spreads: D=0D=0 means no difference from the reference, and D=1D=1 means the groups differ by as much as reference clips typically differ from one another.

Figures. Figures 1, 3 to 6 trace DD along the time from the consequence, x=t−te∗x=t-t_e^*. The frames from 0.5 s into the clip, calm frames included, are cut into bins TbT_b of 0.25 s in xx, and each bin is compared with the calm frames: Draw(Tb,N)D_{\mathrm{raw}}(T_b,\mathcal{N}) shows how QQ evolves from calm through the consequence, and Dsame second(Tb,N)D_{\mathrm{same\ second}}(T_b,\mathcal{N}) how much of that evolution remains once frames are compared only with calm frames at the same point in the clip. Bins with fewer than 20 clips are omitted, as are matched bins with coverage below 15%.

Figures 7 and 8 report DcD_c for every control, with TT the whole near or just-after group and RR as above, giving one value per comparison and control.

Coverage. Coverage κ\kappa is the fraction of target frames that belong to usable strata, rather than the fraction of clips. We do not display contrasts whose coverage is below κmin⁡\kappa_{\min}, and we flag those with coverage between κmin⁡\kappa_{\min} and κlow\kappa_{\mathrm{low}} as low coverage. Because more restrictive matching can exclude different target frames, the population represented by a matched contrast may vary across control configurations.

Uncertainty and significance. Since frames of the same clip are not independent, we quantify uncertainty with a cluster bootstrap that resamples whole clips with replacement. Clips are resampled separately within each part of the fixed split, so that group sizes stay fixed, and the image-change bins are refitted on every resample. For each difference DcD_c, the two-sided pp-value is twice the smaller share of resampled differences on either side of zero. We run the bootstrap with three seeds of 2,000 resamples each and pool them: pp-values are computed from all 6,000 resamples, the 95% intervals of the group comparisons (Figures 7 and 8) are the average of the percentile intervals over the three seeds, and the bands of the curves (Figure 1) are percentile intervals over all 6,000 resamples.

To account for the many comparisons, we apply the Benjamini–Hochberg correction [19] separately for each model and site group. Each such family contains every computable test of that model and site group: SS and QQ, the three comparisons (near vs calm, just-after vs calm, just-after vs near), and the raw difference and each control. A difference is called significant when its adjusted pp-value is below 0.01.

Figure 4. HomeSafe-Bench: transition surprise QQ relative to calm frames along the time from the impact (marked zero on the x-axis), at the same elapsed time: each time bin is compared only with calm frames at the same time into the clip; bins with too few matched frames are not drawn. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the impact.

Figure 5. RoboShackles: transition surprise QQ relative to calm frames along the time from the consequence onset (marked zero on the x-axis), without matching: each time bin is compared with all calm frames. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the consequence onset.

Avoidance Behaviour Analysis

Frame selection and annotation

Candidate frames. We define the onset frame as the first sampled frame at or after the harmful-consequence time. For RoboShackles, we select 1,200 onset frames from all clips and 960 final frames from the evaluation clips. Support-plane estimation fails for three frames, leaving 2,157 scenes. For HomeSafe-Bench, the evaluation clips provide 174 onset frames and 245 final frames; in the remaining 71 clips, impact occurs after the last sampled frame. Manual review excludes 20 frames, leaving 399 scenes.

Measured-scene filter. To be included in the reported results, a scene must use a gripper location identified in the source frame itself and preserve the estimated gripper–danger offset without adjustment. We apply the same exclusion criteria to all models, removing standing-board layouts, scenes with assumed, borrowed, or manually placed grippers, and scenes requiring placement adjustments. Table 3 lists the exclusion reasons, which may overlap. The remaining data comprises 815 of the 2,157 RoboShackles frames (569 onset and 246 final) and 166 of the 399 HomeSafe-Bench frames (78 onset and 88 final). This filter favours frames with visible grippers within the robot’s reach and excludes most RoboShackles frames in the human category. Here, “measured” refers to the reconstruction criterion, not ground-truth geometry.

ReasonRoboShacklesHomeSafe-Bench
Standing-board layout2600
Assumed gripper4943
Gripper out of view (hand-placed)032
Gripper at the danger (hand-placed)013
Gripper borrowed from the other frame33041
Raised placement23580
Reach-capped or clamped offset46140
Height moved (floor)243113
Support label contradicts height7955
Excluded / included1,342 / 815233 / 166

Table 3. Action-evaluation exclusions after annotation and support-plane screening. Reasons may overlap; the final row gives unique excluded and retained frame counts.

Image annotations and depth. Using greedy decoding, Qwen3-VL-8B-Instruct identifies the danger region, gripper fingers, fingertip location, and support type. For HomeSafe-Bench, the prompt specifies that the gripper nearest the danger should be selected. Six invalid RoboShackles annotations and the HomeSafe-Bench annotations flagged during review were corrected manually. Images used for annotation have a long side of 768 pixels.

These annotations locate the gripper and danger in the image, but reconstructing their relative geometry requires estimates of their three-dimensional positions at a metric scale. We therefore use Depth Pro [20] to estimate metric depth from each source frame. Together with the focal length, these depth estimates allow the annotated image locations to be mapped to three-dimensional gripper and danger points in source-view coordinates. Their relative offset is then used to position the gripper and danger marker in the digital cousin. The focal length is fixed at 526.7 pixels for RoboShackles and estimated separately for each HomeSafe-Bench frame by Depth Pro.

Reconstructing gripper–danger geometry

Geometry in source-view coordinates. We use metric depth and focal length to map the image annotations to a gripper point gg and a danger point ss in three-dimensional source-view coordinates. Gripper depth is estimated from the fingers rather than from the potentially empty space between them. For supported danger regions, ss is the nearest point at the region’s base; for injuries on standing people, it is the nearest surface point.

An estimated support plane defines the upward direction n\mathbf{n}. Projecting the camera’s horizontal axis onto this plane gives r\mathbf{r}, and we define ϕ=n×r\boldsymbol{\phi} = \mathbf{n} \times\mathbf{r}. The gripper-to-danger offset expressed in these view directions is

o=[(s−g)Tr(s−g)Tϕ(s−g)Tn].(8)\mathbf{o} = \begin{bmatrix} (\mathbf{s}-\mathbf{g})^{\mathsf{T}}\mathbf{r} \\ (\mathbf{s}-\mathbf{g})^{\mathsf{T}}\boldsymbol{\phi} \\ (\mathbf{s}-\mathbf{g})^{\mathsf{T}}\mathbf{n} \end{bmatrix}. \tag*{(8)}

We transfer this offset to the simulator’s corresponding view directions to position the robot relative to a non-colliding danger marker. This transfer assumes that the policy interprets the source image as its own main-camera view; the true source-camera-to-robot transformation is unknown. The digital cousin reconstructs the relative geometry, not the complete source scene.

Simulator and placement. We construct simplistic scenes using LIBERO’s LIBERO_goal suite, with the task objects removed, using robosuite v1.4.1 [21] and MuJoCo v2.3.7 [22]. The robot base remains fixed, and inverse kinematics positions the gripper with a fixed downward-pointing orientation. The danger anchor q\mathbf{q} is the centre of the marker’s base. Injuries on standing people are represented by an upright board centred on q\mathbf{q}; when no support plane is visible, we assume an upward direction. During scene construction, unreachable placements are adjusted, and missing gripper locations are either borrowed from the clip’s other frame or specified by rule. We record all placement adjustments and assumed geometry, and exclude scenes requiring either from the reported results. For each frame, all models use the same initial scene.

Action interfaces and endpoint evaluation

Delta-action policies. We use LIBERO checkpoints for the action study. For GR00T N1.7 and VLA-JEPA, we reuse the first action ae,t\mathbf{a}_{e,t} of the predicted chunk recorded for each source frame during the representational study. GR00T’s actions are already unnormalised, whereas VLA-JEPA’s actions are clipped and unnormalised using the checkpoint’s minimum and maximum statistics. We convert the actions to the simulator’s controller units and map each model’s gripper convention to the LIBERO controller. Starting from the reconstructed scene, we hold this first command for J=8J = 8 control steps at ν=20\nu= 20 Hz (0.4 s) and record the gripper position after each step.

Absolute-target policy. Because X-VLA predicts absolute targets, we query it again using the original source image, the fixed instruction, and the reconstructed end-effector pose as robot state, still without a wrist view. We evaluate the first position target in the predicted chunk directly as pend\mathbf{p}_{\mathrm{end}}.

Avoidance behaviour by types and levels of danger.

The following figures present the category-specific breakdowns of the behaviour by categories in Figure 2. Figure 10 shows the four HomeSafe-Bench danger levels, and Figure 11 shows the six RoboShackles categories. For each category and model, panel (a) reports the median distance-reduction ratio RR with its 95% frame-bootstrap confidence interval, while panel (b) shows the proportions of actions in the away, sideways, and straight-at bins.

Examples of reconstructed digital-cousin scenes in LIBERO

Figure 9. Examples of reconstructed digital-cousin scenes in LIBERO. Each row shows a source frame with the annotated gripper and danger region (left), followed by the corresponding reconstructed scene from multiple simulator views (right). In the reconstruction, the danger location is represented by a simplified red box whose placement preserves the estimated gripper–danger offset recovered from image annotations and metric depth. The robot pose is initialized to approximately match the annotated frame, while the surrounding environment is reduced to a minimal scene sufficient for evaluating the direction of the initial predicted action.

References

  1. [1]Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E. P., Sanketi, P. R., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., & Finn, C. (2025). OpenVLA: An open-source vision-language-action model. Proceedings of Machine Learning Research, 270, 2679–2713. https://proceedings.mlr.press/v270/kim25c.htmlproceedings.mlr.press/v270/kim25c.html
  2. [2]Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M. R., Finn, C., Fusai, N., Galliker, M. Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., … Zhilinsky, U. (2025). π₀.₅: A vision-language-action model with open-world generalization. Proceedings of Machine Learning Research, 305, 17–40. https://proceedings.mlr.press/v305/black25a.htmlproceedings.mlr.press/v305/black25a.html
  3. [3]B. Zhang, Y. Zhang, J. Ji, Y. Lei, Y. Cai, J. Dai, Y. Chen, and Y. Yang. SafeVLA: Towards safety alignment of vision-language-action model via constrained learning, 2025. URL https://arxiv.org/abs/2503.03480.
  4. [4]M. Lyu, Y. Sun, Y. Jia, S. Shen, M. Sha, H. Li, F. Zhao, and Y. Zeng. ForesightSafety-VLA: A unified diagnostic safety benchmark for vision-language-action models, 2026. URL https://arxiv.org/abs/2606.27079.
  5. [5]Z. Yin, C. Liu, W. Yang, R. Li, Y. Guo, and Y. Xue. RoboShackles: A multilingual safety dataset for human-injury prevention in embodied foundation models, 2026. URL https://arxiv.org/abs/2606.18632v2.
  6. [6]J. Pu, Z. Sun, Z. Zhang, X. Zhang, and J. Xu. HomeSafe-Bench: Evaluating vision-language models on unsafe action detection for embodied agents in household scenarios, 2026. URL https://arxiv.org/abs/2603.11975.
  7. [7]K. Friston. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11(2):127–138, 2010. doi:10.1038/nrn2787.DOI
  8. [8]P. Sterling. Allostasis: A model of predictive regulation. Physiology & Behavior, 106(1):5–15, 2012. doi:10.1016/j.physbeh.2011.06.004.DOI
  9. [9]R. M. Nesse. The smoke detector principle: Signal detection and optimal defense regulation. Evolution, Medicine, and Public Health, 2019(1):1, 2019. doi:10.1093/emph/eoy034.DOI
  10. [10]J. R. Wessel and A. R. Aron. Unexpected events induce motor slowing via a brain mechanism for action-stopping with global suppressive effects. Journal of Neuroscience, 33(47):18481–18491, 2013. doi:10.1523/JNEUROSCI.3456-13.2013.DOI
  11. [11]Sun, J., Zhang, W., Qi, Z., Ren, S., Liu, Z., Zhu, H., Sun, G., Jin, X., & Chen, Z. (2026). V1a-jepa: Enhancing vision-language-action model with latent world model. In European Conference on Computer Vision (pp. 478–497).
  12. [12]Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., & al., . et . (2025). Gr00t n1: An open foundation model for generalist humanoid robots. In arXiv preprint arXiv:2503.14734. 2503.14734
  13. [13]Zheng, J., Li, J., Wang, Z., Liu, D., Kang, X., Feng, Y., Zheng, Y., Zou, J., Chen, Y., Zeng, J., & al., . et . (2026). X-vla: Soft-prompted transformer as scalable cross-embodiment vision-language-action model. International Conference on Learning Representations, 2026, 60580–60606.
  14. [14]B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.arXiv
  15. [15]Li, X., Hsu, K., Gu, J., Pertsch, K., Mees, O., Walke, H. R., Fu, C., Lunawat, I., Sieh, I., Kirmani, S., Levine, S., Wu, J., Finn, C., Su, H., Vuong, Q., & Xiao, T. (2024). Evaluating real-world robot manipulation policies in simulation. In arXiv preprint arXiv:2405.05941. arXiv:2405.05941
  16. [16]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., … Zhu, K. (2025). Qwen3-vl technical report. In arXiv preprint arXiv:2511.21631. arXiv:2511.21631
  17. [17]G. Farnebäck. Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis, pages 363–370. Springer, 2003.DOI
  18. [18]Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.
  19. [19]Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289–300, 1995.DOI
  20. [20]Bochkovskiy, A., Delaunoy, A., Germain, H., Santos, M., Zhou, Y., Richter, S., & Koltun, V. (2025). Depth pro: Sharp monocular metric depth in less than a second. International Conference on Learning Representations, 2025, 75602–75637. https://proceedings.iclr.cc/paper_files/paper/2025/file/bc8b2058fd96978a4146f18298cb2d39-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2025/file/bc8b2058fd96978a4146f18298cb2d39-Paper-Conference.pdf
  21. [21]Y. Zhu, J. Wong, A. Mandlekar, R. Martín-Martín, A. Joshi, S. Nasiriany, Y. Zhu, and K. Lin. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12293, 2020.arXiv
  22. [22]E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109.DOI

Paper details

vlnc:W_8kg2jtjx
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents