WorksComputer Science
Beyond Behaviour: A Stimulus-Response Safety Evaluation of Vision-Language-Action Models
Abstract
Humans exhibit biological sensitivity to safety-critical stimuli that can trigger defensive action. Across three models and two video benchmarks, we investigate whether similar responses emerge in vision-language-action models by examining their internal representations and action tendencies in response to visible hazards and harmful consequences. Our findings suggest that task execution may remain the default response even when danger is internally registered.
Introduction
Recent advances in vision-language-action (VLA) models have enabled robots to execute a wider range of manipulation tasks across different embodiments and environments [1, 2]. However, successful task completion does not necessarily imply safe execution. Existing safety evaluations examine whether VLAs violate safety constraints during execution [3, 4], refuse unsafe instructions [5], or detect unsafe actions in videos [6]. We here approach this problem from a fundamental perspective: why are humans sensitive to danger, and how does this sensitivity shape perception and action?
One biological foundation is the need to maintain physiological conditions compatible with survival. Homeostasis describes how organisms maintain internal physiological conditions within ranges necessary for survival, motivating responses to threats that could disrupt these conditions. Allostasis extends this view by emphasising anticipation of future demands and preparation for potential threats [7, 8]. From an evolutionary standpoint, the smoke detector principle suggests that defensive systems may favour false alarms when failing to detect a threat is substantially more costly than an unnecessary response [9]. Complementing these theories, human experiments show that unexpected events can recruit action-stopping mechanisms and transiently suppress motor output [10]. Together, these accounts connect safety to the processes that register potentially consequential stimuli and prepare an organism to respond.
Inspired by these viewpoints, we investigate a limited functional analogy in VLA policies: (1) do VLA models show internal sensitivity to safety-critical visual stimuli? and (2) do their predicted actions exhibit avoidance behaviour in the presence of danger? To the first question, we measure the level of atypicality in internal representations and their change relative to the baselines estimated from scenes in which harmful consequences are not yet visible. We examine these departures across the vision encoders, vision-language backbones, and action generation heads, comparing responses before and after the onset of harmful consequences. To the second question, we assess whether the initial predicted action moves the gripper of a robot arm towards or away from an estimated danger location. Our analysis therefore complements the existing safety evaluations by characterizing response signatures under safety-critical inputs, yet without assuming a biological equivalence or treating these signatures as evidence of danger recognition or safe behaviour.
Methodology
We address the two research questions through complementary analyses. The representational sensitivity study examines whether the internal representations of a VLA model depart from a calm baseline before and immediately after harmful consequences become visible. The avoidance behaviour study subsequently assesses whether the initial predicted action approaches or retreats from an estimated danger location.
Videos and temporal segments. The evaluation dataset consists of videos depicting household scenarios in which a robot causes harm, for example by dropping a mug, starting a kitchen fire, or stabbing a person. We distinguish a hazard, a potential source of harm, from danger, defined here as the visible occurrence of its physical consequence. Each video contains sampled frames and has an associated harmful-consequence onset time , marking when the consequence first becomes visible. To compare responses around the danger onset, we define , which is positive before onset and negative afterwards, and suppress the clip index when unambiguous. We select three temporal groups of scenes relative to where the danger is present
Here, s is the earliest eligible timestamp, and the calm cutoff is taken from the dataset’s manual annotations. Calm frames may contain hazards but should not show their consequences. The window widths and define the near group before onset and the just-after group at and immediately after onset, respectively.
Representational Sensitivity Analysis
A frozen VLA policy receives one frame and a fixed instruction shared across all clips, without frame history or action feedback. At each selected layer , we extract a hidden representation . We organise these extraction sites into three groups: vision, the encoder output before language projection; language, the fused vision–language backbone; and action generation, the action head at its final denoising step.
We quantify representational “surprise” with a transition surprise score , which measures how atypical the change in a representation is between consecutive sampled frames, relative to the changes seen during calm frames. For a clip and sampled frame time (in seconds), let denote the representation at site , i.e. a module output mapped to a vector. Each frame is processed independently, so depends on frame alone. For a frame with a preceding sampled frame at in the same clip, we define the time-normalized change .
Let denote the calm frames across all clips, i.e. frames before any harm is visible. To ensure that no clip is scored by a baseline fitted on its own frames, we use -fold cross-fitting, where clips are randomly partitioned into folds . For fold , the baseline is fitted on
the changes between consecutive calm frames of clips outside the fold, and is then used to score every frame of the clips in , including their calm frames. Let . For each fold and site, we estimate the mean and coordinate variances of the calm changes:
and use the regularized diagonal covariance , where and is a relative ridge coefficient. The diagonal form remains well defined when is comparable to or larger than , where a full sample covariance would be rank-deficient. Finally, we define as the regularized Mahalanobis distance of a frame’s change from the calm changes:
where and , are those of the fold containing clip .
Avoidance Behaviour Analysis
In response to the second question, we examine whether the initial predicted action is directed towards or away from an estimated danger location. We evaluate the onset and final frames of each video to assess responses when the harmful consequence first becomes visible and after it has occurred. The video datasets do not directly provide the 3D geometry needed to measure gripper-danger distance. We therefore construct a digital cousin: a simplified simulated scene that reproduces the gripper-danger geometry estimated from image annotations and metric depth. This reconstruction provides a common spatial frame for evaluating how the initial predicted action changes that distance.
Let be the initial gripper position, the realized endpoint or predicted target, and a fixed danger anchor at the location of the visible harmful consequence. For policies that predict delta actions, the relative changes in the gripper’s position or orientation, we execute the predicted command for a fixed interval of eight simulation steps ( s) and use the resulting gripper position as . For policies that predict absolute positions, we use the predicted position target directly.
Let . The metric of interest is the net reduction in gripper-danger distance , normalized by the gripper’s displacement magnitude: . For non-zero displacement, , in which positive values indicate approach and negative values indicate retreat. For zero displacement, we assign . To summarize behavioural tendencies, we group the actions into three descriptive bins: away (), sideways (), and straight at (). Note that these labels summarise net point-to-point distance change rather than exact motion directions. Reconstruction and execution details are provided in Appendix § 5.§ 2.
Results and Key Findings
Experimental setup. We use 1,200 RoboShackles videos [5] and 306 HomeSafe-Bench videos [6] in which a hazard develops into a visible harmful consequence. We sample every second frame at 24 fps (about 0.1 s apart). VLA-JEPA [11], GR00T N1.7 [12], and X-VLA [13] are evaluated with LIBERO [14] and SimplerEnv [15] fine-tuned checkpoints. Each frame is paired with the instruction “complete the household task.” Avoidance behaviour evaluation only involves LIBERO checkpoints and the selected onset and final frames with visible grippers.
Internal sensitivity is most evident in vision and language. Compared with all calm frames (Figure 1, grey), the transition scores in the vision and language backbones rise as the consequence approaches and peaks at its onset in all six settings and both datasets, while action generation sites respond less (Figure 3 to Figure 6). This raw comparison is however ambiguous as calm frames come from the first seconds of a clip, hence later frames also reflect how far the generated video has drifted from its start, and the video’s own motion peaks at the same moment. Controlling for the same second into the clip removes the drift (Figure 1, blue). In HomeSafe-Bench, the pre-impact rise flattens to the calm level, while the peak at impact remains. In RoboShackles, the two curves nearly coincide and the rise before onset persists in all six settings. However, object motion persists in the same way, so timing alone does not separate a response to harm from a response to on-screen change.
Figure 1. Transition surprise at the vision encoder. Each curve measures the transition difference from calm frames: the score of frames at time from the consequence minus the score of calm frames (Eq. (7)). Grey refers to the comparisons with all calm frames. Blue refers to the comparisons only with calm frames at the same second into the clip controlling for the progression of the video over time. Detailed metrics are presented in Appendix § 5.1.4.
Figure 3. HomeSafe-Bench: transition surprise relative to calm frames along the time from the impact (marked zero on the x-axis), without matching: each time bin is compared with all calm frames. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the impact.

Figure 6. RoboShackles: transition surprise relative to calm frames along the time from the consequence onset (marked zero on the x-axis), at the same elapsed time: each time bin is compared only with calm frames at the same time into the clip; bins with too few matched frames are not drawn. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the consequence onset.
Sensitivity partly reflects the visual changes over time. Such potential confounding effect motivates the additional control for object motion and appearance change in Figure 7 and Figure 8, which reduces the vision and language difference near the consequence to about a third. Thus, part of the measured sensitivity reflects the visual changes accompanying harmful events rather than necessarily their harmfulness. An exception is GR00T N1.7, whose language states remain significantly atypical after onset in both settings, given the same elapsed time, visual change and danger category.
Figure 7. HomeSafe-Bench: difference in between groups of frames, per model and site group, without matching and under each control (same elapsed time; same image change). Columns: near vs calm, just after vs calm, just after vs near. Units: clip-to-clip spreads of the reference group (calm frames, or near frames in the last column); zero means no difference. Filled markers with thick lines are significant after Benjamini–Hochberg correction. HomeSafe-Bench has no joint elapsed-time and image-change control.
Figure 8. RoboShackles: difference in between groups of frames, per model and site group, without matching and under each control (same elapsed time; same image change; same elapsed time and image change). Columns: near vs calm, just after vs calm, just after vs near. Units: clip-to-clip spreads of the reference group (calm frames, or near frames in the last column); zero means no difference. Filled markers with thick lines are significant after Benjamini–Hochberg correction.
Initial actions predominantly move towards danger. For the second question, the dominant initial response in the evaluated scenes is movement towards danger rather than withdrawal. This holds for all three models on RoboShackles and for GR00T N1.7 and VLA-JEPA on HomeSafe-Bench (Figure 2). X-VLA behaves differently on HomeSafe-Bench, moving towards, away from, or sideways relative to danger with similar frequency. GR00T N1.7 and VLA-JEPA tend to move towards danger across all categories, whereas the behaviour of X-VLA varies more with the type of danger (Figure 10 and Figure 11). Under the shared task instruction, these patterns suggest that task execution may remain the default response even when harmful consequences are visible.

Figure 2. Action direction relative to danger over the onset and final frames across categories. (a) Median distance-reduction ratio with 95% frame-bootstrap confidence intervals; (b) proportion of frames across three types of behaviour.

Figure 10. HomeSafe-Bench: action tendencies by danger level (the higher the level, the more dangerous the event): L1 and L2 in the top row, and L3 and L4 in the bottom row.

Figure 11. RoboShackles: action tendencies by danger type: hand and human in the top row; fire hazard and electrical hazard in the middle row; water safety and falling risk in the bottom row.
Conclusion
This paper examined VLA responses to safety-critical videos in terms of internal sensitivity to danger and avoidance behaviour in action generation. Internal responses were more pronounced in the vision and language backbones and were also influenced by visual changes and video progression. By contrast, safety-related signals remained comparatively weak in the action generation head, while initial actions predominantly moved towards danger under a fixed task instruction across most settings. Together, these findings suggest that heightened internal sensitivity can coexist with movement towards visible harm, while task execution may remain the default response even when safety-critical stimuli are internally registered. However, representational atypicality alone does not establish danger recognition, while action direction without calm-frame comparisons cannot establish intentional avoidance or danger-induced behavioural change.
Experimental Design
Tables 1 and 2 summarize the numerical settings for the representational and behavioural studies, respectively. Details on the experimental design are presented in the following sections.
| Setting | Value |
| Annotation image long side | 768 px |
| RoboShackles focal length | 526.7 px |
| HomeSafe-Bench focal length | Depth Pro’s per-frame estimate |
| Replay steps / frequency | 8 / 20 Hz (0.4 s) |
| Controller position / rotation scales | 0.05 m / 0.5 rad per step |
| Marker extent cap | 0.2 m |
| Initial offset reach cap | 0.855 m |
| Placement tolerance / raised clearance | 0.02 m / 0.03 m |
| Scene centre in front of robot base | 0.55 m |
| Standing-layout centre height | 1.0 m |
| Assumed camera tilt for standing layouts | 29.9° downward |
| Directional cutoff | 0.5 |
| Bootstrap resamples / interval level / seed | 2,000 / 95% / 0 |
Table 2. Settings for avoidance behaviour analysis. Controller scales apply to delta-action replay.
Representational Sensitivity Analysis
Videos, model inputs, and representation extraction
Video selection and sampling. We use 306 Veo-generated household videos from HomeSafe-Bench, covering four levels of harm severity: L1, L2, L3, L4. For RoboShackles, we use all 1,200 test videos: five-second robot-arm videos generated by Wan using initial images derived from DROID. The dataset contains 200 clips in each of six categories: hand, human, fire hazard, electrical hazard, water safety, and falling risk. The sampled frames for RoboShackles also include the final frame of each clip.
Model inputs. For each of the models VLA-JEPA, GR00T N1.7, and X-VLA, we evaluate one LIBERO checkpoint and one SimplerEnv checkpoint. Each frozen policy receives a single source frame and the instruction “complete the household task”, without frame history or action feedback. Robot state is omitted for VLA-JEPA and zero-filled for GR00T (with state dropout) and X-VLA. For LIBERO, the unavailable wrist view is omitted for VLA-JEPA, replaced with a blank view for GR00T, and masked out for X-VLA. We reset the random seed to zero before processing each frame to hold the flow-matching head’s initial noise fixed. These input conventions remain unchanged except in the X-VLA action evaluation described in Appendix § 5.§ 2.
Representation extraction. Each hidden representation is extracted using a fixed token-selection and averaging rule. For vision, we use one site per model at the encoder’s final stage before the language projector, averaging over the image tokens. Language representations are extracted from every fourth backbone layer and the final layer, with averaging over all token positions. Action generation representations are extracted from the action heads at the final denoising step and averaged over action tokens. Checkpoints, preprocessing, and representation mappings are held fixed during baseline fitting and scoring. Predicted actions are used only in the avoidance behaviour study.
Temporal groups, onset annotation, and dataset partitions
Temporal groups. We use the temporal groups defined in Eq. (1), excluding calm frames from the near and just-after groups. These groups need not cover the entire clip. For RoboShackles, we also exclude frames with from the target groups. For HomeSafe-Bench, is the annotated impact time, while denotes the annotated intent onset. RoboShackles onset times are obtained using the procedure below.
RoboShackles onset annotation. All RoboShackles clips last five seconds, so treating the final frame as the event onset would confound proximity to the harmful consequence with elapsed clip time. We therefore estimate an onset time for each clip. Qwen3-VL-8B-Instruct [16] is prompted with a category-specific yes/no question for each frame. The estimated onset is the first time at or after 1.5 s at which three consecutive frames satisfy .
The 208 clips whose onsets remain undetermined, together with a random sample of 30 clips used to assess annotation accuracy, are annotated manually without access to the model outputs. Manual onset times are used for all 238 clips. In the accuracy sample, the estimated onsets are a median of 0.4 s later than the manual annotations and more than 1 s later in 10% of clips. Such timing discrepancies may blur the distinction between the near and just-after groups.
Dataset-preparation partitions. During dataset preparation, we reserve a fraction of eligible clips as donors, which supply calm frames for fixed fitting and calibration partitions. The remaining clips form the evaluation set . For HomeSafe-Bench, donors are selected from clips with intent onset after 0 s and contribute only calm frames; near and just-after frames are drawn from . For RoboShackles, donor clips are included when constructing all three temporal groups. Bootstrap resampling is performed separately within the fixed fitting, calibration, and evaluation partitions.
These preparation partitions are distinct from the cross-fitting used to obtain the reported scores. In cross-fitting, each baseline is fitted using calm frames from both donor and evaluation clips, while excluding every clip in the fold containing the clip being scored. Table 1 reports the partition sizes, the within-donor split, and the random seed used for preparation and fold assignment.
| Setting | Symbol | HomeSafe-Bench | RoboShackles |
| Frame groups and preparation | |||
| Calm start | 0.5 s | 0.5 s | |
| Calm cutoff | s | ||
| Near / just-after window widths | 2 s, 2 s | 1 s, 1 s | |
| Donor fraction | 0.20 | 0.20 | |
| Donor / evaluation clips | — | 61/245 | 240/960 |
| Preparation and fold seed | 0 | 0 | |
| Baselines and controls | |||
| Cross-fitting folds | 5 | 5 | |
| Relative covariance ridge | |||
| Flow resolution / consistency tolerance | 224 px, 1 px | 224 px, 1 px | |
| Elapsed-time bin width | 0.5 s | 0.5 s | |
| Image-change bins | 5 | 7 | |
| Time + image-change bins | — | 7 | |
| Category + time + image-change bins | — | 4 | |
| Minimum clips per group and stratum | 5 | 5 | |
| Inference | |||
| Bootstrap resamples / interval level | 2,000, 0.95 | 2,000, 0.95 | |
| Adjusted significance threshold | 0.01 | 0.01 | |
| Coverage thresholds | 0.15, 0.25 | 0.15, 0.25 | |
| Bootstrap seeds |
Table 1. Settings for the representational sensitivity analysis. Image-change bin counts apply separately to object motion and appearance change. A dash denotes an analysis not performed.
Control variables and matched comparisons
Higher surprise near a harmful consequence may also reflect elapsed clip time, image change, or danger category composition. We therefore compare near versus calm, just-after versus calm, and just-after versus near, both without matching and with separate or joint matching on these variables. In each contrast, the first group is the target and the second is the reference . Elapsed time and danger category are obtained from video timestamps and dataset labels, respectively. For image change, we distinguish camera-compensated object motion from residual appearance change that is not explained by motion.
Image change measurements. We analyze the combination of two aspects of visual change:
Object motion is optical-flow movement after removing the fitted camera motion. It is measured as the mean per-second magnitude of dense Farnebäck optical flow [17] on grayscale frames at resolution , after removing camera motion by fitting a 2D similarity transform to the flow correspondences using RANSAC [18];
Appearance change is pixel change that movement doesn’t explain, such as spills, fire or flashes. It is measured as the mean absolute per-second residual between the current frame and the preceding frame warped using backward flow, after removing the frame-wide brightness shift. Only pixels that pass a forward–backward consistency check with tolerance contribute to the appearance-change measure.
Matching strata and configurations. Each control is defined by its strata. Elapsed-time strata are bins of width of time into the clip, and category strata are the dataset’s danger categories. Image-change strata cross quantile bins of object motion with quantile bins of appearance change, with bin boundaries fitted on . Joint controls cross the corresponding strata, with the image-change bins then fitted within each elapsed-time bin (and each category).
For both datasets, we run separate controls for elapsed time, image change and danger category. Category matching uses the six RoboShackles categories and three HomeSafe-Bench danger types, with L1 and L2 combined because of their small sample sizes. For RoboShackles, we additionally match elapsed time and image change jointly, with and without the danger category. HomeSafe-Bench does not support these joint controls: its calm frames end at the hazard onset, about 1.8 s into the clip, too early to share later elapsed-time bins with the target frames. Bin widths and counts are listed in Table 1.
Detailed computation of metrics
Site-group summaries. We summarize over groups of sites (i.e., vision encoder, vision–language backbone, action generation). The score of each site, denoted as , is first standardized over all frames of the run, so that no site dominates through its dimension, and the group’s sites are then averaged:
where and are the mean and standard deviation of over all frames. The same comparisons below are applied to and, as a model-free reference, to the video’s own object motion.
Groups of frames. Let be the time of the harmful consequence in clip and the time to it ( before the consequence). We compare three groups of frames: near (, the last seconds before the consequence), just after (), and the calm frames , which are kept out of the other two. Here s for HomeSafe-Bench and s for RoboShackles.
Each comparison takes a target group and a reference group : the calm frames , or the near frames when comparing just-after with near. Since frames of the same clip are not independent, each group is first averaged within each clip and then over clips:
where is the set of clips with frames in group and those frames of clip . Frames near or after the consequence also differ from calm frames in nuisance properties, such as how far into the clip they are or how much the image changes.
A control holds one such property fixed by comparing the groups within strata of it, where and are the averages of Eq. (6) taken within stratum . We report
where is the number of target clips in stratum , contains the strata of control with at least five clips in both groups, and is the standard deviation of the reference group’s per-clip means.
The matched difference thus standardizes the reference to the target’s distribution of the nuisance property, and the share of target frames falling in (the coverage) is reported with it. Dividing by expresses in clip-to-clip spreads: means no difference from the reference, and means the groups differ by as much as reference clips typically differ from one another.
Figures. Figures 1, 3 to 6 trace along the time from the consequence, . The frames from 0.5 s into the clip, calm frames included, are cut into bins of 0.25 s in , and each bin is compared with the calm frames: shows how evolves from calm through the consequence, and how much of that evolution remains once frames are compared only with calm frames at the same point in the clip. Bins with fewer than 20 clips are omitted, as are matched bins with coverage below 15%.
Figures 7 and 8 report for every control, with the whole near or just-after group and as above, giving one value per comparison and control.
Coverage. Coverage is the fraction of target frames that belong to usable strata, rather than the fraction of clips. We do not display contrasts whose coverage is below , and we flag those with coverage between and as low coverage. Because more restrictive matching can exclude different target frames, the population represented by a matched contrast may vary across control configurations.
Uncertainty and significance. Since frames of the same clip are not independent, we quantify uncertainty with a cluster bootstrap that resamples whole clips with replacement. Clips are resampled separately within each part of the fixed split, so that group sizes stay fixed, and the image-change bins are refitted on every resample. For each difference , the two-sided -value is twice the smaller share of resampled differences on either side of zero. We run the bootstrap with three seeds of 2,000 resamples each and pool them: -values are computed from all 6,000 resamples, the 95% intervals of the group comparisons (Figures 7 and 8) are the average of the percentile intervals over the three seeds, and the bands of the curves (Figure 1) are percentile intervals over all 6,000 resamples.
To account for the many comparisons, we apply the Benjamini–Hochberg correction [19] separately for each model and site group. Each such family contains every computable test of that model and site group: and , the three comparisons (near vs calm, just-after vs calm, just-after vs near), and the raw difference and each control. A difference is called significant when its adjusted -value is below 0.01.
Figure 4. HomeSafe-Bench: transition surprise relative to calm frames along the time from the impact (marked zero on the x-axis), at the same elapsed time: each time bin is compared only with calm frames at the same time into the clip; bins with too few matched frames are not drawn. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the impact.
Figure 5. RoboShackles: transition surprise relative to calm frames along the time from the consequence onset (marked zero on the x-axis), without matching: each time bin is compared with all calm frames. Rows: language, vision and action-generation sites; last column: the video’s own object motion. Zero is the calm level, in clip-to-clip spreads of the calm frames. Shaded: near and just after the consequence onset.
Avoidance Behaviour Analysis
Frame selection and annotation
Candidate frames. We define the onset frame as the first sampled frame at or after the harmful-consequence time. For RoboShackles, we select 1,200 onset frames from all clips and 960 final frames from the evaluation clips. Support-plane estimation fails for three frames, leaving 2,157 scenes. For HomeSafe-Bench, the evaluation clips provide 174 onset frames and 245 final frames; in the remaining 71 clips, impact occurs after the last sampled frame. Manual review excludes 20 frames, leaving 399 scenes.
Measured-scene filter. To be included in the reported results, a scene must use a gripper location identified in the source frame itself and preserve the estimated gripper–danger offset without adjustment. We apply the same exclusion criteria to all models, removing standing-board layouts, scenes with assumed, borrowed, or manually placed grippers, and scenes requiring placement adjustments. Table 3 lists the exclusion reasons, which may overlap. The remaining data comprises 815 of the 2,157 RoboShackles frames (569 onset and 246 final) and 166 of the 399 HomeSafe-Bench frames (78 onset and 88 final). This filter favours frames with visible grippers within the robot’s reach and excludes most RoboShackles frames in the human category. Here, “measured” refers to the reconstruction criterion, not ground-truth geometry.
| Reason | RoboShackles | HomeSafe-Bench |
| Standing-board layout | 260 | 0 |
| Assumed gripper | 494 | 3 |
| Gripper out of view (hand-placed) | 0 | 32 |
| Gripper at the danger (hand-placed) | 0 | 13 |
| Gripper borrowed from the other frame | 330 | 41 |
| Raised placement | 235 | 80 |
| Reach-capped or clamped offset | 461 | 40 |
| Height moved (floor) | 243 | 113 |
| Support label contradicts height | 79 | 55 |
| Excluded / included | 1,342 / 815 | 233 / 166 |
Table 3. Action-evaluation exclusions after annotation and support-plane screening. Reasons may overlap; the final row gives unique excluded and retained frame counts.
Image annotations and depth. Using greedy decoding, Qwen3-VL-8B-Instruct identifies the danger region, gripper fingers, fingertip location, and support type. For HomeSafe-Bench, the prompt specifies that the gripper nearest the danger should be selected. Six invalid RoboShackles annotations and the HomeSafe-Bench annotations flagged during review were corrected manually. Images used for annotation have a long side of 768 pixels.
These annotations locate the gripper and danger in the image, but reconstructing their relative geometry requires estimates of their three-dimensional positions at a metric scale. We therefore use Depth Pro [20] to estimate metric depth from each source frame. Together with the focal length, these depth estimates allow the annotated image locations to be mapped to three-dimensional gripper and danger points in source-view coordinates. Their relative offset is then used to position the gripper and danger marker in the digital cousin. The focal length is fixed at 526.7 pixels for RoboShackles and estimated separately for each HomeSafe-Bench frame by Depth Pro.
Reconstructing gripper–danger geometry
Geometry in source-view coordinates. We use metric depth and focal length to map the image annotations to a gripper point and a danger point in three-dimensional source-view coordinates. Gripper depth is estimated from the fingers rather than from the potentially empty space between them. For supported danger regions, is the nearest point at the region’s base; for injuries on standing people, it is the nearest surface point.
An estimated support plane defines the upward direction . Projecting the camera’s horizontal axis onto this plane gives , and we define . The gripper-to-danger offset expressed in these view directions is
We transfer this offset to the simulator’s corresponding view directions to position the robot relative to a non-colliding danger marker. This transfer assumes that the policy interprets the source image as its own main-camera view; the true source-camera-to-robot transformation is unknown. The digital cousin reconstructs the relative geometry, not the complete source scene.
Simulator and placement. We construct simplistic scenes using LIBERO’s LIBERO_goal suite, with the task objects removed, using robosuite v1.4.1 [21] and MuJoCo v2.3.7 [22]. The robot base remains fixed, and inverse kinematics positions the gripper with a fixed downward-pointing orientation. The danger anchor is the centre of the marker’s base. Injuries on standing people are represented by an upright board centred on ; when no support plane is visible, we assume an upward direction. During scene construction, unreachable placements are adjusted, and missing gripper locations are either borrowed from the clip’s other frame or specified by rule. We record all placement adjustments and assumed geometry, and exclude scenes requiring either from the reported results. For each frame, all models use the same initial scene.
Action interfaces and endpoint evaluation
Delta-action policies. We use LIBERO checkpoints for the action study. For GR00T N1.7 and VLA-JEPA, we reuse the first action of the predicted chunk recorded for each source frame during the representational study. GR00T’s actions are already unnormalised, whereas VLA-JEPA’s actions are clipped and unnormalised using the checkpoint’s minimum and maximum statistics. We convert the actions to the simulator’s controller units and map each model’s gripper convention to the LIBERO controller. Starting from the reconstructed scene, we hold this first command for control steps at Hz (0.4 s) and record the gripper position after each step.
Absolute-target policy. Because X-VLA predicts absolute targets, we query it again using the original source image, the fixed instruction, and the reconstructed end-effector pose as robot state, still without a wrist view. We evaluate the first position target in the predicted chunk directly as .
Avoidance behaviour by types and levels of danger.
The following figures present the category-specific breakdowns of the behaviour by categories in Figure 2. Figure 10 shows the four HomeSafe-Bench danger levels, and Figure 11 shows the six RoboShackles categories. For each category and model, panel (a) reports the median distance-reduction ratio with its 95% frame-bootstrap confidence interval, while panel (b) shows the proportions of actions in the away, sideways, and straight-at bins.

Figure 9. Examples of reconstructed digital-cousin scenes in LIBERO. Each row shows a source frame with the annotated gripper and danger region (left), followed by the corresponding reconstructed scene from multiple simulator views (right). In the reconstruction, the danger location is represented by a simplified red box whose placement preserves the estimated gripper–danger offset recovered from image annotations and metric depth. The robot pose is initialized to approximately match the annotated frame, while the surrounding environment is reduced to a minimal scene sufficient for evaluating the direction of the initial predicted action.
References
- [1]Kim, M. J., Pertsch, K., Karamcheti, S., Xiao, T., Balakrishna, A., Nair, S., Rafailov, R., Foster, E. P., Sanketi, P. R., Vuong, Q., Kollar, T., Burchfiel, B., Tedrake, R., Sadigh, D., Levine, S., Liang, P., & Finn, C. (2025). OpenVLA: An open-source vision-language-action model. Proceedings of Machine Learning Research, 270, 2679–2713. https://proceedings.mlr.press/v270/kim25c.htmlproceedings.mlr.press/v270/kim25c.html
- [2]Black, K., Brown, N., Darpinian, J., Dhabalia, K., Driess, D., Esmail, A., Equi, M. R., Finn, C., Fusai, N., Galliker, M. Y., Ghosh, D., Groom, L., Hausman, K., Ichter, B., Jakubczak, S., Jones, T., Ke, L., LeBlanc, D., Levine, S., … Zhilinsky, U. (2025). π₀.₅: A vision-language-action model with open-world generalization. Proceedings of Machine Learning Research, 305, 17–40. https://proceedings.mlr.press/v305/black25a.htmlproceedings.mlr.press/v305/black25a.html
- [3]B. Zhang, Y. Zhang, J. Ji, Y. Lei, Y. Cai, J. Dai, Y. Chen, and Y. Yang. SafeVLA: Towards safety alignment of vision-language-action model via constrained learning, 2025. URL https://arxiv.org/abs/2503.03480.
- [4]M. Lyu, Y. Sun, Y. Jia, S. Shen, M. Sha, H. Li, F. Zhao, and Y. Zeng. ForesightSafety-VLA: A unified diagnostic safety benchmark for vision-language-action models, 2026. URL https://arxiv.org/abs/2606.27079.
- [5]Z. Yin, C. Liu, W. Yang, R. Li, Y. Guo, and Y. Xue. RoboShackles: A multilingual safety dataset for human-injury prevention in embodied foundation models, 2026. URL https://arxiv.org/abs/2606.18632v2.
- [6]J. Pu, Z. Sun, Z. Zhang, X. Zhang, and J. Xu. HomeSafe-Bench: Evaluating vision-language models on unsafe action detection for embodied agents in household scenarios, 2026. URL https://arxiv.org/abs/2603.11975.
- [7]K. Friston. The free-energy principle: A unified brain theory? Nature Reviews Neuroscience, 11(2):127–138, 2010. doi:10.1038/nrn2787.DOI
- [8]P. Sterling. Allostasis: A model of predictive regulation. Physiology & Behavior, 106(1):5–15, 2012. doi:10.1016/j.physbeh.2011.06.004.DOI
- [9]R. M. Nesse. The smoke detector principle: Signal detection and optimal defense regulation. Evolution, Medicine, and Public Health, 2019(1):1, 2019. doi:10.1093/emph/eoy034.DOI
- [10]J. R. Wessel and A. R. Aron. Unexpected events induce motor slowing via a brain mechanism for action-stopping with global suppressive effects. Journal of Neuroscience, 33(47):18481–18491, 2013. doi:10.1523/JNEUROSCI.3456-13.2013.DOI
- [11]Sun, J., Zhang, W., Qi, Z., Ren, S., Liu, Z., Zhu, H., Sun, G., Jin, X., & Chen, Z. (2026). V1a-jepa: Enhancing vision-language-action model with latent world model. In European Conference on Computer Vision (pp. 478–497).
- [12]Bjorck, J., Castañeda, F., Cherniadev, N., Da, X., Ding, R., Fan, L., Fang, Y., Fox, D., Hu, F., Huang, S., & al., . et . (2025). Gr00t n1: An open foundation model for generalist humanoid robots. In arXiv preprint arXiv:2503.14734. 2503.14734
- [13]Zheng, J., Li, J., Wang, Z., Liu, D., Kang, X., Feng, Y., Zheng, Y., Zou, J., Chen, Y., Zeng, J., & al., . et . (2026). X-vla: Soft-prompted transformer as scalable cross-embodiment vision-language-action model. International Conference on Learning Representations, 2026, 60580–60606.
- [14]B. Liu, Y. Zhu, C. Gao, Y. Feng, Q. Liu, Y. Zhu, and P. Stone. Libero: Benchmarking knowledge transfer for lifelong robot learning. arXiv preprint arXiv:2306.03310, 2023.arXiv
- [15]Li, X., Hsu, K., Gu, J., Pertsch, K., Mees, O., Walke, H. R., Fu, C., Lunawat, I., Sieh, I., Kirmani, S., Levine, S., Wu, J., Finn, C., Su, H., Vuong, Q., & Xiao, T. (2024). Evaluating real-world robot manipulation policies in simulation. In arXiv preprint arXiv:2405.05941. arXiv:2405.05941
- [16]Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., … Zhu, K. (2025). Qwen3-vl technical report. In arXiv preprint arXiv:2511.21631. arXiv:2511.21631
- [17]G. Farnebäck. Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis, pages 363–370. Springer, 2003.DOI
- [18]Fischler, M. A., & Bolles, R. C. (1981). Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6), 381–395.
- [19]Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal statistical society: series B (Methodological), 57(1):289–300, 1995.DOI
- [20]Bochkovskiy, A., Delaunoy, A., Germain, H., Santos, M., Zhou, Y., Richter, S., & Koltun, V. (2025). Depth pro: Sharp monocular metric depth in less than a second. International Conference on Learning Representations, 2025, 75602–75637. https://proceedings.iclr.cc/paper_files/paper/2025/file/bc8b2058fd96978a4146f18298cb2d39-Paper-Conference.pdfproceedings.iclr.cc/paper_files/paper/2025/file/bc8b2058fd96978a4146f18298cb2d39-Paper-Conference.pdf
- [21]Y. Zhu, J. Wong, A. Mandlekar, R. Martín-Martín, A. Joshi, S. Nasiriany, Y. Zhu, and K. Lin. robosuite: A modular simulation framework and benchmark for robot learning. In arXiv preprint arXiv:2009.12293, 2020.arXiv
- [22]E. Todorov, T. Erez, and Y. Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109.DOI





