EAST: Embodied AI Safety Testing via Systematic Fault Injection in Digital Twins

Introduction

Embodied artificial intelligence (AI) is rapidly evolving towards agents that integrate visual perception, language understanding, and closed-loop physical control [5, 59]. Recent vision-language-action (VLA) models, including RT-1, RT-2, OpenVLA, RDT-1B, and $\pi_0$, have substantially advanced language-conditioned and generalist robotic manipulation [9, 10, 12, 49, 13]. In parallel, manipulation benchmarks and simulation platforms (such as RLBench, CALVIN, ManiSkill2, RoboTwin, RoboTwin 2.0, VLABench, and VLA-Arena) provide increasingly diverse and reproducible environments for evaluating embodied policies [6, 7, 8, 4, 3, 50, 14]. These advances allow embodied AI systems to perform more tasks and support evaluation in more settings. They also make systematic safety assessment more important. Unlike purely digital models, embodied agents act in the physical world, where a wrong decision can lead to unsafe physical actions, such as dropped objects, unstable manipulation, inappropriate interactions, or other problems [63, 59].

A central difficulty for embodied-AI safety assessment is that task success does not imply safe execution. A robot, for example, can complete a requested task while violating safety constraints at an intermediate step during execution. Recent safety-oriented benchmarks have begun to expose this success–safety gap. SafeVLA-Bench [42] evaluated successful trajectories against task-aware safety specifications, reporting successful but unsafe executions; ForesightSafety-VLA explicitly decomposed safe and unsafe success and failure, while measuring process-level risk [15]; and HazardArena constructed matched safe/unsafe contexts to reveal semantic safety failures that may have remained hidden had evaluation been based only on action completion [43]. These studies provide strong evidence that conventional success-oriented evaluation can overlook safety-relevant behavior. They also motivate a shift from asking only if an embodied policy can accomplish a task to examining how the policy behaves when safety-critical conditions are deliberately introduced during testing.

In this paper, we introduce Embodied AI Safety Testing (EAST), a framework for systematic fault-injection testing in digital twins. Figure 1 illustrates the success–safety gap through three representative EAST executions: Operator 3 (Op3) modifies the instruction by swapping object roles; Operator 8 (Op8) triggers a premature release before the receiver is ready; and Operator 7 (Op7) postpones commands to a selected arm by four calls to the policy’s action-generation interface — the delay is counted in interface calls, rather than elapsed time. In all three of these examples, the task-success condition is satisfied despite a detected safety violation. The displayed images are illustrative replays of the fixed test configurations used in the formal experiments: They were not included in the quantitative analyses, which used the original formal execution records.

Previous evaluation and stress-testing approaches addressed complementary parts of this problem, but they had different objectives. Robustness benchmarks and domain-randomization techniques have evaluated whether or not policies remain reliable after changes in appearance, camera pose, object layout, physical properties, and sensor conditions [18, 19, 44, 3, 51]. Such perturbations are valuable for measuring generalization and robustness to changes in simulated conditions, but such environmental changes do not necessarily represent a defined safety fault, or specify which safety requirement should be checked. In contrast, safety benchmarks define hazardous contexts, safety constraints, or safety-oriented metrics for evaluating behavior under specified risk conditions [42, 41, 52, 53]. Adversarial and red-teaming studies, which deliberately probe embodied agents for unsafe behavior [40, 57], complement these approaches: They actively search for inputs or configurations that weaken embodied agents’ decision-making or cause unsafe behavior. Their methods include physical red teaming, embodied jailbreaks, fine-grained adversarial perception attacks, and sensor-level perturbations [40, 56, 57, 58, 60]. Although these approaches can improve embodied-AI evaluation and security, a key testing problem is the need to be able to turn safety knowledge into executable faults for safety testing. A clear record is needed that links the testing evidence, such as fault realization and safety assessment. Systematic safety testing must evaluate more than whether or not a scenario is challenging: It must confirm that an intervention is meaningful for the current task and policy; whether or not it takes effect during execution; and which safety requirement it tests. Finally, it must also link the observed behavior to the injected fault configuration, and assess reproducibility through controlled replay.

Software and cyber-physical-system testing provides a useful foundation for generating safety tests, and for checking execution-level safety requirements. Fault injection has long been used to evaluate system behavior in the face of controlled faults [26, 25]. Automated testing, mutation analysis, fuzzing, and search-based falsification also support systematic test generation, fault seeding, and searches for requirements violations [27, 67, 68]. DeepTest and DeepHunter have applied automated transformation- and coverage-guided test generation for learning-enabled systems [29, 32]. A test oracle enables determination of whether or not an observed execution satisfies the expected requirements: The (test) oracle problem arises when an oracle is unavailable, or available, but not practically applicable [65]. For cyber-physical and robotic systems, DriveFuzz used simulator-based scenario mutation and state-based driving oracles to uncover safety-critical behaviors [37]; RoboFuzz combined semantic mutation, robotic execution, and correctness oracles for Robot Operating System (ROS)-based systems [45]; and PSY-TaLiRo generated tests that falsified temporal requirements [46]. Metamorphic testing uses relations among source and follow-up executions when individual expected outcomes are difficult to specify [66, 1, 2]. It has recently been applied directly to VLA-enabled robots [47]. These studies show the value of systematic test generation, executable oracles, and repeatable simulation. However, modern embodied manipulation also needs a safety-focused approach to turn safety concerns into controlled runtime interventions. These interventions must cover perception, instruction grounding, physical scene configuration, action generation, and the timing and ordering of robot actions and interaction events.

To bridge this gap, we developed EAST around a three-stage safety-testing workflow. EAST first constructs a literature-grounded safety-fault model, maps safety obligations to eight executable fault operators, and generates test specifications subject to applicability, validity, semantic, contextual, deduplication, and diversity constraints. EAST then executes each predefined test specification against the policy under test, applies the corresponding intervention at the observation, instruction, environment, action, or temporal interface, and records operator-specific evidence establishing whether or not the requested fault was realized at runtime. Finally, EAST evaluates the resulting simulator state and execution traces with safety-oriented oracles, separating challenge exposure and safety violation from task outcome: It does this while preserving behavior signatures, coverage information, provenance, and replayable test records. For each test, EAST first verifies whether or not the requested fault actually occurred, and then checks if the resulting execution violated a safety requirement, independently of whether the task was completed. Each finding remains linked to the corresponding test specification and execution evidence.

To evaluate EAST’s performance, we conducted an empirical study on six RoboTwin manipulation tasks with two representative policy families, under matched testing budgets. Generation was compared with four strategies: Random-Legal randomly samples valid configurations from applicable operators; Random-Unconstrained samples a broader configuration space without EAST’s applicability, validity, or semantic filters; Rule-Based applies the first applicable rule with a representative valid configuration; and RoboTwin-DR uses the simulator’s predefined domain-randomization settings. The primary safety-testing comparison used Random-Legal, Rule-Based, and RoboTwin-DR as testing comparators. Clean executions, where each native task was run without an injected EAST fault, were also included as no-fault references. § 4.4 has the details.

The evaluations focused on three aspects: Constructing and operationalizing the safety-fault model; systematically generating applicable, valid, runtime-realized safety tests; and comparing safety-testing effectiveness under matched testing budgets. The equal-budget comparison in Experiment 3 was repeated five times, with a designated master seed for each repetition: We refer to each repetition as a campaign. The experimental results show that EAST included all 74 admissible task–policy–operator combinations in each repetition, obtaining valid execution evidence for 63 of them, with safety-challenge exposure rates close to those of Random-Legal and Rule-Based. The resulting records provide structured and traceable evidence whose selected representative behaviors remain observable under controlled replay. The comparative evaluation also exposed a concrete success–safety gap: 82 valid EAST executions completed their tasks despite a detected safety violation. Ablation results further identified complementary roles for validity-, semantics-, context-, applicability-, deduplication-, taxonomy-, and diversity-aware generation mechanisms. In summary, the results suggest that embodied-AI safety testing should be evaluated not only according to task completion or raw failure counts, but also for test validity, safety-challenge exposure, systematic coverage, behavioral diversity, and reproducibility.

The main contributions of this paper are as follows:

• We provide an evidence-grounded safety-fault model for embodied AI that organizes safety-relevant failure phenomena and operationalizes them through executable fault-injection operators. This provides a bridge from safety knowledge to testable runtime conditions.

• We introduce EAST, a systematic safety-testing framework that generates constraint-valid fault-injection tests, verifies their runtime realization in digital twins, and assesses resulting behaviors independently of task success. For each execution, EAST links the test specification, realized fault, execution trace, and safety outcome in a record that supports controlled replay.

• We report on a multi-dimensional empirical evaluation across six manipulation tasks and two policy families. The results show that EAST covers all 74 admissible task–policy–operator combinations in each of the five repetitions of Experiment 3, with safety-challenge exposure rates close to those of Random-Legal and Rule-Based. Executions that complete their tasks despite a safety violation highlight the need to evaluate task and safety outcomes separately.

The rest of this paper is organized as follows. § 2 introduces the background and preliminaries of embodied AI systems, digital twins, and safety testing. § 3 presents the three-stage EAST workflow. § 4 introduces the research questions, target tasks and policies, comparison strategies, experimental protocols, and evaluation metrics. § 5 reports the empirical results and analyzes the effectiveness of EAST. § 6 discusses threats to the validity of our study, and § 7 reviews related research. Finally, § 8 concludes the paper and outlines possible future research directions.

Preliminaries

This section introduces the basic concepts of embodied AI systems, safety specifications, and fault-injection models that form the foundation for EAST’s design. For ease of description, Table 1 and Table 2 provide the abbreviations and mathematical notation used throughout the paper.

Embodied AI Systems

An embodied AI system is an intelligent agent that perceives the environment, makes decisions, and executes physical actions through interactions with the surrounding world [5]. During closed-loop operations, an embodied AI system uses observations to guide action selections, with the executed actions then modifying the environment from which subsequent observations are obtained.

We model an embodied AI system as: $\Sigma=(S,O,A,\psi,\pi,\delta)$, where $S$, $O$, and $A$ denote the environment’s state space, observation space, and action space, respectively. The observation function $\psi:S\rightarrow O$ maps an environment state to the sensory observation available to the agent; the embodied policy $\pi:O\rightarrow A$ maps an observation to an executable action; and the transition function $\delta:S\times A\rightarrow S$ characterizes the environment dynamics.

At each time step $t$, the agent receives an observation $o_t=\psi(s_t), s_t\in S,\; o_t\in O$, and generates an action $a_t=\pi(o_t)\in A$. After the action is executed, the environment transitions to the next state, according to $s_{t+1}=\delta(s_t,a_t)$. For robotic manipulation scenarios, the environmental state can be decomposed as: $s_t=(r_t,x_t,p_t)$, where $r_t\in R$ denotes the robot state, $x_t\in X$ denotes the state of objects in the environment, and $p_t\in P$ denotes environmental parameters. $R$, $X$, and $P$ denote the corresponding state spaces.

Digital twins are high-fidelity virtual representations of physical systems that maintain state correspondence between the physical system and its simulated counterpart [62]. For embodied AI systems, digital twins provide controllable environments in which robot states, object states, and environmental parameters can be represented, manipulated, and reproduced [4, 3].

Safety Specifications

Safety testing examines whether or not an embodied AI system satisfies predefined safety requirements throughout execution. Unlike task objectives, which characterize completion of the intended task, safety specifications characterize behavioral constraints that must remain satisfied during execution [63, 41].

If $\Phi=\{\phi_1,\phi_2,\ldots,\phi_{N_\Phi}\}$ denotes the set of safety specifications applicable to an execution, where each $\phi_i$ is a Boolean safety predicate representing an individual safety requirement, then we can represent an execution trace as an ordered sequence, $\rho=\langle s_0,a_0,s_1,a_1,\ldots,s_h\rangle$, where $h$ is the final time-step index of the execution, and $\rho_{\leq t}$ denotes the trace prefix up to time step $t$. The execution is considered safe if $\forall\,\phi_i\in\Phi, \forall\,t\in\{0,\ldots,h\}, \phi_i(\rho_{\leq t})=\mathrm{true}$. A safety violation occurs if at least one applicable safety predicate is violated during execution.

A robot’s collision-avoidance requirement with respect to object $j$ can be expressed as $d(r_t,x_t^{(j)}) \geq d_{\mathrm{safe}}$, where $d(\cdot,\cdot)$ denotes the distance between the robot and the relevant object, and $d_{\mathrm{safe}}$ is the minimum allowable separation distance. Similarly, a physical-interaction constraint can be expressed as $f_t^{\mathrm{int}} \leq f_{\max}^{\mathrm{int}}$, where $f_t^{\mathrm{int}}$ denotes the interaction-force magnitude at time $t$, and $f_{\max}^{\mathrm{int}}$ denotes its maximum allowable value. Other safety specifications may depend on temporal relations or event sequences, and could therefore be evaluated over the corresponding trace prefix [64, 41].

A safety-critical failure occurs when an embodied-AI execution violates one or more applicable safety specifications. This evaluation is independent of task completion: Successful completion of the intended task does not imply that all $\phi_i\in\Phi$ remain satisfied throughout the execution. Therefore, evaluation of the task success and the execution safety need to be done separately.

Fault Injection Model

Fault injection evaluates system behavior under controlled abnormal conditions by deliberately perturbing selected components or signals during execution [26, 25]. In embodied AI systems, an intervention may target different parts of the closed-loop interaction, including observations, task instructions, physical scene states, action signals, and temporal control relations.

If $C=\{c_1,c_2,\ldots,c_{N_C}\}$ denotes the set of safety-taxonomy classes used by EAST, and $\Omega=\{\omega_1,\omega_2,\ldots,\omega_{N_\Omega}\}$ denotes the set of executable fault-injection operators, and each taxonomy class is associated with one or more evidence-supported executable operators, then this relationship can be represented as:

where $\mathcal{P}(\Omega)$ denotes the power set of $\Omega$, and $\emptyset\neq\Gamma(c)\subseteq\Omega$ denotes the non-empty set of executable operators supported for operationalizing taxonomy class $c$. The taxonomy distinguishes observable types of unsafe behavior, such as unsafe contact or object release, from contextual hazards, such as compliance with a harmful instruction. The former describes how safety is violated, while the latter describes the hazardous context in which the behavior occurs. Both types of labels may apply to a single execution, as explained in § 5.1.

Each operator $\omega\in\Omega$ acts on an operator-specific target domain $Y_\omega$, and has a corresponding predefined configuration domain $\Theta_\omega$, which specifies the permitted target types, parameter values, and timing settings for that operator. A configuration in this domain must still satisfy the task-specific applicability and validity checks in § 3.2.2. For $y\in Y_\omega$ and $\theta\in\Theta_\omega$, a fault injection is represented as $y'=\omega(y,\theta)$, where $y$ denotes the operator’s intervention target, and $y'$ denotes its modified counterpart. Depending on the operator, the target may be an observation or instruction, a component of the environment state, a physical property, an action channel, or a temporal interaction relation.

The configuration $\theta$ is operator-specific, rather than a single, shared parameterization. It may encode, for example, a target entity, a spatial displacement, a physical-property override, a semantic transformation, a control value, or an event-relative timing parameter, depending on $\omega$’s semantics.

If $K$ denotes the task set, $\Pi$ is the set of embodied policies under test, and $Z$ is the seed-specification set, with each $\pi\in\Pi$ being a policy mapping of the form defined in § 2.1, then the task–policy space can be defined as:

A fault-injection test candidate is represented conceptually as $\eta=(u,c,\omega,\theta,z)$, where $c\in C$ is the targeted safety-taxonomy class, $\omega\in\Gamma(c)$ is the selected injection operator, $\theta\in\Theta_\omega$ is its concrete operator-specific configuration, and $z\in Z$ specifies the pre-determined random seeds required for reproducible test generation and execution.

The objective of fault injection is not merely to modify a system state, but to instantiate an evidence-grounded and executable safety challenge at a controlled intervention point, and evaluate the resulting execution against the safety specifications $\Phi$.

Methodology

This section presents the design and implementation of EAST.

Overview of EAST

As shown in Figure 2, EAST transforms safety knowledge into executable, assessable safety tests through the following three stages:

• Stage : Systematic Safety-Test Generation. This stage transforms external safety knowledge, the embodied task, and the policy under test into executable test specifications. The safety-fault model $\mathcal{M}_{\mathrm{SF}}$ organizes the taxonomy classes and their supported fault-injection operators. EAST then constructs the task-, policy-, and context-specific eligible injection space $\mathcal{X}_{\mathrm{eligible}}$. It checks that the required operator interface is available, the configured intervention is valid, the task remains meaningful, and the required safety context is present. It then removes duplicate configurations and selects diverse candidates. Each accepted candidate is serialized as an immutable TestCaseSpec before execution: Its operator configuration, seeds, and provenance are recorded and remain unchanged during that execution.

• Stage : Fault-Injected Digital-Twin Execution. This stage executes each predefined TestCaseSpec against the policy under test in the digital twin. Fault operators intervene at the corresponding observation, instruction, object, environment, action, timing, or interaction interface. EAST records operator-specific runtime evidence — such as modified inputs, states, command traces, and event timestamps — to verify that the intervention occurred on the intended target.

• Stage : Simulator-State Safety Assessment. This stage checks the recorded simulator states and execution trajectory against predefined safety requirements. These checks, called safety oracles, determine whether or not any requirement was violated. Task completion is evaluated independently. The challenge-exposure status, safety outcome, task outcome, and behavior signature are stored together with the predefined specification and runtime evidence in a reproducible safety-test record.

The following sections explain these three stages. A key design principle of EAST is separating test generation from test execution. Before execution begins, EAST fixes the candidate specifications, operator configurations, random seeds, and test order. Results from earlier tests, including observed failures, safety-oracle outputs, and execution coverage, are not used to alter later candidates. Coverage is measured after execution, to assess the exploration of the targeted safety space. The saved specifications, seeds, and test order allow the planned test sequence to be reconstructed for controlled comparison and replay.

Stage : Systematic Safety-Test Generation

Stage converts safety knowledge into executable test specifications. It comprises safety-fault modeling, construction of applicability and validity constraints, and concrete test generation.

Safety-Fault Model Construction

Current embodied AI evaluation approaches mainly focus on task completion and performance metrics. However, safety-critical failures may be caused by various issues, including perception errors, incorrect decisions, unsafe execution, and unexpected environmental interactions [15, 59]. Without an explicit fault model, safety testing cannot systematically characterize relevant failure sources or derive targeted and executable safety tests.

To address this limitation, EAST constructs a safety-fault model that connects safety evidence to executable fault-injection operators. The construction process addresses RQ1 (Safety-Fault Taxonomy and Operationalization) through three subquestions:

1. What types of safety-critical failures have been reported for embodied AI systems?

2. How can these failure observations be consolidated into general safety-taxonomy classes?

3. How can the resulting taxonomy classes be operationalized through executable interventions in a digital twin?

Building on the definitions in § 2.3, we represent the resulting safety-fault model as $\mathcal{M}_{\mathrm{SF}} = (\mathcal{E}, C, \alpha, \Omega, \Gamma)$, where $\mathcal{E}$ denotes the collected safety-evidence corpus, $C$ denotes the set of safety-taxonomy classes, $\alpha$ denotes the abstraction mapping defined over the taxonomy-mapped subset of the evidence corpus, $\Omega$ denotes the set of executable fault-injection operators, and $\Gamma$ denotes the taxonomy-class-to-operator mapping. RQ1a–RQ1c are addressed through three construction steps: Collecting of safety evidence; abstraction of taxonomy classes; and the mapping of those classes to executable operators.

• Safety Evidence Collection (RQ1a). EAST first collects safety-related evidence from embodied AI, robotics, autonomous systems, and related testing literature, with the resulting evidence corpus denoted by $\mathcal{E}=\{e_1,e_2,\ldots,e_M\}$. For any $i\in\{1,\ldots,M\}$, $e_i\in\mathcal{E}$ denotes an evidence record describing a reported safety-relevant hazard or failure behavior.

  For each record, EAST retains the evidence source, the reported hazard or failure behavior, and the affected aspect of the embodied system. These records provide the empirical basis for identifying recurring safety-failure patterns, rather than defining fault categories solely from manually predefined assumptions.

• Fault Category Abstraction (RQ1b). After collecting safety evidence, EAST abstracts categories to consolidate recurring safety-failure patterns into general safety-taxonomy classes. Taxonomy-mapped evidence records that describe semantically similar failure behaviors, causal mechanisms, and affected system aspects, are grouped into the same class; substantively different failure patterns remain separated.

  If $C=\{c_1,c_2,\ldots,c_{N_C}\}$ denotes the resulting set of safety-taxonomy classes, and $\mathcal{E}_C\subseteq\mathcal{E}$ is the subset of evidence records retained for taxonomy mapping, then the abstraction process is represented as $\alpha:\mathcal{E}_C\rightarrow C$, where $\alpha(e_i)=c_j$ indicates that a taxonomy-mapped evidence record $e_i\in\mathcal{E}_C$ is assigned to class $c_j$.

  Evidence records that characterize auxiliary analytical attributes, or that do not establish an independent safety consequence, are retained separately, rather than being forced into $C$. This abstraction converts the safety-relevant evidence into a structured taxonomy, while preserving traceability from each taxonomy class to its supporting evidence. Two experts independently review the evidence-to-category assignments and operator mappings, following the procedure in § 4.2.1. An evidence record counts as direct taxonomy support only if it establishes a clear safety consequence, or if it identifies a safety condition that can be checked from observed execution states or events, independently of the task completion. Auxiliary analytical attributes and ordinary task failures without such a condition are retained separately.

• Fault Operator Mapping (RQ1c). Because the safety-fault model supports executable safety testing, the abstract taxonomy classes must connect to interventions realizable in a digital twin. EAST therefore maps each safety-taxonomy class to one or more evidence-supported, executable fault-injection operators.

  Using the taxonomy-to-operator mapping $\Gamma$, defined in § 2.3, $\Gamma(c_i)$ lists the implemented operators that can instantiate safety challenges associated with class $c_i$. The condition $\emptyset\neq\Gamma(c_i)\subseteq\Omega$ means that each class is supported by at least one operator, and that every selected operator belongs to the implemented library $\Omega$. The mapping need not be one-to-one: A taxonomy class may be tested using several operators. The operators apply different types of interventions, and the same operator may also support several taxonomy classes. This requires evidence that its perturbation can represent the safety concerns in those classes. Each concrete test candidate selects a single operator, with one operator-specific configuration. Each selected operator is then configured within its predefined configuration domain $\Theta_\omega$, using a permitted target and parameter setting. Task-specific applicability and configuration-validity checks are applied before the candidate is accepted for execution.

Through this evidence-grounded construction, EAST connects reported safety concerns to a structured taxonomy and executable intervention mechanisms, providing the basis for subsequent safety-test generation.

Applicability and Validity Constraints

The safety-fault model specifies which safety phenomena and executable interventions the evidence supports. However, not every operator applies or is meaningful for every embodied task and policy. EAST therefore restricts test generation to combinations of task, policy, safety class, and operator that pass the pre-execution checks defined below. These combinations form the eligible fault-injection space. Constraint-based scene generation, as illustrated by Scenic [69], provides a foundation for expressing admissible configurations and sampling meaningful simulation inputs.

We let $u=(k,\pi)\in U$ denote a task–policy pair: Task $k\in K$ executed by policy $\pi\in\Pi$. We also let $c\in C$ denote a safety-taxonomy class, and $\omega\in\Omega$ an executable fault-injection operator. The operator library $\Omega$ contains the implemented fault-injection operators defined by EAST. Candidate construction considers an operator only in the task–policy contexts where its required interface and task structure are available. It verifies runtime realization during execution.

Before generating a test, EAST checks whether or not (i) the task–policy pair can run, (ii) the operator is applicable, (iii) the required safety context is available, and (iv) the taxonomy supports that operator. We use iff to mean if and only if. The first three checks are defined by the following Boolean predicates:

• $Q(u)$ is true iff the task–policy pair has passed the study’s no-fault runtime checks, establishing that the task, policy checkpoint, and simulator can be initialized and executed together before a fault is introduced.

• $\mathrm{App}(u,\omega)$ is true iff the task–policy pair provides the interface and task structure required by operator $\omega$.

• $\mathrm{Ctx}(u,c,\omega)$ is true iff the task provides the entities, states, events, and observable information required to assess the targeted safety condition.

The additional condition $\omega\in\Gamma(c)$ requires the selected operator to belong to the evidence-supported mapping for safety class $c$. These four requirements combine in the eligibility predicate:

Equation (3) separates evidence-grounded taxonomy compatibility from task- and policy-specific executability. An evidence-supported operator may still not apply to a particular task–policy pair: Passing the pre-execution checks also does not prove that a requested fault actually occurs during a rollout. Runtime realization is verified separately during digital-twin execution.

The eligible fault injection space is defined as:

For each eligible context, EAST generates a concrete configuration $\theta\in\Theta_\omega$. $V(u,\omega,\theta)$ denotes the operator-specific pre-execution validity predicate, which is true iff the configuration satisfies the simulator, physical-feasibility, actuator, and operator-parameter constraints required to execute the intervention. A valid configuration may still create a safety challenge: Validity does not imply safe behavior. EAST retains the resulting candidate only when its configuration validity, semantic-preservation, and context-compatibility checks are satisfied.

To support systematic fault injection, EAST defines eight operators that intervene at different parts of the embodied control loop. For readability, we label these fault-injection operators Op1–Op8, as follows:

• Op1 (visual_target_mask) perturbs visual evidence associated with a target entity.

• Op2 (instruction_target_omission) removes a task-relevant target span from a language instruction.

• Op3 (instruction_role_swap) exchanges valid entity, role, or arm bindings.

• Op4 (object_pose_shift) modifies an object’s position or orientation.

• Op5 (object_friction_override) overrides task-relevant object friction or contact-friction parameters.

• Op6 (gripper_command_override) changes a specified gripper command during a controlled event window.

• Op7 (arm_action_delay) introduces a reproducible delay into a selected action channel.

• Op8 (handover_release_timing) changes the temporal relationship between sender release and receiver readiness during bi-manual handover.

Each operator has its own applicability rule, configuration domain, validity conditions, and runtime realization evidence. For example, an object-pose perturbation must preserve a stable and executable initial scene, whereas an instruction perturbation must preserve a syntactically valid and task-interpretable instruction. Similarly, command- and timing-level operators require the corresponding action or event interface to be observable and controllable. EAST does not impose a common “low–medium–high” severity scale across heterogeneous operators: It defines and validates configurations within each operator’s physical or semantic domain.

Safety-Test-Case Generation and Specification

Given the eligible injection space, EAST generates concrete instances of the fault-injection test candidate defined in § 2.3 through a constraint-aware generation pipeline. Each candidate $\eta=(u,c,\omega,\theta,z)$ consists of a task–policy pair $u\in U$, a targeted safety-taxonomy class $c\in C$, a selected fault-injection operator $\omega\in\Gamma(c)$, and an operator-specific configuration $\theta\in\Theta_\omega$. The seed specification $z\in Z$ records the random-seed values assigned to candidate generation and execution before the test is run.

The generation pipeline contains seven complementary mechanisms, each addressing different sources of ineffective test generation: taxonomy guidance; applicability checking; validity checking; semantic preservation; context compatibility; deduplication; and diversity-aware selection. Taxonomy guidance restricts candidate construction to evidence-supported safety obligations and their mapped operators. Applicability checking removes candidates that lack the required interface, task stage, object, or policy input. For an operator that requires two-arm coordination, the corresponding interaction must also be present: Op8, for example, requires a sender–receiver handover between the two arms. Operator-specific validity checking rejects parameter settings that the simulator or actuator cannot execute, or that violate the operator’s setup constraints: An Op4 pose change, for example, must leave an executable initial scene. Semantic checks ensure that a perturbation does not unintentionally replace the intended test objective with a meaningless or non-executable task. Context checking verifies that the states and relations required by a safety obligation are actually present. Canonicalization converts each candidate into a standard representation of its configuration fields. This enables a consistent comparison of equivalent representations. Deduplication then removes repeated executable configurations before they consume the testing budget. Finally, diversity-aware selection distributes the accepted candidates across tasks, policies, operators, and configuration patterns, using only information available before execution. Importantly, the diversity is determined only using information available before execution: Runtime coverage, previously observed failures, and safety-oracle outputs do not alter the remaining candidate sequence.

Each accepted candidate $\eta$ is saved in a machine-readable TestCaseSpec, denoted $\chi=(\eta,\nu)$: The candidate $\eta=(u,c,\omega,\theta,z)$ specifies what to test; and the provenance bundle $\nu$ records the protocol version, policy-checkpoint identity, simulator build, oracle version, mapping identifiers, and registry hashes. Once saved, the specification fields remain unchanged during execution. Runtime traces and outcomes are recorded separately, as described in § 3.3. The saved specification also records the target entity, affected arm or channel, task stage, and expected realization fields. A hash computed from the standardized specification serves as its identifier for matching test records and replays. The accepted specifications, in their standard representation, are collected into an execution manifest (an ordered list of the specifications to run) $\mathcal{M}_{\mathrm{run}}$. The list and its execution order are fixed before any test outcome is observed. For a fixed protocol and seed set, Stage I therefore produces the same planned test sequence, independently of subsequent policy behavior. Invalid, duplicate, and inapplicable candidate configurations remain in the generation log, allowing each generation attempt to be traced to an accepted test specification or an excluded candidate.

     \caption{Constraint-Aware Safety-Test Specification Generation}
    \label{alg:east_generation}
    \footnotesize
    \begin{algorithmic}[1]
        \Require
        Task--policy space $U$;
        safety-taxonomy classes $C$;
        fault operators $\Omega$;
        taxonomy-to-operator mapping $\Gamma$;
        operator configuration spaces $\{\Theta_\omega\}_{\omega\in\Omega}$;
        seed set $Z$
        
        \Ensure
        Predefined execution manifest $\mathcal{M}_{\mathrm{run}}$
        
        \State $\mathcal{X}_{\mathrm{eligible}}\gets
        \mathrm{ConstructEligibleSpace}(U,C,\Omega,\Gamma)$
        \State $\mathcal{G}\gets\emptyset$
        
        \ForAll{$(u,c,\omega)\in\mathcal{X}_{\mathrm{eligible}}$}
            \State $\mathcal{B}\gets\mathrm{GenerateConfigurations}(\Theta_\omega,Z)$
            \ForAll{$(\theta,z)\in\mathcal{B}$}
                \State $\eta\gets(u,c,\omega,\theta,z)$
                \If{$V(u,\omega,\theta)\land
                \mathrm{SemanticOK}(\eta)\land
                \mathrm{ContextOK}(\eta)$}
                    \State $\mathcal{G}\gets
                    \mathcal{G}\cup\{\mathrm{Canonicalize}(\eta)\}$
                \EndIf
            \EndFor
        \EndFor
        
        \State $\mathcal{G}_{\mathrm{unique}}\gets
        \mathrm{Deduplicate}(\mathcal{G})$
        \State $\mathcal{G}^{\star}\gets
        \mathrm{DiversitySelect}(\mathcal{G}_{\mathrm{unique}})$
        \State $\mathcal{M}_{\mathrm{run}}\gets
        \mathrm{FixSchedule}\bigl(\mathrm{Serialize}(\mathcal{G}^{\star})\bigr)$
        \State \Return $\mathcal{M}_{\mathrm{run}}$
    \end{algorithmic}

Algorithm alg:east_generation presents the pseudocode for the Stage-I generation. The task–policy–operator-level eligibility predicate defines the admissible structural space before proposing concrete configurations. Configuration validity, semantic preservation, and candidate-specific context compatibility are checked before candidates are converted to the standard representation. Duplicates are removed, diverse candidates are selected, and the execution order is fixed. All the operations depend only on pre-execution information.

Stage : Fault-Injected Digital-Twin Execution

Stage executes each predefined test specification in the digital twin, verifying the requested intervention at runtime.

Fault-Injected Digital-Twin Rollout

After test generation, each predefined TestCaseSpec $\chi$ is executed against the policy under test in the RoboTwin digital-twin environment. A fault operator defines a reusable transformation; an intervention is one configured application of that operator to a particular target during execution. For example, Op4 is the object-pose-shift operator, but moving the selected object by the specified displacement is the intervention. EAST applies the configured intervention at the point specified by its operator, leaving non-targeted interfaces unchanged. Scene- and physics-level interventions are applied during environment or actor construction and initialization, before the affected policy interaction begins. Observation-level interventions are applied after observation and before policy inference. Instruction-level operators modify the task text supplied to a language-conditioned policy. In this evaluation, Op2 and Op3 were applied to RDT, whereas ACT had no instruction input. Action-level operators are applied between the policy output and the simulator action interface. Temporal and handover operators act on explicitly recorded action or event relations.

Runtime Realization and Execution Traces

For each execution of a predefined specification $\chi$, EAST records the resulting execution trace $\rho$ and the operator-specific runtime-evidence bundle $\xi$. The corresponding execution record is defined as $\varepsilon=(\chi,\rho,\xi)$. Here, $\eta$ denotes the pre-execution test candidate; $\chi=(\eta,\nu)$ denotes its saved executable TestCaseSpec; and $\varepsilon$ denotes one execution record produced from $\chi$. Exact replay reuses the source specification and all its seeds. Stochastic replay creates a separate replay specification in which the policy seed is changed, but the other experimental settings are retained.

EAST distinguishes a requested injection from a realized injection. Let $\mathrm{Realized}(\varepsilon)$ be a Boolean predicate that is true iff the operator-specific runtime evidence confirms the requested modification, and false otherwise. The evidence may include records of the image masks and object poses after the intervention is applied. It may also include modified instruction inputs or checks of material states. Command traces may show the requested, issued, and observed commands. Timing evidence may include action-queue timestamps or the times of grasp, release, and contact events. A test is not considered successfully instantiated merely because an injection command was issued. If the required recorded states or traces cannot establish that the requested change occurred on the intended target and within its predefined tolerance, the test is classified as invalid rather than being treated as an unaffected policy execution.

\caption{Fault-Injected Digital-Twin Execution and Realization Verification}
    \label{alg:east_execution}
    \footnotesize
    \begin{algorithmic}[1]
        
        \Require
        Predefined test specification $\chi$;
        policy under test $\pi$;
        digital-twin environment $\mathcal{D}$
        
        \Ensure
        Execution record $\varepsilon$;
        %runtime-realization status $r$
        runtime-realization status $q_{\mathrm{real}}$
        
        \State $(u,c,\omega,\theta,z,\nu)
               \gets \mathrm{ResolveSpecification}(\chi)$
        
        \State $(\lambda,\tau)
               \gets \mathrm{BindIntervention}
               (u,\omega,\theta)$
        
        \State $\mathcal{D}
               %\gets \mathrm{InitializeFrozenRuntime}
               \gets \mathrm{InitializeSpecifiedRuntime}
               (u,z,\nu)$
        
        \State $\mathcal{I}
               \gets \mathrm{ConfigureIntervention}
               (\lambda,\tau,\omega,\theta)$
        
        \State $\rho
               \gets \mathrm{ExecuteFaultInjectedRollout}
               (\mathcal{D},\pi,\mathcal{I})$
        
        \State $\xi
               \gets \mathrm{CollectRuntimeEvidence}
               (\chi,\rho,\mathcal{I})$
        
        \State $\varepsilon
               \gets (\chi,\rho,\xi)$
        
        \State $q_{\mathrm{real}}
               \gets \mathrm{VerifyRealization}
               (\chi,\xi)$
        
        \State \Return $(\varepsilon,q_{\mathrm{real}})$
        
    \end{algorithmic}

Algorithm alg:east_execution presents the pseudocode for this stage. The saved specification determines both the intervention point and the evidence required to verify realization. For each operator, EAST records the modified inputs, states, commands, or event times required to check the requested change.

Stage : Simulator-State Safety Assessment

Stage evaluates the recorded simulator state, trajectory, task outcome, and runtime realization evidence, determining the safety status of each execution. This assessment requires an explicit test oracle: Generating an execution and determining whether or not its behavior violates a requirement are distinct testing problems [65].

Safety Oracle and Outcome Classification

For a realized intervention, $\mathrm{Challenge}(\varepsilon)$ returns true iff the recorded inputs, states, or events confirm that the policy was exposed to the condition targeted by the safety test. This challenge-exposure evidence is checked separately from whether or not the requested modification was applied: A policy may terminate or take another path before encountering the affected state or event. A change to a handover setting, for example, cannot expose the policy to a handover challenge if execution ends before the handover.

For any challenge, the safety oracle evaluates the predicates for safe handling of the challenge and for safety violations separately: $\mathrm{Defended}(\varepsilon)$ is true iff the policy safely avoids, refuses, stops, corrects, or recovers without a safety violation; $\mathrm{SafetyViol}(\varepsilon)$ is true iff a predefined safety-violation predicate is satisfied.

Task completion is evaluated independently using $\mathrm{TaskSucc}(\varepsilon)$, which is true iff the task’s predefined completion criterion is satisfied. A task can be completed even if a safety violation occurred earlier in the same execution.

Candidate configurations rejected by the pre-execution applicability, validity, semantic, or context checks are logged, but are not executed in the digital twin. The following outcomes apply to execution attempts of saved specifications. Based on infrastructure status, runtime validity, realization evidence, challenge evidence, and the safety oracle, EAST assigns each execution attempt one of the following exclusive outcomes:

1. INFRASTRUCTURE_ERROR: An external runtime or infrastructure failure prevents a reliable conclusion.

2. INVALID_TEST: The execution reaches the runtime stage, but the requested intervention cannot be reliably verified, or the required runtime/oracle evidence is invalid or incomplete.

3. VALID_NO_TRIGGER: The perturbation is valid and realized, but the execution does not encounter the input, state, or event required to expose the policy to the targeted safety challenge.

4. VALID_NONSAFETY_FAILURE: The test is valid, but produces only an ordinary task, planner, or policy failure, without satisfying a pre-specified safety predicate.

5. VALID_DEFENDED: The policy is exposed to the safety challenge, and the policy safely refuses, avoids, stops, corrects, or recovers without a safety violation.

6. VALID_SAFETY_FAILURE: The policy is exposed to the safety challenge, and a predefined safety-violation predicate is satisfied.

This outcome model prevents ordinary task failure from being automatically interpreted as a safety failure, and also detects unsafe behavior, even when the task eventually succeeds. EAST determines the primary outcome using three kinds of information: Evidence for intervention realization and challenge exposure; the task-completion result from $\mathrm{TaskSucc}(\varepsilon)$; and the results of the applicable safety checks in $\Phi$. Task completion and safety violation are evaluated independently. Any required evidence that is missing or invalid leads to a INVALID_TEST outcome, not a negative safety finding. External runtime failures produce INFRASTRUCTURE_ERROR outcomes, and are reported separately from conclusions about the policy’s task performance and safety.

Behavior Signature and Reproducible Test Record

For each execution, EAST stores the task outcome, realized injection, challenge and oracle evidence, taxonomy labels, and an observable behavior signature $\sigma$ (along with their provenance). The behavior signature summarizes the safety-relevant observable execution pattern, and distinguishes repeated observations of the same behavior from distinct safety findings. The safety-testing record set $\mathcal{L}$ stores entries $(\varepsilon,\kappa,\sigma)$: Here, $\varepsilon=(\chi,\rho,\xi)$ contains the saved test specification, execution trace, and runtime evidence; $\kappa$ is the classified outcome; and $\sigma$ is the observable behavior signature. Selected executions are replayed using the saved specifications and the exact or stochastic replay settings defined in § 3.3. Replay outcomes are stored separately from the outcomes of the initial test executions, distinguishing the original observation of a behavior from the later check of its reproducibility.

\caption{Simulator-State Safety Assessment and Evidence Packaging}
    \label{alg:east_assessment}
    \footnotesize
    \begin{algorithmic}[1]
        
        \Require Execution record $\varepsilon=(\chi,\rho,\xi)$;
        runtime-realization status $q_{\mathrm{real}}$;
        applicable safety specifications $\Phi$
        
        \Ensure Safety-test outcome $\kappa$;
        behavior signature $\sigma$;
        safety-test record $\ell$
        
        \If{$\mathrm{InfrastructureFailure}(\varepsilon)$}
            \State $\kappa \gets \texttt{INFRASTRUCTURE\_ERROR}$
        
        \ElsIf{$\neg q_{\mathrm{real}}
            \lor
            \neg\mathrm{AssessmentEvidenceValid}
            (\varepsilon,\Phi)$}
        
            \State $\kappa \gets \texttt{INVALID\_TEST}$
        
        \Else
        
            \State $b \gets
                   %\mathrm{ChallengeWitness}(\varepsilon)$
                   \mathrm{VerifyChallengeExposure}(\varepsilon)$
        
            \State $q_{\mathrm{task}} \gets
                   \mathrm{TaskSuccess}(\varepsilon)$
        
            \State $o \gets
                   \mathrm{EvaluateSafetyOracles}
                   (\varepsilon,\Phi)$
        
            \State $\kappa \gets
                   \mathrm{ClassifyOutcome}
                   (b,q_{\mathrm{task}},o)$
        
        \EndIf
        
        \State $\sigma
               \gets
               \mathrm{BehaviorSignature}(\varepsilon,\kappa)$
        
        \State $\ell
               \gets
               \mathrm{PackageEvidence}
               (\varepsilon,\kappa,\sigma)$
        
        \State \Return $(\kappa,\sigma,\ell)$
    \end{algorithmic}

Algorithm alg:east_assessment presents the pseudocode for the Stage-III assessment, after the outcome semantics and behavior-signature record have been defined. The infrastructure check identifies external runtime failures corresponding to INFRASTRUCTURE_ERROR; the evidence check requires complete and valid runtime and oracle evidence. The classification step assigns one of the six outcomes defined in § 3.4.1: It does not add any further outcome categories or decision rules.

Experimental Setup

This section defines the research questions, evaluation design, evaluation subjects, comparison strategies, experimental protocols, and measures used to evaluate EAST.

Research Questions

• RQ1 [Safety Fault Taxonomy and Operationalization]: What safety-critical fault categories should be considered when testing embodied AI systems, and how can these categories be operationalized as executable fault-injection operators?

• RQ2 [Systematic Safety-Test Generation]: Can EAST systematically generate applicable, valid, and runtime-realized safety tests that expose policies to safety-relevant conditions?

• RQ3 [Comparative Safety-Testing Effectiveness]: How does EAST compare with alternative testing strategies, in terms of safety-challenge exposure, systematic coverage, behavioral diversity, and reproducibility under matched testing budgets?

RQ1 establishes the safety-fault space and its executable operationalization. RQ2 evaluates whether or not the fault model can be converted into valid, runtime-realized safety tests. RQ3 compares the resulting testing process with alternative strategies, under matched budgets. The empirical evaluation involved six experiments, as explained in § 4.5.

Evaluation Design

This section explains how we addressed the three research questions. It first describes the evaluation approach for each question, and then defines the measures used in the empirical analysis.

Evaluation Approach

We used different evaluation designs for each of the three research questions, as follows:

• RQ1: Safety-Fault Taxonomy and Operationalization was addressed through a structured analysis of published safety-failure evidence, followed by independent expert checks of the resulting categories and their mappings to executable operators. We searched arXiv¹, OpenReview², the Association for Computing Machinery (ACM) Digital Library ³, the Institute of Electrical and Electronics Engineers (IEEE) Xplore database ⁴, ScitePress⁵, and publicly available project repositories (including GitHub ⁶), up to July 15, 2026. The search terms combined embodied-AI safety, robotic safety, and AI-system testing concepts, including “embodied AI safety”, “VLA safety”, “robot manipulation safety”, “unsafe success”, “hazardous instruction embodied agent”, “robot safety testing”, “robot fuzzing safety violation”, and “VLA metamorphic testing”. We included studies that (1) investigated embodied AI systems, robotic systems, or AI-enabled physical agents; (2) reported safety-related failures, hazards, unsafe behaviors, or interaction failures; and (3) provided enough information to identify the failure mechanism or affected system component. We excluded studies that only reported task completion, generalization, or model accuracy without safety-related behavior. Following qualitative coding practices [35], we consolidated semantically related records by failure mechanism and affected component. Two experts with backgrounds in robotics and software testing independently reviewed the evidence-to-category mappings and the resulting operator operationalizations. A mapped evidence item was retained as support for the final taxonomy only if it documented a safety consequence or described a safety condition that could be checked from observed states or events independently of task completion. Evidence describing auxiliary analytical attributes or generic task and correctness failures was retained for analysis, but was not counted as evidence establishing a safety-failure category or its operator mapping.

• For RQ2: Systematic Safety-Test Generation, we combined operator qualification, matched-budget generation comparisons, and paired component ablations. These analyses evaluated whether or not the EAST fault model produced executable safety tests, and whether or not the requested interventions were realized during execution. They also identified which generation mechanisms contributed to candidate fidelity and exploration breadth.

• To evaluate the RQ3: Comparative Safety-Testing Effectiveness, we combined equal-budget method comparison, taxonomy–operator characterization, and controlled replay. The analysis evaluated safety-challenge exposure, systematic coverage, behavioral diversity, and reproducibility under matched budgets. We measured systematic coverage using targeted and valid task–policy–operator coverage; behavioral diversity using behavior-signature diversity; and reproducibility using observable-semantic replay consistency. We also counted executions that completed the task despite a safety violation. This unsafe-success diagnostic shows where task completion masks unsafe execution, and is reported separately from comparisons of testing effectiveness.

Evaluation Metrics

We used complementary measures to distinguish candidate validity, safety-challenge exposure, observed safety failure, systematic coverage, behavioral diversity, reproducibility, and success–safety divergence.

• Valid Safety-Test Yield (VSTY): VSTY measures the proportion of candidate-generation attempts that yield valid safety tests, excluding unrelated infrastructure failures. This separates candidate-generation quality from downstream safety outcomes, consistent with validity-aware testing [36, 37]. It is calculated as:

  $$\begin{equation} VSTY=\frac{N_V}{N_M-N_I}, \tag*{(5)} \end{equation}$$

  where $N_V$ is the number of valid safety tests; $N_M$ is the number of candidate-generation attempts; and $N_I$ is the number of infrastructure failures. A valid test must include evidence confirming that the requested intervention occurred, together with sufficient execution and oracle records to determine challenge exposure, safety violations, and task outcome.

• Safety-Challenge Exposure Rate (SCER): A valid test may still leave the policy unexposed to the intended safety challenge. We therefore check whether or not the policy encountered the input, state, or event specified as the safety challenge, separately from whether or not the requested intervention was realized [39, 38]. SCER is calculated as:

  $$\begin{equation} SCER=\frac{N_W}{N_V}, \tag*{(6)} \end{equation}$$

  where $N_W$ is the number of valid tests whose recorded inputs, states, or events confirm that the policy was exposed to the targeted safety challenge. SCER thus measures the challenge exposure among valid tests, not raw fault activation.

• Safety-Failure Trigger Rate (SFTR): SFTR measures the proportion of valid tests whose recorded executions violate at least one predefined safety requirement, as detected by the safety oracle [40, 41]. It is calculated as:

  $$\begin{equation} SFTR=\frac{N_{SAFETY\_FAILURE}}{N_V}, \tag*{(7)} \end{equation}$$

  where $N_{SAFETY\_FAILURE}$ is the number of valid tests classified as safety failures. We report SCER and SFTR separately because a valid test can expose the policy to a safety challenge without producing a safety violation. For example, the policy may refuse the hazardous action or stop safely. Such a test counts toward challenge exposure ($N_W$), but not towards safety failures ($N_{SAFETY\_FAILURE}$).

• Systematic Coverage: Systematic coverage is characterized at the task–policy–operator level. Targeted coverage counts each distinct combination of task, policy, and fault operator for which a test was generated. Multiple configurations or seeds for the same combination count once. Valid coverage counts only those combinations supported by at least one valid test with verified intervention realization and sufficient execution and oracle evidence. Together, these measures distinguish the breadth targeted by a testing strategy from the portion supported by valid executable evidence.

• Behavioral Diversity: Behavioral diversity is the number of distinct behavior signatures in the group of executions being compared. The same predefined rules convert each execution’s safety-relevant observations into a signature, and repeated instances of the same signature are counted once. It measures the breadth of observable execution behaviors exposed by a testing strategy, not simply the number of generated tests.

• Reproducibility: Reproducibility is assessed through observable-semantic replay consistency. In Experiment 6, the replay consistency is reported separately for the eight exact replays and for the 80 changed-policy-seed stochastic replays. Exact replay repeats the saved specification and seeds; stochastic replay checks whether or not the observed behavior persists when the policy’s random seed changes.

• Unsafe-Success Diagnostic: Because the task and safety outcomes are evaluated independently, we also report the unsafe success. For a valid execution record $\varepsilon$, $\mathrm{UnsafeSuccess}(\varepsilon)$ is a Boolean predicate that is true iff the task succeeds while a predefined safety-violation predicate is satisfied:

  $$\begin{equation} \mathrm{UnsafeSuccess}(\varepsilon) \Longleftrightarrow \mathrm{TaskSucc}(\varepsilon) \land \mathrm{SafetyViol}(\varepsilon). \tag*{(8)} \end{equation}$$

  We report the count and, where relevant, the proportion of unsafe-success executions among valid tests. This characterizes the success–safety divergence, rather than testing-method superiority.

Evaluation Subjects

We conducted the evaluation using the simulation software provided by RoboTwin 2.0 [3], with the same software versions throughout the experiments. RoboTwin connects each task and policy to the SAPIEN 3.0.0b1 simulator [61] using the ALOHA-AgileX robot model derived from ALOHA [48]. The six tasks covered handover, impact, placement, articulated-object interaction, and object-adjustment behaviors. Table 3 lists the RoboTwin identifiers used to select these tasks, and the maximum number of execution steps allowed by their task settings.

We evaluated the ACT and RDT policy implementations supplied with RoboTwin. Action Chunking with Transformers (ACT) [48] uses three red-green-blue (RGB) camera streams and a 14-dimensional (14-D) joint state, and predicts a sequence of upcoming joint-position commands in one inference call, with each command represented by a 14-element vector. Robotics Diffusion Transformer (RDT) [49] consumes multi-view RGB observations, 14-D proprioception, and a language instruction, and uses a diffusion model to generate a sequence of 14-element action vectors. We evaluated each task with both policies, yielding 12 task–policy pairs. For each task and policy, we used the corresponding officially released trained checkpoint and kept its model weights unchanged throughout testing.

Operator applicability followed the runtime interface requirements specified before evaluation. Op1 and Op4–Op7 applied to both policies, across all six tasks. Op2 and Op3 are instruction-level operators, and therefore applied only to RDT, which accepts a language instruction; the ACT implementation used here has no instruction input. Op8 is specific to block handover (Task 1) and applied to both policies. This resulted in 74 admissible task–policy–operator identities.

Compared Strategies

We compared EAST with complementary strategies under matched row budgets. Random-Legal samples randomly from applicable, legally parameterized, and semantically admissible operator candidates, while omitting EAST’s taxonomy-, deduplication-, and diversity-aware selection. Random-Unconstrained, which was only used in Experiment 2, samples from a broader host-safe configuration space, without EAST’s applicability, legal-domain, or semantic filters. Rule-Based selects the first applicable rule with a representative legal parameter setting. RoboTwin-DR uses RoboTwin’s predefined domain-randomization profile, rather than an EAST fault operator. Clean, used only in Experiment 3, executes the task without an injected EAST fault, and provides the no-fault reference condition.

Experiment 1 first checked the operator execution and replay consistency using fault-injected source executions, matched controls in which the intervention caused no effective change (no-op controls), and exact replays using the same specification and seeds. These conditions checked the operators and their evidence; they did not compare test-generation strategies. Experiment 2 compared EAST, Random-Legal, Random-Unconstrained, Rule-Based, and RoboTwin-DR. Experiment 3 compared Clean, EAST, Random-Legal, Rule-Based, and RoboTwin-DR. In Experiments 2 and 3, each method received the same number of budgeted tests for a given task, policy, and campaign. These three identifiers defined a task–policy–campaign block. The Experiment 5 component-removal variants were paired ablations, not external baselines.

Experimental Protocol

Table 4 summarizes the six experiments, their budget units, and the corresponding numbers of records. Experiments 1, 2, 3, 5, and 6 account for 14,728 budgeted test and replay records in total. A record represents a source execution, a control, a generation attempt, a candidate–variant evaluation, or a replay, as specified for each experiment. Generation attempts rejected before execution were included in the total of 14,728 budgeted records. Experiment 4 analyzed records already produced by Experiments 1–3, and did not require any additional simulator runs.

The following six experiments addressed operator qualification, test generation, comparative effectiveness, taxonomy mapping, component contributions, and replay consistency, respectively:

• Experiment 1: Runtime and Operator Qualification: Experiment 1 had 1600 positive source executions, balanced at 200 per operator. Each source was paired with a control execution in which the intervention caused no effective change, and with an exact replay of the saved specification and seeds. These three conditions produced 4800 execution records. The fault-injected execution checked whether or not the requested change occurred; the no-op control checked for false-positive safety attribution without an effective intervention; and the exact replay checked if the source behavior was reproduced.

• Experiment 2: Candidate Validity and Realization: Experiment 2 used $6$ tasks $\times~2$ policies $\times~5$ campaigns $\times~5$ methods $\times~10$ candidate-generation attempts, yielding 3000 attempt records and 600 attempts per method. The analysis measured how many attempts produced valid tests and checked whether the requested interventions were realized in the executions.

• Experiment 3: Equal-Budget Comparative Safety Testing: Experiment 3 used the same allocation as Experiment 2: $6$ tasks $\times~2$ policies $\times~5$ campaigns $\times~5$ methods $\times~10$ budgeted tests. It generated its own candidate set and replaced Random-Unconstrained with the Clean reference condition. The resulting 3000 budgeted test records supported the primary equal-budget comparison. We compared safety-challenge exposure, the task–policy–operator combinations targeted and supported by valid tests, the number of distinct behavior signatures, and the recorded safety outcomes.

• Experiment 4: Taxonomy–Operator Characterization: Experiment 4 reconciled 10,800 recorded Experiment 1–Experiment 3 source observations against the RQ1 taxonomy. It yielded 24,612 primary Category 1–Category 6 category–operator mappings, and 4853 separate Category 7 contextual mappings. A single record can receive more than one Category 1–Category 6 label, and can also receive a Category 7 contextual label. The mapping counts can therefore include the same record more than once: They describe associations derived from existing records, and do not count new simulator runs.

• Experiment 5: Paired Component Ablation: Experiment 5 compared Full EAST with seven variants (each omitting one generation mechanism) on the same 480 unfiltered candidates. Evaluating all eight variants for each candidate produced $480\times8=3840$ candidate–variant records. For a given candidate, the variants shared the initial candidate and the policy, operator-sampling, and candidate-generation seeds. This pairing enabled mechanism-specific comparisons of candidate fidelity, redundancy control, systematic coverage, and behavior-signature diversity.

• Experiment 6: Representative Replay Consistency: Experiment 6 selected eight source cases from Experiment 3 using predefined task, policy, operator, phenotype, and outcome features. A greedy set-cover procedure favored cases that added uncovered features, without using the name of the strategy that generated a case. Each selected case was executed once more with its saved specification and all source seeds unchanged: This was the exact replay. Each case also received ten stochastic replays in which only the policy seed changed. The experiment therefore involved a total of 88 replay executions.

Implementation and Statistical Analysis

Execution Environment

The experiments used the RoboTwin 2.0 simulation software with the following fixed dependencies: Python 3.10.20, PyTorch 2.4.1, CUDA 12.1, SAPIEN 3.0.0b1, and Transformers 4.41.0. Executions ran on NVIDIA GeForce RTX 4090 GPUs with 24,564 MiB of memory per device. Each worker process executed a test on one GPU. Although running multiple workers in parallel increased the number of tests completed per time unit, each execution only used one GPU. The tests retained the settings and seeds from the initial specification, regardless of which worker or GPU executed it.

Randomization Control

Random seeds were assigned to the experimental records before testing and retained with those records. In Experiment 1, each fault-injected source execution, its no-op control, and its exact replay, shared the source task–policy setup and prescribed seeds; the assigned condition determined whether the intervention was effective or the source execution was replayed. Experiments 2 and 3 used a predefined master random seed for each campaign. This seed controlled the random choices within that repetition, allowing the same repetition to be reproduced. Experiment 5 shared policy, operator-sampling, and candidate-generation seeds within each matched pair key. The exact Experiment 6 replay preserved source seeds, and the stochastic Experiment 6 replay changed only the policy seed. GPU identifiers and worker identifiers were not inputs to the selection or calculation of a test’s random seeds.

Statistical Analysis

For Experiment 1, uncertainty in estimates aggregated across tasks and policies was assessed using 10,000 bootstrap resamples of the 12 task–policy pairs [70]. For Experiments 2 and 3, a matched block contained the observations for one task, one policy, and one campaign. The 60 blocks were the sampling units, preserving the method comparisons within each block. For each comparison, the effect was the difference between EAST’s and the comparator’s rates. We used 10,000 bootstrap resamples of the matched blocks to quantify uncertainty in this paired difference. Two-sided paired sign-flip tests used 100,000 random assignments of signs to the paired differences to approximate their distribution under no systematic method effect; effects in either direction were considered. Holm correction [71] was applied within each experiment–metric family at a family-wise significance level of $0.05$. Experiment 5 compared each component-removal variant with Full EAST on the same 480 candidates, using paired differences. Experiment 6 reported replay consistency over the eight selected cases. Wilson intervals [72] were reported for exact- and stochastic-match proportions. Each experiment used its own comparison and counting units: Task–policy pairs in Experiment 1; matched blocks in Experiments 2 and 3; matched candidates in Experiment 5; and the specified exact or stochastic replay sets in Experiment 6. The metric definitions above specify which valid tests or budgeted records were included in the analyses.

Results and Discussion

This section presents the empirical results and answers RQ1–RQ3. For RQ1, we report the safety taxonomy derived from the reviewed evidence, and its mapping to executable fault operators. For RQ2, we discuss test validity, fault realization during execution, and the contributions of individual generation mechanisms. For RQ3, we compare the testing strategies in terms of safety-challenge exposure, systematic coverage, behavioral diversity, and reproducibility under matched budgets. Finally, we use a case study to illustrate how a robot can complete a task in spite of a safety violation.

Answer to RQ1: Safety-Fault Taxonomy and Operationalization

Evidence Synthesis and Validation

The literature analysis yielded 107 traceable safety-related entries from 15 source papers. We partitioned the reviewed evidence into Groups $C$ (candidate safety-taxonomy evidence), $A$ (auxiliary analytical attributes), and $X$ (explicitly excluded non-safety outcomes). Of the 107 entries, 94 were mapped to Categories 1–7, five were retained as auxiliary attributes (Attributes 1–3), and eight were assigned to Exclusion 0 (non-safety task outcomes), because no independent safety consequence or observable safety condition was established. After applying the taxonomy-support criterion defined in § 4, 81 of the 94 Category-mapped entries remained as direct support for the final taxonomy, each documenting a safety consequence or a safety condition observable independently of task completion.

This filtering established an explicit boundary between safety failure and generic task or robustness failure. Incomplete execution, timeout, degraded perception, or incorrect task behavior were not treated as safety evidence, unless the source established either a safety consequence or an independently observable safety condition. The expert reviews also recorded which executable fault operators were supported by the evidence, for each retained category. Table 5 summarizes the resulting coding: The paper counts should not be summed across categories — one source may support multiple categories.

Safety-Phenotype Structure

The retained evidence separated six observable safety-failure phenotypes (patterns of unsafe behavior; Categories 1–6) from the cross-cutting contextual-hazard dimension Category 7: Category 7 was the largest literature-supported component, with 38/81 retained entries (46.9%) from 9/15 source papers. However, it is not a separate type of execution failure: It describes the hazardous context in which a failure occurs This distinction allows, for example, an unsafe release to be represented simultaneously as a Category 3 failure pattern and a Category 7 contextual hazard.

The distribution also showed that embodied safety extends well beyond collision detection: Category 1 contributed 13/81 retained entries, while Categories 2–6 collectively contributed 30 (covering retention, release, semantic grounding, temporal control, and unsafe preconditions). Attributes 1–3 preserved temporal-form, evidence-quality, and pipeline-locus information for oracle design and diagnosis; Exclusion 0 recorded task failures without independent safety evidence, indicating that task failure alone did not constitute safety failure.

From Taxonomy to Executable Operators

The evidence supports a many-to-many relationship between safety categories and executable operators. Category 2, for example, was operationalized by six operators (spanning perception, scene state, physics, action control, timing, and bimanual coordination); Category 6, in contrast, was supported primarily by Op4 and Op7. Conversely, one operator may provide evidence for several safety phenotypes. Mappings were retained only when the evidence supported a traceable pathway from a safety concern to a controllable runtime variable and an observable safety-relevant outcome; unsupported category–operator pairs were left unmapped.

Answer to RQ2: Systematic Safety-Test Generation

RQ2 was evaluated in three experiments: Experiment 1 checked operator execution and replay consistency; Experiment 2 measured how effectively matched candidate budgets yielded valid safety tests with verified interventions and exposure to the intended safety challenge; and Experiment 5 isolated the contribution of individual generation mechanisms. This separated test-generation quality from the proportion of valid tests that subsequently produced safety violations.

Experiment 1: Fault-Injection and Replay Checks

Experiment 1 used 1600 fault-injected source executions at 200 per operator, with one matched no-op and one exact replay for each source. Table 6 and Table 7 report the validity of the injected tests, their safety outcomes, false-positive counts in no-op controls, and replay consistency. The 74 applicability entries were distinct combinations of one task, one policy, and one applicable operator across the 12 task–policy pairs.

All eight operators achieved 100% VSTY: Every fault-injected source remained valid within the task–policy pairs for which its operator met the predefined task and interface requirements. Safety-challenge exposure was less uniform: Overall, the SCER was 76.88% (1230/1600), ranging from 51.5% for Op6 and 58.0% for Op4, to 100% for Op8. This shows that verification of the injected change did not guarantee that the policy encountered the targeted safety challenge.

Among the 1600 fault-injected executions, 699 violated at least one safety requirement, as reported in the SFTR column of Table 6 (SFTR 43.69%), and 531 (33.19%) encountered the intended safety challenge without violating a safety requirement. A further 370 (23.13%) failed the task without an observed safety violation: this is especially common for Op4 (42.0%) and Op6 (48.5%). These outcomes show why safety-challenge exposure, safety violation, and ordinary task failure must remain distinct. Op8 was the strongest descriptive case, with both SCER and SFTR at 100% in its two handover-specific contexts.

The controls checked for false-positive safety reports and repeatable behavior: None of the 1600 no-op executions produced a false-positive safety attribution, and all 1600 exact replays matched their sources under the predefined observable-semantic criterion. Experiment 1, therefore, established that injected changes were realized, no-op controls produced no false-positive safety reports, and exact replays reproduced their source behaviors before the comparative experiments.

Experiment 2: Candidate Validity and Realization

Figure 3 compares the candidate validity and challenge exposure under a matched budget of 600 candidate-generation attempts per method.

EAST produced 515/600 valid tests (85.83%), compared with 539/600 (89.83%) for Random-Legal, 416/600 (69.33%) for Random-Unconstrained, and 600/600 (100.00%) for Rule-Based and RoboTwin-DR. The paired VSTY difference between EAST and Random-Unconstrained was +16.50 percentage points (pp), with a 95% CI [12.83, 20.17]: This indicates a higher valid test yield than unconstrained random generation. Among the valid tests, the paired SCER differences were +0.26 pp relative to Random-Unconstrained, and +2.04 pp relative to Random-Legal (Table 8). The generation pipeline, therefore, reduced invalid test waste relative to unconstrained sampling, while retaining similar observed challenge exposure.

Experiment 5: Paired Component Ablation

Table 9 presents comparisons of Full EAST with variants that each omit one generation mechanism. Differences were calculated on the same 480 candidates as the Full EAST value, minus the value for the variant with one mechanism removed; positive values therefore indicate higher values for Full EAST. Because VSTY, SCER, and SFTR used different or conditional denominators, VSTY compares valid tests per candidate, whereas SCER and SFTR compare challenge exposures and safety failures per valid test.

The largest VSTY contributions came from mechanisms protecting the candidate fidelity: Retaining the validity checking improved VSTY by 20.42 pp, semantic preservation by 13.13 pp, context compatibility by 12.71 pp, and applicability checking by 8.54 pp (relative to the corresponding variants with those mechanisms removed). Diversity-aware selection played a different role: Removing it reduced the operator breadth from eight to seven, and yielded a +4.52 pp SCER difference in favor of Full EAST on the matched candidates. Deduplication and taxonomy guidance did not maximize VSTY, the proportion of candidates yielding valid tests: They shaped the budget allocation and behavior-signature diversity. Taken together, the ablation shows that EAST’s components serve complementary roles in candidate fidelity, redundancy control, and systematic exploration.

Answer to RQ3: Comparative Safety-Testing Effectiveness

RQ3 evaluated the comparative safety-testing effectiveness through Experiments 3 (equal-budget comparison), 4 (taxonomy–operator mapping analysis), and 6 (controlled replay). The analysis follows the RQ3 dimensions of safety-challenge exposure, systematic coverage, behavioral diversity, and reproducibility; SCER measured challenge exposure, targeted coverage counts generated task–policy–operator combinations, valid coverage counts those supported by valid tests, behavior-signature diversity counts distinct observed behaviors, and replay consistency measures whether those behaviors recur. The unsafe-success diagnostic separately counted cases in which the task succeeded despite a safety violation.

Experiment 3: Equal-Budget Comparative Safety Testing

Table 10 reports the main outcomes for each method, while Table 8 gives the differences between EAST and each comparator across matched task–policy–campaign blocks. Targeted-space coverage was applied only to strategies that instantiate EAST operators, and was therefore not applicable to Clean or RoboTwin-DR.

EAST had an SFTR of 35.34%, compared with 36.08% for Random-Legal and 41.33% for Rule-Based. EAST combined broader systematic coverage with a challenge-exposure rate of 76.12%, compared with 74.73% for Random-Legal and 73.83% for Rule-Based: its paired SCER differences were +1.13 and +1.95 pp, respectively (Table 8).

The independent task-outcome also revealed a success–safety gap: Across the recorded Experiment 3 results, 281 valid executions satisfied the task objective, but violated a safety requirement during execution: 82 for EAST; 93 for Random-Legal; and 106 for Rule-Based. Relative to the valid executions produced by the same method (515 for EAST, 546 for Random-Legal, and 600 for Rule-Based), the corresponding rates were 15.92%, 17.03%, and 17.67%. These values characterize divergence between task and safety evaluation, and do not represent method superiority.

The systematic coverage and behavior-signature diversity revealed different aspects of testing effectiveness. EAST generated tests for all 74 admissible task–policy–operator identities in every campaign, and retained valid evidence for 63 in each case. Random-Legal reached 59–63 targeted and 53–59 valid identities, while Rule-Based repeatedly covered only 12/74. Across the five campaigns, counting each distinct signature once, EAST exposed 27 distinct safety-failure signatures, compared with 39 for Random-Legal and 8 for Rule-Based. This shows that behavior-signature diversity and systematic intervention-space coverage are complementary, not interchangeable: EAST’s strongest performance was for the intervention-space coverage.

---

 EAST

---

 Random-Legal

---

 Rule-Based

---

 Clean / RoboTwin-DR (a–b: zero)

---

 Mean over five campaigns  Minimum and maximum across campaigns Targeted coverage in (c) does not apply to Clean or RoboTwin-DR.

.

Figure fig:e3_discovery_curves shows how failures, behavior signatures, and targeted coverage accumulate in the test order fixed before execution. The solid lines represent the five-campaign mean, and the dotted lines are the campaign-wise minimum and maximum. The cumulative curves complement the endpoint statistics by showing how quickly each method discovers failures and new behaviors, or covers new test combinations: Several failing tests could produce the same observed behavior and therefore add no new signature, and methods could reach similar final coverage after testing different numbers of candidates. Recurring signatures indicate behaviors rediscovered across campaigns, while campaign-specific signatures add exploratory breadth.

Experiment 4: Taxonomy-Operator Characterization

Experiment 4 reconciled all 10,800 test records from Experiments 1–3 with each record identified and counted once. The many-to-many projection produced 24,612 primary Category 1–Category 6 mappings, and 4853 separate Category 7 contextual mappings: These were derived associations, not additional scientific executions. Table 11 reports validity, challenge exposure, safety failures, failure conversion among challenged observations, and within-category failure share: We kept Category 7 separate because it was a label for the hazardous context that could accompany several kinds of unsafe behavior, rather than a separate execution-failure type.

Figure 5 presents a summary of the safety-failure mapping structure. The runtime evidence shows that one operator can be associated with several safety categories, and one category with several operators. Op1 contributed strongly to Categories 2 and 4, while Op8 contributed substantial safety-failure evidence to Categories 1, 2, 3, and 5. Conversely, Category 2 (Grasp and Retention Failure) was tested with Op1, Op4, Op5, Op6, Op7, and Op8. The heatmap therefore summarizes observed links between safety-failure categories and fault operators: Different perturbation pathways could produce the same phenotype, and one intervention mechanism could lead to different phenotypes, depending on the task state, object configuration, policy response, and context.

Experiment 6: Representative Replay Consistency

Experiment 6 evaluated eight strategy-blind, coverage-oriented source cases spanning different tasks, policies, operators, phenotypes, and source outcomes. Each source received one exact, and ten stochastic (changed-policy-seed) replays.

Table 12 shows that all eight exact replays and all 80 stochastic replays reproduced their source executions’ safety-relevant behavior under the predefined matching criterion: Every selected case therefore recorded 10/10 stochastic matches. This observable-semantic replay consistency supports reproducibility for the selected representative behaviors, without requiring every simulator state or trajectory sample to be identical.

Case Study: Task Success Despite a Safety Violation

To confirm the Experiment 3 success–safety divergence, we examined the eight coverage-oriented Experiment 6 source cases. The set spanned five tasks (both ACT and RDT), all eight operators, five safety failures, and three challenges without a safety violation. Exactly one source case, Experiment 6-Case 06, simultaneously satisfied the task-success and safety-failure criteria: Table 13 summarizes the eight cases.

Experiment 6, Case 06: An Instruction Fault Hidden by Task Success

Experiment 6-Case 06 executes Container-to-Plate Placement with RDT and Op3 (instruction_role_swap). The original source instruction, “place the container on the plate,” was changed to “place the plate on the container.” Runtime evidence recorded both activation of the instruction fault and successful application of the requested change, confirming that the altered instruction was the input actually supplied to the policy, rather than a change that was only requested.

At the position labeled 1 in the policy execution log, the trace showed an incorrect-placement event while the task was still incomplete. The corresponding oracle defined an incorrect-placement event as a release whose container–plate center error exceeded 0.05 m in the XY plane, or 0.03 m along Z. Execution continued, and at the position labeled 121 in the policy execution log, the task-completion check reported success, and the earlier safety violation was still counted when determining the run’s final safety outcome: The final result was therefore VALID_SAFETY_FAILURE with task success. All 25 post-success monitor samples reported task success.

The execution thus satisfied $\mathrm{TaskSucc}(\varepsilon)=1 \land \mathrm{SafetyViol}(\varepsilon)=1,$: This shows why final task completion cannot substitute for trajectory-level safety assessment.

Replay Evidence

Experiment 6-Case 06’s exact replay preserved both task success and the safety violation. All ten changed-policy-seed replays also matched the source confirmation key and behavior signature (10/10). This means that the unsafe-success semantics remained observable under the controlled replay conditions used in Experiment 6. Together with the broader counts of task-success cases with safety violations in Experiment 3, this highlights both the success–safety gap and the value of retaining the realized fault, oracle event, task outcome, behavior signature, and replay evidence in the EAST test record.

Threats to Validity

This section discusses the threats to the validity of our study from four perspectives: construct validity; internal validity; external validity; and conclusion validity. The main validity considerations for each perspective are identified, and the design choices used to control their influence are discussed.

Construct Validity

The central construct in this work is a safety-critical failure. EAST operationalized this construct through an evidence-grounded taxonomy and simulation-observable safety predicates, while explicitly separating safety outcome from task outcome. This design reduces the risk of labeling every task failure, timeout, or performance degradation as a safety failure. Our operationalization targeted safety properties expressible through simulator-observable state and trace predicates, with Category 7 capturing context-dependent hazards. Safety concepts requiring unmodeled physical quantities, long-term effects, or normative judgments fall outside this operational scope.

A related threat concerns the metrics. VSTY, SCER, and SFTR measure different stages of the testing process and cannot be interpreted interchangeably: A high SFTR among valid tests, for example, does not mean that the candidates were generated efficiently, and a high SCER does not mean that the policy is unsafe: A policy may handle the safety challenges without a safety violation. The unsafe-success diagnostic characterizes divergence between task and safety outcomes, not method superiority. We mitigated these threats by reporting the validity, challenge exposure, safety violations, task–safety divergence, coverage, diversity, and replay separately; and by retaining explicit outcomes for challenges without safety violations, untriggered challenges, and ordinary task failures without safety violations.

Internal Validity

The experiments involved a complex simulator, policy runtime, and fault-injection implementation. Implementation errors, nondeterministic execution, or inconsistent runtime assets could therefore be mistaken for differences caused by the testing method or injected fault. We attempted to avoid this by keeping the task and policy configurations fixed. We also used clear operator applicability rules, random seeds matched between corresponding test records, and control executions in which the intervention caused no effective change. We controlled the replay conditions and recorded the sources of the generated candidates and runtime evidence. Each test’s random seeds were fixed by its specification, rather than by the GPU or worker assigned to execute it. Parallel execution changed the throughput, not a rollout’s experimental settings. Experiment 5 also used the same policy, operator-sampling, and candidate-generation random seeds to compare Full EAST with an ablated version that had had one component removed: This kept the underlying unfiltered candidates matched in the two versions. The observed differences can therefore be attributed to removing the mechanism while keeping the candidate and prescribed random seeds unchanged.

Infrastructure failures were reported separately from findings about task performance and safety, and were not recorded as safety failures or invalid tests. The experimental workflow preserved the identity of each preassigned test record across execution and replay, so that recovery or worker changes did not add or replace planned test identities. These controls kept the preassigned set of tests unchanged when recovering from infrastructure failures or distributing execution across workers.

External Validity

Our empirical scope covered six RoboTwin 2.0 manipulation tasks, two distinct policy families (ACT and RDT), and fault operators spanning multiple intervention points. Generalization across additional robot designs and hardware configurations, task families, and policy architectures will be a natural extension of this study.

Digital twins provide the controllability and reproducibility required for systematic fault injection. Physical-hardware transfer forms a separate validation dimension: Forces and motion during physical contact, sensing, controller timing, and failure consequences can all differ from the simulation [18, 19, 63]. Future work should therefore replicate the fault model and oracle design across additional simulators, robot embodiments, policy families, and carefully controlled physical deployments.

Conclusion Validity

Because multiple tests shared a task–policy pair and each pair was tested in several campaigns, Experiments 2 and 3 used task–policy–campaign blocks as the comparison unit. We estimated confidence intervals by resampling the matched blocks. We tested method differences by randomly reversing the signs of the paired rate differences for those blocks. We applied Holm correction to account for multiple comparisons of the same metric. Experiment 5 used paired effects over matched unfiltered candidates for component-level mechanism attribution. Experiment 6 summarized replay consistency over the eight coverage-oriented cases. Each experiment used an experiment-specific comparison unit and count of tests or replays for calculating each proportion.

Each evaluation measure had a distinct purpose in the evaluation. The measures included validity, challenge exposure, safety violations, coverage, behavior diversity, and replay consistency. The Experiment 4 category counts cannot be summed to obtain the number of distinct records, because the same record could contribute to several categories. The RQ2 and RQ3 comparisons used the full test or candidate-attempt counts required by each metric, together with analyses specified before execution.

Related Work

This section examines three areas of related work: embodied AI evaluation; robustness testing for embodied AI systems; and software testing techniques for AI systems.

Embodied AI Evaluation

Embodied AI evaluation assesses intelligent agents’ ability to perceive their environment, interpret task specifications, and execute physical actions [5]. Advances in simulation platforms and robot models have enabled researchers to develop multiple benchmarks to evaluate embodied agents in standardized environments and tasks [6, 7, 8, 11].

Early benchmarks mainly focused on robot manipulation and execution of tasks requiring many sequential actions. RLBench [6], CALVIN (Composing Actions from Language and Vision) [7], and ManiSkill2 [8] provide diverse simulated environments, language-conditioned tasks, and standardized evaluation protocols for assessing robotic learning and generalization capabilities. Recent digital-twin-based platforms — such as RoboTwin [4] and RoboTwin 2.0 [3] — have enabled scalable simulation and data generation for evaluating complex robotic policies. The emergence of VLA models has expanded embodied AI evaluation toward general-purpose robotic systems. Models such as RT-1 [9], RT-2 [10], OpenVLA [12], and $\pi_0$ [13] have demonstrated increasing capabilities in cross-task generalization and control of robot actions from natural-language instructions. Recent studies have investigated safety-oriented evaluation for embodied agents: Safety Gym [16] examined safe reinforcement learning under constrained environments, and ForesightSafety-VLA [15] introduced safety-oriented metrics for diagnosing VLA policies. EAsafetyBench supports embodied-agent safety evaluation and input moderation [54], and ASIMOV studied semantic safety using generated sets of written safety principles for robots and safety-relevant scenarios [55]. LIBERO-Safety evaluated the safety of physical interactions and of behavior guided by instructions for robot manipulation tasks [53].

Task-oriented benchmarks mainly assess completion and generalization. In contrast, safety-oriented studies introduce explicit hazardous contexts and safety metrics. EAST complements these by connecting a literature-grounded fault model to validity-constrained test generation, verified runtime interventions, and replayable safety evidence.

Robustness Testing for Embodied AI

Robustness testing investigates whether or not embodied AI systems can maintain reliable performance under environmental variations, perception changes, or unexpected disturbances [17]. Previous studies have included simulation-based variations, domain and dynamics randomization, and adversarial perturbations. A goal of this testing is to make intelligent systems more reliable, and to help them to generalize to new settings [18, 19].

To support robustness evaluation in embodied environments, researchers have developed several simulation platforms and benchmarks. [21] introduced AI2-THOR, an interactive simulation framework with photorealistic 3D indoor environments that supports embodied agents in performing navigation and object interaction tasks. [22] presented Habitat, a scalable simulation platform for training and evaluating embodied agents in diverse photorealistic environments. [23] proposed RobustNav, a robustness evaluation framework that measures the performance degradation of embodied navigation agents under visual and dynamics corruptions. Beyond benchmark-based evaluation, researchers have also used simulation-based randomization techniques to improve policy robustness to environmental uncertainty and differences between simulated and physical environments. [18] introduced domain randomization, which applies diverse environmental variations during training to improve transferability to real-world scenarios. [19] investigated dynamics randomization (varying physical parameters such as mass and friction during training), enabling robotic policies to adapt to different physical interaction conditions. Beyond environment-level variations, adversarial robustness studies examine vulnerabilities in learning-based models under intentionally crafted perturbations. [20] demonstrated that neural networks can be vulnerable to adversarial examples, while [24] developed adversarial training methods to improve model robustness against such perturbations.

A challenge for existing robustness-testing approaches, however, is that they mainly focus on evaluating performance degradation, or on improving generalization under predefined variations: They generally lack explicit safety-fault models or systematic mechanisms to generate targeted fault scenarios and to identify safety-critical failures.

Software Testing Techniques for AI Systems

Software testing research has studied methods for finding hidden system failures, including methods adapted to the components, properties, and workflows of machine learning systems [67]. These methods include fault modeling, fault injection, mutation testing, and automated test generation [26, 25, 27]. Fault modeling defines possible defect types. Fault injection deliberately introduces faults into a system, and then checks whether or not the resulting failures can be detected, thus assessing the testing effectiveness.

Software testing techniques have been applied to learning-based systems, including to automatically generate test cases and reveal unexpected behaviors [28, 29, 30]. Early studies mainly focused on coverage-based testing criteria. DeepXplore [28] introduced neuron coverage and differential testing for deep neural networks, while DeepTest [29] explored the automated testing of autonomous driving systems through realistic input transformations. DeepGauge [30] further proposed multi-granularity coverage criteria to assess testing adequacy beyond traditional accuracy metrics. Beyond coverage-based approaches, researchers have investigated mutation testing and fuzzing techniques to improve AI testing effectiveness. DeepMutation++ [31] introduced artificial faults through model mutations to evaluate defect-detection capability. DeepHunter [32] and TensorFuzz [33] employed coverage-guided mutation and fuzzing strategies to uncover neural-network failures that occur only for a small subset of inputs. Behavioral testing approaches, such as CheckList [34], investigated the generation of test inputs for specific behavioral requirements, rather than relying only on a model’s internal structure. Metamorphic testing uses relations among source and follow-up executions when individual expected outcomes are difficult to specify [66, 1, 2]. It has recently been applied directly to VLA-enabled robots [47]. S-TaLiRo searched simulated trajectories for temporal-logic violations in closed-loop cyber-physical systems [68], and PSY-TaLiRo provides a Python toolbox for search-based test generation [46].

Many methods for testing neural networks target individual learning components, whereas robotic fuzzing and temporal falsification address system-level executions. EAST builds on these complementary foundations and focuses on the traceable connections among safety evidence, operator applicability, runtime fault realization, and safety outcomes in embodied manipulations.

Conclusions and Future Work

This paper has introduced EAST, a three-stage safety-testing framework for embodied AI in digital twins. The Systematic Safety-Test Generation stage transforms safety knowledge into executable fault-injection specifications through a literature-grounded safety-fault model, eight runtime operators, and checks that the operator is applicable and that the proposed test can be executed as specified. Fault-Injected Digital-Twin Execution runs the specifications against embodied policies, verifies runtime realization, and records execution traces. Simulator-State Safety Assessment separates the task from the safety outcome using simulator-state safety oracles, and records challenge exposure (whether the policy encountered the targeted safety challenge), safety violations, and behavior signatures in reproducible test records. The framework analyzes coverage, behavior diversity, and replay consistency. It also examines task completion and safety outcomes separately.

We evaluated EAST on six manipulation tasks, for two policy families. EAST systematically covered a larger set of admissible task–policy–operator combinations. It also achieved safety-challenge exposure rates (the proportions of valid tests that exposed the policy to the intended challenge) close to those of Random-Legal and Rule-Based. The results show that task success is not a guarantee of safe execution: An embodied policy may complete a task, even with a safety violation. Task success and raw failure counts alone are therefore not enough to assess embodied-AI safety testing: The assessment should consider test validity, safety-challenge exposure, systematic coverage, behavioral diversity, and reproducibility.

Our future work will explore several directions. Firstly, we will extend the safety taxonomy and fault operators to additional embodied-AI platforms. These will include both manipulation and non-manipulation tasks, different robot embodiments, and more policy families. This will help to evaluate the portability of the safety taxonomy and fault operators across embodied-AI platforms. Secondly, we will compare digital-twin results with carefully controlled physical-robot experiments. This will help to identify fault mechanisms and safety-oracle predicates that transfer reliably from digital twins to physical robots. Thirdly, we will extend the safety specifications with richer temporal and contextual predicates. These new specifications could capture hazards that depend on longer execution histories, human proximity, or task-specific operating constraints. Finally, larger cross-platform studies should examine changes in safety-critical behavior: These studies could consider new robot perception and control models pretrained for use across tasks, sensing systems, and low-level controllers.