WorksEngineering
Co-simulation-Based Safety-Assessment of Black-Box Maritime Cyber-Physical Systems
Abstract
Modern maritime cyber-physical systems consist of tightly interconnected subsystems ranging from multiple engineering domains, making exhaustive safety verification impractical. Simulation-based testing offers a scalable alternative, while co-simulation enables heterogeneous subsystem models, potentially developed by different stakeholders, to be integrated without requiring access to their internal implementations. Adaptive Stress-Testing further improves testing efficiency by formulating failure discovery as a sequential decision-making problem, guiding the search toward the most likely failure scenarios, and giving the counter safety examples using reinforcement learning. However, black-box co-simulation may provide only partial access to the system state required by conventional Markov decision process formulations. To address this limitation, we proposed Event-Based Adaptive Stress-Testing version 2, a co-simulation-based safety-assessment method formulated as a partially observable Markov decision process. This method employs Recurrent Proximal Policy Optimization to exploit observation histories when selecting adversarial actions. The method is evaluated on a ship-in-transit operation in which a collision-avoidance system is stress-tested against multi-adversarial target ships. An extended Intermediate Waypoint Sampling mechanism enables independent multi-adversary trajectory control and supports the generation of realistic collision-inducing scenarios. The experiment demonstrate that the learned stress-testing policy generalizes across different encounter cases, can be further improved through continued training, and can be made effective when the collision- avoidance system under test is replaced by another version. These results demonstrate the method's potential for scalable black-box safety assessment of complex maritime cyber-physical systems.
Introduction
Advances in maritime automation are allowing increasingly autonomous and functionally integrated vessel operations. Modern maritime systems may consist of tightly coupled propulsion, navigation, machinery, energy-management, sensing, and control subsystems, together with intelligent modules such as collision avoidance. Their interaction increasingly gives rise to complex maritime Cyber-Physical Systems (CPSs), whose underlying subsystems may also be developed by different stakeholders. Co-simulation provides an enabling technology for integrating reusable and interchangeable subsystem models into a common virtual prototype, supporting rapid system development and system-level simulation [38].
The increasing integration of these subsystems also makes safety assessment more challenging. Component-level verification alone provides limited evidence of system-level behavior because failures may emerge from interactions between each correctly functioning subsystems. Consequently, complex maritime CPSs require testing approaches that considers all subsystem interactions under given operational conditions.
Simulation-based testing provides a scalable ways for evaluating such systems without exposing physical assets to potentially hazardous scenarios. However, its effectiveness strongly depends on the selection of the test cases. Test scenarios are traditionally selected manually using expert knowledge, operational experience, and risk analysis. On the other hand, systematic experimental-design approaches such as factorial designs, fractional factorial designs, Latin Hypercube Sampling (LHS), and Sobol sequences can be used to sample the input space and characterize system variability [26, 39]. Nevertheless, with a large number of interacting variables and possible environmental conditions exhaustive explorations become computationally infeasible. Automatic search mechanisms that direct simulations toward safety-critical regions are therefore desirable.
Adaptive Stress-Testing (AST) addresses this problem by formulating failure-scenario discovery as a sequential decision-making problem and using an optimization or Reinforcement Learning (RL) agent to search for likely failure trajectories. Different search strategies have been investigated, including Monte Carlo Tree Search (MCTS) [21, 22], the Backward Algorithm [19], Go-Explore [18], and deep RL-based approaches [13, 45]. These methods have demonstrated the capability of AST to stress-test safety-critical systems in domains including automotive, aerospace, and, more recently, maritime applications.
However, applying AST directly to distributed co-simulation introduces an additional challenge. Individual subsystem models may be accessible only through their exposed interfaces, while their complete internal states remain unavailable to the stress-testing agent. Therefore, the agent must make decisions from incomplete observations and infer information relevant to the underlying system state from its interaction history. Under these conditions, the fully observable Markov Decision Process (MDP) formulation is insufficient, and the stress-testing problem should be represented as a Partially Observable Markov Decision Process (POMDP). This motivates an AST formulation that is compatible with black-box co-simulation interfaces by having it capable of retaining temporal information.
Our Approach and Contributions
This article presents Event-Based Adaptive Stress-Testing v2 (EB-ASTv2), an extension of AST for simulation-based failure discovery in partially observable maritime co-simulation. The method is evaluated using a ship-in-transit scenario in which an RL agent controls multiple adversarial target ships to search for collision-inducing scenarios against an own ship equipped with a Collision-Avoidance (COLAV) system.
The principal contributions of this work are:
Partial-observability-aware stress testing. EB-ASTv2 employs Recurrent Proximal Policy Optimization (R-PPO) as its primary policy-optimization method. The recurrent policy retains information from previous observations and actions, enabling the agent to account for temporal information that is not available from the current co-simulation observation alone. The trained recurrent policy’s performance will also be compared with its non-recurrent policy alternatives.
Multi-adversary scenario generation. The framework introduces multiple adversarial control feature to enhance its failure discovery potential. An event-activity masking mechanism allows individual target ships to trigger waypoint-sampling events independently while remaining compatible with a joint policy action space.
Reward-guided generation of plausible adversarial trajectories. An improved reward formulation is introduced to guide the target ships toward interception opportunities while discouraging abrupt trajectory changes. This formulation balances collision-chasing tendencies with realistic target ship behavior.
Generalization across traffic configurations and systems under test. The policy is trained across a diverse collection of traffic configurations to improve its generalization capability. The modular co-simulation architecture further allows the trained stress-testing policy to be applied to different COLAV implementations by replacing the corresponding Functional Mock-up Unit (FMU), enabling comparative safety assessment without redesigning the complete testing framework.
Through these contributions, EB-ASTv2 aims to provide a reusable and scalable safety-assessment method for discovering safety-critical scenarios in partially observable maritime CPSs, while exploiting the modularity offered by co-simulation.
Preliminaries and Backgrounds
Co-simulation
Co-simulation technology is a simulation technique that is based on the construction of systems from loosely-coupled stand-alone models and the simulation across different subsystems. Co-simulation offers the ability for separate subsystems initialization, pre-processing, time-integration, and post-processing with different specialized tools [38]. Co-simulation also enables the use of black-box models [47], where the models is compiled into a low-level machine code that hides the internal algorithm from the users. These enables rapid and accurate model development that make use of already available models/software while at the same time protecting the Intellectual Property Right (IPR) associated to the subsystems.

Figure 1. Illustration of real maritime system modeled as co-simulation. It contains different tools and models, but they are interconnected with each other, run independently, and used in parallel to form a full-system simulation [38].
The subsystems in a co-simulation exchange information through their inputs and outputs. This exchange is based on the time-ordered values computed by the individual simulators during the simulation at discrete communication points [43]. Information exchange between subsystems is performed only at these communication points and can be expressed through interface constraint equations [38]
where and denote the subsystem input and output vectors, respectively, and is a connection graph matrix that maps subsystem outputs to subsystem inputs. The communication points are separated by a macro step size , such that . In practice, the connection structure represented by is specified through an I/O configuration, while an orchestrator algorithm is responsible for executing the co-simulation.
To enable interoperability among different subsystems and simulation tools, co-simulation relies on standardized interfaces or communication protocols. In this work we extensively used the Functional Mock-up Interface (FMI) standards, which has gained increasing attention in the engineering community in recent years [5]. FMI specifies that models should be packaged as FMUs, which are archive files that contain the model code along with its metadata and documentation. This standard is maintained and developed by the Modelica Association. Co-simulation has reached a decent maturity level and has been used across many industries, such as automotive industries [20, 28, 55], aviation industries [48, 56, 57], and recently maritime industries [6, 12, 27, 30, 33, 47].
Failures and Failure Analysis
Failure and failure mode are the two fundamental concepts in system reliability analysis. This section is based on Rausand [36] and summarizes the concepts relevant to failure analysis.
A failure occurs when an item is no longer able to perform one or more of its required functions. An item may be an individual component, a channel, a subsystem, or a safety loop. Hence, failure is the event that takes place when a required function is terminated. A failure may originate from an error, which defined as a “discrepancy between a computed, observed or measured value or condition and the true, specified or theoretically correct value or condition”.
Failure commonly followed by a failed state. The condition responsible for, or associated with, this state is referred to as a fault, where it is characterized by the inability of an item to perform a required function. A fault therefore is a state that may have its origin in a random failure event or in a deficiency related to the item, its location, or its application.
Finally, a failure mode is a description of a fault. It tells how a fault manifests itself. A failure mode lead to one or more consequences, referred to as failure effects. However, these effects can vary considerably in severity. Thus, the main motivation for identifying failure effects is to distinguish the more severe failure modes from less severe ones.

Figure 2. Error, failure, and fault [36].
Simulation-Based Safety-Assessment of Black-Box Cyber-Physical Systems
We first define a system model, denoted by , together with a parameter space , an input space , and a property that the system is required to satisfy. The behavior of the model for a parameter value and an input is denoted by . In general, the input may be time-dependent. The set of all possible behaviors of over the parameter space and input space is denoted by [16].
Testing is the task of evaluating whether the behaviors of the model , for selected subsets and , satisfy the property [16]. Assuming that and are finite, testing can be expressed as:
Verification is the task of proving that all possible behaviors of the model satisfy the property [16]. It therefore provides a formal guarantee of rightness or correctness over the specified parameter and input spaces [4, 40]. Verification can be expressed as:
In this research, we focus on CPSs, which are common in maritime systems because they integrate physical, digital, and networked components [53]. CPSs are difficult to verify exhaustively because they are complex, hybrid, and often black-box. A system is considered a black box when its internal model is unknown or too complex to support explicit reasoning [7], or when it is deliberately concealed from external parties, for example to protect proprietary information.
Kapinski et al. [16] provide an overview on existing system verification method for CPSs, illustrated in Figure 3. The spectrum shows the trade-off between scalability and exhaustiveness of each method. Simulation-based testing is scalable but non-exhaustive. Formal verification is exhaustive but less scalable. In maritime autonomy, Torben et al. [53] show that simulation-based testing is attractive but requires automated test-case selection due to the large scenario space. Thus, Falsification is useful because it searches for counterexamples rather than proving universal correctness.

Figure 3. Spectrum of verification methods for CPSs [16].
After introducing the relevant definitions, we describe a method known as safety validation, introduced by Corso et al. [7]. Safety validation, as illustrated in Figure 4, is the process of ensuring the correct and safe operation of a system within its environment . The desired safety property are expressed in a specification language, and simulation is typically used to identify violations of these properties, which are also referred to as failures. System failures may be caused by disturbances from the adversary , and a model of these disturbances can be used to estimate the probability of failure. A system is deemed safe if no failure is found during a finite exploration or if the estimated failure probability remains below an acceptable threshold. Accordingly, safety validation encompasses three main objectives:

Figure 4. The safety validation problem [7].
Falsification, which is the task of finding a parameter value and an input for which the resulting system behavior violates the property [16]. In other words, the objective is to find a counterexample such that:
Equivalently, the set of all counterexamples may be written as:
Most-Likely Failure Analysis, which is the task to find the counterexample with maximum likelihood:
Failure Probability Estimation, which is the task to compute the probability that a safety property will be violated. Failure probability is given by the expectation of observing failure under the disturbance model
Where,
In the verification literature, falsification is often discussed as part of simulation-based verification because it searches for counterexamples to formalized requirements. In the black-box safety-validation literature, the same activity is interpreted as one goal of safety validation, because the discovered counterexamples reveal unsafe behavior in an operational environment. In this work, falsification is therefore treated as a simulation-based safety-assessment activity: it does not provide a formal proof of correctness, but it supports a scalable verification via counterexamples and validation by identifying requirement-violating trajectories.
Reinforcement Learning Preliminaries
Section 2.4 provides a brief overview of Markov decision processes, largely following the presentation in Sutton and Barto [51].
Markov Decision Process
MDP is the classical of formalization of sequential decision making and serves as the foundation of many modern RL algorithms. MDP is intended as an intuitive framework of the problem of learning through interaction to achieve goal. The framework defines two entities: the “learner” which will be often refer to as the agent, and the environment which consists of everything outside the agent that it interacts with. Together, they create the agent-environment interface where both entities interact continually: The agent selects actions and the environment responds to it, transitioning it to a new situations or states to the agent. The environments then assigns a specific reward signals attribute to this transition.
MDP can be formally defined as a tuple . The agent and environment interact for every discrete time steps, . At each time step , the agent receives some representation of the environment state, . Then based on the states, the agent choose an action, . On the next timestep, in part as a consequence of its action, the agent will find itself in the new state and receive the numerical reward, . The continuous process of this thereby give rise to a sequence of
In a finite MDP, the sets of states, actions, and rewards all have finite number elements. In this case, the next obtained state and reward when starting from a known current state while taking action is random and based on a probability distribution dependent only on the preceding state and action.
for all ; ; and the transition function defines the dynamics of the MDP. Due to this property, a state-action transition not just affect the immediate rewards of one transition but also the future rewards through the subsequent agent-environment interactions. This transition model also satisfies the Markov Property, meaning that, given the current state, future state transitions are independent of past agent-environment interactions. Thus, the state must contain all information from the interaction history that is relevant for predicting future dynamics.
Partially Observable Markov Decision Process
Many real-world problems do not satisfy the assumptions of the MDP framework because the learning agent can only partially observe the state of the environment at each time step [44]. In other words, some components of the state may be inherently hidden from the agent, while others may be observed inaccurately due to environmental noise. Such problem can be modeled as a POMDP [59], which is formally defined as a tuple
The follows the MDP in Section 2.4.1. is the observation space that describe the possible observations, is an observation model where
is the probability of observing given that the agent has reached state after executing action .
One option to address POMDP challenge is to use the agent history as its internal state. This history is composed of previous agent-environment interactions, denoted as
The complete history can be used as an internal Markov state, but it is infeasible for the reason will continually grows over time [44]. To tackle this problem, the concept of a belief state is introduced in the context of nondeterministic and partially observable planning problems [37]. Belief state is the set of actual states in which the agent may be in. In POMDP, the belief state is generalized into a probability distribution over the possible states. Following Shani et al. [44], the belief state is conditioned on the agent’s history, and can be expressed as
where represents the probability that the agent is in state after observing history .
Rewards, Returns and Episodes
In RL, the reward function is formulated to capture the desired objective of the problem. Accordingly, the agent seeks to maximize the cumulative reward over the course of its interaction with the environment, rather than prioritizing immediate reward maximization. If the sequence of rewards for every time step is denoted as , we can then formalized the expected return of that sequence. Formally, the expected return is what the agent aims to maximize. The simplest form of return can be written as
where is a final time step. This approach is sensible in the case when there is a natural notion of final time step, that is when the agent-environment interactions break naturally under a specific condition. This specific condition is generally called as terminal state. Furthermore, the overall sequence from the initial until terminal state is called episodes.
However, in many case the agent-environment interactions go infinitely rather than having a definitive end, which could cause the return to become unbounded. To address this, the discounting concept is introduced to discount the future rewards, emphasizing the importance of the immediate rewards while reducing the influence of the future rewards and preventing the return computation to explode. It is written as
where is called discount rate and bounded with . As approaches 1, the greater the emphasis placed on future rewards, and vice versa, thereby defining the trade-offs between immediate rewards and long-term rewards.
Policies and Value Functions
A policy is formally defined as a mapping from states to probabilities of selecting each possible action. In a stochastic policy, the probability of selecting action in state is denoted by . Thus, when the agent follows policy , the action at time is sampled from .
The value function, , is a function that estimate the value of the agent being in a state under policy . Similarly, action-value function, , also estimate the value of the agent taking one action in a state while following policy . The value here is defined in terms of the expected returns .
Solving RL problem means finding the policy that maximizes the returns collection over the episodes. In that context,there is always at least one policy that is better or equal to all other policies which can be measured by the value function associated to that policy. In essence, if and only if for all . This policy is called the optimal policy, denotes as . They also share the same state-value function and action-value function, namely the optimal state-value function and optimal action-value function respectively. It can be defined as
Policy Evaluation and Policy Improvement
Many RL algorithms can be interpreted as alternating, explicitly or implicitly, between two operations: policy evaluation and policy improvement. Policy evaluation estimates how good the current policy is, usually through a value function or action-value function. Policy improvement then modifies the policy toward actions that are expected to yield higher return. These two sub-processes alternate iteratively until they converge into the optimal policy. Each RL algorithm differs on how they evaluate and improve the policy, some may have an edge than the others depend on the circumstances.
Policy Gradient, Advantage, and Actor-Critic Idea
One could also consider to use a parameterized policy that can select actions without requiring a value functions. Let the policy parameter vector be denoted as . Using this parameter, we parameterized the policy into which can be written as
The policy parameter vector can then be learned by optimizing a scalar performance measure , which quantifies the expected performance of the policy. The value functions may be included in to aid for the policy parameter learning. Since the objective is to maximize , the policy parameters can be updated using stochastic gradient ascent
where is the learning rate and is a stochastic estimate whose expectation approximates the gradient of the performance measure with respect to the policy parameters . Methods that update the policy in this manner are known as policy gradient methods. Based on Marbach and Tsitsiklis in [25] and (Sutton et al.,) in [52], the gradient of the scalar function with respect to the policy parameters can be formalized as:
This gradient increases the likelihood of high returns-action sampling, however it introduces high variances because the gradient is approximated using sampled estimates from trajectories collected by the policy. A common strategy as demonstrated in ([3], [10], [50]) can reduce variance by subtracting the state-value function from the action-value function, such that
This adjustment is referred to as the advantage function, and it describes how much better is action than the average action under the current policy at state . This adjustment does not bias the gradient as well because the value function is independent of the action.
The actor-critic architecture is a method that learns approximations to both of policy learning and value estimation. The actor is a policy network that proposes actions given a state, while the critic consists of value function networks that estimate the expected return of actions taken in given states. This separation allows the actor to be updated using gradients provided by the critic, leading to more stable learning than purely policy-gradient or value-based methods.
Long Short-Term Memory Neural Network
Long Short-Term Memory (LSTM) is an advanced variant of the Recurrent Neural Network (RNN) that is specifically tailored for learning long-term dependencies. LSTM was first introduced by Hochreiter and Schmidhuber [14], and now it is widely use and works remarkably well on many variety of problems.
LSTM addresses the limitations of standard RNN, where it often struggles to learn long-term dependencies due to the vanishing gradient and exploding gradient problems [41]. LSTM solves this by introducing a separate memory pathway known as the cell state. As shown in Figure 5, the cell state is represented by the horizontal connection running through the top of the structure. The flow of information into and out of the cell state is controlled by several gates, which are the forget gate, input gate, and output gate. These gates use sigmoid activation and tanh function to regulate how much information should be retained, added, or exposed. Through this mechanism, LSTM is better able to retain useful long-range information while filtering out irrelevant information.

Figure 5. Long Short-Term Memory Networks [29].
Reinforcement Learning Cores for Event-Based Adaptive Stress-Testing
Proximal Policy Optimization and Recurrent Extension
Proximal Policy Optimization (PPO) is a family of policy-gradient methods that optimizes a surrogate objective using stochastic gradient ascent. Introduced by Schulman et al. [42], PPO was designed to retain some of the data efficiency and reliable performance associated with Trust Region Policy Optimization (TRPO), while avoiding the complexity of second-order constrained optimization. This is achieved through a clipped probability-ratio objective, which discourages excessively large policy updates and improves training stability.
Let denotes the probability ratio between the updated policy and the policy used to collect the data
When , the ratio satisfies . Values greater than one indicate that the updated policy assigns higher probability to action in state while values less than one indicate that the updated policy assigns lower probability to that action.
PPO avoids the explicit constrained Kullback–Leibler (KL) divergence optimization of TRPO by modifying the surrogate objective with a clipping operation
where is a hyperparameter that determines the clipping range. The clipping mechanism limits the incentive for the updated policy to move too far from the old policy by maintaining the balance between the probability ratio and the advantage value while hard-constraining the objective with a predetermined threshold . In this way, PPO approximates the stabilizing effect of TRPO algorithm while using a simpler first-order optimization procedure.
The objective in advantage function estimators can also use value function loss formulated using squared-error loss, denoted as , making it follows the actor-critic formalism
The objective can also be further improved by introducing entropy bonus, maximization, which encourages exploration. Combining this terms, we can obtain a more robust PPO formulation
where and control the strength of the value-function and entropy-bonus terms.
While the standard PPO is designed for fully observable MDPs, many real-life problems provide only for partial observation of the underlying state. Standard PPO generally employs feedforward Multi-Layer Perceptron (MLP) policy and value networks, therefore basing each decision on the current observation. R-PPO extends PPO by leveraging recurrent architecture into the actor and critic networks, enabling the agent to retains memory that can provide information on the history of the current task that is not present in the agent’s most recent observation [34]. In this research, LSTM networks are chosen to represent the policy and the value function while preserving the same optimization objectives as standard PPO (Equation 25).
Soft Actor-Critic
Soft Actor-Critic (SAC) is introduced in [11] by Haarnoja et al. as an off-policy actor-critic algorithm based on the maximum entropy RL framework. For the standard RL algorithm, the goal is to learn the policy to maximizes the returns . On the other hand, the maximum entropy objective generalizes the standard RL objectives by augmenting it with an entropy term, such that the optimal policy additionally maximize its entropy at each visited state:
where is the temperature parameter that determines the strength of the entropy term . SAC algorithm incorporates three key ingredients. First, an actor-critic architecture consisting of a policy network (actor) and a value function network (critic), parameterized by and , respectively. Second, an off-policy formulation that enables reuse of previously collected data through memory buffer for efficiency. Third, an entropy maximization process to encourage stability and exploration.
SAC is trained on the soft policy iteration, which includes soft policy evaluation and soft policy improvement. The main goal of this framework is to simultaneously maximize expected return and entropy to encourage exploration while still discouraging clearly bad actions. Intuitively, it is analogous to succeed at the task while acting randomly to the policy entropy. We encourage reader to visit Haarnoja et al. works [11], where they provide the details about the loss functions and derivations of the SAC algorithm.
Adaptive Stress-Testing
AST is a simulation-based falsification method that aims not only to find a scenario that results in a failure, but also to maximize the probability of the failure modes [23]. One most notable implementation of AST was carried by Lee et al. [22], where AST was introduced as a scalable and black-box compatible failure discovery method to stress-test a prototype air collision avoidance system by identifying the most likely trajectories that leads to mid-air collision.
Borrowing the terminologies from § 2.3, we define a simulator denoted as that contains the system under test (or just system) interacting with the environment . The system with state takes a response based on observation of the environment state . The simulator state is the stacked system and environment states . The system interacts with the environment over discrete timestep . As conventions, we denote the variable at time using subscript and we also denote the sequence of variable over a range of time steps with colon range . The state and action of the system depend on the environment observations, and are modeled as:
The environment state and observation evolve over time depending on the actions of the system and disturbances , where is the dimensionality of the disturbances. The disturbances here can be depicted as any stochastic variables that can affect the environment, hence these terms are very broad and need to be specified. For example, the can be weather loads, sensor noises, or adversarial actors wandering inside the simulation (e.g. pedestrian, target ships, other vehicles)
The disturbances are assumed to be conditionally sampled from a probability density , which may be specified using expert knowledge or estimated from relevant data. Let denote the desired event space, and let denote its complement, corresponding to undesired events. There also exists states that cause the simulation to be terminated which is referred to as terminal state. In this work, we focus on failure, i.e., terminal states satisfying . They signify the end of one episode, and they can belong both in (e.g. ship collision) and (e.g. ship reaches end trajectory). The objective of AST is to find the most likely failure modes by maximizing the reward function. The objective can be formulated as:
AST frames failure discovery as a reinforcement learning problem, shown in Figure 6, where an adversarial agent selects disturbances that challenge the system under test . Meanwhile the simulator evolves the system state and reports whether a failure has occurred. To avoid confusion, AST frames the disturbance sampling as the equivalent of action sampling on typical RL methods. Depending on the extent to which the simulator exposes its internal state, AST can be categorized into two types:

Figure 6. Adaptive Stress-Testing illustration.
Full Observability
If the simulator state is fully observable, we can formulate the AST problem as an MDP as demonstrated in § 2.4.1. In this case, the agent can selects a disturbance from the policy given the true state ,
where the general reward function design by Lee et al. [23] is given by:
The first term of (31) provides a non-negative reward signal, , when the trajectory terminates due to a failure. If the trajectory terminates without encountering a failure, the second term of (31) penalizes the agent by assigning a miss distance, , which measures how close the simulator came to a failure event. This term can significantly accelerate the search by enabling the agent to differentiate the relative desirability of non-failure trajectories [23].
The final term of (31) seeks to maximize the AST objective defined in (29) by rewarding the log-likelihood of each state transition. Recall that unlikely disturbances are associated with lower probabilities, whereas likely disturbances have probabilities closer to one. Consequently, the log-likelihood of an unlikely transition approaches , while that of a highly likely transition approaches . This reward formulation therefore penalizes unlikely transitions while favoring more probable ones, thereby guiding the agent toward discovering failure trajectories that are as likely as possible.
Partial Observability
Not all simulator supports fully observable states. Some simulators are given as a software binary that maintained its internal dynamics hidden over function calls [23]. Example of this are co-simulation. Even though the underlying process can be modeled as an MDP, the agent only sees part of simulator states in form of observations .
This is a problem for typical MDP solvers because the same observation may correspond to multiple underlying hidden states. Consequently, a policy conditioned solely on the current observation may select suboptimal actions, as it lacks sufficient information to infer the true state of the environment. To address this issue, the partially observable AST problem is naturally formulated as a POMDP (see § 2.4.2), where sampled disturbances are conditioned on an observation history (see Equation (12)) rather than only the current observation:
Based on our literature review, we identified two main approaches for handling partial observability in AST:
History-based tree search. This approach employs algorithms that maintain and revisit the action-observation history during planning, allowing the agent to reason over past information when selecting future disturbances. This direction was pioneered by Lee et al. [23] through the introduction of the Monte Carlo Tree Search-Seed Action (MCTS-SA) algorithm.
Recurrent policy gradient methods. This approach augments policy gradient algorithms with RNN architectures, such as LSTM, instead of conventional MLPs. The recurrent layers enable the policy to learn temporal dependencies from the observation history, therefore forming an implicit approximation of the underlying belief state. One example is the work of Koren and Kochenderfer [17], who demonstrated the application of Deep Reinforcement Learning (DRL) algorithms with LSTM layers for partially observable AST.
As for the reward function designs, it mirrors the form and purpose those in Equation 31 [23]:
where the simulator indirectly (due to agent’s limited access to the actual simulator states) provides the transition likelihood , a boolean indicating whether failure has occurred, and the miss distance . Additionally, represents a boolean from a function call that explains whether a terminal state has been reached based on the actual simulator states.
Event-Based Adaptive Stress-Testing
Event-Based Adaptive Stress-Testing (EB-AST) was first introduced as an extension of the conventional AST framework by revisiting its action sampling strategy, and was extensively evaluated for stress-testing maritime collision avoidance systems [45]. In conventional COLAV AST, the adversary is modeled as an RL agent that directly controls the target ship as a disturbance by sampling low-level control actions corresponding to its surge, sway, and yaw accelerations. These actions are updated at every fixed RL decision interval throughout the simulation.
One representative example is the work of Hjelmeland et al. [13], which demonstrated that this formulation is effective in discovering collision-inducing trajectories under several reward function designs. Despite its promising performance, this approach exhibits several limitations. Since the adversary updates its control actions at a higher frequency, the resulting trajectories may become overly discontinuous and operationally unrealistic. This effect is particularly evident in scenarios where the target ship continuously steers toward the own ship or performs excessively aggressive maneuvers during near-collision encounters.
EB-AST considers a typical ship-in-transit scenario in which the own ship executes its nominal voyage while an RL agent controls an adversarial target ship. The objective of the adversary is to interfere with the own ship’s mission and induce collision scenarios. Each of ship asset initially has their own baseline trajectory (which is a straight trajectory between start to end waypoint). The main takeaway is that the target ship is permitted to sample new waypoints in between the start and end of its trajectory to follow, mid simulation. These new waypoints is called Intermediate Waypoints (IWs), and an IW can be sampled only after the target ship enters the current IW’s Radius of Acceptance (RoA). Consequently, IW updates are triggered by navigation events rather than fixed time intervals, giving rise to the term event-based. This waypoint-generation mechanism, referred to as Intermediate Waypoint Sampling (IWS) (illustrated in Figure 7), and constitutes the key characteristics of the EB-AST framework. Additionally, IWS is constrained such that every newly sampled waypoint advances the vessel toward its destination, preventing unrealistic backward maneuvers or oscillatory behavior. The overall AST process is then constructed as MDP and then optimized using SAC algorithm. The objective of EB-AST is to discover critical waypoint sequences that induce collision while preserving realistic target-ship trajectories through the waypoint sampling constraints [45].

Figure 7. Intermediate waypoint sampling process in EB-AST.
The proposed EB-AST framework however was limited to identifying the near-collision events [45] and did not uncover actual collision events. The authors suggest that introducing multiple adversarial vessels may place the own ship’s COLAV system under greater operational stress, which might increasing the likelihood of genuine collision events.
Ship-in-Transit Co-simulation Setup
Ship Model
Figure 8 shows the interconnection structure of the FMUs used in the ship simulation framework. In this research, this structure is specifically used to model the own ship in order to include the system complexity for the stress testing process. Each block in the diagram is implemented as a separate FMU, and exchange data through their defined input and output variables. To orchestrate the communication between each block, we use a Python library called libcosimpy [49], which is a Python wrapper for co-simulation library in C++, libcosim [2], both are developed by Open Simulation Platform (OSP) [1].

Figure 8. FMUs connection diagram for the ship simulation.
The ship model FMU represents the vessel hull dynamics. It receives the thrust force generated by the machinery-system FMU and the rudder force generated by the rudder FMU. Environmental disturbances are modeled as two FMU blocks: surface current and wind, which provides the speed and direction of the surface current and wind to ship model FMU.
The nominal actuator commands are generated by the autopilot and the speed-control subsystems and are subsequently processed by the COLAV FMU. When no collision threat are detected, the nominal throttle and rudder commands are passed directly to the machinery-system and rudder FMU, respectively. When an avoidance action is required, COLAV system modifies these command and output the corrected nominal commands.
The autopilot FMU generates the nominal rudder-angle command using Line-of-Sight (LOS) guidance scheme. The mission-manager FMU selects and supplies the active waypoint segment, consisting of the previous and next waypoints, according to the current vessel position received from the ship model FMU. Using this waypoints, the autopilot determines the desired heading and calculates the corresponding rudder command.
The forward speed control is handled by a cascaded structure consisting of the shaft-speed controller and throttle-controller FMU. The shaft-speed controller compares the actual vessel speed from the ship model FMU and the predefined speed set point from the mission manager FMU. It then generates the desired shaft-speed setpoint that will be used by the throttle-controller to compare with the measured shaft-speed retrieved from the machinery-system FMU. From this, it will output the nominal throttle command, closing the loop.
Simplified Ship Model
Figure 9 shows the interconnection structure of the FMUs used in the simplified ship simulation framework. Like the previous Ship Model structure in § 3.1, all blocks in the diagram are implemented as a separate FMUs, and are orchestrated using the same libcosimpy Python library. In this research, a simplified model is used for the target ships because their role as adversarial actors does not require the same level of system complexity unlike the own ship. Therefore, we stripped off the machinery-system FMU along with the cascading speed-control subsystem and replace it with a simple PI speed controller. Additionally, we also exclude the COLAV system FMU, which enables a direct thrust and rudder command input into the ship model FMU.

Figure 9. FMUs connection diagram for the simplified ship simulation.
Simple Collision-Avoidance System
The simple COLAV implementation follows a relatively simple decision strategy, as shown in Figure 10. When an avoidance maneuver is required, the system commands a half-throttle and a fixed-angle turn to starboard with the intention of passing the selected target ship on the own ship’s port side. This intention is arbitrary and serves as a demonstration only. The avoidance command will considered once the system assess own ship’s collision risk. At each assessment interval, it evaluates the collision risk associated with every target ship and selects the ship presenting the highest risk. The collision risk score computation is based on the Closest Point of Approach (CPA) metrics, and closing speed value.

Figure 10. Process flow diagram of the simple COLAV system to stress-test.
Let the relative north-east position vector and the relative north-east velocity between the own ship and the target ship be defined as and respectively,
Given the own ship north-east position and heading , we compute each target ship’s relative bearing to own ship
and the instantaneous separation distance
Differentiating with respect to time, yields the closing speed
Closing speed explains whether two ships are closing in on each other when is negative or moving apart when is positive.
We also introduce the concept of the CPA, which is widely used in the definition and evaluation of collision-threat parameters and criteria [24]. The CPA denotes the predicted point at which the distance between the own ship and another object, (can be as a target ship or an obstacle), reaches its minimum value [8]. Two quantities associated with the CPA are the Time to Closest Point of Approach () and Distance to Closest Point of Approach ()
During the simulation, the own ship will continuously compute the , , and of each target ship. Given the collision zone radius and the safe look-ahead time , the target ship is considered as threat when its , , and . In the case of multiple target ships are considered as threats, the own ship’s COLAV system will compute The collision risk score and prioritize the target ships that has the highest risk score. is formulated as:
Where , , are all coefficients which control the strength of each terms. The COLAV system can respond to only one target ship during each decision interval. After the interval has elapsed, the are reassessed and the target ship with the highest is selected.
Ship Traffic Generator
Ship Traffic Generator (STG) is a tool developed by Det Norske Veritas (DNV) to generate a structured set of ship encounter scenarios for verifying automated collision and grounding avoidance systems [31]. To generate an encounter scenario involving the own ship, the desired situation must first be specified. This desired situation consists of several user-defined parameters:
Initial Position. The initial position of the own ship, given by latitude and longitude coordinates in decimal degrees.
Speed over Ground (SoG). The speed of the own ship relative to the ground, given in knots.
Course over Ground (CoG). The nominal mission course of the own ship, given in degrees using a north-referenced clockwise convention. For example, a northbound trajectory has a CoG of 0°.
Initial Heading. The initial heading angle of the own ship, given in degrees.
Navigational Status. The navigational status assigned to the simulated vessel.
Vector Time. The time, given in minutes from the start of the simulation, at which the desired encounter situation is expected to occur within a specified range from the own ship.
Relative Bearing . Which is the angle between the own ship heading and the target ship directions, given in degrees.
Relative Speed. The speed of the target ship relative to the own ship, expressed as a fraction of the own ship speed. A value of 1 indicates that the target ship travels at the same speed as the own ship, while values below or above 1 indicate lower or higher relative target-ship speeds, respectively. The minimum value is 0.
The desired situation is typically specified in a JSON file, which serves as input to the STG. The corresponding output is also provided in JSON format and contains relevant vessel settings, such as waypoints and SoG, for all simulated vessels. These settings can subsequently be translated and implemented in the Ship-in-Transit co-simulation framework to generate the desired encounter scenarios. Examples of the generated traffic scenarios are shown in Figure 11.
Figure 11. Examples of the generated traffic scenarios by STG.
In this work, STG is used as an external scenario-generation tool to construct the initial ship encounter configurations used in the simulation study. The complete encounter generation algorithm is not explained here, since the present work does not modify the internal STG procedure. Instead, we summarize the input parameters relevant to our study and describe how the generated encounter cases are integrated into the proposed EB-AStV2 environment. A detailed description of the STG algorithm can be found in Pedersen et al. paper [32].
Case Study: Collision-Avoidance Stress-Testing in a Ship-in-Transit Co-simulation
This case study considers a ship-in-transit operation in which the functionality and failure behavior of a shipboard COLAV system are assessed through simulation-based stress-testing. We are aiming to develop a stress-testing method that automatically searches for scenarios that expose limitations in the COLAV system and lead to collision events. Identifying such scenarios can provide insight into the mechanisms underlying the failures and support subsequent improvements to the COLAV algorithm.
The simulated traffic scenario consists of three vessels. The first vessel is designated as the own ship and is equipped with the COLAV system under test. The remaining two vessels are designated as target ships and are distinguished by numerical identifiers. From the perspective of the own ship, these target ships represent surrounding traffic that must be monitored and avoided during the voyage.
Each vessel is initially assigned a nominal mission defined by two waypoints: a start waypoint and an end waypoint , where all of them attempt to complete their respective missions using their onboard speed and heading controllers. For the own ship, these waypoints define a straight-line route that the vessel is required to follow. Conversely, for each IW-enabled target ship belongs to set , let denotes the target ship identifier, denotes the set of all feasible trajectories as a finite waypoint sequence connecting to .
where is the number of adversarial target ships up to , are the IWs, and is the predetermined maximum number of IWs associated to each target ship . We also define the distance between two consecutive waypoints as
where among the trajectories in , there exists that has the minimum path length.
In the absence of obstacles or additional constraints, is the straight-line route connecting and .
The nominal missions are generated using the STG. Depending on the specified encounter type, the straight-line route may intersect or overlap in a manner that produces head-on, crossing, or overtaking encounters between the own ship and the target ships. The complete set of initial waypoints and vessel states used to initialize an episode is referred to as a spawn request or spawn case.
A scenario is then defined as the combination of the own ship trajectory and the target ships’ trajectories within a single simulation:
Accordingly, the overall scenario space is given by
where the Cartesian product indicates that one trajectory is selected for each target ship.
During a nominal encounter, the own ship relies on its onboard COLAV system to avoid the target ships. However, the simplified COLAV system used in this stress-testing experiment has a known limitation: it cannot simultaneously account for all target ships when multiple collision threats are present (see § 3.3). Instead, collision-avoidance actions are determined according to a sequential prioritization scheme. This behavior introduces a potential vulnerability in encounters involving multiple simultaneous threats.
The objective of the stress-testing method then is to identify the subset of scenarios that exposes limitations of the onboard COLAV system and results in a collision between the own ship and at least one target ship. This subset is defined as
Here, is defined as the set of collision-inducing trajectories for each target ship
where indicates that the trajectory results in a collision involving target ship and the own ship. Thus, a scenario belongs to if at least one of the target-ship trajectory is collision-inducing; the trajectories of the remaining target ships are not required to result in collisions. To identify such scenarios, the stress-testing method searches for waypoint sequences between and along each target ship’s nominal route.
In addition to identifying failures for individual initial conditions, the stress-testing method is designed to produce a scenario generator that can adapt to a diverse range of spawn cases. To achieve this, the stress-testing process is formulated as a RL problem, in which the agent’s policy serves as the scenario generator. A finite collection of traffic configurations is generated using the STG, stored in a spawn-request bank, and sampled during training. These configurations expose the agent to variations in vessel positions, headings, speeds, and encounter geometries, encouraging the policy to learn a generalizable strategy that can operate across multiple traffic configurations rather than overfitting to a single scenario.
Additional experiments are conducted to investigate whether further improvements can be achieved by continuing the training of an existing policy. The trained policy is subsequently used to stress-test different COLAV implementations, allowing us to assess how effectively it can continue to generate collision-inducing scenarios. The proposed solution to this study case is our EB-ASTv2 algorithm, which is introduced in Section 5.
ebast2: An Enhanced Event-Based Adaptive Stress-Testing Framework for Co-Simulation
Section 5 presents the EB-ASTv2 algorithm, an improved iteration of the original EB-AST method [45]. The section describes the proposed algorithm and discusses its key methodological improvements in detail.
Proposed Algorithm and Improvements
We proposed a novel safety-assessment method specifically tailored for black-box CPSs, where the framework diagram of this method is shown in Figure 12. In this paper, we focused on using a maritime Ship-in-Transit co-simulation as a demonstrator. We employ R-PPO algorithm as the primary solver algorithm because of its compatibility with POMDPs, which are often encountered in black-box simulation like co-simulation. The framework consists of two process blocks that interact with each other iteratively that forms an EB-ASTv2 update cycle process: the RL environment wrapper and the DRL agent.

Figure 12. Process diagram of EB-ASTv2 method.
RL environment wrapper is implemented as a Python wrapper that is built around the Ship-in-Transit co-simulation. The wrapper manages the simulation step function calling, reward signals computation, and relevant RL states and termination states compilation for the DRL agent. At each cycle, the wrapper accepts actions (or disturbances seen by the own ship) from the EB-ASTv2 policy and forwards it to the corresponding module that handles IWS mechanism for the target ships. The IWS mechanism is implemented within the Mission Manager FMU of each adversarial target ship, which modifies the ship’s trajectory by sampling IWs to stress-test the own ship’s COLAV system. Once the state-action transition is complete, the wrapper returns observation and its corresponding reward to the DRL agent.
The DRL agent contains the solver which takes the tuples of observation from the state-action transition along with its associated rewards and process it to improve the EB-ASTv2’s policy. In this research, we employs the R-PPO algorithm provided by stable-baselines3, an open-source library which supports variety of RL algorithms [35]. This library offers reliable and easy-to-use RL implementations, well suited for research and RL-related development. Following the R-PPO algorithm, the DRL agent first collects rollout trajectories through repeated interactions with the simulation environment. These rollouts are then used to iteratively train the LSTM-based policy and value networks through the R-PPO policy evaluation and policy improvement.
We specifically adopt the recurrent policy gradient methods like R-PPO rather than the history-based tree search because of the need to learn a reusable policy for a partially observable and continuous scenario space. While history-replay mechanism is effective for online search, it does not directly learn a reusable policy that generalizes across similar histories or scenario instances. In practice, DRL agent is not strictly tied to R-PPO. It can hosts variety of RL algorithms as long as it is compatible with POMDP.
EB-ASTv2 introduces several improvements over its predecessor, EB-AST. In addition to its compatibility with POMDP settings, EB-ASTv2 supports multi-adversary control, unlike EB-AST single-adversary control limit. This allows the EB-ASTv2 agent to coordinate multiple target ships and generate more challenging encounter scenarios that may expose weaknesses in the own ship’s COLAV system. Furthermore, the IWS mechanism is enhanced by introducing a variable IW sampling scope-length as a second action component. The variable IW sampling scope-length allows the agent to adapt the scale of its intervention. Shorter scope-lengths enable local adjustments to the target ship trajectory, while longer scope-lengths allow larger deviations over a wider portion of the encounter scenario. This additional degree of freedom increases the flexibility of IW sampling and enables a more precise exploration of the scenario space.
Gymnasium Environment for Reinforcement Learning
The stable-baselines3 library expected the users to prepare their RL environment based on Gymnasium API. Gymnasium API is a common interface for RL environments, defining how and agent interacts with an standardized function calls that represents MDP (or POMDP) [54]. These main function calls are reset(), step(), and isTerminal().
Note that although Ship-in-Transit co-simulation are wrapped within the RL environment, both entities operate different interpretations of a time-step. In the co-simulation, time-step refers to the time interval between consecutive numerical integration steps (e.g. Euler integration). In contrasts, a time-step in the RL environment represents the interval between two consecutive events. Consequently, the number of co-simulation integration steps between two RL time steps may vary, as different events may occur after different amounts of simulated time.
In this work, we define events as the occurrence of one of the following conditions:
Event 1: the co-simulation reaches the time threshold at which IWS is enabled;
Event 2: a target ship enters the RoA of its next waypoint;
Event 3: the co-simulation reaches a terminal state, either desired or undesired.
The desired terminal states, denoted by , corresponds to outcomes of interest for the stress-testing process. These includes:
own ship collision between one ore more target ships,
own ship prolonged deviation from its intended trajectories beyond a predefined time limit. The deviation boundary is based on the maximum allowable cross-track error of the LOS guidance system.
On the contrary, undesired terminal states are denoted as , correspond to terminal conditions that do not provide relevant information for the stress-testing objective. These includes:
own ship reaches its mission endpoint,
collision between target ships.
The reset() function are typically called when the environment is initialized. Upon calling this function, the Ship-in-Transit co-simulation is advanced until Event 1 is encountered for the first time. At this point, the agent receives the initial observation along with additional information regarding the current environment state.
The step() function executes the selected action from the agent’s policy and updates the environment accordingly. An action is requested when either Event 1 or Event 2 is triggered.
Subsequently, the co-simulation advances until another Event 2 occurs, or until Event 3 is reached. As a result, the agent receives a new observation along with a reward for taking the action.
The isTerminal() function returns two Boolean values, and . The value is TRUE if the current simulator state is terminal, and FALSE otherwise. If is terminal, indicates whether the terminal state is desired. = TRUE if , and = FALSE if .
Additionally, the RL environment also defines the structures such as the observation space and action space, allowing RL libraries like stable-baselines3 to work across different environments through the same interface. This will be further explained in the sections below.
Observation Space
The observation space consists of the simulation states that are observable to the agent, as not all states of the Ship-in-Transit co-simulation are exposed to the agent. Throughout this paper, we are going to refer to the “own ship” and the “target ship” as “OS” and “TS” respectively. Given the number of adversarial target ships and , this space consists of:
own ship COLAV throttle command: ,
own ship throttle command: ,
own ship COLAV rudder angle command: ,
own ship rudder angle command: ,
own ship positions: ,
own ship forward speed: ,
own ship cross-track error: ,
target ships relative positions: ,
target ships forward speed: ,
actions masks: .
The term relative position refers to the difference between the position of a target ship and that of the own ship. Representing the target-ship positions relative to the own ship helps provide a more consistent observation representation across different encounter cases.
In addition, the action masks are key observations that enable multi-adversary control. Each action mask takes a binary value of either 0 or 1, indicating whether the corresponding target ship can currently be acted upon by the agent. The action-masking mechanism is further explained in § 5.3.1.
Action Space
The action space comprises of the available actions for the agent to perform. In the AST context, action here is also referred to as disturbance These actions are the inputs for IWS mechanism, which are responsible for building the collision-inducing trajectories for the target ships. This space consists of:
scope-angle: ,
scope-length: .
where a sampled action from the policy is expressed as
Further details on how these actions are used are shown in § 5.3.
Intermediate Waypoint Sampling
IWS was first developed as a novel action sampling strategy that enables the discovery of realistic collision-inducing trajectories for EB-AST framework through a mechanism referred to as scoping. Conceptually, the method can be compared to a marksman try to pinpoint a target by adjusting its rifle angle and estimating the target’s distance. In this way, IWS provides lower-resolution control over the adversarial target ship’s trajectory, allowing a broader search space to be explored while reducing the agent’s aggressiveness and tendency to deliberately pursue collisions.
We first define two helper functions that will be used extensively for IWS. The first one is getHypot(A, B), used to compute the hypotenuse between point and , given by
The second one is getBeta(A, B), used to compute the angle between the north-axis and the line segment formed by connecting and . This function is formulated as
Each target ship is initialized with a waypoint sequence , containing the initial waypoints and . From this sequence, a waypoint pair is defined and used by the LOS guidance system to provide a heading reference to the target ship’s autopilot. The pair is retrieved using the function getWaypointPairs(i, ζ), such that
where denotes the index of the current waypoint in . As the agent encountered a new event, the advanceWaypointPairs(i) is called to increment by 1 to advance the waypoint pairing. To illustrate how the waypoint pair relates to the current state of the trajectory, consider the initial waypoint sequence before any IWs are introduced.
The IWS starts immediately after Event 1 is triggered (see § 5.2), which automatically calls initiateIWS() function. This function insert the zeroeth IW into based on the target ship’s north-east position when Event 1 is first triggered. Hence the current becomes
The zeroeth IW will then be used as the scoping base to compute the next IW using getNextWaypoint(i, ζ, ψ_t^{\mathrm{sc}}, β). The algorithm goes as follow:
Retrieve the-scope length and scope-angle from the policy, where the scope-angle is constrained so that .
Evaluate the sampled scope-length using
evaluateScopeLength().Retrieve the current waypoint pair using
getWaypointPairs(i, ζ).Compute the and using
getHypot(W_i),getBeta(W_i).Set the coordinates as the base point for the next waypoint .
The next waypoint can then be computed by solving
Insert into at index using
insertNewWaypoint(i, ζ)function.Finally, advance the waypoint pairs using
advanceWaypointPairs(i).
Having called this function, the condition of the trajectory should follows
Furthermore, the getNextWaypoint() function can be called repeatedly as long as Event 2 continues to be triggered by the target ships. This produces the updated trajectory condition defined in Equation (61). In addition, the overall process is illustrated in Figure 13 and Figure 14.

Figure 13. placement & computing .

Figure 14. Computing .
Typically, the IWS mechanism automatically stop once the action-sampling counts reach the predetermined limit of . However, there are special cases in which IWS is forced to stop even before the former condition is not fulfilled. When the sampled scope-length is greater than the length of the remaining untraversed segment-length , the next waypoint will not be computed and the last sampled IW will automatically connect to the end waypoint . Thus, the updated trajectory condition becomes
This case takes place only when the evaluateScopeLength() function returns TRUE, as illustrated in Figure 15. The maximum scope angle, , together with the scope-length evaluation, enables the generation of trajectories that remain flexible while maintaining forward progression, therefore supporting the objectives of EB-ASTv2.

Figure 15. Invalid IWS for . Close the trajectory.
Multi-Adversary Control through Action Masking

Figure 16. Action masking process. Value of 1 is given to the Target Ship(s) that is allowed to sample new IW.
Due to the structure of the EB-ASTv2 action space , the policy cannot sample an action exclusively for a particular IW-enabled target ship. Instead, an action is sampled simultaneously for all target ships represented in the action space. This design choice is intended because two or more target ships may trigger events at the same time, hence a ready supply of action is required at all decision events. In contrast, the multi-adversary control scheme allows each target ship to trigger a decision event independently of the others. Consequently, an event triggered by one target ship also causes the policy to produce actions for target ships that did not trigger an event. Although these additional actions do not result in trajectory modifications, they are still produced by the policy and may therefore introduce irrelevant action–observation associations during training.
Zouitine in his article [58] discusses three approaches for incorporating action masking: action-level masking, feature-level masking and agent-level masking. Action-level masking works by altering the value associated to the impossible action to . Feature-level masking controls which information is available to an agent when constructing its internal state representation. Agent-level masking determines which agents contribute to the policy-loss computation in a multi-agent setting.
To address the asynchronous event triggering in EB-ASTv2, we adopt a mechanism conceptually closer to feature-level masking. Rather than suppressing individual action values or dimensions, an additional observation vector is introduced to indicate which target ships triggered the current decision event. Let
denote this event-activity mask, where each entry corresponds to one IW-enabled target ship . An entry is set to when the corresponding target ship triggers an event and to otherwise. Since target ships trigger events independently, multiple entries may simultaneously take the value 1 at a given decision event.
The mask is included in the observation supplied to the policy. It provides an explicit indication of which target-ship-specific action components are active at the current decision event. Through training, the policy can associate entries with with actions that affect the corresponding target ship, while actions associated with are inactive and do not modify that ship’s trajectory.
Reward Function
The reward function provides the learning signal used to guide the EB-ASTv2 policy toward failure-inducing scenarios. Its design follows the general AST formulation introduced in Equation (35), while incorporating additional shaping terms that are specific to the IWS mechanism.
At each decision event, the reward is determined by whether the episode has terminated and whether the termination corresponds to a desired failure event. Let indicate episode termination and indicate whether the desired failure event has occurred. The reward is then expressed as
where
denotes the reward for a desired termination, denotes the reward for an undesired termination, and denotes the non-terminal reward. Here, “desired” refers to the objective of the stress-testing agent; from the perspective of the system under test, such an event represents a failure.
Moreover, denotes the final own-ship–target-ship separation at an undesired terminal event, while denotes the minimum separation observed up to decision event . The variable specifies the maximum number of IWs that may be sampled for a target ship, and denotes the number already sampled. The counter is initialized to zero at the beginning of each episode and incremented after every successful IW request. Furthermore, denotes the change between consecutive scope-angle commands, whereas denotes the angular difference between the sampled scope angle and the interception scope angle.
Note that both and are obtained by averaging of reward contributions from the IW-enabled target ships contained in the set , hence the superscript . Combined with the action masking protocol, this formulation enables the policy to control multiple adversarial target ships simultaneously. For brevity, the superscript is omitted from the remaining equations in this section unless otherwise stated.
For a desired termination, the terminal reward is set to
If the episode terminates without reaching the desired failure event, the agent instead receives a distance-based penalty
where is a scaling coefficient. The distance transformation is defined as
where denotes the separation threshold corresponding to collision. This function is illustrated in Figure 18.
Figure 18. Miss-distance reward function .
When the episode remains active, the reward serves two complementary purposes. First, it discourages large successive changes in the sampled scope angle, thereby promoting smoother target-ship trajectories. Second, it guides the sampled scope angle towards an interception course when such an interception is kinematically feasible. Accordingly, the non-terminal reward is decomposed as
where is the scope-angle-change term and is the interception-guidance term.
The scope-angle-change component is defined as
where
is the change between two consecutive sampled scope angles. The ratio gradually increases the contribution of this term as additional IWs are sampled. Additionally, is another scaling factor.
The function , as shown in Figure 17, penalizes large changes in scope angle. The change is evaluated under a zero-centered truncated normal distribution, such that small changes are assigned higher likelihood than abrupt changes. Let
Figure 17. Scope-angle change reward function .
denote the corresponding log-likelihood. The normalized scope-angle change function is then
where is the probability density function of a doubly truncated normal distribution [15],
where and denote the probability density function and cumulative distribution function of the standard normal distribution, respectively. plot is given in Figure 17.
The interception-guidance component is defined as
where and scale their respective reward term, and acts as a interception-guiding function. takes inspiration from the first reward design variation based on Goto et al. work [9]. This function is illustrated in Figure 19 and formulated as
Figure 19. Intercepting scope-angle reward function .
This second non-terminal reward term encourages the target ship to select scope angles that are consistent with an interception course through . However, it also fairly penalizes the remaining IW samplings when any interception attempts are no longer feasible. This is done by giving the target ship a reward based on how close it was able to get to the own ship through . At each decision event, a geometrically feasible interception heading is computed from the current relative positions and velocities of the own ship and target ship. The angular discrepancy between the sampled scope angle and the corresponding interception heading is denoted by
The interception heading prediction formula will be presented in Section A.1. Unlike the scope-angle-change term, the weighting of the interception component decreases as additional waypoints are sampled. Consequently, interception guidance dominates the early part of an episode, whereas trajectory smoothness becomes increasingly important as the target ship progresses through its available waypoint requests.
Results
Training Parameters
For the training process, we includes two IW-enabled target ships as the AST stressor. Each of target ship is equipped with IW FMU that allows it to change trajectory mid simulation. The parameter settings of the module for all target ships are shown in Table 1.
| Parameter | Value |
| Event 1 start time | 250 seconds |
| RoA | 300 meters |
| Maximum IWS counts | 5 |
| Maximum | 30.0° |
Table 1. IWS module parameters, for all target ships.
The training parameters for each RL algorithm are presented in Table 2 and Table 3. R-PPO and PPO share the same training because inherently they are derived from the same algorithm. The difference is that R-PPO uses MultiInputLstmPolicy to enable POMDP, whereas both PPO and SAC use MultiInputPolicy due to their MDP property. We aim to establish comparable training conditions across all RL algorithms to ensure a fair comparison. Specifically, we use the same learning rate, number of environments, discount factor , and total number of timesteps across all algorithms. We pick value of 1.0 because the failure-discovery problem is considered as a finite state-action transition problem. Additionally, we enable parallel training by using multiple environments to increase transition data gathering and speed up the training process. We could also identify how many neural network updates for each RL core based on the parameters, where the training happens differently for on-policy and off-policy method.
| Parameter | Value |
| Learning rate | |
| Number of environments | 64 |
| Rollout steps | 512 |
| Batch size | 1024 |
| Number of epochs | 5 |
| Discount factor () | 1.0 |
| Clipping range | 0.2 |
| Entropy coefficient | 0.005 |
| Value function coefficient | 0.5 |
| Maximum gradient norm | 0.5 |
| Advantage normalization | True |
| Total timesteps |
Table 2. R-PPO and PPO training hyperparameters for EB-ASTv2.
| Parameter | Value |
| Learning rate | |
| Number of environments | 64 |
| Replay buffer size | |
| Learning starts | |
| Batch size | 1024 |
| Discount factor () | 1.0 |
| Soft update coefficient () | 0.005 |
| Training frequency | 10 |
| Gradient steps | 128 |
| Entropy coefficient | Auto |
| Total timesteps |
Table 3. SAC training hyperparameters for EB-ASTv2.
For R-PPO and PPO, training was performed per rollout, with each rollout spanning 512 timesteps across 64 parallel environments. Therefore, each rollout collects 512 × 64 = 32,768 transitions. Given a batch size of 1,024, the number of minibatches per rollout is
Since each rollout is optimized for 5 epochs, the number of neural network updates per rollout is
For a total training budget of transitions, the number of rollouts is approximately
Thus, the total number of neural network updates for each R-PPO and PPO training run is approximately
updates. Thus, over 10 million timesteps, R-PPO and PPO performs a total of 48,828 gradient updates.
For SAC, training is performed after every training frequency of 10 environment steps once the initial exploration phase, specified by learning starts, has ended. With 64 parallel environments, each training frequency collects 10 × 64 = 640 transitions, which are stored in the replay buffer. After each collection cycle, SAC performs 128 gradient updates, with each update using a randomly sampled minibatch of 1024 transitions from the replay buffer. Over a total of 10 million timesteps, the number of collection cycles is
The exploration phase lasts for 128,000 timesteps, corresponding to
collection cycles during which no gradient updates are performed. Therefore, the total number of training cycles is
Since each training cycle consists of 128 gradient updates, the total number of gradient updates is
Thus, over 10 million timesteps, SAC performs a total of 1,974,400 gradient updates.
We train the policy using a 100 traffic cases generated with STG. Each traffic case centered on the own ship, whose initial position is located at north-east coordinate of (0, 0). For each generated case, STG parameters are sampled within their respective allowed range. which are specified in a JSON file named encounter_settings. The file’s contents are provided in Appendix Section A.2. At each call to reset(), the RL environment wrapper randomly selects one of the 100 traffic cases. The cases are indexed from 0 to 99, and the selection is performed using a uniform distribution, such that each case has an equal probability of being selected.
RL Cores’ Performance Comparisons
In this section, we discuss each RL core’s performances by comparing their reward collection and episode length trends over the 10 million training timesteps. We evaluate R-PPO’s performance by comparing its results directly with PPO and SAC in the following sections. Note that the maximum episode length will be 10 given that each target ship only allowed to sample 5 IWs per episode.
R-PPO vs PPO
Based on Figure 20, we see that both core yields a nearly-identical trends, where PPO reward is only negligibly higher compared to the R-PPO. Nevertheless, both grow from a low episodic reward of to a value above . This signifies that both implementation are steadily improved overtime, owing to the same objective architecture of R-PPO and PPO. The same goes with how the episode length decrease overtime, showed in Figure 21. Both implementation manage to decrease the episode length to nearly 6 IW samplings per episode before a termination is discovered.
Figure 20. Reward comparison.
Figure 21. Episode length comparison.
Looking more closely at the results in Table 4 and Table 5, R-PPO policy generates a larger number of failure scenarios, with 68 out of 100 cases resulting in own ship collision, compared with 58 out of 100 for the PPO policy. Additionally, the average episode length of the no collision and own ship collision differ slightly between the two methods: R-PPO policy tends to produce shorter no collision episodes of 6.6 and longer own ship collision episodes of 5.6, whereas PPO results exhibit the opposite trend with 6.8 long no collision episodes and 5.3 long own ship collision episodes.
| Outcome | Count | Percentage | Mean Episode Length |
| No Collision | 32 / 100 | 32.00% | 6.6 |
| Target Ship Collision | 0 / 100 | 0.00% | 0.0 |
| Own Ship Collision | 68 / 100 | 68.00% | 5.6 |
Table 4. R-PPO policy’s failure scenario generation training score.
| Outcome | Count | Percentage | Mean Episode Length |
| No Collision | 42 / 100 | 42.00% | 6.8 |
| Target Ship Collision | 0 / 100 | 0.00% | 0.0 |
| Own Ship Collision | 58 / 100 | 58.00% | 5.3 |
Table 5. PPO policy’s failure scenario generation training score.
R-PPO vs SAC
The results reveal a notable difference between the behavior of the R-PPO and SAC methods. Due to its off-policy learning scheme and substantially higher update frequency, the SAC results contain a denser set of training observations with greater variance. For visual clarity, the SAC results are therefore presented as mean curves with shaded variance bands.
As shown in Figure 22, the mean reward achieved by SAC increased rapidly for the first million training timesteps, until the increase slows down considerably for the remainder of the training phase. Eventually, the R-PPO’s mean reward catches up around the timesteps and stay higher compared to the SAC’s mean reward. However, the variability in the SAC results is considerably larger, with the upper range of the shaded bands frequently extending beyond the reward achieved by R-PPO. This indicates that although SAC performs worse on average, some training outcomes are comparable to or higher than those of R-PPO.
Figure 22. Reward comparison.
Figure 23 further shows that SAC produces longer episodes than R-PPO, averaging approximately 7–8 IW samplings per episode. This suggests that SAC generally requires more trajectory modifications before reaching a terminal state compared with R-PPO and PPO. This observation is also reflected in the evaluation results presented in Table 6. In particular, the SAC policy produces longer no collision and own ship collision episodes, with average lengths of 7.9 and 6.9, respectively. In comparison, the corresponding values are 6.6 and 5.6 for R-PPO, and 6.8 and 5.3 for PPO. Overall, these results indicate that SAC tends to require more IW sampling steps to reach a terminal outcome than either R-PPO or PPO.
| Outcome | Count | Percentage | Mean Episode Length |
| No Collision | 46 / 100 | 46.00% | 7.9 |
| Target Ship Collision | 0 / 100 | 0.00% | 0.0 |
| Own Ship Collision | 54 / 100 | 54.00% | 6.9 |
Table 6. SAC policy’s failure scenario generation training score.
Figure 23. Episode length comparison.
Performance Variability Across Independent Training Runs
We then investigate how the performance of each policy varies across independent training runs using identical hyperparameters and training settings. This experiment assesses whether the observed performance is consistent across runs, rather than being the result of particular training converging to a suboptimal local optimum. For each assessment, we compare the collision-inducing score and training time of each method. The score is defined as the percentage of 100 generated scenarios that result in a collision. All runs were performed on Norwegian University of Science and Technology (NTNU)’s IDUN High-Performance Computing systems [46], using 64 CPU cores and 256 GB of RAM per run. We limit the experiment to five runs per policy due to the substantial computational costs, as each training run generally requires more than five days.
Based on Table 7, we observe that R-PPO policy achieves the highest average score of 66.80%, compared to PPO and SAC scores of 65.20% and 51.80% respectively. R-PPO policy also requires the shortest average training time of 5.41 days, while PPO and SAC policies requires averagely 6.02 days and 7.09 days. However, PPO policy outperforms R-PPO policy in several runs, specifically in Trial 2, 3, and 5, and achieves the highest single-run score of 71.0% compared with R-PPO’s highest score of 69.0%.
| R-PPO | PPO | SAC | ||||
| Trial | Score (%) | Time (days) | Score (%) | Time (days) | Score (%) | Time (days) |
| 1 | 68.0 | 6.15 | 58.0 | 6.30 | 54.0 | 7.26 |
| 2 | 64.0 | 4.97 | 68.0 | 6.10 | 57.0 | 8.31 |
| 3 | 65.0 | 4.95 | 66.0 | 6.08 | 51.0 | 5.55 |
| 4 | 68.0 | 4.96 | 63.0 | 6.77 | 43.0 | 7.01 |
| 5 | 69.0 | 6.03 | 71.0 | 4.83 | 54.0 | 7.33 |
| Mean | 66.80 | 5.41 | 65.20 | 6.02 | 51.80 | 7.09 |
Table 7. Training time and performance across independent runs.
To examine this variability more closely, the results’ statistics are summarized in Table 8 and visualized in Figure 24. Although PPO policy occasionally achieves higher score than R-PPO policy, its scores are more dispersed across the independent runs. In contrasts, R-PPO scores are more tightly clustered around the mean, indicating more consistent performance. On the other hand, SAC policy’s score are generally lower and show and even wider spread than those of PPO.
| Method | Mean | Std. | Median | Q25 | Q75 | Min | Max | |
| R-PPO | 5 | 66.80 | 2.17 | 68.00 | 65.00 | 68.00 | 64.00 | 69.00 |
| PPO | 5 | 65.20 | 4.97 | 66.00 | 63.00 | 68.00 | 58.00 | 71.00 |
| SAC | 5 | 51.80 | 5.36 | 54.00 | 51.00 | 54.00 | 43.00 | 57.00 |
Table 8. Performance statistics across independent test runs.
Figure 24. Collision-inducing performance comparison of all EB-ASTv2 policies with 5 repeated trainings for each method.
We can also learn the scenario generation behavior of each policy by comparing the length of the generated collision and non-collision scenarios. Episode length is defined as the number of state-action transitions within an episode. The corresponding statistics are presented in Table 9 with the distribution visualized in Figure 25. The R-PPO policy tends to generate a longer collision scenarios, with episode length of 5.60, but a shorter non-collision scenarios, with a mean of 6.64. The opposite applies to PPO policy, where it tends to generate a shorter collision scenarios, with a mean of 5.36, and slightly longer non-collision scenarios, with a mean of 6.64. The SAC policy generates the longest scenarios for both outcomes, with a mean episode lengths of 6.98 and 7.68 for collision and non-collision respectively. These patterns are consistent with those observed in the initial-run comparisons in § 6.2, suggesting that the scenario-generation characteristics of the EB-ASTv2 policies remain similar across independent training runs.
| Method | Outcome | Mean | Std. | Median | Q25 | Q75 | Min | Max | |
| R-PPO | 5 | Collision | 5.60 | 0.07 | 5.6 | 5.6 | 5.6 | 5.5 | 5.7 |
| No collision | 6.64 | 0.11 | 6.6 | 6.6 | 6.7 | 6.5 | 6.8 | ||
| PPO | 5 | Collision | 5.36 | 0.11 | 5.4 | 5.3 | 5.4 | 5.2 | 5.5 |
| No collision | 6.68 | 0.08 | 6.7 | 6.6 | 6.7 | 6.6 | 6.8 | ||
| SAC | 5 | Collision | 6.98 | 0.08 | 7.0 | 6.9 | 7.0 | 6.9 | 7.1 |
| No collision | 7.68 | 0.15 | 7.7 | 7.6 | 7.7 | 7.5 | 7.9 |
Table 9. Scenario lengths statistics across independent test runs.
Figure 25. Collision-inducing episode lengths comparison of all EB-ASTv2 policies with 5 repeated trainings for each method.
These differences in episode length also suggest distinct scenario-generation behavior across the policies. PPO policy tend to be more conservative when generating collision-inducing scenarios, which reflects its slow-and-stable learning behavior influenced by the clipped surrogate objective. SAC policy tend to generate a more complex failure-inducing scenarios, where it tends to have more IWs that enables pronounced heading change and more flexible course. This behavior may be attributed to the higher variability during training, associated with SAC’s off-policy nature and exploration strategy. R-PPO policy lies between PPO and SAC policies in terms of its failure-inducing scenario generation tendency. This may be related to its recurrent architecture, which can better exploit state-action transition information while retaining the relatively stable learning characteristics of PPO. This ultimately allows R-PPO policy to handle better the limited observations from the co-simulation modules.
Scenario Generation Comparisons
In this section, we compare different generated scenarios by R-PPO, PPO, and SAC policies from a selected number of traffic cases. More specifically, we compare the generated scenarios from Case 50, 66, 96, 77, 89, and 44.
In Case 50, all three policies successfully generate an own ship collision scenario. For the R-PPO policy, the collision is triggered by target ship 2, whereas for both PPO and SAC, the collision involves target ship 1. In all three scenarios, the target ships maneuver toward the own ship and eventually overwhelm its COLAV system, resulting in a collision. However, the trajectories generated by SAC contain more IWs and exhibit more pronounced heading changes than those generated by R-PPO and PPO. This indicates that SAC tends to produce more complex and highly maneuvering trajectories when searching for collision-inducing scenarios.
In some cases, the PPO policy is unable to generate a true own ship collision scenario. This is demonstrated in Case 66, where both R-PPO and SAC successfully identify collision-inducing scenarios, whereas PPO does not. The scenarios generated by R-PPO and SAC share a notable similarity: both contain additional turns and maneuvers before eventually leading to an own ship collision. On the contrary, the trajectory generated by PPO remains
Figure 26. R-PPO trajectory from Case 50 (TS2-collision).
Figure 27. R-PPO trajectory from Case 66 (TS2-collision).
Figure 28. PPO trajectory from Case 50 (TS1-collision).
Figure 29. PPO trajectory from Case 66 (No collision).
Figure 30. SAC trajectory from Case 50 (TS1-collision).
Figure 31. SAC trajectory from Case 66 (TS2-collision).
Figure 32. R-PPO trajectory from Case 96 (TS2-collision).
Figure 33. R-PPO trajectory from Case 77 (No collision).
Figure 34. PPO trajectory from Case 96 (TS1-collision).
Figure 35. PPO trajectory from Case 77 (No collision).
Figure 36. SAC trajectory from Case 96 (No collision).
Figure 37. SAC trajectory from Case 77 (TS1-collision).
Figure 38. R-PPO trajectory from Case 89 (No collision).
Figure 40. PPO trajectory from Case 89 (No collision).
Figure 42. SAC trajectory from Case 89 (No collision).
Figure 39. R-PPO trajectory from Case 44 (No collision).
Figure 41. PPO trajectory from Case 44 (No collision).
Figure 43. SAC trajectory from Case 44 (No collision).
comparatively straight, suggesting that its more conservative maneuvering limits its ability to discover a collision scenario in this case.
Conversely, there are cases in which the SAC policy fails to generate an own ship collision scenario. In Case 96, R-PPO generates a scenario in which the own ship collides with target ship 2, while PPO generates a collision involving target ship 1. The R-PPO trajectories, particularly that of target ship 2, contain more IWs and therefore exhibit more frequent and pronounced heading changes although still maintaining a rather simple course. The PPO trajectories are considerably simpler and succeed in triggering a collision relatively early by maneuvering target ship 1 toward the own ship. The trajectories generated by SAC, on the other hand, are comparatively complex but ultimately fail to produce an own ship collision.
Nevertheless, there are specific cases in which SAC performs better than its R-PPO and PPO counterparts. In Case 77, both R-PPO and PPO struggle to identify collision-inducing scenarios because the own ship travels significantly faster than both target ship 1 and target ship 2. Consequently, a substantially larger interception angle is required for either target ship to reach the own ship. The trajectories generated by R-PPO and PPO remain relatively conservative, with both policies maintaining comparatively straight courses. In contrast, SAC overcomes this limitation by selecting a trajectory with a much larger interception angle, followed by a sharp turn that ultimately results in an own ship collision. This case illustrates how the greater maneuvering flexibility exhibited by SAC can be advantageous when more aggressive trajectory changes are required to induce a collision.
Furthermore, none of the policies is guaranteed to identify a failure scenario for every initial condition. This is illustrated by Cases 89 and 44, in which none of the three policies succeeds in generating a collision-inducing trajectory. In Case 89, target ship 1 is initially located sufficiently far from the own ship that interception is infeasible given its speed, while target ship 2 travels too slowly to intercept the own ship. Similarly, in Case 44, interception is infeasible because the feasible trajectories of the two target ships do not intersect the own ship’s trajectory in a way that allows a collision to occur. Nevertheless, even in these unsuccessful cases, the generated trajectories still exhibit behavior consistent with an attempt to induce a collision: the policies maneuver the target ships toward the own ship and thereby increase the collision risk, despite ultimately being unable to produce an actual collision.
Continued ebast2 Training for Improving the Policy
Figure 45. Episode length comparison.
We then evaluate whether the R-PPO policy can be further improved through continued training. The previously trained policy is trained for an additional 4 million timesteps, and its performance during this second training phase is evaluated. As shown in Table 10, the retrained R-PPO policy achieves a 2% improvement in generating own-ship collision scenarios. This is consistent with the mean reward trend in Figure 44. The gray-shaded region represents the previous training phase, while the orange dashed line shows an ordinary least-squares linear fit to the continued training phase. The mean reward increase are quite modest, whereas the mean episode length remains relatively stable. Although small, this improvement demonstrates the potential for further refinement of an already trained EB-ASTv2 policy.
Figure 44. Reward comparison.
| Outcome | Count | Percentage | Mean Episode Length |
| No Collision | 30 / 100 | 30.00% | 6.5 |
| Target Ship Collision | 0 / 100 | 0.00% | 0.0 |
| Own Ship Collision | 70 / 100 | 70.00% | 5.6 |
Table 10. Second-trained R-PPO policy’s training score.
Continued ebast2 Training with Different colav System
| Outcome | Count | Percentage | Mean Episode Length |
| No Collision | 45 / 100 | 45.00% | 6.8 |
| Target Ship Collision | 0 / 100 | 0.00% | 0.0 |
| Own Ship Collision | 55 / 100 | 55.00% | 5.5 |
Table 12. Second-trained R-PPO policy’s training score using different COLAV system.
We also evaluate whether the trained R-PPO policy can generalize to a different COLAV system during stress testing. For simplicity, the alternative system is based on the implementation in § 3.3, but its response is changed from turning to starboard to turning to port. When the trained R-PPO policy is applied to this modified system, its success rate in generating own ship collision scenarios drops significantly from 68% to 19%, as shown in Table 11. This is consistent with Figure 46, where the mean reward drops sharply after 10 million timesteps. At the same time, the mean episode length increases, indicating that the policy struggles to find effective collision-inducing scenarios.
| Outcome | Count | Percentage | Mean Episode Length |
| No Collision | 81 / 100 | 81.00% | 6.6 |
| Target Ship Collision | 0 / 100 | 0.00% | 0.0 |
| Own Ship Collision | 19 / 100 | 19.00% | 6.4 |
Table 11. First-trained R-PPO policy’s training score using different COLAV system.
Figure 46. Reward comparison.
We therefore retrain the R-PPO policy for another 10 million timesteps. During this training phase, each episode has a 50% chance of using either turning to starboard or turning to port COLAV system, exposing the policy to both behaviors during training. After retraining, the policy achieves a 55% collision generation rate for either configuration. This improvement is also reflected in Figure 46, where the mean reward increases throughout the second training phase, while the mean episode length decreases in Figure 47. These results indicate that the retrained policy generalizes better across different COLAV behaviors while remaining effective at generating collision-inducing scenarios consistent with the AST objective.
Figure 47. Episode length comparison.
Conclusion
Through our experiment, the proposed EB-ASTv2 framework demonstrates its capability as both verification and safety validation methods for CPS by fulfilling these three goals:
Providing counter-examples across a wide range of ship traffic scenarios by adopting AST as the core algorithm.
Supports realistic scenario discovery by generating advancing adversarial trajectories and maintaining a balance between trajectory smoothness, interception intent through IWS.
Demonstrates an increasing ability of the trained policies to generate collision-inducing scenarios, quantified by the failure rate across training cases.
We also compared different RL-algorithm cores, namely R-PPO, PPO and SAC, to demonstrate how the POMDP formulation might be advantageous, compared to its MDP alternatives, as the core of the proposed EB-ASTv2 method to stress test black-box simulation like Co-simulation. The proposed R-PPO policy adoption proves a higher average failure-discovery performance of 66.80% compared to PPO policy (65.20%) and SAC policy (51.80%). Despite this, different RL policy cores exhibit distinct scenario-generation characteristics. Overall, the three RL cores exhibit distinct scenario-generation characteristics. PPO shows more conservative behavior, SAC generates more variable and complex failure scenarios, while R-PPO provides a balance between these characteristics. The recurrent structure of R-PPO appears particularly beneficial in handling the limited observations provided by the co-simulation modules.
Additionally, we demonstrate that a trained policy can be further improved through continued training. The retrained R-PPO policy achieves a higher own-ship collision generation rate, suggesting the potential for an iterative safety-assessment cycle: after a system is stress tested and improved, the EB-ASTv2 policy can be retrained to evaluate the updated system. We also show that the policy can adapt to different system behaviors by exposing it to both the original and modified systems during retraining phase. This is achieved by uniformly sampling the system configuration at each episode reset(). In our experiment, varying the COLAV response enables the retrained policy to generalize across both configurations while still generating meaningful failure scenarios. Together, these results demonstrate the potential of the EB-ASTv2 framework as a safety-assessment method for black-box maritime CPS.
Acknowledgments and Disclosure of Funding
This work was supported by the Research Council of Norway under the project System optimisation and stress testing in co-simulations (OptiStress) (grant no. 344238). The authors would also like to thank Børge Rokseth and Stian Skjong for their valuable support in facilitating this research collaboration between NTNU, SINTEF, and DNV.
Appendix / supplemental material
Interception Heading Prediction
In order to compute the interception-guidance reward term, a procedure has been developed to determine whether a target ship may intercept the own ship at every decision event. If interception is deemed feasible, it then predicts the interception heading accordingly.
Let us define own ship positions (), target ship positions (), own ship velocities (), target ship forward speed (), and target ship directional vector unit (). An interception can be framed from the kinematic perspective as
Refactor this by moving to left side, and we get
where following Equation (36). By applying , we ended up with
By squaring both sides, and arranging the equation by moving all terms to the left side, we get
Above is the quadratic formula that follows . Observe now that we have
From these, we can learn several things about the own-ship–target-ship kinematics. When means that both own ship and target ship travels at a similar speed, hence the only enabling factor for feasible interception are their geometrical positions. We can also compute the discriminant as
where it determines whether interception is infeasible when . If the discriminant is positive, solution exists and the roots of (82) can solved by computing
These terms are all of the possible times-to-intercept. Given the threshold , interception is only feasible when at least one of the root fulfill the condition . Beyond that, the interception either occurred in the past or is too far in the future.
Using the solution , target ship velocity can now be computed by
where the interception heading can be obtained by solving
stg parameters
| Parameter | Value |
| criteria | |
| criteria | |
| criteria | |
| head on | |
| overtaking stand-on | |
| overtaking give-way | |
| crossing stand-on | |
| crossing give-way | |
| Situation length | 90.0 minutes |
| Maximum meeting distance | 5.0 nautical miles |
| Situation develop time | 60.0 minutes |
| Vector time | minutes |
| Own ship SoG | 6.0 knots |
| Own ship CoG | |
| Margin | 2.0° |
| head on | |
| overtaking give-way | |
| overtaking stand-on starboard | |
| overtaking stand-on port | |
| crossing give-way | |
| crossing stand-on | |
| Navigation status | Under way using engine |
Table 13. STG parameters
References
- [1]Open Simulation Platform. URL https://opensimulationplatform.com/.
- [2]libcosim: An open-source co-simulation library for C++, May 2026. URL https: //github.com/open-simulation-platform/libcosim.
- [4]O. Balci. Verification, validation, and certification of modeling and simulation applications. In Proceedings of the 2003 International Conference on Machine Learn- ing and Cybernetics (IEEE Cat. No.03EX693), pages 150–158, New Orleans, LA, USA, 2003. IEEE. ISBN 978-0-7803-8131-5. doi: 10.1109/WSC.2003.1261418. URL http://ieeexplore.ieee.org/document/1261418/.DOI
- [5]Torsten Blockwitz, Martin Otter, Johan Akesson, Martin Arnold, Christoph Clauss, Hilding Elmqvist, Markus Friedrich, Andreas Junghanns, Jakob Mauss, Dietmar Neumerkel, Hans Olsson, and Antoine Viel. Functional Mockup Interface 2.0: The Standard for Tool independent Exchange of Simulation Models. pages 173–184, November 2012. doi: 10.3384/ecp12076173. URL https://ep.liu.se/en/conference-article.aspx?series=ecp&issue=76&Article_No=17.DOI
- [6]Yingguang Chu, Lars Ivar Hatledal, Houxiang Zhang, Vilmar Æsøy, and Sören Ehlers. Virtual prototyping for maritime crane design and operations. Journal of Marine Science and Technology, 23(4):754–766, December 2018. ISSN 0948-4280, 1437-8213. doi: 10.1007/s00773-017-0509-z. URL http://link.springer.com/10.1007/s00773-017-0509-z.DOI
- [7]Anthony Corso, Robert J. Moss, Mark Koren, Ritchie Lee, and Mykel J. Kochenderfer. A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems. Journal of Artificial Intelligence Research, 72, October 2021. ISSN 1076-9757. doi: 10.1613/jair.1.12716. URL http://arxiv.org/abs/2005.02979. arXiv:2005.02979 [cs, eess, stat].
- [8]Angela Errico and Vittorio Di Vito. Methodology for Estimation of Closest Point of Approach between Aircraft in ATM. In AIAA Aviation 2019 Forum, Dallas, Texas, June 2019. American Institute of Aeronautics and Astronautics. ISBN 978-1-62410-589-0. doi: 10.2514/6.2019-2832. URL https://arc.aiaa.org/doi/10.2514/6.2019-2832.DOI
- [9]Takeru Goto, Yuki Kizumi, and Shun Iwasaki. Design of Reward Function on Reinforcement Learning for Automated Driving. IFAC-PapersOnLine, 56(2):7948–7953, 2023. ISSN 2405-8963. doi: 10.1016/j.ifacol.2023.10.902. URL https://linkinghub.elsevier.com/retrieve/pii/S2405896323012855.DOI
- [11]Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Algorithms and Applications, 2018. URL https://arxiv.org/abs/1812.05905. Version Number: 2.
- [12]Vahid Hassani, Martin Rindarøy, Lars T. Kyllingstad, Jørgen B. Nielsen, Severin Simon Sadjina, Stian Skjong, Dariusz Fathi, Trond Johnsen, Vilmar Æsøy, and Eilif Pedersen. Virtual Prototyping of Maritime Systems and Operations. In Volume 7: Ocean Engineering, page V007T06A018, Busan, South Korea, June 2016. American Society of Mechanical Engineers. ISBN 978-0-7918-4998-9. doi: 10.1115/OMAE2016-54886. URL https://asmedigitalcollection.asme.org/OMAE/proceedings/OMAE2016/49998/Busan,%20South%20Korea/281064.DOI
- [13]Hanna W. Hjelmeland, Bjørn-Olav H. Eriksen, Ole J. Mengshoel, and Anastasios M. Lekkas. Identification of Failure Modes in the Collision Avoidance System of an Autonomous Ferry using Adaptive Stress Testing. IFAC-PapersOnLine, 55(31):470–477, 2022. ISSN 24058963. doi: 10.1016/j.ifacol.2022.10.472. URL https://linkinghub.elsevier.com/retrieve/pii/S2405896322025186.DOI
- [14]Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, November 1997. ISSN 0899-7667, 1530-888X. doi: 10.1162/neco.1997.9.8.1735. URL https://direct.mit.edu/neco/article/9/8/1735-1780/6109.DOI
- [15]Norman Lloyd Johnson, Samuel Kotz, and Narayanaswamy Balakrishnan. Distributions in statistics. 1: Continuous univariate distributions / Norman L. Johnson ; Samuel Kotz ; N. Balakrishnan. Wiley series in probability and mathematical statistics. Wiley, New York, 2. ed edition, 1994. ISBN 978-0-471-58495-7. Num Pages: 756.
- [16]James Kapinski, Jyotirmoy V. Deshmukh, Xiaoqing Jin, Hisahiro Ito, and Ken Butts. Simulation-Based Approaches for Verification of Embedded Control Systems: An Overview of Traditional and Advanced Modeling, Testing, and Verification Techniques. IEEE Control Systems, 36(6):45–64, December 2016. ISSN 1066-033X, 1941-000X. doi: 10.1109/MCS.2016.2602089. URL https://ieeexplore.ieee.org/document/7741019/.
- [17]Mark Koren and Mykel Kochenderfer. Efficient Autonomy Validation in Simulation with Adaptive Stress Testing, July 2019. URL http://arxiv.org/abs/1907.06795. arXiv:1907.06795 [cs, eess, stat].
- [18]Mark Koren and Mykel J. Kochenderfer. Adaptive Stress Testing without Domain Heuristics using Go-Explore, June 2020. URL http://arxiv.org/abs/2004.04292. arXiv:2004.04292 [cs, eess, stat].
- [19]Mark Koren, Ahmed Nassar, and Mykel J. Kochenderfer. Finding Failures in High-Fidelity Simulation using Adaptive Stress Testing and the Backward Algorithm, July 2021. URL http://arxiv.org/abs/2107.12940. arXiv:2107.12940 [cs.LG].
- [20]Martin Krammer, Michael Karner, and Anton Fuchs. System design for enhanced forward-engineering possibilities of safety critical embedded systems. In 17th International Symposium on Design and Diagnostics of Electronic Circuits & Systems, pages 234–237, Warsaw, Poland, April 2014. IEEE. ISBN 978-1-4799-4558-0 978-1-4799-4560-3. doi: 10.1109/DDECS.2014.6868797. URL http://ieeexplore.ieee.org/document/6868797/.DOI
- [21]Thomas Steenfeldt Laursen and Ole Jakob Mengshoel. Evolutionary Adaptive Stress Testing for Collision Avoidance in Sustainable Maritime Transportation. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 2215–2223, NH Malaga Hotel Malaga Spain, July 2025. ACM. ISBN 979-8-4007-1464-1. doi: 10.1145/3712255.3734306. URL https://dl.acm.org/doi/10.1145/3712255.3734306.
- [22]Ritchie Lee, Mykel J. Kochenderfer, Ole J. Mengshoel, Guillaume P. Brat, and Michael P. Owen. Adaptive stress testing of airborne collision avoidance systems. In 2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC), pages 6C2–1–6C2–13, Prague, Czech Republic, September 2015. IEEE. ISBN 978-1-4799-8940-9. doi: 10.1109/DASC.2015.7311450. URL http://ieeexplore.ieee.org/document/7311450/.DOI
- [23]Ritchie Lee, Ole J. Mengshoel, Anshu Saksena, Ryan Gardner, Daniel Genin, Joshua Silbermann, Michael Owen, and Mykel J. Kochenderfer. Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning, December 2020. URL http://arxiv.org/abs/1811.02188. arXiv:1811.02188 [cs].
- [24]Andrzej S. Lenart. Analysis of Collision Threat Parameters and Criteria. Journal of Navigation, 68(5):887–896, September 2015. ISSN 0373-4633, 1469-7785. doi: 10.1017/S0373463315000223. URL https://www.cambridge.org/core/product/identifier/S0373463315000223/type/journal_article.DOI
- [25]P. Marbach and J.N. Tsitsiklis. Simulation-based optimization of Markov reward processes. In Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171), volume 3, pages 2698–2703, Tampa, FL, USA, 1998. IEEE. ISBN 978-0-7803-4394-8. doi: 10.1109/CDC.1998.757861. URL http://ieeexplore.ieee.org/document/757861/.
- [26]Douglas C. Montgomery. Design and analysis of experiments. John Wiley & Sons, Inc, Hoboken, NJ, ninth edition edition, 2017. ISBN 978-1-119-11347-8 978-1-119-29936-3 978-1-119-32093-7.
- [27]Rolf Nagel. A Holistic Approach for Energy Flow Simulations in Early Design. Transportation Research Procedia, 14:1601–1610, 2016. ISSN 2352-1465. doi: 10.1016/j.trpro.2016.05.125. URL https://linkinghub.elsevier.com/retrieve/pii/S2352146516301260.DOI
- [28]Demin Nalic, Arno Eichberger, Georg Hanzl, Martin Fellendorf, and Branko Rogic. Development of a Co-Simulation Framework for Systematic Generation of Scenarios for Testing and Validation of Automated Driving Systems. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 1895–1901, Auckland, New Zealand, October 2019. IEEE. ISBN 978-1-5386-7024-8. doi: 10.1109/ITSC.2019.8916839. URL https://ieeexplore.ieee.org/document/8916839/.DOI
- [29]Christopher Olah. Understanding LSTM Networks, August 2015. URL https://colah.github.io/posts/2015-08-Understanding-LSTMs/#fn1.
- [30]Jean-Nicolas Paquin, Wei Li, Jean Belanger, Loic Schoen, Irene Peres, Cristina Olariu, and Hugo Kohmann. A modern and open real-time digital simulator of All-Electric Ships with a multi-platform co-simulation approach. In 2009 IEEE Electric Ship Technologies Symposium, pages 28–35, Baltimore, MD, USA, April 2009. IEEE. ISBN 978-1-4244-3438-1. doi: 10.1109/ESTS.2009.4906490. URL http://ieeexplore.ieee.org/document/4906490/.DOI
- [31]Tom Arne Pedersen, Chanjei Vasanthan, Kristian Karolius, Øystein Engelhardtsen, Koen Pieter Houweling, and Are Jørgensen. Ship Traffic Generator.
- [32]Tom Arne Pedersen, Chanjei Vasanthan, Kristian Karolius, Øystein Engelhardtsen, Koen Pieter Houweling, and Are Jørgensen. Generating Structured Set of Encounters for Verifying Automated Collision and Grounding Avoidance Systems. Journal of Physics: Conference Series, 2618(1):012013, October 2023. ISSN 1742-6596, 1742-6596. doi: 10.1088/1742-6596/2618/1/012013. URL https://iopscience.iop.org/article/10.1088/1742-6596/2618/1/012013.DOI
- [33]Florian Perabo and Mehdi Zadeh. Multiphysics Modeling and Co-Simulation of Ship Electric Power and Propulsion Systems for Virtual Testing and Verification. IEEE Transactions on Transportation Electrification, pages 1–1, 2024. ISSN 2332-7782, 2372-2088. doi: 10.1109/TTE.2024.3480009. URL https://ieeexplore.ieee.org/document/10716430/.DOI
- [34]Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization, 2022. URL https://arxiv.org/abs/2205.11104. Version Number: 1.
- [35]Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, and Maximilian Ernestus. Stable Baselines3, March 2021. URL https://github.com/Stable-Baselines-Team/stable-baselines3-contrib.
- [36]Marvin Rausand. Reliability of Safety-Critical Systems: Theory and Applications. Wiley, 1 edition, January 2014. ISBN 978-1-118-11272-4 978-1-118-77635-3. doi: 10.1002/9781118776353. URL https://onlinelibrary.wiley.com/doi/book/10.1002/9781118776353.DOI
- [37]Stuart J. Russell and Peter Norvig. Artificial intelligence: a modern approach. Pearson Series in Artificial Intelligence. Pearson, Hoboken, NJ, fourth edition edition, 2021. ISBN 978-0-13-461099-3.
- [38]Severin Sadjina, Lars Tandle Kyllingstad, Martin Rindarøy, Stian Skjong, Vilmar Æsoy, and Eilif Pedersen. Distributed Co-Simulation of Maritime Systems and Operations. Journal of Offshore Mechanics and Arctic Engineering, 141(1):011302, February 2019. ISSN 0892-7219, 1528-896X. doi: 10.1115/1.4040473. URL https://asmedigitalcollection.asme.org/offshoremechanics/article/doi/10.1115/1.4040473/366241/Distributed-Co-Simulation-of-Maritime-Systems-and.DOI
- [39]A. Saltelli. Global sensitivity analysis: the primer. John Wiley, Chichester, England, 2008. ISBN 978-0-470-72518-4. OCLC: 212122308.
- [40]Robert G. Sargent. Verification and validation of simulation models. In Proceedings of the 2010 Winter Simulation Conference, pages 166–183, Baltimore, MD, USA, December 2010. IEEE. ISBN 978-1-4244-9866-6. doi: 10.1109/WSC.2010.5679166. URL http://ieeexplore.ieee.org/document/5679166/.DOI
- [41]Robin M. Schmidt. Recurrent Neural Networks (RNNs): A gentle Introduction and Overview, November 2019. URL http://arxiv.org/abs/1912.05911. arXiv:1912.05911 [cs.LG].
- [42]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017. URL http://arxiv.org/abs/1707.06347. arXiv:1707.06347 [cs.LG].
- [43]G. Schweiger, C. Gomes, G. Engel, I. Hafner, J. Schoeggl, A. Posch, and T. Nouidui. An empirical survey on co-simulation: Promising standards, challenges and research needs. Simulation Modelling Practice and Theory, 95:148–163, September 2019. ISSN 1569190X. doi: 10.1016/j.simpat.2019.05.001. URL https://linkinghub.elsevier.com/retrieve/pii/S1569190X1930053X.DOI
- [44]Guy Shani, Joelle Pineau, and Robert Kaplow. A survey of point-based POMDP solvers. Autonomous Agents and Multi-Agent Systems, 27(1):1–51, July 2013. ISSN 1387-2532, 1573-7454. doi: 10.1007/s10458-012-9200-2. URL http://link.springer.com/10.1007/s10458-012-9200-2.DOI
- [45]Andreas R.G. Sitorus and Børge Rokseth. Identifying Ship in Transit Operation Vulnerabilities through Event-Based Adaptive Stress Testing. IFAC-PapersOnLine, 59 (22):716–721, 2025. ISSN 24058963. doi: 10.1016/j.ifacol.2025.11.719. URL https://linkinghub.elsevier.com/retrieve/pii/S2405896325024152.DOI
- [46]Magnus Sjålander, Magnus Jahre, Gunnar Tufte, and Nico Reissmann. EPIC: An Energy-Efficient, High-Performance GPGPU Computing Research Infrastructure, July 2024. URL http://arxiv.org/abs/1912.05848. arXiv:1912.05848 [cs.DC].
- [47]Stian Skjong, Martin Rindarøy, Lars T. Kyllingstad, Vilmar Æsøy, and Eilif Pedersen. Virtual prototyping of maritime systems and operations: applications of distributed co-simulations. Journal of Marine Science and Technology, 23(4):835–853, December 2018. ISSN 0948-4280, 1437-8213. doi: 10.1007/s00773-017-0514-2. URL http://link.springer.com/10.1007/s00773-017-0514-2.DOI
- [48]F.M. Sogandares. Stone axes and warhammers: a decade of distributed simulation in aviation research. In Proceedings 16th Workshop on Parallel and Distributed Simulation, pages 114–121, Washington, DC, USA, 2002. IEEE Comput. Soc. ISBN 978-0-7695-1608-0. doi: 10.1109/PADS.2002.1004208. URL http://ieeexplore.ieee.org/document/1004208/.
- [49]Magnus Stein stø. libcosimpy: Python wrapper for libcosim, June 2026. URL https://github.com/open-simulation-platform/libcosimpy.
- [51]Richard S. Sutton and G. Barto Barto. Reinforcement Learning: an introduction. MIT Press, Cambridge, MA, 2 edition, 2018. ISBN 978-0-262-03924-6.
- [52]Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf.
- [53]Tobias Rye Torben, Jon Arne Glomsrud, Tom Arne Pedersen, Ingrid B Utne, and Asgeir J Sørensen. Automatic simulation-based testing of autonomous ships using Gaussian processes and temporal logic. Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, 237(2):293–313, April 2023. ISSN 1748-006X, 1748-0078. doi: 10.1177/1748006X211069277. URL http://journals.sagepub.com/doi/10.1177/1748006X211069277.DOI
- [54]Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A Standard Interface for Reinforcement Learning Environments, 2024. URL https://arxiv.org/abs/2407.17032. Version Number: 4.
- [55]Michael Winter, Julian Taube, Joachim Froeschl, and Hans-Georg Herzog. From Simulation to Testbench Using the FMI-Standard. In 2015 IEEE Vehicle Power and Propulsion Conference (VPPC), pages 1–5, Montreal, QC, Canada, October 2015. IEEE. ISBN 978-1-4673-7637-2. doi: 10.1109/VPPC.2015.7352951. URL http://ieeexplore.ieee.org/document/7352951/.
- [56]Junjie Zhao, Christopher Conrad, Rodolphe Fremond, Anurag Mukherjee, Quentin Delezennne, Yu Su, Yan Xu, and Antonios Tsourdos. Co-simulation Digital Twin Framework for Testing Future Advanced Air Mobility Concepts: A Study with BlueSky and AirSim. In 2023 IEEE/AIAA 42nd Digital Avionics Systems Conference (DASC), pages 1–10, Barcelona, Spain, October 2023. IEEE. ISBN 978-3-3503-3357-2. doi: 10.1109/DASC58513.2023.10311124. URL https://ieeexplore.ieee.org/document/10311124/.DOI
- [57]Xingyi Zhu, Yanan Wu, Yang Yang, Yafeng Pang, Hongwei Ling, and Dawei Zhang. Real-time risk assessment of aircraft landing based on finite element-virtual prototype-machine learning co-simulation on wet runways. International Journal of Transportation Science and Technology, 13:77–97, March 2024. ISSN 20460430. doi: 10.1016/j.ijts.2023.11.007. URL https://linkinghub.elsevier.com/retrieve/pii/S2046043023001041.DOI
- [58]Adil Zouitine. Masking in Deep Reinforcement Learning, June 2022. URL https://boring-guy.sh/posts/masking-rl/.
- [59]K.J Åström. Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10(1):174–205, February 1965. ISSN 0022247X. doi: 10.1016/0022-247X(65)90154-X. URL https://linkinghub.elsevier.com/retrieve/pii/0022247X6590154X.DOI































