betaFeedback

WorksEngineering

Co-simulation-Based Safety-Assessment of Black-Box Maritime Cyber-Physical Systems

Abstract

Modern maritime cyber-physical systems consist of tightly interconnected subsystems ranging from multiple engineering domains, making exhaustive safety verification impractical. Simulation-based testing offers a scalable alternative, while co-simulation enables heterogeneous subsystem models, potentially developed by different stakeholders, to be integrated without requiring access to their internal implementations. Adaptive Stress-Testing further improves testing efficiency by formulating failure discovery as a sequential decision-making problem, guiding the search toward the most likely failure scenarios, and giving the counter safety examples using reinforcement learning. However, black-box co-simulation may provide only partial access to the system state required by conventional Markov decision process formulations. To address this limitation, we proposed Event-Based Adaptive Stress-Testing version 2, a co-simulation-based safety-assessment method formulated as a partially observable Markov decision process. This method employs Recurrent Proximal Policy Optimization to exploit observation histories when selecting adversarial actions. The method is evaluated on a ship-in-transit operation in which a collision-avoidance system is stress-tested against multi-adversarial target ships. An extended Intermediate Waypoint Sampling mechanism enables independent multi-adversary trajectory control and supports the generation of realistic collision-inducing scenarios. The experiment demonstrate that the learned stress-testing policy generalizes across different encounter cases, can be further improved through continued training, and can be made effective when the collision- avoidance system under test is replaced by another version. These results demonstrate the method's potential for scalable black-box safety assessment of complex maritime cyber-physical systems.

Introduction

Advances in maritime automation are allowing increasingly autonomous and functionally integrated vessel operations. Modern maritime systems may consist of tightly coupled propulsion, navigation, machinery, energy-management, sensing, and control subsystems, together with intelligent modules such as collision avoidance. Their interaction increasingly gives rise to complex maritime Cyber-Physical Systems (CPSs), whose underlying subsystems may also be developed by different stakeholders. Co-simulation provides an enabling technology for integrating reusable and interchangeable subsystem models into a common virtual prototype, supporting rapid system development and system-level simulation [38].

The increasing integration of these subsystems also makes safety assessment more challenging. Component-level verification alone provides limited evidence of system-level behavior because failures may emerge from interactions between each correctly functioning subsystems. Consequently, complex maritime CPSs require testing approaches that considers all subsystem interactions under given operational conditions.

Simulation-based testing provides a scalable ways for evaluating such systems without exposing physical assets to potentially hazardous scenarios. However, its effectiveness strongly depends on the selection of the test cases. Test scenarios are traditionally selected manually using expert knowledge, operational experience, and risk analysis. On the other hand, systematic experimental-design approaches such as factorial designs, fractional factorial designs, Latin Hypercube Sampling (LHS), and Sobol sequences can be used to sample the input space and characterize system variability [26, 39]. Nevertheless, with a large number of interacting variables and possible environmental conditions exhaustive explorations become computationally infeasible. Automatic search mechanisms that direct simulations toward safety-critical regions are therefore desirable.

Adaptive Stress-Testing (AST) addresses this problem by formulating failure-scenario discovery as a sequential decision-making problem and using an optimization or Reinforcement Learning (RL) agent to search for likely failure trajectories. Different search strategies have been investigated, including Monte Carlo Tree Search (MCTS) [21, 22], the Backward Algorithm [19], Go-Explore [18], and deep RL-based approaches [13, 45]. These methods have demonstrated the capability of AST to stress-test safety-critical systems in domains including automotive, aerospace, and, more recently, maritime applications.

However, applying AST directly to distributed co-simulation introduces an additional challenge. Individual subsystem models may be accessible only through their exposed interfaces, while their complete internal states remain unavailable to the stress-testing agent. Therefore, the agent must make decisions from incomplete observations and infer information relevant to the underlying system state from its interaction history. Under these conditions, the fully observable Markov Decision Process (MDP) formulation is insufficient, and the stress-testing problem should be represented as a Partially Observable Markov Decision Process (POMDP). This motivates an AST formulation that is compatible with black-box co-simulation interfaces by having it capable of retaining temporal information.

Our Approach and Contributions

This article presents Event-Based Adaptive Stress-Testing v2 (EB-ASTv2), an extension of AST for simulation-based failure discovery in partially observable maritime co-simulation. The method is evaluated using a ship-in-transit scenario in which an RL agent controls multiple adversarial target ships to search for collision-inducing scenarios against an own ship equipped with a Collision-Avoidance (COLAV) system.

The principal contributions of this work are:

  1. Partial-observability-aware stress testing. EB-ASTv2 employs Recurrent Proximal Policy Optimization (R-PPO) as its primary policy-optimization method. The recurrent policy retains information from previous observations and actions, enabling the agent to account for temporal information that is not available from the current co-simulation observation alone. The trained recurrent policy’s performance will also be compared with its non-recurrent policy alternatives.

  1. Multi-adversary scenario generation. The framework introduces multiple adversarial control feature to enhance its failure discovery potential. An event-activity masking mechanism allows individual target ships to trigger waypoint-sampling events independently while remaining compatible with a joint policy action space.

  1. Reward-guided generation of plausible adversarial trajectories. An improved reward formulation is introduced to guide the target ships toward interception opportunities while discouraging abrupt trajectory changes. This formulation balances collision-chasing tendencies with realistic target ship behavior.

  1. Generalization across traffic configurations and systems under test. The policy is trained across a diverse collection of traffic configurations to improve its generalization capability. The modular co-simulation architecture further allows the trained stress-testing policy to be applied to different COLAV implementations by replacing the corresponding Functional Mock-up Unit (FMU), enabling comparative safety assessment without redesigning the complete testing framework.

Through these contributions, EB-ASTv2 aims to provide a reusable and scalable safety-assessment method for discovering safety-critical scenarios in partially observable maritime CPSs, while exploiting the modularity offered by co-simulation.

Preliminaries and Backgrounds

Co-simulation

Co-simulation technology is a simulation technique that is based on the construction of systems from loosely-coupled stand-alone models and the simulation across different subsystems. Co-simulation offers the ability for separate subsystems initialization, pre-processing, time-integration, and post-processing with different specialized tools [38]. Co-simulation also enables the use of black-box models [47], where the models is compiled into a low-level machine code that hides the internal algorithm from the users. These enables rapid and accurate model development that make use of already available models/software while at the same time protecting the Intellectual Property Right (IPR) associated to the subsystems.

Illustration of real maritime system modeled as co-simulation

Figure 1. Illustration of real maritime system modeled as co-simulation. It contains different tools and models, but they are interconnected with each other, run independently, and used in parallel to form a full-system simulation [38].

The subsystems in a co-simulation exchange information through their inputs and outputs. This exchange is based on the time-ordered values computed by the individual simulators during the simulation at discrete communication points ti∈{t0,t1,…,tN}t_i \in\{t_0,t_1,\ldots,t_N\} [43]. Information exchange between subsystems is performed only at these communication points and can be expressed through interface constraint equations [38]

u(ti)=Ly(ti),(1)\mathbf{u}(t_i) = \mathbf{L}\mathbf{y}(t_i), \tag*{(1)}

where u\mathbf{u} and y\mathbf{y} denote the subsystem input and output vectors, respectively, and L\mathbf{L} is a connection graph matrix that maps subsystem outputs to subsystem inputs. The communication points are separated by a macro step size Δti\Delta t_i, such that ti+1=ti+Δtit_{i+1}=t_i+\Delta t_i. In practice, the connection structure represented by L\mathbf{L} is specified through an I/O configuration, while an orchestrator algorithm is responsible for executing the co-simulation.

To enable interoperability among different subsystems and simulation tools, co-simulation relies on standardized interfaces or communication protocols. In this work we extensively used the Functional Mock-up Interface (FMI) standards, which has gained increasing attention in the engineering community in recent years [5]. FMI specifies that models should be packaged as FMUs, which are archive files that contain the model code along with its metadata and documentation. This standard is maintained and developed by the Modelica Association. Co-simulation has reached a decent maturity level and has been used across many industries, such as automotive industries [20, 28, 55], aviation industries [48, 56, 57], and recently maritime industries [6, 12, 27, 30, 33, 47].

Failures and Failure Analysis

Failure and failure mode are the two fundamental concepts in system reliability analysis. This section is based on Rausand [36] and summarizes the concepts relevant to failure analysis.

A failure occurs when an item is no longer able to perform one or more of its required functions. An item may be an individual component, a channel, a subsystem, or a safety loop. Hence, failure is the event that takes place when a required function is terminated. A failure may originate from an error, which defined as a “discrepancy between a computed, observed or measured value or condition and the true, specified or theoretically correct value or condition”.

Failure commonly followed by a failed state. The condition responsible for, or associated with, this state is referred to as a fault, where it is characterized by the inability of an item to perform a required function. A fault therefore is a state that may have its origin in a random failure event or in a deficiency related to the item, its location, or its application.

Finally, a failure mode is a description of a fault. It tells how a fault manifests itself. A failure mode lead to one or more consequences, referred to as failure effects. However, these effects can vary considerably in severity. Thus, the main motivation for identifying failure effects is to distinguish the more severe failure modes from less severe ones.

Error, failure, and fault

Figure 2. Error, failure, and fault [36].

Simulation-Based Safety-Assessment of Black-Box Cyber-Physical Systems

We first define a system model, denoted by M\mathcal{M}, together with a parameter space P\mathcal{P}, an input space I\mathcal{I}, and a property Ψ\Psi that the system is required to satisfy. The behavior of the model for a parameter value p∈Pp \in\mathcal{P} and an input i∈Ii \in\mathcal{I} is denoted by Φ(M,p,i)\Phi(\mathcal{M},p,i). In general, the input ii may be time-dependent. The set of all possible behaviors of M\mathcal{M} over the parameter space P\mathcal{P} and input space I\mathcal{I} is denoted by Φ(M,P,I)\Phi(\mathcal{M},\mathcal{P},\mathcal{I}) [16].

Testing is the task of evaluating whether the behaviors of the model M\mathcal{M}, for selected subsets P^⊆P\hat{\mathcal{P}} \subseteq\mathcal{P} and I^⊆I\hat{\mathcal{I}} \subseteq\mathcal{I}, satisfy the property Ψ\Psi [16]. Assuming that P^\hat{\mathcal{P}} and I^\hat{\mathcal{I}} are finite, testing can be expressed as:

Φ(M,P^,I^)⊨Ψ.(2)\Phi(\mathcal{M},\hat{\mathcal{P}},\hat{\mathcal{I}}) \models\Psi. \tag*{(2)}

Verification is the task of proving that all possible behaviors of the model M\mathcal{M} satisfy the property Ψ\Psi [16]. It therefore provides a formal guarantee of rightness or correctness over the specified parameter and input spaces [4, 40]. Verification can be expressed as:

Φ(M,P,I)⊨Ψ.(3)\Phi(\mathcal{M},\mathcal{P},\mathcal{I}) \models\Psi. \tag*{(3)}

In this research, we focus on CPSs, which are common in maritime systems because they integrate physical, digital, and networked components [53]. CPSs are difficult to verify exhaustively because they are complex, hybrid, and often black-box. A system is considered a black box when its internal model M\mathcal{M} is unknown or too complex to support explicit reasoning [7], or when it is deliberately concealed from external parties, for example to protect proprietary information.

Kapinski et al. [16] provide an overview on existing system verification method for CPSs, illustrated in Figure 3. The spectrum shows the trade-off between scalability and exhaustiveness of each method. Simulation-based testing is scalable but non-exhaustive. Formal verification is exhaustive but less scalable. In maritime autonomy, Torben et al. [53] show that simulation-based testing is attractive but requires automated test-case selection due to the large scenario space. Thus, Falsification is useful because it searches for counterexamples rather than proving universal correctness.

Spectrum of verification methods for CPSs

Figure 3. Spectrum of verification methods for CPSs [16].

After introducing the relevant definitions, we describe a method known as safety validation, introduced by Corso et al. [7]. Safety validation, as illustrated in Figure 4, is the process of ensuring the correct and safe operation of a system within its environment E\mathcal{E}. The desired safety property are expressed in a specification language, and simulation is typically used to identify violations of these properties, which are also referred to as failures. System failures may be caused by disturbances from the adversary X\mathcal{X}, and a model of these disturbances can be used to estimate the probability of failure. A system is deemed safe if no failure is found during a finite exploration or if the estimated failure probability remains below an acceptable threshold. Accordingly, safety validation encompasses three main objectives:

The safety validation problem

Figure 4. The safety validation problem [7].

  1. Falsification, which is the task of finding a parameter value and an input for which the resulting system behavior violates the property Ψ\Psi [16]. In other words, the objective is to find a counterexample (p,i)(p,i) such that:

p∈P,i∈I,Φ(M,p,i)⊭Ψ.(4)p \in\mathcal{P}, \quad i \in\mathcal{I}, \quad\Phi(\mathcal{M},p,i) \not\models\Psi. \tag*{(4)}

Equivalently, the set of all counterexamples may be written as:

{(p,i)∈P×I∣Φ(M,p,i)⊭Ψ}.(5)\{(p,i) \in\mathcal{P} \times\mathcal{I} \mid\Phi(\mathcal{M},p,i) \not\models\Psi\}. \tag*{(5)}
  1. Most-Likely Failure Analysis, which is the task to find the counterexample with maximum likelihood:

arg⁡max⁡p,i Pr⁡(p,i)such thatΦ(M,p,i)⊭Ψ.(6)\underset{p,i}{\arg\max}\ \Pr(p,i) \quad\text{such that} \quad\Phi(\mathcal{M},p,i) \not\models\Psi. \tag*{(6)}
  1. Failure Probability Estimation, which is the task to compute the probability that a safety property will be violated. Failure probability is given by the expectation of observing failure under the disturbance model

Pr⁡failure=E[1{Φ(M,p,i)⊭Ψ}].(7)\Pr_{\mathit{failure}} = \mathbb{E}\left[\mathbf{1}\{\Phi(\mathcal{M},p,i) \not\models\Psi\}\right]. \tag*{(7)}

Where,

1{Φ(M,p,i)⊭Ψ}={1,if Φ(M,p,i)⊭Ψ0,if Φ(M,p,i)⊨Ψ.(8)\mathbf{1}\{\Phi(\mathcal{M},p,i) \not\models\Psi\} = \begin{cases} 1, & \text{if } \Phi(\mathcal{M},p,i) \not\models\Psi\\ 0, & \text{if } \Phi(\mathcal{M},p,i) \models\Psi. \end{cases} \tag*{(8)}

In the verification literature, falsification is often discussed as part of simulation-based verification because it searches for counterexamples to formalized requirements. In the black-box safety-validation literature, the same activity is interpreted as one goal of safety validation, because the discovered counterexamples reveal unsafe behavior in an operational environment. In this work, falsification is therefore treated as a simulation-based safety-assessment activity: it does not provide a formal proof of correctness, but it supports a scalable verification via counterexamples and validation by identifying requirement-violating trajectories.

Reinforcement Learning Preliminaries

Section 2.4 provides a brief overview of Markov decision processes, largely following the presentation in Sutton and Barto [51].

Markov Decision Process

MDP is the classical of formalization of sequential decision making and serves as the foundation of many modern RL algorithms. MDP is intended as an intuitive framework of the problem of learning through interaction to achieve goal. The framework defines two entities: the “learner” which will be often refer to as the agent, and the environment which consists of everything outside the agent that it interacts with. Together, they create the agent-environment interface where both entities interact continually: The agent selects actions and the environment responds to it, transitioning it to a new situations or states to the agent. The environments then assigns a specific reward signals attribute to this transition.

MDP can be formally defined as a tuple ⟨S,A,T,R⟩\langle S,A,T,R\rangle. The agent and environment interact for every discrete time steps, t=[0,1,2,3,…]t = [0,1,2,3,\ldots]. At each time step tt, the agent receives some representation of the environment state, St∈SS_t \in S. Then based on the states, the agent choose an action, At∈A(St)A_t \in\mathcal{A}(S_t). On the next timestep, in part as a consequence of its action, the agent will find itself in the new state St+1S_{t+1} and receive the numerical reward, Rt+1∈R(St,At)⊂RR_{t+1} \in\mathcal{R}(S_t,A_t) \subset\mathbb{R}. The continuous process of this thereby give rise to a sequence of

S0,A0,R1,S1,A1,R2,S2,A2,R3,S3,….S_0,A_0,R_1,S_1,A_1,R_2,S_2,A_2,R_3,S_3,\ldots.

In a finite MDP, the sets of states, actions, and rewards S,A,R\mathcal{S},\mathcal{A},\mathcal{R} all have finite number elements. In this case, the next obtained state s′∈Ss' \in\mathcal{S} and reward r′∈Rr' \in\mathcal{R} when starting from a known current state s∈Ss \in\mathcal{S} while taking action a∈Aa \in\mathcal{A} is random and based on a probability distribution dependent only on the preceding state and action.

T(s′,r∣s,a):=Pr⁡{St=s′,Rt=r∣St−1=s,At−1=a},(9)T(s',r \mid s,a) := \Pr\{S_t=s',R_t=r \mid S_{t-1}=s,A_{t-1}=a\}, \tag*{(9)}

for all s′,s∈Ss',s \in\mathcal{S}; r∈Rr \in\mathcal{R}; a∈A(s)a \in\mathcal{A}(s) and the transition function TT defines the dynamics of the MDP. Due to this property, a state-action transition not just affect the immediate rewards of one transition but also the future rewards through the subsequent agent-environment interactions. This transition model also satisfies the Markov Property, meaning that, given the current state, future state transitions are independent of past agent-environment interactions. Thus, the state must contain all information from the interaction history that is relevant for predicting future dynamics.

Partially Observable Markov Decision Process

Many real-world problems do not satisfy the assumptions of the MDP framework because the learning agent can only partially observe the state of the environment at each time step [44]. In other words, some components of the state may be inherently hidden from the agent, while others may be observed inaccurately due to environmental noise. Such problem can be modeled as a POMDP [59], which is formally defined as a tuple

⟨S,A,T,R,Ω,O,bo⟩.(10)\langle\mathcal{S},\mathcal{A},T,\mathcal{R},\Omega,O,b_o\rangle. \tag*{(10)}

The ⟨S,A,T,R⟩\langle\mathcal{S},\mathcal{A},T,\mathcal{R}\rangle follows the MDP in Section 2.4.1. Ω\Omega is the observation space that describe the possible observations, OO is an observation model where

O(a,s′,o)=Pr⁡(ot+1∣at,st+1)(11)O(a,s',o)=\Pr(o_{t+1}\mid a_t,s_{t+1}) \tag*{(11)}

is the probability of observing oo given that the agent has reached state s′s' after executing action aa.

One option to address POMDP challenge is to use the agent history as its internal state. This history is composed of previous agent-environment interactions, denoted as

ht=⟨a0,o1,a1,o2,…,at−1,ot⟩.(12)h_t=\langle a_0,o_1,a_1,o_2,\ldots,a_{t-1},o_t\rangle. \tag*{(12)}

The complete history hth_t can be used as an internal Markov state, but it is infeasible for the reason hth_t will continually grows over time [44]. To tackle this problem, the concept of a belief state is introduced in the context of nondeterministic and partially observable planning problems [37]. Belief state is the set of actual states in which the agent may be in. In POMDP, the belief state bb is generalized into a probability distribution over the possible states. Following Shani et al. [44], the belief state is conditioned on the agent’s history, and can be expressed as

bo(s)=Pr⁡(s∣h),(13)b_o(s)=\Pr(s\mid h), \tag*{(13)}

where b(s)b(s) represents the probability that the agent is in state ss after observing history hh.

Rewards, Returns and Episodes

In RL, the reward function is formulated to capture the desired objective of the problem. Accordingly, the agent seeks to maximize the cumulative reward over the course of its interaction with the environment, rather than prioritizing immediate reward maximization. If the sequence of rewards for every time step tt is denoted as Rt+1,Rt+2,Rt+3,…R_{t+1},R_{t+2},R_{t+3},\ldots, we can then formalized the expected return GtG_t of that sequence. Formally, the expected return is what the agent aims to maximize. The simplest form of return can be written as

Gt:=Rt+1+Rt+2+Rt+3+…+RT,(14)G_t := R_{t+1}+R_{t+2}+R_{t+3}+\ldots+R_T, \tag*{(14)}

where TT is a final time step. This approach is sensible in the case when there is a natural notion of final time step, that is when the agent-environment interactions break naturally under a specific condition. This specific condition is generally called as terminal state. Furthermore, the overall sequence from the initial until terminal state is called episodes.

However, in many case the agent-environment interactions go infinitely rather than having a definitive end, which could cause the return to become unbounded. To address this, the discounting concept is introduced to discount the future rewards, emphasizing the importance of the immediate rewards while reducing the influence of the future rewards and preventing the return computation to explode. It is written as

Gt:=Rt+1+γRt+2+γ2Rt+3+…=∑k=0∞γkRt+k+1,(15)G_t := R_{t+1} + \gamma R_{t+2} + \gamma^{2}R_{t+3} + \ldots= \sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1}, \tag*{(15)}

where γ\gamma is called discount rate and bounded with 0≤γ≤10 \le\gamma\le1. As γ\gamma approaches 1, the greater the emphasis placed on future rewards, and vice versa, thereby defining the trade-offs between immediate rewards and long-term rewards.

Policies and Value Functions

A policy is formally defined as a mapping from states to probabilities of selecting each possible action. In a stochastic policy, the probability of selecting action aa in state ss is denoted by π(a∣s)\pi(a \mid s). Thus, when the agent follows policy π\pi, the action AtA_t at time tt is sampled from π(⋅∣St)\pi(\cdot\mid S_t).

The value function, vπ(s)v_\pi(s), is a function that estimate the value of the agent being in a state ss under policy π\pi. Similarly, action-value function, qπ(s,a)q_\pi(s, a), also estimate the value of the agent taking one action aa in a state ss while following policy π\pi. The value here is defined in terms of the expected returns GtG_t.

Solving RL problem means finding the policy that maximizes the returns collection over the episodes. In that context,there is always at least one policy that is better or equal to all other policies which can be measured by the value function associated to that policy. In essence, π≥π′\pi\ge\pi' if and only if vπ(s)≥vπ′(s)v_\pi(s) \ge v_{\pi'}(s) for all s∈Ss \in S. This policy is called the optimal policy, denotes as π∗\pi^*. They also share the same state-value function and action-value function, namely the optimal state-value function v∗(s)v^*(s) and optimal action-value function q∗(s,a)q^*(s, a) respectively. It can be defined as

v∗(s):=max⁡π(vπ(s)),for all s∈S,(16)v^*(s) := \max_{\pi}(v_\pi(s)), \qquad\text{for all }s \in S, \tag*{(16)}
q∗(s,a):=max⁡π(qπ(s,a)),for all s∈S and a∈A.(17)q^*(s,a) := \max_{\pi}(q_\pi(s,a)), \qquad\text{for all }s \in S\text{ and }a \in A. \tag*{(17)}

Policy Evaluation and Policy Improvement

Many RL algorithms can be interpreted as alternating, explicitly or implicitly, between two operations: policy evaluation and policy improvement. Policy evaluation estimates how good the current policy is, usually through a value function or action-value function. Policy improvement then modifies the policy toward actions that are expected to yield higher return. These two sub-processes alternate iteratively until they converge into the optimal policy. Each RL algorithm differs on how they evaluate and improve the policy, some may have an edge than the others depend on the circumstances.

Policy Gradient, Advantage, and Actor-Critic Idea

One could also consider to use a parameterized policy that can select actions without requiring a value functions. Let the policy parameter vector be denoted as θ∈Rn\theta\in\mathbb{R}^{n}. Using this parameter, we parameterized the policy into πθ\pi_\theta which can be written as

πθ(a∣s)=π(a∣s,θ)=Pr⁡{At=a∣St=s,θt=θ}.(18)\pi_\theta(a \mid s) = \pi(a \mid s,\theta) = \Pr\{A_t = a \mid S_t = s,\theta_t = \theta\}. \tag*{(18)}

The policy parameter vector can then be learned by optimizing a scalar performance measure J(πθ)J(\pi_\theta), which quantifies the expected performance of the policy. The value functions may be included in J(πθ)J(\pi_\theta) to aid for the policy parameter learning. Since the objective is to maximize J(πθ)J(\pi_\theta), the policy parameters can be updated using stochastic gradient ascent

θt+1=θ+α∇θJ(πθt),(19)\theta_{t+1} = \theta+ \alpha\nabla_\theta J(\pi_{\theta_t}), \tag*{(19)}

where α\alpha is the learning rate and ∇θJ(πθt)∈Rn\nabla_\theta J(\pi_{\theta_t}) \in\mathbb{R}^n is a stochastic estimate whose expectation approximates the gradient of the performance measure with respect to the policy parameters θt\theta_t. Methods that update the policy in this manner are known as policy gradient methods. Based on Marbach and Tsitsiklis in [25] and (Sutton et al.,) in [52], the gradient of the scalar function J(πθ)J(\pi_\theta) with respect to the policy parameters can be formalized as:

∇θJ(πθ)E(st,at)∼πθ[∇θπθ(at∣st)πθ(at∣st)qπ(st,at)].(20)\nabla_\theta J(\pi_\theta) \mathbb{E}_{(s_t,a_t)\sim\pi_\theta} \left[\frac{\nabla_\theta\pi_\theta(a_t \mid s_t)}{\pi_\theta(a_t \mid s_t)}q_\pi(s_t,a_t)\right]. \tag*{(20)}

This gradient increases the likelihood of high returns-action sampling, however it introduces high variances because the gradient is approximated using sampled estimates from trajectories collected by the policy. A common strategy as demonstrated in ([3], [10], [50]) can reduce variance by subtracting the state-value function from the action-value function, such that

Aπ(st,at):=qπ(st,at)−vπ(st).(21)A_\pi(s_t,a_t) := q_\pi(s_t,a_t) - v_\pi(s_t). \tag*{(21)}

This adjustment is referred to as the advantage function, and it describes how much better is action ata_t than the average action under the current policy at state sts_t. This adjustment does not bias the gradient as well because the value function is independent of the action.

The actor-critic architecture is a method that learns approximations to both of policy learning and value estimation. The actor is a policy network that proposes actions given a state, while the critic consists of value function networks that estimate the expected return of actions taken in given states. This separation allows the actor to be updated using gradients provided by the critic, leading to more stable learning than purely policy-gradient or value-based methods.

Long Short-Term Memory Neural Network

Long Short-Term Memory (LSTM) is an advanced variant of the Recurrent Neural Network (RNN) that is specifically tailored for learning long-term dependencies. LSTM was first introduced by Hochreiter and Schmidhuber [14], and now it is widely use and works remarkably well on many variety of problems.

LSTM addresses the limitations of standard RNN, where it often struggles to learn long-term dependencies due to the vanishing gradient and exploding gradient problems [41]. LSTM solves this by introducing a separate memory pathway known as the cell state. As shown in Figure 5, the cell state is represented by the horizontal connection running through the top of the structure. The flow of information into and out of the cell state is controlled by several gates, which are the forget gate, input gate, and output gate. These gates use sigmoid activation σ\sigma and tanh function σ\sigma to regulate how much information should be retained, added, or exposed. Through this mechanism, LSTM is better able to retain useful long-range information while filtering out irrelevant information.

Long Short-Term Memory Networks

Figure 5. Long Short-Term Memory Networks [29].

Reinforcement Learning Cores for Event-Based Adaptive Stress-Testing

Proximal Policy Optimization and Recurrent Extension

Proximal Policy Optimization (PPO) is a family of policy-gradient methods that optimizes a surrogate objective using stochastic gradient ascent. Introduced by Schulman et al. [42], PPO was designed to retain some of the data efficiency and reliable performance associated with Trust Region Policy Optimization (TRPO), while avoiding the complexity of second-order constrained optimization. This is achieved through a clipped probability-ratio objective, which discourages excessively large policy updates and improves training stability.

Let rt(θ)r_t(\theta) denotes the probability ratio between the updated policy and the policy used to collect the data

rt(θ)=πθ(at∣st)πθold(at∣st).(22)r_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)} . \tag*{(22)}

When θ=θold\theta= \theta_{\mathrm{old}}, the ratio satisfies rt(θold)=1r_t(\theta_{\mathrm{old}}) = 1. Values greater than one indicate that the updated policy assigns higher probability to action ata_t in state sts_t while values less than one indicate that the updated policy assigns lower probability to that action.

PPO avoids the explicit constrained Kullback–Leibler (KL) divergence optimization of TRPO by modifying the surrogate objective with a clipping operation

JCLIP(θ)=Et[min⁡(rt(θ)At,clip⁡(rt(θ),1−ϵ,1+ϵ)At)],(23)J^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t \left[\min\left(r_t(\theta)A_t,\operatorname{clip}\left(r_t(\theta),1-\epsilon,1+\epsilon\right)A_t\right)\right], \tag*{(23)}

where ϵ\epsilon is a hyperparameter that determines the clipping range. The clipping mechanism limits the incentive for the updated policy to move too far from the old policy by maintaining the balance between the probability ratio rt(θ)r_t(\theta) and the advantage value AtA_t while hard-constraining the objective with a predetermined threshold ϵ\epsilon. In this way, PPO approximates the stabilizing effect of TRPO algorithm while using a simpler first-order optimization procedure.

The objective in advantage function estimators can also use value function loss formulated using squared-error loss, denoted as JVFJ^{\mathrm{VF}}, making it follows the actor-critic formalism

JtVF(θ)=(vθ(st)−vttarget)2.(24)J_t^{\mathrm{VF}}(\theta) = \left(v_\theta(s_t)-v_t^{\mathrm{target}}\right)^2 . \tag*{(24)}

The objective can also be further improved by introducing entropy bonus, H(π)\mathcal{H}(\pi) maximization, which encourages exploration. Combining this terms, we can obtain a more robust PPO formulation

JCLIP+VF+H(θ)=Et[JtCLIP(θ)−k1JtVF(θ)+k2H(π(⋅∣st))],(25)J^{\mathrm{CLIP+VF+ \mathcal{H}}}(\theta) = \mathbb{E}_t \left[J_t^{\mathrm{CLIP}}(\theta)-k_1J_t^{\mathrm{VF}}(\theta)+k_2\mathcal{H}(\pi(\cdot\mid s_t))\right], \tag*{(25)}

where k1k_1 and k2k_2 control the strength of the value-function and entropy-bonus terms.

While the standard PPO is designed for fully observable MDPs, many real-life problems provide only for partial observation of the underlying state. Standard PPO generally employs feedforward Multi-Layer Perceptron (MLP) policy and value networks, therefore basing each decision on the current observation. R-PPO extends PPO by leveraging recurrent architecture into the actor and critic networks, enabling the agent to retains memory that can provide information on the history of the current task that is not present in the agent’s most recent observation [34]. In this research, LSTM networks are chosen to represent the policy πθ\pi_\theta and the value function vθv_\theta while preserving the same optimization objectives as standard PPO (Equation 25).

Soft Actor-Critic

Soft Actor-Critic (SAC) is introduced in [11] by Haarnoja et al. as an off-policy actor-critic algorithm based on the maximum entropy RL framework. For the standard RL algorithm, the goal is to learn the policy π\pi to maximizes the returns GtG_t. On the other hand, the maximum entropy objective generalizes the standard RL objectives by augmenting it with an entropy term, such that the optimal policy additionally maximize its entropy at each visited state:

π∗=arg⁡max⁡π∑tE(st,at)∼ρπ[r(st,at)+αH(π(⋅∣st))],(26)\pi^* = \arg\max_{\pi} \sum_t \mathbb{E}_{(s_t,a_t)\sim\rho_\pi}\left[r(s_t,a_t)+\alpha\mathcal{H}(\pi(\cdot\mid s_t))\right], \tag*{(26)}

where α\alpha is the temperature parameter that determines the strength of the entropy term H\mathcal{H}. SAC algorithm incorporates three key ingredients. First, an actor-critic architecture consisting of a policy network πθ(at∣st)\pi_{\theta}(a_t \mid s_t) (actor) and a value function network Qϕ(st,at)Q_{\phi}(s_t, a_t) (critic), parameterized by θ\theta and ϕ\phi, respectively. Second, an off-policy formulation that enables reuse of previously collected data through memory buffer for efficiency. Third, an entropy maximization process to encourage stability and exploration.

SAC is trained on the soft policy iteration, which includes soft policy evaluation and soft policy improvement. The main goal of this framework is to simultaneously maximize expected return and entropy to encourage exploration while still discouraging clearly bad actions. Intuitively, it is analogous to succeed at the task while acting randomly to the policy entropy. We encourage reader to visit Haarnoja et al. works [11], where they provide the details about the loss functions and derivations of the SAC algorithm.

Adaptive Stress-Testing

AST is a simulation-based falsification method that aims not only to find a scenario that results in a failure, but also to maximize the probability of the failure modes [23]. One most notable implementation of AST was carried by Lee et al. [22], where AST was introduced as a scalable and black-box compatible failure discovery method to stress-test a prototype air collision avoidance system by identifying the most likely trajectories that leads to mid-air collision.

Borrowing the terminologies from § 2.3, we define a simulator denoted as S\mathcal{S} that contains the system under test (or just system) M\mathcal{M} interacting with the environment E\mathcal{E}. The system with state μ∈M\mu\in\mathcal{M} takes a response ϱ∈P\varrho\in\mathcal{P} based on observation o∈Ωo \in\Omega of the environment state z∈Zz \in\mathcal{Z}. The simulator state s∈Ss \in\mathcal{S} is the stacked system and environment states [μ,z][\mu,z]. The system interacts with the environment over discrete timestep t∈[0,⋯ ,tend]t \in[0,\cdots,t_{\mathrm{end}}]. As conventions, we denote the variable at time tt using subscript tt and we also denote the sequence of variable over a range of time steps with colon range tstart:tendt_{\mathrm{start}}:t_{\mathrm{end}}. The state and action of the system depend on the environment observations, and are modeled as:

μt+1,ϱt=M(o0:t).(27)\mu_{t+1},\varrho_t=\mathcal{M}(o_{0:t}). \tag*{(27)}

The environment state and observation evolve over time depending on the actions of the system and disturbances x∈X⊆Rnx \in\mathcal{X} \subseteq\mathbb{R}^{n}, where nn is the dimensionality of the disturbances. The disturbances here can be depicted as any stochastic variables that can affect the environment, hence these terms are very broad and need to be specified. For example, the can be weather loads, sensor noises, or adversarial actors wandering inside the simulation (e.g. pedestrian, target ships, other vehicles)

zt+1,ot+1=E(o0:t,x0:t).(28)z_{t+1},o_{t+1}=\mathcal{E}(o_{0:t},x_{0:t}). \tag*{(28)}

The disturbances are assumed to be conditionally sampled from a probability density Pr⁡(x∣s)\Pr(x \mid s), which may be specified using expert knowledge or estimated from relevant data. Let Υ⊆S\Upsilon\subseteq\mathcal{S} denote the desired event space, and let ¬Υ⊆S\neg\Upsilon\subseteq\mathcal{S} denote its complement, corresponding to undesired events. There also exists states that cause the simulation to be terminated which is referred to as terminal state. In this work, we focus on failure, i.e., terminal states satisfying s∈Υs \in\Upsilon. They signify the end of one episode, and they can belong both in Υ\Upsilon (e.g. ship collision) and ¬Υ\neg\Upsilon (e.g. ship reaches end trajectory). The objective of AST is to find the most likely failure modes by maximizing the reward function. The objective can be formulated as:

max⁡x0:tend∏t=0tend−1Pr⁡(xt∣st)subject tostend∈Υ.(29)\max_{x_{0:t_{\mathrm{end}}}} \prod_{t=0}^{t_{\mathrm{end}}-1} \Pr(x_t \mid s_t) \quad\text{subject to} \quad s_{t_{\mathrm{end}}} \in\Upsilon. \tag*{(29)}

AST frames failure discovery as a reinforcement learning problem, shown in Figure 6, where an adversarial agent selects disturbances xtx_t that challenge the system under test M\mathcal{M}. Meanwhile the simulator evolves the system state and reports whether a failure has occurred. To avoid confusion, AST frames the disturbance sampling as the equivalent of action sampling on typical RL methods. Depending on the extent to which the simulator exposes its internal state, AST can be categorized into two types:

Adaptive Stress-Testing illustration

Figure 6. Adaptive Stress-Testing illustration.

Full Observability

If the simulator state is fully observable, we can formulate the AST problem as an MDP as demonstrated in § 2.4.1. In this case, the agent can selects a disturbance xtx_t from the policy π\pi given the true state sts_t,

xt∼π(⋅∣st),(30)x_t \sim\pi(\cdot\mid s_t), \tag*{(30)}

where the general reward function design by Lee et al. [23] is given by:

R(s,x)={RΥif s is terminal and if s∈Υ−dif s is terminal and if s∈¬Υlog⁡(Pr⁡(x∣s))otherwise.(31)\mathcal{R}(s,x)= \begin{cases} R_{\Upsilon} & \text{if }s\text{ is terminal and if }s\in\Upsilon\\ -d & \text{if }s\text{ is terminal and if }s\in\neg\Upsilon\\ \log\left(\Pr(x\mid s)\right) & \text{otherwise}. \end{cases} \tag*{(31)}

The first term of (31) provides a non-negative reward signal, RΥR_{\Upsilon}, when the trajectory terminates due to a failure. If the trajectory terminates without encountering a failure, the second term of (31) penalizes the agent by assigning a miss distance, dd, which measures how close the simulator came to a failure event. This term can significantly accelerate the search by enabling the agent to differentiate the relative desirability of non-failure trajectories [23].

The final term of (31) seeks to maximize the AST objective defined in (29) by rewarding the log-likelihood of each state transition. Recall that unlikely disturbances are associated with lower probabilities, whereas likely disturbances have probabilities closer to one. Consequently, the log-likelihood of an unlikely transition approaches −∞-\infty, while that of a highly likely transition approaches 00. This reward formulation therefore penalizes unlikely transitions while favoring more probable ones, thereby guiding the agent toward discovering failure trajectories that are as likely as possible.

Partial Observability

Not all simulator supports fully observable states. Some simulators are given as a software binary that maintained its internal dynamics hidden over function calls [23]. Example of this are co-simulation. Even though the underlying process can be modeled as an MDP, the agent only sees part of simulator states ss in form of observations oo.

ot∼O(⋅∣st),(32)o_t \sim\mathcal{O}(\cdot\mid s_t), \tag*{(32)}
xt∼π(⋅∣ot).(33)x_t \sim\pi(\cdot\mid o_t). \tag*{(33)}

This is a problem for typical MDP solvers because the same observation may correspond to multiple underlying hidden states. Consequently, a policy conditioned solely on the current observation oto_t may select suboptimal actions, as it lacks sufficient information to infer the true state of the environment. To address this issue, the partially observable AST problem is naturally formulated as a POMDP (see § 2.4.2), where sampled disturbances are conditioned on an observation history hth_t (see Equation (12)) rather than only the current observation:

xt∼π(⋅∣ht).(34)x_t \sim\pi(\cdot\mid h_t). \tag*{(34)}

Based on our literature review, we identified two main approaches for handling partial observability in AST:

  1. History-based tree search. This approach employs algorithms that maintain and revisit the action-observation history during planning, allowing the agent to reason over past information when selecting future disturbances. This direction was pioneered by Lee et al. [23] through the introduction of the Monte Carlo Tree Search-Seed Action (MCTS-SA) algorithm.

  1. Recurrent policy gradient methods. This approach augments policy gradient algorithms with RNN architectures, such as LSTM, instead of conventional MLPs. The recurrent layers enable the policy to learn temporal dependencies from the observation history, therefore forming an implicit approximation of the underlying belief state. One example is the work of Koren and Kochenderfer [17], who demonstrated the application of Deep Reinforcement Learning (DRL) algorithms with LSTM layers for partially observable AST.

As for the reward function designs, it mirrors the form and purpose those in Equation 31 [23]:

R(ρ,f,d,τ)={RΥif τ and f−dif τ and ¬flog⁡ρotherwise(35)\mathcal{R}(\rho, f, d, \tau) = \begin{cases} R_{\Upsilon} & \text{if } \tau\text{ and } f \\ -d & \text{if } \tau\text{ and } \neg f \\ \log\rho& \text{otherwise} \end{cases} \tag*{(35)}

where the simulator indirectly (due to agent’s limited access to the actual simulator states) provides the transition likelihood ρ=Pr⁡(x∣s)\rho= \Pr(x \mid s), a boolean indicating whether failure f=(s∈Υ)f = (s \in\Upsilon) has occurred, and the miss distance dd. Additionally, τ\tau represents a boolean from a function call that explains whether a terminal state has been reached based on the actual simulator states.

Event-Based Adaptive Stress-Testing

Event-Based Adaptive Stress-Testing (EB-AST) was first introduced as an extension of the conventional AST framework by revisiting its action sampling strategy, and was extensively evaluated for stress-testing maritime collision avoidance systems [45]. In conventional COLAV AST, the adversary is modeled as an RL agent that directly controls the target ship as a disturbance by sampling low-level control actions corresponding to its surge, sway, and yaw accelerations. These actions are updated at every fixed RL decision interval throughout the simulation.

One representative example is the work of Hjelmeland et al. [13], which demonstrated that this formulation is effective in discovering collision-inducing trajectories under several reward function designs. Despite its promising performance, this approach exhibits several limitations. Since the adversary updates its control actions at a higher frequency, the resulting trajectories may become overly discontinuous and operationally unrealistic. This effect is particularly evident in scenarios where the target ship continuously steers toward the own ship or performs excessively aggressive maneuvers during near-collision encounters.

EB-AST considers a typical ship-in-transit scenario in which the own ship executes its nominal voyage while an RL agent controls an adversarial target ship. The objective of the adversary is to interfere with the own ship’s mission and induce collision scenarios. Each of ship asset initially has their own baseline trajectory (which is a straight trajectory between start to end waypoint). The main takeaway is that the target ship is permitted to sample new waypoints in between the start and end of its trajectory to follow, mid simulation. These new waypoints is called Intermediate Waypoints (IWs), and an IW can be sampled only after the target ship enters the current IW’s Radius of Acceptance (RoA). Consequently, IW updates are triggered by navigation events rather than fixed time intervals, giving rise to the term event-based. This waypoint-generation mechanism, referred to as Intermediate Waypoint Sampling (IWS) (illustrated in Figure 7), and constitutes the key characteristics of the EB-AST framework. Additionally, IWS is constrained such that every newly sampled waypoint advances the vessel toward its destination, preventing unrealistic backward maneuvers or oscillatory behavior. The overall AST process is then constructed as MDP and then optimized using SAC algorithm. The objective of EB-AST is to discover critical waypoint sequences that induce collision while preserving realistic target-ship trajectories through the waypoint sampling constraints [45].

Intermediate waypoint sampling process in EB-AST

Figure 7. Intermediate waypoint sampling process in EB-AST.

The proposed EB-AST framework however was limited to identifying the near-collision events [45] and did not uncover actual collision events. The authors suggest that introducing multiple adversarial vessels may place the own ship’s COLAV system under greater operational stress, which might increasing the likelihood of genuine collision events.

Ship-in-Transit Co-simulation Setup

Ship Model

Figure 8 shows the interconnection structure of the FMUs used in the ship simulation framework. In this research, this structure is specifically used to model the own ship in order to include the system complexity for the stress testing process. Each block in the diagram is implemented as a separate FMU, and exchange data through their defined input and output variables. To orchestrate the communication between each block, we use a Python library called libcosimpy [49], which is a Python wrapper for co-simulation library in C++, libcosim [2], both are developed by Open Simulation Platform (OSP) [1].

FMUs connection diagram for the ship simulation

Figure 8. FMUs connection diagram for the ship simulation.

The ship model FMU represents the vessel hull dynamics. It receives the thrust force generated by the machinery-system FMU and the rudder force generated by the rudder FMU. Environmental disturbances are modeled as two FMU blocks: surface current and wind, which provides the speed and direction of the surface current and wind to ship model FMU.

The nominal actuator commands are generated by the autopilot and the speed-control subsystems and are subsequently processed by the COLAV FMU. When no collision threat are detected, the nominal throttle and rudder commands are passed directly to the machinery-system and rudder FMU, respectively. When an avoidance action is required, COLAV system modifies these command and output the corrected nominal commands.

The autopilot FMU generates the nominal rudder-angle command using Line-of-Sight (LOS) guidance scheme. The mission-manager FMU selects and supplies the active waypoint segment, consisting of the previous and next waypoints, according to the current vessel position received from the ship model FMU. Using this waypoints, the autopilot determines the desired heading and calculates the corresponding rudder command.

The forward speed control is handled by a cascaded structure consisting of the shaft-speed controller and throttle-controller FMU. The shaft-speed controller compares the actual vessel speed from the ship model FMU and the predefined speed set point from the mission manager FMU. It then generates the desired shaft-speed setpoint that will be used by the throttle-controller to compare with the measured shaft-speed retrieved from the machinery-system FMU. From this, it will output the nominal throttle command, closing the loop.

Simplified Ship Model

Figure 9 shows the interconnection structure of the FMUs used in the simplified ship simulation framework. Like the previous Ship Model structure in § 3.1, all blocks in the diagram are implemented as a separate FMUs, and are orchestrated using the same libcosimpy Python library. In this research, a simplified model is used for the target ships because their role as adversarial actors does not require the same level of system complexity unlike the own ship. Therefore, we stripped off the machinery-system FMU along with the cascading speed-control subsystem and replace it with a simple PI speed controller. Additionally, we also exclude the COLAV system FMU, which enables a direct thrust and rudder command input into the ship model FMU.

FMUs connection diagram for the simplified ship simulation

Figure 9. FMUs connection diagram for the simplified ship simulation.

Simple Collision-Avoidance System

The simple COLAV implementation follows a relatively simple decision strategy, as shown in Figure 10. When an avoidance maneuver is required, the system commands a half-throttle and a fixed-angle turn to starboard with the intention of passing the selected target ship on the own ship’s port side. This intention is arbitrary and serves as a demonstration only. The avoidance command will considered once the system assess own ship’s collision risk. At each assessment interval, it evaluates the collision risk associated with every target ship and selects the ship presenting the highest risk. The collision risk score computation is based on the Closest Point of Approach (CPA) metrics, and closing speed value.

Process flow diagram of the simple COLAV system to stress-test

Figure 10. Process flow diagram of the simple COLAV system to stress-test.

Let the relative north-east position vector and the relative north-east velocity between the own ship and the target ship be defined as rr and ν\nu respectively,

r=pOS−pTS,p:[N,E]T∈R2(36)r = p_{\mathrm{OS}} - p_{\mathrm{TS}}, \quad p : [N,E]^{\mathsf{T}} \in\mathbb{R}^{2} \tag*{(36)}
ν=νOS−νTS,ν:[u,v]T∈R2.(37)\nu= \nu_{\mathrm{OS}} - \nu_{\mathrm{TS}}, \quad\nu: [u,v]^{\mathsf{T}} \in\mathbb{R}^{2}. \tag*{(37)}

Given the own ship north-east position pOSp_{\mathrm{OS}} and heading ψOS\psi_{\mathrm{OS}}, we compute each target ship’s relative bearing to own ship β\beta

β=arctan⁡(EOS−ETS,NOS−NTS)−ψOS,(38)\beta= \arctan(E_{\mathrm{OS}} - E_{\mathrm{TS}},N_{\mathrm{OS}} - N_{\mathrm{TS}}) - \psi_{\mathrm{OS}}, \tag*{(38)}

and the instantaneous separation distance

D=∥r∥.(39)D = \lVert r \rVert. \tag*{(39)}

Differentiating DD with respect to time, yields the closing speed

D˙=rTν∥r∥.(40)\dot{D} = \frac{r^{\mathsf{T}}\nu}{\lVert r \rVert}. \tag*{(40)}

Closing speed explains whether two ships are closing in on each other when D˙\dot{D} is negative or moving apart when D˙\dot{D} is positive.

We also introduce the concept of the CPA, which is widely used in the definition and evaluation of collision-threat parameters and criteria [24]. The CPA denotes the predicted point at which the distance between the own ship and another object, (can be as a target ship or an obstacle), reaches its minimum value [8]. Two quantities associated with the CPA are the Time to Closest Point of Approach (tCPAt_{\mathrm{CPA}}) and Distance to Closest Point of Approach (DCPAD_{\mathrm{CPA}})

tCPA=−rTννTν,(41)t_{\mathrm{CPA}} = -\frac{r^{\mathsf{T}}\nu}{\nu^{\mathsf{T}}\nu}, \tag*{(41)}
DCPA=∥r+νtCPA∥.(42)D_{\mathrm{CPA}} = \lVert r + \nu t_{\mathrm{CPA}}\rVert. \tag*{(42)}

During the simulation, the own ship will continuously compute the tCPAt_{\mathrm{CPA}}, DCPAD_{\mathrm{CPA}}, and D˙\dot{D} of each target ship. Given the collision zone radius DcollisionD_{\mathrm{collision}} and the safe look-ahead time tlook−aheadt_{\mathrm{look-ahead}}, the target ship is considered as threat when its tCPA>0t_{\mathrm{CPA}} > 0, tCPA<tlook−aheadt_{\mathrm{CPA}} < t_{\mathrm{look-ahead}}, DCPA<DcollisionD_{\mathrm{CPA}} < D_{\mathrm{collision}} and D˙<0\dot{D} < 0. In the case of multiple target ships are considered as threats, the own ship’s COLAV system will compute The collision risk score Ξ\Xi and prioritize the target ships that has the highest risk score. Ξ\Xi is formulated as:

Ξ=λtCPAtCPA+λDCPADCPA−λD˙D˙(43)\Xi= \frac{\lambda_{t_{\mathrm{CPA}}}}{t_{\mathrm{CPA}}} + \frac{\lambda_{D_{\mathrm{CPA}}}}{D_{\mathrm{CPA}}} - \lambda_{\dot{D}}\dot{D} \tag*{(43)}

Where λtCPA\lambda_{t_{\mathrm{CPA}}}, λDCPA\lambda_{D_{\mathrm{CPA}}}, λD˙\lambda_{\dot{D}} are all coefficients which control the strength of each terms. The COLAV system can respond to only one target ship during each decision interval. After the interval has elapsed, the Ξ\Xi are reassessed and the target ship with the highest Ξ\Xi is selected.

Ship Traffic Generator

Ship Traffic Generator (STG) is a tool developed by Det Norske Veritas (DNV) to generate a structured set of ship encounter scenarios for verifying automated collision and grounding avoidance systems [31]. To generate an encounter scenario involving the own ship, the desired situation must first be specified. This desired situation consists of several user-defined parameters:

  1. Initial Position. The initial position of the own ship, given by latitude and longitude coordinates in decimal degrees.

  1. Speed over Ground (SoG). The speed of the own ship relative to the ground, given in knots.

  1. Course over Ground (CoG). The nominal mission course of the own ship, given in degrees using a north-referenced clockwise convention. For example, a northbound trajectory has a CoG of 0°.

  1. Initial Heading. The initial heading angle of the own ship, given in degrees.

  1. Navigational Status. The navigational status assigned to the simulated vessel.

  1. Vector Time. The time, given in minutes from the start of the simulation, at which the desired encounter situation is expected to occur within a specified range from the own ship.

  1. Relative Bearing β\beta. Which is the angle between the own ship heading and the target ship directions, given in degrees.

  1. Relative Speed. The speed of the target ship relative to the own ship, expressed as a fraction of the own ship speed. A value of 1 indicates that the target ship travels at the same speed as the own ship, while values below or above 1 indicate lower or higher relative target-ship speeds, respectively. The minimum value is 0.

The desired situation is typically specified in a JSON file, which serves as input to the STG. The corresponding output is also provided in JSON format and contains relevant vessel settings, such as waypoints and SoG, for all simulated vessels. These settings can subsequently be translated and implemented in the Ship-in-Transit co-simulation framework to generate the desired encounter scenarios. Examples of the generated traffic scenarios are shown in Figure 11.

Figure 11. Examples of the generated traffic scenarios by STG.

In this work, STG is used as an external scenario-generation tool to construct the initial ship encounter configurations used in the simulation study. The complete encounter generation algorithm is not explained here, since the present work does not modify the internal STG procedure. Instead, we summarize the input parameters relevant to our study and describe how the generated encounter cases are integrated into the proposed EB-AStV2 environment. A detailed description of the STG algorithm can be found in Pedersen et al. paper [32].

Case Study: Collision-Avoidance Stress-Testing in a Ship-in-Transit Co-simulation

This case study considers a ship-in-transit operation in which the functionality and failure behavior of a shipboard COLAV system are assessed through simulation-based stress-testing. We are aiming to develop a stress-testing method that automatically searches for scenarios that expose limitations in the COLAV system and lead to collision events. Identifying such scenarios can provide insight into the mechanisms underlying the failures and support subsequent improvements to the COLAV algorithm.

The simulated traffic scenario consists of three vessels. The first vessel is designated as the own ship and is equipped with the COLAV system under test. The remaining two vessels are designated as target ships and are distinguished by numerical identifiers. From the perspective of the own ship, these target ships represent surrounding traffic that must be monitored and avoided during the voyage.

Each vessel is initially assigned a nominal mission defined by two waypoints: a start waypoint wstartw_{\mathrm{start}} and an end waypoint wendw_{\mathrm{end}}, where all of them attempt to complete their respective missions using their onboard speed and heading controllers. For the own ship, these waypoints define a straight-line route that the vessel is required to follow. Conversely, for each IW-enabled target ship belongs to set It\mathcal{I}_t, let jj denotes the target ship identifier, T\mathcal{T} denotes the set of all feasible trajectories ζ\zeta as a finite waypoint sequence connecting wstartw_{\mathrm{start}} to wendw_{\mathrm{end}}.

ζj=(wstartj,w0j,w1j,…,wmj,wendj)∈(R2)m+3,wmj=(Nm,Em)(44)\zeta^{j} = \left(w_{\mathrm{start}}^{j}, w_{0}^{j}, w_{1}^{j}, \ldots, w_{m}^{j}, w_{\mathrm{end}}^{j}\right) \in\left(\mathbb{R}^{2}\right)^{m+3}, \quad w_{m}^{j} = (N_{m}, E_{m}) \tag*{(44)}
T≤nwjj=⋃m=0nwj{(w0j,w1j,…,wm+3j)∈(R2)m+3∣w0j=wstartj,wm+3j=wendj,ζj is feasible}.(45)\mathcal{T}_{\leq n_{w}^{j}}^{j} = \bigcup_{m=0}^{n_{w}^{j}} \left\{ \left(w_{0}^{j}, w_{1}^{j}, \ldots, w_{m+3}^{j}\right) \in\left(\mathbb{R}^{2}\right)^{m+3} \mid w_{0}^{j}=w_{\mathrm{start}}^{j}, w_{m+3}^{j}=w_{\mathrm{end}}^{j}, \zeta^{j}\ \text{is feasible} \right\}. \tag*{(45)}
T≤nwjj=T0j∪T1j∪⋯Tnwjj(46)\mathcal{T}_{\leq n_{w}^{j}}^{j} = \mathcal{T}_{0}^{j} \cup\mathcal{T}_{1}^{j} \cup\cdots\mathcal{T}_{n_{w}^{j}}^{j} \tag*{(46)}

where jj is the number of adversarial target ships up to nTSn_{TS}, (w0,w1,…,wm)(w_{0}, w_{1}, \ldots, w_{m}) are the IWs, and nwjn_{w}^{j} is the predetermined maximum number of IWs associated to each target ship jj. We also define the distance between two consecutive waypoints L(ζj)L(\zeta^{j}) as

L(ζj)=∑i=0m∥wi+1j−wij∥,(47)L(\zeta^{j}) = \sum_{i=0}^{m} \left\lVert w_{i+1}^{j} - w_{i}^{j} \right\rVert, \tag*{(47)}

where among the trajectories in T≤nwj\mathcal{T}^{j}_{\le n_w}, there exists ζmin⁡,j\zeta^{\min,j} that has the minimum path length.

ζmin⁡,j=arg min⁡ζj∈T≤nwj L(ζj).(48)\zeta^{\min,j} = \underset{\zeta^j \in\mathcal{T}^{j}_{\le n_w}}{\operatorname{arg\,min}}\ L(\zeta^j). \tag*{(48)}

In the absence of obstacles or additional constraints, ζ0min⁡∗,j∈T0j\zeta^{\min^*,j}_0 \in\mathcal{T}^{j}_0 is the straight-line route connecting wstartjw^j_{\mathrm{start}} and wendjw^j_{\mathrm{end}}.

The nominal missions are generated using the STG. Depending on the specified encounter type, the straight-line route ζmin⁡∗,j\zeta^{\min^*,j} may intersect or overlap in a manner that produces head-on, crossing, or overtaking encounters between the own ship and the target ships. The complete set of initial waypoints and vessel states used to initialize an episode is referred to as a spawn request or spawn case.

A scenario CC is then defined as the combination of the own ship trajectory ζ0∈T0\zeta^0 \in\mathcal{T}^0 and the target ships’ trajectories within a single simulation:

C=(ζ0,ζ1,ζ2,…,ζnTS).(49)C = \left(\zeta^0,\zeta^1,\zeta^2,\ldots,\zeta^{n_{\mathrm{TS}}}\right). \tag*{(49)}

Accordingly, the overall scenario space is given by

C=T0×∏j=1nTST≤nwj.(50)\mathcal{C} = \mathcal{T}^0 \times\prod_{j=1}^{n_{\mathrm{TS}}} \mathcal{T}^{j}_{\le n_w}. \tag*{(50)}

where the Cartesian product indicates that one trajectory is selected for each target ship.

During a nominal encounter, the own ship relies on its onboard COLAV system to avoid the target ships. However, the simplified COLAV system used in this stress-testing experiment has a known limitation: it cannot simultaneously account for all target ships when multiple collision threats are present (see § 3.3). Instead, collision-avoidance actions are determined according to a sequential prioritization scheme. This behavior introduces a potential vulnerability in encounters involving multiple simultaneous threats.

The objective of the stress-testing method then is to identify the subset of scenarios C∗⊆CC^* \subseteq C that exposes limitations of the onboard COLAV system and results in a collision between the own ship and at least one target ship. This subset is defined as

C∗={C∈C∣∃j∈{1,…,nTS}:ζj∈T≤nwj∗}.(51)C^* = \left\{ C \in C \mid\exists j \in\{1,\ldots,n_{\mathrm{TS}}\} : \zeta^j \in\mathcal{T}^{j^*}_{\le n_w} \right\}. \tag*{(51)}

Here, T≤nwj∗\mathcal{T}^{j^*}_{\le n_w} is defined as the set of collision-inducing trajectories for each target ship

T≤nwj∗={ζj∈T≤nwj∣Υ(ζ0,ζj)},(52)\mathcal{T}^{j^*}_{\le n_w} = \left\{ \zeta^j \in\mathcal{T}^{j}_{\le n_w} \mid\Upsilon(\zeta^0,\zeta^j) \right\}, \tag*{(52)}

where Υ(ζ0,ζj)\Upsilon(\zeta^0,\zeta^j) indicates that the trajectory ζj\zeta^j results in a collision involving target ship jj and the own ship. Thus, a scenario belongs to C∗C^* if at least one of the target-ship trajectory is collision-inducing; the trajectories of the remaining target ships are not required to result in collisions. To identify such scenarios, the stress-testing method searches for waypoint sequences between wstartw_{\mathrm{start}} and wendw_{\mathrm{end}} along each target ship’s nominal route.

In addition to identifying failures for individual initial conditions, the stress-testing method is designed to produce a scenario generator that can adapt to a diverse range of spawn cases. To achieve this, the stress-testing process is formulated as a RL problem, in which the agent’s policy serves as the scenario generator. A finite collection of traffic configurations is generated using the STG, stored in a spawn-request bank, and sampled during training. These configurations expose the agent to variations in vessel positions, headings, speeds, and encounter geometries, encouraging the policy to learn a generalizable strategy that can operate across multiple traffic configurations rather than overfitting to a single scenario.

Additional experiments are conducted to investigate whether further improvements can be achieved by continuing the training of an existing policy. The trained policy is subsequently used to stress-test different COLAV implementations, allowing us to assess how effectively it can continue to generate collision-inducing scenarios. The proposed solution to this study case is our EB-ASTv2 algorithm, which is introduced in Section 5.

ebast2: An Enhanced Event-Based Adaptive Stress-Testing Framework for Co-Simulation

Section 5 presents the EB-ASTv2 algorithm, an improved iteration of the original EB-AST method [45]. The section describes the proposed algorithm and discusses its key methodological improvements in detail.

Proposed Algorithm and Improvements

We proposed a novel safety-assessment method specifically tailored for black-box CPSs, where the framework diagram of this method is shown in Figure 12. In this paper, we focused on using a maritime Ship-in-Transit co-simulation as a demonstrator. We employ R-PPO algorithm as the primary solver algorithm because of its compatibility with POMDPs, which are often encountered in black-box simulation like co-simulation. The framework consists of two process blocks that interact with each other iteratively that forms an EB-ASTv2 update cycle process: the RL environment wrapper and the DRL agent.

Process diagram of EB-ASTv2 method

Figure 12. Process diagram of EB-ASTv2 method.

RL environment wrapper is implemented as a Python wrapper that is built around the Ship-in-Transit co-simulation. The wrapper manages the simulation step function calling, reward signals computation, and relevant RL states and termination states compilation for the DRL agent. At each cycle, the wrapper accepts actions (or disturbances seen by the own ship) from the EB-ASTv2 policy and forwards it to the corresponding module that handles IWS mechanism for the target ships. The IWS mechanism is implemented within the Mission Manager FMU of each adversarial target ship, which modifies the ship’s trajectory by sampling IWs to stress-test the own ship’s COLAV system. Once the state-action transition is complete, the wrapper returns observation and its corresponding reward to the DRL agent.

The DRL agent contains the solver which takes the tuples of observation from the state-action transition along with its associated rewards and process it to improve the EB-ASTv2’s policy. In this research, we employs the R-PPO algorithm provided by stable-baselines3, an open-source library which supports variety of RL algorithms [35]. This library offers reliable and easy-to-use RL implementations, well suited for research and RL-related development. Following the R-PPO algorithm, the DRL agent first collects rollout trajectories through repeated interactions with the simulation environment. These rollouts are then used to iteratively train the LSTM-based policy and value networks through the R-PPO policy evaluation and policy improvement.

We specifically adopt the recurrent policy gradient methods like R-PPO rather than the history-based tree search because of the need to learn a reusable policy for a partially observable and continuous scenario space. While history-replay mechanism is effective for online search, it does not directly learn a reusable policy that generalizes across similar histories or scenario instances. In practice, DRL agent is not strictly tied to R-PPO. It can hosts variety of RL algorithms as long as it is compatible with POMDP.

EB-ASTv2 introduces several improvements over its predecessor, EB-AST. In addition to its compatibility with POMDP settings, EB-ASTv2 supports multi-adversary control, unlike EB-AST single-adversary control limit. This allows the EB-ASTv2 agent to coordinate multiple target ships and generate more challenging encounter scenarios that may expose weaknesses in the own ship’s COLAV system. Furthermore, the IWS mechanism is enhanced by introducing a variable IW sampling scope-length as a second action component. The variable IW sampling scope-length allows the agent to adapt the scale of its intervention. Shorter scope-lengths enable local adjustments to the target ship trajectory, while longer scope-lengths allow larger deviations over a wider portion of the encounter scenario. This additional degree of freedom increases the flexibility of IW sampling and enables a more precise exploration of the scenario space.

Gymnasium Environment for Reinforcement Learning

The stable-baselines3 library expected the users to prepare their RL environment based on Gymnasium API. Gymnasium API is a common interface for RL environments, defining how and agent interacts with an standardized function calls that represents MDP (or POMDP) [54]. These main function calls are reset(), step(), and isTerminal().

Note that although Ship-in-Transit co-simulation are wrapped within the RL environment, both entities operate different interpretations of a time-step. In the co-simulation, time-step refers to the time interval between consecutive numerical integration steps (e.g. Euler integration). In contrasts, a time-step in the RL environment represents the interval between two consecutive events. Consequently, the number of co-simulation integration steps between two RL time steps may vary, as different events may occur after different amounts of simulated time.

In this work, we define events as the occurrence of one of the following conditions:

  • Event 1: the co-simulation reaches the time threshold at which IWS is enabled;

  • Event 2: a target ship enters the RoA of its next waypoint;

  • Event 3: the co-simulation reaches a terminal state, either desired or undesired.

The desired terminal states, denoted by Γ\Gamma, corresponds to outcomes of interest for the stress-testing process. These includes:

  • own ship collision between one ore more target ships,

  • own ship prolonged deviation from its intended trajectories beyond a predefined time limit. The deviation boundary is based on the maximum allowable cross-track error of the LOS guidance system.

On the contrary, undesired terminal states are denoted as ¬Γ\neg\Gamma, correspond to terminal conditions that do not provide relevant information for the stress-testing objective. These includes:

  • own ship reaches its mission endpoint,

  • collision between target ships.

The reset() function are typically called when the environment is initialized. Upon calling this function, the Ship-in-Transit co-simulation is advanced until Event 1 is encountered for the first time. At this point, the agent receives the initial observation along with additional information regarding the current environment state.

The step() function executes the selected action from the agent’s policy and updates the environment accordingly. An action is requested when either Event 1 or Event 2 is triggered.

Subsequently, the co-simulation advances until another Event 2 occurs, or until Event 3 is reached. As a result, the agent receives a new observation along with a reward for taking the action.

The isTerminal() function returns two Boolean values, τ\tau and ff. The value τ\tau is TRUE if the current simulator state ss is terminal, and FALSE otherwise. If ss is terminal, ff indicates whether the terminal state is desired. ff = TRUE if s∈Υs \in\Upsilon, and ff = FALSE if s∉Υs \notin\Upsilon.

Additionally, the RL environment also defines the structures such as the observation space and action space, allowing RL libraries like stable-baselines3 to work across different environments through the same interface. This will be further explained in the sections below.

Observation Space

The observation space Ω\Omega consists of the simulation states that are observable to the agent, as not all states of the Ship-in-Transit co-simulation are exposed to the agent. Throughout this paper, we are going to refer to the “own ship” and the “target ship” as “OS” and “TS” respectively. Given the number of adversarial target ships nTSn_{\mathrm{TS}} and j=1,⋯ ,nTSj = 1, \cdots, n_{\mathrm{TS}}, this space consists of:

  • own ship COLAV throttle command: %tCOLAV\%^{\mathrm{COLAV}}_t,

  • own ship throttle command: %t\%_t,

  • own ship COLAV rudder angle command: ψrudder,tCOLAV\psi^{\mathrm{COLAV}}_{\mathrm{rudder},t},

  • own ship rudder angle command: ψrudder,t\psi_{\mathrm{rudder},t},

  • own ship positions: NOS,t,EOS,t,ψOS,tN_{\mathrm{OS},t}, E_{\mathrm{OS},t}, \psi_{\mathrm{OS},t},

  • own ship forward speed: uOS,tu_{\mathrm{OS},t},

  • own ship cross-track error: ect,OS,te_{\mathrm{ct,OS},t},

  • target ships relative positions: NTS/OS1,ETS/OS1,ψTS/OS1,⋯ ,NTS/OSj,ETS/OSj,ψTS/OSjN^{1}_{\mathrm{TS/OS}}, E^{1}_{\mathrm{TS/OS}}, \psi^{1}_{\mathrm{TS/OS}}, \cdots, N^{j}_{\mathrm{TS/OS}}, E^{j}_{\mathrm{TS/OS}}, \psi^{j}_{\mathrm{TS/OS}},

  • target ships forward speed: u1,⋯ ,uju^{1}, \cdots, u^{j},

  • actions masks: mt1,⋯ ,mtjm^{1}_{t}, \cdots, m^{j}_{t}.

The term relative position refers to the difference between the position of a target ship and that of the own ship. Representing the target-ship positions relative to the own ship helps provide a more consistent observation representation across different encounter cases.

In addition, the action masks mm are key observations that enable multi-adversary control. Each action mask takes a binary value of either 0 or 1, indicating whether the corresponding target ship can currently be acted upon by the agent. The action-masking mechanism is further explained in § 5.3.1.

Action Space

The action space A\mathcal{A} comprises of the available actions for the agent to perform. In the AST context, action here is also referred to as disturbance These actions are the inputs for IWS mechanism, which are responsible for building the collision-inducing trajectories for the target ships. This space consists of:

  • scope-angle: ψtsc,1,⋯ ,ψtsc,j\psi^{\mathrm{sc},1}_{t}, \cdots, \psi^{\mathrm{sc},j}_{t},

  • scope-length: ltsc,1,⋯ ,ltsc,jl^{\mathrm{sc},1}_{t}, \cdots, l^{\mathrm{sc},j}_{t}.

where a sampled action from the policy π\pi is expressed as

at=[ψtsc,1,ltsc,1,ψtsc,2,ltsc,2,⋯ ,ψtsc,j,ltsc,j].(53)a_t = \left[\psi^{\mathrm{sc},1}_{t}, l^{\mathrm{sc},1}_{t}, \psi^{\mathrm{sc},2}_{t}, l^{\mathrm{sc},2}_{t}, \cdots, \psi^{\mathrm{sc},j}_{t}, l^{\mathrm{sc},j}_{t}\right]. \tag*{(53)}

Further details on how these actions are used are shown in § 5.3.

Intermediate Waypoint Sampling

IWS was first developed as a novel action sampling strategy that enables the discovery of realistic collision-inducing trajectories for EB-AST framework through a mechanism referred to as scoping. Conceptually, the method can be compared to a marksman try to pinpoint a target by adjusting its rifle angle and estimating the target’s distance. In this way, IWS provides lower-resolution control over the adversarial target ship’s trajectory, allowing a broader search space to be explored while reducing the agent’s aggressiveness and tendency to deliberately pursue collisions.

We first define two helper functions that will be used extensively for IWS. The first one is getHypot(A, B), used to compute the hypotenuse between point AA and BB, given by

lAB=getHypot⁡(A,B)=(NA−NB)2+(EA−EB)2.(54)l_{AB} = \operatorname{getHypot}(A,B) = \sqrt{(N_A-N_B)^2+(E_A-E_B)^2}. \tag*{(54)}

The second one is getBeta(A, B), used to compute the angle β\beta between the north-axis and the line segment formed by connecting AA and BB. This function is formulated as

βAB=getBeta⁡(A,B)=arctan⁡(EA−EBNA−NB).(55)\beta_{AB} = \operatorname{getBeta}(A,B) = \arctan\left(\frac{E_A-E_B}{N_A-N_B}\right). \tag*{(55)}

Each target ship is initialized with a waypoint sequence ζ\zeta, containing the initial waypoints wstartw_{\mathrm{start}} and wendw_{\mathrm{end}}. From this sequence, a waypoint pair W=(wbase,whead)W=(w_{\mathrm{base}},w_{\mathrm{head}}) is defined and used by the LOS guidance system to provide a heading reference to the target ship’s autopilot. The pair is retrieved using the function getWaypointPairs(i, ζ), such that

W=(ζ(i−1),ζ(i)),(56)W=(\zeta(i-1),\zeta(i)), \tag*{(56)}

where i=1,2,…,ni=1,2,\ldots,n denotes the index of the current waypoint in ζ\zeta. As the agent encountered a new event, the advanceWaypointPairs(i) is called to increment ii by 1 to advance the waypoint pairing. To illustrate how the waypoint pair WW relates to the current state of the trajectory, consider the initial waypoint sequence before any IWs are introduced.

ζ=(wstart,wend),i=1.(57)\zeta=(w_{\mathrm{start}},w_{\mathrm{end}}),\quad i=1. \tag*{(57)}

wbasew_{\mathrm{base}}          wheadw_{\mathrm{head}}

The IWS starts immediately after Event 1 is triggered (see § 5.2), which automatically calls initiateIWS() function. This function insert the zeroeth IW w0w_0 into ζ\zeta based on the target ship’s north-east position when Event 1 is first triggered. Hence the current ζ\zeta becomes

ζ=(wstart,w0,wend),i=1.(58)\zeta=(w_{\mathrm{start}},w_0,w_{\mathrm{end}}),\quad i=1. \tag*{(58)}

wbasew_{\mathrm{base}}          wheadw_{\mathrm{head}}

The zeroeth IW will then be used as the scoping base to compute the next IW using getNextWaypoint(i, ζ, ψ_t^{\mathrm{sc}}, β). The algorithm goes as follow:

  1. Retrieve the-scope length ltscl_t^{\mathrm{sc}} and scope-angle ψtsc\psi_t^{\mathrm{sc}} from the policy, where the scope-angle is constrained so that ψtsc≤∣ψtmax⁡∣\psi_t^{\mathrm{sc}}\leq|\psi_t^{\max}|.

  2. Evaluate the sampled scope-length using evaluateScopeLength().

  3. Retrieve the current waypoint pair WiW_i using getWaypointPairs(i, ζ).

  4. Compute the lWil_{W_i} and βWi\beta_{W_i} using getHypot(W_i), getBeta(W_i).

  5. Set the whead,iw_{\mathrm{head},i} coordinates as the base point for the next waypoint whead,i+1w_{\mathrm{head},i+1}.

  6. The next waypoint wi+1=(Ni+1,Ei+1)w_{i+1}=(N_{i+1},E_{i+1}) can then be computed by solving

Nhead,i+1=Nhead,i+lWicos⁡(βWi+ψtsc),Ehead,i+1=Ehead,i+lWisin⁡(βWi+ψtsc).(59)\begin{aligned} N_{\mathrm{head},i+1} &= N_{\mathrm{head},i}+l_{W_i}\cos(\beta_{W_i}+\psi_t^{\mathrm{sc}}),\\ E_{\mathrm{head},i+1} &= E_{\mathrm{head},i}+l_{W_i}\sin(\beta_{W_i}+\psi_t^{\mathrm{sc}}). \tag*{(59)} \end{aligned}
  1. Insert wi+1w_{i+1} into ζ\zeta at index i+1i+1 using insertNewWaypoint(i, ζ) function.

  2. Finally, advance the waypoint pairs using advanceWaypointPairs(i).

Having called this function, the condition of the trajectory ζ\zeta should follows

ζ=(wstart,w0wbase,w1whead,wend),i=2.(60)\zeta= (w_{\mathrm{start}}, \underset{w_{\mathrm{base}}}{w_0}, \underset{w_{\mathrm{head}}}{w_1}, w_{\mathrm{end}}), \quad i = 2. \tag*{(60)}

Furthermore, the getNextWaypoint() function can be called repeatedly as long as Event 2 continues to be triggered by the target ships. This produces the updated trajectory condition defined in Equation (61). In addition, the overall process is illustrated in Figure 13 and Figure 14.

w0 placement and computing w1

Figure 13. w0w_0 placement & computing w1w_1.

Computing w2

Figure 14. Computing w2w_2.

ζ=(wstart,w0wbase,w1,w2whead,wend),i=3(61)\zeta= (w_{\mathrm{start}}, \underset{w_{\mathrm{base}}}{w_0}, w_1, \underset{w_{\mathrm{head}}}{w_2}, w_{\mathrm{end}}), \quad i = 3 \tag*{(61)}

Typically, the IWS mechanism automatically stop once the action-sampling counts reach the predetermined limit of nwn_w. However, there are special cases in which IWS is forced to stop even before the former condition is not fulfilled. When the sampled scope-length ltscl_t^{\mathrm{sc}} is greater than the length of the remaining untraversed segment-length liutl_i^{\mathrm{ut}}, the next waypoint will not be computed and the last sampled IW will automatically connect to the end waypoint wendw_{\mathrm{end}}. Thus, the updated trajectory condition becomes

ζ=(wstart,w0,w1,w2,wend),i=4(62)\zeta= (w_{\mathrm{start}}, w_0, w_1, w_2, w_{\mathrm{end}}), \quad i = 4 \tag*{(62)}

This case takes place only when the evaluateScopeLength() function returns TRUE, as illustrated in Figure 15. The maximum scope angle, ψmax⁡\psi_{\max}, together with the scope-length evaluation, enables the generation of trajectories that remain flexible while maintaining forward progression, therefore supporting the objectives of EB-ASTv2.

Invalid IWS for w3 and closing the trajectory

Figure 15. Invalid IWS for w3w_3. Close the trajectory.

Multi-Adversary Control through Action Masking

Action masking process

Figure 16. Action masking process. Value of 1 is given to the Target Ship(s) that is allowed to sample new IW.

Due to the structure of the EB-ASTv2 action space A\mathcal{A}, the policy cannot sample an action exclusively for a particular IW-enabled target ship. Instead, an action is sampled simultaneously for all target ships represented in the action space. This design choice is intended because two or more target ships may trigger events at the same time, hence a ready supply of action is required at all decision events. In contrast, the multi-adversary control scheme allows each target ship to trigger a decision event independently of the others. Consequently, an event triggered by one target ship also causes the policy to produce actions for target ships that did not trigger an event. Although these additional actions do not result in trajectory modifications, they are still produced by the policy and may therefore introduce irrelevant action–observation associations during training.

Zouitine in his article [58] discusses three approaches for incorporating action masking: action-level masking, feature-level masking and agent-level masking. Action-level masking works by altering the value associated to the impossible action to −∞-\infty. Feature-level masking controls which information is available to an agent when constructing its internal state representation. Agent-level masking determines which agents contribute to the policy-loss computation in a multi-agent setting.

To address the asynchronous event triggering in EB-ASTv2, we adopt a mechanism conceptually closer to feature-level masking. Rather than suppressing individual action values or dimensions, an additional observation vector is introduced to indicate which target ships triggered the current decision event. Let

mt=[mt1,mt2,…,mt∣It∣],mtj∈{0,1},(63)\mathbf{m}_t = \left[m_t^1,m_t^2,\ldots,m_t^{\lvert I_t\rvert}\right], \qquad m_t^j \in\{0,1\}, \tag*{(63)}

denote this event-activity mask, where each entry corresponds to one IW-enabled target ship jj. An entry is set to mtj=1m_t^j = 1 when the corresponding target ship triggers an event and to mtj=0m_t^j = 0 otherwise. Since target ships trigger events independently, multiple entries may simultaneously take the value 1 at a given decision event.

The mask mt\mathbf{m}_t is included in the observation supplied to the policy. It provides an explicit indication of which target-ship-specific action components are active at the current decision event. Through training, the policy can associate entries with mtj=1m_t^j = 1 with actions that affect the corresponding target ship, while actions associated with mtj=0m_t^j = 0 are inactive and do not modify that ship’s trajectory.

Reward Function

The reward function provides the learning signal used to guide the EB-ASTv2 policy toward failure-inducing scenarios. Its design follows the general AST formulation introduced in Equation (35), while incorporating additional shaping terms that are specific to the IWS mechanism.

At each decision event, the reward is determined by whether the episode has terminated and whether the termination corresponds to a desired failure event. Let τt∈{0,1}\tau_t \in\{0,1\} indicate episode termination and ft∈{0,1}f_t \in\{0,1\} indicate whether the desired failure event has occurred. The reward is then expressed as

Rt={RΥ,τt∧ft,R¬Υ(df),τt∧¬ft,RNT,t,¬τt,(64)R_t = \begin{cases} R_{\Upsilon}, & \tau_t \land f_t, \\ R_{\neg\Upsilon}(d_f), & \tau_t \land\neg f_t, \\ R_{\mathrm{NT},t}, & \neg\tau_t, \end{cases} \tag*{(64)}

where

R¬Υ(df)=1∣It∣∑j∈ItR¬Υj(dfj)(65)R_{\neg\Upsilon}(d_f) = \frac{1}{|\mathcal{I}_t|}\sum_{j \in\mathcal{I}_t} R_{\neg\Upsilon}^{j}(d_f^{j}) \tag*{(65)}
RNT,t=1∣It∣∑j∈ItRNT,tj(dfj,dn,tj,nwj,cw,tj,Δψtsc,j,Δψic,tj)(66)R_{\mathrm{NT},t} = \frac{1}{|\mathcal{I}_t|}\sum_{j \in\mathcal{I}_t} R_{\mathrm{NT},t}^{j}\left(d_f^{j},d_{n,t}^{j},n_{\mathrm{w}}^{j},c_{\mathrm{w},t}^{j},\Delta\psi_{t}^{\mathrm{sc},j},\Delta\psi_{\mathrm{ic},t}^{j}\right) \tag*{(66)}

RΥR_{\Upsilon} denotes the reward for a desired termination, R¬ΥR_{\neg\Upsilon} denotes the reward for an undesired termination, and RNT,tR_{\mathrm{NT},t} denotes the non-terminal reward. Here, “desired” refers to the objective of the stress-testing agent; from the perspective of the system under test, such an event represents a failure.

Moreover, dfd_f denotes the final own-ship–target-ship separation at an undesired terminal event, while dn,td_{n,t} denotes the minimum separation observed up to decision event tt. The variable nwn_{\mathrm{w}} specifies the maximum number of IWs that may be sampled for a target ship, and cw,t∈{0,…,nw}c_{\mathrm{w},t} \in\{0,\ldots,n_{\mathrm{w}}\} denotes the number already sampled. The counter is initialized to zero at the beginning of each episode and incremented after every successful IW request. Furthermore, Δψtsc\Delta\psi_t^{\mathrm{sc}} denotes the change between consecutive scope-angle commands, whereas Δψic,t\Delta\psi_{\mathrm{ic},t} denotes the angular difference between the sampled scope angle and the interception scope angle.

Note that both R¬ΥR_{\neg\Upsilon} and RNT,tR_{\mathrm{NT},t} are obtained by averaging of reward contributions from the IW-enabled target ships contained in the set It\mathcal{I}_t, hence the superscript jj. Combined with the action masking protocol, this formulation enables the policy to control multiple adversarial target ships simultaneously. For brevity, the superscript jj is omitted from the remaining equations in this section unless otherwise stated.

For a desired termination, the terminal reward is set to

RΥ=0.(67)R_{\Upsilon} = 0. \tag*{(67)}

If the episode terminates without reaching the desired failure event, the agent instead receives a distance-based penalty

R¬Υ(df)=CρFρ(df),(68)R_{\neg\Upsilon}(d_f) = C_{\rho}F_{\rho}(d_f), \tag*{(68)}

where CρC_{\rho} is a scaling coefficient. The distance transformation is defined as

Fρ(D)=−log⁡(1+(D−Dcollision)),(69)F_{\rho}(D) = -\log\left(1+\left(D-D_{\mathrm{collision}}\right)\right), \tag*{(69)}

where DcollisionD_{\mathrm{collision}} denotes the separation threshold corresponding to collision. This function is illustrated in Figure 18.

Figure 18. Miss-distance reward function FρF_{\rho}.

When the episode remains active, the reward serves two complementary purposes. First, it discourages large successive changes in the sampled scope angle, thereby promoting smoother target-ship trajectories. Second, it guides the sampled scope angle towards an interception course when such an interception is kinematically feasible. Accordingly, the non-terminal reward is decomposed as

RNT,t=RΔψ,t+RΛ,t,(70)R_{\mathrm{NT},t}=R_{\Delta\psi,t}+R_{\Lambda,t}, \tag*{(70)}

where RΔψ,tR_{\Delta\psi,t} is the scope-angle-change term and RΛ,tR_{\Lambda,t} is the interception-guidance term.

The scope-angle-change component is defined as

RΔψ,t=Cℓℓcw,tnwFℓℓ(Δψtsc),(71)R_{\Delta\psi,t}=C_{\ell\ell}\frac{c_{w,t}}{n_w}F_{\ell\ell}\left(\Delta\psi^{\mathrm{sc}}_t\right), \tag*{(71)}

where

Δψtsc=ψtsc−ψt−1sc(72)\Delta\psi^{\mathrm{sc}}_t=\psi^{\mathrm{sc}}_t-\psi^{\mathrm{sc}}_{t-1} \tag*{(72)}

is the change between two consecutive sampled scope angles. The ratio cw,t/nwc_{w,t}/n_w gradually increases the contribution of this term as additional IWs are sampled. Additionally, CℓℓC_{\ell\ell} is another scaling factor.

The function FℓℓF_{\ell\ell}, as shown in Figure 17, penalizes large changes in scope angle. The change is evaluated under a zero-centered truncated normal distribution, such that small changes are assigned higher likelihood than abrupt changes. Let

Figure 17. Scope-angle change reward function FℓℓF_{\ell\ell}.

ℓ(x)=log⁡FTN(x)(73)\ell(x)=\log F_{\mathrm{TN}}(x) \tag*{(73)}

denote the corresponding log-likelihood. The normalized scope-angle change function is then

Fℓℓ(Δψsc)=ℓ(Δψsc)−ℓ(0)ℓ(0)−ℓ(Δψmax⁡sc),(74)F_{\ell\ell}\left(\Delta\psi^{\mathrm{sc}}\right)=\frac{\ell\left(\Delta\psi^{\mathrm{sc}}\right)-\ell(0)}{\ell(0)-\ell\left(\Delta\psi^{\mathrm{sc}}_{\max}\right)}, \tag*{(74)}

where FTNF_{\mathrm{TN}} is the probability density function of a doubly truncated normal distribution [15],

FTN(x)=1σTNϕ(x−μTNσTN)Φ(b−μTNσTN)−Φ(a−μTNσTN),a≤x≤b,(75)F_{\mathrm{TN}}(x)=\frac{\frac{1}{\sigma_{\mathrm{TN}}}\phi\left(\frac{x-\mu_{\mathrm{TN}}}{\sigma_{\mathrm{TN}}}\right)}{\Phi\left(\frac{b-\mu_{\mathrm{TN}}}{\sigma_{\mathrm{TN}}}\right)-\Phi\left(\frac{a-\mu_{\mathrm{TN}}}{\sigma_{\mathrm{TN}}}\right)},\qquad a\le x\le b, \tag*{(75)}

where ϕ(⋅)\phi(\cdot) and Φ(⋅)\Phi(\cdot) denote the probability density function and cumulative distribution function of the standard normal distribution, respectively. FℓℓF_{\ell\ell} plot is given in Figure 17.

The interception-guidance component is defined as

RΛ,t={(nw+1−cw,tnw)CΛFΛ(Δψic,tsc),if interception is feasible,CpassFρ(dn,t),otherwise,(76)R_{\Lambda,t}= \begin{cases} \left(\dfrac{n_w+1-c_{w,t}}{n_w}\right)C_{\Lambda}F_{\Lambda}\left(\Delta\psi^{\mathrm{sc}}_{\mathrm{ic},t}\right), & \text{if interception is feasible},\\ C_{\mathrm{pass}}F_{\rho}(d_{n,t}), & \text{otherwise}, \end{cases} \tag*{(76)}

where CΛC_{\Lambda} and CpassC_{\mathrm{pass}} scale their respective reward term, and FΛF_{\Lambda} acts as a interception-guiding function. FΛF_{\Lambda} takes inspiration from the first reward design variation based on Goto et al. work [9]. This function is illustrated in Figure 19 and formulated as

Figure 19. Intercepting scope-angle reward function FΛF_{\Lambda}.

FΛ(Δψic,tsc)=exp⁡(−∣Δψic,tsc∣κΛ).(77)F_{\Lambda}(\Delta\psi_{\mathrm{ic},t}^{\mathrm{sc}}) = \exp\left(-\frac{\left|\Delta\psi_{\mathrm{ic},t}^{\mathrm{sc}}\right|}{\kappa_{\Lambda}}\right). \tag*{(77)}

This second non-terminal reward term encourages the target ship to select scope angles that are consistent with an interception course through FΛ(Δψic,tsc)F_{\Lambda}(\Delta\psi_{\mathrm{ic},t}^{\mathrm{sc}}). However, it also fairly penalizes the remaining IW samplings when any interception attempts are no longer feasible. This is done by giving the target ship a reward based on how close it was able to get to the own ship through Fρ(dn,t)F_{\rho}(d_{n,t}). At each decision event, a geometrically feasible interception heading is computed from the current relative positions and velocities of the own ship and target ship. The angular discrepancy between the sampled scope angle ψtsc\psi_{t}^{\mathrm{sc}} and the corresponding interception heading ψtic\psi_{t}^{\mathrm{ic}} is denoted by

Δψic,tsc=ψtic−ψtsc.(78)\Delta\psi_{\mathrm{ic},t}^{\mathrm{sc}} = \psi_{t}^{\mathrm{ic}} - \psi_{t}^{\mathrm{sc}}. \tag*{(78)}

The interception heading prediction formula will be presented in Section A.1. Unlike the scope-angle-change term, the weighting of the interception component decreases as additional waypoints are sampled. Consequently, interception guidance dominates the early part of an episode, whereas trajectory smoothness becomes increasingly important as the target ship progresses through its available waypoint requests.

Results

Training Parameters

For the training process, we includes two IW-enabled target ships as the AST stressor. Each of target ship is equipped with IW FMU that allows it to change trajectory mid simulation. The parameter settings of the module for all target ships are shown in Table 1.

ParameterValue
Event 1 start time250 seconds
RoA300 meters
Maximum IWS counts5
Maximum ψsc\psi^{\mathrm{sc}}30.0°

Table 1. IWS module parameters, for all target ships.

The training parameters for each RL algorithm are presented in Table 2 and Table 3. R-PPO and PPO share the same training because inherently they are derived from the same algorithm. The difference is that R-PPO uses MultiInputLstmPolicy to enable POMDP, whereas both PPO and SAC use MultiInputPolicy due to their MDP property. We aim to establish comparable training conditions across all RL algorithms to ensure a fair comparison. Specifically, we use the same learning rate, number of environments, discount factor γ\gamma, and total number of timesteps across all algorithms. We pick γ\gamma value of 1.0 because the failure-discovery problem is considered as a finite state-action transition problem. Additionally, we enable parallel training by using multiple environments to increase transition data gathering and speed up the training process. We could also identify how many neural network updates for each RL core based on the parameters, where the training happens differently for on-policy and off-policy method.

ParameterValue
Learning rate3×10−43 \times10^{-4}
Number of environments64
Rollout steps512
Batch size1024
Number of epochs5
Discount factor (γ\gamma)1.0
Clipping range0.2
Entropy coefficient0.005
Value function coefficient0.5
Maximum gradient norm0.5
Advantage normalizationTrue
Total timesteps10710^7

Table 2. R-PPO and PPO training hyperparameters for EB-ASTv2.

ParameterValue
Learning rate3×10−43 \times10^{-4}
Number of environments64
Replay buffer size1.28×1061.28 \times10^6
Learning starts1.28×1051.28 \times10^5
Batch size1024
Discount factor (γ\gamma)1.0
Soft update coefficient (τ\tau)0.005
Training frequency10
Gradient steps128
Entropy coefficientAuto
Total timesteps10710^7

Table 3. SAC training hyperparameters for EB-ASTv2.

For R-PPO and PPO, training was performed per rollout, with each rollout spanning 512 timesteps across 64 parallel environments. Therefore, each rollout collects 512 × 64 = 32,768 transitions. Given a batch size of 1,024, the number of minibatches per rollout is

32,7681024=32.\frac{32{,}768}{1024}=32.

Since each rollout is optimized for 5 epochs, the number of neural network updates per rollout is

32×5=160.32 \times5=160.

For a total training budget of 10710^{7} transitions, the number of rollouts is approximately

10732,768≈305.18.\frac{10^{7}}{32{,}768}\approx305.18.

Thus, the total number of neural network updates for each R-PPO and PPO training run is approximately

160×305.18≈48,828160 \times305.18 \approx48{,}828

updates. Thus, over 10 million timesteps, R-PPO and PPO performs a total of 48,828 gradient updates.

For SAC, training is performed after every training frequency of 10 environment steps once the initial exploration phase, specified by learning starts, has ended. With 64 parallel environments, each training frequency collects 10 × 64 = 640 transitions, which are stored in the replay buffer. After each collection cycle, SAC performs 128 gradient updates, with each update using a randomly sampled minibatch of 1024 transitions from the replay buffer. Over a total of 10 million timesteps, the number of collection cycles is

107640=15,625.\frac{10^{7}}{640}=15{,}625.

The exploration phase lasts for 128,000 timesteps, corresponding to

128,000640=200\frac{128{,}000}{640}=200

collection cycles during which no gradient updates are performed. Therefore, the total number of training cycles is

15,625−200=15,425.15{,}625-200=15{,}425.

Since each training cycle consists of 128 gradient updates, the total number of gradient updates is

15,425×128=1,974,400.15{,}425 \times128=1{,}974{,}400.

Thus, over 10 million timesteps, SAC performs a total of 1,974,400 gradient updates.

We train the policy using a 100 traffic cases generated with STG. Each traffic case centered on the own ship, whose initial position is located at north-east coordinate of (0, 0). For each generated case, STG parameters are sampled within their respective allowed range. which are specified in a JSON file named encounter_settings. The file’s contents are provided in Appendix Section A.2. At each call to reset(), the RL environment wrapper randomly selects one of the 100 traffic cases. The cases are indexed from 0 to 99, and the selection is performed using a uniform distribution, such that each case has an equal probability of being selected.

RL Cores’ Performance Comparisons

In this section, we discuss each RL core’s performances by comparing their reward collection and episode length trends over the 10 million training timesteps. We evaluate R-PPO’s performance by comparing its results directly with PPO and SAC in the following sections. Note that the maximum episode length will be 10 given that each target ship only allowed to sample 5 IWs per episode.

R-PPO vs PPO

Based on Figure 20, we see that both core yields a nearly-identical trends, where PPO reward is only negligibly higher compared to the R-PPO. Nevertheless, both grow from a low episodic reward of −30-30 to a value above −15-15. This signifies that both implementation are steadily improved overtime, owing to the same objective architecture of R-PPO and PPO. The same goes with how the episode length decrease overtime, showed in Figure 21. Both implementation manage to decrease the episode length to nearly 6 IW samplings per episode before a termination is discovered.

Figure 20. Reward comparison.

Figure 21. Episode length comparison.

Looking more closely at the results in Table 4 and Table 5, R-PPO policy generates a larger number of failure scenarios, with 68 out of 100 cases resulting in own ship collision, compared with 58 out of 100 for the PPO policy. Additionally, the average episode length of the no collision and own ship collision differ slightly between the two methods: R-PPO policy tends to produce shorter no collision episodes of 6.6 and longer own ship collision episodes of 5.6, whereas PPO results exhibit the opposite trend with 6.8 long no collision episodes and 5.3 long own ship collision episodes.

OutcomeCountPercentageMean Episode Length
No Collision32 / 10032.00%6.6
Target Ship Collision0 / 1000.00%0.0
Own Ship Collision68 / 10068.00%5.6

Table 4. R-PPO policy’s failure scenario generation training score.

OutcomeCountPercentageMean Episode Length
No Collision42 / 10042.00%6.8
Target Ship Collision0 / 1000.00%0.0
Own Ship Collision58 / 10058.00%5.3

Table 5. PPO policy’s failure scenario generation training score.

R-PPO vs SAC

The results reveal a notable difference between the behavior of the R-PPO and SAC methods. Due to its off-policy learning scheme and substantially higher update frequency, the SAC results contain a denser set of training observations with greater variance. For visual clarity, the SAC results are therefore presented as mean curves with shaded variance bands.

As shown in Figure 22, the mean reward achieved by SAC increased rapidly for the first million training timesteps, until the increase slows down considerably for the remainder of the training phase. Eventually, the R-PPO’s mean reward catches up around the 7×1067 \times10^{6} timesteps and stay higher compared to the SAC’s mean reward. However, the variability in the SAC results is considerably larger, with the upper range of the shaded bands frequently extending beyond the reward achieved by R-PPO. This indicates that although SAC performs worse on average, some training outcomes are comparable to or higher than those of R-PPO.

Figure 22. Reward comparison.

Figure 23 further shows that SAC produces longer episodes than R-PPO, averaging approximately 7–8 IW samplings per episode. This suggests that SAC generally requires more trajectory modifications before reaching a terminal state compared with R-PPO and PPO. This observation is also reflected in the evaluation results presented in Table 6. In particular, the SAC policy produces longer no collision and own ship collision episodes, with average lengths of 7.9 and 6.9, respectively. In comparison, the corresponding values are 6.6 and 5.6 for R-PPO, and 6.8 and 5.3 for PPO. Overall, these results indicate that SAC tends to require more IW sampling steps to reach a terminal outcome than either R-PPO or PPO.

OutcomeCountPercentageMean Episode Length
No Collision46 / 10046.00%7.9
Target Ship Collision0 / 1000.00%0.0
Own Ship Collision54 / 10054.00%6.9

Table 6. SAC policy’s failure scenario generation training score.

Figure 23. Episode length comparison.

Performance Variability Across Independent Training Runs

We then investigate how the performance of each policy varies across independent training runs using identical hyperparameters and training settings. This experiment assesses whether the observed performance is consistent across runs, rather than being the result of particular training converging to a suboptimal local optimum. For each assessment, we compare the collision-inducing score and training time of each method. The score is defined as the percentage of 100 generated scenarios that result in a collision. All runs were performed on Norwegian University of Science and Technology (NTNU)’s IDUN High-Performance Computing systems [46], using 64 CPU cores and 256 GB of RAM per run. We limit the experiment to five runs per policy due to the substantial computational costs, as each training run generally requires more than five days.

Based on Table 7, we observe that R-PPO policy achieves the highest average score of 66.80%, compared to PPO and SAC scores of 65.20% and 51.80% respectively. R-PPO policy also requires the shortest average training time of 5.41 days, while PPO and SAC policies requires averagely 6.02 days and 7.09 days. However, PPO policy outperforms R-PPO policy in several runs, specifically in Trial 2, 3, and 5, and achieves the highest single-run score of 71.0% compared with R-PPO’s highest score of 69.0%.

R-PPOPPOSAC
TrialScore (%)Time (days)Score (%)Time (days)Score (%)Time (days)
168.06.1558.06.3054.07.26
264.04.9768.06.1057.08.31
365.04.9566.06.0851.05.55
468.04.9663.06.7743.07.01
569.06.0371.04.8354.07.33
Mean66.805.4165.206.0251.807.09

Table 7. Training time and performance across independent runs.

To examine this variability more closely, the results’ statistics are summarized in Table 8 and visualized in Figure 24. Although PPO policy occasionally achieves higher score than R-PPO policy, its scores are more dispersed across the independent runs. In contrasts, R-PPO scores are more tightly clustered around the mean, indicating more consistent performance. On the other hand, SAC policy’s score are generally lower and show and even wider spread than those of PPO.

MethodnnMeanStd.MedianQ25Q75MinMax
R-PPO566.802.1768.0065.0068.0064.0069.00
PPO565.204.9766.0063.0068.0058.0071.00
SAC551.805.3654.0051.0054.0043.0057.00

Table 8. Performance statistics across independent test runs.

Figure 24. Collision-inducing performance comparison of all EB-ASTv2 policies with 5 repeated trainings for each method.

We can also learn the scenario generation behavior of each policy by comparing the length of the generated collision and non-collision scenarios. Episode length is defined as the number of state-action transitions within an episode. The corresponding statistics are presented in Table 9 with the distribution visualized in Figure 25. The R-PPO policy tends to generate a longer collision scenarios, with episode length of 5.60, but a shorter non-collision scenarios, with a mean of 6.64. The opposite applies to PPO policy, where it tends to generate a shorter collision scenarios, with a mean of 5.36, and slightly longer non-collision scenarios, with a mean of 6.64. The SAC policy generates the longest scenarios for both outcomes, with a mean episode lengths of 6.98 and 7.68 for collision and non-collision respectively. These patterns are consistent with those observed in the initial-run comparisons in § 6.2, suggesting that the scenario-generation characteristics of the EB-ASTv2 policies remain similar across independent training runs.

MethodnnOutcomeMeanStd.MedianQ25Q75MinMax
R-PPO5Collision5.600.075.65.65.65.55.7
No collision6.640.116.66.66.76.56.8
PPO5Collision5.360.115.45.35.45.25.5
No collision6.680.086.76.66.76.66.8
SAC5Collision6.980.087.06.97.06.97.1
No collision7.680.157.77.67.77.57.9

Table 9. Scenario lengths statistics across independent test runs.

Figure 25. Collision-inducing episode lengths comparison of all EB-ASTv2 policies with 5 repeated trainings for each method.

These differences in episode length also suggest distinct scenario-generation behavior across the policies. PPO policy tend to be more conservative when generating collision-inducing scenarios, which reflects its slow-and-stable learning behavior influenced by the clipped surrogate objective. SAC policy tend to generate a more complex failure-inducing scenarios, where it tends to have more IWs that enables pronounced heading change and more flexible course. This behavior may be attributed to the higher variability during training, associated with SAC’s off-policy nature and exploration strategy. R-PPO policy lies between PPO and SAC policies in terms of its failure-inducing scenario generation tendency. This may be related to its recurrent architecture, which can better exploit state-action transition information while retaining the relatively stable learning characteristics of PPO. This ultimately allows R-PPO policy to handle better the limited observations from the co-simulation modules.

Scenario Generation Comparisons

In this section, we compare different generated scenarios by R-PPO, PPO, and SAC policies from a selected number of traffic cases. More specifically, we compare the generated scenarios from Case 50, 66, 96, 77, 89, and 44.

In Case 50, all three policies successfully generate an own ship collision scenario. For the R-PPO policy, the collision is triggered by target ship 2, whereas for both PPO and SAC, the collision involves target ship 1. In all three scenarios, the target ships maneuver toward the own ship and eventually overwhelm its COLAV system, resulting in a collision. However, the trajectories generated by SAC contain more IWs and exhibit more pronounced heading changes than those generated by R-PPO and PPO. This indicates that SAC tends to produce more complex and highly maneuvering trajectories when searching for collision-inducing scenarios.

In some cases, the PPO policy is unable to generate a true own ship collision scenario. This is demonstrated in Case 66, where both R-PPO and SAC successfully identify collision-inducing scenarios, whereas PPO does not. The scenarios generated by R-PPO and SAC share a notable similarity: both contain additional turns and maneuvers before eventually leading to an own ship collision. On the contrary, the trajectory generated by PPO remains

Figure 26. R-PPO trajectory from Case 50 (TS2-collision).

Figure 27. R-PPO trajectory from Case 66 (TS2-collision).

Figure 28. PPO trajectory from Case 50 (TS1-collision).

Figure 29. PPO trajectory from Case 66 (No collision).

Figure 30. SAC trajectory from Case 50 (TS1-collision).

Figure 31. SAC trajectory from Case 66 (TS2-collision).

Figure 32. R-PPO trajectory from Case 96 (TS2-collision).

Figure 33. R-PPO trajectory from Case 77 (No collision).

Figure 34. PPO trajectory from Case 96 (TS1-collision).

Figure 35. PPO trajectory from Case 77 (No collision).

Figure 36. SAC trajectory from Case 96 (No collision).

Figure 37. SAC trajectory from Case 77 (TS1-collision).

Figure 38. R-PPO trajectory from Case 89 (No collision).

Figure 40. PPO trajectory from Case 89 (No collision).

Figure 42. SAC trajectory from Case 89 (No collision).

Figure 39. R-PPO trajectory from Case 44 (No collision).

Figure 41. PPO trajectory from Case 44 (No collision).

Figure 43. SAC trajectory from Case 44 (No collision).

comparatively straight, suggesting that its more conservative maneuvering limits its ability to discover a collision scenario in this case.

Conversely, there are cases in which the SAC policy fails to generate an own ship collision scenario. In Case 96, R-PPO generates a scenario in which the own ship collides with target ship 2, while PPO generates a collision involving target ship 1. The R-PPO trajectories, particularly that of target ship 2, contain more IWs and therefore exhibit more frequent and pronounced heading changes although still maintaining a rather simple course. The PPO trajectories are considerably simpler and succeed in triggering a collision relatively early by maneuvering target ship 1 toward the own ship. The trajectories generated by SAC, on the other hand, are comparatively complex but ultimately fail to produce an own ship collision.

Nevertheless, there are specific cases in which SAC performs better than its R-PPO and PPO counterparts. In Case 77, both R-PPO and PPO struggle to identify collision-inducing scenarios because the own ship travels significantly faster than both target ship 1 and target ship 2. Consequently, a substantially larger interception angle is required for either target ship to reach the own ship. The trajectories generated by R-PPO and PPO remain relatively conservative, with both policies maintaining comparatively straight courses. In contrast, SAC overcomes this limitation by selecting a trajectory with a much larger interception angle, followed by a sharp turn that ultimately results in an own ship collision. This case illustrates how the greater maneuvering flexibility exhibited by SAC can be advantageous when more aggressive trajectory changes are required to induce a collision.

Furthermore, none of the policies is guaranteed to identify a failure scenario for every initial condition. This is illustrated by Cases 89 and 44, in which none of the three policies succeeds in generating a collision-inducing trajectory. In Case 89, target ship 1 is initially located sufficiently far from the own ship that interception is infeasible given its speed, while target ship 2 travels too slowly to intercept the own ship. Similarly, in Case 44, interception is infeasible because the feasible trajectories of the two target ships do not intersect the own ship’s trajectory in a way that allows a collision to occur. Nevertheless, even in these unsuccessful cases, the generated trajectories still exhibit behavior consistent with an attempt to induce a collision: the policies maneuver the target ships toward the own ship and thereby increase the collision risk, despite ultimately being unable to produce an actual collision.

Continued ebast2 Training for Improving the Policy

Figure 45. Episode length comparison.

We then evaluate whether the R-PPO policy can be further improved through continued training. The previously trained policy is trained for an additional 4 million timesteps, and its performance during this second training phase is evaluated. As shown in Table 10, the retrained R-PPO policy achieves a 2% improvement in generating own-ship collision scenarios. This is consistent with the mean reward trend in Figure 44. The gray-shaded region represents the previous training phase, while the orange dashed line shows an ordinary least-squares linear fit to the continued training phase. The mean reward increase are quite modest, whereas the mean episode length remains relatively stable. Although small, this improvement demonstrates the potential for further refinement of an already trained EB-ASTv2 policy.

Figure 44. Reward comparison.

OutcomeCountPercentageMean Episode Length
No Collision30 / 10030.00%6.5
Target Ship Collision0 / 1000.00%0.0
Own Ship Collision70 / 10070.00%5.6

Table 10. Second-trained R-PPO policy’s training score.

Continued ebast2 Training with Different colav System

OutcomeCountPercentageMean Episode Length
No Collision45 / 10045.00%6.8
Target Ship Collision0 / 1000.00%0.0
Own Ship Collision55 / 10055.00%5.5

Table 12. Second-trained R-PPO policy’s training score using different COLAV system.

We also evaluate whether the trained R-PPO policy can generalize to a different COLAV system during stress testing. For simplicity, the alternative system is based on the implementation in § 3.3, but its response is changed from turning to starboard to turning to port. When the trained R-PPO policy is applied to this modified system, its success rate in generating own ship collision scenarios drops significantly from 68% to 19%, as shown in Table 11. This is consistent with Figure 46, where the mean reward drops sharply after 10 million timesteps. At the same time, the mean episode length increases, indicating that the policy struggles to find effective collision-inducing scenarios.

OutcomeCountPercentageMean Episode Length
No Collision81 / 10081.00%6.6
Target Ship Collision0 / 1000.00%0.0
Own Ship Collision19 / 10019.00%6.4

Table 11. First-trained R-PPO policy’s training score using different COLAV system.

Figure 46. Reward comparison.

We therefore retrain the R-PPO policy for another 10 million timesteps. During this training phase, each episode has a 50% chance of using either turning to starboard or turning to port COLAV system, exposing the policy to both behaviors during training. After retraining, the policy achieves a 55% collision generation rate for either configuration. This improvement is also reflected in Figure 46, where the mean reward increases throughout the second training phase, while the mean episode length decreases in Figure 47. These results indicate that the retrained policy generalizes better across different COLAV behaviors while remaining effective at generating collision-inducing scenarios consistent with the AST objective.

Figure 47. Episode length comparison.

Conclusion

Through our experiment, the proposed EB-ASTv2 framework demonstrates its capability as both verification and safety validation methods for CPS by fulfilling these three goals:

  • Providing counter-examples across a wide range of ship traffic scenarios by adopting AST as the core algorithm.

  • Supports realistic scenario discovery by generating advancing adversarial trajectories and maintaining a balance between trajectory smoothness, interception intent through IWS.

  • Demonstrates an increasing ability of the trained policies to generate collision-inducing scenarios, quantified by the failure rate across training cases.

We also compared different RL-algorithm cores, namely R-PPO, PPO and SAC, to demonstrate how the POMDP formulation might be advantageous, compared to its MDP alternatives, as the core of the proposed EB-ASTv2 method to stress test black-box simulation like Co-simulation. The proposed R-PPO policy adoption proves a higher average failure-discovery performance of 66.80% compared to PPO policy (65.20%) and SAC policy (51.80%). Despite this, different RL policy cores exhibit distinct scenario-generation characteristics. Overall, the three RL cores exhibit distinct scenario-generation characteristics. PPO shows more conservative behavior, SAC generates more variable and complex failure scenarios, while R-PPO provides a balance between these characteristics. The recurrent structure of R-PPO appears particularly beneficial in handling the limited observations provided by the co-simulation modules.

Additionally, we demonstrate that a trained policy can be further improved through continued training. The retrained R-PPO policy achieves a higher own-ship collision generation rate, suggesting the potential for an iterative safety-assessment cycle: after a system is stress tested and improved, the EB-ASTv2 policy can be retrained to evaluate the updated system. We also show that the policy can adapt to different system behaviors by exposing it to both the original and modified systems during retraining phase. This is achieved by uniformly sampling the system configuration at each episode reset(). In our experiment, varying the COLAV response enables the retrained policy to generalize across both configurations while still generating meaningful failure scenarios. Together, these results demonstrate the potential of the EB-ASTv2 framework as a safety-assessment method for black-box maritime CPS.

Acknowledgments and Disclosure of Funding

This work was supported by the Research Council of Norway under the project System optimisation and stress testing in co-simulations (OptiStress) (grant no. 344238). The authors would also like to thank Børge Rokseth and Stian Skjong for their valuable support in facilitating this research collaboration between NTNU, SINTEF, and DNV.

Appendix / supplemental material

Interception Heading Prediction

In order to compute the interception-guidance reward term, a procedure has been developed to determine whether a target ship may intercept the own ship at every decision event. If interception is deemed feasible, it then predicts the interception heading ψtic\psi^{ic}_{t} accordingly.

Let us define own ship positions (pOS∈R2\mathbf{p}_{\mathrm{OS}} \in\mathbb{R}^{2}), target ship positions (pTS∈R2\mathbf{p}_{\mathrm{TS}} \in\mathbb{R}^{2}), own ship velocities (vOS∈R2\mathbf{v}_{\mathrm{OS}} \in\mathbb{R}^{2}), target ship forward speed (uTS∈R1u_{\mathrm{TS}} \in\mathbb{R}^{1}), and target ship directional vector unit (μTS∈R2\boldsymbol{\mu}_{\mathrm{TS}} \in\mathbb{R}^{2}). An interception can be framed from the kinematic perspective as

pOS+vOSt=pTS+uTSμTSt.(79)\mathbf{p}_{\mathrm{OS}} + \mathbf{v}_{\mathrm{OS}}t = \mathbf{p}_{\mathrm{TS}} + u_{\mathrm{TS}}\boldsymbol{\mu}_{\mathrm{TS}}t. \tag*{(79)}

Refactor this by moving μTS\boldsymbol{\mu}_{\mathrm{TS}} to left side, and we get

μTS=r+vOSuTSt,(80)\boldsymbol{\mu}_{\mathrm{TS}} = \frac{\mathbf{r} + \mathbf{v}_{\mathrm{OS}}}{u_{\mathrm{TS}}t}, \tag*{(80)}

where r=pOS−pTS\mathbf{r} = \mathbf{p}_{\mathrm{OS}} - \mathbf{p}_{\mathrm{TS}} following Equation (36). By applying ∥μTS∥=1\lVert\boldsymbol{\mu}_{\mathrm{TS}}\rVert= 1, we ended up with

∥uTSt∥=∥r+vOS∥.(81)\lVert u_{\mathrm{TS}}\mathbf{t}\rVert= \lVert\mathbf{r} + \mathbf{v}_{\mathrm{OS}}\rVert. \tag*{(81)}

By squaring both sides, and arranging the equation by moving all terms to the left side, we get

(vOS⋅vOS−uTS2)t2+2(r⋅vOS)t+(r⋅r)=0.(82)(\mathbf{v}_{\mathrm{OS}} \cdot\mathbf{v}_{\mathrm{OS}} - u_{\mathrm{TS}}^{2})t^{2} + 2(\mathbf{r} \cdot\mathbf{v}_{\mathrm{OS}})t + (\mathbf{r} \cdot\mathbf{r}) = 0. \tag*{(82)}

Above is the quadratic formula that follows At2+Bt+C=0A t^{2} + B t + C = 0. Observe now that we have

A=vOS⋅vOS−uTS2(83)A = \mathbf{v}_{\mathrm{OS}} \cdot\mathbf{v}_{\mathrm{OS}} - u_{\mathrm{TS}}^{2} \tag*{(83)}
B=2(r⋅vOS)(84)B = 2(\mathbf{r} \cdot\mathbf{v}_{\mathrm{OS}}) \tag*{(84)}
C=r⋅r.(85)C = \mathbf{r} \cdot\mathbf{r}. \tag*{(85)}

From these, we can learn several things about the own-ship–target-ship kinematics. When A≈0A \approx0 means that both own ship and target ship travels at a similar speed, hence the only enabling factor for feasible interception are their geometrical positions. We can also compute the discriminant as

disc⁡=B2−4AC,(86)\operatorname{disc} = B^2 - 4AC, \tag*{(86)}

where it determines whether interception is infeasible when disc⁡<0\operatorname{disc} < 0. If the discriminant is positive, solution exists and the roots of (82) can solved by computing

t1=−B+disc⁡2A(87)t_1 = \frac{-B + \sqrt{\operatorname{disc}}}{2A} \tag*{(87)}
t2=−B−disc⁡2A.(88)t_2 = \frac{-B - \sqrt{\operatorname{disc}}}{2A}. \tag*{(88)}

These terms are all of the possible times-to-intercept. Given the threshold tpredict limitt_{\text{predict limit}}, interception is only feasible when at least one of the root fulfill the condition 0≤tintercept≤tpredict limit0 \leq t_{\text{intercept}} \leq t_{\text{predict limit}}. Beyond that, the interception either occurred in the past or is too far in the future.

Using the solution tinterceptt_{\text{intercept}}, target ship velocity can now be computed by

νTS=r+νOStintercepttintercept,(89)\boldsymbol{\nu}_{TS} = \frac{\mathbf{r} + \boldsymbol{\nu}_{OS}t_{\text{intercept}}}{t_{\text{intercept}}}, \tag*{(89)}

where the interception heading ψtic\psi_t^{ic} can be obtained by solving

ψtic=νTS∥νTS∥.(90)\psi_t^{ic} = \frac{\boldsymbol{\nu}_{TS}}{\lVert\boldsymbol{\nu}_{TS}\rVert}. \tag*{(90)}

stg parameters

ParameterValue
θ13\theta_{13} criteria67.5∘67.5^\circ
θ14\theta_{14} criteria5.0∘5.0^\circ
θ15\theta_{15} criteria5.0∘5.0^\circ
θ15\theta_{15}θ15∈{112.5∘,247.5∘}\theta_{15} \in\{112.5^\circ, 247.5^\circ\}
vrelv_{\mathrm{rel}} head onvrel∈[0.5,1.5]v_{\mathrm{rel}} \in[0.5, 1.5]
vrelv_{\mathrm{rel}} overtaking stand-onvrel∈[1.5,2.0]v_{\mathrm{rel}} \in[1.5, 2.0]
vrelv_{\mathrm{rel}} overtaking give-wayvrel∈[0.25,0.75]v_{\mathrm{rel}} \in[0.25, 0.75]
vrelv_{\mathrm{rel}} crossing stand-onvrel∈[0.5,1.5]v_{\mathrm{rel}} \in[0.5, 1.5]
vrelv_{\mathrm{rel}} crossing give-wayvrel∈[0.5,1.5]v_{\mathrm{rel}} \in[0.5, 1.5]
Situation length90.0 minutes
Maximum meeting distance5.0 nautical miles
Situation develop time60.0 minutes
Vector timetvector∈[40.0,60.0]t_{\mathrm{vector}} \in[40.0, 60.0] minutes
Own ship SoG6.0 knots
Own ship CoGψsog∈{0.0∘,15.0∘,⋯ ,345.0∘,360∘}\psi_{\mathrm{sog}} \in\{0.0^\circ, 15.0^\circ, \cdots, 345.0^\circ, 360^\circ\}
Margin δ\delta2.0°
β\beta head onβ∈[(−θ14+δ),(θ14−δ)]\beta\in[(-\theta_{14}+\delta),(\theta_{14}-\delta)]
β\beta overtaking give-wayβ∈[(−θ13+δ),(θ13−δ)]\beta\in[(-\theta_{13}+\delta),(\theta_{13}-\delta)]
β\beta overtaking stand-on starboardβ∈[(θ15,min⁡+δ),(180.0∘−δ)]\beta\in[(\theta_{15,\min}+\delta),(180.0^\circ-\delta)]
β\beta overtaking stand-on portβ∈[(−180.0∘+δ),(−θ15,min⁡−δ)]\beta\in[(-180.0^\circ+\delta),(-\theta_{15,\min}-\delta)]
β\beta crossing give-wayβ∈[(θ15,crit+δ),(θ15,min⁡−δ)]\beta\in[(\theta_{15,\mathrm{crit}}+\delta),(\theta_{15,\min}-\delta)]
β\beta crossing stand-onβ∈[(−θ15,min⁡+δ),(θ15,crit−δ)]\beta\in[(-\theta_{15,\min}+\delta),(\theta_{15,\mathrm{crit}}-\delta)]
Navigation statusUnder way using engine

Table 13. STG parameters

References

  1. [1]Open Simulation Platform. URL https://opensimulationplatform.com/.
  2. [2]libcosim: An open-source co-simulation library for C++, May 2026. URL https: //github.com/open-simulation-platform/libcosim.
  3. [4]O. Balci. Verification, validation, and certification of modeling and simulation applications. In Proceedings of the 2003 International Conference on Machine Learn- ing and Cybernetics (IEEE Cat. No.03EX693), pages 150–158, New Orleans, LA, USA, 2003. IEEE. ISBN 978-0-7803-8131-5. doi: 10.1109/WSC.2003.1261418. URL http://ieeexplore.ieee.org/document/1261418/.DOI
  4. [5]Torsten Blockwitz, Martin Otter, Johan Akesson, Martin Arnold, Christoph Clauss, Hilding Elmqvist, Markus Friedrich, Andreas Junghanns, Jakob Mauss, Dietmar Neumerkel, Hans Olsson, and Antoine Viel. Functional Mockup Interface 2.0: The Standard for Tool independent Exchange of Simulation Models. pages 173–184, November 2012. doi: 10.3384/ecp12076173. URL https://ep.liu.se/en/conference-article.aspx?series=ecp&issue=76&Article_No=17.DOI
  5. [6]Yingguang Chu, Lars Ivar Hatledal, Houxiang Zhang, Vilmar Æsøy, and Sören Ehlers. Virtual prototyping for maritime crane design and operations. Journal of Marine Science and Technology, 23(4):754–766, December 2018. ISSN 0948-4280, 1437-8213. doi: 10.1007/s00773-017-0509-z. URL http://link.springer.com/10.1007/s00773-017-0509-z.DOI
  6. [7]Anthony Corso, Robert J. Moss, Mark Koren, Ritchie Lee, and Mykel J. Kochenderfer. A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systems. Journal of Artificial Intelligence Research, 72, October 2021. ISSN 1076-9757. doi: 10.1613/jair.1.12716. URL http://arxiv.org/abs/2005.02979. arXiv:2005.02979 [cs, eess, stat].
  7. [8]Angela Errico and Vittorio Di Vito. Methodology for Estimation of Closest Point of Approach between Aircraft in ATM. In AIAA Aviation 2019 Forum, Dallas, Texas, June 2019. American Institute of Aeronautics and Astronautics. ISBN 978-1-62410-589-0. doi: 10.2514/6.2019-2832. URL https://arc.aiaa.org/doi/10.2514/6.2019-2832.DOI
  8. [9]Takeru Goto, Yuki Kizumi, and Shun Iwasaki. Design of Reward Function on Reinforcement Learning for Automated Driving. IFAC-PapersOnLine, 56(2):7948–7953, 2023. ISSN 2405-8963. doi: 10.1016/j.ifacol.2023.10.902. URL https://linkinghub.elsevier.com/retrieve/pii/S2405896323012855.DOI
  9. [11]Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft Actor-Critic Algorithms and Applications, 2018. URL https://arxiv.org/abs/1812.05905. Version Number: 2.
  10. [12]Vahid Hassani, Martin Rindarøy, Lars T. Kyllingstad, Jørgen B. Nielsen, Severin Simon Sadjina, Stian Skjong, Dariusz Fathi, Trond Johnsen, Vilmar Æsøy, and Eilif Pedersen. Virtual Prototyping of Maritime Systems and Operations. In Volume 7: Ocean Engineering, page V007T06A018, Busan, South Korea, June 2016. American Society of Mechanical Engineers. ISBN 978-0-7918-4998-9. doi: 10.1115/OMAE2016-54886. URL https://asmedigitalcollection.asme.org/OMAE/proceedings/OMAE2016/49998/Busan,%20South%20Korea/281064.DOI
  11. [13]Hanna W. Hjelmeland, Bjørn-Olav H. Eriksen, Ole J. Mengshoel, and Anastasios M. Lekkas. Identification of Failure Modes in the Collision Avoidance System of an Autonomous Ferry using Adaptive Stress Testing. IFAC-PapersOnLine, 55(31):470–477, 2022. ISSN 24058963. doi: 10.1016/j.ifacol.2022.10.472. URL https://linkinghub.elsevier.com/retrieve/pii/S2405896322025186.DOI
  12. [14]Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Computation, 9(8):1735–1780, November 1997. ISSN 0899-7667, 1530-888X. doi: 10.1162/neco.1997.9.8.1735. URL https://direct.mit.edu/neco/article/9/8/1735-1780/6109.DOI
  13. [15]Norman Lloyd Johnson, Samuel Kotz, and Narayanaswamy Balakrishnan. Distributions in statistics. 1: Continuous univariate distributions / Norman L. Johnson ; Samuel Kotz ; N. Balakrishnan. Wiley series in probability and mathematical statistics. Wiley, New York, 2. ed edition, 1994. ISBN 978-0-471-58495-7. Num Pages: 756.
  14. [16]James Kapinski, Jyotirmoy V. Deshmukh, Xiaoqing Jin, Hisahiro Ito, and Ken Butts. Simulation-Based Approaches for Verification of Embedded Control Systems: An Overview of Traditional and Advanced Modeling, Testing, and Verification Techniques. IEEE Control Systems, 36(6):45–64, December 2016. ISSN 1066-033X, 1941-000X. doi: 10.1109/MCS.2016.2602089. URL https://ieeexplore.ieee.org/document/7741019/.
  15. [17]Mark Koren and Mykel Kochenderfer. Efficient Autonomy Validation in Simulation with Adaptive Stress Testing, July 2019. URL http://arxiv.org/abs/1907.06795. arXiv:1907.06795 [cs, eess, stat].
  16. [18]Mark Koren and Mykel J. Kochenderfer. Adaptive Stress Testing without Domain Heuristics using Go-Explore, June 2020. URL http://arxiv.org/abs/2004.04292. arXiv:2004.04292 [cs, eess, stat].
  17. [19]Mark Koren, Ahmed Nassar, and Mykel J. Kochenderfer. Finding Failures in High-Fidelity Simulation using Adaptive Stress Testing and the Backward Algorithm, July 2021. URL http://arxiv.org/abs/2107.12940. arXiv:2107.12940 [cs.LG].
  18. [20]Martin Krammer, Michael Karner, and Anton Fuchs. System design for enhanced forward-engineering possibilities of safety critical embedded systems. In 17th International Symposium on Design and Diagnostics of Electronic Circuits & Systems, pages 234–237, Warsaw, Poland, April 2014. IEEE. ISBN 978-1-4799-4558-0 978-1-4799-4560-3. doi: 10.1109/DDECS.2014.6868797. URL http://ieeexplore.ieee.org/document/6868797/.DOI
  19. [21]Thomas Steenfeldt Laursen and Ole Jakob Mengshoel. Evolutionary Adaptive Stress Testing for Collision Avoidance in Sustainable Maritime Transportation. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pages 2215–2223, NH Malaga Hotel Malaga Spain, July 2025. ACM. ISBN 979-8-4007-1464-1. doi: 10.1145/3712255.3734306. URL https://dl.acm.org/doi/10.1145/3712255.3734306.
  20. [22]Ritchie Lee, Mykel J. Kochenderfer, Ole J. Mengshoel, Guillaume P. Brat, and Michael P. Owen. Adaptive stress testing of airborne collision avoidance systems. In 2015 IEEE/AIAA 34th Digital Avionics Systems Conference (DASC), pages 6C2–1–6C2–13, Prague, Czech Republic, September 2015. IEEE. ISBN 978-1-4799-8940-9. doi: 10.1109/DASC.2015.7311450. URL http://ieeexplore.ieee.org/document/7311450/.DOI
  21. [23]Ritchie Lee, Ole J. Mengshoel, Anshu Saksena, Ryan Gardner, Daniel Genin, Joshua Silbermann, Michael Owen, and Mykel J. Kochenderfer. Adaptive Stress Testing: Finding Likely Failure Events with Reinforcement Learning, December 2020. URL http://arxiv.org/abs/1811.02188. arXiv:1811.02188 [cs].
  22. [24]Andrzej S. Lenart. Analysis of Collision Threat Parameters and Criteria. Journal of Navigation, 68(5):887–896, September 2015. ISSN 0373-4633, 1469-7785. doi: 10.1017/S0373463315000223. URL https://www.cambridge.org/core/product/identifier/S0373463315000223/type/journal_article.DOI
  23. [25]P. Marbach and J.N. Tsitsiklis. Simulation-based optimization of Markov reward processes. In Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171), volume 3, pages 2698–2703, Tampa, FL, USA, 1998. IEEE. ISBN 978-0-7803-4394-8. doi: 10.1109/CDC.1998.757861. URL http://ieeexplore.ieee.org/document/757861/.
  24. [26]Douglas C. Montgomery. Design and analysis of experiments. John Wiley & Sons, Inc, Hoboken, NJ, ninth edition edition, 2017. ISBN 978-1-119-11347-8 978-1-119-29936-3 978-1-119-32093-7.
  25. [27]Rolf Nagel. A Holistic Approach for Energy Flow Simulations in Early Design. Transportation Research Procedia, 14:1601–1610, 2016. ISSN 2352-1465. doi: 10.1016/j.trpro.2016.05.125. URL https://linkinghub.elsevier.com/retrieve/pii/S2352146516301260.DOI
  26. [28]Demin Nalic, Arno Eichberger, Georg Hanzl, Martin Fellendorf, and Branko Rogic. Development of a Co-Simulation Framework for Systematic Generation of Scenarios for Testing and Validation of Automated Driving Systems. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages 1895–1901, Auckland, New Zealand, October 2019. IEEE. ISBN 978-1-5386-7024-8. doi: 10.1109/ITSC.2019.8916839. URL https://ieeexplore.ieee.org/document/8916839/.DOI
  27. [29]Christopher Olah. Understanding LSTM Networks, August 2015. URL https://colah.github.io/posts/2015-08-Understanding-LSTMs/#fn1.
  28. [30]Jean-Nicolas Paquin, Wei Li, Jean Belanger, Loic Schoen, Irene Peres, Cristina Olariu, and Hugo Kohmann. A modern and open real-time digital simulator of All-Electric Ships with a multi-platform co-simulation approach. In 2009 IEEE Electric Ship Technologies Symposium, pages 28–35, Baltimore, MD, USA, April 2009. IEEE. ISBN 978-1-4244-3438-1. doi: 10.1109/ESTS.2009.4906490. URL http://ieeexplore.ieee.org/document/4906490/.DOI
  29. [31]Tom Arne Pedersen, Chanjei Vasanthan, Kristian Karolius, Øystein Engelhardtsen, Koen Pieter Houweling, and Are Jørgensen. Ship Traffic Generator.
  30. [32]Tom Arne Pedersen, Chanjei Vasanthan, Kristian Karolius, Øystein Engelhardtsen, Koen Pieter Houweling, and Are Jørgensen. Generating Structured Set of Encounters for Verifying Automated Collision and Grounding Avoidance Systems. Journal of Physics: Conference Series, 2618(1):012013, October 2023. ISSN 1742-6596, 1742-6596. doi: 10.1088/1742-6596/2618/1/012013. URL https://iopscience.iop.org/article/10.1088/1742-6596/2618/1/012013.DOI
  31. [33]Florian Perabo and Mehdi Zadeh. Multiphysics Modeling and Co-Simulation of Ship Electric Power and Propulsion Systems for Virtual Testing and Verification. IEEE Transactions on Transportation Electrification, pages 1–1, 2024. ISSN 2332-7782, 2372-2088. doi: 10.1109/TTE.2024.3480009. URL https://ieeexplore.ieee.org/document/10716430/.DOI
  32. [34]Marco Pleines, Matthias Pallasch, Frank Zimmer, and Mike Preuss. Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization, 2022. URL https://arxiv.org/abs/2205.11104. Version Number: 1.
  33. [35]Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, and Maximilian Ernestus. Stable Baselines3, March 2021. URL https://github.com/Stable-Baselines-Team/stable-baselines3-contrib.
  34. [36]Marvin Rausand. Reliability of Safety-Critical Systems: Theory and Applications. Wiley, 1 edition, January 2014. ISBN 978-1-118-11272-4 978-1-118-77635-3. doi: 10.1002/9781118776353. URL https://onlinelibrary.wiley.com/doi/book/10.1002/9781118776353.DOI
  35. [37]Stuart J. Russell and Peter Norvig. Artificial intelligence: a modern approach. Pearson Series in Artificial Intelligence. Pearson, Hoboken, NJ, fourth edition edition, 2021. ISBN 978-0-13-461099-3.
  36. [38]Severin Sadjina, Lars Tandle Kyllingstad, Martin Rindarøy, Stian Skjong, Vilmar Æsoy, and Eilif Pedersen. Distributed Co-Simulation of Maritime Systems and Operations. Journal of Offshore Mechanics and Arctic Engineering, 141(1):011302, February 2019. ISSN 0892-7219, 1528-896X. doi: 10.1115/1.4040473. URL https://asmedigitalcollection.asme.org/offshoremechanics/article/doi/10.1115/1.4040473/366241/Distributed-Co-Simulation-of-Maritime-Systems-and.DOI
  37. [39]A. Saltelli. Global sensitivity analysis: the primer. John Wiley, Chichester, England, 2008. ISBN 978-0-470-72518-4. OCLC: 212122308.
  38. [40]Robert G. Sargent. Verification and validation of simulation models. In Proceedings of the 2010 Winter Simulation Conference, pages 166–183, Baltimore, MD, USA, December 2010. IEEE. ISBN 978-1-4244-9866-6. doi: 10.1109/WSC.2010.5679166. URL http://ieeexplore.ieee.org/document/5679166/.DOI
  39. [41]Robin M. Schmidt. Recurrent Neural Networks (RNNs): A gentle Introduction and Overview, November 2019. URL http://arxiv.org/abs/1912.05911. arXiv:1912.05911 [cs.LG].
  40. [42]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal Policy Optimization Algorithms, August 2017. URL http://arxiv.org/abs/1707.06347. arXiv:1707.06347 [cs.LG].
  41. [43]G. Schweiger, C. Gomes, G. Engel, I. Hafner, J. Schoeggl, A. Posch, and T. Nouidui. An empirical survey on co-simulation: Promising standards, challenges and research needs. Simulation Modelling Practice and Theory, 95:148–163, September 2019. ISSN 1569190X. doi: 10.1016/j.simpat.2019.05.001. URL https://linkinghub.elsevier.com/retrieve/pii/S1569190X1930053X.DOI
  42. [44]Guy Shani, Joelle Pineau, and Robert Kaplow. A survey of point-based POMDP solvers. Autonomous Agents and Multi-Agent Systems, 27(1):1–51, July 2013. ISSN 1387-2532, 1573-7454. doi: 10.1007/s10458-012-9200-2. URL http://link.springer.com/10.1007/s10458-012-9200-2.DOI
  43. [45]Andreas R.G. Sitorus and Børge Rokseth. Identifying Ship in Transit Operation Vulnerabilities through Event-Based Adaptive Stress Testing. IFAC-PapersOnLine, 59 (22):716–721, 2025. ISSN 24058963. doi: 10.1016/j.ifacol.2025.11.719. URL https://linkinghub.elsevier.com/retrieve/pii/S2405896325024152.DOI
  44. [46]Magnus Sjålander, Magnus Jahre, Gunnar Tufte, and Nico Reissmann. EPIC: An Energy-Efficient, High-Performance GPGPU Computing Research Infrastructure, July 2024. URL http://arxiv.org/abs/1912.05848. arXiv:1912.05848 [cs.DC].
  45. [47]Stian Skjong, Martin Rindarøy, Lars T. Kyllingstad, Vilmar Æsøy, and Eilif Pedersen. Virtual prototyping of maritime systems and operations: applications of distributed co-simulations. Journal of Marine Science and Technology, 23(4):835–853, December 2018. ISSN 0948-4280, 1437-8213. doi: 10.1007/s00773-017-0514-2. URL http://link.springer.com/10.1007/s00773-017-0514-2.DOI
  46. [48]F.M. Sogandares. Stone axes and warhammers: a decade of distributed simulation in aviation research. In Proceedings 16th Workshop on Parallel and Distributed Simulation, pages 114–121, Washington, DC, USA, 2002. IEEE Comput. Soc. ISBN 978-0-7695-1608-0. doi: 10.1109/PADS.2002.1004208. URL http://ieeexplore.ieee.org/document/1004208/.
  47. [49]Magnus Stein stø. libcosimpy: Python wrapper for libcosim, June 2026. URL https://github.com/open-simulation-platform/libcosimpy.
  48. [51]Richard S. Sutton and G. Barto Barto. Reinforcement Learning: an introduction. MIT Press, Cambridge, MA, 2 edition, 2018. ISBN 978-0-262-03924-6.
  49. [52]Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. In S. Solla, T. Leen, and K. Müller, editors, Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf.
  50. [53]Tobias Rye Torben, Jon Arne Glomsrud, Tom Arne Pedersen, Ingrid B Utne, and Asgeir J Sørensen. Automatic simulation-based testing of autonomous ships using Gaussian processes and temporal logic. Proceedings of the Institution of Mechanical Engineers, Part O: Journal of Risk and Reliability, 237(2):293–313, April 2023. ISSN 1748-006X, 1748-0078. doi: 10.1177/1748006X211069277. URL http://journals.sagepub.com/doi/10.1177/1748006X211069277.DOI
  51. [54]Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan, and Omar G. Younis. Gymnasium: A Standard Interface for Reinforcement Learning Environments, 2024. URL https://arxiv.org/abs/2407.17032. Version Number: 4.
  52. [55]Michael Winter, Julian Taube, Joachim Froeschl, and Hans-Georg Herzog. From Simulation to Testbench Using the FMI-Standard. In 2015 IEEE Vehicle Power and Propulsion Conference (VPPC), pages 1–5, Montreal, QC, Canada, October 2015. IEEE. ISBN 978-1-4673-7637-2. doi: 10.1109/VPPC.2015.7352951. URL http://ieeexplore.ieee.org/document/7352951/.
  53. [56]Junjie Zhao, Christopher Conrad, Rodolphe Fremond, Anurag Mukherjee, Quentin Delezennne, Yu Su, Yan Xu, and Antonios Tsourdos. Co-simulation Digital Twin Framework for Testing Future Advanced Air Mobility Concepts: A Study with BlueSky and AirSim. In 2023 IEEE/AIAA 42nd Digital Avionics Systems Conference (DASC), pages 1–10, Barcelona, Spain, October 2023. IEEE. ISBN 978-3-3503-3357-2. doi: 10.1109/DASC58513.2023.10311124. URL https://ieeexplore.ieee.org/document/10311124/.DOI
  54. [57]Xingyi Zhu, Yanan Wu, Yang Yang, Yafeng Pang, Hongwei Ling, and Dawei Zhang. Real-time risk assessment of aircraft landing based on finite element-virtual prototype-machine learning co-simulation on wet runways. International Journal of Transportation Science and Technology, 13:77–97, March 2024. ISSN 20460430. doi: 10.1016/j.ijts.2023.11.007. URL https://linkinghub.elsevier.com/retrieve/pii/S2046043023001041.DOI
  55. [58]Adil Zouitine. Masking in Deep Reinforcement Learning, June 2022. URL https://boring-guy.sh/posts/masking-rl/.
  56. [59]K.J Åström. Optimal control of Markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10(1):174–205, February 1965. ISSN 0022247X. doi: 10.1016/0022-247X(65)90154-X. URL https://linkinghub.elsevier.com/retrieve/pii/0022247X6590154X.DOI

Paper details

vlnc:W_3rx736da
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 0

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Engineering

Not recorded for this paper

View statement
Looking for related papers…
Contents