betaFeedback

WorksComputer Science

Alignment of Autonomous Cyber Defense Agents via Adaptive Policy Shaping

Abstract

Cybersecurity operators increasingly encounter sophisticated and fast-paced attacks, making autonomous cyber defense (ACD) a critical approach for supporting timely and effective decision-making. Reinforcement learning (RL)-based ACD agents have the ability to autonomously observe network states and take actions to defend and recover system assets, making them well suited for ACD. However, these agents often face critical trade-offs, such as improving network security while inadvertently degrading service availability or violating other operational priorities such as confidentiality and integrity, raising the core challenge of aligning agent behavior with human-specified goals and intent. In this work, we study the problem of aligning RL-based ACD agents with human-specified operational objectives without requiring costly retraining. We formalize alignment for ACD as a policy shaping problem, where a pretrained RL decision-making policy can be adjusted at test time using predicted attributes over agent actions. We also propose an adaptive approach that dynamically controls the alignment–performance trade-off as operational priorities shift during an ongoing cyber attack. We evaluate our method in a simulated cyber defense environment, CybORG, analyzing performance across multiple alignment objectives using the Confidentiality, Integrity, and Availability (CIA) information security triad. Our results show that adaptive test-time policy shaping can significantly improve alignment while preserving core cyber defense performance, offering a practical and scalable alternative to training-time alignment methods in RL. Our code is publicly available at https://github.com/ITM-Kitware/alignment-cage-challenge-2.

Introduction

Modern-day cyber attacks have evolved in scale, sophistication, and impact, challenging traditional human-in-the-loop cybersecurity defenses and amplifying the risk of significant economic and societal harm ([23]). In response, a growing body of research has explored autonomous cyber defense (ACD), in which automated decision-making agents support the defense of large-scale networks against adversaries ([36]). ACD is typically formulated as a zero-sum adversarial game, in which a red agent models an attacker attempting to gain access to hosts and impact operational technologies, while a blue agent models a defender responsible for attack detection and recovery ([18]). This adversarial structure has made reinforcement learning (RL) approaches a natural fit for ACD, given the ability of agents to learn effective sequences of actions with limited prior information and their demonstrated success in other adversarial applications [18, 23]. Recent work has demonstrated the effectiveness of RL agents in simulated cyber environments [18], introduced benchmark cyber environments for agent development [33], and explored the generalization of defensive strategies across network configurations and attackers [37].

As cyber defense systems move beyond decision support to fully automated decision-making, the safety and downstream impact of ACD agents become critical concerns. Central to this challenge is alignment: how an agent’s actions correspond to human values, intentions, and priorities. Misalignment is likely when agents are trained to optimize a single reward or objective [16]. For example, a cyber defense agent may choose not to restore a compromised asset to avoid service downtime for end users, yet this decision can compromise confidentiality and increase the risk of data leakage. Similarly, aggressively isolating or rebuilding compromised hosts can limit further intrusion but disrupt availability for legitimate users. Such trade-offs are inherent to cyber defense and reflect broader challenges studied in other safety-critical domains, including clinical decision-making and medical triage [13]. However, alignment for ACD agents has received limited attention. Existing approaches typically optimize a single-objective environment reward and evaluate agents primarily on task performance, without explicitly incorporating human-defined operational objectives (e.g., confidentiality or integrity), while requiring retraining when objectives or network configurations change, thereby limiting their adaptability in deployment.

Our Contributions: This paper addresses these challenges by formulating alignment of ACD agents as a test-time policy shaping problem, as illustrated in Figure 1. The agent’s policy (i.e., its state-to-action decision rule represented as an action distribution) is reweighted at inference time and combined with an alignment-driven policy that promotes operator-specified objectives. By performing these adjustments at test time rather than during training, we avoid costly retraining while enabling operational priorities to be dynamically adjusted as an incident unfolds. This approach also supports the incorporation of additional operational objectives and generalizes as network topologies grow and attack strategies evolve. Overall, the contributions of our work are the following:

CybORG CAGE 2 environment and policy shaping for alignment

Figure 1. In our modified CybORG CAGE 2 environment (Kiely et al., [17]), a red agent attacks network hosts while a blue agent defends. The blue agent’s RL policy (πRL\pi_{\mathrm{RL}}) is trained to maximize reward but has no direct incentive to protect Confidentiality, Integrity, or Availability (CIA)-critical servers: Database (DS), Front-End (WS), and Auth (AS). To align an agent to these CIA operational priorities without retraining, we construct an alignment policy (πalign\pi_{\mathrm{align}}) using attribute predictors that score each action’s CIA impact, and interpolate it with the RL policy at test time using policy shaping.

  • Adaptive test-time alignment of cyber defense agents: We formulate alignment as a test-time policy shaping framework over a fixed RL policy and investigate alignment to operational priorities of the Confidentiality, Integrity, and Availability (CIA) triad without costly retraining. This is in contrast to prior ACD work (e.g., [19, 32]), which does not address agent alignment to human-specified objectives. We further address limitations of existing test-time policy shaping methods [26], which rely on fixed strategies throughout gameplay (i.e., exclusively reward-driven or alignment-driven decisions), by introducing an adaptive alignment variant that dynamically switches between the two policies based on the observed environment state. This adaptive approach dynamically balances reward and alignment when these objectives conflict.

  • Alignment–performance trade-off analysis: Although prior work on cyber resilience [6] quantifies CIA-based operational objectives, it does not explicitly examine how these objectives trade off against agent reward. To address

this gap, we characterize the inherent trade-off between agent reward and alignment, showing that the cost structure is convex and scale-invariant and can vary with the red agent’s attack strategy. Our empirical results suggest that adaptive policy shaping offers a practical approach to balancing these competing objectives.

  • Scalability and generalization to networks and attacks: To evaluate our approach, we train RL agents in the CybORG CAGE 2 environment (Kiely et al., [17]), a popular testbed for developing and evaluating autonomous cyber defense agents (Wolk et al., [37]). This environment has been limited in scale compared to real-world networks, and prior work has not studied how alignment generalizes across network size or adversary strategy (Xie et al., [39]; Kiely et al., [19]). Therefore, to investigate the generalization of our approach, we extend this simulation environment to support configurable network topology generation and systematically vary network size, host importance weights, and red agent attack strategies in our evaluation. We demonstrate that test-time policy shaping generalizes across network sizes (16–61 hosts), attacker strategies, and alignment attributes, whereas training-time reward shaping collapses at larger network scales. This extended simulation environment can be found in our codebase https://github.com/ITM-Kitware/alignment-cage-challenge-2.

Related Work

RL for Autonomous Cyber Defense: Early efforts to combat cyber attacks focused heavily on intrusion detection systems, which rely on techniques to identify malicious behavior by analyzing data such as application traces or network traffic flows to distinguish between normal and anomalous activity (Nguyen and Reddi, [27]; Buczak and Guven, [5]). In contrast, ACD goes beyond detection by employing agents that can also take defensive actions and mitigate attacks in real time (Vyas et al., [36]), enabling more effective responses. to the growing volume and speed of modern cyber threats. Recently, ACD research has heavily relied on RL, particularly deep reinforcement learning (DRL), in which agents learn defensive or offensive strategies within adversarial simulation environments [36, 18, 7, 30]. RL is particularly well suited to cyber defense environments because it enables agents to learn sequential decision-making strategies under uncertainty, balancing long-term objectives while adapting to dynamic and adversarial network conditions.

To support this line of research, several cyber defense environments have been developed, including Yawning Titan [1], CyberBattleSim [35], and CybORG [33]. Prior work has explored a wide range of RL techniques in these environments, including deep Q-networks (DQN) [25], proximal policy optimization (PPO) [31], hierarchical RL [32], multi-agent RL [19], and the co-training of attack and defense agents [7]. Despite this extensive body of work, little attention has been given to analyzing the behavior of ACD agents beyond task performance, particularly with respect to their alignment with human values and objectives.

Human Value Alignment: AI alignment seeks to ensure that AI systems behave in accordance with human intentions, values, and goals [16]. Interest in alignment stems from documented cases of misaligned behavior, including violations of human values, manipulation, unintended resource acquisition, and power-seeking behavior [16, 28]. Prior work has explored objectives such as robustness, interpretability, controllability, and adherence to human principles [16]. To date, much of this research has focused on large language models, using approaches including constitutional AI [4], RL from human feedback [3], and RL from AI feedback [21]. Cyber defense introduces unique alignment challenges, as autonomous agents must balance security objectives against competing operational priorities such as service availability and mission continuity.

RL Alignment: RL agents optimize behavior through interaction with an environment and reward signals [2]. However, prior work has shown that optimizing reward alone can lead RL agents to exhibit undesirable Machiavellian behaviors [28, 26]. Approaches for aligning RL agents have therefore included reward modeling [40, 41], preference learning from human feedback [10, 29, 12], and reward shaping or human-in-the-loop methods such as learning from demonstration [8, 38, 24]. This differs from safe RL, which focuses on enforcing safety constraints and avoiding violations during optimization [14 b], rather than learning or inferring the human preferences and values that define acceptable behavior. Value alignment presents a distinct challenge because human values are often heterogeneous, context-dependent, and difficult to specify compared to explicit safety constraints [13, 26]. While prior work has explored alignment for RL agents in a variety of domains, its application to adversarial and safety-critical settings such as ACD remains relatively underexplored.

Methodology

ACD introduces trade-offs among multiple operational objectives whose relative priority depends on the attack scenario [6]. To balance these trade-offs without costly retraining of defense agents, we propose an adaptive test-time policy shaping approach that dynamically switches the active decision-making policy of an ACD agent during incident response. We evaluate our approach in the CybORG CAGE 2 simulation environment, a widely used platform for ACD research Kiely et al. [17] (Sec. 3.1), and study operational trade-offs through the CIA triad (Sec. 3.2). These attributes also serve as proxies for broader human-aligned objectives that may arise in cybersecurity, such as financial cost, mission impact, or service disruption, to which our approach can naturally extend. In the following sections, we also describe our baseline RL agents (Sec. 3.3), present our test-time policy shaping approach for alignment (Sec. 3.4), present our adaptive policy shaping approach (Sec. 3.5), compare our approach against training-time reward shaping baselines (Sec. 3.6), evaluate generalization across increasing network sizes and unseen adversaries (Sec. 3.7), and define our evaluation metrics (Sec. 3.8).

Cyber Defense Simulation Environment

We use the CybORG simulation environment Standen et al. [33], focusing on the CAGE 2 scenario Kiely et al. [17], which models a network of three subnets (Fig. 1). We select CAGE 2 because it is a widely adopted benchmark for autonomous cyber defense and, through the modifications of Cadet et al. [6], enables the study of resilience and CIA-driven operational trade-offs in a realistic attacker–defender setting. We adopt this modified topology, which introduces servers tied to the CIA alignment attributes. The red agent seeks to identify vulnerable hosts, escalate privileges, and execute impact actions that cause service degradation. The blue agent defends through monitoring, removing malware, deploying decoys, and restoring compromised hosts Kiely et al. [17].

We formalize the problem as a partially observable Markov decision process (POMDP) Sutton and Barto [34], treating the attacker as part of the environment dynamics. The POMDP is defined by ⟨S,A,T,O,rCAGE,b0⟩\langle\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{O}, r^{\mathrm{CAGE}}, b_0 \rangle Kiely et al. [17]; ); Le and Stadler [20]. Here, S\mathcal{S} is the state space representing the network configuration, A\mathcal{A} the set of defensive actions (determined by the number of hosts and per-host actions), T(s′∣s,a)\mathcal{T}(s' \mid s,a) the transition function that implicitly captures agent interactions, and O(o∣s,a)\mathcal{O}(o \mid s,a) the observation function providing the blue agent with partial information about host status and detected red agent sessions. The initial belief b0b_0 encodes uncertainty over the starting configuration. The base blue agent reward in CAGE, rCAGE(s,a)∈Rr^{\mathrm{CAGE}}(s,a) \in\mathbb{R}, reflects penalties incurred from red agent compromise and impact actions, as well as blue agent restore actions (i.e., restoring a machine to its original state, which causes one time step of service disruption):

rtCAGE=−0.1(hostR−1)−serverR−BRestore−10(Rimpact).(1)r_t^{\mathrm{CAGE}} = -0.1(\mathrm{host}_{R}-1) - \mathrm{server}_{R} - \mathrm{B}_{\mathrm{Restore}} - 10(\mathrm{R}_{\mathrm{impact}}). \tag*{(1)}

Here, hostR\mathrm{host}_{R} and serverR\mathrm{server}_{R} denote hosts and servers compromised by the red agent RR, respectively, and BRestore\mathrm{B}_{\mathrm{Restore}} is an indicator equal to 1 if the blue agent BB has run the restore action on any host or server, and 0 otherwise, and Rimpact\mathrm{R}_{\mathrm{impact}} is an indicator equal to 1 if the red agent has run the impact action on any host or server Kiely et al. [17]. The maximum possible reward is 0. Importantly, this reward function excludes hosts and servers associated

BaseHosts_2Hosts_3Hosts_4
Total Hosts16314661
User Subnet5101520
Enterprise Subnet471013
Operational Subnet7142128
CIA-Critical Hosts36912
Auth Servers1234
Database Servers1234
Front-End Servers1234
Steps / Episode200300400500
Action Space ∣A∣\lvert\mathcal{A}\rvert178343508673

caption: Network topologies across scenarios of increasing scale. CIA-critical hosts are scored for Confidentiality, Integrity, and Availability impact. The enterprise subnet includes the defender host.

:::

with the CIA attributes, resulting in an environment that enables the study of the trade-off between operational alignment objectives and the original performance-driven reward.

Confidentiality, Integrity, and Availability (CIA) Attribute Scores

Following Cadet et al. [6], we focus on the CIA attributes as domain-specific alignment objectives, which can also be used to collectively characterize a system’s cyber resilience—its ability to resist, absorb, and recover from compromise. We use preservation of these attributes as a domain-specific proxy for broader alignment objectives, grounding our formulation in prior cyber-resilience research. Confidentiality (C) captures unauthorized access to sensitive data, Integrity (I) reflects unauthorized modification of data, and Availability (A) denotes disruption of services for legitimate users. These attributes are mapped to specific network assets: {AS,DS}\{\mathrm{AS}, \mathrm{DS}\} for Confidentiality, {AS,WS}\{\mathrm{AS}, \mathrm{WS}\} for Integrity, and {AS,DS,WS}\{\mathrm{AS}, \mathrm{DS}, \mathrm{WS}\} for Availability. To quantify each attribute’s score, we adapt the network resilience formulation of Cadet et al. [6], which defines an indicator Nj(t)=1N_j(t)=1 if an adversarial impact on critical asset jj is successful at time tt, and Nj(t)=0N_j(t)=0 otherwise. Each asset jj then carries a cost cost⁡(i,j)\operatorname{cost}(i,j) representing the penalty to attribute i∈{C,I,A}i \in\{\mathrm{C},\mathrm{I},\mathrm{A}\}. We then compute an attribute drop score over a time window Δt\Delta t as the cumulative impact on assets associated with each attribute; for example, the confidentiality drop is defined as

C(B,R,Δt)=∑j∈{AS,DS}(∑t∈ΔtNj(t))⋅cost⁡(C,j),(2)C(B,R,\Delta t)=\sum_{j\in\{\mathrm{AS},\mathrm{DS}\}}\left(\sum_{t\in\Delta t}N_j(t)\right)\cdot\operatorname{cost}(\mathrm{C},j), \tag*{(2)}

where BB and RR denote the blue and red agent actions, respectively. The resilience drop is then defined as the weighted sum of the {C,I,A}\{\mathrm{C},\mathrm{I},\mathrm{A}\} drop scores. Formally, this is defined as the following:

ΔResilienceDrop(B,R,Δt)=∑i∈{C,I,A}wi∑j∈{AS,DS,WS}(∑t∈ΔtNj(t))⋅cost⁡(i,j).(3)\Delta ResilienceDrop(B,R,\Delta t)=\sum_{i\in\{\mathrm{C},\mathrm{I},\mathrm{A}\}}w_i\sum_{j\in\{\mathrm{AS},\mathrm{DS},\mathrm{WS}\}}\left(\sum_{t\in\Delta t}N_j(t)\right)\cdot\operatorname{cost}(i,j). \tag*{(3)}

Whereas Cadet et al. [6] use an indicator of a successful adversarial impact to measure resilience loss, we invert this indicator to measure the “damage prevented” as a proxy for alignment with the CIA attributes. Specifically, we set Nj(t)=1N_j(t)=1 if an adversarial impact on critical asset jj is prevented or resolved at time tt (e.g., the blue agent restores a compromised host), and Nj(t)=0N_j(t)=0 otherwise (e.g., a host was not impacted by the red agent, or the red agent compromises the host and the blue agent fails to restore it). Under this formulation, actions that prevent or remediate red impacts to an asset, such as restoring a compromised host, contribute positively to the corresponding CIA attribute. Similarly, an agent may remove an active malicious process to keep a host online rather than fully restoring it from a clean image, thereby preserving availability. However, this strategy requires monitoring the host to identify the malicious process before it can be removed.

We adopt a multi-objective formulation treating alignment to these attributes and the CAGE reward as distinct objectives. To this end, since defending CIA-associated assets may divert attention from other parts of the network, we construct a modified CAGE scenario that permits red impacts on all hosts and removes CIA-critical hosts from the blue agent reward calculation entirely. This decoupling prevents unintended correlation between alignment and reward, more closely reflecting real-world settings where cyber defenders must balance multiple competing priorities.

Autonomous Cyber Defense RL Agents

Following prior work [17], we train blue agents in the CybORG CAGE 2 environment using the Proximal Policy Optimization (PPO) algorithm [31]. The actor and critic are represented by two feedforward neural networks, each with two hidden layers of 256 units. Each blue agent is trained against a red agent in the base scenario described in § 3.2, using 200-step games in which the blue and red agents each take an action at every step. We train each agent for 500k steps with a learning rate of 0.003, a discount factor of 0.99, and a mini-batch size of 1024 for the SGD optimizer. Agent code, including seeds to reproduce results, is provided in our codebase.

CybORG is packaged with several different heuristic red and blue agents, which we rely upon for training and evaluation. The base Red Meander agent has no knowledge of the network topology and randomly navigates the network, looks for vulnerabilities, and attempts to exploit hosts on the network until it is able to impact a host with the operational technology (OT) service present. For our purpose of training blue agents, we create a new Red Combination agent which switches randomly with probability p=0.5p=0.5 between a policy to impact CAGE-specific hosts (i.e., hosts not tied to CIA attributes) and a policy to impact alignment-specific CIA hosts (i.e., the database, front-end, and authentication servers). Other agents available for testing are described in § 3.7.

Alignment with Test-Time Policy Shaping

Unlike prior work that measures agent success primarily through environment reward or resilience metrics [17, 6], we explicitly model operational objectives as alignment signals that may conflict with reward maximization. Our approach builds on the test-time policy shaping framework of Mujtaba et al. [26], in which the resulting agent policy π\pi is a convex combination of two or more action distributions. Such ensemble-based methods have been shown to improve robustness in decision-making settings [22, 9, 26].

Given a pretrained RL policy πRL\pi_{RL} and an alignment-driven policy πalign\pi_{align}, the shaped policy is defined as:

π(at∣ot)=(1−αt) πRL(at∣ot)+αt πalign(at∣ot)(4)\pi(a_t \mid o_t) = (1-\alpha_t)\,\pi_{RL}(a_t \mid o_t) + \alpha_t\,\pi_{align}(a_t \mid o_t) \tag*{(4)}

where, at time tt, at∈ABluea_t \in\mathcal{A}^{Blue} is a blue agent action and ot∈Oo_t \in\mathcal{O} is the blue agent’s partial observation of the network. The interpolation parameter αt∈[0,1]\alpha_t \in[0,1] controls the degree of attribute-based policy shaping: αt=1\alpha_t = 1 yields full control by the alignment policy, while αt=0\alpha_t = 0 recovers the base RL agent. Varying αt\alpha_t traces the Pareto front between reward maximization and alignment adherence.

To construct the alignment policy πalign\pi_{align}, we compute per-action CIA attribute scores over the blue agent action space ABlue\mathcal{A}^{Blue}. Let {ki}i=1M\{k_i\}_{i=1}^{M} denote the alignment-relevant attributes, with ki∈{C,I,A}k_i \in\{\mathrm{C},\mathrm{I},\mathrm{A}\}. For an action at∈ABluea_t \in\mathcal{A}^{Blue} under observation oto_t, we form an attribute-score vector z(at∣ot)∈RM\mathbf{z}(a_t \mid o_t) \in\mathbb{R}^{M} whose ii-th component measures the one-step, cost-weighted contribution of action ata_t toward attribute kik_i, computed only over the assets associated with that attribute (e.g., {AS,DS}\{\mathrm{AS}, \mathrm{DS}\} for confidentiality).

We define the target (ideal) vector τ∈RM\boldsymbol{\tau} \in\mathbb{R}^{M} as the maximum attainable score for attribute kik_i (i.e., by performing actions that maximize that particular attribute). Each action is scored by its Euclidean distance to the target: d(at∣ot)=∑i=1M(zi(at∣ot)−τi)2d(a_t \mid o_t) = \sqrt{\sum_{i=1}^{M}(z_i(a_t \mid o_t)-\tau_i)^2}. The alignment policy is then defined as a softmax over negative distances:

πalign(at∣ot)=exp⁡(−d(at∣ot))∑a′∈ABlueexp⁡(−d(a′∣ot))(5)\pi_{align}(a_t \mid o_t) = \frac{\exp(-d(a_t \mid o_t))} {\sum_{a' \in\mathcal{A}^{Blue}} \exp(-d(a' \mid o_t))} \tag*{(5)}

This formulation assigns higher probability to actions closer to the desired attribute target while maintaining exploration among similarly aligned actions. For example, under a confidentiality objective, actions that restore the authentication and database servers or remove malicious processes from them may receive higher probability, whereas under an availability objective, actions that maintain critical hosts may be preferred. Although we demonstrate this approach with an on-policy RL agent, the interpolation framework applies equally to off-policy settings [26].

Alignment with Adaptive Policy Shaping

During deployment, the relative priorities of reward maximization and alignment adherence may change as the state of the network evolves. Rather than relying on a fixed interpolation parameter α\alpha, we propose an adaptive test-time policy shaping mechanism that sets αt\alpha_t as a deterministic, rule-based function of the current observation: αt=h(ot)\alpha_t = h(o_t). Given the blue agent’s partial observation, h(ot)h(o_t) flags the state as unsafe if any alignment-relevant host is currently impacted; if so, we set αt=1\alpha_t = 1 to prioritize the alignment policy, and otherwise set αt=0\alpha_t = 0 to revert to the reward-maximizing policy. We use this binary rule as an interpretable, training-free mechanism that isolates the benefit of state-dependent policy shaping without introducing an additional learned controller or associated training assumptions. We also note that, while this effectively amounts to policy switching, our approach can generalize to other h(ot)h(o_t) functions that directly predict context-dependent αt\alpha_t values from observations. The resulting action distribution is then computed via the same convex combination of the reward- and alignment-driven action probabilities, with αt\alpha_t determined by h(ot)h(o_t).

Alignment with Training-Time Reward Shaping

As a baseline comparison to our test-time policy shaping approach, we investigate alignment using training-time reward shaping. This approach modifies the reward function to include an alignment-dependent bonus term, weighted by a coefficient λ\lambda relative to the original CAGE reward, which encourages actions consistent with a specified alignment objective during training. We adopt a bonus-based formulation rather than a weighted sum of independent reward components, since alignment-relevant events (i.e., undesirable impacts to CIA-critical hosts) occur infrequently during training, making an additive bonus more effective. The shaped reward function then becomes:

rt=r~tCAGE+λrtalign(k),k∈{C,I,A},(6)r_t = \tilde{r}^{\mathrm{CAGE}}_t + \lambda r^{\mathrm{align}}_t(k), \qquad k \in\{C,I,A\}, \tag*{(6)}

where r~tCAGE\tilde{r}^{\mathrm{CAGE}}_t is the normalized CAGE environment reward for the blue agent (Eq. (1)), and rtalign(k)r^{\mathrm{align}}_t(k) is an alignment bonus derived from the attribute score (Eq. (2)), which is 1 if the selected action improves the attribute kk and 0 otherwise. For a given target attribute kk, the alignment term is computed from the observed red and blue agent actions and observations at time tt and scaled by the maximum attainable attribute score. To improve training stability and convergence, we normalize the CAGE reward signal using a warm-up period of ten episodes to estimate the running mean and standard deviation, after which rewards are normalized and clipped before being combined with the alignment bonus. PPO agents trained with reward shaping use the same hyperparameters as the baseline.

Generalization and Scalability

A central challenge in autonomous cyber defense is generalization from simulated environments to real-world environments, which often involve larger network topologies, more diverse adversary behaviors, and shifting defensive priorities. To study this in a more controlled setting, we develop a network topology generator for CybORG that enables systematic variation in subnet size and reward weighting. We construct four scenarios – base (the original CAGE 2 topology with 16 hosts), hosts-2 (31 hosts), hosts-3 (46 hosts), and hosts-4 (61 hosts) – each scaling the number of hosts per subnet while proportionally increasing the training budget and the number of steps per episode (see Tab. 1). As network size grows, the number of CIA-critical hosts increases, making sustained adversarial impact more likely and cyber defense more challenging.

We evaluate the robustness of blue agents against several different red agents: a Meander CIA Agent (M-CIA) that randomly navigates the network while targeting CIA-critical hosts; a Meander CAGE Agent (M-CAGE) that targets hosts associated with the CAGE reward but not CIA attributes; and B-line Agents (Bline-C/I/A) that possess prior knowledge of the network topology and can consistently target assets associated with a specific attribute, resulting in more persistent and challenging attack patterns.

Together, these scenarios and red agent types allow us to assess whether alignment-driven defenses generalize across network scales and remain robust under diverse adversarial conditions. Our new CAGE 2 environment is also available for use in our code repository.

Evaluation Metrics

For all experiments, we evaluate both agent performance and operational alignment. The CAGE environment reward (Eq. (1)) measures overall defensive performance, where the maximum achievable reward is 0. We also define an attribute-specific alignment score, which measures the fraction of decision steps in which the blue agent selects an action aligned with a target operational attribute ki∈{C,I,A}k_i \in\{\mathrm{C}, \mathrm{I}, \mathrm{A}\}. Formally, let ata_t denote the action selected by the blue agent at time step tt, and let Aki(ot)⊆ABlueA_{k_i}(o_t) \subseteq\mathcal{A}^{\mathrm{Blue}} denote the set of actions that are aligned with attribute kik_i under observation oto_t. The alignment score for attribute kik_i is then defined as 1T∑t=1T1[at∈Aki(ot)]\frac{1}{T}\sum_{t=1}^{T} \mathbf{1}\left[a_t \in A_{k_i}(o_t)\right], where TT is the episode length and 1[⋅]\mathbf{1}[\cdot] is the indicator function of a selected aligned action.

We also measure overall network resilience drop (§ 3.2) using the weighting scheme of [6] (wC=0.6w_C = 0.6, wI=wA=0.2w_I = w_A = 0.2), and assume equal asset costs. Although these parameters are application-dependent, our focus is on evaluating alignment methods rather than tuning application-specific weights. Agents that successfully prioritize assets associated with the target operational objective achieve lower resilience-drop scores.

Results and Discussion

We evaluate our alignment approaches for ACD agents across four sets of experiments. First, we establish baseline performance of blue monitor (BM) and PPO agents against adversaries with differing operational objectives (§ 4.1). Second, we compare training-time reward shaping (RL-λ\lambda) and test-time policy shaping (RL-α\alpha) across varying λ\lambda and α\alpha values, assessing their effectiveness in balancing alignment and reward (§ 4.2 and § 4.3). Third, we evaluate adaptive policy shaping, which dynamically adjusts alignment priorities to mitigate the trade-off between alignment and reward (§ 4.4). Finally, we assess the generalization and scalability of all alignment approaches under held-out adversaries and increasing network sizes (§ 4.5).

All experiments ran on a single NVIDIA RTX A6000 GPU. For each experiment, five blue agents were trained using different seeds and evaluated on held-out evaluation seeds. The results presented are aggregated across these five trained agents. Seeds and parameters are available in our code repository, with RL training curves provided in the Appendix.

Baseline Blue Agent Performance

The results across different blue agents and red adversaries on the base scenario (see Tab. 1) are presented in Figure 2 for the M-CIA agent that targets at random a CIA asset. The Blue Monitor agent provides a lower bound on the reward, as it only monitors the network and takes no defensive actions. The PPO agent is trained to maximize the CAGE reward and achieves the highest reward by protecting the operational server, but fails to defend CIA-critical hosts. This results in a significant resilience drop, comparable to the BM agent. This gap between ACD agent performance and CIA impact motivates the use of the three alignment approaches: test-time policy shaping (§ 3.4), training-time reward shaping (§ 3.6), and adaptive policy shaping (§ 3.5). These approaches extend the baseline PPO policy to reduce CIA impact while preserving as much reward advantage as possible.

Figure 2. Per-step comparison of blue agent performance against the red M-CIA agent on the base scenario. Each panel shows the Confidentiality drop, Resilience drop, and CAGE reward (mean ± SEM across 100 episodes, each 200 steps); values closer to zero indicate better performance for all metrics. Blue monitor provides a lower bound on performance across metrics; in this baseline, the agent only monitors the network and takes no defensive actions. PPO is the unaligned baseline agent. TT (α=0.5\alpha=0.5) and TT (α=1.0\alpha=1.0) apply test-time policy shaping to interpolate between the Confidentiality-aligned policy and the fixed PPO policy. TR (λ=3.0\lambda=3.0) is trained with a Confidentiality alignment bonus in the reward. Adaptive dynamically switches between the fixed PPO and Confidentiality-aligned policies based on agent observations. All three alignment approaches substantially reduce Confidentiality drop relative to the unaligned PPO baseline.

Training-Time Reward Shaping

A standard approach for steering agents toward different objectives is to shape the reward signal during training. In alignment settings, this can consist of providing a “bonus” for taking actions that align with specific attributes or human-specified objectives [11]. Therefore, we first study the application of alignment at training time to provide a comparison with our test-time approaches. The primary disadvantage of training-time alignment is that, for every change in operational priorities or human values, the agent must be retrained, which can be both time-consuming and computationally expensive.

Results for training-time alignment are illustrated in Figure 3, with additional results and training curves provided in the Appendix. We compare varying values of λ\lambda across different operational priorities, and observe that as λ\lambda increases, the agent learns to better protect CIA-aligned assets while maintaining relatively consistent reward performance. The primary challenge with this approach is that a new agent must be retrained whenever operational priorities change, which can become increasingly time-consuming as network size grows (as illustrated later in § 4.5). Furthermore, training on a specific adversary (e.g., M-CIA) can lead to poorer performance against unseen adversaries (e.g., M-CAGE, which targets CAGE-only assets); this is illustrated in Appendix Tabs. S1–S4, where in the case of attribute “A” (i.e., all CIA assets are prioritized) and λ=3.0\lambda= 3.0, the training-time agent’s reward is worse than the baseline PPO agent across all scenarios. This reflects the trade-off associated with prioritizing the defense of one set of operational objectives over another.

Figure 3. Comparison of scores (mean ±\pm SEM across 100 episodes, against M-CIA red) as the alignment bonus λ\lambda is increased in training-time reward shaping for each CIA attribute. Across all attributes, increasing λ\lambda improves CIA drop scores and reward.

Test-Time Policy Shaping to Operational Priorities

To efficiently and flexibly steer the operational priorities of ACD agents, we propose test-time policy shaping and its adaptive variant as alternatives to training-time alignment. Figure 4 compares test-time policy shaping with training-time alignment and baseline agents against the M-CIA red agent, with additional results in the Appendix. We consider two settings: α=0.5\alpha= 0.5, which assigns equal weight to the original PPO reward-driven and alignment-driven policies, and α=1.0\alpha= 1.0, which uses only the alignment-driven policy. Across all red agent types, test-time policy shaping (α=1.0\alpha= 1.0) achieves the strongest alignment while reducing attribute-associated impacts and resilience drops, albeit at the cost of lower reward. Figure 5 illustrates this trade-off: increasing α\alpha places greater weight on CIA-aligned actions while decreasing agent reward. By contrast, training-time reward shaping (λ=3.0\lambda= 3.0) often achieves the best impact scores, with smaller reductions in reward and alignment scores. Intermediate test-time (α=0.5\alpha= 0.5) and training-time (λ=1.5\lambda= 1.5) settings interpolate between baseline PPO and fully attribute-aligned policies.

Figure 4. Alignment score, reward, and resilience drop of alignment approaches against the M-CIA red agent. Each point shows the mean performance of the method listed on the left, averaged across five seeds with 100 episodes of 200 steps each. Horizontal bars represent the standard deviation across seeds; this variation is very small for all approaches except BM.

Figure 5. Comparison of scores (mean ± SEM across 100 episodes, against M-CIA red) as α\alpha varies in test-time policy shaping for each CIA attribute. As α\alpha increases, alignment improves substantially, but reward decreases.

Figure 6 (top row) further quantifies the test-time alignment-reward trade-off as α\alpha varies. Alignment scores for all three CIA attributes scale nearly linearly with α\alpha and are balanced across CIA at every network scale tested. While the magnitude of the reward changes with scale (reflecting the increased difficulty in defending larger networks), the reward cost remains convex across scales: moderate α\alpha values offer favorable cost-benefit ratios, while α=1.0\alpha= 1.0 guarantees full compliance at a steeper penalty, allowing human operators to tune α\alpha at deployment time without retraining. These trade-offs motivate an adaptive approach that can alternate between the competing objectives.

Figure 6. Test-time policy shaping (top row) and training-time reward shaping (bottom row) results for a blue agent facing the red M-CIA agent in the base (left column) and Hosts3Hosts_3 (right column) scenarios. The results show a trade-off between alignment scores and CAGE reward as α\alpha and λ\lambda vary. Colored bars show alignment scores (left axis) for the Confidentiality (C), Integrity (I), and Availability (A) attributes; the red line shows the average CAGE reward (right axis). For test-time policy shaping, alignment scores consistently improve as network size and α\alpha increase. In contrast, training-time reward shaping maintains reward alongside alignment as λ\lambda increases, but its alignment scores collapse as network size grows.

Adaptive Alignment under Performance Trade-offs

Our adaptive policy shaping approach is demonstrated in Figure 7 for all CIA attributes. This agent uses current observations to infer the red M-CIA agent’s attack strategy and switches between defending the operational server (preserving CAGE reward) and protecting specific CIA-critical hosts (reducing CIA resilience drops). Across these objectives, the drops in Confidentiality and Integrity are mitigated, whereas the Availability drop is more difficult for the agent to counter, likely because Availability is linked to all three CIA-critical hosts, thereby expanding the scope of cyber defense required.

Figure 7. Adaptive blue agent performance against the M-CIA red agent in the base scenario (mean ±\pm SEM across 100 episodes). Rows show Confidentiality-, Integrity-, and Availability-aligned agents using adaptive α\alpha. C- and I-agents reduce their target CIA drops to near zero within 20 steps; suppression by the A-agent is more variable and less complete.

To place these results in context, Figure 4 compares the adaptive approach with the other alignment approaches across CIA objectives and reward against the M-CIA red agent, while Figure 2 illustrates the alignment–reward trade-off for Confidentiality in the base scenario. The unaligned PPO baseline achieves the highest reward but has substantial drops in both Confidentiality and overall alignment as the red agent compromises CIA-critical hosts. Test-time policy shaping, in contrast, progressively reduces Confidentiality drop as α\alpha increases, at the cost of reward, while training-time reward shaping achieves comparable suppression with higher reward. The adaptive agent achieves suppression of Confidentiality loss similar to test-time policy shaping with α=1.0\alpha= 1.0 by activating alignment only after detecting compromise of Confidentiality-associated assets. Behavioral analysis shows that the adaptive agent disrupts the red agent’s attack chain by prioritizing CIA-critical hosts, preventing the lateral movement needed to reach downstream operational servers. Although all three alignment approaches substantially reduce Confidentiality impact relative to the PPO baseline, resilience is not fully recovered because the red agent can still compromise Integrity- and Availability-associated assets that are not explicitly defended under a Confidentiality objective. Additional alignment results are in the Appendix.

The reward cost of fixed-α\alpha test-time policy shaping also depends strongly on the adversary’s strategy, as illustrated by the Pareto fronts and estimated knee-points in Figure 8. Under the red M-CIA agent, which directly targets CIA-tied hosts, the Pareto front is sharply convex and knee-points occur at moderate α\alpha values, beyond which reward degrades steeply. Under the red M-CAGE agent, whose attacks are not CIA-targeted, the Pareto front is substantially flatter, and knee-points shift to higher α\alpha values, allowing the defender to sustain strong alignment with modest reward loss. This contrast indicates that the cost of alignment depends on the adversary’s attack pattern: when the red agent directly targets CIA-tied hosts, the defender must sacrifice more operational reward to maintain the same level of CIA protection. This higher reward cost motivates activating the alignment-driven policy selectively in response to observed attacks, as in our adaptive approach.

Figure 8. Pareto front showing the trade-off between CAGE reward and alignment scores for test-time policy shaping agents on the base scenario. Circles mark knee-points for the CIA attributes (α∗\alpha^{*}). The red M-CIA agent (right) produces a sharply convex trade-off with knee-points at α≈0.75\alpha\approx0.75-0.790.79, while the red M-CAGE agent (left) yields a flatter curve with knee-points at α≈0.65\alpha\approx0.65-0.770.77, reflecting a lower alignment cost when the red agent does not directly target CIA-critical hosts.

This adaptive α\alpha improves the reward–alignment trade-off relative to fixed-α\alpha test-time policy shaping at high α\alpha values by avoiding a constant reward cost. Relative to an intermediate fixed setting (α=0.5\alpha= 0.5), however, the gain in aggregate alignment score is modest (≈0.52\approx0.52 versus ≈0.51\approx0.51 in the base scenario). This modest gain is consistent with the reactive nature of h(ot)h(o_t), which raises αt\alpha_t only once a CIA-critical host has already been impacted. Before this switch, the agent follows the unshaped RL policy. This also introduces a response delay: if the red agent compromises a CIA-tied host at step tt, the blue agent detects this only at step t+1t+1, resulting in a brief unprotected window before the policy switches. The adaptive and fixed-α\alpha agents therefore differ in when they prioritize alignment: the adaptive agent increases its alignment weight after detecting impact, whereas the fixed-α\alpha agent applies the same weight throughout the episode. These results suggest that improving the policy switching condition in h(ot)h(o_t) could yield further gains, motivating anticipatory rules that raise αt\alpha_t on early indicators of compromise rather than on impact alone.

Generalization to Unseen Adversaries and Network Topologies

To evaluate whether alignment approaches generalize beyond the base scenario (16 hosts), we train separate PPO agents on three scenarios of increasing network size: Hosts_2\mathit{Hosts\_2} (31 hosts), Hosts_3\mathit{Hosts\_3} (46 hosts), and Hosts_4\mathit{Hosts\_4} (61 hosts), each against the red M-CIA agent (Tab. 1). Episode duration is scaled proportionally from 200 to 500 steps to accommodate the additional time required by the red agent to traverse larger networks. Furthermore, we also test if alignment approaches generalize to other red agents (i.e., the B-line variants) that were not seen during training.

Alignment as network size increases: Figure 9 compares alignment scores and reward across four network scales (from 16 to 61 hosts). Reward decreases with increasing scale, indicating the challenge of defending larger networks, and the drop is steeper for the adaptive and test-time policy shaping agents. As the standard CAGE reward function does not incentivize protecting CIA-critical hosts, the baseline PPO agent achieves zero alignment scores (not shown). Agents aligned with training-time reward shaping achieve relatively high alignment scores (≈0.91\approx0.91) in the base scenario; however, we observe alignment rapidly degrade at the largest network scales. In contrast, the test-time policy shaping (≈1.0\approx1.0) and adaptive policy shaping (≈0.51\approx0.51-0.520.52) agents maintain relatively stable alignment across all four network sizes. We hypothesize that this may be due to gradient collapse from the alignment reward signal during training. As the network grows, the agent must learn the alignment objective over a larger state and action space, potentially making the alignment reward increasingly sparse or diluted relative to the task reward. In contrast, adaptive policy shaping does not require the agent to learn this trade-off during training, but instead directly increases the influence of the alignment policy when an alignment-relevant impact is observed. While training-time reward shaping learns effective CIA-aligned policies for the base scenario, the alignment reward collapses toward zero and the agents revert to near-unaligned behavior at larger network sizes, making the adaptive and test-time policy shaping approaches more effective at these scales. This trend is further illustrated in Figure 6, which compares test-time alignment across α\alpha (top row) and training-time alignment across λ\lambda (bottom row) for the base scenario (left column) and the hosts-3 scenario (right column). In the base scenario, increasing either alignment parameter improves the alignment score, with the training-time approach achieving higher task reward than the test-time approach. However, in the larger hosts-3 scenario, the training-time approach fails to learn effective CIA-aligned policies, resulting in substantially lower alignment scores despite maintaining higher task reward. In contrast, the test-time approach continues to improve alignment as α\alpha increases, demonstrating greater robustness to increasing network size.

Figure 9. Alignment scores (bars, left axis) and CAGE rewards (lines, right axis) for the baseline PPO, test-time policy shaping (α=0.5\alpha= 0.5, α=1.0\alpha= 1.0), training-time reward shaping (λ=1.5\lambda= 1.5, λ=3.0\lambda= 3.0), and adaptive blue agents. (Left) Results against variants of the red B-line agent, which target CIA-critical hosts using a direct attack path. CAGE rewards are averaged across the Confidentiality, Integrity, and Availability attributes. (Right) Results against the red M-CIA agent across four network scales.

Alignment across targeted red agents: To test generalization to unseen adversaries, we evaluate the performance and alignment of blue agents against various B-line red agents, which target CIA-critical hosts using a direct attack path. We note that blue agents were not trained against any of the red B-line variants. Figure 9 (left) shows alignment scores and reward against each B-line variant in the base scenario, using mid-point settings for training-time alignment (λ=1.5\lambda= 1.5) and test-time alignment (α=0.5\alpha= 0.5), to study trade-offs between the two approaches. Across all methods, reward values remain somewhat consistent across red B-line variants. Test-time policy shaping (≈0.51\approx0.51) and adaptive policy shaping (≈0.52\approx0.52) maintain consistent alignment scores across all three variants. In addition, training-time reward shaping achieves slightly higher alignment scores across B-line red agents (≈0.69\approx0.69-0.740.74) while maintaining a similar reward to PPO across variants. The baseline PPO agent again achieves zero alignment in all cases, confirming that standard RL training provides no CIA protection under direct CIA-targeted attacks. In terms of reward, test-time policy shaping yields the lowest values among the three approaches. In contrast, the adaptive approach achieves substantially higher reward across novel red adversaries while maintaining similar alignment scores, offering a balanced trade-off without requiring retraining.

Conclusion

We formulate alignment of autonomous cyber defense agents as test-time policy shaping over a fixed RL policy, comparing this approach with training-time reward shaping and a new adaptive policy shaping approach. To study this, we focus on alignment to three critical information security attributes or objectives: Confidentiality, Integrity, and Availability. We show that the alignment-reward trade-off is convex and scale-invariant. Our analysis suggests that an ACD agent employing the adaptive alignment approach offers a balanced and practical solution to this trade-off. Notably, alignment based on training-time reward shaping collapses at scale – this result underscores that alignment objectives incorporated during training do not guarantee alignment at deployment. Future work includes learning functions that map observations to αt\alpha_t values for adaptive policy shaping, learning aligned policies from human feedback, and extending our work to multi-attribute alignment with connections to multi-objective reinforcement learning.

Acknowledgments

This material is based upon work supported by the Defense Advanced Research Projects Agency and the Air Force Research Laboratory, contract number(s): FA8650-23-C-7316. Any opinions, findings and conclusions, or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of AFRL or DARPA.

CybORG CAGE 2 Scenario Action Space

Figure S1 shows the set of defensive actions for the blue agent and the ordered list of actions along the attack chain for the red agent.

Figure S1. Available red and blue agent actions in the simulated CybORG cyber defense environment.

Blue Agent Performance Across Network Scales

Tabs. S1–S4 report blue agent performance against the M-CIA, M-CAGE, and B-line red agents across four network scales. Each table reports per-attribute alignment, reward and resilience drop for all alignment methods using the Meander and B-line red agents. From the values presented in these scaled scenario tables, we are able to see how training-time alignment deteriorates as network size increases, while test-time policy shaping maintains consistent alignment across all scales.

Red AgentMetricBMPPOTest-time α=0.5\alpha= 0.5Test-time α=1.0\alpha= 1.0Training-time λ=1.5\lambda= 1.5Training-time λ=3.0\lambda= 3.0Adaptive α∈{0,1}\alpha\in\{0,1\}
M-CIACAlignment (↑\uparrow)0.000.000.531.000.530.900.52
M-CIACReward (↑\uparrow)-3199.12-587.69-991.39-3648.42-657.89-723.16-969.98
M-CIACRes. Drop (↓\downarrow)0.610.610.290.270.280.270.28
M-CIAIAlignment (↑\uparrow)0.000.000.521.000.530.940.52
M-CIAIReward (↑\uparrow)-3199.12-587.69-978.30-3575.89-635.79-653.04-903.32
M-CIAIRes. Drop (↓\downarrow)0.610.610.530.510.570.550.54
M-CIAAAlignment (↑\uparrow)0.000.000.531.000.460.890.52
M-CIAAReward (↑\uparrow)-3199.12-587.69-1946.82-5492.42-1007.24-1145.55-1841.96
M-CIAARes. Drop (↓\downarrow)0.610.610.150.090.110.010.08
M-CAGE†^\daggerCReward (↑\uparrow)-3937.68-729.19-804.28-2963.76-723.64-774.40-731.28
M-CAGE†^\daggerIReward (↑\uparrow)-3937.68-729.19-810.06-2973.65-754.40-722.87-716.25
M-CAGE†^\daggerAReward (↑\uparrow)-3937.68-729.19-816.92-2974.32-928.58-955.00-723.86
Bline-CCAlignment (↑\uparrow)0.000.000.521.000.690.970.53
Bline-CCReward (↑\uparrow)-4039.64-316.63-441.78-3383.29-294.66-375.10-368.28
Bline-CCRes. Drop (↓\downarrow)0.850.830.280.160.080.010.17
Bline-IIAlignment (↑\uparrow)0.000.000.521.000.740.970.53
Bline-IIReward (↑\uparrow)-4039.64-294.70-445.35-3368.02-319.05-304.20-396.33
Bline-IIRes. Drop (↓\downarrow)0.680.660.220.130.060.010.13
Bline-AAAlignment (↑\uparrow)0.000.000.521.000.690.980.53
Bline-AAReward (↑\uparrow)-4039.44-296.31-427.11-3339.64-301.79-346.85-375.05
Bline-AARes. Drop (↓\downarrow)0.650.630.210.120.070.000.13

Table S1. Table S1: Performance analysis of various blue and red agents on the base scenario (16 hosts, 200 steps, 100 episodes), averaged over five training seeds. Specifically, BM is a Blue Monitor agent that monitors the network only and takes no defensive actions. PPO is the unaligned baseline agent. Test-time is the agent aligned with the test-time policy shaping value α\alpha. Training-time is the agent aligned with the training-time reward shaping value λ\lambda. Adaptive is the adaptive policy shaping agent. Metrics are quantified with respect to CIA attributes and include Alignment: fraction of time steps where the agent chose the C/I/A-optimal action; Reward: overall CAGE environment reward; Res. Drop: weighted resilience drop across all CIA attributes. †^\daggerM-CAGE only targets the operational server (which has no CIA scoring), so only the reward is reported.

Red AgentMetricBMPPOTest-time α=0.5\alpha= 0.5Test-time α=1.0\alpha= 1.0Training-time λ=1.5\lambda= 1.5Training-time λ=3.0\lambda= 3.0Adaptive α∈{0,1}\alpha\in\{0,1\}
M-CIACAlignment (↑)0.000.000.531.000.310.720.52
M-CIACReward (↑)-5483.17-1053.83-2042.26-7149.68-1336.29-1537.66-1975.27
M-CIACRes. Drop (↓)0.590.60-0.300.300.310.310.30
M-CIAIAlignment (↑)0.000.000.531.000.290.890.51
M-CIAIReward (↑)-5483.17-1053.83-1973.99-7285.76-1413.32-1508.95-1994.00
M-CIAIRes. Drop (↓)0.590.600.590.590.610.610.60
M-CIAAAlignment (↑)0.000.000.521.000.200.870.52
M-CIAAReward (↑)-5483.17-1053.83-5610.03-15365.54-1960.21-2513.29-5353.18
M-CIAARes. Drop (↓)0.590.600.140.080.230.020.08
M-CAGE†CReward (↑)-5996.48-1406.91-1475.90-4661.63-1320.04-1333.37-1341.90
M-CAGE†IReward (↑)-5996.48-1406.91-1415.35-4654.20-1432.73-1448.63-1330.11
M-CAGE†AReward (↑)-5996.48-1406.91-1448.74-4647.87-1434.85-1916.73-1342.83
Bline-CCAlignment (↑)0.000.000.521.000.600.960.52
Bline-CCReward (↑)-5733.15-503.24-653.34-5132.07-589.24-620.76-604.73
Bline-CCRes. Drop (↓)0.870.860.290.170.130.010.17
Bline-IIAlignment (↑)0.000.000.521.000.610.980.52
Bline-IIReward (↑)-5733.15-507.54-652.46-5117.10-627.78-568.27-601.41
Bline-IIRes. Drop (↓)0.700.690.230.140.110.000.14
Bline-AAAlignment (↑)0.000.000.521.000.340.990.52
Bline-AAReward (↑)-5819.76-497.10-696.17-5094.50-514.87-471.68-606.04
Bline-AARes. Drop (↓)0.660.660.220.130.220.000.13

Table S2. Performance analysis of various blue and red agents on the Hosts2_2 scenario (31 hosts, 300 steps, 100 episodes), averaged over five training seeds. Specifically, BM is a Blue Monitor agent that monitors the network only and takes no defensive actions. PPO is the unaligned baseline agent. Test-time is the agent aligned with test-time policy shaping value α\alpha. Training-time is the agent aligned with training-time reward shaping value λ\lambda. Adaptive is the adaptive policy shaping agent. Metrics are quantified with respect to CIA attributes and include Alignment: fraction of time steps where the agent chose the C/I/A-optimal action; Reward: overall CAGE environment reward; Res. Drop: weighted resilience drop across all CIA attributes. †M-CAGE only targets the operational server (which has no CIA scoring), so only reward is reported.

Red AgentMetricBMPPOTest-time α=0.5\alpha= 0.5Test-time α=1.0\alpha= 1.0Training-time λ=1.5\lambda= 1.5Training-time λ=3.0\lambda= 3.0Adaptive α∈{0,1}\alpha\in\{0,1\}
M-CIACAlignment (↑)0.000.000.511.000.050.550.51
M-CIACReward (↑)-8343.21-2102.34-3655.44-11986.94-2762.10-3135.83-3264.31
M-CIACRes. Drop (↓)0.560.590.310.300.360.320.31
M-CIAIAlignment (↑)0.000.000.511.000.040.740.51
M-CIAIReward (↑)-8343.21-2102.34-3460.37-12834.76-2658.48-3093.05-3413.15
M-CIAIRes. Drop (↓)0.560.590.610.590.630.620.62
M-CIAAAlignment (↑)0.000.000.501.000.010.060.51
M-CIAAReward (↑)-8343.21-2102.34-11627.07-29725.86-2052.61-2974.49-11206.41
M-CIAARes. Drop (↓)0.560.590.140.080.580.480.08
M-CAGE†CReward (↑)-8510.52-2335.89-2360.42-6943.90-2217.40-2303.05-2284.57
M-CAGE†IReward (↑)-8510.52-2335.89-2380.96-6925.75-2208.97-2442.47-2278.56
M-CAGE†AReward (↑)-8510.52-2335.89-2385.60-6894.42-2189.82-2528.40-2294.10
Bline-CCAlignment (↑)0.000.000.521.000.250.920.52
Bline-CCReward (↑)-8444.50-1033.15-1034.86-7381.22-1202.52-1254.60-1314.10
Bline-CCRes. Drop (↓)0.880.870.290.180.390.020.18
Bline-IIAlignment (↑)0.000.000.521.000.320.910.52
Bline-IIReward (↑)-8444.50-1103.63-1233.28-7378.89-1174.75-1204.66-1314.53
Bline-IIRes. Drop (↓)0.700.700.230.140.280.020.14
Bline-AAAlignment (↑)0.000.000.521.000.050.100.52
Bline-AAReward (↑)-8483.08-1062.13-1115.72-7312.82-1106.50-1272.37-1304.62
Bline-AARes. Drop (↓)0.690.680.230.140.560.530.13

Table S3. Performance analysis of various blue and red agents on the Hosts_3 scenario (46 hosts, 400 steps, 100 episodes), averaged over five training seeds. Specifically, BM is a Blue Monitor agent that monitors the network only and takes no defensive actions. PPO is the unaligned baseline agent. Test-time is the agent aligned with test-time policy shaping value α\alpha. Training-time is the agent aligned with training-time reward shaping value λ\lambda. Adaptive is the adaptive policy shaping agent. Metrics are quantified with respect to CIA attributes and include Alignment: fraction of time steps where the agent chose the C/I/A-optimal action; Reward: overall CAGE environment reward; Res. Drop: weighted resilience drop across all CIA attributes. †M-CAGE only targets the operational server (which has no CIA scoring), so only reward is reported.

Red AgentMetricBMPPOTest-timeTraining-timeAdaptive
α=0.5\alpha= 0.5α=1.0\alpha= 1.0λ=1.5\lambda= 1.5λ=3.0\lambda= 3.0α∈{0,1}\alpha\in\{0,1\}
M-CIACAlignment (↑\uparrow)0.000.000.501.000.000.060.52
Reward (↑\uparrow)-9247.90-2358.14-4559.83-16991.56-2852.47-4028.11-4332.24
Res. Drop (↓\downarrow)0.630.610.310.310.540.350.32
IAlignment (↑\uparrow)0.000.000.511.000.010.210.52
Reward (↑\uparrow)-9247.90-2358.14-4638.58-16487.74-2973.00-4224.32-4499.55
Res. Drop (↓\downarrow)0.630.610.610.610.600.630.62
AAlignment (↑\uparrow)0.000.000.511.000.000.000.52
Reward (↑\uparrow)-9247.90-2358.14-17833.46-47764.26-2173.33-2938.43-17073.98
Res. Drop (↓\downarrow)0.630.610.140.080.610.610.08
M-CAGE†^{\dagger}CReward (↑\uparrow)-10253.23-2910.22-2694.31-8149.23-2952.69-2783.82-2629.99
IReward (↑\uparrow)-10253.23-2910.22-2721.87-8115.50-2804.95-2853.31-2663.83
AReward (↑\uparrow)-10253.23-2910.22-2730.70-8145.74-2449.88-3320.91-2610.66
Bline-CCAlignment (↑\uparrow)0.000.000.511.000.020.270.52
Reward (↑\uparrow)-9506.02-1083.17-1325.20-8442.14-1202.86-1045.29-1318.83
Res. Drop (↓\downarrow)0.880.880.290.180.820.380.18
Bline-IIAlignment (↑\uparrow)0.000.000.511.000.060.540.52
Reward (↑\uparrow)-9506.02-1106.57-1305.74-8555.80-1141.21-1267.65-1279.65
Res. Drop (↓\downarrow)0.700.700.230.140.540.140.14
Bline-AAAlignment (↑\uparrow)0.000.000.511.000.000.000.52
Reward (↑\uparrow)-9408.34-1014.99-1593.43-8594.74-1252.23-1141.46-1318.22
Res. Drop (↓\downarrow)0.690.690.230.140.670.670.14

Table S4. Performance analysis of blue and red agents on the Hosts_4 scenario (61 hosts, 500 steps, 100 episodes), averaged over five training seeds. Specifically, BM is a Blue Monitor agent that monitors the network only and takes no defensive actions. PPO is the unaligned baseline agent. Test-time is the agent aligned with test-time policy shaping value α\alpha. Training-time is the agent aligned with training-time reward shaping value λ\lambda. Adaptive is the adaptive policy shaping agent. Metrics are quantified with respect to CIA attributes and include Alignment: fraction of time steps where the agent chose the C/I/A-optimal action; Reward: overall CAGE environment reward; Res. Drop: weighted resilience drop across all CIA attributes. †^{\dagger}M-CAGE only targets the operational server (which has no CIA scoring), so only reward is reported.

Training-Time Reward Shaping Curves

Figs. S2 and S3 show training curves for CAGE reward and alignment reward, respectively, across all four network scales, illustrating how training-time alignment reward convergence degrades with increasing network size.

Figure S2. CAGE reward during training for training-time aligned agents across network sizes (rows) and alignment attributes (columns). Colored curves show agents trained with reward shaping at λ∈[0.5,3.0]\lambda\in[0.5, 3.0]. The gray dashed line shows the PPO baseline (no alignment) for reference. The Blue Monitor baseline reward is annotated with “BM = x ↓”.

Blue Agent Gameplay Results

Figs. S4–S6 compare per-step confidentiality, integrity, and availability drop, resilience drop, and CAGE reward traces across PPO, test-time (α=0.5\alpha=0.5 and α=1.0\alpha=1.0), training-time (λ=1.0\lambda=1.0 and λ=3.0\lambda=3.0, and adaptive agents on the base scenario. Figure S7, Figure S8, and Figure S9 present per-step gameplay traces for the adaptive agent in the three larger network scenarios, showing how CIA impact and resilience evolve over episode steps.

Figure S7. Adaptive policy shaping gameplay on the Hosts_2 scenario (31 hosts, 300 steps).

Figure S8. Adaptive policy shaping gameplay on the Hosts_3 scenario (46 hosts, 400 steps).

Figure S9. Adaptive policy shaping gameplay on the Hosts_4 scenario (61 hosts, 500 steps).

Figure S3. Alignment reward during training for training-time aligned agents across network sizes (rows) and alignment attributes (columns). Colored curves show agents trained with alignment reward shaping at λ∈[0.5,3.0]\lambda\in[0.5, 3.0]. The gray dashed line shows the PPO baseline (zero alignment reward by construction).

Figure S4. Comparing alignment approaches for the Confidentiality agent in the Base scenario (mean ±\pm SEM, 100 episodes). Row 1: PPO baseline. Rows 2–3: test-time policy shaping (TT) with α=0.5\alpha= 0.5 and α=1.0\alpha= 1.0, respectively. Rows 4–5: training-time reward shaping (TR) with λ=1.0\lambda= 1.0 and λ=3.0\lambda= 3.0, respectively. Row 6: adaptive test-time policy shaping. At Base scale, training-time λ=3.0\lambda= 3.0 achieves the strongest CIA suppression at the lowest reward cost; however, training-time shaping requires a separate training run per λ\lambda and fails at larger network sizes, whereas test-time and adaptive shaping operate on a single frozen PPO policy and scale reliably.

Figure S5. Comparing alignment approaches for the Integrity agent in the Base scenario (mean ±\pm SEM, 100 episodes). Row 1: PPO baseline. Rows 2–3: test-time policy shaping (TT) with α=0.5\alpha= 0.5 and α=1.0\alpha= 1.0, respectively. Rows 4–5: training-time reward shaping (TR) with λ=1.0\lambda= 1.0 and λ=3.0\lambda= 3.0, respectively. Row 6: adaptive test-time policy shaping. Integrity shaping follows a similar pattern to Confidentiality: training-time λ=3.0\lambda= 3.0 achieves the strongest suppression at Base scale, whereas test-time and adaptive policy shaping maintain more consistent alignment as network size increases.

Figure S6. Comparing alignment approaches for the Availability agent in the Base scenario (mean ±\pm SEM, 100 episodes). Row 1: PPO baseline. Rows 2–3: test-time policy shaping (TT) with α=0.5\alpha= 0.5 and α=1.0\alpha= 1.0, respectively. Rows 4–5: training-time reward shaping (TR) with λ=1.0\lambda= 1.0 and λ=3.0\lambda= 3.0, respectively. Row 6: adaptive test-time policy shaping. Availability shaping incurs a disproportionately higher reward cost compared to Confidentiality and Integrity, as all three CIA-critical servers contribute to Availability, making it inherently harder to protect without sacrificing task performance.

References

  1. [1]Alex Andrew, Sam Spillard, Joshua Collyer, and Neil Dhir. Developing optimal causal cyber-defence agents via cyber security simulation. In Workshop on Machine Learning for Cybersecurity (ML4Cyber), 07 2022.arxiv.org/abs/2207.12355
  2. [2]Ugurkan Ates. Long-term planning with deep reinforcement learning on autonomous drones. In 2020 Innovations in Intelligent Systems and Applications Conference (ASYU), pages 1–6. IEEE, 2020.arxiv.org/abs/2007.05694
  3. [3]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a.
  4. [4]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b.arxiv.org/abs/2212.08073
  5. [5]Anna L Buczak and Erhan Guven. A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Communications surveys & tutorials, 18(2): 1153–1176, 2015.DOI
  6. [6]Xavier Cadet, Simona Boboila, Edward Koh, Peter Chin, and Alina Oprea. Quantitative resilience modeling for autonomous cyber defense. In Reinforcement Learning Conference, 2025.arxiv.org/abs/2503.02780
  7. [7]Sebastián R Castro, Roberto Campbell, Nancy Lau, Octavio Villalobos, Jiaqi Duan, and Alvaro A Cardenas. Large language models are autonomous cyber defenders. In 2025 IEEE Conference on Artificial Intelligence (CAI), pages 1125–1132. IEEE, 2025.DOI
  8. [8]Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning, pages 3773–3793. PMLR, 2022.arxiv.org/abs/2205.11140
  9. [9]Wojciech M Czarnecki, Gauthier Gidel, Brendan Tracey, Karl Tuyls, Shayegan Omidshafiei, David Balduzzi, and Max Jaderberg. Real world games look like spinning tops. Advances in Neural Information Processing Systems, 33:17443–17454, 2020.arxiv.org/abs/2004.09468
  10. [10]Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: Learning from human feedback without rl. In The Twelfth International Conference on Learning Representations. International Conference on Learning Representations (ICLR), 2024.arxiv.org/abs/2310.13639
  11. [11]Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Patel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, and Jacob Steinhardt. What would jiminy cricket do? towards agents that behave morally. In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1, 2021.
  12. [12]Ilgee Hong, Zichong Li, Alexander Bukhari, Yixiao Li, Haoming Jiang, Tianbao Yang, and Tuo Zhao. Adaptive preference scaling for reinforcement learning with human feedback. Advances in Neural Information Processing Systems, 37:107249–107269, 2024.arxiv.org/abs/2406.02764
  13. [13]Brian Hu, Bill Ray, Alice Leung, Amy Summerville, David Joy, Christopher Funk, and Arslan Basharat. Language models are alignable decision-makers: Dataset and application to the medical triage domain. In Yi Yang, Aida Davani, Avi Sil, and Anoop Kumar, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 213–227, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
  14. [14]Wenhui Huang, Haochen Liu, Zhiyu Huang, and Chen Lv. Safety-aware human-in-the-loop reinforcement learning with shared control for autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 25(11):16181–16192, 2024a.DOI
  15. [16]Jiaming Ji, Tianyi Qiu, Boyuan Chen, Borong Zhang, Hantao Lou, Kaile Wang, Yawen Duan, Zhonghao He, Jiayi Zhou, Zhaowei Zhang, et al. Ai alignment: A comprehensive survey. arXiv preprint arXiv:2310.19852, 2023.arxiv.org/abs/2310.19852
  16. [17]Mitchell Kiely, David Bowman, Maxwell Standen, and Christopher Moir. On autonomous agents in a cyber defence environment. arXiv preprint arXiv:2309.07388, 2023.arxiv.org/abs/2309.07388
  17. [18]Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards, et al. Cage challenge 4: A scalable multi-agent reinforcement learning gym for autonomous cyber defence. AI Magazine, 46(3):e70021, 2025a.DOI
  18. [19]Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards, et al. Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 28907–28913, 2025b.DOI
  19. [20]Duc Huy Le and Rolf Stadler. Learning optimal defender strategies for cage-2 using a pomdp model. In 2025 21st International Conference on Network and Service Management (CNSM), pages 1–9. IEEE, 2025.DOI
  20. [21]Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. Rlaif vs. rlhf: scaling reinforcement learning from human feedback with a feedback. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.arxiv.org/abs/2309.00267
  21. [22]Xuefeng Liu, Takuma Yoneda, Rick Stevens, Matthew Walter, and Yuxin Chen. Blending imitation and reinforcement learning for robust policy improvement. In The Twelfth International Conference on Learning Representations, 2024.arxiv.org/abs/2310.01737
  22. [23]Andrew Lohn, Anna Knack, Ant Burke, and Krystal Jackson. Autonomous cyber defense. A roadmap from lab to ops. Online. Centre for Emerging Technology and Security (CETaS) at The Alan Turing Institute, 2023.DOI
  23. [24]Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 10(105): eads5033, 2025.DOI
  24. [25]Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusу, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529–533, 2015.
  25. [26]Dena Mujtaba, Brian Hu, Anthony Hoogs, and Arslan Basharat. Aligning machiavellian agents: Behavior steering via test-time policy shaping. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 37738–37746, 2026.
  26. [27]Thanh Thi Nguyen and Vijay Janapa Reddi. Deep reinforcement learning for cyber security. IEEE Transactions on Neural Networks and Learning Systems, 34(8):3779–3795, 2021.DOI
  27. [28]Alexander Pan, Jun Shern Chan, Andy Zou, Nathaniel Li, Steven Basart, Thomas Woodside, Hanlin Zhang, Scott Emmons, and Dan Hendrycks. Do the rewards justify the means? measuring trade-offs between rewards and ethical behavior in the machiavelli benchmark. In International conference on machine learning, pages 26837–26867. PMLR, 2023.arxiv.org/abs/2304.03279
  28. [29]Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 52516–52544, 2024.arxiv.org/abs/2408.10075
  29. [30]Ahmad Ridley. Machine learning for autonomous cyber defense. The Next Wave, 22(1): 7–14, 2018.
  30. [31]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  31. [32]Aditya Vikram Singh, Ethan Rathbun, Emma Graham, Lisa Oakley, Simona Boboila, Peter Chin, and Alina Oprea. Hierarchical multi-agent reinforcement learning for cyber network defense. In Reinforcement Learning Conference, 2025.arxiv.org/abs/2410.17351
  32. [33]Maxwell Standen, Martin Lucas, David Bowman, Toby J. Richer, Junae Kim, and Damian Marriott. Cyborg: A gym for the development of autonomous cyber agents. IJCAI-21 1st International Workshop on Adaptive Cyber Defense., 2021.arxiv.org/abs/2108.09118
  33. [34]Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, Cambridge, MA, 1998.DOI
  34. [35]Microsoft Defender Research Team. CyberBattleSim. https://github.com/microsoft/cyberbattlesim, 2021. Created by Christian Seifert, Michael Betser, William Blum, James Bono, Kate Farris, Emily Goren, Justin Grana, Kristian Holsheimer, Brandon Marken, Joshua Neil, Nicole Nichols, Jugal Parikh, Haoran Wei.
  35. [36]Sanyam Vyas, John Hannay, Andrew Bolton, and Professor Pete Burnap. Automated cyber defence: A review. arXiv preprint arXiv:2303.04926, 2023.arxiv.org/abs/2303.04926
  36. [37]Melody Wolk, Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, , and Adrian Webster. Beyond cage: Investigating generalization of learned autonomous network defense policies. In NeurIPS, 2022.
  37. [38]Jingda Wu, Zhiyu Huang, Zhongxu Hu, and Chen Lv. Toward human-in-the-loop ai: Enhancing deep reinforcement learning via real-time human guidance for autonomous driving. Engineering, 21:75–91, 2023.DOI
  38. [39]Qintong Xie, Edward Koh, Xavier Cadet, and Peter Chin. Nash q-network for multi-agent cybersecurity simulation. In International Conference on Game Theory and AI for Security, pages 43–60, 2025.arxiv.org/abs/2509.00678
  39. [40]Rui Yu, Shenghua Wan, Yucen Wang, Chen-Xiao Gao, Le Gan, Zongzhang Zhang, and De-Chuan Zhan. Reward models in deep reinforcement learning: a survey. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, pages 10807–10816, 2025.arxiv.org/abs/2506.15421
  40. [41]Yifan Zhang, Xingyu Lu, Xiao Hu, Chaoyou Fu, Bin Wen, Tianke Zhang, Changyi Liu, Kaiyu Jiang, Kaibing Chen, Kaiyu Tang, et al. R1-reward: Training multimodal reward model through stable reinforcement learning. In International Conference on Learning Representations, volume 2026, pages 31037–31061, 2026.arxiv.org/abs/2505.02835

Paper details

vlnc:W_wkfw7x4q
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY-SA 4.0
Usage

views · 7

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents