WorksComputer Science
Beyond -Norm Sets: Synthesizing Interpretable Non-Uniform Uncertainty Sets for Evaluation-Time State-Adversarial Robustness
Abstract
Deep reinforcement learning (DRL) agents are vulnerable to perturbations of their observed state inputs. Prior works often constrain state adversaries with -norm uncertainty sets and evaluate policy performance under the resulting perturbations. However, in continuous control tasks like robotics control, different state dimensions have distinct physical meanings and influence policy behavior to different degrees, which -norm sets fail to capture or explain. Moreover, it is important to identify when and along which state dimensions a policy is vulnerable, independently of the algorithm used to train it. In this paper, we focus on evaluation-time attacks applied to pretrained policies. We propose Performance-Degradation Non-Uniform Uncertainty Set (\method{}), an interpretable formulation of uncertainty sets for state perturbations. \method{} is state-dependent, dimension-wise non-uniform, and can be synthesized online during policy evaluation. We first formulate an ideal constrained optimization problem for \method{} computation, then introduce a surrogate constraint to make the problem tractable. Experiments across multiple continuous-control environments, policy algorithms, and state attacks show that \method{} reveals variation in policy sensitivity across states and dimensions, while its performance degradation is moderately controlled.
Introduction
Deep reinforcement learning (DRL) has achieved remarkable success in complex control problems, including robotics and autonomous driving. However, even well-trained policies may remain vulnerable to perturbations in observed states, transition dynamics, or action execution, which substantially degrade policy performance and cause unsafe behavior. Improving policy robustness against these perturbations is therefore essential when deploying DRL policies.
In this work, we focus on policy robustness against observed state perturbations in continuous control tasks. While some studies consider more structured formulations (Mandlekar et al. [12]; Everett, Lütjens, and How [5]), prior methods commonly constrain perturbations using an -norm uncertainty set which assumes a fixed perturbation budget across all states and state dimensions, and then evaluate or optimize the resulting performance degradation (Zhang et al. [29]; Oikarinen et al. [16]; Zhang et al. [28]; Liang et al. [11]). Although convenient, the -norm formulation has two important limitations that hinder its interpretability. First, a fixed perturbation budget does not account for variations in policy sensitivity across states and state dimensions. State features can differ substantially in their physical meanings and contributions to policy behavior (see a toy experiment in Table 1). Even perturbations along the same dimension have different effects depending on the current state; for example, a robot is more sensitive to perturbations when it is about to fall than when it is stable. Therefore, -norm produces insufficiently realistic perturbations that are restrictive in some scenarios but damaging in others. Second, in prior work the perturbation budget is often selected empirically and lacks a direct interpretation in terms of performance degradation. An appropriate budget for one environment or policy-learning algorithm can be unsuitable for another. While some works (Zhang et al. [29, 28]) show how to design adversarial perturbations within a specified uncertainty set, they rarely explore how to design a policy-specific uncertainty set which controls the resulting performance degradation.
| Noise () | Hopper Height | Torso Angle | Joint Angles: Thigh | Joint Angles: Leg | Joint Angles: Foot | Torso Linear Velocities: X-Axis | Torso Linear Velocities: Z-Axis | Torso Angular Velocity | Joint Velocities: Thigh | Joint Velocities: Leg | Joint Velocities: Foot |
| 0 | 3547.9 | ||||||||||
| 0.1 | 3282.5 | 3580.5 | 3486.3 | 3497.9 | 3499.9 | 3490.9 | 3435.4 | 3475.6 | 3579.6 | 3561.0 | 3580.4 |
| 0.2 | 2169.2 | 3575.7 | 3440.7 | 3540.5 | 3294.4 | 3245.6 | 3035.1 | 3395.5 | 3484.2 | 3472.8 | 2966.1 |
| 0.3 | 1501.0 | 3468.9 | 2918.6 | 3153.6 | 2632.2 | 3217.5 | 1967.7 | 1627.0 | 3373.2 | 3342.9 | 2523.7 |
| 0.4 | 1093.8 | 3583.8 | 2172.6 | 3098.3 | 2539.1 | 2399.4 | 994.7 | <u>414.6</u> | 2895.7 | 2896.6 | 1744.4 |
Table 1. A toy experiment in which each state dimension is perturbed for a PPO (Schulman et al. [20]) policy trained on Hopper environment. To perturb the -th state dimension, we apply constant noise, i.e. . We report the mean total reward over 20 episodes. For each value of noise , the most and least robust dimensions are shown in bold and <u>underlined</u>, respectively.
The limitations motivate an interpretable formulation of uncertainty sets. We aim to answer the following question:
Given a policy, what is the largest uncertainty set whose admissible perturbations remain consistent with a predefined tolerance on performance degradation?
Our main contributions are summarized as follows:
We propose Performance-Degradation Non-Uniform Uncertainty Set (PD-NUUS), a state-dependent and non-uniform uncertainty set for observed state perturbations. Unlike -norm, PD-NUUS relates local perturbations to a predefined tolerance on degradation in discounted return, providing a performance-based interpretation of the admissible perturbations.
We formulate the synthesis of PD-NUUS as a constrained optimization problem, and derive a tractable surrogate constraint that mitigates occupancy shift with a policy divergence regularizer. The resulting closed-form solution can be directly applied to pretrained policies for robustness evaluation without changing the policy update rule.
Extensive experiments across multiple policy-learning algorithms, environments, and state-adversarial attacks demonstrate that PD-NUUS provides generally reliable performance-level control over the degradation in discounted return.
Related Work
-Norm Uncertainty Set in Robust RL
Robust reinforcement learning (robust RL) studies how policies maintain reliable performance under perturbations to different components of a Markov Decision Process (MDP), including observed states and transition dynamics ([6, 29]). A single type of -norm uncertainty set is most widely used to model perturbations. For example, [29, 16, 28, 11] improve policy robustness against state perturbations constrained within a predefined -norm set. Pattanaik et al. [17] constrain adversarial state noise using an -norm set. Kuang et al. [9] propose an -norm based method to handle dynamics perturbations. Randomized smoothing approaches provide certified policy robustness under -norm perturbations, although the certification is for a smoothed policy rather than the original policy ([10] (Mu et al. 2024)).
Non-Uniform Perturbations
Erdemir et al. [4] introduce non-uniform perturbations for classification and prediction problems, and Wong and Kolter [25] propose to learn a non-uniform perturbation set with conditional variational autoencoders (CVAEs). For vision-based RL, Sun et al. [22] employ diffusion models to generate realistic perturbations semantically distinct from the original images. In robust MDPs where transition dynamics are perturbed, Petrik and Russel [18]; Zhang, Wang, and Boedecker [30] optimize the shape of the uncertainty set using value functions, while Behzadian et al. [3] leverage a percentile-performance criterion. For observed state perturbations, Mandlekar et al. [12] use the fast gradient method to create adversarial perturbations, and Everett, Lutjens, and How [5] certify the robustness of action-value function. Additionally, Korkmaz and Brown-Cohen [8] identify adversarial directions through local quadratic approximations of policy loss, Shen et al. [21] improve robustness through policy smoothness regularization, and Wu et al. (2021) certify action and cumulative-reward robustness through functional smoothing. However, we instead aim to explore how to attack a pretrained policy in a controlled way, and our method constrains the degradation directly in return rather than value functions, motivated by the observation from Beechey, Smith, and Şimşek [2] that contributions of state features to the value function do not necessarily provide faithful insight into contributions to policy performance.
Preliminaries
MDP and SA-MDP
We consider tasks with continuous state and action spaces. An MDP is specified by a tuple , where and denote state and action spaces respectively, is the transition kernel with denoting the set of probability distributions over the given space, is the initial state distribution, is the reward function, and is the discount factor. A policy maps each state to a distribution over actions. The discounted return of is defined as
The advantage function quantifies how much better a specific action is compared to the average action in that state. The normalized discounted future state distribution induced by , also called the occupancy distribution, is defined as
In this work, we focus on the scenarios in which the policy observes perturbed states. The State-Adversarial Markov Decision Process (SA-MDP) ([29]) augments the standard MDP with a state adversary only for the observed states. The action is predicted by a perturbed policy, while the environment still transitions from the true state to the next state. Although SA-MDP shows the influence of a state adversary on the Bellman operator during policy training, we only consider policy evaluation.
-Norm Uncertainty Set of State Adversary
The state adversary is typically constrained to make the problem well-posed. A common choice is an -norm ball around each state : That is, , where , and controls the perturbation budget. Different choices of -norm correspond to different assumptions on the perturbation geometry. A policy observes a perturbed state sampled from and uses as the basis for selecting an action, though the actual state transition proceeds from the true state .
Methodology
Problem Formulation of PD-NUUS
We consider the scenario where a policy is evaluated under observed state perturbations. At each time step, if the policy reaches a state , then a PD-NUUS is synthesized online around after which a perturbation is sampled. Given a continuous state space , we define PD-NUUS with a state-dependent budget function
In other words, a PD-NUUS augments an -norm ball with anisotropic scaling. We leave the correlation of state dimensions for future work. The class of admissible state adversaries induced by a PD-NUUS is denoted as
and attacking the observed state of a clean policy with a state adversary induces a new policy . The synthesis of PD-NUUS is then formulated as a constrained functional optimization problem over .
Uncertainty Set Size as Objective. Since for all state dimensions, we define the objective as the expected 1-norm of over states visited by the policy :
Performance Degradation as Constraint. We constrain the worst-case performance degradation in the expected discounted return. Given a tolerance , all admissible adversaries are required to satisfy
We apply the following performance difference equation which is obtained immediately by setting in Lemma 6.1 of Kakade and Langford [7]:
Therefore, the constraint is equivalent to
Ideal Problem Formulation. Combining objective and constraint, the formal PD-NUUS synthesis problem is:
Ideally, after specifying the hyperparameter , the optimal solution defines PD-NUUS over the visited states such that the worst-case degradation in discounted total return does not exceed . However, directly solving this problem is generally intractable. This is because the constraint contains a state adversary , which has no explicit expression but is only known to be constrained. Although can be approximated through a Taylor expansion, accurately computing generally requires both knowledge of the environment dynamics and an explicitly specified . Alternatively, may be estimated from trajectories sampled by , but repeating this process throughout the optimization is computationally expensive. Moreover, unlike prior works on policy optimization ([19, 1, 20]) which typically assume to guarantee policy improvement within constrained policy change, our formulation admits substantially perturbed policies, especially when is large. This implies may differ considerably from .
Approximate Closed-Form Solution
Surrogate Constraint for Intractable . Inspired by the divergence regularizer in offline RL (Wu, Tucker, and Nachum 2019), we replace the LHS of Eq. (10) by a surrogate
where a constant controls the strength of the Kullback–Leibler (KL) divergence penalty. In offline RL, such regularization handles occupancy shift by encouraging the learned policy to locally stay close to the behavior policy. In our setting, we use this approach to penalize state adversaries that induce large deviations between and . The following lemma motivates the calibration of .
Lemma 1 (Proof in Appendix A.1). Assume that a transition kernel is -Lipschitz continuous w.r.t the action: for all and for all , and a stochastic policy is -Lipschitz continuous from states to action distributions: for all W_1$ is the 1-Wasserstein distance induced by the -norm on the action space . Then for any state adversary ,
Although is over the entire state space and the Lipschitz assumption on transition dynamics is often violated in continuous control tasks, Lemma 1 indicates that the occupancy shift can depend on both the policy sensitivity and environment dynamics. This observation motivates calibrating separately for each environment and policy algorithm. In practice, we determine empirically according to diagnostics obtained from trajectories sampled by , and keep it fixed throughout the synthesis and evaluation of PD-NUUS. We also provide an upper bound on the possible violation of the prescribed degradation tolerance . Increasing reduces this upper bound, although it does not necessarily make the actual degradation decrease monotonically.
Lemma 2 (Proof in Appendix A.2). Consider a fixed policy , , and . After using the surrogate constraint in Eq. (11), the supremum degradation of discounted return satisfies the following upper bound
Reformulation with a Convex Quadratic Constraint
The detailed derivation is provided in Appendix § 10. To express Eq. (11) more directly in terms of , we approximate the perturbed policy using a second-order Taylor expansion, and approximate the local KL divergence using the state Fisher information matrix. Letting , the resulting supremum over becomes
where
and
The matrix is symmetric but not necessarily positive semidefinite. Consequently, Eq. (14) does not generally admit a simple analytical solution. To obtain a closed-form solution, we construct a diagonal positive-definite matrix which upper bounds in the Loewner order, meaning . This indicates that holds for every . We implement as a separable quadratic surrogate (SQS) majorization [14]:
where is a small offset. Since is diagonal with positive diagonal entries, the majorized supremum is separable across dimensions and can be evaluated exactly:
We replace the supremum in the surrogate constraint with the upper bound in Eq. (18). Although the majorization induces a more conservative solution for , it renders the constraint convex and analytically tractable. Finally, we obtain a tractable optimization problem for PD-NUUS with :
Algorithm 1: Evaluate a policy with PD-NUUS.
Input: Policy , PD-NUUS hyperparameters , observation attacker .
Output: Trajectories with PD-NUUS synthesized based on each visited state.
Pre-sample trajectories with and estimate in Eq. (86).
for each episode do
for time step do
Observe the current true state .
Calculate with Eq. (85).
Create a PD-NUUS in Eq. (58) via .
Use attacker to sample a perturbed observed state .
Predict an action: .
Step forward with .
end for
end for
When the optimal solution satisfies for every state dimension , the box constraints are inactive. By complementary slackness, their associate Karush–Kuhn–Tucker (KKT) multipliers vanish and the stationarity condition yields a closed-form solution for
where
is a global scalar estimated from trajectories collected under the clean policy , and and are defined in Eq. (16) and Eq. (17), respectively. We empirically observe that the interior condition is generally satisfied if is appropriately calibrated. However, when a non-negligible portion of reaches , the closed-form solution becomes unreliable. We analyze the failures in Appendix D. The closed-form solution leads us to a simple evaluation procedure with PD-NUUS: We first sample trajectories to estimate , then evaluate the policy while synthesizing PD-NUUS online at each visited state. Algorithm 1 presents the pseudocode.
Experimental Results
In this section, we aim to answer the following questions:
Does PD-NUUS produce different perturbation scales for different policy algorithms?
Is correlated with the empirical worst-case performance degradation, and how does this relationship compare with that of an -norm uncertainty set?
Does addressing the discrepancy between and with non-zero improve the quality of PD-NUUS?
Experiment Setup
Algorithms and Environments. We choose PPO (Schulman et al. [20]), SA-PPO-SGLD (Zhang et al. [29]), ATLA-PPO (Zhang et al. [28]), and WocaR-PPO (Liang et al. [11]) for policy training. Except for PPO, all algorithms incorporate dedicated mechanisms for improving robustness against state perturbations. We conduct experiments on Hopper, HalfCheetah, and Walker2d environments from MuJoCo (Todorov, Erez, and Tassa [24]). For each environment and algorithm, we train policies using four random seeds and report the average results. Since synthesizing PD-NUUS requires estimates of the advantage function , we train an auxiliary advantage network alongside each policy without changing the underlying policy update rule. The discount factor is fixed at throughout the experiments.
State Attackers. We consider four attackers that perturb the observed states within a given uncertainty set:
Random: Uniformly samples from the uncertainty set.
Critic (Zhang et al. [29]): Uses projected gradient descent (PGD) to search for an adversarial state that minimizes the state-value function.
Maximal Action Difference (MAD) (Zhang et al. [29]): A critic-free attack that searches for an adversarial state by maximizing the KL divergence .
PA-AD (Sun et al. [23]): Formulates state perturbations
as a Policy Adversary MDP (PAMDP), where a director learns to select an adversarial action, and an actor finds a feasible state perturbation that induces the desired policy change. An optimal policy of the PAMDP theoretically induces a state adversary that minimizes the clean policy’s expected discounted return.
According to Zhang et al. [29]; Liang et al. [11], Random and Critic are weaker than PA-AD and MAD. All four attacks were originally formulated for -norm uncertainty sets (Zhang et al. [29, 28]; Sun et al. [23]; Liang et al. [11]). To adapt them to PD-NUUS, we replace the budget with for every state dimension .
Implementation Details. For PD-NUUS, Table 2 reports the value of for each environment and algorithm, and we clip the solution to . We parameterize the policy with a Gaussian distribution , where is a multilayer perceptron for predicting the mean and is a diagonal covariance matrix. We exploit this Gaussian policy structure to compute the state Fisher information matrix and approximate the Hessian term in in Eq. (68), using a Gauss-Newton approximation. More implementation details are provided in Appendix C.
| PPO | SA-PPO-SGLD | ATLA-PPO | WocaR-PPO | |
| Hopper | 0.15 | 0.25 | 0.0075 | 0.2 |
| HalfCheetah | 0.06 | 0.2 | 0.02 | 0.075 |
| Walker2d | 0.025 | 0.075 | 0.05 | 0.15 |
Table 2. The coefficient of the KL divergence regularizer in Eq. (11) for each environment and algorithm during policy evaluation with PD-NUUS.
Empirical Results
-Norm vs. PD-NUUS. We report the empirical values for synthesizing PD-NUUS in Appendix D. Table 3 compares the empirical degradation in discounted return on HalfCheetah under -norm uncertainty sets with and PD-NUUS with . Results for Hopper and Walker2d are provided in Appendix D. We use the maximum degradation observed among all attackers as an empirical approximation of the worst-case degradation. As expected, increasing enlarges the -norm uncertainty set and generally admits stronger perturbations. However, the relationship between and return degradation is highly algorithm-dependent: SA-PPO-SGLD and WocaR-PPO experience substantially smaller degradation than PPO and ATLA-PPO under the same . In contrast, the tolerance of PD-NUUS is specified in units of discounted return. The maximum degradation across attackers is better aligned with than with , and generally remains below or moderately above the tolerance .
| Set | Parameter | Attacker | PPO | SA-PPO-SGLD | ATLA-PPO | WocaR-PPO |
| No Attack | — | — | 421.5 ± 57.5 | 349.4 ± 40.3 | 345.7 ± 24.9 | 331.4 ± 10.4 |
| -Norm | Random | 415.0 ± 46.7 (6.5) | 350.5 ± 40.7 (-1.1) | 342.9 ± 21.0 (2.8) | 331.3 ± 10.1 (0.1) | |
| -Norm | Critic | 404.7 ± 72.6 (16.8) | 350.4 ± 39.3 (-1.0) | 344.3 ± 18.8 (1.4) | 331.6 ± 10.7 (-0.2) | |
| -Norm | MAD | 336.7 ± 47.7 (84.8) | 349.9 ± 37.5 (-0.5) | 339.1 ± 21.4 (6.6) | 328.4 ± 9.3 (3.0) | |
| -Norm | PA-AD | 338.3 ± 67.0 (83.2) | 348.8 ± 38.6 (0.6) | 313.1 ± 38.0 (32.6) | 324.8 ± 7.1 (6.6) | |
| -Norm | Random | 382.9 ± 44.2 (38.6) | 350.6 ± 37.9 (-1.2) | 345.2 ± 19.7 (0.5) | 328.2 ± 9.0 (3.2) | |
| -Norm | Critic | 367.3 ± 98.8 (54.2) | 350.5 ± 41.5 (-1.1) | 334.1 ± 17.5 (11.6) | 329.5 ± 8.1 (1.9) | |
| -Norm | MAD | 224.6 ± 58.6 (196.9) | 348.2 ± 40.0 (1.2) | 293.0 ± 42.8 (52.7) | 322.6 ± 7.0 (8.8) | |
| -Norm | PA-AD | 168.0 ± 59.8 (253.5) | 348.0 ± 42.6 (1.4) | 243.7 ± 44.2 (102.0) | 328.4 ± 8.0 (3.0) | |
| -Norm | Random | 368.3 ± 36.5 (53.2) | 351.2 ± 38.2 (-1.8) | 339.6 ± 24.5 (6.1) | 326.7 ± 9.8 (4.7) | |
| -Norm | Critic | 332.2 ± 92.5 (89.3) | 351.1 ± 39.7 (-1.7) | 321.5 ± 15.7 (24.2) | 329.5 ± 10.1 (1.9) | |
| -Norm | MAD | 162.1 ± 55.2 (259.4) | 344.6 ± 37.6 (4.8) | 232.8 ± 39.9 (112.9) | 315.3 ± 8.4 (16.1) | |
| -Norm | PA-AD | 100.3 ± 47.7 (321.2) | 339.7 ± 27.3 (9.7) | 173.8 ± 53.7 (171.9) | 300.9 ± 17.2 (30.5) | |
| -Norm | Random | 321.8 ± 38.5 (99.7) | 348.5 ± 40.1 (0.9) | 328.3 ± 26.3 (17.4) | 324.3 ± 8.5 (7.1) | |
| -Norm | Critic | 280.5 ± 100.5 (141.0) | 347.2 ± 42.7 (2.2) | 295.5 ± 16.0 (50.2) | 324.7 ± 12.2 (6.7) | |
| -Norm | MAD | 117.1 ± 42.6 (304.4) | 342.5 ± 38.2 (6.9) | 164.5 ± 36.9 (181.2) | 303.2 ± 6.8 (28.2) | |
| -Norm | PA-AD | 22.3 ± 33.6 (399.2) | 337.8 ± 42.7 (11.6) | 86.6 ± 55.7 (259.1) | 287.0 ± 12.2 (44.4) | |
| PD-NUUS | Random | 421.2 ± 60.9 (0.3) | 350.0 ± 39.9 (-0.6) | 349.8 ± 22.7 (-4.1) | 330.1 ± 11.9 (1.3) | |
| PD-NUUS | Critic | 418.1 ± 56.6 (3.4) | 350.4 ± 41.3 (-1.0) | 350.2 ± 20.1 (-4.5) | 332.1 ± 9.4 (-0.7) | |
| PD-NUUS | MAD | 409.2 ± 65.3 (12.3) | 349.6 ± 41.4 (-0.2) | 349.4 ± 20.0 (-3.7) | 329.6 ± 9.2 (1.8) | |
| PD-NUUS | PA-AD | 418.2 ± 60.8 (3.3) | 346.2 ± 41.2 (3.2) | 347.9 ± 21.7 (-2.2) | 329.9 ± 8.9 (1.5) | |
| PD-NUUS | Random | 417.6 ± 58.1 (3.9) | 348.3 ± 41.2 (1.1) | 350.1 ± 19.2 (-4.4) | 330.4 ± 9.6 (1.0) | |
| PD-NUUS | Critic | 417.9 ± 59.9 (3.6) | 351.5 ± 40.4 (-2.1) | 351.4 ± 18.3 (-5.7) | 333.4 ± 10.3 (-2.0) | |
| PD-NUUS | MAD | 422.4 ± 60.0 (-0.9) | 348.1 ± 38.8 (1.3) | 346.3 ± 23.8 (-0.6) | 329.2 ± 11.4 (2.2) | |
| PD-NUUS | PA-AD | 424.4 ± 58.1 (-2.9) | 348.6 ± 38.8 (0.8) | 344.2 ± 21.2 (1.5) | 325.6 ± 8.6 (5.8) | |
| PD-NUUS | Random | 409.3 ± 59.1 (12.2) | 347.7 ± 39.3 (1.7) | 350.3 ± 21.2 (-4.6) | 329.8 ± 10.2 (1.6) | |
| PD-NUUS | Critic | 414.8 ± 65.6 (6.7) | 350.5 ± 40.4 (-1.1) | 346.1 ± 20.4 (-0.4) | 331.4 ± 11.2 (0.0) | |
| PD-NUUS | MAD | 413.9 ± 68.6 (7.6) | 346.5 ± 36.8 (2.9) | 344.1 ± 24.5 (1.6) | 326.8 ± 12.2 (4.6) | |
| PD-NUUS | PA-AD | 404.3 ± 54.5 (17.2) | 345.2 ± 38.7 (4.2) | 332.3 ± 19.0 (13.4) | 311.5 ± 13.6 (19.9) | |
| PD-NUUS | Random | 416.7 ± 59.0 (4.8) | 347.1 ± 38.1 (2.3) | 349.6 ± 20.7 (-3.9) | 321.3 ± 12.8 (10.1) | |
| PD-NUUS | Critic | 413.6 ± 60.6 (7.9) | 340.4 ± 36.2 (9.0) | 335.2 ± 23.4 (10.5) | 324.0 ± 9.1 (7.4) | |
| PD-NUUS | MAD | 405.4 ± 61.6 (16.1) | 330.2 ± 29.4 (19.2) | 335.8 ± 24.6 (9.9) | 310.0 ± 11.9 (21.2) | |
| PD-NUUS | PA-AD | 387.3 ± 68.3 (34.2) | 327.0 ± 44.3 (22.4) | 319.6 ± 19.1 (26.1) | 287.0 ± 24.8 (44.4) |
Table 3. Comparison of -norm uncertainty sets and PD-NUUS on HalfCheetah. Each entry reports the discounted return, with degradation in parentheses. The largest degradation across attackers is in bold. The episode rewards of clean policies are 5275.1 ± 1586.0 (PPO), 4320.3 ± 611.3 (SA-PPO-SGLD), 4267.1 ± 854.5 (ATLA-PPO) and 3952.2 ± 566.3 (WocaR-PPO).
Tolerance Calibration. Table 4 summarizes the empirical maximum degradation in discounted return among all attackers. Although several failure cases occur when , PD-NUUS generally provides more reliable control of empirical degradation for different algorithms. We next quantitatively evaluate whether the tolerance controls the maximum degradation in Table 4. Requiring the maximum degradation to stay strictly below as in Eq. (6) is too restrictive, because the surrogate constraint, local approximation errors, SQS majorization and finite-horizon trajectories can introduce discrepancies between the predefined and observed degradations. Therefore, we define the relaxed-tolerance frequency
| Environment | Algorithm | -Norm, | PD-NUUS, | ||||||
| Hopper | PPO | 52.0 | 162.3 | 168.9 | 198.5 | 1.7 | 10.0 | 2.7 | 21.0 |
| SA-PPO-SGLD | 0.6 | 0.6 | 0.6 | 28.4 | 3.9 | 5.8 | 14.6 | 30.8 | |
| ATLA-PPO | 17.3 | 78.7 | 103.0 | 136.8 | 1.0 | 2.5 | 24.5 | 127.7 | |
| WocaR-PPO | 1.2 | 10.6 | 25.7 | 151.7 | 2.7 | 2.4 | 9.3 | 21.7 | |
| HalfCheetah | PPO | 84.8 | 253.5 | 321.2 | 399.2 | 12.3 | 3.9 | 17.2 | 34.2 |
| SA-PPO-SGLD | 0.6 | 1.4 | 9.7 | 11.6 | 3.2 | 1.3 | 4.2 | 22.4 | |
| ATLA-PPO | 32.6 | 102.0 | 171.9 | 259.1 | -2.2 | 1.5 | 13.4 | 26.1 | |
| WocaR-PPO | 6.6 | 8.8 | 30.5 | 44.4 | 1.8 | 5.8 | 19.9 | 44.4 | |
| Walker2d | PPO | 50.3 | 139.0 | 162.3 | 205.3 | 2.8 | 7.7 | 60.5 | 105.6 |
| SA-PPO-SGLD | 1.0 | 23.0 | 50.3 | 128.9 | 16.0 | 14.9 | 15.4 | 128.3 | |
| ATLA-PPO | 49.0 | 125.7 | 165.1 | 191.5 | 0.1 | 3.2 | 7.9 | 12.3 | |
| WocaR-PPO | 2.9 | 42.3 | 135.2 | 139.0 | 5.6 | 1.7 | 16.7 | 50.0 |
Table 4. Maximum empirical degradation in discounted return, observed among all state attackers for each environment, algorithm and uncertainty set parameter.
where is the total number of evaluations with PD-NUUS, is the empirical maximum degradation in setting , and if is true and otherwise. Thus, denotes the fraction of settings in which the empirical maximum degradation exceeds the tolerance by no more than a relative margin . Table 5 indicates that provides informative control over the scale of empirical degradation.
| 0 | 0.1 | 0.2 | 0.25 | |
| 77.08% | 77.08% | 81.25% | 85.42% |
Table 5. values calculated across all PD-NUUS tests.
Ablation Study of the KL Divergence Regularizer. We conduct an ablation study to evaluate the role of the KL divergence regularizer in Eq. (11). Omitting the regularizer induces a solution that completely ignores the occupancy shift for any value (assuming ). Table 6 shows that the policy performance degrades substantially without the KL divergence regularizer for both values and all algorithms.
| Environment | Algorithm | ||||
| w/ | w/o | w/ | w/o | ||
| Hopper | PPO | 2.7 | 228.7 | 21.0 | 241.4 |
| Hopper | SA-PPO-SGLD | 14.6 | 55.0 | 30.8 | 72.4 |
| Hopper | ATLA-PPO | 24.5 | 128.1 | 127.7 | 176.7 |
| Hopper | WocaR-PPO | 9.3 | 134.5 | 21.7 | 183.2 |
| HalfCheetah | PPO | 17.2 | 285.3 | 34.2 | 341.7 |
| HalfCheetah | SA-PPO-SGLD | 4.2 | 112.8 | 22.4 | 151.5 |
| HalfCheetah | ATLA-PPO | 13.4 | 245.1 | 26.1 | 284.3 |
| HalfCheetah | WocaR-PPO | 19.9 | 129.4 | 44.4 | 157.5 |
| Walker2d | PPO | 60.5 | 256.0 | 105.6 | 273.4 |
| Walker2d | SA-PPO-SGLD | 15.4 | 252.9 | 128.3 | 251.9 |
| Walker2d | ATLA-PPO | 7.9 | 211.5 | 12.3 | 232.5 |
| Walker2d | WocaR-PPO | 16.7 | 218.7 | 50.0 | 203.4 |
Table 6. Empirical degradation in discounted total returns using PD-NUUS with (w/) and without (w/o) the KL divergence regularizer. With regularization, we report the maximum empirical degradation across four attackers; without regularization (), we report degradation under PA-AD. Even under this conservative comparison, degradation without regularization substantially exceeds .
PD-NUUS Visualization Results. In Figure 1, we visualize the synthesized PD-NUUS budgets for four algorithms under PA-AD attack. As increases, the budgets generally increase for all algorithms. SA-PPO-SGLD admits larger perturbations than other algorithms nearly for all dimensions. Under PA-AD attack, PD-NUUS at reduces episode rewards from 3471.7 391.7 to 2489.8 1287.3 for PPO, from 3203.0 378.5 to 1198.2 1085.3 for SA-PPO-SGLD, from 3227.5 634.9 to 924.3 1323.3 for ATLA-PPO, and from 3358.3 452.8 to 1663.1 1081.5 for WocaR-PPO compared with no attack.
Figure 1. We report the mean and confidence range of the calculated budget function for each state dimension across four algorithms and on Hopper under PA-AD attack. The statistics are computed over 200 sampled trajectories.
Limitations and Conclusion
We discuss several limitations of our work here. First, PD-NUUS selects the degradation in expected discounted return as the criterion. However, the discount factor significantly underweight rewards at later time steps. To address this issue, we may extend PD-NUUS to average-return scenarios. In addition, while the surrogate constraint makes the problem tractable, it introduces additional approximations. Finally, in this work we only use PD-NUUS for policy evaluation, and the effects of PD-NUUS on policy training remain unknown. Meanwhile for training, the pre-sampling phase at early training stages may result in unreliable PD-NUUS. Future work should address these concerns for robust policy training.
In summary, we introduce PD-NUUS, an interpretable uncertainty set formulation for observed state perturbations. By formulating its synthesis as a constrained optimization problem, we derive an efficient closed-form solution for analyzing policy robustness. In empirical results, PD-NUUS provides anisotropic uncertainty sets with generally constrained performance degradation.
Proofs
Proof of Lemma 1
Lemma 1. Assume that a transition kernel is -Lipschitz continuous w.r.t. the action: for all and for all , and a stochastic policy is -Lipschitz continuous from states to action distributions: for all where is the -Wasserstein distance induced by the -norm on the action space . Then for any state adversary ,
Proof. The proof is motivated by Achiam et al. [1]; Mao et al. [13]. Define
The -Wasserstein induced by -norm on the action space is expressed as
where denotes the set of couplings whose marginals are and . Then, for any coupling we upper bound the -norm distance
which holds for every coupling . Besides, for we have for . Taking the infimum over all couplings yields
The occupancy is also expressed as:
Then
Let then use the triangle inequality
This indicates
Since , from Eq. (35) we get
This completes the proof.
Proof of Lemma 2
Lemma 2. Consider a fixed policy , , and . After using the surrogate constraint in Eq. (11), the supremum degradation of discounted return satisfies the following upper bound
Proof. Fix an arbitrary adversary and denote
Surrogate feasibility implies
By the performance difference lemma,
::: Denote , for every ,
Applying Corollary 1 and 3 of Achiam et al. [1] with , together with , gives
where
For , Young’s inequality yields
Equivalently, this follows by expanding . Then
Dividing by and substituting the definition of gives
Since was arbitrary and is independent of the adversary class , taking the supremum over proves the lemma.
Derivation of a Closed-Form Solution
Reformulate the Surrogate Constraint
The original constrained optimization problem is intractable because it relies on which requires the knowledge of both the transition dynamics and the perturbed policy , while the exact is unknown but only known to be constrained by a PD-NUUS and an admissible set of state adversary
We propose a surrogate constraint
to make the problem tractable. This constraint is still dependent on , and we express it as a function of .
Let , then the perturbed policy is approximated by a second-order Taylor expansion
The KL divergence is approximated by using the state Fisher information matrix
Since , we get that
where
and
Then the surrogate constraint is equivalent to
where holds for each state dimension . However, is not necessarily positive semidefinite, and if this happens then the problem generally does not have a closed-form solution. To derive a closed-form solution, we construct a positive definite matrix , satisfying both and . This indicates that holds for every . We implement using SQP majorization as in Section 4.2. Then we obtain a convex constraint by:
Finally, the problem of PD-NUUS is reformulated as a tractable constrained problem:
Derive a Closed-Form Solution
The Lagrangian is constructed with , and .
When the optimal solution satisfies for every state dimension , by complementary slackness we have
for all sampled states and each state dimension . Since , the only solution is that . Then we differentiate with respect to
For the optimal solution we have this gradient equals 0, meaning
Since , and , does not allow this to happen. Therefore, holds and we
and Eq. (80) is equivalent to
Solving in Eq. (83) and (84) derives our closed-form solution
where
However, the closed-form solution is based on , and we have to calibrate to guarantee that
Additional Implementation Details
Exploit Gaussian Policy for PD-NUUS
In our experiments, the policy is parameterized as a Gaussian distribution where is a neural network for predicting the mean, and is a learnable but state-independent diagonal covariance matrix. The terms and in Eq. (68) and (69) involve , , and in the expectation terms. We leverage the structure of Gaussian policy to avoid explicitly estimating and . Let denote the Jacobian matrix of the policy mean w.r.t. the state, our implementation uses
and
Experiment Setup
All experiments were conducted on Quadro RTX 6000 GPU. We build on the official implementation of SA-PPO, ATLA-PPO and Wocar-PPO. For all methods, the actor and critic are parameterized by multilayer perceptrons with hidden layers [64, 64], and each policy is trained for environment steps. We made two changes compared to the official implementation. First, we train an auxiliary advantage network of hidden layers [64, 64] alongside the policy. Second, while the official implementation enables reward normalization, we disable it because naively reversing reward normalization on the learned advantage network does not provide reliable advantage estimates. We train each algorithm using four random seeds.
We train each PA-AD attacker for environment steps. Its actor and critic use the same MLP architecture, with hidden layers [64, 64], and are both optimized with a learning rate of .
Additional Experimental Results
Table 7, Table 8 and Table 9 present the discounted total return and the degradation compared to the clean policy under Random, Critic, MAD and PA-AD attackers on Hopper, HalfCheetah and Walker2d environments, respectively. We use these results to generate Table 4 in the main paper by taking the largest degradation across all state attackers.
| Set | Parameter | Attacker | PPO | SA-PPO-SGLD | ATLA-PPO | WocaR-PPO |
| No Attack | – | – | 269.2 ± 10.0 | 256.4 ± 6.5 | 257.5 ± 6.2 | 267.9 ± 4.5 |
| -Norm | Random | 268.2 ± 9.1 (1.0) | 256.1 ± 6.3 (0.3) | 257.2 ± 5.8 (0.3) | 268.0 ± 4.3 (-0.1) | |
| -Norm | Critic | 269.0 ± 11.0 (0.2) | 255.9 ± 6.2 (0.5) | 258.3 ± 5.7 (-0.8) | 268.4 ± 5.7 (-0.5) | |
| -Norm | MAD | 265.9 ± 9.1 (3.3) | 256.1 ± 5.5 (0.3) | 254.8 ± 6.5 (2.7) | 267.4 ± 4.3 (0.5) | |
| -Norm | PA-AD | 217.2 ± 29.3 (52.0) | 255.8 ± 4.8 (0.6) | 240.2 ± 17.7 (17.3) | 266.7 ± 2.1 (1.2) | |
| -Norm | Random | 267.5 ± 8.8 (1.7) | 256.0 ± 6.3 (0.4) | 256.3 ± 6.5 (1.2) | 267.8 ± 4.3 (0.1) | |
| -Norm | Critic | 266.0 ± 11.2 (3.2) | 255.9 ± 5.7 (0.5) | 252.7 ± 8.3 (4.8) | 270.7 ± 3.5 (-2.8) | |
| -Norm | MAD | 241.4 ± 14.9 (27.8) | 255.8 ± 5.5 (0.6) | 243.5 ± 9.7 (14.0) | 265.4 ± 4.2 (2.5) | |
| -Norm | PA-AD | 106.9 ± 64.9 (162.3) | 260.4 ± 7.4 (-4.0) | 178.8 ± 60.7 (78.7) | 257.3 ± 8.1 (10.6) | |
| -Norm | Random | 264.0 ± 8.3 (5.2) | 256.1 ± 5.8 (0.3) | 256.4 ± 6.3 (1.1) | 267.6 ± 4.0 (0.3) | |
| -Norm | Critic | 260.9 ± 11.6 (8.3) | 255.8 ± 5.8 (0.6) | 237.5 ± 28.0 (20.0) | 270.7 ± 3.4 (-2.8) | |
| -Norm | MAD | 186.0 ± 46.0 (83.2) | 257.0 ± 4.9 (-0.6) | 230.8 ± 15.9 (26.7) | 262.6 ± 3.4 (5.3) | |
| -Norm | PA-AD | 100.3 ± 66.5 (168.9) | 256.2 ± 13.1 (0.2) | 154.5 ± 69.2 (103.0) | 242.2 ± 16.2 (25.7) | |
| -Norm | Random | 263.7 ± 7.8 (5.5) | 255.7 ± 6.4 (0.7) | 252.6 ± 7.4 (4.9) | 266.6 ± 4.5 (1.3) | |
| -Norm | Critic | 242.5 ± 25.8 (26.7) | 255.0 ± 6.6 (1.4) | 233.5 ± 26.3 (24.0) | 270.0 ± 3.5 (-2.1) | |
| -Norm | MAD | 162.8 ± 54.2 (106.4) | 255.2 ± 3.0 (1.2) | 223.1 ± 15.1 (34.4) | 258.0 ± 6.4 (9.9) | |
| -Norm | PA-AD | 70.7 ± 34.0 (198.5) | 228.0 ± 29.4 (28.4) | 120.7 ± 64.8 (136.8) | 116.2 ± 43.2 (151.7) | |
| PD-NUUS | Random | 268.9 ± 10.3 (0.3) | 255.5 ± 5.6 (0.9) | 257.3 ± 5.8 (0.2) | 268.0 ± 4.2 (-0.1) | |
| PD-NUUS | Critic | 268.6 ± 11.2 (0.6) | 253.6 ± 6.3 (2.8) | 257.3 ± 5.4 (0.2) | 265.2 ± 7.5 (2.7) | |
| PD-NUUS | MAD | 267.5 ± 10.3 (1.7) | 252.5 ± 8.8 (3.9) | 256.5 ± 5.6 (1.0) | 267.4 ± 4.6 (0.5) | |
| PD-NUUS | PA-AD | 268.5 ± 10.8 (0.7) | 257.4 ± 6.6 (-1.0) | 256.6 ± 6.3 (0.9) | 266.5 ± 5.9 (1.4) | |
| PD-NUUS | Random | 268.9 ± 10.3 (0.3) | 256.1 ± 5.5 (0.3) | 257.3 ± 6.0 (0.2) | 267.3 ± 4.8 (0.6) | |
| PD-NUUS | Critic | 268.3 ± 11.6 (0.9) | 252.3 ± 7.4 (4.1) | 256.7 ± 6.1 (0.8) | 265.5 ± 6.0 (2.4) | |
| PD-NUUS | MAD | 267.2 ± 9.5 (2.0) | 250.6 ± 12.0 (5.8) | 255.0 ± 6.0 (2.5) | 267.1 ± 4.6 (0.8) | |
| PD-NUUS | PA-AD | 259.2 ± 17.5 (10.0) | 251.5 ± 5.6 (4.9) | 257.1 ± 5.6 (0.4) | 266.0 ± 6.5 (1.9) | |
| PD-NUUS | Random | 268.5 ± 9.5 (0.7) | 255.6 ± 6.4 (0.8) | 256.1 ± 6.7 (1.4) | 267.3 ± 4.6 (0.6) | |
| PD-NUUS | Critic | 269.0 ± 11.6 (0.2) | 250.6 ± 8.3 (5.8) | 257.2 ± 6.0 (0.3) | 262.7 ± 10.9 (5.2) | |
| PD-NUUS | MAD | 266.5 ± 9.4 (2.7) | 241.8 ± 20.3 (14.6) | 249.7 ± 9.4 (7.8) | 259.1 ± 16.1 (8.8) | |
| PD-NUUS | PA-AD | 268.4 ± 11.9 (0.8) | 249.2 ± 11.5 (7.2) | 233.0 ± 23.6 (24.5) | 258.6 ± 13.7 (9.3) | |
| PD-NUUS | Random | 268.4 ± 9.5 (0.8) | 254.5 ± 6.1 (1.9) | 257.2 ± 5.3 (0.3) | 266.6 ± 4.7 (1.3) | |
| PD-NUUS | Critic | 267.6 ± 12.2 (1.6) | 248.2 ± 5.8 (8.2) | 252.4 ± 7.5 (5.1) | 261.4 ± 11.9 (6.5) | |
| PD-NUUS | MAD | 261.3 ± 12.9 (7.9) | 225.6 ± 27.0 (30.8) | 238.8 ± 17.3 (18.7) | 253.5 ± 22.8 (14.4) | |
| PD-NUUS | PA-AD | 248.2 ± 16.6 (21.0) | 226.7 ± 12.4 (29.7) | 129.8 ± 95.9 (127.7) | 246.2 ± 8.6 (21.7) |
Table 7. Discounted returns () under state attackers on Hopper environment. When using PD-NUUS, the values of for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO are 0.15, 0.25, 0.0075 and 0.2, respectively. The values in parentheses represent degradation compared to no attack.
| Set | Parameter | Attacker | PPO | SA-PPO-SGLD | ATLA-PPO | WocaR-PPO |
| No Attack | – | – | 421.5 ± 57.5 | 349.4 ± 40.3 | 345.7 ± 24.9 | 331.4 ± 10.4 |
| -Norm | Random | 415.0 ± 46.7 (6.5) | 350.5 ± 40.7 (-1.1) | 342.9 ± 21.0 (2.8) | 331.3 ± 10.1 (0.1) | |
| -Norm | Critic | 404.7 ± 72.6 (16.8) | 350.4 ± 39.3 (-1.0) | 344.3 ± 18.8 (1.4) | 331.6 ± 10.7 (-0.2) | |
| -Norm | MAD | 336.7 ± 47.7 (84.8) | 349.9 ± 37.5 (-0.5) | 339.1 ± 21.4 (6.6) | 328.4 ± 9.3 (3.0) | |
| -Norm | PA-AD | 338.3 ± 67.0 (83.2) | 348.8 ± 38.6 (0.6) | 313.1 ± 38.0 (32.6) | 324.8 ± 7.1 (6.6) | |
| -Norm | Random | 382.9 ± 44.2 (38.6) | 350.6 ± 37.9 (-1.2) | 345.2 ± 19.7 (0.5) | 328.2 ± 9.0 (3.2) | |
| -Norm | Critic | 367.3 ± 98.8 (54.2) | 350.5 ± 41.5 (-1.1) | 334.1 ± 17.5 (11.6) | 329.5 ± 8.1 (1.9) | |
| -Norm | MAD | 224.6 ± 58.6 (196.9) | 348.2 ± 40.0 (1.2) | 293.0 ± 42.8 (52.7) | 322.6 ± 7.0 (8.8) | |
| -Norm | PA-AD | 168.0 ± 59.8 (253.5) | 348.0 ± 42.6 (1.4) | 243.7 ± 44.2 (102.0) | 328.4 ± 8.0 (3.0) | |
| -Norm | Random | 368.3 ± 36.5 (53.2) | 351.2 ± 38.2 (-1.8) | 339.6 ± 24.5 (6.1) | 326.7 ± 9.8 (4.7) | |
| -Norm | Critic | 332.2 ± 92.5 (89.3) | 351.1 ± 39.7 (-1.7) | 321.5 ± 15.7 (24.2) | 329.5 ± 10.1 (1.9) | |
| -Norm | MAD | 162.1 ± 55.2 (259.4) | 344.6 ± 37.6 (4.8) | 232.8 ± 39.9 (112.9) | 315.3 ± 8.4 (16.1) | |
| -Norm | PA-AD | 100.3 ± 47.7 (321.2) | 339.7 ± 27.3 (9.7) | 173.8 ± 53.7 (171.9) | 300.9 ± 17.2 (30.5) | |
| -Norm | Random | 321.8 ± 38.5 (99.7) | 348.5 ± 40.1 (0.9) | 328.3 ± 26.3 (17.4) | 324.3 ± 8.5 (7.1) | |
| -Norm | Critic | 280.5 ± 100.5 (141.0) | 347.2 ± 42.7 (2.2) | 295.5 ± 16.0 (50.2) | 324.7 ± 12.2 (6.7) | |
| -Norm | MAD | 117.1 ± 42.6 (304.4) | 342.5 ± 38.2 (6.9) | 164.5 ± 36.9 (181.2) | 303.2 ± 6.8 (28.2) | |
| -Norm | PA-AD | 22.3 ± 33.6 (399.2) | 337.8 ± 42.7 (11.6) | 86.6 ± 55.7 (259.1) | 287.0 ± 12.2 (44.4) | |
| PD-NUUS | Random | 421.2 ± 60.9 (0.3) | 350.0 ± 39.9 (-0.6) | 349.8 ± 22.7 (-4.1) | 330.1 ± 11.9 (1.3) | |
| PD-NUUS | Critic | 418.1 ± 56.6 (3.4) | 350.4 ± 41.3 (-1.0) | 350.2 ± 20.1 (-4.5) | 332.1 ± 9.4 (-0.7) | |
| PD-NUUS | MAD | 409.2 ± 65.3 (12.3) | 349.6 ± 41.4 (-0.2) | 349.4 ± 20.0 (-3.7) | 329.6 ± 9.2 (1.8) | |
| PD-NUUS | PA-AD | 418.2 ± 60.8 (3.3) | 346.2 ± 41.2 (3.2) | 347.9 ± 21.7 (-2.2) | 329.9 ± 8.9 (1.5) | |
| PD-NUUS | Random | 417.6 ± 58.1 (3.9) | 348.3 ± 41.2 (1.1) | 350.1 ± 19.2 (-4.4) | 330.4 ± 9.6 (1.0) | |
| PD-NUUS | Critic | 417.9 ± 59.9 (3.6) | 351.5 ± 40.4 (-2.1) | 351.4 ± 18.3 (-5.7) | 333.4 ± 10.3 (-2.0) | |
| PD-NUUS | MAD | 422.4 ± 60.0 (-0.9) | 348.1 ± 38.8 (1.3) | 346.3 ± 23.8 (-0.6) | 329.2 ± 11.4 (2.2) | |
| PD-NUUS | PA-AD | 424.4 ± 58.1 (-2.9) | 348.6 ± 38.8 (0.8) | 344.2 ± 21.2 (1.5) | 325.6 ± 8.6 (5.8) | |
| PD-NUUS | Random | 409.3 ± 59.1 (12.2) | 347.7 ± 39.3 (1.7) | 350.3 ± 21.2 (-4.6) | 329.8 ± 10.2 (1.6) | |
| PD-NUUS | Critic | 414.8 ± 65.6 (6.7) | 350.5 ± 40.4 (-1.1) | 346.1 ± 20.4 (-0.4) | 331.4 ± 11.2 (0.0) | |
| PD-NUUS | MAD | 413.9 ± 68.6 (7.6) | 346.5 ± 36.8 (2.9) | 344.1 ± 24.5 (1.6) | 326.8 ± 12.2 (4.6) | |
| PD-NUUS | PA-AD | 404.3 ± 54.5 (17.2) | 345.2 ± 38.7 (4.2) | 332.3 ± 19.0 (13.4) | 311.5 ± 13.6 (19.9) | |
| PD-NUUS | Random | 416.7 ± 59.0 (4.8) | 347.1 ± 38.1 (2.3) | 349.6 ± 20.7 (-3.9) | 321.3 ± 12.8 (10.1) | |
| PD-NUUS | Critic | 413.6 ± 60.6 (7.9) | 340.4 ± 36.2 (9.0) | 335.2 ± 23.4 (10.5) | 324.0 ± 9.1 (7.4) | |
| PD-NUUS | MAD | 405.4 ± 61.6 (16.1) | 330.2 ± 29.4 (19.2) | 335.8 ± 24.6 (9.9) | 310.2 ± 11.9 (21.2) | |
| PD-NUUS | PA-AD | 387.3 ± 68.3 (34.2) | 327.0 ± 44.3 (22.4) | 319.6 ± 19.1 (26.1) | 287.0 ± 24.8 (44.4) |
Table 8. Discounted returns () under state attackers on HalfCheetah environment. When using PD-NUUS, the value of for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO are 0.06, 0.2, 0.02 and 0.075, respectively. The values in parentheses represent degradation compared to no attack.
| Set | Parameter | Attacker | PPO | SA-PPO-SGLD | ATLA-PPO | WocaR-PPO |
| No Attack | – | – | 278.5 8.5 | 298.3 12.6 | 276.7 14.0 | 287.9 17.6 |
| -Norm | Random | 278.2 9.6 (0.3) | 299.0 12.7 (-0.7) | 278.6 11.5 (-1.9) | 287.6 17.8 (0.3) | |
| -Norm | Critic | 275.9 9.4 (2.6) | 302.6 10.8 (-4.3) | 275.6 16.3 (1.1) | 289.5 19.0 (-1.6) | |
| -Norm | MAD | 265.1 14.5 (13.4) | 299.6 10.4 (-1.3) | 260.3 10.4 (16.4) | 286.9 17.6 (1.0) | |
| -Norm | PA-AD | 228.2 44.6 (50.3) | 297.3 14.4 (1.0) | 227.7 35.1 (49.0) | 285.0 19.0 (2.9) | |
| -Norm | Random | 277.7 9.7 (0.8) | 298.2 12.8 (0.1) | 274.8 12.2 (1.9) | 287.6 17.8 (0.3) | |
| -Norm | Critic | 260.5 21.8 (18.0) | 296.6 9.7 (1.7) | 255.0 23.7 (21.7) | 292.3 18.8 (-4.4) | |
| -Norm | MAD | 229.9 29.4 (48.6) | 295.3 11.1 (3.0) | 222.1 14.4 (54.6) | 282.8 17.3 (5.1) | |
| -Norm | PA-AD | 139.5 16.5 (139.0) | 275.3 42.0 (23.0) | 151.0 17.6 (125.7) | 245.6 38.1 (42.3) | |
| -Norm | Random | 268.5 13.3 (10.0) | 298.1 11.7 (0.2) | 268.9 11.7 (7.8) | 286.7 18.4 (1.2) | |
| -Norm | Critic | 246.6 25.1 (31.9) | 287.1 28.2 (11.2) | 232.9 29.7 (43.8) | 294.5 17.8 (-6.6) | |
| -Norm | MAD | 197.4 23.0 (81.1) | 290.8 8.6 (7.5) | 186.4 19.4 (90.3) | 275.3 17.2 (12.6) | |
| -Norm | PA-AD | 116.2 31.7 (162.3) | 248.0 17.9 (50.3) | 111.6 23.0 (165.1) | 152.7 60.8 (135.2) | |
| -Norm | Random | 267.1 13.0 (11.4) | 297.9 10.4 (0.4) | 264.0 11.4 (12.7) | 286.9 17.9 (1.0) | |
| -Norm | Critic | 227.1 40.0 (51.4) | 271.8 40.7 (26.5) | 221.1 27.2 (55.6) | 297.7 13.8 (-9.8) | |
| -Norm | MAD | 178.5 17.6 (100.0) | 278.5 10.1 (19.8) | 162.7 21.9 (114.0) | 272.0 19.9 (15.9) | |
| -Norm | PA-AD | 73.2 33.5 (205.3) | 169.4 23.7 (128.9) | 85.2 41.7 (191.5) | 148.9 57.7 (139.0) | |
| PD-NUUS | Random | 279.2 7.6 (-0.7) | 298.0 14.5 (0.3) | 276.8 14.2 (-0.1) | 287.6 17.9 (0.3) | |
| PD-NUUS | Critic | 276.5 8.0 (2.0) | 305.3 9.8 (-7.0) | 276.6 15.5 (0.1) | 289.3 18.4 (-1.4) | |
| PD-NUUS | MAD | 279.3 8.5 (-0.8) | 296.4 14.4 (1.9) | 276.8 11.9 (-0.1) | 287.9 17.3 (0.0) | |
| PD-NUUS | PA-AD | 275.7 6.4 (2.8) | 282.3 17.3 (16.0) | 277.8 10.0 (-1.1) | 282.3 19.8 (5.6) | |
| PD-NUUS | Random | 278.0 8.9 (0.5) | 297.6 14.8 (0.7) | 277.6 13.7 (-0.9) | 288.0 17.3 (-0.1) | |
| PD-NUUS | Critic | 274.8 10.6 (3.7) | 304.1 7.1 (-5.8) | 276.1 14.7 (0.6) | 289.7 18.3 (-1.8) | |
| PD-NUUS | MAD | 276.2 7.7 (2.3) | 297.0 12.8 (1.3) | 276.0 12.8 (0.7) | 287.7 17.3 (0.2) | |
| PD-NUUS | PA-AD | 270.8 11.9 (7.7) | 283.4 16.2 (14.9) | 273.5 15.5 (3.2) | 286.2 20.7 (1.7) | |
| PD-NUUS | Random | 277.9 10.1 (0.6) | 297.6 12.5 (0.7) | 278.2 11.7 (-1.5) | 288.0 17.8 (-0.1) | |
| PD-NUUS | Critic | 268.9 13.7 (9.6) | 296.1 12.4 (2.2) | 276.2 14.0 (0.5) | 289.9 18.9 (-2.0) | |
| PD-NUUS | MAD | 269.8 10.2 (8.7) | 296.6 6.1 (1.7) | 277.7 13.2 (-1.0) | 286.6 18.3 (1.3) | |
| PD-NUUS | PA-AD | 218.0 40.7 (60.5) | 282.9 15.6 (15.4) | 268.8 11.3 (7.9) | 271.1 26.6 (16.7) | |
| PD-NUUS | Random | 277.2 10.0 (1.3) | 296.3 8.4 (2.0) | 275.2 11.8 (1.5) | 287.4 17.8 (0.5) | |
| PD-NUUS | Critic | 263.7 18.3 (14.8) | 288.0 8.8 (10.3) | 274.0 16.1 (2.7) | 292.2 17.3 (-4.3) | |
| PD-NUUS | MAD | 255.6 8.3 (22.9) | 280.5 10.1 (17.8) | 272.1 11.8 (4.6) | 282.6 17.8 (5.3) | |
| PD-NUUS | PA-AD | 172.9 44.8 (105.6) | 170.0 81.1 (128.3) | 264.4 22.6 (12.3) | 237.9 50.4 (50.0) |
Table 9. Discounted returns () under state attackers on Walker2d environment. When using PD-NUUS, the value of for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO are 0.025, 0.075, 0.05 and 0.15, respectively. The values in parentheses represent degradation compared to no attack.
In Table 10 we present the degradation in discounted return caused by different values, and the percentage of perturbation budget reaching the upper bound .
| Environment | Policy | Budget at (%) | Degradation in Discounted Return | |
| Hopper | PPO | 0 | 36.219 | 241.4 |
| Hopper | PPO | 0.015 | 4.855 | 234.7 |
| Hopper | PPO | 0.137 | 21.0 | |
| Hopper | SA-PPO-SGLD | 0 | 18.343 | 72.3 |
| Hopper | SA-PPO-SGLD | 7.515 | 29.6 | |
| Hopper | SA-PPO-SGLD | 0.3 | 8.758 | 22.4 |
| Hopper | ATLA-PPO | 0 | 15.982 | 176.7 |
| Hopper | ATLA-PPO | 0.325 | 127.7 | |
| Hopper | ATLA-PPO | 0.02 | 0.033 | 17.7 |
| Hopper | WocaR-PPO | 0 | 18.400 | 183.2 |
| Hopper | WocaR-PPO | 0.15 | 4.021 | 189.0 |
| Hopper | WocaR-PPO | 1.145 | 21.7 |
Table 10. Percentage of perturbation budgets that reach the upper bound under PA-AD attacks with PD-NUUS at . We compare the selected value (with *) of with other values. An inappropriate choice of yields degradation in discounted total return substantially greater than and cause a larger fraction of the learned budgets to saturate at .
Table 11 reports the empirical values of global multiplier for PD-NUUS.
| Environment | PPO | SA-PPO-SGLD | ATLA-PPO | WocaR-PPO | |
| Hopper | 5 | ||||
| Hopper | 10 | ||||
| Hopper | 20 | ||||
| Hopper | 40 | ||||
| HalfCheetah | 5 | ||||
| HalfCheetah | 10 | ||||
| HalfCheetah | 20 | ||||
| HalfCheetah | 40 | ||||
| Walker2d | 5 | ||||
| Walker2d | 10 | ||||
| Walker2d | 20 | ||||
| Walker2d | 40 |
Table 11. The mean and standard deviation of values estimated from sampled trajectories, when synthesizing PD-NUUS.
Table 12 reports the episode rewards of clean policies and policies perturbed by PD-NUUS () and PA-AD attacker.
| Environment | Attacker | PPO | SA-PPO-SGLD | ATLA-PPO | WocaR-PPO | |
| Hopper | – | – | ||||
| Hopper | PD-NUUS | 20 | ||||
| Hopper | PD-NUUS | 40 | ||||
| HalfCheetah | – | – | ||||
| HalfCheetah | PD-NUUS | 20 | ||||
| HalfCheetah | PD-NUUS | 40 | ||||
| Walker2d | – | – | ||||
| Walker2d | PD-NUUS | 20 | ||||
| Walker2d | PD-NUUS | 40 |
Table 12. Episode rewards without attack and under PA-AD attack with PD-NUUS ().
In Figure 2 we show the visualized budgets of PD-NUUS for PPO, SA-PPO-SGLD, ATLA-PPO and WocaR-PPO policies when .
Figure 2. The mean and confidence range of calculated budget of policies under PA-AD attack sampled from PD-NUUS with on Hopper. SA-PPO-SGLD, ATLA-PPO and WocaR-PPO admit higher perturbation tolerance than PPO in nearly all state dimensions.
References
- [1]Achiam, J.; Held, D.; Tamar, A.; and Abbeel, P. 2017. Constrained policy optimization. In International conference on machine learning, 22–31. PMLR.arxiv.org/abs/1705.10528
- [2]Beechey, D.; Smith, T. M.; and Şimşek, Ö. 2023. Explaining reinforcement learning with shapley values. In International Conference on Machine Learning, 2003–2014. PMLR.
- [3]Behzadian, B.; Russel, R. H.; Petrik, M.; and Ho, C. P. 2021. Optimizing percentile criterion using robust MDPs. In International Conference on Artificial Intelligence and Statistics, 1009–1017. PMLR.
- [4]Erdemir, E.; Bickford, J.; Melis, L.; and Aydore, S. 2021. Adversarial robustness with non-uniform perturbations. Advances in Neural Information Processing Systems, 34: 19147–19159.
- [5]Everett, M.; Lütjens, B.; and How, J. P. 2021. Certifiable robustness to adversarial state uncertainty in deep reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems, 33(9): 4184–4198.
- [6]Iyengar, G. N. 2005. Robust dynamic programming. Mathematics of Operations Research, 30(2): 257–280.
- [7]Kakade, S.; and Langford, J. 2002. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, 267–274.
- [8]Korkmaz, E.; and Brown-Cohen, J. 2023. Detecting adversarial directions in deep reinforcement learning to make robust decisions. In International Conference on Machine Learning, 17534–17543. PMLR.
- [9]Kuang, Y.; Lu, M.; Wang, J.; Zhou, Q.; Li, B.; and Li, H. 2022. Learning robust policy against disturbance in transition dynamics via state-conservative policy optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 7247–7254.
- [10]Kumar, A.; Levine, A.; and Feizi, S. 2022. Policy Smoothing for Provably Robust Reinforcement Learning. In International Conference on Learning Representations (ICLR).
- [11]Liang, Y.; Sun, Y.; Zheng, R.; and Huang, F. 2022. Efficient adversarial training without attacking: Worst-case-aware robust reinforcement learning. Advances in neural information processing systems, 35: 22547–22561.
- [12]Mandlekar, A.; Zhu, Y.; Garg, A.; Fei-Fei, L.; and Savarese, S. 2017. Adversarially robust policy learning: Active construction of physically-plausible perturbations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 3932–3939. IEEE.
- [13]Mao, Y.; Qu, Y.; Wang, Q.; and Ji, X. 2026. Adaptive neighborhood-constrained q learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 38: 26008–26041.arxiv.org/abs/2511.02567
- [14]McGaffin, M. G.; and Fessler, J. A. 2015. Algorithmic design of majorizers for large-scale inverse problems. arXiv preprint arXiv:1508.02958.arxiv.org/abs/1508.02958
- [16]Oikarinen, T.; Zhang, W.; Megretski, A.; Daniel, L.; and Weng, T.-W. 2021. Robust deep reinforcement learning through adversarial loss. Advances in Neural Information Processing Systems, 34: 26156–26167.
- [17]Pattanaik, A.; Tang, Z.; Liu, S.; Bommanna, G.; and Chowdhary, G. 2017. Robust deep reinforcement learning with adversarial attacks. arXiv preprint arXiv:1712.03632.arxiv.org/abs/1712.03632
- [18]Petrik, M.; and Russel, R. H. 2019. Beyond confidence regions: Tight bayesian ambiguity sets for robust mdps. Advances in neural information processing systems, 32.
- [19]Schulman, J.; Levine, S.; Abbeel, P.; Jordan, M.; and Moritz, P. 2015. Trust region policy optimization. In International conference on machine learning, 1889–1897. PMLR.
- [20]Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.arxiv.org/abs/1707.06347
- [21]Shen, Q.; Li, Y.; Jiang, H.; Wang, Z.; and Zhao, T. 2020. Deep reinforcement learning with robust and smooth policy. In International Conference on Machine Learning, 8707–8718. PMLR.
- [22]Sun, X.; Liu, F.; Ding, Z.; and Zheng, Z. 2026. Diffusion Guided Adversarial State Perturbations in Reinforcement Learning. Advances in Neural Information Processing Systems, 38: 43780–43815.
- [23]Sun, Y.; Zheng, R.; Liang, Y.; and Huang, F. 2022. Who Is the Strongest Enemy? Towards Optimal and Efficient Evasion Attacks in Deep RL. In International Conference on Learning Representations.
- [24]Todorov, E.; Erez, T.; and Tassa, Y. 2012. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, 5026–5033. IEEE.
- [25]Wong, E.; and Kolter, J. Z. 2020. Learning perturbation sets for robust machine learning. arXiv preprint arXiv:2007.08450.arxiv.org/abs/2007.08450
- [28]Zhang, H.; Chen, H.; Boning, D. S.; and Hsieh, C.-J. 2021. Robust Reinforcement Learning on State Observations with Learned Optimal Adversary. In International Conference on Learning Representations.
- [29]Zhang, H.; Chen, H.; Xiao, C.; Li, B.; Liu, M.; Boning, D.; and Hsieh, C.-J. 2020. Robust deep reinforcement learning against adversarial perturbations on state observations. Advances in Neural Information Processing Systems, 33: 21024–21037.
- [30]Zhang, Y.; Wang, J.; and Boedecker, J. 2023. Robust reinforcement learning in continuous control tasks with uncertainty set regularization. In Conference on Robot Learning, 1400–1424. PMLR.
