betaFeedback

WorksComputer Science

Structural Dual Adaptation for Safe Reinforcement Learning

Abstract

Primal-dual methods enforce safety constraints in reinforcement learning through Lagrangian relaxation. For a fixed policy, the exact bounded multiplier response exhibits bang-bang switching at the constraint boundary. Practical gradient updates can oscillate through coupling with policy optimization. We propose Structural DuaL Adaptation (SDLA), which replaces this bang-bang response with a unique continuous response through current-cost centering, trajectory weighting, and quadratic regularization. We characterize activation thresholds and response sensitivity. Ternary search computes multipliers without a dual gradient stepsize. For entropy-regularized tabular policies, we bound equilibrium constraint violations alongside reward gaps. We establish convergence of delayed updates under mild assumptions. Neural PPO/SAC experiments on OmniSafe and HASARD demonstrate competitive rewards with improved constraint tracking. On HASARD, PPO-SDLA achieves higher reported mean reward than the human baseline in six of eighteen settings while meeting the respective mean-cost budgets, including four with lower mean cost. To the best of our knowledge, this is the first report of higher mean reward at lower mean cost than the human baseline across multiple HASARD settings.

Introduction

Safe reinforcement learning seeks high reward while respecting cost constraints [1], [2]. Lagrangian methods introduce a multiplier λ\lambda that controls the cost penalty during policy improvement [3], [4], [5]. For a fixed policy with mean cost cc, the multiplier objective is λ(d−c)\lambda(d-c). Its exact minimizer on [0,λmax⁡][0,\lambda_{\max}] switches from zero to λmax⁡\lambda_{\max} as cc crosses the budget dd. This fixed-policy bang-bang response arises even with exact cost estimates.

Quadratic regularization makes the fixed-policy response continuous. Its penalty stays zero for c≤dc \le d and activates after a mean-cost violation. This violation-triggered dual activation can lag policy changes when cost estimates are delayed. We seek smooth feedback that can activate within the feasible region based on current trajectories. Incremental gradient updates can oscillate through policy coupling. PID uses tuned gains to combine damping and cost-trend feedback [6], [7]. Entropy regularization can smooth the dual function when the policy is reoptimized for each multiplier [8], [9]. With the policy held fixed, the multiplier objective remains linear, so its exact minimizer exhibits bang-bang switching. Table 1 compares these methods.

MethodInner SolveAvoids Bang-Bang
LagrangianO(1)O(1) (closed form)✗
Lagrangian-PIDNo closed form—
Entropy-RegularizedO(1)O(1) (closed form)✗
SDLA (ours)O(log⁡(λmax⁡/ελ))O(\log(\lambda_{\max}/\varepsilon_\lambda))✓

Table 1. Single-constraint multiplier responses with policy and return statistics fixed. Inner solve counts scalar optimization steps to multiplier accuracy ελ\varepsilon_\lambda. Linear objectives admit an exact closed-form solution (4); SDLA uses ternary search. Bang-bang refers to the exact response. For PID, “No closed form” denotes feedback updates without a closed-form optimal response; — denotes not applicable. Entropy-Regularized refers to policy entropy.

We propose Structural DuaL Adaptation (SDLA) to replace fixed-policy bang-bang switching with structured smooth feedback. Its trajectory-dependent surrogate combines log-sum-exp weighting of penalized returns with current-cost centering. Quadratic regularization and trajectory-weighted cost variance control response sensitivity, yielding a unique continuous multiplier response. Ternary search computes this response without a dual gradient stepsize.

Our main contributions are:

Structured multiplier response. We construct a strongly convex surrogate whose unique multiplier response replaces fixed-policy bang-bang switching with a continuous transition. Current-cost centering and trajectory weighting determine its activation and saturation regimes. A scalar inner solve computes the response without a dual gradient stepsize.

Sensitivity characterization. We bound response sensitivity, including clipped endpoints, and identify the interior gain through regularization and trajectory-weighted cost variance. For entropy-regularized tabular policies, we bound fixed-parameter equilibrium feasibility and reward gaps and establish convergence of delayed updates with actor and dual errors under mild assumptions.

Practical actor–critic evaluation. On HASARD, PPO-SDLA achieves higher reported mean reward than the human baseline in 6 of 18 task–difficulty settings, including four

with lower mean cost. All six settings meet their respective mean-cost budgets. To the best of our knowledge, this is the first report of simultaneously higher mean reward and lower mean cost than the human baseline across multiple HASARD settings. Ablations examine the surrogate components, while training-time diagnostics quantify multiplier changes and constraint tracking.

Preliminary

Constrained Markov Decision Process

We consider a single-constraint discounted CMDP (S,A,P,r,c,γ,d)(\mathcal{S},\mathcal{A},P,r,c,\gamma,d) [10], with 0≤r≤Rmax⁡0 \le r \le R_{\max}, 0≤c≤Cmax⁡0 \le c \le C_{\max}, γ∈[0,1)\gamma\in[0,1), and a fixed initial distribution. Let R=∑t≥0γtr(st,at)R = \sum_{t\geq0}\gamma^t r(s_t,a_t) and C=∑t≥0γtc(st,at)C = \sum_{t\geq0}\gamma^t c(s_t,a_t) denote trajectory returns, and write Jr(π)=EπRJ^r(\pi)=\mathbb{E}_{\pi}R, Jc(π)=EπCJ^c(\pi)=\mathbb{E}_{\pi}C. The objective is

max⁡πJr(π)s.t.Jc(π)≤d.(1)\max_{\pi} J^r(\pi) \qquad\text{s.t.}\qquad J^c(\pi)\leq d. \tag*{(1)}

Returns satisfy R≤Rmax⁡/(1−γ)R \leq R_{\max}/(1-\gamma) and C≤Cˉ:=Cmax⁡/(1−γ)C \leq\bar{C}:=C_{\max}/(1-\gamma). Discounted policy entropy is H(π)=Eπ∑t≥0γt[−log⁡π(at∣st)]H(\pi)=\mathbb{E}_{\pi}\sum_{t\geq0}\gamma^t[-\log\pi(a_t\mid s_t)].

Related Methods

Gradient primal–dual updates accumulate cost errors, and coupling with policy optimization can cause oscillation and stepsize sensitivity [6], [11]. PID control adds proportional damping and derivative feedback based on cost trends [6]. Augmented penalties [12], critic calibration [13], and regularized primal–dual methods [14], [15] provide other stabilization mechanisms. SDLA derives its response by optimizing a scalar surrogate on a frozen trajectory batch, with explicit activation and sensitivity bounds.

Penalty methods require selecting a penalty strength [16]. Constrained or projected methods introduce additional actor optimization steps [17], [18], [19], [20], while state augmentation requires budget-conditioned policy inputs [21], [22]. For constrained multi-objective RL, Gu et al. [23] combine gradient manipulation across reward objectives with cost-reducing policy updates when constraints are violated. Appendix E details these tradeoffs. SDLA changes the multiplier response within an actor–critic implementation.

Policy Dual Smoothness and Fixed-Policy Responses

Entropy regularization improves the optimization landscape of CMDPs by smoothing policy optimization. With entropy coefficient α>0\alpha>0, the entropy-regularized problem is

max⁡π{Jr(π)+αH(π)}s.t.Jc(π)≤d.(2)\max_{\pi}\{J^r(\pi)+\alpha H(\pi)\}\qquad\text{s.t.}\qquad J^c(\pi)\leq d. \tag*{(2)}

Lagrangian dual. The corresponding policy dual is [8]

gstd(λ):=sup⁡π{Jr(π)+αH(π)−λ(Jc(π)−d)}.(3)g_{\mathrm{std}}(\lambda):=\sup_{\pi}\{J^r(\pi)+\alpha H(\pi)-\lambda(J^c(\pi)-d)\}. \tag*{(3)}

In entropy-regularized tabular formulations, policy-dual smoothness enables accelerated optimization [8], [9]. These guarantees involve solving, or approximating, the underlying policy optimization problem.

Fixed-policy dual response. For a frozen policy πcur\pi_{\mathrm{cur}}, both Jr(πcur)J^r(\pi_{\mathrm{cur}}) and αH(πcur)\alpha H(\pi_{\mathrm{cur}}) are constant with respect to λ\lambda. Writing c=Jc(πcur)c=J^c(\pi_{\mathrm{cur}}), the exact multiplier response is

arg⁡min⁡0≤λ≤λmax⁡ λ(d−c)={{0},c<d,[0,λmax⁡],c=d,{λmax⁡},c>d.(4)\underset{0\leq\lambda\leq\lambda_{\max}}{\arg\min}\ \lambda(d-c)= \begin{cases} \{0\}, & c<d,\\ [0,\lambda_{\max}], & c=d,\\ \{\lambda_{\max}\}, & c>d. \end{cases} \tag*{(4)}

This bang-bang response concerns the fixed-policy subproblem. The policy dual in (3) reoptimizes the policy as λ\lambda changes. Quadratic regularization smooths the fixed-policy response and retains the activation boundary c=dc=d: the multiplier remains zero for c≤dc\leq d and activates after a mean-cost violation. SDLA builds on this curvature to construct structured smooth feedback from current mean and trajectory-weighted costs. Its response has controlled sensitivity and can also activate within the feasible region.

Structural Dual Adaptation

Surrogate Dual Formulation

We first define SDLA through expectations under a fixed policy π\pi. Write cπ=Jc(π)c_{\pi}=J^c(\pi) and fix τ,β,λmax⁡>0\tau,\beta,\lambda_{\max}>0. Returns satisfy the bounds in § 2.

Definition 3.1 (Population Surrogate Dual). The population surrogate dual is

gπ(λ)=τlog⁡Eπe(R−λC)/τ+λ(d−cπ)+λ22β,(5)g_{\pi}(\lambda)=\tau\log\mathbb{E}_{\pi}e^{(R-\lambda C)/\tau}+\lambda(d-c_{\pi})+\frac{\lambda^2}{2\beta}, \tag*{(5)}

for λ∈[0,λmax⁡]\lambda\in[0,\lambda_{\max}]. Its multiplier response is

Λ(π)=arg⁡min⁡0≤λ≤λmax⁡ gπ(λ).(6)\Lambda(\pi)=\underset{0\leq\lambda\leq\lambda_{\max}}{\arg\min}\ g_{\pi}(\lambda). \tag*{(6)}

Write Fπ(λ)=τlog⁡Eπe(R−λC)/τF_{\pi}(\lambda)=\tau\log\mathbb{E}_{\pi}e^{(R-\lambda C)/\tau}. Its derivative supplies the trajectory-weighted cost

mπ(λ):=−Fπ′(λ)=Eπ[Ce(R−λC)/τ]Eπe(R−λC)/τ.(7)m_{\pi}(\lambda):=-F'_{\pi}(\lambda)=\frac{\mathbb{E}_{\pi}[C e^{(R-\lambda C)/\tau}]}{\mathbb{E}_{\pi}e^{(R-\lambda C)/\tau}}. \tag*{(7)}

Trajectories with higher penalized returns R−λCR-\lambda C receive greater weight. Centering by λ(d−cπ)\lambda(d-c_{\pi}) combines mean and trajectory-weighted costs into the signal cπ+mπ(λ)−dc_{\pi}+m_{\pi}(\lambda)-d. Quadratic regularization and trajectory-weighted cost variance control the response curvature.

Structural Properties and Activation

Lemma 3.2 (Curvature and continuous activation). For any fixed policy with 0≤C≤Cˉ0\leq C\leq\bar{C}, the population surrogate satisfies

gπ′(λ)=d−cπ−mπ(λ)+λ/β,(8)g'_{\pi}(\lambda)=d-c_{\pi}-m_{\pi}(\lambda)+\lambda/\beta, \tag*{(8)}
gπ′′(λ)=Var⁡qπ,λ(C)τ+1β.(9)g''_{\pi}(\lambda)=\frac{\operatorname{Var}_{q_{\pi,\lambda}}(C)}{\tau}+\frac{1}{\beta}. \tag*{(9)}

Here qπ,λq_{\pi,\lambda} is the trajectory law reweighted by e(R−λC)/τe^{(R-\lambda C)/\tau}. Hence gπg_{\pi} is 1/β1/\beta-strongly convex and (1/β+Cˉ2/(4τ))(1/\beta+\bar{C}^{2}/(4\tau))-smooth. Its response Λ(π)\Lambda(\pi) is unique and satisfies

Λ(π)>0⟺cπ+mπ(0)>d.(10)\Lambda(\pi)>0 \quad\Longleftrightarrow\quad c_{\pi}+m_{\pi}(0)>d. \tag*{(10)}

Moreover, 0≤Λ(π)≤min⁡{λmax⁡,β[cπ+mπ(0)−d]+}0\leq\Lambda(\pi)\leq\min\{\lambda_{\max},\beta[c_{\pi}+m_{\pi}(0)-d]_{+}\}.

Feasible-region activation. Condition (10) permits penalties before the current policy’s mean cost exceeds the budget.

Proposition 4.2 quantifies equilibrium constraint violations and regularized reward gaps for fixed parameters. The endpoint derivatives distinguish three response regimes:

  • Inactive: gπ′(0)≥0g_{\pi}'(0) \ge0 gives Λ(π)=0\Lambda(\pi) = 0.

  • Transitional: gπ′(0)<0<gπ′(λmax⁡)g_{\pi}'(0) < 0 < g_{\pi}'(\lambda_{\max}) gives 0<Λ(π)<λmax⁡0 < \Lambda(\pi) < \lambda_{\max} and cπ+mπ(Λ(π))=d+Λ(π)/βc_{\pi} + m_{\pi}(\Lambda(\pi)) = d + \Lambda(\pi)/\beta.

  • Saturated: gπ′(λmax⁡)≤0g_{\pi}'(\lambda_{\max}) \le0 gives Λ(π)=λmax⁡\Lambda(\pi) = \lambda_{\max}.

The lemma implies Λ(π)→0\Lambda(\pi) \to0 as [cπ+mπ(0)−d]+→0[c_{\pi} + m_{\pi}(0) - d]_{+} \to0. Section 3.3 quantifies sensitivity across these regimes; Appendix B gives the proofs. Section 6.4 compares empirical variants with fixed and adaptive centering.

Closed-Loop Multiplier Feedback

References

  1. [1]J. García and F. Fernández, “A comprehensive survey on safe reinforcement learning,” Journal of Machine Learning Research, vol. 16, no. 42, pp. 1437–1480, 2015.DOI
  2. [2]S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, and A. Knoll, “A review of safe reinforcement learning: Methods, theories and applications,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 46, no. 12, pp. 11 216–11 235, 2024.DOI
  3. [3]A. Ray, J. Achiam, and D. Amodei, “Benchmarking safe exploration in deep reinforcement learning,” OpenAI, Tech. Rep., 2019. [Online]. Available: https://cdn.openai.com/safexp-short.pdf
  4. [4]W. Chen, J. Onvijay, L. Yu, L. Hoang, D. Subramanian, K. Kar, S. Mishra, and S. Paternaïn, “Adaptive primal-dual method for safe reinforcement learning,” arXiv preprint arXiv:2402.00355, 2024.arxiv.org/abs/2402.00355
  5. [5]S. Paternain, M. Calvo-Fullana, L. F. Chamon, and A. Ribeiro, “Safe policies for reinforcement learning via primal-dual methods,” IEEE Transactions on Automatic Control, vol. 68, no. 3, pp. 1321–1336, 2023.arxiv.org/abs/1911.09101
  6. [6]A. Stooke, J. Achiam, and P. Abbeel, “Responsive safety in reinforcement learning by pid lagrangian methods,” in International Conference on Machine Learning. PMLR, 2020, pp. 9133–9143.arxiv.org/abs/2007.03964
  7. [7]L. Spoor, Á. Serra-Gómez, A. Plaat, and T. Moerland, “An empirical study of lagrangian methods in safe reinforcement learning,” arXiv preprint arXiv:2510.17564v1, 2025.
  8. [8]D. Ying, Y. Ding, and J. Lavaei, “A dual approach to constrained markov decision processes with entropy regularization,” in International Conference on Artificial Intelligence and Statistics. PMLR, 2022, pp. 1887–1909. [Online]. Available: https://proceedings.mlr.press/v151/ying22a.htmlarxiv.org/abs/2110.08923
  9. [9]T. Li, Z. Guan, S. Zou, T. Xu, Y. Liang, and G. Lan, “Faster algorithm and sharper analysis for constrained markov decision process,” Operations Research Letters, vol. 54, p. 107107, 2024.DOI
  10. [10]E. Altman, Constrained Markov decision processes. Routledge, 2021.DOI
  11. [11]T. Lin, C. Jin, and M. Jordan, “On gradient descent ascent for nonconvex-concave minimax problems,” in International conference on machine learning. PMLR, 2020, pp. 6083–6093.arxiv.org/abs/1906.00331
  12. [12]J. Dai, J. Ji, L. Yang, Q. Zheng, and G. Pan, “Augmented proximal policy optimization for safe reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 6, 2023, pp. 7288–7295.DOI
  13. [13]Z. Wu, B. Tang, Q. Lin, C. Yu, S. Mao, Q. Xie, X. Wang, and D. Wang, “Off-policy primal-dual safe reinforcement learning,” arXiv preprint arXiv:2401.14758, 2024.arxiv.org/abs/2401.14758
  14. [14]D. Ding, C.-Y. Wei, K. Zhang, and A. Ribeiro, “Last-iterate convergent policy gradient primal-dual methods for constrained mdps,” Advances in Neural Information Processing Systems, vol. 36, pp. 66 138–66 200, 2023. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2023/hash/d0949cbec31c09431610553a284f94a-Abstract-Conference.htmlarxiv.org/abs/2306.11700
  15. [15]A. Montenegro, M. Mussi, M. Papini, and A. M. Metelli, “Last-iterate global convergence of policy gradients for constrained reinforcement learning,” Advances in Neural Information Processing Systems, vol. 37, pp. 126 363–126 416, 2024. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2024/hash/e47470631adb188a30f14c9738fe157b-Abstract-Conference.html
  16. [16]L. Zhang, L. Shen, L. Yang, S. Chen, B. Yuan, X. Wang, and D. Tao, “Penalized proximal policy optimization for safe reinforcement learning,” arXiv preprint arXiv:2205.11814, 2022.arxiv.org/abs/2205.11814
  17. [17]J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in International conference on machine learning. PMLR, 2017, pp. 22–31.arxiv.org/abs/1705.10528
  18. [18]T.-Y. Yang, J. Rosca, K. Narasimhan, and P. J. Ramadge, “Projection-based constrained policy optimization,” arXiv preprint arXiv:2010.03152, 2020.arxiv.org/abs/2010.03152
  19. [19]Y. Zhang, Q. Vuong, and K. Ross, “First order constrained optimization in policy space,” Advances in Neural Information Processing Systems, vol. 33, pp. 15 338–15 349, 2020.arxiv.org/abs/2002.06506
  20. [20]L. Yang, J. Ji, J. Dai, L. Zhang, B. Zhou, P. Li, Y. Yang, and G. Pan, “Constrained update projection approach to safe policy optimization,” Advances in Neural Information Processing Systems, vol. 35, pp. 9111–9124, 2022.arxiv.org/abs/2209.07089
  21. [21]A. Sootla, A. I. Cowen-Rivers, T. Jafferjee, Z. Wang, D. H. Mguni, J. Wang, and H. Ammar, “Almost sure safe reinforcement learning using state augmentation,” in International Conference on Machine Learning. PMLR, 2022, pp. 20 423–20 443.arxiv.org/abs/2202.06558
  22. [22]A. Sootla, A. Cowen-Rivers, J. Wang, and H. Bou Ammar, “Enhancing safe exploration using safety state augmentation,” Advances in Neural Information Processing Systems, vol. 35, pp. 34 464–34 477, 2022.DOI
  23. [23]S. Gu, B. Sel, Y. Ding, L. Wang, Q. Lin, A. Knoll, and M. Jin, “Safe and balanced: A framework for constrained multi-objective reinforcement learning,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 47, no. 5, pp. 3322–3331, 2025.DOI
  24. [24]T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in International conference on machine learning. Pmlr, 2018, pp. 1861–1870.
  25. [25]B. Liu, Q. Cai, Z. Yang, and Z. Wang, “Neural proximal/trust region policy optimization attains globally optimal policy,” in Advances in Neural Information Processing Systems, vol. 32, 2019. [Online]. Available: https://arxiv.org/abs/1906.10306
  26. [26]S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi, “Fast global convergence of natural policy gradient methods with entropy regularization,” Operations Research, vol. 70, no. 4, pp. 2563–2578, 2022. [Online]. Available: https://yunchen2020.github.io/publications/NPG_reg.pdfarxiv.org/abs/2007.06558
  27. [27]X. Guo, A. Hu, R. Xu, and J. Zhang, “Learning mean-field games,” in Advances in Neural Information Processing Systems, vol. 32, 2019. [Online]. Available: https://papers.neurips.cc/paper/8742-learning-mean-field-games.pdfarxiv.org/abs/1901.09585
  28. [28]Q. Xie, Z. Yang, Z. Wang, and A. Minca, “Learning while playing in mean-field games: Convergence and optimality,” in International Conference on Machine Learning, vol. 139. PMLR, 2021, pp. 11 436–11 447. [Online]. Available: https://proceedings.mlr.press/v139/xie21g.html
  29. [29]D. Mandal, S. Triantafyllou, and G. Radanovic, “Performative reinforcement learning,” in International Conference on Machine Learning, vol. 202. PMLR, 2023, pp. 23 642–23 680. [Online]. Available: https://proceedings.mlr.press/v202/mandal23a.htmlarxiv.org/abs/2207.00046
  30. [30]A. Olshevsky and B. Charesard, “A small gain analysis of single timescale actor critic,” SIAM Journal on Control and Optimization, vol. 61, no. 2, pp. 980–1007, 2023.
  31. [32]T. Antczak, “A lower bound for the penalty parameter in the exact minimax penalty function method for solving nondifferentiable extremum problems,” Journal of Optimization Theory and Applications, vol. 159, pp. 437–453, 2013. [Online]. Available: https://link.springer.com/article/10.1007/s10957-013-0335-3DOI
  32. [33]J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017.arxiv.org/abs/1707.06347
  33. [34]Z. Liu, Z. Cen, V. I. Eysenbaev, W. Liu, S. Wu, B. Li, and D. Zhao, “Constrained variational policy optimization for safe reinforcement learning,” in Proceedings of the 39th International Conference on Machine Learning, vol. 162. PMLR, 2022, pp. 13 644–13 668. [Online]. Available: https://proceedings.mlr.press/v162/liu22b.htmlarxiv.org/abs/2201.11927
  34. [35]T. Xu, Y. Liang, and G. Lan, “CRPO: A new approach for safe reinforcement learning with convergence guarantee,” in Proceedings of the 38th International Conference on Machine Learning, vol. 139. PMLR, 2021, pp. 11 480–11 491. [Online]. Available: https://proceedings.mlr.press/v139/xu21a.htmlarxiv.org/abs/2011.05869
  35. [36]T. Tomlin, M. Fang, and M. Pechenizkiy, “Hasard: A benchmark for vision-based safe reinforcement learning in embodied agents,” in The Thirteenth International Conference on Learning Representations, 2025.
  36. [37]J. Ji, J. Zhou, B. Zhang, J. Dai, X. Pan, R. Sun, W. Huang, Y. Geng, M. Liu, and Y. Yang, “Omission-safe an infrastructure for accelerating safe reinforcement learning research,” Journal of Machine Learning Research, vol. 25, no. 285, pp. 1–6, 2024.

Paper details

vlnc:W_q9s4au6y
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 2

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

OpenAI ChatGPT/Codex

View statement
Looking for related papers…
Contents