betaFeedback

WorksMathematics

KL-Regularized Policy Iteration for Adaptive LQR: Gain Updates and Probing from One Policy Improvement

Abstract

Policy-gradient methods for adaptive linear quadratic control improve the feedback gain of an unknown plant online by gradient-type steps computed from closed-loop data. A recent framework of such methods, policy-gradient adaptive control (PGAC), takes one gradient-type step per sample, which its indirect form computes from an estimated model and its direct form directly from the data. Because data generated by a fixed linear feedback cannot identify the plant, PGAC adds a probing perturbation to the input and leaves its covariance as a design parameter. We study the adaptive LQR problem through a KL-regularized formulation, in which the probing is the randomness of a Gaussian policy kept close to a Gaussian reference policy, so that the cost of the probing is part of the objective. One policy improvement of this formulation gives a gain update together with a probing covariance. We show that, for suitable references centred at the current gain, this gain update is exactly PGAC's Gauss-Newton or natural-gradient update with any step size below a bound. We repeat this policy improvement, each time against a tempered copy of the current policy, and refer to this scheme as KL-regularized policy iteration (KLPI). On a known plant and with tempering, the cost of KLPI's gains converges linearly to the optimal cost, and the probing covariance keeps a floor proportional to the regularization weight. On an estimated plant, under conditions like PGAC's (small updates and sufficiently informative data), the closed loop stays stable, the cost of the gains converges linearly up to a bias, and the probing covariance again keeps this floor. In simulations at equal probing cost, the shape of KLPI's probing covariance, which PGAC can adopt, lowers the extra cost caused by suboptimal gains on nearly all tested plants, whereas KLPI's use of past iterations in its gain update changes this cost little.

Introduction

The linear quadratic regulator (LQR) asks for the state-feedback gain KK that minimizes the infinite-horizon average quadratic cost C(K)C(K) of a linear plant driven by Gaussian process noise. When the plant is known, the optimal gain K⋆K^{\star} follows from a Riccati equation, or from Hewer’s policy iteration [2], which alternates evaluating the current gain (policy evaluation) with choosing the input that minimizes its state-action value (policy improvement). Policy-gradient methods search over gains directly. The cost C(K)C(K) is not convex in KK, but Fazel et al. [3] showed that it is gradient dominated, that is, the gap C(K)−C(K⋆)C(K)-C(K^{\star}) is at most a multiple of the squared norm of the gradient. With suitable step sizes, the gradient, natural-gradient and Gauss–Newton iterations therefore converge linearly from any stabilizing gain. The Gauss–Newton step rescales the gradient on the left by the inverse Hessian of the state-action value with respect to the input and on the right by the inverse stationary state covariance, and with step size one half it is Hewer’s step [4].

In adaptive LQR the plant is unknown and the controller improves its gain from closed-loop data, which raises two difficulties. Firstly, the switching between gains that each stabilize the plant can still destabilize the closed loop. The state stays bounded when the gains are uniformly strongly stable and consecutive gains stay close, a property used in [5] that we call sequential stability in this paper. A gradient method secures it by taking small steps. Secondly, the data generated by a fixed linear feedback u=Kxu=Kx cannot identify the plant, because several plants produce the same closed loop on the subspace where these data lie [6]. The controller must therefore add to its input a probing perturbation, that is, a component that is not a linear function of the state.

Policy-gradient adaptive control (PGAC) [1] is a framework of policy-gradient methods for adaptive LQR that addresses both difficulties. At every sample, a PGAC method applies the current gain plus a probing perturbation and then takes one gradient, natural-gradient or Gauss–Newton step on the gain. The indirect form computes this step from a recursive least-squares estimate of the plant, and the direct form computes it from the sample covariance of the data. Computing a step requires solving only Lyapunov equations, whereas recomputing the optimal gain of the changing plant estimate at every sample would require solving a Riccati equation. Zhao et al. [1] also cite the robustness of small updates. Zhao et al. [1] prove that, if the step size is small and the data have a large enough signal-to-noise ratio, the switching gains keep the closed loop stable and the cost of the gain converges linearly up to a bias that shrinks as this ratio grows. In their analysis the probing matters only through the data it produces, and its cost is not part of the cost that the gain update decreases. The covariance of the probing perturbation is a design parameter, constant and isotropic in the experiments of [1], and both PGAC and the earlier data-enabled policy optimization (DeePO) [7] leave its choice to the designer.

This paper studies adaptive LQR through the lens of KL regularization, which is common in reinforcement learning [8] and in probabilistic approaches to control [9]. A KL-regularized problem minimizes the expected cost plus a penalty equal to a positive regularization weight times the KL divergence of the policy from a reference policy. Two of its properties matter for adaptive LQR. Firstly, when the reference is a Gaussian policy, the optimal policy is also randomized, as for the LQR with an entropy bonus [10], so the problem produces the probing and counts its cost in the objective. Secondly, when the reference is the current policy, the KL penalty keeps the new policy close to it, which gives the small steps that sequential stability needs. Just as PGAC takes one gradient-type step per sample, we take one policy improvement per sample instead of solving the regularized problem. Repeating this policy improvement with the current policy as reference and a fixed regularization weight makes the probing die out, because each repetition adds to the precision (inverse covariance) of the policy. Tempering the reference, that is, raising the density of the current policy to a power α<1\alpha< 1 and renormalizing, gives a policy improvement that penalizes the KL divergence to the current policy and rewards the entropy [11, 12, 13]. For the LQR, tempering keeps the policy covariance bounded away from zero when the regularization weight is bounded away from zero. For brevity, we refer to this scheme, in which the policy improvement is repeated against the tempered current policy, as KL-regularized policy iteration (KLPI).

Related work: KLPI applies to adaptive LQR the KL-and entropy-regularized policy iteration of reinforcement learning [11, 12]. This policy iteration is a policy mirror-descent method [12]. Without tempering and with a fixed regularization weight, the repeated policy improvement reweights the initial policy exponentially in the summed state-action values of earlier policies, as in POLITEX [14] for finite Markov decision processes. The model-free LQR method MFLQ [15] instead plays the gain that minimizes its sum of past estimated state-action values and probes with scheduled random inputs. In trajectory optimization, Mahmoudi Filabadi et al. [13] regularize the backward pass of differential dynamic programming with an entropy bonus and a KL penalty to the policy of the previous iteration. Over the iterations, their Gaussian policies accumulate the input gradients and Hessians of the local state-action values with geometrically decaying weights, and the authors interpret this accumulation as momentum. Closest to this paper, Szpruch et al. [16] probe only through entropy-regularized Gaussian policies when controlling an unknown continuous-time linear-quadratic system over finite-horizon episodes, and one of their algorithms penalizes the KL divergence to the previous episode’s policy. In this paper, every policy improvement evaluates the current gain on a least-squares plant estimate updated at every sample of a single trajectory, and our stability analysis follows that of [1].

Contributions: (i) We show that one policy improvement of the KL-regularized LQR returns a gain update and a probing covariance, and that for suitable references the gain update is PGAC’s Gauss–Newton or natural-gradient step with any step size below a bound (Propositions 1 and 3). (ii) With a known plant and tempering, we prove that the cost of KLPI’s gains converges linearly and that its probing covariance keeps a floor proportional to the regularization weight (Theorem 1). (iii) On an estimated plant, under conditions like those in PGAC’s analysis, we prove that KLPI keeps the closed loop stable and that its cost converges linearly up to a bias (Theorem 2), and that its probing covariance keeps a floor proportional to the regularization weight (Corollary 1). (iv) In simulations at equal probing cost, the shape of KLPI’s probing covariance, which PGAC can adopt, lowers the extra cost caused by suboptimal gains on nearly all tested plants, while KLPI’s use of past iterations in its gain update changes this cost little (Section VI).

Organization: Section II recalls the LQR, its policy-gradient updates and PGAC. Section III introduces the KL-regularized LQR problem and its policy improvement. Section IV introduces KLPI, analyses it when the plant is known and compares it with PGAC’s Gauss–Newton iterations. Section V analyses KLPI on an estimated plant. Section VI compares KLPI and PGAC in simulations, and Section VII concludes.

Notation: ∥⋅∥\lVert\cdot\rVert is the spectral norm, σ(⋅)\sigma(\cdot) the smallest singular value, λmin⁡\lambda_{\min}, λmax⁡\lambda_{\max} the extreme eigenvalues of a symmetric matrix, and M⪰NM \succeq N means that M−NM-N is positive semidefinite. N(m,Ψ)\mathcal{N}(m,\Psi) is the Gaussian distribution with mean mm and covariance Ψ\Psi. Capitals XtX_t, UtU_t, WtW_t, Ωt\Omega_t denote the closed-loop state, input, process noise and probing perturbation, lowercase xx, uu denote values, and Zt:=[Ut;Xt]Z_t := [U_t; X_t]. DKL(μ′∥μ)D_{\mathrm{KL}}(\mu'\Vert\mu) is the KL divergence of a distribution μ′\mu' from a distribution μ\mu, and h(μ)h(\mu) is the differential entropy of μ\mu. For stationary policies π\pi and ρ\rho, we write κρπ(x):=DKL(π(⋅∣x)∥ρ(⋅∣x))\kappa^\pi_\rho(x) := D_{\mathrm{KL}}(\pi(\cdot\mid x)\Vert\rho(\cdot\mid x)) for the KL divergence of π\pi from ρ\rho in the state xx.

Policy-gradient methods for LQR & PGAC

This section sets up the plant and its cost, then states the policy-gradient updates that PGAC takes and its condition on the data, which Section V also adopts.

Plant and cost: We consider the plant

Xt+1=AXt+BUt+Wt,Wt∼N(0,ΣW),(1)X_{t+1}=AX_t+BU_t+W_t,\qquad W_t\sim\mathcal{N}(0,\Sigma_W), \tag*{(1)}

with Xt∈RnX_t\in\mathbb{R}^n, Ut∈RmU_t\in\mathbb{R}^m, a process-noise covariance ΣW≻0\Sigma_W\succ0, (Wt)(W_t) independent over time and of the initial state X0X_0, which has a finite second moment, (A,B)(A,B) controllable, and the stage cost c(x,u):=xTQx+uTRuc(x,u):=x^\mathsf{T}Qx+u^\mathsf{T}Ru with Q,R≻0Q,R\succ0. A stationary policy π\pi assigns to each state xx a distribution π(⋅∣x)\pi(\cdot\mid x) of the input, and P\mathbb{P} denotes the law of the closed loop (1) with Ut∼π(⋅∣Xt)U_t\sim\pi(\cdot\mid X_t). Its average cost is

C(π):=lim sup⁡T→∞1TEP[∑t<Tc(Xt,Ut)].(2)C(\pi):=\limsup_{T\to\infty}\frac{1}{T}\mathbb{E}_{\mathbb{P}}\left[\sum_{t<T}c(X_t,U_t)\right]. \tag*{(2)}

A gain KK defines the deterministic policy u=Kxu=Kx and the closed-loop matrix AK:=A+BKA_K:=A+BK, and S\mathcal{S} is the set of gains for which AKA_K is Schur. Adding a Gaussian probing perturbation of covariance Ψ\Psi, drawn independently of the past, gives the linear-Gaussian policy N(Kx,Ψ)\mathcal{N}(Kx,\Psi).

Matrices of a gain: For K∈SK\in\mathcal{S}, the cost matrix PKP_K solves PK=Q+KTRK+AKTPKAKP_K=Q+K^\mathsf{T}RK+A_K^\mathsf{T}P_KA_K, the stationary state covariance solves ΣK=ΣW+AKΣKAKT\Sigma_K=\Sigma_W+A_K\Sigma_KA_K^\mathsf{T}, so that ΣK⪰ΣW\Sigma_K\succeq\Sigma_W, and C(K)=Tr⁡(PKΣW)=Tr⁡((Q+KTRK)ΣK)C(K)=\operatorname{Tr}(P_K\Sigma_W)=\operatorname{Tr}((Q+K^\mathsf{T}RK)\Sigma_K). For K∉SK\notin\mathcal{S}, C(K)=∞C(K)=\infty. The state-action value of KK is the cost of applying uu once in state xx and KxKx afterwards, relative to the average cost,

QK(x,u):=c(x,u)−C(K)+EX′∼N(Ax+Bu,ΣW)[X′TPKX′]=uTSKu+2uTNKx+xT(Q+ATPKA)x,(3)\begin{aligned} Q_K(x,u):=c(x,u)-C(K)+\mathbb{E}_{X'\sim\mathcal{N}(Ax+Bu,\Sigma_W)}[X'^\mathsf{T}P_KX'] \\ &= u^\mathsf{T}S_Ku+2u^\mathsf{T}N_Kx+x^\mathsf{T}(Q+A^\mathsf{T}P_KA)x, \tag*{(3)} \end{aligned}

with the input Hessian SK:=R+BTPKBS_K:=R+B^\mathsf{T}P_KB (half the Hessian of QKQ_K in uu) and NK:=BTPKAN_K:=B^\mathsf{T}P_KA. Its gradient in uu at u=Kxu=Kx is $2E_Kx$, with the slope EK:=SKK+NKE_K:=S_KK+N_K, which vanishes exactly at K⋆K^\star. We write P⋆P^\star, S⋆S^\star, Σ⋆\Sigma^\star, $C^\star for the objects of K⋆K^\star, and γ⋆:=∥Σ⋆∥/σ(ΣW)≥1\gamma^\star:= \|\Sigma^\star\|/\sigma(\Sigma_W) \ge1. Since PK⪰QP_K \succeq Q, every input Hessian satisfies SK⪰sIS_K \succeq sI with s:=λmin⁡(R+BTQB)s := \lambda_{\min}(R+B^\mathsf{T}QB).

Policy-gradient updates of the gain: The gradient of the cost is ∇C(K)=2EKΣK\nabla C(K)=2E_K\Sigma_K [3]. With a step size η>0\eta>0, the natural-gradient and Gauss–Newton steps of [3] are

K−2ηEK,K−2ηSK−1EK.(4)K-2\eta E_K,\qquad K-2\eta S_K^{-1}E_K. \tag*{(4)}

Both steps of (4) are preconditioned steps K−H−1EKK-H^{-1}E_K, in which a symmetric positive definite matrix HH (the preconditioner) sets the size and direction of the step, with H=12ηIH=\frac{1}{2\eta}I and H=SK2ηH=\frac{S_K}{2\eta} respectively. For η=12\eta=\frac{1}{2} the Gauss–Newton step minimizes (3) over uu, which is Hewer’s step [2, 4].

References

  1. [1]F. Zhao, A. Chiuso, and F. Dörfler, “Policy gradient adaptive control for the LQR: Indirect and direct approaches,” arXiv:2505.03706, 2025.arxiv.org/abs/2505.03706
  2. [2]G. A. Hewes, “An iterative technique for the computation of the steady state gains for the discrete optimal regulator,” IEEE Trans. Autom. Control, vol. 16, no. 4, pp. 382–384, 1971.
  3. [3]M. Fazel, R. Ge, S. Kakade, and M. Mesbahi, “Global convergence of policy gradient methods for the linear quadratic regulator,” in Proc. ICML, vol. 80, 2018, pp. 1467–1476.arxiv.org/abs/1801.05039
  4. [4]J. Bu, A. Mesbahi, M. Fazel, and M. Mesbahi, “LQR through the lens of first order methods: Discrete-time case,” arXiv:1907.08921, 2019.arxiv.org/abs/1907.08921
  5. [5]A. Cohen, T. Koren, and Y. Mansour, “Learning linear-quadratic regulators efficiently with only √T regret,” in Proc. ICML, vol. 97, 2019, pp. 1300–1309.
  6. [6]M. Simchowitz and D. Foster, “Naïve exploration is optimal for online LQR,” in Proc. ICML, vol. 119, 2020, pp. 8937–8948.arxiv.org/abs/2001.09576
  7. [7]F. Zhao, F. Dörfler, A. Chiuso, and K. You, “Data-enabled policy optimization for direct adaptive learning of the LQR,” arXiv:2401.14871, 2024.arxiv.org/abs/2401.14871
  8. [8]M. Geist, B. Scherrer, and O. Pietquin, “A theory of regularized Markov decision processes,” in Proc. ICML, vol. 97, 2019, pp. 2160–2169.arxiv.org/abs/1901.11275
  9. [9]T. Lefebvre, “Probabilistic control and majorisation of optimal control,” Syst. Control Lett., vol. 190, p. 105837, 2024.DOI
  10. [10]X. Guo, X. Li, and R. Xu, “Fast policy learning for linear quadratic control with entropy regularization,” arXiv:2311.14168, 2023.arxiv.org/abs/2311.14168
  11. [11]T. Kozuno, E. Uchibe, and K. Doya, “Theoretical analysis of efficiency and robustness of softmax and gap-increasing operators in reinforcement learning,” in Proc. AISTATS, 2019.
  12. [12]N. Vieillard, T. Kozuno, B. Scherrer, O. Pietquin, R. Munos, and M. Geist, “Leverage the average: an analysis of KL regularization in reinforcement learning,” in Proc. NeurIPS, 2020.
  13. [13]M. Mahmoudi Filabadi, G. Crevecoeur, and T. Lefebvre, “How KL regularization induces momentum in differential dynamic programming,” IEEE Control Syst. Lett., 2026.
  14. [14]Y. Abbasi-Yadkori, P. Bartlett, K. Bhatia, N. Lazić, C. Szepesvári, and G. Weisz, “POLITEX: Regret bounds for policy iteration using expert prediction,” in Proc. ICML, ser. PMLR, vol. 97, 2019, pp. 3692–3702.
  15. [15]Y. Abbasi-Yadkori, N. Lazić, and C. Szepesvári, “Model-free linear quadratic control via reduction to expert prediction,” in Proc. AISTATS, ser. PMLR, vol. 89, 2019, pp. 3108–3117.
  16. [16]L. Szpruch, T. Tretyakov, and Y. Zhang, “Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning,” SIAM J. Control Optim., vol. 62, no. 1, pp. 135–166, 2024.DOI
  17. [17]K. Lange, M. M Optimization Algorithms, SIAM, 2016.
  18. [18]E. A. Theodorou, “Nonlinear stochastic control and information theoretic dualities: Connections, interdependencies and thermodynamic interpretations,” Entropy, vol. 17, no. 5, pp. 3352–3375, 2015.DOI
  19. [19]G. Lan, “Policy mirror descent for reinforcement learning: Linear convergence, new sampling complexity, and generalized problem classes,” Math. Program., vol. 198, pp. 1059–1106, 2023.arxiv.org/abs/2102.00135
  20. [20]S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi, “Fast global convergence of natural policy gradient methods with entropy regularization,” Oper. Res., vol. 70, no. 4, pp. 2563–2578, 2022.arxiv.org/abs/2007.06558

Paper details

vlnc:W_3ew74vk2
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 5

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

Claude

View statement
Looking for related papers…
Contents