betaFeedback

WorksMathematics

Turnpike Properties for Exploratory Stochastic Linear-Quadratic Optimal Control with Indefinite Cost Functionals

Abstract

Continuous-time exploratory stochastic control provides a stochastic control framework for the exploration–exploitation trade-off in reinforcement learning. In this paper, we study the long-time behavior of finite-horizon exploratory stochastic linear-quadratic (LQ) optimal control problems as the time horizon T→∞T\to\infty, and establish so-called turnpike properties for such problems. In the finite-horizon setting, we derive a feedback representation of the exploratory stochastic LQ problem and analyze its long-time structure. We formulate an exploratory static distribution optimization problem analogous to the classical static optimization problem and identify the corresponding static reference object. We prove a turnpike property for the optimal state process relative to a stochastic reference process, as well as a Wasserstein turnpike property for the optimal control distribution. In addition to the exploratory control formulation, another key feature of our results is that, by introducing a joint coercivity condition, we allow the cost functional to be indefinite and the controlled state system to be merely stabilizable. Consequently, the derived turnpike property is novel even within the classical stochastic LQ control framework.

Introduction

Let (Ω,F,P)(\Omega,\mathcal{F},\mathbb{P}) be a complete probability space on which the Brownian motions used in this paper are defined. Here W={W(t):t≥0}W = \{W(t): t \ge0\} is a standard one-dimensional Brownian motion, W^={W^(t):t≥0}\widehat{W} = \{\widehat{W}(t): t \ge0\} is a standard nn-dimensional Brownian motion, and W~={W~(t):t≥0}\widetilde{W} = \{\widetilde{W}(t): t \ge0\} is a standard mm-dimensional Brownian motion. We assume that WW and W~\widetilde{W} are independent. Let F={Ft}t≥0\mathbb{F} = \{\mathcal{F}_t\}_{t \ge0} be the usual augmentation of the natural filtration generated by these Brownian motions. All random variables and processes appearing below are assumed to be measurable and adapted to F\mathbb{F}; admissible control distribution processes are F\mathbb{F}-progressively measurable.

This paper studies continuous-time exploratory stochastic linear-quadratic (LQ) control problems. We start from a general controlled diffusion. Consider the stochastic differential equation (SDE) with point-valued controls

{dXu(t)=b(Xu(t),u(t))dt+σ(Xu(t),u(t))dW(t),t≥0,Xu(0)=x,\left\{ \begin{aligned} dX^{u}(t) &= b(X^{u}(t),u(t))dt+\sigma(X^{u}(t),u(t))dW(t), \qquad t \ge0,\\ X^{u}(0) &= x, \end{aligned} \right.

where bb and σ\sigma are the drift and diffusion coefficients. The usual measurability, growth, and integrability conditions are assumed to make this state equation well posed.

Following Wang–Zariphopoulou–Zhou [44], for a control distribution π∈P(U)\pi\in\mathcal{P}(U) and a pointwise running cost ρ(x,u)\rho(x,u), we define the averaged drift, diffusion covariance, and running cost, as well as the differential entropy when π\pi has density pp:

b~(x,π)=∫Ub(x,u) π(du),ρ~(x,π)=∫Uρ(x,u) π(du),\widetilde{b}(x,\pi)=\int_U b(x,u)\,\pi(du), \qquad\widetilde{\rho}(x,\pi)=\int_U \rho(x,u)\,\pi(du),
σ~(x,π)σ~(x,π)T=∫Uσ(x,u)σ(x,u)T π(du),Ent⁡(π)=−∫Up(u)log⁡p(u) du.\widetilde{\sigma}(x,\pi)\widetilde{\sigma}(x,\pi)^{\mathsf{T}}=\int_U \sigma(x,u)\sigma(x,u)^{\mathsf{T}}\,\pi(du), \qquad\operatorname{Ent}(\pi)=-\int_U p(u)\log p(u)\,du.

The exploratory state equation is

{dXπ(t)=b~(Xπ(t),πt)dt+σ~(Xπ(t),πt)dW~(t),Xπ(0)=x.(1)\begin{cases} dX^\pi(t)=\widetilde{b}(X^\pi(t),\pi_t)dt+\widetilde{\sigma}(X^\pi(t),\pi_t)d\widetilde{W}(t),\\ X^\pi(0)=x. \tag*{(1)} \end{cases}

We now specialize to the LQ setting with U=RmU=\mathbb{R}^m:

b(x,u)=Ax+Bu+b,σ(x,u)=Cx+Du+σ,b(x,u)=Ax+Bu+b,\qquad\sigma(x,u)=Cx+Du+\sigma,
ρ(x,u)=xTQx+2xTSTu+uTRu+2qTx+2rTu.(2)\rho(x,u)=x^{\mathsf{T}}Qx+2x^{\mathsf{T}}S^{\mathsf{T}}u+u^{\mathsf{T}}Ru+2q^{\mathsf{T}}x+2r^{\mathsf{T}}u. \tag*{(2)}

In the exploratory framework, the control variable is a process of probability distributions π={πt:t∈[0,T]}\pi=\{\pi_t:t\in[0,T]\} on the control space Rm\mathbb{R}^m. Denote the mean and covariance of πt\pi_t by

u‾(t)=∫Rmu πt(du),Λ(t)=∫Rm(u−u‾(t))(u−u‾(t))Tπt(du).(3)\overline{u}(t)=\int_{\mathbb{R}^m}u\,\pi_t(du),\qquad\Lambda(t)=\int_{\mathbb{R}^m}(u-\overline{u}(t))(u-\overline{u}(t))^{\mathsf{T}}\pi_t(du). \tag*{(3)}

Substituting the LQ coefficients (2) and the control moments (3) into the general exploratory state equation (1) gives the following equivalent exploratory LQ state equation:

{dXπ(t)=(AXπ(t)+Bu‾(t)+b)dt+(CXπ(t)+Du‾(t)+σ)dW(t)+DΛ(t)1/2dW~(t),t∈[0,T],Xπ(0)=x.\begin{cases} dX^\pi(t)=(AX^\pi(t)+B\overline{u}(t)+b)dt+(CX^\pi(t)+D\overline{u}(t)+\sigma)dW(t)\\ \qquad{}+D\Lambda(t)^{1/2}d\widetilde{W}(t),\qquad t\in[0,T],\\ X^\pi(0)=x. \end{cases}

Here WW represents the original system noise and W~\widetilde{W} represents the additional noise generated by the covariance of the randomized control.

The entropy-regularized cost on [0,T][0,T] is

JTλ(x;π)=E∫0T∫Rm[Xπ(t)TQXπ(t)+2Xπ(t)TSTu+uTRu+2qTXπ(t)+2rTu]πt(du) dt−λE∫0TEnt⁡(πt) dt,J_T^\lambda(x;\pi)=\mathbb{E}\int_0^T\int_{\mathbb{R}^m}\left[X^\pi(t)^{\mathsf{T}}QX^\pi(t)+2X^\pi(t)^{\mathsf{T}}S^{\mathsf{T}}u+u^{\mathsf{T}}Ru+2q^{\mathsf{T}}X^\pi(t)+2r^{\mathsf{T}}u\right]\pi_t(du)\,dt-\lambda\mathbb{E}\int_0^T\operatorname{Ent}(\pi_t)\,dt,

where λ>0\lambda>0 is the exploration weight. Since the cost is minimized, the entropy term encourages exploration: a larger λ\lambda gives more weight to exploration, while a smaller λ\lambda brings the model closer to the classical point-control problem.

The finite-horizon exploratory LQ problem can be stated as follows.

Problem (ESLQ)T_T. For a given initial state x∈Rnx\in\mathbb{R}^n, find π‾T\overline{\pi}^T in the admissible class of exploratory control distributions such that

JTλ(x;π‾T)=inf⁡πJTλ(x;π)≡VTλ(x).J_T^\lambda(x;\overline{\pi}^T)=\inf_{\pi}J_T^\lambda(x;\pi)\equiv V_T^\lambda(x).

If such a distribution process exists, it is called an optimal exploratory control distribution of Problem (ESLQ)$_T$, and the corresponding state process is denoted by X‾T\overline{X}^T.

Our main question concerns the long-time behavior of the optimal pair (X‾T,π‾T)(\overline{X}^{T},\overline{\pi}^{T}) as T→∞T \to\infty. More precisely, we seek static or stationary reference objects that are independent of TT, such that the finite-horizon optimal processes remain close to these objects for most of the interval [0,T][0,T]. Since the control object in this paper is a probability distribution, a reasonable turnpike property should compare not only the state and the mean control but also the full control distributions. Thus, we use the Wasserstein distance W2W_2 to measure the difference between control distributions.

The main result of this paper is that, under the stabilizable condition (H1) and the joint coercivity condition (H2), we establish an estimate of the form

E∣X‾T(t)−X∗(t)∣2+EW22(π‾tT,πt∗)≤K(e−μt+e−μ(T−t)),t∈[0,T],(4)\mathbb{E}\lvert\overline{X}^{T}(t)-X^{*}(t)\rvert^{2}+\mathbb{E}W_{2}^{2}(\overline{\pi}^{T}_{t},\pi^{*}_{t})\le K\left(e^{-\mu t}+e^{-\mu(T-t)}\right),\qquad t\in[0,T], \tag*{(4)}

where K,μ>0K,\mu>0 are independent of T,tT,t, and the reference process X∗X^{*} and control distributions π∗\pi^{*} are constructed from the static optimizer and the limiting feedback parameters (see Theorem 5.10). This estimate is called a turnpike property for the exploratory stochastic LQ problem. It means that, away from the initial and terminal endpoints, the finite-horizon optimal objects are exponentially close to the reference objects (see Corollary 5.11).

The study of turnpike properties began with long-run economic growth problems [7, 26, 28], and has since been developed widely in optimal control, variational problems, and control of partial differential equations; see [8, 24, 30, 32, 40, 41, 42, 47]. For deterministic and discrete-time LQ problems, the static optimization problem, the Riccati equation, and the closed-loop feedback have a clear relation. Hence the turnpike property can be described by a static optimal point; related results can be found in [2, 5, 10, 11]. For stochastic LQ problems, Sun–Wang–Yong [34] studied turnpike properties for stochastic LQ problems and pointed out that the correct form of the static problem is a key difference between the stochastic and deterministic cases. Furthermore, Sun–Yong [36] proved turnpike properties for mean-field LQ problems and stressed that the proper reference object in stochastic systems is usually not a deterministic point, but a TT-independent stochastic reference process. Further studies of stochastic LQ turnpike properties can be found in [1, 17, 27, 31, 37].

Alongside these developments in turnpike theory, the entropy-regularized exploratory-control framework of Wang–Zariphopoulou–Zhou [44] has led to a growing literature on reinforcement learning. In particular, Jia–Zhou [21, 22, 23] systematically developed a martingale approach to policy evaluation and temporal-difference learning, and subsequently established policy-gradient, actor–critic, and continuous-time qq-learning theories under the exploratory formulation. In LQ settings, Hambly–Xu–Yang [15] studied policy-gradient methods for finite-horizon noisy problems in discrete time. For entropy-regularized LQ problems, Guo–Li–Xu [13] and Giegrich–Reisinger–Zhang [9] analyzed policy learning in discrete and continuous time, respectively. Huang–Jia–Zhou [18] obtained a sublinear regret bound for a model-free algorithm, and Spruch–Treanțapleț–Zhang [38] studied the optimal scheduling of the entropy regularizer. Huang–Zhou [20] further proposed a data-driven adaptive exploration mechanism. The literature also includes studies of mean-field games [12, 14] and applications to mean–variance portfolio selection [4, 19, 45]. These works mainly study learning and exploration mechanisms. This paper focuses on the turnpike property, namely the estimate (4), for exploratory stochastic LQ control. This can be viewed as a fine-grained behavioral characterization of the optimal state processes and optimal control distributions.

Compared with existing results, our work has two main features.

  1. In the exploratory framework, the control variable is a process of probability distributions. Thus, the turnpike property of the controls in our paper needs to be established in the Wasserstein space.

To this end, we propose an exploratory static problem defined over general distributions rather than over the Euclidean space as in [34]. Specifically, it takes the following form:

min⁡x,π∫RmF0(x,u) π(du)−λEnt⁡(π)s.t.Ax+Buˉπ+b=0,(5)\begin{aligned} \min_{x,\pi}\quad&\int_{\mathbb{R}^{m}} F_0(x,u)\,\pi(du)-\lambda\operatorname{Ent}(\pi)\\ \text{s.t.}\quad&Ax+B\bar{u}_{\pi}+b=0, \tag*{(5)} \end{aligned}

where

F0(x,u)=x⊤Qx+2x⊤S⊤u+u⊤Ru+2q⊤x+2r⊤u+(Cx+Du+σ)⊤P∞(Cx+Du+σ),(6)F_0(x,u)=x^{\top}Qx+2x^{\top}S^{\top}u+u^{\top}Ru+2q^{\top}x+2r^{\top}u+(Cx+Du+\sigma)^{\top}P_{\infty}(Cx+Du+\sigma), \tag*{(6)}

with P∞P_{\infty} being the unique stabilizing solution of the algebraic Riccati equation (22), and

References

  1. [1]E. Bayraktar and J. Jian, Convergence and turnpike properties of linear-quadratic mean field control problems with common noise, arXiv preprint, arXiv:2601.07815, 2026.arxiv.org/abs/2601.07815
  2. [2]T. Breiten and L. Pfeiffer, On the turnpike property and the receding-horizon method for linear-quadratic optimal control problems, SIAM J. Control Optim., 58 (2020), pp. 1077–1102.arxiv.org/abs/1811.02421
  3. [3]T. M. Cover and J. A. Thomas, Elements of Information Theory, 2nd ed., Wiley-Interscience, 2006.
  4. [4]M. Dai, Y. Dong, and Y. Jia, Learning equilibrium mean-variance strategy, Math. Finance, 33 (2023), pp. 1166–1212.
  5. [5]T. Damm, L. Grüne, M. Stieler, and K. Worthmann, An exponential turnpike theorem for dissipative discrete time optimal control problems, SIAM J. Control Optim., 52 (2014), pp. 1935–1957.DOI
  6. [6]T. Damm and B. Jacob, On coercivity and the frequency domain condition in indefinite LQ-control, Ann. Acad. Rom. Sci. Ser. Math. Appl., 12 (2020), pp. 553–563.arxiv.org/abs/2010.07401
  7. [7]R. Dorfman, P. A. Samuelson, and R. M. Solow, Linear Programming and Economic Analysis, McGraw-Hill, New York, 1958.
  8. [8]T. Faulwasser and L. Grüne, Turnpike properties in optimal control: An overview of discrete-time and continuous-time results, in Numerical Control: Part A, (eds E. Trélat and E. Zuazua), Handbook of Numerical Analysis, 23, North-Holland, Amsterdam, (2022), pp. 367–400.
  9. [9]M. Giegrich, C. Reisinger, and Y. Zhang, Convergence of policy gradient methods for finite-horizon exploratory linear-quadratic control problems, SIAM J. Control Optim., 62 (2024), pp. 1060–1092.arxiv.org/abs/2211.00617
  10. [10]L. Grüne and R. Guglielmi, Turnpike properties and strict dissipativity for discrete time linear quadratic optimal control problems, SIAM J. Control Optim., 56 (2018), pp. 1282–1302.DOI
  11. [11]L. Grüne and R. Guglielmi, On the relation between turnpike properties and dissipativity for continuous time linear quadratic optimal control problems, Math. Control & Relat. Fields, 11 (2021), pp. 169–188.DOI
  12. [12]X. Guo, A. Hu, R. Xu, and J. Zhang, A general framework for learning mean-field games, Math. Oper. Res., 48 (2023), pp. 656–686.arxiv.org/abs/2003.06069
  13. [13]X. Guo, X. Li, and R. Xu, Fast policy learning for linear-quadratic control with entropy regularization, SIAM J. Control Optim., 64 (2026), pp. 124–151.
  14. [14]X. Guo, R. Xu, and T. Zariphopoulou, Entropy regularization for mean field games with learning, Math. Oper. Res., 47 (2022), pp. 3239–3260.
  15. [15]B. Hambly, R. Xu, and H. Yang, Policy gradient methods for the noisy linear quadratic regulator over a finite horizon, SIAM J. Control Optim., 59 (2021), pp. 3359–3391.
  16. [16]N. J. Higham, Functions of Matrices: Theory and Computation, SIAM, Philadelphia, 2008.
  17. [17]R. Hu and H. Wu, Turnpike properties for stochastic linear–quadratic optimal control problems with partial observation, Statist. Probab. Lett., 233 (2026), Art. 110660.DOI
  18. [18]Y. Huang, Y. Jia, and X. Y. Zhou, Sublinear regret for a class of continuous-time linear–quadratic reinforcement learning problems, SIAM J. Control Optim., 63 (2025), pp. 3452–3474.arxiv.org/abs/2407.17226
  19. [19]Y. Huang, Y. Jia, and X. Y. Zhou, Mean-variance portfolio selection by continuous-time reinforcement learning: Algorithms, regret analysis, and empirical study, arXiv preprint, arXiv:2412.16175v3, 2026.arxiv.org/abs/2412.16175
  20. [20]Y. Huang and X. Y. Zhou, Data-driven exploration for a class of continuous-time indefinite linear–quadratic reinforcement learning problems, IEEE Trans. Automat. Control, 2026, pp. 1–16, Early Access.
  21. [21]Y. Jia and X. Y. Zhou, Policy evaluation and temporal-difference learning in continuous time and space: A martingale approach, J. Mach. Learn. Res., 23 (2022), no. 154, pp. 1–55.arxiv.org/abs/2108.06655
  22. [22]Y. Jia and X. Y. Zhou, Policy gradient and actor-critic learning in continuous time and space: Theory and algorithms, J. Mach. Learn. Res., 23 (2022), no. 275, pp. 1–50.arxiv.org/abs/2111.11232
  23. [23]Y. Jia and X. Y. Zhou, q-learning in continuous time, J. Mach. Learn. Res., 24 (2023), no. 161, pp. 1–61.arxiv.org/abs/2207.00713
  24. [24]M. E. Lewis and A. Paul, Uniform turnpike theorems for finite Markov decision processes, Math. Oper. Res., 44 (2019), pp. 1145–1160.
  25. [25]H. Liu, J. Hu, Y. Li, and Z. Wen, Optimization: Modeling, Algorithm and Theory, Higher Education Press, Beijing, 2020 (in Chinese).
  26. [26]L. W. McKenzie, Turnpike theory, Econometrica, 44 (1976), pp. 841–865.
  27. [27]H. Mei, R. Wang, and J. Yong, Turnpike property of stochastic linear–quadratic optimal control problems in large horizons with regime switching I: Homogeneous cases, arXiv preprint ,arXiv:2506.09337, 2025.arxiv.org/abs/2506.09337
  28. [28]J. von Neumann, A model of general economic equilibrium, Rev. Econ. Stud., 13 (1945), pp. 1–9.
  29. [29]G. Peyré and M. Cuturi, Computational optimal transport: With applications to data science, Foundations and Trends in Machine Learning, 11 (2019), pp. 355–607.
  30. [30]A. Porretta and E. Zuazua, Long time versus steady state optimal control, SIAM J. Control Optim., 51 (2013), pp. 4242–4273.DOI
  31. [31]J. Schiebl, R. Ou, T. Faulwasser, M. H. Baumann, and L. Grüne, Turnpike and dissipativity in generalized discrete-time stochastic linear–quadratic optimal control, SIAM J. Control Optim., 63 (2025), pp. 1432–1457.
  32. [32]S. Sethi, H. M. Soner, Q. Zhang, and H. Jiang, Turnpike sets and their analysis in stochastic production planning problems, Math. Oper. Res., 17 (1992), pp. 932–950.DOI
  33. [33]J. Sun, X. Li, and J. Yong, Open-loop and closed-loop solvabilities for stochastic linear quadratic optimal control problems, SIAM J. Control Optim., 54 (2016), pp. 2274–2308.arxiv.org/abs/1508.02163
  34. [34]J. Sun, H. Wang, and J. Yong, Turnpike properties for stochastic linear–quadratic optimal control problems, Chinese Ann. Math. Ser. B, 43 (2022), pp. 999–1022.arxiv.org/abs/2202.12699
  35. [35]J. Sun and J. Yong, Stochastic linear quadratic optimal control problems in infinite horizon, Appl. Math. Optim., 78 (2018), pp. 145–183.arxiv.org/abs/1610.05021
  36. [36]J. Sun and J. Yong, Turnpike properties for mean-field linear-quadratic optimal control problems, SIAM J. Control Optim., 62 (2024), pp. 752–775.arxiv.org/abs/2209.11466
  37. [37]J. Sun and J. Yong, Long-time behavior of zero-sum linear-quadratic stochastic differential games, SIAM J. Control Optim., 63 (2025), pp. 3961–3989.arxiv.org/abs/2406.02089
  38. [38]L. Szpruch, T. Tchuindjep, and Y. Zhang, Optimal scheduling of entropy regularizer for continuous-time linear-quadratic reinforcement learning, SIAM J. Control Optim., 62 (2024), pp. 135–166.DOI
  39. [39]W. Tang, Y. P. Zhang, and X. Y. Zhou, Exploratory HJB equations and their convergence, SIAM J. Control Optim., 60 (2022), pp. 3191–3216.arxiv.org/abs/2109.10269
  40. [40]E. Trélat and C. Zhang, Integral and measure-turnpike properties for infinite-dimensional optimal control systems, Math. Control Signals Syst., 30 (2018), Art. 3.arxiv.org/abs/1705.02762
  41. [41]E. Trélat and E. Zuazua, Turnpike in optimal control and beyond: a survey, arXiv preprint, arXiv:2503.20342, 2025.arxiv.org/abs/2503.20342
  42. [42]E. Trélat and E. Zuazua, The turnpike property in finite-dimensional nonlinear optimal control, J. Differential Equations, 258 (2015), pp. 81–114.arxiv.org/abs/1402.3263
  43. [43]C. Villani, Optimal Transport: Old and New, Grundlehren Math. Wiss. 338, Springer, Berlin, Heidelberg, 2009.
  44. [44]H. Wang, T. Zariphopoulou, and X. Y. Zhou, Reinforcement learning in continuous time and space: A stochastic control approach, J. Mach. Learn. Res., 21 (2020), no. 198, pp. 1–34.
  45. [45]H. Wang and X. Y. Zhou, Continuous-time mean-variance portfolio selection: A reinforcement learning framework, Math. Finance, 30 (2020), pp. 1273–1308.
  46. [46]J. Yong, Optimization Theory: A Concise Introduction, World Scientific, 2018.
  47. [47]A. J. Zaslavski, Turnpike Properties in the Calculus of Variations and Optimal Control, Non-convex Optim. Appl. 80, Springer, New York, 2006.

Paper details

vlnc:W_er2z9auw
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 1

A view counts once per browser per day, a download each time the PDF is requested.

  • Mathematics

AI-assisted tools

View statement
Looking for related papers…
Contents