Introduction
Let ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) be a complete probability space on which the Brownian motions used in this paper are defined. Here W = { W ( t ) : t ≥ 0 } W = \{W(t): t \ge0\} W = { W ( t ) : t ≥ 0 } is a standard one-dimensional Brownian motion, W ^ = { W ^ ( t ) : t ≥ 0 } \widehat{W} = \{\widehat{W}(t): t \ge0\} W = { W ( t ) : t ≥ 0 } is a standard n n n -dimensional Brownian motion, and W ~ = { W ~ ( t ) : t ≥ 0 } \widetilde{W} = \{\widetilde{W}(t): t \ge0\} W = { W ( t ) : t ≥ 0 } is a standard m m m -dimensional Brownian motion. We assume that W W W and W ~ \widetilde{W} W are independent. Let F = { F t } t ≥ 0 \mathbb{F} = \{\mathcal{F}_t\}_{t \ge0} F = { F t } t ≥ 0 be the usual augmentation of the natural filtration generated by these Brownian motions. All random variables and processes appearing below are assumed to be measurable and adapted to F \mathbb{F} F ; admissible control distribution processes are F \mathbb{F} F -progressively measurable.
This paper studies continuous-time exploratory stochastic linear-quadratic (LQ) control problems. We start from a general controlled diffusion. Consider the stochastic differential equation (SDE) with point-valued controls
{ d X u ( t ) = b ( X u ( t ) , u ( t ) ) d t + σ ( X u ( t ) , u ( t ) ) d W ( t ) , t ≥ 0 , X u ( 0 ) = x , \left\{
\begin{aligned}
dX^{u}(t) &= b(X^{u}(t),u(t))dt+\sigma(X^{u}(t),u(t))dW(t), \qquad t \ge0,\\
X^{u}(0) &= x,
\end{aligned}
\right. { d X u ( t ) X u ( 0 ) = b ( X u ( t ) , u ( t )) d t + σ ( X u ( t ) , u ( t )) d W ( t ) , t ≥ 0 , = x ,
where b b b and σ \sigma σ are the drift and diffusion coefficients. The usual measurability, growth, and integrability conditions are assumed to make this state equation well posed.
Following Wang–Zariphopoulou–Zhou [44 ] , for a control distribution π ∈ P ( U ) \pi\in\mathcal{P}(U) π ∈ P ( U ) and a pointwise running cost ρ ( x , u ) \rho(x,u) ρ ( x , u ) , we define the averaged drift, diffusion covariance, and running cost, as well as the differential entropy when π \pi π has density p p p :
b ~ ( x , π ) = ∫ U b ( x , u ) π ( d u ) , ρ ~ ( x , π ) = ∫ U ρ ( x , u ) π ( d u ) , \widetilde{b}(x,\pi)=\int_U b(x,u)\,\pi(du), \qquad\widetilde{\rho}(x,\pi)=\int_U \rho(x,u)\,\pi(du), b ( x , π ) = ∫ U b ( x , u ) π ( d u ) , ρ ( x , π ) = ∫ U ρ ( x , u ) π ( d u ) ,
σ ~ ( x , π ) σ ~ ( x , π ) T = ∫ U σ ( x , u ) σ ( x , u ) T π ( d u ) , Ent ( π ) = − ∫ U p ( u ) log p ( u ) d u . \widetilde{\sigma}(x,\pi)\widetilde{\sigma}(x,\pi)^{\mathsf{T}}=\int_U \sigma(x,u)\sigma(x,u)^{\mathsf{T}}\,\pi(du), \qquad\operatorname{Ent}(\pi)=-\int_U p(u)\log p(u)\,du. σ ( x , π ) σ ( x , π ) T = ∫ U σ ( x , u ) σ ( x , u ) T π ( d u ) , Ent ( π ) = − ∫ U p ( u ) log p ( u ) d u .
The exploratory state equation is
{ d X π ( t ) = b ~ ( X π ( t ) , π t ) d t + σ ~ ( X π ( t ) , π t ) d W ~ ( t ) , X π ( 0 ) = x . (1) \begin{cases}
dX^\pi(t)=\widetilde{b}(X^\pi(t),\pi_t)dt+\widetilde{\sigma}(X^\pi(t),\pi_t)d\widetilde{W}(t),\\
X^\pi(0)=x. \tag*{(1)}
\end{cases} { d X π ( t ) = b ( X π ( t ) , π t ) d t + σ ( X π ( t ) , π t ) d W ( t ) , X π ( 0 ) = x . (1)
We now specialize to the LQ setting with U = R m U=\mathbb{R}^m U = R m :
b ( x , u ) = A x + B u + b , σ ( x , u ) = C x + D u + σ , b(x,u)=Ax+Bu+b,\qquad\sigma(x,u)=Cx+Du+\sigma, b ( x , u ) = A x + B u + b , σ ( x , u ) = C x + D u + σ ,
ρ ( x , u ) = x T Q x + 2 x T S T u + u T R u + 2 q T x + 2 r T u . (2) \rho(x,u)=x^{\mathsf{T}}Qx+2x^{\mathsf{T}}S^{\mathsf{T}}u+u^{\mathsf{T}}Ru+2q^{\mathsf{T}}x+2r^{\mathsf{T}}u. \tag*{(2)} ρ ( x , u ) = x T Q x + 2 x T S T u + u T R u + 2 q T x + 2 r T u . (2)
In the exploratory framework, the control variable is a process of probability distributions π = { π t : t ∈ [ 0 , T ] } \pi=\{\pi_t:t\in[0,T]\} π = { π t : t ∈ [ 0 , T ]} on the control space R m \mathbb{R}^m R m . Denote the mean and covariance of π t \pi_t π t by
u ‾ ( t ) = ∫ R m u π t ( d u ) , Λ ( t ) = ∫ R m ( u − u ‾ ( t ) ) ( u − u ‾ ( t ) ) T π t ( d u ) . (3) \overline{u}(t)=\int_{\mathbb{R}^m}u\,\pi_t(du),\qquad\Lambda(t)=\int_{\mathbb{R}^m}(u-\overline{u}(t))(u-\overline{u}(t))^{\mathsf{T}}\pi_t(du). \tag*{(3)} u ( t ) = ∫ R m u π t ( d u ) , Λ ( t ) = ∫ R m ( u − u ( t )) ( u − u ( t ) ) T π t ( d u ) . (3)
Substituting the LQ coefficients (2) and the control moments (3) into the general exploratory state equation (1) gives the following equivalent exploratory LQ state equation:
{ d X π ( t ) = ( A X π ( t ) + B u ‾ ( t ) + b ) d t + ( C X π ( t ) + D u ‾ ( t ) + σ ) d W ( t ) + D Λ ( t ) 1 / 2 d W ~ ( t ) , t ∈ [ 0 , T ] , X π ( 0 ) = x . \begin{cases}
dX^\pi(t)=(AX^\pi(t)+B\overline{u}(t)+b)dt+(CX^\pi(t)+D\overline{u}(t)+\sigma)dW(t)\\
\qquad{}+D\Lambda(t)^{1/2}d\widetilde{W}(t),\qquad t\in[0,T],\\
X^\pi(0)=x.
\end{cases} ⎩ ⎨ ⎧ d X π ( t ) = ( A X π ( t ) + B u ( t ) + b ) d t + ( C X π ( t ) + D u ( t ) + σ ) d W ( t ) + D Λ ( t ) 1/2 d W ( t ) , t ∈ [ 0 , T ] , X π ( 0 ) = x .
Here W W W represents the original system noise and W ~ \widetilde{W} W represents the additional noise generated by the covariance of the randomized control.
The entropy-regularized cost on [ 0 , T ] [0,T] [ 0 , T ] is
J T λ ( x ; π ) = E ∫ 0 T ∫ R m [ X π ( t ) T Q X π ( t ) + 2 X π ( t ) T S T u + u T R u + 2 q T X π ( t ) + 2 r T u ] π t ( d u ) d t − λ E ∫ 0 T Ent ( π t ) d t , J_T^\lambda(x;\pi)=\mathbb{E}\int_0^T\int_{\mathbb{R}^m}\left[X^\pi(t)^{\mathsf{T}}QX^\pi(t)+2X^\pi(t)^{\mathsf{T}}S^{\mathsf{T}}u+u^{\mathsf{T}}Ru+2q^{\mathsf{T}}X^\pi(t)+2r^{\mathsf{T}}u\right]\pi_t(du)\,dt-\lambda\mathbb{E}\int_0^T\operatorname{Ent}(\pi_t)\,dt, J T λ ( x ; π ) = E ∫ 0 T ∫ R m [ X π ( t ) T Q X π ( t ) + 2 X π ( t ) T S T u + u T R u + 2 q T X π ( t ) + 2 r T u ] π t ( d u ) d t − λ E ∫ 0 T Ent ( π t ) d t ,
where λ > 0 \lambda>0 λ > 0 is the exploration weight. Since the cost is minimized, the entropy term encourages exploration: a larger λ \lambda λ gives more weight to exploration, while a smaller λ \lambda λ brings the model closer to the classical point-control problem.
The finite-horizon exploratory LQ problem can be stated as follows.
Problem (ESLQ)T _T T . For a given initial state x ∈ R n x\in\mathbb{R}^n x ∈ R n , find π ‾ T \overline{\pi}^T π T in the admissible class of exploratory control distributions such that
J T λ ( x ; π ‾ T ) = inf π J T λ ( x ; π ) ≡ V T λ ( x ) . J_T^\lambda(x;\overline{\pi}^T)=\inf_{\pi}J_T^\lambda(x;\pi)\equiv V_T^\lambda(x). J T λ ( x ; π T ) = π inf J T λ ( x ; π ) ≡ V T λ ( x ) . If such a distribution process exists, it is called an optimal exploratory control distribution of Problem (ESLQ)$_T$, and the corresponding state process is denoted by X ‾ T \overline{X}^T X T .
Our main question concerns the long-time behavior of the optimal pair ( X ‾ T , π ‾ T ) (\overline{X}^{T},\overline{\pi}^{T}) ( X T , π T ) as T → ∞ T \to\infty T → ∞ . More precisely, we seek static or stationary reference objects that are independent of T T T , such that the finite-horizon optimal processes remain close to these objects for most of the interval [ 0 , T ] [0,T] [ 0 , T ] . Since the control object in this paper is a probability distribution, a reasonable turnpike property should compare not only the state and the mean control but also the full control distributions. Thus, we use the Wasserstein distance W 2 W_2 W 2 to measure the difference between control distributions.
The main result of this paper is that, under the stabilizable condition (H1) and the joint coercivity condition (H2), we establish an estimate of the form
E ∣ X ‾ T ( t ) − X ∗ ( t ) ∣ 2 + E W 2 2 ( π ‾ t T , π t ∗ ) ≤ K ( e − μ t + e − μ ( T − t ) ) , t ∈ [ 0 , T ] , (4) \mathbb{E}\lvert\overline{X}^{T}(t)-X^{*}(t)\rvert^{2}+\mathbb{E}W_{2}^{2}(\overline{\pi}^{T}_{t},\pi^{*}_{t})\le K\left(e^{-\mu t}+e^{-\mu(T-t)}\right),\qquad t\in[0,T], \tag*{(4)} E ∣ X T ( t ) − X ∗ ( t ) ∣ 2 + E W 2 2 ( π t T , π t ∗ ) ≤ K ( e − μ t + e − μ ( T − t ) ) , t ∈ [ 0 , T ] , (4)
where K , μ > 0 K,\mu>0 K , μ > 0 are independent of T , t T,t T , t , and the reference process X ∗ X^{*} X ∗ and control distributions π ∗ \pi^{*} π ∗ are constructed from the static optimizer and the limiting feedback parameters (see Theorem 5.10 ). This estimate is called a turnpike property for the exploratory stochastic LQ problem. It means that, away from the initial and terminal endpoints, the finite-horizon optimal objects are exponentially close to the reference objects (see Corollary 5.11 ).
The study of turnpike properties began with long-run economic growth problems [7 , 26 , 28 ] , and has since been developed widely in optimal control, variational problems, and control of partial differential equations; see [8 , 24 , 30 , 32 , 40 , 41 , 42 , 47 ] . For deterministic and discrete-time LQ problems, the static optimization problem, the Riccati equation, and the closed-loop feedback have a clear relation. Hence the turnpike property can be described by a static optimal point; related results can be found in [2 , 5 , 10 , 11 ] . For stochastic LQ problems, Sun–Wang–Yong [34 ] studied turnpike properties for stochastic LQ problems and pointed out that the correct form of the static problem is a key difference between the stochastic and deterministic cases. Furthermore, Sun–Yong [36 ] proved turnpike properties for mean-field LQ problems and stressed that the proper reference object in stochastic systems is usually not a deterministic point, but a T T T -independent stochastic reference process. Further studies of stochastic LQ turnpike properties can be found in [1 , 17 , 27 , 31 , 37 ] .
Alongside these developments in turnpike theory, the entropy-regularized exploratory-control framework of Wang–Zariphopoulou–Zhou [44 ] has led to a growing literature on reinforcement learning. In particular, Jia–Zhou [21 , 22 , 23 ] systematically developed a martingale approach to policy evaluation and temporal-difference learning, and subsequently established policy-gradient, actor–critic, and continuous-time q q q -learning theories under the exploratory formulation. In LQ settings, Hambly–Xu–Yang [15 ] studied policy-gradient methods for finite-horizon noisy problems in discrete time. For entropy-regularized LQ problems, Guo–Li–Xu [13 ] and Giegrich–Reisinger–Zhang [9 ] analyzed policy learning in discrete and continuous time, respectively. Huang–Jia–Zhou [18 ] obtained a sublinear regret bound for a model-free algorithm, and Spruch–Treanțapleț–Zhang [38 ] studied the optimal scheduling of the entropy regularizer. Huang–Zhou [20 ] further proposed a data-driven adaptive exploration mechanism. The literature also includes studies of mean-field games [12 , 14 ] and applications to mean–variance portfolio selection [4 , 19 , 45 ] . These works mainly study learning and exploration mechanisms. This paper focuses on the turnpike property, namely the estimate (4) , for exploratory stochastic LQ control. This can be viewed as a fine-grained behavioral characterization of the optimal state processes and optimal control distributions.
Compared with existing results, our work has two main features.
In the exploratory framework, the control variable is a process of probability distributions. Thus, the turnpike property of the controls in our paper needs to be established in the Wasserstein space.
To this end, we propose an exploratory static problem defined over general distributions rather than over the Euclidean space as in [34 ] . Specifically, it takes the following form:
min x , π ∫ R m F 0 ( x , u ) π ( d u ) − λ Ent ( π ) s.t. A x + B u ˉ π + b = 0 , (5) \begin{aligned}
\min_{x,\pi}\quad&\int_{\mathbb{R}^{m}} F_0(x,u)\,\pi(du)-\lambda\operatorname{Ent}(\pi)\\
\text{s.t.}\quad&Ax+B\bar{u}_{\pi}+b=0, \tag*{(5)}
\end{aligned} x , π min s.t. ∫ R m F 0 ( x , u ) π ( d u ) − λ Ent ( π ) A x + B u ˉ π + b = 0 , (5)
where
F 0 ( x , u ) = x ⊤ Q x + 2 x ⊤ S ⊤ u + u ⊤ R u + 2 q ⊤ x + 2 r ⊤ u + ( C x + D u + σ ) ⊤ P ∞ ( C x + D u + σ ) , (6) F_0(x,u)=x^{\top}Qx+2x^{\top}S^{\top}u+u^{\top}Ru+2q^{\top}x+2r^{\top}u+(Cx+Du+\sigma)^{\top}P_{\infty}(Cx+Du+\sigma), \tag*{(6)} F 0 ( x , u ) = x ⊤ Q x + 2 x ⊤ S ⊤ u + u ⊤ R u + 2 q ⊤ x + 2 r ⊤ u + ( C x + D u + σ ) ⊤ P ∞ ( C x + D u + σ ) , (6)
with P ∞ P_{\infty} P ∞ being the unique stabilizing solution of the algebraic Riccati equation (22) , and
u ˉ π = ∫ R m u π ( d u ) , Λ π = ∫ R m ( u − u ˉ π ) ( u − u ˉ π ) ⊤ π ( d u ) , Ent ( π ) = − ∫ R m p ( u ) log p ( u ) d u , \bar{u}_{\pi}=\int_{\mathbb{R}^{m}}u\,\pi(du),\qquad\Lambda_{\pi}=\int_{\mathbb{R}^{m}}(u-\bar{u}_{\pi})(u-\bar{u}_{\pi})^{\top}\pi(du),\qquad\operatorname{Ent}(\pi)=-\int_{\mathbb{R}^{m}}p(u)\log p(u)\,du, u ˉ π = ∫ R m u π ( d u ) , Λ π = ∫ R m ( u − u ˉ π ) ( u − u ˉ π ) ⊤ π ( d u ) , Ent ( π ) = − ∫ R m p ( u ) log p ( u ) d u ,
with π ( d u ) = p ( u ) d u \pi(du)=p(u)\,du π ( d u ) = p ( u ) d u .
Note that in (5) , π \pi π ranges over proper probability distributions, and the equality constraint is expressed in terms of the mean of π \pi π . Thus, it is still an infinite-dimensional constrained optimization problem. Using [3 Theorem 8.6.5] (see Lemma 3.3 ), we show that the exploratory static distribution optimization problem is equivalent to the following finite-dimensional one:
min x , u , Λ F 0 ( x , u ) + tr ( M ∞ Λ ) − λ 2 log ( ( 2 π e ) m det Λ ) s.t. A x + B u + b = 0 , (7) \begin{aligned}
\min_{x,u,\Lambda}\quad&F_0(x,u)+\operatorname{tr}(M_{\infty}\Lambda)-\frac{\lambda}{2}\log\left((2\pi e)^{m}\det\Lambda\right)\\
\text{s.t.}\quad&Ax+Bu+b=0, \tag*{(7)}
\end{aligned} x , u , Λ min s.t. F 0 ( x , u ) + tr ( M ∞ Λ ) − 2 λ log ( ( 2 π e ) m det Λ ) A x + B u + b = 0 , (7)
where M ∞ : = R + D ⊤ P ∞ D M_{\infty}:=R+D^{\top}P_{\infty}D M ∞ := R + D ⊤ P ∞ D , and u u u and Λ \Lambda Λ are the mean and covariance of the distribution π \pi π , respectively.
(ii) The second feature is that the cost functional is indefinite, meaning that the weighting matrices need not be positive semidefinite. We remark that the indefinite case is also novel even for the turnpike property of classical stochastic LQ control problems without exploration.
We assume that the controlled system [ A , C ; B , D ] [A,C;B,D] [ A , C ; B , D ] is stabilizable (see (H1)), which includes the stable system as a special case. A main contribution of this paper is the introduction of the so-called joint coercivity condition (see (H2)). On each fixed finite horizon, this condition is equivalent to the well-known uniform convexity condition [33 ] , with constants that may depend on the horizon (see Remark 2.10 ). If the uncontrolled system is L 2 L^{2} L 2 -stable, joint coercivity is equivalent to uniform convexity with a constant independent of the horizon (see Proposition 2.11 ). Thus, the joint coercivity condition is a natural extension of the uniform convexity condition, enabling us to handle infinite-horizon stabilizable systems in the indefinite cost functional case.
The indefinite cost functional gives rise to several significant difficulties in establishing the turnpike property. Since the matrices Q Q Q and P ∞ P_{\infty} P ∞ in (6) may be indefinite, solving the static optimization problem (7) is itself nontrivial. By a careful convexity analysis of F 0 F_0 F 0 , we show that problem (7) still admits a unique solution ( x ∗ , u ∗ , Λ ∗ ) (x_{*},u_{*},\Lambda_{*}) ( x ∗ , u ∗ , Λ ∗ ) , even when P ∞ P_{\infty} P ∞ is indefinite (see Propositions 3.1 and 3.4 ). Then, under Assumptions (H1)–(H2), we prove that (5) admits a unique optimal solution ( x ∗ , π ∗ λ ) (x_{*},\pi_{*}^{\lambda}) ( x ∗ , π ∗ λ ) , with the optimal control distribution being Gaussian, without assuming Gaussianity a priori:
π ∗ λ = N ( u ∗ , λ 2 ( R + D ⊤ P ∞ D ) − 1 ) . \pi_{*}^{\lambda}=\mathcal{N}\left(u_{*},\frac{\lambda}{2}(R+D^{\top}P_{\infty}D)^{-1}\right). π ∗ λ = N ( u ∗ , 2 λ ( R + D ⊤ P ∞ D ) − 1 ) .
This shows that the exploratory regularization does not shift the static center. Instead, it yields Gaussian exploration around the static control point u ∗ u_{*} u ∗ , with covariance determined by λ \lambda λ and P ∞ P_{\infty} P ∞ .
To establish the turnpike property (4) (see Theorem 5.10 ), we use the unique optimal solution ( x ∗ , π ∗ λ ) (x_{*},\pi_{*}^{\lambda}) ( x ∗ , π ∗ λ ) to problem (5) , establish the convergence of the Riccati equations (see Lemma 4.1 and Proposition 4.2 ), and exploit certain refined properties of the Wasserstein distance between Gaussian distributions (see Lemma 5.7 ). It is worth pointing out that, since the cost functional is indefinite, the method employed by Sun–Wang–Yong [34 ] cannot be applied to establish the convergence of the Riccati equations in Lemma 4.1 and Proposition 4.2 . We transform the stabilizable system into a stable one and, inspired by Sun–Yong [37 ] , overcome this difficulty by means of a Hilbert-space approach. More importantly, we prove that the convergence of the Riccati equations is equivalent to the satisfaction of the joint coercivity condition (see Theorem 4.3 ). Since the convergence of the Riccati equations is fundamental to the turnpike property, this shows that the joint coercivity condition introduced in this paper is sharp.
The remainder of the paper is organized as follows. Subsection 2.1 presents the precise formulation of the exploratory stochastic LQ problem. Subsection 2.2 reviews the stability of stochastic linear systems and derives the optimal feedback for the infinite-horizon stochastic LQ problem. Section 3 establishes the existence and uniqueness of the solution to the classical static problem without requiring the cost functional to be positive semidefinite, and constructs its unique Gaussian exploratory extension. Section 4 proves exponential convergence of the solutions of the finite-horizon Riccati equations, the feedback parameters, and the exploration covariance to their infinite-horizon counterparts. Section 5 establishes the turnpike property for the optimal state process and the Wasserstein turnpike property for the optimal control distributions.
Preliminaries
Exploratory Stochastic LQ Problems
We first fix some notation used below. Let R n × m \mathbb{R}^{n \times m} R n × m be the space of n × m n \times m n × m real matrices, with the Frobenius inner product ⟨ M , N ⟩ = tr ( M T N ) \langle M,N\rangle= \operatorname{tr}(M^{\mathsf{T}}N) ⟨ M , N ⟩ = tr ( M T N ) , M , N ∈ R n × m M,N \in\mathbb{R}^{n \times m} M , N ∈ R n × m . The norm induced by this inner product is denoted by ∣ M ∣ |M| ∣ M ∣ or ∥ ⋅ ∥ F \lVert\cdot\rVert_{\mathrm{F}} ∥ ⋅ ∥ F . For matrices, ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ denotes the spectral norm; for linear operators, ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ denotes the induced operator norm. Let S k \mathbb{S}^{k} S k be the space of k × k k \times k k × k real symmetric matrices, and let S + k \mathbb{S}^{k}_{+} S + k and S + + k \mathbb{S}^{k}_{++} S ++ k be the sets of positive semidefinite and positive definite matrices, respectively. For P , Q ∈ S k P,Q \in\mathbb{S}^{k} P , Q ∈ S k , we write P ≥ Q P \ge Q P ≥ Q if P − Q ∈ S + k P-Q \in\mathbb{S}^{k}_{+} P − Q ∈ S + k , and P > Q P > Q P > Q if P − Q ∈ S + + k P-Q \in\mathbb{S}^{k}_{++} P − Q ∈ S ++ k . The identity matrix is denoted by I k I_{k} I k . We use the probability space and filtration introduced in Section 1, and all processes involved are assumed to be F \mathbb{F} F -progressively measurable.
Let P 2 ( R m ) \mathcal{P}_{2}(\mathbb{R}^{m}) P 2 ( R m ) denote the probability distributions on R m \mathbb{R}^{m} R m with finite second moment. For π ∈ P 2 ( R m ) \pi\in\mathcal{P}_{2}(\mathbb{R}^{m}) π ∈ P 2 ( R m ) , define its mean and covariance by
u ‾ ( π ) = ∫ R m u π ( d u ) , Λ ( π ) = ∫ R m ( u − u ‾ ( π ) ) ( u − u ‾ ( π ) ) T π ( d u ) . \overline{u}(\pi)=\int_{\mathbb{R}^{m}}u\,\pi(\mathrm{d}u), \qquad\Lambda(\pi)=\int_{\mathbb{R}^{m}}\left(u-\overline{u}(\pi)\right)\left(u-\overline{u}(\pi)\right)^{\mathsf{T}}\pi(\mathrm{d}u). u ( π ) = ∫ R m u π ( d u ) , Λ ( π ) = ∫ R m ( u − u ( π ) ) ( u − u ( π ) ) T π ( d u ) .
Then ∫ R m u u T π ( d u ) = u ‾ ( π ) u ‾ ( π ) T + Λ ( π ) \int_{\mathbb{R}^{m}}uu^{\mathsf{T}}\,\pi(\mathrm{d}u)=\overline{u}(\pi)\overline{u}(\pi)^{\mathsf{T}}+\Lambda(\pi) ∫ R m u u T π ( d u ) = u ( π ) u ( π ) T + Λ ( π ) . If π \pi π has a Lebesgue density p p p , its differential entropy and the Gaussian entropy formula are
Ent ( π ) = − ∫ R m p ( u ) log p ( u ) d u , Ent ( N ( u , Λ ) ) = 1 2 log ( ( 2 π e ) m det Λ ) , \operatorname{Ent}(\pi)=-\int_{\mathbb{R}^{m}}p(u)\log p(u)\,\mathrm{d}u, \qquad\operatorname{Ent}(\mathcal{N}(u,\Lambda))=\frac{1}{2}\log\left((2\pi e)^{m}\det\Lambda\right), Ent ( π ) = − ∫ R m p ( u ) log p ( u ) d u , Ent ( N ( u , Λ )) = 2 1 log ( ( 2 π e ) m det Λ ) ,
where Λ ∈ S + + m \Lambda\in\mathbb{S}^{m}_{++} Λ ∈ S ++ m in the Gaussian case.
For the finite-horizon LQ problem, let A , C ∈ R n × n A,C \in\mathbb{R}^{n \times n} A , C ∈ R n × n , B , D ∈ R n × m B,D \in\mathbb{R}^{n \times m} B , D ∈ R n × m , b , σ , q ∈ R n b,\sigma,q \in\mathbb{R}^{n} b , σ , q ∈ R n , r ∈ R m r \in\mathbb{R}^{m} r ∈ R m , Q ∈ S n Q \in\mathbb{S}^{n} Q ∈ S n , R ∈ S m R \in\mathbb{S}^{m} R ∈ S m , and S ∈ R m × n S \in\mathbb{R}^{m \times n} S ∈ R m × n . Given a control distribution process π = { π t } 0 ≤ t ≤ T \pi=\{\pi_{t}\}_{0\le t\le T} π = { π t } 0 ≤ t ≤ T , set u ‾ ( t ) = u ‾ ( π t ) \overline{u}(t)=\overline{u}(\pi_{t}) u ( t ) = u ( π t ) and Λ ( t ) = Λ ( π t ) \Lambda(t)=\Lambda(\pi_{t}) Λ ( t ) = Λ ( π t ) . The exploratory state process X π X^{\pi} X π satisfies
d X π ( t ) = ( A X π ( t ) + B u ‾ ( t ) + b ) d t + ( C X π ( t ) + D u ‾ ( t ) + σ ) d W ( t ) + D Λ ( t ) 1 / 2 d W ~ ( t ) , t ∈ [ 0 , T ] , X π ( 0 ) = x . (8) \begin{aligned}
\mathrm{d}X^{\pi}(t) &= \left(AX^{\pi}(t)+B\overline{u}(t)+b\right)\mathrm{d}t+\left(CX^{\pi}(t)+D\overline{u}(t)+\sigma\right)\mathrm{d}W(t) \\
&\quad+D\Lambda(t)^{1/2}\mathrm{d}\widetilde{W}(t), \qquad t\in[0,T], \\
X^{\pi}(0)&=x. \tag*{(8)}
\end{aligned} d X π ( t ) X π ( 0 ) = ( A X π ( t ) + B u ( t ) + b ) d t + ( C X π ( t ) + D u ( t ) + σ ) d W ( t ) + D Λ ( t ) 1/2 d W ( t ) , t ∈ [ 0 , T ] , = x . (8)
For x ∈ R n x\in\mathbb{R}^{n} x ∈ R n and π ∈ P 2 ( R m ) \pi\in\mathcal{P}_{2}(\mathbb{R}^{m}) π ∈ P 2 ( R m ) , define the running cost under the control distribution π \pi π by
ℓ ( x , π ) = ∫ R m [ x T Q x + 2 x T S T u + u T R u + 2 q T x + 2 r T u ] π ( d u ) . \ell(x,\pi)=\int_{\mathbb{R}^{m}}\left[x^{\mathsf{T}}Qx+2x^{\mathsf{T}}S^{\mathsf{T}}u+u^{\mathsf{T}}Ru+2q^{\mathsf{T}}x+2r^{\mathsf{T}}u\right]\pi(\mathrm{d}u). ℓ ( x , π ) = ∫ R m [ x T Q x + 2 x T S T u + u T R u + 2 q T x + 2 r T u ] π ( d u ) .
For a given initial state x ∈ R n x \in\mathbb{R}^{n} x ∈ R n and exploration weight λ > 0 \lambda> 0 λ > 0 , a control distribution process π = { π t } 0 ≤ t ≤ T \pi= \{\pi_t\}_{0 \le t \le T} π = { π t } 0 ≤ t ≤ T belongs to A T λ ( x ) \mathcal{A}^{\lambda}_{T}(x) A T λ ( x ) if it satisfies the following conditions:
(i) for each t ∈ [ 0 , T ] t \in[0,T] t ∈ [ 0 , T ] , π t ∈ P 2 ( R m ) \pi_t \in\mathcal{P}_2(\mathbb{R}^{m}) π t ∈ P 2 ( R m ) and π t \pi_t π t is absolutely continuous with respect to the Lebesgue measure, with density p t p_t p t ;
(ii) for every Borel set E ⊂ R m E \subset\mathbb{R}^{m} E ⊂ R m , the process t ↦ π t ( E ) t \mapsto\pi_t(E) t ↦ π t ( E ) is F \mathbb{F} F -progressively measurable;
(iii) the mean and covariance satisfy E ∫ 0 T ( ∣ u ˉ ( t ) ∣ 2 + tr Λ ( t ) ) d t < ∞ \mathbb{E}\int_0^T \left(|\bar{u}(t)|^2+\operatorname{tr}\Lambda(t)\right)dt < \infty E ∫ 0 T ( ∣ u ˉ ( t ) ∣ 2 + tr Λ ( t ) ) d t < ∞ ;
(iv) the entropy regularization term is integrable, namely E ∫ 0 T ∣ ∫ R m p t ( u ) log p t ( u ) d u ∣ d t < ∞ \mathbb{E}\int_0^T \left|\int_{\mathbb{R}^{m}}p_t(u)\log p_t(u)\,du\right|dt < \infty E ∫ 0 T ∫ R m p t ( u ) log p t ( u ) d u d t < ∞ ;
(v) under π \pi π , the state equation (8) is well posed.
For the linear state equation, condition (iii) yields the usual second-moment estimate for X π X^\pi X π . Hence, together with condition (iv), the above assumptions ensure that all terms in the cost functional below are integrable.
Problem (ESLQ)T _T T . For a given x ∈ R n x \in\mathbb{R}^{n} x ∈ R n , over all π ∈ A T λ ( x ) \pi\in\mathcal{A}^{\lambda}_{T}(x) π ∈ A T λ ( x ) , minimize
J T λ ( x ; π ) = E ∫ 0 T [ ℓ ( X π ( t ) , π t ) + λ ∫ R m p t ( u ) log p t ( u ) d u ] d t . J^\lambda_T(x;\pi)=\mathbb{E}\int_0^T\left[\ell(X^\pi(t),\pi_t)+\lambda\int_{\mathbb{R}^{m}}p_t(u)\log p_t(u)\,du\right]dt. J T λ ( x ; π ) = E ∫ 0 T [ ℓ ( X π ( t ) , π t ) + λ ∫ R m p t ( u ) log p t ( u ) d u ] d t .
The value function is defined by
V T λ ( x ) = inf π ∈ A T λ ( x ) J T λ ( x ; π ) . V^\lambda_T(x)=\inf_{\pi\in\mathcal{A}^{\lambda}_{T}(x)}J^\lambda_T(x;\pi). V T λ ( x ) = π ∈ A T λ ( x ) inf J T λ ( x ; π ) .
Remark 2.1. If there exists π T , λ , ∗ ∈ A T λ ( x ) \pi^{T,\lambda,*}\in\mathcal{A}^{\lambda}_{T}(x) π T , λ , ∗ ∈ A T λ ( x ) such that J T λ ( x ; π T , λ , ∗ ) = V T λ ( x ) J^\lambda_T(x;\pi^{T,\lambda,*})=V^\lambda_T(x) J T λ ( x ; π T , λ , ∗ ) = V T λ ( x ) , then π T , λ , ∗ \pi^{T,\lambda,*} π T , λ , ∗ is called an optimal control distribution of Problem (ESLQ)T _T T .
The exploratory control framework and the entropy-regularized objective were first introduced in [44 ] . The exploratory HJB equation and its well-posedness theory can be found in [39 , 44 ] . In this section, we use these results only in the LQ case. Consider the following Riccati equation:
P ˙ T ( t ) + P T ( t ) A + A T P T ( t ) + C T P T ( t ) C + Q − ( P T ( t ) B + C T P T ( t ) D + S T ) ⋅ ( R + D T P T ( t ) D ) − 1 ( B T P T ( t ) + D T P T ( t ) C + S ) = 0 , t ∈ [ 0 , T ] , P T ( T ) = 0. (9) \begin{aligned}
\dot{P}_T(t)+P_T(t)A+A^{\mathsf{T}}P_T(t)+C^{\mathsf{T}}P_T(t)C+Q
&-\left(P_T(t)B+C^{\mathsf{T}}P_T(t)D+S^{\mathsf{T}}\right)\\
&\cdot\left(R+D^{\mathsf{T}}P_T(t)D\right)^{-1}
\left(B^{\mathsf{T}}P_T(t)+D^{\mathsf{T}}P_T(t)C+S\right)=0,\quad t\in[0,T],\\
P_T(T)&=0. \tag*{(9)}
\end{aligned} P ˙ T ( t ) + P T ( t ) A + A T P T ( t ) + C T P T ( t ) C + Q P T ( T ) − ( P T ( t ) B + C T P T ( t ) D + S T ) ⋅ ( R + D T P T ( t ) D ) − 1 ( B T P T ( t ) + D T P T ( t ) C + S ) = 0 , t ∈ [ 0 , T ] , = 0. (9)
A solution P T ∈ C ( [ 0 , T ] ; S n ) P_T\in C([0,T];\mathbb{S}^{n}) P T ∈ C ([ 0 , T ] ; S n ) of (9) is called strongly regular if there exists a constant δ T > 0 \delta_T>0 δ T > 0 such that
R + D T P T ( t ) D ≥ δ T I m , t ∈ [ 0 , T ] . R+D^{\mathsf{T}}P_T(t)D\ge\delta_T I_m,\quad t\in[0,T]. R + D T P T ( t ) D ≥ δ T I m , t ∈ [ 0 , T ] .
The next proposition gives the Gaussian feedback law and the linear terminal-value equation for φ T \varphi_T φ T . Related results and proof details can be found in [9 , 33 , 39 , 44 ] .
Proposition 2.2. Let P T P_T P T be a strongly regular solution of the Riccati equation (9) . Define
M T ( t ) = R + D T P T ( t ) D , N T ( t ) = B T P T ( t ) + D T P T ( t ) C + S , K T ( t ) = M T ( t ) − 1 N T ( t ) . M_T(t)=R+D^{\mathsf{T}}P_T(t)D,\qquad N_T(t)=B^{\mathsf{T}}P_T(t)+D^{\mathsf{T}}P_T(t)C+S,\qquad K_T(t)=M_T(t)^{-1}N_T(t). M T ( t ) = R + D T P T ( t ) D , N T ( t ) = B T P T ( t ) + D T P T ( t ) C + S , K T ( t ) = M T ( t ) − 1 N T ( t ) . Then the following linear terminal-value problem admits a unique solution φ T \varphi_T φ T :
φ ˙ T ( t ) + ( A − B K T ( t ) ) T φ T ( t ) + ( C − D K T ( t ) ) T P T ( t ) σ − K T ( t ) T r + P T ( t ) b + q = 0 , t ∈ [ 0 , T ] , φ T ( T ) = 0. (10) \begin{aligned}
\dot{\varphi}_T(t)+(A-BK_T(t))^{\mathsf{T}}\varphi_T(t)
+(C-DK_T(t))^{\mathsf{T}}P_T(t)\sigma
-K_T(t)^{\mathsf{T}}r+P_T(t)b+q&=0,\quad t\in[0,T],\\
\varphi_T(T)&=0. \tag*{(10)}
\end{aligned} φ ˙ T ( t ) + ( A − B K T ( t ) ) T φ T ( t ) + ( C − D K T ( t ) ) T P T ( t ) σ − K T ( t ) T r + P T ( t ) b + q φ T ( T ) = 0 , t ∈ [ 0 , T ] , = 0. (10) Set
d T ( t ) = B T φ T ( t ) + D T P T ( t ) σ + r , ℓ T ( t ) = M T ( t ) − 1 d T ( t ) , Λ T λ ( t ) = λ 2 M T ( t ) − 1 . d_T(t)=B^{\mathsf{T}}\varphi_T(t)+D^{\mathsf{T}}P_T(t)\sigma+r,\qquad
\ell_T(t)=M_T(t)^{-1}d_T(t),\qquad
\Lambda^\lambda_T(t)=\frac{\lambda}{2}M_T(t)^{-1}. d T ( t ) = B T φ T ( t ) + D T P T ( t ) σ + r , ℓ T ( t ) = M T ( t ) − 1 d T ( t ) , Λ T λ ( t ) = 2 λ M T ( t ) − 1 . Then the optimal feedback control distribution is given by
π T λ , ∗ ( d u ∣ t , x ) = N ( − K T ( t ) x − ℓ T ( t ) , Λ T λ ( t ) ) ( d u ) . (11) \pi_T^{\lambda,*}(du\mid t,x)=\mathcal{N}\left(-K_T(t)x-\ell_T(t),\Lambda^\lambda_T(t)\right)(du). \tag*{(11)} π T λ , ∗ ( d u ∣ t , x ) = N ( − K T ( t ) x − ℓ T ( t ) , Λ T λ ( t ) ) ( d u ) . (11)
Remark 2.3. By (11) , the mean − K T ( t ) x − ℓ T ( t ) -K_T(t)x-\ell_T(t) − K T ( t ) x − ℓ T ( t ) of the optimal feedback distribution is independent of λ \lambda λ , while its covariance Λ T λ ( t ) = λ 2 M T ( t ) − 1 \Lambda_T^\lambda(t)=\frac{\lambda}{2}M_T(t)^{-1} Λ T λ ( t ) = 2 λ M T ( t ) − 1 is proportional to λ \lambda λ .
Infinite-Horizon Problem
We recall the stability of stochastic linear systems. Consider the homogeneous uncontrolled system
{ d X ( t ) = A X ( t ) d t + C X ( t ) d W ( t ) , t ≥ 0 , X ( 0 ) = x . \begin{cases}
dX(t)=AX(t)dt+CX(t)dW(t), & t\ge0,\\
X(0)=x.
\end{cases} { d X ( t ) = A X ( t ) d t + C X ( t ) d W ( t ) , X ( 0 ) = x . t ≥ 0 ,
For any initial value x ∈ R n x\in\mathbb{R}^n x ∈ R n , denote its solution by X ( ⋅ ; x ) X(\cdot;x) X ( ⋅ ; x ) .
Definition 2.4. (i) The system [ A , C ] [A,C] [ A , C ] is called L 2 L^2 L 2 -stable if, for every x ∈ R n x\in\mathbb{R}^n x ∈ R n ,
E ∫ 0 ∞ ∣ X ( t ; x ) ∣ 2 d t < ∞ . \mathbb{E}\int_0^\infty|X(t;x)|^2dt<\infty. E ∫ 0 ∞ ∣ X ( t ; x ) ∣ 2 d t < ∞. (ii) The system [ A , C ] [A,C] [ A , C ] is called mean-square exponentially stable if there exists β > 0 \beta>0 β > 0 such that, for every x ∈ R n x\in\mathbb{R}^n x ∈ R n ,
sup t ≥ 0 e β t E ∣ X ( t ; x ) ∣ 2 < ∞ . \sup_{t\ge0}e^{\beta t}\mathbb{E}|X(t;x)|^2<\infty. t ≥ 0 sup e β t E ∣ X ( t ; x ) ∣ 2 < ∞.
The following characterization will be used below; see Sun–Yong [35 ] .
Lemma 2.5. The following conditions are equivalent:
(i) [ A , C ] [A,C] [ A , C ] is mean-square exponentially stable;
(ii) [ A , C ] [A,C] [ A , C ] is L 2 L^2 L 2 -stable;
(iii) there exists P ∈ S + + n P\in\mathbb{S}_{++}^n P ∈ S ++ n such that P A + A ⊤ P + C ⊤ P C < 0 PA+A^\top P+C^\top PC<0 P A + A ⊤ P + C ⊤ P C < 0 .
Moreover, from Sun–Wang–Yong [34 Corollary 3.1] , we have the following result.
Corollary 2.6. If [ A , C ] [A,C] [ A , C ] is L 2 L^2 L 2 -stable, then there exist constants α , β > 0 \alpha,\beta>0 α , β > 0 such that ∣ e A t ∣ ≤ α e − β t / 2 |e^{At}|\le\alpha e^{-\beta t/2} ∣ e A t ∣ ≤ α e − β t /2 , t ≥ 0 t\ge0 t ≥ 0 .
Now we consider the following controlled homogeneous linear system:
{ d X ( t ) = ( A X ( t ) + B u ( t ) ) d t + ( C X ( t ) + D u ( t ) ) d W ( t ) , t ∈ [ 0 , ∞ ) , X ( 0 ) = x . (12) \begin{cases}
dX(t)=(AX(t)+Bu(t))dt+(CX(t)+Du(t))dW(t), & t\in[0,\infty),\\
X(0)=x. \tag*{(12)}
\end{cases} { d X ( t ) = ( A X ( t ) + B u ( t )) d t + ( C X ( t ) + D u ( t )) d W ( t ) , X ( 0 ) = x . t ∈ [ 0 , ∞ ) , (12)
For the infinite-horizon problem, the admissible control set is
U a d ( x ) = { u ∈ L F 2 ( 0 , ∞ ; R m ) : X x , u ∈ L F 2 ( 0 , ∞ ; R n ) } . (13) \mathcal{U}_{ad}(x)=\left\{u\in L_{\mathbb{F}}^2(0,\infty;\mathbb{R}^m):X^{x,u}\in L_{\mathbb{F}}^2(0,\infty;\mathbb{R}^n)\right\}. \tag*{(13)} U a d ( x ) = { u ∈ L F 2 ( 0 , ∞ ; R m ) : X x , u ∈ L F 2 ( 0 , ∞ ; R n ) } . (13)
The cost functional and the value function are given by
J ∞ 0 ( x ; u ) = E ∫ 0 ∞ [ X ( t ) ⊤ Q X ( t ) + 2 u ( t ) ⊤ S X ( t ) + u ( t ) ⊤ R u ( t ) ] d t , J_\infty^0(x;u)=\mathbb{E}\int_0^\infty\left[X(t)^\top QX(t)+2u(t)^\top SX(t)+u(t)^\top Ru(t)\right]dt, J ∞ 0 ( x ; u ) = E ∫ 0 ∞ [ X ( t ) ⊤ QX ( t ) + 2 u ( t ) ⊤ S X ( t ) + u ( t ) ⊤ R u ( t ) ] d t ,
V ∞ 0 ( x ) = inf u ∈ U a d ( x ) J ∞ 0 ( x ; u ) . (14) V_\infty^0(x)=\inf_{u\in\mathcal{U}_{ad}(x)}J_\infty^0(x;u). \tag*{(14)} V ∞ 0 ( x ) = u ∈ U a d ( x ) inf J ∞ 0 ( x ; u ) . (14)
The stability of the uncontrolled system also gives the following estimate uniformly over the time horizon. The proof can be found in Sun–Yong [37 ] .
Lemma 2.7. If [ A , C ] [A,C] [ A , C ] is L 2 L^2 L 2 -stable, then for any 0 ≤ t < T 0\le t<T 0 ≤ t < T , x ∈ R n x\in\mathbb{R}^n x ∈ R n , and u ∈ L F 2 ( t , T ; R m ) u\in L_{\mathbb{F}}^2(t,T;\mathbb{R}^m) u ∈ L F 2 ( t , T ; R m ) , the corresponding homogeneous controlled state satisfies
E ∫ t T ∣ X ( s ) ∣ 2 d s + E ∣ X ( T ) ∣ 2 ≤ C ( ∣ x ∣ 2 + E ∫ t T ∣ u ( s ) ∣ 2 d s ) , \mathbb{E}\int_t^T|X(s)|^2ds+\mathbb{E}|X(T)|^2\le C\left(|x|^2+\mathbb{E}\int_t^T|u(s)|^2ds\right), E ∫ t T ∣ X ( s ) ∣ 2 d s + E ∣ X ( T ) ∣ 2 ≤ C ( ∣ x ∣ 2 + E ∫ t T ∣ u ( s ) ∣ 2 d s ) , where C > 0 C>0 C > 0 is independent of t t t and T T T .
Corollary 2.8. If [ A , C ] [A,C] [ A , C ] is L 2 L^{2} L 2 -stable, then there exist constants C , γ > 0 C,\gamma>0 C , γ > 0 such that, for every 0 ≤ t ≤ s 0\leq t\leq s 0 ≤ t ≤ s and x ∈ R n x\in\mathbb{R}^{n} x ∈ R n , the uncontrolled homogeneous state starting from x x x at time t t t satisfies
E ∣ X ( s ; t , x ) ∣ 2 ≤ C e − 2 γ ( s − t ) ∣ x ∣ 2 . \mathbb{E}|X(s;t,x)|^{2}\leq Ce^{-2\gamma(s-t)}|x|^{2}. E ∣ X ( s ; t , x ) ∣ 2 ≤ C e − 2 γ ( s − t ) ∣ x ∣ 2 .
We denote the system (12) by [ A , C ; B , D ] [A,C;B,D] [ A , C ; B , D ] , and recall the following notion.
Definition 2.9. The system [ A , C ; B , D ] [A,C;B,D] [ A , C ; B , D ] is called L 2 L^{2} L 2 -stabilizable if there exists a matrix Θ ∈ R m × n \Theta\in\mathbb{R}^{m\times n} Θ ∈ R m × n such that the closed-loop system [ A + B Θ , C + D Θ ] [A+B\Theta,C+D\Theta] [ A + B Θ , C + D Θ ] is L 2 L^{2} L 2 -stable. Such a matrix Θ \Theta Θ is called a stabilizer of [ A , C ; B , D ] [A,C;B,D] [ A , C ; B , D ] .
For T > 0 T>0 T > 0 and u ∈ L F 2 ( 0 , T ; R m ) u\in L_{\mathbb{F}}^{2}(0,T;\mathbb{R}^{m}) u ∈ L F 2 ( 0 , T ; R m ) , let X X X be the solution of the homogeneous finite-horizon state equation
{ d X ( t ) = ( A X ( t ) + B u ( t ) ) d t + ( C X ( t ) + D u ( t ) ) d W ( t ) , t ∈ [ 0 , T ] , X ( 0 ) = 0 , \begin{cases}
dX(t)=(AX(t)+Bu(t))dt+(CX(t)+Du(t))dW(t), & t\in[0,T],\\
X(0)=0,
\end{cases} { d X ( t ) = ( A X ( t ) + B u ( t )) d t + ( C X ( t ) + D u ( t )) d W ( t ) , X ( 0 ) = 0 , t ∈ [ 0 , T ] ,
and define
J T 0 ( 0 ; u ) = E ∫ 0 T [ X ( t ) ⊤ Q X ( t ) + 2 X ( t ) ⊤ S ⊤ u ( t ) + u ( t ) ⊤ R u ( t ) ] d t . J_{T}^{0}(0;u)=\mathbb{E}\int_{0}^{T}\left[X(t)^{\top}QX(t)+2X(t)^{\top}S^{\top}u(t)+u(t)^{\top}Ru(t)\right]dt. J T 0 ( 0 ; u ) = E ∫ 0 T [ X ( t ) ⊤ QX ( t ) + 2 X ( t ) ⊤ S ⊤ u ( t ) + u ( t ) ⊤ R u ( t ) ] d t .
We impose the following assumptions.
(H1) The system [ A , C ; B , D ] [A,C;B,D] [ A , C ; B , D ] is L 2 L^{2} L 2 -stabilizable.
(H2) There exists a constant δ > 0 \delta>0 δ > 0 , independent of T T T and u u u , such that
J T 0 ( 0 ; u ) ≥ δ E ∫ 0 T ( ∣ X 0 , u ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , ∀ T > 0 , ∀ u ∈ L F 2 ( 0 , T ; R m ) . (15) J_{T}^{0}(0;u)\geq\delta\mathbb{E}\int_{0}^{T}\left(|X^{0,u}(t)|^{2}+|u(t)|^{2}\right)dt,\qquad\forall T>0,\quad\forall u\in L_{\mathbb{F}}^{2}(0,T;\mathbb{R}^{m}). \tag*{(15)} J T 0 ( 0 ; u ) ≥ δ E ∫ 0 T ( ∣ X 0 , u ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , ∀ T > 0 , ∀ u ∈ L F 2 ( 0 , T ; R m ) . (15)
We refer to Assumption (H2) as the joint coercivity (JC) condition. This terminology is motivated by the following coercivity condition, introduced by Damm and Jacob [6 ] : There exists δ > 0 \delta>0 δ > 0 such that
J ∞ 0 ( 0 ; u ) ≥ δ E ∫ 0 ∞ ∣ X 0 , u ( t ) ∣ 2 d t , ∀ u ∈ U a d ( 0 ) . (16) J_{\infty}^{0}(0;u)\geq\delta\mathbb{E}\int_{0}^{\infty}|X^{0,u}(t)|^{2}dt,\qquad\forall u\in U_{\mathrm{ad}}(0). \tag*{(16)} J ∞ 0 ( 0 ; u ) ≥ δ E ∫ 0 ∞ ∣ X 0 , u ( t ) ∣ 2 d t , ∀ u ∈ U ad ( 0 ) . (16)
A related condition commonly imposed in the literature is the uniform convexity (UC) of the cost functional with respect to the control; see [33 ] . The homogeneous cost functional satisfies the uniform convexity condition, if for any T > 0 T>0 T > 0 , there exists a constant δ T > 0 \delta_{T}>0 δ T > 0 , independent of u u u , such that
J T 0 ( 0 ; u ) ≥ δ T E ∫ 0 T ∣ u ( t ) ∣ 2 d t , u ∈ L F 2 ( 0 , T ; R m ) . (17) J_{T}^{0}(0;u)\geq\delta_{T}\mathbb{E}\int_{0}^{T}|u(t)|^{2}dt,\qquad u\in L_{\mathbb{F}}^{2}(0,T;\mathbb{R}^{m}). \tag*{(17)} J T 0 ( 0 ; u ) ≥ δ T E ∫ 0 T ∣ u ( t ) ∣ 2 d t , u ∈ L F 2 ( 0 , T ; R m ) . (17)
Proposition 2.11. Suppose that the uncontrolled system [ A , C ] [A,C] [ A , C ] is L 2 L^{2} L 2 -stable. Then (15) is equivalent to (17) holding with a common constant δ T = δ 0 > 0 \delta_{T}=\delta_{0}>0 δ T = δ 0 > 0 independent of T T T .
Proof. It suffices to prove the reverse implication. Suppose that (17) holds with a common constant δ 0 > 0 \delta_0>0 δ 0 > 0 independent of T T T . By Lemma 2.7 , there exists a constant C > 0 C>0 C > 0 , independent of T T T and u u u , such that, for a zero initial state,
E ∫ 0 T ∣ X ( t ) ∣ 2 d t ≤ C E ∫ 0 T ∣ u ( t ) ∣ 2 d t . \mathbb{E}\int_0^T |X(t)|^2dt\leq C\mathbb{E}\int_0^T |u(t)|^2dt. E ∫ 0 T ∣ X ( t ) ∣ 2 d t ≤ C E ∫ 0 T ∣ u ( t ) ∣ 2 d t .
Combining this estimate with (17) , we obtain
E ∫ 0 T ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t ≤ ( 1 + C ) E ∫ 0 T ∣ u ( t ) ∣ 2 d t ≤ 1 + C δ 0 J T 0 ( 0 ; u ) . \mathbb{E}\int_0^T \left(|X(t)|^2+|u(t)|^2\right)dt\leq(1+C)\mathbb{E}\int_0^T |u(t)|^2dt\leq\frac{1+C}{\delta_0}J_T^0(0;u). E ∫ 0 T ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t ≤ ( 1 + C ) E ∫ 0 T ∣ u ( t ) ∣ 2 d t ≤ δ 0 1 + C J T 0 ( 0 ; u ) .
Taking δ = δ 0 / ( 1 + C ) \delta=\delta_0/(1+C) δ = δ 0 / ( 1 + C ) gives (15) . ■
In finite-horizon stochastic LQ theory, the standard condition requires the terminal weight matrix to be non-negative and assumes that there exists a constant δ 0 > 0 \delta_0>0 δ 0 > 0 such that
R ≥ δ 0 I m , Q − S T R − 1 S ≥ 0. (18) R\geq\delta_0 I_m,\qquad Q-S^\mathsf{T}R^{-1}S\geq0. \tag*{(18)} R ≥ δ 0 I m , Q − S T R − 1 S ≥ 0. (18)
On every fixed finite horizon [ 0 , T ] [0,T] [ 0 , T ] , the standard condition implies the uniform convexity of the homogeneous cost functional u ↦ J T 0 ( 0 ; u ) u\mapsto J_T^0(0;u) u ↦ J T 0 ( 0 ; u ) , where the convexity constant may depend on T T T ; this uniform convexity in turn guarantees that the corresponding Riccati equation admits a unique strongly regular solution. These results can be found in Sun–Li–Yong [33 Proposition 3.5, Theorem 4.5 and Remark 4.6] . However, the standard condition does not provide a single convexity constant valid for all T > 0 T>0 T > 0 and therefore does not imply (H2). The following example illustrates this point.
Example 2.12. The standard condition does not imply (H2). Consider the deterministic scalar system
{ X ˙ ( t ) = − X ( t ) + 3 u ( t ) , 0 ≤ t ≤ T , X ( 0 ) = 0 (19) \left\{\begin{aligned}
\dot{X}(t)&=-X(t)+3u(t),\qquad0\leq t\leq T,\\
X(0)&=0
\end{aligned}\right. \tag*{(19)} { X ˙ ( t ) X ( 0 ) = − X ( t ) + 3 u ( t ) , 0 ≤ t ≤ T , = 0 (19) with cost functional
J T 0 ( 0 ; u ) = ∫ 0 T ( u ( t ) − X ( t ) ) 2 d t . J_T^0(0;u)=\int_0^T\left(u(t)-X(t)\right)^2dt. J T 0 ( 0 ; u ) = ∫ 0 T ( u ( t ) − X ( t ) ) 2 d t . Here A = − 1 A=-1 A = − 1 , B = 3 B=3 B = 3 , C = D = 0 C=D=0 C = D = 0 , and R = 1 R=1 R = 1 , S = − 1 S=-1 S = − 1 , Q = 1 Q=1 Q = 1 . Thus the free system is exponentially stable and Q − S T R − 1 S = 0 Q-S^\mathsf{T}R^{-1}S=0 Q − S T R − 1 S = 0 , so the standard condition (18) holds. For each T > 0 T>0 T > 0 , let u T ( t ) − X ( t ) = 3 e − t u_T(t)-X(t)=3e^{-t} u T ( t ) − X ( t ) = 3 e − t . It follows from (19) that
X ( t ) = 3 ( e 2 t − e − t ) , u T ( t ) = 3 e 2 t . X(t)=3\left(e^{2t}-e^{-t}\right),\qquad u_T(t)=3e^{2t}. X ( t ) = 3 ( e 2 t − e − t ) , u T ( t ) = 3 e 2 t . Consequently,
J T 0 ( 0 ; u T ) ∫ 0 T ∣ u T ( t ) ∣ 2 d t = ∫ 0 T 9 e − 2 t d t ∫ 0 T 9 e 4 t d t = 2 ( 1 − e − 2 T ) e 4 T − 1 ⟶ 0 , as T → ∞ . \frac{J_T^0(0;u_T)}{\int_0^T |u_T(t)|^2dt}=\frac{\int_0^T 9e^{-2t}dt}{\int_0^T 9e^{4t}dt}=\frac{2(1-e^{-2T})}{e^{4T}-1}\longrightarrow0,\qquad\text{as }T\to\infty. ∫ 0 T ∣ u T ( t ) ∣ 2 d t J T 0 ( 0 ; u T ) = ∫ 0 T 9 e 4 t d t ∫ 0 T 9 e − 2 t d t = e 4 T − 1 2 ( 1 − e − 2 T ) ⟶ 0 , as T → ∞. Hence no positive constant δ \delta δ independent of T T T can satisfy (H2).
The following example shows that a system may still satisfy the J C JC J C condition while its uncontrolled dynamic system is unstable and its cost functional fails to satisfy the standard condition.
Example 2.14. Consider the system
{ X ˙ ( t ) = X ( t ) + u ( t ) , 0 ≤ t ≤ T , X ( 0 ) = 0 , \begin{cases}
\dot{X}(t) = X(t) + u(t), & 0 \le t \le T,\\
X(0) = 0,
\end{cases} { X ˙ ( t ) = X ( t ) + u ( t ) , X ( 0 ) = 0 , 0 ≤ t ≤ T , and the cost functional
J T 0 ( 0 ; u ) = ∫ 0 T ( 6 X ( t ) 2 + 6 X ( t ) u ( t ) + u ( t ) 2 ) d t . J_T^0(0;u)=\int_0^T \left(6X(t)^2+6X(t)u(t)+u(t)^2\right)\mathrm{d}t. J T 0 ( 0 ; u ) = ∫ 0 T ( 6 X ( t ) 2 + 6 X ( t ) u ( t ) + u ( t ) 2 ) d t . The uncontrolled system is unstable. However, the feedback u ( t ) = − 2 X ( t ) u(t)=-2X(t) u ( t ) = − 2 X ( t ) yields the closed-loop equation X ˙ ( t ) = − X ( t ) \dot{X}(t)=-X(t) X ˙ ( t ) = − X ( t ) , so the system is stabilizable. In this case, Q = 6 Q=6 Q = 6 , S = 3 S=3 S = 3 , R = 1 R=1 R = 1 , and Q − S T R − 1 S = 6 − 9 = − 3 < 0 Q-S^{\mathsf T}R^{-1}S=6-9=-3<0 Q − S T R − 1 S = 6 − 9 = − 3 < 0 , so the standard condition fails. For any T > 0 T>0 T > 0 and u ∈ L 2 ( 0 , T ) u\in L^2(0,T) u ∈ L 2 ( 0 , T ) , differentiation gives
d d t X ( t ) 2 = 2 X ( t ) 2 + 2 X ( t ) u ( t ) ≥ X ( t ) 2 − u ( t ) 2 . \frac{\mathrm{d}}{\mathrm{d}t}X(t)^2=2X(t)^2+2X(t)u(t)\ge X(t)^2-u(t)^2. d t d X ( t ) 2 = 2 X ( t ) 2 + 2 X ( t ) u ( t ) ≥ X ( t ) 2 − u ( t ) 2 . Integrating and using X ( 0 ) = 0 X(0)=0 X ( 0 ) = 0 , we obtain ∫ 0 T X ( t ) 2 d t ≤ ∫ 0 T u ( t ) 2 d t + X ( T ) 2 \int_0^T X(t)^2\mathrm{d}t\le\int_0^T u(t)^2\mathrm{d}t+X(T)^2 ∫ 0 T X ( t ) 2 d t ≤ ∫ 0 T u ( t ) 2 d t + X ( T ) 2 . Moreover, the state equation yields
J T 0 ( 0 ; u ) = 3 ∫ 0 T d d t X ( t ) 2 d t + ∫ 0 T u ( t ) 2 d t = 3 X ( T ) 2 + ∫ 0 T u ( t ) 2 d t ≥ 1 2 ∫ 0 T ( X ( t ) 2 + u ( t ) 2 ) d t . J_T^0(0;u)=3\int_0^T\frac{\mathrm{d}}{\mathrm{d}t}X(t)^2\mathrm{d}t+\int_0^T u(t)^2\mathrm{d}t=3X(T)^2+\int_0^T u(t)^2\mathrm{d}t\ge\frac{1}{2}\int_0^T\left(X(t)^2+u(t)^2\right)\mathrm{d}t. J T 0 ( 0 ; u ) = 3 ∫ 0 T d t d X ( t ) 2 d t + ∫ 0 T u ( t ) 2 d t = 3 X ( T ) 2 + ∫ 0 T u ( t ) 2 d t ≥ 2 1 ∫ 0 T ( X ( t ) 2 + u ( t ) 2 ) d t .
Proposition 2.15. Let ( H 2 ) (H2) ( H 2 ) hold. Then, for every T > 0 T>0 T > 0 , the Riccati equation (9) admits a unique strongly regular solution P T ∈ C ( [ 0 , T ] ; S n ) P_T\in C([0,T];\mathcal{S}^n) P T ∈ C ([ 0 , T ] ; S n ) . Moreover,
R + D T P T ( t ) D ≥ δ I m , 0 ≤ t ≤ T . (20) R+D^{\mathsf T}P_T(t)D\ge\delta I_m,\qquad0\le t\le T. \tag*{(20)} R + D T P T ( t ) D ≥ δ I m , 0 ≤ t ≤ T . (20) In particular, R ≥ δ I m R\ge\delta I_m R ≥ δ I m .
Remark 2.16. Condition (15) implies (17) with δ T ≡ δ \delta_T\equiv\delta δ T ≡ δ . Then the above result is given by Sun–Li–Yong [33 Theorem 4.5 and Remark 4.6] immediately. The result R ≥ δ I m R\ge\delta I_m R ≥ δ I m is obtained by (2.13) and the fact P T ( T ) = 0 P_T(T)=0 P T ( T ) = 0 .
Proposition 2.17. Assumption ( H 2 ) (H2) ( H 2 ) implies
J ∞ 0 ( 0 ; u ) ≥ δ E ∫ 0 ∞ ( ∣ X 0 , u ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , ∀ u ∈ U a d ( 0 ) . (21) J_\infty^0(0;u)\ge\delta\mathbb{E}\int_0^\infty\left(\lvert X^{0,u}(t)\rvert^2+\lvert u(t)\rvert^2\right)\mathrm{d}t,\qquad\forall u\in\mathcal{U}_{\mathrm{ad}}(0). \tag*{(21)} J ∞ 0 ( 0 ; u ) ≥ δ E ∫ 0 ∞ ( ∣ X 0 , u ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , ∀ u ∈ U ad ( 0 ) . (21)
This follows by letting T → ∞ T\to\infty T → ∞ in (15) for u ∈ U a d ( 0 ) u\in\mathcal{U}_{\mathrm{ad}}(0) u ∈ U ad ( 0 ) . Now we return to the homogeneous infinite-horizon problem defined by (12) –(13) –(14) , and determine P ∞ P_\infty P ∞ and the stabilizing feedback. The theorem follows the approach of Damm–Jacob [6 Theorem 2] , where their diffusion coefficient does not depend on the control; we establish the result for the general case. The proof is given in the Appendix.
Consider the algebraic Riccati equation
0 = A T P ∞ + P ∞ A + C T P ∞ C + Q − ( P ∞ B + C T P ∞ D + S T ) ( R + D T P ∞ D ) − 1 ( B T P ∞ + D T P ∞ C + S ) . (22) \begin{aligned}
0={}&A^{\mathsf T}P_\infty+P_\infty A+C^{\mathsf T}P_\infty C+Q\\
&-(P_\infty B+C^{\mathsf T}P_\infty D+S^{\mathsf T})(R+D^{\mathsf T}P_\infty D)^{-1}(B^{\mathsf T}P_\infty+D^{\mathsf T}P_\infty C+S). \tag*{(22)}
\end{aligned} 0 = A T P ∞ + P ∞ A + C T P ∞ C + Q − ( P ∞ B + C T P ∞ D + S T ) ( R + D T P ∞ D ) − 1 ( B T P ∞ + D T P ∞ C + S ) . (22)
We call P ∞ ∈ S n P_\infty\in\mathcal{S}^n P ∞ ∈ S n a stabilizing solution of (22) if it satisfies this equation and the corresponding closed-loop system [ A − B K ∞ , C − D K ∞ ] [A-BK_\infty,C-DK_\infty] [ A − B K ∞ , C − D K ∞ ] is L 2 L^2 L 2 -stable, where
M ∞ = R + D T P ∞ D > 0 , N ∞ = B T P ∞ + D T P ∞ C + S , K ∞ = M ∞ − 1 N ∞ . (23) M_\infty=R+D^{\mathsf T}P_\infty D>0,\qquad N_\infty=B^{\mathsf T}P_\infty+D^{\mathsf T}P_\infty C+S,\qquad K_\infty=M_\infty^{-1}N_\infty. \tag*{(23)} M ∞ = R + D T P ∞ D > 0 , N ∞ = B T P ∞ + D T P ∞ C + S , K ∞ = M ∞ − 1 N ∞ . (23)
Theorem 2.18. Suppose that the system satisfies ( H 1 ) (H1) ( H 1 ) and that R > 0 R>0 R > 0 . Then the following three statements are equivalent:
(i) The coercivity condition (16) holds for some δ > 0 \delta> 0 δ > 0 .
(ii) There exists P ∞ ∈ S n P_{\infty} \in\mathbb{S}^{n} P ∞ ∈ S n satisfying (22) such that the system [ A − B K ∞ , C − D K ∞ ] [A - BK_{\infty}, C - DK_{\infty}] [ A − B K ∞ , C − D K ∞ ] is mean-square exponentially stable.
(iii) There exists δ > 0 \delta> 0 δ > 0 such that
J ∞ 0 ( 0 ; u ) ≥ δ E ∫ 0 ∞ ( ∣ X 0 , u ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , ∀ u ∈ U a d ( 0 ) . (24) J_{\infty}^{0}(0;u) \geq\delta\mathbb{E}\int_{0}^{\infty}\left(\lvert X^{0,u}(t)\rvert^{2}+\lvert u(t)\rvert^{2}\right)\,\mathrm{d}t,\qquad\forall u\in\mathcal{U}_{\mathrm{ad}}(0). \tag*{(24)} J ∞ 0 ( 0 ; u ) ≥ δ E ∫ 0 ∞ ( ∣ X 0 , u ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , ∀ u ∈ U ad ( 0 ) . (24)
In this case, P ∞ P_{\infty} P ∞ is the unique stabilizing solution of (22) , and
V ∞ 0 ( x ) = x T P ∞ x , u ∗ ( t ) = − K ∞ X ∗ ( t ) (25) V_{\infty}^{0}(x)=x^{\mathsf{T}}P_{\infty}x,\qquad u^{*}(t)=-K_{\infty}X^{*}(t) \tag*{(25)} V ∞ 0 ( x ) = x T P ∞ x , u ∗ ( t ) = − K ∞ X ∗ ( t ) (25)
give the optimal value and the unique optimal control for each initial state. The matrix P ∞ P_{\infty} P ∞ may be indefinite.
Corollary 2.19. If (H1) and (H2) hold, then (22) admits a unique stabilizing solution, and all the conclusions of Theorem 2.18 hold.
Proof. By Proposition 2.15 , (20) gives R ≥ δ I m > 0 R \geq\delta I_{m} > 0 R ≥ δ I m > 0 . Moreover, (21) in Proposition 2.17 implies the coercivity condition (16) . Thus the assumptions of Theorem 2.18 are satisfied, and its conclusions follow. ■
Static Optimization Problem
We formulate the static problem over general control distributions. We then use the maximum-entropy property to show that its optimizer is Gaussian, so the problem can be reduced to the variables ( x , u , Λ ) (x,u,\Lambda) ( x , u , Λ ) , where x ∈ R n x \in\mathbb{R}^{n} x ∈ R n , u ∈ R m u \in\mathbb{R}^{m} u ∈ R m , and Λ ∈ S + + m \Lambda\in\mathbb{S}_{++}^{m} Λ ∈ S ++ m . Here u u u is the mean of the static control distribution, and Λ \Lambda Λ is the static exploration covariance.
We first give a static optimization problem. Define the pointwise running part
L 0 ( x , u ) = x T Q x + 2 x T S T u + u T R u + 2 q T x + 2 r T u . L_{0}(x,u)=x^{\mathsf{T}}Qx+2x^{\mathsf{T}}S^{\mathsf{T}}u+u^{\mathsf{T}}Ru+2q^{\mathsf{T}}x+2r^{\mathsf{T}}u. L 0 ( x , u ) = x T Q x + 2 x T S T u + u T R u + 2 q T x + 2 r T u .
Define the feasible set
V = { ( x , u ) ∈ R n × R m ∣ A x + B u + b = 0 } . \mathcal{V}=\{(x,u)\in\mathbb{R}^{n}\times\mathbb{R}^{m}\mid Ax+Bu+b=0\}. V = {( x , u ) ∈ R n × R m ∣ A x + B u + b = 0 } .
Let P ∞ P_{\infty} P ∞ be the unique stabilizing solution of (22) . Define
F 0 ( x , u ) = L 0 ( x , u ) + ( C x + D u + σ ) T P ∞ ( C x + D u + σ ) . (26) F_{0}(x,u)=L_{0}(x,u)+(Cx+Du+\sigma)^{\mathsf{T}}P_{\infty}(Cx+Du+\sigma). \tag*{(26)} F 0 ( x , u ) = L 0 ( x , u ) + ( C x + D u + σ ) T P ∞ ( C x + D u + σ ) . (26)
The classical stochastic LQ static problem is
min ( x , u ) ∈ V F 0 ( x , u ) . (27) \min_{(x,u)\in\mathcal{V}} F_{0}(x,u). \tag*{(27)} ( x , u ) ∈ V min F 0 ( x , u ) . (27)
Proposition 3.1. Let (H1)–(H2) hold. Then the static problem (27) has a unique solution, denoted by ( x ∗ , u ∗ ) (x_{*},u_{*}) ( x ∗ , u ∗ ) . Moreover, there exists a Lagrange multiplier ζ ∗ ∈ R n \zeta^{*}\in\mathbb{R}^{n} ζ ∗ ∈ R n satisfying the Karush–Kuhn–Tucker (KKT) conditions
A x ∗ + B u ∗ + b = 0 , Q x ∗ + S T u ∗ + q + C T P ∞ ( C x ∗ + D u ∗ + σ ) + A T ζ ∗ = 0 , S x ∗ + R u ∗ + r + D T P ∞ ( C x ∗ + D u ∗ + σ ) + B T ζ ∗ = 0. (28) \begin{aligned}
Ax_{*}+Bu_{*}+b&=0,\\
Qx_{*}+S^{\mathsf{T}}u_{*}+q+C^{\mathsf{T}}P_{\infty}(Cx_{*}+Du_{*}+\sigma)+A^{\mathsf{T}}\zeta^{*}&=0,\\
Sx_{*}+Ru_{*}+r+D^{\mathsf{T}}P_{\infty}(Cx_{*}+Du_{*}+\sigma)+B^{\mathsf{T}}\zeta^{*}&=0. \tag*{(28)}
\end{aligned} A x ∗ + B u ∗ + b Q x ∗ + S T u ∗ + q + C T P ∞ ( C x ∗ + D u ∗ + σ ) + A T ζ ∗ S x ∗ + R u ∗ + r + D T P ∞ ( C x ∗ + D u ∗ + σ ) + B T ζ ∗ = 0 , = 0 , = 0. (28)
Conversely, if a triple ( x , u , ζ ) (x,u,\zeta) ( x , u , ζ ) satisfies the system obtained from (28) by replacing ( x ∗ , u ∗ , ζ ∗ ) (x_{*},u_{*},\zeta^{*}) ( x ∗ , u ∗ , ζ ∗ ) with ( x , u , ζ ) (x,u,\zeta) ( x , u , ζ ) , then ( x , u ) (x,u) ( x , u ) is the unique minimizer, that is, ( x ∗ , u ∗ ) = ( x , u ) (x_{*},u_{*})=(x,u) ( x ∗ , u ∗ ) = ( x , u ) .
Remark 3.2. In contrast to Sun–Wang–Yong [34 ] , the matrices Q Q Q and P ∞ P_{\infty} P ∞ in (26) can be indefinite. Therefore, we need to introduce new ideas and perform a careful convexity analysis of F 0 F_0 F 0 .
Proof. By Corollary 2.19 and Corollary 2.6 , A − B K ∞ A-BK_{\infty} A − B K ∞ is Hurwitz and hence invertible. Let
x 0 = − ( A − B K ∞ ) − 1 b , u 0 = − K ∞ x 0 . x^0=-(A-BK_{\infty})^{-1}b,\qquad u^0=-K_{\infty}x^0. x 0 = − ( A − B K ∞ ) − 1 b , u 0 = − K ∞ x 0 .
Then
A x 0 + B u 0 + b = ( A − B K ∞ ) x 0 + b = 0. Ax^0+Bu^0+b=(A-BK_{\infty})x^0+b=0. A x 0 + B u 0 + b = ( A − B K ∞ ) x 0 + b = 0.
Hence ( x 0 , u 0 ) ∈ V (x^0,u^0)\in\mathcal{V} ( x 0 , u 0 ) ∈ V . Moreover,
[ A B ] ( I n − K ∞ ) = A − B K ∞ [A\ \ B]\begin{pmatrix}I_n\\-K_{\infty}\end{pmatrix}=A-BK_{\infty} [ A B ] ( I n − K ∞ ) = A − B K ∞
is invertible, so [ A B ] [A\ \ B] [ A B ] has full row rank.
Fix ( x , u ) ∈ V (x,u)\in\mathcal{V} ( x , u ) ∈ V . For any ( ξ , v ) ∈ R n × R m (\xi,v)\in\mathbb{R}^n\times\mathbb{R}^m ( ξ , v ) ∈ R n × R m ,
( x + ξ , u + v ) ∈ V ⟺ A ξ + B v = 0. (29) (x+\xi,u+v)\in\mathcal{V}\quad\Longleftrightarrow\quad A\xi+Bv=0. \tag*{(29)} ( x + ξ , u + v ) ∈ V ⟺ A ξ + B v = 0. (29)
Let T : = { ( ξ , v ) ∈ R n × R m ∣ A ξ + B v = 0 } \mathcal{T}:=\{(\xi,v)\in\mathbb{R}^n\times\mathbb{R}^m\mid A\xi+Bv=0\} T := {( ξ , v ) ∈ R n × R m ∣ A ξ + B v = 0 } be the tangent space. The Taylor expansion of the quadratic function F 0 F_0 F 0 in (26) gives
F 0 ( x + ξ , u + v ) − F 0 ( x , u ) = 2 f x ( x , u ) T ξ + 2 f u ( x , u ) T v + f x u ( ξ , v ) . (30) F_0(x+\xi,u+v)-F_0(x,u)=2f_x(x,u)^{\mathsf T}\xi+2f_u(x,u)^{\mathsf T}v+f_{xu}(\xi,v). \tag*{(30)} F 0 ( x + ξ , u + v ) − F 0 ( x , u ) = 2 f x ( x , u ) T ξ + 2 f u ( x , u ) T v + f xu ( ξ , v ) . (30)
Here 2 f x 2f_x 2 f x and 2 f u 2f_u 2 f u are the corresponding gradient components, with
f x ( x , u ) = Q x + S T u + q + C T P ∞ ( C x + D u + σ ) , f_x(x,u)=Qx+S^{\mathsf T}u+q+C^{\mathsf T}P_{\infty}(Cx+Du+\sigma), f x ( x , u ) = Q x + S T u + q + C T P ∞ ( C x + D u + σ ) ,
f u ( x , u ) = S x + R u + r + D T P ∞ ( C x + D u + σ ) . f_u(x,u)=Sx+Ru+r+D^{\mathsf T}P_{\infty}(Cx+Du+\sigma). f u ( x , u ) = S x + R u + r + D T P ∞ ( C x + D u + σ ) .
The second-order term is
f x u ( ξ , v ) = ξ T ( Q + C T P ∞ C ) ξ + 2 ξ T ( S T + C T P ∞ D ) v + v T ( R + D T P ∞ D ) v . f_{xu}(\xi,v)=\xi^{\mathsf T}(Q+C^{\mathsf T}P_{\infty}C)\xi+2\xi^{\mathsf T}(S^{\mathsf T}+C^{\mathsf T}P_{\infty}D)v+v^{\mathsf T}(R+D^{\mathsf T}P_{\infty}D)v. f xu ( ξ , v ) = ξ T ( Q + C T P ∞ C ) ξ + 2 ξ T ( S T + C T P ∞ D ) v + v T ( R + D T P ∞ D ) v .
For ( ξ , v ) ∈ T (\xi,v)\in\mathcal{T} ( ξ , v ) ∈ T , the algebraic Riccati equation (22) , the symmetry of P ∞ P_{\infty} P ∞ , the tangent constraint (29) , and the definitions in (23) yield
f x u ( ξ , v ) = ξ T N ∞ T M ∞ − 1 N ∞ ξ + 2 ξ T N ∞ T v + v T M ∞ v = ( v + K ∞ ξ ) T M ∞ ( v + K ∞ ξ ) . f_{xu}(\xi,v)=\xi^{\mathsf T}N_{\infty}^{\mathsf T}M_{\infty}^{-1}N_{\infty}\xi+2\xi^{\mathsf T}N_{\infty}^{\mathsf T}v+v^{\mathsf T}M_{\infty}v=(v+K_{\infty}\xi)^{\mathsf T}M_{\infty}(v+K_{\infty}\xi). f xu ( ξ , v ) = ξ T N ∞ T M ∞ − 1 N ∞ ξ + 2 ξ T N ∞ T v + v T M ∞ v = ( v + K ∞ ξ ) T M ∞ ( v + K ∞ ξ ) .
Since M ∞ > 0 M_{\infty}>0 M ∞ > 0 , we have f x u ( ξ , v ) ≥ 0 f_{xu}(\xi,v)\geq0 f xu ( ξ , v ) ≥ 0 . If equality holds, then v = − K ∞ ξ v=-K_{\infty}\xi v = − K ∞ ξ . Together with (29) , this gives
( A − B K ∞ ) ξ = 0. (A-BK_{\infty})\xi=0. ( A − B K ∞ ) ξ = 0.
The invertibility of A − B K ∞ A-BK_{\infty} A − B K ∞ implies ξ = 0 \xi=0 ξ = 0 and then v = 0 v=0 v = 0 . Hence f x u ( ξ , v ) > 0 f_{xu}(\xi,v)>0 f xu ( ξ , v ) > 0 for all ( ξ , v ) ∈ T ∖ { ( 0 , 0 ) } (\xi,v)\in\mathcal{T}\setminus\{(0,0)\} ( ξ , v ) ∈ T ∖ {( 0 , 0 )} .
The unit sphere S T : = { ( ξ , v ) ∈ T ∣ ∣ ξ ∣ 2 + ∣ v ∣ 2 = 1 } S_{\mathcal{T}}:=\{(\xi,v)\in\mathcal{T}\mid|\xi|^2+|v|^2=1\} S T := {( ξ , v ) ∈ T ∣ ∣ ξ ∣ 2 + ∣ v ∣ 2 = 1 } is compact, and therefore c : = min ( ξ , v ) ∈ S T f x u ( ξ , v ) > 0 c:=\min_{(\xi,v)\in S_{\mathcal{T}}}f_{xu}(\xi,v)>0 c := min ( ξ , v ) ∈ S T f xu ( ξ , v ) > 0 . For any nonzero ( ξ , v ) ∈ T (\xi,v)\in\mathcal{T} ( ξ , v ) ∈ T , let r = ∣ ξ ∣ 2 + ∣ v ∣ 2 r=\sqrt{|\xi|^2+|v|^2} r = ∣ ξ ∣ 2 + ∣ v ∣ 2 . The quadratic homogeneity of f x u f_{xu} f xu gives
f x u ( ξ , v ) = r 2 f x u ( ξ r , v r ) ≥ c r 2 = c ( ∣ ξ ∣ 2 + ∣ v ∣ 2 ) . (31) f_{xu}(\xi,v)=r^2f_{xu}\left(\frac{\xi}{r},\frac{v}{r}\right)\geq cr^2=c(|\xi|^2+|v|^2). \tag*{(31)} f xu ( ξ , v ) = r 2 f xu ( r ξ , r v ) ≥ c r 2 = c ( ∣ ξ ∣ 2 + ∣ v ∣ 2 ) . (31)
The assertion is immediate for ( ξ , v ) = ( 0 , 0 ) (\xi,v)=(0,0) ( ξ , v ) = ( 0 , 0 ) .
Every ( x , u ) ∈ V (x,u)\in\mathcal{V} ( x , u ) ∈ V can be written as ( x , u ) = ( x 0 , u 0 ) + ( ξ , v ) (x,u)=(x^0,u^0)+(\xi,v) ( x , u ) = ( x 0 , u 0 ) + ( ξ , v ) with ( ξ , v ) ∈ T (\xi,v)\in\mathcal{T} ( ξ , v ) ∈ T . By (30) and (31) , there exists a constant C > 0 C>0 C > 0 , independent of ( ξ , v ) (\xi,v) ( ξ , v ) , such that
F 0 ( x 0 + ξ , u 0 + v ) ≥ c ( ∣ ξ ∣ 2 + ∣ v ∣ 2 ) − C ( ∣ ξ ∣ + ∣ v ∣ ) − C . F_0(x^0+\xi,u^0+v)\geq c(|\xi|^2+|v|^2)-C(|\xi|+|v|)-C. F 0 ( x 0 + ξ , u 0 + v ) ≥ c ( ∣ ξ ∣ 2 + ∣ v ∣ 2 ) − C ( ∣ ξ ∣ + ∣ v ∣ ) − C .
Thus F 0 F_0 F 0 is coercive on the closed affine set V V V . Since F 0 F_0 F 0 is continuous, it attains its minimum on V V V . Moreover, (31) gives f x u ( ξ , v ) > 0 f_{xu}(\xi,v)>0 f xu ( ξ , v ) > 0 for every nonzero feasible direction ( ξ , v ) ∈ T (\xi,v)\in T ( ξ , v ) ∈ T . Hence F 0 F_0 F 0 is strictly convex on V V V , so it has at most one minimizer. Therefore the static problem (3.2 admits a unique minimizer ( x ∗ , u ∗ ) (x_*,u_*) ( x ∗ , u ∗ ) .
Since [ A B ] [A\ \ B] [ A B ] has full row rank, the finite-dimensional Lagrange multiplier theorem for equality constraints (see Yong [46 ] ) yields a ζ ∗ ∈ R n \zeta^*\in\mathbb{R}^n ζ ∗ ∈ R n such that
∇ F 0 ( x ∗ , u ∗ ) + 2 [ A B ] ⊤ ζ ∗ = 0. \nabla F_0(x_*,u_*)+2[A\ \ B]^\top\zeta^*=0. ∇ F 0 ( x ∗ , u ∗ ) + 2 [ A B ] ⊤ ζ ∗ = 0.
Since ∇ x F 0 = 2 f x \nabla_xF_0=2f_x ∇ x F 0 = 2 f x and ∇ u F 0 = 2 f u \nabla_uF_0=2f_u ∇ u F 0 = 2 f u , this condition, together with feasibility, gives (28) .
Conversely, let ( x , u , ζ ) (x,u,\zeta) ( x , u , ζ ) satisfy (28) and let ( ξ , v ) ∈ T (\xi,v)\in T ( ξ , v ) ∈ T . The KKT equations give
f x ( x , u ) = − A ⊤ ζ , f u ( x , u ) = − B ⊤ ζ . f_x(x,u)=-A^\top\zeta,\qquad f_u(x,u)=-B^\top\zeta. f x ( x , u ) = − A ⊤ ζ , f u ( x , u ) = − B ⊤ ζ .
f x ( x , u ) ⊤ ξ + f u ( x , u ) ⊤ v = − ζ ⊤ ( A ξ + B v ) = 0. f_x(x,u)^\top\xi+f_u(x,u)^\top v=-\zeta^\top(A\xi+Bv)=0. f x ( x , u ) ⊤ ξ + f u ( x , u ) ⊤ v = − ζ ⊤ ( A ξ + B v ) = 0.
By (30) and (31) , for every nonzero ( ξ , v ) ∈ T (\xi,v)\in T ( ξ , v ) ∈ T ,
F 0 ( x + ξ , u + v ) − F 0 ( x , u ) = f x u ( ξ , v ) ≥ c ( ∣ ξ ∣ 2 + ∣ v ∣ 2 ) > 0. F_0(x+\xi,u+v)-F_0(x,u)=f_{xu}(\xi,v)\geq c\left(|\xi|^2+|v|^2\right)>0. F 0 ( x + ξ , u + v ) − F 0 ( x , u ) = f xu ( ξ , v ) ≥ c ( ∣ ξ ∣ 2 + ∣ v ∣ 2 ) > 0.
Thus ( x , u ) (x,u) ( x , u ) is the unique minimizer, and therefore ( x , u ) = ( x ∗ , u ∗ ) (x,u)=(x_*,u_*) ( x , u ) = ( x ∗ , u ∗ ) .
For a probability distribution π \pi π on R m \mathbb{R}^m R m with finite second moment, set
u ˉ π = ∫ R m u π ( d u ) , Λ π = ∫ R m ( u − u ˉ π ) ( u − u ˉ π ) ⊤ π ( d u ) . \bar{u}_{\pi}=\int_{\mathbb{R}^m}u\,\pi(du),\qquad
\Lambda_{\pi}=\int_{\mathbb{R}^m}(u-\bar{u}_{\pi})(u-\bar{u}_{\pi})^\top\pi(du). u ˉ π = ∫ R m u π ( d u ) , Λ π = ∫ R m ( u − u ˉ π ) ( u − u ˉ π ) ⊤ π ( d u ) .
Denote
F λ ( x , π ) : = ∫ R m F 0 ( x , u ) π ( d u ) − λ Ent ( π ) . (32) F_{\lambda}(x,\pi):=\int_{\mathbb{R}^m}F_0(x,u)\,\pi(du)-\lambda\operatorname{Ent}(\pi). \tag*{(32)} F λ ( x , π ) := ∫ R m F 0 ( x , u ) π ( d u ) − λ Ent ( π ) . (32)
The exploratory static problem over general distributions is
min x , π F λ ( x , π ) s . t . A x + B u ˉ π + b = 0. (33) \min_{x,\pi}\ F_{\lambda}(x,\pi)\qquad\mathrm{s.t.}\qquad Ax+B\bar{u}_{\pi}+b=0. \tag*{(33)} x , π min F λ ( x , π ) s.t. A x + B u ˉ π + b = 0. (33)
Here π \pi π ranges over probability distributions that are absolutely continuous with respect to Lebesgue measure and have finite second moment and finite differential entropy.
Expanding F 0 ( x , u ) F_0(x,u) F 0 ( x , u ) with respect to the control variable gives
F 0 ( x , u ) = c ∞ ( x ) + 2 g ∞ ( x ) ⊤ u + u ⊤ M ∞ u . F_0(x,u)=c_{\infty}(x)+2g_{\infty}(x)^\top u+u^\top M_{\infty}u. F 0 ( x , u ) = c ∞ ( x ) + 2 g ∞ ( x ) ⊤ u + u ⊤ M ∞ u .
where
g ∞ ( x ) = S x + r + D ⊤ P ∞ ( C x + σ ) , g_{\infty}(x)=Sx+r+D^\top P_{\infty}(Cx+\sigma), g ∞ ( x ) = S x + r + D ⊤ P ∞ ( C x + σ ) ,
c ∞ ( x ) = x ⊤ Q x + 2 q ⊤ x + ( C x + σ ) ⊤ P ∞ ( C x + σ ) . c_{\infty}(x)=x^\top Qx+2q^\top x+(Cx+\sigma)^\top P_{\infty}(Cx+\sigma). c ∞ ( x ) = x ⊤ Q x + 2 q ⊤ x + ( C x + σ ) ⊤ P ∞ ( C x + σ ) .
Taking expectation under a general distribution π \pi π and using the second-moment decomposition ∫ R m u u ⊤ π ( d u ) = u ˉ π u ˉ π ⊤ + Λ π \int_{\mathbb{R}^m}uu^\top\pi(du)=\bar{u}_{\pi}\bar{u}_{\pi}^\top+\Lambda_{\pi} ∫ R m u u ⊤ π ( d u ) = u ˉ π u ˉ π ⊤ + Λ π , we obtain
∫ R m F 0 ( x , u ) π ( d u ) = c ∞ ( x ) + 2 g ∞ ( x ) ⊤ u ˉ π + u ˉ π ⊤ M ∞ u ˉ π + tr ( M ∞ Λ π ) = F 0 ( x , u ˉ π ) + tr ( M ∞ Λ π ) . \begin{aligned}
\int_{\mathbb{R}^m}F_0(x,u)\,\pi(du)
&=c_{\infty}(x)+2g_{\infty}(x)^\top\bar{u}_{\pi}
+\bar{u}_{\pi}^\top M_{\infty}\bar{u}_{\pi}
+\operatorname{tr}(M_{\infty}\Lambda_{\pi})\\
&=F_0(x,\bar{u}_{\pi})+\operatorname{tr}(M_{\infty}\Lambda_{\pi}).
\end{aligned} ∫ R m F 0 ( x , u ) π ( d u ) = c ∞ ( x ) + 2 g ∞ ( x ) ⊤ u ˉ π + u ˉ π ⊤ M ∞ u ˉ π + tr ( M ∞ Λ π ) = F 0 ( x , u ˉ π ) + tr ( M ∞ Λ π ) .
By (32) , the objective in (33) can be written as
F λ ( x , π ) = F 0 ( x , u ˉ π ) + tr ( M ∞ Λ π ) − λ Ent ( π ) . (34) F_{\lambda}(x,\pi)=F_0(x,\bar{u}_{\pi})+\operatorname{tr}(M_{\infty}\Lambda_{\pi})-\lambda\operatorname{Ent}(\pi). \tag*{(34)} F λ ( x , π ) = F 0 ( x , u ˉ π ) + tr ( M ∞ Λ π ) − λ Ent ( π ) . (34)
Except for the entropy term, the objective only depends on the first moment and the centered second moment of the control distribution.
The following is taken from Cover–Thomas [3 ] .
Lemma 3.3. Let π ( d u ) = p ( u ) d u \pi(\mathrm{d}u)=p(u)\mathrm{d}u π ( d u ) = p ( u ) d u be a probability distribution on R m \mathbb{R}^{m} R m with finite second moment and finite entropy. Let u ∈ R m u\in\mathbb{R}^{m} u ∈ R m and Λ ∈ S + + m \Lambda\in\mathbb{S}_{++}^{m} Λ ∈ S ++ m be its mean and covariance, respectively. Then
Ent ( π ) ≤ Ent ( N ( u , Λ ) ) = 1 2 log ( ( 2 π e ) m det Λ ) . \operatorname{Ent}(\pi)\leq\operatorname{Ent}\left(\mathcal{N}(u,\Lambda)\right)=\frac{1}{2}\log\left((2\pi e)^{m}\det\Lambda\right). Ent ( π ) ≤ Ent ( N ( u , Λ ) ) = 2 1 log ( ( 2 π e ) m det Λ ) . Equality holds if and only if π = N ( u , Λ ) \pi=\mathcal{N}(u,\Lambda) π = N ( u , Λ ) .
Take any feasible pair ( x , π ) (x,\pi) ( x , π ) , and set u = u ˉ π u=\bar{u}_{\pi} u = u ˉ π , Λ = Λ π \Lambda=\Lambda_{\pi} Λ = Λ π , and π G = N ( u , Λ ) \pi_{G}=\mathcal{N}(u,\Lambda) π G = N ( u , Λ ) . Since π G \pi_{G} π G and π \pi π have the same mean and covariance, the drift constraint remains unchanged: A x + B u ˉ π G + b = A x + B u ˉ π + b Ax+B\bar{u}_{\pi_{G}}+b=Ax+B\bar{u}_{\pi}+b A x + B u ˉ π G + b = A x + B u ˉ π + b . Also, all terms in (34) except the entropy term are unchanged. By Lemma 3.3, Ent ( π G ) ≥ Ent ( π ) \operatorname{Ent}(\pi_{G})\geq\operatorname{Ent}(\pi) Ent ( π G ) ≥ Ent ( π ) , and hence
F λ ( x , π G ) ≤ F λ ( x , π ) . F_{\lambda}(x,\pi_{G})\leq F_{\lambda}(x,\pi). F λ ( x , π G ) ≤ F λ ( x , π ) .
If π \pi π is not Gaussian, the above inequality is strict. Thus the general distribution problem (33) is equivalent to the following Gaussian moment problem.
Problem (O).
min x , u , Λ > 0 F λ ( x , u , Λ ) s.t. A x + B u + b = 0 , (35) \min_{x,u,\Lambda>0}\quad F_{\lambda}(x,u,\Lambda)\qquad\text{s.t.}\quad Ax+Bu+b=0, \tag*{(35)} x , u , Λ > 0 min F λ ( x , u , Λ ) s.t. A x + B u + b = 0 , (35)
where Λ > 0 \Lambda>0 Λ > 0 means Λ ∈ S + + m \Lambda\in\mathbb{S}_{++}^{m} Λ ∈ S ++ m , and
F λ ( x , u , Λ ) : = F 0 ( x , u ) + tr ( M ∞ Λ ) − λ 2 log ( ( 2 π e ) m det Λ ) . (36) F_{\lambda}(x,u,\Lambda):=F_{0}(x,u)+\operatorname{tr}(M_{\infty}\Lambda)-\frac{\lambda}{2}\log\left((2\pi e)^{m}\det\Lambda\right). \tag*{(36)} F λ ( x , u , Λ ) := F 0 ( x , u ) + tr ( M ∞ Λ ) − 2 λ log ( ( 2 π e ) m det Λ ) . (36)
We separate the terms in (35) that contain Λ \Lambda Λ . By (36) , we can write
F λ ( x , u , Λ ) = F 0 ( x , u ) + G λ ( Λ ) − λ 2 m log ( 2 π e ) , (37) F_{\lambda}(x,u,\Lambda)=F_{0}(x,u)+G_{\lambda}(\Lambda)-\frac{\lambda}{2}m\log(2\pi e), \tag*{(37)} F λ ( x , u , Λ ) = F 0 ( x , u ) + G λ ( Λ ) − 2 λ m log ( 2 π e ) , (37)
where
G λ ( Λ ) = tr ( M ∞ Λ ) − λ 2 log det Λ . (38) G_{\lambda}(\Lambda)=\operatorname{tr}(M_{\infty}\Lambda)-\frac{\lambda}{2}\log\det\Lambda. \tag*{(38)} G λ ( Λ ) = tr ( M ∞ Λ ) − 2 λ log det Λ. (38)
The last term of (37) only depends on λ \lambda λ and the dimension m m m , and does not affect the location of the optimizer. Hence the state–mean variables ( x , u ) (x,u) ( x , u ) and the exploration covariance variable Λ \Lambda Λ can be optimized separately.
Proposition 3.4. The covariance part G λ G_{\lambda} G λ , defined by (38) , has a unique minimizer on S + + m \mathbb{S}_{++}^{m} S ++ m , and this minimizer is
Λ ∗ λ = Λ ∞ λ : = λ 2 M ∞ − 1 = λ 2 ( R + D T P ∞ D ) − 1 . (39) \Lambda_{\ast}^{\lambda}=\Lambda_{\infty}^{\lambda}:=\frac{\lambda}{2}M_{\infty}^{-1}=\frac{\lambda}{2}\left(R+D^{\mathsf{T}}P_{\infty}D\right)^{-1}. \tag*{(39)} Λ ∗ λ = Λ ∞ λ := 2 λ M ∞ − 1 = 2 λ ( R + D T P ∞ D ) − 1 . (39) Proof. Let H = H T H=H^{\mathsf{T}} H = H T be any symmetric perturbation. By linearity of the trace and the differentiation formulas for matrix-variable functions (see [25 Section 2.2.2] ), setting f ( Λ ) = log det Λ f(\Lambda)=\log\det\Lambda f ( Λ ) = log det Λ gives
∇ Λ tr ( M ∞ Λ ) = M ∞ , ∇ f ( Λ ) = ( Λ − 1 ) T = Λ − 1 , ∇ 2 f ( Λ ) H = − Λ − 1 H Λ − 1 . \nabla_{\Lambda}\operatorname{tr}(M_{\infty}\Lambda)=M_{\infty},\qquad\nabla f(\Lambda)=(\Lambda^{-1})^{\mathsf{T}}=\Lambda^{-1},\qquad\nabla^{2}f(\Lambda)H=-\Lambda^{-1}H\Lambda^{-1}. ∇ Λ tr ( M ∞ Λ ) = M ∞ , ∇ f ( Λ ) = ( Λ − 1 ) T = Λ − 1 , ∇ 2 f ( Λ ) H = − Λ − 1 H Λ − 1 . It follows that ∇ G λ ( Λ ) = M ∞ − λ 2 Λ − 1 \nabla G_{\lambda}(\Lambda)=M_{\infty}-\frac{\lambda}{2}\Lambda^{-1} ∇ G λ ( Λ ) = M ∞ − 2 λ Λ − 1 . The first-order condition for an interior minimizer is M ∞ − λ 2 Λ − 1 = 0 M_{\infty}-\frac{\lambda}{2}\Lambda^{-1}=0 M ∞ − 2 λ Λ − 1 = 0 , which gives (39) ; since M ∞ > 0 M_{\infty}>0 M ∞ > 0 , this stationary point belongs to S + + m \mathbb{S}_{++}^{m} S ++ m . Moreover, for every symmetric matrix H ≠ 0 H\ne0 H = 0 ,
⟨ ∇ 2 G λ ( Λ ) H , H ⟩ = λ 2 tr ( Λ − 1 H Λ − 1 H ) = λ 2 ∥ Λ − 1 / 2 H Λ − 1 / 2 ∥ F 2 > 0. \left\langle\nabla^{2}G_{\lambda}(\Lambda)H,H\right\rangle=\frac{\lambda}{2}\operatorname{tr}\left(\Lambda^{-1}H\Lambda^{-1}H\right)=\frac{\lambda}{2}\left\|\Lambda^{-1/2}H\Lambda^{-1/2}\right\|_{F}^{2}>0. ⟨ ∇ 2 G λ ( Λ ) H , H ⟩ = 2 λ tr ( Λ − 1 H Λ − 1 H ) = 2 λ Λ − 1/2 H Λ − 1/2 F 2 > 0. Thus G λ G_{\lambda} G λ is strictly convex on S + + m \mathbb{S}_{++}^{m} S ++ m , and the stationary point in (39) is its unique global minimizer.
Since the optimized covariance part only contributes a constant independent of ( x , u ) (x,u) ( x , u ) , minimizing with respect to ( x , u ) (x,u) ( x , u ) is exactly the classical stochastic static problem (27) . Therefore the static mean center of the exploratory problem is unchanged:
( x ⋆ λ , u ⋆ λ ) = ( x ⋆ , u ⋆ ) . (x_\star^\lambda,u_\star^\lambda)=(x_\star,u_\star). ( x ⋆ λ , u ⋆ λ ) = ( x ⋆ , u ⋆ ) .
Hence the unique optimizer of the Gaussian moment problem (35) is
( x ⋆ , u ⋆ , Λ ⋆ λ ) , (x_\star,u_\star,\Lambda_\star^\lambda), ( x ⋆ , u ⋆ , Λ ⋆ λ ) ,
and the corresponding static Gaussian control distribution is
π ⋆ λ ( ⋅ ) = N ( u ⋆ , Λ ⋆ λ ) . \pi_\star^\lambda(\cdot)=\mathcal{N}(u_\star,\Lambda_\star^\lambda). π ⋆ λ ( ⋅ ) = N ( u ⋆ , Λ ⋆ λ ) .
Proposition 3.5. Under ( H 1 ) (H1) ( H 1 ) –( H 2 ) (H2) ( H 2 ) , the Problem ( O ) (O) ( O ) defined by (35) has a unique optimizer ( x ⋆ , u ⋆ , Λ ⋆ λ ) (x_\star,u_\star,\Lambda_\star^\lambda) ( x ⋆ , u ⋆ , Λ ⋆ λ ) , where ( x ⋆ , u ⋆ ) (x_\star,u_\star) ( x ⋆ , u ⋆ ) is the unique solution of the classical stochastic static problem (27) , and Λ ⋆ λ = λ 2 ( R + D ⊤ P ∞ D ) − 1 \Lambda_\star^\lambda=\frac{\lambda}{2}(R+D^\top P_\infty D)^{-1} Λ ⋆ λ = 2 λ ( R + D ⊤ P ∞ D ) − 1 . Equivalently, the static exploratory control distribution of (33) is given by N ( u ⋆ , Λ ⋆ λ ) \mathcal{N}(u_\star,\Lambda_\star^\lambda) N ( u ⋆ , Λ ⋆ λ ) .
Proof. The separation formula (37) shows that, up to an additive constant, the minimization over ( x , u ) (x,u) ( x , u ) and the minimization over Λ \Lambda Λ are independent. The ( x , u ) (x,u) ( x , u ) part is exactly (27) , whose unique solution is ( x ⋆ , u ⋆ ) (x_\star,u_\star) ( x ⋆ , u ⋆ ) by Proposition 3.1. The covariance part has the unique solution (39) by Proposition 3.4. The proof is complete. ■
Convergence of Riccati Equations
This section studies the convergence of finite-horizon parameters as T → ∞ T \to\infty T → ∞ . The Gaussian feedback law in Proposition 2.2 is determined by the same Riccati equation (9) as in the corresponding classical stochastic LQ problem. In this section, the constants C C C , γ > 0 \gamma> 0 γ > 0 may change from line to line, but they are always independent of the time horizon T T T . Unless otherwise stated, the exploration weight λ > 0 \lambda> 0 λ > 0 is fixed.
Recall that P T P_T P T is the unique strongly regular solution of (9) , and P ∞ P_\infty P ∞ is the unique stabilizing solution of (22) . Similar to Sun–Wang–Yong [34 ] , we next establish an estimate for the difference between P T P_T P T and P ∞ P_\infty P ∞ . However, since the cost functional is indefinite, the method employed in Sun–Wang–Yong [34 ] cannot be applied here. Motivated by Sun–Yong [37 ] , we overcome this difficulty via a Hilbert space approach.
Lemma 4.1. Suppose that the uncontrolled system [ A , C ] [A,C] [ A , C ] is L 2 L^2 L 2 -stable and that uniform convexity (17) holds with a common constant δ T = δ 0 > 0 \delta_T=\delta_0>0 δ T = δ 0 > 0 independent of T T T . Let P T P_T P T be the unique strongly regular solution of (9) , and let P ∞ P_\infty P ∞ be the unique stabilizing solution of (22) . Then there exist constants C , γ > 0 C,\gamma>0 C , γ > 0 , independent of T T T , such that
∥ P T ( t ) − P ∞ ∥ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T . (40) \lVert P_T(t)-P_\infty\rVert\le Ce^{-\gamma(T-t)}, \qquad0\le t\le T. \tag*{(40)} ∥ P T ( t ) − P ∞ ∥ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T . (40) Moreover, P T ( t ) P_T(t) P T ( t ) is uniformly bounded with respect to both T T T and t t t .
Proof. By Proposition 2.11, the assumed horizon-independent lower bound implies (15) with a constant δ > 0 \delta>0 δ > 0 independent of T T T . We use this joint coercivity constant below. Proposition 2.15 and Corollary 2.19 then ensure the existence of P T P_T P T and the stabilizing solution P ∞ P_\infty P ∞ .
Fix 0 ≤ t < T 0\le t<T 0 ≤ t < T , and let
H t , T = L 2 ( t , T ; R m ) , ∥ u ∥ H t , T 2 = E ∫ t T ∣ u ( s ) ∣ 2 d s . \mathcal{H}_{t,T}=L^2(t,T;\mathbb{R}^m), \qquad\lVert u\rVert_{\mathcal{H}_{t,T}}^2=\mathbb{E}\int_t^T\lvert u(s)\rvert^2\,\mathrm{d}s. H t , T = L 2 ( t , T ; R m ) , ∥ u ∥ H t , T 2 = E ∫ t T ∣ u ( s ) ∣ 2 d s .
Consider the following homogeneous controlled state equation
{ d X ( s ) = ( A X ( s ) + B u ( s ) ) d s + ( C X ( s ) + D u ( s ) ) d W ( s ) , s ∈ [ t , T ] , X ( t ) = x . (41) \begin{cases}
\mathrm{d}X(s)=(AX(s)+Bu(s))\mathrm{d}s+(CX(s)+Du(s))\mathrm{d}W(s), & s\in[t,T],\\
X(t)=x. \tag*{(41)}
\end{cases} { d X ( s ) = ( A X ( s ) + B u ( s )) d s + ( C X ( s ) + D u ( s )) d W ( s ) , X ( t ) = x . s ∈ [ t , T ] , (41)
Denote the corresponding finite-horizon cost functional by
J t , T 0 ( x ; u ) = E ∫ t T [ X ( s ) ⊤ Q X ( s ) + 2 X ( s ) ⊤ S ⊤ u ( s ) + u ( s ) ⊤ R u ( s ) ] d s . J^0_{t,T}(x;u)=\mathbb{E}\int_t^T\left[X(s)^\top QX(s)+2X(s)^\top S^\top u(s)+u(s)^\top Ru(s)\right]\mathrm{d}s. J t , T 0 ( x ; u ) = E ∫ t T [ X ( s ) ⊤ QX ( s ) + 2 X ( s ) ⊤ S ⊤ u ( s ) + u ( s ) ⊤ R u ( s ) ] d s .
By classical LQ theory [33 ] , the corresponding value function is
V T 0 ( t , x ) : = inf u ∈ H t , T J t , T 0 ( x ; u ) = x ⊤ P T ( t ) x . V^0_T(t,x):=\inf_{u\in\mathcal{H}_{t,T}}J^0_{t,T}(x;u)=x^\top P_T(t)x. V T 0 ( t , x ) := u ∈ H t , T inf J t , T 0 ( x ; u ) = x ⊤ P T ( t ) x .
We denote the solution of the state equation (41) with initial pair ( t , x ) (t,x) ( t , x ) and control u u u by X t , x ; u X^{t,x;u} X t , x ; u . Define the zero-initial-state trajectory operator and the terminal-state operator by
L t , T u = X t , 0 ; u ( ⋅ ) , L t , T : H t , T ⟶ L 2 ( t , T ; R n ) , \mathcal{L}_{t,T}u=X^{t,0;u}(\cdot),\qquad\mathcal{L}_{t,T}:\mathcal{H}_{t,T}\longrightarrow L^2(t,T;\mathbb{R}^n), L t , T u = X t , 0 ; u ( ⋅ ) , L t , T : H t , T ⟶ L 2 ( t , T ; R n ) ,
L ^ t , T u = X t , 0 ; u ( T ) , L ^ t , T : H t , T ⟶ L F T 2 ( Ω ; R n ) . \widehat{\mathcal{L}}_{t,T}u=X^{t,0;u}(T),\qquad\widehat{\mathcal{L}}_{t,T}:\mathcal{H}_{t,T}\longrightarrow L^2_{\mathcal{F}_T}(\Omega;\mathbb{R}^n). L t , T u = X t , 0 ; u ( T ) , L t , T : H t , T ⟶ L F T 2 ( Ω ; R n ) .
By Lemma 2.7, there exists a constant C > 0 C>0 C > 0 , independent of the time horizon [ t , T ] [t,T] [ t , T ] , such that
∥ L t , T ∥ + ∥ L ^ t , T ∥ ≤ C . (42) \|\mathcal{L}_{t,T}\|+\|\widehat{\mathcal{L}}_{t,T}\|\le C. \tag*{(42)} ∥ L t , T ∥ + ∥ L t , T ∥ ≤ C . (42)
By linearity and uniqueness of the state equation, X t , x ; u = X t , x ; 0 + L t , T u X^{t,x;u}=X^{t,x;0}+\mathcal{L}_{t,T}u X t , x ; u = X t , x ; 0 + L t , T u . Thus the initial state contributes only to the affine terms below, whereas the quadratic part in u u u is exactly the part determined by the zero-initial-state operator. Pulling the state terms back to the control space, we can write J t , T 0 ( x ; ⋅ ) J^0_{t,T}(x;\cdot) J t , T 0 ( x ; ⋅ ) uniquely as
J t , T 0 ( x ; u ) = ⟨ M t , T u , u ⟩ + 2 ⟨ h t , T ( x ) , u ⟩ + c t , T ( x ) , (43) J^0_{t,T}(x;u)=\langle\mathcal{M}_{t,T}u,u\rangle+2\langle h_{t,T}(x),u\rangle+c_{t,T}(x), \tag*{(43)} J t , T 0 ( x ; u ) = ⟨ M t , T u , u ⟩ + 2 ⟨ h t , T ( x ) , u ⟩ + c t , T ( x ) , (43)
where M t , T \mathcal{M}_{t,T} M t , T is a bounded self-adjoint operator and
M t , T = L t , T ∗ Q L t , T + L t , T ∗ S ⊤ + S L t , T + R . \mathcal{M}_{t,T}=\mathcal{L}_{t,T}^*Q\mathcal{L}_{t,T}+\mathcal{L}_{t,T}^*S^\top+S\mathcal{L}_{t,T}+R. M t , T = L t , T ∗ Q L t , T + L t , T ∗ S ⊤ + S L t , T + R .
By time homogeneity and the joint coercivity condition (15) established above, we have
M t , T ≥ δ I , ∥ M t , T − 1 ∥ ≤ δ − 1 . (44) \mathcal{M}_{t,T}\ge\delta I,\qquad\|\mathcal{M}_{t,T}^{-1}\|\le\delta^{-1}. \tag*{(44)} M t , T ≥ δ I , ∥ M t , T − 1 ∥ ≤ δ − 1 . (44)
Consequently, the finite-horizon problem with zero terminal cost admits a unique optimal control u T u_T u T . Denote its optimal state by X T : = X t , x ; u T X_T:=X^{t,x;u_T} X T := X t , x ; u T . The first-order optimality condition is
M t , T u T + h t , T ( x ) = 0. (45) \mathcal{M}_{t,T}u_T+h_{t,T}(x)=0. \tag*{(45)} M t , T u T + h t , T ( x ) = 0. (45)
We next compare this problem with the infinite-horizon problem on the same control space. Let u ∞ = − K ∞ X ∞ u_\infty=-K_\infty X_\infty u ∞ = − K ∞ X ∞ be the restriction to [ t , T ] [t,T] [ t , T ] of the infinite-horizon optimal control. By the dynamic programming principle for the infinite-horizon problem and the identity V ∞ 0 ( x ) = x ⊤ P ∞ x V^0_\infty(x)=x^\top P_\infty x V ∞ 0 ( x ) = x ⊤ P ∞ x ,
x ⊤ P ∞ x = inf u ∈ H t , T { J t , T 0 ( x ; u ) + E [ X ( T ) ⊤ P ∞ X ( T ) ] } , (46) x^\top P_\infty x=\inf_{u\in\mathcal{H}_{t,T}}\left\{J^0_{t,T}(x;u)+\mathbb{E}\left[X(T)^\top P_\infty X(T)\right]\right\}, \tag*{(46)} x ⊤ P ∞ x = u ∈ H t , T inf { J t , T 0 ( x ; u ) + E [ X ( T ) ⊤ P ∞ X ( T ) ] } , (46)
and the minimizer is precisely u ∞ u_\infty u ∞ . Taking the Fréchet derivative of the terminal quadratic term E [ X ( T ) ⊤ P ∞ X ( T ) ] \mathbb{E}[X(T)^\top P_\infty X(T)] E [ X ( T ) ⊤ P ∞ X ( T )] in (46) , we obtain, for every v ∈ H t , T v\in\mathcal{H}_{t,T} v ∈ H t , T ,
D u [ E [ X ( T ) ⊤ P ∞ X ( T ) ] ] ( u ∞ ) [ v ] = 2 E [ X ∞ ( T ) ⊤ P ∞ L ^ t , T v ] = 2 ⟨ L ^ t , T ∗ ( P ∞ X ∞ ( T ) ) , v ⟩ H t , T . D_u\left[\mathbb{E}[X(T)^\top P_\infty X(T)]\right](u_\infty)[v]=2\mathbb{E}\left[X_\infty(T)^\top P_\infty\widehat{\mathcal{L}}_{t,T}v\right]=2\left\langle\widehat{\mathcal{L}}_{t,T}^*(P_\infty X_\infty(T)),v\right\rangle_{\mathcal{H}_{t,T}}. D u [ E [ X ( T ) ⊤ P ∞ X ( T )] ] ( u ∞ ) [ v ] = 2 E [ X ∞ ( T ) ⊤ P ∞ L t , T v ] = 2 ⟨ L t , T ∗ ( P ∞ X ∞ ( T )) , v ⟩ H t , T .
On the other hand, (43) gives the derivative of the running-cost term J t , T 0 ( x ; u ) J^0_{t,T}(x;u) J t , T 0 ( x ; u ) :
D u J t , T 0 ( x ; u ∞ ) [ v ] = 2 ⟨ M t , T u ∞ + h t , T ( x ) , v ⟩ H t , T , v ∈ H t , T . D_uJ^0_{t,T}(x;u_\infty)[v]=2\left\langle\mathcal{M}_{t,T}u_\infty+h_{t,T}(x),v\right\rangle_{\mathcal{H}_{t,T}},\qquad v\in\mathcal{H}_{t,T}. D u J t , T 0 ( x ; u ∞ ) [ v ] = 2 ⟨ M t , T u ∞ + h t , T ( x ) , v ⟩ H t , T , v ∈ H t , T .
Therefore, the minimizer u ∞ u_{\infty} u ∞ satisfies the first-order optimality condition for (46)
M t , T u ∞ + h t , T ( x ) + L ^ t , T ∗ ( P ∞ X ∞ ( T ) ) = 0. (47) \mathcal{M}_{t,T}u_{\infty}+h_{t,T}(x)+\widehat{\mathcal{L}}_{t,T}^{*}(P_{\infty}X_{\infty}(T))=0. \tag*{(47)} M t , T u ∞ + h t , T ( x ) + L t , T ∗ ( P ∞ X ∞ ( T )) = 0. (47)
Subtracting (45) from (47) and using (44) , we obtain
u T − u ∞ = M t , T − 1 L ^ t , T ∗ ( P ∞ X ∞ ( T ) ) , (48) u_T-u_{\infty}=\mathcal{M}_{t,T}^{-1}\widehat{\mathcal{L}}_{t,T}^{*}(P_{\infty}X_{\infty}(T)), \tag*{(48)} u T − u ∞ = M t , T − 1 L t , T ∗ ( P ∞ X ∞ ( T )) , (48)
∥ u T − u ∞ ∥ H t , T ≤ δ − 1 ∥ L ^ t , T ∥ ∥ P ∞ ∥ ( E ∣ X ∞ ( T ) ∣ 2 ) 1 / 2 . \lVert u_T-u_{\infty}\rVert_{H_{t,T}}\leq\delta^{-1}\lVert\widehat{\mathcal{L}}_{t,T}\rVert\lVert P_{\infty}\rVert\bigl(\mathbb{E}|X_{\infty}(T)|^2\bigr)^{1/2}. ∥ u T − u ∞ ∥ H t , T ≤ δ − 1 ∥ L t , T ∥ ∥ P ∞ ∥ ( E ∣ X ∞ ( T ) ∣ 2 ) 1/2 .
By Corollary 2.19, the closed-loop system [ A − B K ∞ , C − D K ∞ ] [A-BK_{\infty},C-DK_{\infty}] [ A − B K ∞ , C − D K ∞ ] is L 2 L^2 L 2 -stable. Applying Corollary 2.8 to this closed-loop system with initial time t t t and terminal time T T T , we obtain
E ∣ X ∞ ( T ) ∣ 2 ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 . (49) \mathbb{E}|X_{\infty}(T)|^2\leq Ce^{-2\gamma(T-t)}|x|^2. \tag*{(49)} E ∣ X ∞ ( T ) ∣ 2 ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 . (49)
It follows from (48) that
∥ u T − u ∞ ∥ H t , T ≤ C e − γ ( T − t ) ∣ x ∣ . (50) \lVert u_T-u_{\infty}\rVert_{H_{t,T}}\leq Ce^{-\gamma(T-t)}|x|. \tag*{(50)} ∥ u T − u ∞ ∥ H t , T ≤ C e − γ ( T − t ) ∣ x ∣. (50)
The two optimal states have the same initial condition, and therefore
X T ( T ) − X ∞ ( T ) = L ^ t , T ( u T − u ∞ ) . X_T(T)-X_{\infty}(T)=\widehat{\mathcal{L}}_{t,T}(u_T-u_{\infty}). X T ( T ) − X ∞ ( T ) = L t , T ( u T − u ∞ ) .
Combining (42) , (49) , and (50) , we obtain
E ∣ X T ( T ) ∣ 2 ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 . (51) \mathbb{E}|X_T(T)|^2\leq Ce^{-2\gamma(T-t)}|x|^2. \tag*{(51)} E ∣ X T ( T ) ∣ 2 ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 . (51)
It remains to compare the two value functions. For the first comparison, take the infinite-horizon optimal control on [ t , ∞ ) [t,\infty) [ t , ∞ ) and restrict it to [ t , T ] [t,T] [ t , T ] . We continue to denote this restriction by u ∞ u_{\infty} u ∞ . It is admissible for the finite-horizon problem with zero terminal cost. Therefore,
V T 0 ( t , x ) − V ∞ ( x ) ≤ J t , T 0 ( x ; u ∞ ) − V ∞ ( x ) = − E [ X ∞ ( T ) ⊤ P ∞ X ∞ ( T ) ] ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 . V_T^0(t,x)-V^{\infty}(x)\leq J_{t,T}^0(x;u_{\infty})-V^{\infty}(x)=-\mathbb{E}\left[X_{\infty}(T)^{\top}P_{\infty}X_{\infty}(T)\right]\leq Ce^{-2\gamma(T-t)}|x|^2. V T 0 ( t , x ) − V ∞ ( x ) ≤ J t , T 0 ( x ; u ∞ ) − V ∞ ( x ) = − E [ X ∞ ( T ) ⊤ P ∞ X ∞ ( T ) ] ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 .
Conversely, using u T u_T u T as a competitor in (46) and the identity J t , T 0 ( x ; u T ) = V T 0 ( t , x ) J_{t,T}^0(x;u_T)=V_T^0(t,x) J t , T 0 ( x ; u T ) = V T 0 ( t , x ) , we obtain
V ∞ ( x ) ≤ J t , T 0 ( x ; u T ) + E [ X T ( T ) ⊤ P ∞ X T ( T ) ] = V T 0 ( t , x ) + E [ X T ( T ) ⊤ P ∞ X T ( T ) ] . V^{\infty}(x)\leq J_{t,T}^0(x;u_T)+\mathbb{E}\left[X_T(T)^{\top}P_{\infty}X_T(T)\right]=V_T^0(t,x)+\mathbb{E}\left[X_T(T)^{\top}P_{\infty}X_T(T)\right]. V ∞ ( x ) ≤ J t , T 0 ( x ; u T ) + E [ X T ( T ) ⊤ P ∞ X T ( T ) ] = V T 0 ( t , x ) + E [ X T ( T ) ⊤ P ∞ X T ( T ) ] .
Using (51) , we obtain
V ∞ ( x ) − V T 0 ( t , x ) ≤ E [ X T ( T ) ⊤ P ∞ X T ( T ) ] ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 . V^{\infty}(x)-V_T^0(t,x)\leq\mathbb{E}\left[X_T(T)^{\top}P_{\infty}X_T(T)\right]\leq Ce^{-2\gamma(T-t)}|x|^2. V ∞ ( x ) − V T 0 ( t , x ) ≤ E [ X T ( T ) ⊤ P ∞ X T ( T ) ] ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 .
Both estimates use only ∣ z ⊤ P ∞ z ∣ ≤ ∥ P ∞ ∥ ∣ z ∣ 2 |z^{\top}P_{\infty}z|\leq\lVert P_{\infty}\rVert|z|^2 ∣ z ⊤ P ∞ z ∣ ≤ ∥ P ∞ ∥ ∣ z ∣ 2 and do not use any nonnegativity property of P T ( t ) P_T(t) P T ( t ) or P ∞ P_{\infty} P ∞ . Thus,
∣ x ⊤ ( P T ( t ) − P ∞ ) x ∣ ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 . |x^{\top}(P_T(t)-P_{\infty})x|\leq Ce^{-2\gamma(T-t)}|x|^2. ∣ x ⊤ ( P T ( t ) − P ∞ ) x ∣ ≤ C e − 2 γ ( T − t ) ∣ x ∣ 2 .
Since P T ( t ) − P ∞ P_T(t)-P_{\infty} P T ( t ) − P ∞ is a real symmetric matrix, taking the supremum over ∣ x ∣ = 1 |x|=1 ∣ x ∣ = 1 and relabeling the exponent gives (40) . The same estimate also yields the uniform boundedness of P T ( t ) P_T(t) P T ( t ) .
Proposition 4.2. Suppose that (H1) and (H2) hold. Let P T P_T P T be the unique strongly regular solution of (9) , and let P ∞ P_{\infty} P ∞ be the unique stabilizing solution of (22) . Then there exist constants C , γ > 0 C,\gamma>0 C , γ > 0 , independent of T T T and t t t , such that
∥ P T ( t ) − P ∞ ∥ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T . (52) \lVert P_T(t)-P_{\infty}\rVert\leq Ce^{-\gamma(T-t)},\qquad0\leq t\leq T. \tag*{(52)} ∥ P T ( t ) − P ∞ ∥ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T . (52) Proof. By stabilizability, there exists a stabilizing feedback matrix Θ \Theta Θ . Set A Θ = A + B Θ A_{\Theta}=A+B\Theta A Θ = A + B Θ and C Θ = C + D Θ C_{\Theta}=C+D\Theta C Θ = C + D Θ . We denote the transformed finite-horizon cost by J Θ , T 0 ( x ; v ) J_{\Theta,T}^0(x;v) J Θ , T 0 ( x ; v ) , its value matrix by P Θ , T ( t ) P_{\Theta,T}(t) P Θ , T ( t ) , and the infinite-horizon value matrix by P Θ , ∞ P_{\Theta,\infty} P Θ , ∞ . Under the change of control u ( t ) = Θ X ( t ) + v ( t ) u(t)=\Theta X(t)+v(t) u ( t ) = Θ X ( t ) + v ( t ) , the state equation (41) and the cost coefficients become
{ d X ( t ) = ( A Θ X ( t ) + B v ( t ) ) d t + ( C Θ X ( t ) + D v ( t ) ) d W ( t ) , 0 ≤ t ≤ T , X ( 0 ) = x . (53) \begin{cases}
dX(t)=(A_{\Theta}X(t)+Bv(t))\,dt+(C_{\Theta}X(t)+Dv(t))\,dW(t), & 0\leq t\leq T,\\
X(0)=x. \tag*{(53)}
\end{cases} { d X ( t ) = ( A Θ X ( t ) + B v ( t )) d t + ( C Θ X ( t ) + D v ( t )) d W ( t ) , X ( 0 ) = x . 0 ≤ t ≤ T , (53)
Q Θ = Q + S T Θ + Θ T S + Θ T R Θ , S Θ = S + R Θ , R Θ = R . (54) Q_{\Theta}=Q+S^{\mathsf T}\Theta+\Theta^{\mathsf T}S+\Theta^{\mathsf T}R\Theta,\qquad S_{\Theta}=S+R\Theta,\qquad R_{\Theta}=R. \tag*{(54)} Q Θ = Q + S T Θ + Θ T S + Θ T R Θ , S Θ = S + R Θ , R Θ = R . (54)
For each fixed T T T , the transformations u ( t ) = Θ X ( t ) + v ( t ) u(t)=\Theta X(t)+v(t) u ( t ) = Θ X ( t ) + v ( t ) and v ( t ) = u ( t ) − Θ X ( t ) v(t)=u(t)-\Theta X(t) v ( t ) = u ( t ) − Θ X ( t ) are mutually inverse and preserve the initial state, the state trajectory, and the cost. The assumption (H2), together with ∣ v ( t ) ∣ 2 ≤ ( 1 + ∥ Θ ∥ 2 ) ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) |v(t)|^2\leq(1+\|\Theta\|^2)(|X(t)|^2+|u(t)|^2) ∣ v ( t ) ∣ 2 ≤ ( 1 + ∥Θ ∥ 2 ) ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) , yields
J Θ , T 0 ( 0 ; v ) ≥ δ 1 + ∥ Θ ∥ 2 E ∫ 0 T ∣ v ( t ) ∣ 2 d t . (55) J_{\Theta,T}^{0}(0;v)\geq\frac{\delta}{1+\|\Theta\|^2}\mathbb{E}\int_0^T|v(t)|^2\mathrm{d}t. \tag*{(55)} J Θ , T 0 ( 0 ; v ) ≥ 1 + ∥Θ ∥ 2 δ E ∫ 0 T ∣ v ( t ) ∣ 2 d t . (55)
On the infinite horizon, by Sun–Yong [35 Proposition 3.3] , the transformation u ( t ) = Θ X ( t ) + v ( t ) u(t)=\Theta X(t)+v(t) u ( t ) = Θ X ( t ) + v ( t ) defines a bijection from L F 2 ( 0 , ∞ ; R m ) L_{\mathbb{F}}^2(0,\infty;\mathbb{R}^m) L F 2 ( 0 , ∞ ; R m ) onto U a d ( x ) \mathcal{U}_{\mathrm{ad}}(x) U ad ( x ) , with inverse v ( t ) = u ( t ) − Θ X ( t ) v(t)=u(t)-\Theta X(t) v ( t ) = u ( t ) − Θ X ( t ) . Thus, on both finite and infinite horizons, corresponding controls yield the same state trajectory and the same cost. The value functions therefore coincide. Their quadratic representations give P Θ , T ( 0 ) = P T ( 0 ) P_{\Theta,T}(0)=P_T(0) P Θ , T ( 0 ) = P T ( 0 ) and P Θ , ∞ = P ∞ P_{\Theta,\infty}=P_\infty P Θ , ∞ = P ∞ . Moreover, time homogeneity yields P Θ , T ( t ) = P Θ , T − t ( 0 ) = P T − t ( 0 ) = P T ( t ) P_{\Theta,T}(t)=P_{\Theta,T-t}(0)=P_{T-t}(0)=P_T(t) P Θ , T ( t ) = P Θ , T − t ( 0 ) = P T − t ( 0 ) = P T ( t ) . The value matrices before and after the transformation satisfy
P Θ , T ( t ) = P T ( t ) , P Θ , ∞ = P ∞ , 0 ≤ t ≤ T . P_{\Theta,T}(t)=P_T(t),\qquad P_{\Theta,\infty}=P_\infty,\qquad0\leq t\leq T. P Θ , T ( t ) = P T ( t ) , P Θ , ∞ = P ∞ , 0 ≤ t ≤ T .
Since [ A Θ , C Θ ] [A_\Theta,C_\Theta] [ A Θ , C Θ ] is L 2 L^2 L 2 -stable and (55) holds with a constant independent of T T T , Lemma 4.1 applies to (53) with the cost coefficients (54) . Using P Θ , T ( t ) = P T ( t ) P_{\Theta,T}(t)=P_T(t) P Θ , T ( t ) = P T ( t ) and P Θ , ∞ = P ∞ P_{\Theta,\infty}=P_\infty P Θ , ∞ = P ∞ , we obtain (52) . ■
The above implies that the J C \mathcal{JC} J C condition (H2) implies the convergence estimate (52) of Riccati equations. This convergence estimate will play the most important role in the study of the turnpike property. Next, we prove the converse result; that is the convergence of Riccati equations implies the J C \mathcal{JC} J C condition (H2) holds.
Theorem 4.3. Consider the finite-horizon Riccati equation (9) , with the notation M T M_T M T , N T N_T N T , K T K_T K T in Proposition 2.2. Suppose that the following conditions hold:
(i) For every T > 0 T>0 T > 0 , the Riccati equation (9) admits a strongly regular solution P T ∈ C ( [ 0 , T ] ; S n ) P_T\in C([0,T];\mathbb{S}^n) P T ∈ C ([ 0 , T ] ; S n ) . More precisely, there exists μ T > 0 \mu_T>0 μ T > 0 , such that
M T ( t ) = R + D T P T ( t ) D ≥ μ T I m , 0 ≤ t ≤ T . M_T(t)=R+D^{\mathsf T}P_T(t)D\geq\mu_T I_m,\qquad0\leq t\leq T. M T ( t ) = R + D T P T ( t ) D ≥ μ T I m , 0 ≤ t ≤ T . (ii) There exists P ∞ ∈ S n P_\infty\in\mathbb{S}^n P ∞ ∈ S n such that
lim T → ∞ ∥ P T ( 0 ) − P ∞ ∥ = 0 , \lim_{T\to\infty}\|P_T(0)-P_\infty\|=0, T → ∞ lim ∥ P T ( 0 ) − P ∞ ∥ = 0 , and [ A − B K ∞ , C − D K ∞ ] [A-BK_\infty,C-DK_\infty] [ A − B K ∞ , C − D K ∞ ] is L 2 L^2 L 2 -stable, where
M ∞ ≔ R + D T P ∞ D > 0 , K ∞ = M ∞ − 1 ( B T P ∞ + D T P ∞ C + S ) . M_\infty\coloneqq R+D^{\mathsf T}P_\infty D>0,\qquad K_\infty=M_\infty^{-1}(B^{\mathsf T}P_\infty+D^{\mathsf T}P_\infty C+S). M ∞ : = R + D T P ∞ D > 0 , K ∞ = M ∞ − 1 ( B T P ∞ + D T P ∞ C + S ) . Then P ∞ P_\infty P ∞ is the unique stabilizing solution of the algebraic Riccati equation (22) , and there exists δ > 0 \delta>0 δ > 0 , independent of T T T , such that
J T 0 ( 0 ; u ) ≥ δ E ∫ 0 T ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , T > 0 , u ∈ L F 2 ( 0 , T ; R m ) . J_T^0(0;u)\geq\delta\mathbb{E}\int_0^T\left(|X(t)|^2+|u(t)|^2\right)\mathrm{d}t,\qquad T>0,\quad u\in L_{\mathbb{F}}^2(0,T;\mathbb{R}^m). J T 0 ( 0 ; u ) ≥ δ E ∫ 0 T ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , T > 0 , u ∈ L F 2 ( 0 , T ; R m ) . That is, (H2) holds.
Proof. We first establish a uniform positive lower bound for M T ( t ) M_T(t) M T ( t ) . Since the coefficients are constant and P T ( T ) = 0 P_T(T)=0 P T ( T ) = 0 for every T > 0 T>0 T > 0 , time reversal transforms the finite-horizon Riccati equations into the same initial-value problem; see [34 , 35 ] . By uniqueness, there exists a function Π ∈ C 1 ( [ 0 , ∞ ) ; S n ) \Pi\in C^1([0,\infty);\mathbb{S}^n) Π ∈ C 1 ([ 0 , ∞ ) ; S n ) such that
P T ( t ) = Π ( T − t ) , 0 ≤ t ≤ T , Π ( 0 ) = 0. P_T(t)=\Pi(T-t),\qquad0\leq t\leq T,\qquad\Pi(0)=0. P T ( t ) = Π ( T − t ) , 0 ≤ t ≤ T , Π ( 0 ) = 0.
In particular, Π ( T ) = P T ( 0 ) \Pi(T)=P_T(0) Π ( T ) = P T ( 0 ) , so the assumed convergence implies that Π ( s ) → P ∞ \Pi(s)\to P_\infty Π ( s ) → P ∞ as s → ∞ s\to\infty s → ∞ .
Define
M ( s ) = R + D T Π ( s ) D , N ( s ) = B T Π ( s ) + D T Π ( s ) C + S , K ( s ) = M ( s ) − 1 N ( s ) . M(s)=R+D^{\mathsf T}\Pi(s)D,\qquad N(s)=B^{\mathsf T}\Pi(s)+D^{\mathsf T}\Pi(s)C+S,\qquad K(s)=M(s)^{-1}N(s). M ( s ) = R + D T Π ( s ) D , N ( s ) = B T Π ( s ) + D T Π ( s ) C + S , K ( s ) = M ( s ) − 1 N ( s ) .
Then M T ( t ) = M ( T − t ) M_T(t)=M(T-t) M T ( t ) = M ( T − t ) and K T ( t ) = K ( T − t ) K_T(t)=K(T-t) K T ( t ) = K ( T − t ) for 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T . From the Riccati equation (9) , we obtain the following time-reversed equation:
{ Π ′ ( s ) = Π ( s ) A + A T Π ( s ) + C T Π ( s ) C + Q − N ( s ) T M ( s ) − 1 N ( s ) , s ≥ 0 , Π ( 0 ) = 0. (56) \begin{cases}
\Pi'(s)=\Pi(s)A+A^{\mathsf T}\Pi(s)+C^{\mathsf T}\Pi(s)C+Q-N(s)^{\mathsf T}M(s)^{-1}N(s), & s\ge0,\\
\Pi(0)=0. \tag*{(56)}
\end{cases} { Π ′ ( s ) = Π ( s ) A + A T Π ( s ) + C T Π ( s ) C + Q − N ( s ) T M ( s ) − 1 N ( s ) , Π ( 0 ) = 0. s ≥ 0 , (56)
Strong regularity on each finite horizon gives M ( s ) > 0 M(s)>0 M ( s ) > 0 for all s ≥ 0 s\ge0 s ≥ 0 . Moreover, M ( s ) → M ∞ > 0 M(s)\to M_\infty>0 M ( s ) → M ∞ > 0 as s → ∞ s\to\infty s → ∞ , so there exists s 0 > 0 s_0>0 s 0 > 0 such that
M ( s ) ≥ 1 2 λ min ( M ∞ ) I m , s ≥ s 0 . M(s)\ge\frac{1}{2}\lambda_{\min}(M_\infty)I_m,\qquad s\ge s_0. M ( s ) ≥ 2 1 λ m i n ( M ∞ ) I m , s ≥ s 0 .
Since M ( s ) M(s) M ( s ) is continuous and positive definite on the compact interval [ 0 , s 0 ] [0,s_0] [ 0 , s 0 ] , we also have μ 0 : = min 0 ≤ s ≤ s 0 λ min ( M ( s ) ) > 0 \mu_0:=\min_{0\le s\le s_0}\lambda_{\min}(M(s))>0 μ 0 := min 0 ≤ s ≤ s 0 λ m i n ( M ( s )) > 0 . Combining these bounds and setting μ = min { μ 0 , 1 2 λ min ( M ∞ ) } > 0 \mu=\min\{\mu_0,\frac{1}{2}\lambda_{\min}(M_\infty)\}>0 μ = min { μ 0 , 2 1 λ m i n ( M ∞ )} > 0 , we obtain
M ( s ) ≥ μ I m , ∥ M ( s ) − 1 ∥ ≤ μ − 1 , s ≥ 0. M(s)\ge\mu I_m,\qquad\lVert M(s)^{-1}\rVert\le\mu^{-1},\qquad s\ge0. M ( s ) ≥ μ I m , ∥ M ( s ) − 1 ∥ ≤ μ − 1 , s ≥ 0.
Since Π ( s ) \Pi(s) Π ( s ) is continuous and convergent, N ( s ) N(s) N ( s ) is bounded on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) . The uniform bound on M ( s ) − 1 M(s)^{-1} M ( s ) − 1 then implies that K ( s ) K(s) K ( s ) is also bounded. Moreover, the definitions of K ( s ) K(s) K ( s ) and K ∞ K_\infty K ∞ give
K ( s ) − K ∞ = M ( s ) − 1 [ B T ( Π ( s ) − P ∞ ) + D T ( Π ( s ) − P ∞ ) ( C − D K ∞ ) ] . K(s)-K_\infty=M(s)^{-1}\left[B^{\mathsf T}(\Pi(s)-P_\infty)+D^{\mathsf T}(\Pi(s)-P_\infty)(C-DK_\infty)\right]. K ( s ) − K ∞ = M ( s ) − 1 [ B T ( Π ( s ) − P ∞ ) + D T ( Π ( s ) − P ∞ ) ( C − D K ∞ ) ] .
Using ∥ M ( s ) − 1 ∥ ≤ μ − 1 \lVert M(s)^{-1}\rVert\le\mu^{-1} ∥ M ( s ) − 1 ∥ ≤ μ − 1 , we obtain
∥ K ( s ) − K ∞ ∥ ≤ C ∥ Π ( s ) − P ∞ ∥ ⟶ 0 , s → ∞ , \lVert K(s)-K_\infty\rVert\le C\lVert\Pi(s)-P_\infty\rVert\longrightarrow0,\qquad s\to\infty, ∥ K ( s ) − K ∞ ∥ ≤ C ∥ Π ( s ) − P ∞ ∥ ⟶ 0 , s → ∞ ,
where C > 0 C>0 C > 0 is independent of s s s .
We next show that P ∞ P_\infty P ∞ satisfies (22) . Since Π ( s ) → P ∞ \Pi(s)\to P_\infty Π ( s ) → P ∞ and M ( s ) → M ∞ > 0 M(s)\to M_\infty>0 M ( s ) → M ∞ > 0 , every term on the right-hand side of (56) converges. Hence Π ′ ( s ) \Pi'(s) Π ′ ( s ) has a limit L ∈ S n L\in\mathbb{S}^n L ∈ S n . The identity
Π ( s + 1 ) − Π ( s ) = ∫ s s + 1 Π ′ ( r ) d r \Pi(s+1)-\Pi(s)=\int_s^{s+1}\Pi'(r)\,dr Π ( s + 1 ) − Π ( s ) = ∫ s s + 1 Π ′ ( r ) d r
then implies L = 0 L=0 L = 0 , since the left-hand side tends to zero and the right-hand side tends to L L L as s → ∞ s\to\infty s → ∞ . Thus Π ′ ( s ) → 0 \Pi'(s)\to0 Π ′ ( s ) → 0 , and passing to the limit in (56) yields (22) for P ∞ P_\infty P ∞ .
Fix T > 0 T>0 T > 0 and u ∈ L F 2 ( 0 , T ; R m ) u\in L^2_F(0,T;\mathbb{R}^m) u ∈ L F 2 ( 0 , T ; R m ) , and let X X X be the corresponding solution of (12) with x = 0 x=0 x = 0 , namely,
{ d X ( t ) = ( A X ( t ) + B u ( t ) ) d t + ( C X ( t ) + D u ( t ) ) d W ( t ) , 0 ≤ t ≤ T , X ( 0 ) = 0. (57) \begin{cases}
dX(t)=(AX(t)+Bu(t))\,dt+(CX(t)+Du(t))\,dW(t), & 0\le t\le T,\\
X(0)=0. \tag*{(57)}
\end{cases} { d X ( t ) = ( A X ( t ) + B u ( t )) d t + ( C X ( t ) + D u ( t )) d W ( t ) , X ( 0 ) = 0. 0 ≤ t ≤ T , (57)
Set
v ( t ) = u ( t ) + K T ( t ) X ( t ) , 0 ≤ t ≤ T . (58) v(t)=u(t)+K_T(t)X(t),\qquad0\le t\le T. \tag*{(58)} v ( t ) = u ( t ) + K T ( t ) X ( t ) , 0 ≤ t ≤ T . (58)
The standard finite-horizon estimate gives X ∈ L F 2 ( 0 , T ; R n ) X\in L^2_F(0,T;\mathbb{R}^n) X ∈ L F 2 ( 0 , T ; R n ) . Since K T K_T K T is bounded on [ 0 , T ] [0,T] [ 0 , T ] , we also have v ∈ L F 2 ( 0 , T ; R m ) v\in L^2_F(0,T;\mathbb{R}^m) v ∈ L F 2 ( 0 , T ; R m ) . Applying Itô's formula to X ( t ) T P T ( t ) X ( t ) X(t)^{\mathsf T}P_T(t)X(t) X ( t ) T P T ( t ) X ( t ) , taking expectations, and using (9) , we obtain
d d t E [ X ( t ) T P T ( t ) X ( t ) ] = E [ X ( t ) T ( P ˙ T ( t ) + A T P T ( t ) + P T ( t ) A + C T P T ( t ) C ) X ( t ) + 2 u ( t ) T ( B T P T ( t ) + D T P T ( t ) C ) X ( t ) + u ( t ) T D T P T ( t ) D u ( t ) ] = E [ X ( t ) T ( K T ( t ) T M T ( t ) K T ( t ) − Q ) X ( t ) + 2 u ( t ) T ( M T ( t ) K T ( t ) − S ) X ( t ) + u ( t ) T ( M T ( t ) − R ) u ( t ) ] = E [ ( u ( t ) + K T ( t ) X ( t ) ) T M T ( t ) ( u ( t ) + K T ( t ) X ( t ) ) − X ( t ) T Q X ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] = E [ v ( t ) T M T ( t ) v ( t ) − X ( t ) T Q X ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] . \begin{aligned}
\frac{d}{dt}\mathbb{E}[X(t)^{\mathsf T}P_T(t)X(t)]
={}&\mathbb{E}\bigl[X(t)^{\mathsf T}\bigl(\dot P_T(t)+A^{\mathsf T}P_T(t)+P_T(t)A+C^{\mathsf T}P_T(t)C\bigr)X(t)\\
&\quad+2u(t)^{\mathsf T}\bigl(B^{\mathsf T}P_T(t)+D^{\mathsf T}P_T(t)C\bigr)X(t)
+u(t)^{\mathsf T}D^{\mathsf T}P_T(t)Du(t)\bigr]\\
={}&\mathbb{E}\bigl[X(t)^{\mathsf T}\bigl(K_T(t)^{\mathsf T}M_T(t)K_T(t)-Q\bigr)X(t)\\
&\quad+2u(t)^{\mathsf T}\bigl(M_T(t)K_T(t)-S\bigr)X(t)
+u(t)^{\mathsf T}\bigl(M_T(t)-R\bigr)u(t)\bigr]\\
={}&\mathbb{E}\bigl[(u(t)+K_T(t)X(t))^{\mathsf T}M_T(t)(u(t)+K_T(t)X(t))\\
&\quad-X(t)^{\mathsf T}QX(t)-2u(t)^{\mathsf T}SX(t)-u(t)^{\mathsf T}Ru(t)\bigr]\\
={}&\mathbb{E}\bigl[v(t)^{\mathsf T}M_T(t)v(t)-X(t)^{\mathsf T}QX(t)-2u(t)^{\mathsf T}SX(t)-u(t)^{\mathsf T}Ru(t)\bigr].
\end{aligned} d t d E [ X ( t ) T P T ( t ) X ( t )] = = = = E [ X ( t ) T ( P ˙ T ( t ) + A T P T ( t ) + P T ( t ) A + C T P T ( t ) C ) X ( t ) + 2 u ( t ) T ( B T P T ( t ) + D T P T ( t ) C ) X ( t ) + u ( t ) T D T P T ( t ) D u ( t ) ] E [ X ( t ) T ( K T ( t ) T M T ( t ) K T ( t ) − Q ) X ( t ) + 2 u ( t ) T ( M T ( t ) K T ( t ) − S ) X ( t ) + u ( t ) T ( M T ( t ) − R ) u ( t ) ] E [ ( u ( t ) + K T ( t ) X ( t ) ) T M T ( t ) ( u ( t ) + K T ( t ) X ( t )) − X ( t ) T QX ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] E [ v ( t ) T M T ( t ) v ( t ) − X ( t ) T QX ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] .
Integrating over [ 0 , T ] [0,T] [ 0 , T ] gives
E [ X ( T ) T P T ( T ) X ( T ) ] − X ( 0 ) T P T ( 0 ) X ( 0 ) = E ∫ 0 T [ v ( t ) T M T ( t ) v ( t ) − X ( t ) T Q X ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] d t . \begin{aligned}
\mathbb{E}[X(T)^{\mathsf T}P_T(T)X(T)]-X(0)^{\mathsf T}P_T(0)X(0) \\
&=\mathbb{E}\int_0^T\left[v(t)^{\mathsf T}M_T(t)v(t)-X(t)^{\mathsf T}QX(t)-2u(t)^{\mathsf T}SX(t)-u(t)^{\mathsf T}Ru(t)\right]\mathrm{d}t.
\end{aligned} E [ X ( T ) T P T ( T ) X ( T )] − X ( 0 ) T P T ( 0 ) X ( 0 ) = E ∫ 0 T [ v ( t ) T M T ( t ) v ( t ) − X ( t ) T QX ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] d t .
Since X ( 0 ) = 0 X(0)=0 X ( 0 ) = 0 , P T ( T ) = 0 P_T(T)=0 P T ( T ) = 0 , and M T ( t ) ≥ μ I m M_T(t)\geq\mu I_m M T ( t ) ≥ μ I m , it follows that
J T 0 ( 0 ; u ) = E ∫ 0 T v ( t ) T M T ( t ) v ( t ) d t ≥ μ E ∫ 0 T ∣ v ( t ) ∣ 2 d t . (59) J_T^0(0;u)=\mathbb{E}\int_0^T v(t)^{\mathsf T}M_T(t)v(t)\mathrm{d}t\geq\mu\mathbb{E}\int_0^T|v(t)|^2\mathrm{d}t. \tag*{(59)} J T 0 ( 0 ; u ) = E ∫ 0 T v ( t ) T M T ( t ) v ( t ) d t ≥ μ E ∫ 0 T ∣ v ( t ) ∣ 2 d t . (59)
Let h > 0 h>0 h > 0 be a constant to be specified later, independently of T T T , and set t 0 = max { T − h , 0 } t_0=\max\{T-h,0\} t 0 = max { T − h , 0 } . We first estimate the state on the terminal interval [ t 0 , T ] [t_0,T] [ t 0 , T ] , whose length is at most h h h . Substituting u ( t ) = v ( t ) − K T ( t ) X ( t ) u(t)=v(t)-K_T(t)X(t) u ( t ) = v ( t ) − K T ( t ) X ( t ) into (57) and setting ξ = X ( t 0 ) \xi=X(t_0) ξ = X ( t 0 ) , we see that the same state process X X X satisfies
{ d X ( t ) = [ ( A − B K T ( t ) ) X ( t ) + B v ( t ) ] d t + [ ( C − D K T ( t ) ) X ( t ) + D v ( t ) ] d W ( t ) , t 0 ≤ t ≤ T , X ( t 0 ) = ξ . (60) \begin{cases}
\mathrm{d}X(t)=[(A-BK_T(t))X(t)+Bv(t)]\mathrm{d}t+[(C-DK_T(t))X(t)+Dv(t)]\mathrm{d}W(t), & t_0\leq t\leq T,\\
X(t_0)=\xi. \tag*{(60)}
\end{cases} { d X ( t ) = [( A − B K T ( t )) X ( t ) + B v ( t )] d t + [( C − D K T ( t )) X ( t ) + D v ( t )] d W ( t ) , X ( t 0 ) = ξ . t 0 ≤ t ≤ T , (60)
Applying Itô’s formula to ∣ X ( t ) ∣ 2 |X(t)|^2 ∣ X ( t ) ∣ 2 along the solution of (60) and taking expectations, we obtain, for almost every t ∈ [ t 0 , T ] t\in[t_0,T] t ∈ [ t 0 , T ] ,
d d t E ∣ X ( t ) ∣ 2 = E [ X ( t ) T ( ( A − B K T ( t ) ) T + ( A − B K T ( t ) ) + ( C − D K T ( t ) ) T ( C − D K T ( t ) ) ) X ( t ) + v ( t ) T D T D v ( t ) + 2 v ( t ) T ( B T + D T ( C − D K T ( t ) ) ) X ( t ) ] ≤ a E ∣ X ( t ) ∣ 2 + b E ∣ v ( t ) ∣ 2 , \begin{aligned}
\frac{\mathrm{d}}{\mathrm{d}t}\mathbb{E}|X(t)|^2
={}&\mathbb{E}\Big[X(t)^{\mathsf T}\big((A-BK_T(t))^{\mathsf T}+(A-BK_T(t))\\
&\quad+(C-DK_T(t))^{\mathsf T}(C-DK_T(t))\big)X(t)\\
&\quad+v(t)^{\mathsf T}D^{\mathsf T}Dv(t)+2v(t)^{\mathsf T}\big(B^{\mathsf T}+D^{\mathsf T}(C-DK_T(t))\big)X(t)\Big]\\
\leq{}&a\mathbb{E}|X(t)|^2+b\mathbb{E}|v(t)|^2,
\end{aligned} d t d E ∣ X ( t ) ∣ 2 = ≤ E [ X ( t ) T ( ( A − B K T ( t ) ) T + ( A − B K T ( t )) + ( C − D K T ( t ) ) T ( C − D K T ( t )) ) X ( t ) + v ( t ) T D T D v ( t ) + 2 v ( t ) T ( B T + D T ( C − D K T ( t )) ) X ( t ) ] a E ∣ X ( t ) ∣ 2 + b E ∣ v ( t ) ∣ 2 ,
where the last inequality follows from Young’s inequality and the uniform boundedness of K T ( t ) = K ( T − t ) K_T(t)=K(T-t) K T ( t ) = K ( T − t ) , and the constants a > 0 a>0 a > 0 and b ≥ 0 b\geq0 b ≥ 0 are independent of T T T , u u u , and h h h . Gronwall’s inequality gives, for t 0 ≤ t ≤ T t_0\leq t\leq T t 0 ≤ t ≤ T ,
E ∣ X ( t ) ∣ 2 ≤ e a ( t − t 0 ) E ∣ X ( t 0 ) ∣ 2 + b E ∫ t 0 t e a ( t − s ) ∣ v ( s ) ∣ 2 d s ≤ e a h [ E ∣ X ( t 0 ) ∣ 2 + b E ∫ t 0 T ∣ v ( s ) ∣ 2 d s ] . \begin{aligned}
\mathbb{E}|X(t)|^2
&\leq e^{a(t-t_0)}\mathbb{E}|X(t_0)|^2+b\mathbb{E}\int_{t_0}^t e^{a(t-s)}|v(s)|^2\mathrm{d}s\\
&\leq e^{ah}\left[\mathbb{E}|X(t_0)|^2+b\mathbb{E}\int_{t_0}^T|v(s)|^2\mathrm{d}s\right].
\end{aligned} E ∣ X ( t ) ∣ 2 ≤ e a ( t − t 0 ) E ∣ X ( t 0 ) ∣ 2 + b E ∫ t 0 t e a ( t − s ) ∣ v ( s ) ∣ 2 d s ≤ e ah [ E ∣ X ( t 0 ) ∣ 2 + b E ∫ t 0 T ∣ v ( s ) ∣ 2 d s ] .
Integrating over [ t 0 , T ] [t_0,T] [ t 0 , T ] and using T − t 0 ≤ h T-t_0\leq h T − t 0 ≤ h , we obtain
E ∫ t 0 T ∣ X ( t ) ∣ 2 d t ≤ h e a h [ E ∣ X ( t 0 ) ∣ 2 + b E ∫ t 0 T ∣ v ( t ) ∣ 2 d t ] . (61) \mathbb{E}\int_{t_0}^T|X(t)|^2\mathrm{d}t\leq he^{ah}\left[\mathbb{E}|X(t_0)|^2+b\mathbb{E}\int_{t_0}^T|v(t)|^2\mathrm{d}t\right]. \tag*{(61)} E ∫ t 0 T ∣ X ( t ) ∣ 2 d t ≤ h e ah [ E ∣ X ( t 0 ) ∣ 2 + b E ∫ t 0 T ∣ v ( t ) ∣ 2 d t ] . (61)
We next estimate the state on [ 0 , t 0 ] [0,t_0] [ 0 , t 0 ] . Set
A ∞ = A − B K ∞ , C ∞ = C − D K ∞ , w ( t ) = u ( t ) + K ∞ X ( t ) . A_\infty=A-BK_\infty,\qquad C_\infty=C-DK_\infty,\qquad w(t)=u(t)+K_\infty X(t). A ∞ = A − B K ∞ , C ∞ = C − D K ∞ , w ( t ) = u ( t ) + K ∞ X ( t ) .
Then the same state process X X X as in (57) satisfies
{ d X ( t ) = ( A ∞ X ( t ) + B w ( t ) ) d t + ( C ∞ X ( t ) + D w ( t ) ) d W ( t ) , 0 ≤ t ≤ T , X ( 0 ) = 0. (62) \begin{cases}
\mathrm{d}X(t)=(A_\infty X(t)+Bw(t))\mathrm{d}t+(C_\infty X(t)+Dw(t))\mathrm{d}W(t), & 0\leq t\leq T,\\
X(0)=0. \tag*{(62)}
\end{cases} { d X ( t ) = ( A ∞ X ( t ) + B w ( t )) d t + ( C ∞ X ( t ) + D w ( t )) d W ( t ) , X ( 0 ) = 0. 0 ≤ t ≤ T , (62)
Since [ A ∞ , C ∞ ] [A_\infty,C_\infty] [ A ∞ , C ∞ ] is L 2 L^2 L 2 -stable and w ∈ L 2 ( 0 , T ; R m ) w\in L^2(0,T;\mathbb{R}^m) w ∈ L 2 ( 0 , T ; R m ) , Lemma 2.7, applied to (62) , gives a constant C > 0 C>0 C > 0 , independent of T T T , u u u , and h h h , such that
E ∫ 0 s ∣ X ( t ) ∣ 2 d t + E ∣ X ( s ) ∣ 2 ≤ C E ∫ 0 s ∣ w ( t ) ∣ 2 d t , 0 ≤ s ≤ T . (63) \mathbb{E}\int_0^s|X(t)|^2\mathrm{d}t+\mathbb{E}|X(s)|^2\leq C\mathbb{E}\int_0^s|w(t)|^2\mathrm{d}t,\qquad0\leq s\leq T. \tag*{(63)} E ∫ 0 s ∣ X ( t ) ∣ 2 d t + E ∣ X ( s ) ∣ 2 ≤ C E ∫ 0 s ∣ w ( t ) ∣ 2 d t , 0 ≤ s ≤ T . (63)
For h ≥ 0 h \ge0 h ≥ 0 , define
ε ( h ) = sup s ≥ h ∥ K ( s ) − K ∞ ∥ . \varepsilon(h)=\sup_{s\ge h}\lVert K(s)-K_{\infty}\rVert. ε ( h ) = s ≥ h sup ∥ K ( s ) − K ∞ ∥ .
Since K ( s ) → K ∞ K(s)\to K_{\infty} K ( s ) → K ∞ as s → ∞ s\to\infty s → ∞ , we have ε ( h ) → 0 \varepsilon(h)\to0 ε ( h ) → 0 as h → ∞ h\to\infty h → ∞ . Moreover, K T ( t ) = K ( T − t ) K_T(t)=K(T-t) K T ( t ) = K ( T − t ) implies that
∥ K T ( t ) − K ∞ ∥ ≤ ε ( h ) , 0 ≤ t ≤ T − h , T > h . \lVert K_T(t)-K_{\infty}\rVert\le\varepsilon(h),\qquad0\le t\le T-h,\quad T>h. ∥ K T ( t ) − K ∞ ∥ ≤ ε ( h ) , 0 ≤ t ≤ T − h , T > h .
Choose h > 0 h>0 h > 0 , independently of T T T and u u u , sufficiently large that 2 C ε ( h ) 2 ≤ 1 / 2 2C\varepsilon(h)^2\le1/2 2 C ε ( h ) 2 ≤ 1/2 . If T > h T>h T > h , then t 0 = T − h t_0=T-h t 0 = T − h . By (58) , we have
w ( t ) = v ( t ) + ( K ∞ − K T ( t ) ) X ( t ) , 0 ≤ t ≤ t 0 , w(t)=v(t)+(K_{\infty}-K_T(t))X(t),\qquad0\le t\le t_0, w ( t ) = v ( t ) + ( K ∞ − K T ( t )) X ( t ) , 0 ≤ t ≤ t 0 ,
and hence
∣ w ( t ) ∣ 2 ≤ 2 ∣ v ( t ) ∣ 2 + 2 ε ( h ) 2 ∣ X ( t ) ∣ 2 , 0 ≤ t ≤ t 0 . |w(t)|^2\le2|v(t)|^2+2\varepsilon(h)^2|X(t)|^2,\qquad0\le t\le t_0. ∣ w ( t ) ∣ 2 ≤ 2∣ v ( t ) ∣ 2 + 2 ε ( h ) 2 ∣ X ( t ) ∣ 2 , 0 ≤ t ≤ t 0 .
Taking s = t 0 s=t_0 s = t 0 in (63) , we obtain
E ∫ 0 t 0 ∣ X ( t ) ∣ 2 d t + E ∣ X ( t 0 ) ∣ 2 ≤ 2 C E ∫ 0 t 0 ∣ v ( t ) ∣ 2 d t + 2 C ε ( h ) 2 E ∫ 0 t 0 ∣ X ( t ) ∣ 2 d t \mathbb{E}\int_0^{t_0}|X(t)|^2\,dt+\mathbb{E}|X(t_0)|^2
\le2C\mathbb{E}\int_0^{t_0}|v(t)|^2\,dt+2C\varepsilon(h)^2\mathbb{E}\int_0^{t_0}|X(t)|^2\,dt E ∫ 0 t 0 ∣ X ( t ) ∣ 2 d t + E ∣ X ( t 0 ) ∣ 2 ≤ 2 C E ∫ 0 t 0 ∣ v ( t ) ∣ 2 d t + 2 C ε ( h ) 2 E ∫ 0 t 0 ∣ X ( t ) ∣ 2 d t
≤ 2 C E ∫ 0 t 0 ∣ v ( t ) ∣ 2 d t + 1 2 E ∫ 0 t 0 ∣ X ( t ) ∣ 2 d t . \le2C\mathbb{E}\int_0^{t_0}|v(t)|^2\,dt+\frac{1}{2}\mathbb{E}\int_0^{t_0}|X(t)|^2\,dt. ≤ 2 C E ∫ 0 t 0 ∣ v ( t ) ∣ 2 d t + 2 1 E ∫ 0 t 0 ∣ X ( t ) ∣ 2 d t .
Absorbing the last term into the left-hand side yields
E ∫ 0 t 0 ∣ X ( t ) ∣ 2 d t ≤ 4 C E ∫ 0 t 0 ∣ v ( t ) ∣ 2 d t , E ∣ X ( t 0 ) ∣ 2 ≤ 2 C E ∫ 0 t 0 ∣ v ( t ) ∣ 2 d t . (64) \mathbb{E}\int_0^{t_0}|X(t)|^2\,dt\le4C\mathbb{E}\int_0^{t_0}|v(t)|^2\,dt,\qquad
\mathbb{E}|X(t_0)|^2\le2C\mathbb{E}\int_0^{t_0}|v(t)|^2\,dt. \tag*{(64)} E ∫ 0 t 0 ∣ X ( t ) ∣ 2 d t ≤ 4 C E ∫ 0 t 0 ∣ v ( t ) ∣ 2 d t , E ∣ X ( t 0 ) ∣ 2 ≤ 2 C E ∫ 0 t 0 ∣ v ( t ) ∣ 2 d t . (64)
If T ≤ h T\le h T ≤ h , then t 0 = 0 t_0=0 t 0 = 0 , and (64) remains valid because X ( 0 ) = 0 X(0)=0 X ( 0 ) = 0 .
Combining (61) with (64) , we obtain
E ∫ 0 T ∣ X ( t ) ∣ 2 d t ≤ C E ∫ 0 T ∣ v ( t ) ∣ 2 d t . (65) \mathbb{E}\int_0^T|X(t)|^2\,dt\le C\mathbb{E}\int_0^T|v(t)|^2\,dt. \tag*{(65)} E ∫ 0 T ∣ X ( t ) ∣ 2 d t ≤ C E ∫ 0 T ∣ v ( t ) ∣ 2 d t . (65)
where C > 0 C>0 C > 0 is independent of T T T and u u u and is chosen large enough that ∥ K T ( t ) ∥ ≤ C \lVert K_T(t)\rVert\le C ∥ K T ( t )∥ ≤ C for all T > 0 T>0 T > 0 and 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T . Since u ( t ) = v ( t ) − K T ( t ) X ( t ) u(t)=v(t)-K_T(t)X(t) u ( t ) = v ( t ) − K T ( t ) X ( t ) by (58) , we have
∣ u ( t ) ∣ 2 ≤ 2 ∣ v ( t ) ∣ 2 + 2 C 2 ∣ X ( t ) ∣ 2 , 0 ≤ t ≤ T . |u(t)|^2\le2|v(t)|^2+2C^2|X(t)|^2,\qquad0\le t\le T. ∣ u ( t ) ∣ 2 ≤ 2∣ v ( t ) ∣ 2 + 2 C 2 ∣ X ( t ) ∣ 2 , 0 ≤ t ≤ T .
Combining (65) and (59) , we obtain
E ∫ 0 T ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t ≤ [ 2 + ( 1 + 2 C 2 ) C ] E ∫ 0 T ∣ v ( t ) ∣ 2 d t ≤ 2 + ( 1 + 2 C 2 ) C μ J T 0 ( 0 ; u ) . \mathbb{E}\int_0^T\left(|X(t)|^2+|u(t)|^2\right)\,dt
\le[2+(1+2C^2)C]\mathbb{E}\int_0^T|v(t)|^2\,dt
\le\frac{2+(1+2C^2)C}{\mu}J_T^0(0;u). E ∫ 0 T ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t ≤ [ 2 + ( 1 + 2 C 2 ) C ] E ∫ 0 T ∣ v ( t ) ∣ 2 d t ≤ μ 2 + ( 1 + 2 C 2 ) C J T 0 ( 0 ; u ) .
Thus, taking δ = μ / [ 2 + ( 1 + 2 C 2 ) C ] > 0 \delta=\mu/[2+(1+2C^2)C]>0 δ = μ / [ 2 + ( 1 + 2 C 2 ) C ] > 0 , we obtain (H2). ■
Remark 4.4. Assumption (H2) implies (21) and, for every T > 0 T>0 T > 0 , a finite-horizon lower bound with a constant δ T > 0 \delta_T>0 δ T > 0 :
J T 0 ( 0 ; u ) ≥ δ T E ∫ 0 T ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , T > 0 , u ∈ L F 2 ( 0 , T ; R m ) . (66) J_T^0(0;u)\ge\delta_T\mathbb{E}\int_0^T\left(|X(t)|^2+|u(t)|^2\right)\,dt,\qquad T>0,\quad u\in L_{\mathbb{F}}^2(0,T;\mathbb{R}^m). \tag*{(66)} J T 0 ( 0 ; u ) ≥ δ T E ∫ 0 T ( ∣ X ( t ) ∣ 2 + ∣ u ( t ) ∣ 2 ) d t , T > 0 , u ∈ L F 2 ( 0 , T ; R m ) . (66)
Under (21) and (66) , the Riccati equations associated with both the finite-horizon and infinite-horizon problems are solvable. However, (21) and (66) do not imply that (H2) holds. Consequently, the corresponding Riccati equations need not converge to the stabilizing solution. By Theorem 4.3 , Example 2.12 is readily seen to provide a counterexample.
Remark 4.5. By Remark 2.10 , (66) and (17) are equivalent on each fixed horizon. As shown in Sun–Li–Yong [33 ] , (17) is almost necessary for the solvability of the Riccati equation (9) . We establish the convergence result in Proposition 4.2 under the J C JC J C condition (H2), which is slightly stronger than the uniform convexity condition (17) . Furthermore, by Theorem 4.3 , the J C JC J C condition (H2) is necessary for the convergence of Riccati equations. Thus, the J C JC J C condition (H2) is sharp.
For the remainder of this section, suppose that Assumptions (H1) and (H2) hold. Recall the finite-horizon coefficients M T M_T M T , N T N_T N T , K T K_T K T from Proposition 2.2 and the limiting coefficients M ∞ M_\infty M ∞ , N ∞ N_\infty N ∞ , K ∞ K_\infty K ∞ from (23) . For every T > 0 T>0 T > 0 , Proposition 2.15 gives M T ( t ) ≥ δ I m M_T(t) \geq\delta I_m M T ( t ) ≥ δ I m for all 0 ≤ t ≤ T 0 \leq t \leq T 0 ≤ t ≤ T . Taking t = 0 t=0 t = 0 , letting T → ∞ T \to\infty T → ∞ , and using (52) in Proposition 4.2 , we obtain M ∞ ≥ δ I m M_\infty\geq\delta I_m M ∞ ≥ δ I m . The following corollary follows from standard matrix-norm estimates.
Corollary 4.6. There exist constants C , γ > 0 C,\gamma>0 C , γ > 0 , independent of T T T , such that
∥ M T ( t ) − M ∞ ∥ + ∥ N T ( t ) − N ∞ ∥ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T , ∥ M T ( t ) − 1 − M ∞ − 1 ∥ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T . (67) \begin{aligned}
\lVert M_T(t)-M_\infty\rVert+\lVert N_T(t)-N_\infty\rVert&\leq Ce^{-\gamma(T-t)}, && 0\leq t\leq T,\\
\lVert M_T(t)^{-1}-M_\infty^{-1}\rVert&\leq Ce^{-\gamma(T-t)}, && 0\leq t\leq T. \tag*{(67)}
\end{aligned} ∥ M T ( t ) − M ∞ ∥ + ∥ N T ( t ) − N ∞ ∥ ∥ M T ( t ) − 1 − M ∞ − 1 ∥ ≤ C e − γ ( T − t ) , ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T , 0 ≤ t ≤ T . (67)
Corollary 4.7. There exist constants C , γ > 0 C,\gamma>0 C , γ > 0 , independent of T T T , such that
∥ K T ( t ) − K ∞ ∥ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T . \lVert K_T(t)-K_\infty\rVert\leq Ce^{-\gamma(T-t)},\qquad0\leq t\leq T. ∥ K T ( t ) − K ∞ ∥ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T .
Recall that φ T \varphi_T φ T solves the terminal-value problem (10) . Since P T ( t ) P_T(t) P T ( t ) and K T ( t ) K_T(t) K T ( t ) will converge to P ∞ P_\infty P ∞ and K ∞ K_\infty K ∞ as T − t → ∞ T-t\to\infty T − t → ∞ , we consider the following algebraic equation:
( A − B K ∞ ) ⊤ φ ∞ + ( C − D K ∞ ) ⊤ P ∞ σ − K ∞ ⊤ r + P ∞ b + q = 0. (68) (A-BK_\infty)^\top\varphi_\infty+(C-DK_\infty)^\top P_\infty\sigma-K_\infty^\top r+P_\infty b+q=0. \tag*{(68)} ( A − B K ∞ ) ⊤ φ ∞ + ( C − D K ∞ ) ⊤ P ∞ σ − K ∞ ⊤ r + P ∞ b + q = 0. (68)
By Corollary 2.19, the closed-loop system [ A − B K ∞ , C − D K ∞ ] [A-BK_\infty,C-DK_\infty] [ A − B K ∞ , C − D K ∞ ] is L 2 L^2 L 2 -stable. Corollary 2.6 therefore implies that A − B K ∞ A-BK_\infty A − B K ∞ is Hurwitz and hence invertible. Thus (68) has a unique solution, which we denote by φ ∞ \varphi_\infty φ ∞ .
Lemma 4.8. Let φ T \varphi_T φ T and φ ∞ \varphi_\infty φ ∞ be the unique solutions of (10) and (68) , respectively. There exist constants C , γ > 0 C,\gamma>0 C , γ > 0 , independent of T T T , such that
∣ φ T ( t ) − φ ∞ ∣ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T . \lvert\varphi_T(t)-\varphi_\infty\rvert\leq Ce^{-\gamma(T-t)},\qquad0\leq t\leq T. ∣ φ T ( t ) − φ ∞ ∣ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T .
Proof. The proof is the same as that in Sun–Wang–Yong [34 Lemma 5.1] . ■
Define
d T ( t ) = B ⊤ φ T ( t ) + D ⊤ P T ( t ) σ + r , d ∞ = B ⊤ φ ∞ + D ⊤ P ∞ σ + r , ℓ T ( t ) = M T ( t ) − 1 d T ( t ) , ℓ ∞ = M ∞ − 1 d ∞ , u ˉ T ( t , x ) = − K T ( t ) x − ℓ T ( t ) , u ˉ ∞ ( x ) = − K ∞ x − ℓ ∞ . \begin{aligned}
d_T(t)&=B^\top\varphi_T(t)+D^\top P_T(t)\sigma+r, & d_\infty&=B^\top\varphi_\infty+D^\top P_\infty\sigma+r,\\
\ell_T(t)&=M_T(t)^{-1}d_T(t), & \ell_\infty&=M_\infty^{-1}d_\infty,\\
\bar{u}_T(t,x)&=-K_T(t)x-\ell_T(t), & \bar{u}_\infty(x)&=-K_\infty x-\ell_\infty.
\end{aligned} d T ( t ) ℓ T ( t ) u ˉ T ( t , x ) = B ⊤ φ T ( t ) + D ⊤ P T ( t ) σ + r , = M T ( t ) − 1 d T ( t ) , = − K T ( t ) x − ℓ T ( t ) , d ∞ ℓ ∞ u ˉ ∞ ( x ) = B ⊤ φ ∞ + D ⊤ P ∞ σ + r , = M ∞ − 1 d ∞ , = − K ∞ x − ℓ ∞ .
With the above estimates, we can obtain the following corollary.
Corollary 4.9. There exist constants C , γ > 0 C,\gamma>0 C , γ > 0 , independent of T T T , such that
∥ K T ( t ) − K ∞ ∥ + ∣ ℓ T ( t ) − ℓ ∞ ∣ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T . \lVert K_T(t)-K_\infty\rVert+\lvert\ell_T(t)-\ell_\infty\rvert\leq Ce^{-\gamma(T-t)},\qquad0\leq t\leq T. ∥ K T ( t ) − K ∞ ∥ + ∣ ℓ T ( t ) − ℓ ∞ ∣ ≤ C e − γ ( T − t ) , 0 ≤ t ≤ T . Thus, for any x ∈ R n x\in\mathbb{R}^n x ∈ R n ,
∣ u ˉ T ( t , x ) − u ˉ ∞ ( x ) ∣ ≤ C ( 1 + ∣ x ∣ ) e − γ ( T − t ) . \lvert\bar{u}_T(t,x)-\bar{u}_\infty(x)\rvert\leq C(1+\lvert x\rvert)e^{-\gamma(T-t)}. ∣ u ˉ T ( t , x ) − u ˉ ∞ ( x )∣ ≤ C ( 1 + ∣ x ∣) e − γ ( T − t ) .
Recall the exploration covariances from Proposition 2.2 and (39) .
Lemma 4.10. There exist constants C , γ > 0 C,\gamma>0 C , γ > 0 , independent of T T T and λ \lambda λ , such that
∥ ( Λ T λ ( t ) − Λ ∞ λ ) ∥ ≤ C λ e − γ ( T − t ) , 0 ≤ t ≤ T , ∥ ( Λ T λ ( t ) ) 1 / 2 − ( Λ ∞ λ ) 1 / 2 ∥ ≤ C λ e − γ ( T − t ) , 0 ≤ t ≤ T . (69) \begin{aligned}
\lVert(\Lambda_T^\lambda(t)-\Lambda_\infty^\lambda)\rVert&\leq C\lambda e^{-\gamma(T-t)}, && 0\leq t\leq T,\\
\lVert(\Lambda_T^\lambda(t))^{1/2}-(\Lambda_\infty^\lambda)^{1/2}\rVert&\leq C\sqrt{\lambda}e^{-\gamma(T-t)}, && 0\leq t\leq T. \tag*{(69)}
\end{aligned} ∥( Λ T λ ( t ) − Λ ∞ λ )∥ ∥( Λ T λ ( t ) ) 1/2 − ( Λ ∞ λ ) 1/2 ∥ ≤ C λ e − γ ( T − t ) , ≤ C λ e − γ ( T − t ) , 0 ≤ t ≤ T , 0 ≤ t ≤ T . (69)
Proof. Using the covariance formulas in Proposition 2.2 and (39) , together with the bounds M T ( t ) , M ∞ ≥ δ I m M_T(t),M_\infty\ge\delta I_m M T ( t ) , M ∞ ≥ δ I m and (67) , we have
∥ Λ T λ ( t ) − Λ ∞ λ ∥ = λ 2 ∥ M T ( t ) − 1 − M ∞ − 1 ∥ = λ 2 ∥ M T ( t ) − 1 ( M ∞ − M T ( t ) ) M ∞ − 1 ∥ ≤ λ 2 δ 2 ∥ M T ( t ) − M ∞ ∥ ≤ C λ e − γ ( T − t ) . (70) \left\|\Lambda_T^\lambda(t)-\Lambda_\infty^\lambda\right\|=\frac{\lambda}{2}\left\|M_T(t)^{-1}-M_\infty^{-1}\right\|=\frac{\lambda}{2}\left\|M_T(t)^{-1}(M_\infty-M_T(t))M_\infty^{-1}\right\|
\le\frac{\lambda}{2\delta^2}\left\|M_T(t)-M_\infty\right\|\le C\lambda e^{-\gamma(T-t)}. \tag*{(70)} Λ T λ ( t ) − Λ ∞ λ = 2 λ M T ( t ) − 1 − M ∞ − 1 = 2 λ M T ( t ) − 1 ( M ∞ − M T ( t )) M ∞ − 1 ≤ 2 δ 2 λ ∥ M T ( t ) − M ∞ ∥ ≤ C λ e − γ ( T − t ) . (70)
This proves (69) . By the uniform boundedness of P T ( t ) P_T(t) P T ( t ) in Proposition 4.2 and the definitions of M T ( t ) M_T(t) M T ( t ) and M ∞ M_\infty M ∞ , there exists a constant β > 0 \beta>0 β > 0 , independent of T , t , λ T,t,\lambda T , t , λ , such that
δ I m ≤ M T ( t ) ≤ β I m , δ I m ≤ M ∞ ≤ β I m . \delta I_m\le M_T(t)\le\beta I_m,\qquad\delta I_m\le M_\infty\le\beta I_m. δ I m ≤ M T ( t ) ≤ β I m , δ I m ≤ M ∞ ≤ β I m .
Hence
M T ( t ) − 1 ≥ β − 1 I m , M ∞ − 1 ≥ β − 1 I m , Λ T λ ( t ) ≥ λ 2 β I m , Λ ∞ λ ≥ λ 2 β I m . (71) M_T(t)^{-1}\ge\beta^{-1}I_m,\qquad M_\infty^{-1}\ge\beta^{-1}I_m,\qquad\Lambda_T^\lambda(t)\ge\frac{\lambda}{2\beta}I_m,\qquad\Lambda_\infty^\lambda\ge\frac{\lambda}{2\beta}I_m. \tag*{(71)} M T ( t ) − 1 ≥ β − 1 I m , M ∞ − 1 ≥ β − 1 I m , Λ T λ ( t ) ≥ 2 β λ I m , Λ ∞ λ ≥ 2 β λ I m . (71)
Note from Higham [16 Theorem 6.2] that
∥ Λ 1 / 2 − Λ ^ 1 / 2 ∥ ≤ ∥ Λ − Λ ^ ∥ λ min ( Λ ) + λ min ( Λ ^ ) , \left\|\Lambda^{1/2}-\widehat{\Lambda}^{1/2}\right\|\le\frac{\left\|\Lambda-\widehat{\Lambda}\right\|}{\sqrt{\lambda_{\min}(\Lambda)}+\sqrt{\lambda_{\min}(\widehat{\Lambda})}}, Λ 1/2 − Λ 1/2 ≤ λ m i n ( Λ ) + λ m i n ( Λ ) Λ − Λ ,
for any Λ , Λ ^ ∈ S + + m \Lambda,\widehat{\Lambda}\in\mathbb{S}_{++}^m Λ , Λ ∈ S ++ m , where λ min ( ⋅ ) \lambda_{\min}(\cdot) λ m i n ( ⋅ ) denotes the smallest eigenvalue of the corresponding matrix. Combining this inequality with (70) and (71) gives (4.31), with constants independent of T , t , λ T,t,\lambda T , t , λ .
Combining the preceding results, we conclude the following exponential convergence estimate between the finite-horizon and infinite-horizon parameters.
Proposition 4.11. Let (H1)–(H2) hold. Then there exist constants C , γ > 0 C,\gamma>0 C , γ > 0 , independent of T T T , such that for all 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T ,
∣ P T ( t ) − P ∞ ∣ + ∣ φ T ( t ) − φ ∞ ∣ + ∥ K T ( t ) − K ∞ ∥ + ∣ ℓ T ( t ) − ℓ ∞ ∣ + λ − 1 ∥ Λ T λ ( t ) − Λ ∞ λ ∥ + λ − 1 / 2 ∥ ( Λ T λ ( t ) ) 1 / 2 − ( Λ ∞ λ ) 1 / 2 ∥ ≤ C e − γ ( T − t ) . (72) \begin{aligned}
&\lvert P_T(t)-P_\infty\rvert+\lvert\varphi_T(t)-\varphi_\infty\rvert+\lVert K_T(t)-K_\infty\rVert+\lvert\ell_T(t)-\ell_\infty\rvert+\lambda^{-1}\left\|\Lambda_T^\lambda(t)-\Lambda_\infty^\lambda\right\|\\
&\qquad+\lambda^{-1/2}\left\|(\Lambda_T^\lambda(t))^{1/2}-(\Lambda_\infty^\lambda)^{1/2}\right\|\le Ce^{-\gamma(T-t)}. \tag*{(72)}
\end{aligned} ∣ P T ( t ) − P ∞ ∣ + ∣ φ T ( t ) − φ ∞ ∣ + ∥ K T ( t ) − K ∞ ∥ + ∣ ℓ T ( t ) − ℓ ∞ ∣ + λ − 1 Λ T λ ( t ) − Λ ∞ λ + λ − 1/2 ( Λ T λ ( t ) ) 1/2 − ( Λ ∞ λ ) 1/2 ≤ C e − γ ( T − t ) . (72) Therefore, for any x ∈ R n x\in\mathbb{R}^n x ∈ R n ,
∣ u ˉ T ( t , x ) − u ˉ ∞ ( x ) ∣ ≤ C ( 1 + ∣ x ∣ ) e − γ ( T − t ) . \left|\bar u_T(t,x)-\bar u_\infty(x)\right|\le C(1+|x|)e^{-\gamma(T-t)}. ∣ u ˉ T ( t , x ) − u ˉ ∞ ( x ) ∣ ≤ C ( 1 + ∣ x ∣ ) e − γ ( T − t ) .
Turnpike Properties
Using the Gaussian feedback and parameter convergence established above, we construct the stochastic reference process X ∗ , λ X^{*,\lambda} X ∗ , λ from the infinite-horizon closed-loop system starting at x ∗ x_* x ∗ . We first prove the state-process turnpike estimate and then derive the Wasserstein estimate for the associated control distributions. The reference object is a process because diffusion and exploration noise generally persist even when the state starts at the static center.
Using the long-time feedback parameters from § 4 , define the infinite-horizon Gaussian feedback distribution
π ∞ λ , ∗ ( ⋅ ∣ x ) = N ( − K ∞ x − ℓ ∞ , Λ ∞ λ ) . (73) \pi_\infty^{\lambda,*}(\cdot\mid x)=\mathcal{N}\left(-K_\infty x-\ell_\infty,\Lambda_\infty^\lambda\right). \tag*{(73)} π ∞ λ , ∗ ( ⋅ ∣ x ) = N ( − K ∞ x − ℓ ∞ , Λ ∞ λ ) . (73)
Before constructing the reference process, we first show that this long-time feedback is compatible, at the static center, with the static distribution optimizer from § 3 .
Proposition 5.1. Let (H1)–(H2) hold. Let ( x ∗ , u ∗ ) (x_*,u_*) ( x ∗ , u ∗ ) be the unique optimizer of the static problem (27) , and let ζ ∗ \zeta^* ζ ∗ be a Lagrange multiplier satisfying the KKT system (28) . Then
ζ ∗ = P ∞ x ∗ + φ ∞ , (74) \zeta^*=P_\infty x_*+\varphi_\infty, \tag*{(74)} ζ ∗ = P ∞ x ∗ + φ ∞ , (74)
u ∗ = − K ∞ x ∗ − ℓ ∞ , (75) u_*=-K_\infty x_*-\ell_\infty, \tag*{(75)} u ∗ = − K ∞ x ∗ − ℓ ∞ , (75)
and the static balance equation
( A − B K ∞ ) x + b − B ℓ ∞ = 0 (A-BK_\infty)x+b-B\ell_\infty=0 ( A − B K ∞ ) x + b − B ℓ ∞ = 0
admits the unique solution x = x ∗ x=x_* x = x ∗ . Moreover,
π ∞ λ , ∗ ( ⋅ ∣ x ∗ ) = N ( u ∗ , Λ ∗ λ ) . (76) \pi_\infty^{\lambda,*}(\cdot\mid x_*)=\mathcal{N}(u_*,\Lambda_*^\lambda). \tag*{(76)} π ∞ λ , ∗ ( ⋅ ∣ x ∗ ) = N ( u ∗ , Λ ∗ λ ) . (76)
Proof. We first prove that (74) holds. Let η ∗ : = ζ ∗ − P ∞ x ∗ − φ ∞ \eta^*:=\zeta^*-P_\infty x_*-\varphi_\infty η ∗ := ζ ∗ − P ∞ x ∗ − φ ∞ . The u u u -equation in the KKT system (28) gives
0 = S x ∗ + R u ∗ + r + D ⊤ P ∞ ( C x ∗ + D u ∗ + σ ) + B ⊤ ζ ∗ = N ∞ x ∗ + M ∞ u ∗ + d ∞ + B ⊤ η ∗ . (77) \begin{aligned}
0&=Sx_*+Ru_*+r+D^\top P_\infty(Cx_*+Du_*+\sigma)+B^\top\zeta^*\\
&=N_\infty x_*+M_\infty u_*+d_\infty+B^\top\eta^*. \tag*{(77)}
\end{aligned} 0 = S x ∗ + R u ∗ + r + D ⊤ P ∞ ( C x ∗ + D u ∗ + σ ) + B ⊤ ζ ∗ = N ∞ x ∗ + M ∞ u ∗ + d ∞ + B ⊤ η ∗ . (77)
Since M ∞ > 0 M_\infty>0 M ∞ > 0 , (77) is equivalent to
u ∗ + M ∞ − 1 ( N ∞ x ∗ + d ∞ ) = − M ∞ − 1 B ⊤ η ∗ . (78) u_*+M_\infty^{-1}(N_\infty x_*+d_\infty)=-M_\infty^{-1}B^\top\eta^*. \tag*{(78)} u ∗ + M ∞ − 1 ( N ∞ x ∗ + d ∞ ) = − M ∞ − 1 B ⊤ η ∗ . (78)
Moreover, the stationary linear equation (68) can be written as
0 = ( A − B K ∞ ) ⊤ φ ∞ + ( C − D K ∞ ) ⊤ P ∞ σ − K ∞ ⊤ r + P ∞ b + q = A ⊤ φ ∞ + P ∞ b + C ⊤ P ∞ σ + q − N ∞ ⊤ M ∞ − 1 d ∞ . (79) \begin{aligned}
0&=(A-BK_\infty)^\top\varphi_\infty+(C-DK_\infty)^\top P_\infty\sigma-K_\infty^\top r+P_\infty b+q\\
&=A^\top\varphi_\infty+P_\infty b+C^\top P_\infty\sigma+q-N_\infty^\top M_\infty^{-1}d_\infty. \tag*{(79)}
\end{aligned} 0 = ( A − B K ∞ ) ⊤ φ ∞ + ( C − D K ∞ ) ⊤ P ∞ σ − K ∞ ⊤ r + P ∞ b + q = A ⊤ φ ∞ + P ∞ b + C ⊤ P ∞ σ + q − N ∞ ⊤ M ∞ − 1 d ∞ . (79)
Using the x x x -equation in (28) , the algebraic Riccati equation (22) , the static drift balance, and (78) –(79) , we obtain
0 = Q x ∗ + S ⊤ u ∗ + q + C ⊤ P ∞ ( C x ∗ + D u ∗ + σ ) + A ⊤ ζ ∗ = − P ∞ ( A x ∗ + B u ∗ ) + N ∞ ⊤ ( M ∞ − 1 N ∞ x ∗ + u ∗ ) + q + C ⊤ P ∞ σ + A ⊤ φ ∞ + A ⊤ η ∗ = N ∞ ⊤ [ u ∗ + M ∞ − 1 ( N ∞ x ∗ + d ∞ ) ] + A ⊤ η ∗ = − N ∞ ⊤ M ∞ − 1 B ⊤ η ∗ + A ⊤ η ∗ = ( A − B K ∞ ) ⊤ η ∗ . \begin{aligned}
0&=Qx_*+S^\top u_*+q+C^\top P_\infty(Cx_*+Du_*+\sigma)+A^\top\zeta^*\\
&=-P_\infty(Ax_*+Bu_*)+N_\infty^\top(M_\infty^{-1}N_\infty x_*+u_*)+q+C^\top P_\infty\sigma+A^\top\varphi_\infty+A^\top\eta^*\\
&=N_\infty^\top\left[u_*+M_\infty^{-1}(N_\infty x_*+d_\infty)\right]+A^\top\eta^*\\
&=-N_\infty^\top M_\infty^{-1}B^\top\eta^*+A^\top\eta^*\\
&=(A-BK_\infty)^\top\eta^*.
\end{aligned} 0 = Q x ∗ + S ⊤ u ∗ + q + C ⊤ P ∞ ( C x ∗ + D u ∗ + σ ) + A ⊤ ζ ∗ = − P ∞ ( A x ∗ + B u ∗ ) + N ∞ ⊤ ( M ∞ − 1 N ∞ x ∗ + u ∗ ) + q + C ⊤ P ∞ σ + A ⊤ φ ∞ + A ⊤ η ∗ = N ∞ ⊤ [ u ∗ + M ∞ − 1 ( N ∞ x ∗ + d ∞ ) ] + A ⊤ η ∗ = − N ∞ ⊤ M ∞ − 1 B ⊤ η ∗ + A ⊤ η ∗ = ( A − B K ∞ ) ⊤ η ∗ .
Since P ∞ P_\infty P ∞ is the stabilizing solution of (22) , Corollary 2.6 implies that A − B K ∞ A-BK_\infty A − B K ∞ is invertible. Hence η ∗ = 0 \eta^*=0 η ∗ = 0 , and (74) follows.
Next, substituting η ∗ = 0 \eta^*=0 η ∗ = 0 into (77) , we get
u ∗ = − M ∞ − 1 ( N ∞ x ∗ + d ∞ ) = − K ∞ x ∗ − ℓ ∞ . u_*=-M_\infty^{-1}(N_\infty x_*+d_\infty)=-K_\infty x_*-\ell_\infty. u ∗ = − M ∞ − 1 ( N ∞ x ∗ + d ∞ ) = − K ∞ x ∗ − ℓ ∞ .
This is (75) . Moreover,
( A − B K ∞ ) x ∗ + b − B ℓ ∞ = A x ∗ + B ( − K ∞ x ∗ − ℓ ∞ ) + b = A x ∗ + B u ∗ + b = 0. \begin{aligned}
(A-BK_\infty)x_*+b-B\ell_\infty&=Ax_*+B(-K_\infty x_*-\ell_\infty)+b\\
&=Ax_*+Bu_*+b=0.
\end{aligned} ( A − B K ∞ ) x ∗ + b − B ℓ ∞ = A x ∗ + B ( − K ∞ x ∗ − ℓ ∞ ) + b = A x ∗ + B u ∗ + b = 0.
Since A − B K ∞ A-BK_\infty A − B K ∞ is invertible, this linear equation has a unique solution.
Finally, by the infinite-horizon Gaussian feedback formula (73) , the mean compatibility (75) , and the covariance compatibility (39) ,
π ∞ λ , ∗ ( ⋅ ∣ x ∗ ) = N ( − K ∞ x ∗ − ℓ ∞ , Λ ∗ λ ) = N ( u ∗ , Λ ∗ λ ) , \pi_\infty^{\lambda,*}(\cdot\mid x_*)=\mathcal{N}(-K_\infty x_*-\ell_\infty,\Lambda_*^\lambda)=\mathcal{N}(u_*,\Lambda_*^\lambda), π ∞ λ , ∗ ( ⋅ ∣ x ∗ ) = N ( − K ∞ x ∗ − ℓ ∞ , Λ ∗ λ ) = N ( u ∗ , Λ ∗ λ ) ,
which is (76) . ■
The closed-loop system of (8) induced by the finite-horizon optimal feedback is given by
d X T , λ ( t ) = [ ( A − B K T ( t ) ) X T , λ ( t ) + b − B ℓ T ( t ) ] d t + [ ( C − D K T ( t ) ) X T , λ ( t ) + σ − D ℓ T ( t ) ] d W ( t ) + D ( Λ T ( t ) ) 1 / 2 d W ~ ( t ) , t ∈ [ 0 , T ] , X T , λ ( 0 ) = x . (80) \begin{aligned}
dX^{T,\lambda}(t) &= [(A-BK_T(t))X^{T,\lambda}(t)+b-B\ell_T(t)]dt \\
&\quad+[(C-DK_T(t))X^{T,\lambda}(t)+\sigma-D\ell_T(t)]dW(t) \\
&\quad+D(\Lambda_T(t))^{1/2}d\widetilde{W}(t), \qquad t\in[0,T],\\
X^{T,\lambda}(0)&=x. \tag*{(80)}
\end{aligned} d X T , λ ( t ) X T , λ ( 0 ) = [( A − B K T ( t )) X T , λ ( t ) + b − B ℓ T ( t )] d t + [( C − D K T ( t )) X T , λ ( t ) + σ − D ℓ T ( t )] d W ( t ) + D ( Λ T ( t ) ) 1/2 d W ( t ) , t ∈ [ 0 , T ] , = x . (80)
The infinite-horizon closed-loop system induced by the feedback distribution (73) is defined by
d X ∞ , λ , x ( t ) = [ ( A − B K ∞ ) X ∞ , λ , x ( t ) + b − B ℓ ∞ ] d t + [ ( C − D K ∞ ) X ∞ , λ , x ( t ) + σ − D ℓ ∞ ] d W ( t ) + D ( Λ ∞ λ ) 1 / 2 d W ~ ( t ) , t ∈ [ 0 , ∞ ) , X ∞ , λ , x ( 0 ) = x . (81) \begin{aligned}
dX^{\infty,\lambda,x}(t) &= [(A-BK_\infty)X^{\infty,\lambda,x}(t)+b-B\ell_\infty]dt \\
&\quad+[(C-DK_\infty)X^{\infty,\lambda,x}(t)+\sigma-D\ell_\infty]dW(t) \\
&\quad+D(\Lambda_\infty^\lambda)^{1/2}d\widetilde{W}(t), \qquad t\in[0,\infty),\\
X^{\infty,\lambda,x}(0)&=x. \tag*{(81)}
\end{aligned} d X ∞ , λ , x ( t ) X ∞ , λ , x ( 0 ) = [( A − B K ∞ ) X ∞ , λ , x ( t ) + b − B ℓ ∞ ] d t + [( C − D K ∞ ) X ∞ , λ , x ( t ) + σ − D ℓ ∞ ] d W ( t ) + D ( Λ ∞ λ ) 1/2 d W ( t ) , t ∈ [ 0 , ∞ ) , = x . (81)
By (75) and the first equation in (28) , we obtain
( A − B K ∞ ) x ∗ + b − B ℓ ∞ = 0. (82) (A-BK_\infty)x_*+b-B\ell_\infty=0. \tag*{(82)} ( A − B K ∞ ) x ∗ + b − B ℓ ∞ = 0. (82)
Again using the above feedback distribution, define the stochastic reference process by
d X ∗ , λ ( t ) = [ ( A − B K ∞ ) X ∗ , λ ( t ) + b − B ℓ ∞ ] d t + [ ( C − D K ∞ ) X ∗ , λ ( t ) + σ − D ℓ ∞ ] d W ( t ) + D ( Λ ∞ λ ) 1 / 2 d W ~ ( t ) , t ∈ [ 0 , ∞ ) , X ∗ , λ ( 0 ) = x ∗ . (83) \begin{aligned}
dX^{*,\lambda}(t) &= [(A-BK_\infty)X^{*,\lambda}(t)+b-B\ell_\infty]dt \\
&\quad+[(C-DK_\infty)X^{*,\lambda}(t)+\sigma-D\ell_\infty]dW(t) \\
&\quad+D(\Lambda_\infty^\lambda)^{1/2}d\widetilde{W}(t), \qquad t\in[0,\infty),\\
X^{*,\lambda}(0)&=x_*. \tag*{(83)}
\end{aligned} d X ∗ , λ ( t ) X ∗ , λ ( 0 ) = [( A − B K ∞ ) X ∗ , λ ( t ) + b − B ℓ ∞ ] d t + [( C − D K ∞ ) X ∗ , λ ( t ) + σ − D ℓ ∞ ] d W ( t ) + D ( Λ ∞ λ ) 1/2 d W ( t ) , t ∈ [ 0 , ∞ ) , = x ∗ . (83)
Proposition 5.2. Let ( H 1 ) (H1) ( H 1 ) –( H 2 ) (H2) ( H 2 ) hold. Let ( x ∗ , u ∗ ) (x_*,u_*) ( x ∗ , u ∗ ) be the unique optimizer of the static problem (27) , X ∗ , λ X^{*,\lambda} X ∗ , λ be the unique solution of (83) , and let u ˉ ∗ , λ ( t ) \bar{u}^{*,\lambda}(t) u ˉ ∗ , λ ( t ) be the mean of the feedback distribution (73) evaluated along X ∗ , λ ( t ) X^{*,\lambda}(t) X ∗ , λ ( t ) , that is, u ˉ ∗ , λ ( t ) : = u ˉ ∞ ( X ∗ , λ ( t ) ) = − K ∞ X ∗ , λ ( t ) − ℓ ∞ \bar{u}^{*,\lambda}(t):=\bar{u}_\infty(X^{*,\lambda}(t))=-K_\infty X^{*,\lambda}(t)-\ell_\infty u ˉ ∗ , λ ( t ) := u ˉ ∞ ( X ∗ , λ ( t )) = − K ∞ X ∗ , λ ( t ) − ℓ ∞ . Then
E X ∗ , λ ( t ) = x ∗ , E u ˉ ∗ , λ ( t ) = u ∗ , t ∈ [ 0 , ∞ ) . \mathbb{E}X^{*,\lambda}(t)=x_*, \qquad\mathbb{E}\bar{u}^{*,\lambda}(t)=u_*, \qquad t\in[0,\infty). E X ∗ , λ ( t ) = x ∗ , E u ˉ ∗ , λ ( t ) = u ∗ , t ∈ [ 0 , ∞ ) .
Proof. Taking expectation in (83) and using (82) , we obtain
d d t E X ∗ , λ ( t ) = ( A − B K ∞ ) E X ∗ , λ ( t ) + b − B ℓ ∞ = ( A − B K ∞ ) ( E X ∗ , λ ( t ) − x ∗ ) , E X ∗ , λ ( 0 ) = x ∗ . \frac{d}{dt}\mathbb{E}X^{*,\lambda}(t)=(A-BK_\infty)\mathbb{E}X^{*,\lambda}(t)+b-B\ell_\infty=(A-BK_\infty)(\mathbb{E}X^{*,\lambda}(t)-x_*), \qquad\mathbb{E}X^{*,\lambda}(0)=x_*. d t d E X ∗ , λ ( t ) = ( A − B K ∞ ) E X ∗ , λ ( t ) + b − B ℓ ∞ = ( A − B K ∞ ) ( E X ∗ , λ ( t ) − x ∗ ) , E X ∗ , λ ( 0 ) = x ∗ .
Note that the above is a linear ordinary differential equation. Then its unique solution is given by
E X ∗ , λ ( t ) = x ∗ . \mathbb{E}X^{*,\lambda}(t)=x_*. E X ∗ , λ ( t ) = x ∗ .
Moreover, from (75) , we get
E u ˉ ∗ , λ ( t ) = − K ∞ E X ∗ , λ ( t ) − ℓ ∞ = − K ∞ x ∗ − ℓ ∞ = u ∗ . \mathbb{E}\bar{u}^{*,\lambda}(t)=-K_\infty\mathbb{E}X^{*,\lambda}(t)-\ell_\infty=-K_\infty x_*-\ell_\infty=u_*. E u ˉ ∗ , λ ( t ) = − K ∞ E X ∗ , λ ( t ) − ℓ ∞ = − K ∞ x ∗ − ℓ ∞ = u ∗ .
This completes the proof. ■
Remark 5.3. The means of X ∗ , λ X^{*,\lambda} X ∗ , λ and u ˉ ∗ , λ \bar{u}^{*,\lambda} u ˉ ∗ , λ stay at the static optimal points x ∗ x_* x ∗ and u ∗ u_* u ∗ , respectively, although the processes themselves are generally not constant.
By Corollary 2.19 and Lemma 2.5, the homogeneous closed-loop system d X ( t ) = ( A − B K ∞ ) X ( t ) d t + ( C − D K ∞ ) X ( t ) d W ( t ) dX(t)=(A-BK_\infty)X(t)dt+(C-DK_\infty)X(t)dW(t) d X ( t ) = ( A − B K ∞ ) X ( t ) d t + ( C − D K ∞ ) X ( t ) d W ( t ) is mean-square exponentially stable. That is, there exist constants C , κ > 0 C,\kappa>0 C , κ > 0 such that
E ∣ X ( t ) ∣ 2 ≤ C e − κ t ∣ X ( 0 ) ∣ 2 , t ≥ 0. (84) \mathbb{E}|X(t)|^2\le Ce^{-\kappa t}|X(0)|^2,\qquad t\ge0. \tag*{(84)} E ∣ X ( t ) ∣ 2 ≤ C e − κ t ∣ X ( 0 ) ∣ 2 , t ≥ 0. (84)
Equivalently, one can choose H ∈ S + + n H \in\mathbb{S}_{++}^{n} H ∈ S ++ n such that
( A − B K ∞ ) T H + H ( A − B K ∞ ) + ( C − D K ∞ ) T H ( C − D K ∞ ) ≤ − I n . (85) (A-BK_{\infty})^{\mathsf{T}}H+H(A-BK_{\infty})+(C-DK_{\infty})^{\mathsf{T}}H(C-DK_{\infty})\leq-I_n. \tag*{(85)} ( A − B K ∞ ) T H + H ( A − B K ∞ ) + ( C − D K ∞ ) T H ( C − D K ∞ ) ≤ − I n . (85)
Let h − : = λ min ( H ) > 0 h_-:=\lambda_{\min}(H)>0 h − := λ m i n ( H ) > 0 and h + : = λ max ( H ) > 0 h_+:=\lambda_{\max}(H)>0 h + := λ m a x ( H ) > 0 . Then
h − ∣ z ∣ 2 ≤ z T H z ≤ h + ∣ z ∣ 2 , z ∈ R n . (86) h_-|z|^2\leq z^{\mathsf{T}}Hz\leq h_+|z|^2,\qquad z\in\mathbb{R}^n. \tag*{(86)} h − ∣ z ∣ 2 ≤ z T H z ≤ h + ∣ z ∣ 2 , z ∈ R n . (86)
Lemma 5.4. Let ( H 1 ) (H1) ( H 1 ) –( H 2 ) (H2) ( H 2 ) hold. Then there exists a constant C > 0 C>0 C > 0 , independent of t t t , such that for any initial value x ∈ R n x\in\mathbb{R}^n x ∈ R n ,
sup t ≥ 0 E ∣ X ∞ , λ , x ( t ) ∣ 2 ≤ C ( 1 + ∣ x ∣ 2 + λ ) . (87) \sup_{t\geq0}\mathbb{E}|X^{\infty,\lambda,x}(t)|^2\leq C(1+|x|^2+\lambda). \tag*{(87)} t ≥ 0 sup E ∣ X ∞ , λ , x ( t ) ∣ 2 ≤ C ( 1 + ∣ x ∣ 2 + λ ) . (87) In particular,
sup t ≥ 0 E ∣ X ∗ , λ ( t ) ∣ 2 ≤ C ( 1 + λ ) . (88) \sup_{t\geq0}\mathbb{E}|X^{*,\lambda}(t)|^2\leq C(1+\lambda). \tag*{(88)} t ≥ 0 sup E ∣ X ∗ , λ ( t ) ∣ 2 ≤ C ( 1 + λ ) . (88)
Proof. Let Y ( t ) = X ∞ , λ , x ( t ) − x ∗ Y(t)=X^{\infty,\lambda,x}(t)-x_* Y ( t ) = X ∞ , λ , x ( t ) − x ∗ . Set ξ ∗ : = ( C − D K ∞ ) x ∗ + σ − D ℓ ∞ = C x ∗ + D u ∗ + σ \xi_*:=(C-DK_{\infty})x_*+\sigma-D\ell_{\infty}=Cx_*+Du_*+\sigma ξ ∗ := ( C − D K ∞ ) x ∗ + σ − D ℓ ∞ = C x ∗ + D u ∗ + σ . Then by (82) , we have
d Y ( t ) = ( A − B K ∞ ) Y ( t ) d t + [ ( C − D K ∞ ) Y ( t ) + ξ ∗ ] d W ( t ) + D ( Λ ∞ λ ) 1 / 2 d W ~ ( t ) , t ∈ [ 0 , ∞ ) , Y ( 0 ) = x − x ∗ . \begin{aligned}
dY(t)&=(A-BK_{\infty})Y(t)dt+\left[(C-DK_{\infty})Y(t)+\xi_*\right]dW(t)\\
&\quad+D(\Lambda_{\infty}^{\lambda})^{1/2}d\widetilde{W}(t),\qquad t\in[0,\infty),\\
Y(0)&=x-x_*.
\end{aligned} d Y ( t ) Y ( 0 ) = ( A − B K ∞ ) Y ( t ) d t + [ ( C − D K ∞ ) Y ( t ) + ξ ∗ ] d W ( t ) + D ( Λ ∞ λ ) 1/2 d W ( t ) , t ∈ [ 0 , ∞ ) , = x − x ∗ .
Applying Itô’s formula to Y ( t ) T H Y ( t ) Y(t)^{\mathsf{T}}HY(t) Y ( t ) T H Y ( t ) , and using the independence of W W W and W ~ \widetilde{W} W , we obtain
d d t E ( Y ( t ) T H Y ( t ) ) = E [ Y ( t ) T ( ( A − B K ∞ ) T H + H ( A − B K ∞ ) ) Y ( t ) ] + E [ Y ( t ) T ( C − D K ∞ ) T H ( C − D K ∞ ) Y ( t ) ] + 2 E [ Y ( t ) T ( C − D K ∞ ) T H ξ ∗ ] + ξ ∗ T H ξ ∗ + tr ( D T H D Λ ∞ λ ) . \begin{aligned}
\frac{d}{dt}\mathbb{E}(Y(t)^{\mathsf{T}}HY(t))
={}&\mathbb{E}\left[Y(t)^{\mathsf{T}}\left((A-BK_{\infty})^{\mathsf{T}}H+H(A-BK_{\infty})\right)Y(t)\right]\\
&+\mathbb{E}\left[Y(t)^{\mathsf{T}}(C-DK_{\infty})^{\mathsf{T}}H(C-DK_{\infty})Y(t)\right]\\
&+2\mathbb{E}\left[Y(t)^{\mathsf{T}}(C-DK_{\infty})^{\mathsf{T}}H\xi_*\right]+\xi_*^{\mathsf{T}}H\xi_*+\operatorname{tr}\left(D^{\mathsf{T}}HD\Lambda_{\infty}^{\lambda}\right).
\end{aligned} d t d E ( Y ( t ) T H Y ( t )) = E [ Y ( t ) T ( ( A − B K ∞ ) T H + H ( A − B K ∞ ) ) Y ( t ) ] + E [ Y ( t ) T ( C − D K ∞ ) T H ( C − D K ∞ ) Y ( t ) ] + 2 E [ Y ( t ) T ( C − D K ∞ ) T H ξ ∗ ] + ξ ∗ T H ξ ∗ + tr ( D T H D Λ ∞ λ ) .
By (85) , the quadratic terms are bounded by − E ∣ Y ( t ) ∣ 2 -\mathbb{E}|Y(t)|^2 − E ∣ Y ( t ) ∣ 2 . For the cross term, Young’s inequality gives
2 Y ( t ) T ( C − D K ∞ ) T H ξ ∗ ≤ 1 2 ∣ Y ( t ) ∣ 2 + C ∣ ξ ∗ ∣ 2 . 2Y(t)^{\mathsf{T}}(C-DK_{\infty})^{\mathsf{T}}H\xi_*\leq\frac{1}{2}|Y(t)|^2+C|\xi_*|^2. 2 Y ( t ) T ( C − D K ∞ ) T H ξ ∗ ≤ 2 1 ∣ Y ( t ) ∣ 2 + C ∣ ξ ∗ ∣ 2 .
The exploration-noise trace term satisfies
tr ( D T H D Λ ∞ λ ) = λ 2 tr ( D T H D M − 1 ) ≤ C λ . \operatorname{tr}(D^{\mathsf{T}}HD\Lambda_{\infty}^{\lambda})=\frac{\lambda}{2}\operatorname{tr}(D^{\mathsf{T}}HDM^{-1})\leq C\lambda. tr ( D T H D Λ ∞ λ ) = 2 λ tr ( D T H D M − 1 ) ≤ C λ .
Thus
d d t E ( Y ( t ) T H Y ( t ) ) ≤ − 1 2 E ∣ Y ( t ) ∣ 2 + C ( 1 + λ ) . \frac{d}{dt}\mathbb{E}(Y(t)^{\mathsf{T}}HY(t))\leq-\frac{1}{2}\mathbb{E}|Y(t)|^2+C(1+\lambda). d t d E ( Y ( t ) T H Y ( t )) ≤ − 2 1 E ∣ Y ( t ) ∣ 2 + C ( 1 + λ ) .
From (86) , we have ∣ y ∣ 2 ≥ h + − 1 y T H y |y|^2\geq h_+^{-1}y^{\mathsf{T}}Hy ∣ y ∣ 2 ≥ h + − 1 y T H y . Consequently,
d d t E ( Y ( t ) T H Y ( t ) ) ≤ − c E ( Y ( t ) T H Y ( t ) ) + C ( 1 + λ ) . \frac{d}{dt}\mathbb{E}(Y(t)^{\mathsf{T}}HY(t))\leq-c\mathbb{E}(Y(t)^{\mathsf{T}}HY(t))+C(1+\lambda). d t d E ( Y ( t ) T H Y ( t )) ≤ − c E ( Y ( t ) T H Y ( t )) + C ( 1 + λ ) .
Gronwall’s inequality then yields
E ∣ Y ( t ) ∣ 2 ≤ C e − c t ∣ x − x ∗ ∣ 2 + C ( 1 + λ ) , t ≥ 0. \mathbb{E}|Y(t)|^2\leq Ce^{-ct}|x-x_*|^2+C(1+\lambda),\qquad t\geq0. E ∣ Y ( t ) ∣ 2 ≤ C e − c t ∣ x − x ∗ ∣ 2 + C ( 1 + λ ) , t ≥ 0.
Since x ∗ x_* x ∗ is fixed, (87) follows. Setting x = x ∗ x=x_* x = x ∗ gives (88) . ■
Theorem 5.5. Let ( H 1 ) (H1) ( H 1 ) –( H 2 ) (H2) ( H 2 ) hold. Let X T , λ X^{T,\lambda} X T , λ be the unique solution of the finite-horizon closed-loop system (80) , and let X ∗ , λ X^{*,\lambda} X ∗ , λ be the stochastic reference process given by (83) . Then, for any fixed initial state x x x and exploration weight λ > 0 \lambda>0 λ > 0 , there exist constants K = K x , λ > 0 K=K_{x,\lambda}>0 K = K x , λ > 0 and μ > 0 \mu>0 μ > 0 , independent of T , t T,t T , t , such that for all 0 ≤ t ≤ T 0\leq t\leq T 0 ≤ t ≤ T ,
E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 ≤ K ( e − μ t + e − μ ( T − t ) ) . (89) \mathbb{E}|X^{T,\lambda}(t)-X^{*,\lambda}(t)|^2\leq K\left(e^{-\mu t}+e^{-\mu(T-t)}\right). \tag*{(89)} E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 ≤ K ( e − μ t + e − μ ( T − t ) ) . (89)
Proof. We first compare X T , λ X^{T,\lambda} X T , λ with the infinite-horizon intermediate process X ∞ , λ , x X^{\infty,\lambda,x} X ∞ , λ , x with the same initial value. Let Z ( t ) = X T , λ ( t ) − X ∞ , λ , x ( t ) Z(t)=X^{T,\lambda}(t)-X^{\infty,\lambda,x}(t) Z ( t ) = X T , λ ( t ) − X ∞ , λ , x ( t ) . Subtracting (81) from (80) , we obtain
d Z ( t ) = [ ( A − B K ∞ ) Z ( t ) − B Δ R t ] d t + [ ( C − D K ∞ ) Z ( t ) − D Δ R t ] d W ( t ) + D Δ Λ t d W ~ ( t ) , t ∈ [ 0 , T ] , Z ( 0 ) = 0 , (90) \begin{aligned}
dZ(t)={}&\left[(A-BK_{\infty})Z(t)-B\Delta R_t\right]dt\\
&+\left[(C-DK_{\infty})Z(t)-D\Delta R_t\right]dW(t)+D\Delta\Lambda_t\,d\widetilde W(t),\qquad t\in[0,T],\\
Z(0)={}&0, \tag*{(90)}
\end{aligned} d Z ( t ) = Z ( 0 ) = [ ( A − B K ∞ ) Z ( t ) − B Δ R t ] d t + [ ( C − D K ∞ ) Z ( t ) − D Δ R t ] d W ( t ) + D Δ Λ t d W ( t ) , t ∈ [ 0 , T ] , 0 , (90)
where
Δ R t = ( K T ( t ) − K ∞ ) X T , λ ( t ) + ( ℓ T ( t ) − ℓ ∞ ) , Δ Λ t = ( Λ T λ ( t ) ) 1 / 2 − ( Λ ∞ λ ) 1 / 2 . \Delta R_t=(K_T(t)-K_{\infty})X^{T,\lambda}(t)+(\ell_T(t)-\ell_{\infty}),\qquad
\Delta\Lambda_t=(\Lambda_T^\lambda(t))^{1/2}-(\Lambda_{\infty}^{\lambda})^{1/2}. Δ R t = ( K T ( t ) − K ∞ ) X T , λ ( t ) + ( ℓ T ( t ) − ℓ ∞ ) , Δ Λ t = ( Λ T λ ( t ) ) 1/2 − ( Λ ∞ λ ) 1/2 .
By Proposition 4.11, there exist constants C , γ 0 > 0 C,\gamma_0>0 C , γ 0 > 0 such that
∥ K T ( t ) − K ∞ ∥ + ∣ ℓ T ( t ) − ℓ ∞ ∣ ≤ C δ t , ∥ Δ Λ t ∥ ≤ C λ δ t , (91) \lVert K_T(t)-K_{\infty}\rVert+|\ell_T(t)-\ell_{\infty}|\le C\delta_t,\qquad\lVert\Delta\Lambda_t\rVert\le C\sqrt{\lambda}\,\delta_t, \tag*{(91)} ∥ K T ( t ) − K ∞ ∥ + ∣ ℓ T ( t ) − ℓ ∞ ∣ ≤ C δ t , ∥ Δ Λ t ∥ ≤ C λ δ t , (91)
where δ t = e − γ 0 ( T − t ) \delta_t=e^{-\gamma_0(T-t)} δ t = e − γ 0 ( T − t ) . Since X T , λ ( t ) = Z ( t ) + X ∞ , λ , x ( t ) X^{T,\lambda}(t)=Z(t)+X^{\infty,\lambda,x}(t) X T , λ ( t ) = Z ( t ) + X ∞ , λ , x ( t ) , we have
Δ R t = ( K T ( t ) − K ∞ ) Z ( t ) + ( K T ( t ) − K ∞ ) X ∞ , λ , x ( t ) + ( ℓ T ( t ) − ℓ ∞ ) . \Delta R_t=(K_T(t)-K_{\infty})Z(t)+(K_T(t)-K_{\infty})X^{\infty,\lambda,x}(t)+(\ell_T(t)-\ell_{\infty}). Δ R t = ( K T ( t ) − K ∞ ) Z ( t ) + ( K T ( t ) − K ∞ ) X ∞ , λ , x ( t ) + ( ℓ T ( t ) − ℓ ∞ ) .
Then by Lemma 5.4, we have
E ∣ Δ R t ∣ 2 ≤ C δ t 2 ( E ∣ Z ( t ) ∣ 2 + E ∣ X ∞ , λ , x ( t ) ∣ 2 + 1 ) ≤ C δ t 2 E ∣ Z ( t ) ∣ 2 + C ( 1 + ∣ x ∣ 2 + λ ) δ t 2 . (92) \begin{aligned}
\mathbb{E}|\Delta R_t|^2
&\le C\delta_t^2\left(\mathbb{E}|Z(t)|^2+\mathbb{E}|X^{\infty,\lambda,x}(t)|^2+1\right)\\
&\le C\delta_t^2\mathbb{E}|Z(t)|^2+C(1+|x|^2+\lambda)\delta_t^2. \tag*{(92)}
\end{aligned} E ∣Δ R t ∣ 2 ≤ C δ t 2 ( E ∣ Z ( t ) ∣ 2 + E ∣ X ∞ , λ , x ( t ) ∣ 2 + 1 ) ≤ C δ t 2 E ∣ Z ( t ) ∣ 2 + C ( 1 + ∣ x ∣ 2 + λ ) δ t 2 . (92)
We next apply Itô’s formula to Z ( t ) ⊤ H Z ( t ) Z(t)^\top HZ(t) Z ( t ) ⊤ H Z ( t ) . From (90) , we have
d d t E ( Z ( t ) ⊤ H Z ( t ) ) = E [ Z ( t ) ⊤ ( ( A − B K ∞ ) ⊤ H + H ( A − B K ∞ ) ) Z ( t ) ] + E [ Z ( t ) ⊤ ( C − D K ∞ ) ⊤ H ( C − D K ∞ ) Z ( t ) ] − 2 E [ Z ( t ) ⊤ H B Δ R t ] − 2 E [ Z ( t ) ⊤ ( C − D K ∞ ) ⊤ H D Δ R t ] + E [ ( Δ R t ) ⊤ D ⊤ H D Δ R t ] + tr ( ( Δ Λ t ) ⊤ D ⊤ H D Δ Λ t ) . \begin{aligned}
\frac{d}{dt}\mathbb{E}(Z(t)^\top HZ(t))
={}&\mathbb{E}\left[Z(t)^\top\left((A-BK_{\infty})^\top H+H(A-BK_{\infty})\right)Z(t)\right]\\
&+\mathbb{E}\left[Z(t)^\top(C-DK_{\infty})^\top H(C-DK_{\infty})Z(t)\right]\\
&-2\mathbb{E}[Z(t)^\top HB\Delta R_t]-2\mathbb{E}[Z(t)^\top(C-DK_{\infty})^\top HD\Delta R_t]\\
&+\mathbb{E}[(\Delta R_t)^\top D^\top HD\Delta R_t]+\operatorname{tr}\left((\Delta\Lambda_t)^\top D^\top HD\Delta\Lambda_t\right).
\end{aligned} d t d E ( Z ( t ) ⊤ H Z ( t )) = E [ Z ( t ) ⊤ ( ( A − B K ∞ ) ⊤ H + H ( A − B K ∞ ) ) Z ( t ) ] + E [ Z ( t ) ⊤ ( C − D K ∞ ) ⊤ H ( C − D K ∞ ) Z ( t ) ] − 2 E [ Z ( t ) ⊤ H B Δ R t ] − 2 E [ Z ( t ) ⊤ ( C − D K ∞ ) ⊤ H D Δ R t ] + E [( Δ R t ) ⊤ D ⊤ H D Δ R t ] + tr ( ( Δ Λ t ) ⊤ D ⊤ H D Δ Λ t ) .
Using (85) and Young’s inequality, we get
d d t E ( Z ( t ) ⊤ H Z ( t ) ) ≤ − 1 2 E ∣ Z ( t ) ∣ 2 + C E ∣ Δ R t ∣ 2 + C ∥ Δ Λ t ∥ 2 . \frac{d}{dt}\mathbb{E}(Z(t)^\top HZ(t))\le-\frac{1}{2}\mathbb{E}|Z(t)|^2+C\mathbb{E}|\Delta R_t|^2+C\|\Delta\Lambda_t\|^2. d t d E ( Z ( t ) ⊤ H Z ( t )) ≤ − 2 1 E ∣ Z ( t ) ∣ 2 + C E ∣Δ R t ∣ 2 + C ∥Δ Λ t ∥ 2 .
Let V Z ( t ) = E ( Z ( t ) ⊤ H Z ( t ) ) V_Z(t)=\mathbb{E}(Z(t)^\top HZ(t)) V Z ( t ) = E ( Z ( t ) ⊤ H Z ( t )) . Substituting (91) and (92) into the above, we obtain
V ˙ Z ( t ) ≤ [ − c + C δ t 2 ] V Z ( t ) + C ( 1 + ∣ x ∣ 2 + λ ) δ t 2 , V Z ( 0 ) = 0. \dot V_Z(t)\le\left[-c+C\delta_t^2\right]V_Z(t)+C(1+|x|^2+\lambda)\delta_t^2,\qquad V_Z(0)=0. V ˙ Z ( t ) ≤ [ − c + C δ t 2 ] V Z ( t ) + C ( 1 + ∣ x ∣ 2 + λ ) δ t 2 , V Z ( 0 ) = 0.
By Gronwall’s inequality,
V Z ( t ) ≤ C ( 1 + ∣ x ∣ 2 + λ ) ∫ 0 t exp ( ∫ s t [ − c + C δ r 2 ] d r ) δ s 2 d s . V_Z(t)\le C(1+|x|^2+\lambda)\int_0^t\exp\left(\int_s^t[-c+C\delta_r^2]\,dr\right)\delta_s^2\,ds. V Z ( t ) ≤ C ( 1 + ∣ x ∣ 2 + λ ) ∫ 0 t exp ( ∫ s t [ − c + C δ r 2 ] d r ) δ s 2 d s .
Note that
∫ s t δ r 2 d r = ∫ s t e − 2 γ 0 ( T − r ) d r = e − 2 γ 0 ( T − t ) − e − 2 γ 0 ( T − s ) 2 γ 0 ≤ C . \int_s^t\delta_r^2\,dr=\int_s^t e^{-2\gamma_0(T-r)}\,dr=\frac{e^{-2\gamma_0(T-t)}-e^{-2\gamma_0(T-s)}}{2\gamma_0}\le C. ∫ s t δ r 2 d r = ∫ s t e − 2 γ 0 ( T − r ) d r = 2 γ 0 e − 2 γ 0 ( T − t ) − e − 2 γ 0 ( T − s ) ≤ C .
Hence
exp ( ∫ s t [ − c + C δ r 2 ] d r ) = e − c ( t − s ) exp ( C ∫ s t δ r 2 d r ) ≤ C e − c ( t − s ) . \exp\left(\int_s^t[-c+C\delta_r^2]\,dr\right)=e^{-c(t-s)}\exp\left(C\int_s^t\delta_r^2\,dr\right)\le Ce^{-c(t-s)}. exp ( ∫ s t [ − c + C δ r 2 ] d r ) = e − c ( t − s ) exp ( C ∫ s t δ r 2 d r ) ≤ C e − c ( t − s ) .
It follows that
V Z ( t ) ≤ C ( 1 + ∣ x ∣ 2 + λ ) ∫ 0 t e − c ( t − s ) e − 2 γ 0 ( T − s ) d s ≤ C ( 1 + ∣ x ∣ 2 + λ ) e − 2 γ 0 ( T − t ) . V_Z(t)\le C(1+|x|^2+\lambda)\int_0^t e^{-c(t-s)}e^{-2\gamma_0(T-s)}\,ds\le C(1+|x|^2+\lambda)e^{-2\gamma_0(T-t)}. V Z ( t ) ≤ C ( 1 + ∣ x ∣ 2 + λ ) ∫ 0 t e − c ( t − s ) e − 2 γ 0 ( T − s ) d s ≤ C ( 1 + ∣ x ∣ 2 + λ ) e − 2 γ 0 ( T − t ) .
Using (86) , we get
E ∣ X T , λ ( t ) − X ∞ , λ , x ( t ) ∣ 2 ≤ C ( 1 + ∣ x ∣ 2 + λ ) e − 2 γ 0 ( T − t ) . (93) \mathbb{E}|X^{T,\lambda}(t)-X^{\infty,\lambda,x}(t)|^2 \le C(1+|x|^2+\lambda)e^{-2\gamma_0(T-t)}. \tag*{(93)} E ∣ X T , λ ( t ) − X ∞ , λ , x ( t ) ∣ 2 ≤ C ( 1 + ∣ x ∣ 2 + λ ) e − 2 γ 0 ( T − t ) . (93)
Together with Lemma 5.4, this also gives the uniform second-moment estimate for the finite-horizon state process X T , λ X^{T,\lambda} X T , λ :
sup 0 ≤ t ≤ T E ∣ X T , λ ( t ) ∣ 2 ≤ C ( 1 + ∣ x ∣ 2 + λ ) . (94) \sup_{0\le t\le T}\mathbb{E}|X^{T,\lambda}(t)|^2 \le C(1+|x|^2+\lambda). \tag*{(94)} 0 ≤ t ≤ T sup E ∣ X T , λ ( t ) ∣ 2 ≤ C ( 1 + ∣ x ∣ 2 + λ ) . (94)
On the other hand, X ∞ , λ , x X^{\infty,\lambda,x} X ∞ , λ , x and X ∗ , λ X^{*,\lambda} X ∗ , λ satisfy the same infinite-horizon closed-loop equation and are driven by the same Brownian motions. Their only difference is the initial value. Let Y ( t ) = X ∞ , λ , x ( t ) − X ∗ , λ ( t ) Y(t)=X^{\infty,\lambda,x}(t)-X^{*,\lambda}(t) Y ( t ) = X ∞ , λ , x ( t ) − X ∗ , λ ( t ) . Then
d Y ( t ) = ( A − B K ∞ ) Y ( t ) d t + ( C − D K ∞ ) Y ( t ) d W ( t ) , t ∈ [ 0 , ∞ ) , dY(t)=(A-BK_\infty)Y(t)dt+(C-DK_\infty)Y(t)dW(t),\qquad t\in[0,\infty), d Y ( t ) = ( A − B K ∞ ) Y ( t ) d t + ( C − D K ∞ ) Y ( t ) d W ( t ) , t ∈ [ 0 , ∞ ) ,
with Y ( 0 ) = x − x ∗ Y(0)=x-x_* Y ( 0 ) = x − x ∗ . By (84) , we obtain the estimate
E ∣ X ∞ , λ , x ( t ) − X ∗ , λ ( t ) ∣ 2 ≤ C e − κ t ∣ x − x ∗ ∣ 2 . (95) \mathbb{E}|X^{\infty,\lambda,x}(t)-X^{*,\lambda}(t)|^2\le Ce^{-\kappa t}|x-x_*|^2. \tag*{(95)} E ∣ X ∞ , λ , x ( t ) − X ∗ , λ ( t ) ∣ 2 ≤ C e − κ t ∣ x − x ∗ ∣ 2 . (95)
Finally,
X T , λ ( t ) − X ∗ , λ ( t ) = ( X T , λ ( t ) − X ∞ , λ , x ( t ) ) + ( X ∞ , λ , x ( t ) − X ∗ , λ ( t ) ) . X^{T,\lambda}(t)-X^{*,\lambda}(t)=\left(X^{T,\lambda}(t)-X^{\infty,\lambda,x}(t)\right)+\left(X^{\infty,\lambda,x}(t)-X^{*,\lambda}(t)\right). X T , λ ( t ) − X ∗ , λ ( t ) = ( X T , λ ( t ) − X ∞ , λ , x ( t ) ) + ( X ∞ , λ , x ( t ) − X ∗ , λ ( t ) ) .
Using ∣ a + b ∣ 2 ≤ 2 ∣ a ∣ 2 + 2 ∣ b ∣ 2 |a+b|^2\le2|a|^2+2|b|^2 ∣ a + b ∣ 2 ≤ 2∣ a ∣ 2 + 2∣ b ∣ 2 , and combining (93) with (95) , we obtain (89) . ■
We now derive a Wasserstein comparison for the control distributions from the state-process estimate. The finite-horizon optimal feedback distribution is
π T λ , ∗ ( d u ∣ t , x ) = N ( u ˉ T ( t , x ) , Λ T λ ( t ) ) ( d u ) , u ˉ T ( t , x ) = − K T ( t ) x − ℓ T ( t ) , \pi_T^{\lambda,*}(du\mid t,x)=\mathcal{N}\left(\bar{u}_T(t,x),\Lambda_T^\lambda(t)\right)(du),\qquad\bar{u}_T(t,x)=-K_T(t)x-\ell_T(t), π T λ , ∗ ( d u ∣ t , x ) = N ( u ˉ T ( t , x ) , Λ T λ ( t ) ) ( d u ) , u ˉ T ( t , x ) = − K T ( t ) x − ℓ T ( t ) ,
and the infinite-horizon Gaussian feedback distribution defined by the long-time parameters is
π ∞ λ , ∗ ( d u ∣ x ) = N ( u ˉ ∞ ( x ) , Λ ∞ λ ) ( d u ) , u ˉ ∞ ( x ) = − K ∞ x − ℓ ∞ . \pi_\infty^{\lambda,*}(du\mid x)=\mathcal{N}\left(\bar{u}_\infty(x),\Lambda_\infty^\lambda\right)(du),\qquad\bar{u}_\infty(x)=-K_\infty x-\ell_\infty. π ∞ λ , ∗ ( d u ∣ x ) = N ( u ˉ ∞ ( x ) , Λ ∞ λ ) ( d u ) , u ˉ ∞ ( x ) = − K ∞ x − ℓ ∞ .
To compare the finite-horizon control distribution with its infinite-horizon reference distribution, we use the W 2 W_2 W 2 distance [43 ] .
Definition 5.6. Let μ , ν \mu,\nu μ , ν be two Borel probability measures on R m \mathbb{R}^m R m with finite second moments, and let Π ( μ , ν ) \Pi(\mu,\nu) Π ( μ , ν ) be the set of all joint Borel probability distributions π \pi π on R m × R m \mathbb{R}^m\times\mathbb{R}^m R m × R m with first marginal μ \mu μ and second marginal ν \nu ν . The W 2 W_2 W 2 distance between μ \mu μ and ν \nu ν is defined by
W 2 ( μ , ν ) = ( inf π ∈ Π ( μ , ν ) ∫ R m × R m ∣ u − v ∣ 2 d π ( u , v ) ) 1 / 2 , W_2(\mu,\nu)=\left(\inf_{\pi\in\Pi(\mu,\nu)}\int_{\mathbb{R}^m\times\mathbb{R}^m}|u-v|^2d\pi(u,v)\right)^{1/2}, W 2 ( μ , ν ) = ( π ∈ Π ( μ , ν ) inf ∫ R m × R m ∣ u − v ∣ 2 d π ( u , v ) ) 1/2 , where ∣ ⋅ ∣ |\cdot| ∣ ⋅ ∣ is the Euclidean norm on R m \mathbb{R}^m R m .
When the feedback distributions are Gaussian, the following formula, which is taken from Peyré–Cuturi [29 ] , allows us to compare them through their means and covariance matrices.
Lemma 5.7. Let μ i = N ( m i , Σ i ) \mu_i=\mathcal{N}(m_i,\Sigma_i) μ i = N ( m i , Σ i ) , i = 1 , 2 i=1,2 i = 1 , 2 , where m i ∈ R m m_i\in\mathbb{R}^m m i ∈ R m and Σ i ∈ S + + m \Sigma_i\in\mathbb{S}^{m}_{++} Σ i ∈ S ++ m .
W 2 2 ( μ 1 , μ 2 ) = ∣ m 1 − m 2 ∣ 2 + B 2 ( Σ 1 , Σ 2 ) , W_2^2(\mu_1,\mu_2)=|m_1-m_2|^2+\mathcal{B}^2(\Sigma_1,\Sigma_2), W 2 2 ( μ 1 , μ 2 ) = ∣ m 1 − m 2 ∣ 2 + B 2 ( Σ 1 , Σ 2 ) , B 2 ( Σ 1 , Σ 2 ) = tr [ Σ 1 + Σ 2 − 2 ( Σ 1 1 / 2 Σ 2 Σ 1 1 / 2 ) 1 / 2 ] , \mathcal{B}^2(\Sigma_1,\Sigma_2)=\operatorname{tr}\left[\Sigma_1+\Sigma_2-2\left(\Sigma_1^{1/2}\Sigma_2\Sigma_1^{1/2}\right)^{1/2}\right], B 2 ( Σ 1 , Σ 2 ) = tr [ Σ 1 + Σ 2 − 2 ( Σ 1 1/2 Σ 2 Σ 1 1/2 ) 1/2 ] , B ( Σ 1 , Σ 2 ) ≤ ∥ Σ 1 1 / 2 − Σ 2 1 / 2 ∥ F . \mathcal{B}(\Sigma_1,\Sigma_2)\le\left\|\Sigma_1^{1/2}-\Sigma_2^{1/2}\right\|_F. B ( Σ 1 , Σ 2 ) ≤ Σ 1 1/2 − Σ 2 1/2 F .
Combining Lemma 5.7 with Proposition 4.11 yields the following estimate.
Lemma 5.8. Let ( H 1 ) (H1) ( H 1 ) –( H 2 ) (H2) ( H 2 ) hold. Then for any x ∈ R n x\in\mathbb{R}^n x ∈ R n and 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T ,
W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , x ) , π ∞ λ , ∗ ( ⋅ ∣ x ) ) ≤ C ( 1 + ∣ x ∣ 2 + λ ) e − 2 γ ( T − t ) . (96) W_2^2\left(\pi_T^{\lambda,*}(\cdot\mid t,x),\pi_\infty^{\lambda,*}(\cdot\mid x)\right)\le C(1+|x|^2+\lambda)e^{-2\gamma(T-t)}. \tag*{(96)} W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , x ) , π ∞ λ , ∗ ( ⋅ ∣ x ) ) ≤ C ( 1 + ∣ x ∣ 2 + λ ) e − 2 γ ( T − t ) . (96)
Proof. By Lemma 5.7 , we get
W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , x ) , π ∞ λ , ∗ ( ⋅ ∣ x ) ) ≤ ∣ u ˉ T ( t , x ) − u ˉ ∞ ( x ) ∣ 2 + ∥ ( Λ T λ ( t ) ) 1 / 2 − ( Λ ∞ λ ) 1 / 2 ∥ F 2 . W_2^2\left(\pi_T^{\lambda,*}(\cdot\mid t,x),\pi_\infty^{\lambda,*}(\cdot\mid x)\right)\leq\left|\bar{u}_T(t,x)-\bar{u}_\infty(x)\right|^2+\left\|\left(\Lambda_T^\lambda(t)\right)^{1/2}-\left(\Lambda_\infty^\lambda\right)^{1/2}\right\|_F^2. W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , x ) , π ∞ λ , ∗ ( ⋅ ∣ x ) ) ≤ ∣ u ˉ T ( t , x ) − u ˉ ∞ ( x ) ∣ 2 + ( Λ T λ ( t ) ) 1/2 − ( Λ ∞ λ ) 1/2 F 2 .
Combining this with Proposition 4.11 and (4.31) yields (96) .
Along the state processes X T , λ X^{T,\lambda} X T , λ and X ∗ , λ X^{*,\lambda} X ∗ , λ , define the finite-horizon control mean and the reference control mean by
u ˉ T , λ ( t ) = u ˉ T ( t , X T , λ ( t ) ) = − K T ( t ) X T , λ ( t ) − ℓ T ( t ) , u ˉ ∗ , λ ( t ) = u ˉ ∞ ( X ∗ , λ ( t ) ) = − K ∞ X ∗ , λ ( t ) − ℓ ∞ . (97) \begin{aligned}
\bar{u}^{T,\lambda}(t)&=\bar{u}_T(t,X^{T,\lambda}(t))=-K_T(t)X^{T,\lambda}(t)-\ell_T(t),\\
\bar{u}^{*,\lambda}(t)&=\bar{u}_\infty(X^{*,\lambda}(t))=-K_\infty X^{*,\lambda}(t)-\ell_\infty. \tag*{(97)}
\end{aligned} u ˉ T , λ ( t ) u ˉ ∗ , λ ( t ) = u ˉ T ( t , X T , λ ( t )) = − K T ( t ) X T , λ ( t ) − ℓ T ( t ) , = u ˉ ∞ ( X ∗ , λ ( t )) = − K ∞ X ∗ , λ ( t ) − ℓ ∞ . (97)
Lemma 5.9. Under the conditions of Theorem 5.5 , we have
E ∣ u ˉ T , λ ( t ) − u ˉ ∗ , λ ( t ) ∣ 2 ≤ C e − γ t ∣ x − x ∗ ∣ 2 + C ( 1 + ∣ x ∣ 2 + λ ) e − γ ( T − t ) . (98) \mathbb{E}\left|\bar{u}^{T,\lambda}(t)-\bar{u}^{*,\lambda}(t)\right|^2\leq Ce^{-\gamma t}|x-x_*|^2+C(1+|x|^2+\lambda)e^{-\gamma(T-t)}. \tag*{(98)} E u ˉ T , λ ( t ) − u ˉ ∗ , λ ( t ) 2 ≤ C e − γ t ∣ x − x ∗ ∣ 2 + C ( 1 + ∣ x ∣ 2 + λ ) e − γ ( T − t ) . (98) Proof. By (97) , we have
u ˉ T , λ ( t ) − u ˉ ∗ , λ ( t ) = − ( K T ( t ) − K ∞ ) X T , λ ( t ) − K ∞ ( X T , λ ( t ) − X ∗ , λ ( t ) ) − ( ℓ T ( t ) − ℓ ∞ ) . \bar{u}^{T,\lambda}(t)-\bar{u}^{*,\lambda}(t)=-(K_T(t)-K_\infty)X^{T,\lambda}(t)-K_\infty\left(X^{T,\lambda}(t)-X^{*,\lambda}(t)\right)-\left(\ell_T(t)-\ell_\infty\right). u ˉ T , λ ( t ) − u ˉ ∗ , λ ( t ) = − ( K T ( t ) − K ∞ ) X T , λ ( t ) − K ∞ ( X T , λ ( t ) − X ∗ , λ ( t ) ) − ( ℓ T ( t ) − ℓ ∞ ) . Then by Proposition 4.11, we get
E ∣ u ˉ T , λ ( t ) − u ˉ ∗ , λ ( t ) ∣ 2 ≤ C ∥ K T ( t ) − K ∞ ∥ 2 E ∣ X T , λ ( t ) ∣ 2 + C E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 + C ∣ ℓ T ( t ) − ℓ ∞ ∣ 2 ≤ C e − γ t ∣ x − x ∗ ∣ 2 + C ( 1 + ∣ x ∣ 2 + λ ) e − γ ( T − t ) . \begin{aligned}
\mathbb{E}\left|\bar{u}^{T,\lambda}(t)-\bar{u}^{*,\lambda}(t)\right|^2
&\leq C\|K_T(t)-K_\infty\|^2\mathbb{E}|X^{T,\lambda}(t)|^2+C\mathbb{E}|X^{T,\lambda}(t)-X^{*,\lambda}(t)|^2+C|\ell_T(t)-\ell_\infty|^2\\
&\leq Ce^{-\gamma t}|x-x_*|^2+C(1+|x|^2+\lambda)e^{-\gamma(T-t)}.
\end{aligned} E u ˉ T , λ ( t ) − u ˉ ∗ , λ ( t ) 2 ≤ C ∥ K T ( t ) − K ∞ ∥ 2 E ∣ X T , λ ( t ) ∣ 2 + C E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 + C ∣ ℓ T ( t ) − ℓ ∞ ∣ 2 ≤ C e − γ t ∣ x − x ∗ ∣ 2 + C ( 1 + ∣ x ∣ 2 + λ ) e − γ ( T − t ) . After decreasing γ \gamma γ if necessary, this is (98) .
Now we are ready to present the main result of this paper.
Theorem 5.10. Under the conditions of Theorem 5.5 , for any fixed initial state x x x and exploration weight λ > 0 \lambda>0 λ > 0 , there exist constants K = K x , λ > 0 K=K_{x,\lambda}>0 K = K x , λ > 0 and μ > 0 \mu>0 μ > 0 , independent of T , t T,t T , t , such that for all 0 ≤ t ≤ T 0\leq t\leq T 0 ≤ t ≤ T ,
E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 + E W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , X T , λ ( t ) ) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t ) ) ) ≤ K ( e − μ t + e − μ ( T − t ) ) . (99) \mathbb{E}|X^{T,\lambda}(t)-X^{*,\lambda}(t)|^2+\mathbb{E}W_2^2\left(\pi_T^{\lambda,*}(\cdot\mid t,X^{T,\lambda}(t)),\pi_\infty^{\lambda,*}(\cdot\mid X^{*,\lambda}(t))\right)
\leq K\left(e^{-\mu t}+e^{-\mu(T-t)}\right). \tag*{(99)} E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 + E W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , X T , λ ( t )) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t )) ) ≤ K ( e − μ t + e − μ ( T − t ) ) . (99) Proof. By Lemma 5.8 , we obtain
W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , X T , λ ( t ) ) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t ) ) ) ≤ 2 W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , X T , λ ( t ) ) , π ∞ λ , ∗ ( ⋅ ∣ X T , λ ( t ) ) ) + 2 W 2 2 ( π ∞ λ , ∗ ( ⋅ ∣ X T , λ ( t ) ) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t ) ) ) ≤ C ( 1 + ∣ X T , λ ( t ) ∣ 2 + λ ) e − 2 γ ( T − t ) + C ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 . \begin{aligned}
W_2^2\left(\pi_T^{\lambda,*}(\cdot\mid t,X^{T,\lambda}(t)),\pi_\infty^{\lambda,*}(\cdot\mid X^{*,\lambda}(t))\right)
&\leq2W_2^2\left(\pi_T^{\lambda,*}(\cdot\mid t,X^{T,\lambda}(t)),\pi_\infty^{\lambda,*}(\cdot\mid X^{T,\lambda}(t))\right)\\
&\quad+2W_2^2\left(\pi_\infty^{\lambda,*}(\cdot\mid X^{T,\lambda}(t)),\pi_\infty^{\lambda,*}(\cdot\mid X^{*,\lambda}(t))\right)\\
&\leq C(1+|X^{T,\lambda}(t)|^2+\lambda)e^{-2\gamma(T-t)}+C|X^{T,\lambda}(t)-X^{*,\lambda}(t)|^2.
\end{aligned} W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , X T , λ ( t )) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t )) ) ≤ 2 W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , X T , λ ( t )) , π ∞ λ , ∗ ( ⋅ ∣ X T , λ ( t )) ) + 2 W 2 2 ( π ∞ λ , ∗ ( ⋅ ∣ X T , λ ( t )) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t )) ) ≤ C ( 1 + ∣ X T , λ ( t ) ∣ 2 + λ ) e − 2 γ ( T − t ) + C ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 . The last inequality also uses the fact that the two infinite-horizon Gaussian distributions have the same covariance, so their Wasserstein distance equals the distance between their means. Then by (94) and Theorem 5.5 , we obtain
E W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , X T , λ ( t ) ) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t ) ) ) ≤ C ( 1 + E ∣ X T , λ ( t ) ∣ 2 + λ ) e − 2 γ ( T − t ) + C E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 ≤ K ( e − μ t + e − μ ( T − t ) ) , \begin{aligned}
\mathbb{E}W_2^2\left(\pi_T^{\lambda,*}(\cdot\mid t,X^{T,\lambda}(t)),\pi_\infty^{\lambda,*}(\cdot\mid X^{*,\lambda}(t))\right)
&\leq C(1+\mathbb{E}|X^{T,\lambda}(t)|^2+\lambda)e^{-2\gamma(T-t)}+C\mathbb{E}|X^{T,\lambda}(t)-X^{*,\lambda}(t)|^2\\
&\leq K\left(e^{-\mu t}+e^{-\mu(T-t)}\right),
\end{aligned} E W 2 2 ( π T λ , ∗ ( ⋅ ∣ t , X T , λ ( t )) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t )) ) ≤ C ( 1 + E ∣ X T , λ ( t ) ∣ 2 + λ ) e − 2 γ ( T − t ) + C E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 ≤ K ( e − μ t + e − μ ( T − t ) ) , for some K , μ > 0 K,\mu>0 K , μ > 0 . Combining this with the state-process estimate in Theorem 5.5 yields (99) .
In particular, by Proposition 5.1, π ∞ λ , ∗ ( ⋅ ∣ x ∗ ) = N ( u ∗ , Λ λ ) \pi_\infty^{\lambda,*}(\cdot\mid x_*)=\mathcal{N}(u_*,\Lambda^\lambda) π ∞ λ , ∗ ( ⋅ ∣ x ∗ ) = N ( u ∗ , Λ λ ) . Thus, when the reference state is at the static center, the above result is compatible with the static Gaussian optimal distribution constructed in § 3 .
Corollary 5.11. For any δ ∈ ( 0 , 1 / 2 ) \delta\in(0,1/2) δ ∈ ( 0 , 1/2 ) , there exist constants K = K x , λ , δ > 0 K = K_{x,\lambda,\delta} > 0 K = K x , λ , δ > 0 and μ = μ δ > 0 \mu= \mu_\delta> 0 μ = μ δ > 0 , independent of T , t T,t T , t , such that for any t ∈ [ δ T , ( 1 − δ ) T ] t \in[\delta T,(1-\delta)T] t ∈ [ δ T , ( 1 − δ ) T ] ,
E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 + W 2 2 ( π ∗ T , λ ( ⋅ ∣ t , X T , λ ( t ) ) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t ) ) ) ≤ K e − μ T . \mathbb{E}|X^{T,\lambda}(t)-X^{*,\lambda}(t)|^2+\mathcal{W}_2^2\left(\pi_*^{T,\lambda}(\cdot\mid t,X^{T,\lambda}(t)),\pi_\infty^{\lambda,*}(\cdot\mid X^{*,\lambda}(t))\right)\leq Ke^{-\mu T}. E ∣ X T , λ ( t ) − X ∗ , λ ( t ) ∣ 2 + W 2 2 ( π ∗ T , λ ( ⋅ ∣ t , X T , λ ( t )) , π ∞ λ , ∗ ( ⋅ ∣ X ∗ , λ ( t )) ) ≤ K e − μ T . Proof. If t ∈ [ δ T , ( 1 − δ ) T ] t \in[\delta T,(1-\delta)T] t ∈ [ δ T , ( 1 − δ ) T ] , then t ≥ δ T t \geq\delta T t ≥ δ T and T − t ≥ δ T T-t \geq\delta T T − t ≥ δ T . Using e − μ t ≤ e − μ δ T e^{-\mu t} \leq e^{-\mu\delta T} e − μ t ≤ e − μ δ T and e − μ ( T − t ) ≤ e − μ δ T e^{-\mu(T-t)} \leq e^{-\mu\delta T} e − μ ( T − t ) ≤ e − μ δ T in the state estimate (89) and the Wasserstein estimate (99) gives the result. ■
Proof of Theorem 2.18
Write ∥ z ∥ ∞ 2 = E ∫ 0 ∞ ∣ z ( t ) ∣ 2 d t \|z\|_{\infty}^2=\mathbb{E}\int_0^\infty|z(t)|^2\,dt ∥ z ∥ ∞ 2 = E ∫ 0 ∞ ∣ z ( t ) ∣ 2 d t .
( i ) ⇒ ( i i ) (i)\Rightarrow(ii) ( i ) ⇒ ( ii ) . We first construct P 1 P_1 P 1 . Set
α = 1 + ∥ Q − S T R − 1 S ∥ , Q 1 = S T R − 1 S + α I n , Q 2 = Q 1 − Q . \alpha=1+\|Q-S^{\mathsf{T}}R^{-1}S\|,\qquad Q_1=S^{\mathsf{T}}R^{-1}S+\alpha I_n,\qquad Q_2=Q_1-Q. α = 1 + ∥ Q − S T R − 1 S ∥ , Q 1 = S T R − 1 S + α I n , Q 2 = Q 1 − Q .
Then
Q = Q 1 − Q 2 , Q 2 ≥ I n , Q 1 − S T R − 1 S = α I n > 0. Q=Q_1-Q_2,\qquad Q_2\geq I_n,\qquad Q_1-S^{\mathsf{T}}R^{-1}S=\alpha I_n>0. Q = Q 1 − Q 2 , Q 2 ≥ I n , Q 1 − S T R − 1 S = α I n > 0.
For x ∈ R n x\in\mathbb{R}^n x ∈ R n , consider (12) with the auxiliary cost functional obtained by replacing Q Q Q with Q 1 Q_1 Q 1 :
J 1 , ∞ 0 ( x ; u ) = E ∫ 0 ∞ [ X ( t ) T Q 1 X ( t ) + 2 u ( t ) T S X ( t ) + u ( t ) T R u ( t ) ] d t , J_{1,\infty}^{0}(x;u)=\mathbb{E}\int_0^\infty\left[X(t)^{\mathsf{T}}Q_1X(t)+2u(t)^{\mathsf{T}}SX(t)+u(t)^{\mathsf{T}}Ru(t)\right]\,dt, J 1 , ∞ 0 ( x ; u ) = E ∫ 0 ∞ [ X ( t ) T Q 1 X ( t ) + 2 u ( t ) T S X ( t ) + u ( t ) T R u ( t ) ] d t ,
where u ∈ U a d ( x ) u\in\mathcal{U}_{\mathrm{ad}}(x) u ∈ U ad ( x ) . This auxiliary problem satisfies the strong standard condition. By the L 2 L^2 L 2 -stabilizability assumption and Sun–Wang–Yong [34 Proposition 3.1] , there exists a positive definite stabilizing solution P 1 P_1 P 1 of the auxiliary algebraic Riccati equation
A T P 1 + P 1 A + C T P 1 C + Q 1 = K 1 T M 1 K 1 . (100) A^{\mathsf{T}}P_1+P_1A+C^{\mathsf{T}}P_1C+Q_1=K_1^{\mathsf{T}}M_1K_1. \tag*{(100)} A T P 1 + P 1 A + C T P 1 C + Q 1 = K 1 T M 1 K 1 . (100)
where
M 1 = R + D T P 1 D > 0 , K 1 = M 1 − 1 ( B T P 1 + D T P 1 C + S ) , M_1=R+D^{\mathsf{T}}P_1D>0,\qquad K_1=M_1^{-1}(B^{\mathsf{T}}P_1+D^{\mathsf{T}}P_1C+S), M 1 = R + D T P 1 D > 0 , K 1 = M 1 − 1 ( B T P 1 + D T P 1 C + S ) ,
A 1 = A − B K 1 , C 1 = C − D K 1 , (101) A_1=A-BK_1,\qquad C_1=C-DK_1, \tag*{(101)} A 1 = A − B K 1 , C 1 = C − D K 1 , (101)
and [ A 1 , C 1 ] [A_1,C_1] [ A 1 , C 1 ] is mean-square exponentially stable.
Let u ( t ) = v ( t ) − K 1 X ( t ) u(t)=v(t)-K_1X(t) u ( t ) = v ( t ) − K 1 X ( t ) . Substitution into (12) gives
{ d X ( t ) = [ A 1 X ( t ) + B v ( t ) ] d t + [ C 1 X ( t ) + D v ( t ) ] d W ( t ) , t ≥ 0 , X ( 0 ) = x . (102) \begin{cases}
dX(t)=[A_1X(t)+Bv(t)]\,dt+[C_1X(t)+Dv(t)]\,dW(t), & t\geq0,\\
X(0)=x. \tag*{(102)}
\end{cases} { d X ( t ) = [ A 1 X ( t ) + B v ( t )] d t + [ C 1 X ( t ) + D v ( t )] d W ( t ) , X ( 0 ) = x . t ≥ 0 , (102)
Since [ A 1 , C 1 ] [A_1,C_1] [ A 1 , C 1 ] is L 2 L^2 L 2 -stable, every v ∈ L F 2 ( 0 , ∞ ; R m ) v\in L_F^2(0,\infty;\mathbb{R}^m) v ∈ L F 2 ( 0 , ∞ ; R m ) yields a state X ∈ L F 2 ( 0 , ∞ ; R n ) X\in L_F^2(0,\infty;\mathbb{R}^n) X ∈ L F 2 ( 0 , ∞ ; R n ) ; see Sun–Yong [35 ] . Consequently, u = v − K 1 X u=v-K_1X u = v − K 1 X belongs to U a d ( x ) \mathcal{U}_{\mathrm{ad}}(x) U ad ( x ) . Conversely, every u ∈ U a d ( x ) u\in\mathcal{U}_{\mathrm{ad}}(x) u ∈ U ad ( x ) and its corresponding state X X X determine v = u + K 1 X ∈ L F 2 ( 0 , ∞ ; R m ) v=u+K_1X\in L_F^2(0,\infty;\mathbb{R}^m) v = u + K 1 X ∈ L F 2 ( 0 , ∞ ; R m ) . These two transformations are inverses of each other.
Applying Itô’s formula to X ( t ) T P 1 X ( t ) X(t)^{\mathsf{T}}P_1X(t) X ( t ) T P 1 X ( t ) along the solution of (A.3) and using (A.1)–(A.2), we obtain
d d t E [ X ( t ) T P 1 X ( t ) ] = E [ X ( t ) T ( A T P 1 + P 1 A + C T P 1 C ) X ( t ) + 2 u ( t ) T ( B T P 1 + D T P 1 C ) X ( t ) + u ( t ) T D T P 1 D u ( t ) ] = E [ X ( t ) T ( K 1 T M 1 K 1 − Q 1 ) X ( t ) + 2 u ( t ) T ( M 1 K 1 − S ) X ( t ) + u ( t ) T ( M 1 − R ) u ( t ) ] = E [ ( u ( t ) + K 1 X ( t ) ) T M 1 ( u ( t ) + K 1 X ( t ) ) − X ( t ) T Q 1 X ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] = E [ v ( t ) T M 1 v ( t ) − X ( t ) T Q 1 X ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] . (103) \begin{aligned}
\frac{d}{dt}\mathbb{E}[X(t)^{\mathsf{T}}P_1X(t)]
={}&\mathbb{E}\big[X(t)^{\mathsf{T}}(A^{\mathsf{T}}P_1+P_1A+C^{\mathsf{T}}P_1C)X(t)\\
&\quad+2u(t)^{\mathsf{T}}(B^{\mathsf{T}}P_1+D^{\mathsf{T}}P_1C)X(t)+u(t)^{\mathsf{T}}D^{\mathsf{T}}P_1Du(t)\big]\\
={}&\mathbb{E}\big[X(t)^{\mathsf{T}}(K_1^{\mathsf{T}}M_1K_1-Q_1)X(t)+2u(t)^{\mathsf{T}}(M_1K_1-S)X(t)+u(t)^{\mathsf{T}}(M_1-R)u(t)\big]\\
={}&\mathbb{E}\big[(u(t)+K_1X(t))^{\mathsf{T}}M_1(u(t)+K_1X(t))-X(t)^{\mathsf{T}}Q_1X(t)\\
&\quad-2u(t)^{\mathsf{T}}SX(t)-u(t)^{\mathsf{T}}Ru(t)\big]\\
={}&\mathbb{E}\big[v(t)^{\mathsf{T}}M_1v(t)-X(t)^{\mathsf{T}}Q_1X(t)-2u(t)^{\mathsf{T}}SX(t)-u(t)^{\mathsf{T}}Ru(t)\big]. \tag*{(103)}
\end{aligned} d t d E [ X ( t ) T P 1 X ( t )] = = = = E [ X ( t ) T ( A T P 1 + P 1 A + C T P 1 C ) X ( t ) + 2 u ( t ) T ( B T P 1 + D T P 1 C ) X ( t ) + u ( t ) T D T P 1 D u ( t ) ] E [ X ( t ) T ( K 1 T M 1 K 1 − Q 1 ) X ( t ) + 2 u ( t ) T ( M 1 K 1 − S ) X ( t ) + u ( t ) T ( M 1 − R ) u ( t ) ] E [ ( u ( t ) + K 1 X ( t ) ) T M 1 ( u ( t ) + K 1 X ( t )) − X ( t ) T Q 1 X ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] E [ v ( t ) T M 1 v ( t ) − X ( t ) T Q 1 X ( t ) − 2 u ( t ) T S X ( t ) − u ( t ) T R u ( t ) ] . (103)
Substituting Q = Q 1 − Q 2 Q = Q_1 - Q_2 Q = Q 1 − Q 2 and integrating over [ 0 , T ] [0,T] [ 0 , T ] , we obtain
J T 0 ( x ; u ) + E [ X ( T ) ⊤ P 1 X ( T ) ] = x ⊤ P 1 x + E ∫ 0 T [ v ( t ) ⊤ M 1 v ( t ) − X ( t ) ⊤ Q 2 X ( t ) ] d t . (104) J_T^0(x;u) + \mathbb{E}[X(T)^\top P_1X(T)] = x^\top P_1x + \mathbb{E}\int_0^T [v(t)^\top M_1v(t) - X(t)^\top Q_2X(t)]\,\mathrm{d}t. \tag*{(104)} J T 0 ( x ; u ) + E [ X ( T ) ⊤ P 1 X ( T )] = x ⊤ P 1 x + E ∫ 0 T [ v ( t ) ⊤ M 1 v ( t ) − X ( t ) ⊤ Q 2 X ( t )] d t . (104)
Since [ A 1 , C 1 ] [A_1,C_1] [ A 1 , C 1 ] is L 2 L^2 L 2 -stable and v v v is square-integrable, we have E ∣ X ( T ) ∣ 2 → 0 \mathbb{E}|X(T)|^2 \to0 E ∣ X ( T ) ∣ 2 → 0 as T → ∞ T \to\infty T → ∞ . The square-integrability of X X X , u u u , and v v v also ensures that the running-cost integrals converge. Thus, letting T → ∞ T \to\infty T → ∞ in (A.5) yields
J ∞ 0 ( x ; u ) = x ⊤ P 1 x + J 2 , ∞ ( x ; v ) , (105) J_\infty^0(x;u) = x^\top P_1x + J_{2,\infty}(x;v), \tag*{(105)} J ∞ 0 ( x ; u ) = x ⊤ P 1 x + J 2 , ∞ ( x ; v ) , (105)
J 2 , ∞ ( x ; v ) : = E ∫ 0 ∞ [ v ( t ) ⊤ M 1 v ( t ) − X ( t ) ⊤ Q 2 X ( t ) ] d t . J_{2,\infty}(x;v) := \mathbb{E}\int_0^\infty[v(t)^\top M_1v(t) - X(t)^\top Q_2X(t)]\,\mathrm{d}t. J 2 , ∞ ( x ; v ) := E ∫ 0 ∞ [ v ( t ) ⊤ M 1 v ( t ) − X ( t ) ⊤ Q 2 X ( t )] d t .
Now we construct P 2 P_2 P 2 . Fix v ∈ L F 2 ( 0 , ∞ ; R m ) v \in L^2_{\mathbb{F}}(0,\infty;\mathbb{R}^m) v ∈ L F 2 ( 0 , ∞ ; R m ) , and let X X X solve (A.3) with X ( 0 ) = 0 X(0)=0 X ( 0 ) = 0 . By the preceding control transformation, u ( t ) = v ( t ) − K 1 X ( t ) u(t)=v(t)-K_1X(t) u ( t ) = v ( t ) − K 1 X ( t ) belongs to U a d ( 0 ) U_{\mathrm{ad}}(0) U ad ( 0 ) . The coercivity condition (16) and the cost decomposition (A.6) therefore give
J 2 , ∞ ( 0 ; v ) = J ∞ 0 ( 0 ; u ) ≥ δ ∥ X ∥ ∞ 2 . J_{2,\infty}(0;v) = J_\infty^0(0;u) \ge\delta\|X\|_\infty^2. J 2 , ∞ ( 0 ; v ) = J ∞ 0 ( 0 ; u ) ≥ δ ∥ X ∥ ∞ 2 .
Consequently,
λ min ( M 1 ) ∥ v ∥ ∞ 2 ≤ E ∫ 0 ∞ v ( t ) ⊤ M 1 v ( t ) d t = J 2 , ∞ ( 0 ; v ) + E ∫ 0 ∞ X ( t ) ⊤ Q 2 X ( t ) d t ≤ J 2 , ∞ ( 0 ; v ) + ∥ Q 2 ∥ ∥ X ∥ ∞ 2 ≤ ( 1 + ∥ Q 2 ∥ δ ) J 2 , ∞ ( 0 ; v ) . \begin{aligned}
\lambda_{\min}(M_1)\|v\|_\infty^2 \le\mathbb{E}\int_0^\infty v(t)^\top M_1v(t)\,\mathrm{d}t = J_{2,\infty}(0;v) + \mathbb{E}\int_0^\infty X(t)^\top Q_2X(t)\,\mathrm{d}t \\
&\le J_{2,\infty}(0;v) + \|Q_2\|\|X\|_\infty^2 \le\left(1+\frac{\|Q_2\|}{\delta}\right)J_{2,\infty}(0;v).
\end{aligned} λ m i n ( M 1 ) ∥ v ∥ ∞ 2 ≤ E ∫ 0 ∞ v ( t ) ⊤ M 1 v ( t ) d t = J 2 , ∞ ( 0 ; v ) + E ∫ 0 ∞ X ( t ) ⊤ Q 2 X ( t ) d t ≤ J 2 , ∞ ( 0 ; v ) + ∥ Q 2 ∥∥ X ∥ ∞ 2 ≤ ( 1 + δ ∥ Q 2 ∥ ) J 2 , ∞ ( 0 ; v ) .
It follows that
J 2 , ∞ ( 0 ; v ) ≥ δ λ min ( M 1 ) δ + ∥ Q 2 ∥ ∥ v ∥ ∞ 2 , ∀ v ∈ L F 2 ( 0 , ∞ ; R m ) . (106) J_{2,\infty}(0;v) \ge\frac{\delta\lambda_{\min}(M_1)}{\delta+\|Q_2\|}\|v\|_\infty^2,\qquad\forall v\in L^2_{\mathbb{F}}(0,\infty;\mathbb{R}^m). \tag*{(106)} J 2 , ∞ ( 0 ; v ) ≥ δ + ∥ Q 2 ∥ δ λ m i n ( M 1 ) ∥ v ∥ ∞ 2 , ∀ v ∈ L F 2 ( 0 , ∞ ; R m ) . (106)
For the system (A.3), the uncontrolled system [ A 1 , C 1 ] [A_1,C_1] [ A 1 , C 1 ] is L 2 L^2 L 2 -stable, and (A.7) establishes uniform convexity of the associated cost functional J 2 , ∞ J_{2,\infty} J 2 , ∞ . By Sun–Yong [35, Theorem 5.3], its algebraic Riccati equation admits a stabilizing solution P 2 ∈ S n P_2\in\mathbb{S}^n P 2 ∈ S n satisfying
A 1 ⊤ P 2 + P 2 A 1 + C 1 ⊤ P 2 C 1 − Q 2 = K 2 ⊤ M 2 K 2 , (107) A_1^\top P_2 + P_2A_1 + C_1^\top P_2C_1 - Q_2 = K_2^\top M_2K_2, \tag*{(107)} A 1 ⊤ P 2 + P 2 A 1 + C 1 ⊤ P 2 C 1 − Q 2 = K 2 ⊤ M 2 K 2 , (107)
where
M 2 = M 1 + D ⊤ P 2 D > 0 , K 2 = M 2 − 1 ( B ⊤ P 2 + D ⊤ P 2 C 1 ) , (108) M_2 = M_1 + D^\top P_2D > 0,\qquad K_2 = M_2^{-1}(B^\top P_2 + D^\top P_2C_1), \tag*{(108)} M 2 = M 1 + D ⊤ P 2 D > 0 , K 2 = M 2 − 1 ( B ⊤ P 2 + D ⊤ P 2 C 1 ) , (108)
and [ A 1 − B K 2 , C 1 − D K 2 ] [A_1-BK_2,C_1-DK_2] [ A 1 − B K 2 , C 1 − D K 2 ] is L 2 L^2 L 2 -stable.
We now combine (A.1) and (A.8). Set P = P 1 + P 2 P=P_1+P_2 P = P 1 + P 2 and K = K 1 + K 2 K=K_1+K_2 K = K 1 + K 2 . Using (A.2), we obtain
A ⊤ P 2 + P 2 A + C ⊤ P 2 C = A 1 ⊤ P 2 + P 2 A 1 + C 1 ⊤ P 2 C 1 + K 1 ⊤ B ⊤ P 2 + P 2 B K 1 + K 1 ⊤ D ⊤ P 2 C 1 + C 1 ⊤ P 2 D K 1 + K 1 ⊤ D ⊤ P 2 D K 1 = Q 2 + K 2 ⊤ M 2 K 2 + K 1 ⊤ ( B ⊤ P 2 + D ⊤ P 2 C 1 ) + ( P 2 B + C 1 ⊤ P 2 D ) K 1 + K 1 ⊤ D ⊤ P 2 D K 1 = Q 2 + K 2 ⊤ M 2 K 2 + K 1 ⊤ M 2 K 2 + K 2 ⊤ M 2 K 1 + K 1 ⊤ D ⊤ P 2 D K 1 . \begin{aligned}
A^\top P_2 + P_2A + C^\top P_2C
&= A_1^\top P_2 + P_2A_1 + C_1^\top P_2C_1 \\
&\quad+ K_1^\top B^\top P_2 + P_2BK_1 + K_1^\top D^\top P_2C_1 + C_1^\top P_2DK_1 + K_1^\top D^\top P_2DK_1 \\
&= Q_2 + K_2^\top M_2K_2 + K_1^\top(B^\top P_2+D^\top P_2C_1) + (P_2B+C_1^\top P_2D)K_1 + K_1^\top D^\top P_2DK_1 \\
&= Q_2 + K_2^\top M_2K_2 + K_1^\top M_2K_2 + K_2^\top M_2K_1 + K_1^\top D^\top P_2DK_1.
\end{aligned} A ⊤ P 2 + P 2 A + C ⊤ P 2 C = A 1 ⊤ P 2 + P 2 A 1 + C 1 ⊤ P 2 C 1 + K 1 ⊤ B ⊤ P 2 + P 2 B K 1 + K 1 ⊤ D ⊤ P 2 C 1 + C 1 ⊤ P 2 D K 1 + K 1 ⊤ D ⊤ P 2 D K 1 = Q 2 + K 2 ⊤ M 2 K 2 + K 1 ⊤ ( B ⊤ P 2 + D ⊤ P 2 C 1 ) + ( P 2 B + C 1 ⊤ P 2 D ) K 1 + K 1 ⊤ D ⊤ P 2 D K 1 = Q 2 + K 2 ⊤ M 2 K 2 + K 1 ⊤ M 2 K 2 + K 2 ⊤ M 2 K 1 + K 1 ⊤ D ⊤ P 2 D K 1 .
Adding (A.1) and using Q = Q 1 − Q 2 Q=Q_1-Q_2 Q = Q 1 − Q 2 , we obtain
A ⊤ P + P A + C ⊤ P C + Q = K 1 ⊤ ( M 1 + D ⊤ P 2 D ) K 1 + K 1 ⊤ M 2 K 2 + K 2 ⊤ M 2 K 1 + K 2 ⊤ M 2 K 2 = ( K 1 + K 2 ) ⊤ M 2 ( K 1 + K 2 ) = K ⊤ M 2 K . (109) \begin{aligned}
A^\top P + PA + C^\top PC + Q
&= K_1^\top(M_1+D^\top P_2D)K_1 + K_1^\top M_2K_2 + K_2^\top M_2K_1 + K_2^\top M_2K_2 \\
&= (K_1+K_2)^\top M_2(K_1+K_2)=K^\top M_2K. \tag*{(109)}
\end{aligned} A ⊤ P + P A + C ⊤ P C + Q = K 1 ⊤ ( M 1 + D ⊤ P 2 D ) K 1 + K 1 ⊤ M 2 K 2 + K 2 ⊤ M 2 K 1 + K 2 ⊤ M 2 K 2 = ( K 1 + K 2 ) ⊤ M 2 ( K 1 + K 2 ) = K ⊤ M 2 K . (109)
Moreover,
R + D ⊤ P D = R + D ⊤ P 1 D + D ⊤ P 2 D = M 2 > 0 , R+D^\top PD=R+D^\top P_1D+D^\top P_2D=M_2>0, R + D ⊤ P D = R + D ⊤ P 1 D + D ⊤ P 2 D = M 2 > 0 ,
B ⊤ P + D ⊤ P C + S = M 1 K 1 + B ⊤ P 2 + D ⊤ P 2 ( C 1 + D K 1 ) = M 1 K 1 + M 2 K 2 + D ⊤ P 2 D K 1 = ( M 1 + D ⊤ P 2 D ) K 1 + M 2 K 2 = M 2 K , \begin{aligned}
B^\top P+D^\top PC+S
&=M_1K_1+B^\top P_2+D^\top P_2(C_1+DK_1) \\
&=M_1K_1+M_2K_2+D^\top P_2DK_1 \\
&=(M_1+D^\top P_2D)K_1+M_2K_2=M_2K,
\end{aligned} B ⊤ P + D ⊤ P C + S = M 1 K 1 + B ⊤ P 2 + D ⊤ P 2 ( C 1 + D K 1 ) = M 1 K 1 + M 2 K 2 + D ⊤ P 2 D K 1 = ( M 1 + D ⊤ P 2 D ) K 1 + M 2 K 2 = M 2 K ,
K = ( R + D ⊤ P D ) − 1 ( B ⊤ P + D ⊤ P C + S ) . (110) K=(R+D^\top PD)^{-1}(B^\top P+D^\top PC+S). \tag*{(110)} K = ( R + D ⊤ P D ) − 1 ( B ⊤ P + D ⊤ P C + S ) . (110)
Substituting the expressions for M 2 M_2 M 2 and K K K in (A.11) into (A.10) yields (22) for P P P . Note that
A − B K = A 1 − B K 2 , C − D K = C 1 − D K 2 , A-BK=A_1-BK_2,\qquad C-DK=C_1-DK_2, A − B K = A 1 − B K 2 , C − D K = C 1 − D K 2 ,
so [ A − B K , C − D K ] [A-BK,C-DK] [ A − B K , C − D K ] is L 2 L^2 L 2 -stable and hence mean-square exponentially stable. Taking P ∞ = P P_\infty=P P ∞ = P and K ∞ = K K_\infty=K K ∞ = K proves (ii).
(ii) ⇒ \Rightarrow ⇒ (iii). Fix x ∈ R n x\in\mathbb{R}^n x ∈ R n and u ∈ U a d ( x ) u\in\mathcal{U}_{\mathrm{ad}}(x) u ∈ U ad ( x ) , and let X X X solve (12) . Set v ( t ) = u ( t ) + K ∞ X ( t ) v(t)=u(t)+K_\infty X(t) v ( t ) = u ( t ) + K ∞ X ( t ) . Then the state equation can be written as
d X ( t ) = [ ( A − B K ∞ ) X ( t ) + B v ( t ) ] d t + [ ( C − D K ∞ ) X ( t ) + D v ( t ) ] d W ( t ) , t ≥ 0 , X ( 0 ) = x . \begin{aligned}
dX(t)&=\left[(A-BK_\infty)X(t)+Bv(t)\right]dt\\
&\quad+\left[(C-DK_\infty)X(t)+Dv(t)\right]dW(t),\qquad t\geq0,\\
X(0)&=x.
\end{aligned} d X ( t ) X ( 0 ) = [ ( A − B K ∞ ) X ( t ) + B v ( t ) ] d t + [ ( C − D K ∞ ) X ( t ) + D v ( t ) ] d W ( t ) , t ≥ 0 , = x .
Since u ∈ U a d ( x ) u\in\mathcal{U}_{\mathrm{ad}}(x) u ∈ U ad ( x ) , both X X X and u u u are square-integrable on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) , and hence v ∈ L F 2 ( 0 , ∞ ; R m ) v\in L^2_{\mathbb{F}}(0,\infty;\mathbb{R}^m) v ∈ L F 2 ( 0 , ∞ ; R m ) . The L 2 L^2 L 2 -stability of [ A − B K ∞ , C − D K ∞ ] [A-BK_\infty,C-DK_\infty] [ A − B K ∞ , C − D K ∞ ] , together with the square-integrability of v v v , implies that lim T → ∞ E ∣ X ( T ) ∣ 2 = 0 \lim_{T\to\infty}\mathbb{E}|X(T)|^2=0 lim T → ∞ E ∣ X ( T ) ∣ 2 = 0 . Applying Itô’s formula as in (A.4) and using (22) –(23) , we obtain
d d t E [ X ( t ) ⊤ P ∞ X ( t ) ] + E [ X ( t ) ⊤ Q X ( t ) + 2 u ( t ) ⊤ S X ( t ) + u ( t ) ⊤ R u ( t ) ] = E [ v ( t ) ⊤ M ∞ v ( t ) ] . \frac{d}{dt}\mathbb{E}[X(t)^\top P_\infty X(t)]+\mathbb{E}\left[X(t)^\top QX(t)+2u(t)^\top SX(t)+u(t)^\top Ru(t)\right]=\mathbb{E}[v(t)^\top M_\infty v(t)]. d t d E [ X ( t ) ⊤ P ∞ X ( t )] + E [ X ( t ) ⊤ QX ( t ) + 2 u ( t ) ⊤ S X ( t ) + u ( t ) ⊤ R u ( t ) ] = E [ v ( t ) ⊤ M ∞ v ( t )] .
Integrating over [ 0 , T ] [0,T] [ 0 , T ] and letting T → ∞ T\to\infty T → ∞ yields
J ∞ 0 ( x ; u ) = x ⊤ P ∞ x + E ∫ 0 ∞ v ( t ) ⊤ M ∞ v ( t ) d t . (111) J^0_\infty(x;u)=x^\top P_\infty x+\mathbb{E}\int_0^\infty v(t)^\top M_\infty v(t)\,dt. \tag*{(111)} J ∞ 0 ( x ; u ) = x ⊤ P ∞ x + E ∫ 0 ∞ v ( t ) ⊤ M ∞ v ( t ) d t . (111)
Now take x = 0 x=0 x = 0 . Applying Lemma 2.7 to the stable closed-loop system and letting the time horizon tend to infinity, we obtain a constant C > 0 C>0 C > 0 , independent of u u u , such that
∥ X ∥ ∞ 2 ≤ C ∥ v ∥ ∞ 2 . \|X\|_\infty^2\leq C\|v\|_\infty^2. ∥ X ∥ ∞ 2 ≤ C ∥ v ∥ ∞ 2 .
Since u ( t ) = v ( t ) − K ∞ X ( t ) u(t)=v(t)-K_\infty X(t) u ( t ) = v ( t ) − K ∞ X ( t ) , we have
∥ u ∥ ∞ 2 ≤ 2 ∥ v ∥ ∞ 2 + 2 ∥ K ∞ ∥ 2 ∥ X ∥ ∞ 2 ≤ ( 2 + 2 C ∥ K ∞ ∥ 2 ) ∥ v ∥ ∞ 2 . \|u\|_\infty^2\leq2\|v\|_\infty^2+2\|K_\infty\|^2\|X\|_\infty^2\leq(2+2C\|K_\infty\|^2)\|v\|_\infty^2. ∥ u ∥ ∞ 2 ≤ 2∥ v ∥ ∞ 2 + 2∥ K ∞ ∥ 2 ∥ X ∥ ∞ 2 ≤ ( 2 + 2 C ∥ K ∞ ∥ 2 ) ∥ v ∥ ∞ 2 .
Combining these estimates with (A.12) and M ∞ > 0 M_\infty>0 M ∞ > 0 , we obtain
J ∞ 0 ( 0 ; u ) ≥ λ min ( M ∞ ) ∥ v ∥ ∞ 2 ≥ λ min ( M ∞ ) 2 + C ( 1 + 2 ∥ K ∞ ∥ 2 ) ( ∥ X ∥ ∞ 2 + ∥ u ∥ ∞ 2 ) . J^0_\infty(0;u)\geq\lambda_{\min}(M_\infty)\|v\|_\infty^2\geq\frac{\lambda_{\min}(M_\infty)}{2+C(1+2\|K_\infty\|^2)}\left(\|X\|_\infty^2+\|u\|_\infty^2\right). J ∞ 0 ( 0 ; u ) ≥ λ m i n ( M ∞ ) ∥ v ∥ ∞ 2 ≥ 2 + C ( 1 + 2∥ K ∞ ∥ 2 ) λ m i n ( M ∞ ) ( ∥ X ∥ ∞ 2 + ∥ u ∥ ∞ 2 ) .
This proves (ii) ⇒ \Rightarrow ⇒ (iii).
The implication (iii) ⇒ \Rightarrow ⇒ (i) follows immediately by dropping the nonnegative control term from the lower bound in (iii).
Under these equivalent conditions, (A.12) and M ∞ > 0 M_\infty>0 M ∞ > 0 imply that
J ∞ 0 ( x ; u ) ≥ x ⊤ P ∞ x , ∀ u ∈ U a d ( x ) , J^0_\infty(x;u)\geq x^\top P_\infty x,\qquad\forall u\in\mathcal{U}_{\mathrm{ad}}(x), J ∞ 0 ( x ; u ) ≥ x ⊤ P ∞ x , ∀ u ∈ U ad ( x ) ,
with equality if and only if u = − K ∞ X u=-K_\infty X u = − K ∞ X . The mean-square exponential stability of the closed-loop system ensures that this feedback control is admissible. Thus (25) gives the optimal value and the unique optimal control for each initial state x x x .
To prove uniqueness of the stabilizing solution, let P ^ ∈ S n \widehat{P}\in\mathbb{S}^n P ∈ S n be another solution of (22) such that R + D ⊤ P ^ D > 0 R+D^\top\widehat{P}D>0 R + D ⊤ P D > 0 and the associated closed-loop system is mean-square exponentially stable. Applying the same verification argument to P ^ \widehat{P} P yields
x ⊤ P ^ x = V ∞ 0 ( x ) = x ⊤ P ∞ x , ∀ x ∈ R n . x^\top\widehat{P}x=V^0_\infty(x)=x^\top P_\infty x,\qquad\forall x\in\mathbb{R}^n. x ⊤ P x = V ∞ 0 ( x ) = x ⊤ P ∞ x , ∀ x ∈ R n .
Since both matrices are symmetric, this identity implies P ^ = P ∞ \widehat{P}=P_\infty P = P ∞ .
Acknowledgments
YH is supported in part by National Key Research and Development Program of China (2025YFA1016903), National Natural Science Foundation of China (12571327, 12426311), Project of Educational Commission of Guangdong Province (2025ZDZX2059), Shenzhen Science and Technology Program (RCJC20221008092753082). HW is supported in part by National Natural Science Foundation of China (12522121, 12631016). JX is supported in part by National Key Research and Development Program of China (2022YFA1006102) and National Natural Science Foundation of China (12471418).
Statement on AI use: This manuscript was written entirely by the authors. AI-assisted tools were used solely for language polishing and grammatical correction, and not for generating scientific content.