Introduction
In sequential control problems, feedback allows a controller to adapt its actions based on observations collected over time, rather than committing to an action sequence in advance. In partially observed control, the controller must choose actions based on incomplete information about the state. Moreover, an action may influence not only the state, but also the quality of future observations. This creates a dual effect Bar-Shalom and Tse, [3 ] : a controller’s action affects both its immediate performance and the information available for future decisions. Consequently, an action that is not immediately optimal for control may be worthwhile to improve future decision-making. In the classical linear-quadratic-Gaussian (LQG) problem this tradeoff does not arise due to action-independent observations, and the separation principle allows estimation and control to be treated separately [26 , 2 ] . When observations are action-dependent, this separation can fail [22 ] , raising the question of when the benefit of actively acquiring information is large enough to justify its cost.
The informational role of actions is also central in sequential decision-making. In regret-minimization bandits, performance is measured by cumulative sub-optimality relative to a benchmark, and actions balance immediate reward against acquiring information useful for future decisions [5 ] . In pure exploration problems such as best-arm identification, the sampling strategy is designed directly around collecting enough information to distinguish among alternatives [8 ] . Interestingly, in some of these problems, strong guarantees can be achieved using a sampling allocation fixed in advance, without adapting to observations [18 ] . This raises a broader question in sequential decision-making: when is a policy fixed in advance sufficient, and when does adaptation to observations provide a genuine advantage?
In dynamic or stateful environments, this distinction also determines which policy class provides an appropriate benchmark for regret analysis. Benchmarks may range from fixed action policies, to an open-loop action sequence chosen in advance, to a fully history-dependent closed-loop policy. These choices can lead to substantially different notions of performance. For example, in restless bandits, weak regret compares against a fixed-arm benchmark [25 ] , whereas strong regret compares against an optimal closed-loop policy [24 , 19 ] . Optimal closed-loop policies, however, can be hard to compute. In some structured settings, this difficulty can be avoided by weakening the benchmark: when state evolution is deterministic, the state trajectory is determined by the action sequence, so an open-loop benchmark suffices. This is the case in several stateful bandit models with deterministic state evolution [16 , 14 , 21 , 4 , 13 ] . Thus, whether adaptation provides a meaningful advantage depends on the structure of the problem.
Related questions about benchmark choice arise in recent works on bandits with latent dynamics. Choi et al. [7 ] study a non-stationary bandit problem with latent linear dynamics and bilinear rewards, measuring regret against an optimal open-loop action sequence. Hwang et al. [11 ] develop an adaptive optimism-based algorithm for this model while retaining the same open-loop benchmark. Other dynamic learning and online-control formulations make different comparator choices. One common notion in non-stationary bandits is dynamic regret, which compares against a time-varying oracle [17 ] ; related formulations have also been studied in linear control [9 ] . In online and non-stochastic control, regret is often instead measured relative to a restricted class of feedback-controllers [1 , 23 ] . Together, these formulations emphasize that there is no universal choice of benchmark for sequential problems with dynamics.
In partially observed dynamical systems, this benchmark question leads to several natural policy classes with different uses of feedback. Open-loop control commits to an action sequence without adapting to observations, while certainty-equivalent (CE) control responds to observations through a state estimate, applying the fully observed controller as if that estimate were the true state. The optimal policy may instead depend on the full information state, and explicitly account for how actions affect future information. While CE is known to be optimal in the classical LQG setting [15 ] , recent work shows that CE can be suboptimal in linear-quadratic control with bilinear observations [22 ] . However, this result does not characterize how the resulting performance gap scales with the horizon. We ask when the performance gaps between open-loop, CE, and optimal control grow with the horizon.
We study this question in the scalar linear-quadratic setting with bilinear observations, where observation quality depends on the action. This structure creates a failure mode for CE: when the state estimate is small, CE takes a small action, resulting in weak observations and little new information. We identify an instance-dependent information strength parameter q q q that captures how strongly information acquired under CE contributes to the growth rate of the state estimate, and hence to the informativeness of future observations. We show that the relative performance of open-loop, CE, and optimal control varies qualitatively across different ranges of q q q . Our contributions are as follows.
We establish four instance-dependent thresholds q 1 < q 2 < q 3 < q 4 q_1 < q_2 < q_3 < q_4 q 1 < q 2 < q 3 < q 4 that separate different performance regimes. When q < q 1 q < q_1 q < q 1 no policy can achieve more than an O ( 1 ) O(1) O ( 1 ) advantage over open-loop control. When q < q 3 q < q_3 q < q 3 , CE achieves at most an O ( 1 ) O(1) O ( 1 ) advantage over open-loop control. Conversely, when q > q 4 q > q_4 q > q 4 , CE achieves an Ω ( T ) \Omega(T) Ω ( T ) advantage over open-loop control.
Our main finding is that when q > q 2 q > q_2 q > q 2 optimal control has an Ω ( T ) \Omega(T) Ω ( T ) advantage over open-loop control. Consequently, when q 2 < q < q 3 q_2 < q < q_3 q 2 < q < q 3 , the performance gap between optimal control and CE control is Ω ( T ) \Omega(T) Ω ( T ) . To establish this result, we construct a policy that occasionally deviates from CE to probe for information and show that when q > q 2 q > q_2 q > q 2 it achieves an Ω ( T ) \Omega(T) Ω ( T ) advantage over open-loop control.
Table Table 1 summarizes the established bounds.
Table 1. R T ( π , π ′ ) R_T(\pi,\pi') R T ( π , π ′ ) refers to the regret of policy π ′ \pi' π ′ with respect to π \pi π as the benchmark. π O L \pi^{OL} π O L , π C E \pi^{CE} π C E , and π ⋆ \pi^\star π ⋆ refer to the optimal OL, CE, and optimal policies, respectively.
Preliminaries
Notation. We denote the positive integers, integers, and nonnegative integers by N \mathbb{N} N , Z \mathbb{Z} Z , and Z ≥ 0 \mathbb{Z}_{\ge0} Z ≥ 0 , respectively. For T ∈ Z ≥ 0 T \in\mathbb{Z}_{\ge0} T ∈ Z ≥ 0 , let [ T ] : = { 0 , 1 , … , T } [T] := \{0, 1, \ldots, T\} [ T ] := { 0 , 1 , … , T } . For a set A A A , let Δ ( A ) \Delta(A) Δ ( A ) denote the set of probability distributions over A A A , and let 1 { E } \mathbb{1}\{E\} 1 { E } denote the indicator of an event E E E . For a matrix A A A , tr ( A ) \operatorname{tr}(A) tr ( A ) , ρ ( A ) \rho(A) ρ ( A ) , and ∥ A ∥ \lVert A\rVert ∥ A ∥ denote its trace, spectral radius, and spectral norm, respectively, while ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 denotes the Euclidean norm. For symmetric matrices A A A and B B B , we write A ⪰ B A \succeq B A ⪰ B (resp. A ≻ B A \succ B A ≻ B ) if A − B A-B A − B is positive semidefinite (resp. positive definite). Finally, X = d Y X \overset{d}{=} Y X = d Y denotes equality in distribution.
General problem setting. We first introduce a general partially observable control setting to define the three policies being compared. A partially observable control problem is defined by ( X , U , Y , f , g , c , μ , T ) (\mathcal{X},\mathcal{U},\mathcal{Y},f,g,c,\mu,T) ( X , U , Y , f , g , c , μ , T ) , where X , U , Y \mathcal{X},\mathcal{U},\mathcal{Y} X , U , Y are the state, action, and observation spaces respectively, f : X × U → Δ ( X ) f:\mathcal{X}\times\mathcal{U}\to\Delta(\mathcal{X}) f : X × U → Δ ( X ) is the state transition kernel, g : X × U → Δ ( Y ) g:\mathcal{X}\times\mathcal{U}\to\Delta(\mathcal{Y}) g : X × U → Δ ( Y ) is the observation kernel, c : X × U → R c:\mathcal{X}\times\mathcal{U}\to\mathbb{R} c : X × U → R is the stage cost, μ ∈ Δ ( X ) \mu\in\Delta(\mathcal{X}) μ ∈ Δ ( X ) is the initial state distribution, and T T T is the horizon. Interaction is as follows. We start at initial state x 0 ∼ μ ( ⋅ ) x_0\sim\mu(\cdot) x 0 ∼ μ ( ⋅ ) , which is unobserved. Then at each time step t = 0 , … , T t=0,\ldots,T t = 0 , … , T , a controller takes action u t u_t u t based on previous observations, incurs cost c ( x t , u t ) c(x_t,u_t) c ( x t , u t ) , and obtains observation y t ∼ g ( ⋅ ∣ x t , u t ) y_t\sim g(\cdot\mid x_t,u_t) y t ∼ g ( ⋅ ∣ x t , u t ) . The state of the system transitions to x t + 1 ∼ f ( ⋅ ∣ x t , u t ) x_{t+1}\sim f(\cdot\mid x_t,u_t) x t + 1 ∼ f ( ⋅ ∣ x t , u t ) .
Policies. We define the history of observations available when choosing u t u_t u t as
H t : = ( u 0 , y 0 , … , u t − 1 , y t − 1 ) H_t := (u_0,y_0,\ldots,u_{t-1},y_{t-1}) H t := ( u 0 , y 0 , … , u t − 1 , y t − 1 )
with H 0 = ∅ H_0 = \emptyset H 0 = ∅ . In particular, y t y_t y t is observed only after u t u_t u t has been chosen. Let H t = ( U × Y ) t \mathcal{H}_t = (\mathcal{U} \times\mathcal{Y})^t H t = ( U × Y ) t denote the space of histories. Let Π \Pi Π denote the class of policies π = { π t } t = 0 T \pi= \{\pi_t\}_{t=0}^T π = { π t } t = 0 T where each π t : H t → U \pi_t : \mathcal{H}_t \to\mathcal{U} π t : H t → U is measurable and u t = π t ( H t ) u_t = \pi_t(H_t) u t = π t ( H t ) . We call a policy open-loop if its actions do not depend on the history [15 ] . Formally, we define the class of open-loop policies
Π O L : = { π ∈ Π : π t ( H ) = π t ( H ′ ) , ∀ H , H ′ ∈ H t , t ∈ [ T ] } . \Pi^{OL} := \{\pi\in\Pi: \pi_t(H) = \pi_t(H'),\ \forall H,H' \in\mathcal{H}_t,\ t \in[T]\}. Π O L := { π ∈ Π : π t ( H ) = π t ( H ′ ) , ∀ H , H ′ ∈ H t , t ∈ [ T ]} .
Equivalently, open-loop policies commit to a deterministic action sequence before interaction begins. For a policy π \pi π , let E π \mathbb{E}_{\pi} E π denote expectation with respect to the trajectory distribution induced by the initial distribution μ \mu μ , transition kernel f f f , observation kernel g g g , and actions selected according to π \pi π .
Objectives. We define the expected cumulative cost of a policy π ∈ Π \pi\in\Pi π ∈ Π as
J T ( π ) = E π [ ∑ t = 0 T c ( x t , u t ) ] . J_T(\pi) = \mathbb{E}_{\pi}\left[\sum_{t=0}^{T} c(x_t,u_t)\right]. J T ( π ) = E π [ t = 0 ∑ T c ( x t , u t ) ] .
We adopt the convention that a policy designer aims to minimize J T J_T J T . For π , π ′ \pi,\pi' π , π ′ we define the regret of π ′ \pi' π ′ with respect to π \pi π as
R T ( π , π ′ ) = J T ( π ′ ) − J T ( π ) . R_T(\pi,\pi') = J_T(\pi') - J_T(\pi). R T ( π , π ′ ) = J T ( π ′ ) − J T ( π ) .
Alternatively, R T R_T R T can be considered the advantage of π \pi π over π ′ \pi' π ′ .
State-action cost-to-go. In order to define CE we consider the corresponding fully observed problem where the state is known. For each t t t , define the state-action cost-to-go function Q ~ t : X × U → R \widetilde{Q}_t : \mathcal{X} \times\mathcal{U} \to\mathbb{R} Q t : X × U → R recursively by:
Q ~ T ( x , u ) = c ( x , u ) \widetilde{Q}_T(x,u) = c(x,u) Q T ( x , u ) = c ( x , u )
and for each t ∈ [ T − 1 ] t \in[T-1] t ∈ [ T − 1 ] ,
Q ~ t ( x , u ) = c ( x , u ) + E x ′ ∼ f ( ⋅ ∣ x , u ) [ min u ′ ∈ U Q ~ t + 1 ( x ′ , u ′ ) ] . \widetilde{Q}_t(x,u) = c(x,u) + \mathbb{E}_{x' \sim f(\cdot\mid x,u)}\left[\min_{u' \in\mathcal{U}} \widetilde{Q}_{t+1}(x',u')\right]. Q t ( x , u ) = c ( x , u ) + E x ′ ∼ f ( ⋅ ∣ x , u ) [ u ′ ∈ U min Q t + 1 ( x ′ , u ′ ) ] .
Then,
arg min u ∈ U Q ~ t ( x , u ) \underset{u \in\mathcal{U}}{\arg\min}\ \widetilde{Q}_t(x,u) u ∈ U arg min Q t ( x , u )
denotes the set of optimal actions at time step t t t and state x x x in the fully observed problem. The CE policy will apply this fully observed decision rule to a state estimate.
State estimation. The final ingredient needed to define CE is the state estimate. Because the system is only partially observed, the controller generally does not know x t x_t x t when choosing u t u_t u t and must instead estimate it from the history. We use the conditional mean
x ^ t : = E [ x t ∣ H t ] \hat{x}_t := \mathbb{E}[x_t \mid H_t] x ^ t := E [ x t ∣ H t ]
as the state estimate. More generally, CE control may be defined with an alternative state estimator when the conditional mean is unavailable or does not belong to the state space.
Benchmark policies. Now we define the benchmark policies that we consider in this paper.
The optimal policy π ⋆ \pi^\star π ⋆ is defined as
π ⋆ ∈ arg min π ∈ Π J T ( π ) \pi^\star\in\arg\min_{\pi\in\Pi} J_T(\pi) π ⋆ ∈ arg π ∈ Π min J T ( π )
The certainty-equivalent (CE) policy π C E \pi^{CE} π C E applies the optimal fully observed policy to the estimated state x ^ t \hat{x}_t x ^ t and is defined as
π t C E ( H t ) ∈ arg min u Q ~ t ( x ^ t , u ) , ∀ t \pi_t^{CE}(H_t) \in\arg\min_u \widetilde{Q}_t(\hat{x}_t,u), \ \forall t π t C E ( H t ) ∈ arg u min Q t ( x ^ t , u ) , ∀ t
The optimal open-loop (OL) policy π O L \pi^{OL} π O L is
π O L ∈ arg min π ∈ Π O L J T ( π ) \pi^{OL} \in\arg\min_{\pi\in\Pi^{OL}} J_T(\pi) π O L ∈ arg π ∈ Π O L min J T ( π )
Linear-Gaussian Control With Bilinear Observation
In this section, we introduce the setting we analyze. We first summarize the main performance regimes and explain the mechanism underlying them. We then introduce the linear-quadratic Gaussian setting with bilinear observations, the associated Kalman filter, and the corresponding CE and OL policies.
Overview of the Main Results
Before introducing our main result, we begin with the following theorem about the classical LQG setting with action-independent observations.
Theorem 1. In the LQG setting, we have
(a) π ⋆ = π C E \pi^\star= \pi^{CE} π ⋆ = π C E
(b) R T ( π ⋆ , π O L ) = Ω ( T ) R_T(\pi^\star,\pi^{OL}) = \Omega(T) R T ( π ⋆ , π O L ) = Ω ( T ) for all (non-degenerate) instances
Theorem 1(a) follows from the classical separation principle (Witsenhausen, [26 ] ), while Theorem 1(b) follows from action-independent observations, and its proof is deferred to Appendix § 6 . Theorem 1 shows that in LQG, feedback provides a linear advantage over open-loop control, and since π ⋆ = π C E \pi^\star= \pi^{CE} π ⋆ = π C E , CE fully realizes this advantage. However, under action-dependent feedback, different relationships between open-loop, CE, and optimal control can arise. The following theorem summarizes the three possibilities that we realize in our bilinear feedback setting.
Theorem 2. For each case below, there exists a partially observed control problem satisfying the bounds:
We show that all three cases in Theorem 2 arise within the bilinear observation model introduced in § 3.2 . This sharply contrasts with LQG. Theorem 2(i) implies there exist instances in which optimal control fails to achieve more than a constant advantage over open-loop. More strikingly, Theorem 2(ii) implies feedback can provide a linear advantage over open-loop control, but CE fails to realize it. Theorem 2(iii) recovers the LQG behavior in which CE obtains a linear advantage over open-loop control.
In § 3.4 we quantify the strength of information by an instance-dependent parameter q q q , and § 4 gives sufficient conditions on q q q for realizing the three cases above.
Model and Kalman Filtering
Our Model. We study a linear-quadratic Gaussian setting with bilinear feedback. This is instantiated as follows. Let X = R n \mathcal{X} = \mathbb{R}^{n} X = R n , U = R p \mathcal{U} = \mathbb{R}^{p} U = R p , and Y = R \mathcal{Y} = \mathbb{R} Y = R . An optimal control problem is specified by ( A , B , C , Σ w , σ z 2 , Q , R , S , μ x ) (A, B, C, \Sigma_{w}, \sigma_{z}^{2}, Q, R, S, \mu_{x}) ( A , B , C , Σ w , σ z 2 , Q , R , S , μ x ) through
f ( ⋅ ∣ x , u ) = N ( A x + B u , Σ w ) , g ( ⋅ ∣ x , u ) = N ( u T C x , σ z 2 ) , c ( x , u ) = x T Q x + u T R u + 2 u T S x , μ ( ⋅ ) = N ( μ x , Σ ˉ ) . \begin{aligned}
f(\cdot\mid x,u) &= \mathcal{N}(Ax + Bu, \Sigma_{w}),\\
g(\cdot\mid x,u) &= \mathcal{N}(u^{\mathsf{T}}Cx, \sigma_{z}^{2}),\\
c(x,u) &= x^{\mathsf{T}}Qx + u^{\mathsf{T}}Ru + 2u^{\mathsf{T}}Sx,\\
\mu(\cdot) &= \mathcal{N}(\mu_{x}, \bar{\Sigma}).
\end{aligned} f ( ⋅ ∣ x , u ) g ( ⋅ ∣ x , u ) c ( x , u ) μ ( ⋅ ) = N ( A x + B u , Σ w ) , = N ( u T C x , σ z 2 ) , = x T Q x + u T R u + 2 u T S x , = N ( μ x , Σ ˉ ) .
We assume ρ ( A ) < 1 \rho(A) < 1 ρ ( A ) < 1 , R ≻ 0 R \succ0 R ≻ 0 , Q − S T R − 1 S ≻ 0 Q - S^{\mathsf{T}}R^{-1}S \succ0 Q − S T R − 1 S ≻ 0 , Σ w ≻ 0 \Sigma_{w} \succ0 Σ w ≻ 0 , and σ z 2 > 0 \sigma_{z}^{2} > 0 σ z 2 > 0 . Here, Σ ˉ \bar{\Sigma} Σ ˉ is the stationary open-loop covariance, uniquely determined by Σ ˉ = A Σ ˉ A T + Σ w \bar{\Sigma} = A\bar{\Sigma}A^{\mathsf{T}} + \Sigma_{w} Σ ˉ = A Σ ˉ A T + Σ w .
Kalman Filtering. To compute the state estimate used by π C E \pi^{CE} π C E , we recall the Kalman filtering results of Sattar et al. [22 ] . Under any policy π \pi π , the posterior state distribution is
x t ∣ H t ∼ N ( x ^ t , Σ t ) . x_t \mid H_t \sim\mathcal{N}(\hat{x}_t, \Sigma_t). x t ∣ H t ∼ N ( x ^ t , Σ t ) .
Starting from x ^ 0 = μ x \hat{x}_0 = \mu_x x ^ 0 = μ x and Σ 0 = Σ ˉ \Sigma_0 = \bar{\Sigma} Σ 0 = Σ ˉ , the conditional mean and covariance can be computed through the following recursion:
x ^ t + 1 = A x ^ t + B u t + A Σ t C T u t ( y t − u t T C x ^ t ) σ z 2 + u t T C Σ t C T u t , Σ t + 1 = A Σ t A T + Σ w − F ( Σ t , u t ) , (1) \begin{aligned}
\hat{x}_{t+1} &= A\hat{x}_t + Bu_t + \frac{A\Sigma_t C^{\mathsf{T}}u_t(y_t - u_t^{\mathsf{T}}C\hat{x}_t)}{\sigma_z^2 + u_t^{\mathsf{T}}C\Sigma_t C^{\mathsf{T}}u_t},\\
\Sigma_{t+1} &= A\Sigma_t A^{\mathsf{T}} + \Sigma_w - F(\Sigma_t,u_t), \tag*{(1)}
\end{aligned} x ^ t + 1 Σ t + 1 = A x ^ t + B u t + σ z 2 + u t T C Σ t C T u t A Σ t C T u t ( y t − u t T C x ^ t ) , = A Σ t A T + Σ w − F ( Σ t , u t ) , (1)
where
F ( Σ , u ) = A Σ C T u u T C Σ A T σ z 2 + u T C Σ C T u ≥ 0 F(\Sigma,u) = \frac{A\Sigma C^{\mathsf{T}}uu^{\mathsf{T}}C\Sigma A^{\mathsf{T}}}{\sigma_z^2 + u^{\mathsf{T}}C\Sigma C^{\mathsf{T}}u} \ge0 F ( Σ , u ) = σ z 2 + u T C Σ C T u A Σ C T u u T C Σ A T ≥ 0
These recursions form a variant of the classical Kalman filter (Kalman [12 ] ). Unlike in classical LQG, the observation map is input-dependent, so both the Kalman gain and the covariance update depend on the input.
We note that F ( Σ , u ) F(\Sigma,u) F ( Σ , u ) represents the next-step covariance reduction relative to the open-loop covariance A Σ A T + Σ w A\Sigma A^{\mathsf{T}} + \Sigma_w A Σ A T + Σ w . The following lemma gives another interpretation of F ( Σ , u ) F(\Sigma,u) F ( Σ , u ) .
Lemma 1.
E [ x ^ t + 1 x ^ t + 1 T ∣ H t ] = ( A x ^ t + B u t ) ( A x ^ t + B u t ) T + F ( Σ t , u t ) \mathbb{E}[\hat{x}_{t+1}\hat{x}_{t+1}^{\mathsf{T}} \mid H_t] = (A\hat{x}_t + Bu_t)(A\hat{x}_t + Bu_t)^{\mathsf{T}} + F(\Sigma_t,u_t) E [ x ^ t + 1 x ^ t + 1 T ∣ H t ] = ( A x ^ t + B u t ) ( A x ^ t + B u t ) T + F ( Σ t , u t )
Lemma 1 implies that F ( Σ , u ) F(\Sigma,u) F ( Σ , u ) can also be interpreted as influencing the growth rate of the second moment of the state estimate.
Benchmark Policies
In this section, we realize the certainty-equivalent (CE) policy and the open-loop (OL) policy. Set P T + 1 = 0 P_{T+1}=0 P T + 1 = 0 and define for t = T , … , 0 t=T,\ldots,0 t = T , … , 0 ,
N t = R + B ⊤ P t + 1 B , K t = − N t − 1 ( B ⊤ P t + 1 A + S ) , P t = A ⊤ P t + 1 A + Q − K t ⊤ N t K t . (2) \begin{aligned}
N_t &= R+B^\top P_{t+1}B,\\
K_t &= -N_t^{-1}(B^\top P_{t+1}A+S),\\
P_t &= A^\top P_{t+1}A+Q-K_t^\top N_tK_t. \tag*{(2)}
\end{aligned} N t K t P t = R + B ⊤ P t + 1 B , = − N t − 1 ( B ⊤ P t + 1 A + S ) , = A ⊤ P t + 1 A + Q − K t ⊤ N t K t . (2)
These are the finite-horizon Riccati recursions, with P t ≻ 0 P_t \succ0 P t ≻ 0 . For the fully observed optimal control problem, it is well known ([15 ] ) that
K t x ∈ arg min u Q ~ t ( x , u ) . K_tx\in\underset{u}{\arg\min}\ \widetilde{Q}_t(x,u). K t x ∈ u arg min Q t ( x , u ) .
From the definitions in § 2 , the corresponding benchmark policies are as follows.
The certainty-equivalent (CE) policy π C E \pi^{CE} π C E is
π t C E ( H t ) = K t x ^ t . \pi_t^{CE}(H_t)=K_t\widehat{x}_t. π t C E ( H t ) = K t x t .
The optimal open-loop (OL) policy π O L \pi^{OL} π O L is
π t O L ( H t ) = K t m t \pi_t^{OL}(H_t)=K_tm_t π t O L ( H t ) = K t m t
where m 0 = μ x m_0=\mu_x m 0 = μ x and m t + 1 = ( A + B K t ) m t m_{t+1}=(A+BK_t)m_t m t + 1 = ( A + B K t ) m t .
Both CE and OL use the gains K t K_t K t from the fully observed linear-quadratic problem. CE applies these gains to the estimate x ^ t \widehat{x}_t x t , which incorporates observations, whereas OL applies them to the unconditional state prediction m t m_t m t , which evolves without incorporating observations. Appendix § 6 provides the derivation of the gains and the resulting CE and OL policies.
We note that characterizing the optimal policy π ⋆ \pi^\star π ⋆ is more challenging. Previous works ([22 , 6 ] ) discuss the need to balance control and information gathering, since actions affect both the state dynamics and future estimation uncertainty.
Information Strength q q q
Under π C E \pi^{CE} π C E , when the state estimate x ^ t \hat{x}_t x ^ t is small, the CE action K t x ^ t K_t\hat{x}_t K t x ^ t is small as well, providing a weak observation. In the scalar setting when p = n = 1 p=n=1 p = n = 1 , the covariance reduction is
F ( Σ t , K t x ^ t ) = A 2 K t 2 x ^ t 2 C 2 Σ t 2 σ z 2 + K t 2 x ^ t 2 C 2 Σ t 2 . F(\Sigma_t,K_t\hat{x}_t)=\frac{A^2K_t^2\hat{x}_t^2C^2\Sigma_t^2}{\sigma_z^2+K_t^2\hat{x}_t^2C^2\Sigma_t^2}. F ( Σ t , K t x ^ t ) = σ z 2 + K t 2 x ^ t 2 C 2 Σ t 2 A 2 K t 2 x ^ t 2 C 2 Σ t 2 .
For ( x ^ t , Σ t ) (\hat{x}_t,\Sigma_t) ( x ^ t , Σ t ) near ( 0 , Σ ˉ ) (0,\bar{\Sigma}) ( 0 , Σ ˉ ) , we have
F ( Σ t , K t x ^ t ) ≈ q x ^ t 2 with q = A 2 K 2 C 2 Σ ˉ 2 σ z 2 . F(\Sigma_t,K_t\hat{x}_t)\approx q\hat{x}_t^2 \quad\text{with} \quad q=\frac{A^2K^2C^2\bar{\Sigma}^2}{\sigma_z^2}. F ( Σ t , K t x ^ t ) ≈ q x ^ t 2 with q = σ z 2 A 2 K 2 C 2 Σ ˉ 2 .
Hence by Lemma 1 it follows that
E [ x ^ t + 1 2 ∣ H t ] ≈ ( D 2 + q ) x ^ t 2 \mathbb{E}[\hat{x}_{t+1}^2\mid H_t]\approx(D^2+q)\hat{x}_t^2 E [ x ^ t + 1 2 ∣ H t ] ≈ ( D 2 + q ) x ^ t 2
for D D D in Remark 1. The parameter q q q can be viewed as the information-feedback strength under CE. Indeed, for beliefs near ( 0 , Σ ˉ ) (0,\bar{\Sigma}) ( 0 , Σ ˉ ) , the signal-to-noise ratio of an observation generated by CE is proportional to
K 2 C 2 Σ ˉ σ z 2 x ^ t 2 , (3) \frac{K^2C^2\bar{\Sigma}}{\sigma_z^2}\hat{x}_t^2, \tag*{(3)} σ z 2 K 2 C 2 Σ ˉ x ^ t 2 , (3)
while q q q additionally accounts for how information propagates through the dynamics to the next state estimate. Equivalently, D 2 + q D^2+q D 2 + q is the local growth rate factor of the second moment of the CE state estimate. This motivates the following study of how the performance gaps between π O L \pi^{OL} π O L , π C E \pi^{CE} π C E , and π ⋆ \pi^\star π ⋆ vary with q q q . § 4 establishes sufficient conditions on q q q for bounded and linear advantages over open-loop control, thereby realizing the three cases of Theorem 2.
Results
All results in this section concern the scalar setting where p = n = 1 p=n=1 p = n = 1 .
Regret decomposition
Throughout our analysis, we use π O L \pi^{OL} π O L as a common reference. Since π O L \pi^{OL} π O L is optimal among policies that do not use observations R T ( π , π O L ) R_T(\pi,\pi^{OL}) R T ( π , π O L ) measures the advantage π \pi π gains from using feedback. In this section, we present in Proposition 1 a regret decomposition with respect to π O L \pi^{OL} π O L , which will be used to derive our main results. We first define the ingredients of the first decomposition. For x ^ , u ∈ R \hat{x},u\in\mathbb{R} x ^ , u ∈ R , Σ > 0 \Sigma>0 Σ > 0 , define
b t ( Σ , u ) = Γ t + 1 F ( Σ , u ) p t ( x ^ , u ) = N t ( u − K t x ^ ) 2 (4) \begin{aligned}
b_t(\Sigma,u)&=\Gamma_{t+1}F(\Sigma,u)\\
p_t(\hat{x},u)&=N_t(u-K_t\hat{x})^2 \tag*{(4)}
\end{aligned} b t ( Σ , u ) p t ( x ^ , u ) = Γ t + 1 F ( Σ , u ) = N t ( u − K t x ^ ) 2 (4)
where Γ t \Gamma_t Γ t is defined recursively:
Γ t = N t K t 2 + A 2 Γ t + 1 , Γ T + 1 = 0. (5) \Gamma_t=N_tK_t^2+A^2\Gamma_{t+1},\qquad\Gamma_{T+1}=0. \tag*{(5)} Γ t = N t K t 2 + A 2 Γ t + 1 , Γ T + 1 = 0. (5)
Proposition 1. For x ^ , u ∈ R \hat{x},u\in\mathbb{R} x ^ , u ∈ R and Σ > 0 \Sigma>0 Σ > 0 , define
r t ( x ^ , Σ , u ) = b t ( Σ , u ) − p t ( x ^ , u ) . (6) r_t(\hat{x},\Sigma,u)=b_t(\Sigma,u)-p_t(\hat{x},u). \tag*{(6)} r t ( x ^ , Σ , u ) = b t ( Σ , u ) − p t ( x ^ , u ) . (6) For any policy π \pi π ,
R T ( π , π O L ) = ∑ t = 0 T E π [ r t ( x ^ t , Σ t , u t ) ] . R_T(\pi,\pi^{OL})=\sum_{t=0}^{T}\mathbb{E}_{\pi}[r_t(\hat{x}_t,\Sigma_t,u_t)]. R T ( π , π O L ) = t = 0 ∑ T E π [ r t ( x ^ t , Σ t , u t )] .
The proof of Proposition 1 is deferred to Appendix § 8 . We can interpret the summands r t r_t r t as follows. The values p t ( x ^ , u ) p_t(\hat{x},u) p t ( x ^ , u ) and b t ( Σ , u ) b_t(\Sigma,u) b t ( Σ , u ) are always nonnegative. The term p t ( x ^ , u ) p_t(\hat{x},u) p t ( x ^ , u ) represents the penalty of deviating from certainty-equivalence, while b t ( Σ , u ) b_t(\Sigma,u) b t ( Σ , u ) represents the weighted reduction in next-step estimation uncertainty due to the observation. The two terms create a tradeoff in deciding whether to incur cost by deviating from certainty equivalence to obtain information for future benefit, a dilemma created by action-dependent observations.
Bounded Optimal-Control Advantage
In this section, we show that unlike classical LQG, π ⋆ \pi^\star π ⋆ need not achieve a linear improvement over π O L \pi^{OL} π O L . This is implied by the following theorem.
Theorem 3 (Low Information Regime). Define
q 1 : = ( 1 − A 2 ) K 2 Q ( R − S 2 Q ) . q_1 := \frac{(1-A^2)K^2}{Q}\left(R-\frac{S^2}{Q}\right). q 1 := Q ( 1 − A 2 ) K 2 ( R − Q S 2 ) . If q < q 1 q < q_1 q < q 1 , then for any policy π \pi π
r t ( x ^ t , Σ t , u t ) ≤ P t x ^ t 2 − P t + 1 E π [ x ^ t + 1 2 ∣ H t ] . (7) r_t(\hat{x}_t,\Sigma_t,u_t) \le P_t\hat{x}_t^2-P_{t+1}\mathbb{E}^{\pi}\left[\hat{x}_{t+1}^2 \mid H_t\right]. \tag*{(7)} r t ( x ^ t , Σ t , u t ) ≤ P t x ^ t 2 − P t + 1 E π [ x ^ t + 1 2 ∣ H t ] . (7) Consequently, we have
R T ( π ⋆ , π O L ) = O ( 1 ) . R_T(\pi^\star,\pi^{OL})=O(1). R T ( π ⋆ , π O L ) = O ( 1 ) .
The proof of Theorem 3 is deferred to Appendix § 9 . When q < q 1 q < q_1 q < q 1 , the proof shows that the weighted reduction in estimation uncertainty is dominated by the state-and-action cost. Consequently, no policy, including the optimal policy π ⋆ \pi^\star π ⋆ , can reduce the total cost relative to open-loop control by more than a constant that does not grow with the horizon. This regime arises, for example, when measurement noise is sufficiently large. In the next section, we identify an even wider range of instances in which π C E \pi^{CE} π C E achieves only a bounded advantage over open-loop control.
Bounded CE Advantage
In this section, we seek an upper bound on the advantage of π C E \pi^{CE} π C E over π O L \pi^{OL} π O L . Under the policy π C E \pi^{CE} π C E , we note that the value p t ( x ^ , u ) p_t(\hat{x},u) p t ( x ^ , u ) , defined in (4) , is always zero. This leads to a key identity for bounding this advantage:
r t ( x ^ , Σ , K t x ^ ) = Γ t + 1 A 2 K t 2 x ^ 2 C 2 Σ 2 σ z 2 + K t 2 x ^ 2 C 2 Σ . (8) r_t(\hat{x},\Sigma,K_t\hat{x})=\Gamma_{t+1}\frac{A^2K_t^2\hat{x}^2C^2\Sigma^2}{\sigma_z^2+K_t^2\hat{x}^2C^2\Sigma}. \tag*{(8)} r t ( x ^ , Σ , K t x ^ ) = Γ t + 1 σ z 2 + K t 2 x ^ 2 C 2 Σ A 2 K t 2 x ^ 2 C 2 Σ 2 . (8)
where r t r_t r t is defined in (6) that aggregates to regret by Proposition 1. The quantity (8) is nondecreasing in both x ^ 2 \hat{x}^2 x ^ 2 and Σ \Sigma Σ , and vanishes when x ^ = 0 \hat{x}=0 x ^ = 0 . One key attribute of the bilinear setting is that there exists a process { M t } t ≥ 0 \{M_t\}_{t\geq0} { M t } t ≥ 0 under π C E \pi^{CE} π C E such that
x ^ t + 1 = M t x ^ t . (9) \hat{x}_{t+1}=M_t\hat{x}_t. \tag*{(9)} x ^ t + 1 = M t x ^ t . (9)
See Lemma 10 in Appendix § 7 for details of M t M_t M t . If μ x = 0 \mu_x=0 μ x = 0 , then x ^ t = 0 \hat{x}_t=0 x ^ t = 0 for every t t t . Hence, r t ( x ^ t , Σ t , K t x ^ t ) = 0 r_t(\hat{x}_t,\Sigma_t,K_t\hat{x}_t)=0 r t ( x ^ t , Σ t , K t x ^ t ) = 0 , and Proposition 1 gives R T ( π C E , π O L ) = 0 R_T(\pi^{CE},\pi^{OL})=0 R T ( π C E , π O L ) = 0 .
In terms of feedback, this is easy to interpret. A zero estimate leads to the zero action, so the observation is pure noise and carries no information about the state. Thus π C E \pi^{CE} π C E receives no useful feedback and coincides with π O L \pi^{OL} π O L . This contrasts with LQG with action-independent observations, where the zero action does not make the observations uninformative, and where μ x = 0 \mu_x=0 μ x = 0 can be assumed without loss of generality.
A more interesting failure of π C E \pi^{CE} π C E occurs even when μ x ≠ 0 \mu_x \ne0 μ x = 0 . When each E [ M t 2 ∣ H t ] ≤ γ \mathbb{E}[M_t^2 \mid H_t] \le\gamma E [ M t 2 ∣ H t ] ≤ γ is uniformly below 1, Eq. (9) implies E [ x ^ t 2 ] \mathbb{E}[\hat{x}_t^2] E [ x ^ t 2 ] decays geometrically. Then, r t ( x ^ t , Σ t , K t x ^ t ) r_t(\hat{x}_t,\Sigma_t,K_t\hat{x}_t) r t ( x ^ t , Σ t , K t x ^ t ) decays geometrically, and its sum over time remains bounded. In other words, π C E \pi^{CE} π C E gradually loses the feedback that gave it an advantage over π O L \pi^{OL} π O L . We formalize this as follows.
Proposition 2. Define γ : = ( D 2 + q + 1 ) / 2 \gamma:= (D^2+q+1)/2 γ := ( D 2 + q + 1 ) /2 and
q 3 : = 1 − D 2 . q_3 := 1-D^2. q 3 := 1 − D 2 . If q < q 3 q<q_3 q < q 3 , then γ < 1 \gamma<1 γ < 1 and there exists a positive integer t 0 t_0 t 0 such that E [ M t 2 ∣ H t ] ≤ γ \mathbb{E}[M_t^2 \mid H_t]\le\gamma E [ M t 2 ∣ H t ] ≤ γ for all t ≤ T − t 0 t\le T-t_0 t ≤ T − t 0 . Consequently
R T ( π C E , π O L ) = O ( 1 ) . R_T(\pi^{CE},\pi^{OL})=O(1). R T ( π C E , π O L ) = O ( 1 ) .
The proof of Proposition 2 is deferred to Appendix E.
Linear CE Advantage
Our next result shows that the previous negative result is not universal for π C E \pi^{CE} π C E . We show that as long as q q q is large enough, a linear advantage of π C E \pi^{CE} π C E over π O L \pi^{OL} π O L can be established. Similar to the analysis of π C E \pi^{CE} π C E in the previous section, the result primarily depends on the behavior of x ^ t \hat{x}_t x ^ t . We show that for large enough q q q , either x ^ t 2 \hat{x}_t^2 x ^ t 2 or Σ ˉ − Σ t \bar{\Sigma}-\Sigma_t Σ ˉ − Σ t spends sufficient time away from zero, leading to linear regret. Define the set
M δ : = { ( x ^ , Σ ) : x ^ 2 ≤ δ , Σ ˉ − Σ ≤ δ } . (10) \mathcal{M}_{\delta}:=\{(\hat{x},\Sigma):\hat{x}^2\le\delta,\ \bar{\Sigma}-\Sigma\le\delta\}. \tag*{(10)} M δ := {( x ^ , Σ ) : x ^ 2 ≤ δ , Σ ˉ − Σ ≤ δ } . (10)
We then have the following proposition.
Proposition 3. Fix δ , ε > 0 \delta,\varepsilon>0 δ , ε > 0 with δ ≤ Σ ˉ \delta\le\bar{\Sigma} δ ≤ Σ ˉ . Assume μ x ≠ 0 \mu_x\ne0 μ x = 0 , D ≠ 0 D\ne0 D = 0 , and suppose
D 2 + A 2 ( Σ ˉ − δ ) 2 K 2 C 2 σ z 2 + K 2 C 2 ( Σ ˉ − δ ) δ > 2 ε ε + γ E (11) D^2+\frac{A^2(\bar{\Sigma}-\delta)^2K^2C^2}{\sigma_z^2+K^2C^2(\bar{\Sigma}-\delta)\delta}>2\varepsilon^{\varepsilon+\gamma_E} \tag*{(11)} D 2 + σ z 2 + K 2 C 2 ( Σ ˉ − δ ) δ A 2 ( Σ ˉ − δ ) 2 K 2 C 2 > 2 ε ε + γ E (11) where γ E \gamma_E γ E is Euler’s constant. Then
∑ t = 0 T Pr ( ( x ^ t , Σ t ) ∉ M δ ) = Ω ( T ) . (12) \sum_{t=0}^{T}\Pr((\hat{x}_t,\Sigma_t)\notin\mathcal{M}_{\delta})=\Omega(T). \tag*{(12)} t = 0 ∑ T Pr (( x ^ t , Σ t ) ∈ / M δ ) = Ω ( T ) . (12) Furthermore, it follows that
R T ( π C E , π O L ) = Ω ( T ) . R_T(\pi^{CE},\pi^{OL})=\Omega(T). R T ( π C E , π O L ) = Ω ( T ) .
The proof of Proposition 3 is deferred to Appendix F. Proposition 3 identifies a sufficient condition under which the observations generated by the actions of π C E \pi^{CE} π C E itself remain informative enough for π C E \pi^{CE} π C E to achieve a linear advantage over π O L \pi^{OL} π O L . The set M δ \mathcal{M}_{\delta} M δ represents beliefs where both the state estimate is small, leading to small CE actions, and the estimation uncertainty is close to the open-loop covariance, indicating little information retained from past observations. Under condition (11) , π C E \pi^{CE} π C E spends a positive fraction of the horizon, in expectation, outside this set. Outside this set, either the estimated state is large enough to support informative CE actions, or past observations have already reduced uncertainty about the current state. These two sources of benefit yield a cumulative cost advantage over π O L \pi^{OL} π O L that grows linearly with the horizon.
By choosing δ \delta δ and ε \varepsilon ε sufficiently close to zero, we obtain the following sufficient condition in terms of the information strength q q q .
Theorem 4 (High Information Regime). Assume μ x ≠ 0 \mu_x \ne0 μ x = 0 and D ≠ 0 D \ne0 D = 0 . Define
q 4 : = 2 e γ E − D 2 . q_4 := 2e^{\gamma_E} - D^2. q 4 := 2 e γ E − D 2 . If q > q 4 q > q_4 q > q 4 , then
R T ( π C E , π O L ) = Ω ( T ) . R_T(\pi^{CE}, \pi^{OL}) = \Omega(T). R T ( π C E , π O L ) = Ω ( T ) .
The proof of Theorem 4 is deferred to Appendix § 11 . The results so far identify both limitations and benefits of π C E \pi^{CE} π C E over π O L \pi^{OL} π O L . When q < q 1 q < q_1 q < q 1 , Theorem 3 shows that no policy can improve on π O L \pi^{OL} π O L by more than a constant. For π C E \pi^{CE} π C E , Proposition 2 establishes a bounded advantage when q < q 3 q < q_3 q < q 3 , whereas Theorem 4 guarantees a linear advantage when q > q 4 q > q_4 q > q 4 , under its stated assumptions. Thus, sufficiently strong observations allow CE to accumulate a benefit from feedback even without active information gathering. These sufficient conditions leave CE’s behavior between q 3 q_3 q 3 and q 4 q_4 q 4 uncharacterized. The next section explores whether actively probing to gather information can improve on the performance of π C E \pi^{CE} π C E .
Probing Policy
To motivate this next section, we start with the following proposition.
Proposition 4. We have that q 1 < q 3 q_1 < q_3 q 1 < q 3 .
Proposition 4 leaves open whether π ⋆ \pi^\star π ⋆ can achieve a linear advantage over π O L \pi^{OL} π O L for q 1 < q < q 3 q_1 < q < q_3 q 1 < q < q 3 , where the advantage of π C E \pi^{CE} π C E remains bounded. Directly characterizing π ⋆ \pi^\star π ⋆ is challenging because actions affect both the state dynamics and future estimation uncertainty. Previous works (Sattar et al., [22 ] ; Cao et al., [6 ] ) show that the resulting control optimization can be nonconvex and intractable. For our purpose, however, it suffices to construct a policy with a linear advantage over π O L \pi^{OL} π O L , since the optimal policy performs at least as well. The policy achieves this through “probing”; exploratory actions aimed at increasing information. At each time step t t t , the policy operates as follows:
Take a probing action if all of the following hold: (a) no probing action has been taken recently; (b) x ^ t 2 \hat{x}_t^2 x ^ t 2 is small; and (c) Σ ˉ − Σ t \bar{\Sigma} - \Sigma_t Σ ˉ − Σ t is small.
Otherwise, take the CE action K t x ^ t K_t\hat{x}_t K t x ^ t .
We formalize this policy using stopping times. Conditions (b) and (c) together say that the belief lies in the set M δ M_\delta M δ defined in (10) , and condition (a) is enforced by requiring consecutive probes to be at least L L L steps apart. Fix δ , a , L , t P > 0 \delta, a, L, t_P > 0 δ , a , L , t P > 0 , and let τ 0 : = − L \tau_0 := -L τ 0 := − L , and for k ≥ 1 k \ge1 k ≥ 1 define
τ k : = inf t ∈ I k { ( x ^ t , Σ t ) ∈ M δ } \tau_k := \inf_{t \in I_k}\{(\hat{x}_t,\Sigma_t) \in M_\delta\} τ k := t ∈ I k inf {( x ^ t , Σ t ) ∈ M δ }
I k : = { t ∈ Z : τ k − 1 + L ≤ t ≤ T − t P − L } (13) I_k := \{t \in\mathbb{Z}: \tau_{k-1} + L \le t \le T - t_P - L\} \tag*{(13)} I k := { t ∈ Z : τ k − 1 + L ≤ t ≤ T − t P − L } (13)
with inf ∅ = ∞ \inf\varnothing= \infty inf ∅ = ∞ . Let T : = { τ k : k ≥ 1 , τ k < ∞ } \mathcal{T} := \{\tau_k : k \ge1, \tau_k < \infty\} T := { τ k : k ≥ 1 , τ k < ∞ } . The probing policy is defined as
π t P ( H t ) = K t x ^ t + a ⋅ 1 { t ∈ T } . (14) \pi_t^P(H_t) = K_t\hat{x}_t + a \cdot1\{t \in\mathcal{T}\}. \tag*{(14)} π t P ( H t ) = K t x ^ t + a ⋅ 1 { t ∈ T } . (14)
Here, a a a is the size of each probe, and L L L is the minimum spacing between probes. The parameter δ \delta δ sets how close the belief must be to ( 0 , Σ ˉ ) (0,\bar{\Sigma}) ( 0 , Σ ˉ ) for a probe to occur. The cutoff parameter t P t_P t P sets when to stop probing near the end of the horizon. The following proposition characterizes when π P \pi^P π P achieves a linear advantage over open-loop control.
Proposition 5. Define
q 2 ≔ ( 1 − A 2 ) ( 1 − D 2 ) 2 ( 1 − A D ) . q_2 \coloneqq\frac{(1-A^2)(1-D^2)}{2(1-AD)}. q 2 : = 2 ( 1 − A D ) ( 1 − A 2 ) ( 1 − D 2 ) . If q > q 2 q > q_2 q > q 2 , then there exist choices δ \delta δ , a a a , L L L , t P > 0 t_{\mathrm{P}} > 0 t P > 0 independent of T T T such that π P \pi^{\mathrm{P}} π P satisfies
R T ( π P , π O L ) = Ω ( T ) . R_T(\pi^{\mathrm{P}},\pi^{\mathrm{OL}})=\Omega(T). R T ( π P , π OL ) = Ω ( T ) .
The proof of Proposition 5 is deferred to Appendix § 12 . The result addresses the failure mode of π C E \pi^{CE} π C E identified earlier: when the belief enters M δ \mathcal{M}_{\delta} M δ , the CE action is small while estimation uncertainty remains high. A probe incurs a cost for deviating from this action, but produces an informative observation and can move the state estimate away from zero. Subsequent CE actions can then gather further informative observations. The condition q > q 2 q > q_2 q > q 2 ensures that the probe size a a a and block length L L L can be chosen so that these expected benefits outweigh the probing cost. The policy repeats this intervention when small actions and high uncertainty recur. Outside probing blocks, CE benefits either from informative actions or from information retained from earlier observations. These gains accumulate into a cost advantage over open-loop control that grows linearly with the horizon.
Building on Propositions 2 and 5, the following theorem establishes an Ω ( T ) \Omega(T) Ω ( T ) gap between optimal and CE control when q 2 < q < q 3 q_2 < q < q_3 q 2 < q < q 3 .
Theorem 5 (Separation Regime). We have that q 2 < q 3 q_2 < q_3 q 2 < q 3 and that for q ∈ ( q 2 , q 3 ) q \in(q_2,q_3) q ∈ ( q 2 , q 3 ) ,
(i) R T ( π ⋆ , π O L ) = Ω ( T ) R_T(\pi^\star,\pi^{OL})=\Omega(T) R T ( π ⋆ , π O L ) = Ω ( T )
(ii) R T ( π C E , π O L ) = O ( 1 ) R_T(\pi^{CE},\pi^{OL})=O(1) R T ( π C E , π O L ) = O ( 1 ) .
The proof of Theorem 5 is deferred to Appendix § 12 . Figure 1 compares the performance of the probing policy and CE as q q q varies. Finally, for completeness, we have the following result.
Figure 1. (a) Depicts the advantage of CE and probing over open-loop; (b) depicts the advantage of probing over CE. The advantages are normalized by T T T , hence values bounded away from zero are consistent with linear advantage, while values near zero are consistent with bounded advantage. The observed behavior is consistent with the theoretical regimes: probing begins to outperform open-loop control around q 2 q_2 q 2 , while CE’s advantage begins around q 4 q_4 q 4 . For small q q q , probing incurs costs that outweigh its informational benefits. For large q q q , CE and probing perform similarly. Appendix § 13 provides the experimental setup and additional plots illustrating how these advantages scale with the horizon.
Proposition 6. We have that q 1 < q 2 q_1 < q_2 q 1 < q 2 .
Conclusion
We studied the performance gaps between open-loop, certainty-equivalent, and optimal control in linear-quadratic systems with bilinear observations. In the scalar setting, we identified information-strength regimes in which feedback provides only a bounded advantage, CE achieves a linear advantage, or active probing achieves a linear advantage that CE fails to realize. Natural directions for future work include extending the theory beyond the scalar setting, refining the probing policy, and studying adaptive settings in which the system dynamics or observation model must be learned online.
Acknowledgements. MF was supported in part by awards NSF TRIPODS II DMS-2023166, CCF 2212261, CCF 2312775, and the Moorthy Professorship at the UW. She also works at Amazon as an Amazon Scholar. SD was supported in part by NSF CCF-2312774, NSF OAC-2311521, NSF IIS-2442137, and an AI2050 Early Career Fellowship program at Schmidt Sciences.
Dynamic programming and Policies
In this section, we connect the policy definitions in § 2 to their explicit forms in the linear-quadratic model. We first derive the Riccati recursion from the fully observed Bellman equation and obtain the certainty-equivalent controller by substituting the conditional mean for the state. We then show that the optimal open-loop policy is obtained from the same Riccati recursion applied to the deterministic mean trajectory. Finally, we prove the filtering results used to describe the evolution of the belief state. Throughout this section, we work in the general setting of § 2 .
Riccati Recursion and Certainty-Equivalent Control
Let Q ~ t \widetilde{Q}_t Q t be the fully observed state-action cost-to-go function defined in § 2 , and define
V ~ t ( x ) = min u ∈ R p Q ~ t ( x , u ) , V ~ T + 1 ( x ) : = 0. \widetilde{V}_t(x)=\min_{u\in\mathbb{R}^p}\widetilde{Q}_t(x,u),\qquad\widetilde{V}_{T+1}(x):=0. V t ( x ) = u ∈ R p min Q t ( x , u ) , V T + 1 ( x ) := 0.
The Bellman equation becomes
Q ~ t ( x , u ) = c ( x , u ) + E w ∼ N ( 0 , Σ w ) [ V ~ t + 1 ( A x + B u + w ) ] . \widetilde{Q}_t(x,u)=c(x,u)+\mathbb{E}_{w\sim\mathcal{N}(0,\Sigma_w)}\left[\widetilde{V}_{t+1}(Ax+Bu+w)\right]. Q t ( x , u ) = c ( x , u ) + E w ∼ N ( 0 , Σ w ) [ V t + 1 ( A x + B u + w ) ] .
We show by backward induction that
V ~ t ( x ) = x ⊤ P t x + ∑ s = t + 1 T tr ( P s Σ w ) , (15) \widetilde{V}_t(x)=x^\top P_t x+\sum_{s=t+1}^{T}\operatorname{tr}(P_s\Sigma_w), \tag*{(15)} V t ( x ) = x ⊤ P t x + s = t + 1 ∑ T tr ( P s Σ w ) , (15)
where P t ⪰ 0 P_t\succeq0 P t ⪰ 0 and P T + 1 = 0 P_{T+1}=0 P T + 1 = 0 . At t = T + 1 t=T+1 t = T + 1 , the sum is empty and the identity holds. Suppose it holds at t + 1 t+1 t + 1 . Since w w w has zero mean and covariance Σ w \Sigma_w Σ w ,
Q ~ t ( x , u ) = x ⊤ Q x + u ⊤ R u + 2 u ⊤ S x + ( A x + B u ) ⊤ P t + 1 ( A x + B u ) + tr ( P t + 1 Σ w ) + ∑ s = t + 2 T tr ( P s Σ w ) = x ⊤ ( Q + A ⊤ P t + 1 A ) x + 2 u ⊤ ( B ⊤ P t + 1 A + S ) x + u ⊤ ( R + B ⊤ P t + 1 B ) u + ∑ s = t + 1 T tr ( P s Σ w ) . \begin{aligned}
\widetilde{Q}_t(x,u)={}&x^\top Qx+u^\top Ru+2u^\top Sx+(Ax+Bu)^\top P_{t+1}(Ax+Bu)+\operatorname{tr}(P_{t+1}\Sigma_w)+\sum_{s=t+2}^{T}\operatorname{tr}(P_s\Sigma_w)\\
={}&x^\top(Q+A^\top P_{t+1}A)x+2u^\top(B^\top P_{t+1}A+S)x+u^\top(R+B^\top P_{t+1}B)u+\sum_{s=t+1}^{T}\operatorname{tr}(P_s\Sigma_w).
\end{aligned} Q t ( x , u ) = = x ⊤ Q x + u ⊤ R u + 2 u ⊤ S x + ( A x + B u ) ⊤ P t + 1 ( A x + B u ) + tr ( P t + 1 Σ w ) + s = t + 2 ∑ T tr ( P s Σ w ) x ⊤ ( Q + A ⊤ P t + 1 A ) x + 2 u ⊤ ( B ⊤ P t + 1 A + S ) x + u ⊤ ( R + B ⊤ P t + 1 B ) u + s = t + 1 ∑ T tr ( P s Σ w ) .
Define
N t : = R + B ⊤ P t + 1 B , (16) N_t:=R+B^\top P_{t+1}B, \tag*{(16)} N t := R + B ⊤ P t + 1 B , (16)
K t : = − N t − 1 ( B ⊤ P t + 1 A + S ) , (17) K_t:=-N_t^{-1}(B^\top P_{t+1}A+S), \tag*{(17)} K t := − N t − 1 ( B ⊤ P t + 1 A + S ) , (17)
P t : = A ⊤ P t + 1 A + Q − K t ⊤ N t K t . (18) P_t:=A^\top P_{t+1}A+Q-K_t^\top N_tK_t. \tag*{(18)} P t := A ⊤ P t + 1 A + Q − K t ⊤ N t K t . (18)
Since P t + 1 ⪰ 0 P_{t+1}\succeq0 P t + 1 ⪰ 0 and R ≻ 0 R\succ0 R ≻ 0 , we have N t ⪰ R ≻ 0 N_t\succeq R\succ0 N t ⪰ R ≻ 0 . Completing the square gives
Q ~ t ( x , u ) = x ⊤ P t x + ( u − K t x ) ⊤ N t ( u − K t x ) + ∑ s = t + 1 T tr ( P s Σ w ) . (19) \widetilde{Q}_t(x,u)=x^\top P_tx+(u-K_tx)^\top N_t(u-K_tx)+\sum_{s=t+1}^{T}\operatorname{tr}(P_s\Sigma_w). \tag*{(19)} Q t ( x , u ) = x ⊤ P t x + ( u − K t x ) ⊤ N t ( u − K t x ) + s = t + 1 ∑ T tr ( P s Σ w ) . (19)
Thus, the unique minimizing action is u = K t x u=K_tx u = K t x , and minimizing over u u u gives (A.1). Moreover,
x ⊤ P t x = min u { c ( x , u ) + ( A x + B u ) ⊤ P t + 1 ( A x + B u ) } > 0 , for every x ∈ R n . x^\top P_tx=\min_u\left\{c(x,u)+(Ax+Bu)^\top P_{t+1}(Ax+Bu)\right\}>0,\qquad\text{for every }x\in\mathbb{R}^n. x ⊤ P t x = u min { c ( x , u ) + ( A x + B u ) ⊤ P t + 1 ( A x + B u ) } > 0 , for every x ∈ R n .
This is because the conditions Q ⪰ 0 Q\succeq0 Q ⪰ 0 , R ≻ 0 R\succ0 R ≻ 0 , and Q − S ⊤ R − 1 S ≻ 0 Q-S^\top R^{-1}S\succ0 Q − S ⊤ R − 1 S ≻ 0 imply c ( x , u ) > 0 c(x,u)>0 c ( x , u ) > 0 . Hence P t ≻ 0 P_t\succ0 P t ≻ 0 , completing the induction. In fact, the proof implies that P t ≻ 0 P_t\succ0 P t ≻ 0 for t = 0 , 1 , … , T t=0,1,\ldots,T t = 0 , 1 , … , T and P T + 1 = 0 P_{T+1}=0 P T + 1 = 0 .
Equations (A.2)–(A.4), together with P T + 1 = 0 P_{T+1}=0 P T + 1 = 0 , are the finite-horizon Riccati recursion. By the definition of certainty-equivalent control in § 2 , we evaluate the fully observed minimizing action at x ^ t = E [ x t ∣ H t ] \hat{x}_t=\mathbb{E}[x_t\mid H_t] x ^ t = E [ x t ∣ H t ] . Therefore,
π t C E ( H t ) = K t x ^ t . \pi_t^{CE}(H_t)=K_t\hat{x}_t. π t C E ( H t ) = K t x ^ t .
Stationary quantities. Lemma 5 establishes convergence of P t P_t P t to P P P as the remaining horizon grows. Since N t ⪰ R ≻ 0 N_t \succeq R \succ0 N t ⪰ R ≻ 0 , continuity of matrix inversion also gives convergence of N t N_t N t and K t K_t K t . Their limits satisfy
N = R + B T P B , K = − N − 1 ( B T P A + S ) , P = A T P A + Q − K T N K . \begin{aligned}
N &= R + B^{\mathsf{T}}PB,\\
K &= -N^{-1}(B^{\mathsf{T}}PA + S),\\
P &= A^{\mathsf{T}}PA + Q - K^{\mathsf{T}}NK.
\end{aligned} N K P = R + B T P B , = − N − 1 ( B T P A + S ) , = A T P A + Q − K T N K .
The last equation is the discrete algebraic Riccati equation (DARE). For the closed-loop dynamics, we write D t = A + B K t D_t = A + BK_t D t = A + B K t and D = A + B K D = A + BK D = A + B K .
Closed-loop stability In addition to the convergence result in Lemma 5, the gain K K K stabilizes the closed-loop dynamics, ρ ( D ) < 1 \rho(D) < 1 ρ ( D ) < 1 , under some assumptions. This follows from [10 Theorem 5] where we apply after the change of variables u t ← u t + R − 1 S x t u_t \leftarrow u_t + R^{-1}Sx_t u t ← u t + R − 1 S x t that removes the state-action cross term. We state the corresponding lemma for completeness where we defer the proof to [10 ] .
Lemma 2. If ( A , B ) (A, B) ( A , B ) is stabilizable1 , R ≻ 0 R \succ0 R ≻ 0 , and Q − S T R − 1 S ≻ 0 Q - S^{\mathsf{T}}R^{-1}S \succ0 Q − S T R − 1 S ≻ 0 , then ρ ( D ) < 1 \rho(D) < 1 ρ ( D ) < 1 .
Optimal Open-loop Control
Consider any deterministic action sequence ( u 0 , … , u T ) (u_0, \ldots, u_T) ( u 0 , … , u T ) chosen before interaction, and let m t : = E [ x t ] m_t := \mathbb{E}[x_t] m t := E [ x t ] . The state dynamics imply
m t + 1 = A m t + B u t with m 0 = μ x . m_{t+1} = Am_t + Bu_t \quad\text{with } m_0 = \mu_x. m t + 1 = A m t + B u t with m 0 = μ x .
Because the actions are deterministic, the unconditional state covariance satisfies
Cov ( x t + 1 ) = A Cov ( x t ) A T + Σ w . \operatorname{Cov}(x_{t+1}) = A\operatorname{Cov}(x_t)A^{\mathsf{T}} + \Sigma_w. Cov ( x t + 1 ) = A Cov ( x t ) A T + Σ w .
Since Cov ( x 0 ) = Σ ˉ \operatorname{Cov}(x_0) = \bar{\Sigma} Cov ( x 0 ) = Σ ˉ and Σ ˉ = A Σ ˉ A T + Σ w \bar{\Sigma} = A\bar{\Sigma}A^{\mathsf{T}} + \Sigma_w Σ ˉ = A Σ ˉ A T + Σ w , it follows that
Cov ( x t ) = Σ ˉ for every t . \operatorname{Cov}(x_t) = \bar{\Sigma} \quad\text{for every } t. Cov ( x t ) = Σ ˉ for every t .
This is the unconditional state covariance, distinct from the conditional covariance Σ t = Cov ( x t ∣ H t ) \Sigma_t = \operatorname{Cov}(x_t \mid H_t) Σ t = Cov ( x t ∣ H t ) . Consequently, the expected stage cost is now
E [ c ( x t , u t ) ] = c ( m t , u t ) + tr ( Q Σ ˉ ) , \mathbb{E}[c(x_t, u_t)] = c(m_t, u_t) + \operatorname{tr}(Q\bar{\Sigma}), E [ c ( x t , u t )] = c ( m t , u t ) + tr ( Q Σ ˉ ) ,
so that the expected cumulative cost of the open-loop policy is
J T ( π ) = ∑ t = 0 T c ( m t , u t ) + ( T + 1 ) tr ( Q Σ ˉ ) . (20) J_T(\pi) = \sum_{t=0}^{T} c(m_t, u_t) + (T+1)\operatorname{tr}(Q\bar{\Sigma}). \tag*{(20)} J T ( π ) = t = 0 ∑ T c ( m t , u t ) + ( T + 1 ) tr ( Q Σ ˉ ) . (20)
The covariance term is independent of the action sequence. Thus, optimizing the open-loop policy reduces to a deterministic linear-quadratic control problem for the mean trajectory. To solve this problem, we use a similar technique to decompose the term c ( m t , u t ) c(m_t, u_t) c ( m t , u t ) :
c ( m t , u t ) = ( u t − K t m t ) T N t ( u t − K t m t ) − m t + 1 T P t + 1 m t + 1 + m t T P t m t . (21) c(m_t, u_t) = (u_t - K_tm_t)^{\mathsf{T}}N_t(u_t - K_tm_t) - m_{t+1}^{\mathsf{T}}P_{t+1}m_{t+1} + m_t^{\mathsf{T}}P_tm_t. \tag*{(21)} c ( m t , u t ) = ( u t − K t m t ) T N t ( u t − K t m t ) − m t + 1 T P t + 1 m t + 1 + m t T P t m t . (21)
Summing over t t t , the quadratic terms telescope, and since P T + 1 = 0 P_{T+1}=0 P T + 1 = 0 and m 0 = μ x m_0=\mu_x m 0 = μ x , we obtain
∑ t = 0 T c ( m t , u t ) = μ x ⊤ P 0 μ x + ∑ t = 0 T ( u t − K t m t ) ⊤ N t ( u t − K t m t ) \sum_{t=0}^{T} c(m_t,u_t)=\mu_x^\top P_0\mu_x+\sum_{t=0}^{T}(u_t-K_tm_t)^\top N_t(u_t-K_tm_t) t = 0 ∑ T c ( m t , u t ) = μ x ⊤ P 0 μ x + t = 0 ∑ T ( u t − K t m t ) ⊤ N t ( u t − K t m t )
Substituting into (A.6) yields
J T ( π ) = μ x ⊤ P 0 μ x + ( T + 1 ) tr ( Q Σ ‾ ) + ∑ t = 0 T ( u t − K t m t ) ⊤ N t ( u t − K t m t ) . J_T(\pi)=\mu_x^\top P_0\mu_x+(T+1)\operatorname{tr}(Q\overline{\Sigma})+\sum_{t=0}^{T}(u_t-K_tm_t)^\top N_t(u_t-K_tm_t). J T ( π ) = μ x ⊤ P 0 μ x + ( T + 1 ) tr ( Q Σ ) + t = 0 ∑ T ( u t − K t m t ) ⊤ N t ( u t − K t m t ) .
Each term in the final sum is nonnegative because N t ≻ 0 N_t\succ0 N t ≻ 0 . The unique minimizing action sequence therefore satisfies
u t O L = K t m t , m t + 1 = ( A + B K t ) m t = D t m t , m 0 = μ x . u_t^{OL}=K_tm_t,\qquad m_{t+1}=(A+BK_t)m_t=D_tm_t,\qquad m_0=\mu_x. u t O L = K t m t , m t + 1 = ( A + B K t ) m t = D t m t , m 0 = μ x .
This sequence is deterministic and can be computed before interaction, so it defines an admissible open-loop policy. In particular,
π t O L ( H t ) = K t m t , J T ( π O L ) = μ x ⊤ P 0 μ x + ( T + 1 ) tr ( Q Σ ‾ ) . \pi_t^{OL}(H_t)=K_tm_t,\qquad J_T(\pi^{OL})=\mu_x^\top P_0\mu_x+(T+1)\operatorname{tr}(Q\overline{\Sigma}). π t O L ( H t ) = K t m t , J T ( π O L ) = μ x ⊤ P 0 μ x + ( T + 1 ) tr ( Q Σ ) .
The certainty-equivalent and open-loop policies therefore use the same gain K t K_t K t , but apply it to different quantities. The policy π C E \pi^{CE} π C E uses the conditional mean x ^ t \hat{x}_t x ^ t , which incorporates past observations, whereas the policy π O L \pi^{OL} π O L uses the deterministic mean m t m_t m t .
Riccati Equation Tools
In this section, we provide technical lemmas related to Riccati recursion that are useful to prove the main results.
Lemma 3. Define G t = Q + A 2 G t + 1 G_t=Q+A^2G_{t+1} G t = Q + A 2 G t + 1 with G T + 1 = 0 G_{T+1}=0 G T + 1 = 0 . Then, we have Γ t = G t − P t \Gamma_t=G_t-P_t Γ t = G t − P t .
Proof. Recall Γ t \Gamma_t Γ t defined in (5) . Since G T + 1 = P T + 1 = 0 G_{T+1}=P_{T+1}=0 G T + 1 = P T + 1 = 0 , we trivially have that Γ T + 1 = G T + 1 − P T + 1 = 0 \Gamma_{T+1}=G_{T+1}-P_{T+1}=0 Γ T + 1 = G T + 1 − P T + 1 = 0 . Assuming Γ t + 1 = G t + 1 − P t + 1 \Gamma_{t+1}=G_{t+1}-P_{t+1} Γ t + 1 = G t + 1 − P t + 1 , we get
Γ t = N t K t 2 + A 2 Γ t + 1 = A 2 P t + 1 − P t + Q + A 2 Γ t + 1 = A 2 G t + 1 + Q − P t = G t − P t . \Gamma_t=N_tK_t^2+A^2\Gamma_{t+1}=A^2P_{t+1}-P_t+Q+A^2\Gamma_{t+1}=A^2G_{t+1}+Q-P_t=G_t-P_t. Γ t = N t K t 2 + A 2 Γ t + 1 = A 2 P t + 1 − P t + Q + A 2 Γ t + 1 = A 2 G t + 1 + Q − P t = G t − P t . The proof follows by induction.
Lemma 4. Recall that D t = A + B K t D_t=A+BK_t D t = A + B K t . For each t t t ,
D t = A R − B S N t and ∣ D t ∣ ≤ ∣ D T ∣ = ∣ A R − B S ∣ R . (22) D_t=\frac{AR-BS}{N_t}\quad\text{and}\quad|D_t|\leq|D_T|=\frac{|AR-BS|}{R}. \tag*{(22)} D t = N t A R − B S and ∣ D t ∣ ≤ ∣ D T ∣ = R ∣ A R − B S ∣ . (22)
Proof. First, since K T = − S R K_T=-\frac{S}{R} K T = − R S as P T + 1 = 0 P_{T+1}=0 P T + 1 = 0 , we get
D T = A + B K T = A R − B S R and D t = A + B K t = A R − B S R + B 2 P t + 1 . D_T=A+BK_T=\frac{AR-BS}{R}\quad\text{and}\quad D_t=A+BK_t=\frac{AR-BS}{R+B^2P_{t+1}}. D T = A + B K T = R A R − B S and D t = A + B K t = R + B 2 P t + 1 A R − B S .
Then the result is clear.
:::
Lemma 5. There exists C P < ∞ C_P<\infty C P < ∞ and ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) such that for all T , t T,t T , t
∥ P t − P ∥ ≤ C P ρ T − t . \lVert P_t-P\rVert\leq C_P\rho^{T-t}. ∥ P t − P ∥ ≤ C P ρ T − t .
Proof. Denote P 0 ( T ) P_0^{(T)} P 0 ( T ) the solution to the DARE with time horizon T T T . From the proof of Corollary 2 of Hager and Horowitz, [10 ] we have that there exists C P C_P C P and 0 < ρ < 1 0 < \rho< 1 0 < ρ < 1 such that
∥ P 0 ( T ) − P ∥ ≤ C P ρ T (23) \left\lVert P_0^{(T)} - P \right\rVert\le C_P \rho^T \tag*{(23)} P 0 ( T ) − P ≤ C P ρ T (23)
Observing that P 0 ( T − t ) = P t ( T ) P_0^{(T-t)} = P_t^{(T)} P 0 ( T − t ) = P t ( T ) , it follows that
∥ P t ( T ) − P ∥ = ∥ P 0 ( T − t ) − P ∥ ≤ C P ρ T − t . (24) \left\lVert P_t^{(T)} - P \right\rVert= \left\lVert P_0^{(T-t)} - P \right\rVert\le C_P \rho^{T-t}. \tag*{(24)} P t ( T ) − P = P 0 ( T − t ) − P ≤ C P ρ T − t . (24)
This completes the proof.
We introduce two corollaries of Lemma 5, the second of which will play a key role in our regret analyses. Define
Γ = K 2 N 1 − A 2 \Gamma= \frac{K^2N}{1-A^2} Γ = 1 − A 2 K 2 N
which is a stationary counterpart of Γ t \Gamma_t Γ t defined in (5) . We have the following two lemmas.
Lemma 6. There exists C Γ < ∞ C_\Gamma< \infty C Γ < ∞ and η ∈ ( 0 , 1 ) \eta\in(0,1) η ∈ ( 0 , 1 ) such that, for all T , t T,t T , t ,
∣ Γ t − Γ ∣ ≤ C Γ η T − t . \lvert\Gamma_t - \Gamma\rvert\le C_\Gamma\eta^{T-t}. ∣ Γ t − Γ ∣ ≤ C Γ η T − t . Proof. By Lemma 3, we have Γ t = G t − P t \Gamma_t = G_t - P_t Γ t = G t − P t . The recursion of G t G_t G t in Lemma 3 implies that
G t = Q ( 1 − A 2 ( T − t + 1 ) ) 1 − A 2 . G_t = \frac{Q(1-A^{2(T-t+1)})}{1-A^2}. G t = 1 − A 2 Q ( 1 − A 2 ( T − t + 1 ) ) . Hence, we have
Γ t + 1 = G t + 1 − P t + 1 = Q ( 1 − A 2 ( T − t ) ) 1 − A 2 − P t + 1 . \Gamma_{t+1} = G_{t+1} - P_{t+1} = \frac{Q(1-A^{2(T-t)})}{1-A^2} - P_{t+1}. Γ t + 1 = G t + 1 − P t + 1 = 1 − A 2 Q ( 1 − A 2 ( T − t ) ) − P t + 1 . Furthermore
Γ = N K 2 1 − A 2 = Q + P A 2 − P 1 − A 2 = Q 1 − A 2 − P . \Gamma= \frac{NK^2}{1-A^2} = \frac{Q+PA^2-P}{1-A^2} = \frac{Q}{1-A^2} - P. Γ = 1 − A 2 N K 2 = 1 − A 2 Q + P A 2 − P = 1 − A 2 Q − P . So
∣ Γ t + 1 − Γ ∣ = ∣ Q ( 1 − A 2 ( T − t ) ) 1 − A 2 − P t + 1 − Q 1 − A 2 + P ∣ ≤ Q 1 − A 2 ∣ A 2 ( T − t ) ∣ + ∣ P t + 1 − P ∣ ≤ Q 1 − A 2 ∣ A 2 ∣ T − t + C P ρ T − t − 1 \begin{aligned}
\lvert\Gamma_{t+1} - \Gamma\rvert
&= \left\lvert\frac{Q(1-A^{2(T-t)})}{1-A^2} - P_{t+1} - \frac{Q}{1-A^2} + P \right\rvert\\
&\le\frac{Q}{1-A^2}\lvert A^{2(T-t)}\rvert+ \lvert P_{t+1}-P\rvert\\
&\le\frac{Q}{1-A^2}\lvert A^2\rvert^{T-t} + C_P\rho^{T-t-1}
\end{aligned} ∣ Γ t + 1 − Γ ∣ = 1 − A 2 Q ( 1 − A 2 ( T − t ) ) − P t + 1 − 1 − A 2 Q + P ≤ 1 − A 2 Q ∣ A 2 ( T − t ) ∣ + ∣ P t + 1 − P ∣ ≤ 1 − A 2 Q ∣ A 2 ∣ T − t + C P ρ T − t − 1 (Lemma 5) for some C P < ∞ C_P < \infty C P < ∞ and ρ ∈ ( 0 , 1 ) \rho\in(0,1) ρ ∈ ( 0 , 1 ) . Choosing C Γ = 2 max { Q A 2 1 − A 2 , C P } C_\Gamma= 2\max\left\{\frac{QA^2}{1-A^2}, C_P\right\} C Γ = 2 max { 1 − A 2 Q A 2 , C P } and η = max { ρ , A 2 } \eta= \max\{\rho,A^2\} η = max { ρ , A 2 } finishes the proof.
Lemma 7. Let h : R 2 → R h:\mathbb{R}^2 \to\mathbb{R} h : R 2 → R be continuous at ( P , Γ ) (P,\Gamma) ( P , Γ ) . If a < h ( P , Γ ) < b a < h(P,\Gamma) < b a < h ( P , Γ ) < b , then there exists t 0 ∈ N t_0 \in\mathbb{N} t 0 ∈ N such that
h ( P , Γ ) + a 2 < h ( P t , Γ t ) < h ( P , Γ ) + b 2 \frac{h(P,\Gamma)+a}{2} < h(P_t,\Gamma_t) < \frac{h(P,\Gamma)+b}{2} 2 h ( P , Γ ) + a < h ( P t , Γ t ) < 2 h ( P , Γ ) + b for all t , T t,T t , T satisfying t ≤ T − t 0 t \le T-t_0 t ≤ T − t 0 .
Proof. By the continuity of h h h , there exists δ \delta δ such that
∣ h ( P , Γ ) − h ( P ′ , Γ ′ ) ∣ < min { h ( P , Γ ) − a , b − h ( P , Γ ) } 2 (25) \left|h(P,\Gamma)-h(P',\Gamma')\right|<\frac{\min\{h(P,\Gamma)-a,\ b-h(P,\Gamma)\}}{2} \tag*{(25)} ∣ h ( P , Γ ) − h ( P ′ , Γ ′ ) ∣ < 2 min { h ( P , Γ ) − a , b − h ( P , Γ )} (25)
whenever ∣ P − P ′ ∣ + ∣ Γ − Γ ′ ∣ < δ \left|P-P'\right|+\left|\Gamma-\Gamma'\right|<\delta ∣ P − P ′ ∣ + ∣ Γ − Γ ′ ∣ < δ . That is if ∣ P − P ′ ∣ + ∣ Γ − Γ ′ ∣ < δ \left|P-P'\right|+\left|\Gamma-\Gamma'\right|<\delta ∣ P − P ′ ∣ + ∣ Γ − Γ ′ ∣ < δ then
h ( P ′ , Γ ′ ) ≤ h ( P , Γ ) + ∣ h ( P , Γ ) − h ( P ′ , Γ ′ ) ∣ < h ( P , Γ ) + b − h ( P , Γ ) 2 = h ( P , Γ ) + b 2 , h(P',\Gamma')\leq h(P,\Gamma)+\left|h(P,\Gamma)-h(P',\Gamma')\right|<h(P,\Gamma)+\frac{b-h(P,\Gamma)}{2}=\frac{h(P,\Gamma)+b}{2}, h ( P ′ , Γ ′ ) ≤ h ( P , Γ ) + ∣ h ( P , Γ ) − h ( P ′ , Γ ′ ) ∣ < h ( P , Γ ) + 2 b − h ( P , Γ ) = 2 h ( P , Γ ) + b ,
and
h ( P ′ , Γ ′ ) ≥ h ( P , Γ ) − ∣ h ( P , Γ ) − h ( P ′ , Γ ′ ) ∣ > h ( P , Γ ) − h ( P , Γ ) − a 2 = h ( P , Γ ) + a 2 . h(P',\Gamma')\geq h(P,\Gamma)-\left|h(P,\Gamma)-h(P',\Gamma')\right|>h(P,\Gamma)-\frac{h(P,\Gamma)-a}{2}=\frac{h(P,\Gamma)+a}{2}. h ( P ′ , Γ ′ ) ≥ h ( P , Γ ) − ∣ h ( P , Γ ) − h ( P ′ , Γ ′ ) ∣ > h ( P , Γ ) − 2 h ( P , Γ ) − a = 2 h ( P , Γ ) + a .
By Lemmas 5 and 6, choose t 0 t_0 t 0 such that C P ρ t 0 + C Γ η t 0 < δ C_P\rho^{t_0}+C_\Gamma\eta^{t_0}<\delta C P ρ t 0 + C Γ η t 0 < δ where C P , C Γ > 0 C_P,C_\Gamma>0 C P , C Γ > 0 and ρ , η ∈ ( 0 , 1 ) \rho,\eta\in(0,1) ρ , η ∈ ( 0 , 1 ) satisfy
∣ P t − P ∣ ≤ C P ρ T − t ≤ C P ρ t 0 and ∣ Γ t − Γ ∣ ≤ C Γ η T − t ≤ C Γ η t 0 . \left|P_t-P\right|\leq C_P\rho^{T-t}\leq C_P\rho^{t_0}\quad\text{and}\quad\left|\Gamma_t-\Gamma\right|\leq C_\Gamma\eta^{T-t}\leq C_\Gamma\eta^{t_0}. ∣ P t − P ∣ ≤ C P ρ T − t ≤ C P ρ t 0 and ∣ Γ t − Γ ∣ ≤ C Γ η T − t ≤ C Γ η t 0 .
Applying the continuity estimate above with ( P ′ , Γ ′ ) = ( P t , Γ t ) (P',\Gamma')=(P_t,\Gamma_t) ( P ′ , Γ ′ ) = ( P t , Γ t ) gives
h ( P , Γ ) + a 2 < h ( P t , Γ t ) < h ( P , Γ ) + b 2 . \frac{h(P,\Gamma)+a}{2}<h(P_t,\Gamma_t)<\frac{h(P,\Gamma)+b}{2}. 2 h ( P , Γ ) + a < h ( P t , Γ t ) < 2 h ( P , Γ ) + b .
as claimed.
Remark 2. Note that since N t N_t N t , K t K_t K t , D t D_t D t are continuous mappings of P t + 1 P_{t+1} P t + 1 , Lemma 7 holds for continuous functions of N t N_t N t , K t K_t K t , D t D_t D t as well.
Proof of Theorem 1
In the proof of Theorem 1, we assume that ρ ( A ) < 1 \rho(A)<1 ρ ( A ) < 1 , as we assumed for our bilinear setting.
Proof. By the completion of squares used by Eq. (A.7)
c ( x , u ) = ( u − K t x ) T N t ( u − K t x ) − ( A x + B u ) T P t + 1 ( A x + B u ) + x T P t x . (26) c(x,u)=(u-K_tx)^{\mathsf{T}}N_t(u-K_tx)-(Ax+Bu)^{\mathsf{T}}P_{t+1}(Ax+Bu)+x^{\mathsf{T}}P_tx. \tag*{(26)} c ( x , u ) = ( u − K t x ) T N t ( u − K t x ) − ( A x + B u ) T P t + 1 ( A x + B u ) + x T P t x . (26)
Hence
c ( x t , u t ) = ( u t − K t x t ) T N t ( u t − K t x t ) − ( A x t + B u t ) T P t + 1 ( A x t + B u t ) + x t T P t x t . (27) c(x_t,u_t)=(u_t-K_tx_t)^{\mathsf{T}}N_t(u_t-K_tx_t)-(Ax_t+Bu_t)^{\mathsf{T}}P_{t+1}(Ax_t+Bu_t)+x_t^{\mathsf{T}}P_tx_t. \tag*{(27)} c ( x t , u t ) = ( u t − K t x t ) T N t ( u t − K t x t ) − ( A x t + B u t ) T P t + 1 ( A x t + B u t ) + x t T P t x t . (27)
Since x t + 1 ∼ N ( A x t + B u t , Σ w ) x_{t+1}\sim\mathcal{N}(Ax_t+Bu_t,\Sigma_w) x t + 1 ∼ N ( A x t + B u t , Σ w ) we have
c ( x t , u t ) = ( u t − K t x t ) T N t ( u t − K t x t ) − E [ x t + 1 T P t + 1 x t + 1 ] + x t T P t x t + tr P t + 1 Σ w . (28) c(x_t,u_t)=(u_t-K_tx_t)^{\mathsf{T}}N_t(u_t-K_tx_t)-\mathbb{E}[x_{t+1}^{\mathsf{T}}P_{t+1}x_{t+1}]+x_t^{\mathsf{T}}P_tx_t+\operatorname{tr}P_{t+1}\Sigma_w. \tag*{(28)} c ( x t , u t ) = ( u t − K t x t ) T N t ( u t − K t x t ) − E [ x t + 1 T P t + 1 x t + 1 ] + x t T P t x t + tr P t + 1 Σ w . (28)
Summing and applying the tower rule, we have
J T ( π ) = C T + ∑ t = 0 T E π [ ( u t − K t x t ) T N t ( u t − K t x t ) ] (29) J_T(\pi)=C_T+\sum_{t=0}^{T}\mathbb{E}_{\pi}\left[(u_t-K_tx_t)^{\mathsf{T}}N_t(u_t-K_tx_t)\right] \tag*{(29)} J T ( π ) = C T + t = 0 ∑ T E π [ ( u t − K t x t ) T N t ( u t − K t x t ) ] (29)
where
C T = E [ x 0 T P 0 x 0 ] + ∑ t = 0 T tr ( P t + 1 Σ w ) . (30) C_T=\mathbb{E}\left[x_0^{\mathsf{T}}P_0x_0\right]+\sum_{t=0}^{T}\operatorname{tr}\left(P_{t+1}\Sigma_w\right). \tag*{(30)} C T = E [ x 0 T P 0 x 0 ] + t = 0 ∑ T tr ( P t + 1 Σ w ) . (30)
Observe that
∑ t = 0 T E π C E [ ( u t − K t x t ) ⊤ N t ( u t − K t x t ) ] = ∑ t = 0 T E π C E [ ( x ^ t − x t ) ⊤ K t ⊤ N t K t ( x ^ t − x t ) ] = ∑ t = 0 T tr ( K t ⊤ N t K t Σ t C E ) . \sum_{t=0}^{T}\mathbb{E}_{\pi^{CE}}\left[(u_t-K_tx_t)^\top N_t(u_t-K_tx_t)\right]
=
\sum_{t=0}^{T}\mathbb{E}_{\pi^{CE}}\left[(\hat{x}_t-x_t)^\top K_t^\top N_tK_t(\hat{x}_t-x_t)\right]
=
\sum_{t=0}^{T}\operatorname{tr}\left(K_t^\top N_tK_t\Sigma_t^{CE}\right). t = 0 ∑ T E π C E [ ( u t − K t x t ) ⊤ N t ( u t − K t x t ) ] = t = 0 ∑ T E π C E [ ( x ^ t − x t ) ⊤ K t ⊤ N t K t ( x ^ t − x t ) ] = t = 0 ∑ T tr ( K t ⊤ N t K t Σ t C E ) .
Similarly,
∑ t = 0 T E π O L [ ( u t − K t x t ) ⊤ N t ( u t − K t x t ) ] = ∑ t = 0 T E π O L [ ( m t − x t ) ⊤ K t ⊤ N t K t ( m t − x t ) ] = ∑ t = 0 T tr ( K t ⊤ N t K t Σ ˉ ) . \sum_{t=0}^{T}\mathbb{E}_{\pi^{OL}}\left[(u_t-K_tx_t)^\top N_t(u_t-K_tx_t)\right]
=
\sum_{t=0}^{T}\mathbb{E}_{\pi^{OL}}\left[(m_t-x_t)^\top K_t^\top N_tK_t(m_t-x_t)\right]
=
\sum_{t=0}^{T}\operatorname{tr}\left(K_t^\top N_tK_t\bar{\Sigma}\right). t = 0 ∑ T E π O L [ ( u t − K t x t ) ⊤ N t ( u t − K t x t ) ] = t = 0 ∑ T E π O L [ ( m t − x t ) ⊤ K t ⊤ N t K t ( m t − x t ) ] = t = 0 ∑ T tr ( K t ⊤ N t K t Σ ˉ ) .
Hence
R T ( π C E , π O L ) = ∑ t = 0 T tr ( K t ⊤ N t K t ( Σ ˉ − Σ t C E ) ) (31) R_T(\pi^{CE},\pi^{OL})=\sum_{t=0}^{T}\operatorname{tr}\left(K_t^\top N_tK_t\left(\bar{\Sigma}-\Sigma_t^{CE}\right)\right) \tag*{(31)} R T ( π C E , π O L ) = t = 0 ∑ T tr ( K t ⊤ N t K t ( Σ ˉ − Σ t C E ) ) (31)
Observe that since observations are action-independent, Σ t C E \Sigma_t^{CE} Σ t C E is deterministic, and it converges to Σ ∞ \Sigma_\infty Σ ∞ under stability assumption, ρ ( A ) < 1 \rho(A)<1 ρ ( A ) < 1 . Moreover, continuity argument in Lemma 7 implies that, as long as χ : = tr ( K ⊤ N K ( Σ ˉ − Σ ∞ ) ) > 0 \chi:=\operatorname{tr}\left(K^\top NK\left(\bar{\Sigma}-\Sigma_\infty\right)\right)>0 χ := tr ( K ⊤ N K ( Σ ˉ − Σ ∞ ) ) > 0 , we can choose s 0 s_0 s 0 and s 1 s_1 s 1 , independent of T T T , such that
tr ( K t ⊤ N t K t ( Σ ˉ − Σ t C E ) ) ≥ χ 2 \operatorname{tr}\left(K_t^\top N_tK_t\left(\bar{\Sigma}-\Sigma_t^{CE}\right)\right)\geq\frac{\chi}{2} tr ( K t ⊤ N t K t ( Σ ˉ − Σ t C E ) ) ≥ 2 χ
whenever s 0 ≤ t ≤ T − s 1 s_0\leq t\leq T-s_1 s 0 ≤ t ≤ T − s 1 . Therefore, we get R T ( π C E , π O L ) = Ω ( T ) R_T(\pi^{CE},\pi^{OL})=\Omega(T) R T ( π C E , π O L ) = Ω ( T ) .
Remark 3. In the proof of Theorem 1, we assumed that A A A is stable for the sake of clear comparison. However, under stabilizability and detectability assumptions, a similar result can be shown when A A A is unstable. The stability assumption ensures that the open-loop covariance remains bounded. Under stabilizability and detectability, unstable systems can exhibit a faster-growing advantage over open-loop control. Here we restrict attention to stable systems.
Bilinear Kalman Filtering Tools
In this section, we provide technical lemmas related to Kalman filtering that are useful to prove the main results. Throughout the remaining sections, define F t = σ ( H t ) \mathcal{F}_t=\sigma(H_t) F t = σ ( H t ) , the σ \sigma σ -algebra generated by the history H t H_t H t .
Proof of Lemma 1
Proof. Define the innovation and its conditional variance by
e t : = y t − u t ⊤ C x ^ t = u t ⊤ C ( x t − x ^ t ) + z t , s t : = σ z 2 + u t ⊤ C Σ t C ⊤ u t e_t:=y_t-u_t^\top C\hat{x}_t=u_t^\top C(x_t-\hat{x}_t)+z_t,\qquad s_t:=\sigma_z^2+u_t^\top C\Sigma_tC^\top u_t e t := y t − u t ⊤ C x ^ t = u t ⊤ C ( x t − x ^ t ) + z t , s t := σ z 2 + u t ⊤ C Σ t C ⊤ u t
respectively. Since u t u_t u t is F t \mathcal{F}_t F t -measurable, the posterior state distribution and the observation model imply
E [ e t ∣ F t ] = 0 and E [ e t 2 ∣ F t ] = s t . \mathbb{E}[e_t\mid\mathcal{F}_t]=0\qquad\text{and}\qquad\mathbb{E}[e_t^2\mid\mathcal{F}_t]=s_t. E [ e t ∣ F t ] = 0 and E [ e t 2 ∣ F t ] = s t .
By (1) , we have
x ^ t + 1 = A x ^ t + B u t + A Σ t C T u t s t e t . \hat{x}_{t+1}=A\hat{x}_t+Bu_t+\frac{A\Sigma_t C^{\mathsf T}u_t}{s_t}e_t. x ^ t + 1 = A x ^ t + B u t + s t A Σ t C T u t e t .
All coefficients in this expression are H t \mathcal{H}_t H t -measurable. Therefore,
E [ x ^ t + 1 x ^ t + 1 T ∣ F t ] = ( A x ^ t + B u t ) ( A x ^ t + B u t ) T + A Σ t C T u t u t T C Σ t A T s t 2 E [ e t 2 ∣ F t ] = ( A x ^ t + B u t ) ( A x ^ t + B u t ) T + F ( Σ t , u t ) , \begin{aligned}
\mathbb{E}[\hat{x}_{t+1}\hat{x}_{t+1}^{\mathsf T}\mid\mathcal{F}_t]=(A\hat{x}_t+Bu_t)(A\hat{x}_t+Bu_t)^{\mathsf T}+\frac{A\Sigma_t C^{\mathsf T}u_tu_t^{\mathsf T}C\Sigma_t A^{\mathsf T}}{s_t^2}\mathbb{E}[e_t^2\mid\mathcal{F}_t] \\
&=(A\hat{x}_t+Bu_t)(A\hat{x}_t+Bu_t)^{\mathsf T}+F(\Sigma_t,u_t),
\end{aligned} E [ x ^ t + 1 x ^ t + 1 T ∣ F t ] = ( A x ^ t + B u t ) ( A x ^ t + B u t ) T + s t 2 A Σ t C T u t u t T C Σ t A T E [ e t 2 ∣ F t ] = ( A x ^ t + B u t ) ( A x ^ t + B u t ) T + F ( Σ t , u t ) ,
where the last equality follows from the definition of F F F .
Lemma 8. For t ≥ 0 t\geq0 t ≥ 0
Σ w ⪯ Σ t ⪯ Σ ˉ . \Sigma_w\preceq\Sigma_t\preceq\bar{\Sigma}. Σ w ⪯ Σ t ⪯ Σ ˉ . Proof. For the base case, observe that
Σ w ⪯ A Σ ˉ A T + Σ w = Σ ˉ = Σ 0 . \Sigma_w\preceq A\bar{\Sigma}A^{\mathsf T}+\Sigma_w=\bar{\Sigma}=\Sigma_0. Σ w ⪯ A Σ ˉ A T + Σ w = Σ ˉ = Σ 0 . Let t ≥ 0 t\geq0 t ≥ 0 . Assume Σ t ⪯ Σ ˉ \Sigma_t\preceq\bar{\Sigma} Σ t ⪯ Σ ˉ . For the upper bound, positive semidefiniteness of F ( Σ , u ) F(\Sigma,u) F ( Σ , u ) , the induction hypothesis, and the definition of Σ ˉ \bar{\Sigma} Σ ˉ imply
Σ t + 1 = A Σ t A T + Σ w − F ( Σ t , u t ) ⪯ A Σ t A T + Σ w ⪯ A Σ ˉ A T + Σ w = Σ ˉ . \Sigma_{t+1}=A\Sigma_tA^{\mathsf T}+\Sigma_w-F(\Sigma_t,u_t)\preceq A\Sigma_tA^{\mathsf T}+\Sigma_w\preceq A\bar{\Sigma}A^{\mathsf T}+\Sigma_w=\bar{\Sigma}. Σ t + 1 = A Σ t A T + Σ w − F ( Σ t , u t ) ⪯ A Σ t A T + Σ w ⪯ A Σ ˉ A T + Σ w = Σ ˉ . For the lower bound we aim to show
Σ t + 1 = A Σ t A T + Σ w − F ( Σ t , u t ) = A ( Σ t − Σ t C T u t u t T C Σ t σ z 2 + u t T C Σ t C T u t ) A T + Σ w ⪰ Σ w . \Sigma_{t+1}=A\Sigma_tA^{\mathsf T}+\Sigma_w-F(\Sigma_t,u_t)=A\left(\Sigma_t-\frac{\Sigma_tC^{\mathsf T}u_tu_t^{\mathsf T}C\Sigma_t}{\sigma_z^2+u_t^{\mathsf T}C\Sigma_tC^{\mathsf T}u_t}\right)A^{\mathsf T}+\Sigma_w\succeq\Sigma_w. Σ t + 1 = A Σ t A T + Σ w − F ( Σ t , u t ) = A ( Σ t − σ z 2 + u t T C Σ t C T u t Σ t C T u t u t T C Σ t ) A T + Σ w ⪰ Σ w . It suffices to show
Σ t C T u t u t T C Σ t σ z 2 + u t T C Σ t C T u t ⪯ Σ t , \frac{\Sigma_tC^{\mathsf T}u_tu_t^{\mathsf T}C\Sigma_t}{\sigma_z^2+u_t^{\mathsf T}C\Sigma_tC^{\mathsf T}u_t}\preceq\Sigma_t, σ z 2 + u t T C Σ t C T u t Σ t C T u t u t T C Σ t ⪯ Σ t , i.e.,
Σ t 1 / 2 C T u t u t T C Σ t 1 / 2 σ z 2 + u t T C Σ t C T u t ⪯ I . \frac{\Sigma_t^{1/2}C^{\mathsf T}u_tu_t^{\mathsf T}C\Sigma_t^{1/2}}{\sigma_z^2+u_t^{\mathsf T}C\Sigma_tC^{\mathsf T}u_t}\preceq I. σ z 2 + u t T C Σ t C T u t Σ t 1/2 C T u t u t T C Σ t 1/2 ⪯ I . Observe that
∥ Σ t 1 / 2 C T u t u t T C Σ t 1 / 2 σ z 2 + u t T C Σ t C T u t ∥ = u t T C Σ t C T u t σ z 2 + u t T C Σ t C T u t ≤ 1. \left\|\frac{\Sigma_t^{1/2}C^{\mathsf T}u_tu_t^{\mathsf T}C\Sigma_t^{1/2}}{\sigma_z^2+u_t^{\mathsf T}C\Sigma_tC^{\mathsf T}u_t}\right\|=\frac{u_t^{\mathsf T}C\Sigma_tC^{\mathsf T}u_t}{\sigma_z^2+u_t^{\mathsf T}C\Sigma_tC^{\mathsf T}u_t}\leq1. σ z 2 + u t T C Σ t C T u t Σ t 1/2 C T u t u t T C Σ t 1/2 = σ z 2 + u t T C Σ t C T u t u t T C Σ t C T u t ≤ 1. The result follows from the fact that, for M ⪰ 0 M\succeq0 M ⪰ 0 we have ∥ M ∥ ≤ 1 ⟺ M ⪯ I \|M\|\leq1\Longleftrightarrow M\preceq I ∥ M ∥ ≤ 1 ⟺ M ⪯ I .
Lemma 9. For each t t t , define
Z t = y t − u t T C x ^ t σ z 2 + C 2 Σ t u t 2 Z_t=\frac{y_t-u_t^{\mathsf T}C\hat{x}_t}{\sqrt{\sigma_z^2+C^2\Sigma_tu_t^2}} Z t = σ z 2 + C 2 Σ t u t 2 y t − u t T C x ^ t where u t u_t u t is F t \mathcal{F}_t F t -measurable. Then Z 0 , Z 1 , … Z_0,Z_1,\ldots Z 0 , Z 1 , … are independent standard Gaussian random variables, and each Z t Z_t Z t is independent of F t \mathcal{F}_t F t . Moreover, almost surely,
x ^ t + 1 = A x ^ t + B u t + A Σ t C u t σ z 2 + C 2 Σ t u t 2 Z t . (32) \hat{x}_{t+1}=A\hat{x}_t+Bu_t+\frac{A\Sigma_tCu_t}{\sqrt{\sigma_z^2+C^2\Sigma_tu_t^2}}Z_t. \tag*{(32)} x ^ t + 1 = A x ^ t + B u t + σ z 2 + C 2 Σ t u t 2 A Σ t C u t Z t . (32) In particular,
x ^ t + 1 ∣ F t ∼ N ( A x ^ t + B u t , F ( Σ t , u t ) ) . \hat{x}_{t+1}\mid\mathcal{F}_t\sim\mathcal{N}(A\hat{x}_t+Bu_t,F(\Sigma_t,u_t)). x ^ t + 1 ∣ F t ∼ N ( A x ^ t + B u t , F ( Σ t , u t )) .
Proof. Since y t = u t C x t + z t y_t=u_t Cx_t+z_t y t = u t C x t + z t , the innovation can be written as
y t − u t C x ^ t = u t C ( x t − x ^ t ) + z t . y_t-u_tC\hat{x}_t=u_tC\left(x_t-\hat{x}_t\right)+z_t. y t − u t C x ^ t = u t C ( x t − x ^ t ) + z t .
Condition on F t \mathcal{F}_t F t . The action u t u_t u t is F t \mathcal{F}_t F t -measurable, x t − x ^ t ∣ F t ∼ N ( 0 , Σ t ) x_t-\hat{x}_t\mid\mathcal{F}_t\sim\mathcal{N}(0,\Sigma_t) x t − x ^ t ∣ F t ∼ N ( 0 , Σ t ) . The measurement noise z t ∼ N ( 0 , σ z 2 ) z_t\sim\mathcal{N}(0,\sigma_z^2) z t ∼ N ( 0 , σ z 2 ) is independent of ( x t , F t ) (x_t,\mathcal{F}_t) ( x t , F t ) . Hence
y t − u t C x ^ t ∣ F t ∼ N ( 0 , σ z 2 + C 2 Σ t u t 2 ) , y_t-u_tC\hat{x}_t\mid\mathcal{F}_t\sim\mathcal{N}\left(0,\sigma_z^2+C^2\Sigma_tu_t^2\right), y t − u t C x ^ t ∣ F t ∼ N ( 0 , σ z 2 + C 2 Σ t u t 2 ) ,
and therefore Z t ∣ F t ∼ N ( 0 , 1 ) Z_t\mid\mathcal{F}_t\sim\mathcal{N}(0,1) Z t ∣ F t ∼ N ( 0 , 1 ) . This conditional law does not depend on F t \mathcal{F}_t F t , so Z t Z_t Z t is independent of F t \mathcal{F}_t F t and Z t ∼ N ( 0 , 1 ) Z_t\sim\mathcal{N}(0,1) Z t ∼ N ( 0 , 1 ) . Since y t y_t y t , u t u_t u t , x ^ t \hat{x}_t x ^ t and Σ t \Sigma_t Σ t are all F t + 1 \mathcal{F}_{t+1} F t + 1 -measurable, so is Z t Z_t Z t . Finally, substituting
y t − u t C x ^ t = σ z 2 + C 2 Σ t u t 2 Z t y_t-u_tC\hat{x}_t=\sqrt{\sigma_z^2+C^2\Sigma_tu_t^2}Z_t y t − u t C x ^ t = σ z 2 + C 2 Σ t u t 2 Z t
into the recursion (1) gives
x ^ t + 1 = A x ^ t + B u t + A Σ t C u t σ z 2 + C 2 Σ t u t 2 σ z 2 + C 2 Σ t u t 2 Z t , \hat{x}_{t+1}=A\hat{x}_t+Bu_t+\frac{A\Sigma_tCu_t}{\sigma_z^2+C^2\Sigma_tu_t^2}\sqrt{\sigma_z^2+C^2\Sigma_tu_t^2}Z_t, x ^ t + 1 = A x ^ t + B u t + σ z 2 + C 2 Σ t u t 2 A Σ t C u t σ z 2 + C 2 Σ t u t 2 Z t ,
which simplifies to (B.1).
Lemma 10. Under π C E \pi^{CE} π C E , define
M t = D t + A Σ t C K t σ z 2 + K t 2 C 2 Σ t x ^ t 2 ( y t − K t C x ^ t 2 ) , q t = ∣ A C K t ∣ Σ t σ z 2 + K t 2 C 2 Σ t x ^ t 2 . M_t=D_t+\frac{A\Sigma_tCK_t}{\sigma_z^2+K_t^2C^2\Sigma_t\hat{x}_t^2}\left(y_t-K_tC\hat{x}_t^2\right),\qquad q_t=\frac{\lvert ACK_t\rvert\Sigma_t}{\sqrt{\sigma_z^2+K_t^2C^2\Sigma_t\hat{x}_t^2}}. M t = D t + σ z 2 + K t 2 C 2 Σ t x ^ t 2 A Σ t C K t ( y t − K t C x ^ t 2 ) , q t = σ z 2 + K t 2 C 2 Σ t x ^ t 2 ∣ A C K t ∣ Σ t . Then,
(i) x ^ t + 1 = M t x ^ t \hat{x}_{t+1}=M_t\hat{x}_t x ^ t + 1 = M t x ^ t for all t t t .
(ii) Conditionally on F t \mathcal{F}_t F t , M t = d D t + q t Z M_t\mathrel{\overset{d}{=}}D_t+q_tZ M t = d D t + q t Z , where Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) is independent of F t \mathcal{F}_t F t .
Proof.
(i) Under π C E \pi^{CE} π C E , we have y t − u t C x ^ t = y t − K t C x ^ t 2 y_t-u_tC\hat{x}_t=y_t-K_tC\hat{x}_t^2 y t − u t C x ^ t = y t − K t C x ^ t 2 . Substituting into the recursion (1) ,
x ^ t + 1 = ( A + B K t ) x ^ t + A Σ t C K t x ^ t σ z 2 + K t 2 C 2 Σ t x ^ t 2 ( y t − K t C x ^ t 2 ) = M t x ^ t . \hat{x}_{t+1}=(A+BK_t)\hat{x}_t+\frac{A\Sigma_tCK_t\hat{x}_t}{\sigma_z^2+K_t^2C^2\Sigma_t\hat{x}_t^2}\left(y_t-K_tC\hat{x}_t^2\right)=M_t\hat{x}_t. x ^ t + 1 = ( A + B K t ) x ^ t + σ z 2 + K t 2 C 2 Σ t x ^ t 2 A Σ t C K t x ^ t ( y t − K t C x ^ t 2 ) = M t x ^ t . (ii) By the definition of Z t Z_t Z t in Lemma 9 and u t = K t x ^ t u_t=K_t\hat{x}_t u t = K t x ^ t ,
M t = D t + A Σ t C K t σ z 2 + K t 2 C 2 Σ t x ^ t 2 Z t M_t=D_t+\frac{A\Sigma_tCK_t}{\sqrt{\sigma_z^2+K_t^2C^2\Sigma_t\hat{x}_t^2}}Z_t M t = D t + σ z 2 + K t 2 C 2 Σ t x ^ t 2 A Σ t C K t Z t where Z t ∼ N ( 0 , 1 ) Z_t\sim\mathcal{N}(0,1) Z t ∼ N ( 0 , 1 ) is independent of F t \mathcal{F}_t F t . The coefficient of Z t Z_t Z t is F t \mathcal{F}_t F t -measurable and its square equals q t 2 q_t^2 q t 2 . Therefore,
M t ∣ F t ∼ N ( D t , q t 2 ) , M_t\mid\mathcal{F}_t\sim\mathcal{N}(D_t,q_t^2), M t ∣ F t ∼ N ( D t , q t 2 ) , which is the conditional distribution of D t + q t Z D_t+q_tZ D t + q t Z for Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) independent of F t \mathcal{F}_t F t .
Lemma 11. Under π C E \pi^{CE} π C E , there exists an integer s ≥ 1 s\geq1 s ≥ 1 , independent of T T T , such that
E π C E [ x ^ t 2 ] ≤ C C E , with C C E : = M 2 s ( μ x 2 + A 2 Σ ( 1 1 − ( D ′ ) 2 + s ) ) (33) \mathbb{E}_{\pi^{CE}}[\hat{x}_t^2]\leq C_{CE},\qquad\text{with }C_{CE}:=M^{2s}\left(\mu_x^2+A^2\Sigma\left(\frac{1}{1-(D')^2}+s\right)\right) \tag*{(33)} E π C E [ x ^ t 2 ] ≤ C C E , with C C E := M 2 s ( μ x 2 + A 2 Σ ( 1 − ( D ′ ) 2 1 + s ) ) (33) for every horizon T T T and 0 ≤ t ≤ T 0\leq t\leq T 0 ≤ t ≤ T , where
D ′ = 1 + D 2 2 and M = max { 1 , ∣ A R − B S ∣ R } . (34) D'=\frac{1+D^2}{2}\qquad\text{and}\qquad M=\max\left\{1,\frac{\lvert AR-BS\rvert}{R}\right\}. \tag*{(34)} D ′ = 2 1 + D 2 and M = max { 1 , R ∣ A R − B S ∣ } . (34)
Proof. Write b t = E π C E [ x ^ t 2 ] b_t = \mathbb{E}_{\pi_{CE}}[\hat{x}_t^2] b t = E π C E [ x ^ t 2 ] , so that b 0 = μ x 2 b_0 = \mu_x^2 b 0 = μ x 2 . By Lemma 2, we have ∣ D ∣ < 1 |D| < 1 ∣ D ∣ < 1 , and hence,
∣ D ∣ < D ′ = 1 + D 2 2 < 1. |D| < D' = \frac{1 + D^2}{2} < 1. ∣ D ∣ < D ′ = 2 1 + D 2 < 1.
From Lemma 7, choose an integer s ≥ 1 s \ge1 s ≥ 1 , independent of T T T , such that ∣ D t ∣ ≤ D ′ |D_t| \le D' ∣ D t ∣ ≤ D ′ whenever t ≤ T − s t \le T-s t ≤ T − s . Also, Lemma 4 gives ∣ D t ∣ ≤ M |D_t| \le M ∣ D t ∣ ≤ M for every t t t . By Lemma 8, we have
0 ≤ F ( Σ t , K t x ^ t ) = A 2 K t 2 C 2 Σ t 2 x ^ t 2 σ z 2 + K t 2 C 2 Σ t x ^ t 2 ≤ A 2 Σ t ≤ A 2 Σ ˉ . 0 \le F(\Sigma_t, K_t\hat{x}_t) = \frac{A^2K_t^2C^2\Sigma_t^2\hat{x}_t^2}{\sigma_z^2 + K_t^2C^2\Sigma_t\hat{x}_t^2} \le A^2\Sigma_t \le A^2\bar{\Sigma}. 0 ≤ F ( Σ t , K t x ^ t ) = σ z 2 + K t 2 C 2 Σ t x ^ t 2 A 2 K t 2 C 2 Σ t 2 x ^ t 2 ≤ A 2 Σ t ≤ A 2 Σ ˉ .
Therefore, Lemma 1 and the tower property give
b t + 1 = D t 2 b t + E π C E [ F ( Σ t , K t x ^ t ) ] ≤ D t 2 b t + A 2 Σ ˉ . b_{t+1} = D_t^2b_t + \mathbb{E}_{\pi_{CE}}[F(\Sigma_t, K_t\hat{x}_t)] \le D_t^2b_t + A^2\bar{\Sigma}. b t + 1 = D t 2 b t + E π C E [ F ( Σ t , K t x ^ t )] ≤ D t 2 b t + A 2 Σ ˉ .
Since ∣ D t ∣ ≤ D ′ |D_t| \le D' ∣ D t ∣ ≤ D ′ for 0 ≤ t ≤ T − s 0 \le t \le T-s 0 ≤ t ≤ T − s , iteration gives
b t ≤ ( D ′ ) 2 t μ x 2 + A 2 Σ ˉ ∑ i = 0 t − 1 ( D ′ ) 2 i ≤ μ x 2 + A 2 Σ ˉ 1 − ( D ′ ) 2 = : C 0 b_t \le(D')^{2t}\mu_x^2 + A^2\bar{\Sigma}\sum_{i=0}^{t-1}(D')^{2i} \le\mu_x^2 + \frac{A^2\bar{\Sigma}}{1-(D')^2} =: C_0 b t ≤ ( D ′ ) 2 t μ x 2 + A 2 Σ ˉ i = 0 ∑ t − 1 ( D ′ ) 2 i ≤ μ x 2 + 1 − ( D ′ ) 2 A 2 Σ ˉ =: C 0
for 0 ≤ t ≤ T − s 0 \le t \le T-s 0 ≤ t ≤ T − s .
There are at most s s s remaining updates, so using ∣ D t ∣ ≤ M |D_t| \le M ∣ D t ∣ ≤ M and M ≥ 1 M \ge1 M ≥ 1 , for 0 ≤ j ≤ s 0 \le j \le s 0 ≤ j ≤ s we obtain
b T − s + j ≤ M 2 j b T − s + A 2 Σ ˉ ∑ i = 0 j − 1 M 2 i ≤ M 2 s ( C 0 + s A 2 Σ ˉ ) = M 2 s ( μ x 2 + A 2 Σ ˉ ( 1 1 − ( D ′ ) 2 + s ) ) = C C E . \begin{aligned}
b_{T-s+j} \le M^{2j}b_{T-s} + A^2\bar{\Sigma}\sum_{i=0}^{j-1}M^{2i} \\
&\le M^{2s}(C_0+sA^2\bar{\Sigma}) \\
&= M^{2s}\left(\mu_x^2 + A^2\bar{\Sigma}\left(\frac{1}{1-(D')^2}+s\right)\right) = C_{CE}.
\end{aligned} b T − s + j ≤ M 2 j b T − s + A 2 Σ ˉ i = 0 ∑ j − 1 M 2 i ≤ M 2 s ( C 0 + s A 2 Σ ˉ ) = M 2 s ( μ x 2 + A 2 Σ ˉ ( 1 − ( D ′ ) 2 1 + s ) ) = C C E .
Since C 0 ≤ C C E C_0 \le C_{CE} C 0 ≤ C C E , the bound holds for every 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T .
Regret decomposition
This section proves the regret decompositions in Proposition 1 and 7. Define the conditional expected stage cost by
c ˉ ( x ^ , Σ , u ) : = Q x ^ 2 + Q Σ + R u 2 + 2 u S x ^ . (35) \bar{c}(\hat{x}, \Sigma, u) := Q\hat{x}^2 + Q\Sigma+ Ru^2 + 2uS\hat{x}. \tag*{(35)} c ˉ ( x ^ , Σ , u ) := Q x ^ 2 + Q Σ + R u 2 + 2 u S x ^ . (35)
Since x ^ t = E [ x t ∣ F t ] \hat{x}_t = \mathbb{E}[x_t \mid\mathcal{F}_t] x ^ t = E [ x t ∣ F t ] and Σ t = Var ( x t ∣ F t ) \Sigma_t = \operatorname{Var}(x_t \mid\mathcal{F}_t) Σ t = Var ( x t ∣ F t ) , we have
E [ c ( x t , u t ) ∣ F t ] = c ˉ ( x ^ t , Σ t , u t ) . \mathbb{E}[c(x_t,u_t) \mid\mathcal{F}_t] = \bar{c}(\hat{x}_t,\Sigma_t,u_t). E [ c ( x t , u t ) ∣ F t ] = c ˉ ( x ^ t , Σ t , u t ) .
The proof of Proposition 1 uses an auxiliary quantity V t = P t x ^ t 2 + G t Σ t V_t = P_t\hat{x}_t^2 + G_t\Sigma_t V t = P t x ^ t 2 + G t Σ t to separate the cumulative cost into a policy-independent term and the accumulated quantities r t r_t r t . Lemma 13 provides a key identity for the separation, while Lemma 14 shows that the accumulated quantities r t r_t r t have zero expectation under the open-loop policy. Proposition 7 simply follows from Lemma 12, which shows that covariance reductions and uncertainty reductions give the same total weighted benefit. Figure 2 describes the proof dependencies for Section C.
Figure 2. Main proof dependencies for Section C. An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Proof of Proposition 1
Proof. We consider two cases: J T ( π ) < ∞ J_T(\pi)<\infty J T ( π ) < ∞ and J T ( π ) = ∞ J_T(\pi)=\infty J T ( π ) = ∞ .
First, suppose that J T ( π ) < ∞ J_T(\pi)<\infty J T ( π ) < ∞ . The tower property gives
J T ( π ) = ∑ t = 0 T E π [ c ~ ( x ^ t , Σ t , u t ) ] . J_T(\pi)=\sum_{t=0}^{T}\mathbb{E}_{\pi}\left[\tilde{c}(\hat{x}_t,\Sigma_t,u_t)\right]. J T ( π ) = t = 0 ∑ T E π [ c ~ ( x ^ t , Σ t , u t ) ] .
Recall V t V_t V t from Lemma 13 . Letting V T + 1 = 0 V_{T+1}=0 V T + 1 = 0 , we get a telescoping sum by Lemma 13 :
J T ( π ) = ∑ t = 0 T E π [ V t − V t + 1 ] + ∑ t = 0 T G t + 1 − ∑ t = 0 T E π [ r t ( x ^ t , Σ t , u t ) ] (by tower property) = P 0 μ x 2 + G 0 Σ ˉ + ∑ t = 1 T + 1 G t − ∑ t = 0 T E π [ r t ( x ^ t , Σ t , u t ) ] . (36) \begin{aligned}
J_T(\pi)
&=\sum_{t=0}^{T}\mathbb{E}_{\pi}[V_t-V_{t+1}]+\sum_{t=0}^{T}G_{t+1}-\sum_{t=0}^{T}\mathbb{E}_{\pi}[r_t(\hat{x}_t,\Sigma_t,u_t)] &&\text{(by tower property)}\\
&=P_0\mu_x^2+G_0\bar{\Sigma}+\sum_{t=1}^{T+1}G_t-\sum_{t=0}^{T}\mathbb{E}_{\pi}[r_t(\hat{x}_t,\Sigma_t,u_t)]. \tag*{(36)}
\end{aligned} J T ( π ) = t = 0 ∑ T E π [ V t − V t + 1 ] + t = 0 ∑ T G t + 1 − t = 0 ∑ T E π [ r t ( x ^ t , Σ t , u t )] = P 0 μ x 2 + G 0 Σ ˉ + t = 1 ∑ T + 1 G t − t = 0 ∑ T E π [ r t ( x ^ t , Σ t , u t )] . (by tower property) (36)
This computation was possible because the inequality
c ( x , u ) ≥ R ( u + S R x ) 2 , c(x,u)\ge R\left(u+\frac{S}{R}x\right)^2, c ( x , u ) ≥ R ( u + R S x ) 2 ,
the state recursion, and E [ x 0 2 ] < ∞ \mathbb{E}[x_0^2]<\infty E [ x 0 2 ] < ∞ imply inductively that x t x_t x t , u t u_t u t , and x ^ t \hat{x}_t x ^ t have finite second moments. Therefore, we get
R T ( π , π O L ) = J T ( π O L ) − J T ( π ) = ∑ t = 0 T E π [ r t ( x ^ t , Σ t , u t ) ] − ∑ t = 0 T E π O L [ r t ( x ^ t , Σ t , u t ) ] (by xref‘Eq. (C.2)‘) = ∑ t = 0 T E π [ r t ( x ^ t , Σ t , u t ) ] . \begin{aligned}
R_T(\pi,\pi^{OL})
&=J_T(\pi^{OL})-J_T(\pi)\\
&=\sum_{t=0}^{T}\mathbb{E}_{\pi}[r_t(\hat{x}_t,\Sigma_t,u_t)]-\sum_{t=0}^{T}\mathbb{E}_{\pi^{OL}}[r_t(\hat{x}_t,\Sigma_t,u_t)] &&\text{(by {xref}`Eq. (C.2)`)}\\
&=\sum_{t=0}^{T}\mathbb{E}_{\pi}[r_t(\hat{x}_t,\Sigma_t,u_t)].
\end{aligned} R T ( π , π O L ) = J T ( π O L ) − J T ( π ) = t = 0 ∑ T E π [ r t ( x ^ t , Σ t , u t )] − t = 0 ∑ T E π O L [ r t ( x ^ t , Σ t , u t )] = t = 0 ∑ T E π [ r t ( x ^ t , Σ t , u t )] . (by xref ‘Eq. (C.2)‘)
(Lemma 14)
Next, we consider the case J T ( π ) = ∞ J_T(\pi)=\infty J T ( π ) = ∞ . If E π [ u t 2 ] < ∞ \mathbb{E}_{\pi}[u_t^2]<\infty E π [ u t 2 ] < ∞ for every t = 0 , … , T t=0,\ldots,T t = 0 , … , T , then the state recursion would imply finite second moments for every x t x_t x t , and hence finite expected quadratic stage costs. This would contradict J T ( π ) = ∞ J_T(\pi)=\infty J T ( π ) = ∞ . Thus, let
s : = min { t ∈ { 0 , … , T } : E π [ u t 2 ] = ∞ } . s:=\min\{t\in\{0,\ldots,T\}:\mathbb{E}_{\pi}[u_t^2]=\infty\}. s := min { t ∈ { 0 , … , T } : E π [ u t 2 ] = ∞ } .
By the definition of s s s , every earlier action has a finite second moment. Since
x s = A s x 0 + ∑ j = 0 s − 1 A s − 1 − j ( B u j + w j ) , x_s=A^s x_0+\sum_{j=0}^{s-1}A^{s-1-j}(Bu_j+w_j), x s = A s x 0 + j = 0 ∑ s − 1 A s − 1 − j ( B u j + w j ) ,
and x 0 x_0 x 0 and the process noises have finite second moments, we obtain E π [ x s 2 ] < ∞ \mathbb{E}_{\pi}[x_s^2] < \infty E π [ x s 2 ] < ∞ . In particular, x s x_s x s is integrable, and conditional Jensen’s inequality and the tower property give
E π [ x ^ s 2 ] = E π [ ( E π [ x s ∣ F s ] ) 2 ] ≤ E π [ E π [ x s 2 ∣ F s ] ] = E π [ x s 2 ] < ∞ . \mathbb{E}_{\pi}[\hat{x}_s^2] = \mathbb{E}_{\pi}\left[\left(\mathbb{E}_{\pi}[x_s \mid\mathcal{F}_s]\right)^2\right]
\le\mathbb{E}_{\pi}\left[\mathbb{E}_{\pi}[x_s^2 \mid\mathcal{F}_s]\right]
= \mathbb{E}_{\pi}[x_s^2] < \infty. E π [ x ^ s 2 ] = E π [ ( E π [ x s ∣ F s ] ) 2 ] ≤ E π [ E π [ x s 2 ∣ F s ] ] = E π [ x s 2 ] < ∞.
Next, since u s 2 ≤ 2 ( u s − K s x ^ s ) 2 + 2 K s 2 x ^ s 2 u_s^2 \le2(u_s-K_s\hat{x}_s)^2+2K_s^2\hat{x}_s^2 u s 2 ≤ 2 ( u s − K s x ^ s ) 2 + 2 K s 2 x ^ s 2 , we get
∞ = E π [ u s 2 ] ≤ 2 E π [ ( u s − K s x ^ s ) 2 ] + 2 K s 2 E π [ x ^ s 2 ] . \infty= \mathbb{E}_{\pi}[u_s^2] \le2\mathbb{E}_{\pi}[(u_s-K_s\hat{x}_s)^2] + 2K_s^2\mathbb{E}_{\pi}[\hat{x}_s^2]. ∞ = E π [ u s 2 ] ≤ 2 E π [( u s − K s x ^ s ) 2 ] + 2 K s 2 E π [ x ^ s 2 ] .
The last term is finite, so E π [ ( u s − K s x ^ s ) 2 ] = ∞ \mathbb{E}_{\pi}[(u_s-K_s\hat{x}_s)^2]=\infty E π [( u s − K s x ^ s ) 2 ] = ∞ . Since N s ≥ R > 0 N_s \ge R > 0 N s ≥ R > 0 , it follows that
E π [ p s ( x ^ s , u s ) ] = N s E π [ ( u s − K s x ^ s ) 2 ] = ∞ . \mathbb{E}_{\pi}[p_s(\hat{x}_s,u_s)] = N_s\mathbb{E}_{\pi}[(u_s-K_s\hat{x}_s)^2] = \infty. E π [ p s ( x ^ s , u s )] = N s E π [( u s − K s x ^ s ) 2 ] = ∞.
On the other hand, the covariance bound gives, for every t t t ,
0 ≤ b t ( Σ t , u t ) = Γ t + 1 F ( Σ t , u t ) ≤ Γ t + 1 A ˉ 2 Σ ˉ < ∞ . 0 \le b_t(\Sigma_t,u_t)=\Gamma_{t+1}F(\Sigma_t,u_t)\le\Gamma_{t+1}\bar{A}^2\bar{\Sigma}<\infty. 0 ≤ b t ( Σ t , u t ) = Γ t + 1 F ( Σ t , u t ) ≤ Γ t + 1 A ˉ 2 Σ ˉ < ∞.
Thus, we can see that the expectation of r t ( x ^ t , Σ t , u t ) r_t(\hat{x}_t,\Sigma_t,u_t) r t ( x ^ t , Σ t , u t ) is now well-defined in the extended real system, and cannot equal + ∞ +\infty + ∞ . In particular, at time s s s , we get
E π [ r s ( x ^ s , Σ s , u s ) ] = − ∞ . \mathbb{E}_{\pi}[r_s(\hat{x}_s,\Sigma_s,u_s)]=-\infty. E π [ r s ( x ^ s , Σ s , u s )] = − ∞.
Consequently, using J T ( π O L ) < ∞ J_T(\pi^{OL})<\infty J T ( π O L ) < ∞ ,
∑ t = 0 T E π [ r t ( x ^ t , Σ t , u t ) ] = − ∞ = J T ( π O L ) − J T ( π ) = R T ( π , π O L ) . \sum_{t=0}^{T}\mathbb{E}_{\pi}[r_t(\hat{x}_t,\Sigma_t,u_t)] = -\infty= J_T(\pi^{OL})-J_T(\pi)=R_T(\pi,\pi^{OL}). t = 0 ∑ T E π [ r t ( x ^ t , Σ t , u t )] = − ∞ = J T ( π O L ) − J T ( π ) = R T ( π , π O L ) .
This completes the proof.
Alternate Regret Decomposition
For proving lower bounds, it will be useful to have access to a more flexible regret decomposition. Define
d t ( Σ ) = N t K t 2 ( Σ ˉ − Σ ) . d_t(\Sigma)=N_tK_t^2(\bar{\Sigma}-\Sigma). d t ( Σ ) = N t K t 2 ( Σ ˉ − Σ ) .
The term b t ( Σ t , u t ) b_t(\Sigma_t,u_t) b t ( Σ t , u t ) corresponds to the immediate covariance reduced at t t t , whereas d t ( Σ t ) d_t(\Sigma_t) d t ( Σ t ) corresponds to the information retained up to t t t . In Appendix § 8 , Lemma 12 implies that these terms accumulate identically. This allows us to obtain the following generalization of Proposition 1.
Proposition 7. For λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , define
r t , λ ( x ^ , Σ , u ) = λ b t ( Σ , u ) + ( 1 − λ ) d t ( Σ ) − p t ( x ^ , u ) r_{t,\lambda}(\hat{x},\Sigma,u)=\lambda b_t(\Sigma,u)+(1-\lambda)d_t(\Sigma)-p_t(\hat{x},u) r t , λ ( x ^ , Σ , u ) = λ b t ( Σ , u ) + ( 1 − λ ) d t ( Σ ) − p t ( x ^ , u ) for x ^ , u ∈ R \hat{x},u\in\mathbb{R} x ^ , u ∈ R and Σ > 0 \Sigma>0 Σ > 0 . For any policy π \pi π ,
R T ( π , π O L ) = ∑ t = 0 T E π [ r t , λ ( x ^ t , Σ t , u t ) ] . R_T(\pi,\pi^{OL})=\sum_{t=0}^{T}\mathbb{E}_{\pi}[r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)]. R T ( π , π O L ) = t = 0 ∑ T E π [ r t , λ ( x ^ t , Σ t , u t )] .
Proof. Lemma 12 implies the following equality:
∑ t = 0 T b t ( Σ t , u t ) = ∑ t = 0 T d t ( Σ t ) + λ ( ∑ t = 0 T b t ( Σ t , u t ) − ∑ t = 0 T d t ( Σ t ) ) = λ ∑ t = 0 T b t ( Σ t , u t ) + ( 1 − λ ) ∑ t = 0 T d t ( Σ t ) \sum_{t=0}^{T} b_t(\Sigma_t,u_t) = \sum_{t=0}^{T} d_t(\Sigma_t) + \lambda\left(\sum_{t=0}^{T} b_t(\Sigma_t,u_t) - \sum_{t=0}^{T} d_t(\Sigma_t)\right)
= \lambda\sum_{t=0}^{T} b_t(\Sigma_t,u_t) + (1-\lambda)\sum_{t=0}^{T} d_t(\Sigma_t) t = 0 ∑ T b t ( Σ t , u t ) = t = 0 ∑ T d t ( Σ t ) + λ ( t = 0 ∑ T b t ( Σ t , u t ) − t = 0 ∑ T d t ( Σ t ) ) = λ t = 0 ∑ T b t ( Σ t , u t ) + ( 1 − λ ) t = 0 ∑ T d t ( Σ t )
Plugging this quantity into Proposition 1 completes the proof.
This regret decomposition shows that regret accumulates linearly if either b t b_t b t or d t d_t d t is bounded away from zero, giving us more flexibility in our lower bound derivations.
Lemmas
Lemma 12. For any policy,
∑ t = 0 T b t ( Σ t , u t ) = ∑ t = 0 T d t ( Σ t ) . \sum_{t=0}^{T} b_t(\Sigma_t,u_t) = \sum_{t=0}^{T} d_t(\Sigma_t). t = 0 ∑ T b t ( Σ t , u t ) = t = 0 ∑ T d t ( Σ t ) .
Proof. Using the fact that Σ w = ( 1 − A 2 ) Σ ˉ \Sigma_w = (1-A^2)\bar{\Sigma} Σ w = ( 1 − A 2 ) Σ ˉ , we get
F ( Σ t , u t ) = ( Σ ˉ − Σ t + 1 ) − A 2 ( Σ ˉ − Σ t ) . F(\Sigma_t,u_t) = (\bar{\Sigma}-\Sigma_{t+1}) - A^2(\bar{\Sigma}-\Sigma_t). F ( Σ t , u t ) = ( Σ ˉ − Σ t + 1 ) − A 2 ( Σ ˉ − Σ t ) .
Then, this equality implies that
∑ t = 0 T b t ( Σ t , u t ) = ∑ t = 0 T ( Γ t + 1 ( Σ ˉ − Σ t + 1 ) − Γ t + 1 A 2 ( Σ ˉ − Σ t ) ) = ∑ t = 0 T ( Γ t + 1 ( Σ ˉ − Σ t + 1 ) − Γ t ( Σ ˉ − Σ t ) + N t K t 2 ( Σ ˉ − Σ t ) ) ( ∵ Γ t = N t K t 2 + A 2 Γ t + 1 ) = ∑ t = 0 T d t ( Σ t ) \begin{aligned}
\sum_{t=0}^{T} b_t(\Sigma_t,u_t) = \sum_{t=0}^{T}\left(\Gamma_{t+1}(\bar{\Sigma}-\Sigma_{t+1})-\Gamma_{t+1}A^2(\bar{\Sigma}-\Sigma_t)\right) \\
&= \sum_{t=0}^{T}\left(\Gamma_{t+1}(\bar{\Sigma}-\Sigma_{t+1})-\Gamma_t(\bar{\Sigma}-\Sigma_t)+N_tK_t^2(\bar{\Sigma}-\Sigma_t)\right) \qquad(\because\Gamma_t=N_tK_t^2+A^2\Gamma_{t+1}) \\
&= \sum_{t=0}^{T} d_t(\Sigma_t)
\end{aligned} t = 0 ∑ T b t ( Σ t , u t ) = t = 0 ∑ T ( Γ t + 1 ( Σ ˉ − Σ t + 1 ) − Γ t + 1 A 2 ( Σ ˉ − Σ t ) ) = t = 0 ∑ T ( Γ t + 1 ( Σ ˉ − Σ t + 1 ) − Γ t ( Σ ˉ − Σ t ) + N t K t 2 ( Σ ˉ − Σ t ) ) ( ∵ Γ t = N t K t 2 + A 2 Γ t + 1 ) = t = 0 ∑ T d t ( Σ t )
where the last equality follows since Γ T + 1 = d 0 = 0 \Gamma_{T+1}=d_0=0 Γ T + 1 = d 0 = 0 and Σ 0 = Σ ˉ \Sigma_0=\bar{\Sigma} Σ 0 = Σ ˉ .
Lemma 13. Recall that G t = Q + A 2 G t + 1 G_t=Q+A^2G_{t+1} G t = Q + A 2 G t + 1 with G T + 1 = 0 G_{T+1}=0 G T + 1 = 0 . Define
V t = P t x ^ t 2 + G t Σ t . V_t=P_t\hat{x}_t^2+G_t\Sigma_t. V t = P t x ^ t 2 + G t Σ t . For any t = 0 , 1 , … , T t=0,1,\ldots,T t = 0 , 1 , … , T , we have
c ˉ ( x ^ t , Σ t , u t ) = V t − E [ V t + 1 ∣ x ^ t , Σ t , u t ] + G t + 1 Σ w − r t ( x ^ t , Σ t , u t ) . \bar{c}(\hat{x}_t,\Sigma_t,u_t)=V_t-\mathbb{E}[V_{t+1}\mid\hat{x}_t,\Sigma_t,u_t]+G_{t+1}\Sigma_w-r_t(\hat{x}_t,\Sigma_t,u_t). c ˉ ( x ^ t , Σ t , u t ) = V t − E [ V t + 1 ∣ x ^ t , Σ t , u t ] + G t + 1 Σ w − r t ( x ^ t , Σ t , u t ) .
Proof. Recall that c ˉ ( x ^ t , Σ t , u t ) = Q x ^ t 2 + Q Σ t + R u t 2 + 2 u t S x ^ t \bar{c}(\hat{x}_t,\Sigma_t,u_t)=Q\hat{x}_t^2+Q\Sigma_t+Ru_t^2+2u_tS\hat{x}_t c ˉ ( x ^ t , Σ t , u t ) = Q x ^ t 2 + Q Σ t + R u t 2 + 2 u t S x ^ t . For the terms related to x ^ t \hat{x}_t x ^ t in c ˉ \bar{c} c ˉ , using Lemma 1, observe the following equality:
Q x ^ t 2 + R u t 2 + 2 u t S x ^ t + P t + 1 E [ x ^ t + 1 2 ∣ x ^ t , Σ t , u t ] − P t x ^ t 2 = ( R + B 2 P t + 1 ) u t 2 + 2 ( S + A B P t + 1 ) u t x ^ t + ( Q + A 2 P t + 1 − P t ) x ^ t 2 + P t + 1 F ( Σ t , u t ) = N t ( u t − K t x ^ t ) 2 + P t + 1 F ( Σ t , u t ) . \begin{aligned}
Q\hat{x}_t^2+Ru_t^2+2u_tS\hat{x}_t+P_{t+1}\mathbb{E}[\hat{x}_{t+1}^2\mid\hat{x}_t,\Sigma_t,u_t]-P_t\hat{x}_t^2 \\
&=(R+B^2P_{t+1})u_t^2+2(S+ABP_{t+1})u_t\hat{x}_t+(Q+A^2P_{t+1}-P_t)\hat{x}_t^2+P_{t+1}F(\Sigma_t,u_t) \\
&= N_t(u_t-K_t\hat{x}_t)^2+P_{t+1}F(\Sigma_t,u_t).
\end{aligned} Q x ^ t 2 + R u t 2 + 2 u t S x ^ t + P t + 1 E [ x ^ t + 1 2 ∣ x ^ t , Σ t , u t ] − P t x ^ t 2 = ( R + B 2 P t + 1 ) u t 2 + 2 ( S + A B P t + 1 ) u t x ^ t + ( Q + A 2 P t + 1 − P t ) x ^ t 2 + P t + 1 F ( Σ t , u t ) = N t ( u t − K t x ^ t ) 2 + P t + 1 F ( Σ t , u t ) .
For the term related to Σ t \Sigma_t Σ t , we also observe the following equality:
Q Σ t + G t + 1 Σ t + 1 − G t Σ t = Q Σ t + G t + 1 ( A 2 Σ t + Σ w − F ( Σ t , u t ) ) − ( Q + A 2 G t + 1 ) Σ t = G t + 1 Σ w − G t + 1 F ( Σ t , u t ) . \begin{aligned}
Q\Sigma_t+G_{t+1}\Sigma_{t+1}-G_t\Sigma_t
&=Q\Sigma_t+G_{t+1}\left(A^2\Sigma_t+\Sigma_w-F(\Sigma_t,u_t)\right)-(Q+A^2G_{t+1})\Sigma_t\\
&=G_{t+1}\Sigma_w-G_{t+1}F(\Sigma_t,u_t).
\end{aligned} Q Σ t + G t + 1 Σ t + 1 − G t Σ t = Q Σ t + G t + 1 ( A 2 Σ t + Σ w − F ( Σ t , u t ) ) − ( Q + A 2 G t + 1 ) Σ t = G t + 1 Σ w − G t + 1 F ( Σ t , u t ) .
Hence, combining two equalities gives
c ˉ ( x ^ t , Σ t , u t ) + E [ V t + 1 ∣ x ^ t , Σ t , u t ] − V t = N t ( u t − K t x ^ t ) 2 + G t + 1 Σ w + ( P t + 1 − G t + 1 ) F ( Σ t , u t ) = N t ( u t − K t x ^ t ) 2 + G t + 1 Σ w − Γ t + 1 F ( Σ t , u t ) (Lemma 3) = G t + 1 Σ w − r t ( x ^ t , Σ t , u t ) \begin{aligned}
\bar{c}(\hat{x}_t,\Sigma_t,u_t)+\mathbb{E}[V_{t+1}\mid\hat{x}_t,\Sigma_t,u_t]-V_t
&=N_t(u_t-K_t\hat{x}_t)^2+G_{t+1}\Sigma_w+(P_{t+1}-G_{t+1})F(\Sigma_t,u_t)\\
&=N_t(u_t-K_t\hat{x}_t)^2+G_{t+1}\Sigma_w-\Gamma_{t+1}F(\Sigma_t,u_t)\quad\text{(Lemma 3)}\\
&=G_{t+1}\Sigma_w-r_t(\hat{x}_t,\Sigma_t,u_t)
\end{aligned} c ˉ ( x ^ t , Σ t , u t ) + E [ V t + 1 ∣ x ^ t , Σ t , u t ] − V t = N t ( u t − K t x ^ t ) 2 + G t + 1 Σ w + ( P t + 1 − G t + 1 ) F ( Σ t , u t ) = N t ( u t − K t x ^ t ) 2 + G t + 1 Σ w − Γ t + 1 F ( Σ t , u t ) (Lemma 3) = G t + 1 Σ w − r t ( x ^ t , Σ t , u t )
and rearranging completes the proof.
Lemma 14. We have
∑ t = 0 T E π O L [ r t ( x ^ t , Σ t , u t ) ] = 0 (37) \sum_{t=0}^{T}\mathbb{E}_{\pi^{OL}}[r_t(\hat{x}_t,\Sigma_t,u_t)]=0 \tag*{(37)} t = 0 ∑ T E π O L [ r t ( x ^ t , Σ t , u t )] = 0 (37) Proof. From the law of total variance, we get Var ( x ^ t ) = Var ( x t ) − Σ t \operatorname{Var}(\hat{x}_t)=\operatorname{Var}(x_t)-\Sigma_t Var ( x ^ t ) = Var ( x t ) − Σ t under π O L \pi^{OL} π O L . Also, each recursion of x t x_t x t and Σ t \Sigma_t Σ t gives the equalities:
Var ( x t + 1 ) = A 2 Var ( x t ) + Σ w and Σ t + 1 = A 2 Σ t + Σ w − F ( Σ t , u t ) . \operatorname{Var}(x_{t+1})=A^2\operatorname{Var}(x_t)+\Sigma_w
\quad\text{and}\quad
\Sigma_{t+1}=A^2\Sigma_t+\Sigma_w-F(\Sigma_t,u_t). Var ( x t + 1 ) = A 2 Var ( x t ) + Σ w and Σ t + 1 = A 2 Σ t + Σ w − F ( Σ t , u t ) . It follows that
Var ( x ^ t + 1 ) = A 2 Var ( x ^ t ) + F ( Σ t , u t ) , i.e., F ( Σ t , u t ) = Var ( x ^ t + 1 ) − A 2 Var ( x ^ t ) . (38) \operatorname{Var}(\hat{x}_{t+1})=A^2\operatorname{Var}(\hat{x}_t)+F(\Sigma_t,u_t),
\quad\text{i.e.,}\quad
F(\Sigma_t,u_t)=\operatorname{Var}(\hat{x}_{t+1})-A^2\operatorname{Var}(\hat{x}_t). \tag*{(38)} Var ( x ^ t + 1 ) = A 2 Var ( x ^ t ) + F ( Σ t , u t ) , i.e., F ( Σ t , u t ) = Var ( x ^ t + 1 ) − A 2 Var ( x ^ t ) . (38) For the OL policy u t = K t m t = K t E [ x t ] u_t=K_tm_t=K_t\mathbb{E}[x_t] u t = K t m t = K t E [ x t ] , observe that
E [ ( u t − K t x ^ t ) 2 ] = K t 2 E [ ( x ^ t − E [ x ^ t ] ) 2 ] = K t 2 Var ( x ^ t ) . (39) \mathbb{E}\left[(u_t-K_t\hat{x}_t)^2\right]
=K_t^2\mathbb{E}\left[(\hat{x}_t-\mathbb{E}[\hat{x}_t])^2\right]
=K_t^2\operatorname{Var}(\hat{x}_t). \tag*{(39)} E [ ( u t − K t x ^ t ) 2 ] = K t 2 E [ ( x ^ t − E [ x ^ t ] ) 2 ] = K t 2 Var ( x ^ t ) . (39) Therefore, the quantity can be reduced as follows:
∑ t = 0 T E π O L [ r t ( x ^ t , Σ t , u t ) ] = ∑ t = 0 T E [ Γ t + 1 F ( Σ t , u t ) − N t ( u t − K t x ^ t ) 2 ] = ∑ t = 0 T E [ Γ t + 1 Var ( x ^ t + 1 ) − ( N t K t 2 + A 2 Γ t + 1 ) Var ( x ^ t ) ] (by xref‘Eq. (C.4)‘ and xref‘(C.5)‘) = ∑ t = 0 T E [ Γ t + 1 Var ( x ^ t + 1 ) − Γ t Var ( x ^ t ) ] = Γ T + 1 Var ( x ^ T + 1 ) − Γ 0 Var ( x ^ 0 ) . \begin{aligned}
\sum_{t=0}^{T}\mathbb{E}_{\pi^{OL}}[r_t(\hat{x}_t,\Sigma_t,u_t)]
&=\sum_{t=0}^{T}\mathbb{E}\left[\Gamma_{t+1}F(\Sigma_t,u_t)-N_t(u_t-K_t\hat{x}_t)^2\right]\\
&=\sum_{t=0}^{T}\mathbb{E}\left[\Gamma_{t+1}\operatorname{Var}(\hat{x}_{t+1})-(N_tK_t^2+A^2\Gamma_{t+1})\operatorname{Var}(\hat{x}_t)\right]
\quad\text{(by {xref}`Eq. (C.4)` and {xref}`(C.5)`)}\\
&=\sum_{t=0}^{T}\mathbb{E}\left[\Gamma_{t+1}\operatorname{Var}(\hat{x}_{t+1})-\Gamma_t\operatorname{Var}(\hat{x}_t)\right]\\
&=\Gamma_{T+1}\operatorname{Var}(\hat{x}_{T+1})-\Gamma_0\operatorname{Var}(\hat{x}_0).
\end{aligned} t = 0 ∑ T E π O L [ r t ( x ^ t , Σ t , u t )] = t = 0 ∑ T E [ Γ t + 1 F ( Σ t , u t ) − N t ( u t − K t x ^ t ) 2 ] = t = 0 ∑ T E [ Γ t + 1 Var ( x ^ t + 1 ) − ( N t K t 2 + A 2 Γ t + 1 ) Var ( x ^ t ) ] (by xref ‘Eq. (C.4)‘ and xref ‘(C.5)‘) = t = 0 ∑ T E [ Γ t + 1 Var ( x ^ t + 1 ) − Γ t Var ( x ^ t ) ] = Γ T + 1 Var ( x ^ T + 1 ) − Γ 0 Var ( x ^ 0 ) . Since Γ T + 1 = 0 \Gamma_{T+1}=0 Γ T + 1 = 0 and Var ( x ^ 0 ) = 0 \operatorname{Var}(\hat{x}_0)=0 Var ( x ^ 0 ) = 0 , we complete the proof.
Bounded Optimal-Control Advantage
This section establishes a uniform upper bound on the cost advantage of any policy over open-loop control when q < q 1 q < q_1 q < q 1 . The proof is largely algebraic and relies on the Riccati equations to show that q < q 1 q < q_1 q < q 1 implies
r t ( x ^ t , Σ t , u t ) ≤ P t x ^ t 2 − P t + 1 E π [ x ^ t + 1 2 ∣ F t ] . (40) r_t(\hat{x}_t,\Sigma_t,u_t) \le P_t\hat{x}_t^2-P_{t+1}\mathbb{E}_{\pi}[\hat{x}_{t+1}^2\mid\mathcal{F}_t]. \tag*{(40)} r t ( x ^ t , Σ t , u t ) ≤ P t x ^ t 2 − P t + 1 E π [ x ^ t + 1 2 ∣ F t ] . (40)
which is shown in Lemma 15. Under Eq. (D.1), the regret upper bound telescopes to P 0 μ x 2 P_0\mu_x^2 P 0 μ x 2 , which is constant due to Lemma 5. Consequently, as the policy is arbitrary, even the optimal policy can improve on open-loop policy by at most a constant independent of the horizon. Figure 3 describes the proof dependencies for Section D.
Figure 3. Main proof dependencies for Section D. An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Proof of Theorem 3
Proof. We have
R T ( π , π O L ) = ∑ t = 0 T E π [ r t ( x ^ t , Σ t , u t ) ] (Proposition 1) ≤ ∑ t = 0 T E π [ P t x ^ t 2 − E [ P t + 1 x ^ t + 1 2 ∣ x ^ t , Σ t , u t ] ] (Lemma 15) = P 0 μ x 2 \begin{aligned}
R_T(\pi,\pi^{OL}) &= \sum_{t=0}^{T}\mathbb{E}_{\pi}[r_t(\hat{x}_t,\Sigma_t,u_t)] && \text{(Proposition 1)}\\
&\le\sum_{t=0}^{T}\mathbb{E}_{\pi}[P_t\hat{x}_t^2-\mathbb{E}[P_{t+1}\hat{x}_{t+1}^2\mid\hat{x}_t,\Sigma_t,u_t]] && \text{(Lemma 15)}\\
&=P_0\mu_x^2
\end{aligned} R T ( π , π O L ) = t = 0 ∑ T E π [ r t ( x ^ t , Σ t , u t )] ≤ t = 0 ∑ T E π [ P t x ^ t 2 − E [ P t + 1 x ^ t + 1 2 ∣ x ^ t , Σ t , u t ]] = P 0 μ x 2 (Proposition 1) (Lemma 15)
where the last equality is obtained by telescoping and that P T + 1 = 0 P_{T+1}=0 P T + 1 = 0 . By Lemma 5, for
P 0 μ x 2 ≤ ( P + C P ) μ x 2 P_0\mu_x^2 \le(P+C_P)\mu_x^2 P 0 μ x 2 ≤ ( P + C P ) μ x 2
for some C P > 0 C_P>0 C P > 0 , which is independent of T T T and π \pi π .
Lemmas
Lemma 15. If q < q 1 q<q_1 q < q 1 , then
r t ( x ^ t , Σ t , u t ) ≤ P t x ^ t 2 − E [ P t + 1 x ^ t + 1 2 ∣ x ^ t , Σ t , u t ] . (41) r_t(\hat{x}_t,\Sigma_t,u_t)\le P_t\hat{x}_t^2-\mathbb{E}[P_{t+1}\hat{x}_{t+1}^2\mid\hat{x}_t,\Sigma_t,u_t]. \tag*{(41)} r t ( x ^ t , Σ t , u t ) ≤ P t x ^ t 2 − E [ P t + 1 x ^ t + 1 2 ∣ x ^ t , Σ t , u t ] . (41)
Proof. Denote
ψ ( x ^ t , Σ t , u t ) = P t x ^ t 2 − E [ P t + 1 x ^ t + 1 2 ∣ x ^ t , Σ t , u t ] − r t ( x ^ t , Σ t , u t ) (42) \psi(\hat{x}_t,\Sigma_t,u_t)=P_t\hat{x}_t^2-\mathbb{E}[P_{t+1}\hat{x}_{t+1}^2\mid\hat{x}_t,\Sigma_t,u_t]-r_t(\hat{x}_t,\Sigma_t,u_t) \tag*{(42)} ψ ( x ^ t , Σ t , u t ) = P t x ^ t 2 − E [ P t + 1 x ^ t + 1 2 ∣ x ^ t , Σ t , u t ] − r t ( x ^ t , Σ t , u t ) (42)
We aim to show ψ ( x ^ , Σ , u ) ≥ 0 \psi(\hat{x},\Sigma,u)\ge0 ψ ( x ^ , Σ , u ) ≥ 0 . We have
ψ ( x ^ , Σ , u ) \psi(\hat{x},\Sigma,u) ψ ( x ^ , Σ , u )
= P t x ^ t 2 − E [ P t + 1 x ^ t + 1 2 ∣ x ^ t , Σ t , u t ] − r t ( x ^ t , Σ t , u t ) = P t x ^ t 2 − P t + 1 ( ( A x ^ t + B u t ) 2 + F ( Σ t , u t ) ) + N t ( u t − K t x ^ t ) 2 − Γ t + 1 F ( Σ t , u t ) (Lemma 1) = P t x ^ t 2 − P t + 1 ( A x ^ t + B u t ) 2 + N t ( u t − K t x ^ t ) 2 − G t + 1 F ( Σ t , u t ) ( ∵ Γ t + 1 = G t + 1 − P t + 1 ) = ( P t − A 2 P t + 1 + N t K t 2 ) x ^ t 2 − 2 ( P t + 1 A B + N t K t ) x ^ t u t + ( N t − P t + 1 B 2 ) u t 2 − G t + 1 F ( Σ t , u t ) \begin{aligned}
&= P_t\hat{x}_t^2-\mathbb{E}[P_{t+1}\hat{x}_{t+1}^2\mid\hat{x}_t,\Sigma_t,u_t]-r_t(\hat{x}_t,\Sigma_t,u_t)\\
&=P_t\hat{x}_t^2-P_{t+1}\left((A\hat{x}_t+Bu_t)^2+F(\Sigma_t,u_t)\right)+N_t(u_t-K_t\hat{x}_t)^2-\Gamma_{t+1}F(\Sigma_t,u_t) \qquad\text{(Lemma 1)}\\
&=P_t\hat{x}_t^2-P_{t+1}(A\hat{x}_t+Bu_t)^2+N_t(u_t-K_t\hat{x}_t)^2-G_{t+1}F(\Sigma_t,u_t) \qquad(\because\ \Gamma_{t+1}=G_{t+1}-P_{t+1})\\
&=(P_t-A^2P_{t+1}+N_tK_t^2)\hat{x}_t^2-2(P_{t+1}AB+N_tK_t)\hat{x}_tu_t+(N_t-P_{t+1}B^2)u_t^2-G_{t+1}F(\Sigma_t,u_t)
\end{aligned} = P t x ^ t 2 − E [ P t + 1 x ^ t + 1 2 ∣ x ^ t , Σ t , u t ] − r t ( x ^ t , Σ t , u t ) = P t x ^ t 2 − P t + 1 ( ( A x ^ t + B u t ) 2 + F ( Σ t , u t ) ) + N t ( u t − K t x ^ t ) 2 − Γ t + 1 F ( Σ t , u t ) (Lemma 1) = P t x ^ t 2 − P t + 1 ( A x ^ t + B u t ) 2 + N t ( u t − K t x ^ t ) 2 − G t + 1 F ( Σ t , u t ) ( ∵ Γ t + 1 = G t + 1 − P t + 1 ) = ( P t − A 2 P t + 1 + N t K t 2 ) x ^ t 2 − 2 ( P t + 1 A B + N t K t ) x ^ t u t + ( N t − P t + 1 B 2 ) u t 2 − G t + 1 F ( Σ t , u t )
Recall the following equalities:
P t = A 2 P t + 1 + Q − K t 2 N t , N t = R + B 2 P t + 1 , − K t N t = S + B P t + 1 A . P_t=A^2P_{t+1}+Q-K_t^2N_t,\qquad N_t=R+B^2P_{t+1},\qquad-K_tN_t=S+BP_{t+1}A. P t = A 2 P t + 1 + Q − K t 2 N t , N t = R + B 2 P t + 1 , − K t N t = S + B P t + 1 A .
It follows that
ψ ( x ^ t , Σ t , u t ) = Q x ^ t 2 + 2 S x ^ t u t + R u t 2 − G t + 1 F ( Σ t , u t ) . \psi(\hat{x}_t,\Sigma_t,u_t)=Q\hat{x}_t^2+2S\hat{x}_tu_t+Ru_t^2-G_{t+1}F(\Sigma_t,u_t). ψ ( x ^ t , Σ t , u t ) = Q x ^ t 2 + 2 S x ^ t u t + R u t 2 − G t + 1 F ( Σ t , u t ) .
From the two inequalities
F ( Σ t , u t ) ≤ A 2 Σ ˉ 2 C 2 u t 2 σ z 2 and G t + 1 = Q ∑ j = 0 T − t − 1 A 2 j ≤ Q ∑ j = 0 ∞ A 2 j = Q 1 − A 2 , F(\Sigma_t,u_t)\leq\frac{A^2\bar{\Sigma}^2C^2u_t^2}{\sigma_z^2}
\qquad\text{and}\qquad
G_{t+1}=Q\sum_{j=0}^{T-t-1}A^{2j}\leq Q\sum_{j=0}^{\infty}A^{2j}=\frac{Q}{1-A^2}, F ( Σ t , u t ) ≤ σ z 2 A 2 Σ ˉ 2 C 2 u t 2 and G t + 1 = Q j = 0 ∑ T − t − 1 A 2 j ≤ Q j = 0 ∑ ∞ A 2 j = 1 − A 2 Q ,
we get
ψ ( x ^ t , Σ t , u t ) ≥ ( R − Q 1 − A 2 ⋅ A 2 C 2 Σ ˉ 2 σ z 2 ) u t 2 + 2 S u t x ^ t + Q x ^ t 2 . \psi(\hat{x}_t,\Sigma_t,u_t)\geq\left(R-\frac{Q}{1-A^2}\cdot\frac{A^2C^2\bar{\Sigma}^2}{\sigma_z^2}\right)u_t^2+2Su_t\hat{x}_t+Q\hat{x}_t^2. ψ ( x ^ t , Σ t , u t ) ≥ ( R − 1 − A 2 Q ⋅ σ z 2 A 2 C 2 Σ ˉ 2 ) u t 2 + 2 S u t x ^ t + Q x ^ t 2 .
Call α : = R − Q A 2 C 2 Σ ˉ 2 σ z 2 ( 1 − A 2 ) \alpha:=R-\frac{QA^2C^2\bar{\Sigma}^2}{\sigma_z^2(1-A^2)} α := R − σ z 2 ( 1 − A 2 ) Q A 2 C 2 Σ ˉ 2 , then we have
ψ ( x ^ , Σ , u ) ≥ 2 S u t x ^ t + Q x ^ t 2 + α u t 2 = 1 Q u t 2 S 2 + 2 u t S x ^ t + Q x ^ t 2 + ( α − 1 Q S 2 ) u t 2 = ( 1 Q u t S + Q x ^ t ) 2 + ( α − 1 Q S 2 ) u t 2 \begin{aligned}
\psi(\hat{x},\Sigma,u)&\geq2Su_t\hat{x}_t+Q\hat{x}_t^2+\alpha u_t^2\\
&=\frac{1}{Q}u_t^2S^2+2u_tS\hat{x}_t+Q\hat{x}_t^2+\left(\alpha-\frac{1}{Q}S^2\right)u_t^2\\
&=\left(\frac{1}{\sqrt{Q}}u_tS+\sqrt{Q}\hat{x}_t\right)^2+\left(\alpha-\frac{1}{Q}S^2\right)u_t^2
\end{aligned} ψ ( x ^ , Σ , u ) ≥ 2 S u t x ^ t + Q x ^ t 2 + α u t 2 = Q 1 u t 2 S 2 + 2 u t S x ^ t + Q x ^ t 2 + ( α − Q 1 S 2 ) u t 2 = ( Q 1 u t S + Q x ^ t ) 2 + ( α − Q 1 S 2 ) u t 2
Hence if α − 1 Q S 2 ≥ 0 \alpha-\frac{1}{Q}S^2\geq0 α − Q 1 S 2 ≥ 0 , then ψ ( x ^ t , Σ t , u t ) ≥ 0 \psi(\hat{x}_t,\Sigma_t,u_t)\geq0 ψ ( x ^ t , Σ t , u t ) ≥ 0 . That is, we require
R − Q A 2 C 2 Σ ˉ 2 σ z 2 ( 1 − A 2 ) − 1 Q S 2 ≥ 0. (43) R-\frac{QA^2C^2\bar{\Sigma}^2}{\sigma_z^2(1-A^2)}-\frac{1}{Q}S^2\geq0. \tag*{(43)} R − σ z 2 ( 1 − A 2 ) Q A 2 C 2 Σ ˉ 2 − Q 1 S 2 ≥ 0. (43)
Rearranging yields the result. □
Bounded CE Advantage
This section proves Proposition 2, which shows that π C E \pi^{CE} π C E achieves only a bounded advantage over π O L \pi^{OL} π O L when q < q 3 q<q_3 q < q 3 . Under π C E \pi^{CE} π C E , the term p t p_t p t vanishes, so by Proposition 1, the regret is equal to
∑ t = 0 T Γ t + 1 E [ F ( Σ t , K t x ^ t ) ] \sum_{t=0}^{T}\Gamma_{t+1}\mathbb{E}[F(\Sigma_t,K_t\hat{x}_t)] t = 0 ∑ T Γ t + 1 E [ F ( Σ t , K t x ^ t )]
and it suffices to show that how this sum is bounded. The key is the multiplicative recursion x ^ t + 1 = M t x ^ t \hat{x}_{t+1}=M_t\hat{x}_t x ^ t + 1 = M t x ^ t of Lemma 10 . When q < q 3 q<q_3 q < q 3 , Lemma 7 shows that E [ M t 2 ∣ H t ] ≤ γ < 1 \mathbb{E}[M_t^2\mid H_t]\leq\gamma<1 E [ M t 2 ∣ H t ] ≤ γ < 1 for all t t t away from the horizon, so E [ x ^ t 2 ] \mathbb{E}[\hat{x}_t^2] E [ x ^ t 2 ] decays geometrically. From the observation that F ( Σ t , K t x ^ t ) F(\Sigma_t,K_t\hat{x}_t) F ( Σ t , K t x ^ t ) is at most proportional to x ^ t 2 \hat{x}_t^2 x ^ t 2 , it follows that the regret is dominated by a convergent geometric series. Figure 4 describes the proof dependencies for Section E.
Figure 4. Main proof dependencies for Section E. An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Proof of Proposition 2
Proof. Assume q < q 3 q < q_3 q < q 3 . Let us analyze F ( Σ t , K t x ^ t ) F(\Sigma_t, K_t\hat{x}_t) F ( Σ t , K t x ^ t ) . Since Σ t ≤ Σ ˉ \Sigma_t \le\bar{\Sigma} Σ t ≤ Σ ˉ , we have
F ( Σ t , K t x ^ t ) = A 2 K t 2 x ^ t 2 C 2 Σ t 2 σ z 2 + K t 2 x ^ t 2 C 2 Σ t ≤ A 2 K t 2 x ^ t 2 C 2 Σ ˉ 2 σ z 2 ≤ A 2 K t 2 x ^ t 2 C 2 Σ ˉ 2 σ z 2 = A 2 K t 2 C 2 Σ ˉ 2 σ z 2 x ^ t 2 . F(\Sigma_t, K_t\hat{x}_t) = \frac{A^2K_t^2\hat{x}_t^2C^2\Sigma_t^2}{\sigma_z^2 + K_t^2\hat{x}_t^2C^2\Sigma_t} \le\frac{A^2K_t^2\hat{x}_t^2C^2\bar{\Sigma}^2}{\sigma_z^2} \le\frac{A^2K_t^2\hat{x}_t^2C^2\bar{\Sigma}^2}{\sigma_z^2} = \frac{A^2K_t^2C^2\bar{\Sigma}^2}{\sigma_z^2}\hat{x}_t^2. F ( Σ t , K t x ^ t ) = σ z 2 + K t 2 x ^ t 2 C 2 Σ t A 2 K t 2 x ^ t 2 C 2 Σ t 2 ≤ σ z 2 A 2 K t 2 x ^ t 2 C 2 Σ ˉ 2 ≤ σ z 2 A 2 K t 2 x ^ t 2 C 2 Σ ˉ 2 = σ z 2 A 2 K t 2 C 2 Σ ˉ 2 x ^ t 2 .
Define the function h h h by
h ( k ) : = ( A + B k ) 2 + A 2 k 2 C 2 Σ ˉ 2 σ z 2 . h(k) := (A + Bk)^2 + \frac{A^2k^2C^2\bar{\Sigma}^2}{\sigma_z^2}. h ( k ) := ( A + B k ) 2 + σ z 2 A 2 k 2 C 2 Σ ˉ 2 .
Observe that by definition
q < q 3 ⟺ h ( K ) = D 2 + q < 1. q < q_3 \Longleftrightarrow h(K) = D^2 + q < 1. q < q 3 ⟺ h ( K ) = D 2 + q < 1.
The function h h h is clearly continuous in k k k , thus by Lemma 7 there exists an integer t 0 t_0 t 0 , depending only on the problem parameters and not on T T T , such that h ( K t ) < D 2 + q + 1 2 h(K_t) < \frac{D^2+q+1}{2} h ( K t ) < 2 D 2 + q + 1 for all t ≤ T − t 0 t \le T-t_0 t ≤ T − t 0 . Denote γ : = ( D 2 + q + 1 ) / 2 \gamma:= (D^2+q+1)/2 γ := ( D 2 + q + 1 ) /2 so that h ( K t ) ≤ γ h(K_t) \le\gamma h ( K t ) ≤ γ for all t ≤ T − t 0 t \le T-t_0 t ≤ T − t 0 . By Lemma 10(ii), we have
E [ M t 2 ∣ F t ] = D t 2 + q t 2 = ( A + B K t ) 2 + A 2 K t 2 C 2 Σ t 2 σ z 2 + K t 2 x ^ t 2 C 2 Σ t ≤ ( A + B K t ) 2 + A 2 K t 2 C 2 Σ ˉ 2 σ z 2 = h ( K t ) . \mathbb{E}[M_t^2 \mid\mathcal{F}_t] = D_t^2 + q_t^2 = (A + BK_t)^2 + \frac{A^2K_t^2C^2\Sigma_t^2}{\sigma_z^2 + K_t^2\hat{x}_t^2C^2\Sigma_t} \le(A + BK_t)^2 + \frac{A^2K_t^2C^2\bar{\Sigma}^2}{\sigma_z^2} = h(K_t). E [ M t 2 ∣ F t ] = D t 2 + q t 2 = ( A + B K t ) 2 + σ z 2 + K t 2 x ^ t 2 C 2 Σ t A 2 K t 2 C 2 Σ t 2 ≤ ( A + B K t ) 2 + σ z 2 A 2 K t 2 C 2 Σ ˉ 2 = h ( K t ) .
Then, for t ≤ T − t 0 t \le T-t_0 t ≤ T − t 0 , we have
E [ M t 2 ∣ H t ] ≤ h ( K t ) ≤ γ . \mathbb{E}[M_t^2 \mid\mathcal{H}_t] \le h(K_t) \le\gamma. E [ M t 2 ∣ H t ] ≤ h ( K t ) ≤ γ .
Now, Lemma 10(i) and tower property give
E [ x ^ t + 1 2 ] = E [ x ^ t 2 E [ M t 2 ∣ H t ] ] ≤ γ E [ x ^ t 2 ] . \mathbb{E}[\hat{x}_{t+1}^2] = \mathbb{E}[\hat{x}_t^2\mathbb{E}[M_t^2 \mid\mathcal{H}_t]] \le\gamma\mathbb{E}[\hat{x}_t^2]. E [ x ^ t + 1 2 ] = E [ x ^ t 2 E [ M t 2 ∣ H t ]] ≤ γ E [ x ^ t 2 ] .
Thus, for any t ≤ T − t 0 + 1 t \le T-t_0+1 t ≤ T − t 0 + 1 we have
E [ x ^ t 2 ] ≤ μ x 2 γ t . \mathbb{E}[\hat{x}_t^2] \le\mu_x^2\gamma^t. E [ x ^ t 2 ] ≤ μ x 2 γ t .
To connect this to regret, observe that
E [ F ( Σ t , K t x ^ t ) ] ≤ A 2 K t 2 C 2 Σ ˉ 2 σ z 2 E [ x ^ t 2 ] ≤ h ( K t ) E [ x ^ t 2 ] ≤ μ x 2 γ t + 1 , \mathbb{E}[F(\Sigma_t, K_t\hat{x}_t)] \le\frac{A^2K_t^2C^2\bar{\Sigma}^2}{\sigma_z^2}\mathbb{E}[\hat{x}_t^2] \le h(K_t)\mathbb{E}[\hat{x}_t^2] \le\mu_x^2\gamma^{t+1}, E [ F ( Σ t , K t x ^ t )] ≤ σ z 2 A 2 K t 2 C 2 Σ ˉ 2 E [ x ^ t 2 ] ≤ h ( K t ) E [ x ^ t 2 ] ≤ μ x 2 γ t + 1 ,
so that
∑ t = 0 T − t 0 E [ F ( Σ t , K t x ^ t ) ] ≤ μ x 2 ∑ t = 0 T − t 0 γ t + 1 ≤ μ x 2 ∑ t = 0 ∞ γ t + 1 = μ x 2 γ 1 − γ . \sum_{t=0}^{T-t_0}\mathbb{E}[F(\Sigma_t, K_t\hat{x}_t)] \le\mu_x^2\sum_{t=0}^{T-t_0}\gamma^{t+1} \le\mu_x^2\sum_{t=0}^{\infty}\gamma^{t+1} = \frac{\mu_x^2\gamma}{1-\gamma}. t = 0 ∑ T − t 0 E [ F ( Σ t , K t x ^ t )] ≤ μ x 2 t = 0 ∑ T − t 0 γ t + 1 ≤ μ x 2 t = 0 ∑ ∞ γ t + 1 = 1 − γ μ x 2 γ .
For t > T − t 0 + 1 t>T-t_0+1 t > T − t 0 + 1 , we consider the natural crude bound:
E [ F ( Σ t , K t x ^ t ) ] ≤ A 2 Σ ˉ . \mathbb{E}[F(\Sigma_t,K_t\widehat{x}_t)]\le A^2\bar{\Sigma}. E [ F ( Σ t , K t x t )] ≤ A 2 Σ ˉ .
For the coefficient Γ t \Gamma_t Γ t , we consider its uniform bound as follows: Lemma 5 implies there exists C 1 < ∞ C_1<\infty C 1 < ∞ such that max t { N t , K t } ≤ C 1 \max_t\{N_t,K_t\}\le C_1 max t { N t , K t } ≤ C 1 , so that
Γ t = ∑ s = t T A 2 ( s − t ) N s K s 2 ≤ C 1 3 ∑ s = t T A 2 ( s − t ) ≤ C 1 3 1 − A 2 . \Gamma_t=\sum_{s=t}^{T}A^{2(s-t)}N_sK_s^2\le C_1^3\sum_{s=t}^{T}A^{2(s-t)}\le\frac{C_1^3}{1-A^2}. Γ t = s = t ∑ T A 2 ( s − t ) N s K s 2 ≤ C 1 3 s = t ∑ T A 2 ( s − t ) ≤ 1 − A 2 C 1 3 .
Therefore, the regret now can be bounded as follows:
∑ t = 0 T E π [ r t ( x ^ t , Σ t , u t ) ] = ∑ t = 0 T E [ Γ t + 1 F ( Σ t , K t x ^ t ) ] ≤ C 1 3 1 − A 2 ⋅ ∑ t = 0 T E [ F ( Σ t , K t x ^ t ) ] = C 1 3 1 − A 2 ( μ γ 2 γ 1 − γ + t 0 A 2 Σ ˉ ) . \begin{aligned}
\sum_{t=0}^{T}\mathbb{E}_{\pi}[r_t(\widehat{x}_t,\Sigma_t,u_t)]=\sum_{t=0}^{T}\mathbb{E}\left[\Gamma_{t+1}F(\Sigma_t,K_t\widehat{x}_t)\right] \\
&\le\frac{C_1^3}{1-A^2}\cdot\sum_{t=0}^{T}\mathbb{E}[F(\Sigma_t,K_t\widehat{x}_t)] \\
&=\frac{C_1^3}{1-A^2}\left(\frac{\mu_\gamma^2\gamma}{1-\gamma}+t_0A^2\bar{\Sigma}\right).
\end{aligned} t = 0 ∑ T E π [ r t ( x t , Σ t , u t )] = t = 0 ∑ T E [ Γ t + 1 F ( Σ t , K t x t ) ] ≤ 1 − A 2 C 1 3 ⋅ t = 0 ∑ T E [ F ( Σ t , K t x t )] = 1 − A 2 C 1 3 ( 1 − γ μ γ 2 γ + t 0 A 2 Σ ˉ ) .
Note that t 0 t_0 t 0 is chosen independent of T T T . This completes the proof.
Linear CE Advantage
This section proves Theorem 4, which shows that π C E \pi^{CE} π C E achieves a linear advantage over π O L \pi^{OL} π O L when q > q 4 q>q_4 q > q 4 , through the more general Proposition 3. Here, we show that the region M δ \mathcal{M}_\delta M δ , in which x ^ t 2 \widehat{x}_t^2 x t 2 and Σ ˉ − Σ t \bar{\Sigma}-\Sigma_t Σ ˉ − Σ t are small, is unstable under π C E \pi^{CE} π C E . Since x ^ t + 1 = M t x ^ t \widehat{x}_{t+1}=M_t\widehat{x}_t x t + 1 = M t x t is a multiplicative recursion, we measure growth on a logarithmic scale through the drift
Δ t : = E [ log x ^ t + 1 2 − log x ^ t 2 ∣ F t ] = E [ log M t 2 ∣ F t ] . \Delta_t:=\mathbb{E}\left[\log\widehat{x}_{t+1}^2-\log\widehat{x}_t^2\mid\mathcal{F}_t\right]=\mathbb{E}\left[\log M_t^2\mid\mathcal{F}_t\right]. Δ t := E [ log x t + 1 2 − log x t 2 ∣ F t ] = E [ log M t 2 ∣ F t ] .
Because M t M_t M t is conditionally Gaussian (Lemma 10 ), this drift can be bounded using moments of the logarithm of a noncentral chi-squared variable (Lemmas 16 and 17).
The argument proceeds in three steps. First, under condition (11) , the drift is at least ε \varepsilon ε whenever the belief lies in M δ \mathcal{M}_\delta M δ (Lemma 20 ), while it is bounded below by − ℓ -\ell − ℓ everywhere (Lemma 19 ); Lemma 18 ensures that log x ^ t 2 \log\widehat{x}_t^2 log x t 2 is well defined. Second, summing the drift Δ t \Delta_t Δ t over time, the left-hand side telescopes, and E [ log x ^ t 2 ] \mathbb{E}[\log\widehat{x}_t^2] E [ log x t 2 ] is bounded above by Lemma 11 . Hence the belief must spend a linear expected number of steps outside M δ \mathcal{M}_\delta M δ , which is (12) (Lemma 21). Third, at every such step either x ^ t 2 > δ \widehat{x}_t^2>\delta x t 2 > δ , so that b t b_t b t is bounded below, or Σ ˉ − Σ t > δ \bar{\Sigma}-\Sigma_t>\delta Σ ˉ − Σ t > δ , so that d t d_t d t is bounded below. Proposition 7 with λ = 1 \lambda=1 λ = 1 or λ = 0 \lambda=0 λ = 0 , respectively, then turns this into a linear advantage over π O L \pi^{OL} π O L . Finally, as δ , ε → 0 \delta,\varepsilon\to0 δ , ε → 0 , the left-hand side of (11) converges to D 2 + q D^2+q D 2 + q , so the condition holds for some sufficiently small δ , ε > 0 \delta,\varepsilon>0 δ , ε > 0 whenever q > q 4 q>q_4 q > q 4 , which proves Theorem 4. Figure 5 describes the proof dependencies for Section F.
Figure 5. Main proof dependencies for Section F. An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Proof of Theorem 4
Proof. Since q > q 4 = 2 e γ E − D 2 q>q_4=2e^{\gamma_E}-D^2 q > q 4 = 2 e γ E − D 2 , we have D 2 + q > 2 e γ E D^2+q>2e^{\gamma_E} D 2 + q > 2 e γ E . Hence
κ : = log ( D 2 + q ) − γ E − log 2 > 0. \kappa:=\log(D^2+q)-\gamma_E-\log2>0. κ := log ( D 2 + q ) − γ E − log 2 > 0.
For δ ∈ [ 0 , Σ ‾ ] \delta\in[0,\overline{\Sigma}] δ ∈ [ 0 , Σ ] define
ϕ ( δ ) : = A 2 ( Σ ‾ − δ ) 2 K 2 C 2 σ z 2 + K 2 C 2 ( Σ ‾ − δ ) δ . \phi(\delta) := \frac{A^{2}(\overline{\Sigma}-\delta)^{2}K^{2}C^{2}}{\sigma_{z}^{2}+K^{2}C^{2}(\overline{\Sigma}-\delta)\delta}. ϕ ( δ ) := σ z 2 + K 2 C 2 ( Σ − δ ) δ A 2 ( Σ − δ ) 2 K 2 C 2 .
On [ 0 , Σ ‾ ] [0,\overline{\Sigma}] [ 0 , Σ ] we have ( Σ ‾ − δ ) δ ≥ 0 (\overline{\Sigma}-\delta)\delta\ge0 ( Σ − δ ) δ ≥ 0 , so the denominator is at least σ z 2 > 0 \sigma_{z}^{2}>0 σ z 2 > 0 . Hence ϕ \phi ϕ is continuous on [ 0 , Σ ‾ ] [0,\overline{\Sigma}] [ 0 , Σ ] , and ϕ ( 0 ) = q \phi(0)=q ϕ ( 0 ) = q . Since D ≠ 0 D\ne0 D = 0 , we have D 2 + ϕ ( δ ) ≥ D 2 > 0 D^{2}+\phi(\delta)\ge D^{2}>0 D 2 + ϕ ( δ ) ≥ D 2 > 0 , so δ ↦ log ( D 2 + ϕ ( δ ) ) \delta\mapsto\log(D^{2}+\phi(\delta)) δ ↦ log ( D 2 + ϕ ( δ )) is continuous on [ 0 , Σ ‾ ] [0,\overline{\Sigma}] [ 0 , Σ ] and equals log ( D 2 + q ) \log(D^{2}+q) log ( D 2 + q ) at δ = 0 \delta=0 δ = 0 . Therefore there exists δ ∈ ( 0 , Σ ‾ ] \delta\in(0,\overline{\Sigma}] δ ∈ ( 0 , Σ ] such that
log ( D 2 + ϕ ( δ ) ) > log ( D 2 + q ) − κ 2 . \log(D^{2}+\phi(\delta))>\log(D^{2}+q)-\frac{\kappa}{2}. log ( D 2 + ϕ ( δ )) > log ( D 2 + q ) − 2 κ .
Set ε : = κ / 4 > 0 \varepsilon:=\kappa/4>0 ε := κ /4 > 0 . Then we have
log ( D 2 + ϕ ( δ ) ) − γ E − log 2 − ε > ( log ( D 2 + q ) − γ E − log 2 ) − κ 2 − κ 4 = κ 4 > 0 \log(D^{2}+\phi(\delta))-\gamma_E-\log2-\varepsilon>\left(\log(D^{2}+q)-\gamma_E-\log2\right)-\frac{\kappa}{2}-\frac{\kappa}{4}=\frac{\kappa}{4}>0 log ( D 2 + ϕ ( δ )) − γ E − log 2 − ε > ( log ( D 2 + q ) − γ E − log 2 ) − 2 κ − 4 κ = 4 κ > 0
which rearranges to (11) . Note that the pair ( δ , ε ) (\delta,\varepsilon) ( δ , ε ) is independent of T T T . Together with the assumptions μ x ≠ 0 \mu_x\ne0 μ x = 0 and D ≠ 0 D\ne0 D = 0 , all hypotheses of Proposition 3 hold, and Proposition 3 gives R T ( π C E , π O L ) = Ω ( T ) R_T(\pi^{CE},\pi^{OL})=\Omega(T) R T ( π C E , π O L ) = Ω ( T ) .
Proof of Proposition 3
Proof. Lemma 21 proves (12) in Proposition 3. Now we bound the regret. By Lemma 7, there exists t 5 t_5 t 5 such that K t 2 > K 2 / 2 K_t^2>K^2/2 K t 2 > K 2 /2 , N t > N / 2 N_t>N/2 N t > N /2 , Γ t + 1 > Γ / 2 \Gamma_{t+1}>\Gamma/2 Γ t + 1 > Γ/2 for all t ≤ T − t 5 t\le T-t_5 t ≤ T − t 5 . Take t 1 = max { t 4 , t 5 } t_1=\max\{t_4,t_5\} t 1 = max { t 4 , t 5 } where t 4 t_4 t 4 is obtained in Lemma 21. It follows from Lemma 21 that
max { ∑ t = 0 T − t 1 Pr ( x ^ t 2 > δ ) , ∑ t = 0 T − t 1 Pr ( Σ ‾ − Σ t > δ ) } ≥ ( T − t 1 ) ε + log ( μ x 2 C C E 2 ) 2 ( ε + ℓ ) (44) \max\left\{\sum_{t=0}^{T-t_1}\Pr(\hat{x}_t^2>\delta),\sum_{t=0}^{T-t_1}\Pr(\overline{\Sigma}-\Sigma_t>\delta)\right\}\ge\frac{(T-t_1)\varepsilon+\log\left(\frac{\mu_x^2}{C_{CE}^2}\right)}{2(\varepsilon+\ell)} \tag*{(44)} max { t = 0 ∑ T − t 1 Pr ( x ^ t 2 > δ ) , t = 0 ∑ T − t 1 Pr ( Σ − Σ t > δ ) } ≥ 2 ( ε + ℓ ) ( T − t 1 ) ε + log ( C C E 2 μ x 2 ) (44)
When ∑ t = 0 T − t 1 Pr ( x ^ t 2 > δ ) \sum_{t=0}^{T-t_1}\Pr(\hat{x}_t^2>\delta) ∑ t = 0 T − t 1 Pr ( x ^ t 2 > δ ) is larger applying Proposition 1 we have
R T ( π C E , π O L ) = ∑ t = 0 T E π C E [ r t ( x ^ t , Σ t , u t ) ] ≥ ∑ t = 0 T − t 1 E π C E [ r t ( x ^ t , Σ t , u t ) ] = ∑ t = 0 T − t 1 Γ t + 1 E [ A 2 K t 2 x ^ t 2 C 2 Σ t 2 σ z 2 + K t 2 x ^ t 2 C 2 Σ t 2 ] (for π C E , p t ( x ^ t , u t ) = 0 for all t ) \begin{aligned}
R_T(\pi^{CE},\pi^{OL})=\sum_{t=0}^{T}\mathbb{E}_{\pi_{CE}}[r_t(\hat{x}_t,\Sigma_t,u_t)] \\
&\ge\sum_{t=0}^{T-t_1}\mathbb{E}_{\pi_{CE}}[r_t(\hat{x}_t,\Sigma_t,u_t)] \\
&=\sum_{t=0}^{T-t_1}\Gamma_{t+1}\mathbb{E}\left[\frac{A^2K_t^2\hat{x}_t^2C^2\Sigma_t^2}{\sigma_z^2+K_t^2\hat{x}_t^2C^2\Sigma_t^2}\right]\qquad\text{(for }\pi^{CE}\text{, }p_t(\hat{x}_t,u_t)=0\text{ for all }t\text{)}
\end{aligned} R T ( π C E , π O L ) = t = 0 ∑ T E π C E [ r t ( x ^ t , Σ t , u t )] ≥ t = 0 ∑ T − t 1 E π C E [ r t ( x ^ t , Σ t , u t )] = t = 0 ∑ T − t 1 Γ t + 1 E [ σ z 2 + K t 2 x ^ t 2 C 2 Σ t 2 A 2 K t 2 x ^ t 2 C 2 Σ t 2 ] (for π C E , p t ( x ^ t , u t ) = 0 for all t )
≥ Γ 2 ∑ t = 0 T − t 1 E [ 1 2 A 2 K 2 x ^ t 2 C 2 Σ t 2 σ z 2 + 1 2 K 2 x ^ t 2 C 2 Σ t ] ≥ Γ 2 ∑ t = 0 T − t 1 E [ 1 2 A 2 K 2 x ^ t 2 C 2 Σ t 2 σ z 2 + 1 2 K 2 x ^ t 2 C 2 Σ t 1 { x ^ t 2 > δ } ] ≥ Γ 2 ∑ t = 0 T − t 1 1 2 A 2 K 2 δ C 2 Σ w 2 σ z 2 + 1 2 K 2 δ C 2 Σ w E [ 1 { x ^ t 2 > δ } ] = Γ 2 ⋅ 1 2 A 2 K 2 δ C 2 Σ w 2 σ z 2 + 1 2 K 2 δ C 2 Σ w ∑ t = 0 T − t 1 Pr ( x ^ t 2 > δ ) ≥ Γ 2 ⋅ 1 2 A 2 K 2 δ C 2 Σ w 2 σ z 2 + 1 2 K 2 δ C 2 Σ w ⋅ ( T − t 1 ) ε + log ( μ x 2 C C E 2 ) 2 ( ε + ℓ ) . \begin{aligned}
&\ge\frac{\Gamma}{2}\sum_{t=0}^{T-t_1}\mathbb{E}\left[\frac{\frac{1}{2}A^2K^2\hat{x}_t^2C^2\Sigma_t^2}{\sigma_z^2+\frac{1}{2}K^2\hat{x}_t^2C^2\Sigma_t}\right] \\
&\ge\frac{\Gamma}{2}\sum_{t=0}^{T-t_1}\mathbb{E}\left[\frac{\frac{1}{2}A^2K^2\hat{x}_t^2C^2\Sigma_t^2}{\sigma_z^2+\frac{1}{2}K^2\hat{x}_t^2C^2\Sigma_t}1\{\hat{x}_t^2>\delta\}\right] \\
&\ge\frac{\Gamma}{2}\sum_{t=0}^{T-t_1}\frac{\frac{1}{2}A^2K^2\delta C^2\Sigma_w^2}{\sigma_z^2+\frac{1}{2}K^2\delta C^2\Sigma_w}\mathbb{E}\left[1\{\hat{x}_t^2>\delta\}\right] \\
&= \frac{\Gamma}{2}\cdot\frac{\frac{1}{2}A^2K^2\delta C^2\Sigma_w^2}{\sigma_z^2+\frac{1}{2}K^2\delta C^2\Sigma_w}\sum_{t=0}^{T-t_1}\Pr(\hat{x}_t^2>\delta) \\
&\ge\frac{\Gamma}{2}\cdot\frac{\frac{1}{2}A^2K^2\delta C^2\Sigma_w^2}{\sigma_z^2+\frac{1}{2}K^2\delta C^2\Sigma_w}\cdot\frac{(T-t_1)\varepsilon+\log\left(\frac{\mu_x^2}{C_{CE}^2}\right)}{2(\varepsilon+\ell)}.
\end{aligned} ≥ 2 Γ t = 0 ∑ T − t 1 E [ σ z 2 + 2 1 K 2 x ^ t 2 C 2 Σ t 2 1 A 2 K 2 x ^ t 2 C 2 Σ t 2 ] ≥ 2 Γ t = 0 ∑ T − t 1 E [ σ z 2 + 2 1 K 2 x ^ t 2 C 2 Σ t 2 1 A 2 K 2 x ^ t 2 C 2 Σ t 2 1 { x ^ t 2 > δ } ] ≥ 2 Γ t = 0 ∑ T − t 1 σ z 2 + 2 1 K 2 δ C 2 Σ w 2 1 A 2 K 2 δ C 2 Σ w 2 E [ 1 { x ^ t 2 > δ } ] = 2 Γ ⋅ σ z 2 + 2 1 K 2 δ C 2 Σ w 2 1 A 2 K 2 δ C 2 Σ w 2 t = 0 ∑ T − t 1 Pr ( x ^ t 2 > δ ) ≥ 2 Γ ⋅ σ z 2 + 2 1 K 2 δ C 2 Σ w 2 1 A 2 K 2 δ C 2 Σ w 2 ⋅ 2 ( ε + ℓ ) ( T − t 1 ) ε + log ( C C E 2 μ x 2 ) .
When ∑ t = 0 T Pr ( Σ ˉ − Σ t > δ ) \sum_{t=0}^{T}\Pr(\bar{\Sigma}-\Sigma_t>\delta) ∑ t = 0 T Pr ( Σ ˉ − Σ t > δ ) is larger, applying Proposition 7 with λ = 0 \lambda=0 λ = 0 we have
R T ( π C E , π O L ) = ∑ t = 0 T K t 2 N t E [ Σ ˉ − Σ t ] ≥ ∑ t = 0 T − t 1 K t 2 N t E [ Σ ˉ − Σ t ] ≥ K 2 N 4 ∑ t = 0 T − t 1 E [ Σ ˉ − Σ t ] ≥ K 2 N 4 ⋅ δ ⋅ ∑ t = 0 T − t 1 Pr ( Σ ˉ − Σ t > δ ) ≥ K 2 N 4 ⋅ δ ⋅ ( T − t 1 ) ε + log ( μ x 2 C C E 2 ) 2 ( ε + ℓ ) . \begin{aligned}
R_T(\pi^{CE},\pi^{OL}) &= \sum_{t=0}^{T}K_t^2N_t\mathbb{E}\left[\bar{\Sigma}-\Sigma_t\right] \\
&\ge\sum_{t=0}^{T-t_1}K_t^2N_t\mathbb{E}\left[\bar{\Sigma}-\Sigma_t\right] \\
&\ge\frac{K^2N}{4}\sum_{t=0}^{T-t_1}\mathbb{E}\left[\bar{\Sigma}-\Sigma_t\right] \\
&\ge\frac{K^2N}{4}\cdot\delta\cdot\sum_{t=0}^{T-t_1}\Pr(\bar{\Sigma}-\Sigma_t>\delta) \\
&\ge\frac{K^2N}{4}\cdot\delta\cdot\frac{(T-t_1)\varepsilon+\log\left(\frac{\mu_x^2}{C_{CE}^2}\right)}{2(\varepsilon+\ell)}.
\end{aligned} R T ( π C E , π O L ) = t = 0 ∑ T K t 2 N t E [ Σ ˉ − Σ t ] ≥ t = 0 ∑ T − t 1 K t 2 N t E [ Σ ˉ − Σ t ] ≥ 4 K 2 N t = 0 ∑ T − t 1 E [ Σ ˉ − Σ t ] ≥ 4 K 2 N ⋅ δ ⋅ t = 0 ∑ T − t 1 Pr ( Σ ˉ − Σ t > δ ) ≥ 4 K 2 N ⋅ δ ⋅ 2 ( ε + ℓ ) ( T − t 1 ) ε + log ( C C E 2 μ x 2 ) .
It follows that R T ( π C E , π O L ) = Ω ( T ) R_T(\pi^{CE},\pi^{OL})=\Omega(T) R T ( π C E , π O L ) = Ω ( T ) .
Lemmas
Lemma 16. Define the function f f f on Z ≥ 0 \mathbb{Z}_{\ge0} Z ≥ 0 and the function g g g on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) given by
f ( n ) = { ∑ j = 0 n − 1 2 2 j + 1 n = 1 , 2 , … 0 n = 0 , and g ( μ ) = { E Y ∼ Pois ( μ ) [ f ( Y ) ] μ > 0 0 μ = 0. f(n)=
\begin{cases}
\sum_{j=0}^{n-1}\frac{2}{2j+1} & n=1,2,\ldots\\
0 & n=0,
\end{cases}
\qquad\text{and}\qquad
g(\mu)=
\begin{cases}
\mathbb{E}_{Y\sim\operatorname{Pois}(\mu)}[f(Y)] & \mu>0 \\
0 & \mu=0.
\end{cases} f ( n ) = { ∑ j = 0 n − 1 2 j + 1 2 0 n = 1 , 2 , … n = 0 , and g ( μ ) = { E Y ∼ Pois ( μ ) [ f ( Y )] 0 μ > 0 μ = 0. Then, g g g is right-continuous at μ = 0 \mu=0 μ = 0 and
g ′ ( μ ) = E Y ∼ Pois ( μ ) [ f ( Y + 1 ) − f ( Y ) ] = E Y ∼ Pois ( μ ) [ 2 2 Y + 1 ] ≥ 2 2 μ + 1 . (45) g'(\mu)=\mathbb{E}_{Y\sim\operatorname{Pois}(\mu)}[f(Y+1)-f(Y)]=\mathbb{E}_{Y\sim\operatorname{Pois}(\mu)}\left[\frac{2}{2Y+1}\right]\ge\frac{2}{2\mu+1}. \tag*{(45)} g ′ ( μ ) = E Y ∼ Pois ( μ ) [ f ( Y + 1 ) − f ( Y )] = E Y ∼ Pois ( μ ) [ 2 Y + 1 2 ] ≥ 2 μ + 1 2 . (45) This also implies that g ( μ ) ≥ log ( 1 + 2 μ ) g(\mu)\ge\log(1+2\mu) g ( μ ) ≥ log ( 1 + 2 μ ) for any μ ≥ 0 \mu\ge0 μ ≥ 0 .
Proof. By definition, we have
g ( μ ) = E Y ∼ Pois ( μ ) [ f ( Y ) ] = ∑ y = 0 ∞ f ( y ) e − μ μ y y ! . g(\mu)=\mathbb{E}_{Y\sim\operatorname{Pois}(\mu)}[f(Y)]=\sum_{y=0}^{\infty}f(y)\frac{e^{-\mu}\mu^y}{y!}. g ( μ ) = E Y ∼ Pois ( μ ) [ f ( Y )] = y = 0 ∑ ∞ f ( y ) y ! e − μ μ y .
Each term of f f f is at most 2, so 0 ≤ f ( y ) ≤ 2 y 0 \le f(y) \le2y 0 ≤ f ( y ) ≤ 2 y for all y ∈ Z ≥ 0 y \in\mathbb{Z}_{\ge0} y ∈ Z ≥ 0 . Hence, for every μ > 0 \mu> 0 μ > 0 ,
0 ≤ g ( μ ) ≤ E Y ∼ Pois ( μ ) [ 2 Y ] = 2 μ < ∞ . 0 \le g(\mu) \le\mathbb{E}_{Y \sim\operatorname{Pois}(\mu)}[2Y] = 2\mu< \infty. 0 ≤ g ( μ ) ≤ E Y ∼ Pois ( μ ) [ 2 Y ] = 2 μ < ∞.
Letting μ → 0 + \mu\to0+ μ → 0 + gives lim μ → 0 + g ( μ ) = 0 = g ( 0 ) \lim_{\mu\to0+} g(\mu) = 0 = g(0) lim μ → 0 + g ( μ ) = 0 = g ( 0 ) , which proves right-continuity at μ = 0 \mu= 0 μ = 0 . Next, we compute g ′ ( μ ) g'(\mu) g ′ ( μ ) . Define the function h ( μ ) = e μ g ( μ ) h(\mu) = e^\mu g(\mu) h ( μ ) = e μ g ( μ ) . For every R > 0 R > 0 R > 0 , we have
h ( R ) = ∑ y = 0 ∞ f ( y ) R y y ! ≤ 2 ∑ y = 1 ∞ R y ( y − 1 ) ! = 2 R e R < ∞ , h(R) = \sum_{y=0}^{\infty} f(y) \frac{R^y}{y!} \le2\sum_{y=1}^{\infty} \frac{R^y}{(y-1)!} = 2Re^R < \infty, h ( R ) = y = 0 ∑ ∞ f ( y ) y ! R y ≤ 2 y = 1 ∑ ∞ ( y − 1 )! R y = 2 R e R < ∞ ,
that is, h h h is a power series whose radius of convergence is infinite. Therefore, we may compute the derivative of h h h by differentiating each summands:
h ′ ( μ ) = ∑ y = 1 ∞ f ( y ) μ y − 1 ( y − 1 ) ! = ∑ y = 0 ∞ f ( y + 1 ) μ y y ! . h'(\mu) = \sum_{y=1}^{\infty} f(y) \frac{\mu^{y-1}}{(y-1)!} = \sum_{y=0}^{\infty} f(y+1) \frac{\mu^y}{y!}. h ′ ( μ ) = y = 1 ∑ ∞ f ( y ) ( y − 1 )! μ y − 1 = y = 0 ∑ ∞ f ( y + 1 ) y ! μ y .
Then, since g ( μ ) = e − μ h ( μ ) g(\mu) = e^{-\mu}h(\mu) g ( μ ) = e − μ h ( μ ) , we get
g ′ ( μ ) = ∑ y = 0 ∞ [ f ( y + 1 ) − f ( y ) ] e − μ μ y y ! = E Y ∼ Pois ( μ ) [ 2 2 Y + 1 ] ≥ 2 2 E Y ∼ Pois ( μ ) [ Y ] + 1 = 2 2 μ + 1 . g'(\mu) = \sum_{y=0}^{\infty} [f(y+1)-f(y)]\frac{e^{-\mu}\mu^y}{y!}
= \mathbb{E}_{Y \sim\operatorname{Pois}(\mu)}\left[\frac{2}{2Y+1}\right]
\ge\frac{2}{2\mathbb{E}_{Y \sim\operatorname{Pois}(\mu)}[Y]+1}
= \frac{2}{2\mu+1}. g ′ ( μ ) = y = 0 ∑ ∞ [ f ( y + 1 ) − f ( y )] y ! e − μ μ y = E Y ∼ Pois ( μ ) [ 2 Y + 1 2 ] ≥ 2 E Y ∼ Pois ( μ ) [ Y ] + 1 2 = 2 μ + 1 2 .
where the inequality is obtained from Jensen’s inequality. It follows that, for μ > 0 \mu> 0 μ > 0 ,
g ( μ ) = g ( μ ) − g ( 0 ) = ∫ 0 μ g ′ ( x ) d x ≥ ∫ 0 μ 2 2 x + 1 d x = log ( 1 + 2 μ ) . g(\mu) = g(\mu)-g(0) = \int_0^\mu g'(x)\,dx \ge\int_0^\mu\frac{2}{2x+1}\,dx = \log(1+2\mu). g ( μ ) = g ( μ ) − g ( 0 ) = ∫ 0 μ g ′ ( x ) d x ≥ ∫ 0 μ 2 x + 1 2 d x = log ( 1 + 2 μ ) .
If μ = 0 \mu= 0 μ = 0 , then g ( 0 ) = 0 = log 1 g(0) = 0 = \log1 g ( 0 ) = 0 = log 1 . Therefore, the proof is done.
Lemma 17. Let X ∼ N ( 0 , 1 ) X \sim\mathcal{N}(0,1) X ∼ N ( 0 , 1 ) . Then, for any c ∈ R c \in\mathbb{R} c ∈ R , we have
E [ log ( X + c ) 2 ] ≥ E [ log X 2 ] + log ( 1 + c 2 ) (46) \mathbb{E}[\log(X+c)^2] \ge\mathbb{E}[\log X^2] + \log(1+c^2) \tag*{(46)} E [ log ( X + c ) 2 ] ≥ E [ log X 2 ] + log ( 1 + c 2 ) (46) Proof. If c = 0 c=0 c = 0 , then it is trivial. Assume that c ≠ 0 c \ne0 c = 0 . Let Z i Z_i Z i denote the central chi-squared random variable with i i i degrees of freedom. Let ψ \psi ψ be the digamma function. It is well known [20 ] that
E [ log Z i ] = log 2 + ψ ( i / 2 ) . (47) \mathbb{E}[\log Z_i] = \log2 + \psi(i/2). \tag*{(47)} E [ log Z i ] = log 2 + ψ ( i /2 ) . (47) Observe that ( X + c ) 2 (X+c)^2 ( X + c ) 2 follows a non-central chi-squared distribution with non-centrality parameter c 2 c^2 c 2 and 1 degree of freedom. Let Y Y Y be a Poisson random variable with mean c 2 2 \frac{c^2}{2} 2 c 2 . It is also well known [20 ] that
( X + c ) 2 = d Z 1 + 2 Y . (48) (X+c)^2 \overset{d}{=} Z_{1+2Y}. \tag*{(48)} ( X + c ) 2 = d Z 1 + 2 Y . (48) Using (F.4) and (F.5), we get
E [ log ( X + c ) 2 ] = E [ log Z 1 + 2 Y ] = E Y [ E [ log Z 1 + 2 Y ∣ Y ] ] = E Y [ E [ log 2 + ψ ( 1 2 + Y ) ∣ Y ] ] = log 2 + E Y [ ψ ( 1 2 + Y ) ] \begin{aligned}
\mathbb{E}[\log(X+c)^2] &= \mathbb{E}[\log Z_{1+2Y}] \\
&= \mathbb{E}_Y[\mathbb{E}[\log Z_{1+2Y}\mid Y]] \\
&= \mathbb{E}_Y\left[\mathbb{E}\left[\left.\log2+\psi\left(\frac{1}{2}+Y\right)\right|Y\right]\right] \\
&= \log2+\mathbb{E}_Y\left[\psi\left(\frac{1}{2}+Y\right)\right]
\end{aligned} E [ log ( X + c ) 2 ] = E [ log Z 1 + 2 Y ] = E Y [ E [ log Z 1 + 2 Y ∣ Y ]] = E Y [ E [ log 2 + ψ ( 2 1 + Y ) Y ] ] = log 2 + E Y [ ψ ( 2 1 + Y ) ]
By the recurrence of ψ \psi ψ , we have
E Y [ ψ ( 1 2 + Y ) ] = ψ ( 1 / 2 ) + E Y [ ∑ j = 0 Y − 1 2 2 j + 1 ] . (49) \mathbb{E}_{Y}\left[\psi\left(\frac{1}{2}+Y\right)\right]=\psi(1/2)+\mathbb{E}_{Y}\left[\sum_{j=0}^{Y-1}\frac{2}{2j+1}\right]. \tag*{(49)} E Y [ ψ ( 2 1 + Y ) ] = ψ ( 1/2 ) + E Y [ j = 0 ∑ Y − 1 2 j + 1 2 ] . (49)
Using the functions f f f and g g g defined in Lemma 16 , the lemma implies
E Y [ ψ ( 1 2 + Y ) ] = ψ ( 1 / 2 ) + E [ f ( Y ) ] = ψ ( 1 / 2 ) + g ( c 2 / 2 ) ≥ ψ ( 1 / 2 ) + log ( 1 + c 2 ) . \mathbb{E}_{Y}\left[\psi\left(\frac{1}{2}+Y\right)\right]=\psi(1/2)+\mathbb{E}[f(Y)]=\psi(1/2)+g(c^{2}/2)\geq\psi(1/2)+\log(1+c^{2}). E Y [ ψ ( 2 1 + Y ) ] = ψ ( 1/2 ) + E [ f ( Y )] = ψ ( 1/2 ) + g ( c 2 /2 ) ≥ ψ ( 1/2 ) + log ( 1 + c 2 ) .
Therefore, we have
E [ log ( X + c ) 2 ] = log 2 + E Y [ ψ ( 1 2 + Y ) ] ≥ log 2 + ψ ( 1 / 2 ) + log ( 1 + c 2 ) = E [ log X 2 ] + log ( 1 + c 2 ) \begin{aligned}
\mathbb{E}\left[\log(X+c)^{2}\right]&=\log2+\mathbb{E}_{Y}\left[\psi\left(\frac{1}{2}+Y\right)\right]\\
&\geq\log2+\psi(1/2)+\log(1+c^{2})\\
&=\mathbb{E}\left[\log X^{2}\right]+\log(1+c^{2})
\end{aligned} E [ log ( X + c ) 2 ] = log 2 + E Y [ ψ ( 2 1 + Y ) ] ≥ log 2 + ψ ( 1/2 ) + log ( 1 + c 2 ) = E [ log X 2 ] + log ( 1 + c 2 )
where the last equality is obtained from (F.4). This completes the proof.
Lemma 18. Assume that μ x ≠ 0 \mu_x\ne0 μ x = 0 and D ≠ 0 D\ne0 D = 0 . Under the policy π C E \pi^{CE} π C E , we have
P ( x ^ t 2 = 0 ) = 0 \mathbb{P}(\hat{x}_t^{2}=0)=0 P ( x ^ t 2 = 0 ) = 0 for all t ∈ Z ≥ 0 t\in\mathbb{Z}_{\geq0} t ∈ Z ≥ 0 .
Proof. By Lemma 10 (i), since x ^ 0 = μ x \hat{x}_0=\mu_x x ^ 0 = μ x , it follows that
x ^ t = μ x ∏ s = 0 t − 1 M s . \hat{x}_t=\mu_x\prod_{s=0}^{t-1}M_s. x ^ t = μ x s = 0 ∏ t − 1 M s .
As μ x ≠ 0 \mu_x\ne0 μ x = 0 , the event { x ^ t = 0 } \{\hat{x}_t=0\} { x ^ t = 0 } is contained in ⋃ s = 0 t − 1 { M s = 0 } \bigcup_{s=0}^{t-1}\{M_s=0\} ⋃ s = 0 t − 1 { M s = 0 } . Fix s s s . By Lemma 10 (ii), we have
P ( M s = 0 ∣ F s ) = P Z ∼ N ( 0 , 1 ) ( D s + q s Z = 0 ) = { 0 if q s > 0 , 1 { D s = 0 } if q s = 0. \mathbb{P}(M_s=0\mid\mathcal{F}_s)=\mathbb{P}_{Z\sim\mathcal{N}(0,1)}(D_s+q_sZ=0)=
\begin{cases}
0 & \text{if }q_s>0,\\
\mathbf{1}\{D_s=0\} & \text{if }q_s=0.
\end{cases} P ( M s = 0 ∣ F s ) = P Z ∼ N ( 0 , 1 ) ( D s + q s Z = 0 ) = { 0 1 { D s = 0 } if q s > 0 , if q s = 0.
Recall that, from Lemma 4, we have
D s = A R − B S R + B 2 P s + 1 s and D = A R − B S R + B 2 P . D_s=\frac{AR-BS}{R+B^{2}P_{s+1}^{s}} \quad\text{and}\quad D=\frac{AR-BS}{R+B^{2}P}. D s = R + B 2 P s + 1 s A R − B S and D = R + B 2 P A R − B S .
Since D ≠ 0 D\ne0 D = 0 , we have A R ≠ B S AR\ne BS A R = B S , and therefore, D s ≠ 0 D_s\ne0 D s = 0 . Hence, in both cases, we have P ( M s = 0 ∣ F s ) = 0 \mathbb{P}(M_s=0\mid\mathcal{F}_s)=0 P ( M s = 0 ∣ F s ) = 0 , and so P ( M s = 0 ) = 0 \mathbb{P}(M_s=0)=0 P ( M s = 0 ) = 0 . A union bound gives
P ( x ^ t = 0 ) ≤ ∑ s = 0 t − 1 P ( M s = 0 ) = 0 \mathbb{P}(\hat{x}_t=0)\leq\sum_{s=0}^{t-1}\mathbb{P}(M_s=0)=0 P ( x ^ t = 0 ) ≤ s = 0 ∑ t − 1 P ( M s = 0 ) = 0
which completes the proof.
Lemma 19. Assume D ≠ 0 D \ne0 D = 0 . Under the policy π C E \pi^{CE} π C E , there exists constants t 2 , ℓ > 0 t_2,\ell> 0 t 2 , ℓ > 0 such that
E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] ≥ − ℓ (50) \mathbb{E}\left[\log\hat{x}_{t+1}^{2}-\log\hat{x}_{t}^{2}\mid H_t\right] \ge-\ell \tag*{(50)} E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] ≥ − ℓ (50) for all t ≤ T − t 2 t \le T-t_2 t ≤ T − t 2 and x ^ t 2 > 0 \hat{x}_t^2>0 x ^ t 2 > 0 .
Proof. Since D 2 > 0 D^2>0 D 2 > 0 , it follows by Lemma 7 there exists t 2 > 0 t_2>0 t 2 > 0 such that D t 2 > D 2 2 D_t^2>\frac{D^2}{2} D t 2 > 2 D 2 for all t ≤ T − t 2 t\le T-t_2 t ≤ T − t 2 .
Let t ≤ T − t 2 t\le T-t_2 t ≤ T − t 2 . When q t = 0 q_t=0 q t = 0 , we have E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] = log D t 2 > log D 2 2 \mathbb{E}\left[\log\hat{x}_{t+1}^{2}-\log\hat{x}_{t}^{2}\mid H_t\right]=\log D_t^2>\log\frac{D^2}{2} E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] = log D t 2 > log 2 D 2 . When q t > 0 q_t>0 q t > 0 we have
E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] = E [ log ( D t + q t Z ) 2 ∣ H t ] ≥ inf a > 0 E [ log ( D t + a Z ) 2 ] = inf a > 0 { log a 2 + E [ log ( D t a + Z ) 2 ] } ≥ inf a > 0 { log a 2 + E [ log Z 2 ] + log ( 1 + D t 2 a 2 ) } = inf a > 0 log ( a 2 + D t 2 ) + E [ log Z 2 ] = log D t 2 + E [ log Z 2 ] = log D t 2 − log 2 − γ E > log D 2 2 − log 2 − γ E . \begin{aligned}
\mathbb{E}\left[\log\hat{x}_{t+1}^{2}-\log\hat{x}_{t}^{2}\mid H_t\right]
&=\mathbb{E}\left[\log(D_t+q_tZ)^2\mid H_t\right]\\
&\ge\inf_{a>0}\mathbb{E}\left[\log(D_t+aZ)^2\right]\\
&=\inf_{a>0}\left\{\log a^2+\mathbb{E}\left[\log\left(\frac{D_t}{a}+Z\right)^2\right]\right\}\\
&\ge\inf_{a>0}\left\{\log a^2+\mathbb{E}[\log Z^2]+\log\left(1+\frac{D_t^2}{a^2}\right)\right\}\\
&=\inf_{a>0}\log(a^2+D_t^2)+\mathbb{E}[\log Z^2]\\
&=\log D_t^2+\mathbb{E}[\log Z^2]\\
&=\log D_t^2-\log2-\gamma_E\\
&>\log\frac{D^2}{2}-\log2-\gamma_E.
\end{aligned} E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] = E [ log ( D t + q t Z ) 2 ∣ H t ] ≥ a > 0 inf E [ log ( D t + a Z ) 2 ] = a > 0 inf { log a 2 + E [ log ( a D t + Z ) 2 ] } ≥ a > 0 inf { log a 2 + E [ log Z 2 ] + log ( 1 + a 2 D t 2 ) } = a > 0 inf log ( a 2 + D t 2 ) + E [ log Z 2 ] = log D t 2 + E [ log Z 2 ] = log D t 2 − log 2 − γ E > log 2 D 2 − log 2 − γ E .
Hence we may take ℓ = log 2 + γ E − log D 2 2 > 0 \ell=\log2+\gamma_E-\log\frac{D^2}{2}>0 ℓ = log 2 + γ E − log 2 D 2 > 0 since ∣ D ∣ < 1 |D|<1 ∣ D ∣ < 1 .
Lemma 20. Fix ε , δ > 0 \varepsilon,\delta>0 ε , δ > 0 , with δ ≤ Σ ˉ \delta\le\bar{\Sigma} δ ≤ Σ ˉ . Suppose
log ( A 2 ( Σ ˉ − δ ) 2 K 2 C 2 σ z 2 + K 2 C 2 ( Σ ˉ − δ ) δ + D 2 ) − γ E − log 2 − ε > 0. (51) \log\left(\frac{A^2(\bar{\Sigma}-\delta)^2K^2C^2}{\sigma_z^2+K^2C^2(\bar{\Sigma}-\delta)\delta}+D^2\right)-\gamma_E-\log2-\varepsilon>0. \tag*{(51)} log ( σ z 2 + K 2 C 2 ( Σ ˉ − δ ) δ A 2 ( Σ ˉ − δ ) 2 K 2 C 2 + D 2 ) − γ E − log 2 − ε > 0. (51) Then there exists t 3 t_3 t 3 such that, for any t ≤ T − t 3 t\le T-t_3 t ≤ T − t 3 ,
E π C E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] > ε (52) \mathbb{E}_{\pi^{CE}}\left[\log\hat{x}_{t+1}^{2}-\log\hat{x}_{t}^{2}\mid H_t\right]>\varepsilon \tag*{(52)} E π C E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] > ε (52) whenever 0 < x ^ t 2 ≤ δ 0<\hat{x}_t^2\le\delta 0 < x ^ t 2 ≤ δ and Σ ˉ − Σ t ≤ δ \bar{\Sigma}-\Sigma_t\le\delta Σ ˉ − Σ t ≤ δ .
Proof. Define a function g g g on R \mathbb{R} R such that
g ( k ) = log ( A 2 ( Σ ˉ − δ ) 2 k 2 C 2 σ z 2 + k 2 C 2 ( Σ ˉ − δ ) δ + ( A + B k ) 2 ) − γ E − log 2 − ε . (53) g(k)=\log\left(\frac{A^2(\bar{\Sigma}-\delta)^2k^2C^2}{\sigma_z^2+k^2C^2(\bar{\Sigma}-\delta)\delta}+(A+Bk)^2\right)-\gamma_E-\log2-\varepsilon. \tag*{(53)} g ( k ) = log ( σ z 2 + k 2 C 2 ( Σ ˉ − δ ) δ A 2 ( Σ ˉ − δ ) 2 k 2 C 2 + ( A + B k ) 2 ) − γ E − log 2 − ε . (53)
Note that g g g is continuous in k k k and g ( K ) > 0 g(K)>0 g ( K ) > 0 by assumption. It follows by Lemma 7 that there exists t 3 t_3 t 3 such that g ( K t ) > 0 g(K_t)>0 g ( K t ) > 0 for all t ≤ T − t 3 t\le T-t_3 t ≤ T − t 3 . Fix a t t t with t ≤ T − t 3 t\le T-t_3 t ≤ T − t 3 . We have
E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] = E [ log M t 2 ∣ H t ] = E [ log ( D t + q t Z ) 2 ∣ H t ] = log ( q t 2 ) + E [ log ( D t q t + Z ) 2 ∣ H t ] . \begin{aligned}
\mathbb{E}\left[\log\hat{x}_{t+1}^{2}-\log\hat{x}_{t}^{2}\mid H_t\right]
&=\mathbb{E}\left[\log M_t^2\mid H_t\right]\\
&=\mathbb{E}\left[\log(D_t+q_tZ)^2\mid H_t\right]\\
&=\log(q_t^2)+\mathbb{E}\left[\log\left(\frac{D_t}{q_t}+Z\right)^2\mid H_t\right].
\end{aligned} E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] = E [ log M t 2 ∣ H t ] = E [ log ( D t + q t Z ) 2 ∣ H t ] = log ( q t 2 ) + E [ log ( q t D t + Z ) 2 ∣ H t ] .
By Lemma 17 we have
E [ log ( D t q t + Z ) 2 ∣ H t ] ≥ log ( 1 + ( D t q t ) 2 ) + E [ log Z 2 ] = log ( 1 + ( D t q t ) 2 ) − γ E − log 2. \mathbb{E}\left[\left.\log\left(\frac{D_t}{q_t}+Z\right)^2\right|H_t\right]\ge\log\left(1+\left(\frac{D_t}{q_t}\right)^2\right)+\mathbb{E}[\log Z^2]=\log\left(1+\left(\frac{D_t}{q_t}\right)^2\right)-\gamma_E-\log2. E [ log ( q t D t + Z ) 2 H t ] ≥ log ( 1 + ( q t D t ) 2 ) + E [ log Z 2 ] = log ( 1 + ( q t D t ) 2 ) − γ E − log 2.
Hence, we get
E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] ≥ log ( q t 2 ) + log ( 1 + ( D t q t ) 2 ) − γ E − log 2 = log ( q t 2 + D t 2 ) − γ E − log 2. \begin{aligned}
\mathbb{E}\left[\left.\log\hat{x}_{t+1}^2-\log\hat{x}_t^2\right|H_t\right]\ge\log(q_t^2)+\log\left(1+\left(\frac{D_t}{q_t}\right)^2\right)-\gamma_E-\log2 \\
&=\log\left(q_t^2+D_t^2\right)-\gamma_E-\log2.
\end{aligned} E [ log x ^ t + 1 2 − log x ^ t 2 H t ] ≥ log ( q t 2 ) + log ( 1 + ( q t D t ) 2 ) − γ E − log 2 = log ( q t 2 + D t 2 ) − γ E − log 2.
If x ^ t 2 ≤ δ \hat{x}_t^2\le\delta x ^ t 2 ≤ δ and Σ ˉ − Σ t ≤ δ \bar{\Sigma}-\Sigma_t\le\delta Σ ˉ − Σ t ≤ δ we have
q t 2 ≥ A 2 ( Σ ˉ − δ ) 2 K t 2 C 2 σ z 2 + K t 2 C 2 ( Σ ˉ − δ ) δ . q_t^2\ge\frac{A^2(\bar{\Sigma}-\delta)^2K_t^2C^2}{\sigma_z^2+K_t^2C^2(\bar{\Sigma}-\delta)\delta}. q t 2 ≥ σ z 2 + K t 2 C 2 ( Σ ˉ − δ ) δ A 2 ( Σ ˉ − δ ) 2 K t 2 C 2 .
Therefore, we conclude
E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] ≥ log ( q t 2 + D t 2 ) − γ E − log 2 ≥ log ( A 2 ( Σ ˉ − δ ) 2 K t 2 C 2 σ z 2 + K t 2 C 2 ( Σ ˉ − δ ) δ + D t 2 ) − γ E − log 2 = g ( K t ) + ε > ε \begin{aligned}
\mathbb{E}\left[\left.\log\hat{x}_{t+1}^2-\log\hat{x}_t^2\right|H_t\right]\ge\log(q_t^2+D_t^2)-\gamma_E-\log2 \\
&\ge\log\left(\frac{A^2(\bar{\Sigma}-\delta)^2K_t^2C^2}{\sigma_z^2+K_t^2C^2(\bar{\Sigma}-\delta)\delta}+D_t^2\right)-\gamma_E-\log2 \\
&=g(K_t)+\varepsilon\\
&>\varepsilon
\end{aligned} E [ log x ^ t + 1 2 − log x ^ t 2 H t ] ≥ log ( q t 2 + D t 2 ) − γ E − log 2 ≥ log ( σ z 2 + K t 2 C 2 ( Σ ˉ − δ ) δ A 2 ( Σ ˉ − δ ) 2 K t 2 C 2 + D t 2 ) − γ E − log 2 = g ( K t ) + ε > ε
and the proof is done. □
Lemma 21. Fix ε , δ > 0 \varepsilon,\delta>0 ε , δ > 0 , with δ ≤ Σ ˉ \delta\le\bar{\Sigma} δ ≤ Σ ˉ . Assume μ x ≠ 0 \mu_x\ne0 μ x = 0 , D ≠ 0 D\ne0 D = 0 , and suppose
log ( A 2 ( Σ ˉ − δ ) 2 K 2 C 2 σ z 2 + K 2 C 2 ( Σ ˉ − δ ) δ + D 2 ) − γ E − log 2 − ε > 0. (54) \log\left(\frac{A^2(\bar{\Sigma}-\delta)^2K^2C^2}{\sigma_z^2+K^2C^2(\bar{\Sigma}-\delta)\delta}+D^2\right)-\gamma_E-\log2-\varepsilon>0. \tag*{(54)} log ( σ z 2 + K 2 C 2 ( Σ ˉ − δ ) δ A 2 ( Σ ˉ − δ ) 2 K 2 C 2 + D 2 ) − γ E − log 2 − ε > 0. (54) Then there exists constants t 4 , ℓ > 0 t_4,\ell>0 t 4 , ℓ > 0 such that
∑ t = 0 T − t 4 { P ( x ^ t 2 > δ ) + P ( Σ ˉ − Σ t > δ ) } ≥ ( T − t 4 ) ε + log ( μ x 2 C C E ) ε + ℓ . (55) \sum_{t=0}^{T-t_4}\left\{\mathbb{P}\left(\hat{x}_t^2>\delta\right)+\mathbb{P}\left(\bar{\Sigma}-\Sigma_t>\delta\right)\right\}\ge\frac{(T-t_4)\varepsilon+\log\left(\frac{\mu_x^2}{C_{CE}}\right)}{\varepsilon+\ell}. \tag*{(55)} t = 0 ∑ T − t 4 { P ( x ^ t 2 > δ ) + P ( Σ ˉ − Σ t > δ ) } ≥ ε + ℓ ( T − t 4 ) ε + log ( C C E μ x 2 ) . (55)
Proof. Take t 4 = max { t 2 , t 3 } t_4=\max\{t_2,t_3\} t 4 = max { t 2 , t 3 } where t 2 , t 3 t_2,t_3 t 2 , t 3 are defined in Lemmas 19 and 20 . Fix a t t t with t ≤ T − t 4 t\le T-t_4 t ≤ T − t 4 . Recall the set M δ \mathcal{M}_\delta M δ defined in (10) . Then,
E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] = 1 { ( x ^ t , Σ t ) ∉ M δ } E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] + 1 { ( x ^ t , Σ t ) ∈ M δ } E [ log x ^ t + 1 2 − log x ^ t 2 ∣ H t ] ≥ − 1 { ( x ^ t , Σ t ) ∉ M δ } ℓ + ε 1 { ( x ^ t , Σ t ) ∈ M δ } (Lemmas 19 and 20) \begin{aligned}
\mathbb{E}\left[\left.\log\hat{x}_{t+1}^2-\log\hat{x}_t^2\right|H_t\right]
={}&\mathbf{1}\{(\hat{x}_t,\Sigma_t)\notin\mathcal{M}_\delta\}\mathbb{E}\left[\left.\log\hat{x}_{t+1}^2-\log\hat{x}_t^2\right|H_t\right]\\
&+\mathbf{1}\{(\hat{x}_t,\Sigma_t)\in\mathcal{M}_\delta\}\mathbb{E}\left[\left.\log\hat{x}_{t+1}^2-\log\hat{x}_t^2\right|H_t\right]\\
\ge{}&-\mathbf{1}\{(\hat{x}_t,\Sigma_t)\notin\mathcal{M}_\delta\}\ell+\varepsilon\mathbf{1}\{(\hat{x}_t,\Sigma_t)\in\mathcal{M}_\delta\}\qquad\text{(Lemmas 19 and 20)}
\end{aligned} E [ log x ^ t + 1 2 − log x ^ t 2 H t ] = ≥ 1 {( x ^ t , Σ t ) ∈ / M δ } E [ log x ^ t + 1 2 − log x ^ t 2 H t ] + 1 {( x ^ t , Σ t ) ∈ M δ } E [ log x ^ t + 1 2 − log x ^ t 2 H t ] − 1 {( x ^ t , Σ t ) ∈ / M δ } ℓ + ε 1 {( x ^ t , Σ t ) ∈ M δ } (Lemmas 19 and 20)
Since μ x ≠ 0 \mu_x\ne0 μ x = 0 , by Lemma 18 we have x ^ t 2 > 0 \hat{x}_t^2>0 x ^ t 2 > 0 almost surely for all t t t . Hence, the tower property says
E [ log x ^ t + 1 2 − log x ^ t 2 ] ≥ − P ( ( x ^ t , Σ t ) ∉ M δ ) ℓ + ε P ( ( x ^ t , Σ t ) ∈ M δ ) = − P ( ( x ^ t , Σ t ) ∉ M δ ) ℓ + ε ( 1 − P ( ( x ^ t , Σ t ) ∉ M δ ) ) = ε − P ( ( x ^ t , Σ t ) ∉ M δ ) ( ε + ℓ ) . \begin{aligned}
\mathbb{E}\left[\log\hat{x}_{t+1}^2-\log\hat{x}_t^2\right]
\ge{}&-\mathbb{P}\left((\hat{x}_t,\Sigma_t)\notin\mathcal{M}_\delta\right)\ell+\varepsilon\mathbb{P}\left((\hat{x}_t,\Sigma_t)\in\mathcal{M}_\delta\right)\\
={}&-\mathbb{P}\left((\hat{x}_t,\Sigma_t)\notin\mathcal{M}_\delta\right)\ell+\varepsilon\left(1-\mathbb{P}\left((\hat{x}_t,\Sigma_t)\notin\mathcal{M}_\delta\right)\right)\\
={}&\varepsilon-\mathbb{P}\left((\hat{x}_t,\Sigma_t)\notin\mathcal{M}_\delta\right)(\varepsilon+\ell).
\end{aligned} E [ log x ^ t + 1 2 − log x ^ t 2 ] ≥ = = − P ( ( x ^ t , Σ t ) ∈ / M δ ) ℓ + ε P ( ( x ^ t , Σ t ) ∈ M δ ) − P ( ( x ^ t , Σ t ) ∈ / M δ ) ℓ + ε ( 1 − P ( ( x ^ t , Σ t ) ∈ / M δ ) ) ε − P ( ( x ^ t , Σ t ) ∈ / M δ ) ( ε + ℓ ) .
By rearranging the term and summing over t t t from 0 0 0 to T − t 4 T-t_4 T − t 4 , we have
∑ t = 0 T − t 4 P ( ( x ^ t , Σ t ) ∉ M δ ) ≥ ( T − t 4 ) ε − E [ log x ^ t + 1 2 ] + log μ x 2 ε + ℓ \sum_{t=0}^{T-t_4} \mathbb{P}\left((\hat{x}_t,\Sigma_t)\notin\mathcal{M}_\delta\right) \ge\frac{(T-t_4)\varepsilon-\mathbb{E}[\log\hat{x}_{t+1}^2]+\log\mu_x^2}{\varepsilon+\ell} t = 0 ∑ T − t 4 P ( ( x ^ t , Σ t ) ∈ / M δ ) ≥ ε + ℓ ( T − t 4 ) ε − E [ log x ^ t + 1 2 ] + log μ x 2
By the union bound, we get
∑ t = 0 T − t 4 { P ( x ^ t 2 > δ ) + P ( Σ ˉ − Σ t > δ ) } ≥ ( T − t 4 ) ε − E [ log x ^ t + 1 2 ] + log μ x 2 ε + ℓ . (56) \sum_{t=0}^{T-t_4}\left\{\mathbb{P}\left(\hat{x}_t^2>\delta\right)+\mathbb{P}\left(\bar{\Sigma}-\Sigma_t>\delta\right)\right\}\ge\frac{(T-t_4)\varepsilon-\mathbb{E}[\log\hat{x}_{t+1}^2]+\log\mu_x^2}{\varepsilon+\ell}. \tag*{(56)} t = 0 ∑ T − t 4 { P ( x ^ t 2 > δ ) + P ( Σ ˉ − Σ t > δ ) } ≥ ε + ℓ ( T − t 4 ) ε − E [ log x ^ t + 1 2 ] + log μ x 2 . (56)
Using that E [ x ^ t + 1 2 ] ≤ C C E \mathbb{E}[\hat{x}_{t+1}^2]\le C_{CE} E [ x ^ t + 1 2 ] ≤ C C E by Lemma 11 and Jensen’s inequality finishes the proof.
Probing Policy
This section proves Proposition 5, which shows that the probing policy π P \pi^P π P achieves a linear advantage over π O L \pi^{OL} π O L when q > q 2 q>q_2 q > q 2 , together with Theorem 5 and the threshold comparisons in Propositions 4 and 6.
After each probe, π P \pi^P π P plays the CE action for at least the next L − 1 L-1 L − 1 steps, since probes are at least L L L steps apart. We call the probe together with these L − 1 L-1 L − 1 steps a probe block . A probe incurs its cost immediately, while its benefit arrives over the rest of the block, as the subsequent CE actions exploit the information it generated. A single step may therefore contribute negatively, so we analyze each probe block as a whole.
The main step is to show that, when q > q 2 q>q_2 q > q 2 , every probe block pays off in expectation (Lemma 26 ). For a small probe, both its cost and the value of the information it triggers scale with the square of the probe size, so whether the block pays off depends only on the instance through q q q . We first establish this for a block starting at the steady state (Lemma 24), where the computation is explicit, and then extend it by continuity (Lemma 25 ) to the blocks that actually occur, which start near the steady state. Every other time step lies outside M δ \mathcal{M}_\delta M δ and contributes a constant advantage on average. Summing over the horizon gives Proposition 5. The remaining results then follow quickly from Proposition 5, Proposition 2, and the stationary Riccati identities. Figure 6 describes the proof dependencies in Section G.
Figure 6. Main proof dependencies for Section G. An arrow points from a result to a result whose proof uses it. Common preliminary results are omitted.
Proof of Theorem 5
Proof. Recall that ∣ A ∣ < 1 |A|<1 ∣ A ∣ < 1 and ∣ D ∣ < 1 |D|<1 ∣ D ∣ < 1 . In particular, 1 − A D > 0 1-AD>0 1 − A D > 0 , and
q 3 − q 2 = ( 1 − D 2 ) − ( 1 − A 2 ) ( 1 − D 2 ) 2 ( 1 − A D ) = ( 1 − D 2 ) ( ( A − D ) 2 + 1 − D 2 ) 2 ( 1 − A D ) > 0. q_3-q_2=(1-D^2)-\frac{(1-A^2)(1-D^2)}{2(1-AD)}=\frac{(1-D^2)((A-D)^2+1-D^2)}{2(1-AD)}>0. q 3 − q 2 = ( 1 − D 2 ) − 2 ( 1 − A D ) ( 1 − A 2 ) ( 1 − D 2 ) = 2 ( 1 − A D ) ( 1 − D 2 ) (( A − D ) 2 + 1 − D 2 ) > 0.
e n d s m i d = 1 {ends_mid=1} e n d s m i d = 1
Hence q 2 < q 3 q_2 < q_3 q 2 < q 3 . For q > q 2 q > q_2 q > q 2 , by Proposition 5, there exists a probing policy π P \pi^{P} π P such that R T ( π P , π O L ) = Ω ( T ) R_T(\pi^{P},\pi^{OL}) = \Omega(T) R T ( π P , π O L ) = Ω ( T ) . By optimality of π ⋆ \pi^\star π ⋆ , we have J T ( π ⋆ ) ≤ J T ( π P ) J_T(\pi^\star) \le J_T(\pi^{P}) J T ( π ⋆ ) ≤ J T ( π P ) . Therefore, we have
R T ( π ⋆ , π O L ) = J T ( π O L ) − J T ( π ⋆ ) ≥ J T ( π O L ) − J T ( π P ) = R T ( π P , π O L ) = Ω ( T ) , R_T(\pi^\star,\pi^{OL}) = J_T(\pi^{OL}) - J_T(\pi^\star) \ge J_T(\pi^{OL}) - J_T(\pi^{P}) = R_T(\pi^{P},\pi^{OL}) = \Omega(T), R T ( π ⋆ , π O L ) = J T ( π O L ) − J T ( π ⋆ ) ≥ J T ( π O L ) − J T ( π P ) = R T ( π P , π O L ) = Ω ( T ) ,
which proves Theorem 5(i). The result of Theorem 5(ii) is clear by Proposition 2.
Proof of Proposition 4
Proof. This is directly implied by Theorem 5 and Proposition 6.
Proof of Proposition 5
Proof. Denote
B k = ∑ t = 0 L − 1 r τ k + t , λ ( x ^ τ k + t , Σ τ k + t , u τ k + t ) . (57) B_k = \sum_{t=0}^{L-1} r_{\tau_k+t,\lambda}(\hat{x}_{\tau_k+t},\Sigma_{\tau_k+t},u_{\tau_k+t}). \tag*{(57)} B k = t = 0 ∑ L − 1 r τ k + t , λ ( x ^ τ k + t , Σ τ k + t , u τ k + t ) . (57)
By Lemma 26 , there exist integers L ≥ 1 L \ge1 L ≥ 1 , t 6 ≥ 0 t_6 \ge0 t 6 ≥ 0 , constants δ , a , c 1 > 0 \delta,a,c_1 > 0 δ , a , c 1 > 0 , and λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) , all independent of T T T , such that π P \pi^{P} π P defined with parameters δ , a , L , t 6 \delta,a,L,t_6 δ , a , L , t 6 satisfies
E π P [ B k ∣ H τ k ] ≥ c 1 a 2 (58) \mathbb{E}_{\pi^{P}}[B_k \mid H_{\tau_k}] \ge c_1 a^2 \tag*{(58)} E π P [ B k ∣ H τ k ] ≥ c 1 a 2 (58)
on τ k < ∞ \tau_k < \infty τ k < ∞ , for every k ≥ 1 k \ge1 k ≥ 1 . Here, the probing times use the cutoff T − t 6 − L T-t_6-L T − t 6 − L and M δ \mathcal{M}_{\delta} M δ . By Lemma 7, there exists t 5 > 0 t_5 > 0 t 5 > 0 such that
K t 2 > K 2 2 , N t > N 2 , Γ t + 1 > Γ 2 (59) K_t^2 > \frac{K^2}{2}, \qquad N_t > \frac{N}{2}, \qquad\Gamma_{t+1} > \frac{\Gamma}{2} \tag*{(59)} K t 2 > 2 K 2 , N t > 2 N , Γ t + 1 > 2 Γ (59)
for all t ≤ T − t 5 t \le T-t_5 t ≤ T − t 5 . Let t P : = max { t 5 , t 6 } + L t_P := \max\{t_5,t_6\}+L t P := max { t 5 , t 6 } + L . Recall T = { τ k : k ≥ 1 , τ k < ∞ } \mathcal{T} = \{\tau_k : k \ge1,\ \tau_k < \infty\} T = { τ k : k ≥ 1 , τ k < ∞ } , and let M = ∣ T ∣ M = |\mathcal{T}| M = ∣ T ∣ . For the following set
Y : = ⋃ k : τ k < ∞ { τ k , … , τ k + L − 1 } , (60) \mathcal{Y} := \bigcup_{k:\tau_k<\infty}\{\tau_k,\ldots,\tau_k+L-1\}, \tag*{(60)} Y := k : τ k < ∞ ⋃ { τ k , … , τ k + L − 1 } , (60)
we have ∣ Y ∣ = L M |\mathcal{Y}| = LM ∣ Y ∣ = L M . Notice that we have
∑ t ∈ Y r t , λ ( x ^ t , Σ t , u t ) = ∑ k ≥ 1 1 { τ k < ∞ } B k \sum_{t\in\mathcal{Y}} r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t) = \sum_{k\ge1} 1\{\tau_k < \infty\}B_k t ∈ Y ∑ r t , λ ( x ^ t , Σ t , u t ) = k ≥ 1 ∑ 1 { τ k < ∞ } B k
From (G.2), the tower property gives the following bound
E π P [ 1 { τ k < ∞ } B k ] = E π P [ 1 { τ k < ∞ } E π P [ B k ∣ H τ k ] ] ≥ c 1 a 2 ⋅ E [ 1 { τ k < ∞ } ] \mathbb{E}_{\pi^{P}}[1\{\tau_k < \infty\}B_k] = \mathbb{E}_{\pi^{P}}[1\{\tau_k < \infty\}\mathbb{E}_{\pi^{P}}[B_k \mid H_{\tau_k}]]
\ge c_1a^2 \cdot\mathbb{E}[1\{\tau_k < \infty\}] E π P [ 1 { τ k < ∞ } B k ] = E π P [ 1 { τ k < ∞ } E π P [ B k ∣ H τ k ]] ≥ c 1 a 2 ⋅ E [ 1 { τ k < ∞ }]
Since M = ∑ k ≥ 1 1 { τ k < ∞ } M = \sum_{k\ge1}1\{\tau_k < \infty\} M = ∑ k ≥ 1 1 { τ k < ∞ } , we have
E [ ∑ t ∈ Y r t , λ ( x ^ t , Σ t , u t ) ] ≥ c 1 a 2 E [ M ] . \mathbb{E}\left[\sum_{t\in\mathcal{Y}}r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)\right] \ge c_1a^2\mathbb{E}[M]. E [ t ∈ Y ∑ r t , λ ( x ^ t , Σ t , u t ) ] ≥ c 1 a 2 E [ M ] .
Define
V = { t ∈ Z : 0 ≤ t ≤ T − t P } and Z : = V ∖ Y . \mathcal{V} = \{t\in\mathbb{Z}: 0\le t\le T-t_P\} \qquad\text{and} \qquad\mathcal{Z} := \mathcal{V}\setminus\mathcal{Y}. V = { t ∈ Z : 0 ≤ t ≤ T − t P } and Z := V ∖ Y .
For each t ∈ Z t \in\mathcal{Z} t ∈ Z we have ( x ^ t , Σ t ) ∉ M δ (\hat{x}_t,\Sigma_t) \notin\mathcal{M}_{\delta} ( x ^ t , Σ t ) ∈ / M δ . This is because, since t ∈ V t \in\mathcal{V} t ∈ V , if ( x ^ t , Σ t ) ∈ M δ (\hat{x}_t,\Sigma_t) \in\mathcal{M}_{\delta} ( x ^ t , Σ t ) ∈ M δ then either a probe will happen at t t t or has occurred fewer than L L L steps ago, implying t ∈ Y t \in\mathcal{Y} t ∈ Y , which is a contradiction. Thus
t ∈ Z ⟹ x ^ t 2 > δ or Σ ˉ − Σ t > δ t \in\mathcal{Z} \Longrightarrow\hat{x}_t^2 > \delta\text{ or } \bar{\Sigma}-\Sigma_t > \delta t ∈ Z ⟹ x ^ t 2 > δ or Σ ˉ − Σ t > δ
For the case when x ^ t 2 > δ \hat{x}_t^2 > \delta x ^ t 2 > δ , we have
r t , λ ( x ^ t , Σ t , u t ) = λ Γ t + 1 F ( Σ t , K t x ^ t ) + ( 1 − λ ) N t K t 2 [ Σ ˉ − Σ t ] ≥ λ Γ t + 1 A 2 K t 2 δ 2 C 2 Σ w 2 σ z 2 + K t 2 δ 2 C 2 Σ w ( x ^ t 2 > δ , Σ t ≥ Σ w ) ≥ λ ⋅ 1 2 A 2 K t 2 δ 2 C 2 Σ t 2 1 2 σ z 2 + 1 2 K t 2 δ 2 C 2 Σ t (by xref‘(G.3)‘). \begin{aligned}
r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)
&= \lambda\Gamma_{t+1}F(\Sigma_t,K_t\hat{x}_t)+(1-\lambda)N_tK_t^2[\bar{\Sigma}-\Sigma_t] \\
&\ge\lambda\Gamma_{t+1}\frac{A^2K_t^2\delta^2C^2\Sigma_w^2}{\sigma_z^2+K_t^2\delta^2C^2\Sigma_w}
\qquad(\hat{x}_t^2>\delta,\ \Sigma_t\ge\Sigma_w) \\
&\ge\lambda\cdot\frac{\frac{1}{2}A^2K_t^2\delta^2C^2\Sigma_t^2}{\frac{1}{2}\sigma_z^2+\frac{1}{2}K_t^2\delta^2C^2\Sigma_t}
\qquad\text{(by {xref}`(G.3)`).}
\end{aligned} r t , λ ( x ^ t , Σ t , u t ) = λ Γ t + 1 F ( Σ t , K t x ^ t ) + ( 1 − λ ) N t K t 2 [ Σ ˉ − Σ t ] ≥ λ Γ t + 1 σ z 2 + K t 2 δ 2 C 2 Σ w A 2 K t 2 δ 2 C 2 Σ w 2 ( x ^ t 2 > δ , Σ t ≥ Σ w ) ≥ λ ⋅ 2 1 σ z 2 + 2 1 K t 2 δ 2 C 2 Σ t 2 1 A 2 K t 2 δ 2 C 2 Σ t 2 (by xref ‘(G.3)‘).
When Σ ˉ − Σ t > δ \bar{\Sigma}-\Sigma_t>\delta Σ ˉ − Σ t > δ we have
r t , λ ( x ^ t , Σ t , u t ) = λ Γ t + 1 F ( Σ , K t x ^ t ) + ( 1 − λ ) N t K t 2 [ Σ ˉ − Σ t ] ≥ ( 1 − λ ) N t K t 2 [ Σ ˉ − Σ t ] ≥ ( 1 − λ ) N K 2 4 [ Σ ˉ − Σ t ] (by xref‘(G.3)‘) ≥ ( 1 − λ ) ⋅ N K 2 4 δ . \begin{aligned}
r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)
&= \lambda\Gamma_{t+1}F(\Sigma,K_t\hat{x}_t)+(1-\lambda)N_tK_t^2[\bar{\Sigma}-\Sigma_t] \\
&\ge(1-\lambda)N_tK_t^2[\bar{\Sigma}-\Sigma_t] \\
&\ge(1-\lambda)\frac{NK^2}{4}[\bar{\Sigma}-\Sigma_t]
\qquad\text{(by {xref}`(G.3)`) }\\
&\ge(1-\lambda)\cdot\frac{NK^2}{4}\delta.
\end{aligned} r t , λ ( x ^ t , Σ t , u t ) = λ Γ t + 1 F ( Σ , K t x ^ t ) + ( 1 − λ ) N t K t 2 [ Σ ˉ − Σ t ] ≥ ( 1 − λ ) N t K t 2 [ Σ ˉ − Σ t ] ≥ ( 1 − λ ) 4 N K 2 [ Σ ˉ − Σ t ] (by xref ‘(G.3)‘) ≥ ( 1 − λ ) ⋅ 4 N K 2 δ .
Let
c 2 : = min { λ ⋅ 1 2 A 2 K 2 δ 2 C 2 Σ w 2 σ z 2 + 1 2 K 2 δ C 2 Σ w , ( 1 − λ ) ⋅ N K 2 4 δ } > 0 c_2 := \min\left\{\lambda\cdot\frac{\frac{1}{2}A^2K^2\delta^2C^2\Sigma_w^2}{\sigma_z^2+\frac{1}{2}K^2\delta C^2\Sigma_w},(1-\lambda)\cdot\frac{NK^2}{4}\delta\right\}>0 c 2 := min { λ ⋅ σ z 2 + 2 1 K 2 δ C 2 Σ w 2 1 A 2 K 2 δ 2 C 2 Σ w 2 , ( 1 − λ ) ⋅ 4 N K 2 δ } > 0
It follows that r t , λ ( x ^ t , Σ t , u t ) ≥ c 2 r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)\ge c_2 r t , λ ( x ^ t , Σ t , u t ) ≥ c 2 for all t ∈ Z t\in\mathcal{Z} t ∈ Z . So
E [ ∑ t ∈ Z r t , λ ( x ^ t , Σ t , u t ) ] ≥ c 2 E [ ∣ Z ∣ ] . \mathbb{E}\left[\sum_{t\in\mathcal{Z}}r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)\right]\ge c_2\mathbb{E}[|\mathcal{Z}|]. E [ t ∈ Z ∑ r t , λ ( x ^ t , Σ t , u t ) ] ≥ c 2 E [ ∣ Z ∣ ] .
Finally, let E = { 0 , … , T } ∖ ( Y ∪ Z ) E=\{0,\ldots,T\}\setminus(\mathcal{Y}\cup\mathcal{Z}) E = { 0 , … , T } ∖ ( Y ∪ Z ) . Note for all t ∈ E t\in E t ∈ E , u t = K t x ^ t u_t=K_t\hat{x}_t u t = K t x ^ t so r t , λ ( x ^ t , Σ t , u t ) ≥ 0 r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)\ge0 r t , λ ( x ^ t , Σ t , u t ) ≥ 0 . Hence, we have
∑ t = 0 T E π P [ r t , λ ( x ^ t , Σ t , u t ) ] = E π P [ ∑ t ∈ E r t , λ ( x ^ t , Σ t , u t ) + ∑ t ∈ Y r t , λ ( x ^ t , Σ t , u t ) + ∑ t ∈ Z r t , λ ( x ^ t , Σ t , u t ) ] ≥ c 1 a 2 E [ M ] + c 2 E [ ∣ Z ∣ ] = c 1 a 2 E [ ∣ Y ∣ L ] + c 2 E [ ∣ Z ∣ ] ≥ min { c 1 a 2 L , c 2 } ⋅ E [ ∣ Y ∣ + ∣ Z ∣ ] ≥ min { c 1 a 2 L , c 2 } ⋅ E [ ∣ V ∣ ] = min { c 1 a 2 L , c 2 } ⋅ ( T − t P + 1 ) . \begin{aligned}
\sum_{t=0}^{T}\mathbb{E}_{\pi^P}[r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)]
&=\mathbb{E}_{\pi^P}\left[\sum_{t\in E}r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)+\sum_{t\in\mathcal{Y}}r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)+\sum_{t\in\mathcal{Z}}r_{t,\lambda}(\hat{x}_t,\Sigma_t,u_t)\right]\\
&\ge c_1a^2\mathbb{E}[M]+c_2\mathbb{E}[|\mathcal{Z}|]\\
&=c_1a^2\mathbb{E}\left[\frac{|\mathcal{Y}|}{L}\right]+c_2\mathbb{E}[|\mathcal{Z}|]\\
&\ge\min\left\{\frac{c_1a^2}{L},c_2\right\}\cdot\mathbb{E}[|\mathcal{Y}|+|\mathcal{Z}|]\\
&\ge\min\left\{\frac{c_1a^2}{L},c_2\right\}\cdot\mathbb{E}[|\mathcal{V}|]\\
&=\min\left\{\frac{c_1a^2}{L},c_2\right\}\cdot(T-t_P+1).
\end{aligned} t = 0 ∑ T E π P [ r t , λ ( x ^ t , Σ t , u t )] = E π P [ t ∈ E ∑ r t , λ ( x ^ t , Σ t , u t ) + t ∈ Y ∑ r t , λ ( x ^ t , Σ t , u t ) + t ∈ Z ∑ r t , λ ( x ^ t , Σ t , u t ) ] ≥ c 1 a 2 E [ M ] + c 2 E [ ∣ Z ∣ ] = c 1 a 2 E [ L ∣ Y ∣ ] + c 2 E [ ∣ Z ∣ ] ≥ min { L c 1 a 2 , c 2 } ⋅ E [ ∣ Y ∣ + ∣ Z ∣ ] ≥ min { L c 1 a 2 , c 2 } ⋅ E [ ∣ V ∣ ] = min { L c 1 a 2 , c 2 } ⋅ ( T − t P + 1 ) .
Therefore, by Proposition 7, we now conclude that R T ( π P , π O L ) = Ω ( T ) R_T(\pi^P,\pi^{OL})=\Omega(T) R T ( π P , π O L ) = Ω ( T ) .
Proof of Proposition 6
Proof. Throughout the proof, we use the fact that ∣ A ∣ < 1 |A|<1 ∣ A ∣ < 1 and ∣ D ∣ < 1 |D|<1 ∣ D ∣ < 1 . Recall the quantities that we want to compare:
q 1 = K 2 ⋅ 1 − A 2 Q ⋅ ( R − S 2 Q ) , q 2 = ( 1 − A 2 ) ( 1 − D 2 ) 2 ( 1 − A D ) . q_1 = K^2 \cdot\frac{1-A^2}{Q} \cdot\left(R-\frac{S^2}{Q}\right), \qquad q_2 = \frac{(1-A^2)(1-D^2)}{2(1-AD)}. q 1 = K 2 ⋅ Q 1 − A 2 ⋅ ( R − Q S 2 ) , q 2 = 2 ( 1 − A D ) ( 1 − A 2 ) ( 1 − D 2 ) .
We use the following 3 identities
N = R + B 2 P (61) N=R+B^2P \tag*{(61)} N = R + B 2 P (61)
− K N = S + B P A (62) -KN=S+BPA \tag*{(62)} − K N = S + B P A (62)
P = A 2 P + Q − K 2 N . (63) P=A^2P+Q-K^2N. \tag*{(63)} P = A 2 P + Q − K 2 N . (63)
The identities (G.6) and (G.7) give different expressions for K 2 N K^2N K 2 N :
K 2 N = Q − ( 1 − A 2 ) P (64) K^2N=Q-(1-A^2)P \tag*{(64)} K 2 N = Q − ( 1 − A 2 ) P (64)
K 2 N = − K ( S + B P A ) = − K S − D P A + P A 2 K^2N=-K(S+BPA)=-KS-DPA+PA^2 K 2 N = − K ( S + B P A ) = − K S − D P A + P A 2
where we used B K = D − A BK=D-A B K = D − A in the second expression. Equating the two expressions for K 2 N K^2N K 2 N and solving for K S KS K S , we get
K S = ( 1 − A D ) P − Q . KS=(1-AD)P-Q. K S = ( 1 − A D ) P − Q .
Further, the identities (G.5) and (G.8) give
K 2 R = K 2 N − K 2 B 2 P = Q − ( 1 − A 2 ) P − ( D − A ) 2 P = Q − ( 1 − 2 A D + D 2 ) P K^2R=K^2N-K^2B^2P=Q-(1-A^2)P-(D-A)^2P=Q-(1-2AD+D^2)P K 2 R = K 2 N − K 2 B 2 P = Q − ( 1 − A 2 ) P − ( D − A ) 2 P = Q − ( 1 − 2 A D + D 2 ) P
where we again used B K = D − A BK=D-A B K = D − A . Hence, we get
K 2 ( R − S 2 Q ) = K 2 R − ( K S ) 2 Q = Q − ( 1 − 2 A D + D 2 ) P − ( ( 1 − A D ) P − Q ) 2 Q = Q − ( 1 − 2 A D + D 2 ) P − Q + 2 ( 1 − D A ) P − ( 1 − A D ) 2 P 2 Q = P ( 1 − D 2 ) − ( 1 − A D ) 2 P 2 Q . \begin{aligned}
K^2\left(R-\frac{S^2}{Q}\right)&=K^2R-\frac{(KS)^2}{Q}\\
&=Q-(1-2AD+D^2)P-\frac{\left((1-AD)P-Q\right)^2}{Q}\\
&=Q-(1-2AD+D^2)P-Q+2(1-DA)P-\frac{(1-AD)^2P^2}{Q}\\
&=P(1-D^2)-\frac{(1-AD)^2P^2}{Q}.
\end{aligned} K 2 ( R − Q S 2 ) = K 2 R − Q ( K S ) 2 = Q − ( 1 − 2 A D + D 2 ) P − Q ( ( 1 − A D ) P − Q ) 2 = Q − ( 1 − 2 A D + D 2 ) P − Q + 2 ( 1 − D A ) P − Q ( 1 − A D ) 2 P 2 = P ( 1 − D 2 ) − Q ( 1 − A D ) 2 P 2 .
Multiplying by ( 1 − A 2 ) / Q (1-A^2)/Q ( 1 − A 2 ) / Q gives
q 1 = K 2 ⋅ 1 − A 2 Q ⋅ ( R − S 2 Q ) = 1 − A 2 Q ( P ( 1 − D 2 ) − ( 1 − A D ) 2 P 2 Q ) = ( 1 − A 2 ) ( ( 1 − D 2 ) 2 4 ( 1 − A D ) 2 − ( 1 − D 2 ) 2 4 ( 1 − A D ) 2 + P ( 1 − D 2 ) ( 1 − A D ) Q ( 1 − A D ) − ( 1 − A D ) 2 P 2 Q 2 ) = ( 1 − A 2 ) ( ( 1 − D 2 ) 2 4 ( 1 − A D ) 2 − ( 1 − D 2 2 ( 1 − A D ) − ( 1 − A D ) P Q ) 2 ) ≤ ( 1 − A 2 ) ( 1 − D 2 ) 2 4 ( 1 − A D ) 2 = q 2 1 − D 2 2 ( 1 − A D ) \begin{aligned}
q_1&=K^2\cdot\frac{1-A^2}{Q}\cdot\left(R-\frac{S^2}{Q}\right)\\
&=\frac{1-A^2}{Q}\left(P(1-D^2)-\frac{(1-AD)^2P^2}{Q}\right)\\
&=(1-A^2)\left(\frac{(1-D^2)^2}{4(1-AD)^2}-\frac{(1-D^2)^2}{4(1-AD)^2}+\frac{P(1-D^2)(1-AD)}{Q(1-AD)}-\frac{(1-AD)^2P^2}{Q^2}\right)\\
&=(1-A^2)\left(\frac{(1-D^2)^2}{4(1-AD)^2}-\left(\frac{1-D^2}{2(1-AD)}-\frac{(1-AD)P}{Q}\right)^2\right) \\
&\le(1-A^2)\frac{(1-D^2)^2}{4(1-AD)^2} \\
&= q_2\frac{1-D^2}{2(1-AD)}
\end{aligned} q 1 = K 2 ⋅ Q 1 − A 2 ⋅ ( R − Q S 2 ) = Q 1 − A 2 ( P ( 1 − D 2 ) − Q ( 1 − A D ) 2 P 2 ) = ( 1 − A 2 ) ( 4 ( 1 − A D ) 2 ( 1 − D 2 ) 2 − 4 ( 1 − A D ) 2 ( 1 − D 2 ) 2 + Q ( 1 − A D ) P ( 1 − D 2 ) ( 1 − A D ) − Q 2 ( 1 − A D ) 2 P 2 ) = ( 1 − A 2 ) ( 4 ( 1 − A D ) 2 ( 1 − D 2 ) 2 − ( 2 ( 1 − A D ) 1 − D 2 − Q ( 1 − A D ) P ) 2 ) ≤ ( 1 − A 2 ) 4 ( 1 − A D ) 2 ( 1 − D 2 ) 2 = q 2 2 ( 1 − A D ) 1 − D 2
One can observe that 2 ( 1 − A D ) − ( 1 − D 2 ) ≥ 1 − A 2 > 0 2(1-AD)-(1-D^2)\ge1-A^2>0 2 ( 1 − A D ) − ( 1 − D 2 ) ≥ 1 − A 2 > 0 , that is,
1 − D 2 2 ( 1 − A D ) < 1. \frac{1-D^2}{2(1-AD)}<1. 2 ( 1 − A D ) 1 − D 2 < 1.
Therefore, we conclude that q 1 < q 2 q_1<q_2 q 1 < q 2 .
Lemmas
Throughout this section, we analyze a single probe block in isolation. Fix a block length L L L , a positive integer, and a probe size a > 0 a>0 a > 0 . The Riccati coefficients used within the block are collected in a vector
θ = ( p 1 , … , p L , γ 1 , … , γ L ) ∈ [ 0 , ∞ ) 2 L , (65) \theta=(p_1,\ldots,p_L,\gamma_1,\ldots,\gamma_L)\in[0,\infty)^{2L}, \tag*{(65)} θ = ( p 1 , … , p L , γ 1 , … , γ L ) ∈ [ 0 , ∞ ) 2 L , (65)
where, for a block starting at time τ \tau τ , the coordinates p t + 1 p_{t+1} p t + 1 and γ t + 1 \gamma_{t+1} γ t + 1 play the roles of P τ + t + 1 P_{\tau+t+1} P τ + t + 1 and Γ τ + t + 1 \Gamma_{\tau+t+1} Γ τ + t + 1 . For t = 0 , … , L − 1 t=0,\ldots,L-1 t = 0 , … , L − 1 , the corresponding gain is
n t = R + B 2 p t + 1 , k t = − n t − 1 ( B p t + 1 A + S ) , (66) n_t=R+B^2p_{t+1},\qquad k_t=-n_t^{-1}(Bp_{t+1}A+S), \tag*{(66)} n t = R + B 2 p t + 1 , k t = − n t − 1 ( B p t + 1 A + S ) , (66)
where n t ≥ R > 0 n_t\ge R>0 n t ≥ R > 0 since p t + 1 ≥ 0 p_{t+1}\ge0 p t + 1 ≥ 0 . The block policy π θ , a \pi^{\theta,a} π θ , a plays the role of π P \pi^{\mathrm{P}} π P during one block: it probes once at the start and then follows the CE action,
π 0 θ , a ( H 0 ) = k 0 x ^ 0 + a , π t θ , a ( H t ) = k t x ^ t ( 1 ≤ t ≤ L − 1 ) . \pi^{\theta,a}_0(H_0)=k_0\hat{x}_0+a,\qquad\pi^{\theta,a}_t(H_t)=k_t\hat{x}_t\quad(1\le t\le L-1). π 0 θ , a ( H 0 ) = k 0 x ^ 0 + a , π t θ , a ( H t ) = k t x ^ t ( 1 ≤ t ≤ L − 1 ) .
For λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , the per-step payoff and the expected payoff of a block are
r t , λ θ ( x ^ , Σ , u ) = − n t ( u − k t x ^ ) 2 + ( λ ⋅ γ t + 1 ⋅ F ( Σ , u ) + ( 1 − λ ) ⋅ n t k t 2 [ Σ ˉ − Σ ] ) , (67) r_{t,\lambda}^{\theta}(\hat{x},\Sigma,u)=-n_t(u-k_t\hat{x})^2+\left(\lambda\cdot\gamma_{t+1}\cdot F(\Sigma,u)+(1-\lambda)\cdot n_tk_t^2[\bar{\Sigma}-\Sigma]\right), \tag*{(67)} r t , λ θ ( x ^ , Σ , u ) = − n t ( u − k t x ^ ) 2 + ( λ ⋅ γ t + 1 ⋅ F ( Σ , u ) + ( 1 − λ ) ⋅ n t k t 2 [ Σ ˉ − Σ ] ) , (67)
G L , λ ( a ; x ^ , Σ , θ ) = E π θ , a [ ∑ t = 0 L − 1 r t , λ θ ( x ^ t , Σ t , u t ) ∣ x ^ 0 = x ^ , Σ 0 = Σ ] . (68) G_{L,\lambda}(a;\hat{x},\Sigma,\theta)=\mathbb{E}_{\pi^{\theta,a}}\left[\sum_{t=0}^{L-1}r_{t,\lambda}^{\theta}(\hat{x}_t,\Sigma_t,u_t)\mid\hat{x}_0=\hat{x},\Sigma_0=\Sigma\right]. \tag*{(68)} G L , λ ( a ; x ^ , Σ , θ ) = E π θ , a [ t = 0 ∑ L − 1 r t , λ θ ( x ^ t , Σ t , u t ) ∣ x ^ 0 = x ^ , Σ 0 = Σ ] . (68)
The stationary coefficients θ ˉ \bar{\theta} θ ˉ is defined as
θ ˉ = ( P , … , P , Γ , … , Γ ) ∈ R 2 L , (69) \bar{\theta}=(P,\ldots,P,\Gamma,\ldots,\Gamma)\in\mathbb{R}^{2L}, \tag*{(69)} θ ˉ = ( P , … , P , Γ , … , Γ ) ∈ R 2 L , (69)
and the expected payoff of the block starting at the steady state is denoted as
f L ( a ) = G L , 1 ( a ; 0 , Σ ˉ , θ ˉ ) . (70) f_L(a)=G_{L,1}(a;0,\bar{\Sigma},\bar{\theta}). \tag*{(70)} f L ( a ) = G L , 1 ( a ; 0 , Σ ˉ , θ ˉ ) . (70)
Lemma 22. We have
lim inf a ↓ 0 f L ( a ) a 2 ≥ N ( g ( L ) − 1 ) , (71) \liminf_{a\downarrow0}\frac{f_L(a)}{a^2}\ge N\bigl(g(L)-1\bigr), \tag*{(71)} a ↓ 0 lim inf a 2 f L ( a ) ≥ N ( g ( L ) − 1 ) , (71) where
g ( L ) = q 1 − A 2 ( 1 + ( ( D − A ) 2 + q ) ∑ j = 0 L − 2 ( D 2 + q ) j ) (72) g(L)=\frac{q}{1-A^2}\left(1+\bigl((D-A)^2+q\bigr)\sum_{j=0}^{L-2}(D^2+q)^j\right) \tag*{(72)} g ( L ) = 1 − A 2 q ( 1 + ( ( D − A ) 2 + q ) j = 0 ∑ L − 2 ( D 2 + q ) j ) (72)
Proof. Fix independent standard Gaussian random variables Z 0 , … , Z L − 1 Z_0,\ldots,Z_{L-1} Z 0 , … , Z L − 1 . For every a > 0 a>0 a > 0 , construct the process { x ^ t ( a ) , Σ t ( a ) , u t ( a ) } t ≥ 0 \{\hat{x}_t(a),\Sigma_t(a),u_t(a)\}_{t\geq0} { x ^ t ( a ) , Σ t ( a ) , u t ( a ) } t ≥ 0 by x ^ 0 ( a ) = 0 \hat{x}_0(a)=0 x ^ 0 ( a ) = 0 , Σ 0 ( a ) = Σ ˉ \Sigma_0(a)=\bar{\Sigma} Σ 0 ( a ) = Σ ˉ , u 0 ( a ) = a u_0(a)=a u 0 ( a ) = a , and for t ≥ 0 t\geq0 t ≥ 0 ,
x ^ t + 1 ( a ) = A x ^ t ( a ) + B u t ( a ) + A C Σ t ( a ) u t ( a ) σ z 2 + u t ( a ) 2 C 2 Σ t ( a ) Z t , Σ t + 1 ( a ) = A 2 Σ t ( a ) + Σ w − F ( Σ t ( a ) , u t ( a ) ) , u t + 1 ( a ) = K x ^ t + 1 ( a ) . \begin{aligned}
\hat{x}_{t+1}(a)&=A\hat{x}_t(a)+Bu_t(a)+\frac{AC\Sigma_t(a)u_t(a)}{\sqrt{\sigma_z^2+u_t(a)^2C^2\Sigma_t(a)}}Z_t,\\
\Sigma_{t+1}(a)&=A^2\Sigma_t(a)+\Sigma_w-F\left(\Sigma_t(a),u_t(a)\right),\\
u_{t+1}(a)&=K\hat{x}_{t+1}(a).
\end{aligned} x ^ t + 1 ( a ) Σ t + 1 ( a ) u t + 1 ( a ) = A x ^ t ( a ) + B u t ( a ) + σ z 2 + u t ( a ) 2 C 2 Σ t ( a ) A C Σ t ( a ) u t ( a ) Z t , = A 2 Σ t ( a ) + Σ w − F ( Σ t ( a ) , u t ( a ) ) , = K x ^ t + 1 ( a ) .
with the same Gaussian sequence.
By Lemma 9 and the covariance recursion, for each fixed a a a , the constructed process has the same joint distribution as the belief-state and action process under π ˉ θ , a ˉ \bar{\pi}^{\theta,\bar{a}} π ˉ θ , a ˉ , initialized at ( 0 , Σ ˉ ) (0,\bar{\Sigma}) ( 0 , Σ ˉ ) . Consequently,
f L ( a ) = − N a 2 + Γ E [ ∑ t = 0 L − 1 F ( Σ t ( a ) , u t ( a ) ) ] . (73) f_L(a)=-Na^2+\Gamma\mathbb{E}\left[\sum_{t=0}^{L-1}F\left(\Sigma_t(a),u_t(a)\right)\right]. \tag*{(73)} f L ( a ) = − N a 2 + Γ E [ t = 0 ∑ L − 1 F ( Σ t ( a ) , u t ( a ) ) ] . (73)
Since all processes are driven by the same Z Z Z , we may fix a realization of Z Z Z and study pointwise limits as a ↓ 0 a\downarrow0 a ↓ 0 .
For the initial values, we trivially have
x ^ 0 ( a ) = 0 → 0 , Σ 0 ( a ) = Σ ˉ → Σ ˉ , u 0 ( a ) = a → 0 as a ↓ 0. \hat{x}_0(a)=0\to0,\qquad\Sigma_0(a)=\bar{\Sigma}\to\bar{\Sigma},\qquad u_0(a)=a\to0\quad\text{as }a\downarrow0. x ^ 0 ( a ) = 0 → 0 , Σ 0 ( a ) = Σ ˉ → Σ ˉ , u 0 ( a ) = a → 0 as a ↓ 0.
Furthermore
F ( Σ 0 ( a ) , u 0 ( a ) ) = A 2 u 0 ( a ) 2 C 2 Σ 0 ( a ) 2 σ z 2 + u 0 ( a ) 2 C 2 Σ 0 ( a ) → 0 as a ↓ 0. (74) F\left(\Sigma_0(a),u_0(a)\right)=\frac{A^2u_0(a)^2C^2\Sigma_0(a)^2}{\sigma_z^2+u_0(a)^2C^2\Sigma_0(a)}\to0\quad\text{as }a\downarrow0. \tag*{(74)} F ( Σ 0 ( a ) , u 0 ( a ) ) = σ z 2 + u 0 ( a ) 2 C 2 Σ 0 ( a ) A 2 u 0 ( a ) 2 C 2 Σ 0 ( a ) 2 → 0 as a ↓ 0. (74)
Inductively, it is clear that for all t t t ,
x ^ t ( a ) → 0 , Σ t ( a ) → Σ ˉ , u t ( a ) → 0 as a ↓ 0. \hat{x}_t(a)\to0,\qquad\Sigma_t(a)\to\bar{\Sigma},\qquad u_t(a)\to0\quad\text{as }a\downarrow0. x ^ t ( a ) → 0 , Σ t ( a ) → Σ ˉ , u t ( a ) → 0 as a ↓ 0.
Define Y t ( a ) = x ^ t ( a ) / a Y_t(a)=\hat{x}_t(a)/a Y t ( a ) = x ^ t ( a ) / a , so that Y 0 ( a ) = 0 Y_0(a)=0 Y 0 ( a ) = 0 and
Y 1 ( a ) = B + A C Σ ˉ σ z 2 + a 2 C 2 Σ ˉ Z 0 . Y_1(a)=B+\frac{AC\bar{\Sigma}}{\sqrt{\sigma_z^2+a^2C^2\bar{\Sigma}}}Z_0. Y 1 ( a ) = B + σ z 2 + a 2 C 2 Σ ˉ A C Σ ˉ Z 0 .
For t ≥ 1 t\geq1 t ≥ 1 we have u t ( a ) = K x ^ t ( a ) = a K Y t ( a ) u_t(a)=K\hat{x}_t(a)=aKY_t(a) u t ( a ) = K x ^ t ( a ) = a K Y t ( a ) . Dividing the recursion for x ^ t ( a ) \hat{x}_t(a) x ^ t ( a ) by a a a ,
Y t + 1 ( a ) = A Y t ( a ) + B K Y t ( a ) + A C Σ t ( a ) K Y t ( a ) σ z 2 + K 2 x ^ t ( a ) 2 C 2 Σ t ( a ) Z t . (75) Y_{t+1}(a)=AY_t(a)+BKY_t(a)+\frac{AC\Sigma_t(a)KY_t(a)}{\sqrt{\sigma_z^2+K^2\hat{x}_t(a)^2C^2\Sigma_t(a)}}Z_t. \tag*{(75)} Y t + 1 ( a ) = A Y t ( a ) + B K Y t ( a ) + σ z 2 + K 2 x ^ t ( a ) 2 C 2 Σ t ( a ) A C Σ t ( a ) K Y t ( a ) Z t . (75)
We define the limiting process by
Y 1 = B + A C Σ ˉ σ z Z 0 , Y t + 1 = ( D + A C K Σ ˉ σ z Z t ) Y t ( t ≥ 1 ) . Y_1=B+\frac{AC\bar{\Sigma}}{\sigma_z}Z_0,\qquad Y_{t+1}=\left(D+\frac{ACK\bar{\Sigma}}{\sigma_z}Z_t\right)Y_t\quad(t\geq1). Y 1 = B + σ z A C Σ ˉ Z 0 , Y t + 1 = ( D + σ z A C K Σ ˉ Z t ) Y t ( t ≥ 1 ) .
Inductively, it is clear to see that Y t ( a ) → Y t Y_t(a)\to Y_t Y t ( a ) → Y t as a ↓ 0 a\downarrow0 a ↓ 0 for all t ≥ 1 t\geq1 t ≥ 1 .
For each t ≥ 1 t\geq1 t ≥ 1 , Y t Y_t Y t is a function of Z 0 , … , Z t − 1 Z_0,\ldots,Z_{t-1} Z 0 , … , Z t − 1 , so it is independent of Z t Z_t Z t . Therefore, we get
E [ Y 1 2 ] = B 2 + A 2 C 2 Σ ˉ 2 σ z 2 , E [ Y t + 1 2 ] = ( D 2 + q ) E [ Y t 2 ] ( t ≥ 1 ) \mathbb{E}[Y_1^2]=B^2+\frac{A^2C^2\bar{\Sigma}^2}{\sigma_z^2},\qquad\mathbb{E}[Y_{t+1}^2]=(D^2+q)\mathbb{E}[Y_t^2]\quad(t\geq1) E [ Y 1 2 ] = B 2 + σ z 2 A 2 C 2 Σ ˉ 2 , E [ Y t + 1 2 ] = ( D 2 + q ) E [ Y t 2 ] ( t ≥ 1 )
hence
E [ Y t 2 ] = ( B 2 + A 2 C 2 Σ ˉ 2 σ z 2 ) ( D 2 + q ) t − 1 ( t ≥ 1 ) . \mathbb{E}[Y_t^2]=\left(B^2+\frac{A^2C^2\bar{\Sigma}^2}{\sigma_z^2}\right)(D^2+q)^{t-1}\quad(t\geq1). E [ Y t 2 ] = ( B 2 + σ z 2 A 2 C 2 Σ ˉ 2 ) ( D 2 + q ) t − 1 ( t ≥ 1 ) .
Observe that for t ≥ 1 t \ge1 t ≥ 1
F ( Σ t ( a ) , u t ( a ) ) a 2 = A 2 K 2 C 2 Σ t ( a ) 2 σ z 2 + K 2 x ^ t ( a ) 2 C 2 Σ t ( a ) Y t ( a ) 2 . \frac{F\left(\Sigma_t(a),u_t(a)\right)}{a^2}=\frac{A^2K^2C^2\Sigma_t(a)^2}{\sigma_z^2+K^2\hat{x}_t(a)^2C^2\Sigma_t(a)}Y_t(a)^2. a 2 F ( Σ t ( a ) , u t ( a ) ) = σ z 2 + K 2 x ^ t ( a ) 2 C 2 Σ t ( a ) A 2 K 2 C 2 Σ t ( a ) 2 Y t ( a ) 2 .
which implies that
lim a → 0 F ( Σ t ( a ) , u t ( a ) ) a 2 = A 2 K 2 C 2 Σ ‾ 2 σ z 2 Y t 2 = q Y t 2 . \lim_{a\to0}\frac{F\left(\Sigma_t(a),u_t(a)\right)}{a^2}=\frac{A^2K^2C^2\overline{\Sigma}^2}{\sigma_z^2}Y_t^2=qY_t^2. a → 0 lim a 2 F ( Σ t ( a ) , u t ( a ) ) = σ z 2 A 2 K 2 C 2 Σ 2 Y t 2 = q Y t 2 .
By Fatou’s Lemma, for t ≥ 1 t \ge1 t ≥ 1 , we have
lim inf a → 0 E [ F ( Σ t ( a ) , u t ( a ) ) a 2 ] ≥ E [ lim inf a → 0 F ( Σ t ( a ) , u t ( a ) ) a 2 ] = q E [ Y t 2 ] = q ( B 2 + A 2 C 2 Σ ‾ 2 σ z 2 ) ( D 2 + q ) t − 1 . \liminf_{a\to0}\mathbb{E}\left[\frac{F\left(\Sigma_t(a),u_t(a)\right)}{a^2}\right]\ge\mathbb{E}\left[\liminf_{a\to0}\frac{F\left(\Sigma_t(a),u_t(a)\right)}{a^2}\right]=q\mathbb{E}\left[Y_t^2\right]=q\left(B^2+\frac{A^2C^2\overline{\Sigma}^2}{\sigma_z^2}\right)(D^2+q)^{t-1}. a → 0 lim inf E [ a 2 F ( Σ t ( a ) , u t ( a ) ) ] ≥ E [ a → 0 lim inf a 2 F ( Σ t ( a ) , u t ( a ) ) ] = q E [ Y t 2 ] = q ( B 2 + σ z 2 A 2 C 2 Σ 2 ) ( D 2 + q ) t − 1 .
while for t = 0 t=0 t = 0 , we get
lim a → 0 F ( Σ 0 ( a ) , u 0 ( a ) ) a 2 = A 2 C 2 Σ ‾ 2 σ z 2 . \lim_{a\to0}\frac{F\left(\Sigma_0(a),u_0(a)\right)}{a^2}=\frac{A^2C^2\overline{\Sigma}^2}{\sigma_z^2}. a → 0 lim a 2 F ( Σ 0 ( a ) , u 0 ( a ) ) = σ z 2 A 2 C 2 Σ 2 .
Therefore,
lim inf a → 0 f L ( a ) a 2 ≥ − N + Γ ∑ t = 0 L − 1 lim inf a → 0 E [ F ( Σ t ( a ) , u t ( a ) ) a 2 ] ≥ − N + Γ A 2 C 2 Σ ‾ 2 σ z 2 + Γ ∑ t = 1 L − 1 q ( B 2 + A 2 C 2 Σ ‾ 2 σ z 2 ) ( D 2 + q ) t − 1 = − N + N K 2 1 − A 2 A 2 C 2 Σ ‾ 2 σ z 2 + N K 2 1 − A 2 q ( B 2 + A 2 C 2 Σ ‾ 2 σ z 2 ) ∑ t = 0 L − 2 ( D 2 + q ) t = N ( − 1 + q 1 − A 2 + q 1 − A 2 ( ( D − A ) 2 + q ) ∑ t = 0 L − 2 ( D 2 + q ) t ) = N ( g ( L ) − 1 ) \begin{aligned}
\liminf_{a\to0}\frac{f_L(a)}{a^2}\ge-N+\Gamma\sum_{t=0}^{L-1}\liminf_{a\to0}\mathbb{E}\left[\frac{F\left(\Sigma_t(a),u_t(a)\right)}{a^2}\right] \\
&\ge-N+\Gamma\frac{A^2C^2\overline{\Sigma}^2}{\sigma_z^2}+\Gamma\sum_{t=1}^{L-1}q\left(B^2+\frac{A^2C^2\overline{\Sigma}^2}{\sigma_z^2}\right)(D^2+q)^{t-1} \\
&= -N+\frac{NK^2}{1-A^2}\frac{A^2C^2\overline{\Sigma}^2}{\sigma_z^2}+\frac{NK^2}{1-A^2}q\left(B^2+\frac{A^2C^2\overline{\Sigma}^2}{\sigma_z^2}\right)\sum_{t=0}^{L-2}(D^2+q)^t \\
&=N\left(-1+\frac{q}{1-A^2}+\frac{q}{1-A^2}\left((D-A)^2+q\right)\sum_{t=0}^{L-2}(D^2+q)^t\right) \\
&=N(g(L)-1)
\end{aligned} a → 0 lim inf a 2 f L ( a ) ≥ − N + Γ t = 0 ∑ L − 1 a → 0 lim inf E [ a 2 F ( Σ t ( a ) , u t ( a ) ) ] ≥ − N + Γ σ z 2 A 2 C 2 Σ 2 + Γ t = 1 ∑ L − 1 q ( B 2 + σ z 2 A 2 C 2 Σ 2 ) ( D 2 + q ) t − 1 = − N + 1 − A 2 N K 2 σ z 2 A 2 C 2 Σ 2 + 1 − A 2 N K 2 q ( B 2 + σ z 2 A 2 C 2 Σ 2 ) t = 0 ∑ L − 2 ( D 2 + q ) t = N ( − 1 + 1 − A 2 q + 1 − A 2 q ( ( D − A ) 2 + q ) t = 0 ∑ L − 2 ( D 2 + q ) t ) = N ( g ( L ) − 1 )
which completes the proof.
Lemma 23. If q > q 2 q>q_2 q > q 2 , then there exists finite L L L such that g ( L ) > 1 g(L)>1 g ( L ) > 1 .
Proof. In the case when D 2 + q ≥ 1 D^2+q\ge1 D 2 + q ≥ 1 we have that g ( L ) → ∞ g(L)\to\infty g ( L ) → ∞ as L → ∞ L\to\infty L → ∞ and the claim follows trivially. When D 2 + q < 1 D^2+q<1 D 2 + q < 1 we have
g ( L ) → q 1 − A 2 ( 1 + ( D − A ) 2 + q 1 − D 2 − q ) as L → ∞ . g(L)\to\frac{q}{1-A^2}\left(1+\frac{(D-A)^2+q}{1-D^2-q}\right)\quad\text{as }L\to\infty. g ( L ) → 1 − A 2 q ( 1 + 1 − D 2 − q ( D − A ) 2 + q ) as L → ∞. Observe that
q 1 − A 2 ( 1 + ( D − A ) 2 + q 1 − D 2 − q ) > 1 ⟺ q > ( 1 − A 2 ) ( 1 − D 2 ) 2 ( 1 − A D ) = q 2 . (76) \frac{q}{1-A^2}\left(1+\frac{(D-A)^2+q}{1-D^2-q}\right)>1\quad\Longleftrightarrow\quad q>\frac{(1-A^2)(1-D^2)}{2(1-AD)}=q_2. \tag*{(76)} 1 − A 2 q ( 1 + 1 − D 2 − q ( D − A ) 2 + q ) > 1 ⟺ q > 2 ( 1 − A D ) ( 1 − A 2 ) ( 1 − D 2 ) = q 2 . (76) Therefore, if q > q 2 q>q_2 q > q 2 , (G.20) shows that the limit of g ( L ) g(L) g ( L ) exceeds 1 1 1 , so by the definition of limit, there exists a finite L L L such that g ( L ) > 1 g(L)>1 g ( L ) > 1 .
Lemma 24. If q > q 2 q>q_2 q > q 2 there exist finite L , a , c 0 > 0 L,a,c_0>0 L , a , c 0 > 0 such that G L , 1 ( a ; 0 , Σ ‾ , θ ˉ ) ≥ c 0 ⋅ a 2 G_{L,1}(a;0,\overline{\Sigma},\bar{\theta})\ge c_0\cdot a^2 G L , 1 ( a ; 0 , Σ , θ ˉ ) ≥ c 0 ⋅ a 2 .
Proof. If q > q 2 q>q_2 q > q 2 , there exists a finite L 0 L_0 L 0 such that N ( g ( L 0 ) − 1 ) > 0 N(g(L_0)-1)>0 N ( g ( L 0 ) − 1 ) > 0 by Lemma 23. Next, by Lemma 22 , there exists a 0 a_0 a 0 such that if 0 < a < a 0 0<a<a_0 0 < a < a 0 then
f L 0 ( a ) a 2 ≥ 1 2 N ( g ( L 0 ) − 1 ) . (77) \frac{f_{L_0}(a)}{a^2}\ge\frac{1}{2}N(g(L_0)-1). \tag*{(77)} a 2 f L 0 ( a ) ≥ 2 1 N ( g ( L 0 ) − 1 ) . (77)
by the definition of limit inferior. Fix a ∈ ( 0 , a 0 ) a\in(0,a_0) a ∈ ( 0 , a 0 ) . It follows that
G L 0 , 1 ( a ; 0 , Σ ˉ , θ ˉ ) = f L 0 ( a ) ≥ a 2 1 2 N ( g ( L 0 ) − 1 ) > 0 (78) \mathcal{G}_{L_0,1}(a;0,\bar{\Sigma},\bar{\theta})=f_{L_0}(a)\ge a^2\frac{1}{2}N(g(L_0)-1)>0 \tag*{(78)} G L 0 , 1 ( a ; 0 , Σ ˉ , θ ˉ ) = f L 0 ( a ) ≥ a 2 2 1 N ( g ( L 0 ) − 1 ) > 0 (78)
Choosing c 0 = 1 2 N ( g ( L 0 ) − 1 ) c_0=\frac{1}{2}N(g(L_0)-1) c 0 = 2 1 N ( g ( L 0 ) − 1 ) completes the proof.
Lemma 25. Fix L ∈ Z > 0 L\in\mathbb{Z}_{>0} L ∈ Z > 0 , a > 0 a>0 a > 0 and λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] . Then, the mapping
( x ^ , Σ , θ ) ↦ G L , λ ( a ; x ^ , Σ , θ ) (\hat{x},\Sigma,\theta)\mapsto\mathcal{G}_{L,\lambda}(a;\hat{x},\Sigma,\theta) ( x ^ , Σ , θ ) ↦ G L , λ ( a ; x ^ , Σ , θ ) is continuous on R × [ Σ w , Σ ˉ ] × [ 0 , ∞ ) 2 L \mathbb{R}\times[\Sigma_w,\bar{\Sigma}]\times[0,\infty)^{2L} R × [ Σ w , Σ ˉ ] × [ 0 , ∞ ) 2 L .
Proof. For Z = ( Z 0 , … , Z L − 1 ) ∈ R L Z=(Z_0,\ldots,Z_{L-1})\in\mathbb{R}^L Z = ( Z 0 , … , Z L − 1 ) ∈ R L , define the trajectory recursively by
x ^ 0 = x ^ , Σ 0 = Σ , u 0 = k 0 x ^ 0 + a \hat{x}_0=\hat{x},\qquad\Sigma_0=\Sigma,\qquad u_0=k_0\hat{x}_0+a x ^ 0 = x ^ , Σ 0 = Σ , u 0 = k 0 x ^ 0 + a
and, for 0 ≤ t ≤ L − 2 0\le t\le L-2 0 ≤ t ≤ L − 2 ,
x ^ t + 1 = A x ^ t + B u t + A Σ t C u t σ z 2 + C 2 Σ t u t 2 Z t Σ t + 1 = A 2 Σ t + Σ w − F ( Σ t , u t ) u t + 1 = k t + 1 x ^ t + 1 . (79) \begin{aligned}
\hat{x}_{t+1}&=A\hat{x}_t+Bu_t+\frac{A\Sigma_tCu_t}{\sqrt{\sigma_z^2+C^2\Sigma_tu_t^2}}Z_t\\
\Sigma_{t+1}&=A^2\Sigma_t+\Sigma_w-F(\Sigma_t,u_t)\\
u_{t+1}&=k_{t+1}\hat{x}_{t+1}. \tag*{(79)}
\end{aligned} x ^ t + 1 Σ t + 1 u t + 1 = A x ^ t + B u t + σ z 2 + C 2 Σ t u t 2 A Σ t C u t Z t = A 2 Σ t + Σ w − F ( Σ t , u t ) = k t + 1 x ^ t + 1 . (79)
Using the trajectory, define
Φ L ( x ^ , Σ , θ , Z ) = − n 0 a 2 + ∑ t = 0 L − 1 [ λ γ t + 1 F ( Σ t , u t ) + ( 1 − λ ) n t k t 2 ( Σ ˉ − Σ t ) ] . \Phi_L(\hat{x},\Sigma,\theta,Z)=-n_0a^2+\sum_{t=0}^{L-1}\left[\lambda\gamma_{t+1}F(\Sigma_t,u_t)+(1-\lambda)n_tk_t^2(\bar{\Sigma}-\Sigma_t)\right]. Φ L ( x ^ , Σ , θ , Z ) = − n 0 a 2 + t = 0 ∑ L − 1 [ λ γ t + 1 F ( Σ t , u t ) + ( 1 − λ ) n t k t 2 ( Σ ˉ − Σ t ) ] .
then this equals
∑ t = 0 L − 1 r t , λ θ ( x ^ t , Σ t , u t ) . \sum_{t=0}^{L-1}r_{t,\lambda}^{\theta}(\hat{x}_t,\Sigma_t,u_t). t = 0 ∑ L − 1 r t , λ θ ( x ^ t , Σ t , u t ) .
By Lemma 9 , when Z ∼ N ( 0 , I L ) Z\sim\mathcal{N}(0,I_L) Z ∼ N ( 0 , I L ) , the constructed trajectory above has the same joint distribution as the belief-state and action process under π θ , a \pi^{\theta,a} π θ , a , initialized at ( x ^ , Σ ) (\hat{x},\Sigma) ( x ^ , Σ ) . Hence,
G L , λ ( a ; x ^ , Σ , θ ) = E Z ∼ N ( 0 , I L ) [ Φ L ( x ^ , Σ , θ , Z ) ] . \mathcal{G}_{L,\lambda}(a;\hat{x},\Sigma,\theta)=\mathbb{E}_{Z\sim\mathcal{N}(0,I_L)}\left[\Phi_L(\hat{x},\Sigma,\theta,Z)\right]. G L , λ ( a ; x ^ , Σ , θ ) = E Z ∼ N ( 0 , I L ) [ Φ L ( x ^ , Σ , θ , Z ) ] .
Note that, for every t t t , the covariance gives
Σ w ≤ Σ t ≤ Σ ˉ , and 0 ≤ F ( Σ t , u t ) ≤ A 2 Σ ˉ . \Sigma_w\le\Sigma_t\le\bar{\Sigma},\qquad\text{and}\qquad0\le F(\Sigma_t,u_t)\le A^2\bar{\Sigma}. Σ w ≤ Σ t ≤ Σ ˉ , and 0 ≤ F ( Σ t , u t ) ≤ A 2 Σ ˉ .
Let ( x ^ n , Σ m , θ m ) → ( x ^ , Σ , θ ) (\hat{x}_n,\Sigma_m,\theta_m)\to(\hat{x},\Sigma,\theta) ( x ^ n , Σ m , θ m ) → ( x ^ , Σ , θ ) be an arbitrary convergent sequence in R × [ Σ w , Σ ˉ ] × [ 0 , ∞ ) 2 L \mathbb{R}\times[\Sigma_w,\bar{\Sigma}]\times[0,\infty)^{2L} R × [ Σ w , Σ ˉ ] × [ 0 , ∞ ) 2 L . Since { θ m } m = 1 ∞ \{\theta_m\}_{m=1}^{\infty} { θ m } m = 1 ∞ converges, its coordinates are uniformly bounded. For each parameter vector θ m \theta_m θ m , the corresponding coefficients satisfy
n t = R + B 2 p t + 1 ≥ R > 0 , and k t = − B p t + 1 A + S n t . n_t=R+B^2p_{t+1}\ge R>0,\qquad\text{and}\qquad k_t=-\frac{Bp_{t+1}A+S}{n_t}. n t = R + B 2 p t + 1 ≥ R > 0 , and k t = − n t B p t + 1 A + S .
Hence, there exists M < ∞ M < \infty M < ∞ such that
max { ∣ n t ∣ , ∣ k t ∣ , ∣ γ t + 1 ∣ } ≤ M \max\{|n_t|, |k_t|, |\gamma_{t+1}|\} \le M max { ∣ n t ∣ , ∣ k t ∣ , ∣ γ t + 1 ∣ } ≤ M
uniformly over all m m m and all 0 ≤ t < L 0 \le t < L 0 ≤ t < L . Consequently,
∣ Φ L ( x ^ m , Σ m , θ m , Z ) ∣ ≤ M a 2 + L ( M A 2 Σ ˉ + M 3 Σ ˉ ) (80) |\Phi_L(\hat{x}_m, \Sigma_m, \theta_m, Z)| \le Ma^2 + L\left(MA^2\bar{\Sigma} + M^3\bar{\Sigma}\right) \tag*{(80)} ∣ Φ L ( x ^ m , Σ m , θ m , Z ) ∣ ≤ M a 2 + L ( M A 2 Σ ˉ + M 3 Σ ˉ ) (80)
independently of m m m and Z Z Z .
Next, we show pointwise convergence for each fixed Z Z Z . By the definitions of n t n_t n t and k t k_t k t , together with n t ≥ R > 0 n_t \ge R > 0 n t ≥ R > 0 , these coefficients are continuous functions of θ \theta θ . This also implies that the map ( Σ , u ) ↦ F ( Σ , u ) (\Sigma,u) \mapsto F(\Sigma,u) ( Σ , u ) ↦ F ( Σ , u ) is continuous. Moreover, σ z 2 + C 2 Σ t u t 2 ≥ σ z 2 > 0 \sigma_z^2+C^2\Sigma_tu_t^2 \ge\sigma_z^2 > 0 σ z 2 + C 2 Σ t u t 2 ≥ σ z 2 > 0 . Hence the updates in (G.23) are continuous in ( x ^ t , Σ t , u t , θ ) (\hat{x}_t,\Sigma_t,u_t,\theta) ( x ^ t , Σ t , u t , θ ) for each fixed Z Z Z , and so is the initial action u 0 = k 0 x ^ 0 + a u_0=k_0\hat{x}_0+a u 0 = k 0 x ^ 0 + a . By induction on t t t , each of x ^ t \hat{x}_t x ^ t , Σ t \Sigma_t Σ t and u t u_t u t is a continuous function of ( x ^ , Σ , θ ) (\hat{x},\Sigma,\theta) ( x ^ , Σ , θ ) for each fixed Z Z Z . Since Φ L \Phi_L Φ L is a finite sum of continuous functions of these quantities and of θ \theta θ ,
Φ L ( x ^ m , Σ m , θ m , Z ) → Φ L ( x ^ , Σ , θ , Z ) as m → ∞ for every Z ∈ R L . (81) \Phi_L(\hat{x}_m,\Sigma_m,\theta_m,Z) \to\Phi_L(\hat{x},\Sigma,\theta,Z)
\quad\text{as }m\to\infty\text{ for every }Z\in\mathbb{R}^L. \tag*{(81)} Φ L ( x ^ m , Σ m , θ m , Z ) → Φ L ( x ^ , Σ , θ , Z ) as m → ∞ for every Z ∈ R L . (81)
The right-hand side of (G.24) is a constant, so it is integrable with respect to N ( 0 , I L ) \mathcal{N}(0,I_L) N ( 0 , I L ) . By the dominated convergence theorem,
lim m → ∞ G L , λ ( a ; x ^ m , Σ m , θ m ) = lim m → ∞ E Z ∼ N ( 0 , I L ) [ Φ L ( x ^ m , Σ m , θ m , Z ) ] = E Z ∼ N ( 0 , I L ) [ lim m → ∞ Φ L ( x ^ m , Σ m , θ m , Z ) ] = E Z ∼ N ( 0 , I L ) [ Φ L ( x ^ , Σ , θ , Z ) ] = G L , λ ( a ; x ^ , Σ , θ ) . \begin{aligned}
\lim_{m\to\infty}G_{L,\lambda}(a;\hat{x}_m,\Sigma_m,\theta_m)
&=\lim_{m\to\infty}\mathbb{E}_{Z\sim\mathcal{N}(0,I_L)}\left[\Phi_L(\hat{x}_m,\Sigma_m,\theta_m,Z)\right]\\
&=\mathbb{E}_{Z\sim\mathcal{N}(0,I_L)}\left[\lim_{m\to\infty}\Phi_L(\hat{x}_m,\Sigma_m,\theta_m,Z)\right]\\
&=\mathbb{E}_{Z\sim\mathcal{N}(0,I_L)}\left[\Phi_L(\hat{x},\Sigma,\theta,Z)\right]\\
&=G_{L,\lambda}(a;\hat{x},\Sigma,\theta).
\end{aligned} m → ∞ lim G L , λ ( a ; x ^ m , Σ m , θ m ) = m → ∞ lim E Z ∼ N ( 0 , I L ) [ Φ L ( x ^ m , Σ m , θ m , Z ) ] = E Z ∼ N ( 0 , I L ) [ m → ∞ lim Φ L ( x ^ m , Σ m , θ m , Z ) ] = E Z ∼ N ( 0 , I L ) [ Φ L ( x ^ , Σ , θ , Z ) ] = G L , λ ( a ; x ^ , Σ , θ ) .
Since the convergent sequence is chosen arbitrarily, the mapping ( x ^ , Σ , θ ) ↦ G L , λ ( a ; x ^ , Σ , θ ) (\hat{x},\Sigma,\theta)\mapsto G_{L,\lambda}(a;\hat{x},\Sigma,\theta) ( x ^ , Σ , θ ) ↦ G L , λ ( a ; x ^ , Σ , θ ) is continuous on R × [ Σ ‾ w , Σ ˉ ] × [ 0 , ∞ ) 2 L \mathbb{R}\times[\underline{\Sigma}_w,\bar{\Sigma}]\times[0,\infty)^{2L} R × [ Σ w , Σ ˉ ] × [ 0 , ∞ ) 2 L .
Lemma 26. If q > q 2 q>q_2 q > q 2 , there exist integers L ≥ 1 L\ge1 L ≥ 1 , t 6 ≥ 0 t_6\ge0 t 6 ≥ 0 , constants δ , a , c 1 > 0 \delta,a,c_1>0 δ , a , c 1 > 0 , and λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) , all independent of T T T , such that the probing policy π P \pi^P π P defined with parameters δ , a , L , t 6 \delta,a,L,t_6 δ , a , L , t 6 satisfies
E π P [ ∑ t = 0 L − 1 r τ k + t , λ ( x ^ τ k + t , Σ τ k + t , u τ k + t ) ∣ H τ k ] ≥ c 1 ⋅ a 2 (82) \mathbb{E}_{\pi^P}\left[\left.\sum_{t=0}^{L-1}r_{\tau_k+t,\lambda}(\hat{x}_{\tau_k+t},\Sigma_{\tau_k+t},u_{\tau_k+t})\right|H_{\tau_k}\right]\ge c_1\cdot a^2 \tag*{(82)} E π P [ t = 0 ∑ L − 1 r τ k + t , λ ( x ^ τ k + t , Σ τ k + t , u τ k + t ) H τ k ] ≥ c 1 ⋅ a 2 (82) on τ k < ∞ \tau_k<\infty τ k < ∞ , for every k ≥ 1 k\ge1 k ≥ 1 . Here, the probing times use the cutoff T − t 6 − L T-t_6-L T − t 6 − L and M δ \mathcal{M}_\delta M δ .
Proof. For any L L L and λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) , observe that
G L , λ ( a ; x ^ , Σ , θ ) = λ G L , 1 ( a ; x ^ , Σ , θ ) + ( 1 − λ ) G L , 0 ( a ; x ^ , Σ , θ ) G_{L,\lambda}(a;\hat{x},\Sigma,\theta)=\lambda G_{L,1}(a;\hat{x},\Sigma,\theta)+(1-\lambda)G_{L,0}(a;\hat{x},\Sigma,\theta) G L , λ ( a ; x ^ , Σ , θ ) = λ G L , 1 ( a ; x ^ , Σ , θ ) + ( 1 − λ ) G L , 0 ( a ; x ^ , Σ , θ )
for all ( x ^ , Σ , θ ) (\hat{x},\Sigma,\theta) ( x ^ , Σ , θ ) . Also, note that G L , 0 ( a ; 0 , Σ ˉ , θ ˉ ) ≥ − N a 2 G_{L,0}(a;0,\bar{\Sigma},\bar{\theta})\ge-Na^2 G L , 0 ( a ; 0 , Σ ˉ , θ ˉ ) ≥ − N a 2 . Choose L L L and c 0 > 0 c_0>0 c 0 > 0 such that G L , 1 ( a ; 0 , Σ ˉ , θ ˉ ) ≥ c 0 a 2 G_{L,1}(a;0,\bar{\Sigma},\bar{\theta})\ge c_0a^2 G L , 1 ( a ; 0 , Σ ˉ , θ ˉ ) ≥ c 0 a 2 . This is guaranteed by Lemma 24. For such L L L and c 0 > 0 c_0>0 c 0 > 0 , we get
G L , λ ( a ; 0 , Σ ˉ , θ ˉ ) ≥ λ c 0 a 2 − ( 1 − λ ) N a 2 (83) G_{L,\lambda}(a;0,\bar{\Sigma},\bar{\theta})\ge\lambda c_0a^2-(1-\lambda)Na^2 \tag*{(83)} G L , λ ( a ; 0 , Σ ˉ , θ ˉ ) ≥ λ c 0 a 2 − ( 1 − λ ) N a 2 (83)
for any λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) . Choose λ ∈ ( 0 , 1 ) \lambda\in(0,1) λ ∈ ( 0 , 1 ) such that N c 0 + N < λ < 1 \frac{N}{c_0+N}<\lambda<1 c 0 + N N < λ < 1 so that c 1 : = 1 2 ( λ c 0 − ( 1 − λ ) N ) > 0 c_1:=\frac{1}{2}\left(\lambda c_0-(1-\lambda)N\right)>0 c 1 := 2 1 ( λ c 0 − ( 1 − λ ) N ) > 0 . Continuity from Lemma 25 implies that there exists ε \varepsilon ε such that
G L , λ ( a ; x ^ , Σ , θ ) ≥ c 1 a 2 whenever x ^ 2 + ∣ Σ − Σ ˉ ∣ + ∥ θ − θ ˉ ∥ ≤ ε . G_{L,\lambda}(a;\hat{x},\Sigma,\theta)\ge c_1a^2
\quad\text{whenever }\hat{x}^2+|\Sigma-\bar{\Sigma}|+\|\theta-\bar{\theta}\|\le\varepsilon. G L , λ ( a ; x ^ , Σ , θ ) ≥ c 1 a 2 whenever x ^ 2 + ∣Σ − Σ ˉ ∣ + ∥ θ − θ ˉ ∥ ≤ ε .
Let θ s = ( P s + 1 , … , P s + L , Γ s + 1 , … , Γ s + L ) \theta_s = (P_{s+1}, \ldots, P_{s+L}, \Gamma_{s+1}, \ldots, \Gamma_{s+L}) θ s = ( P s + 1 , … , P s + L , Γ s + 1 , … , Γ s + L ) . By Lemmas 5 and 6, we can choose t 6 t_6 t 6 , independent of T T T , such that ∥ θ s − θ ˉ ∥ ≤ ε / 2 \lVert\theta_s-\bar{\theta}\rVert\le\varepsilon/2 ∥ θ s − θ ˉ ∥ ≤ ε /2 for all s ≤ T − t 6 − L s \le T-t_6-L s ≤ T − t 6 − L . Choose δ > 0 \delta>0 δ > 0 such that 2 δ ≤ ε / 2 2\delta\le\varepsilon/2 2 δ ≤ ε /2 . For every ( x ^ , Σ ) ∈ M δ (\hat{x},\Sigma)\in\mathcal{M}_{\delta} ( x ^ , Σ ) ∈ M δ ,
x ^ 2 + ∣ Σ − Σ ˉ ∣ + ∥ θ s − θ ˉ ∥ ≤ δ + δ + ε 2 ≤ ε . \hat{x}^{2}+\lvert\Sigma-\bar{\Sigma}\rvert+\lVert\theta_s-\bar{\theta}\rVert\le\delta+\delta+\frac{\varepsilon}{2}\le\varepsilon. x ^ 2 + ∣ Σ − Σ ˉ ∣ + ∥ θ s − θ ˉ ∥ ≤ δ + δ + 2 ε ≤ ε .
Therefore,
G L , λ ( a ; x ^ , Σ , θ s ) ≥ c 1 a 2 (84) G_{L,\lambda}(a;\hat{x},\Sigma,\theta_s)\ge c_1a^2 \tag*{(84)} G L , λ ( a ; x ^ , Σ , θ s ) ≥ c 1 a 2 (84)
for all s ≤ T − t 6 − L s\le T-t_6-L s ≤ T − t 6 − L and ( x ^ , Σ ) ∈ M δ (\hat{x},\Sigma)\in\mathcal{M}_{\delta} ( x ^ , Σ ) ∈ M δ . Take the cutoff parameter in the definition of π P \pi^{P} π P to be t 6 t_6 t 6 . Fix k ≥ 1 k\ge1 k ≥ 1 and consider the event { τ k < ∞ } \{\tau_k<\infty\} { τ k < ∞ } . For each deterministic starting time s s s , the corresponding local coefficients satisfy
n t = N s + t , k t = K s + t , γ t + 1 = Γ s + t + 1 , 0 ≤ t < L . n_t=N_{s+t},\qquad k_t=K_{s+t},\qquad\gamma_{t+1}=\Gamma_{s+t+1},\qquad0\le t<L. n t = N s + t , k t = K s + t , γ t + 1 = Γ s + t + 1 , 0 ≤ t < L .
In particular,
r t , λ θ s ( x ^ , Σ , u ) = r s + t , λ ( x ^ , Σ , u ) . r^{\theta_s}_{t,\lambda}(\hat{x},\Sigma,u)=r_{s+t,\lambda}(\hat{x},\Sigma,u). r t , λ θ s ( x ^ , Σ , u ) = r s + t , λ ( x ^ , Σ , u ) .
On the event { τ k = s } \{\tau_k=s\} { τ k = s } , the actions within the block therefore coincide with those of π θ s , a \pi^{\theta_s,a} π θ s , a , initialized at ( x ^ s , Σ s ) (\hat{x}_s,\Sigma_s) ( x ^ s , Σ s ) . Hence, on each event { τ k = s } \{\tau_k=s\} { τ k = s } ,
E π P [ ∑ t = 0 L − 1 r τ k + t , λ ( x ^ τ k + t , Σ τ k + t , u τ k + t ) ∣ F τ k ] = G L , λ ( a ; x ^ τ k , Σ τ k , θ τ k ) ≥ c 1 a 2 \mathbb{E}_{\pi^P}\left[\left.\sum_{t=0}^{L-1}r_{\tau_k+t,\lambda}(\hat{x}_{\tau_k+t},\Sigma_{\tau_k+t},u_{\tau_k+t})\right|\mathcal{F}_{\tau_k}\right]
=G_{L,\lambda}(a;\hat{x}_{\tau_k},\Sigma_{\tau_k},\theta_{\tau_k})\ge c_1a^2 E π P [ t = 0 ∑ L − 1 r τ k + t , λ ( x ^ τ k + t , Σ τ k + t , u τ k + t ) F τ k ] = G L , λ ( a ; x ^ τ k , Σ τ k , θ τ k ) ≥ c 1 a 2
since τ k ≤ T − t 6 − L \tau_k\le T-t_6-L τ k ≤ T − t 6 − L and ( x ^ τ k , Σ τ k ) ∈ M δ (\hat{x}_{\tau_k},\Sigma_{\tau_k})\in\mathcal{M}_{\delta} ( x ^ τ k , Σ τ k ) ∈ M δ . This completes the proof.
Numerical Simulations
Regime visualization
In this section we numerically visualize the key regimes characterized by our results. We fix the scalar instance
A = 0.8 B = C = Q = R = Σ w = μ x = 1 S = 0 (85) A = 0.8 \quad B = C = Q = R = \Sigma_w = \mu_x = 1 \quad S = 0 \tag*{(85)} A = 0.8 B = C = Q = R = Σ w = μ x = 1 S = 0 (85)
and vary q q q by varying σ z 2 \sigma_z^2 σ z 2 . We use the fixed, heuristically chosen probing parameters
a = 0.25 δ = 0.03 L = 5 t P = 20 (86) a = 0.25 \quad\delta= 0.03 \quad L = 5 \quad t_P = 20 \tag*{(86)} a = 0.25 δ = 0.03 L = 5 t P = 20 (86)
throughout all experiments. Figure 1 sweeps over different values of q q q at T = 800 T = 800 T = 800 . The result exhibits behavior predicted by our theory. For small q q q , neither CE nor probing significantly improves over OL; for intermediate values of q q q , probing improves linearly over open loop, while the advantage of CE remains bounded; for large values of q q q both probing and CE achieve linear advantage over open-loop. Figure 7 complements this by visualizing cumulative advantage as T T T varies for fixed values of q q q . For q = 0.5 q = 0.5 q = 0.5 , which lies between q 2 q_2 q 2 and q 3 q_3 q 3 , the advantage of CE over open-loop remains bounded, while the advantage of probing grows linearly. For q = 4 q = 4 q = 4 , which lies above q 4 q_4 q 4 , both the advantage of CE and probing grow linearly. All experiments use 2000 independent Monte Carlo trajectories per policy. Error bands denote 95% normal confidence intervals for the trajectory-level mean.
Figure 7. Numerical illustration of CE and probing behavior in two different regimes. We plot the cumulative regret R T ( π , π O L ) R_T(\pi,\pi^{OL}) R T ( π , π O L ) over the open-loop policy as a function of the horizon T T T . In the intermediate q regime q 2 < q = 0.50 < q 3 q_2 < q = 0.50 < q_3 q 2 < q = 0.50 < q 3 (left), the CE regret remains bounded while the probing policy accumulates regret linearly. In the high q regime q = 4.00 > q 4 q = 4.00 > q_4 q = 4.00 > q 4 (right), CE itself accumulates linear regret. The same probing-policy parameters are used in both panels.
Multivariate dynamical system
We next investigate whether this phenomenon persists in a multivariate system. We consider a dynamical system with p = n = 3 p=n=3 p = n = 3 , with:
A = [ 0.8 0.1 0 0 0.75 0.1 0.05 0 0.7 ] B = C = Q = R = Σ w = I 3 S = 0 μ x = 1 3 . (87) A=\begin{bmatrix} 0.8 & 0.1 & 0 \\ 0 & 0.75 & 0.1 \\ 0.05 & 0 & 0.7 \end{bmatrix}
\qquad B=C=Q=R=\Sigma_w=I_3 \qquad S=0 \qquad\mu_x=1_3. \tag*{(87)} A = 0.8 0 0.05 0.1 0.75 0 0 0.1 0.7 B = C = Q = R = Σ w = I 3 S = 0 μ x = 1 3 . (87)
We use the multivariate analog of q q q :
q = ∥ A Σ ˉ C T K ∥ 2 2 σ z 2 (88) q=\frac{\lVert A\bar{\Sigma}C^{\mathsf{T}}K\rVert_2^2}{\sigma_z^2} \tag*{(88)} q = σ z 2 ∥ A Σ ˉ C T K ∥ 2 2 (88)
and we vary q q q by varying σ z 2 \sigma_z^2 σ z 2 similarly to the scalar experiment. We emphasize that here q q q is only a convenient measure of information strength, and our theoretical thresholds q 1 , … , q 4 q_1,\ldots,q_4 q 1 , … , q 4 do not necessarily extend to this multivariate setting. We generalize π P \pi^P π P as follows. Define
M δ = { ( x ^ , Σ ) : ∥ x ^ ∥ 2 2 ≤ δ , tr ( Σ ˉ − Σ ) ≤ δ } . (89) \mathcal{M}_{\delta}=\left\{(\hat{x},\Sigma):\lVert\hat{x}\rVert_2^2\leq\delta,\operatorname{tr}(\bar{\Sigma}-\Sigma)\leq\delta\right\}. \tag*{(89)} M δ = { ( x ^ , Σ ) : ∥ x ^ ∥ 2 2 ≤ δ , tr ( Σ ˉ − Σ ) ≤ δ } . (89)
Whenever the probing conditions of Eq (13) are satisfied π P \pi^P π P takes the action
K t x ^ t + a ⋅ 1 3 . (90) K_t\hat{x}_t+a\cdot1_3. \tag*{(90)} K t x ^ t + a ⋅ 1 3 . (90)
We use the same probing parameters a = 0.25 a=0.25 a = 0.25 , δ = 0.03 \delta=0.03 δ = 0.03 , L = 5 L=5 L = 5 , t P = 20 t_P=20 t P = 20 as the previous section.
Figure 8 shows that the same qualitative phenomenon persists in this multivariate example. For small values of q q q both policies do no better than open-loop. As q q q increases, probing begins to achieve a linear advantage over open-loop before CE does. For large values of q q q , CE eventually achieves a linear advantage over open-loop, and its performance approaches the probing policy. This experiment suggests that the separation between CE and probing is not restricted only to scalar settings. All experiments use 2000 independent Monte Carlo trajectories per policy. Error bands denote 95% normal confidence intervals for the trajectory-level mean.
Figure 8. (a) Depicts the normalized advantage of CE and probing over open-loop; (b) depicts the normalized advantage of probing over CE. The policy performance as q q q varies is similar to the scalar experiment in Figure 1 ; we see a clear separation between the two policies for intermediate values of q q q .
Nomenclature
Table 2. Main notation for the scalar setting (n = p = 1 n = p = 1 n = p = 1 ). Constants local to individual proofs are omitted.