Introduction
Q-learning [45 ] is one of the most fundamental algorithms in reinforcement learning (RL). It learns an optimal action-value function directly from sampled transitions and rewards. In this work, we study the convergence rate of Gaussian approximation for Q-learning as a function of the iteration number n n n , which is also the number of samples in the asynchronous setting. Considerable progress has been made in understanding the finite-sample guarantees for Q-learning [13 , 7 LWC+20, 23 ] . However, this line of work only bounds the distance between the estimate and the optimal Q-function, which does not determine the distribution of its error. Gaussian approximation addresses a different question. For the average Q ˉ n = n − 1 ∑ k = 1 n Q k \bar{Q}_n = n^{-1} \sum_{k=1}^{n} Q_k Q ˉ n = n − 1 ∑ k = 1 n Q k , we study how close the distribution of n ( Q ˉ n − Q ⋆ ) \sqrt{n}(\bar{Q}_n - Q^\star) n ( Q ˉ n − Q ⋆ ) is to a centered Gaussian distribution. This provides a theoretical foundation for statistical inference, including the construction of confidence intervals and hypothesis tests [39 ] . It also plays an important role in designing UCB- and LCB-based algorithms.
Stochastic approximation (SA) [3 , 4 ] is a general iterative framework to solve fixed-point equation problems. Since the Bellman operator in RL is a contraction map with a unique fixed point, many RL algorithms can be studied under this framework. A classic example is temporal-difference (TD) learning [38 ] , which is an instance of linear SA. In contrast, Q-learning involves maximizing over actions, making the underlying Bellman operator nonlinear. Q-learning is typically studied in two settings, synchronous and asynchronous, which correspond to SA with martingale difference noise and Markovian noise, respectively. In synchronous Q-learning, all state-action pairs are updated at each iteration using samples from a generative model. This requires the ability to sample transitions from arbitrary state-action pairs, which is typically only available in simulations. In contrast, asynchronous Q-learning updates only the state-action pair visited at each iteration, making it a realistic formulation for interaction with the real environment. However, this gives rise to Markovian noise. In this work, we focus mainly on the more challenging asynchronous setting and then extend our results to the synchronous setting.
Gaussian approximation for SA algorithms has been extensively studied [29 , 43 , 17 , 16 , 22 , 1 , 41 SMS+24, 25 ] . However, existing results may not directly apply to specific RL algorithms, which can present additional challenges. Moreover, existing SA results often rely on more restrictive assumptions and do not fully exploit problem-specific structure and parameters. The lower-bound constructions therein are also not applicable to RL algorithms. Non-asymptotic Gaussian approximations for TD learning have been established in [41 SMS+24] . For Q-learning, functional central limit theorems have been shown [47 , 28 , 49 , 34 ] . The first non-asymptotic Gaussian approximation bound for Q-learning was proved in [24 ] , which obtained a rate of O ~ ( n − 1 / 6 ) \widetilde{O}(n^{-1/6}) O ( n − 1/6 ) in Wasserstein distance under the asynchronous setting. Subsequently, [37 ] established a bound with the same rate on n n n in a different distance. These results leave open the question of whether the convergence rate can be further improved and, more fundamentally,
What is the sharpest achievable rate of Gaussian approximation for Q-learning in terms of n n n ?
Overview of Main Results
We study Q-learning with Polyak–Ruppert averaging and polynomial stepsizes α t = c 0 ( t + b ) − β \alpha_t = c_0(t+b)^{-\beta} α t = c 0 ( t + b ) − β . Our objective is to quantify how accurately a centered Gaussian approximates the normalized error X n = n ( Q ˉ n − Q ⋆ ) X_n = \sqrt{n}(\bar{Q}_n - Q^\star) X n = n ( Q ˉ n − Q ⋆ ) . We first consider asynchronous updates along a single Markovian trajectory. Theorem 3.1 gives the following bound in Wasserstein distance, up to logarithmic factors,
W 1 ( X n , Z ∞ ) ≲ n − 1 / 4 ⏟ Gaussian approximation + n β − 1 ⏟ covariance discrepancy + n 1 / 2 − β ⏟ Bellman nonlinearity and Markovian noise . W_1(X_n, Z_\infty) \lesssim\underbrace{n^{-1/4}}_{\text{Gaussian approximation}} + \underbrace{n^{\beta-1}}_{\text{covariance discrepancy}} + \underbrace{n^{1/2-\beta}}_{\text{Bellman nonlinearity and Markovian noise}}. W 1 ( X n , Z ∞ ) ≲ Gaussian approximation n − 1/4 + covariance discrepancy n β − 1 + Bellman nonlinearity and Markovian noise n 1/2 − β .
where Z ∞ ∼ N ( 0 , Σ ∞ ) Z_\infty\sim\mathcal{N}(0,\Sigma_\infty) Z ∞ ∼ N ( 0 , Σ ∞ ) , Σ ∞ \Sigma_\infty Σ ∞ is the limiting covariance, and 1 2 < β < 1 \frac{1}{2} < \beta< 1 2 1 < β < 1 .
The three terms reflect different parts of the approximation. First, we approximate a weighted sum M n M_n M n of fixed-point martingale noises by a Gaussian vector Z n ∼ N ( 0 , C n ) Z_n \sim\mathcal{N}(0,C_n) Z n ∼ N ( 0 , C n ) , where the deterministic covariance C n C_n C n retains the finite horizon. Second, we compare C n C_n C n with its limit Σ ∞ \Sigma_\infty Σ ∞ . Third, we analyze the errors from the nonlinear Bellman operator and the Markovian noises.
A key step in the analysis is to retain the exact averaging weights when constructing the intermediate Gaussian approximation. Replacing these weights by their limits and using a second-moment coupling gives an error bound of order n ( β − 1 ) / 2 n^{(\beta-1)/2} n ( β − 1 ) /2 . We instead first approximate M n M_n M n by Z n Z_n Z n and then compare them through their covariance matrices. The latter error is proportional to ∥ C n − Σ ∞ ∥ F = O ( n β − 1 ) \lVert C_n-\Sigma_\infty\rVert_F = O(n^{\beta-1}) ∥ C n − Σ ∞ ∥ F = O ( n β − 1 ) . Avoiding the additional square root allows a sharper balance with the n 1 / 2 − β n^{1/2-\beta} n 1/2 − β term and improves the previously established polynomial rate from n − 1 / 6 n^{-1/6} n − 1/6 to n − 1 / 4 n^{-1/4} n − 1/4 . We also carry out a numerical experiment (Figure 1 ) to validate our theoretical results.
Figure 1. Gaussian approximation for asynchronous Q Q Q -learning on a four-state, three-action MDP with β = 3 / 4 \beta= 3/4 β = 3/4 . (a) Empirical projected Wasserstein distance d ^ n \hat{d}_n d ^ n and its theoretical rate n − 1 / 4 B n^{-1/4}B n − 1/4 B , where B ≈ 73.13670 B \approx73.13670 B ≈ 73.13670 is computed from the given MDP and stepsize. (b) Covariance error ∥ C n − Σ ∞ ∥ F \lVert C_n - \Sigma_\infty\rVert_{\mathrm{F}} ∥ C n − Σ ∞ ∥ F and its theoretical rate n − 1 / 4 ∥ G ∥ F n^{-1/4}\lVert G\rVert_{\mathrm{F}} n − 1/4 ∥ G ∥ F , where ∥ G ∥ F ≈ 24.44716 \lVert G\rVert_{\mathrm{F}} \approx24.44716 ∥ G ∥ F ≈ 24.44716 .
The stepsize creates a trade-off between the last two terms. Increasing β \beta β makes the stepsizes smaller and reduces the accumulated nonlinear and Markovian errors of order n 1 / 2 − β n^{1/2-\beta} n 1/2 − β . However, it increases the covariance discrepancy of order n β − 1 n^{\beta-1} n β − 1 . Smaller updates give samples collected near the end less time to influence the average, increasing the difference between the finite-sample covariance and its limit. The two orders balance at β = 3 / 4 \beta= 3/4 β = 3/4 , giving the upper bound O ~ ( n − 1 / 4 ) \widetilde{O}(n^{-1/4}) O ( n − 1/4 ) .
To determine whether this rate can be improved, we construct a fixed two-state, one-action MDP and establish matching lower bounds for every fixed β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) in Theorem 3.2. On this example, the mean bias has order n α n \sqrt{n}\alpha_n n α n , and the covariance has the expansion C n = Σ ∞ + n β − 1 G + o ( n β − 1 ) C_n = \Sigma_\infty+ n^{\beta-1}G + o(n^{\beta-1}) C n = Σ ∞ + n β − 1 G + o ( n β − 1 ) with G ≻ 0 G \succ0 G ≻ 0 . These two results give a lower bound of order max { n 1 / 2 − β , n β − 1 } \max\{n^{1/2-\beta}, n^{\beta-1}\} max { n 1/2 − β , n β − 1 } . Since there is only one action, the Bellman operator is linear. Thus, the mean-bias lower bound isolates the effect of Markovian sampling. Together, the upper and lower bounds establish the optimal polynomial rate n − 1 / 4 n^{-1/4} n − 1/4 up to logarithmic factors.
We then study synchronous Q-learning, which uses independent transition samples for every state-action pair at each iteration. Under certain conditions and up to logarithmic factors,
W 1 ( X n , Z ∞ s y n ) ≲ n − 1 / 2 + n − β + n β − 1 , W_1(X_n, Z^{\mathrm{syn}}_\infty) \lesssim n^{-1/2} + n^{-\beta} + n^{\beta-1}, W 1 ( X n , Z ∞ syn ) ≲ n − 1/2 + n − β + n β − 1 ,
for 1 2 ≤ β < 1 \frac{1}{2} \le\beta< 1 2 1 ≤ β < 1 . Independent sampling removes the Markovian mean bias. This allows β = 1 / 2 \beta= 1/2 β = 1/2 and yields a better rate O ~ ( n − 1 / 2 ) \widetilde{O}(n^{-1/2}) O ( n − 1/2 ) . Switching the same lower-bound MDP to the synchronous setting gives an Ω ( n − 1 / 2 ) \Omega(n^{-1/2}) Ω ( n − 1/2 ) lower bound at β = 1 / 2 \beta= 1/2 β = 1/2 and an Ω ( n β − 1 ) \Omega(n^{\beta-1}) Ω ( n β − 1 ) lower bound for every fixed β > 1 / 2 \beta> 1/2 β > 1/2 . Thus the synchronous rate is also sharp up to logarithmic factors.
Remark 1.1. In asynchronous Q-learning, each iteration uses one sample, and therefore n n n denotes both the number of iterations and the number of samples. In the synchronous setting, we use n n n to count iterations, each of which requires one sample per state-action pair. The total sample size is therefore N = n ∣ S ∣ ∣ A ∣ N = n|\mathcal{S}||\mathcal{A}| N = n ∣ S ∣∣ A ∣ . Thus, faster convergence in iterations does not by itself imply better sample complexity. A comparison at the same sample budget must also account for the dimension dependence hidden in our bounds, which we leave for future work.
Preliminaries
An infinite-horizon discounted Markov decision process (MDP) is denoted by M \mathcal{M} M , and is defined by the tuple ⟨ S , A , P , r , γ ⟩ \langle\mathcal{S}, \mathcal{A}, P, r, \gamma\rangle ⟨ S , A , P , r , γ ⟩ where S \mathcal{S} S is the set of states, A \mathcal{A} A is the set of actions, P : S × A → Δ S P : \mathcal{S} \times\mathcal{A} \to\Delta_{\mathcal{S}} P : S × A → Δ S is the transition probability function, r ( s , a ) ∈ [ 0 , 1 ] r(s,a) \in[0,1] r ( s , a ) ∈ [ 0 , 1 ] is the reward, and γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) is the discount factor. Let d = ∣ S ∣ ∣ A ∣ d = |\mathcal{S}||\mathcal{A}| d = ∣ S ∣∣ A ∣ and H = ( 1 − γ ) − 1 H = (1-\gamma)^{-1} H = ( 1 − γ ) − 1 . Let Δ A \Delta_{\mathcal{A}} Δ A denotes the simplex over the action space. The action-value function (Q-function) of a stationary and stochastic policy π : S → Δ A \pi: \mathcal{S} \to\Delta_{\mathcal{A}} π : S → Δ A is defined as Q π ( s , a ) = E [ ∑ t = 0 ∞ γ t r ( s t , a t ) | s 0 = s , a 0 = a ] Q^\pi(s,a) = \mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^t r(s_t,a_t)\middle|s_0=s,a_0=a\right] Q π ( s , a ) = E [ ∑ t = 0 ∞ γ t r ( s t , a t ) ∣ s 0 = s , a 0 = a ] , where a t ∼ π ( ⋅ ∣ s t ) a_t \sim\pi(\cdot|s_t) a t ∼ π ( ⋅ ∣ s t ) and s t + 1 ∼ P ( ⋅ ∣ s t , a t ) s_{t+1} \sim P(\cdot|s_t,a_t) s t + 1 ∼ P ( ⋅ ∣ s t , a t ) . The optimal Q-function is defined as Q ⋆ : = max π Q π Q^\star:= \max_\pi Q^\pi Q ⋆ := max π Q π . We identify a Q-function with a vector in R d \mathbb{R}^d R d . For q ∈ R d q \in\mathbb{R}^d q ∈ R d , let V ( q ) ∈ R ∣ S ∣ V(q) \in\mathbb{R}^{|\mathcal{S}|} V ( q ) ∈ R ∣ S ∣ be its associated value vector and let T : R d → R d \mathcal{T} : \mathbb{R}^d \to\mathbb{R}^d T : R d → R d be the Bellman operator:
V ( q ) ( s ) = max a ∈ A q ( s , a ) , ( T q ) ( s , a ) = r ( s , a ) + γ ∑ s ′ ∈ S P ( s ′ ∣ s , a ) V ( q ) ( s ′ ) . (1) V(q)(s) = \max_{a \in\mathcal{A}} q(s,a), \qquad(\mathcal{T}q)(s,a) = r(s,a) + \gamma\sum_{s' \in\mathcal{S}} P(s' \mid s,a)V(q)(s'). \tag*{(1)} V ( q ) ( s ) = a ∈ A max q ( s , a ) , ( T q ) ( s , a ) = r ( s , a ) + γ s ′ ∈ S ∑ P ( s ′ ∣ s , a ) V ( q ) ( s ′ ) . (1)
The operator T \mathcal{T} T is a γ \gamma γ -contraction in the maximum norm. It therefore has a unique fixed point Q ⋆ Q^\star Q ⋆ . The reward bound gives Q ⋆ ∈ [ 0 , H ] d Q^\star\in[0,H]^d Q ⋆ ∈ [ 0 , H ] d . We write V ⋆ = V ( Q ⋆ ) V^\star= V(Q^\star) V ⋆ = V ( Q ⋆ ) for the optimal value function.
Samples are collected using a fixed behavior policy π b \pi_b π b . Given a state s t s_t s t , an action is sampled from π b ( ⋅ ∣ s t ) \pi_b(\cdot\mid s_t) π b ( ⋅ ∣ s t ) , and a next-state s t + 1 s_{t+1} s t + 1 is sampled from P ( ⋅ ∣ s t , a t ) P(\cdot\mid s_t,a_t) P ( ⋅ ∣ s t , a t ) . The transition triples Y t = ( s t , a t , s t + 1 ) Y_t = (s_t,a_t,s_{t+1}) Y t = ( s t , a t , s t + 1 ) form a Markov chain with kernel K K K . For y = ( s , a , s ′ ) y = (s,a,s') y = ( s , a , s ′ ) and y ~ = ( s ~ , a ~ , s ~ ′ ) \widetilde{y} = (\widetilde{s},\widetilde{a},\widetilde{s}') y = ( s , a , s ′ ) , this kernel is
K ( y , y ~ ) = 1 { s ~ = s ′ } π b ( a ~ ∣ s ~ ) P ( s ~ ′ ∣ s ~ , a ~ ) . K(y,\widetilde{y}) = \mathbf{1}\{\widetilde{s} = s'\}\pi_b(\widetilde{a} \mid\widetilde{s})P(\widetilde{s}' \mid\widetilde{s},\widetilde{a}). K ( y , y ) = 1 { s = s ′ } π b ( a ∣ s ) P ( s ′ ∣ s , a ) .
Let μ ˉ \bar{\mu} μ ˉ be its stationary distribution and let μ \mu μ be the stationary distribution of the state-action pair. Thus
μ ( s , a ) = ∑ s ′ ∈ S μ ˉ ( s , a , s ′ ) , μ ˉ ( s , a , s ′ ) = μ ( s , a ) P ( s ′ ∣ s , a ) . \mu(s,a) = \sum_{s' \in\mathcal{S}} \bar{\mu}(s,a,s'), \qquad\bar{\mu}(s,a,s') = \mu(s,a)P(s' \mid s,a). μ ( s , a ) = s ′ ∈ S ∑ μ ˉ ( s , a , s ′ ) , μ ˉ ( s , a , s ′ ) = μ ( s , a ) P ( s ′ ∣ s , a ) .
Assumption 2.1 (Mixing and exploration). There exist constants C m i x , c m i x > 0 C_{\mathrm{mix}},c_{\mathrm{mix}}>0 C mix , c mix > 0 such that
sup y ∥ K j ( y , ⋅ ) − μ ˉ ∥ T V ≤ C m i x exp ( − c m i x j ) , j ≥ 0. (2) \sup_y\left\|K^j(y,\cdot)-\bar{\mu}\right\|_{\mathrm{TV}}\le C_{\mathrm{mix}}\exp(-c_{\mathrm{mix}}j),\qquad j\ge0. \tag*{(2)} y sup K j ( y , ⋅ ) − μ ˉ TV ≤ C mix exp ( − c mix j ) , j ≥ 0. (2) Moreover, ρ : = min s , a μ ( s , a ) > 0 \rho:=\min_{s,a}\mu(s,a)>0 ρ := min s , a μ ( s , a ) > 0 .
Here ∥ ν − ν ′ ∥ T V = sup E ∣ ν ( E ) − ν ′ ( E ) ∣ \|\nu-\nu'\|_{\mathrm{TV}}=\sup_E|\nu(E)-\nu'(E)| ∥ ν − ν ′ ∥ TV = sup E ∣ ν ( E ) − ν ′ ( E ) ∣ , and the supremum in (S.2) ranges over admissible triples ( s , a , s ′ ) (s,a,s') ( s , a , s ′ ) with P ( s ′ ∣ s , a ) > 0 P(s' \mid s,a)>0 P ( s ′ ∣ s , a ) > 0 . We will use the following notations throughout the paper,
D = diag ( μ ( s , a ) : ( s , a ) ∈ S × A ) , a ⋆ = ρ ( 1 − γ ) > 0. D=\operatorname{diag}(\mu(s,a):(s,a)\in\mathcal{S}\times\mathcal{A}),\qquad a_{\star}=\rho(1-\gamma)>0. D = diag ( μ ( s , a ) : ( s , a ) ∈ S × A ) , a ⋆ = ρ ( 1 − γ ) > 0.
Assumption 2.1 serves two purposes. The mixing condition ensures the existence of bounded solutions to Poisson equations, supporting martingale approximations of the Markovian noise. The condition ρ > 0 \rho>0 ρ > 0 ensures that every state-action pair is visited with positive probability and that D D D is nonsingular. Neither condition requires every transition probability P ( s ′ ∣ s , a ) P(s'\mid s,a) P ( s ′ ∣ s , a ) to be positive. Assumption 2.1 is commonly used in the analysis of asynchronous Q-learning [LWC+20, QW20, CMSS21, ZDCL22, ZX24].
Notation. For vectors, ∥ ⋅ ∥ ∞ \|\cdot\|_{\infty} ∥ ⋅ ∥ ∞ and ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 denote the maximum and Euclidean norms. For matrices, ∥ ⋅ ∥ ∞ \|\cdot\|_{\infty} ∥ ⋅ ∥ ∞ is the induced maximum absolute row-sum norm, while ∥ ⋅ ∥ F \|\cdot\|_F ∥ ⋅ ∥ F and ∥ ⋅ ∥ o p \|\cdot\|_{\mathrm{op}} ∥ ⋅ ∥ op are the Frobenius and Euclidean operator norms. For p ≥ 1 p\ge1 p ≥ 1 , we write ∥ Z ∥ L p ( ℓ ∞ ) = ( E ∥ Z ∥ ∞ p ) 1 / p \|Z\|_{L^p(\ell_{\infty})}=(\mathbb{E}\|Z\|_{\infty}^p)^{1/p} ∥ Z ∥ L p ( ℓ ∞ ) = ( E ∥ Z ∥ ∞ p ) 1/ p . For random vectors with finite first moments, W 1 W_1 W 1 is the Wasserstein distance between their laws with Euclidean cost; equivalently,
W 1 ( X , Z ) = sup Lip 2 ( h ) ≤ 1 ∣ E h ( X ) − E h ( Z ) ∣ . W_1(X,Z)=\sup_{\operatorname{Lip}_2(h)\le1}\left|\mathbb{E}h(X)-\mathbb{E}h(Z)\right|. W 1 ( X , Z ) = Lip 2 ( h ) ≤ 1 sup ∣ E h ( X ) − E h ( Z ) ∣ .
where Lip 2 \operatorname{Lip}_2 Lip 2 is the Lipschitz constant relative to ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 . For the sake of simplicity, we use C C C to hide constants and some parameters. Throughout, C C C may change from line to line and may depend on many factors, but not on n , t , p n,t,p n , t , p .
Main Results
For each ( s , a ) (s,a) ( s , a ) , let u s , a ∈ R d \mathbf{u}_{s,a}\in\mathbb{R}^d u s , a ∈ R d be the standard basis vector with a 1 in coordinate ( s , a ) (s,a) ( s , a ) and zeros elsewhere. For y = ( s , a , s ′ ) y=(s,a,s') y = ( s , a , s ′ ) , define the sampled update
U ( q , y ) = u s , a { r ( s , a ) + γ V ( q ) ( s ′ ) − q ( s , a ) } . U(q,y)=\mathbf{u}_{s,a}\{r(s,a)+\gamma V(q)(s')-q(s,a)\}. U ( q , y ) = u s , a { r ( s , a ) + γ V ( q ) ( s ′ ) − q ( s , a )} .
Starting from an initial Q 0 ∈ [ 0 , H ] d Q_0\in[0,H]^d Q 0 ∈ [ 0 , H ] d , we run
Q t + 1 = Q t + α t U ( Q t , Y t ) , α t = c 0 ( t + b ) − β , 1 2 < β < 1 , (3) Q_{t+1}=Q_t+\alpha_tU(Q_t,Y_t),\qquad\alpha_t=c_0(t+b)^{-\beta},\qquad\frac{1}{2}<\beta<1, \tag*{(3)} Q t + 1 = Q t + α t U ( Q t , Y t ) , α t = c 0 ( t + b ) − β , 2 1 < β < 1 , (3)
where c 0 > 0 c_0>0 c 0 > 0 , α 0 ≤ 1 \alpha_0\le1 α 0 ≤ 1 , and the fixed shift b ≥ 1 b\ge1 b ≥ 1 satisfies β c 0 b β − 1 ≤ a ⋆ 4 \frac{\beta}{c_0}b^{\beta-1}\le\frac{a_\star}{4} c 0 β b β − 1 ≤ 4 a ⋆ . Note that b b b does not depend on n n n . The range β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) ensures ∑ t α t = ∞ \sum_t\alpha_t=\infty ∑ t α t = ∞ and ∑ t α t 2 < ∞ \sum_t\alpha_t^2<\infty ∑ t α t 2 < ∞ . The polynomial stepsize and the restriction β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) are chosen to satisfy the standard convergence conditions for SA with Polyak–Ruppert averaging [33 , 28 ] . Polyak-Ruppert averaging is a classical technique for reducing variance and improving the statistical efficiency of SA. We study Q-learning with Polyak-Ruppert averaging and define the iterate error, averaged estimate, and normalized error as
Δ t = Q t − Q ⋆ , Q ˉ n = 1 n ∑ k = 1 n Q k , X n = n ( Q ˉ n − Q ⋆ ) . (4) \Delta_t=Q_t-Q^\star,\qquad\bar{Q}_n=\frac{1}{n}\sum_{k=1}^{n}Q_k,\qquad X_n=\sqrt{n}(\bar{Q}_n-Q^\star). \tag*{(4)} Δ t = Q t − Q ⋆ , Q ˉ n = n 1 k = 1 ∑ n Q k , X n = n ( Q ˉ n − Q ⋆ ) . (4)
We use the filtration F t = σ ( Q 0 , Y 0 , Y 1 , … , Y t ) \mathcal{F}_t=\sigma(Q_0,Y_0,Y_1,\ldots,Y_t) F t = σ ( Q 0 , Y 0 , Y 1 , … , Y t ) .
Assumptions
We discuss Assumption 3.1 and Assumption 3.2, which are needed to establish our main results. Meanwhile, we introduce some key quantities, the nonlinear remainder φ \varphi φ , the martingale noise ϵ \epsilon ϵ , and the limiting covariance matrix Σ ∞ \Sigma_{\infty} Σ ∞ , that are important throughout our analysis. We first discuss Assumption 3.1, which controls a remainder term φ \varphi φ arising from the nonlinearity of the Bellman operator. Recall that U U U denotes the update and D D D accounts for the different update frequencies of the coordinates. For a fixed q ∈ R d q \in\mathbb{R}^{d} q ∈ R d , we define the stationary mean update as
B ( q ) = E Y ∼ μ ˉ [ U ( q , Y ) ] = D ( T q − q ) . B(q) = \mathbb{E}_{Y\sim\bar{\mu}}[U(q,Y)] = D(\mathcal{T}q-q). B ( q ) = E Y ∼ μ ˉ [ U ( q , Y )] = D ( T q − q ) .
Note that B ( Q ⋆ ) = 0 B(Q^{\star})=0 B ( Q ⋆ ) = 0 and E [ U ( q , Y t ) ∣ F t − 1 ] ≠ B ( q ) \mathbb{E}[U(q,Y_t)\mid\mathcal{F}_{t-1}]\ne B(q) E [ U ( q , Y t ) ∣ F t − 1 ] = B ( q ) . Fix a deterministic optimal policy π ⋆ ( s ) ∈ arg max a ∈ A Q ⋆ ( s , a ) \pi^{\star}(s)\in\arg\max_{a\in\mathcal{A}}Q^{\star}(s,a) π ⋆ ( s ) ∈ arg max a ∈ A Q ⋆ ( s , a ) , we define
( P π ⋆ q ) ( s , a ) = ∑ s ′ ∈ S P ( s ′ ∣ s , a ) q ( s ′ , π ⋆ ( s ′ ) ) , A = D ( I − γ P π ⋆ ) . (5) (P^{\pi^{\star}}q)(s,a)=\sum_{s'\in\mathcal{S}}P(s'\mid s,a)q(s',\pi^{\star}(s')), \qquad A=D(I-\gamma P^{\pi^{\star}}). \tag*{(5)} ( P π ⋆ q ) ( s , a ) = s ′ ∈ S ∑ P ( s ′ ∣ s , a ) q ( s ′ , π ⋆ ( s ′ )) , A = D ( I − γ P π ⋆ ) . (5)
Since Q ⋆ = r + γ P π ⋆ Q ⋆ Q^{\star}=r+\gamma P^{\pi^{\star}}Q^{\star} Q ⋆ = r + γ P π ⋆ Q ⋆ , if the update rule U U U does not take maximization over actions in each iteration, we have B ( Q ⋆ + δ ) = D [ r + γ P π ⋆ ( Q ⋆ + δ ) − Q ⋆ − δ ] = − D ( I − γ P π ⋆ ) δ = − A δ B(Q^{\star}+\delta)=D[r+\gamma P^{\pi^{\star}}(Q^{\star}+\delta)-Q^{\star}-\delta]=-D(I-\gamma P^{\pi^{\star}})\delta=-A\delta B ( Q ⋆ + δ ) = D [ r + γ P π ⋆ ( Q ⋆ + δ ) − Q ⋆ − δ ] = − D ( I − γ P π ⋆ ) δ = − A δ for a perturbation δ ∈ R d \delta\in\mathbb{R}^{d} δ ∈ R d . In this case, the problem becomes linear SA. Now we use φ ( δ ) \varphi(\delta) φ ( δ ) to denote the remainder term due to the nonlinearity of Q Q Q -learning. Since B ( Q ⋆ + δ ) = − A δ + φ ( δ ) B(Q^{\star}+\delta)=-A\delta+\varphi(\delta) B ( Q ⋆ + δ ) = − A δ + φ ( δ ) for some φ ( δ ) \varphi(\delta) φ ( δ ) , we write
φ ( δ ) : = B ( Q ⋆ + δ ) + A δ . \varphi(\delta):=B(Q^{\star}+\delta)+A\delta. φ ( δ ) := B ( Q ⋆ + δ ) + A δ .
The remainder has the explicit form
φ ( δ ) ( s , a ) = γ μ ( s , a ) ∑ s ′ ∈ S P ( s ′ ∣ s , a ) { V ( Q ⋆ + δ ) ( s ′ ) − V ⋆ ( s ′ ) − δ ( s ′ , π ⋆ ( s ′ ) ) } . \varphi(\delta)(s,a)=\gamma\mu(s,a)\sum_{s'\in\mathcal{S}}P(s'\mid s,a)\left\{V(Q^{\star}+\delta)(s')-V^{\star}(s')-\delta(s',\pi^{\star}(s'))\right\}. φ ( δ ) ( s , a ) = γ μ ( s , a ) s ′ ∈ S ∑ P ( s ′ ∣ s , a ) { V ( Q ⋆ + δ ) ( s ′ ) − V ⋆ ( s ′ ) − δ ( s ′ , π ⋆ ( s ′ )) } .
Now we state our Assumption 3.1, which controls the nonsmoothness caused by maximizing over actions in the stationary mean update.
Assumption 3.1. There is a finite constant L L L such that ∥ φ ( δ ) ∥ ∞ ≤ L ∥ δ ∥ ∞ 2 \|\varphi(\delta)\|_{\infty}\le L\|\delta\|_{\infty}^{2} ∥ φ ( δ ) ∥ ∞ ≤ L ∥ δ ∥ ∞ 2 for every δ \delta δ with Q ⋆ + δ ∈ [ 0 , H ] d Q^{\star}+\delta\in[0,H]^{d} Q ⋆ + δ ∈ [ 0 , H ] d .
Next, we discuss the limiting covariance in our Gaussian approximation target. We first note that the noise in the update U U U does not vanish at Q ⋆ Q^{\star} Q ⋆ . We denote this noise by
ϵ t = U ( Q ⋆ , Y t ) = γ u s t , a t { V ⋆ ( s t + 1 ) − ∑ s ′ ∈ S P ( s ′ ∣ s t , a t ) V ⋆ ( s ′ ) } . (6) \epsilon_t=U(Q^{\star},Y_t)=\gamma u_{s_t,a_t}\left\{V^{\star}(s_{t+1})-\sum_{s'\in\mathcal{S}}P(s'\mid s_t,a_t)V^{\star}(s')\right\}. \tag*{(6)} ϵ t = U ( Q ⋆ , Y t ) = γ u s t , a t { V ⋆ ( s t + 1 ) − s ′ ∈ S ∑ P ( s ′ ∣ s t , a t ) V ⋆ ( s ′ ) } . (6)
Note that this is a martingale noise. To see this, let G t = σ ( Q 0 , s 0 , a 0 , … , s t , a t ) \mathcal{G}_t=\sigma(Q_0,s_0,a_0,\ldots,s_t,a_t) G t = σ ( Q 0 , s 0 , a 0 , … , s t , a t ) denote the history before observing s t + 1 s_{t+1} s t + 1 . Conditional on G t \mathcal{G}_t G t , the pair ( s t , a t ) (s_t,a_t) ( s t , a t ) is known and the next state follows P ( ⋅ ∣ s t , a t ) P(\cdot\mid s_t,a_t) P ( ⋅ ∣ s t , a t ) . Thus, we have E [ ϵ t ∣ G t ] = 0 \mathbb{E}[\epsilon_t\mid\mathcal{G}_t]=0 E [ ϵ t ∣ G t ] = 0 . Since F t − 1 ⊆ G t \mathcal{F}_{t-1}\subseteq\mathcal{G}_t F t − 1 ⊆ G t for t ≥ 1 t\ge1 t ≥ 1 , this implies E ϵ 0 = 0 \mathbb{E}\epsilon_0=0 E ϵ 0 = 0 and E [ ϵ t ∣ F t − 1 ] = 0 \mathbb{E}[\epsilon_t\mid\mathcal{F}_{t-1}]=0 E [ ϵ t ∣ F t − 1 ] = 0 under any initial state distribution. Thus the fixed-point noise is a martingale difference sequence even though the transitions { Y t } t ∈ [ T ] \{Y_t\}_{t\in[T]} { Y t } t ∈ [ T ] are temporally dependent. Moreover, the centered update U ( Q t , Y t ) − B ( Q t ) U(Q_t,Y_t)-B(Q_t) U ( Q t , Y t ) − B ( Q t ) need not be a martingale difference.
Now we denote S = E μ ˉ [ ϵ 0 ϵ 0 ⊤ ] S=\mathbb{E}_{\bar{\mu}}[\epsilon_0\epsilon_0^{\top}] S = E μ ˉ [ ϵ 0 ϵ 0 ⊤ ] . The martingale difference property gives E [ ϵ t ϵ u ⊤ ] = 0 \mathbb{E}[\epsilon_t\epsilon_u^{\top}]=0 E [ ϵ t ϵ u ⊤ ] = 0 for t ≠ u t\ne u t = u . Consequently, under stationarity the covariance of a sum of these increments has no additional lag-covariance terms. Moreover, ϵ t \epsilon_t ϵ t has only one potentially nonzero coordinate. Therefore, S S S is diagonal with S ( s , a ) , ( s , a ) = μ ( s , a ) γ 2 Var s ′ ∼ P ( ⋅ ∣ s , a ) ( V ∗ ( s ′ ) ) S_{(s,a),(s,a)}=\mu(s,a)\gamma^2\operatorname{Var}_{s'\sim P(\cdot\mid s,a)}(V^*(s')) S ( s , a ) , ( s , a ) = μ ( s , a ) γ 2 Var s ′ ∼ P ( ⋅ ∣ s , a ) ( V ∗ ( s ′ )) . The covariance of our Gaussian approximation target N ( 0 , Σ ∞ ) \mathcal{N}(0,\Sigma_\infty) N ( 0 , Σ ∞ ) is
Σ ∞ = A − 1 S A − ⊤ , \Sigma_\infty=A^{-1}SA^{-\top}, Σ ∞ = A − 1 S A − ⊤ ,
where A − ⊤ = ( A − 1 ) ⊤ A^{-\top}=(A^{-1})^\top A − ⊤ = ( A − 1 ) ⊤ . We make the following assumption.
Assumption 3.2. The matrix Σ ∞ \Sigma_\infty Σ ∞ is positive definite.
Upper Bound
We now state our main result.
Theorem 3.1. Under Assumptions 2.1 2.1 2.1 , 3.1 3.1 3.1 , and 3.2 3.2 3.2 , let Z ∞ ∼ N ( 0 , Σ ∞ ) Z_\infty\sim\mathcal{N}(0,\Sigma_\infty) Z ∞ ∼ N ( 0 , Σ ∞ ) . For every n ≥ 2 n\ge2 n ≥ 2 ,
W 1 ( X n , Z ∞ ) ≤ C { n − 1 / 4 + n β − 1 + n 1 / 2 − β } . (7) W_1(X_n,Z_\infty)\le C\{n^{-1/4}+n^{\beta-1}+n^{1/2-\beta}\}. \tag*{(7)} W 1 ( X n , Z ∞ ) ≤ C { n − 1/4 + n β − 1 + n 1/2 − β } . (7) In particular, β = 3 / 4 \beta=3/4 β = 3/4 gives C n − 1 / 4 Cn^{-1/4} C n − 1/4 .
We next discuss the sources of error in the bound and present the proof sketch of the theorem. For i ≤ j i\le j i ≤ j , define Φ i : j = ∏ u = i j ( I − α u A ) \Phi_{i:j}=\prod_{u=i}^{j}(I-\alpha_u A) Φ i : j = ∏ u = i j ( I − α u A ) . We use the convention that Φ i : j = I \Phi_{i:j}=I Φ i : j = I when i > j i>j i > j . We define
W n , t = α t ∑ k = t + 1 n Φ t + 1 : k − 1 , M n = 1 n ∑ t < n W n , t ϵ t , C n = 1 n ∑ t < n W n , t S W n , t ⊤ . (8) \begin{aligned}
W_{n,t}&=\alpha_t\sum_{k=t+1}^{n}\Phi_{t+1:k-1},\qquad
M_n=\frac{1}{\sqrt{n}}\sum_{t<n}W_{n,t}\epsilon_t,\qquad
C_n=\frac{1}{n}\sum_{t<n}W_{n,t}SW_{n,t}^{\top}. \tag*{(8)}
\end{aligned} W n , t = α t k = t + 1 ∑ n Φ t + 1 : k − 1 , M n = n 1 t < n ∑ W n , t ϵ t , C n = n 1 t < n ∑ W n , t S W n , t ⊤ . (8)
We recall that Δ t = Q t − Q ∗ \Delta_t=Q_t-Q^* Δ t = Q t − Q ∗ and X n = n ( Q ˉ n − Q ∗ ) X_n=\sqrt{n}(\bar{Q}_n-Q^*) X n = n ( Q ˉ n − Q ∗ ) . The first step of the proof is to decompose X n X_n X n as
X n = M n + R n , X_n=M_n+R_n, X n = M n + R n ,
where R n R_n R n is a remainder term that accounts for several sources of error. Denote Z n ∼ N ( 0 , C n ) Z_n\sim\mathcal{N}(0,C_n) Z n ∼ N ( 0 , C n ) . Then, we have
W 1 ( X n , Z ∞ ) = W 1 ( M n + R n , Z ∞ ) ≤ W 1 ( M n + R n , M n ) + W 1 ( M n , Z n ) + W 1 ( Z n , Z ∞ ) . (9) W_1(X_n,Z_\infty)=W_1(M_n+R_n,Z_\infty)\le W_1(M_n+R_n,M_n)+W_1(M_n,Z_n)+W_1(Z_n,Z_\infty). \tag*{(9)} W 1 ( X n , Z ∞ ) = W 1 ( M n + R n , Z ∞ ) ≤ W 1 ( M n + R n , M n ) + W 1 ( M n , Z n ) + W 1 ( Z n , Z ∞ ) . (9)
We can see from (S.9) that we apply Gaussian approximation to M n M_n M n , which is a weighted sum of martingale difference noises. The error due to the reduction from Markovian noise to martingale difference noise is captured by the remainder term R n R_n R n . We now discuss each term on the right-hand side of (S.9) separately.
The Error Remainder R n R_n R n For the first term, we have W 1 ( M n + R n , M n ) ≤ E ∥ ( M n + R n ) − M n ∥ 2 = E ∥ R n ∥ 2 W_1(M_n + R_n, M_n) \le\mathbb{E}\lVert(M_n + R_n) - M_n\rVert_2 = \mathbb{E}\lVert R_n\rVert_2 W 1 ( M n + R n , M n ) ≤ E ∥( M n + R n ) − M n ∥ 2 = E ∥ R n ∥ 2 . Thus, we only need to upper bound the norm of R n R_n R n . We begin by observing that
Δ t + 1 = ( I − α t A ) Δ t + α t { ε t + f Δ t ( Y t ) + φ ( Δ t ) } (10) \Delta_{t+1} = (I - \alpha_t A)\Delta_t + \alpha_t\{\varepsilon_t + f_{\Delta_t}(Y_t) + \varphi(\Delta_t)\} \tag*{(10)} Δ t + 1 = ( I − α t A ) Δ t + α t { ε t + f Δ t ( Y t ) + φ ( Δ t )} (10)
where φ \varphi φ and ε \varepsilon ε are discussed in § 3.1 , and
f δ ( y ) = U ( Q ⋆ + δ , y ) − U ( Q ⋆ , y ) − E Y ∼ μ ˉ [ U ( Q ⋆ + δ , Y ) − U ( Q ⋆ , Y ) ] . f_\delta(y) = U(Q^\star+ \delta, y) - U(Q^\star, y) - \mathbb{E}_{Y\sim\bar{\mu}}\left[U(Q^\star+ \delta, Y) - U(Q^\star, Y)\right]. f δ ( y ) = U ( Q ⋆ + δ , y ) − U ( Q ⋆ , y ) − E Y ∼ μ ˉ [ U ( Q ⋆ + δ , Y ) − U ( Q ⋆ , Y ) ] .
Therefore, f Δ t ( Y t ) f_{\Delta_t}(Y_t) f Δ t ( Y t ) is the fluctuation of U ( Q t , Y t ) − U ( Q ⋆ , Y t ) U(Q_t, Y_t) - U(Q^\star, Y_t) U ( Q t , Y t ) − U ( Q ⋆ , Y t ) around its stationary average B ( Q t ) B(Q_t) B ( Q t ) , which is a Markovian noise term. Iterating the recursion for Δ t \Delta_t Δ t , and using X n = n − 1 / 2 ∑ k = 1 n Δ k X_n = n^{-1/2}\sum_{k=1}^{n}\Delta_k X n = n − 1/2 ∑ k = 1 n Δ k and R n = X n − M n R_n = X_n - M_n R n = X n − M n , we obtain
R n = R n i n i t + R n n o n l i n + R n M a r k o v . R_n = R_n^{\mathrm{init}} + R_n^{\mathrm{nonlin}} + R_n^{\mathrm{Markov}}. R n = R n init + R n nonlin + R n Markov .
where
R n i n i t = 1 n ∑ k = 1 n Φ 0 : k − 1 Δ 0 , R n n o n l i n = 1 n ∑ t < n W n , t φ ( Δ t ) , R n M a r k o v = 1 n ∑ t < n W n , t f Δ t ( Y t ) . R_n^{\mathrm{init}} = \frac{1}{\sqrt{n}}\sum_{k=1}^{n}\Phi_{0:k-1}\Delta_0,\qquad R_n^{\mathrm{nonlin}} = \frac{1}{\sqrt{n}}\sum_{t<n}W_{n,t}\varphi(\Delta_t),\qquad R_n^{\mathrm{Markov}} = \frac{1}{\sqrt{n}}\sum_{t<n}W_{n,t}f_{\Delta_t}(Y_t). R n init = n 1 k = 1 ∑ n Φ 0 : k − 1 Δ 0 , R n nonlin = n 1 t < n ∑ W n , t φ ( Δ t ) , R n Markov = n 1 t < n ∑ W n , t f Δ t ( Y t ) .
Thus, the term R n R_n R n can be decomposed into three sources of errors, arising from initialization, the nonlinearity of the Bellman operator, and Markovian noise, respectively. We next discuss each term separately.
The Initialization Error R n i n i t R_n^{\mathrm{init}} R n init The analysis of Φ i : j \Phi_{i:j} Φ i : j and W n , t W_{n,t} W n , t uses deterministic estimates for polynomial stepsizes, building on standard techniques in stochastic approximation (see, e.g., [33 , 28 , 41 , 24 ] ). The decay of the propagators bounds the initialization term
∥ R n i n i t ∥ 2 ≤ d ∥ Δ 0 ∥ ∞ n ∑ k = 1 n ∥ Φ 0 : k − 1 ∥ ∞ ≤ C n − 1 / 2 . \lVert R_n^{\mathrm{init}}\rVert_2 \le\frac{\sqrt{d}\lVert\Delta_0\rVert_\infty}{\sqrt{n}}\sum_{k=1}^{n}\lVert\Phi_{0:k-1}\rVert_\infty\le Cn^{-1/2}. ∥ R n init ∥ 2 ≤ n d ∥ Δ 0 ∥ ∞ k = 1 ∑ n ∥ Φ 0 : k − 1 ∥ ∞ ≤ C n − 1/2 .
The Nonlinearity Error R n n o n l i n R_n^{\mathrm{nonlin}} R n nonlin For the nonlinearity term, Assumption 3.1 gives ∥ φ ( δ ) ∥ ∞ ≤ L ∥ δ ∥ ∞ \|\varphi(\delta)\|_{\infty} \le L\|\delta\|_{\infty} ∥ φ ( δ ) ∥ ∞ ≤ L ∥ δ ∥ ∞ . The last-iterate moment bound proved in the appendix gives E ∥ Δ t ∥ ∞ 2 ≤ C α t \mathbb{E}\|\Delta_t\|_{\infty}^{2} \le C\alpha_t E ∥ Δ t ∥ ∞ 2 ≤ C α t . Combining these estimates with the uniform bound on W n , t W_{n,t} W n , t , we obtain
E ∥ R n n o n l i n ∥ 2 ≤ C n ∑ t < n E ∥ Δ t ∥ ∞ 2 ≤ C n ∑ t < n α t ≤ C n 1 / 2 − β . \mathbb{E}\|R_n^{\mathrm{nonlin}}\|_2 \le\frac{C}{\sqrt{n}}\sum_{t<n}\mathbb{E}\|\Delta_t\|_{\infty}^{2} \le\frac{C}{\sqrt{n}}\sum_{t<n}\alpha_t \le Cn^{1/2-\beta}. E ∥ R n nonlin ∥ 2 ≤ n C t < n ∑ E ∥ Δ t ∥ ∞ 2 ≤ n C t < n ∑ α t ≤ C n 1/2 − β .
The Markovian Error R n M a r k o v R_n^{\mathrm{Markov}} R n Markov We now apply the Poisson equation technique [15 , 31 , 12 , 8 ] to handle the Markovian term R n M a r k o v R_n^{\mathrm{Markov}} R n Markov . Recall that K K K denotes the transition kernel of the observation chain Y t Y_t Y t . The technique provides a solution to the Poisson equation h δ − K h δ = f δ h_\delta-Kh_\delta=f_\delta h δ − K h δ = f δ with sup y ∥ h δ ( y ) ∥ ∞ + sup y ∥ K h δ ( y ) ∥ ∞ ≤ C ∥ δ ∥ ∞ \sup_y\|h_\delta(y)\|_{\infty}+\sup_y\|Kh_\delta(y)\|_{\infty}\le C\|\delta\|_{\infty} sup y ∥ h δ ( y ) ∥ ∞ + sup y ∥ K h δ ( y ) ∥ ∞ ≤ C ∥ δ ∥ ∞ . We denote v δ : = K h δ v_\delta:=Kh_\delta v δ := K h δ . Adding and subtracting v Δ t ( Y t − 1 ) v_{\Delta_t}(Y_{t-1}) v Δ t ( Y t − 1 ) gives
f Δ t ( Y t ) = h Δ t ( Y t ) − v Δ t ( Y t − 1 ) ⏟ m t + v Δ t ( Y t − 1 ) − v Δ t ( Y t ) . f_{\Delta_t}(Y_t)=\underbrace{h_{\Delta_t}(Y_t)-v_{\Delta_t}(Y_{t-1})}_{m_t}+v_{\Delta_t}(Y_{t-1})-v_{\Delta_t}(Y_t). f Δ t ( Y t ) = m t h Δ t ( Y t ) − v Δ t ( Y t − 1 ) + v Δ t ( Y t − 1 ) − v Δ t ( Y t ) .
Since Δ t \Delta_t Δ t is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, the Markov property implies E [ h Δ t ( Y t ) ∣ F t − 1 ] = ( K h Δ t ) ( Y t − 1 ) = v Δ t ( Y t − 1 ) \mathbb{E}[h_{\Delta_t}(Y_t)\mid\mathcal{F}_{t-1}]=(Kh_{\Delta_t})(Y_{t-1})=v_{\Delta_t}(Y_{t-1}) E [ h Δ t ( Y t ) ∣ F t − 1 ] = ( K h Δ t ) ( Y t − 1 ) = v Δ t ( Y t − 1 ) . Thus m t m_t m t is a martingale difference, and the Poisson bounds give ∥ m t ∥ ∞ ≤ C ∥ Δ t ∥ ∞ \|m_t\|_{\infty}\le C\|\Delta_t\|_{\infty} ∥ m t ∥ ∞ ≤ C ∥ Δ t ∥ ∞ . Since the weights are uniformly bounded and E ∥ Δ t ∥ 2 2 ≤ C α t \mathbb{E}\|\Delta_t\|_2^2\le C\alpha_t E ∥ Δ t ∥ 2 2 ≤ C α t , we have
E ∥ 1 n ∑ t = 1 n − 1 W n , t m t ∥ 2 ≤ C ( 1 n ∑ t < n E ∥ Δ t ∥ 2 2 ) 1 / 2 ≤ C n − β / 2 . \mathbb{E}\left\|\frac{1}{\sqrt{n}}\sum_{t=1}^{n-1}W_{n,t}m_t\right\|_2\le C\left(\frac{1}{n}\sum_{t<n}\mathbb{E}\|\Delta_t\|_2^2\right)^{1/2}\le Cn^{-\beta/2}. E n 1 t = 1 ∑ n − 1 W n , t m t 2 ≤ C ( n 1 t < n ∑ E ∥ Δ t ∥ 2 2 ) 1/2 ≤ C n − β /2 .
For the remaining difference T n : = 1 n ∑ t = 1 n − 1 W n , t { v Δ t ( Y t − 1 ) − v Δ t ( Y t ) } T_n:=\frac{1}{\sqrt{n}}\sum_{t=1}^{n-1}W_{n,t}\{v_{\Delta_t}(Y_{t-1})-v_{\Delta_t}(Y_t)\} T n := n 1 ∑ t = 1 n − 1 W n , t { v Δ t ( Y t − 1 ) − v Δ t ( Y t )} , shifting the index in the first sum and separating the endpoints gives
T n = 1 n { W n , 1 v Δ 1 ( Y 0 ) − W n , n − 1 v Δ n − 1 ( Y n − 1 ) } + 1 n ∑ t = 1 n − 2 { W n , t + 1 v Δ t + 1 ( Y t ) − W n , t v Δ t ( Y t ) } . T_n=\frac{1}{\sqrt{n}}\{W_{n,1}v_{\Delta_1}(Y_0)-W_{n,n-1}v_{\Delta_{n-1}}(Y_{n-1})\}+\frac{1}{\sqrt{n}}\sum_{t=1}^{n-2}\{W_{n,t+1}v_{\Delta_{t+1}}(Y_t)-W_{n,t}v_{\Delta_t}(Y_t)\}. T n = n 1 { W n , 1 v Δ 1 ( Y 0 ) − W n , n − 1 v Δ n − 1 ( Y n − 1 )} + n 1 t = 1 ∑ n − 2 { W n , t + 1 v Δ t + 1 ( Y t ) − W n , t v Δ t ( Y t )} .
By the uniform boundedness of the weights and v Δ t v_{\Delta_t} v Δ t , the first part on the right-hand side can be bounded by C n − 1 / 2 Cn^{-1/2} C n − 1/2 . For the second part, adding and subtracting W n , t + 1 v Δ t ( Y t ) W_{n,t+1}v_{\Delta_t}(Y_t) W n , t + 1 v Δ t ( Y t ) in each interior term yields
W n , t + 1 v Δ t + 1 ( Y t ) − W n , t v Δ t ( Y t ) = ( W n , t + 1 − W n , t ) v Δ t ( Y t ) + W n , t + 1 { v Δ t + 1 ( Y t ) − v Δ t ( Y t ) } . W_{n,t+1}v_{\Delta_{t+1}}(Y_t)-W_{n,t}v_{\Delta_t}(Y_t)=(W_{n,t+1}-W_{n,t})v_{\Delta_t}(Y_t)+W_{n,t+1}\{v_{\Delta_{t+1}}(Y_t)-v_{\Delta_t}(Y_t)\}. W n , t + 1 v Δ t + 1 ( Y t ) − W n , t v Δ t ( Y t ) = ( W n , t + 1 − W n , t ) v Δ t ( Y t ) + W n , t + 1 { v Δ t + 1 ( Y t ) − v Δ t ( Y t )} .
The first term accounts for changes in the weights, while the second accounts for changes in the iterate. The total variation bound on the weights gives C n − 1 / 2 Cn^{-1/2} C n − 1/2 for the weight-variation sum. For the other sum, Lipschitz continuity gives
∥ v Δ t + 1 ( Y t ) − v Δ t ( Y t ) ∥ 2 ≤ C ∥ Δ t + 1 − Δ t ∥ 2 ≤ C α t . \|v_{\Delta_{t+1}}(Y_t)-v_{\Delta_t}(Y_t)\|_2\le C\|\Delta_{t+1}-\Delta_t\|_2\le C\alpha_t. ∥ v Δ t + 1 ( Y t ) − v Δ t ( Y t ) ∥ 2 ≤ C ∥ Δ t + 1 − Δ t ∥ 2 ≤ C α t .
Consequently,
∥ T n ∥ 2 ≤ C n ( 1 + 1 + ∑ t < n α t ) ≤ C { n − 1 / 2 + n 1 / 2 − β } . \|T_n\|_2\le\frac{C}{\sqrt{n}}\left(1+1+\sum_{t<n}\alpha_t\right)\le C\{n^{-1/2}+n^{1/2-\beta}\}. ∥ T n ∥ 2 ≤ n C ( 1 + 1 + t < n ∑ α t ) ≤ C { n − 1/2 + n 1/2 − β } .
Combining all the above, we have
E ∥ R n M a r k o v ∥ 2 ≤ C { n − β / 2 + n 1 / 2 − β + n − 1 / 2 } . \mathbb{E}\|R_n^{\mathrm{Markov}}\|_2\le C\{n^{-\beta/2}+n^{1/2-\beta}+n^{-1/2}\}. E ∥ R n Markov ∥ 2 ≤ C { n − β /2 + n 1/2 − β + n − 1/2 } .
Gaussian Approximation for M n M_n M n . We now turn to the remaining two terms on the right-hand side of (S.9). Recall that M n = 1 n ∑ t < n W n , t ε t M_n = \frac{1}{\sqrt{n}}\sum_{t<n} W_{n,t}\varepsilon_t M n = n 1 ∑ t < n W n , t ε t is a weighted sum of martingale noises. A natural approach is to replace the time-varying weights W n , t W_{n,t} W n , t by A − 1 A^{-1} A − 1 and introduce M ~ n = 1 n ∑ t < n A − 1 ε t \widetilde{M}_n = \frac{1}{\sqrt{n}}\sum_{t<n} A^{-1}\varepsilon_t M n = n 1 ∑ t < n A − 1 ε t . This allows us to apply existing Gaussian approximation results for sums of martingale differences (e.g., [41 ] ) by separately bounding W 1 ( M n , M ~ n ) W_1(M_n,\widetilde{M}_n) W 1 ( M n , M n ) and W 1 ( M ~ n , Z ∞ ) W_1(\widetilde{M}_n,Z_\infty) W 1 ( M n , Z ∞ ) . However, this is why previous work [24 , 37 ] obtained a loose rate of C n − 1 / 6 Cn^{-1/6} C n − 1/6 . To obtain a sharper rate, we instead retain the exact weights W n , t W_{n,t} W n , t and first approximate M n M_n M n by the Gaussian vector Z n Z_n Z n with covariance C n C_n C n . We then bound the error from replacing C n C_n C n by its limit Σ ∞ \Sigma_\infty Σ ∞ . This leads us to bound W 1 ( M n , Z n ) W_1(M_n,Z_n) W 1 ( M n , Z n ) and W 1 ( Z n , Z ∞ ) W_1(Z_n,Z_\infty) W 1 ( Z n , Z ∞ ) separately. We first explain why the previous approach leads to a slower convergence rate. By the coupling inequality for Wasserstein distance, we have
W 1 ( M n , M ~ n ) ≤ ( E ∥ M n − M ~ n ∥ 2 2 ) 1 / 2 . W_1(M_n,\widetilde{M}_n) \le\left(\mathbb{E}\lVert M_n-\widetilde{M}_n\rVert_2^2\right)^{1/2}. W 1 ( M n , M n ) ≤ ( E ∥ M n − M n ∥ 2 2 ) 1/2 .
Since
E ∥ M n − M ~ n ∥ 2 2 ≤ C n ∑ t < n ∥ W n , t − A − 1 ∥ ∞ ≤ C n β − 1 , \mathbb{E}\lVert M_n-\widetilde{M}_n\rVert_2^2 \le\frac{C}{n}\sum_{t<n}\lVert W_{n,t}-A^{-1}\rVert_\infty\le Cn^{\beta-1}, E ∥ M n − M n ∥ 2 2 ≤ n C t < n ∑ ∥ W n , t − A − 1 ∥ ∞ ≤ C n β − 1 ,
we obtain W 1 ( M n , M ~ n ) ≤ C n ( β − 1 ) / 2 W_1(M_n,\widetilde{M}_n) \le Cn^{(\beta-1)/2} W 1 ( M n , M n ) ≤ C n ( β − 1 ) /2 . Balancing this term with another source of error of order n 1 / 2 − β n^{1/2-\beta} n 1/2 − β requires choosing β = 2 / 3 \beta= 2/3 β = 2/3 , leading to the rate C n − 1 / 6 Cn^{-1/6} C n − 1/6 . Our approach avoids this by retaining the exact weights W n , t W_{n,t} W n , t and directly approximating M n M_n M n by a centered Gaussian vector Z n ∼ N ( 0 , C n ) Z_n \sim\mathcal{N}(0,C_n) Z n ∼ N ( 0 , C n ) , where C n C_n C n approximates the limiting covariance Σ ∞ \Sigma_\infty Σ ∞ . We first establish the Gaussian approximation for M n M_n M n , W 1 ( M n , Z n ) ≤ C n − 1 / 4 W_1(M_n,Z_n) \le Cn^{-1/4} W 1 ( M n , Z n ) ≤ C n − 1/4 . We then bound the distance between the two Gaussian distributions through their covariance matrices,
W 1 ( Z n , Z ∞ ) ≤ C ∥ C n − Σ ∞ ∥ F ≤ C n ∑ t < n ∥ W n , t − A − 1 ∥ ∞ ≤ C n β − 1 . W_1(Z_n,Z_\infty) \le C\lVert C_n-\Sigma_\infty\rVert_F \le\frac{C}{n}\sum_{t<n}\lVert W_{n,t}-A^{-1}\rVert_\infty\le Cn^{\beta-1}. W 1 ( Z n , Z ∞ ) ≤ C ∥ C n − Σ ∞ ∥ F ≤ n C t < n ∑ ∥ W n , t − A − 1 ∥ ∞ ≤ C n β − 1 .
This bound does not have an additional square root on the power. Consequently, balancing n β − 1 n^{\beta-1} n β − 1 with n 1 / 2 − β n^{1/2-\beta} n 1/2 − β allows us to choose β = 3 / 4 \beta= 3/4 β = 3/4 , resulting in the improved rate C n − 1 / 4 Cn^{-1/4} C n − 1/4 . The key insight from this proof technique is that bounding the second moment of the difference between two random vectors can yield a much looser estimate than directly bounding the difference between their covariance matrices.
Putting everything together. Combining all of the above, we obtain
W 1 ( X n , Z ∞ ) ≤ C { n − β / 2 + n 1 / 2 − β + n − 1 / 2 + n − 1 / 4 + n β − 1 } . W_1(X_n,Z_\infty) \le C\{n^{-\beta/2}+n^{1/2-\beta}+n^{-1/2}+n^{-1/4}+n^{\beta-1}\}. W 1 ( X n , Z ∞ ) ≤ C { n − β /2 + n 1/2 − β + n − 1/2 + n − 1/4 + n β − 1 } .
Since β > 1 / 2 \beta> 1/2 β > 1/2 , the terms n − β / 2 n^{-\beta/2} n − β /2 and n − 1 / 2 n^{-1/2} n − 1/2 are absorbed by n − 1 / 4 n^{-1/4} n − 1/4 , proving Theorem 3.1. The three terms left in the theorem have separate sources,
n − 1 / 4 ⏟ Gaussian approximation of M n , n β − 1 ⏟ finite-sample covariance , n 1 / 2 − β ⏟ Markovian noises . \underbrace{n^{-1/4}}_{\text{Gaussian approximation of }M_n},\qquad
\underbrace{n^{\beta-1}}_{\text{finite-sample covariance},\qquad}
\underbrace{n^{1/2-\beta}}_{\text{Markovian noises}}. Gaussian approximation of M n n − 1/4 , finite-sample covariance , n β − 1 Markovian noises n 1/2 − β .
In the current analysis, the last term arises from both Markovian noises and Bellman nonlinearity. In the next section, we show that Markovian noise is the fundamental source of this error.
Lower Bound
In this section, we establish matching lower bounds over the full stepsize range β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) . We use an MDP with one action and two states, for which the Bellman operator is linear. The updates remain asynchronous and the noises are Markovian. This construction shows that Markovian sampling can contribute to the Gaussian approximation error even in the absence of Bellman nonlinearity. We construct the following hard instance,
γ = 4 5 , P = ( 4 / 5 1 / 5 3 / 10 7 / 10 ) , r = ( 0 1 ) , μ = ( 3 / 5 2 / 5 ) . (11) \gamma= \frac{4}{5}, \qquad P = \begin{pmatrix} 4/5 & 1/5 \\ 3/10 & 7/10 \end{pmatrix}, \qquad r = \begin{pmatrix} 0 \\ 1 \end{pmatrix}, \qquad\mu= \begin{pmatrix} 3/5 \\ 2/5 \end{pmatrix}. \tag*{(11)} γ = 5 4 , P = ( 4/5 3/10 1/5 7/10 ) , r = ( 0 1 ) , μ = ( 3/5 2/5 ) . (11)
Since there is only one action, we have P π ⋆ = P P^{\pi^\star}=P P π ⋆ = P , V ⋆ ( i ) = Q i ⋆ V^\star(i)=Q^\star_i V ⋆ ( i ) = Q i ⋆ , u i = u i , a u_i=u_{i,a} u i = u i , a , and μ \mu μ is the stationary state distribution. Initialize s 0 ∼ μ s_0 \sim\mu s 0 ∼ μ and Q 0 = 0 Q_0=0 Q 0 = 0 . Direct calculation gives
Q ⋆ = ( 4 / 3 3 ) , A = 1 125 ( 27 − 12 − 12 22 ) , S = diag ( 64 / 375 , 56 / 375 ) . (12) Q^\star= \begin{pmatrix} 4/3 \\ 3 \end{pmatrix}, \qquad A = \frac{1}{125}\begin{pmatrix} 27 & -12 \\ -12 & 22 \end{pmatrix}, \qquad S = \operatorname{diag}(64/375,56/375). \tag*{(12)} Q ⋆ = ( 4/3 3 ) , A = 125 1 ( 27 − 12 − 12 22 ) , S = diag ( 64/375 , 56/375 ) . (12)
The chain is geometrically mixing, both states are visited, and A , S ≻ 0 A,S \succ0 A , S ≻ 0 . Thus all assumptions of the upper bound hold. The same problem will be used for every choice of stepsize parameters. Here a ⋆ = 2 / 25 a_\star=2/25 a ⋆ = 2/25 , and the admissible shift satisfies b ≥ 1 b \ge1 b ≥ 1 , α 0 ≤ 1 \alpha_0 \le1 α 0 ≤ 1 , β c 0 b β − 1 ≤ 1 50 \frac{\beta}{c_0}b^{\beta-1} \le\frac{1}{50} c 0 β b β − 1 ≤ 50 1 .
Theorem 3.2. For the problem in (S.11), every fixed β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) , and c 0 > 0 c_0>0 c 0 > 0 , there exist c > 0 c>0 c > 0 and n 0 < ∞ n_0<\infty n 0 < ∞ such that, for n ≥ n 0 n \ge n_0 n ≥ n 0 ,
W 1 ( X n , Z ∞ ) ≥ c { n 1 / 2 − β , 1 / 2 < β ≤ 3 / 4 , n β − 1 , 3 / 4 < β < 1. (13) W_1(X_n,Z_\infty) \ge c \begin{cases} n^{1/2-\beta}, & 1/2 < \beta\le3/4, \\ n^{\beta-1}, & 3/4 < \beta< 1. \end{cases} \tag*{(13)} W 1 ( X n , Z ∞ ) ≥ c { n 1/2 − β , n β − 1 , 1/2 < β ≤ 3/4 , 3/4 < β < 1. (13) The problem and initialization are independent of β , c 0 , b \beta,c_0,b β , c 0 , b . The constants may depend on these parameters.
The lower-bound argument consists of two parts, which we analyze separately below. The first part comes from the mean bias.
Proposition 3.1. On (S.11), with stationary state initialization,
E Δ t = α t v + O ( α t 2 + ( t + b ) − 1 ) , v = ( 248 / 1125 − 34 / 375 ) ≠ 0. (14) \mathbb{E}\Delta_t = \alpha_t v + O\left(\alpha_t^2+(t+b)^{-1}\right), \qquad v = \begin{pmatrix} 248/1125 \\ -34/375 \end{pmatrix} \ne0. \tag*{(14)} E Δ t = α t v + O ( α t 2 + ( t + b ) − 1 ) , v = ( 248/1125 − 34/375 ) = 0. (14) For any fixed bounded deterministic Q 0 Q_0 Q 0 , this implies
E X n = c 0 1 − β v n 1 / 2 − β + o ( n 1 / 2 − β ) . (15) \mathbb{E}X_n = \frac{c_0}{1-\beta}v n^{1/2-\beta} + o(n^{1/2-\beta}). \tag*{(15)} E X n = 1 − β c 0 v n 1/2 − β + o ( n 1/2 − β ) . (15)
Recall that the Markovian remainder R n M a r k o v = 1 n ∑ t < n W n , t f Δ t ( Y t ) R_n^{\mathrm{Markov}}=\frac{1}{\sqrt{n}}\sum_{t<n}W_{n,t}f_{\Delta_t}(Y_t) R n Markov = n 1 ∑ t < n W n , t f Δ t ( Y t ) contributes an n 1 / 2 − β n^{1/2-\beta} n 1/2 − β term to the upper bound. Although E μ ˉ f δ ( Y ) = 0 \mathbb{E}_{\bar{\mu}}f_\delta(Y)=0 E μ ˉ f δ ( Y ) = 0 for each fixed δ \delta δ , the dependence between Δ t \Delta_t Δ t and Y t Y_t Y t can make E f Δ t ( Y t ) \mathbb{E}f_{\Delta_t}(Y_t) E f Δ t ( Y t ) nonzero. Proposition 3.1 shows that this dependence produces a systematic mean bias on our fixed example. There is no Bellman nonlinearity, E M n = 0 \mathbb{E}M_n=0 E M n = 0 , and the initialization error term is O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) , so the leading bias comes entirely from R n M a r k o v R_n^{\mathrm{Markov}} R n Markov . Equation (S.14) identifies its order α t \alpha_t α t and fixed nonzero direction v v v . These biases accumulate under averaging to give (S.15). Since Z ∞ Z_\infty Z ∞ is centered,
W 1 ( X n , Z ∞ ) ≥ ∥ E X n ∥ 2 ≍ n 1 / 2 − β . W_1(X_n,Z_\infty) \ge\lVert\mathbb{E}X_n\rVert_2 \asymp n^{1/2-\beta}. W 1 ( X n , Z ∞ ) ≥ ∥ E X n ∥ 2 ≍ n 1/2 − β .
Thus, the n 1 / 2 − β n^{1/2-\beta} n 1/2 − β contribution in the upper bound can be unavoidable even when the Bellman operator is affine.
The second part comes from the finite-sample covariance approximation error, contributing the same order as in the upper bound. Previous work [42 ] showed that the discrepancy between the finite-sample and limiting variances yields an Ω ( n β − 1 ) \Omega(n^{\beta-1}) Ω ( n β − 1 ) lower bound in convex distance for Polyak–Ruppert averaged SGD on a scalar quadratic problem. [2 ] used the same example for linear stochastic approximation. On our two-state MDP, it identifies the leading covariance correction for the weighted martingale term. This correction has the same Lyapunov form as in [42 , 2 ] . Let X = ∫ 0 ∞ e − A s S e − A T s d s \mathcal{X}=\int_{0}^{\infty}e^{-As}Se^{-A^{\mathsf T}s}\,\mathrm{d}s X = ∫ 0 ∞ e − A s S e − A T s d s denote the unique positive-definite solution of the Lyapunov equation A X + X A T = S A\mathcal{X}+\mathcal{X}A^{\mathsf T}=S A X + X A T = S [19 ] .
Proposition 3.2. For the weights in (S.8) on (S.11),
C n = Σ ∞ + n β − 1 G + o ( n β − 1 ) , G = 1 c 0 A − 1 X A − T ≻ 0. (16) C_n=\Sigma_\infty+n^{\beta-1}G+o(n^{\beta-1}),\qquad G=\frac{1}{c_0}A^{-1}\mathcal{X}A^{-\mathsf T}\succ0. \tag*{(16)} C n = Σ ∞ + n β − 1 G + o ( n β − 1 ) , G = c 0 1 A − 1 X A − T ≻ 0. (16)
This expansion shows that the covariance error is genuinely of order n β − 1 n^{\beta-1} n β − 1 . Recall that Z n ∼ N ( 0 , C n ) Z_n\sim N(0,C_n) Z n ∼ N ( 0 , C n ) . Since G ≻ 0 G\succ0 G ≻ 0 , every diagonal entry satisfies G j j > 0 G_{jj}>0 G j j > 0 . Projection onto coordinate j j j and using the formula W 1 ( N ( 0 , σ 2 ) , N ( 0 , τ 2 ) ) = 2 π ∣ σ − τ ∣ W_1(N(0,\sigma^2),N(0,\tau^2))=\sqrt{\frac{2}{\pi}}|\sigma-\tau| W 1 ( N ( 0 , σ 2 ) , N ( 0 , τ 2 )) = π 2 ∣ σ − τ ∣ give
W 1 ( Z n , Z ∞ ) ≥ 2 π ∣ ( C n ) j j − ( Σ ∞ ) j j ∣ ≳ n β − 1 . W_1(Z_n,Z_\infty)\ge\sqrt{\frac{2}{\pi}}\left|\sqrt{(C_n)_{jj}}-\sqrt{(\Sigma_\infty)_{jj}}\right|\gtrsim n^{\beta-1}. W 1 ( Z n , Z ∞ ) ≥ π 2 ( C n ) j j − ( Σ ∞ ) j j ≳ n β − 1 .
Thus, the covariance comparison used in the upper bound has the correct order on this example. For β > 3 / 4 \beta>3/4 β > 3/4 , the finite-covariance approximation established in the upper-bound proof gives W 1 ( X n , Z n ) = o ( n β − 1 ) W_1(X_n,Z_n)=o(n^{\beta-1}) W 1 ( X n , Z n ) = o ( n β − 1 ) . The reverse triangle inequality then yields
W 1 ( X n , Z ∞ ) ≥ W 1 ( Z n , Z ∞ ) − W 1 ( X n , Z n ) ≳ n β − 1 . W_1(X_n,Z_\infty)\ge W_1(Z_n,Z_\infty)-W_1(X_n,Z_n)\gtrsim n^{\beta-1}. W 1 ( X n , Z ∞ ) ≥ W 1 ( Z n , Z ∞ ) − W 1 ( X n , Z n ) ≳ n β − 1 .
Combining both the upper and lower bounds, and letting β = 3 / 4 \beta=3/4 β = 3/4 , we have the following corollary.
Corollary 3.1. When β = 3 / 4 \beta=3/4 β = 3/4 and n ≥ n 0 n\ge n_0 n ≥ n 0 , we have
c n − 1 / 4 ≤ W 1 ( X n , Z ∞ ) ≤ C n − 1 / 4 log n . cn^{-1/4}\le W_1(X_n,Z_\infty)\le Cn^{-1/4}\log n. c n − 1/4 ≤ W 1 ( X n , Z ∞ ) ≤ C n − 1/4 log n .
The stepsize creates a trade-off between the accumulated Markovian mean bias and the finite-sample covariance discrepancy. Smaller stepsizes reduce the former, of order n α n \sqrt{n}\,\alpha_n n α n , but increase the latter, of order ( n α n ) − 1 (n\alpha_n)^{-1} ( n α n ) − 1 . This happens because smaller stepsizes make the updates slower, so samples collected near the end have less time to influence the average. This increases the discrepancy between the finite-sample covariance and its limit. Two orders n 1 / 2 − β n^{1/2-\beta} n 1/2 − β and n β − 1 n^{\beta-1} n β − 1 balance at β = 3 / 4 \beta=3/4 β = 3/4 , which gives the optimal polynomial Gaussian approximation rate n − 1 / 4 n^{-1/4} n − 1/4 .
Remark 3.3. Our lower bound also applies to Polyak–Ruppert averaged tabular TD(0), since asynchronous Q-learning coincides with tabular TD(0) on a one-action MDP. Specifically, it yields the same Wasserstein lower bound for TD(0) with arithmetic averaging and polynomial stepsizes α t = c 0 ( t + b ) − β \alpha_t=c_0(t+b)^{-\beta} α t = c 0 ( t + b ) − β , where β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) . On our fixed two-state instance, no fixed choice of β \beta β in this range gives a Gaussian approximation rate faster than n − 1 / 4 n^{-1/4} n − 1/4 . This complements existing O ( n − 1 / 4 ) O(n^{-1/4}) O ( n − 1/4 ) upper bounds for Markovian linear stochastic approximation in Kolmogorov distance [43 ] .
Synchronous Q-Learning
We now study Gaussian approximation for synchronous Q-learning. In this setting, every state-action pair is updated at each iteration using independent samples. Unlike in the asynchronous setting, here n n n denotes only the number of iterations, not the sample size. We retain the MDP and the notation Δ t \Delta_t Δ t , Q ˉ n \bar{Q}_n Q ˉ n , and X n X_n X n in (S.4), now defined using the synchronous iterates. We use the superscript syn to distinguish the drift and covariance matrices from their asynchronous counterparts.
At each iteration t t t , independently sample S t ′ ( s , a ) ∼ P ( ⋅ ∣ s , a ) S'_t(s,a) \sim P(\cdot\mid s,a) S t ′ ( s , a ) ∼ P ( ⋅ ∣ s , a ) for every ( s , a ) (s,a) ( s , a ) . The sample blocks ω t = ( S t ′ ( s , a ) ) s , a \omega_t = (S'_t(s,a))_{s,a} ω t = ( S t ′ ( s , a ) ) s , a are independent across iterations, and the samples within each block are independent. Starting from a deterministic Q 0 ∈ [ 0 , H ] d Q_0 \in[0,H]^d Q 0 ∈ [ 0 , H ] d , we run
Q t + 1 ( s , a ) = ( 1 − α t ) Q t ( s , a ) + α t { r ( s , a ) + γ V ( Q t ) ( S t ′ ( s , a ) ) } , α t = c 0 ( t + b ) − β . (17) Q_{t+1}(s,a) = (1-\alpha_t)Q_t(s,a) + \alpha_t\{r(s,a) + \gamma V(Q_t)(S'_t(s,a))\}, \qquad\alpha_t = c_0(t+b)^{-\beta}. \tag*{(17)} Q t + 1 ( s , a ) = ( 1 − α t ) Q t ( s , a ) + α t { r ( s , a ) + γ V ( Q t ) ( S t ′ ( s , a ))} , α t = c 0 ( t + b ) − β . (17)
As before, c 0 > 0 c_0 > 0 c 0 > 0 and b ≥ 1 b \ge1 b ≥ 1 are fixed independently of n n n . Write a s y n = 1 − γ a_{\mathrm{syn}} = 1-\gamma a syn = 1 − γ and choose parameters to satisfy 1 2 ≤ β < 1 \frac{1}{2} \le\beta< 1 2 1 ≤ β < 1 , α 0 ≤ 1 \alpha_0 \le1 α 0 ≤ 1 , β c 0 b β − 1 ≤ a s y n 4 \frac{\beta}{c_0}b^{\beta-1} \le\frac{a_{\mathrm{syn}}}{4} c 0 β b β − 1 ≤ 4 a syn . Unlike the asynchronous setting, the endpoint β = 1 / 2 \beta= 1/2 β = 1/2 is included. The iterates remain in [ 0 , H ] d [0,H]^d [ 0 , H ] d . Fix a deterministic optimal policy π ⋆ \pi^\star π ⋆ as in (S.5), and define ( P ^ t π ⋆ q ) ( s , a ) = q ( S t ′ ( s , a ) , π ⋆ ( S t ′ ( s , a ) ) ) (\widehat{P}^{\pi^\star}_tq)(s,a) = q(S'_t(s,a),\pi^\star(S'_t(s,a))) ( P t π ⋆ q ) ( s , a ) = q ( S t ′ ( s , a ) , π ⋆ ( S t ′ ( s , a ))) . The synchronous drift, fixed-point noise, and covariance matrices are
A s y n = I − γ P π ⋆ , ε t s y n = γ ( P ^ t π ⋆ − P π ⋆ ) Q ⋆ , A^{\mathrm{syn}} = I-\gamma P^{\pi^\star}, \qquad\varepsilon^{\mathrm{syn}}_t = \gamma(\widehat{P}^{\pi^\star}_t-P^{\pi^\star})Q^\star, A syn = I − γ P π ⋆ , ε t syn = γ ( P t π ⋆ − P π ⋆ ) Q ⋆ ,
S s y n = E [ ε t s y n ( ε t s y n ) ⊤ ] , Σ ∞ s y n = ( A s y n ) − 1 S s y n ( A s y n ) − ⊤ . (18) S^{\mathrm{syn}} = \mathbb{E}[\varepsilon^{\mathrm{syn}}_t(\varepsilon^{\mathrm{syn}}_t)^\top], \qquad\Sigma^{\mathrm{syn}}_\infty= (A^{\mathrm{syn}})^{-1}S^{\mathrm{syn}}(A^{\mathrm{syn}})^{-\top}. \tag*{(18)} S syn = E [ ε t syn ( ε t syn ) ⊤ ] , Σ ∞ syn = ( A syn ) − 1 S syn ( A syn ) − ⊤ . (18)
Since all coordinates are updated at every iteration, the visitation matrix D D D does not appear in A s y n A^{\mathrm{syn}} A syn . Row independence gives
S s y n = diag ( γ 2 Var s ′ ∼ P ( ⋅ ∣ s , a ) ( V ⋆ ( s ′ ) ) : ( s , a ) ∈ S × A ) . S^{\mathrm{syn}} = \operatorname{diag}\left(\gamma^2\operatorname{Var}_{s'\sim P(\cdot\mid s,a)}(V^\star(s')):(s,a)\in\mathcal{S}\times\mathcal{A}\right). S syn = diag ( γ 2 Var s ′ ∼ P ( ⋅ ∣ s , a ) ( V ⋆ ( s ′ )) : ( s , a ) ∈ S × A ) .
We impose the following action-gap and covariance conditions.
Assumption 4.1. There is one action, or every state has a unique optimal action with
κ = min s ∈ S min a ≠ π ⋆ ( s ) { Q ⋆ ( s , π ⋆ ( s ) ) − Q ⋆ ( s , a ) } > 0. \kappa= \min_{s\in\mathcal{S}}\min_{a\ne\pi^\star(s)}\{Q^\star(s,\pi^\star(s))-Q^\star(s,a)\} > 0. κ = s ∈ S min a = π ⋆ ( s ) min { Q ⋆ ( s , π ⋆ ( s )) − Q ⋆ ( s , a )} > 0. Moreover, $\Sigma^{\mathrm{syn}}_\infty \succ 0.
We will discuss this assumption in the appendix. This assumption allows us to express each sampled Bellman update as an exact linear recursion near Q ⋆ Q^\star Q ⋆ , a key step in the proof for synchronous Q-learning. Assumption 3.1 bounds the remainder in the stationary mean update and applying it would give an error of order n 1 / 2 − β n^{1/2-\beta} n 1/2 − β , which does not vanish at β = 1 / 2 \beta= 1/2 β = 1/2 . The covariance condition is the synchronous counterpart of Assumption 3.2 and is equivalent to requiring every diagonal entry of S s y n S^{\mathrm{syn}} S syn to be positive. No mixing, exploration, or stationary-initialization assumption is needed.
Upper Bound
Theorem 4.1. Under Assumption 4.1, let Z ∞ s y n ∼ N ( 0 , Σ ∞ s y n ) Z^{\mathrm{syn}}_\infty\sim\mathcal{N}(0,\Sigma^{\mathrm{syn}}_\infty) Z ∞ syn ∼ N ( 0 , Σ ∞ syn ) . For every n ≥ 2 n \ge2 n ≥ 2 ,
W 1 ( X n , Z ∞ s y n ) ≤ C { n − 1 / 2 log ( e n ) + n − β + n β − 1 } . (19) W_1(X_n,Z^{\mathrm{syn}}_\infty) \le C\{n^{-1/2}\log(en)+n^{-\beta}+n^{\beta-1}\}. \tag*{(19)} W 1 ( X n , Z ∞ syn ) ≤ C { n − 1/2 log ( e n ) + n − β + n β − 1 } . (19) In particular, β = 1 / 2 \beta= 1/2 β = 1/2 gives C n − 1 / 2 log ( e n ) Cn^{-1/2}\log(en) C n − 1/2 log ( e n ) . For every fixed β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) , the bound is O ( n β − 1 ) O(n^{\beta-1}) O ( n β − 1 ) .
We next describe the sources of error in this bound. To handle the Bellman nonlinearity, we compare Q t Q_t Q t with a fixed-policy recursion driven by the same samples:
L 0 = Q 0 , L t + 1 = ( 1 − α t ) L t + α t ( r + γ P ^ t π ⋆ L t ) , X n l i n = 1 n ∑ k = 1 n ( L k − Q ⋆ ) . (20) L_0=Q_0,\qquad L_{t+1}=(1-\alpha_t)L_t+\alpha_t(r+\gamma\widehat{P}^{\pi^\star}_tL_t),\qquad X^{\mathrm{lin}}_n=\frac{1}{\sqrt{n}}\sum_{k=1}^{n}(L_k-Q^\star). \tag*{(20)} L 0 = Q 0 , L t + 1 = ( 1 − α t ) L t + α t ( r + γ P t π ⋆ L t ) , X n lin = n 1 k = 1 ∑ n ( L k − Q ⋆ ) . (20)
The sampled Bellman map at Q t Q_t Q t agrees with its fixed-policy counterpart whenever ∥ Q t − Q ⋆ ∥ ∞ < κ / 2 \lVert Q_t-Q^\star\rVert_\infty< \kappa/2 ∥ Q t − Q ⋆ ∥ ∞ < κ /2 . For the covariance comparison, define the synchronous versions of the weights and covariance in (S.8):
Φ i : j s y n = ∏ u = i j ( I − α u A s y n ) , W n , t s y n = α t ∑ k = t + 1 n Φ t + 1 : k − 1 s y n , C n s y n = 1 n ∑ t < n W n , t s y n S s y n ( W n , t s y n ) T , (21) \Phi^{\mathrm{syn}}_{i:j}=\prod_{u=i}^{j}(I-\alpha_u A^{\mathrm{syn}}),\qquad W^{\mathrm{syn}}_{n,t}=\alpha_t\sum_{k=t+1}^{n}\Phi^{\mathrm{syn}}_{t+1:k-1},\qquad C^{\mathrm{syn}}_n=\frac{1}{n}\sum_{t<n}W^{\mathrm{syn}}_{n,t}S^{\mathrm{syn}}(W^{\mathrm{syn}}_{n,t})^{\mathsf{T}}, \tag*{(21)} Φ i : j syn = u = i ∏ j ( I − α u A syn ) , W n , t syn = α t k = t + 1 ∑ n Φ t + 1 : k − 1 syn , C n syn = n 1 t < n ∑ W n , t syn S syn ( W n , t syn ) T , (21)
with the same empty-product convention. Write Z n s y n ∼ N ( 0 , C n s y n ) Z^{\mathrm{syn}}_n\sim\mathcal{N}(0,C^{\mathrm{syn}}_n) Z n syn ∼ N ( 0 , C n syn ) .
Proposition 4.1. Under the assumptions of Theorem 4.1, for every n ≥ 2 n\ge2 n ≥ 2 , we have
E ∥ X n − X n l i n ∥ 2 ≤ C n − 1 / 2 , (22) \mathbb{E}\lVert X_n-X^{\mathrm{lin}}_n\rVert_2\le Cn^{-1/2}, \tag*{(22)} E ∥ X n − X n lin ∥ 2 ≤ C n − 1/2 , (22) W 1 ( X n l i n , Z n s y n ) ≤ C { n − 1 / 2 log ( e n ) + n − β } , (23) W_1(X^{\mathrm{lin}}_n,Z^{\mathrm{syn}}_n)\le C\{n^{-1/2}\log(en)+n^{-\beta}\}, \tag*{(23)} W 1 ( X n lin , Z n syn ) ≤ C { n − 1/2 log ( e n ) + n − β } , (23) ∥ C n s y n − Σ ∞ s y n ∥ F ≤ C n β − 1 . (24) \lVert C^{\mathrm{syn}}_n-\Sigma^{\mathrm{syn}}_\infty\rVert_F\le Cn^{\beta-1}. \tag*{(24)} ∥ C n syn − Σ ∞ syn ∥ F ≤ C n β − 1 . (24) The first term bounds the difference between X n X_n X n and X n l i n X^{\mathrm{lin}}_n X n lin . The difference is zero when there is one action. When ∥ Q t − Q ⋆ ∥ ∞ < κ / 2 \lVert Q_t-Q^\star\rVert_\infty< \kappa/2 ∥ Q t − Q ⋆ ∥ ∞ < κ /2 , the policy π ⋆ \pi^\star π ⋆ is greedy with respect to Q t Q_t Q t , so the maximization introduces no additional error. Such errors arise only when Q t Q_t Q t lies outside this neighborhood. These exceptional updates become rare enough that their expected errors have a finite sum, giving an O ( n − 1 / 2 ) O(n^{-1/2}) O ( n − 1/2 ) contribution to the normalized average. This gives a sharper bound than using the quadratic estimate of the Bellman remainder.
The second term is the Gaussian approximation error for X n l i n X^{\mathrm{lin}}_n X n lin . We use a resampling argument based on the independent sample blocks to obtain the n − 1 / 2 log ( e n ) n^{-1/2}\log(en) n − 1/2 log ( e n ) term. The additional O ( n − β ) O(n^{-\beta}) O ( n − β ) term accounts for the difference between the true covariance of X n l i n X^{\mathrm{lin}}_n X n lin and the deterministic matrix C n s y n C^{\mathrm{syn}}_n C n syn used in the Gaussian approximation.
The last term is the finite-sample covariance discrepancy, which also appears in the asynchronous bound. Independent sampling removes the accumulated Markovian mean bias of order n 1 / 2 − β n^{1/2-\beta} n 1/2 − β . Together with the sharper treatment of the sampled recursion and Bellman nonlinearity, this allows the choice β = 1 / 2 \beta=1/2 β = 1/2 and improves the optimal polynomial rate from n − 1 / 4 n^{-1/4} n − 1/4 to n − 1 / 2 n^{-1/2} n − 1/2 .
Lower Bound
We use the same two-state, one-action MDP in (S.11), now drawing independent samples from both transition rows at every iteration and initializing Q 0 = 0 Q_0=0 Q 0 = 0 . Thus, the MDP and Q Q Q -function initialization are unchanged, while the sampling scheme is synchronous.
Theorem 4.2. For the MDP in (S.11) with synchronous sampling and Q 0 = 0 Q_0=0 Q 0 = 0 , and for fixed choice of c 0 c_0 c 0 , b b b , β \beta β , constants c > 0 c>0 c > 0 and n 0 < ∞ n_0<\infty n 0 < ∞ such that, for n ≥ n 0 n\ge n_0 n ≥ n 0 ,
W 1 ( X n , Z ∞ s y n ) ≥ c { n − 1 / 2 , β = 1 / 2 , n β − 1 , 1 / 2 < β < 1. (25) W_1(X_n,Z^{\mathrm{syn}}_\infty)\ge c\begin{cases}
n^{-1/2}, & \beta=1/2,\\
n^{\beta-1}, & 1/2<\beta<1.
\end{cases} \tag*{(25)} W 1 ( X n , Z ∞ syn ) ≥ c { n − 1/2 , n β − 1 , β = 1/2 , 1/2 < β < 1. (25) The MDP and initialization are independent of the stepsize parameters.
The difference from the asynchronous lower bound is the absence of the Markovian mean bias. Under asynchronous sampling, the dependence between the current iterate and the visited state produces a bias of order n 1 / 2 − β n^{1/2-\beta} n 1/2 − β on this same one-action MDP. Under synchronous sampling, each new sample block is independent of the current iterate, so the mean error on this linear instance follows a homogeneous deterministic recursion. Thus, the lower bound no longer contains the
n 1 / 2 − β n^{1/2-\beta} n 1/2 − β term that forces the asynchronous choice β = 3 / 4 \beta= 3/4 β = 3/4 . For β > 1 / 2 \beta> 1/2 β > 1/2 , the lower bound comes from the finite-sample covariance discrepancy. The deterministic covariance expansion used in Proposition 3.2 also applies to the synchronous matrices on this example. Letting β = 1 / 2 \beta= 1/2 β = 1/2 , we have the following corollary.
Corollary 4.1. For all sufficiently large n n n , we have
c n − 1 / 2 ≤ W 1 ( X n , Z ∞ s y n ) ≤ C n − 1 / 2 log ( e n ) . cn^{-1/2} \leq W_1(X_n, Z_{\infty}^{\mathrm{syn}}) \leq Cn^{-1/2}\log(en). c n − 1/2 ≤ W 1 ( X n , Z ∞ syn ) ≤ C n − 1/2 log ( e n ) .
Conclusion and Future Work
In this paper, we established sharp non-asymptotic Gaussian approximation rates for Polyak-Ruppert averaged Q-learning. For polynomial stepsizes α t = c 0 ( t + b ) − β \alpha_t = c_0(t+b)^{-\beta} α t = c 0 ( t + b ) − β , we proved an O ~ ( n − 1 / 4 + n β − 1 + n 1 / 2 − β ) \widetilde{O}(n^{-1/4} + n^{\beta-1} + n^{1/2-\beta}) O ( n − 1/4 + n β − 1 + n 1/2 − β ) upper bound for asynchronous Q-learning and an O ~ ( n − 1 / 2 + n β − 1 + n − β ) \widetilde{O}(n^{-1/2} + n^{\beta-1} + n^{-\beta}) O ( n − 1/2 + n β − 1 + n − β ) upper bound for synchronous Q-learning, together with matching lower bounds up to logarithmic factors. Our analysis identifies the main sources of Gaussian approximation error, including Bellman nonlinearity, Markovian noise, and the finite-sample covariance discrepancy. In particular, we show that Markovian sampling can produce an unavoidable mean bias even when the Bellman operator is linear. The trade-off between this bias and the covariance discrepancy explains the optimal choice β = 3 / 4 \beta= 3/4 β = 3/4 and the resulting n − 1 / 4 n^{-1/4} n − 1/4 rate in the asynchronous setting. These results establish the fundamental convergence limits of Gaussian approximation for Q-learning and provide a theoretical foundation for finite-sample statistical inference in reinforcement learning.
We believe that the analysis and techniques developed in this work can be extended to other metrics beyond W 1 W_1 W 1 without major modifications. Nevertheless, there are several important and interesting questions that remain for future work.
First, this work only focuses on n n n , while the optimal dependence on other problem-specific parameters, such as the state-action space size, the discount factor γ \gamma γ , and the exploration parameter ρ \rho ρ , remains unknown.
Second, although our assumptions are standard in the literature, relaxing them remains important, as they exclude certain classes of MDPs. Assumption 3.1 is slightly weaker than the corresponding assumptions in previous work [28 , 24 , 37 ] , but it still excludes MDPs with multiple optimal actions. Assumption 3.2, which requires the limiting covariance matrix to be positive definite, is also commonly used when there are Markovian noises. However, it excludes MDPs with deterministic transitions, since Σ ∞ ≻ 0 ⟺ Var s ′ ∼ P ( ⋅ ∣ s , a ) [ V ⋆ ( s ′ ) ] > 0 \Sigma_{\infty} \succ0 \Longleftrightarrow\operatorname{Var}_{s' \sim P(\cdot\mid s,a)}[V^{\star}(s')] > 0 Σ ∞ ≻ 0 ⟺ Var s ′ ∼ P ( ⋅ ∣ s , a ) [ V ⋆ ( s ′ )] > 0 , ∀ ( s , a ) \forall(s,a) ∀ ( s , a ) . Relaxing these assumptions while preserving sharp Gaussian approximation rates is an important but challenging direction for future research.
Third, an interesting problem is to study non-asymptotic Gaussian approximation under other stepsize schedules, particularly constant and linearly decaying stepsizes. Following the seminal work of [33 ] , classical convergence results for Polyak-Ruppert averaging typically rely on the following sufficient conditions: (i) 0 ≤ sup t α t ≤ 1 0 \leq\sup_t \alpha_t \leq1 0 ≤ sup t α t ≤ 1 , α t ↓ 0 \alpha_t \downarrow0 α t ↓ 0 , and α t → ∞ \alpha_t \to\infty α t → ∞ ; (ii) α t − 1 α t = o ( α t − 1 ) \frac{\alpha_{t-1}}{\alpha_t} = o(\alpha_{t-1}) α t α t − 1 = o ( α t − 1 ) ; (iii) 1 T ∑ t = 0 T α t → 0 \frac{1}{\sqrt{T}}\sum_{t=0}^{T}\alpha_t \to0 T 1 ∑ t = 0 T α t → 0 ; and (iv) ∑ t = 0 T α t T α T ≤ C \frac{\sum_{t=0}^{T}\alpha_t}{T\alpha_T} \leq C T α T ∑ t = 0 T α t ≤ C . Polynomial stepsizes α t = c 0 ( t + b ) − β \alpha_t = c_0(t+b)^{-\beta} α t = c 0 ( t + b ) − β with β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) satisfy these conditions, whereas constant and linearly decaying stepsizes do not. Nevertheless, these conditions are not necessary for Gaussian approximation, and both alternative stepsize schedules may admit Gaussian limits under alternative assumptions. Previous works [28 , 49 ] have studied Gaussian approximation for asynchronous and synchronous Q-learning beyond polynomial stepsize, but the results are asymptotic. Establishing non-asymptotic Gaussian approximation rates for Q-learning under different stepsize schedules remains open.
Last but not least, Gaussian approximation for the last iterate, beyond the Polyak–Ruppert averaging scheme, is an important direction and has recently attracted increasing attention [11 , 20 , 6 , 46 ] .
AI Disclosure. The proof was developed with assistance from GPT-5.6 Sol and GPT-6 Astra, which was verified and rewritten by the author. The author takes full responsibility for the correctness, presentation, and communication of the results. The author also made an effort to identify and cite, to the best of the author’s knowledge, prior work whose results were used in the proof but were not cited by the model.
Proofs for Section 3.1
In this section, we first prove that the assumptions imposed in [24 , 37 ] are sufficient conditions for Assumption 3.1 to be satisfied. Moreover, we show that these three assumptions are equivalent whenever ∑ x ∈ S ∑ b ∈ A μ ( x , b ) P ( s ∣ x , b ) > 0 \sum_{x\in\mathcal{S}}\sum_{b\in\mathcal{A}}\mu(x,b)P(s\mid x,b)>0 ∑ x ∈ S ∑ b ∈ A μ ( x , b ) P ( s ∣ x , b ) > 0 for every s ∈ S s\in\mathcal{S} s ∈ S .
Assumption A.1 (Assumption from [24 ] ). There exists a constant C < ∞ C<\infty C < ∞ such that
∥ ( P π q − P π ⋆ ) ( q − Q ⋆ ) ∥ ∞ ≤ C ∥ q − Q ⋆ ∥ ∞ 2 \left\lVert(P^{\pi_q}-P^{\pi^\star})(q-Q^\star)\right\rVert_{\infty}\leq C\left\lVert q-Q^\star\right\rVert_{\infty}^{2} ( P π q − P π ⋆ ) ( q − Q ⋆ ) ∞ ≤ C ∥ q − Q ⋆ ∥ ∞ 2
for every q ∈ [ 0 , H ] d q \in[0,H]^d q ∈ [ 0 , H ] d , where π q \pi_q π q is a deterministic greedy policy with respect to q q q .
Assumption A.2 (Assumption from [37 ] ). Every state has a unique optimal action, and the minimum optimal action gap satisfies
κ : = min s ∈ S min a ≠ π ⋆ ( s ) { Q ⋆ ( s , π ⋆ ( s ) ) − Q ⋆ ( s , a ) } > 0. \kappa:= \min_{s \in S} \min_{a \ne\pi^\star(s)} \left\{Q^\star(s,\pi^\star(s)) - Q^\star(s,a)\right\} > 0. κ := s ∈ S min a = π ⋆ ( s ) min { Q ⋆ ( s , π ⋆ ( s )) − Q ⋆ ( s , a ) } > 0.
Lemma A.1. Assumption 3.1 holds if either Assumption A.1 or Assumption A.2 holds.
Proof. Recall that
B ( q ) = D ( T q − q ) , A = D ( I − γ P π ⋆ ) , B(q) = D(Tq-q), \qquad A = D(I-\gamma P^{\pi^\star}), B ( q ) = D ( T q − q ) , A = D ( I − γ P π ⋆ ) ,
and
φ ( δ ) = B ( Q ⋆ + δ ) + A δ . \varphi(\delta) = B(Q^\star+\delta)+A\delta. φ ( δ ) = B ( Q ⋆ + δ ) + A δ .
Let q = Q ⋆ + δ ∈ [ 0 , H ] d q=Q^\star+\delta\in[0,H]^d q = Q ⋆ + δ ∈ [ 0 , H ] d . Since T Q ⋆ = Q ⋆ TQ^\star=Q^\star T Q ⋆ = Q ⋆ , we have
φ ( δ ) = D ( T q − q ) + D ( I − γ P π ⋆ ) δ = D ( T q − T Q ⋆ − γ P π ⋆ δ ) . \varphi(\delta) = D(Tq-q)+D(I-\gamma P^{\pi^\star})\delta= D(Tq-TQ^\star-\gamma P^{\pi^\star}\delta). φ ( δ ) = D ( T q − q ) + D ( I − γ P π ⋆ ) δ = D ( T q − T Q ⋆ − γ P π ⋆ δ ) .
Using
T q = r + γ P π q q , T Q ⋆ = r + γ P π ⋆ Q ⋆ , Tq = r+\gamma P^{\pi_q}q, \qquad TQ^\star= r+\gamma P^{\pi^\star}Q^\star, T q = r + γ P π q q , T Q ⋆ = r + γ P π ⋆ Q ⋆ ,
we obtain
φ ( δ ) = γ D ( P π q − P π ⋆ ) q . (26) \varphi(\delta) = \gamma D(P^{\pi_q}-P^{\pi^\star})q. \tag*{(26)} φ ( δ ) = γ D ( P π q − P π ⋆ ) q . (26)
By the definition of π q \pi_q π q , for every state s s s ,
q ( s , π q ( s ) ) ≥ q ( s , π ⋆ ( s ) ) . q(s,\pi_q(s)) \ge q(s,\pi^\star(s)). q ( s , π q ( s )) ≥ q ( s , π ⋆ ( s )) .
Similarly, the optimality of π ⋆ \pi^\star π ⋆ implies
Q ⋆ ( s , π q ( s ) ) ≤ Q ⋆ ( s , π ⋆ ( s ) ) . Q^\star(s,\pi_q(s)) \le Q^\star(s,\pi^\star(s)). Q ⋆ ( s , π q ( s )) ≤ Q ⋆ ( s , π ⋆ ( s )) .
It follows that, componentwise,
( P π q − P π ⋆ ) q ≥ 0 , ( P π q − P π ⋆ ) Q ⋆ ≤ 0. (P^{\pi_q}-P^{\pi^\star})q \ge0, \qquad(P^{\pi_q}-P^{\pi^\star})Q^\star\le0. ( P π q − P π ⋆ ) q ≥ 0 , ( P π q − P π ⋆ ) Q ⋆ ≤ 0.
Consequently,
0 ≤ ( P π q − P π ⋆ ) q ≤ ( P π q − P π ⋆ ) ( q − Q ⋆ ) . (27) 0 \le(P^{\pi_q}-P^{\pi^\star})q \le(P^{\pi_q}-P^{\pi^\star})(q-Q^\star). \tag*{(27)} 0 ≤ ( P π q − P π ⋆ ) q ≤ ( P π q − P π ⋆ ) ( q − Q ⋆ ) . (27)
We first consider Assumption A.1. Combining (A.1) and (A.2), and writing μ max : = max s , a μ ( s , a ) \mu_{\max}:=\max_{s,a}\mu(s,a) μ m a x := max s , a μ ( s , a ) , we obtain
∥ φ ( δ ) ∥ ∞ ≤ γ μ max ∥ ( P π q − P π ⋆ ) ( q − Q ⋆ ) ∥ ∞ ≤ γ μ max C ∥ q − Q ⋆ ∥ ∞ 2 = γ μ max C ∥ δ ∥ ∞ 2 . \begin{aligned}
\|\varphi(\delta)\|_\infty
&\le\gamma\mu_{\max}\left\|(P^{\pi_q}-P^{\pi^\star})(q-Q^\star)\right\|_\infty\\
&\le\gamma\mu_{\max}C\|q-Q^\star\|_\infty^2 \\
&= \gamma\mu_{\max}C\|\delta\|_\infty^2.
\end{aligned} ∥ φ ( δ ) ∥ ∞ ≤ γ μ m a x ( P π q − P π ⋆ ) ( q − Q ⋆ ) ∞ ≤ γ μ m a x C ∥ q − Q ⋆ ∥ ∞ 2 = γ μ m a x C ∥ δ ∥ ∞ 2 .
Thus Assumption 3.1 holds with L = γ μ max C L=\gamma\mu_{\max}C L = γ μ m a x C . We next consider Assumption A.2. Suppose first that ∥ δ ∥ ∞ < κ 2 \|\delta\|_\infty< \frac{\kappa}{2} ∥ δ ∥ ∞ < 2 κ . For every state s s s and every a ≠ π ⋆ ( s ) a \ne\pi^\star(s) a = π ⋆ ( s ) ,
q ( s , π ⋆ ( s ) ) − q ( s , a ) = Q ⋆ ( s , π ⋆ ( s ) ) − Q ⋆ ( s , a ) + δ ( s , π ⋆ ( s ) ) − δ ( s , a ) ≥ κ − 2 ∥ δ ∥ ∞ > 0. q(s,\pi^\star(s))-q(s,a) = Q^\star(s,\pi^\star(s))-Q^\star(s,a)+\delta(s,\pi^\star(s))-\delta(s,a) \ge\kappa-2\|\delta\|_\infty> 0. q ( s , π ⋆ ( s )) − q ( s , a ) = Q ⋆ ( s , π ⋆ ( s )) − Q ⋆ ( s , a ) + δ ( s , π ⋆ ( s )) − δ ( s , a ) ≥ κ − 2∥ δ ∥ ∞ > 0.
Hence π q = π ⋆ \pi_q=\pi^\star π q = π ⋆ , and (A.1) gives
φ ( δ ) = 0 if ∥ δ ∥ ∞ < κ 2 . (28) \varphi(\delta)=0 \quad\text{if } \|\delta\|_\infty< \frac{\kappa}{2}. \tag*{(28)} φ ( δ ) = 0 if ∥ δ ∥ ∞ < 2 κ . (28)
Recall that V ( q ) ( s ) = max a q ( s , a ) V(q)(s) = \max_a q(s, a) V ( q ) ( s ) = max a q ( s , a ) . Since V ( Q ⋆ ) ( s ) = Q ⋆ ( s , π ⋆ ( s ) ) V(Q^\star)(s) = Q^\star(s, \pi^\star(s)) V ( Q ⋆ ) ( s ) = Q ⋆ ( s , π ⋆ ( s )) , we have
V ( Q ⋆ + δ ) ( s ) − V ( Q ⋆ ) ( s ) − δ ( s , π ⋆ ( s ) ) = max a { Q ⋆ ( s , a ) + δ ( s , a ) } − Q ⋆ ( s , π ⋆ ( s ) ) − δ ( s , π ⋆ ( s ) ) V(Q^\star+ \delta)(s) - V(Q^\star)(s) - \delta(s, \pi^\star(s))
= \max_a\{Q^\star(s, a) + \delta(s, a)\} - Q^\star(s, \pi^\star(s)) - \delta(s, \pi^\star(s)) V ( Q ⋆ + δ ) ( s ) − V ( Q ⋆ ) ( s ) − δ ( s , π ⋆ ( s )) = a max { Q ⋆ ( s , a ) + δ ( s , a )} − Q ⋆ ( s , π ⋆ ( s )) − δ ( s , π ⋆ ( s ))
for an arbitrary perturbation δ \delta δ . The expression on the right is nonnegative, since the maximum includes π ⋆ ( s ) \pi^\star(s) π ⋆ ( s ) . Moreover, by the optimality of π ⋆ \pi^\star π ⋆ ,
V ( Q ⋆ + δ ) ( s ) − V ( Q ⋆ ) ( s ) − δ ( s , π ⋆ ( s ) ) ≤ max a δ ( s , a ) − δ ( s , π ⋆ ( s ) ) ≤ 2 ∥ δ ∥ ∞ . V(Q^\star+ \delta)(s) - V(Q^\star)(s) - \delta(s, \pi^\star(s)) \le\max_a \delta(s, a) - \delta(s, \pi^\star(s)) \le2\lVert\delta\rVert_\infty. V ( Q ⋆ + δ ) ( s ) − V ( Q ⋆ ) ( s ) − δ ( s , π ⋆ ( s )) ≤ a max δ ( s , a ) − δ ( s , π ⋆ ( s )) ≤ 2 ∥ δ ∥ ∞ .
Using the explicit representation of φ ( δ ) \varphi(\delta) φ ( δ ) , we therefore obtain
∥ φ ( δ ) ∥ ∞ ≤ 2 γ μ max ∥ δ ∥ ∞ . (29) \lVert\varphi(\delta)\rVert_\infty\le2\gamma\mu_{\max}\lVert\delta\rVert_\infty. \tag*{(29)} ∥ φ ( δ ) ∥ ∞ ≤ 2 γ μ m a x ∥ δ ∥ ∞ . (29)
Combining (A.3) and (A.4), we conclude that
∥ φ ( δ ) ∥ ∞ ≤ 2 γ μ max ∥ δ ∥ ∞ 1 { ∥ δ ∥ ∞ ≥ κ 2 } ≤ 4 γ μ max κ ∥ δ ∥ ∞ 2 . \lVert\varphi(\delta)\rVert_\infty\le2\gamma\mu_{\max}\lVert\delta\rVert_\infty\mathbf{1}\left\{\lVert\delta\rVert_\infty\ge\frac{\kappa}{2}\right\} \le\frac{4\gamma\mu_{\max}}{\kappa}\lVert\delta\rVert_\infty^2. ∥ φ ( δ ) ∥ ∞ ≤ 2 γ μ m a x ∥ δ ∥ ∞ 1 { ∥ δ ∥ ∞ ≥ 2 κ } ≤ κ 4 γ μ m a x ∥ δ ∥ ∞ 2 .
Thus Assumption 3.1 holds with L = 4 γ μ max / κ L = 4\gamma\mu_{\max}/\kappa L = 4 γ μ m a x / κ . If there is only one action, the Bellman operator is affine and φ \varphi φ vanishes identically, and the assumption holds with L = 0 L = 0 L = 0 .
Lemma A.2. Suppose that S \mathcal{S} S and A \mathcal{A} A are finite, ∣ A ∣ ≥ 2 |\mathcal{A}| \ge2 ∣ A ∣ ≥ 2 , 0 < γ < 1 0 < \gamma< 1 0 < γ < 1 , H > 0 H > 0 H > 0 , and Q ⋆ ∈ [ 0 , H ] d Q^\star\in[0,H]^d Q ⋆ ∈ [ 0 , H ] d . Let D = diag ( μ ) D = \operatorname{diag}(\mu) D = diag ( μ ) , where μ ( s , a ) ≥ 0 \mu(s,a) \ge0 μ ( s , a ) ≥ 0 , and assume that
∑ x ∈ S ∑ b ∈ A μ ( x , b ) P ( s ∣ x , b ) > 0 for every s ∈ S . (30) \sum_{x \in\mathcal{S}}\sum_{b \in\mathcal{A}} \mu(x,b)P(s \mid x,b) > 0 \qquad\text{for every }s \in\mathcal{S}. \tag*{(30)} x ∈ S ∑ b ∈ A ∑ μ ( x , b ) P ( s ∣ x , b ) > 0 for every s ∈ S . (30) Fix a deterministic optimal policy π ⋆ \pi^\star π ⋆ . For each q q q , let π q \pi_q π q be a deterministic greedy policy with respect to q q q . Then, Assumption A.1, Assumption A.2, and Assumption 3.1 are equivalent. In particular, one may take C = 4 κ C = \frac{4}{\kappa} C = κ 4 , L = 4 γ μ max κ L = \frac{4\gamma\mu_{\max}}{\kappa} L = κ 4 γ μ m a x , μ max : = max s , a μ ( s , a ) \mu_{\max} := \max_{s,a}\mu(s,a) μ m a x := max s , a μ ( s , a ) for Assumption A.2.
Proof. Recall that
B ( q ) = D ( T q − q ) , A = D ( I − γ P π ⋆ ) , φ ( δ ) = B ( Q ⋆ + δ ) + A δ . B(q) = D(Tq - q), \qquad A = D(I - \gamma P^{\pi^\star}), \qquad\varphi(\delta) = B(Q^\star+ \delta) + A\delta. B ( q ) = D ( T q − q ) , A = D ( I − γ P π ⋆ ) , φ ( δ ) = B ( Q ⋆ + δ ) + A δ .
For q = Q ⋆ + δ q = Q^\star+ \delta q = Q ⋆ + δ , the Bellman fixed-point equation gives
φ ( δ ) = D ( T q − T Q ⋆ − γ P π ⋆ δ ) = γ D ( P π q − P π ⋆ ) q . (31) \varphi(\delta) = D(Tq - TQ^\star- \gamma P^{\pi^\star}\delta) = \gamma D(P^{\pi_q} - P^{\pi^\star})q. \tag*{(31)} φ ( δ ) = D ( T q − T Q ⋆ − γ P π ⋆ δ ) = γ D ( P π q − P π ⋆ ) q . (31)
We first prove Assumption A.2 ⇒ \Rightarrow ⇒ Assumption A.1. Write ρ = ∥ q − Q ⋆ ∥ ∞ \rho= \lVert q - Q^\star\rVert_\infty ρ = ∥ q − Q ⋆ ∥ ∞ . If ρ < κ / 2 \rho< \kappa/2 ρ < κ /2 , then, for every state s s s and every a ≠ π ⋆ ( s ) a \ne\pi^\star(s) a = π ⋆ ( s ) ,
q ( s , π ⋆ ( s ) ) − q ( s , a ) ≥ Q ⋆ ( s , π ⋆ ( s ) ) − Q ⋆ ( s , a ) − 2 ρ ≥ κ − 2 ρ > 0. q(s,\pi^\star(s)) - q(s,a) \ge Q^\star(s,\pi^\star(s)) - Q^\star(s,a) - 2\rho\ge\kappa- 2\rho> 0. q ( s , π ⋆ ( s )) − q ( s , a ) ≥ Q ⋆ ( s , π ⋆ ( s )) − Q ⋆ ( s , a ) − 2 ρ ≥ κ − 2 ρ > 0.
Thus π q = π ⋆ \pi_q = \pi^\star π q = π ⋆ , and
( P π q − P π ⋆ ) ( q − Q ⋆ ) = 0. (P^{\pi_q} - P^{\pi^\star})(q - Q^\star) = 0. ( P π q − P π ⋆ ) ( q − Q ⋆ ) = 0.
If ρ ≥ κ / 2 \rho\ge\kappa/2 ρ ≥ κ /2 , the stochasticity of P π q P^{\pi_q} P π q and P π ⋆ P^{\pi^\star} P π ⋆ implies
∥ ( P π q − P π ⋆ ) ( q − Q ⋆ ) ∥ ∞ ≤ 2 ρ ≤ 4 κ ρ 2 . \left\lVert(P^{\pi_q} - P^{\pi^\star})(q - Q^\star)\right\rVert_\infty\le2\rho\le\frac{4}{\kappa}\rho^2. ( P π q − P π ⋆ ) ( q − Q ⋆ ) ∞ ≤ 2 ρ ≤ κ 4 ρ 2 .
Therefore Assumption A.1 holds with C = 4 / κ C = 4/\kappa C = 4/ κ .
We next prove Assumption A.1 ⇒ \Rightarrow ⇒ Assumption 3.1. The greediness of π q \pi_q π q and the optimality of π ⋆ \pi^\star π ⋆ imply, componentwise,
( P π q − P π ⋆ ) q ≥ 0 , ( P π q − P π ⋆ ) Q ⋆ ≤ 0. (P^{\pi_q}-P^{\pi^\star})q \ge0,\qquad(P^{\pi_q}-P^{\pi^\star})Q^\star\le0. ( P π q − P π ⋆ ) q ≥ 0 , ( P π q − P π ⋆ ) Q ⋆ ≤ 0.
Consequently,
0 ≤ ( P π q − P π ⋆ ) q ≤ ( P π q − P π ⋆ ) ( q − Q ⋆ ) . 0 \le(P^{\pi_q}-P^{\pi^\star})q \le(P^{\pi_q}-P^{\pi^\star})(q-Q^\star). 0 ≤ ( P π q − P π ⋆ ) q ≤ ( P π q − P π ⋆ ) ( q − Q ⋆ ) .
Combining this inequality with (A.6) yields
∥ φ ( δ ) ∥ ∞ ≤ γ μ max ∥ ( P π q − P π ⋆ ) ( q − Q ⋆ ) ∥ ∞ ≤ γ μ max C ∥ δ ∥ ∞ 2 . \|\varphi(\delta)\|_\infty\le\gamma\mu_{\max}\|(P^{\pi_q}-P^{\pi^\star})(q-Q^\star)\|_\infty\le\gamma\mu_{\max}C\|\delta\|_\infty^2. ∥ φ ( δ ) ∥ ∞ ≤ γ μ m a x ∥ ( P π q − P π ⋆ ) ( q − Q ⋆ ) ∥ ∞ ≤ γ μ m a x C ∥ δ ∥ ∞ 2 .
Thus Assumption 3.1 holds with L = γ μ max C L=\gamma\mu_{\max}C L = γ μ m a x C .
It remains to prove Assumption 3.1 ⇒ \Rightarrow ⇒ Assumption A.2. For any feasible perturbation δ \delta δ , define
h δ ( s ) : = max a ∈ A { Q ⋆ ( s , a ) + δ ( s , a ) } − Q ⋆ ( s , π ⋆ ( s ) ) − δ ( s , π ⋆ ( s ) ) . h_\delta(s):=\max_{a\in\mathcal{A}}\{Q^\star(s,a)+\delta(s,a)\}-Q^\star(s,\pi^\star(s))-\delta(s,\pi^\star(s)). h δ ( s ) := a ∈ A max { Q ⋆ ( s , a ) + δ ( s , a )} − Q ⋆ ( s , π ⋆ ( s )) − δ ( s , π ⋆ ( s )) .
By the Bellman equation,
φ ( δ ) ( x , b ) = γ μ ( x , b ) ∑ s ∈ S P ( s ∣ x , b ) h δ ( s ) . (32) \varphi(\delta)(x,b)=\gamma\mu(x,b)\sum_{s\in\mathcal{S}}P(s\mid x,b)h_\delta(s). \tag*{(32)} φ ( δ ) ( x , b ) = γ μ ( x , b ) s ∈ S ∑ P ( s ∣ x , b ) h δ ( s ) . (32)
Suppose, towards a contradiction, that some state s 0 s_0 s 0 has an optimal action a 1 ≠ a 0 : = π ⋆ ( s 0 ) a_1\ne a_0:=\pi^\star(s_0) a 1 = a 0 := π ⋆ ( s 0 ) . Write
v : = Q ⋆ ( s 0 , a 0 ) = Q ⋆ ( s 0 , a 1 ) . v:=Q^\star(s_0,a_0)=Q^\star(s_0,a_1). v := Q ⋆ ( s 0 , a 0 ) = Q ⋆ ( s 0 , a 1 ) .
Let e s , a e_{s,a} e s , a denote the coordinate vector corresponding to ( s , a ) (s,a) ( s , a ) . For sufficiently small t > 0 t>0 t > 0 , define
δ t = { t e s 0 , a 1 , v < H , − t e s 0 , a 0 , v = H . \delta_t=
\begin{cases}
te_{s_0,a_1}, & v<H,\\
-te_{s_0,a_0}, & v=H.
\end{cases} δ t = { t e s 0 , a 1 , − t e s 0 , a 0 , v < H , v = H .
In the first case, take 0 < t < H − v 0<t<H-v 0 < t < H − v ; in the second, take 0 < t < H 0<t<H 0 < t < H . Then Q ⋆ + δ t ∈ [ 0 , H ] d Q^\star+\delta_t\in[0,H]^d Q ⋆ + δ t ∈ [ 0 , H ] d and ∥ δ t ∥ ∞ = t \|\delta_t\|_\infty=t ∥ δ t ∥ ∞ = t . In the first case, increasing the value of a 1 a_1 a 1 raises the maximum at s 0 s_0 s 0 by t t t , while leaving the value of a 0 a_0 a 0 unchanged. In the second case, decreasing the value of a 0 a_0 a 0 leaves the maximum unchanged, since a 1 a_1 a 1 remains optimal at value H H H . Hence, in both cases,
h δ t ( s ) = { t , s = s 0 , 0 , s ≠ s 0 . h_{\delta_t}(s)=
\begin{cases}
t, & s=s_0,\\
0, & s\ne s_0.
\end{cases} h δ t ( s ) = { t , 0 , s = s 0 , s = s 0 .
By (A.5), there exists ( x , b ) (x,b) ( x , b ) such that
μ ( x , b ) P ( s 0 ∣ x , b ) > 0. \mu(x,b)P(s_0\mid x,b)>0. μ ( x , b ) P ( s 0 ∣ x , b ) > 0.
Using (A.7) and Assumption 3.1, we obtain
γ μ ( x , b ) P ( s 0 ∣ x , b ) t ≤ ∥ φ ( δ t ) ∥ ∞ ≤ L t 2 . \gamma\mu(x,b)P(s_0\mid x,b)t\le\|\varphi(\delta_t)\|_\infty\le Lt^2. γ μ ( x , b ) P ( s 0 ∣ x , b ) t ≤ ∥ φ ( δ t ) ∥ ∞ ≤ L t 2 .
Dividing by t t t and letting t ↓ 0 t\downarrow0 t ↓ 0 gives
γ μ ( x , b ) P ( s 0 ∣ x , b ) ≤ 0 , \gamma\mu(x,b)P(s_0\mid x,b)\le0, γ μ ( x , b ) P ( s 0 ∣ x , b ) ≤ 0 ,
a contradiction. Therefore, every state has a unique optimal action. Since the state and action spaces are finite, the minimum optimal action gap is strictly positive. This proves Assumption A.2 and completes the proof.
Proof of Theorem 3.1
In this section, we prove Theorem 3.1. The proof of Theorem 3.1 is built on Proposition B.1 and Proposition B.2 . Proposition B.1 proves upper bounds on the error remainder term R n R_n R n and the finite-sample covariance discrepancy ∥ C n − Σ ∞ ∥ F \lVert C_n-\Sigma_\infty\rVert_F ∥ C n − Σ ∞ ∥ F . Proposition B.2 provides an upper bound on the Gaussian approximation of M n M_n M n . Section B.3 contains two problem-specific bounds. Lemma B.1 provides moment bounds for the iterate error, showing that ∥ Δ t ∥ L p ( ℓ ∞ ) ≤ C p α t \lVert\Delta_t\rVert_{L^p(\ell_\infty)} \le C\sqrt{p}\alpha^t ∥ Δ t ∥ L p ( ℓ ∞ ) ≤ C p α t , which is essential for controlling the Bellman nonlinearity and Markovian noise. Lemma B.2 establishes uniform boundedness, bounded total variation, and convergence properties of the Polyak–Ruppert averaging weights W n , t W_{n,t} W n , t . Section B.4 provides supporting lemmas, which are mainly probability tools.
Theorem 3.1. Under Assumptions 2.1, 3.1, and 3.2, let Z ∞ ∼ N ( 0 , Σ ∞ ) Z_\infty\sim\mathcal{N}(0,\Sigma_\infty) Z ∞ ∼ N ( 0 , Σ ∞ ) . For every n ≥ 2 n \ge2 n ≥ 2 ,
W 1 ( X n , Z ∞ ) ≤ C { n − 1 / 4 + n β − 1 + n 1 / 2 − β } . (33) W_1(X_n,Z_\infty) \le C\{n^{-1/4}+n^{\beta-1}+n^{1/2-\beta}\}. \tag*{(33)} W 1 ( X n , Z ∞ ) ≤ C { n − 1/4 + n β − 1 + n 1/2 − β } . (33) In particular, β = 3 / 4 \beta=3/4 β = 3/4 gives C n − 1 / 4 Cn^{-1/4} C n − 1/4 .
Proof of Theorem 3.1. We first compare the finite-sample covariance with the limiting covariance. Denote λ ∞ = λ min ( Σ ∞ ) > 0 \lambda_\infty=\lambda_{\min}(\Sigma_\infty)>0 λ ∞ = λ m i n ( Σ ∞ ) > 0 . By Proposition B.1, there exists a constant C c o v C_{\mathrm{cov}} C cov such that
∥ C n − Σ ∞ ∥ F ≤ C c o v n β − 1 . \lVert C_n-\Sigma_\infty\rVert_F \le C_{\mathrm{cov}}n^{\beta-1}. ∥ C n − Σ ∞ ∥ F ≤ C cov n β − 1 .
Since β < 1 \beta<1 β < 1 , we can choose n 0 n_0 n 0 such that the above bound is at most λ ∞ / 2 \lambda_\infty/2 λ ∞ /2 for n ≥ n 0 n\ge n_0 n ≥ n 0 and Proposition B.2 holds in this range. Using the fact that the operator norm is bounded by the Frobenius norm, for every unit vector u u u , we have
u T C n u ≥ u T Σ ∞ u − ∥ C n − Σ ∞ ∥ o p ≥ λ ∞ 2 . u^{\mathsf{T}}C_nu \ge u^{\mathsf{T}}\Sigma_\infty u-\lVert C_n-\Sigma_\infty\rVert_{\mathrm{op}} \ge\frac{\lambda_\infty}{2}. u T C n u ≥ u T Σ ∞ u − ∥ C n − Σ ∞ ∥ op ≥ 2 λ ∞ .
Thus, C n C_n C n is positive definite for all n ≥ n 0 n\ge n_0 n ≥ n 0 , and its smallest eigenvalue is bounded below by a constant independent of n n n . By Lemma B.11 , we obtain
W 1 ( Z n , Z ∞ ) ≤ ∥ C n − Σ ∞ ∥ F λ min ( C n ) + λ min ( Σ ∞ ) ≤ C c o v n β − 1 λ ∞ / 2 + λ ∞ ≤ C n β − 1 . (34) W_1(Z_n,Z_\infty) \le\frac{\lVert C_n-\Sigma_\infty\rVert_F}{\sqrt{\lambda_{\min}(C_n)}+\sqrt{\lambda_{\min}(\Sigma_\infty)}}
\le\frac{C_{\mathrm{cov}}n^{\beta-1}}{\sqrt{\lambda_\infty/2}+\sqrt{\lambda_\infty}}
\le Cn^{\beta-1}. \tag*{(34)} W 1 ( Z n , Z ∞ ) ≤ λ m i n ( C n ) + λ m i n ( Σ ∞ ) ∥ C n − Σ ∞ ∥ F ≤ λ ∞ /2 + λ ∞ C cov n β − 1 ≤ C n β − 1 . (34)
Next, recall that X n = M n + R n X_n=M_n+R_n X n = M n + R n holds on the original probability space. We use this decomposition to couple X n X_n X n and M n M_n M n when bounding W 1 ( X n , M n ) W_1(X_n,M_n) W 1 ( X n , M n ) . By the comparison between the Euclidean and maximum norms and Cauchy–Schwarz, we have
W 1 ( X n , M n ) ≤ E ∥ X n − M n ∥ 2 = E ∥ R n ∥ 2 ≤ d E ∥ R n ∥ ∞ ≤ d ( E ∥ R n ∥ ∞ 2 ) 1 / 2 = d ∥ R n ∥ L 2 ( ℓ ∞ ) . (35) \begin{aligned}
W_1(X_n,M_n) &\le\mathbb{E}\lVert X_n-M_n\rVert_2=\mathbb{E}\lVert R_n\rVert_2\\
&\le\sqrt{d}\,\mathbb{E}\lVert R_n\rVert_\infty\le\sqrt{d}\,(\mathbb{E}\lVert R_n\rVert_\infty^2)^{1/2}=\sqrt{d}\,\lVert R_n\rVert_{L^2(\ell_\infty)}. \tag*{(35)}
\end{aligned} W 1 ( X n , M n ) ≤ E ∥ X n − M n ∥ 2 = E ∥ R n ∥ 2 ≤ d E ∥ R n ∥ ∞ ≤ d ( E ∥ R n ∥ ∞ 2 ) 1/2 = d ∥ R n ∥ L 2 ( ℓ ∞ ) . (35)
Using (B.4) at p = 2 p=2 p = 2 , we obtain
W 1 ( X n , M n ) ≤ C { n − β / 2 + n 1 / 2 − β + ℓ n n − 1 / 2 } . W_1(X_n,M_n)\le C\{n^{-\beta/2}+n^{1/2-\beta}+\ell_n n^{-1/2}\}. W 1 ( X n , M n ) ≤ C { n − β /2 + n 1/2 − β + ℓ n n − 1/2 } .
Combining the triangle inequality, Proposition B.2 , and (B.1), we obtain
W 1 ( X n , Z ∞ ) ≤ W 1 ( X n , M n ) + W 1 ( M n , Z n ) + W 1 ( Z n , Z ∞ ) ≤ C { n − β / 2 + n 1 / 2 − β + ℓ n n − 1 / 2 + ℓ n n − 1 / 4 + n β − 1 } . (36) \begin{aligned}
W_1(X_n,Z_\infty)&\le W_1(X_n,M_n)+W_1(M_n,Z_n)+W_1(Z_n,Z_\infty)\\
&\le C\{n^{-\beta/2}+n^{1/2-\beta}+\ell_n n^{-1/2}+\ell_n n^{-1/4}+n^{\beta-1}\}. \tag*{(36)}
\end{aligned} W 1 ( X n , Z ∞ ) ≤ W 1 ( X n , M n ) + W 1 ( M n , Z n ) + W 1 ( Z n , Z ∞ ) ≤ C { n − β /2 + n 1/2 − β + ℓ n n − 1/2 + ℓ n n − 1/4 + n β − 1 } . (36)
Since n ≥ 1 n \ge1 n ≥ 1 , β > 1 / 2 \beta> 1/2 β > 1/2 , and ℓ n ≥ 1 \ell_n \ge1 ℓ n ≥ 1 , we have
n − β / 2 ≤ n − 1 / 4 , ℓ n n − 1 / 2 ≤ ℓ n n − 1 / 4 , n 1 / 2 − β + n β − 1 ≤ ℓ n ( n 1 / 2 − β + n β − 1 ) . n^{-\beta/2} \le n^{-1/4}, \qquad\ell_n n^{-1/2} \le\ell_n n^{-1/4}, \qquad n^{1/2-\beta} + n^{\beta-1} \le\ell_n(n^{1/2-\beta} + n^{\beta-1}). n − β /2 ≤ n − 1/4 , ℓ n n − 1/2 ≤ ℓ n n − 1/4 , n 1/2 − β + n β − 1 ≤ ℓ n ( n 1/2 − β + n β − 1 ) .
Substituting these inequalities into (B.3) proves (S.7) for n ≥ n 0 n \ge n_0 n ≥ n 0 . We now consider the remaining finitely many horizons. Since Q k , Q ⋆ ∈ [ 0 , H ] d Q_k, Q^\star\in[0,H]^d Q k , Q ⋆ ∈ [ 0 , H ] d , we have ∥ X n ∥ 2 ≤ H d n \lVert X_n\rVert_2 \le H\sqrt{dn} ∥ X n ∥ 2 ≤ H d n . Also, E ∥ Z ∞ ∥ 2 ≤ tr ( Σ ∞ ) \mathbb{E}\lVert Z_\infty\rVert_2 \le\sqrt{\operatorname{tr}(\Sigma_\infty)} E ∥ Z ∞ ∥ 2 ≤ tr ( Σ ∞ ) . Taking an independent coupling gives
W 1 ( X n , Z ∞ ) ≤ H d n + tr ( Σ ∞ ) . W_1(X_n,Z_\infty) \le H\sqrt{dn}+\sqrt{\operatorname{tr}(\Sigma_\infty)}. W 1 ( X n , Z ∞ ) ≤ H d n + tr ( Σ ∞ ) .
Denote a n = ℓ n ( n − 1 / 4 + n β − 1 + n 1 / 2 − β ) a_n=\ell_n(n^{-1/4}+n^{\beta-1}+n^{1/2-\beta}) a n = ℓ n ( n − 1/4 + n β − 1 + n 1/2 − β ) . Since this quantity is positive, the maximum of [ H d n + tr ( Σ ∞ ) ] / a n [H\sqrt{dn}+\sqrt{\operatorname{tr}(\Sigma_\infty)}]/a_n [ H d n + tr ( Σ ∞ ) ] / a n over the finite set 2 ≤ n < n 0 2\le n<n_0 2 ≤ n < n 0 is finite. Increasing C C C to include this maximum proves the result for all the remaining horizons. Finally, when β = 3 / 4 \beta=3/4 β = 3/4 , both n β − 1 n^{\beta-1} n β − 1 and n 1 / 2 − β n^{1/2-\beta} n 1/2 − β are equal to n − 1 / 4 n^{-1/4} n − 1/4 . For fixed d , b d,b d , b , we have ℓ n = O ( log n ) \ell_n=O(\log n) ℓ n = O ( log n ) , which gives the stated rate and completes the proof.
Proof of Proposition B.1
Proposition B.1. There is an exact decomposition X n = M n + R n X_n=M_n+R_n X n = M n + R n . For n ≥ 1 n\ge1 n ≥ 1 and p ≥ 2 p\ge2 p ≥ 2 ,
∥ R n ∥ L p ( ℓ ∞ ) ≤ C { p 3 / 2 n − β / 2 + p n 1 / 2 − β + ℓ n n − 1 / 2 } , ∥ C n − Σ ∞ ∥ F ≤ C n β − 1 . (37) \lVert R_n\rVert_{L^p(\ell_\infty)} \le C\{p^{3/2}n^{-\beta/2}+pn^{1/2-\beta}+\ell_n n^{-1/2}\}, \qquad\lVert C_n-\Sigma_\infty\rVert_F \le Cn^{\beta-1}. \tag*{(37)} ∥ R n ∥ L p ( ℓ ∞ ) ≤ C { p 3/2 n − β /2 + p n 1/2 − β + ℓ n n − 1/2 } , ∥ C n − Σ ∞ ∥ F ≤ C n β − 1 . (37)
Proof. We first derive the decomposition of the averaged error. We then bound the initialization term, the nonlinear term, and the centered Markov term using Lemmas B.1 and B.2. The covariance bound follows from the same deterministic weight estimates.
For a deterministic δ \delta δ in E = { δ : Q ⋆ + δ ∈ [ 0 , H ] d } \mathcal{E}=\{\delta:Q^\star+\delta\in[0,H]^d\} E = { δ : Q ⋆ + δ ∈ [ 0 , H ] d } , define
U δ ( y ) = U ( Q ⋆ + δ , y ) − U ( Q ⋆ , y ) , f δ ( y ) = U δ ( y ) − E μ ˉ U δ ( Y ) . (38) U_\delta(y)=U(Q^\star+\delta,y)-U(Q^\star,y), \qquad f_\delta(y)=U_\delta(y)-\mathbb{E}_{\bar{\mu}}U_\delta(Y). \tag*{(38)} U δ ( y ) = U ( Q ⋆ + δ , y ) − U ( Q ⋆ , y ) , f δ ( y ) = U δ ( y ) − E μ ˉ U δ ( Y ) . (38)
Recall that ∥ V ( q ) − V ( q ′ ) ∥ ∞ ≤ ∥ q − q ′ ∥ ∞ \lVert V(q)-V(q')\rVert_\infty\le\lVert q-q'\rVert_\infty ∥ V ( q ) − V ( q ′ ) ∥ ∞ ≤ ∥ q − q ′ ∥ ∞ . By the definition of the sampled update, we have
sup y ∥ U ( q , y ) − U ( q ′ , y ) ∥ ∞ ≤ ( 1 + γ ) ∥ q − q ′ ∥ ∞ . \sup_y\lVert U(q,y)-U(q',y)\rVert_\infty\le(1+\gamma)\lVert q-q'\rVert_\infty. y sup ∥ U ( q , y ) − U ( q ′ , y ) ∥ ∞ ≤ ( 1 + γ ) ∥ q − q ′ ∥ ∞ .
Subtracting the stationary means on both sides, we obtain
sup y ∥ f δ ( y ) − f δ ′ ( y ) ∥ ∞ ≤ 2 ( 1 + γ ) ∥ δ − δ ′ ∥ ∞ . (39) \sup_y\lVert f_\delta(y)-f_{\delta'}(y)\rVert_\infty\le2(1+\gamma)\lVert\delta-\delta'\rVert_\infty. \tag*{(39)} y sup ∥ f δ ( y ) − f δ ′ ( y ) ∥ ∞ ≤ 2 ( 1 + γ ) ∥ δ − δ ′ ∥ ∞ . (39)
By definition, μ ˉ f δ = 0 \bar{\mu}f_\delta=0 μ ˉ f δ = 0 and f 0 = 0 f_0=0 f 0 = 0 . Thus, sup y ∥ f δ ( y ) ∥ ∞ ≤ C ∥ δ ∥ ∞ \sup_y\lVert f_\delta(y)\rVert_\infty\le C\lVert\delta\rVert_\infty sup y ∥ f δ ( y ) ∥ ∞ ≤ C ∥ δ ∥ ∞ . Since B ( Q ⋆ ) = 0 B(Q^\star)=0 B ( Q ⋆ ) = 0 , we also have
E μ ˉ U δ ( Y ) = B ( Q ⋆ + δ ) = − A δ + φ ( δ ) . \mathbb{E}_{\bar{\mu}}U_\delta(Y)=B(Q^\star+\delta)=-A\delta+\varphi(\delta). E μ ˉ U δ ( Y ) = B ( Q ⋆ + δ ) = − A δ + φ ( δ ) .
Substituting δ = Δ t \delta=\Delta_t δ = Δ t into (B.5) and using the update rule, we obtain
Δ t + 1 = ( I − α t A ) Δ t + α t { ϵ t + f Δ t ( Y t ) + φ ( Δ t ) } . (40) \Delta_{t+1}=(I-\alpha_t A)\Delta_t+\alpha_t\{\epsilon_t+f_{\Delta_t}(Y_t)+\varphi(\Delta_t)\}. \tag*{(40)} Δ t + 1 = ( I − α t A ) Δ t + α t { ϵ t + f Δ t ( Y t ) + φ ( Δ t )} . (40)
The terms φ \varphi φ and f Δ t f_{\Delta_t} f Δ t in this identity account for the full difference between the original update and the linear drift. Thus, the recursion above is exact.
Denote ζ t = ϵ t + f Δ t ( Y t ) + φ ( Δ t ) \zeta_t=\epsilon_t+f_{\Delta_t}(Y_t)+\varphi(\Delta_t) ζ t = ϵ t + f Δ t ( Y t ) + φ ( Δ t ) . By recursion in (B.7), we have
Δ k = Φ 0 : k − 1 Δ 0 + ∑ t = 0 k − 1 α t Φ t + 1 : k − 1 ζ t . \Delta_k=\Phi_{0:k-1}\Delta_0+\sum_{t=0}^{k-1}\alpha_t\Phi_{t+1:k-1}\zeta_t. Δ k = Φ 0 : k − 1 Δ 0 + t = 0 ∑ k − 1 α t Φ t + 1 : k − 1 ζ t .
Recall that X n = n − 1 / 2 ∑ k = 1 n Δ k X_n=n^{-1/2}\sum_{k=1}^{n}\Delta_k X n = n − 1/2 ∑ k = 1 n Δ k . Summing the above identity and interchanging the two finite sums gives
1 n ∑ k = 1 n ∑ t = 0 k − 1 α t Φ t + 1 : k − 1 ζ t = 1 n ∑ t = 0 n − 1 ( α t ∑ k = t + 1 n Φ t + 1 : k − 1 ) ζ t = 1 n ∑ t = 0 n − 1 W n , t ζ t . \frac{1}{\sqrt{n}}\sum_{k=1}^{n}\sum_{t=0}^{k-1}\alpha_t\Phi_{t+1:k-1}\zeta_t
=
\frac{1}{\sqrt{n}}\sum_{t=0}^{n-1}\left(\alpha_t\sum_{k=t+1}^{n}\Phi_{t+1:k-1}\right)\zeta_t
=
\frac{1}{\sqrt{n}}\sum_{t=0}^{n-1}W_{n,t}\zeta_t. n 1 k = 1 ∑ n t = 0 ∑ k − 1 α t Φ t + 1 : k − 1 ζ t = n 1 t = 0 ∑ n − 1 ( α t k = t + 1 ∑ n Φ t + 1 : k − 1 ) ζ t = n 1 t = 0 ∑ n − 1 W n , t ζ t .
Therefore, the decomposition holds with
R n = R n i n i t + R n n o n l i n + R n M a r k o v , (41) R_n=R_n^{\mathrm{init}}+R_n^{\mathrm{nonlin}}+R_n^{\mathrm{Markov}}, \tag*{(41)} R n = R n init + R n nonlin + R n Markov , (41)
R n i n i t = 1 n ∑ k = 1 n Φ 0 : k − 1 Δ 0 , (42) R_n^{\mathrm{init}}=\frac{1}{\sqrt{n}}\sum_{k=1}^{n}\Phi_{0:k-1}\Delta_0, \tag*{(42)} R n init = n 1 k = 1 ∑ n Φ 0 : k − 1 Δ 0 , (42)
R n n o n l i n = 1 n ∑ t < n W n , t φ ( Δ t ) , (43) R_n^{\mathrm{nonlin}}=\frac{1}{\sqrt{n}}\sum_{t<n}W_{n,t}\varphi(\Delta_t), \tag*{(43)} R n nonlin = n 1 t < n ∑ W n , t φ ( Δ t ) , (43)
R n M a r k o v = 1 n ∑ t < n W n , t f Δ t ( Y t ) . (44) R_n^{\mathrm{Markov}}=\frac{1}{\sqrt{n}}\sum_{t<n}W_{n,t}f\Delta_t(Y_t). \tag*{(44)} R n Markov = n 1 t < n ∑ W n , t f Δ t ( Y t ) . (44)
We bound the three terms in this decomposition separately. For the initialization term, integral comparison gives
∑ u = 0 k − 1 α u ≥ c 0 ∫ 0 k ( x + b ) − β d x = c 0 1 − β ( ( k + b ) 1 − β − b 1 − β ) . \sum_{u=0}^{k-1}\alpha_u\ge c_0\int_0^k(x+b)^{-\beta}\,dx=\frac{c_0}{1-\beta}\left((k+b)^{1-\beta}-b^{1-\beta}\right). u = 0 ∑ k − 1 α u ≥ c 0 ∫ 0 k ( x + b ) − β d x = 1 − β c 0 ( ( k + b ) 1 − β − b 1 − β ) .
By (B.79), we have ∥ Φ 0 : k − 1 ∥ ∞ ≤ C e − c k 1 − β \lVert\Phi_{0:k-1}\rVert_\infty\le Ce^{-ck^{1-\beta}} ∥ Φ 0 : k − 1 ∥ ∞ ≤ C e − c k 1 − β . To bound the sum ∑ k ≥ 1 e − c k 1 − β \sum_{k\ge1}e^{-ck^{1-\beta}} ∑ k ≥ 1 e − c k 1 − β , let r = 1 − β > 0 r=1-\beta>0 r = 1 − β > 0 . By a change of variable in the corresponding integral, we have
∫ 0 ∞ e − c x r d x = c − 1 / r r ∫ 0 ∞ e − z z 1 / r − 1 d z = c − 1 / r r Γ ( 1 / r ) < ∞ . \int_0^\infty e^{-cx^r}\,dx=\frac{c^{-1/r}}{r}\int_0^\infty e^{-z}z^{1/r-1}\,dz=\frac{c^{-1/r}}{r}\Gamma(1/r)<\infty. ∫ 0 ∞ e − c x r d x = r c − 1/ r ∫ 0 ∞ e − z z 1/ r − 1 d z = r c − 1/ r Γ ( 1/ r ) < ∞.
It follows by integral comparison that the series is bounded by a constant independent of the horizon. Since ∥ Δ 0 ∥ ∞ ≤ H \lVert\Delta_0\rVert_\infty\le H ∥ Δ 0 ∥ ∞ ≤ H , we obtain
∥ R n i n i t ∥ L p ( ℓ ∞ ) ≤ H n ∑ k = 1 ∞ ∥ Φ 0 : k − 1 ∥ ∞ ≤ C n − 1 / 2 . (45) \left\lVert R_n^{\mathrm{init}}\right\rVert_{L^p(\ell_\infty)}
\le\frac{H}{\sqrt{n}}\sum_{k=1}^{\infty}\lVert\Phi_{0:k-1}\rVert_\infty
\le Cn^{-1/2}. \tag*{(45)} R n init L p ( ℓ ∞ ) ≤ n H k = 1 ∑ ∞ ∥ Φ 0 : k − 1 ∥ ∞ ≤ C n − 1/2 . (45)
The above estimate holds deterministically, so its constant is independent of p p p . For the nonlinear term, by the triangle inequality in L p L^p L p , the uniform weight bound, and the quadratic remainder assumption, we have
∥ R n n o n l i n ∥ L p ( ℓ ∞ ) ≤ C n ∑ t < n ∥ ∥ Δ t ∥ ∞ 2 ∥ L p ≤ C p n 1 / 2 − β . (46) \left\lVert R_n^{\mathrm{nonlin}}\right\rVert_{L^p(\ell_\infty)}
\le\frac{C}{\sqrt{n}}\sum_{t<n}\left\lVert\lVert\Delta_t\rVert_\infty^2\right\rVert_{L^p}
\le Cp n^{1/2-\beta}. \tag*{(46)} R n nonlin L p ( ℓ ∞ ) ≤ n C t < n ∑ ∥ Δ t ∥ ∞ 2 L p ≤ C p n 1/2 − β . (46)
where the second inequality follows from (B.55). Thus, summing the quadratic remainders gives a contribution of order n 1 / 2 − β n^{1/2-\beta} n 1/2 − β .
We next consider the centered Markov term. For every fixed δ \delta δ , f δ f_\delta f δ is centered under μ ˉ \bar{\mu} μ ˉ . However, after substituting the iterate, its conditional mean is
E [ f Δ t ( Y t ) ∣ F t − 1 ] = ( K f Δ t ) ( Y t − 1 ) , \mathbb{E}[f_{\Delta_t}(Y_t) \mid\mathcal{F}_{t-1}] = (Kf_{\Delta_t})(Y_{t-1}), E [ f Δ t ( Y t ) ∣ F t − 1 ] = ( K f Δ t ) ( Y t − 1 ) ,
which is not necessarily zero. We therefore use the Poisson equation to express this term as a martingale difference and two remaining differences. For each δ ∈ E \delta\in\mathcal{E} δ ∈ E , define
h δ = ∑ j = 0 ∞ K j f δ , v δ = K h δ . (47) h_\delta= \sum_{j=0}^{\infty} K^j f_\delta,\qquad v_\delta= Kh_\delta. \tag*{(47)} h δ = j = 0 ∑ ∞ K j f δ , v δ = K h δ . (47)
Since μ ˉ f δ = 0 \bar{\mu}f_\delta= 0 μ ˉ f δ = 0 , f 0 = 0 f_0 = 0 f 0 = 0 , and (B.6) holds, Lemma B.9 implies that the series converges absolutely and satisfies
sup y ∥ h δ ( y ) ∥ ∞ ≤ C ∥ δ ∥ ∞ , (48) \sup_y \|h_\delta(y)\|_\infty\le C\|\delta\|_\infty, \tag*{(48)} y sup ∥ h δ ( y ) ∥ ∞ ≤ C ∥ δ ∥ ∞ , (48)
sup y ∥ h δ ( y ) − h δ ′ ( y ) ∥ ∞ ≤ C ∥ δ − δ ′ ∥ ∞ . (49) \sup_y \|h_\delta(y)-h_{\delta'}(y)\|_\infty\le C\|\delta-\delta'\|_\infty. \tag*{(49)} y sup ∥ h δ ( y ) − h δ ′ ( y ) ∥ ∞ ≤ C ∥ δ − δ ′ ∥ ∞ . (49)
Since a Markov kernel does not increase the sup norm, both bounds also hold for v δ v_\delta v δ . Applying K K K to the convergent series term by term, we obtain
h δ − K h δ = f δ . (50) h_\delta- Kh_\delta= f_\delta. \tag*{(50)} h δ − K h δ = f δ . (50)
The Lipschitz bounds imply measurability in the parameter, so these identities can also be evaluated at the random iterate Δ t \Delta_t Δ t . For t ≥ 1 t \ge1 t ≥ 1 , define
m t = h Δ t ( Y t ) − v Δ t ( Y t − 1 ) . (51) m_t = h_{\Delta_t}(Y_t)-v_{\Delta_t}(Y_{t-1}). \tag*{(51)} m t = h Δ t ( Y t ) − v Δ t ( Y t − 1 ) . (51)
Since Δ t \Delta_t Δ t is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, the Markov property gives
E [ h Δ t ( Y t ) ∣ F t − 1 ] = ∑ y h Δ t ( y ) K ( Y t − 1 , y ) = v Δ t ( Y t − 1 ) . \mathbb{E}[h_{\Delta_t}(Y_t)\mid\mathcal{F}_{t-1}]
= \sum_y h_{\Delta_t}(y)K(Y_{t-1},y)
= v_{\Delta_t}(Y_{t-1}). E [ h Δ t ( Y t ) ∣ F t − 1 ] = y ∑ h Δ t ( y ) K ( Y t − 1 , y ) = v Δ t ( Y t − 1 ) .
Thus, E [ m t ∣ F t − 1 ] = 0 \mathbb{E}[m_t\mid\mathcal{F}_{t-1}]=0 E [ m t ∣ F t − 1 ] = 0 . By (B.15), we also have ∥ m t ∥ ∞ ≤ C ∥ Δ t ∥ ∞ \|m_t\|_\infty\le C\|\Delta_t\|_\infty ∥ m t ∥ ∞ ≤ C ∥ Δ t ∥ ∞ . Substituting the definition into the Poisson identity gives
f Δ t ( Y t ) = m t + v Δ t ( Y t − 1 ) − v Δ t ( Y t ) . (52) f_{\Delta_t}(Y_t) = m_t + v_{\Delta_t}(Y_{t-1}) - v_{\Delta_t}(Y_t). \tag*{(52)} f Δ t ( Y t ) = m t + v Δ t ( Y t − 1 ) − v Δ t ( Y t ) . (52)
Since W n , t W_{n,t} W n , t is deterministic, each coordinate of W n , t m t W_{n,t}m_t W n , t m t is a martingale difference. Applying Lemma B.10 coordinatewise and using (B.75), we have
∥ 1 n ∑ t = 1 n − 1 W n , t m t ∥ L p ( ℓ ∞ ) ≤ C p n ( ∑ t = 1 n − 1 ∥ Δ t ∥ L p ( ℓ ∞ ) 2 ) 1 / 2 ≤ C p n ( C p n 1 − β ) 1 / 2 ≤ C p 3 / 2 n − β / 2 . (53) \left\|\frac{1}{\sqrt{n}}\sum_{t=1}^{n-1} W_{n,t}m_t\right\|_{L^p(\ell_\infty)}
\le\frac{C_p}{\sqrt{n}}\left(\sum_{t=1}^{n-1}\|\Delta_t\|_{L^p(\ell_\infty)}^2\right)^{1/2}
\le\frac{C_p}{\sqrt{n}}\left(Cpn^{1-\beta}\right)^{1/2}
\le Cp^{3/2}n^{-\beta/2}. \tag*{(53)} n 1 t = 1 ∑ n − 1 W n , t m t L p ( ℓ ∞ ) ≤ n C p ( t = 1 ∑ n − 1 ∥ Δ t ∥ L p ( ℓ ∞ ) 2 ) 1/2 ≤ n C p ( C p n 1 − β ) 1/2 ≤ C p 3/2 n − β /2 . (53)
The coordinatewise estimate changes the constant only through its dependence on d d d . We use the factor C p C_p C p in the martingale bound, although the scalar calculation also gives the factor C p C\sqrt{p} C p . The former is sufficient for the claimed remainder estimate.
For n ≥ 2 n \ge2 n ≥ 2 , denote g t = v Δ t g_t = v_{\Delta_t} g t = v Δ t . By summation by parts, we have
∑ t = 1 n − 1 W n , t { g t ( Y t − 1 ) − g t ( Y t ) } = W n , 1 g 1 ( Y 0 ) − W n , n − 1 g n − 1 ( Y n − 1 ) + ∑ t = 1 n − 2 { W n , t + 1 g t + 1 ( Y t ) − W n , t g t ( Y t ) } . (54) \begin{aligned}
\sum_{t=1}^{n-1} W_{n,t}\{g_t(Y_{t-1}) - g_t(Y_t)\}
&= W_{n,1}g_1(Y_0) - W_{n,n-1}g_{n-1}(Y_{n-1}) \\
&\quad+ \sum_{t=1}^{n-2}\{W_{n,t+1}g_{t+1}(Y_t) - W_{n,t}g_t(Y_t)\}. \tag*{(54)}
\end{aligned} t = 1 ∑ n − 1 W n , t { g t ( Y t − 1 ) − g t ( Y t )} = W n , 1 g 1 ( Y 0 ) − W n , n − 1 g n − 1 ( Y n − 1 ) + t = 1 ∑ n − 2 { W n , t + 1 g t + 1 ( Y t ) − W n , t g t ( Y t )} . (54)
Since ∥ Δ t ∥ ∞ ≤ H \|\Delta_t\|_\infty\le H ∥ Δ t ∥ ∞ ≤ H , both endpoint terms are bounded by C C C . For each interior term, we have the decomposition
W n , t + 1 g t + 1 ( Y t ) − W n , t g t ( Y t ) = ( W n , t + 1 − W n , t ) g t ( Y t ) + W n , t + 1 { g t + 1 ( Y t ) − g t ( Y t ) } . (55) \begin{aligned}
W_{n,t+1}g_{t+1}(Y_t) - W_{n,t}g_t(Y_t)
&= (W_{n,t+1} - W_{n,t})g_t(Y_t) \\
&\quad+ W_{n,t+1}\{g_{t+1}(Y_t) - g_t(Y_t)\}. \tag*{(55)}
\end{aligned} W n , t + 1 g t + 1 ( Y t ) − W n , t g t ( Y t ) = ( W n , t + 1 − W n , t ) g t ( Y t ) + W n , t + 1 { g t + 1 ( Y t ) − g t ( Y t )} . (55)
For the first term, by (B.76), we have
∑ t = 1 n − 2 ∥ ( W n , t + 1 − W n , t ) g t ( Y t ) ∥ ∞ ≤ C ∑ t = 1 n − 2 ∥ W n , t + 1 − W n , t ∥ ∞ ≤ C ℓ n . \sum_{t=1}^{n-2}\|(W_{n,t+1} - W_{n,t})g_t(Y_t)\|_\infty
\le C\sum_{t=1}^{n-2}\|W_{n,t+1} - W_{n,t}\|_\infty
\le C\ell_n. t = 1 ∑ n − 2 ∥ ( W n , t + 1 − W n , t ) g t ( Y t ) ∥ ∞ ≤ C t = 1 ∑ n − 2 ∥ W n , t + 1 − W n , t ∥ ∞ ≤ C ℓ n .
For the second term, using the Lipschitz bound for v δ v_\delta v δ and the one-step movement bound in (B.54), we obtain
sup y ∥ g t + 1 ( y ) − g t ( y ) ∥ ∞ ≤ C ∥ Δ t + 1 − Δ t ∥ ∞ ≤ C H α t . \sup_y \|g_{t+1}(y) - g_t(y)\|_\infty
\le C\|\Delta_{t+1} - \Delta_t\|_\infty
\le CH\alpha_t. y sup ∥ g t + 1 ( y ) − g t ( y ) ∥ ∞ ≤ C ∥ Δ t + 1 − Δ t ∥ ∞ ≤ C H α t .
Combining the endpoint estimates and the two bounds above, we obtain, for every sample path,
∥ ∑ t = 1 n − 1 W n , t { v Δ t ( Y t − 1 ) − v Δ t ( Y t ) } ∥ ∞ ≤ C ( 1 + ℓ n + n 1 − β ) . (56) \left\|\sum_{t=1}^{n-1} W_{n,t}\{v_{\Delta_t}(Y_{t-1}) - v_{\Delta_t}(Y_t)\}\right\|_\infty
\le C(1+\ell_n+n^{1-\beta}). \tag*{(56)} t = 1 ∑ n − 1 W n , t { v Δ t ( Y t − 1 ) − v Δ t ( Y t )} ∞ ≤ C ( 1 + ℓ n + n 1 − β ) . (56)
Dividing by n \sqrt{n} n and taking the L p L^p L p norm gives
C { ℓ n n − 1 / 2 + n 1 / 2 − β } . (57) C\{\ell_n n^{-1/2}+n^{1/2-\beta}\}. \tag*{(57)} C { ℓ n n − 1/2 + n 1/2 − β } . (57)
These bounds hold for every sample path and do not use conditional centering of the two remaining differences.
The Poisson decomposition was applied from t = 1 t=1 t = 1 onward, without introducing Y − 1 Y_{-1} Y − 1 . We bound the time-zero term directly as follows.
∥ 1 n W n , 0 f Δ 0 ( Y 0 ) ∥ ∞ ≤ C n ∥ Δ 0 ∥ ∞ ≤ C n − 1 / 2 . (58) \left\|\frac{1}{\sqrt{n}}W_{n,0}f_{\Delta_0}(Y_0)\right\|_\infty
\le\frac{C}{\sqrt{n}}\|\Delta_0\|_\infty
\le Cn^{-1/2}. \tag*{(58)} n 1 W n , 0 f Δ 0 ( Y 0 ) ∞ ≤ n C ∥ Δ 0 ∥ ∞ ≤ C n − 1/2 . (58)
Combining (B.19), (B.20), (B.24), and (B.25), we have
∥ R n M a r k o v ∥ L p ( ℓ ∞ ) ≤ C { p 3 / 2 n − β / 2 + n 1 / 2 − β + ℓ n n − 1 / 2 } . \|R_n^{\mathrm{Markov}}\|_{L^p(\ell_\infty)}
\le C\{p^{3/2}n^{-\beta/2}+n^{1/2-\beta}+\ell_n n^{-1/2}\}. ∥ R n Markov ∥ L p ( ℓ ∞ ) ≤ C { p 3/2 n − β /2 + n 1/2 − β + ℓ n n − 1/2 } .
Combining this estimate with (B.8), (B.12), and (B.13), we obtain
∥ R n ∥ L p ( ℓ ∞ ) ≤ C { p 3 / 2 n − β / 2 + ( p + 1 ) n 1 / 2 − β + ( ℓ n + 1 ) n − 1 / 2 } . \|R_n\|_{L^p(\ell_\infty)}
\le C\{p^{3/2}n^{-\beta/2}+(p+1)n^{1/2-\beta}+(\ell_n+1)n^{-1/2}\}. ∥ R n ∥ L p ( ℓ ∞ ) ≤ C { p 3/2 n − β /2 + ( p + 1 ) n 1/2 − β + ( ℓ n + 1 ) n − 1/2 } .
Using p ≥ 2 p \ge2 p ≥ 2 and ℓ n ≥ 1 \ell_n \ge1 ℓ n ≥ 1 , the above inequality gives the first inequality in (B.4). When n = 1 n=1 n = 1 , the martingale and coboundary sums are empty. The bounded initialization, nonlinear, and time-zero terms give the same inequality after increasing C C C .
We now prove the second inequality in (B.4). Denote T = A − 1 T=A^{-1} T = A − 1 . For any matrix B B B , we have
B S B T − T S T T = ( B − T ) S B T + T S ( B − T ) T . (59) BSB^{\mathsf{T}}-TST^{\mathsf{T}}=(B-T)SB^{\mathsf{T}}+TS(B-T)^{\mathsf{T}}. \tag*{(59)} B S B T − T S T T = ( B − T ) S B T + T S ( B − T ) T . (59)
Expanding the right-hand side verifies the identity, since the two cross terms cancel. Using ∥ X Y Z ∥ F ≤ ∥ X ∥ F ∥ Y ∥ o p ∥ Z ∥ o p \lVert XYZ\rVert_F\le\lVert X\rVert_F\lVert Y\rVert_{\mathrm{op}}\lVert Z\rVert_{\mathrm{op}} ∥ X Y Z ∥ F ≤ ∥ X ∥ F ∥ Y ∥ op ∥ Z ∥ op and the same inequality after transposing, we obtain
∥ B S B T − T S T T ∥ F ≤ ∥ S ∥ o p ( ∥ B ∥ o p + ∥ T ∥ o p ) ∥ B − T ∥ F . \left\lVert BSB^{\mathsf{T}}-TST^{\mathsf{T}}\right\rVert_F\le\lVert S\rVert_{\mathrm{op}}\left(\lVert B\rVert_{\mathrm{op}}+\lVert T\rVert_{\mathrm{op}}\right)\lVert B-T\rVert_F. B S B T − T S T T F ≤ ∥ S ∥ op ( ∥ B ∥ op + ∥ T ∥ op ) ∥ B − T ∥ F .
Taking B = W n , t B=W_{n,t} B = W n , t , the two operator norms are uniformly bounded by (B.75), (B.79), and norm equivalence in fixed dimension. Also, ∥ B − T ∥ F ≤ d ∥ B − T ∥ ∞ \lVert B-T\rVert_F\le\sqrt{d}\lVert B-T\rVert_\infty ∥ B − T ∥ F ≤ d ∥ B − T ∥ ∞ . Therefore, we have
∥ C n − Σ ∞ ∥ F ≤ 1 n ∑ t = 0 n − 1 ∥ W n , t S W n , t T − T S T T ∥ F ≤ C n ∑ t = 0 n − 1 ∥ W n , t − A − 1 ∥ ∞ ≤ C n β − 1 , \begin{aligned}
\lVert C_n-\Sigma_\infty\rVert_F
&\le\frac{1}{n}\sum_{t=0}^{n-1}\left\lVert W_{n,t}SW_{n,t}^{\mathsf{T}}-TST^{\mathsf{T}}\right\rVert_F\\
&\le\frac{C}{n}\sum_{t=0}^{n-1}\lVert W_{n,t}-A^{-1}\rVert_\infty\le Cn^{\beta-1},
\end{aligned} ∥ C n − Σ ∞ ∥ F ≤ n 1 t = 0 ∑ n − 1 W n , t S W n , t T − T S T T F ≤ n C t = 0 ∑ n − 1 ∥ W n , t − A − 1 ∥ ∞ ≤ C n β − 1 ,
where the last inequality follows from (B.77). This completes the proof.
Proof of Proposition B.2
Proposition B.2. For all sufficiently large n n n , if Z n ∼ N ( 0 , C n ) Z_n\sim\mathcal{N}(0,C_n) Z n ∼ N ( 0 , C n ) , then
W 1 ( M n , Z n ) ≤ C ℓ n n − 1 / 4 . (60) W_1(M_n,Z_n)\le C\ell_n n^{-1/4}. \tag*{(60)} W 1 ( M n , Z n ) ≤ C ℓ n n − 1/4 . (60)
Proof. By Lemma B.2 and the boundedness of the fixed-point noise, there exists a constant K 0 < ∞ K_0<\infty K 0 < ∞ such that
∥ W n , t ϵ t ∥ 2 ≤ K 0 , ∥ W n , t ϵ t ′ ∥ 2 ≤ K 0 (61) \lVert W_{n,t}\epsilon_t\rVert_2\le K_0,\qquad\lVert W_{n,t}\epsilon'_t\rVert_2\le K_0 \tag*{(61)} ∥ W n , t ϵ t ∥ 2 ≤ K 0 , ∥ W n , t ϵ t ′ ∥ 2 ≤ K 0 (61)
almost surely, uniformly in n , t n,t n , t . We first partition the trajectory into retained intervals separated by gaps. Let
q = ⌈ n ⌉ , m = ⌈ C 1 log ( e n ) ⌉ , (62) q=\lceil\sqrt{n}\rceil,\qquad m=\lceil C_1\log(en)\rceil, \tag*{(62)} q = ⌈ n ⌉ , m = ⌈ C 1 log ( e n )⌉ , (62)
where C 1 C_1 C 1 is a sufficiently large constant to be chosen below. For all sufficiently large n n n , we have m < q m<q m < q . We partition { 0 , … , n − 1 } \{0,\ldots,n-1\} { 0 , … , n − 1 } by first discarding m m m indices, then retaining the next q q q indices, discarding the next m m m indices, and retaining the next q q q indices. We continue alternating in this way. The last retained interval can contain fewer than q q q indices. All final indices that are not retained are included in the discarded set. Denote the retained intervals by I 1 , … , I N I_1,\ldots,I_N I 1 , … , I N and let
I = ⋃ j = 1 N I j , J = { 0 , … , n − 1 } ∖ I . I=\bigcup_{j=1}^{N}I_j,\qquad J=\{0,\ldots,n-1\}\setminus I. I = j = 1 ⋃ N I j , J = { 0 , … , n − 1 } ∖ I .
By this construction, the number of retained intervals is bounded by N ≤ C ( n / q + 1 ) N\le C(n/q+1) N ≤ C ( n / q + 1 ) . Each gap contains at most m m m indices, so the total number of discarded indices satisfies
∣ J ∣ ≤ C ( n / q + 1 ) m ≤ C n ℓ n . (63) \lvert J\rvert\le C(n/q+1)m\le C\sqrt{n}\ell_n. \tag*{(63)} ∣ J ∣ ≤ C ( n / q + 1 ) m ≤ C n ℓ n . (63)
Define
B j = 1 n ∑ t ∈ I j W n , t ε t , G n g a p = 1 n ∑ t ∈ J W n , t ε t . (64) B_j=\frac{1}{\sqrt{n}}\sum_{t\in I_j}W_{n,t}\varepsilon_t,\qquad G_n^{\mathrm{gap}}=\frac{1}{\sqrt{n}}\sum_{t\in J}W_{n,t}\varepsilon_t. \tag*{(64)} B j = n 1 t ∈ I j ∑ W n , t ε t , G n gap = n 1 t ∈ J ∑ W n , t ε t . (64)
By the definitions above, we have
M n = ∑ j = 1 N B j + G n g a p . (65) M_n=\sum_{j=1}^{N}B_j+G_n^{\mathrm{gap}}. \tag*{(65)} M n = j = 1 ∑ N B j + G n gap . (65)
Applying Lemma B.12 to the trajectory segments ( Y t : t ∈ I j ) (Y_t:t\in I_j) ( Y t : t ∈ I j ) , we obtain independent stationary copies of the retained segments. Each copy has the same dependence within its interval as a stationary trajectory of that length. Using the same deterministic weights, define
B j ′ = 1 n ∑ t ∈ I j W n , t ε t ′ . (66) B'_j=\frac{1}{\sqrt{n}}\sum_{t\in I_j}W_{n,t}\varepsilon'_t. \tag*{(66)} B j ′ = n 1 t ∈ I j ∑ W n , t ε t ′ . (66)
By choosing the fixed constant C 1 C_1 C 1 sufficiently large, we have, for all sufficiently large n n n ,
P ( E n c ) ≤ C N e − c m i x m ≤ n − 5 , E n = { B j = B j ′ for every j } . (67) \mathbb{P}(\mathcal{E}_n^c)\le CNe^{-c_{\mathrm{mix}}m}\le n^{-5},\qquad\mathcal{E}_n=\{B_j=B'_j\text{ for every }j\}. \tag*{(67)} P ( E n c ) ≤ C N e − c mix m ≤ n − 5 , E n = { B j = B j ′ for every j } . (67)
The copied segments are defined on an extension of the original probability space. In this construction, the original discarded observations and retained observations keep their joint law.
We first bound the sum over the discarded indices. By the martingale difference property of (S.6), increments at distinct times are orthogonal, including when the indices in J J J are not consecutive. Thus, using (B.28), we have
E ∥ G n g a p ∥ 2 2 = 1 n ∑ t ∈ J E ∥ W n , t ε t ∥ 2 2 ≤ K 0 2 ∣ J ∣ n , (68) \mathbb{E}\lVert G_n^{\mathrm{gap}}\rVert_2^2=\frac{1}{n}\sum_{t\in J}\mathbb{E}\lVert W_{n,t}\varepsilon_t\rVert_2^2\le\frac{K_0^2|J|}{n}, \tag*{(68)} E ∥ G n gap ∥ 2 2 = n 1 t ∈ J ∑ E ∥ W n , t ε t ∥ 2 2 ≤ n K 0 2 ∣ J ∣ , (68)
E ∥ G n g a p ∥ 2 ≤ K 0 ∣ J ∣ n ≤ C n − 1 / 4 ℓ n . (69) \mathbb{E}\lVert G_n^{\mathrm{gap}}\rVert_2\le K_0\sqrt{\frac{|J|}{n}}\le Cn^{-1/4}\sqrt{\ell_n}. \tag*{(69)} E ∥ G n gap ∥ 2 ≤ K 0 n ∣ J ∣ ≤ C n − 1/4 ℓ n . (69)
The term at t = 0 t=0 t = 0 also has mean zero, so it is included in the orthogonality calculation of (S.6). Alternatively, it can be bounded directly by K 0 / n K_0/\sqrt{n} K 0 / n . The above calculation does not require independence between G n g a p G_n^{\mathrm{gap}} G n gap and the sum over the retained intervals.
Next, denote L j = ∣ I j ∣ ≤ q L_j=|I_j|\le q L j = ∣ I j ∣ ≤ q . Within each stationary copied interval, the noise increments are bounded martingale differences. Applying Azuma–Hoeffding coordinatewise and integrating the tail gives, for every fixed p ≥ 2 p\ge2 p ≥ 2 ,
( E ∥ ∑ t ∈ I j W n , t ε t ′ ∥ 2 p ) 1 / p ≤ C d , p L j . (70) \left(\mathbb{E}\left\lVert\sum_{t\in I_j}W_{n,t}\varepsilon'_t\right\rVert_2^p\right)^{1/p}\le C_{d,p}\sqrt{L_j}. \tag*{(70)} E t ∈ I j ∑ W n , t ε t ′ 2 p 1/ p ≤ C d , p L j . (70)
Taking p = 3 p=3 p = 3 , we obtain
E ∥ B j ′ ∥ 2 3 ≤ C n − 3 / 2 L j 3 / 2 . (71) \mathbb{E}\lVert B'_j\rVert_2^3\le Cn^{-3/2}L_j^{3/2}. \tag*{(71)} E ∥ B j ′ ∥ 2 3 ≤ C n − 3/2 L j 3/2 . (71)
Summing over the retained intervals and using L j 3 / 2 ≤ q L j L_j^{3/2}\le\sqrt{q}L_j L j 3/2 ≤ q L j , we have
∑ j = 1 N E ∥ B j ′ ∥ 2 3 ≤ C n − 3 / 2 ∑ j L j 3 / 2 ≤ C n − 3 / 2 q ∑ j L j ≤ C q / n ≤ C n − 1 / 4 . (72) \begin{aligned}
\sum_{j=1}^{N}\mathbb{E}\lVert B'_j\rVert_2^3\le Cn^{-3/2}\sum_jL_j^{3/2}\le Cn^{-3/2}\sqrt{q}\sum_jL_j \\
&\le C\sqrt{q/n}\le Cn^{-1/4}. \tag*{(72)}
\end{aligned} j = 1 ∑ N E ∥ B j ′ ∥ 2 3 ≤ C n − 3/2 j ∑ L j 3/2 ≤ C n − 3/2 q j ∑ L j ≤ C q / n ≤ C n − 1/4 . (72)
{#eq:B.40}
Thus, the sum of the third moments has order n − 1 / 4 n^{-1/4} n − 1/4 , which will determine the polynomial order in the Gaussian approximation.
Denote B n ′ = ∑ j B j ′ B'_n=\sum_j B'_j B n ′ = ∑ j B j ′ . Each copied sum has mean zero. Using independence between the copied intervals and martingale orthogonality within each interval, we obtain
Cov ( B n ′ ) = C n I , C n I = 1 n ∑ t ∈ I W n , t S W n , t T . (73) \operatorname{Cov}(B'_n)=C^I_n,\qquad C^I_n=\frac{1}{n}\sum_{t\in I}W_{n,t}SW_{n,t}^{\mathsf T}. \tag*{(73)} Cov ( B n ′ ) = C n I , C n I = n 1 t ∈ I ∑ W n , t S W n , t T . (73)
where the single-time second moment is S S S by stationarity and (S.6). Since the terms omitted from the full covariance are precisely those indexed by J J J , we have
∥ C n I − C n ∥ F = ∥ 1 n ∑ t ∈ J W n , t S W n , t T ∥ F ≤ C ∣ J ∣ / n ≤ C ℓ n n − 1 / 2 . (74) \left\|C^I_n-C_n\right\|_F=\left\|\frac{1}{n}\sum_{t\in J}W_{n,t}SW_{n,t}^{\mathsf T}\right\|_F\le C|J|/n\le C\ell_n n^{-1/2}. \tag*{(74)} C n I − C n F = n 1 t ∈ J ∑ W n , t S W n , t T F ≤ C ∣ J ∣/ n ≤ C ℓ n n − 1/2 . (74)
Combining the above bound with (B.4), we obtain
∥ C n I − Σ ∞ ∥ F ≤ C ( n β − 1 + ℓ n n − 1 / 2 ) = o ( 1 ) . (75) \left\|C^I_n-\Sigma_\infty\right\|_F\le C\left(n^{\beta-1}+\ell_n n^{-1/2}\right)=o(1). \tag*{(75)} C n I − Σ ∞ F ≤ C ( n β − 1 + ℓ n n − 1/2 ) = o ( 1 ) . (75)
Since Σ ∞ ≻ 0 \Sigma_\infty\succ0 Σ ∞ ≻ 0 , there exist fixed constants 0 < λ ∗ < Λ ∗ < ∞ 0<\lambda_*<\Lambda_*<\infty 0 < λ ∗ < Λ ∗ < ∞ such that, for all sufficiently large n n n ,
λ ∗ I d ⪯ C n I , C n ⪯ Λ ∗ I d . (76) \lambda_*I_d\preceq C^I_n,C_n\preceq\Lambda_*I_d. \tag*{(76)} λ ∗ I d ⪯ C n I , C n ⪯ Λ ∗ I d . (76)
This is where Assumption 3.2 is used in the proof. We now apply the Gaussian approximation bound for independent sums. Using the covariance of the retained sum, define
ξ j = ( C n I ) − 1 / 2 B j ′ , ∑ j E ξ j ξ j T = I d . (77) \xi_j=(C^I_n)^{-1/2}B'_j,\qquad\sum_j\mathbb{E}\xi_j\xi_j^{\mathsf T}=I_d. \tag*{(77)} ξ j = ( C n I ) − 1/2 B j ′ , j ∑ E ξ j ξ j T = I d . (77)
By (B.44) and (B.40),
ρ 3 = ∑ j E ∥ ξ j ∥ 2 3 ≤ λ ∗ − 3 / 2 ∑ j E ∥ B j ′ ∥ 2 3 ≤ C n − 1 / 4 . (78) \rho_3=\sum_j\mathbb{E}\|\xi_j\|_2^3\le\lambda_*^{-3/2}\sum_j\mathbb{E}\|B'_j\|_2^3\le Cn^{-1/4}. \tag*{(78)} ρ 3 = j ∑ E ∥ ξ j ∥ 2 3 ≤ λ ∗ − 3/2 j ∑ E ∥ B j ′ ∥ 2 3 ≤ C n − 1/4 . (78)
For sufficiently large n n n , we have ρ 3 ≤ 1 / 2 \rho_3\le1/2 ρ 3 ≤ 1/2 . By Lemma B.13 , the extension to a general covariance given after its proof, and (B.46), we obtain
W 1 ( B n ′ , N ( 0 , C n I ) ) ≤ C ρ 3 ( 1 + ∣ log ρ 3 ∣ ) ≤ C ℓ n n − 1 / 4 . (79) W_1(B'_n,\mathcal{N}(0,C^I_n))\le C\rho_3(1+|\log\rho_3|)\le C\ell_n n^{-1/4}. \tag*{(79)} W 1 ( B n ′ , N ( 0 , C n I )) ≤ C ρ 3 ( 1 + ∣ log ρ 3 ∣ ) ≤ C ℓ n n − 1/4 . (79)
where the last inequality follows because x ↦ x ( 1 − log x ) x\mapsto x(1-\log x) x ↦ x ( 1 − log x ) is increasing on ( 0 , 1 ) (0,1) ( 0 , 1 ) and ρ 3 ≤ C n − 1 / 4 \rho_3\le Cn^{-1/4} ρ 3 ≤ C n − 1/4 . For all sufficiently large n n n , using the eigenvalue bounds in (B.44), we apply Lemma B.11 to obtain
W 1 ( N ( 0 , C n I ) , N ( 0 , C n ) ) ≤ C ∥ C n I − C n ∥ F ≤ C ℓ n n − 1 / 2 . (80) W_1(\mathcal{N}(0,C^I_n),\mathcal{N}(0,C_n))\le C\left\|C^I_n-C_n\right\|_F\le C\ell_n n^{-1/2}. \tag*{(80)} W 1 ( N ( 0 , C n I ) , N ( 0 , C n )) ≤ C C n I − C n F ≤ C ℓ n n − 1/2 . (80)
It remains to compare the copied sum with the original sum. Denote B n = ∑ j B j B_n=\sum_j B_j B n = ∑ j B j , so M n = B n + G n n M_n=B_n+G^n_n M n = B n + G n n . There are at most n n n summands in each of B n B_n B n and B n ′ B'_n B n ′ . By (B.28), we have
∥ B n ∥ 2 ≤ K 0 n , ∥ B n ′ ∥ 2 ≤ K 0 n . \|B_n\|_2\le K_0\sqrt{n},\qquad\|B'_n\|_2\le K_0\sqrt{n}. ∥ B n ∥ 2 ≤ K 0 n , ∥ B n ′ ∥ 2 ≤ K 0 n .
Since B n = B n ′ B_n=B'_n B n = B n ′ on the event E n \mathcal{E}_n E n , we have
E ∥ B n − B n ′ ∥ 2 = E [ ∥ B n − B n ′ ∥ 2 1 E n c ] ≤ 2 K 0 n P ( E n c ) ≤ C n − 9 / 2 . (81) \mathbb{E}\|B_n-B'_n\|_2=\mathbb{E}\left[\|B_n-B'_n\|_2\mathbf{1}_{\mathcal{E}_n^c}\right]\le2K_0\sqrt{n}\,\mathbb{P}(\mathcal{E}_n^c)\le Cn^{-9/2}. \tag*{(81)} E ∥ B n − B n ′ ∥ 2 = E [ ∥ B n − B n ′ ∥ 2 1 E n c ] ≤ 2 K 0 n P ( E n c ) ≤ C n − 9/2 . (81)
Let Z n I ∼ N ( 0 , C n I ) Z_n^I \sim\mathcal{N}(0,C_n^I) Z n I ∼ N ( 0 , C n I ) and Z n ∼ N ( 0 , C n ) Z_n \sim\mathcal{N}(0,C_n) Z n ∼ N ( 0 , C n ) . We couple the original and copied retained sums as above, and couple M n M_n M n and B n B_n B n through their decomposition on the original space. Combining the preceding estimates with the triangle inequality gives
W 1 ( M n , Z n ) ≤ E ∥ G n a p p ∥ 2 + E ∥ B n − B n ′ ∥ 2 + W 1 ( B n ′ , Z n I ) + W 1 ( Z n I , Z n ) (82) W_1(M_n,Z_n) \le\mathbb{E}\lVert G_n^{\mathrm{app}}\rVert_2+\mathbb{E}\lVert B_n-B_n'\rVert_2+W_1(B_n',Z_n^I)+W_1(Z_n^I,Z_n) \tag*{(82)} W 1 ( M n , Z n ) ≤ E ∥ G n app ∥ 2 + E ∥ B n − B n ′ ∥ 2 + W 1 ( B n ′ , Z n I ) + W 1 ( Z n I , Z n ) (82)
≤ C { n − 1 / 4 ℓ n + n − 9 / 2 + n − 1 / 4 ℓ n + n − 1 / 2 ℓ n } (83) \le C\left\{n^{-1/4}\sqrt{\ell_n}+n^{-9/2}+n^{-1/4}\ell_n+n^{-1/2}\ell_n\right\} \tag*{(83)} ≤ C { n − 1/4 ℓ n + n − 9/2 + n − 1/4 ℓ n + n − 1/2 ℓ n } (83)
≤ C n − 1 / 4 ℓ n . (84) \le Cn^{-1/4}\ell_n. \tag*{(84)} ≤ C n − 1/4 ℓ n . (84)
This proves (B.27) and completes the proof. □
Problem Specific Bounds
Lemma B.1. Under Assumption 2.1, for every t ≥ 0 t \ge0 t ≥ 0 and p ≥ 2 p \ge2 p ≥ 2 ,
∥ Δ t ∥ L p ( ℓ ∞ ) ≤ C p α t . (85) \lVert\Delta_t\rVert_{L^p(\ell_\infty)} \le C\sqrt{p\alpha_t}. \tag*{(85)} ∥ Δ t ∥ L p ( ℓ ∞ ) ≤ C p α t . (85) Also, pathwise,
∥ Δ t ∥ ∞ ≤ H , ∥ Δ t + 1 − Δ t ∥ ∞ ≤ H α t . (86) \lVert\Delta_t\rVert_\infty\le H,\qquad\lVert\Delta_{t+1}-\Delta_t\rVert_\infty\le H\alpha_t. \tag*{(86)} ∥ Δ t ∥ ∞ ≤ H , ∥ Δ t + 1 − Δ t ∥ ∞ ≤ H α t . (86) Consequently,
∑ t < n ∥ Δ t ∥ L p ( ℓ ∞ ) 2 ≤ C p n 1 − β , ∑ t < n ∥ ∥ Δ t ∥ ∞ 2 ∥ L p ≤ C p n 1 − β . (87) \sum_{t<n}\lVert\Delta_t\rVert_{L^p(\ell_\infty)}^2\le Cpn^{1-\beta},\qquad
\sum_{t<n}\left\lVert\lVert\Delta_t\rVert_\infty^2\right\rVert_{L^p}\le Cpn^{1-\beta}. \tag*{(87)} t < n ∑ ∥ Δ t ∥ L p ( ℓ ∞ ) 2 ≤ C p n 1 − β , t < n ∑ ∥ Δ t ∥ ∞ 2 L p ≤ C p n 1 − β . (87)
Proof. We first bound the cumulative centered update over an arbitrary interval. We then apply this bound on intervals with fixed cumulative stepsize and compare the iterates with the mean recursion.
We begin by showing that the iterates remain bounded. Suppose Q t ∈ [ 0 , H ] d Q_t\in[0,H]^d Q t ∈ [ 0 , H ] d . For the updated coordinate i = ( s t , a t ) i=(s_t,a_t) i = ( s t , a t ) , the update rule gives
Q t + 1 ( i ) = ( 1 − α t ) Q t ( i ) + α t [ r ( i ) + γ V ( Q t ) ( s t + 1 ) ] . Q_{t+1}(i)=(1-\alpha_t)Q_t(i)+\alpha_t\left[r(i)+\gamma V(Q_t)(s_{t+1})\right]. Q t + 1 ( i ) = ( 1 − α t ) Q t ( i ) + α t [ r ( i ) + γ V ( Q t ) ( s t + 1 ) ] .
Since 0 ≤ r ( i ) + γ V ( Q t ) ( s t + 1 ) ≤ 1 + γ H = H 0\le r(i)+\gamma V(Q_t)(s_{t+1})\le1+\gamma H=H 0 ≤ r ( i ) + γ V ( Q t ) ( s t + 1 ) ≤ 1 + γ H = H and 0 ≤ α t ≤ 1 0\le\alpha_t\le1 0 ≤ α t ≤ 1 , the updated coordinate is a convex combination of two values in [ 0 , H ] [0,H] [ 0 , H ] . The other coordinates remain unchanged. By induction from Q 0 Q_0 Q 0 , we have Q t ∈ [ 0 , H ] d Q_t\in[0,H]^d Q t ∈ [ 0 , H ] d for all t t t . Since Q ⋆ ∈ [ 0 , H ] d Q^\star\in[0,H]^d Q ⋆ ∈ [ 0 , H ] d , this gives ∥ Δ t ∥ ∞ ≤ H \lVert\Delta_t\rVert_\infty\le H ∥ Δ t ∥ ∞ ≤ H . The update changes only one coordinate and its magnitude is at most H α t H\alpha_t H α t . Hence,
∥ Q t + 1 − Q t ∥ ∞ = ∥ Δ t + 1 − Δ t ∥ ∞ ≤ H α t . \lVert Q_{t+1}-Q_t\rVert_\infty=\lVert\Delta_{t+1}-\Delta_t\rVert_\infty\le H\alpha_t. ∥ Q t + 1 − Q t ∥ ∞ = ∥ Δ t + 1 − Δ t ∥ ∞ ≤ H α t .
This proves (B.54).
We now express the update as its stationary mean and a centered perturbation. For y = ( s , a , s ′ ) y=(s,a,s') y = ( s , a , s ′ ) , recall that
U ( q , y ) = u s , a [ r ( s , a ) + γ V ( q ) ( s ′ ) − q ( s , a ) ] , B ( q ) = E μ ˉ U ( q , Y ) . U(q,y)=u_{s,a}\left[r(s,a)+\gamma V(q)(s')-q(s,a)\right],\qquad B(q)=\mathbb{E}_{\bar{\mu}}U(q,Y). U ( q , y ) = u s , a [ r ( s , a ) + γ V ( q ) ( s ′ ) − q ( s , a ) ] , B ( q ) = E μ ˉ U ( q , Y ) .
By the definition of D = diag ( μ ) D=\operatorname{diag}(\mu) D = diag ( μ ) , we have B ( q ) = D ( T q − q ) B(q)=D(Tq-q) B ( q ) = D ( T q − q ) and B ( Q ⋆ ) = 0 B(Q^\star)=0 B ( Q ⋆ ) = 0 . For any q , q ′ ∈ [ 0 , H ] d q,q'\in[0,H]^d q , q ′ ∈ [ 0 , H ] d , the Lipschitz property of the maximum gives
∥ U ( q , y ) − U ( q ′ , y ) ∥ ∞ ≤ ( 1 + γ ) ∥ q − q ′ ∥ ∞ ≤ 2 ∥ q − q ′ ∥ ∞ , \lVert U(q,y)-U(q',y)\rVert_\infty\le(1+\gamma)\lVert q-q'\rVert_\infty\le2\lVert q-q'\rVert_\infty, ∥ U ( q , y ) − U ( q ′ , y ) ∥ ∞ ≤ ( 1 + γ ) ∥ q − q ′ ∥ ∞ ≤ 2 ∥ q − q ′ ∥ ∞ ,
∥ B ( q ) − B ( q ′ ) ∥ ∞ ≤ ( 1 + γ ) ∥ q − q ′ ∥ ∞ ≤ 2 ∥ q − q ′ ∥ ∞ . \lVert B(q)-B(q')\rVert_\infty\le(1+\gamma)\lVert q-q'\rVert_\infty\le2\lVert q-q'\rVert_\infty. ∥ B ( q ) − B ( q ′ ) ∥ ∞ ≤ ( 1 + γ ) ∥ q − q ′ ∥ ∞ ≤ 2 ∥ q − q ′ ∥ ∞ .
Both functions are uniformly bounded on the cube. Define
F q ( y ) = U ( q , y ) − B ( q ) . F_q(y)=U(q,y)-B(q). F q ( y ) = U ( q , y ) − B ( q ) .
By the bounds above, μ ˉ F q = 0 \bar{\mu}F_q = 0 μ ˉ F q = 0 , sup q , y ∥ F q ( y ) ∥ ∞ ≤ C \sup_{q,y}\|F_q(y)\|_{\infty} \le C sup q , y ∥ F q ( y ) ∥ ∞ ≤ C , and sup y ∥ F q ( y ) − F q ′ ( y ) ∥ ∞ ≤ C ∥ q − q ′ ∥ ∞ \sup_y\|F_q(y)-F_{q'}(y)\|_{\infty} \le C\|q-q'\|_{\infty} sup y ∥ F q ( y ) − F q ′ ( y ) ∥ ∞ ≤ C ∥ q − q ′ ∥ ∞ . Substituting the definition into the original recursion, we have
Q t + 1 = Q t + α t B ( Q t ) + α t F Q t ( Y t ) . (88) Q_{t+1}=Q_t+\alpha_tB(Q_t)+\alpha_tF_{Q_t}(Y_t). \tag*{(88)} Q t + 1 = Q t + α t B ( Q t ) + α t F Q t ( Y t ) . (88)
Although F Q t ( Y t ) F_{Q_t}(Y_t) F Q t ( Y t ) is centered under μ ˉ \bar{\mu} μ ˉ for each fixed parameter, it is not necessarily a martingale difference. The iterate Q t Q_t Q t and the observation Y t Y_t Y t both depend on the past trajectory. We therefore apply the Poisson equation. For every q ∈ [ 0 , H ] d q\in[0,H]^d q ∈ [ 0 , H ] d , define
ψ q ( y ) = ∑ j = 0 ∞ ( K j F q ) ( y ) , ( K j f ) ( y ) = ∫ f ( z ) K j ( y , d z ) . (89) \psi_q(y)=\sum_{j=0}^{\infty}(K^jF_q)(y),\qquad(K^jf)(y)=\int f(z)K^j(y,dz). \tag*{(89)} ψ q ( y ) = j = 0 ∑ ∞ ( K j F q ) ( y ) , ( K j f ) ( y ) = ∫ f ( z ) K j ( y , d z ) . (89)
Since F q F_q F q is centered, uniformly bounded, and Lipschitz in q q q , Lemma B.9 gives
sup y ∥ ψ q ( y ) ∥ ∞ ≤ C , sup y ∥ ψ q ( y ) − ψ q ′ ( y ) ∥ ∞ ≤ C ∥ q − q ′ ∥ ∞ . (90) \sup_y\|\psi_q(y)\|_{\infty}\le C,\qquad\sup_y\|\psi_q(y)-\psi_{q'}(y)\|_{\infty}\le C\|q-q'\|_{\infty}. \tag*{(90)} y sup ∥ ψ q ( y ) ∥ ∞ ≤ C , y sup ∥ ψ q ( y ) − ψ q ′ ( y ) ∥ ∞ ≤ C ∥ q − q ′ ∥ ∞ . (90)
Applying K K K to the series term by term gives
ψ q − K ψ q = F q . (91) \psi_q-K\psi_q=F_q. \tag*{(91)} ψ q − K ψ q = F q . (91)
Denote g q = K ψ q g_q=K\psi_q g q = K ψ q . Since a Markov kernel does not increase the sup norm, the same boundedness and Lipschitz estimates hold for g q g_q g q .
Let F t = σ ( Q 0 , Y 0 , … , Y t ) \mathcal{F}_t=\sigma(Q_0,Y_0,\ldots,Y_t) F t = σ ( Q 0 , Y 0 , … , Y t ) . For t ≥ 1 t\ge1 t ≥ 1 , Q t Q_t Q t is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable. By (B.59), we have
F Q t ( Y t ) = M t + g Q t ( Y t − 1 ) − g Q t ( Y t ) , M t : = ψ Q t ( Y t ) − g Q t ( Y t − 1 ) . (92) F_{Q_t}(Y_t)=M_t+g_{Q_t}(Y_{t-1})-g_{Q_t}(Y_t),\qquad M_t:=\psi_{Q_t}(Y_t)-g_{Q_t}(Y_{t-1}). \tag*{(92)} F Q t ( Y t ) = M t + g Q t ( Y t − 1 ) − g Q t ( Y t ) , M t := ψ Q t ( Y t ) − g Q t ( Y t − 1 ) . (92)
Since Q t Q_t Q t is measurable with respect to the past, the Markov property gives
E [ M t ∣ F t − 1 ] = K ψ Q t ( Y t − 1 ) − g Q t ( Y t − 1 ) = 0. \mathbb{E}[M_t\mid\mathcal{F}_{t-1}]=K\psi_{Q_t}(Y_{t-1})-g_{Q_t}(Y_{t-1})=0. E [ M t ∣ F t − 1 ] = K ψ Q t ( Y t − 1 ) − g Q t ( Y t − 1 ) = 0.
By (B.58), we also have ∥ M t ∥ ∞ ≤ C \|M_t\|_{\infty}\le C ∥ M t ∥ ∞ ≤ C almost surely. Thus, ( M t ) t ≥ 1 (M_t)_{t\ge1} ( M t ) t ≥ 1 is a bounded vector-valued martingale difference sequence.
For integers 1 ≤ u < v 1\le u<v 1 ≤ u < v , define
S k M = ∑ t = u k − 1 α t M t , u ≤ k ≤ v . S_k^M=\sum_{t=u}^{k-1}\alpha_tM_t,\qquad u\le k\le v. S k M = t = u ∑ k − 1 α t M t , u ≤ k ≤ v .
Applying the maximal Azuma–Hoeffding inequality to each coordinate, we obtain
P ( max u ≤ k ≤ v ∣ ( S k M ) i ∣ > x ) ≤ 2 exp ( − c x 2 ∑ t = u v − 1 α t 2 ) . \mathbb{P}\left(\max_{u\le k\le v}|(S_k^M)_i|>x\right)\le2\exp\left(-\frac{cx^2}{\sum_{t=u}^{v-1}\alpha_t^2}\right). P ( u ≤ k ≤ v max ∣ ( S k M ) i ∣ > x ) ≤ 2 exp ( − ∑ t = u v − 1 α t 2 c x 2 ) .
Taking a union bound over the fixed d d d coordinates and integrating p x p − 1 P ( ⋅ > x ) px^{p-1}\mathbb{P}(\cdot>x) p x p − 1 P ( ⋅ > x ) over x > 0 x>0 x > 0 , we have, for p ≥ 2 p\ge2 p ≥ 2 ,
∥ max u ≤ k ≤ v ∥ S k M ∥ ∞ ∥ L p ≤ C p ∑ t = u v − 1 α t 2 . (93) \left\|\max_{u\le k\le v}\|S_k^M\|_{\infty}\right\|_{L_p}\le C\sqrt{p\sum_{t=u}^{v-1}\alpha_t^2}. \tag*{(93)} u ≤ k ≤ v max ∥ S k M ∥ ∞ L p ≤ C p t = u ∑ v − 1 α t 2 . (93)
The dependence on the fixed dimension is included in C C C . We now bound the remaining differences in the Poisson decomposition. Denote g t = g Q t g_t = gQ_t g t = g Q t . For u < k ≤ v u < k \le v u < k ≤ v , summation by parts gives
∑ t = u k − 1 α t [ g t ( Y t − 1 ) − g t ( Y t ) ] = α u g u ( Y u − 1 ) − α k − 1 g k − 1 ( Y k − 1 ) + ∑ t = u k − 2 [ α t + 1 g t + 1 ( Y t ) − α t g t ( Y t ) ] . (94) \sum_{t=u}^{k-1} \alpha_t[g_t(Y_{t-1})-g_t(Y_t)]
= \alpha_u g_u(Y_{u-1})-\alpha_{k-1}g_{k-1}(Y_{k-1})+\sum_{t=u}^{k-2}[\alpha_{t+1}g_{t+1}(Y_t)-\alpha_tg_t(Y_t)]. \tag*{(94)} t = u ∑ k − 1 α t [ g t ( Y t − 1 ) − g t ( Y t )] = α u g u ( Y u − 1 ) − α k − 1 g k − 1 ( Y k − 1 ) + t = u ∑ k − 2 [ α t + 1 g t + 1 ( Y t ) − α t g t ( Y t )] . (94)
When k = u k=u k = u , the sum is empty and equals zero, so this index does not change the maximum over u ≤ k ≤ v u \le k \le v u ≤ k ≤ v . Since α t \alpha_t α t is decreasing and g q g_q g q is uniformly bounded, both endpoint terms have sup norm at most C α u C\alpha_u C α u . For the interior terms, we use the decomposition
α t + 1 g t + 1 ( Y t ) − α t g t ( Y t ) = ( α t + 1 − α t ) g t ( Y t ) + α t + 1 [ g t + 1 ( Y t ) − g t ( Y t ) ] . \alpha_{t+1}g_{t+1}(Y_t)-\alpha_tg_t(Y_t)=(\alpha_{t+1}-\alpha_t)g_t(Y_t)+\alpha_{t+1}[g_{t+1}(Y_t)-g_t(Y_t)]. α t + 1 g t + 1 ( Y t ) − α t g t ( Y t ) = ( α t + 1 − α t ) g t ( Y t ) + α t + 1 [ g t + 1 ( Y t ) − g t ( Y t )] .
For the first term, by monotonicity of the stepsizes, we have
∑ t = u k − 2 ∣ α t + 1 − α t ∣ ∥ g t ∥ ∞ ≤ C ∑ t = u k − 2 ( α t − α t + 1 ) ≤ C α u . \sum_{t=u}^{k-2}|\alpha_{t+1}-\alpha_t|\lVert g_t\rVert_\infty\le C\sum_{t=u}^{k-2}(\alpha_t-\alpha_{t+1})\le C\alpha_u. t = u ∑ k − 2 ∣ α t + 1 − α t ∣ ∥ g t ∥ ∞ ≤ C t = u ∑ k − 2 ( α t − α t + 1 ) ≤ C α u .
For the second term, using (B.58) and (B.54), we have
∥ g t + 1 − g t ∥ ∞ ≤ C ∥ Q t + 1 − Q t ∥ ∞ ≤ C α t . \lVert g_{t+1}-g_t\rVert_\infty\le C\lVert Q_{t+1}-Q_t\rVert_\infty\le C\alpha_t. ∥ g t + 1 − g t ∥ ∞ ≤ C ∥ Q t + 1 − Q t ∥ ∞ ≤ C α t .
Substituting this inequality into the sum gives
∑ t = u k − 2 α t + 1 ∥ g t + 1 − g t ∥ ∞ ≤ C ∑ t = u v − 1 α t 2 . \sum_{t=u}^{k-2}\alpha_{t+1}\lVert g_{t+1}-g_t\rVert_\infty\le C\sum_{t=u}^{v-1}\alpha_t^2. t = u ∑ k − 2 α t + 1 ∥ g t + 1 − g t ∥ ∞ ≤ C t = u ∑ v − 1 α t 2 .
All the preceding estimates hold for each sample path and simultaneously over k k k . Combining them, we obtain
max u ≤ k ≤ v ∥ ∑ t = u k − 1 α t [ g t ( Y t − 1 ) − g t ( Y t ) ] ∥ ∞ ≤ C ( α u + ∑ t = u v − 1 α t 2 ) . (95) \max_{u\le k\le v}\left\lVert\sum_{t=u}^{k-1}\alpha_t[g_t(Y_{t-1})-g_t(Y_t)]\right\rVert_\infty
\le C\left(\alpha_u+\sum_{t=u}^{v-1}\alpha_t^2\right). \tag*{(95)} u ≤ k ≤ v max t = u ∑ k − 1 α t [ g t ( Y t − 1 ) − g t ( Y t )] ∞ ≤ C ( α u + t = u ∑ v − 1 α t 2 ) . (95)
Combining (B.60), (B.61), and (B.63), we conclude that
∥ max u ≤ k ≤ v ∥ ∑ t = u k − 1 α t F Q t ( Y t ) ∥ ∞ ∥ L p ≤ C ( p ∑ t = u v − 1 α t 2 + α u + ∑ t = u v − 1 α t 2 ) . (96) \left\lVert\max_{u\le k\le v}\left\lVert\sum_{t=u}^{k-1}\alpha_tF_{Q_t}(Y_t)\right\rVert_\infty\right\rVert_{L^p}
\le C\left(\sqrt{p\sum_{t=u}^{v-1}\alpha_t^2}+\alpha_u+\sum_{t=u}^{v-1}\alpha_t^2\right). \tag*{(96)} u ≤ k ≤ v max t = u ∑ k − 1 α t F Q t ( Y t ) ∞ L p ≤ C p t = u ∑ v − 1 α t 2 + α u + t = u ∑ v − 1 α t 2 . (96)
For u = 0 u=0 u = 0 , the first term α 0 F Q 0 ( Y 0 ) \alpha_0F_{Q_0}(Y_0) α 0 F Q 0 ( Y 0 ) has norm at most C α 0 C\alpha_0 C α 0 . We bound this term directly and apply the preceding calculation from t = 1 t=1 t = 1 onward. Thus, (B.64) also holds for u = 0 u=0 u = 0 after increasing C C C if necessary.
We next divide the time indices into intervals with a fixed cumulative stepsize. Let
T 0 = log 4 a ⋆ , t 0 = 0 , t j + 1 = min { v > t j : ∑ t = t j v − 1 α t ≥ T 0 } . T_0=\frac{\log4}{a_\star},\qquad t_0=0,\qquad t_{j+1}=\min\left\{v>t_j:\sum_{t=t_j}^{v-1}\alpha_t\ge T_0\right\}. T 0 = a ⋆ log 4 , t 0 = 0 , t j + 1 = min ⎩ ⎨ ⎧ v > t j : t = t j ∑ v − 1 α t ≥ T 0 ⎭ ⎬ ⎫ .
Since ∑ t α t = ∞ \sum_t \alpha_t = \infty ∑ t α t = ∞ for β < 1 \beta< 1 β < 1 , each endpoint is well defined. For a block [ u , v ] = [ t j , t j + 1 ] [u,v] = [t_j,t_{j+1}] [ u , v ] = [ t j , t j + 1 ] , by the minimality of v v v and the bound α t ≤ 1 \alpha_t \le1 α t ≤ 1 , we have
T 0 ≤ ∑ t = u v − 1 α t < T 0 + 1. (97) T_0 \le\sum_{t=u}^{v-1} \alpha_t < T_0 + 1. \tag*{(97)} T 0 ≤ t = u ∑ v − 1 α t < T 0 + 1. (97)
We first compare the stepsizes at the two endpoints. Using α t = c 0 ( t + b ) − β \alpha_t = c_0(t+b)^{-\beta} α t = c 0 ( t + b ) − β and the shift condition, we have
log α u α v = β ∑ t = u v − 1 log ( 1 + 1 t + b ) ≤ β ∑ t = u v − 1 1 t + b = β c 0 ∑ t = u v − 1 ( t + b ) β − 1 α t ≤ β c 0 ( u + b ) β − 1 ∑ t = u v − 1 α t ≤ a ⋆ 4 ( T 0 + 1 ) = log 4 + a ⋆ 4 < log 2. (98) \begin{aligned}
\log\frac{\alpha_u}{\alpha_v}
&= \beta\sum_{t=u}^{v-1} \log\left(1+\frac{1}{t+b}\right) \\
&\le\beta\sum_{t=u}^{v-1} \frac{1}{t+b} \\
&= \frac{\beta}{c_0}\sum_{t=u}^{v-1}(t+b)^{\beta-1}\alpha_t \\
&\le\frac{\beta}{c_0}(u+b)^{\beta-1}\sum_{t=u}^{v-1}\alpha_t \\
&\le\frac{a_\star}{4}(T_0+1)=\frac{\log4+a_\star}{4}<\log2. \tag*{(98)}
\end{aligned} log α v α u = β t = u ∑ v − 1 log ( 1 + t + b 1 ) ≤ β t = u ∑ v − 1 t + b 1 = c 0 β t = u ∑ v − 1 ( t + b ) β − 1 α t ≤ c 0 β ( u + b ) β − 1 t = u ∑ v − 1 α t ≤ 4 a ⋆ ( T 0 + 1 ) = 4 log 4 + a ⋆ < log 2. (98)
where the last inequality follows from a ⋆ = ρ ( 1 − γ ) ≤ 1 a_\star=\rho(1-\gamma)\le1 a ⋆ = ρ ( 1 − γ ) ≤ 1 . Therefore,
α u α v < 2 , \frac{\alpha_u}{\alpha_v}<2, α v α u < 2 ,
and hence α u ≤ 2 α t ( u ≤ t ≤ v ) \alpha_u\le2\alpha_t\quad(u\le t\le v) α u ≤ 2 α t ( u ≤ t ≤ v ) .
Since the stepsizes are nonincreasing, we also have
∑ t = u v − 1 α t 2 ≤ α u ∑ t = u v − 1 α t ≤ ( T 0 + 1 ) α u . (99) \sum_{t=u}^{v-1}\alpha_t^2\le\alpha_u\sum_{t=u}^{v-1}\alpha_t\le(T_0+1)\alpha_u. \tag*{(99)} t = u ∑ v − 1 α t 2 ≤ α u t = u ∑ v − 1 α t ≤ ( T 0 + 1 ) α u . (99)
Substituting (B.67) into (B.64) gives a bound for the noise on each block. Since α u ≤ 1 \alpha_u\le1 α u ≤ 1 and p ≥ 2 p\ge2 p ≥ 2 , we have α u ≤ p α u \alpha_u\le\sqrt{p\alpha_u} α u ≤ p α u . Hence,
∥ max u ≤ k ≤ v ∥ ∑ t = u k − 1 α t F Q t ( Y t ) ∥ ∥ ∞ , L p ≤ C p α u . (100) \left\|\max_{u\le k\le v}\left\|\sum_{t=u}^{k-1}\alpha_tF_{Q_t}(Y_t)\right\|\right\|_{\infty,L^p}\le C\sqrt{p\alpha_u}. \tag*{(100)} u ≤ k ≤ v max t = u ∑ k − 1 α t F Q t ( Y t ) ∞ , L p ≤ C p α u . (100)
For the same block [ u , v ] [u,v] [ u , v ] , we define the mean recursion by
z u = Q u , z t + 1 = z t + α t B ( z t ) ( u ≤ t < v ) . z_u=Q_u,\qquad z_{t+1}=z_t+\alpha_tB(z_t)\quad(u\le t<v). z u = Q u , z t + 1 = z t + α t B ( z t ) ( u ≤ t < v ) .
We first show that the mean-update map G α ( q ) = q + α B ( q ) G_\alpha(q)=q+\alpha B(q) G α ( q ) = q + α B ( q ) preserves [ 0 , H ] d [0,H]^d [ 0 , H ] d . Its i i i th coordinate is ( 1 − α μ i ) q i + α μ i ( T q ) i (1-\alpha\mu_i)q_i+\alpha\mu_i(Tq)_i ( 1 − α μ i ) q i + α μ i ( T q ) i , which is a convex combination of two values in [ 0 , H ] [0,H] [ 0 , H ] . For q ∈ [ 0 , H ] d q\in[0,H]^d q ∈ [ 0 , H ] d and 0 ≤ α ≤ 1 0\le\alpha\le1 0 ≤ α ≤ 1 , the Bellman contraction gives
∣ G α ( q ) i − Q i ⋆ ∣ ≤ ( 1 − α μ i ) ∣ q i − Q i ⋆ ∣ + α μ i ∣ ( T q ) i − ( T Q ⋆ ) i ∣ ≤ [ 1 − α μ i ( 1 − γ ) ] ∥ q − Q ⋆ ∥ ∞ ≤ ( 1 − a ⋆ α ) ∥ q − Q ⋆ ∥ ∞ . \begin{aligned}
|G_\alpha(q)_i-Q_i^\star|
&\le(1-\alpha\mu_i)|q_i-Q_i^\star|+\alpha\mu_i|(Tq)_i-(TQ^\star)_i| \\
&\le[1-\alpha\mu_i(1-\gamma)]\|q-Q^\star\|_\infty\\
&\le(1-a_\star\alpha)\|q-Q^\star\|_\infty.
\end{aligned} ∣ G α ( q ) i − Q i ⋆ ∣ ≤ ( 1 − α μ i ) ∣ q i − Q i ⋆ ∣ + α μ i ∣ ( T q ) i − ( T Q ⋆ ) i ∣ ≤ [ 1 − α μ i ( 1 − γ )] ∥ q − Q ⋆ ∥ ∞ ≤ ( 1 − a ⋆ α ) ∥ q − Q ⋆ ∥ ∞ .
Taking the maximum over i i i and iterating over the block, we obtain
∥ z v − Q ⋆ ∥ ∞ ≤ ∏ t = u v − 1 ( 1 − a ⋆ α t ) ∥ Q u − Q ⋆ ∥ ∞ ≤ exp ( − a ⋆ ∑ t = u v − 1 α t ) ∥ Δ u ∥ ∞ ≤ 1 4 ∥ Δ u ∥ ∞ . (101) \lVert z_v-Q^\star\rVert_\infty\le\prod_{t=u}^{v-1}(1-a_\star\alpha_t)\lVert Q_u-Q^\star\rVert_\infty
\le\exp\left(-a_\star\sum_{t=u}^{v-1}\alpha_t\right)\lVert\Delta_u\rVert_\infty\le\frac{1}{4}\lVert\Delta_u\rVert_\infty. \tag*{(101)} ∥ z v − Q ⋆ ∥ ∞ ≤ t = u ∏ v − 1 ( 1 − a ⋆ α t ) ∥ Q u − Q ⋆ ∥ ∞ ≤ exp ( − a ⋆ t = u ∑ v − 1 α t ) ∥ Δ u ∥ ∞ ≤ 4 1 ∥ Δ u ∥ ∞ . (101)
Denote E k = Q k − z k E_k=Q_k-z_k E k = Q k − z k . Subtracting the two recursions and using E u = 0 E_u=0 E u = 0 , we have
E k = ∑ t = u k − 1 α t F Q t ( Y t ) + ∑ t = u k − 1 α t [ B ( Q t ) − B ( z t ) ] . (102) E_k=\sum_{t=u}^{k-1}\alpha_tF_{Q_t}(Y_t)+\sum_{t=u}^{k-1}\alpha_t[B(Q_t)-B(z_t)]. \tag*{(102)} E k = t = u ∑ k − 1 α t F Q t ( Y t ) + t = u ∑ k − 1 α t [ B ( Q t ) − B ( z t )] . (102)
Set
N u , v = max u ≤ k ≤ v ∥ ∑ t = u k − 1 α t F Q t ( Y t ) ∥ ∞ . N_{u,v}=\max_{u\le k\le v}\left\lVert\sum_{t=u}^{k-1}\alpha_tF_{Q_t}(Y_t)\right\rVert_\infty. N u , v = u ≤ k ≤ v max t = u ∑ k − 1 α t F Q t ( Y t ) ∞ .
Since B B B is 2-Lipschitz, the preceding identity implies
∥ E k ∥ ∞ ≤ N u , v + 2 ∑ t = u k − 1 α t ∥ E t ∥ ∞ . \lVert E_k\rVert_\infty\le N_{u,v}+2\sum_{t=u}^{k-1}\alpha_t\lVert E_t\rVert_\infty. ∥ E k ∥ ∞ ≤ N u , v + 2 t = u ∑ k − 1 α t ∥ E t ∥ ∞ .
Applying the discrete Gronwall inequality to each sample path, we obtain
max u ≤ k ≤ v ∥ E k ∥ ∞ ≤ N u , v ∏ t = u v − 1 ( 1 + 2 α t ) ≤ N u , v exp ( 2 ∑ t = u v − 1 α t ) ≤ e 2 ( T 0 + 1 ) N u , v . \max_{u\le k\le v}\lVert E_k\rVert_\infty\le N_{u,v}\prod_{t=u}^{v-1}(1+2\alpha_t)\le N_{u,v}\exp\left(2\sum_{t=u}^{v-1}\alpha_t\right)\le e^{2(T_0+1)}N_{u,v}. u ≤ k ≤ v max ∥ E k ∥ ∞ ≤ N u , v t = u ∏ v − 1 ( 1 + 2 α t ) ≤ N u , v exp ( 2 t = u ∑ v − 1 α t ) ≤ e 2 ( T 0 + 1 ) N u , v .
Taking the L p L^p L p norm and using (B.68), we conclude that
∥ max u ≤ k ≤ v ∥ Q k − z k ∥ ∞ ∥ L p ≤ C p α u . (103) \left\lVert\max_{u\le k\le v}\lVert Q_k-z_k\rVert_\infty\right\rVert_{L^p}\le C\sqrt{p\alpha_u}. \tag*{(103)} u ≤ k ≤ v max ∥ Q k − z k ∥ ∞ L p ≤ C p α u . (103)
In the calculation above, we first sum the noise increments and then take the norm. This retains cancellation in the cumulative noise. Taking the norm of each increment before summing would not give the square-root bound.
We now bound the error at the block endpoints. Combining the triangle inequality, (B.69), and (B.71), we have
∥ Δ v ∥ L p ( ℓ ∞ ) ≤ 1 4 ∥ Δ u ∥ L p ( ℓ ∞ ) + C p α u . (104) \lVert\Delta_v\rVert_{L^p(\ell_\infty)}\le\frac{1}{4}\lVert\Delta_u\rVert_{L^p(\ell_\infty)}+C\sqrt{p\alpha_u}. \tag*{(104)} ∥ Δ v ∥ L p ( ℓ ∞ ) ≤ 4 1 ∥ Δ u ∥ L p ( ℓ ∞ ) + C p α u . (104)
To compare the error with the current stepsize, define
d j = ∥ Δ t j ∥ L p ( ℓ ∞ ) p α t j . d_j=\frac{\lVert\Delta_{t_j}\rVert_{L^p(\ell_\infty)}}{\sqrt{p\alpha_{t_j}}}. d j = p α t j ∥ Δ t j ∥ L p ( ℓ ∞ ) .
By (B.66) and (B.72), we obtain
d j + 1 ≤ 1 4 α t j α t j + 1 d j + C α t j α t j + 1 ≤ 2 4 d j + C 2 ≤ 1 2 d j + C . (105) d_{j+1}\le\frac{1}{4}\sqrt{\frac{\alpha_{t_j}}{\alpha_{t_{j+1}}}}d_j+C\sqrt{\frac{\alpha_{t_j}}{\alpha_{t_{j+1}}}}\le\frac{\sqrt{2}}{4}d_j+C\sqrt{2}\le\frac{1}{2}d_j+C. \tag*{(105)} d j + 1 ≤ 4 1 α t j + 1 α t j d j + C α t j + 1 α t j ≤ 4 2 d j + C 2 ≤ 2 1 d j + C . (105)
Iterating the last inequality gives
d j ≤ 2 − j d 0 + C ∑ i = 0 j − 1 2 − i ≤ d 0 + 2 C . d_j \le2^{-j}d_0+C\sum_{i=0}^{j-1}2^{-i}\le d_0+2C. d j ≤ 2 − j d 0 + C i = 0 ∑ j − 1 2 − i ≤ d 0 + 2 C .
Since d 0 ≤ H / p α 0 ≤ H / 2 α 0 d_0\le H/\sqrt{p\alpha_0}\le H/\sqrt{2\alpha_0} d 0 ≤ H / p α 0 ≤ H / 2 α 0 and α 0 \alpha_0 α 0 is fixed, the initial normalized error is bounded by a constant independent of p p p . Therefore,
∥ Δ t j ∥ L p ( ℓ ∞ ) ≤ C p α t j for all j ≥ 0. (106) \|\Delta_{t_j}\|_{L_p(\ell_\infty)}\le C\sqrt{p\alpha_{t_j}}\qquad\text{for all }j\ge0. \tag*{(106)} ∥ Δ t j ∥ L p ( ℓ ∞ ) ≤ C p α t j for all j ≥ 0. (106)
Finally, we extend the endpoint bound to all intermediate times. Fix t ≥ 0 t\ge0 t ≥ 0 and choose j j j such that u = t j ≤ t ≤ t j + 1 = v u=t_j\le t\le t_{j+1}=v u = t j ≤ t ≤ t j + 1 = v . By contraction of the mean recursion up to time t t t and the comparison bound, we have
∥ Δ t ∥ L p ( ℓ ∞ ) ≤ ∥ Q t − z t ∥ L p ( ℓ ∞ ) + ∥ z t − Q ∗ ∥ L p ( ℓ ∞ ) ≤ C p α u + ∥ Δ u ∥ L p ( ℓ ∞ ) ≤ C p α u . \|\Delta_t\|_{L_p(\ell_\infty)}
\le\|Q_t-z_t\|_{L_p(\ell_\infty)}+\|z_t-Q^*\|_{L_p(\ell_\infty)}
\le C\sqrt{p\alpha_u}+\|\Delta_u\|_{L_p(\ell_\infty)}
\le C\sqrt{p\alpha_u}. ∥ Δ t ∥ L p ( ℓ ∞ ) ≤ ∥ Q t − z t ∥ L p ( ℓ ∞ ) + ∥ z t − Q ∗ ∥ L p ( ℓ ∞ ) ≤ C p α u + ∥ Δ u ∥ L p ( ℓ ∞ ) ≤ C p α u .
Using α u ≤ 2 α t \alpha_u\le2\alpha_t α u ≤ 2 α t from (B.66), we obtain
∥ Δ t ∥ L p ( ℓ ∞ ) ≤ C p α t , \|\Delta_t\|_{L_p(\ell_\infty)}\le C\sqrt{p\alpha_t}, ∥ Δ t ∥ L p ( ℓ ∞ ) ≤ C p α t ,
which proves (B.53).
It remains to bound the sums of quadratic moments. By (B.53) and the sum of the power stepsizes, we have
∑ t = 0 n − 1 ∥ Δ t ∥ L p ( ℓ ∞ ) 2 ≤ C p ∑ t = 0 n − 1 α t ≤ C p n 1 − β . \sum_{t=0}^{n-1}\|\Delta_t\|_{L_p(\ell_\infty)}^2
\le Cp\sum_{t=0}^{n-1}\alpha_t
\le Cpn^{1-\beta}. t = 0 ∑ n − 1 ∥ Δ t ∥ L p ( ℓ ∞ ) 2 ≤ C p t = 0 ∑ n − 1 α t ≤ C p n 1 − β .
For the second bound, by Minkowski’s inequality and the identity ∥ ∥ Δ t ∥ ∞ 2 ∥ L p = ∥ Δ t ∥ L 2 p ( ℓ ∞ ) 2 \left\|\|\Delta_t\|_\infty^2\right\|_{L_p}=\|\Delta_t\|_{L_{2p}(\ell_\infty)}^2 ∥ Δ t ∥ ∞ 2 L p = ∥ Δ t ∥ L 2 p ( ℓ ∞ ) 2 , we obtain
∥ ∑ t = 0 n − 1 ∥ Δ t ∥ ∞ 2 ∥ L p ≤ ∑ t = 0 n − 1 ∥ ∥ Δ t ∥ ∞ 2 ∥ L p = ∑ t = 0 n − 1 ∥ Δ t ∥ L 2 p ( ℓ ∞ ) 2 ≤ C ( 2 p ) ∑ t = 0 n − 1 α t ≤ C p n 1 − β . \left\|\sum_{t=0}^{n-1}\|\Delta_t\|_\infty^2\right\|_{L_p}
\le\sum_{t=0}^{n-1}\left\|\|\Delta_t\|_\infty^2\right\|_{L_p}
= \sum_{t=0}^{n-1}\|\Delta_t\|_{L_{2p}(\ell_\infty)}^2
\le C(2p)\sum_{t=0}^{n-1}\alpha_t
\le Cpn^{1-\beta}. t = 0 ∑ n − 1 ∥ Δ t ∥ ∞ 2 L p ≤ t = 0 ∑ n − 1 ∥ Δ t ∥ ∞ 2 L p = t = 0 ∑ n − 1 ∥ Δ t ∥ L 2 p ( ℓ ∞ ) 2 ≤ C ( 2 p ) t = 0 ∑ n − 1 α t ≤ C p n 1 − β .
This proves (B.55) and completes the proof. □
Lemma B.2. For all n ≥ 1 n\ge1 n ≥ 1 ,
sup t < n ∥ W n , t ∥ ∞ ≤ C , (107) \sup_{t<n}\|W_{n,t}\|_\infty\le C, \tag*{(107)} t < n sup ∥ W n , t ∥ ∞ ≤ C , (107) ∑ t = 0 n − 2 ∥ W n , t + 1 − W n , t ∥ ∞ ≤ C ℓ n , (108) \sum_{t=0}^{n-2}\|W_{n,t+1}-W_{n,t}\|_\infty\le C\ell_n, \tag*{(108)} t = 0 ∑ n − 2 ∥ W n , t + 1 − W n , t ∥ ∞ ≤ C ℓ n , (108) 1 n ∑ t < n ∥ W n , t − A − 1 ∥ ∞ ≤ C n β − 1 . (109) \frac{1}{n}\sum_{t<n}\|W_{n,t}-A^{-1}\|_\infty\le Cn^{\beta-1}. \tag*{(109)} n 1 t < n ∑ ∥ W n , t − A − 1 ∥ ∞ ≤ C n β − 1 . (109)
The sum in (B.76) is zero for n = 1 n=1 n = 1 . For every 0 ≤ t < n 0 \le t < n 0 ≤ t < n ,
∥ W n , t − A − 1 ∥ ∞ ≤ C ( ( t + b ) β − 1 + exp { − a ⋆ 2 ∑ u = t + 1 n α u } ) . (110) \lVert W_{n,t}-A^{-1}\rVert_{\infty}\le C\left((t+b)^{\beta-1}+\exp\left\{-\frac{a_{\star}}{2}\sum_{u=t+1}^{n}\alpha_u\right\}\right). \tag*{(110)} ∥ W n , t − A − 1 ∥ ∞ ≤ C ( ( t + b ) β − 1 + exp { − 2 a ⋆ u = t + 1 ∑ n α u } ) . (110)
Also,
∥ Φ i : j ∥ ∞ ≤ exp ( − a ⋆ ∑ u = i j α u ) , ∥ A − 1 ∥ ∞ ≤ a ⋆ − 1 . (111) \lVert\Phi_{i:j}\rVert_{\infty}\le\exp\left(-a_{\star}\sum_{u=i}^{j}\alpha_u\right),\qquad\lVert A^{-1}\rVert_{\infty}\le a_{\star}^{-1}. \tag*{(111)} ∥ Φ i : j ∥ ∞ ≤ exp ( − a ⋆ u = i ∑ j α u ) , ∥ A − 1 ∥ ∞ ≤ a ⋆ − 1 . (111)
Proof. We first bound the matrix products and compare the stepsizes at different indices. We then use these estimates to compare the infinite-horizon weights with A − 1 A^{-1} A − 1 and bound the finite-sample tail. Finally, we prove the bound for adjacent weights.
Recall that A = D ( I − γ P π ⋆ ) A=D(I-\gamma P^{\pi^\star}) A = D ( I − γ P π ⋆ ) . For 0 ≤ α ≤ 1 0\le\alpha\le1 0 ≤ α ≤ 1 , we have
I − α A = I − α D + α γ D P π ⋆ . I-\alpha A=I-\alpha D+\alpha\gamma DP^{\pi^\star}. I − α A = I − α D + α γ D P π ⋆ .
Since μ i ≤ 1 \mu_i\le1 μ i ≤ 1 , all diagonal entries of I − α D I-\alpha D I − α D are nonnegative. The entries of D P π ⋆ DP^{\pi^\star} D P π ⋆ are also nonnegative, so the full matrix has nonnegative entries. For row i i i , the row sum is
1 − α μ i + α γ μ i = 1 − α μ i ( 1 − γ ) ≤ 1 − a ⋆ α . 1-\alpha\mu_i+\alpha\gamma\mu_i=1-\alpha\mu_i(1-\gamma)\le1-a_{\star}\alpha. 1 − α μ i + α γ μ i = 1 − α μ i ( 1 − γ ) ≤ 1 − a ⋆ α .
The infinity norm of a nonnegative matrix equals its largest row sum. Thus, by submultiplicativity and 1 − x ≤ e − x 1-x\le e^{-x} 1 − x ≤ e − x , we have
∥ Φ i : j ∥ ∞ ≤ ∏ u = i j ( 1 − a ⋆ α u ) ≤ exp ( − a ⋆ ∑ u = i j α u ) . (112) \lVert\Phi_{i:j}\rVert_{\infty}\le\prod_{u=i}^{j}(1-a_{\star}\alpha_u)\le\exp\left(-a_{\star}\sum_{u=i}^{j}\alpha_u\right). \tag*{(112)} ∥ Φ i : j ∥ ∞ ≤ u = i ∏ j ( 1 − a ⋆ α u ) ≤ exp ( − a ⋆ u = i ∑ j α u ) . (112)
When i > j i>j i > j , the same inequality holds by the empty-product convention. The matrix A A A is invertible since ∥ P π ⋆ ∥ ∞ = 1 \lVert P^{\pi^\star}\rVert_{\infty}=1 ∥ P π ⋆ ∥ ∞ = 1 and γ < 1 \gamma<1 γ < 1 . Its inverse can be written as
A − 1 = ( I − γ P π ⋆ ) − 1 D − 1 = ( ∑ r = 0 ∞ γ r ( P π ⋆ ) r ) D − 1 . A^{-1}=(I-\gamma P^{\pi^\star})^{-1}D^{-1}=\left(\sum_{r=0}^{\infty}\gamma^r(P^{\pi^\star})^r\right)D^{-1}. A − 1 = ( I − γ P π ⋆ ) − 1 D − 1 = ( r = 0 ∑ ∞ γ r ( P π ⋆ ) r ) D − 1 .
Taking the infinity norm, we obtain
∥ A − 1 ∥ ∞ ≤ 1 1 − γ 1 ρ = 1 a ⋆ . (113) \lVert A^{-1}\rVert_{\infty}\le\frac{1}{1-\gamma}\frac{1}{\rho}=\frac{1}{a_{\star}}. \tag*{(113)} ∥ A − 1 ∥ ∞ ≤ 1 − γ 1 ρ 1 = a ⋆ 1 . (113)
Since ∑ u α u = ∞ \sum_u\alpha_u=\infty ∑ u α u = ∞ for β < 1 \beta<1 β < 1 , it follows from (B.80) that Φ i : j → 0 \Phi_{i:j}\to0 Φ i : j → 0 as j → ∞ j\to\infty j → ∞ for each fixed i i i . For k > t k>t k > t , define the cumulative stepsize and its comparison parameter by
s t , k = ∑ u = t + 1 k − 1 α u , η t = β c 0 ( t + b ) β − 1 . (114) s_{t,k}=\sum_{u=t+1}^{k-1}\alpha_u,\qquad\eta_t=\frac{\beta}{c_0}(t+b)^{\beta-1}. \tag*{(114)} s t , k = u = t + 1 ∑ k − 1 α u , η t = c 0 β ( t + b ) β − 1 . (114)
Since β − 1 \beta-1 β − 1 is negative, η t \eta_t η t decreases with t t t . Note that b b b is chosen to satisfy β c 0 b β − 1 ≤ a ⋆ 4 \frac{\beta}{c_0}b^{\beta-1}\le\frac{a_{\star}}{4} c 0 β b β − 1 ≤ 4 a ⋆ , and thus we have η t ≤ η 0 ≤ a ⋆ / 4 \eta_t\le\eta_0\le a_{\star}/4 η t ≤ η 0 ≤ a ⋆ /4 . Using the formula for the stepsizes, we obtain
log α t α k = β log k + b t + b = β ∑ u = t k − 1 log ( 1 + 1 u + b ) ≤ β ∑ u = t k − 1 1 u + b = ∑ u = t k − 1 β c 0 ( u + b ) β − 1 α u ≤ η t ∑ u = t k − 1 α u = η t ( α t + s t , k ) . (115) \begin{aligned}
\log\frac{\alpha_t}{\alpha_k}=\beta\log\frac{k+b}{t+b}=\beta\sum_{u=t}^{k-1}\log\left(1+\frac{1}{u+b}\right) \\
&\le\beta\sum_{u=t}^{k-1}\frac{1}{u+b}=\sum_{u=t}^{k-1}\frac{\beta}{c_0}(u+b)^{\beta-1}\alpha_u \\
&\le\eta_t\sum_{u=t}^{k-1}\alpha_u=\eta_t(\alpha_t+s_{t,k}). \tag*{(115)}
\end{aligned} log α k α t = β log t + b k + b = β u = t ∑ k − 1 log ( 1 + u + b 1 ) ≤ β u = t ∑ k − 1 u + b 1 = u = t ∑ k − 1 c 0 β ( u + b ) β − 1 α u ≤ η t u = t ∑ k − 1 α u = η t ( α t + s t , k ) . (115)
Thus, the relative change in the stepsize can be bounded using the cumulative stepsize that appears in the product bound. Exponentiating the preceding inequality gives
α t α k ≤ e η t α t e η t s t , k ≤ e a ⋆ / 4 e ( a ⋆ / 4 ) s t , k . (116) \frac{\alpha_t}{\alpha_k} \le e^{\eta_t\alpha_t}e^{\eta_t s_{t,k}} \le e^{a_\star/4}e^{(a_\star/4)s_{t,k}}. \tag*{(116)} α k α t ≤ e η t α t e η t s t , k ≤ e a ⋆ /4 e ( a ⋆ /4 ) s t , k . (116)
The exponent in this bound has a smaller coefficient than the contraction exponent in (B.80). We use this difference to bound the sums below.
For fixed t t t and λ > 0 \lambda> 0 λ > 0 , the cumulative sums s t , k s_{t,k} s t , k satisfy
s t , t + 1 = 0 , s t , k + 1 − s t , k = α k . s_{t,t+1}=0,\qquad s_{t,k+1}-s_{t,k}=\alpha_k. s t , t + 1 = 0 , s t , k + 1 − s t , k = α k .
Since the cumulative sum diverges, the intervals [ s t , k , s t , k + 1 ] [s_{t,k},s_{t,k+1}] [ s t , k , s t , k + 1 ] , k = t + 1 , t + 2 , … k=t+1,t+2,\ldots k = t + 1 , t + 2 , … , partition [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) up to their common endpoints. Each interval has length α k ≤ 1 \alpha_k \le1 α k ≤ 1 . For x ∈ [ s t , k , s t , k + 1 ] x \in[s_{t,k},s_{t,k+1}] x ∈ [ s t , k , s t , k + 1 ] , we have
e − λ s t , k ≤ e λ e − λ x , 1 + s t , k ≤ 1 + x . e^{-\lambda s_{t,k}} \le e^\lambda e^{-\lambda x},\qquad1+s_{t,k}\le1+x. e − λ s t , k ≤ e λ e − λ x , 1 + s t , k ≤ 1 + x .
Integrating over the interval, we obtain
α k ( 1 + s t , k ) e − λ s t , k ≤ e λ ∫ s t , k s t , k + 1 ( 1 + x ) e − λ x d x . \alpha_k(1+s_{t,k})e^{-\lambda s_{t,k}} \le e^\lambda\int_{s_{t,k}}^{s_{t,k+1}}(1+x)e^{-\lambda x}\,\mathrm{d}x. α k ( 1 + s t , k ) e − λ s t , k ≤ e λ ∫ s t , k s t , k + 1 ( 1 + x ) e − λ x d x .
Summing these inequalities over k k k gives
∑ k > t α k ( 1 + s t , k ) e − λ s t , k ≤ e λ ∫ 0 ∞ ( 1 + x ) e − λ x d x = e λ ( 1 λ + 1 λ 2 ) . (117) \sum_{k>t}\alpha_k(1+s_{t,k})e^{-\lambda s_{t,k}} \le e^\lambda\int_0^\infty(1+x)e^{-\lambda x}\,\mathrm{d}x=e^\lambda\left(\frac{1}{\lambda}+\frac{1}{\lambda^2}\right). \tag*{(117)} k > t ∑ α k ( 1 + s t , k ) e − λ s t , k ≤ e λ ∫ 0 ∞ ( 1 + x ) e − λ x d x = e λ ( λ 1 + λ 2 1 ) . (117)
Since the factor 1 + s t , k 1+s_{t,k} 1 + s t , k is at least one, the sum without it is also bounded uniformly in t t t . We now define the infinite-horizon weight by
W ∞ , t = α t ∑ k = t + 1 ∞ Φ t + 1 : k − 1 . (118) W_{\infty,t}=\alpha_t\sum_{k=t+1}^{\infty}\Phi_{t+1:k-1}. \tag*{(118)} W ∞ , t = α t k = t + 1 ∑ ∞ Φ t + 1 : k − 1 . (118)
To show that this series converges, we insert the factor α k / α k \alpha_k/\alpha_k α k / α k and apply (B.80) and (B.84). We have
α t ∑ k > t ∥ Φ t + 1 : k − 1 ∥ ∞ ≤ ∑ k > t α k α t α k e − a ⋆ s t , k ≤ e a ⋆ / 4 ∑ k > t α k e − ( 3 a ⋆ / 4 ) s t , k ≤ C . \alpha_t\sum_{k>t}\lVert\Phi_{t+1:k-1}\rVert_\infty
\le\sum_{k>t}\alpha_k\frac{\alpha_t}{\alpha_k}e^{-a_\star s_{t,k}}
\le e^{a_\star/4}\sum_{k>t}\alpha_k e^{-(3a_\star/4)s_{t,k}}
\le C. α t k > t ∑ ∥ Φ t + 1 : k − 1 ∥ ∞ ≤ k > t ∑ α k α k α t e − a ⋆ s t , k ≤ e a ⋆ /4 k > t ∑ α k e − ( 3 a ⋆ /4 ) s t , k ≤ C .
where the last inequality follows from (B.85) with λ = 3 a ⋆ / 4 \lambda=3a_\star/4 λ = 3 a ⋆ /4 . This proves absolute convergence and gives
sup t ≥ 0 ∥ W ∞ , t ∥ ∞ ≤ C . \sup_{t\ge0}\lVert W_{\infty,t}\rVert_\infty\le C. t ≥ 0 sup ∥ W ∞ , t ∥ ∞ ≤ C .
The same calculation bounds every finite partial sum and therefore proves (B.75). Next, we compare the infinite-horizon weight with the inverse drift. Since every factor in the product commutes with A A A , for k > t k>t k > t , we have
α k A Φ t + 1 : k − 1 = Φ t + 1 : k − 1 − Φ t + 1 : k . \alpha_k A\Phi_{t+1:k-1}=\Phi_{t+1:k-1}-\Phi_{t+1:k}. α k A Φ t + 1 : k − 1 = Φ t + 1 : k − 1 − Φ t + 1 : k .
Summing over k k k from the first index to m > t m>t m > t gives
A ∑ k = t + 1 m α k Φ t + 1 : k − 1 = I − Φ t + 1 : m . (119) A\sum_{k=t+1}^{m}\alpha_k\Phi_{t+1:k-1}=I-\Phi_{t+1:m}. \tag*{(119)} A k = t + 1 ∑ m α k Φ t + 1 : k − 1 = I − Φ t + 1 : m . (119)
Taking m → ∞ m \to\infty m → ∞ and using (B.80), we obtain
∑ k > t α k Φ t + 1 : k − 1 = A − 1 . (120) \sum_{k>t}\alpha_k\Phi_{t+1:k-1}=A^{-1}. \tag*{(120)} k > t ∑ α k Φ t + 1 : k − 1 = A − 1 . (120)
The identity above uses α k \alpha_k α k in each summand, while the definition of the infinite-horizon weight uses α t \alpha_t α t in every summand. Their difference therefore measures the effect of changing stepsizes. Subtracting (B.88) from (B.86), we have
W ∞ , t − A − 1 = ∑ k > t α k ( α t α k − 1 ) Φ t + 1 : k − 1 . (121) W_{\infty,t}-A^{-1}=\sum_{k>t}\alpha_k\left(\frac{\alpha_t}{\alpha_k}-1\right)\Phi_{t+1:k-1}. \tag*{(121)} W ∞ , t − A − 1 = k > t ∑ α k ( α k α t − 1 ) Φ t + 1 : k − 1 . (121)
Since the stepsizes decrease, the ratio is at least one. For x ≥ 0 x \ge0 x ≥ 0 , we have e x − 1 = ∫ 0 x e y d y ≤ x e x e^x-1=\int_0^x e^y\,\mathrm{d}y\le xe^x e x − 1 = ∫ 0 x e y d y ≤ x e x . Taking x = log ( α t / α k ) x=\log(\alpha_t/\alpha_k) x = log ( α t / α k ) and applying (B.83), we obtain
0 ≤ α t α k − 1 ≤ η t ( α t + s t , k ) e η t ( α t + s t , k ) . 0\le\frac{\alpha_t}{\alpha_k}-1\le\eta_t(\alpha_t+s_{t,k})e^{\eta_t(\alpha_t+s_{t,k})}. 0 ≤ α k α t − 1 ≤ η t ( α t + s t , k ) e η t ( α t + s t , k ) .
Substituting this inequality into the difference above gives
∥ W ∞ , t − A − 1 ∥ ∞ ≤ η t e η t α t ∑ k > t α k ( α t + s t , k ) e − ( a ⋆ − η t ) s t , k ≤ C η t ∑ k > t α k ( 1 + s t , k ) e − ( 3 a ⋆ / 4 ) s t , k ≤ C η t ≤ C ( t + b ) β − 1 . (122) \begin{aligned}
\left\|W_{\infty,t}-A^{-1}\right\|_\infty
\le\eta_t e^{\eta_t\alpha_t}\sum_{k>t}\alpha_k(\alpha_t+s_{t,k})e^{-(a_\star-\eta_t)s_{t,k}} \\
&\le C\eta_t\sum_{k>t}\alpha_k(1+s_{t,k})e^{-(3a_\star/4)s_{t,k}} \\
&\le C\eta_t\le C(t+b)^{\beta-1}. \tag*{(122)}
\end{aligned} W ∞ , t − A − 1 ∞ ≤ η t e η t α t k > t ∑ α k ( α t + s t , k ) e − ( a ⋆ − η t ) s t , k ≤ C η t k > t ∑ α k ( 1 + s t , k ) e − ( 3 a ⋆ /4 ) s t , k ≤ C η t ≤ C ( t + b ) β − 1 . (122)
where we used α t ≤ 1 \alpha_t\le1 α t ≤ 1 , η t ≤ a ⋆ / 4 \eta_t\le a_\star/4 η t ≤ a ⋆ /4 , and (B.85). This proves the first part of the pointwise estimate (B.78).
We next bound the tail beyond the finite horizon. Denote T n , t = ∑ u = t + 1 n α u T_{n,t}=\sum_{u=t+1}^n\alpha_u T n , t = ∑ u = t + 1 n α u . Factoring the products at time n n n , we have
W ∞ , t − W n , t = α t ∑ k = n + 1 ∞ Φ t + 1 : k − 1 = α t Φ t + 1 : n ∑ k = n + 1 ∞ Φ n + 1 : k − 1 = α t α n Φ t + 1 : n W ∞ , n . (123) W_{\infty,t}-W_{n,t}
=\alpha_t\sum_{k=n+1}^{\infty}\Phi_{t+1:k-1}
=\alpha_t\Phi_{t+1:n}\sum_{k=n+1}^{\infty}\Phi_{n+1:k-1}
=\frac{\alpha_t}{\alpha_n}\Phi_{t+1:n}W_{\infty,n}. \tag*{(123)} W ∞ , t − W n , t = α t k = n + 1 ∑ ∞ Φ t + 1 : k − 1 = α t Φ t + 1 : n k = n + 1 ∑ ∞ Φ n + 1 : k − 1 = α n α t Φ t + 1 : n W ∞ , n . (123)
The factor α t / α n \alpha_t/\alpha_n α t / α n is not uniformly bounded, so we estimate it together with the contracting product. Taking k = n k=n k = n in (B.83), we obtain
α t α n ≤ exp { η t ( α t + ∑ u = t + 1 n − 1 α u ) } ≤ e a ⋆ / 4 e ( a ⋆ / 4 ) T n , t . \frac{\alpha_t}{\alpha_n}\le\exp\left\{\eta_t\left(\alpha_t+\sum_{u=t+1}^{n-1}\alpha_u\right)\right\}\le e^{a_\star/4}e^{(a_\star/4)T_{n,t}}. α n α t ≤ exp { η t ( α t + u = t + 1 ∑ n − 1 α u ) } ≤ e a ⋆ /4 e ( a ⋆ /4 ) T n , t .
Combining this bound with (B.80) and the uniform bound for W ∞ , n W_{\infty,n} W ∞ , n , we have
∥ W ∞ , t − W n , t ∥ ∞ ≤ C e − ( 3 a ⋆ / 4 ) T n , t ≤ C e − ( a ⋆ / 2 ) T n , t . (124) \left\|W_{\infty,t}-W_{n,t}\right\|_\infty\le Ce^{-(3a_\star/4)T_{n,t}}\le Ce^{-(a_\star/2)T_{n,t}}. \tag*{(124)} ∥ W ∞ , t − W n , t ∥ ∞ ≤ C e − ( 3 a ⋆ /4 ) T n , t ≤ C e − ( a ⋆ /2 ) T n , t . (124)
By adding and subtracting the infinite-horizon weight, we have
W n , t − A − 1 = ( W ∞ , t − A − 1 ) − ( W ∞ , t − W n , t ) . W_{n,t}-A^{-1}=(W_{\infty,t}-A^{-1})-(W_{\infty,t}-W_{n,t}). W n , t − A − 1 = ( W ∞ , t − A − 1 ) − ( W ∞ , t − W n , t ) .
Combining (B.90) and (B.92) with the triangle inequality proves (B.78).
We now average this pointwise bound over the time indices. For the first term, using b ≥ 1 b \ge1 b ≥ 1 and β − 1 < 0 \beta- 1 < 0 β − 1 < 0 , we have
∑ t = 0 n − 1 ( t + b ) β − 1 ≤ ∑ j = 1 n j β − 1 ≤ 1 + ∫ 1 n x β − 1 d x ≤ n β β . (125) \sum_{t=0}^{n-1}(t+b)^{\beta-1} \le\sum_{j=1}^{n}j^{\beta-1} \le1+\int_{1}^{n}x^{\beta-1}\,\mathrm{d}x \le\frac{n^\beta}{\beta}. \tag*{(125)} t = 0 ∑ n − 1 ( t + b ) β − 1 ≤ j = 1 ∑ n j β − 1 ≤ 1 + ∫ 1 n x β − 1 d x ≤ β n β . (125)
For the second term, let c = a ⋆ / 2 c=a_{\star}/2 c = a ⋆ /2 . Since α u ≥ α n \alpha_u \ge\alpha_n α u ≥ α n for u ≤ n u \le n u ≤ n , we have T n , t ≥ ( n − t ) α n T_{n,t} \ge(n-t)\alpha_n T n , t ≥ ( n − t ) α n . It follows that
∑ t = 0 n − 1 e − c T n , t ≤ ∑ j = 1 n e − c j α n ≤ ∑ j = 1 ∞ e − c j α n = 1 e c α n − 1 ≤ 1 c α n = ( n + b ) β c c 0 ≤ C n β . (126) \begin{aligned}
\sum_{t=0}^{n-1}e^{-cT_{n,t}} \le\sum_{j=1}^{n}e^{-cj\alpha_n} \le\sum_{j=1}^{\infty}e^{-cj\alpha_n} = \frac{1}{e^{c\alpha_n}-1} \\
&\le\frac{1}{c\alpha_n}=\frac{(n+b)^\beta}{cc_0}\le Cn^\beta. \tag*{(126)}
\end{aligned} t = 0 ∑ n − 1 e − c T n , t ≤ j = 1 ∑ n e − c j α n ≤ j = 1 ∑ ∞ e − c j α n = e c α n − 1 1 ≤ c α n 1 = c c 0 ( n + b ) β ≤ C n β . (126)
The last constant can depend on the fixed shift b b b . In the last step, we used n + b ≤ ( 1 + b ) n n+b \le(1+b)n n + b ≤ ( 1 + b ) n for n ≥ 1 n \ge1 n ≥ 1 . Summing (B.93) and (B.94) and dividing by n n n proves (B.77).
It remains to prove (B.76). The sum is empty when n = 1 n=1 n = 1 , so we assume n ≥ 2 n \ge2 n ≥ 2 below. For 0 ≤ t ≤ n − 2 0 \le t \le n-2 0 ≤ t ≤ n − 2 , separating the k = t + 1 k=t+1 k = t + 1 term in (S.8) and factoring I − α t + 1 A I-\alpha_{t+1}A I − α t + 1 A out of the remaining products gives
W n , t = α t I + α t α t + 1 ( I − α t + 1 A ) W n , t + 1 . (127) W_{n,t}=\alpha_t I+\frac{\alpha_t}{\alpha_{t+1}}(I-\alpha_{t+1}A)W_{n,t+1}. \tag*{(127)} W n , t = α t I + α t + 1 α t ( I − α t + 1 A ) W n , t + 1 . (127)
Subtracting W n , t + 1 W_{n,t+1} W n , t + 1 from both sides, we obtain
W n , t − W n , t + 1 = α t ( I − A W n , t + 1 ) + ( α t α t + 1 − 1 ) W n , t + 1 . (128) W_{n,t}-W_{n,t+1}=\alpha_t(I-AW_{n,t+1})+\left(\frac{\alpha_t}{\alpha_{t+1}}-1\right)W_{n,t+1}. \tag*{(128)} W n , t − W n , t + 1 = α t ( I − A W n , t + 1 ) + ( α t + 1 α t − 1 ) W n , t + 1 . (128)
For the first term, use the identity I − A W n , t + 1 = A ( A − 1 − W n , t + 1 ) I-AW_{n,t+1}=A(A^{-1}-W_{n,t+1}) I − A W n , t + 1 = A ( A − 1 − W n , t + 1 ) . Applying (B.78) at t + 1 t+1 t + 1 , we have
α t ∥ I − A W n , t + 1 ∥ ∞ ≤ C α t ( t + b + 1 ) β − 1 + C α t exp ( − c ∑ u = t + 2 n α u ) ≤ C t + b + C α t exp ( − c ∑ u = t + 2 n α u ) . \begin{aligned}
\alpha_t\lVert I-AW_{n,t+1}\rVert_{\infty}\le C\alpha_t(t+b+1)^{\beta-1}+C\alpha_t\exp\left(-c\sum_{u=t+2}^{n}\alpha_u\right) \\
&\le\frac{C}{t+b}+C\alpha_t\exp\left(-c\sum_{u=t+2}^{n}\alpha_u\right).
\end{aligned} α t ∥ I − A W n , t + 1 ∥ ∞ ≤ C α t ( t + b + 1 ) β − 1 + C α t exp ( − c u = t + 2 ∑ n α u ) ≤ t + b C + C α t exp ( − c u = t + 2 ∑ n α u ) .
where the second inequality follows from ( t + b + 1 ) β − 1 ≤ ( t + b ) β − 1 (t+b+1)^{\beta-1}\le(t+b)^{\beta-1} ( t + b + 1 ) β − 1 ≤ ( t + b ) β − 1 and α t = c 0 ( t + b ) − β \alpha_t=c_0(t+b)^{-\beta} α t = c 0 ( t + b ) − β . For the second term in (B.96), by concavity of x ↦ x β x\mapsto x^\beta x ↦ x β , we have
0 ≤ α t α t + 1 − 1 = ( 1 + 1 t + b ) β − 1 ≤ β t + b . 0\le\frac{\alpha_t}{\alpha_{t+1}}-1=\left(1+\frac{1}{t+b}\right)^\beta-1\le\frac{\beta}{t+b}. 0 ≤ α t + 1 α t − 1 = ( 1 + t + b 1 ) β − 1 ≤ t + b β .
Combining the two estimates with (B.75), we obtain
∥ W n , t − W n , t + 1 ∥ ∞ ≤ C t + b + C α t exp ( − c ∑ u = t + 2 n α u ) . (129) \lVert W_{n,t}-W_{n,t+1}\rVert_{\infty}\le\frac{C}{t+b}+C\alpha_t\exp\left(-c\sum_{u=t+2}^{n}\alpha_u\right). \tag*{(129)} ∥ W n , t − W n , t + 1 ∥ ∞ ≤ t + b C + C α t exp ( − c u = t + 2 ∑ n α u ) . (129)
For the sum of the first term in (B.97), the harmonic-sum bound gives
∑ t = 0 n − 2 1 t + b ≤ C log ( e ( n + b ) ) ≤ C ℓ n . \sum_{t=0}^{n-2}\frac{1}{t+b}\le C\log(e(n+b))\le C\ell_n. t = 0 ∑ n − 2 t + b 1 ≤ C log ( e ( n + b )) ≤ C ℓ n .
We bound the sum of the second term by telescoping. Define
R t = ∑ u = t + 2 n α u ( 0 ≤ t ≤ n − 2 ) , R n − 1 = 0. R_t = \sum_{u=t+2}^{n} \alpha_u \quad(0 \le t \le n-2), \qquad R_{n-1}=0. R t = u = t + 2 ∑ n α u ( 0 ≤ t ≤ n − 2 ) , R n − 1 = 0.
By definition, R t − R t + 1 = α t + 2 R_t-R_{t+1}=\alpha_{t+2} R t − R t + 1 = α t + 2 . Since b ≥ 1 b \ge1 b ≥ 1 , we also have
α t α t + 2 = ( 1 + 2 t + b ) β ≤ 3 β . \frac{\alpha_t}{\alpha_{t+2}}=\left(1+\frac{2}{t+b}\right)^\beta\le3^\beta. α t + 2 α t = ( 1 + t + b 2 ) β ≤ 3 β .
Using e x − 1 ≥ x e^x-1 \ge x e x − 1 ≥ x , we obtain
e − c R t + 1 − e − c R t = e − c R t ( e c α t + 2 − 1 ) ≥ c α t + 2 e − c R t . e^{-cR_{t+1}}-e^{-cR_t}=e^{-cR_t}\left(e^{c\alpha_{t+2}}-1\right)\ge c\alpha_{t+2}e^{-cR_t}. e − c R t + 1 − e − c R t = e − c R t ( e c α t + 2 − 1 ) ≥ c α t + 2 e − c R t .
Combining the two inequalities above, we have
∑ t = 0 n − 2 α t e − c R t ≤ 3 β c ∑ t = 0 n − 2 ( e − c R t + 1 − e − c R t ) = 3 β c ( 1 − e − c R 0 ) ≤ 3 β c . (130) \sum_{t=0}^{n-2}\alpha_t e^{-cR_t}\le\frac{3^\beta}{c}\sum_{t=0}^{n-2}\left(e^{-cR_{t+1}}-e^{-cR_t}\right)
=\frac{3^\beta}{c}\left(1-e^{-cR_0}\right)\le\frac{3^\beta}{c}. \tag*{(130)} t = 0 ∑ n − 2 α t e − c R t ≤ c 3 β t = 0 ∑ n − 2 ( e − c R t + 1 − e − c R t ) = c 3 β ( 1 − e − c R 0 ) ≤ c 3 β . (130)
Summing (B.97) and using the preceding bounds proves (B.76). This completes the proof. □
Supporting Lemmas
Lemma B.3 (Theorem 2.3 in [15 ] ). We use the finite-state, uniformly geometrically ergodic specialization of the bounded fundamental-kernel conclusion. Let K K K have invariant law μ ˉ \bar{\mu} μ ˉ and satisfy (S.2), and put Π F = μ ˉ F \Pi F=\bar{\mu}F Π F = μ ˉ F . On bounded centered functions, the potential operator is
R F = ( I − K + Π ) − 1 F = ∑ j ≥ 0 K j F , ( I − K ) R F = F . \mathcal{R}F=(I-K+\Pi)^{-1}F=\sum_{j\ge0}K^jF,\qquad(I-K)\mathcal{R}F=F. R F = ( I − K + Π ) − 1 F = j ≥ 0 ∑ K j F , ( I − K ) R F = F . It is linear and bounded. In the present TV convention its bound can be taken as
∥ R F ∥ s u p ≤ C P ∥ F ∥ s u p , C P = 2 C m i x 1 − e − c m i x . \|\mathcal{R}F\|_{\mathrm{sup}}\le C_P\|F\|_{\mathrm{sup}},\qquad C_P=\frac{2C_{\mathrm{mix}}}{1-e^{-c_{\mathrm{mix}}}}. ∥ R F ∥ sup ≤ C P ∥ F ∥ sup , C P = 1 − e − c mix 2 C mix . The displayed constant follows directly by summing ∥ K j F ∥ s u p ≤ 2 C m i x e − c m i x j ∥ F ∥ s u p \|K^jF\|_{\mathrm{sup}}\le2C_{\mathrm{mix}}e^{-c_{\mathrm{mix}}j}\|F\|_{\mathrm{sup}} ∥ K j F ∥ sup ≤ 2 C mix e − c mix j ∥ F ∥ sup ; thus the series converges in sup norm. Vector-valued functions are treated coordinatewise.
Lemma B.4 (Theorems 3.5 and 4.1 in [32 ] ). We use their scalar specializations, including equation (4.2) of the cited paper. Let d j d_j d j be real martingale differences, S k = ∑ j ≤ k d j S_k=\sum_{j\le k}d_j S k = ∑ j ≤ k d j , d ∗ = max j ∣ d j ∣ d^*=\max_j|d_j| d ∗ = max j ∣ d j ∣ , and s 2 = ∑ j E [ d j 2 ∣ F j − 1 ] s^2=\sum_j\mathbb{E}[d_j^2\mid\mathcal{F}_{j-1}] s 2 = ∑ j E [ d j 2 ∣ F j − 1 ] .
(i) For p ≥ 2 p\ge2 p ≥ 2 and finite p p p th moments,
∥ max k ∣ S k ∣ ∥ L p ≤ C { p ∥ d ∗ ∥ L p + p ∥ s ∥ L p } , \left\|\max_k|S_k|\right\|_{L^p}\le C\{p\|d^*\|_{L^p}+\sqrt{p}\|s\|_{L^p}\}, k max ∣ S k ∣ L p ≤ C { p ∥ d ∗ ∥ L p + p ∥ s ∥ L p } , with an absolute constant C C C independent of p p p and the number of increments.
(ii) If ∣ d j ∣ ≤ a j |d_j|\le a_j ∣ d j ∣ ≤ a j almost surely for deterministic a j a_j a j and V = ∑ j a j 2 > 0 V=\sum_j a_j^2>0 V = ∑ j a j 2 > 0 , then for x ≥ 0 x\ge0 x ≥ 0 ,
Pr { max k ∣ S k ∣ ≥ x } ≤ 2 e − x 2 / ( 2 V ) . \Pr\left\{\max_k|S_k|\ge x\right\}\le2e^{-x^2/(2V)}. Pr { k max ∣ S k ∣ ≥ x } ≤ 2 e − x 2 / ( 2 V ) .
Lemma B.5 (Theorem 9.2 in [5 ] ). We specialize the cited result to symmetric positive-definite matrices, using the identification A = U A=U A = U , B = − V B=-V B = − V in that paper. The Frobenius bound is an immediate consequence of the integral representation; see also Section 10 of the cited paper. For real symmetric positive-definite U , V U,V U , V and any real matrix H H H of compatible size, the equation U E + E V = H UE+EV=H U E + E V = H has the unique solution
E = ∫ 0 ∞ e − t U H e − t V d t , ∥ E ∥ F ≤ ∥ H ∥ F λ min ( U ) + λ min ( V ) . E=\int_{0}^{\infty}e^{-tU}He^{-tV}\,\mathrm{d}t,\qquad\|E\|_{F}\leq\frac{\|H\|_{F}}{\lambda_{\min}(U)+\lambda_{\min}(V)}. E = ∫ 0 ∞ e − t U H e − t V d t , ∥ E ∥ F ≤ λ m i n ( U ) + λ m i n ( V ) ∥ H ∥ F . The integrand’s Frobenius norm is at most e − t ( λ min ( U ) + λ min ( V ) ) ∥ H ∥ F e^{-t(\lambda_{\min}(U)+\lambda_{\min}(V))}\|H\|_{F} e − t ( λ m i n ( U ) + λ m i n ( V )) ∥ H ∥ F . No commutativity of U , V , H U,V,H U , V , H is assumed.
Lemma B.6 (Proposition 4.7 in [26 ] ). Part (i) is the maximal coupling assertion; see also Remark 4.8 of the cited book. Part (ii) is a direct consequence of the definition of total variation. For probability laws ν , ν ′ \nu,\nu' ν , ν ′ on a finite space:
(i) There is a coupling ( Z , Z ′ ) (Z,Z') ( Z , Z ′ ) with these marginals such that Pr ( Z ≠ Z ′ ) = ∥ ν − ν ′ ∥ T V \Pr(Z\ne Z')=\|\nu-\nu'\|_{\mathrm{TV}} Pr ( Z = Z ′ ) = ∥ ν − ν ′ ∥ TV .
(ii) For any Markov kernel T T T , ∥ ν T − ν ′ T ∥ T V ≤ ∥ ν − ν ′ ∥ T V \|\nu T-\nu'T\|_{\mathrm{TV}}\leq\|\nu-\nu'\|_{\mathrm{TV}} ∥ ν T − ν ′ T ∥ TV ≤ ∥ ν − ν ′ ∥ TV .
The second assertion follows by integrating T ( ⋅ , E ) ∈ [ 0 , 1 ] T(\cdot,E)\in[0,1] T ( ⋅ , E ) ∈ [ 0 , 1 ] against ν − ν ′ \nu-\nu' ν − ν ′ . Both assertions apply to finite trajectory-segment spaces as well as to individual states.
Lemma B.7 (Equation 3.5 in [Rail19]). We use the bound in Section 3 of arXiv version 2. Let ξ j ∈ R d \xi_j\in\mathbb{R}^{d} ξ j ∈ R d be independent centered vectors satisfying ∑ j E ξ j ξ j T = I d \sum_j\mathbb{E}\xi_j\xi_j^{\mathsf{T}}=I_d ∑ j E ξ j ξ j T = I d , and let Z ∼ N ( 0 , I d ) Z\sim\mathcal{N}(0,I_d) Z ∼ N ( 0 , I d ) . Writing k d = 11.1 + 0.83 log d k_d=11.1+0.83\log d k d = 11.1 + 0.83 log d , the stated bound implies
W 1 ( ∑ j ξ j , Z ) ≤ ∑ j E [ ∥ ξ j ∥ 2 2 min { 4.5 , k d ∥ ξ j ∥ 2 } ] ≤ k d ∑ j E ∥ ξ j ∥ 2 3 . W_1\left(\sum_j\xi_j,Z\right)\leq\sum_j\mathbb{E}\left[\|\xi_j\|_2^2\min\{4.5,k_d\|\xi_j\|_2\}\right]\leq k_d\sum_j\mathbb{E}\|\xi_j\|_2^3. W 1 ( j ∑ ξ j , Z ) ≤ j ∑ E [ ∥ ξ j ∥ 2 2 min { 4.5 , k d ∥ ξ j ∥ 2 } ] ≤ k d j ∑ E ∥ ξ j ∥ 2 3 . The last inequality is used when the third absolute moments are finite. No identical-distribution assumption or smallness condition on their sum is required.
Lemma B.8 (Theorem 5.10 in [44 ] ). We specialize Kantorovich duality in Chapter 5 to Euclidean distance as cost. The map and mean bounds below are immediate consequences. For probability laws with finite first moments,
W 1 ( U , V ) = inf π ∈ Γ ( L ( U ) , L ( V ) ) ∫ ∥ u − v ∥ 2 d π = sup Lip 2 ( h ) ≤ 1 ∣ E h ( U ) − E h ( V ) ∣ . W_1(U,V)=\inf_{\pi\in\Gamma(\mathcal{L}(U),\mathcal{L}(V))}\int\|u-v\|_2\,\mathrm{d}\pi=\sup_{\operatorname{Lip}_2(h)\leq1}|\mathbb{E}h(U)-\mathbb{E}h(V)|. W 1 ( U , V ) = π ∈ Γ ( L ( U ) , L ( V )) inf ∫ ∥ u − v ∥ 2 d π = Lip 2 ( h ) ≤ 1 sup ∣ E h ( U ) − E h ( V ) ∣. Consequently, an L L L -Lipschitz map T T T satisfies W 1 ( T U , T V ) ≤ L W 1 ( U , V ) W_1(TU,TV)\leq LW_1(U,V) W 1 ( T U , T V ) ≤ L W 1 ( U , V ) , and W 1 ( U , V ) ≥ ∥ E U − E V ∥ 2 W_1(U,V)\geq\|\mathbb{E}U-\mathbb{E}V\|_2 W 1 ( U , V ) ≥ ∥ E U − E V ∥ 2 . These include coordinate projection and deterministic linear maps, with constants 1 1 1 and ∥ T ∥ o p \|T\|_{\mathrm{op}} ∥ T ∥ op respectively.
Lemma B.9 (Poisson equation). Suppose (S.2) holds. Let F θ F_\theta F θ be a family of bounded vector-valued functions with μ ˉ F θ = 0 \bar{\mu}F_\theta=0 μ ˉ F θ = 0 . Write ∥ F ∥ s u p = sup y ∥ F ( y ) ∥ ∞ \|F\|_{\mathrm{sup}}=\sup_y\|F(y)\|_\infty ∥ F ∥ sup = sup y ∥ F ( y ) ∥ ∞ . Then
h θ = ∑ j = 0 ∞ K j F θ , v θ = K h θ h_\theta=\sum_{j=0}^{\infty}K^jF_\theta,\qquad v_\theta=Kh_\theta h θ = j = 0 ∑ ∞ K j F θ , v θ = K h θ are well defined and satisfy h θ − K h θ = F θ h_\theta-Kh_\theta=F_\theta h θ − K h θ = F θ . Furthermore,
∥ h θ ∥ s u p + ∥ v θ ∥ s u p ≤ C ∥ F θ ∥ s u p . (131) \|h_\theta\|_{\mathrm{sup}}+\|v_\theta\|_{\mathrm{sup}}\leq C\|F_\theta\|_{\mathrm{sup}}. \tag*{(131)} ∥ h θ ∥ sup + ∥ v θ ∥ sup ≤ C ∥ F θ ∥ sup . (131) ∥ h θ − h θ ′ ∥ s u p + ∥ v θ − v θ ′ ∥ s u p ≤ C ∥ F θ − F θ ′ ∥ s u p . (132) \|h_\theta-h_{\theta'}\|_{\mathrm{sup}}+\|v_\theta-v_{\theta'}\|_{\mathrm{sup}}\leq C\|F_\theta-F_{\theta'}\|_{\mathrm{sup}}. \tag*{(132)} ∥ h θ − h θ ′ ∥ sup + ∥ v θ − v θ ′ ∥ sup ≤ C ∥ F θ − F θ ′ ∥ sup . (132)
If θ t \theta_t θ t is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, the family is jointly measurable, and E ∥ F θ t ∥ sup < ∞ \mathbb{E}\|F_{\theta_t}\|_{\sup}<\infty E ∥ F θ t ∥ s u p < ∞ , then for t ≥ 1 t\geq1 t ≥ 1 ,
h θ t ( Y t ) − v θ t ( Y t − 1 ) h_{\theta_t}(Y_t)-v_{\theta_t}(Y_{t-1}) h θ t ( Y t ) − v θ t ( Y t − 1 )
is a martingale difference.
Proof. Apply the bounded linear potential operator in Lemma B.3 to each centered F θ F_\theta F θ . Its norm bound and linearity give both the size estimate and the same estimate applied to F θ − F θ ′ F_\theta-F_{\theta'} F θ − F θ ′ ; K K K is a sup-norm contraction. For a predictable parameter, integrability follows from (B.99), and the Markov property gives
E [ h θ t ( Y t ) ∣ F t − 1 ] = ( K h θ t ) ( Y t − 1 ) = v θ t ( Y t − 1 ) . \mathbb{E}[h_{\theta_t}(Y_t)\mid\mathcal{F}_{t-1}]=(Kh_{\theta_t})(Y_{t-1})=v_{\theta_t}(Y_{t-1}). E [ h θ t ( Y t ) ∣ F t − 1 ] = ( K h θ t ) ( Y t − 1 ) = v θ t ( Y t − 1 ) .
Thus the asserted difference is a martingale difference. The decomposition used in Section B.1 is the exact identity
F θ t ( Y t ) = { h θ t ( Y t ) − v θ t ( Y t − 1 ) } + v θ t ( Y t − 1 ) − v θ t ( Y t ) . F_{\theta_t}(Y_t)=\{h_{\theta_t}(Y_t)-v_{\theta_t}(Y_{t-1})\}+v_{\theta_t}(Y_{t-1})-v_{\theta_t}(Y_t). F θ t ( Y t ) = { h θ t ( Y t ) − v θ t ( Y t − 1 )} + v θ t ( Y t − 1 ) − v θ t ( Y t ) .
The dependence on θ t \theta_t θ t is retained in the last two terms when summing over time.
Lemma B.10 (Martingale moment bounds). Let ( D j ) j = 1 N (D_j)_{j=1}^{N} ( D j ) j = 1 N be vector-valued martingale differences in R d \mathbb{R}^{d} R d with finite p p p th moments, where p ≥ 2 p\geq2 p ≥ 2 . Then
∥ ∑ j = 1 N D j ∥ L p ( ℓ ∞ ) ≤ C d p ( ∑ j = 1 N ∥ D j ∥ L p ( ℓ ∞ ) 2 ) 1 / 2 . (133) \left\|\sum_{j=1}^{N}D_j\right\|_{L^p(\ell_\infty)}
\leq C_{d p}\left(\sum_{j=1}^{N}\|D_j\|_{L^p(\ell_\infty)}^2\right)^{1/2}. \tag*{(133)} j = 1 ∑ N D j L p ( ℓ ∞ ) ≤ C d p ( j = 1 ∑ N ∥ D j ∥ L p ( ℓ ∞ ) 2 ) 1/2 . (133) If ∥ D j ∥ ∞ ≤ a j \|D_j\|_\infty\leq a_j ∥ D j ∥ ∞ ≤ a j almost surely for deterministic a j a_j a j , then
∥ max 0 ≤ k ≤ N ∥ ∑ j = 1 k D j ∥ ∞ ∥ L p ≤ C d p ∑ j = 1 N a j 2 . (134) \left\|\max_{0\leq k\leq N}\left\|\sum_{j=1}^{k}D_j\right\|_\infty\right\|_{L^p}
\leq C_d\sqrt{p\sum_{j=1}^{N}a_j^2}. \tag*{(134)} 0 ≤ k ≤ N max j = 1 ∑ k D j ∞ L p ≤ C d p j = 1 ∑ N a j 2 . (134) Proof. Apply Lemma B.4 (i) coordinatewise. For scalar martingale differences d j d_j d j , conditional Jensen and Minkowski give
∥ max j ∣ d j ∣ ∥ L p ≤ ( ∑ j ∥ d j ∥ L p 2 ) 1 / 2 , ∥ ( ∑ j E [ d j 2 ∣ F j − 1 ] ) 1 / 2 ∥ L p ≤ ( ∑ j ∥ d j ∥ L p 2 ) 1 / 2 . \|\max_j|d_j|\|_{L^p}\leq\left(\sum_j\|d_j\|_{L^p}^2\right)^{1/2},
\qquad
\left\|\left(\sum_j\mathbb{E}[d_j^2\mid\mathcal{F}_{j-1}]\right)^{1/2}\right\|_{L^p}
\leq\left(\sum_j\|d_j\|_{L^p}^2\right)^{1/2}. ∥ j max ∣ d j ∣ ∥ L p ≤ ( j ∑ ∥ d j ∥ L p 2 ) 1/2 , ( j ∑ E [ d j 2 ∣ F j − 1 ] ) 1/2 L p ≤ ( j ∑ ∥ d j ∥ L p 2 ) 1/2 . Since ∥ Z ∥ L p ( ℓ ∞ ) ≤ d 1 / p max i ∥ Z i ∥ L p \|Z\|_{L^p(\ell_\infty)}\leq d^{1/p}\max_i\|Z_i\|_{L^p} ∥ Z ∥ L p ( ℓ ∞ ) ≤ d 1/ p max i ∥ Z i ∥ L p , the first assertion follows, with the required factor C d p C_{d p} C d p . For the second assertion, part (ii) of Lemma B.4 and a coordinate union bound yield, with V = ∑ j a j 2 > 0 V=\sum_j a_j^2>0 V = ∑ j a j 2 > 0 ,
Pr { max k ≤ N ∥ ∑ j ≤ k D j ∥ ∞ > x } ≤ 2 d e − x 2 / ( 2 V ) . \Pr\left\{\max_{k\leq N}\left\|\sum_{j\leq k}D_j\right\|_\infty>x\right\}\leq2d e^{-x^2/(2V)}. Pr ⎩ ⎨ ⎧ k ≤ N max j ≤ k ∑ D j ∞ > x ⎭ ⎬ ⎫ ≤ 2 d e − x 2 / ( 2 V ) . Integrating this tail gives an L p L^p L p bound by ( 2 d ) 1 / p ( 2 V ) 1 / 2 Γ ( 1 + p / 2 ) 1 / p ≤ C d p V (2d)^{1/p}(2V)^{1/2}\Gamma(1+p/2)^{1/p}\leq C_d\sqrt{pV} ( 2 d ) 1/ p ( 2 V ) 1/2 Γ ( 1 + p /2 ) 1/ p ≤ C d p V . The case V = 0 V=0 V = 0 is immediate.
Lemma B.11 (Gaussian covariance comparison). For positive-definite matrices V 1 , V 2 V_1,V_2 V 1 , V 2 ,
W 1 ( N ( 0 , V 1 ) , N ( 0 , V 2 ) ) ≤ ∥ V 1 − V 2 ∥ F λ min ( V 1 ) + λ min ( V 2 ) . (135) W_1\bigl(\mathcal{N}(0,V_1),\mathcal{N}(0,V_2)\bigr)
\leq\frac{\|V_1-V_2\|_F}{\sqrt{\lambda_{\min}(V_1)}+\sqrt{\lambda_{\min}(V_2)}}. \tag*{(135)} W 1 ( N ( 0 , V 1 ) , N ( 0 , V 2 ) ) ≤ λ m i n ( V 1 ) + λ m i n ( V 2 ) ∥ V 1 − V 2 ∥ F . (135)
Proof. Let U = V 1 1 / 2 U = V_1^{1/2} U = V 1 1/2 , V = V 2 1 / 2 V = V_2^{1/2} V = V 2 1/2 , and E = U − V E = U - V E = U − V . Then U E + E V = V 1 − V 2 UE + EV = V_1 - V_2 U E + E V = V 1 − V 2 , without a commutativity assumption. Lemma B.5 gives
∥ E ∥ F ≤ ∥ V 1 − V 2 ∥ F λ min ( V 1 ) + λ min ( V 2 ) . \|E\|_F \le\frac{\|V_1 - V_2\|_F}{\sqrt{\lambda_{\min}(V_1)} + \sqrt{\lambda_{\min}(V_2)}}. ∥ E ∥ F ≤ λ m i n ( V 1 ) + λ m i n ( V 2 ) ∥ V 1 − V 2 ∥ F .
Couple the Gaussians as U Z , V Z UZ,VZ U Z , V Z with Z ∼ N ( 0 , I d ) Z \sim\mathcal{N}(0,I_d) Z ∼ N ( 0 , I d ) . Then W 1 ( U Z , V Z ) ≤ E ∥ E Z ∥ 2 ≤ ∥ E ∥ F W_1(UZ,VZ) \le\mathbb{E}\|EZ\|_2 \le\|E\|_F W 1 ( U Z , V Z ) ≤ E ∥ E Z ∥ 2 ≤ ∥ E ∥ F , proving the claim.
Lemma B.12 (Coupling separated trajectory intervals). Suppose (S.2) holds. Let I 1 , … , I N I_1,\ldots,I_N I 1 , … , I N be ordered intervals of time indices. Suppose at least m m m indices precede I 1 I_1 I 1 and at least m m m indices lie between successive intervals. There are independent stationary trajectory segments ( Y t j : t ∈ I j ) (Y_t^j:t\in I_j) ( Y t j : t ∈ I j ) , each of the corresponding length, that can be coupled with the original chain so that
P { some retained segment differs from its copy } ≤ C N e − c m i x m . (136) \mathbb{P}\{\text{some retained segment differs from its copy}\} \le CNe^{-c_{\mathrm{mix}}m}. \tag*{(136)} P { some retained segment differs from its copy } ≤ C N e − c mix m . (136)
Proof. Write I j = { a j , … , b j } I_j=\{a_j,\ldots,b_j\} I j = { a j , … , b j } , let Y j \mathcal{Y}_j Y j be its trajectory segment, and let ν j \nu_j ν j be its stationary segment law. For j ≥ 2 j\ge2 j ≥ 2 , conditional on the preceding retained segments, the law at a j a_j a j is K a j − b j − 1 ( Y b j − 1 , ⋅ ) K^{a_j-b_{j-1}}(Y_{b_{j-1}},\cdot) K a j − b j − 1 ( Y b j − 1 , ⋅ ) . The gap condition and (S.2) bound its total variation distance from μ ˉ \bar{\mu} μ ˉ by C e − c m i x m Ce^{-c_{\mathrm{mix}}m} C e − c mix m . The initial discarded interval gives the same bound for j = 1 j=1 j = 1 . Applying the segment transition kernel preserves this bound, by Lemma B.6 (ii). Telescoping the joint law against the product of the ν j \nu_j ν j therefore gives
∥ L ( Y 1 , … , Y N ) − ⨂ j = 1 N ν j ∥ T V ≤ C N e − c m i x m . \left\|\mathcal{L}(\mathcal{Y}_1,\ldots,\mathcal{Y}_N)-\bigotimes_{j=1}^{N}\nu_j\right\|_{\mathrm{TV}} \le CNe^{-c_{\mathrm{mix}}m}. L ( Y 1 , … , Y N ) − j = 1 ⨂ N ν j TV ≤ C N e − c mix m .
The maximal coupling in Lemma B.6 (i) proves the desired bound. Its product marginal consists of independent stationary segments. Sampling the copies conditionally on the retained original segments realizes this coupling on an extension of the original probability space; all retained and discarded original observations keep their original joint law.
Lemma B.13 (Independent-sum Gaussian approximation). Let ξ 1 , … , ξ N \xi_1,\ldots,\xi_N ξ 1 , … , ξ N be independent centered vectors in R d \mathbb{R}^d R d , with ∑ j E ξ j ξ j T = I d \sum_j\mathbb{E}\xi_j\xi_j^{\mathsf{T}}=I_d ∑ j E ξ j ξ j T = I d . Let ρ 3 = ∑ j E ∥ ξ j ∥ 2 3 \rho_3=\sum_j\mathbb{E}\|\xi_j\|_2^3 ρ 3 = ∑ j E ∥ ξ j ∥ 2 3 and Z ∼ N ( 0 , I d ) Z\sim\mathcal{N}(0,I_d) Z ∼ N ( 0 , I d ) . If 0 < ρ 3 ≤ 1 / 2 0<\rho_3\le1/2 0 < ρ 3 ≤ 1/2 , then
W 1 ( ∑ j ξ j , Z ) ≤ C d ρ 3 ( 1 + ∣ log ρ 3 ∣ ) . (137) W_1\left(\sum_j\xi_j,Z\right)\le Cd\rho_3(1+|\log\rho_3|). \tag*{(137)} W 1 ( j ∑ ξ j , Z ) ≤ C d ρ 3 ( 1 + ∣ log ρ 3 ∣ ) . (137)
Proof. Lemma B.7 , with k d = 11.1 + 0.83 log d k_d=11.1+0.83\log d k d = 11.1 + 0.83 log d , gives the stronger estimate
W 1 ( ∑ j ξ j , Z ) ≤ k d ∑ j E ∥ ξ j ∥ 2 3 = k d ρ 3 . W_1\left(\sum_j\xi_j,Z\right)\le k_d\sum_j\mathbb{E}\|\xi_j\|_2^3=k_d\rho_3. W 1 ( j ∑ ξ j , Z ) ≤ k d j ∑ E ∥ ξ j ∥ 2 3 = k d ρ 3 .
This implies (B.105). Independence, centering, and identity total covariance are exactly the hypotheses of that lemma; identical distributions are not required.
Proof of Theorem 3.2
In this section, we establish the lower bound. Proposition 3.1 and Proposition 3.2 are discussed in the main body. In the proofs below, vector norms without a subscript are Euclidean norms, and matrix norms without a subscript are operator norms.
Theorem 3.2. For the problem in (S.11), every fixed β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) , and c 0 > 0 c_0 > 0 c 0 > 0 , there exist c > 0 c > 0 c > 0 and n 0 < ∞ n_0 < \infty n 0 < ∞ such that, for n ≥ n 0 n \ge n_0 n ≥ n 0 ,
W 1 ( X n , Z ∞ ) ≥ c { n 1 / 2 − β , 1 / 2 < β ≤ 3 / 4 , n β − 1 , 3 / 4 < β < 1. (138) W_1(X_n,Z_\infty) \ge c
\begin{cases}
n^{1/2-\beta}, & 1/2 < \beta\le3/4,\\
n^{\beta-1}, & 3/4 < \beta< 1.
\end{cases} \tag*{(138)} W 1 ( X n , Z ∞ ) ≥ c { n 1/2 − β , n β − 1 , 1/2 < β ≤ 3/4 , 3/4 < β < 1. (138) The problem and initialization are independent of β , c 0 , b \beta,c_0,b β , c 0 , b . The constants may depend on these parameters.
Proof of Theorem 3.2. We first verify the assumptions for the fixed problem. Solving μ ⊤ P = μ ⊤ \mu^\top P=\mu^\top μ ⊤ P = μ ⊤ gives μ = ( 3 / 5 , 2 / 5 ) ⊤ \mu=(3/5,2/5)^\top μ = ( 3/5 , 2/5 ) ⊤ . Let Π = 1 μ ⊤ \Pi=\mathbf{1}\mu^\top Π = 1 μ ⊤ . By direct multiplication,
P = Π + 1 2 ( I − Π ) , Π 2 = Π , Π ( I − Π ) = ( I − Π ) Π = 0. P=\Pi+\frac{1}{2}(I-\Pi),\qquad\Pi^2=\Pi,\qquad\Pi(I-\Pi)=(I-\Pi)\Pi=0. P = Π + 2 1 ( I − Π ) , Π 2 = Π , Π ( I − Π ) = ( I − Π ) Π = 0.
Therefore,
P j = Π + 2 − j ( I − Π ) , sup i ∥ P j ( i , ⋅ ) − μ ∥ T V ≤ 2 − j . (139) P^j=\Pi+2^{-j}(I-\Pi),\qquad\sup_i\left\|P^j(i,\cdot)-\mu\right\|_{\mathrm{TV}}\le2^{-j}. \tag*{(139)} P j = Π + 2 − j ( I − Π ) , i sup P j ( i , ⋅ ) − μ TV ≤ 2 − j . (139)
With the unique action suppressed, we identify the transition triple Y t Y_t Y t with the pair ( s t , s t + 1 ) (s_t,s_{t+1}) ( s t , s t + 1 ) . Conditional on Y 0 = ( i , j ) Y_0=(i,j) Y 0 = ( i , j ) , its law after h ≥ 1 h\ge1 h ≥ 1 steps is obtained by starting the state chain at j j j , applying h − 1 h-1 h − 1 transitions, and then drawing one further transition. Applying a Markov kernel cannot increase total variation distance. Thus its distance from the stationary pair law is at most 2 − ( h − 1 ) 2^{-(h-1)} 2 − ( h − 1 ) . Increasing the constant also covers h = 0 h=0 h = 0 . This proves geometric mixing for the transition chain. Exploration holds since min i μ i = 2 / 5 \min_i\mu_i=2/5 min i μ i = 2/5 .
The Bellman system and its inverse are
I − γ P = 1 25 ( 9 − 4 − 6 11 ) , ( I − γ P ) − 1 = 1 3 ( 11 4 6 9 ) . I-\gamma P=\frac{1}{25}\begin{pmatrix}9&-4\\-6&11\end{pmatrix},\qquad
(I-\gamma P)^{-1}=\frac{1}{3}\begin{pmatrix}11&4\\6&9\end{pmatrix}. I − γ P = 25 1 ( 9 − 6 − 4 11 ) , ( I − γ P ) − 1 = 3 1 ( 11 6 4 9 ) .
Multiplying the inverse by r r r gives Q ⋆ = ( 4 / 3 , 3 ) ⊤ Q^\star=(4/3,3)^\top Q ⋆ = ( 4/3 , 3 ) ⊤ . Also,
A = D ( I − γ P ) = 1 125 ( 27 − 12 − 12 22 ) , λ ± ( A ) = 49 ± 601 250 > 0. A=D(I-\gamma P)=\frac{1}{125}\begin{pmatrix}27&-12\\-12&22\end{pmatrix},\qquad
\lambda_\pm(A)=\frac{49\pm\sqrt{601}}{250}>0. A = D ( I − γ P ) = 125 1 ( 27 − 12 − 12 22 ) , λ ± ( A ) = 250 49 ± 601 > 0.
Since there is one action, the mean update at Q ⋆ + δ Q^\star+\delta Q ⋆ + δ is exactly − A δ -A\delta − A δ . Hence the quadratic remainder assumption holds with zero remainder.
To calculate the fixed-point noise covariance, write ξ i j = r i + γ Q j ⋆ − Q i ⋆ \xi_{ij}=r_i+\gamma Q^\star_j-Q^\star_i ξ ij = r i + γ Q j ⋆ − Q i ⋆ . The four values are
( ξ i j ) i , j = 1 2 = ( − 4 / 15 16 / 15 − 14 / 15 2 / 5 ) . (140) (\xi_{ij})^2_{i,j=1}=\begin{pmatrix}-4/15&16/15\\-14/15&2/5\end{pmatrix}. \tag*{(140)} ( ξ ij ) i , j = 1 2 = ( − 4/15 − 14/15 16/15 2/5 ) . (140)
Their conditional means are zero in both rows. Only the coordinate corresponding to s t s_t s t is updated, so the off-diagonal entries of S S S vanish. Its diagonal entries are
S 11 = 3 5 { 4 5 16 225 + 1 5 256 225 } = 64 375 , S_{11}=\frac{3}{5}\left\{\frac{4}{5}\frac{16}{225}+\frac{1}{5}\frac{256}{225}\right\}=\frac{64}{375}, S 11 = 5 3 { 5 4 225 16 + 5 1 225 256 } = 375 64 ,
S 22 = 2 5 { 3 10 196 225 + 7 10 4 25 } = 56 375 . S_{22}=\frac{2}{5}\left\{\frac{3}{10}\frac{196}{225}+\frac{7}{10}\frac{4}{25}\right\}=\frac{56}{375}. S 22 = 5 2 { 10 3 225 196 + 10 7 25 4 } = 375 56 .
Thus S ≻ 0 S\succ0 S ≻ 0 , and invertibility of A A A gives Σ ∞ ≻ 0 \Sigma_\infty\succ0 Σ ∞ ≻ 0 . For reference, the inverse and the limiting covariance are
A − 1 = ( 55 / 9 10 / 3 10 / 3 15 / 2 ) , Σ ∞ = 1 243 ( 1952 1752 1752 2502 ) . (141) A^{-1}=\begin{pmatrix}55/9&10/3\\10/3&15/2\end{pmatrix},\qquad
\Sigma_\infty=\frac{1}{243}\begin{pmatrix}1952&1752\\1752&2502\end{pmatrix}. \tag*{(141)} A − 1 = ( 55/9 10/3 10/3 15/2 ) , Σ ∞ = 243 1 ( 1952 1752 1752 2502 ) . (141)
The initialization Q 0 = 0 Q_0 = 0 Q 0 = 0 belongs to [ 0 , 5 ] 2 [0,5]^2 [ 0 , 5 ] 2 . Since α t ≤ 1 \alpha_t \le1 α t ≤ 1 , each updated coordinate is a convex combination of its old value and a value in this interval. Thus the iterates remain bounded and all the first moments used below are finite.
Suppose first that 1 / 2 < β ≤ 3 / 4 1/2 < \beta\le3/4 1/2 < β ≤ 3/4 . Let w = v / ∥ v ∥ 2 w = v / \lVert v\rVert_2 w = v / ∥ v ∥ 2 , which is a fixed unit vector. For every coupling of X n X_n X n and Z ∞ Z_\infty Z ∞ , we have
E ∥ X n − Z ∞ ∥ 2 ≥ ∣ E [ w ⊤ ( X n − Z ∞ ) ] ∣ . \mathbb{E}\lVert X_n-Z_\infty\rVert_2 \ge\left\lvert\mathbb{E}\left[w^\top(X_n-Z_\infty)\right]\right\rvert. E ∥ X n − Z ∞ ∥ 2 ≥ E [ w ⊤ ( X n − Z ∞ ) ] .
Taking the infimum over couplings and using E Z ∞ = 0 \mathbb{E}Z_\infty= 0 E Z ∞ = 0 , Proposition 3.1 gives
W 1 ( X n , Z ∞ ) ≥ ∣ w ⊤ E X n ∣ = c 0 ∥ v ∥ 2 1 − β n 1 / 2 − β + o ( n 1 / 2 − β ) ≥ c 0 ∥ v ∥ 2 2 ( 1 − β ) n 1 / 2 − β . (142) W_1(X_n,Z_\infty) \ge\lvert w^\top\mathbb{E}X_n\rvert
= \frac{c_0\lVert v\rVert_2}{1-\beta}n^{1/2-\beta}+o(n^{1/2-\beta})
\ge\frac{c_0\lVert v\rVert_2}{2(1-\beta)}n^{1/2-\beta}. \tag*{(142)} W 1 ( X n , Z ∞ ) ≥ ∣ w ⊤ E X n ∣ = 1 − β c 0 ∥ v ∥ 2 n 1/2 − β + o ( n 1/2 − β ) ≥ 2 ( 1 − β ) c 0 ∥ v ∥ 2 n 1/2 − β . (142)
for all sufficiently large n n n . The coefficient is strictly positive because v ≠ 0 v\ne0 v = 0 . This proves the first regime, including β = 3 / 4 \beta=3/4 β = 3/4 .
Now suppose that 3 / 4 < β < 1 3/4 < \beta< 1 3/4 < β < 1 and put δ n = n β − 1 \delta_n = n^{\beta-1} δ n = n β − 1 . By Proposition 3.2, C n → Σ ∞ C_n\to\Sigma_\infty C n → Σ ∞ . Since the latter is positive definite, the eigenvalues of C n C_n C n are bounded below by a fixed positive constant for all sufficiently large n n n . We can therefore use the finite-covariance approximation from Propositions B.1 and B.2 . The identity X n = M n + R n X_n=M_n+R_n X n = M n + R n gives a coupling on the original space, so
W 1 ( X n , Z n ) ≤ E ∥ R n ∥ 2 + W 1 ( M n , Z n ) ≤ C { n − β / 2 + n 1 / 2 − β + ℓ n n − 1 / 2 + ℓ n n − 1 / 4 } . (143) W_1(X_n,Z_n) \le\mathbb{E}\lVert R_n\rVert_2+W_1(M_n,Z_n)
\le C\{\smash{n^{-\beta/2}+n^{1/2-\beta}+\ell_n n^{-1/2}+\ell_n n^{-1/4}}\}. \tag*{(143)} W 1 ( X n , Z n ) ≤ E ∥ R n ∥ 2 + W 1 ( M n , Z n ) ≤ C { n − β /2 + n 1/2 − β + ℓ n n − 1/2 + ℓ n n − 1/4 } . (143)
We compare each term with δ n \delta_n δ n . The four ratios are
n 1 − 3 β / 2 , n 3 / 2 − 2 β , ℓ n n 1 / 2 − β , ℓ n n 3 / 4 − β . (144) n^{1-3\beta/2},\qquad n^{3/2-2\beta},\qquad\ell_n n^{1/2-\beta},\qquad\ell_n n^{3/4-\beta}. \tag*{(144)} n 1 − 3 β /2 , n 3/2 − 2 β , ℓ n n 1/2 − β , ℓ n n 3/4 − β . (144)
All four converge to zero in the stated range. For the first ratio it is enough that β > 2 / 3 \beta>2/3 β > 2/3 . The second and fourth require β > 3 / 4 \beta>3/4 β > 3/4 , while the third requires only β > 1 / 2 \beta>1/2 β > 1/2 . Since ℓ n = O ( log n ) \ell_n=O(\log n) ℓ n = O ( log n ) for fixed b b b , it does not change any of these strict exponent comparisons. Consequently,
W 1 ( X n , Z n ) = o ( δ n ) . (145) W_1(X_n,Z_n)=o(\delta_n). \tag*{(145)} W 1 ( X n , Z n ) = o ( δ n ) . (145)
Fix a coordinate j j j and write σ j 2 = ( Σ ∞ ) j j > 0 \sigma_j^2=(\Sigma_\infty)_{jj}>0 σ j 2 = ( Σ ∞ ) j j > 0 and g j = G j j > 0 g_j=G_{jj}>0 g j = G j j > 0 . The covariance expansion gives
( C n ) j j = σ j 2 + g j δ n + o ( δ n ) . (C_n)_{jj}=\sigma_j^2+g_j\delta_n+o(\delta_n). ( C n ) j j = σ j 2 + g j δ n + o ( δ n ) .
Let σ n , j = ( C n ) j j \sigma_{n,j}=\sqrt{(C_n)_{jj}} σ n , j = ( C n ) j j . Rationalizing the difference of square roots, we obtain
σ n , j − σ j = ( C n ) j j − σ j 2 σ n , j + σ j = g j 2 σ j δ n + o ( δ n ) . (146) \sigma_{n,j}-\sigma_j
=\frac{(C_n)_{jj}-\sigma_j^2}{\sigma_{n,j}+\sigma_j}
=\frac{g_j}{2\sigma_j}\delta_n+o(\delta_n). \tag*{(146)} σ n , j − σ j = σ n , j + σ j ( C n ) j j − σ j 2 = 2 σ j g j δ n + o ( δ n ) . (146)
By Lemma C.10 , projection onto coordinate j j j and the exact scalar Gaussian formula give
W 1 ( Z n , Z ∞ ) ≥ 2 π ∣ σ n , j − σ j ∣ = 2 π g j 2 σ j δ n + o ( δ n ) . (147) W_1(Z_n,Z_\infty)\ge\sqrt{\frac{2}{\pi}}\lvert\sigma_{n,j}-\sigma_j\rvert
=\sqrt{\frac{2}{\pi}}\frac{g_j}{2\sigma_j}\delta_n+o(\delta_n). \tag*{(147)} W 1 ( Z n , Z ∞ ) ≥ π 2 ∣ σ n , j − σ j ∣ = π 2 2 σ j g j δ n + o ( δ n ) . (147)
The leading coefficient is strictly positive. Applying the reverse triangle inequality and (C.7), we conclude that
W 1 ( X n , Z ∞ ) ≥ W 1 ( Z n , Z ∞ ) − W 1 ( X n , Z n ) ≥ c δ n W_1(X_n,Z_\infty)\ge W_1(Z_n,Z_\infty)-W_1(X_n,Z_n)\ge c\delta_n W 1 ( X n , Z ∞ ) ≥ W 1 ( Z n , Z ∞ ) − W 1 ( X n , Z n ) ≥ c δ n
for all sufficiently large n n n . At β = 3 / 4 \beta=3/4 β = 3/4 , the last ratio in (C.6) does not tend to zero under the available bound. That endpoint has already been covered by (C.4). Increasing the threshold n 0 n_0 n 0 to include all the preceding requirements completes the proof.
Proof of Proposition 3.1
Proposition 3.1. On (S.11), with stationary state initialization ,
E Δ t = α t v + O ( α t 2 + ( t + b ) − 1 ) , v = ( 248 / 1125 − 34 / 375 ) ≠ 0. (148) \mathbb{E}\Delta_t=\alpha_t v+O(\alpha_t^2+(t+b)^{-1}),\qquad v=\begin{pmatrix}248/1125\\-34/375\end{pmatrix}\ne0. \tag*{(148)} E Δ t = α t v + O ( α t 2 + ( t + b ) − 1 ) , v = ( 248/1125 − 34/375 ) = 0. (148) For any fixed bounded deterministic Q 0 Q_0 Q 0 , this implies
E X n = c 0 1 − β v n 1 / 2 − β + o ( n 1 / 2 − β ) . (149) \mathbb{E}X_n=\frac{c_0}{1-\beta}v n^{1/2-\beta}+o(n^{1/2-\beta}). \tag*{(149)} E X n = 1 − β c 0 v n 1/2 − β + o ( n 1/2 − β ) . (149)
Proof. We first derive a closed recursion for the joint first moments. For i ∈ { 1 , 2 } i\in\{1,2\} i ∈ { 1 , 2 } , define the two-dimensional vector z t ( i ) = E [ Δ t 1 { s t = i } ] z_t^{(i)}=\mathbb{E}[\Delta_t1\{s_t=i\}] z t ( i ) = E [ Δ t 1 { s t = i }] , and let
z t = ( z t ( 1 ) z t ( 2 ) ) = ( E [ Δ t , 1 1 { s t = 1 } ] E [ Δ t , 2 1 { s t = 1 } ] E [ Δ t , 1 1 { s t = 2 } ] E [ Δ t , 2 1 { s t = 2 } ] ) , L 0 = ( 1 0 1 0 0 1 0 1 ) . (150) z_t=\begin{pmatrix}z_t^{(1)}\\z_t^{(2)}\end{pmatrix}
=\begin{pmatrix}
\mathbb{E}[\Delta_{t,1}1\{s_t=1\}]\\
\mathbb{E}[\Delta_{t,2}1\{s_t=1\}]\\
\mathbb{E}[\Delta_{t,1}1\{s_t=2\}]\\
\mathbb{E}[\Delta_{t,2}1\{s_t=2\}]
\end{pmatrix},
\qquad
L_0=\begin{pmatrix}1&0&1&0\\0&1&0&1\end{pmatrix}. \tag*{(150)} z t = ( z t ( 1 ) z t ( 2 ) ) = E [ Δ t , 1 1 { s t = 1 }] E [ Δ t , 2 1 { s t = 1 }] E [ Δ t , 1 1 { s t = 2 }] E [ Δ t , 2 1 { s t = 2 }] , L 0 = ( 1 0 0 1 1 0 0 1 ) . (150)
By definition, L 0 z t = E Δ t L_0z_t=\mathbb{E}\Delta_t L 0 z t = E Δ t . The iterate and the current state need not be independent. We keep their joint moments when taking expectations.
Conditional on s t = i s_t=i s t = i , s t + 1 = j s_{t+1}=j s t + 1 = j , the error update is
Δ t + 1 = { I + α t u i ( γ u j − u i ) T } Δ t + α t u i ξ i j , (151) \Delta_{t+1}=\{I+\alpha_tu_i(\gamma u_j-u_i)^\mathsf{T}\}\Delta_t+\alpha_tu_i\xi_{ij}, \tag*{(151)} Δ t + 1 = { I + α t u i ( γ u j − u i ) T } Δ t + α t u i ξ ij , (151)
where ξ i j \xi_{ij} ξ ij is given in (C.2). Given the current state and the past, the next state has law P ( s t , ⋅ ) P(s_t,\cdot) P ( s t , ⋅ ) . Therefore,
E [ Δ t 1 { s t = i , s t + 1 = j } ] = P i j z t ( i ) . \mathbb{E}[\Delta_t1\{s_t=i,s_{t+1}=j\}]=P_{ij}z_t^{(i)}. E [ Δ t 1 { s t = i , s t + 1 = j }] = P ij z t ( i ) .
The state law is stationary at every time, so P ( s t = i , s t + 1 = j ) = μ i P i j \mathbb{P}(s_t=i,s_{t+1}=j)=\mu_iP_{ij} P ( s t = i , s t + 1 = j ) = μ i P ij . Multiplying (C.11) by the indicator of the next state and taking expectations gives
z t + 1 ( j ) = ∑ i = 1 2 P i j { I + α t u i ( γ u j − u i ) T } z t ( i ) + α t ∑ i = 1 2 μ i P i j u i ξ i j . (152) z_{t+1}^{(j)}=\sum_{i=1}^{2}P_{ij}\{I+\alpha_tu_i(\gamma u_j-u_i)^\mathsf{T}\}z_t^{(i)}+\alpha_t\sum_{i=1}^{2}\mu_iP_{ij}u_i\xi_{ij}. \tag*{(152)} z t + 1 ( j ) = i = 1 ∑ 2 P ij { I + α t u i ( γ u j − u i ) T } z t ( i ) + α t i = 1 ∑ 2 μ i P ij u i ξ ij . (152)
The forcing is constant in time because of stationary initialization. Writing z t , k z_{t,k} z t , k for the k k k th coordinate of z t z_t z t , the four rows of (C.12) are
z t + 1 , 1 = 4 5 z t , 1 + 3 10 z t , 3 − α t 4 25 z t , 1 − α t 16 125 , z t + 1 , 2 = 4 5 z t , 2 + 3 10 z t , 4 + α t ( 6 25 z t , 3 − 3 10 z t , 4 ) − α t 14 125 , z t + 1 , 3 = 1 5 z t , 1 + 7 10 z t , 3 + α t ( − 1 5 z t , 1 + 4 25 z t , 2 ) + α t 16 125 , z t + 1 , 4 = 1 5 z t , 2 + 7 10 z t , 4 − α t 7 50 z t , 4 + α t 14 125 . (153) \begin{aligned}
z_{t+1,1}&=\frac{4}{5}z_{t,1}+\frac{3}{10}z_{t,3}-\alpha_t\frac{4}{25}z_{t,1}-\alpha_t\frac{16}{125},\\
z_{t+1,2}&=\frac{4}{5}z_{t,2}+\frac{3}{10}z_{t,4}+\alpha_t\left(\frac{6}{25}z_{t,3}-\frac{3}{10}z_{t,4}\right)-\alpha_t\frac{14}{125},\\
z_{t+1,3}&=\frac{1}{5}z_{t,1}+\frac{7}{10}z_{t,3}+\alpha_t\left(-\frac{1}{5}z_{t,1}+\frac{4}{25}z_{t,2}\right)+\alpha_t\frac{16}{125},\\
z_{t+1,4}&=\frac{1}{5}z_{t,2}+\frac{7}{10}z_{t,4}-\alpha_t\frac{7}{50}z_{t,4}+\alpha_t\frac{14}{125}. \tag*{(153)}
\end{aligned} z t + 1 , 1 z t + 1 , 2 z t + 1 , 3 z t + 1 , 4 = 5 4 z t , 1 + 10 3 z t , 3 − α t 25 4 z t , 1 − α t 125 16 , = 5 4 z t , 2 + 10 3 z t , 4 + α t ( 25 6 z t , 3 − 10 3 z t , 4 ) − α t 125 14 , = 5 1 z t , 1 + 10 7 z t , 3 + α t ( − 5 1 z t , 1 + 25 4 z t , 2 ) + α t 125 16 , = 5 1 z t , 2 + 10 7 z t , 4 − α t 50 7 z t , 4 + α t 125 14 . (153)
For example, the forcing in the first row is μ 1 P 11 ξ 11 = − 16 / 125 \mu_1P_{11}\xi_{11}=-16/125 μ 1 P 11 ξ 11 = − 16/125 . For the second row it is μ 2 P 21 ξ 21 = − 14 / 125 \mu_2P_{21}\xi_{21}=-14/125 μ 2 P 21 ξ 21 = − 14/125 . The other two forcing entries have the opposite signs.
Thus, the recursion can be written as
z t + 1 = ( T 0 + α t T 1 ) z t + α t g , g = 1 125 ( − 16 − 14 16 14 ) , (154) z_{t+1}=(T_0+\alpha_tT_1)z_t+\alpha_t g,\qquad g=\frac{1}{125}\begin{pmatrix}-16\\-14\\16\\14\end{pmatrix}, \tag*{(154)} z t + 1 = ( T 0 + α t T 1 ) z t + α t g , g = 125 1 − 16 − 14 16 14 , (154)
where
T 0 = ( 4 / 5 0 3 / 10 0 0 4 / 5 0 3 / 10 1 / 5 0 7 / 10 0 0 1 / 5 0 7 / 10 ) , T 1 = ( − 4 / 25 0 0 0 0 0 6 / 25 − 3 / 10 − 1 / 5 4 / 25 0 0 0 0 0 − 7 / 50 ) . (155) T_0=\begin{pmatrix}4/5&0&3/10&0\\0&4/5&0&3/10\\1/5&0&7/10&0\\0&1/5&0&7/10\end{pmatrix},\qquad T_1=\begin{pmatrix}-4/25&0&0&0\\0&0&6/25&-3/10\\-1/5&4/25&0&0\\0&0&0&-7/50\end{pmatrix}. \tag*{(155)} T 0 = 4/5 0 1/5 0 0 4/5 0 1/5 3/10 0 7/10 0 0 3/10 0 7/10 , T 1 = − 4/25 0 − 1/5 0 0 0 4/25 0 0 6/25 0 0 0 − 3/10 0 − 7/50 . (155)
We next change coordinates so that the overall mean is the first block. Define
J 0 = ( 3 / 5 0 0 3 / 5 2 / 5 0 0 2 / 5 ) , F 0 = ( 1 0 0 1 − 1 0 0 − 1 ) , M 0 = [ J 0 F 0 ] . (156) J_0=\begin{pmatrix}3/5&0\\0&3/5\\2/5&0\\0&2/5\end{pmatrix},\qquad F_0=\begin{pmatrix}1&0\\0&1\\-1&0\\0&-1\end{pmatrix},\qquad M_0=[J_0\ F_0]. \tag*{(156)} J 0 = 3/5 0 2/5 0 0 3/5 0 2/5 , F 0 = 1 0 − 1 0 0 1 0 − 1 , M 0 = [ J 0 F 0 ] . (156)
The inverse is
M 0 − 1 = ( 1 0 1 0 0 1 0 1 2 / 5 0 − 3 / 5 0 0 2 / 5 0 − 3 / 5 ) . M_0^{-1}=\begin{pmatrix}1&0&1&0\\0&1&0&1\\2/5&0&-3/5&0\\0&2/5&0&-3/5\end{pmatrix}. M 0 − 1 = 1 0 2/5 0 0 1 0 2/5 1 0 − 3/5 0 0 1 0 − 3/5 .
Write z = M 0 ( u , f ) T z=M_0(u,f)^\mathsf{T} z = M 0 ( u , f ) T . Then
u = z ( 1 ) + z ( 2 ) = L 0 z , f = 2 5 z ( 1 ) − 3 5 z ( 2 ) , z ( 1 ) = 3 5 u + f , z ( 2 ) = 2 5 u − f . (157) u=z^{(1)}+z^{(2)}=L_0z,\qquad f=\frac{2}{5}z^{(1)}-\frac{3}{5}z^{(2)},\qquad z^{(1)}=\frac{3}{5}u+f,\qquad z^{(2)}=\frac{2}{5}u-f. \tag*{(157)} u = z ( 1 ) + z ( 2 ) = L 0 z , f = 5 2 z ( 1 ) − 5 3 z ( 2 ) , z ( 1 ) = 5 3 u + f , z ( 2 ) = 5 2 u − f . (157)
In particular, u t = E Δ t u_t=\mathbb{E}\Delta_t u t = E Δ t . The second block is zero when each joint moment equals the stationary state probability times the overall mean.
Multiplying the matrices in (C.15) by this change of coordinates gives
M 0 − 1 ( T 0 + α T 1 ) M 0 = ( I − α A a B 0 a C 0 1 2 I + a E 0 ) , M 0 − 1 g = ( 0 g f ) , (158) M_0^{-1}(T_0+\alpha T_1)M_0=\begin{pmatrix}I-\alpha A&aB_0\\aC_0&\frac{1}{2}I+aE_0\end{pmatrix},\qquad M_0^{-1}g=\begin{pmatrix}0\\g_f\end{pmatrix}, \tag*{(158)} M 0 − 1 ( T 0 + α T 1 ) M 0 = ( I − α A a C 0 a B 0 2 1 I + a E 0 ) , M 0 − 1 g = ( 0 g f ) , (158)
where
B 0 = 1 25 ( − 9 4 − 6 11 ) , C 0 = 1 625 ( 21 − 36 24 − 9 ) , B_0=\frac{1}{25}\begin{pmatrix}-9&4\\-6&11\end{pmatrix},\qquad C_0=\frac{1}{625}\begin{pmatrix}21&-36\\24&-9\end{pmatrix}, B 0 = 25 1 ( − 9 − 6 4 11 ) , C 0 = 625 1 ( 21 24 − 36 − 9 ) ,
E 0 = 1 250 ( 14 − 24 − 24 9 ) , g f = 1 125 ( − 16 − 14 ) . (159) E_0=\frac{1}{250}\begin{pmatrix}14&-24\\-24&9\end{pmatrix},\qquad g_f=\frac{1}{125}\begin{pmatrix}-16\\-14\end{pmatrix}. \tag*{(159)} E 0 = 250 1 ( 14 − 24 − 24 9 ) , g f = 125 1 ( − 16 − 14 ) . (159)
The zero in the forcing is a two-dimensional vector. Thus the mean has no direct additive forcing, but it is coupled to the second block through a B 0 f aB_0f a B 0 f .
For a constant stepsize a > 0 a>0 a > 0 , consider an equilibrium z ( a ) z(a) z ( a ) of (C.14). In the transformed coordinates, its equations are
u = ( I − a A ) u + a B 0 f , u=(I-aA)u+aB_0f, u = ( I − a A ) u + a B 0 f ,
f = a C 0 u + ( 1 2 I + a E 0 ) f + a g f . f=aC_0u+\left(\frac{1}{2}I+aE_0\right)f+ag_f. f = a C 0 u + ( 2 1 I + a E 0 ) f + a g f .
Dividing the first equation by a a a gives A u = B 0 f Au = B_0f A u = B 0 f . Substituting u = A − 1 B 0 f u = A^{-1}B_0f u = A − 1 B 0 f into the second gives
H ( a ) f = a g f , H ( a ) = 1 2 I − a N 0 , N 0 = E 0 + C 0 A − 1 B 0 = 1 375 ( 48 − 88 − 63 28 ) . (160) \mathcal{H}(a)f = ag_f,\qquad\mathcal{H}(a) = \frac{1}{2}I-aN_0,\qquad N_0 = E_0+C_0A^{-1}B_0 = \frac{1}{375}\begin{pmatrix}48 & -88 \\ -63 & 28\end{pmatrix}. \tag*{(160)} H ( a ) f = a g f , H ( a ) = 2 1 I − a N 0 , N 0 = E 0 + C 0 A − 1 B 0 = 375 1 ( 48 − 63 − 88 28 ) . (160)
Since H ( 0 ) = I / 2 \mathcal{H}(0)=I/2 H ( 0 ) = I /2 , this matrix is invertible in a neighborhood of zero. The equilibrium is therefore uniquely given by
f ( a ) = a H ( a ) − 1 g f , u ( a ) = A − 1 B 0 f ( a ) , z ( a ) = M 0 ( u ( a ) f ( a ) ) . (161) f(a) = a\mathcal{H}(a)^{-1}g_f,\qquad u(a) = A^{-1}B_0f(a),\qquad z(a) = M_0\begin{pmatrix}u(a) \\ f(a)\end{pmatrix}. \tag*{(161)} f ( a ) = a H ( a ) − 1 g f , u ( a ) = A − 1 B 0 f ( a ) , z ( a ) = M 0 ( u ( a ) f ( a ) ) . (161)
The entries are analytic near zero and all three vectors vanish at zero. These properties allow us to control the change in the equilibrium as the stepsize decreases.
We give the exact mean to verify the first-order coefficient. For the two-by-two system,
det H ( a ) = 1 4 − 38 375 a − 56 1875 a 2 , 7500 det H ( a ) = 1875 − 760 a − 224 a 2 . \det\mathcal{H}(a) = \frac{1}{4}-\frac{38}{375}a-\frac{56}{1875}a^2,\qquad7500\det\mathcal{H}(a) = 1875-760a-224a^2. det H ( a ) = 4 1 − 375 38 a − 1875 56 a 2 , 7500 det H ( a ) = 1875 − 760 a − 224 a 2 .
Using the inverse formula for a two-by-two matrix gives
f ( a ) = a 3125 det H ( a ) ( 112 a − 200 112 a − 175 ) . f(a) = \frac{a}{3125\det\mathcal{H}(a)}\begin{pmatrix}112a-200 \\ 112a-175\end{pmatrix}. f ( a ) = 3125 det H ( a ) a ( 112 a − 200 112 a − 175 ) .
Multiplying by A − 1 B 0 A^{-1}B_0 A − 1 B 0 and collecting the numerator terms, we obtain
u 1 ( a ) = 8 a ( 56 a − 155 ) 3 ( 224 a 2 + 760 a − 1875 ) , u 2 ( a ) = 2 a ( 85 − 112 a ) 224 a 2 + 760 a − 1875 . (162) u_1(a) = \frac{8a(56a-155)}{3(224a^2+760a-1875)},\qquad u_2(a) = \frac{2a(85-112a)}{224a^2+760a-1875}. \tag*{(162)} u 1 ( a ) = 3 ( 224 a 2 + 760 a − 1875 ) 8 a ( 56 a − 155 ) , u 2 ( a ) = 224 a 2 + 760 a − 1875 2 a ( 85 − 112 a ) . (162)
Alternatively, expanding the inverse in (C.21) gives
f ( a ) = 2 a g f + O ( a 2 ) , f(a) = 2ag_f+O(a^2), f ( a ) = 2 a g f + O ( a 2 ) ,
u ( a ) = 2 a A − 1 B 0 g f + O ( a 2 ) = a ( 248 / 1125 − 34 / 375 ) + O ( a 2 ) . (163) u(a) = 2aA^{-1}B_0g_f+O(a^2)=a\begin{pmatrix}248/1125 \\ -34/375\end{pmatrix}+O(a^2). \tag*{(163)} u ( a ) = 2 a A − 1 B 0 g f + O ( a 2 ) = a ( 248/1125 − 34/375 ) + O ( a 2 ) . (163)
For the last equality, one can first compute
B 0 g f = 1 3125 ( 88 − 58 ) . B_0g_f=\frac{1}{3125}\begin{pmatrix}88 \\ -58\end{pmatrix}. B 0 g f = 3125 1 ( 88 − 58 ) .
and then use the inverse in (C.3). This verifies that the vector v v v in the proposition is nonzero.
It remains to compare the nonconstant-stepsize recursion with these equilibria. Lemma C.2 , applied to (C.18), gives a single norm in which all sufficiently small stepsize matrices contract. Its product bound is
∥ ( T 0 + α t T 1 ) ⋯ ( T 0 + α k T 1 ) ∥ ≤ C exp ( − c ∑ u = k t α u ) , k ≤ t . (164) \left\|(T_0+\alpha_tT_1)\cdots(T_0+\alpha_kT_1)\right\|\le C\exp\left(-c\sum_{u=k}^{t}\alpha_u\right),\qquad k\le t. \tag*{(164)} ∥ ( T 0 + α t T 1 ) ⋯ ( T 0 + α k T 1 ) ∥ ≤ C exp ( − c u = k ∑ t α u ) , k ≤ t . (164)
The finitely many earlier factors are included in the constant. The proof of that lemma gives the quadratic estimates for both blocks and verifies the bound for the full varying product.
Choose a fixed t 0 t_0 t 0 so large that z ( α t ) z(\alpha_t) z ( α t ) is defined in the analytic neighborhood above for every t ≥ t 0 t\ge t_0 t ≥ t 0 . Let χ t = z t − z ( α t ) \chi_t=z_t-z(\alpha_t) χ t = z t − z ( α t ) . Subtracting the equilibrium equation from (C.14), we have
χ t + 1 = ( T 0 + α t T 1 ) χ t + z ( α t ) − z ( α t + 1 ) . (165) \chi_{t+1}=(T_0+\alpha_tT_1)\chi_t+z(\alpha_t)-z(\alpha_{t+1}). \tag*{(165)} χ t + 1 = ( T 0 + α t T 1 ) χ t + z ( α t ) − z ( α t + 1 ) . (165)
The derivative of z ( a ) z(a) z ( a ) is bounded on a sufficiently small closed interval. Hence
∥ z ( α t ) − z ( α t + 1 ) ∥ ≤ C ∣ α t − α t + 1 ∣ ≤ C α t t + b . (166) \lVert z(\alpha_t)-z(\alpha_{t+1})\rVert\le C\lvert\alpha_t-\alpha_{t+1}\rvert\le C\frac{\alpha_t}{t+b}. \tag*{(166)} ∥ z ( α t ) − z ( α t + 1 )∥ ≤ C ∣ α t − α t + 1 ∣ ≤ C t + b α t . (166)
The last inequality follows from the mean value theorem applied to c 0 x − β c_0x^{-\beta} c 0 x − β . Unrolling (C.25) and using (C.24) gives
∥ χ t ∥ ≤ C exp ( − c ∑ u = t 0 t − 1 α u ) ∥ χ t 0 ∥ + C ∑ k = t 0 t − 1 exp ( − c ∑ u = k + 1 t − 1 α u ) α k k + b . (167) \lVert\chi_t\rVert\le C\exp\left(-c\sum_{u=t_0}^{t-1}\alpha_u\right)\lVert\chi_{t_0}\rVert+C\sum_{k=t_0}^{t-1}\exp\left(-c\sum_{u=k+1}^{t-1}\alpha_u\right)\frac{\alpha_k}{k+b}. \tag*{(167)} ∥ χ t ∥ ≤ C exp ( − c u = t 0 ∑ t − 1 α u ) ∥ χ t 0 ∥ + C k = t 0 ∑ t − 1 exp ( − c u = k + 1 ∑ t − 1 α u ) k + b α k . (167)
Lemma C.7 proves that the sum on the right is O ( ( t + b ) − 1 ) O((t+b)^{-1}) O (( t + b ) − 1 ) . It does so by separating the early indices, for which the remaining product is exponentially small, from the later indices, for which a geometric sum is uniformly bounded. The initial term is also exponentially small. Consequently,
∥ z t − z ( α t ) ∥ = O ( ( t + b ) − 1 ) . (168) \lVert z_t-z(\alpha_t)\rVert=O((t+b)^{-1}). \tag*{(168)} ∥ z t − z ( α t )∥ = O (( t + b ) − 1 ) . (168)
Applying L 0 L_0 L 0 and combining this estimate with (C.23), we obtain
E Δ t = α t v + O ( α t 2 + ( t + b ) − 1 ) , \mathbb{E}\Delta_t=\alpha_t v+O(\alpha_t^2+(t+b)^{-1}), E Δ t = α t v + O ( α t 2 + ( t + b ) − 1 ) ,
which proves (S.14).
Finally, sum this expansion over t = 1 , … , n t=1,\ldots,n t = 1 , … , n . By Lemma C.6,
∑ t = 1 n α t = c 0 1 − β n 1 − β + O ( 1 ) , \sum_{t=1}^{n}\alpha_t=\frac{c_0}{1-\beta}n^{1-\beta}+O(1), t = 1 ∑ n α t = 1 − β c 0 n 1 − β + O ( 1 ) ,
∑ t = 1 n α t 2 = O ( 1 ) , ∑ t = 1 n ( t + b ) − 1 = O ( log ( e n ) ) . \sum_{t=1}^{n}\alpha_t^2=O(1),\qquad\sum_{t=1}^{n}(t+b)^{-1}=O(\log(en)). t = 1 ∑ n α t 2 = O ( 1 ) , t = 1 ∑ n ( t + b ) − 1 = O ( log ( e n )) .
After division by n \sqrt{n} n , we have
E X n = c 0 1 − β v n 1 / 2 − β + O ( log ( e n ) n ) . \mathbb{E}X_n=\frac{c_0}{1-\beta}v\,n^{1/2-\beta}+O\left(\frac{\log(en)}{\sqrt{n}}\right). E X n = 1 − β c 0 v n 1/2 − β + O ( n log ( e n ) ) .
The ratio of the last term to n 1 / 2 − β n^{1/2-\beta} n 1/2 − β is bounded by a constant times log ( e n ) n β − 1 \log(en)n^{\beta-1} log ( e n ) n β − 1 , which tends to zero. This proves (S.15).
The only dependence on the deterministic initialization occurs in the fixed initial error in (C.27). For stationary states and deterministic Δ 0 \Delta_0 Δ 0 , one has z 0 = J 0 Δ 0 z_0=J_0\Delta_0 z 0 = J 0 Δ 0 . Every fixed bounded initialization therefore gives the same leading coefficient. This completes the proof.
Proof of Proposition 3.2
Proposition 3.2. For the weights in (S.8) on (S.11),`
C n = Σ ∞ + n β − 1 G + o ( n β − 1 ) , G = 1 c 0 A − 1 X A − T ≻ 0. (169) C_n=\Sigma_\infty+n^{\beta-1}G+o(n^{\beta-1}),\qquad G=\frac{1}{c_0}A^{-1}\mathcal{X}A^{-\mathsf{T}}\succ0. \tag*{(169)} C n = Σ ∞ + n β − 1 G + o ( n β − 1 ) , G = c 0 1 A − 1 X A − T ≻ 0. (169)
Proof. We first expand the infinite-horizon weight and then calculate the effect of truncating it at time n n n . All limits below are taken with c 0 c_0 c 0 , b b b , β \beta β fixed. Let
W ∞ , t = α t ∑ k = t + 1 ∞ Φ t + 1 : k − 1 , R = A − 1 , τ = t + b , a = α t . W_{\infty,t}=\alpha_t\sum_{k=t+1}^{\infty}\Phi_{t+1:k-1},\qquad R=A^{-1},\qquad\tau=t+b,\qquad a=\alpha_t. W ∞ , t = α t k = t + 1 ∑ ∞ Φ t + 1 : k − 1 , R = A − 1 , τ = t + b , a = α t .
Lemma C.1 proves convergence of this series and the uniform bounds used below. It also proves the telescoping identity
∑ k = t + 1 ∞ α k Φ t + 1 : k − 1 = A − 1 . \sum_{k=t+1}^{\infty} \alpha_k\Phi_{t+1:k-1}=A^{-1}. k = t + 1 ∑ ∞ α k Φ t + 1 : k − 1 = A − 1 .
Subtracting this identity from the definition of the infinite-horizon weight gives
W ∞ , t − A − 1 = ∑ j = 1 ∞ ( α t − α t + j ) Φ t + 1 : t + j − 1 . (170) W_{\infty,t}-A^{-1}=\sum_{j=1}^{\infty}(\alpha_t-\alpha_{t+j})\Phi_{t+1:t+j-1}. \tag*{(170)} W ∞ , t − A − 1 = j = 1 ∑ ∞ ( α t − α t + j ) Φ t + 1 : t + j − 1 . (170)
We will prove
W ∞ , t = A − 1 + β c 0 ( t + b ) β − 1 A − 2 + o ( ( t + b ) β − 1 ) . (171) W_{\infty,t}=A^{-1}+\frac{\beta}{c_0}(t+b)^{\beta-1}A^{-2}+o\left((t+b)^{\beta-1}\right). \tag*{(171)} W ∞ , t = A − 1 + c 0 β ( t + b ) β − 1 A − 2 + o ( ( t + b ) β − 1 ) . (171)
Fix L > 0 L>0 L > 0 and first consider the summands with 1 ≤ j ≤ ⌊ L / a ⌋ 1\le j\le\lfloor L/a\rfloor 1 ≤ j ≤ ⌊ L / a ⌋ . Since τ a = c 0 τ 1 − β → ∞ \tau a=c_0\tau^{1-\beta}\to\infty τ a = c 0 τ 1 − β → ∞ , we have
max j ≤ L / a j τ ≤ L τ a ⟶ 0. \max_{j\le L/a}\frac{j}{\tau}\le\frac{L}{\tau a}\longrightarrow0. j ≤ L / a max τ j ≤ τ a L ⟶ 0.
The Taylor expansion of ( 1 + x ) − β (1+x)^{-\beta} ( 1 + x ) − β at zero therefore gives, uniformly over these indices,
α t − α t + j = a { 1 − ( 1 + j / τ ) − β } = β a j τ ( 1 + O ( j τ ) ) . (172) \alpha_t-\alpha_{t+j}=a\{1-(1+j/\tau)^{-\beta}\}=\frac{\beta aj}{\tau}\left(1+O\left(\frac{j}{\tau}\right)\right). \tag*{(172)} α t − α t + j = a { 1 − ( 1 + j / τ ) − β } = τ β aj ( 1 + O ( τ j ) ) . (172)
The error is uniform for each fixed L L L because the second derivative of ( 1 + x ) − β (1+x)^{-\beta} ( 1 + x ) − β is bounded on a fixed neighborhood of zero.
We also need a uniform approximation to the matrix product. For 1 ≤ h ≤ j − 1 1\le h\le j-1 1 ≤ h ≤ j − 1 , the mean value theorem gives
∣ α t + h − a ∣ ≤ C a h τ . |\alpha_{t+h}-a|\le C\frac{ah}{\tau}. ∣ α t + h − a ∣ ≤ C τ ah .
Summing this inequality and using j ≤ L / a j\le L/a j ≤ L / a gives
∣ ∑ h = 1 j − 1 α t + h − a ( j − 1 ) ∣ ≤ C a j 2 τ ≤ C L 2 τ a ⟶ 0. (173) \left|\sum_{h=1}^{j-1}\alpha_{t+h}-a(j-1)\right|\le C\frac{aj^2}{\tau}\le C\frac{L^2}{\tau a}\longrightarrow0. \tag*{(173)} h = 1 ∑ j − 1 α t + h − a ( j − 1 ) ≤ C τ a j 2 ≤ C τ a L 2 ⟶ 0. (173)
Since the stepsizes decrease,
∑ h = 1 j − 1 α t + h 2 ≤ j a 2 ≤ L a ⟶ 0. (174) \sum_{h=1}^{j-1}\alpha_{t+h}^{2}\le ja^2\le La\longrightarrow0. \tag*{(174)} h = 1 ∑ j − 1 α t + h 2 ≤ j a 2 ≤ L a ⟶ 0. (174)
The matrix A A A is symmetric and positive definite. For sufficiently large t t t , the convergent matrix logarithm series gives
log ( I − α u A ) = − α u A + E u , ∥ E u ∥ ≤ C α u 2 , u ≥ t . \log(I-\alpha_u A)=-\alpha_u A+\mathcal{E}_u,\qquad\|\mathcal{E}_u\|\le C\alpha_u^2,\qquad u\ge t. log ( I − α u A ) = − α u A + E u , ∥ E u ∥ ≤ C α u 2 , u ≥ t .
For example, once α t ∥ A ∥ ≤ 1 / 2 \alpha_t\|A\|\le1/2 α t ∥ A ∥ ≤ 1/2 , the terms of degree at least two have norm bounded by
∑ r ≥ 2 ( α u ∥ A ∥ ) r r ≤ C α u 2 . \sum_{r\ge2}\frac{(\alpha_u\|A\|)^r}{r}\le C\alpha_u^2. r ≥ 2 ∑ r ( α u ∥ A ∥ ) r ≤ C α u 2 .
All these matrices are functions of A A A and commute. Thus their logarithms can be added in the product, giving
Φ t + 1 : t + j − 1 = exp { − A a ( j − 1 ) + E t , j } , max j ≤ L / a ∥ E t , j ∥ ≤ C ( L 2 τ a + L a ) ⟶ 0. \Phi_{t+1:t+j-1}=\exp\{-Aa(j-1)+\mathcal{E}_{t,j}\},\qquad
\max_{j\le L/a}\|\mathcal{E}_{t,j}\|\le C\left(\frac{L^2}{\tau a}+La\right)\longrightarrow0. Φ t + 1 : t + j − 1 = exp { − A a ( j − 1 ) + E t , j } , j ≤ L / a max ∥ E t , j ∥ ≤ C ( τ a L 2 + L a ) ⟶ 0.
The main exponent is bounded on this range of indices. Since the two exponents commute, the exponential series gives
∥ e B + E − e B ∥ = ∥ e B ( e E − I ) ∥ ≤ e ∥ B ∥ ( e ∥ E ∥ − 1 ) ≤ e ∥ B ∥ + ∥ E ∥ ∥ E ∥ . \lVert e^{B+E}-e^B\rVert=\lVert e^B(e^E-I)\rVert\leq e^{\lVert B\rVert}(e^{\lVert E\rVert}-1)\leq e^{\lVert B\rVert+\lVert E\rVert}\lVert E\rVert. ∥ e B + E − e B ∥ = ∥ e B ( e E − I )∥ ≤ e ∥ B ∥ ( e ∥ E ∥ − 1 ) ≤ e ∥ B ∥ + ∥ E ∥ ∥ E ∥ .
Taking B = − A a ( j − 1 ) B=-Aa(j-1) B = − A a ( j − 1 ) and E = E t , j E=\mathcal{E}_{t,j} E = E t , j , we obtain
max 1 ≤ j ≤ L / a ∥ Φ t + 1 : t + j − 1 − e − A a ( j − 1 ) ∥ ⟶ 0. (175) \max_{1\leq j\leq L/a}\lVert\Phi_{t+1:t+j-1}-e^{-Aa(j-1)}\rVert\longrightarrow0. \tag*{(175)} 1 ≤ j ≤ L / a max ∥ Φ t + 1 : t + j − 1 − e − A a ( j − 1 ) ∥ ⟶ 0. (175)
This calculation includes the empty product at j = 1 j=1 j = 1 . Multiply the truncated sum in (C.29) by τ a \tau a τ a . By (C.31),
( τ a ) ( α t − α t + j ) = β a 2 j ( 1 + o ( 1 ) ) (\tau a)(\alpha_t-\alpha_{t+j})=\beta a^2j(1+o(1)) ( τ a ) ( α t − α t + j ) = β a 2 j ( 1 + o ( 1 ))
uniformly for j ≤ L / a j\leq L/a j ≤ L / a . The sum of a 2 j a^2j a 2 j over this range is bounded by a constant depending only on L L L . We can therefore use (C.34) term by term in the truncated sum. We obtain
τ a ∑ j = 1 ⌊ L / a ⌋ ( α t − α t + j ) Φ t + 1 : t + j − 1 = β ∑ j = 1 ⌊ L / a ⌋ a ( a j ) e − A a ( j − 1 ) + o ( 1 ) ⟶ β ∫ 0 L x e − A x d x . (176) \begin{aligned}
\tau a\sum_{j=1}^{\lfloor L/a\rfloor}(\alpha_t-\alpha_{t+j})\Phi_{t+1:t+j-1} \\
&=\beta\sum_{j=1}^{\lfloor L/a\rfloor}a(aj)e^{-Aa(j-1)}+o(1)\longrightarrow\beta\int_0^L xe^{-Ax}\,dx. \tag*{(176)}
\end{aligned} τ a j = 1 ∑ ⌊ L / a ⌋ ( α t − α t + j ) Φ t + 1 : t + j − 1 = β j = 1 ∑ ⌊ L / a ⌋ a ( aj ) e − A a ( j − 1 ) + o ( 1 ) ⟶ β ∫ 0 L x e − A x d x . (176)
The last step is a Riemann sum with mesh a a a . Replacing a ( j − 1 ) a(j-1) a ( j − 1 ) by a j aj aj in the exponential changes the sum by O ( a ) O(a) O ( a ) for fixed L L L , since the matrix exponential has a bounded derivative on that interval. We now control the omitted part of the infinite sum. For k > t k>t k > t , let
s t , k = ∑ u = t + 1 k − 1 α u , η t = β c 0 ( t + b ) β − 1 = β τ a . s_{t,k}=\sum_{u=t+1}^{k-1}\alpha_u,\qquad\eta_t=\frac{\beta}{c_0}(t+b)^{\beta-1}=\frac{\beta}{\tau a}. s t , k = u = t + 1 ∑ k − 1 α u , η t = c 0 β ( t + b ) β − 1 = τ a β .
By Lemma C.1 ,
0 ≤ α t − α k ≤ α k η t ( α t + s t , k ) e η t ( α t + s t , k ) . 0\leq\alpha_t-\alpha_k\leq\alpha_k\eta_t(\alpha_t+s_{t,k})e^{\eta_t(\alpha_t+s_{t,k})}. 0 ≤ α t − α k ≤ α k η t ( α t + s t , k ) e η t ( α t + s t , k ) .
Let λ = λ min ( A ) \lambda=\lambda_{\min}(A) λ = λ m i n ( A ) . The shift condition gives η t ≤ λ / 4 \eta_t\leq\lambda/4 η t ≤ λ /4 , and the product is bounded by e − λ s t , k e^{-\lambda s_{t,k}} e − λ s t , k . Since τ a η t = β \tau a\eta_t=\beta τ a η t = β and α t ≤ 1 \alpha_t\leq1 α t ≤ 1 , the normalized tail has norm at most
C ∑ k > t + ⌊ L / a ⌋ α k ( 1 + s t , k ) e − ( 3 λ / 4 ) s t , k . (177) C\sum_{k>t+\lfloor L/a\rfloor}\alpha_k(1+s_{t,k})e^{-(3\lambda/4)s_{t,k}}. \tag*{(177)} C k > t + ⌊ L / a ⌋ ∑ α k ( 1 + s t , k ) e − ( 3 λ /4 ) s t , k . (177)
At the first index in this tail, the cumulative stepsize tends to L L L . This follows from (C.32) with the corresponding number of factors. The interval comparison in Lemma C.1 consequently bounds the limit superior of (C.36) by
C ∫ L / 2 ∞ ( 1 + x ) e − c x d x . C\int_{L/2}^{\infty}(1+x)e^{-cx}\,dx. C ∫ L /2 ∞ ( 1 + x ) e − c x d x .
The integral tends to zero as L → ∞ L\to\infty L → ∞ . Taking the limit first in t t t for fixed L L L and then in L L L , we obtain
τ a ( W ∞ , t − A − 1 ) ⟶ β ∫ 0 ∞ x e − A x d x = β A − 2 . (178) \tau a(W_{\infty,t}-A^{-1})\longrightarrow\beta\int_0^\infty xe^{-Ax}\,dx=\beta A^{-2}. \tag*{(178)} τ a ( W ∞ , t − A − 1 ) ⟶ β ∫ 0 ∞ x e − A x d x = β A − 2 . (178)
The matrix integral is evaluated in Lemma C.9 . Since ( τ a ) − 1 = c 0 − 1 ( t + b ) β − 1 (\tau a)^{-1}=c_0^{-1}(t+b)^{\beta-1} ( τ a ) − 1 = c 0 − 1 ( t + b ) β − 1 , this proves (C.30). The same estimates give the domination
∥ W ∞ , t − A − 1 ∥ ≤ C ( t + b ) β − 1 , (179) \lVert W_{\infty,t}-A^{-1}\rVert\le C(t+b)^{\beta-1}, \tag*{(179)} ∥ W ∞ , t − A − 1 ∥ ≤ C ( t + b ) β − 1 , (179)
including the finitely many early indices after increasing C C C .
We next average the covariance of these infinite-horizon weights. Let q t = ( t + b ) β − 1 q_t=(t+b)^{\beta-1} q t = ( t + b ) β − 1 and write
W ∞ , t = R + β c 0 q t A − 2 + E t , ∥ E t ∥ = o ( q t ) , ∥ W ∞ , t − R ∥ ≤ C q t . W_{\infty,t}=R+\frac{\beta}{c_0}q_tA^{-2}+E_t,\qquad\lVert E_t\rVert=o(q_t),\qquad\lVert W_{\infty,t}-R\rVert\le Cq_t. W ∞ , t = R + c 0 β q t A − 2 + E t , ∥ E t ∥ = o ( q t ) , ∥ W ∞ , t − R ∥ ≤ C q t .
Expanding the two weights on the two sides of S S S gives
W ∞ , t S W ∞ , t T − R S R T = β c 0 q t { A − 2 S A − T + A − 1 S ( A − 2 ) T } + H t , W_{\infty,t}SW_{\infty,t}^{\mathsf T}-RSR^{\mathsf T}
=\frac{\beta}{c_0}q_t\{A^{-2}SA^{-\mathsf T}+A^{-1}S(A^{-2})^{\mathsf T}\}+H_t, W ∞ , t S W ∞ , t T − R S R T = c 0 β q t { A − 2 S A − T + A − 1 S ( A − 2 ) T } + H t ,
∥ H t ∥ ≤ C ∥ E t ∥ + C q t 2 . (180) \lVert H_t\rVert\le C\lVert E_t\rVert+Cq_t^2. \tag*{(180)} ∥ H t ∥ ≤ C ∥ E t ∥ + C q t 2 . (180)
The first two remainder terms are E t S R T E_tSR^{\mathsf T} E t S R T and R S E t T RSE_t^{\mathsf T} R S E t T . The remaining term is the product of the two weight differences, which is bounded by C q t 2 Cq_t^2 C q t 2 . By Lemma C.6 ,
1 n ∑ t = 0 n − 1 q t = 1 β n β − 1 + O ( n − 1 ) , 1 n ∑ t = 0 n − 1 q t 2 = O ( n 2 β − 2 ) = o ( n β − 1 ) . \frac{1}{n}\sum_{t=0}^{n-1}q_t=\frac{1}{\beta}n^{\beta-1}+O(n^{-1}),\qquad
\frac{1}{n}\sum_{t=0}^{n-1}q_t^2=O(n^{2\beta-2})=o(n^{\beta-1}). n 1 t = 0 ∑ n − 1 q t = β 1 n β − 1 + O ( n − 1 ) , n 1 t = 0 ∑ n − 1 q t 2 = O ( n 2 β − 2 ) = o ( n β − 1 ) .
The same lemma, applied to E t = o ( q t ) E_t=o(q_t) E t = o ( q t ) , gives n − 1 ∑ t < n ∥ E t ∥ = o ( n β − 1 ) n^{-1}\sum_{t<n}\lVert E_t\rVert=o(n^{\beta-1}) n − 1 ∑ t < n ∥ E t ∥ = o ( n β − 1 ) . Therefore,
1 n ∑ t = 0 n − 1 { W ∞ , t S W ∞ , t T − Σ ∞ } = n β − 1 c 0 { A − 2 S A − T + A − 1 S ( A − 2 ) T } + o ( n β − 1 ) . (181) \frac{1}{n}\sum_{t=0}^{n-1}\{W_{\infty,t}SW_{\infty,t}^{\mathsf T}-\Sigma_\infty\}
=\frac{n^{\beta-1}}{c_0}\{A^{-2}SA^{-\mathsf T}+A^{-1}S(A^{-2})^{\mathsf T}\}+o(n^{\beta-1}). \tag*{(181)} n 1 t = 0 ∑ n − 1 { W ∞ , t S W ∞ , t T − Σ ∞ } = c 0 n β − 1 { A − 2 S A − T + A − 1 S ( A − 2 ) T } + o ( n β − 1 ) . (181)
The factor β \beta β from the local weight expansion cancels with the factor 1 / β 1/\beta 1/ β in the power sum.
It remains to replace the infinite-horizon weights by the weights truncated at time n n n . Let a n = α n a_n=\alpha_n a n = α n and write t = n − k t=n-k t = n − k . For fixed L > 0 L>0 L > 0 , consider 1 ≤ k ≤ L / a n 1\le k\le L/a_n 1 ≤ k ≤ L / a n . Since n a n → ∞ na_n\to\infty n a n → ∞ , all these indices satisfy k / n → 0 k/n\to0 k / n → 0 uniformly. For n − k ≤ u ≤ n n-k\le u\le n n − k ≤ u ≤ n , we have
1 ≤ α u a n ≤ ( 1 − k n + b ) − β = 1 + o ( 1 ) (182) 1\le\frac{\alpha_u}{a_n}\le\left(1-\frac{k}{n+b}\right)^{-\beta}=1+o(1) \tag*{(182)} 1 ≤ a n α u ≤ ( 1 − n + b k ) − β = 1 + o ( 1 ) (182)
uniformly over this range. The product comparison used in (C.32)–(C.34) can now be applied with the common stepsize a n a_n a n throughout the terminal interval. The cumulative stepsize error is at most
L max n − k ≤ u ≤ n ∣ α u / a n − 1 ∣ = o ( 1 ) , L\max_{n-k\le u\le n}\lvert\alpha_u/a_n-1\rvert=o(1), L n − k ≤ u ≤ n max ∣ α u / a n − 1 ∣ = o ( 1 ) ,
and the sum of squared stepsizes is O ( L a n ) = o ( 1 ) O(La_n)=o(1) O ( L a n ) = o ( 1 ) . Thus all the products in the finite sum are uniformly approximated by the corresponding constant-stepsize exponentials.
In the notation x k = k a n x_k=ka_n x k = k a n , this gives
W n , n − k = α n − k a n { a n ∑ j = 1 k e − A a n ( j − 1 ) + o ( 1 ) } = ∫ 0 x k e − A s d s + o ( 1 ) = R ( I − e − A x k ) + o ( 1 ) , (183) \begin{aligned}
W_{n,n-k}=\frac{\alpha_{n-k}}{a_n}\left\{a_n\sum_{j=1}^{k}e^{-Aa_n(j-1)}+o(1)\right\} \\
&=\int_0^{x_k}e^{-As}\,ds+o(1)=R(I-e^{-Ax_k})+o(1), \tag*{(183)}
\end{aligned} W n , n − k = a n α n − k { a n j = 1 ∑ k e − A a n ( j − 1 ) + o ( 1 ) } = ∫ 0 x k e − A s d s + o ( 1 ) = R ( I − e − A x k ) + o ( 1 ) , (183)
uniformly for 1 ≤ k ≤ L / a n 1 \le k \le L/a_n 1 ≤ k ≤ L / a n . The error in summing the product approximation is bounded by k a n ka_n k a n times the largest product error, so it tends to zero uniformly on this range. The Riemann sum for the integral is also uniform in its upper endpoint because the integrand has bounded derivative on [ 0 , L ] [0,L] [ 0 , L ] . By (C.38),
max 1 ≤ k ≤ L / a n ∥ W ∞ , n − k − R ∥ ⟶ 0 , (184) \max_{1\le k\le L/a_n}\left\|W_{\infty,n-k}-R\right\|\longrightarrow0, \tag*{(184)} 1 ≤ k ≤ L / a n max ∥ W ∞ , n − k − R ∥ ⟶ 0 , (184)
since the smallest time index in this range tends to infinity. These estimates also cover grid points tending to zero.
Define the difference of the two covariance summands by
F n , k = W n , n − k S W n , n − k T − W ∞ , n − k S W ∞ , n − k T , 1 ≤ k ≤ n . F_{n,k}=W_{n,n-k}SW_{n,n-k}^{\mathsf T}-W_{\infty,n-k}SW_{\infty,n-k}^{\mathsf T},\qquad1\le k\le n. F n , k = W n , n − k S W n , n − k T − W ∞ , n − k S W ∞ , n − k T , 1 ≤ k ≤ n .
The two uniform limits above imply convergence on every bounded interval of the grid x k = k a n x_k=ka_n x k = k a n to the continuous matrix function
F ( x ) = R ( I − e − A x ) S ( I − e − A T x ) R T − Σ ∞ . (185) F(x)=R(I-e^{-Ax})S(I-e^{-A^{\mathsf T}x})R^{\mathsf T}-\Sigma_\infty. \tag*{(185)} F ( x ) = R ( I − e − A x ) S ( I − e − A T x ) R T − Σ ∞ . (185)
To justify summation over all k k k , we use the exact tail identity from Lemma C.1 :
W ∞ , t − W n , t = α t a n Φ t + 1 : n W ∞ , n . W_{\infty,t}-W_{n,t}=\frac{\alpha_t}{a_n}\Phi_{t+1:n}W_{\infty,n}. W ∞ , t − W n , t = a n α t Φ t + 1 : n W ∞ , n .
Its exponential bound and uniform boundedness of the weights give
∥ F n , k ∥ ≤ C ∥ W n , n − k − W ∞ , n − k ∥ ≤ C exp ( − c ∑ u = n − k + 1 n α u ) ≤ C e − c k a n . (186) \|F_{n,k}\|\le C\|W_{n,n-k}-W_{\infty,n-k}\|
\le C\exp\left(-c\sum_{u=n-k+1}^{n}\alpha_u\right)\le Ce^{-cka_n}. \tag*{(186)} ∥ F n , k ∥ ≤ C ∥ W n , n − k − W ∞ , n − k ∥ ≤ C exp ( − c u = n − k + 1 ∑ n α u ) ≤ C e − c k a n . (186)
In the last inequality, every stepsize in the sum is at least a n a_n a n . The limiting integrand has the same type of bound. Expanding its two factors and using ∥ e − A x ∥ ≤ e − λ x \|e^{-Ax}\|\le e^{-\lambda x} ∥ e − A x ∥ ≤ e − λ x gives
∥ F ( x ) ∥ ≤ ∥ R ∥ 2 ∥ S ∥ { 2 e − λ x + e − 2 λ x } ≤ C e − λ x . \|F(x)\|\le\|R\|^2\|S\|\{2e^{-\lambda x}+e^{-2\lambda x}\}\le Ce^{-\lambda x}. ∥ F ( x ) ∥ ≤ ∥ R ∥ 2 ∥ S ∥ { 2 e − λ x + e − 2 λ x } ≤ C e − λ x .
Thus the continuous tail is also integrable.
We can now apply Lemma C.8 with mesh a n a_n a n . Its hypotheses are precisely the local uniform convergence in (C.42)–(C.43) and the domination (C.45). It gives
a n ∑ k = 1 n F n , k ⟶ ∫ 0 ∞ F ( x ) d x . a_n\sum_{k=1}^{n}F_{n,k}\longrightarrow\int_{0}^{\infty}F(x)\,dx. a n k = 1 ∑ n F n , k ⟶ ∫ 0 ∞ F ( x ) d x .
Since
1 n a n = ( n + b ) β c 0 n = n β − 1 c 0 ( 1 + o ( 1 ) ) , \frac{1}{na_n}=\frac{(n+b)^\beta}{c_0n}=\frac{n^{\beta-1}}{c_0}(1+o(1)), n a n 1 = c 0 n ( n + b ) β = c 0 n β − 1 ( 1 + o ( 1 )) ,
the terminal contribution is
1 n ∑ t = 0 n − 1 { W n , t S W n , t T − W ∞ , t S W ∞ , t T } = n β − 1 c 0 ∫ 0 ∞ F ( x ) d x + o ( n β − 1 ) . (187) \frac{1}{n}\sum_{t=0}^{n-1}\{W_{n,t}SW_{n,t}^{\mathsf T}-W_{\infty,t}SW_{\infty,t}^{\mathsf T}\}
=\frac{n^{\beta-1}}{c_0}\int_{0}^{\infty}F(x)\,dx+o(n^{\beta-1}). \tag*{(187)} n 1 t = 0 ∑ n − 1 { W n , t S W n , t T − W ∞ , t S W ∞ , t T } = c 0 n β − 1 ∫ 0 ∞ F ( x ) d x + o ( n β − 1 ) . (187)
We evaluate the integral without assuming that S S S commutes with A A A . Expanding (C.44) gives
F ( x ) = − R e − A x S R T − R S e − A T x R T + R e − A x S e − A T x R T . F(x)=-R e^{-Ax} S R^{\mathsf{T}}-R S e^{-A^{\mathsf{T}}x}R^{\mathsf{T}}+R e^{-Ax}S e^{-A^{\mathsf{T}}x}R^{\mathsf{T}}. F ( x ) = − R e − A x S R T − R S e − A T x R T + R e − A x S e − A T x R T .
Using ∫ 0 ∞ e − A x d x = A − 1 \int_{0}^{\infty}e^{-Ax}\,\mathrm{d}x=A^{-1} ∫ 0 ∞ e − A x d x = A − 1 and the definition of X \mathcal{X} X , we obtain
∫ 0 ∞ F ( x ) d x = − A − 2 S A − T − A − 1 S ( A − 2 ) T + A − 1 X A − T . (188) \int_{0}^{\infty}F(x)\,\mathrm{d}x=-A^{-2}SA^{-\mathsf{T}}-A^{-1}S(A^{-2})^{\mathsf{T}}+A^{-1}\mathcal{X}A^{-\mathsf{T}}. \tag*{(188)} ∫ 0 ∞ F ( x ) d x = − A − 2 S A − T − A − 1 S ( A − 2 ) T + A − 1 X A − T . (188)
Adding (C.40) and (C.46), the first two terms in (C.47) cancel the interior correction. We are left with
C n − Σ ∞ = n β − 1 c 0 A − 1 X A − T + o ( n β − 1 ) . C_n-\Sigma_{\infty}=\frac{n^{\beta-1}}{c_0}A^{-1}\mathcal{X}A^{-\mathsf{T}}+o(n^{\beta-1}). C n − Σ ∞ = c 0 n β − 1 A − 1 X A − T + o ( n β − 1 ) .
Finally, S ≻ 0 S\succ0 S ≻ 0 implies X ≻ 0 \mathcal{X}\succ0 X ≻ 0 by Lemma C.9 . Since A A A is invertible and c 0 > 0 c_0>0 c 0 > 0 , the same is true of G G G . This proves Proposition 3.2.
Problem Specific Bounds
Lemma C.1. For the matrix A A A in (S.12), let λ = λ min ( A ) \lambda=\lambda_{\min}(A) λ = λ m i n ( A ) . The infinite-horizon weights are well defined and uniformly bounded. More precisely,
∥ Φ i : j ∥ ≤ exp ( − λ ∑ u = i j α u ) , ∥ A − 1 ∥ ≤ λ − 1 , \lVert\Phi_{i:j}\rVert\le\exp\left(-\lambda\sum_{u=i}^{j}\alpha_u\right),\qquad\lVert A^{-1}\rVert\le\lambda^{-1}, ∥ Φ i : j ∥ ≤ exp ( − λ u = i ∑ j α u ) , ∥ A − 1 ∥ ≤ λ − 1 , sup t ∥ W ∞ , t ∥ + sup n , t < n ∥ W n , t ∥ ≤ C , ∥ W ∞ , t − A − 1 ∥ ≤ C ( t + b ) β − 1 . (189) \sup_t\lVert W_{\infty,t}\rVert+\sup_{n,t<n}\lVert W_{n,t}\rVert\le C,\qquad\lVert W_{\infty,t}-A^{-1}\rVert\le C(t+b)^{\beta-1}. \tag*{(189)} t sup ∥ W ∞ , t ∥ + n , t < n sup ∥ W n , t ∥ ≤ C , ∥ W ∞ , t − A − 1 ∥ ≤ C ( t + b ) β − 1 . (189) The exact identities are
∑ k = t + 1 ∞ α k Φ t + 1 : k − 1 = A − 1 , (190) \sum_{k=t+1}^{\infty}\alpha_k\Phi_{t+1:k-1}=A^{-1}, \tag*{(190)} k = t + 1 ∑ ∞ α k Φ t + 1 : k − 1 = A − 1 , (190) W ∞ , t − W n , t = α t α n Φ t + 1 : n W ∞ , n . (191) W_{\infty,t}-W_{n,t}=\frac{\alpha_t}{\alpha_n}\Phi_{t+1:n}W_{\infty,n}. \tag*{(191)} W ∞ , t − W n , t = α n α t Φ t + 1 : n W ∞ , n . (191) The tail satisfies
∥ W ∞ , t − W n , t ∥ ≤ C exp ( − 3 λ 4 ∑ u = t + 1 n α u ) , t < n . (192) \lVert W_{\infty,t}-W_{n,t}\rVert\le C\exp\left(-\frac{3\lambda}{4}\sum_{u=t+1}^{n}\alpha_u\right),\qquad t<n. \tag*{(192)} ∥ W ∞ , t − W n , t ∥ ≤ C exp ( − 4 3 λ u = t + 1 ∑ n α u ) , t < n . (192) For s t , k = ∑ u = t + 1 k − 1 α u s_{t,k}=\sum_{u=t+1}^{k-1}\alpha_u s t , k = ∑ u = t + 1 k − 1 α u and any d > 0 d>0 d > 0 , one also has
∑ k ≥ m α k ( 1 + s t , k ) e − d s t , k ≤ e d ∫ s t , m ∞ ( 1 + x ) e − d x d x , m ≥ t + 1. (193) \sum_{k\ge m}\alpha_k(1+s_{t,k})e^{-d s_{t,k}}\le e^d\int_{s_{t,m}}^{\infty}(1+x)e^{-dx}\,\mathrm{d}x,\qquad m\ge t+1. \tag*{(193)} k ≥ m ∑ α k ( 1 + s t , k ) e − d s t , k ≤ e d ∫ s t , m ∞ ( 1 + x ) e − d x d x , m ≥ t + 1. (193)
Proof. The explicit eigenvalues satisfy 0 < λ ≤ λ max ( A ) < 1 0<\lambda\le\lambda_{\max}(A)<1 0 < λ ≤ λ m a x ( A ) < 1 and λ ≥ a ⋆ = 2 / 25 \lambda\ge a_{\star}=2/25 λ ≥ a ⋆ = 2/25 . For 0 ≤ α ≤ 1 0\le\alpha\le1 0 ≤ α ≤ 1 , the matrix I − α A I-\alpha A I − α A is symmetric with nonnegative eigenvalues. Its largest eigenvalue is 1 − λ α 1-\lambda\alpha 1 − λ α , so
∥ I − α A ∥ = 1 − λ α ≤ e − λ α . \lVert I-\alpha A\rVert=1-\lambda\alpha\le e^{-\lambda\alpha}. ∥ I − α A ∥ = 1 − λ α ≤ e − λ α .
Multiplying this inequality proves the product bound. The inverse bound follows from the smallest eigenvalue. Both assertions also respect the empty-product convention.
To compare stepsizes, define
η t = β c 0 ( t + b ) β − 1 . \eta_t = \frac{\beta}{c_0}(t+b)^{\beta-1}. η t = c 0 β ( t + b ) β − 1 .
The shift condition and λ ≥ a ⋆ \lambda\ge a_\star λ ≥ a ⋆ give η t ≤ η 0 ≤ λ / 4 \eta_t \le\eta_0 \le\lambda/4 η t ≤ η 0 ≤ λ /4 . For k > t k > t k > t , using log ( 1 + x ) ≤ x \log(1+x) \le x log ( 1 + x ) ≤ x , we have
log α t α k = β ∑ u = t k − 1 log ( 1 + 1 u + b ) ≤ β c 0 ∑ u = t k − 1 ( u + b ) β − 1 α u ≤ η t ∑ u = t k − 1 α u = η t ( α t + s t , k ) . (194) \begin{aligned}
\log\frac{\alpha_t}{\alpha_k}
&= \beta\sum_{u=t}^{k-1} \log\left(1+\frac{1}{u+b}\right) \\
&\le\frac{\beta}{c_0}\sum_{u=t}^{k-1}(u+b)^{\beta-1}\alpha_u \\
&\le\eta_t\sum_{u=t}^{k-1}\alpha_u
= \eta_t(\alpha_t+s_{t,k}). \tag*{(194)}
\end{aligned} log α k α t = β u = t ∑ k − 1 log ( 1 + u + b 1 ) ≤ c 0 β u = t ∑ k − 1 ( u + b ) β − 1 α u ≤ η t u = t ∑ k − 1 α u = η t ( α t + s t , k ) . (194)
The second inequality uses the fact that β − 1 < 0 \beta-1<0 β − 1 < 0 . Exponentiating gives
α t α k ≤ e λ / 4 e ( λ / 4 ) s t , k . (195) \frac{\alpha_t}{\alpha_k} \le e^{\lambda/4}e^{(\lambda/4)s_{t,k}}. \tag*{(195)} α k α t ≤ e λ /4 e ( λ /4 ) s t , k . (195)
We next prove (C.52). The intervals [ s t , k , s t , k + 1 ] [s_{t,k},s_{t,k+1}] [ s t , k , s t , k + 1 ] have lengths α k ≤ 1 \alpha_k \le1 α k ≤ 1 . Since ∑ u α u = ∞ \sum_u\alpha_u=\infty ∑ u α u = ∞ , they partition [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) up to endpoints. For a point x x x in the interval indexed by k k k ,
e − d s t , k ≤ e d e − d x , 1 + s t , k ≤ 1 + x . e^{-d s_{t,k}} \le e^d e^{-d x}, \qquad1+s_{t,k} \le1+x. e − d s t , k ≤ e d e − d x , 1 + s t , k ≤ 1 + x .
Multiplying and integrating over that interval gives
α k ( 1 + s t , k ) e − d s t , k ≤ e d ∫ s t , k s t , k + 1 ( 1 + x ) e − d x d x . \alpha_k(1+s_{t,k})e^{-d s_{t,k}}
\le e^d\int_{s_{t,k}}^{s_{t,k+1}}(1+x)e^{-d x}\,\mathrm{d}x. α k ( 1 + s t , k ) e − d s t , k ≤ e d ∫ s t , k s t , k + 1 ( 1 + x ) e − d x d x .
Summing from k = m k=m k = m onward proves the stated bound. In particular, the full sum is uniformly bounded, since
∫ 0 ∞ ( 1 + x ) e − d x d x = 1 d + 1 d 2 . \int_0^\infty(1+x)e^{-d x}\,\mathrm{d}x=\frac{1}{d}+\frac{1}{d^2}. ∫ 0 ∞ ( 1 + x ) e − d x d x = d 1 + d 2 1 .
Insert α k / α k \alpha_k/\alpha_k α k / α k into the infinite weight series. By the product estimate and (C.54),
α t ∑ k > t ∥ Φ t + 1 : k − 1 ∥ ≤ ∑ k > t α k α t α k e − λ s t , k ≤ e λ / 4 ∑ k > t α k e − ( 3 λ / 4 ) s t , k ≤ C . \alpha_t\sum_{k>t}\|\Phi_{t+1:k-1}\|
\le\sum_{k>t}\alpha_k\frac{\alpha_t}{\alpha_k}e^{-\lambda s_{t,k}}
\le e^{\lambda/4}\sum_{k>t}\alpha_k e^{-(3\lambda/4)s_{t,k}}
\le C. α t k > t ∑ ∥ Φ t + 1 : k − 1 ∥ ≤ k > t ∑ α k α k α t e − λ s t , k ≤ e λ /4 k > t ∑ α k e − ( 3 λ /4 ) s t , k ≤ C .
This proves absolute convergence and the uniform bound for W ∞ , t W_{\infty,t} W ∞ , t . The same estimate applies to every finite partial sum, proving the uniform bound for W n , t W_{n,t} W n , t .
Since every product factor is a polynomial in A A A , it commutes with A A A . For k > t k>t k > t ,
α k A Φ t + 1 : k − 1 = Φ t + 1 : k − 1 − Φ t + 1 : k . \alpha_k A\Phi_{t+1:k-1}=\Phi_{t+1:k-1}-\Phi_{t+1:k}. α k A Φ t + 1 : k − 1 = Φ t + 1 : k − 1 − Φ t + 1 : k .
Summing to a finite upper index m m m gives
A ∑ k = t + 1 m α k Φ t + 1 : k − 1 = I − Φ t + 1 : m . A\sum_{k=t+1}^{m}\alpha_k\Phi_{t+1:k-1}=I-\Phi_{t+1:m}. A k = t + 1 ∑ m α k Φ t + 1 : k − 1 = I − Φ t + 1 : m .
The product on the right tends to zero as m → ∞ m\to\infty m → ∞ . This proves (C.49).
Subtracting the telescoping identity from the infinite weight series gives
W ∞ , t − A − 1 = ∑ k > t α k ( α t α k − 1 ) Φ t + 1 : k − 1 . W_{\infty,t}-A^{-1}=\sum_{k>t}\alpha_k\left(\frac{\alpha_t}{\alpha_k}-1\right)\Phi_{t+1:k-1}. W ∞ , t − A − 1 = k > t ∑ α k ( α k α t − 1 ) Φ t + 1 : k − 1 .
For x ≥ 0 x\ge0 x ≥ 0 , e x − 1 ≤ x e x e^x-1\le xe^x e x − 1 ≤ x e x . Using this inequality and (C.53), we obtain
0 ≤ α t α k − 1 ≤ η t ( α t + s t , k ) e η t ( α t + s t , k ) . 0\le\frac{\alpha_t}{\alpha_k}-1\le\eta_t(\alpha_t+s_{t,k})e^{\eta_t(\alpha_t+s_{t,k})}. 0 ≤ α k α t − 1 ≤ η t ( α t + s t , k ) e η t ( α t + s t , k ) .
Thus, by α t ≤ 1 \alpha_t\le1 α t ≤ 1 and η t ≤ λ / 4 \eta_t\le\lambda/4 η t ≤ λ /4 ,
∥ W ∞ , t − A − 1 ∥ ≤ C η t ∑ k > t α k ( 1 + s t , k ) e − ( 3 λ / 4 ) s t , k ≤ C η t ≤ C ( t + b ) β − 1 . \left\|W_{\infty,t}-A^{-1}\right\|\le C\eta_t\sum_{k>t}\alpha_k(1+s_{t,k})e^{-(3\lambda/4)s_{t,k}}\le C\eta_t\le C(t+b)^{\beta-1}. W ∞ , t − A − 1 ≤ C η t k > t ∑ α k ( 1 + s t , k ) e − ( 3 λ /4 ) s t , k ≤ C η t ≤ C ( t + b ) β − 1 .
This proves the remaining bound in (C.48).
For the omitted tail, factor the products at time n n n :
W ∞ , t − W n , t = α t ∑ k = n + 1 ∞ Φ t + 1 : k − 1 = α t Φ t + 1 : n ∑ k = n + 1 ∞ Φ n + 1 : k − 1 = α t α n Φ t + 1 : n W ∞ , n . W_{\infty,t}-W_{n,t}=\alpha_t\sum_{k=n+1}^{\infty}\Phi_{t+1:k-1}=\alpha_t\Phi_{t+1:n}\sum_{k=n+1}^{\infty}\Phi_{n+1:k-1}=\frac{\alpha_t}{\alpha_n}\Phi_{t+1:n}W_{\infty,n}. W ∞ , t − W n , t = α t k = n + 1 ∑ ∞ Φ t + 1 : k − 1 = α t Φ t + 1 : n k = n + 1 ∑ ∞ Φ n + 1 : k − 1 = α n α t Φ t + 1 : n W ∞ , n .
This proves (C.50). Taking k = n k=n k = n in (C.53), we have
α t α n ≤ e λ / 4 exp ( λ 4 ∑ u = t + 1 n α u ) . \frac{\alpha_t}{\alpha_n}\le e^{\lambda/4}\exp\left(\frac{\lambda}{4}\sum_{u=t+1}^{n}\alpha_u\right). α n α t ≤ e λ /4 exp ( 4 λ u = t + 1 ∑ n α u ) .
Combining this with the product estimate and the uniform bound for W ∞ , n W_{\infty,n} W ∞ , n proves (C.51).
Under the stationary initialization used in the lower bound, C n C_n C n is also the actual covariance of M n M_n M n . For this one-action model with deterministic Q 0 Q_0 Q 0 , the filtration defined in § 2 reduces to F t = σ ( s 0 , … , s t + 1 ) \mathcal{F}_t=\sigma(s_0,\ldots,s_{t+1}) F t = σ ( s 0 , … , s t + 1 ) . The Bellman equation gives E [ ε t ∣ F t − 1 ] = 0 \mathbb{E}[\varepsilon_t\mid\mathcal{F}_{t-1}]=0 E [ ε t ∣ F t − 1 ] = 0 for t ≥ 1 t\ge1 t ≥ 1 and E ε 0 = 0 \mathbb{E}\varepsilon_0=0 E ε 0 = 0 . For s < t s<t s < t , the earlier increment is measurable with respect to F t − 1 \mathcal{F}_{t-1} F t − 1 , so
E [ ε s ε t T ] = E { ε s E [ ε t T ∣ F t − 1 ] } = 0. \mathbb{E}[\varepsilon_s\varepsilon_t^{\mathsf{T}}]=\mathbb{E}\{\varepsilon_s\mathbb{E}[\varepsilon_t^{\mathsf{T}}\mid\mathcal{F}_{t-1}]\}=0. E [ ε s ε t T ] = E { ε s E [ ε t T ∣ F t − 1 ]} = 0.
The stationary marginal second moment equals S S S at every time. Since the weights are deterministic, expanding the covariance therefore gives
Cov ( M n ) = 1 n ∑ s , t < n W n , s E [ ε s ε t T ] W n , t T = 1 n ∑ t < n W n , t S W n , t T = C n . \operatorname{Cov}(M_n)=\frac{1}{n}\sum_{s,t<n}W_{n,s}\mathbb{E}[\varepsilon_s\varepsilon_t^{\mathsf{T}}]W_{n,t}^{\mathsf{T}}=\frac{1}{n}\sum_{t<n}W_{n,t}SW_{n,t}^{\mathsf{T}}=C_n. Cov ( M n ) = n 1 s , t < n ∑ W n , s E [ ε s ε t T ] W n , t T = n 1 t < n ∑ W n , t S W n , t T = C n .
Lemma C.2. Let A ≻ 0 A\succ0 A ≻ 0 be symmetric and let B 0 B_0 B 0 , C 0 C_0 C 0 , E 0 E_0 E 0 be fixed matrices of compatible sizes. Define
B ( a ) = ( I − a A a B 0 a C 0 1 2 I + a E 0 ) . \mathcal{B}(a)=\begin{pmatrix}I-aA&aB_0\\aC_0&\frac{1}{2}I+aE_0\end{pmatrix}. B ( a ) = ( I − a A a C 0 a B 0 2 1 I + a E 0 ) . There exist a 0 , c > 0 a_0,c>0 a 0 , c > 0 such that, for 0 < a ≤ a 0 0<a\le a_0 0 < a ≤ a 0 ,
∥ B ( a ) ( u , f ) T ∥ 2 2 ≤ ( 1 − c a ) ( ∥ u ∥ 2 2 + ∥ f ∥ 2 2 ) . (196) \left\|\mathcal{B}(a)(u,f)^{\mathsf{T}}\right\|_2^2\le(1-ca)(\|u\|_2^2+\|f\|_2^2). \tag*{(196)} B ( a ) ( u , f ) T 2 2 ≤ ( 1 − c a ) ( ∥ u ∥ 2 2 + ∥ f ∥ 2 2 ) . (196)
For the matrices in (C.15) and every decreasing power stepsize, this implies
∥ ( T 0 + α t T 1 ) ⋯ ( T 0 + α k T 1 ) ∥ ≤ C exp ( − c ′ ∑ u = k t α u ) , k ≤ t . \left\|(T_0+\alpha_tT_1)\cdots(T_0+\alpha_kT_1)\right\|\le C\exp\left(-c'\sum_{u=k}^{t}\alpha_u\right),\qquad k\le t. ∥ ( T 0 + α t T 1 ) ⋯ ( T 0 + α k T 1 ) ∥ ≤ C exp ( − c ′ u = k ∑ t α u ) , k ≤ t .
Proof. Let λ = λ min ( A ) > 0 \lambda= \lambda_{\min}(A) > 0 λ = λ m i n ( A ) > 0 , L = ∥ A ∥ L = \lVert A\rVert L = ∥ A ∥ , and denote K B = ∥ B 0 ∥ K_B = \lVert B_0\rVert K B = ∥ B 0 ∥ , K C = ∥ C 0 ∥ K_C = \lVert C_0\rVert K C = ∥ C 0 ∥ , and K E = ∥ E 0 ∥ K_E = \lVert E_0\rVert K E = ∥ E 0 ∥ . Write the two blocks after applying the matrix as
u ′ = ( I − a A ) u + a B 0 f , f ′ = a C 0 u + ( 1 2 I + a E 0 ) f . u' = (I-aA)u + aB_0f,\qquad f' = aC_0u + \left(\tfrac{1}{2}I+aE_0\right)f. u ′ = ( I − a A ) u + a B 0 f , f ′ = a C 0 u + ( 2 1 I + a E 0 ) f .
Choose a a a small enough that a L ≤ 1 aL \le1 a L ≤ 1 and a L 2 ≤ λ / 2 aL^2 \le\lambda/2 a L 2 ≤ λ /2 . Positive definiteness and symmetry give
∥ ( I − a A ) u ∥ 2 2 ≤ ( 1 − 2 λ a + L 2 a 2 ) ∥ u ∥ 2 2 , ∥ I − a A ∥ ≤ 1. \lVert(I-aA)u\rVert_2^2 \le(1-2\lambda a+L^2a^2)\lVert u\rVert_2^2,\qquad\lVert I-aA\rVert\le1. ∥( I − a A ) u ∥ 2 2 ≤ ( 1 − 2 λa + L 2 a 2 ) ∥ u ∥ 2 2 , ∥ I − a A ∥ ≤ 1.
Expanding the square of the first block and applying Young’s inequality to its cross term, we have
∥ u ′ ∥ 2 2 ≤ ( 1 − 2 λ a + L 2 a 2 ) ∥ u ∥ 2 2 + 2 a K B ∥ u ∥ 2 ∥ f ∥ 2 + a 2 K B 2 ∥ f ∥ 2 2 ≤ ( 1 − 2 λ a + L 2 a 2 + λ a / 2 ) ∥ u ∥ 2 2 + ( 2 K B 2 a / λ + K B 2 a 2 ) ∥ f ∥ 2 2 ≤ ( 1 − λ a ) ∥ u ∥ 2 2 + C B a ∥ f ∥ 2 2 , \begin{aligned}
\lVert u'\rVert_2^2
&\le(1-2\lambda a+L^2a^2)\lVert u\rVert_2^2+2aK_B\lVert u\rVert_2\lVert f\rVert_2+a^2K_B^2\lVert f\rVert_2^2\\
&\le(1-2\lambda a+L^2a^2+\lambda a/2)\lVert u\rVert_2^2+(2K_B^2a/\lambda+K_B^2a^2)\lVert f\rVert_2^2\\
&\le(1-\lambda a)\lVert u\rVert_2^2+C_Ba\lVert f\rVert_2^2,
\end{aligned} ∥ u ′ ∥ 2 2 ≤ ( 1 − 2 λa + L 2 a 2 ) ∥ u ∥ 2 2 + 2 a K B ∥ u ∥ 2 ∥ f ∥ 2 + a 2 K B 2 ∥ f ∥ 2 2 ≤ ( 1 − 2 λa + L 2 a 2 + λa /2 ) ∥ u ∥ 2 2 + ( 2 K B 2 a / λ + K B 2 a 2 ) ∥ f ∥ 2 2 ≤ ( 1 − λa ) ∥ u ∥ 2 2 + C B a ∥ f ∥ 2 2 ,
where we also require a ≤ 1 a \le1 a ≤ 1 and can take C B = K B 2 ( 1 + 2 / λ ) C_B=K_B^2(1+2/\lambda) C B = K B 2 ( 1 + 2/ λ ) .
For the second block, choose a K E ≤ 1 / 8 aK_E \le1/8 a K E ≤ 1/8 . Then ∥ I / 2 + a E 0 ∥ ≤ 5 / 8 \lVert I/2+aE_0\rVert\le5/8 ∥ I /2 + a E 0 ∥ ≤ 5/8 . Using ∥ x + y ∥ 2 2 ≤ ( 3 / 2 ) ∥ x ∥ 2 2 + 3 ∥ y ∥ 2 2 \lVert x+y\rVert_2^2\le(3/2)\lVert x\rVert_2^2+3\lVert y\rVert_2^2 ∥ x + y ∥ 2 2 ≤ ( 3/2 ) ∥ x ∥ 2 2 + 3 ∥ y ∥ 2 2 with x = ( I / 2 + a E 0 ) f x=(I/2+aE_0)f x = ( I /2 + a E 0 ) f and y = a C 0 u y=aC_0u y = a C 0 u , we obtain
∥ f ′ ∥ 2 2 ≤ 3 2 ( 5 8 ) 2 ∥ f ∥ 2 2 + 3 K C 2 a 2 ∥ u ∥ 2 2 ≤ 3 4 ∥ f ∥ 2 2 + C C a 2 ∥ u ∥ 2 2 , C C = 3 K C 2 . \lVert f'\rVert_2^2\le\frac{3}{2}\left(\frac{5}{8}\right)^2\lVert f\rVert_2^2+3K_C^2a^2\lVert u\rVert_2^2\le\frac{3}{4}\lVert f\rVert_2^2+C_Ca^2\lVert u\rVert_2^2,\qquad C_C=3K_C^2. ∥ f ′ ∥ 2 2 ≤ 2 3 ( 8 5 ) 2 ∥ f ∥ 2 2 + 3 K C 2 a 2 ∥ u ∥ 2 2 ≤ 4 3 ∥ f ∥ 2 2 + C C a 2 ∥ u ∥ 2 2 , C C = 3 K C 2 .
Adding the two estimates gives
∥ u ′ ∥ 2 2 + ∥ f ′ ∥ 2 2 ≤ ( 1 − λ a + C C a 2 ) ∥ u ∥ 2 2 + ( 3 / 4 + C B a ) ∥ f ∥ 2 2 . \lVert u'\rVert_2^2+\lVert f'\rVert_2^2\le(1-\lambda a+C_Ca^2)\lVert u\rVert_2^2+(3/4+C_Ba)\lVert f\rVert_2^2. ∥ u ′ ∥ 2 2 + ∥ f ′ ∥ 2 2 ≤ ( 1 − λa + C C a 2 ) ∥ u ∥ 2 2 + ( 3/4 + C B a ) ∥ f ∥ 2 2 .
Decrease a 0 a_0 a 0 so that, in addition to the earlier conditions,
C C a 0 ≤ λ / 2 , ( C B + λ / 2 ) a 0 ≤ 1 / 4. C_Ca_0\le\lambda/2,\qquad(C_B+\lambda/2)a_0\le1/4. C C a 0 ≤ λ /2 , ( C B + λ /2 ) a 0 ≤ 1/4.
Both coefficients are then at most 1 − λ a / 2 1-\lambda a/2 1 − λa /2 . This proves (C.55) with c = λ / 2 c=\lambda/2 c = λ /2 . Taking square roots and using 1 − x ≤ e − x / 2 \sqrt{1-x}\le e^{-x/2} 1 − x ≤ e − x /2 gives
∥ B ( a ) ∥ ≤ e − λ a / 4 , 0 < a ≤ a 0 . \lVert B(a)\rVert\le e^{-\lambda a/4},\qquad0<a\le a_0. ∥ B ( a )∥ ≤ e − λa /4 , 0 < a ≤ a 0 .
This estimate holds in the same Euclidean norm for every such a a a . Thus, once the stepsizes are at most a 0 a_0 a 0 , multiplying the norm bounds gives an exponential in their sum. By (C.18), the original matrices are conjugate to these block matrices by the fixed invertible matrix M 0 M_0 M 0 . Conjugating the full product contributes the fixed factor ∥ M 0 ∥ ∥ M 0 − 1 ∥ \lVert M_0\rVert\lVert M_0^{-1}\rVert ∥ M 0 ∥ ∥ M 0 − 1 ∥ .
There are only finitely many indices before the stepsizes enter this range. Products containing these earlier factors can be split into a later contracting part and a finite earlier part. The norms of all possible earlier parts, multiplied by the exponential of their finite accumulated stepsizes, have a finite maximum. Including this maximum in C C C proves the bound for all k ≤ t k\le t k ≤ t . The common norm estimate just established is what allows the factors to have different stepsizes.
Supporting Lemmas
Lemma C.3 (Proposition A.1 in [18 ] ). If f f f is continuous and non-increasing on [ a , b + 1 ] [a,b+1] [ a , b + 1 ] , where a ≤ b a\le b a ≤ b are integers, then
∫ a b + 1 f ( x ) d x ≤ ∑ k = a b f ( k ) ≤ f ( a ) + ∫ a b f ( x ) d x . \int_a^{b+1} f(x)\,\mathrm{d}x\le\sum_{k=a}^{b}f(k)\le f(a)+\int_a^b f(x)\,\mathrm{d}x. ∫ a b + 1 f ( x ) d x ≤ k = a ∑ b f ( k ) ≤ f ( a ) + ∫ a b f ( x ) d x . In particular, this applies to every decreasing power function used in Lemma C.6 .
Lemma C.4 (Lemma 2.4(a) in [18 ] ). We use the asymptotic consequence of the cited lemma in arXiv version 2, with the shift denoted by η \eta η . Let 0 < ν < 1 0<\nu<1 0 < ν < 1 , q , c , d > 0 q,c,d>0 q , c , d > 0 , and η > 0 \eta>0 η > 0 . Suppose a nonnegative scalar sequence satisfies, from some fixed index onward,
a k + 1 ≤ ( 1 − c ( k + η ) ν ) a k + d ( k + η ) ν + q . a_{k+1}\leq\left(1-\frac{c}{(k+\eta)^\nu}\right)a_k+\frac{d}{(k+\eta)^{\nu+q}}. a k + 1 ≤ ( 1 − ( k + η ) ν c ) a k + ( k + η ) ν + q d . Then a k = O ( ( k + η ) − q ) a_k=O((k+\eta)^{-q}) a k = O (( k + η ) − q ) . More precisely, after shifting the initial index so that η ≥ c 1 / ν \eta\geq c^{1/\nu} η ≥ c 1/ ν and η > ( q / c ) 1 / ( 1 − ν ) \eta>(q/c)^{1/(1-\nu)} η > ( q / c ) 1/ ( 1 − ν ) , the cited lemma bounds a k a_k a k by a constant times ( k + η ) − q + exp [ − c ( k + η ) 1 − ν / ( 1 − ν ) ] (k+\eta)^{-q}+\exp[-c(k+\eta)^{1-\nu}/(1-\nu)] ( k + η ) − q + exp [ − c ( k + η ) 1 − ν / ( 1 − ν )] . The finite prefix affects only the constants.
Lemma C.5 (Theorem 2.24 in [14 ] ). If measurable scalar functions f n f_n f n converge almost everywhere to f f f and ∣ f n ∣ ≤ g |f_n|\leq g ∣ f n ∣ ≤ g for some g ∈ L 1 g\in L^1 g ∈ L 1 , then f ∈ L 1 f\in L^1 f ∈ L 1 and ∫ f n → ∫ f \int f_n\to\int f ∫ f n → ∫ f . The same conclusion holds in any fixed matrix norm for fixed-size matrix-valued functions with ∥ F n ∥ ≤ g \|F_n\|\leq g ∥ F n ∥ ≤ g and F n → F F_n\to F F n → F almost everywhere.
Lemma C.6. Let α t = c 0 ( t + b ) − β \alpha_t=c_0(t+b)^{-\beta} α t = c 0 ( t + b ) − β with b ≥ 1 b\geq1 b ≥ 1 and 1 / 2 < β < 1 1/2<\beta<1 1/2 < β < 1 . For fixed c 0 , b , β c_0,b,\beta c 0 , b , β ,
∑ t = 1 n α t = c 0 1 − β n 1 − β + O ( 1 ) , ∑ t = 1 n α t 2 = O ( 1 ) , \sum_{t=1}^{n}\alpha_t=\frac{c_0}{1-\beta}n^{1-\beta}+O(1),\qquad\sum_{t=1}^{n}\alpha_t^2=O(1), t = 1 ∑ n α t = 1 − β c 0 n 1 − β + O ( 1 ) , t = 1 ∑ n α t 2 = O ( 1 ) , ∑ t = 1 n ( t + b ) − 1 = O ( log ( e n ) ) , ∑ t = 0 ∞ α t t + b < ∞ . (197) \sum_{t=1}^{n}(t+b)^{-1}=O(\log(en)),\qquad\sum_{t=0}^{\infty}\frac{\alpha_t}{t+b}<\infty. \tag*{(197)} t = 1 ∑ n ( t + b ) − 1 = O ( log ( e n )) , t = 0 ∑ ∞ t + b α t < ∞. (197) If q t = ( t + b ) β − 1 q_t=(t+b)^{\beta-1} q t = ( t + b ) β − 1 , then
∑ t = 0 n − 1 q t = 1 β n β + O ( 1 ) , ∑ t = 0 n − 1 q t 2 = O ( n 2 β − 1 ) . (198) \sum_{t=0}^{n-1}q_t=\frac{1}{\beta}n^\beta+O(1),\qquad\sum_{t=0}^{n-1}q_t^2=O(n^{2\beta-1}). \tag*{(198)} t = 0 ∑ n − 1 q t = β 1 n β + O ( 1 ) , t = 0 ∑ n − 1 q t 2 = O ( n 2 β − 1 ) . (198) For any sequence of matrices with ∥ E t ∥ = o ( q t ) \|E_t\|=o(q_t) ∥ E t ∥ = o ( q t ) ,
1 n ∑ t = 0 n − 1 ∥ E t ∥ = o ( n β − 1 ) . (199) \frac{1}{n}\sum_{t=0}^{n-1}\|E_t\|=o(n^{\beta-1}). \tag*{(199)} n 1 t = 0 ∑ n − 1 ∥ E t ∥ = o ( n β − 1 ) . (199) Proof. Apply the integral test in Lemma C.3 to ( x + b ) − β (x+b)^{-\beta} ( x + b ) − β , ( x + b ) − 2 β (x+b)^{-2\beta} ( x + b ) − 2 β , ( x + b ) − 1 (x+b)^{-1} ( x + b ) − 1 , ( x + b ) − 1 − β (x+b)^{-1-\beta} ( x + b ) − 1 − β , ( x + b ) β − 1 (x+b)^{\beta-1} ( x + b ) β − 1 , and ( x + b ) 2 β − 2 (x+b)^{2\beta-2} ( x + b ) 2 β − 2 . Their elementary integrals give (C.56)–(C.57); fixed shifts change the displayed divergent power sums by O ( 1 ) O(1) O ( 1 ) . For (C.58), choose a fixed T T T with ∥ E t ∥ ≤ ϵ q t \|E_t\|\leq\epsilon q_t ∥ E t ∥ ≤ ϵ q t for t ≥ T t\geq T t ≥ T and write
n − β ∑ t < n ∥ E t ∥ ≤ n − β ∑ t < T ∥ E t ∥ + ϵ n − β ∑ T ≤ t < n q t . n^{-\beta}\sum_{t<n}\|E_t\|\leq n^{-\beta}\sum_{t<T}\|E_t\|+\epsilon n^{-\beta}\sum_{T\leq t<n}q_t. n − β t < n ∑ ∥ E t ∥ ≤ n − β t < T ∑ ∥ E t ∥ + ϵ n − β T ≤ t < n ∑ q t . Its limit superior is at most ϵ / β \epsilon/\beta ϵ / β ; let ϵ ↓ 0 \epsilon\downarrow0 ϵ ↓ 0 .
Lemma C.7. Let α t = c 0 ( t + b ) − β \alpha_t=c_0(t+b)^{-\beta} α t = c 0 ( t + b ) − β with 0 < β < 1 0<\beta<1 0 < β < 1 . Suppose the deterministic matrices B t B_t B t satisfy
∥ B t ⋯ B k ∥ ≤ C exp ( − c ∑ u = k t α u ) , k ≤ t , \|B_t\cdots B_k\|\leq C\exp\left(-c\sum_{u=k}^{t}\alpha_u\right),\qquad k\leq t, ∥ B t ⋯ B k ∥ ≤ C exp ( − c u = k ∑ t α u ) , k ≤ t , and, from a fixed time t 0 t_0 t 0 onward,
χ t + 1 = B t χ t + d t , ∥ d t ∥ ≤ C α t t + b . \chi_{t+1}=B_t\chi_t+d_t,\qquad\|d_t\|\leq C\frac{\alpha_t}{t+b}. χ t + 1 = B t χ t + d t , ∥ d t ∥ ≤ C t + b α t . For every fixed initial error, ∥ χ t ∥ = O ( ( t + b ) − 1 ) \|\chi_t\|=O((t+b)^{-1}) ∥ χ t ∥ = O (( t + b ) − 1 ) .
Proof. Unroll the matrix recursion using the assumed product bound. For a sufficiently large fixed C C C , the resulting scalar upper bound is the solution of
u t 0 = C ∥ χ t 0 ∥ , u t + 1 = e − c α t u t + C α t t + b , ∥ χ t ∥ ≤ u t . u_{t_0}=C\lVert\chi_{t_0}\rVert,\qquad u_{t+1}=e^{-c\alpha_t}u_t+C\frac{\alpha_t}{t+b},\qquad\lVert\chi_t\rVert\le u_t. u t 0 = C ∥ χ t 0 ∥ , u t + 1 = e − c α t u t + C t + b α t , ∥ χ t ∥ ≤ u t .
The last inequality follows from the unrolled products, and does not require a one-step contraction bound on B t B_t B t . Eventually e − c α t ≤ 1 − ( c / 2 ) α t e^{-c\alpha_t}\le1-(c/2)\alpha_t e − c α t ≤ 1 − ( c /2 ) α t , so
u t + 1 ≤ ( 1 − c c 0 / 2 ( t + b ) β ) u t + C c 0 ( t + b ) β + 1 . u_{t+1}\le\left(1-\frac{cc_0/2}{(t+b)^\beta}\right)u_t+\frac{Cc_0}{(t+b)^{\beta+1}}. u t + 1 ≤ ( 1 − ( t + b ) β c c 0 /2 ) u t + ( t + b ) β + 1 C c 0 .
Lemma C.4, with ν = β \nu=\beta ν = β , q = 1 q=1 q = 1 , and shift η = b \eta=b η = b , gives u t = O ( ( t + b ) − 1 ) u_t=O((t+b)^{-1}) u t = O (( t + b ) − 1 ) . A finite initial prefix only changes the constant.
Lemma C.8. Let a n > 0 a_n>0 a n > 0 satisfy a n → 0 a_n\to0 a n → 0 and n a n → ∞ na_n\to\infty n a n → ∞ . Let F n , k F_{n,k} F n , k , 1 ≤ k ≤ n 1\le k\le n 1 ≤ k ≤ n , be matrices of a fixed size. Suppose there is a continuous matrix function F F F on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) such that, for every fixed L > 0 L>0 L > 0 ,
max 1 ≤ k ≤ ⌊ L / a n ⌋ ∥ F n , k − F ( k a n ) ∥ ⟶ 0. (200) \max_{1\le k\le\lfloor L/a_n\rfloor}\lVert F_{n,k}-F(ka_n)\rVert\longrightarrow0. \tag*{(200)} 1 ≤ k ≤ ⌊ L / a n ⌋ max ∥ F n , k − F ( k a n )∥ ⟶ 0. (200) Assume also that ∥ F n , k ∥ ≤ C e − c k a n \lVert F_{n,k}\rVert\le Ce^{-cka_n} ∥ F n , k ∥ ≤ C e − c k a n and ∥ F ( x ) ∥ ≤ C e − c x \lVert F(x)\rVert\le Ce^{-cx} ∥ F ( x )∥ ≤ C e − c x for fixed C , c > 0 C,c>0 C , c > 0 . Then
a n ∑ k = 1 n F n , k ⟶ ∫ 0 ∞ F ( x ) d x . (201) a_n\sum_{k=1}^{n}F_{n,k}\longrightarrow\int_{0}^{\infty}F(x)\,dx. \tag*{(201)} a n k = 1 ∑ n F n , k ⟶ ∫ 0 ∞ F ( x ) d x . (201)
Proof. Define the matrix-valued step function
F ~ n ( x ) = { F n , ⌈ x / a n ⌉ , 0 < x ≤ n a n , 0 , x > n a n . \widetilde F_n(x)=
\begin{cases}
F_{n,\lceil x/a_n\rceil}, & 0<x\le na_n,\\
0, & x>na_n.
\end{cases} F n ( x ) = { F n , ⌈ x / a n ⌉ , 0 , 0 < x ≤ n a n , x > n a n .
For every fixed x > 0 x>0 x > 0 , n a n → ∞ na_n\to\infty n a n → ∞ , the local uniform approximation, and continuity of F F F imply F ~ n ( x ) → F ( x ) \widetilde F_n(x)\to F(x) F n ( x ) → F ( x ) . Moreover, ∥ F ~ n ( x ) ∥ ≤ C e − c x \lVert\widetilde F_n(x)\rVert\le Ce^{-cx} ∥ F n ( x )∥ ≤ C e − c x , since a n ⌈ x / a n ⌉ ≥ x a_n\lceil x/a_n\rceil\ge x a n ⌈ x / a n ⌉ ≥ x . Lemma C.5, applied entrywise, now gives
a n ∑ k = 1 n F n , k = ∫ 0 ∞ F ~ n ( x ) d x ⟶ ∫ 0 ∞ F ( x ) d x . a_n\sum_{k=1}^{n}F_{n,k}=\int_{0}^{\infty}\widetilde F_n(x)\,dx\longrightarrow\int_{0}^{\infty}F(x)\,dx. a n k = 1 ∑ n F n , k = ∫ 0 ∞ F n ( x ) d x ⟶ ∫ 0 ∞ F ( x ) d x .
Lemma C.9. Let A ≻ 0 A\succ0 A ≻ 0 be symmetric and S ≻ 0 S\succ0 S ≻ 0 . Then
∫ 0 ∞ e − A s d s = A − 1 , ∫ 0 ∞ s e − A s d s = A − 2 . (202) \int_{0}^{\infty}e^{-As}\,ds=A^{-1},\qquad\int_{0}^{\infty}se^{-As}\,ds=A^{-2}. \tag*{(202)} ∫ 0 ∞ e − A s d s = A − 1 , ∫ 0 ∞ s e − A s d s = A − 2 . (202) The matrix X = ∫ 0 ∞ e − A s S e − A T s d s X=\int_{0}^{\infty}e^{-As}Se^{-A^{\mathsf T}s}\,ds X = ∫ 0 ∞ e − A s S e − A T s d s is the unique solution to A X + X A T = S AX+XA^{\mathsf T}=S A X + X A T = S , and X ≻ 0 X\succ0 X ≻ 0 . For (S.12), its value is
X = 1 6615 ( 3392 1752 1752 3762 ) . (203) X=\frac{1}{6615}
\begin{pmatrix}
3392 & 1752\\
1752 & 3762
\end{pmatrix}. \tag*{(203)} X = 6615 1 ( 3392 1752 1752 3762 ) . (203)
Proof. Diagonalizing the symmetric positive-definite A A A and applying
∫ 0 ∞ e − λ s d s = λ − 1 and ∫ 0 ∞ s e − λ s d s = λ − 2 \int_{0}^{\infty}e^{-\lambda s}\,ds=\lambda^{-1}\qquad\text{and}\qquad\int_{0}^{\infty}se^{-\lambda s}\,ds=\lambda^{-2} ∫ 0 ∞ e − λ s d s = λ − 1 and ∫ 0 ∞ s e − λ s d s = λ − 2
proves (C.61). Lemma B.5, with U = V = A U=V=A U = V = A , gives the covariance integral and uniqueness. For u ≠ 0 u\ne0 u = 0 , its quadratic form is ∫ 0 ∞ ∥ S 1 / 2 e − A T s u ∥ 2 d s > 0 \int_{0}^{\infty}\lVert S^{1/2}e^{-A^{\mathsf T}s}u\rVert^2\,ds>0 ∫ 0 ∞ ∥ S 1/2 e − A T s u ∥ 2 d s > 0 , proving positivity. For the particular matrices in (S.12), direct multiplication of the matrix in (C.62) gives A X + X A T = diag ( 64 / 375 , 56 / 375 ) = S AX+XA^{\mathsf T}=\operatorname{diag}(64/375,56/375)=S A X + X A T = diag ( 64/375 , 56/375 ) = S ; uniqueness identifies it as the claimed covariance.
Lemma C.10. For random vectors U U U , V V V with finite first moments and a 1-Lipschitz function h h h ,
∣ E h ( U ) − E h ( V ) ∣ ≤ W 1 ( U , V ) . (204) |\mathbb{E}h(U)-\mathbb{E}h(V)| \le W_1(U,V). \tag*{(204)} ∣ E h ( U ) − E h ( V ) ∣ ≤ W 1 ( U , V ) . (204) In particular, W 1 ( U , V ) ≥ ∥ E U − E V ∥ 2 W_1(U,V) \ge\lVert\mathbb{E}U-\mathbb{E}V\rVert_2 W 1 ( U , V ) ≥ ∥ E U − E V ∥ 2 . For centered scalar Gaussians with standard deviations σ , τ ≥ 0 \sigma,\tau\ge0 σ , τ ≥ 0 ,
W 1 ( N ( 0 , σ 2 ) , N ( 0 , τ 2 ) ) = 2 π ∣ σ − τ ∣ . (205) W_1(\mathcal{N}(0,\sigma^2),\mathcal{N}(0,\tau^2))=\sqrt{\frac{2}{\pi}}|\sigma-\tau|. \tag*{(205)} W 1 ( N ( 0 , σ 2 ) , N ( 0 , τ 2 )) = π 2 ∣ σ − τ ∣. (205) Projection onto any coordinate does not increase W 1 W_1 W 1 . Consequently, if V n → V ≻ 0 V_n \to V \succ0 V n → V ≻ 0 and ( V n ) j j = V j j + g j δ n + o ( δ n ) (V_n)_{jj}=V_{jj}+g_j\delta_n+o(\delta_n) ( V n ) j j = V j j + g j δ n + o ( δ n ) with g j > 0 g_j>0 g j > 0 and δ n ↓ 0 \delta_n\downarrow0 δ n ↓ 0 , then
W 1 ( N ( 0 , V n ) , N ( 0 , V ) ) ≥ 2 π g j 2 V j j δ n + o ( δ n ) . (206) W_1(\mathcal{N}(0,V_n),\mathcal{N}(0,V))\ge\sqrt{\frac{2}{\pi}}\frac{g_j}{2\sqrt{V_{jj}}}\delta_n+o(\delta_n). \tag*{(206)} W 1 ( N ( 0 , V n ) , N ( 0 , V )) ≥ π 2 2 V j j g j δ n + o ( δ n ) . (206)
Proof. Lemma B.8 gives the Lipschitz test inequality, the mean lower bound, and contraction under coordinate projection. If G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) , the coupling ( σ G , τ G ) (\sigma G,\tau G) ( σ G , τ G ) gives an upper bound 2 / π ∣ σ − τ ∣ \sqrt{2/\pi}|\sigma-\tau| 2/ π ∣ σ − τ ∣ ; the 1-Lipschitz test x ↦ ∣ x ∣ x\mapsto|x| x ↦ ∣ x ∣ gives the same lower bound. This proves (C.64). For the variance expansion, write σ n = ( V n ) j j \sigma_n=\sqrt{(V_n)_{jj}} σ n = ( V n ) j j and σ = V j j > 0 \sigma=\sqrt{V_{jj}}>0 σ = V j j > 0 . Then
σ n − σ = ( V n ) j j − V j j σ n + σ = g j 2 V j j δ n + o ( δ n ) . \sigma_n-\sigma=\frac{(V_n)_{jj}-V_{jj}}{\sigma_n+\sigma}=\frac{g_j}{2\sqrt{V_{jj}}}\delta_n+o(\delta_n). σ n − σ = σ n + σ ( V n ) j j − V j j = 2 V j j g j δ n + o ( δ n ) .
Projection and the scalar formula give (C.65).
Proofs for Section 4
In this section, we prove the synchronous upper and lower bounds. The upper bound follows from Proposition 4.1, which separates the Bellman nonlinearity, the Gaussian approximation of the sampled fixed-policy recursion, and the deterministic covariance comparison. The lower bound uses the same MDP as in the asynchronous setting. Section D.4 provides supporting lemmas used below. Throughout this section, Q t Q_t Q t and X n X_n X n refer to the synchronous algorithm (S.17).
Proof of Theorem 4.1
Theorem 4.1. Under Assumption 4.1, let Z ∞ s y n ∼ N ( 0 , Σ ∞ s y n ) Z^{\mathrm{syn}}_\infty\sim\mathcal{N}(0,\Sigma^{\mathrm{syn}}_\infty) Z ∞ syn ∼ N ( 0 , Σ ∞ syn ) . For every n ≥ 2 n\ge2 n ≥ 2 ,
W 1 ( X n , Z ∞ s y n ) ≤ C { n − 1 / 2 log ( e n ) + n − β + n β − 1 } . (207) W_1(X_n,Z^{\mathrm{syn}}_\infty)\le C\{n^{-1/2}\log(en)+n^{-\beta}+n^{\beta-1}\}. \tag*{(207)} W 1 ( X n , Z ∞ syn ) ≤ C { n − 1/2 log ( e n ) + n − β + n β − 1 } . (207) In particular, β = 1 / 2 \beta=1/2 β = 1/2 gives C n − 1 / 2 log ( e n ) Cn^{-1/2}\log(en) C n − 1/2 log ( e n ) . For every fixed β ∈ ( 1 / 2 , 1 ) \beta\in(1/2,1) β ∈ ( 1/2 , 1 ) , the bound is O ( n β − 1 ) O(n^{\beta-1}) O ( n β − 1 ) .
Proof. Let λ s y n = λ min ( Σ ∞ s y n ) > 0 \lambda_{\mathrm{syn}}=\lambda_{\min}(\Sigma^{\mathrm{syn}}_\infty)>0 λ syn = λ m i n ( Σ ∞ syn ) > 0 . By (S.24), there exists n 0 n_0 n 0 such that λ min ( C n s y n ) ≥ λ s y n / 2 \lambda_{\min}(C_n^{\mathrm{syn}})\ge\lambda_{\mathrm{syn}}/2 λ m i n ( C n syn ) ≥ λ syn /2 for n ≥ n 0 n\ge n_0 n ≥ n 0 . Lemma B.11 then gives
W 1 ( Z n s y n , Z ∞ s y n ) ≤ C ∥ C n s y n − Σ ∞ s y n ∥ F ≤ C n β − 1 . W_1(Z_n^{\mathrm{syn}},Z^{\mathrm{syn}}_\infty)\le C\lVert C_n^{\mathrm{syn}}-\Sigma^{\mathrm{syn}}_\infty\rVert_F\le Cn^{\beta-1}. W 1 ( Z n syn , Z ∞ syn ) ≤ C ∥ C n syn − Σ ∞ syn ∥ F ≤ C n β − 1 .
Using the coupling in (S.20), the triangle inequality, and Proposition 4.1, we obtain
W 1 ( X n , Z ∞ s y n ) ≤ E ∥ X n − X n l i n ∥ 2 + W 1 ( X n l i n , Z n s y n ) + W 1 ( Z n s y n , Z ∞ s y n ) ≤ C { n − 1 / 2 log ( e n ) + n − β + n β − 1 } . \begin{aligned}
W_1(X_n,Z^{\mathrm{syn}}_\infty)
&\le\mathbb{E}\lVert X_n-X_n^{\mathrm{lin}}\rVert_2+W_1(X_n^{\mathrm{lin}},Z_n^{\mathrm{syn}})+W_1(Z_n^{\mathrm{syn}},Z^{\mathrm{syn}}_\infty)\\
&\le C\{n^{-1/2}\log(en)+n^{-\beta}+n^{\beta-1}\}.
\end{aligned} W 1 ( X n , Z ∞ syn ) ≤ E ∥ X n − X n lin ∥ 2 + W 1 ( X n lin , Z n syn ) + W 1 ( Z n syn , Z ∞ syn ) ≤ C { n − 1/2 log ( e n ) + n − β + n β − 1 } .
The finitely many 2 ≤ n < n 0 2\le n<n_0 2 ≤ n < n 0 are covered by increasing C C C , since ∥ X n ∥ 2 ≤ H d n \lVert X_n\rVert_2\le H\sqrt{dn} ∥ X n ∥ 2 ≤ H d n and Z ∞ s y n Z^{\mathrm{syn}}_\infty Z ∞ syn has a finite first moment. Substitution of β = 1 / 2 \beta=1/2 β = 1/2 proves the endpoint bound. For fixed β > 1 / 2 \beta>1/2 β > 1/2 , both n − 1 / 2 log ( e n ) n^{-1/2}\log(en) n − 1/2 log ( e n ) and n − β n^{-\beta} n − β are o ( n β − 1 ) o(n^{\beta-1}) o ( n β − 1 ) .
Proof of Proposition 4.1
Proposition 4.1. Under the assumptions of Theorem 4.1, for every n ≥ 2 n \ge2 n ≥ 2 , we have
E ∥ X n − X n l i n ∥ 2 ≤ C n − 1 / 2 , (208) \mathbb{E}\lVert X_n-X_n^{\mathrm{lin}}\rVert_2 \le Cn^{-1/2}, \tag*{(208)} E ∥ X n − X n lin ∥ 2 ≤ C n − 1/2 , (208) W 1 ( X n l i n , Z n s y n ) ≤ C { n − 1 / 2 log ( e n ) + n − β } , (209) W_1(X_n^{\mathrm{lin}},Z_n^{\mathrm{syn}}) \le C\{n^{-1/2}\log(en)+n^{-\beta}\}, \tag*{(209)} W 1 ( X n lin , Z n syn ) ≤ C { n − 1/2 log ( e n ) + n − β } , (209) ∥ C n s y n − Σ ∞ s y n ∥ F ≤ C n β − 1 . (210) \lVert C_n^{\mathrm{syn}}-\Sigma_\infty^{\mathrm{syn}}\rVert_F \le Cn^{\beta-1}. \tag*{(210)} ∥ C n syn − Σ ∞ syn ∥ F ≤ C n β − 1 . (210)
Proof. Write F t s y n = σ ( Q 0 , ω 0 , … , ω t ) \mathcal{F}_t^{\mathrm{syn}}=\sigma(Q_0,\omega_0,\ldots,\omega_t) F t syn = σ ( Q 0 , ω 0 , … , ω t ) and F t − 1 s y n = σ ( Q 0 ) \mathcal{F}_{t-1}^{\mathrm{syn}}=\sigma(Q_0) F t − 1 syn = σ ( Q 0 ) . The current block ω t \omega_t ω t is independent of F t − 1 s y n \mathcal{F}_{t-1}^{\mathrm{syn}} F t − 1 syn . Both Q t Q_t Q t and L t L_t L t are F t − 1 s y n \mathcal{F}_{t-1}^{\mathrm{syn}} F t − 1 syn -measurable and remain in [ 0 , H ] d [0,H]^d [ 0 , H ] d . For later use, set
B t = ( 1 − α t ) I + α t γ P ^ t π ⋆ , ρ i : j = ∏ u = i j ( 1 − α s y n α u ) , B_t=(1-\alpha_t)I+\alpha_t\gamma\widehat{P}_t^{\pi^\star},\qquad\rho_{i:j}=\prod_{u=i}^{j}(1-\alpha_{\mathrm{syn}}\alpha_u), B t = ( 1 − α t ) I + α t γ P t π ⋆ , ρ i : j = u = i ∏ j ( 1 − α syn α u ) ,
with ρ i : j = 1 \rho_{i:j}=1 ρ i : j = 1 when i > j i>j i > j . Every sampled matrix is nonnegative with row sums 1 − α s y n α t 1-\alpha_{\mathrm{syn}}\alpha_t 1 − α syn α t . Thus, for every configuration of the sample blocks, including arbitrary replacements of any of them,
∥ B j ⋯ B i ∥ ∞ ≤ ρ i : j . (211) \lVert B_j\cdots B_i\rVert_\infty\le\rho_{i:j}. \tag*{(211)} ∥ B j ⋯ B i ∥ ∞ ≤ ρ i : j . (211)
Reduction to the fixed-policy recursion. Let
( T ^ t q ) ( s , a ) = r ( s , a ) + γ V ( q ) ( S t ′ ( s , a ) ) , g t ( q ) = T ^ t q − ( r + γ P ^ t π ⋆ q ) . (\widehat{T}_tq)(s,a)=r(s,a)+\gamma V(q)(S'_t(s,a)),\qquad g_t(q)=\widehat{T}_tq-(r+\gamma\widehat{P}_t^{\pi^\star}q). ( T t q ) ( s , a ) = r ( s , a ) + γ V ( q ) ( S t ′ ( s , a )) , g t ( q ) = T t q − ( r + γ P t π ⋆ q ) .
With one action, g t = 0 g_t=0 g t = 0 . Otherwise, optimality of π ⋆ \pi^\star π ⋆ gives, for every q ∈ [ 0 , H ] d q\in[0,H]^d q ∈ [ 0 , H ] d ,
∥ g t ( q ) ∥ ∞ ≤ 2 γ ∥ q − Q ⋆ ∥ ∞ 1 { ∥ q − Q ⋆ ∥ ∞ ≥ κ / 2 } . (212) \lVert g_t(q)\rVert_\infty\le2\gamma\lVert q-Q^\star\rVert_\infty\mathbf{1}\{\lVert q-Q^\star\rVert_\infty\ge\kappa/2\}. \tag*{(212)} ∥ g t ( q ) ∥ ∞ ≤ 2 γ ∥ q − Q ⋆ ∥ ∞ 1 {∥ q − Q ⋆ ∥ ∞ ≥ κ /2 } . (212)
Below this threshold the greedy action is π ⋆ \pi^\star π ⋆ at every state, and in general 0 ≤ V ( q ) ( s ) − q ( s , π ⋆ ( s ) ) ≤ 2 ∥ q − Q ⋆ ∥ ∞ 0\le V(q)(s)-q(s,\pi^\star(s))\le2\lVert q-Q^\star\rVert_\infty 0 ≤ V ( q ) ( s ) − q ( s , π ⋆ ( s )) ≤ 2 ∥ q − Q ⋆ ∥ ∞ . Lemma D.2 therefore implies
E ∥ g t ( Q t ) ∥ ∞ ≤ 2 γ ( 2 / κ ) 5 E ∥ Q t − Q ⋆ ∥ ∞ 6 ≤ C α t 3 . \mathbb{E}\lVert g_t(Q_t)\rVert_\infty\le2\gamma(2/\kappa)^5\mathbb{E}\lVert Q_t-Q^\star\rVert_\infty^6\le C\alpha_t^3. E ∥ g t ( Q t ) ∥ ∞ ≤ 2 γ ( 2/ κ ) 5 E ∥ Q t − Q ⋆ ∥ ∞ 6 ≤ C α t 3 .
Since 3 β > 1 3\beta>1 3 β > 1 , these expectations are summable, including at β = 1 / 2 \beta=1/2 β = 1/2 . Subtracting (S.20) from (S.17) gives
Q t + 1 − L t + 1 = B t ( Q t − L t ) + α t g t ( Q t ) . Q_{t+1}-L_{t+1}=B_t(Q_t-L_t)+\alpha_tg_t(Q_t). Q t + 1 − L t + 1 = B t ( Q t − L t ) + α t g t ( Q t ) .
Unrolling this recursion, applying (D.1), and then using the response bound in Lemma D.1 , we obtain
E ∥ X n − X n l i n ∥ 2 ≤ d n ∑ t < n α t ∑ k = t + 1 n ρ t + 1 : k − 1 E ∥ g t ( Q t ) ∥ ∞ ≤ C n ∑ t < n α t 3 ≤ C n − 1 / 2 . \begin{aligned}
\mathbb{E}\lVert X_n-X_n^{\mathrm{lin}}\rVert_2
&\le\frac{\sqrt{d}}{\sqrt{n}}\sum_{t<n}\alpha_t\sum_{k=t+1}^{n}\rho_{t+1:k-1}\mathbb{E}\lVert g_t(Q_t)\rVert_\infty\\
&\le\frac{C}{\sqrt{n}}\sum_{t<n}\alpha_t^3\le Cn^{-1/2}.
\end{aligned} E ∥ X n − X n lin ∥ 2 ≤ n d t < n ∑ α t k = t + 1 ∑ n ρ t + 1 : k − 1 E ∥ g t ( Q t ) ∥ ∞ ≤ n C t < n ∑ α t 3 ≤ C n − 1/2 .
This proves (S.22). The covariance bound (S.24) is part of Lemma D.1 .
Mean and covariance of the fixed-policy average. Write
e t = L t − Q ∗ , ζ t = γ ( P π ∗ − P ^ t π ∗ ) , η t = ε t s y n − ζ t e t . e_t = L_t - Q^*, \qquad\zeta_t = \gamma(P^{\pi^*} - \widehat{P}_t^{\pi^*}), \qquad\eta_t = \varepsilon_t^{\mathrm{syn}} - \zeta_t e_t. e t = L t − Q ∗ , ζ t = γ ( P π ∗ − P t π ∗ ) , η t = ε t syn − ζ t e t .
Then
e t + 1 = ( I − α t A s y n ) e t + α t η t , E [ η t ∣ F t − 1 s y n ] = 0. (213) e_{t+1} = (I - \alpha_t A^{\mathrm{syn}})e_t + \alpha_t\eta_t, \qquad\mathbb{E}[\eta_t \mid\mathcal{F}_{t-1}^{\mathrm{syn}}] = 0. \tag*{(213)} e t + 1 = ( I − α t A syn ) e t + α t η t , E [ η t ∣ F t − 1 syn ] = 0. (213)
Since Q 0 Q_0 Q 0 is deterministic, the mean is m t : = E e t = Φ 0 : t − 1 s y n e 0 m_t := \mathbb{E}e_t = \Phi_{0:t-1}^{\mathrm{syn}}e_0 m t := E e t = Φ 0 : t − 1 syn e 0 . Product contraction and β < 1 \beta< 1 β < 1 give
∥ m t ∥ 2 ≤ C exp { − c ( t + b ) 1 − β } , ∥ E X n l i n ∥ 2 ≤ C n − 1 / 2 . (214) \lVert m_t\rVert_2 \leq C\exp\{-c(t+b)^{1-\beta}\}, \qquad\lVert\mathbb{E}X_n^{\mathrm{lin}}\rVert_2 \leq Cn^{-1/2}. \tag*{(214)} ∥ m t ∥ 2 ≤ C exp { − c ( t + b ) 1 − β } , ∥ E X n lin ∥ 2 ≤ C n − 1/2 . (214)
The constant in the first bound absorbs the fixed initial shift. Set
F n = X n l i n − E X n l i n , Γ n = Cov ( F n ) , J t = E [ η t η t T ] . F_n = X_n^{\mathrm{lin}} - \mathbb{E}X_n^{\mathrm{lin}}, \qquad\Gamma_n = \operatorname{Cov}(F_n), \qquad J_t = \mathbb{E}[\eta_t\eta_t^{\mathsf{T}}]. F n = X n lin − E X n lin , Γ n = Cov ( F n ) , J t = E [ η t η t T ] .
Although ζ t \zeta_t ζ t and ε t s y n \varepsilon_t^{\mathrm{syn}} ε t syn use the same block, that block is independent of e t e_t e t . Expanding the covariance accordingly gives the exact identity
J t − S s y n = − E [ ε t s y n m t T ζ t T ] − E [ ζ t m t ( ε t s y n ) T ] + E [ ζ t e t e t T ζ t T ] . J_t - S^{\mathrm{syn}} = -\mathbb{E}[\varepsilon_t^{\mathrm{syn}}m_t^{\mathsf{T}}\zeta_t^{\mathsf{T}}] - \mathbb{E}[\zeta_t m_t(\varepsilon_t^{\mathrm{syn}})^{\mathsf{T}}] + \mathbb{E}[\zeta_t e_t e_t^{\mathsf{T}}\zeta_t^{\mathsf{T}}]. J t − S syn = − E [ ε t syn m t T ζ t T ] − E [ ζ t m t ( ε t syn ) T ] + E [ ζ t e t e t T ζ t T ] .
Boundedness of the sampled matrices, Lemma D.2 , and (D.4) imply
∥ J t − S s y n ∥ F ≤ C { ∥ m t ∥ 2 + E ∥ e t ∥ 2 2 } ≤ C α t . (215) \lVert J_t - S^{\mathrm{syn}}\rVert_F \leq C\{\lVert m_t\rVert_2 + \mathbb{E}\lVert e_t\rVert_2^2\} \leq C\alpha_t. \tag*{(215)} ∥ J t − S syn ∥ F ≤ C {∥ m t ∥ 2 + E ∥ e t ∥ 2 2 } ≤ C α t . (215)
In particular, the cross terms involve the mean m t m_t m t , rather than E ∥ e t ∥ 2 \mathbb{E}\lVert e_t\rVert_2 E ∥ e t ∥ 2 . Unrolling (D.3) and using martingale orthogonality gives
F n = 1 n ∑ t < n W n , t s y n η t , Γ n = 1 n ∑ t < n W n , t s y n J t ( W n , t s y n ) T . (216) F_n = \frac{1}{\sqrt{n}}\sum_{t<n} W_{n,t}^{\mathrm{syn}}\eta_t, \qquad\Gamma_n = \frac{1}{n}\sum_{t<n} W_{n,t}^{\mathrm{syn}}J_t(W_{n,t}^{\mathrm{syn}})^{\mathsf{T}}. \tag*{(216)} F n = n 1 t < n ∑ W n , t syn η t , Γ n = n 1 t < n ∑ W n , t syn J t ( W n , t syn ) T . (216)
Uniform boundedness of the weights now yields
∥ Γ n − C n s y n ∥ F ≤ C n ∑ t < n α t ≤ C n − β . (217) \lVert\Gamma_n - C_n^{\mathrm{syn}}\rVert_F \leq\frac{C}{n}\sum_{t<n}\alpha_t \leq Cn^{-\beta}. \tag*{(217)} ∥ Γ n − C n syn ∥ F ≤ n C t < n ∑ α t ≤ C n − β . (217)
Sensitivity to the independent sample blocks. Regard F n = f n ( ω 0 , … , ω n − 1 ) F_n = f_n(\omega_0,\ldots,\omega_{n-1}) F n = f n ( ω 0 , … , ω n − 1 ) . Use D i D_i D i for the difference caused by replacing input block i i i by another admissible block; the bounds below are uniform over the original blocks and all replacements. For distinct i , j i,j i , j , D i D j D_iD_j D i D j is the corresponding four-term difference. We claim that
∥ D i f n ∥ 2 ≤ C n − 1 / 2 , ∥ D i D j f n ∥ 2 ≤ b i j : = C n − 1 / 2 α min ( i , j ) ρ min ( i , j ) + 1 : max ( i , j ) − 1 . (218) \lVert D_i f_n\rVert_2 \leq Cn^{-1/2}, \qquad\lVert D_iD_j f_n\rVert_2 \leq b_{ij} := Cn^{-1/2}\alpha_{\min(i,j)}\rho_{\min(i,j)+1:\max(i,j)-1}. \tag*{(218)} ∥ D i f n ∥ 2 ≤ C n − 1/2 , ∥ D i D j f n ∥ 2 ≤ b ij := C n − 1/2 α m i n ( i , j ) ρ m i n ( i , j ) + 1 : m a x ( i , j ) − 1 . (218)
A replacement at time i i i changes L i + 1 L_{i+1} L i + 1 by at most C α i C\alpha_i C α i , since L i L_i L i stays in the cube. At a later time k k k , its effect is at most C α i ρ i + 1 : k − 1 C\alpha_i\rho_{i+1:k-1} C α i ρ i + 1 : k − 1 . Summing over the average and applying Lemma D.1 proves the first bound. For i < j i<j i < j , the four-term difference first appears at time j + 1 j+1 j + 1 . At that time it is α j γ ( P ^ j π ∗ − P ^ j π ′ ∗ ) \alpha_j\gamma(\widehat{P}_j^{\pi^*}-\widehat{P}_j^{\pi^{\prime*}}) α j γ ( P j π ∗ − P j π ′ ∗ ) times the change in L j L_j L j caused by block i i i . Its norm is at most C α i α j ρ i + 1 : j − 1 C\alpha_i\alpha_j\rho_{i+1:j-1} C α i α j ρ i + 1 : j − 1 . Subsequent common updates are linear and satisfy (D.1). The response bound starting at j j j proves the second assertion.
The row sums of these deterministic sensitivity bounds satisfy
max i ∑ j ≠ i b i j ≤ C n − 1 / 2 . (219) \max_i\sum_{j\ne i}b_{ij}\leq Cn^{-1/2}. \tag*{(219)} i max j = i ∑ b ij ≤ C n − 1/2 . (219)
For j > i j>i j > i , this follows from the response bound in Lemma D.1. For j < i j<i j < i , use its telescoping identity ∑ j < i α j ρ j + 1 : i − 1 ≤ a s y n − 1 \sum_{j<i}\alpha_j\rho_{j+1:i-1}\leq a_{\mathrm{syn}}^{-1} ∑ j < i α j ρ j + 1 : i − 1 ≤ a syn − 1 .
Gaussian approximation by resampling. Let ω ′ \omega' ω ′ be an independent copy of ω \omega ω . For J ⊆ I n : = { 0 , … , n − 1 } \mathcal{J} \subseteq\mathcal{I}_n := \{0,\ldots,n-1\} J ⊆ I n := { 0 , … , n − 1 } , let ω J \omega^{\mathcal{J}} ω J replace the blocks in J \mathcal{J} J by their copies. For j ∉ J j \notin\mathcal{J} j ∈ / J , write D j f n ( ω J ) = f n ( ω J ) − f n ( ω J ∪ { j } ) \mathcal{D}_j f_n(\omega^{\mathcal{J}})=f_n(\omega^{\mathcal{J}})-f_n(\omega^{\mathcal{J}\cup\{j\}}) D j f n ( ω J ) = f n ( ω J ) − f n ( ω J ∪ { j } ) , and put
K n = 1 2 ∑ J ⊆ I n ϑ J ∑ j ∉ J D j f n ( ω J ) ( D j f n ( ω ) ) ⊤ , ϑ J = 1 ( n ∣ J ∣ ) ( n − ∣ J ∣ ) . (220) \mathcal{K}_n=\frac{1}{2}\sum_{\mathcal{J}\subseteq\mathcal{I}_n}\vartheta_{\mathcal{J}}\sum_{j\notin\mathcal{J}}\mathcal{D}_j f_n(\omega^{\mathcal{J}})(\mathcal{D}_j f_n(\omega))^\top,\qquad\vartheta_{\mathcal{J}}=\frac{1}{\binom{n}{|\mathcal{J}|}(n-|\mathcal{J}|)}. \tag*{(220)} K n = 2 1 J ⊆ I n ∑ ϑ J j ∈ / J ∑ D j f n ( ω J ) ( D j f n ( ω ) ) ⊤ , ϑ J = ( ∣ J ∣ n ) ( n − ∣ J ∣ ) 1 . (220)
Lemma D.3, applied coordinatewise, gives E K n = Γ n \mathbb{E}\mathcal{K}_n=\Gamma_n E K n = Γ n and ∑ J : j ∉ J ϑ J = 1 \sum_{\mathcal{J}:j\notin\mathcal{J}}\vartheta_{\mathcal{J}}=1 ∑ J : j ∈ / J ϑ J = 1 for every j j j . Replacing one of the 2 n 2n 2 n independent blocks in ( ω , ω ′ ) (\omega,\omega') ( ω , ω ′ ) changes K n \mathcal{K}_n K n in Frobenius norm by at most
C { n − 1 + n − 1 / 2 ∑ j ≠ i b i j } ≤ C n − 1 . C\left\{n^{-1}+n^{-1/2}\sum_{j\ne i}b_{ij}\right\}\le Cn^{-1}. C ⎩ ⎨ ⎧ n − 1 + n − 1/2 j = i ∑ b ij ⎭ ⎬ ⎫ ≤ C n − 1 .
Here the terms j = i j=i j = i use the first bound in (D.8); all other terms use the four-term bound there. The variance inequality in Lemma D.3 therefore implies
E ∥ K n − Γ n ∥ F 2 ≤ C / n , E ∥ K n − C n s y n ∥ F ≤ C ( n − 1 / 2 + n − β ) , (221) \mathbb{E}\lVert\mathcal{K}_n-\Gamma_n\rVert_F^2\le C/n,\qquad\mathbb{E}\lVert\mathcal{K}_n-C_n^{\mathrm{syn}}\rVert_F\le C(n^{-1/2}+n^{-\beta}), \tag*{(221)} E ∥ K n − Γ n ∥ F 2 ≤ C / n , E ∥ K n − C n syn ∥ F ≤ C ( n − 1/2 + n − β ) , (221)
where the second estimate also uses (D.7).
For large n n n , the eigenvalues of C n s y n C_n^{\mathrm{syn}} C n syn are bounded above and below by fixed positive constants, by (S.24). Fix a Euclidean 1-Lipschitz function h h h and 0 < δ ≤ 1 / 2 0<\delta\le1/2 0 < δ ≤ 1/2 . Lemma D.4, with V = C n s y n V=C_n^{\mathrm{syn}} V = C n syn , supplies a smoothed test h δ h_\delta h δ and a solution ψ δ \psi_\delta ψ δ to
tr ( C n s y n D 2 ψ δ ( x ) ) − x ⊤ ∇ ψ δ ( x ) = h δ ( x ) − E h δ ( Z n s y n ) , \operatorname{tr}(C_n^{\mathrm{syn}}D^2\psi_\delta(x))-x^\top\nabla\psi_\delta(x)=h_\delta(x)-\mathbb{E}h_\delta(Z_n^{\mathrm{syn}}), tr ( C n syn D 2 ψ δ ( x )) − x ⊤ ∇ ψ δ ( x ) = h δ ( x ) − E h δ ( Z n syn ) ,
with
∥ h δ − h ∥ sup ≤ C δ , ∥ D 2 ψ δ ∥ sup ≤ C , ∥ D 3 ψ δ ∥ sup ≤ C ( 1 + ∣ log δ ∣ ) . \lVert h_\delta-h\rVert_{\sup}\le C\delta,\qquad\lVert D^2\psi_\delta\rVert_{\sup}\le C,\qquad\lVert D^3\psi_\delta\rVert_{\sup}\le C(1+|\log\delta|). ∥ h δ − h ∥ s u p ≤ C δ , ∥ D 2 ψ δ ∥ s u p ≤ C , ∥ D 3 ψ δ ∥ s u p ≤ C ( 1 + ∣ log δ ∣ ) .
Apply the covariance identity in Lemma D.3 with g = ∂ a ψ δ ∘ f n g=\partial_a\psi_\delta\circ f_n g = ∂ a ψ δ ∘ f n and f = f n , a f=f_{n,a} f = f n , a , and sum over a a a . Taylor expansion at F n = f n ( ω ) F_n=f_n(\omega) F n = f n ( ω ) gives
E [ F n ⊤ ∇ ψ δ ( F n ) ] = E ⟨ K n , D 2 ψ δ ( F n ) ⟩ F + E n , ∣ E n ∣ ≤ C ( 1 + ∣ log δ ∣ ) n − 1 / 2 . (222) \mathbb{E}[F_n^\top\nabla\psi_\delta(F_n)]=\mathbb{E}\langle\mathcal{K}_n,D^2\psi_\delta(F_n)\rangle_F+\mathcal{E}_n,\qquad|\mathcal{E}_n|\le C(1+|\log\delta|)n^{-1/2}. \tag*{(222)} E [ F n ⊤ ∇ ψ δ ( F n )] = E ⟨ K n , D 2 ψ δ ( F n ) ⟩ F + E n , ∣ E n ∣ ≤ C ( 1 + ∣ log δ ∣ ) n − 1/2 . (222)
Note that each Taylor remainder is bounded by C ∥ D 3 ψ δ ∥ sup ∥ D j f n ( ω ) ∥ 2 ∥ D j f n ( ω J ) ∥ 2 C\lVert D^3\psi_\delta\rVert_{\sup}\lVert\mathcal{D}_j f_n(\omega)\rVert_2\lVert\mathcal{D}_j f_n(\omega^{\mathcal{J}})\rVert_2 C ∥ D 3 ψ δ ∥ s u p ∥ D j f n ( ω ) ∥ 2 ∥ D j f n ( ω J ) ∥ 2 . The first-difference bound makes this C ( 1 + ∣ log δ ∣ ) n − 3 / 2 C(1+|\log\delta|)n^{-3/2} C ( 1 + ∣ log δ ∣ ) n − 3/2 , and the subset weights sum to one for each of the n n n blocks. Combining the Stein equation, (D.11), and (D.12), we obtain
∣ E h ( F n ) − E h ( Z n s y n ) ∣ ≤ C { δ + n − β + n − 1 / 2 ( 1 + ∣ log δ ∣ ) } . |\mathbb{E}h(F_n)-\mathbb{E}h(Z_n^{\mathrm{syn}})|\le C\{\delta+n^{-\beta}+n^{-1/2}(1+|\log\delta|)\}. ∣ E h ( F n ) − E h ( Z n syn ) ∣ ≤ C { δ + n − β + n − 1/2 ( 1 + ∣ log δ ∣ )} .
Taking δ = n − 1 / 2 \delta=n^{-1/2} δ = n − 1/2 for n ≥ 4 n\ge4 n ≥ 4 , then the supremum over h h h , and restoring the mean using (D.4), proves (S.23) for all sufficiently large n n n . Cube invariance and boundedness of C n s y n C_n^{\mathrm{syn}} C n syn cover the finitely many remaining horizons after increasing C C C .
Proof of Theorem 4.2
Theorem 4.2. For the MDP in (S.11) with synchronous sampling and Q 0 = 0 Q_0=0 Q 0 = 0 , and for fixed choice of c 0 c_0 c 0 , b b b , β \beta β , constants c > 0 c>0 c > 0 and n 0 < ∞ n_0<\infty n 0 < ∞ such that, for n ≥ n 0 n\ge n_0 n ≥ n 0 ,
W 1 ( X n , Z ∞ s y n ) ≥ c { n − 1 / 2 , β = 1 / 2 , n β − 1 , 1 / 2 < β < 1. (223) W_1(X_n,Z_\infty^{\mathrm{syn}})\ge c\begin{cases} n^{-1/2}, & \beta=1/2, \\ n^{\beta-1}, & 1/2<\beta<1. \end{cases} \tag*{(223)} W 1 ( X n , Z ∞ syn ) ≥ c { n − 1/2 , n β − 1 , β = 1/2 , 1/2 < β < 1. (223) The MDP and initialization are independent of the stepsize parameters.
Proof. There is one action, so Q t = L t Q_t=L_t Q t = L t and X_n=X_n^{\lin} . On the MDP in (S.11), direct calculation gives
Q^\star=\begin{pmatrix}4/3\\3\end{pmatrix},\qquad A^{\syn}=\frac{1}{25}\begin{pmatrix}9&-4\\-6&11\end{pmatrix},\qquad S^{\syn}=\operatorname{diag}(64/225,28/75)\succ0. \tag*{(224)}
Thus Assumption 4.1 holds.
First suppose 1 / 2 < β < 1 1/2<\beta<1 1/2 < β < 1 . We apply the deterministic covariance expansion in the proof of Proposition 3.2 after symmetrizing the drift. Let
T=\operatorname{diag}(\sqrt{3/5},\sqrt{2/5}),\qquad\widetilde{S}=TS^{\syn}T^\top=\operatorname{diag}(64/375,56/375),
\widetilde{A}=TA^{\syn}T^{-1}=\frac{1}{25}\begin{pmatrix}9&-2\sqrt{6}\\-2\sqrt{6}&11\end{pmatrix}. \tag*{(225)}
The matrix A ~ \widetilde{A} A is symmetric positive definite with eigenvalues 1 / 5 1/5 1/5 and 3 / 5 3/5 3/5 . The stepsize assumptions give
α 0 λ max ( A ~ ) ≤ 1 , β c 0 β β − 1 ≤ 1 20 = λ min ( A ~ ) 4 . \alpha_0\lambda_{\max}(\widetilde{A})\le1,\qquad\frac{\beta}{c_0}\beta^{\beta-1}\le\frac{1}{20}=\frac{\lambda_{\min}(\widetilde{A})}{4}. α 0 λ m a x ( A ) ≤ 1 , c 0 β β β − 1 ≤ 20 1 = 4 λ m i n ( A ) .
These are the stability and stepsize comparisons used in Section C.2. That calculation uses only a fixed symmetric positive-definite drift, a positive-definite noise covariance, and these comparisons; it does not use the particular entries of the asynchronous matrices. The weights are equivariant under similarity, so the weights formed with A ~ \widetilde{A} A are TW_{n,t}^{\syn}T^{-1} , and their covariance is TC_n^{\syn}T^\top . Applying the expansion of Section C.2 to ( A ~ , S ~ ) (\widetilde{A},\widetilde{S}) ( A , S ) and transforming back yields
C_n^{\syn}=\Sigma_\infty^{\syn}+n^{\beta-1}G^{\syn}+o(n^{\beta-1}),\qquad G^{\syn}=\frac{1}{c_0}T^{-1}\widetilde{A}^{-1}\widetilde{\chi}\widetilde{A}^{-\top}T^{-\top}\succ0, \tag*{(226)}
where χ ~ \widetilde{\chi} χ is the unique solution of A ~ χ ~ + χ ~ A ~ ⊤ = S ~ \widetilde{A}\widetilde{\chi}+\widetilde{\chi}\widetilde{A}^{\top}=\widetilde{S} A χ + χ A ⊤ = S . Its positive definiteness follows from Lemma C.9. In particular, (G^{\syn})_{11}>0 . The one-dimensional Gaussian comparison in Lemma C.10, applied after projection onto the first coordinate, implies
W_1(Z_n^{\syn},Z_\infty^{\syn})\ge cn^{\beta-1}
for all sufficiently large n n n . The reverse triangle inequality and (S.23) now give
\begin{aligned}
W_1(X_n,Z_\infty^{\syn})\ge W_1(Z_n^{\syn},Z_\infty^{\syn})-W_1(X_n^{\lin},Z_n^{\syn}) \\
&\ge cn^{\beta-1}-C\{n^{-1/2}\log(en)+n^{-\beta}\}\ge c'n^{\beta-1},
\end{aligned}
since the subtracted terms are o ( n β − 1 ) o(n^{\beta-1}) o ( n β − 1 ) for every fixed β > 1 / 2 \beta>1/2 β > 1/2 .
At β = 1 / 2 \beta=1/2 β = 1/2 , use the mean instead. By (D.3) and Q 0 = 0 Q_0=0 Q 0 = 0 ,
\mathbb{E}X_n=-\frac{1}{\sqrt{n}}\sum_{k=1}^{n}\Phi_{0:k-1}^{\syn}Q^\star=-\frac{v}{\sqrt{n}}+o(n^{-1/2}),\qquad v=\sum_{k=1}^{\infty}\Phi_{0:k-1}^{\syn}Q^\star.
The series converges by product contraction. Every factor I-\alpha_tA^{\syn} is nonnegative with strictly positive row sums, and every coordinate of Q ⋆ Q^\star Q ⋆ is positive. Consequently v v v has strictly positive coordinates and is nonzero. The difference-of-means bound in Lemma C.10 gives
W_1(X_n,Z_\infty^{\syn})\ge\lVert\mathbb{E}X_n\rVert_2\ge cn^{-1/2}
for all sufficiently large $n. The MDP and Q 0 Q_0 Q 0 are the same for every admissible choice of stepsize parameters.
Supporting Lemmas
Lemma D.1 (Synchronous deterministic weights). The weights in (S.21) satisfy
sup n , t < n ∥ W n , t s y n ∥ ∞ ≤ C , ∥ C n s y n − Σ ∞ s y n ∥ F ≤ C n β − 1 , (227) \sup_{n,t<n}\lVert W_{n,t}^{\mathrm{syn}}\rVert_{\infty}\le C,\qquad\lVert C_{n}^{\mathrm{syn}}-\Sigma_{\infty}^{\mathrm{syn}}\rVert_{F}\le Cn^{\beta-1}, \tag*{(227)} n , t < n sup ∥ W n , t syn ∥ ∞ ≤ C , ∥ C n syn − Σ ∞ syn ∥ F ≤ C n β − 1 , (227) sup i ≥ 0 α i ∑ k > i ρ i + 1 : k − 1 ≤ C , ∑ j < i α j ρ j + 1 : i − 1 = 1 − ρ 0 : i − 1 a s y n ≤ a s y n − 1 . (228) \sup_{i\ge0}\alpha_i\sum_{k>i}\rho_{i+1:k-1}\le C,\qquad\sum_{j<i}\alpha_j\rho_{j+1:i-1}=\frac{1-\rho_{0:i-1}}{a_{\mathrm{syn}}}\le a_{\mathrm{syn}}^{-1}. \tag*{(228)} i ≥ 0 sup α i k > i ∑ ρ i + 1 : k − 1 ≤ C , j < i ∑ α j ρ j + 1 : i − 1 = a syn 1 − ρ 0 : i − 1 ≤ a syn − 1 . (228) Moreover,
∥ Φ i : j s y n ∥ ∞ ≤ ρ i : j ≤ exp ( − a s y n ∑ u = i j α u ) , ∥ ( A s y n ) − 1 ∥ ∞ ≤ a s y n − 1 . (229) \lVert\Phi_{i:j}^{\mathrm{syn}}\rVert_{\infty}\le\rho_{i:j}\le\exp\left(-a_{\mathrm{syn}}\sum_{u=i}^{j}\alpha_u\right),\qquad\lVert(A^{\mathrm{syn}})^{-1}\rVert_{\infty}\le a_{\mathrm{syn}}^{-1}. \tag*{(229)} ∥ Φ i : j syn ∥ ∞ ≤ ρ i : j ≤ exp ( − a syn u = i ∑ j α u ) , ∥( A syn ) − 1 ∥ ∞ ≤ a syn − 1 . (229)
Proof. The mean update matrices are nonnegative with row sums 1 − a s y n α t 1-a_{\mathrm{syn}}\alpha_t 1 − a syn α t , giving the product bound. The inverse bound follows from the Neumann series for I − γ P π ⋆ I-\gamma P^{\pi^\star} I − γ P π ⋆ . The proof of Lemma B.2, with D = I D=I D = I and a ⋆ = a s y n a_\star=a_{\mathrm{syn}} a ⋆ = a syn , gives
∥ W n , t s y n − ( A s y n ) − 1 ∥ ∞ ≤ C { ( t + b ) β − 1 + exp ( − a s y n 2 ∑ u = t + 1 n α u ) } , (230) \lVert W_{n,t}^{\mathrm{syn}}-(A^{\mathrm{syn}})^{-1}\rVert_{\infty}\le C\left\{(t+b)^{\beta-1}+\exp\left(-\frac{a_{\mathrm{syn}}}{2}\sum_{u=t+1}^{n}\alpha_u\right)\right\}, \tag*{(230)} ∥ W n , t syn − ( A syn ) − 1 ∥ ∞ ≤ C { ( t + b ) β − 1 + exp ( − 2 a syn u = t + 1 ∑ n α u ) } , (230)
as well as uniform boundedness of the weights. Those arguments require 0 < β < 1 0<\beta<1 0 < β < 1 and the shift comparison, and do not use ∑ t α t 2 < ∞ \sum_t\alpha_t^2<\infty ∑ t α t 2 < ∞ ; hence they also cover β = 1 / 2 \beta=1/2 β = 1/2 . For completeness, averaging the first term in (D.19) gives O ( n β − 1 ) O(n^{\beta-1}) O ( n β − 1 ) by the integral test, and the exponential term satisfies
∑ t < n exp ( − a s y n 2 ∑ u = t + 1 n α u ) ≤ ∑ k ≥ 1 e − a s y n k α n / 2 ≤ C α n − 1 ≤ C n β . \sum_{t<n}\exp\left(-\frac{a_{\mathrm{syn}}}{2}\sum_{u=t+1}^{n}\alpha_u\right)\le\sum_{k\ge1}e^{-a_{\mathrm{syn}}k\alpha_n/2}\le C\alpha_n^{-1}\le Cn^\beta. t < n ∑ exp ( − 2 a syn u = t + 1 ∑ n α u ) ≤ k ≥ 1 ∑ e − a syn k α n /2 ≤ C α n − 1 ≤ C n β .
Since the weights and ( A s y n ) − 1 (A^{\mathrm{syn}})^{-1} ( A syn ) − 1 are bounded, expanding the two covariance factors and averaging proves the covariance bound in (D.16).
To prove the response bound directly, set s i , k = ∑ u = i + 1 k − 1 α u s_{i,k}=\sum_{u=i+1}^{k-1}\alpha_u s i , k = ∑ u = i + 1 k − 1 α u . The stepsize comparison used in Lemma B.2 gives
α i / α k ≤ C exp { a s y n s i , k / 4 } , k > i . \alpha_i/\alpha_k\le C\exp\{a_{\mathrm{syn}}s_{i,k}/4\},\qquad k>i. α i / α k ≤ C exp { a syn s i , k /4 } , k > i .
Thus
α i ∑ k > i ρ i + 1 : k − 1 ≤ C ∑ k > i α k e − 3 a s y n s i , k / 4 ≤ C . \alpha_i\sum_{k>i}\rho_{i+1:k-1}\le C\sum_{k>i}\alpha_k e^{-3a_{\mathrm{syn}}s_{i,k}/4}\le C. α i k > i ∑ ρ i + 1 : k − 1 ≤ C k > i ∑ α k e − 3 a syn s i , k /4 ≤ C .
The last sum is bounded by an exponential integral, since the intervals [ s i , k , s i , k + 1 ] [s_{i,k},s_{i,k+1}] [ s i , k , s i , k + 1 ] partition [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) and have lengths α k ≤ 1 \alpha_k\le1 α k ≤ 1 . Finally, a s y n α j ρ j + 1 : i − 1 = ρ j + 1 : i − 1 − ρ j : i − 1 a_{\mathrm{syn}}\alpha_j\rho_{j+1:i-1}=\rho_{j+1:i-1}-\rho_{j:i-1} a syn α j ρ j + 1 : i − 1 = ρ j + 1 : i − 1 − ρ j : i − 1 . Summing over j < i j<i j < i proves the second identity in (D.17).
Lemma D.2 (Synchronous iterate moments). Both recursions (S.17) and (S.20) satisfy
E ∥ Q t − Q ⋆ ∥ ∞ p + E ∥ L t − Q ⋆ ∥ ∞ p ≤ C α t p / 2 , p ∈ { 2 , 4 , 6 } , t ≥ 0. (231) \mathbb{E}\lVert Q_t-Q^\star\rVert_{\infty}^{p}+\mathbb{E}\lVert L_t-Q^\star\rVert_{\infty}^{p}\le C\alpha_t^{p/2},\qquad p\in\{2,4,6\},\quad t\ge0. \tag*{(231)} E ∥ Q t − Q ⋆ ∥ ∞ p + E ∥ L t − Q ⋆ ∥ ∞ p ≤ C α t p /2 , p ∈ { 2 , 4 , 6 } , t ≥ 0. (231) The action-gap and covariance assumptions are not needed for this estimate.
Proof. Choose a fixed q ≥ 2 q\ge2 q ≥ 2 sufficiently large that θ = γ d 1 / q < 1 \theta=\gamma d^{1/q}<1 θ = γ d 1/ q < 1 . Both mean Bellman maps, optimal and fixed-policy, are θ \theta θ -contractions in ℓ q \ell_q ℓ q , since
∥ T x − T y ∥ q ≤ d 1 / q γ ∥ x − y ∥ ∞ ≤ θ ∥ x − y ∥ q , \lVert\mathcal{T}x-\mathcal{T}y\rVert_q\le d^{1/q}\gamma\lVert x-y\rVert_\infty\le\theta\lVert x-y\rVert_q, ∥ T x − T y ∥ q ≤ d 1/ q γ ∥ x − y ∥ ∞ ≤ θ ∥ x − y ∥ q ,
and the same bound holds for r + γ P π ⋆ x r + \gamma P^{\pi^\star}x r + γ P π ⋆ x . For either error sequence e t e_t e t , write its update as
e t + 1 = e ˉ t + α t ξ t , ∥ e ˉ t ∥ q ≤ [ 1 − ( 1 − θ ) α t ] ∥ e t ∥ q , E [ ξ t ∣ F t − 1 s y n ] = 0. e_{t+1} = \bar{e}_t + \alpha_t \xi_t, \qquad\lVert\bar{e}_t \rVert_q \le[1 - (1-\theta)\alpha_t]\lVert e_t \rVert_q, \qquad\mathbb{E}[\xi_t \mid\mathcal{F}^{\mathrm{syn}}_{t-1}] = 0. e t + 1 = e ˉ t + α t ξ t , ∥ e ˉ t ∥ q ≤ [ 1 − ( 1 − θ ) α t ] ∥ e t ∥ q , E [ ξ t ∣ F t − 1 syn ] = 0.
The noise is uniformly bounded by cube invariance, and e ˉ t \bar{e}_t e ˉ t is measurable with respect to the past. For p ∈ { 2 , 4 , 6 } p \in\{2,4,6\} p ∈ { 2 , 4 , 6 } , the first-order Taylor remainder for x ↦ ∥ x ∥ q p x \mapsto\lVert x\rVert_q^p x ↦ ∥ x ∥ q p is at most C ( ∥ x ∥ q p − 2 ∥ y ∥ q 2 + ∥ y ∥ q p ) C(\lVert x\rVert_q^{p-2}\lVert y\rVert_q^2 + \lVert y\rVert_q^p) C (∥ x ∥ q p − 2 ∥ y ∥ q 2 + ∥ y ∥ q p ) . This follows by integrating its second directional derivative along x + u y x+uy x + u y ; its absolute value is bounded by C ∥ x + u y ∥ q p − 2 ∥ y ∥ q 2 C\lVert x+uy\rVert_q^{p-2}\lVert y\rVert_q^2 C ∥ x + u y ∥ q p − 2 ∥ y ∥ q 2 . Taking conditional expectations removes the linear term. Consequently, with M p , t = E ∥ e t ∥ q p M_{p,t} = \mathbb{E}\lVert e_t\rVert_q^p M p , t = E ∥ e t ∥ q p and M 0 , t = 1 M_{0,t}=1 M 0 , t = 1 ,
M p , t + 1 ≤ ( 1 − c α t ) M p , t + C α t 2 M p − 2 , t + C α t p . M_{p,t+1} \le(1-c\alpha_t)M_{p,t} + C\alpha_t^2 M_{p-2,t} + C\alpha_t^p. M p , t + 1 ≤ ( 1 − c α t ) M p , t + C α t 2 M p − 2 , t + C α t p .
Inductively in p p p , the forcing terms are O ( α t p / 2 + 1 ) O(\alpha_t^{p/2+1}) O ( α t p /2 + 1 ) . Also, α t + 1 p / 2 / α t p / 2 = 1 − O ( ( t + b ) − 1 ) \alpha_{t+1}^{p/2}/\alpha_t^{p/2} = 1-O((t+b)^{-1}) α t + 1 p /2 / α t p /2 = 1 − O (( t + b ) − 1 ) and ( t + b ) − 1 = o ( α t ) (t+b)^{-1}=o(\alpha_t) ( t + b ) − 1 = o ( α t ) because β < 1 \beta<1 β < 1 . Comparing with a sufficiently large multiple of α t p / 2 \alpha_t^{p/2} α t p /2 proves M p , t ≤ C α t p / 2 M_{p,t}\le C\alpha_t^{p/2} M p , t ≤ C α t p /2 from some fixed time onward. Increasing C C C covers the finite prefix and both recursions. The maximum norm is bounded by the ℓ q \ell_q ℓ q norm, proving (D.20). The argument applies without change at β = 1 / 2 \beta=1/2 β = 1/2 .
Lemma D.3 (Resampling covariance identity). Let ω = ( ω 0 , … , ω n − 1 ) \omega=(\omega_0,\ldots,\omega_{n-1}) ω = ( ω 0 , … , ω n − 1 ) have independent coordinates and let ω ′ \omega' ω ′ be an independent copy. Use the resampling notation and weights of (D.10). For square-integrable scalar functions g , f g,f g , f ,
Cov ( g ( ω ) , f ( ω ) ) = 1 2 ∑ J ⊊ I n ϑ J ∑ j ∉ J E [ D j g ( ω ) D j f ( ω J ) ] . (232) \operatorname{Cov}(g(\omega),f(\omega)) = \frac{1}{2}\sum_{\mathcal{J}\subsetneq\mathcal{I}_n}\vartheta_{\mathcal{J}}\sum_{j\notin\mathcal{J}}\mathbb{E}[D_jg(\omega)D_jf(\omega^{\mathcal{J}})]. \tag*{(232)} Cov ( g ( ω ) , f ( ω )) = 2 1 J ⊊ I n ∑ ϑ J j ∈ / J ∑ E [ D j g ( ω ) D j f ( ω J )] . (232) For each j j j , ∑ J : j ∉ J ϑ J = 1 \sum_{\mathcal{J}:j\notin\mathcal{J}}\vartheta_{\mathcal{J}}=1 ∑ J : j ∈ / J ϑ J = 1 . The identity is Lemma 2.3 in [9 ] . We also use the variance inequality: if G G G is a square-integrable matrix-valued function of independent inputs U 1 , … , U m U_1,\ldots,U_m U 1 , … , U m , and G [ i ] G^{[i]} G [ i ] replaces input i i i by an independent copy, then
E ∥ G − E G ∥ F 2 ≤ 1 2 ∑ i = 1 m E ∥ G − G [ i ] ∥ F 2 . (233) \mathbb{E}\lVert G-\mathbb{E}G\rVert_F^2 \le\frac{1}{2}\sum_{i=1}^m\mathbb{E}\lVert G-G^{[i]}\rVert_F^2. \tag*{(233)} E ∥ G − E G ∥ F 2 ≤ 2 1 i = 1 ∑ m E ∥ G − G [ i ] ∥ F 2 . (233)
Proof. Average the telescoping replacement of all coordinates over a uniform random permutation. The probability that the coordinates before j j j form exactly J \mathcal{J} J is ∣ J ∣ ! ( n − ∣ J ∣ − 1 ) ! / n ! = ϑ J |\mathcal{J}|!(n-|\mathcal{J}|-1)!/n!=\vartheta_{\mathcal{J}} ∣ J ∣ ! ( n − ∣ J ∣ − 1 )! / n ! = ϑ J . Thus
f ( ω ) − f ( ω ′ ) = ∑ J ⊊ I n ϑ J ∑ j ∉ J D j f ( ω J ) . f(\omega)-f(\omega')=\sum_{\mathcal{J}\subsetneq\mathcal{I}_n}\vartheta_{\mathcal{J}}\sum_{j\notin\mathcal{J}}D_jf(\omega^{\mathcal{J}}). f ( ω ) − f ( ω ′ ) = J ⊊ I n ∑ ϑ J j ∈ / J ∑ D j f ( ω J ) .
Multiply by g ( ω ) g(\omega) g ( ω ) and take expectations. In each summand, interchanging ω j \omega_j ω j and ω j ′ \omega'_j ω j ′ preserves the joint law and reverses the sign of D j f ( ω J ) D_jf(\omega^{\mathcal{J}}) D j f ( ω J ) . Averaging before and after this interchange gives (D.21). The subset-weight identity follows because exactly one predecessor set occurs for each j j j in every permutation. Finally, the Doob martingale for G G G in the order U 1 , … , U m U_1,\ldots,U_m U 1 , … , U m has orthogonal differences. For each entry, the squared norm of its i i i th difference is at most E Var ( G ∣ U 1 , … , U i − 1 , U i + 1 , … , U m ) \mathbb{E}\operatorname{Var}(G\mid U_1,\ldots,U_{i-1},U_{i+1},\ldots,U_m) E Var ( G ∣ U 1 , … , U i − 1 , U i + 1 , … , U m ) by conditional Jensen. This conditional variance equals half the expected squared resampling difference. Summing over i i i and matrix entries proves (D.22).
Lemma D.4 (Gaussian smoothing and Stein equation). Let V ≻ 0 V\succ0 V ≻ 0 have eigenvalues in a fixed interval [ λ , Λ ] ⊂ ( 0 , ∞ ) [\lambda,\Lambda]\subset(0,\infty) [ λ , Λ ] ⊂ ( 0 , ∞ ) , and let Z V ∼ N ( 0 , V ) Z_V\sim\mathcal{N}(0,V) Z V ∼ N ( 0 , V ) . For every Euclidean 1-Lipschitz function h h h and 0 < δ ≤ 1 / 2 0<\delta\le1/2 0 < δ ≤ 1/2 , there are smooth functions h δ , ψ δ h_\delta,\psi_\delta h δ , ψ δ such that
tr ( V D 2 ψ δ ( x ) ) − x ⊤ ∇ ψ δ ( x ) = h δ ( x ) − E h δ ( Z V ) , (234) \operatorname{tr}(V D^2\psi_\delta(x)) - x^\top\nabla\psi_\delta(x) = h_\delta(x)-\mathbb{E}h_\delta(Z_V), \tag*{(234)} tr ( V D 2 ψ δ ( x )) − x ⊤ ∇ ψ δ ( x ) = h δ ( x ) − E h δ ( Z V ) , (234) ∥ h δ − h ∥ s u p ≤ C δ , ∥ D 2 ψ δ ∥ s u p ≤ C , ∥ D 3 ψ δ ∥ s u p ≤ C ( 1 + ∣ log δ ∣ ) . (235) \lVert h_\delta-h\rVert_{\mathrm{sup}}\le C\delta,\qquad\lVert D^2\psi_\delta\rVert_{\mathrm{sup}}\le C,\qquad\lVert D^3\psi_\delta\rVert_{\mathrm{sup}}\le C(1+|\log\delta|). \tag*{(235)} ∥ h δ − h ∥ sup ≤ C δ , ∥ D 2 ψ δ ∥ sup ≤ C , ∥ D 3 ψ δ ∥ sup ≤ C ( 1 + ∣ log δ ∣ ) . (235) The derivative norms are Euclidean operator norms, and C C C depends only on $d,\lambda,\Lambda.
Proof. Set h δ ( x ) = E h ( x + δ Z V ) h_\delta(x) = \mathbb{E}h(x + \delta Z_V) h δ ( x ) = E h ( x + δ Z V ) . The smoothing error is at most δ E ∥ Z V ∥ 2 ≤ δ d Λ \delta\mathbb{E}\lVert Z_V\rVert_2 \le\delta\sqrt{d\Lambda} δ E ∥ Z V ∥ 2 ≤ δ d Λ . For the Gaussian semigroup
( P s h δ ) ( x ) = E h δ ( e − s x + 1 − e − 2 s Z V ) , (P_s h_\delta)(x) = \mathbb{E}h_\delta\left(e^{-s}x + \sqrt{1-e^{-2s}}Z_V\right), ( P s h δ ) ( x ) = E h δ ( e − s x + 1 − e − 2 s Z V ) ,
define
ψ δ ( x ) = − ∫ 0 ∞ { P s h δ ( x ) − E h δ ( Z V ) } d s . \psi_\delta(x) = -\int_0^\infty\{P_s h_\delta(x) - \mathbb{E}h_\delta(Z_V)\}\,\mathrm{d}s. ψ δ ( x ) = − ∫ 0 ∞ { P s h δ ( x ) − E h δ ( Z V )} d s .
The integral converges by the Lipschitz property and Gaussian coupling. Its generator is tr ( V D 2 ) − x ⊤ ∇ \operatorname{tr}(V D^2)-x^\top\nabla tr ( V D 2 ) − x ⊤ ∇ , and integration of ∂ s P s h δ \partial_s P_s h_\delta ∂ s P s h δ proves (D.23). To bound derivatives, combine the two Gaussian convolutions:
P s h δ ( x ) = E h ( e − s x + 1 − e − 2 s + δ 2 Z V ) . P_s h_\delta(x) = \mathbb{E}h\left(e^{-s}x + \sqrt{1-e^{-2s}+\delta^2}Z_V\right). P s h δ ( x ) = E h ( e − s x + 1 − e − 2 s + δ 2 Z V ) .
Use the bounded first derivative of a Lipschitz function and integrate the Gaussian density by parts once and twice, respectively. This gives
∥ D 2 P s h δ ∥ s u p ≤ C e − 2 s 1 − e − 2 s + δ 2 , ∥ D 3 P s h δ ∥ s u p ≤ C e − 3 s 1 − e − 2 s + δ 2 . \lVert D^2P_s h_\delta\rVert_{\mathrm{sup}} \le\frac{C e^{-2s}}{\sqrt{1-e^{-2s}+\delta^2}}, \qquad\lVert D^3P_s h_\delta\rVert_{\mathrm{sup}} \le\frac{C e^{-3s}}{1-e^{-2s}+\delta^2}. ∥ D 2 P s h δ ∥ sup ≤ 1 − e − 2 s + δ 2 C e − 2 s , ∥ D 3 P s h δ ∥ sup ≤ 1 − e − 2 s + δ 2 C e − 3 s .
The first bound has a uniformly finite integral over s ≥ 0 s \ge0 s ≥ 0 . The integral of the second is at most C ( 1 + ∣ log δ ∣ ) C(1+|\log\delta|) C ( 1 + ∣ log δ ∣ ) , since its denominator is comparable to s + δ 2 s+\delta^2 s + δ 2 on 0 ≤ s ≤ 1 0\le s\le1 0 ≤ s ≤ 1 and its numerator decays exponentially for s ≥ 1 s\ge1 s ≥ 1 . Integrating these bounds proves (D.24). For nonsmooth h h h , the integrations by parts use its bounded weak gradient; equivalently, first smooth h h h further and pass to the limit.
Details of Experiments
The setup of the MDP and Simulation. We consider a fixed discounted MDP with state space S = { 0 , 1 , 2 , 3 } \mathcal{S}=\{0,1,2,3\} S = { 0 , 1 , 2 , 3 } , action space A = { 0 , 1 , 2 } \mathcal{A}=\{0,1,2\} A = { 0 , 1 , 2 } , and discount factor γ = 0.8 \gamma=0.8 γ = 0.8 . Let ν = ( 0.1 , 0.2 , 0.3 , 0.4 ) ⊤ \nu=(0.1,0.2,0.3,0.4)^\top ν = ( 0.1 , 0.2 , 0.3 , 0.4 ) ⊤ . The transition matrix associated with action a a a is
P a = p a I + ( 1 − p a ) 1 ν ⊤ , ( p 0 , p 1 , p 2 ) = ( 0.8 , 0.6 , 0.4 ) , P_a=p_a I+(1-p_a)1\nu^\top,\qquad(p_0,p_1,p_2)=(0.8,0.6,0.4), P a = p a I + ( 1 − p a ) 1 ν ⊤ , ( p 0 , p 1 , p 2 ) = ( 0.8 , 0.6 , 0.4 ) ,
and the deterministic reward matrix, with rows indexed by states and columns by actions, is
r = ( 0 0.4 0.1 1 0 0.2 0.1 0.2 1 0.3 0.9 0 ) . r=\begin{pmatrix}
0 & 0.4 & 0.1\\
1 & 0 & 0.2\\
0.1 & 0.2 & 1\\
0.3 & 0.9 & 0
\end{pmatrix}. r = 0 1 0.1 0.3 0.4 0 0.2 0.9 0.1 0.2 1 0 .
The behavior policy selects each action with probability 1 / 3 1/3 1/3 at every state. Its state transition matrix is P b = 0.6 I + 0.41 ν ⊤ P_b=0.6I+0.4 1\nu^\top P b = 0.6 I + 0.41 ν ⊤ , whose stationary distribution is ν \nu ν . The optimal policy is π ⋆ = ( 1 , 0 , 2 , 1 ) \pi^\star=(1,0,2,1) π ⋆ = ( 1 , 0 , 2 , 1 ) . We use the stepsize α t = c 0 ( t + b ) − β \alpha_t=c_0(t+b)^{-\beta} α t = c 0 ( t + b ) − β , β = 3 4 \beta=\frac{3}{4} β = 4 3 , b = 512 b=512 b = 512 , c 0 = 512 3 / 4 c_0=512^{3/4} c 0 = 51 2 3/4 . These choices give α 0 = 1 \alpha_0=1 α 0 = 1 and satisfy β c 0 b β − 1 ≤ ρ ( 1 − γ ) 4 \frac{\beta}{c_0}b^{\beta-1}\le\frac{\rho(1-\gamma)}{4} c 0 β b β − 1 ≤ 4 ρ ( 1 − γ ) . We simulate R = 32,768 R=32{,}768 R = 32 , 768 independent trajectories through n max = 10 8 n_{\max}=10^8 n m a x = 1 0 8 .
Measuring Gaussian approximation. We represent Q-functions as twelve-dimensional vectors. Recall that D = diag ( μ ) D=\operatorname{diag}(\mu) D = diag ( μ ) , A = D ( I − γ P π ⋆ ) A=D(I-\gamma P^{\pi^\star}) A = D ( I − γ P π ⋆ ) , P ( s , a ) , ( s ′ , a ′ ) π ⋆ = P ( s ′ ∣ s , a ) 1 { a ′ = π ⋆ ( s ′ ) } P^{\pi^\star}_{(s,a),(s',a')}=P(s'\mid s,a)1\{a'=\pi^\star(s')\} P ( s , a ) , ( s ′ , a ′ ) π ⋆ = P ( s ′ ∣ s , a ) 1 { a ′ = π ⋆ ( s ′ )} , S ( s , a ) , ( s , a ) = μ ( s , a ) γ 2 Var S ′ ∼ P ( ⋅ ∣ s , a ) [ V ⋆ ( S ′ ) ] S_{(s,a),(s,a)}=\mu(s,a)\gamma^2\operatorname{Var}_{S'\sim P(\cdot\mid s,a)}[V^\star(S')] S ( s , a ) , ( s , a ) = μ ( s , a ) γ 2 Var S ′ ∼ P ( ⋅ ∣ s , a ) [ V ⋆ ( S ′ )] , V ⋆ ( s ) = max a ∈ A Q ⋆ ( s , a ) V^\star(s)=\max_{a\in\mathcal{A}}Q^\star(s,a) V ⋆ ( s ) = max a ∈ A Q ⋆ ( s , a ) , and Σ ∞ = A − 1 S A − ⊤ \Sigma_\infty=A^{-1}SA^{-\top} Σ ∞ = A − 1 S A − ⊤ . To measure Gaussian approximation, we consider a fixed one-dimensional projection u = v / ∥ v ∥ 2 u=v/\lVert v\rVert_2 u = v / ∥ v ∥ 2 , where v v v is the first-order stationary mean-bias coefficient of the locally linearized Markov recursion. We obtain v v v by solving a finite system of 48 scalar equations for the joint first moments of the state and the twelve-dimensional error. The projection is therefore determined entirely by the MDP and is fixed before simulation. At each horizon n n n , we estimate the projected Wasserstein distance by
d ^ n = W 1 ( 1 R ∑ j = 1 R δ u ⊤ X n ( j ) , N ( 0 , u ⊤ Σ ∞ u ) ) , \widehat{d}_n = W_1\left(\frac{1}{R}\sum_{j=1}^{R}\delta_{u^\top X_n^{(j)}},\mathcal{N}(0,u^\top\Sigma_\infty u)\right), d n = W 1 ( R 1 j = 1 ∑ R δ u ⊤ X n ( j ) , N ( 0 , u ⊤ Σ ∞ u ) ) ,
where the target Gaussian has standard deviation approximately 5.22327. Figure 1 (a) compares d ^ n \widehat{d}_n d n with the benchmark n − 1 / 4 B n^{-1/4}B n − 1/4 B , where
B = c 0 1 − β ∣ u ⊤ v ∣ ≈ 73.13670. B = \frac{c_0}{1-\beta}\lvert u^\top v\rvert\approx73.13670. B = 1 − β c 0 ∣ u ⊤ v ∣ ≈ 73.13670.
This coefficient is computed from the MDP and stepsize parameters before simulation.
Covariance comparison. To examine the covariance error in our Gaussian approximation analysis, we consider the linear noise recursion
ℓ 0 = 0 , ℓ t + 1 = ( I − α t A ) ℓ t + α t ϵ t , M n = 1 n ∑ k = 1 n ℓ k , \ell_0 = 0,\qquad\ell_{t+1} = (I-\alpha_t A)\ell_t+\alpha_t\epsilon_t,\qquad M_n = \frac{1}{\sqrt{n}}\sum_{k=1}^{n}\ell_k, ℓ 0 = 0 , ℓ t + 1 = ( I − α t A ) ℓ t + α t ϵ t , M n = n 1 k = 1 ∑ n ℓ k ,
where ϵ t \epsilon_t ϵ t is the Bellman innovation at Q ⋆ Q^\star Q ⋆ along the stationary behavior trajectory. These innovations form a martingale difference sequence with unconditional covariance S S S . We compute C n = Cov ( M n ) C_n=\operatorname{Cov}(M_n) C n = Cov ( M n ) using a deterministic finite-step covariance recursion, without Monte Carlo approximation. This allows us to directly examine the covariance error ∥ C n − Σ ∞ ∥ F \lVert C_n-\Sigma_\infty\rVert_F ∥ C n − Σ ∞ ∥ F appearing in the Gaussian approximation decomposition in § 3.2 . To obtain the first-order covariance benchmark, we solve the Lyapunov equation A H + H A ⊤ = S AH+HA^\top=S A H + H A ⊤ = S and define
G = c 0 − 1 A − 1 H A − ⊤ , ∥ G ∥ F ≈ 24.44716. G = c_0^{-1}A^{-1}HA^{-\top},\qquad\lVert G\rVert_F\approx24.44716. G = c 0 − 1 A − 1 H A − ⊤ , ∥ G ∥ F ≈ 24.44716.
Figure 1 (b) compares the covariance error ∥ C n − Σ ∞ ∥ F \lVert C_n-\Sigma_\infty\rVert_F ∥ C n − Σ ∞ ∥ F with the reference curve n − 1 / 4 ∥ G ∥ F n^{-1/4}\lVert G\rVert_F n − 1/4 ∥ G ∥ F , illustrating the covariance error due to finite-sample approximation.