Introduction
We study online statistical inference for a local minimizer of stochastic optimization problem on Riemannian manifold. Let ξ ∼ D \xi\sim\mathcal{D} ξ ∼ D be a random data sample, and let f ( x , ξ ) f(x,\xi) f ( x , ξ ) be the corresponding stochastic loss. We consider
min x ∈ M F ( x ) : = E ξ ∼ D [ f ( x , ξ ) ] , (1) \min_{x \in\mathcal{M}} F(x) := \mathbb{E}_{\xi\sim\mathcal{D}}[f(x,\xi)], \tag*{(1)} x ∈ M min F ( x ) := E ξ ∼ D [ f ( x , ξ )] , (1)
where the parameter of interest x ⋆ x_\star x ⋆ is a local minimizer of the population objective F F F on the manifold M \mathcal{M} M . Many problems in statistics, machine learning, and signal processing can be modeled on manifolds, such as principal component analysis with orthogonality constraints [15 ] , dictionary learning on the sphere [37 ] , low-rank matrix completion [10 ] and low multilinear rank tensor approximation [22 , 42 ] .
The classical theory for stochastic approximation is often restricted to Euclidean space. In this setting (M = R d \mathcal{M} = \mathbb{R}^{d} M = R d ), stochastic gradient descent (SGD) is one of the most standard methods for solving (1) , which replaces the population gradient ∇ F ( x ) \nabla F(x) ∇ F ( x ) with an unbiased estimate ∇ f ( x , ξ ) \nabla f(x,\xi) ∇ f ( x , ξ ) . In particular, given independent observations ξ n ∼ D \xi_n \sim\mathcal{D} ξ n ∼ D , an initial point x 0 x_0 x 0 , and step sizes γ n > 0 \gamma_n > 0 γ n > 0 , SGD performs the update
x n = x n − 1 − γ n ∇ f ( x n − 1 , ξ n ) . (2) x_n = x_{n-1} - \gamma_n \nabla f(x_{n-1}, \xi_n). \tag*{(2)} x n = x n − 1 − γ n ∇ f ( x n − 1 , ξ n ) . (2)
SGD is widely used for large-scale parameter estimation and online learning because of its low computational and memory cost [8 ] . Let x ˉ n = n − 1 ∑ i = 1 n x i \bar{x}_n = n^{-1}\sum_{i=1}^{n} x_i x ˉ n = n − 1 ∑ i = 1 n x i denote the averaged SGD iterate. Ruppert [35 ] and Polyak and Juditsky [33 ] showed that this averaged iterate is asymptotically normal and attains the optimal covariance in the local asymptotic minimax sense of Hájek and Le Cam [20 , 27 ] . Specifically, consider step sizes γ n = γ n − α \gamma_n = \gamma n^{-\alpha} γ n = γ n − α with γ > 0 \gamma> 0 γ > 0 and α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2, 1) α ∈ ( 1/2 , 1 ) . Under suitable conditions,
n ( x ˉ n − x ⋆ ) ⟶ d N ( 0 , Σ ) , Σ = A − 1 S A − 1 , (3) \sqrt{n}(\bar{x}_n - x_\star) \mathrel{\mathop{\longrightarrow}^{\mathrm{d}}} \mathcal{N}(0,\Sigma), \qquad\Sigma= A^{-1} S A^{-1}, \tag*{(3)} n ( x ˉ n − x ⋆ ) ⟶ d N ( 0 , Σ ) , Σ = A − 1 S A − 1 , (3)
where
A : = ∇ 2 F ( x ⋆ ) and S : = E [ ∇ f ( x ⋆ , ξ ) ∇ f ( x ⋆ , ξ ) T ] A := \nabla^2 F(x_\star) \qquad\text{and} \qquad S := \mathbb{E}\left[\nabla f(x_\star,\xi)\nabla f(x_\star,\xi)^\mathsf{T}\right] A := ∇ 2 F ( x ⋆ ) and S := E [ ∇ f ( x ⋆ , ξ ) ∇ f ( x ⋆ , ξ ) T ]
denote the Hessian of the population objective at x ⋆ x_\star x ⋆ and the covariance of the stochastic gradient noise, respectively. This result characterizes the estimation error at the n − 1 / 2 n^{-1/2} n − 1/2 scale and provides the foundation for statistical inference.
To construct valid confidence intervals based on (3) , one needs to estimate the asymptotic covariance Σ \Sigma Σ . Chen et al. [11 ] proposed two consistent estimators: a plug-in estimator that requires Hessian information and a batch-means estimator that uses only SGD iterates. Zhu et al. [46 ] further developed overlapping (OL) and non-overlapping (NOL) variants of the batch-means estimator that can be updated without prior knowledge of the terminal sample size, thereby enabling fully online covariance estimation. The key idea is to divide the iterates into blocks of increasing size, with each block providing an approximately independent estimate of the covariance matrix. Specifically, let B j B_j B j denote a batch ending at iteration j j j , with length ℓ j = ∣ B j ∣ \ell_j = |B_j| ℓ j = ∣ B j ∣ and average x ˉ j B = ℓ j − 1 ∑ x ∈ B j x \bar{x}_j^B = \ell_j^{-1}\sum_{x\in B_j} x x ˉ j B = ℓ j − 1 ∑ x ∈ B j x , and let E n E_n E n index the batches used at iteration n n n . Both the OL and NOL estimators take the form
Σ ^ n = ∑ j ∈ E n ℓ j 2 ( x ˉ j B − x ˉ n ) ( x ˉ j B − x ˉ n ) T ∑ j ∈ E n ℓ j . \widehat{\Sigma}_n = \frac{\sum_{j\in E_n} \ell_j^2(\bar{x}_j^B-\bar{x}_n)(\bar{x}_j^B-\bar{x}_n)^\mathsf{T}}{\sum_{j\in E_n}\ell_j}. Σ n = ∑ j ∈ E n ℓ j ∑ j ∈ E n ℓ j 2 ( x ˉ j B − x ˉ n ) ( x ˉ j B − x ˉ n ) T .
With a suitable choice of batch size sequence (see § 3.3 ), Zhu et al. [46 ] proved that, under suitable conditions, both estimators satisfy
E ∥ Σ ^ n − Σ ∥ ≲ n − ( 1 − α ) / 4 . \mathbb{E}\lVert\widehat{\Sigma}_n-\Sigma\rVert\lesssim n^{-(1-\alpha)/4}. E ∥ Σ n − Σ ∥ ≲ n − ( 1 − α ) /4 .
with the convergence rate approaching n − 1 / 8 n^{-1/8} n − 1/8 as α → 1 / 2 \alpha\to1/2 α → 1/2 .
Despite these advances in online statistical inference in Euclidean space, stochastic approximation on a general Riemannian manifold is more complicated because the Euclidean SGD update in (2) does not, in general, preserve the manifold constraint. At a point x ∈ M x \in\mathcal{M} x ∈ M , the Riemannian gradient grad f ( x , ξ ) \operatorname{grad} f(x,\xi) grad f ( x , ξ ) lies in the tangent space T x M T_x\mathcal{M} T x M . Riemannian stochastic gradient descent (RSGD) therefore uses a retraction R x R_x R x to map a tangent vector, such as a step along the negative Riemannian gradient, back onto the manifold. Precisely, the RSGD update takes the form
x n = R x n − 1 ( − γ n grad f ( x n − 1 , ξ n ) ) . (4) x_n = R_{x_{n-1}}\left(-\gamma_n \operatorname{grad} f(x_{n-1},\xi_n)\right). \tag*{(4)} x n = R x n − 1 ( − γ n grad f ( x n − 1 , ξ n ) ) . (4)
Bonnabel [7 ] established the convergence of RSGD, while, more recently, Tripuraneni et al. [38 ] extended iterate averaging from SGD to RSGD for statistical inference. The authors used the retraction and its local inverse to recursively update the average as each new iterate becomes available, yielding
x ~ n = R x ~ n − 1 ( n − 1 R x ~ n − 1 − 1 ( x n ) ) . (5) \widetilde{x}_n = R_{\widetilde{x}_{n-1}}\left(n^{-1}R_{\widetilde{x}_{n-1}}^{-1}(x_n)\right). \tag*{(5)} x n = R x n − 1 ( n − 1 R x n − 1 − 1 ( x n ) ) . (5)
The average thus moves toward the new iterate with weight 1 / n 1/n 1/ n . For example, when M = R d \mathcal{M} = \mathbb{R}^d M = R d , we can take R x ( v ) = x + v R_x(v) = x + v R x ( v ) = x + v and R x − 1 ( y ) = y − x R_x^{-1}(y) = y - x R x − 1 ( y ) = y − x . Then, (4) reduces to the standard SGD update x n = x n − 1 − γ n ∇ f ( x n − 1 , ξ n ) x_n = x_{n-1} - \gamma_n \nabla f(x_{n-1}, \xi_n) x n = x n − 1 − γ n ∇ f ( x n − 1 , ξ n ) , and (5) reduces to the ordinary recursive average x ~ n = x ~ n − 1 + n − 1 ( x n − x ~ n − 1 ) \widetilde{x}_n = \widetilde{x}_{n-1} + n^{-1}(x_n - \widetilde{x}_{n-1}) x n = x n − 1 + n − 1 ( x n − x n − 1 ) . Section 2 2 2 provides the geometric definitions and assumptions used throughout this paper.
Under suitable local smoothness and stability assumptions, Tripuraneni et al. [38 ] proved asymptotic normality of the average x ~ n \widetilde{x}_n x n in (5) . Mapping the averaged point back to the tangent space T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M via R x ⋆ − 1 R_{x_\star}^{-1} R x ⋆ − 1 and expressing the resulting tangent vector in an orthonormal basis of the tangent space gives
n R x ⋆ − 1 ( x ~ n ) ⟶ d N ( 0 , Σ ) , Σ = A − 1 S A − 1 . \sqrt{n}R_{x_\star}^{-1}(\widetilde{x}_n) \mathrel{\mathop{\longrightarrow}\limits^{d}} \mathcal{N}(0,\Sigma), \qquad\Sigma= A^{-1}SA^{-1}. n R x ⋆ − 1 ( x n ) ⟶ d N ( 0 , Σ ) , Σ = A − 1 S A − 1 .
Here A = Hess F ( x ⋆ ) A = \operatorname{Hess} F(x_\star) A = Hess F ( x ⋆ ) is the Riemannian Hessian and S S S is the covariance of grad f ( x ⋆ , ξ ) \operatorname{grad} f(x_\star,\xi) grad f ( x ⋆ , ξ ) ; both act on T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M . This asymptotic behavior of RSGD provides a basis for statistical inference on Riemannian manifolds. Motivated by the design of batch-means covariance estimator in Zhu et al. [46 ] , we raise the following question:
Can the batch-means covariance estimator be adapted to the averaged RSGD, and can the resulting estimator retain the same convergence rate as in the Euclidean setting?
In this paper, we provide an affirmative answer to the above question. In particular, we use retraction maps and vector transports to extend the online OL and NOL batch-means constructions of Zhu et al. [46 ] to averaged RSGD. To this end, we compute for each batch B j B_j B j its iterate average c j c_j c j using the retraction-based averaging rule in (5) , and construct the covariance estimator from the vectors b j , n = R x ~ n − 1 ( c j ) b_{j,n} = R_{\widetilde{x}_n}^{-1}(c_j) b j , n = R x n − 1 ( c j ) , which represent the batch averages in the tangent space at the overall average x ~ n \widetilde{x}_n x n . This extension presents two challenges. (i) As the overall average is updated, the tangent space containing the vectors { b j , n } j \{b_{j,n}\}_j { b j , n } j changes from T x ~ n M T_{\widetilde{x}_n}\mathcal{M} T x n M to T x ~ n + 1 M T_{\widetilde{x}_{n+1}}\mathcal{M} T x n + 1 M . Thus, computing each b j , n + 1 = R x ~ n + 1 − 1 ( c j ) b_{j,n+1} = R_{\widetilde{x}_{n+1}}^{-1}(c_j) b j , n + 1 = R x n + 1 − 1 ( c j ) directly would require revisiting all previous batch averages and contradict the online spirit. To obtain an online update, we approximate b j , n b_{j,n} b j , n by b ~ j , n \widetilde{b}_{j,n} b j , n (see (13) ) and update these approximations recursively by accounting for the change in the overall average and applying vector transport to move the resulting vectors to the new tangent space. This avoids repeatedly recomputing the inverse retractions of previous batch averages. (ii) These approximate updates introduce errors that may accumulate over time. To control them, we separate the Euclidean batch-means error from the additional errors induced by the manifold geometry. Under standard conditions (cf. Theorem 3.1 ), we show that these geometric errors are of higher order than the leading Euclidean error and therefore do not affect the overall convergence rate. Altogether, we establish that the convergence rate of covariance estimation on manifolds matches that in Euclidean space.
Main contributions. The main contributions of the paper are threefold. First, we construct an online batch-means covariance estimator Σ ^ n \widehat{\Sigma}_n Σ n for averaged RSGD, with OL and NOL versions. To our knowledge, this is the first estimator of the limiting covariance of averaged RSGD. Our estimator can be updated online and requires neither Riemannian Hessian computations nor a prespecified terminal sample size. Second, we establish the convergence rate for our estimator. Under the conditions of Theorem 3.1 , choosing the batch exponent β = 2 / ( 1 − α ) \beta= 2/(1-\alpha) β = 2/ ( 1 − α ) yields the bound
E [ ∥ ( Γ x ⋆ x ~ n ) − 1 ∘ Σ ^ n ∘ Γ x ⋆ x ~ n − Σ ∥ ] ≲ n − ( 1 − α ) / 4 . \mathbb{E}\left[\left\|(\Gamma_{x_\star}^{\widetilde{x}_n})^{-1} \circ\widehat{\Sigma}_n \circ\Gamma_{x_\star}^{\widetilde{x}_n} - \Sigma\right\|\right] \lesssim n^{-(1-\alpha)/4}. E [ ( Γ x ⋆ x n ) − 1 ∘ Σ n ∘ Γ x ⋆ x n − Σ ] ≲ n − ( 1 − α ) /4 .
Here, Γ x x ~ n \Gamma_{x}^{\widetilde{x}_n} Γ x x n denotes the parallel transport from T x M T_x\mathcal{M} T x M to T x ~ n M T_{\widetilde{x}_n}\mathcal{M} T x n M . This rate matches the corresponding rate of Zhu et al. [46 ] in the Euclidean setting and approaches n − 1 / 8 n^{-1/8} n − 1/8 as α → 1 / 2 \alpha\to1/2 α → 1/2 . Finally, we combine covariance consistency with asymptotic normality of averaged RSGD to develop online statistical inference on manifolds. In particular, we construct asymptotically valid confidence intervals for smooth scalar functions, as well as joint confidence regions.
Related work
Inference for stochastic approximation. The asymptotic theory of stochastic approximation in the Euclidean setting has led to extensive research on online statistical inference. For SGD, alternatives to the batch-means covariance estimators discussed above include online bootstrap and random scaling ([17 , 28 ] ). Subsequent work has reduced the computational cost of online bootstrap by using fewer perturbed SGD runs ([26 ] ) and extended random scaling to weighted averages of the iterates ([41 ] ).
Statistical inference under constraints has also received growing attention. Asymptotic analyses have examined how constraint geometry affects the statistical efficiency of stochastic optimization methods ([14 ] ). Stochastic sequential quadratic programming (SSQP) methods have been developed for problems with nonlinear equality and inequality constraints ((Na et al., 2023a,b; Fang et al., 2024)). Building on these designs, recent work has established asymptotic normality of SSQP iterates and developed covariance estimation and random-scaling procedures for online inference ([30 , 19 , 29 ] (Du et al. 2025)). Related developments for nonsmooth stochastic approximation include asymptotic normality and online batch-means covariance estimation ([12 , 24 ] ). Although manifold optimization can often be formulated as constrained optimization, our analysis focuses on the averaged iterate generated intrinsically by RSGD and on a covariance estimator evolving across the changing tangent spaces at this average.
Riemannian optimization. Riemannian optimization has received significant attention in the machine learning community. Comprehensive treatments can be found in Udrişte [39 ] , Absil et al. [3 ] , Boumal [9 ] . In the deterministic setting, convergence guarantees have been established for gradient, trust-region, and quasi-Newton methods ([2 , 34 ] (Zhang and Sra 2016)). For stochastic objectives, Bonnabel [7 ] established convergence guarantees for RSGD under suitable conditions. Stochastic variance reduction methods have also been developed for finite-sum objectives ([44 , 36 ] ). As discussed above, Tripuraneni et al. [38 ] extended iterate averaging to RSGD and established asymptotic normality of the averaged iterate. This result provides the theoretical foundation for our study of online covariance estimation for averaged RSGD.
Paper organization
§ 2 reviews preliminaries on Riemannian manifolds, introduces the assumptions used throughout the paper, and recalls the asymptotic normality result for averaged RSGD. § 3 constructs the OL and NOL online batch-means covariance estimators, establishes their convergence rates, and outlines the main proof sketch. § 4 develops confidence intervals and joint confidence regions on manifolds based on these results, using streaming principal component analysis on Grassmann manifolds as an illustrative example. § 5 evaluates covariance estimation and statistical inference through numerical experiments. § 6 concludes the paper and discusses directions for future work.
Notation
We write N = { 1 , 2 , … } \mathbb{N}=\{1,2,\ldots\} N = { 1 , 2 , … } for the positive integers, ∣ B ∣ |B| ∣ B ∣ for the cardinality of a finite set B B B , and ⌊ t ⌋ \lfloor t\rfloor ⌊ t ⌋ for the floor of t ∈ R t\in\mathbb{R} t ∈ R . We use I I I to denote the identity map or matrix, with a space or dimension subscript when needed, and e i e_i e i to denote the i i i th canonical basis vector. In a fixed real inner product space, ⟨ u , v ⟩ \langle u,v\rangle ⟨ u , v ⟩ denotes the inner product and ∥ v ∥ = ⟨ v , v ⟩ 1 / 2 \|v\|=\langle v,v\rangle^{1/2} ∥ v ∥ = ⟨ v , v ⟩ 1/2 its induced norm. The d d d -dimensional unit sphere is S d = { x ∈ R d + 1 : ∥ x ∥ = 1 } \mathbb{S}^{d}=\{x\in\mathbb{R}^{d+1}:\|x\|=1\} S d = { x ∈ R d + 1 : ∥ x ∥ = 1 } , and v ⊥ = { w : ⟨ v , w ⟩ = 0 } v^\perp=\{w:\langle v,w\rangle=0\} v ⊥ = { w : ⟨ v , w ⟩ = 0 } denotes the orthogonal complement of the span of v v v . For a linear map L : V → W L:V\to W L : V → W between finite-dimensional real inner product spaces, ∥ L ∥ = sup ∥ v ∥ = 1 ∥ L v ∥ \|L\|=\sup_{\|v\|=1}\|Lv\| ∥ L ∥ = sup ∥ v ∥ = 1 ∥ Lv ∥ is its operator norm. Its adjoint L ∗ : W → V L^*:W\to V L ∗ : W → V is defined by ⟨ L v , w ⟩ W = ⟨ v , L ∗ w ⟩ V \langle Lv,w\rangle_W=\langle v,L^*w\rangle_V ⟨ Lv , w ⟩ W = ⟨ v , L ∗ w ⟩ V for all v ∈ V v\in V v ∈ V and w ∈ W w\in W w ∈ W . For invertible L L L , L ∗ = L − 1 L^*=L^{-1} L ∗ = L − 1 if and only if L L L preserves inner products. We define the outer product by ( u ⊗ v ) w = ⟨ v , w ⟩ u (u\otimes v)w=\langle v,w\rangle u ( u ⊗ v ) w = ⟨ v , w ⟩ u , with u ⊗ 2 = u ⊗ u u^{\otimes2}=u\otimes u u ⊗ 2 = u ⊗ u . Thus, ( L u ) ⊗ 2 = L ∘ u ⊗ 2 ∘ L ∗ (Lu)^{\otimes2}=L\circ u^{\otimes2}\circ L^* ( Lu ) ⊗ 2 = L ∘ u ⊗ 2 ∘ L ∗ . In orthonormal bases, u ⊗ v u\otimes v u ⊗ v is represented by u v ⊤ uv^\top u v ⊤ , and L ∗ L^* L ∗ by the transpose of the matrix representing L L L .
For a n , b n ≥ 0 a_n,b_n\geq0 a n , b n ≥ 0 , a n ≲ b n a_n\lesssim b_n a n ≲ b n or a n = O ( b n ) a_n=O(b_n) a n = O ( b n ) means a n ≤ C b n a_n\leq Cb_n a n ≤ C b n for some constant C > 0 C>0 C > 0 and all sufficiently large n n n . We write a n = o ( b n ) a_n=o(b_n) a n = o ( b n ) if a n / b n → 0 a_n/b_n\to0 a n / b n → 0 , and a n ≍ b n a_n\asymp b_n a n ≍ b n if both a n ≲ b n a_n\lesssim b_n a n ≲ b n and b n ≲ a n b_n\lesssim a_n b n ≲ a n . For notational simplicity, we use C C C to denote generic constants that may take different values in different expressions. We write E n ( ⋅ ) = E ( ⋅ ∣ F n ) \mathbb{E}_n(\cdot)=\mathbb{E}(\cdot\mid\mathcal{F}_n) E n ( ⋅ ) = E ( ⋅ ∣ F n ) for the conditional expectation, where F n = σ ( ξ 1 , … , ξ n ) \mathcal{F}_n=\sigma(\xi_1,\ldots,\xi_n) F n = σ ( ξ 1 , … , ξ n ) is the σ \sigma σ -algebra generated by ξ 1 , … , ξ n \xi_1,\ldots,\xi_n ξ 1 , … , ξ n . The symbol z 1 − δ / 2 z_{1-\delta/2} z 1 − δ /2 denotes the ( 1 − δ / 2 ) (1-\delta/2) ( 1 − δ /2 ) -quantile of the standard normal distribution. Similarly, χ d , 1 − δ 2 \chi^2_{d,1-\delta} χ d , 1 − δ 2 denotes the ( 1 − δ ) (1-\delta) ( 1 − δ ) -quantile of the chi-squared distribution with d d d degrees of freedom.
Problem formulation and preliminaries
We first introduce the geometric operations required for averaged RSGD and covariance estimation. We then state the assumptions used throughout the paper and recall the asymptotic normality result for averaged RSGD.
Riemannian geometry
We briefly review the geometric notions related to Riemannian manifold, while we point the reader to Absil et al. [3 ] and Boumal [9 ] for a more detailed review.
Riemannian manifolds and smooth maps. Throughout the paper, ( M , g ) (\mathcal{M},g) ( M , g ) denotes a connected, d d d -dimensional Riemannian manifold, that is, a smooth manifold M \mathcal{M} M equipped with a Riemannian metric g g g . For each x ∈ M x\in\mathcal{M} x ∈ M , we use T x M T_x\mathcal{M} T x M to denote the tangent space at x x x . The metric g g g smoothly assigns an inner product ⟨ u , v ⟩ x : = g x ( u , v ) \langle u,v\rangle_x:=g_x(u,v) ⟨ u , v ⟩ x := g x ( u , v ) to each tangent space T x M T_x\mathcal{M} T x M , with induced norm ∥ v ∥ x : = ⟨ v , v ⟩ x 1 / 2 \|v\|_x:=\langle v,v\rangle_x^{1/2} ∥ v ∥ x := ⟨ v , v ⟩ x 1/2 . We write T M : = ⨆ x ∈ M T x M T\mathcal{M}:=\bigsqcup_{x\in\mathcal{M}}T_x\mathcal{M} T M := ⨆ x ∈ M T x M for the tangent bundle and 0 x 0_x 0 x for the zero vector in T x M T_x\mathcal{M} T x M . A map Φ : M → M ′ \Phi:\mathcal{M}\to\mathcal{M}' Φ : M → M ′ between smooth manifolds is smooth if its representation in local coordinates is smooth. A smooth curve on M \mathcal{M} M is a smooth map c : [ 0 , 1 ] → M c:[0,1]\to\mathcal{M} c : [ 0 , 1 ] → M . Its velocity at t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] is denoted by c ˙ ( t ) : = d d t c ( t ) ∈ T c ( t ) M \dot{c}(t):=\frac{d}{dt}c(t)\in T_{c(t)}\mathcal{M} c ˙ ( t ) := d t d c ( t ) ∈ T c ( t ) M , where d / d t d/dt d / d t denotes ordinary differentiation in local coordinates. The length of c c c is defined by L ( c ) : = ∫ 0 1 ∥ c ˙ ( t ) ∥ c ( t ) d t L(c):=\int_0^1\|\dot{c}(t)\|_{c(t)}\,dt L ( c ) := ∫ 0 1 ∥ c ˙ ( t ) ∥ c ( t ) d t . The Riemannian distance d ( x , y ) d(x,y) d ( x , y ) is the infimum of L ( c ) L(c) L ( c ) over all piecewise smooth curves c c c joining x x x and y y y .
Differential calculus and connections. For a smooth map Φ : M → M ′ \Phi:\mathcal{M}\to\mathcal{M}' Φ : M → M ′ , the differential D Φ ( x ) : T x M → T Φ ( x ) M ′ D\Phi(x):T_x\mathcal{M}\to T_{\Phi(x)}\mathcal{M}' D Φ ( x ) : T x M → T Φ ( x ) M ′ is the linear map characterized by D Φ ( x ) [ v ] = d d t Φ ( c ( t ) ) ∣ t = 0 D\Phi(x)[v]=\left.\frac{d}{dt}\Phi(c(t))\right|_{t=0} D Φ ( x ) [ v ] = d t d Φ ( c ( t )) t = 0 , for any v ∈ T x M v\in T_x\mathcal{M} v ∈ T x M and any smooth curve c : [ 0 , 1 ] → M c:[0,1]\to\mathcal{M} c : [ 0 , 1 ] → M satisfying c ( 0 ) = x c(0)=x c ( 0 ) = x and c ˙ ( 0 ) = v \dot{c}(0)=v c ˙ ( 0 ) = v . A connection specifies how vector fields are differentiated on a manifold. We use ∇ \nabla ∇ to denote the Levi-Civita connection, the unique torsion-free connection compatible with g g g . In particular, for a smooth vector field Y : M → T M Y:\mathcal{M}\to T\mathcal{M} Y : M → T M with Y ( x ) ∈ T x M Y(x) \in T_x\mathcal{M} Y ( x ) ∈ T x M , ∇ v Y ∈ T x M \nabla_vY \in T_x\mathcal{M} ∇ v Y ∈ T x M with v ∈ T x M v \in T_x\mathcal{M} v ∈ T x M denotes its covariant derivative at x x x in the direction v v v . Along a smooth curve c : [ 0 , 1 ] → M c : [0,1] \to\mathcal{M} c : [ 0 , 1 ] → M , the induced covariant derivative of a smooth field V : [ 0 , 1 ] → T M V : [0,1] \to T\mathcal{M} V : [ 0 , 1 ] → T M with V ( t ) ∈ T c ( t ) M V(t) \in T_{c(t)}\mathcal{M} V ( t ) ∈ T c ( t ) M is denoted by D d t V ( t ) : = ∇ c ˙ ( t ) V ∈ T c ( t ) M \frac{D}{dt}V(t) := \nabla_{\dot{c}(t)}V \in T_{c(t)}\mathcal{M} d t D V ( t ) := ∇ c ˙ ( t ) V ∈ T c ( t ) M . For a smooth function h : M → R h : \mathcal{M} \to\mathbb{R} h : M → R , its Riemannian gradient grad h ( x ) ∈ T x M \operatorname{grad} h(x) \in T_x\mathcal{M} grad h ( x ) ∈ T x M is the unique vector satisfying the first relation below, while its Riemannian Hessian Hess h ( x ) : T x M → T x M \operatorname{Hess} h(x) : T_x\mathcal{M} \to T_x\mathcal{M} Hess h ( x ) : T x M → T x M is the linear map defined by the second:
⟨ grad h ( x ) , v ⟩ x = D h ( x ) [ v ] , Hess h ( x ) [ v ] : = ∇ v grad h , for all v ∈ T x M . \langle\operatorname{grad} h(x),v\rangle_x = Dh(x)[v], \qquad\operatorname{Hess} h(x)[v] := \nabla_v\operatorname{grad} h, \qquad\text{for all } v \in T_x\mathcal{M}. ⟨ grad h ( x ) , v ⟩ x = D h ( x ) [ v ] , Hess h ( x ) [ v ] := ∇ v grad h , for all v ∈ T x M .
The Riemannian Hessian Hess h ( x ) \operatorname{Hess} h(x) Hess h ( x ) is self-adjoint with respect to the Riemannian metric g g g ; that is,
⟨ Hess h ( x ) [ u ] , v ⟩ x = ⟨ u , Hess h ( x ) [ v ] ⟩ x , for all u , v ∈ T x M . \langle\operatorname{Hess} h(x)[u],v\rangle_x = \langle u,\operatorname{Hess} h(x)[v]\rangle_x, \qquad\text{for all } u,v \in T_x\mathcal{M}. ⟨ Hess h ( x ) [ u ] , v ⟩ x = ⟨ u , Hess h ( x ) [ v ] ⟩ x , for all u , v ∈ T x M .
Exponential maps and retractions. To turn a tangent vector into a step on the manifold, we can follow a geodesic. A smooth curve γ : [ 0 , 1 ] → M \gamma: [0,1] \to\mathcal{M} γ : [ 0 , 1 ] → M is a geodesic if D d t γ ˙ ( t ) = 0 γ ( t ) \frac{D}{dt}\dot{\gamma}(t) = 0_{\gamma(t)} d t D γ ˙ ( t ) = 0 γ ( t ) for all t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] . Given x ∈ M x \in\mathcal{M} x ∈ M and v ∈ T x M v \in T_x\mathcal{M} v ∈ T x M , let γ x , v : [ 0 , 1 ] → M \gamma_{x,v} : [0,1] \to\mathcal{M} γ x , v : [ 0 , 1 ] → M denote the unique geodesic satisfying γ x , v ( 0 ) = x \gamma_{x,v}(0) = x γ x , v ( 0 ) = x and γ ˙ x , v ( 0 ) = v \dot{\gamma}_{x,v}(0) = v γ ˙ x , v ( 0 ) = v , whenever it exists on the entire interval. If γ x , v \gamma_{x,v} γ x , v exists at time one, the exponential map is defined by Exp x ( v ) : = γ x , v ( 1 ) \operatorname{Exp}_x(v) := \gamma_{x,v}(1) Exp x ( v ) := γ x , v ( 1 ) . Restricted to a sufficiently small ball around 0 x 0_x 0 x in T x M T_x\mathcal{M} T x M , Exp x \operatorname{Exp}_x Exp x is a diffeomorphism onto a neighborhood of x x x and satisfies d ( x , Exp x ( v ) ) = ∥ v ∥ x d(x,\operatorname{Exp}_x(v)) = \|v\|_x d ( x , Exp x ( v )) = ∥ v ∥ x . A retraction provides a computationally cheaper alternative to the exponential map for mapping tangent vectors to the manifold. For an integer k ≥ 1 k \ge1 k ≥ 1 , a C k C^k C k map R : T M → M R : T\mathcal{M} \to\mathcal{M} R : T M → M is a retraction if, for every x ∈ M x \in\mathcal{M} x ∈ M , its restriction R x : T x M → M R_x : T_x\mathcal{M} \to\mathcal{M} R x : T x M → M satisfies
R x ( 0 x ) = x , D R x ( 0 x ) = I T x M . R_x(0_x) = x, \qquad DR_x(0_x) = I_{T_x\mathcal{M}}. R x ( 0 x ) = x , D R x ( 0 x ) = I T x M .
These conditions ensure that, for every v ∈ T x M v \in T_x\mathcal{M} v ∈ T x M , the curve t ↦ R x ( t v ) t \mapsto R_x(tv) t ↦ R x ( t v ) starts at x x x with initial velocity v v v . When k ≥ 2 k \ge2 k ≥ 2 , R R R is said to be a second-order retraction at x x x if, in addition, D d t c ˙ η ( t ) ∣ t = 0 = 0 x \left.\frac{D}{dt}\dot{c}_\eta(t)\right|_{t=0} = 0_x d t D c ˙ η ( t ) t = 0 = 0 x for every η ∈ T x M \eta\in T_x\mathcal{M} η ∈ T x M , where c η : [ 0 , 1 ] → M c_\eta: [0,1] \to\mathcal{M} c η : [ 0 , 1 ] → M is defined as c η ( t ) : = R x ( t η ) c_\eta(t) := R_x(t\eta) c η ( t ) := R x ( t η ) . By the inverse function theorem, R x R_x R x admits a local inverse, denoted by R x − 1 R_x^{-1} R x − 1 , from a neighborhood of x x x in M \mathcal{M} M to a neighborhood of 0 x 0_x 0 x in T x M T_x\mathcal{M} T x M . Thus, R x − 1 ( y ) R_x^{-1}(y) R x − 1 ( y ) represents a nearby point y y y as a tangent vector in T x M T_x\mathcal{M} T x M . As y → x y \to x y → x , it satisfies ∥ R x − 1 ( y ) ∥ x = d ( x , y ) + o ( d ( x , y ) ) \|R_x^{-1}(y)\|_x = d(x,y) + o(d(x,y)) ∥ R x − 1 ( y ) ∥ x = d ( x , y ) + o ( d ( x , y )) .
Parallel transport and vector transport. Comparing tangent vectors at different points requires maps between the corresponding tangent spaces. For a smooth curve c : [ 0 , 1 ] → M c : [0,1] \to\mathcal{M} c : [ 0 , 1 ] → M and s , t ∈ [ 0 , 1 ] s,t \in[0,1] s , t ∈ [ 0 , 1 ] , the parallel transport P s → t c : T c ( s ) M → T c ( t ) M \mathcal{P}^{c}_{s\to t} : T_{c(s)}\mathcal{M} \to T_{c(t)}\mathcal{M} P s → t c : T c ( s ) M → T c ( t ) M is characterized, for each v ∈ T c ( s ) M v \in T_{c(s)}\mathcal{M} v ∈ T c ( s ) M , by D d t P s → t c ( v ) = 0 \frac{D}{dt}\mathcal{P}^{c}_{s\to t}(v) = 0 d t D P s → t c ( v ) = 0 and P s → s c ( v ) = v \mathcal{P}^{c}_{s\to s}(v) = v P s → s c ( v ) = v . Parallel transport is a linear isometry and therefore preserves inner products and norms. Whenever R x − 1 ( y ) R_x^{-1}(y) R x − 1 ( y ) is well defined, we consider the retraction curve c x , y R : [ 0 , 1 ] → M c^R_{x,y} : [0,1] \to\mathcal{M} c x , y R : [ 0 , 1 ] → M , defined by c x , y R ( t ) : = R x { t R x − 1 ( y ) } c^R_{x,y}(t) := R_x\{tR_x^{-1}(y)\} c x , y R ( t ) := R x { t R x − 1 ( y )} for t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] , and define two maps from T x M T_x\mathcal{M} T x M to T y M T_y\mathcal{M} T y M :
Γ x y : = P 0 → 1 c x , y R , Λ x y : = D R x { R x − 1 ( y ) } . (6) \Gamma^y_x := \mathcal{P}^{c^R_{x,y}}_{0\to1}, \qquad\Lambda^y_x := DR_x\{R_x^{-1}(y)\}. \tag*{(6)} Γ x y := P 0 → 1 c x , y R , Λ x y := D R x { R x − 1 ( y )} . (6)
Here, Γ x y \Gamma^y_x Γ x y is the parallel transport along c x , y R c^R_{x,y} c x , y R , while Λ x y \Lambda^y_x Λ x y is the differential of R x R_x R x with respect to its tangent vector argument at R x − 1 ( y ) R_x^{-1}(y) R x − 1 ( y ) . We denote by Λ \Lambda Λ the family of maps { Λ x y } \{\Lambda^y_x\} { Λ x y } , defined for pairs ( x , y ) (x,y) ( x , y ) for which R x − 1 ( y ) R_x^{-1}(y) R x − 1 ( y ) exists. Both maps are linear, but only Γ x y \Gamma^y_x Γ x y is guaranteed to be an isometry. The inverses of these maps should be distinguished from the corresponding maps obtained by interchanging x x x and y y y . Specifically, Γ x y \Gamma^y_x Γ x y is invertible, with ( Γ x y ) − 1 = P 1 → 0 c x , y R (\Gamma^y_x)^{-1} = \mathcal{P}^{c^R_{x,y}}_{1\to0} ( Γ x y ) − 1 = P 1 → 0 c x , y R , given by parallel transport back along the same curve. In general, this inverse need not coincide with Γ y x \Gamma^x_y Γ y x , since c x , y R c^R_{x,y} c x , y R need not be the reverse of c y , x R c^R_{y,x} c y , x R . Similarly, if R x R_x R x is a local diffeomorphism at R x − 1 ( y ) R_x^{-1}(y) R x − 1 ( y ) , then Λ x y \Lambda^y_x Λ x y is invertible, with ( Λ x y ) − 1 = D ( R x − 1 ) ( y ) (\Lambda^y_x)^{-1} = D(R_x^{-1})(y) ( Λ x y ) − 1 = D ( R x − 1 ) ( y ) . In general, this inverse need not coincide with Λ y x \Lambda^x_y Λ y x , since the latter is obtained by differentiating R y R_y R y rather than R x − 1 R_x^{-1} R x − 1 .
Illustration on the sphere. We next illustrate these definitions on the sphere, where the maps have explicit formulas. Consider the unit sphere S d ⊆ R d + 1 \mathbb{S}^{d} \subseteq\mathbb{R}^{d+1} S d ⊆ R d + 1 with the metric induced by the Euclidean inner product. For x ∈ S d x \in\mathbb{S}^{d} x ∈ S d , tangent vectors are orthogonal to x x x , so T x S d = { v ∈ R d + 1 : x T v = 0 } T_x\mathbb{S}^{d} = \{v \in\mathbb{R}^{d+1} : x^{\mathsf{T}}v = 0\} T x S d = { v ∈ R d + 1 : x T v = 0 } . Thus, P x : = I − x x T P_x := I - xx^{\mathsf{T}} P x := I − x x T is the orthogonal projection from R d + 1 \mathbb{R}^{d+1} R d + 1 onto T x S d T_x\mathbb{S}^{d} T x S d . As x x x varies over the sphere, these tangent spaces form the tangent bundle T S d = { ( x , v ) : ∥ x ∥ = 1 , x T v = 0 } T\mathbb{S}^{d} = \{(x,v) : \lVert x\rVert= 1, x^{\mathsf{T}}v = 0\} T S d = {( x , v ) : ∥ x ∥ = 1 , x T v = 0 } . For the Euclidean-induced metric, the inner product and corresponding distance are
⟨ u , v ⟩ x = u T v , d ( x , y ) = arccos ( x T y ) . \langle u,v\rangle_x = u^{\mathsf{T}}v,\qquad d(x,y) = \arccos(x^{\mathsf{T}}y). ⟨ u , v ⟩ x = u T v , d ( x , y ) = arccos ( x T y ) .
For a smooth function h : R d + 1 → R h : \mathbb{R}^{d+1} \to\mathbb{R} h : R d + 1 → R , its restriction on S d \mathbb{S}^{d} S d , h ∣ S d : S d → R h|_{\mathbb{S}^{d}} : \mathbb{S}^{d} \to\mathbb{R} h ∣ S d : S d → R satisfies
grad ( h ∣ S d ) ( x ) = P x ∇ h ( x ) , Hess ( h ∣ S d ) ( x ) [ v ] = P x ∇ 2 h ( x ) v − { x T ∇ h ( x ) } v . \operatorname{grad}(h|_{\mathbb{S}^{d}})(x) = P_x\nabla h(x),\qquad\operatorname{Hess}(h|_{\mathbb{S}^{d}})(x)[v] = P_x\nabla^{2}h(x)v - \{x^{\mathsf{T}}\nabla h(x)\}v. grad ( h ∣ S d ) ( x ) = P x ∇ h ( x ) , Hess ( h ∣ S d ) ( x ) [ v ] = P x ∇ 2 h ( x ) v − { x T ∇ h ( x )} v .
Here ∇ h \nabla h ∇ h and ∇ 2 h \nabla^{2}h ∇ 2 h denote the Euclidean gradient and Hessian, respectively. On the sphere, geodesics lie on great circles. In this example, we use the second-order retraction obtained by normalizing x + v x+v x + v . For v ∈ T x S d v \in T_x\mathbb{S}^{d} v ∈ T x S d with ∥ v ∥ > 0 \lVert v\rVert> 0 ∥ v ∥ > 0 , the exponential map and the retraction map we consider are
Exp x ( v ) = cos ( ∥ v ∥ ) x + sin ( ∥ v ∥ ) ∥ v ∥ v , R x ( v ) = x + v ( 1 + ∥ v ∥ 2 ) 1 / 2 . (7) \operatorname{Exp}_x(v) = \cos(\lVert v\rVert)x + \frac{\sin(\lVert v\rVert)}{\lVert v\rVert}v,\qquad R_x(v) = \frac{x+v}{(1+\lVert v\rVert^2)^{1/2}}. \tag*{(7)} Exp x ( v ) = cos (∥ v ∥) x + ∥ v ∥ sin (∥ v ∥) v , R x ( v ) = ( 1 + ∥ v ∥ 2 ) 1/2 x + v . (7)
The exponential map moves from x x x in the direction v / ∥ v ∥ v/\lVert v\rVert v / ∥ v ∥ along a great circle, traveling an arc of length ∥ v ∥ \lVert v\rVert ∥ v ∥ . For the retraction, x T v = 0 x^{\mathsf{T}}v = 0 x T v = 0 gives ∥ x + v ∥ = ( 1 + ∥ v ∥ 2 ) 1 / 2 \lVert x+v\rVert= (1+\lVert v\rVert^2)^{1/2} ∥ x + v ∥ = ( 1 + ∥ v ∥ 2 ) 1/2 , which explains the denominator in R x ( v ) R_x(v) R x ( v ) . In particular, R x ( v ) = Exp x { arctan ( ∥ v ∥ ) v / ∥ v ∥ } R_x(v) = \operatorname{Exp}_x\{\arctan(\lVert v\rVert)v/\lVert v\rVert\} R x ( v ) = Exp x { arctan (∥ v ∥) v / ∥ v ∥} , so the retraction follows the same great circle through an angle arctan ( ∥ v ∥ ) \arctan(\lVert v\rVert) arctan (∥ v ∥) instead of ∥ v ∥ \lVert v\rVert ∥ v ∥ . Conversely, for y ∈ S d y \in\mathbb{S}^{d} y ∈ S d , the local inverses Exp x − 1 \operatorname{Exp}_x^{-1} Exp x − 1 and R x − 1 ( y ) R_x^{-1}(y) R x − 1 ( y ) are both well defined when x T y > 0 x^{\mathsf{T}}y > 0 x T y > 0 and y ≠ x y \ne x y = x . Formally,
Exp x − 1 ( y ) = d ( x , y ) sin d ( x , y ) P x y , R x − 1 ( y ) = y x T y − x . \operatorname{Exp}_x^{-1}(y) = \frac{d(x,y)}{\sin d(x,y)}P_xy,\qquad R_x^{-1}(y) = \frac{y}{x^{\mathsf{T}}y} - x. Exp x − 1 ( y ) = sin d ( x , y ) d ( x , y ) P x y , R x − 1 ( y ) = x T y y − x .
Let θ : = d ( x , y ) = arccos ( x T y ) \theta:= d(x,y) = \arccos(x^{\mathsf{T}}y) θ := d ( x , y ) = arccos ( x T y ) denote the angle between x x x and y y y in radians. We can see both vectors point in the direction P x y P_xy P x y , but have lengths θ \theta θ and tan θ \tan\theta tan θ , respectively. Finally, for u ∈ T x S d u \in T_x\mathbb{S}^{d} u ∈ T x S d , the parallel transition map and vector transport map from T x S d T_x\mathbb{S}^{d} T x S d to T y S d T_y\mathbb{S}^{d} T y S d are
Γ x y u = u − u T y 1 + x T y ( x + y ) , Λ x y u = ( x T y ) ( I − y y T ) u . \Gamma_x^y u = u - \frac{u^{\mathsf{T}}y}{1+x^{\mathsf{T}}y}(x+y),\qquad\Lambda_x^y u = (x^{\mathsf{T}}y)(I-yy^{\mathsf{T}})u. Γ x y u = u − 1 + x T y u T y ( x + y ) , Λ x y u = ( x T y ) ( I − y y T ) u .
Averaged RSGD and asymptotic normality
In this section, we introduce the averaged RSGD method for Problem (1) and state the assumptions under which the iterate exhibits asymptotic normality. Let { ξ n } n ≥ 1 \{\xi_n\}_{n\geq1} { ξ n } n ≥ 1 be independent copies of ξ ∼ D \xi\sim\mathcal{D} ξ ∼ D and x 0 ∈ M x_0 \in\mathcal{M} x 0 ∈ M be the initial point. We set x ~ 0 = x 0 \widetilde{x}_0 = x_0 x 0 = x 0 . For n ≥ 1 n \geq1 n ≥ 1 , we define
x n = R x n − 1 ( − γ n grad f ( x n − 1 , ξ n ) ) , x ~ n = R x ~ n − 1 ( 1 n R x ~ n − 1 − 1 ( x n ) ) . (8) x_n = R_{x_{n-1}}\left(-\gamma_n\operatorname{grad}f(x_{n-1},\xi_n)\right),\qquad\widetilde{x}_n = R_{\widetilde{x}_{n-1}}\left(\frac{1}{n}R_{\widetilde{x}_{n-1}}^{-1}(x_n)\right). \tag*{(8)} x n = R x n − 1 ( − γ n grad f ( x n − 1 , ξ n ) ) , x n = R x n − 1 ( n 1 R x n − 1 − 1 ( x n ) ) . (8)
Here, grad f \operatorname{grad} f grad f is the Riemannian gradient with respect to x x x , and γ n \gamma_n γ n is the step size. The second recursion moves the current average x ~ n − 1 \widetilde{x}_{n-1} x n − 1 toward x n x_n x n along the retraction curve, with weight 1 / n 1/n 1/ n . In the Euclidean case, R x ( v ) = x + v R_x(v)=x+v R x ( v ) = x + v , so the two recursions reduce to the standard SGD update and the recursive average x ~ n = x ~ n − 1 + n − 1 ( x n − x ~ n − 1 ) \widetilde{x}_n=\widetilde{x}_{n-1}+n^{-1}(x_n-\widetilde{x}_{n-1}) x n = x n − 1 + n − 1 ( x n − x n − 1 ) , respectively. Throughout the paper, we take step sizes γ n = γ n − α \gamma_n=\gamma n^{-\alpha} γ n = γ n − α for some γ > 0 \gamma> 0 γ > 0 and α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2, 1) α ∈ ( 1/2 , 1 ) , which satisfy the standard conditions for stochastic approximation step sizes, ∑ n ≥ 1 γ n = ∞ \sum_{n \ge1} \gamma_n = \infty ∑ n ≥ 1 γ n = ∞ and ∑ n ≥ 1 γ n 2 < ∞ \sum_{n \ge1} \gamma_n^2 < \infty ∑ n ≥ 1 γ n 2 < ∞ (see, e.g., Benveniste et al., [6 ] ).
In what follows, we review assumptions for asymptotic normality established in Tripuraneni et al. [38 ] . We begin with geometric setting in their Assumption 2, stating the conditions on a compact set X \mathcal{X} X and retraction R R R explicitly. Note that the update of the average requires R x ~ n − 1 − 1 ( x n ) R_{\widetilde{x}_{n-1}}^{-1}(x_n) R x n − 1 − 1 ( x n ) to exist at each iteration. Since this inverse need not exist globally, we assume that the iterates remain in a common set X \mathcal{X} X where the algorithm “behaves well”.
Assumption 2.1 (Manifold and retraction). There exists a compact, connected subset X ⊆ M \mathcal{X} \subseteq\mathcal{M} X ⊆ M containing x ⋆ x_\star x ⋆ such that, almost surely, x n ∈ X x_n \in\mathcal{X} x n ∈ X for every n ≥ 0 n \ge0 n ≥ 0 . Moreover:
(a) The set X \mathcal{X} X is uniformly totally retractive with respect to R R R : there exists ρ r e t > 0 \rho_{\mathrm{ret}} > 0 ρ ret > 0 such that for any x ∈ X x \in\mathcal{X} x ∈ X , the restriction R x ∣ B x ( 0 x , ρ r e t ) R_x|_{B_x(0_x,\rho_{\mathrm{ret}})} R x ∣ B x ( 0 x , ρ ret ) is a diffeomorphism onto its image and X ⊆ R x ( B x ( 0 x , ρ r e t ) ) \mathcal{X} \subseteq R_x(B_x(0_x,\rho_{\mathrm{ret}})) X ⊆ R x ( B x ( 0 x , ρ ret )) , where B x ( 0 x , ρ r e t ) : = { η ∈ T x M : ∥ η ∥ x < ρ r e t } B_x(0_x,\rho_{\mathrm{ret}}) := \{\eta\in T_x\mathcal{M} : \|\eta\|_x < \rho_{\mathrm{ret}}\} B x ( 0 x , ρ ret ) := { η ∈ T x M : ∥ η ∥ x < ρ ret } .
(b) The set X \mathcal{X} X is retraction convex with respect to R R R : ∀ x , y ∈ X \forall x,y \in\mathcal{X} ∀ x , y ∈ X and t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] , R x ( t R x − 1 ( y ) ) ∈ X R_x(tR_x^{-1}(y)) \in\mathcal{X} R x ( t R x − 1 ( y )) ∈ X .
(c) For every y ∈ X y \in\mathcal{X} y ∈ X , the function z ↦ 1 2 ∥ R y − 1 ( z ) ∥ y 2 z \mapsto\frac{1}{2}\|R_y^{-1}(z)\|_y^2 z ↦ 2 1 ∥ R y − 1 ( z ) ∥ y 2 is retraction strongly convex on X \mathcal{X} X with respect to R R R : for any x ∈ X x \in\mathcal{X} x ∈ X and unit vector η ∈ T x M \eta\in T_x\mathcal{M} η ∈ T x M , the function t ↦ 1 2 ∥ R y − 1 { R x ( t η ) } ∥ y 2 t \mapsto\frac{1}{2}\|R_y^{-1}\{R_x(t\eta)\}\|_y^2 t ↦ 2 1 ∥ R y − 1 { R x ( t η )} ∥ y 2 is strongly convex on every interval on which R x ( t η ) ∈ X R_x(t\eta) \in\mathcal{X} R x ( t η ) ∈ X .
(d) The retraction R R R is second order at every x ∈ X x \in\mathcal{X} x ∈ X .
(e) The map ( x , η ) ↦ R x ( η ) (x,\eta) \mapsto R_x(\eta) ( x , η ) ↦ R x ( η ) is C 3 C^3 C 3 on an open neighborhood in T M T\mathcal{M} T M of { ( x , R x − 1 ( y ) ) : x , y ∈ X } \{(x,R_x^{-1}(y)):x,y \in\mathcal{X}\} {( x , R x − 1 ( y )) : x , y ∈ X } .
The requirement that { x n } n ⊆ X \{x_n\}_n \subseteq\mathcal{X} { x n } n ⊆ X is common in stochastic approximation on manifolds (Bonnabel, [7 ] ; Sato et al., [36 ] ; Tripuraneni et al., [38 ] ). Under this stability condition, item (a) makes R x ~ n − 1 − 1 ( x n ) R_{\widetilde{x}_{n-1}}^{-1}(x_n) R x n − 1 − 1 ( x n ) well defined whenever x ~ n − 1 ∈ X \widetilde{x}_{n-1} \in\mathcal{X} x n − 1 ∈ X , while item (b) then keeps x ~ n \widetilde{x}_n x n in X \mathcal{X} X . Since x ~ 0 = x 0 ∈ X \widetilde{x}_0 = x_0 \in\mathcal{X} x 0 = x 0 ∈ X , induction shows that every average x ~ n \widetilde{x}_n x n is well defined and belongs to X \mathcal{X} X . The following sphere example illustrates items (a) and (b).
Consider M = S 2 \mathcal{M} = \mathbb{S}^2 M = S 2 with the normalization retraction in (7) . In the left panel of Figure 1 , let X \mathcal{X} X be the shaded compact, connected region with maximum pairwise distance δ : = max x , y ∈ X d ( x , y ) < π / 2 \delta:= \max_{x,y\in\mathcal{X}} d(x,y) < \pi/2 δ := max x , y ∈ X d ( x , y ) < π /2 . For any fixed x ∈ X x \in\mathcal{X} x ∈ X , the inverse retraction formula in § 2.1 gives ∥ R x − 1 ( y ) ∥ x = tan d ( x , y ) ≤ tan δ \|R_x^{-1}(y)\|_x = \tan d(x,y) \le\tan\delta ∥ R x − 1 ( y ) ∥ x = tan d ( x , y ) ≤ tan δ for all y ∈ X y \in\mathcal{X} y ∈ X . We can therefore choose a common radius ρ r e t > tan δ \rho_{\mathrm{ret}} > \tan\delta ρ ret > tan δ such that R x − 1 ( X ) ⊆ B x ( 0 x , ρ r e t ) R_x^{-1}(\mathcal{X}) \subseteq B_x(0_x,\rho_{\mathrm{ret}}) R x − 1 ( X ) ⊆ B x ( 0 x , ρ ret ) for every x ∈ X x \in\mathcal{X} x ∈ X , which verifies item (a). In the right panel, let X \mathcal{X} X be the shaded region, which appears elliptical in the figure. For any x , y ∈ X x,y \in\mathcal{X} x , y ∈ X , the vectors t R x − 1 ( y ) tR_x^{-1}(y) t R x − 1 ( y ) , t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] , form the line segment from 0 x 0_x 0 x to R x − 1 ( y ) R_x^{-1}(y) R x − 1 ( y ) in T x M T_x\mathcal{M} T x M . The retraction R x R_x R x maps this segment to the short great-circle arc from x x x to y y y , which stays in the displayed region. Hence, R x ( t R x − 1 ( y ) ) ∈ X R_x(tR_x^{-1}(y)) \in\mathcal{X} R x ( t R x − 1 ( y )) ∈ X for all t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] , verifying the retraction convexity in item (b).
Figure 1. Left: Assumption 2.1(a). The inverse image R x − 1 ( X ) R_x^{-1}(\mathcal{X}) R x − 1 ( X ) lies in the tangent ball B x ( 0 x , ρ r e t ) B_x(0_x,\rho_{\mathrm{ret}}) B x ( 0 x , ρ ret ) for every x ∈ X x \in\mathcal{X} x ∈ X , with a common radius ρ r e t > tan ( max x , y ∈ X d ( x , y ) ) \rho_{\mathrm{ret}} > \tan(\max_{x,y\in\mathcal{X}} d(x,y)) ρ ret > tan ( max x , y ∈ X d ( x , y )) . Right: Assumption 2.1(b). For any x , y ∈ X x,y \in\mathcal{X} x , y ∈ X , the normalization retraction maps the line segment { t R x − 1 ( y ) : t ∈ [ 0 , 1 ] } \{tR_x^{-1}(y):t\in[0,1]\} { t R x − 1 ( y ) : t ∈ [ 0 , 1 ]} in T x M T_x\mathcal{M} T x M onto the short great circle arc from x x x to y y y , which remains in X \mathcal{X} X , illustrating its retraction convexity.
Item (c) is restated from (Tripuraneni et al., [38 ] , Assumption 2) and adopts the notion of strong convexity along retraction curves introduced by Huang et al. [21 ] . Item (d) requires the retraction to be second order, which holds for the normalization retraction on the sphere and the Grassmann polar retraction used in § 4.2 (Absil and Malick, [1 ] ). Finally, item (e) provides the regularity conditions needed for the Taylor expansions used to control the geometric errors.
For the objective function F F F , we restate the smoothness and local strong convexity conditions in Tripuraneni et al. [38 ] , Assumptions 3–4).
Assumption 2.2 (Objective function). The objective function F F F satisfies the following conditions:
(a) F F F is twice continuously differentiable on an open neighborhood of X \mathcal{X} X and grad F ( x ⋆ ) = 0 \operatorname{grad} F(x_\star)=0 grad F ( x ⋆ ) = 0 .
(b) Let A : = Hess F ( x ⋆ ) : T x ⋆ M → T x ⋆ M A := \operatorname{Hess} F(x_\star):T_{x_\star}\mathcal{M}\to T_{x_\star}\mathcal{M} A := Hess F ( x ⋆ ) : T x ⋆ M → T x ⋆ M . There exists μ > 0 \mu>0 μ > 0 such that for any η ∈ T x ⋆ M \eta\in T_{x_\star}\mathcal{M} η ∈ T x ⋆ M ,
⟨ η , A η ⟩ x ⋆ ≥ μ ∥ η ∥ x ⋆ 2 . \langle\eta,A\eta\rangle_{x_\star}\ge\mu\|\eta\|_{x_\star}^{2}. ⟨ η , A η ⟩ x ⋆ ≥ μ ∥ η ∥ x ⋆ 2 .
Items (a) and (b) state that x ⋆ x_\star x ⋆ is a stationary point with a positive definite Riemannian Hessian. Consequently, x ⋆ x_\star x ⋆ is a strict local minimizer and A A A is invertible. Item (c) generalizes the Lipschitz Hessian assumption to Riemannian manifolds, with one point fixed at x ⋆ x_\star x ⋆ .
The corresponding noise conditions in Tripuraneni et al. [38 ] (Assumptions 5–6) take the following form for the stochastic gradients { grad f ( ⋅ , ξ n ) } n ≥ 1 \{\operatorname{grad} f(\cdot,\xi_n)\}_{n\ge1} { grad f ( ⋅ , ξ n ) } n ≥ 1 .
Assumption 2.3 (Gradient noise). We assume the stochastic gradients satisfy the following conditions:
(a) The random vector fields { grad f ( ⋅ , ξ n ) } n ≥ 1 \{\operatorname{grad} f(\cdot,\xi_n)\}_{n\ge1} { grad f ( ⋅ , ξ n ) } n ≥ 1 are F n \mathcal{F}_n F n -measurable. For every x ∈ X x\in\mathcal{X} x ∈ X and n ≥ 1 n\ge1 n ≥ 1 , the random vector grad f ( x , ξ n ) \operatorname{grad} f(x,\xi_n) grad f ( x , ξ n ) is unbiased: E n − 1 [ grad f ( x , ξ n ) ] = grad F ( x ) \mathbb{E}_{n-1}[\operatorname{grad} f(x,\xi_n)]=\operatorname{grad} F(x) E n − 1 [ grad f ( x , ξ n )] = grad F ( x ) .
(b) There exists τ < ∞ \tau<\infty τ < ∞ such that for any x ∈ X x\in\mathcal{X} x ∈ X and n ≥ 1 n\ge1 n ≥ 1 ,
E n − 1 [ ∥ grad f ( x , ξ n ) ∥ x 4 ] ≤ τ 4 . \mathbb{E}_{n-1}[\|\operatorname{grad} f(x,\xi_n)\|_{x}^{4}]\le\tau^{4}. E n − 1 [ ∥ grad f ( x , ξ n ) ∥ x 4 ] ≤ τ 4 .
These conditions extend standard noise assumptions for Euclidean stochastic approximation (Fabian [16 ] ) to the manifold setting (Tripuraneni et al. [38 ] ). In particular, the operator S S S in item (d) is the covariance operator of the gradient noise at x ⋆ x_\star x ⋆ , since grad F ( x ⋆ ) = 0 \operatorname{grad} F(x_\star)=0 grad F ( x ⋆ ) = 0 .
Finally, we state the moment bound conditions used in Tripuraneni et al. [38 ] (Assumption 1).
Assumption 2.4 (Moment bounds for the RSGD iterates). The RSGD iterates satisfy the following moment conditions. Let Δ n : = R x ⋆ − 1 ( x n ) ∈ T x ⋆ M \Delta_n := R_{x_\star}^{-1}(x_n) \in T_{x_\star}\mathcal{M} Δ n := R x ⋆ − 1 ( x n ) ∈ T x ⋆ M .
(a) The iterates satisfy E ∥ Δ n ∥ x ⋆ 2 = O ( γ n ) \mathbb{E}\lVert\Delta_n\rVert_{x_\star}^{2} = O(\gamma_n) E ∥ Δ n ∥ x ⋆ 2 = O ( γ n ) .
(b) The iterates satisfy E ∥ Δ n ∥ x ⋆ 4 = O ( γ n 2 ) \mathbb{E}\lVert\Delta_n\rVert_{x_\star}^{4} = O(\gamma_n^{2}) E ∥ Δ n ∥ x ⋆ 4 = O ( γ n 2 ) .
Since x ⋆ x_\star x ⋆ is unknown, the tangent vector Δ n ∈ T x ⋆ M \Delta_n \in T_{x_\star}\mathcal{M} Δ n ∈ T x ⋆ M cannot be computed by the algorithm and is introduced only for the analysis. The second-moment bound in item (a) is required to establish asymptotic normality, while the fourth-moment bound in item (b) is required to estimate the covariance (a requirement needed even in the Euclidean setting). Consistent with these roles, Tripuraneni et al. [38 ] uses the fourth-moment bound to establish a nonasymptotic error rate for the averaged iterates. In the Euclidean setting, the second- and fourth-moment bounds follow from strong convexity, Lipschitz continuity of the gradient, and suitable moment bounds on the stochastic noise; see Chen et al. [11 ] , Zhu et al. [46 ] , and (Kuang et al., [25 ] ). The same moment bounds in Assumption 2.4 have been verified for a variety of manifold optimization problems; see [5 , 45 ] .
Under Assumptions 2.1–2.3 and the second-moment condition in Assumption 2.4(a), the asymptotic normality result of Tripuraneni et al. [38 ] takes the following form.
Theorem 2.5 (Asymptotic normality of averaged RSGD). Consider the RSGD iterates { x n } n \{x_n\}_n { x n } n and recursive averages { x ~ n } n \{\widetilde{x}_n\}_n { x n } n in (8) , with step sizes γ n = γ n − α \gamma_n = \gamma n^{-\alpha} γ n = γ n − α for γ > 0 \gamma> 0 γ > 0 and α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2,1) α ∈ ( 1/2 , 1 ) . Under Assumptions 2.1–2.3, 2.4(a), define Δ ~ n : = R x ⋆ − 1 ( x ~ n ) ∈ T x ⋆ M \widetilde{\Delta}_n := R_{x_\star}^{-1}(\widetilde{x}_n) \in T_{x_\star}\mathcal{M} Δ n := R x ⋆ − 1 ( x n ) ∈ T x ⋆ M . Then, with respect to any orthonormal basis of T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M , we have
n Δ ~ n → d N ( 0 , Σ ) with Σ : = A − 1 S A − 1 . \sqrt{n}\,\widetilde{\Delta}_n \xrightarrow{\mathrm{d}} \mathcal{N}(0,\Sigma) \qquad\text{with} \qquad\Sigma:= A^{-1}SA^{-1}. n Δ n d N ( 0 , Σ ) with Σ := A − 1 S A − 1 .
Theorem 2.5 shows that the error of the averaged RSGD iterate exhibits asymptotic normality in T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M . Its limiting covariance, however, depends on the unknown operators A A A and S S S . To enable practical inference, § 3 develops consistent online batch-means estimators of this covariance Σ \Sigma Σ using only the RSGD iterates, without evaluating the Riemannian Hessian. Combining these estimators with Theorem 2.5 yields the confidence intervals and confidence regions constructed in § 4 .
Online covariance estimation on Riemannian manifolds
In this section, we develop online batch-means estimators of the asymptotic covariance Σ \Sigma Σ in Theorem 2.5 . The target covariance Σ \Sigma Σ acts on T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M , but this space is unknown because x ⋆ x_\star x ⋆ is unknown. A natural choice is therefore to construct the estimator on T x ~ n M T_{\widetilde{x}_n}\mathcal{M} T x n M , using the averaged iterate x ~ n \widetilde{x}_n x n as our current estimate of x ⋆ x_\star x ⋆ . Unlike in Euclidean space, both the operator and the space on which it acts change as the average x ~ n \widetilde{x}_n x n is updated, creating a central challenge for online estimation. We first present a general framework for constructing online batch-means estimators on manifolds, then specialize it to two estimators: overlapping (OL) and non-overlapping (NOL). § 3.2 presents a common online implementation. § 3.3 establishes convergence rates for both estimators, and § 3.4 outlines the proof strategy, with full details deferred to Appendix § 11 .
Online batch-means covariance estimators
We first introduce the notation and components used to construct our batch-means estimator. Consider the RSGD iterates { x j } j ≥ 1 \{x_j\}_{j\ge1} { x j } j ≥ 1 generated by (8) . Let { a m } m ≥ 1 \{a_m\}_{m\ge1} { a m } m ≥ 1 be a user-selected, strictly increasing sequence of positive integers with a 1 = 1 a_1 = 1 a 1 = 1 . For each j ≥ 1 j \ge1 j ≥ 1 , define m j : = max { m : a m ≤ j } m_j := \max\{m : a_m \le j\} m j := max { m : a m ≤ j } and let t j : = a m j t_j := a_{m_j} t j := a m j be the largest value of a m a_m a m not exceeding j j j . The construction consists of three components:
Batches. We adopt the same batch construction as Zhu et al. [46 ] in the Euclidean setting. For each j ≥ 1 j \ge1 j ≥ 1 , define the batch
B j : = { x t j , x t j + 1 , … , x j } . (9) B_j := \{x_{t_j}, x_{t_j+1}, \ldots, x_j\}. \tag*{(9)} B j := { x t j , x t j + 1 , … , x j } . (9)
Its length is ℓ j : = ∣ B j ∣ = j − t j + 1 \ell_j := |B_j| = j - t_j + 1 ℓ j := ∣ B j ∣ = j − t j + 1 . For example, with a m = m 2 a_m = m^2 a m = m 2 we have t j = ⌊ j ⌋ 2 t_j = \lfloor\sqrt{j}\rfloor^2 t j = ⌊ j ⌋ 2 and
B 1 = { x 1 } , B 2 = { x 1 , x 2 } , B 3 = { x 1 , x 2 , x 3 } , B 4 = { x 4 } , B 5 = { x 4 , x 5 } , … , B 8 = { x 4 , x 5 , x 6 , x 7 , x 8 } , B 9 = { x 9 } , B 10 = { x 9 , x 10 } , B 11 = { x 9 , x 10 , x 11 } , B 12 = { x 9 , x 10 , x 11 , x 12 } , … . (10) \begin{aligned}
B_1 &= \{x_1\}, \quad B_2 = \{x_1,x_2\}, \quad B_3 = \{x_1,x_2,x_3\},\\
B_4 &= \{x_4\}, \quad B_5 = \{x_4,x_5\}, \quad\ldots,\quad B_8 = \{x_4,x_5,x_6,x_7,x_8\},\\
B_9 &= \{x_9\}, \quad B_{10} = \{x_9,x_{10}\}, \quad B_{11} = \{x_9,x_{10},x_{11}\}, \quad B_{12} = \{x_9,x_{10},x_{11},x_{12}\}, \quad\ldots. \tag*{(10)}
\end{aligned} B 1 B 4 B 9 = { x 1 } , B 2 = { x 1 , x 2 } , B 3 = { x 1 , x 2 , x 3 } , = { x 4 } , B 5 = { x 4 , x 5 } , … , B 8 = { x 4 , x 5 , x 6 , x 7 , x 8 } , = { x 9 } , B 10 = { x 9 , x 10 } , B 11 = { x 9 , x 10 , x 11 } , B 12 = { x 9 , x 10 , x 11 , x 12 } , … . (10)
In general, for a m ≤ j < a m + 1 a_m \le j < a_{m+1} a m ≤ j < a m + 1 , the batch B j B_j B j contains the iterates x a m , … , x j x_{a_m}, \ldots, x_j x a m , … , x j . Thus, successive batches within this interval share a starting index and increase in length by one. At j = a m + 1 j = a_{m+1} j = a m + 1 , the starting index changes and B a m + 1 = { x a m + 1 } B_{a_{m+1}} = \{x_{a_{m+1}}\} B a m + 1 = { x a m + 1 } .
Batch means. Since arithmetic averaging is not intrinsically defined on a general Riemannian manifold, we use the same retraction-based averaging rule as for the overall average x ~ n \tilde{x}_n x ~ n in (8) . We define the mean c j c_j c j of batch B j B_j B j recursively by
c j : = { x j , j = t j , R c j − 1 ( ℓ j − 1 R c j − 1 − 1 ( x j ) ) , j > t j . (11) c_j := \begin{cases}
x_j, & j=t_j,\\
R_{c_{j-1}}\left(\ell_j^{-1}R^{-1}_{c_{j-1}}(x_j)\right), & j>t_j.
\end{cases} \tag*{(11)} c j := { x j , R c j − 1 ( ℓ j − 1 R c j − 1 − 1 ( x j ) ) , j = t j , j > t j . (11)
Assumption 2.1 ensures that this recursion is well defined and that c j ∈ X c_j \in\mathcal{X} c j ∈ X . In Euclidean space, with the standard retraction R x ( v ) = x + v R_x(v) = x + v R x ( v ) = x + v , it reduces to the usual arithmetic batch mean. Equation (11) also provides the online update for c j c_j c j used in our algorithm.
Batch deviations. We next measure the deviation of each batch mean c j c_j c j from the overall average x ~ n \tilde{x}_n x ~ n . Since subtraction is not intrinsically defined on a general Riemannian manifold, we express this deviation as a tangent vector using the inverse retraction:
b j , n : = R x ~ n − 1 ( c j ) ∈ T x ~ n M , n ≥ j . (12) b_{j,n} := R^{-1}_{\tilde{x}_n}(c_j) \in T_{\tilde{x}_n}\mathcal{M}, \qquad n \ge j. \tag*{(12)} b j , n := R x ~ n − 1 ( c j ) ∈ T x ~ n M , n ≥ j . (12)
In Euclidean space, with R x ( v ) = x + v R_x(v) = x+v R x ( v ) = x + v , we have b j , n = c j − x ˉ n = ℓ j − 1 ∑ i = t j j x i − x ˉ n b_{j,n} = c_j-\bar{x}_n = \ell_j^{-1}\sum_{i=t_j}^{j}x_i-\bar{x}_n b j , n = c j − x ˉ n = ℓ j − 1 ∑ i = t j j x i − x ˉ n , where x ˉ n = n − 1 ∑ i = 1 n x i \bar{x}_n = n^{-1}\sum_{i=1}^{n}x_i x ˉ n = n − 1 ∑ i = 1 n x i is the arithmetic average of all the iterates. For a fixed j j j , the batch mean c j c_j c j remains unchanged after time j j j . However, as x ~ n \tilde{x}_n x ~ n is updated, both the value of b j , n b_{j,n} b j , n and the tangent space to which it belongs change. Evaluating (12) directly for all selected batches at each time n n n would require storing and revisiting their means, with memory and per-iteration costs growing with the number of selected batches. To enable an online implementation, we instead approximate b j , n b_{j,n} b j , n by the recursively updated vector
b ~ j , k : = { b j , j , k = j , Λ ~ x ~ k − 1 x ~ k ( b ~ j , k − 1 − u k ) , k = j + 1 , … , n , (13) \tilde{b}_{j,k} := \begin{cases}
b_{j,j}, & k=j,\\
\widetilde{\Lambda}^{\tilde{x}_k}_{\tilde{x}_{k-1}}\left(\tilde{b}_{j,k-1}-u_k\right), & k=j+1,\ldots,n,
\end{cases} \tag*{(13)} b ~ j , k := { b j , j , Λ x ~ k − 1 x ~ k ( b ~ j , k − 1 − u k ) , k = j , k = j + 1 , … , n , (13)
where u k : = R x ~ k − 1 − 1 ( x ~ k ) = k − 1 R x ~ k − 1 − 1 ( x k ) ∈ T x ~ k − 1 M u_k := R^{-1}_{\tilde{x}_{k-1}}(\tilde{x}_k) = k^{-1}R^{-1}_{\tilde{x}_{k-1}}(x_k) \in T_{\tilde{x}_{k-1}}\mathcal{M} u k := R x ~ k − 1 − 1 ( x ~ k ) = k − 1 R x ~ k − 1 − 1 ( x k ) ∈ T x ~ k − 1 M and Λ ~ x ~ k − 1 x ~ k \widetilde{\Lambda}^{\tilde{x}_k}_{\tilde{x}_{k-1}} Λ x ~ k − 1 x ~ k is the vector transport defined in (6) with x = x ~ k − 1 x=\tilde{x}_{k-1} x = x ~ k − 1 and y = x ~ k y=\tilde{x}_k y = x ~ k . For k > j k>j k > j , the update first subtracts u k u_k u k from b ~ j , k − 1 \tilde{b}_{j,k-1} b ~ j , k − 1 to account for the movement of the overall average from x ~ k − 1 \tilde{x}_{k-1} x ~ k − 1 to x ~ k \tilde{x}_k x ~ k . It then applies the linear map Λ x ~ k − 1 x ~ k \Lambda^{\tilde{x}_k}_{\tilde{x}_{k-1}} Λ x ~ k − 1 x ~ k to transport the adjusted vector from T x ~ k − 1 M T_{\tilde{x}_{k-1}}\mathcal{M} T x ~ k − 1 M to T x ~ k M T_{\tilde{x}_k}\mathcal{M} T x ~ k M . In Euclidean space, Λ x ˉ k − 1 x ˉ k \Lambda^{\bar{x}_k}_{\bar{x}_{k-1}} Λ x ˉ k − 1 x ˉ k is the identity map and u k = x ˉ k − x ˉ k − 1 u_k=\bar{x}_k-\bar{x}_{k-1} u k = x ˉ k − x ˉ k − 1 . Thus, (13) reproduces the exact identity b j , k = b j , k − 1 − ( x ˉ k − x ˉ k − 1 ) b_{j,k}=b_{j,k-1}-(\bar{x}_k-\bar{x}_{k-1}) b j , k = b j , k − 1 − ( x ˉ k − x ˉ k − 1 ) , yielding b ~ j , n = b j , n \tilde{b}_{j,n}=b_{j,n} b ~ j , n = b j , n . On a general manifold, b ~ j , n \tilde{b}_{j,n} b ~ j , n only approximates b j , n b_{j,n} b j , n , but our analysis shows that the resulting error does not affect the convergence rate of the covariance estimator.
The covariance estimator need not use every batch. Let E n ⊂ { 1 , … , n } E_n\subset\{1,\ldots,n\} E n ⊂ { 1 , … , n } index the batches selected at time n n n . We assume that the prescribed sequence { E n } n ≥ 1 \{E_n\}_{n\ge1} { E n } n ≥ 1 satisfies E 1 = { 1 } E_1=\{1\} E 1 = { 1 } and that E n + 1 E_{n+1} E n + 1 is either E n ∪ { n + 1 } E_n\cup\{n+1\} E n ∪ { n + 1 } or ( E n ∖ { n } ) ∪ { n + 1 } (E_n\setminus\{n\})\cup\{n+1\} ( E n ∖ { n }) ∪ { n + 1 } for every n ≥ 1 n\ge1 n ≥ 1 . Using the recursively updated deviations from (13) , we define our general batch-means estimator as
Σ ~ n : = 1 v n ∑ j ∈ E n ℓ j 2 b ~ j , n ⊗ 2 , v n : = ∑ j ∈ E n ℓ j . (14) \tilde{\Sigma}_n:=\frac{1}{v_n}\sum_{j\in E_n}\ell_j^2\tilde{b}_{j,n}^{\otimes2},\qquad v_n:=\sum_{j\in E_n}\ell_j. \tag*{(14)} Σ ~ n := v n 1 j ∈ E n ∑ ℓ j 2 b ~ j , n ⊗ 2 , v n := j ∈ E n ∑ ℓ j . (14)
We now specialize the choice of E n E_n E n and introduce two main estimators we consider in this work.
Overlapping estimator. We take E n = E n O L : = { 1 , … , n } E_n=E_n^{\mathrm{OL}}:=\{1,\ldots,n\} E n = E n OL := { 1 , … , n } to obtain the OL estimator. In this case, every batch B j B_j B j is selected. The general covariance estimator in (14) then becomes
Σ ~ n , O L : = 1 v n ∑ j ∈ E n O L ℓ j 2 b ~ j , n ⊗ 2 . (15) \tilde{\Sigma}_{n,\mathrm{OL}}:=\frac{1}{v_n}\sum_{j\in E_n^{\mathrm{OL}}}\ell_j^2\tilde{b}_{j,n}^{\otimes2}. \tag*{(15)} Σ ~ n , OL := v n 1 j ∈ E n OL ∑ ℓ j 2 b ~ j , n ⊗ 2 . (15)
Here v n = ∑ j ∈ E n O L ℓ j v_n=\sum_{j\in E_n^{\mathrm{OL}}}\ell_j v n = ∑ j ∈ E n OL ℓ j is the sum of the batch lengths, including overlaps. In Euclidean space, with R x ( v ) = x + v R_x(v)=x+v R x ( v ) = x + v , the estimator recovers the overlapping estimator introduced in Zhu et al. [46 ] .
Non-overlapping estimator. For the non-overlapping version, we take
E n N O L : = { a m + 1 − 1 : m ≥ 1 , a m + 1 ≤ n } ∪ { n } . (16) E_n^{\mathrm{NOL}}:=\{a_{m+1}-1:m\ge1,\ a_{m+1}\le n\}\cup\{n\}. \tag*{(16)} E n NOL := { a m + 1 − 1 : m ≥ 1 , a m + 1 ≤ n } ∪ { n } . (16)
In contrast, the NOL estimator selects only the past batches that end immediately before a new starting index, together with the current batch B n B_n B n . The batches we select are pairwise disjoint and partition { x 1 , … , x n } \{x_1,\ldots,x_n\} { x 1 , … , x n } , so their total length is v n = n v_n=n v n = n . With this choice, the covariance estimator in (14) becomes
Σ ~ n , N O L : = 1 n ∑ j ∈ E n N O L ℓ j 2 b ~ j , n ⊗ 2 . (17) \tilde{\Sigma}_{n,\mathrm{NOL}}:=\frac{1}{n}\sum_{j\in E_n^{\mathrm{NOL}}}\ell_j^2\tilde{b}_{j,n}^{\otimes2}. \tag*{(17)} Σ ~ n , NOL := n 1 j ∈ E n NOL ∑ ℓ j 2 b ~ j , n ⊗ 2 . (17)
In Euclidean space, with R x ( v ) = x + v R_x(v)=x+v R x ( v ) = x + v , the estimator recovers the non-overlapping estimator introduced in Zhu et al. [46 ] .
For comparison, consider the example with a m = m 2 a_m=m^2 a m = m 2 in (10) . At time 9, the OL estimator selects B 1 , … , B 9 B_1,\ldots,B_9 B 1 , … , B 9 , while the NOL estimator selects only B 3 B_3 B 3 , B 8 B_8 B 8 , B 9 B_9 B 9 . At time 10, OL keeps B 9 B_9 B 9 and adds B 10 B_{10} B 10 , whereas NOL replaces B 9 B_9 B 9 with B 10 B_{10} B 10 . In the next section, we present a concrete algorithm to calculate Σ ~ n \tilde{\Sigma}_n Σ ~ n in an online fashion.
Online implementation
To illustrate the method, we separate the current batch B n B_n B n from the other selected batches. At time n n n , let H n : = E n ∖ { n } H_n:=E_n\setminus\{n\} H n := E n ∖ { n } denote the indices of these other batches. We call the batches B j B_j B j with j ∈ H n j\in H_n j ∈ H n retained batches. Under the assumed selection rule that E n + 1 E_{n+1} E n + 1 is either E n ∪ { n + 1 } E_n \cup\{n+1\} E n ∪ { n + 1 } or ( E n ∖ { n } ) ∪ { n + 1 } (E_n \setminus\{n\}) \cup\{n+1\} ( E n ∖ { n }) ∪ { n + 1 } , only the current index may be removed at each update. Hence, j ∈ H n j \in H_n j ∈ H n implies j ∈ E k j \in E_k j ∈ E k for every k ≥ n k \ge n k ≥ n , so a retained batch remains selected at all subsequent times. The algorithm handles the retained batches and the current batch separately.
Alongside the iterate x n x_n x n and overall average x ~ n \tilde{x}_n x ~ n , the algorithm maintains m n m_n m n , which identifies the starting index a m n a_{m_n} a m n , and the current batch length ℓ n \ell_n ℓ n , mean c n c_n c n , and deviation approximation b ~ n , n \tilde{b}_{n,n} b ~ n , n . For the retained batches, it maintains their total length s n : = ∑ j ∈ H n ℓ j = v n − ℓ n s_n := \sum_{j\in H_n}\ell_j = v_n-\ell_n s n := ∑ j ∈ H n ℓ j = v n − ℓ n and the weighted sums
q n : = ∑ j ∈ H n ℓ j 2 , P n : = ∑ j ∈ H n ℓ j 2 b ~ j , n , V n : = ∑ j ∈ H n ℓ j 2 b ~ j , n ⊗ 2 . q_n := \sum_{j\in H_n}\ell_j^2,\qquad P_n := \sum_{j\in H_n}\ell_j^2\tilde{b}_{j,n},\qquad V_n := \sum_{j\in H_n}\ell_j^2\tilde{b}_{j,n}^{\otimes2}. q n := j ∈ H n ∑ ℓ j 2 , P n := j ∈ H n ∑ ℓ j 2 b ~ j , n , V n := j ∈ H n ∑ ℓ j 2 b ~ j , n ⊗ 2 .
Here, V n V_n V n collects the contributions of the retained batches to the numerator of (14) , while s n s_n s n accounts for their contribution to the normalizing factor. The auxiliary sums P n P_n P n and q n q_n q n can be maintained recursively and allow V n V_n V n to be updated without revisiting individual retained batches. Algorithm 1 combines these updates with those of the current batch to compute the estimator in an online fashion.
Table 1.
For n ≥ 1 n\ge1 n ≥ 1 , step 2 determines whether B n B_n B n becomes a retained batch. If n ∈ E n + 1 n\in E_{n+1} n ∈ E n + 1 , then H n + 1 = H n ∪ { n } H_{n+1}=H_n\cup\{n\} H n + 1 = H n ∪ { n } , and its contribution is added to the retained-batch sums. Otherwise, H n + 1 = H n H_{n+1}=H_n H n + 1 = H n , and these sums remain unchanged. Step 3 then computes the mean c n + 1 c_{n+1} c n + 1 of B n + 1 B_{n+1} B n + 1 . If n + 1 = a m n + 1 n+1=a_{m_{n+1}} n + 1 = a m n + 1 , this batch contains only the iterate x n + 1 x_{n+1} x n + 1 , so c n + 1 = x n + 1 c_{n+1}=x_{n+1} c n + 1 = x n + 1 . Otherwise, B n + 1 B_{n+1} B n + 1 extends B n B_n B n by one iterate, and c n + 1 c_{n+1} c n + 1 is computed from c n c_n c n using (11) . The algorithm then evaluates the deviation b ~ n + 1 , n + 1 = R x ~ n + 1 − 1 ( c n + 1 ) \tilde{b}_{n+1,n+1}=R_{\tilde{x}_{n+1}}^{-1}(c_{n+1}) b ~ n + 1 , n + 1 = R x ~ n + 1 − 1 ( c n + 1 ) .
The deviation of B n + 1 B_{n+1} B n + 1 already lies in T x ~ n + 1 M T_{\tilde{x}_{n+1}}\mathcal{M} T x ~ n + 1 M , but the retained-batch sums P n P_n P n and V n V_n V n assembled in step 2 still use the deviations b ~ j , n ∈ T x ~ n M \tilde{b}_{j,n}\in T_{\tilde{x}_n}\mathcal{M} b ~ j , n ∈ T x ~ n M . Step 4 adjusts these sums for the movement of the overall average through the common shift and transport in (13) . For the weighted sum P n P_n P n , implies
P n + 1 = ∑ j ∈ H n + 1 ℓ j 2 b ~ j , n + 1 = Λ n + 1 { ∑ j ∈ H n + 1 ℓ j 2 b ~ j , n − ( ∑ j ∈ H n + 1 ℓ j 2 ) u n + 1 } = Λ n + 1 ( P n − q n u n + 1 ) . P_{n+1}=\sum_{j\in H_{n+1}}\ell_j^2\widetilde{b}_{j,n+1}
=\Lambda_{n+1}\left\{\sum_{j\in H_{n+1}}\ell_j^2\widetilde{b}_{j,n}-\left(\sum_{j\in H_{n+1}}\ell_j^2\right)u_{n+1}\right\}
=\Lambda_{n+1}(P_n-q_nu_{n+1}). P n + 1 = j ∈ H n + 1 ∑ ℓ j 2 b j , n + 1 = Λ n + 1 ⎩ ⎨ ⎧ j ∈ H n + 1 ∑ ℓ j 2 b j , n − j ∈ H n + 1 ∑ ℓ j 2 u n + 1 ⎭ ⎬ ⎫ = Λ n + 1 ( P n − q n u n + 1 ) .
For the second-order sum, expanding the weighted outer products yields
∑ j ∈ H n + 1 ℓ j 2 ( b ~ j , n − u n + 1 ) ⊗ 2 = ∑ j ∈ H n + 1 ℓ j 2 b ~ j , n ⊗ 2 − ( ∑ j ∈ H n + 1 ℓ j 2 b ~ j , n ) ⊗ u n + 1 − u n + 1 ⊗ ( ∑ j ∈ H n + 1 ℓ j 2 b ~ j , n ) + ( ∑ j ∈ H n + 1 ℓ j 2 ) u n + 1 ⊗ 2 = V n − P n ⊗ u n + 1 − u n + 1 ⊗ P n + q n u n + 1 ⊗ 2 . \begin{aligned}
\sum_{j\in H_{n+1}}\ell_j^2(\widetilde{b}_{j,n}-u_{n+1})^{\otimes2}
&=\sum_{j\in H_{n+1}}\ell_j^2\widetilde{b}_{j,n}^{\otimes2}
-\left(\sum_{j\in H_{n+1}}\ell_j^2\widetilde{b}_{j,n}\right)\otimes u_{n+1}\\
&\quad-u_{n+1}\otimes\left(\sum_{j\in H_{n+1}}\ell_j^2\widetilde{b}_{j,n}\right)
+\left(\sum_{j\in H_{n+1}}\ell_j^2\right)u_{n+1}^{\otimes2}\\
&=V_n-P_n\otimes u_{n+1}-u_{n+1}\otimes P_n+q_nu_{n+1}^{\otimes2}.
\end{aligned} j ∈ H n + 1 ∑ ℓ j 2 ( b j , n − u n + 1 ) ⊗ 2 = j ∈ H n + 1 ∑ ℓ j 2 b j , n ⊗ 2 − j ∈ H n + 1 ∑ ℓ j 2 b j , n ⊗ u n + 1 − u n + 1 ⊗ j ∈ H n + 1 ∑ ℓ j 2 b j , n + j ∈ H n + 1 ∑ ℓ j 2 u n + 1 ⊗ 2 = V n − P n ⊗ u n + 1 − u n + 1 ⊗ P n + q n u n + 1 ⊗ 2 .
For z ∈ T x ~ n M z\in T_{\widetilde{x}_n}\mathcal{M} z ∈ T x n M , the transport identity ( Λ n + 1 z ) ⊗ 2 = Λ n + 1 ∘ z ⊗ 2 ∘ Λ n + 1 ∗ (\Lambda_{n+1}z)^{\otimes2}=\Lambda_{n+1}\circ z^{\otimes2}\circ\Lambda_{n+1}^{*} ( Λ n + 1 z ) ⊗ 2 = Λ n + 1 ∘ z ⊗ 2 ∘ Λ n + 1 ∗ implies
V n + 1 = ∑ j ∈ H n + 1 ℓ j 2 b ~ j , n + 1 ⊗ 2 = Λ n + 1 ∘ { ∑ j ∈ H n + 1 ℓ j 2 ( b ~ j , n − u n + 1 ) ⊗ 2 } ∘ Λ n + 1 ∗ = Λ n + 1 ∘ { V n − P n ⊗ u n + 1 − u n + 1 ⊗ P n + q n u n + 1 ⊗ 2 } ∘ Λ n + 1 ∗ . \begin{aligned}
V_{n+1}=\sum_{j\in H_{n+1}}\ell_j^2\widetilde{b}_{j,n+1}^{\otimes2}
=\Lambda_{n+1}\circ\left\{\sum_{j\in H_{n+1}}\ell_j^2(\widetilde{b}_{j,n}-u_{n+1})^{\otimes2}\right\}\circ\Lambda_{n+1}^{*} \\
&=\Lambda_{n+1}\circ\left\{V_n-P_n\otimes u_{n+1}-u_{n+1}\otimes P_n+q_nu_{n+1}^{\otimes2}\right\}\circ\Lambda_{n+1}^{*}.
\end{aligned} V n + 1 = j ∈ H n + 1 ∑ ℓ j 2 b j , n + 1 ⊗ 2 = Λ n + 1 ∘ ⎩ ⎨ ⎧ j ∈ H n + 1 ∑ ℓ j 2 ( b j , n − u n + 1 ) ⊗ 2 ⎭ ⎬ ⎫ ∘ Λ n + 1 ∗ = Λ n + 1 ∘ { V n − P n ⊗ u n + 1 − u n + 1 ⊗ P n + q n u n + 1 ⊗ 2 } ∘ Λ n + 1 ∗ .
The updates of P n P_n P n and V n V_n V n require only the aggregate sums, allowing the estimator to be updated online without revisiting individual retained batches.
Finally, at each time n ≥ 1 n\ge1 n ≥ 1 , the output combines the retained-batch sum V n V_n V n with the contribution of B n B_n B n . The decomposition E n = H n ∪ { n } E_n=H_n\cup\{n\} E n = H n ∪ { n } and the identity v n = s n + ℓ n v_n=s_n+\ell_n v n = s n + ℓ n imply
V n + ℓ n 2 b ~ n , n ⊗ 2 s n + ℓ n = 1 v n ∑ j ∈ E n ℓ j 2 b ~ j , n ⊗ 2 = Σ ~ n . \frac{V_n+\ell_n^2\widetilde{b}_{n,n}^{\otimes2}}{s_n+\ell_n}
=\frac{1}{v_n}\sum_{j\in E_n}\ell_j^2\widetilde{b}_{j,n}^{\otimes2}
=\widetilde{\Sigma}_n. s n + ℓ n V n + ℓ n 2 b n , n ⊗ 2 = v n 1 j ∈ E n ∑ ℓ j 2 b j , n ⊗ 2 = Σ n .
Thus the output is exactly the estimator in (14) ; updating the weighted sums introduces no further approximation beyond the batch vectors b ~ j , n \widetilde{b}_{j,n} b j , n already used in its definition. We evaluate the numerical performance of the estimators in § 5 .
Convergence rates
In this section, we establish convergence guarantees for the OL and NOL estimators defined in (15) and (17), respectively. Recall that the estimators act on T x ~ n M T_{\widetilde{x}_n}\mathcal{M} T x n M , whereas Σ \Sigma Σ acts on T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M . We therefore use parallel transport to express the estimators on T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M for comparison with Σ \Sigma Σ . This transport is used only in the analysis; computing the estimators requires neither parallel transport nor knowledge of x ⋆ x_\star x ⋆ . We now state our main convergence guarantee.
Theorem 3.1 (Convergence rates of the OL and NOL estimators). Consider the RSGD iterates { x n } n \{x_n\}_n { x n } n and recursive averages { x ~ n } n \{\widetilde{x}_n\}_n { x n } n in (8), with step sizes γ n = γ n − α \gamma_n=\gamma n^{-\alpha} γ n = γ n − α for γ > 0 \gamma>0 γ > 0 and α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2,1) α ∈ ( 1/2 , 1 ) . Under Assumptions 2.1–2.4, let Σ ~ n , O L \widetilde{\Sigma}_{n,\mathrm{OL}} Σ n , OL and Σ ~ n , N O L \widetilde{\Sigma}_{n,\mathrm{NOL}} Σ n , NOL be the estimators defined in (15) and (17), respectively,
with batch starts a 1 = 1 a_1=1 a 1 = 1 and a m = ⌊ c m β ⌋ a_m=\lfloor cm^\beta\rfloor a m = ⌊ c m β ⌋ for m ≥ 2 m\ge2 m ≥ 2 , where c ≥ 1 c\ge1 c ≥ 1 and β > 1 / ( 1 − α ) \beta>1/(1-\alpha) β > 1/ ( 1 − α ) . Then, for ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } , we have
E ∥ [ ( Γ x ⋆ x ~ n ) − 1 ∘ Σ ~ n , ⋄ ∘ Γ x ⋆ x ~ n ] − Σ ∥ ≤ n − 1 / ( 2 β ) + n ( α − 1 ) / 2 + 1 / ( 2 β ) . \mathbb{E}\left\Vert\left[(\Gamma_{x_\star}^{\tilde{x}_n})^{-1}\circ\widetilde{\Sigma}_{n,\diamond}\circ\Gamma_{x_\star}^{\tilde{x}_n}\right]-\Sigma\right\Vert\le n^{-1/(2\beta)}+n^{(\alpha-1)/2+1/(2\beta)}. E [ ( Γ x ⋆ x ~ n ) − 1 ∘ Σ n , ⋄ ∘ Γ x ⋆ x ~ n ] − Σ ≤ n − 1/ ( 2 β ) + n ( α − 1 ) /2 + 1/ ( 2 β ) .
We defer the proof for OL and NOL to Appendices D. Theorem 3.1 shows that, as n → ∞ n\to\infty n → ∞ , the OL and NOL estimators converge to the asymptotic covariance Σ \Sigma Σ of averaged RSGD. The convergence rate is associated with the step-size parameter α \alpha α and the batch parameter β \beta β . For each fixed α \alpha α , optimizing the choice of β \beta β leads to the following corollary.
Corollary 3.2 (Choice of batch exponent). Consider the RSGD iterates { x n } n \{x_n\}_n { x n } n and recursive averages { x ~ n } n \{\tilde{x}_n\}_n { x ~ n } n in (8) , with step sizes γ n = γ n − α \gamma_n=\gamma n^{-\alpha} γ n = γ n − α for γ > 0 \gamma>0 γ > 0 and α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2,1) α ∈ ( 1/2 , 1 ) . Under Assumptions 2.1–2.4, let Σ ~ n , O L \widetilde{\Sigma}_{n,\mathrm{OL}} Σ n , OL and Σ ~ n , N O L \widetilde{\Sigma}_{n,\mathrm{NOL}} Σ n , NOL be the estimators defined in (15) and (17) , respectively, with batch starts a 1 = 1 a_1=1 a 1 = 1 and a m = ⌊ c m β ⌋ a_m=\lfloor cm^\beta\rfloor a m = ⌊ c m β ⌋ for m ≥ 2 m\ge2 m ≥ 2 , where c ≥ 1 c\ge1 c ≥ 1 and β = 2 / ( 1 − α ) \beta=2/(1-\alpha) β = 2/ ( 1 − α ) . Then, for ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } , we have
E ∥ [ ( Γ x ⋆ x ~ n ) − 1 ∘ Σ ~ n , ⋄ ∘ Γ x ⋆ x ~ n ] − Σ ∥ ≤ n − ( 1 − α ) / 4 . \mathbb{E}\left\Vert\left[(\Gamma_{x_\star}^{\tilde{x}_n})^{-1}\circ\widetilde{\Sigma}_{n,\diamond}\circ\Gamma_{x_\star}^{\tilde{x}_n}\right]-\Sigma\right\Vert\le n^{-(1-\alpha)/4}. E [ ( Γ x ⋆ x ~ n ) − 1 ∘ Σ n , ⋄ ∘ Γ x ⋆ x ~ n ] − Σ ≤ n − ( 1 − α ) /4 .
The convergence rates in Theorem 3.1 and Corollary 3.2 match those established by Zhu et al. [46 ] for the corresponding Euclidean batch-means estimators. Thus, under our assumptions, the manifold geometry does not slow the convergence of the covariance estimators. In particular, the optimized rate approaches n − 1 / 8 n^{-1/8} n − 1/8 as α \alpha α approaches 1 / 2 1/2 1/2 from above.
Proof strategy
In this section, we provide a brief overview of the proof of Theorem 3.1 for the OL estimator; the NOL argument follows the same strategy. The key is to separate the additional errors introduced by the manifold geometry from the batch-means estimation error already present in Euclidean space. To this end, we introduce the following notation and intermediate quantities.
Recall the errors of the RSGD iterates, Δ k = R x ⋆ − 1 ( x k ) \Delta_k=R_{x_\star}^{-1}(x_k) Δ k = R x ⋆ − 1 ( x k ) , and the error of the averaged iterate, Δ ~ n = R x ⋆ − 1 ( x ~ n ) \widetilde{\Delta}_n=R_{x_\star}^{-1}(\tilde{x}_n) Δ n = R x ⋆ − 1 ( x ~ n ) . For each batch B j B_j B j in (9) and its recursive mean c j c_j c j in (11) , define
Δ ˉ n : = 1 n ∑ k = 1 n Δ k , Δ ˉ j B : = 1 ℓ j ∑ k = t j j Δ k , Δ ~ j B : = R x ⋆ − 1 ( c j ) , (18) \bar{\Delta}_n:=\frac{1}{n}\sum_{k=1}^{n}\Delta_k,\qquad
\bar{\Delta}_j^B:=\frac{1}{\ell_j}\sum_{k=t_j}^{j}\Delta_k,\qquad
\widetilde{\Delta}_j^B:=R_{x_\star}^{-1}(c_j), \tag*{(18)} Δ ˉ n := n 1 k = 1 ∑ n Δ k , Δ ˉ j B := ℓ j 1 k = t j ∑ j Δ k , Δ j B := R x ⋆ − 1 ( c j ) , (18)
where Δ ˉ n \bar{\Delta}_n Δ ˉ n and Δ ˉ j B \bar{\Delta}_j^B Δ ˉ j B are arithmetic averages of the tangent vectors Δ k \Delta_k Δ k , while Δ ~ n \widetilde{\Delta}_n Δ n and Δ ~ j B \widetilde{\Delta}_j^B Δ j B are obtained by averaging on the manifold before mapping the averages to T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M . Using the batch indices E n O L E_n^{\mathrm{OL}} E n OL and normalizing factor v n v_n v n from (15) , we define four operators on this fixed tangent space:
Σ ^ n , O L ( 0 ) : = 1 v n ∑ j ∈ E n O L ℓ j 2 ( ( Γ x ⋆ x ~ n ) − 1 b ~ j , n ) ⊗ 2 , \widehat{\Sigma}_{n,\mathrm{OL}}^{(0)}:=\frac{1}{v_n}\sum_{j\in E_n^{\mathrm{OL}}}\ell_j^2\left((\Gamma_{x_\star}^{\tilde{x}_n})^{-1}\tilde{b}_{j,n}\right)^{\otimes2}, Σ n , OL ( 0 ) := v n 1 j ∈ E n OL ∑ ℓ j 2 ( ( Γ x ⋆ x ~ n ) − 1 b ~ j , n ) ⊗ 2 ,
Σ ^ n , O L ( 1 ) : = 1 v n ∑ j ∈ E n O L ℓ j 2 ( ( Γ x ⋆ x ~ n ) − 1 b j , n ) ⊗ 2 , \widehat{\Sigma}_{n,\mathrm{OL}}^{(1)}:=\frac{1}{v_n}\sum_{j\in E_n^{\mathrm{OL}}}\ell_j^2\left((\Gamma_{x_\star}^{\tilde{x}_n})^{-1}b_{j,n}\right)^{\otimes2}, Σ n , OL ( 1 ) := v n 1 j ∈ E n OL ∑ ℓ j 2 ( ( Γ x ⋆ x ~ n ) − 1 b j , n ) ⊗ 2 ,
Σ ^ n , O L ( 2 ) : = 1 v n ∑ j ∈ E n O L ℓ j 2 ( Δ ~ j B − Δ ~ n ) ⊗ 2 , \widehat{\Sigma}_{n,\mathrm{OL}}^{(2)}:=\frac{1}{v_n}\sum_{j\in E_n^{\mathrm{OL}}}\ell_j^2\left(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n\right)^{\otimes2}, Σ n , OL ( 2 ) := v n 1 j ∈ E n OL ∑ ℓ j 2 ( Δ j B − Δ n ) ⊗ 2 ,
Σ ^ n , O L ( 3 ) : = 1 v n ∑ j ∈ E n O L ℓ j 2 ( Δ ˉ j B − Δ ˉ n ) ⊗ 2 . (19) \widehat{\Sigma}_{n,\mathrm{OL}}^{(3)}:=\frac{1}{v_n}\sum_{j\in E_n^{\mathrm{OL}}}\ell_j^2\left(\bar{\Delta}_j^B-\bar{\Delta}_n\right)^{\otimes2}. \tag*{(19)} Σ n , OL ( 3 ) := v n 1 j ∈ E n OL ∑ ℓ j 2 ( Δ ˉ j B − Δ ˉ n ) ⊗ 2 . (19)
In (19) , Σ ~ ^ n , O L ( 0 ) = ( Γ x n x ~ n ) − 1 ∘ Σ ~ n , O L ∘ Γ x n x ~ n \widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(0)} = (\Gamma_{x_n}^{\widetilde{x}_n})^{-1} \circ\widetilde{\Sigma}_{n,\mathrm{OL}} \circ\Gamma_{x_n}^{\widetilde{x}_n} Σ n , OL ( 0 ) = ( Γ x n x n ) − 1 ∘ Σ n , OL ∘ Γ x n x n is the OL estimator transported to T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M . The last operator, Σ ~ ^ n , O L ( 3 ) \widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(3)} Σ n , OL ( 3 ) , has the Euclidean OL batch-means form applied to the vectors Δ k \Delta_k Δ k . The triangle inequality separates the total error into the three differences between successive operators and the remaining batch-means error:
E ∥ Σ ~ ^ n , O L ( 0 ) − Σ ∥ ≤ E ∥ Σ ~ ^ n , O L ( 0 ) − Σ ~ ^ n , O L ( 1 ) ∥ + E ∥ Σ ~ ^ n , O L ( 1 ) − Σ ~ ^ n , O L ( 2 ) ∥ + E ∥ Σ ~ ^ n , O L ( 2 ) − Σ ~ ^ n , O L ( 3 ) ∥ + E ∥ Σ ~ ^ n , O L ( 3 ) − Σ ∥ . \mathbb{E}\lVert\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(0)}-\Sigma\rVert\leq\mathbb{E}\lVert\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(0)}-\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(1)}\rVert+ \mathbb{E}\lVert\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(1)}-\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(2)}\rVert
+\mathbb{E}\lVert\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(2)}-\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(3)}\rVert+ \mathbb{E}\lVert\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(3)}-\Sigma\rVert. E ∥ Σ n , OL ( 0 ) − Σ ∥ ≤ E ∥ Σ n , OL ( 0 ) − Σ n , OL ( 1 ) ∥ + E ∥ Σ n , OL ( 1 ) − Σ n , OL ( 2 ) ∥ + E ∥ Σ n , OL ( 2 ) − Σ n , OL ( 3 ) ∥ + E ∥ Σ n , OL ( 3 ) − Σ ∥ .
The first three terms measure the additional errors introduced by the manifold and vanish in the standard Euclidean setting. The last term is the batch-means estimation error that must still be controlled even in Euclidean space. We begin with the first three terms.
From Σ ~ ^ n , O L ( 0 ) \widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(0)} Σ n , OL ( 0 ) to Σ ~ ^ n , O L ( 1 ) \widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(1)} Σ n , OL ( 1 ) , we replace the recursively updated vector b ~ j , n \widetilde{b}_{j,n} b j , n with the exact deviation b j , n = R x ~ n − 1 ( c j ) b_{j,n}=R_{\widetilde{x}_n}^{-1}(c_j) b j , n = R x n − 1 ( c j ) . This isolates the approximation introduced by the online update in (13) . Proposition D.9 bounds the resulting covariance error.
From Σ ~ ^ n , O L ( 1 ) \widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(1)} Σ n , OL ( 1 ) to Σ ~ ^ n , O L ( 2 ) \widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(2)} Σ n , OL ( 2 ) , we compare two ways of expressing the deviation of c j c_j c j from x ~ n \widetilde{x}_n x n in the fixed tangent space T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M :
( Γ x ~ n x ⋆ ) − 1 b j , n = ( Γ x ~ n x ⋆ ) − 1 R x ~ n − 1 ( c j ) , Δ ~ j B − Δ ~ n = R x ⋆ − 1 ( c j ) − R x ⋆ − 1 ( x ~ n ) . (\Gamma_{\widetilde{x}_n}^{x_\star})^{-1}b_{j,n}=(\Gamma_{\widetilde{x}_n}^{x_\star})^{-1}R_{\widetilde{x}_n}^{-1}(c_j), \qquad\widetilde{\Delta}_j^B-\widetilde{\Delta}_n=R_{x_\star}^{-1}(c_j)-R_{x_\star}^{-1}(\widetilde{x}_n). ( Γ x n x ⋆ ) − 1 b j , n = ( Γ x n x ⋆ ) − 1 R x n − 1 ( c j ) , Δ j B − Δ n = R x ⋆ − 1 ( c j ) − R x ⋆ − 1 ( x n ) .
The first forms the deviation using the inverse retraction at x ~ n \widetilde{x}_n x n and then transports it to T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M . The second maps both averages to T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M using R x ⋆ − 1 R_{x_\star}^{-1} R x ⋆ − 1 before subtracting the resulting vectors. Proposition D.7 bounds the resulting covariance error.
From Σ ~ ^ n , O L ( 2 ) \widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(2)} Σ n , OL ( 2 ) to Σ ~ ^ n , O L ( 3 ) \widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(3)} Σ n , OL ( 3 ) , we replace the mapped manifold averages ( Δ ~ j B , Δ ~ n ) (\widetilde{\Delta}_j^B,\widetilde{\Delta}_n) ( Δ j B , Δ n ) with the arithmetic averages ( Δ ‾ j B , Δ ‾ n ) (\overline{\Delta}_j^B,\overline{\Delta}_n) ( Δ j B , Δ n ) in (18) . This isolates the difference between averaging on the manifold and averaging the tangent vectors Δ k \Delta_k Δ k . Proposition D.5 bounds the resulting covariance error.
The propositions above bound each of these three errors by a constant multiple of n 1 / 2 − 1 / ( 2 β ) − α n^{1/2-1/(2\beta)-\alpha} n 1/2 − 1/ ( 2 β ) − α . It remains to control the batch-means error E ∥ Σ ~ ^ n , O L ( 3 ) − Σ ∥ \mathbb{E}\lVert\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(3)}-\Sigma\rVert E ∥ Σ n , OL ( 3 ) − Σ ∥ . We modify the Euclidean batch-means analysis of Zhu et al. [46 ] and show in Proposition D.4 that
E ∥ Σ ~ ^ n , O L ( 3 ) − Σ ∥ ≲ n − 1 / ( 2 β ) + n ( α − 1 ) / 2 + 1 / ( 2 β ) . \mathbb{E}\lVert\widehat{\widetilde{\Sigma}}_{n,\mathrm{OL}}^{(3)}-\Sigma\rVert\lesssim n^{-1/(2\beta)}+n^{(\alpha-1)/2+1/(2\beta)}. E ∥ Σ n , OL ( 3 ) − Σ ∥ ≲ n − 1/ ( 2 β ) + n ( α − 1 ) /2 + 1/ ( 2 β ) .
Since α > 1 / 2 \alpha>1/2 α > 1/2 , we have n 1 / 2 − 1 / ( 2 β ) − α = o ( n − 1 / ( 2 β ) ) n^{1/2-1/(2\beta)-\alpha}=o(n^{-1/(2\beta)}) n 1/2 − 1/ ( 2 β ) − α = o ( n − 1/ ( 2 β ) ) , so the errors introduced by the manifold do not affect the convergence rate. Combining the four bounds establishes the OL result in Theorem 3.1 .
Online statistical inference on manifolds
In this section, we combine the asymptotic normality result in Theorem 2.5 with our covariance estimator to perform online statistical inference on manifolds. We first present a general strategy for constructing confidence intervals and regions when operating on a Riemannian manifold, then specialize it to streaming principal component analysis on a Grassmann manifold.
Confidence intervals and regions
We first construct a confidence interval for a scalar quantity h ( x ⋆ ) h(x_\star) h ( x ⋆ ) , where h : M → R h:\mathcal{M}\to\mathbb{R} h : M → R is a smooth function. For example, in Euclidean setting where M = R d \mathcal{M}=\mathbb{R}^d M = R d , the function h ( x ) = e i T x h(x)=e_i^{\mathsf{T}}x h ( x ) = e i T x selects the i i i -th coordinate of x ⋆ x_\star x ⋆ . We estimate h ( x ⋆ ) h(x_\star) h ( x ⋆ ) by h ( x ~ n ) h(\widetilde{x}_n) h ( x n ) . For a prescribed level 1 − δ 1-\delta 1 − δ , with δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , we seek an interval whose probability of containing h ( x ⋆ ) h(x_\star) h ( x ⋆ ) approaches 1 − δ 1-\delta 1 − δ as n n n increases.
For either covariance estimator Σ ~ n , ⋄ \widetilde{\Sigma}_{n,\diamond} Σ n , ⋄ , with ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } , we estimate the asymptotic variance by
σ ^ n , h , ⋄ 2 : = ⟨ grad h ( x ~ n ) , Σ ~ n , ⋄ grad h ( x ~ n ) ⟩ x ~ n . (20) \widehat{\sigma}_{n,h,\diamond}^{2}:=\langle\operatorname{grad}h(\widetilde{x}_n),\widetilde{\Sigma}_{n,\diamond}\operatorname{grad}h(\widetilde{x}_n)\rangle_{\widetilde{x}_n}. \tag*{(20)} σ n , h , ⋄ 2 := ⟨ grad h ( x n ) , Σ n , ⋄ grad h ( x n ) ⟩ x n . (20)
The gradient accounts for the first-order change in h h h along tangent directions. Let σ ^ n , h , ⋄ ≥ 0 \widehat{\sigma}_{n,h,\diamond}\geq0 σ n , h , ⋄ ≥ 0 denote the square root of this variance estimate. Then σ ^ n , h , ⋄ / n \widehat{\sigma}_{n,h,\diamond}/\sqrt{n} σ n , h , ⋄ / n is the estimated standard error of h ( x ~ n ) h(\widetilde{x}_n) h ( x n ) , which determines the width of the confidence interval
I n , h , ⋄ ( 1 − δ ) : = [ h ( x ~ n ) − z 1 − δ / 2 σ ^ n , h , ⋄ n , h ( x ~ n ) + z 1 − δ / 2 σ ^ n , h , ⋄ n ] , (21) \mathcal{I}_{n,h,\diamond}(1-\delta):=\left[h(\widetilde{x}_n)-z_{1-\delta/2}\frac{\widehat{\sigma}_{n,h,\diamond}}{\sqrt{n}},\ h(\widetilde{x}_n)+z_{1-\delta/2}\frac{\widehat{\sigma}_{n,h,\diamond}}{\sqrt{n}}\right], \tag*{(21)} I n , h , ⋄ ( 1 − δ ) := [ h ( x n ) − z 1 − δ /2 n σ n , h , ⋄ , h ( x n ) + z 1 − δ /2 n σ n , h , ⋄ ] , (21)
where z 1 − δ / 2 z_{1-\delta/2} z 1 − δ /2 is the ( 1 − δ / 2 ) (1-\delta/2) ( 1 − δ /2 ) -quantile of the standard normal distribution.
For joint inference, we seek a confidence region for x ⋆ x_\star x ⋆ itself, consisting of points on M \mathcal{M} M . Recall that Σ ~ n , ⋄ \widetilde{\Sigma}_{n,\diamond} Σ n , ⋄ acts on T x ~ n M T_{\widetilde{x}_n}\mathcal{M} T x n M . We therefore first construct an ellipsoid of tangent vectors around 0 x ~ n 0_{\widetilde{x}_n} 0 x n , with its shape determined by the covariance estimate. To obtain a region on the manifold, we apply the retraction R x ~ n R_{\widetilde{x}_n} R x n to this ellipsoid. When Σ ~ n , ⋄ \widetilde{\Sigma}_{n,\diamond} Σ n , ⋄ is positive definite, the two sets are
E n , ⋄ ( 1 − δ ) : = { v ∈ T x ~ n M : n ⟨ v , Σ ~ n , ⋄ − 1 v ⟩ x ~ n ≤ χ d , 1 − δ 2 } , \mathcal{E}_{n,\diamond}(1-\delta):=\left\{v\in T_{\widetilde{x}_n}\mathcal{M}:n\langle v,\widetilde{\Sigma}_{n,\diamond}^{-1}v\rangle_{\widetilde{x}_n}\leq\chi_{d,1-\delta}^{2}\right\}, E n , ⋄ ( 1 − δ ) := { v ∈ T x n M : n ⟨ v , Σ n , ⋄ − 1 v ⟩ x n ≤ χ d , 1 − δ 2 } ,
C n , ⋄ ( 1 − δ ) : = R x ~ n { E n , ⋄ ( 1 − δ ) } . (22) \mathcal{C}_{n,\diamond}(1-\delta):=R_{\widetilde{x}_n}\left\{\mathcal{E}_{n,\diamond}(1-\delta)\right\}. \tag*{(22)} C n , ⋄ ( 1 − δ ) := R x n { E n , ⋄ ( 1 − δ ) } . (22)
Here, d = dim M d=\dim\mathcal{M} d = dim M and χ d , 1 − δ 2 \chi_{d,1-\delta}^{2} χ d , 1 − δ 2 is the ( 1 − δ ) (1-\delta) ( 1 − δ ) -quantile of the chi-squared distribution with d d d degrees of freedom. Figure 2 illustrates this construction.
Figure 2. Construction of a confidence region on the manifold. The retraction R x ~ n R_{\widetilde{x}_n} R x n maps the tangent ellipsoid E n , ⋄ ( 1 − δ ) \mathcal{E}_{n,\diamond}(1-\delta) E n , ⋄ ( 1 − δ ) to the shaded region C n , ⋄ ( 1 − δ ) \mathcal{C}_{n,\diamond}(1-\delta) C n , ⋄ ( 1 − δ ) , and its center 0 x ~ n 0_{\widetilde{x}_n} 0 x n to x ~ n \widetilde{x}_n x n . The right panel separates the tangent space from the manifold to show corresponding points, connected by the light gray dashed lines. The estimator subscript ⋄ \diamond ⋄ is omitted in the figure.
In Euclidean space, with R x ( v ) = x + v R_x(v)=x+v R x ( v ) = x + v and h ( x ) = e i ⊤ x h(x)=e_i^\top x h ( x ) = e i ⊤ x , the interval in (21) and the region in (22) reduce to the coordinatewise interval and ellipsoidal region in Zhu et al. [46 ] .
The following proposition combines the asymptotic normality in Theorem 2.5 with the covariance consistency in Theorem 3.1 to establish asymptotically correct coverage for both constructions. The proof is deferred to Appendix § 12 .
Proposition 4.1. Let M \mathcal{M} M be a Riemannian manifold with dimension d d d . Under the conditions of Theorem 3.1 , fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and let h : M → R h:\mathcal{M}\to\mathbb{R} h : M → R be continuously differentiable on an open neighborhood of X \mathcal{X} X , with grad h ( x ⋆ ) ≠ 0 x ⋆ \operatorname{grad}h(x_\star)\ne0_{x_\star} grad h ( x ⋆ ) = 0 x ⋆ . For ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } , consider the interval in (21) and the region in (22) , setting C n , ⋄ ( 1 − δ ) = { x ~ n } C_{n,\diamond}(1-\delta)=\{\tilde{x}_n\} C n , ⋄ ( 1 − δ ) = { x ~ n } if Σ ~ n , ⋄ \widetilde{\Sigma}_{n,\diamond} Σ n , ⋄ is singular. Then, as n → ∞ n\to\infty n → ∞ ,
P { h ( x ⋆ ) ∈ I n , h , ⋄ ( 1 − δ ) } ⟶ 1 − δ , \mathbb{P}\{h(x_\star)\in I_{n,h,\diamond}(1-\delta)\}\longrightarrow1-\delta, P { h ( x ⋆ ) ∈ I n , h , ⋄ ( 1 − δ )} ⟶ 1 − δ , P { x ⋆ ∈ C n , ⋄ ( 1 − δ ) } ⟶ 1 − δ . \mathbb{P}\{x_\star\in C_{n,\diamond}(1-\delta)\}\longrightarrow1-\delta. P { x ⋆ ∈ C n , ⋄ ( 1 − δ )} ⟶ 1 − δ .
We next specialize these inference procedures to streaming principal component analysis.
Streaming principal component analysis (PCA)
Streaming k k k -PCA estimates a leading k k k -dimensional principal subspace from sequentially observed data ([23 , 4 ] ). Given independent and identically distributed symmetric positive semidefinite matrices H n ∈ R p × p H_n\in\mathbb{R}^{p\times p} H n ∈ R p × p , the goal is to estimate the subspace spanned by the eigenvectors corresponding to the k k k largest eigenvalues of H = E H n H=\mathbb{E}H_n H = E H n , where 1 ≤ k < p 1\le k<p 1 ≤ k < p . We focus on the standard outer-product case H n = ξ n ξ n ⊤ H_n=\xi_n\xi_n^\top H n = ξ n ξ n ⊤ , where { ξ n } n ≥ 1 \{\xi_n\}_{n\ge1} { ξ n } n ≥ 1 are independent samples from a common distribution on R p \mathbb{R}^p R p with finite second moment, so H = E ( ξ n ξ n ⊤ ) H=\mathbb{E}(\xi_n\xi_n^\top) H = E ( ξ n ξ n ⊤ ) . Write H = Q diag ( λ 1 , … , λ p ) Q ⊤ H=Q\operatorname{diag}(\lambda_1,\ldots,\lambda_p)Q^\top H = Q diag ( λ 1 , … , λ p ) Q ⊤ , where Q Q Q is orthogonal and λ 1 ≥ ⋯ ≥ λ p ≥ 0 \lambda_1\ge\cdots\ge\lambda_p\ge0 λ 1 ≥ ⋯ ≥ λ p ≥ 0 , and let Q 1 ∈ R p × k Q_1\in\mathbb{R}^{p\times k} Q 1 ∈ R p × k contain the first k k k columns of Q Q Q . The columns of Q 1 Q_1 Q 1 span the subspace we seek to estimate. Different orthonormal bases can represent the same subspace, and the particular choice does not matter. So, this problem can be formulated on the Grassmann manifold G p , k G_{p,k} G p , k , whose points are the k k k -dimensional subspaces of R p \mathbb{R}^p R p ; see Bonnabel [7 ] and Tripuraneni et al. [38 ] .
The dimension of G p , k G_{p,k} G p , k is k ( p − k ) k(p-k) k ( p − k ) . We write a point in G p , k G_{p,k} G p , k as [ X ] [X] [ X ] , where X ∈ R p × k X\in\mathbb{R}^{p\times k} X ∈ R p × k has orthonormal columns and spans [ X ] [X] [ X ] . The orthogonal projection of an observation ξ n \xi_n ξ n onto [ X ] [X] [ X ] is X X ⊤ ξ n XX^\top\xi_n X X ⊤ ξ n , with squared length ∥ X ⊤ ξ n ∥ 2 \lVert X^\top\xi_n\rVert^2 ∥ X ⊤ ξ n ∥ 2 . PCA seeks the subspace that minimizes the expected squared norm of the residual ξ n − X X ⊤ ξ n \xi_n-XX^\top\xi_n ξ n − X X ⊤ ξ n . Equivalently, it maximizes E ∥ X ⊤ ξ n ∥ 2 = tr ( X ⊤ H X ) \mathbb{E}\lVert X^\top\xi_n\rVert^2=\operatorname{tr}(X^\top H X) E ∥ X ⊤ ξ n ∥ 2 = tr ( X ⊤ H X ) . The corresponding population optimization problem is
min [ X ] ∈ G p , k F ( [ X ] ) ≔ − 1 2 tr ( X ⊤ H X ) = E [ − 1 2 ∥ X ⊤ ξ n ∥ 2 ] . (23) \min_{[X]\in G_{p,k}} F([X])\coloneqq-\frac{1}{2}\operatorname{tr}(X^\top H X)=\mathbb{E}\left[-\frac{1}{2}\lVert X^\top\xi_n\rVert^2\right]. \tag*{(23)} [ X ] ∈ G p , k min F ([ X ]) : = − 2 1 tr ( X ⊤ H X ) = E [ − 2 1 ∥ X ⊤ ξ n ∥ 2 ] . (23)
Under the eigengap condition λ k > λ k + 1 \lambda_k>\lambda_{k+1} λ k > λ k + 1 , the leading principal subspace is unique, and the minimizer is x ⋆ = [ Q 1 ] x_\star=[Q_1] x ⋆ = [ Q 1 ] .
We now specify the geometric maps and Riemannian gradient needed to implement RSGD and Algorithm 1 on G p , k G_{p,k} G p , k .
Tangent space and Riemannian metric. We equip G p , k G_{p,k} G p , k with its standard Riemannian metric; see Boumal [9 ] . For a fixed orthonormal representative X X X , the tangent space, inner product, and orthogonal projection Π X : R p × k → T [ X ] G p , k \Pi_X:\mathbb{R}^{p\times k}\to T_{[X]}G_{p,k} Π X : R p × k → T [ X ] G p , k have the following matrix representations:
T [ X ] G p , k = { V ∈ R p × k : X ⊤ V = 0 } , T_{[X]}G_{p,k}=\{V\in\mathbb{R}^{p\times k}:X^\top V=0\}, T [ X ] G p , k = { V ∈ R p × k : X ⊤ V = 0 } ,
⟨ V , W ⟩ [ X ] = tr ( V ⊤ W ) , Π X Z = ( I p − X X ⊤ ) Z . (24) \langle V,W\rangle_{[X]}=\operatorname{tr}(V^\top W),\qquad\Pi_X Z=(I_p-XX^\top)Z. \tag*{(24)} ⟨ V , W ⟩ [ X ] = tr ( V ⊤ W ) , Π X Z = ( I p − X X ⊤ ) Z . (24)
Here V , W ∈ T [ X ] G p , k V,W \in\mathrm{T}_{[X]}G_{p,k} V , W ∈ T [ X ] G p , k and Z ∈ R p × k Z \in\mathbb{R}^{p \times k} Z ∈ R p × k . Although the matrix representations depend on the choice of X X X , the inner product of the corresponding tangent vectors does not.
Retraction and inverse retraction. For a tangent vector V ∈ T [ X ] G p , k V \in\mathrm{T}_{[X]}G_{p,k} V ∈ T [ X ] G p , k , the columns of X + V X+V X + V need not be orthonormal. We orthonormalize them using the polar retraction R [ X ] : T [ X ] G p , k → G p , k R_{[X]}:\mathrm{T}_{[X]}G_{p,k} \to G_{p,k} R [ X ] : T [ X ] G p , k → G p , k :
R [ X ] ( V ) = [ ( X + V ) ( I k + V T V ) − 1 / 2 ] . (25) R_{[X]}(V)=\left[(X+V)(I_k+V^{\mathsf T}V)^{-1/2}\right]. \tag*{(25)} R [ X ] ( V ) = [ ( X + V ) ( I k + V T V ) − 1/2 ] . (25)
The factor ( I k + V T V ) − 1 / 2 (I_k+V^{\mathsf T}V)^{-1/2} ( I k + V T V ) − 1/2 changes the basis but not the span of X + V X+V X + V . As established in Boumal [9 ] , this retraction is second order.
The averaging and covariance updates also require the local inverse retraction. For a nearby subspace [ Y ] [Y] [ Y ] with orthonormal representative Y 0 Y_0 Y 0 , write X T Y 0 = K O X^{\mathsf T}Y_0=KO X T Y 0 = K O in polar form, where K ≻ 0 K \succ0 K ≻ 0 and O O O is orthogonal. The factor K K K is unique because it is the unique positive definite square root of X T Y 0 Y 0 T X X^{\mathsf T}Y_0Y_0^{\mathsf T}X X T Y 0 Y 0 T X . With the aligned representative Y = Y 0 O T Y=Y_0O^{\mathsf T} Y = Y 0 O T , the inverse retraction takes the form
R [ X ] − 1 ( [ Y ] ) = Y K − 1 − X . R_{[X]}^{-1}([Y])=YK^{-1}-X. R [ X ] − 1 ([ Y ]) = Y K − 1 − X .
Vector transport and its adjoint. The covariance updates require transporting tangent vectors between the tangent spaces of successive averages. With the aligned representatives X , Y X,Y X , Y and the matrix K K K specified in item 2, the vector transport in (6) and its adjoint are represented by
Λ [ X ] [ Y ] ( W ) = Π Y W K , ( Λ [ X ] [ Y ] ) ∗ ( Z ) = Π X Z K . \Lambda_{[X]}^{[Y]}(W)=\Pi_YWK,\qquad\left(\Lambda_{[X]}^{[Y]}\right)^*(Z)=\Pi_XZK. Λ [ X ] [ Y ] ( W ) = Π Y W K , ( Λ [ X ] [ Y ] ) ∗ ( Z ) = Π X Z K .
Here W ∈ T [ X ] G p , k W \in\mathrm{T}_{[X]}G_{p,k} W ∈ T [ X ] G p , k and Z ∈ T [ Y ] G p , k Z \in\mathrm{T}_{[Y]}G_{p,k} Z ∈ T [ Y ] G p , k . The transport maps tangent vectors at [ X ] [X] [ X ] to tangent vectors at [ Y ] [Y] [ Y ] , while its adjoint acts in the opposite direction.
Riemannian gradient. For the sample loss f ( [ X ] , ξ ) : = − ∥ X T ξ ∥ 2 / 2 f([X],\xi):=-\lVert X^{\mathsf T}\xi\rVert^2/2 f ([ X ] , ξ ) := − ∥ X T ξ ∥ 2 /2 in (23) , the Euclidean gradient with respect to X X X is − ξ ξ T X -\xi\xi^{\mathsf T}X − ξ ξ T X . Under the metric above, the Riemannian gradient is its projection onto the tangent space: grad f ( [ X ] , ξ ) = − Π X ξ ξ T X \operatorname{grad} f([X],\xi)=-\Pi_X\xi\xi^{\mathsf T}X grad f ([ X ] , ξ ) = − Π X ξ ξ T X .
We write the RSGD iterate and its recursive average as x n = [ X n ] x_n=[X_n] x n = [ X n ] and x ~ n = [ X ~ n ] \widetilde{x}_n=[\widetilde{X}_n] x n = [ X n ] , respectively. With the components above, the RSGD and averaging updates in (8) become
[ X n ] = R [ X n − 1 ] { γ n Π X n − 1 ξ n ξ n T X n − 1 } , [ X ~ n ] = R [ X ~ n − 1 ] { 1 n R [ X ~ n − 1 ] − 1 ( [ X n ] ) } . (26) [X_n]=R_{[X_{n-1}]}\left\{\gamma_n\Pi_{X_{n-1}}\xi_n\xi_n^{\mathsf T}X_{n-1}\right\},\qquad[\widetilde{X}_n]=R_{[\widetilde{X}_{n-1}]}\left\{\frac{1}{n}R^{-1}_{[\widetilde{X}_{n-1}]}([X_n])\right\}. \tag*{(26)} [ X n ] = R [ X n − 1 ] { γ n Π X n − 1 ξ n ξ n T X n − 1 } , [ X n ] = R [ X n − 1 ] { n 1 R [ X n − 1 ] − 1 ([ X n ]) } . (26)
Algorithm 1 uses these iterates and the geometric maps above to compute the OL and NOL covariance estimators on T [ X ~ n ] G p , k \mathrm{T}_{[\widetilde{X}_n]}G_{p,k} T [ X n ] G p , k . We next use these estimates to construct confidence regions for the principal subspace and confidence intervals for its scalar features.
Confidence regions for the principal subspace. To construct a confidence region for x ⋆ = [ Q 1 ] x_\star=[Q_1] x ⋆ = [ Q 1 ] , we form the tangent ellipsoid E n , ⋄ ( 1 − δ ) \mathcal{E}_{n,\diamond}(1-\delta) E n , ⋄ ( 1 − δ ) in (22) at x ~ n = [ X ~ n ] \widetilde{x}_n=[\widetilde{X}_n] x n = [ X n ] , with d = k ( p − k ) d=k(p-k) d = k ( p − k ) . The polar retraction maps this ellipsoid to a set of subspaces. Since the orthonormalization in (25) does not change the column span, the resulting region is
C n , ⋄ ( 1 − δ ) = { span ( X ~ n + V ) : V ∈ E n , ⋄ ( 1 − δ ) } . \mathcal{C}_{n,\diamond}(1-\delta)=\left\{\operatorname{span}(\widetilde{X}_n+V):V\in\mathcal{E}_{n,\diamond}(1-\delta)\right\}. C n , ⋄ ( 1 − δ ) = { span ( X n + V ) : V ∈ E n , ⋄ ( 1 − δ ) } .
Here we represent each tangent vector V V V by a matrix in R p × k \mathbb{R}^{p \times k} R p × k satisfying X ~ n T V = 0 \widetilde{X}_n^{\mathsf{T}}V=0 X n T V = 0 , and span ( X ~ n + V ) \operatorname{span}(\widetilde{X}_n+V) span ( X n + V ) denotes the subspace spanned by the columns of X ~ n + V \widetilde{X}_n+V X n + V . As V V V ranges over the tangent ellipsoid, the subspaces span ( X ~ n + V ) \operatorname{span}(\widetilde{X}_n+V) span ( X n + V ) form the confidence region. This region is independent of the choice of orthonormal representative X ~ n \widetilde{X}_n X n .
Figure 3 illustrates this construction on G 3 , 1 G_{3,1} G 3 , 1 , whose points are lines through the origin in R 3 \mathbb{R}^3 R 3 . As shown in the left panel, each line intersects the unit sphere S 2 S^2 S 2 at two opposite points z z z and − z -z − z , either of which represents the same subspace. For lines near the estimate x ~ n = [ X ~ n ] \widetilde{x}_n=[\widetilde{X}_n] x n = [ X n ] , we choose the representative satisfying X ~ n T z > 0 \widetilde{X}_n^{\mathsf{T}}z>0 X n T z > 0 . This represents each such line by a single point in the hemisphere centered at X ~ n \widetilde{X}_n X n . The right panel shows the confidence-region construction using these representatives. The tangent ellipsoid E n , ⋄ ( 1 − δ ) E_{n,\diamond}(1-\delta) E n , ⋄ ( 1 − δ ) is an ellipse in the tangent plane T X ~ n S 2 T_{\widetilde{X}_n}S^2 T X n S 2 . For each vector v v v in this ellipse, the polar retraction in (25) divides X ~ n + v \widetilde{X}_n+v X n + v by its length to obtain the unit vector z = ( X ~ n + v ) / 1 + ∥ v ∥ 2 z=(\widetilde{X}_n+v)/\sqrt{1+\lVert v\rVert^2} z = ( X n + v ) / 1 + ∥ v ∥ 2 and returns the line [ z ] = [ X ~ n + v ] [z]=[\widetilde{X}_n+v] [ z ] = [ X n + v ] . As v v v ranges over the ellipse, these representatives form the shaded spherical cap. The corresponding lines form the confidence region C n , ⋄ ( 1 − δ ) C_{n,\diamond}(1-\delta) C n , ⋄ ( 1 − δ ) .
Figure 3. Spherical representation of a confidence region on G 3 , 1 G_{3,1} G 3 , 1 . Left: the black and orange lines represent the estimate [ X ~ n ] [\widetilde{X}_n] [ X n ] and a subspace [ z ] [z] [ z ] in the confidence region, respectively. Right: the polar retraction maps each tangent vector v v v to the line represented by z = ( X ~ n + v ) / 1 + ∥ v ∥ 2 z=(\widetilde{X}_n+v)/\sqrt{1+\lVert v\rVert^2} z = ( X n + v ) / 1 + ∥ v ∥ 2 . These unit representatives form the shaded spherical cap depicting the confidence region. The subscript ⋄ \diamond ⋄ is omitted.
Confidence intervals for a scalar feature. To illustrate the scalar confidence-interval construction in § 4.1 , we consider how closely a fixed unit vector q ∈ R p q\in\mathbb{R}^p q ∈ R p aligns with the unknown principal subspace x ⋆ = [ Q 1 ] x_\star=[Q_1] x ⋆ = [ Q 1 ] . Our target is ∥ Q 1 T q ∥ 2 \lVert Q_1^{\mathsf{T}}q\rVert^2 ∥ Q 1 T q ∥ 2 , the squared length of the projection of q q q onto this subspace. We express it as a smooth function on G p , k G_{p,k} G p , k by defining
h q ( [ X ] ) : = q T X X T q = ∥ X T q ∥ 2 . (27) h_q([X]) := q^{\mathsf{T}}XX^{\mathsf{T}}q = \lVert X^{\mathsf{T}}q\rVert^2. \tag*{(27)} h q ([ X ]) := q T X X T q = ∥ X T q ∥ 2 . (27)
Because X X T XX^{\mathsf{T}} X X T is the orthogonal projector onto [ X ] [X] [ X ] , the function h q h_q h q does not depend on the choice of representative X X X . Its value lies in [ 0 , 1 ] [0,1] [ 0 , 1 ] , with larger values indicating closer alignment. The target ∥ Q 1 T q ∥ 2 \lVert Q_1^{\mathsf{T}}q\rVert^2 ∥ Q 1 T q ∥ 2 can be written as h q ( x ⋆ ) h_q(x_\star) h q ( x ⋆ ) , and we estimate it by h q ( [ X ~ n ] ) h_q([\widetilde{X}_n]) h q ([ X n ]) . To quantify the uncertainty of this estimate, we use the Riemannian gradient of h q h_q h q , represented by
grad h q ( [ X ] ) = 2 ( I p − X X T ) q q T X . (28) \operatorname{grad} h_q([X]) = 2(I_p - XX^{\mathsf{T}})qq^{\mathsf{T}}X. \tag*{(28)} grad h q ([ X ]) = 2 ( I p − X X T ) q q T X . (28)
Evaluating (28) at [ X ~ n ] [\widetilde{X}_n] [ X n ] and combining it with either the OL or NOL covariance estimator in (20) yields the estimated asymptotic variance. This determines the width of the confidence interval in (21) , centered at h q ( [ X ~ n ] ) h_q([\widetilde{X}_n]) h q ([ X n ]) . In § 5 , we will evaluate the empirical coverage and length of these intervals through numerical experiments on Gaussian streaming PCA.
Numerical experiments
This section examines the finite-sample performance of the OL and NOL covariance estimators in (15) and (17) using the streaming k k k -PCA model from § 4.2 with Gaussian observations. In this model, the limiting covariance is available in closed form, allowing us to quantify covariance estimation errors and compare the resulting confidence intervals with oracle intervals constructed using the true asymptotic variance. We first describe the experimental setup in § 5.1 , then compare covariance estimation errors in § 5.2 , and finally examine confidence interval coverage and length in § 5.3 . We defer additional covariance estimation and statistical inference results for more settings to Appendix § 13 .
Experimental setup
Recall from § 4.2 that streaming k k k -PCA uses observations ξ n ∈ R p \xi_n \in\mathbb{R}^p ξ n ∈ R p that arrive sequentially to estimate the subspace spanned by the eigenvectors associated with the k k k largest eigenvalues of H = E ( ξ n ξ n T ) H = \mathbb{E}(\xi_n \xi_n^{\mathsf{T}}) H = E ( ξ n ξ n T ) . The parameter space is the Grassmann manifold G p , k G_{p,k} G p , k , whose dimension is d = k ( p − k ) d = k(p-k) d = k ( p − k ) . In our experiments, we generate i.i.d. observations ξ 1 , ξ 2 , … ∼ N ( 0 , H ) \xi_1, \xi_2, \ldots\sim\mathcal{N}(0,H) ξ 1 , ξ 2 , … ∼ N ( 0 , H ) , where H = Q diag ( λ 1 , … , λ p ) Q T H = Q\operatorname{diag}(\lambda_1,\ldots,\lambda_p)Q^{\mathsf{T}} H = Q diag ( λ 1 , … , λ p ) Q T with a fixed orthogonal matrix Q Q Q , and set k = ⌊ p ⌋ k = \lfloor\sqrt{p}\rfloor k = ⌊ p ⌋ in the streaming k k k -PCA problem. Following the choice of eigenvalues in Tripuraneni et al. [38 ] , Section 7, we choose Δ λ > 0 \Delta_\lambda> 0 Δ λ > 0 and set
λ j = j − 1 + Δ λ , 1 ≤ j ≤ k , λ j = ( j − 1 ) − 1 , k < j ≤ p . \lambda_j = j^{-1} + \Delta_\lambda,\quad1 \le j \le k,\qquad\lambda_j = (j-1)^{-1},\quad k < j \le p. λ j = j − 1 + Δ λ , 1 ≤ j ≤ k , λ j = ( j − 1 ) − 1 , k < j ≤ p .
Thus, the eigengap is λ k − λ k + 1 = Δ λ \lambda_k-\lambda_{k+1}=\Delta_\lambda λ k − λ k + 1 = Δ λ . A smaller eigengap makes the principal subspace more sensitive to sampling noise and therefore harder to estimate accurately.
At iteration n n n , we use the new observation ξ n \xi_n ξ n to update the RSGD iterate and its recursive average using (26) , with the polar retraction in (25) . Following Zhu et al. [46 ] , Section 4, we set α = 0.505 \alpha= 0.505 α = 0.505 and use step size γ n = 0.5 n − α \gamma_n = 0.5n^{-\alpha} γ n = 0.5 n − α . The batch starting points are a 1 = 1 a_1 = 1 a 1 = 1 and a m = ⌊ c m β ⌋ a_m = \lfloor cm^\beta\rfloor a m = ⌊ c m β ⌋ for m ≥ 2 m \ge2 m ≥ 2 , with β = 2 / ( 1 − α ) \beta= 2/(1-\alpha) β = 2/ ( 1 − α ) as recommended in Corollary 3.2 . The constant c c c controls the spacing between successive batch starts. We vary c ∈ { 1 , 2 , 5 } c \in\{1,2,5\} c ∈ { 1 , 2 , 5 } for covariance estimation to examine the effect of this choice. Each run contains N = 3 × 10 5 N = 3 \times10^5 N = 3 × 1 0 5 observations. All results are based on 200 independent runs for each setting. For each method and setting, the initialization is fixed across runs.
We first give an explicit representation of the limiting covariance Σ = A − 1 S A − 1 \Sigma= A^{-1}SA^{-1} Σ = A − 1 S A − 1 from Theorem 2.5 . Recall from § 4.2 that x ⋆ = [ Q 1 ] x_\star= [Q_1] x ⋆ = [ Q 1 ] is the population principal subspace, where Q 1 Q_1 Q 1 contains the first k k k columns of Q Q Q and [ Q 1 ] [Q_1] [ Q 1 ] denotes their span. At x ⋆ = [ Q 1 ] x_\star= [Q_1] x ⋆ = [ Q 1 ] , Σ \Sigma Σ acts on the tangent space T x ⋆ G p , k T_{x_\star}G_{p,k} T x ⋆ G p , k , which has dimension d = k ( p − k ) d = k(p-k) d = k ( p − k ) . We therefore represent Σ \Sigma Σ by a d × d d \times d d × d matrix in an orthonormal basis. We choose the orthonormal basis matrices q i e j T q_i e_j^{\mathsf{T}} q i e j T , indexed by $1 \le j \le k < i \le p$, where q i q_i q i is column i i i of Q Q Q and e j e_j e j is the j j j th coordinate vector in R k \mathbb{R}^k R k . Since the columns of Q Q Q and the vectors e j e_j e j are orthonormal, these d d d matrices lie in the tangent space and form an orthonormal basis under the inner product in (24) .
In this basis, we compute the Hessian A = Hess F ( x ⋆ ) A = \operatorname{Hess} F(x_\star) A = Hess F ( x ⋆ ) from the objective in (23) . Differentiating its Riemannian gradient at the matrix representative X = Q 1 X = Q_1 X = Q 1 of x ⋆ x_\star x ⋆ and projecting the result onto T x ⋆ G p , k T_{x_\star}G_{p,k} T x ⋆ G p , k gives A [ V ] = V ( Q 1 T H Q 1 ) − H V A[V] = V(Q_1^{\mathsf{T}}HQ_1) - HV A [ V ] = V ( Q 1 T H Q 1 ) − H V for V ∈ T x ⋆ G p , k V \in T_{x_\star}G_{p,k} V ∈ T x ⋆ G p , k . Since H q i = λ i q i Hq_i = \lambda_i q_i H q i = λ i q i , we have A [ q i e j T ] = ( λ j − λ i ) q i e j T A[q_i e_j^{\mathsf{T}}] = (\lambda_j - \lambda_i)q_i e_j^{\mathsf{T}} A [ q i e j T ] = ( λ j − λ i ) q i e j T . This identity shows that each basis vector is an eigenvector of A A A , so the matrix of A A A is diagonal in this basis, with diagonal entries λ j − λ i \lambda_j - \lambda_i λ j − λ i .
We next compute the gradient-noise covariance S S S in the same basis. The gradient noise grad f ( x ⋆ , ξ n ) \operatorname{grad} f(x_\star,\xi_n) grad f ( x ⋆ , ξ n ) has coordinate − ( q i T ξ n ) ( q j T ξ n ) -(q_i^{\mathsf{T}}\xi_n)(q_j^{\mathsf{T}}\xi_n) − ( q i T ξ n ) ( q j T ξ n ) along q i e j T q_i e_j^{\mathsf{T}} q i e j T . Since Q T ξ n Q^{\mathsf{T}}\xi_n Q T ξ n has independent centered Gaussian entries with variances λ 1 , … , λ p \lambda_1,\ldots,\lambda_p λ 1 , … , λ p , this coordinate has mean zero and variance E [ ( q i T ξ n ) 2 ( q j T ξ n ) 2 ] = λ i λ j \mathbb{E}[(q_i^{\mathsf{T}}\xi_n)^2(q_j^{\mathsf{T}}\xi_n)^2] = \lambda_i\lambda_j E [( q i T ξ n ) 2 ( q j T ξ n ) 2 ] = λ i λ j . For two distinct noise coordinates, at least one centered Gaussian factor appears only once in their product; thus, independence then implies zero covariance between them. Finally, the gradient-noise covariance S S S is diagonal with entries λ i λ j \lambda_i\lambda_j λ i λ j . Substituting the diagonal entries of A A A and S S S into Σ = A − 1 S A − 1 \Sigma= A^{-1}SA^{-1} Σ = A − 1 S A − 1 , we know
Σ = diag ( σ i j 2 ) , σ i j 2 = λ i λ j ( λ j − λ i ) 2 , 1 ≤ j ≤ k < i ≤ p . (29) \Sigma= \operatorname{diag}(\sigma_{ij}^{2}), \qquad\sigma_{ij}^{2} = \frac{\lambda_i\lambda_j}{(\lambda_j-\lambda_i)^2}, \qquad1 \le j \le k < i \le p. \tag*{(29)} Σ = diag ( σ ij 2 ) , σ ij 2 = ( λ j − λ i ) 2 λ i λ j , 1 ≤ j ≤ k < i ≤ p . (29)
Note that this covariance is used only to evaluate estimation errors and construct oracle intervals. It is not used in the OL or NOL updates.
Covariance estimation
We first compare the OL and NOL covariance estimators on G 3 , 1 G_{3,1} G 3 , 1 , G 7 , 2 G_{7,2} G 7 , 2 , and G 10 , 3 G_{10,3} G 10 , 3 , using eigengaps Δ λ ∈ { 1 , 0.25 } \Delta_\lambda\in\{1,0.25\} Δ λ ∈ { 1 , 0.25 } and batch constants c ∈ { 1 , 2 , 5 } c \in\{1,2,5\} c ∈ { 1 , 2 , 5 } . To compare each estimate with the limiting covariance Σ \Sigma Σ in (29) , we parallel transport the estimate from T x ~ n G p , k T_{\tilde{x}_n}G_{p,k} T x ~ n G p , k to T x ⋆ G p , k T_{x_\star}G_{p,k} T x ⋆ G p , k , as in Theorem 3.1 , and compute the operator-norm error in the orthonormal basis described above. Figure 4 plots the logarithm of the average error over 200 runs against log n \log n log n . The black dashed lines have slope − 1 / 8 -1/8 − 1/8 , which is close to the exponent − ( 1 − α ) / 4 = − 0.12375 -(1-\alpha)/4 = -0.12375 − ( 1 − α ) /4 = − 0.12375 in Corollary 3.2 , and are included as reference lines.
Figure 4. Gaussian streaming PCA: log mean covariance error (operator norm) against log sample size. Columns correspond to G 3 , 1 G_{3,1} G 3 , 1 , G 7 , 2 G_{7,2} G 7 , 2 , and G 10 , 3 G_{10,3} G 10 , 3 , and the top and bottom rows use eigengaps 1 and 0.25, respectively. The colored curves compare OL and NOL with c ∈ { 1 , 2 , 5 } c \in\{1, 2, 5\} c ∈ { 1 , 2 , 5 } . Each mean is based on 200 independent runs. The black dashed lines indicate the reference slope − 1 / 8 -1/8 − 1/8 .
Figure 4 shows that the errors of both estimators generally decrease as more observations are received, despite some fluctuations along the trajectories. The overall slopes of the log-log curves are close to the reference slope − 1 / 8 -1/8 − 1/8 , consistent with the convergence rate in Corollary 3.2 . Comparing the two rows of Figure 4 , we also observe larger absolute errors for the smaller eigengap. However, the magnitude of Σ \Sigma Σ in (29) also increases as the eigengap decreases, so the larger absolute errors do not necessarily indicate lower relative accuracy.
We next compare the different choices of c c c and the two covariance estimators. At n = 3 × 10 5 n = 3 \times10^5 n = 3 × 1 0 5 , c = 1 c = 1 c = 1 yields the smallest mean error among the three batch constants for both estimators across all settings considered. Based on these results, we use c = 1 c = 1 c = 1 for both estimators in the confidence interval experiments below.
Statistical inference
We next examine the empirical coverage and mean length of the nominal 95% confidence intervals in (21) . We consider G 10 , 3 G_{10,3} G 10 , 3 and G 20 , 4 G_{20,4} G 20 , 4 , with Δ λ = 0.25 \Delta_\lambda= 0.25 Δ λ = 0.25 and c = 1 c = 1 c = 1 . For each manifold, we draw v ∼ N ( 0 , H ) v \sim\mathcal{N}(0,H) v ∼ N ( 0 , H ) independently of the RSGD streams, set q = v / ∥ v ∥ q = v/\lVert v\rVert q = v / ∥ v ∥ , and hold q q q fixed across runs. Recall from § 4.2 that h q ( x ⋆ ) = ∥ Q 1 T q ∥ 2 h_q(x_\star)=\lVert Q_1^\mathsf{T}q\rVert^2 h q ( x ⋆ ) = ∥ Q 1 T q ∥ 2 in (27) is the squared length of the projection of q q q onto the principal subspace x ⋆ x_\star x ⋆ . We estimate this quantity by h q ( x ~ n ) h_q(\widetilde{x}_n) h q ( x n ) and construct its confidence interval using either the OL or NOL covariance estimate.
For comparison, we also construct oracle intervals by using the true scalar asymptotic variance σ h q 2 = ⟨ grad h q ( x ⋆ ) , Σ grad h q ( x ⋆ ) ⟩ x ⋆ \sigma_{h_q}^2=\langle\operatorname{grad} h_q(x_\star),\Sigma\operatorname{grad} h_q(x_\star)\rangle_{x_\star} σ h q 2 = ⟨ grad h q ( x ⋆ ) , Σ grad h q ( x ⋆ ) ⟩ x ⋆ in the confidence interval formula (21) , where Σ \Sigma Σ is given by (29) . We report the empirical coverage, defined as the fraction of the 200 intervals that contain h q ( x ⋆ ) h_q(x_\star) h q ( x ⋆ ) , together with the mean interval length. Figure 5 displays both quantities as the sample size increases. Table 1 reports the coverage at four sample sizes, with Monte Carlo standard errors p ^ ( 1 − p ^ ) / 200 \sqrt{\widehat{p}(1-\widehat{p})/200} p ( 1 − p ) /200 shown in parentheses, where p ^ \widehat{p} p denotes the empirical coverage.
Figure 5. Gaussian streaming PCA: empirical coverage (top) and mean length (bottom) of 95% confidence intervals for h q ( x ⋆ ) h_q(x_\star) h q ( x ⋆ ) . Columns correspond to G 10 , 3 G_{10,3} G 10 , 3 and G 20 , 4 G_{20,4} G 20 , 4 , with Δ λ = 0.25 \Delta_\lambda= 0.25 Δ λ = 0.25 and c = 1 c = 1 c = 1 . The blue, orange, and purple curves show OL, NOL, and Oracle, respectively, with results based on 200 independent runs. The small plots inside the top panels show the later coverage values, and the red dashed lines mark the nominal coverage level 0.95.
Table 1 (PDF p. 23). Empirical coverage of nominal 95% confidence intervals for h q ( x ⋆ ) h_q(x_\star) h q ( x ⋆ ) in Gaussian streaming PCA, with Δ λ = 0.25 \Delta_\lambda=0.25 Δ λ = 0.25 and c = 1 c=1 c = 1 . Results are based on 200 independent runs, with Monte Carlo standard errors in parentheses. Here d = k ( p − k ) d=k(p-k) d = k ( p − k ) is the dimension of G p , k G_{p,k} G p , k .
Figure 5 shows that OL and NOL achieve similar coverage on both manifolds. For example, at n = 3 × 10 5 n = 3 \times10^5 n = 3 × 1 0 5 , Table 1 shows that the OL and NOL coverage rates are 0.895 and 0.920 on G 10 , 3 G_{10,3} G 10 , 3 , respectively, while both methods attain coverage of 0.895 on G 20 , 4 G_{20,4} G 20 , 4 . Consistent with the observations in Zhu et al. [46 ] , although these coverage rates are below the nominal 95% level, both methods achieve coverage of about 90% at this sample size, indicating reasonable finite-sample performance in these higher-dimensional settings with the small eigengap Δ λ = 0.25 \Delta_\lambda= 0.25 Δ λ = 0.25 . The mean interval lengths decrease as more observations are received and are similar for OL and NOL. Additional experimental results for different choices of dimension and eigengap are reported in Appendix § 13 .
Conclusion and future directions
In this paper, we studied asymptotic covariance estimation for averaged Riemannian stochastic gradient descent (RSGD). Specifically, we extended the Euclidean online batch-means estimator of Zhu et al. [46 ] to Riemannian manifolds, developing both overlapping (OL) and non-overlapping (NOL) versions. To our knowledge, these are the first covariance estimators of this type for averaged RSGD. Both estimators use only the RSGD iterates and require neither Riemannian Hessian calculations nor a prespecified terminal sample size. Our analysis further shows that the geometric errors induced by the manifold structure are of smaller order than the leading batch-means error. Consequently, under standard assumptions, an appropriate choice of the batch-size control sequence yields the convergence rate n − ( 1 − α ) / 4 n^{-(1-\alpha)/4} n − ( 1 − α ) /4 , matching that of the Euclidean estimator of Zhu et al. [46 ] . Combining this consistency result with the asymptotic normality established by Tripuraneni et al. [38 ] , we obtained asymptotically valid confidence intervals for smooth scalar functions of x ⋆ x_\star x ⋆ and joint confidence regions for x ⋆ x_\star x ⋆ . We illustrated these online inference procedures through streaming PCA on Grassmann manifolds. In this setting, our Gaussian experiments support the consistency of the proposed covariance estimators and demonstrate their use in constructing confidence intervals in higher-dimensional settings.
One direction for future research is to develop online covariance estimators for averaged RSGD with faster convergence rates. For example, extending the bias-reduced estimator of Wei et al. [40 ] to averaged RSGD could yield a faster rate, as the Euclidean estimator achieves n − ( 1 − α ) / 2 log n n^{-(1-\alpha)/2}\sqrt{\log n} n − ( 1 − α ) /2 log n under stronger moment conditions. As α ↓ 1 / 2 \alpha\downarrow1/2 α ↓ 1/2 , the polynomial factor in this rate approaches n − 1 / 4 n^{-1/4} n − 1/4 , compared with n − 1 / 8 n^{-1/8} n − 1/8 in our bound. Faster covariance estimation may also improve the finite-sample performance of the resulting inference procedures. Another direction is to establish lower bounds for covariance estimation based on RSGD iterates and characterize the minimax optimal convergence rate on Riemannian manifolds.
Disclosure of AI usage
The methodology, results, and proofs were developed by the authors. All writing was done by the authors. OpenAI’s GPT 5.6 Sol and some earlier models were used to assist with the proofs of some technical results (Lemmas A.6 and D.8), to help write code for the simulations, and to proofread the final draft of the paper.
Supplemental Materials: Online Covariance Estimation for Averaged Stochastic Gradient Descent on Riemannian Manifolds
The supplementary material is organized as follows. In Appendix § 8 , we present some useful technical lemmas. Appendices § 9 and § 10 provide estimates for geometric maps and the RSGD iterates, respectively. In Appendix § 11 , we prove Theorem 3.1 , our main theorem for the OL and NOL covariance estimators, and derive Corollary 3.2 . Appendix § 12 proves the statistical inference results in Proposition 4.1 . Additional numerical experiments are presented in Appendix § 13 . Throughout the appendices, C C C denotes a finite positive constant whose value may change between displays.
Technical lemmas
Lemma A.1. Let A ∈ R d × d A \in\mathbb{R}^{d \times d} A ∈ R d × d be symmetric and positive definite. For i ∈ Z ≥ 0 i \in\mathbb{Z}_{\ge0} i ∈ Z ≥ 0 , define Y i i : = I d Y_i^i := I_d Y i i := I d and, for j > i j > i j > i , set
Y i j : = ∏ k = i + 1 j ( I d − γ k A ) , Y_i^j := \prod_{k=i+1}^{j}(I_d-\gamma_k A), Y i j := k = i + 1 ∏ j ( I d − γ k A ) , where γ k : = γ k − α \gamma_k := \gamma k^{-\alpha} γ k := γ k − α , γ > 0 \gamma> 0 γ > 0 and α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2,1) α ∈ ( 1/2 , 1 ) . There exists a constant C > 0 C > 0 C > 0 such that, for all 0 ≤ i ≤ j 0 \le i \le j 0 ≤ i ≤ j ,
∥ Y i j ∥ ≤ C exp ( − γ λ min ( A ) ∑ k = i + 1 j k − α ) ≤ C exp [ − γ λ min ( A ) 1 − α ( ( j + 1 ) 1 − α − ( i + 1 ) 1 − α ) ] . \left\|Y_i^j\right\| \le C \exp\left(-\gamma\lambda_{\min}(A)\sum_{k=i+1}^{j} k^{-\alpha}\right) \le C\exp\left[-\frac{\gamma\lambda_{\min}(A)}{1-\alpha}\left((j+1)^{1-\alpha}-(i+1)^{1-\alpha}\right)\right]. Y i j ≤ C exp ( − γ λ m i n ( A ) k = i + 1 ∑ j k − α ) ≤ C exp [ − 1 − α γ λ m i n ( A ) ( ( j + 1 ) 1 − α − ( i + 1 ) 1 − α ) ] .
Proof. Set η : = γ λ min ( A ) \eta:= \gamma\lambda_{\min}(A) η := γ λ m i n ( A ) and choose k A : = ⌈ ( γ λ max ( A ) ) 1 / α ⌉ + 1 k_A := \left\lceil(\gamma\lambda_{\max}(A))^{1/\alpha}\right\rceil+1 k A := ⌈ ( γ λ m a x ( A ) ) 1/ α ⌉ + 1 . Let λ 1 , … , λ d \lambda_1,\ldots,\lambda_d λ 1 , … , λ d denote the eigenvalues of A A A . Then, for every k ≥ k A k \ge k_A k ≥ k A ,
∥ I d − γ k A ∥ = max 1 ≤ r ≤ d ∣ 1 − γ k − α λ r ∣ ≤ 1 − η k − α ≤ e − η k − α . \left\|I_d-\gamma_k A\right\|=\max_{1\le r\le d}\left|1-\gamma k^{-\alpha}\lambda_r\right|\le1-\eta k^{-\alpha}\le e^{-\eta k^{-\alpha}}. ∥ I d − γ k A ∥ = 1 ≤ r ≤ d max 1 − γ k − α λ r ≤ 1 − η k − α ≤ e − η k − α .
The first inequality uses 0 ≤ γ k λ r ≤ 1 0 \le\gamma_k\lambda_r \le1 0 ≤ γ k λ r ≤ 1 and λ r ≥ λ min ( A ) \lambda_r \ge\lambda_{\min}(A) λ r ≥ λ m i n ( A ) ; the last uses 1 − x ≤ e − x 1-x \le e^{-x} 1 − x ≤ e − x for x ∈ [ 0 , 1 ] x \in[0,1] x ∈ [ 0 , 1 ] . Set
C : = ( ∏ k = 1 k A − 1 max { 1 , ∥ I d − γ k A ∥ } ) exp ( η ∑ k = 1 k A − 1 k − α ) . C:=\left(\prod_{k=1}^{k_A-1}\max\{1,\left\|I_d-\gamma_kA\right\|\}\right)\exp\left(\eta\sum_{k=1}^{k_A-1}k^{-\alpha}\right). C := ( k = 1 ∏ k A − 1 max { 1 , ∥ I d − γ k A ∥ } ) exp ( η k = 1 ∑ k A − 1 k − α ) .
By submultiplicativity, for j > i j>i j > i , we have
∥ Y i j ∥ ≤ ∏ k = i + 1 j ∥ I d − γ k A ∥ = exp ( − η ∑ k = i + 1 j k − α ) ∏ k = i + 1 j ( e η k − α ∥ I d − γ k A ∥ ) ≤ C exp ( − η ∑ k = i + 1 j k − α ) . \left\|Y_i^j\right\|\le\prod_{k=i+1}^{j}\left\|I_d-\gamma_kA\right\|=\exp\left(-\eta\sum_{k=i+1}^{j}k^{-\alpha}\right)\prod_{k=i+1}^{j}\left(e^{\eta k^{-\alpha}}\left\|I_d-\gamma_kA\right\|\right)\le C\exp\left(-\eta\sum_{k=i+1}^{j}k^{-\alpha}\right). Y i j ≤ k = i + 1 ∏ j ∥ I d − γ k A ∥ = exp ( − η k = i + 1 ∑ j k − α ) k = i + 1 ∏ j ( e η k − α ∥ I d − γ k A ∥ ) ≤ C exp ( − η k = i + 1 ∑ j k − α ) .
For j = i j=i j = i , the bound follows from C ≥ 1 C\ge1 C ≥ 1 . Finally, integral comparison yields
∑ k = i + 1 j k − α ≥ ∫ i + 1 j + 1 x − α d x = ( j + 1 ) 1 − α − ( i + 1 ) 1 − α 1 − α . \sum_{k=i+1}^{j}k^{-\alpha}\ge\int_{i+1}^{j+1}x^{-\alpha}\,dx=\frac{(j+1)^{1-\alpha}-(i+1)^{1-\alpha}}{1-\alpha}. k = i + 1 ∑ j k − α ≥ ∫ i + 1 j + 1 x − α d x = 1 − α ( j + 1 ) 1 − α − ( i + 1 ) 1 − α .
Substituting this lower bound into the exponential proves the result. ■
Lemma A.2. Under the assumptions of Lemma A.1, for each fixed q > 0 q>0 q > 0 , there exists C ( q ) < ∞ C(q)<\infty C ( q ) < ∞ such that the matrix products Y i j Y_i^j Y i j defined there satisfy
∑ i = 1 n ∥ Y i n ∥ i − q ≤ C ( q ) n α − q , n ≥ 1. \sum_{i=1}^{n}\left\|Y_i^n\right\|^{i-q}\le C(q)n^{\alpha-q},\qquad n\ge1. i = 1 ∑ n ∥ Y i n ∥ i − q ≤ C ( q ) n α − q , n ≥ 1.
Proof. Set η : = γ λ min ( A ) > 0 \eta:= \gamma\lambda_{\min}(A) > 0 η := γ λ m i n ( A ) > 0 . The case n = 1 n = 1 n = 1 is covered by choosing C ( q ) ≥ 1 C(q) \ge1 C ( q ) ≥ 1 . For n ≥ 2 n \ge2 n ≥ 2 , split the sum at ⌊ n / 2 ⌋ \lfloor n/2\rfloor ⌊ n /2 ⌋ . If i ≤ ⌊ n / 2 ⌋ i \le\lfloor n/2\rfloor i ≤ ⌊ n /2 ⌋ , then ∑ k = i + 1 n k − α ≥ ( n − i ) n − α ≥ n 1 − α / 2 \sum_{k=i+1}^{n} k^{-\alpha} \ge(n-i)n^{-\alpha} \ge n^{1-\alpha}/2 ∑ k = i + 1 n k − α ≥ ( n − i ) n − α ≥ n 1 − α /2 and i − q ≤ 1 i^{-q} \le1 i − q ≤ 1 , so
∑ i = 1 ⌊ n / 2 ⌋ ∥ Y i n ∥ i − q ≤ C n e − η n 1 − α / 2 . (30) \sum_{i=1}^{\lfloor n/2\rfloor} \lVert Y_i^n\rVert i^{-q} \le C n e^{-\eta n^{1-\alpha}/2}. \tag*{(30)} i = 1 ∑ ⌊ n /2 ⌋ ∥ Y i n ∥ i − q ≤ C n e − η n 1 − α /2 . (30)
Dividing by n α − q n^{\alpha-q} n α − q leaves the bounded factor z ( 1 − α + q ) / ( 1 − α ) e − η z / 2 z^{(1-\alpha+q)/(1-\alpha)}e^{-\eta z/2} z ( 1 − α + q ) / ( 1 − α ) e − η z /2 , where z = n 1 − α ≥ 1 z = n^{1-\alpha} \ge1 z = n 1 − α ≥ 1 , so
∑ i = 1 ⌊ n / 2 ⌋ ∥ Y i n ∥ i − q ≤ C n α − q . \sum_{i=1}^{\lfloor n/2\rfloor} \lVert Y_i^n\rVert i^{-q} \le C n^{\alpha-q}. i = 1 ∑ ⌊ n /2 ⌋ ∥ Y i n ∥ i − q ≤ C n α − q .
If i > ⌊ n / 2 ⌋ i > \lfloor n/2\rfloor i > ⌊ n /2 ⌋ , then i − q ≤ 2 q n − q i^{-q} \le2^q n^{-q} i − q ≤ 2 q n − q and ∑ k = i + 1 n k − α ≥ ( n − i ) n − α \sum_{k=i+1}^{n} k^{-\alpha} \ge(n-i)n^{-\alpha} ∑ k = i + 1 n k − α ≥ ( n − i ) n − α . Extending the nonnegative sum to a geometric series, we obtain
∑ i = ⌊ n / 2 ⌋ + 1 n ∥ Y i n ∥ i − q ≤ C 2 q n − q ∑ r = 0 ∞ e − η r n − α = C 2 q n − q 1 − e − η n − α ≤ C 2 q 1 − e − η n α − q . (31) \sum_{i=\lfloor n/2\rfloor+1}^{n} \lVert Y_i^n\rVert i^{-q}
\le C2^q n^{-q}\sum_{r=0}^{\infty}e^{-\eta r n^{-\alpha}}
= \frac{C2^q n^{-q}}{1-e^{-\eta n^{-\alpha}}}
\le\frac{C2^q}{1-e^{-\eta}}n^{\alpha-q}. \tag*{(31)} i = ⌊ n /2 ⌋ + 1 ∑ n ∥ Y i n ∥ i − q ≤ C 2 q n − q r = 0 ∑ ∞ e − η r n − α = 1 − e − η n − α C 2 q n − q ≤ 1 − e − η C 2 q n α − q . (31)
The last inequality uses 1 − e − η n − α ≥ n − α ( 1 − e − η ) 1-e^{-\eta n^{-\alpha}} \ge n^{-\alpha}(1-e^{-\eta}) 1 − e − η n − α ≥ n − α ( 1 − e − η ) , by concavity of x ↦ 1 − e − η x x\mapsto1-e^{-\eta x} x ↦ 1 − e − η x on [ 0 , 1 ] [0,1] [ 0 , 1 ] . Combining the two parts proves the claim. ■
Lemma A.3. Under the assumptions of Lemma A.1, let ( F n ) n ≥ 0 (\mathcal{F}_n)_{n\ge0} ( F n ) n ≥ 0 be a filtration. Let U 0 , ζ 1 , ζ 2 , … ∈ R d U_0,\zeta_1,\zeta_2,\ldots\in\mathbb{R}^d U 0 , ζ 1 , ζ 2 , … ∈ R d be random vectors such that U 0 U_0 U 0 is F 0 \mathcal{F}_0 F 0 -measurable and ζ n \zeta_n ζ n is F n \mathcal{F}_n F n -measurable for every n ≥ 1 n \ge1 n ≥ 1 . Assume E ∥ U 0 ∥ 2 < ∞ \mathbb{E}\lVert U_0\rVert^2 < \infty E ∥ U 0 ∥ 2 < ∞ , E [ ζ n ∣ F n − 1 ] = 0 \mathbb{E}[\zeta_n\mid\mathcal{F}_{n-1}] = 0 E [ ζ n ∣ F n − 1 ] = 0 , and E ∥ ζ n ∥ 2 ≤ K ζ < ∞ \mathbb{E}\lVert\zeta_n\rVert^2 \le K_\zeta< \infty E ∥ ζ n ∥ 2 ≤ K ζ < ∞ for every n ≥ 1 n \ge1 n ≥ 1 . Consider
U n = ( I d − γ n A ) U n − 1 + γ n ζ n , γ n = γ n − α . U_n = (I_d-\gamma_n A)U_{n-1}+\gamma_n\zeta_n,\qquad\gamma_n=\gamma n^{-\alpha}. U n = ( I d − γ n A ) U n − 1 + γ n ζ n , γ n = γ n − α .
Then there exists a constant C > 0 C > 0 C > 0 depending on E ∥ U 0 ∥ 2 \mathbb{E}\lVert U_0\rVert^2 E ∥ U 0 ∥ 2 such that, for every n ≥ 1 n \ge1 n ≥ 1 ,
E ∥ U n ∥ 2 ≤ C γ n . \mathbb{E}\lVert U_n\rVert^2 \le C\gamma_n. E ∥ U n ∥ 2 ≤ C γ n .
Proof. With Y i n Y_i^n Y i n from Lemma A.1, iterating the recursion yields
U n = Y 0 n U 0 + ∑ i = 1 n Y i n γ i ζ i . U_n=Y_0^nU_0+\sum_{i=1}^{n}Y_i^n\gamma_i\zeta_i. U n = Y 0 n U 0 + i = 1 ∑ n Y i n γ i ζ i .
For i < j i<j i < j , the i i i th summand is F j − 1 \mathcal{F}_{j-1} F j − 1 -measurable and E ( ζ j ∣ F j − 1 ) = 0 \mathbb{E}(\zeta_j\mid\mathcal{F}_{j-1})=0 E ( ζ j ∣ F j − 1 ) = 0 , so
E ⟨ Y i n γ i ζ i , Y j n γ j ζ j ⟩ = E ⟨ Y i n γ i ζ i , Y j n γ j E ( ζ j ∣ F j − 1 ) ⟩ = 0. \mathbb{E}\langle Y_i^n\gamma_i\zeta_i,Y_j^n\gamma_j\zeta_j\rangle
=\mathbb{E}\langle Y_i^n\gamma_i\zeta_i,Y_j^n\gamma_j\mathbb{E}(\zeta_j\mid\mathcal{F}_{j-1})\rangle=0. E ⟨ Y i n γ i ζ i , Y j n γ j ζ j ⟩ = E ⟨ Y i n γ i ζ i , Y j n γ j E ( ζ j ∣ F j − 1 )⟩ = 0.
The initial term is also orthogonal to each increment. Thus, using E ∥ ζ i ∥ 2 ≤ K ζ \mathbb{E}\lVert\zeta_i\rVert^2\le K_\zeta E ∥ ζ i ∥ 2 ≤ K ζ ,
E ∥ U n ∥ 2 ≤ ∥ Y 0 n ∥ 2 E ∥ U 0 ∥ 2 + K ζ ∑ i = 1 n γ i 2 ∥ Y i n ∥ 2 . \mathbb{E}\lVert U_n\rVert^2\le\lVert Y_0^n\rVert^2\mathbb{E}\lVert U_0\rVert^2+K_\zeta\sum_{i=1}^{n}\gamma_i^2\lVert Y_i^n\rVert^2. E ∥ U n ∥ 2 ≤ ∥ Y 0 n ∥ 2 E ∥ U 0 ∥ 2 + K ζ i = 1 ∑ n γ i 2 ∥ Y i n ∥ 2 .
Since ∑ k = 1 n k − α ≥ n 1 − α \sum_{k=1}^{n}k^{-\alpha}\ge n^{1-\alpha} ∑ k = 1 n k − α ≥ n 1 − α , (A.1) bounds the first term by C 2 e − 2 γ λ min ( A ) n 1 − α E ∥ U 0 ∥ 2 C^2e^{-2\gamma\lambda_{\min}(A)n^{1-\alpha}}\mathbb{E}\lVert U_0\rVert^2 C 2 e − 2 γ λ m i n ( A ) n 1 − α E ∥ U 0 ∥ 2 . For the second term, we have
∑ i = 1 n γ i 2 ∥ Y i n ∥ 2 ≤ γ 2 C ∑ i = 1 n ∥ Y i n ∥ i − 2 α . (32) \sum_{i=1}^{n}\gamma_i^2\lVert Y_i^n\rVert^2\le\gamma^2C\sum_{i=1}^{n}\lVert Y_i^n\rVert i^{-2\alpha}. \tag*{(32)} i = 1 ∑ n γ i 2 ∥ Y i n ∥ 2 ≤ γ 2 C i = 1 ∑ n ∥ Y i n ∥ i − 2 α . (32)
Applying (A.2) with q = 2 α q = 2\alpha q = 2 α and absorbing fixed factors into C C C , we obtain
E ∥ U n ∥ 2 ≤ C e − 2 γ λ min ( A ) n 1 − α E ∥ U 0 ∥ 2 + C K ζ n − α . \mathbb{E}\lVert U_n\rVert^2 \le C e^{-2\gamma\lambda_{\min}(A)n^{1-\alpha}}\mathbb{E}\lVert U_0\rVert^2 + CK\zeta n^{-\alpha}. E ∥ U n ∥ 2 ≤ C e − 2 γ λ m i n ( A ) n 1 − α E ∥ U 0 ∥ 2 + C K ζ n − α .
Since sup z ≥ 1 z α / ( 1 − α ) e − 2 γ λ min ( A ) z < ∞ \sup_{z\ge1} z^{\alpha/(1-\alpha)}e^{-2\gamma\lambda_{\min}(A)z}<\infty sup z ≥ 1 z α / ( 1 − α ) e − 2 γ λ m i n ( A ) z < ∞ , setting z = n 1 − α z=n^{1-\alpha} z = n 1 − α bounds the first term by C n − α E ∥ U 0 ∥ 2 Cn^{-\alpha}\mathbb{E}\lVert U_0\rVert^2 C n − α E ∥ U 0 ∥ 2 . This proves the claim. ■
Lemma A.4. Under the assumptions of Lemma A.3, let ( U n ) n ≥ 0 (U_n)_{n\ge0} ( U n ) n ≥ 0 be the sequence defined there. For 1 ≤ s ≤ t 1\le s\le t 1 ≤ s ≤ t , define
r s , t U : = U t − U s − 1 + A − 1 ( γ s − 1 U s − 1 − γ t − 1 U t + ∑ k = s t − 1 ( γ k + 1 − 1 − γ k − 1 ) U k ) . r^U_{s,t}:=U_t-U_{s-1}+A^{-1}\left(\gamma_s^{-1}U_{s-1}-\gamma_t^{-1}U_t+\sum_{k=s}^{t-1}(\gamma_{k+1}^{-1}-\gamma_k^{-1})U_k\right). r s , t U := U t − U s − 1 + A − 1 ( γ s − 1 U s − 1 − γ t − 1 U t + k = s ∑ t − 1 ( γ k + 1 − 1 − γ k − 1 ) U k ) . Here and below, the sum from s s s to t − 1 t-1 t − 1 is understood to be zero when s = t s=t s = t . Then there exists a constant C > 0 C>0 C > 0 depending on E ∥ U 0 ∥ 2 \mathbb{E}\lVert U_0\rVert^2 E ∥ U 0 ∥ 2 such that, for every pair of integers 1 ≤ s ≤ t 1\le s\le t 1 ≤ s ≤ t ,
E ∥ r s , t U ∥ 2 ≤ C t α . \mathbb{E}\lVert r^U_{s,t}\rVert^2\le Ct^\alpha. E ∥ r s , t U ∥ 2 ≤ C t α .
Proof. Note that the recursion implies U k − 1 = A − 1 ζ k + A − 1 γ k − 1 ( U k − 1 − U k ) U_{k-1}=A^{-1}\zeta_k+A^{-1}\gamma_k^{-1}(U_{k-1}-U_k) U k − 1 = A − 1 ζ k + A − 1 γ k − 1 ( U k − 1 − U k ) . Since ∑ k = s t U k = ∑ k = s t U k − 1 + U t − U s − 1 \sum_{k=s}^t U_k=\sum_{k=s}^t U_{k-1}+U_t-U_{s-1} ∑ k = s t U k = ∑ k = s t U k − 1 + U t − U s − 1 , discrete summation by parts yields
∑ k = s t γ k − 1 ( U k − 1 − U k ) = γ s − 1 U s − 1 − γ t − 1 U t + ∑ k = s t − 1 ( γ k + 1 − 1 − γ k − 1 ) U k . \sum_{k=s}^{t}\gamma_k^{-1}(U_{k-1}-U_k)=\gamma_s^{-1}U_{s-1}-\gamma_t^{-1}U_t+\sum_{k=s}^{t-1}(\gamma_{k+1}^{-1}-\gamma_k^{-1})U_k. k = s ∑ t γ k − 1 ( U k − 1 − U k ) = γ s − 1 U s − 1 − γ t − 1 U t + k = s ∑ t − 1 ( γ k + 1 − 1 − γ k − 1 ) U k .
This also holds for s = t s=t s = t , with an empty sum. Substitution yields
∑ k = s t U k = A − 1 ∑ k = s t ζ k + A − 1 ∑ k = s t γ k − 1 ( U k − 1 − U k ) + U t − U s − 1 = A − 1 ∑ k = s t ζ k + r s , t U . \sum_{k=s}^{t}U_k=A^{-1}\sum_{k=s}^{t}\zeta_k+A^{-1}\sum_{k=s}^{t}\gamma_k^{-1}(U_{k-1}-U_k)+U_t-U_{s-1}=A^{-1}\sum_{k=s}^{t}\zeta_k+r^U_{s,t}. k = s ∑ t U k = A − 1 k = s ∑ t ζ k + A − 1 k = s ∑ t γ k − 1 ( U k − 1 − U k ) + U t − U s − 1 = A − 1 k = s ∑ t ζ k + r s , t U .
By Lemma A.3, including k = 0 k=0 k = 0 by enlarging C C C , we have
( E ∥ U k ∥ 2 ) 1 / 2 ≤ C ( k + 1 ) − α / 2 , k ≥ 0. \left(\mathbb{E}\lVert U_k\rVert^2\right)^{1/2}\le C(k+1)^{-\alpha/2},\qquad k\ge0. ( E ∥ U k ∥ 2 ) 1/2 ≤ C ( k + 1 ) − α /2 , k ≥ 0.
Also, concavity of x ↦ x α x\mapsto x^\alpha x ↦ x α implies
0 ≤ γ k + 1 − 1 − γ k − 1 ≤ α γ k − 1 k α − 1 . 0\le\gamma_{k+1}^{-1}-\gamma_k^{-1}\le\alpha\gamma_k^{-1}k^{\alpha-1}. 0 ≤ γ k + 1 − 1 − γ k − 1 ≤ α γ k − 1 k α − 1 .
Minkowski’s inequality and these two bounds yield
( E ∥ r s , t U ∥ 2 ) 1 / 2 ≤ ( E ∥ U t ∥ 2 ) 1 / 2 + ( E ∥ U s − 1 ∥ 2 ) 1 / 2 + ∥ A − 1 ∥ ( γ s − 1 ( E ∥ U s − 1 ∥ 2 ) 1 / 2 + γ t − 1 ( E ∥ U t ∥ 2 ) 1 / 2 + ∑ k = s t − 1 ( γ k + 1 − 1 − γ k − 1 ) ( E ∥ U k ∥ 2 ) 1 / 2 ) \left(\mathbb{E}\lVert r^U_{s,t}\rVert^2\right)^{1/2}\le\left(\mathbb{E}\lVert U_t\rVert^2\right)^{1/2}+\left(\mathbb{E}\lVert U_{s-1}\rVert^2\right)^{1/2}+\lVert A^{-1}\rVert\left(\gamma_s^{-1}\left(\mathbb{E}\lVert U_{s-1}\rVert^2\right)^{1/2}+\gamma_t^{-1}\left(\mathbb{E}\lVert U_t\rVert^2\right)^{1/2}+\sum_{k=s}^{t-1}(\gamma_{k+1}^{-1}-\gamma_k^{-1})\left(\mathbb{E}\lVert U_k\rVert^2\right)^{1/2}\right) ( E ∥ r s , t U ∥ 2 ) 1/2 ≤ ( E ∥ U t ∥ 2 ) 1/2 + ( E ∥ U s − 1 ∥ 2 ) 1/2 + ∥ A − 1 ∥ ( γ s − 1 ( E ∥ U s − 1 ∥ 2 ) 1/2 + γ t − 1 ( E ∥ U t ∥ 2 ) 1/2 + k = s ∑ t − 1 ( γ k + 1 − 1 − γ k − 1 ) ( E ∥ U k ∥ 2 ) 1/2 )
≤ 2 C + ∥ A − 1 ∥ γ − 1 C ( s α / 2 + t α / 2 + α ∑ k = s t − 1 k α / 2 − 1 ) . \le2C+\lVert A^{-1}\rVert\gamma^{-1}C\left(s^{\alpha/2}+t^{\alpha/2}+\alpha\sum_{k=s}^{t-1}k^{\alpha/2-1}\right). ≤ 2 C + ∥ A − 1 ∥ γ − 1 C ( s α /2 + t α /2 + α k = s ∑ t − 1 k α /2 − 1 ) .
Since x ↦ x α / 2 − 1 x \mapsto x^{\alpha/2-1} x ↦ x α /2 − 1 is decreasing,
∑ k = s t − 1 k α / 2 − 1 ≤ 1 + ∫ s t x α / 2 − 1 d x ≤ 1 + 2 α t α / 2 . \sum_{k=s}^{t-1} k^{\alpha/2-1} \le1+\int_s^t x^{\alpha/2-1}\,dx \le1+\frac{2}{\alpha}t^{\alpha/2}. k = s ∑ t − 1 k α /2 − 1 ≤ 1 + ∫ s t x α /2 − 1 d x ≤ 1 + α 2 t α /2 .
Using 1 ≤ s ≤ t 1\le s\le t 1 ≤ s ≤ t in the preceding two displays, we obtain
( E ∥ r s , t U ∥ 2 ) 1 / 2 ≤ C t α / 2 . \left(\mathbb{E}\lVert r^U_{s,t}\rVert^2\right)^{1/2}\le Ct^{\alpha/2}. ( E ∥ r s , t U ∥ 2 ) 1/2 ≤ C t α /2 .
Squaring proves the claim. ■
The following inequality compares two finite weighted sums of outer products.
Lemma A.5. Let m ≥ 1 m\ge1 m ≥ 1 be an integer, let a 1 , … , a m ≥ 0 a_1,\ldots,a_m\ge0 a 1 , … , a m ≥ 0 and D > 0 D>0 D > 0 be deterministic, and let σ i , e i ∈ R d \sigma_i,e_i\in\mathbb{R}^d σ i , e i ∈ R d , 1 ≤ i ≤ m 1\le i\le m 1 ≤ i ≤ m , be square-integrable random vectors. Define
Q : = D − 1 ∑ i = 1 m a i ( σ i + e i ) ⊗ 2 , Q 0 : = D − 1 ∑ i = 1 m a i σ i ⊗ 2 , δ : = D − 1 ∑ i = 1 m a i E ∥ e i ∥ 2 . Q:=D^{-1}\sum_{i=1}^m a_i(\sigma_i+e_i)^{\otimes2},\qquad Q_0:=D^{-1}\sum_{i=1}^m a_i\sigma_i^{\otimes2},\qquad\delta:=D^{-1}\sum_{i=1}^m a_i\mathbb{E}\lVert e_i\rVert^2. Q := D − 1 i = 1 ∑ m a i ( σ i + e i ) ⊗ 2 , Q 0 := D − 1 i = 1 ∑ m a i σ i ⊗ 2 , δ := D − 1 i = 1 ∑ m a i E ∥ e i ∥ 2 . Then
E ∥ Q − Q 0 ∥ ≤ 2 ( E ∥ Q 0 ∥ ) 1 / 2 δ 1 / 2 + δ . (33) \mathbb{E}\lVert Q-Q_0\rVert\le2(\mathbb{E}\lVert Q_0\rVert)^{1/2}\delta^{1/2}+\delta. \tag*{(33)} E ∥ Q − Q 0 ∥ ≤ 2 ( E ∥ Q 0 ∥ ) 1/2 δ 1/2 + δ . (33) Proof. Let S , R ∈ R d × m S,R\in\mathbb{R}^{d\times m} S , R ∈ R d × m have columns D − 1 / 2 a i 1 / 2 σ i D^{-1/2}a_i^{1/2}\sigma_i D − 1/2 a i 1/2 σ i and D − 1 / 2 a i 1 / 2 e i D^{-1/2}a_i^{1/2}e_i D − 1/2 a i 1/2 e i , respectively. Then Q 0 = S S ⊤ Q_0=SS^\top Q 0 = S S ⊤ and
Q − Q 0 = S R ⊤ + R S ⊤ + R R ⊤ . Q-Q_0=SR^\top+RS^\top+RR^\top. Q − Q 0 = S R ⊤ + R S ⊤ + R R ⊤ . The trace bound and submultiplicativity imply
E ∥ R R ⊤ ∥ ≤ E tr ( R R ⊤ ) = D − 1 ∑ i = 1 m a i E ∥ e i ∥ 2 = δ , \mathbb{E}\lVert RR^\top\rVert\le\mathbb{E}\operatorname{tr}(RR^\top)=D^{-1}\sum_{i=1}^m a_i\mathbb{E}\lVert e_i\rVert^2=\delta, E ∥ R R ⊤ ∥ ≤ E tr ( R R ⊤ ) = D − 1 i = 1 ∑ m a i E ∥ e i ∥ 2 = δ , ∥ S R ⊤ ∥ ≤ ∥ S ∥ ∥ R ∥ = ∥ Q 0 ∥ 1 / 2 ∥ R R ⊤ ∥ 1 / 2 . \lVert SR^\top\rVert\le\lVert S\rVert\lVert R\rVert=\lVert Q_0\rVert^{1/2}\lVert RR^\top\rVert^{1/2}. ∥ S R ⊤ ∥ ≤ ∥ S ∥ ∥ R ∥ = ∥ Q 0 ∥ 1/2 ∥ R R ⊤ ∥ 1/2 . Since ∥ R S ⊤ ∥ = ∥ S R ⊤ ∥ \lVert RS^\top\rVert=\lVert SR^\top\rVert ∥ R S ⊤ ∥ = ∥ S R ⊤ ∥ , Cauchy–Schwarz yields the second inequality below:
E ∥ Q − Q 0 ∥ ≤ 2 E [ ∥ Q 0 ∥ 1 / 2 ∥ R R ⊤ ∥ 1 / 2 ] + E ∥ R R ⊤ ∥ ≤ 2 ( E ∥ Q 0 ∥ ) 1 / 2 ( E ∥ R R ⊤ ∥ ) 1 / 2 + E ∥ R R ⊤ ∥ \mathbb{E}\lVert Q-Q_0\rVert\le2\mathbb{E}\left[\lVert Q_0\rVert^{1/2}\lVert RR^\top\rVert^{1/2}\right]+\mathbb{E}\lVert RR^\top\rVert
\le2(\mathbb{E}\lVert Q_0\rVert)^{1/2}(\mathbb{E}\lVert RR^\top\rVert)^{1/2}+\mathbb{E}\lVert RR^\top\rVert E ∥ Q − Q 0 ∥ ≤ 2 E [ ∥ Q 0 ∥ 1/2 ∥ R R ⊤ ∥ 1/2 ] + E ∥ R R ⊤ ∥ ≤ 2 ( E ∥ Q 0 ∥ ) 1/2 ( E ∥ R R ⊤ ∥ ) 1/2 + E ∥ R R ⊤ ∥ ≤ 2 ( E ∥ Q 0 ∥ ) 1 / 2 δ 1 / 2 + δ . \le2(\mathbb{E}\lVert Q_0\rVert)^{1/2}\delta^{1/2}+\delta. ≤ 2 ( E ∥ Q 0 ∥ ) 1/2 δ 1/2 + δ . This proves the quadratic comparison bound. ■
The following lemma collects the deterministic block bounds used in the OL and NOL proofs. Recall the batches B j = { x t j , x t j + 1 , … , x j } B_j=\{x_{t_j},x_{t_j+1},\ldots,x_j\} B j = { x t j , x t j + 1 , … , x j } in (9) , with lengths ℓ j = ∣ B j ∣ = j − t j + 1 \ell_j=|B_j|=j-t_j+1 ℓ j = ∣ B j ∣ = j − t j + 1 . The set I j I_j I j below contains their indices.
Lemma A.6. Let α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2,1) α ∈ ( 1/2 , 1 ) and β > 1 / ( 1 − α ) \beta>1/(1-\alpha) β > 1/ ( 1 − α ) . Let ( a m ) m ≥ 1 (a_m)_{m\ge1} ( a m ) m ≥ 1 be an increasing integer sequence with a 1 = 1 a_1=1 a 1 = 1 . Suppose that there are constants 0 < c a ≤ C a < ∞ 0<c_a\le C_a<\infty 0 < c a ≤ C a < ∞ and 0 < c g ≤ C g < ∞ 0<c_g\le C_g<\infty 0 < c g ≤ C g < ∞ such that, for all m ≥ 1 m\ge1 m ≥ 1 ,
c a m β ≤ a m ≤ C a m β , c g m β − 1 ≤ a m + 1 − a m ≤ C g m β − 1 . c_a m^\beta\le a_m\le C_a m^\beta,\qquad c_gm^{\beta-1}\le a_{m+1}-a_m\le C_gm^{\beta-1}. c a m β ≤ a m ≤ C a m β , c g m β − 1 ≤ a m + 1 − a m ≤ C g m β − 1 .
For each integer j ≥ 1 j \ge1 j ≥ 1 , write
m j : = max { m : a m ≤ j } , t j : = a m j , m_j := \max\{m : a_m \le j\}, \qquad t_j := a_{m_j}, m j := max { m : a m ≤ j } , t j := a m j ,
I j : = { t j , t j + 1 , … , j } , ℓ j : = ∣ B j ∣ = j − t j + 1. I_j := \{t_j,t_j+1,\ldots,j\}, \qquad\ell_j := |B_j| = j-t_j+1. I j := { t j , t j + 1 , … , j } , ℓ j := ∣ B j ∣ = j − t j + 1.
At time n ≥ 1 n \ge1 n ≥ 1 , let L m : = min { a m + 1 − 1 , n } − a m + 1 L_m := \min\{a_{m+1}-1,n\}-a_m+1 L m := min { a m + 1 − 1 , n } − a m + 1 for 1 ≤ m ≤ m n 1 \le m \le m_n 1 ≤ m ≤ m n , and define
E n O L : = { 1 , … , n } , E n N O L : = { min { a m + 1 − 1 , n } : 1 ≤ m ≤ m n } , E_n^{\mathrm{OL}} := \{1,\ldots,n\}, \qquad E_n^{\mathrm{NOL}} := \{\min\{a_{m+1}-1,n\}:1 \le m \le m_n\}, E n OL := { 1 , … , n } , E n NOL := { min { a m + 1 − 1 , n } : 1 ≤ m ≤ m n } ,
v n , O L : = ∑ j ∈ E n O L ℓ j , v n , N O L : = n . v_{n,\mathrm{OL}} := \sum_{j\in E_n^{\mathrm{OL}}}\ell_j, \qquad v_{n,\mathrm{NOL}} := n. v n , OL := j ∈ E n OL ∑ ℓ j , v n , NOL := n .
Then there exists C ≥ 1 C \ge1 C ≥ 1 such that, for every integer n ≥ 1 n \ge1 n ≥ 1 and ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } ,
(i)
1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 ≤ C n 1 − 1 / β . \frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j^2 \le Cn^{1-1/\beta}. v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 ≤ C n 1 − 1/ β .
(ii)
1 v n , O L 2 ∑ m = 1 m n L m 4 ≤ C n − 1 / β . \frac{1}{v_{n,\mathrm{OL}}^2}\sum_{m=1}^{m_n}L_m^4 \le Cn^{-1/\beta}. v n , OL 2 1 m = 1 ∑ m n L m 4 ≤ C n − 1/ β .
(iii)
1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 t j − 2 α ≤ C n 1 − 1 / β − 2 α . \frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j^{2t_j-2\alpha} \le Cn^{1-1/\beta-2\alpha}. v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 t j − 2 α ≤ C n 1 − 1/ β − 2 α .
(iv)
1 v n , ⋄ ∑ j ∈ E n ⋄ j α ≤ C n α − 1 + 1 / β . \frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}j^\alpha\le Cn^{\alpha-1+1/\beta}. v n , ⋄ 1 j ∈ E n ⋄ ∑ j α ≤ C n α − 1 + 1/ β .
These bounds apply in particular to a 1 = 1 a_1=1 a 1 = 1 and a m = ⌊ c m β ⌋ a_m=\lfloor cm^\beta\rfloor a m = ⌊ c m β ⌋ for m ≥ 2 m\ge2 m ≥ 2 with c ≥ 1 c\ge1 c ≥ 1 , as in Theorem 3.1 .
Proof. For a m = ⌊ c m β ⌋ a_m=\lfloor cm^\beta\rfloor a m = ⌊ c m β ⌋ and m ≥ 2 m\ge2 m ≥ 2 , the floor error is less than one, so a m ≍ m β a_m\asymp m^\beta a m ≍ m β . The mean-value theorem also implies
c β m β − 1 − 1 ≤ a m + 1 − a m ≤ c β ( m + 1 ) β − 1 + 1. c\beta m^{\beta-1}-1 \le a_{m+1}-a_m \le c\beta(m+1)^{\beta-1}+1. c β m β − 1 − 1 ≤ a m + 1 − a m ≤ c β ( m + 1 ) β − 1 + 1.
Since β > 2 \beta>2 β > 2 and c ≥ 1 c\ge1 c ≥ 1 , these differences are positive and comparable to m β − 1 m^{\beta-1} m β − 1 . Adjusting the constants covers m = 1 m=1 m = 1 and verifies the hypotheses. For any sequence satisfying the hypotheses, we use
∑ m = 1 N m r ≍ N r + 1 , r ≥ 0 , N ≥ 1. (34) \sum_{m=1}^{N}m^r\asymp N^{r+1}, \qquad r\ge0,\quad N\ge1. \tag*{(34)} m = 1 ∑ N m r ≍ N r + 1 , r ≥ 0 , N ≥ 1. (34)
Indeed, since x r x^r x r is nondecreasing, we have
N r + 1 r + 1 = ∫ 0 N x r d x ≤ ∑ m = 1 N m r ≤ N ⋅ N r = N r + 1 . \frac{N^{r+1}}{r+1}=\int_0^N x^r\,\mathrm{d}x\le\sum_{m=1}^{N}m^r\le N\cdot N^r=N^{r+1}. r + 1 N r + 1 = ∫ 0 N x r d x ≤ m = 1 ∑ N m r ≤ N ⋅ N r = N r + 1 .
First, a m j ≤ j < a m j + 1 a_{m_j}\le j<a_{m_j+1} a m j ≤ j < a m j + 1 and m j + 1 ≤ 2 m j m_j+1\le2m_j m j + 1 ≤ 2 m j imply
c a m j β ≤ j < C a ( 2 m j ) β . c_a m_j^\beta\le j<C_a(2m_j)^\beta. c a m j β ≤ j < C a ( 2 m j ) β .
Thus m j ≍ j 1 / β m_j\asymp j^{1/\beta} m j ≍ j 1/ β . The NOL indices lie in distinct blocks, so ∣ E n N O L ∣ = m n ≍ n 1 / β |E_n^{\mathrm{NOL}}|=m_n\asymp n^{1/\beta} ∣ E n NOL ∣ = m n ≍ n 1/ β . Equivalently,
E n N O L = { a m + 1 − 1 : m ≥ 1 , a m + 1 ≤ n } ∪ { n } , E_n^{\mathrm{NOL}}=\{a_{m+1}-1:m\ge1,\ a_{m+1}\le n\}\cup\{n\}, E n NOL = { a m + 1 − 1 : m ≥ 1 , a m + 1 ≤ n } ∪ { n } ,
as in (16) . The first set excludes n n n , so n n n is counted once even at a block endpoint. Since t j = a m j ≥ c a m j β t_j=a_{m_j}\ge ca_{m_j}^{\beta} t j = a m j ≥ c a m j β ,
ℓ j ≤ C g m j β − 1 ≤ C g c a − ( β − 1 ) / β t j 1 − 1 / β ≤ C g c a − ( β − 1 ) / β j 1 − 1 / β . \ell_j\le C_gm_j^{\beta-1}\le C_gc_a^{-(\beta-1)/\beta}t_j^{1-1/\beta}\le C_gc_a^{-(\beta-1)/\beta}j^{1-1/\beta}. ℓ j ≤ C g m j β − 1 ≤ C g c a − ( β − 1 ) / β t j 1 − 1/ β ≤ C g c a − ( β − 1 ) / β j 1 − 1/ β .
For the normalizing factors, use L m ≤ C g m β − 1 L_m\le C_gm^{\beta-1} L m ≤ C g m β − 1 for all blocks and L m ≥ g m β − 1 L_m\ge gm^{\beta-1} L m ≥ g m β − 1 only for completed blocks. The OL batches in block m m m have lengths 1 , … , L m 1,\ldots,L_m 1 , … , L m , so
v n , O L = ∑ m = 1 m n L m ( L m + 1 ) 2 ≲ ∑ m = 1 m n m 2 β − 2 ≲ m n 2 β − 1 . v_{n,\mathrm{OL}}=\sum_{m=1}^{m_n}\frac{L_m(L_m+1)}{2}\lesssim\sum_{m=1}^{m_n}m^{2\beta-2}\lesssim m_n^{2\beta-1}. v n , OL = m = 1 ∑ m n 2 L m ( L m + 1 ) ≲ m = 1 ∑ m n m 2 β − 2 ≲ m n 2 β − 1 .
x r e f ‘ ( A .4 ) ‘ {xref}`(A.4)` x r e f ‘ ( A .4 ) ‘
The inequality uses L m ( L m + 1 ) / 2 ≤ L m 2 L_m(L_m+1)/2\le L_m^2 L m ( L m + 1 ) /2 ≤ L m 2 . For m n ≥ 2 m_n\ge2 m n ≥ 2 , the completed blocks and m n − 1 ≥ m n / 2 m_n-1\ge m_n/2 m n − 1 ≥ m n /2 yield
v n , O L ≥ ( c g 2 / 2 ) ∑ m = 1 m n − 1 m 2 β − 2 ≳ m n 2 β − 1 . v_{n,\mathrm{OL}}\ge(c_g^2/2)\sum_{m=1}^{m_n-1}m^{2\beta-2}\gtrsim m_n^{2\beta-1}. v n , OL ≥ ( c g 2 /2 ) m = 1 ∑ m n − 1 m 2 β − 2 ≳ m n 2 β − 1 .
x r e f ‘ ( A .4 ) ‘ {xref}`(A.4)` x r e f ‘ ( A .4 ) ‘
Adjusting the constants covers the finitely many times with m n = 1 m_n=1 m n = 1 . Hence v n , O L ≍ m n 2 β − 1 ≍ n 2 − 1 / β v_{n,\mathrm{OL}}\asymp m_n^{2\beta-1}\asymp n^{2-1/\beta} v n , OL ≍ m n 2 β − 1 ≍ n 2 − 1/ β . For NOL, the index sets { I j : j ∈ E n N O L } \{I_j:j\in E_n^{\mathrm{NOL}}\} { I j : j ∈ E n NOL } partition { 1 , … , n } \{1,\ldots,n\} { 1 , … , n } , so ∑ j ∈ E n N O L ℓ j = n \sum_{j\in E_n^{\mathrm{NOL}}}\ell_j=n ∑ j ∈ E n NOL ℓ j = n .
For (i), both batch rules satisfy ∑ j ∈ E n ⋄ ℓ j = v n , ⋄ \sum_{j\in E_n^\diamond}\ell_j=v_{n,\diamond} ∑ j ∈ E n ⋄ ℓ j = v n , ⋄ . The batch-length bound above therefore implies
1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 ≤ max j ≤ n ℓ j ≤ C g c a − ( β − 1 ) / β n 1 − 1 / β . \frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j^2\le\max_{j\le n}\ell_j\le C_gc_a^{-(\beta-1)/\beta}n^{1-1/\beta}. v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 ≤ j ≤ n max ℓ j ≤ C g c a − ( β − 1 ) / β n 1 − 1/ β .
For (ii), we have
∑ m = 1 m n L m 4 v n , O L 2 ≲ m n 4 β − 3 m n 4 β − 2 = m n − 1 ≲ n − 1 / β . \frac{\sum_{m=1}^{m_n}L_m^4}{v_{n,\mathrm{OL}}^2}\lesssim\frac{m_n^{4\beta-3}}{m_n^{4\beta-2}}=m_n^{-1}\lesssim n^{-1/\beta}. v n , OL 2 ∑ m = 1 m n L m 4 ≲ m n 4 β − 2 m n 4 β − 3 = m n − 1 ≲ n − 1/ β .
The first inequality uses L m ≤ C g m β − 1 L_m\le C_gm^{\beta-1} L m ≤ C g m β − 1 , (A.4) with exponent 4 β − 4 4\beta-4 4 β − 4 , and v n , O L ≍ m n 2 β − 1 v_{n,\mathrm{OL}}\asymp m_n^{2\beta-1} v n , OL ≍ m n 2 β − 1 . The last uses m n ≍ n 1 / β m_n\asymp n^{1/\beta} m n ≍ n 1/ β .
For (iii), t j = a m ≍ m β t_j=a_m\asymp m^\beta t j = a m ≍ m β throughout block m m m . Therefore,
∑ j ∈ E n O L ℓ j 2 t j − 2 α ≲ ∑ m = 1 m n m 3 β − 3 − 2 α β ≲ m n 3 β − 2 − 2 α β . \sum_{j\in E_n^{\mathrm{OL}}}\ell_j^2t_j^{-2\alpha}\lesssim\sum_{m=1}^{m_n}m^{3\beta-3-2\alpha\beta}\lesssim m_n^{3\beta-2-2\alpha\beta}. j ∈ E n OL ∑ ℓ j 2 t j − 2 α ≲ m = 1 ∑ m n m 3 β − 3 − 2 α β ≲ m n 3 β − 2 − 2 α β .
x r e f ‘ ( A .4 ) ‘ {xref}`(A.4)` x r e f ‘ ( A .4 ) ‘
\sum_{j\in E_n^{\mathrm{NOL}}}\ell_j^2t_j^{-2\alpha}\lesssim\sum_{m=1}^{m_n}m^{2\beta-2-2\alpha\beta}\lesssim m_n^{2\beta-1-2\alpha\beta}.
x r e f ‘ ( A .4 ) ‘ {xref}`(A.4)` x r e f ‘ ( A .4 ) ‘
The first inequalities use L m ≤ C g m β − 1 L_m\le C_gm^{\beta-1} L m ≤ C g m β − 1 and a m ≥ c a m β a_m\ge ca_m^\beta a m ≥ c a m β , with ∑ r = 1 L m r 2 ≤ L m 3 \sum_{r=1}^{L_m}r^2\le L_m^3 ∑ r = 1 L m r 2 ≤ L m 3 for OL and one batch of length L m L_m L m for NOL. Both exponents are positive: 2 β ( 1 − α ) − 2 > 0 2\beta(1-\alpha)-2>0 2 β ( 1 − α ) − 2 > 0 , and the OL exponent is larger by β − 1 \beta-1 β − 1 . Using v n , O L ≍ m n 2 β − 1 v_{n,\mathrm{OL}}\asymp m_n^{2\beta-1} v n , OL ≍ m n 2 β − 1 and v n , N O L = n ≍ m n β v_{n,\mathrm{NOL}}=n\asymp m_n^\beta v n , NOL = n ≍ m n β yields
1 v n , O L ∑ j ∈ E n O L ℓ j 2 t j − 2 α ≲ m n 3 β − 2 − 2 α β m n 2 β − 1 = m n β − 1 − 2 α β ≲ n 1 − 1 / β − 2 α , \frac{1}{v_{n,\mathrm{OL}}}\sum_{j\in E_n^{\mathrm{OL}}}\ell_j^2t_j^{-2\alpha}\lesssim\frac{m_n^{3\beta-2-2\alpha\beta}}{m_n^{2\beta-1}}=m_n^{\beta-1-2\alpha\beta}\lesssim n^{1-1/\beta-2\alpha}, v n , OL 1 j ∈ E n OL ∑ ℓ j 2 t j − 2 α ≲ m n 2 β − 1 m n 3 β − 2 − 2 α β = m n β − 1 − 2 α β ≲ n 1 − 1/ β − 2 α ,
1 v n , N O L ∑ j ∈ E n N O L ℓ j 2 t j − 2 α ≲ m n 2 β − 1 − 2 α β m n β = m n β − 1 − 2 α β ≲ n 1 − 1 / β − 2 α . \frac{1}{v_{n,\mathrm{NOL}}}\sum_{j\in E_n^{\mathrm{NOL}}}\ell_j^2t_j^{-2\alpha}\lesssim\frac{m_n^{2\beta-1-2\alpha\beta}}{m_n^\beta}=m_n^{\beta-1-2\alpha\beta}\lesssim n^{1-1/\beta-2\alpha}. v n , NOL 1 j ∈ E n NOL ∑ ℓ j 2 t j − 2 α ≲ m n β m n 2 β − 1 − 2 α β = m n β − 1 − 2 α β ≲ n 1 − 1/ β − 2 α .
Finally, for (iv), ∑ j = 1 n j α ≲ n α + 1 \sum_{j=1}^{n}j^{\alpha}\lesssim n^{\alpha+1} ∑ j = 1 n j α ≲ n α + 1 and ∣ E n N O L ∣ ≲ n 1 / β \lvert E_{n}^{\mathrm{NOL}}\rvert\lesssim n^{1/\beta} ∣ E n NOL ∣ ≲ n 1/ β imply
∑ j ∈ E n O L j α v n , O L ≲ n α − 1 + 1 / β , ∑ j ∈ E n N O L j α n ≤ n α − 1 ∣ E n N O L ∣ ≲ n α − 1 + 1 / β . \frac{\sum_{j\in E_{n}^{\mathrm{OL}}}j^{\alpha}}{v_{n,\mathrm{OL}}}\lesssim n^{\alpha-1+1/\beta},\qquad\frac{\sum_{j\in E_{n}^{\mathrm{NOL}}}j^{\alpha}}{n}\le n^{\alpha-1}\lvert E_{n}^{\mathrm{NOL}}\rvert\lesssim n^{\alpha-1+1/\beta}. v n , OL ∑ j ∈ E n OL j α ≲ n α − 1 + 1/ β , n ∑ j ∈ E n NOL j α ≤ n α − 1 ∣ E n NOL ∣ ≲ n α − 1 + 1/ β .
The bounds use j α ≤ n α j^{\alpha}\le n^{\alpha} j α ≤ n α , v n , O L ≍ n 2 − 1 / β v_{n,\mathrm{OL}}\asymp n^{2-1/\beta} v n , OL ≍ n 2 − 1/ β , and ∣ E n N O L ∣ ≍ n 1 / β \lvert E_{n}^{\mathrm{NOL}}\rvert\asymp n^{1/\beta} ∣ E n NOL ∣ ≍ n 1/ β . Choosing a common C ≥ 1 C\ge1 C ≥ 1 proves all four claims. ■
Manifold lemmas
This appendix collects the deterministic geometric estimates used in the proofs. Under Assumption 2.1 2.1 2.1 (a), R x − 1 ( y ) R_{x}^{-1}(y) R x − 1 ( y ) denotes the unique vector in B x ( 0 x , ρ r e t ) B_{x}(0_{x},\rho_{\mathrm{ret}}) B x ( 0 x , ρ ret ) such that R x ( R x − 1 ( y ) ) = y R_{x}(R_{x}^{-1}(y))=y R x ( R x − 1 ( y )) = y . For the maps used below, D D D denotes the total differential. If the domain is a product, D i D_{i} D i denotes the partial differential with respect to the i i i th argument, and D i j 2 : = D j D i D_{ij}^{2}:=D_{j}D_{i} D ij 2 := D j D i denotes the corresponding second partial differential. For a map or a parametrized family of maps with one displayed variable, D D D and D 2 D^{2} D 2 denote the first and second differentials with respect to that variable. Covariant differentiation along a curve is denoted by D / d t D/dt D / d t , as in § 2.1 . The base point of each vector norm is indicated explicitly, and ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ denotes the operator norm induced by the Riemannian metrics on the indicated tangent spaces. For a bilinear map B : V 1 × V 2 → W B:V_{1}\times V_{2}\to W B : V 1 × V 2 → W between finite-dimensional inner-product spaces, ∥ B ∥ : = sup ∥ u ∥ V 1 ≤ 1 , ∥ v ∥ V 2 ≤ 1 ∥ B [ u , v ] ∥ W \lVert B\rVert:=\sup_{\lVert u\rVert_{V_{1}}\le1,\lVert v\rVert_{V_{2}}\le1}\lVert B[u,v]\rVert_{W} ∥ B ∥ := sup ∥ u ∥ V 1 ≤ 1 , ∥ v ∥ V 2 ≤ 1 ∥ B [ u , v ] ∥ W .
Lemma B.1. Let X ⊆ M \mathcal{X}\subseteq\mathcal{M} X ⊆ M be compact and suppose that Assumption 2.1 2.1 2.1 (a) holds. Assume that ( x , v ) ↦ R x ( v ) (x,v)\mapsto R_{x}(v) ( x , v ) ↦ R x ( v ) is C 2 C^{2} C 2 on an open neighborhood of { ( x , R x − 1 ( y ) ) : x , y ∈ X } \{(x,R_{x}^{-1}(y)):x,y\in\mathcal{X}\} {( x , R x − 1 ( y )) : x , y ∈ X } in T M T\mathcal{M} T M . Then the following conclusions hold.
(i) The inverse retraction ( x , y ) ↦ R x − 1 ( y ) (x,y)\mapsto R_{x}^{-1}(y) ( x , y ) ↦ R x − 1 ( y ) has a jointly C 2 C^{2} C 2 extension to an open neighborhood of X × X \mathcal{X}\times\mathcal{X} X × X . More precisely, there exist an open set U R ⊆ M × M U_{R}\subseteq\mathcal{M}\times\mathcal{M} U R ⊆ M × M containing X × X \mathcal{X}\times\mathcal{X} X × X and a C 2 C^{2} C 2 map I : U R → T M \mathcal{I}:U_{R}\to T\mathcal{M} I : U R → T M such that
I ( x , y ) ∈ T x M , R x ( I ( x , y ) ) = y , ( x , y ) ∈ U R , \mathcal{I}(x,y)\in T_{x}\mathcal{M},\qquad R_{x}(\mathcal{I}(x,y))=y,\qquad(x,y)\in U_{R}, I ( x , y ) ∈ T x M , R x ( I ( x , y )) = y , ( x , y ) ∈ U R , and I ( x , y ) = R x − 1 ( y ) \mathcal{I}(x,y)=R_{x}^{-1}(y) I ( x , y ) = R x − 1 ( y ) for every x , y ∈ X x,y\in\mathcal{X} x , y ∈ X .
(ii) Using the extension in (i), the map Φ x , y : = R y − 1 ∘ R x \Phi_{x,y}:=R_{y}^{-1}\circ R_{x} Φ x , y := R y − 1 ∘ R x is C 2 C^{2} C 2 on a neighborhood of R x − 1 ( X ) R_{x}^{-1}(\mathcal{X}) R x − 1 ( X ) for every x , y ∈ X x,y\in\mathcal{X} x , y ∈ X . Moreover, there exists C < ∞ C<\infty C < ∞ such that
∥ D 2 Φ x , y ( v ) ∥ ≤ C , x , y ∈ X , v ∈ R x − 1 ( X ) . \lVert D^{2}\Phi_{x,y}(v)\rVert\le C,\qquad x,y\in\mathcal{X},\quad v\in R_{x}^{-1}(\mathcal{X}). ∥ D 2 Φ x , y ( v )∥ ≤ C , x , y ∈ X , v ∈ R x − 1 ( X ) . Here the derivatives are taken with respect to v ∈ T x M v\in T_{x}\mathcal{M} v ∈ T x M .
If R R R also has the C 3 C^{3} C 3 regularity in Assumption 2.1 2.1 2.1 (e), both extensions can be chosen to be C 3 C^{3} C 3 .
Proof. For (i), consider ( x , v ) ↦ ( x , R x ( v ) ) (x,v)\mapsto(x,R_{x}(v)) ( x , v ) ↦ ( x , R x ( v )) . At ( x , R x − 1 ( y ) ) (x,R_{x}^{-1}(y)) ( x , R x − 1 ( y )) , x , y ∈ X x,y\in\mathcal{X} x , y ∈ X , its differential has the block form
( I 0 ∗ D R x ( R x − 1 ( y ) ) ) \begin{pmatrix}
I & 0\\
* & DR_{x}(R_{x}^{-1}(y))
\end{pmatrix} ( I ∗ 0 D R x ( R x − 1 ( y )) )
The lower-right block is invertible by Assumption 2.1 2.1 2.1 (a), so the inverse function theorem supplies local C 2 C^{2} C 2 inverses. Shrink their domains so that the tangent components have norm less than ρ r e t \rho_{\mathrm{ret}} ρ ret . Uniqueness in Assumption 2.1(a) identifies these components with R x − 1 ( y ) R_x^{-1}(y) R x − 1 ( y ) whenever x , y ∈ X x,y\in\mathcal{X} x , y ∈ X . Thus ( x , y ) ↦ R x − 1 ( y ) (x,y)\mapsto R_x^{-1}(y) ( x , y ) ↦ R x − 1 ( y ) is continuous on X × X \mathcal{X}\times\mathcal{X} X × X , and { ( x , R x − 1 ( y ) ) : x , y ∈ X } \{(x,R_x^{-1}(y)):x,y\in\mathcal{X}\} {( x , R x − 1 ( y )) : x , y ∈ X } is compact. By the inverse function theorem, ( x , v ) ↦ ( x , R x ( v ) ) (x,v)\mapsto(x,R_x(v)) ( x , v ) ↦ ( x , R x ( v )) is locally invertible near this compact set. Since it is one-to-one on the set, compactness implies that it is one-to-one on a sufficiently small open neighborhood. Its image is an open set U R U_R U R containing X × X \mathcal{X}\times\mathcal{X} X × X , and its C 2 C^2 C 2 inverse has the form ( x , y ) ↦ ( x , I ( x , y ) ) (x,y)\mapsto(x,\mathcal{I}(x,y)) ( x , y ) ↦ ( x , I ( x , y )) . This proves (i).
For (ii), the set { ( x , y , R x − 1 ( z ) ) : x , y , z ∈ X } \{(x,y,R_x^{-1}(z)):x,y,z\in\mathcal{X}\} {( x , y , R x − 1 ( z )) : x , y , z ∈ X } is compact by continuity. On this set, ( y , R x ( v ) ) ∈ U R (y,R_x(v))\in U_R ( y , R x ( v )) ∈ U R , so Φ x , y ( v ) = I ( y , R x ( v ) ) ∈ T y M \Phi_{x,y}(v)=\mathcal{I}(y,R_x(v))\in T_y\mathcal{M} Φ x , y ( v ) = I ( y , R x ( v )) ∈ T y M is jointly C 2 C^2 C 2 on a neighborhood. Compactness therefore bounds D 2 Φ x , y D^2\Phi_{x,y} D 2 Φ x , y uniformly on the set. Under the C 3 C^3 C 3 condition, the same argument yields C 3 C^3 C 3 extensions. This completes the proof. ■
Lemma B.2. Under the assumptions of Lemma B.1 , there exist constants 0 < c M ≤ C M < ∞ 0<c_M\le C_M<\infty 0 < c M ≤ C M < ∞ such that, for every x , y ∈ X x,y\in\mathcal{X} x , y ∈ X ,
c M d ( x , y ) ≤ ∥ R x − 1 ( y ) ∥ x ≤ C M d ( x , y ) . (35) c_M d(x,y)\le\lVert R_x^{-1}(y)\rVert_x\le C_M d(x,y). \tag*{(35)} c M d ( x , y ) ≤ ∥ R x − 1 ( y ) ∥ x ≤ C M d ( x , y ) . (35)
Consequently, for fixed x ⋆ ∈ X x_\star\in\mathcal{X} x ⋆ ∈ X and ∀ x , y ∈ X \forall x,y\in\mathcal{X} ∀ x , y ∈ X , ∥ R x − 1 ( y ) ∥ x ≤ C M c M − 1 ( ∥ R x ⋆ − 1 ( x ) ∥ x ⋆ + ∥ R x ⋆ − 1 ( y ) ∥ x ⋆ ) \lVert R_x^{-1}(y)\rVert_x\le C_M c_M^{-1}\left(\lVert R_{x_\star}^{-1}(x)\rVert_{x_\star}+\lVert R_{x_\star}^{-1}(y)\rVert_{x_\star}\right) ∥ R x − 1 ( y ) ∥ x ≤ C M c M − 1 ( ∥ R x ⋆ − 1 ( x ) ∥ x ⋆ + ∥ R x ⋆ − 1 ( y ) ∥ x ⋆ ) .
Proof. For x ≠ y x\ne y x = y , set h ( x , y ) : = ∥ R x − 1 ( y ) ∥ x / d ( x , y ) h(x,y):=\lVert R_x^{-1}(y)\rVert_x/d(x,y) h ( x , y ) := ∥ R x − 1 ( y ) ∥ x / d ( x , y ) , which is continuous by Lemma B.1 . We show that h ( x , y ) → 1 h(x,y)\to1 h ( x , y ) → 1 as y → x y\to x y → x , uniformly in x ∈ X x\in\mathcal{X} x ∈ X . The local properties of the exponential map and compactness of X \mathcal{X} X supply a common ρ > 0 \rho>0 ρ > 0 : for every x ∈ X x\in\mathcal{X} x ∈ X , d ( x , y ) ≤ ρ d(x,y)\le\rho d ( x , y ) ≤ ρ implies y = Exp x ( v ) y=\operatorname{Exp}_x(v) y = Exp x ( v ) for a unique v v v with ∥ v ∥ x ≤ ρ \lVert v\rVert_x\le\rho ∥ v ∥ x ≤ ρ , and
d ( x , Exp x ( v ) ) = ∥ v ∥ x , ∥ v ∥ x ≤ ρ . (36) d(x,\operatorname{Exp}_x(v))=\lVert v\rVert_x,\qquad\lVert v\rVert_x\le\rho. \tag*{(36)} d ( x , Exp x ( v )) = ∥ v ∥ x , ∥ v ∥ x ≤ ρ . (36)
Here we use R x − 1 R_x^{-1} R x − 1 also for its C 2 C^2 C 2 extension in Lemma B.1 . Reducing ρ \rho ρ if necessary, ( x , v ) ↦ R x − 1 ( Exp x ( v ) ) (x,v)\mapsto R_x^{-1}(\operatorname{Exp}_x(v)) ( x , v ) ↦ R x − 1 ( Exp x ( v )) is C 2 C^2 C 2 on an open neighborhood of the compact set { ( x , v ) ∈ T M : x ∈ X , ∥ v ∥ x ≤ ρ } \{(x,v)\in T\mathcal{M}:x\in\mathcal{X},\lVert v\rVert_x\le\rho\} {( x , v ) ∈ T M : x ∈ X , ∥ v ∥ x ≤ ρ } , and
R x − 1 ( Exp x ( 0 x ) ) = 0 x , D ( R x − 1 ∘ Exp x ) ( 0 x ) = I T x M . R_x^{-1}(\operatorname{Exp}_x(0_x))=0_x,\qquad D(R_x^{-1}\circ\operatorname{Exp}_x)(0_x)=I_{T_x\mathcal{M}}. R x − 1 ( Exp x ( 0 x )) = 0 x , D ( R x − 1 ∘ Exp x ) ( 0 x ) = I T x M .
Applying the fundamental theorem of calculus twice to t ↦ R x − 1 ( Exp x ( t v ) ) t\mapsto R_x^{-1}(\operatorname{Exp}_x(tv)) t ↦ R x − 1 ( Exp x ( t v )) yields, for ∥ v ∥ x ≤ ρ \lVert v\rVert_x\le\rho ∥ v ∥ x ≤ ρ ,
R x − 1 ( Exp x ( v ) ) − v = ∫ 0 1 ( 1 − t ) D 2 ( R x − 1 ∘ Exp x ) ( t v ) [ v , v ] d t . R_x^{-1}(\operatorname{Exp}_x(v))-v=\int_0^1(1-t)D^2(R_x^{-1}\circ\operatorname{Exp}_x)(tv)[v,v]\,dt. R x − 1 ( Exp x ( v )) − v = ∫ 0 1 ( 1 − t ) D 2 ( R x − 1 ∘ Exp x ) ( t v ) [ v , v ] d t .
Consequently, we obtain
∥ R x − 1 ( Exp x ( v ) ) − v ∥ x ≤ 1 2 sup x ∈ X ∥ u ∥ x ≤ ρ ∥ D 2 ( R x − 1 ∘ Exp x ) ( u ) ∥ ∥ v ∥ x 2 . (37) \lVert R_x^{-1}(\operatorname{Exp}_x(v))-v\rVert_x\le\frac{1}{2}\sup_{\substack{x\in\mathcal{X}\\\lVert u\rVert_x\le\rho}}\lVert D^2(R_x^{-1}\circ\operatorname{Exp}_x)(u)\rVert\lVert v\rVert_x^2. \tag*{(37)} ∥ R x − 1 ( Exp x ( v )) − v ∥ x ≤ 2 1 x ∈ X ∥ u ∥ x ≤ ρ sup ∥ D 2 ( R x − 1 ∘ Exp x ) ( u )∥ ∥ v ∥ x 2 . (37)
The supremum is finite by compactness. For 0 < ∥ v ∥ x ≤ ρ 0<\lVert v\rVert_x\le\rho 0 < ∥ v ∥ x ≤ ρ ,
∣ ∥ R x − 1 ( Exp x ( v ) ) ∥ x ∥ v ∥ x − 1 ∣ ≤ ∥ R x − 1 ( Exp x ( v ) ) − v ∥ x ∥ v ∥ x ≤ C ∥ v ∥ x . \left|\frac{\lVert R_x^{-1}(\operatorname{Exp}_x(v))\rVert_x}{\lVert v\rVert_x}-1\right|\le\frac{\lVert R_x^{-1}(\operatorname{Exp}_x(v))-v\rVert_x}{\lVert v\rVert_x}\le C\lVert v\rVert_x. ∥ v ∥ x ∥ R x − 1 ( Exp x ( v )) ∥ x − 1 ≤ ∥ v ∥ x ∥ R x − 1 ( Exp x ( v )) − v ∥ x ≤ C ∥ v ∥ x .
Together with (B.2), this uniform estimate shows that h ( x , x ) : = 1 h(x,x):=1 h ( x , x ) := 1 extends h h h continuously to X × X \mathcal{X}\times\mathcal{X} X × X . Since R x − 1 ( y ) = 0 x R_x^{-1}(y)=0_x R x − 1 ( y ) = 0 x only when x = y x=y x = y , this extension is positive. Compactness then yields 0 < c M : = min h ≤ C M : = max h < ∞ 0<c_M:=\min h\le C_M:=\max h<\infty 0 < c M := min h ≤ C M := max h < ∞ . Finally, we have
∥ R x − 1 ( y ) ∥ x ≤ (B.1) C M d ( x , y ) ≤ C M ( d ( x , x ⋆ ) + d ( x ⋆ , y ) ) ≤ (B.1) C M c M − 1 ( ∥ R x ⋆ − 1 ( x ) ∥ x ⋆ + ∥ R x ⋆ − 1 ( y ) ∥ x ⋆ ) , \lVert R_x^{-1}(y)\rVert_x\mathrel{\overset{\text{(B.1)}}{\le}} C_Md(x,y)\le C_M\bigl(d(x,x_\star)+d(x_\star,y)\bigr)\mathrel{\overset{\text{(B.1)}}{\le}} C_Mc_M^{-1}\bigl(\lVert R_{x_\star}^{-1}(x)\rVert_{x_\star}+\lVert R_{x_\star}^{-1}(y)\rVert_{x_\star}\bigr), ∥ R x − 1 ( y ) ∥ x ≤ (B.1) C M d ( x , y ) ≤ C M ( d ( x , x ⋆ ) + d ( x ⋆ , y ) ) ≤ (B.1) C M c M − 1 ( ∥ R x ⋆ − 1 ( x ) ∥ x ⋆ + ∥ R x ⋆ − 1 ( y ) ∥ x ⋆ ) ,
which proves the second claim. ■
Recall from § 2.1 that Λ x y = D R x ( R x − 1 ( y ) ) \Lambda_x^y = DR_x\left(R_x^{-1}(y)\right) Λ x y = D R x ( R x − 1 ( y ) ) for x , y ∈ X x,y \in\mathcal{X} x , y ∈ X .
Lemma B.3. Under the assumptions of Lemma B.1, suppose additionally that items (b), (d), and (e) of Assumption 2.1 hold. Then there exists a constant C L < ∞ C_L < \infty C L < ∞ such that, for every x , y ∈ X x,y \in\mathcal{X} x , y ∈ X ,
∥ Λ x y ∥ + ∥ [ Λ x y ] − 1 ∥ ≤ C L . (38) \|\Lambda_x^y\|+\|[\Lambda_x^y]^{-1}\| \leq C_L. \tag*{(38)} ∥ Λ x y ∥ + ∥ [ Λ x y ] − 1 ∥ ≤ C L . (38) Moreover,
∥ Λ x y ∥ ≤ 1 + C L ∥ R x − 1 ( y ) ∥ x 2 . (39) \|\Lambda_x^y\| \leq1+C_L\|R_x^{-1}(y)\|_x^2. \tag*{(39)} ∥ Λ x y ∥ ≤ 1 + C L ∥ R x − 1 ( y ) ∥ x 2 . (39)
Proof. Let I \mathcal{I} I be the C 2 C^2 C 2 extension in Lemma B.1. For u = R x − 1 ( y ) u=R_x^{-1}(y) u = R x − 1 ( y ) , the identity I ( x , R x ( u ) ) = u \mathcal{I}(x,R_x(u))=u I ( x , R x ( u )) = u and the chain rule imply
D 2 I ( x , y ) ∘ D R x ( u ) = I T x M . D_2\mathcal{I}(x,y)\circ DR_x(u)=I_{T_x\mathcal{M}}. D 2 I ( x , y ) ∘ D R x ( u ) = I T x M .
Since D R x ( u ) = Λ x y DR_x(u)=\Lambda_x^y D R x ( u ) = Λ x y by definition, we have [ Λ x y ] − 1 = D 2 I ( x , y ) [\Lambda_x^y]^{-1}=D_2\mathcal{I}(x,y) [ Λ x y ] − 1 = D 2 I ( x , y ) . Both Λ x y \Lambda_x^y Λ x y and its inverse are continuous on the compact set X × X \mathcal{X}\times\mathcal{X} X × X , proving (B.4) for some C L C_L C L .
For the second bound, the second covariant derivative of R x R_x R x at 0 x 0_x 0 x is symmetric because the Levi-Civita connection is torsion-free. Its values on equal arguments vanish by the second-order retraction property, so polarization implies
D d t ( D R x ( t h ) [ v ] ) ∣ t = 0 = 0 x , h , v ∈ T x M . (40) \left.\frac{D}{\mathrm{d}t}(DR_x(th)[v])\right|_{t=0}=0_x,\qquad h,v\in T_x\mathcal{M}. \tag*{(40)} d t D ( D R x ( t h ) [ v ]) t = 0 = 0 x , h , v ∈ T x M . (40)
where the derivative is taken along t ↦ R x ( t h ) t\mapsto R_x(th) t ↦ R x ( t h ) . For ∥ v ∥ x = 1 \|v\|_x=1 ∥ v ∥ x = 1 , the retraction property, metric compatibility, and (B.6) imply
∥ D R x ( 0 x ) [ v ] ∥ x 2 = 1 , d d s ∥ D R x ( s u ) [ v ] ∥ R x ( s u ) 2 ∣ s = 0 = 0. \|DR_x(0_x)[v]\|_x^2=1,\qquad\left.\frac{\mathrm{d}}{\mathrm{d}s}\|DR_x(su)[v]\|_{R_x(su)}^2\right|_{s=0}=0. ∥ D R x ( 0 x ) [ v ] ∥ x 2 = 1 , d s d ∥ D R x ( s u ) [ v ] ∥ R x ( s u ) 2 s = 0 = 0.
Retraction convexity and injectivity imply R x ( s u ) ∈ X R_x(su)\in\mathcal{X} R x ( s u ) ∈ X and s u = R x − 1 ( R x ( s u ) ) su=R_x^{-1}(R_x(su)) s u = R x − 1 ( R x ( s u )) for s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] . These points lie in the compact set { ( x , R x − 1 ( z ) ) : x , z ∈ X } \{(x,R_x^{-1}(z)):x,z\in\mathcal{X}\} {( x , R x − 1 ( z )) : x , z ∈ X } . By C 3 C^3 C 3 regularity, ∣ d 2 d s 2 ∥ D R x ( s u ) [ v ] ∥ R x ( s u ) 2 ∣ ≤ C ∥ u ∥ x 2 \left|\frac{\mathrm{d}^2}{\mathrm{d}s^2}\|DR_x(su)[v]\|_{R_x(su)}^2\right|\leq C\|u\|_x^2 d s 2 d 2 ∥ D R x ( s u ) [ v ] ∥ R x ( s u ) 2 ≤ C ∥ u ∥ x 2 uniformly for x , y ∈ X x,y\in\mathcal{X} x , y ∈ X , s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] , and ∥ v ∥ x = 1 \|v\|_x=1 ∥ v ∥ x = 1 . Taylor’s formula therefore yields
∥ D R x ( u ) [ v ] ∥ y 2 = 1 + ∫ 0 1 ( 1 − s ) d 2 d s 2 ∥ D R x ( s u ) [ v ] ∥ R x ( s u ) 2 d s ≤ 1 + C 2 ∥ u ∥ x 2 . (41) \|DR_x(u)[v]\|_y^2=1+\int_0^1(1-s)\frac{\mathrm{d}^2}{\mathrm{d}s^2}\|DR_x(su)[v]\|_{R_x(su)}^2\,\mathrm{d}s\leq1+\frac{C}{2}\|u\|_x^2. \tag*{(41)} ∥ D R x ( u ) [ v ] ∥ y 2 = 1 + ∫ 0 1 ( 1 − s ) d s 2 d 2 ∥ D R x ( s u ) [ v ] ∥ R x ( s u ) 2 d s ≤ 1 + 2 C ∥ u ∥ x 2 . (41)
Taking the supremum over ∥ v ∥ x = 1 \|v\|_x=1 ∥ v ∥ x = 1 , we obtain
∥ Λ x y ∥ ≤ ( 1 + C 2 ∥ u ∥ x 2 ) 1 / 2 ≤ 1 + C 4 ∥ u ∥ x 2 . \|\Lambda_x^y\|\leq\left(1+\frac{C}{2}\|u\|_x^2\right)^{1/2}\leq1+\frac{C}{4}\|u\|_x^2. ∥ Λ x y ∥ ≤ ( 1 + 2 C ∥ u ∥ x 2 ) 1/2 ≤ 1 + 4 C ∥ u ∥ x 2 .
Choosing C L C_L C L large enough proves both bounds. ■
Lemma B.4. Under the assumptions of Lemma B.1, suppose additionally that items (b) and (e) of Assumption 2.1 hold. Then there exists a constant C Ψ < ∞ C_\Psi<\infty C Ψ < ∞ such that, for every x ∈ X x\in\mathcal{X} x ∈ X and every u , w ∈ T x M u,w\in T_x\mathcal{M} u , w ∈ T x M with u , u + w ∈ R x − 1 ( X ) u,u+w\in R_x^{-1}(\mathcal{X}) u , u + w ∈ R x − 1 ( X ) , we have
∥ [ Λ R x ( u ) R x ( u ) ] − 1 R R x ( u ) − 1 ( R x ( u + w ) ) − w ∥ x ≤ C Ψ ∥ u ∥ x ∥ w ∥ x . (42) \left\|[\Lambda_{R_x(u)}^{R_x(u)}]^{-1}R_{R_x(u)}^{-1}(R_x(u+w))-w\right\|_x\leq C_\Psi\|u\|_x\|w\|_x. \tag*{(42)} [ Λ R x ( u ) R x ( u ) ] − 1 R R x ( u ) − 1 ( R x ( u + w )) − w x ≤ C Ψ ∥ u ∥ x ∥ w ∥ x . (42)
Proof. For x ∈ X x \in\mathcal{X} x ∈ X and u , v ∈ R x − 1 ( X ) u,v \in R_x^{-1}(\mathcal{X}) u , v ∈ R x − 1 ( X ) , define
Ψ x ( u , v ) : = [ Λ x R x ( u ) ] − 1 R R x ( u ) − 1 ( R x ( v ) ) − ( v − u ) ∈ T x M . \Psi_x(u,v) := [\Lambda_x^{R_x(u)}]^{-1}R_{R_x(u)}^{-1}(R_x(v))-(v-u) \in T_x\mathcal{M}. Ψ x ( u , v ) := [ Λ x R x ( u ) ] − 1 R R x ( u ) − 1 ( R x ( v )) − ( v − u ) ∈ T x M .
These triples form a compact set by Lemma B.1 . Since Λ x R x ( u ) = D R x ( u ) \Lambda_x^{R_x(u)}=DR_x(u) Λ x R x ( u ) = D R x ( u ) , the inverse extension in that lemma and Assumption 2.1 (e) make Ψ \Psi Ψ jointly C 2 C^2 C 2 on an open neighborhood of this set. Moreover,
Ψ x ( 0 x , v ) = 0 x , Ψ x ( u , u ) = 0 x . \Psi_x(0_x,v)=0_x,\qquad\Psi_x(u,u)=0_x. Ψ x ( 0 x , v ) = 0 x , Ψ x ( u , u ) = 0 x .
Retraction convexity and injectivity imply s u ∈ R x − 1 ( X ) su\in R_x^{-1}(\mathcal{X}) s u ∈ R x − 1 ( X ) for s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] . Thus ( x , s u , u + w ) (x,su,u+w) ( x , s u , u + w ) and ( x , s u , u ) (x,su,u) ( x , s u , u ) lie in the compact set above. Since D 1 Ψ D_1\Psi D 1 Ψ is C 1 C^1 C 1 on an open neighborhood, it is Lipschitz in the second variable on this compact set, uniformly in x x x and the first variable. Hence there exists C < ∞ C<\infty C < ∞ such that
∥ D 1 Ψ x ( s u , u + w ) − D 1 Ψ x ( s u , u ) ∥ x ≤ C ∥ w ∥ x , s ∈ [ 0 , 1 ] . \lVert D_1\Psi_x(su,u+w)-D_1\Psi_x(su,u)\rVert_x\le C\lVert w\rVert_x,\qquad s\in[0,1]. ∥ D 1 Ψ x ( s u , u + w ) − D 1 Ψ x ( s u , u ) ∥ x ≤ C ∥ w ∥ x , s ∈ [ 0 , 1 ] .
The fundamental theorem of calculus and the two identities above yield
Ψ x ( u , u + w ) = Ψ x ( u , u + w ) − Ψ x ( 0 x , u + w ) − Ψ x ( u , u ) + Ψ x ( 0 x , u ) = ∫ 0 1 ( D 1 Ψ x ( s u , u + w ) − D 1 Ψ x ( s u , u ) ) [ u ] d s . \begin{aligned}
\Psi_x(u,u+w)
&=\Psi_x(u,u+w)-\Psi_x(0_x,u+w)-\Psi_x(u,u)+\Psi_x(0_x,u)\\
&=\int_0^1\left(D_1\Psi_x(su,u+w)-D_1\Psi_x(su,u)\right)[u]\,\mathrm{d}s.
\end{aligned} Ψ x ( u , u + w ) = Ψ x ( u , u + w ) − Ψ x ( 0 x , u + w ) − Ψ x ( u , u ) + Ψ x ( 0 x , u ) = ∫ 0 1 ( D 1 Ψ x ( s u , u + w ) − D 1 Ψ x ( s u , u ) ) [ u ] d s .
The derivative bound then implies
∥ Ψ x ( u , u + w ) ∥ x ≤ ∫ 0 1 C ∥ w ∥ x ∥ u ∥ x d s = C ∥ u ∥ x ∥ w ∥ x . \lVert\Psi_x(u,u+w)\rVert_x\le\int_0^1 C\lVert w\rVert_x\lVert u\rVert_x\,\mathrm{d}s=C\lVert u\rVert_x\lVert w\rVert_x. ∥ Ψ x ( u , u + w ) ∥ x ≤ ∫ 0 1 C ∥ w ∥ x ∥ u ∥ x d s = C ∥ u ∥ x ∥ w ∥ x .
Taking C Ψ = C C_\Psi=C C Ψ = C completes the proof. ■
We next compare Λ x y \Lambda_x^y Λ x y with parallel transport Γ x y \Gamma_x^y Γ x y along the curve t ↦ R x ( t R x − 1 ( y ) ) t\mapsto R_x(tR_x^{-1}(y)) t ↦ R x ( t R x − 1 ( y )) , t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] . Its inverse ( Γ x y ) − 1 (\Gamma_x^y)^{-1} ( Γ x y ) − 1 denotes parallel transport along the same curve in reverse.
Lemma B.5. Under Assumption 2.1 , there exists a constant C Γ , Λ < ∞ C_{\Gamma,\Lambda}<\infty C Γ , Λ < ∞ such that, for every x , y ∈ X x,y\in\mathcal{X} x , y ∈ X ,
∥ ( Γ x y ) − 1 Λ x y − I T x M ∥ ≤ C Γ , Λ ∥ R x − 1 ( y ) ∥ x 2 . (43) \left\lVert(\Gamma_x^y)^{-1}\Lambda_x^y-I_{T_x\mathcal{M}}\right\rVert\le C_{\Gamma,\Lambda}\lVert R_x^{-1}(y)\rVert_x^2. \tag*{(43)} ( Γ x y ) − 1 Λ x y − I T x M ≤ C Γ , Λ ∥ R x − 1 ( y ) ∥ x 2 . (43) In particular, for every y ∈ X y\in\mathcal{X} y ∈ X ,
∥ ( Γ x y ) − 1 − [ Λ x y ] − 1 ∥ ≤ C Γ , Λ ∥ R x − 1 ( y ) ∥ x 2 . (44) \left\lVert(\Gamma_x^y)^{-1}-[\Lambda_x^y]^{-1}\right\rVert\le C_{\Gamma,\Lambda}\lVert R_x^{-1}(y)\rVert_x^2. \tag*{(44)} ( Γ x y ) − 1 − [ Λ x y ] − 1 ≤ C Γ , Λ ∥ R x − 1 ( y ) ∥ x 2 . (44)
Proof. Set u = R x − 1 ( y ) u=R_x^{-1}(y) u = R x − 1 ( y ) . Retraction convexity and injectivity imply R x ( s u ) ∈ X R_x(su)\in\mathcal{X} R x ( s u ) ∈ X and s u = R x − 1 ( R x ( s u ) ) su=R_x^{-1}(R_x(su)) s u = R x − 1 ( R x ( s u )) for s ∈ [ 0 , 1 ] s\in[0,1] s ∈ [ 0 , 1 ] . Thus ( x , s u ) (x,su) ( x , s u ) remains in the compact set { ( x , R x − 1 ( z ) ) : x , z ∈ X } \{(x,R_x^{-1}(z)):x,z\in\mathcal{X}\} {( x , R x − 1 ( z )) : x , z ∈ X } from Lemma B.1 . For nearby ( x , v ) (x,v) ( x , v ) , let Γ x R x ( v ) \Gamma_x^{R_x(v)} Γ x R x ( v ) denote transport along t ↦ R x ( t v ) t\mapsto R_x(tv) t ↦ R x ( t v ) . It satisfies a linear differential equation with coefficients that are C 2 C^2 C 2 in ( x , v ) (x,v) ( x , v ) by Assumption 2.1 (e), so its solution and inverse are C 2 C^2 C 2 . Consequently, ( Γ x R x ( v ) ) − 1 D R x ( v ) (\Gamma_x^{R_x(v)})^{-1}DR_x(v) ( Γ x R x ( v ) ) − 1 D R x ( v ) is jointly C 2 C^2 C 2 near this compact set. Since Γ x R x ( s u ) \Gamma_x^{R_x(su)} Γ x R x ( s u ) follows the curve s ↦ R x ( s u ) s\mapsto R_x(su) s ↦ R x ( s u ) up to time s s s , (B.6) implies, for every v ∈ T x M v\in T_x\mathcal{M} v ∈ T x M ,
d d s ( ( Γ x R x ( s u ) ) − 1 D R x ( s u ) [ v ] ) ∣ s = 0 = D d s ( D R x ( s u ) [ v ] ) ∣ s = 0 = 0 x . (45) \left.\frac{\mathrm{d}}{\mathrm{d}s}\left((\Gamma_x^{R_x(su)})^{-1}DR_x(su)[v]\right)\right|_{s=0}
=
\left.\frac{\mathrm{D}}{\mathrm{d}s}\left(DR_x(su)[v]\right)\right|_{s=0}
=0_x. \tag*{(45)} d s d ( ( Γ x R x ( s u ) ) − 1 D R x ( s u ) [ v ] ) s = 0 = d s D ( D R x ( s u ) [ v ] ) s = 0 = 0 x . (45)
The operator at s = 0 s=0 s = 0 is Γ T x M \Gamma_{T_x\mathcal{M}} Γ T x M , and its second derivative in s s s has norm at most C ∥ u ∥ x 2 C\lVert u\rVert_x^2 C ∥ u ∥ x 2 by the C 2 C^2 C 2 regularity and compactness above. Taylor’s formula then yields
∥ ( Γ x y ) − 1 Λ x y − I T x ⋆ M ∥ = ∥ ∫ 0 1 ( 1 − s ) d 2 d s 2 ( ( Γ x R x ( s u ) ) − 1 D R x ( s u ) ) d s ∥ ≤ C 2 ∥ u ∥ x 2 . \left\lVert(\Gamma_x^y)^{-1}\Lambda_x^y-I_{T_{x_\star}\mathcal{M}}\right\rVert=\left\lVert\int_0^1(1-s)\frac{d^2}{ds^2}\left((\Gamma_x^{R_x(su)})^{-1}D R_x(su)\right)\,ds\right\rVert\leq\frac{C}{2}\lVert u\rVert_x^2. ( Γ x y ) − 1 Λ x y − I T x ⋆ M = ∫ 0 1 ( 1 − s ) d s 2 d 2 ( ( Γ x R x ( s u ) ) − 1 D R x ( s u ) ) d s ≤ 2 C ∥ u ∥ x 2 .
This proves the first bound. For the inverse transports,
[ Λ x ⋆ y ] − 1 − ( Γ x ⋆ y ) − 1 = [ Λ x ⋆ y ] − 1 ( Γ x ⋆ y − Λ x ⋆ y ) ( Γ x ⋆ y ) − 1 . [\Lambda_{x_\star}^y]^{-1}-(\Gamma_{x_\star}^y)^{-1}=[\Lambda_{x_\star}^y]^{-1}(\Gamma_{x_\star}^y-\Lambda_{x_\star}^y)(\Gamma_{x_\star}^y)^{-1}. [ Λ x ⋆ y ] − 1 − ( Γ x ⋆ y ) − 1 = [ Λ x ⋆ y ] − 1 ( Γ x ⋆ y − Λ x ⋆ y ) ( Γ x ⋆ y ) − 1 .
Submultiplicativity and the isometry of parallel transport imply
∥ [ Λ x ⋆ y ] − 1 − ( Γ x ⋆ y ) − 1 ∥ ≤ C L ∥ Λ x ⋆ y − Γ x ⋆ y ∥ = C L ∥ ( Γ x ⋆ y ) − 1 Λ x ⋆ y − I T x ⋆ M ∥ . (46) \left\lVert[\Lambda_{x_\star}^y]^{-1}-(\Gamma_{x_\star}^y)^{-1}\right\rVert\leq C_L\lVert\Lambda_{x_\star}^y-\Gamma_{x_\star}^y\rVert=C_L\left\lVert(\Gamma_{x_\star}^y)^{-1}\Lambda_{x_\star}^y-I_{T_{x_\star}\mathcal{M}}\right\rVert. \tag*{(46)} [ Λ x ⋆ y ] − 1 − ( Γ x ⋆ y ) − 1 ≤ C L ∥ Λ x ⋆ y − Γ x ⋆ y ∥ = C L ( Γ x ⋆ y ) − 1 Λ x ⋆ y − I T x ⋆ M . (46)
Combining this with the first bound and choosing C Γ , Λ C_{\Gamma,\Lambda} C Γ , Λ large enough proves both claims. ■
Algorithmic estimates
Recall Δ n = R x ⋆ − 1 ( x n ) \Delta_n=R_{x_\star}^{-1}(x_n) Δ n = R x ⋆ − 1 ( x n ) and Δ ~ n = R x ⋆ − 1 ( x ~ n ) \widetilde{\Delta}_n=R_{x_\star}^{-1}(\widetilde{x}_n) Δ n = R x ⋆ − 1 ( x n ) , both in T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M . In Appendices § 10 –§ 11 , we use a fixed orthonormal basis of T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M , with ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ denoting the vector or operator norm and ∥ ⋅ ∥ F \lVert\cdot\rVert_F ∥ ⋅ ∥ F the Frobenius norm. Write g n : = grad f ( x n − 1 , ξ n ) ∈ T x n − 1 M g_n:=\operatorname{grad}f(x_{n-1},\xi_n)\in T_{x_{n-1}}\mathcal{M} g n := grad f ( x n − 1 , ξ n ) ∈ T x n − 1 M and D X : = sup x ∈ X ∥ R x ⋆ − 1 ( x ) ∥ D_{\mathcal{X}}:=\sup_{x\in\mathcal{X}}\lVert R_{x_\star}^{-1}(x)\rVert D X := sup x ∈ X ∥ R x ⋆ − 1 ( x )∥ .
Lemma C.1. Let γ n = γ n − α \gamma_n=\gamma n^{-\alpha} γ n = γ n − α , where γ > 0 \gamma>0 γ > 0 and α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2,1) α ∈ ( 1/2 , 1 ) . Then, under Assumptions 2.1 and 2.3, there exist a constant C < ∞ C<\infty C < ∞ and F n \mathcal{F}_n F n -measurable vectors r Δ , n ∈ T x ⋆ M r_{\Delta,n}\in T_{x_\star}\mathcal{M} r Δ , n ∈ T x ⋆ M , n ≥ 1 n\geq1 n ≥ 1 , such that, for every n ≥ 1 n\geq1 n ≥ 1 ,
Δ n = Δ n − 1 − γ n [ Λ x ⋆ x n − 1 ] − 1 g n + γ n r Δ , n , \Delta_n=\Delta_{n-1}-\gamma_n[\Lambda_{x_\star}^{x_{n-1}}]^{-1}g_n+\gamma_n r_{\Delta,n}, Δ n = Δ n − 1 − γ n [ Λ x ⋆ x n − 1 ] − 1 g n + γ n r Δ , n ,
and, almost surely,
E n − 1 ∥ r Δ , n ∥ ≤ C γ n , E n − 1 ∥ r Δ , n ∥ 2 ≤ C γ n 2 . \mathbb{E}_{n-1}\lVert r_{\Delta,n}\rVert\leq C\gamma_n,\qquad\mathbb{E}_{n-1}\lVert r_{\Delta,n}\rVert^2\leq C\gamma_n^2. E n − 1 ∥ r Δ , n ∥ ≤ C γ n , E n − 1 ∥ r Δ , n ∥ 2 ≤ C γ n 2 .
Proof. Following the linearization argument of Tripuraneni et al. [38 ] , define r Δ , n r_{\Delta,n} r Δ , n by
r Δ , n : = γ n − 1 ( Δ n − Δ n − 1 ) + [ Λ x ⋆ x n − 1 ] − 1 g n . r_{\Delta,n}:=\gamma_n^{-1}(\Delta_n-\Delta_{n-1})+[\Lambda_{x_\star}^{x_{n-1}}]^{-1}g_n. r Δ , n := γ n − 1 ( Δ n − Δ n − 1 ) + [ Λ x ⋆ x n − 1 ] − 1 g n .
This remainder is F n \mathcal{F}_n F n -measurable and satisfies the recursion. Recall Φ x , x ⋆ = R x ⋆ − 1 ∘ R x \Phi_{x,x_\star}=R_{x_\star}^{-1}\circ R_x Φ x , x ⋆ = R x ⋆ − 1 ∘ R x from Lemma B.1 . Retraction convexity and injectivity imply t R x ⋆ − 1 ( z ) = R x ⋆ − 1 ( R x ( t R x − 1 ( z ) ) ) tR_{x_\star}^{-1}(z)=R_{x_\star}^{-1}(R_x(tR_x^{-1}(z))) t R x ⋆ − 1 ( z ) = R x ⋆ − 1 ( R x ( t R x − 1 ( z ))) with R x ( t R x − 1 ( z ) ) ∈ X R_x(tR_x^{-1}(z))\in\mathcal{X} R x ( t R x − 1 ( z )) ∈ X . The lemma therefore bounds ∥ D 2 Φ x , x ⋆ ( t R x − 1 ( z ) ) ∥ \lVert D^2\Phi_{x,x_\star}(tR_x^{-1}(z))\rVert ∥ D 2 Φ x , x ⋆ ( t R x − 1 ( z ))∥ uniformly over x , z ∈ X x,z\in\mathcal{X} x , z ∈ X and t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] by some C < ∞ C<\infty C < ∞ . The chain rule and the inverse function theorem yield
D Φ x , x ⋆ ( 0 x ) = D ( R x ⋆ − 1 ) ( x ) ∘ D R x ( 0 x ) = [ D R x ⋆ ( R x ⋆ − 1 ( x ) ) ] − 1 = [ Λ x ⋆ x ] − 1 . D\Phi_{x,x_\star}(0_x)=D(R_{x_\star}^{-1})(x)\circ D R_x(0_x)=[D R_{x_\star}(R_{x_\star}^{-1}(x))]^{-1}=[\Lambda_{x_\star}^x]^{-1}. D Φ x , x ⋆ ( 0 x ) = D ( R x ⋆ − 1 ) ( x ) ∘ D R x ( 0 x ) = [ D R x ⋆ ( R x ⋆ − 1 ( x )) ] − 1 = [ Λ x ⋆ x ] − 1 .
If γ n ∥ g n ∥ x n − 1 < ρ r e t \gamma_n\lVert g_n\rVert_{x_{n-1}}<\rho_{\mathrm{ret}} γ n ∥ g n ∥ x n − 1 < ρ ret , injectivity implies R x n − 1 − 1 ( x n ) = − γ n g n R_{x_{n-1}}^{-1}(x_n)=-\gamma_n g_n R x n − 1 − 1 ( x n ) = − γ n g n . Taylor’s formula at 0 x n − 1 0_{x_{n-1}} 0 x n − 1 then yields
Δ n = Δ n − 1 − γ n [ Λ x ⋆ x n − 1 ] − 1 g n + γ n 2 ∫ 0 1 ( 1 − t ) D 2 Φ x n − 1 , x ⋆ ( − t γ n g n ) [ g n , g n ] d t . \Delta_n=\Delta_{n-1}-\gamma_n[\Lambda_{x_\star}^{x_{n-1}}]^{-1}g_n+\gamma_n^2\int_0^1(1-t)D^2\Phi_{x_{n-1},x_\star}(-t\gamma_n g_n)[g_n,g_n]\,dt. Δ n = Δ n − 1 − γ n [ Λ x ⋆ x n − 1 ] − 1 g n + γ n 2 ∫ 0 1 ( 1 − t ) D 2 Φ x n − 1 , x ⋆ ( − t γ n g n ) [ g n , g n ] d t .
Dividing the remainder by γ n \gamma_n γ n , we obtain
∥ r Δ , n ∥ ≤ C 2 γ n ∥ g n ∥ x n − 1 2 . \lVert r_{\Delta,n}\rVert\leq\frac{C}{2}\gamma_n\lVert g_n\rVert_{x_{n-1}}^2. ∥ r Δ , n ∥ ≤ 2 C γ n ∥ g n ∥ x n − 1 2 .
If γ n ∥ g n ∥ x n − 1 ≥ ρ r e t \gamma_n\lVert g_n\rVert_{x_{n-1}} \ge\rho_{\mathrm{ret}} γ n ∥ g n ∥ x n − 1 ≥ ρ ret , compactness and Lemma B.1 give D X < ∞ D_{\mathcal{X}} < \infty D X < ∞ , and the definition of r Δ , n r_{\Delta,n} r Δ , n yields
∥ r Δ , n ∥ ≤ 2 D X γ n − 1 + C L ∥ g n ∥ x n − 1 ≤ ( 2 D X ρ r e t 2 + C L ρ r e t ) γ n ∥ g n ∥ x n − 1 2 . (47) \lVert r_{\Delta,n}\rVert\le2D_{\mathcal{X}}\gamma_n^{-1}+C_L\lVert g_n\rVert_{x_{n-1}} \le\left(\frac{2D_{\mathcal{X}}}{\rho_{\mathrm{ret}}^2}+\frac{C_L}{\rho_{\mathrm{ret}}}\right)\gamma_n\lVert g_n\rVert_{x_{n-1}}^2. \tag*{(47)} ∥ r Δ , n ∥ ≤ 2 D X γ n − 1 + C L ∥ g n ∥ x n − 1 ≤ ( ρ ret 2 2 D X + ρ ret C L ) γ n ∥ g n ∥ x n − 1 2 . (47)
Combining the two cases, we obtain
∥ r Δ , n ∥ ≤ C γ n ∥ g n ∥ x n − 1 2 . (48) \lVert r_{\Delta,n}\rVert\le C\gamma_n\lVert g_n\rVert_{x_{n-1}}^2. \tag*{(48)} ∥ r Δ , n ∥ ≤ C γ n ∥ g n ∥ x n − 1 2 . (48)
The loss is jointly measurable by continuity in x x x and measurability in the data. Its partial derivatives are limits of measurable difference quotients, so the smooth metric makes the stochastic gradient jointly measurable. Since x n − 1 x_{n-1} x n − 1 is F n − 1 \mathcal{F}_{n-1} F n − 1 -measurable and ξ n \xi_n ξ n is independent of F n − 1 \mathcal{F}_{n-1} F n − 1 , Assumption 2.3(b) implies
E n − 1 ∥ g n ∥ x n − 1 4 = E ξ ∼ D [ ∥ grad f ( x n − 1 , ξ ) ∥ x n − 1 4 ] ≤ τ 4 . (49) \mathbb{E}_{n-1}\lVert g_n\rVert_{x_{n-1}}^4=\mathbb{E}_{\xi\sim D}\left[\lVert\operatorname{grad} f(x_{n-1},\xi)\rVert_{x_{n-1}}^4\right]\le\tau^4. \tag*{(49)} E n − 1 ∥ g n ∥ x n − 1 4 = E ξ ∼ D [ ∥ grad f ( x n − 1 , ξ ) ∥ x n − 1 4 ] ≤ τ 4 . (49)
almost surely, where the expectation over ξ \xi ξ holds x n − 1 x_{n-1} x n − 1 fixed. Taking conditional expectations in the pointwise bound and its square yields, respectively,
E n − 1 ∥ r Δ , n ∥ ≤ C γ n E n − 1 ∥ g n ∥ x n − 1 2 , (50) \mathbb{E}_{n-1}\lVert r_{\Delta,n}\rVert\le C\gamma_n\mathbb{E}_{n-1}\lVert g_n\rVert_{x_{n-1}}^2, \tag*{(50)} E n − 1 ∥ r Δ , n ∥ ≤ C γ n E n − 1 ∥ g n ∥ x n − 1 2 , (50)
E n − 1 ∥ r Δ , n ∥ 2 ≤ C 2 γ n 2 E n − 1 ∥ g n ∥ x n − 1 4 . (51) \mathbb{E}_{n-1}\lVert r_{\Delta,n}\rVert^2 \le C^2\gamma_n^2\mathbb{E}_{n-1}\lVert g_n\rVert_{x_{n-1}}^4. \tag*{(51)} E n − 1 ∥ r Δ , n ∥ 2 ≤ C 2 γ n 2 E n − 1 ∥ g n ∥ x n − 1 4 . (51)
The first bound uses only the conditional second moment of g n g_n g n , whereas the second uses its conditional fourth moment. Conditional Jensen’s inequality and (C.2) bound these moments by τ 2 \tau^2 τ 2 and τ 4 \tau^4 τ 4 , respectively. Choosing a common C C C proves both claims. ■
Lemma C.2. Recall that A = Hess F ( x ⋆ ) A=\operatorname{Hess} F(x_\star) A = Hess F ( x ⋆ ) . Under Assumptions 2.1–2.3, there exist a constant C r , F < ∞ C_{r,F}<\infty C r , F < ∞ , F n \mathcal{F}_n F n -measurable vectors ϵ ~ n ∈ T x ⋆ M \widetilde{\epsilon}_n\in T_{x_\star}\mathcal{M} ϵ n ∈ T x ⋆ M , and F n − 1 \mathcal{F}_{n-1} F n − 1 -measurable vectors r F , n ∈ T x ⋆ M r_{F,n}\in T_{x_\star}\mathcal{M} r F , n ∈ T x ⋆ M , n ≥ 1 n\ge1 n ≥ 1 , such that, for every n ≥ 1 n\ge1 n ≥ 1 ,
( Γ x ⋆ x n − 1 ) − 1 g n = A Δ n − 1 + grad f ( x ⋆ , ξ n ) + ϵ ~ n + r F , n , (\Gamma_{x_\star}^{x_{n-1}})^{-1}g_n=A\Delta_{n-1}+\operatorname{grad} f(x_\star,\xi_n)+\widetilde{\epsilon}_n+r_{F,n}, ( Γ x ⋆ x n − 1 ) − 1 g n = A Δ n − 1 + grad f ( x ⋆ , ξ n ) + ϵ n + r F , n , and, almost surely,
E n − 1 ϵ ~ n = 0 , E n − 1 ∥ ϵ ~ n ∥ 2 ≤ L g 2 ∥ Δ n − 1 ∥ 2 , ∥ r F , n ∥ ≤ C r , F ∥ Δ n − 1 ∥ 2 . \mathbb{E}_{n-1}\widetilde{\epsilon}_n=0,\qquad\mathbb{E}_{n-1}\lVert\widetilde{\epsilon}_n\rVert^2\le L_g^2\lVert\Delta_{n-1}\rVert^2,\qquad\lVert r_{F,n}\rVert\le C_{r,F}\lVert\Delta_{n-1}\rVert^2. E n − 1 ϵ n = 0 , E n − 1 ∥ ϵ n ∥ 2 ≤ L g 2 ∥ Δ n − 1 ∥ 2 , ∥ r F , n ∥ ≤ C r , F ∥ Δ n − 1 ∥ 2 . Proof. Define the centered perturbation of the stochastic gradient by
ϵ ~ n : = ( Γ x ⋆ x n − 1 ) − 1 g n − grad f ( x ⋆ , ξ n ) − ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) . \widetilde{\epsilon}_n:=(\Gamma_{x_\star}^{x_{n-1}})^{-1}g_n-\operatorname{grad} f(x_\star,\xi_n)-(\Gamma_{x_\star}^{x_{n-1}})^{-1}\operatorname{grad} F(x_{n-1}). ϵ n := ( Γ x ⋆ x n − 1 ) − 1 g n − grad f ( x ⋆ , ξ n ) − ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) . Conditioning on F n − 1 \mathcal{F}_{n-1} F n − 1 as in the preceding proof, Assumption 2.3(a) yields
E n − 1 g n = E ξ ∼ D [ grad f ( x n − 1 , ξ ) ] = grad F ( x n − 1 ) . (52) \mathbb{E}_{n-1}g_n=\mathbb{E}_{\xi\sim D}[\operatorname{grad} f(x_{n-1},\xi)]=\operatorname{grad} F(x_{n-1}). \tag*{(52)} E n − 1 g n = E ξ ∼ D [ grad f ( x n − 1 , ξ )] = grad F ( x n − 1 ) . (52) The fourth-moment bound ensures integrability, and the inverse transport is F n − 1 \mathcal{F}_{n-1} F n − 1 -measurable and isometric. Thus (C.3) and grad F ( x ⋆ ) = 0 \operatorname{grad} F(x_\star)=0 grad F ( x ⋆ ) = 0 imply
E n − 1 ϵ ~ n = ( Γ x ⋆ x n − 1 ) − 1 E n − 1 g n − E n − 1 grad f ( x ⋆ , ξ n ) − ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) \mathbb{E}_{n-1}\widetilde{\epsilon}_n=(\Gamma_{x_\star}^{x_{n-1}})^{-1}\mathbb{E}_{n-1}g_n-\mathbb{E}_{n-1}\operatorname{grad} f(x_\star,\xi_n)-(\Gamma_{x_\star}^{x_{n-1}})^{-1}\operatorname{grad} F(x_{n-1}) E n − 1 ϵ n = ( Γ x ⋆ x n − 1 ) − 1 E n − 1 g n − E n − 1 grad f ( x ⋆ , ξ n ) − ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) = ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) − grad F ( x ⋆ ) − ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) = 0. =(\Gamma_{x_\star}^{x_{n-1}})^{-1}\operatorname{grad} F(x_{n-1})-\operatorname{grad} F(x_\star)-(\Gamma_{x_\star}^{x_{n-1}})^{-1}\operatorname{grad} F(x_{n-1})=0. = ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) − grad F ( x ⋆ ) − ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) = 0.
For the conditional second moment, we have
E n − 1 ∥ ϵ ~ n ∥ 2 = E n − 1 ∥ ( Γ x ⋆ x n − 1 ) − 1 g n − grad f ( x ⋆ , ξ n ) ∥ 2 − ∥ ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) ∥ 2 ≤ E n − 1 ∥ ( Γ x ⋆ x n − 1 ) − 1 g n − grad f ( x ⋆ , ξ n ) ∥ 2 ≤ L g 2 ∥ Δ n − 1 ∥ 2 . \mathbb{E}_{n-1}\|\tilde{\epsilon}_n\|^2 = \mathbb{E}_{n-1}\left\|\left(\Gamma_{x_\star}^{x_{n-1}}\right)^{-1}g_n-\operatorname{grad}f(x_\star,\xi_n)\right\|^2-\left\|\left(\Gamma_{x_\star}^{x_{n-1}}\right)^{-1}\operatorname{grad}F(x_{n-1})\right\|^2
\leq\mathbb{E}_{n-1}\left\|\left(\Gamma_{x_\star}^{x_{n-1}}\right)^{-1}g_n-\operatorname{grad}f(x_\star,\xi_n)\right\|^2 \leq L_g^2\|\Delta_{n-1}\|^2. E n − 1 ∥ ϵ ~ n ∥ 2 = E n − 1 ( Γ x ⋆ x n − 1 ) − 1 g n − grad f ( x ⋆ , ξ n ) 2 − ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) 2 ≤ E n − 1 ( Γ x ⋆ x n − 1 ) − 1 g n − grad f ( x ⋆ , ξ n ) 2 ≤ L g 2 ∥ Δ n − 1 ∥ 2 .
The equality is the conditional variance identity; the last inequality uses Assumption 2.3(c) at x n − 1 x_{n-1} x n − 1 . To expand grad F \operatorname{grad}F grad F , set y ( t ) = y t : = R x ⋆ ( t Δ n − 1 ) y(t)=y_t:=R_{x_\star}(t\Delta_{n-1}) y ( t ) = y t := R x ⋆ ( t Δ n − 1 ) for t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] . Retraction convexity and injectivity imply y t ∈ X y_t\in\mathcal{X} y t ∈ X and R x ⋆ − 1 ( y t ) = t Δ n − 1 R_{x_\star}^{-1}(y_t)=t\Delta_{n-1} R x ⋆ − 1 ( y t ) = t Δ n − 1 . Since Γ x ⋆ y t \Gamma_{x_\star}^{y_t} Γ x ⋆ y t follows the subcurve s ↦ y ( t s ) s\mapsto y(ts) s ↦ y ( t s ) , differentiation yields
d d t ( ( Γ x ⋆ y t ) − 1 grad F ( y t ) ) = ( Γ x ⋆ y t ) − 1 ∘ Hess F ( y t ) ∘ Λ x ⋆ y t Δ n − 1 . \frac{\mathrm{d}}{\mathrm{d}t}\left(\left(\Gamma_{x_\star}^{y_t}\right)^{-1}\operatorname{grad}F(y_t)\right)=\left(\Gamma_{x_\star}^{y_t}\right)^{-1}\circ\operatorname{Hess}F(y_t)\circ\Lambda_{x_\star}^{y_t}\Delta_{n-1}. d t d ( ( Γ x ⋆ y t ) − 1 grad F ( y t ) ) = ( Γ x ⋆ y t ) − 1 ∘ Hess F ( y t ) ∘ Λ x ⋆ y t Δ n − 1 .
Since grad F ( x ⋆ ) = 0 \operatorname{grad}F(x_\star)=0 grad F ( x ⋆ ) = 0 and y 1 = x n − 1 y_1=x_{n-1} y 1 = x n − 1 , the fundamental theorem of calculus yields
( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) = ∫ 0 1 ( Γ x ⋆ y t ) − 1 ∘ Hess F ( y t ) ∘ Λ x ⋆ y t Δ n − 1 d t . \left(\Gamma_{x_\star}^{x_{n-1}}\right)^{-1}\operatorname{grad}F(x_{n-1})=\int_0^1\left(\Gamma_{x_\star}^{y_t}\right)^{-1}\circ\operatorname{Hess}F(y_t)\circ\Lambda_{x_\star}^{y_t}\Delta_{n-1}\,\mathrm{d}t. ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) = ∫ 0 1 ( Γ x ⋆ y t ) − 1 ∘ Hess F ( y t ) ∘ Λ x ⋆ y t Δ n − 1 d t .
Define the F n − 1 \mathcal{F}_{n-1} F n − 1 -measurable remainder r F , n : = ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) − A Δ n − 1 r_{F,n}:=\left(\Gamma_{x_\star}^{x_{n-1}}\right)^{-1}\operatorname{grad}F(x_{n-1})-A\Delta_{n-1} r F , n := ( Γ x ⋆ x n − 1 ) − 1 grad F ( x n − 1 ) − A Δ n − 1 . Adding and subtracting Γ x ⋆ y t \Gamma_{x_\star}^{y_t} Γ x ⋆ y t yields
r F , n = ∫ 0 1 ( ( Γ x ⋆ y t ) − 1 ∘ Hess F ( y t ) ∘ Γ x ⋆ y t − A ) Δ n − 1 d t + ∫ 0 1 ( Γ x ⋆ y t ) − 1 ∘ Hess F ( y t ) ∘ ( Λ x ⋆ y t − Γ x ⋆ y t ) Δ n − 1 d t . r_{F,n}=\int_0^1\left(\left(\Gamma_{x_\star}^{y_t}\right)^{-1}\circ\operatorname{Hess}F(y_t)\circ\Gamma_{x_\star}^{y_t}-A\right)\Delta_{n-1}\,\mathrm{d}t
+\int_0^1\left(\Gamma_{x_\star}^{y_t}\right)^{-1}\circ\operatorname{Hess}F(y_t)\circ\left(\Lambda_{x_\star}^{y_t}-\Gamma_{x_\star}^{y_t}\right)\Delta_{n-1}\,\mathrm{d}t. r F , n = ∫ 0 1 ( ( Γ x ⋆ y t ) − 1 ∘ Hess F ( y t ) ∘ Γ x ⋆ y t − A ) Δ n − 1 d t + ∫ 0 1 ( Γ x ⋆ y t ) − 1 ∘ Hess F ( y t ) ∘ ( Λ x ⋆ y t − Γ x ⋆ y t ) Δ n − 1 d t .
Assumption 2.2 (c) bounds the transported Hessian by ∥ A ∥ + L H D X \|A\|+L_HD_{\mathcal{X}} ∥ A ∥ + L H D X and the first integrand by L H t ∥ Δ n − 1 ∥ 2 L_Ht\|\Delta_{n-1}\|^2 L H t ∥ Δ n − 1 ∥ 2 . For the second integrand,
∥ ( Γ x ⋆ y t ) − 1 Λ x ⋆ y t − I ∥ ≤ C Γ , Λ t 2 ∥ Δ n − 1 ∥ 2 (53) \left\|\left(\Gamma_{x_\star}^{y_t}\right)^{-1}\Lambda_{x_\star}^{y_t}-I\right\|\leq C_{\Gamma,\Lambda}t^2\|\Delta_{n-1}\|^2 \tag*{(53)} ( Γ x ⋆ y t ) − 1 Λ x ⋆ y t − I ≤ C Γ , Λ t 2 ∥ Δ n − 1 ∥ 2 (53)
Thus, using ∥ Δ n − 1 ∥ ≤ D X \|\Delta_{n-1}\|\leq D_{\mathcal{X}} ∥ Δ n − 1 ∥ ≤ D X ,
∥ r F , n ∥ ≤ ∫ 0 1 L H t ∥ Δ n − 1 ∥ 2 d t + ∫ 0 1 ( ∥ A ∥ + L H D X ) C Γ , Λ t 2 ∥ Δ n − 1 ∥ 3 d t ≤ ( L H 2 + ( ∥ A ∥ + L H D X ) C Γ , Λ D X 3 ) ∥ Δ n − 1 ∥ 2 = : C r , F ∥ Δ n − 1 ∥ 2 . \begin{aligned}
\|r_{F,n}\|&\leq\int_0^1L_Ht\|\Delta_{n-1}\|^2\,\mathrm{d}t+\int_0^1(\|A\|+L_HD_{\mathcal{X}})C_{\Gamma,\Lambda}t^2\|\Delta_{n-1}\|^3\,\mathrm{d}t\\
&\leq\left(\frac{L_H}{2}+\frac{(\|A\|+L_HD_{\mathcal{X}})C_{\Gamma,\Lambda}D_{\mathcal{X}}}{3}\right)\|\Delta_{n-1}\|^2=:C_{r,F}\|\Delta_{n-1}\|^2.
\end{aligned} ∥ r F , n ∥ ≤ ∫ 0 1 L H t ∥ Δ n − 1 ∥ 2 d t + ∫ 0 1 ( ∥ A ∥ + L H D X ) C Γ , Λ t 2 ∥ Δ n − 1 ∥ 3 d t ≤ ( 2 L H + 3 ( ∥ A ∥ + L H D X ) C Γ , Λ D X ) ∥ Δ n − 1 ∥ 2 =: C r , F ∥ Δ n − 1 ∥ 2 .
Combining the definitions of ϵ ~ n \tilde{\epsilon}_n ϵ ~ n and r F , n r_{F,n} r F , n proves the expansion. ■
Theorem C.3. Under the assumptions of Lemma C.2 , let γ n = γ n − α \gamma_n=\gamma n^{-\alpha} γ n = γ n − α , where γ > 0 \gamma>0 γ > 0 and α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2,1) α ∈ ( 1/2 , 1 ) . Then there exist F n \mathcal{F}_n F n -measurable sequences { ϵ n } n ≥ 1 \{\epsilon_n\}_{n\geq1} { ϵ n } n ≥ 1 and { e n } n ≥ 1 \{e_n\}_{n\geq1} { e n } n ≥ 1 in T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M and a constant C < ∞ C<\infty C < ∞ such that, for every n ≥ 1 n\geq1 n ≥ 1 ,
Δ n = ( I − γ n A ) Δ n − 1 + γ n ( ζ n + ϵ n + e n ) , (54) \Delta_n=(I-\gamma_nA)\Delta_{n-1}+\gamma_n(\zeta_n+\epsilon_n+e_n), \tag*{(54)} Δ n = ( I − γ n A ) Δ n − 1 + γ n ( ζ n + ϵ n + e n ) , (54) where ζ n : = − grad f ( x ⋆ , ξ n ) \zeta_n:=-\operatorname{grad}f(x_\star,\xi_n) ζ n := − grad f ( x ⋆ , ξ n ) . For every n ≥ 1 n\geq1 n ≥ 1 , the three terms satisfy
E n − 1 ζ n = 0 , E n − 1 ( ζ n ⊗ 2 ) = S , E n − 1 ∥ ζ n ∥ 4 ≤ τ 4 , (55) \mathbb{E}_{n-1}\zeta_n=0,\qquad\mathbb{E}_{n-1}(\zeta_n^{\otimes2})=S,\qquad\mathbb{E}_{n-1}\|\zeta_n\|^4\leq\tau^4, \tag*{(55)} E n − 1 ζ n = 0 , E n − 1 ( ζ n ⊗ 2 ) = S , E n − 1 ∥ ζ n ∥ 4 ≤ τ 4 , (55) E n − 1 ϵ n = 0 , E n − 1 ∥ ϵ n ∥ 2 ≤ C ∥ Δ n − 1 ∥ 2 , (56) \mathbb{E}_{n-1}\epsilon_n=0,\qquad\mathbb{E}_{n-1}\|\epsilon_n\|^2\leq C\|\Delta_{n-1}\|^2, \tag*{(56)} E n − 1 ϵ n = 0 , E n − 1 ∥ ϵ n ∥ 2 ≤ C ∥ Δ n − 1 ∥ 2 , (56) and
E n − 1 ∥ e n ∥ ≤ C ( ∥ Δ n − 1 ∥ 2 + γ n ) , E n − 1 ∥ e n ∥ 2 ≤ C ( ∥ Δ n − 1 ∥ 4 + γ n 2 ) . (57) \mathbb{E}_{n-1}\|e_n\|\leq C(\|\Delta_{n-1}\|^2+\gamma_n),\qquad\mathbb{E}_{n-1}\|e_n\|^2\leq C(\|\Delta_{n-1}\|^4+\gamma_n^2). \tag*{(57)} E n − 1 ∥ e n ∥ ≤ C ( ∥ Δ n − 1 ∥ 2 + γ n ) , E n − 1 ∥ e n ∥ 2 ≤ C ( ∥ Δ n − 1 ∥ 4 + γ n 2 ) . (57)
Proof. Since Λ x ⋆ x n − 1 \Lambda_{x_\star}^{x_{n-1}} Λ x ⋆ x n − 1 is an isometry,
∥ [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ∥ ≤ C Γ , Λ ∥ Δ n − 1 ∥ 2 . (58) \left\|\left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}\Gamma_{x_\star}^{x_{n-1}}-I\right\|\le C_{\Gamma,\Lambda}\|\Delta_{n-1}\|^2. \tag*{(58)} [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ≤ C Γ , Λ ∥ Δ n − 1 ∥ 2 . (58)
Define
ϵ n : = − ϵ ~ n − ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) ( grad f ( x ⋆ , ξ n ) + ϵ ~ n ) , \epsilon_n:=-\widetilde{\epsilon}_n-\left(\left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}\Gamma_{x_\star}^{x_{n-1}}-I\right)\left(\operatorname{grad} f(x_\star,\xi_n)+\widetilde{\epsilon}_n\right), ϵ n := − ϵ n − ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) ( grad f ( x ⋆ , ξ n ) + ϵ n ) ,
and
e n : = r Δ , n − r F , n − ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) ( A Δ n − 1 + r F , n ) . e_n:=r_{\Delta,n}-r_{F,n}-\left(\left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}\Gamma_{x_\star}^{x_{n-1}}-I\right)\left(A\Delta_{n-1}+r_{F,n}\right). e n := r Δ , n − r F , n − ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) ( A Δ n − 1 + r F , n ) .
Both sequences are F n \mathcal{F}_n F n -measurable. Substituting
[ Λ x ⋆ x n − 1 ] − 1 = [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 ( Γ x ⋆ x n − 1 ) − 1 , \left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}=\left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}\Gamma_{x_\star}^{x_{n-1}}\left(\Gamma_{x_\star}^{x_{n-1}}\right)^{-1}, [ Λ x ⋆ x n − 1 ] − 1 = [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 ( Γ x ⋆ x n − 1 ) − 1 ,
into the recursion for Δ n \Delta_n Δ n and using the gradient expansion in Lemma C.2, we obtain
Δ n = Δ n − 1 + γ n r Δ , n − γ n [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 ( A Δ n − 1 + grad f ( x ⋆ , ξ n ) + ϵ ~ n + r F , n ) = ( I − γ n A ) Δ n − 1 − γ n grad f ( x ⋆ , ξ n ) + γ n ( r Δ , n − ϵ ~ n − r F , n ) − γ n ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) ( A Δ n − 1 + grad f ( x ⋆ , ξ n ) + ϵ ~ n + r F , n ) = ( I − γ n A ) Δ n − 1 + γ n ( ζ n + ϵ n + e n ) . \begin{aligned}
\Delta_n={}&\Delta_{n-1}+\gamma_n r_{\Delta,n}-\gamma_n\left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}\Gamma_{x_\star}^{x_{n-1}}\left(A\Delta_{n-1}+\operatorname{grad} f(x_\star,\xi_n)+\widetilde{\epsilon}_n+r_{F,n}\right)\\
={}&(I-\gamma_n A)\Delta_{n-1}-\gamma_n\operatorname{grad} f(x_\star,\xi_n)+\gamma_n(r_{\Delta,n}-\widetilde{\epsilon}_n-r_{F,n})\\
&-\gamma_n\left(\left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}\Gamma_{x_\star}^{x_{n-1}}-I\right)\left(A\Delta_{n-1}+\operatorname{grad} f(x_\star,\xi_n)+\widetilde{\epsilon}_n+r_{F,n}\right)\\
={}&(I-\gamma_n A)\Delta_{n-1}+\gamma_n(\zeta_n+\epsilon_n+e_n).
\end{aligned} Δ n = = = Δ n − 1 + γ n r Δ , n − γ n [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 ( A Δ n − 1 + grad f ( x ⋆ , ξ n ) + ϵ n + r F , n ) ( I − γ n A ) Δ n − 1 − γ n grad f ( x ⋆ , ξ n ) + γ n ( r Δ , n − ϵ n − r F , n ) − γ n ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) ( A Δ n − 1 + grad f ( x ⋆ , ξ n ) + ϵ n + r F , n ) ( I − γ n A ) Δ n − 1 + γ n ( ζ n + ϵ n + e n ) .
which proves (C.4). Assumption 2.3(a), (b), and (d) at x ⋆ x_\star x ⋆ , together with grad F ( x ⋆ ) = 0 \operatorname{grad} F(x_\star)=0 grad F ( x ⋆ ) = 0 from Assumption 2.2(a), imply (C.5). For ϵ n \epsilon_n ϵ n , the operator
[ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I \left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}\Gamma_{x_\star}^{x_{n-1}}-I [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I
is F n − 1 \mathcal{F}_{n-1} F n − 1 -measurable. Assumption 2.3(a) and Lemma C.2 therefore imply
E n − 1 ϵ n = − E n − 1 ϵ ~ n − ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) E n − 1 [ grad f ( x ⋆ , ξ n ) + ϵ ~ n ] = − 0 − ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) ( grad F ( x ⋆ ) + 0 ) = 0. \begin{aligned}
\mathbb{E}_{n-1}\epsilon_n
&=-\mathbb{E}_{n-1}\widetilde{\epsilon}_n-\left(\left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}\Gamma_{x_\star}^{x_{n-1}}-I\right)\mathbb{E}_{n-1}\left[\operatorname{grad} f(x_\star,\xi_n)+\widetilde{\epsilon}_n\right]\\
&=-0-\left(\left[\Lambda_{x_\star}^{x_{n-1}}\right]^{-1}\Gamma_{x_\star}^{x_{n-1}}-I\right)\left(\operatorname{grad} F(x_\star)+0\right)=0.
\end{aligned} E n − 1 ϵ n = − E n − 1 ϵ n − ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) E n − 1 [ grad f ( x ⋆ , ξ n ) + ϵ n ] = − 0 − ( [ Λ x ⋆ x n − 1 ] − 1 Γ x ⋆ x n − 1 − I ) ( grad F ( x ⋆ ) + 0 ) = 0.
Conditional Cauchy–Schwarz bounds the noise second moment by τ 2 \tau^2 τ 2 . Using Lemma C.2 and ∥ Δ n − 1 ∥ ≤ D X \|\Delta_{n-1}\|\le D_X ∥ Δ n − 1 ∥ ≤ D X , we obtain
E n − 1 ∥ grad f ( x ⋆ , ξ n ) + ϵ ~ n ∥ 2 ≤ 2 τ 2 + 2 L g 2 D X 2 . \mathbb{E}_{n-1}\left\|\operatorname{grad} f(x_\star,\xi_n)+\widetilde{\epsilon}_n\right\|^2\le2\tau^2+2L_g^2D_X^2. E n − 1 ∥ grad f ( x ⋆ , ξ n ) + ϵ n ∥ 2 ≤ 2 τ 2 + 2 L g 2 D X 2 .
The transport bound then yields
E n − 1 ∥ ϵ n ∥ 2 ≤ 2 L g 2 ∥ Δ n − 1 ∥ 2 + C ∥ Δ n − 1 ∥ 4 ≤ ( 2 L g 2 + C D X 2 ) ∥ Δ n − 1 ∥ 2 . \mathbb{E}_{n-1}\|\epsilon_n\|^2\le2L_g^2\|\Delta_{n-1}\|^2+C\|\Delta_{n-1}\|^4\le(2L_g^2+CD_X^2)\|\Delta_{n-1}\|^2. E n − 1 ∥ ϵ n ∥ 2 ≤ 2 L g 2 ∥ Δ n − 1 ∥ 2 + C ∥ Δ n − 1 ∥ 4 ≤ ( 2 L g 2 + C D X 2 ) ∥ Δ n − 1 ∥ 2 .
For e n e_n e n , the remainder and transport bounds, together with ∥ Δ n − 1 ∥ ≤ D X \|\Delta_{n-1}\|\le D_X ∥ Δ n − 1 ∥ ≤ D X , imply
∥ e n − r Δ , n ∥ ≤ C r , F ∥ Δ n − 1 ∥ 2 + C Γ , Λ ∥ Δ n − 1 ∥ 2 ( ∥ A ∥ ∥ Δ n − 1 ∥ + C r , F ∥ Δ n − 1 ∥ 2 ) ≤ C ∥ Δ n − 1 ∥ 2 . \|e_n-r_{\Delta,n}\|\le C_{r,F}\|\Delta_{n-1}\|^2+C_{\Gamma,\Lambda}\|\Delta_{n-1}\|^2(\|A\|\|\Delta_{n-1}\|+C_{r,F}\|\Delta_{n-1}\|^2)\le C\|\Delta_{n-1}\|^2. ∥ e n − r Δ , n ∥ ≤ C r , F ∥ Δ n − 1 ∥ 2 + C Γ , Λ ∥ Δ n − 1 ∥ 2 ( ∥ A ∥∥ Δ n − 1 ∥ + C r , F ∥ Δ n − 1 ∥ 2 ) ≤ C ∥ Δ n − 1 ∥ 2 .
Lemma C.1 now implies
E n − 1 ∥ e n ∥ ≤ C ( ∥ Δ n − 1 ∥ 2 + γ n ) , \mathbb{E}_{n-1}\|e_n\|\le C(\|\Delta_{n-1}\|^2+\gamma_n), E n − 1 ∥ e n ∥ ≤ C ( ∥ Δ n − 1 ∥ 2 + γ n ) ,
E n − 1 ∥ e n ∥ 2 ≤ C ( ∥ Δ n − 1 ∥ 4 + γ n 2 ) . \mathbb{E}_{n-1}\|e_n\|^2\le C(\|\Delta_{n-1}\|^4+\gamma_n^2). E n − 1 ∥ e n ∥ 2 ≤ C ( ∥ Δ n − 1 ∥ 4 + γ n 2 ) .
Choosing a common C C C completes the proof. ■
We now compare the perturbed recursion with its linear part. Under Assumption 2.4 (a) and (b), respectively, choose C Δ , 2 , C Δ , 4 < ∞ C_{\Delta,2}, C_{\Delta,4} < \infty C Δ , 2 , C Δ , 4 < ∞ , enlarged to cover the finitely many initial indices, such that, for every n ≥ 1 n \ge1 n ≥ 1 ,
E ∥ Δ n ∥ 2 ≤ C Δ , 2 γ n , E ∥ Δ n ∥ 4 ≤ C Δ , 4 γ n 2 . (59) \mathbb{E}\lVert\Delta_n\rVert^2 \le C_{\Delta,2}\gamma_n,\qquad\mathbb{E}\lVert\Delta_n\rVert^4 \le C_{\Delta,4}\gamma_n^2. \tag*{(59)} E ∥ Δ n ∥ 2 ≤ C Δ , 2 γ n , E ∥ Δ n ∥ 4 ≤ C Δ , 4 γ n 2 . (59)
Lemma C.4. Under the assumptions of Theorem C.3 and Assumption 2.4 , define the linear recursion in T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M by U 0 : = Δ 0 U_0 := \Delta_0 U 0 := Δ 0 and
U n = ( I − γ n A ) U n − 1 + γ n ζ n , n ≥ 1. (60) U_n = (I-\gamma_n A)U_{n-1}+\gamma_n\zeta_n,\qquad n\ge1. \tag*{(60)} U n = ( I − γ n A ) U n − 1 + γ n ζ n , n ≥ 1. (60) Set d n : = Δ n − U n d_n := \Delta_n-U_n d n := Δ n − U n for n ≥ 0 n\ge0 n ≥ 0 . Then there exists a constant C < ∞ C<\infty C < ∞ such that, for every n ≥ 1 n\ge1 n ≥ 1 ,
( E ∥ d n ∥ 2 ) 1 / 2 ≤ C n − α . \left(\mathbb{E}\lVert d_n\rVert^2\right)^{1/2}\le Cn^{-\alpha}. ( E ∥ d n ∥ 2 ) 1/2 ≤ C n − α .
Proof. Subtracting the linear recursion (C.9) from the perturbed recursion (C.4), we have d 0 = 0 d_0=0 d 0 = 0 and
d n = ( I − γ n A ) d n − 1 + γ n ( ϵ n + e n ) . d_n=(I-\gamma_n A)d_{n-1}+\gamma_n(\epsilon_n+e_n). d n = ( I − γ n A ) d n − 1 + γ n ( ϵ n + e n ) .
Taking expectations of the conditional bounds yields, for n ≥ 2 n\ge2 n ≥ 2 ,
E ∥ ϵ n ∥ 2 = E [ E n − 1 ∥ ϵ n ∥ 2 ] ≤ C E ∥ Δ n − 1 ∥ 2 , (61) \mathbb{E}\lVert\epsilon_n\rVert^2=\mathbb{E}\left[\mathbb{E}_{n-1}\lVert\epsilon_n\rVert^2\right]\le C\mathbb{E}\lVert\Delta_{n-1}\rVert^2, \tag*{(61)} E ∥ ϵ n ∥ 2 = E [ E n − 1 ∥ ϵ n ∥ 2 ] ≤ C E ∥ Δ n − 1 ∥ 2 , (61)
E ∥ e n ∥ 2 = E [ E n − 1 ∥ e n ∥ 2 ] ≤ C ( E ∥ Δ n − 1 ∥ 4 + γ n 2 ) . (62) \mathbb{E}\lVert e_n\rVert^2=\mathbb{E}\left[\mathbb{E}_{n-1}\lVert e_n\rVert^2\right]\le C\left(\mathbb{E}\lVert\Delta_{n-1}\rVert^4+\gamma_n^2\right). \tag*{(62)} E ∥ e n ∥ 2 = E [ E n − 1 ∥ e n ∥ 2 ] ≤ C ( E ∥ Δ n − 1 ∥ 4 + γ n 2 ) . (62)
Combining these bounds with (C.8) and γ n − 1 ≤ 2 α γ n \gamma_{n-1}\le2^\alpha\gamma_n γ n − 1 ≤ 2 α γ n yields
E ∥ ϵ n ∥ 2 ≤ C n − α , E ∥ e n ∥ 2 ≤ C n − 2 α . (63) \mathbb{E}\lVert\epsilon_n\rVert^2\le Cn^{-\alpha},\qquad\mathbb{E}\lVert e_n\rVert^2\le Cn^{-2\alpha}. \tag*{(63)} E ∥ ϵ n ∥ 2 ≤ C n − α , E ∥ e n ∥ 2 ≤ C n − 2 α . (63)
The case n = 1 n=1 n = 1 is covered by ∥ Δ 0 ∥ ≤ D X \lVert\Delta_0\rVert\le D_X ∥ Δ 0 ∥ ≤ D X . Let us recall Y i n : = ∏ k = i + 1 n ( I − γ k A ) Y_i^n:=\prod_{k=i+1}^n(I-\gamma_kA) Y i n := ∏ k = i + 1 n ( I − γ k A ) and Y n n = I Y_n^n=I Y n n = I ; then the iteration yields
d n = ∑ i = 1 n Y i n γ i ϵ i + ∑ i = 1 n Y i n γ i e i . d_n=\sum_{i=1}^nY_i^n\gamma_i\epsilon_i+\sum_{i=1}^nY_i^n\gamma_i e_i. d n = i = 1 ∑ n Y i n γ i ϵ i + i = 1 ∑ n Y i n γ i e i .
For i < j i<j i < j , past measurability and conditional centering imply
E ⟨ Y i n γ i ϵ i , Y j n γ j ϵ j ⟩ = E ⟨ Y i n γ i ϵ i , Y j n γ j E j − 1 ϵ j ⟩ = 0. \mathbb{E}\langle Y_i^n\gamma_i\epsilon_i,Y_j^n\gamma_j\epsilon_j\rangle
=\mathbb{E}\langle Y_i^n\gamma_i\epsilon_i,Y_j^n\gamma_j\mathbb{E}_{j-1}\epsilon_j\rangle=0. E ⟨ Y i n γ i ϵ i , Y j n γ j ϵ j ⟩ = E ⟨ Y i n γ i ϵ i , Y j n γ j E j − 1 ϵ j ⟩ = 0.
Using this orthogonality, we obtain
E ∥ ∑ i = 1 n Y i n γ i ϵ i ∥ 2 = ∑ i = 1 n γ i 2 E ∥ Y i n ϵ i ∥ 2 ≤ C ∑ i = 1 n ∥ Y i n ∥ 2 i − 3 α . \mathbb{E}\left\lVert\sum_{i=1}^nY_i^n\gamma_i\epsilon_i\right\rVert^2
=\sum_{i=1}^n\gamma_i^2\mathbb{E}\lVert Y_i^n\epsilon_i\rVert^2
\le C\sum_{i=1}^n\lVert Y_i^n\rVert^2i^{-3\alpha}. E i = 1 ∑ n Y i n γ i ϵ i 2 = i = 1 ∑ n γ i 2 E ∥ Y i n ϵ i ∥ 2 ≤ C i = 1 ∑ n ∥ Y i n ∥ 2 i − 3 α .
The inequality combines (C.10) with (A.1), which implies ∥ Y i n ∥ 2 ≤ C ∥ Y n n ∥ \lVert Y_i^n\rVert^2\le C\lVert Y_n^n\rVert ∥ Y i n ∥ 2 ≤ C ∥ Y n n ∥ . For the remainder sum, Minkowski’s inequality yields
( E ∥ ∑ i = 1 n Y i n γ i e i ∥ 2 ) 1 / 2 ≤ ∑ i = 1 n ∥ Y i n ∥ γ i ( E ∥ e i ∥ 2 ) 1 / 2 ≤ C ∑ i = 1 n ∥ Y i n ∥ i − 2 α . (64) \left(\mathbb{E}\left\lVert\sum_{i=1}^nY_i^n\gamma_i e_i\right\rVert^2\right)^{1/2}
\le\sum_{i=1}^n\lVert Y_i^n\rVert\gamma_i\left(\mathbb{E}\lVert e_i\rVert^2\right)^{1/2}
\le C\sum_{i=1}^n\lVert Y_i^n\rVert i^{-2\alpha}. \tag*{(64)} E i = 1 ∑ n Y i n γ i e i 2 1/2 ≤ i = 1 ∑ n ∥ Y i n ∥ γ i ( E ∥ e i ∥ 2 ) 1/2 ≤ C i = 1 ∑ n ∥ Y i n ∥ i − 2 α . (64)
Absorbing the fixed factors for q = 3 α q = 3\alpha q = 3 α and q = 2 α q = 2\alpha q = 2 α into C C C , we obtain
E ∥ ∑ i = 1 n Y i n γ i ϵ i ∥ 2 ≤ (A.2) C n − 2 α , ( E ∥ ∑ i = 1 n Y i n γ i ϵ i ∥ 2 ) 1 / 2 ≤ (A.2) C n − α . \mathbb{E}\left\lVert\sum_{i=1}^{n}Y_i^n\gamma_i\epsilon_i\right\rVert^2 \overset{\text{(A.2)}}{\le} Cn^{-2\alpha}, \qquad\left(\mathbb{E}\left\lVert\sum_{i=1}^{n}Y_i^n\gamma_i\epsilon_i\right\rVert^2\right)^{1/2} \overset{\text{(A.2)}}{\le} Cn^{-\alpha}. E i = 1 ∑ n Y i n γ i ϵ i 2 ≤ (A.2) C n − 2 α , E i = 1 ∑ n Y i n γ i ϵ i 2 1/2 ≤ (A.2) C n − α .
Taking the square root of the first bound and adding the second proves the claim. ■
We next compare arithmetic and recursive averages over an interval. For each s ≥ 1 s \ge1 s ≥ 1 , set c s : s : = x s c_{s:s} := x_s c s : s := x s and, for j > s j > s j > s , define
c s : j : = R c s : j − 1 ( 1 j − s + 1 R c s : j − 1 − 1 ( x j ) ) . c_{s:j} := R_{c_{s:j-1}}\left(\frac{1}{j-s+1}R_{c_{s:j-1}}^{-1}(x_j)\right). c s : j := R c s : j − 1 ( j − s + 1 1 R c s : j − 1 − 1 ( x j ) ) .
For 1 ≤ s ≤ j 1 \le s \le j 1 ≤ s ≤ j , write
Δ ˉ s : j : = 1 j − s + 1 ∑ k = s j Δ k , Δ ~ s : j : = R x ∗ − 1 ( c s : j ) . \bar{\Delta}_{s:j} := \frac{1}{j-s+1}\sum_{k=s}^{j}\Delta_k,\qquad\widetilde{\Delta}_{s:j} := R_{x_*}^{-1}(c_{s:j}). Δ ˉ s : j := j − s + 1 1 k = s ∑ j Δ k , Δ s : j := R x ∗ − 1 ( c s : j ) .
Thus Δ ˉ 1 : n = Δ ˉ n \bar{\Delta}_{1:n}=\bar{\Delta}_n Δ ˉ 1 : n = Δ ˉ n and Δ ~ 1 : n = Δ ~ n \widetilde{\Delta}_{1:n}=\widetilde{\Delta}_n Δ 1 : n = Δ n for the global averages, while Δ ˉ t j : j = Δ ˉ j B \bar{\Delta}_{t_j:j}=\bar{\Delta}^{B}_j Δ ˉ t j : j = Δ ˉ j B and Δ ~ t j : j = Δ ~ j B \widetilde{\Delta}_{t_j:j}=\widetilde{\Delta}^{B}_j Δ t j : j = Δ j B for the batch averages in (18) .
Lemma C.5. Let γ n = γ n − α \gamma_n=\gamma n^{-\alpha} γ n = γ n − α , where γ > 0 \gamma>0 γ > 0 and α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) . Then, under Assumptions 2.1 2.1 2.1 and 2.4 2.4 2.4 (a), for every n ≥ 1 n\ge1 n ≥ 1 and 1 ≤ s ≤ j 1\le s\le j 1 ≤ s ≤ j ,
E ∥ Δ ˉ n ∥ 2 ≤ C Δ , 2 γ n 1 − α , E ∥ Δ ~ n ∥ 2 ≤ C Δ , 2 γ n 1 − α , \mathbb{E}\lVert\bar{\Delta}_n\rVert^2 \le\frac{C_{\Delta,2}\gamma_n}{1-\alpha},\qquad\mathbb{E}\lVert\widetilde{\Delta}_n\rVert^2 \le\frac{C_{\Delta,2}\gamma_n}{1-\alpha}, E ∥ Δ ˉ n ∥ 2 ≤ 1 − α C Δ , 2 γ n , E ∥ Δ n ∥ 2 ≤ 1 − α C Δ , 2 γ n , E ∥ Δ ˉ s : j ∥ 2 ≤ C Δ , 2 γ s , E ∥ Δ ~ s : j ∥ 2 ≤ C Δ , 2 γ s . \mathbb{E}\lVert\bar{\Delta}_{s:j}\rVert^2 \le C_{\Delta,2}\gamma_s,\qquad\mathbb{E}\lVert\widetilde{\Delta}_{s:j}\rVert^2 \le C_{\Delta,2}\gamma_s. E ∥ Δ ˉ s : j ∥ 2 ≤ C Δ , 2 γ s , E ∥ Δ s : j ∥ 2 ≤ C Δ , 2 γ s . If Assumption 2.4 2.4 2.4 (b) also holds, then
E ∥ Δ ˉ n ∥ 4 ≤ C Δ , 4 γ n 2 ( 1 − α ) 2 , E ∥ Δ ~ n ∥ 4 ≤ C Δ , 4 γ n 2 ( 1 − α ) 2 , \mathbb{E}\lVert\bar{\Delta}_n\rVert^4 \le\frac{C_{\Delta,4}\gamma_n^2}{(1-\alpha)^2},\qquad\mathbb{E}\lVert\widetilde{\Delta}_n\rVert^4 \le\frac{C_{\Delta,4}\gamma_n^2}{(1-\alpha)^2}, E ∥ Δ ˉ n ∥ 4 ≤ ( 1 − α ) 2 C Δ , 4 γ n 2 , E ∥ Δ n ∥ 4 ≤ ( 1 − α ) 2 C Δ , 4 γ n 2 , E ∥ Δ ˉ s : j ∥ 4 ≤ C Δ , 4 γ s 2 , E ∥ Δ ~ s : j ∥ 4 ≤ C Δ , 4 γ s 2 . \mathbb{E}\lVert\bar{\Delta}_{s:j}\rVert^4 \le C_{\Delta,4}\gamma_s^2,\qquad\mathbb{E}\lVert\widetilde{\Delta}_{s:j}\rVert^4 \le C_{\Delta,4}\gamma_s^2. E ∥ Δ ˉ s : j ∥ 4 ≤ C Δ , 4 γ s 2 , E ∥ Δ s : j ∥ 4 ≤ C Δ , 4 γ s 2 . The constants C Δ , 2 C_{\Delta,2} C Δ , 2 and C Δ , 4 C_{\Delta,4} C Δ , 4 are those in (C.8).
Proof. By Assumption 2.1 2.1 2.1 (b), c s : j ∈ X c_{s:j}\in\mathcal{X} c s : j ∈ X . Convexity in Assumption 2.1 2.1 2.1 (c) gives, for j > s j>s j > s ,
∥ Δ ~ s : j ∥ 2 ≤ j − s j − s + 1 ∥ Δ ~ s : j − 1 ∥ 2 + 1 j − s + 1 ∥ Δ j ∥ 2 . \lVert\widetilde{\Delta}_{s:j}\rVert^2 \le\frac{j-s}{j-s+1}\lVert\widetilde{\Delta}_{s:j-1}\rVert^2+\frac{1}{j-s+1}\lVert\Delta_j\rVert^2. ∥ Δ s : j ∥ 2 ≤ j − s + 1 j − s ∥ Δ s : j − 1 ∥ 2 + j − s + 1 1 ∥ Δ j ∥ 2 .
Starting from Δ ~ s : s = Δ s \widetilde{\Delta}_{s:s}=\Delta_s Δ s : s = Δ s , iteration yields
∥ Δ ~ s : j ∥ 2 ≤ 1 j − s + 1 ∑ k = s j ∥ Δ k ∥ 2 . (65) \lVert\widetilde{\Delta}_{s:j}\rVert^2\le\frac{1}{j-s+1}\sum_{k=s}^{j}\lVert\Delta_k\rVert^2. \tag*{(65)} ∥ Δ s : j ∥ 2 ≤ j − s + 1 1 k = s ∑ j ∥ Δ k ∥ 2 . (65)
Jensen’s inequality gives the same bound for ∥ Δ ˉ s : j ∥ 2 \lVert\bar{\Delta}_{s:j}\rVert^2 ∥ Δ ˉ s : j ∥ 2 . Taking expectations and using (C.8), we obtain
E ∥ Δ ˉ s : j ∥ 2 ≤ C Δ , 2 j − s + 1 ∑ k = s j γ k , E ∥ Δ ~ s : j ∥ 2 ≤ C Δ , 2 j − s + 1 ∑ k = s j γ k . \mathbb{E}\lVert\bar{\Delta}_{s:j}\rVert^2\le\frac{C_{\Delta,2}}{j-s+1}\sum_{k=s}^{j}\gamma_k,\qquad\mathbb{E}\lVert\widetilde{\Delta}_{s:j}\rVert^2\le\frac{C_{\Delta,2}}{j-s+1}\sum_{k=s}^{j}\gamma_k. E ∥ Δ ˉ s : j ∥ 2 ≤ j − s + 1 C Δ , 2 k = s ∑ j γ k , E ∥ Δ s : j ∥ 2 ≤ j − s + 1 C Δ , 2 k = s ∑ j γ k .
The interval bounds follow from γ k ≤ γ s \gamma_k \le\gamma_s γ k ≤ γ s for k ≥ s k \ge s k ≥ s . Taking s = 1 s = 1 s = 1 and j = n j = n j = n gives the global bounds because
1 n ∑ k = 1 n γ k ≤ γ n ∫ 0 n t − α d t = γ n 1 − α . (66) \frac{1}{n}\sum_{k=1}^{n}\gamma_k \le\frac{\gamma}{n}\int_0^n t^{-\alpha}\,\mathrm{d}t = \frac{\gamma_n}{1-\alpha}. \tag*{(66)} n 1 k = 1 ∑ n γ k ≤ n γ ∫ 0 n t − α d t = 1 − α γ n . (66)
If Assumption 2.4 (b) also holds, Minkowski’s inequality yields
( E ∥ Δ ~ s : j ∥ 4 ) 1 / 2 ≤ (C.11) 1 j − s + 1 ∑ k = s j ( E ∥ Δ k ∥ 4 ) 1 / 2 ≤ (C.8) C γ 1 , 4 1 / 2 j − s + 1 ∑ k = s j γ k . \left(\mathbb{E}\left\|\widetilde{\Delta}_{s:j}\right\|^4\right)^{1/2} \mathrel{\overset{\text{(C.11)}}{\le}} \frac{1}{j-s+1}\sum_{k=s}^{j}\left(\mathbb{E}\left\|\Delta_k\right\|^4\right)^{1/2} \mathrel{\overset{\text{(C.8)}}{\le}} \frac{C\gamma_{1,4}^{1/2}}{j-s+1}\sum_{k=s}^{j}\gamma_k. ( E Δ s : j 4 ) 1/2 ≤ (C.11) j − s + 1 1 k = s ∑ j ( E ∥ Δ k ∥ 4 ) 1/2 ≤ (C.8) j − s + 1 C γ 1 , 4 1/2 k = s ∑ j γ k .
The same argument applies to Δ ‾ s : j \overline{\Delta}_{s:j} Δ s : j . Squaring and using γ k ≤ γ s \gamma_k \le\gamma_s γ k ≤ γ s for the interval averages and (C.12) for the global averages proves the fourth-moment bounds. This completes the proof. ■
Lemma C.6. Under the assumptions of Lemma C.5 , there exists a constant C < ∞ C < \infty C < ∞ such that, for every n ≥ 1 n \ge1 n ≥ 1 and 1 ≤ s ≤ j 1 \le s \le j 1 ≤ s ≤ j ,
E ∥ Δ ~ n − Δ ‾ n ∥ ≤ C γ n , E ∥ Δ ~ s : j − Δ ‾ s : j ∥ ≤ C γ s . \mathbb{E}\left\|\widetilde{\Delta}_n-\overline{\Delta}_n\right\| \le C\gamma_n,\qquad\mathbb{E}\left\|\widetilde{\Delta}_{s:j}-\overline{\Delta}_{s:j}\right\| \le C\gamma_s. E Δ n − Δ n ≤ C γ n , E Δ s : j − Δ s : j ≤ C γ s . If Assumption 2.4 (b) also holds, then, after enlarging C C C if necessary,
E ∥ Δ ~ n − Δ ‾ n ∥ 2 ≤ C γ n 2 , E ∥ Δ ~ s : j − Δ ‾ s : j ∥ 2 ≤ C γ s 2 . \mathbb{E}\left\|\widetilde{\Delta}_n-\overline{\Delta}_n\right\|^2 \le C\gamma_n^2,\qquad\mathbb{E}\left\|\widetilde{\Delta}_{s:j}-\overline{\Delta}_{s:j}\right\|^2 \le C\gamma_s^2. E Δ n − Δ n 2 ≤ C γ n 2 , E Δ s : j − Δ s : j 2 ≤ C γ s 2 .
Proof. Following Tripuraneni et al. [38 ] (Appendix C.3), we first bound the error in one averaging step. Fix x , z ∈ X x,z \in\mathcal{X} x , z ∈ X and set v = R x − 1 ( z ) v=R_x^{-1}(z) v = R x − 1 ( z ) . Recall Φ x , x ⋆ = R x ⋆ − 1 ∘ R x \Phi_{x,x_\star}=R_{x_\star}^{-1}\circ R_x Φ x , x ⋆ = R x ⋆ − 1 ∘ R x from Lemma B.1 . Retraction convexity gives t v ∈ R x − 1 ( X ) tv\in R_x^{-1}(\mathcal{X}) t v ∈ R x − 1 ( X ) for t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] , so that lemma bounds D 2 Φ x , x ⋆ ( t v ) D^2\Phi_{x,x_\star}(tv) D 2 Φ x , x ⋆ ( t v ) uniformly. Taylor’s formula therefore gives
∥ Φ x , x ⋆ ( t v ) − Φ x , x ⋆ ( 0 x ) − t D Φ x , x ⋆ ( 0 x ) [ v ] ∥ ≤ C 2 t 2 ∥ v ∥ x 2 , 0 ≤ t ≤ 1. \left\|\Phi_{x,x_\star}(tv)-\Phi_{x,x_\star}(0_x)-tD\Phi_{x,x_\star}(0_x)[v]\right\| \le\frac{C}{2}t^2\|v\|_x^2,\qquad0\le t\le1. ∥ Φ x , x ⋆ ( t v ) − Φ x , x ⋆ ( 0 x ) − t D Φ x , x ⋆ ( 0 x ) [ v ] ∥ ≤ 2 C t 2 ∥ v ∥ x 2 , 0 ≤ t ≤ 1.
For θ ∈ [ 0 , 1 ] \theta\in[0,1] θ ∈ [ 0 , 1 ] , subtracting θ \theta θ times the remainder at t = 1 t=1 t = 1 from the remainder at t = θ t=\theta t = θ cancels the linear terms and yields
∥ Φ x , x ⋆ ( θ v ) − ( 1 − θ ) Φ x , x ⋆ ( 0 x ) − θ Φ x , x ⋆ ( v ) ∥ ≤ C 2 ( θ 2 + θ ) ∥ v ∥ x 2 ≤ C θ ∥ v ∥ x 2 . \left\|\Phi_{x,x_\star}(\theta v)-(1-\theta)\Phi_{x,x_\star}(0_x)-\theta\Phi_{x,x_\star}(v)\right\| \le\frac{C}{2}(\theta^2+\theta)\|v\|_x^2 \le C\theta\|v\|_x^2. ∥ Φ x , x ⋆ ( θ v ) − ( 1 − θ ) Φ x , x ⋆ ( 0 x ) − θ Φ x , x ⋆ ( v ) ∥ ≤ 2 C ( θ 2 + θ ) ∥ v ∥ x 2 ≤ C θ ∥ v ∥ x 2 .
Assumption 2.1 (a), Lemma B.1 , and compactness give D X < ρ r e t D_{\mathcal{X}}<\rho_{\mathrm{ret}} D X < ρ ret , so D R x ⋆ DR_{x_\star} D R x ⋆ is bounded on ∥ u ∥ ≤ D X \|u\|\le D_{\mathcal{X}} ∥ u ∥ ≤ D X . The curve t ↦ R x ⋆ ( ( 1 − t ) R x ⋆ − 1 ( x ) + t R x ⋆ − 1 ( z ) ) t\mapsto R_{x_\star}((1-t)R_{x_\star}^{-1}(x)+tR_{x_\star}^{-1}(z)) t ↦ R x ⋆ (( 1 − t ) R x ⋆ − 1 ( x ) + t R x ⋆ − 1 ( z )) , t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] , then has length at most C ∥ R x ⋆ − 1 ( z ) − R x ⋆ − 1 ( x ) ∥ C\|R_{x_\star}^{-1}(z)-R_{x_\star}^{-1}(x)\| C ∥ R x ⋆ − 1 ( z ) − R x ⋆ − 1 ( x ) ∥ . Combining this with (B.1) gives
∥ R x − 1 ( z ) ∥ x ≤ C ∥ R x ⋆ − 1 ( z ) − R x ⋆ − 1 ( x ) ∥ . (67) \left\|R_x^{-1}(z)\right\|_x \le C\left\|R_{x_\star}^{-1}(z)-R_{x_\star}^{-1}(x)\right\|. \tag*{(67)} R x − 1 ( z ) x ≤ C R x ⋆ − 1 ( z ) − R x ⋆ − 1 ( x ) . (67)
Since Φ x , x ⋆ ( 0 x ) = R x ⋆ − 1 ( x ) \Phi_{x,x_\star}(0_x)=R_{x_\star}^{-1}(x) Φ x , x ⋆ ( 0 x ) = R x ⋆ − 1 ( x ) and Φ x , x ⋆ ( v ) = R x ⋆ − 1 ( z ) \Phi_{x,x_\star}(v)=R_{x_\star}^{-1}(z) Φ x , x ⋆ ( v ) = R x ⋆ − 1 ( z ) , the Taylor estimate and (C.13) imply
∥ R x ⋆ − 1 ( R x ( θ R x − 1 ( z ) ) ) − ( 1 − θ ) R x ⋆ − 1 ( x ) − θ R x ⋆ − 1 ( z ) ∥ ≤ C θ ∥ R x ⋆ − 1 ( z ) − R x ⋆ − 1 ( x ) ∥ 2 . (68) \left\|R_{x_\star}^{-1}\left(R_x(\theta R_x^{-1}(z))\right)-(1-\theta)R_{x_\star}^{-1}(x)-\theta R_{x_\star}^{-1}(z)\right\| \le C\theta\left\|R_{x_\star}^{-1}(z)-R_{x_\star}^{-1}(x)\right\|^2. \tag*{(68)} R x ⋆ − 1 ( R x ( θ R x − 1 ( z )) ) − ( 1 − θ ) R x ⋆ − 1 ( x ) − θ R x ⋆ − 1 ( z ) ≤ C θ R x ⋆ − 1 ( z ) − R x ⋆ − 1 ( x ) 2 . (68)
We now accumulate these errors over an interval. For k ≥ s k\ge s k ≥ s , take x = c s : k x=c_{s:k} x = c s : k , z = x k + 1 z=x_{k+1} z = x k + 1 , and θ = ( k − s + 2 ) − 1 \theta=(k-s+2)^{-1} θ = ( k − s + 2 ) − 1 . Subtracting the arithmetic update gives
( k − s + 2 ) ∥ Δ ~ s : k + 1 − Δ ‾ s : k + 1 ∥ ≤ (C.14) ( k − s + 1 ) ∥ Δ ~ s : k − Δ ‾ s : k ∥ + C ∥ Δ k + 1 − Δ ~ s : k ∥ 2 . (k-s+2)\left\|\widetilde{\Delta}_{s:k+1}-\overline{\Delta}_{s:k+1}\right\| \mathrel{\overset{\text{(C.14)}}{\le}} (k-s+1)\left\|\widetilde{\Delta}_{s:k}-\overline{\Delta}_{s:k}\right\|+C\left\|\Delta_{k+1}-\widetilde{\Delta}_{s:k}\right\|^2. ( k − s + 2 ) Δ s : k + 1 − Δ s : k + 1 ≤ (C.14) ( k − s + 1 ) Δ s : k − Δ s : k + C Δ k + 1 − Δ s : k 2 .
Since Δ ~ s : s = Δ ‾ s : s = Δ s \widetilde{\Delta}_{s:s}=\overline{\Delta}_{s:s}=\Delta_s Δ s : s = Δ s : s = Δ s , iteration yields
∥ Δ ~ s : j − Δ ‾ s : j ∥ ≤ C j − s + 1 ∑ k = s j − 1 ∥ Δ k + 1 − Δ ~ s : k ∥ 2 . (69) \lVert\widetilde{\Delta}_{s:j}-\overline{\Delta}_{s:j}\rVert\le\frac{C}{j-s+1}\sum_{k=s}^{j-1}\lVert\Delta_{k+1}-\widetilde{\Delta}_{s:k}\rVert^2. \tag*{(69)} ∥ Δ s : j − Δ s : j ∥ ≤ j − s + 1 C k = s ∑ j − 1 ∥ Δ k + 1 − Δ s : k ∥ 2 . (69)
For s = j s=j s = j , the sum is empty and both sides are zero. The second-moment bounds in (C.8) and Lemma C.5 give, for 1 ≤ s ≤ k 1\le s\le k 1 ≤ s ≤ k ,
E ∥ Δ k + 1 − Δ ~ k ∥ 2 ≤ C γ k , E ∥ Δ k + 1 − Δ ~ s : k ∥ 2 ≤ C γ s . \mathbb{E}\lVert\Delta_{k+1}-\widetilde{\Delta}_k\rVert^2\le C\gamma_k,\qquad\mathbb{E}\lVert\Delta_{k+1}-\widetilde{\Delta}_{s:k}\rVert^2\le C\gamma_s. E ∥ Δ k + 1 − Δ k ∥ 2 ≤ C γ k , E ∥ Δ k + 1 − Δ s : k ∥ 2 ≤ C γ s .
Taking expectations in (C.15), first with s = 1 s=1 s = 1 , j = n j=n j = n and then with arbitrary s , j s,j s , j , yields
E ∥ Δ ~ n − Δ ‾ n ∥ ≤ C n ∑ k = 1 n − 1 γ k ≤ C 1 − α γ n , (70) \mathbb{E}\lVert\widetilde{\Delta}_n-\overline{\Delta}_n\rVert\le\frac{C}{n}\sum_{k=1}^{n-1}\gamma_k\le\frac{C}{1-\alpha}\gamma_n, \tag*{(70)} E ∥ Δ n − Δ n ∥ ≤ n C k = 1 ∑ n − 1 γ k ≤ 1 − α C γ n , (70)
E ∥ Δ ~ s : j − Δ ‾ s : j ∥ ≤ C j − s j − s + 1 γ s ≤ C γ s . \mathbb{E}\lVert\widetilde{\Delta}_{s:j}-\overline{\Delta}_{s:j}\rVert\le C\frac{j-s}{j-s+1}\gamma_s\le C\gamma_s. E ∥ Δ s : j − Δ s : j ∥ ≤ C j − s + 1 j − s γ s ≤ C γ s .
This proves the first-moment bounds. If Assumption 2.4(b) also holds, the fourth-moment bounds in (C.8) and Lemma C.5 give
( E ∥ Δ k + 1 − Δ ~ k ∥ 4 ) 1 / 2 ≤ C γ k , ( E ∥ Δ k + 1 − Δ ~ s : k ∥ 4 ) 1 / 2 ≤ C γ s . \left(\mathbb{E}\lVert\Delta_{k+1}-\widetilde{\Delta}_k\rVert^4\right)^{1/2}\le C\gamma_k,\qquad\left(\mathbb{E}\lVert\Delta_{k+1}-\widetilde{\Delta}_{s:k}\rVert^4\right)^{1/2}\le C\gamma_s. ( E ∥ Δ k + 1 − Δ k ∥ 4 ) 1/2 ≤ C γ k , ( E ∥ Δ k + 1 − Δ s : k ∥ 4 ) 1/2 ≤ C γ s .
Minkowski’s inequality yields
( E ∥ Δ ~ s : j − Δ ‾ s : j ∥ 2 ) 1 / 2 ≤ C j − s + 1 ∑ k = s j − 1 ( E ∥ Δ k + 1 − Δ ~ s : k ∥ 4 ) 1 / 2 . \left(\mathbb{E}\lVert\widetilde{\Delta}_{s:j}-\overline{\Delta}_{s:j}\rVert^2\right)^{1/2}\le\frac{C}{j-s+1}\sum_{k=s}^{j-1}\left(\mathbb{E}\lVert\Delta_{k+1}-\widetilde{\Delta}_{s:k}\rVert^4\right)^{1/2}. ( E ∥ Δ s : j − Δ s : j ∥ 2 ) 1/2 ≤ j − s + 1 C k = s ∑ j − 1 ( E ∥ Δ k + 1 − Δ s : k ∥ 4 ) 1/2 .
Substituting the preceding fourth-moment bounds and using (C.12) for the global average gives
( E ∥ Δ ~ n − Δ ‾ n ∥ 2 ) 1 / 2 ≤ C γ n , ( E ∥ Δ ~ s : j − Δ ‾ s : j ∥ 2 ) 1 / 2 ≤ C γ s . \left(\mathbb{E}\lVert\widetilde{\Delta}_n-\overline{\Delta}_n\rVert^2\right)^{1/2}\le C\gamma_n,\qquad\left(\mathbb{E}\lVert\widetilde{\Delta}_{s:j}-\overline{\Delta}_{s:j}\rVert^2\right)^{1/2}\le C\gamma_s. ( E ∥ Δ n − Δ n ∥ 2 ) 1/2 ≤ C γ n , ( E ∥ Δ s : j − Δ s : j ∥ 2 ) 1/2 ≤ C γ s .
Squaring and enlarging C C C proves the second-moment bounds and completes the proof. ■
Proof of Theorem 3.1
We prove the result for both OL and NOL using the four-term decomposition in § 3.4 . Fix ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } . Recall the selected indices E n ⋄ E_n^\diamond E n ⋄ from (15) and (16) , and write
v n , ⋄ : = ∑ j ∈ E n ⋄ ℓ j , v n , O L = ∑ j = 1 n ℓ j , v n , N O L = n . v_{n,\diamond}:=\sum_{j\in E_n^\diamond}\ell_j,\qquad v_{n,\mathrm{OL}}=\sum_{j=1}^{n}\ell_j,\qquad v_{n,\mathrm{NOL}}=n. v n , ⋄ := j ∈ E n ⋄ ∑ ℓ j , v n , OL = j = 1 ∑ n ℓ j , v n , NOL = n .
Here B j = { x t j , … , x j } B_j=\{x_{t_j},\ldots,x_j\} B j = { x t j , … , x j } is a batch of iterates, ℓ j = ∣ B j ∣ \ell_j=\lvert B_j\rvert ℓ j = ∣ B j ∣ , and I j = { t j , … , j } I_j=\{t_j,\ldots,j\} I j = { t j , … , j } is its index set, as in Lemma A.6 . Recall its mean c j c_j c j from (11) , the exact vector b j , n = R x ~ n − 1 ( c j ) b_{j,n}=R^{-1}_{\widetilde{x}_n}(c_j) b j , n = R x n − 1 ( c j ) , and its approximation b ~ j , n \tilde b_{j,n} b ~ j , n from (13) . Using the averages in (18) , extend the four operators in (19) to either batch selection:
Σ ^ n , ⋄ ( 0 ) : = ( Γ x ⋆ x ~ ⋆ ) − 1 ∘ Σ ~ n , ⋄ ∘ Γ x ⋆ x ~ ⋆ = 1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 ( ( Γ x ⋆ x ~ ⋆ ) − 1 b ~ j , n ) ⊗ 2 , \widehat{\Sigma}^{(0)}_{n,\diamond}:=(\Gamma^{\tilde{x}_{\star}}_{x_{\star}})^{-1}\circ\widetilde{\Sigma}_{n,\diamond}\circ\Gamma^{\tilde{x}_{\star}}_{x_{\star}}=\frac{1}{v_{n,\diamond}}\sum_{j\in E^\diamond_n}\ell_j^2\left((\Gamma^{\tilde{x}_{\star}}_{x_{\star}})^{-1}\tilde{b}_{j,n}\right)^{\otimes2}, Σ n , ⋄ ( 0 ) := ( Γ x ⋆ x ~ ⋆ ) − 1 ∘ Σ n , ⋄ ∘ Γ x ⋆ x ~ ⋆ = v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 ( ( Γ x ⋆ x ~ ⋆ ) − 1 b ~ j , n ) ⊗ 2 ,
Σ ^ n , ⋄ ( 1 ) : = 1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 ( ( Γ x ⋆ x ~ ⋆ ) − 1 b j , n ) ⊗ 2 , \widehat{\Sigma}^{(1)}_{n,\diamond}:=\frac{1}{v_{n,\diamond}}\sum_{j\in E^\diamond_n}\ell_j^2\left((\Gamma^{\tilde{x}_{\star}}_{x_{\star}})^{-1}b_{j,n}\right)^{\otimes2}, Σ n , ⋄ ( 1 ) := v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 ( ( Γ x ⋆ x ~ ⋆ ) − 1 b j , n ) ⊗ 2 ,
Σ ^ n , ⋄ ( 2 ) : = 1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 ( Δ ~ j B − Δ ~ n ) ⊗ 2 , \widehat{\Sigma}^{(2)}_{n,\diamond}:=\frac{1}{v_{n,\diamond}}\sum_{j\in E^\diamond_n}\ell_j^2\left(\widetilde{\Delta}^{B}_j-\widetilde{\Delta}_n\right)^{\otimes2}, Σ n , ⋄ ( 2 ) := v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 ( Δ j B − Δ n ) ⊗ 2 ,
Σ ^ n , ⋄ ( 3 ) : = 1 v n , ⋄ ∑ j ∈ E n ⋄ ( ∑ k = t j j Δ k − ℓ j Δ ~ n ) ⊗ 2 = 1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 ( Δ ~ j B − Δ ~ n ) ⊗ 2 . \widehat{\Sigma}^{(3)}_{n,\diamond}:=\frac{1}{v_{n,\diamond}}\sum_{j\in E^\diamond_n}\left(\sum_{k=t_j}^{j}\Delta_k-\ell_j\widetilde{\Delta}_n\right)^{\otimes2}=\frac{1}{v_{n,\diamond}}\sum_{j\in E^\diamond_n}\ell_j^2\left(\widetilde{\Delta}^{B}_j-\widetilde{\Delta}_n\right)^{\otimes2}. Σ n , ⋄ ( 3 ) := v n , ⋄ 1 j ∈ E n ⋄ ∑ k = t j ∑ j Δ k − ℓ j Δ n ⊗ 2 = v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 ( Δ j B − Δ n ) ⊗ 2 .
All four operators act on T x ⋆ M T_{x_{\star}}\mathcal{M} T x ⋆ M . Isometry of parallel transport identifies its inverse with its adjoint in the representation of Σ ^ n , ⋄ ( 0 ) \widehat{\Sigma}^{(0)}_{n,\diamond} Σ n , ⋄ ( 0 ) . Hence
E ∥ Σ ^ n , ⋄ ( 0 ) − Σ ∥ ≤ E ∥ Σ ^ n , ⋄ ( 0 ) − Σ ^ n , ⋄ ( 1 ) ∥ + E ∥ Σ ^ n , ⋄ ( 1 ) − Σ ^ n , ⋄ ( 2 ) ∥ + E ∥ Σ ^ n , ⋄ ( 2 ) − Σ ^ n , ⋄ ( 3 ) ∥ + E ∥ Σ ^ n , ⋄ ( 3 ) − Σ ∥ . (71) \mathbb{E}\left\|\widehat{\Sigma}^{(0)}_{n,\diamond}-\Sigma\right\|\leq\mathbb{E}\left\|\widehat{\Sigma}^{(0)}_{n,\diamond}-\widehat{\Sigma}^{(1)}_{n,\diamond}\right\|+\mathbb{E}\left\|\widehat{\Sigma}^{(1)}_{n,\diamond}-\widehat{\Sigma}^{(2)}_{n,\diamond}\right\|+\mathbb{E}\left\|\widehat{\Sigma}^{(2)}_{n,\diamond}-\widehat{\Sigma}^{(3)}_{n,\diamond}\right\|+\mathbb{E}\left\|\widehat{\Sigma}^{(3)}_{n,\diamond}-\Sigma\right\|. \tag*{(71)} E Σ n , ⋄ ( 0 ) − Σ ≤ E Σ n , ⋄ ( 0 ) − Σ n , ⋄ ( 1 ) + E Σ n , ⋄ ( 1 ) − Σ n , ⋄ ( 2 ) + E Σ n , ⋄ ( 2 ) − Σ n , ⋄ ( 3 ) + E Σ n , ⋄ ( 3 ) − Σ . (71)
The first three terms correspond to online vector updates, changes of base point, and averaging on the manifold. The last is the batch-means error in the fixed tangent space. We first bound that last term in Section D.1, obtaining E ∥ Σ ^ n , ⋄ ( 3 ) − Σ ∥ ≤ C ( n − 1 / ( 2 β ) + n ( α − 1 ) / 2 + 1 / ( 2 β ) ) \mathbb{E}\left\|\widehat{\Sigma}^{(3)}_{n,\diamond}-\Sigma\right\|\leq C\left(n^{-1/(2\beta)}+n^{(\alpha-1)/2+1/(2\beta)}\right) E Σ n , ⋄ ( 3 ) − Σ ≤ C ( n − 1/ ( 2 β ) + n ( α − 1 ) /2 + 1/ ( 2 β ) ) . Section D.2 then bounds each of the first three terms by C n 1 / 2 − 1 / ( 2 β ) − α Cn^{1/2-1/(2\beta)-\alpha} C n 1/2 − 1/ ( 2 β ) − α . We combine these bounds at the end of the appendix.
Batch-means error in the fixed tangent space
We first bound the noise-based estimator, then compare it with the linear and RSGD estimators. These bounds also control the expected operator norms needed below. Set ζ k : = − grad f ( x ⋆ , ξ k ) \zeta_k:=-\operatorname{grad}f(x_{\star},\xi_k) ζ k := − grad f ( x ⋆ , ξ k ) , k ≥ 1 k\geq1 k ≥ 1 ; these vectors are independent and identically distributed by § 2.2 . Recall that
A = Hess F ( x ⋆ ) , S = E ξ ∼ D [ grad f ( x ⋆ , ξ ) ⊗ 2 ] , Σ = A − 1 S A − 1 = E [ ( A − 1 ζ 1 ) ⊗ 2 ] . A=\operatorname{Hess}F(x_{\star}),\qquad S=\mathbb{E}_{\xi\sim\mathcal{D}}\left[\operatorname{grad}f(x_{\star},\xi)^{\otimes2}\right],\qquad\Sigma=A^{-1}SA^{-1}=\mathbb{E}\left[(A^{-1}\zeta_1)^{\otimes2}\right]. A = Hess F ( x ⋆ ) , S = E ξ ∼ D [ grad f ( x ⋆ , ξ ) ⊗ 2 ] , Σ = A − 1 S A − 1 = E [ ( A − 1 ζ 1 ) ⊗ 2 ] .
Let ( U k ) (U_k) ( U k ) be the coupled linear recursion in (C.9), and set U ‾ n : = n − 1 ∑ k = 1 n U k \overline{U}_n:=n^{-1}\sum_{k=1}^{n}U_k U n := n − 1 ∑ k = 1 n U k . For every n ≥ 1 n\geq1 n ≥ 1 , define
Σ ^ n , ⋄ ( 3 , 1 ) : = 1 v n , ⋄ ∑ j ∈ E n ⋄ ( ∑ k = t j j U k − ℓ j U ‾ n ) ⊗ 2 , \widehat{\Sigma}^{(3,1)}_{n,\diamond}:=\frac{1}{v_{n,\diamond}}\sum_{j\in E^\diamond_n}\left(\sum_{k=t_j}^{j}U_k-\ell_j\overline{U}_n\right)^{\otimes2}, Σ n , ⋄ ( 3 , 1 ) := v n , ⋄ 1 j ∈ E n ⋄ ∑ k = t j ∑ j U k − ℓ j U n ⊗ 2 ,
Σ ^ n , ⋄ ( 3 , 2 ) : = 1 v n , ⋄ ∑ j ∈ E n ⋄ ( ∑ k = t j j A − 1 ζ k − ℓ j n ∑ k = 1 n A − 1 ζ k ) ⊗ 2 . (72) \widehat{\Sigma}^{(3,2)}_{n,\diamond}:=\frac{1}{v_{n,\diamond}}\sum_{j\in E^\diamond_n}\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k-\frac{\ell_j}{n}\sum_{k=1}^{n}A^{-1}\zeta_k\right)^{\otimes2}. \tag*{(72)} Σ n , ⋄ ( 3 , 2 ) := v n , ⋄ 1 j ∈ E n ⋄ ∑ k = t j ∑ j A − 1 ζ k − n ℓ j k = 1 ∑ n A − 1 ζ k ⊗ 2 . (72)
Thus, Σ ^ n , ⋄ ( 3 , 1 ) \widehat{\Sigma}^{(3,1)}_{n,\diamond} Σ n , ⋄ ( 3 , 1 ) replaces Δ k \Delta_k Δ k in Σ ^ n , ⋄ ( 3 ) \widehat{\Sigma}^{(3)}_{n,\diamond} Σ n , ⋄ ( 3 ) by U k U_k U k , whereas Σ ^ n , ⋄ ( 3 , 2 ) \widehat{\Sigma}^{(3,2)}_{n,\diamond} Σ n , ⋄ ( 3 , 2 ) replaces U k U_k U k by A − 1 ζ k A^{-1}\zeta_k A − 1 ζ k .
Lemma D.1. Let α ∈ ( 1 / 2 , 1 ) \alpha\in(1/2,1) α ∈ ( 1/2 , 1 ) , and let a 1 = 1 a_1=1 a 1 = 1 and a m = ⌊ c m β ⌋ a_m=\lfloor cm^\beta\rfloor a m = ⌊ c m β ⌋ for m ≥ 2 m\geq2 m ≥ 2 , where c ≥ 1 c\geq1 c ≥ 1 and β > 1 / ( 1 − α ) \beta>1/(1-\alpha) β > 1/ ( 1 − α ) . Then, under Assumptions 2.2 and 2.3, there exists C < ∞ C<\infty C < ∞ such that the noise-based covariance estimator defined in (D.2) satisfies, for every n ≥ 1 n\geq1 n ≥ 1 and ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } ,
E ∥ Σ ^ n , ⋄ ( 3 , 2 ) − Σ ∥ ≤ C n − 1 / ( 2 β ) . \mathbb{E}\left\|\widehat{\Sigma}^{(3,2)}_{n,\diamond}-\Sigma\right\|\leq Cn^{-1/(2\beta)}. E Σ n , ⋄ ( 3 , 2 ) − Σ ≤ C n − 1/ ( 2 β ) .
Proof. By Assumptions 2.2 and 2.3, E ( A − 1 ζ k ) = 0 \mathbb{E}(A^{-1}\zeta_k)=0 E ( A − 1 ζ k ) = 0 , E ( A − 1 ζ k ) ⊗ 2 = Σ \mathbb{E}(A^{-1}\zeta_k)^{\otimes2}=\Sigma E ( A − 1 ζ k ) ⊗ 2 = Σ , and E ∥ A − 1 ζ k ∥ 4 ≤ μ − 4 τ 4 \mathbb{E}\lVert A^{-1}\zeta_k\rVert^4\leq\mu^{-4}\tau^4 E ∥ A − 1 ζ k ∥ 4 ≤ μ − 4 τ 4 , using ∥ A − 1 ∥ ≤ μ − 1 \lVert A^{-1}\rVert\leq\mu^{-1} ∥ A − 1 ∥ ≤ μ − 1 . First omit the common centering term:
Σ ^ n , ∘ ( 3 , 2 , 0 ) : = v n , ∘ − 1 ∑ j ∈ E n ∘ ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 . \widehat{\Sigma}_{n,\circ}^{(3,2,0)}:=v_{n,\circ}^{-1}\sum_{j\in E_n^\circ}\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}. Σ n , ∘ ( 3 , 2 , 0 ) := v n , ∘ − 1 j ∈ E n ∘ ∑ k = t j ∑ j A − 1 ζ k ⊗ 2 .
Independence and centering imply E ( A − 1 ζ k ) ( A − 1 ζ r ) T = 0 \mathbb{E}(A^{-1}\zeta_k)(A^{-1}\zeta_r)^{\mathsf{T}}=0 E ( A − 1 ζ k ) ( A − 1 ζ r ) T = 0 for k ≠ r k\neq r k = r , so
E ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 = ∑ k , r = t j j E ( ( A − 1 ζ k ) ( A − 1 ζ r ) T ) = ℓ j Σ . \mathbb{E}\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}=\sum_{k,r=t_j}^{j}\mathbb{E}\left((A^{-1}\zeta_k)(A^{-1}\zeta_r)^{\mathsf{T}}\right)=\ell_j\Sigma. E k = t j ∑ j A − 1 ζ k ⊗ 2 = k , r = t j ∑ j E ( ( A − 1 ζ k ) ( A − 1 ζ r ) T ) = ℓ j Σ.
Since ∑ j ∈ E n ∘ ℓ j = v n , ∘ \sum_{j\in E_n^\circ}\ell_j=v_{n,\circ} ∑ j ∈ E n ∘ ℓ j = v n , ∘ , the uncentered estimator has expectation Σ \Sigma Σ . In the fourth-moment expansion, independence and centering remove every term with an index appearing only once. For k ≠ r k\neq r k = r , independence yields
E ( ∥ A − 1 ζ k ∥ 2 ∥ A − 1 ζ r ∥ 2 ) = ( E ∥ A − 1 ζ k ∥ 2 ) ( E ∥ A − 1 ζ r ∥ 2 ) = ( tr Σ ) 2 , \mathbb{E}\left(\lVert A^{-1}\zeta_k\rVert^2\lVert A^{-1}\zeta_r\rVert^2\right)=\left(\mathbb{E}\lVert A^{-1}\zeta_k\rVert^2\right)\left(\mathbb{E}\lVert A^{-1}\zeta_r\rVert^2\right)=(\operatorname{tr}\Sigma)^2, E ( ∥ A − 1 ζ k ∥ 2 ∥ A − 1 ζ r ∥ 2 ) = ( E ∥ A − 1 ζ k ∥ 2 ) ( E ∥ A − 1 ζ r ∥ 2 ) = ( tr Σ ) 2 ,
E ⟨ A − 1 ζ k , A − 1 ζ r ⟩ 2 = tr ( E [ ( A − 1 ζ k ) ( A − 1 ζ k ) T ( A − 1 ζ r ) ( A − 1 ζ r ) T ] ) = tr ( Σ 2 ) . \mathbb{E}\langle A^{-1}\zeta_k,A^{-1}\zeta_r\rangle^2=\operatorname{tr}\left(\mathbb{E}\left[(A^{-1}\zeta_k)(A^{-1}\zeta_k)^{\mathsf{T}}(A^{-1}\zeta_r)(A^{-1}\zeta_r)^{\mathsf{T}}\right]\right)=\operatorname{tr}(\Sigma^2). E ⟨ A − 1 ζ k , A − 1 ζ r ⟩ 2 = tr ( E [ ( A − 1 ζ k ) ( A − 1 ζ k ) T ( A − 1 ζ r ) ( A − 1 ζ r ) T ] ) = tr ( Σ 2 ) .
The remaining terms in the expansion therefore yield
E ∥ ∑ k = t j j A − 1 ζ k ∥ 4 = ∑ k = t j j E ∥ A − 1 ζ k ∥ 4 + 2 ∑ t j ≤ k < r ≤ j E ( ∥ A − 1 ζ k ∥ 2 ∥ A − 1 ζ r ∥ 2 ) + 4 ∑ t j ≤ k < r ≤ j E ⟨ A − 1 ζ k , A − 1 ζ r ⟩ 2 = ℓ j E ∥ A − 1 ζ 1 ∥ 4 + ℓ j ( ℓ j − 1 ) ( ( tr Σ ) 2 + 2 tr ( Σ 2 ) ) ≤ C ℓ j 2 . (73) \begin{aligned}
\mathbb{E}\left\lVert\sum_{k=t_j}^{j}A^{-1}\zeta_k\right\rVert^4
&=\sum_{k=t_j}^{j}\mathbb{E}\lVert A^{-1}\zeta_k\rVert^4
+2\sum_{t_j\leq k<r\leq j}\mathbb{E}\left(\lVert A^{-1}\zeta_k\rVert^2\lVert A^{-1}\zeta_r\rVert^2\right)\\
&\quad+4\sum_{t_j\leq k<r\leq j}\mathbb{E}\langle A^{-1}\zeta_k,A^{-1}\zeta_r\rangle^2\\
&=\ell_j\mathbb{E}\lVert A^{-1}\zeta_1\rVert^4+\ell_j(\ell_j-1)\left((\operatorname{tr}\Sigma)^2+2\operatorname{tr}(\Sigma^2)\right)\leq C\ell_j^2. \tag*{(73)}
\end{aligned} E k = t j ∑ j A − 1 ζ k 4 = k = t j ∑ j E ∥ A − 1 ζ k ∥ 4 + 2 t j ≤ k < r ≤ j ∑ E ( ∥ A − 1 ζ k ∥ 2 ∥ A − 1 ζ r ∥ 2 ) + 4 t j ≤ k < r ≤ j ∑ E ⟨ A − 1 ζ k , A − 1 ζ r ⟩ 2 = ℓ j E ∥ A − 1 ζ 1 ∥ 4 + ℓ j ( ℓ j − 1 ) ( ( tr Σ ) 2 + 2 tr ( Σ 2 ) ) ≤ C ℓ j 2 . (73)
The inequality uses the fourth-moment bound and ℓ j ≥ 1 \ell_j\geq1 ℓ j ≥ 1 . Since ∥ M ∥ F 2 = tr ( M T M ) \lVert M\rVert_{\mathrm{F}}^2=\operatorname{tr}(M^{\mathsf{T}}M) ∥ M ∥ F 2 = tr ( M T M ) and ∥ v v T ∥ F 2 = ∥ v ∥ 4 \lVert vv^{\mathsf{T}}\rVert_{\mathrm{F}}^2=\lVert v\rVert^4 ∥ v v T ∥ F 2 = ∥ v ∥ 4 , the second-moment identity implies
E ∥ ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) ∥ F 2 = E ∥ ∑ k = t j j A − 1 ζ k ∥ 4 − 2 ℓ j tr E [ ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 Σ ] + ℓ j 2 ∥ Σ ∥ F 2 = E ∥ ∑ k = t j j A − 1 ζ k ∥ 4 − ℓ j 2 ∥ Σ ∥ F 2 ≤ C ℓ j 2 . (74) \begin{aligned}
\mathbb{E}\left\lVert\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\right\rVert_{\mathrm{F}}^2
&=\mathbb{E}\left\lVert\sum_{k=t_j}^{j}A^{-1}\zeta_k\right\rVert^4
-2\ell_j\operatorname{tr}\mathbb{E}\left[\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}\Sigma\right]
+\ell_j^2\lVert\Sigma\rVert_{\mathrm{F}}^2\\
&=\mathbb{E}\left\lVert\sum_{k=t_j}^{j}A^{-1}\zeta_k\right\rVert^4-\ell_j^2\lVert\Sigma\rVert_{\mathrm{F}}^2\leq C\ell_j^2. \tag*{(74)}
\end{aligned} E k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ F 2 = E k = t j ∑ j A − 1 ζ k 4 − 2 ℓ j tr E k = t j ∑ j A − 1 ζ k ⊗ 2 Σ + ℓ j 2 ∥ Σ ∥ F 2 = E k = t j ∑ j A − 1 ζ k 4 − ℓ j 2 ∥ Σ ∥ F 2 ≤ C ℓ j 2 . (74)
We now bound the cross terms separately for OL and NOL.
OL. If I j I_j I j and I q I_q I q lie in different blocks, independence and centering imply
E tr [ ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ] = tr [ E ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) E ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ] = 0. \mathbb{E}\operatorname{tr}\left[\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\right]
=\operatorname{tr}\left[\mathbb{E}\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\mathbb{E}\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\right]=0. E tr k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ = tr E k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ E k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ = 0.
Within one block, let j ≤ q j \le q j ≤ q . Since t j = t q t_j = t_q t j = t q , we have
( ∑ k = t q q A − 1 ζ k ) = ( ∑ k = t j j A − 1 ζ k ) + ∑ k = j + 1 q ( A − 1 ζ k ) , ℓ q − ℓ j = q − j . \left(\sum_{k=t_q}^{q} A^{-1}\zeta_k\right)=\left(\sum_{k=t_j}^{j} A^{-1}\zeta_k\right)+\sum_{k=j+1}^{q}(A^{-1}\zeta_k),\qquad\ell_q-\ell_j=q-j. k = t q ∑ q A − 1 ζ k = k = t j ∑ j A − 1 ζ k + k = j + 1 ∑ q ( A − 1 ζ k ) , ℓ q − ℓ j = q − j .
The first term is F j \mathcal{F}_j F j -measurable, while the remaining observations are independent of F j \mathcal{F}_j F j . Thus
E [ ∑ k = j + 1 q ( A − 1 ζ k ) ∣ F j ] = 0 , E [ ( ∑ k = j + 1 q ( A − 1 ζ k ) ) ⊗ 2 ∣ F j ] = ( ℓ q − ℓ j ) Σ . \mathbb{E}\left[\sum_{k=j+1}^{q}(A^{-1}\zeta_k)\mid\mathcal{F}_j\right]=0,\qquad
\mathbb{E}\left[\left(\sum_{k=j+1}^{q}(A^{-1}\zeta_k)\right)^{\otimes2}\mid\mathcal{F}_j\right]=(\ell_q-\ell_j)\Sigma. E k = j + 1 ∑ q ( A − 1 ζ k ) ∣ F j = 0 , E k = j + 1 ∑ q ( A − 1 ζ k ) ⊗ 2 ∣ F j = ( ℓ q − ℓ j ) Σ.
Expanding the outer product and conditioning on F j \mathcal{F}_j F j , we obtain
E [ ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ∣ F j ] = ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 + ( ∑ k = t j j A − 1 ζ k ) E [ ∑ k = j + 1 q A − 1 ζ k ∣ F j ] T + E [ ∑ k = j + 1 q A − 1 ζ k ∣ F j ] ( ∑ k = t j j A − 1 ζ k ) T + E [ ( ∑ k = j + 1 q A − 1 ζ k ) ⊗ 2 ∣ F j ] − ℓ q Σ = ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 + ( ℓ q − ℓ j ) Σ − ℓ q Σ = ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) . \begin{aligned}
\mathbb{E}\left[\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\mid\mathcal{F}_j\right]
={}&\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}
+\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)\mathbb{E}\left[\sum_{k=j+1}^{q}A^{-1}\zeta_k\mid\mathcal{F}_j\right]^{\mathsf T}\\
&+\mathbb{E}\left[\sum_{k=j+1}^{q}A^{-1}\zeta_k\mid\mathcal{F}_j\right]\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\mathsf T}\\
&+\mathbb{E}\left[\left(\sum_{k=j+1}^{q}A^{-1}\zeta_k\right)^{\otimes2}\mid\mathcal{F}_j\right]-\ell_q\Sigma\\
={}&\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}+(\ell_q-\ell_j)\Sigma-\ell_q\Sigma\\
={}&\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right).
\end{aligned} E k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ ∣ F j = = = k = t j ∑ j A − 1 ζ k ⊗ 2 + k = t j ∑ j A − 1 ζ k E k = j + 1 ∑ q A − 1 ζ k ∣ F j T + E k = j + 1 ∑ q A − 1 ζ k ∣ F j k = t j ∑ j A − 1 ζ k T + E k = j + 1 ∑ q A − 1 ζ k ⊗ 2 ∣ F j − ℓ q Σ k = t j ∑ j A − 1 ζ k ⊗ 2 + ( ℓ q − ℓ j ) Σ − ℓ q Σ k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ .
Since the centered outer product for batch j j j is F j \mathcal{F}_j F j -measurable, the tower property yields
E tr [ ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ] = E ( E [ tr ( ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ) ∣ F j ] ) = E tr [ ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) E [ ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ∣ F j ] ] = E tr [ ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) 2 ] = E ∥ ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ∥ F 2 . (75) \begin{aligned}
\mathbb{E}\operatorname{tr}\left[\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\right]
={}&\mathbb{E}\left(\mathbb{E}\left[\operatorname{tr}\left(\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\right)\mid\mathcal{F}_j\right]\right)\\
={}&\mathbb{E}\operatorname{tr}\left[\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\mathbb{E}\left[\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\mid\mathcal{F}_j\right]\right]\\
={}&\mathbb{E}\operatorname{tr}\left[\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)^2\right]
=\mathbb{E}\left\|\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right\|_{\mathrm{F}}^2. \tag*{(75)}
\end{aligned} E tr k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ = = = E E tr k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ ∣ F j E tr k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ E k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ ∣ F j E tr k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ 2 = E k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ F 2 . (75)
Cauchy–Schwarz and (D.4) ensure integrability. The identities also hold for j = q j=q j = q , with empty sums. Recall the block count m n m_n m n and lengths L m = min { a m + 1 − 1 , n } − a m + 1 L_m=\min\{a_{m+1}-1,n\}-a_m+1 L m = min { a m + 1 − 1 , n } − a m + 1 from Lemma A.6. Summing over batch lengths r , s r,s r , s within each block yields
E ∥ Σ ^ n , O L ( 3 , 2 , 0 ) − Σ ∥ F 2 = 1 v n , O L 2 ∑ j , q ∈ E n O L E tr [ ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ] ≤ C v n , O L 2 ∑ m = 1 m n ∑ r , s = 1 L m min ( r , s ) 2 ≤ C v n , O L 2 ∑ m = 1 m n L m 4 . \begin{aligned}
\mathbb{E}\left\|\widehat{\Sigma}_{n,\mathrm{OL}}^{(3,2,0)}-\Sigma\right\|_{\mathrm{F}}^2
=\frac{1}{v_{n,\mathrm{OL}}^2}\sum_{j,q\in E_n^{\mathrm{OL}}}\mathbb{E}\operatorname{tr}\left[\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\right] \\
&\le\frac{C}{v_{n,\mathrm{OL}}^2}\sum_{m=1}^{m_n}\sum_{r,s=1}^{L_m}\min(r,s)^2
\le\frac{C}{v_{n,\mathrm{OL}}^2}\sum_{m=1}^{m_n}L_m^4.
\end{aligned} E Σ n , OL ( 3 , 2 , 0 ) − Σ F 2 = v n , OL 2 1 j , q ∈ E n OL ∑ E tr k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ ≤ v n , OL 2 C m = 1 ∑ m n r , s = 1 ∑ L m min ( r , s ) 2 ≤ v n , OL 2 C m = 1 ∑ m n L m 4 .
The first inequality combines (D.5) and (D.4) within blocks, exchanging j , q j,q j , q if needed; cross terms between blocks are zero. The second uses min ( r , s ) 2 ≤ L m 2 \min(r,s)^2\le L_m^2 min ( r , s ) 2 ≤ L m 2 .
NOL. In contrast, the sets I j I_j I j , j ∈ E n N O L j\in E_n^{\mathrm{NOL}} j ∈ E n NOL , are disjoint, so their noise sums are independent. For distinct selected indices j , q j,q j , q ,
E tr [ ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ] = tr [ E ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) E ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ] = 0. \mathbb{E}\operatorname{tr}\left[\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\right]
=\operatorname{tr}\left[\mathbb{E}\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\mathbb{E}\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\right]=0. E tr k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ = tr E k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ E k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ = 0.
Consequently, we have
E ∥ Σ ^ n , N O L ( 3 , 2 , 0 ) − Σ ∥ F 2 = 1 n 2 ∑ j , q ∈ E n N O L E tr [ ( ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ) ( ( ∑ k = t q q A − 1 ζ k ) ⊗ 2 − ℓ q Σ ) ] = 1 n 2 ∑ j ∈ E n N O L E ∥ ( ∑ k = t j j A − 1 ζ k ) ⊗ 2 − ℓ j Σ ∥ F 2 ≤ C n 2 ∑ j ∈ E n N O L ℓ j 2 . \begin{aligned}
\mathbb{E}\left\|\widehat{\Sigma}_{n,\mathrm{NOL}}^{(3,2,0)}-\Sigma\right\|_{\mathrm{F}}^2
=\frac{1}{n^2}\sum_{j,q\in E_n^{\mathrm{NOL}}}\mathbb{E}\operatorname{tr}\left[\left(\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right)\left(\left(\sum_{k=t_q}^{q}A^{-1}\zeta_k\right)^{\otimes2}-\ell_q\Sigma\right)\right] \\
&=\frac{1}{n^2}\sum_{j\in E_n^{\mathrm{NOL}}}\mathbb{E}\left\|\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\otimes2}-\ell_j\Sigma\right\|_{\mathrm{F}}^2 \\
&\le\frac{C}{n^2}\sum_{j\in E_n^{\mathrm{NOL}}}\ell_j^2.
\end{aligned} E Σ n , NOL ( 3 , 2 , 0 ) − Σ F 2 = n 2 1 j , q ∈ E n NOL ∑ E tr k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ k = t q ∑ q A − 1 ζ k ⊗ 2 − ℓ q Σ = n 2 1 j ∈ E n NOL ∑ E k = t j ∑ j A − 1 ζ k ⊗ 2 − ℓ j Σ F 2 ≤ n 2 C j ∈ E n NOL ∑ ℓ j 2 .
Lemma A.6(ii) for OL and (i) for NOL now imply
E ∥ Σ ^ n , ⋄ ( 3 , 2 , 0 ) − Σ ∥ F 2 ≤ C n − 1 / β , ⋄ ∈ { O L , N O L } . \mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3,2,0)}-\Sigma\right\|_{\mathrm{F}}^2\le Cn^{-1/\beta},\qquad\diamond\in\{\mathrm{OL},\mathrm{NOL}\}. E Σ n , ⋄ ( 3 , 2 , 0 ) − Σ F 2 ≤ C n − 1/ β , ⋄ ∈ { OL , NOL } .
For either choice of ⋄ \diamond ⋄ , Cauchy–Schwarz and ∥ ⋅ ∥ ≤ ∥ ⋅ ∥ F \|\cdot\|\le\|\cdot\|_{\mathrm{F}} ∥ ⋅ ∥ ≤ ∥ ⋅ ∥ F now yield
E ∥ Σ ^ n , ⋄ ( 3 , 2 , 0 ) − Σ ∥ ≤ ( E ∥ Σ ^ n , ⋄ ( 3 , 2 , 0 ) − Σ ∥ F 2 ) 1 / 2 ≤ C n − 1 / ( 2 β ) . (76) \mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3,2,0)}-\Sigma\right\|
\le\left(\mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3,2,0)}-\Sigma\right\|_{\mathrm{F}}^2\right)^{1/2}
\le Cn^{-1/(2\beta)}. \tag*{(76)} E Σ n , ⋄ ( 3 , 2 , 0 ) − Σ ≤ ( E Σ n , ⋄ ( 3 , 2 , 0 ) − Σ F 2 ) 1/2 ≤ C n − 1/ ( 2 β ) . (76)
In particular, sup n , ⋄ E ∥ Σ ^ n , ⋄ ( 3 , 2 , 0 ) ∥ < ∞ \sup_{n,\diamond}\mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3,2,0)}\right\|<\infty sup n , ⋄ E Σ n , ⋄ ( 3 , 2 , 0 ) < ∞ . To restore the centering term, independence and centering imply
E ∥ 1 n ∑ k = 1 n A − 1 ζ k ∥ 2 = 1 n 2 ∑ k , r = 1 n E ⟨ A − 1 ζ k , A − 1 ζ r ⟩ = 1 n 2 ∑ k = 1 n E ∥ A − 1 ζ k ∥ 2 = tr ( Σ ) n . \mathbb{E}\left\|\frac{1}{n}\sum_{k=1}^{n}A^{-1}\zeta_k\right\|^2
=\frac{1}{n^2}\sum_{k,r=1}^{n}\mathbb{E}\langle A^{-1}\zeta_k,A^{-1}\zeta_r\rangle
=\frac{1}{n^2}\sum_{k=1}^{n}\mathbb{E}\|A^{-1}\zeta_k\|^2
=\frac{\operatorname{tr}(\Sigma)}{n}. E n 1 k = 1 ∑ n A − 1 ζ k 2 = n 2 1 k , r = 1 ∑ n E ⟨ A − 1 ζ k , A − 1 ζ r ⟩ = n 2 1 k = 1 ∑ n E ∥ A − 1 ζ k ∥ 2 = n tr ( Σ ) .
Lemma A.6 (i) then implies
1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 E ∥ ( 1 n ∑ r = 1 n A − 1 ζ r ) ∥ 2 = tr ( Σ ) n v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 ≤ C n − 1 / β . (77) \frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j^2\mathbb{E}\left\|\left(\frac{1}{n}\sum_{r=1}^{n}A^{-1}\zeta_r\right)\right\|^2=\frac{\operatorname{tr}(\Sigma)}{nv_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j^2\le Cn^{-1/\beta}. \tag*{(77)} v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 E ( n 1 r = 1 ∑ n A − 1 ζ r ) 2 = n v n , ⋄ tr ( Σ ) j ∈ E n ⋄ ∑ ℓ j 2 ≤ C n − 1/ β . (77)
The centering changes the estimator by
Σ ^ n , ⋄ ( 3 , 2 ) − Σ ^ n , ⋄ ( 3 , 2 , 0 ) = 1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 ( 1 n ∑ r = 1 n A − 1 ζ r ) ⊗ 2 − 1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j ( ∑ k = t j j A − 1 ζ k ) ( 1 n ∑ r = 1 n A − 1 ζ r ) T \widehat{\Sigma}_{n,\diamond}^{(3,2)}-\widehat{\Sigma}_{n,\diamond}^{(3,2,0)}
=\frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j^2\left(\frac{1}{n}\sum_{r=1}^{n}A^{-1}\zeta_r\right)^{\otimes2}
-\frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)\left(\frac{1}{n}\sum_{r=1}^{n}A^{-1}\zeta_r\right)^{\mathsf T} Σ n , ⋄ ( 3 , 2 ) − Σ n , ⋄ ( 3 , 2 , 0 ) = v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 ( n 1 r = 1 ∑ n A − 1 ζ r ) ⊗ 2 − v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j k = t j ∑ j A − 1 ζ k ( n 1 r = 1 ∑ n A − 1 ζ r ) T
− 1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j ( 1 n ∑ r = 1 n A − 1 ζ r ) ( ∑ k = t j j A − 1 ζ k ) T . -\frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j\left(\frac{1}{n}\sum_{r=1}^{n}A^{-1}\zeta_r\right)\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)^{\mathsf T}. − v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j ( n 1 r = 1 ∑ n A − 1 ζ r ) k = t j ∑ j A − 1 ζ k T .
Independence and centering imply
E [ ( ∑ k = t j j A − 1 ζ k ) ( 1 n ∑ r = 1 n A − 1 ζ r ) T ] = 1 n ∑ k = t j j ∑ r = 1 n E [ ( A − 1 ζ k ) ( A − 1 ζ r ) T ] = 1 n ∑ k = t j j Σ = ℓ j n Σ . \mathbb{E}\left[\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k\right)\left(\frac{1}{n}\sum_{r=1}^{n}A^{-1}\zeta_r\right)^{\mathsf T}\right]
=\frac{1}{n}\sum_{k=t_j}^{j}\sum_{r=1}^{n}\mathbb{E}\left[(A^{-1}\zeta_k)(A^{-1}\zeta_r)^{\mathsf T}\right]
=\frac{1}{n}\sum_{k=t_j}^{j}\Sigma
=\frac{\ell_j}{n}\Sigma. E k = t j ∑ j A − 1 ζ k ( n 1 r = 1 ∑ n A − 1 ζ r ) T = n 1 k = t j ∑ j r = 1 ∑ n E [ ( A − 1 ζ k ) ( A − 1 ζ r ) T ] = n 1 k = t j ∑ j Σ = n ℓ j Σ.
These mixed terms need not vanish. Apply Lemma A.5 over E n ⋄ E_n^\diamond E n ⋄ , with D = v n , ⋄ D=v_{n,\diamond} D = v n , ⋄ , a j = 1 a_j=1 a j = 1 , σ j = ∑ k = t j j A − 1 ζ k \sigma_j=\sum_{k=t_j}^{j}A^{-1}\zeta_k σ j = ∑ k = t j j A − 1 ζ k , and e j = − ℓ j n − 1 ∑ r = 1 n A − 1 ζ r e_j=-\ell_jn^{-1}\sum_{r=1}^{n}A^{-1}\zeta_r e j = − ℓ j n − 1 ∑ r = 1 n A − 1 ζ r . By (D.7) and the bounded expected norm following (D.6),
E ∥ Σ ^ n , ⋄ ( 3 , 2 ) − Σ ^ n , ⋄ ( 3 , 2 , 0 ) ∥ ≤ C ( n − 1 / ( 2 β ) + n − 1 / β ) ≤ 2 C n − 1 / ( 2 β ) . \mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3,2)}-\widehat{\Sigma}_{n,\diamond}^{(3,2,0)}\right\|
\le C\left(n^{-1/(2\beta)}+n^{-1/\beta}\right)
\le2Cn^{-1/(2\beta)}. E Σ n , ⋄ ( 3 , 2 ) − Σ n , ⋄ ( 3 , 2 , 0 ) ≤ C ( n − 1/ ( 2 β ) + n − 1/ β ) ≤ 2 C n − 1/ ( 2 β ) .
Since n − 1 / β ≤ n − 1 / ( 2 β ) n^{-1/\beta}\le n^{-1/(2\beta)} n − 1/ β ≤ n − 1/ ( 2 β ) , combining this estimate with (D.6) proves the result. ■
Lemma D.2. Under the assumptions of Lemma D.1, suppose that Assumptions 2.1 and 2.4 also hold, and let γ k = γ k − α \gamma_k=\gamma k^{-\alpha} γ k = γ k − α with γ > 0 \gamma>0 γ > 0 . Then there exists C < ∞ C<\infty C < ∞ such that the covariance estimators based on U k U_k U k and A − 1 ζ k A^{-1}\zeta_k A − 1 ζ k in (D.2) satisfy, for every n ≥ 1 n\ge1 n ≥ 1 and ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } ,
E ∥ Σ ^ n , ⋄ ( 3 , 1 ) − Σ ^ n , ⋄ ( 3 , 2 ) ∥ ≤ C n ( α − 1 ) / 2 + 1 / ( 2 β ) . \mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3,1)}-\widehat{\Sigma}_{n,\diamond}^{(3,2)}\right\|\le Cn^{(\alpha-1)/2+1/(2\beta)}. E Σ n , ⋄ ( 3 , 1 ) − Σ n , ⋄ ( 3 , 2 ) ≤ C n ( α − 1 ) /2 + 1/ ( 2 β ) .
Proof. Lemma A.4 supplies E ∥ r s , t U ∥ 2 ≤ C t α \mathbb{E}\|r_{s,t}^{U}\|^2\le Ct^\alpha E ∥ r s , t U ∥ 2 ≤ C t α , and its proof shows r s , t U = ∑ k = s t ( U k − A − 1 ζ k ) r_{s,t}^{U}=\sum_{k=s}^{t}(U_k-A^{-1}\zeta_k) r s , t U = ∑ k = s t ( U k − A − 1 ζ k ) . The centered batch sums differ by
( ∑ k = t j j U k − ℓ j U ‾ n ) − ( ∑ k = t j j A − 1 ζ k − ℓ j n ∑ k = 1 n A − 1 ζ k ) = ∑ k = t j j ( U k − A − 1 ζ k ) − ℓ j n ∑ k = 1 n ( U k − A − 1 ζ k ) − r t j , j U − ℓ j n r 1 , n U . \begin{aligned}
\left(\sum_{k=t_j}^{j}U_k-\ell_j\overline{U}_n\right)
-\left(\sum_{k=t_j}^{j}A^{-1}\zeta_k-\frac{\ell_j}{n}\sum_{k=1}^{n}A^{-1}\zeta_k\right) \\
&=\sum_{k=t_j}^{j}(U_k-A^{-1}\zeta_k)-\frac{\ell_j}{n}\sum_{k=1}^{n}(U_k-A^{-1}\zeta_k)-r_{t_j,j}^{U}-\frac{\ell_j}{n}r_{1,n}^{U}.
\end{aligned} k = t j ∑ j U k − ℓ j U n − k = t j ∑ j A − 1 ζ k − n ℓ j k = 1 ∑ n A − 1 ζ k = k = t j ∑ j ( U k − A − 1 ζ k ) − n ℓ j k = 1 ∑ n ( U k − A − 1 ζ k ) − r t j , j U − n ℓ j r 1 , n U .
For either batch rule, the remainder bound yields
1 v n , ⋄ ∑ j ∈ E n ⋄ E ∥ r t j , j U − ℓ j n r 1 , n U ∥ 2 ≤ C v n , ⋄ ∑ j ∈ E n ⋄ j α + C n α n 2 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 . \frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\mathbb{E}\left\|r_{t_j,j}^{U}-\frac{\ell_j}{n}r_{1,n}^{U}\right\|^2
\le\frac{C}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}j^\alpha+\frac{Cn^\alpha}{n^2v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j^2. v n , ⋄ 1 j ∈ E n ⋄ ∑ E r t j , j U − n ℓ j r 1 , n U 2 ≤ v n , ⋄ C j ∈ E n ⋄ ∑ j α + n 2 v n , ⋄ C n α j ∈ E n ⋄ ∑ ℓ j 2 .
Applying Lemma A.6 (iv) to the first sum and (i) to the second yields
1 v n , ⋄ ∑ j ∈ E n ⋄ E ∥ ℓ t j , j U − ℓ j n r 1 , n U ∥ 2 ≤ C ( n ( α − 1 ) + 1 / β + n α − 1 − 1 / β ) ≤ 2 C n α − 1 + 1 / β . (78) \frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\mathbb{E}\left\| \ell_{t_j,j}^{U}-\frac{\ell_j}{n}r_{1,n}^{U}\right\|^2 \le C\left(n^{(\alpha-1)+1/\beta}+n^{\alpha-1-1/\beta}\right)\le2Cn^{\alpha-1+1/\beta}. \tag*{(78)} v n , ⋄ 1 j ∈ E n ⋄ ∑ E ℓ t j , j U − n ℓ j r 1 , n U 2 ≤ C ( n ( α − 1 ) + 1/ β + n α − 1 − 1/ β ) ≤ 2 C n α − 1 + 1/ β . (78)
The last step uses β > 0 \beta>0 β > 0 . In Lemma A.5 , take indices E n ⋄ E_n^\diamond E n ⋄ , D = v n , ⋄ D=v_{n,\diamond} D = v n , ⋄ , a j = 1 a_j=1 a j = 1 , and
σ j : = ∑ k = t j j A − 1 ζ k − ℓ j n ∑ k = 1 n A − 1 ζ k , e j : = r t j , j U − ℓ j n r 1 , n U . \sigma_j:=\sum_{k=t_j}^{j}A^{-1}\zeta_k-\frac{\ell_j}{n}\sum_{k=1}^{n}A^{-1}\zeta_k,\qquad e_j:=r_{t_j,j}^{U}-\frac{\ell_j}{n}r_{1,n}^{U}. σ j := k = t j ∑ j A − 1 ζ k − n ℓ j k = 1 ∑ n A − 1 ζ k , e j := r t j , j U − n ℓ j r 1 , n U .
Then Q = Σ ^ n , ⋄ ( 3 , 1 ) Q=\widehat{\Sigma}_{n,\diamond}^{(3,1)} Q = Σ n , ⋄ ( 3 , 1 ) and Q 0 = Σ ^ n , ⋄ ( 3 , 2 ) Q_0=\widehat{\Sigma}_{n,\diamond}^{(3,2)} Q 0 = Σ n , ⋄ ( 3 , 2 ) , with sup n E ∥ Q 0 ∥ < ∞ \sup_n\mathbb{E}\|Q_0\|<\infty sup n E ∥ Q 0 ∥ < ∞ by Lemma D.1. Combining (A.3) and (D.8) yields
E ∥ Σ ^ n , ⋄ ( 3 , 1 ) − Σ ^ n , ⋄ ( 3 , 2 ) ∥ ≤ C ( n ( α − 1 ) / 2 + 1 / ( 2 β ) + n α − 1 + 1 / β ) ≤ 2 C n ( α − 1 ) / 2 + 1 / ( 2 β ) . \mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3,1)}-\widehat{\Sigma}_{n,\diamond}^{(3,2)}\right\|\le C\left(n^{(\alpha-1)/2+1/(2\beta)}+n^{\alpha-1+1/\beta}\right)\le2Cn^{(\alpha-1)/2+1/(2\beta)}. E Σ n , ⋄ ( 3 , 1 ) − Σ n , ⋄ ( 3 , 2 ) ≤ C ( n ( α − 1 ) /2 + 1/ ( 2 β ) + n α − 1 + 1/ β ) ≤ 2 C n ( α − 1 ) /2 + 1/ ( 2 β ) .
The last step uses α − 1 + 1 / β < 0 \alpha-1+1/\beta<0 α − 1 + 1/ β < 0 , proving the claim. ■
Lemma D.3. Under the assumptions of Lemma D.2 , there exists C < ∞ C<\infty C < ∞ such that the covariance estimators based on Δ k \Delta_k Δ k and its linear approximation U k U_k U k satisfy, for every n ≥ 1 n\ge1 n ≥ 1 and ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } ,
E ∥ Σ ^ n , ⋄ ( 3 ) − Σ ^ n , ⋄ ( 3 , 1 ) ∥ ≤ C n 1 / 2 − 1 / ( 2 β ) − α . \mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3)}-\widehat{\Sigma}_{n,\diamond}^{(3,1)}\right\|\le Cn^{1/2-1/(2\beta)-\alpha}. E Σ n , ⋄ ( 3 ) − Σ n , ⋄ ( 3 , 1 ) ≤ C n 1/2 − 1/ ( 2 β ) − α . Proof. Recall d k = Δ k − U k d_k=\Delta_k-U_k d k = Δ k − U k . Since Δ ‾ n − U ‾ n = n − 1 ∑ i = 1 n d i \overline{\Delta}_n-\overline{U}_n=n^{-1}\sum_{i=1}^{n}d_i Δ n − U n = n − 1 ∑ i = 1 n d i , Minkowski’s inequality and Lemma C.4 yield
( E ∥ ∑ k = t j j d k − ℓ j n ∑ i = 1 n d i ∥ 2 ) 1 / 2 ≤ C ∑ k = t j j k − α + C ℓ j n ∑ i = 1 n i − α ≤ C ( 1 + 1 1 − α ) ℓ j t j − α . \left(\mathbb{E}\left\|\sum_{k=t_j}^{j}d_k-\frac{\ell_j}{n}\sum_{i=1}^{n}d_i\right\|^2\right)^{1/2}\le C\sum_{k=t_j}^{j}k^{-\alpha}+\frac{C\ell_j}{n}\sum_{i=1}^{n}i^{-\alpha}\le C\left(1+\frac{1}{1-\alpha}\right)\ell_jt_j^{-\alpha}. E k = t j ∑ j d k − n ℓ j i = 1 ∑ n d i 2 1/2 ≤ C k = t j ∑ j k − α + n C ℓ j i = 1 ∑ n i − α ≤ C ( 1 + 1 − α 1 ) ℓ j t j − α . The last step uses k ≥ t j k\ge t_j k ≥ t j , ∑ i = 1 n i − α ≤ n 1 − α / ( 1 − α ) \sum_{i=1}^{n}i^{-\alpha}\le n^{1-\alpha}/(1-\alpha) ∑ i = 1 n i − α ≤ n 1 − α / ( 1 − α ) , and t j ≤ n t_j\le n t j ≤ n . Squaring and applying Lemma A.6 (iii) yields
1 v n , ⋄ ∑ j ∈ E n ⋄ E ∥ ∑ k = t j j d k − ℓ j n ∑ i = 1 n d i ∥ 2 ≤ C n 1 − 1 / β − 2 α . (79) \frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\mathbb{E}\left\|\sum_{k=t_j}^{j}d_k-\frac{\ell_j}{n}\sum_{i=1}^{n}d_i\right\|^2\le Cn^{1-1/\beta-2\alpha}. \tag*{(79)} v n , ⋄ 1 j ∈ E n ⋄ ∑ E k = t j ∑ j d k − n ℓ j i = 1 ∑ n d i 2 ≤ C n 1 − 1/ β − 2 α . (79) In Lemma A.5 , take indices E n ⋄ E_n^\diamond E n ⋄ , D = v n , ⋄ D=v_{n,\diamond} D = v n , ⋄ , a j = 1 a_j=1 a j = 1 , σ j = ∑ k = t j j U k − ℓ j U ‾ n \sigma_j=\sum_{k=t_j}^{j}U_k-\ell_j\overline{U}_n σ j = ∑ k = t j j U k − ℓ j U n , and e j = ∑ k = t j j d k − ℓ j n ∑ i = 1 n d i e_j=\sum_{k=t_j}^{j}d_k-\frac{\ell_j}{n}\sum_{i=1}^{n}d_i e j = ∑ k = t j j d k − n ℓ j ∑ i = 1 n d i . Then Q = Σ ^ n , ⋄ ( 3 ) Q=\widehat{\Sigma}_{n,\diamond}^{(3)} Q = Σ n , ⋄ ( 3 ) and Q 0 = Σ ^ n , ⋄ ( 3 , 1 ) Q_0=\widehat{\Sigma}_{n,\diamond}^{(3,1)} Q 0 = Σ n , ⋄ ( 3 , 1 ) , with sup n E ∥ Q 0 ∥ < ∞ \sup_n\mathbb{E}\|Q_0\|<\infty sup n E ∥ Q 0 ∥ < ∞ by Lemmas D.1 and D.2 . Combining (A.3) and (D.9) yields
E ∥ Σ ^ n , ⋄ ( 3 ) − Σ ^ n , ⋄ ( 3 , 1 ) ∥ ≤ C ( n 1 / 2 − 1 / ( 2 β ) − α + n 1 − 1 / β − 2 α ) ≤ 2 C n 1 / 2 − 1 / ( 2 β ) − α . \mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3)}-\widehat{\Sigma}_{n,\diamond}^{(3,1)}\right\|\le C\left(n^{1/2-1/(2\beta)-\alpha}+n^{1-1/\beta-2\alpha}\right)\le2Cn^{1/2-1/(2\beta)-\alpha}. E Σ n , ⋄ ( 3 ) − Σ n , ⋄ ( 3 , 1 ) ≤ C ( n 1/2 − 1/ ( 2 β ) − α + n 1 − 1/ β − 2 α ) ≤ 2 C n 1/2 − 1/ ( 2 β ) − α . The last step uses 1 − 1 / β − 2 α < 0 1-1/\beta-2\alpha<0 1 − 1/ β − 2 α < 0 , proving the claim. ■
Proposition D.4. Under the assumptions of Lemma D.2 , there exists C < ∞ C<\infty C < ∞ such that, for every n ≥ 1 n\ge1 n ≥ 1 and ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } ,
E ∥ Σ ^ n , ⋄ ( 3 ) − Σ ∥ ≤ C ( n − 1 / ( 2 β ) + n ( α − 1 ) / 2 + 1 / ( 2 β ) ) . \mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3)}-\Sigma\right\|\le C\left(n^{-1/(2\beta)}+n^{(\alpha-1)/2+1/(2\beta)}\right). E Σ n , ⋄ ( 3 ) − Σ ≤ C ( n − 1/ ( 2 β ) + n ( α − 1 ) /2 + 1/ ( 2 β ) ) . Proof. Lemmas D.1, D.2 , and D.3 yield
E ∥ Σ ^ n , ⋄ ( 3 ) − Σ ∥ ≤ C ( n − 1 / ( 2 β ) + n ( α − 1 ) / 2 + 1 / ( 2 β ) + n 1 / 2 − 1 / ( 2 β ) − α ) . \mathbb{E}\left\|\widehat{\Sigma}_{n,\diamond}^{(3)}-\Sigma\right\|\le C\left(n^{-1/(2\beta)}+n^{(\alpha-1)/2+1/(2\beta)}+n^{1/2-1/(2\beta)-\alpha}\right). E Σ n , ⋄ ( 3 ) − Σ ≤ C ( n − 1/ ( 2 β ) + n ( α − 1 ) /2 + 1/ ( 2 β ) + n 1/2 − 1/ ( 2 β ) − α ) . Since α > 1 / 2 \alpha>1/2 α > 1/2 , the last term equals n − 1 / ( 2 β ) n 1 / 2 − α n^{-1/(2\beta)}n^{1/2-\alpha} n − 1/ ( 2 β ) n 1/2 − α and can be absorbed into the first. Both remaining exponents are negative because β > 1 / ( 1 − α ) \beta>1/(1-\alpha) β > 1/ ( 1 − α ) , so sup n E ∥ Σ ^ n , ⋄ ( 3 ) ∥ < ∞ \sup_n\mathbb{E}\|\widehat{\Sigma}_{n,\diamond}^{(3)}\|<\infty sup n E ∥ Σ n , ⋄ ( 3 ) ∥ < ∞ . This proves the claim. ■
Geometric errors
Starting from Proposition D.4 , we bound the first three terms of (D.1) in reverse order: averaging, change of base point, and online vector updates. This establishes sup n E ∥ Q 0 ∥ < ∞ \sup_n \mathbb{E}\lVert Q_0\rVert< \infty sup n E ∥ Q 0 ∥ < ∞ before each use of Lemma A.5 . All three applications take D = v n , ⋄ D=v_{n,\diamond} D = v n , ⋄ and a j = ℓ j 2 a_j=\ell_j^2 a j = ℓ j 2 for j ∈ E n ⋄ j\in E_n^\diamond j ∈ E n ⋄ , with Q 0 Q_0 Q 0 formed from σ j \sigma_j σ j and Q Q Q from σ j + e j \sigma_j+e_j σ j + e j . If E ∥ e j ∥ 2 ≤ C t j − 2 α \mathbb{E}\lVert e_j\rVert^2\le Ct_j^{-2\alpha} E ∥ e j ∥ 2 ≤ C t j − 2 α , then
δ = 1 v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 E ∥ e j ∥ 2 ≤ C v n , ⋄ ∑ j ∈ E n ⋄ ℓ j 2 t j − 2 α . \delta=\frac{1}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j^2\mathbb{E}\lVert e_j\rVert^2\le\frac{C}{v_{n,\diamond}}\sum_{j\in E_n^\diamond}\ell_j^2t_j^{-2\alpha}. δ = v n , ⋄ 1 j ∈ E n ⋄ ∑ ℓ j 2 E ∥ e j ∥ 2 ≤ v n , ⋄ C j ∈ E n ⋄ ∑ ℓ j 2 t j − 2 α .
Lemma A.6 (iii) therefore yields
δ ≤ C n 1 − 1 / β − 2 α . (80) \delta\le Cn^{1-1/\beta-2\alpha}. \tag*{(80)} δ ≤ C n 1 − 1/ β − 2 α . (80)
If sup n E ∥ Q 0 ∥ < ∞ \sup_n\mathbb{E}\lVert Q_0\rVert<\infty sup n E ∥ Q 0 ∥ < ∞ , we obtain
E ∥ Q − Q 0 ∥ ≤ C ( δ 1 / 2 + δ ) . (81) \mathbb{E}\lVert Q-Q_0\rVert\le C(\delta^{1/2}+\delta). \tag*{(81)} E ∥ Q − Q 0 ∥ ≤ C ( δ 1/2 + δ ) . (81)
Since 1 − 1 / β − 2 α < 0 1-1/\beta-2\alpha<0 1 − 1/ β − 2 α < 0 , n 1 − 1 / β − 2 α ≤ n 1 / 2 − 1 / ( 2 β ) − α n^{1-1/\beta-2\alpha}\le n^{1/2-1/(2\beta)-\alpha} n 1 − 1/ β − 2 α ≤ n 1/2 − 1/ ( 2 β ) − α . Substituting the bound on δ \delta δ and absorbing the smaller term yields
E ∥ Q − Q 0 ∥ ≤ C n 1 / 2 − 1 / ( 2 β ) − α . (82) \mathbb{E}\lVert Q-Q_0\rVert\le Cn^{1/2-1/(2\beta)-\alpha}. \tag*{(82)} E ∥ Q − Q 0 ∥ ≤ C n 1/2 − 1/ ( 2 β ) − α . (82)
Each proof below identifies the reference and error vectors and verifies the required bound on Q 0 Q_0 Q 0 .
Error from averaging on the manifold. We first bound the difference between recursive retraction averaging and arithmetic averaging in T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M .
Proposition D.5. Under the assumptions of Lemma D.2 , there exists C < ∞ C<\infty C < ∞ such that, for every n ≥ 1 n\ge1 n ≥ 1 and ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } ,
E ∥ Σ ~ n , ⋄ ( 2 ) − Σ ~ n , ⋄ ( 3 ) ∥ ≤ C n 1 / 2 − 1 / ( 2 β ) − α . \mathbb{E}\lVert\widetilde{\Sigma}_{n,\diamond}^{(2)}-\widetilde{\Sigma}_{n,\diamond}^{(3)}\rVert\le Cn^{1/2-1/(2\beta)-\alpha}. E ∥ Σ n , ⋄ ( 2 ) − Σ n , ⋄ ( 3 ) ∥ ≤ C n 1/2 − 1/ ( 2 β ) − α .
Proof. For j ∈ E n ⋄ j\in E_n^\diamond j ∈ E n ⋄ , Lemma C.6 yields
E ∥ ( Δ ~ j B − Δ ~ n ) − ( Δ j B − Δ n ) ∥ 2 ≤ C ( γ t j 2 + γ n 2 ) ≤ 2 C γ 2 t j − 2 α . \mathbb{E}\left\lVert(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n)-(\Delta_j^B-\Delta_n)\right\rVert^2\le C(\gamma_{t_j}^2+\gamma_n^2)\le2C\gamma^2t_j^{-2\alpha}. E ( Δ j B − Δ n ) − ( Δ j B − Δ n ) 2 ≤ C ( γ t j 2 + γ n 2 ) ≤ 2 C γ 2 t j − 2 α .
The second inequality uses γ n ≤ γ t j = γ t j − α \gamma_n\le\gamma_{t_j}=\gamma t_j^{-\alpha} γ n ≤ γ t j = γ t j − α . In (D.11), take σ j = Δ j B − Δ n \sigma_j=\Delta_j^B-\Delta_n σ j = Δ j B − Δ n and e j = ( Δ ~ j B − Δ ~ n ) − ( Δ j B − Δ n ) e_j=(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n)-(\Delta_j^B-\Delta_n) e j = ( Δ j B − Δ n ) − ( Δ j B − Δ n ) , so Q 0 = Σ ~ n , ⋄ ( 3 ) Q_0=\widetilde{\Sigma}_{n,\diamond}^{(3)} Q 0 = Σ n , ⋄ ( 3 ) and Q = Σ ~ n , ⋄ ( 2 ) Q=\widetilde{\Sigma}_{n,\diamond}^{(2)} Q = Σ n , ⋄ ( 2 ) . Proposition D.4 implies sup n E ∥ Q 0 ∥ < ∞ \sup_n\mathbb{E}\lVert Q_0\rVert<\infty sup n E ∥ Q 0 ∥ < ∞ , verifying the remaining condition and proving the claim. ■
Error from changing the base point. We next compare ( Γ x ~ n x ⋆ ) − 1 b j , n (\Gamma_{\widetilde{x}_n}^{x_\star})^{-1}b_{j,n} ( Γ x n x ⋆ ) − 1 b j , n with Δ ~ j B − Δ ~ n \widetilde{\Delta}_j^B-\widetilde{\Delta}_n Δ j B − Δ n for each batch and then sum the resulting bounds with weights ℓ j 2 / v n , ⋄ \ell_j^2/v_{n,\diamond} ℓ j 2 / v n , ⋄ .
Lemma D.6. Under the assumptions of Lemma C.5 , suppose additionally that Assumption 2.4 (b) holds. For each batch B j = { x t j , … , x j } B_j=\{x_{t_j},\ldots,x_j\} B j = { x t j , … , x j } , let c j c_j c j be its recursive mean from (11) . Recall that b j , n = R x ~ n − 1 ( c j ) b_{j,n}=R_{\widetilde{x}_n}^{-1}(c_j) b j , n = R x n − 1 ( c j ) and Δ ~ j B = R x ⋆ − 1 ( c j ) \widetilde{\Delta}_j^B=R_{x_\star}^{-1}(c_j) Δ j B = R x ⋆ − 1 ( c j ) represent this mean at x ~ n \widetilde{x}_n x n and x ⋆ x_\star x ⋆ , respectively. Then there exists C > 0 C>0 C > 0 such that, for every n ≥ 1 n\ge1 n ≥ 1 and 1 ≤ j ≤ n 1\le j\le n 1 ≤ j ≤ n ,
E ∥ ( Γ x ~ n x ⋆ ) − 1 b j , n − ( Δ ~ j B − Δ ~ n ) ∥ 2 ≤ C t j − 2 α . \mathbb{E}\left\lVert(\Gamma_{\widetilde{x}_n}^{x_\star})^{-1}b_{j,n}-(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n)\right\rVert^2\le Ct_j^{-2\alpha}. E ( Γ x n x ⋆ ) − 1 b j , n − ( Δ j B − Δ n ) 2 ≤ C t j − 2 α .
Proof. Fix 1 ≤ j ≤ n 1 \le j \le n 1 ≤ j ≤ n . We have
( Γ x ⋆ x ~ n ) − 1 b j , n − ( Δ ~ j B − Δ ~ n ) = ( [ Λ x ⋆ x ~ n ] − 1 b j , n − ( Δ ~ j B − Δ ~ n ) ) + ( ( Γ x ⋆ x ~ n ) − 1 − [ Λ x ⋆ x ~ n ] − 1 ) b j , n . (\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}b_{j,n}-(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n)=\left([\Lambda_{x_\star}^{\widetilde{x}_n}]^{-1}b_{j,n}-(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n)\right)+\left((\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}-[\Lambda_{x_\star}^{\widetilde{x}_n}]^{-1}\right)b_{j,n}. ( Γ x ⋆ x n ) − 1 b j , n − ( Δ j B − Δ n ) = ( [ Λ x ⋆ x n ] − 1 b j , n − ( Δ j B − Δ n ) ) + ( ( Γ x ⋆ x n ) − 1 − [ Λ x ⋆ x n ] − 1 ) b j , n .
Apply (B.8) with x = x ⋆ x=x_\star x = x ⋆ , u = Δ ~ n u=\widetilde{\Delta}_n u = Δ n , and w = Δ ~ j B − Δ ~ n w=\widetilde{\Delta}_j^B-\widetilde{\Delta}_n w = Δ j B − Δ n to the first term, and (B.10) to the second. The domain conditions hold since u u u and u + w u+w u + w represent points in X \mathcal{X} X . Thus
∥ ( Γ x ⋆ x ~ n ) − 1 b j , n − ( Δ ~ j B − Δ ~ n ) ∥ ≤ C Ψ ∥ Δ ~ n ∥ ∥ Δ ~ j B − Δ ~ n ∥ + C Γ , Λ ∥ Δ ~ n ∥ 2 ∥ b j , n ∥ x ~ n . (83) \left\|(\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}b_{j,n}-(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n)\right\|\le C_\Psi\|\widetilde{\Delta}_n\|\|\widetilde{\Delta}_j^B-\widetilde{\Delta}_n\|+C_{\Gamma,\Lambda}\|\widetilde{\Delta}_n\|^2\|b_{j,n}\|_{\widetilde{x}_n}. \tag*{(83)} ( Γ x ⋆ x n ) − 1 b j , n − ( Δ j B − Δ n ) ≤ C Ψ ∥ Δ n ∥∥ Δ j B − Δ n ∥ + C Γ , Λ ∥ Δ n ∥ 2 ∥ b j , n ∥ x n . (83)
Taking x = x ~ n x=\widetilde{x}_n x = x n and z = c j z=c_j z = c j , we obtain
∥ b j , n ∥ x ~ n ≤ C ∥ Δ ~ j B − Δ ~ n ∥ . (84) \|b_{j,n}\|_{\widetilde{x}_n}\le C\|\widetilde{\Delta}_j^B-\widetilde{\Delta}_n\|. \tag*{(84)} ∥ b j , n ∥ x n ≤ C ∥ Δ j B − Δ n ∥. (84)
Combining this with (D.12) and ∥ Δ ~ n ∥ ≤ D X \|\widetilde{\Delta}_n\|\le D_{\mathcal{X}} ∥ Δ n ∥ ≤ D X bounds the pointwise error by C ∥ Δ ~ n ∥ ∥ Δ ~ j B − Δ ~ n ∥ C\|\widetilde{\Delta}_n\|\|\widetilde{\Delta}_j^B-\widetilde{\Delta}_n\| C ∥ Δ n ∥∥ Δ j B − Δ n ∥ . Cauchy–Schwarz then yields
E ∥ ( Γ x ⋆ x ~ n ) − 1 b j , n − ( Δ ~ j B − Δ ~ n ) ∥ 2 ≤ C ( E ∥ Δ ~ n ∥ 4 ) 1 / 2 ( E ∥ Δ ~ j B − Δ ~ n ∥ 4 ) 1 / 2 . \mathbb{E}\left\|(\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}b_{j,n}-(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n)\right\|^2\le C\left(\mathbb{E}\|\widetilde{\Delta}_n\|^4\right)^{1/2}\left(\mathbb{E}\|\widetilde{\Delta}_j^B-\widetilde{\Delta}_n\|^4\right)^{1/2}. E ( Γ x ⋆ x n ) − 1 b j , n − ( Δ j B − Δ n ) 2 ≤ C ( E ∥ Δ n ∥ 4 ) 1/2 ( E ∥ Δ j B − Δ n ∥ 4 ) 1/2 .
Lemma C.5 and γ n ≤ γ t j \gamma_n\le\gamma_{t_j} γ n ≤ γ t j bound the two fourth moments by C γ n 2 C\gamma_n^2 C γ n 2 and C γ t j 2 C\gamma_{t_j}^2 C γ t j 2 , respectively. Therefore,
E ∥ ( Γ x ⋆ x ~ n ) − 1 b j , n − ( Δ ~ j B − Δ ~ n ) ∥ 2 ≤ C γ n γ t j ≤ C γ t j 2 t j − 2 α . \mathbb{E}\left\|(\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}b_{j,n}-(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n)\right\|^2\le C\gamma_n\gamma_{t_j}\le C\gamma_{t_j}^2t_j^{-2\alpha}. E ( Γ x ⋆ x n ) − 1 b j , n − ( Δ j B − Δ n ) 2 ≤ C γ n γ t j ≤ C γ t j 2 t j − 2 α .
The last inequality uses γ n ≤ γ t j = γ t j − α \gamma_n\le\gamma_{t_j}=\gamma^{t_j^{-\alpha}} γ n ≤ γ t j = γ t j − α . This proves the bound for the change of base point. ■
Proposition D.7. Under the assumptions of Lemma D.2, there exists C < ∞ C<\infty C < ∞ such that, for every n ≥ 1 n\ge1 n ≥ 1 and ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } ,
E ∥ Σ ^ n , ⋄ ( 1 ) − Σ ^ n , ⋄ ( 2 ) ∥ ≤ C n 1 / 2 − 1 / ( 2 β ) − α . \mathbb{E}\|\widehat{\Sigma}_{n,\diamond}^{(1)}-\widehat{\Sigma}_{n,\diamond}^{(2)}\|\le Cn^{1/2-1/(2\beta)-\alpha}. E ∥ Σ n , ⋄ ( 1 ) − Σ n , ⋄ ( 2 ) ∥ ≤ C n 1/2 − 1/ ( 2 β ) − α .
Proof. In (D.11), take σ j = Δ ~ j B − Δ ~ n \sigma_j=\widetilde{\Delta}_j^B-\widetilde{\Delta}_n σ j = Δ j B − Δ n and e j = ( Γ x ⋆ x ~ n ) − 1 b j , n − ( Δ ~ j B − Δ ~ n ) e_j=(\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}b_{j,n}-(\widetilde{\Delta}_j^B-\widetilde{\Delta}_n) e j = ( Γ x ⋆ x n ) − 1 b j , n − ( Δ j B − Δ n ) . Then Q 0 = Σ ^ n , ⋄ ( 2 ) Q_0=\widehat{\Sigma}_{n,\diamond}^{(2)} Q 0 = Σ n , ⋄ ( 2 ) and Q = Σ ^ n , ⋄ ( 1 ) Q=\widehat{\Sigma}_{n,\diamond}^{(1)} Q = Σ n , ⋄ ( 1 ) . Lemma D.6 implies E ∥ e j ∥ 2 ≤ C t j − 2 α \mathbb{E}\|e_j\|^2\le Ct_j^{-2\alpha} E ∥ e j ∥ 2 ≤ C t j − 2 α , and Propositions D.4 and D.5 ensure sup n E ∥ Q 0 ∥ < ∞ \sup_n\mathbb{E}\|Q_0\|<\infty sup n E ∥ Q 0 ∥ < ∞ . Thus (D.11) proves the claim. ■
Error from the online vector updates. Finally, for each batch j j j , we bound b ~ j , n − b j , n \widetilde{b}_{j,n}-b_{j,n} b j , n − b j , n and then sum these bounds with weights ℓ j 2 / v n , ⋄ \ell_j^2/v_{n,\diamond} ℓ j 2 / v n , ⋄ .
Lemma D.8. Under the assumptions of Lemma D.6, let b ~ j , n \widetilde{b}_{j,n} b j , n be the recursive approximation of b j , n b_{j,n} b j , n from (13) , initialized by b ~ j , j = b j , j \widetilde{b}_{j,j}=b_{j,j} b j , j = b j , j . Then there exists C > 0 C>0 C > 0 such that, for every n ≥ 1 n\ge1 n ≥ 1 and 1 ≤ j ≤ n 1\le j\le n 1 ≤ j ≤ n ,
E ∥ ( Γ x ⋆ x ~ n ) − 1 ( b ~ j , n − b j , n ) ∥ 2 ≤ C t j − 2 α . \mathbb{E}\left\|(\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}(\widetilde{b}_{j,n}-b_{j,n})\right\|^2\le Ct_j^{-2\alpha}. E ( Γ x ⋆ x n ) − 1 ( b j , n − b j , n ) 2 ≤ C t j − 2 α .
Proof. For k ≥ j k\ge j k ≥ j , set d j , k : = b ~ j , k − b j , k ∈ T x ~ k M d_{j,k}:=\widetilde{b}_{j,k}-b_{j,k}\in T_{\widetilde{x}_k}M d j , k := b j , k − b j , k ∈ T x k M , with d j , j = 0 d_{j,j}=0 d j , j = 0 . First bound the products of later transports. Recall u k = k x ~ k − 1 − 1 R x ~ k − 1 − 1 ( x k ) u_k=k\widetilde{x}_{k-1}^{-1}R_{\widetilde{x}_{k-1}}^{-1}(x_k) u k = k x k − 1 − 1 R x k − 1 − 1 ( x k ) . Assumption 2.1(a) implies ∥ u k ∥ x ~ k − 1 < ρ r e t / k \|u_k\|_{\widetilde{x}_{k-1}}<\rho_{\mathrm{ret}}/k ∥ u k ∥ x k − 1 < ρ ret / k and u k = x ~ k − 1 − 1 R x ~ k − 1 − 1 ( x ~ k ) u_k=\widetilde{x}_{k-1}^{-1}R_{\widetilde{x}_{k-1}}^{-1}(\widetilde{x}_k) u k = x k − 1 − 1 R x k − 1 − 1 ( x k ) . By (B.5), ∥ Λ x ~ k − 1 x ~ k ∥ ≤ 1 + C L ρ r e t 2 k − 2 \|\Lambda_{\widetilde{x}_{k-1}}^{\widetilde{x}_k}\|\le1+C_L\rho_{\mathrm{ret}}^2k^{-2} ∥ Λ x k − 1 x k ∥ ≤ 1 + C L ρ ret 2 k − 2 . Submultiplicativity and 1 + t ≤ e t 1+t\le e^t 1 + t ≤ e t yield, for k < n k<n k < n ,
∥ Λ x ~ n − 1 x ~ n ∘ ⋯ ∘ Λ x ~ k + 1 x ~ k ∥ ≤ ∏ r = k + 1 n ( 1 + C L ρ r e t 2 r − 2 ) ≤ exp ( C L ρ r e t 2 ∑ r = 1 ∞ r − 2 ) ≤ C < ∞ . (85) \left\|\Lambda_{\widetilde{x}_{n-1}}^{\widetilde{x}_n}\circ\cdots\circ\Lambda_{\widetilde{x}_{k+1}}^{\widetilde{x}_k}\right\|\le\prod_{r=k+1}^{n}(1+C_L\rho_{\mathrm{ret}}^2r^{-2})\le\exp\left(C_L\rho_{\mathrm{ret}}^2\sum_{r=1}^{\infty}r^{-2}\right)\le C<\infty. \tag*{(85)} Λ x n − 1 x n ∘ ⋯ ∘ Λ x k + 1 x k ≤ r = k + 1 ∏ n ( 1 + C L ρ ret 2 r − 2 ) ≤ exp ( C L ρ ret 2 r = 1 ∑ ∞ r − 2 ) ≤ C < ∞. (85)
This deterministic bound is uniform in k , n k,n k , n and also covers k = n k=n k = n , when the composition is the identity. Next, for k > j k>j k > j , define the error in one linear update by
ρ j , k : = b j , k − Λ x ~ k − 1 x ~ k ( b j , k − 1 − u k ) . \rho_{j,k}:=b_{j,k}-\Lambda^{\widetilde{x}_k}_{\widetilde{x}_{k-1}}(b_{j,k-1}-u_k). ρ j , k := b j , k − Λ x k − 1 x k ( b j , k − 1 − u k ) .
Apply (B.8) with x = x ~ k − 1 x=\widetilde{x}_{k-1} x = x k − 1 , u = u k u=u_k u = u k , and w = b j , k − 1 − u k w=b_{j,k-1}-u_k w = b j , k − 1 − u k ; the domain conditions hold since u u u and u + w u+w u + w represent x ~ k , c j ∈ X \widetilde{x}_k,c_j\in\mathcal{X} x k , c j ∈ X . Multiplying by Λ x ~ k − 1 x ~ k \Lambda^{\widetilde{x}_k}_{\widetilde{x}_{k-1}} Λ x k − 1 x k and using (B.4) yields
∥ ρ j , k ∥ x ~ k ≤ C ∥ u k ∥ x ~ k − 1 ( ∥ b j , k − 1 ∥ x ~ k − 1 + ∥ u k ∥ x ~ k − 1 ) . (86) \lVert\rho_{j,k}\rVert_{\widetilde{x}_k}\le C\lVert u_k\rVert_{\widetilde{x}_{k-1}}\left(\lVert b_{j,k-1}\rVert_{\widetilde{x}_{k-1}}+\lVert u_k\rVert_{\widetilde{x}_{k-1}}\right). \tag*{(86)} ∥ ρ j , k ∥ x k ≤ C ∥ u k ∥ x k − 1 ( ∥ b j , k − 1 ∥ x k − 1 + ∥ u k ∥ x k − 1 ) . (86)
The distance comparison in Lemma B.2 implies
E ∥ u k ∥ x ~ k − 1 4 ≤ C k − 4 ( E ∥ Δ ~ k − 1 ∥ 4 + E ∥ Δ k ∥ 4 ) , \mathbb{E}\lVert u_k\rVert_{\widetilde{x}_{k-1}}^4\le Ck^{-4}\left(\mathbb{E}\lVert\widetilde{\Delta}_{k-1}\rVert^4+\mathbb{E}\lVert\Delta_k\rVert^4\right), E ∥ u k ∥ x k − 1 4 ≤ C k − 4 ( E ∥ Δ k − 1 ∥ 4 + E ∥ Δ k ∥ 4 ) ,
E ∥ b j , k − 1 ∥ x ~ k − 1 4 ≤ C ( E ∥ Δ ~ k − 1 ∥ 4 + E ∥ Δ ~ j B ∥ 4 ) . \mathbb{E}\lVert b_{j,k-1}\rVert_{\widetilde{x}_{k-1}}^4\le C\left(\mathbb{E}\lVert\widetilde{\Delta}_{k-1}\rVert^4+\mathbb{E}\lVert\widetilde{\Delta}^{B}_j\rVert^4\right). E ∥ b j , k − 1 ∥ x k − 1 4 ≤ C ( E ∥ Δ k − 1 ∥ 4 + E ∥ Δ j B ∥ 4 ) .
Lemma C.5 , (C.8), k > j ≥ t j k>j\ge t_j k > j ≥ t j , and γ k − 1 ≤ 2 α γ k \gamma_{k-1}\le2^\alpha\gamma_k γ k − 1 ≤ 2 α γ k then imply
E ∥ u k ∥ x ~ k − 1 4 ≤ C k − 4 γ k 2 , E ∥ b j , k − 1 ∥ x ~ k − 1 4 ≤ C γ t j 2 . \mathbb{E}\lVert u_k\rVert_{\widetilde{x}_{k-1}}^4\le Ck^{-4}\gamma_k^2,\qquad\mathbb{E}\lVert b_{j,k-1}\rVert_{\widetilde{x}_{k-1}}^4\le C\gamma_{t_j}^2. E ∥ u k ∥ x k − 1 4 ≤ C k − 4 γ k 2 , E ∥ b j , k − 1 ∥ x k − 1 4 ≤ C γ t j 2 .
Minkowski yields the first inequality below, and Cauchy–Schwarz yields the second:
( E ∥ ρ j , k ∥ x ~ k 2 ) 1 / 2 ≤ C ( ( E [ ∥ u k ∥ x ~ k − 1 ∥ b j , k − 1 ∥ x ~ k − 1 ] ) 1 / 2 + ( E ∥ u k ∥ x ~ k − 1 2 ) 1 / 2 ) ≤ C ( ( E ∥ u k ∥ x ~ k − 1 4 ) 1 / 4 ( E ∥ b j , k − 1 ∥ x ~ k − 1 4 ) 1 / 4 + ( E ∥ u k ∥ x ~ k − 1 2 ) 1 / 2 ) . \begin{aligned}
\left(\mathbb{E}\lVert\rho_{j,k}\rVert_{\widetilde{x}_k}^2\right)^{1/2}
\le C\left(\left(\mathbb{E}\left[\lVert u_k\rVert_{\widetilde{x}_{k-1}}\lVert b_{j,k-1}\rVert_{\widetilde{x}_{k-1}}\right]\right)^{1/2}
+\left(\mathbb{E}\lVert u_k\rVert_{\widetilde{x}_{k-1}}^2\right)^{1/2}\right) \\
&\le C\left(\left(\mathbb{E}\lVert u_k\rVert_{\widetilde{x}_{k-1}}^4\right)^{1/4}\left(\mathbb{E}\lVert b_{j,k-1}\rVert_{\widetilde{x}_{k-1}}^4\right)^{1/4}
+\left(\mathbb{E}\lVert u_k\rVert_{\widetilde{x}_{k-1}}^2\right)^{1/2}\right).
\end{aligned} ( E ∥ ρ j , k ∥ x k 2 ) 1/2 ≤ C ( ( E [ ∥ u k ∥ x k − 1 ∥ b j , k − 1 ∥ x k − 1 ] ) 1/2 + ( E ∥ u k ∥ x k − 1 2 ) 1/2 ) ≤ C ( ( E ∥ u k ∥ x k − 1 4 ) 1/4 ( E ∥ b j , k − 1 ∥ x k − 1 4 ) 1/4 + ( E ∥ u k ∥ x k − 1 2 ) 1/2 ) .
Substitute the two fourth-moment bounds above and use k − 2 γ k ≤ k − 1 γ k 1 / 2 γ t j 1 / 2 k^{-2}\gamma_k\le k^{-1}\gamma_k^{1/2}\gamma_{t_j}^{1/2} k − 2 γ k ≤ k − 1 γ k 1/2 γ t j 1/2 , since k > j ≥ t j k>j\ge t_j k > j ≥ t j . We obtain
( E ∥ ρ j , k ∥ x ~ k 2 ) 1 / 2 ≤ C k − 1 γ k 1 / 2 γ t j 1 / 2 . (87) \left(\mathbb{E}\lVert\rho_{j,k}\rVert_{\widetilde{x}_k}^2\right)^{1/2}\le Ck^{-1}\gamma_k^{1/2}\gamma_{t_j}^{1/2}. \tag*{(87)} ( E ∥ ρ j , k ∥ x k 2 ) 1/2 ≤ C k − 1 γ k 1/2 γ t j 1/2 . (87)
Subtracting the exact update from (13) and iterating from d j , j = 0 d_{j,j}=0 d j , j = 0 yields
d j , k = Λ x ~ k − 1 x ~ k d j , k − 1 − ρ j , k , d_{j,k}=\Lambda^{\widetilde{x}_k}_{\widetilde{x}_{k-1}}d_{j,k-1}-\rho_{j,k}, d j , k = Λ x k − 1 x k d j , k − 1 − ρ j , k ,
d j , n = − ∑ k = j + 1 n ( Λ x ~ n − 1 x ~ n ∘ ⋯ ∘ Λ x ~ k x ~ k + 1 ) ρ j , k . d_{j,n}=-\sum_{k=j+1}^{n}\left(\Lambda^{\widetilde{x}_n}_{\widetilde{x}_{n-1}}\circ\cdots\circ\Lambda^{\widetilde{x}_{k+1}}_{\widetilde{x}_k}\right)\rho_{j,k}. d j , n = − k = j + 1 ∑ n ( Λ x n − 1 x n ∘ ⋯ ∘ Λ x k x k + 1 ) ρ j , k .
Taking norms and applying Minkowski’s inequality yields
( E ∥ d j , n ∥ x ~ n 2 ) 1 / 2 ≤ C ∑ k = j + 1 n ( E ∥ ρ j , k ∥ x ~ k 2 ) 1 / 2 . \left(\mathbb{E}\lVert d_{j,n}\rVert_{\widetilde{x}_n}^2\right)^{1/2}\le C\sum_{k=j+1}^{n}\left(\mathbb{E}\lVert\rho_{j,k}\rVert_{\widetilde{x}_k}^2\right)^{1/2}. ( E ∥ d j , n ∥ x n 2 ) 1/2 ≤ C k = j + 1 ∑ n ( E ∥ ρ j , k ∥ x k 2 ) 1/2 .
Using γ k = γ k − α \gamma_k=\gamma k^{-\alpha} γ k = γ k − α in the one-step error bound, we obtain
( E ∥ d j , n ∥ x ~ n 2 ) 1 / 2 ≤ C t j − α / 2 ∑ k = j + 1 n k − 1 − α / 2 ≤ C t j − α / 2 ∫ j ∞ x − 1 − α / 2 d x = 2 C α j − α / 2 t j − α / 2 ≤ 2 C α t j − α . \begin{aligned}
\left(\mathbb{E}\lVert d_{j,n}\rVert_{\widetilde{x}_n}^2\right)^{1/2}\le Ct_j^{-\alpha/2}\sum_{k=j+1}^{n}k^{-1-\alpha/2} \\
&\le Ct_j^{-\alpha/2}\int_j^\infty x^{-1-\alpha/2}\,dx=\frac{2C}{\alpha}j^{-\alpha/2}t_j^{-\alpha/2}\le\frac{2C}{\alpha}t_j^{-\alpha}.
\end{aligned} ( E ∥ d j , n ∥ x n 2 ) 1/2 ≤ C t j − α /2 k = j + 1 ∑ n k − 1 − α /2 ≤ C t j − α /2 ∫ j ∞ x − 1 − α /2 d x = α 2 C j − α /2 t j − α /2 ≤ α 2 C t j − α .
The integral comparison uses that x − 1 − α / 2 x^{-1-\alpha/2} x − 1 − α /2 is decreasing; the last step uses j ≥ t j ≥ 1 j\ge t_j\ge1 j ≥ t j ≥ 1 . Squaring and using the isometry of parallel transport proves the claim, including n = j n=j n = j , when the error is zero. ■
Proposition D.9. Under the assumptions of Lemma D.2, there exists C < ∞ C < \infty C < ∞ such that, for every n ≥ 1 n \ge1 n ≥ 1 and ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } ,
E ∥ Σ ^ n , ⋄ ( 0 ) − Σ ^ n , ⋄ ( 1 ) ∥ ≤ C n 1 / 2 − 1 / ( 2 β ) − α . \mathbb{E}\left\|\widehat{\Sigma}^{(0)}_{n,\diamond}-\widehat{\Sigma}^{(1)}_{n,\diamond}\right\| \le Cn^{1/2-1/(2\beta)-\alpha}. E Σ n , ⋄ ( 0 ) − Σ n , ⋄ ( 1 ) ≤ C n 1/2 − 1/ ( 2 β ) − α .
Proof. In (D.11), take σ j = ( Γ x ⋆ x ~ n ) − 1 b j , n \sigma_j=(\Gamma^{\widetilde{x}_n}_{x_\star})^{-1}b_{j,n} σ j = ( Γ x ⋆ x n ) − 1 b j , n and e j = ( Γ x ⋆ x ~ n ) − 1 ( b ~ j , n − b j , n ) e_j=(\Gamma^{\widetilde{x}_n}_{x_\star})^{-1}(\widetilde{b}_{j,n}-b_{j,n}) e j = ( Γ x ⋆ x n ) − 1 ( b j , n − b j , n ) , so Q 0 = Σ ^ n , ⋄ ( 1 ) Q_0=\widehat{\Sigma}^{(1)}_{n,\diamond} Q 0 = Σ n , ⋄ ( 1 ) and Q = Σ ^ n , ⋄ ( 0 ) Q=\widehat{\Sigma}^{(0)}_{n,\diamond} Q = Σ n , ⋄ ( 0 ) . Lemma D.8 implies E ∥ e j ∥ 2 ≤ C t j − 2 α \mathbb{E}\|e_j\|^2\le Ct_j^{-2\alpha} E ∥ e j ∥ 2 ≤ C t j − 2 α , including j = n j=n j = n . Propositions D.4, D.5, and D.7 imply sup n E ∥ Q 0 ∥ < ∞ \sup_n\mathbb{E}\|Q_0\|<\infty sup n E ∥ Q 0 ∥ < ∞ . Thus (D.11) proves the claim. ■
Proof of Theorem 3.1. For either ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } , apply Propositions D.9, D.7, D.5, and D.4 to the four terms in (D.1), respectively. We obtain
E ∥ Σ ^ n , ⋄ ( 0 ) − Σ ∥ ≤ C ( n − 1 / ( 2 β ) + n ( α − 1 ) / 2 + 1 / ( 2 β ) + n 1 / 2 − 1 / ( 2 β ) − α ) . \mathbb{E}\left\|\widehat{\Sigma}^{(0)}_{n,\diamond}-\Sigma\right\|\le C\left(n^{-1/(2\beta)}+n^{(\alpha-1)/2+1/(2\beta)}+n^{1/2-1/(2\beta)-\alpha}\right). E Σ n , ⋄ ( 0 ) − Σ ≤ C ( n − 1/ ( 2 β ) + n ( α − 1 ) /2 + 1/ ( 2 β ) + n 1/2 − 1/ ( 2 β ) − α ) .
The ratio of the last term to n − 1 / ( 2 β ) n^{-1/(2\beta)} n − 1/ ( 2 β ) is n 1 / 2 − α → 0 n^{1/2-\alpha}\to0 n 1/2 − α → 0 . Absorbing it into the first term proves the theorem. ■
Proof of Corollary 3.2. The two decay exponents in Theorem 3.1 are 1 / ( 2 β ) 1/(2\beta) 1/ ( 2 β ) and ( 1 − α ) / 2 − 1 / ( 2 β ) (1-\alpha)/2-1/(2\beta) ( 1 − α ) /2 − 1/ ( 2 β ) . Equating them yields β = 2 / ( 1 − α ) \beta=2/(1-\alpha) β = 2/ ( 1 − α ) , which satisfies the required strict lower bound. Substituting this value into either exponent yields ( 1 − α ) / 4 (1-\alpha)/4 ( 1 − α ) /4 , as claimed. ■
Proof of Proposition 4.1
Proof of Proposition 4.1. Fix ⋄ ∈ { O L , N O L } \diamond\in\{\mathrm{OL},\mathrm{NOL}\} ⋄ ∈ { OL , NOL } . For the joint region, compare R x ~ n − 1 ( x ⋆ ) R_{\widetilde{x}_n}^{-1}(x_\star) R x n − 1 ( x ⋆ ) with Δ ~ n = R x ⋆ − 1 ( x ~ n ) \widetilde{\Delta}_n=R_{x_\star}^{-1}(\widetilde{x}_n) Δ n = R x ⋆ − 1 ( x n ) after transport to T x ⋆ M T_{x_\star}\mathcal{M} T x ⋆ M :
∥ ( Γ x ⋆ x ~ n ) − 1 R x ~ n − 1 ( x ⋆ ) + Δ ~ n ∥ x ⋆ ≤ ∥ [ Λ x ⋆ x ~ n ] − 1 R x ~ n − 1 ( x ⋆ ) + Δ ~ n ∥ x ⋆ + ∥ ( ( Γ x ⋆ x ~ n ) − 1 − [ Λ x ⋆ x ~ n ] − 1 ) R x ~ n − 1 ( x ⋆ ) ∥ x ⋆ ≤ C Ψ ∥ Δ ~ n ∥ x ⋆ 2 + C Γ , Λ ∥ Δ ~ n ∥ x ⋆ 2 ∥ R x ~ n − 1 ( x ⋆ ) ∥ x ~ n ≲ ∥ Δ ~ n ∥ x ⋆ 2 . \begin{aligned}
\left\|(\Gamma^{\widetilde{x}_n}_{x_\star})^{-1}R_{\widetilde{x}_n}^{-1}(x_\star)+\widetilde{\Delta}_n\right\|_{x_\star}\le\left\|[\Lambda^{\widetilde{x}_n}_{x_\star}]^{-1}R_{\widetilde{x}_n}^{-1}(x_\star)+\widetilde{\Delta}_n\right\|_{x_\star}+\left\|((\Gamma^{\widetilde{x}_n}_{x_\star})^{-1}-[\Lambda^{\widetilde{x}_n}_{x_\star}]^{-1})R_{\widetilde{x}_n}^{-1}(x_\star)\right\|_{x_\star} \\
&\le C_{\Psi}\|\widetilde{\Delta}_n\|_{x_\star}^{2}+C_{\Gamma,\Lambda}\|\widetilde{\Delta}_n\|_{x_\star}^{2}\|R_{\widetilde{x}_n}^{-1}(x_\star)\|_{\widetilde{x}_n}\lesssim\|\widetilde{\Delta}_n\|_{x_\star}^{2}.
\end{aligned} ( Γ x ⋆ x n ) − 1 R x n − 1 ( x ⋆ ) + Δ n x ⋆ ≤ [ Λ x ⋆ x n ] − 1 R x n − 1 ( x ⋆ ) + Δ n x ⋆ + (( Γ x ⋆ x n ) − 1 − [ Λ x ⋆ x n ] − 1 ) R x n − 1 ( x ⋆ ) x ⋆ ≤ C Ψ ∥ Δ n ∥ x ⋆ 2 + C Γ , Λ ∥ Δ n ∥ x ⋆ 2 ∥ R x n − 1 ( x ⋆ ) ∥ x n ≲ ∥ Δ n ∥ x ⋆ 2 .
The second inequality applies (B.8) at ( x , u , w ) = ( x ⋆ , Δ ~ n , − Δ ~ n ) (x,u,w)=(x_\star,\widetilde{\Delta}_n,-\widetilde{\Delta}_n) ( x , u , w ) = ( x ⋆ , Δ n , − Δ n ) and (B.10) to the two terms, respectively. The last uses ∥ R x ~ n − 1 ( x ⋆ ) ∥ x ~ n ≤ C M C M − 1 ∥ Δ ~ n ∥ x ⋆ \|R_{\widetilde{x}_n}^{-1}(x_\star)\|_{\widetilde{x}_n}\le C_{\mathcal{M}}C_{\mathcal{M}}^{-1}\|\widetilde{\Delta}_n\|_{x_\star} ∥ R x n − 1 ( x ⋆ ) ∥ x n ≤ C M C M − 1 ∥ Δ n ∥ x ⋆ from (B.1) and boundedness of R x ⋆ − 1 ( X ) R_{x_\star}^{-1}(\mathcal{X}) R x ⋆ − 1 ( X ) . By Theorem 2.5, Δ ~ n = O P ( n − 1 / 2 ) \widetilde{\Delta}_n=O_{\mathbb{P}}(n^{-1/2}) Δ n = O P ( n − 1/2 ) and x ~ n → P x ⋆ \widetilde{x}_n\xrightarrow{\mathbb{P}}x_\star x n P x ⋆ , so
n ( Γ x ⋆ x ~ n ) − 1 R x ~ n − 1 ( x ⋆ ) = − n Δ ~ n + o P ( 1 ) → d N ( 0 , Σ ) . (88) \sqrt{n}(\Gamma^{\widetilde{x}_n}_{x_\star})^{-1}R_{\widetilde{x}_n}^{-1}(x_\star)=-\sqrt{n}\widetilde{\Delta}_n+o_{\mathbb{P}}(1)\xrightarrow{\mathrm{d}}\mathcal{N}(0,\Sigma). \tag*{(88)} n ( Γ x ⋆ x n ) − 1 R x n − 1 ( x ⋆ ) = − n Δ n + o P ( 1 ) d N ( 0 , Σ ) . (88)
Set Σ ^ n , ⋄ : = ( Γ x ⋆ x ~ n ) − 1 ∘ Σ ~ n , ⋄ ∘ Γ x ⋆ x ~ n \widehat{\Sigma}_{n,\diamond}:=(\Gamma^{\widetilde{x}_n}_{x_\star})^{-1}\circ\widetilde{\Sigma}_{n,\diamond}\circ\Gamma^{\widetilde{x}_n}_{x_\star} Σ n , ⋄ := ( Γ x ⋆ x n ) − 1 ∘ Σ n , ⋄ ∘ Γ x ⋆ x n . Theorem 3.1 implies
Σ ^ n , ⋄ → P Σ . (89) \widehat{\Sigma}_{n,\diamond}\xrightarrow{\mathbb{P}}\Sigma. \tag*{(89)} Σ n , ⋄ P Σ. (89)
Positive definiteness of Σ \Sigma Σ , the isometry of parallel transport, and Weyl’s inequality imply P ( λ min ( Σ ~ n , ⋄ ) ≥ λ min ( Σ ) / 2 ) → 1 \mathbb{P}\left(\lambda_{\min}(\widetilde{\Sigma}_{n,\diamond})\ge\lambda_{\min}(\Sigma)/2\right)\to1 P ( λ m i n ( Σ n , ⋄ ) ≥ λ m i n ( Σ ) /2 ) → 1 . Continuity of inversion implies Σ ^ n , ⋄ − 1 → P Σ − 1 \widehat{\Sigma}_{n,\diamond}^{-1}\xrightarrow{\mathbb{P}}\Sigma^{-1} Σ n , ⋄ − 1 P Σ − 1 . Combining this with (E.1), Slutsky’s theorem and continuity of the quadratic form yield
n ⟨ R x ~ n − 1 ( x ⋆ ) , Σ ~ n , ⋄ − 1 R x ~ n − 1 ( x ⋆ ) ⟩ x ~ n → d χ d 2 . (90) n\left\langle R_{\widetilde{x}_n}^{-1}(x_\star),\widetilde{\Sigma}_{n,\diamond}^{-1}R_{\widetilde{x}_n}^{-1}(x_\star)\right\rangle_{\widetilde{x}_n}\xrightarrow{\mathrm{d}}\chi_d^2. \tag*{(90)} n ⟨ R x n − 1 ( x ⋆ ) , Σ n , ⋄ − 1 R x n − 1 ( x ⋆ ) ⟩ x n d χ d 2 . (90)
Recall C n , ⋄ ( 1 − δ ) = R x ~ n ( E n , ⋄ ( 1 − δ ) ) C_{n,\diamond}(1-\delta)=R_{\widetilde{x}_n}\left(\mathcal{E}_{n,\diamond}(1-\delta)\right) C n , ⋄ ( 1 − δ ) = R x n ( E n , ⋄ ( 1 − δ ) ) . By (E.2) and Weyl’s inequality, λ max ( Σ ~ n , ⋄ ) = O P ( 1 ) \lambda_{\max}(\widetilde{\Sigma}_{n,\diamond})=O_{\mathrm{P}}(1) λ m a x ( Σ n , ⋄ ) = O P ( 1 ) . On the positive-definite event, the spectral theorem implies ∥ v ∥ x ~ n 2 ≤ λ max ( Σ ~ n , ⋄ ) ⟨ v , Σ ~ n , ⋄ − 1 v ⟩ x ~ n \lVert v\rVert_{\widetilde{x}_n}^{2}\leq\lambda_{\max}(\widetilde{\Sigma}_{n,\diamond})\langle v,\widetilde{\Sigma}_{n,\diamond}^{-1}v\rangle_{\widetilde{x}_n} ∥ v ∥ x n 2 ≤ λ m a x ( Σ n , ⋄ ) ⟨ v , Σ n , ⋄ − 1 v ⟩ x n for every v ∈ T x ~ n M v\in T_{\widetilde{x}_n}\mathcal{M} v ∈ T x n M . The ellipsoid definition in (22) therefore yields
sup v ∈ E n , ⋄ ( 1 − δ ) ∥ v ∥ x ~ n 2 ≤ χ d , 1 − δ 2 λ max ( Σ ~ n , ⋄ ) n = O P ( n − 1 ) . \sup_{v\in\mathcal{E}_{n,\diamond}(1-\delta)}\lVert v\rVert_{\widetilde{x}_n}^{2}\leq\frac{\chi_{d,1-\delta}^{2}\lambda_{\max}(\widetilde{\Sigma}_{n,\diamond})}{n}=O_{\mathrm{P}}(n^{-1}). v ∈ E n , ⋄ ( 1 − δ ) sup ∥ v ∥ x n 2 ≤ n χ d , 1 − δ 2 λ m a x ( Σ n , ⋄ ) = O P ( n − 1 ) .
Thus the ellipsoid lies in B x ~ n ( 0 x ~ n , ρ r e t ) B_{\widetilde{x}_n}(0_{\widetilde{x}_n},\rho_{\mathrm{ret}}) B x n ( 0 x n , ρ ret ) with probability tending to one. Assumption 2.1(a) makes R x ~ n R_{\widetilde{x}_n} R x n one-to-one on this ball, which also contains R x ~ n − 1 ( x ⋆ ) R_{\widetilde{x}_n}^{-1}(x_\star) R x n − 1 ( x ⋆ ) . On this event, x ⋆ ∈ C n , ⋄ ( 1 − δ ) x_\star\in C_{n,\diamond}(1-\delta) x ⋆ ∈ C n , ⋄ ( 1 − δ ) if and only if R x ~ n − 1 ( x ⋆ ) ∈ E n , ⋄ ( 1 − δ ) R_{\widetilde{x}_n}^{-1}(x_\star)\in\mathcal{E}_{n,\diamond}(1-\delta) R x n − 1 ( x ⋆ ) ∈ E n , ⋄ ( 1 − δ ) . Hence
P ( x ⋆ ∈ C n , ⋄ ( 1 − δ ) ) = P ( n ⟨ R x ~ n − 1 ( x ⋆ ) , Σ ~ n , ⋄ − 1 R x ~ n − 1 ( x ⋆ ) ⟩ x ~ n ≤ χ d , 1 − δ 2 ) + o ( 1 ) → (E.3) 1 − δ . \mathbb{P}\left(x_\star\in C_{n,\diamond}(1-\delta)\right)=\mathbb{P}\left(n\left\langle R_{\widetilde{x}_n}^{-1}(x_\star),\widetilde{\Sigma}_{n,\diamond}^{-1}R_{\widetilde{x}_n}^{-1}(x_\star)\right\rangle_{\widetilde{x}_n}\leq\chi_{d,1-\delta}^{2}\right)+o(1)\xrightarrow{\text{(E.3)}}{}1-\delta. P ( x ⋆ ∈ C n , ⋄ ( 1 − δ ) ) = P ( n ⟨ R x n − 1 ( x ⋆ ) , Σ n , ⋄ − 1 R x n − 1 ( x ⋆ ) ⟩ x n ≤ χ d , 1 − δ 2 ) + o ( 1 ) (E.3) 1 − δ .
For the scalar interval, a first-order expansion of h ∘ R x ⋆ h\circ R_{x_\star} h ∘ R x ⋆ and Theorem 2.5 yield
n ( h ( x ~ n ) − h ( x ⋆ ) ) = D h ( x ⋆ ) [ n R x ⋆ − 1 ( x ~ n ) ] + o P ( 1 ) → d N ( 0 , σ h 2 ) , \sqrt{n}\left(h(\widetilde{x}_n)-h(x_\star)\right)=Dh(x_\star)\left[\sqrt{n}R_{x_\star}^{-1}(\widetilde{x}_n)\right]+o_{\mathrm{P}}(1)\xrightarrow{d}N(0,\sigma_h^{2}), n ( h ( x n ) − h ( x ⋆ ) ) = D h ( x ⋆ ) [ n R x ⋆ − 1 ( x n ) ] + o P ( 1 ) d N ( 0 , σ h 2 ) ,
where σ h 2 : = ⟨ grad h ( x ⋆ ) , Σ grad h ( x ⋆ ) ⟩ x ⋆ > 0 \sigma_h^{2}:=\langle\operatorname{grad}h(x_\star),\Sigma\operatorname{grad}h(x_\star)\rangle_{x_\star}>0 σ h 2 := ⟨ grad h ( x ⋆ ) , Σ grad h ( x ⋆ ) ⟩ x ⋆ > 0 . Here D R x ⋆ ( 0 x ⋆ ) = I DR_{x_\star}(0_{x_\star})=I D R x ⋆ ( 0 x ⋆ ) = I , and the remainder satisfies n o ( ∥ Δ ~ n ∥ x ⋆ ) = o P ( 1 ) \sqrt{n}o(\lVert\widetilde{\Delta}_n\rVert_{x_\star})=o_{\mathrm{P}}(1) n o (∥ Δ n ∥ x ⋆ ) = o P ( 1 ) . By the isometry of parallel transport, (20) becomes
σ ^ n , h , ⋄ 2 = ⟨ ( Γ x ⋆ x ~ n ) − 1 grad h ( x ~ n ) , Σ ~ n , ⋄ ( Γ x ⋆ x ~ n ) − 1 grad h ( x ~ n ) ⟩ x ⋆ . \widehat{\sigma}_{n,h,\diamond}^{2}=\left\langle(\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}\operatorname{grad}h(\widetilde{x}_n),\widetilde{\Sigma}_{n,\diamond}(\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}\operatorname{grad}h(\widetilde{x}_n)\right\rangle_{x_\star}. σ n , h , ⋄ 2 = ⟨ ( Γ x ⋆ x n ) − 1 grad h ( x n ) , Σ n , ⋄ ( Γ x ⋆ x n ) − 1 grad h ( x n ) ⟩ x ⋆ .
Continuity of the gradient and parallel transport implies ( Γ x ⋆ x ~ n ) − 1 grad h ( x ~ n ) → P grad h ( x ⋆ ) (\Gamma_{x_\star}^{\widetilde{x}_n})^{-1}\operatorname{grad}h(\widetilde{x}_n)\xrightarrow{\mathrm{P}}\operatorname{grad}h(x_\star) ( Γ x ⋆ x n ) − 1 grad h ( x n ) P grad h ( x ⋆ ) , and (E.2) therefore implies σ ^ n , h , ⋄ 2 → P σ h 2 \widehat{\sigma}_{n,h,\diamond}^{2}\xrightarrow{\mathrm{P}}\sigma_h^{2} σ n , h , ⋄ 2 P σ h 2 . Thus, σ ^ n , h , ⋄ > 0 \widehat{\sigma}_{n,h,\diamond}>0 σ n , h , ⋄ > 0 with probability tending to one. Set the following ratio to zero if its denominator is zero. Slutsky’s theorem then yields
n ( h ( x ~ n ) − h ( x ⋆ ) ) σ ^ n , h , ⋄ → d N ( 0 , 1 ) . \frac{\sqrt{n}\left(h(\widetilde{x}_n)-h(x_\star)\right)}{\widehat{\sigma}_{n,h,\diamond}}\xrightarrow{d}N(0,1). σ n , h , ⋄ n ( h ( x n ) − h ( x ⋆ ) ) d N ( 0 , 1 ) .
Since P ( σ ^ n , h , ⋄ > 0 ) → 1 \mathbb{P}(\widehat{\sigma}_{n,h,\diamond}>0)\to1 P ( σ n , h , ⋄ > 0 ) → 1 and the standard normal distribution is continuous, we have
P ( h ( x ⋆ ) ∈ I n , h , ⋄ ( 1 − δ ) ) = P ( ∣ n ( h ( x ~ n ) − h ( x ⋆ ) ) σ ^ n , h , ⋄ ∣ ≤ z 1 − δ / 2 ) + o ( 1 ) ⟶ 1 − δ . \mathbb{P}\left(h(x_\star)\in I_{n,h,\diamond}(1-\delta)\right)=\mathbb{P}\left(\left|\frac{\sqrt{n}\left(h(\widetilde{x}_n)-h(x_\star)\right)}{\widehat{\sigma}_{n,h,\diamond}}\right|\leq z_{1-\delta/2}\right)+o(1)\longrightarrow1-\delta. P ( h ( x ⋆ ) ∈ I n , h , ⋄ ( 1 − δ ) ) = P ( σ n , h , ⋄ n ( h ( x n ) − h ( x ⋆ ) ) ≤ z 1 − δ /2 ) + o ( 1 ) ⟶ 1 − δ .
This proves the assertion for the scalar confidence interval.
Additional numerical experiments
This appendix supplements § 5 with further experiments on covariance estimation and statistical inference. We first compare covariance errors in higher dimensions, then examine interval coverage and length across different dimensions and eigengaps. All experiments use the Gaussian model and algorithm settings described in § 5.1 , with 200 independent runs for each setting.
Covariance estimation in higher dimensions
We extend the covariance estimation experiments in § 5.2 to G 15 , 3 G_{15,3} G 15 , 3 , G 20 , 4 G_{20,4} G 20 , 4 , and G 25 , 5 G_{25,5} G 25 , 5 , with eigengaps Δ λ ∈ { 1 , 0.25 } \Delta_{\lambda} \in\{1, 0.25\} Δ λ ∈ { 1 , 0.25 } and batch constants c ∈ { 1 , 2 , 5 } c \in\{1, 2, 5\} c ∈ { 1 , 2 , 5 } . Recall from § 4.2 that G p , k G_{p,k} G p , k is the space of k k k -dimensional subspaces of R p \mathbb{R}^{p} R p , with dimension d = k ( p − k ) d = k(p-k) d = k ( p − k ) . Thus, the dimensions of these three manifolds are 36, 64, and 100, respectively.
Figure 6 shows an overall decrease in the mean errors of both estimators as the sample size increases. At larger sample sizes, the slopes of the log–log curves are close to the reference slope − 1 / 8 -1/8 − 1/8 . At n = 3 × 10 5 n = 3 \times10^{5} n = 3 × 1 0 5 , c = 1 c = 1 c = 1 gives the lowest mean error among the three batch constants for both estimators in all six settings.
Figure 6. Gaussian streaming PCA: log mean covariance error (operator norm) against log sample size. Columns correspond to G 15 , 3 G_{15,3} G 15 , 3 , G 20 , 4 G_{20,4} G 20 , 4 , and G 25 , 5 G_{25,5} G 25 , 5 , and the top and bottom rows use eigengaps 1 and 0.25, respectively. As in Figure 4 , darker and lighter curves represent OL and NOL, with blue, orange, and green indicating c = 1 , 2 , 5 c = 1, 2, 5 c = 1 , 2 , 5 , respectively. Each mean is based on 200 independent runs. The black dashed lines indicate the reference slope − 1 / 8 -1/8 − 1/8 .
Additional scalar confidence intervals
We next extend the confidence-interval comparison in § 5.3 to a wider range of dimensions and eigengaps. Recall from § 4.2 that h q ( [ X ] ) = ∥ X T q ∥ 2 h_q([X]) = \lVert X^{\mathsf{T}}q \rVert^2 h q ([ X ]) = ∥ X T q ∥ 2 is the squared length of the projection of a fixed unit vector q q q onto the subspace [ X ] [X] [ X ] . We select q q q as in § 5.3 and construct nominal 95% confidence intervals for h q ( x ⋆ ) h_q(x_{\star}) h q ( x ⋆ ) using the OL and NOL variance estimates, with c = 1 c = 1 c = 1 . For comparison, we also include oracle intervals based on the true scalar asymptotic variance.
Table 2 reports empirical coverage and mean interval length at n = 3 × 10 5 n = 3 \times10^{5} n = 3 × 1 0 5 for all twelve settings, including the two reported in Table 1. OL and NOL have similar mean interval lengths within each setting, and their empirical coverage ranges from 0.880 to 0.925. In particular, on G 25 , 5 G_{25,5} G 25 , 5 with the smaller eigengap Δ λ = 0.25 \Delta_{\lambda} = 0.25 Δ λ = 0.25 , the coverage rates are 0.885 and 0.880 for OL and NOL, respectively. Although these coverage rates are below the nominal 95% level, coverage for both methods is close to 90% at this sample size, even in this higher-dimensional problem with a small eigengap.
Table 2. Empirical coverage and mean length of nominal 95% confidence intervals for h q ( x ⋆ ) h_q(x_\star) h q ( x ⋆ ) in Gaussian streaming PCA, with n = 3 × 10 5 n = 3 \times10^5 n = 3 × 1 0 5 and c = 1 c = 1 c = 1 . Results are based on 200 independent runs, with Monte Carlo standard errors in parentheses. Mean lengths are multiplied by 10 3 10^3 1 0 3 .
Figure 7 complements Table 2 by showing how coverage and mean interval length change with the sample size on G 3 , 1 G_{3,1} G 3 , 1 and G 25 , 5 G_{25,5} G 25 , 5 . For both eigengaps, the intervals become shorter on average as the sample size increases. OL and NOL have similar coverage and mean interval lengths, with empirical coverage around 90% at larger sample sizes.
Figure 7. Gaussian streaming PCA: empirical coverage and mean length of 95% confidence intervals for h q ( x ⋆ ) h_q(x_\star) h q ( x ⋆ ) , with c = 1 c = 1 c = 1 . Columns correspond to G 3 , 1 G_{3,1} G 3 , 1 and G 25 , 5 G_{25,5} G 25 , 5 ; the upper and lower pairs of rows use eigengaps 1 and 0.25, respectively, with coverage above mean length in each pair. The blue, orange, and purple curves show OL, NOL, and Oracle, respectively, with results based on 200 independent runs. The small plots inside the coverage panels show the later coverage values, and the red dashed lines mark the nominal coverage level 0.95.
Together, these Gaussian experiments complement the local theory by showing decreasing covariance errors and similar interval performance for OL and NOL across the settings considered.