Introduction
We consider the stochastic optimization problem:
min x ∈ R d f ( x ) = E ξ ∼ P [ F ( x ; ξ ) ] , (1) \min_{x\in\mathbb{R}^{d}} f(x) = \mathbb{E}_{\xi\sim\mathcal{P}}[F(x;\xi)], \tag*{(1)} x ∈ R d min f ( x ) = E ξ ∼ P [ F ( x ; ξ )] , (1)
where f : R d → R f:\mathbb{R}^{d}\to\mathbb{R} f : R d → R is a stochastic objective and F ( ⋅ ; ξ ) : R d → R F(\cdot;\xi):\mathbb{R}^{d}\to\mathbb{R} F ( ⋅ ; ξ ) : R d → R is its realization with ξ ∼ P \xi\sim\mathcal{P} ξ ∼ P . Problem (1) is one of the most fundamental building blocks for many decision-making tasks in statistics and machine learning, such as online recommendation [22 ] , precision medicine [19 ] , and portfolio allocation [11 ] . In these applications, (1) is interpreted as a parameter estimation problem, where x x x denotes the model parameter, ξ \xi ξ denotes an observation, and the minimizer x ⋆ = arg min x ∈ R d f ( x ) x^\star=\operatorname*{arg\,min}_{x\in\mathbb{R}^{d}}f(x) x ⋆ = arg min x ∈ R d f ( x ) denotes the true model parameter.
The ubiquity of streaming data has made online methods increasingly attractive, where only one single sample is processed at each step. Stochastic gradient descent (SGD) and its many acceleration variates are prevalent examples [29 ] , with extensively studied convergence properties [27 , 23 ] . Although first-order methods enjoy inexpensive updates, their performance can be sensitive to ill-conditioning and heterogeneous gradient noise. Moreover, online inference with SGD still requires updating a proper covariance matrix, incurring O ( d 2 ) O(d^2) O ( d 2 ) time and memory complexity. The resulting confidence intervals can also exhibit substantial undercoverage even in relatively low-dimensional regression problems [36 ] . Online Newton methods address these issues by preconditioning each stochastic gradient with curvature information [3 ] :
x t + 1 = x t + φ t Δ x t , with B t Δ x t = − ∇ F ( x t ; ξ t ) , (2) x_{t+1}=x_t+\varphi_t\Delta x_t,\qquad\text{with}\qquad B_t\Delta x_t=-\nabla F(x_t;\xi_t), \tag*{(2)} x t + 1 = x t + φ t Δ x t , with B t Δ x t = − ∇ F ( x t ; ξ t ) , (2)
*School of Industrial and Systems Engineering, Georgia Institute of Technology.
†School of Mathematics, Georgia Institute of Technology. where B t B_t B t approximates the Hessian ∇ 2 f ( x t ) \nabla^2 f(x_t) ∇ 2 f ( x t ) .
Beyond convergence, reliable decision-making also requires uncertainty quantification for the resulting estimator. A standard approach for SGD is Polyak–Ruppert averaging, which averages the iterates to yield an asymptotically normal estimator [30 , 28 ] . Building on prior works, the asymptotic properties of the last and averaged iterates of online Newton methods have also been established in unconstrained and constrained settings [21 , 12 ] . In particular, under suitable regularity conditions and B t → B ⋆ = ∇ 2 f ( x ⋆ ) B_t \to B^\star= \nabla^2 f(x^\star) B t → B ⋆ = ∇ 2 f ( x ⋆ ) , we have
1 φ t ( x t − x ⋆ ) → d N ( 0 , ι ⋅ Ω ) and t ( x ˉ t − x ⋆ ) → d N ( 0 , Ω ) , (3) \frac{1}{\sqrt{\varphi_t}}(x_t-x^\star) \xrightarrow{d} \mathcal{N}(0,\iota\cdot\Omega) \qquad\text{and} \qquad\sqrt{t}(\bar{x}_t-x^\star) \xrightarrow{d} \mathcal{N}(0,\Omega), \tag*{(3)} φ t 1 ( x t − x ⋆ ) d N ( 0 , ι ⋅ Ω ) and t ( x ˉ t − x ⋆ ) d N ( 0 , Ω ) , (3)
where x ˉ t : = 1 t ∑ i = 0 t − 1 x i \bar{x}_t := \frac{1}{t}\sum_{i=0}^{t-1}x_i x ˉ t := t 1 ∑ i = 0 t − 1 x i , Ω = ( B ⋆ ) − 1 Cov ( ∇ F ( x ⋆ ; ξ ) ) ( B ⋆ ) − 1 \Omega=(B^\star)^{-1}\operatorname{Cov}(\nabla F(x^\star;\xi))(B^\star)^{-1} Ω = ( B ⋆ ) − 1 Cov ( ∇ F ( x ⋆ ; ξ )) ( B ⋆ ) − 1 (see (9) ), and ι = C φ / ( 2 C φ − 1 ) \iota=C_\varphi/(2C_\varphi-1) ι = C φ / ( 2 C φ − 1 ) if φ t = C φ / ( t + 1 ) φ \varphi_t=C_\varphi/(t+1)^\varphi φ t = C φ / ( t + 1 ) φ with φ = 1 \varphi=1 φ = 1 while ι = 0.5 \iota=0.5 ι = 0.5 if φ ∈ ( 0.5 , 1 ) \varphi\in(0.5,1) φ ∈ ( 0.5 , 1 ) . The limiting covariance Ω \Omega Ω is further known to be asymptotic minimax optimal in Hájek and Le Cam sense [9 , 5 ] . Though averaged SGD also attains Ω \Omega Ω [30 , 28 ] , the last iterate of SGD is not asymptotically optimal and its limiting covariance solves a Lyapunov equation involving the Hessian. For online Newton, preconditioning gives the scalar form ι Ω \iota\Omega ι Ω in (3) . In particular, when φ t = 1 / ( t + 1 ) \varphi_t=1/(t+1) φ t = 1/ ( t + 1 ) , the last iterate attains Ω \Omega Ω , so inference can be based directly on it without iterate averaging [1 , 21 ] . Appendix A compares the rate orders and last-iterate stepsize conditions of SGD and online Newton. Online inference also requires covariance estimation. For averaged SGD, the batch-means estimator studied in Zhu et al. [36 ] has rate O ( 1 / t φ t ) O(1/\sqrt{t\varphi_t}) O ( 1/ t φ t ) . For online Newton methods, the weighted sample covariance estimators studied in Kuang et al. [20 ] ; ); Wang et al. [34 ] achieve O ( 1 / t φ t ) O(1/\sqrt{t\varphi_t}) O ( 1/ t φ t ) , a quadratic improvement over the batch-means rate. Despite promising statistical properties, a key computational bottleneck of online Newton methods is solving the Newton system in (2) , which requires O ( d 3 ) O(d^3) O ( d 3 ) flops at each step.
Online sketched Newton methods approximately solve the Newton system using an inner randomized iterative solver that performs a fixed number of sketch-and-project updates, reducing the per-step cost to O ( d 2 ) O(d^2) O ( d 2 ) [24 ] . A recent Nesterov-accelerated variant further improves the decay rate of the solver’s approximation error while preserving the same order of computational cost [6 ] . Despite these computational gains, the sketching solver degrades the favorable statistical efficiency of the outer Newton iterates. In particular, Wang et al. [34 ] and Du et al. [8 ] established asymptotic normality analogous to (3) for the last and averaged sketched Newton iterates, with limiting covariances Σ g , a s ℓ \Sigma^\ell_{g,\mathrm{as}} Σ g , as ℓ and Σ g , a s a \Sigma^a_{g,\mathrm{as}} Σ g , as a , respectively. Their results reveal an interesting computational–statistical trade-off: randomized sketching avoids the exact Newton solve but introduces an additional covariance term that inflates the limiting covariance. Specifically, the last and averaged iterates have limiting covariances satisfying Ω ⪯ Σ g , a s ℓ ⪯ Σ g , a s a \Omega\preceq\Sigma^\ell_{g,\mathrm{as}} \preceq\Sigma^a_{g,\mathrm{as}} Ω ⪯ Σ g , as ℓ ⪯ Σ g , as a . Therefore, we ask:
Can sketching solver reduce the cost of online Newton without inflating the limiting covariance?
In this paper, we provide an affirmative answer to this question by replacing the single-sample gradient g t : = ∇ F ( x t ; ξ t ) g_t := \nabla F(x_t;\xi_t) g t := ∇ F ( x t ; ξ t ) with its moving average m t = ( 1 − η t ) m t − 1 + η t g t m_t=(1-\eta_t)m_{t-1}+\eta_tg_t m t = ( 1 − η t ) m t − 1 + η t g t . Gradient averaging has long been used to smooth stochastic estimates (Ruzaykin & Sly, 1983), and recent work studies its interaction with Polyak–Ruppert averaging in first-order methods [33 ] . Here, however, we use gradient averaging for quite a different purpose: to weaken the interaction between sampling noise and the randomness introduced by the sketching solver in Newton methods. We establish convergence and asymptotic normality of both the last and averaged sketched Newton iterates with gradient averaging. In particular, under weaker assumptions than Wang et al. [34 ] ; Du et al. [8 ] , we show that (i) the last-iterate covariance satisfies Σ m , a s ℓ ⪯ Σ g , a s ℓ \Sigma^\ell_{m,\mathrm{as}}\preceq\Sigma^\ell_{g,\mathrm{as}} Σ m , as ℓ ⪯ Σ g , as ℓ , where Σ m , a s ℓ \Sigma^\ell_{m,\mathrm{as}} Σ m , as ℓ and Σ g , a s ℓ \Sigma^\ell_{g,\mathrm{as}} Σ g , as ℓ denote the last-iterate covariances of accelerated sketched Newton using moving-average and single-sample gradients, respectively; and (ii) the averaged-iterate covariance satisfies Σ m , a s a = Ω \Sigma^a_{m,\mathrm{as}}=\Omega Σ m , as a = Ω . Thus, gradient averaging mitigates the computational–statistical trade-off for the last iterate, and even eliminates it for the averaged iterate. In this sense, gradient averaging provides a “free lunch” compared to existing sketched Newton methods: our covariance gains require only an O ( d ) O(d) O ( d ) vector update per iteration, without additional gradient, Hessian, or sketching evaluations, and hold under weaker assumptions.
The analysis requires addressing the following challenges introduced by gradient averaging.
(a) m t m_t m t averages stochastic gradients evaluated at different iterates and is therefore not conditionally unbiased for ∇ f ( x t ) \nabla f(x_t) ∇ f ( x t ) , unlike g t g_t g t , whose centered noise forms crucial martingale difference structure underlying central limit theorem arguments. More precisely, the tracking error e t : = m t − ∇ f ( x t ) e_t:=m_t-\nabla f(x_t) e t := m t − ∇ f ( x t ) contains inherited error from earlier steps, gradient drift, and martingale noise, and must be analyzed jointly with the optimization error Δ t : = x t − x ⋆ \Delta_t := x_t-x^\star Δ t := x t − x ⋆ . As such, we derive coupled recursions for ( e t , Δ t ) (e_t,\Delta_t) ( e t , Δ t ) and control their evolution on the two timescales governed by η t \eta_t η t and φ t \varphi_t φ t .
(b) To characterize how gradient averaging changes the leading sketching noise, we must separate sampling noise, sketching fluctuations, and variations in the Hessian approximation and sketching parameters across iterations. Since the resulting perturbation terms are sufficiently small only when ( Δ t , B t − B ⋆ , e t ) (\Delta_t,B_t-B^\star,e_t) ( Δ t , B t − B ⋆ , e t ) remain in a local neighborhood of zero, we introduce a stopping time at the first exit from this neighborhood and localize the convergence rate analysis (see (D.25) in the proof of Lemma 4.2). The effect of the localization is then removed by our almost-sure convergence guarantee.
(c) To isolate ∇ F ( x ⋆ ; ξ t ) \nabla F(x^\star;\xi_t) ∇ F ( x ⋆ ; ξ t ) as the leading stochastic noise in the asymptotic linear representations established in Theorem 4.3, we rewrite the outer Newton recursion in terms of the stochastic gradient g t g_t g t . This introduces weighted differences m t − m t − 1 m_t-m_{t-1} m t − m t − 1 that do not appear in the analysis of vanilla online Newton methods. We use summation by parts to transfer the differences from the moving-average gradients to their weights, and then bound the resulting boundary and weighted-sum terms to show that they are asymptotically negligible under both the last- and averaged-iterate normalizations.
Our main contributions are summarized as follows.
(a) We establish global almost-sure convergence of online sketched Newton with gradient averaging. Under standard two-timescale conditions, we show that the iterate x t x_t x t converges to x ⋆ x^\star x ⋆ and the moving-average gradient m t m_t m t tracks ∇ f ( x t ) \nabla f(x_t) ∇ f ( x t ) almost surely (Theorem 3.5). Thus, using gradients evaluated at earlier iterates does not prevent the outer iterates from converging. Moreover, gradient averaging requires no additional oracle evaluations and preserves the inner-solver contraction bound.
(b) We establish last-iterate asymptotic normality, 1 / φ t ⋅ ( x t − x ⋆ ) ⟶ d N ( 0 , Σ m , a s ℓ ) 1/\sqrt{\varphi_t}\cdot(x_t-x^\star)\mathrel{\mathop{\longrightarrow}^{d}} \mathcal{N}(0,\Sigma^\ell_{m,\mathrm{as}}) 1/ φ t ⋅ ( x t − x ⋆ ) ⟶ d N ( 0 , Σ m , as ℓ ) (Theorem 4.5), and prove Σ m , a s ℓ ⪯ Σ g , a s ℓ \Sigma^\ell_{m,\mathrm{as}}\preceq\Sigma^\ell_{g,\mathrm{as}} Σ m , as ℓ ⪯ Σ g , as ℓ (Prop. 4.6). Thus, gradient averaging yields no larger asymptotic variance in any direction than vanilla online sketched Newton. This covariance reduction (same for (c) below) uses the same inner sketching solver and the same number of sketching steps as existing methods using single-sample gradients. Our local convergence rates and asymptotic linear representations reveal the mechanism behind this variance reduction: the random matrix I − K ~ i ⋆ I-\widetilde{K}_{i}^\star I − K i ⋆ , which captures the limiting effect of the inner solver on gradient noise, is replaced by its mean I − K ⋆ I-K^\star I − K ⋆ (Theorem 4.3). Consequently, sketching enters the limiting covariance only through its mean effect.
(c) We further establish averaged-iterate asymptotic normality, t ( x ˉ t − x ⋆ ) ⟶ d N ( 0 , Σ m , a s a ) \sqrt{t}\,(\bar{x}_t-x^\star)\mathrel{\mathop{\longrightarrow}^{d}}\mathcal{N}(0,\Sigma^a_{m,\mathrm{as}}) t ( x ˉ t − x ⋆ ) ⟶ d N ( 0 , Σ m , as a ) , with Σ m , a s a = Ω ⪯ Σ g , a s a \Sigma^a_{m,\mathrm{as}}=\Omega\preceq\Sigma^a_{g,\mathrm{as}} Σ m , as a = Ω ⪯ Σ g , as a (Theorem 4.7). Polyak–Ruppert averaging eliminates the remaining effect of sketching and recovers the covariance Ω \Omega Ω of exact online Newton method, which is asymptotically minimax optimal ([9 ] ). Therefore, with a fixed number of sketching steps per iteration, our method gains computational efficiency without sacrificing statistical efficiency: its averaged iterate has the same limiting covariance as if each Newton system were solved exactly.
Notation. We use ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ to denote the Euclidean norm for vectors and the spectral norm for matrices. We write O ( ⋅ ) O(\cdot) O ( ⋅ ) and o ( ⋅ ) o(\cdot) o ( ⋅ ) (O p ( ⋅ ) O_p(\cdot) O p ( ⋅ ) and o p ( ⋅ ) o_p(\cdot) o p ( ⋅ ) ) for the usual big-O O O and small-o o o notation (in probability). We use I I I and 0 0 0 to denote identity matrices and zero vectors/matrices, with dimensions clear from context. For symmetric A A A and B B B , A ⪰ B A\succeq B A ⪰ B means A − B A-B A − B is positive semidefinite. We abbreviate ∇ f t = ∇ f ( x t ) \nabla f_t=\nabla f(x_t) ∇ f t = ∇ f ( x t ) and ∇ 2 f t = ∇ 2 f ( x t ) \nabla^2 f_t=\nabla^2 f(x_t) ∇ 2 f t = ∇ 2 f ( x t ) . We write Σ u , s q \Sigma^q_{u,s} Σ u , s q for limiting covariances, where u ∈ { g , m } u\in\{g,m\} u ∈ { g , m } denotes the stochastic gradient or its moving average; s ∈ { e , a s } s\in\{e,\mathrm{as}\} s ∈ { e , as } denotes the solver (exact or accelerated sketching); and q ∈ { ℓ , a } q\in\{\ell,a\} q ∈ { ℓ , a } denotes the iterate (last or averaged). In particular, we know from (1.3) that Ω = Σ g , e a \Omega=\Sigma^a_{g,e} Ω = Σ g , e a .
Preliminaries
We review online sketched Newton methods, which replace the exact Newton solve with an inexact randomized sketching solver ([24 , 20 ] ). These methods were subsequently improved by leveraging Nesterov’s acceleration ([34 , 8 ] ). In what follows, we first introduce the vanilla online Newton, then describe the accelerated sketching solver (of which the unaccelerated solver is a special case), and finally present the preliminary guarantees.
• Online Newton with a Hessian approximation. Online Newton methods precondition stochastic gradients using curvature information. Given x t x_t x t , we form a positive definite matrix B t B_t B t from past information to approximate ∇ 2 f t \nabla^2 f_t ∇ 2 f t . We then draw ξ t ∼ P \xi_t\sim\mathcal{P} ξ t ∼ P independently of the past and compute g t : = ∇ F ( x t ; ξ t ) g_t:=\nabla F(x_t;\xi_t) g t := ∇ F ( x t ; ξ t ) and H t : = ∇ 2 F ( x t ; ξ t ) H_t:=\nabla^2 F(x_t;\xi_t) H t := ∇ 2 F ( x t ; ξ t ) . A common starting point for constructing B t B_t B t is the Hessian average B ~ t : = 1 t ∑ i = 0 t − 1 H i \widetilde{B}_t := \frac{1}{t}\sum_{i=0}^{t-1} H_i B t := t 1 ∑ i = 0 t − 1 H i , which stabilizes the curvature estimate and improves local convergence behavior [25 ] . Section 4 uses a clipped version of this average to form B t B_t B t .
Given a prescribed stepsize sequence { φ t } t ≥ 0 \{\varphi_t\}_{t\geq0} { φ t } t ≥ 0 , the vanilla online Newton method updates as
x t + 1 = x t + φ t Δ x t , with B t Δ x t = − g t . (4) x_{t+1}=x_t+\varphi_t\Delta x_t,\qquad\text{with}\qquad B_t\Delta x_t=-g_t. \tag*{(4)} x t + 1 = x t + φ t Δ x t , with B t Δ x t = − g t . (4)
Solving this Newton system costs O ( d 3 ) O(d^3) O ( d 3 ) flops per step, motivating the sketching solver next.
Nesterov’s accelerated sketch-and-project. The inner solver for (4) combines randomized sketch-and-project updates with Nesterov’s acceleration, yielding faster decay rate of the approximation error at the same order of per-step cost [14 , 6 ] . We suppress the outer index t t t and present the accelerated solver NASketch( B , − g ; α , β , γ , τ ) (B,-g;\alpha,\beta,\gamma,\tau) ( B , − g ; α , β , γ , τ ) in Algorithm 1, where ( α , β , γ ) (\alpha,\beta,\gamma) ( α , β , γ ) are the acceleration parameters and τ \tau τ is the number of sketching steps.
At each outer iteration, ( B , g ) (B,g) ( B , g ) are held fixed, and we draw S j ∼ i i d S S_j\mathrel{\overset{\mathrm{iid}}{\sim}} S S j ∼ iid S for 0 ≤ j ≤ τ − 1 0\leq j\leq\tau-1 0 ≤ j ≤ τ − 1 . Without acceleration, the sketch-and-project solver performs the update z j + 1 = z j − B S j ( S j T B 2 S j ) † S j T ( B z j + g ) z_{j+1}=z_j-BS_j(S_j^{\mathsf T}B^2S_j)^\dagger S_j^{\mathsf T}(Bz_j+g) z j + 1 = z j − B S j ( S j T B 2 S j ) † S j T ( B z j + g ) , where † \dagger † denotes the pseudoinverse. Nesterov’s acceleration introduces an auxiliary momentum sequence v j v_j v j . Given ( z j , v j ) (z_j,v_j) ( z j , v j ) , Algorithm 1 first forms the middle point y j = α v j + ( 1 − α ) z j y_j=\alpha v_j+(1-\alpha)z_j y j = α v j + ( 1 − α ) z j , computes a sketching correction at y j y_j y j , and updates ( z j , v j ) (z_j,v_j) ( z j , v j ) jointly. By Algorithm 1, the setup ( α , β , γ ) = ( 0.5 , 0 , 1 ) (\alpha,\beta,\gamma)=(0.5,0,1) ( α , β , γ ) = ( 0.5 , 0 , 1 ) leads to v j = z j v_j=z_j v j = z j and reduces the method to unaccelerated sketch-and-project.
Algorithm 1 Sketch-and-Project with Nesterov’s Acceleration: NASketch( B , − g ; α , β , γ , τ ) (B,-g;\alpha,\beta,\gamma,\tau) ( B , − g ; α , β , γ , τ )
Initialize: z 0 = v 0 = 0 z_0=v_0=0 z 0 = v 0 = 0 , α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , β ∈ [ 0 , 1 ) \beta\in[0,1) β ∈ [ 0 , 1 ) , γ ≥ 1 \gamma\geq1 γ ≥ 1 , τ ≥ 1 \tau\geq1 τ ≥ 1 .
for j = 0 , 1 , … , τ − 1 j=0,1,\ldots,\tau-1 j = 0 , 1 , … , τ − 1 do
y j ← α v j + ( 1 − α ) z j y_j\leftarrow\alpha v_j+(1-\alpha)z_j y j ← α v j + ( 1 − α ) z j and draw a sketching matrix S j ∼ S S_j\sim S S j ∼ S , where S j ∈ R d × s S_j\in\mathbb{R}^{d\times s} S j ∈ R d × s , 1 ≤ s ≤ d 1\leq s\leq d 1 ≤ s ≤ d .
w j ← B S j ( S j T B 2 S j ) † S j T ( B y j + g ) w_j\leftarrow BS_j(S_j^{\mathsf T}B^2S_j)^\dagger S_j^{\mathsf T}(By_j+g) w j ← B S j ( S j T B 2 S j ) † S j T ( B y j + g ) .
( z j + 1 , v j + 1 ) ← ( y j − w j , β v j + ( 1 − β ) y j − γ w j ) (z_{j+1},v_{j+1})\leftarrow(y_j-w_j,\beta v_j+(1-\beta)y_j-\gamma w_j) ( z j + 1 , v j + 1 ) ← ( y j − w j , β v j + ( 1 − β ) y j − γ w j ) .
end for
Output: z τ z_\tau z τ .
Following Gower et al. [14 ] ; ); Derezinski et al. [6 ] , we parameterize ( α , β , γ ) (\alpha,\beta,\gamma) ( α , β , γ ) by ( μ , ν ) (\mu,\nu) ( μ , ν ) as
α = 1 1 + γ ν , β = 1 − μ ν , γ = 1 μ ν . (5) \alpha=\frac{1}{1+\gamma\nu},\qquad\beta=1-\sqrt{\frac{\mu}{\nu}},\qquad\gamma=\frac{1}{\sqrt{\mu\nu}}. \tag*{(5)} α = 1 + γ ν 1 , β = 1 − ν μ , γ = μν 1 . (5)
For a generic sketching matrix S S S , define Z ~ : = B S ( S T B 2 S ) † S T B \widetilde{Z}:=BS(S^{\mathsf T}B^2S)^\dagger S^{\mathsf T}B Z := B S ( S T B 2 S ) † S T B and Z : = E [ Z ~ ] Z:=\mathbb{E}[\widetilde{Z}] Z := E [ Z ] , where the expectations below are taken over S S S . When Z ≻ 0 Z\succ0 Z ≻ 0 , define
μ : = inf a ≠ 0 a T Z a a T a , ν : = sup a ≠ 0 a T E [ Z ~ Z ~ − 1 Z ~ ] a a T Z a . (6) \mu:=\inf_{a\ne0}\frac{a^{\mathsf T}Za}{a^{\mathsf T}a},\qquad
\nu:=\sup_{a\ne0}\frac{a^{\mathsf T}\mathbb{E}[\widetilde{Z}\widetilde{Z}^{-1}\widetilde{Z}]a}{a^{\mathsf T}Za}. \tag*{(6)} μ := a = 0 inf a T a a T Z a , ν := a = 0 sup a T Z a a T E [ Z Z − 1 Z ] a . (6)
At outer iteration t t t , evaluating (6) at B t B_t B t gives ( μ t , ν t ) (\mu_t,\nu_t) ( μ t , ν t ) and hence ( α t , β t , γ t ) (\alpha_t,\beta_t,\gamma_t) ( α t , β t , γ t ) . Since the exact computation of these parameters requires expectations over the sketching distribution, (6) primarily serves as a theoretical parameterization for analyzing the solver’s convergence; implementations may approximate these quantities using periodically updated estimates or fixed parameter choices [34 ] . With specified parameters, the online sketched Newton method replaces the exact Newton direction Δ x t \Delta x_t Δ x t in (4) with z t , τ z_{t,\tau} z t , τ :
z t , τ = NASketch ( B t , − g t ; α t , β t , γ t , τ ) , x t + 1 = x t + φ t z t , τ . (7) z_{t,\tau}=\operatorname{NASketch}(B_t,-g_t;\alpha_t,\beta_t,\gamma_t,\tau),\qquad x_{t+1}=x_t+\varphi_tz_{t,\tau}. \tag*{(7)} z t , τ = NASketch ( B t , − g t ; α t , β t , γ t , τ ) , x t + 1 = x t + φ t z t , τ . (7)
Theory of accelerated sketched Newton. We next review the inner-solver contraction bound and existing asymptotic normality results for online sketched Newton methods. At any fixed outer iteration t t t , each inner update of the solution approximation in Algorithm 1 is linear in g t g_t g t . More specifically, by Wang et al. [34 ] , there exists a realized coefficient factor matrix K ~ t \widetilde{K}_t K t for the solution approximation error, independent of g t g_t g t , such that (see (C.2) for the precise form of K ~ t \widetilde{K}_t K t )
z t , τ − Δ x t = K ~ t ( z t , 0 − Δ x t ) , Δ x t = − B t − 1 g t . (8) z_{t,\tau}-\Delta x_t=\widetilde{K}_t(z_{t,0}-\Delta x_t),\qquad\Delta x_t=-B_t^{-1}g_t. \tag*{(8)} z t , τ − Δ x t = K t ( z t , 0 − Δ x t ) , Δ x t = − B t − 1 g t . (8)
With the above identity, the accelerated inner solver satisfies the following contraction bound.
Lemma 2.1 (Derezinski et al. [6 ] ). Suppose Z t = E [ Z ~ t ] = E [ B t S ( S T B t 2 S ) † S T B t ] ≻ 0 Z_t=\mathbb{E}[\widetilde{Z}_t]=\mathbb{E}[B_tS(S^{\mathsf T}B_t^2S)^\dagger S^{\mathsf T}B_t]\succ0 Z t = E [ Z t ] = E [ B t S ( S T B t 2 S ) † S T B t ] ≻ 0 , then
E [ ∥ z t , j − Δ x t ∥ 2 ] ≤ 2 ( 1 − μ t ν t ) j ∥ z t , 0 − Δ x t ∥ 2 , for any t ≥ 0 , j ≥ 0 , \mathbb{E}[\|z_{t,j}-\Delta x_t\|^2]\leq2\left(1-\sqrt{\frac{\mu_t}{\nu_t}}\right)^j\|z_{t,0}-\Delta x_t\|^2,\qquad\text{for any }t\geq0,\ j\geq0, E [ ∥ z t , j − Δ x t ∥ 2 ] ≤ 2 ( 1 − ν t μ t ) j ∥ z t , 0 − Δ x t ∥ 2 , for any t ≥ 0 , j ≥ 0 , where the expectation is taken over the inner sketches { S j } j \{S_j\}_j { S j } j .
By Gower et al. [14 ] , 1 ≤ ν t ≤ 1 / μ t 1 \le\nu_t \le1/\mu_t 1 ≤ ν t ≤ 1/ μ t ; thus, 1 − μ t / ν t ≤ 1 − μ t 1-\sqrt{\mu_t/\nu_t} \le1-\mu_t 1 − μ t / ν t ≤ 1 − μ t , where the right-hand side is the contraction rate of vanilla unaccelerated sketch-and-project (Gower & Richtárik, [15 ] ).
To introduce asymptotic normality, we need additional notation. Let B ⋆ : = ∇ 2 f ( x ⋆ ) B^\star:= \nabla^2 f(x^\star) B ⋆ := ∇ 2 f ( x ⋆ ) be the Hessian at x ⋆ x^\star x ⋆ and ( μ ⋆ , ν ⋆ , α ⋆ , β ⋆ , γ ⋆ ) (\mu^\star,\nu^\star,\alpha^\star,\beta^\star,\gamma^\star) ( μ ⋆ , ν ⋆ , α ⋆ , β ⋆ , γ ⋆ ) be parameters obtained from (5) and (6) with B = B ⋆ B=B^\star B = B ⋆ . We also define K ~ ⋆ \widetilde{K}^\star K ⋆ by evaluating (8) at these limiting quantities (cf. (C.2)), K ⋆ : = E [ K ~ ⋆ ] K^\star:= \mathbb{E}[\widetilde{K}^\star] K ⋆ := E [ K ⋆ ] , and two matrices
Ω = ( B ⋆ ) − 1 E [ ∇ F ( x ⋆ ; ξ ) ∇ F ( x ⋆ ; ξ ) ⊤ ] ( B ⋆ ) − 1 and Γ g , a s : = E [ ( I − K ~ ⋆ ) Ω ( I − K ~ ⋆ ) ⊤ ] . (9) \Omega= (B^\star)^{-1}\mathbb{E}[\nabla F(x^\star;\xi)\nabla F(x^\star;\xi)^\top](B^\star)^{-1}
\quad\text{and}\quad
\Gamma_{g,\mathrm{as}} := \mathbb{E}[(I-\widetilde{K}^\star)\Omega(I-\widetilde{K}^\star)^\top]. \tag*{(9)} Ω = ( B ⋆ ) − 1 E [ ∇ F ( x ⋆ ; ξ ) ∇ F ( x ⋆ ; ξ ) ⊤ ] ( B ⋆ ) − 1 and Γ g , as := E [( I − K ⋆ ) Ω ( I − K ⋆ ) ⊤ ] . (9)
Let φ t : = C φ / ( t + 1 ) φ \varphi_t := C_\varphi/(t+1)^\varphi φ t := C φ / ( t + 1 ) φ , ζ φ : = 1 { φ = 1 } / ( 2 C φ ) \zeta_\varphi:= \mathbb{1}_{\{\varphi=1\}}/(2C_\varphi) ζ φ := 1 { φ = 1 } / ( 2 C φ ) , and A φ ⋆ : = ( I − K ⋆ ) − ζ φ I A_\varphi^\star:= (I-K^\star)-\zeta_\varphi I A φ ⋆ := ( I − K ⋆ ) − ζ φ I . The following theorem extends the existing normality results for both the last and the averaged iterates.
Theorem 2.2 (Wang et al. [34 ] ; Du et al. [8 ] ). Under the assumptions in Theorem 4.3 , we have
1 / φ t ( x t − x ⋆ ) → d N ( 0 , Σ g , a s ℓ ) , where ( A φ ⋆ ) ⊤ Σ g , a s ℓ + Σ g , a s ℓ A φ ⋆ = Γ g , a s . (10) 1/\sqrt{\varphi_t}\,(x_t-x^\star) \xrightarrow{d} \mathcal{N}(0,\Sigma^\ell_{g,\mathrm{as}}), \qquad\text{where}\qquad(A_\varphi^\star)^\top\Sigma^\ell_{g,\mathrm{as}}+\Sigma^\ell_{g,\mathrm{as}}A_\varphi^\star=\Gamma_{g,\mathrm{as}}. \tag*{(10)} 1/ φ t ( x t − x ⋆ ) d N ( 0 , Σ g , as ℓ ) , where ( A φ ⋆ ) ⊤ Σ g , as ℓ + Σ g , as ℓ A φ ⋆ = Γ g , as . (10) Moreover, if φ ∈ ( 0.5 , 1 ) \varphi\in(0.5,1) φ ∈ ( 0.5 , 1 ) , then the averaged iterate x ˉ t = ∑ i = 0 t − 1 x i / t \bar{x}_t=\sum_{i=0}^{t-1}x_i/t x ˉ t = ∑ i = 0 t − 1 x i / t satisfies
t ( x ˉ t − x ⋆ ) → d N ( 0 , Σ g , a s a ) , where Σ g , a s a : = ( I − K ⋆ ) − 1 Γ g , a s ( I − K ⋆ ) − 1 . (11) \sqrt{t}(\bar{x}_t-x^\star)\xrightarrow{d}\mathcal{N}(0,\Sigma^a_{g,\mathrm{as}}), \qquad\text{where}\qquad\Sigma^a_{g,\mathrm{as}}:=(I-K^\star)^{-1}\Gamma_{g,\mathrm{as}}(I-K^\star)^{-1}. \tag*{(11)} t ( x ˉ t − x ⋆ ) d N ( 0 , Σ g , as a ) , where Σ g , as a := ( I − K ⋆ ) − 1 Γ g , as ( I − K ⋆ ) − 1 . (11)
We clarify that the results of Wang et al. [34 ] ; Du et al. [8 ] were established under uniform bounds on sampled Hessian ∇ 2 F ( ⋅ ; ξ ) \nabla^2F(\cdot;\xi) ∇ 2 F ( ⋅ ; ξ ) . For clipped B t B_t B t mentioned in § 4 , our global convergence and local rate arguments extend to their update that uses g t g_t g t and provide the bounds needed in their proofs. Their asymptotic normality arguments then yield (10) and (11) under the conditions stated here.
For the last iterate, setting K ~ ⋆ = K ⋆ = 0 \widetilde{K}^\star=K^\star=0 K ⋆ = K ⋆ = 0 in (10) recovers the asymptotic covariance of exact online Newton (3) . In general, however, when φ = C φ = 1 \varphi=C_\varphi=1 φ = C φ = 1 , it has been shown that Σ g , a s ℓ ⪰ Ω \Sigma^\ell_{g,\mathrm{as}}\succeq\Omega Σ g , as ℓ ⪰ Ω . Since Ω \Omega Ω is the classical Polyak–Ruppert covariance and the local asymptotic minimax bound (Duchi & Ruan, [9 ] ), this ordering reveals a computational–statistical trade-off: randomized sketching reduces the cost of the Newton solve but inflates the limiting covariance.
Analogously, the averaged-iterate result (11) also recovers exact online Newton (3) . However, it has also been shown that Σ g , a s ℓ ⪰ Σ g , a s a ⪰ Ω \Sigma^\ell_{g,\mathrm{as}}\succeq\Sigma^a_{g,\mathrm{as}}\succeq\Omega Σ g , as ℓ ⪰ Σ g , as a ⪰ Ω . Thus, while iterate averaging reduces limiting covariance relative to the last iterate, it does not fully eliminate the sketch-induced covariance inflation, and the trade-off persists. In this paper, we completely eliminate the trade-off by gradient averaging.
Global Convergence with Gradient Averaging
In this section, we incorporate a moving-average gradient into accelerated sketched Newton method, introduce assumptions, and establish almost-sure convergence of the method. Specifically, instead of using g t g_t g t directly, we initialize m 0 = g 0 m_0=g_0 m 0 = g 0 and update
m t = ( 1 − η t ) m t − 1 + η t g t , η t ∈ ( 0 , 1 ] , t ≥ 1 , (12) m_t=(1-\eta_t)m_{t-1}+\eta_t g_t,\qquad\eta_t\in(0,1],\quad t\ge1, \tag*{(12)} m t = ( 1 − η t ) m t − 1 + η t g t , η t ∈ ( 0 , 1 ] , t ≥ 1 , (12)
where { η t } t ≥ 1 \{\eta_t\}_{t\ge1} { η t } t ≥ 1 is a prescribed averaging sequence. We then replace g t g_t g t in (7) with m t m_t m t and obtain
z t , τ = NASKetch ( B t , − m t ; α t , β t , γ t ) , x t + 1 = x t + φ t z t , τ . (13) z_{t,\tau}=\operatorname{NASKetch}(B_t,-m_t;\alpha_t,\beta_t,\gamma_t),\qquad x_{t+1}=x_t+\varphi_tz_{t,\tau}. \tag*{(13)} z t , τ = NASKetch ( B t , − m t ; α t , β t , γ t ) , x t + 1 = x t + φ t z t , τ . (13)
This replacement changes only the right-hand side of (4) ; hence, B t B_t B t , ( α t , β t , γ t ) (\alpha_t,\beta_t,\gamma_t) ( α t , β t , γ t ) , and the inner sketching procedure remain unchanged, as does the contraction bound in Lemma 2.1 . Moreover, updating m t m_t m t requires no additional oracle evaluations and incurs only O ( d ) O(d) O ( d ) computation, preserving the per-step cost order. Since m t m_t m t averages gradients evaluated at different iterates, it is not conditionally unbiased for ∇ f t \nabla f_t ∇ f t and destroys the crucial martingale difference structure. Thus, establishing global convergence requires joint control of the optimization error Δ t = x t − x ⋆ \Delta_t=x_t-x^\star Δ t = x t − x ⋆ and the gradient-tracking error e t = m t − ∇ f t e_t=m_t-\nabla f_t e t = m t − ∇ f t . We first specify the filtration and assumptions used here and in § 4 .
For t ≥ 0 t\ge0 t ≥ 0 , we define σ \sigma σ -algebras F t − 0.5 = σ ( { ξ i , { S i , j } j = 1 t − 1 } ∪ ξ t ) \mathcal{F}_{t-0.5}=\sigma(\{\xi_i,\{S_{i,j}\}_{j=1}^{t-1}\}\cup\xi_t) F t − 0.5 = σ ({ ξ i , { S i , j } j = 1 t − 1 } ∪ ξ t ) and F t = σ ( { ξ i , { S i , j } j = 0 t ) \mathcal{F}_t=\sigma(\{\xi_i,\{S_{i,j}\}_{j=0}^{t}) F t = σ ({ ξ i , { S i , j } j = 0 t ) . For consistency, let ( x 0 , B 0 ) (x_0,B_0) ( x 0 , B 0 ) be deterministic and F − 1 : = σ ( x 0 , B 0 ) \mathcal{F}_{-1}:=\sigma(x_0,B_0) F − 1 := σ ( x 0 , B 0 ) be the trivial σ \sigma σ -algebra. Then, we know ( g t , H t , m t ) (g_t,H_t,m_t) ( g t , H t , m t ) and ( z t , τ , x t + 1 , B t + 1 ) (z_{t,\tau},x_{t+1},B_{t+1}) ( z t , τ , x t + 1 , B t + 1 ) are F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 - and F t \mathcal{F}_t F t -measurable, respectively.
The first assumption regards the strong convexity and Lipschitz smoothness of the objective.
Assumption 3.1. We assume f f f and its realization F ( ⋅ ; ξ ) F(\cdot;\xi) F ( ⋅ ; ξ ) are twice continuously differentiable with γ H I ⪯ ∇ 2 f ( x ) ⪯ γ H I \gamma_H I\preceq\nabla^2f(x)\preceq\gamma_H I γ H I ⪯ ∇ 2 f ( x ) ⪯ γ H I for any x x x and some constants γ H , Υ H > 0 \gamma_H,\Upsilon_H>0 γ H , Υ H > 0 , and ∇ 2 f \nabla^2f ∇ 2 f is Υ L \Upsilon_L Υ L -Lipschitz continuous. We further assume there exist two positive sequences { γ B , t , Γ B , t } t ≥ 0 \{\gamma_{B,t},\Gamma_{B,t}\}_{t\ge0} { γ B , t , Γ B , t } t ≥ 0 such that B t B_t B t , γ B , t \gamma_{B,t} γ B , t and Γ B , t \Gamma_{B,t} Γ B , t are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable and the following hold almost surely: γ B , t I ⪯ B t ⪯ Γ B , t I \gamma_{B,t}I\preceq B_t\preceq\Gamma_{B,t}I γ B , t I ⪯ B t ⪯ Γ B , t I for all t ≥ 0 t\ge0 t ≥ 0 ; ∑ t = 0 ∞ φ t / γ B , t = ∞ \sum_{t=0}^{\infty}\varphi_t/\gamma_{B,t}=\infty ∑ t = 0 ∞ φ t / γ B , t = ∞ and ∑ t = 0 ∞ ( φ t / γ B , t ) 2 < ∞ \sum_{t=0}^{\infty}(\varphi_t/\gamma_{B,t})^2<\infty ∑ t = 0 ∞ ( φ t / γ B , t ) 2 < ∞ ; and lim sup t → ∞ Γ B , t / γ B , t < ∞ \limsup_{t\to\infty}\Gamma_{B,t}/\gamma_{B,t}<\infty lim sup t → ∞ Γ B , t / γ B , t < ∞ .
We next impose growth conditions on the moments of the gradient and Hessian noises.
Assumption 3.2. For any t ≥ 0 t \ge0 t ≥ 0 , we assume the gradient estimate is unbiased E [ g t ∣ F t − 1 ] = ∇ f t \mathbb{E}[g_t \mid\mathcal{F}_{t-1}] = \nabla f_t E [ g t ∣ F t − 1 ] = ∇ f t and satisfies a growth condition on its q g q_g q g -th moment: for some constants C g , 1 , C g , 2 > 0 C_{g,1}, C_{g,2} > 0 C g , 1 , C g , 2 > 0 ,
E [ ∥ g t − ∇ f t ∥ q g ∣ F t − 1 ] ≤ C g , 1 ∥ x t − x ∗ ∥ q g + C g , 2 . \mathbb{E}[\lVert g_t-\nabla f_t\rVert^{q_g}\mid\mathcal{F}_{t-1}]\le C_{g,1}\lVert x_t-x^\ast\rVert^{q_g}+C_{g,2}. E [∥ g t − ∇ f t ∥ q g ∣ F t − 1 ] ≤ C g , 1 ∥ x t − x ∗ ∥ q g + C g , 2 .
Assumption 3.3. For any t ≥ 0 t \ge0 t ≥ 0 , we assume the Hessian estimate is unbiased E [ H t ∣ F t − 1 ] = ∇ 2 f t \mathbb{E}[H_t \mid\mathcal{F}_{t-1}] = \nabla^2 f_t E [ H t ∣ F t − 1 ] = ∇ 2 f t and satisfies a growth condition on its second moment: for some constants C H , 1 , C H , 2 > 0 C_{H,1}, C_{H,2} > 0 C H , 1 , C H , 2 > 0 ,
E [ ∥ H t − ∇ 2 f t ∥ 2 ∣ F t − 1 ] ≤ C H , 1 ∥ x t − x ∗ ∥ 2 + C H , 2 . \mathbb{E}[\lVert H_t-\nabla^2 f_t\rVert^2\mid\mathcal{F}_{t-1}]\le C_{H,1}\lVert x_t-x^\ast\rVert^2+C_{H,2}. E [∥ H t − ∇ 2 f t ∥ 2 ∣ F t − 1 ] ≤ C H , 1 ∥ x t − x ∗ ∥ 2 + C H , 2 .
We will specify q g q_g q g in the theorem statements later. In particular, q g = 2 q_g=2 q g = 2 suffices for global convergence and q g > 2 q_g>2 q g > 2 for asymptotic normality; Assumption 3.3 will be used only for asymptotic normality in § 4 . Finally, we present an assumption on the sketching distribution.
Assumption 3.4. For any t ≥ 0 t \ge0 t ≥ 0 , we assume there exists γ S > 0 \gamma_S>0 γ S > 0 such that { S t , j } j = 0 τ − 1 ∼ i i d S \{S_{t,j}\}_{j=0}^{\tau-1}\overset{\mathrm{iid}}{\sim} S { S t , j } j = 0 τ − 1 ∼ iid S satisfying
Z t : = E [ B t S ( S ⊤ B t S ) † S ⊤ B t ∣ F t − 1 ] ⪰ γ S I , Z_t:=\mathbb{E}[B_tS(S^\top B_tS)^\dagger S^\top B_t\mid\mathcal{F}_{t-1}]\succeq\gamma_S I, Z t := E [ B t S ( S ⊤ B t S ) † S ⊤ B t ∣ F t − 1 ] ⪰ γ S I , and { S t , j } j = 0 τ − 1 \{S_{t,j}\}_{j=0}^{\tau-1} { S t , j } j = 0 τ − 1 are independent of ξ t \xi_t ξ t . Moreover, the condition number of S S S has bounded expectation: E [ ∥ S ∥ ∥ S † ∥ ] ≤ Υ S \mathbb{E}[\lVert S\rVert\lVert S^\dagger\rVert]\le\Upsilon_S E [∥ S ∥ ∥ S † ∥] ≤ Υ S for Υ S > 0 \Upsilon_S>0 Υ S > 0 .
Assumptions 3.1–3.4 are taken (in fact, weaken) from existing literature on sketched Newton inference ([20 , 34 , 8 ] ), and the population strong convexity and smoothness conditions are also used in SGD inference ([4 , 36 ] ). In particular, Assumption 3.1 has two parts. (i) The strong convexity and smoothness are imposed only on the population objective f f f , and no spectral bounds are required on sampled Hessians ∇ 2 F \nabla^2F ∇ 2 F . We note that global lower bound on ∇ 2 f \nabla^2f ∇ 2 f is used only to establish x t → a . s . x ∗ x_t\xrightarrow{\mathrm{a.s.}}x^\ast x t a.s. x ∗ in Theorem 3.5 , while inference is local in nature: the curvature entering Ω \Omega Ω in (9) is the Hessian at the solution, B ∗ = ∇ 2 f ( x ∗ ) B^\ast=\nabla^2f(x^\ast) B ∗ = ∇ 2 f ( x ∗ ) . (ii) The conditions on the Hessian approximation B t B_t B t are taken directly from conditioned SGD (Leluc & Portier, [21 ] , Assumption 9). It is strictly weaker than its counterparts in sketched Newton inference ([20 , 34 , 8 ] ), which impose uniform bounds on ∇ 2 F \nabla^2F ∇ 2 F . In linear regression with covariates a ∼ N ( 0 , Σ a ) a\sim\mathcal{N}(0,\Sigma_a) a ∼ N ( 0 , Σ a ) , the sample Hessian a a ⊤ aa^\top a a ⊤ is rank one and unbounded, so uniform bounds fail while ∇ 2 f ( x ) = Σ a \nabla^2f(x)=\Sigma_a ∇ 2 f ( x ) = Σ a satisfies Assumption 3.1. Assumptions 3.2–3.4 follow sketched Newton literature, and the gradient condition in Assumption 3.2 is also standard in SGD inference. Appendix § 8 further discusses Hessian conditions and their relation to conditioned SGD.
For regression problems, the gradient moment conditions can be verified by the arguments in Chen et al. [4 ] , Appendix A). The lower bound on expected projection matrix Z t Z_t Z t is required to ensure convergence of sketching solvers ([14 ] ). These conditions hold for various sketching distributions, including Gaussian and Kaczmarz sketching ([32 ] ).
Under the assumptions shown above, the following theorem establishes joint global almost-sure convergence of the optimization error Δ t \Delta_t Δ t and tracking error e t e_t e t .
Theorem 3.5. Consider the Newton scheme (13) with φ t = C φ / ( t + 1 ) φ \varphi_t=C_\varphi/(t+1)^\varphi φ t = C φ / ( t + 1 ) φ , η t = C η / ( t + 1 ) η \eta_t=C_\eta/(t+1)^\eta η t = C η / ( t + 1 ) η satisfying C φ > 0 C_\varphi>0 C φ > 0 , C η ∈ ( 0 , 1 ] C_\eta\in(0,1] C η ∈ ( 0 , 1 ] and η > 0.5 \eta>0.5 η > 0.5 . Suppose Assumptions 3.1, 3.2, and 3.4 hold with q g = 2 q_g=2 q g = 2 , φ t / ( γ B , t η t ) → 0 \varphi_t/(\gamma_{B,t}\eta_t)\to0 φ t / ( γ B , t η t ) → 0 almost surely, and τ \tau τ satisfies τ ( 1 − μ t / ν t ) τ − 2 ≤ γ B , t / ( 8 Γ B , t ) \tau(1-\sqrt{\mu_t/\nu_t})\tau^{-2}\le\gamma_{B,t}/(8\Gamma_{B,t}) τ ( 1 − μ t / ν t ) τ − 2 ≤ γ B , t / ( 8 Γ B , t ) for all t ≥ 0 t\ge0 t ≥ 0 . Then, we have e t = m t − ∇ f t → a . s . 0 e_t=m_t-\nabla f_t\xrightarrow{\mathrm{a.s.}}0 e t = m t − ∇ f t a.s. 0 and x t → a . s . x ∗ x_t\xrightarrow{\mathrm{a.s.}}x^\ast x t a.s. x ∗ as t → ∞ t\to\infty t → ∞ .
Theorem 3.5 requires only the bounded variance condition on the stochastic gradient, which is standard in stochastic optimization ([13 ] ). The condition φ t / ( γ B , t η t ) → 0 \varphi_t/(\gamma_{B,t}\eta_t)\to0 φ t / ( γ B , t η t ) → 0 ensures that m t m_t m t adopts an averaging sequence η t \eta_t η t that asymptotically dominates the effective outer Newton stepsize φ t / γ B , t \varphi_t/\gamma_{B,t} φ t / γ B , t , so as to preserve as much gradient information from g t g_t g t as possible despite the continued evolution of x t x_t x t . Moreover, Lemma 2.1 and the condition on τ \tau τ ensure that the approximate Newton direction is sufficiently accurate for global convergence. By Wang et al. [34 ] , 1 − μ t / ν t ≤ 1 − γ S < 1 1-\sqrt{\mu_t/\nu_t}\le1-\gamma_S<1 1 − μ t / ν t ≤ 1 − γ S < 1 . Assumption 3.1 gives sup t ≥ 0 Γ B , t / γ B , t < ∞ \sup_{t\ge0}\Gamma_{B,t}/\gamma_{B,t}<\infty sup t ≥ 0 Γ B , t / γ B , t < ∞ almost surely. Hence, along almost every realization, the sketching steps needed to meet the condition on τ \tau τ are uniformly bounded in t t t ; the required inner accuracy doesn’t increase with t t t . See Appendix § 8 for more on τ \tau τ . Overall, Theorem 3.5 suggests that though m t m_t m t is not conditionally unbiased for ∇ f t \nabla f_t ∇ f t , it tracks ∇ f t \nabla f_t ∇ f t almost surely and x t x_t x t also converges to x ∗ x^\ast x ∗ . This global result underlies the local analysis in § 4 .
Optimal Statistical Efficiency via Gradient Averaging
We now study how moving-average gradient estimators in (12) change the statistical effect of sketching. We first introduce a spectrally clipped Hessian average B t B_t B t and then establish local asymptotic rates and show that the centered sketching fluctuation is asymptotically negligible with gradient averaging. We then derive asymptotic normality of the last and averaged Newton iterates. The resulting limits reveal a sharp distinction (Theorem 4.3 ): gradient averaging reduces the random sketching factor in the leading stochastic noise to its mean, thereby reducing the sketching-induced fluctuation of the last iterate, while averaging the outer iterates further cancels the remaining mean factor and completely eliminates the inflation in asymptotic uncertainty.
Throughout this section, we consider the following B t B_t B t based on Hessian averaging. For t ≥ 1 t \ge1 t ≥ 1 , let
B ~ t : = t − 1 t B ~ t − 1 + 1 t H t − 1 = 1 t ∑ i = 0 t − 1 H i , B t : = Clip ( B ~ t ; γ B , t , Υ B , t ) . (14) \widetilde{B}_t := \frac{t-1}{t}\widetilde{B}_{t-1}+\frac{1}{t}H_{t-1}=\frac{1}{t}\sum_{i=0}^{t-1}H_i,\qquad B_t:=\operatorname{Clip}(\widetilde{B}_t;\gamma_{B,t},\Upsilon_{B,t}). \tag*{(14)} B t := t t − 1 B t − 1 + t 1 H t − 1 = t 1 i = 0 ∑ t − 1 H i , B t := Clip ( B t ; γ B , t , Υ B , t ) . (14)
For a symmetric matrix A = U diag ( λ 1 , … , λ d ) U ⊤ A=U\operatorname{diag}(\lambda_1,\ldots,\lambda_d)U^\top A = U diag ( λ 1 , … , λ d ) U ⊤ and 0 < γ ≤ Υ 0<\gamma\le\Upsilon 0 < γ ≤ Υ , we define Clip ( A ; γ , Υ ) : = U diag ( { min { Υ , max { γ , λ j } } } j = 1 d ) U ⊤ \operatorname{Clip}(A;\gamma,\Upsilon):=U\operatorname{diag}\left(\{\min\{\Upsilon,\max\{\gamma,\lambda_j\}\}\}_{j=1}^d\right)U^\top Clip ( A ; γ , Υ ) := U diag ( { min { Υ , max { γ , λ j }} } j = 1 d ) U ⊤ . In practice, we refresh the clipping once every d d d iterations and keep B t B_t B t , γ B , t \gamma_{B,t} γ B , t , and Υ B , t \Upsilon_{B,t} Υ B , t fixed between refreshes. A spectral clipping costs O ( d 3 ) O(d^3) O ( d 3 ) , while updating B ~ t \widetilde{B}_t B t costs O ( d 2 ) O(d^2) O ( d 2 ) per iteration. Thus, the amortized matrix-update cost is O ( d 2 ) O(d^2) O ( d 2 ) per iteration. The clipping in (14) enforces the spectral bounds on B t B_t B t used to control the search direction in the global convergence analysis. To identify the limiting covariance matrices, we also need the clipped Hessian average to approach B ⋆ B^\star B ⋆ as x t → x ⋆ x_t\to x^\star x t → x ⋆ . We therefore assume that almost surely γ B , t ≤ λ min ( B ⋆ ) \gamma_{B,t}\le\lambda_{\min}(B^\star) γ B , t ≤ λ m i n ( B ⋆ ) and Υ B , t ≥ λ max ( B ⋆ ) \Upsilon_{B,t}\ge\lambda_{\max}(B^\star) Υ B , t ≥ λ m a x ( B ⋆ ) for all sufficiently large t t t .
The sketching cost depends on sketching distribution. For coordinate sketches S = e j S=e_j S = e j considered in Section 5, each update in Algorithm 1 accesses only one column of B t B_t B t and costs O ( d ) O(d) O ( d ) , so τ \tau τ updates cost O ( τ d ) O(\tau d) O ( τ d ) . Moreover, for these sketches, (6) yields μ t ≥ 1 / ( d κ t 2 ) \mu_t\ge1/(d\kappa_t^2) μ t ≥ 1/ ( d κ t 2 ) and ν t = d \nu_t=d ν t = d , where κ t : = λ max ( B t ) / λ min ( B t ) ≤ Υ B , t / γ B , t \kappa_t:=\lambda_{\max}(B_t)/\lambda_{\min}(B_t)\le\Upsilon_{B,t}/\gamma_{B,t} κ t := λ m a x ( B t ) / λ m i n ( B t ) ≤ Υ B , t / γ B , t is bounded uniformly in t t t (Gower et al., [14 ] ; Derezinski et al., [6 ] ; Wang et al., [34 ] ). Thus, the conditions on τ \tau τ in Theorems 3.5 and 4.3 are met by τ \tau τ of order d d d , yielding an O ( d 2 ) O(d^2) O ( d 2 ) sketching solver cost per outer iteration, compared with O ( d 3 ) O(d^3) O ( d 3 ) for an exact Newton solve. The constant hidden in this O ( d 2 ) O(d^2) O ( d 2 ) bound depends on the condition number, which is where sketching acceleration helps: it improves the worst-case dependence from κ t 2 \kappa_t^2 κ t 2 to κ t \kappa_t κ t . Including the O ( d ) O(d) O ( d ) moving average update, the total amortized cost per outer iteration remains O ( d 2 ) O(d^2) O ( d 2 ) in terms of the dimension. See Appendix A for details. We emphasize that the conditions on τ \tau τ are sufficient, worst-case conditions and can be conservative in practice. In our experiment, even τ ∈ { 5 , 10 } \tau\in\{5,10\} τ ∈ { 5 , 10 } suffices to achieve the predicted covariance reduction.
Local rates and sketching noise reduction
We begin by showing the almost-sure convergence of the clipped Hessian average B t B_t B t , the acceleration parameters ( α t , β t , γ t ) (\alpha_t,\beta_t,\gamma_t) ( α t , β t , γ t ) , and, more importantly, the evolution coefficient matrix of the sketching approximation error, K t : = E [ K ~ t ∣ F t − 1 ] K_t:=\mathbb{E}[\widetilde{K}_t\mid\mathcal{F}_{t-1}] K t := E [ K t ∣ F t − 1 ] , from (8) . Explicit form of K ~ t \widetilde{K}_t K t is given in (C.2) and not crucial here.
Lemma 4.1. Under the conditions of Theorem 3.5 and Assumption 3.3 , we have B t → a . s . B ⋆ B_t\xrightarrow{\mathrm{a.s.}}B^\star B t a.s. B ⋆ , ( α t , β t , γ t ) → a . s . ( α ⋆ , β ⋆ , γ ⋆ ) (\alpha_t,\beta_t,\gamma_t)\xrightarrow{\mathrm{a.s.}}(\alpha^\star,\beta^\star,\gamma^\star) ( α t , β t , γ t ) a.s. ( α ⋆ , β ⋆ , γ ⋆ ) , K t → a . s . K ⋆ K_t\xrightarrow{\mathrm{a.s.}}K^\star K t a.s. K ⋆ ; and K ⋆ K^\star K ⋆ is symmetric with ∥ K ⋆ ∥ ≤ 1 / 4 \lVert K^\star\rVert\le1/4 ∥ K ⋆ ∥ ≤ 1/4 .
With the limiting quantities in place, we establish the local rates needed for asymptotic normality.
Lemma 4.2. Suppose Assumptions 3.1–3.4 hold with a g ≥ 2 a_g\ge2 a g ≥ 2 ; τ \tau τ satisfies τ ( 1 − μ t / ν t ) − τ − 2 ≤ γ B , t / ( 8 Υ B , t ) \tau(1-\sqrt{\mu_t/\nu_t})^{-\tau-2}\le\gamma_{B,t}/(8\Upsilon_{B,t}) τ ( 1 − μ t / ν t ) − τ − 2 ≤ γ B , t / ( 8 Υ B , t ) for all t ≥ 0 t\ge0 t ≥ 0 ; and the stepsize parameters satisfy C η ∈ ( 0 , 1 ] C_\eta\in(0,1] C η ∈ ( 0 , 1 ] and 0.5 < η < φ ≤ 1 0.5<\eta<\varphi\le1 0.5 < η < φ ≤ 1 . If φ = 1 \varphi=1 φ = 1 , we further suppose C φ > 1 / { 2 λ min ( I − K ⋆ ) } C_\varphi>1/\{2\lambda_{\min}(I-K^\star)\} C φ > 1/ { 2 λ m i n ( I − K ⋆ )} . Then, we have
max { ∥ m t − ∇ f t ∥ , ∥ x t − x ⋆ ∥ , ∥ B t − B ⋆ ∥ , ∥ ( α t , β t , γ t ) − ( α ⋆ , β ⋆ , γ ⋆ ) ∥ , ∥ K t − K ⋆ ∥ } = O p ( η t ) . \max\left\{\lVert m_t-\nabla f_t\rVert,\lVert x_t-x^\star\rVert,\lVert B_t-B^\star\rVert,\lVert(\alpha_t,\beta_t,\gamma_t)-(\alpha^\star,\beta^\star,\gamma^\star)\rVert,\lVert K_t-K^\star\rVert\right\}=O_p(\sqrt{\eta_t}). max { ∥ m t − ∇ f t ∥ , ∥ x t − x ⋆ ∥ , ∥ B t − B ⋆ ∥ , ∥( α t , β t , γ t ) − ( α ⋆ , β ⋆ , γ ⋆ )∥ , ∥ K t − K ⋆ ∥ } = O p ( η t ) . The rates in Lemma 4.2 are obtained via a localization argument. Specifically, we introduce the first exit time in (D.25) to restrict the analysis to trajectories for which the iterate and gradient-tracking errors remain within a sufficiently small neighborhood of zero. This localization enables control of the resulting perturbation terms. The almost-sure convergence established in Theorem 3.5 and Lemma 4.1 ensures that the exit probability can be made arbitrarily small, thereby converting the localized bounds
into the stated O p ( η t ) O_p(\sqrt{\eta_t}) O p ( η t ) rates. By Lemma 4.2 , the centered sketching residual ( K ~ t − K t ) B t − 1 m t (\widetilde{K}_t-K_t)B_t^{-1}m_t ( K t − K t ) B t − 1 m t forms a martingale difference sequence whose conditional second moment is of order O p ( η t ) O_p(\eta_t) O p ( η t ) . Thus, its cumulative contribution is negligible under the iterate normalization.
The following theorem states the resulting asymptotic linear representations.
Theorem 4.3. Suppose Assumptions 3.1–3.4 hold with q g > 2 q_g>2 q g > 2 and τ \tau τ satisfies τ ( 1 − μ t / ν t ) τ − 2 ≤ γ B , t / ( 8 Υ B , t ) \tau(1-\sqrt{\mu_t/\nu_t})\tau^{-2}\leq\gamma_{B,t}/(8\Upsilon_{B,t}) τ ( 1 − μ t / ν t ) τ − 2 ≤ γ B , t / ( 8 Υ B , t ) for all t ≥ 0 t\geq0 t ≥ 0 . Let C ϕ > 0 C_\phi>0 C ϕ > 0 , C η ∈ ( 0 , 1 ] C_\eta\in(0,1] C η ∈ ( 0 , 1 ] , 0.5 < η < ϕ ≤ 1 0.5<\eta<\phi\leq1 0.5 < η < ϕ ≤ 1 , and C ϕ > 1 / { 2 λ min ( I − K ⋆ ) } C_\phi>1/\{2\lambda_{\min}(I-K^\star)\} C ϕ > 1/ { 2 λ m i n ( I − K ⋆ )} when ϕ = 1 \phi=1 ϕ = 1 . Then, the last-iterate admits the linear representation
1 ϕ t ( x t − x ⋆ ) = − 1 ϕ t ∑ i = 0 t − 1 Π t , i + 1 φ i ( I − K ⋆ ) ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) , (15) \frac{1}{\sqrt{\phi_t}}(x_t-x^\star)=-\frac{1}{\sqrt{\phi_t}}\sum_{i=0}^{t-1}\Pi_{t,i+1}\varphi_i(I-K^\star)(B^\star)^{-1}\nabla F(x^\star;\xi_i)+o_p(1), \tag*{(15)} ϕ t 1 ( x t − x ⋆ ) = − ϕ t 1 i = 0 ∑ t − 1 Π t , i + 1 φ i ( I − K ⋆ ) ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) , (15) where Π n , i : = ∏ j = i n − 1 { I − φ j ( I − K ⋆ ) } \Pi_{n,i}:=\prod_{j=i}^{n-1}\{I-\varphi_j(I-K^\star)\} Π n , i := ∏ j = i n − 1 { I − φ j ( I − K ⋆ )} for n > i n>i n > i with Π n , n : = I \Pi_{n,n}:=I Π n , n := I . Moreover, if φ < 1 \varphi<1 φ < 1 , we have
t ( x ˉ t − x ⋆ ) = 1 t ∑ i = 0 t − 1 ( x i − x ⋆ ) = − 1 t ∑ i = 0 t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) . (16) \sqrt{t}(\bar{x}_t-x^\star)=\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}(x_i-x^\star)=-\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)+o_p(1). \tag*{(16)} t ( x ˉ t − x ⋆ ) = t 1 i = 0 ∑ t − 1 ( x i − x ⋆ ) = − t 1 i = 0 ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) . (16)
Remark 4.4 (Interaction between gradient averaging and randomized sketching). The conditions of Theorem 4.3 relax the sampled Hessian bounds imposed for vanilla online sketched Newton methods (Wang et al., [34 ] ; Du et al., [8 ] ) (except that we have an additional averaging sequence η t \eta_t η t satisfying φ t = o ( η t ) \varphi_t=o(\eta_t) φ t = o ( η t ) ). To isolate the effect of gradient averaging, consider vanilla Newton sketch that uses g t g_t g t in place of m t m_t m t . For each t ≥ 0 t\geq0 t ≥ 0 , let K ~ t ⋆ \widetilde{K}^{\star}_t K t ⋆ denote the coefficient matrix in (8) constructed from the step-t t t sketches and evaluated at B ⋆ B^\star B ⋆ and ( α ⋆ , β ⋆ , γ ⋆ ) (\alpha^\star,\beta^\star,\gamma^\star) ( α ⋆ , β ⋆ , γ ⋆ ) . Building upon Wang et al. [34 ] , we can derive the following linear representation for vanilla Newton sketch:
1 ϕ t ( x t − x ⋆ ) = − 1 ϕ t ∑ i = 0 t − 1 Π t , i + 1 φ i ( I − K ~ i ⋆ ) ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) . (17) \frac{1}{\sqrt{\phi_t}}(x_t-x^\star)=-\frac{1}{\sqrt{\phi_t}}\sum_{i=0}^{t-1}\Pi_{t,i+1}\varphi_i(I-\widetilde{K}^{\star}_i)(B^\star)^{-1}\nabla F(x^\star;\xi_i)+o_p(1). \tag*{(17)} ϕ t 1 ( x t − x ⋆ ) = − ϕ t 1 i = 0 ∑ t − 1 Π t , i + 1 φ i ( I − K i ⋆ ) ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) . (17)
Comparing (15) and (17) reveals the effect of gradient averaging. For our method, the localized moment bounds in Lemma 4.2 imply that the centered sketching term ( K ~ t − K t ) B t − 1 m t (\widetilde{K}_t-K_t)B_t^{-1}m_t ( K t − K t ) B t − 1 m t has a second moment of order O ( η t ) O(\eta_t) O ( η t ) and therefore does not contribute to the leading stochastic sum. In contrast, for vanilla Newton sketch, the corresponding centered sketching term ( K ~ t − K t ) B t − 1 g t (\widetilde{K}_t-K_t)B_t^{-1}g_t ( K t − K t ) B t − 1 g t has a constant order of second moment, which inevitably inflates the leading stochastic sum.
The results in (15) –(17) reveal two successive effects. First, replacing the stochastic gradient with its moving average turns the sketching factor I − K ~ i ⋆ I-\widetilde{K}^{\star}_i I − K i ⋆ into its mean I − K ⋆ I-K^\star I − K ⋆ in the leading stochastic term of the last iterate. Second, averaging the Newton iterates further introduces the factor ( I − K ⋆ ) − 1 (I-K^\star)^{-1} ( I − K ⋆ ) − 1 , which cancels the remaining mean sketching factor. Consequently, (16) contains no sketch-dependent matrix and has the same leading stochastic term as exact online Newton. In contrast, the averaged iterate of vanilla sketched Newton still contains the centered sketching factor ( I − K ⋆ ) − 1 ( K ~ i ⋆ − K ⋆ ) (I-K^\star)^{-1}(\widetilde{K}^{\star}_i-K^\star) ( I − K ⋆ ) − 1 ( K i ⋆ − K ⋆ ) in its leading term, so the sketching randomness persists in its asymptotic distribution.
Asymptotic normality and covariance comparisons
In this section, we build upon Theorem 4.3 to establish asymptotic normality and to explore how gradient averaging and randomized sketching affect statistical efficiency. We begin with the last iterate.
Theorem 4.5. Under the conditions of Theorem 4.3, let Γ m , a s l = ( I − K ⋆ ) Ω ( I − K ⋆ ) \Gamma_{m,\mathrm{as}}^l=(I-K^\star)\Omega(I-K^\star) Γ m , as l = ( I − K ⋆ ) Ω ( I − K ⋆ ) and we have
1 ϕ t ( x t − x ⋆ ) → d N ( 0 , Σ m , a s l ) , where A φ ⋆ Σ m , a s l + Σ m , a s l A φ ⋆ = Γ m , a s . (18) \frac{1}{\sqrt{\phi_t}}(x_t-x^\star)\xrightarrow{d}\mathcal{N}(0,\Sigma_{m,\mathrm{as}}^l),\qquad\text{where}\qquad A_\varphi^\star\Sigma_{m,\mathrm{as}}^l+\Sigma_{m,\mathrm{as}}^l A_\varphi^\star=\Gamma_{m,\mathrm{as}}. \tag*{(18)} ϕ t 1 ( x t − x ⋆ ) d N ( 0 , Σ m , as l ) , where A φ ⋆ Σ m , as l + Σ m , as l A φ ⋆ = Γ m , as . (18)
Recall from Theorem 2.2 that A φ ⋆ = ( I − K ⋆ ) − ζ φ I A_\varphi^\star=(I-K^\star)-\zeta_\varphi I A φ ⋆ = ( I − K ⋆ ) − ζ φ I and ζ φ = 1 { φ = 1 } / ( 2 C φ ) \zeta_\varphi=1_{\{\varphi=1\}}/(2C_\varphi) ζ φ = 1 { φ = 1 } / ( 2 C φ ) . The result (18) characterizes Σ m , a s l \Sigma_{m,\mathrm{as}}^l Σ m , as l through a Lyapunov equation in which randomized sketching enters only through the mean factor matrix K ⋆ K^\star K ⋆ . When φ = 1 \varphi=1 φ = 1 , the condition C φ > 1 / { 2 λ min ( I − K ⋆ ) } C_\varphi>1/\{2\lambda_{\min}(I-K^\star)\} C φ > 1/ { 2 λ m i n ( I − K ⋆ )} ensures that A φ ⋆ ≻ 0 A_\varphi^\star\succ0 A φ ⋆ ≻ 0 . Hence, the Lyapunov equation in (18) admits a unique positive semidefinite solution.
We next show the reduction in Σ m , a s l \Sigma_{m,\mathrm{as}}^l Σ m , as l compared to Σ g , a s l \Sigma_{g,\mathrm{as}}^l Σ g , as l (the asymptotic covariance of vanilla online Newton sketch), and we relate both to the asymptotic covariance of exact online Newton.
Proposition 4.6. Under the conditions of Theorem 4.3, the following statements hold.
(a) We have Γ m , a s ⪯ Γ g , a s \Gamma_{m,\mathrm{as}} \preceq\Gamma_{g,\mathrm{as}} Γ m , as ⪯ Γ g , as and Σ m , a s l ⪯ Σ g , a s l \Sigma^{l}_{m,\mathrm{as}} \preceq\Sigma^{l}_{g,\mathrm{as}} Σ m , as l ⪯ Σ g , as l .
(b) If the Newton system (4) is solved exactly (K ~ ⋆ = K ⋆ = 0 \widetilde{K}^{\star}=K^{\star}=0 K ⋆ = K ⋆ = 0 ), the last-iterate covariances reduce to Σ m , e l = Σ g , e l = 0.5 Ω \Sigma^{l}_{m,e}=\Sigma^{l}_{g,e}=0.5\Omega Σ m , e l = Σ g , e l = 0.5Ω for φ ∈ ( 0.5 , 1 ) \varphi\in(0.5,1) φ ∈ ( 0.5 , 1 ) and Σ m , e l = Σ g , e l = C φ 2 C φ − 1 ⋅ Ω \Sigma^{l}_{m,e}=\Sigma^{l}_{g,e}=\frac{C_{\varphi}}{2C_{\varphi}-1}\cdot\Omega Σ m , e l = Σ g , e l = 2 C φ − 1 C φ ⋅ Ω for φ = 1 \varphi=1 φ = 1 .
(c) (t \sqrt{t} t -consistency) Let φ t = 1 / t \varphi_t=1/t φ t = 1/ t (i.e., φ = C φ = 1 \varphi=C_{\varphi}=1 φ = C φ = 1 ). Then, Ω = Σ g , e l = Σ m , e l ⪯ Σ m , a s l ⪯ Σ g , a s l \Omega=\Sigma^{l}_{g,e}=\Sigma^{l}_{m,e}\preceq\Sigma^{l}_{m,\mathrm{as}}\preceq\Sigma^{l}_{g,\mathrm{as}} Ω = Σ g , e l = Σ m , e l ⪯ Σ m , as l ⪯ Σ g , as l .
Part (a) shows the covariance reduction achieved by gradient averaging. The gap Σ g , a s l − Σ m , a s l \Sigma^{l}_{g,\mathrm{as}}-\Sigma^{l}_{m,\mathrm{as}} Σ g , as l − Σ m , as l is positive semidefinite and arises from the centered sketching fluctuation K ⋆ − K ~ ⋆ K^{\star}-\widetilde{K}^{\star} K ⋆ − K ⋆ . Part (b) shows that gradient averaging leaves the last-iterate covariance of exact online Newton in (3) unchanged. Part (c) compares sketched and exact Newton with φ t = 1 / t \varphi_t=1/t φ t = 1/ t , so that all four last-iterate covariances correspond to the same t \sqrt{t} t normalization. In this regime, the remaining gap Σ m , a s l − Ω \Sigma^{l}_{m,\mathrm{as}}-\Omega Σ m , as l − Ω stems from the factor I − K ⋆ I-K^{\star} I − K ⋆ in the Lyapunov equation (18) . Thus, gradient averaging eliminates the covariance contribution from the centered sketching fluctuation K ⋆ − K ~ ⋆ K^{\star}-\widetilde{K}^{\star} K ⋆ − K ⋆ , while the residual dependence on I − K ⋆ I-K^{\star} I − K ⋆ in (15) prevents the covariance from further reducing to that of exact Newton method.
Lastly, we turn to the Polyak–Ruppert average x ˉ t \bar{x}_t x ˉ t studied in (16) .
Theorem 4.7. Under the conditions of Theorem 4.3 , suppose that φ < 1 \varphi<1 φ < 1 . Then, we have
t ( x ˉ t − x ⋆ ) → d N ( 0 , Σ m , a s a ) , where Σ m , a s a = Ω . (19) \sqrt{t}\left(\bar{x}_t-x^{\star}\right)\mathrel{\xrightarrow{d}}\mathcal{N}\left(0,\Sigma^{a}_{m,\mathrm{as}}\right), \qquad\text{where }\Sigma^{a}_{m,\mathrm{as}}=\Omega. \tag*{(19)} t ( x ˉ t − x ⋆ ) d N ( 0 , Σ m , as a ) , where Σ m , as a = Ω. (19)
The result (16) reduces the averaged iterate to a normalized sum of i.i.d. mean-zero random vectors up to an o p ( 1 ) o_p(1) o p ( 1 ) term. Therefore, the multivariate central limit theorem and Slutsky’s theorem yield (19) . To quantify the effect of gradient averaging, consider vanilla online sketched Newton in (7) . By Theorem 2.2 and the definition of Γ g , a s \Gamma_{g,\mathrm{as}} Γ g , as in (9) , its averaged-iterate covariance is
Σ g , a s a ≔ ( I − K ⋆ ) − 1 Γ g , a s ( I − K ⋆ ) − 1 = Ω + ( I − K ⋆ ) − 1 E [ ( K ~ ⋆ − K ⋆ ) Ω ( K ~ ⋆ − K ⋆ ) T ] ( I − K ⋆ ) − 1 ⪰ Ω . (20) \begin{aligned}
\Sigma^{a}_{g,\mathrm{as}} \coloneqq(I-K^{\star})^{-1}\Gamma_{g,\mathrm{as}}(I-K^{\star})^{-1} \\
&= \Omega+(I-K^{\star})^{-1}\mathbb{E}\left[(\widetilde{K}^{\star}-K^{\star})\Omega(\widetilde{K}^{\star}-K^{\star})^{\mathsf{T}}\right](I-K^{\star})^{-1}\succeq\Omega. \tag*{(20)}
\end{aligned} Σ g , as a : = ( I − K ⋆ ) − 1 Γ g , as ( I − K ⋆ ) − 1 = Ω + ( I − K ⋆ ) − 1 E [ ( K ⋆ − K ⋆ ) Ω ( K ⋆ − K ⋆ ) T ] ( I − K ⋆ ) − 1 ⪰ Ω. (20)
Comparing (19) with (20) reveals a sharp statistical distinction. Vanilla online sketched Newton suffers from a positive semidefinite covariance inflation induced by the centered sketching fluctuation, while Σ m , a s a = Ω \Sigma^{a}_{m,\mathrm{as}}=\Omega Σ m , as a = Ω . The covariance Ω \Omega Ω coincides with that of averaged exact online Newton in (3) and that of averaged SGD ([30 , 28 ] ), which is known to be asymptotically minimax optimal ([9 , 5 ] ). Therefore, for every fixed τ \tau τ satisfying the conditions of Theorem 3.5, accelerated sketching incurs no computational–statistical trade-off for the averaged iterate: it significantly reduces the cost of Newton system solve while preserving the asymptotically optimal statistical efficiency achieved by averaged exact Newton and SGD.
Numerical Experiments
We compare online Newton methods with and without gradient averaging, using either accelerated sketching or exact solves, and include SGD as a baseline. We consider linear and logistic regression with Gaussian designs, vary d ∈ { 20 , 40 } d\in\{20,40\} d ∈ { 20 , 40 } , and vary the design covariance structure among identity, equicorrelation, or Toeplitz. Kaczmarz and Gaussian sketches with τ ∈ { 5 , 10 } \tau\in\{5,10\} τ ∈ { 5 , 10 } yield 48 matched settings. Each configuration uses R = 200 R=200 R = 200 independent runs of T = 10 5 T=10^5 T = 1 0 5 iterations. All methods share the same data streams, and matched sketching methods additionally share the same sketches. Σ \Sigma Σ denotes the centered empirical covariances, computed across runs, of ( x t − x ⋆ ) / φ t (x_t-x^{\star})/\sqrt{\varphi_t} ( x t − x ⋆ ) / φ t and t ( x ˉ t − x ⋆ ) \sqrt{t}(\bar{x}_t-x^{\star}) t ( x ˉ t − x ⋆ ) . Bias and mean squared error (MSE) are evaluated separately. Appendix E provides complete experimental setups and additional results, including estimation-error comparisons with SGD and comparisons with averaged SGD on confidence-interval coverage, length, and computational cost.
Covariance reduction with gradient averaging. Figure 1 (a,Figure 1 c) shows that gradient averaging reduces the last-iterate covariance trace with the same data streams, sketches, inner sketching solve budgets, and outer stepsizes. In particular, each point on the plot corresponds to a specific configuration ratio, and all points lie below the line y = x y=x y = x . Thus, the reduction holds across design covariance structures and sketching distributions, including higher-dimensional settings reported in Appendix E. By comparison, Table 2 shows that the covariance trace of exact Newton remains nearly unchanged under gradient averaging. These observations align with Proposition 4.6 : gradient averaging reduces the last-iterate limiting covariance of sketched Newton while leaving that of exact Newton unchanged.
Figure 1. Covariance comparisons (d = 20 d = 20 d = 20 , T = 10 5 T = 10^5 T = 1 0 5 , R = 200 R = 200 R = 200 ). Panels (a–d) compare covariance traces without (horizontal axis) and with (vertical axis) gradient averaging across 12 settings. All averaged-iterate covariance traces are normalized by tr ( Ω ) \operatorname{tr}(\Omega) tr ( Ω ) . For identity design covariance, Gaussian sketches, and τ = 10 \tau= 10 τ = 10 , panels (e,f) show covariance trace trajectories with pointwise 95% bootstrap intervals, while panels (g,h) show standard normal QQ plots of T 1 T ( x ˉ T − x ⋆ ) / 1 T Ω 1 \sqrt{T}\,\mathbf{1}^{\mathsf{T}}(\bar{x}_T-x^\star)/\sqrt{\mathbf{1}^{\mathsf{T}}\Omega\mathbf{1}} T 1 T ( x ˉ T − x ⋆ ) / 1 T Ω 1 without recentering. Diagonal lines indicate equality, and horizontal lines indicate one. The inset magnifies the six points corresponding to τ = 10 \tau= 10 τ = 10 . Sketching uses g t g_t g t without gradient averaging and m t m_t m t with gradient averaging, while exact Newton uses g t g_t g t .
Table 2. Moving-average/vanilla ratios at T = 10 5 T = 10^{5} T = 1 0 5 over R = 200 R = 200 R = 200 runs per configuration. Entries give median [minimum, maximum] over 48 sketching settings or 12 exact-solve settings. All sketching ratios are below one. Directional variance uses 1; bias, MSE, and objective gap use unnormalized errors.
Comparison with exact online Newton. Figure 1 (b,d,e,f) shows that gradient averaging brings the averaged-iterate covariance trace closer to the exact Newton benchmark, since not only do all points lie below y = x y = x y = x , but more points also lie close to y = 1 y = 1 y = 1 . Note that both vanilla sketched Newton and our method adopt iterate averaging, so the improvement is purely attributable to gradient averaging. Theorem 4.7 explains this comparison: gradient averaging along with iterate averaging recovers Ω \Omega Ω asymptotically, while vanilla sketched Newton retains an additional sketch-dependent covariance contribution. The trajectories in panels (e,f) further support this conclusion: with gradient averaging, the covariance trace approaches tr ( Ω ) \operatorname{tr}(\Omega) tr ( Ω ) while the number of sketching steps per iteration remains fixed.
Estimation accuracy and normal approximation. From Table 1 , we observe that gradient averaging reduces the MSE of both the last and averaged iterates across all tested settings. The bias comparisons in Appendix § 12 show similar improvements, indicating that the covariance reduction is accompanied by improved estimation accuracy. Increasing τ \tau τ also brings the averaged-iterate covariance trace closer to tr ( Ω ) \operatorname{tr}(\Omega) tr ( Ω ) . Thus, although the limiting covariance is independent of sketching, more accurate inner solves remain beneficial at finite iterations. Figure 1 (g,h) further complements these comparisons. In particular, with gradient averaging, the logistic regression QQ plot closely follows the standard normal reference. The linear regression QQ plot exhibits a similar pattern but is shifted due to finite-sample bias; this shift is still smaller with gradient averaging.
Table 1: *Comparisons at T = 10 5 T = 10^5 T = 1 0 5 with R = 200 R = 200 R = 200 . Covariance-trace and MSE ratios compare sketched Newton with gradient averaging to its vanilla counterpart. The central columns report tr ( Σ ^ u , a s ) / tr ( Ω ) \operatorname{tr}(\widehat{\Sigma}_{u,\mathrm{as}})/\operatorname{tr}(\Omega) tr ( Σ u , as ) / tr ( Ω ) for u ∈ { g , m } u \in\{g,m\} u ∈ { g , m } . Entries are medians of six settingwise ratios across three design covariances and two sketching distributions.
Table 1.
Conclusion
We studied statistical efficiency of online Newton methods with Nesterov accelerated sketching and gradient averaging. We established almost-sure convergence and asymptotic normality of the last and averaged iterates. We showed that gradient averaging yields a last-iterate covariance no larger than that of vanilla online sketched Newton, while subsequent averaging of the outer iterates further achieves the minimax optimal limiting covariance of exact online Newton. Thus, for averaged iterates, a fixed number of sketching steps can reduce the cost of an expensive Newton solve without inflating the limiting covariance. Gradient averaging requires no extra oracle evaluations and leaves the contraction bound of the inner sketching solver unchanged. Experiments on linear and logistic regression support our free-lunch finding and further demonstrate improvements in bias and mean squared error. We should mention that our asymptotic normality only provides the first step toward practical inference. Establishing theoretical guarantees for online covariance estimation or random scaling based on the gradient-averaged iterates remains an important direction for future work.
AI Use Statement
We used generative AI tools to assist with writing and polishing the manuscript. We have reviewed all AI-assisted text and take responsibility for the final manuscript, including text, claims, proofs, and experimental results.
Appendix: “A Free Lunch: Gradient Averaging Achieves Optimal Statistical Efficiency in Newton Sketch”
Additional Discussion
In this section, we provide additional discussion of several aspects of the main text: the convergence rates and stepsize conditions for SGD and online Newton methods; Assumption 3.1, particularly the population strong convexity of the objective and the relationship between the conditions on B t B_t B t and those used in conditioned SGD; the choice of stepsizes for statistical inference; and the computational complexity, including the required number of sketching steps and the cost per outer iteration.
Rates of SGD and online Newton. In the online setting with vanishing stepsizes, SGD and online Newton methods share the same convergence rates, which are governed by the stepsize: for both families, the last iterate converges at rate O p ( φ t ) O_p(\sqrt{\varphi_t}) O p ( φ t ) , while the averaged iterate converges at rate O p ( 1 / t ) O_p(1/\sqrt{t}) O p ( 1/ t ) . Their distinction for statistical inference lies instead in the limiting covariance of the iterates and the accuracy of covariance estimation, as discussed in § 1 . However, a difference arises in the stepsize requirements for last-iterate asymptotic normality. When φ t = C φ / ( t + 1 ) \varphi_t=C_\varphi/(t+1) φ t = C φ / ( t + 1 ) , SGD requires C φ > 1 / { 2 λ min ( B ⋆ ) } C_\varphi>1/\{2\lambda_{\min}(B^\star)\} C φ > 1/ { 2 λ m i n ( B ⋆ )} , for which Assumption 3.1 yields the sufficient condition C φ > 1 / ( 2 γ H ) C_\varphi>1/(2\gamma_H) C φ > 1/ ( 2 γ H ) . In either case, the requirement depends on the unknown curvature of the objective. In contrast, exact online Newton requires only C φ > 1 / 2 C_\varphi>1/2 C φ > 1/2 due to affine invariance, as Hessian preconditioning removes the dependence on the objective curvature. For accelerated sketched Newton, the corresponding condition is C φ > 1 / { 2 λ min ( I − K ⋆ ) } C_\varphi>1/\{2\lambda_{\min}(I-K^\star)\} C φ > 1/ { 2 λ m i n ( I − K ⋆ )} (cf. Lemma 4.2 and Theorem 4.3 ), which reduces to C φ > 1 / 2 C_\varphi>1/2 C φ > 1/2 when the Newton systems are solved exactly, i.e., K ⋆ = 0 K^\star=0 K ⋆ = 0 .
Population curvature and sample Hessians. Assumption 3.1 requires strong convexity and smoothness only of the population objective f f f , without imposing such conditions on the sample functions F ( ⋅ ; ξ ) F(\cdot;\xi) F ( ⋅ ; ξ ) . These are standard conditions in SGD-based inference [28 , 4 , 36 ] . Although SGD can converge under much weaker conditions, including for nonconvex problems, strong convexity remains a common and convenient assumption for inference, for two reasons. First, inference is inherently local: it quantifies the uncertainty of the iterates around a strict local minimizer x ⋆ x^\star x ⋆ satisfying B ⋆ = ∇ 2 f ( x ⋆ ) ≻ 0 B^\star=\nabla^2f(x^\star)\succ0 B ⋆ = ∇ 2 f ( x ⋆ ) ≻ 0 . By continuity of ∇ 2 f \nabla^2f ∇ 2 f , strong convexity then holds in a neighborhood of x ⋆ x^\star x ⋆ . The global lower bound on ∇ 2 f \nabla^2f ∇ 2 f is imposed here to establish x t → a . s . x ⋆ x_t\xrightarrow{\mathrm{a.s.}}x^\star x t a.s. x ⋆ in Theorem 3.5 , while the essential curvature requirement for inference is local. Second, positive definiteness of the population Hessian at the solution is essential for establishing meaningful statistical lower bounds. At such a minimizer, Ω \Omega Ω is well defined and characterizes the local asymptotic minimax covariance [9 , 5 ] . Recent instance-dependent nonasymptotic (local minimax) lower bounds are likewise established for strongly convex population objectives [18 ] . Thus, our conditions on the objective are no stronger than those commonly imposed for SGD inference.
For the sample functions, Assumption 3.1 requires only twice continuous differentiability, and Assumption 3.3 controls the second moment of the Hessian noise. In particular, we do not require the sample Hessians to be positive definite or uniformly bounded [20 , 34 , 8 ] . Linear regression illustrates the importance of this relaxation. Consider F ( x ; ( a , y ) ) = ( a ⊤ x − y ) 2 / 2 F(x;(a,y))=(a^\top x-y)^2/2 F ( x ; ( a , y )) = ( a ⊤ x − y ) 2 /2 with a ∼ N ( 0 , Σ a ) a\sim\mathcal{N}(0,\Sigma_a) a ∼ N ( 0 , Σ a ) and Σ a ≻ 0 \Sigma_a\succ0 Σ a ≻ 0 . The sample Hessian a a ⊤ aa^\top a a ⊤ is rank one and has no uniform upper bound, so uniform sample-level spectral bounds cannot hold. In contrast, the population Hessian satisfies ∇ 2 f ( x ) = E [ a a ⊤ ] = Σ a ≻ 0 \nabla^2f(x)=\mathbb{E}[aa^\top]=\Sigma_a\succ0 ∇ 2 f ( x ) = E [ a a ⊤ ] = Σ a ≻ 0 , and the clipped Hessian average B t B_t B t in (14) satisfies the conditions on B t B_t B t in Assumption 3.1 by construction, without requiring spectral bounds on individual sample Hessians. We also note that, as is standard, a regularization term is included in logistic regression to ensure the global lower bound on ∇ 2 f \nabla^2f ∇ 2 f . See Appendix § 12 .§ 1 .
Relation to conditioned SGD. For the Hessian approximation B t B_t B t , Assumption 3.1 adopts exactly the same conditions used in conditioned SGD [21 Assumption 9] : B t B_t B t is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable and has lower and upper bounds of the same order. No additional conditions on B t B_t B t are imposed. In particular, we do not require any sample Hessian to be positive definite. To see the correspondence, note that the conditioning matrix applied to the gradient in our Newton update is B t − 1 B_{t}^{-1} B t − 1 , and the bounds in Assumption 3.1 give 1 / Γ B , t ⋅ I ⪯ B t − 1 ⪯ 1 / γ B , t ⋅ I 1/\Gamma_{B,t}\cdot I\preceq B_t^{-1}\preceq1/\gamma_{B,t}\cdot I 1/ Γ B , t ⋅ I ⪯ B t − 1 ⪯ 1/ γ B , t ⋅ I . Leluc & Portier [21 ] states the stepsize conditions through the upper bound of the conditioning matrix, which is 1 / γ B , t 1/\gamma_{B,t} 1/ γ B , t here, so the two series conditions become exactly ∑ t φ t / γ B , t = ∞ \sum_t\varphi_t/\gamma_{B,t}=\infty ∑ t φ t / γ B , t = ∞ and ∑ t ( φ t / γ B , t ) 2 < ∞ \sum_t(\varphi_t/\gamma_{B,t})^2<\infty ∑ t ( φ t / γ B , t ) 2 < ∞ . Likewise, the ratio of the upper and lower bounds, which is required to be bounded, is Γ B , t / γ B , t \Gamma_{B,t}/\gamma_{B,t} Γ B , t / γ B , t .
Choice of stepsizes for inference. There is no universally optimal exponent φ \varphi φ : its choice depends on whether the last or averaged iterate is used for inference and reflects a trade-off between nonasymptotic and asymptotic performance. (i) For nonasymptotic convergence, φ t ∝ 1 / t \varphi_t \propto1/\sqrt{t} φ t ∝ 1/ t , i.e., φ = 0.5 \varphi= 0.5 φ = 0.5 , is known to achieve the optimal convergence rate [23 ] . (ii) For the averaged iterate, any φ ∈ ( 0.5 , 1 ) \varphi\in(0.5,1) φ ∈ ( 0.5 , 1 ) yields t \sqrt{t} t -consistency with the same optimal limiting covariance Ω \Omega Ω [28 ] , as also established for our method in Theorem 4.7. Thus, when using the averaged iterate for inference, one would favor φ \varphi φ close to 0.5, which improves nonasymptotic performance and allows the averaged iterate to enter its asymptotic regime sooner, as larger stepsizes diminish the effect of initialization more rapidly. (iii) For the last iterate, the situation is reversed. Its asymptotic normality is established for ( x t − x ⋆ ) / φ t (x_t-x^\star)/\sqrt{\varphi_t} ( x t − x ⋆ ) / φ t , so its variance is of order φ t = C φ / ( t + 1 ) φ \varphi_t = C_\varphi/(t+1)^\varphi φ t = C φ / ( t + 1 ) φ . An exponent close to 0.5 therefore leads to a larger variance, while t \sqrt{t} t -consistency with the optimal last-iterate covariance is attained only when φ t ∝ 1 / t \varphi_t \propto1/t φ t ∝ 1/ t (Proposition 4.6 (c)). Thus, when using the last iterate for inference, one would favor φ \varphi φ close to 1, at the expense of slower nonasymptotic convergence. At φ = 1 \varphi= 1 φ = 1 , the condition on C φ C_\varphi C φ in Theorem 4.3 becomes relevant, and Newton methods impose weaker requirements on C φ C_\varphi C φ than SGD, as discussed above. In our experiments, we choose φ = 0.8 \varphi= 0.8 φ = 0.8 as a compromise that allows both estimators to be evaluated using the same runs.
Constant stepsizes are common when optimization is the primary objective, as in training. However, they are generally unsuitable for achieving optimal statistical efficiency in inference, which is the focus of this paper. With a constant stepsize, SGD iterates typically do not converge to x ⋆ x^\star x ⋆ but instead approach a stationary distribution whose mean is biased away from x ⋆ x^\star x ⋆ by an amount of the order of the stepsize. Iterate averaging reduces the variance but does not eliminate this bias [7 , 17 ] (Yu et al. 2021). Consequently, the resulting estimator is generally not consistent, let alone t \sqrt{t} t -consistent with a limiting covariance comparable to Ω \Omega Ω . For this reason, the existing literature on online inference with SGD and Newton methods, upon which we build, adopts decaying stepsizes [4 , 36 , 21 , 20 , 34 , 8 ] . Removing the bias induced by constant stepsizes requires additional techniques, such as Richardson–Romberg extrapolation [7 , 17 ] , which are beyond the scope of the present study.
Per-iteration computational cost. Gradient averaging does not alter the sketching solver, so the discussion below follows the existing literature on sketch-and-project methods [15 , 14 , 6 ] . The computational cost of the sketching solver is determined by the number of sketching steps and the cost per step.
Theorems 3.5 and 4.3 require τ \tau τ satisfying τ ( 1 − μ t / ν t ) τ − 2 ≤ γ B t / ( δ Y t ) \tau(1-\sqrt{\mu_t/\nu_t})^{\tau-2} \le\gamma\mathcal{B}_t/(\delta\mathcal{Y}_t) τ ( 1 − μ t / ν t ) τ − 2 ≤ γ B t / ( δ Y t ) for all t t t . Since 1 − μ t / ν t = 1 − γ S < 1 1-\sqrt{\mu_t/\nu_t}=1-\gamma_S<1 1 − μ t / ν t = 1 − γ S < 1 by Assumption 3.4 and sup t ≥ 0 B t / Y t < ∞ \sup_{t\ge0}\mathcal{B}_{t}/\mathcal{Y}_{t}<\infty sup t ≥ 0 B t / Y t < ∞ almost surely by Assumption 3.1, the required number of sketching steps is uniformly bounded in t t t . In particular, we do not require the approximation error in solving the Newton systems to vanish over time. At each sketching step, the expected squared error contracts by a factor of 1 − μ t 1-\mu_t 1 − μ t without acceleration and 1 − μ t / ν t ≤ 1 − μ t 1-\sqrt{\mu_t/\nu_t}\le1-\mu_t 1 − μ t / ν t ≤ 1 − μ t with acceleration (Lemma 2.1 ). For randomized coordinate sketching S = e j \mathcal{S} = e_j S = e j , we have μ t ≥ 1 / ( d κ t 2 ) \mu_t \ge1/(d\kappa_t^2) μ t ≥ 1/ ( d κ t 2 ) and ν t = d \nu_t = d ν t = d , where κ t = λ max ( B t ) / λ min ( B t ) \kappa_t = \lambda_{\max}(B_t)/\lambda_{\min}(B_t) κ t = λ m a x ( B t ) / λ m i n ( B t ) . Consequently, acceleration reduces the number of steps required to reach a prescribed accuracy ϵ \epsilon ϵ from O ( d κ t 2 log ( 1 / ϵ ) ) O(d\kappa_t^2\log(1/\epsilon)) O ( d κ t 2 log ( 1/ ϵ )) to O ( d κ t log ( 1 / ϵ ) ) O(d\kappa_t\log(1/\epsilon)) O ( d κ t log ( 1/ ϵ )) .
The per sketching step cost depends on the sketching distribution and the sparsity of the sketching matrix. For randomized coordinate sketching, each step accesses only one column of B t B_t B t and costs O ( d ) O(d) O ( d ) , with or without acceleration.
Finally, by (i) and (ii), the sketching solver costs O ( d ) × O ( d ) = O ( d 2 ) O(d)\times O(d)=O(d^2) O ( d ) × O ( d ) = O ( d 2 ) per outer iteration, compared with O ( d 3 ) O(d^3) O ( d 3 ) for an exact Newton solve. The constant hidden in this bound depends on the condition number κ t \kappa_t κ t , and acceleration improves the worst-case dependence from κ t 2 \kappa_t^2 κ t 2 to κ t \kappa_t κ t . The remaining components do not increase the computational order: updating the Hessian average costs O ( d 2 ) O(d^2) O ( d 2 ) , spectral clipping, performed once every d d d iterations, incurs an amortized cost of O ( d 2 ) O(d^2) O ( d 2 ) , and gradient averaging adds only an O ( d ) O(d) O ( d ) vector update.
Preparation Lemmas
Lemma B.1 (Kronecker’s lemma). Suppose { A t } t ≥ 0 \{A_t\}_{t\geq0} { A t } t ≥ 0 satisfies ∑ t = 0 ∞ A t < ∞ \sum_{t=0}^{\infty} A_t < \infty ∑ t = 0 ∞ A t < ∞ . Then, for any positive, non-decreasing sequence { a t } t ≥ 0 \{a_t\}_{t\geq0} { a t } t ≥ 0 satisfying a t → ∞ a_t \to\infty a t → ∞ , we have 1 a T ∑ t = 1 T a t A t → 0 \frac{1}{a_T}\sum_{t=1}^{T} a_t A_t \to0 a T 1 ∑ t = 1 T a t A t → 0 as T → ∞ T \to\infty T → ∞ .
Lemma B.2 (Stolz–Cesaro theorem). Let { a t } t ≥ 1 \{a_t\}_{t\geq1} { a t } t ≥ 1 and { b t } t ≥ 1 \{b_t\}_{t\geq1} { b t } t ≥ 1 be two sequences of real numbers. Suppose that { b t } t ≥ 1 \{b_t\}_{t\geq1} { b t } t ≥ 1 is strictly monotone and divergent, and that the following limit exists:
lim t → ∞ a t + 1 − a t b t + 1 − b t = ℓ . \lim_{t\to\infty} \frac{a_{t+1}-a_t}{b_{t+1}-b_t}=\ell. t → ∞ lim b t + 1 − b t a t + 1 − a t = ℓ . Then,
lim t → ∞ a t b t = ℓ . \lim_{t\to\infty} \frac{a_t}{b_t}=\ell. t → ∞ lim b t a t = ℓ .
Lemma B.3. ([21 Lemma 3] ) Let { β t } t ≥ 1 \{\beta_t\}_{t\geq1} { β t } t ≥ 1 be a nonnegative sequence converging to zero, and let λ > 0 \lambda>0 λ > 0 , m ≥ 1 m\geq1 m ≥ 1 , and q ≥ 0 q\geq0 q ≥ 0 . Consider two nonnegative sequences { A t } t ≥ 0 \{A_t\}_{t\geq0} { A t } t ≥ 0 and { ε t } t ≥ 1 \{\varepsilon_t\}_{t\geq1} { ε t } t ≥ 1 such that, for all sufficiently large t t t ,
A t = ( 1 − λ β t ) m A t − 1 + β t q + 1 ε t . A_t=(1-\lambda\beta_t)^m A_{t-1}+\beta_t^{q+1}\varepsilon_t. A t = ( 1 − λ β t ) m A t − 1 + β t q + 1 ε t . If β t = β 0 / ( t + 1 ) a \beta_t=\beta_0/(t+1)^a β t = β 0 / ( t + 1 ) a with a ∈ ( 0 , 1 ) a\in(0,1) a ∈ ( 0 , 1 ) , then
lim sup t → ∞ A t β t q ≤ 1 m λ lim sup t → ∞ ε t . \limsup_{t\to\infty}\frac{A_t}{\beta_t^q}\leq\frac{1}{m\lambda}\limsup_{t\to\infty}\varepsilon_t. t → ∞ lim sup β t q A t ≤ mλ 1 t → ∞ lim sup ε t . If β t = β 0 / ( t + 1 ) \beta_t=\beta_0/(t+1) β t = β 0 / ( t + 1 ) and q < m λ β 0 q<m\lambda\beta_0 q < mλ β 0 , then
lim sup t → ∞ A t β t q ≤ β 0 m λ β 0 − q lim sup t → ∞ ε t . \limsup_{t\to\infty}\frac{A_t}{\beta_t^q}\leq\frac{\beta_0}{m\lambda\beta_0-q}\limsup_{t\to\infty}\varepsilon_t. t → ∞ lim sup β t q A t ≤ mλ β 0 − q β 0 t → ∞ lim sup ε t .
Lemma B.4. ([16 Corollary 3.1] ) For every t ≥ 1 t\geq1 t ≥ 1 , let { W t , i } i = 0 t − 1 \{W_{t,i}\}_{i=0}^{t-1} { W t , i } i = 0 t − 1 be a triangular array of R d \mathbb{R}^d R d -valued random vectors such that W t , i W_{t,i} W t , i is F i \mathcal{F}_i F i -measurable and
E [ W t , i ∣ F i − 1 ] = 0 , (21) \mathbb{E}[W_{t,i}\mid\mathcal{F}_{i-1}]=0, \tag*{(21)} E [ W t , i ∣ F i − 1 ] = 0 , (21) ∑ i = 0 t − 1 E [ W t , i W t , i ⊤ ∣ F i − 1 ] → p V ⋆ ⪰ 0 , (22) \sum_{i=0}^{t-1}\mathbb{E}[W_{t,i}W_{t,i}^{\top}\mid\mathcal{F}_{i-1}]\xrightarrow{p}V^\star\succeq0, \tag*{(22)} i = 0 ∑ t − 1 E [ W t , i W t , i ⊤ ∣ F i − 1 ] p V ⋆ ⪰ 0 , (22) ∑ i = 0 t − 1 E [ ∥ W t , i ∥ 2 1 { ∥ W t , i ∥ > ϵ } ∣ F i − 1 ] → p 0 , ∀ ϵ > 0. (23) \sum_{i=0}^{t-1}\mathbb{E}[\lVert W_{t,i}\rVert^2\mathbf{1}_{\{\lVert W_{t,i}\rVert>\epsilon\}}\mid\mathcal{F}_{i-1}]\xrightarrow{p}0,\qquad\forall\ \epsilon>0. \tag*{(23)} i = 0 ∑ t − 1 E [∥ W t , i ∥ 2 1 {∥ W t , i ∥ > ϵ } ∣ F i − 1 ] p 0 , ∀ ϵ > 0. (23) Then, we have
∑ i = 0 t − 1 W t , i → d N ( 0 , V ⋆ ) . \sum_{i=0}^{t-1}W_{t,i}\xrightarrow{d}\mathcal{N}(0,V^\star). i = 0 ∑ t − 1 W t , i d N ( 0 , V ⋆ ) .
Proofs of Section 3
Throughout this section, we write f t : = f ( x t ) f_t:=f(x_t) f t := f ( x t ) and f ⋆ : = f ( x ⋆ ) f^\star:=f(x^\star) f ⋆ := f ( x ⋆ ) and use 1 { ⋅ } \mathbf{1}_{\{\cdot\}} 1 { ⋅ } to denote the indicator function.
Proof of Theorem 3.5
We first introduce the inner-solver quantities used in the proof. For every t ≥ 0 t\geq0 t ≥ 0 , we define
Z ~ t , j : = B t S t , j ( S t , j ⊤ B t 2 S t , j ) − 1 S t , j ⊤ B t ∈ R d × d \widetilde{Z}_{t,j}:=B_tS_{t,j}(S_{t,j}^{\top}B_t^2S_{t,j})^{-1}S_{t,j}^{\top}B_t\in\mathbb{R}^{d\times d} Z t , j := B t S t , j ( S t , j ⊤ B t 2 S t , j ) − 1 S t , j ⊤ B t ∈ R d × d
and
C ~ t , j : = ( 1 − α t α t ( 1 − α t ) ( 1 − β t ) α t + β t − α t β t ) ⊗ I d − ( 1 − α t α t ( 1 − α t ) γ t α t γ t ) ⊗ Z ~ t , j . (24) \widetilde{C}_{t,j}:=
\begin{pmatrix}
1-\alpha_t & \alpha_t\\
(1-\alpha_t)(1-\beta_t) & \alpha_t+\beta_t-\alpha_t\beta_t
\end{pmatrix}\otimes I_d-
\begin{pmatrix}
1-\alpha_t & \alpha_t\\
(1-\alpha_t)\gamma_t & \alpha_t\gamma_t
\end{pmatrix}\otimes\widetilde{Z}_{t,j}. \tag*{(24)} C t , j := ( 1 − α t ( 1 − α t ) ( 1 − β t ) α t α t + β t − α t β t ) ⊗ I d − ( 1 − α t ( 1 − α t ) γ t α t α t γ t ) ⊗ Z t , j . (24)
Let C ~ t : = C ~ t , τ − 1 ⋯ C ~ t , 0 \widetilde{C}_t:=\widetilde{C}_{t,\tau-1}\cdots\widetilde{C}_{t,0} C t := C t , τ − 1 ⋯ C t , 0 and write [ C ~ t ] a , b [\widetilde{C}_t]_{a,b} [ C t ] a , b for its ( a , b ) (a,b) ( a , b ) -th d × d d\times d d × d block. The realized residual matrix introduced in (8) has the explicit representation
K ~ t : = [ C ~ t ] 1 , 1 + [ C ~ t ] 1 , 2 = ( I 0 ) C ~ t ( I I ) . (25) \widetilde{K}_t:=[\widetilde{C}_t]_{1,1}+[\widetilde{C}_t]_{1,2}=(I\quad0)\widetilde{C}_t
\begin{pmatrix}
I\\
I
\end{pmatrix}. \tag*{(25)} K t := [ C t ] 1 , 1 + [ C t ] 1 , 2 = ( I 0 ) C t ( I I ) . (25)
Moreover, we define
M t : = ( ( 1 − α t ( 1 − α t ) ( 1 − β t ) α t α t + β t − α t β t ) ⊗ I d − ( 1 − α t ( 1 − α t ) γ t α t α t γ t ) ⊗ Z t ) . (26) M_t := \left(\begin{pmatrix} \frac{1-\alpha_t}{(1-\alpha_t)(1-\beta_t)} & \frac{\alpha_t}{\alpha_t+\beta_t-\alpha_t\beta_t} \end{pmatrix} \otimes I_d - \begin{pmatrix} \frac{1-\alpha_t}{(1-\alpha_t)\gamma_t} & \frac{\alpha_t}{\alpha_t\gamma_t} \end{pmatrix} \otimes Z_t\right). \tag*{(26)} M t := ( ( ( 1 − α t ) ( 1 − β t ) 1 − α t α t + β t − α t β t α t ) ⊗ I d − ( ( 1 − α t ) γ t 1 − α t α t γ t α t ) ⊗ Z t ) . (26)
The conditional independence of the inner sketches gives
C t : = E [ C ~ t ∣ F t − 1 ] = M t τ , K t : = E [ K ~ t ∣ F t − 1 ] = ( I 0 ) C t ( I I ) . C_t := \mathbb{E}[\widetilde{C}_t \mid\mathcal{F}_{t-1}] = M_t^{\tau}, \qquad K_t := \mathbb{E}[\widetilde{K}_t \mid\mathcal{F}_{t-1}] = (I \quad0) C_t \begin{pmatrix} I \\ I \end{pmatrix}. C t := E [ C t ∣ F t − 1 ] = M t τ , K t := E [ K t ∣ F t − 1 ] = ( I 0 ) C t ( I I ) .
The matrices K ~ t \widetilde{K}_t K t and K t K_t K t are precisely the realized and mean residual matrices introduced in § 2 . Indeed, the block matrices in (C.1) and their conditional means depend on B t B_t B t , the acceleration parameters, and the current sketches or their distribution, but not on the right-hand side of the Newton system. Therefore, replacing g t g_t g t with m t m_t m t changes neither K ~ t \widetilde{K}_t K t nor K t K_t K t . Consequently, the bound 1 − μ t / ν t ≤ 1 − γ S < 1 1-\sqrt{\mu_t/\nu_t}\le1-\gamma_S<1 1 − μ t / ν t ≤ 1 − γ S < 1 from [34 Lemma 3.6] and the mean-residual bound from [34 Lemma 3.7] continue to hold without modification. For the moving-average gradient estimates, the exact direction Δ x t = − B t − 1 m t \Delta x_t=-B_t^{-1}m_t Δ x t = − B t − 1 m t , and the conditional-mean of the approximate direction z t , τ z_{t,\tau} z t , τ must be formulated with respect to F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 .
The following auxiliary lemma records these two relations. Its proof is deferred to Appendix § 10 .§ 2 .
Lemma C.1. Define Δ x t : = − B t − 1 m t \Delta x_t := -B_t^{-1}m_t Δ x t := − B t − 1 m t . Then, for every t ≥ 0 t\ge0 t ≥ 0 , we have
z t , τ − Δ x t = K ~ t ( z t , 0 − Δ x t ) , E [ z t , τ ∣ F t − 0.5 ] = − ( I − K t ) B t − 1 m t . z_{t,\tau}-\Delta x_t=\widetilde{K}_t(z_{t,0}-\Delta x_t), \qquad\mathbb{E}[z_{t,\tau}\mid\mathcal{F}_{t-0.5}]=-(I-K_t)B_t^{-1}m_t. z t , τ − Δ x t = K t ( z t , 0 − Δ x t ) , E [ z t , τ ∣ F t − 0.5 ] = − ( I − K t ) B t − 1 m t . Consequently, ( z t , τ − ( I − K t ) Δ x t , F t ) t ≥ 0 (z_{t,\tau}-(I-K_t)\Delta x_t,\mathcal{F}_t)_{t\ge0} ( z t , τ − ( I − K t ) Δ x t , F t ) t ≥ 0 forms a martingale difference sequence. Furthermore, if γ t = 1 \gamma_t=1 γ t = 1 , then K ~ t \widetilde{K}_t K t has a simple form of K ~ t = ( I − Z ~ t , τ − 1 ) ⋯ ( I − Z ~ t , 0 ) \widetilde{K}_t=(I-\widetilde{Z}_{t,\tau-1})\cdots(I-\widetilde{Z}_{t,0}) K t = ( I − Z t , τ − 1 ) ⋯ ( I − Z t , 0 ) .
The realized block matrices in (C.1) and K ~ t \widetilde{K}_t K t depend on B t B_t B t , the acceleration parameters, and the inner sketches, while their conditional means depend on B t B_t B t , the acceleration parameters, and the sketch distribution. None of these matrices depends on the right-hand side of the Newton system. Therefore, replacing g t g_t g t with m t m_t m t does not affect the bound 1 − μ t / ν t ≤ 1 − γ S < 1 1-\sqrt{\mu_t/\nu_t}\le1-\gamma_S<1 1 − μ t / ν t ≤ 1 − γ S < 1 from [34 Lemma 3.6] or the mean-residual bound from [34 Lemma 3.7); only the conditional-mean relation in {xref}] Lemma C.1` must be stated with respect to F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 .
By Assumption 3.1, f ( x ) f(x) f ( x ) is strongly convex. By the basic property of strong convexity [26 ] , we have the following:
γ H 2 ∥ x t − x ⋆ ∥ 2 ≤ f t − f ⋆ ≤ 1 2 γ H ∥ ∇ f t ∥ 2 . (27) \frac{\gamma_H}{2}\lVert x_t-x^\star\rVert^2\le f_t-f^\star\le\frac{1}{2\gamma_H}\lVert\nabla f_t\rVert^2. \tag*{(27)} 2 γ H ∥ x t − x ⋆ ∥ 2 ≤ f t − f ⋆ ≤ 2 γ H 1 ∥ ∇ f t ∥ 2 . (27)
Still, by the γ H \gamma_H γ H -Lipschitz continuity of ∇ f ( x ) \nabla f(x) ∇ f ( x ) implied by Assumption 3.1, we have
1 2 γ H ∥ ∇ f t ∥ 2 ≤ f t − f ⋆ ≤ γ H 2 ∥ x t − x ⋆ ∥ 2 . (28) \frac{1}{2\gamma_H}\lVert\nabla f_t\rVert^2\le f_t-f^\star\le\frac{\gamma_H}{2}\lVert x_t-x^\star\rVert^2. \tag*{(28)} 2 γ H 1 ∥ ∇ f t ∥ 2 ≤ f t − f ⋆ ≤ 2 γ H ∥ x t − x ⋆ ∥ 2 . (28)
Furthermore, the γ H \gamma_H γ H -Lipschitz continuity of ∇ f ( x ) \nabla f(x) ∇ f ( x ) yields
f t + 1 − f ⋆ ≤ f t − f ⋆ + φ t ∇ f t ⊤ z t , τ + γ H 2 φ t 2 ∥ z t , τ ∥ 2 . f_{t+1}-f^\star\le f_t-f^\star+\varphi_t\nabla f_t^\top z_{t,\tau}+\frac{\gamma_H}{2}\varphi_t^2\lVert z_{t,\tau}\rVert^2. f t + 1 − f ⋆ ≤ f t − f ⋆ + φ t ∇ f t ⊤ z t , τ + 2 γ H φ t 2 ∥ z t , τ ∥ 2 .
Taking the conditional expectation E [ ⋅ ∣ F t − 0.5 ] \mathbb{E}[\cdot\mid\mathcal{F}_{t-0.5}] E [ ⋅ ∣ F t − 0.5 ] on both sides yields
E [ f t + 1 − f ⋆ ∣ F t − 0.5 ] ≤ f t − f ⋆ + φ t E [ ∇ f t ⊤ z t , τ ∣ F t − 0.5 ] + γ H 2 φ t 2 E [ ∥ z t , τ ∥ 2 ∣ F t − 0.5 ] . (29) \mathbb{E}[f_{t+1}-f^\star\mid\mathcal{F}_{t-0.5}]\le f_t-f^\star+\varphi_t\mathbb{E}[\nabla f_t^\top z_{t,\tau}\mid\mathcal{F}_{t-0.5}]+\frac{\gamma_H}{2}\varphi_t^2\mathbb{E}[\lVert z_{t,\tau}\rVert^2\mid\mathcal{F}_{t-0.5}]. \tag*{(29)} E [ f t + 1 − f ⋆ ∣ F t − 0.5 ] ≤ f t − f ⋆ + φ t E [ ∇ f t ⊤ z t , τ ∣ F t − 0.5 ] + 2 γ H φ t 2 E [∥ z t , τ ∥ 2 ∣ F t − 0.5 ] . (29)
In addition, by Assumption 3.1, γ B , t I ⪯ B t ⪯ Υ B , t I \gamma_{B,t}I\preceq B_t\preceq\Upsilon_{B,t}I γ B , t I ⪯ B t ⪯ Υ B , t I and hence Υ B , t − 1 I ⪯ B t − 1 ⪯ γ B , t − 1 I \Upsilon_{B,t}^{-1}I\preceq B_t^{-1}\preceq\gamma_{B,t}^{-1}I Υ B , t − 1 I ⪯ B t − 1 ⪯ γ B , t − 1 I almost surely for every t ≥ 0 t\ge0 t ≥ 0 .
We first control the conditional mean of the search direction. If μ t = 1 \mu_t=1 μ t = 1 , the eigendirection representation in the proof of [34 Lemma 3.7] gives K t = 0 K_t=0 K t = 0 . If μ t < 1 \mu_t<1 μ t < 1 , [34 Lemma 3.7] and the condition on τ \tau τ give
∥ K t ∥ ≤ 2 τ ( 1 − μ t ν t ) τ − 2 ≤ 2 ⋅ γ B , t 8 Υ B , t = γ B , t 4 Υ B , t . \lVert K_t\rVert\le2\tau\left(1-\sqrt{\frac{\mu_t}{\nu_t}}\right)^{\tau-2}\le2\cdot\frac{\gamma_{B,t}}{8\Upsilon_{B,t}}=\frac{\gamma_{B,t}}{4\Upsilon_{B,t}}. ∥ K t ∥ ≤ 2 τ ( 1 − ν t μ t ) τ − 2 ≤ 2 ⋅ 8 Υ B , t γ B , t = 4 Υ B , t γ B , t .
Therefore, in either case, it holds almost surely that ∥ K t ∥ ≤ γ B , t 4 Υ B , t \lVert K_t\rVert\le\frac{\gamma_{B,t}}{4\Upsilon_{B,t}} ∥ K t ∥ ≤ 4 Υ B , t γ B , t for every t ≥ 0 t \ge0 t ≥ 0 . By Lemma C.1 , we obtain
φ t E [ ∇ f t T z t , τ ∣ F t − 0.5 ] = − φ t ∇ f t T ( I − K t ) B t − 1 ( ∇ f t + e t ) ≤ − φ t γ B , t ∥ ∇ f t ∥ 2 + φ t γ B , t ∥ K t ∥ ∥ ∇ f t ∥ 2 + φ t γ B , t ( 1 + ∥ K t ∥ ) ∥ ∇ f t ∥ ∥ e t ∥ ≤ − φ t 2 γ B , t ∥ ∇ f t ∥ 2 + φ t γ B , t ( 1 + γ B , t 4 Υ B , t ) ∥ ∇ f t ∥ ∥ e t ∥ ≤ − φ t 4 Υ B , t ∥ ∇ f t ∥ 2 + γ B , t φ t γ B , t 2 ( 1 + γ B , t 4 Υ B , t ) 2 ∥ e t ∥ 2 . (30) \begin{aligned}
\varphi_t\mathbb{E}[\nabla f_t^{\mathsf{T}}z_{t,\tau}\mid\mathcal{F}_{t-0.5}]
&=-\varphi_t\nabla f_t^{\mathsf{T}}(I-K_t)B_t^{-1}(\nabla f_t+e_t)\\
&\le-\frac{\varphi_t}{\gamma_{B,t}}\lVert\nabla f_t\rVert^2+\frac{\varphi_t}{\gamma_{B,t}}\lVert K_t\rVert\lVert\nabla f_t\rVert^2+\frac{\varphi_t}{\gamma_{B,t}}(1+\lVert K_t\rVert)\lVert\nabla f_t\rVert\lVert e_t\rVert\\
&\le-\frac{\varphi_t}{2\gamma_{B,t}}\lVert\nabla f_t\rVert^2+\frac{\varphi_t}{\gamma_{B,t}}\left(1+\frac{\gamma_{B,t}}{4\Upsilon_{B,t}}\right)\lVert\nabla f_t\rVert\lVert e_t\rVert\\
&\le-\frac{\varphi_t}{4\Upsilon_{B,t}}\lVert\nabla f_t\rVert^2+\frac{\gamma_{B,t}\varphi_t}{\gamma_{B,t}^2}\left(1+\frac{\gamma_{B,t}}{4\Upsilon_{B,t}}\right)^2\lVert e_t\rVert^2. \tag*{(30)}
\end{aligned} φ t E [ ∇ f t T z t , τ ∣ F t − 0.5 ] = − φ t ∇ f t T ( I − K t ) B t − 1 ( ∇ f t + e t ) ≤ − γ B , t φ t ∥ ∇ f t ∥ 2 + γ B , t φ t ∥ K t ∥ ∥ ∇ f t ∥ 2 + γ B , t φ t ( 1 + ∥ K t ∥) ∥ ∇ f t ∥ ∥ e t ∥ ≤ − 2 γ B , t φ t ∥ ∇ f t ∥ 2 + γ B , t φ t ( 1 + 4 Υ B , t γ B , t ) ∥ ∇ f t ∥ ∥ e t ∥ ≤ − 4 Υ B , t φ t ∥ ∇ f t ∥ 2 + γ B , t 2 γ B , t φ t ( 1 + 4 Υ B , t γ B , t ) 2 ∥ e t ∥ 2 . (30)
where the last inequality follows from the Young’s inequality
1 γ B , t ( 1 + γ B , t 4 Υ B , t ) a b ≤ 1 4 Υ B , t a 2 + γ B , t γ B , t 2 ( 1 + γ B , t 4 Υ B , t ) 2 b 2 . \frac{1}{\gamma_{B,t}}\left(1+\frac{\gamma_{B,t}}{4\Upsilon_{B,t}}\right)ab\le\frac{1}{4\Upsilon_{B,t}}a^2+\frac{\gamma_{B,t}}{\gamma_{B,t}^2}\left(1+\frac{\gamma_{B,t}}{4\Upsilon_{B,t}}\right)^2b^2. γ B , t 1 ( 1 + 4 Υ B , t γ B , t ) ab ≤ 4 Υ B , t 1 a 2 + γ B , t 2 γ B , t ( 1 + 4 Υ B , t γ B , t ) 2 b 2 .
To control the second moment of the search direction, we first establish a uniform bound for ∥ K ~ t ∥ \lVert\widetilde{K}_t\rVert ∥ K t ∥ . Each Z ~ t , j \widetilde{Z}_{t,j} Z t , j is an orthogonal projection, and the definitions of the acceleration parameters together with 1 ≤ ν t ≤ 1 / μ t 1\le\nu_t\le1/\mu_t 1 ≤ ν t ≤ 1/ μ t and μ t ≥ γ S \mu_t\ge\gamma_S μ t ≥ γ S , imply
∥ Z ~ t , j ∥ ≤ 1 , ∥ I − Z ~ t , j ∥ ≤ 1 , 1 ≤ γ t = 1 μ t ν t ≤ 1 μ t ≤ 1 γ S . \lVert\widetilde{Z}_{t,j}\rVert\le1,\qquad\lVert I-\widetilde{Z}_{t,j}\rVert\le1,\qquad1\le\gamma_t=\frac{1}{\sqrt{\mu_t\nu_t}}\le\frac{1}{\sqrt{\mu_t}}\le\frac{1}{\sqrt{\gamma_S}}. ∥ Z t , j ∥ ≤ 1 , ∥ I − Z t , j ∥ ≤ 1 , 1 ≤ γ t = μ t ν t 1 ≤ μ t 1 ≤ γ S 1 .
Consequently,
∥ K ~ t ∥ = ∥ ( I 0 ) C ~ t ( I I ) ∥ ≤ 2 ∥ C ~ t ∥ ≤ 2 ∏ j = 0 τ − 1 ∥ C ~ t , j ∥ ≤ 2 ∏ j = 0 τ − 1 { ∥ I − Z ~ t , j ∥ + ( 1 − α t ) ( 1 − β t ) + ( 1 − α t ) γ t ∥ Z ~ t , j ∥ + ( α t + β t − α t β t ) + α t γ t ∥ Z ~ t , j ∥ } ≤ 2 ∏ j = 0 τ − 1 ( ∥ I − Z ~ t , j ∥ + 1 + γ t ∥ Z ~ t , j ∥ ) ≤ 2 ( 2 + 1 γ S ) τ = : C K . (31) \begin{aligned}
\lVert\widetilde{K}_t\rVert
&=\left\lVert(I\ \ 0)\widetilde{C}_t\binom{I}{I}\right\rVert
\le\sqrt{2}\lVert\widetilde{C}_t\rVert
\le\sqrt{2}\prod_{j=0}^{\tau-1}\lVert\widetilde{C}_{t,j}\rVert\\
&\le\sqrt{2}\prod_{j=0}^{\tau-1}\left\{\lVert I-\widetilde{Z}_{t,j}\rVert+(1-\alpha_t)(1-\beta_t)+(1-\alpha_t)\gamma_t\lVert\widetilde{Z}_{t,j}\rVert+(\alpha_t+\beta_t-\alpha_t\beta_t)+\alpha_t\gamma_t\lVert\widetilde{Z}_{t,j}\rVert\right\}\\
&\le\sqrt{2}\prod_{j=0}^{\tau-1}\left(\lVert I-\widetilde{Z}_{t,j}\rVert+1+\gamma_t\lVert\widetilde{Z}_{t,j}\rVert\right)\\
&\le\sqrt{2}\left(2+\frac{1}{\sqrt{\gamma_S}}\right)^\tau=:C_K. \tag*{(31)}
\end{aligned} ∥ K t ∥ = ( I 0 ) C t ( I I ) ≤ 2 ∥ C t ∥ ≤ 2 j = 0 ∏ τ − 1 ∥ C t , j ∥ ≤ 2 j = 0 ∏ τ − 1 { ∥ I − Z t , j ∥ + ( 1 − α t ) ( 1 − β t ) + ( 1 − α t ) γ t ∥ Z t , j ∥ + ( α t + β t − α t β t ) + α t γ t ∥ Z t , j ∥ } ≤ 2 j = 0 ∏ τ − 1 ( ∥ I − Z t , j ∥ + 1 + γ t ∥ Z t , j ∥ ) ≤ 2 ( 2 + γ S 1 ) τ =: C K . (31)
By Lemma C.1 and (C.8), we immediately have
E [ ∥ z t , τ ∥ 2 ∣ F t − 0.5 ] ≤ 2 ( 1 + C K ) 2 γ B , t 2 ( ∥ ∇ f t ∥ 2 + ∥ e t ∥ 2 ) . (32) \mathbb{E}[\lVert z_{t,\tau}\rVert^2\mid\mathcal{F}_{t-0.5}]
\le\frac{2(1+C_K)^2}{\gamma_{B,t}^2}\left(\lVert\nabla f_t\rVert^2+\lVert e_t\rVert^2\right). \tag*{(32)} E [∥ z t , τ ∥ 2 ∣ F t − 0.5 ] ≤ γ B , t 2 2 ( 1 + C K ) 2 ( ∥ ∇ f t ∥ 2 + ∥ e t ∥ 2 ) . (32)
Substituting (C.7) and (C.9) into (C.6) yields
E [ f t + 1 − f ∗ ∣ F t − 0.5 ] ≤ f t − f ∗ − { 1 4 Υ B , t − Υ H ( 1 + C K ) 2 γ B , t 2 φ t } φ t ∥ ∇ f t ∥ 2 + { γ B , t γ B , t 2 ( 1 + γ B , t 4 Υ B , t ) 2 + Υ H ( 1 + C K ) 2 γ B , t 2 φ t } φ t ∥ e t ∥ 2 . \begin{aligned}
\mathbb{E}[f_{t+1}-f^*\mid\mathcal{F}_{t-0.5}]
\le{}&f_t-f^*-\left\{\frac{1}{4\Upsilon_{B,t}}-\frac{\Upsilon_H(1+C_K)^2}{\gamma_{B,t}^2}\varphi_t\right\}\varphi_t\lVert\nabla f_t\rVert^2\\
&+\left\{\frac{\gamma_{B,t}}{\gamma_{B,t}^2}\left(1+\frac{\gamma_{B,t}}{4\Upsilon_{B,t}}\right)^2+\frac{\Upsilon_H(1+C_K)^2}{\gamma_{B,t}^2}\varphi_t\right\}\varphi_t\lVert e_t\rVert^2.
\end{aligned} E [ f t + 1 − f ∗ ∣ F t − 0.5 ] ≤ f t − f ∗ − { 4 Υ B , t 1 − γ B , t 2 Υ H ( 1 + C K ) 2 φ t } φ t ∥ ∇ f t ∥ 2 + { γ B , t 2 γ B , t ( 1 + 4 Υ B , t γ B , t ) 2 + γ B , t 2 Υ H ( 1 + C K ) 2 φ t } φ t ∥ e t ∥ 2 .
Since ∑ t ( φ t / γ B , t ) 2 < ∞ \sum_t(\varphi_t/\gamma_{B,t})^2<\infty ∑ t ( φ t / γ B , t ) 2 < ∞ and lim sup t → ∞ Υ B , t / γ B , t < ∞ \limsup_{t\to\infty}\Upsilon_{B,t}/\gamma_{B,t}<\infty lim sup t → ∞ Υ B , t / γ B , t < ∞ almost surely, we have φ t / γ B , t → 0 \varphi_t/\gamma_{B,t}\to0 φ t / γ B , t → 0 and φ t Υ B , t / γ B , t 2 → 0 \varphi_t\Upsilon_{B,t}/\gamma_{B,t}^2\to0 φ t Υ B , t / γ B , t 2 → 0 almost surely. Thus, almost surely for all sufficiently large t t t , φ t / γ B , t ≤ 1 \varphi_t/\gamma_{B,t}\le1 φ t / γ B , t ≤ 1 and Υ H ( 1 + C K ) 2 φ t Υ B , t / γ B , t 2 ≤ 1 / 8 \Upsilon_H(1+C_K)^2\varphi_t\Upsilon_{B,t}/\gamma_{B,t}^2\le1/8 Υ H ( 1 + C K ) 2 φ t Υ B , t / γ B , t 2 ≤ 1/8 . For such sufficiently large t t t , we have
E [ f t + 1 − f ∗ ∣ F t − 0.5 ] ≤ f t − f ∗ − φ t 8 Υ B , t ∥ ∇ f t ∥ 2 + { 25 16 + Υ H ( 1 + C K ) 2 } φ t Υ B , t γ B , t 2 ∥ e t ∥ 2 . (33) \begin{aligned}
\mathbb{E}[f_{t+1}-f^*\mid\mathcal{F}_{t-0.5}]
\le{}&f_t-f^*-\frac{\varphi_t}{8\Upsilon_{B,t}}\lVert\nabla f_t\rVert^2\\
&+\left\{\frac{25}{16}+\Upsilon_H(1+C_K)^2\right\}\frac{\varphi_t\Upsilon_{B,t}}{\gamma_{B,t}^2}\lVert e_t\rVert^2. \tag*{(33)}
\end{aligned} E [ f t + 1 − f ∗ ∣ F t − 0.5 ] ≤ f t − f ∗ − 8 Υ B , t φ t ∥ ∇ f t ∥ 2 + { 16 25 + Υ H ( 1 + C K ) 2 } γ B , t 2 φ t Υ B , t ∥ e t ∥ 2 . (33)
We next control the moving-average error e t e_t e t , which admits the recursion
e t + 1 = ( 1 − η t + 1 ) ( e t + ∇ f t − ∇ f t + 1 ) + η t + 1 ( g t + 1 − ∇ f t + 1 ) . (34) e_{t+1}=(1-\eta_{t+1})(e_t+\nabla f_t-\nabla f_{t+1})+\eta_{t+1}(g_{t+1}-\nabla f_{t+1}). \tag*{(34)} e t + 1 = ( 1 − η t + 1 ) ( e t + ∇ f t − ∇ f t + 1 ) + η t + 1 ( g t + 1 − ∇ f t + 1 ) . (34)
Conditional on F t \mathcal{F}_t F t , the first term on the right-hand side is fixed. By Assumption 3.2, we have
E [ ∥ e t + 1 ∥ 2 ∣ F t ] = ( 1 − η t + 1 ) 2 ∥ e t + ∇ f t − ∇ f t + 1 ∥ 2 + η t + 1 2 E [ ∥ g t + 1 − ∇ f t + 1 ∥ 2 ∣ F t ] . (35) \mathbb{E}[\lVert e_{t+1}\rVert^2\mid\mathcal{F}_t]=(1-\eta_{t+1})^2\lVert e_t+\nabla f_t-\nabla f_{t+1}\rVert^2+\eta_{t+1}^2\mathbb{E}[\lVert g_{t+1}-\nabla f_{t+1}\rVert^2\mid\mathcal{F}_t]. \tag*{(35)} E [∥ e t + 1 ∥ 2 ∣ F t ] = ( 1 − η t + 1 ) 2 ∥ e t + ∇ f t − ∇ f t + 1 ∥ 2 + η t + 1 2 E [∥ g t + 1 − ∇ f t + 1 ∥ 2 ∣ F t ] . (35)
For η t + 1 ∈ ( 0 , 1 ) \eta_{t+1}\in(0,1) η t + 1 ∈ ( 0 , 1 ) , applying Young’s inequality to the cross term gives
( 1 − η t + 1 ) 2 ∥ e t + ∇ f t − ∇ f t + 1 ∥ 2 = ( 1 − η t + 1 ) 2 { ∥ e t ∥ 2 + 2 e t ⊤ ( ∇ f t − ∇ f t + 1 ) + ∥ ∇ f t − ∇ f t + 1 ∥ 2 } ≤ ( 1 − η t + 1 ) 2 { ( 1 + η t + 1 1 − η t + 1 ) ∥ e t ∥ 2 + ( 1 + 1 − η t + 1 η t + 1 ) ∥ ∇ f t − ∇ f t + 1 ∥ 2 } = ( 1 − η t + 1 ) ∥ e t ∥ 2 + ( 1 − η t + 1 ) 2 η t + 1 ∥ ∇ f t − ∇ f t + 1 ∥ 2 ≤ ( 1 − η t + 1 ) ∥ e t ∥ 2 + η t + 1 − 1 ∥ ∇ f t − ∇ f t + 1 ∥ 2 . \begin{aligned}
(1-\eta_{t+1})^2\lVert e_t+\nabla f_t-\nabla f_{t+1}\rVert^2
&=(1-\eta_{t+1})^2\{\lVert e_t\rVert^2+2e_t^\top(\nabla f_t-\nabla f_{t+1})+\lVert\nabla f_t-\nabla f_{t+1}\rVert^2\}\\
&\le(1-\eta_{t+1})^2\left\{\left(1+\frac{\eta_{t+1}}{1-\eta_{t+1}}\right)\lVert e_t\rVert^2+\left(1+\frac{1-\eta_{t+1}}{\eta_{t+1}}\right)\lVert\nabla f_t-\nabla f_{t+1}\rVert^2\right\}\\
&=(1-\eta_{t+1})\lVert e_t\rVert^2+\frac{(1-\eta_{t+1})^2}{\eta_{t+1}}\lVert\nabla f_t-\nabla f_{t+1}\rVert^2\\
&\le(1-\eta_{t+1})\lVert e_t\rVert^2+\eta_{t+1}^{-1}\lVert\nabla f_t-\nabla f_{t+1}\rVert^2.
\end{aligned} ( 1 − η t + 1 ) 2 ∥ e t + ∇ f t − ∇ f t + 1 ∥ 2 = ( 1 − η t + 1 ) 2 {∥ e t ∥ 2 + 2 e t ⊤ ( ∇ f t − ∇ f t + 1 ) + ∥ ∇ f t − ∇ f t + 1 ∥ 2 } ≤ ( 1 − η t + 1 ) 2 { ( 1 + 1 − η t + 1 η t + 1 ) ∥ e t ∥ 2 + ( 1 + η t + 1 1 − η t + 1 ) ∥ ∇ f t − ∇ f t + 1 ∥ 2 } = ( 1 − η t + 1 ) ∥ e t ∥ 2 + η t + 1 ( 1 − η t + 1 ) 2 ∥ ∇ f t − ∇ f t + 1 ∥ 2 ≤ ( 1 − η t + 1 ) ∥ e t ∥ 2 + η t + 1 − 1 ∥ ∇ f t − ∇ f t + 1 ∥ 2 .
When η t + 1 = 1 \eta_{t+1}=1 η t + 1 = 1 , the left-hand side is zero, so the same inequality holds trivially. Moreover, the Υ H \Upsilon_H Υ H -Lipschitz continuity of ∇ f ( x ) \nabla f(x) ∇ f ( x ) and the outer update imply
∥ ∇ f t + 1 − ∇ f t ∥ ≤ Υ H ∥ x t + 1 − x t ∥ = Υ H ϕ t ∥ z t , τ ∥ . \lVert\nabla f_{t+1}-\nabla f_t\rVert\le\Upsilon_H\lVert x_{t+1}-x_t\rVert=\Upsilon_H\phi_t\lVert z_{t,\tau}\rVert. ∥ ∇ f t + 1 − ∇ f t ∥ ≤ Υ H ∥ x t + 1 − x t ∥ = Υ H ϕ t ∥ z t , τ ∥ .
Applying these inequalities to (C.12), taking the conditional expectation with respect to F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 , and using the tower property and (C.9), we obtain
E [ ∥ e t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ ( 1 − η t + 1 ) ∥ e t ∥ 2 + 2 Υ H 2 ( 1 + C K ) 2 ϕ t 2 γ B , t 2 η t + 1 ( ∥ ∇ f t ∥ 2 + ∥ e t ∥ 2 ) + η t + 1 2 E [ ∥ g t + 1 − ∇ f t + 1 ∥ 2 ∣ F t − 0.5 ] . \mathbb{E}[\lVert e_{t+1}\rVert^2\mid\mathcal{F}_{t-0.5}]
\le(1-\eta_{t+1})\lVert e_t\rVert^2+\frac{2\Upsilon_H^2(1+C_K)^2\phi_t^2}{\gamma_{B,t}^2\eta_{t+1}}(\lVert\nabla f_t\rVert^2+\lVert e_t\rVert^2)+\eta_{t+1}^2\mathbb{E}[\lVert g_{t+1}-\nabla f_{t+1}\rVert^2\mid\mathcal{F}_{t-0.5}]. E [∥ e t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ ( 1 − η t + 1 ) ∥ e t ∥ 2 + γ B , t 2 η t + 1 2 Υ H 2 ( 1 + C K ) 2 ϕ t 2 (∥ ∇ f t ∥ 2 + ∥ e t ∥ 2 ) + η t + 1 2 E [∥ g t + 1 − ∇ f t + 1 ∥ 2 ∣ F t − 0.5 ] .
By Assumption 3.2 with q g = 2 q_g=2 q g = 2 , we have
E [ ∥ g t + 1 − ∇ f t + 1 ∥ 2 ∣ F t ] ≤ C g , 1 ∥ x t + 1 − x ⋆ ∥ 2 + C g , 2 . \mathbb{E}[\lVert g_{t+1}-\nabla f_{t+1}\rVert^2\mid\mathcal{F}_t]\le C_{g,1}\lVert x_{t+1}-x^\star\rVert^2+C_{g,2}. E [∥ g t + 1 − ∇ f t + 1 ∥ 2 ∣ F t ] ≤ C g , 1 ∥ x t + 1 − x ⋆ ∥ 2 + C g , 2 .
Since ϕ t / γ B , t ≤ 1 \phi_t/\gamma_{B,t}\le1 ϕ t / γ B , t ≤ 1 almost surely for all sufficiently large t t t , (C.4), (C.5), and (C.9) yield
E [ ∥ x t + 1 − x ⋆ ∥ 2 ∣ F t − 0.5 ] ≤ 2 ∥ x t − x ⋆ ∥ 2 + 2 ϕ t 2 E [ ∥ z t , τ ∥ 2 ∣ F t − 0.5 ] ≤ { 4 γ H + 8 Υ H ( 1 + C K ) 2 } ( f t − f ⋆ ) + 4 ( 1 + C K ) 2 ∥ e t ∥ 2 . \mathbb{E}[\lVert x_{t+1}-x^\star\rVert^2\mid\mathcal{F}_{t-0.5}]
\le2\lVert x_t-x^\star\rVert^2+2\phi_t^2\mathbb{E}[\lVert z_{t,\tau}\rVert^2\mid\mathcal{F}_{t-0.5}]
\le\left\{\frac{4}{\gamma_H}+8\Upsilon_H(1+C_K)^2\right\}(f_t-f^\star)+4(1+C_K)^2\lVert e_t\rVert^2. E [∥ x t + 1 − x ⋆ ∥ 2 ∣ F t − 0.5 ] ≤ 2 ∥ x t − x ⋆ ∥ 2 + 2 ϕ t 2 E [∥ z t , τ ∥ 2 ∣ F t − 0.5 ] ≤ { γ H 4 + 8 Υ H ( 1 + C K ) 2 } ( f t − f ⋆ ) + 4 ( 1 + C K ) 2 ∥ e t ∥ 2 .
Therefore, by the tower property,
E [ ∥ g t + 1 − ∇ f t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ C g , 1 { 4 γ H + 8 Υ H ( 1 + C K ) 2 } ( f t − f ⋆ ) + 4 C g , 1 ( 1 + C K ) 2 ∥ e t ∥ 2 + C g , 2 . \mathbb{E}[\lVert g_{t+1}-\nabla f_{t+1}\rVert^2\mid\mathcal{F}_{t-0.5}]
\le C_{g,1}\left\{\frac{4}{\gamma_H}+8\Upsilon_H(1+C_K)^2\right\}(f_t-f^\star)+4C_{g,1}(1+C_K)^2\lVert e_t\rVert^2+C_{g,2}. E [∥ g t + 1 − ∇ f t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ C g , 1 { γ H 4 + 8 Υ H ( 1 + C K ) 2 } ( f t − f ⋆ ) + 4 C g , 1 ( 1 + C K ) 2 ∥ e t ∥ 2 + C g , 2 .
Substituting this bound into the preliminary tracking recursion yields
E [ ∥ e t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ ( 1 − η t + 1 ) ∥ e t ∥ 2 + 2 Υ H 2 ( 1 + C K ) 2 ϕ t 2 γ B , t 2 η t + 1 ( ∥ ∇ f t ∥ 2 + ∥ e t ∥ 2 ) + C g , 1 { 4 γ H + 8 Υ H ( 1 + C K ) 2 } η t + 1 2 ( f t − f ⋆ ) + 4 C g , 1 ( 1 + C K ) 2 η t + 1 2 ∥ e t ∥ 2 + C g , 2 η t + 1 2 . (36) \begin{aligned}
\mathbb{E}[\lVert e_{t+1}\rVert^2\mid\mathcal{F}_{t-0.5}]
&\le(1-\eta_{t+1})\lVert e_t\rVert^2+\frac{2\Upsilon_H^2(1+C_K)^2\phi_t^2}{\gamma_{B,t}^2\eta_{t+1}}(\lVert\nabla f_t\rVert^2+\lVert e_t\rVert^2)\\
&\quad+C_{g,1}\left\{\frac{4}{\gamma_H}+8\Upsilon_H(1+C_K)^2\right\}\eta_{t+1}^2(f_t-f^\star)+4C_{g,1}(1+C_K)^2\eta_{t+1}^2\lVert e_t\rVert^2+C_{g,2}\eta_{t+1}^2. \tag*{(36)}
\end{aligned} E [∥ e t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ ( 1 − η t + 1 ) ∥ e t ∥ 2 + γ B , t 2 η t + 1 2 Υ H 2 ( 1 + C K ) 2 ϕ t 2 (∥ ∇ f t ∥ 2 + ∥ e t ∥ 2 ) + C g , 1 { γ H 4 + 8 Υ H ( 1 + C K ) 2 } η t + 1 2 ( f t − f ⋆ ) + 4 C g , 1 ( 1 + C K ) 2 η t + 1 2 ∥ e t ∥ 2 + C g , 2 η t + 1 2 . (36)
Since η t / η t + 1 → 1 \eta_t/\eta_{t+1}\to1 η t / η t + 1 → 1 , the theorem’s condition gives us ϕ t / ( γ B , t η t + 1 ) → 0 \phi_t/(\gamma_{B,t}\eta_{t+1})\to0 ϕ t / ( γ B , t η t + 1 ) → 0 almost surely. Together with lim sup t → ∞ Υ B , t / γ B , t < ∞ \limsup_{t\to\infty}\Upsilon_{B,t}/\gamma_{B,t}<\infty lim sup t → ∞ Υ B , t / γ B , t < ∞ , this implies that almost surely for all sufficiently large t t t ,
2 Υ H 2 ( 1 + C K ) 2 ϕ t Υ B , t γ B , t 2 η t + 1 ≤ 1 16 , 2\Upsilon_H^2(1+C_K)^2\frac{\phi_t\Upsilon_{B,t}}{\gamma_{B,t}^2\eta_{t+1}}\le\frac{1}{16}, 2 Υ H 2 ( 1 + C K ) 2 γ B , t 2 η t + 1 ϕ t Υ B , t ≤ 16 1 ,
{ 25 16 + Υ H ( 1 + C K ) 2 } ϕ t Υ B , t γ B , t 2 η t + 1 ≤ 1 4 , \left\{\frac{25}{16}+\Upsilon_H(1+C_K)^2\right\}\frac{\phi_t\Upsilon_{B,t}}{\gamma_{B,t}^2\eta_{t+1}}\le\frac{1}{4}, { 16 25 + Υ H ( 1 + C K ) 2 } γ B , t 2 η t + 1 ϕ t Υ B , t ≤ 4 1 ,
2 Υ H 2 ( 1 + C K ) 2 ( ϕ t γ B , t η t + 1 ) 2 ≤ 1 4 . 2\Upsilon_H^2(1+C_K)^2\left(\frac{\phi_t}{\gamma_{B,t}\eta_{t+1}}\right)^2\le\frac{1}{4}. 2 Υ H 2 ( 1 + C K ) 2 ( γ B , t η t + 1 ϕ t ) 2 ≤ 4 1 .
Adding (C.10) and (C.13) and using these three inequalities gives us
E [ f t + 1 − f ⋆ + ∥ e t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ [ 1 + C g , 1 { 4 γ H + 8 γ H ( 1 + C K ) 2 + 4 ( 1 + C K ) 2 } η t + 1 2 ] ( f t − f ⋆ + ∥ e t ∥ 2 ) − φ t 16 Υ B , t ∥ ∇ f t ∥ 2 − η t + 1 2 ∥ e t ∥ 2 + C g , 2 η t + 1 2 . (37) \mathbb{E}[f_{t+1}-f^\star+\lVert e_{t+1}\rVert^2\mid\mathcal{F}_{t-0.5}]
\leq\left[1+C_{g,1}\left\{\frac{4}{\gamma_H}+8\gamma_H(1+C_K)^2+4(1+C_K)^2\right\}\eta_{t+1}^2\right](f_t-f^\star+\lVert e_t\rVert^2)-\frac{\varphi_t}{16\Upsilon_{B,t}}\lVert\nabla f_t\rVert^2-\frac{\eta_{t+1}}{2}\lVert e_t\rVert^2+C_{g,2}\eta_{t+1}^2. \tag*{(37)} E [ f t + 1 − f ⋆ + ∥ e t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ [ 1 + C g , 1 { γ H 4 + 8 γ H ( 1 + C K ) 2 + 4 ( 1 + C K ) 2 } η t + 1 2 ] ( f t − f ⋆ + ∥ e t ∥ 2 ) − 16 Υ B , t φ t ∥ ∇ f t ∥ 2 − 2 η t + 1 ∥ e t ∥ 2 + C g , 2 η t + 1 2 . (37)
Since 2 η > 1 2\eta>1 2 η > 1 , we have ∑ t = 0 ∞ η t + 1 2 < ∞ \sum_{t=0}^{\infty}\eta_{t+1}^2<\infty ∑ t = 0 ∞ η t + 1 2 < ∞ . Let L t : = f t − f ⋆ + ∥ e t ∥ 2 \mathcal{L}_t:=f_t-f^\star+\lVert e_t\rVert^2 L t := f t − f ⋆ + ∥ e t ∥ 2 . For each integer m ≥ 1 m\geq1 m ≥ 1 , choose a deterministic δ m ∈ ( 0 , 1 ] \delta_m\in(0,1] δ m ∈ ( 0 , 1 ] sufficiently small that (C.10) and the three inequalities preceding (C.14) hold whenever Υ B , t / γ B , t ≤ m \Upsilon_{B,t}/\gamma_{B,t}\leq m Υ B , t / γ B , t ≤ m and φ t / ( γ B , t η t + 1 ) ≤ δ m \varphi_t/(\gamma_{B,t}\eta_{t+1})\leq\delta_m φ t / ( γ B , t η t + 1 ) ≤ δ m . Such a choice is possible since η t + 1 ≤ 1 \eta_{t+1}\leq1 η t + 1 ≤ 1 and these two bounds imply
φ t γ B , t ≤ δ m , φ t Υ B , t γ B , t 2 ≤ m δ m , φ t Υ B , t γ B , t 2 η t + 1 ≤ m δ m , ( φ t γ B , t η t + 1 ) 2 ≤ δ m 2 . \frac{\varphi_t}{\gamma_{B,t}}\leq\delta_m,\qquad\frac{\varphi_t\Upsilon_{B,t}}{\gamma_{B,t}^2}\leq m\delta_m,\qquad\frac{\varphi_t\Upsilon_{B,t}}{\gamma_{B,t}^2\eta_{t+1}}\leq m\delta_m,\qquad\left(\frac{\varphi_t}{\gamma_{B,t}\eta_{t+1}}\right)^2\leq\delta_m^2. γ B , t φ t ≤ δ m , γ B , t 2 φ t Υ B , t ≤ m δ m , γ B , t 2 η t + 1 φ t Υ B , t ≤ m δ m , ( γ B , t η t + 1 φ t ) 2 ≤ δ m 2 .
For any k ≥ 0 k\geq0 k ≥ 0 , we define
σ k , m : = inf { t ≥ k : Υ B , t γ B , t > m O R φ t γ B , t η t + 1 > δ m } , inf ∅ = ∞ . (38) \sigma_{k,m}:=\inf\left\{t\geq k:\frac{\Upsilon_{B,t}}{\gamma_{B,t}}>m\ \mathrm{OR}\ \frac{\varphi_t}{\gamma_{B,t}\eta_{t+1}}>\delta_m\right\},\qquad\inf\varnothing=\infty. \tag*{(38)} σ k , m := inf { t ≥ k : γ B , t Υ B , t > m OR γ B , t η t + 1 φ t > δ m } , inf ∅ = ∞. (38)
Since Υ B , t / γ B , t \Upsilon_{B,t}/\gamma_{B,t} Υ B , t / γ B , t and φ t / ( γ B , t η t + 1 ) \varphi_t/(\gamma_{B,t}\eta_{t+1}) φ t / ( γ B , t η t + 1 ) are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, σ k , m \sigma_{k,m} σ k , m is a stopping time and { σ k , m > t } ∈ F t − 1 \{\sigma_{k,m}>t\}\in\mathcal{F}_{t-1} { σ k , m > t } ∈ F t − 1 . Hence, (C.14) holds on { σ k , m > t } \{\sigma_{k,m}>t\} { σ k , m > t } . For any ℓ ≥ 1 \ell\geq1 ℓ ≥ 1 , t ≥ k t\geq k t ≥ k , the stopped process satisfies
E [ L ( t + 1 ) ∧ σ k , m 1 { L k ≤ ℓ } ∣ F t − 0.5 ] ≤ [ 1 + C g , 1 { 4 γ H + 8 γ H ( 1 + C K ) 2 + 4 ( 1 + C K ) 2 } η t + 1 2 ] L t ∧ σ k , m 1 { L k ≤ ℓ } − { φ t 16 Υ B , t ∥ ∇ f t ∥ 2 + η t + 1 2 ∥ e t ∥ 2 } 1 { σ k , m > t } 1 { L k ≤ ℓ } + C g , 2 η t + 1 2 1 { L k ≤ ℓ } . (39) \mathbb{E}[\mathcal{L}_{(t+1)\wedge\sigma_{k,m}}\mathbf{1}_{\{\mathcal{L}_k\leq\ell\}}\mid\mathcal{F}_{t-0.5}]
\leq\left[1+C_{g,1}\left\{\frac{4}{\gamma_H}+8\gamma_H(1+C_K)^2+4(1+C_K)^2\right\}\eta_{t+1}^2\right]\mathcal{L}_{t\wedge\sigma_{k,m}}\mathbf{1}_{\{\mathcal{L}_k\leq\ell\}}
-\left\{\frac{\varphi_t}{16\Upsilon_{B,t}}\lVert\nabla f_t\rVert^2+\frac{\eta_{t+1}}{2}\lVert e_t\rVert^2\right\}\mathbf{1}_{\{\sigma_{k,m}>t\}}\mathbf{1}_{\{\mathcal{L}_k\leq\ell\}}+C_{g,2}\eta_{t+1}^2\mathbf{1}_{\{\mathcal{L}_k\leq\ell\}}. \tag*{(39)} E [ L ( t + 1 ) ∧ σ k , m 1 { L k ≤ ℓ } ∣ F t − 0.5 ] ≤ [ 1 + C g , 1 { γ H 4 + 8 γ H ( 1 + C K ) 2 + 4 ( 1 + C K ) 2 } η t + 1 2 ] L t ∧ σ k , m 1 { L k ≤ ℓ } − { 16 Υ B , t φ t ∥ ∇ f t ∥ 2 + 2 η t + 1 ∥ e t ∥ 2 } 1 { σ k , m > t } 1 { L k ≤ ℓ } + C g , 2 η t + 1 2 1 { L k ≤ ℓ } . (39)
where a ∧ b : = min { a , b } a\wedge b:=\min\{a,b\} a ∧ b := min { a , b } . Since L k 1 { L k ≤ ℓ } ≤ ℓ \mathcal{L}_k\mathbf{1}_{\{\mathcal{L}_k\leq\ell\}}\leq\ell L k 1 { L k ≤ ℓ } ≤ ℓ , applying the Robbins–Siegmund theorem [10 Theorem 1.3.12] to (C.16) gives lim t → ∞ L t ∧ σ k , m 1 { L k ≤ ℓ } ∈ [ 0 , ∞ ) \lim_{t\to\infty}\mathcal{L}_{t\wedge\sigma_{k,m}}\mathbf{1}_{\{\mathcal{L}_k\leq\ell\}}\in[0,\infty) lim t → ∞ L t ∧ σ k , m 1 { L k ≤ ℓ } ∈ [ 0 , ∞ ) ,
∑ t = k ∞ φ t Υ B , t ∥ ∇ f t ∥ 2 1 { σ k , m > t } 1 { L k ≤ ℓ } < ∞ , ∑ t = k ∞ η t + 1 ∥ e t ∥ 2 1 { σ k , m > t } 1 { L k ≤ ℓ } < ∞ \sum_{t=k}^{\infty}\frac{\varphi_t}{\Upsilon_{B,t}}\lVert\nabla f_t\rVert^2\mathbf{1}_{\{\sigma_{k,m}>t\}}\mathbf{1}_{\{\mathcal{L}_k\leq\ell\}}<\infty,\qquad
\sum_{t=k}^{\infty}\eta_{t+1}\lVert e_t\rVert^2\mathbf{1}_{\{\sigma_{k,m}>t\}}\mathbf{1}_{\{\mathcal{L}_k\leq\ell\}}<\infty t = k ∑ ∞ Υ B , t φ t ∥ ∇ f t ∥ 2 1 { σ k , m > t } 1 { L k ≤ ℓ } < ∞ , t = k ∑ ∞ η t + 1 ∥ e t ∥ 2 1 { σ k , m > t } 1 { L k ≤ ℓ } < ∞
almost surely. By Assumption 3.1, lim sup t → ∞ Υ B , t / γ B , t < ∞ \limsup_{t\to\infty}\Upsilon_{B,t}/\gamma_{B,t}<\infty lim sup t → ∞ Υ B , t / γ B , t < ∞ almost surely. For each such realization, choose an integer m > lim sup t → ∞ Υ B , t / γ B , t m>\limsup_{t\to\infty}\Upsilon_{B,t}/\gamma_{B,t} m > lim sup t → ∞ Υ B , t / γ B , t . Since φ t / ( γ B , t η t + 1 ) → 0 \varphi_t/(\gamma_{B,t}\eta_{t+1})\to0 φ t / ( γ B , t η t + 1 ) → 0 and δ m > 0 \delta_m>0 δ m > 0 , we can then choose k k k such that σ k , m = ∞ \sigma_{k,m}=\infty σ k , m = ∞ . Finally, L k < ∞ \mathcal{L}_k<\infty L k < ∞ allows us to choose an integer ℓ ≥ L k \ell\geq\mathcal{L}_k ℓ ≥ L k . As a result, we have
P ( ⋃ k ≥ 0 ⋃ m , ℓ ≥ 1 { σ k , m = ∞ , L k ≤ ℓ } ) = 1. \mathbb{P}\left(\bigcup_{k\geq0}\bigcup_{m,\ell\geq1}\{\sigma_{k,m}=\infty,\mathcal{L}_k\leq\ell\}\right)=1. P k ≥ 0 ⋃ m , ℓ ≥ 1 ⋃ { σ k , m = ∞ , L k ≤ ℓ } = 1.
Since ( k , m , ℓ ) (k,m,\ell) ( k , m , ℓ ) ranges over a countable set, the three conclusions obtained from the Robbins–Siegmund theorem hold almost surely for all ( k , m , ℓ ) (k,m,\ell) ( k , m , ℓ ) . For almost every realization, choose ( k , m , ℓ ) (k,m,\ell) ( k , m , ℓ ) such that σ k , m = ∞ \sigma_{k,m}=\infty σ k , m = ∞ and L k ≤ ℓ \mathcal{L}_k\leq\ell L k ≤ ℓ . For this choice, t ∧ σ k , m = t t\wedge\sigma_{k,m}=t t ∧ σ k , m = t and 1 { σ k , m > t } = 1 { L k ≤ ℓ } = 1 \mathbf{1}_{\{\sigma_{k,m}>t\}}=\mathbf{1}_{\{\mathcal{L}_k\leq\ell\}}=1 1 { σ k , m > t } = 1 { L k ≤ ℓ } = 1 for every t ≥ k t\geq k t ≥ k . Therefore, L t \mathcal{L}_t L t converges to a finite random variable, and both sums over t ≥ k t\geq k t ≥ k are finite. Adding the finitely many terms with t < k t<k t < k , we obtain
∑ t = 0 ∞ φ t Υ B , t ∥ ∇ f t ∥ 2 < ∞ , ∑ t = 0 ∞ η t + 1 ∥ e t ∥ 2 < ∞ (40) \sum_{t=0}^{\infty}\frac{\varphi_t}{\Upsilon_{B,t}}\lVert\nabla f_t\rVert^2<\infty,\qquad
\sum_{t=0}^{\infty}\eta_{t+1}\lVert e_t\rVert^2<\infty \tag*{(40)} t = 0 ∑ ∞ Υ B , t φ t ∥ ∇ f t ∥ 2 < ∞ , t = 0 ∑ ∞ η t + 1 ∥ e t ∥ 2 < ∞ (40)
almost surely. In particular, f t − f ⋆ f_t-f^\star f t − f ⋆ and ∥ e t ∥ 2 \lVert e_t\rVert^2 ∥ e t ∥ 2 are bounded almost surely. Moreover,
2 γ H 2 ( 1 + C K ) 2 ( φ t γ B , t η t + 1 ) 2 + 4 C g , 1 ( 1 + C K ) 2 η t + 1 → 0. 2\gamma_H^2(1+C_K)^2\left(\frac{\varphi_t}{\gamma_{B,t}\eta_{t+1}}\right)^2+4C_{g,1}(1+C_K)^2\eta_{t+1}\to0. 2 γ H 2 ( 1 + C K ) 2 ( γ B , t η t + 1 φ t ) 2 + 4 C g , 1 ( 1 + C K ) 2 η t + 1 → 0.
Hence, for all sufficiently large t t t , (C.13) implies
E [ ∥ e t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ ( 1 − η t + 1 2 ) ∥ e t ∥ 2 + 2 Υ H 2 ( 1 + C K ) 2 φ t 2 γ B , t η t + 1 ∥ ∇ f t ∥ 2 + C g , 1 { 4 γ H + 8 Υ H ( 1 + C K ) 2 } η t + 1 2 ( f t − f ⋆ ) + C g , 2 η t + 1 2 . (41) \mathbb{E}[\lVert e_{t+1}\rVert^2 \mid\mathcal{F}_{t-0.5}] \le\left(1-\frac{\eta_{t+1}}{2}\right)\lVert e_t\rVert^2 + \frac{2\Upsilon_H^2(1+C_K)^2\varphi_t^2}{\gamma_{B,t}\eta_{t+1}}\lVert\nabla f_t\rVert^2 + C_{g,1}\left\{\frac{4}{\gamma_H}+8\Upsilon_H(1+C_K)^2\right\}\eta_{t+1}^2(f_t-f^\star)+C_{g,2}\eta_{t+1}^2. \tag*{(41)} E [∥ e t + 1 ∥ 2 ∣ F t − 0.5 ] ≤ ( 1 − 2 η t + 1 ) ∥ e t ∥ 2 + γ B , t η t + 1 2 Υ H 2 ( 1 + C K ) 2 φ t 2 ∥ ∇ f t ∥ 2 + C g , 1 { γ H 4 + 8 Υ H ( 1 + C K ) 2 } η t + 1 2 ( f t − f ⋆ ) + C g , 2 η t + 1 2 . (41)
The positive terms on the right-hand side of (C.18) are almost surely summable. Indeed, by (C.17),
2 Υ H 2 ( 1 + C K ) 2 ∑ t = 0 ∞ φ t 2 γ B , t 2 η t + 1 ∥ ∇ f t ∥ 2 = 2 Υ H 2 ( 1 + C K ) 2 ∑ t = 0 ∞ ( γ B , t γ B , t φ t γ B , t η t + 1 ) ( φ t γ B , t ∥ ∇ f t ∥ 2 ) < ∞ 2\Upsilon_H^2(1+C_K)^2\sum_{t=0}^{\infty}\frac{\varphi_t^2}{\gamma_{B,t}^2\eta_{t+1}}\lVert\nabla f_t\rVert^2
=2\Upsilon_H^2(1+C_K)^2\sum_{t=0}^{\infty}\left(\frac{\gamma_{B,t}}{\gamma_{B,t}}\frac{\varphi_t}{\gamma_{B,t}\eta_{t+1}}\right)\left(\frac{\varphi_t}{\gamma_{B,t}}\lVert\nabla f_t\rVert^2\right)<\infty 2 Υ H 2 ( 1 + C K ) 2 t = 0 ∑ ∞ γ B , t 2 η t + 1 φ t 2 ∥ ∇ f t ∥ 2 = 2 Υ H 2 ( 1 + C K ) 2 t = 0 ∑ ∞ ( γ B , t γ B , t γ B , t η t + 1 φ t ) ( γ B , t φ t ∥ ∇ f t ∥ 2 ) < ∞
almost surely. Furthermore, the almost-sure boundedness of f t − f ⋆ f_t-f^\star f t − f ⋆ and ∑ t + 1 η t + 1 2 < ∞ \sum_{t+1}\eta_{t+1}^2<\infty ∑ t + 1 η t + 1 2 < ∞ also imply
∑ t = 0 ∞ η t + 1 2 [ C g , 1 { 4 γ H + 8 Υ H ( 1 + C K ) 2 } ( f t − f ⋆ ) + C g , 2 ] < ∞ \sum_{t=0}^{\infty}\eta_{t+1}^2\left[C_{g,1}\left\{\frac{4}{\gamma_H}+8\Upsilon_H(1+C_K)^2\right\}(f_t-f^\star)+C_{g,2}\right]<\infty t = 0 ∑ ∞ η t + 1 2 [ C g , 1 { γ H 4 + 8 Υ H ( 1 + C K ) 2 } ( f t − f ⋆ ) + C g , 2 ] < ∞
almost surely. Applying Robbins–Siegmund again to the same stopped processes and then removing the localization, we conclude that ∥ e t ∥ 2 \lVert e_t\rVert^2 ∥ e t ∥ 2 converges almost surely; the positive terms are summable by the preceding two bounds. Moreover, ∑ t = 0 ∞ η t + 1 = ∞ \sum_{t=0}^{\infty}\eta_{t+1}=\infty ∑ t = 0 ∞ η t + 1 = ∞ . Otherwise, φ t / ( γ B , t η t + 1 ) → 0 \varphi_t/(\gamma_{B,t}\eta_{t+1})\to0 φ t / ( γ B , t η t + 1 ) → 0 would give φ t / γ B , t ≤ η t + 1 \varphi_t/\gamma_{B,t}\leq\eta_{t+1} φ t / γ B , t ≤ η t + 1 eventually, contradicting ∑ t φ t / γ B , t = ∞ \sum_t\varphi_t/\gamma_{B,t}=\infty ∑ t φ t / γ B , t = ∞ in Assumption 3.1. Together with ∑ t = 0 ∞ η t + 1 ∥ e t ∥ 2 < ∞ \sum_{t=0}^{\infty}\eta_{t+1}\lVert e_t\rVert^2<\infty ∑ t = 0 ∞ η t + 1 ∥ e t ∥ 2 < ∞ , this gives us
e t = m t − ∇ f t → a . s . 0. e_t=m_t-\nabla f_t \xrightarrow{\mathrm{a.s.}} 0. e t = m t − ∇ f t a.s. 0.
Finally, the almost-sure convergence of f t − f ⋆ + ∥ e t ∥ 2 f_t-f^\star+\lVert e_t\rVert^2 f t − f ⋆ + ∥ e t ∥ 2 and ∥ e t ∥ 2 → 0 \lVert e_t\rVert^2\to0 ∥ e t ∥ 2 → 0 imply that f t − f ⋆ f_t-f^\star f t − f ⋆ also converges almost surely. Since ∑ t = 0 ∞ φ t / Υ B , t = ∞ \sum_{t=0}^{\infty}\varphi_t/\Upsilon_{B,t}=\infty ∑ t = 0 ∞ φ t / Υ B , t = ∞ and ∑ t = 0 ∞ ( φ t / Υ B , t ) ∥ ∇ f t ∥ 2 < ∞ \sum_{t=0}^{\infty}(\varphi_t/\Upsilon_{B,t})\lVert\nabla f_t\rVert^2<\infty ∑ t = 0 ∞ ( φ t / Υ B , t ) ∥ ∇ f t ∥ 2 < ∞ , we have
lim inf t → ∞ ∥ ∇ f t ∥ = 0 \liminf_{t\to\infty}\lVert\nabla f_t\rVert=0 t → ∞ lim inf ∥ ∇ f t ∥ = 0
almost surely. It follows from (C.4) that lim inf t → ∞ ( f t − f ⋆ ) = 0 \liminf_{t\to\infty}(f_t-f^\star)=0 lim inf t → ∞ ( f t − f ⋆ ) = 0 almost surely. Given that f t − f ⋆ f_t-f^\star f t − f ⋆ converges almost surely, we conclude that f t − f ⋆ → 0 f_t-f^\star\to0 f t − f ⋆ → 0 almost surely. Again, we can apply the lower bound in (C.4) and obtain
x t → a . s . x ⋆ , x_t\xrightarrow{\mathrm{a.s.}}x^\star, x t a.s. x ⋆ ,
which completes the proof.
Proof of Lemma C.1
Fix an outer iteration t t t . In § 2 , we temporarily suppress the outer iteration index. Below, we write { z t , j , v t , j , y t , j , w t , j } j = 0 \{z_{t,j},v_{t,j},y_{t,j},w_{t,j}\}_{j=0} { z t , j , v t , j , y t , j , w t , j } j = 0 to make the dependence on t t t explicit. Recall that B t Δ x t = − m t B_t\Delta x_t=-m_t B t Δ x t = − m t . It follows from the definition of w t , j w_{t,j} w t , j in Algorithm 1 that
w t , j = B t S t , j ( S t , j T B t 2 S t , j ) † S t , j T B t ( y t , j − Δ x t ) = Z ~ t , j ( y t , j − Δ x t ) . (42) w_{t,j}=B_tS_{t,j}(S_{t,j}^{\mathsf{T}}B_t^2S_{t,j})^\dagger S_{t,j}^{\mathsf{T}}B_t(y_{t,j}-\Delta x_t)=\widetilde{Z}_{t,j}(y_{t,j}-\Delta x_t). \tag*{(42)} w t , j = B t S t , j ( S t , j T B t 2 S t , j ) † S t , j T B t ( y t , j − Δ x t ) = Z t , j ( y t , j − Δ x t ) . (42)
Consequently, by Algorithm 1,
z t , j + 1 − Δ x t = y t , j − Δ x t − w t , j = ( I − Z ~ t , j ) { ( 1 − α t ) ( z t , j − Δ x t ) + α t ( v t , j − Δ x t ) } , z_{t,j+1}-\Delta x_t=y_{t,j}-\Delta x_t-w_{t,j}
=(I-\widetilde{Z}_{t,j})\{(1-\alpha_t)(z_{t,j}-\Delta x_t)+\alpha_t(v_{t,j}-\Delta x_t)\}, z t , j + 1 − Δ x t = y t , j − Δ x t − w t , j = ( I − Z t , j ) {( 1 − α t ) ( z t , j − Δ x t ) + α t ( v t , j − Δ x t )} ,
and
v t , j + 1 − Δ x t = β t ( v t , j − Δ x t ) + ( 1 − β t ) ( y t , j − Δ x t ) − γ t w t , j = { ( 1 − α t ) ( 1 − β t ) I − ( 1 − α t ) γ t Z ~ t , j } ( z t , j − Δ x t ) + { ( α t + β t − α t β t ) I − α t γ t Z ~ t , j } ( v t , j − Δ x t ) . \begin{aligned}
v_{t,j+1}-\Delta x_t
&=\beta_t(v_{t,j}-\Delta x_t)+(1-\beta_t)(y_{t,j}-\Delta x_t)-\gamma_t w_{t,j}\\
&=\{(1-\alpha_t)(1-\beta_t)I-(1-\alpha_t)\gamma_t\widetilde{Z}_{t,j}\}(z_{t,j}-\Delta x_t)\\
&\quad+\{(\alpha_t+\beta_t-\alpha_t\beta_t)I-\alpha_t\gamma_t\widetilde{Z}_{t,j}\}(v_{t,j}-\Delta x_t).
\end{aligned} v t , j + 1 − Δ x t = β t ( v t , j − Δ x t ) + ( 1 − β t ) ( y t , j − Δ x t ) − γ t w t , j = {( 1 − α t ) ( 1 − β t ) I − ( 1 − α t ) γ t Z t , j } ( z t , j − Δ x t ) + {( α t + β t − α t β t ) I − α t γ t Z t , j } ( v t , j − Δ x t ) .
Combining the two displays, we obtain the block recursion
\begin{pmatrix}
z_{t,j+1}-\Delta x_t\\
v_{t,j+1}-\Delta x_t
\end{pmatrix}
=\widetilde{C}_{t,j}
\begin{pmatrix}
z_{t,j}-\Delta x_t\\
v_{t,j}-\Delta x_t
\end{pmatrix}. \tag*{(43)}\end{pmatrix}
where C ~ t , j \widetilde{C}_{t,j} C t , j is defined in (C.1). Iterating (C.20) yields
\begin{pmatrix} z_{t,\tau}-\Delta x_t \\ v_{t,\tau}-\Delta x_t \end{pmatrix}
=
\widetilde{C}_{t,\tau-1}\cdots\widetilde{C}_{t,0}
\begin{pmatrix} z_{t,0}-\Delta x_t \\ v_{t,0}-\Delta x_t \end{pmatrix}. \tag*{(44)}\end{pmatrix}
Since z t , 0 = v t , 0 = 0 z_{t,0}=v_{t,0}=0 z t , 0 = v t , 0 = 0 , C ~ t = ∏ j = 0 τ − 1 C ~ t , j \widetilde{C}_t=\prod_{j=0}^{\tau-1}\widetilde{C}_{t,j} C t = ∏ j = 0 τ − 1 C t , j , taking the first block row in (C.21) and using (C.2) gives
z t , τ − Δ x t = − K ~ t Δ x t = K ~ t ( z t , 0 − Δ x t ) , z_{t,\tau}-\Delta x_t=-\widetilde{K}_t\Delta x_t=\widetilde{K}_t(z_{t,0}-\Delta x_t), z t , τ − Δ x t = − K t Δ x t = K t ( z t , 0 − Δ x t ) ,
or, equivalently,
z t , τ = ( I − K ~ t ) Δ x t = − ( I − K ~ t ) B t − 1 m t . (45) z_{t,\tau}=(I-\widetilde{K}_t)\Delta x_t=-(I-\widetilde{K}_t)B_t^{-1}m_t. \tag*{(45)} z t , τ = ( I − K t ) Δ x t = − ( I − K t ) B t − 1 m t . (45)
Conditional on F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 , the quantities ( B t , m t , α t , β t , γ t ) (B_t,m_t,\alpha_t,\beta_t,\gamma_t) ( B t , m t , α t , β t , γ t ) are fixed, and the only randomness in (C.22) comes from the sketching matrices. By Assumption 3.4, we have
E [ K ~ t ∣ F t − 0.5 ] = E [ K ~ t ∣ F t − 1 ] = K t . \mathbb{E}[\widetilde{K}_t\mid\mathcal{F}_{t-0.5}]
=
\mathbb{E}[\widetilde{K}_t\mid\mathcal{F}_{t-1}]
=
K_t. E [ K t ∣ F t − 0.5 ] = E [ K t ∣ F t − 1 ] = K t .
Consequently,
E [ z t , τ ∣ F t − 0.5 ] = − ( I − K t ) B t − 1 m t . (46) \mathbb{E}[z_{t,\tau}\mid\mathcal{F}_{t-0.5}]
=-(I-K_t)B_t^{-1}m_t. \tag*{(46)} E [ z t , τ ∣ F t − 0.5 ] = − ( I − K t ) B t − 1 m t . (46)
Moreover,
E [ z t , τ − ( I − K t ) Δ x t ∣ F t − 0.5 ] = 0. \mathbb{E}[z_{t,\tau}-(I-K_t)\Delta x_t\mid\mathcal{F}_{t-0.5}]=0. E [ z t , τ − ( I − K t ) Δ x t ∣ F t − 0.5 ] = 0.
By the tower property,
E [ z t , τ − ( I − K t ) Δ x t ∣ F t − 1 ] = E [ E [ z t , τ − ( I − K t ) Δ x t ∣ F t − 0.5 ] ∣ F t − 1 ] = 0 , \mathbb{E}[z_{t,\tau}-(I-K_t)\Delta x_t\mid\mathcal{F}_{t-1}]
=
\mathbb{E}[\mathbb{E}[z_{t,\tau}-(I-K_t)\Delta x_t\mid\mathcal{F}_{t-0.5}]\mid\mathcal{F}_{t-1}]
=0, E [ z t , τ − ( I − K t ) Δ x t ∣ F t − 1 ] = E [ E [ z t , τ − ( I − K t ) Δ x t ∣ F t − 0.5 ] ∣ F t − 1 ] = 0 ,
which proves the martingale-difference statement with respect to { F t } t ≥ 0 \{\mathcal{F}_t\}_{t\geq0} { F t } t ≥ 0 .
Finally, suppose that γ t = 1 \gamma_t=1 γ t = 1 . For any Y ∈ R d × d Y\in\mathbb{R}^{d\times d} Y ∈ R d × d , direct multiplication in (C.1) gives
C ~ t , j { ( 1 1 ) ⊗ Y } = ( 1 1 ) ⊗ { ( I − Z ~ t , j ) Y } . (47) \widetilde{C}_{t,j}
\left\{
\begin{pmatrix}1\\1\end{pmatrix}\otimes Y
\right\}
=
\begin{pmatrix}1\\1\end{pmatrix}\otimes\{(I-\widetilde{Z}_{t,j})Y\}. \tag*{(47)} C t , j { ( 1 1 ) ⊗ Y } = ( 1 1 ) ⊗ {( I − Z t , j ) Y } . (47)
By the above property, we then obtain
C ~ t ( I d I d ) = C ~ t , τ − 1 C ~ t , τ − 2 ⋯ C ~ t , 0 { ( 1 1 ) ⊗ I d } = ( 1 1 ) ⊗ ( ∏ j = 0 τ − 1 ( I d − Z ~ t , j ) ) . \widetilde{C}_t
\begin{pmatrix}I_d\\I_d\end{pmatrix}
=
\widetilde{C}_{t,\tau-1}\widetilde{C}_{t,\tau-2}\cdots\widetilde{C}_{t,0}
\left\{
\begin{pmatrix}1\\1\end{pmatrix}\otimes I_d
\right\}
=
\begin{pmatrix}1\\1\end{pmatrix}
\otimes
\left(
\prod_{j=0}^{\tau-1}(I_d-\widetilde{Z}_{t,j})
\right). C t ( I d I d ) = C t , τ − 1 C t , τ − 2 ⋯ C t , 0 { ( 1 1 ) ⊗ I d } = ( 1 1 ) ⊗ ( j = 0 ∏ τ − 1 ( I d − Z t , j ) ) .
Applying (C.24) successively and then using (C.2), we obtain
K ~ t = ( I 0 ) C ~ t ( I I ) = ( I − Z ~ t , τ − 1 ) ⋯ ( I − Z ~ t , 0 ) . \widetilde{K}_t=(I\ \ 0)\widetilde{C}_t
\begin{pmatrix}I\\I\end{pmatrix}
=
(I-\widetilde{Z}_{t,\tau-1})\cdots(I-\widetilde{Z}_{t,0}). K t = ( I 0 ) C t ( I I ) = ( I − Z t , τ − 1 ) ⋯ ( I − Z t , 0 ) .
This completes the proof.
Proofs of Section 4
Throughout this section, let Δ t : = x t − x ⋆ \Delta_t:=x_t-x^\star Δ t := x t − x ⋆ , R : = I − K ⋆ \mathcal{R}:=I-K^\star R := I − K ⋆ and λ m : = λ min ( R ) \lambda_m:=\lambda_{\min}(\mathcal{R}) λ m := λ m i n ( R ) . We use a t = O ( b t ) a_t=O(b_t) a t = O ( b t ) and a t ≲ b t a_t\lesssim b_t a t ≲ b t interchangeably to simplify notation, where the implicit constants are deterministic and independent of t t t . We also use 1 { ⋅ } \mathbb{1}\{\cdot\} 1 { ⋅ } to denote the indicator function. In the proofs below, B t B_t B t is clipped at every iteration as in (14) . The periodically refreshed implementation can be treated by the same arguments: for fixed d d d , its latest refresh differs from t t t by at most d − 1 d-1 d − 1 iterations, so the corresponding Hessian average has the same almost-sure limit and O p ( η t ) O_p(\sqrt{\eta_t}) O p ( η t ) local rate.
Proof of Lemma 4.1
∥ B ~ t − B ⋆ ∥ = ∥ 1 t ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) + 1 t ∑ i = 0 t − 1 ( ∇ 2 f i − B ⋆ ) ∥ ≤ ∥ 1 t ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) ∥ + ∥ 1 t ∑ i = 0 t − 1 ( ∇ 2 f i − B ⋆ ) ∥ ≤ ∥ 1 t ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) ∥ + γ L t ∑ i = 0 t − 1 ∥ x i − x ⋆ ∥ . (48) \begin{aligned}
\left\|\widetilde{B}_t-B^\star\right\|
=
\left\|
\frac{1}{t}\sum_{i=0}^{t-1}(H_i-\nabla^2 f_i)
+
\frac{1}{t}\sum_{i=0}^{t-1}(\nabla^2 f_i-B^\star)
\right\| \\
&\leq
\left\|
\frac{1}{t}\sum_{i=0}^{t-1}(H_i-\nabla^2 f_i)
\right\|
+
\left\|
\frac{1}{t}\sum_{i=0}^{t-1}(\nabla^2 f_i-B^\star)
\right\| \\
&\leq
\left\|
\frac{1}{t}\sum_{i=0}^{t-1}(H_i-\nabla^2 f_i)
\right\|
+
\frac{\gamma L}{t}\sum_{i=0}^{t-1}\|x_i-x^\star\|. \tag*{(48)}
\end{aligned} B t − B ⋆ = t 1 i = 0 ∑ t − 1 ( H i − ∇ 2 f i ) + t 1 i = 0 ∑ t − 1 ( ∇ 2 f i − B ⋆ ) ≤ t 1 i = 0 ∑ t − 1 ( H i − ∇ 2 f i ) + t 1 i = 0 ∑ t − 1 ( ∇ 2 f i − B ⋆ ) ≤ t 1 i = 0 ∑ t − 1 ( H i − ∇ 2 f i ) + t γ L i = 0 ∑ t − 1 ∥ x i − x ⋆ ∥. (48)
By Theorem 3.5, x i ⟶ a . s . x ⋆ x_i \mathrel{\overset{\mathrm{a.s.}}{\longrightarrow}} x^\star x i ⟶ a.s. x ⋆ . Hence, the second term on the right-hand side of (D.1) converges to zero almost surely by Lemma B.2. For the first term, { H i − ∇ 2 f i } i ≥ 0 \{H_i-\nabla^2 f_i\}_{i\geq0} { H i − ∇ 2 f i } i ≥ 0 is a matrix-valued martingale difference sequence since E [ H i ∣ F i − 1 ] = ∇ 2 f i \mathbb{E}[H_i\mid\mathcal{F}_{i-1}]=\nabla^2 f_i E [ H i ∣ F i − 1 ] = ∇ 2 f i . For each integer m ≥ 1 m\geq1 m ≥ 1 , 1 { ∥ x i − x ⋆ ∥ ≤ m } \mathbf{1}_{\{\|x_i-x^\star\|\leq m\}} 1 { ∥ x i − x ⋆ ∥ ≤ m } is F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable. By Assumption 3.3, ∥ A ∥ F 2 ≤ d ∥ A ∥ 2 \|A\|_F^2\leq d\|A\|^2 ∥ A ∥ F 2 ≤ d ∥ A ∥ 2 and martingale orthogonality, we have
E [ ∥ ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) 1 { ∥ x i − x ⋆ ∥ ≤ m } i + 1 ∥ F 2 ] = ∑ i = 0 t − 1 E [ ∥ H i − ∇ 2 f i ∥ F 2 1 { ∥ x i − x ⋆ ∥ ≤ m } ] ( i + 1 ) 2 ≤ d ( C H , 1 m 2 + C H , 2 ) ∑ i = 0 ∞ 1 ( i + 1 ) 2 < ∞ . \mathbb{E}\left[\left\|\sum_{i=0}^{t-1}\frac{(H_i-\nabla^2 f_i)\mathbf{1}_{\{\|x_i-x^\star\|\leq m\}}}{i+1}\right\|_F^2\right]
=\sum_{i=0}^{t-1}\frac{\mathbb{E}\left[\|H_i-\nabla^2 f_i\|_F^2\mathbf{1}_{\{\|x_i-x^\star\|\leq m\}}\right]}{(i+1)^2}
\leq d(C_{H,1}m^2+C_{H,2})\sum_{i=0}^{\infty}\frac{1}{(i+1)^2}<\infty. E i = 0 ∑ t − 1 i + 1 ( H i − ∇ 2 f i ) 1 { ∥ x i − x ⋆ ∥ ≤ m } F 2 = i = 0 ∑ t − 1 ( i + 1 ) 2 E [ ∥ H i − ∇ 2 f i ∥ F 2 1 { ∥ x i − x ⋆ ∥ ≤ m } ] ≤ d ( C H , 1 m 2 + C H , 2 ) i = 0 ∑ ∞ ( i + 1 ) 2 1 < ∞.
where the equality follows from the orthogonality of the martingale differences under the Frobenius inner product. Hence, for every m ≥ 1 m\geq1 m ≥ 1 , { ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) 1 { ∥ x i − x ⋆ ∥ ≤ m } / ( i + 1 ) } t ≥ 1 \left\{\sum_{i=0}^{t-1}(H_i-\nabla^2 f_i)\mathbf{1}_{\{\|x_i-x^\star\|\leq m\}}/(i+1)\right\}_{t\geq1} { ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) 1 { ∥ x i − x ⋆ ∥ ≤ m } / ( i + 1 ) } t ≥ 1 is an L 2 L^2 L 2 -bounded martingale. By Doob’s martingale convergence theorem [16 Corollary 2.2] , ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) 1 { ∥ x i − x ⋆ ∥ ≤ m } / ( i + 1 ) \sum_{i=0}^{t-1}(H_i-\nabla^2 f_i)\mathbf{1}_{\{\|x_i-x^\star\|\leq m\}}/(i+1) ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) 1 { ∥ x i − x ⋆ ∥ ≤ m } / ( i + 1 ) converges almost surely.
Since x i ⟶ a . s . x ⋆ x_i\mathrel{\overset{\mathrm{a.s.}}{\longrightarrow}}x^\star x i ⟶ a.s. x ⋆ and the positive integers are countable, almost surely sup i ≥ 0 ∥ x i − x ⋆ ∥ < ∞ \sup_{i\geq0}\|x_i-x^\star\|<\infty sup i ≥ 0 ∥ x i − x ⋆ ∥ < ∞ and the truncated weighted sum converges for every m ≥ 1 m\geq1 m ≥ 1 . For each such realization, we could choose an integer m > sup i ≥ 0 ∥ x i − x ⋆ ∥ m>\sup_{i\geq0}\|x_i-x^\star\| m > sup i ≥ 0 ∥ x i − x ⋆ ∥ . Then 1 { ∥ x i − x ⋆ ∥ ≤ m } = 1 \mathbf{1}_{\{\|x_i-x^\star\|\leq m\}}=1 1 { ∥ x i − x ⋆ ∥ ≤ m } = 1 for every i ≥ 0 i\geq0 i ≥ 0 , so ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) / ( i + 1 ) \sum_{i=0}^{t-1}(H_i-\nabla^2 f_i)/(i+1) ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) / ( i + 1 ) converges. Applying Lemma B.1 further yields
1 t ∑ i = 0 t − 1 ( H i − ∇ 2 f i ) ⟶ a . s . 0. \frac{1}{t}\sum_{i=0}^{t-1}(H_i-\nabla^2 f_i)\mathrel{\overset{\mathrm{a.s.}}{\longrightarrow}}0. t 1 i = 0 ∑ t − 1 ( H i − ∇ 2 f i ) ⟶ a.s. 0.
Combining the preceding conclusion with (D.1), we obtain B ~ t ⟶ a . s . B ⋆ \widetilde{B}_t\mathrel{\overset{\mathrm{a.s.}}{\longrightarrow}}B^\star B t ⟶ a.s. B ⋆ . Since B ⋆ B^\star B ⋆ belongs to the clipping interval for all sufficiently large t t t almost surely, the projection property of Clip ( ⋅ ) \operatorname{Clip}(\cdot) Clip ( ⋅ ) gives us
∥ B t − B ⋆ ∥ ≤ ∥ B t − B ⋆ ∥ F ≤ ∥ B ~ t − B ⋆ ∥ F ≤ d ∥ B ~ t − B ⋆ ∥ ⟶ a . s . 0. \|B_t-B^\star\|\leq\|B_t-B^\star\|_F\leq\|\widetilde{B}_t-B^\star\|_F\leq\sqrt{d}\|\widetilde{B}_t-B^\star\|\mathrel{\overset{\mathrm{a.s.}}{\longrightarrow}}0. ∥ B t − B ⋆ ∥ ≤ ∥ B t − B ⋆ ∥ F ≤ ∥ B t − B ⋆ ∥ F ≤ d ∥ B t − B ⋆ ∥ ⟶ a.s. 0.
Therefore,
B t ⟶ a . s . B ⋆ . (49) B_t\mathrel{\overset{\mathrm{a.s.}}{\longrightarrow}}B^\star. \tag*{(49)} B t ⟶ a.s. B ⋆ . (49)
Convergence of ( α t , β t , γ t ) (\alpha_t,\beta_t,\gamma_t) ( α t , β t , γ t ) . Recall from (5) that the acceleration parameters ( α t , β t , γ t ) (\alpha_t,\beta_t,\gamma_t) ( α t , β t , γ t ) (the distinction between ( α t , β t , γ t ) (\alpha_t,\beta_t,\gamma_t) ( α t , β t , γ t ) and ( α ⋆ , β ⋆ , γ ⋆ ) (\alpha^\star,\beta^\star,\gamma^\star) ( α ⋆ , β ⋆ , γ ⋆ ) is solely due to the evaluation at ( μ t , ν t ) (\mu_t,\nu_t) ( μ t , ν t ) v.s. ( μ ⋆ , ν ⋆ ) (\mu^\star,\nu^\star) ( μ ⋆ , ν ⋆ ) ) can be regarded as functions of ( μ t , ν t ) (\mu_t,\nu_t) ( μ t , ν t ) as follows:
α ( μ t , ν t ) = 1 1 + ν t / μ t , β ( μ t , ν t ) = 1 − μ t ν t , γ ( μ t , ν t ) = 1 μ t ν t . \alpha(\mu_t,\nu_t)=\frac{1}{1+\sqrt{\nu_t/\mu_t}},\qquad
\beta(\mu_t,\nu_t)=1-\sqrt{\frac{\mu_t}{\nu_t}},\qquad
\gamma(\mu_t,\nu_t)=\frac{1}{\sqrt{\mu_t\nu_t}}. α ( μ t , ν t ) = 1 + ν t / μ t 1 , β ( μ t , ν t ) = 1 − ν t μ t , γ ( μ t , ν t ) = μ t ν t 1 .
Each of these functions is continuously differentiable on ( 0 , ∞ ) 2 (0,\infty)^2 ( 0 , ∞ ) 2 . Since ( μ t , ν t ) ∈ [ γ S , 1 ] × [ 1 , 1 / γ S ] (\mu_t,\nu_t)\in[\gamma_S,1]\times[1,1/\gamma_S] ( μ t , ν t ) ∈ [ γ S , 1 ] × [ 1 , 1/ γ S ] for all t ≥ 0 t\geq0 t ≥ 0 , their gradients are uniformly bounded on this compact and convex set. Therefore, by the mean-value theorem, we further have
∣ α t − α ⋆ ∣ + ∣ β t − β ⋆ ∣ + ∣ γ t − γ ⋆ ∣ ≲ ∣ μ t − μ ⋆ ∣ + ∣ ν t − ν ⋆ ∣ . (50) |\alpha_t-\alpha^\star|+|\beta_t-\beta^\star|+|\gamma_t-\gamma^\star|\lesssim|\mu_t-\mu^\star|+|\nu_t-\nu^\star|. \tag*{(50)} ∣ α t − α ⋆ ∣ + ∣ β t − β ⋆ ∣ + ∣ γ t − γ ⋆ ∣ ≲ ∣ μ t − μ ⋆ ∣ + ∣ ν t − ν ⋆ ∣. (50)
Thus, it suffices to study the convergence of ( μ t , ν t ) (\mu_t,\nu_t) ( μ t , ν t ) . For a generic sketching matrix S S S , define
Z ~ t ( S ) : = B t S ( S ⊤ B t 2 S ) † S ⊤ B t , Z t : = E [ Z ~ t ( S ) ∣ F t − 1 ] , \widetilde{Z}_t(S):=B_tS(S^\top B_t^2S)^\dagger S^\top B_t,\qquad
Z_t:=\mathbb{E}[\widetilde{Z}_t(S)\mid\mathcal{F}_{t-1}], Z t ( S ) := B t S ( S ⊤ B t 2 S ) † S ⊤ B t , Z t := E [ Z t ( S ) ∣ F t − 1 ] ,
Z ~ ⋆ ( S ) : = B ⋆ S ( S ⊤ ( B ⋆ ) 2 S ) † S ⊤ B ⋆ , Z ⋆ : = E [ Z ~ ⋆ ( S ) ] . (51) \widetilde{Z}^\star(S):=B^\star S(S^\top(B^\star)^2S)^\dagger S^\top B^\star,\qquad
Z^\star:=\mathbb{E}[\widetilde{Z}^\star(S)]. \tag*{(51)} Z ⋆ ( S ) := B ⋆ S ( S ⊤ ( B ⋆ ) 2 S ) † S ⊤ B ⋆ , Z ⋆ := E [ Z ⋆ ( S )] . (51)
We first recall the following perturbation result for the projection matrix.
Lemma D.1. [24 Lemma 5.2] Suppose B t , B ⋆ ∈ R d × d B_t,B^\star\in\mathbb{R}^{d\times d} B t , B ⋆ ∈ R d × d are both nonsingular. Then, for any sketching matrix S ∈ R d × s S\in\mathbb{R}^{d\times s} S ∈ R d × s , we have
∥ Z ~ t ( S ) − Z ~ ⋆ ( S ) ∥ ≤ 2 ∥ B t − B ⋆ ∥ σ min ( B ⋆ ) ∥ S ∥ ∥ S ⊤ ∥ , \|\widetilde{Z}_t(S)-\widetilde{Z}^\star(S)\|\leq\frac{2\|B_t-B^\star\|}{\sigma_{\min}(B^\star)}\|S\|\|S^\top\|, ∥ Z t ( S ) − Z ⋆ ( S ) ∥ ≤ σ m i n ( B ⋆ ) 2∥ B t − B ⋆ ∥ ∥ S ∥∥ S ⊤ ∥ , where σ min ( ⋅ ) \sigma_{\min}(\cdot) σ m i n ( ⋅ ) denotes the smallest singular value.
By Lemma D.1 , Assumption 3.4 , and the independence between S S S and F t − 1 \mathcal{F}_{t-1} F t − 1 , we have
E [ ∥ Z ~ t ( S ) − Z ~ ⋆ ( S ) ∥ ∣ F t − 1 ] ≤ 2 ∥ B t − B ⋆ ∥ γ H 2 E [ ∥ S ∥ ∥ S † ∥ ] ≲ ∥ B t − B ⋆ ∥ . (52) \mathbb{E}\left[\left\|\widetilde{Z}_t(S)-\widetilde{Z}^{\star}(S)\right\|\mid\mathcal{F}_{t-1}\right]\leq\frac{2\left\|B_t-B^{\star}\right\|}{\gamma_H^2}\mathbb{E}\left[\|S\|\|S^{\dagger}\|\right]\lesssim\left\|B_t-B^{\star}\right\|. \tag*{(52)} E [ Z t ( S ) − Z ⋆ ( S ) ∣ F t − 1 ] ≤ γ H 2 2 ∥ B t − B ⋆ ∥ E [ ∥ S ∥∥ S † ∥ ] ≲ ∥ B t − B ⋆ ∥ . (52)
•• Convergence of μ t \mu_t μ t . By the definitions of μ t \mu_t μ t and μ ⋆ \mu^{\star} μ ⋆ , we have μ t = λ min ( Z t ) \mu_t=\lambda_{\min}(Z_t) μ t = λ m i n ( Z t ) and μ ⋆ = λ min ( Z ⋆ ) \mu^{\star}=\lambda_{\min}(Z^{\star}) μ ⋆ = λ m i n ( Z ⋆ ) . Applying Weyl’s perturbation theorem [2 Corollary III.2.6] , we obtain
∣ μ t − μ ⋆ ∣ ≤ ∥ Z t − Z ⋆ ∥ = ∥ E [ Z ~ t ( S ) − Z ~ ⋆ ( S ) ∣ F t − 1 ] ∥ ≤ E [ ∥ Z ~ t ( S ) − Z ~ ⋆ ( S ) ∥ ∣ F t − 1 ] ≲ ∥ B t − B ⋆ ∥ . (53) \begin{aligned}
|\mu_t-\mu^{\star}|\leq\|Z_t-Z^{\star}\|=\left\|\mathbb{E}\left[\widetilde{Z}_t(S)-\widetilde{Z}^{\star}(S)\mid\mathcal{F}_{t-1}\right]\right\| \\
&\leq\mathbb{E}\left[\left\|\widetilde{Z}_t(S)-\widetilde{Z}^{\star}(S)\right\|\mid\mathcal{F}_{t-1}\right]\lesssim\left\|B_t-B^{\star}\right\|. \tag*{(53)}
\end{aligned} ∣ μ t − μ ⋆ ∣ ≤ ∥ Z t − Z ⋆ ∥ = E [ Z t ( S ) − Z ⋆ ( S ) ∣ F t − 1 ] ≤ E [ Z t ( S ) − Z ⋆ ( S ) ∣ F t − 1 ] ≲ ∥ B t − B ⋆ ∥ . (53)
Since we have shown B t ⟶ a . s . B ⋆ B_t \mathrel{\mathop{\longrightarrow}\limits^{\mathrm{a.s.}}} B^{\star} B t ⟶ a.s. B ⋆ , it follows that μ t ⟶ a . s . μ ⋆ \mu_t \mathrel{\mathop{\longrightarrow}\limits^{\mathrm{a.s.}}} \mu^{\star} μ t ⟶ a.s. μ ⋆ .
•• Convergence of ν t \nu_t ν t . Define
A t : = E [ Z ~ t ( S ) Z t − 1 Z ~ t ( S ) ∣ F t − 1 ] , A ⋆ : = E [ Z ~ ⋆ ( S ) ( Z ⋆ ) − 1 Z ~ ⋆ ( S ) ] . A_t:=\mathbb{E}\left[\widetilde{Z}_t(S)Z_t^{-1}\widetilde{Z}_t(S)\mid\mathcal{F}_{t-1}\right],\qquad A^{\star}:=\mathbb{E}\left[\widetilde{Z}^{\star}(S)(Z^{\star})^{-1}\widetilde{Z}^{\star}(S)\right]. A t := E [ Z t ( S ) Z t − 1 Z t ( S ) ∣ F t − 1 ] , A ⋆ := E [ Z ⋆ ( S ) ( Z ⋆ ) − 1 Z ⋆ ( S ) ] .
The matrices A t A_t A t and A ⋆ A^{\star} A ⋆ are symmetric positive semidefinite. Letting b = Z t 1 / 2 a b=Z_t^{1/2}a b = Z t 1/2 a in (2.3), we obtain
ν t = sup a ≠ 0 a T A t a a T Z t a = sup b ≠ 0 b T Z t − 1 / 2 A t Z t − 1 / 2 b b T b = λ max ( Z t − 1 / 2 A t Z t − 1 / 2 ) = ∥ Z t − 1 / 2 A t Z t − 1 / 2 ∥ . (54) \begin{aligned}
\nu_t=\sup_{a\neq0}\frac{a^{\mathsf{T}}A_ta}{a^{\mathsf{T}}Z_ta}=\sup_{b\neq0}\frac{b^{\mathsf{T}}Z_t^{-1/2}A_tZ_t^{-1/2}b}{b^{\mathsf{T}}b} \\
&=\lambda_{\max}\left(Z_t^{-1/2}A_tZ_t^{-1/2}\right)=\left\|Z_t^{-1/2}A_tZ_t^{-1/2}\right\|. \tag*{(54)}
\end{aligned} ν t = a = 0 sup a T Z t a a T A t a = b = 0 sup b T b b T Z t − 1/2 A t Z t − 1/2 b = λ m a x ( Z t − 1/2 A t Z t − 1/2 ) = Z t − 1/2 A t Z t − 1/2 . (54)
Similarly, we have ν ⋆ = ∥ ( Z ⋆ ) − 1 / 2 A ⋆ ( Z ⋆ ) − 1 / 2 ∥ \nu^{\star}=\left\|(Z^{\star})^{-1/2}A^{\star}(Z^{\star})^{-1/2}\right\| ν ⋆ = ( Z ⋆ ) − 1/2 A ⋆ ( Z ⋆ ) − 1/2 . Consequently,
∣ ν t − ν ⋆ ∣ = ∣ ∥ Z t − 1 / 2 A t Z t − 1 / 2 ∥ − ∥ ( Z ⋆ ) − 1 / 2 A ⋆ ( Z ⋆ ) − 1 / 2 ∥ ∣ ≤ ∥ Z t − 1 / 2 A t Z t − 1 / 2 − ( Z ⋆ ) − 1 / 2 A ⋆ ( Z ⋆ ) − 1 / 2 ∥ ≤ ∥ ( Z t − 1 / 2 − ( Z ⋆ ) − 1 / 2 ) A t Z t − 1 / 2 ∥ + ∥ ( Z ⋆ ) − 1 / 2 ( A t − A ⋆ ) Z t − 1 / 2 ∥ + ∥ ( Z ⋆ ) − 1 / 2 A ⋆ ( Z t − 1 / 2 − ( Z ⋆ ) − 1 / 2 ) ∥ . (55) \begin{aligned}
|\nu_t-\nu^{\star}|=\left|\left\|Z_t^{-1/2}A_tZ_t^{-1/2}\right\|-\left\|(Z^{\star})^{-1/2}A^{\star}(Z^{\star})^{-1/2}\right\|\right| \\
&\leq\left\|Z_t^{-1/2}A_tZ_t^{-1/2}-(Z^{\star})^{-1/2}A^{\star}(Z^{\star})^{-1/2}\right\| \\
&\leq\left\|\left(Z_t^{-1/2}-(Z^{\star})^{-1/2}\right)A_tZ_t^{-1/2}\right\|+\left\|(Z^{\star})^{-1/2}(A_t-A^{\star})Z_t^{-1/2}\right\|+\left\|(Z^{\star})^{-1/2}A^{\star}\left(Z_t^{-1/2}-(Z^{\star})^{-1/2}\right)\right\|. \tag*{(55)}
\end{aligned} ∣ ν t − ν ⋆ ∣ = Z t − 1/2 A t Z t − 1/2 − ( Z ⋆ ) − 1/2 A ⋆ ( Z ⋆ ) − 1/2 ≤ Z t − 1/2 A t Z t − 1/2 − ( Z ⋆ ) − 1/2 A ⋆ ( Z ⋆ ) − 1/2 ≤ ( Z t − 1/2 − ( Z ⋆ ) − 1/2 ) A t Z t − 1/2 + ( Z ⋆ ) − 1/2 ( A t − A ⋆ ) Z t − 1/2 + ( Z ⋆ ) − 1/2 A ⋆ ( Z t − 1/2 − ( Z ⋆ ) − 1/2 ) . (55)
To control the three terms on the right-hand side, we use the following preparation lemma.
Lemma D.2. Under Assumption 3.4 , we have
∥ Z t − 1 − ( Z ⋆ ) − 1 ∥ ≤ 1 γ S 2 ∥ Z t − Z ⋆ ∥ , ∥ Z t − 1 / 2 − ( Z ⋆ ) − 1 / 2 ∥ ≤ 1 2 γ S 3 / 2 ∥ Z t − Z ⋆ ∥ . \left\|Z_t^{-1}-(Z^{\star})^{-1}\right\|\leq\frac{1}{\gamma_S^2}\left\|Z_t-Z^{\star}\right\|,\qquad\left\|Z_t^{-1/2}-(Z^{\star})^{-1/2}\right\|\leq\frac{1}{2\gamma_S^{3/2}}\left\|Z_t-Z^{\star}\right\|. Z t − 1 − ( Z ⋆ ) − 1 ≤ γ S 2 1 ∥ Z t − Z ⋆ ∥ , Z t − 1/2 − ( Z ⋆ ) − 1/2 ≤ 2 γ S 3/2 1 ∥ Z t − Z ⋆ ∥ .
••• The first term in (D.8). Since Z ~ t ( S ) \widetilde{Z}_t(S) Z t ( S ) is an orthogonal projection and Z t ⪰ γ S I Z_t\succeq\gamma_S I Z t ⪰ γ S I , we have ∥ A t ∥ ≤ 1 / γ S \|A_t\|\leq1/\gamma_S ∥ A t ∥ ≤ 1/ γ S and ∥ Z t − 1 / 2 ∥ ≤ 1 / γ S \|Z_t^{-1/2}\|\leq1/\sqrt{\gamma_S} ∥ Z t − 1/2 ∥ ≤ 1/ γ S . Therefore, Lemma D.2 implies that
∥ ( Z t − 1 / 2 − ( Z ⋆ ) − 1 / 2 ) A t Z t − 1 / 2 ∥ ≲ ∥ Z t − Z ⋆ ∥ . (56) \left\|\left(Z_t^{-1/2}-(Z^{\star})^{-1/2}\right)A_tZ_t^{-1/2}\right\|\lesssim\left\|Z_t-Z^{\star}\right\|. \tag*{(56)} ( Z t − 1/2 − ( Z ⋆ ) − 1/2 ) A t Z t − 1/2 ≲ ∥ Z t − Z ⋆ ∥ . (56)
••• The third term in (D.8). Similarly, ∥ A ⋆ ∥ ≤ 1 / γ S \|A^{\star}\|\leq1/\gamma_S ∥ A ⋆ ∥ ≤ 1/ γ S and ∥ ( Z ⋆ ) − 1 / 2 ∥ ≤ 1 / γ S \|(Z^{\star})^{-1/2}\|\leq1/\sqrt{\gamma_S} ∥ ( Z ⋆ ) − 1/2 ∥ ≤ 1/ γ S . Therefore,
∥ ( Z ⋆ ) − 1 / 2 A ⋆ ( Z t − 1 / 2 − ( Z ⋆ ) − 1 / 2 ) ∥ ≲ ∥ Z t − Z ⋆ ∥ . (57) \left\|(Z^{\star})^{-1/2}A^{\star}\left(Z_t^{-1/2}-(Z^{\star})^{-1/2}\right)\right\|\lesssim\left\|Z_t-Z^{\star}\right\|. \tag*{(57)} ( Z ⋆ ) − 1/2 A ⋆ ( Z t − 1/2 − ( Z ⋆ ) − 1/2 ) ≲ ∥ Z t − Z ⋆ ∥ . (57)
∥ ( Z ⋆ ) − 1 / 2 ( A t − A ⋆ ) Z t − 1 / 2 ∥ ≤ 1 γ S ∥ A t − A ⋆ ∥ = 1 γ S ∥ E [ Z ~ t ( S ) Z t − 1 Z ~ t ( S ) ∣ F t − 1 ] − E [ Z ~ ⋆ ( S ) ( Z ⋆ ) − 1 Z ~ ⋆ ( S ) ] ∥ = 1 γ S ∥ E [ Z ~ t ( S ) Z t − 1 Z ~ t ( S ) − Z ~ ⋆ ( S ) ( Z ⋆ ) − 1 Z ~ ⋆ ( S ) ∣ F t − 1 ] ∥ = 1 γ S ∥ E [ ( Z ~ t ( S ) − Z ~ ⋆ ( S ) ) Z t − 1 Z ~ t ( S ) + Z ~ ⋆ ( S ) { Z t − 1 − ( Z ⋆ ) − 1 } Z ~ t ( S ) + Z ~ ⋆ ( S ) ( Z ⋆ ) − 1 ( Z ~ t ( S ) − Z ~ ⋆ ( S ) ) ∣ F t − 1 ] ∥ ≤ 1 γ S { 1 γ S E [ ∥ Z ~ t ( S ) − Z ~ ⋆ ( S ) ∥ ∣ F t − 1 ] + ∥ Z t − 1 − ( Z ⋆ ) − 1 ∥ + 1 γ S E [ ∥ Z ~ t ( S ) − Z ~ ⋆ ( S ) ∥ ∣ F t − 1 ] } ≤ 1 γ S 2 ∥ Z t − Z ⋆ ∥ + 2 γ S 2 E [ ∥ Z ~ t ( S ) − Z ~ ⋆ ( S ) ∥ ∣ F t − 1 ] . (58) \begin{aligned}
\left\lVert(Z^\star)^{-1/2}(A_t-A^\star)Z_t^{-1/2}\right\rVert
&\le\frac{1}{\gamma_S}\lVert A_t-A^\star\rVert\\
&= \frac{1}{\gamma_S}\left\lVert\mathbb{E}\left[\widetilde{Z}_t(S)Z_t^{-1}\widetilde{Z}_t(S)\mid\mathcal{F}_{t-1}\right]-\mathbb{E}\left[\widetilde{Z}^\star(S)(Z^\star)^{-1}\widetilde{Z}^\star(S)\right]\right\rVert\\
&= \frac{1}{\gamma_S}\left\lVert\mathbb{E}\left[\widetilde{Z}_t(S)Z_t^{-1}\widetilde{Z}_t(S)-\widetilde{Z}^\star(S)(Z^\star)^{-1}\widetilde{Z}^\star(S)\mid\mathcal{F}_{t-1}\right]\right\rVert\\
&= \frac{1}{\gamma_S}\left\lVert\mathbb{E}\left[(\widetilde{Z}_t(S)-\widetilde{Z}^\star(S))Z_t^{-1}\widetilde{Z}_t(S)+\widetilde{Z}^\star(S)\{Z_t^{-1}-(Z^\star)^{-1}\}\widetilde{Z}_t(S)\right.\right.\\
&\qquad\left.\left.{}+\widetilde{Z}^\star(S)(Z^\star)^{-1}(\widetilde{Z}_t(S)-\widetilde{Z}^\star(S))\mid\mathcal{F}_{t-1}\right]\right\rVert\\
&\le\frac{1}{\gamma_S}\left\{\frac{1}{\gamma_S}\mathbb{E}\left[\lVert\widetilde{Z}_t(S)-\widetilde{Z}^\star(S)\rVert\mid\mathcal{F}_{t-1}\right]+\lVert Z_t^{-1}-(Z^\star)^{-1}\rVert\right.\\
&\qquad\left.{}+\frac{1}{\gamma_S}\mathbb{E}\left[\lVert\widetilde{Z}_t(S)-\widetilde{Z}^\star(S)\rVert\mid\mathcal{F}_{t-1}\right]\right\} \\
&\le\frac{1}{\gamma_S^2}\lVert Z_t-Z^\star\rVert+\frac{2}{\gamma_S^2}\mathbb{E}\left[\lVert\widetilde{Z}_t(S)-\widetilde{Z}^\star(S)\rVert\mid\mathcal{F}_{t-1}\right]. \tag*{(58)}
\end{aligned} ( Z ⋆ ) − 1/2 ( A t − A ⋆ ) Z t − 1/2 ≤ γ S 1 ∥ A t − A ⋆ ∥ = γ S 1 E [ Z t ( S ) Z t − 1 Z t ( S ) ∣ F t − 1 ] − E [ Z ⋆ ( S ) ( Z ⋆ ) − 1 Z ⋆ ( S ) ] = γ S 1 E [ Z t ( S ) Z t − 1 Z t ( S ) − Z ⋆ ( S ) ( Z ⋆ ) − 1 Z ⋆ ( S ) ∣ F t − 1 ] = γ S 1 E [ ( Z t ( S ) − Z ⋆ ( S )) Z t − 1 Z t ( S ) + Z ⋆ ( S ) { Z t − 1 − ( Z ⋆ ) − 1 } Z t ( S ) + Z ⋆ ( S ) ( Z ⋆ ) − 1 ( Z t ( S ) − Z ⋆ ( S )) ∣ F t − 1 ] ≤ γ S 1 { γ S 1 E [ ∥ Z t ( S ) − Z ⋆ ( S )∥ ∣ F t − 1 ] + ∥ Z t − 1 − ( Z ⋆ ) − 1 ∥ + γ S 1 E [ ∥ Z t ( S ) − Z ⋆ ( S )∥ ∣ F t − 1 ] } ≤ γ S 2 1 ∥ Z t − Z ⋆ ∥ + γ S 2 2 E [ ∥ Z t ( S ) − Z ⋆ ( S )∥ ∣ F t − 1 ] . (58)
Substituting (D.9), (D.10), and (D.11) into (D.8), we obtain
∣ ν t − ν ⋆ ∣ ≲ ∥ Z t − Z ⋆ ∥ + E [ ∥ Z ~ t ( S ) − Z ~ ⋆ ( S ) ∥ ∣ F t − 1 ] ≲ ∥ B t − B ⋆ ∥ , (59) \lvert\nu_t-\nu^\star\rvert\lesssim\lVert Z_t-Z^\star\rVert+\mathbb{E}\left[\lVert\widetilde{Z}_t(S)-\widetilde{Z}^\star(S)\rVert\mid\mathcal{F}_{t-1}\right]\lesssim\lVert B_t-B^\star\rVert, \tag*{(59)} ∣ ν t − ν ⋆ ∣ ≲ ∥ Z t − Z ⋆ ∥ + E [ ∥ Z t ( S ) − Z ⋆ ( S )∥ ∣ F t − 1 ] ≲ ∥ B t − B ⋆ ∥ , (59)
where the last inequality follows from (D.5). Since B t → a . s . B ⋆ B_t \mathrel{\mathop{\to}^{\mathrm{a.s.}}} B^\star B t → a.s. B ⋆ , we can conclude that ν t → a . s . ν ⋆ \nu_t \mathrel{\mathop{\to}^{\mathrm{a.s.}}} \nu^\star ν t → a.s. ν ⋆ .
Combining (D.3), (D.6), and (D.12), we obtain
∣ α t − α ⋆ ∣ + ∣ β t − β ⋆ ∣ + ∣ γ t − γ ⋆ ∣ ≲ ∥ B t − B ⋆ ∥ . (60) \lvert\alpha_t-\alpha^\star\rvert+\lvert\beta_t-\beta^\star\rvert+\lvert\gamma_t-\gamma^\star\rvert\lesssim\lVert B_t-B^\star\rVert. \tag*{(60)} ∣ α t − α ⋆ ∣ + ∣ β t − β ⋆ ∣ + ∣ γ t − γ ⋆ ∣ ≲ ∥ B t − B ⋆ ∥ . (60)
Together with (D.2), this further implies
( α t , β t , γ t ) → a . s . ( α ⋆ , β ⋆ , γ ⋆ ) . (\alpha_t,\beta_t,\gamma_t)\mathrel{\mathop{\to}^{\mathrm{a.s.}}}(\alpha^\star,\beta^\star,\gamma^\star). ( α t , β t , γ t ) → a.s. ( α ⋆ , β ⋆ , γ ⋆ ) .
C t = M t τ , C ⋆ = ( M ⋆ ) τ , C_t=M_t^\tau,\qquad C^\star=(M^\star)^\tau, C t = M t τ , C ⋆ = ( M ⋆ ) τ ,
where M t M_t M t is defined in (C.3), and M ⋆ M^\star M ⋆ is defined analogously by replacing ( α t , β t , γ t , Z t ) (\alpha_t,\beta_t,\gamma_t,Z_t) ( α t , β t , γ t , Z t ) with ( α ⋆ , β ⋆ , γ ⋆ , Z ⋆ ) (\alpha^\star,\beta^\star,\gamma^\star,Z^\star) ( α ⋆ , β ⋆ , γ ⋆ , Z ⋆ ) . From the block representation of M t M_t M t , we have
∥ M t − M ⋆ ∥ ≤ ∥ ( 1 − α t ) ( I − Z t ) − ( 1 − α ⋆ ) ( I − Z ⋆ ) ∥ + ∥ α t ( I − Z t ) − α ⋆ ( I − Z ⋆ ) ∥ + ∥ ( 1 − α t ) ( 1 − β t ) I − ( 1 − α t ) γ t Z t − ( 1 − α ⋆ ) ( 1 − β ⋆ ) I + ( 1 − α ⋆ ) γ ⋆ Z ⋆ ∥ + ∥ ( α t + β t − α t β t ) I − α t γ t Z t − ( α ⋆ + β ⋆ − α ⋆ β ⋆ ) I + α ⋆ γ ⋆ Z ⋆ ∥ ≲ ∣ α t − α ⋆ ∣ + ∣ β t − β ⋆ ∣ + ∣ γ t − γ ⋆ ∣ + ∥ Z t − Z ⋆ ∥ ≲ ∥ B t − B ⋆ ∥ . (61) \begin{aligned}
\lVert M_t-M^\star\rVert
&\le\lVert(1-\alpha_t)(I-Z_t)-(1-\alpha^\star)(I-Z^\star)\rVert+\lVert\alpha_t(I-Z_t)-\alpha^\star(I-Z^\star)\rVert\\
&\quad+\lVert(1-\alpha_t)(1-\beta_t)I-(1-\alpha_t)\gamma_tZ_t-(1-\alpha^\star)(1-\beta^\star)I+(1-\alpha^\star)\gamma^\star Z^\star\rVert\\
&\quad+\lVert(\alpha_t+\beta_t-\alpha_t\beta_t)I-\alpha_t\gamma_tZ_t-(\alpha^\star+\beta^\star-\alpha^\star\beta^\star)I+\alpha^\star\gamma^\star Z^\star\rVert\\
&\lesssim\lvert\alpha_t-\alpha^\star\rvert+\lvert\beta_t-\beta^\star\rvert+\lvert\gamma_t-\gamma^\star\rvert+\lVert Z_t-Z^\star\rVert\lesssim\lVert B_t-B^\star\rVert. \tag*{(61)}
\end{aligned} ∥ M t − M ⋆ ∥ ≤ ∥( 1 − α t ) ( I − Z t ) − ( 1 − α ⋆ ) ( I − Z ⋆ )∥ + ∥ α t ( I − Z t ) − α ⋆ ( I − Z ⋆ )∥ + ∥( 1 − α t ) ( 1 − β t ) I − ( 1 − α t ) γ t Z t − ( 1 − α ⋆ ) ( 1 − β ⋆ ) I + ( 1 − α ⋆ ) γ ⋆ Z ⋆ ∥ + ∥( α t + β t − α t β t ) I − α t γ t Z t − ( α ⋆ + β ⋆ − α ⋆ β ⋆ ) I + α ⋆ γ ⋆ Z ⋆ ∥ ≲ ∣ α t − α ⋆ ∣ + ∣ β t − β ⋆ ∣ + ∣ γ t − γ ⋆ ∣ + ∥ Z t − Z ⋆ ∥ ≲ ∥ B t − B ⋆ ∥ . (61)
where the last inequality follows from (D.5) and (D.13). Furthermore, using ∥ I − Z t ∥ ≤ 1 \lVert I-Z_t\rVert\le1 ∥ I − Z t ∥ ≤ 1 , ∥ Z t ∥ ≤ 1 \lVert Z_t\rVert\le1 ∥ Z t ∥ ≤ 1 , 0 < α t < 1 0<\alpha_t<1 0 < α t < 1 , 0 ≤ β t < 1 0\le\beta_t<1 0 ≤ β t < 1 , and γ t ≤ 1 / γ S \gamma_t\le1/\sqrt{\gamma_S} γ t ≤ 1/ γ S , we obtain
∥ M t ∥ ≤ ∥ I − Z t ∥ + ( 1 − α t ) ( 1 − β t ) + ( 1 − α t ) γ t ∥ Z t ∥ + ( α t + β t − α t β t ) + α t γ t ∥ Z t ∥ ≤ 2 + 1 γ S . \lVert M_t\rVert\le\lVert I-Z_t\rVert+(1-\alpha_t)(1-\beta_t)+(1-\alpha_t)\gamma_t\lVert Z_t\rVert+(\alpha_t+\beta_t-\alpha_t\beta_t)+\alpha_t\gamma_t\lVert Z_t\rVert\le2+\frac{1}{\sqrt{\gamma_S}}. ∥ M t ∥ ≤ ∥ I − Z t ∥ + ( 1 − α t ) ( 1 − β t ) + ( 1 − α t ) γ t ∥ Z t ∥ + ( α t + β t − α t β t ) + α t γ t ∥ Z t ∥ ≤ 2 + γ S 1 .
The same bound also holds for ∥ M ⋆ ∥ \lVert M^\star\rVert ∥ M ⋆ ∥ . Therefore, a telescoping expansion gives us
∥ C t − C ⋆ ∥ = ∥ M t τ − ( M ⋆ ) τ ∥ = ∥ ∑ k = 0 τ − 1 M t τ − 1 − k ( M t − M ⋆ ) ( M ⋆ ) k ∥ ≤ ( ∑ k = 0 τ − 1 ∥ M t ∥ τ − 1 − k ∥ M ⋆ ∥ k ) ∥ M t − M ⋆ ∥ ≲ ∥ M t − M ⋆ ∥ ≲ ∥ B t − B ⋆ ∥ . (62) \begin{aligned}
\lVert C_t-C^\star\rVert
&=\lVert M_t^\tau-(M^\star)^\tau\rVert\\
&=\left\lVert\sum_{k=0}^{\tau-1}M_t^{\tau-1-k}(M_t-M^\star)(M^\star)^k\right\rVert\\
&\le\left(\sum_{k=0}^{\tau-1}\lVert M_t\rVert^{\tau-1-k}\lVert M^\star\rVert^k\right)\lVert M_t-M^\star\rVert\\
&\lesssim\lVert M_t-M^\star\rVert\lesssim\lVert B_t-B^\star\rVert. \tag*{(62)}
\end{aligned} ∥ C t − C ⋆ ∥ = ∥ M t τ − ( M ⋆ ) τ ∥ = k = 0 ∑ τ − 1 M t τ − 1 − k ( M t − M ⋆ ) ( M ⋆ ) k ≤ ( k = 0 ∑ τ − 1 ∥ M t ∥ τ − 1 − k ∥ M ⋆ ∥ k ) ∥ M t − M ⋆ ∥ ≲ ∥ M t − M ⋆ ∥ ≲ ∥ B t − B ⋆ ∥ . (62)
Finally, by the definitions of K t K_t K t and K ⋆ K^\star K ⋆ , there exists a deterministic constant L K > 0 L_K > 0 L K > 0 such that
∥ K t − K ⋆ ∥ = ∥ ( I 0 ) ( C t − C ⋆ ) ( I I ) ∥ ≤ 2 ∥ C t − C ⋆ ∥ ≤ L K ∥ B t − B ⋆ ∥ . (63) \lVert K_t-K^\star\rVert=\left\lVert(I\ \ 0)(C_t-C^\star)\begin{pmatrix} I \\ I \end{pmatrix}\right\rVert\leq\sqrt{2}\lVert C_t-C^\star\rVert\leq L_K\lVert B_t-B^\star\rVert. \tag*{(63)} ∥ K t − K ⋆ ∥ = ( I 0 ) ( C t − C ⋆ ) ( I I ) ≤ 2 ∥ C t − C ⋆ ∥ ≤ L K ∥ B t − B ⋆ ∥ . (63)
Since B t → a . s . B ⋆ B_t \mathrel{\overset{\mathrm{a.s.}}{\to}} B^\star B t → a.s. B ⋆ , we conclude that
K t → a . s . K ⋆ . K_t \mathrel{\overset{\mathrm{a.s.}}{\to}} K^\star. K t → a.s. K ⋆ .
Symmetry and bound of K ⋆ K^\star K ⋆ . Recall from the proof of Wang et al. [34 ] that U t ⊤ K t U t = diag ( p t , τ ( z t , 1 ) , … , p t , τ ( z t , d ) ) U_t^\top K_t U_t=\operatorname{diag}(p_{t,\tau}(z_{t,1}),\ldots,p_{t,\tau}(z_{t,d})) U t ⊤ K t U t = diag ( p t , τ ( z t , 1 ) , … , p t , τ ( z t , d )) , implying K t K_t K t is symmetric. Since K t → a . s . K ⋆ K_t \mathrel{\overset{\mathrm{a.s.}}{\to}} K^\star K t → a.s. K ⋆ , K ⋆ K^\star K ⋆ is symmetric.
If μ t = 1 \mu_t=1 μ t = 1 , Wang et al. [34 ] gives K t = 0 K_t=0 K t = 0 . If μ t < 1 \mu_t<1 μ t < 1 , Wang et al. [34 ] and the condition on τ \tau τ give
∥ K t ∥ ≤ 2 τ ( 1 − μ t ν t ) τ − 2 ≤ γ B , t 4 Γ B , t . \lVert K_t\rVert\leq2\tau\left(1-\sqrt{\frac{\mu_t}{\nu_t}}\right)^{\tau-2}\leq\frac{\gamma_{B,t}}{4\Gamma_{B,t}}. ∥ K t ∥ ≤ 2 τ ( 1 − ν t μ t ) τ − 2 ≤ 4 Γ B , t γ B , t .
Therefore, ∥ K t ∥ ≤ 1 / 4 \lVert K_t\rVert\leq1/4 ∥ K t ∥ ≤ 1/4 for every t ≥ 0 t\geq0 t ≥ 0 . Taking the almost-sure limit yields
∥ K ⋆ ∥ ≤ 1 4 < 1. \lVert K^\star\rVert\leq\frac{1}{4}<1. ∥ K ⋆ ∥ ≤ 4 1 < 1.
This completes the proof.
Proof of Lemma D.2
For the first result, we have
∥ Z t − 1 − ( Z ⋆ ) − 1 ∥ = ∥ Z t − 1 ( Z ⋆ − Z t ) ( Z ⋆ ) − 1 ∥ ≤ ∥ Z t − 1 ∥ ∥ Z t − Z ⋆ ∥ ∥ ( Z ⋆ ) − 1 ∥ ≤ 1 γ S 2 ∥ Z t − Z ⋆ ∥ , \left\lVert Z_t^{-1}-(Z^\star)^{-1}\right\rVert=\left\lVert Z_t^{-1}(Z^\star-Z_t)(Z^\star)^{-1}\right\rVert\leq\lVert Z_t^{-1}\rVert\lVert Z_t-Z^\star\rVert\lVert(Z^\star)^{-1}\rVert\leq\frac{1}{\gamma_S^2}\lVert Z_t-Z^\star\rVert, Z t − 1 − ( Z ⋆ ) − 1 = Z t − 1 ( Z ⋆ − Z t ) ( Z ⋆ ) − 1 ≤ ∥ Z t − 1 ∥ ∥ Z t − Z ⋆ ∥ ∥( Z ⋆ ) − 1 ∥ ≤ γ S 2 1 ∥ Z t − Z ⋆ ∥ ,
where the last inequality follows from Z t ⪰ γ S I Z_t\succeq\gamma_S I Z t ⪰ γ S I in Assumption 3.4 and Z ⋆ ⪰ γ S I Z^\star\succeq\gamma_S I Z ⋆ ⪰ γ S I , which is due to Z t → a . s . Z ⋆ Z_t\mathrel{\overset{\mathrm{a.s.}}{\to}}Z^\star Z t → a.s. Z ⋆ established above.
For the second result, we proceed in two steps.
X − 1 / 2 = 1 π ∫ 0 ∞ s − 1 / 2 ( X + s I ) − 1 d s . (64) X^{-1/2}=\frac{1}{\pi}\int_0^\infty s^{-1/2}(X+sI)^{-1}\,\mathrm{d}s. \tag*{(64)} X − 1/2 = π 1 ∫ 0 ∞ s − 1/2 ( X + s I ) − 1 d s . (64)
Indeed, for any scalar x > 0 x>0 x > 0 , the following identity holds:
x − 1 / 2 = 1 π ∫ 0 ∞ s − 1 / 2 ( x + s ) − 1 d s . x^{-1/2}=\frac{1}{\pi}\int_0^\infty s^{-1/2}(x+s)^{-1}\,\mathrm{d}s. x − 1/2 = π 1 ∫ 0 ∞ s − 1/2 ( x + s ) − 1 d s .
To verify this identity, let s = x u 2 s=xu^2 s = x u 2 , so that d s = 2 x u d u \mathrm{d}s=2xu\,\mathrm{d}u d s = 2 xu d u . Then,
1 π ∫ 0 ∞ s − 1 / 2 ( x + s ) − 1 d s = 1 π ∫ 0 ∞ ( x u 2 ) − 1 / 2 ( x + x u 2 ) − 1 2 x u d u = 1 π ∫ 0 ∞ 2 x 1 / 2 ( 1 + u 2 ) d u = 2 π x 1 / 2 arctan ( u ) ∣ 0 ∞ = x − 1 / 2 . (65) \begin{aligned}
\frac{1}{\pi}\int_0^\infty s^{-1/2}(x+s)^{-1}\,\mathrm{d}s
&=\frac{1}{\pi}\int_0^\infty(xu^2)^{-1/2}(x+xu^2)^{-1}2xu\,\mathrm{d}u\\
&=\frac{1}{\pi}\int_0^\infty\frac{2}{x^{1/2}(1+u^2)}\,\mathrm{d}u\\
&=\frac{2}{\pi x^{1/2}}\arctan(u)\bigg|_0^\infty=x^{-1/2}. \tag*{(65)}
\end{aligned} π 1 ∫ 0 ∞ s − 1/2 ( x + s ) − 1 d s = π 1 ∫ 0 ∞ ( x u 2 ) − 1/2 ( x + x u 2 ) − 1 2 xu d u = π 1 ∫ 0 ∞ x 1/2 ( 1 + u 2 ) 2 d u = π x 1/2 2 arctan ( u ) 0 ∞ = x − 1/2 . (65)
We now extend (D.18) to matrices. Since X X X is positive definite, it admits an eigendecomposition X = U D U ⊤ X=UDU^\top X = U D U ⊤ , where D D D is diagonal and U U U is orthogonal. Then, X − 1 / 2 = U D − 1 / 2 U ⊤ X^{-1/2}=UD^{-1/2}U^\top X − 1/2 = U D − 1/2 U ⊤ , and
1 π ∫ 0 ∞ s − 1 / 2 ( X + s I ) − 1 d s = 1 π ∫ 0 ∞ s − 1 / 2 ( U D U ⊤ + s U I U ⊤ ) − 1 d s = 1 π ∫ 0 ∞ s − 1 / 2 { U ( D + s I ) U ⊤ } − 1 d s = 1 π ∫ 0 ∞ s − 1 / 2 U ( D + s I ) − 1 U ⊤ d s = U { 1 π ∫ 0 ∞ s − 1 / 2 ( D + s I ) − 1 d s } U ⊤ . (66) \begin{aligned}
\frac{1}{\pi}\int_0^\infty s^{-1/2}(X+sI)^{-1}\,\mathrm{d}s
&=\frac{1}{\pi}\int_0^\infty s^{-1/2}(UDU^\top+sUIU^\top)^{-1}\,\mathrm{d}s\\
&=\frac{1}{\pi}\int_0^\infty s^{-1/2}\{U(D+sI)U^\top\}^{-1}\,\mathrm{d}s\\
&=\frac{1}{\pi}\int_0^\infty s^{-1/2}U(D+sI)^{-1}U^\top\,\mathrm{d}s\\
&=U\left\{\frac{1}{\pi}\int_0^\infty s^{-1/2}(D+sI)^{-1}\,\mathrm{d}s\right\}U^\top. \tag*{(66)}
\end{aligned} π 1 ∫ 0 ∞ s − 1/2 ( X + s I ) − 1 d s = π 1 ∫ 0 ∞ s − 1/2 ( U D U ⊤ + s U I U ⊤ ) − 1 d s = π 1 ∫ 0 ∞ s − 1/2 { U ( D + s I ) U ⊤ } − 1 d s = π 1 ∫ 0 ∞ s − 1/2 U ( D + s I ) − 1 U ⊤ d s = U { π 1 ∫ 0 ∞ s − 1/2 ( D + s I ) − 1 d s } U ⊤ . (66)
Since ( D + s I ) − 1 (D+sI)^{-1} ( D + s I ) − 1 is diagonal, the integral is evaluated entry-wise. By (D.18), its i i i th diagonal entry equals λ i − 1 / 2 \lambda_i^{-1/2} λ i − 1/2 , where λ i \lambda_i λ i is the i i i th eigenvalue of X X X . Therefore, applying (D.19) gives us
1 π ∫ 0 ∞ s − 1 / 2 ( X + s I ) − 1 d s = U { 1 π ∫ 0 ∞ s − 1 / 2 ( D + s I ) − 1 d s } U ⊤ = U D − 1 / 2 U ⊤ = X − 1 / 2 . \frac{1}{\pi}\int_{0}^{\infty}s^{-1/2}(X+sI)^{-1}\,ds=U\left\{\frac{1}{\pi}\int_{0}^{\infty}s^{-1/2}(D+sI)^{-1}\,ds\right\}U^{\top}=UD^{-1/2}U^{\top}=X^{-1/2}. π 1 ∫ 0 ∞ s − 1/2 ( X + s I ) − 1 d s = U { π 1 ∫ 0 ∞ s − 1/2 ( D + s I ) − 1 d s } U ⊤ = U D − 1/2 U ⊤ = X − 1/2 .
∥ Z t − 1 / 2 − ( Z ⋆ ) − 1 / 2 ∥ = ∥ 1 π ∫ 0 ∞ y − 1 / 2 { ( Z t + y I ) − 1 − ( Z ⋆ + y I ) − 1 } d y ∥ = ∥ 1 π ∫ 0 ∞ y − 1 / 2 ( Z t + y I ) − 1 ( Z ⋆ − Z t ) ( Z ⋆ + y I ) − 1 d y ∥ ≤ ∥ Z ⋆ − Z t ∥ π ∫ 0 ∞ y − 1 / 2 ∥ ( Z t + y I ) − 1 ∥ ∥ ( Z ⋆ + y I ) − 1 ∥ d y ≤ ∥ Z ⋆ − Z t ∥ π ∫ 0 ∞ y − 1 / 2 ( γ S + y ) 2 d y , \begin{aligned}
\left\|Z_t^{-1/2}-(Z^\star)^{-1/2}\right\|
&=\left\|\frac{1}{\pi}\int_{0}^{\infty}y^{-1/2}\left\{(Z_t+yI)^{-1}-(Z^\star+yI)^{-1}\right\}\,dy\right\|\\
&=\left\|\frac{1}{\pi}\int_{0}^{\infty}y^{-1/2}(Z_t+yI)^{-1}(Z^\star-Z_t)(Z^\star+yI)^{-1}\,dy\right\|\\
&\leq\frac{\left\|Z^\star-Z_t\right\|}{\pi}\int_{0}^{\infty}y^{-1/2}\left\|(Z_t+yI)^{-1}\right\|\left\|(Z^\star+yI)^{-1}\right\|\,dy\\
&\leq\frac{\left\|Z^\star-Z_t\right\|}{\pi}\int_{0}^{\infty}\frac{y^{-1/2}}{(\gamma_S+y)^2}\,dy,
\end{aligned} Z t − 1/2 − ( Z ⋆ ) − 1/2 = π 1 ∫ 0 ∞ y − 1/2 { ( Z t + y I ) − 1 − ( Z ⋆ + y I ) − 1 } d y = π 1 ∫ 0 ∞ y − 1/2 ( Z t + y I ) − 1 ( Z ⋆ − Z t ) ( Z ⋆ + y I ) − 1 d y ≤ π ∥ Z ⋆ − Z t ∥ ∫ 0 ∞ y − 1/2 ( Z t + y I ) − 1 ( Z ⋆ + y I ) − 1 d y ≤ π ∥ Z ⋆ − Z t ∥ ∫ 0 ∞ ( γ S + y ) 2 y − 1/2 d y ,
where the last inequality follows from Z t ⪰ γ S I Z_t\succeq\gamma_S I Z t ⪰ γ S I and Z ⋆ ⪰ γ S I Z^\star\succeq\gamma_S I Z ⋆ ⪰ γ S I . To evaluate the integral, we let y = γ S u 2 y=\gamma_Su^2 y = γ S u 2 and hence obtain the following:
∫ 0 ∞ y − 1 / 2 ( γ S + y ) 2 d y = ∫ 0 ∞ ( γ S u 2 ) − 1 / 2 ( γ S + γ S u 2 ) 2 ( 2 γ S u ) d u = 2 γ S 3 / 2 ∫ 0 ∞ 1 ( 1 + u 2 ) 2 d u . \int_{0}^{\infty}\frac{y^{-1/2}}{(\gamma_S+y)^2}\,dy
=\int_{0}^{\infty}\frac{(\gamma_Su^2)^{-1/2}}{(\gamma_S+\gamma_Su^2)^2}(2\gamma_Su)\,du
=\frac{2}{\gamma_S^{3/2}}\int_{0}^{\infty}\frac{1}{(1+u^2)^2}\,du. ∫ 0 ∞ ( γ S + y ) 2 y − 1/2 d y = ∫ 0 ∞ ( γ S + γ S u 2 ) 2 ( γ S u 2 ) − 1/2 ( 2 γ S u ) d u = γ S 3/2 2 ∫ 0 ∞ ( 1 + u 2 ) 2 1 d u .
Letting u = tan θ u=\tan\theta u = tan θ further yields
∫ 0 ∞ 1 ( 1 + u 2 ) 2 d u = ∫ 0 π / 2 sec 2 θ ( 1 + tan 2 θ ) 2 d θ = ∫ 0 π / 2 1 sec 2 θ d θ = ∫ 0 π / 2 cos 2 θ d θ = ∫ 0 π / 2 1 + cos ( 2 θ ) 2 d θ = π 4 . \begin{aligned}
\int_{0}^{\infty}\frac{1}{(1+u^2)^2}\,du
&=\int_{0}^{\pi/2}\frac{\sec^2\theta}{(1+\tan^2\theta)^2}\,d\theta
=\int_{0}^{\pi/2}\frac{1}{\sec^2\theta}\,d\theta\\
&=\int_{0}^{\pi/2}\cos^2\theta\,d\theta
=\int_{0}^{\pi/2}\frac{1+\cos(2\theta)}{2}\,d\theta
=\frac{\pi}{4}.
\end{aligned} ∫ 0 ∞ ( 1 + u 2 ) 2 1 d u = ∫ 0 π /2 ( 1 + tan 2 θ ) 2 sec 2 θ d θ = ∫ 0 π /2 sec 2 θ 1 d θ = ∫ 0 π /2 cos 2 θ d θ = ∫ 0 π /2 2 1 + cos ( 2 θ ) d θ = 4 π .
Combining the preceding displays, we conclude that
∥ Z t − 1 / 2 − ( Z ⋆ ) − 1 / 2 ∥ ≤ ∥ Z ⋆ − Z t ∥ π ∫ 0 ∞ y − 1 / 2 ( γ S + y ) 2 d y = 1 2 γ S 3 / 2 ∥ Z ⋆ − Z t ∥ . \left\|Z_t^{-1/2}-(Z^\star)^{-1/2}\right\|
\leq\frac{\left\|Z^\star-Z_t\right\|}{\pi}\int_{0}^{\infty}\frac{y^{-1/2}}{(\gamma_S+y)^2}\,dy
=\frac{1}{2\gamma_S^{3/2}}\left\|Z^\star-Z_t\right\|. Z t − 1/2 − ( Z ⋆ ) − 1/2 ≤ π ∥ Z ⋆ − Z t ∥ ∫ 0 ∞ ( γ S + y ) 2 y − 1/2 d y = 2 γ S 3/2 1 ∥ Z ⋆ − Z t ∥ .
This completes the proof.
Proof of Lemma 4.2
Recall that Δ t = x t − x ⋆ \Delta_t=x_t-x^\star Δ t = x t − x ⋆ , R = I − K ⋆ \mathcal{R}=I-K^\star R = I − K ⋆ and λ m = λ min ( R ) \lambda_m=\lambda_{\min}(\mathcal{R}) λ m = λ m i n ( R ) . By Lemma 4.1, K ⋆ K^\star K ⋆ is symmetric and ∥ K ⋆ ∥ < 1 \left\|K^\star\right\|<1 ∥ K ⋆ ∥ < 1 . Hence, R \mathcal{R} R is symmetric positive definite and λ m > 0 \lambda_m>0 λ m > 0 .
Also, define the realized and conditional-mean inner-solver operators as
Q ~ t : = ( I − K ~ t ) B t − 1 , Q t : = ( I − K t ) B t − 1 . \widetilde{Q}_t:=(I-\widetilde{K}_t)B_t^{-1},\qquad Q_t:=(I-K_t)B_t^{-1}. Q t := ( I − K t ) B t − 1 , Q t := ( I − K t ) B t − 1 .
Conditional on F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 , B t B_t B t is fixed. Therefore, by Lemma C.1, we obtain
E [ Q ~ t ∣ F t − 0.5 ] = E [ ( I − K ~ t ) B t − 1 ∣ F t − 0.5 ] = { I − E [ K ~ t ∣ F t − 0.5 ] } B t − 1 = ( I − K t ) B t − 1 = Q t . (67) \begin{aligned}
\mathbb{E}[\widetilde{Q}_t\mid\mathcal{F}_{t-0.5}]
&=\mathbb{E}[(I-\widetilde{K}_t)B_t^{-1}\mid\mathcal{F}_{t-0.5}]\\
&=\left\{I-\mathbb{E}[\widetilde{K}_t\mid\mathcal{F}_{t-0.5}]\right\}B_t^{-1}
=(I-K_t)B_t^{-1}=Q_t. \tag*{(67)}
\end{aligned} E [ Q t ∣ F t − 0.5 ] = E [( I − K t ) B t − 1 ∣ F t − 0.5 ] = { I − E [ K t ∣ F t − 0.5 ] } B t − 1 = ( I − K t ) B t − 1 = Q t . (67)
Let r ∈ ( 0 , γ H / 2 ) r\in(0,\gamma_H/2) r ∈ ( 0 , γ H /2 ) be chosen below. On the event { ∥ B t − B ⋆ ∥ ≤ r } \{\left\|B_t-B^\star\right\|\leq r\} { ∥ B t − B ⋆ ∥ ≤ r } , Assumption 3.1 gives B ⋆ ⪰ γ H I B^\star\succeq\gamma_HI B ⋆ ⪰ γ H I and hence B t ⪰ ( γ H − r ) I ⪰ ( γ H / 2 ) I B_t\succeq(\gamma_H-r)I\succeq(\gamma_H/2)I B t ⪰ ( γ H − r ) I ⪰ ( γ H /2 ) I . Therefore, ∥ B t − 1 ∥ ≤ 2 / γ H \left\|B_t^{-1}\right\|\leq2/\gamma_H B t − 1 ≤ 2/ γ H , and the following bounds hold on this event. By (C.8), we have ∥ K ~ t ∥ ≤ C K \left\|\widetilde{K}_t\right\|\leq C_K K t ≤ C K . Moreover, Wang et al. [34 ] and the condition on τ \tau τ ensure that ∥ K t ∥ ≤ 2 τ ( 1 − μ t / ν t ) − 2 ≤ γ B , t / ( 4 Υ B , t ) ≤ 1 / 4 \left\|K_t\right\|\leq2\tau\left(1-\sqrt{\mu_t/\nu_t}\right)^{-2}\leq\gamma_{B,t}/(4\Upsilon_{B,t})\leq1/4 ∥ K t ∥ ≤ 2 τ ( 1 − μ t / ν t ) − 2 ≤ γ B , t / ( 4 Υ B , t ) ≤ 1/4 . On { ∥ B t − B ⋆ ∥ ≤ r } \{\left\|B_t-B^\star\right\|\leq r\} { ∥ B t − B ⋆ ∥ ≤ r } , it follows that
∥ Q ~ t ∥ = ∥ ( I − K ~ t ) B t − 1 ∥ ≤ ( 1 + ∥ K ~ t ∥ ) ∥ B t − 1 ∥ ≤ 2 ( 1 + C K ) γ H ≲ 1 , \left\|\widetilde{Q}_t\right\|=\left\|(I-\widetilde{K}_t)B_t^{-1}\right\|\leq(1+\left\|\widetilde{K}_t\right\|)\left\|B_t^{-1}\right\|\leq\frac{2(1+C_K)}{\gamma_H}\lesssim1, Q t = ( I − K t ) B t − 1 ≤ ( 1 + K t ) B t − 1 ≤ γ H 2 ( 1 + C K ) ≲ 1 ,
∥ Q t ∥ = ∥ ( I − K t ) B t − 1 ∥ ≤ ( 1 + ∥ K t ∥ ) ∥ B t − 1 ∥ ≤ 5 2 γ H ≲ 1. (68) \left\|Q_t\right\|=\left\|(I-K_t)B_t^{-1}\right\|\leq(1+\left\|K_t\right\|)\left\|B_t^{-1}\right\|\leq\frac{5}{2\gamma_H}\lesssim1. \tag*{(68)} ∥ Q t ∥ = ( I − K t ) B t − 1 ≤ ( 1 + ∥ K t ∥ ) B t − 1 ≤ 2 γ H 5 ≲ 1. (68)
On the event { ∥ B t − B ⋆ ∥ ≤ r } \{\lVert B_t-B^\star\rVert\le r\} {∥ B t − B ⋆ ∥ ≤ r } , we further bound the deviation of the conditional-mean operator from its limit as follows:
∥ Q t B ⋆ − R ∥ = ∥ ( I − K t ) B t − 1 B ⋆ − ( I − K ⋆ ) ∥ = ∥ ( I − K t ) B t − 1 ( B ⋆ − B t ) + ( K ⋆ − K t ) ∥ ≤ ∥ I − K t ∥ ∥ B t − 1 ∥ ∥ B ⋆ − B t ∥ + ∥ K t − K ⋆ ∥ ≤ { 2 ( 1 + ∥ K t ∥ ) γ H + L K } ∥ B t − B ⋆ ∥ ≤ { 5 2 γ H + L K } ∥ B t − B ⋆ ∥ . (69) \begin{aligned}
\lVert Q_tB^\star-R\rVert
&=\lVert(I-K_t)B_t^{-1}B^\star-(I-K^\star)\rVert\\
&=\lVert(I-K_t)B_t^{-1}(B^\star-B_t)+(K^\star-K_t)\rVert\\
&\le\lVert I-K_t\rVert\lVert B_t^{-1}\rVert\lVert B^\star-B_t\rVert+\lVert K_t-K^\star\rVert\\
&\le\left\{\frac{2(1+\lVert K_t\rVert)}{\gamma_H}+L_K\right\}\lVert B_t-B^\star\rVert\\
&\le\left\{\frac{5}{2\gamma_H}+L_K\right\}\lVert B_t-B^\star\rVert. \tag*{(69)}
\end{aligned} ∥ Q t B ⋆ − R ∥ = ∥( I − K t ) B t − 1 B ⋆ − ( I − K ⋆ )∥ = ∥( I − K t ) B t − 1 ( B ⋆ − B t ) + ( K ⋆ − K t )∥ ≤ ∥ I − K t ∥ ∥ B t − 1 ∥ ∥ B ⋆ − B t ∥ + ∥ K t − K ⋆ ∥ ≤ { γ H 2 ( 1 + ∥ K t ∥) + L K } ∥ B t − B ⋆ ∥ ≤ { 2 γ H 5 + L K } ∥ B t − B ⋆ ∥ . (69)
Here, the first equality follows from the definitions of Q t Q_t Q t and R R R ; the fourth inequality uses ∥ B t − 1 ∥ ≤ 2 / γ H \lVert B_t^{-1}\rVert\le2/\gamma_H ∥ B t − 1 ∥ ≤ 2/ γ H on { ∥ B t − B ⋆ ∥ ≤ r } \{\lVert B_t-B^\star\rVert\le r\} {∥ B t − B ⋆ ∥ ≤ r } and (D.16); and the last inequality follows from ∥ K t ∥ ≤ 1 / 4 \lVert K_t\rVert\le1/4 ∥ K t ∥ ≤ 1/4 .
Also, by the Υ L \Upsilon_L Υ L -Lipschitz continuity of ∇ 2 f \nabla^2f ∇ 2 f , we obtain
∥ ∇ f t − B ⋆ Δ t ∥ = ∥ ∫ 0 1 { ∇ 2 f ( x ⋆ + s Δ t ) − B ⋆ } Δ t d s ∥ ≤ ∫ 0 1 ∥ ∇ 2 f ( x ⋆ + s Δ t ) − ∇ 2 f ( x ⋆ ) ∥ ∥ Δ t ∥ d s ≤ Υ L ∫ 0 1 s ∥ Δ t ∥ 2 d s = Υ L 2 ∥ Δ t ∥ 2 . (70) \begin{aligned}
\lVert\nabla f_t-B^\star\Delta_t\rVert
&=\left\lVert\int_0^1\{\nabla^2f(x^\star+s\Delta_t)-B^\star\}\Delta_t\,ds\right\rVert\\
&\le\int_0^1\lVert\nabla^2f(x^\star+s\Delta_t)-\nabla^2f(x^\star)\rVert\lVert\Delta_t\rVert\,ds\\
&\le\Upsilon_L\int_0^1s\lVert\Delta_t\rVert^2\,ds
=\frac{\Upsilon_L}{2}\lVert\Delta_t\rVert^2. \tag*{(70)}
\end{aligned} ∥ ∇ f t − B ⋆ Δ t ∥ = ∫ 0 1 { ∇ 2 f ( x ⋆ + s Δ t ) − B ⋆ } Δ t d s ≤ ∫ 0 1 ∥ ∇ 2 f ( x ⋆ + s Δ t ) − ∇ 2 f ( x ⋆ )∥ ∥ Δ t ∥ d s ≤ Υ L ∫ 0 1 s ∥ Δ t ∥ 2 d s = 2 Υ L ∥ Δ t ∥ 2 . (70)
Combining (D.21), (D.22), and (D.23), we further obtain the following bound on { ∥ B t − B ⋆ ∥ ≤ r } \{\lVert B_t-B^\star\rVert\le r\} {∥ B t − B ⋆ ∥ ≤ r } :
∥ Q t ∇ f t − R Δ t ∥ = ∥ Q t ( ∇ f t − B ⋆ Δ t ) + ( Q t B ⋆ − R ) Δ t ∥ ≤ ∥ Q t ∥ ∥ ∇ f t − B ⋆ Δ t ∥ + ∥ Q t B ⋆ − R ∥ ∥ Δ t ∥ ≤ 5 Υ L 4 γ H ∥ Δ t ∥ 2 + { 5 2 γ H + L K } ∥ B t − B ⋆ ∥ ∥ Δ t ∥ ≲ { ∥ Δ t ∥ + ∥ B t − B ⋆ ∥ } ∥ Δ t ∥ . (71) \begin{aligned}
\lVert Q_t\nabla f_t-R\Delta_t\rVert
&=\lVert Q_t(\nabla f_t-B^\star\Delta_t)+(Q_tB^\star-R)\Delta_t\rVert\\
&\le\lVert Q_t\rVert\lVert\nabla f_t-B^\star\Delta_t\rVert+\lVert Q_tB^\star-R\rVert\lVert\Delta_t\rVert\\
&\le\frac{5\Upsilon_L}{4\gamma_H}\lVert\Delta_t\rVert^2+\left\{\frac{5}{2\gamma_H}+L_K\right\}\lVert B_t-B^\star\rVert\lVert\Delta_t\rVert\\
&\lesssim\{\lVert\Delta_t\rVert+\lVert B_t-B^\star\rVert\}\lVert\Delta_t\rVert. \tag*{(71)}
\end{aligned} ∥ Q t ∇ f t − R Δ t ∥ = ∥ Q t ( ∇ f t − B ⋆ Δ t ) + ( Q t B ⋆ − R ) Δ t ∥ ≤ ∥ Q t ∥ ∥ ∇ f t − B ⋆ Δ t ∥ + ∥ Q t B ⋆ − R ∥ ∥ Δ t ∥ ≤ 4 γ H 5 Υ L ∥ Δ t ∥ 2 + { 2 γ H 5 + L K } ∥ B t − B ⋆ ∥ ∥ Δ t ∥ ≲ {∥ Δ t ∥ + ∥ B t − B ⋆ ∥} ∥ Δ t ∥ . (71)
Now we localize the analysis by introducing the stopping time: for k ≥ 1 k\ge1 k ≥ 1 and the radius r ∈ ( 0 , γ H / 2 ) r\in(0,\gamma_H/2) r ∈ ( 0 , γ H /2 ) which will be specified below,
τ k , r : = inf { t ≥ k : ∥ Δ t ∥ > r O R ∥ B t − B ⋆ ∥ > r O R ∥ e t − 1 ∥ > r O R ∥ Δ t − 1 ∥ > r O R ∥ B ~ t − B ⋆ ∥ F > r O R γ B , t > λ min ( B ⋆ ) O R Υ B , t < λ max ( B ⋆ ) } . (72) \tau_{k,r}:=\inf\left\{t\ge k:
\begin{array}{l}
\lVert\Delta_t\rVert>r\ \mathrm{OR}\ \lVert B_t-B^\star\rVert>r\ \mathrm{OR}\ \lVert e_{t-1}\rVert>r\\
\mathrm{OR}\ \lVert\Delta_{t-1}\rVert>r\ \mathrm{OR}\ \lVert\widetilde B_t-B^\star\rVert_F>r\\
\mathrm{OR}\ \gamma_{B,t}>\lambda_{\min}(B^\star)\ \mathrm{OR}\ \Upsilon_{B,t}<\lambda_{\max}(B^\star)
\end{array}
\right\}. \tag*{(72)} τ k , r := inf ⎩ ⎨ ⎧ t ≥ k : ∥ Δ t ∥ > r OR ∥ B t − B ⋆ ∥ > r OR ∥ e t − 1 ∥ > r OR ∥ Δ t − 1 ∥ > r OR ∥ B t − B ⋆ ∥ F > r OR γ B , t > λ m i n ( B ⋆ ) OR Υ B , t < λ m a x ( B ⋆ ) ⎭ ⎬ ⎫ . (72)
Since Δ t − 1 \Delta_{t-1} Δ t − 1 , Δ t \Delta_t Δ t , B ~ t \widetilde B_t B t , B t B_t B t , e t − 1 e_{t-1} e t − 1 , γ B , t \gamma_{B,t} γ B , t , and Υ B , t \Upsilon_{B,t} Υ B , t are F t − 1 \mathcal F_{t-1} F t − 1 -measurable, τ k , r \tau_{k,r} τ k , r is a stopping time and { τ k , r > t } ∈ F t − 1 \{\tau_{k,r}>t\}\in\mathcal F_{t-1} { τ k , r > t } ∈ F t − 1 . On { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , we have ∥ B t − B ⋆ ∥ ≤ r \lVert B_t-B^\star\rVert\le r ∥ B t − B ⋆ ∥ ≤ r , so the local bounds above hold. The clipping interval also contains B ⋆ B^\star B ⋆ , and hence ∥ B t − B ⋆ ∥ F ≤ ∥ B ~ t − B ⋆ ∥ F \lVert B_t-B^\star\rVert_F\le\lVert\widetilde B_t-B^\star\rVert_F ∥ B t − B ⋆ ∥ F ≤ ∥ B t − B ⋆ ∥ F . Moreover, Theorem 3.5 and Lemma 4.1 together ensure
Δ t → a . s . 0 , B t → a . s . B ⋆ , e t → a . s . 0. \Delta_t\xrightarrow{\mathrm{a.s.}}0,\qquad B_t\xrightarrow{\mathrm{a.s.}}B^\star,\qquad e_t\xrightarrow{\mathrm{a.s.}}0. Δ t a.s. 0 , B t a.s. B ⋆ , e t a.s. 0.
The proof of Lemma 4.1 also gives ∥ B ~ t − B ⋆ ∥ F → a . s . 0 \lVert\widetilde B_t-B^\star\rVert_F\xrightarrow{\mathrm{a.s.}}0 ∥ B t − B ⋆ ∥ F a.s. 0 , and the clipping interval contains B ⋆ B^\star B ⋆ for all sufficiently large t t t almost surely. Thus, for each fixed r ∈ ( 0 , γ H / 2 ) r\in(0,\gamma_H/2) r ∈ ( 0 , γ H /2 ) , τ k , r = ∞ \tau_{k,r}=\infty τ k , r = ∞ for all sufficiently large k k k almost surely, which implies
P ( τ k , r < ∞ ) ⟶ 0 as k → ∞ . (73) \mathbb{P}(\tau_{k,r}<\infty)\longrightarrow0\qquad\text{as }k\to\infty. \tag*{(73)} P ( τ k , r < ∞ ) ⟶ 0 as k → ∞. (73)
Once r r r is chosen below, for any ϵ > 0 \epsilon>0 ϵ > 0 we can choose k = k ( ϵ , r ) k=k(\epsilon,r) k = k ( ϵ , r ) sufficiently large such that P ( τ k , r < ∞ ) < ϵ / 2 \mathbb{P}(\tau_{k,r}<\infty)<\epsilon/2 P ( τ k , r < ∞ ) < ϵ /2 . For every t ≥ k t\ge k t ≥ k , we then have
P ( τ k , r ≤ t ) ≤ P ( τ k , r < ∞ ) < ϵ 2 . (74) \mathbb{P}(\tau_{k,r}\le t)\le\mathbb{P}(\tau_{k,r}<\infty)<\frac{\epsilon}{2}. \tag*{(74)} P ( τ k , r ≤ t ) ≤ P ( τ k , r < ∞ ) < 2 ϵ . (74)
Next we determine the radius r r r and index k k k step by step. We first choose a margin ε 0 > 0 \varepsilon_0 > 0 ε 0 > 0 . If φ < 1 \varphi< 1 φ < 1 , we could choose any ε 0 ∈ ( 0 , 1 ) \varepsilon_0 \in(0,1) ε 0 ∈ ( 0 , 1 ) . If φ = 1 \varphi= 1 φ = 1 , we choose ε 0 ∈ ( 0 , 1 ) \varepsilon_0 \in(0,1) ε 0 ∈ ( 0 , 1 ) sufficiently small such that
2 C φ ( 1 − ε 0 ) λ m > η , (75) 2C_{\varphi}(1-\varepsilon_0)\lambda_m > \eta, \tag*{(75)} 2 C φ ( 1 − ε 0 ) λ m > η , (75)
which is possible since C φ > η / ( 2 λ m ) C_{\varphi} > \eta/(2\lambda_m) C φ > η / ( 2 λ m ) . The precise choice of the localization radius r r r will be specified after deriving the preliminary recursion for the iterate error.
By the outer update, Lemma C.1 , and the definition of Q ~ t \widetilde{Q}_t Q t , we have
Δ t + 1 = Δ t − φ t Q ~ t ( ∇ f t + e t ) . \Delta_{t+1} = \Delta_t - \varphi_t\widetilde{Q}_t(\nabla f_t + e_t). Δ t + 1 = Δ t − φ t Q t ( ∇ f t + e t ) .
Conditional on F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 , the quantities ( Δ t , ∇ f t , e t , Q t ) (\Delta_t,\nabla f_t,e_t,Q_t) ( Δ t , ∇ f t , e t , Q t ) are fixed. Therefore, we obtain
E [ ∥ Δ t + 1 ∥ 2 ∣ F t − 0.5 ] = ∥ Δ t − φ t Q t ( ∇ f t + e t ) ∥ 2 + φ t 2 E [ ∥ ( Q ~ t − Q t ) ( ∇ f t + e t ) ∥ 2 ∣ F t − 0.5 ] . (76) \mathbb{E}[\|\Delta_{t+1}\|^2 \mid\mathcal{F}_{t-0.5}] = \|\Delta_t-\varphi_tQ_t(\nabla f_t+e_t)\|^2 + \varphi_t^2\mathbb{E}[\|(\widetilde{Q}_t-Q_t)(\nabla f_t+e_t)\|^2 \mid\mathcal{F}_{t-0.5}]. \tag*{(76)} E [ ∥ Δ t + 1 ∥ 2 ∣ F t − 0.5 ] = ∥ Δ t − φ t Q t ( ∇ f t + e t ) ∥ 2 + φ t 2 E [ ∥ ( Q t − Q t ) ( ∇ f t + e t ) ∥ 2 ∣ F t − 0.5 ] . (76)
Here, the cross term vanishes since E [ Q ~ t − Q t ∣ F t − 0.5 ] = 0 \mathbb{E}[\widetilde{Q}_t-Q_t\mid\mathcal{F}_{t-0.5}]=0 E [ Q t − Q t ∣ F t − 0.5 ] = 0 given in (D.20). Moreover,
Δ t − φ t Q t ( ∇ f t + e t ) = ( I − φ t R ) Δ t − φ t ( Q t ∇ f t − R Δ t ) − φ t Q t e t . \Delta_t-\varphi_tQ_t(\nabla f_t+e_t)=(I-\varphi_tR)\Delta_t-\varphi_t(Q_t\nabla f_t-R\Delta_t)-\varphi_tQ_te_t. Δ t − φ t Q t ( ∇ f t + e t ) = ( I − φ t R ) Δ t − φ t ( Q t ∇ f t − R Δ t ) − φ t Q t e t .
On the event { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , since R \mathcal{R} R is symmetric positive definite, for all sufficiently large t t t such that ∥ I − φ t R ∥ ≤ 1 \|I-\varphi_t\mathcal{R}\|\leq1 ∥ I − φ t R ∥ ≤ 1 , we have
∥ Δ t − φ t Q t ( ∇ f t + e t ) ∥ 2 = ∥ ( I − φ t R ) Δ t − φ t ( Q t ∇ f t − R Δ t ) − φ t Q t e t ∥ 2 ≤ ∥ ( I − φ t R ) Δ t ∥ 2 + 2 φ t ∥ Δ t ∥ ∥ Q t ∇ f t − R Δ t ∥ + 5 φ t γ H ∥ Δ t ∥ ∥ e t ∥ + 2 φ t 2 ∥ Q t ∇ f t − R Δ t ∥ 2 + 25 φ t 2 2 γ H 2 ∥ e t ∥ 2 ≤ ( 1 − 2 λ m φ t + ∥ R ∥ 2 φ t 2 ) ∥ Δ t ∥ 2 + 2 φ t ∥ Δ t ∥ ∥ Q t ∇ f t − R Δ t ∥ + 5 φ t γ H ∥ Δ t ∥ ∥ e t ∥ + 2 φ t 2 ∥ Q t ∇ f t − R Δ t ∥ 2 + 25 φ t 2 2 γ H 2 ∥ e t ∥ 2 , (77) \begin{aligned}
\|\Delta_t-\varphi_tQ_t(\nabla f_t+e_t)\|^2
&=\|(I-\varphi_t\mathcal{R})\Delta_t-\varphi_t(Q_t\nabla f_t-\mathcal{R}\Delta_t)-\varphi_tQ_te_t\|^2\\
&\leq\|(I-\varphi_t\mathcal{R})\Delta_t\|^2+2\varphi_t\|\Delta_t\|\|Q_t\nabla f_t-\mathcal{R}\Delta_t\|\\
&\quad+\frac{5\varphi_t}{\gamma_H}\|\Delta_t\|\|e_t\|+2\varphi_t^2\|Q_t\nabla f_t-\mathcal{R}\Delta_t\|^2+\frac{25\varphi_t^2}{2\gamma_H^2}\|e_t\|^2\\
&\leq(1-2\lambda_m\varphi_t+\|\mathcal{R}\|^2\varphi_t^2)\|\Delta_t\|^2+2\varphi_t\|\Delta_t\|\|Q_t\nabla f_t-\mathcal{R}\Delta_t\|\\
&\quad+\frac{5\varphi_t}{\gamma_H}\|\Delta_t\|\|e_t\|+2\varphi_t^2\|Q_t\nabla f_t-\mathcal{R}\Delta_t\|^2+\frac{25\varphi_t^2}{2\gamma_H^2}\|e_t\|^2, \tag*{(77)}
\end{aligned} ∥ Δ t − φ t Q t ( ∇ f t + e t ) ∥ 2 = ∥ ( I − φ t R ) Δ t − φ t ( Q t ∇ f t − R Δ t ) − φ t Q t e t ∥ 2 ≤ ∥ ( I − φ t R ) Δ t ∥ 2 + 2 φ t ∥ Δ t ∥∥ Q t ∇ f t − R Δ t ∥ + γ H 5 φ t ∥ Δ t ∥∥ e t ∥ + 2 φ t 2 ∥ Q t ∇ f t − R Δ t ∥ 2 + 2 γ H 2 25 φ t 2 ∥ e t ∥ 2 ≤ ( 1 − 2 λ m φ t + ∥ R ∥ 2 φ t 2 ) ∥ Δ t ∥ 2 + 2 φ t ∥ Δ t ∥∥ Q t ∇ f t − R Δ t ∥ + γ H 5 φ t ∥ Δ t ∥∥ e t ∥ + 2 φ t 2 ∥ Q t ∇ f t − R Δ t ∥ 2 + 2 γ H 2 25 φ t 2 ∥ e t ∥ 2 , (77)
where the second inequality uses the Cauchy–Schwarz inequality, ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 \|a+b\|^2\leq2\|a\|^2+2\|b\|^2 ∥ a + b ∥ 2 ≤ 2∥ a ∥ 2 + 2∥ b ∥ 2 , and the local bound ∥ Q t ∥ ≤ 5 / ( 2 γ H ) \|Q_t\|\leq5/(2\gamma_H) ∥ Q t ∥ ≤ 5/ ( 2 γ H ) from (D.21), and the last inequality holds due to the fact that
∥ ( I − φ t R ) a ∥ 2 ≤ ( 1 − 2 λ m φ t + ∥ R ∥ 2 φ t 2 ) ∥ a ∥ 2 . \|(I-\varphi_t\mathcal{R})a\|^2\leq(1-2\lambda_m\varphi_t+\|\mathcal{R}\|^2\varphi_t^2)\|a\|^2. ∥ ( I − φ t R ) a ∥ 2 ≤ ( 1 − 2 λ m φ t + ∥ R ∥ 2 φ t 2 ) ∥ a ∥ 2 .
On { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , by the definition of τ k , r \tau_{k,r} τ k , r and (D.24), we have
∥ Q t ∇ f t − R Δ t ∥ ≤ { 5 Υ L 4 γ H + 5 2 γ H + L K r } ∥ Δ t ∥ . \|Q_t\nabla f_t-\mathcal{R}\Delta_t\|\leq\left\{\frac{5\Upsilon_L}{4\gamma_H}+\frac{5}{2\gamma_H}+L_Kr\right\}\|\Delta_t\|. ∥ Q t ∇ f t − R Δ t ∥ ≤ { 4 γ H 5 Υ L + 2 γ H 5 + L K r } ∥ Δ t ∥.
Moreover, on { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , Young’s inequality directly implies
5 φ t γ H ∥ Δ t ∥ ∥ e t ∥ ≤ ε 0 λ m φ t ∥ Δ t ∥ 2 + 25 φ t 4 ε 0 λ m γ H 2 ∥ e t ∥ 2 . \frac{5\varphi_t}{\gamma_H}\|\Delta_t\|\|e_t\|\leq\varepsilon_0\lambda_m\varphi_t\|\Delta_t\|^2+\frac{25\varphi_t}{4\varepsilon_0\lambda_m\gamma_H^2}\|e_t\|^2. γ H 5 φ t ∥ Δ t ∥∥ e t ∥ ≤ ε 0 λ m φ t ∥ Δ t ∥ 2 + 4 ε 0 λ m γ H 2 25 φ t ∥ e t ∥ 2 .
On { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , substituting these two bounds into (D.30) yields
∥ Δ t − φ t Q t ( ∇ f t + e t ) ∥ 2 ≤ [ 1 − ( 2 − ε 0 ) λ m φ t + 2 { 5 Υ L 4 γ H + 5 2 γ H + L K } r φ t + ( ∥ R ∥ 2 + 2 { 5 Υ L 4 γ H + 5 2 γ H + L K } 2 r 2 ) φ t 2 ] ∥ Δ t ∥ 2 + 25 4 γ H 2 ( φ t ε 0 λ m + 2 φ t 2 ) ∥ e t ∥ 2 . (78) \begin{aligned}
\|\Delta_t-\varphi_tQ_t(\nabla f_t+e_t)\|^2
&\leq\left[1-(2-\varepsilon_0)\lambda_m\varphi_t+2\left\{\frac{5\Upsilon_L}{4\gamma_H}+\frac{5}{2\gamma_H}+L_K\right\}r\varphi_t\right.\\
&\qquad\left.+\left(\|\mathcal{R}\|^2+2\left\{\frac{5\Upsilon_L}{4\gamma_H}+\frac{5}{2\gamma_H}+L_K\right\}^2r^2\right)\varphi_t^2\right]\|\Delta_t\|^2\\
&\quad+\frac{25}{4\gamma_H^2}\left(\frac{\varphi_t}{\varepsilon_0\lambda_m}+2\varphi_t^2\right)\|e_t\|^2. \tag*{(78)}
\end{aligned} ∥ Δ t − φ t Q t ( ∇ f t + e t ) ∥ 2 ≤ [ 1 − ( 2 − ε 0 ) λ m φ t + 2 { 4 γ H 5 Υ L + 2 γ H 5 + L K } r φ t + ( ∥ R ∥ 2 + 2 { 4 γ H 5 Υ L + 2 γ H 5 + L K } 2 r 2 ) φ t 2 ] ∥ Δ t ∥ 2 + 4 γ H 2 25 ( ε 0 λ m φ t + 2 φ t 2 ) ∥ e t ∥ 2 . (78)
We next control the second term in (D.29). On the event { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , since ( ∇ f t , e t ) (\nabla f_t,e_t) ( ∇ f t , e t ) is F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 -measurable, (D.21) gives
E [ ∥ ( Q ~ t − Q t ) ( ∇ f t + e t ) ∥ 2 ∣ F t − 0.5 ] ≤ E [ ∥ Q ~ t − Q t ∥ 2 ∣ F t − 0.5 ] ∥ ∇ f t + e t ∥ 2 ≤ 2 { 2 ( 1 + C K ) γ H + 5 2 γ H } 2 ( ∥ ∇ f t ∥ 2 + ∥ e t ∥ 2 ) ≤ 2 { 2 ( 1 + C K ) γ H + 5 2 γ H } 2 ( Γ H 2 ∥ Δ t ∥ 2 + ∥ e t ∥ 2 ) . (79) \begin{aligned}
\mathbb{E}[\|(\widetilde{Q}_t-Q_t)(\nabla f_t+e_t)\|^2\mid\mathcal{F}_{t-0.5}]
&\leq\mathbb{E}[\|\widetilde{Q}_t-Q_t\|^2\mid\mathcal{F}_{t-0.5}]\|\nabla f_t+e_t\|^2\\
&\leq2\left\{\frac{2(1+C_K)}{\gamma_H}+\frac{5}{2\gamma_H}\right\}^2(\|\nabla f_t\|^2+\|e_t\|^2)\\
&\leq2\left\{\frac{2(1+C_K)}{\gamma_H}+\frac{5}{2\gamma_H}\right\}^2(\Gamma_H^2\|\Delta_t\|^2+\|e_t\|^2). \tag*{(79)}
\end{aligned} E [ ∥ ( Q t − Q t ) ( ∇ f t + e t ) ∥ 2 ∣ F t − 0.5 ] ≤ E [ ∥ Q t − Q t ∥ 2 ∣ F t − 0.5 ] ∥∇ f t + e t ∥ 2 ≤ 2 { γ H 2 ( 1 + C K ) + 2 γ H 5 } 2 ( ∥∇ f t ∥ 2 + ∥ e t ∥ 2 ) ≤ 2 { γ H 2 ( 1 + C K ) + 2 γ H 5 } 2 ( Γ H 2 ∥ Δ t ∥ 2 + ∥ e t ∥ 2 ) . (79)
Combining (D.29), (D.31) and (D.32), we obtain
E [ ∥ Δ t + 1 ∥ 2 ∣ F t − 0.5 ] 1 { τ k , r > t } ≤ [ 1 − ( 2 − ε 0 ) λ m φ t + 2 { 5 Υ L 4 γ H + 5 2 γ H + L K } r φ t + ( ∥ R ∥ 2 + 2 { 5 Υ L 4 γ H + 5 2 γ H + L K } 2 r 2 + 2 γ H 2 { 9 + 4 C K 2 γ H } 2 ) φ t 2 ] ∥ Δ t ∥ 2 1 { τ k , r > t } + [ 25 4 ε 0 λ m γ H 2 φ t + 2 { 25 4 γ H 2 + { 9 + 4 C K 2 γ H } 2 } φ t 2 ] ∥ e t ∥ 2 1 { τ k , r > t } . (80) \begin{aligned}
\mathbb{E}[\|\Delta_{t+1}\|^2 \mid\mathcal{F}_{t-0.5}]\mathbf{1}_{\{\tau_{k,r}>t\}}
\le{}& \left[1-(2-\varepsilon_0)\lambda_m\varphi_t+2\left\{\frac{5\Upsilon_L}{4\gamma_H}+\frac{5}{2\gamma_H}+L_K\right\}r\varphi_t \right.\\
&\left. +\left(\|\mathcal{R}\|^2+2\left\{\frac{5\Upsilon_L}{4\gamma_H}+\frac{5}{2\gamma_H}+L_K\right\}^2r^2+2\gamma_H^2\left\{\frac{9+4C_K}{2\gamma_H}\right\}^2\right)\varphi_t^2\right]\|\Delta_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\\
&+\left[\frac{25}{4\varepsilon_0\lambda_m\gamma_H^2}\varphi_t+2\left\{\frac{25}{4\gamma_H^2}+\left\{\frac{9+4C_K}{2\gamma_H}\right\}^2\right\}\varphi_t^2\right]\|e_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}. \tag*{(80)}
\end{aligned} E [ ∥ Δ t + 1 ∥ 2 ∣ F t − 0.5 ] 1 { τ k , r > t } ≤ [ 1 − ( 2 − ε 0 ) λ m φ t + 2 { 4 γ H 5 Υ L + 2 γ H 5 + L K } r φ t + ( ∥ R ∥ 2 + 2 { 4 γ H 5 Υ L + 2 γ H 5 + L K } 2 r 2 + 2 γ H 2 { 2 γ H 9 + 4 C K } 2 ) φ t 2 ] ∥ Δ t ∥ 2 1 { τ k , r > t } + [ 4 ε 0 λ m γ H 2 25 φ t + 2 { 4 γ H 2 25 + { 2 γ H 9 + 4 C K } 2 } φ t 2 ] ∥ e t ∥ 2 1 { τ k , r > t } . (80)
For notational conciseness, define the following deterministic constants:
L Δ : = 5 Υ L 4 γ H + 5 2 γ H + L K , L S : = 2 ( 1 + C K ) γ H + 5 2 γ H , L_\Delta:=\frac{5\Upsilon_L}{4\gamma_H}+\frac{5}{2\gamma_H}+L_K,\qquad
L_S:=\frac{2(1+C_K)}{\gamma_H}+\frac{5}{2\gamma_H}, L Δ := 4 γ H 5 Υ L + 2 γ H 5 + L K , L S := γ H 2 ( 1 + C K ) + 2 γ H 5 ,
D Δ ( r ) : = ∥ R ∥ 2 + 2 L Δ 2 r 2 + 2 γ H 2 L S 2 , D e : = 25 4 ε 0 λ m γ H 2 + 2 { 25 4 γ H 2 + L S 2 } . D_\Delta(r):=\|\mathcal{R}\|^2+2L_\Delta^2r^2+2\gamma_H^2L_S^2,\qquad
D_e:=\frac{25}{4\varepsilon_0\lambda_m\gamma_H^2}+2\left\{\frac{25}{4\gamma_H^2}+L_S^2\right\}. D Δ ( r ) := ∥ R ∥ 2 + 2 L Δ 2 r 2 + 2 γ H 2 L S 2 , D e := 4 ε 0 λ m γ H 2 25 + 2 { 4 γ H 2 25 + L S 2 } .
Therefore, for all sufficiently large t t t such that φ t ≤ 1 \varphi_t\leq1 φ t ≤ 1 , (D.33) implies
E [ ∥ Δ t + 1 ∥ 2 ∣ F t − 0.5 ] 1 { τ k , r > t } ≤ { [ 1 − ( 2 − ε 0 ) λ m φ t + 2 L Δ r φ t + D Δ ( r ) φ t 2 ] ∥ Δ t ∥ 2 + D e φ t ∥ e t ∥ 2 } 1 { τ k , r > t } . (81) \mathbb{E}[\|\Delta_{t+1}\|^2\mid\mathcal{F}_{t-0.5}]\mathbf{1}_{\{\tau_{k,r}>t\}}
\leq\left\{\left[1-(2-\varepsilon_0)\lambda_m\varphi_t+2L_\Delta r\varphi_t+D_\Delta(r)\varphi_t^2\right]\|\Delta_t\|^2+D_e\varphi_t\|e_t\|^2\right\}\mathbf{1}_{\{\tau_{k,r}>t\}}. \tag*{(81)} E [ ∥ Δ t + 1 ∥ 2 ∣ F t − 0.5 ] 1 { τ k , r > t } ≤ { [ 1 − ( 2 − ε 0 ) λ m φ t + 2 L Δ r φ t + D Δ ( r ) φ t 2 ] ∥ Δ t ∥ 2 + D e φ t ∥ e t ∥ 2 } 1 { τ k , r > t } . (81)
We now choose r ∈ ( 0 , min { 1 , γ H / 2 } ) r\in(0,\min\{1,\gamma_H/2\}) r ∈ ( 0 , min { 1 , γ H /2 }) sufficiently small such that
2 L Δ r ≤ ε 0 λ m 2 . (82) 2L_\Delta r\leq\frac{\varepsilon_0\lambda_m}{2}. \tag*{(82)} 2 L Δ r ≤ 2 ε 0 λ m . (82)
For this fixed r r r and any given ϵ > 0 \epsilon>0 ϵ > 0 , we first choose k = k ( ϵ , r ) k=k(\epsilon,r) k = k ( ϵ , r ) sufficiently large so that P ( τ k , r < ∞ ) < ϵ / 2 \mathbb{P}(\tau_{k,r}<\infty)<\epsilon/2 P ( τ k , r < ∞ ) < ϵ /2 , which guarantees (D.27) for every t ≥ k t\geq k t ≥ k . Additionally, for every t ≥ k t\geq k t ≥ k , we require
φ t ≤ 1 , φ t ∥ R ∥ ≤ 1 , D Δ ( r ) φ t ≤ ε 0 λ m 2 , 0 < 2 ( 1 − ε 0 ) λ m φ t < 1. (83) \varphi_t\leq1,\qquad\varphi_t\|\mathcal{R}\|\leq1,\qquad D_\Delta(r)\varphi_t\leq\frac{\varepsilon_0\lambda_m}{2},\qquad0<2(1-\varepsilon_0)\lambda_m\varphi_t<1. \tag*{(83)} φ t ≤ 1 , φ t ∥ R ∥ ≤ 1 , D Δ ( r ) φ t ≤ 2 ε 0 λ m , 0 < 2 ( 1 − ε 0 ) λ m φ t < 1. (83)
Specifically, the third condition allows the second-order term D Δ ( r ) φ t 2 D_\Delta(r)\varphi_t^2 D Δ ( r ) φ t 2 in (D.34) to be absorbed into the first-order contraction margin, thereby preserving the desired contraction coefficient.
For these fixed ( k , r ) (k,r) ( k , r ) , we define the following localized second moments
u t : = E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] , v t : = E [ ∥ e t ∥ 2 1 { τ k , r > t } ] , w t : = E [ ∥ B t − B ⋆ ∥ 2 1 { τ k , r > t } ] . (84) u_t:=\mathbb{E}[\|\Delta_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}],\qquad
v_t:=\mathbb{E}[\|e_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}],\qquad
w_t:=\mathbb{E}[\|B_t-B^\star\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}]. \tag*{(84)} u t := E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] , v t := E [ ∥ e t ∥ 2 1 { τ k , r > t } ] , w t := E [ ∥ B t − B ⋆ ∥ 2 1 { τ k , r > t } ] . (84)
P ( ∥ B t − B ⋆ ∥ > M η t ) = P ( ∥ B t − B ⋆ ∥ 2 > M 2 η t ) = P ( { ∥ B t − B ⋆ ∥ 2 > M 2 η t } ∩ { τ k , r > t } ) + P ( { ∥ B t − B ⋆ ∥ 2 > M 2 η t } ∩ { τ k , r ≤ t } ) ≤ P ( τ k , r ≤ t ) + P ( ∥ B t − B ⋆ ∥ 2 1 { τ k , r > t } > M 2 η t ) ≤ ϵ 2 + w t M 2 η t . (85) \begin{aligned}
\mathbb{P}(\|B_t-B^\star\|>M\sqrt{\eta_t})
&=\mathbb{P}(\|B_t-B^\star\|^2>M^2\eta_t)\\
&=\mathbb{P}(\{\|B_t-B^\star\|^2>M^2\eta_t\}\cap\{\tau_{k,r}>t\})\\
&\quad+\mathbb{P}(\{\|B_t-B^\star\|^2>M^2\eta_t\}\cap\{\tau_{k,r}\leq t\})\\
&\leq\mathbb{P}(\tau_{k,r}\leq t)+\mathbb{P}(\|B_t-B^\star\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}>M^2\eta_t)\\
&\leq\frac{\epsilon}{2}+\frac{w_t}{M^2\eta_t}. \tag*{(85)}
\end{aligned} P ( ∥ B t − B ⋆ ∥ > M η t ) = P ( ∥ B t − B ⋆ ∥ 2 > M 2 η t ) = P ({ ∥ B t − B ⋆ ∥ 2 > M 2 η t } ∩ { τ k , r > t }) + P ({ ∥ B t − B ⋆ ∥ 2 > M 2 η t } ∩ { τ k , r ≤ t }) ≤ P ( τ k , r ≤ t ) + P ( ∥ B t − B ⋆ ∥ 2 1 { τ k , r > t } > M 2 η t ) ≤ 2 ϵ + M 2 η t w t . (85)
Therefore, in order to obtain O p ( η t ) O_p(\sqrt{\eta_t}) O p ( η t ) , it suffices to establish w t = O ( η t ) w_t=O(\eta_t) w t = O ( η t ) . Recall that k = k ( ϵ , r ) k=k(\epsilon,r) k = k ( ϵ , r ) is fixed. On { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , we have ∥ B ~ k − B ⋆ ∥ F ≤ r \|\widetilde{B}_k-B^\star\|_F\leq r ∥ B k − B ⋆ ∥ F ≤ r . By the projection property of Clip and the decomposition of B ~ t \widetilde{B}_t B t at index k k k , we obtain
w t ≤ E [ ∥ B ~ t − B ⋆ ∥ F 2 1 { τ k , r > t } ] = E [ ∥ k t ( B ~ k − B ⋆ ) + 1 t ∑ i = k t − 1 ( H i − ∇ 2 f i ) + 1 t ∑ i = k t − 1 ( ∇ 2 f i − B ⋆ ) ∥ F 2 1 { τ k , r > t } ] ≲ k 2 r 2 t 2 + 1 t 2 E [ ∥ ∑ i = k t − 1 ( H i − ∇ 2 f i ) 1 { τ k , r > i } ∥ F 2 ] + 1 t 2 E [ ( ∑ i = k t − 1 ∥ Δ i ∥ 1 { τ k , r > i } ) 2 ] ≲ k 2 r 2 t 2 + 1 t 2 ∑ i = k t − 1 E [ ∥ H i − ∇ 2 f i ∥ F 2 1 { τ k , r > i } ] + 1 t 2 ( ∑ i = k t − 1 u i ) 2 ≲ k 2 r 2 t 2 + 1 t + 1 t 2 ( ∑ i = k t − 1 u i ) 2 . (86) \begin{aligned}
w_t &\leq\mathbb{E}\left[\left\|\widetilde{B}_t-B^\star\right\|_{\mathrm{F}}^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right] \\
&= \mathbb{E}\left[\left\|\frac{k}{t}(\widetilde{B}_k-B^\star)+\frac{1}{t}\sum_{i=k}^{t-1}(H_i-\nabla^2 f_i)+\frac{1}{t}\sum_{i=k}^{t-1}(\nabla^2 f_i-B^\star)\right\|_{\mathrm{F}}^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right] \\
&\lesssim\frac{k^2r^2}{t^2}+\frac{1}{t^2}\mathbb{E}\left[\left\|\sum_{i=k}^{t-1}(H_i-\nabla^2 f_i)\mathbf{1}_{\{\tau_{k,r}>i\}}\right\|_{\mathrm{F}}^2\right]+\frac{1}{t^2}\mathbb{E}\left[\left(\sum_{i=k}^{t-1}\|\Delta_i\|\mathbf{1}_{\{\tau_{k,r}>i\}}\right)^2\right] \\
&\lesssim\frac{k^2r^2}{t^2}+\frac{1}{t^2}\sum_{i=k}^{t-1}\mathbb{E}\left[\|H_i-\nabla^2 f_i\|_{\mathrm{F}}^2\mathbf{1}_{\{\tau_{k,r}>i\}}\right]+\frac{1}{t^2}\left(\sum_{i=k}^{t-1}\sqrt{u_i}\right)^2 \\
&\lesssim\frac{k^2r^2}{t^2}+\frac{1}{t}+\frac{1}{t^2}\left(\sum_{i=k}^{t-1}\sqrt{u_i}\right)^2. \tag*{(86)}
\end{aligned} w t ≤ E [ B t − B ⋆ F 2 1 { τ k , r > t } ] = E t k ( B k − B ⋆ ) + t 1 i = k ∑ t − 1 ( H i − ∇ 2 f i ) + t 1 i = k ∑ t − 1 ( ∇ 2 f i − B ⋆ ) F 2 1 { τ k , r > t } ≲ t 2 k 2 r 2 + t 2 1 E i = k ∑ t − 1 ( H i − ∇ 2 f i ) 1 { τ k , r > i } F 2 + t 2 1 E ( i = k ∑ t − 1 ∥ Δ i ∥ 1 { τ k , r > i } ) 2 ≲ t 2 k 2 r 2 + t 2 1 i = k ∑ t − 1 E [ ∥ H i − ∇ 2 f i ∥ F 2 1 { τ k , r > i } ] + t 2 1 ( i = k ∑ t − 1 u i ) 2 ≲ t 2 k 2 r 2 + t 1 + t 2 1 ( i = k ∑ t − 1 u i ) 2 . (86)
Here, ( H i − ∇ 2 f i ) 1 { τ k , r > i } (H_i-\nabla^2 f_i)\mathbf{1}_{\{\tau_{k,r}>i\}} ( H i − ∇ 2 f i ) 1 { τ k , r > i } is a matrix-valued martingale difference sequence because 1 { τ k , r > i } \mathbf{1}_{\{\tau_{k,r}>i\}} 1 { τ k , r > i } is F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable. Therefore, the third inequality follows from the Υ L \Upsilon_L Υ L -Lipschitz continuity of ∇ 2 f \nabla^2 f ∇ 2 f , ∥ B ~ k − B ⋆ ∥ F ≤ r \|\widetilde{B}_k-B^\star\|_{\mathrm{F}}\leq r ∥ B k − B ⋆ ∥ F ≤ r on { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , and { τ k , r > t } ⊆ { τ k , r > i } \{\tau_{k,r}>t\}\subseteq\{\tau_{k,r}>i\} { τ k , r > t } ⊆ { τ k , r > i } for i ≤ t i\leq t i ≤ t ; the fourth inequality follows from martingale orthogonality under the Frobenius inner product and Minkowski’s inequality; and the last inequality follows from Assumption 3.3 and ∥ Δ i ∥ ≤ r \|\Delta_i\|\leq r ∥ Δ i ∥ ≤ r on { τ k , r > i } \{\tau_{k,r}>i\} { τ k , r > i } . Then, it remains to establish the rates of u t u_t u t and v t v_t v t .
To complete the proof, we present the following two auxiliary lemmas. And their proofs will be deferred to Appendices § 11 .§ 4 and § 11 .§ 5 , respectively.
Lemma D.3. For ( k , r ) (k,r) ( k , r ) chosen above and all sufficiently large t t t , we have
u t + 1 ≤ { 1 − 2 ( 1 − ε 0 ) λ m ϕ t } u t + D e ϕ t v t . (87) u_{t+1}\leq\{1-2(1-\varepsilon_0)\lambda_m\phi_t\}u_t+D_e\phi_tv_t. \tag*{(87)} u t + 1 ≤ { 1 − 2 ( 1 − ε 0 ) λ m ϕ t } u t + D e ϕ t v t . (87) and
v t ≲ { ∏ j = k + 1 t ( 1 − η j ) } 2 v k + ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) 2 } η i 2 + { ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } ϕ i − 1 ( u i − 1 + v i − 1 ) } 2 . (88) \begin{aligned}
v_t \lesssim{}& \left\{\prod_{j=k+1}^{t}(1-\eta_j)\right\}^2v_k+\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)^2\right\}\eta_i^2 \\
&+\left\{\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}\phi_{i-1}\left(\sqrt{u_{i-1}}+\sqrt{v_{i-1}}\right)\right\}^2. \tag*{(88)}
\end{aligned} v t ≲ ⎩ ⎨ ⎧ j = k + 1 ∏ t ( 1 − η j ) ⎭ ⎬ ⎫ 2 v k + i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) 2 } η i 2 + { i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } ϕ i − 1 ( u i − 1 + v i − 1 ) } 2 . (88) Note that ∏ j = m n ( 1 − η j ) : = 1 \prod_{j=m}^{n}(1-\eta_j):=1 ∏ j = m n ( 1 − η j ) := 1 when m > n m>n m > n .
Lemma D.4. Under the conditions of Lemma 4.2 , for every integer q ≥ 0 q\geq0 q ≥ 0 , there exists a deterministic constant C q ( k , r ) > 0 C_q(k,r)>0 C q ( k , r ) > 0 such that
max { u t , v t } ≤ C q ( k , r ) { η t + ( ϕ t η t ) 2 q } (89) \max\{u_t,v_t\}\leq C_q(k,r)\left\{\eta_t+\left(\frac{\phi_t}{\eta_t}\right)^{2q}\right\} \tag*{(89)} max { u t , v t } ≤ C q ( k , r ) { η t + ( η t ϕ t ) 2 q } (89) for all sufficiently large t t t .
Since ϕ > η \phi>\eta ϕ > η , we can choose the finite integer q ⋆ : = ⌊ η / [ 2 ( ϕ − η ) ] ⌋ + 1 q_\star:=\lfloor\eta/[2(\phi-\eta)]\rfloor+1 q ⋆ := ⌊ η / [ 2 ( ϕ − η )]⌋ + 1 , which satisfies 2 q ⋆ ( ϕ − η ) > η 2q_\star(\phi-\eta)>\eta 2 q ⋆ ( ϕ − η ) > η . Consequently, ( ϕ t / η t ) 2 q ⋆ = o ( η t ) (\phi_t/\eta_t)^{2q_\star}=o(\eta_t) ( ϕ t / η t ) 2 q ⋆ = o ( η t ) . Applying Lemma D.4 with q = q ⋆ q=q_\star q = q ⋆ gives
u t = O ( η t ) , v t = O ( η t ) . (90) u_t=O(\eta_t),\qquad v_t=O(\eta_t). \tag*{(90)} u t = O ( η t ) , v t = O ( η t ) . (90)
Returning to (D.39) and using (D.43), we obtain
w t ≲ k 2 r 2 t 2 + 1 t + 1 t 2 ( ∑ i = k t − 1 η i ) 2 ≲ 1 t + η t ≲ η t . (91) w_t\lesssim\frac{k^2r^2}{t^2}+\frac{1}{t}+\frac{1}{t^2}\left(\sum_{i=k}^{t-1}\sqrt{\eta_i}\right)^2\lesssim\frac{1}{t}+\eta_t\lesssim\eta_t. \tag*{(91)} w t ≲ t 2 k 2 r 2 + t 1 + t 2 1 ( i = k ∑ t − 1 η i ) 2 ≲ t 1 + η t ≲ η t . (91)
Therefore, for all sufficiently large t t t , there exists C B , k , r > 0 C_{B,k,r} > 0 C B , k , r > 0 such that
P ( ∥ B t − B ⋆ ∥ > M η t ) ≤ ϵ 2 + C B , k , r M 2 . \mathbb{P}(\lVert B_t - B^\star\rVert> M\sqrt{\eta_t}) \le\frac{\epsilon}{2} + \frac{C_{B,k,r}}{M^2}. P (∥ B t − B ⋆ ∥ > M η t ) ≤ 2 ϵ + M 2 C B , k , r .
Choosing M ≥ 2 C B , k , r / ϵ M \ge\sqrt{2C_{B,k,r}/\epsilon} M ≥ 2 C B , k , r / ϵ ensures P ( ∥ B t − B ⋆ ∥ > M η t ) < ϵ \mathbb{P}(\lVert B_t - B^\star\rVert> M\sqrt{\eta_t}) < \epsilon P (∥ B t − B ⋆ ∥ > M η t ) < ϵ for t t t large enough. Therefore,
∥ B t − B ⋆ ∥ = O p ( η t ) . (92) \lVert B_t - B^\star\rVert= O_p(\sqrt{\eta_t}). \tag*{(92)} ∥ B t − B ⋆ ∥ = O p ( η t ) . (92)
Convergence rates of m t m_t m t and Δ t \Delta_t Δ t . By (D.43) and the same stopping-time decomposition as above, for the fixed ( k , r ) (k,r) ( k , r ) and t t t large enough, there exist constants C Δ , k , r , C e , k , r > 0 C_{\Delta,k,r}, C_{e,k,r} > 0 C Δ , k , r , C e , k , r > 0 such that
P ( ∥ Δ t ∥ > M η t ) ≤ ϵ 2 + C Δ , k , r M 2 , P ( ∥ e t ∥ > M η t ) ≤ ϵ 2 + C e , k , r M 2 . \mathbb{P}(\lVert\Delta_t\rVert> M\sqrt{\eta_t}) \le\frac{\epsilon}{2} + \frac{C_{\Delta,k,r}}{M^2}, \qquad\mathbb{P}(\lVert e_t\rVert> M\sqrt{\eta_t}) \le\frac{\epsilon}{2} + \frac{C_{e,k,r}}{M^2}. P (∥ Δ t ∥ > M η t ) ≤ 2 ϵ + M 2 C Δ , k , r , P (∥ e t ∥ > M η t ) ≤ 2 ϵ + M 2 C e , k , r .
Again, choosing M ≥ 2 max { C Δ , k , r , C e , k , r } / ϵ M \ge\sqrt{2\max\{C_{\Delta,k,r},C_{e,k,r}\}/\epsilon} M ≥ 2 max { C Δ , k , r , C e , k , r } / ϵ ensures that
∥ Δ t ∥ = ∥ x t − x ⋆ ∥ = O p ( η t ) , ∥ e t ∥ = O p ( η t ) . (93) \lVert\Delta_t\rVert= \lVert x_t - x^\star\rVert= O_p(\sqrt{\eta_t}), \qquad\lVert e_t\rVert= O_p(\sqrt{\eta_t}). \tag*{(93)} ∥ Δ t ∥ = ∥ x t − x ⋆ ∥ = O p ( η t ) , ∥ e t ∥ = O p ( η t ) . (93)
Since m t = e t + ∇ f t m_t = e_t + \nabla f_t m t = e t + ∇ f t and ∥ ∇ f t ∥ ≤ Υ H ∥ Δ t ∥ \lVert\nabla f_t\rVert\le\Upsilon_H\lVert\Delta_t\rVert ∥ ∇ f t ∥ ≤ Υ H ∥ Δ t ∥ , (D.46) further yields
∥ m t ∥ ≤ ∥ e t ∥ + Υ H ∥ Δ t ∥ = O p ( η t ) . \lVert m_t\rVert\le\lVert e_t\rVert+ \Upsilon_H\lVert\Delta_t\rVert= O_p(\sqrt{\eta_t}). ∥ m t ∥ ≤ ∥ e t ∥ + Υ H ∥ Δ t ∥ = O p ( η t ) .
Convergence rates of ( α t , β t , γ t ) (\alpha_t,\beta_t,\gamma_t) ( α t , β t , γ t ) . Combining (D.13) and (D.45), we obtain
max { ∣ α t − α ⋆ ∣ , ∣ β t − β ⋆ ∣ , ∣ γ t − γ ⋆ ∣ } = O p ( η t ) . \max\{|\alpha_t-\alpha^\star|,|\beta_t-\beta^\star|,|\gamma_t-\gamma^\star|\} = O_p(\sqrt{\eta_t}). max { ∣ α t − α ⋆ ∣ , ∣ β t − β ⋆ ∣ , ∣ γ t − γ ⋆ ∣ } = O p ( η t ) .
∥ K t − K ⋆ ∥ ≤ L K ∥ B t − B ⋆ ∥ = O p ( η t ) . \lVert K_t-K^\star\rVert\le L_K\lVert B_t-B^\star\rVert= O_p(\sqrt{\eta_t}). ∥ K t − K ⋆ ∥ ≤ L K ∥ B t − B ⋆ ∥ = O p ( η t ) .
Together with (D.45) and (D.46), these bounds yield
max { ∥ m t − ∇ f t ∥ , ∥ m t ∥ , ∥ x t − x ⋆ ∥ , ∥ B t − B ⋆ ∥ , ∣ α t − α ⋆ ∣ , ∣ β t − β ⋆ ∣ , ∣ γ t − γ ⋆ ∣ , ∥ K t − K ⋆ ∥ } = O p ( η t ) , \max\{\lVert m_t-\nabla f_t\rVert,\lVert m_t\rVert,\lVert x_t-x^\star\rVert,\lVert B_t-B^\star\rVert,
|\alpha_t-\alpha^\star|,|\beta_t-\beta^\star|,|\gamma_t-\gamma^\star|,\lVert K_t-K^\star\rVert\}
= O_p(\sqrt{\eta_t}), max {∥ m t − ∇ f t ∥ , ∥ m t ∥ , ∥ x t − x ⋆ ∥ , ∥ B t − B ⋆ ∥ , ∣ α t − α ⋆ ∣ , ∣ β t − β ⋆ ∣ , ∣ γ t − γ ⋆ ∣ , ∥ K t − K ⋆ ∥} = O p ( η t ) ,
which completes the proof of Lemma 4.2 .
Proof of Lemma D.3
Since 1 { τ k , r > t + 1 } ≤ 1 { τ k , r > t } \mathbf{1}_{\{\tau_{k,r}>t+1\}} \le\mathbf{1}_{\{\tau_{k,r}>t\}} 1 { τ k , r > t + 1 } ≤ 1 { τ k , r > t } and 1 { τ k , r > t } \mathbf{1}_{\{\tau_{k,r}>t\}} 1 { τ k , r > t } is F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 -measurable, (D.34)–(D.36) yield
u t + 1 = E [ ∥ Δ t + 1 ∥ 2 1 { τ k , r > t + 1 } ] ≤ E [ E [ ∥ Δ t + 1 ∥ 2 ∣ F t − 0.5 ] 1 { τ k , r > t } ] ≤ { 1 − 2 ( 1 − ϵ 0 ) λ m φ t } u t + D e φ t v t , t ≥ k . (94) u_{t+1}=\mathbb{E}[\lVert\Delta_{t+1}\rVert^2\mathbf{1}_{\{\tau_{k,r}>t+1\}}]\le\mathbb{E}\left[\mathbb{E}[\lVert\Delta_{t+1}\rVert^2\mid\mathcal{F}_{t-0.5}]\mathbf{1}_{\{\tau_{k,r}>t\}}\right]
\le\{1-2(1-\epsilon_0)\lambda_m\varphi_t\}u_t+D_e\varphi_tv_t,\qquad t\ge k. \tag*{(94)} u t + 1 = E [∥ Δ t + 1 ∥ 2 1 { τ k , r > t + 1 } ] ≤ E [ E [∥ Δ t + 1 ∥ 2 ∣ F t − 0.5 ] 1 { τ k , r > t } ] ≤ { 1 − 2 ( 1 − ϵ 0 ) λ m φ t } u t + D e φ t v t , t ≥ k . (94)
This proves the first result. It remains to establish (D.41). By the moving-average update,
e t = ( 1 − η t ) e t − 1 + η t ( g t − ∇ f t ) + ( 1 − η t ) ( ∇ f t − 1 − ∇ f t ) . e_t=(1-\eta_t)e_{t-1}+\eta_t(g_t-\nabla f_t)+(1-\eta_t)(\nabla f_{t-1}-\nabla f_t). e t = ( 1 − η t ) e t − 1 + η t ( g t − ∇ f t ) + ( 1 − η t ) ( ∇ f t − 1 − ∇ f t ) .
On { τ k , r > k } \{\tau_{k,r}>k\} { τ k , r > k } , we have ∥ e k − 1 ∥ ≤ r \lVert e_{k-1}\rVert\le r ∥ e k − 1 ∥ ≤ r and ∥ Δ k − 1 ∥ , ∥ Δ k ∥ ≤ r \lVert\Delta_{k-1}\rVert,\lVert\Delta_k\rVert\le r ∥ Δ k − 1 ∥ , ∥ Δ k ∥ ≤ r . Thus, ∥ ∇ f k − 1 − ∇ f k ∥ ≤ 2 Υ H r \lVert\nabla f_{k-1}-\nabla f_k\rVert\le2\Upsilon_Hr ∥ ∇ f k − 1 − ∇ f k ∥ ≤ 2 Υ H r , and the preceding recursion and Assumption 3.2 give v k ≲ r 2 + η k 2 < ∞ v_k\lesssim r^2+\eta_k^2<\infty v k ≲ r 2 + η k 2 < ∞ . Iterating this identity from the fixed index k k k gives
e t = { ∏ j = k + 1 t ( 1 − η j ) } e k + ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } η i ( g i − ∇ f i ) + ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } ( 1 − η i ) ( ∇ f i − 1 − ∇ f i ) . (95) \begin{aligned}
e_t={}&\left\{\prod_{j=k+1}^{t}(1-\eta_j)\right\}e_k
+\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}\eta_i(g_i-\nabla f_i)\\
&+\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}(1-\eta_i)(\nabla f_{i-1}-\nabla f_i). \tag*{(95)}
\end{aligned} e t = ⎩ ⎨ ⎧ j = k + 1 ∏ t ( 1 − η j ) ⎭ ⎬ ⎫ e k + i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } η i ( g i − ∇ f i ) + i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } ( 1 − η i ) ( ∇ f i − 1 − ∇ f i ) . (95)
Since 1 { τ k , r > i } 1_{\{\tau_{k,r}>i\}} 1 { τ k , r > i } is F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable and E [ g i − ∇ f i ∣ F i − 1 ] = 0 \mathbb{E}[g_i-\nabla f_i\mid\mathcal{F}_{i-1}]=0 E [ g i − ∇ f i ∣ F i − 1 ] = 0 , { ( g i − ∇ f i ) 1 { τ k , r > i } } i ≥ k + 1 \{(g_i-\nabla f_i)1_{\{\tau_{k,r}>i\}}\}_{i\ge k+1} {( g i − ∇ f i ) 1 { τ k , r > i } } i ≥ k + 1 forms a martingale difference sequence. Moreover, E [ 1 { τ k , r > t } ∣ F i − 1 ] ≤ 1 { τ k , r > i } \mathbb{E}[1_{\{\tau_{k,r}>t\}}\mid\mathcal{F}_{i-1}]\le1_{\{\tau_{k,r}>i\}} E [ 1 { τ k , r > t } ∣ F i − 1 ] ≤ 1 { τ k , r > i } for every i ≤ t i\le t i ≤ t . As a result, by Jensen’s inequality and Assumption 3.2 , we obtain
E [ ∥ ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } η i ( g i − ∇ f i ) ∥ 2 1 { τ k , r > t } ] ≤ E [ ∥ ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } η i ( g i − ∇ f i ) 1 { τ k , r > i } ∥ 2 ] = ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) 2 } η i 2 E [ ∥ g i − ∇ f i ∥ 2 1 { τ k , r > i } ] ≲ ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) 2 } η i 2 . (96) \begin{aligned}
\mathbb{E}\left[\left\|\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}\eta_i(g_i-\nabla f_i)\right\|^2 1_{\{\tau_{k,r}>t\}}\right]
&\le\mathbb{E}\left[\left\|\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}\eta_i(g_i-\nabla f_i)1_{\{\tau_{k,r}>i\}}\right\|^2\right]\\
&=\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)^2\right\}\eta_i^2\mathbb{E}\left[\|g_i-\nabla f_i\|^2 1_{\{\tau_{k,r}>i\}}\right]\\
&\lesssim\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)^2\right\}\eta_i^2. \tag*{(96)}
\end{aligned} E i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } η i ( g i − ∇ f i ) 2 1 { τ k , r > t } ≤ E i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } η i ( g i − ∇ f i ) 1 { τ k , r > i } 2 = i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) 2 } η i 2 E [ ∥ g i − ∇ f i ∥ 2 1 { τ k , r > i } ] ≲ i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) 2 } η i 2 . (96)
For the last term in (D.48), by the T H \mathcal{T}_H T H -Lipschitz continuity of ∇ f \nabla f ∇ f and Lemma C.1 , we have
∥ ∇ f i − ∇ f i − 1 ∥ ≤ T H φ i − 1 ∥ z i − 1 , τ ∥ = T H φ i − 1 ∥ Q ~ i − 1 ( ∇ f i − 1 + e i − 1 ) ∥ . \|\nabla f_i-\nabla f_{i-1}\|\le\mathcal{T}_H\varphi_{i-1}\|z_{i-1,\tau}\|=\mathcal{T}_H\varphi_{i-1}\|\widetilde{Q}_{i-1}(\nabla f_{i-1}+e_{i-1})\|. ∥∇ f i − ∇ f i − 1 ∥ ≤ T H φ i − 1 ∥ z i − 1 , τ ∥ = T H φ i − 1 ∥ Q i − 1 ( ∇ f i − 1 + e i − 1 ) ∥.
Then, by (D.21) and 1 { τ k , r > i } ≤ 1 { τ k , r > i − 1 } 1_{\{\tau_{k,r}>i\}}\le1_{\{\tau_{k,r}>i-1\}} 1 { τ k , r > i } ≤ 1 { τ k , r > i − 1 } , we obtain
E [ ∥ ∇ f i − ∇ f i − 1 ∥ 2 1 { τ k , r > i } ] ≤ T H 2 φ i − 1 2 E [ ∥ z i − 1 , τ ∥ 2 1 { τ k , r > i − 1 } ] = T H 2 φ i − 1 2 E [ ∥ Q ~ i − 1 ( ∇ f i − 1 + e i − 1 ) ∥ 2 1 { τ k , r > i − 1 } ] ≤ 4 T H 2 ( 1 + C K ) 2 γ H 2 φ i − 1 2 E [ ∥ ∇ f i − 1 + e i − 1 ∥ 2 1 { τ k , r > i − 1 } ] ≤ 8 T H 2 ( 1 + C K ) 2 γ H 2 φ i − 1 2 { E [ ∥ ∇ f i − 1 ∥ 2 1 { τ k , r > i − 1 } ] + v i − 1 } ≤ 8 T H 2 ( 1 + C K ) 2 γ H 2 φ i − 1 2 ( γ H 2 u i − 1 + v i − 1 ) ≲ φ i − 1 2 ( u i − 1 + v i − 1 ) . (97) \begin{aligned}
\mathbb{E}[\|\nabla f_i-\nabla f_{i-1}\|^2 1_{\{\tau_{k,r}>i\}}]
&\le\mathcal{T}_H^2\varphi_{i-1}^2\mathbb{E}[\|z_{i-1,\tau}\|^2 1_{\{\tau_{k,r}>i-1\}}]\\
&=\mathcal{T}_H^2\varphi_{i-1}^2\mathbb{E}[\|\widetilde{Q}_{i-1}(\nabla f_{i-1}+e_{i-1})\|^2 1_{\{\tau_{k,r}>i-1\}}]\\
&\le\frac{4\mathcal{T}_H^2(1+C_K)^2}{\gamma_H^2}\varphi_{i-1}^2\mathbb{E}[\|\nabla f_{i-1}+e_{i-1}\|^2 1_{\{\tau_{k,r}>i-1\}}]\\
&\le\frac{8\mathcal{T}_H^2(1+C_K)^2}{\gamma_H^2}\varphi_{i-1}^2\left\{\mathbb{E}[\|\nabla f_{i-1}\|^2 1_{\{\tau_{k,r}>i-1\}}]+v_{i-1}\right\}\\
&\le\frac{8\mathcal{T}_H^2(1+C_K)^2}{\gamma_H^2}\varphi_{i-1}^2\left(\gamma_H^2u_{i-1}+v_{i-1}\right)\lesssim\varphi_{i-1}^2(u_{i-1}+v_{i-1}). \tag*{(97)}
\end{aligned} E [ ∥∇ f i − ∇ f i − 1 ∥ 2 1 { τ k , r > i } ] ≤ T H 2 φ i − 1 2 E [ ∥ z i − 1 , τ ∥ 2 1 { τ k , r > i − 1 } ] = T H 2 φ i − 1 2 E [ ∥ Q i − 1 ( ∇ f i − 1 + e i − 1 ) ∥ 2 1 { τ k , r > i − 1 } ] ≤ γ H 2 4 T H 2 ( 1 + C K ) 2 φ i − 1 2 E [ ∥∇ f i − 1 + e i − 1 ∥ 2 1 { τ k , r > i − 1 } ] ≤ γ H 2 8 T H 2 ( 1 + C K ) 2 φ i − 1 2 { E [ ∥∇ f i − 1 ∥ 2 1 { τ k , r > i − 1 } ] + v i − 1 } ≤ γ H 2 8 T H 2 ( 1 + C K ) 2 φ i − 1 2 ( γ H 2 u i − 1 + v i − 1 ) ≲ φ i − 1 2 ( u i − 1 + v i − 1 ) . (97)
where the third inequality follows from ∥ Q ~ i − 1 ∥ ≤ 2 ( 1 + C K ) / γ H \|\widetilde{Q}_{i-1}\|\le2(1+C_K)/\gamma_H ∥ Q i − 1 ∥ ≤ 2 ( 1 + C K ) / γ H on { τ k , r > i − 1 } \{\tau_{k,r}>i-1\} { τ k , r > i − 1 } in (D.21), and the last inequality uses ∥ ∇ f i − 1 ∥ ≤ T H ∥ Δ i − 1 ∥ \|\nabla f_{i-1}\|\le\mathcal{T}_H\|\Delta_{i-1}\| ∥∇ f i − 1 ∥ ≤ T H ∥ Δ i − 1 ∥ together with the definitions of u i − 1 u_{i-1} u i − 1 and v i − 1 v_{i-1} v i − 1 .
Applying Minkowski’s inequality to (D.48) and using (D.49) and (D.50), we obtain
v t ≲ { ∏ j = k + 1 t ( 1 − η j ) } v k + { ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) 2 } η i 2 } 1 / 2 + ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } φ i − 1 ( u i − 1 + v i − 1 ) . \sqrt{v_t}\lesssim\left\{\prod_{j=k+1}^{t}(1-\eta_j)\right\}\sqrt{v_k}+\left\{\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)^2\right\}\eta_i^2\right\}^{1/2}
+\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}\varphi_{i-1}\left(\sqrt{u_{i-1}}+\sqrt{v_{i-1}}\right). v t ≲ ⎩ ⎨ ⎧ j = k + 1 ∏ t ( 1 − η j ) ⎭ ⎬ ⎫ v k + { i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) 2 } η i 2 } 1/2 + i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } φ i − 1 ( u i − 1 + v i − 1 ) .
Squaring both sides and using ( a + b + c ) 2 ≤ 3 ( a 2 + b 2 + c 2 ) (a+b+c)^2\le3(a^2+b^2+c^2) ( a + b + c ) 2 ≤ 3 ( a 2 + b 2 + c 2 ) yields
v t ≲ { ∏ j = k + 1 t ( 1 − η j ) 2 } v k + ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) 2 } η i 2 + { ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } φ i − 1 ( u i − 1 + v i − 1 ) } 2 , \begin{aligned}
v_t\lesssim{}&\left\{\prod_{j=k+1}^{t}(1-\eta_j)^2\right\}v_k+\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)^2\right\}\eta_i^2\\
&+\left\{\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}\varphi_{i-1}\left(\sqrt{u_{i-1}}+\sqrt{v_{i-1}}\right)\right\}^2,
\end{aligned} v t ≲ ⎩ ⎨ ⎧ j = k + 1 ∏ t ( 1 − η j ) 2 ⎭ ⎬ ⎫ v k + i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) 2 } η i 2 + { i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } φ i − 1 ( u i − 1 + v i − 1 ) } 2 ,
which proves (D.41) and completes the proof.
Proof of Lemma D.4
We proceed by induction on q q q . For q = 0 q=0 q = 0 , the right-hand side of (D.42) is of constant order. By the definition of the stopping time, v t = E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] ≤ r 2 v_t=\mathbb{E}[\lVert\Delta_t\rVert^2\mathbf{1}_{\{\tau_{k,r}>t\}}]\le r^2 v t = E [∥ Δ t ∥ 2 1 { τ k , r > t } ] ≤ r 2 . Moreover, on { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } ,
∥ e t − 1 ∥ ≤ r , ∥ Δ t − 1 ∥ ≤ r , ∥ Δ t ∥ ≤ r . \lVert e_{t-1}\rVert\le r,\qquad\lVert\Delta_{t-1}\rVert\le r,\qquad\lVert\Delta_t\rVert\le r. ∥ e t − 1 ∥ ≤ r , ∥ Δ t − 1 ∥ ≤ r , ∥ Δ t ∥ ≤ r .
Using the recursion for e t e_t e t , Assumption 3.2 and the bound that ∥ ∇ f t − 1 − ∇ f t ∥ ≤ 2 H r \lVert\nabla f_{t-1}-\nabla f_t\rVert\le2H_r ∥ ∇ f t − 1 − ∇ f t ∥ ≤ 2 H r on the event { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , we obtain
v t = E [ ∥ e t ∥ 2 1 { τ k , r > t } ] ≲ E [ { ∥ e t − 1 ∥ 2 + η t 2 ∥ g t − ∇ f t ∥ 2 + ∥ ∇ f t − 1 − ∇ f t ∥ 2 } 1 { τ k , r > t } ] ≲ r 2 + η t 2 ≲ 1. \begin{aligned}
v_t&=\mathbb{E}[\lVert e_t\rVert^2\mathbf{1}_{\{\tau_{k,r}>t\}}]\\
&\lesssim\mathbb{E}\left[\left\{\lVert e_{t-1}\rVert^2+\eta_t^2\lVert g_t-\nabla f_t\rVert^2+\lVert\nabla f_{t-1}-\nabla f_t\rVert^2\right\}\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\\
&\lesssim r^2+\eta_t^2\lesssim1.
\end{aligned} v t = E [∥ e t ∥ 2 1 { τ k , r > t } ] ≲ E [ { ∥ e t − 1 ∥ 2 + η t 2 ∥ g t − ∇ f t ∥ 2 + ∥ ∇ f t − 1 − ∇ f t ∥ 2 } 1 { τ k , r > t } ] ≲ r 2 + η t 2 ≲ 1.
Therefore, (D.42) holds for q = 0 q=0 q = 0 .
Suppose that (D.42) holds for some integer q ≥ 0 q\ge0 q ≥ 0 . Then, we have
u i + v i ≲ η i + ( φ i η i ) q . (98) \sqrt{u_i}+\sqrt{v_i}\lesssim\sqrt{\eta_i}+\left(\frac{\varphi_i}{\eta_i}\right)^q. \tag*{(98)} u i + v i ≲ η i + ( η i φ i ) q . (98)
Since η < φ ≤ 1 \eta<\varphi\le1 η < φ ≤ 1 , by Lemma B.3, we obtain
{ ∏ j = k + 1 t ( 1 − η j ) } 2 = o ( η t ) , ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } φ i − 1 η i − 1 ≲ φ t η t η t , \left\{\prod_{j=k+1}^{t}(1-\eta_j)\right\}^2=o(\eta_t),\qquad
\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}\varphi_{i-1}\sqrt{\eta_{i-1}}\lesssim\frac{\varphi_t}{\eta_t}\sqrt{\eta_t}, ⎩ ⎨ ⎧ j = k + 1 ∏ t ( 1 − η j ) ⎭ ⎬ ⎫ 2 = o ( η t ) , i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } φ i − 1 η i − 1 ≲ η t φ t η t ,
∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } 2 η i 2 ≲ η t , ∑ i = k + 1 t { ∏ j = i + 1 t ( 1 − η j ) } φ i − 1 ( φ i − 1 η i − 1 ) q ≲ ( φ t η t ) q + 1 . (99) \sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}^2\eta_i^2\lesssim\eta_t,\qquad
\sum_{i=k+1}^{t}\left\{\prod_{j=i+1}^{t}(1-\eta_j)\right\}\varphi_{i-1}\left(\frac{\varphi_{i-1}}{\eta_{i-1}}\right)^q\lesssim\left(\frac{\varphi_t}{\eta_t}\right)^{q+1}. \tag*{(99)} i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } 2 η i 2 ≲ η t , i = k + 1 ∑ t { j = i + 1 ∏ t ( 1 − η j ) } φ i − 1 ( η i − 1 φ i − 1 ) q ≲ ( η t φ t ) q + 1 . (99)
Substituting (D.51) and (D.52) into (D.41), we obtain
v t ≲ η t + η t ( φ t η t ) 2 + ( φ t η t ) 2 ( q + 1 ) ≲ η t + ( φ t η t ) 2 ( q + 1 ) , (100) v_t\lesssim\eta_t+\eta_t\left(\frac{\varphi_t}{\eta_t}\right)^2+\left(\frac{\varphi_t}{\eta_t}\right)^{2(q+1)}\lesssim\eta_t+\left(\frac{\varphi_t}{\eta_t}\right)^{2(q+1)}, \tag*{(100)} v t ≲ η t + η t ( η t φ t ) 2 + ( η t φ t ) 2 ( q + 1 ) ≲ η t + ( η t φ t ) 2 ( q + 1 ) , (100)
where the last inequality uses the fact that φ t / η t → 0 \varphi_t/\eta_t\to0 φ t / η t → 0 . Consequently, there exists a deterministic constant C v , q + 1 ( k , r ) > 0 C_{v,q+1}(k,r)>0 C v , q + 1 ( k , r ) > 0 such that
v t ≤ C v , q + 1 ( k , r ) { η t + ( φ t η t ) 2 ( q + 1 ) } (101) v_t\le C_{v,q+1}(k,r)\left\{\eta_t+\left(\frac{\varphi_t}{\eta_t}\right)^{2(q+1)}\right\} \tag*{(101)} v t ≤ C v , q + 1 ( k , r ) { η t + ( η t φ t ) 2 ( q + 1 ) } (101)
for all sufficiently large t t t .
Combining (D.54) with (D.40), we obtain a deterministic constant C u , q + 1 ( k , r ) > 0 C_{u,q+1}(k,r)>0 C u , q + 1 ( k , r ) > 0 such that
u t + 1 ≤ { 1 − 2 C φ ( 1 − ϵ 0 ) λ m ( t + 1 ) φ } u t + C u , q + 1 ( k , r ) ( t + 1 ) φ + min { η , 2 ( q + 1 ) ( φ − η ) } . (102) u_{t+1}\le\left\{1-\frac{2C_\varphi(1-\epsilon_0)\lambda_m}{(t+1)^\varphi}\right\}u_t+\frac{C_{u,q+1}(k,r)}{(t+1)^{\varphi+\min\{\eta,2(q+1)(\varphi-\eta)\}}}. \tag*{(102)} u t + 1 ≤ { 1 − ( t + 1 ) φ 2 C φ ( 1 − ϵ 0 ) λ m } u t + ( t + 1 ) φ + m i n { η , 2 ( q + 1 ) ( φ − η )} C u , q + 1 ( k , r ) . (102)
By Lemma B.3, we obtain for φ ∈ ( 0.5 , 1 ] \varphi\in(0.5,1] φ ∈ ( 0.5 , 1 ]
lim sup t → ∞ u t η t + ( φ t / η t ) 2 ( q + 1 ) ≲ { C u , q + 1 ( k , r ) 2 C φ ( 1 − ϵ 0 ) λ m , φ ∈ ( 0.5 , 1 ) , C u , q + 1 ( k , r ) 2 C φ ( 1 − ϵ 0 ) λ m − min { η , 2 ( q + 1 ) ( φ − η ) } , φ = 1 , \limsup_{t\to\infty}\frac{u_t}{\eta_t+(\varphi_t/\eta_t)^{2(q+1)}}\lesssim
\begin{cases}
\frac{C_{u,q+1}(k,r)}{2C_\varphi(1-\epsilon_0)\lambda_m}, & \varphi\in(0.5,1),\\[1.2ex]
\frac{C_{u,q+1}(k,r)}{2C_\varphi(1-\epsilon_0)\lambda_m-\min\{\eta,2(q+1)(\varphi-\eta)\}}, & \varphi=1,
\end{cases} t → ∞ lim sup η t + ( φ t / η t ) 2 ( q + 1 ) u t ≲ ⎩ ⎨ ⎧ 2 C φ ( 1 − ϵ 0 ) λ m C u , q + 1 ( k , r ) , 2 C φ ( 1 − ϵ 0 ) λ m − m i n { η , 2 ( q + 1 ) ( φ − η )} C u , q + 1 ( k , r ) , φ ∈ ( 0.5 , 1 ) , φ = 1 ,
⟹ u t ≲ η t + ( φ t η t ) 2 ( q + 1 ) . (103) \Longrightarrow\qquad u_t\lesssim\eta_t+\left(\frac{\varphi_t}{\eta_t}\right)^{2(q+1)}. \tag*{(103)} ⟹ u t ≲ η t + ( η t φ t ) 2 ( q + 1 ) . (103)
When φ = 1 \varphi=1 φ = 1 , the denominator is positive since
min { η , 2 ( q + 1 ) ( φ − η ) } ≤ η < 2 C φ ( 1 − ϵ 0 ) λ m , \min\{\eta,2(q+1)(\varphi-\eta)\}\le\eta<2C_\varphi(1-\epsilon_0)\lambda_m, min { η , 2 ( q + 1 ) ( φ − η )} ≤ η < 2 C φ ( 1 − ϵ 0 ) λ m ,
which is ensured by the choice of ϵ 0 \epsilon_0 ϵ 0 in (D.28).
Combining the above bound with (D.53), we obtain (D.42) with q q q replaced by q + 1 q+1 q + 1 . This completes the induction and hence the proof.
Proof of Theorem 4.3
Recall that R : = I − K ⋆ \mathcal{R} := I-K^{\star} R := I − K ⋆ , which is symmetric positive definite. Fix r > 0 r>0 r > 0 as chosen in the proof of Lemma 4.2 and choose k = k ( ϵ , r ) k=k(\epsilon,r) k = k ( ϵ , r ) . The localized rates established in (D.43) and (D.44 give
max { E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] , E [ ∥ e t ∥ 2 1 { τ k , r > t } ] , E [ ∥ B t − B ⋆ ∥ 2 1 { τ k , r > t } ] } ≲ η t . (104) \max\left\{\mathbb{E}\left[\|\Delta_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right],\mathbb{E}\left[\|e_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right],\mathbb{E}\left[\|B_t-B^{\star}\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\right\}\lesssim\eta_t. \tag*{(104)} max { E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] , E [ ∥ e t ∥ 2 1 { τ k , r > t } ] , E [ ∥ B t − B ⋆ ∥ 2 1 { τ k , r > t } ] } ≲ η t . (104)
Since m t = e t + ∇ f t m_t=e_t+\nabla f_t m t = e t + ∇ f t and ∥ ∇ f t ∥ ≤ Υ H ∥ Δ t ∥ \|\nabla f_t\|\leq\Upsilon_H\|\Delta_t\| ∥∇ f t ∥ ≤ Υ H ∥ Δ t ∥ , it follows that
E [ ∥ m t ∥ 2 1 { τ k , r > t } ] ≤ 2 E [ ∥ e t ∥ 2 1 { τ k , r > t } ] + 2 Υ H 2 E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] ≲ η t . (105) \mathbb{E}\left[\|m_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\leq2\mathbb{E}\left[\|e_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]+2\Upsilon_H^2\mathbb{E}\left[\|\Delta_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\lesssim\eta_t. \tag*{(105)} E [ ∥ m t ∥ 2 1 { τ k , r > t } ] ≤ 2 E [ ∥ e t ∥ 2 1 { τ k , r > t } ] + 2 Υ H 2 E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] ≲ η t . (105)
Since x t → a . s . x ⋆ x_t\xrightarrow{\mathrm{a.s.}}x^{\star} x t a.s. x ⋆ and ξ t \xi_t ξ t is independent of F t − 1 \mathcal{F}_{t-1} F t − 1 , Assumptions 3.1 and 3.2 as well as Fatou’s lemma together imply that E [ ∥ ∇ F ( x ⋆ ; ξ ) ∥ q g ] ≤ C g , 2 \mathbb{E}[\|\nabla F(x^{\star};\xi)\|^{q_g}]\leq C_{g,2} E [ ∥∇ F ( x ⋆ ; ξ ) ∥ q g ] ≤ C g , 2 . The q g q_g q g -moment bound also gives uniform integrability, so the unbiasedness extends to x ⋆ x^{\star} x ⋆ . Using ∇ f ( x ⋆ ) = 0 \nabla f(x^{\star})=0 ∇ f ( x ⋆ ) = 0 , we obtain
E [ ∇ F ( x ⋆ ; ξ t ) ∣ F t − 1 ] = 0 , E [ ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∣ F t − 1 ] = 0. (106) \mathbb{E}[\nabla F(x^{\star};\xi_t)\mid\mathcal{F}_{t-1}]=0,\qquad\mathbb{E}[(g_t-\nabla f_t)-\nabla F(x^{\star};\xi_t)\mid\mathcal{F}_{t-1}]=0. \tag*{(106)} E [ ∇ F ( x ⋆ ; ξ t ) ∣ F t − 1 ] = 0 , E [( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∣ F t − 1 ] = 0. (106)
Moreover, since x t → a . s . x ⋆ x_t\xrightarrow{\mathrm{a.s.}}x^{\star} x t a.s. x ⋆ , the continuity of ∇ F ( ⋅ ; ξ ) \nabla F(\cdot;\xi) ∇ F ( ⋅ ; ξ ) and ∇ f \nabla f ∇ f gives pointwise convergence of the gradient difference to zero. The q g q_g q g -moment bounds above with q g > 2 q_g>2 q g > 2 imply uniform integrability. Since ξ t \xi_t ξ t is independent of F t − 1 \mathcal{F}_{t-1} F t − 1 , we obtain
E [ ∥ ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∥ 2 ∣ F t − 1 ] → a . s . 0. \mathbb{E}\left[\|(g_t-\nabla f_t)-\nabla F(x^{\star};\xi_t)\|^2\mid\mathcal{F}_{t-1}\right]\xrightarrow{\mathrm{a.s.}}0. E [ ∥ ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∥ 2 ∣ F t − 1 ] a.s. 0.
On { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , Assumption 3.2 and ∥ Δ t ∥ ≤ r \|\Delta_t\|\leq r ∥ Δ t ∥ ≤ r bound this conditional second moment by 2 ( C g , 1 r q g + C g , 2 ) 2 / q g + 2 C g , 2 2 / q g 2(C_{g,1}r^{q_g}+C_{g,2})^{2/q_g}+2C_{g,2}^{2/q_g} 2 ( C g , 1 r q g + C g , 2 ) 2/ q g + 2 C g , 2 2/ q g . Since { τ k , r > t } ∈ F t − 1 \{\tau_{k,r}>t\}\in\mathcal{F}_{t-1} { τ k , r > t } ∈ F t − 1 , dominated convergence yields
E [ ∥ ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∥ 2 1 { τ k , r > t } ] = o ( 1 ) . (107) \mathbb{E}\left[\|(g_t-\nabla f_t)-\nabla F(x^{\star};\xi_t)\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]=o(1). \tag*{(107)} E [ ∥ ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∥ 2 1 { τ k , r > t } ] = o ( 1 ) . (107)
By (D.20), B t − 1 m t B_t^{-1}m_t B t − 1 m t is F t − 0.5 \mathcal{F}_{t-0.5} F t − 0.5 -measurable, while 1 { τ k , r > t } \mathbf{1}_{\{\tau_{k,r}>t\}} 1 { τ k , r > t } is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, and hence
E [ ( K ~ t − K t ) B t − 1 m t 1 { τ k , r > t } ∣ F t − 0.5 ] = E [ K ~ t − K t ∣ F t − 0.5 ] B t − 1 m t 1 { τ k , r > t } = 0. \mathbb{E}\left[(\widetilde{K}_t-K_t)B_t^{-1}m_t\mathbf{1}_{\{\tau_{k,r}>t\}}\mid\mathcal{F}_{t-0.5}\right]=\mathbb{E}\left[\widetilde{K}_t-K_t\mid\mathcal{F}_{t-0.5}\right]B_t^{-1}m_t\mathbf{1}_{\{\tau_{k,r}>t\}}=0. E [ ( K t − K t ) B t − 1 m t 1 { τ k , r > t } ∣ F t − 0.5 ] = E [ K t − K t ∣ F t − 0.5 ] B t − 1 m t 1 { τ k , r > t } = 0.
Then, by the tower property and (D.59), we have
E [ ( K ~ t − K t ) B t − 1 m t 1 { τ k , r > t } − R ( B ⋆ ) − 1 { ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) } 1 { τ k , r > t } ∣ F t − 1 ] = E [ E [ ( K ~ t − K t ) B t − 1 m t 1 { τ k , r > t } ∣ F t − 0.5 ] ∣ F t − 1 ] − R ( B ⋆ ) − 1 1 { τ k , r > t } E [ ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∣ F t − 1 ] = 0. (108) \begin{aligned}
&\mathbb{E}\Big[(\widetilde{K}_t-K_t)B_t^{-1}m_t\mathbf{1}_{\{\tau_{k,r}>t\}}-\mathcal{R}(B^{\star})^{-1}\{(g_t-\nabla f_t)-\nabla F(x^{\star};\xi_t)\}\mathbf{1}_{\{\tau_{k,r}>t\}}\mid\mathcal{F}_{t-1}\Big]\\
&=\mathbb{E}\Big[\mathbb{E}\big[(\widetilde{K}_t-K_t)B_t^{-1}m_t\mathbf{1}_{\{\tau_{k,r}>t\}}\mid\mathcal{F}_{t-0.5}\big]\mid\mathcal{F}_{t-1}\Big]\\
&\quad-\mathcal{R}(B^{\star})^{-1}\mathbf{1}_{\{\tau_{k,r}>t\}}\mathbb{E}[(g_t-\nabla f_t)-\nabla F(x^{\star};\xi_t)\mid\mathcal{F}_{t-1}]=0. \tag*{(108)}
\end{aligned} E [ ( K t − K t ) B t − 1 m t 1 { τ k , r > t } − R ( B ⋆ ) − 1 {( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t )} 1 { τ k , r > t } ∣ F t − 1 ] = E [ E [ ( K t − K t ) B t − 1 m t 1 { τ k , r > t } ∣ F t − 0.5 ] ∣ F t − 1 ] − R ( B ⋆ ) − 1 1 { τ k , r > t } E [( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∣ F t − 1 ] = 0. (108)
As a result, by (D.61), we could conclude that the expression inside the conditional expectation forms a martingale difference sequence.
We next control its localized second moment. By (C.8), (D.60), the Cauchy–Schwarz inequality and the bounds ∥ K t ∥ ≤ 1 / 4 \|K_t\|\leq1/4 ∥ K t ∥ ≤ 1/4 , ∥ ( B ⋆ ) − 1 ∥ ≤ 1 / γ H \|(B^{\star})^{-1}\|\leq1/\gamma_H ∥ ( B ⋆ ) − 1 ∥ ≤ 1/ γ H , and ∥ B t − 1 ∥ ≤ 2 / γ H \|B_t^{-1}\|\leq2/\gamma_H ∥ B t − 1 ∥ ≤ 2/ γ H on { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , we obtain
E [ ∥ ( K ~ t − K t ) B t − 1 m t − R ( B ⋆ ) − 1 { ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) } ∥ 2 1 { τ k , r > t } ] ≤ 2 E [ ∥ ( K ~ t − K t ) B t − 1 m t ∥ 2 1 { τ k , r > t } ] + 2 E [ ∥ R ( B ⋆ ) − 1 { ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) } ∥ 2 1 { τ k , r > t } ] ≤ 8 γ H 2 ( C K + 1 4 ) 2 E [ ∥ m t ∥ 2 1 { τ k , r > t } ] + 2 ∥ R ∥ 2 γ H 2 E [ ∥ ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∥ 2 1 { τ k , r > t } ] = o ( 1 ) , (109) \begin{aligned}
&\mathbb{E}\Big[\big\|(\widetilde{K}_t-K_t)B_t^{-1}m_t-\mathcal{R}(B^{\star})^{-1}\{(g_t-\nabla f_t)-\nabla F(x^{\star};\xi_t)\}\big\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\Big]\\
&\leq2\mathbb{E}\left[\|(\widetilde{K}_t-K_t)B_t^{-1}m_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]+2\mathbb{E}\left[\|\mathcal{R}(B^{\star})^{-1}\{(g_t-\nabla f_t)-\nabla F(x^{\star};\xi_t)\}\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\\
&\leq\frac{8}{\gamma_H^2}\left(C_K+\frac14\right)^2\mathbb{E}\left[\|m_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]+\frac{2\|\mathcal{R}\|^2}{\gamma_H^2}\mathbb{E}\left[\|(g_t-\nabla f_t)-\nabla F(x^{\star};\xi_t)\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]=o(1), \tag*{(109)}
\end{aligned} E [ ( K t − K t ) B t − 1 m t − R ( B ⋆ ) − 1 {( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t )} 2 1 { τ k , r > t } ] ≤ 2 E [ ∥ ( K t − K t ) B t − 1 m t ∥ 2 1 { τ k , r > t } ] + 2 E [ ∥ R ( B ⋆ ) − 1 {( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t )} ∥ 2 1 { τ k , r > t } ] ≤ γ H 2 8 ( C K + 4 1 ) 2 E [ ∥ m t ∥ 2 1 { τ k , r > t } ] + γ H 2 2∥ R ∥ 2 E [ ∥ ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) ∥ 2 1 { τ k , r > t } ] = o ( 1 ) , (109)
where the last inequality follows from (D.58) and (D.60).
Moreover, on { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , by the definition of Q t Q_t Q t , (D.22), and ∥ ( B ⋆ ) − 1 ∥ ≤ 1 / γ H \|(B^{\star})^{-1}\|\leq1/\gamma_H ∥ ( B ⋆ ) − 1 ∥ ≤ 1/ γ H , we obtain
∥ ( I − K t ) B t − 1 − R ( B ⋆ ) − 1 ∥ = ∥ ( I − K t ) B t − 1 B ⋆ − R ∥ ( B ⋆ ) − 1 = ∥ ( Q t B ⋆ − R ) ( B ⋆ ) − 1 ∥ ≤ ∥ Q t B ⋆ − R ∥ ∥ ( B ⋆ ) − 1 ∥ ≤ 1 γ H { 5 2 γ H + L K } ∥ B t − B ⋆ ∥ ≲ ∥ B t − B ⋆ ∥ . (110) \begin{aligned}
\|(I-K_t)B_t^{-1}-\mathcal{R}(B^{\star})^{-1}\|
&=\|(I-K_t)B_t^{-1}B^{\star}-\mathcal{R}\|(B^{\star})^{-1}\\
&=\|(Q_tB^{\star}-\mathcal{R})(B^{\star})^{-1}\|\\
&\leq\|Q_tB^{\star}-\mathcal{R}\|\|(B^{\star})^{-1}\|\\
&\leq\frac{1}{\gamma_H}\left\{\frac{5}{2\gamma_H}+L_K\right\}\|B_t-B^{\star}\|\lesssim\|B_t-B^{\star}\|. \tag*{(110)}
\end{aligned} ∥ ( I − K t ) B t − 1 − R ( B ⋆ ) − 1 ∥ = ∥ ( I − K t ) B t − 1 B ⋆ − R ∥ ( B ⋆ ) − 1 = ∥ ( Q t B ⋆ − R ) ( B ⋆ ) − 1 ∥ ≤ ∥ Q t B ⋆ − R ∥∥ ( B ⋆ ) − 1 ∥ ≤ γ H 1 { 2 γ H 5 + L K } ∥ B t − B ⋆ ∥ ≲ ∥ B t − B ⋆ ∥. (110)
Additionally, by the triangle inequality, (D.23), and (D.63), we have
E [ ∥ R ( B ⋆ ) − 1 ( ∇ f t − B ⋆ Δ t ) + { ( I − K t ) B t − 1 − R ( B ⋆ ) − 1 } m t ∥ 1 { τ k , r > t } ] ≤ ∥ R ∥ ∥ ( B ⋆ ) − 1 ∥ E [ ∥ ∇ f t − B ⋆ Δ t ∥ 1 { τ k , r > t } ] + E [ ∥ ( I − K t ) B t − 1 − R ( B ⋆ ) − 1 ∥ ∥ m t ∥ 1 { τ k , r > t } ] ≲ E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] + E [ ∥ B t − B ⋆ ∥ ∥ m t ∥ 1 { τ k , r > t } ] ≤ E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] + { E [ ∥ B t − B ⋆ ∥ 2 1 { τ k , r > t } ] } 1 / 2 { E [ ∥ m t ∥ 2 1 { τ k , r > t } ] } 1 / 2 ≲ η t . (111) \begin{aligned}
\mathbb{E}\bigl[\|\mathcal{R}(B^\star)^{-1}(\nabla f_t-B^\star\Delta_t)+\{(I-K_t)B_t^{-1}-\mathcal{R}(B^\star)^{-1}\}m_t\|\mathbf{1}_{\{\tau_{k,r}>t\}}\bigr]
&\le\|\mathcal{R}\|\|(B^\star)^{-1}\|\mathbb{E}[\|\nabla f_t-B^\star\Delta_t\|\mathbf{1}_{\{\tau_{k,r}>t\}}] \\
&\quad+\mathbb{E}[\|(I-K_t)B_t^{-1}-\mathcal{R}(B^\star)^{-1}\|\|m_t\|\mathbf{1}_{\{\tau_{k,r}>t\}}] \\
&\lesssim\mathbb{E}[\|\Delta_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}]+\mathbb{E}[\|B_t-B^\star\|\|m_t\|\mathbf{1}_{\{\tau_{k,r}>t\}}] \\
&\le\mathbb{E}[\|\Delta_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}]+\{\mathbb{E}[\|B_t-B^\star\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}]\}^{1/2}\{\mathbb{E}[\|m_t\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}]\}^{1/2}\lesssim\eta_t . \tag*{(111)}
\end{aligned} E [ ∥ R ( B ⋆ ) − 1 ( ∇ f t − B ⋆ Δ t ) + {( I − K t ) B t − 1 − R ( B ⋆ ) − 1 } m t ∥ 1 { τ k , r > t } ] ≤ ∥ R ∥∥ ( B ⋆ ) − 1 ∥ E [ ∥∇ f t − B ⋆ Δ t ∥ 1 { τ k , r > t } ] + E [ ∥ ( I − K t ) B t − 1 − R ( B ⋆ ) − 1 ∥∥ m t ∥ 1 { τ k , r > t } ] ≲ E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] + E [ ∥ B t − B ⋆ ∥∥ m t ∥ 1 { τ k , r > t } ] ≤ E [ ∥ Δ t ∥ 2 1 { τ k , r > t } ] + { E [ ∥ B t − B ⋆ ∥ 2 1 { τ k , r > t } ] } 1/2 { E [ ∥ m t ∥ 2 1 { τ k , r > t } ] } 1/2 ≲ η t . (111)
where the second-to-last inequality is due to the Cauchy–Schwarz inequality, and the last inequality directly follows from (D.57) and (D.58).
m t = g t − 1 − η t η t ( m t − m t − 1 ) . m_t=g_t-\frac{1-\eta_t}{\eta_t}(m_t-m_{t-1}). m t = g t − η t 1 − η t ( m t − m t − 1 ) .
Moreover, we have
g t = B ⋆ Δ t + ∇ F ( x ⋆ ; ξ t ) + { ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) } + ( ∇ f t − B ⋆ Δ t ) . g_t=B^\star\Delta_t+\nabla F(x^\star;\xi_t)+\{(g_t-\nabla f_t)-\nabla F(x^\star;\xi_t)\}+(\nabla f_t-B^\star\Delta_t). g t = B ⋆ Δ t + ∇ F ( x ⋆ ; ξ t ) + {( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t )} + ( ∇ f t − B ⋆ Δ t ) .
Substituting the above two identities into the outer update and rearranging terms, we obtain
Δ t + 1 = Δ t − φ t ( I − K ~ t ) B t − 1 m t = Δ t − φ t R ( B ⋆ ) − 1 m t + φ t ( K ~ t − K t ) B t − 1 m t − φ t { ( I − K t ) B t − 1 − R ( B ⋆ ) − 1 } m t = Δ t − φ t R ( B ⋆ ) − 1 g t + φ t 1 − η t η t R ( B ⋆ ) − 1 ( m t − m t − 1 ) + φ t ( K ~ t − K t ) B t − 1 m t − φ t { ( I − K t ) B t − 1 − R ( B ⋆ ) − 1 } m t = ( I − φ t R ) Δ t − φ t R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ t ) + φ t 1 − η t η t R ( B ⋆ ) − 1 ( m t − m t − 1 ) + φ t [ ( K ~ t − K t ) B t − 1 m t − R ( B ⋆ ) − 1 { ( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t ) } ] − φ t [ R ( B ⋆ ) − 1 ( ∇ f t − B ⋆ Δ t ) + { ( I − K t ) B t − 1 − R ( B ⋆ ) − 1 } m t ] . (112) \begin{aligned}
\Delta_{t+1}
&=\Delta_t-\varphi_t(I-\widetilde K_t)B_t^{-1}m_t\\
&=\Delta_t-\varphi_t\mathcal{R}(B^\star)^{-1}m_t+\varphi_t(\widetilde K_t-K_t)B_t^{-1}m_t-\varphi_t\{(I-K_t)B_t^{-1}-\mathcal{R}(B^\star)^{-1}\}m_t\\
&=\Delta_t-\varphi_t\mathcal{R}(B^\star)^{-1}g_t+\varphi_t\frac{1-\eta_t}{\eta_t}\mathcal{R}(B^\star)^{-1}(m_t-m_{t-1})+\varphi_t(\widetilde K_t-K_t)B_t^{-1}m_t\\
&\quad-\varphi_t\{(I-K_t)B_t^{-1}-\mathcal{R}(B^\star)^{-1}\}m_t\\
&=(I-\varphi_t\mathcal{R})\Delta_t-\varphi_t\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_t)+\varphi_t\frac{1-\eta_t}{\eta_t}\mathcal{R}(B^\star)^{-1}(m_t-m_{t-1})\\
&\quad+\varphi_t\left[(\widetilde K_t-K_t)B_t^{-1}m_t-\mathcal{R}(B^\star)^{-1}\{(g_t-\nabla f_t)-\nabla F(x^\star;\xi_t)\}\right]\\
&\quad-\varphi_t\left[\mathcal{R}(B^\star)^{-1}(\nabla f_t-B^\star\Delta_t)+\{(I-K_t)B_t^{-1}-\mathcal{R}(B^\star)^{-1}\}m_t\right]. \tag*{(112)}
\end{aligned} Δ t + 1 = Δ t − φ t ( I − K t ) B t − 1 m t = Δ t − φ t R ( B ⋆ ) − 1 m t + φ t ( K t − K t ) B t − 1 m t − φ t {( I − K t ) B t − 1 − R ( B ⋆ ) − 1 } m t = Δ t − φ t R ( B ⋆ ) − 1 g t + φ t η t 1 − η t R ( B ⋆ ) − 1 ( m t − m t − 1 ) + φ t ( K t − K t ) B t − 1 m t − φ t {( I − K t ) B t − 1 − R ( B ⋆ ) − 1 } m t = ( I − φ t R ) Δ t − φ t R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ t ) + φ t η t 1 − η t R ( B ⋆ ) − 1 ( m t − m t − 1 ) + φ t [ ( K t − K t ) B t − 1 m t − R ( B ⋆ ) − 1 {( g t − ∇ f t ) − ∇ F ( x ⋆ ; ξ t )} ] − φ t [ R ( B ⋆ ) − 1 ( ∇ f t − B ⋆ Δ t ) + {( I − K t ) B t − 1 − R ( B ⋆ ) − 1 } m t ] . (112)
Iterating (D.65) from the fixed index k k k yields
Δ t = Π t , k Δ k − ∑ i = k t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + ∑ i = k t − 1 Π t , i + 1 φ i 1 − η i η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) + ∑ i = k t − 1 Π t , i + 1 φ i [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] − ∑ i = k t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] . (113) \begin{aligned}
\Delta_t={}&\Pi_{t,k}\Delta_k-\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\\
&+\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\frac{1-\eta_i}{\eta_i}\mathcal{R}(B^\star)^{-1}(m_i-m_{i-1})\\
&+\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\left[(\widetilde K_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\\
&-\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]. \tag*{(113)}
\end{aligned} Δ t = Π t , k Δ k − i = k ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + i = k ∑ t − 1 Π t , i + 1 φ i η i 1 − η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) + i = k ∑ t − 1 Π t , i + 1 φ i [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] − i = k ∑ t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] . (113)
We next derive bounds for Π t , i \Pi_{t,i} Π t , i and the weighted sums appearing in (D.66).
Without loss of generality, we may assume that φ j ∥ R ∥ ≤ 1 \varphi_j\|\mathcal{R}\|\le1 φ j ∥ R ∥ ≤ 1 for every j ≥ k j\ge k j ≥ k . Since R \mathcal{R} R is symmetric positive definite, for every i ≥ k i\ge k i ≥ k , we obtain
∥ I − φ j R ∥ = 1 − λ m φ j , ∥ Π n , i ∥ ≤ ∏ j = i n − 1 ( 1 − λ m φ j ) . (114) \|I-\varphi_j\mathcal{R}\|=1-\lambda_m\varphi_j,\qquad\|\Pi_{n,i}\|\le\prod_{j=i}^{n-1}(1-\lambda_m\varphi_j). \tag*{(114)} ∥ I − φ j R ∥ = 1 − λ m φ j , ∥ Π n , i ∥ ≤ j = i ∏ n − 1 ( 1 − λ m φ j ) . (114)
We first consider the case where φ < 1 \varphi<1 φ < 1 . For every fixed i 0 ≥ k i_0\ge k i 0 ≥ k , by (D.67), we obtain
∥ Π t , i 0 ∥ ≤ exp { − λ m ∑ j = i 0 t − 1 φ j } ≤ exp [ − C { ( t + 1 ) 1 − φ − ( i 0 + 1 ) 1 − φ } ] = o ( φ t ) , \|\Pi_{t,i_0}\|\le\exp\left\{-\lambda_m\sum_{j=i_0}^{t-1}\varphi_j\right\}\le\exp\left[-C\left\{(t+1)^{1-\varphi}-(i_0+1)^{1-\varphi}\right\}\right]=o(\sqrt{\varphi_t}), ∥ Π t , i 0 ∥ ≤ exp { − λ m j = i 0 ∑ t − 1 φ j } ≤ exp [ − C { ( t + 1 ) 1 − φ − ( i 0 + 1 ) 1 − φ } ] = o ( φ t ) ,
where C > 0 C > 0 C > 0 is some constant independent of t t t . Since η > φ / 2 \eta> \varphi/2 η > φ /2 , we have η t = o ( φ t ) \eta_t = o(\sqrt{\varphi_t}) η t = o ( φ t ) . In addition, φ t / η t → 0 \varphi_t/\eta_t \to0 φ t / η t → 0 implies that
φ t η t = φ t φ t η t = o ( φ t ) . \frac{\varphi_t}{\sqrt{\eta_t}} = \sqrt{\varphi_t}\sqrt{\frac{\varphi_t}{\eta_t}} = o(\sqrt{\varphi_t}). η t φ t = φ t η t φ t = o ( φ t ) .
Therefore, by Lemma B.3 , we have
∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 ≲ φ t , ∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 η i ≲ φ t η t = o ( φ t ) , ∑ i = k t − 1 ∥ Π t , i + 1 ∥ φ i η i ≲ η t = o ( φ t ) , ∑ i = k t − 2 ∥ Π t , i + 2 ∥ φ i 2 η i ≲ φ t η t = o ( φ t ) . (115) \begin{aligned}
\sum_{i=k}^{t-1} \lVert\Pi_{t,i+1}\rVert^2\varphi_i^2 &\lesssim\varphi_t, &
\sum_{i=k}^{t-1} \lVert\Pi_{t,i+1}\rVert^2\varphi_i^2\eta_i &\lesssim\varphi_t\eta_t=o(\varphi_t),\\
\sum_{i=k}^{t-1} \lVert\Pi_{t,i+1}\rVert\varphi_i\eta_i &\lesssim\eta_t=o(\sqrt{\varphi_t}), &
\sum_{i=k}^{t-2} \lVert\Pi_{t,i+2}\rVert\frac{\varphi_i^2}{\sqrt{\eta_i}} &\lesssim\frac{\varphi_t}{\sqrt{\eta_t}}=o(\sqrt{\varphi_t}). \tag*{(115)}
\end{aligned} i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 i = k ∑ t − 1 ∥ Π t , i + 1 ∥ φ i η i ≲ φ t , ≲ η t = o ( φ t ) , i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 η i i = k ∑ t − 2 ∥ Π t , i + 2 ∥ η i φ i 2 ≲ φ t η t = o ( φ t ) , ≲ η t φ t = o ( φ t ) . (115)
Then we consider the case where φ = 1 \varphi= 1 φ = 1 . For every fixed i 0 ≥ k i_0 \ge k i 0 ≥ k , by (D.67), we obtain
∥ Π t , i 0 ∥ ≤ exp { − λ m ∑ j = i 0 t − 1 φ j } ≲ ( i 0 + 1 t + 1 ) C φ λ m ≲ ( t + 1 ) − C φ λ m = o ( t − 1 / 2 ) = o ( φ t ) , \lVert\Pi_{t,i_0}\rVert\le\exp\left\{-\lambda_m\sum_{j=i_0}^{t-1}\varphi_j\right\}\lesssim\left(\frac{i_0+1}{t+1}\right)^{C_\varphi\lambda_m}\lesssim(t+1)^{-C_\varphi\lambda_m}=o(t^{-1/2})=o(\sqrt{\varphi_t}), ∥ Π t , i 0 ∥ ≤ exp { − λ m j = i 0 ∑ t − 1 φ j } ≲ ( t + 1 i 0 + 1 ) C φ λ m ≲ ( t + 1 ) − C φ λ m = o ( t − 1/2 ) = o ( φ t ) ,
where the last two relations follow from C φ λ m > 1 / 2 C_\varphi\lambda_m>1/2 C φ λ m > 1/2 . Moreover, since φ t ≍ t − 1 \varphi_t\asymp t^{-1} φ t ≍ t − 1 , we obtain
∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 ≲ ( t + 1 ) − 2 C φ λ m ∑ i = k t − 1 ( i + 1 ) 2 C φ λ m − 2 ≲ t − 1 ≲ φ t , ∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 η i ≲ ( t + 1 ) − 2 C φ λ m ∑ i = k t − 1 ( i + 1 ) 2 C φ λ m − 2 − η ≲ t − min { 2 C φ λ m , 1 + η } ( 1 + log t ) = o ( t − 1 ) = o ( φ t ) , ∑ i = k t − 1 ∥ Π t , i + 1 ∥ φ i η i ≲ ( t + 1 ) − C φ λ m ∑ i = k t − 1 ( i + 1 ) C φ λ m − 1 − η ≲ t − min { C φ λ m , η } ( 1 + log t ) = o ( t − 1 / 2 ) = o ( φ t ) , ∑ i = k t − 2 ∥ Π t , i + 2 ∥ φ i 2 η i ≲ ( t + 1 ) − C φ λ m ∑ i = k t − 2 ( i + 1 ) C φ λ m − 2 + η / 2 ≲ t − min { C φ λ m , 1 − η / 2 } ( 1 + log t ) = o ( t − 1 / 2 ) = o ( φ t ) . (116) \begin{aligned}
\sum_{i=k}^{t-1}\lVert\Pi_{t,i+1}\rVert^2\varphi_i^2
&\lesssim(t+1)^{-2C_\varphi\lambda_m}\sum_{i=k}^{t-1}(i+1)^{2C_\varphi\lambda_m-2}
\lesssim t^{-1}\lesssim\varphi_t,\\
\sum_{i=k}^{t-1}\lVert\Pi_{t,i+1}\rVert^2\varphi_i^2\eta_i
&\lesssim(t+1)^{-2C_\varphi\lambda_m}\sum_{i=k}^{t-1}(i+1)^{2C_\varphi\lambda_m-2-\eta}\\
&\lesssim t^{-\min\{2C_\varphi\lambda_m,1+\eta\}}(1+\log t)
=o(t^{-1})=o(\varphi_t),\\
\sum_{i=k}^{t-1}\lVert\Pi_{t,i+1}\rVert\varphi_i\eta_i
&\lesssim(t+1)^{-C_\varphi\lambda_m}\sum_{i=k}^{t-1}(i+1)^{C_\varphi\lambda_m-1-\eta}\\
&\lesssim t^{-\min\{C_\varphi\lambda_m,\eta\}}(1+\log t)
=o(t^{-1/2})=o(\sqrt{\varphi_t}),\\
\sum_{i=k}^{t-2}\lVert\Pi_{t,i+2}\rVert\frac{\varphi_i^2}{\sqrt{\eta_i}}
&\lesssim(t+1)^{-C_\varphi\lambda_m}\sum_{i=k}^{t-2}(i+1)^{C_\varphi\lambda_m-2+\eta/2}\\
&\lesssim t^{-\min\{C_\varphi\lambda_m,1-\eta/2\}}(1+\log t)
=o(t^{-1/2})=o(\sqrt{\varphi_t}). \tag*{(116)}
\end{aligned} i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 η i i = k ∑ t − 1 ∥ Π t , i + 1 ∥ φ i η i i = k ∑ t − 2 ∥ Π t , i + 2 ∥ η i φ i 2 ≲ ( t + 1 ) − 2 C φ λ m i = k ∑ t − 1 ( i + 1 ) 2 C φ λ m − 2 ≲ t − 1 ≲ φ t , ≲ ( t + 1 ) − 2 C φ λ m i = k ∑ t − 1 ( i + 1 ) 2 C φ λ m − 2 − η ≲ t − m i n { 2 C φ λ m , 1 + η } ( 1 + log t ) = o ( t − 1 ) = o ( φ t ) , ≲ ( t + 1 ) − C φ λ m i = k ∑ t − 1 ( i + 1 ) C φ λ m − 1 − η ≲ t − m i n { C φ λ m , η } ( 1 + log t ) = o ( t − 1/2 ) = o ( φ t ) , ≲ ( t + 1 ) − C φ λ m i = k ∑ t − 2 ( i + 1 ) C φ λ m − 2 + η /2 ≲ t − m i n { C φ λ m , 1 − η /2 } ( 1 + log t ) = o ( t − 1/2 ) = o ( φ t ) . (116)
Here, 1 + log t 1+\log t 1 + log t is included to cover the situation in which the exponent of a corresponding power sum equals − 1 -1 − 1 ; the little-o o o relations follow from C φ λ m > 1 / 2 C_\varphi\lambda_m>1/2 C φ λ m > 1/2 , η > 1 / 2 \eta>1/2 η > 1/2 and 1 − η / 2 > 1 / 2 1-\eta/2>1/2 1 − η /2 > 1/2 .
Combining the cases φ < 1 \varphi<1 φ < 1 and φ = 1 \varphi=1 φ = 1 , for every fixed i 0 ≥ k i_0\ge k i 0 ≥ k , we obtain ∥ Π n , i 0 ∥ = o ( φ n ) \lVert\Pi_{n,i_0}\rVert=o(\sqrt{\varphi_n}) ∥ Π n , i 0 ∥ = o ( φ n ) and
∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 ≲ φ t , ∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 η i = o ( φ t ) , ∑ i = k t − 1 ∥ Π t , i + 1 ∥ φ i η i = o ( φ t ) , ∑ i = k t − 2 ∥ Π t , i + 2 ∥ φ i 2 η i = o ( φ t ) . (117) \begin{aligned}
\sum_{i=k}^{t-1}\lVert\Pi_{t,i+1}\rVert^2\varphi_i^2 &\lesssim\varphi_t, &
\sum_{i=k}^{t-1}\lVert\Pi_{t,i+1}\rVert^2\varphi_i^2\eta_i &=o(\varphi_t),\\
\sum_{i=k}^{t-1}\lVert\Pi_{t,i+1}\rVert\varphi_i\eta_i &=o(\sqrt{\varphi_t}), &
\sum_{i=k}^{t-2}\lVert\Pi_{t,i+2}\rVert\frac{\varphi_i^2}{\sqrt{\eta_i}}&=o(\sqrt{\varphi_t}). \tag*{(117)}
\end{aligned} i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 i = k ∑ t − 1 ∥ Π t , i + 1 ∥ φ i η i ≲ φ t , = o ( φ t ) , i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 η i i = k ∑ t − 2 ∥ Π t , i + 2 ∥ η i φ i 2 = o ( φ t ) , = o ( φ t ) . (117)
In (D.66), the second term on the right-hand side is the leading stochastic term. It remains to show the other three sums are o p ( φ t ) o_p(\sqrt{\varphi_t}) o p ( φ t ) . We control these sums in the same order as they appear in (D.66).
We first consider the moving-average difference term. For the prescribed sequences, we have
φ t 1 − η t η t = C φ C η ( t + 1 ) − ( φ − η ) − C φ ( t + 1 ) − φ ≲ φ t η t . \varphi_t\frac{1-\eta_t}{\eta_t}
= \frac{C_\varphi}{C_\eta}(t+1)^{-(\varphi-\eta)}-C_\varphi(t+1)^{-\varphi}
\lesssim\frac{\varphi_t}{\eta_t}. φ t η t 1 − η t = C η C φ ( t + 1 ) − ( φ − η ) − C φ ( t + 1 ) − φ ≲ η t φ t .
Furthermore, by the mean-value theorem and φ ≤ 1 \varphi\le1 φ ≤ 1 , we have
∣ φ t + 1 1 − η t + 1 η t + 1 − φ t 1 − η t η t ∣ ≲ ( t + 1 ) − ( φ − η ) − 1 + ( t + 1 ) − φ − 1 ≲ φ t 2 η t . (118) \left|\varphi_{t+1}\frac{1-\eta_{t+1}}{\eta_{t+1}}-\varphi_t\frac{1-\eta_t}{\eta_t}\right|
\lesssim(t+1)^{-(\varphi-\eta)-1}+(t+1)^{-\varphi-1}
\lesssim\frac{\varphi_t^2}{\eta_t}. \tag*{(118)} φ t + 1 η t + 1 1 − η t + 1 − φ t η t 1 − η t ≲ ( t + 1 ) − ( φ − η ) − 1 + ( t + 1 ) − φ − 1 ≲ η t φ t 2 . (118)
Applying discrete summation by parts to the third term on the right-hand side of (D.66), we obtain
∑ i = k t − 1 Π t , i + 1 φ i 1 − η i η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) = φ t − 1 ( 1 − η t − 1 ) η t − 1 R ( B ⋆ ) − 1 m t − 1 − Π t , k + 1 φ k ( 1 − η k ) η k R ( B ⋆ ) − 1 m k − 1 + ∑ i = k t − 2 { Π t , i + 1 φ i ( 1 − η i ) η i − Π t , i + 2 φ i + 1 ( 1 − η i + 1 ) η i + 1 } R ( B ⋆ ) − 1 m i . (119) \begin{aligned}
\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\frac{1-\eta_i}{\eta_i}\mathcal{R}(B^\star)^{-1}(\mathbf{m}_i-\mathbf{m}_{i-1})
&=\frac{\varphi_{t-1}(1-\eta_{t-1})}{\eta_{t-1}}\mathcal{R}(B^\star)^{-1}\mathbf{m}_{t-1}-\Pi_{t,k+1}\frac{\varphi_k(1-\eta_k)}{\eta_k}\mathcal{R}(B^\star)^{-1}\mathbf{m}_{k-1}\\
&\quad+\sum_{i=k}^{t-2}\left\{\Pi_{t,i+1}\frac{\varphi_i(1-\eta_i)}{\eta_i}-\Pi_{t,i+2}\frac{\varphi_{i+1}(1-\eta_{i+1})}{\eta_{i+1}}\right\}\mathcal{R}(B^\star)^{-1}\mathbf{m}_i. \tag*{(119)}
\end{aligned} i = k ∑ t − 1 Π t , i + 1 φ i η i 1 − η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) = η t − 1 φ t − 1 ( 1 − η t − 1 ) R ( B ⋆ ) − 1 m t − 1 − Π t , k + 1 η k φ k ( 1 − η k ) R ( B ⋆ ) − 1 m k − 1 + i = k ∑ t − 2 { Π t , i + 1 η i φ i ( 1 − η i ) − Π t , i + 2 η i + 1 φ i + 1 ( 1 − η i + 1 ) } R ( B ⋆ ) − 1 m i . (119)
Since Π t , i + 1 = Π t , i + 2 ( I − φ i + 1 R ) \Pi_{t,i+1}=\Pi_{t,i+2}(I-\varphi_{i+1}\mathcal{R}) Π t , i + 1 = Π t , i + 2 ( I − φ i + 1 R ) , for the last line of (D.72), we have
∥ Π t , i + 1 φ i ( 1 − η i ) η i − Π t , i + 2 φ i + 1 ( 1 − η i + 1 ) η i + 1 ∥ ≤ ∥ Π t , i + 2 ∥ { ∥ φ i ( 1 − η i ) η i − φ i + 1 ( 1 − η i + 1 ) η i + 1 ∥ + φ i + 1 φ i ( 1 − η i ) η i ∥ R ∥ } ≲ ∥ Π t , i + 2 ∥ φ i 2 η i . (120) \left\|\Pi_{t,i+1}\frac{\varphi_i(1-\eta_i)}{\eta_i}-\Pi_{t,i+2}\frac{\varphi_{i+1}(1-\eta_{i+1})}{\eta_{i+1}}\right\|
\le\|\Pi_{t,i+2}\|\left\{\left\|\frac{\varphi_i(1-\eta_i)}{\eta_i}-\frac{\varphi_{i+1}(1-\eta_{i+1})}{\eta_{i+1}}\right\|+\varphi_{i+1}\frac{\varphi_i(1-\eta_i)}{\eta_i}\|\mathcal{R}\|\right\}\lesssim\|\Pi_{t,i+2}\|\frac{\varphi_i^2}{\eta_i}. \tag*{(120)} Π t , i + 1 η i φ i ( 1 − η i ) − Π t , i + 2 η i + 1 φ i + 1 ( 1 − η i + 1 ) ≤ ∥ Π t , i + 2 ∥ { η i φ i ( 1 − η i ) − η i + 1 φ i + 1 ( 1 − η i + 1 ) + φ i + 1 η i φ i ( 1 − η i ) ∥ R ∥ } ≲ ∥ Π t , i + 2 ∥ η i φ i 2 . (120)
where the last inequality follows from (D.71).
Since 1 { τ k , r > t } ≤ 1 { τ k , r > i } \mathbf{1}_{\{\tau_{k,r}>t\}}\le\mathbf{1}_{\{\tau_{k,r}>i\}} 1 { τ k , r > t } ≤ 1 { τ k , r > i } for every i ≤ t i\le t i ≤ t , Jensen’s inequality directly yields
E [ ∥ m i ∥ 1 { τ k , r > t } ] ≤ { E [ ∥ m i ∥ 2 1 { τ k , r > i } ] } 1 / 2 . \mathbb{E}[\|\mathbf{m}_i\|\mathbf{1}_{\{\tau_{k,r}>t\}}]\le\left\{\mathbb{E}[\|\mathbf{m}_i\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}]\right\}^{1/2}. E [ ∥ m i ∥ 1 { τ k , r > t } ] ≤ { E [ ∥ m i ∥ 2 1 { τ k , r > i } ] } 1/2 .
Therefore, applying the triangle inequality to (D.72) and then using (D.58) and (D.73), we obtain
E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i 1 − η i η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) ∥ 1 { τ k , r > t } ] ≤ φ t − 1 ( 1 − η t − 1 ) η t − 1 ∥ R ( B ⋆ ) − 1 ∥ E [ ∥ m t − 1 ∥ 1 { τ k , r > t } ] + ∥ Π t , k + 1 ∥ φ k ( 1 − η k ) η k ∥ R ( B ⋆ ) − 1 ∥ E [ ∥ m k − 1 ∥ 1 { τ k , r > t } ] + ∑ i = k t − 2 ∥ Π t , i + 1 φ i ( 1 − η i ) η i − Π t , i + 2 φ i + 1 ( 1 − η i + 1 ) η i + 1 ∥ ∥ R ( B ⋆ ) − 1 ∥ E [ ∥ m i ∥ 1 { τ k , r > t } ] ≲ φ t − 1 η t − 1 { E [ ∥ m t − 1 ∥ 2 1 { τ k , r > t − 1 } ] } 1 / 2 + ∥ Π t , k + 1 ∥ E [ ∥ m k − 1 ∥ 1 { τ k , r > k } ] + ∑ i = k t − 2 ∥ Π t , i + 2 ∥ φ i 2 η i { E [ ∥ m i ∥ 2 1 { τ k , r > i } ] } 1 / 2 ≲ φ t − 1 η t − 1 + ∥ Π t , k + 1 ∥ + ∑ i = k t − 2 ∥ Π t , i + 2 ∥ φ i 2 η i = o ( φ t ) . (121) \begin{aligned}
&\mathbb{E}\left[\left\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\frac{1-\eta_i}{\eta_i}\mathcal{R}(B^\star)^{-1}(\mathbf{m}_i-\mathbf{m}_{i-1})\right\|\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\\
&\le\frac{\varphi_{t-1}(1-\eta_{t-1})}{\eta_{t-1}}\|\mathcal{R}(B^\star)^{-1}\|\mathbb{E}[\|\mathbf{m}_{t-1}\|\mathbf{1}_{\{\tau_{k,r}>t\}}]\\
&\quad+\|\Pi_{t,k+1}\|\frac{\varphi_k(1-\eta_k)}{\eta_k}\|\mathcal{R}(B^\star)^{-1}\|\mathbb{E}[\|\mathbf{m}_{k-1}\|\mathbf{1}_{\{\tau_{k,r}>t\}}]\\
&\quad+\sum_{i=k}^{t-2}\left\|\Pi_{t,i+1}\frac{\varphi_i(1-\eta_i)}{\eta_i}-\Pi_{t,i+2}\frac{\varphi_{i+1}(1-\eta_{i+1})}{\eta_{i+1}}\right\|\|\mathcal{R}(B^\star)^{-1}\|\mathbb{E}[\|\mathbf{m}_i\|\mathbf{1}_{\{\tau_{k,r}>t\}}]\\
&\lesssim\frac{\varphi_{t-1}}{\eta_{t-1}}\left\{\mathbb{E}[\|\mathbf{m}_{t-1}\|^2\mathbf{1}_{\{\tau_{k,r}>t-1\}}]\right\}^{1/2}
+\|\Pi_{t,k+1}\|\mathbb{E}[\|\mathbf{m}_{k-1}\|\mathbf{1}_{\{\tau_{k,r}>k\}}]\\
&\quad+\sum_{i=k}^{t-2}\|\Pi_{t,i+2}\|\frac{\varphi_i^2}{\eta_i}\left\{\mathbb{E}[\|\mathbf{m}_i\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}]\right\}^{1/2}\\
&\lesssim\frac{\varphi_{t-1}}{\sqrt{\eta_{t-1}}}+\|\Pi_{t,k+1}\|+\sum_{i=k}^{t-2}\|\Pi_{t,i+2}\|\frac{\varphi_i^2}{\sqrt{\eta_i}}=o(\sqrt{\varphi_t}). \tag*{(121)}
\end{aligned} E [ i = k ∑ t − 1 Π t , i + 1 φ i η i 1 − η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) 1 { τ k , r > t } ] ≤ η t − 1 φ t − 1 ( 1 − η t − 1 ) ∥ R ( B ⋆ ) − 1 ∥ E [ ∥ m t − 1 ∥ 1 { τ k , r > t } ] + ∥ Π t , k + 1 ∥ η k φ k ( 1 − η k ) ∥ R ( B ⋆ ) − 1 ∥ E [ ∥ m k − 1 ∥ 1 { τ k , r > t } ] + i = k ∑ t − 2 Π t , i + 1 η i φ i ( 1 − η i ) − Π t , i + 2 η i + 1 φ i + 1 ( 1 − η i + 1 ) ∥ R ( B ⋆ ) − 1 ∥ E [ ∥ m i ∥ 1 { τ k , r > t } ] ≲ η t − 1 φ t − 1 { E [ ∥ m t − 1 ∥ 2 1 { τ k , r > t − 1 } ] } 1/2 + ∥ Π t , k + 1 ∥ E [ ∥ m k − 1 ∥ 1 { τ k , r > k } ] + i = k ∑ t − 2 ∥ Π t , i + 2 ∥ η i φ i 2 { E [ ∥ m i ∥ 2 1 { τ k , r > i } ] } 1/2 ≲ η t − 1 φ t − 1 + ∥ Π t , k + 1 ∥ + i = k ∑ t − 2 ∥ Π t , i + 2 ∥ η i φ i 2 = o ( φ t ) . (121)
where the last two steps follow from (D.58), (D.70) and φ t / η t → 0 \varphi_t/\eta_t\to0 φ t / η t → 0 . On the event { τ k , r > k } \{\tau_{k,r}>k\} { τ k , r > k } , ∥ m k − 1 ∥ ≤ ∥ e k − 1 ∥ + Υ H ∥ Δ k − 1 ∥ ≤ ( 1 + Υ H ) r \|\mathbf{m}_{k-1}\|\le\|\mathbf{e}_{k-1}\|+\Upsilon_H\|\boldsymbol{\Delta}_{k-1}\|\le(1+\Upsilon_H)r ∥ m k − 1 ∥ ≤ ∥ e k − 1 ∥ + Υ H ∥ Δ k − 1 ∥ ≤ ( 1 + Υ H ) r .
We next control the centered sum in (D.66). By (D.61), the bracketed expressions multiplied by 1 { τ k , r > i } \mathbf{1}_{\{\tau_{k,r}>i\}} 1 { τ k , r > i } form a martingale difference sequence. Therefore, by (D.62) and (D.70), we have
E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] 1 { τ k , r > t } ∥ 2 ] ≤ E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] 1 { τ k , r > i } ∥ 2 ] = ∑ i = k t − 1 E [ ∥ Π t , i + 1 φ i [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] ∥ 2 1 { τ k , r > i } ] ≤ ∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 E [ ∥ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ∥ 2 1 { τ k , r > i } ] ≲ ∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 η i + ∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 E [ ∥ ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) ∥ 2 1 { τ k , r > i } ] = o ( φ t ) . (122) \begin{aligned}
\mathbb{E}\Bigg[\Bigg\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\Big[(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\Big]\mathbf{1}_{\{\tau_{k,r}>t\}}\Bigg\|^2\Bigg]\\
\leq\mathbb{E}\Bigg[\Bigg\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\Big[(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\Big]\mathbf{1}_{\{\tau_{k,r}>i\}}\Bigg\|^2\Bigg]\\
= \sum_{i=k}^{t-1}\mathbb{E}\Bigg[\Bigg\|\Pi_{t,i+1}\varphi_i\Big[(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\Big]\Bigg\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}\Bigg]\\
\leq\sum_{i=k}^{t-1}\|\Pi_{t,i+1}\|^2\varphi_i^2\mathbb{E}\Bigg[\Big\|(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\Big\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}\Bigg]\\
\lesssim\sum_{i=k}^{t-1}\|\Pi_{t,i+1}\|^2\varphi_i^2\eta_i+\sum_{i=k}^{t-1}\|\Pi_{t,i+1}\|^2\varphi_i^2\mathbb{E}\big[\|(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}\big]=o(\varphi_t). \tag*{(122)}
\end{aligned} E [ i = k ∑ t − 1 Π t , i + 1 φ i [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 1 { τ k , r > t } 2 ] ≤ E [ i = k ∑ t − 1 Π t , i + 1 φ i [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 1 { τ k , r > i } 2 ] = i = k ∑ t − 1 E [ Π t , i + 1 φ i [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 2 1 { τ k , r > i } ] ≤ i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 E [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} 2 1 { τ k , r > i } ] ≲ i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 η i + i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 E [ ∥ ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) ∥ 2 1 { τ k , r > i } ] = o ( φ t ) . (122)
The first sum is o ( φ t ) o(\varphi_t) o ( φ t ) by (D.70). As for the second one, (D.60) implies that, for any ε > 0 \varepsilon>0 ε > 0 , there exists N > k N>k N > k such that E [ ∥ ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) ∥ 2 1 { τ k , r > i } ] ≤ ε \mathbb{E}[\|(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}]\leq\varepsilon E [ ∥ ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) ∥ 2 1 { τ k , r > i } ] ≤ ε , i ≥ N i\geq N i ≥ N . Therefore,
∑ i = N t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 E [ ∥ ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) ∥ 2 1 { τ k , r > i } ] ≤ ε ∑ i = N t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 ≲ ε φ t . \sum_{i=N}^{t-1}\|\Pi_{t,i+1}\|^2\varphi_i^2\mathbb{E}[\|(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}]\leq\varepsilon\sum_{i=N}^{t-1}\|\Pi_{t,i+1}\|^2\varphi_i^2\lesssim\varepsilon\varphi_t. i = N ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 E [ ∥ ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) ∥ 2 1 { τ k , r > i } ] ≤ ε i = N ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 ≲ ε φ t .
For the finitely many terms with i < N i<N i < N , ∥ Π t , i + 1 ∥ = o ( φ t ) \|\Pi_{t,i+1}\|=o(\sqrt{\varphi_t}) ∥ Π t , i + 1 ∥ = o ( φ t ) , so their sum is o ( φ t ) o(\varphi_t) o ( φ t ) . Then, since ε \varepsilon ε is arbitrary, the second sum is also o ( φ t ) o(\varphi_t) o ( φ t ) .
For the last sum in (D.66), by the triangle inequality, (D.64) and (D.70), we obtain
E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } ∥ ] ≤ ∑ i = k t − 1 ∥ Π t , i + 1 ∥ φ i E [ ∥ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ∥ 1 { τ k , r > i } ] ≲ ∑ i = k t − 1 ∥ Π t , i + 1 ∥ φ i η i = o ( φ t ) . (123) \begin{aligned}
\mathbb{E}\Bigg[\Bigg\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\Big[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\Big]\mathbf{1}_{\{\tau_{k,r}>t\}}\Bigg\|\Bigg]\\
\leq\sum_{i=k}^{t-1}\|\Pi_{t,i+1}\|\varphi_i\mathbb{E}\Big[\|\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\|\mathbf{1}_{\{\tau_{k,r}>i\}}\Big]\\
\lesssim\sum_{i=k}^{t-1}\|\Pi_{t,i+1}\|\varphi_i\eta_i=o(\sqrt{\varphi_t}). \tag*{(123)}
\end{aligned} E [ i = k ∑ t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } ] ≤ i = k ∑ t − 1 ∥ Π t , i + 1 ∥ φ i E [ ∥ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ∥ 1 { τ k , r > i } ] ≲ i = k ∑ t − 1 ∥ Π t , i + 1 ∥ φ i η i = o ( φ t ) . (123)
Equations (D.74), (D.75), and (D.76) control the three remainder sums in (D.66) on { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } . We next remove the stopping time. For every ζ > 0 \zeta>0 ζ > 0 , (D.66) gives us the following:
P ( ∥ Δ t − Π t , k Δ k + ∑ i = k t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ φ t 2 ) ≤ P ( τ k , r ≤ t ) + P ( ∥ Δ t − Π t , k Δ k + ∑ i = k t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ 1 { τ k , r > t } > ζ φ t 2 ) . \begin{aligned}
\mathbb{P}\left(\left\|\Delta_t-\Pi_{t,k}\Delta_k+\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|>\frac{\zeta\sqrt{\varphi_t}}{2}\right) \\
&\leq\mathbb{P}(\tau_{k,r}\leq t)+\mathbb{P}\left(\left\|\Delta_t-\Pi_{t,k}\Delta_k+\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|\mathbf{1}_{\{\tau_{k,r}>t\}}>\frac{\zeta\sqrt{\varphi_t}}{2}\right).
\end{aligned} P ( Δ t − Π t , k Δ k + i = k ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > 2 ζ φ t ) ≤ P ( τ k , r ≤ t ) + P ( Δ t − Π t , k Δ k + i = k ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) 1 { τ k , r > t } > 2 ζ φ t ) .
For the fixed ( k , r ) (k,r) ( k , r ) , the vector inside the second probability is the sum of the three remainder terms in (D.66). By the triangle inequality and the union bound, it suffices to show that the probability associated with each remainder term, with threshold ζ φ t / 6 \zeta\sqrt{\varphi_t}/6 ζ φ t /6 , converges to zero.
For the moving-average difference term, by Markov’s inequality and (D.74), we have
P ( ∥ ∑ i = k t − 1 Π t , i + 1 φ i 1 − η i η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) 1 { τ k , r > t } ∥ > ζ φ t 6 ) ≤ 6 ζ φ t E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i 1 − η i η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) ∥ 1 { τ k , r > t } ] = o ( 1 ) . \begin{aligned}
\mathbb{P}\left(\left\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\frac{1-\eta_i}{\eta_i}\mathcal{R}(B^\star)^{-1}(m_i-m_{i-1})\mathbf{1}_{\{\tau_{k,r}>t\}}\right\|>\frac{\zeta\sqrt{\varphi_t}}{6}\right) \\
&\leq\frac{6}{\zeta\sqrt{\varphi_t}}\mathbb{E}\left[\left\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\frac{1-\eta_i}{\eta_i}\mathcal{R}(B^\star)^{-1}(m_i-m_{i-1})\right\|\mathbf{1}_{\{\tau_{k,r}>t\}}\right]=o(1).
\end{aligned} P ( i = k ∑ t − 1 Π t , i + 1 φ i η i 1 − η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) 1 { τ k , r > t } > 6 ζ φ t ) ≤ ζ φ t 6 E [ i = k ∑ t − 1 Π t , i + 1 φ i η i 1 − η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) 1 { τ k , r > t } ] = o ( 1 ) .
For the centered term, by Chebyshev’s inequality and (D.75), we have
P ( ∥ ∑ i = k t − 1 Π t , i + 1 φ i [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] 1 { τ k , r > t } ∥ > ζ φ t 6 ) ≤ 36 ζ 2 φ t E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] 1 { τ k , r > t } ∥ 2 ] = o ( 1 ) . \begin{aligned}
\mathbb{P}\left(\left\Vert\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\left[(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\mathbf{1}_{\{\tau_{k,r}>t\}}\right\Vert>\frac{\zeta\sqrt{\varphi_t}}{6}\right) \\
&\leq\frac{36}{\zeta^2\varphi_t}\mathbb{E}\left[\left\Vert\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\left[(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\mathbf{1}_{\{\tau_{k,r}>t\}}\right\Vert^2\right]=o(1).
\end{aligned} P ( i = k ∑ t − 1 Π t , i + 1 φ i [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 1 { τ k , r > t } > 6 ζ φ t ) ≤ ζ 2 φ t 36 E i = k ∑ t − 1 Π t , i + 1 φ i [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 1 { τ k , r > t } 2 = o ( 1 ) .
For the noncentered term, by Markov’s inequality and (D.76), we have
P ( ∥ ∑ i = k t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } ∥ > ζ φ t 6 ) ≤ 6 ζ φ t E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } ∥ ] = o ( 1 ) . \begin{aligned}
\mathbb{P}\left(\left\Vert\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]\mathbf{1}_{\{\tau_{k,r}>t\}}\right\Vert>\frac{\zeta\sqrt{\varphi_t}}{6}\right) \\
&\leq\frac{6}{\zeta\sqrt{\varphi_t}}\mathbb{E}\left[\left\Vert\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]\mathbf{1}_{\{\tau_{k,r}>t\}}\right\Vert\right]=o(1).
\end{aligned} P ( i = k ∑ t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } > 6 ζ φ t ) ≤ ζ φ t 6 E [ i = k ∑ t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } ] = o ( 1 ) .
Since second probability in the stopping-time decomposition converges to zero, we have
lim sup t → ∞ P ( ∥ Δ t − Π t , k Δ k + ∑ i = k t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ φ t 2 ) ≤ P ( τ k , r < ∞ ) < ϵ 2 . \limsup_{t\to\infty}\mathbb{P}\left(\left\Vert\Delta_t-\Pi_{t,k}\Delta_k+\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\Vert>\frac{\zeta\sqrt{\varphi_t}}{2}\right)\leq\mathbb{P}(\tau_{k,r}<\infty)<\frac{\epsilon}{2}. t → ∞ lim sup P ( Δ t − Π t , k Δ k + i = k ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > 2 ζ φ t ) ≤ P ( τ k , r < ∞ ) < 2 ϵ .
Lastly, we extend the stochastic sum from i = k i=k i = k to i = 0 i=0 i = 0 and we have
Δ t + ∑ i = 0 t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) = Δ t − Π t , k Δ k + ∑ i = k t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + Π t , k { Δ k + ∑ i = 0 k − 1 Π k , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } . \begin{aligned}
\Delta_t+\sum_{i=0}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i) \\
&=\Delta_t-\Pi_{t,k}\Delta_k+\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)
+\Pi_{t,k}\left\{\Delta_k+\sum_{i=0}^{k-1}\Pi_{k,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\}.
\end{aligned} Δ t + i = 0 ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) = Δ t − Π t , k Δ k + i = k ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + Π t , k { Δ k + i = 0 ∑ k − 1 Π k , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } .
For fixed k k k , the quantity inside the braces does not depend on t t t and is finite almost surely. Moreover, (D.70) gives ∥ Π t , k ∥ = o ( φ t ) \|\Pi_{t,k}\|=o(\sqrt{\varphi_t}) ∥ Π t , k ∥ = o ( φ t ) . Therefore, we obtain
1 φ t ∥ Π t , k { Δ k + ∑ i = 0 k − 1 Π k , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } ∥ → a . s . 0. \frac{1}{\sqrt{\varphi_t}}\left\Vert\Pi_{t,k}\left\{\Delta_k+\sum_{i=0}^{k-1}\Pi_{k,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\}\right\|\xrightarrow{\mathrm{a.s.}}0. φ t 1 Π t , k { Δ k + i = 0 ∑ k − 1 Π k , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } a.s. 0.
Hence, the second term on the right-hand side of the preceding decomposition is o p ( φ t ) o_p(\sqrt{\varphi_t}) o p ( φ t ) .
For every ζ > 0 \zeta>0 ζ > 0 , the triangle inequality further yields
P ( ∥ Δ t + ∑ i = 0 t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ φ t ) ≤ P ( ∥ Δ t − Π t , k Δ k + ∑ i = k t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ φ t 2 ) + P ( ∥ Π t , k { Δ k + ∑ i = 0 k − 1 Π k , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } ∥ > ζ φ t 2 ) . \begin{aligned}
\mathbb{P}\left(\left\Vert\Delta_t+\sum_{i=0}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\Vert>\zeta\sqrt{\varphi_t}\right) \\
&\leq\mathbb{P}\left(\left\Vert\Delta_t-\Pi_{t,k}\Delta_k+\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\Vert>\frac{\zeta\sqrt{\varphi_t}}{2}\right) \\
&+\mathbb{P}\left(\left\Vert\Pi_{t,k}\left\{\Delta_k+\sum_{i=0}^{k-1}\Pi_{k,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\}\right\Vert>\frac{\zeta\sqrt{\varphi_t}}{2}\right).
\end{aligned} P ( Δ t + i = 0 ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > ζ φ t ) ≤ P ( Δ t − Π t , k Δ k + i = k ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > 2 ζ φ t ) + P ( Π t , k { Δ k + i = 0 ∑ k − 1 Π k , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } > 2 ζ φ t ) .
For the fixed k = k ( ϵ , r ) k=k(\epsilon,r) k = k ( ϵ , r ) , the preceding bound controls the first probability, whereas the second probability converges to zero by the almost-sure convergence established above. It follows that
lim sup t → ∞ P ( ∥ Δ t + ∑ i = 0 t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ φ t ) ≤ P ( τ k , r < ∞ ) < ϵ 2 . \limsup_{t\to\infty}\mathbb{P}\left(\left\Vert\Delta_t+\sum_{i=0}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\Vert>\zeta\sqrt{\varphi_t}\right)\leq\mathbb{P}(\tau_{k,r}<\infty)<\frac{\epsilon}{2}. t → ∞ lim sup P ( Δ t + i = 0 ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > ζ φ t ) ≤ P ( τ k , r < ∞ ) < 2 ϵ .
Since ϵ > 0 \epsilon> 0 ϵ > 0 is arbitrary and the term inside the probability is independent of k k k , we conclude that
Δ t + ∑ i = 0 t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) = o p ( φ t ) . \Delta_t + \sum_{i=0}^{t-1} \Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i) = o_p(\sqrt{\varphi_t}). Δ t + i = 0 ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) = o p ( φ t ) .
Recalling that R = I − K ⋆ \mathcal{R}=I-K^\star R = I − K ⋆ and dividing both sides by φ t \sqrt{\varphi_t} φ t yield
1 φ t Δ t = − 1 φ t ∑ i = 0 t − 1 Π t , i + 1 φ i ( I − K ⋆ ) ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) . \frac{1}{\sqrt{\varphi_t}}\Delta_t=-\frac{1}{\sqrt{\varphi_t}}\sum_{i=0}^{t-1}\Pi_{t,i+1}\varphi_i(I-K^\star)(B^\star)^{-1}\nabla F(x^\star;\xi_i)+o_p(1). φ t 1 Δ t = − φ t 1 i = 0 ∑ t − 1 Π t , i + 1 φ i ( I − K ⋆ ) ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) .
Then we complete the proof of (15) .
• Proof of (16) . Suppose throughout this part that φ < 1 \varphi< 1 φ < 1 . We return to the fixed localized system ( k , r ) (k,r) ( k , r ) used above. Before averaging (D.65), we first derive a localized first-moment bound for Δ t \Delta_t Δ t .
By Assumptions 3.1 and 3.2, ( ∇ F ( x ⋆ ; ξ i ) , F i ) i ≥ 0 (\nabla F(x^\star;\xi_i),\mathcal{F}_i)_{i\ge0} ( ∇ F ( x ⋆ ; ξ i ) , F i ) i ≥ 0 forms a martingale difference sequence and E [ ∥ ∇ F ( x ⋆ ; ξ i ) ∥ 2 ] ≲ 1 \mathbb{E}[\|\nabla F(x^\star;\xi_i)\|^2]\lesssim1 E [ ∥∇ F ( x ⋆ ; ξ i ) ∥ 2 ] ≲ 1 . Therefore, by (D.70), we have
E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ 2 ] = ∑ i = k t − 1 E [ ∥ Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ 2 ] ≲ ∑ i = k t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 ≲ φ t . (124) \mathbb{E}\left[\left\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|^2\right]
=\sum_{i=k}^{t-1}\mathbb{E}\left[\left\|\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|^2\right]
\lesssim\sum_{i=k}^{t-1}\|\Pi_{t,i+1}\|^2\varphi_i^2\lesssim\varphi_t. \tag*{(124)} E i = k ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) 2 = i = k ∑ t − 1 E [ Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) 2 ] ≲ i = k ∑ t − 1 ∥ Π t , i + 1 ∥ 2 φ i 2 ≲ φ t . (124)
Taking norms in (D.66), multiplying by 1 { τ k , r > t } \mathbf{1}_{\{\tau_{k,r}>t\}} 1 { τ k , r > t } , and using 1 { τ k , r > t } ≤ 1 { τ k , r > i } \mathbf{1}_{\{\tau_{k,r}>t\}}\le\mathbf{1}_{\{\tau_{k,r}>i\}} 1 { τ k , r > t } ≤ 1 { τ k , r > i } for every i ≤ t i\le t i ≤ t , the triangle inequality and the Cauchy–Schwarz inequality yield
E [ ∥ Δ t ∥ 1 { τ k , r > t } ] ≤ r ∥ Π t , k ∥ + { E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ 2 ] } 1 / 2 + E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i 1 − η i η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) ∥ 1 { τ k , r > t } ] + { E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] 1 { τ k , r > i } ∥ 2 ] } 1 / 2 + E [ ∥ ∑ i = k t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] ∥ 1 { τ k , r > t } ] ≲ r ∥ Π t , k ∥ + φ t + o ( φ t ) + { o ( φ t ) } 1 / 2 + o ( φ t ) ≲ φ t . (125) \begin{aligned}
\mathbb{E}[\|\Delta_t\|\mathbf{1}_{\{\tau_{k,r}>t\}}]
&\le r\|\Pi_{t,k}\|+
\left\{\mathbb{E}\left[\left\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|^2\right]\right\}^{1/2}\\
&\quad+\mathbb{E}\left[\left\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\frac{1-\eta_i}{\eta_i}\mathcal{R}(B^\star)^{-1}(m_i-m_{i-1})\right\|\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\\
&\quad+\left\{\mathbb{E}\left[\left\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\left[(\widetilde K_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\mathbf{1}_{\{\tau_{k,r}>i\}}\right\|^2\right]\right\}^{1/2}\\
&\quad+\mathbb{E}\left[\left\|\sum_{i=k}^{t-1}\Pi_{t,i+1}\varphi_i\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]\right\|\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\\
&\lesssim r\|\Pi_{t,k}\|+\sqrt{\varphi_t}+o(\sqrt{\varphi_t})+\{o(\varphi_t)\}^{1/2}+o(\sqrt{\varphi_t})\lesssim\sqrt{\varphi_t}. \tag*{(125)}
\end{aligned} E [ ∥ Δ t ∥ 1 { τ k , r > t } ] ≤ r ∥ Π t , k ∥ + ⎩ ⎨ ⎧ E i = k ∑ t − 1 Π t , i + 1 φ i R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) 2 ⎭ ⎬ ⎫ 1/2 + E [ i = k ∑ t − 1 Π t , i + 1 φ i η i 1 − η i R ( B ⋆ ) − 1 ( m i − m i − 1 ) 1 { τ k , r > t } ] + ⎩ ⎨ ⎧ E i = k ∑ t − 1 Π t , i + 1 φ i [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 1 { τ k , r > i } 2 ⎭ ⎬ ⎫ 1/2 + E [ i = k ∑ t − 1 Π t , i + 1 φ i [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } ] ≲ r ∥ Π t , k ∥ + φ t + o ( φ t ) + { o ( φ t ) } 1/2 + o ( φ t ) ≲ φ t . (125)
where the first inequality uses ∥ Δ k ∥ ≤ r \|\Delta_k\|\le r ∥ Δ k ∥ ≤ r on { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , the second inequality follows from (D.74), (D.75), (D.76) and (D.77), while the last inequality uses ∥ Π t , k ∥ = o ( φ t ) \|\Pi_{t,k}\|=o(\sqrt{\varphi_t}) ∥ Π t , k ∥ = o ( φ t ) from (D.70).
We now rearrange (D.65) as follows,
Δ i = − ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) − R − 1 Δ i + 1 − Δ i φ i + 1 − η i η i ( B ⋆ ) − 1 ( m i − m i − 1 ) + R − 1 [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] − R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] . (126) \begin{aligned}
\Delta_i={}&-(B^\star)^{-1}\nabla F(x^\star;\xi_i)-\mathcal{R}^{-1}\frac{\Delta_{i+1}-\Delta_i}{\varphi_i}+\frac{1-\eta_i}{\eta_i}(B^\star)^{-1}(m_i-m_{i-1})\\
&+\mathcal{R}^{-1}\left[(\widetilde K_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\\
&-\mathcal{R}^{-1}\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]. \tag*{(126)}
\end{aligned} Δ i = − ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) − R − 1 φ i Δ i + 1 − Δ i + η i 1 − η i ( B ⋆ ) − 1 ( m i − m i − 1 ) + R − 1 [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] − R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] . (126)
Since k k k is some fixed index, the averaged iterate can be decomposed as
t ( x ˉ t − x ⋆ ) = 1 t ∑ i = 0 t − 1 Δ i = 1 t ∑ i = 0 k − 1 Δ i + 1 t ∑ i = k t − 1 Δ i . \sqrt{t}(\bar{x}_t-x^\star)=\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}\Delta_i=\frac{1}{\sqrt{t}}\sum_{i=0}^{k-1}\Delta_i+\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\Delta_i. t ( x ˉ t − x ⋆ ) = t 1 i = 0 ∑ t − 1 Δ i = t 1 i = 0 ∑ k − 1 Δ i + t 1 i = k ∑ t − 1 Δ i .
We first analyze the second term and restore the fixed finite prefix at the end. Summing (D.79) from i = k i=k i = k to t − 1 t-1 t − 1 and dividing both sides by t \sqrt{t} t , we obtain
1 t ∑ i = k t − 1 Δ i = − 1 t ∑ i = k t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) − 1 t R − 1 ∑ i = k t − 1 Δ i + 1 − Δ i φ i + 1 t ( B ⋆ ) − 1 ∑ i = k t − 1 1 − η i η i ( m i − m i − 1 ) + 1 t R − 1 ∑ i = k t − 1 [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] − 1 t R − 1 ∑ i = k t − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] . (127) \begin{aligned}
\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\Delta_i
={}&-\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)-\frac{1}{\sqrt{t}}\mathcal{R}^{-1}\sum_{i=k}^{t-1}\frac{\Delta_{i+1}-\Delta_i}{\varphi_i}\\
&+\frac{1}{\sqrt{t}}(B^\star)^{-1}\sum_{i=k}^{t-1}\frac{1-\eta_i}{\eta_i}(m_i-m_{i-1})\\
&+\frac{1}{\sqrt{t}}\mathcal{R}^{-1}\sum_{i=k}^{t-1}\left[(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\\
&-\frac{1}{\sqrt{t}}\mathcal{R}^{-1}\sum_{i=k}^{t-1}\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]. \tag*{(127)}
\end{aligned} t 1 i = k ∑ t − 1 Δ i = − t 1 i = k ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) − t 1 R − 1 i = k ∑ t − 1 φ i Δ i + 1 − Δ i + t 1 ( B ⋆ ) − 1 i = k ∑ t − 1 η i 1 − η i ( m i − m i − 1 ) + t 1 R − 1 i = k ∑ t − 1 [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] − t 1 R − 1 i = k ∑ t − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] . (127)
The first term on the right-hand side is the leading stochastic term, and it remains to show that each of the other four terms is o P ( 1 ) o_P(1) o P ( 1 ) .
For the second term on the right-hand side of (D.80), summation by parts gives us
∑ i = k t − 1 Δ i + 1 − Δ i φ i = Δ t φ t − 1 − Δ k φ k + ∑ i = k + 1 t − 1 ( 1 φ i − 1 − 1 φ i ) Δ i . (128) \sum_{i=k}^{t-1}\frac{\Delta_{i+1}-\Delta_i}{\varphi_i}
=\frac{\Delta_t}{\varphi_{t-1}}-\frac{\Delta_k}{\varphi_k}
+\sum_{i=k+1}^{t-1}\left(\frac{1}{\varphi_{i-1}}-\frac{1}{\varphi_i}\right)\Delta_i. \tag*{(128)} i = k ∑ t − 1 φ i Δ i + 1 − Δ i = φ t − 1 Δ t − φ k Δ k + i = k + 1 ∑ t − 1 ( φ i − 1 1 − φ i 1 ) Δ i . (128)
Since φ i − 1 = ( i + 1 ) φ / C φ \varphi_i^{-1}=(i+1)^\varphi/C_\varphi φ i − 1 = ( i + 1 ) φ / C φ , by the mean-value theorem, we have
∣ 1 φ i − 1 φ i − 1 ∣ = 1 C φ ∣ ( i + 1 ) φ − i φ ∣ ≲ ( i + 1 ) φ − 1 . \left|\frac{1}{\varphi_i}-\frac{1}{\varphi_{i-1}}\right|
=\frac{1}{C_\varphi}\left|(i+1)^\varphi-i^\varphi\right|
\lesssim(i+1)^{\varphi-1}. φ i 1 − φ i − 1 1 = C φ 1 ∣ ( i + 1 ) φ − i φ ∣ ≲ ( i + 1 ) φ − 1 .
Therefore, by (D.78) and 1 { τ k , r > t } ≤ 1 { τ k , r > i } \mathbf{1}_{\{\tau_{k,r}>t\}}\leq\mathbf{1}_{\{\tau_{k,r}>i\}} 1 { τ k , r > t } ≤ 1 { τ k , r > i } , we obtain
1 t E [ ∥ ∑ i = k t − 1 Δ i + 1 − Δ i φ i ∥ 1 { τ k , r > t } ] ≤ 1 t { E [ ∥ Δ t ∥ 1 { τ k , r > t } ] φ t − 1 + r φ k + ∑ i = k + 1 t − 1 ∣ 1 φ i − 1 − 1 φ i ∣ E [ ∥ Δ i ∥ 1 { τ k , r > i } ] } ≲ 1 t { 1 φ t + 1 + ∑ i = k + 1 t − 1 ( i + 1 ) φ − 1 φ i } ≲ 1 t ( 1 − φ ) / 2 = o ( 1 ) . (129) \begin{aligned}
\frac{1}{\sqrt{t}}\mathbb{E}\left[\left\|\sum_{i=k}^{t-1}\frac{\Delta_{i+1}-\Delta_i}{\varphi_i}\right\|\mathbf{1}_{\{\tau_{k,r}>t\}}\right]
&\leq\frac{1}{\sqrt{t}}\left\{\frac{\mathbb{E}[\|\Delta_t\|\mathbf{1}_{\{\tau_{k,r}>t\}}]}{\varphi_{t-1}}+\frac{r}{\varphi_k}+\sum_{i=k+1}^{t-1}\left|\frac{1}{\varphi_{i-1}}-\frac{1}{\varphi_i}\right|\mathbb{E}[\|\Delta_i\|\mathbf{1}_{\{\tau_{k,r}>i\}}]\right\}\\
&\lesssim\frac{1}{\sqrt{t}}\left\{\frac{1}{\sqrt{\varphi_t}}+1+\sum_{i=k+1}^{t-1}(i+1)^{\varphi-1}\sqrt{\varphi_i}\right\}
\lesssim\frac{1}{t^{(1-\varphi)/2}}=o(1). \tag*{(129)}
\end{aligned} t 1 E [ i = k ∑ t − 1 φ i Δ i + 1 − Δ i 1 { τ k , r > t } ] ≤ t 1 { φ t − 1 E [ ∥ Δ t ∥ 1 { τ k , r > t } ] + φ k r + i = k + 1 ∑ t − 1 φ i − 1 1 − φ i 1 E [ ∥ Δ i ∥ 1 { τ k , r > i } ] } ≲ t 1 { φ t 1 + 1 + i = k + 1 ∑ t − 1 ( i + 1 ) φ − 1 φ i } ≲ t ( 1 − φ ) /2 1 = o ( 1 ) . (129)
Here, r / φ k r/\varphi_k r / φ k follows from ∥ Δ k ∥ ≤ r \|\Delta_k\|\leq r ∥ Δ k ∥ ≤ r on { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } .
For the third term on the right-hand side of (D.80), again we apply summation by parts and obtain
∑ i = k t − 1 1 − η i η i ( m i − m i − 1 ) = 1 − η t − 1 η t − 1 m t − 1 − 1 − η k η k m k − 1 + ∑ i = k t − 2 ( 1 − η i η i − 1 − η i + 1 η i + 1 ) m i . (130) \sum_{i=k}^{t-1}\frac{1-\eta_i}{\eta_i}(m_i-m_{i-1})
=\frac{1-\eta_{t-1}}{\eta_{t-1}}m_{t-1}-\frac{1-\eta_k}{\eta_k}m_{k-1}
+\sum_{i=k}^{t-2}\left(\frac{1-\eta_i}{\eta_i}-\frac{1-\eta_{i+1}}{\eta_{i+1}}\right)m_i. \tag*{(130)} i = k ∑ t − 1 η i 1 − η i ( m i − m i − 1 ) = η t − 1 1 − η t − 1 m t − 1 − η k 1 − η k m k − 1 + i = k ∑ t − 2 ( η i 1 − η i − η i + 1 1 − η i + 1 ) m i . (130)
In addition, the mean-value theorem gives us
∣ 1 − η i η i − 1 − η i + 1 η i + 1 ∣ ≲ ( i + 1 ) η − 1 ≲ 1 ( i + 1 ) η i . \left|\frac{1-\eta_i}{\eta_i}-\frac{1-\eta_{i+1}}{\eta_{i+1}}\right|
\lesssim(i+1)^{\eta-1}\lesssim\frac{1}{(i+1)\eta_i}. η i 1 − η i − η i + 1 1 − η i + 1 ≲ ( i + 1 ) η − 1 ≲ ( i + 1 ) η i 1 .
Using (D.58), 1 { τ k , r > t } ≤ 1 { τ k , r > i } \mathbf{1}_{\{\tau_{k,r}>t\}}\le\mathbf{1}_{\{\tau_{k,r}>i\}} 1 { τ k , r > t } ≤ 1 { τ k , r > i } , and Jensen’s inequality, we further obtain
1 t E [ ∥ ∑ i = k t − 1 1 − η i η i ( m i − m i − 1 ) 1 { τ k , r > t } ∥ ] ≤ 1 t { 1 η t − 1 E [ ∥ m t − 1 ∥ 1 { τ k , r > t } ] + 1 − η k η k E [ ∥ m k − 1 ∥ 1 { τ k , r > k } ] + ∑ i = k t − 2 1 ( i + 1 ) η i E [ ∥ m i ∥ 1 { τ k , r > t } ] } ≤ 1 t { 1 η t − 1 { E [ ∥ m t − 1 ∥ 2 1 { τ k , r > t − 1 } ] } 1 / 2 + 1 − η k η k E [ ∥ m k − 1 ∥ 1 { τ k , r > k } ] + ∑ i = k t − 2 1 ( i + 1 ) η i { E [ ∥ m i ∥ 2 1 { τ k , r > i } ] } 1 / 2 } ≲ 1 t { η t − 1 η t − 1 + 1 + ∑ i = k t − 2 η i ( i + 1 ) η i } = 1 t { 1 η t − 1 + 1 + ∑ i = k t − 2 1 ( i + 1 ) η i } ≲ 1 t { t η / 2 + 1 + ∑ i = k t − 2 ( i + 1 ) − 1 + η / 2 } ≲ 1 t { t η / 2 + 1 + ( t + 1 ) η / 2 } ≲ 1 t ( 1 − η ) / 2 = o ( 1 ) . (131) \begin{aligned}
\frac{1}{\sqrt{t}}\mathbb{E}\left[\left\|\sum_{i=k}^{t-1}\frac{1-\eta_i}{\eta_i}(m_i-m_{i-1})\mathbf{1}_{\{\tau_{k,r}>t\}}\right\|\right]
&\le\frac{1}{\sqrt{t}}\left\{\frac{1}{\eta_{t-1}}\mathbb{E}[\|m_{t-1}\|\mathbf{1}_{\{\tau_{k,r}>t\}}]+\frac{1-\eta_k}{\eta_k}\mathbb{E}[\|m_{k-1}\|\mathbf{1}_{\{\tau_{k,r}>k\}}]\right.\\
&\qquad\left.+\sum_{i=k}^{t-2}\frac{1}{(i+1)\eta_i}\mathbb{E}[\|m_i\|\mathbf{1}_{\{\tau_{k,r}>t\}}]\right\}\\
&\le\frac{1}{\sqrt{t}}\left\{\frac{1}{\eta_{t-1}}\left\{\mathbb{E}[\|m_{t-1}\|^2\mathbf{1}_{\{\tau_{k,r}>t-1\}}]\right\}^{1/2}+\frac{1-\eta_k}{\eta_k}\mathbb{E}[\|m_{k-1}\|\mathbf{1}_{\{\tau_{k,r}>k\}}]\right.\\
&\qquad\left.+\sum_{i=k}^{t-2}\frac{1}{(i+1)\eta_i}\left\{\mathbb{E}[\|m_i\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}]\right\}^{1/2}\right\}\\
&\lesssim\frac{1}{\sqrt{t}}\left\{\frac{\sqrt{\eta_{t-1}}}{\eta_{t-1}}+1+\sum_{i=k}^{t-2}\frac{\sqrt{\eta_i}}{(i+1)\eta_i}\right\}
=\frac{1}{\sqrt{t}}\left\{\frac{1}{\sqrt{\eta_{t-1}}}+1+\sum_{i=k}^{t-2}\frac{1}{(i+1)\sqrt{\eta_i}}\right\}\\
&\lesssim\frac{1}{\sqrt{t}}\left\{t^{\eta/2}+1+\sum_{i=k}^{t-2}(i+1)^{-1+\eta/2}\right\}
\lesssim\frac{1}{\sqrt{t}}\left\{t^{\eta/2}+1+(t+1)^{\eta/2}\right\}\\
&\lesssim\frac{1}{t^{(1-\eta)/2}}=o(1). \tag*{(131)}
\end{aligned} t 1 E [ i = k ∑ t − 1 η i 1 − η i ( m i − m i − 1 ) 1 { τ k , r > t } ] ≤ t 1 { η t − 1 1 E [ ∥ m t − 1 ∥ 1 { τ k , r > t } ] + η k 1 − η k E [ ∥ m k − 1 ∥ 1 { τ k , r > k } ] + i = k ∑ t − 2 ( i + 1 ) η i 1 E [ ∥ m i ∥ 1 { τ k , r > t } ] } ≤ t 1 { η t − 1 1 { E [ ∥ m t − 1 ∥ 2 1 { τ k , r > t − 1 } ] } 1/2 + η k 1 − η k E [ ∥ m k − 1 ∥ 1 { τ k , r > k } ] + i = k ∑ t − 2 ( i + 1 ) η i 1 { E [ ∥ m i ∥ 2 1 { τ k , r > i } ] } 1/2 } ≲ t 1 { η t − 1 η t − 1 + 1 + i = k ∑ t − 2 ( i + 1 ) η i η i } = t 1 { η t − 1 1 + 1 + i = k ∑ t − 2 ( i + 1 ) η i 1 } ≲ t 1 { t η /2 + 1 + i = k ∑ t − 2 ( i + 1 ) − 1 + η /2 } ≲ t 1 { t η /2 + 1 + ( t + 1 ) η /2 } ≲ t ( 1 − η ) /2 1 = o ( 1 ) . (131)
Here, ∥ m k − 1 ∥ ≤ ( 1 + Υ H ) r \|m_{k-1}\|\le(1+\Upsilon_H)r ∥ m k − 1 ∥ ≤ ( 1 + Υ H ) r on { τ k , r > k } \{\tau_{k,r}>k\} { τ k , r > k } , so the fixed boundary term is absorbed into the hidden constant; the last equality directly follows from η < φ < 1 \eta<\varphi<1 η < φ < 1 .
For the fourth term on the right-hand side of (D.80), by (D.61) and (D.62), we obtain
E [ ∥ 1 t ∑ i = k t − 1 R − 1 [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] ∥ 2 1 { τ k , r > t } ] ≤ E [ ∥ 1 t ∑ i = k t − 1 R − 1 [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] 1 { τ k , r > i } ∥ 2 ] = 1 t ∑ i = k t − 1 E [ ∥ R − 1 [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] ∥ 2 1 { τ k , r > i } ] ≲ 1 t ∑ i = k t − 1 η i + 1 t ∑ i = k t − 1 E [ ∥ ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) ∥ 2 1 { τ k , r > i } ] = o ( 1 ) . (132) \begin{aligned}
&\mathbb{E}\left[\left\|\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\mathcal{R}^{-1}\left[(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\right\|^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\\
&\le\mathbb{E}\left[\left\|\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\mathcal{R}^{-1}\left[(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\mathbf{1}_{\{\tau_{k,r}>i\}}\right\|^2\right]\\
&=\frac{1}{t}\sum_{i=k}^{t-1}\mathbb{E}\left[\left\|\mathcal{R}^{-1}\left[(\widetilde{K}_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\right\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}\right]\\
&\lesssim\frac{1}{t}\sum_{i=k}^{t-1}\eta_i+\frac{1}{t}\sum_{i=k}^{t-1}\mathbb{E}[\|(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\|^2\mathbf{1}_{\{\tau_{k,r}>i\}}]=o(1). \tag*{(132)}
\end{aligned} E t 1 i = k ∑ t − 1 R − 1 [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 2 1 { τ k , r > t } ≤ E t 1 i = k ∑ t − 1 R − 1 [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 1 { τ k , r > i } 2 = t 1 i = k ∑ t − 1 E [ R − 1 [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 2 1 { τ k , r > i } ] ≲ t 1 i = k ∑ t − 1 η i + t 1 i = k ∑ t − 1 E [ ∥ ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) ∥ 2 1 { τ k , r > i } ] = o ( 1 ) . (132)
Here, the first sum is O ( t − η ) O(t^{-\eta}) O ( t − η ) since η < 1 \eta<1 η < 1 ; the second is o ( 1 ) o(1) o ( 1 ) by (D.60) and Cesàro’s lemma.
For the final term on the right-hand side of (D.80), by the triangle inequality and (D.64), we obtain
E [ ∥ 1 t ∑ i = k t − 1 R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] ∥ 1 { τ k , r > t } ] ≤ ∥ R − 1 ∥ t ∑ i = k t − 1 E [ ∥ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ∥ 1 { τ k , r > i } ] ≲ 1 t ∑ i = k t − 1 η i ≲ 1 t η − 1 / 2 = o ( 1 ) , (133) \begin{aligned}
&\mathbb{E}\left[\left\|\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\mathcal{R}^{-1}\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]\right\|\mathbf{1}_{\{\tau_{k,r}>t\}}\right]\\
&\le\frac{\|\mathcal{R}^{-1}\|}{\sqrt{t}}\sum_{i=k}^{t-1}\mathbb{E}\left[\left\|\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right\|\mathbf{1}_{\{\tau_{k,r}>i\}}\right]\\
&\lesssim\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\eta_i\lesssim\frac{1}{t^{\eta-1/2}}=o(1), \tag*{(133)}
\end{aligned} E [ t 1 i = k ∑ t − 1 R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } ] ≤ t ∥ R − 1 ∥ i = k ∑ t − 1 E [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i 1 { τ k , r > i } ] ≲ t 1 i = k ∑ t − 1 η i ≲ t η − 1/2 1 = o ( 1 ) , (133)
where the last equality follows from η > 1 / 2 \eta>1/2 η > 1/2 .
Similarly to the proof of (15) , it remains to remove the stopping time and restore the finitely many indices preceding k k k . For every ζ > 0 \zeta> 0 ζ > 0 , the stopping-time decomposition gives
P ( ∥ 1 t ∑ i = k t − 1 Δ i + 1 t ∑ i = k t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ 2 ) ≤ P ( τ k , r ≤ t ) + P ( ∥ 1 t ∑ i = k t − 1 Δ i + 1 t ∑ i = k t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ 1 { τ k , r > t } > ζ 2 ) . \mathbb{P}\left(\left\Vert\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\Delta_i+\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\Vert>\frac{\zeta}{2}\right)
\leq\mathbb{P}(\tau_{k,r}\leq t)+\mathbb{P}\left(\left\Vert\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\Delta_i+\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}>\frac{\zeta}{2}\right). P ( t 1 i = k ∑ t − 1 Δ i + t 1 i = k ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > 2 ζ ) ≤ P ( τ k , r ≤ t ) + P ( t 1 i = k ∑ t − 1 Δ i + t 1 i = k ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) 1 { τ k , r > t } > 2 ζ ) .
On { τ k , r > t } \{\tau_{k,r}>t\} { τ k , r > t } , by (D.80) and the triangle inequality, we have
∥ 1 t ∑ i = k t − 1 Δ i + 1 t ∑ i = k t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ 1 { τ k , r > t } ≤ ∥ R − 1 ∥ t ∥ ∑ i = k t − 1 Δ i + 1 − Δ i φ i ∥ 1 { τ k , r > t } + ∥ ( B ⋆ ) − 1 ∥ t ∥ ∑ i = k t − 1 1 − η i η i ( m i − m i − 1 ) ∥ 1 { τ k , r > t } + ∥ 1 t ∑ i = k t − 1 R − 1 [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] ∥ 1 { τ k , r > t } + ∥ 1 t ∑ i = k t − 1 R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] ∥ 1 { τ k , r > t } . \begin{aligned}
\left\Vert\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\Delta_i+\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}
&\leq\frac{\|\mathcal{R}^{-1}\|}{\sqrt{t}}\left\Vert\sum_{i=k}^{t-1}\frac{\Delta_{i+1}-\Delta_i}{\varphi_i}\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}\\
&\quad+\frac{\|(B^\star)^{-1}\|}{\sqrt{t}}\left\Vert\sum_{i=k}^{t-1}\frac{1-\eta_i}{\eta_i}(m_i-m_{i-1})\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}\\
&\quad+\left\Vert\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\mathcal{R}^{-1}\left[(\widetilde K_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}\\
&\quad+\left\Vert\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\mathcal{R}^{-1}\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}.
\end{aligned} t 1 i = k ∑ t − 1 Δ i + t 1 i = k ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) 1 { τ k , r > t } ≤ t ∥ R − 1 ∥ i = k ∑ t − 1 φ i Δ i + 1 − Δ i 1 { τ k , r > t } + t ∥ ( B ⋆ ) − 1 ∥ i = k ∑ t − 1 η i 1 − η i ( m i − m i − 1 ) 1 { τ k , r > t } + t 1 i = k ∑ t − 1 R − 1 [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 1 { τ k , r > t } + t 1 i = k ∑ t − 1 R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } .
Therefore, if the left-hand side is greater than ζ / 2 \zeta/2 ζ /2 , then at least one term on the right-hand side must be greater than ζ / 8 \zeta/8 ζ /8 . By the union bound, it remains to control the corresponding four probabilities.
For the iterate-difference term, by Markov’s inequality and (D.82), we have
P ( ∥ R − 1 ∥ t ∥ ∑ i = k t − 1 Δ i + 1 − Δ i φ i ∥ 1 { τ k , r > t } > ζ 8 ) ≤ 8 ∥ R − 1 ∥ ζ t E [ ∥ ∑ i = k t − 1 Δ i + 1 − Δ i φ i ∥ 1 { τ k , r > t } ] = o ( 1 ) . \mathbb{P}\left(\frac{\|\mathcal{R}^{-1}\|}{\sqrt{t}}\left\Vert\sum_{i=k}^{t-1}\frac{\Delta_{i+1}-\Delta_i}{\varphi_i}\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}>\frac{\zeta}{8}\right)
\leq\frac{8\|\mathcal{R}^{-1}\|}{\zeta\sqrt{t}}\mathbb{E}\left[\left\Vert\sum_{i=k}^{t-1}\frac{\Delta_{i+1}-\Delta_i}{\varphi_i}\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}\right]=o(1). P ( t ∥ R − 1 ∥ i = k ∑ t − 1 φ i Δ i + 1 − Δ i 1 { τ k , r > t } > 8 ζ ) ≤ ζ t 8∥ R − 1 ∥ E [ i = k ∑ t − 1 φ i Δ i + 1 − Δ i 1 { τ k , r > t } ] = o ( 1 ) .
For the moving-average difference term, by Markov’s inequality and (D.84), we have
P ( ∥ ( B ⋆ ) − 1 ∥ t ∥ ∑ i = k t − 1 1 − η i η i ( m i − m i − 1 ) ∥ 1 { τ k , r > t } > ζ 8 ) ≤ 8 ∥ ( B ⋆ ) − 1 ∥ ζ t E [ ∥ ∑ i = k t − 1 1 − η i η i ( m i − m i − 1 ) ∥ 1 { τ k , r > t } ] = o ( 1 ) . \mathbb{P}\left(\frac{\|(B^\star)^{-1}\|}{\sqrt{t}}\left\Vert\sum_{i=k}^{t-1}\frac{1-\eta_i}{\eta_i}(m_i-m_{i-1})\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}>\frac{\zeta}{8}\right)
\leq\frac{8\|(B^\star)^{-1}\|}{\zeta\sqrt{t}}\mathbb{E}\left[\left\Vert\sum_{i=k}^{t-1}\frac{1-\eta_i}{\eta_i}(m_i-m_{i-1})\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}\right]=o(1). P ( t ∥ ( B ⋆ ) − 1 ∥ i = k ∑ t − 1 η i 1 − η i ( m i − m i − 1 ) 1 { τ k , r > t } > 8 ζ ) ≤ ζ t 8∥ ( B ⋆ ) − 1 ∥ E [ i = k ∑ t − 1 η i 1 − η i ( m i − m i − 1 ) 1 { τ k , r > t } ] = o ( 1 ) .
For the centered term, by Chebyshev’s inequality and (D.85), we have
P ( ∥ 1 t ∑ i = k t − 1 R − 1 [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] ∥ 1 { τ k , r > t } > ζ 8 ) ≤ 64 ζ 2 E [ ∥ 1 t ∑ i = k t − 1 R − 1 [ ( K ~ i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 { ( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i ) } ] ∥ 2 1 { τ k , r > t } ] = o ( 1 ) . \begin{aligned}
&\mathbb{P}\left(\left\Vert\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\mathcal{R}^{-1}\left[(\widetilde K_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}>\frac{\zeta}{8}\right)\\
&\quad\leq\frac{64}{\zeta^2}\mathbb{E}\left[\left\Vert\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\mathcal{R}^{-1}\left[(\widetilde K_i-K_i)B_i^{-1}m_i-\mathcal{R}(B^\star)^{-1}\{(g_i-\nabla f_i)-\nabla F(x^\star;\xi_i)\}\right]\right\Vert^2\mathbf{1}_{\{\tau_{k,r}>t\}}\right]=o(1).
\end{aligned} P ( t 1 i = k ∑ t − 1 R − 1 [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 1 { τ k , r > t } > 8 ζ ) ≤ ζ 2 64 E t 1 i = k ∑ t − 1 R − 1 [ ( K i − K i ) B i − 1 m i − R ( B ⋆ ) − 1 {( g i − ∇ f i ) − ∇ F ( x ⋆ ; ξ i )} ] 2 1 { τ k , r > t } = o ( 1 ) .
For the noncentered term, by Markov’s inequality and (D.86), we have
P ( ∥ 1 t ∑ i = k t − 1 R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] ∥ 1 { τ k , r > t } > ζ 8 ) ≤ 8 ζ E [ ∥ 1 t ∑ i = k t − 1 R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + { ( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] ∥ 1 { τ k , r > t } ] = o ( 1 ) . \begin{aligned}
&\mathbb{P}\left(\left\Vert\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\mathcal{R}^{-1}\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}>\frac{\zeta}{8}\right)\\
&\quad\leq\frac{8}{\zeta}\mathbb{E}\left[\left\Vert\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\mathcal{R}^{-1}\left[\mathcal{R}(B^\star)^{-1}(\nabla f_i-B^\star\Delta_i)+\{(I-K_i)B_i^{-1}-\mathcal{R}(B^\star)^{-1}\}m_i\right]\right\Vert\mathbf{1}_{\{\tau_{k,r}>t\}}\right]=o(1).
\end{aligned} P ( t 1 i = k ∑ t − 1 R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } > 8 ζ ) ≤ ζ 8 E [ t 1 i = k ∑ t − 1 R − 1 [ R ( B ⋆ ) − 1 ( ∇ f i − B ⋆ Δ i ) + {( I − K i ) B i − 1 − R ( B ⋆ ) − 1 } m i ] 1 { τ k , r > t } ] = o ( 1 ) .
Since all four probabilities converge to zero, the second probability in the stopping-time decomposition converges to zero. As a result, we obtain
lim sup t → ∞ P ( ∥ 1 t ∑ i = k t − 1 Δ i + 1 t ∑ i = k t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ 2 ) ≤ P ( τ k , r < ∞ ) < ϵ 2 . \limsup_{t\to\infty}\mathbb{P}\left(\left\|\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\Delta_i+\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|>\frac{\zeta}{2}\right)\leq\mathbb{P}(\tau_{k,r}<\infty)<\frac{\epsilon}{2}. t → ∞ lim sup P ( t 1 i = k ∑ t − 1 Δ i + t 1 i = k ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > 2 ζ ) ≤ P ( τ k , r < ∞ ) < 2 ϵ .
Lastly, we extend both sums from i = k i=k i = k to i = 0 i=0 i = 0 . For the fixed k k k , we have
1 t ∑ i = 0 k − 1 { Δ i + ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } → a.s. 0. \frac{1}{\sqrt{t}}\sum_{i=0}^{k-1}\left\{\Delta_i+(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\}\xrightarrow{\text{a.s.}}0. t 1 i = 0 ∑ k − 1 { Δ i + ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } a.s. 0.
Therefore, by the triangle inequality and the union bound,
P ( ∥ 1 t ∑ i = 0 t − 1 Δ i + 1 t ∑ i = 0 t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ ) ≤ P ( ∥ 1 t ∑ i = k t − 1 Δ i + 1 t ∑ i = k t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ 2 ) + P ( ∥ 1 t ∑ i = 0 k − 1 { Δ i + ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } ∥ > ζ 2 ) . \begin{aligned}
\mathbb{P}\left(\left\|\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}\Delta_i+\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|>\zeta\right) \\
&\leq\mathbb{P}\left(\left\|\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}\Delta_i+\frac{1}{\sqrt{t}}\sum_{i=k}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|>\frac{\zeta}{2}\right) \\
&+\mathbb{P}\left(\left\|\frac{1}{\sqrt{t}}\sum_{i=0}^{k-1}\left\{\Delta_i+(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\}\right\|>\frac{\zeta}{2}\right).
\end{aligned} P ( t 1 i = 0 ∑ t − 1 Δ i + t 1 i = 0 ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > ζ ) ≤ P ( t 1 i = k ∑ t − 1 Δ i + t 1 i = k ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > 2 ζ ) + P ( t 1 i = 0 ∑ k − 1 { Δ i + ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) } > 2 ζ ) .
Since the second probability on the right-hand side converges to zero by the preceding almost-sure convergence, then it follows that
lim sup t → ∞ P ( ∥ 1 t ∑ i = 0 t − 1 Δ i + 1 t ∑ i = 0 t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ > ζ ) ≤ P ( τ k , r < ∞ ) < ϵ 2 . \limsup_{t\to\infty}\mathbb{P}\left(\left\|\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}\Delta_i+\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|>\zeta\right)\leq\mathbb{P}(\tau_{k,r}<\infty)<\frac{\epsilon}{2}. t → ∞ lim sup P ( t 1 i = 0 ∑ t − 1 Δ i + t 1 i = 0 ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) > ζ ) ≤ P ( τ k , r < ∞ ) < 2 ϵ .
Since ϵ > 0 \epsilon>0 ϵ > 0 is arbitrary, we conclude that
t ( x ˉ t − x ⋆ ) = 1 t ∑ i = 0 t − 1 ( x i − x ⋆ ) = − 1 t ∑ i = 0 t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) , \sqrt{t}(\bar{x}_t-x^\star)=\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}(x_i-x^\star)=-\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)+o_p(1), t ( x ˉ t − x ⋆ ) = t 1 i = 0 ∑ t − 1 ( x i − x ⋆ ) = − t 1 i = 0 ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) ,
which proves (16) and completes the proof.
Proof of Theorem 4.5
By Lemma 4.1 , R \mathcal{R} R is symmetric positive definite. By (15) and Slutsky’s theorem, it suffices to establish the asymptotic normality of its leading stochastic term. For every 0 ≤ i ≤ t − 1 0\leq i\leq t-1 0 ≤ i ≤ t − 1 , define
W t , i : = − φ i φ t Π t , i + 1 R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) . (134) W_{t,i}:=-\frac{\varphi_i}{\sqrt{\varphi_t}}\Pi_{t,i+1}\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i). \tag*{(134)} W t , i := − φ t φ i Π t , i + 1 R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) . (134)
E [ ∇ F ( x ⋆ ; ξ i ) ∣ F i − 1 ] = ∇ f ( x ⋆ ) = 0. \mathbb{E}[\nabla F(x^\star;\xi_i)\mid\mathcal{F}_{i-1}]=\nabla f(x^\star)=0. E [ ∇ F ( x ⋆ ; ξ i ) ∣ F i − 1 ] = ∇ f ( x ⋆ ) = 0.
Since Π t , i + 1 \Pi_{t,i+1} Π t , i + 1 is deterministic, it follows that
E [ W t , i ∣ F i − 1 ] = − φ i φ t Π t , i + 1 R ( B ⋆ ) − 1 E [ ∇ F ( x ⋆ ; ξ i ) ∣ F i − 1 ] = 0. \mathbb{E}[W_{t,i}\mid\mathcal{F}_{i-1}]=-\frac{\varphi_i}{\sqrt{\varphi_t}}\Pi_{t,i+1}\mathcal{R}(B^\star)^{-1}\mathbb{E}[\nabla F(x^\star;\xi_i)\mid\mathcal{F}_{i-1}]=0. E [ W t , i ∣ F i − 1 ] = − φ t φ i Π t , i + 1 R ( B ⋆ ) − 1 E [ ∇ F ( x ⋆ ; ξ i ) ∣ F i − 1 ] = 0.
Verification of (B.2). By the definitions of Ω \Omega Ω and Γ m , a s \Gamma_{m,\mathrm{as}} Γ m , as , we have
E [ R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∇ F ( x ⋆ ; ξ i ) ⊤ ( B ⋆ ) − 1 R ⊤ ∣ F i − 1 ] = R Ω R ⊤ = Γ m , a s . \mathbb{E}\left[\mathcal{R}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\nabla F(x^\star;\xi_i)^\top(B^\star)^{-1}\mathcal{R}^\top\mid\mathcal{F}_{i-1}\right]=\mathcal{R}\Omega\mathcal{R}^\top=\Gamma_{m,\mathrm{as}}. E [ R ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∇ F ( x ⋆ ; ξ i ) ⊤ ( B ⋆ ) − 1 R ⊤ ∣ F i − 1 ] = R Ω R ⊤ = Γ m , as .
Therefore, the conditional covariance sum of the triangular array is deterministic and satisfies
V t : = ∑ i = 0 t − 1 E [ W t , i W t , i ⊤ ∣ F i − 1 ] = 1 φ t ∑ i = 0 t − 1 Π t , i + 1 φ i 2 Γ m , a s Π t , i + 1 ⊤ . (135) \mathcal{V}_t:=\sum_{i=0}^{t-1}\mathbb{E}[W_{t,i}W_{t,i}^\top\mid\mathcal{F}_{i-1}]=\frac{1}{\varphi_t}\sum_{i=0}^{t-1}\Pi_{t,i+1}\varphi_i^2\Gamma_{m,\mathrm{as}}\Pi_{t,i+1}^\top. \tag*{(135)} V t := i = 0 ∑ t − 1 E [ W t , i W t , i ⊤ ∣ F i − 1 ] = φ t 1 i = 0 ∑ t − 1 Π t , i + 1 φ i 2 Γ m , as Π t , i + 1 ⊤ . (135)
Since Π t + 1 , i + 1 = ( I − φ t R ) Π t , i + 1 \Pi_{t+1,i+1}=(I-\varphi_t\mathcal{R})\Pi_{t,i+1} Π t + 1 , i + 1 = ( I − φ t R ) Π t , i + 1 for every i ≤ t − 1 i\le t-1 i ≤ t − 1 , we obtain the following recursion:
V t + 1 = φ t φ t + 1 ( I − φ t R ) V t ( I − φ t R ) ⊤ + φ t 2 φ t + 1 Γ m , a s . (136) \mathcal{V}_{t+1}=\frac{\varphi_t}{\varphi_{t+1}}(I-\varphi_t\mathcal{R})\mathcal{V}_t(I-\varphi_t\mathcal{R})^{\top}+\frac{\varphi_t^2}{\varphi_{t+1}}\Gamma_{m,\mathrm{as}}. \tag*{(136)} V t + 1 = φ t + 1 φ t ( I − φ t R ) V t ( I − φ t R ) ⊤ + φ t + 1 φ t 2 Γ m , as . (136)
If φ < 1 \varphi<1 φ < 1 , the mean-value theorem gives φ t / φ t + 1 − 1 = O ( ( t + 1 ) − 1 ) = o ( φ t ) \varphi_t/\varphi_{t+1}-1=O((t+1)^{-1})=o(\varphi_t) φ t / φ t + 1 − 1 = O (( t + 1 ) − 1 ) = o ( φ t ) . If φ = 1 \varphi=1 φ = 1 , we have φ t / φ t + 1 = 1 + φ t / C φ \varphi_t/\varphi_{t+1}=1+\varphi_t/C_\varphi φ t / φ t + 1 = 1 + φ t / C φ . Hence, in both cases, the following relations hold:
φ t φ t + 1 = 1 + 2 ζ φ φ t + o ( φ t ) , φ t 2 φ t + 1 = φ t + o ( φ t ) . (137) \frac{\varphi_t}{\varphi_{t+1}}=1+2\zeta_\varphi\varphi_t+o(\varphi_t),\qquad\frac{\varphi_t^2}{\varphi_{t+1}}=\varphi_t+o(\varphi_t). \tag*{(137)} φ t + 1 φ t = 1 + 2 ζ φ φ t + o ( φ t ) , φ t + 1 φ t 2 = φ t + o ( φ t ) . (137)
Moreover, the condition on C φ C_\varphi C φ when φ = 1 \varphi=1 φ = 1 ensures that λ m − ζ φ > 0 \lambda_m-\zeta_\varphi>0 λ m − ζ φ > 0 . For all sufficiently large t t t , we also have ∥ I − φ t R ∥ = 1 − λ m φ t \lVert I-\varphi_t\mathcal{R}\rVert=1-\lambda_m\varphi_t ∥ I − φ t R ∥ = 1 − λ m φ t . Consequently,
φ t φ t + 1 ∥ I − φ t R ∥ 2 = 1 − 2 ( λ m − ζ φ ) φ t + o ( φ t ) . \frac{\varphi_t}{\varphi_{t+1}}\lVert I-\varphi_t\mathcal{R}\rVert^2=1-2(\lambda_m-\zeta_\varphi)\varphi_t+o(\varphi_t). φ t + 1 φ t ∥ I − φ t R ∥ 2 = 1 − 2 ( λ m − ζ φ ) φ t + o ( φ t ) .
Therefore, there exists a sufficiently large fixed integer T T T such that, for every t ≥ T t\ge T t ≥ T ,
φ t φ t + 1 ∥ I − φ t R ∥ 2 ≤ 1 − ( λ m − ζ φ ) φ t , φ t 2 φ t + 1 ≤ 2 φ t , 0 < ( λ m − ζ φ ) φ t ≤ 1. \frac{\varphi_t}{\varphi_{t+1}}\lVert I-\varphi_t\mathcal{R}\rVert^2\le1-(\lambda_m-\zeta_\varphi)\varphi_t,\qquad\frac{\varphi_t^2}{\varphi_{t+1}}\le2\varphi_t,\qquad0<(\lambda_m-\zeta_\varphi)\varphi_t\le1. φ t + 1 φ t ∥ I − φ t R ∥ 2 ≤ 1 − ( λ m − ζ φ ) φ t , φ t + 1 φ t 2 ≤ 2 φ t , 0 < ( λ m − ζ φ ) φ t ≤ 1.
Then, we take norms in (D.89) and obtain
∥ V t + 1 ∥ ≤ φ t φ t + 1 ∥ I − φ t R ∥ 2 ∥ V t ∥ + φ t 2 φ t + 1 ∥ Γ m , a s ∥ ≤ { 1 − ( λ m − ζ φ ) φ t } ∥ V t ∥ + 2 ∥ Γ m , a s ∥ φ t ≤ ( 1 − c φ t ) ∥ V t ∥ + C φ t , \begin{aligned}
\lVert\mathcal{V}_{t+1}\rVert\le\frac{\varphi_t}{\varphi_{t+1}}\lVert I-\varphi_t\mathcal{R}\rVert^2\lVert\mathcal{V}_t\rVert+\frac{\varphi_t^2}{\varphi_{t+1}}\lVert\Gamma_{m,\mathrm{as}}\rVert\le\{1-(\lambda_m-\zeta_\varphi)\varphi_t\}\lVert\mathcal{V}_t\rVert+2\lVert\Gamma_{m,\mathrm{as}}\rVert\varphi_t \\
&\le(1-c\varphi_t)\lVert\mathcal{V}_t\rVert+C\varphi_t,
\end{aligned} ∥ V t + 1 ∥ ≤ φ t + 1 φ t ∥ I − φ t R ∥ 2 ∥ V t ∥ + φ t + 1 φ t 2 ∥ Γ m , as ∥ ≤ { 1 − ( λ m − ζ φ ) φ t } ∥ V t ∥ + 2 ∥ Γ m , as ∥ φ t ≤ ( 1 − c φ t ) ∥ V t ∥ + C φ t ,
where c : = λ m − ζ φ > 0 c:=\lambda_m-\zeta_\varphi>0 c := λ m − ζ φ > 0 and C : = 2 ( ∥ Γ m , a s ∥ ∨ 1 ) > 0 C:=2(\lVert\Gamma_{m,\mathrm{as}}\rVert\vee1)>0 C := 2 (∥ Γ m , as ∥ ∨ 1 ) > 0 . Now, we define
M : = max { max 1 ≤ s ≤ T ∥ V s ∥ , C c } . M:=\max\left\{\max_{1\le s\le T}\lVert\mathcal{V}_s\rVert,\frac{C}{c}\right\}. M := max { 1 ≤ s ≤ T max ∥ V s ∥ , c C } .
Suppose that ∥ V t ∥ ≤ M \lVert\mathcal{V}_t\rVert\le M ∥ V t ∥ ≤ M for some t ≥ T t\ge T t ≥ T . Then, we have
∥ V t + 1 ∥ ≤ ( 1 − c φ t ) M + C φ t = M − c φ t ( M − C c ) ≤ M . \lVert\mathcal{V}_{t+1}\rVert\le(1-c\varphi_t)M+C\varphi_t=M-c\varphi_t\left(M-\frac{C}{c}\right)\le M. ∥ V t + 1 ∥ ≤ ( 1 − c φ t ) M + C φ t = M − c φ t ( M − c C ) ≤ M .
Since ∥ V T ∥ ≤ M \lVert\mathcal{V}_T\rVert\le M ∥ V T ∥ ≤ M , we can conclude that ∥ V t ∥ ≤ M \lVert\mathcal{V}_t\rVert\le M ∥ V t ∥ ≤ M for every t ≥ T t\ge T t ≥ T by induction. Together with the finitely many indices preceding T T T , this proves that { V t } t ≥ 1 \{\mathcal{V}_t\}_{t\ge1} { V t } t ≥ 1 is uniformly bounded.
Expanding (D.89) and using (D.90) together with the uniform boundedness of V t \mathcal{V}_t V t , we further obtain
V t + 1 = V t − φ t [ { R − ζ φ I } V t + V t { R − ζ φ I } − Γ m , a s ] + o ( φ t ) . (138) \mathcal{V}_{t+1}=\mathcal{V}_t-\varphi_t\left[\{\mathcal{R}-\zeta_\varphi I\}\mathcal{V}_t+\mathcal{V}_t\{\mathcal{R}-\zeta_\varphi I\}-\Gamma_{m,\mathrm{as}}\right]+o(\varphi_t). \tag*{(138)} V t + 1 = V t − φ t [ { R − ζ φ I } V t + V t { R − ζ φ I } − Γ m , as ] + o ( φ t ) . (138)
Let R φ : = R − ζ φ I \mathcal{R}_\varphi:=\mathcal{R}-\zeta_\varphi I R φ := R − ζ φ I and Q φ : = I ⊗ R φ + R φ ⊗ I Q_\varphi:=I\otimes\mathcal{R}_\varphi+\mathcal{R}_\varphi\otimes I Q φ := I ⊗ R φ + R φ ⊗ I . Since R φ \mathcal{R}_\varphi R φ is symmetric positive definite, Q φ Q_\varphi Q φ is symmetric positive definite with λ min ( Q φ ) = 2 ( λ m − ζ φ ) > 0 \lambda_{\min}(Q_\varphi)=2(\lambda_m-\zeta_\varphi)>0 λ m i n ( Q φ ) = 2 ( λ m − ζ φ ) > 0 . Define
s t : = vec ( V t ) , s ⋆ : = vec ( Σ m , a s l ) = Q φ − 1 vec ( Γ m , a s ) . s_t:=\operatorname{vec}(\mathcal{V}_t),\qquad s^\star:=\operatorname{vec}(\Sigma^l_{m,\mathrm{as}})=Q_\varphi^{-1}\operatorname{vec}(\Gamma_{m,\mathrm{as}}). s t := vec ( V t ) , s ⋆ := vec ( Σ m , as l ) = Q φ − 1 vec ( Γ m , as ) .
By vectorizing the recursion (D.91), we obtain
s t + 1 − s ⋆ = ( I − φ t Q φ ) ( s t − s ⋆ ) + φ t ϵ t , s_{t+1}-s^\star=(I-\varphi_tQ_\varphi)(s_t-s^\star)+\varphi_t\epsilon_t, s t + 1 − s ⋆ = ( I − φ t Q φ ) ( s t − s ⋆ ) + φ t ϵ t ,
where ϵ t → 0 \epsilon_t\to0 ϵ t → 0 . Therefore, for all sufficiently large t t t ,
∥ s t + 1 − s ⋆ ∥ ≤ { 1 − 2 ( λ m − ζ φ ) φ t } ∥ s t − s ⋆ ∥ + φ t ∥ ϵ t ∥ . \lVert s_{t+1}-s^\star\rVert\le\{1-2(\lambda_m-\zeta_\varphi)\varphi_t\}\lVert s_t-s^\star\rVert+\varphi_t\lVert\epsilon_t\rVert. ∥ s t + 1 − s ⋆ ∥ ≤ { 1 − 2 ( λ m − ζ φ ) φ t } ∥ s t − s ⋆ ∥ + φ t ∥ ϵ t ∥ .
Applying Lemma B.3 to the preceding recursion, for every φ ∈ ( 0.5 , 1 ] \varphi\in(0.5,1] φ ∈ ( 0.5 , 1 ] , we obtain
lim sup t → ∞ ∥ s t − s ⋆ ∥ ≤ 1 2 ( λ m − ζ φ ) lim sup t → ∞ ∥ ϵ t ∥ = 0. \limsup_{t\to\infty}\lVert s_t-s^\star\rVert\le\frac{1}{2(\lambda_m-\zeta_\varphi)}\limsup_{t\to\infty}\lVert\epsilon_t\rVert=0. t → ∞ lim sup ∥ s t − s ⋆ ∥ ≤ 2 ( λ m − ζ φ ) 1 t → ∞ lim sup ∥ ϵ t ∥ = 0.
Therefore, s t → s ⋆ s_t\to s^\star s t → s ⋆ , which is equivalent to V t → Σ m , a s l \mathcal{V}_t\to\Sigma^l_{m,\mathrm{as}} V t → Σ m , as l .
Moreover, since s ⋆ = vec ( Σ m , a s l ) s^\star=\operatorname{vec}(\Sigma^l_{m,\mathrm{as}}) s ⋆ = vec ( Σ m , as l ) , the equation Q φ s ⋆ = vec ( Γ m , a s ) Q_\varphi s^\star=\operatorname{vec}(\Gamma_{m,\mathrm{as}}) Q φ s ⋆ = vec ( Γ m , as ) is equivalent to
( R − ζ φ I ) Σ m , a s l + Σ m , a s l ( R − ζ φ I ) = Γ m , a s , (\mathcal{R}-\zeta_\varphi I)\Sigma^l_{m,\mathrm{as}}+\Sigma^l_{m,\mathrm{as}}(\mathcal{R}-\zeta_\varphi I)=\Gamma_{m,\mathrm{as}}, ( R − ζ φ I ) Σ m , as l + Σ m , as l ( R − ζ φ I ) = Γ m , as ,
which is precisely (18) . Since Q φ Q_{\varphi} Q φ is positive definite, the equation Q φ s = vec ( Γ m , a s ) Q_{\varphi}s=\operatorname{vec}(\Gamma_{m,\mathrm{as}}) Q φ s = vec ( Γ m , as ) admits a unique solution, and hence so does (18) . Finally, each V t \mathcal{V}_t V t is positive semidefinite because it is a sum of conditional covariance matrices. Therefore, the convergence of V t \mathcal{V}_t V t implies that Σ m , a s l \Sigma_{m,\mathrm{as}}^{l} Σ m , as l is also positive semidefinite, and condition (B.2) in Lemma B.4 is satisfied.
Verification of (B.3). Let δ : = q g − 2 > 0 \delta:= q_g-2>0 δ := q g − 2 > 0 . The argument before (D.59) establishes a q g q_g q g -moment bound for ∇ F ( x ⋆ ; ξ ) \nabla F(x^{\star};\xi) ∇ F ( x ⋆ ; ξ ) . Since q g = 2 + δ q_g=2+\delta q g = 2 + δ , we have
E [ ∥ ∇ F ( x ⋆ ; ξ ) ∥ 2 + δ ] ≤ C g , 2 < ∞ . (139) \mathbb{E}\left[\left\|\nabla F(x^{\star};\xi)\right\|^{2+\delta}\right]\le C_{g,2}<\infty. \tag*{(139)} E [ ∥ ∇ F ( x ⋆ ; ξ ) ∥ 2 + δ ] ≤ C g , 2 < ∞. (139)
Define A t : = ∑ i = 0 t − 1 φ i 2 + δ ∥ Π t , i + 1 ∥ 2 + δ A_t:=\sum_{i=0}^{t-1}\varphi_i^{2+\delta}\left\|\Pi_{t,i+1}\right\|^{2+\delta} A t := ∑ i = 0 t − 1 φ i 2 + δ ∥ Π t , i + 1 ∥ 2 + δ . Then, for all sufficiently large t t t , we have
A t + 1 ≤ ( 1 − λ m φ t ) 2 + δ A t + φ t 2 + δ = ( 1 − λ m φ t ) 2 + δ A t + φ t 2 + δ / 2 φ t δ / 2 . A_{t+1}\le(1-\lambda_m\varphi_t)^{2+\delta}A_t+\varphi_t^{2+\delta}=(1-\lambda_m\varphi_t)^{2+\delta}A_t+\varphi_t^{2+\delta/2}\varphi_t^{\delta/2}. A t + 1 ≤ ( 1 − λ m φ t ) 2 + δ A t + φ t 2 + δ = ( 1 − λ m φ t ) 2 + δ A t + φ t 2 + δ /2 φ t δ /2 .
By Lemma B.3, for φ ∈ ( 0.5 , 1 ] \varphi\in(0.5,1] φ ∈ ( 0.5 , 1 ] , we obtain
lim sup t → ∞ A t φ t 1 + δ / 2 ≤ { 1 ( 2 + δ ) λ m lim sup t → ∞ φ t δ / 2 , φ ∈ ( 0.5 , 1 ) , C φ ( 2 + δ ) C φ λ m − ( 1 + δ / 2 ) lim sup t → ∞ φ t δ / 2 , φ = 1 , = 0. (140) \limsup_{t\to\infty}\frac{A_t}{\varphi_t^{1+\delta/2}}\le
\begin{cases}
\frac{1}{(2+\delta)\lambda_m}\limsup_{t\to\infty}\varphi_t^{\delta/2}, & \varphi\in(0.5,1),\\[6pt]
\frac{C_{\varphi}}{(2+\delta)C_{\varphi}\lambda_m-(1+\delta/2)}\limsup_{t\to\infty}\varphi_t^{\delta/2}, & \varphi=1,
\end{cases}
=0. \tag*{(140)} t → ∞ lim sup φ t 1 + δ /2 A t ≤ ⎩ ⎨ ⎧ ( 2 + δ ) λ m 1 lim sup t → ∞ φ t δ /2 , ( 2 + δ ) C φ λ m − ( 1 + δ /2 ) C φ lim sup t → ∞ φ t δ /2 , φ ∈ ( 0.5 , 1 ) , φ = 1 , = 0. (140)
Here, the denominator in the second case is positive because
1 + δ 2 = 2 + δ 2 < ( 2 + δ ) C φ λ m , 1+\frac{\delta}{2}=\frac{2+\delta}{2}<(2+\delta)C_{\varphi}\lambda_m, 1 + 2 δ = 2 2 + δ < ( 2 + δ ) C φ λ m ,
where the inequality follows from C φ λ m > 1 / 2 C_{\varphi}\lambda_m>1/2 C φ λ m > 1/2 . Consequently, we obtain A t = o ( φ t 1 + δ / 2 ) A_t=o(\varphi_t^{1+\delta/2}) A t = o ( φ t 1 + δ /2 ) .
For every ϵ > 0 \epsilon>0 ϵ > 0 , by the inequality a 2 1 { a > ϵ } ≤ a 2 + δ / ϵ δ a^2\mathbf{1}_{\{a>\epsilon\}}\le a^{2+\delta}/\epsilon^\delta a 2 1 { a > ϵ } ≤ a 2 + δ / ϵ δ , (D.92) and (D.93), we obtain
∑ i = 0 t − 1 E [ ∥ W t , i ∥ 2 1 { ∥ W t , i ∥ > ϵ } ∣ F i − 1 ] ≤ 1 ϵ δ ∑ i = 0 t − 1 E [ ∥ W t , i ∥ 2 + δ ∣ F i − 1 ] \sum_{i=0}^{t-1}\mathbb{E}\left[\left\|W_{t,i}\right\|^2\mathbf{1}_{\{\left\|W_{t,i}\right\|>\epsilon\}}\mid\mathcal{F}_{i-1}\right]
\le\frac{1}{\epsilon^\delta}\sum_{i=0}^{t-1}\mathbb{E}\left[\left\|W_{t,i}\right\|^{2+\delta}\mid\mathcal{F}_{i-1}\right] i = 0 ∑ t − 1 E [ ∥ W t , i ∥ 2 1 { ∥ W t , i ∥ > ϵ } ∣ F i − 1 ] ≤ ϵ δ 1 i = 0 ∑ t − 1 E [ ∥ W t , i ∥ 2 + δ ∣ F i − 1 ]
≲ 1 ϵ δ φ t 1 + δ / 2 ∑ i = 0 t − 1 φ i 2 + δ ∥ Π t , i + 1 ∥ 2 + δ = A t ϵ δ φ t 1 + δ / 2 → 0. (141) \lesssim\frac{1}{\epsilon^\delta\varphi_t^{1+\delta/2}}\sum_{i=0}^{t-1}\varphi_i^{2+\delta}\left\|\Pi_{t,i+1}\right\|^{2+\delta}
=\frac{A_t}{\epsilon^\delta\varphi_t^{1+\delta/2}}\to0. \tag*{(141)} ≲ ϵ δ φ t 1 + δ /2 1 i = 0 ∑ t − 1 φ i 2 + δ ∥ Π t , i + 1 ∥ 2 + δ = ϵ δ φ t 1 + δ /2 A t → 0. (141)
Therefore, condition (B.3) is satisfied.
Combining (B.1)–(B.3), Lemma B.4 yields
∑ i = 0 t − 1 W t , i = − 1 φ t ∑ i = 0 t − 1 Π t , i + 1 φ i ( I − K ⋆ ) ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) → d N ( 0 , Σ m , a s l ) . (142) \sum_{i=0}^{t-1}W_{t,i}=-\frac{1}{\sqrt{\varphi_t}}\sum_{i=0}^{t-1}\Pi_{t,i+1}\varphi_i(I-K^{\star})(B^{\star})^{-1}\nabla F(x^{\star};\xi_i)\xrightarrow{d}\mathcal{N}(0,\Sigma_{m,\mathrm{as}}^{l}). \tag*{(142)} i = 0 ∑ t − 1 W t , i = − φ t 1 i = 0 ∑ t − 1 Π t , i + 1 φ i ( I − K ⋆ ) ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) d N ( 0 , Σ m , as l ) . (142)
Finally, combining (15) and (D.95) and applying Slutsky’s theorem, we obtain
1 φ t ( x t − x ⋆ ) → d N ( 0 , Σ m , a s l ) , \frac{1}{\sqrt{\varphi_t}}(x_t-x^{\star})\xrightarrow{d}\mathcal{N}(0,\Sigma_{m,\mathrm{as}}^{l}), φ t 1 ( x t − x ⋆ ) d N ( 0 , Σ m , as l ) ,
And we complete the proof.
Proof of Proposition 4.6
Recall that A φ ⋆ = ( I − K ⋆ ) − ζ φ I A_{\varphi}^{\star}=(I-K^{\star})-\zeta_{\varphi}I A φ ⋆ = ( I − K ⋆ ) − ζ φ I . By Lemma 4.1, K ⋆ K^{\star} K ⋆ is symmetric and I − K ⋆ ≻ 0 I-K^{\star}\succ0 I − K ⋆ ≻ 0 . If φ < 1 \varphi<1 φ < 1 , then ζ φ = 0 \zeta_{\varphi}=0 ζ φ = 0 . If φ = 1 \varphi=1 φ = 1 , the condition on C φ C_{\varphi} C φ in Theorem 4.3 gives ζ φ = 1 / ( 2 C φ ) < λ min ( I − K ⋆ ) \zeta_{\varphi}=1/(2C_{\varphi})<\lambda_{\min}(I-K^{\star}) ζ φ = 1/ ( 2 C φ ) < λ m i n ( I − K ⋆ ) . Therefore, A φ ⋆ ≻ 0 A_{\varphi}^{\star}\succ0 A φ ⋆ ≻ 0 under both stepsize regimes.
Proof of part (a). Since K ⋆ = E [ K ~ ⋆ ] K^{\star}=\mathbb{E}[\widetilde{K}^{\star}] K ⋆ = E [ K ⋆ ] , we have E [ K ~ ⋆ − K ⋆ ] = 0 \mathbb{E}[\widetilde{K}^{\star}-K^{\star}]=0 E [ K ⋆ − K ⋆ ] = 0 . Also, we have
Γ g , a s = Γ m , a s − ( I − K ⋆ ) Ω E [ ( K ~ ⋆ − K ⋆ ) T ] − E [ K ~ ⋆ − K ⋆ ] Ω ( I − K ⋆ ) T + E [ ( K ~ ⋆ − K ⋆ ) Ω ( K ~ ⋆ − K ⋆ ) T ] = Γ m , a s + E [ ( K ~ ⋆ − K ⋆ ) Ω ( K ~ ⋆ − K ⋆ ) T ] . \begin{aligned}
\Gamma_{g,\mathrm{as}}=\Gamma_{m,\mathrm{as}}-(I-K^{\star})\Omega\mathbb{E}\left[(\widetilde{K}^{\star}-K^{\star})^{\mathsf{T}}\right]
-\mathbb{E}\left[\widetilde{K}^{\star}-K^{\star}\right]\Omega(I-K^{\star})^{\mathsf{T}}
+\mathbb{E}\left[(\widetilde{K}^{\star}-K^{\star})\Omega(\widetilde{K}^{\star}-K^{\star})^{\mathsf{T}}\right] \\
&=\Gamma_{m,\mathrm{as}}+\mathbb{E}\left[(\widetilde{K}^{\star}-K^{\star})\Omega(\widetilde{K}^{\star}-K^{\star})^{\mathsf{T}}\right].
\end{aligned} Γ g , as = Γ m , as − ( I − K ⋆ ) Ω E [ ( K ⋆ − K ⋆ ) T ] − E [ K ⋆ − K ⋆ ] Ω ( I − K ⋆ ) T + E [ ( K ⋆ − K ⋆ ) Ω ( K ⋆ − K ⋆ ) T ] = Γ m , as + E [ ( K ⋆ − K ⋆ ) Ω ( K ⋆ − K ⋆ ) T ] .
Since Ω ⪰ 0 \Omega\succeq0 Ω ⪰ 0 , this identity implies Γ m , a s ⪯ Γ g , a s \Gamma_{m,\mathrm{as}} \preceq\Gamma_{g,\mathrm{as}} Γ m , as ⪯ Γ g , as .
Subtracting (18) from the defining Lyapunov equation for Σ g , a s l \Sigma^{l}_{g,\mathrm{as}} Σ g , as l , we obtain
( R − ζ φ I ) ( Σ g , a s l − Σ m , a s l ) + ( Σ g , a s l − Σ m , a s l ) ( R − ζ φ I ) = E [ ( K ~ ⋆ − K ⋆ ) Ω ( K ~ ⋆ − K ⋆ ) T ] ⪰ 0. (\mathcal{R}-\zeta_{\varphi}I)(\Sigma^{l}_{g,\mathrm{as}}-\Sigma^{l}_{m,\mathrm{as}})+(\Sigma^{l}_{g,\mathrm{as}}-\Sigma^{l}_{m,\mathrm{as}})(\mathcal{R}-\zeta_{\varphi}I)=\mathbb{E}\left[(\widetilde{K}^{\star}-K^{\star})\Omega(\widetilde{K}^{\star}-K^{\star})^{\mathsf{T}}\right]\succeq0. ( R − ζ φ I ) ( Σ g , as l − Σ m , as l ) + ( Σ g , as l − Σ m , as l ) ( R − ζ φ I ) = E [ ( K ⋆ − K ⋆ ) Ω ( K ⋆ − K ⋆ ) T ] ⪰ 0.
Since R − ζ φ I ≻ 0 \mathcal{R}-\zeta_{\varphi}I\succ0 R − ζ φ I ≻ 0 , the unique solution of this Lyapunov equation admits the representation
Σ g , a s l − Σ m , a s l = ∫ 0 ∞ exp { − ( R − ζ φ I ) s } E [ ( K ~ ⋆ − K ⋆ ) Ω ( K ~ ⋆ − K ⋆ ) T ] exp { − ( R − ζ φ I ) s } d s ⪰ 0 , \Sigma^{l}_{g,\mathrm{as}}-\Sigma^{l}_{m,\mathrm{as}}=\int_{0}^{\infty}\exp\{-(\mathcal{R}-\zeta_{\varphi}I)s\}\mathbb{E}\left[(\widetilde{K}^{\star}-K^{\star})\Omega(\widetilde{K}^{\star}-K^{\star})^{\mathsf{T}}\right]\exp\{-(\mathcal{R}-\zeta_{\varphi}I)s\}\,\mathrm{d}s\succeq0, Σ g , as l − Σ m , as l = ∫ 0 ∞ exp { − ( R − ζ φ I ) s } E [ ( K ⋆ − K ⋆ ) Ω ( K ⋆ − K ⋆ ) T ] exp { − ( R − ζ φ I ) s } d s ⪰ 0 ,
which completes the proof of part (a).
Proof of part (b). Suppose that the Newton system is solved exactly. Then, we have K ⋆ = K ~ ⋆ = 0 K^{\star}=\widetilde{K}^{\star}=0 K ⋆ = K ⋆ = 0 and Γ m , a s = Γ g , a s = Ω \Gamma_{m,\mathrm{as}}=\Gamma_{g,\mathrm{as}}=\Omega Γ m , as = Γ g , as = Ω . In this regime, the Lyapunov equations in (18) and (10) reduce to
2 ( 1 − ζ φ ) Σ m , e l = 2 ( 1 − ζ φ ) Σ g , e l = Ω . 2(1-\zeta_{\varphi})\Sigma^{l}_{m,e}=2(1-\zeta_{\varphi})\Sigma^{l}_{g,e}=\Omega. 2 ( 1 − ζ φ ) Σ m , e l = 2 ( 1 − ζ φ ) Σ g , e l = Ω.
If φ < 1 \varphi<1 φ < 1 , then ζ φ = 0 \zeta_{\varphi}=0 ζ φ = 0 , which gives Σ m , e l = Σ g , e l = 0.5 Ω \Sigma^{l}_{m,e}=\Sigma^{l}_{g,e}=0.5\Omega Σ m , e l = Σ g , e l = 0.5Ω . If φ = 1 \varphi=1 φ = 1 , then ζ φ = 1 / ( 2 C φ ) \zeta_{\varphi}=1/(2C_{\varphi}) ζ φ = 1/ ( 2 C φ ) and hence
Σ m , e l = Σ g , e l = 1 2 { 1 − 1 / ( 2 C φ ) } Ω = C φ 2 C φ − 1 Ω . \Sigma^{l}_{m,e}=\Sigma^{l}_{g,e}=\frac{1}{2\{1-1/(2C_{\varphi})\}}\Omega=\frac{C_{\varphi}}{2C_{\varphi}-1}\Omega. Σ m , e l = Σ g , e l = 2 { 1 − 1/ ( 2 C φ )} 1 Ω = 2 C φ − 1 C φ Ω.
This completes the proof of part (b).
Proof of part (c). When φ = C φ = 1 \varphi=C_{\varphi}=1 φ = C φ = 1 , part (b) gives Σ m , e l = Σ g , e l = Ω \Sigma^{l}_{m,e}=\Sigma^{l}_{g,e}=\Omega Σ m , e l = Σ g , e l = Ω , and t φ t → 1 t\varphi_t\to1 t φ t → 1 . Therefore, by Theorems 4.5 and 2.2 and Slutsky’s theorem, exact online Newton, with or without gradient averaging, satisfies
t ( x t − x ⋆ ) ⟶ d N ( 0 , Ω ) . \sqrt{t}(x_t-x^{\star})\mathrel{\mathop{\longrightarrow}^{\mathrm{d}}}\mathcal{N}(0,\Omega). t ( x t − x ⋆ ) ⟶ d N ( 0 , Ω ) .
We next establish the covariance ordering for sketched Newton with the same stepsizes. In this case, ζ φ = 1 / 2 \zeta_{\varphi}=1/2 ζ φ = 1/2 , and Lemma 4.1 gives
λ m = λ min ( I − K ⋆ ) ≥ 3 4 > 1 2 , 1 2 I − K ⋆ ≻ 0. \lambda_m=\lambda_{\min}(I-K^{\star})\ge\frac{3}{4}>\frac{1}{2},\qquad\frac{1}{2}I-K^{\star}\succ0. λ m = λ m i n ( I − K ⋆ ) ≥ 4 3 > 2 1 , 2 1 I − K ⋆ ≻ 0.
Moreover, by the definition of Γ m , a s \Gamma_{m,\mathrm{as}} Γ m , as , we have
Γ m , a s − { ( 1 2 I − K ⋆ ) Ω + Ω ( 1 2 I − ( K ⋆ ) T ) } = ( I − K ⋆ ) Ω ( I − K ⋆ ) T − { ( 1 2 I − K ⋆ ) Ω + Ω ( 1 2 I − ( K ⋆ ) T ) } = K ⋆ Ω ( K ⋆ ) T . \Gamma_{m,\mathrm{as}}-\left\{\left(\frac{1}{2}I-K^{\star}\right)\Omega+\Omega\left(\frac{1}{2}I-(K^{\star})^{\mathsf{T}}\right)\right\}
=(I-K^{\star})\Omega(I-K^{\star})^{\mathsf{T}}-\left\{\left(\frac{1}{2}I-K^{\star}\right)\Omega+\Omega\left(\frac{1}{2}I-(K^{\star})^{\mathsf{T}}\right)\right\}=K^{\star}\Omega(K^{\star})^{\mathsf{T}}. Γ m , as − { ( 2 1 I − K ⋆ ) Ω + Ω ( 2 1 I − ( K ⋆ ) T ) } = ( I − K ⋆ ) Ω ( I − K ⋆ ) T − { ( 2 1 I − K ⋆ ) Ω + Ω ( 2 1 I − ( K ⋆ ) T ) } = K ⋆ Ω ( K ⋆ ) T .
Subtracting the expression inside the braces from (18) gives
( 1 2 I − K ⋆ ) ( Σ m , a s l − Ω ) + ( Σ m , a s l − Ω ) ( 1 2 I − ( K ⋆ ) T ) = K ⋆ Ω ( K ⋆ ) T ⪰ 0. \left(\frac{1}{2}I-K^{\star}\right)(\Sigma^{l}_{m,\mathrm{as}}-\Omega)+(\Sigma^{l}_{m,\mathrm{as}}-\Omega)\left(\frac{1}{2}I-(K^{\star})^{\mathsf{T}}\right)=K^{\star}\Omega(K^{\star})^{\mathsf{T}}\succeq0. ( 2 1 I − K ⋆ ) ( Σ m , as l − Ω ) + ( Σ m , as l − Ω ) ( 2 1 I − ( K ⋆ ) T ) = K ⋆ Ω ( K ⋆ ) T ⪰ 0.
Since 1 2 I − K ⋆ ≻ 0 \frac{1}{2}I-K^{\star}\succ0 2 1 I − K ⋆ ≻ 0 , it follows that
Σ m , a s l − Ω = ∫ 0 ∞ exp { − ( 1 2 I − K ⋆ ) s } K ⋆ Ω ( K ⋆ ) T exp { − ( 1 2 I − ( K ⋆ ) T ) s } d s ⪰ 0. \Sigma^{l}_{m,\mathrm{as}}-\Omega=\int_{0}^{\infty}\exp\left\{-\left(\frac{1}{2}I-K^{\star}\right)s\right\}K^{\star}\Omega(K^{\star})^{\mathsf{T}}\exp\left\{-\left(\frac{1}{2}I-(K^{\star})^{\mathsf{T}}\right)s\right\}\,\mathrm{d}s\succeq0. Σ m , as l − Ω = ∫ 0 ∞ exp { − ( 2 1 I − K ⋆ ) s } K ⋆ Ω ( K ⋆ ) T exp { − ( 2 1 I − ( K ⋆ ) T ) s } d s ⪰ 0.
Therefore, Ω ⪯ Σ m , a s l \Omega\preceq\Sigma^{l}_{m,\mathrm{as}} Ω ⪯ Σ m , as l . Combining this inequality with parts (a) and (b) gives
Ω = Σ g , e l = Σ m , e l ⪯ Σ m , a s l ⪯ Σ g , a s l , \Omega=\Sigma^{l}_{g,e}=\Sigma^{l}_{m,e}\preceq\Sigma^{l}_{m,\mathrm{as}}\preceq\Sigma^{l}_{g,\mathrm{as}}, Ω = Σ g , e l = Σ m , e l ⪯ Σ m , as l ⪯ Σ g , as l ,
which completes the proof of part (c).
Proof of Theorem 4.7
Since ∇ f ( x ⋆ ) = 0 \nabla f(x^\star)=0 ∇ f ( x ⋆ ) = 0 , we have
E [ ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ] = ( B ⋆ ) − 1 ∇ f ( x ⋆ ) = 0 , \mathbb{E}\left[(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right]=(B^\star)^{-1}\nabla f(x^\star)=0, E [ ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ] = ( B ⋆ ) − 1 ∇ f ( x ⋆ ) = 0 ,
E [ ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∇ F ( x ⋆ ; ξ i ) ⊤ ( B ⋆ ) − 1 ] = Ω . \mathbb{E}\left[(B^\star)^{-1}\nabla F(x^\star;\xi_i)\nabla F(x^\star;\xi_i)^\top(B^\star)^{-1}\right]=\Omega. E [ ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∇ F ( x ⋆ ; ξ i ) ⊤ ( B ⋆ ) − 1 ] = Ω.
Moreover, by (D.92) and Jensen’s inequality, we further obtain
E [ ∥ ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) ∥ 2 ] ≤ 1 γ H 2 E [ ∥ ∇ F ( x ⋆ ; ξ i ) ∥ 2 ] ≤ 1 γ H 2 { E [ ∥ ∇ F ( x ⋆ ; ξ i ) ∥ 2 + δ ] } 2 / ( 2 + δ ) < ∞ . \begin{aligned}
\mathbb{E}\left[\left\|(B^\star)^{-1}\nabla F(x^\star;\xi_i)\right\|^2\right]\leq\frac{1}{\gamma_H^2}\mathbb{E}\left[\|\nabla F(x^\star;\xi_i)\|^2\right] \\
&\leq\frac{1}{\gamma_H^2}\left\{\mathbb{E}\left[\|\nabla F(x^\star;\xi_i)\|^{2+\delta}\right]\right\}^{2/(2+\delta)}<\infty.
\end{aligned} E [ ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) 2 ] ≤ γ H 2 1 E [ ∥∇ F ( x ⋆ ; ξ i ) ∥ 2 ] ≤ γ H 2 1 { E [ ∥∇ F ( x ⋆ ; ξ i ) ∥ 2 + δ ] } 2/ ( 2 + δ ) < ∞.
Therefore, the central limit theorem for i.i.d. random variables gives us
− 1 t ∑ i = 0 t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) → d N ( 0 , Ω ) . -\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)\xrightarrow{d}\mathcal{N}(0,\Omega). − t 1 i = 0 ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) d N ( 0 , Ω ) .
Finally, by (16) , we have
t ( x ˉ t − x ⋆ ) = − 1 t ∑ i = 0 t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) . \sqrt{t}(\bar{x}_t-x^\star)=-\frac{1}{\sqrt{t}}\sum_{i=0}^{t-1}(B^\star)^{-1}\nabla F(x^\star;\xi_i)+o_p(1). t ( x ˉ t − x ⋆ ) = − t 1 i = 0 ∑ t − 1 ( B ⋆ ) − 1 ∇ F ( x ⋆ ; ξ i ) + o p ( 1 ) .
Combining the preceding two equations and applying Slutsky’s theorem, we conclude that
t ( x ˉ t − x ⋆ ) → d N ( 0 , Σ m , a s ) , \sqrt{t}(\bar{x}_t-x^\star)\xrightarrow{d}\mathcal{N}(0,\Sigma_{m,\mathrm{as}}), t ( x ˉ t − x ⋆ ) d N ( 0 , Σ m , as ) ,
where Σ m , a s = Ω \Sigma_{m,\mathrm{as}}=\Omega Σ m , as = Ω . This proves (19) and completes the proof.
Experimental Details and Additional Results
We provide the experimental details and additional results for § 5 . The comparisons below complement the main paper with higher-dimensional problems, full-matrix covariance errors, method-specific normal QQ plots, and estimation-error trajectories.
Detailed experimental setup
Regression models. For both models, we set x ⋆ = ( 0 , 1 / ( d − 1 ) , … , 1 ) ⊤ x^\star=(0,1/(d-1),\ldots,1)^\top x ⋆ = ( 0 , 1/ ( d − 1 ) , … , 1 ) ⊤ and generate a ∼ N ( 0 , Σ a ) a\sim\mathcal{N}(0,\Sigma_a) a ∼ N ( 0 , Σ a ) . We consider d ∈ { 20 , 40 } d\in\{20,40\} d ∈ { 20 , 40 } and three covariate covariances: Σ a = I \Sigma_a=I Σ a = I , the equicorrelation matrix ( 1 − ρ ) I + ρ 11 ⊤ (1-\rho)I+\rho\mathbf{1}\mathbf{1}^\top ( 1 − ρ ) I + ρ 1 1 ⊤ , and the Toeplitz matrix with entries ( Σ a ) j k = ρ ∣ j − k ∣ (\Sigma_a)_{jk}=\rho^{|j-k|} ( Σ a ) j k = ρ ∣ j − k ∣ , where ρ = 0.4 \rho=0.4 ρ = 0.4 . For linear regression, y = a ⊤ x ⋆ + ϵ y=a^\top x^\star+\epsilon y = a ⊤ x ⋆ + ϵ with independent ϵ ∼ N ( 0 , 1 ) \epsilon\sim\mathcal{N}(0,1) ϵ ∼ N ( 0 , 1 ) , and
F ( x ; ( a , y ) ) = 1 2 ( a ⊤ x − y ) 2 . F(x;(a,y))=\frac{1}{2}(a^\top x-y)^2. F ( x ; ( a , y )) = 2 1 ( a ⊤ x − y ) 2 .
For logistic regression, y ∣ a ∼ Bernoulli { σ ( a ⊤ x ⋆ ) } y\mid a\sim\operatorname{Bernoulli}\{\sigma(a^\top x^\star)\} y ∣ a ∼ Bernoulli { σ ( a ⊤ x ⋆ )} , where σ ( z ) = ( 1 + e − z ) − 1 \sigma(z)=(1+e^{-z})^{-1} σ ( z ) = ( 1 + e − z ) − 1 , and
F ( x ; ( a , y ) ) = log { 1 + exp ( a ⊤ x ) } − y a ⊤ x + c 2 ∥ x − x ⋆ ∥ 2 , c = 2. F(x;(a,y))=\log\{1+\exp(a^\top x)\}-ya^\top x+\frac{c}{2}\|x-x^\star\|^2,\qquad c=2. F ( x ; ( a , y )) = log { 1 + exp ( a ⊤ x )} − y a ⊤ x + 2 c ∥ x − x ⋆ ∥ 2 , c = 2.
The penalty is centered at x ⋆ x^\star x ⋆ so that the population minimizer remains unchanged. We use this penalty only in simulations, since x ⋆ x^\star x ⋆ is unknown in practice. In both models, observations are drawn from a fixed distribution, and the gradient and Hessian are evaluated at the current iterate using the same observation. The Gaussian covariates are unbounded, and the linear sample Hessian is rank one. Thus, the uniform sampled Hessian bounds imposed in the existing sketched Newton literature do not hold in these experiments, while the population-level conditions in Assumption 3.1 do.
Methods and sketching parameters. We compare online Newton with and without gradient averaging, using either accelerated sketching or exact solves, and include SGD. All methods start from x 0 = 1 x_0=1 x 0 = 1 . Newton methods use the Hessian average B t = { I + ∑ i = 0 t − 1 H i } / ( t + 1 ) B_t=\{I+\sum_{i=0}^{t-1}H_i\}/(t+1) B t = { I + ∑ i = 0 t − 1 H i } / ( t + 1 ) , which is fixed
before ξ t \xi_t ξ t is drawn. The term I / ( t + 1 ) I/(t+1) I / ( t + 1 ) keeps B t B_t B t nonsingular and vanishes as t t t increases. We use Kaczmarz sketches S = e j S=e_j S = e j with j j j uniform on { 1 , … , d } \{1,\ldots,d\} { 1 , … , d } , or Gaussian sketches S ∼ N ( 0 , I ) S\sim\mathcal{N}(0,I) S ∼ N ( 0 , I ) , with τ ∈ { 5 , 10 } \tau\in\{5,10\} τ ∈ { 5 , 10 } . Each inner solve starts from z t , 0 = v t , 0 = 0 z_{t,0}=v_{t,0}=0 z t , 0 = v t , 0 = 0 . We compute the acceleration parameters from the current B t B_t B t at t = 0 , 500 , 1000 , … t=0,500,1000,\ldots t = 0 , 500 , 1000 , … and keep them fixed between updates. The expectations in (6) are computed exactly by averaging over all d d d coordinates for Kaczmarz sketches. For Gaussian sketches, we use the same M a c c = 16,384 M_{\mathrm{acc}}=16{,}384 M acc = 16 , 384 Gaussian vectors at each update, generated independently of the training data and inner sketches.
Stepsizes. We set φ = 0.8 \varphi=0.8 φ = 0.8 for all methods and use η t = ( t + 1 ) − 0.6 \eta_t=(t+1)^{-0.6} η t = ( t + 1 ) − 0.6 with m 0 = g 0 m_0=g_0 m 0 = g 0 for gradient averaging. For logistic regression, the Newton stepsize is φ t = ( t + 1 ) − 0.8 \varphi_t=(t+1)^{-0.8} φ t = ( t + 1 ) − 0.8 . For linear regression, all four Newton methods use the same initial stepsizes, obtained by interpolating log φ t \log\varphi_t log φ t linearly in log ( t + 1 ) \log(t+1) log ( t + 1 ) through
( t , φ t ) ∈ { ( 0 , 0.02 ) , ( 39 , 0.0005 ) , ( 99 , 0.01 ) , ( 199 , 0.014427 ) , ( 499 , 0.020794 ) } , (t,\varphi_t)\in\{(0,0.02),(39,0.0005),(99,0.01),(199,0.014427),(499,0.020794)\}, ( t , φ t ) ∈ {( 0 , 0.02 ) , ( 39 , 0.0005 ) , ( 99 , 0.01 ) , ( 199 , 0.014427 ) , ( 499 , 0.020794 )} ,
where the final two stepsizes are rounded. For t ≥ 500 t\ge500 t ≥ 500 , we use φ t = 3 / ( t + 1 ) 0.8 \varphi_t=3/(t+1)^{0.8} φ t = 3/ ( t + 1 ) 0.8 . Linear SGD uses min { ( t + 1 ) − 0.8 , 0.02 } \min\{(t+1)^{-0.8},0.02\} min {( t + 1 ) − 0.8 , 0.02 } for the first 500 updates and ( t + 1 ) − 0.8 (t+1)^{-0.8} ( t + 1 ) − 0.8 thereafter, while logistic SGD uses 0.4 / ( t + 1 ) 0.8 0.4/(t+1)^{0.8} 0.4/ ( t + 1 ) 0.8 . These choices are fixed across runs. Within each problem setting, the Newton methods use the same stepsizes with and without gradient averaging, for both sketching distributions and both values of τ \tau τ .
Runs and paired comparisons. The two models, two dimensions, and three covariate covariance matrices give 12 problem settings. For each setting, we compare eight accelerated-sketching configurations, two exact-Newton configurations, and SGD, giving 132 configurations in total. For each configuration, we conduct R = 200 R=200 R = 200 independent runs of T = 10 5 T=10^5 T = 1 0 5 iterations. Within each setting, all methods use the same observations in each run. Sketched Newton with and without gradient averaging also uses the same sketches for the same sketching distribution and τ \tau τ . The observations and sketches are generated independently.
Covariance estimation and statistical comparisons
We use the averaged iterate x ˉ t = t − 1 ∑ i = 0 t − 1 x i \bar{x}_t=t^{-1}\sum_{i=0}^{t-1}x_i x ˉ t = t − 1 ∑ i = 0 t − 1 x i . At iteration t t t , we compute centered sample covariances across the R R R runs of
Z r , t l = x t ( r ) − x ∗ φ t , Z r , t a = t ( x ˉ t ( r ) − x ∗ ) , Z^l_{r,t}=\frac{x_t^{(r)}-x^*}{\sqrt{\varphi_t}},\qquad Z^a_{r,t}=\sqrt{t}\left(\bar{x}_t^{(r)}-x^*\right), Z r , t l = φ t x t ( r ) − x ∗ , Z r , t a = t ( x ˉ t ( r ) − x ∗ ) ,
with denominator R − 1 R-1 R − 1 . For the last iterate, normalization uses C / ( t + 1 ) 0.8 C/(t+1)^{0.8} C / ( t + 1 ) 0.8 with the corresponding constant C C C for each method, rather than the modified initial stepsizes. We measure directional variance along 1 1 1 . To compare estimation accuracy, we also report the empirical bias norm, MSE, and mean population objective gap using unnormalized errors. The bias norm is the norm of the mean error across runs.
For the last iterate, we compare each empirical covariance with the corresponding theoretical covariance, which generally differs from Ω \Omega Ω . For the averaged iterate, online sketched Newton with gradient averaging and exact online Newton have limiting covariance Ω \Omega Ω , whereas vanilla sketched Newton has an additional covariance term. We therefore compare averaged-iterate covariances both with their respective theoretical values and with Ω \Omega Ω . We use covariance traces to compare total variance and Frobenius errors to measure differences between covariance matrices.
We construct pointwise 95% confidence intervals using 1,000 bootstrap resamples of the independent runs. Within each problem setting, we use the same resampled run indices for all methods and iterations. These intervals measure uncertainty across runs, with the theoretical covariances held fixed. For the ratio tables, we compute each ratio within a matched setting and then summarize these point estimates across settings. For the QQ plots, we use theoretical rather than empirical standard deviations and do not subtract the empirical means.
We compute population moments analytically for linear regression and by numerical Gaussian integration for logistic regression. The acceleration parameters used in the theoretical covariances are computed at B ∗ B^* B ∗ using the same procedure as in the experiments. We estimate the sketch-dependent covariance terms from M K = 131,072 M_K=131{,}072 M K = 131 , 072 independent realizations of the inner solver at B ∗ B^* B ∗ , using sketches generated independently of the training and parameter-estimation samples. For these Monte
Carlo estimates, we require both the relative Frobenius standard error and the relative standard error of the variance along 1 to be below 2%. The theoretical covariances are computed using the known data distributions and the true parameter x ⋆ x^{\star} x ⋆ .
Additional covariance comparisons
From Figure 2 , we observe smaller covariance traces with gradient averaging for both the last and averaged iterates in all 24 matched settings with d = 40 d = 40 d = 40 . As in the d = 20 d = 20 d = 20 results in Figure 1 (a–Figure 1 d), this reduction holds for both models, both sketching distributions, and all three covariate covariance matrices.
Figure 2. Covariance comparisons for d = 40 d = 40 d = 40 , T = 10 5 T = 10^{5} T = 1 0 5 , and R = 200 R = 200 R = 200 . Horizontal and vertical axes show traces without and with gradient averaging, respectively; averaged-iterate traces are divided by tr ( Ω ) \operatorname{tr}(\Omega) tr ( Ω ) . Each panel contains 12 settings. Colors identify covariate covariances, circles/squares distinguish τ = 5 / 10 \tau= 5/10 τ = 5/10 , and filled/open markers denote Kaczmarz/Gaussian sketches. Diagonals indicate equality; dotted horizontal lines indicate one. Open markers are enlarged to distinguish overlaps without changing their positions.
Table 2 reports additional comparisons of directional variance, bias, and objective gap, together with the ranges of the ratios across settings. We observe smaller values with gradient averaging for both the last and averaged iterates in all 48 sketching settings. Therefore, gradient averaging improves estimation accuracy as well as reducing covariance. In contrast, the covariance-trace ratios for exact online Newton remain close to one, consistent with Proposition 4.6 .
We also compare the averaged-iterate covariance matrices with Ω \Omega Ω using the relative Frobenius error E u = ∥ Σ ^ u , a s − Ω ∥ F / ∥ Ω ∥ F E_u = \lVert\widehat{\Sigma}_{u,\mathrm{as}}-\Omega\rVert_F/\lVert\Omega\rVert_F E u = ∥ Σ u , as − Ω ∥ F / ∥ Ω ∥ F for u ∈ { g , m } u \in\{g,m\} u ∈ { g , m } , where ∥ ⋅ ∥ F \lVert\cdot\rVert_F ∥ ⋅ ∥ F denotes the Frobenius norm. From Table 3 , we observe smaller errors with gradient averaging in all 48 matched settings. Thus, gradient averaging brings the covariance matrices closer to Ω \Omega Ω , not only their traces. Increasing τ \tau τ further reduces these errors, although some settings still have large errors at T = 10 5 T = 10^{5} T = 1 0 5 .
Table 3. Relative Frobenius errors of averaged-iterate covariances against Ω \Omega Ω at T = 10 5 T = 10^5 T = 1 0 5 . Entries give median [minimum, maximum] over 12 dimension–covariate-covariance–sketch combinations. The final column aggregates settingwise error ratios, not ratios of the two median errors.
Table 4 reports the final covariance-trace ratios and 95% bootstrap intervals for Figure 1 (e,Figure 1 f). We observe that gradient averaging brings the covariance traces closer to those of exact online Newton. The bootstrap intervals for sketched Newton with and without gradient averaging do not overlap.
Table 4. **Averaged-iterate trace divided by tr ( Ω ) \operatorname{tr}(\Omega) tr ( Ω ) , with pointwise 95% bootstrap intervals, for d = 20 d = 20 d = 20 , identity covariate covariance, Gaussian sketches, τ = 10 \tau= 10 τ = 10 , T = 10 5 T = 10^5 T = 1 0 5 , and R = 200 R = 200 R = 200 . Exact online Newton uses g t g_t g t .
Normal QQ plots with method-specific covariances
Figure 3 shows normal QQ plots using each method’s theoretical covariance. We consider d = 20 d = 20 d = 20 , identity covariate covariance, Gaussian sketches, and τ = 10 \tau= 10 τ = 10 , as in Figure 1 (e–Figure 1 h). For each method and iterate, we project Z r , T q Z^{q}_{r,T} Z r , T q onto 1 \mathbf{1} 1 and divide by the theoretical standard deviation 1 T Σ u , s 1 \sqrt{\mathbf{1}^{\mathsf{T}}\Sigma_{u,s}\mathbf{1}} 1 T Σ u , s 1 . Unlike the averaged-iterate QQ plots in the main paper, vanilla sketched Newton is normalized by its own larger covariance rather than by Ω \Omega Ω .
Figure 3. Method-specific normal QQ plots for d = 20 d = 20 d = 20 , identity covariate covariance, Gaussian sketches, τ = 10 \tau= 10 τ = 10 , T = 10 5 T = 10^{5} T = 1 0 5 , and R = 200 R = 200 R = 200 . Horizontal axes show standard normal quantiles; vertical axes show quantiles of 1 ⊤ Z r , T q / 1 ⊤ Σ u , s 1 q 1 \mathbf{1}^{\top}Z_{r,T}^{q}/\sqrt{\mathbf{1}^{\top}\Sigma_{u,s1}^{q}\mathbf{1}} 1 ⊤ Z r , T q / 1 ⊤ Σ u , s 1 q 1 , using each method’s own theoretical covariance. Sketching uses g t / m t g_t/m_t g t / m t without/with gradient averaging; exact Newton uses g t g_t g t . Dashed lines indicate equality. Empirical means are retained.
From panels (a,c,d), we observe that the empirical quantiles closely follow the standard normal reference over most quantiles. Panel (b) shows a mean shift for the linear averaged iterate, which remains visible because we do not subtract the empirical means. These plots examine the normal approximation for each method using its corresponding limiting covariance.
Estimation error and computational cost
Figure 4 and Figure 5 show bias, MSE, and population objective gap over iterations for the same representative problems. We include SGD and exact online Newton as first- and second-order baselines, respectively. Sketched Newton with and without gradient averaging uses the same observations, sketches, and outer stepsizes. Across both figures, the methods exhibit comparable error decay rates, consistent with our asymptotic results. Among the baselines, SGD performs better in the simpler, well-conditioned linear regression setting of Figure 4 , where second-order information provides limited additional benefits, whereas exact Newton performs comparably to SGD in the more complex logistic regression setting of Figure 5 and even achieves lower averaged-iterate bias, benefiting from its ability to adapt to varying curvature. We also observe smaller MSE and objective gaps with gradient averaging toward the end of the runs. Table 2 also shows smaller empirical bias in every matched sketching setting. Therefore, although the moving-average gradient is generally conditionally biased, the resulting parameter estimates have smaller bias in these experiments.
Figure 4. Estimation errors for linear regression with d = 20 d = 20 d = 20 , identity covariate covariance, Gaussian sketches, τ = 10 \tau= 10 τ = 10 , and R = 200 R = 200 R = 200 . Columns show empirical bias norm, MSE, and population objective gap; rows show last and averaged iterates. Sketching uses g t / m t g_t/m_t g t / m t without/with gradient averaging; exact Newton uses g t g_t g t , and SGD is the first-order baseline. All quantities use unnormalized errors.
Figure 5. Estimation errors for logistic regression with d = 20 d = 20 d = 20 , identity covariate covariance, Gaussian sketches, τ = 10 \tau= 10 τ = 10 , and R = 200 R = 200 R = 200 . Methods, quantities, and layout follow Figure 4 .
Gradient averaging requires no additional derivative evaluations and only O ( d ) O(d) O ( d ) extra operations per iteration, preserving the O ( d 2 ) O(d^2) O ( d 2 ) cost for a fixed sketching budget. Across the 48 matched sketching settings, the per-iteration time ratio with versus without gradient averaging has median 1.038 and range [0.877, 1.564]. Thus, the improvements in covariance and estimation accuracy require neither additional observations nor more sketching steps, with a median running-time increase of only 3.8%.
Statistical inference and computational cost
We study statistical inference for the coordinate average d − 1 ∑ j = 1 d x j ⋆ d^{-1}\sum_{j=1}^{d}x_j^\star d − 1 ∑ j = 1 d x j ⋆ to complement the covariance and estimation error comparisons in § 5 . The experiments use the same linear and logistic regression settings at d = 40 d = 40 d = 40 , with all three design covariance structures. For the sketched Newton methods, we use Kaczmarz sketching with τ = 10 \tau= 10 τ = 10 steps at each iteration. After 100,000 updates, we construct nominal 95% confidence intervals with a burn-in of 5,000 updates. The comparison includes both the last and averaged iterates of the exact and sketched Newton methods, as well as averaged SGD (ASGD). Table 5 reports empirical coverage, mean interval length, and mean absolute error (MAE) over 200 runs per setting. It also reports mean runtime and GFLOPs per run, including both iterate updates and confidence interval construction.
Table 5. Inference for the coordinate average d − 1 ∑ j = 1 d x j ∗ d^{-1}\sum_{j=1}^{d}x_j^* d − 1 ∑ j = 1 d x j ∗ after 100,000 updates, with a 5,000-update burn-in. Empirical coverage is reported for nominal 95% confidence intervals. Each entry summarizes 200 runs. CI length is the mean full interval length. For each setting, we highlight a method’s coverage rate and mean CI length if no other method achieves comparably good coverage with a shorter mean interval. When both measures are comparable, we favor the method with fewer FLOPs per iteration.
For exact and sketched Newton, Table 5 shows that the methods achieve comparably high estimation accuracy. Most MAEs are on the order of 10 − 4 10^{-4} 1 0 − 4 or smaller, and all are below 2 × 10 − 3 2 \times10^{-3} 2 × 1 0 − 3 . Empirical coverage for the Newton methods ranges from 89.5% to 98.5%, with most values close to the nominal level of 95% and only the lowest one slightly below 90%. Interval lengths reflect the uncertainty in the estimated coordinate average. The averaged iterates of exact Newton and Sketch (m t m_t m t ) generally yield shorter intervals than the other Newton estimators. For example, in linear regression with identity covariance, the averaged iterates of exact Newton and Sketch (m t m_t m t ) yield mean interval lengths of 2.567 × 10 − 3 2.567 \times10^{-3} 2.567 × 1 0 − 3 and 2.357 × 10 − 3 2.357 \times10^{-3} 2.357 × 1 0 − 3 , respectively. The mean interval lengths of the other Newton estimators in this setting range from 3.455 × 10 − 3 3.455 \times10^{-3} 3.455 × 1 0 − 3 to 7.626 × 10 − 3 7.626 \times10^{-3} 7.626 × 1 0 − 3 . This pattern is consistent with Theorem 4.7, which shows that the averaged iterate of Sketch (m t m_t m t ) attains the same optimal limiting covariance Ω \Omega Ω as averaged exact Newton.
The results also show that sketching reduces the computational cost of Newton inference, as measured by both FLOPs and runtime. Across all six settings, exact Newton uses 2.733–2.750 GFLOPs per run. Compared with exact Newton in the corresponding settings, Sketch (g t g_t g t ) uses about 74% fewer FLOPs and reduces runtime by 27–53%, while Sketch (m t m_t m t ) uses about 73% fewer FLOPs and reduces runtime by 24–39%. These savings support our cost analysis, which gives an average cost per outer iteration of O ( d 2 ) O(d^2) O ( d 2 ) for coordinate sketching with fixed τ \tau τ and O ( d 3 ) O(d^3) O ( d 3 ) for exact Newton, and illustrate the computational advantage of sketching solvers.
Table 5 also demonstrates the “free lunch” from gradient averaging: it improves both estimation accuracy and statistical efficiency with negligible additional computational cost. With the solver and sketching budget held fixed, Sketch (m t m_t m t ) yields lower MAEs and shorter confidence intervals than Sketch (g t g_t g t ) for both the last and averaged iterates across all six settings. For example, in linear regression with Toeplitz covariance, gradient averaging reduces the MAE of the averaged iterate from 0.464 × 10 − 3 0.464 \times10^{-3} 0.464 × 1 0 − 3 to 0.279 × 10 − 3 0.279 \times10^{-3} 0.279 × 1 0 − 3 and its mean interval length from 2.704 × 10 − 3 2.704 \times10^{-3} 2.704 × 1 0 − 3 to 1.654 × 10 − 3 1.654 \times10^{-3} 1.654 × 1 0 − 3 , with the same 93% empirical coverage for both methods. These substantial improvements in estimation accuracy and statistical efficiency come with little extra computational cost, with the reported FLOP count increasing by only about 1.7%.
As to ASGD, the results show that its estimation accuracy is comparable to that of the Newton methods. However, for inference, its nominal 95% batch-means confidence intervals exhibit substantial undercoverage in all six settings, with empirical coverage of only 57–65%. For example, in linear regression with identity covariance, despite the similar MAEs, ASGD achieves only 58.5% empirical coverage, compared with 93.5% for the averaged iterate of exact Newton. At 100,000 updates, ASGD indeed uses fewer FLOPs and runs faster than the Newton methods, as shown in Table 5 . However, its savings over sketched Newton are modest when measured in runtime. For example, in linear regression with equicorrelated covariates, ASGD takes 1.173 seconds per run on average, compared with 1.285 seconds for Sketch (g t g_t g t ). These time savings offer limited benefit given the observed substantial undercoverage.
More importantly, for inference the relevant comparison is the time and computation required to reach a common coverage target. Under this requirement, ASGD shows no computational advantage over sketched Newton. We run an experiment in which we evaluate coverage every 1,000 updates and identify the first checkpoint with at least 90% empirical coverage. Table 6 reports the stopping step, mean runtime, and mean GFLOPs per run. All exact and sketched Newton methods reach the target within 100,000 updates. Many of these methods attain the required coverage at an early stage, with some reaching the target even within the first 10,000 updates. Their mean runtimes to reach the target range from 0.111 to 2.151 seconds per run. In contrast, ASGD does not reach 90% coverage in any of the six settings within the 30-second limit per run. By that limit, ASGD has spent substantially more time than any Newton method needs to reach the target and has already used more FLOPs than the corresponding Newton method in almost all comparisons. In the few comparisons with lower FLOP counts for ASGD, those counts reflect only the computation performed before ASGD was stopped at the time limit without reaching the target, so they do not establish a computational advantage at the target coverage. The runtime and FLOP advantages of ASGD at a fixed number of updates therefore do not carry over to inference under a common coverage requirement.
Table 6. Time and computational cost to reach 90% empirical coverage at d = 40 d = 40 d = 40 , with 200 runs per setting. Coverage is evaluated every 1,000 updates. The table reports the first checkpoint with at least 90% coverage, along with the corresponding runtime and GFLOPs. If the threshold is not reached, the reported step is the mean number of updates completed by the 30-second cap, rounded to the nearest integer. Runtime and GFLOPs are per-run means that include iteration updates and confidence interval construction.
Notably, Table 6 also shows the benefits of gradient averaging at a common empirical coverage target of 90%. In all six settings, the averaged iterate of Sketch (m t m_t m t ) reaches the required empirical coverage rate in fewer updates, with less time and fewer FLOPs than the averaged iterate of Sketch (g t g_t g t ). These results complement the comparison at 100,000 updates: compared with sketched Newton without gradient averaging, the method with gradient averaging achieves higher statistical efficiency for a similar computational cost after a fixed number of updates and reaches the common coverage target sooner while using less computation.