Introduction
Understanding how first-order stochastic optimization generalizes in high- and infinite-dimensional feature spaces is a central theme in modern statistical learning theory. For offline empirical risk minimization in linear regression, Bartlett et al. [2 ] and Tsigler and Bartlett [33 ] characterized benign overfitting —when minimum-ℓ 2 \ell_2 ℓ 2 -norm interpolators and ridge regression generalize well despite fitting noisy training data—in terms of the effective ranks of the population data covariance operator H \mathbf{H} H ([3 , 11 ] ). In the complementary setting of fresh-sample online stochastic approximation, Zou et al. [37 ] established an instance-dependent excess risk analysis for constant-stepsize stochastic gradient descent (SGD) with full iterate averaging, demonstrating that online SGD enjoys an implicit spectral regularization effect governed by a closely related spectral partition of H \mathbf{H} H (while noting that for polynomially decaying spectra such as λ i = i − 2 \lambda_i=i^{-2} λ i = i − 2 , online SGD achieves vanishing excess risk by forgetting noisy labels rather than interpolating training data; see also Wu et al. (2025)).
In large-scale stochastic optimization, however, averaging all iterates from the initial point w 0 \mathbf{w}_0 w 0 suffers from slow polynomial decay of its deterministic drift bias 1 2 ∥ E [ w ‾ 0 : N ] − w ∗ ∥ H 2 \frac{1}{2}\|\mathbb{E}[\overline{\mathbf{w}}_{0:N}]-\mathbf{w}^{*}\|_{\mathbf{H}}^{2} 2 1 ∥ E [ w 0 : N ] − w ∗ ∥ H 2 —decaying at best as O ( N − 2 ) O(N^{-2}) O ( N − 2 ) when ∥ w 0 − w ∗ ∥ H − 1 < ∞ \|\mathbf{w}_0-\mathbf{w}^{*}\|_{\mathbf{H}^{-1}}<\infty ∥ w 0 − w ∗ ∥ H − 1 < ∞ (or in finite dimensions with λ d > 0 \lambda_d>0 λ d > 0 ), and even more slowly (e.g., Θ ( N − 3 / 2 ) \Theta(N^{-3/2}) Θ ( N − 3/2 ) when λ i = i − 2 \lambda_i=i^{-2} λ i = i − 2 and ( w 0 − w ∗ ) i 2 = i − 2 (\mathbf{w}_0-\mathbf{w}^{*})_i^2=i^{-2} ( w 0 − w ∗ ) i 2 = i − 2 ) when slow spectral modes remain uncontracted. To accelerate the decay of initial error while preserving statistical efficiency, Rakhlin et al. [29 ] introduced suffix (tail) averaging for stochastic gradient descent, and Jain et al. [15 , 14 ] developed its operator-theoretic analysis for least-squares linear regression; in modern deep learning and language model pretraining, its checkpoint-based counterpart Latest Weight Averaging (LAWA) ([16 , 32 ] ) averages recent checkpoints along the trajectory to achieve rapid training speedups. Tail-averaging discards the first s s s iterates as a burn-in phase and averages the subsequent N N N iterates:
w ‾ s : s + N = 1 N ∑ t = s s + N − 1 w t , where w t = w t − 1 + γ ( y t − ⟨ w t − 1 , x t ⟩ ) x t . \overline{\mathbf{w}}_{s:s+N}=\frac{1}{N}\sum_{t=s}^{s+N-1}\mathbf{w}_t,\qquad\text{where}\qquad\mathbf{w}_t=\mathbf{w}_{t-1}+\gamma\left(\mathbf{y}_t-\langle\mathbf{w}_{t-1},\mathbf{x}_t\rangle\right)\mathbf{x}_t. w s : s + N = N 1 t = s ∑ s + N − 1 w t , where w t = w t − 1 + γ ( y t − ⟨ w t − 1 , x t ⟩ ) x t .
Zou et al. [37 ] extended their spectral excess risk analysis to tail-averaged SGD. Let { λ i } i > 1 \{\lambda_i\}_{i>1} { λ i } i > 1 denote the eigenvalues of H \mathbf{H} H in non-increasing order, and define the effective dimension cutoffs k ∗ = max { k : λ k ≥ ( γ N ) − 1 } k^{*}=\max\{k:\lambda_k\geq(\gamma N)^{-1}\} k ∗ = max { k : λ k ≥ ( γ N ) − 1 } and k † = max { k : λ k ≥ ( γ ( s + N ) ) − 1 } k^{\dagger}=\max\{k:\lambda_k\geq(\gamma(s+N))^{-1}\} k † = max { k : λ k ≥ ( γ ( s + N ) ) − 1 } . Up to universal multiplicative constants, when the burn-in length satisfies s = Θ ( N ) s=\Theta(N) s = Θ ( N ) , Zou et al. [37 ] upper-bounds the bias-induced variance contribution by
α ( ∥ w 0 − w ∗ ∥ I 0 : k † 2 + ( s + N ) γ ∥ w 0 − w ∗ ∥ H k † : ∞ 2 ) N γ ( 1 − γ α tr ( H ) ) ⋅ ( k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 ) , (1) \frac{\alpha\left(\|\mathbf{w}_0-\mathbf{w}^{*}\|_{\mathbf{I}_{0:k^{\dagger}}}^{2}+(s+N)\gamma\|\mathbf{w}_0-\mathbf{w}^{*}\|_{\mathbf{H}_{k^{\dagger}:\infty}}^{2}\right)}{N\gamma(1-\gamma\alpha\operatorname{tr}(\mathbf{H}))}\cdot\left(\frac{k^{*}}{N}+N\gamma^{2}\sum_{i>k^{*}}\lambda_i^{2}\right), \tag*{(1)} N γ ( 1 − γ α tr ( H )) α ( ∥ w 0 − w ∗ ∥ I 0 : k † 2 + ( s + N ) γ ∥ w 0 − w ∗ ∥ H k † : ∞ 2 ) ⋅ ( N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 ) , (1)
whereas their lower bound (Zou et al. [37 ] ) yields only
β ∥ w 0 − w ∗ ∥ H k † : ∞ 2 ⋅ ( k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 ) . (2) \beta\|\mathbf{w}_0-\mathbf{w}^{*}\|_{\mathbf{H}_{k^{\dagger}:\infty}}^{2}\cdot\left(\frac{k^{*}}{N}+N\gamma^{2}\sum_{i>k^{*}}\lambda_i^{2}\right). \tag*{(2)} β ∥ w 0 − w ∗ ∥ H k † : ∞ 2 ⋅ ( N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 ) . (2)
Comparing (1) and (2) , Zou et al. [37 ] observed that the two bounds fail to match whenever the initial error w 0 − w ∗ \mathbf{w}_0-\mathbf{w}^{*} w 0 − w ∗ has significant energy in the head subspace I 0 : k † \mathbf{I}_{0:k^{\dagger}} I 0 : k † , and conjectured that the upper bound (1) can be improved. The same discrepancy on the head initialization subspace persists in the exponential moving average (EMA) analysis of Li and Gu [20 ] and in the simplified diagonal covariance analysis of Meterez et al. [22 ] .
Why Both Existing Upper and Lower Bounds Are Loose
In this paper, we show that closing the gap between (1) and (2) requires sharpening both the upper bound in Zou et al. [37 ] and the lower bound in Zou et al. [37 ] . Specifically, expanding the outer product in (1) between the initialization subspaces { I 0 : k † , H k † : ∞ } \{\mathbf{I}_{0:k^{\dagger}},\mathbf{H}_{k^{\dagger}:\infty}\} { I 0 : k † , H k † : ∞ } and the fast and slow variance components { k ∗ / N , N γ 2 ∑ i > k ∗ λ i 2 } \{k^{*}/N,N\gamma^{2}\sum_{i>k^{*}}\lambda_i^{2}\} { k ∗ / N , N γ 2 ∑ i > k ∗ λ i 2 } reveals two distinct physical mechanisms during the burn-in phase:
Looseness in the Upper Bound (Undamped Fast-Mode Terms): The product term 1 γ N ∥ w 0 − w ∗ ∥ I 0 : k † 2 ⋅ k ∗ N \frac{1}{\gamma N}\|\mathbf{w}_0-\mathbf{w}^{*}\|_{\mathbf{I}_{0:k^{\dagger}}}^{2}\cdot\frac{k^{*}}{N} γ N 1 ∥ w 0 − w ∗ ∥ I 0 : k † 2 ⋅ N k ∗ in (1) does not decay with the burn-in length s s s . Consequently, even when
all eigenvalues satisfy λ i ≥ ( γ N ) − 1 \lambda_i \ge(\gamma N)^{-1} λ i ≥ ( γ N ) − 1 (k ∗ = d k^* = d k ∗ = d and H k ∗ : ∞ = 0 \mathbf{H}_{k^*:\infty} = 0 H k ∗ : ∞ = 0 ), (1) predicts a polynomial rate O ( d / ( γ N 2 ) ) O(d/(\gamma N^2)) O ( d / ( γ N 2 )) instead of exponential burn-in decay [14 ] . This looseness arises because prior upper-bound analyses drop the deterministic burn-in contraction acting on multiplicative gradient fluctuations after they are excited. Retaining the exact burn-in propagation shows that the single-hop (i ≤ k ∗ , j ≤ k ∗ i \le k^*, j \le k^* i ≤ k ∗ , j ≤ k ∗ ) contribution is exponentially damped in s s s through ( I − γ H ) s ( w 0 − w ∗ ) (\mathbf{I} - \gamma\mathbf{H})^s(\mathbf{w}_0 - \mathbf{w}^*) ( I − γ H ) s ( w 0 − w ∗ ) and a burn-in cross-spectral convolution (Theorems 4.1 and 4.2), while multi-hop feedback (including energy that scatters into slow modes i > k ∗ i > k^* i > k ∗ and subsequently returns to fast modes) is absorbed into the total excess risk whenever our spectral feedback condition χ s , N ≤ χ 0 < 1 \chi_{s,N} \le\chi_0 < 1 χ s , N ≤ χ 0 < 1 holds.
Looseness in the Lower Bound (Burn-In Head-to-Tail Transfer): Conversely, whenever slow spectral modes i > k ∗ i > k^* i > k ∗ exist (k ∗ < d k^* < d k ∗ < d ), the cross-interaction between the initial head error term ( γ N ) − 1 ∥ w 0 − w ∗ ∥ 0 : k † 2 (\gamma N)^{-1}\|\mathbf{w}_0 - \mathbf{w}^*\|^2_{0:k^\dagger} ( γ N ) − 1 ∥ w 0 − w ∗ ∥ 0 : k † 2 and the slow-spectrum factor N γ 2 ∑ i > k ∗ λ i 2 N\gamma^2\sum_{i>k^*}\lambda_i^2 N γ 2 ∑ i > k ∗ λ i 2 cannot be removed from the total-risk upper bound; instead, it is a genuine lower bound contribution missing from (2) . During the first Θ ( 1 / ( γ λ j ) ) \Theta(1/(\gamma\lambda_j)) Θ ( 1/ ( γ λ j )) steps of the burn-in phase, an initial error component along a fast eigenvector v j \mathbf{v}_j v j (j ≤ k † j \le k^\dagger j ≤ k † ) undergoes multiplicative gradient fluctuations ( x t x t T − H ) ( w t − 1 − w ∗ ) (\mathbf{x}_t\mathbf{x}_t^{\mathsf T} - \mathbf{H})(\mathbf{w}_{t-1} - \mathbf{w}^*) ( x t x t T − H ) ( w t − 1 − w ∗ ) . Under Assumption 3.4 , this process scatters fourth-moment energy proportional to β γ ∥ w 0 − w ∗ ∥ 0 : k † 2 H \beta\gamma\|\mathbf{w}_0 - \mathbf{w}^*\|^2_{0:k^\dagger}\mathbf{H} β γ ∥ w 0 − w ∗ ∥ 0 : k † 2 H across all eigendirections of H \mathbf{H} H . While the single-hop energy scattered into fast modes (i ≤ k ∗ i \le k^* i ≤ k ∗ ) decays over the remaining burn-in steps, the energy scattered into slow modes (i > k † i > k^\dagger i > k † , where λ i < ( γ ( s + N ) ) − 1 \lambda_i < (\gamma(s+N))^{-1} λ i < ( γ ( s + N ) ) − 1 ) experiences negligible contraction ( 1 − γ λ i ) 2 ( s − 1 − t ) ≥ e − 2 (1-\gamma\lambda_i)^{2(s-1-t)} \ge e^{-2} ( 1 − γ λ i ) 2 ( s − 1 − t ) ≥ e − 2 and survives undamped into the averaging window—even in two-dimensional noiseless Gaussian regression (Proposition 4.7(1)).
Main Contributions. Our main contributions are summarized as follows (see Table 1 in Section 4.3 for a detailed term-by-term comparison with Zou et al. [37 ] ):
Table 1. Comparison of the fast-mode (i ≤ k ∗ i \le k^* i ≤ k ∗ ) and slow-mode (i > k ∗ i > k^* i > k ∗ ) index components in the total bias-induced variance bound for tail-averaged SGD w ‾ s : s + N \overline{\mathbf{w}}_{s:s+N} w s : s + N in the canonical regime c 0 N ≤ s ≤ C 0 N c_0N \le s \le C_0N c 0 N ≤ s ≤ C 0 N , omitting multiplicative constants depending on α \alpha α , β \beta β , c 0 c_0 c 0 , C 0 C_0 C 0 , and the stability margin ( 1 − γ α tr ( H ) ) (1-\gamma\alpha\operatorname{tr}(\mathbf{H})) ( 1 − γ α tr ( H )) in Zou et al. [37 ] , and 1 − χ 0 1-\chi_0 1 − χ 0 in Corollary 4.5 under χ s , N ≤ χ 0 < 1 \chi_{s,N}\le\chi_0<1 χ s , N ≤ χ 0 < 1 .
Matching Excess Risk Bounds: Under the distributional assumptions of Zou et al. [37 ] —Assumptions 3.1–3.3 for the upper bound (Theorem 4.1) and the well-specified model (3.4) with Assumptions 3.1, 3.2, and 3.4 for the lower bound (Theorem 4.2 )—we derive an exact trajectory operator-polynomial identity (Lemma 5.1 ) and control repeated fourth-moment interactions via a scalar spectral feedback ratio χ s , N \chi_{s,N} χ s , N (Lemma 5.2 ). Whenever χ s , N ≤ χ 0 < 1 \chi_{s,N} \le\chi_0 < 1 χ s , N ≤ χ 0 < 1 , the multi-hop feedback terms are absorbed within a factor of ( 1 − χ s , N ) − 1 (1-\chi_{s,N})^{-1} ( 1 − χ s , N ) − 1 , yielding explicit spectral upper and lower bounds for EffectiveBias and EffectiveVar that match up to constants depending only on α \alpha α , β \beta β , and 1 − χ 0 1-\chi_0 1 − χ 0 across all dimensions, initializations w 0 − w ∗ \mathbf{w}_0-\mathbf{w}^* w 0 − w ∗ , burn-in lengths s ≥ 0 s \ge0 s ≥ 0 , and window sizes N ≥ 2 N \ge2 N ≥ 2 (Theorems 4.1 –4.2; and up to constants also depending on c 0 c_0 c 0 , C 0 C_0 C 0 in the canonical specialization c 0 N ≤ s ≤ C 0 N c_0N \le s \le C_0N c 0 N ≤ s ≤ C 0 N of Corollary 4.5 ).
Stepsize Rules for Spectral Feedback Control: Because the tail-averaging filter weights F t F_t F t increase during the burn-in phase 1 ≤ t ≤ s 1 \le t \le s 1 ≤ t ≤ s , standard monotone renewal arguments cannot bound χ s , N \chi_{s,N} χ s , N . By expanding F t F_t F t via a pair-midpoint identity, we establish two concrete constant-stepsize regimes guaranteeing χ s , N ≤ 1 / 2 \chi_{s,N} \le1/2 χ s , N ≤ 1/2 (Proposition 4.3 ): (i) for arbitrary spectra H \mathbf{H} H and any finite horizon T = s + N − 1 ≥ 2 T=s+N-1 \ge2 T = s + N − 1 ≥ 2 , positive correlation of the kernel modes yields χ s , N ≤ α γ 2 ( T − 1 ) tr ( H 2 ) ( 1 − γ λ 1 ) − 2 \chi_{s,N} \le\alpha\gamma^2(T-1)\operatorname{tr}(\mathbf{H}^2)(1-\gamma\lambda_1)^{-2} χ s , N ≤ α γ 2 ( T − 1 ) tr ( H 2 ) ( 1 − γ λ 1 ) − 2 , so any pre-run stepsize γ ≤ min { ( 2 λ 1 ) − 1 , ( 8 α ( T − 1 ) tr ( H 2 ) ) − 1 / 2 } \gamma\le\min\{(2\lambda_1)^{-1},(8\alpha(T-1)\operatorname{tr}(\mathbf{H}^2))^{-1/2}\} γ ≤ min {( 2 λ 1 ) − 1 , ( 8 α ( T − 1 ) tr ( H 2 ) ) − 1/2 } guarantees χ s , N ≤ 1 / 2 \chi_{s,N} \le1/2 χ s , N ≤ 1/2 ; and (ii) splitting the inner convolution at the midpoint bounds χ s , N \chi_{s,N} χ s , N through the kernel doubling constant D T = sup 1 ≤ h ≤ T k ( ( h − 1 ) / 2 ) / k ( h ) D_T = \sup_{1\le h\le T} k((h-1)/2)/k(h) D T = sup 1 ≤ h ≤ T k (( h − 1 ) /2 ) / k ( h ) . For polynomially decaying spectra θ Λ i − a ≤ λ i ≤ Λ i − a \theta\Lambda i^{-a} \le\lambda_i \le\Lambda i^{-a} θ Λ i − a ≤ λ i ≤ Λ i − a (a ≥ 1 a \ge1 a ≥ 1 ), we prove D T ≤ D ∗ ( a , θ ) < ∞ D_T \le D_*(a,\theta) < \infty D T ≤ D ∗ ( a , θ ) < ∞ uniformly in T T T , so a single horizon-independent stepsize ensures χ s , N ≤ 1 / 2 \chi_{s,N} \le1/2 χ s , N ≤ 1/2 across all s , N s,N s , N , and w 0 − w ∗ \mathbf{w}_0-\mathbf{w}^* w 0 − w ∗ (Corollary 4.4 ).
Separation Constructions and Conditional Bounds: We establish two explicit separation results (Proposition 4.7): (i) in two-dimensional noiseless Gaussian regression with H = diag ( 1 , ( 4 γ / N ) − 1 ) \mathbf{H}=\operatorname{diag}(1,(4\gamma/N)^{-1}) H = diag ( 1 , ( 4 γ / N ) − 1 ) and w 0 − w ∗ = e 1 \mathbf{w}_0-\mathbf{w}^*=\mathbf{e}_1 w 0 − w ∗ = e 1 , burn-in transfer from mode 1 to mode 2 forces excess risk Ω ( 1 / ( γ N 2 ) ) \Omega(1/(\gamma N^2)) Ω ( 1/ ( γ N 2 )) , whereas Zou et al. [37 ] gives the exponentially small bound O ( ( 1 − γ ) 2 N / ( γ 2 N 2 ) ) O((1-\gamma)^{2N}/(\gamma^2N^2)) O (( 1 − γ ) 2 N / ( γ 2 N 2 )) ; and (ii) in one-dimensional regression where k ( h ) k(h) k ( h ) decays exponentially and D T → ∞ D_T\to\infty D T → ∞ , two distributions sharing the same ( H , w 0 − w ∗ , α , β ) (\mathbf{H},\mathbf{w}_0-\mathbf{w}^*,\alpha,\beta) ( H , w 0 − w ∗ , α , β ) exhibit an exponential ratio gap ( r 3 / r 2 ) N → ∞ (r_3/r_2)^N\to\infty ( r 3 / r 2 ) N → ∞ , proving that no single expression depending only on ( H , w 0 − w ∗ , γ , s , N , α , β , σ n o i s e 2 ) (\mathbf{H},\mathbf{w}_0-\mathbf{w}^*,\gamma,s,N,\alpha,\beta,\sigma_{\mathrm{noise}}^2) ( H , w 0 − w ∗ , γ , s , N , α , β , σ noise 2 ) can match both risks without an additional structural condition (such as χ s , N ≤ χ 0 < 1 \chi_{s,N}\leq\chi_0<1 χ s , N ≤ χ 0 < 1 ) or state information (such as B s \mathbf{B}_s B s in Proposition 4.8).
Notation. We use lowercase letters ( a , b , γ , λ ) (a,b,\gamma,\lambda) ( a , b , γ , λ ) to denote scalars, boldface lowercase letters ( w , x , η ) (\mathbf{w},\mathbf{x},\boldsymbol{\eta}) ( w , x , η ) to denote vectors in the separable Hilbert space H \mathcal{H} H (where H = R d \mathcal{H}=\mathbb{R}^d H = R d or ℓ 2 \ell_2 ℓ 2 ), boldface uppercase letters ( A , B , H , I ) (\mathbf{A},\mathbf{B},\mathbf{H},\mathbf{I}) ( A , B , H , I ) to denote bounded self-adjoint linear operators (matrices) on H \mathcal{H} H , and calligraphic letters ( I , M , S , T , V ) (\mathcal{I},\mathcal{M},\mathcal{S},\mathcal{T},\mathcal{V}) ( I , M , S , T , V ) to denote linear super-operators or scalar risk functionals. For a vector w ∈ H \mathbf{w}\in\mathcal{H} w ∈ H , we write ( w ) i : = ⟨ v i , w ⟩ (\mathbf{w})_i:=\langle\mathbf{v}_i,\mathbf{w}\rangle ( w ) i := ⟨ v i , w ⟩ for its i i i -th coordinate in the eigenbasis of H \mathbf{H} H , and for a positive semi-definite (PSD) operator A ⪰ 0 \mathbf{A}\succeq0 A ⪰ 0 , we write ∥ w ∥ A : = ⟨ w , A w ⟩ \|\mathbf{w}\|_{\mathbf{A}}:=\sqrt{\langle\mathbf{w},\mathbf{A}\mathbf{w}\rangle} ∥ w ∥ A := ⟨ w , Aw ⟩ , where ⟨ u , v ⟩ = u T v \langle\mathbf{u},\mathbf{v}\rangle=\mathbf{u}^{\mathsf{T}}\mathbf{v} ⟨ u , v ⟩ = u T v . For two vectors u , v ∈ H \mathbf{u},\mathbf{v}\in\mathcal{H} u , v ∈ H , their outer product is the rank-one operator u ⊗ v : = u v T \mathbf{u}\otimes\mathbf{v}:=\mathbf{u}\mathbf{v}^{\mathsf{T}} u ⊗ v := u v T ; for two bounded operators A , B \mathbf{A},\mathbf{B} A , B on H \mathcal{H} H , A ⊗ B \mathbf{A}\otimes\mathbf{B} A ⊗ B denotes the linear super-operator acting on operators C \mathbf{C} C via ( A ⊗ B ) ∘ C : = A C B T (\mathbf{A}\otimes\mathbf{B})\circ\mathbf{C}:=\mathbf{A}\mathbf{C}\mathbf{B}^{\mathsf{T}} ( A ⊗ B ) ∘ C := AC B T (so ( H ⊗ I + I ⊗ H ) ∘ C = H C + C H (\mathbf{H}\otimes\mathbf{I}+\mathbf{I}\otimes\mathbf{H})\circ\mathbf{C}=\mathbf{H}\mathbf{C}+\mathbf{C}\mathbf{H} ( H ⊗ I + I ⊗ H ) ∘ C = HC + CH preserves self-adjointness); and for two operators A \mathbf{A} A and B \mathbf{B} B whose product A T B \mathbf{A}^{\mathsf{T}}\mathbf{B} A T B is trace-class, their trace inner product is ⟨ A , B ⟩ : = tr ( A T B ) \langle\mathbf{A},\mathbf{B}\rangle:=\operatorname{tr}(\mathbf{A}^{\mathsf{T}}\mathbf{B}) ⟨ A , B ⟩ := tr ( A T B ) . Given the spectral decomposition H = ∑ i ≥ 1 λ i v i v i T \mathbf{H}=\sum_{i\geq1}\lambda_i\mathbf{v}_i\mathbf{v}_i^{\mathsf{T}} H = ∑ i ≥ 1 λ i v i v i T with λ 1 ≥ λ 2 ≥ ⋯ > 0 \lambda_1\geq\lambda_2\geq\cdots>0 λ 1 ≥ λ 2 ≥ ⋯ > 0 , for any index pair 0 ≤ k 1 ≤ k 2 ≤ ∞ 0\leq k_1\leq k_2\leq\infty 0 ≤ k 1 ≤ k 2 ≤ ∞ and finite head cutoff 0 ≤ k < ∞ 0\leq k<\infty 0 ≤ k < ∞ , we define the spectral projections and truncated operators
I k 1 : k 2 : = ∑ k 1 < i ≤ k 2 v i v i T , H k 1 : k 2 : = ∑ k 1 < i ≤ k 2 λ i v i v i T , H 0 : k − 1 : = ∑ 1 ≤ i ≤ k λ i − 1 v i v i T , \mathbf{I}_{k_1:k_2}:=\sum_{k_1<i\leq k_2}\mathbf{v}_i\mathbf{v}_i^{\mathsf{T}},\qquad\mathbf{H}_{k_1:k_2}:=\sum_{k_1<i\leq k_2}\lambda_i\mathbf{v}_i\mathbf{v}_i^{\mathsf{T}},\qquad\mathbf{H}_{0:k}^{-1}:=\sum_{1\leq i\leq k}\lambda_i^{-1}\mathbf{v}_i\mathbf{v}_i^{\mathsf{T}}, I k 1 : k 2 := k 1 < i ≤ k 2 ∑ v i v i T , H k 1 : k 2 := k 1 < i ≤ k 2 ∑ λ i v i v i T , H 0 : k − 1 := 1 ≤ i ≤ k ∑ λ i − 1 v i v i T ,
and write ∥ w ∥ H − 1 2 : = ∑ i ≥ 1 λ i − 1 ( w ) i 2 ∈ [ 0 , ∞ ] \|\mathbf{w}\|_{\mathbf{H}^{-1}}^2:=\sum_{i\geq1}\lambda_i^{-1}(\mathbf{w})_i^2\in[0,\infty] ∥ w ∥ H − 1 2 := ∑ i ≥ 1 λ i − 1 ( w ) i 2 ∈ [ 0 , ∞ ] for the inverse-covariance quadratic form. Throughout the paper, any spectral cutoff max { k : λ k ≥ τ } \max\{k:\lambda_k\geq\tau\} max { k : λ k ≥ τ } is set to 0 when the defining set is empty. For sequences f n , g n ≥ 0 f_n,g_n\geq0 f n , g n ≥ 0 , we write f n ≲ g n f_n\lesssim g_n f n ≲ g n (or f n = O ( g n ) f_n=O(g_n) f n = O ( g n ) ) if f n ≤ C g n f_n\leq Cg_n f n ≤ C g n for a positive constant C C C , f n ≳ g n f_n\gtrsim g_n f n ≳ g n (or f n = Ω ( g n ) f_n=\Omega(g_n) f n = Ω ( g n ) ) if g n ≲ f n g_n\lesssim f_n g n ≲ f n , and f n ≍ g n f_n\asymp g_n f n ≍ g n (or f n = Θ ( g n ) f_n=\Theta(g_n) f n = Θ ( g n ) ) if both f n ≲ g n f_n\lesssim g_n f n ≲ g n and f n ≳ g n f_n\gtrsim g_n f n ≳ g n hold. Finally, for real numbers a , b ∈ R a,b\in\mathbb{R} a , b ∈ R , we write a ∧ b : = min { a , b } a\wedge b:=\min\{a,b\} a ∧ b := min { a , b } and a ∨ b : = max { a , b } a\vee b:=\max\{a,b\} a ∨ b := max { a , b } .
Related Work
Stochastic Approximation and Suffix/Tail Averaging. Stochastic gradient descent and Polyak–Ruppert iterate averaging have a rich history in stochastic approximation ([30 , 28 , 31 , 25 ] ). Because full-iterate averaging forgets initial conditions only at a polynomial rate, Rakhlin et al. [29 ] introduced suffix (tail) averaging for strongly convex stochastic optimization. For least-squares regression in finite and non-parametric dimensions, Bach and Moulines [1 ] , Défossez and Bach [7 ] , and Dieuleveut and Bach [8 ] ; Dieuleveut et al. [9 ] established convergence guarantees for constant- and decaying-stepsize SGD using operator recursions. Jain et al. [13 ] developed a Markov chain operator perspective for constant-stepsize least-squares SGD, proving minimax optimal rates and analyzing tail-averaging to achieve exponential forgetting of the initial error in finite dimensions. Jain et al. [14 ] and Ge et al. (2019) further studied accelerated and step-decay schedules for least-squares SGD, while Lakshminarayanan and Szepesvári [17 ] examined constant-stepsize linear stochastic approximation with iterate averaging. In high-dimensional least-squares settings, Paquette et al. [26 , 27 ] characterized exact average-case SGD risk trajectories via Volterra integral equations.
Benign Overfitting and Implicit Regularization in Linear Regression. The phenomenon of benign overfitting in offline linear regression was formalized by Bartlett et al. [2 ] , who proved matching upper and lower variance bounds and effective-rank characterizations for the minimum-ℓ 2 \ell_2 ℓ 2 -norm interpolator in terms of the data covariance H \mathbf{H} H . Tsigler and Bartlett [33 ] extended these sharp characterizations to ridge regression and minimum-ℓ 2 \ell_2 ℓ 2 -norm interpolation, while Hastie et al. [11 ] and Muthukumar et al. [24 ] studied asymptotic and finite-sample high-dimensional regression regimes (and Chatterji and Long [6 ] analyzed overparameterized linear classification). More recently, Wu et al. (2025) established instance-wise risk comparisons showing that implicit regularization via gradient methods can dominate explicit ridge regularization in linear regression.
Sharp Risk Bounds for SGD with Weight Averaging in Linear Regression. Zou et al. [37 ] bridged the Markov operator framework of Jain et al. [13 ] with the spectral partitioning technique of Bartlett et al. [2 ] to derive instance-dependent upper and lower bounds for constant-stepsize online SGD with full iterate averaging (Zou et al., [37 ] ) and tail-averaging (Zou et al., [37 ] ). Related works have studied several complementary directions: Berthier et al. [4 ] studied tight non-parametric rates in noiseless linear models; Wu et al. (2022) analyzed the last-iterate risk of SGD with polynomial and step-decay stepsize schedules; Li et al. (2024) established instance-dependent risk bounds for accelerated SGD with Nesterov momentum and tail averaging; Lin et al. [21 ] and Li et al. [20 ] studied scaling laws and learning-rate schedules under power-law spectral conditions; and Li and Gu [20 ] analyzed SGD with exponential moving average (EMA), where a parallel gap between upper and lower bounds on the head initialization subspace appears between their Theorems 4.1 and 4.2. In large-scale deep learning and language model pretraining, checkpoint-based tail averaging—termed Latest Weight Averaging (LAWA) (Kaddour, [16 ] ; Sanyal et al., [32 ] )—has been used widely to accelerate convergence, motivating a sharp theoretical understanding of tail-averaged SGD. Methodologically, Meterez et al. [22 ] recently simplified the proof of Zou et al. [37 ] by working in the diagonal eigenbasis of H \mathbf{H} H while retaining the same gap on the head initialization subspace, and Meterez et al. [23 ] studied horizon-free learning-rate schedules with weight averaging for anytime pretraining.
Problem Setup and Preliminaries
We consider linear regression over a separable Hilbert space H \mathcal{H} H (which may be R d \mathbb{R}^{d} R d or infinite-dimensional). Let D \mathcal{D} D be an unknown population distribution over feature-response pairs ( x , y ) ∈ H × R (\mathbf{x},y) \in\mathcal{H} \times\mathbb{R} ( x , y ) ∈ H × R . The population risk of a weight vector w ∈ H \mathbf{w} \in\mathcal{H} w ∈ H is given by the expected squared loss
L ( w ) = 1 2 E ( x , y ) ∼ D [ ( y − ⟨ w , x ⟩ ) 2 ] . L(\mathbf{w}) = \frac{1}{2}\mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}}\left[(y-\langle\mathbf{w},\mathbf{x}\rangle)^2\right]. L ( w ) = 2 1 E ( x , y ) ∼ D [ ( y − ⟨ w , x ⟩ ) 2 ] .
Tail-Averaged Stochastic Gradient Descent
Given an initialization w 0 ∈ H \mathbf{w}_0 \in\mathcal{H} w 0 ∈ H and a stream of independent and identically distributed (i.i.d.) samples { ( x t , y t ) } t ≥ 1 \{(\mathbf{x}_t,y_t)\}_{t\geq1} {( x t , y t ) } t ≥ 1 drawn from D \mathcal{D} D , constant-stepsize SGD updates the weight vector at iteration t ≥ 1 t \geq1 t ≥ 1 according to
w t = w t − 1 + γ ( y t − ⟨ w t − 1 , x t ⟩ ) x t , (3) \mathbf{w}_t = \mathbf{w}_{t-1} + \gamma\left(y_t-\langle\mathbf{w}_{t-1},\mathbf{x}_t\rangle\right)\mathbf{x}_t, \tag*{(3)} w t = w t − 1 + γ ( y t − ⟨ w t − 1 , x t ⟩ ) x t , (3)
where γ > 0 \gamma> 0 γ > 0 is the learning rate (stepsize). Given a burn-in period of length s ≥ 0 s \ge0 s ≥ 0 and an averaging window of length N ≥ 1 N \ge1 N ≥ 1 , SGD with tail-averaging outputs the average of the iterates from step s s s to step s + N − 1 s+N-1 s + N − 1 :
w ‾ s : s + N = 1 N ∑ t = s s + N − 1 w t . (4) \overline{\mathbf{w}}_{s:s+N} = \frac{1}{N}\sum_{t=s}^{s+N-1}\mathbf{w}_t. \tag*{(4)} w s : s + N = N 1 t = s ∑ s + N − 1 w t . (4)
During the first s s s burn-in iterations, the algorithm executes constant-stepsize SGD updates (3) without recording the iterates into the running average; from iteration t = s t=s t = s to s + N − 1 s+N-1 s + N − 1 , it averages the subsequent N N N iterates to form w ‾ s : s + N \overline{\mathbf{w}}_{s:s+N} w s : s + N in (4) . When s = 0 s=0 s = 0 , (4) reduces to standard full-iterate averaging w ‾ 0 : N = 1 N ∑ t = 0 N − 1 w t \overline{\mathbf{w}}_{0:N}=\frac{1}{N}\sum_{t=0}^{N-1}\mathbf{w}_t w 0 : N = N 1 ∑ t = 0 N − 1 w t [37 ] .
Model Assumptions
Following Zou et al. [37 ] , we adopt standard second- and fourth-moment regularity conditions on the feature distribution and label noise. Specifically, our upper bounds rely solely on Assumptions 3.1, 3.2, and 3.3, while our lower bounds rely on the well-specified linear model (6) together with Assumptions 3.1, 3.2, and 3.4.
Assumption 3.1 (Regularity Conditions; Zou et al. [37 ] , Assumption 2.1). The second moment operator H : = E ( x , y ) ∼ D [ x x T ] \mathbf{H}:=\mathbb{E}_{(x,y)\sim\mathcal{D}}[\mathbf{x}\mathbf{x}^{\mathsf{T}}] H := E ( x , y ) ∼ D [ x x T ] is strictly positive definite (H ≻ 0 \mathbf{H}\succ0 H ≻ 0 ) and has finite trace tr ( H ) < ∞ \operatorname{tr}(\mathbf{H})<\infty tr ( H ) < ∞ . Moreover, the population risk L ( w ) L(\mathbf{w}) L ( w ) admits a unique global minimizer w ∗ = arg min w ∈ H L ( w ) \mathbf{w}^{*}=\operatorname*{arg\,min}_{\mathbf{w}\in\mathcal{H}}L(\mathbf{w}) w ∗ = arg min w ∈ H L ( w ) satisfying L ( w ∗ ) < ∞ L(\mathbf{w}^{*})<\infty L ( w ∗ ) < ∞ .
Let H = ∑ i ≥ 1 λ i v i v i T \mathbf{H}=\sum_{i\ge1}\lambda_i\mathbf{v}_i\mathbf{v}_i^{\mathsf{T}} H = ∑ i ≥ 1 λ i v i v i T be the eigendecomposition of H \mathbf{H} H , where the eigenvalues are ordered non-increasingly as λ 1 ≥ λ 2 ≥ ⋯ > 0 \lambda_1\ge\lambda_2\ge\cdots>0 λ 1 ≥ λ 2 ≥ ⋯ > 0 and ∑ i ≥ 1 λ i = tr ( H ) < ∞ \sum_{i\ge1}\lambda_i=\operatorname{tr}(\mathbf{H})<\infty ∑ i ≥ 1 λ i = tr ( H ) < ∞ . To control the fourth-moment behavior of the stochastic rank-one updates x t x t T \mathbf{x}_t\mathbf{x}_t^{\mathsf{T}} x t x t T , define the fourth-moment PSD super-operators M \mathcal{M} M and M ~ \widetilde{\mathcal{M}} M acting on any symmetric operator A \mathbf{A} A by
M ∘ A : = E ( x , y ) ∼ D [ x x T A x x T ] , M ~ ∘ A : = H A H . (5) \mathcal{M}\circ\mathbf{A}:=\mathbb{E}_{(x,y)\sim\mathcal{D}}[\mathbf{x}\mathbf{x}^{\mathsf{T}}\mathbf{A}\mathbf{x}\mathbf{x}^{\mathsf{T}}],\qquad\widetilde{\mathcal{M}}\circ\mathbf{A}:=\mathbf{H}\mathbf{A}\mathbf{H}. \tag*{(5)} M ∘ A := E ( x , y ) ∼ D [ x x T Ax x T ] , M ∘ A := HAH . (5)
Assumption 3.2 (Fourth-Moment Upper Bound; Zou et al. [37 ] , Assumption 2.2). There exists a constant α > 0 \alpha>0 α > 0 such that for any PSD operator A ⪰ 0 \mathbf{A}\succeq0 A ⪰ 0 , it holds that
M ∘ A = E ( x , y ) ∼ D [ x x T A x x T ] ⪯ α tr ( H A ) H . \mathcal{M}\circ\mathbf{A}=\mathbb{E}_{(x,y)\sim\mathcal{D}}[\mathbf{x}\mathbf{x}^{\mathsf{T}}\mathbf{A}\mathbf{x}\mathbf{x}^{\mathsf{T}}]\preceq\alpha\operatorname{tr}(\mathbf{H}\mathbf{A})\mathbf{H}. M ∘ A = E ( x , y ) ∼ D [ x x T Ax x T ] ⪯ α tr ( HA ) H .
As shown in Lemma C.4 (restated from Zou et al. [37 ] ), Assumption 3.2 holds with α = 16 σ z 4 \alpha=16\sigma_z^4 α = 16 σ z 4 whenever the standardized projections of x \mathbf{x} x are σ z 2 \sigma_z^2 σ z 2 -sub-Gaussian (i.e., E [ exp ( ⟨ u , x ⟩ ) ] ≤ exp ( σ z 2 ∥ u ∥ H 2 / 2 ) \mathbb{E}[\exp(\langle\mathbf{u},\mathbf{x}\rangle)]\le\exp(\sigma_z^2\|\mathbf{u}\|_{\mathbf{H}}^2/2) E [ exp (⟨ u , x ⟩)] ≤ exp ( σ z 2 ∥ u ∥ H 2 /2 ) for all u ∈ H \mathbf{u}\in\mathcal{H} u ∈ H , or z : = H − 1 / 2 x \mathbf{z}:=\mathbf{H}^{-1/2}\mathbf{x} z := H − 1/2 x is σ z 2 \sigma_z^2 σ z 2 -sub-Gaussian in finite dimension), and with α = 3 \alpha=3 α = 3 when x ∼ N ( 0 , H ) \mathbf{x}\sim\mathcal{N}(0,\mathbf{H}) x ∼ N ( 0 , H ) . Note also that taking A = v 1 v 1 T \mathbf{A}=\mathbf{v}_1\mathbf{v}_1^{\mathsf{T}} A = v 1 v 1 T in H A H = M ~ ∘ A ⪯ M ∘ A ⪯ α tr ( H A ) H \mathbf{H}\mathbf{A}\mathbf{H}=\widetilde{\mathcal{M}}\circ\mathbf{A}\preceq\mathcal{M}\circ\mathbf{A}\preceq\alpha\operatorname{tr}(\mathbf{H}\mathbf{A})\mathbf{H} HAH = M ∘ A ⪯ M ∘ A ⪯ α tr ( HA ) H yields λ 1 2 ≤ α λ 1 2 \lambda_1^2\le\alpha\lambda_1^2 λ 1 2 ≤ α λ 1 2 , so α ≥ 1 \alpha\ge1 α ≥ 1 always holds; moreover, ( M − M ~ ) ∘ A ⪯ M ∘ A ⪯ α tr ( H A ) H (\mathcal{M}-\widetilde{\mathcal{M}})\circ\mathbf{A}\preceq\mathcal{M}\circ\mathbf{A}\preceq\alpha\operatorname{tr}(\mathbf{H}\mathbf{A})\mathbf{H} ( M − M ) ∘ A ⪯ M ∘ A ⪯ α tr ( HA ) H for all A ⪰ 0 \mathbf{A}\succeq0 A ⪰ 0 .
Assumption 3.3 (Noise Condition; Zou et al. [37 ] , Assumption 2.3). Define the residual noise ξ : = y − ⟨ w ∗ , x ⟩ \xi:=y-\langle\mathbf{w}^{*},\mathbf{x}\rangle ξ := y − ⟨ w ∗ , x ⟩ and the noise-weighted covariance operator Σ : = E ( x , y ) ∼ D [ ξ 2 x x T ] \mathbf{\Sigma}:=\mathbb{E}_{(x,y)\sim\mathcal{D}}[\xi^2\mathbf{x}\mathbf{x}^{\mathsf{T}}] Σ := E ( x , y ) ∼ D [ ξ 2 x x T ] . Suppose that
σ 2 : = ∥ H − 1 / 2 Σ H − 1 / 2 ∥ 2 2 < ∞ , \sigma^2:=\left\|\mathbf{H}^{-1/2}\mathbf{\Sigma}\mathbf{H}^{-1/2}\right\|_2^2<\infty, σ 2 := H − 1/2 Σ H − 1/2 2 2 < ∞ , or equivalently, Σ ⪯ σ 2 H \mathbf{\Sigma}\preceq\sigma^2\mathbf{H} Σ ⪯ σ 2 H in the Loewner PSD order.
Assumption 3.3 accommodates model misspecification and heteroscedastic noise, requiring only that E [ ξ x ] = ∇ L ( w ∗ ) = 0 \mathbb{E}[\xi\mathbf{x}]=\nabla L(\mathbf{w}^{*})=0 E [ ξ x ] = ∇ L ( w ∗ ) = 0 and Σ ⪯ σ 2 H \boldsymbol{\Sigma}\preceq\sigma^{2}\mathbf{H} Σ ⪯ σ 2 H . For the lower bound analysis, we consider the well-specified homoscedastic linear regression model (slightly relaxing Zou et al. [37 ] , which specifies Gaussian noise ξ ∼ N ( 0 , σ n o i s e 2 ) \xi\sim\mathcal{N}(0,\sigma_{\mathrm{noise}}^{2}) ξ ∼ N ( 0 , σ noise 2 ) , to arbitrary centered finite-variance noise independent of x \mathbf{x} x ):
y = ⟨ w ∗ , x ⟩ + ξ , ξ ⊥ x , E [ ξ ] = 0 , and E [ ξ 2 ] = σ n o i s e 2 < ∞ . (6) y=\langle\mathbf{w}^{*},\mathbf{x}\rangle+\xi,\qquad\xi\perp\mathbf{x},\qquad\mathbb{E}[\xi]=0,\qquad\text{and}\qquad\mathbb{E}[\xi^{2}]=\sigma_{\mathrm{noise}}^{2}<\infty. \tag*{(6)} y = ⟨ w ∗ , x ⟩ + ξ , ξ ⊥ x , E [ ξ ] = 0 , and E [ ξ 2 ] = σ noise 2 < ∞. (6)
Under (6) , Σ = σ n o i s e 2 H \boldsymbol{\Sigma}=\sigma_{\mathrm{noise}}^{2}\mathbf{H} Σ = σ noise 2 H , so Assumption 3.3 holds with equality ( σ 2 = σ n o i s e 2 ) (\sigma^{2}=\sigma_{\mathrm{noise}}^{2}) ( σ 2 = σ noise 2 ) . Finally, to establish lower bounds on the fourth-moment bias-variance interaction, we adopt the fourth-moment lower bound condition of Zou et al. [37 ] .
Assumption 3.4 (Fourth-Moment Lower Bound; Zou et al. [37 ] ). There exists a constant β ∈ ( 0 , α ] \beta\in(0,\alpha] β ∈ ( 0 , α ] such that for any PSD operator A ⪰ 0 \mathbf{A}\succeq0 A ⪰ 0 , it holds that
( M − M ~ ) ∘ A = E ( x , y ) ∼ D [ x x T A x x T ] − H A H ⪰ β tr ( H A ) H . (\mathcal{M}-\widetilde{\mathcal{M}})\circ\mathbf{A}=\mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}}[\mathbf{x}\mathbf{x}^{\mathsf{T}}\mathbf{A}\mathbf{x}\mathbf{x}^{\mathsf{T}}]-\mathbf{H}\mathbf{A}\mathbf{H}\succeq\beta\operatorname{tr}(\mathbf{H}\mathbf{A})\mathbf{H}. ( M − M ) ∘ A = E ( x , y ) ∼ D [ x x T Ax x T ] − HAH ⪰ β tr ( HA ) H . When x ∼ N ( 0 , H ) \mathbf{x}\sim\mathcal{N}(0,\mathbf{H}) x ∼ N ( 0 , H ) , Isserlis’ theorem yields M ∘ A = 2 H A H + tr ( H A ) H \mathcal{M}\circ\mathbf{A}=2\mathbf{H}\mathbf{A}\mathbf{H}+\operatorname{tr}(\mathbf{H}\mathbf{A})\mathbf{H} M ∘ A = 2 HAH + tr ( HA ) H , so ( M − M ~ ) ∘ A = H A H + tr ( H A ) H ⪰ tr ( H A ) H (\mathcal{M}-\widetilde{\mathcal{M}})\circ\mathbf{A}=\mathbf{H}\mathbf{A}\mathbf{H}+\operatorname{tr}(\mathbf{H}\mathbf{A})\mathbf{H}\succeq\operatorname{tr}(\mathbf{H}\mathbf{A})\mathbf{H} ( M − M ) ∘ A = HAH + tr ( HA ) H ⪰ tr ( HA ) H and Assumption 3.4 holds with β = 1 \beta=1 β = 1 (and with β = 2 \beta=2 β = 2 in scalar Gaussian regression d = 1 d=1 d = 1 ).
Bias-Variance Decomposition and Effective Risk Components
Following Jain et al. [15 ] , Zou et al. [37 ] , and Li and Gu [20 ] , let η t : = w t − w ∗ \eta_t:=\mathbf{w}_t-\mathbf{w}^{*} η t := w t − w ∗ denote the estimation error at iteration t t t and η ˉ s : s + N : = w ˉ s : s + N − w ∗ = 1 N ∑ t = s s + N − 1 η t \bar{\eta}_{s:s+N}:=\bar{\mathbf{w}}_{s:s+N}-\mathbf{w}^{*}=\frac{1}{N}\sum_{t=s}^{s+N-1}\eta_t η ˉ s : s + N := w ˉ s : s + N − w ∗ = N 1 ∑ t = s s + N − 1 η t . By linearity of (3) , η t = η t b i a s + η t v a r i a n c e \eta_t=\eta_t^{\mathrm{bias}}+\eta_t^{\mathrm{variance}} η t = η t bias + η t variance , where the noiseless bias process and zero-initialized variance process satisfy
η t b i a s = ( I − γ x t x t T ) η t − 1 b i a s , η 0 b i a s = w 0 − w ∗ , (7) \eta_t^{\mathrm{bias}}=(\mathbf{I}-\gamma\mathbf{x}_t\mathbf{x}_t^{\mathsf{T}})\eta_{t-1}^{\mathrm{bias}},\qquad\eta_0^{\mathrm{bias}}=\mathbf{w}_0-\mathbf{w}^{*}, \tag*{(7)} η t bias = ( I − γ x t x t T ) η t − 1 bias , η 0 bias = w 0 − w ∗ , (7)
η t v a r i a n c e = ( I − γ x t x t T ) η t − 1 v a r i a n c e + γ ξ t x t , η 0 v a r i a n c e = 0. (8) \eta_t^{\mathrm{variance}}=(\mathbf{I}-\gamma\mathbf{x}_t\mathbf{x}_t^{\mathsf{T}})\eta_{t-1}^{\mathrm{variance}}+\gamma\xi_t\mathbf{x}_t,\qquad\eta_0^{\mathrm{variance}}=0. \tag*{(8)} η t variance = ( I − γ x t x t T ) η t − 1 variance + γ ξ t x t , η 0 variance = 0. (8)
Here ξ t : = y t − ⟨ w ∗ , x t ⟩ \xi_t:=y_t-\langle\mathbf{w}^{*},\mathbf{x}_t\rangle ξ t := y t − ⟨ w ∗ , x t ⟩ , and the single-step second-moment operators are B t : = E [ η t b i a s ⊗ η t b i a s ] \mathbf{B}_t:=\mathbb{E}[\eta_t^{\mathrm{bias}}\otimes\eta_t^{\mathrm{bias}}] B t := E [ η t bias ⊗ η t bias ] and C t : = E [ η t v a r i a n c e ⊗ η t v a r i a n c e ] \mathbf{C}_t:=\mathbb{E}[\eta_t^{\mathrm{variance}}\otimes\eta_t^{\mathrm{variance}}] C t := E [ η t variance ⊗ η t variance ] (initialized at B 0 = ( w 0 − w ∗ ) ⊗ ( w 0 − w ∗ ) \mathbf{B}_0=(\mathbf{w}_0-\mathbf{w}^{*})\otimes(\mathbf{w}_0-\mathbf{w}^{*}) B 0 = ( w 0 − w ∗ ) ⊗ ( w 0 − w ∗ ) and C 0 = 0 \mathbf{C}_0=0 C 0 = 0 ). Letting η ˉ s : s + N b i a s : = 1 N ∑ t = s s + N − 1 η t b i a s \bar{\eta}_{s:s+N}^{\mathrm{bias}}:=\frac{1}{N}\sum_{t=s}^{s+N-1}\eta_t^{\mathrm{bias}} η ˉ s : s + N bias := N 1 ∑ t = s s + N − 1 η t bias and η ˉ s : s + N v a r i a n c e : = 1 N ∑ t = s s + N − 1 η t v a r i a n c e \bar{\eta}_{s:s+N}^{\mathrm{variance}}:=\frac{1}{N}\sum_{t=s}^{s+N-1}\eta_t^{\mathrm{variance}} η ˉ s : s + N variance := N 1 ∑ t = s s + N − 1 η t variance , Lemma C.6 (Jain et al., [15 ] ; Zou et al. [37 ] shows that E [ L ( w ˉ s : s + N ) ] − L ( w ∗ ) ≤ 2 ⋅ b i a s + 2 ⋅ v a r i a n c e \mathbb{E}[L(\bar{\mathbf{w}}_{s:s+N})]-L(\mathbf{w}^{*})\le2\cdot\mathrm{bias}+2\cdot\mathrm{variance} E [ L ( w ˉ s : s + N )] − L ( w ∗ ) ≤ 2 ⋅ bias + 2 ⋅ variance under Assumptions 3.1–3.3, and E [ L ( w ˉ s : s + N ) ] − L ( w ∗ ) = b i a s + v a r i a n c e \mathbb{E}[L(\bar{\mathbf{w}}_{s:s+N})]-L(\mathbf{w}^{*})=\mathrm{bias}+\mathrm{variance} E [ L ( w ˉ s : s + N )] − L ( w ∗ ) = bias + variance under the well-specified model (6) , where
b i a s : = 1 2 ⟨ H , E [ η ˉ s : s + N b i a s ⊗ η ˉ s : s + N b i a s ] ⟩ , v a r i a n c e : = 1 2 ⟨ H , E [ η ˉ s : s + N v a r i a n c e ⊗ η ˉ s : s + N v a r i a n c e ] ⟩ . (9) \mathrm{bias}:=\frac{1}{2}\langle\mathbf{H},\mathbb{E}[\bar{\eta}_{s:s+N}^{\mathrm{bias}}\otimes\bar{\eta}_{s:s+N}^{\mathrm{bias}}]\rangle,\qquad\mathrm{variance}:=\frac{1}{2}\langle\mathbf{H},\mathbb{E}[\bar{\eta}_{s:s+N}^{\mathrm{variance}}\otimes\bar{\eta}_{s:s+N}^{\mathrm{variance}}]\rangle. \tag*{(9)} bias := 2 1 ⟨ H , E [ η ˉ s : s + N bias ⊗ η ˉ s : s + N bias ]⟩ , variance := 2 1 ⟨ H , E [ η ˉ s : s + N variance ⊗ η ˉ s : s + N variance ]⟩ . (9)
Because the rank-one updates x t x t T \mathbf{x}_t\mathbf{x}_t^{\mathsf{T}} x t x t T in (7) fluctuate around H = E [ x t x t T ] \mathbf{H}=\mathbb{E}[\mathbf{x}_t\mathbf{x}_t^{\mathsf{T}}] H = E [ x t x t T ] , the bias process has mean E [ η t b i a s ] = ( I − γ H ) t ( w 0 − w ∗ ) \mathbb{E}[\eta_t^{\mathrm{bias}}]=(\mathbf{I}-\gamma\mathbf{H})^t(\mathbf{w}_0-\mathbf{w}^{*}) E [ η t bias ] = ( I − γ H ) t ( w 0 − w ∗ ) together with stochastic fluctuations η t b i a s − E [ η t b i a s ] \eta_t^{\mathrm{bias}}-\mathbb{E}[\eta_t^{\mathrm{bias}}] η t bias − E [ η t bias ] driven by multiplicative gradient noise ( x t x t T − H ) η t − 1 b i a s (\mathbf{x}_t\mathbf{x}_t^{\mathsf{T}}-\mathbf{H})\eta_{t-1}^{\mathrm{bias}} ( x t x t T − H ) η t − 1 bias . Following Zou et al. [37 ] and Li and Gu [20 ] , we therefore decompose b i a s + v a r i a n c e = E f f e c t i v e B i a s + E f f e c t i v e V a r \mathrm{bias}+\mathrm{variance}=\mathrm{EffectiveBias}+\mathrm{EffectiveVar} bias + variance = EffectiveBias + EffectiveVar into the deterministic drift bias EffectiveBias and the total stochastic variance error EffectiveVar:
E f f e c t i v e B i a s : = 1 2 E ∥ η ˉ s : s + N b i a s ∥ H 2 = ∑ i = 1 ∞ ( w 0 − w ∗ ) i 2 2 γ 2 N 2 λ i ( 1 − γ λ i ) 2 s ( 1 − ( 1 − γ λ i ) N ) 2 , (10) \mathrm{EffectiveBias}:=\frac{1}{2}\mathbb{E}\left\|\bar{\eta}_{s:s+N}^{\mathrm{bias}}\right\|_{\mathbf{H}}^{2}=\sum_{i=1}^{\infty}\frac{(\mathbf{w}_0-\mathbf{w}^{*})_i^{2}}{2\gamma^{2}N^{2}\lambda_i}(1-\gamma\lambda_i)^{2s}\left(1-(1-\gamma\lambda_i)^N\right)^2, \tag*{(10)} EffectiveBias := 2 1 E η ˉ s : s + N bias H 2 = i = 1 ∑ ∞ 2 γ 2 N 2 λ i ( w 0 − w ∗ ) i 2 ( 1 − γ λ i ) 2 s ( 1 − ( 1 − γ λ i ) N ) 2 , (10)
E f f e c t i v e V a r : = ( b i a s − E f f e c t i v e B i a s ) + v a r i a n c e = 1 2 E ∥ η ˉ s : s + N b i a s − E [ η ˉ s : s + N b i a s ] ∥ H 2 + v a r i a n c e , (11) \mathrm{EffectiveVar}:=(\mathrm{bias}-\mathrm{EffectiveBias})+\mathrm{variance}=\frac{1}{2}\mathbb{E}\left\|\bar{\eta}_{s:s+N}^{\mathrm{bias}}-\mathbb{E}[\bar{\eta}_{s:s+N}^{\mathrm{bias}}]\right\|_{\mathbf{H}}^{2}+\mathrm{variance}, \tag*{(11)} EffectiveVar := ( bias − EffectiveBias ) + variance = 2 1 E η ˉ s : s + N bias − E [ η ˉ s : s + N bias ] H 2 + variance , (11)
where E f f e c t i v e B i a s = 1 2 ∥ 1 N ∑ t = s s + N − 1 ( I − γ H ) t ( w 0 − w ∗ ) ∥ H 2 \mathrm{EffectiveBias}=\frac{1}{2}\left\|\frac{1}{N}\sum_{t=s}^{s+N-1}(\mathbf{I}-\gamma\mathbf{H})^t(\mathbf{w}_0-\mathbf{w}^{*})\right\|_{\mathbf{H}}^{2} EffectiveBias = 2 1 N 1 ∑ t = s s + N − 1 ( I − γ H ) t ( w 0 − w ∗ ) H 2 is the excess risk of deterministic gradient descent averaged over [ s , s + N − 1 ] [s,s+N-1] [ s , s + N − 1 ] . The effective variance in (11) is the sum of the bias-induced variance b i a s − E f f e c t i v e B i a s \mathrm{bias}-\mathrm{EffectiveBias} bias − EffectiveBias and the additive-noise variance v a r i a n c e \mathrm{variance} variance .
Main Results
In this section, we present our main upper and lower bounds on the excess risk E [ L ( w ‾ s : s + N ) ] − L ( w ∗ ) \mathbb{E}[L(\overline{w}_{s:s+N})]-L(w^*) E [ L ( w s : s + N )] − L ( w ∗ ) of SGD with tail-averaging (§ 4.1 ), establish concrete stepsize rules guaranteeing the spectral feedback condition for general and polynomially decaying spectra (Section 4.2), specialize our bounds to the canonical regime s = Θ ( N ) s=\Theta(N) s = Θ ( N ) to compare directly with Zou et al. [37 ] (Section 4.3), and present our separation constructions and conditional bounds (§ 4.5 ).
Before stating our theorems, recall from § 3.3 the deterministic drift bias EffectiveBias in (10) and the effective variance EffectiveVar in (11) , and let T : = s + N − 1 T:=s+N-1 T := s + N − 1 denote the final iteration index in (4) . For each step t ∈ { 1 , … , T } t\in\{1,\ldots,T\} t ∈ { 1 , … , T } and real lag h ≥ 0 h\ge0 h ≥ 0 , we define the tail-averaging filter polynomial q t ( λ ) q_t(\lambda) q t ( λ ) , the receiving sensitivity weight F t F_t F t , and the propagation kernel k ( h ) k(h) k ( h ) by1
q t ( λ ) : = 1 N ∑ m = s ∨ t T ( 1 − γ λ ) m − t , F t : = ∑ i = 1 ∞ λ i 2 q t ( λ i ) 2 , k ( h ) : = ∑ i = 1 ∞ λ i 2 ( 1 − γ λ i ) 2 h , (12) q_t(\lambda):=\frac{1}{N}\sum_{m=s\vee t}^{T}(1-\gamma\lambda)^{m-t},\qquad F_t:=\sum_{i=1}^{\infty}\lambda_i^2q_t(\lambda_i)^2,\qquad k(h):=\sum_{i=1}^{\infty}\lambda_i^2(1-\gamma\lambda_i)^{2h}, \tag*{(12)} q t ( λ ) := N 1 m = s ∨ t ∑ T ( 1 − γ λ ) m − t , F t := i = 1 ∑ ∞ λ i 2 q t ( λ i ) 2 , k ( h ) := i = 1 ∑ ∞ λ i 2 ( 1 − γ λ i ) 2 h , (12)
together with the spectral feedback coefficient
χ s , N : = α γ 2 max 1 ≤ r < T ∑ t = r + 1 T k ( t − r − 1 ) F t F r . (13) \chi_{s,N}:=\alpha\gamma^2\max_{1\le r<T}\frac{\sum_{t=r+1}^{T}k(t-r-1)F_t}{F_r}. \tag*{(13)} χ s , N := α γ 2 1 ≤ r < T max F r ∑ t = r + 1 T k ( t − r − 1 ) F t . (13)
Here F t F_t F t measures the sensitivity of w ‾ s : s + N \overline{w}_{s:s+N} w s : s + N to a stochastic gradient perturbation at step t t t , while χ s , N \chi_{s,N} χ s , N measures the worst-case single-hop amplification of fourth-moment energy across the trajectory (see § 5.1 for a detailed derivation).
Upper and Lower Bounds of Excess Risk
We first present our upper bound on the excess risk of SGD with tail-averaging.
Theorem 4.1 (Upper Bound). Consider SGD with tail-averaging ((4) ). Suppose Assumptions 3.1, 3.2, and 3.3 hold, and that the stepsize and spectral feedback coefficient satisfy γ ≤ 1 / ( 2 λ 1 ) \gamma\le1/(2\lambda_1) γ ≤ 1/ ( 2 λ 1 ) and χ s , N < 1 \chi_{s,N}<1 χ s , N < 1 . Then for any s ≥ 0 s\ge0 s ≥ 0 and N ≥ 1 N\ge1 N ≥ 1 , the excess risk can be upper bounded as follows:
E [ L ( w ‾ s : s + N ) ] − L ( w ∗ ) ≤ 2 ⋅ E f f e c t i v e B i a s + 2 ⋅ E f f e c t i v e V a r , \mathbb{E}\left[L(\overline{w}_{s:s+N})\right]-L(w^*)\le2\cdot\mathrm{EffectiveBias}+2\cdot\mathrm{EffectiveVar}, E [ L ( w s : s + N ) ] − L ( w ∗ ) ≤ 2 ⋅ EffectiveBias + 2 ⋅ EffectiveVar ,
where the factor of 2 can be omitted under the well-specified model (6) , the effective bias in (10) satisfies
E f f e c t i v e B i a s ≤ 1 2 γ 2 N 2 ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H 0 : k ∗ 2 + 1 2 ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 , (14) \mathrm{EffectiveBias}\le\frac{1}{2\gamma^2N^2}\left\|(I-\gamma H)^s(w_0-w^*)\right\|_{H_{0:k^*}}^2+\frac{1}{2}\left\|(I-\gamma H)^s(w_0-w^*)\right\|_{H_{k^*:\infty}}^2, \tag*{(14)} EffectiveBias ≤ 2 γ 2 N 2 1 ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H 0 : k ∗ 2 + 2 1 ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 , (14)
and the effective variance in (11) satisfies2
E f f e c t i v e V a r ≤ α ( ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ I 0 : k ∗ 2 + N γ ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 ) N γ ( 1 − χ s , N ) ( k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 ) + α 1 − χ s , N ∑ i , j = 1 ∞ min { 1 N 2 , γ 2 λ i 2 } λ j ( w 0 − w ∗ ) j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) s − 1 ∨ γ ∣ λ i − λ j ∣ . (15) \begin{aligned}
\mathrm{EffectiveVar}\le{}&
\frac{\alpha\left(\left\|(I-\gamma H)^s(w_0-w^*)\right\|_{I_{0:k^*}}^2+N\gamma\left\|(I-\gamma H)^s(w_0-w^*)\right\|_{H_{k^*:\infty}}^2\right)}
{N\gamma(1-\chi_{s,N})}
\left(\frac{k^*}{N}+N\gamma^2\sum_{i>k^*}\lambda_i^2\right)\\
&+\frac{\alpha}{1-\chi_{s,N}}\sum_{i,j=1}^{\infty}\min\left\{\frac{1}{N^2},\gamma^2\lambda_i^2\right\}
\frac{\lambda_j(w_0-w^*)_j^2(1-\gamma(\lambda_i\wedge\lambda_j))^{2(s-1)}}
{s^{-1}\vee\gamma|\lambda_i-\lambda_j|}. \tag*{(15)}
\end{aligned} EffectiveVar ≤ N γ ( 1 − χ s , N ) α ( ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ I 0 : k ∗ 2 + N γ ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 ) ( N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 ) + 1 − χ s , N α i , j = 1 ∑ ∞ min { N 2 1 , γ 2 λ i 2 } s − 1 ∨ γ ∣ λ i − λ j ∣ λ j ( w 0 − w ∗ ) j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) . (15)
+ σ 2 1 − χ s , N ( k ∗ N + γ ∑ k ∗ < i ≤ k † λ i + ( s + N ) γ 2 ∑ i > k † λ i 2 ) , (16) +\frac{\sigma^2}{1-\chi_{s,N}}\left(\frac{k^*}{N}+\gamma\sum_{k^*<i\le k^\dagger}\lambda_i+(s+N)\gamma^2\sum_{i>k^\dagger}\lambda_i^2\right), \tag*{(16)} + 1 − χ s , N σ 2 N k ∗ + γ k ∗ < i ≤ k † ∑ λ i + ( s + N ) γ 2 i > k † ∑ λ i 2 , (16)
where the eigenvalue cutoffs are defined as
k ∗ = max { k : λ k ≥ 1 γ N } , k † = max { k : λ k ≥ 1 γ ( s + N ) } . (17) k^*=\max\left\{k:\lambda_k\ge\frac{1}{\gamma N}\right\},\qquad k^\dagger=\max\left\{k:\lambda_k\ge\frac{1}{\gamma(s+N)}\right\}. \tag*{(17)} k ∗ = max { k : λ k ≥ γ N 1 } , k † = max { k : λ k ≥ γ ( s + N ) 1 } . (17)
We next establish our matching lower bound on the excess risk of SGD with tail-averaging:
Theorem 4.2 (Lower Bound). Consider SGD with tail-averaging (4) under the well-specified model (6) . Suppose Assumptions 3.1, 3.2, and 3.4 hold, and the hyperparameters satisfy γ ≤ 1 / ( 2 λ 1 ) \gamma\le1/(2\lambda_1) γ ≤ 1/ ( 2 λ 1 ) and N ≥ 2 N \ge2 N ≥ 2 . Then for any s ≥ 0 s \ge0 s ≥ 0 , the excess risk decomposes as
E [ L ( w ‾ s : s + N ) ] − L ( w ∗ ) = EffectiveBias + EffectiveVar , \mathbb{E}\left[L(\overline{\mathbf{w}}_{s:s+N})\right]-L(\mathbf{w}^*)=\operatorname{EffectiveBias}+\operatorname{EffectiveVar}, E [ L ( w s : s + N ) ] − L ( w ∗ ) = EffectiveBias + EffectiveVar , where the effective bias in (10) satisfies
EffectiveBias ≥ 1 8 γ 2 N 2 ⋅ ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H 0 : k ∗ 2 + 1 8 ⋅ ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 , (18) \operatorname{EffectiveBias}\ge\frac{1}{8\gamma^2N^2}\cdot\left\|(I-\gamma H)^s(\mathbf{w}_0-\mathbf{w}^*)\right\|_{H_{0:k^*}}^2+\frac{1}{8}\cdot\left\|(I-\gamma H)^s(\mathbf{w}_0-\mathbf{w}^*)\right\|_{H_{k^*:\infty}}^2, \tag*{(18)} EffectiveBias ≥ 8 γ 2 N 2 1 ⋅ ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H 0 : k ∗ 2 + 8 1 ⋅ ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 , (18) and the effective variance in (11) satisfies
EffectiveVar ≥ β ( ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ I 0 : k ∗ 2 + N γ ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 ) 864 N γ ( k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 ) + β 108 ∑ i , j = 1 ∞ min { 1 N 2 , γ 2 λ i 2 } λ j ( w 0 − w ∗ ) j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) s − 1 ∨ γ ∣ λ i − λ j ∣ + σ n o i s e 2 216 ( k ∗ N + γ ∑ k ∗ < i ≤ k † λ i + ( s + N ) γ 2 ∑ i > k † λ i 2 ) , (19) \begin{aligned}
\operatorname{EffectiveVar}\ge{}&\frac{\beta\left(\left\|(I-\gamma H)^s(\mathbf{w}_0-\mathbf{w}^*)\right\|_{I_{0:k^*}}^2+N\gamma\left\|(I-\gamma H)^s(\mathbf{w}_0-\mathbf{w}^*)\right\|_{H_{k^*:\infty}}^2\right)}{864N\gamma}\left(\frac{k^*}{N}+N\gamma^2\sum_{i>k^*}\lambda_i^2\right)\\
&+\frac{\beta}{108}\sum_{i,j=1}^{\infty}\min\left\{\frac{1}{N^2},\gamma^2\lambda_i^2\right\}\frac{\lambda_j(\mathbf{w}_0-\mathbf{w}^*)_j^2(1-\gamma(\lambda_i\wedge\lambda_j))^{2(s-1)}}{s^{-1}\vee\gamma|\lambda_i-\lambda_j|}\\
&+\frac{\sigma_{\mathrm{noise}}^2}{216}\left(\frac{k^*}{N}+\gamma\sum_{k^*<i\le k^\dagger}\lambda_i+(s+N)\gamma^2\sum_{i>k^\dagger}\lambda_i^2\right), \tag*{(19)}
\end{aligned} EffectiveVar ≥ 864 N γ β ( ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ I 0 : k ∗ 2 + N γ ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 ) ( N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 ) + 108 β i , j = 1 ∑ ∞ min { N 2 1 , γ 2 λ i 2 } s − 1 ∨ γ ∣ λ i − λ j ∣ λ j ( w 0 − w ∗ ) j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) + 216 σ noise 2 N k ∗ + γ k ∗ < i ≤ k † ∑ λ i + ( s + N ) γ 2 i > k † ∑ λ i 2 , (19) with k ∗ k^* k ∗ and k † k^\dagger k † defined in (17) .
Structure and Interpretation of the Bounds
The proofs of Theorems 4.1 and 4.2 are given in Appendix § 10 . The bounds on EffectiveBias and EffectiveVar separate the following contributions:
Effective Bias: In (14) and (18) , the initial error w 0 − w ∗ \mathbf{w}_0-\mathbf{w}^* w 0 − w ∗ is first contracted by ( I − γ H ) s (I-\gamma H)^s ( I − γ H ) s during the s s s burn-in steps, and then enjoys O ( ( γ 2 N 2 λ i ) − 1 ) O((\gamma^2N^2\lambda_i)^{-1}) O (( γ 2 N 2 λ i ) − 1 ) averaging decay on the head subspace I 0 : k ∗ I_{0:k^*} I 0 : k ∗ while remaining unaveraged on the tail subspace H k ∗ : ∞ H_{k^*:\infty} H k ∗ : ∞ .
Effective Variance: Both (4.4) and (19) consist of three matching components: (i) the averaging-window multiplicative variance (first line), which has the exact form of full-averaging SGD restarted from the burn-in drifted vector ( I − γ H ) s ( w 0 − w ∗ ) (I-\gamma H)^s(\mathbf{w}_0-\mathbf{w}^*) ( I − γ H ) s ( w 0 − w ∗ ) at step s s s ; (ii) the burn-in cross-spectral transfer variance (second line), where initial error in source mode j j j scatters fourth-moment energy into mode i i i during the s s s burn-in steps with effective interaction duration min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } \min\{s,(\gamma|\lambda_i-\lambda_j|)^{-1}\} min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } and slower-mode damping ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) (1-\gamma(\lambda_i\wedge\lambda_j))^{2(s-1)} ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) ; and (iii) the additive label-noise variance (third line), partitioned across i ≤ k ∗ i\le k^* i ≤ k ∗ , k ∗ < i ≤ k † k^*<i\le k^\dagger k ∗ < i ≤ k † , and i > k † i>k^\dagger i > k † .
Whenever χ s , N ≤ χ 0 < 1 \chi_{s,N}\le\chi_0<1 χ s , N ≤ χ 0 < 1 for a constant χ 0 < 1 \chi_0<1 χ 0 < 1 , every term in Theorems 4.1 and 4.2 matches up to a multiplicative factor depending only on α \alpha α , β \beta β , and 1 − χ 0 1-\chi_0 1 − χ 0 across all dimensions, spectra, initializations, and horizons s ≥ 0 s\ge0 s ≥ 0 , N ≥ 2 N\ge2 N ≥ 2 . Moreover, when s = 0 s=0 s = 0 , the burn-in sum vanishes and Theorems 4.1 and 4.2 recover Zou et al. [37 ] (while extending the lower bound from N ≥ 500 N\ge500 N ≥ 500 to all N ≥ 2 N\ge2 N ≥ 2 ).
Guaranteeing the Spectral Feedback Condition χ s , N ≤ χ 0 < 1 \chi_{s,N} \le\chi_0 < 1 χ s , N ≤ χ 0 < 1
We now show how the spectral feedback condition χ s , N ≤ 1 / 2 \chi_{s,N} \le1/2 χ s , N ≤ 1/2 can be guaranteed prior to execution, both for arbitrary covariance spectra H \mathbf{H} H (under a horizon-dependent constant stepsize) and for polynomially decaying spectra λ i ≍ i − a \lambda_i \asymp i^{-a} λ i ≍ i − a (under a single horizon-independent constant stepsize). Recall k ( h ) = ∑ i ≥ 1 λ i 2 ( 1 − γ λ i ) 2 h k(h) = \sum_{i\ge1} \lambda_i^2(1-\gamma\lambda_i)^{2h} k ( h ) = ∑ i ≥ 1 λ i 2 ( 1 − γ λ i ) 2 h for real h ≥ 0 h \ge0 h ≥ 0 , and define its doubling constant over [ 1 , T ] [1,T] [ 1 , T ] by
D T : = sup 1 ≤ h ≤ T k ( ( h − 1 ) / 2 ) k ( h ) . (20) D_T := \sup_{1\le h\le T} \frac{k((h-1)/2)}{k(h)}. \tag*{(20)} D T := 1 ≤ h ≤ T sup k ( h ) k (( h − 1 ) /2 ) . (20)
Proposition 4.3 (Pair-Midpoint Convolution Bounds on χ s , N \chi_{s,N} χ s , N and General-Spectrum Stepsize Rule). Suppose Assumption 3.1 holds and 0 < γ < 1 / λ 1 0 < \gamma< 1/\lambda_1 0 < γ < 1/ λ 1 . Then for every s ≥ 0 s \ge0 s ≥ 0 and N ≥ 1 N \ge1 N ≥ 1 (with T = s + N − 1 ≥ 0 T = s + N - 1 \ge0 T = s + N − 1 ≥ 0 ), the receiving weights F t F_t F t in (4.1) satisfy the exact pair-midpoint identity
F t = 1 N 2 ∑ a = s ∨ t T ∑ b = s ∨ t T k ( a + b 2 − t ) , ∀ t ∈ { 1 , … , T } . (21) F_t = \frac{1}{N^2}\sum_{a=s\vee t}^{T}\sum_{b=s\vee t}^{T} k\left(\frac{a+b}{2}-t\right), \qquad\forall t \in\{1,\ldots,T\}. \tag*{(21)} F t = N 2 1 a = s ∨ t ∑ T b = s ∨ t ∑ T k ( 2 a + b − t ) , ∀ t ∈ { 1 , … , T } . (21) Consequently, χ s , N = 0 \chi_{s,N}=0 χ s , N = 0 whenever T ≤ 1 T\le1 T ≤ 1 , and for all T = s + N − 1 ≥ 2 T=s+N-1\ge2 T = s + N − 1 ≥ 2 , the spectral feedback coefficient χ s , N \chi_{s,N} χ s , N in (4.2) satisfies both:
(General-Spectrum Horizon-Dependent Bound): For any covariance operator H \mathbf{H} H and T ≥ 2 T\ge2 T ≥ 2 ,
χ s , N ≤ α γ 2 ( T − 1 ) tr ( H 2 ) ( 1 − γ λ 1 ) 2 , (22) \chi_{s,N} \le\frac{\alpha\gamma^2(T-1)\operatorname{tr}(\mathbf{H}^2)}{(1-\gamma\lambda_1)^2}, \tag*{(22)} χ s , N ≤ ( 1 − γ λ 1 ) 2 α γ 2 ( T − 1 ) tr ( H 2 ) , (22) so choosing any constant pre-run stepsize satisfying
γ ≤ min { 1 2 λ 1 , 1 8 α ( T − 1 ) tr ( H 2 ) } ⟹ χ s , N ≤ 1 2 . (23) \gamma\le\min\left\{\frac{1}{2\lambda_1},\frac{1}{\sqrt{8\alpha(T-1)\operatorname{tr}(\mathbf{H}^2)}}\right\} \quad\Longrightarrow\quad\chi_{s,N}\le\frac{1}{2}. \tag*{(23)} γ ≤ min { 2 λ 1 1 , 8 α ( T − 1 ) tr ( H 2 ) 1 } ⟹ χ s , N ≤ 2 1 . (23) (Kernel-Doubling Bound): In terms of the doubling constant D T D_T D T in (4.8),
χ s , N ≤ 2 α D T γ 2 ∑ n = 0 ∞ k ( n ) = 2 α D T γ 2 ∑ i = 1 ∞ λ i 2 − γ λ i ≤ 2 α D T γ 2 tr ( H ) 2 − γ λ 1 . (24) \chi_{s,N} \le2\alpha D_T\gamma^2\sum_{n=0}^{\infty}k(n)=2\alpha D_T\gamma^2\sum_{i=1}^{\infty}\frac{\lambda_i}{2-\gamma\lambda_i}\le\frac{2\alpha D_T\gamma^2\operatorname{tr}(\mathbf{H})}{2-\gamma\lambda_1}. \tag*{(24)} χ s , N ≤ 2 α D T γ 2 n = 0 ∑ ∞ k ( n ) = 2 α D T γ 2 i = 1 ∑ ∞ 2 − γ λ i λ i ≤ 2 − γ λ 1 2 α D T γ 2 tr ( H ) . (24)
Proposition 4.3 (1) guarantees χ s , N ≤ 1 / 2 \chi_{s,N}\le1/2 χ s , N ≤ 1/2 (and hence matching upper and lower bounds in Theorems 4.1 and 4.2 ) for arbitrary spectra H \mathbf{H} H under the horizon-dependent constant stepsize (4.11). Moreover, when the eigenvalues of H \mathbf{H} H decay polynomially as λ i ≍ i − a \lambda_i\asymp i^{-a} λ i ≍ i − a (a > 1 a>1 a > 1 )—the standard power-law family studied in Caponnetto and De Vito [5 ] ; Dieuleveut and Bach [8 ] ; Bartlett et al. [2 ] ; Zou et al. [37 ] ; Li and Gu [20 ] —the propagation kernel decays polynomially as k ( h ) ≍ Λ 2 ( 1 + γ Λ h ) − ( 2 − 1 / a ) k(h)\asymp\Lambda^2(1+\gamma\Lambda h)^{-(2-1/a)} k ( h ) ≍ Λ 2 ( 1 + γ Λ h ) − ( 2 − 1/ a ) . In this regime, Proposition 4.3 (2) strengthens (4.11) to a single horizon-independent stepsize γ \gamma γ ensuring χ s , N ≤ 1 / 2 \chi_{s,N}\le1/2 χ s , N ≤ 1/2 across all s s s and N N N .
Corollary 4.4 (Uniform Horizon-Free Matching under Polynomial Eigenvalue Decay). Suppose there exist constants a > 1 a>1 a > 1 , Λ > 0 \Lambda>0 Λ > 0 , and θ ∈ ( 0 , 1 ] \theta\in(0,1] θ ∈ ( 0 , 1 ] such that θ Λ i − a ≤ λ i ≤ Λ i − a \theta\Lambda i^{-a}\le\lambda_i\le\Lambda i^{-a} θ Λ i − a ≤ λ i ≤ Λ i − a for all i ≥ 1 i\ge1 i ≥ 1 (in infinite dimension) or for 1 ≤ i ≤ d 1\le i\le d 1 ≤ i ≤ d (in finite dimension d ≥ 2 d\ge2 d ≥ 2 , over horizons γ Λ T ≤ ( d / 2 ) a \gamma\Lambda T\le(d/2)^a γ Λ T ≤ ( d /2 ) a ). Then there exists a finite constant D ∗ = D ∗ ( a , θ ) ≥ 1 D_*=D_*(a,\theta)\ge1 D ∗ = D ∗ ( a , θ ) ≥ 1 such that γ Λ ≤ 1 / 2 \gamma\Lambda\le1/2 γ Λ ≤ 1/2 implies D T ≤ D ∗ D_T\le D_* D T ≤ D ∗ , and hence any fixed stepsize satisfying
γ ≤ min { 1 2 Λ , 1 4 α D ∗ tr ( H ) } (25) \gamma\le\min\left\{\frac{1}{2\Lambda},\frac{1}{4\alpha D_*\operatorname{tr}(\mathbf{H})}\right\} \tag*{(25)} γ ≤ min { 2Λ 1 , 4 α D ∗ tr ( H ) 1 } (25) ensures χ s , N ≤ 1 / 2 \chi_{s,N}\le1/2 χ s , N ≤ 1/2 . Consequently, under (3.4) and Assumptions 3.1–3.4, the upper bound of Theorem 4.1 and the lower bound of Theorem 4.2 match up to constants depending only on α \alpha α and β \beta β uniformly over all w 0 − w ∗ \mathbf{w}_0-\mathbf{w}^* w 0 − w ∗ , s ≥ 0 s\ge0 s ≥ 0 , and N ≥ 2 N\ge2 N ≥ 2 (and Corollary 4.5 below holds whenever c 0 N ≤ s ≤ C 0 N c_0N\le s\le C_0N c 0 N ≤ s ≤ C 0 N up to constants depending only on α , β , c 0 , C 0 \alpha,\beta,c_0,C_0 α , β , c 0 , C 0 ).
Canonical Regime s = Θ ( N ) s = \Theta(N) s = Θ ( N ) and Comparison with Zou et al. [37 ]
In the standard tail-averaging regime where s = Θ ( N ) s = \Theta(N) s = Θ ( N ) (for instance, s = N s = N s = N , which averages the second half of a 2 N 2N 2 N -step run as in Jain et al. [14 ] and Zou et al. [37 ] ), the eigenvalue cutoffs k ∗ k^* k ∗ and k † k^\dagger k † have thresholds ( γ N ) − 1 (\gamma N)^{-1} ( γ N ) − 1 and ( γ ( s + N ) ) − 1 (\gamma(s+N))^{-1} ( γ ( s + N ) ) − 1 within a constant factor of each other, so ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 ≍ ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 \lVert w_0-w^*\rVert^2_{H_{k^*:\infty}} \asymp\lVert(I-\gamma H)^s(w_0-w^*)\rVert^2_{H_{k^*:\infty}} ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 ≍ ∥( I − γ H ) s ( w 0 − w ∗ ) ∥ H k ∗ : ∞ 2 and γ λ i ≍ N γ 2 λ i 2 ≍ N − 1 \gamma\lambda_i \asymp N\gamma^2\lambda_i^2 \asymp N^{-1} γ λ i ≍ N γ 2 λ i 2 ≍ N − 1 for all k ∗ < i ≤ k † k^* < i \le k^\dagger k ∗ < i ≤ k † . Splitting the burn-in variance sum in (4.4) and (19) into fast index modes (i ≤ k ∗ i \le k^* i ≤ k ∗ , where min { N − 2 , γ 2 λ i 2 } = N − 2 \min\{N^{-2},\gamma^2\lambda_i^2\}=N^{-2} min { N − 2 , γ 2 λ i 2 } = N − 2 ) and slow index modes (i > k ∗ i > k^* i > k ∗ , where min { N − 2 , γ 2 λ i 2 } = γ 2 λ i 2 \min\{N^{-2},\gamma^2\lambda_i^2\}=\gamma^2\lambda_i^2 min { N − 2 , γ 2 λ i 2 } = γ 2 λ i 2 ) yields the following corollary and the term-by-term comparison in Table 1 .
Corollary 4.5 (Matching Excess Risk Bounds when s = Θ ( N ) s = \Theta(N) s = Θ ( N ) ). Suppose c 0 N ≤ s ≤ C 0 N c_0N \le s \le C_0N c 0 N ≤ s ≤ C 0 N for constants 0 < c 0 ≤ C 0 < ∞ 0 < c_0 \le C_0 < \infty 0 < c 0 ≤ C 0 < ∞ , and suppose the assumptions of Theorems 4.1 and 4.2 hold with χ s , N ≤ χ 0 < 1 \chi_{s,N} \le\chi_0 < 1 χ s , N ≤ χ 0 < 1 . Then the excess risk satisfies
EffectiveBias = Θ ( 1 γ 2 N 2 ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ H 0 : k ∗ 2 + ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 ) , (26) \operatorname{EffectiveBias} = \Theta\left(\frac{1}{\gamma^2N^2}\lVert(I-\gamma H)^s(w_0-w^*)\rVert^2_{H_{0:k^*}}+\lVert w_0-w^*\rVert^2_{H_{k^*:\infty}}\right), \tag*{(26)} EffectiveBias = Θ ( γ 2 N 2 1 ∥( I − γ H ) s ( w 0 − w ∗ ) ∥ H 0 : k ∗ 2 + ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 ) , (26) EffectiveVar = Θ ( ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 + N γ ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 N γ ⋅ ( N γ 2 ∑ i > k ∗ λ i 2 ) + ∥ ( I − γ H ) s ( w 0 − w ∗ ) ∥ I 0 : k ∗ 2 + N γ ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 N γ ⋅ k ∗ N + 1 N 2 ∑ i ≤ k ∗ ∑ j ≤ k ∗ λ j ( w 0 − w ∗ ) j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) min { s , 1 γ ∣ λ i − λ j ∣ } + σ n o i s e 2 ⋅ ( k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 ) ) . (27) \begin{aligned}
\operatorname{EffectiveVar} = \Theta\Biggl(&\frac{\lVert w_0-w^*\rVert^2_{I_{0:k^*}}+N\gamma\lVert w_0-w^*\rVert^2_{H_{k^*:\infty}}}{N\gamma}\cdot\left(N\gamma^2\sum_{i>k^*}\lambda_i^2\right)\\
&+\frac{\lVert(I-\gamma H)^s(w_0-w^*)\rVert^2_{I_{0:k^*}}+N\gamma\lVert w_0-w^*\rVert^2_{H_{k^*:\infty}}}{N\gamma}\cdot\frac{k^*}{N}\\
&+\frac{1}{N^2}\sum_{i\le k^*}\sum_{j\le k^*}\lambda_j(w_0-w^*)_j^2(1-\gamma(\lambda_i\wedge\lambda_j))^{2(s-1)}\min\left\{s,\frac{1}{\gamma|\lambda_i-\lambda_j|}\right\}\\
&+\sigma_{\mathrm{noise}}^2\cdot\left(\frac{k^*}{N}+N\gamma^2\sum_{i>k^*}\lambda_i^2\right)\Biggr). \tag*{(27)}
\end{aligned} EffectiveVar = Θ ( N γ ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 + N γ ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 ⋅ ( N γ 2 i > k ∗ ∑ λ i 2 ) + N γ ∥( I − γ H ) s ( w 0 − w ∗ ) ∥ I 0 : k ∗ 2 + N γ ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 ⋅ N k ∗ + N 2 1 i ≤ k ∗ ∑ j ≤ k ∗ ∑ λ j ( w 0 − w ∗ ) j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) min { s , γ ∣ λ i − λ j ∣ 1 } + σ noise 2 ⋅ ( N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 ) ) . (27) where the constants hidden in Θ ( ⋅ ) \Theta(\cdot) Θ ( ⋅ ) depend only on α \alpha α , β \beta β , 1 − χ 0 1-\chi_0 1 − χ 0 , c 0 c_0 c 0 , and C 0 C_0 C 0 .
Remark 4.6 (Resolution of the Total-Risk Gap in Zou et al. [37 ] and Role of Multi-Hop Return Paths). Comparing (27) with Zou et al. [37 ] (see Table 1 ) clarifies both why the rank-one factored form in prior work could not match and how the terms of (27) combine in the total excess risk:
Presence of Slow Spectral Modes (k ∗ < d k^* < d k ∗ < d , so ∑ i > k ∗ λ i 2 > 0 \sum_{i>k^*}\lambda_i^2 > 0 ∑ i > k ∗ λ i 2 > 0 ): In Panel B (i > k ∗ i > k^* i > k ∗ ), the first line of (27) retains the undamped head initialization contribution 1 γ N ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 ⋅ ( N γ 2 ∑ i > k ∗ λ i 2 ) \frac{1}{\gamma N}\lVert w_0-w^*\rVert^2_{I_{0:k^*}}\cdot\left(N\gamma^2\sum_{i>k^*}\lambda_i^2\right) γ N 1 ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 ⋅ ( N γ 2 ∑ i > k ∗ λ i 2 ) , which is missing from Zou et al. [37 ] . In multi-dimensional problems with k ∗ < d k^* < d k ∗ < d , energy scattered into slow modes i > k ∗ i > k^* i > k ∗ can also scatter back into fast modes i ≤ k ∗ i \le k^* i ≤ k ∗ ; under χ s , N ≤ χ 0 < 1 \chi_{s,N} \le\chi_0 < 1 χ s , N ≤ χ 0 < 1 , this multi-hop return contribution is absorbed into the total-risk bound via ( 1 − χ s , N ) − 1 (1-\chi_{s,N})^{-1} ( 1 − χ s , N ) − 1 multiplying the full single-hop expression in (27) .3
Well-Resolved Spectral Regime (d ≤ k ∗ d \le k^* d ≤ k ∗ , so H k ∗ : ∞ = 0 H_{k^*:\infty}=0 H k ∗ : ∞ = 0 ): Conversely, when all eigenvalues satisfy λ i ≥ ( γ N ) − 1 \lambda_i \ge(\gamma N)^{-1} λ i ≥ ( γ N ) − 1 , the slow subspace is empty (∑ i > k ∗ λ i 2 = 0 \sum_{i>k^*}\lambda_i^2=0 ∑ i > k ∗ λ i 2 = 0 ), so Panel B vanishes and no slow-mode return path exists. Every remaining bias-induced variance term in Panel A (the second and third lines of (27) ) is exponentially damped in s s s by ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) (1-\gamma(\lambda_i\wedge\lambda_j))^{2(s-1)} ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) , eliminating the undamped 1 γ N ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 ⋅ k ∗ N \frac{1}{\gamma N}\lVert w_0-w^*\rVert^2_{I_{0:k^*}}\cdot\frac{k^*}{N} γ N 1 ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 ⋅ N k ∗ artifact in Zou et al. [37 ] .
Sharp Separation Examples and Conditional Averaging-Window Bounds
We conclude Section 4 4 4 with two explicit separation constructions (Proposition 4.7 4.7 4.7 ) that demonstrate (i) an exponential-versus-polynomial gap between our lower bound in Theorem 4.2 4.2 4.2 and Zou et al. [37 ] , and (ii) why fourth-moment assumptions alone cannot determine the asymptotic exponential burn-in rate without an additional structural condition (such as χ s , N ≤ χ 0 < 1 \chi_{s,N}\le\chi_0<1 χ s , N ≤ χ 0 < 1 ) or state information (such as B s B_s B s in Proposition 4.8 4.8 4.8 ).
Proposition 4.7 (Sharp Separation Examples for Cross-Spectral Transfer and Burn-In Decay).
(Exponential-vs.-Polynomial Gap in the Lower Bound of Zou et al. [37 ] ): Fix 0 < γ ≤ 1 / 6 0<\gamma\le1/6 0 < γ ≤ 1/6 and integer N ≥ max { 500 , 1 / γ } N\ge\max\{500,1/\gamma\} N ≥ max { 500 , 1/ γ } . For x ∼ N ( 0 , H ) x\sim\mathcal{N}(0,\mathbf{H}) x ∼ N ( 0 , H ) in R 2 \mathbb{R}^2 R 2 with H = diag ( 1 , ( 4 γ N ) − 1 ) \mathbf{H}=\operatorname{diag}(1,(4\gamma N)^{-1}) H = diag ( 1 , ( 4 γ N ) − 1 ) , y = ⟨ w ∗ , x ⟩ y=\langle\mathbf{w}^*,x\rangle y = ⟨ w ∗ , x ⟩ (σ n o i s e = 0 \sigma_{\mathrm{noise}}=0 σ noise = 0 ), w 0 − w ∗ = ( 1 , 0 ) ⊤ \mathbf{w}_0-\mathbf{w}^*=(1,0)^\top w 0 − w ∗ = ( 1 , 0 ) ⊤ , and s = N s=N s = N , constant-stepsize tail-averaged SGD satisfies
E [ L ( w ‾ N : 2 N ) ] − L ( w ∗ ) ≥ 1 1024 γ N 2 , (28) \mathbb{E}\left[L(\overline{\mathbf{w}}_{N:2N})\right]-L(\mathbf{w}^*)\ge\frac{1}{1024\gamma N^2}, \tag*{(28)} E [ L ( w N : 2 N ) ] − L ( w ∗ ) ≥ 1024 γ N 2 1 , (28) whereas Zou et al. [37 ] yields the exponentially small lower bound ( 1 − γ ) 2 N / ( 100 γ 2 N 2 ) (1-\gamma)^{2N}/(100\gamma^2N^2) ( 1 − γ ) 2 N / ( 100 γ 2 N 2 ) .
(Fourth Moments Shift Scalar Exponential Decay When χ s , N \chi_{s,N} χ s , N Is Uncontrolled): Fix 0 < γ < 1 / 3 0<\gamma<1/3 0 < γ < 1/3 and y = 0 y=0 y = 0 , w ∗ = 0 \mathbf{w}^*=0 w ∗ = 0 , w 0 = 1 \mathbf{w}_0=1 w 0 = 1 , H = 1 \mathbf{H}=1 H = 1 . Let D 1 D_1 D 1 be the three-point distribution P ( x = 0 ) = 1 / 2 \mathbb{P}(x=0)=1/2 P ( x = 0 ) = 1/2 , P ( x = ± 2 ) = 1 / 4 \mathbb{P}(x=\pm\sqrt{2})=1/4 P ( x = ± 2 ) = 1/4 (where E [ x 2 ] = 1 \mathbb{E}[x^2]=1 E [ x 2 ] = 1 , E [ x 4 ] = 2 \mathbb{E}[x^4]=2 E [ x 4 ] = 2 ) and let D 2 D_2 D 2 be N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) (where E [ x 2 ] = 1 \mathbb{E}[x^2]=1 E [ x 2 ] = 1 , E [ x 4 ] = 3 \mathbb{E}[x^4]=3 E [ x 4 ] = 3 ). Both satisfy Assumptions 3.1, 3.2, and 3.4 with α = 3 \alpha=3 α = 3 and β = 1 \beta=1 β = 1 , yet their exact
tail-averaged risks R κ ( s , N ) R_{\kappa}(s,N) R κ ( s , N ) (κ ∈ { 2 , 3 } \kappa\in\{2,3\} κ ∈ { 2 , 3 } ) satisfy R κ ( s , N ) ≍ γ r κ s / N 2 R_{\kappa}(s,N) \asymp_{\gamma} r_{\kappa}^{s}/N^{2} R κ ( s , N ) ≍ γ r κ s / N 2 with r κ = 1 − 2 γ + κ γ 2 r_{\kappa}=1-2\gamma+\kappa\gamma^{2} r κ = 1 − 2 γ + κ γ 2 , so
R 3 ( N , N ) R 2 ( N , N ) ≳ γ ( 1 − 2 γ + 3 γ 2 1 − 2 γ + 2 γ 2 ) N ⟶ ∞ as N → ∞ . (29) \frac{R_{3}(N,N)}{R_{2}(N,N)} \gtrsim_{\gamma} \left(\frac{1-2\gamma+3\gamma^{2}}{1-2\gamma+2\gamma^{2}}\right)^{N} \longrightarrow\infty\qquad\text{as } N \to\infty. \tag*{(29)} R 2 ( N , N ) R 3 ( N , N ) ≳ γ ( 1 − 2 γ + 2 γ 2 1 − 2 γ + 3 γ 2 ) N ⟶ ∞ as N → ∞. (29)
Consequently, no single expression depending only on ( H , w 0 − w ∗ , γ , s , N , α , β , σ n o i s e 2 ) (\mathbf{H},\mathbf{w}_{0}-\mathbf{w}^{*},\gamma,s,N,\alpha,\beta,\sigma_{\mathrm{noise}}^{2}) ( H , w 0 − w ∗ , γ , s , N , α , β , σ noise 2 ) can match both risks uniformly across N N N without an additional structural condition (such as χ s , N ≤ χ 0 < 1 \chi_{s,N}\leq\chi_{0}<1 χ s , N ≤ χ 0 < 1 ) or conditioning on B s \mathbf{B}_{s} B s .
When χ s , N ≥ 1 \chi_{s,N}\geq1 χ s , N ≥ 1 (as in the large-s s s scalar regime of Proposition 4.7(2)), conditioning at the start of the averaging window t = s t=s t = s still yields upper and lower bounds in terms of the burn-in bias covariance B s = E [ η s b i a s ⊗ η s b i a s ] \mathbf{B}_{s}=\mathbb{E}[\eta_{s}^{\mathrm{bias}}\otimes\eta_{s}^{\mathrm{bias}}] B s = E [ η s bias ⊗ η s bias ] from § 3.3 that match up to constants depending only on α \alpha α , β \beta β , and 1 − γ α tr ( H ) 1-\gamma\alpha\operatorname{tr}(\mathbf{H}) 1 − γ α tr ( H ) for all s ≥ 0 s\geq0 s ≥ 0 and N ≥ 2 N\geq2 N ≥ 2 (recovering at the level of orders the result of restarting the full-averaging bounds of Zou et al. [37 ] at B s \mathbf{B}_{s} B s while extending coverage to all N ≥ 2 N\geq2 N ≥ 2 ).
Proposition 4.8 (Conditional Upper and Lower Bounds in Terms of B s \mathbf{B}_{s} B s ). Under Assumptions 3.1–3.3 with γ ≤ 1 / ( 2 α tr ( H ) ) \gamma\leq1/(2\alpha\operatorname{tr}(\mathbf{H})) γ ≤ 1/ ( 2 α tr ( H )) , for all s ≥ 0 s\geq0 s ≥ 0 and N ≥ 1 N\geq1 N ≥ 1 , the excess risk satisfies
E [ L ( w ‾ s : s + N ) ] − L ( w ∗ ) ≤ 1 γ 2 N 2 tr ( H 0 : k ∗ − 1 B s ) + tr ( H k ∗ : ∞ B s ) \mathbb{E}\left[L(\overline{\mathbf{w}}_{s:s+N})\right]-L(\mathbf{w}^{*}) \leq\frac{1}{\gamma^{2}N^{2}}\operatorname{tr}\left(\mathbf{H}_{0:k^{*}}^{-1}\mathbf{B}_{s}\right)+\operatorname{tr}\left(\mathbf{H}_{k^{*}:\infty}\mathbf{B}_{s}\right) E [ L ( w s : s + N ) ] − L ( w ∗ ) ≤ γ 2 N 2 1 tr ( H 0 : k ∗ − 1 B s ) + tr ( H k ∗ : ∞ B s )
+ 2 α 1 − γ α tr ( H ) ⋅ tr ( I 0 : k ∗ B s ) + N γ tr ( H k ∗ : ∞ B s ) N γ ⋅ ( k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 ) {}+\frac{2\alpha}{1-\gamma\alpha\operatorname{tr}(\mathbf{H})}\cdot\frac{\operatorname{tr}(\mathbf{I}_{0:k^{*}}\mathbf{B}_{s})+N\gamma\operatorname{tr}(\mathbf{H}_{k^{*}:\infty}\mathbf{B}_{s})}{N\gamma}\cdot\left(\frac{k^{*}}{N}+N\gamma^{2}\sum_{i>k^{*}}\lambda_{i}^{2}\right) + 1 − γ α tr ( H ) 2 α ⋅ N γ tr ( I 0 : k ∗ B s ) + N γ tr ( H k ∗ : ∞ B s ) ⋅ ( N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 )
+ 2 σ 2 1 − γ α tr ( H ) ⋅ ( k ∗ N + γ ∑ k ∗ < i ≤ k † λ i + ( s + N ) γ 2 ∑ i > k † λ i 2 ) . (30) {}+\frac{2\sigma^{2}}{1-\gamma\alpha\operatorname{tr}(\mathbf{H})}\cdot\left(\frac{k^{*}}{N}+\gamma\sum_{k^{*}<i\leq k^{\dagger}}\lambda_{i}+(s+N)\gamma^{2}\sum_{i>k^{\dagger}}\lambda_{i}^{2}\right). \tag*{(30)} + 1 − γ α tr ( H ) 2 σ 2 ⋅ N k ∗ + γ k ∗ < i ≤ k † ∑ λ i + ( s + N ) γ 2 i > k † ∑ λ i 2 . (30)
and under (3.4) and Assumptions 3.1, 3.2, and 3.4 with γ ≤ 1 / ( 2 λ 1 ) \gamma\leq1/(2\lambda_{1}) γ ≤ 1/ ( 2 λ 1 ) and N ≥ 2 N\geq2 N ≥ 2 ,
E [ L ( w ‾ s : s + N ) ] − L ( w ∗ ) ≥ 1 8 γ 2 N 2 tr ( H 0 : k ∗ − 1 B s ) + 1 8 tr ( H k ∗ : ∞ B s ) \mathbb{E}\left[L(\overline{\mathbf{w}}_{s:s+N})\right]-L(\mathbf{w}^{*}) \geq\frac{1}{8\gamma^{2}N^{2}}\operatorname{tr}\left(\mathbf{H}_{0:k^{*}}^{-1}\mathbf{B}_{s}\right)+\frac{1}{8}\operatorname{tr}\left(\mathbf{H}_{k^{*}:\infty}\mathbf{B}_{s}\right) E [ L ( w s : s + N ) ] − L ( w ∗ ) ≥ 8 γ 2 N 2 1 tr ( H 0 : k ∗ − 1 B s ) + 8 1 tr ( H k ∗ : ∞ B s )
+ β 864 ⋅ tr ( I 0 : k ∗ B s ) + N γ tr ( H k ∗ : ∞ B s ) N γ ⋅ ( k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 ) {}+\frac{\beta}{864}\cdot\frac{\operatorname{tr}(\mathbf{I}_{0:k^{*}}\mathbf{B}_{s})+N\gamma\operatorname{tr}(\mathbf{H}_{k^{*}:\infty}\mathbf{B}_{s})}{N\gamma}\cdot\left(\frac{k^{*}}{N}+N\gamma^{2}\sum_{i>k^{*}}\lambda_{i}^{2}\right) + 864 β ⋅ N γ tr ( I 0 : k ∗ B s ) + N γ tr ( H k ∗ : ∞ B s ) ⋅ ( N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 )
+ σ n o i s e 2 216 ⋅ ( k ∗ N + γ ∑ k ∗ < i ≤ k † λ i + ( s + N ) γ 2 ∑ i > k † λ i 2 ) . (31) {}+\frac{\sigma_{\mathrm{noise}}^{2}}{216}\cdot\left(\frac{k^{*}}{N}+\gamma\sum_{k^{*}<i\leq k^{\dagger}}\lambda_{i}+(s+N)\gamma^{2}\sum_{i>k^{\dagger}}\lambda_{i}^{2}\right). \tag*{(31)} + 216 σ noise 2 ⋅ N k ∗ + γ k ∗ < i ≤ k † ∑ λ i + ( s + N ) γ 2 i > k † ∑ λ i 2 . (31)
Overview of Proof Techniques
In this section, we present the analytical framework underlying Theorems 4.1 and 4.2 , Proposition 4.3 , Corollary 4.4 , and Propositions 4.7–4.8. Recall from § 3.3 the bias and variance error processes { η t b i a s } t ≥ 0 \{\eta_{t}^{\mathrm{bias}}\}_{t\geq0} { η t bias } t ≥ 0 and { η t v a r i a n c e } t ≥ 0 \{\eta_{t}^{\mathrm{variance}}\}_{t\geq0} { η t variance } t ≥ 0 in (3.5)–(3.6) and their single-step second-moment operators B t = E [ η t b i a s ⊗ η t b i a s ] \mathbf{B}_{t}=\mathbb{E}[\eta_{t}^{\mathrm{bias}}\otimes\eta_{t}^{\mathrm{bias}}] B t = E [ η t bias ⊗ η t bias ] and C t = E [ η t v a r i a n c e ⊗ η t v a r i a n c e ] \mathbf{C}_{t}=\mathbb{E}[\eta_{t}^{\mathrm{variance}}\otimes\eta_{t}^{\mathrm{variance}}] C t = E [ η t variance ⊗ η t variance ] . Following Jain et al. [15 ] and Zou et al. [37 ] , we define the linear transition super-operators T \mathcal{T} T and T ~ \widetilde{\mathcal{T}} T acting on symmetric operators A \mathbf{A} A by
T : = H ⊗ I + I ⊗ H − γ M , T ~ : = H ⊗ I + I ⊗ H − γ M ~ , (32) \mathcal{T}:=\mathbf{H}\otimes\mathbf{I}+\mathbf{I}\otimes\mathbf{H}-\gamma\mathcal{M},\qquad\widetilde{\mathcal{T}}:=\mathbf{H}\otimes\mathbf{I}+\mathbf{I}\otimes\mathbf{H}-\gamma\widetilde{\mathcal{M}}, \tag*{(32)} T := H ⊗ I + I ⊗ H − γ M , T := H ⊗ I + I ⊗ H − γ M , (32)
so that ( I − γ T ) ∘ A = E [ ( I − γ x x T ) A ( I − γ x x T ) ] (\mathbf{I}-\gamma\mathcal{T})\circ\mathbf{A}=\mathbb{E}[(\mathbf{I}-\gamma\mathbf{x}\mathbf{x}^{\mathsf{T}})\mathbf{A}(\mathbf{I}-\gamma\mathbf{x}\mathbf{x}^{\mathsf{T}})] ( I − γ T ) ∘ A = E [( I − γ x x T ) A ( I − γ x x T )] is the single-step stochastic covariance transition and ( I − γ T ~ ) ∘ A = ( I − γ H ) A ( I − γ H ) (\mathbf{I}-\gamma\widetilde{\mathcal{T}})\circ\mathbf{A}=(\mathbf{I}-\gamma\mathbf{H})\mathbf{A}(\mathbf{I}-\gamma\mathbf{H}) ( I − γ T ) ∘ A = ( I − γ H ) A ( I − γ H ) is its deterministic counterpart. Their difference isolates the fourth-moment fluctuation super-operator:
( I − γ T ) − ( I − γ T ~ ) = γ ( M − M ~ ) . (33) (\mathbf{I}-\gamma\mathcal{T})-(\mathbf{I}-\gamma\widetilde{\mathcal{T}})=\gamma(\mathcal{M}-\widetilde{\mathcal{M}}). \tag*{(33)} ( I − γ T ) − ( I − γ T ) = γ ( M − M ) . (33)
Let F t : = σ ( { ( x u , y u ) } u = 1 t ) \mathcal{F}_t := \sigma(\{(x_u,y_u)\}_{u=1}^t) F t := σ ({( x u , y u ) } u = 1 t ) denote the natural filtration generated by the first t t t samples (with F 0 \mathcal{F}_0 F 0 trivial). Because ( x t , y t ) (x_t,y_t) ( x t , y t ) is independent of F t − 1 \mathcal{F}_{t-1} F t − 1 and first-order optimality implies E [ ξ t x t ] = ∇ L ( w ∗ ) = 0 \mathbb{E}[\xi_t x_t] = \nabla L(w^*) = 0 E [ ξ t x t ] = ∇ L ( w ∗ ) = 0 , taking expectations in (8) from η 0 v a r i a n c e = 0 \eta_0^{\mathrm{variance}} = 0 η 0 variance = 0 gives E [ η t v a r i a n c e ] = ( I − γ H ) E [ η t − 1 v a r i a n c e ] = 0 \mathbb{E}[\eta_t^{\mathrm{variance}}] = (I-\gamma H)\mathbb{E}[\eta_{t-1}^{\mathrm{variance}}] = 0 E [ η t variance ] = ( I − γ H ) E [ η t − 1 variance ] = 0 for all t ≥ 1 t \ge1 t ≥ 1 . Together with sample independence, this cancels the cross terms in C t C_t C t , so taking outer products and expectations in (7) –(8) yields the exact single-step operator recurrences [15 , 37 ] :
B t = ( I − γ T ) ∘ B t − 1 , C t = ( I − γ T ) ∘ C t − 1 + γ 2 Σ , ∀ t ≥ 1. (34) B_t = (I-\gamma\mathcal{T}) \circ B_{t-1}, \qquad C_t = (I-\gamma\mathcal{T}) \circ C_{t-1} + \gamma^2\Sigma, \qquad\forall t \ge1. \tag*{(34)} B t = ( I − γ T ) ∘ B t − 1 , C t = ( I − γ T ) ∘ C t − 1 + γ 2 Σ , ∀ t ≥ 1. (34)
Our analysis is organized into two main components. In Section 5.1, we develop a perturbation-theoretic trajectory decomposition (Lemma 5.1) that unrolls (33) across the full horizon and leads directly to a discrete Volterra feedback absorption inequality (Lemma 5.2, whose complete proof we present below). In Section 5.2, we analyze the resulting propagation kernel k ( h ) k(h) k ( h ) and explain how the pair-midpoint convolution identity and kernel doubling constant D T D_{\mathcal{T}} D T control χ s , N \chi_{s,N} χ s , N across general and polynomially decaying spectra.
Perturbation Decomposition and Discrete Volterra Feedback Absorption
In Zou et al. [37 ] , the tail-averaged bias error is analyzed by splitting the trajectory at the end of the burn-in phase t = s t=s t = s : one first unrolls (34) over 0 ≤ t ≤ s − 1 0 \le t \le s-1 0 ≤ t ≤ s − 1 via the Duhamel expansion
B s = ( I − γ H ) s B 0 ( I − γ H ) s + γ 2 ∑ t = 0 s − 1 ( I − γ H ) s − 1 − t ( ( M − M ~ ) ∘ B t ) ( I − γ H ) s − 1 − t , (35) B_s = (I-\gamma H)^s B_0 (I-\gamma H)^s + \gamma^2 \sum_{t=0}^{s-1}(I-\gamma H)^{s-1-t}\bigl((\mathcal{M}-\widetilde{\mathcal{M}})\circ B_t\bigr)(I-\gamma H)^{s-1-t}, \tag*{(35)} B s = ( I − γ H ) s B 0 ( I − γ H ) s + γ 2 t = 0 ∑ s − 1 ( I − γ H ) s − 1 − t ( ( M − M ) ∘ B t ) ( I − γ H ) s − 1 − t , (35)
and then restarts a full-averaging bound from B s B_s B s over [ s , s + N − 1 ] [s,s+N-1] [ s , s + N − 1 ] . The subsequent relaxations in this two-stage argument cause both the upper- and lower-bound gaps in (1) –(2) : Zou et al. [37 ] upper-bounds (35) by replacing ( I − γ H ) 2 ( s − 1 − t ) H (I-\gamma H)^{2(s-1-t)}H ( I − γ H ) 2 ( s − 1 − t ) H with H H H (and Zou et al. [37 ] replaces the averaging-window trace drop tr ( B s , s + N ) \operatorname{tr}(B_{s,s+N}) tr ( B s , s + N ) by tr ( B 0 , s + N ) \operatorname{tr}(B_{0,s+N}) tr ( B 0 , s + N ) , where B a , b : = B a − ( I − γ H ) b − a B a ( I − γ H ) b − a B_{a,b} := B_a-(I-\gamma H)^{b-a}B_a(I-\gamma H)^{b-a} B a , b := B a − ( I − γ H ) b − a B a ( I − γ H ) b − a ), which destroys the remaining burn-in contraction on the fast modes; conversely, Zou et al. [37 ] lower-bounds (35) by dropping the non-negative sum over t ∈ { 0 , … , s − 1 } t \in\{0,\ldots,s-1\} t ∈ { 0 , … , s − 1 } altogether, which discards all burn-in fourth-moment energy scattered from the head initialization into slow modes.
Inspired by the operator-polynomial reformulations developed for momentum and exponential moving average SGD in Li et al. (2024); Li and Gu [20 ] , we bypass the intermediate boundary t = s t=s t = s and unroll the perturbation identity (33) directly across the entire trajectory r ∈ { 0 , … , s + N − 2 } r \in\{0,\ldots,s+N-2\} r ∈ { 0 , … , s + N − 2 } . Physically, a stochastic gradient fluctuation injected at step t = r + 1 t=r+1 t = r + 1 is propagated by the deterministic transition operator ( I − γ H ) m − t (I-\gamma H)^{m-t} ( I − γ H ) m − t to every subsequent iterate m ∈ { s ∨ t , … , T } m \in\{s \lor t,\ldots,T\} m ∈ { s ∨ t , … , T } in the averaging window, accumulating the filter polynomial Q r , s , N = q t ( H ) Q_{r,s,N}=q_t(H) Q r , s , N = q t ( H ) from (4.1). This yields the following exact covariance decomposition, proved in Appendix A.1.
Lemma 5.1 (Exact Operator-Polynomial Decomposition for Tail-Averaging). Under Assumptions 3.1–3.3, for any stepsize γ > 0 \gamma> 0 γ > 0 , burn-in length s ≥ 0 s \ge0 s ≥ 0 , and averaging window N ≥ 1 N \ge1 N ≥ 1 , define the deterministic operator polynomials
P s , N : = 1 N ∑ t = s s + N − 1 ( I − γ H ) t , Q r , s , N : = 1 N ∑ t = s ∨ ( r + 1 ) s + N − 1 ( I − γ H ) t − 1 − r , ∀ r ∈ { 0 , … , s + N − 2 } . (36) P_{s,N} := \frac{1}{N}\sum_{t=s}^{s+N-1}(I-\gamma H)^t,\qquad Q_{r,s,N} := \frac{1}{N}\sum_{t=s\vee(r+1)}^{s+N-1}(I-\gamma H)^{t-1-r},\qquad\forall r \in\{0,\ldots,s+N-2\}. \tag*{(36)} P s , N := N 1 t = s ∑ s + N − 1 ( I − γ H ) t , Q r , s , N := N 1 t = s ∨ ( r + 1 ) ∑ s + N − 1 ( I − γ H ) t − 1 − r , ∀ r ∈ { 0 , … , s + N − 2 } . (36)
Then the tail-averaged bias and variance covariance operators satisfy the exact identities
E [ η ˉ s : s + N b i a s ⊗ η ˉ s : s + N b i a s ] = P s , N B 0 P s , N + γ 2 ∑ r = 0 s + N − 2 Q r , s , N ( ( M − M ~ ) ∘ B r ) Q r , s , N , (37) \mathbb{E}\left[\bar{\eta}^{\mathrm{bias}}_{s:s+N}\otimes\bar{\eta}^{\mathrm{bias}}_{s:s+N}\right]=\mathbf{P}_{s,N}\mathbf{B}_0\mathbf{P}_{s,N}+\gamma^2\sum_{r=0}^{s+N-2}\mathbf{Q}_{r,s,N}\left((\mathcal{M}-\widetilde{\mathcal{M}})\circ\mathbf{B}_r\right)\mathbf{Q}_{r,s,N}, \tag*{(37)} E [ η ˉ s : s + N bias ⊗ η ˉ s : s + N bias ] = P s , N B 0 P s , N + γ 2 r = 0 ∑ s + N − 2 Q r , s , N ( ( M − M ) ∘ B r ) Q r , s , N , (37)
E [ η ˉ s : s + N v a r i a n c e ⊗ η ˉ s : s + N v a r i a n c e ] = γ 2 ∑ r = 0 s + N − 2 Q r , s , N ( ( M − M ~ ) ∘ C r + Σ ) Q r , s , N . (38) \mathbb{E}\left[\bar{\eta}^{\mathrm{variance}}_{s:s+N}\otimes\bar{\eta}^{\mathrm{variance}}_{s:s+N}\right]=\gamma^2\sum_{r=0}^{s+N-2}\mathbf{Q}_{r,s,N}\left((\mathcal{M}-\widetilde{\mathcal{M}})\circ\mathbf{C}_r+\mathbf{\Sigma}\right)\mathbf{Q}_{r,s,N}. \tag*{(38)} E [ η ˉ s : s + N variance ⊗ η ˉ s : s + N variance ] = γ 2 r = 0 ∑ s + N − 2 Q r , s , N ( ( M − M ) ∘ C r + Σ ) Q r , s , N . (38)
Lemma 5.1 explains why the receiving weights F t F_t F t , propagation kernel k ( h ) k(h) k ( h ) , and spectral feedback ratio χ s , N \chi_{s,N} χ s , N in (12) –(13) arise from first principles. Taking the trace inner product of (37) and (38) with 1 2 H \frac{1}{2}\mathbf{H} 2 1 H , the deterministic term 1 2 ⟨ H , P s , N B 0 P s , N ⟩ \frac{1}{2}\langle\mathbf{H},\mathbf{P}_{s,N}\mathbf{B}_0\mathbf{P}_{s,N}\rangle 2 1 ⟨ H , P s , N B 0 P s , N ⟩ equals EffectiveBias in (10) , while the remaining stochastic terms sum to EffectiveVar = = = (bias − - − EffectiveBias) + + + variance in (11) . Under Assumptions 3.2–3.4, the fourth-moment fluctuation operators ( M − M ~ ) ∘ B t − 1 (\mathcal{M}-\widetilde{\mathcal{M}})\circ\mathbf{B}_{t-1} ( M − M ) ∘ B t − 1 and ( M − M ~ ) ∘ C t − 1 (\mathcal{M}-\widetilde{\mathcal{M}})\circ\mathbf{C}_{t-1} ( M − M ) ∘ C t − 1 are sandwiched by multiples of tr ( H B t − 1 ) H \operatorname{tr}(\mathbf{H}\mathbf{B}_{t-1})\mathbf{H} tr ( H B t − 1 ) H and tr ( H C t − 1 ) H \operatorname{tr}(\mathbf{H}\mathbf{C}_{t-1})\mathbf{H} tr ( H C t − 1 ) H . Consequently, each fluctuation excited along H \mathbf{H} H at step t = r + 1 ∈ { 1 , … , T } t=r+1\in\{1,\ldots,T\} t = r + 1 ∈ { 1 , … , T } is weighted at the output by
⟨ H , Q t − 1 , s , N H Q t − 1 , s , N ⟩ = tr ( H 2 q t ( H ) 2 ) = ∑ i = 1 ∞ λ i 2 q t ( λ i ) 2 = F t . \langle\mathbf{H},\mathbf{Q}_{t-1,s,N}\mathbf{H}\mathbf{Q}_{t-1,s,N}\rangle=\operatorname{tr}\left(\mathbf{H}^2q_t(\mathbf{H})^2\right)=\sum_{i=1}^{\infty}\lambda_i^2q_t(\lambda_i)^2=F_t. ⟨ H , Q t − 1 , s , N H Q t − 1 , s , N ⟩ = tr ( H 2 q t ( H ) 2 ) = i = 1 ∑ ∞ λ i 2 q t ( λ i ) 2 = F t .
When B t − 1 \mathbf{B}_{t-1} B t − 1 is replaced by its unperturbed deterministic drift ( I − γ H ) t − 1 B 0 ( I − γ H ) t − 1 (\mathbf{I}-\gamma\mathbf{H})^{t-1}\mathbf{B}_0(\mathbf{I}-\gamma\mathbf{H})^{t-1} ( I − γ H ) t − 1 B 0 ( I − γ H ) t − 1 , the resulting single-hop initialization-transfer functional S s , N \mathcal{S}_{s,N} S s , N and additive-noise functional V s , N \mathcal{V}_{s,N} V s , N take the form
S s , N : = γ 2 2 ∑ t = 1 T ∥ ( I − γ H ) t − 1 ( w 0 − w ∗ ) ∥ H 2 F t , V s , N : = γ 2 2 ∑ t = 1 T F t . (39) \mathcal{S}_{s,N}:=\frac{\gamma^2}{2}\sum_{t=1}^{T}\left\|(\mathbf{I}-\gamma\mathbf{H})^{t-1}(\mathbf{w}_0-\mathbf{w}^*)\right\|_{\mathbf{H}}^2F_t,\qquad\mathcal{V}_{s,N}:=\frac{\gamma^2}{2}\sum_{t=1}^{T}F_t. \tag*{(39)} S s , N := 2 γ 2 t = 1 ∑ T ( I − γ H ) t − 1 ( w 0 − w ∗ ) H 2 F t , V s , N := 2 γ 2 t = 1 ∑ T F t . (39)
To bound the multi-hop contributions in EffectiveVar, we must control how the covariance traces f r : = tr ( H B r ) f_r:=\operatorname{tr}(\mathbf{H}\mathbf{B}_r) f r := tr ( H B r ) and g r : = tr ( H C r ) g_r:=\operatorname{tr}(\mathbf{H}\mathbf{C}_r) g r := tr ( H C r ) amplify over time. Unrolling B t − 1 \mathbf{B}_{t-1} B t − 1 as in (35) (and C t − 1 \mathbf{C}_{t-1} C t − 1 analogously) and taking the trace inner product with H \mathbf{H} H reveals that a fluctuation of magnitude f r − 1 H f_{r-1}\mathbf{H} f r − 1 H injected at an earlier step r < t r<t r < t is contracted by ( I − γ H ) t − r − 1 (\mathbf{I}-\gamma\mathbf{H})^{t-r-1} ( I − γ H ) t − r − 1 before being tested against H \mathbf{H} H at step t − 1 t-1 t − 1 , producing the two-time trace propagation kernel
tr ( H ( I − γ H ) t − r − 1 H ( I − γ H ) t − r − 1 ) = ∑ i = 1 ∞ λ i 2 ( 1 − γ λ i ) 2 ( t − r − 1 ) = k ( t − r − 1 ) . \operatorname{tr}\left(\mathbf{H}(\mathbf{I}-\gamma\mathbf{H})^{t-r-1}\mathbf{H}(\mathbf{I}-\gamma\mathbf{H})^{t-r-1}\right)=\sum_{i=1}^{\infty}\lambda_i^2(1-\gamma\lambda_i)^{2(t-r-1)}=k(t-r-1). tr ( H ( I − γ H ) t − r − 1 H ( I − γ H ) t − r − 1 ) = i = 1 ∑ ∞ λ i 2 ( 1 − γ λ i ) 2 ( t − r − 1 ) = k ( t − r − 1 ) .
Thus { f t } \{f_t\} { f t } and { g t } \{g_t\} { g t } obey a discrete-time Volterra convolution recursion driven by the kernel k ( t − r − 1 ) k(t-r-1) k ( t − r − 1 ) —a non-asymptotic counterpart of the Volterra integral equations that govern high-dimensional SGD dynamics [26 , 27 , 20 ] . Crucially, because the receiving weights F t F_t F t increase with t t t during the burn-in phase 1 ≤ t ≤ s 1\le t\le s 1 ≤ t ≤ s , bounding ∑ t = 1 T F t f t − 1 \sum_{t=1}^{T}F_tf_{t-1} ∑ t = 1 T F t f t − 1 via an unweighted trace sum ∑ t = 1 T f t − 1 \sum_{t=1}^{T}f_{t-1} ∑ t = 1 T f t − 1 would require replacing F t F_t F t by max t F t \max_tF_t max t F t and losing burn-in contraction. Inspired by the Volterra convolution self-bounding strategy of Li et al. [20 ] , we instead bound the F t F_t F t -weighted trajectory sum directly as a self-bounding functional.
Lemma 5.2 (Weighted Trajectory Feedback Absorption). Suppose Assumption 3.1 holds and 0 < γ < 1 / λ 1 0<\gamma<1/\lambda_1 0 < γ < 1/ λ 1 , and recall EffectiveBias, EffectiveVar, χ s , N \chi_{s,N} χ s , N , S s , N \mathcal{S}_{s,N} S s , N , and V s , N \mathcal{V}_{s,N} V s , N from (10) , (11) , (13) , and (39) .
Under Assumptions 3.2 and 3.3, if χ s , N < 1 \chi_{s,N}<1 χ s , N < 1 , then
E f f e c t i v e V a r ≤ α 1 − χ s , N S s , N + σ 2 1 − χ s , N V s , N . (40) \mathrm{EffectiveVar}\leq\frac{\alpha}{1-\chi_{s,N}}\mathcal{S}_{s,N}+\frac{\sigma^2}{1-\chi_{s,N}}\mathcal{V}_{s,N}. \tag*{(40)} EffectiveVar ≤ 1 − χ s , N α S s , N + 1 − χ s , N σ 2 V s , N . (40)
Under the well-specified model (6) and Assumptions 3.2 and 3.4 (without requiring χ s , N < 1 \chi_{s,N}<1 χ s , N < 1 ),
EffectiveVar ≥ β S s , N + σ n o i s e 2 V s , N . (41) \operatorname{EffectiveVar} \ge\beta\mathcal{S}_{s,N}+\sigma_{\mathrm{noise}}^{2}\mathcal{V}_{s,N}. \tag*{(41)} EffectiveVar ≥ β S s , N + σ noise 2 V s , N . (41)
Proof of Lemma 5.2. For each integer r ≥ 0 r \ge0 r ≥ 0 , let f r : = tr ( H B r ) f_r:=\operatorname{tr}(\mathbf{H}\mathbf{B}_r) f r := tr ( H B r ) , g r : = tr ( H C r ) g_r:=\operatorname{tr}(\mathbf{H}\mathbf{C}_r) g r := tr ( H C r ) , and b r : = tr ( H ( I − γ H ) 2 r B 0 ) = ∥ ( I − γ H ) r ( w 0 − w ∗ ) ∥ H 2 b_r:=\operatorname{tr}(\mathbf{H}(\mathbf{I}-\gamma\mathbf{H})^{2r}\mathbf{B}_0)=\|(\mathbf{I}-\gamma\mathbf{H})^r(\mathbf{w}_0-\mathbf{w}^*)\|_{\mathbf{H}}^2 b r := tr ( H ( I − γ H ) 2 r B 0 ) = ∥ ( I − γ H ) r ( w 0 − w ∗ ) ∥ H 2 . Taking the trace inner product of (37) and (38) in Lemma 5.1 with 1 2 H \frac{1}{2}\mathbf{H} 2 1 H and using 1 2 ⟨ H , P s , N B 0 P s , N ⟩ = EffectiveBias \frac{1}{2}\langle\mathbf{H},\mathbf{P}_{s,N}\mathbf{B}_0\mathbf{P}_{s,N}\rangle=\operatorname{EffectiveBias} 2 1 ⟨ H , P s , N B 0 P s , N ⟩ = EffectiveBias together with the index shift t = r + 1 ∈ { 1 , … , T } t=r+1\in\{1,\ldots,T\} t = r + 1 ∈ { 1 , … , T } , we obtain the exact representation
EffectiveVar = γ 2 2 ∑ t = 1 T ⟨ H , Q t − 1 , s , N ( ( M − M ~ ) ∘ ( B t − 1 + C t − 1 ) + Σ ) Q t − 1 , s , N ⟩ . (42) \operatorname{EffectiveVar}=\frac{\gamma^2}{2}\sum_{t=1}^{T}\left\langle\mathbf{H},\mathbf{Q}_{t-1,s,N}\left((\mathcal{M}-\widetilde{\mathcal{M}})\mathbin{\circ}(\mathbf{B}_{t-1}+\mathbf{C}_{t-1})+\mathbf{\Sigma}\right)\mathbf{Q}_{t-1,s,N}\right\rangle. \tag*{(42)} EffectiveVar = 2 γ 2 t = 1 ∑ T ⟨ H , Q t − 1 , s , N ( ( M − M ) ∘ ( B t − 1 + C t − 1 ) + Σ ) Q t − 1 , s , N ⟩ . (42)
We first prove the upper bound (40) . Applying Assumptions 3.2 and 3.3 to the inner operator in (42) gives ( M − M ~ ) ∘ B t − 1 ⪯ α f t − 1 H (\mathcal{M}-\widetilde{\mathcal{M}})\mathbin{\circ}\mathbf{B}_{t-1}\preceq\alpha f_{t-1}\mathbf{H} ( M − M ) ∘ B t − 1 ⪯ α f t − 1 H and ( M − M ~ ) ∘ C t − 1 + Σ ⪯ ( α g t − 1 + σ 2 ) H (\mathcal{M}-\widetilde{\mathcal{M}})\mathbin{\circ}\mathbf{C}_{t-1}+\mathbf{\Sigma}\preceq(\alpha g_{t-1}+\sigma^2)\mathbf{H} ( M − M ) ∘ C t − 1 + Σ ⪯ ( α g t − 1 + σ 2 ) H . Substituting these Loewner upper bounds into (42) yields
EffectiveVar ≤ γ 2 2 ∑ t = 1 T [ α ( f t − 1 + g t − 1 ) + σ 2 ] ⟨ H , Q t − 1 , s , N H Q t − 1 , s , N ⟩ = γ 2 2 ∑ t = 1 T F t [ α ( f t − 1 + g t − 1 ) + σ 2 ] = : U . (43) \begin{aligned}
\operatorname{EffectiveVar}\le\frac{\gamma^2}{2}\sum_{t=1}^{T}\left[\alpha(f_{t-1}+g_{t-1})+\sigma^2\right]\langle\mathbf{H},\mathbf{Q}_{t-1,s,N}\mathbf{H}\mathbf{Q}_{t-1,s,N}\rangle\\
&=\frac{\gamma^2}{2}\sum_{t=1}^{T}F_t\left[\alpha(f_{t-1}+g_{t-1})+\sigma^2\right]=:U. \tag*{(43)}
\end{aligned} EffectiveVar ≤ 2 γ 2 t = 1 ∑ T [ α ( f t − 1 + g t − 1 ) + σ 2 ] ⟨ H , Q t − 1 , s , N H Q t − 1 , s , N ⟩ = 2 γ 2 t = 1 ∑ T F t [ α ( f t − 1 + g t − 1 ) + σ 2 ] =: U . (43)
where the equality uses ⟨ H , Q t − 1 , s , N H Q t − 1 , s , N ⟩ = F t \langle\mathbf{H},\mathbf{Q}_{t-1,s,N}\mathbf{H}\mathbf{Q}_{t-1,s,N}\rangle=F_t ⟨ H , Q t − 1 , s , N H Q t − 1 , s , N ⟩ = F t . Next, unrolling the single-step recurrences (34) via the perturbation identity (33) up to step t − 1 t-1 t − 1 gives
B t − 1 = ( I − γ H ) t − 1 B 0 ( I − γ H ) t − 1 + γ 2 ∑ r = 1 t − 1 ( I − γ H ) t − 1 − r ( ( M − M ~ ) ∘ B r − 1 ) ( I − γ H ) t − 1 − r , \mathbf{B}_{t-1}=(\mathbf{I}-\gamma\mathbf{H})^{t-1}\mathbf{B}_0(\mathbf{I}-\gamma\mathbf{H})^{t-1}+\gamma^2\sum_{r=1}^{t-1}(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}\left((\mathcal{M}-\widetilde{\mathcal{M}})\mathbin{\circ}\mathbf{B}_{r-1}\right)(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}, B t − 1 = ( I − γ H ) t − 1 B 0 ( I − γ H ) t − 1 + γ 2 r = 1 ∑ t − 1 ( I − γ H ) t − 1 − r ( ( M − M ) ∘ B r − 1 ) ( I − γ H ) t − 1 − r ,
C t − 1 = γ 2 ∑ r = 1 t − 1 ( I − γ H ) t − 1 − r ( ( M − M ~ ) ∘ C r − 1 + Σ ) ( I − γ H ) t − 1 − r . \mathbf{C}_{t-1}=\gamma^2\sum_{r=1}^{t-1}(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}\left((\mathcal{M}-\widetilde{\mathcal{M}})\mathbin{\circ}\mathbf{C}_{r-1}+\mathbf{\Sigma}\right)(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}. C t − 1 = γ 2 r = 1 ∑ t − 1 ( I − γ H ) t − 1 − r ( ( M − M ) ∘ C r − 1 + Σ ) ( I − γ H ) t − 1 − r .
Taking the trace inner product of both expansions with H \mathbf{H} H and applying Assumptions 3.2 and 3.3 together with tr ( H ( I − γ H ) t − 1 − r H ( I − γ H ) t − 1 − r ) = k ( t − r − 1 ) \operatorname{tr}(\mathbf{H}(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}\mathbf{H}(\mathbf{I}-\gamma\mathbf{H})^{t-1-r})=k(t-r-1) tr ( H ( I − γ H ) t − 1 − r H ( I − γ H ) t − 1 − r ) = k ( t − r − 1 ) , we obtain for every t ∈ { 1 , … , T } t\in\{1,\ldots,T\} t ∈ { 1 , … , T } the coupled Volterra trace inequalities
f t − 1 ≤ b t − 1 + α γ 2 ∑ r = 1 t − 1 k ( t − r − 1 ) f r − 1 , g t − 1 ≤ γ 2 ∑ r = 1 t − 1 k ( t − r − 1 ) ( α g r − 1 + σ 2 ) . (44) f_{t-1}\le b_{t-1}+\alpha\gamma^2\sum_{r=1}^{t-1}k(t-r-1)f_{r-1},\qquad
g_{t-1}\le\gamma^2\sum_{r=1}^{t-1}k(t-r-1)(\alpha g_{r-1}+\sigma^2). \tag*{(44)} f t − 1 ≤ b t − 1 + α γ 2 r = 1 ∑ t − 1 k ( t − r − 1 ) f r − 1 , g t − 1 ≤ γ 2 r = 1 ∑ t − 1 k ( t − r − 1 ) ( α g r − 1 + σ 2 ) . (44)
Substituting (44) into the definition of U U U in (43) yields
U ≤ γ 2 2 ∑ t = 1 T F t ( α b t − 1 + σ 2 ) + γ 2 2 ∑ t = 1 T F t α γ 2 ∑ r = 1 t − 1 k ( t − r − 1 ) [ α ( f r − 1 + g r − 1 ) + σ 2 ] = α S s , N + σ 2 V s , N + γ 2 2 ∑ r = 1 T − 1 [ α ( f r − 1 + g r − 1 ) + σ 2 ] ( α γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t ) ≤ α S s , N + σ 2 V s , N + χ s , N ⋅ γ 2 2 ∑ r = 1 T − 1 F r [ α ( f r − 1 + g r − 1 ) + σ 2 ] ≤ α S s , N + σ 2 V s , N + χ s , N U , (45) \begin{aligned}
U\le\frac{\gamma^2}{2}\sum_{t=1}^{T}F_t(\alpha b_{t-1}+\sigma^2)+\frac{\gamma^2}{2}\sum_{t=1}^{T}F_t\alpha\gamma^2\sum_{r=1}^{t-1}k(t-r-1)\left[\alpha(f_{r-1}+g_{r-1})+\sigma^2\right] \\
&=\alpha\mathcal{S}_{s,N}+\sigma^2\mathcal{V}_{s,N}+\frac{\gamma^2}{2}\sum_{r=1}^{T-1}\left[\alpha(f_{r-1}+g_{r-1})+\sigma^2\right]\left(\alpha\gamma^2\sum_{t=r+1}^{T}k(t-r-1)F_t\right) \\
&\le\alpha\mathcal{S}_{s,N}+\sigma^2\mathcal{V}_{s,N}+\chi_{s,N}\cdot\frac{\gamma^2}{2}\sum_{r=1}^{T-1}F_r\left[\alpha(f_{r-1}+g_{r-1})+\sigma^2\right] \\
&\le\alpha\mathcal{S}_{s,N}+\sigma^{2}\mathcal{V}_{s,N}+\chi_{s,N}U, \tag*{(45)}
\end{aligned} U ≤ 2 γ 2 t = 1 ∑ T F t ( α b t − 1 + σ 2 ) + 2 γ 2 t = 1 ∑ T F t α γ 2 r = 1 ∑ t − 1 k ( t − r − 1 ) [ α ( f r − 1 + g r − 1 ) + σ 2 ] = α S s , N + σ 2 V s , N + 2 γ 2 r = 1 ∑ T − 1 [ α ( f r − 1 + g r − 1 ) + σ 2 ] ( α γ 2 t = r + 1 ∑ T k ( t − r − 1 ) F t ) ≤ α S s , N + σ 2 V s , N + χ s , N ⋅ 2 γ 2 r = 1 ∑ T − 1 F r [ α ( f r − 1 + g r − 1 ) + σ 2 ] ≤ α S s , N + σ 2 V s , N + χ s , N U , (45)
where the first equality uses the definitions of S s , N \mathcal{S}_{s,N} S s , N and V s , N \mathcal{V}_{s,N} V s , N in (39) and interchanges the finite sums over 1 ≤ r < t ≤ T 1\le r<t\le T 1 ≤ r < t ≤ T , the second inequality uses α γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t ≤ χ s , N F r \alpha\gamma^{2}\sum_{t=r+1}^{T}k(t-r-1)F_t\le\chi_{s,N}F_r α γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t ≤ χ s , N F r from the definition of χ s , N \chi_{s,N} χ s , N in (13) , and the last inequality extends the non-negative sum over r r r to T T T . Since χ s , N < 1 \chi_{s,N}<1 χ s , N < 1 , rearranging (45) and combining with (43) establishes (40) .
Finally, to prove the lower bound (41) , Lemma C.5(3) implies B t − 1 ⪰ ( I − γ H ) t − 1 B 0 ( I − γ H ) t − 1 \mathbf{B}_{t-1}\succeq(\mathbf{I}-\gamma\mathbf{H})^{t-1}\mathbf{B}_0(\mathbf{I}-\gamma\mathbf{H})^{t-1} B t − 1 ⪰ ( I − γ H ) t − 1 B 0 ( I − γ H ) t − 1 (so f t − 1 ≥ b t − 1 f_{t-1}\ge b_{t-1} f t − 1 ≥ b t − 1 ) and ( M − M ˉ ) ∘ C t − 1 ⪰ 0 (\mathcal{M}-\bar{\mathcal{M}})\circ\mathbf{C}_{t-1}\succeq0 ( M − M ˉ ) ∘ C t − 1 ⪰ 0 . Applying Assumption 3.4 and Σ = σ n o i s e 2 H \mathbf{\Sigma}=\sigma_{\mathrm{noise}}^{2}\mathbf{H} Σ = σ noise 2 H under (6) to (42) gives
EffectiveVar ≥ γ 2 2 ∑ t = 1 T ⟨ H , Q t − 1 , s , N ( β f t − 1 H + σ n o i s e 2 H ) Q t − 1 , s , N ⟩ ≥ γ 2 2 ∑ t = 1 T F t ( β b t − 1 + σ n o i s e 2 ) = β S s , N + σ n o i s e 2 V s , N , \begin{aligned}
\operatorname{EffectiveVar}
&\ge\frac{\gamma^{2}}{2}\sum_{t=1}^{T}\left\langle\mathbf{H},\mathbf{Q}_{t-1,s,N}\left(\beta f_{t-1}\mathbf{H}+\sigma_{\mathrm{noise}}^{2}\mathbf{H}\right)\mathbf{Q}_{t-1,s,N}\right\rangle\\
&\ge\frac{\gamma^{2}}{2}\sum_{t=1}^{T}F_t\left(\beta b_{t-1}+\sigma_{\mathrm{noise}}^{2}\right)=\beta\mathcal{S}_{s,N}+\sigma_{\mathrm{noise}}^{2}\mathcal{V}_{s,N},
\end{aligned} EffectiveVar ≥ 2 γ 2 t = 1 ∑ T ⟨ H , Q t − 1 , s , N ( β f t − 1 H + σ noise 2 H ) Q t − 1 , s , N ⟩ ≥ 2 γ 2 t = 1 ∑ T F t ( β b t − 1 + σ noise 2 ) = β S s , N + σ noise 2 V s , N ,
which completes the proof of Lemma 5.2 .
Propagation Kernel Geometry, Pair-Midpoint Convolution, and Spectral Regimes
With Lemma 5.2 in hand, the remaining analytical challenge is to bound the spectral feedback ratio χ s , N \chi_{s,N} χ s , N in (13) without losing the denominator weight F r F_r F r . During the burn-in phase 1 ≤ r < t ≤ s 1\le r<t\le s 1 ≤ r < t ≤ s , the receiving sensitivity F t F_t F t increases with t t t because later perturbations undergo fewer burn-in contractions before entering the averaging window [ s , T ] [s,T] [ s , T ] . Consequently, bounding F t / F r F_t/F_r F t / F r pointwise would incur an exponential blow-up in s s s .
Our key observation (Proposition 4.3 ) is that F t F_t F t and the step-to-step kernel k ( h ) k(h) k ( h ) are governed by the same functional k ( ⋅ ) k(\cdot) k ( ⋅ ) . Expanding the square of the filter polynomial q t ( λ i ) = 1 N ∑ a = s ∨ t T ( 1 − γ λ i ) a − t q_t(\lambda_i)=\frac{1}{N}\sum_{a=s\vee t}^{T}(1-\gamma\lambda_i)^{a-t} q t ( λ i ) = N 1 ∑ a = s ∨ t T ( 1 − γ λ i ) a − t in (12) and summing against λ i 2 \lambda_i^2 λ i 2 expresses F t F_t F t as the average of k ( ⋅ ) k(\cdot) k ( ⋅ ) evaluated at the pair midpoint a + b 2 − t \frac{a+b}{2}-t 2 a + b − t :
F t = ∑ i = 1 ∞ λ i 2 ( 1 N ∑ a = s ∨ t T ( 1 − γ λ i ) a − t ) ( 1 N ∑ b = s ∨ t T ( 1 − γ λ i ) b − t ) = 1 N 2 ∑ a , b = s ∨ t T k ( a + b 2 − t ) . F_t=\sum_{i=1}^{\infty}\lambda_i^2\left(\frac{1}{N}\sum_{a=s\vee t}^{T}(1-\gamma\lambda_i)^{a-t}\right)\left(\frac{1}{N}\sum_{b=s\vee t}^{T}(1-\gamma\lambda_i)^{b-t}\right)=\frac{1}{N^2}\sum_{a,b=s\vee t}^{T}k\left(\frac{a+b}{2}-t\right). F t = i = 1 ∑ ∞ λ i 2 ( N 1 a = s ∨ t ∑ T ( 1 − γ λ i ) a − t ) ( N 1 b = s ∨ t ∑ T ( 1 − γ λ i ) b − t ) = N 2 1 a , b = s ∨ t ∑ T k ( 2 a + b − t ) .
Substituting this pair-midpoint representation into γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t \gamma^2\sum_{t=r+1}^{T}k(t-r-1)F_t γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t and setting j : = t − r − 1 j:=t-r-1 j := t − r − 1 and h : = a + b 2 − r ∈ [ 1 , T ] h:=\frac{a+b}{2}-r\in[1,T] h := 2 a + b − r ∈ [ 1 , T ] reduces the inner sum over t t t to the discrete self-convolution ∑ j = 0 min ( a , b ) − r − 1 k ( j ) k ( h − 1 − j ) \sum_{j=0}^{\min(a,b)-r-1}k(j)k(h-1-j) ∑ j = 0 m i n ( a , b ) − r − 1 k ( j ) k ( h − 1 − j ) at total lag h − 1 h-1 h − 1 . This structure yields two complementary mechanisms for controlling χ s , N \chi_{s,N} χ s , N (proved in Appendix § 10 .§ 2 ):
General Trace-Class Spectra via Mode-Wise Positive Correlation (Proposition 4.3 (1)): Under the main-theorem restriction γ λ 1 ≤ 1 / 2 \gamma\lambda_1\le1/2 γ λ 1 ≤ 1/2 , write a i : = 1 − γ λ i ∈ [ 1 / 2 , 1 ) a_i:=1-\gamma\lambda_i\in[1/2,1) a i := 1 − γ λ i ∈ [ 1/2 , 1 ) . Because slower modes contract more slowly, the sequences { a i 2 j } i ≥ 1 \{a_i^{2j}\}_{i\ge1} { a i 2 j } i ≥ 1 and { a i 2 ( h − 1 − j ) } i ≥ 1 \{a_i^{2(h-1-j)}\}_{i\ge1} { a i 2 ( h − 1 − j ) } i ≥ 1 are both non-decreasing in i i i . By Chebyshev’s sum inequality under the positive weights { λ i 2 } i ≥ 1 \{\lambda_i^2\}_{i\ge1} { λ i 2 } i ≥ 1 (Lemma C.1 ),
k ( j ) k ( h − 1 − j ) ≤ tr ( H 2 ) k ( h − 1 ) ≤ tr ( H 2 ) ( 1 − γ λ 1 ) 2 k ( a + b 2 − r ) ≲ tr ( H 2 ) k ( a + b 2 − r ) . k(j)k(h-1-j)\le\operatorname{tr}(\mathbf{H}^2)k(h-1)\le\frac{\operatorname{tr}(\mathbf{H}^2)}{(1-\gamma\lambda_1)^2}k\left(\frac{a+b}{2}-r\right)\lesssim\operatorname{tr}(\mathbf{H}^2)k\left(\frac{a+b}{2}-r\right). k ( j ) k ( h − 1 − j ) ≤ tr ( H 2 ) k ( h − 1 ) ≤ ( 1 − γ λ 1 ) 2 tr ( H 2 ) k ( 2 a + b − r ) ≲ tr ( H 2 ) k ( 2 a + b − r ) .
Summing over the at most T − 1 T-1 T − 1 indices j j j and averaging over a , b ≥ s ∨ ( r + 1 ) ≥ s ∨ r a,b\ge s\vee(r+1)\ge s\vee r a , b ≥ s ∨ ( r + 1 ) ≥ s ∨ r reconstructs F r = 1 N 2 ∑ a , b = s ∨ r T k ( a + b 2 − r ) F_r=\frac{1}{N^2}\sum_{a,b=s\vee r}^{T}k\left(\frac{a+b}{2}-r\right) F r = N 2 1 ∑ a , b = s ∨ r T k ( 2 a + b − r ) , canceling the denominator F r F_r F r in (13) and yielding χ s , N ≲ α γ 2 T tr ( H 2 ) \chi_{s,N}\lesssim\alpha\gamma^2T\operatorname{tr}(\mathbf{H}^2) χ s , N ≲ α γ 2 T tr ( H 2 ) .
Polynomial Spectra via Midpoint Convolution Splitting and Kernel Doubling (Proposition 4.3 (2) and Corollary 4.4 ): In every term of the self-convolution ∑ j = 0 h − 1 k ( j ) k ( h − 1 − j ) \sum_{j=0}^{h-1} k(j)k(h-1-j) ∑ j = 0 h − 1 k ( j ) k ( h − 1 − j ) , at least one of the two lags j j j or h − 1 − j h-1-j h − 1 − j is at least ( h − 1 ) / 2 (h-1)/2 ( h − 1 ) /2 . Since k ( ⋅ ) k(\cdot) k ( ⋅ ) is non-increasing, splitting the convolution at the midpoint ( h − 1 ) / 2 (h-1)/2 ( h − 1 ) /2 (a discrete counterpart of regular-variation splitting for power-law Volterra kernels; cf. Li et al. [20 ] ) factors out k ( ( h − 1 ) / 2 ) ≤ D T k ( h ) k((h-1)/2) \le D_T k(h) k (( h − 1 ) /2 ) ≤ D T k ( h ) while the remaining factor sums to γ 2 ∑ n = 0 ∞ k ( n ) ≲ γ tr ( H ) \gamma^2 \sum_{n=0}^{\infty} k(n) \lesssim\gamma\operatorname{tr}(\mathbf{H}) γ 2 ∑ n = 0 ∞ k ( n ) ≲ γ tr ( H ) :
γ 2 ∑ j = 0 min ( a , b ) − r − 1 k ( j ) k ( h − 1 − j ) ≤ 2 k ( h − 1 2 ) γ 2 ∑ n = 0 ∞ k ( n ) ≲ D T γ tr ( H ) k ( a + b 2 − r ) . \gamma^2 \sum_{j=0}^{\min(a,b)-r-1} k(j)k(h-1-j) \le2k\left(\frac{h-1}{2}\right)\gamma^2\sum_{n=0}^{\infty} k(n) \lesssim D_T\gamma\operatorname{tr}(\mathbf{H})k\left(\frac{a+b}{2}-r\right). γ 2 j = 0 ∑ m i n ( a , b ) − r − 1 k ( j ) k ( h − 1 − j ) ≤ 2 k ( 2 h − 1 ) γ 2 n = 0 ∑ ∞ k ( n ) ≲ D T γ tr ( H ) k ( 2 a + b − r ) .
Averaging over ( a , b ) (a,b) ( a , b ) again recovers F r F_r F r and gives the horizon-free bound χ s , N ≲ α D T γ tr ( H ) \chi_{s,N} \lesssim\alpha D_T\gamma\operatorname{tr}(\mathbf{H}) χ s , N ≲ α D T γ tr ( H ) . Under two-sided polynomial eigenvalue decay λ i ≍ Λ i − a \lambda_i \asymp\Lambda i^{-a} λ i ≍ Λ i − a (a > 1 a>1 a > 1 ), integral comparison over the active spectral scale i ≍ ( γ Λ h ) 1 / a i \asymp(\gamma\Lambda h)^{1/a} i ≍ ( γ Λ h ) 1/ a shows that k ( h ) ≍ Λ 2 ( 1 + γ Λ h ) − ( 2 − 1 / a ) k(h) \asymp\Lambda^2(1+\gamma\Lambda h)^{-(2-1/a)} k ( h ) ≍ Λ 2 ( 1 + γ Λ h ) − ( 2 − 1/ a ) decays as a power law, so its doubling ratio D T D_T D T is uniformly bounded by a constant D ∗ ( a , θ ) < ∞ D_*(a,\theta)<\infty D ∗ ( a , θ ) < ∞ for all T ≥ 1 T\ge1 T ≥ 1 (whereas in d = 1 d=1 d = 1 , k ( h ) = λ 1 2 ( 1 − γ λ 1 ) 2 h k(h)=\lambda_1^{2}(1-\gamma\lambda_1)^{2h} k ( h ) = λ 1 2 ( 1 − γ λ 1 ) 2 h decays exponentially so D T D_T D T grows exponentially in T T T , giving rise to the scalar separation in Proposition 4.7(2)).
Finally, in Appendix § 8 .§ 2 (Lemma A.1 ), we evaluate the single-hop functionals EffectiveBias, V s , N V_{s,N} V s , N , and S s , N S_{s,N} S s , N within universal constants across the spectral cutoffs k ∗ k^* k ∗ and k † k^\dagger k † , which combined with Lemma 5.2 yields Theorems 4.1 and 4.2 .
Conclusion
We have established initialization-explicit matching upper and lower excess risk bounds for constant-stepsize stochastic gradient descent with tail-averaging in high-dimensional linear regression under the distributional assumptions of Zou et al. [37 ] . By controlling cumulative fourth-moment trajectory feedback through the spectral feedback ratio χ s , N ≤ χ 0 < 1 \chi_{s,N}\le\chi_0<1 χ s , N ≤ χ 0 < 1 and bounding χ s , N \chi_{s,N} χ s , N via the pair-midpoint convolution identity, we proved that matching bounds hold for arbitrary covariance spectra under a horizon-dependent constant stepsize chosen before the run, and hold uniformly across all horizons s ≥ 0 s\ge0 s ≥ 0 and N ≥ 2 N\ge2 N ≥ 2 at a single horizon-independent stepsize for polynomially decaying spectra λ i ≍ i − a \lambda_i\asymp i^{-a} λ i ≍ i − a (a > 1 a>1 a > 1 ). Complementing these positive results, our two-dimensional Gaussian separation construction demonstrates an exponential-versus-polynomial improvement over the lower bound of Zou et al. [37 ] due to burn-in head-to-tail transfer, while our scalar separation construction shows why additional structural conditions (such as χ s , N ≤ χ 0 < 1 \chi_{s,N}\le\chi_0<1 χ s , N ≤ χ 0 < 1 ) or state information (B s B_s B s ) are required over the unrestricted fourth-moment class when D T D_T D T grows exponentially.
Acknowledgment
The proof ideas are developed through an interactive brainstorm among GPT-6 Astra, Gemini 4 Argon, and the authors. Gemini 4 Argon are employed to further streamline the canonical pillars underlying the proof roadmap. The authors simplified the arguments, re-organized the proofs, verified the details, and prepared the draft with further assistance from Gemini 4 Argon and GPT-6 Astra. The authors take full responsibility for its contents.
Trajectory Decomposition and Spectral Evaluation
In this appendix, we establish the exact operator-polynomial decomposition (Lemma 5.1 in Section A.1) and the two-sided spectral evaluations of EffectiveBias, V s , N V_{s,N} V s , N , and S s , N S_{s,N} S s , N (Lemma A.1 in Section A.2). Elementary algebraic and convolution lemmas as well as foundational operator lemmas from prior work are collected in Appendix C.
Additional Notations. Throughout the appendices, for any real number z ∈ R z \in\mathbb{R} z ∈ R , we write ⌊ z ⌋ \lfloor z \rfloor ⌊ z ⌋ and ⌈ z ⌉ \lceil z \rceil ⌈ z ⌉ for the floor and ceiling of z z z , respectively. Recall from Section 5 the natural filtration F t = σ ( { ( x u , y u ) } u = 1 t ) \mathcal{F}_t = \sigma(\{(x_u,y_u)\}_{u=1}^t) F t = σ ({( x u , y u ) } u = 1 t ) (with F 0 \mathcal{F}_0 F 0 trivial) and the eigenvalue cutoffs k ∗ = max { k : λ k ≥ ( γ N ) − 1 } k^* = \max\{k : \lambda_k \ge(\gamma N)^{-1}\} k ∗ = max { k : λ k ≥ ( γ N ) − 1 } and k † = max { k : λ k ≥ ( γ ( s + N ) ) − 1 } k^\dagger= \max\{k : \lambda_k \ge(\gamma(s+N))^{-1}\} k † = max { k : λ k ≥ ( γ ( s + N ) ) − 1 } from (17) . To streamline the spectral evaluations across Appendices A and B, we define the burn-in drifted error vector, initial eigen-coordinates, and window effective-variance multiplier by
w s d r i f t : = ( I − γ H ) s ( w 0 − w ∗ ) , ω j : = ( w 0 − w ∗ ) j = ⟨ v j , w 0 − w ∗ ⟩ , Ξ N : = k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 . (46) \mathbf{w}^{\mathrm{drift}}_s := (\mathbf{I}-\gamma\mathbf{H})^s(\mathbf{w}_0-\mathbf{w}^*), \qquad\omega_j := (\mathbf{w}_0-\mathbf{w}^*)_j = \langle\mathbf{v}_j,\mathbf{w}_0-\mathbf{w}^*\rangle, \qquad\Xi_N := \frac{k^*}{N}+N\gamma^2\sum_{i>k^*}\lambda_i^2. \tag*{(46)} w s drift := ( I − γ H ) s ( w 0 − w ∗ ) , ω j := ( w 0 − w ∗ ) j = ⟨ v j , w 0 − w ∗ ⟩ , Ξ N := N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 . (46)
and for each r ∈ { 0 , … , s + N − 2 } r \in\{0,\ldots,s+N-2\} r ∈ { 0 , … , s + N − 2 } , we define the step-( r + 1 ) (r+1) ( r + 1 ) scaled receiving weight
Ψ r , s , N : = γ 2 F r + 1 = 1 N 2 ∑ i = 1 ∞ ( 1 − γ λ i ) 2 ( ( s − 1 − r ) ∨ 0 ) ( 1 − ( 1 − γ λ i ) ( s + N − 1 − r ) ∧ N ) 2 . (47) \Psi_{r,s,N} := \gamma^2 F_{r+1} = \frac{1}{N^2}\sum_{i=1}^{\infty}(1-\gamma\lambda_i)^{2((s-1-r)\vee0)}\left(1-(1-\gamma\lambda_i)^{(s+N-1-r)\wedge N}\right)^2. \tag*{(47)} Ψ r , s , N := γ 2 F r + 1 = N 2 1 i = 1 ∑ ∞ ( 1 − γ λ i ) 2 (( s − 1 − r ) ∨ 0 ) ( 1 − ( 1 − γ λ i ) ( s + N − 1 − r ) ∧ N ) 2 . (47)
Proof of Lemma 5.1
Proof of Lemma 5.1. We first establish the bias covariance identity (37) . For each r ≥ 1 r \ge1 r ≥ 1 , define the zero-mean random operator fluctuation
Δ r : = ( I − γ x r x r T ) − ( I − γ H ) = γ ( H − x r x r T ) . \Delta_r := (\mathbf{I}-\gamma\mathbf{x}_r\mathbf{x}_r^{\mathsf{T}})-(\mathbf{I}-\gamma\mathbf{H})=\gamma(\mathbf{H}-\mathbf{x}_r\mathbf{x}_r^{\mathsf{T}}). Δ r := ( I − γ x r x r T ) − ( I − γ H ) = γ ( H − x r x r T ) .
Then the single-step bias recursion (7) can be written as η t b i a s = ( I − γ H ) η t − 1 b i a s + Δ t η t − 1 b i a s \eta_t^{\mathrm{bias}}=(\mathbf{I}-\gamma\mathbf{H})\eta_{t-1}^{\mathrm{bias}}+\Delta_t\eta_{t-1}^{\mathrm{bias}} η t bias = ( I − γ H ) η t − 1 bias + Δ t η t − 1 bias . Unrolling this linear recursion from step 0 0 0 to step t ≥ 0 t\ge0 t ≥ 0 gives
η t b i a s = ( I − γ H ) t η 0 b i a s + ∑ r = 0 t − 1 ( I − γ H ) t − 1 − r Δ r + 1 η r b i a s . (48) \eta_t^{\mathrm{bias}}=(\mathbf{I}-\gamma\mathbf{H})^t\eta_0^{\mathrm{bias}}+\sum_{r=0}^{t-1}(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}\Delta_{r+1}\eta_r^{\mathrm{bias}}. \tag*{(48)} η t bias = ( I − γ H ) t η 0 bias + r = 0 ∑ t − 1 ( I − γ H ) t − 1 − r Δ r + 1 η r bias . (48)
Averaging (A.3) over t ∈ { s , s + 1 , … , s + N − 1 } t\in\{s,s+1,\ldots,s+N-1\} t ∈ { s , s + 1 , … , s + N − 1 } and exchanging the order of summation between t t t and r ∈ { 0 , … , s + N − 2 } r\in\{0,\ldots,s+N-2\} r ∈ { 0 , … , s + N − 2 } , we obtain
η ‾ s : s + N b i a s = 1 N ∑ t = s s + N − 1 ( I − γ H ) t η 0 b i a s + 1 N ∑ t = s s + N − 1 ∑ r = 0 t − 1 ( I − γ H ) t − 1 − r Δ r + 1 η r b i a s = P s , N η 0 b i a s + ∑ r = 0 s + N − 2 ( 1 N ∑ t = s ∨ ( r + 1 ) s + N − 1 ( I − γ H ) t − 1 − r ) Δ r + 1 η r b i a s = P s , N η 0 b i a s + ∑ r = 0 s + N − 2 Q r , s , N Δ r + 1 η r b i a s , (49) \begin{aligned}
\overline{\eta}_{s:s+N}^{\mathrm{bias}}
&=\frac{1}{N}\sum_{t=s}^{s+N-1}(\mathbf{I}-\gamma\mathbf{H})^t\eta_0^{\mathrm{bias}}+\frac{1}{N}\sum_{t=s}^{s+N-1}\sum_{r=0}^{t-1}(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}\Delta_{r+1}\eta_r^{\mathrm{bias}}\\
&=\mathbf{P}_{s,N}\eta_0^{\mathrm{bias}}+\sum_{r=0}^{s+N-2}\left(\frac{1}{N}\sum_{t=s\vee(r+1)}^{s+N-1}(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}\right)\Delta_{r+1}\eta_r^{\mathrm{bias}}\\
&=\mathbf{P}_{s,N}\eta_0^{\mathrm{bias}}+\sum_{r=0}^{s+N-2}\mathbf{Q}_{r,s,N}\Delta_{r+1}\eta_r^{\mathrm{bias}}, \tag*{(49)}
\end{aligned} η s : s + N bias = N 1 t = s ∑ s + N − 1 ( I − γ H ) t η 0 bias + N 1 t = s ∑ s + N − 1 r = 0 ∑ t − 1 ( I − γ H ) t − 1 − r Δ r + 1 η r bias = P s , N η 0 bias + r = 0 ∑ s + N − 2 N 1 t = s ∨ ( r + 1 ) ∑ s + N − 1 ( I − γ H ) t − 1 − r Δ r + 1 η r bias = P s , N η 0 bias + r = 0 ∑ s + N − 2 Q r , s , N Δ r + 1 η r bias , (49)
where the third equality uses the definitions of P s , N \mathbf{P}_{s,N} P s , N and Q r , s , N \mathbf{Q}_{r,s,N} Q r , s , N in (36) . Since η r b i a s \eta_r^{\mathrm{bias}} η r bias is F r \mathcal{F}_r F r -measurable and ( x r + 1 , y r + 1 ) (x_{r+1},y_{r+1}) ( x r + 1 , y r + 1 ) is independent of F r \mathcal{F}_r F r with E [ Δ r + 1 ∣ F r ] = 0 \mathbb{E}[\Delta_{r+1}\mid\mathcal{F}_r]=0 E [ Δ r + 1 ∣ F r ] = 0 , the sequence { Q r , s , N Δ r + 1 η r b i a s } r = 0 s + N − 2 \{\mathbf{Q}_{r,s,N}\Delta_{r+1}\eta_r^{\mathrm{bias}}\}_{r=0}^{s+N-2} { Q r , s , N Δ r + 1 η r bias } r = 0 s + N − 2 is a martingale difference sequence with respect to { F r + 1 } r ≥ 0 \{\mathcal{F}_{r+1}\}_{r\ge0} { F r + 1 } r ≥ 0 . Consequently, for any 0 ≤ r < r ′ ≤ s + N − 2 0\le r<r'\le s+N-2 0 ≤ r < r ′ ≤ s + N − 2 ,
E [ ( Q r ′ , s , N Δ r ′ + 1 η r ′ b i a s ) ⊗ ( Q r , s , N Δ r + 1 η r b i a s ) ] = 0 , E [ ( Q r , s , N Δ r + 1 η r b i a s ) ⊗ ( P s , N η 0 b i a s ) ] = 0. \mathbb{E}\left[(Q_{r',s,N}\Delta_{r'+1}\eta_{r'}^{\mathrm{bias}})\otimes(Q_{r,s,N}\Delta_{r+1}\eta_r^{\mathrm{bias}})\right]=0,\qquad\mathbb{E}\left[(Q_{r,s,N}\Delta_{r+1}\eta_r^{\mathrm{bias}})\otimes(P_{s,N}\eta_0^{\mathrm{bias}})\right]=0. E [ ( Q r ′ , s , N Δ r ′ + 1 η r ′ bias ) ⊗ ( Q r , s , N Δ r + 1 η r bias ) ] = 0 , E [ ( Q r , s , N Δ r + 1 η r bias ) ⊗ ( P s , N η 0 bias ) ] = 0.
Furthermore, for each r ∈ { 0 , … , s + N − 2 } r\in\{0,\ldots,s+N-2\} r ∈ { 0 , … , s + N − 2 } , conditioning on F r \mathcal{F}_r F r and using (5) gives
E [ ( Δ r + 1 η r b i a s ) ⊗ ( Δ r + 1 η r b i a s ) ] = γ 2 E [ E [ ( x r + 1 x r + 1 T − H ) ( η r b i a s ⊗ η r b i a s ) ( x r + 1 x r + 1 T − H ) ∣ F r ] ] = γ 2 ( M − M ~ ) ∘ E [ η r b i a s ⊗ η r b i a s ] = γ 2 ( M − M ~ ) ∘ B r . (50) \begin{aligned}
\mathbb{E}\left[(\Delta_{r+1}\eta_r^{\mathrm{bias}})\otimes(\Delta_{r+1}\eta_r^{\mathrm{bias}})\right]=\gamma^2\mathbb{E}\left[\mathbb{E}\left[(x_{r+1}x_{r+1}^{\mathsf T}-\mathbf{H})(\eta_r^{\mathrm{bias}}\otimes\eta_r^{\mathrm{bias}})(x_{r+1}x_{r+1}^{\mathsf T}-\mathbf{H})\mid\mathcal{F}_r\right]\right] \\
&=\gamma^2(\mathcal{M}-\widetilde{\mathcal{M}})\circ\mathbb{E}[\eta_r^{\mathrm{bias}}\otimes\eta_r^{\mathrm{bias}}]=\gamma^2(\mathcal{M}-\widetilde{\mathcal{M}})\circ\mathbf{B}_r. \tag*{(50)}
\end{aligned} E [ ( Δ r + 1 η r bias ) ⊗ ( Δ r + 1 η r bias ) ] = γ 2 E [ E [ ( x r + 1 x r + 1 T − H ) ( η r bias ⊗ η r bias ) ( x r + 1 x r + 1 T − H ) ∣ F r ] ] = γ 2 ( M − M ) ∘ E [ η r bias ⊗ η r bias ] = γ 2 ( M − M ) ∘ B r . (50)
Taking the outer product of (A.4) with itself in expectation and substituting (A.5) yields (37) .
To prove the variance covariance identity (38) under Assumption 3.3 (without assuming independence of ξ t \xi_t ξ t and x t x_t x t ), we work with C t = E [ η t v a r i a n c e ⊗ η t v a r i a n c e ] C_t=\mathbb{E}[\eta_t^{\mathrm{variance}}\otimes\eta_t^{\mathrm{variance}}] C t = E [ η t variance ⊗ η t variance ] . Since η t v a r i a n c e \eta_t^{\mathrm{variance}} η t variance is F t \mathcal{F}_t F t -measurable and the tower property with E [ ξ u x u ∣ F u − 1 ] = ∇ L ( w ∗ ) = 0 \mathbb{E}[\xi_u x_u\mid\mathcal{F}_{u-1}]=\nabla L(\mathbf{w}^*)=\mathbf{0} E [ ξ u x u ∣ F u − 1 ] = ∇ L ( w ∗ ) = 0 implies E [ ξ u x u ∣ F t ] = 0 \mathbb{E}[\xi_u x_u\mid\mathcal{F}_t]=0 E [ ξ u x u ∣ F t ] = 0 for all u > t u>t u > t , taking conditional expectations given F t \mathcal{F}_t F t in (8) yields E [ η t ′ v a r i a n c e ∣ F t ] = ( I − γ H ) t ′ − t η t v a r i a n c e \mathbb{E}[\eta_{t'}^{\mathrm{variance}}\mid\mathcal{F}_t]=(\mathbf{I}-\gamma\mathbf{H})^{t'-t}\eta_t^{\mathrm{variance}} E [ η t ′ variance ∣ F t ] = ( I − γ H ) t ′ − t η t variance for all t ′ ≥ t ≥ 0 t'\ge t\ge0 t ′ ≥ t ≥ 0 , and hence
E [ η t ′ v a r i a n c e ⊗ η t v a r i a n c e ] = E [ E [ η t ′ v a r i a n c e ∣ F t ] ⊗ η t v a r i a n c e ] = ( I − γ H ) t ′ − t C t , ∀ t ′ ≥ t ≥ 0. (51) \mathbb{E}[\eta_{t'}^{\mathrm{variance}}\otimes\eta_t^{\mathrm{variance}}]=\mathbb{E}\left[\mathbb{E}[\eta_{t'}^{\mathrm{variance}}\mid\mathcal{F}_t]\otimes\eta_t^{\mathrm{variance}}\right]=(\mathbf{I}-\gamma\mathbf{H})^{t'-t}C_t,\qquad\forall t'\ge t\ge0. \tag*{(51)} E [ η t ′ variance ⊗ η t variance ] = E [ E [ η t ′ variance ∣ F t ] ⊗ η t variance ] = ( I − γ H ) t ′ − t C t , ∀ t ′ ≥ t ≥ 0. (51)
On the other hand, iterating C t = ( I − γ T ~ ) ∘ C t − 1 + γ 2 ( ( M − M ~ ) ∘ C t − 1 + Σ ) C_t=(\mathbf{I}-\gamma\widetilde{\mathbf{T}})\circ C_{t-1}+\gamma^2((\mathcal{M}-\widetilde{\mathcal{M}})\circ C_{t-1}+\mathbf{\Sigma}) C t = ( I − γ T ) ∘ C t − 1 + γ 2 (( M − M ) ∘ C t − 1 + Σ ) from C 0 = 0 C_0=\mathbf{0} C 0 = 0 gives
C t = γ 2 ∑ r = 0 t − 1 ( I − γ H ) t − 1 − r D r ( I − γ H ) t − 1 − r , where D r : = ( M − M ~ ) ∘ C r + Σ . (52) C_t=\gamma^2\sum_{r=0}^{t-1}(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}\mathbf{D}_r(\mathbf{I}-\gamma\mathbf{H})^{t-1-r},\qquad\text{where}\quad\mathbf{D}_r:=(\mathcal{M}-\widetilde{\mathcal{M}})\circ C_r+\mathbf{\Sigma}. \tag*{(52)} C t = γ 2 r = 0 ∑ t − 1 ( I − γ H ) t − 1 − r D r ( I − γ H ) t − 1 − r , where D r := ( M − M ) ∘ C r + Σ . (52)
Substituting (A.7) into (A.6) (and taking transposes when t ′ < t t'<t t ′ < t ), we see that for every pair ( t , t ′ ) ∈ { s , … , s + N − 1 } 2 (t,t')\in\{s,\ldots,s+N-1\}^2 ( t , t ′ ) ∈ { s , … , s + N − 1 } 2 ,
E [ η t ′ v a r i a n c e ⊗ η t v a r i a n c e ] = γ 2 ∑ r = 0 ( t ∧ t ′ ) − 1 ( I − γ H ) t ′ − 1 − r D r ( I − γ H ) t − 1 − r . \mathbb{E}[\eta_{t'}^{\mathrm{variance}}\otimes\eta_t^{\mathrm{variance}}]=\gamma^2\sum_{r=0}^{(t\wedge t')-1}(\mathbf{I}-\gamma\mathbf{H})^{t'-1-r}\mathbf{D}_r(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}. E [ η t ′ variance ⊗ η t variance ] = γ 2 r = 0 ∑ ( t ∧ t ′ ) − 1 ( I − γ H ) t ′ − 1 − r D r ( I − γ H ) t − 1 − r .
Averaging over all t , t ′ ∈ { s , … , s + N − 1 } t,t'\in\{s,\ldots,s+N-1\} t , t ′ ∈ { s , … , s + N − 1 } and exchanging the sum over r ∈ { 0 , … , s + N − 2 } r\in\{0,\ldots,s+N-2\} r ∈ { 0 , … , s + N − 2 } with the double sum over t , t ′ ≥ s ∨ ( r + 1 ) t,t'\ge s\vee(r+1) t , t ′ ≥ s ∨ ( r + 1 ) yields
E [ η ‾ s : s + N v a r i a n c e ⊗ η ‾ s : s + N v a r i a n c e ] = γ 2 ∑ r = 0 s + N − 2 ( 1 N ∑ t ′ = s ∨ ( r + 1 ) s + N − 1 ( I − γ H ) t ′ − 1 − r ) D r ( 1 N ∑ t = s ∨ ( r + 1 ) s + N − 1 ( I − γ H ) t − 1 − r ) = γ 2 ∑ r = 0 s + N − 2 Q r , s , N ( ( M − M ~ ) ∘ C r + Σ ) Q r , s , N , \begin{aligned}
\mathbb{E}[\overline{\eta}_{s:s+N}^{\mathrm{variance}}\otimes\overline{\eta}_{s:s+N}^{\mathrm{variance}}]=\gamma^2\sum_{r=0}^{s+N-2}\left(\frac{1}{N}\sum_{t'=s\vee(r+1)}^{s+N-1}(\mathbf{I}-\gamma\mathbf{H})^{t'-1-r}\right)\mathbf{D}_r\left(\frac{1}{N}\sum_{t=s\vee(r+1)}^{s+N-1}(\mathbf{I}-\gamma\mathbf{H})^{t-1-r}\right) \\
&=\gamma^2\sum_{r=0}^{s+N-2}Q_{r,s,N}\bigl((\mathcal{M}-\widetilde{\mathcal{M}})\circ C_r+\mathbf{\Sigma}\bigr)Q_{r,s,N},
\end{aligned} E [ η s : s + N variance ⊗ η s : s + N variance ] = γ 2 r = 0 ∑ s + N − 2 N 1 t ′ = s ∨ ( r + 1 ) ∑ s + N − 1 ( I − γ H ) t ′ − 1 − r D r N 1 t = s ∨ ( r + 1 ) ∑ s + N − 1 ( I − γ H ) t − 1 − r = γ 2 r = 0 ∑ s + N − 2 Q r , s , N ( ( M − M ) ∘ C r + Σ ) Q r , s , N ,
which completes the proof of (38) . □
Two-Sided Spectral Evaluation of Trajectory Functionals
Lemma A.1 (Two-Sided Spectral Evaluation of EffectiveBias, V s , N V_{s,N} V s , N , and S s , N S_{s,N} S s , N ). Suppose Assumption 3.1 holds and 0 < γ ≤ 1 / ( 2 λ 1 ) 0<\gamma\le1/(2\lambda_1) 0 < γ ≤ 1/ ( 2 λ 1 ) , and recall EffectiveBias, S s , N S_{s,N} S s , N , V s , N V_{s,N} V s , N , and ( w s d r i f t , ω j , Ξ N ) (\mathbf{w}^{\mathrm{drift}}_s,\omega_j,\Xi_N) ( w s drift , ω j , Ξ N ) from (10) , (39) , and (A.1). Define the burn-in cross-spectral transfer functional
Φ b u r n ( s , N ) : = ∑ i , j = 1 ∞ min { 1 N 2 , γ 2 λ i 2 } λ j ω j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) min { s , 1 γ ∣ λ i − λ j ∣ } . (53) \Phi_{\mathrm{burn}}(s,N):=\sum_{i,j=1}^{\infty}\min\left\{\frac{1}{N^2},\gamma^2\lambda_i^2\right\}\lambda_j\omega_j^2(1-\gamma(\lambda_i\wedge\lambda_j))^{2(s-1)}\min\left\{s,\frac{1}{\gamma|\lambda_i-\lambda_j|}\right\}. \tag*{(53)} Φ burn ( s , N ) := i , j = 1 ∑ ∞ min { N 2 1 , γ 2 λ i 2 } λ j ω j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) min { s , γ ∣ λ i − λ j ∣ 1 } . (53)
Then for all s ≥ 0 s \ge0 s ≥ 0 and N ≥ 1 N \ge1 N ≥ 1 (and N ≥ 2 N \ge2 N ≥ 2 for the lower bounds on V s , N \mathcal{V}_{s,N} V s , N and S s , N \mathcal{S}_{s,N} S s , N ),4
1 8 ( 1 γ 2 N 2 ∥ w s d r i f t ∥ H 0 : k ∗ − 1 2 + ∥ w s d r i f t ∥ H k ∗ : ∞ 2 ) ≤ EffectiveBias ≤ 1 2 ( 1 γ 2 N 2 ∥ w s d r i f t ∥ H 0 : k ∗ − 1 2 + ∥ w s d r i f t ∥ H k ∗ : ∞ 2 ) (54) \frac{1}{8}\left(\frac{1}{\gamma^{2}N^{2}}\lVert w_s^{\mathrm{drift}}\rVert_{H_{0:k^*}^{-1}}^{2}+\lVert w_s^{\mathrm{drift}}\rVert_{H_{k^*:\infty}}^{2}\right)\le\operatorname{EffectiveBias}
\le\frac{1}{2}\left(\frac{1}{\gamma^{2}N^{2}}\lVert w_s^{\mathrm{drift}}\rVert_{H_{0:k^*}^{-1}}^{2}+\lVert w_s^{\mathrm{drift}}\rVert_{H_{k^*:\infty}}^{2}\right) \tag*{(54)} 8 1 ( γ 2 N 2 1 ∥ w s drift ∥ H 0 : k ∗ − 1 2 + ∥ w s drift ∥ H k ∗ : ∞ 2 ) ≤ EffectiveBias ≤ 2 1 ( γ 2 N 2 1 ∥ w s drift ∥ H 0 : k ∗ − 1 2 + ∥ w s drift ∥ H k ∗ : ∞ 2 ) (54)
1 216 ( k ∗ N + γ ∑ k ∗ < i ≤ k † λ i + ( s + N ) γ 2 ∑ i > k † λ i 2 ) ≤ V s , N ≤ k ∗ N + γ ∑ k ∗ < i ≤ k † λ i + ( s + N ) γ 2 ∑ i > k † λ i 2 (55) \frac{1}{216}\left(\frac{k^*}{N}+\gamma\sum_{k^*<i\le k^\dagger}\lambda_i+(s+N)\gamma^2\sum_{i>k^\dagger}\lambda_i^2\right)\le\mathcal{V}_{s,N}\le\frac{k^*}{N}+\gamma\sum_{k^*<i\le k^\dagger}\lambda_i+(s+N)\gamma^2\sum_{i>k^\dagger}\lambda_i^2 \tag*{(55)} 216 1 N k ∗ + γ k ∗ < i ≤ k † ∑ λ i + ( s + N ) γ 2 i > k † ∑ λ i 2 ≤ V s , N ≤ N k ∗ + γ k ∗ < i ≤ k † ∑ λ i + ( s + N ) γ 2 i > k † ∑ λ i 2 (55)
and
S s , N ≥ 1 864 ( 1 γ N ∥ w s d r i f t ∥ I 0 : k ∗ 2 + ∥ w s d r i f t ∥ H k ∗ : ∞ 2 ) Ξ N + 1 108 Φ b u r n ( s , N ) , \mathcal{S}_{s,N}\ge\frac{1}{864}\left(\frac{1}{\gamma N}\lVert w_s^{\mathrm{drift}}\rVert_{I_{0:k^*}}^{2}+\lVert w_s^{\mathrm{drift}}\rVert_{H_{k^*:\infty}}^{2}\right)\Xi_N+\frac{1}{108}\Phi_{\mathrm{burn}}(s,N), S s , N ≥ 864 1 ( γ N 1 ∥ w s drift ∥ I 0 : k ∗ 2 + ∥ w s drift ∥ H k ∗ : ∞ 2 ) Ξ N + 108 1 Φ burn ( s , N ) ,
S s , N ≤ 1 2 ( 1 γ N ∥ w s d r i f t ∥ I 0 : k ∗ 2 + ∥ w s d r i f t ∥ H k ∗ : ∞ 2 ) Ξ N + 1 2 Φ b u r n ( s , N ) . (56) \mathcal{S}_{s,N}\le\frac{1}{2}\left(\frac{1}{\gamma N}\lVert w_s^{\mathrm{drift}}\rVert_{I_{0:k^*}}^{2}+\lVert w_s^{\mathrm{drift}}\rVert_{H_{k^*:\infty}}^{2}\right)\Xi_N+\frac{1}{2}\Phi_{\mathrm{burn}}(s,N). \tag*{(56)} S s , N ≤ 2 1 ( γ N 1 ∥ w s drift ∥ I 0 : k ∗ 2 + ∥ w s drift ∥ H k ∗ : ∞ 2 ) Ξ N + 2 1 Φ burn ( s , N ) . (56)
Proof of Lemma A.1. We establish (A.9), (A.10)}, and (A.11)` in three parts.
Part 1: Evaluation of EffectiveBias
For each mode i ≥ 1 i\ge1 i ≥ 1 , the elementary inequalities 1 2 ( 1 ∧ γ N λ i ) ≤ ( 1 − e − 1 ) ( 1 ∧ γ N λ i ) ≤ 1 − ( 1 − γ λ i ) N ≤ 1 ∧ ( γ N λ i ) \frac{1}{2}(1\wedge\gamma N\lambda_i)\le(1-e^{-1})(1\wedge\gamma N\lambda_i)\le1-(1-\gamma\lambda_i)^N\le1\wedge(\gamma N\lambda_i) 2 1 ( 1 ∧ γ N λ i ) ≤ ( 1 − e − 1 ) ( 1 ∧ γ N λ i ) ≤ 1 − ( 1 − γ λ i ) N ≤ 1 ∧ ( γ N λ i ) imply
1 4 ( 1 { i ≤ k ∗ } + γ 2 N 2 λ i 2 1 { i > k ∗ } ) ≤ ( 1 − ( 1 − γ λ i ) N ) 2 ≤ 1 { i ≤ k ∗ } + γ 2 N 2 λ i 2 1 { i > k ∗ } . (57) \frac{1}{4}\left(\mathbf{1}_{\{i\le k^*\}}+\gamma^2N^2\lambda_i^2\mathbf{1}_{\{i>k^*\}}\right)\le\left(1-(1-\gamma\lambda_i)^N\right)^2\le\mathbf{1}_{\{i\le k^*\}}+\gamma^2N^2\lambda_i^2\mathbf{1}_{\{i>k^*\}}. \tag*{(57)} 4 1 ( 1 { i ≤ k ∗ } + γ 2 N 2 λ i 2 1 { i > k ∗ } ) ≤ ( 1 − ( 1 − γ λ i ) N ) 2 ≤ 1 { i ≤ k ∗ } + γ 2 N 2 λ i 2 1 { i > k ∗ } . (57)
Multiplying (A.12) by ω i 2 2 γ 2 N 2 λ i ( 1 − γ λ i ) 2 s = ( v i ⊤ w s d r i f t ) 2 2 γ 2 N 2 λ i \frac{\omega_i^2}{2\gamma^2N^2\lambda_i}(1-\gamma\lambda_i)^{2s}=\frac{(\mathbf{v}_i^\top w_s^{\mathrm{drift}})^2}{2\gamma^2N^2\lambda_i} 2 γ 2 N 2 λ i ω i 2 ( 1 − γ λ i ) 2 s = 2 γ 2 N 2 λ i ( v i ⊤ w s drift ) 2 and summing over i ≥ 1 i\ge1 i ≥ 1 in (10) directly yields (A.9).
Part 2: Evaluation of V s , N \mathcal{V}_{s,N} V s , N Across the Three Spectral Regimes
Using Ψ r , s , N = γ 2 F r + 1 \Psi_{r,s,N}=\gamma^2F_{r+1} Ψ r , s , N = γ 2 F r + 1 from (A.2), we have V s , N = 1 2 ∑ r = 0 s + N − 2 Ψ r , s , N \mathcal{V}_{s,N}=\frac{1}{2}\sum_{r=0}^{s+N-2}\Psi_{r,s,N} V s , N = 2 1 ∑ r = 0 s + N − 2 Ψ r , s , N . We split this sum into the averaging window r ∈ { s , … , s + N − 2 } r\in\{s,\ldots,s+N-2\} r ∈ { s , … , s + N − 2 } and the burn-in phase r ∈ { 0 , … , s − 1 } r\in\{0,\ldots,s-1\} r ∈ { 0 , … , s − 1 } :
Averaging-window contribution (s ≤ r ≤ s + N − 2 s\le r\le s+N-2 s ≤ r ≤ s + N − 2 ): Setting m = s + N − 1 − r ∈ { 1 , … , N − 1 } m=s+N-1-r\in\{1,\ldots,N-1\} m = s + N − 1 − r ∈ { 1 , … , N − 1 } gives
1 2 ∑ r = s s + N − 2 Ψ r , s , N = 1 2 N 2 ∑ m = 1 N − 1 ∑ i = 1 ∞ ( 1 − ( 1 − γ λ i ) m ) 2 ≤ N − 1 2 N 2 ∑ i = 1 ∞ ( 1 ∧ γ 2 N 2 λ i 2 ) ≤ 1 2 Ξ N , \begin{aligned}
\frac{1}{2}\sum_{r=s}^{s+N-2}\Psi_{r,s,N}=\frac{1}{2N^2}\sum_{m=1}^{N-1}\sum_{i=1}^{\infty}\left(1-(1-\gamma\lambda_i)^m\right)^2 \\
&\le\frac{N-1}{2N^2}\sum_{i=1}^{\infty}(1\wedge\gamma^2N^2\lambda_i^2)\le\frac{1}{2}\Xi_N,
\end{aligned} 2 1 r = s ∑ s + N − 2 Ψ r , s , N = 2 N 2 1 m = 1 ∑ N − 1 i = 1 ∑ ∞ ( 1 − ( 1 − γ λ i ) m ) 2 ≤ 2 N 2 N − 1 i = 1 ∑ ∞ ( 1 ∧ γ 2 N 2 λ i 2 ) ≤ 2 1 Ξ N ,
where the first inequality uses 1 − ( 1 − γ λ i ) m ≤ 1 ∧ ( γ N λ i ) 1-(1-\gamma\lambda_i)^m\le1\wedge(\gamma N\lambda_i) 1 − ( 1 − γ λ i ) m ≤ 1 ∧ ( γ N λ i ) . Conversely, for N ≥ 2 N\ge2 N ≥ 2 , restricting the sum to r ∈ { s , … , s + ⌊ N / 2 ⌋ − 1 } r\in\{s,\ldots,s+\lfloor N/2\rfloor-1\} r ∈ { s , … , s + ⌊ N /2 ⌋ − 1 } ensures m ≥ ⌊ N / 2 ⌋ m\ge\lfloor N/2\rfloor m ≥ ⌊ N /2 ⌋ , so
1 2 ∑ r = s s + N − 2 Ψ r , s , N ≥ 1 2 N 2 ∑ r = s s + ⌊ N / 2 ⌋ − 1 ∑ i = 1 ∞ ( 1 − ( 1 − γ λ i ) ⌊ N / 2 ⌋ ) 2 \frac{1}{2}\sum_{r=s}^{s+N-2}\Psi_{r,s,N}\ge\frac{1}{2N^2}\sum_{r=s}^{s+\lfloor N/2\rfloor-1}\sum_{i=1}^{\infty}\left(1-(1-\gamma\lambda_i)^{\lfloor N/2\rfloor}\right)^2 2 1 r = s ∑ s + N − 2 Ψ r , s , N ≥ 2 N 2 1 r = s ∑ s + ⌊ N /2 ⌋ − 1 i = 1 ∑ ∞ ( 1 − ( 1 − γ λ i ) ⌊ N /2 ⌋ ) 2
≥ ⌊ N / 2 ⌋ 18 N 2 ∑ i = 1 ∞ ( 1 − ( 1 − γ λ i ) N ) 2 ≥ 1 216 Ξ N , \ge\frac{\lfloor N/2\rfloor}{18N^2}\sum_{i=1}^{\infty}\left(1-\left(1-\gamma\lambda_i\right)^N\right)^2 \ge\frac{1}{216}\Xi_N, ≥ 18 N 2 ⌊ N /2 ⌋ i = 1 ∑ ∞ ( 1 − ( 1 − γ λ i ) N ) 2 ≥ 216 1 Ξ N ,
where the second inequality applies Lemma C.2 and the third inequality uses (A.12) together with ⌊ N / 2 ⌋ ≥ N / 3 \lfloor N/2\rfloor\ge N/3 ⌊ N /2 ⌋ ≥ N /3 for N ≥ 2 N \ge2 N ≥ 2 . Combining both bounds gives
1 216 ( k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 ) ≤ 1 2 ∑ r = s s + N − 2 Ψ r , s , N ≤ 1 2 ( k ∗ N + N γ 2 ∑ i > k ∗ λ i 2 ) . (58) \frac{1}{216}\left(\frac{k^*}{N}+N\gamma^2\sum_{i>k^*}\lambda_i^2\right)\le\frac{1}{2}\sum_{r=s}^{s+N-2}\Psi_{r,s,N}\le\frac{1}{2}\left(\frac{k^*}{N}+N\gamma^2\sum_{i>k^*}\lambda_i^2\right). \tag*{(58)} 216 1 ( N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 ) ≤ 2 1 r = s ∑ s + N − 2 Ψ r , s , N ≤ 2 1 ( N k ∗ + N γ 2 i > k ∗ ∑ λ i 2 ) . (58)
1 2 ∑ r = 0 s − 1 Ψ r , s , N = 1 2 N 2 ∑ i = 1 ∞ ( 1 − ( 1 − γ λ i ) N ) 2 1 − ( 1 − γ λ i ) 2 s γ λ i ( 2 − γ λ i ) . \frac{1}{2}\sum_{r=0}^{s-1}\Psi_{r,s,N}=\frac{1}{2N^2}\sum_{i=1}^{\infty}\left(1-\left(1-\gamma\lambda_i\right)^N\right)^2\frac{1-\left(1-\gamma\lambda_i\right)^{2s}}{\gamma\lambda_i(2-\gamma\lambda_i)}. 2 1 r = 0 ∑ s − 1 Ψ r , s , N = 2 N 2 1 i = 1 ∑ ∞ ( 1 − ( 1 − γ λ i ) N ) 2 γ λ i ( 2 − γ λ i ) 1 − ( 1 − γ λ i ) 2 s .
Since 1 4 ( s ∧ ( γ λ i ) − 1 ) ≤ 1 − ( 1 − γ λ i ) 2 s γ λ i ( 2 − γ λ i ) ≤ s ∧ ( γ λ i ) − 1 \frac{1}{4}\left(s\wedge(\gamma\lambda_i)^{-1}\right)\le\frac{1-\left(1-\gamma\lambda_i\right)^{2s}}{\gamma\lambda_i(2-\gamma\lambda_i)}\le s\wedge(\gamma\lambda_i)^{-1} 4 1 ( s ∧ ( γ λ i ) − 1 ) ≤ γ λ i ( 2 − γ λ i ) 1 − ( 1 − γ λ i ) 2 s ≤ s ∧ ( γ λ i ) − 1 , applying (A.12) shows that 1 2 ∑ r = 0 s − 1 Ψ r , s , N \frac{1}{2}\sum_{r=0}^{s-1}\Psi_{r,s,N} 2 1 ∑ r = 0 s − 1 Ψ r , s , N is sandwiched between 1 32 ∑ i ≥ 1 v i \frac{1}{32}\sum_{i\ge1}v_i 32 1 ∑ i ≥ 1 v i and 1 2 ∑ i ≥ 1 v i \frac{1}{2}\sum_{i\ge1}v_i 2 1 ∑ i ≥ 1 v i , where v i : = 1 N 2 ( 1 ∧ γ 2 N 2 λ i 2 ) min { s , ( γ λ i ) − 1 } v_i:=\frac{1}{N^2}(1\wedge\gamma^2N^2\lambda_i^2)\min\{s,(\gamma\lambda_i)^{-1}\} v i := N 2 1 ( 1 ∧ γ 2 N 2 λ i 2 ) min { s , ( γ λ i ) − 1 } . Adapting the three-regime spectral partitioning strategy of Zou et al. [37 ] , we evaluate v i v_i v i across the cutoffs k ∗ k^* k ∗ and k † k^\dagger k † :
Case 1 (i ≤ k ∗ i\le k^* i ≤ k ∗ , so γ λ i ≥ N − 1 \gamma\lambda_i\ge N^{-1} γ λ i ≥ N − 1 ): Here 1 ∧ γ 2 N 2 λ i 2 = 1 1\wedge\gamma^2N^2\lambda_i^2=1 1 ∧ γ 2 N 2 λ i 2 = 1 and min { s , ( γ λ i ) − 1 } ≤ ( γ λ i ) − 1 ≤ N \min\{s,(\gamma\lambda_i)^{-1}\}\le(\gamma\lambda_i)^{-1}\le N min { s , ( γ λ i ) − 1 } ≤ ( γ λ i ) − 1 ≤ N , so 0 ≤ v i ≤ 1 N 0\le v_i\le\frac{1}{N} 0 ≤ v i ≤ N 1 .
Case 2 (k ∗ < i ≤ k † k^*<i\le k^\dagger k ∗ < i ≤ k † , so ( s + N ) − 1 ≤ γ λ i < N − 1 (s+N)^{-1}\le\gamma\lambda_i<N^{-1} ( s + N ) − 1 ≤ γ λ i < N − 1 ): Here 1 ∧ γ 2 N 2 λ i 2 = γ 2 N 2 λ i 2 1\wedge\gamma^2N^2\lambda_i^2=\gamma^2N^2\lambda_i^2 1 ∧ γ 2 N 2 λ i 2 = γ 2 N 2 λ i 2 , so v i = γ 2 λ i 2 min { s , ( γ λ i ) − 1 } v_i=\gamma^2\lambda_i^2\min\{s,(\gamma\lambda_i)^{-1}\} v i = γ 2 λ i 2 min { s , ( γ λ i ) − 1 } . In the upper bound, min { s , ( γ λ i ) − 1 } ≤ ( γ λ i ) − 1 \min\{s,(\gamma\lambda_i)^{-1}\}\le(\gamma\lambda_i)^{-1} min { s , ( γ λ i ) − 1 } ≤ ( γ λ i ) − 1 gives v i ≤ γ λ i v_i\le\gamma\lambda_i v i ≤ γ λ i . In the lower bound, if s ≥ ( γ λ i ) − 1 s\ge(\gamma\lambda_i)^{-1} s ≥ ( γ λ i ) − 1 , then v i = γ λ i v_i=\gamma\lambda_i v i = γ λ i ; if s < ( γ λ i ) − 1 ≤ s + N s<(\gamma\lambda_i)^{-1}\le s+N s < ( γ λ i ) − 1 ≤ s + N , then adding the averaging-window contribution N γ 2 λ i 2 N\gamma^2\lambda_i^2 N γ 2 λ i 2 from (A.13) gives v i + N γ 2 λ i 2 = ( s + N ) γ 2 λ i 2 ≥ γ λ i v_i+N\gamma^2\lambda_i^2=(s+N)\gamma^2\lambda_i^2\ge\gamma\lambda_i v i + N γ 2 λ i 2 = ( s + N ) γ 2 λ i 2 ≥ γ λ i .
Case 3 (i > k † i>k^\dagger i > k † , so γ λ i < ( s + N ) − 1 ≤ s − 1 \gamma\lambda_i<(s+N)^{-1}\le s^{-1} γ λ i < ( s + N ) − 1 ≤ s − 1 ): Here 1 ∧ γ 2 N 2 λ i 2 = γ 2 N 2 λ i 2 1\wedge\gamma^2N^2\lambda_i^2=\gamma^2N^2\lambda_i^2 1 ∧ γ 2 N 2 λ i 2 = γ 2 N 2 λ i 2 and min { s , ( γ λ i ) − 1 } = s \min\{s,(\gamma\lambda_i)^{-1}\}=s min { s , ( γ λ i ) − 1 } = s , so v i = s γ 2 λ i 2 v_i=s\gamma^2\lambda_i^2 v i = s γ 2 λ i 2 .
Summing Cases 1–3 and combining with (A.13) (noting that for k ∗ < i ≤ k † k^*<i\le k^\dagger k ∗ < i ≤ k † , N γ 2 λ i 2 ≤ γ λ i N\gamma^2\lambda_i^2\le\gamma\lambda_i N γ 2 λ i 2 ≤ γ λ i in the upper bound) yields (A.10).
Part 3: Evaluation of S s , N S_{s,N} S s , N
Writing b r = ∥ ( I − γ H ) r ( w 0 − w ∗ ) ∥ 2 2 = ∑ j ≥ 1 λ j ω j 2 ( 1 − γ λ j ) 2 r b_r=\|(I-\gamma H)^r(w_0-w^*)\|_2^2=\sum_{j\ge1}\lambda_j\omega_j^2(1-\gamma\lambda_j)^{2r} b r = ∥ ( I − γ H ) r ( w 0 − w ∗ ) ∥ 2 2 = ∑ j ≥ 1 λ j ω j 2 ( 1 − γ λ j ) 2 r , we have S s , N = 1 2 ∑ r = 0 s + N − 2 b r Ψ r , s , N S_{s,N}=\frac{1}{2}\sum_{r=0}^{s+N-2}b_r\Psi_{r,s,N} S s , N = 2 1 ∑ r = 0 s + N − 2 b r Ψ r , s , N . We split this sum into the averaging window r = s + k r=s+k r = s + k (0 ≤ k ≤ N − 2 0\le k\le N-2 0 ≤ k ≤ N − 2 ) and the burn-in phase 0 ≤ r ≤ s − 1 0\le r\le s-1 0 ≤ r ≤ s − 1 :
Averaging-window contribution (r = s + k r=s+k r = s + k , 0 ≤ k ≤ N − 2 0\le k\le N-2 0 ≤ k ≤ N − 2 ): For N ≥ 2 N\ge2 N ≥ 2 ,5 note first from Part 2 that Ψ s + k , s , N ≤ Ψ s , s , N ≤ 1 3 Ξ N \Psi_{s+k,s,N}\le\Psi_{s,s,N}\le\frac{1}{3}\Xi_N Ψ s + k , s , N ≤ Ψ s , s , N ≤ 3 1 Ξ N for all 0 ≤ k ≤ N − 2 0\le k\le N-2 0 ≤ k ≤ N − 2 , while Lemma C.2 and (A.12) give Ψ s + k , s , N ≥ 1 36 Ξ N \Psi_{s+k,s,N}\ge\frac{1}{36}\Xi_N Ψ s + k , s , N ≥ 36 1 Ξ N for all 0 ≤ k ≤ ⌊ N / 2 ⌋ − 1 0\le k\le\lfloor N/2\rfloor-1 0 ≤ k ≤ ⌊ N /2 ⌋ − 1 . Moreover, since b s + k = ∑ j ≥ 1 λ j ( v j T w s d r i f t ) 2 ( 1 − γ λ j ) 2 k b_{s+k}=\sum_{j\ge1}\lambda_j(v_j^\mathsf{T}w_s^{\mathrm{drift}})^2(1-\gamma\lambda_j)^{2k} b s + k = ∑ j ≥ 1 λ j ( v j T w s drift ) 2 ( 1 − γ λ j ) 2 k , summing the geometric series over k k k gives
γ ∑ k = 0 N − 1 b s + k = ∑ j = 1 ∞ 1 − ( 1 − γ λ j ) 2 N 2 − γ λ j ( v j T w s d r i f t ) 2 ≤ ∥ w s d r i f t ∥ 0 : k ∗ 2 + γ N ∥ w s d r i f t ∥ H k ∗ : ∞ 2 . \begin{aligned}
\gamma\sum_{k=0}^{N-1}b_{s+k}=\sum_{j=1}^{\infty}\frac{1-(1-\gamma\lambda_j)^{2N}}{2-\gamma\lambda_j}(v_j^\mathsf{T}w_s^{\mathrm{drift}})^2 \\
&\le\|w_s^{\mathrm{drift}}\|_{0:k^*}^2+\gamma N\|w_s^{\mathrm{drift}}\|_{H_{k^*: \infty}}^2.
\end{aligned} γ k = 0 ∑ N − 1 b s + k = j = 1 ∑ ∞ 2 − γ λ j 1 − ( 1 − γ λ j ) 2 N ( v j T w s drift ) 2 ≤ ∥ w s drift ∥ 0 : k ∗ 2 + γ N ∥ w s drift ∥ H k ∗ : ∞ 2 .
where the inequality uses 1 − ( 1 − γ λ j ) 2 N 2 − γ λ j ≤ 1 \frac{1-(1-\gamma\lambda_j)^{2N}}{2-\gamma\lambda_j}\le1 2 − γ λ j 1 − ( 1 − γ λ j ) 2 N ≤ 1 on j ≤ k ∗ j\le k^* j ≤ k ∗ (since 2 − γ λ j ≥ 1 2-\gamma\lambda_j\ge1 2 − γ λ j ≥ 1 ) and γ ∑ k = 0 N − 1 λ j ( 1 − γ λ j ) 2 k ≤ γ N λ j \gamma\sum_{k=0}^{N-1}\lambda_j(1-\gamma\lambda_j)^{2k}\le\gamma N\lambda_j γ ∑ k = 0 N − 1 λ j ( 1 − γ λ j ) 2 k ≤ γ N λ j on j > k ∗ j>k^* j > k ∗ . Conversely, restricting the sum to 0 ≤ k ≤ ⌊ N / 2 ⌋ − 1 0\le k\le\lfloor N/2\rfloor-1 0 ≤ k ≤ ⌊ N /2 ⌋ − 1 yields
γ ∑ k = 0 ⌊ N / 2 ⌋ − 1 b s + k = ∑ j = 1 ∞ 1 − ( 1 − γ λ j ) 2 ⌊ N / 2 ⌋ 2 − γ λ j ( v j T w s d r i f t ) 2 ≥ 1 6 ∑ j = 1 ∞ [ 1 − ( 1 − γ λ j ) 2 N ] ( v j T w s d r i f t ) 2 ≥ 1 12 ( ∥ w s d r i f t ∥ I 0 : k ∗ 2 + γ N ∥ w s d r i f t ∥ H k ∗ : ∞ 2 ) , \begin{aligned}
\gamma\sum_{k=0}^{\lfloor N/2\rfloor-1}b_{s+k}
=\sum_{j=1}^{\infty}\frac{1-(1-\gamma\lambda_j)^{2\lfloor N/2\rfloor}}{2-\gamma\lambda_j}(\mathbf{v}_j^{\mathsf{T}}\mathbf{w}_s^{\mathrm{drift}})^2 \\
&\ge\frac{1}{6}\sum_{j=1}^{\infty}\left[1-(1-\gamma\lambda_j)^{2N}\right](\mathbf{v}_j^{\mathsf{T}}\mathbf{w}_s^{\mathrm{drift}})^2 \\
&\ge\frac{1}{12}\left(\|\mathbf{w}_s^{\mathrm{drift}}\|_{I_{0:k^*}}^2+\gamma N\|\mathbf{w}_s^{\mathrm{drift}}\|_{H_{k^*:\infty}}^2\right),
\end{aligned} γ k = 0 ∑ ⌊ N /2 ⌋ − 1 b s + k = j = 1 ∑ ∞ 2 − γ λ j 1 − ( 1 − γ λ j ) 2 ⌊ N /2 ⌋ ( v j T w s drift ) 2 ≥ 6 1 j = 1 ∑ ∞ [ 1 − ( 1 − γ λ j ) 2 N ] ( v j T w s drift ) 2 ≥ 12 1 ( ∥ w s drift ∥ I 0 : k ∗ 2 + γ N ∥ w s drift ∥ H k ∗ : ∞ 2 ) ,
where the first inequality applies Lemma C.2 together with 2 − γ λ j ≤ 2 2-\gamma\lambda_j\le2 2 − γ λ j ≤ 2 , and the second inequality uses 1 − ( 1 − γ λ j ) 2 N ≥ 1 − ( 1 − γ λ j ) N ≥ 1 2 ( 1 ∧ γ N λ j ) 1-(1-\gamma\lambda_j)^{2N}\ge1-(1-\gamma\lambda_j)^N\ge\frac{1}{2}(1\wedge\gamma N\lambda_j) 1 − ( 1 − γ λ j ) 2 N ≥ 1 − ( 1 − γ λ j ) N ≥ 2 1 ( 1 ∧ γ N λ j ) . Combining the filter and geometric-sum bounds yields
1 864 ( 1 γ N ∥ w s d r i f t ∥ I 0 : k ∗ 2 + ∥ w s d r i f t ∥ H k ∗ : ∞ 2 ) Ξ N ≤ 1 2 ∑ k = 0 N − 2 b s + k Ψ s + k , s , N ≤ 1 2 ( 1 γ N ∥ w s d r i f t ∥ I 0 : k ∗ 2 + ∥ w s d r i f t ∥ H k ∗ : ∞ 2 ) Ξ N . (59) \begin{aligned}
\frac{1}{864}\left(\frac{1}{\gamma N}\|\mathbf{w}_s^{\mathrm{drift}}\|_{I_{0:k^*}}^2+\|\mathbf{w}_s^{\mathrm{drift}}\|_{H_{k^*:\infty}}^2\right)\Xi_N
\le\frac{1}{2}\sum_{k=0}^{N-2}b_{s+k}\Psi_{s+k,s,N} \\
&\le\frac{1}{2}\left(\frac{1}{\gamma N}\|\mathbf{w}_s^{\mathrm{drift}}\|_{I_{0:k^*}}^2+\|\mathbf{w}_s^{\mathrm{drift}}\|_{H_{k^*:\infty}}^2\right)\Xi_N. \tag*{(59)}
\end{aligned} 864 1 ( γ N 1 ∥ w s drift ∥ I 0 : k ∗ 2 + ∥ w s drift ∥ H k ∗ : ∞ 2 ) Ξ N ≤ 2 1 k = 0 ∑ N − 2 b s + k Ψ s + k , s , N ≤ 2 1 ( γ N 1 ∥ w s drift ∥ I 0 : k ∗ 2 + ∥ w s drift ∥ H k ∗ : ∞ 2 ) Ξ N . (59)
Burn-in contribution (0 ≤ r ≤ s − 1 0\le r\le s-1 0 ≤ r ≤ s − 1 , when s ≥ 1 s\ge1 s ≥ 1 ): Expanding b r b_r b r and Ψ r , s , N \Psi_{r,s,N} Ψ r , s , N gives
1 2 ∑ r = 0 s − 1 b r Ψ r , s , N = 1 2 N 2 ∑ i = 1 ∞ ( 1 − ( 1 − γ λ i ) N ) 2 ∑ j = 1 ∞ λ j ω j 2 ∑ r = 0 s − 1 ( 1 − γ λ j ) 2 r ( 1 − γ λ i ) 2 ( s − 1 − r ) . \frac{1}{2}\sum_{r=0}^{s-1}b_r\Psi_{r,s,N}
=\frac{1}{2N^2}\sum_{i=1}^{\infty}\left(1-(1-\gamma\lambda_i)^N\right)^2\sum_{j=1}^{\infty}\lambda_j\omega_j^2\sum_{r=0}^{s-1}(1-\gamma\lambda_j)^{2r}(1-\gamma\lambda_i)^{2(s-1-r)}. 2 1 r = 0 ∑ s − 1 b r Ψ r , s , N = 2 N 2 1 i = 1 ∑ ∞ ( 1 − ( 1 − γ λ i ) N ) 2 j = 1 ∑ ∞ λ j ω j 2 r = 0 ∑ s − 1 ( 1 − γ λ j ) 2 r ( 1 − γ λ i ) 2 ( s − 1 − r ) .
For each mode pair ( i , j ) (i,j) ( i , j ) , let p : = ( 1 − γ λ j ) 2 p:=(1-\gamma\lambda_j)^2 p := ( 1 − γ λ j ) 2 and q : = ( 1 − γ λ i ) 2 q:=(1-\gamma\lambda_i)^2 q := ( 1 − γ λ i ) 2 , so that M : = p ∨ q = ( 1 − γ ( λ i ∧ λ j ) ) 2 ∈ [ 1 / 4 , 1 ] M:=p\vee q=(1-\gamma(\lambda_i\wedge\lambda_j))^2\in[1/4,1] M := p ∨ q = ( 1 − γ ( λ i ∧ λ j ) ) 2 ∈ [ 1/4 , 1 ] (since γ λ 1 ≤ 1 / 2 \gamma\lambda_1\le1/2 γ λ 1 ≤ 1/2 ) and M − m = γ ∣ λ i − λ j ∣ ( 2 − γ ( λ i + λ j ) ) ∈ [ γ ∣ λ i − λ j ∣ , 2 γ ∣ λ i − λ j ∣ ] M-m=\gamma|\lambda_i-\lambda_j|(2-\gamma(\lambda_i+\lambda_j))\in[\gamma|\lambda_i-\lambda_j|,2\gamma|\lambda_i-\lambda_j|] M − m = γ ∣ λ i − λ j ∣ ( 2 − γ ( λ i + λ j )) ∈ [ γ ∣ λ i − λ j ∣ , 2 γ ∣ λ i − λ j ∣ ] . Consequently,
1 8 γ ∣ λ i − λ j ∣ ≤ M M − m ≤ 1 γ ∣ λ i − λ j ∣ . \frac{1}{8\gamma|\lambda_i-\lambda_j|}\le\frac{M}{M-m}\le\frac{1}{\gamma|\lambda_i-\lambda_j|}. 8 γ ∣ λ i − λ j ∣ 1 ≤ M − m M ≤ γ ∣ λ i − λ j ∣ 1 .
Applying Lemma C.3 to the inner geometric convolution over r r r and (A.12) to ( 1 − ( 1 − γ λ i ) N ) 2 (1-(1-\gamma\lambda_i)^N)^2 ( 1 − ( 1 − γ λ i ) N ) 2 term-by-term yields
1 108 Φ b u r n ( s , N ) ≤ 1 − e − 1 64 Φ b u r n ( s , N ) ≤ 1 2 ∑ r = 0 s − 1 b r Ψ r , s , N ≤ 1 2 Φ b u r n ( s , N ) . (60) \frac{1}{108}\Phi_{\mathrm{burn}}(s,N)\le\frac{1-e^{-1}}{64}\Phi_{\mathrm{burn}}(s,N)\le\frac{1}{2}\sum_{r=0}^{s-1}b_r\Psi_{r,s,N}\le\frac{1}{2}\Phi_{\mathrm{burn}}(s,N). \tag*{(60)} 108 1 Φ burn ( s , N ) ≤ 64 1 − e − 1 Φ burn ( s , N ) ≤ 2 1 r = 0 ∑ s − 1 b r Ψ r , s , N ≤ 2 1 Φ burn ( s , N ) . (60)
Adding (A.14) and (A.15) establishes (A.11)`.
Proofs of Main Theorems, Propositions, and Corollaries
Proof of Theorem 4.1, Theorem 4.2, and Corollary 4.5
Proof of Theorem 4.1 and Theorem 4.2. For Theorem 4.1, Lemma C.6(1) gives E [ L ( w ‾ s : s + N ) ] − L ( w ∗ ) ≤ 2 ⋅ b i a s + 2 ⋅ v a r i a n c e = 2 ⋅ E f f e c t i v e B i a s + 2 ⋅ E f f e c t i v e V a r \mathbb{E}[L(\overline{\mathbf{w}}_{s:s+N})]-L(\mathbf{w}^*)\le2\cdot\mathrm{bias}+2\cdot\mathrm{variance}=2\cdot\mathrm{EffectiveBias}+2\cdot\mathrm{EffectiveVar} E [ L ( w s : s + N )] − L ( w ∗ ) ≤ 2 ⋅ bias + 2 ⋅ variance = 2 ⋅ EffectiveBias + 2 ⋅ EffectiveVar under Assumptions 3.1–3.3 (and equality without the factor of 2 under the well-specified model (6) by Lemma C.6(2)). Invoking Lemma 5.2 (1) and substituting the upper bounds (A.9)–(A.11) from Lemma A.1 directly establishes (14) and (4.4).
For Theorem 4.2, Lemma C.6(2) gives the exact decomposition E [ L ( w ‾ s : s + N ) ] − L ( w ∗ ) = E f f e c t i v e B i a s + E f f e c t i v e V a r \mathbb{E}[L(\overline{\mathbf{w}}_{s:s+N})]-L(\mathbf{w}^*)=\mathrm{EffectiveBias}+\mathrm{EffectiveVar} E [ L ( w s : s + N )] − L ( w ∗ ) = EffectiveBias + EffectiveVar under (6) . Invoking Lemma 5.2 (2) and substituting the lower bounds (A.9)–(A.11) from Lemma A.1 directly establishes (18) and (19) .
Proof of Corollary 4.5. Suppose c 0 N ≤ s ≤ C 0 N c_0N \le s \le C_0N c 0 N ≤ s ≤ C 0 N , and recall w s d r i f t \mathbf{w}^{\mathrm{drift}}_s w s drift , ω j \omega_j ω j , and Φ b u r n ( s , N ) \Phi_{\mathrm{burn}}(s,N) Φ burn ( s , N ) from (A.1) and (A.8). For every slow mode i > k ∗ i > k^* i > k ∗ , we have γ λ i < N − 1 ≤ C 0 / s \gamma\lambda_i < N^{-1} \le C_0/s γ λ i < N − 1 ≤ C 0 / s , which implies ( 1 − γ λ i ) 2 s ∈ [ e − 4 C 0 , 1 ] (1-\gamma\lambda_i)^{2s} \in[e^{-4C_0},1] ( 1 − γ λ i ) 2 s ∈ [ e − 4 C 0 , 1 ] and hence ∥ w s d r i f t ∥ H k ∗ : ∞ 2 ≍ ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 \lVert\mathbf{w}^{\mathrm{drift}}_s\rVert_{H_{k^*:\infty}}^2 \asymp\lVert\mathbf{w}_0-\mathbf{w}^*\rVert_{H_{k^*:\infty}}^2 ∥ w s drift ∥ H k ∗ : ∞ 2 ≍ ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 . Similarly, for every intermediate mode i ∈ ( k ∗ , k † ] i \in(k^*,k^\dagger] i ∈ ( k ∗ , k † ] , we have ( ( 1 + C 0 ) N ) − 1 ≤ γ λ i < N − 1 ((1+C_0)N)^{-1} \le\gamma\lambda_i < N^{-1} (( 1 + C 0 ) N ) − 1 ≤ γ λ i < N − 1 , so γ λ i ≍ N γ 2 λ i 2 ≍ N − 1 \gamma\lambda_i \asymp N\gamma^2\lambda_i^2 \asymp N^{-1} γ λ i ≍ N γ 2 λ i 2 ≍ N − 1 .
To evaluate Φ b u r n ( s , N ) \Phi_{\mathrm{burn}}(s,N) Φ burn ( s , N ) in (A.8), we partition the double sum over ( i , j ) (i,j) ( i , j ) into three blocks:
Fast-to-fast block (i ≤ k ∗ i \le k^* i ≤ k ∗ and j ≤ k ∗ j \le k^* j ≤ k ∗ ): Since min { N − 2 , γ 2 λ i 2 } = N − 2 \min\{N^{-2},\gamma^2\lambda_i^2\}=N^{-2} min { N − 2 , γ 2 λ i 2 } = N − 2 on i ≤ k ∗ i \le k^* i ≤ k ∗ , this block equals the third line of (27) .
Slow-to-fast block (i ≤ k ∗ i \le k^* i ≤ k ∗ and j > k ∗ j > k^* j > k ∗ ): Using min { N − 2 , γ 2 λ i 2 } = N − 2 \min\{N^{-2},\gamma^2\lambda_i^2\}=N^{-2} min { N − 2 , γ 2 λ i 2 } = N − 2 and ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) ≤ 1 (1-\gamma(\lambda_i\wedge\lambda_j))^{2(s-1)}\le1 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) ≤ 1 , we upper-bound this block by
1 N 2 ∑ i ≤ k ∗ ∑ j > k ∗ λ j ω j 2 min { s , 1 γ ∣ λ i − λ j ∣ } ≤ 2 ∨ C 0 N ∑ i ≤ k ∗ ∑ j > k ∗ λ j ω j 2 = ( 2 ∨ C 0 ) ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 k ∗ N . \begin{aligned}
\frac{1}{N^2}\sum_{i\le k^*}\sum_{j>k^*}\lambda_j\omega_j^2\min\left\{s,\frac{1}{\gamma\lvert\lambda_i-\lambda_j\rvert}\right\}\le\frac{2\vee C_0}{N}\sum_{i\le k^*}\sum_{j>k^*}\lambda_j\omega_j^2 \\
&= (2\vee C_0)\lVert\mathbf{w}_0-\mathbf{w}^*\rVert_{H_{k^*:\infty}}^2\frac{k^*}{N}.
\end{aligned} N 2 1 i ≤ k ∗ ∑ j > k ∗ ∑ λ j ω j 2 min { s , γ ∣ λ i − λ j ∣ 1 } ≤ N 2 ∨ C 0 i ≤ k ∗ ∑ j > k ∗ ∑ λ j ω j 2 = ( 2 ∨ C 0 ) ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 N k ∗ .
where the inequality uses min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } ≤ 2 ( γ λ i ) − 1 ≤ 2 N \min\{s,(\gamma\lvert\lambda_i-\lambda_j\rvert)^{-1}\}\le2(\gamma\lambda_i)^{-1}\le2N min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } ≤ 2 ( γ λ i ) − 1 ≤ 2 N when λ j ≤ λ i / 2 \lambda_j\le\lambda_i/2 λ j ≤ λ i /2 and min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } ≤ s ≤ C 0 N \min\{s,(\gamma\lvert\lambda_i-\lambda_j\rvert)^{-1}\}\le s\le C_0N min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } ≤ s ≤ C 0 N when λ j > λ i / 2 \lambda_j>\lambda_i/2 λ j > λ i /2 . This contribution is absorbed into the second line of (27) .
Slow receiving block (i > k ∗ i > k^* i > k ∗ and j ≥ 1 j \ge1 j ≥ 1 ): On i > k ∗ i > k^* i > k ∗ , we have min { N − 2 , γ 2 λ i 2 } = γ 2 λ i 2 \min\{N^{-2},\gamma^2\lambda_i^2\}=\gamma^2\lambda_i^2 min { N − 2 , γ 2 λ i 2 } = γ 2 λ i 2 and λ i ∧ λ j ≤ λ i < ( γ N ) − 1 \lambda_i\wedge\lambda_j\le\lambda_i<(\gamma N)^{-1} λ i ∧ λ j ≤ λ i < ( γ N ) − 1 , so ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) ∈ [ e − 4 C 0 , 1 ] (1-\gamma(\lambda_i\wedge\lambda_j))^{2(s-1)}\in[e^{-4C_0},1] ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) ∈ [ e − 4 C 0 , 1 ] . Moreover, for j ≤ k ∗ j\le k^* j ≤ k ∗ , we have min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } ≍ ( γ λ j ) − 1 \min\{s,(\gamma\lvert\lambda_i-\lambda_j\rvert)^{-1}\}\asymp(\gamma\lambda_j)^{-1} min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } ≍ ( γ λ j ) − 1 (both when λ j ≥ 2 λ i \lambda_j\ge2\lambda_i λ j ≥ 2 λ i and when λ i < λ j < 2 λ i \lambda_i<\lambda_j<2\lambda_i λ i < λ j < 2 λ i , since s ≍ N ≍ ( γ λ j ) − 1 s\asymp N\asymp(\gamma\lambda_j)^{-1} s ≍ N ≍ ( γ λ j ) − 1 in the latter case), whereas for j > k ∗ j>k^* j > k ∗ , both γ λ i < N − 1 \gamma\lambda_i<N^{-1} γ λ i < N − 1 and γ λ j < N − 1 \gamma\lambda_j<N^{-1} γ λ j < N − 1 imply min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } ≍ N \min\{s,(\gamma\lvert\lambda_i-\lambda_j\rvert)^{-1}\}\asymp N min { s , ( γ ∣ λ i − λ j ∣ ) − 1 } ≍ N . Summing over j ≥ 1 j\ge1 j ≥ 1 and i > k ∗ i>k^* i > k ∗ therefore gives
∑ i > k ∗ γ 2 λ i 2 ∑ j = 1 ∞ λ j ω j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) min { s , 1 γ ∣ λ i − λ j ∣ } \sum_{i>k^*}\gamma^2\lambda_i^2\sum_{j=1}^{\infty}\lambda_j\omega_j^2(1-\gamma(\lambda_i\wedge\lambda_j))^{2(s-1)}\min\left\{s,\frac{1}{\gamma\lvert\lambda_i-\lambda_j\rvert}\right\} i > k ∗ ∑ γ 2 λ i 2 j = 1 ∑ ∞ λ j ω j 2 ( 1 − γ ( λ i ∧ λ j ) ) 2 ( s − 1 ) min { s , γ ∣ λ i − λ j ∣ 1 }
≍ ( ∑ i > k ∗ γ 2 λ i 2 ) ( 1 γ ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 + N ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 ) = ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 + N γ ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 N γ ( N γ 2 ∑ i > k ∗ λ i 2 ) , \begin{aligned}
\asymp\left(\sum_{i>k^*}\gamma^2\lambda_i^2\right)\left(\frac{1}{\gamma}\lVert\mathbf{w}_0-\mathbf{w}^*\rVert_{I_{0:k^*}}^2+N\lVert\mathbf{w}_0-\mathbf{w}^*\rVert_{H_{k^*:\infty}}^2\right) \\
&= \frac{\lVert\mathbf{w}_0-\mathbf{w}^*\rVert_{I_{0:k^*}}^2+N\gamma\lVert\mathbf{w}_0-\mathbf{w}^*\rVert_{H_{k^*:\infty}}^2}{N\gamma}\left(N\gamma^2\sum_{i>k^*}\lambda_i^2\right),
\end{aligned} ≍ ( i > k ∗ ∑ γ 2 λ i 2 ) ( γ 1 ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 + N ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 ) = N γ ∥ w 0 − w ∗ ∥ I 0 : k ∗ 2 + N γ ∥ w 0 − w ∗ ∥ H k ∗ : ∞ 2 ( N γ 2 i > k ∗ ∑ λ i 2 ) ,
which yields the first line of (27) .
Substituting these evaluations into Theorems 4.1 and 4.2 establishes (26) –(27) .
Proof of Proposition 4.3 and Corollary 4.4
Proof of Proposition 4.3. Expanding q t ( λ i ) 2 = 1 N 2 ∑ a = s ∨ t T ∑ b = s ∨ t T ( 1 − γ λ i ) a + b − 2 t q_t(\lambda_i)^2=\frac{1}{N^2}\sum_{a=s\vee t}^{T}\sum_{b=s\vee t}^{T}(1-\gamma\lambda_i)^{a+b-2t} q t ( λ i ) 2 = N 2 1 ∑ a = s ∨ t T ∑ b = s ∨ t T ( 1 − γ λ i ) a + b − 2 t in (12) and summing against λ i 2 \lambda_i^2 λ i 2 yields the pair-midpoint identity (21) . When T ≤ 1 T\le1 T ≤ 1 , the maximum in (13) is empty so χ s , N = 0 \chi_{s,N}=0 χ s , N = 0 . Suppose T ≥ 2 T\ge2 T ≥ 2 and fix any r ∈ { 1 , … , T − 1 } r\in\{1,\ldots,T-1\} r ∈ { 1 , … , T − 1 } . Since s ∨ t ≥ s ∨ ( r + 1 ) s\vee t\ge s\vee(r+1) s ∨ t ≥ s ∨ ( r + 1 ) for all t ≥ r + 1 t\ge r+1 t ≥ r + 1 , substituting (21) into γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t \gamma^2\sum_{t=r+1}^{T}k(t-r-1)F_t γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t and exchanging the order of summation gives
γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t = γ 2 N 2 ∑ a = s ∨ ( r + 1 ) T ∑ b = s ∨ ( r + 1 ) T ∑ t = r + 1 min ( a , b ) k ( t − r − 1 ) k ( a + b 2 − t ) . (61) \gamma^2\sum_{t=r+1}^{T}k(t-r-1)F_t=\frac{\gamma^2}{N^2}\sum_{a=s\vee(r+1)}^{T}\sum_{b=s\vee(r+1)}^{T}\sum_{t=r+1}^{\min(a,b)}k(t-r-1)k\left(\frac{a+b}{2}-t\right). \tag*{(61)} γ 2 t = r + 1 ∑ T k ( t − r − 1 ) F t = N 2 γ 2 a = s ∨ ( r + 1 ) ∑ T b = s ∨ ( r + 1 ) ∑ T t = r + 1 ∑ m i n ( a , b ) k ( t − r − 1 ) k ( 2 a + b − t ) . (61)
For each pair ( a , b ) (a,b) ( a , b ) , set j : = t − r − 1 ≥ 0 j := t-r-1 \ge0 j := t − r − 1 ≥ 0 and h : = a + b 2 − r ∈ [ 1 , T ] h := \frac{a+b}{2}-r \in[1,T] h := 2 a + b − r ∈ [ 1 , T ] , so the inner sum in (B.1) runs over 0 ≤ j ≤ min ( a , b ) − r − 1 ≤ T − 2 0 \le j \le\min(a,b)-r-1 \le T-2 0 ≤ j ≤ min ( a , b ) − r − 1 ≤ T − 2 with h − 1 − j = a + b 2 − t ≥ 0 h-1-j=\frac{a+b}{2}-t \ge0 h − 1 − j = 2 a + b − t ≥ 0 .
For part (1), for every 0 ≤ j ≤ min ( a , b ) − r − 1 0 \le j \le\min(a,b)-r-1 0 ≤ j ≤ min ( a , b ) − r − 1 , we have
k ( j ) k ( h − 1 − j ) ≤ tr ( H 2 ) k ( h − 1 ) = tr ( H 2 ) ∑ i = 1 ∞ λ i 2 ( 1 − γ λ i ) 2 h − 2 ≤ tr ( H 2 ) ( 1 − γ λ 1 ) 2 k ( h ) . \begin{aligned}
k(j)k(h-1-j) \le\operatorname{tr}(\mathbf{H}^{2})k(h-1) \\
&= \operatorname{tr}(\mathbf{H}^{2})\sum_{i=1}^{\infty}\lambda_i^{2}(1-\gamma\lambda_i)^{2h-2}
\le\frac{\operatorname{tr}(\mathbf{H}^{2})}{(1-\gamma\lambda_1)^2}k(h).
\end{aligned} k ( j ) k ( h − 1 − j ) ≤ tr ( H 2 ) k ( h − 1 ) = tr ( H 2 ) i = 1 ∑ ∞ λ i 2 ( 1 − γ λ i ) 2 h − 2 ≤ ( 1 − γ λ 1 ) 2 tr ( H 2 ) k ( h ) .
where the first inequality follows from Chebyshev’s sum inequality (Lemma C.1 with u = j u=j u = j and v = h − 1 − j v=h-1-j v = h − 1 − j ) and the second inequality uses ( 1 − γ λ i ) − 2 ≤ ( 1 − γ λ 1 ) − 2 (1-\gamma\lambda_i)^{-2} \le(1-\gamma\lambda_1)^{-2} ( 1 − γ λ i ) − 2 ≤ ( 1 − γ λ 1 ) − 2 . Since the inner sum over j j j in (B.1) contains at most T − 1 T-1 T − 1 terms and 1 N 2 ∑ a , b = s ∨ ( r + 1 ) T k ( a + b 2 − r ) ≤ 1 N 2 ∑ a , b = s ∨ r T k ( a + b 2 − r ) = F r \frac{1}{N^2}\sum_{a,b=s\vee(r+1)}^T k\left(\frac{a+b}{2}-r\right) \le\frac{1}{N^2}\sum_{a,b=s\vee r}^T k\left(\frac{a+b}{2}-r\right)=F_r N 2 1 ∑ a , b = s ∨ ( r + 1 ) T k ( 2 a + b − r ) ≤ N 2 1 ∑ a , b = s ∨ r T k ( 2 a + b − r ) = F r , summing over a , b a,b a , b yields
γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t ≤ γ 2 ( T − 1 ) tr ( H 2 ) ( 1 − γ λ 1 ) 2 F r , \gamma^2\sum_{t=r+1}^{T}k(t-r-1)F_t \le\frac{\gamma^2(T-1)\operatorname{tr}(\mathbf{H}^2)}{(1-\gamma\lambda_1)^2}F_r, γ 2 t = r + 1 ∑ T k ( t − r − 1 ) F t ≤ ( 1 − γ λ 1 ) 2 γ 2 ( T − 1 ) tr ( H 2 ) F r ,
which establishes (22) and (23) .
For part (2), since k ( ⋅ ) k(\cdot) k ( ⋅ ) is non-negative and non-increasing on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) , splitting the inner convolution over 0 ≤ j ≤ min ( a , b ) − r − 1 ≤ h − 1 0 \le j \le\min(a,b)-r-1 \le h-1 0 ≤ j ≤ min ( a , b ) − r − 1 ≤ h − 1 at the midpoint ( h − 1 ) / 2 (h-1)/2 ( h − 1 ) /2 yields
∑ j = 0 min ( a , b ) − r − 1 k ( j ) k ( h − 1 − j ) ≤ ∑ 0 ≤ j ≤ ( h − 1 ) / 2 k ( j ) k ( h − 1 2 ) + ∑ ( h − 1 ) / 2 < j ≤ h − 1 k ( h − 1 2 ) k ( h − 1 − j ) \sum_{j=0}^{\min(a,b)-r-1} k(j)k(h-1-j)
\le\sum_{0\le j\le(h-1)/2} k(j)k\left(\frac{h-1}{2}\right)
+\sum_{(h-1)/2<j\le h-1} k\left(\frac{h-1}{2}\right)k(h-1-j) j = 0 ∑ m i n ( a , b ) − r − 1 k ( j ) k ( h − 1 − j ) ≤ 0 ≤ j ≤ ( h − 1 ) /2 ∑ k ( j ) k ( 2 h − 1 ) + ( h − 1 ) /2 < j ≤ h − 1 ∑ k ( 2 h − 1 ) k ( h − 1 − j )
≤ 2 k ( h − 1 2 ) ∑ n = 0 ∞ k ( n ) ≤ 2 D T k ( h ) ∑ n = 0 ∞ k ( n ) , \le2k\left(\frac{h-1}{2}\right)\sum_{n=0}^{\infty}k(n)
\le2D_Tk(h)\sum_{n=0}^{\infty}k(n), ≤ 2 k ( 2 h − 1 ) n = 0 ∑ ∞ k ( n ) ≤ 2 D T k ( h ) n = 0 ∑ ∞ k ( n ) ,
where the first inequality bounds k ( h − 1 − j ) ≤ k ( ( h − 1 ) / 2 ) k(h-1-j) \le k((h-1)/2) k ( h − 1 − j ) ≤ k (( h − 1 ) /2 ) on the first half and k ( j ) ≤ k ( ( h − 1 ) / 2 ) k(j) \le k((h-1)/2) k ( j ) ≤ k (( h − 1 ) /2 ) on the second half, and the third inequality uses k ( ( h − 1 ) / 2 ) ≤ D T k ( h ) k((h-1)/2) \le D_Tk(h) k (( h − 1 ) /2 ) ≤ D T k ( h ) from (20) . Substituting this bound into (B.1) and using 1 N 2 ∑ a , b = s ∨ ( r + 1 ) T k ( a + b 2 − r ) ≤ F r \frac{1}{N^2}\sum_{a,b=s\vee(r+1)}^T k\left(\frac{a+b}{2}-r\right)\le F_r N 2 1 ∑ a , b = s ∨ ( r + 1 ) T k ( 2 a + b − r ) ≤ F r gives
γ 2 ∑ t = r + 1 T k ( t − r − 1 ) F t ≤ 2 D T γ 2 ( ∑ n = 0 ∞ k ( n ) ) F r . \gamma^2\sum_{t=r+1}^{T}k(t-r-1)F_t \le2D_T\gamma^2\left(\sum_{n=0}^{\infty}k(n)\right)F_r. γ 2 t = r + 1 ∑ T k ( t − r − 1 ) F t ≤ 2 D T γ 2 ( n = 0 ∑ ∞ k ( n ) ) F r .
Multiplying by α / F r \alpha/F_r α / F r , taking the maximum over 1 ≤ r < T 1\le r<T 1 ≤ r < T , and evaluating the geometric series γ 2 ∑ n = 0 ∞ k ( n ) = γ ∑ i ≥ 1 λ i 2 − γ λ i = γ tr ( H ) 2 − γ λ 1 \gamma^2\sum_{n=0}^{\infty}k(n)=\gamma\sum_{i\ge1}\frac{\lambda_i}{2-\gamma\lambda_i}=\frac{\gamma\operatorname{tr}(\mathbf{H})}{2-\gamma\lambda_1} γ 2 ∑ n = 0 ∞ k ( n ) = γ ∑ i ≥ 1 2 − γ λ i λ i = 2 − γ λ 1 γ tr ( H ) establishes (24) .
Proof of Corollary 4.4. Let ν : = 2 − 1 / a ∈ ( 1 , 2 ) \nu:=2-1/a\in(1,2) ν := 2 − 1/ a ∈ ( 1 , 2 ) and z : = γ Λ h ≥ 0 z:=\gamma\Lambda h\ge0 z := γ Λ h ≥ 0 . Since γ λ i ≤ γ Λ ≤ 1 / 2 \gamma\lambda_i\le\gamma\Lambda\le1/2 γ λ i ≤ γ Λ ≤ 1/2 , the elementary inequalities − 2 x ≤ log ( 1 − x ) ≤ − x -2x\le\log(1-x)\le-x − 2 x ≤ log ( 1 − x ) ≤ − x for x ∈ [ 0 , 1 / 2 ] x\in[0,1/2] x ∈ [ 0 , 1/2 ] imply e − 4 γ h λ i ≤ ( 1 − γ λ i ) 2 h ≤ e − 2 γ h λ i e^{-4\gamma h\lambda_i}\le(1-\gamma\lambda_i)^{2h}\le e^{-2\gamma h\lambda_i} e − 4 γ h λ i ≤ ( 1 − γ λ i ) 2 h ≤ e − 2 γ h λ i .
Upper bound on k ( h ) k(h) k ( h ) : For z ∈ [ 0 , 1 ] z\in[0,1] z ∈ [ 0 , 1 ] , k ( h ) ≤ k ( 0 ) ≤ Λ 2 ∑ i > 1 i − 2 a ≤ 2 ν Λ 2 ( ∑ i > 1 i − 2 a ) ( 1 + z ) − ν k(h)\le k(0)\le\Lambda^2\sum_{i>1}i^{-2a}\le2^\nu\Lambda^2\left(\sum_{i>1}i^{-2a}\right)(1+z)^{-\nu} k ( h ) ≤ k ( 0 ) ≤ Λ 2 ∑ i > 1 i − 2 a ≤ 2 ν Λ 2 ( ∑ i > 1 i − 2 a ) ( 1 + z ) − ν .
For z > 1 z>1 z > 1 , set M : = ⌈ z 1 / a ⌉ ≥ 2 M:=\lceil z^{1/a}\rceil\ge2 M := ⌈ z 1/ a ⌉ ≥ 2 and split the sum at M M M :
k ( h ) ≤ Λ 2 ∑ i = 1 M i − 2 a e − 2 θ z i − a + Λ 2 ∑ i = M + 1 ∞ i − 2 a ≤ Λ 2 ( 2 θ z ) − 2 ∑ i = 1 M 1 + Λ 2 ∫ M ∞ x − 2 a d x ≤ Λ 2 ( 2 θ − 2 + 1 2 a − 1 ) z − ν ≤ C ( a , θ ) Λ 2 ( 1 + γ Λ h ) − ν , \begin{aligned}
k(h)\le\Lambda^2\sum_{i=1}^{M}i^{-2a}e^{-2\theta zi^{-a}}+\Lambda^2\sum_{i=M+1}^{\infty}i^{-2a} \\
&\le\Lambda^2(2\theta z)^{-2}\sum_{i=1}^{M}1+\Lambda^2\int_M^\infty x^{-2a}\,dx \\
&\le\Lambda^2\left(2\theta^{-2}+\frac{1}{2a-1}\right)z^{-\nu}
\le C(a,\theta)\Lambda^2(1+\gamma\Lambda h)^{-\nu},
\end{aligned} k ( h ) ≤ Λ 2 i = 1 ∑ M i − 2 a e − 2 θ z i − a + Λ 2 i = M + 1 ∑ ∞ i − 2 a ≤ Λ 2 ( 2 θ z ) − 2 i = 1 ∑ M 1 + Λ 2 ∫ M ∞ x − 2 a d x ≤ Λ 2 ( 2 θ − 2 + 2 a − 1 1 ) z − ν ≤ C ( a , θ ) Λ 2 ( 1 + γ Λ h ) − ν ,
where the second inequality uses v 2 e − v ≤ 4 e − 2 ≤ 1 v^2e^{-v}\le4e^{-2}\le1 v 2 e − v ≤ 4 e − 2 ≤ 1 with v = θ z i − a v=\theta zi^{-a} v = θ z i − a on 1 ≤ i ≤ M 1\le i\le M 1 ≤ i ≤ M and integral comparison on i > M i>M i > M , the third inequality uses M = ⌈ z 1 / a ⌉ ≤ 2 z 1 / a M=\lceil z^{1/a}\rceil\le2z^{1/a} M = ⌈ z 1/ a ⌉ ≤ 2 z 1/ a and M 1 − 2 a ≤ z − ν M^{1-2a}\le z^{-\nu} M 1 − 2 a ≤ z − ν , and the last inequality uses z − ν ≤ 2 ν ( 1 + z ) − ν z^{-\nu}\le2^\nu(1+z)^{-\nu} z − ν ≤ 2 ν ( 1 + z ) − ν for z > 1 z>1 z > 1 .
Lower bound on k ( h ) k(h) k ( h ) : For z ∈ [ 0 , 1 ] z \in[0,1] z ∈ [ 0 , 1 ] , retaining i = 1 i=1 i = 1 gives k ( h ) ≥ θ 2 Λ 2 e − 4 z ≥ θ 2 e − 4 Λ 2 ( 1 + z ) − ν k(h) \ge\theta^2\Lambda^2e^{-4z} \ge\theta^2e^{-4}\Lambda^2(1+z)^{-\nu} k ( h ) ≥ θ 2 Λ 2 e − 4 z ≥ θ 2 e − 4 Λ 2 ( 1 + z ) − ν . For 1 ≤ z ≤ ( d / 2 ) a 1 \le z \le(d/2)^a 1 ≤ z ≤ ( d /2 ) a (with d = ∞ d=\infty d = ∞ in infinite dimension), let m : = ⌊ z 1 / a ⌋ ≥ 1 m:=\lfloor z^{1/a}\rfloor\ge1 m := ⌊ z 1/ a ⌋ ≥ 1 , so that 2 m ≤ d 2m\le d 2 m ≤ d and m + 1 > z 1 / a m+1>z^{1/a} m + 1 > z 1/ a . Restricting the sum to m + 1 ≤ i ≤ 2 m m+1\le i\le2m m + 1 ≤ i ≤ 2 m , where z i − a ≤ z ( m + 1 ) − a < 1 zi^{-a}\le z(m+1)^{-a}<1 z i − a ≤ z ( m + 1 ) − a < 1 , gives
k ( h ) ≥ θ 2 Λ 2 ∑ i = m + 1 2 m i − 2 a e − 4 z i − a ≥ θ 2 Λ 2 e − 4 ∑ i = m + 1 2 m i − 2 a ≥ θ 2 Λ 2 e − 4 m ( 2 m ) − 2 a ≥ θ 2 e − 4 2 − 2 a Λ 2 ( 1 + γ Λ h ) − ν . \begin{aligned}
k(h)&\ge\theta^2\Lambda^2\sum_{i=m+1}^{2m}i^{-2a}e^{-4zi^{-a}}
\ge\theta^2\Lambda^2e^{-4}\sum_{i=m+1}^{2m}i^{-2a}\\
&\ge\theta^2\Lambda^2e^{-4}m(2m)^{-2a}
\ge\theta^2e^{-4}2^{-2a}\Lambda^2(1+\gamma\Lambda h)^{-\nu}.
\end{aligned} k ( h ) ≥ θ 2 Λ 2 i = m + 1 ∑ 2 m i − 2 a e − 4 z i − a ≥ θ 2 Λ 2 e − 4 i = m + 1 ∑ 2 m i − 2 a ≥ θ 2 Λ 2 e − 4 m ( 2 m ) − 2 a ≥ θ 2 e − 4 2 − 2 a Λ 2 ( 1 + γ Λ h ) − ν .
where the last inequality uses m ≤ z 1 / a ≤ ( 1 + z ) 1 / a m\le z^{1/a}\le(1+z)^{1/a} m ≤ z 1/ a ≤ ( 1 + z ) 1/ a and 1 − 2 a < 0 1-2a<0 1 − 2 a < 0 .
Combining the upper and lower kernel bounds, for every 1 ≤ h ≤ T 1\le h\le T 1 ≤ h ≤ T (with γ Λ T ≤ ( d / 2 ) a \gamma\Lambda T\le(d/2)^a γ Λ T ≤ ( d /2 ) a ),
k ( ( h − 1 ) / 2 ) k ( h ) ≤ C ( a , θ ) θ 2 e − 4 2 − 2 a ( 1 + γ Λ h 1 + γ Λ ( h − 1 ) / 2 ) ν ≤ 2 ν C ( a , θ ) θ 2 e − 4 2 − 2 a = : D ∗ ( a , θ ) . \frac{k((h-1)/2)}{k(h)}
\le\frac{C(a,\theta)}{\theta^2e^{-4}2^{-2a}}
\left(\frac{1+\gamma\Lambda h}{1+\gamma\Lambda(h-1)/2}\right)^\nu
\le\frac{2^\nu C(a,\theta)}{\theta^2e^{-4}2^{-2a}}
=:D_*(a,\theta). k ( h ) k (( h − 1 ) /2 ) ≤ θ 2 e − 4 2 − 2 a C ( a , θ ) ( 1 + γ Λ ( h − 1 ) /2 1 + γ Λ h ) ν ≤ θ 2 e − 4 2 − 2 a 2 ν C ( a , θ ) =: D ∗ ( a , θ ) .
Consequently, D T ≤ D ∗ D_T\le D_* D T ≤ D ∗ , and for any γ \gamma γ satisfying (25) , Proposition 4.3(2) yields χ s , N ≤ 2 α D ∗ γ tr ( H ) 2 − γ λ 1 ≤ 2 α D ∗ γ tr ( H ) ≤ 1 / 2 \chi_{s,N}\le\frac{2\alpha D_*\gamma\operatorname{tr}(\mathbf{H})}{2-\gamma\lambda_1}\le2\alpha D_*\gamma\operatorname{tr}(\mathbf{H})\le1/2 χ s , N ≤ 2 − γ λ 1 2 α D ∗ γ tr ( H ) ≤ 2 α D ∗ γ tr ( H ) ≤ 1/2 .
Proof of Proposition 4.7 and Proposition 4.8
Proof of Proposition 4.7. For part (1), we have λ 1 = 1 \lambda_1=1 λ 1 = 1 , λ 2 = ( 4 γ N ) − 1 \lambda_2=(4\gamma N)^{-1} λ 2 = ( 4 γ N ) − 1 , ω 1 = 1 \omega_1=1 ω 1 = 1 , ω 2 = 0 \omega_2=0 ω 2 = 0 , σ n o i s e 2 = 0 \sigma_{\mathrm{noise}}^2=0 σ noise 2 = 0 , and β = 1 \beta=1 β = 1 (since x ∼ N ( 0 , H ) \mathbf{x}\sim\mathcal{N}(\mathbf{0},\mathbf{H}) x ∼ N ( 0 , H ) ). Let m : = ⌊ 1 / ( 4 γ ) ⌋ + 1 ≤ N m:=\lfloor1/(4\gamma)\rfloor+1\le N m := ⌊ 1/ ( 4 γ )⌋ + 1 ≤ N . For every 1 ≤ t ≤ m 1\le t\le m 1 ≤ t ≤ m , we have t − 1 ≤ 1 / ( 4 γ ) t-1\le1/(4\gamma) t − 1 ≤ 1/ ( 4 γ ) , so Bernoulli’s inequality gives ( 1 − γ ) 2 ( t − 1 ) ≥ 1 − 2 γ ( t − 1 ) ≥ 1 / 2 (1-\gamma)^{2(t-1)}\ge1-2\gamma(t-1)\ge1/2 ( 1 − γ ) 2 ( t − 1 ) ≥ 1 − 2 γ ( t − 1 ) ≥ 1/2 . Meanwhile, for every 1 ≤ t ≤ N 1\le t\le N 1 ≤ t ≤ N and N ≤ k ≤ 2 N − 1 N\le k\le2N-1 N ≤ k ≤ 2 N − 1 , we have k − t ≤ 2 N k-t\le2N k − t ≤ 2 N , so ( 1 − γ λ 2 ) k − t = ( 1 − 1 4 N ) k − t ≥ 1 − k − t 4 N ≥ 1 / 2 (1-\gamma\lambda_2)^{k-t}=(1-\frac{1}{4N})^{k-t}\ge1-\frac{k-t}{4N}\ge1/2 ( 1 − γ λ 2 ) k − t = ( 1 − 4 N 1 ) k − t ≥ 1 − 4 N k − t ≥ 1/2 , which implies q t ( λ 2 ) = 1 N ∑ k = N 2 N − 1 ( 1 − γ λ 2 ) k − t ≥ 1 / 2 q_t(\lambda_2)=\frac{1}{N}\sum_{k=N}^{2N-1}(1-\gamma\lambda_2)^{k-t}\ge1/2 q t ( λ 2 ) = N 1 ∑ k = N 2 N − 1 ( 1 − γ λ 2 ) k − t ≥ 1/2 and F t ≥ λ 2 2 q t ( λ 2 ) 2 ≥ 1 4 ( 4 γ N ) − 2 = 1 64 γ 2 N 2 F_t\ge\lambda_2^2q_t(\lambda_2)^2\ge\frac14(4\gamma N)^{-2}=\frac{1}{64\gamma^2N^2} F t ≥ λ 2 2 q t ( λ 2 ) 2 ≥ 4 1 ( 4 γ N ) − 2 = 64 γ 2 N 2 1 . Substituting these lower bounds into (41) of Lemma 5.2 with β = 1 \beta=1 β = 1 yields
E [ L ( w N : 2 N ) ] − L ( w ∗ ) ≥ S N , N ≥ γ 2 2 ∑ t = 1 m ( 1 − γ ) 2 ( t − 1 ) F t ≥ γ 2 2 ⋅ m ⋅ 1 2 ⋅ 1 64 γ 2 N 2 ≥ 1 1024 γ N 2 , \begin{aligned}
\mathbb{E}[L(\mathbf{w}_{N:2N})]-L(\mathbf{w}^*)\ge S_{N,N}\ge\frac{\gamma^2}{2}\sum_{t=1}^{m}(1-\gamma)^{2(t-1)}F_t \\
&\ge\frac{\gamma^2}{2}\cdot m\cdot\frac12\cdot\frac{1}{64\gamma^2N^2}\ge\frac{1}{1024\gamma N^2},
\end{aligned} E [ L ( w N : 2 N )] − L ( w ∗ ) ≥ S N , N ≥ 2 γ 2 t = 1 ∑ m ( 1 − γ ) 2 ( t − 1 ) F t ≥ 2 γ 2 ⋅ m ⋅ 2 1 ⋅ 64 γ 2 N 2 1 ≥ 1024 γ N 2 1 ,
where the last inequality uses m > 1 / ( 4 γ ) m>1/(4\gamma) m > 1/ ( 4 γ ) . By contrast, since λ 1 = 1 ≥ ( 2 γ N ) − 1 > λ 2 = ( 4 γ N ) − 1 \lambda_1=1\ge(2\gamma N)^{-1}>\lambda_2=(4\gamma N)^{-1} λ 1 = 1 ≥ ( 2 γ N ) − 1 > λ 2 = ( 4 γ N ) − 1 , we have k ∗ = k † = 1 k^*=k^\dagger=1 k ∗ = k † = 1 and ∥ w 0 − w ∗ ∥ H k † : ∞ 2 = λ 2 ω 2 2 = 0 \lVert\mathbf{w}_0-\mathbf{w}^*\rVert_{\mathbf{H}_{k^\dagger:\infty}}^2=\lambda_2\omega_2^2=0 ∥ w 0 − w ∗ ∥ H k † : ∞ 2 = λ 2 ω 2 2 = 0 , so Zou et al. [37 ] drops the variance contribution entirely and retains only the bias lower bound 1 100 γ 2 N 2 ∥ ( I − γ H ) N ( w 0 − w ∗ ) ∥ H 0 : k ∗ − 1 2 = ( 1 − γ ) 2 N 100 γ 2 N 2 \frac{1}{100\gamma^2N^2}\lVert(\mathbf{I}-\gamma\mathbf{H})^N(\mathbf{w}_0-\mathbf{w}^*)\rVert_{\mathbf{H}_{0:k^*}^{-1}}^2=\frac{(1-\gamma)^{2N}}{100\gamma^2N^2} 100 γ 2 N 2 1 ∥( I − γ H ) N ( w 0 − w ∗ ) ∥ H 0 : k ∗ − 1 2 = 100 γ 2 N 2 ( 1 − γ ) 2 N , proving (28) .
For part (2), both D 1 D_1 D 1 and D 2 D_2 D 2 satisfy E [ x 2 ] = 1 \mathbb{E}[x^2]=1 E [ x 2 ] = 1 , E [ x 4 ] ≤ 3 = α \mathbb{E}[x^4]\le3=\alpha E [ x 4 ] ≤ 3 = α , and E [ ( x 2 − 1 ) 2 ] = κ − 1 ≥ 1 = β \mathbb{E}[(x^2-1)^2]=\kappa-1\ge1=\beta E [( x 2 − 1 ) 2 ] = κ − 1 ≥ 1 = β for κ : = E [ x 4 ] ∈ { 2 , 3 } \kappa:=\mathbb{E}[x^4]\in\{2,3\} κ := E [ x 4 ] ∈ { 2 , 3 } . Since y t = 0 y_t=0 y t = 0 and w 0 = 1 w_0=1 w 0 = 1 , the scalar iterate satisfies w t = ( 1 − γ x t 2 ) w t − 1 w_t=(1-\gamma x_t^2)w_{t-1} w t = ( 1 − γ x t 2 ) w t − 1 with E [ 1 − γ x t 2 ] = 1 − γ > 0 \mathbb{E}[1-\gamma x_t^2]=1-\gamma>0 E [ 1 − γ x t 2 ] = 1 − γ > 0 and E [ ( 1 − γ x t 2 ) 2 ] = 1 − 2 γ + κ γ 2 = r κ ∈ ( 0 , 1 ) \mathbb{E}[(1-\gamma x_t^2)^2]=1-2\gamma+\kappa\gamma^2=r_\kappa\in(0,1) E [( 1 − γ x t 2 ) 2 ] = 1 − 2 γ + κ γ 2 = r κ ∈ ( 0 , 1 ) for 0 < γ < 1 / 3 0<\gamma<1/3 0 < γ < 1/3 . By independence of { x t } \{x_t\} { x t } , we have E [ w t 2 ] = r κ t \mathbb{E}[w_t^2]=r_\kappa^t E [ w t 2 ] = r κ t and E [ w t w t + h ] = r κ t ( 1 − γ ) h \mathbb{E}[w_tw_{t+h}]=r_\kappa^t(1-\gamma)^h E [ w t w t + h ] = r κ t ( 1 − γ ) h for all t , h ≥ 0 t,h\ge0 t , h ≥ 0 . Expanding w ‾ s : s + N 2 = 1 N 2 ∑ t = s s + N − 1 w t 2 + 2 N 2 ∑ t = s s + N − 1 ∑ h = 1 s + N − 1 − t w t w t + h \overline{w}_{s:s+N}^2=\frac{1}{N^2}\sum_{t=s}^{s+N-1}w_t^2+\frac{2}{N^2}\sum_{t=s}^{s+N-1}\sum_{h=1}^{s+N-1-t}w_tw_{t+h} w s : s + N 2 = N 2 1 ∑ t = s s + N − 1 w t 2 + N 2 2 ∑ t = s s + N − 1 ∑ h = 1 s + N − 1 − t w t w t + h gives the exact risk
R κ ( s , N ) = 1 2 E [ w ‾ s : s + N 2 ] = r κ s 2 N 2 ∑ j = 0 N − 1 r κ j ( 1 + 2 ∑ h = 1 N − 1 − j ( 1 − γ ) h ) . R_\kappa(s,N)=\frac12\mathbb{E}[\overline{w}_{s:s+N}^2]
=\frac{r_\kappa^s}{2N^2}\sum_{j=0}^{N-1}r_\kappa^j\left(1+2\sum_{h=1}^{N-1-j}(1-\gamma)^h\right). R κ ( s , N ) = 2 1 E [ w s : s + N 2 ] = 2 N 2 r κ s j = 0 ∑ N − 1 r κ j ( 1 + 2 h = 1 ∑ N − 1 − j ( 1 − γ ) h ) .
Retaining j = 0 j=0 j = 0 and dropping h ≥ 1 h\ge1 h ≥ 1 gives R κ ( s , N ) ≥ r κ s 2 N 2 R_\kappa(s,N)\ge\frac{r_\kappa^s}{2N^2} R κ ( s , N ) ≥ 2 N 2 r κ s , while bounding 1 + 2 ∑ h = 1 ∞ ( 1 − γ ) h = 2 − γ γ 1+2\sum_{h=1}^{\infty}(1-\gamma)^h=\frac{2-\gamma}{\gamma} 1 + 2 ∑ h = 1 ∞ ( 1 − γ ) h = γ 2 − γ and ∑ j = 0 ∞ r κ j = 1 1 − r κ \sum_{j=0}^{\infty}r_\kappa^j=\frac{1}{1-r_\kappa} ∑ j = 0 ∞ r κ j = 1 − r κ 1 gives R κ ( s , N ) ≤ ( 2 − γ ) r κ s 2 γ ( 1 − r κ ) N 2 R_\kappa(s,N)\le\frac{(2-\gamma)r_\kappa^s}{2\gamma(1-r_\kappa)N^2} R κ ( s , N ) ≤ 2 γ ( 1 − r κ ) N 2 ( 2 − γ ) r κ s , establishing (29) .
Proof of Proposition 4.8. Applying the martingale-difference expansion of Lemma 5.1 to the shifted bias process { η s + k b i a s } k = 0 N − 1 \{\eta^{\mathrm{bias}}_{s+k}\}_{k=0}^{N-1} { η s + k bias } k = 0 N − 1 initialized at η s b i a s \eta^{\mathrm{bias}}_s η s bias with E [ η s b i a s ⊗ η s b i a s ] = B s \mathbb{E}[\eta^{\mathrm{bias}}_s \otimes\eta^{\mathrm{bias}}_s] = B_s E [ η s bias ⊗ η s bias ] = B s yields the conditional identity
E [ η ‾ s : s + N b i a s ⊗ η ‾ s : s + N b i a s ] = P 0 , N B s P 0 , N + γ 2 ∑ k = 0 N − 2 Q k , 0 , N ( ( M − M ~ ) ∘ B s + k ) Q k , 0 , N . (62) \mathbb{E}\left[\overline{\eta}^{\mathrm{bias}}_{s:s+N} \otimes\overline{\eta}^{\mathrm{bias}}_{s:s+N}\right] = P_{0,N}B_sP_{0,N}+\gamma^2\sum_{k=0}^{N-2}Q_{k,0,N}\left((\mathcal{M}-\widetilde{\mathcal{M}})\circ B_{s+k}\right)Q_{k,0,N}. \tag*{(62)} E [ η s : s + N bias ⊗ η s : s + N bias ] = P 0 , N B s P 0 , N + γ 2 k = 0 ∑ N − 2 Q k , 0 , N ( ( M − M ) ∘ B s + k ) Q k , 0 , N . (62)
Let a i ( s ) : = v i T B s v i ≥ 0 a_i^{(s)} := v_i^{\mathsf{T}}B_sv_i \ge0 a i ( s ) := v i T B s v i ≥ 0 denote the i i i -th diagonal entry of B s B_s B s in the eigenbasis of H H H , and recall Ξ N \Xi_N Ξ N and Ψ r , s , N \Psi_{r,s,N} Ψ r , s , N from (A.1)–(A.2). Taking the trace inner product of (B.2) with 1 2 H \frac{1}{2}H 2 1 H gives
b i a s = 1 2 ⟨ H , P 0 , N B s P 0 , N ⟩ + γ 2 2 ∑ k = 0 N − 2 ⟨ H , Q k , 0 , N ( ( M − M ~ ) ∘ B s + k ) Q k , 0 , N ⟩ . \mathrm{bias}=\frac{1}{2}\langle H,P_{0,N}B_sP_{0,N}\rangle+\frac{\gamma^2}{2}\sum_{k=0}^{N-2}\left\langle H,Q_{k,0,N}\left((\mathcal{M}-\widetilde{\mathcal{M}})\circ B_{s+k}\right)Q_{k,0,N}\right\rangle. bias = 2 1 ⟨ H , P 0 , N B s P 0 , N ⟩ + 2 γ 2 k = 0 ∑ N − 2 ⟨ H , Q k , 0 , N ( ( M − M ) ∘ B s + k ) Q k , 0 , N ⟩ .
For the first term, since P 0 , N v i = 1 − ( 1 − γ λ i ) N γ N λ i v i P_{0,N}v_i=\frac{1-(1-\gamma\lambda_i)^N}{\gamma N\lambda_i}v_i P 0 , N v i = γ N λ i 1 − ( 1 − γ λ i ) N v i , applying (A.12) yields
1 8 γ 2 N 2 tr ( H 0 : k − 1 B s ) + 1 8 tr ( H k : ∞ B s ) ≤ 1 2 ⟨ H , P 0 , N B s P 0 , N ⟩ ≤ 1 2 γ 2 N 2 tr ( H 0 : k − 1 B s ) + 1 2 tr ( H k : ∞ B s ) . (63) \begin{aligned}
\frac{1}{8\gamma^2N^2}\operatorname{tr}\left(H_{0:k}^{-1}B_s\right)+\frac{1}{8}\operatorname{tr}\left(H_{k:\infty}B_s\right)\leq\frac{1}{2}\left\langle H,P_{0,N}B_sP_{0,N}\right\rangle\\
&\leq\frac{1}{2\gamma^2N^2}\operatorname{tr}\left(H_{0:k}^{-1}B_s\right)+\frac{1}{2}\operatorname{tr}\left(H_{k:\infty}B_s\right). \tag*{(63)}
\end{aligned} 8 γ 2 N 2 1 tr ( H 0 : k − 1 B s ) + 8 1 tr ( H k : ∞ B s ) ≤ 2 1 ⟨ H , P 0 , N B s P 0 , N ⟩ ≤ 2 γ 2 N 2 1 tr ( H 0 : k − 1 B s ) + 2 1 tr ( H k : ∞ B s ) . (63)
For the second term, note that γ 2 ⟨ H , Q k , 0 , N H Q k , 0 , N ⟩ = Ψ s + k , s , N \gamma^2\langle H,Q_{k,0,N}HQ_{k,0,N}\rangle=\Psi_{s+k,s,N} γ 2 ⟨ H , Q k , 0 , N H Q k , 0 , N ⟩ = Ψ s + k , s , N .6 In the upper bound, we have
γ 2 2 ∑ k = 0 N − 2 ⟨ H , Q k , 0 , N ( ( M − M ~ ) ∘ B s + k ) Q k , 0 , N ⟩ ≤ α 2 ∑ k = 0 N − 2 tr ( H B s + k ) Ψ s + k , s , N ≤ α Ξ N 4 γ N ( 1 − γ α tr ( H ) ) tr ( B s , s + N ) ≤ α 2 ( 1 − γ α tr ( H ) ) ( 1 γ N tr ( I 0 : k B s ) + tr ( H k : ∞ B s ) ) Ξ N , \begin{aligned}
\frac{\gamma^2}{2}\sum_{k=0}^{N-2}\left\langle H,Q_{k,0,N}\left((\mathcal{M}-\widetilde{\mathcal{M}})\circ B_{s+k}\right)Q_{k,0,N}\right\rangle\\
&\leq\frac{\alpha}{2}\sum_{k=0}^{N-2}\operatorname{tr}(HB_{s+k})\Psi_{s+k,s,N}\leq\frac{\alpha\Xi_N}{4\gamma N(1-\gamma\alpha\operatorname{tr}(H))}\operatorname{tr}(B_{s,s+N}) \\
&\leq\frac{\alpha}{2(1-\gamma\alpha\operatorname{tr}(H))}\left(\frac{1}{\gamma N}\operatorname{tr}(I_{0:k}B_s)+\operatorname{tr}(H_{k:\infty}B_s)\right)\Xi_N,
\end{aligned} 2 γ 2 k = 0 ∑ N − 2 ⟨ H , Q k , 0 , N ( ( M − M ) ∘ B s + k ) Q k , 0 , N ⟩ ≤ 2 α k = 0 ∑ N − 2 tr ( H B s + k ) Ψ s + k , s , N ≤ 4 γ N ( 1 − γ α tr ( H )) α Ξ N tr ( B s , s + N ) ≤ 2 ( 1 − γ α tr ( H )) α ( γ N 1 tr ( I 0 : k B s ) + tr ( H k : ∞ B s ) ) Ξ N ,
where the first inequality applies Assumption 3.2, the second inequality uses Ψ s + k , s , N ≤ Ψ s , s , N ≤ 1 N Ξ N \Psi_{s+k,s,N}\leq\Psi_{s,s,N}\leq\frac{1}{N}\Xi_N Ψ s + k , s , N ≤ Ψ s , s , N ≤ N 1 Ξ N and Lemma C.7 on [ s , s + N ] [s,s+N] [ s , s + N ] , and the third inequality uses 1 − ( 1 − γ λ i ) 2 N ≤ 1 ∧ ( 2 γ N λ i ) 1-(1-\gamma\lambda_i)^{2N}\leq1\wedge(2\gamma N\lambda_i) 1 − ( 1 − γ λ i ) 2 N ≤ 1 ∧ ( 2 γ N λ i ) . Meanwhile, for the variance error in (5.7), Lemma C.8 and Assumption 3.2 imply ( M − M ~ ) ∘ C r + Σ ⪯ σ 2 1 − γ α tr ( H ) H (\mathcal{M}-\widetilde{\mathcal{M}})\circ C_r+\Sigma\preceq\frac{\sigma^2}{1-\gamma\alpha\operatorname{tr}(H)}H ( M − M ) ∘ C r + Σ ⪯ 1 − γ α tr ( H ) σ 2 H for all r ≥ 0 r\geq0 r ≥ 0 , so v a r i a n c e ≤ σ 2 1 − γ α tr ( H ) V s , N \mathrm{variance}\leq\frac{\sigma^2}{1-\gamma\alpha\operatorname{tr}(H)}V_{s,N} variance ≤ 1 − γ α tr ( H ) σ 2 V s , N . Combining these bounds with E [ L ( w s : s + N ) − L ( w ∗ ) ] ≤ 2 ⋅ b i a s + 2 ⋅ v a r i a n c e \mathbb{E}[L(w_{s:s+N})-L(w^*)]\leq2\cdot\mathrm{bias}+2\cdot\mathrm{variance} E [ L ( w s : s + N ) − L ( w ∗ )] ≤ 2 ⋅ bias + 2 ⋅ variance and (A.10) yields (4.18).
In the lower bound (N ≥ 2 N\geq2 N ≥ 2 and γ ≤ 1 / ( 2 λ 1 ) \gamma\leq1/(2\lambda_1) γ ≤ 1/ ( 2 λ 1 ) ), let m : = ⌊ N / 2 ⌋ ≥ 1 m:=\lfloor N/2\rfloor\geq1 m := ⌊ N /2 ⌋ ≥ 1 and G : = I − γ H G:=I-\gamma H G := I − γ H . Restricting the sum over k k k to 0 ≤ k ≤ m − 1 0\leq k\leq m-1 0 ≤ k ≤ m − 1 , we have Ψ s + k , s , N ≥ 1 36 N Ξ N \Psi_{s+k,s,N}\geq\frac{1}{36N}\Xi_N Ψ s + k , s , N ≥ 36 N 1 Ξ N from Part 3 of the proof of Lemma A.1, while Assumption 3.4 gives ( M − M ~ ) ∘ B s + k ⪰ β tr ( H B s + k ) H (\mathcal{M}-\widetilde{\mathcal{M}})\circ B_{s+k}\succeq\beta\operatorname{tr}(HB_{s+k})H ( M − M ) ∘ B s + k ⪰ β tr ( H B s + k ) H . Summing γ tr ( H B s + k ) \gamma\operatorname{tr}(HB_{s+k}) γ tr ( H B s + k ) over 0 ≤ k ≤ m − 1 0\leq k\leq m-1 0 ≤ k ≤ m − 1 gives
γ ∑ k = 0 m − 1 tr ( H B s + k ) ≥ γ ∑ k = 0 m − 1 tr ( H G 2 k B s ) = ∑ i = 1 ∞ 1 − ( 1 − γ λ i ) 2 m 2 − γ λ i a i ( s ) ≥ 1 2 tr ( B s − G 2 m B s ) ≥ 1 6 tr ( B s , s + N ) ≥ 1 12 ( tr ( I 0 : k B s ) + N γ tr ( H k : ∞ B s ) ) , (64) \begin{aligned}
\gamma\sum_{k=0}^{m-1}\operatorname{tr}(HB_{s+k})\geq\gamma\sum_{k=0}^{m-1}\operatorname{tr}(HG^{2k}B_s)=\sum_{i=1}^{\infty}\frac{1-(1-\gamma\lambda_i)^{2m}}{2-\gamma\lambda_i}a_i^{(s)} \\
&\geq\frac{1}{2}\operatorname{tr}(B_s-G^{2m}B_s)\geq\frac{1}{6}\operatorname{tr}(B_{s,s+N}) \\
&\geq\frac{1}{12}\left(\operatorname{tr}(I_{0:k}B_s)+N\gamma\operatorname{tr}(H_{k:\infty}B_s)\right), \tag*{(64)}
\end{aligned} γ k = 0 ∑ m − 1 tr ( H B s + k ) ≥ γ k = 0 ∑ m − 1 tr ( H G 2 k B s ) = i = 1 ∑ ∞ 2 − γ λ i 1 − ( 1 − γ λ i ) 2 m a i ( s ) ≥ 2 1 tr ( B s − G 2 m B s ) ≥ 6 1 tr ( B s , s + N ) ≥ 12 1 ( tr ( I 0 : k B s ) + N γ tr ( H k : ∞ B s ) ) , (64)
where the first inequality uses B s + k = ( I − γ T ) k ∘ B s ⪰ G k B s G k \mathbf{B}_{s+k}=(\mathcal{I}-\gamma\mathcal{T})^k\circ\mathbf{B}_s\succeq G^k\mathbf{B}_sG^k B s + k = ( I − γ T ) k ∘ B s ⪰ G k B s G k from Lemma C.5(3), the second inequality uses 2 − γ λ i ≤ 2 2-\gamma\lambda_i\le2 2 − γ λ i ≤ 2 , the third inequality applies Lemma C.2, and the fourth inequality uses 1 − ( 1 − γ λ i ) 2 N ≥ 1 2 ( 1 ∧ γ N λ i ) 1-(1-\gamma\lambda_i)^{2N}\ge\frac{1}{2}(1\wedge\gamma N\lambda_i) 1 − ( 1 − γ λ i ) 2 N ≥ 2 1 ( 1 ∧ γ N λ i ) . Multiplying this trace bound by β 2 γ ⋅ 1 36 N Ξ N = β 72 γ N Ξ N \frac{\beta}{2\gamma}\cdot\frac{1}{36N}\Xi_N=\frac{\beta}{72\gamma N}\Xi_N 2 γ β ⋅ 36 N 1 Ξ N = 72 γ N β Ξ N gives β 864 γ N ( tr ( I 0 : k ∗ B s ) + N γ tr ( H k ∗ : ∞ B s ) ) Ξ N \frac{\beta}{864\gamma N}\left(\operatorname{tr}(\mathcal{I}_{0:k}^*\mathbf{B}_s)+N\gamma\operatorname{tr}(\mathbf{H}_{k^*:\infty}\mathbf{B}_s)\right)\Xi_N 864 γ N β ( tr ( I 0 : k ∗ B s ) + N γ tr ( H k ∗ : ∞ B s ) ) Ξ N . Combining this with (B.3), variance ≥ σ n o i s e 2 V s , N \ge\sigma_{\mathrm{noise}}^2\mathcal{V}_{s,N} ≥ σ noise 2 V s , N , and (A.10) establishes (31) .
Auxiliary Lemmas
In this appendix, we collect the general-purpose elementary algebraic and convolution lemmas (Section C.1) and the foundational operator lemmas from Jain et al. [13 ] and Zou et al. [37 ] (Section C.2) invoked in Appendices A–B.
Elementary Algebraic and Convolution Lemmas
Lemma C.1 (Chebyshev’s Sum Inequality for Weighted Monotone Sequences). Let { w i } i ≥ 1 \{w_i\}_{i\ge1} { w i } i ≥ 1 be non-negative weights with W : = ∑ i = 1 ∞ w i < ∞ W:=\sum_{i=1}^{\infty}w_i<\infty W := ∑ i = 1 ∞ w i < ∞ , and let { x i } i ≥ 1 \{x_i\}_{i\ge1} { x i } i ≥ 1 and { y i } i ≥ 1 \{y_i\}_{i\ge1} { y i } i ≥ 1 be bounded non-negative sequences that are both non-decreasing in i i i . Then
( ∑ i = 1 ∞ w i x i ) ( ∑ i = 1 ∞ w i y i ) ≤ W ∑ i = 1 ∞ w i x i y i . (65) \left(\sum_{i=1}^{\infty}w_ix_i\right)\left(\sum_{i=1}^{\infty}w_iy_i\right)\le W\sum_{i=1}^{\infty}w_ix_iy_i. \tag*{(65)} ( i = 1 ∑ ∞ w i x i ) ( i = 1 ∑ ∞ w i y i ) ≤ W i = 1 ∑ ∞ w i x i y i . (65) In particular, under Assumption 3.1 with $0<\gamma<1/\lambda_1$, the propagation kernel k ( h ) = ∑ i ≥ 1 λ i 2 ( 1 − γ λ i ) 2 h k(h)=\sum_{i\ge1}\lambda_i^2(1-\gamma\lambda_i)^{2h} k ( h ) = ∑ i ≥ 1 λ i 2 ( 1 − γ λ i ) 2 h satisfies k ( u ) k ( v ) ≤ k ( 0 ) k ( u + v ) = tr ( H 2 ) k ( u + v ) k(u)k(v)\le k(0)k(u+v)=\operatorname{tr}(\mathbf{H}^2)k(u+v) k ( u ) k ( v ) ≤ k ( 0 ) k ( u + v ) = tr ( H 2 ) k ( u + v ) for all $u,v\ge 0.
Proof of Lemma C.1. Since { x i } \{x_i\} { x i } and { y i } \{y_i\} { y i } are both non-decreasing, ( x i − x j ) ( y i − y j ) ≥ 0 (x_i-x_j)(y_i-y_j)\ge0 ( x i − x j ) ( y i − y j ) ≥ 0 for all i , j ≥ 1 i,j\ge1 i , j ≥ 1 . Multiplying by w i w j ≥ 0 w_iw_j\ge0 w i w j ≥ 0 and summing over all i , j ≥ 1 i,j\ge1 i , j ≥ 1 gives
0 ≤ 1 2 ∑ i = 1 ∞ ∑ j = 1 ∞ w i w j ( x i − x j ) ( y i − y j ) = W ∑ i = 1 ∞ w i x i y i − ( ∑ i = 1 ∞ w i x i ) ( ∑ j = 1 ∞ w j y j ) , 0\le\frac{1}{2}\sum_{i=1}^{\infty}\sum_{j=1}^{\infty}w_iw_j(x_i-x_j)(y_i-y_j)=W\sum_{i=1}^{\infty}w_ix_iy_i-\left(\sum_{i=1}^{\infty}w_ix_i\right)\left(\sum_{j=1}^{\infty}w_jy_j\right), 0 ≤ 2 1 i = 1 ∑ ∞ j = 1 ∑ ∞ w i w j ( x i − x j ) ( y i − y j ) = W i = 1 ∑ ∞ w i x i y i − ( i = 1 ∑ ∞ w i x i ) ( j = 1 ∑ ∞ w j y j ) ,
which proves (C.1). Taking w i = λ i 2 w_i=\lambda_i^2 w i = λ i 2 , x i = ( 1 − γ λ i ) 2 u x_i=(1-\gamma\lambda_i)^{2u} x i = ( 1 − γ λ i ) 2 u , and y i = ( 1 − γ λ i ) 2 v y_i=(1-\gamma\lambda_i)^{2v} y i = ( 1 − γ λ i ) 2 v (which are non-decreasing in i i i since λ 1 ≥ λ 2 ≥ ⋯ > 0 \lambda_1\ge\lambda_2\ge\cdots>0 λ 1 ≥ λ 2 ≥ ⋯ > 0 ) yields k ( u ) k ( v ) ≤ tr ( H 2 ) k ( u + v ) k(u)k(v)\le\operatorname{tr}(\mathbf{H}^2)k(u+v) k ( u ) k ( v ) ≤ tr ( H 2 ) k ( u + v ) .
Lemma C.2 (Polynomial Exponent-Halving Sandwich). For every x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] and integer N ≥ 2 N\ge2 N ≥ 2 with m : = ⌊ N / 2 ⌋ ≥ 1 m:=\lfloor N/2\rfloor\ge1 m := ⌊ N /2 ⌋ ≥ 1 , it holds that 1 − x m ≤ 1 − x N ≤ 3 ( 1 − x m ) 1-x^m\le1-x^N\le3(1-x^m) 1 − x m ≤ 1 − x N ≤ 3 ( 1 − x m ) . Consequently, for any PSD operator H \mathbf{H} H with ∥ γ H ∥ 2 ≤ 1 \|\gamma\mathbf{H}\|_2\le1 ∥ γ H ∥ 2 ≤ 1 and any PSD operator A ⪰ 0 \mathbf{A}\succeq0 A ⪰ 0 ,
1 3 tr ( ( I − ( I − γ H ) 2 N ) A ) ≤ tr ( ( I − ( I − γ H ) 2 ⌊ N / 2 ⌋ ) A ) ≤ tr ( ( I − ( I − γ H ) 2 N ) A ) . (66) \frac{1}{3}\operatorname{tr}\left((\mathbf{I}-(\mathbf{I}-\gamma\mathbf{H})^{2N})\mathbf{A}\right)\le\operatorname{tr}\left((\mathbf{I}-(\mathbf{I}-\gamma\mathbf{H})^{2\lfloor N/2\rfloor})\mathbf{A}\right)\le\operatorname{tr}\left((\mathbf{I}-(\mathbf{I}-\gamma\mathbf{H})^{2N})\mathbf{A}\right). \tag*{(66)} 3 1 tr ( ( I − ( I − γ H ) 2 N ) A ) ≤ tr ( ( I − ( I − γ H ) 2 ⌊ N /2 ⌋ ) A ) ≤ tr ( ( I − ( I − γ H ) 2 N ) A ) . (66)
Proof of Lemma C.2. Since m = ⌊ N / 2 ⌋ ≥ 1 m=\lfloor N/2\rfloor\ge1 m = ⌊ N /2 ⌋ ≥ 1 satisfies m ≤ N ≤ 3 m m\le N\le3m m ≤ N ≤ 3 m and x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] , we have 1 − x m ≤ 1 − x N ≤ 1 − x 3 m = ( 1 − x m ) ( 1 + x m + x 2 m ) ≤ 3 ( 1 − x m ) 1-x^m\le1-x^N\le1-x^{3m}=(1-x^m)(1+x^m+x^{2m})\le3(1-x^m) 1 − x m ≤ 1 − x N ≤ 1 − x 3 m = ( 1 − x m ) ( 1 + x m + x 2 m ) ≤ 3 ( 1 − x m ) . Applying this to each eigenvalue x = ( 1 − γ λ i ) 2 ∈ [ 0 , 1 ] x=(1-\gamma\lambda_i)^2\in[0,1] x = ( 1 − γ λ i ) 2 ∈ [ 0 , 1 ] yields (C.2).
Lemma C.3 (Exact Two-Sided Bound for Finite Geometric Convolutions). For any scalars p , q ∈ ( 0 , 1 ] p,q\in(0,1] p , q ∈ ( 0 , 1 ] and any integer s ≥ 1 s\ge1 s ≥ 1 , let M : = p ∨ q M:=p\vee q M := p ∨ q and m : = p ∧ q m:=p\wedge q m := p ∧ q . Then
( 1 − e − 1 ) M s − 1 min { s , M M − m } ≤ ∑ r = 0 s − 1 p r q s − 1 − r ≤ M s − 1 min { s , M M − m } , (67) (1-e^{-1})M^{s-1}\min\left\{s,\frac{M}{M-m}\right\}\le\sum_{r=0}^{s-1}p^rq^{s-1-r}\le M^{s-1}\min\left\{s,\frac{M}{M-m}\right\}, \tag*{(67)} ( 1 − e − 1 ) M s − 1 min { s , M − m M } ≤ r = 0 ∑ s − 1 p r q s − 1 − r ≤ M s − 1 min { s , M − m M } , (67) where M / ( M − m ) M/(M-m) M / ( M − m ) is interpreted as + ∞ +\infty + ∞ when p = q p=q p = q .
Proof of Lemma C.3. By symmetry, assume p ≤ q p \le q p ≤ q , so M = q M=q M = q , m = p m=p m = p , and set u : = p / q ∈ ( 0 , 1 ] u:=p/q \in(0,1] u := p / q ∈ ( 0 , 1 ] . Then ∑ r = 0 s − 1 p r q s − 1 − r = q s − 1 ∑ r = 0 s − 1 u r \sum_{r=0}^{s-1}p^rq^{s-1-r}=q^{s-1}\sum_{r=0}^{s-1}u^r ∑ r = 0 s − 1 p r q s − 1 − r = q s − 1 ∑ r = 0 s − 1 u r . Since u ∈ ( 0 , 1 ] u \in(0,1] u ∈ ( 0 , 1 ] , we have ∑ r = 0 s − 1 u r ≤ s ∧ ( 1 − u ) − 1 = min { s , q / ( q − p ) } \sum_{r=0}^{s-1}u^r \le s\wedge(1-u)^{-1}=\min\{s,q/(q-p)\} ∑ r = 0 s − 1 u r ≤ s ∧ ( 1 − u ) − 1 = min { s , q / ( q − p )} . Conversely, if 1 − u ≤ 1 / s 1-u \le1/s 1 − u ≤ 1/ s , then u ≥ 1 − 1 / s u \ge1-1/s u ≥ 1 − 1/ s gives ∑ r = 0 s − 1 u r ≥ s ( 1 − ( 1 − 1 / s ) s ) ≥ ( 1 − e − 1 ) s \sum_{r=0}^{s-1}u^r \ge s(1-(1-1/s)^s)\ge(1-e^{-1})s ∑ r = 0 s − 1 u r ≥ s ( 1 − ( 1 − 1/ s ) s ) ≥ ( 1 − e − 1 ) s ; if 1 − u > 1 / s 1-u>1/s 1 − u > 1/ s , then u s < ( 1 − 1 / s ) s < e − 1 u^s<(1-1/s)^s<e^{-1} u s < ( 1 − 1/ s ) s < e − 1 gives ∑ r = 0 s − 1 u r = ( 1 − u s ) / ( 1 − u ) ≥ ( 1 − e − 1 ) q / ( q − p ) \sum_{r=0}^{s-1}u^r=(1-u^s)/(1-u)\ge(1-e^{-1})q/(q-p) ∑ r = 0 s − 1 u r = ( 1 − u s ) / ( 1 − u ) ≥ ( 1 − e − 1 ) q / ( q − p ) . □
Operator Lemmas from Prior Work
Lemma C.4 (Sub-Gaussian Implies Fourth-Moment Upper Bound; [37 , 12 , 34 Proposition 2.5.2] ). Suppose E [ x x ⊤ ] = H ≻ 0 \mathbb{E}[\mathbf{x}\mathbf{x}^{\top}]=\mathbf{H}\succ0 E [ x x ⊤ ] = H ≻ 0 and the standardized feature projections are σ z 2 \sigma_z^2 σ z 2 -sub-Gaussian, in the sense that E [ exp ( ⟨ u , x ⟩ ) ] ≤ exp ( σ z 2 ∥ u ∥ H 2 / 2 ) \mathbb{E}[\exp(\langle\mathbf{u},\mathbf{x}\rangle)]\le\exp(\sigma_z^2\|\mathbf{u}\|_{\mathbf{H}}^2/2) E [ exp (⟨ u , x ⟩)] ≤ exp ( σ z 2 ∥ u ∥ H 2 /2 ) for all u ∈ H \mathbf{u}\in\mathcal{H} u ∈ H (equivalently, E [ exp ( λ ⟨ v , H − 1 / 2 P m x ⟩ ) ] ≤ exp ( λ 2 σ z 2 / 2 ) \mathbb{E}[\exp(\lambda\langle\mathbf{v},\mathbf{H}^{-1/2}P_m\mathbf{x}\rangle)]\le\exp(\lambda^2\sigma_z^2/2) E [ exp ( λ ⟨ v , H − 1/2 P m x ⟩)] ≤ exp ( λ 2 σ z 2 /2 ) for all finite-rank spectral projections P m = I 0 : m P_m=I_{0:m} P m = I 0 : m , unit vectors v ∈ H \mathbf{v}\in\mathcal{H} v ∈ H , and λ ∈ R \lambda\in\mathbb{R} λ ∈ R ). Then for any PSD operator A ⪰ 0 \mathbf{A}\succeq0 A ⪰ 0 , Assumption 3.2 holds with α = 16 σ z 4 \alpha=16\sigma_z^4 α = 16 σ z 4 :
M ∘ A = E [ ( x ⊤ A x ) x x ⊤ ] ⪯ 16 σ z 4 tr ( H A ) H . \mathcal{M}\circ\mathbf{A}=\mathbb{E}\left[(\mathbf{x}^{\top}\mathbf{A}\mathbf{x})\mathbf{x}\mathbf{x}^{\top}\right]\preceq16\sigma_z^4\operatorname{tr}(\mathbf{H}\mathbf{A})\mathbf{H}. M ∘ A = E [ ( x ⊤ Ax ) x x ⊤ ] ⪯ 16 σ z 4 tr ( HA ) H .
Lemma C.5 (Properties of PSD Super-Operators; [37 b] (Lemmas 4.1 and B.1; Jain et al. 2017a)). A linear super-operator O \mathcal{O} O on symmetric operators is called a PSD mapping if A ⪰ 0 \mathbf{A}\succeq0 A ⪰ 0 implies O ∘ A ⪰ 0 \mathcal{O}\circ\mathbf{A}\succeq0 O ∘ A ⪰ 0 . Under Assumptions 3.1 and 3.2, the super-operators defined in (5) and (32) are well-defined on symmetric trace-class operators and satisfy:
M \mathcal{M} M and M ~ \widetilde{\mathcal{M}} M are both PSD mappings.
I − γ T \mathcal{I}-\gamma\mathcal{T} I − γ T and I − γ T ~ \mathcal{I}-\gamma\widetilde{\mathcal{T}} I − γ T are both PSD mappings for all γ > 0 \gamma>0 γ > 0 .
M − M ~ \mathcal{M}-\widetilde{\mathcal{M}} M − M and T ~ − T = γ ( M − M ~ ) \widetilde{\mathcal{T}}-\mathcal{T}=\gamma(\mathcal{M}-\widetilde{\mathcal{M}}) T − T = γ ( M − M ) are both PSD mappings; in particular, ( I − γ T ) ∘ A ⪰ ( I − γ T ~ ) ∘ A = ( I − γ H ) A ( I − γ H ) (\mathcal{I}-\gamma\mathcal{T})\circ\mathbf{A}\succeq(\mathcal{I}-\gamma\widetilde{\mathcal{T}})\circ\mathbf{A}=(\mathcal{I}-\gamma\mathbf{H})\mathbf{A}(\mathcal{I}-\gamma\mathbf{H}) ( I − γ T ) ∘ A ⪰ ( I − γ T ) ∘ A = ( I − γ H ) A ( I − γ H ) for all A ⪰ 0 \mathbf{A}\succeq0 A ⪰ 0 .
Lemma C.6 (Bias-Variance Risk Decomposition; [37 Lemmas B.2, C.1, D.1, and D.6] ). Consider SGD with tail-averaging (4) and recall the definitions of bias and variance in (9) . Under Assumptions 3.1, 3.2, and 3.3 (which ensure E ∥ η t b i a s ∥ H 2 < ∞ \mathbb{E}\|\eta_t^{\mathrm{bias}}\|_{\mathbf{H}}^2<\infty E ∥ η t bias ∥ H 2 < ∞ and E ∥ η t v a r i a n c e ∥ H 2 < ∞ \mathbb{E}\|\eta_t^{\mathrm{variance}}\|_{\mathbf{H}}^2<\infty E ∥ η t variance ∥ H 2 < ∞ for all t ≥ 0 t\ge0 t ≥ 0 ) :
For any data distribution D \mathcal{D} D , the expected excess risk satisfies
E [ L ( w ‾ s : s + N ) − L ( w ∗ ) ] = 1 2 ⟨ H , E [ η ‾ s : s + N ⊗ η ‾ s : s + N ] ⟩ ≤ ( b i a s + v a r i a n c e ) 2 ≤ 2 ⋅ b i a s + 2 ⋅ v a r i a n c e . \mathbb{E}\left[L(\overline{\mathbf{w}}_{s:s+N})-L(\mathbf{w}^*)\right]
=\frac{1}{2}\left\langle\mathbf{H},\mathbb{E}\left[\overline{\boldsymbol{\eta}}_{s:s+N}\otimes\overline{\boldsymbol{\eta}}_{s:s+N}\right]\right\rangle
\le\left(\sqrt{\mathrm{bias}}+\sqrt{\mathrm{variance}}\right)^2
\le2\cdot\mathrm{bias}+2\cdot\mathrm{variance}. E [ L ( w s : s + N ) − L ( w ∗ ) ] = 2 1 ⟨ H , E [ η s : s + N ⊗ η s : s + N ] ⟩ ≤ ( bias + variance ) 2 ≤ 2 ⋅ bias + 2 ⋅ variance .
If the residual noise ξ t = y t − ⟨ w ∗ , x t ⟩ \xi_t=y_t-\langle\mathbf{w}^*,\mathbf{x}_t\rangle ξ t = y t − ⟨ w ∗ , x t ⟩ is well-specified as in (6) (so E [ ξ t ∣ x t ] = 0 \mathbb{E}[\xi_t\mid\mathbf{x}_t]=0 E [ ξ t ∣ x t ] = 0 ), then E [ η ‾ s : s + N b i a s ⊗ η ‾ s : s + N v a r i a n c e ] = 0 \mathbb{E}[\overline{\boldsymbol{\eta}}_{s:s+N}^{\mathrm{bias}}\otimes\overline{\boldsymbol{\eta}}_{s:s+N}^{\mathrm{variance}}]=0 E [ η s : s + N bias ⊗ η s : s + N variance ] = 0 and the excess risk decomposes with exact equality:
E [ L ( w ‾ s : s + N ) − L ( w ∗ ) ] = b i a s + v a r i a n c e . \mathbb{E}\left[L(\overline{\mathbf{w}}_{s:s+N})-L(\mathbf{w}^*)\right]=\mathrm{bias}+\mathrm{variance}. E [ L ( w s : s + N ) − L ( w ∗ ) ] = bias + variance .
Lemma C.7 (Single-Step Trace Identity and Interval Telescoping Inequality; adapted from [37 Eqs. (B.1) and (D.4] ). Under Assumptions 3.1 and 3.2, if γ < 1 / ( α tr ( H ) ) \gamma<1/(\alpha\operatorname{tr}(\mathbf{H})) γ < 1/ ( α tr ( H )) , then for any integers 0 ≤ a < b 0\le a<b 0 ≤ a < b and B a , b : = B a − ( I − γ H ) b − a B a ( I − γ H ) b − a \mathbf{B}_{a,b}:=\mathbf{B}_a-(\mathcal{I}-\gamma\mathbf{H})^{b-a}\mathbf{B}_a(\mathcal{I}-\gamma\mathbf{H})^{b-a} B a , b := B a − ( I − γ H ) b − a B a ( I − γ H ) b − a , the bias covariance sequence { B u } u ≥ 0 \{\mathbf{B}_u\}_{u\ge0} { B u } u ≥ 0 satisfies
γ ∑ u = a b − 1 tr ( H B u ) ≤ tr ( B a ) − tr ( B b ) 2 − γ α tr ( H ) ≤ 1 2 ( 1 − γ α tr ( H ) ) tr ( B a , b ) . (68) \gamma\sum_{u=a}^{b-1}\operatorname{tr}(\mathbf{H}\mathbf{B}_u)
\le\frac{\operatorname{tr}(\mathbf{B}_a)-\operatorname{tr}(\mathbf{B}_b)}{2-\gamma\alpha\operatorname{tr}(\mathbf{H})}
\le\frac{1}{2(1-\gamma\alpha\operatorname{tr}(\mathbf{H}))}\operatorname{tr}(\mathbf{B}_{a,b}). \tag*{(68)} γ u = a ∑ b − 1 tr ( H B u ) ≤ 2 − γ α tr ( H ) tr ( B a ) − tr ( B b ) ≤ 2 ( 1 − γ α tr ( H )) 1 tr ( B a , b ) . (68)
Proof of Lemma C.7. We adapt the telescoping argument of Zou et al. [37 ] to a general interval [ a , b ] [a,b] [ a , b ] for completeness. For each step u ≥ 0 u \ge0 u ≥ 0 , taking the trace of B u + 1 = ( I − γ T ) ∘ B u = B u − γ ( H B u + B u H ) + γ 2 M ∘ B u \mathbf{B}_{u+1} = (\mathcal{I}-\gamma\mathcal{T})\circ\mathbf{B}_u = \mathbf{B}_u-\gamma(\mathbf{H}\mathbf{B}_u+\mathbf{B}_u\mathbf{H})+\gamma^2\mathcal{M}\circ\mathbf{B}_u B u + 1 = ( I − γ T ) ∘ B u = B u − γ ( H B u + B u H ) + γ 2 M ∘ B u and applying Assumption 3.2 (tr ( M ∘ B u ) ≤ α tr ( H ) tr ( H B u ) \operatorname{tr}(\mathcal{M}\circ\mathbf{B}_u) \le\alpha\operatorname{tr}(\mathbf{H})\operatorname{tr}(\mathbf{H}\mathbf{B}_u) tr ( M ∘ B u ) ≤ α tr ( H ) tr ( H B u ) ) gives
tr ( B u + 1 ) ≤ tr ( B u ) − γ ( 2 − γ α tr ( H ) ) tr ( H B u ) . \operatorname{tr}(\mathbf{B}_{u+1}) \le\operatorname{tr}(\mathbf{B}_u)-\gamma\left(2-\gamma\alpha\operatorname{tr}(\mathbf{H})\right)\operatorname{tr}(\mathbf{H}\mathbf{B}_u). tr ( B u + 1 ) ≤ tr ( B u ) − γ ( 2 − γ α tr ( H ) ) tr ( H B u ) .
Summing this inequality over u ∈ { a , … , b − 1 } u \in\{a,\ldots,b-1\} u ∈ { a , … , b − 1 } and dividing by 2 − γ α tr ( H ) > 2 ( 1 − γ α tr ( H ) ) > 0 2-\gamma\alpha\operatorname{tr}(\mathbf{H})>2(1-\gamma\alpha\operatorname{tr}(\mathbf{H}))>0 2 − γ α tr ( H ) > 2 ( 1 − γ α tr ( H )) > 0 yields the first inequality in (C.4). Finally, since B b = ( I − γ T ) b − a ∘ B a ⪰ ( I − γ H ) b − a B a ( I − γ H ) b − a \mathbf{B}_b=(\mathcal{I}-\gamma\mathcal{T})^{b-a}\circ\mathbf{B}_a \succeq(\mathbf{I}-\gamma\mathbf{H})^{b-a}\mathbf{B}_a(\mathbf{I}-\gamma\mathbf{H})^{b-a} B b = ( I − γ T ) b − a ∘ B a ⪰ ( I − γ H ) b − a B a ( I − γ H ) b − a by Lemma C.5(3), we have tr ( B a ) − tr ( B b ) ≤ tr ( B a , b ) \operatorname{tr}(\mathbf{B}_a)-\operatorname{tr}(\mathbf{B}_b)\le\operatorname{tr}(\mathbf{B}_{a,b}) tr ( B a ) − tr ( B b ) ≤ tr ( B a , b ) , establishing the second inequality in (C.4). □
Lemma C.8 (Upper Bounds on Constant-Stepsize Variance Iterates; [37 Lemmas B.4 and B.5] ). Under Assumptions 3.1, 3.2, and 3.3, if the constant stepsize satisfies 0 < γ < 1 / ( α tr ( H ) ) 0<\gamma<1/(\alpha\operatorname{tr}(\mathbf{H})) 0 < γ < 1/ ( α tr ( H )) , then the variance covariance sequence { C t } t ≥ 0 \{\mathbf{C}_t\}_{t\ge0} { C t } t ≥ 0 defined in (34) satisfies
C t ⪯ γ σ 2 1 − γ α tr ( H ) ( I − ( I − γ H ) t ) ⪯ γ σ 2 1 − γ α tr ( H ) I , ∀ t ≥ 0. \mathbf{C}_t \preceq\frac{\gamma\sigma^2}{1-\gamma\alpha\operatorname{tr}(\mathbf{H})}\left(\mathbf{I}-(\mathbf{I}-\gamma\mathbf{H})^t\right) \preceq\frac{\gamma\sigma^2}{1-\gamma\alpha\operatorname{tr}(\mathbf{H})}\mathbf{I}, \qquad\forall t\ge0. C t ⪯ 1 − γ α tr ( H ) γ σ 2 ( I − ( I − γ H ) t ) ⪯ 1 − γ α tr ( H ) γ σ 2 I , ∀ t ≥ 0.