Introduction
Accelerated stochastic optimization is well understood under classical smoothness, where the gradient is globally Lipschitz. Two additional difficulties arise in settings relevant to modern learning. First, the local smoothness of the objective may grow with the gradient norm, a dependence observed by Zhang et al. [24 ] along training trajectories. Second, stochastic gradient noise can be heavy-tailed, as observed, for example, in BERT pre-training ([25 ] ). Together, these effects make acceleration challenging: high curvature may require smaller steps, while large stochastic gradient errors can be amplified by momentum. Gradient clipping can limit the effect of such errors, but introduces bias that must also be controlled. Under classical L L L -smoothness, high-probability accelerated guarantees under heavy-tailed noise have been established by Sadiev et al. [18 ] ; Nguyen et al. [17 ] .
The ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness condition provides a natural model for such nonuniform curvature. For twice-differentiable objectives, this dependence is captured by
∥ ∇ 2 f ( x ) ∥ ≤ L 0 + L 1 ∥ ∇ f ( x ) ∥ \lVert\nabla^2 f(x) \rVert\le L_0 + L_1 \lVert\nabla f(x) \rVert ∥ ∇ 2 f ( x )∥ ≤ L 0 + L 1 ∥ ∇ f ( x )∥
(Zhang et al. [24 ] ). First-order formulations for continuously differentiable functions were studied by Zhang et al. [23 ] and Chen et al. [1 ] ; we adopt the formulation in Assumption 2. In this model, the curvature can be large when the gradient norm is large, while becoming comparable to L 0 L_0 L 0 when ∥ ∇ f ( x ) ∥ \lVert\nabla f(x) \rVert ∥ ∇ f ( x )∥ is of order L 0 / L 1 L_0/L_1 L 0 / L 1 .
A natural approach is to take an accelerated method with guarantees under classical smoothness, show that its iterates remain in a ball of radius O ( R ) O(R) O ( R ) (Sadiev et al. [18 ] ), and use ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness to obtain an effective smoothness constant over this ball. One could then apply the standard accelerated analysis with this local constant. However, the resulting smoothness parameter can be as large as L 0 exp ( O ( L 1 R ) ) L_0 \exp(O(L_1R)) L 0 exp ( O ( L 1 R )) , leading to an exponential dependence on L 1 R L_1R L 1 R .
Deterministic acceleration under generalized smoothness has been studied in several recent works. Gorbunov et al. [9 ] proposed an accelerated Similar Triangles method for convex ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smooth optimization, although its convergence guarantee retains an exponential dependence on L 1 R L_1R L 1 R , where R R R bounds the initial distance to a minimizer. Vankov et al. [21 ] subsequently established accelerated guarantees without such an exponential dependence, using a method that requires solving a one-dimensional line-search subproblem at each iteration, making a direct extension to stochastic gradients nontrivial. More recently, Tyurin [20 ] obtained deterministic accelerated guarantees under an ℓ \ell ℓ -smoothness assumption that generalizes ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness. They consider both a gradient-descent warm start followed by acceleration and a method that starts directly from the initial point, adjusting its stepsizes to control the initial high-curvature regime. Under ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness, these methods recover an L 1 L_1 L 1 -independent leading high-accuracy rate after an initial transient. Our two-phase and one-phase approaches follow similar principles, but use Clip-SSTM with different weight choices. The additional challenge is to control curvature and accumulated clipping errors simultaneously along the accelerated trajectory, with high probability under heavy-tailed noise.
Stochastic accelerated-gradient methods under ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness have also been analyzed recently. Yu et al. (2025) study randomized stochastic accelerated gradient methods under ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness and relaxed affine noise assumptions, obtaining high-probability convergence guarantees for convex optimization; their analysis, however, does not yield an accelerated convergence rate in this setting. Dvinskikh et al. [4 ] establish high-probability accelerated guarantees for convex ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smooth optimization under unbiased stochastic gradients with additive norm-sub-Gaussian noise, while avoiding an exponential dependence on L 1 R L_1R L 1 R . Their guarantees assume light-tailed noise, whereas we assume only a bounded central α \alpha α -th moment with α ∈ ( 1 , 2 ] \alpha\in(1,2] α ∈ ( 1 , 2 ] , which is a weaker condition.
This gap raises a natural question:
Can classical clipped accelerated schemes achieve high-probability acceleration under ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness and heavy-tailed noise without exponential dependence on L 1 R L_1R L 1 R ?
We answer affirmatively.
Our contributions. Under Assumptions 1 (convexity), Assumptions 2 (( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness), and Assumptions 3 (bounded central α \alpha α -th moment, α ∈ ( 1 , 2 ] \alpha\in(1,2] α ∈ ( 1 , 2 ] ), we establish high-probability accelerated guarantees for two approaches. Let ∥ x 0 − x ⋆ ∥ ≤ R 0 \lVert x_0-x^\star\rVert\le R_0 ∥ x 0 − x ⋆ ∥ ≤ R 0 . Our total complexity bounds avoid exponential dependence on L 1 R 0 L_1R_0 L 1 R 0 .
Two-phase acceleration. We use a nonaccelerated method as a warm start to obtain a point x ˉ \bar{x} x ˉ satisfying f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 f(\bar{x})-f^\star\le cL_0/L_1^2 f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 , an accuracy-independent threshold implying ∥ ∇ f ( x ˉ ) ∥ = O ( L 0 / L 1 ) \lVert\nabla f(\bar{x})\rVert= O(L_0/L_1) ∥ ∇ f ( x ˉ )∥ = O ( L 0 / L 1 ) with high probability. Starting from x ˉ \bar{x} x ˉ , Clip-SSTM satisfies, with probability at least 1 − δ 1-\delta 1 − δ ,
f ( y K ) − f ⋆ ≤ O ~ ( L 0 R 2 / K 2 + σ R K − ( α − 1 ) / α ) f(y_K)-f^\star\le\widetilde{O}\left(L_0R^2/K^2+\sigma RK^{-(\alpha-1)/\alpha}\right) f ( y K ) − f ⋆ ≤ O ( L 0 R 2 / K 2 + σ R K − ( α − 1 ) / α )
after K K K additional stochastic gradient evaluations, where ∥ x ˉ − x ⋆ ∥ ≤ R \lVert\bar{x}-x^\star\rVert\le R ∥ x ˉ − x ⋆ ∥ ≤ R and O ~ \widetilde{O} O hides logarithmic factors in K K K and 1 / δ 1/\delta 1/ δ . The analysis couples two potentials, anchored at x ˉ \bar{x} x ˉ and x ⋆ x^\star x ⋆ , to keep the accelerated trajectory in the low-curvature region despite momentum.
One-phase acceleration. Clip-SSTM can also start directly from x 0 x_0 x 0 , using a modified deterministic weight schedule to control the high-curvature transient before recovering acceleration. With the prescribed initialization, it recovers the classical L L L -smooth accelerated rate after a transient that is accuracy-independent up to logarithmic factors. Here the distance bound is R 0 R_0 R 0 .
Complexity guarantees. For the two-phase method, the warm start also guarantees ∥ x ˉ − x ⋆ ∥ ≤ 2 R 0 \lVert\bar{x}-x^\star\rVert\le2R_0 ∥ x ˉ − x ⋆ ∥ ≤ 2 R 0 on its success event, so the accelerated phase can use R = 2 R 0 R=2R_0 R = 2 R 0 . Including the warm-start cost, Corollary 1 ensures f ( y ) − f ⋆ ≤ ε f(y)-f^\star\le\varepsilon f ( y ) − f ⋆ ≤ ε with probability at least 1 − δ 1-\delta 1 − δ using
O ~ ( 1 + ( L 1 R 0 ) 2 + L 0 R 0 2 ε + ( σ R 0 ε ) α α − 1 ) \widetilde{O}\left(1+(L_1R_0)^2+\sqrt{\frac{L_0R_0^2}{\varepsilon}}+\left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}\right) O ( 1 + ( L 1 R 0 ) 2 + ε L 0 R 0 2 + ( ε σ R 0 ) α − 1 α )
stochastic gradient evaluations. For the one-phase method, Theorem 2 with the initialization in Remark 1 ensures f ( y K ) − f ⋆ ≤ ε f(y_K)-f^\star\le\varepsilon f ( y K ) − f ⋆ ≤ ε with the same probability using
O ~ ( K 0 + L 0 R 0 2 ε + ( σ R 0 ε ) α α − 1 ) \widetilde{O}\left(K_0+\sqrt{\frac{L_0R_0^2}{\varepsilon}}+\left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}\right) O ( K 0 + ε L 0 R 0 2 + ( ε σ R 0 ) α − 1 α )
stochastic gradient evaluations. The transient K 0 K_0 K 0 depends logarithmically on the initial-gap bound. Using the worst-case gap bound gives K 0 = O ~ ( ( 1 + L 1 R 0 ) 2 ) K_0=\widetilde{O}((1+L_1R_0)^2) K 0 = O (( 1 + L 1 R 0 ) 2 ) ; a tighter bound can give a smaller transient. Up to logarithmic factors and an additional initial transient, both methods recover the complexity of accelerated methods for classical L L L -smooth convex optimization under heavy-tailed noise, with L L L replaced by L 0 L_0 L 0 . The dependence on L 1 L_1 L 1 is confined to the transient.
Table 1 : Comparison with related results for convex optimization. HP denotes a high-probability guarantee. Here α \alpha α -moment means bounded central α \alpha α -th moment noise, α ∈ ( 1 , 2 ] \alpha\in(1, 2] α ∈ ( 1 , 2 ] , and “No exp. dep.” means no exponential dependence on L 1 R L_{1}R L 1 R (or H 1 R \sqrt{H_{1}}R H 1 R under ( H 0 , H 1 ) (H_{0}, H_{1}) ( H 0 , H 1 ) -smoothness).
Table 1.
Det. = deterministic. 1 Vankov et al. [21 ] use a one-dimensional line search with function values for their accelerated method. 2 Yu et al. (2025) analyze stochastic accelerated-gradient schemes under relaxed affine almost-sure noise, with a sub-Gaussian extension. Their convex high-probability guarantee does not establish an accelerated rate. 3 Concurrent work. Lobanov et al. [13 ] consider ( H 0 , H 1 ) (H_{0}, H_{1}) ( H 0 , H 1 ) -smoothness and a generalized central moment bound with constant, gradient-dependent, and objective-gap-dependent terms. Their method uses projections and phase restarts; our methods require neither projections nor repeated restarts. See the discussion of concurrent work for details.
Preliminaries
Notation. We work in R d \mathbb{R}^{d} R d , equipped with the standard Euclidean inner product ⟨ ⋅ , ⋅ ⟩ \langle\cdot, \cdot\rangle ⟨ ⋅ , ⋅ ⟩ and norm ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ . For λ > 0 \lambda> 0 λ > 0 , let B ( 0 , λ ) : = { x ∈ R d : ∥ x ∥ ≤ λ } B(0, \lambda) := \{x \in\mathbb{R}^{d} : \lVert x \rVert\le\lambda\} B ( 0 , λ ) := { x ∈ R d : ∥ x ∥ ≤ λ } . We define the clipping operator clip ( x , λ ) \operatorname{clip}(x, \lambda) clip ( x , λ ) as the Euclidean projection onto this ball:
clip ( x , λ ) : = Π B ( 0 , λ ) ( x ) = { x , ∥ x ∥ ≤ λ , λ ∥ x ∥ x , ∥ x ∥ > λ . \operatorname{clip}(x, \lambda) := \Pi_{B(0,\lambda)}(x) =
\begin{cases}
x, & \lVert x \rVert\le\lambda,\\
\frac{\lambda}{\lVert x \rVert}x, & \lVert x \rVert> \lambda.
\end{cases} clip ( x , λ ) := Π B ( 0 , λ ) ( x ) = { x , ∥ x ∥ λ x , ∥ x ∥ ≤ λ , ∥ x ∥ > λ .
Thus, clipping preserves the direction of a nonzero vector while limiting its norm to λ \lambda λ . We assume that f f f admits a minimizer, fix x ⋆ ∈ arg min x ∈ R d f ( x ) x^{\star} \in\arg\min_{x \in\mathbb{R}^{d}} f(x) x ⋆ ∈ arg min x ∈ R d f ( x ) , and denote f ⋆ : = f ( x ⋆ ) f^{\star} := f(x^{\star}) f ⋆ := f ( x ⋆ ) . We use L 0 , L 1 > 0 L_{0}, L_{1} > 0 L 0 , L 1 > 0 for the constants in the ( L 0 , L 1 ) (L_{0}, L_{1}) ( L 0 , L 1 ) -smoothness condition introduced below. Finally, let ν ∈ ( 0 , 1 ) \nu\in(0, 1) ν ∈ ( 0 , 1 ) be the unique positive solution of ν e ν = 1 \nu e^{\nu} = 1 ν e ν = 1 .
Problem. We minimize f f f over R d \mathbb{R}^{d} R d using stochastic gradient samples ∇ f ( x , ξ ) \nabla f(x, \xi) ∇ f ( x , ξ ) , under the assumptions stated below. We establish high-probability objective-gap bounds for Clip-SSTM with two initialization and weight choices: standard accelerated weights after a warm start in a low-curvature regime, and modified weights that allow a direct start from x 0 x_{0} x 0 without a small-gap requirement.
Common algorithm. Both procedures use Algorithm 1, where v 0 v_{0} v 0 is the starting point, A 0 ≥ 0 A_{0} \ge0 A 0 ≥ 0 the initial accumulated weight, α k + 1 > 0 \alpha_{k+1} > 0 α k + 1 > 0 the added weight, and λ k + 1 > 0 \lambda_{k+1} > 0 λ k + 1 > 0 the clipping level. Their initialization and weight choices are specified below.
Algorithm 1 Clip-SSTM
Input: Starting point v 0 v_0 v 0 , iteration budget K ≥ 1 K \ge1 K ≥ 1 , initial weight A 0 ≥ 0 A_0 \ge0 A 0 ≥ 0 , positive weights { α k } k = 1 K \{\alpha_k\}_{k=1}^{K} { α k } k = 1 K , and positive clipping levels { λ k } k = 1 K \{\lambda_k\}_{k=1}^{K} { λ k } k = 1 K .
y 0 = z 0 = v 0 y_0=z_0=v_0 y 0 = z 0 = v 0 .
for k = 0 , … , K − 1 k=0,\ldots,K-1 k = 0 , … , K − 1 do
A k + 1 = A k + α k + 1 A_{k+1}=A_k+\alpha_{k+1} A k + 1 = A k + α k + 1 .
x k + 1 = A k y k + α k + 1 z k A k + 1 x_{k+1}=\frac{A_k y_k+\alpha_{k+1}z_k}{A_{k+1}} x k + 1 = A k + 1 A k y k + α k + 1 z k .
Draw a fresh sample ξ k + 1 \xi_{k+1} ξ k + 1 , independent of the past.
g k + 1 = clip ( ∇ f ( x k + 1 , ξ k + 1 ) , λ k + 1 ) g_{k+1}=\operatorname{clip}(\nabla f(x_{k+1},\xi_{k+1}),\lambda_{k+1}) g k + 1 = clip ( ∇ f ( x k + 1 , ξ k + 1 ) , λ k + 1 ) .
z k + 1 = z k − α k + 1 g k + 1 z_{k+1}=z_k-\alpha_{k+1}g_{k+1} z k + 1 = z k − α k + 1 g k + 1 .
y k + 1 = A k y k + α k + 1 z k + 1 A k + 1 y_{k+1}=\frac{A_k y_k+\alpha_{k+1}z_{k+1}}{A_{k+1}} y k + 1 = A k + 1 A k y k + α k + 1 z k + 1 .
end for
Output: y K y_K y K .
Two-phase procedure. Phase I runs a warm-start method and returns a point x ˉ \bar{x} x ˉ . On its success event,
{ ∥ x ˉ − x ⋆ ∥ ≤ R , f ( x ˉ ) − f ⋆ ≤ c L 0 L 1 2 . \left\{
\begin{aligned}
\lVert\bar{x}-x^\star\rVert&\le R,\\
f(\bar{x})-f^\star&\le c\frac{L_0}{L_1^2}.
\end{aligned}
\right. ⎩ ⎨ ⎧ ∥ x ˉ − x ⋆ ∥ f ( x ˉ ) − f ⋆ ≤ R , ≤ c L 1 2 L 0 .
where c > 0 c>0 c > 0 is a universal constant specified in the corresponding convergence theorem.
Phase II runs Clip-SSTM using fresh stochastic-gradient samples. For a parameter a > 0 a>0 a > 0 , we initialize y 0 = z 0 = x ˉ y_0=z_0=\bar{x} y 0 = z 0 = x ˉ , A 0 = 0 A_0=0 A 0 = 0 , and, for k ≥ 0 k\ge0 k ≥ 0 , choose α k + 1 = k + 2 a L 0 \alpha_{k+1}=\frac{k+2}{aL_0} α k + 1 = a L 0 k + 2 . For this choice, A k = k ( k + 3 ) 2 a L 0 A_k=\frac{k(k+3)}{2aL_0} A k = 2 a L 0 k ( k + 3 ) , and A k + 1 ≥ a L 0 2 α k + 1 2 A_{k+1}\ge\frac{aL_0}{2}\alpha_{k+1}^2 A k + 1 ≥ 2 a L 0 α k + 1 2 . Thus A k A_k A k grows quadratically from the beginning of Phase II. The small-gap condition at x ˉ \bar{x} x ˉ places the accelerated method in a local low-curvature regime, allowing us to use from the first iteration the standard accelerated weight schedule of Sadiev et al. [18 ] .
One-phase procedure. The one-phase procedure runs Clip-SSTM directly from x 0 x_0 x 0 . We assume ∥ x 0 − x ⋆ ∥ ≤ R 0 \lVert x_0-x^\star\rVert\le R_0 ∥ x 0 − x ⋆ ∥ ≤ R 0 , but impose no small-gap condition. Let Δ 0 : = f ( x 0 ) − f ⋆ \Delta_0:=f(x_0)-f^\star Δ 0 := f ( x 0 ) − f ⋆ , and let Δ ˉ 0 ≥ Δ 0 \bar{\Delta}_0\ge\Delta_0 Δ ˉ 0 ≥ Δ 0 be any deterministic upper bound on the initial objective gap.
Let a > 0 a>0 a > 0 and ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 be parameters, whose specific choices are given in the corresponding convergence theorem; in particular, the theorem sets ℓ = 8 log ( 4 K / δ ) \ell=8\log(4K/\delta) ℓ = 8 log ( 4 K / δ ) . We initialize y 0 = z 0 = x 0 y_0=z_0=x_0 y 0 = z 0 = x 0 and choose A 0 > 0 A_0>0 A 0 > 0 satisfying A 0 ≤ 1 a L 0 A_0\le\frac{1}{aL_0} A 0 ≤ a L 0 1 and A 0 Δ ˉ 0 ≤ R 0 2 64 A_0\bar{\Delta}_0\le\frac{R_0^2}{64} A 0 Δ ˉ 0 ≤ 64 R 0 2 . In particular, A 0 ( f ( x 0 ) − f ⋆ ) ≤ A 0 Δ ˉ 0 ≤ R 0 2 64 A_0(f(x_0)-f^\star)\le A_0\bar{\Delta}_0\le\frac{R_0^2}{64} A 0 ( f ( x 0 ) − f ⋆ ) ≤ A 0 Δ ˉ 0 ≤ 64 R 0 2 , so the initial objective gap contributes only a controlled amount to the potential used in the analysis. For k ≥ 0 k\ge0 k ≥ 0 , we use the weight rule
α k + 1 = min { A k 64 ℓ ( 1 + L 1 R 0 ) , A k a L 0 } . \alpha_{k+1}=\min\left\{\frac{A_k}{64\ell(1+L_1R_0)},\sqrt{\frac{A_k}{aL_0}}\right\}. α k + 1 = min { 64 ℓ ( 1 + L 1 R 0 ) A k , a L 0 A k } .
Consequently, α k + 1 A k ≤ 1 64 ℓ ( 1 + L 1 R 0 ) \frac{\alpha_{k+1}}{A_k}\le\frac{1}{64\ell(1+L_1R_0)} A k α k + 1 ≤ 64 ℓ ( 1 + L 1 R 0 ) 1 and A k + 1 ≥ A k ≥ a L 0 α k + 1 2 A_{k+1}\ge A_k\ge aL_0\alpha_{k+1}^2 A k + 1 ≥ A k ≥ a L 0 α k + 1 2 . The two terms in the minimum correspond to two regimes. If
A k < [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 , A_k<\frac{[64\ell(1+L_1R_0)]^2}{aL_0}, A k < a L 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 ,
then A k 64 ℓ ( 1 + L 1 R 0 ) < A k a L 0 \frac{A_k}{64\ell(1+L_1R_0)}<\sqrt{\frac{A_k}{aL_0}} 64 ℓ ( 1 + L 1 R 0 ) A k < a L 0 A k , so the method uses α k + 1 = A k 64 ℓ ( 1 + L 1 R 0 ) \alpha_{k+1}=\frac{A_k}{64\ell(1+L_1R_0)} α k + 1 = 64 ℓ ( 1 + L 1 R 0 ) A k . Thus, in this regime, the weight α k + 1 \alpha_{k+1} α k + 1 is deliberately chosen smaller than the standard accelerated square-root weight A k / ( a L 0 ) \sqrt{A_k/(aL_0)} A k / ( a L 0 ) .
Once
A k ≥ [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 , A_k \ge\frac{[64\ell(1+L_1R_0)]^2}{aL_0}, A k ≥ a L 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 ,
the square-root term becomes the smaller one, and hence α k + 1 = A k a L 0 \alpha_{k+1}=\sqrt{\frac{A_k}{aL_0}} α k + 1 = a L 0 A k . Since A k + 1 ≥ A k A_{k+1}\ge A_k A k + 1 ≥ A k , the square-root choice remains active thereafter. The method then follows the standard accelerated square-root schedule
A k + 1 = A k + A k a L 0 , A_{k+1}=A_k+\sqrt{\frac{A_k}{aL_0}}, A k + 1 = A k + a L 0 A k ,
for which A k A_k A k grows quadratically with the number of subsequent iterations.
Thus, the modified weight rule uses smaller-than-standard accelerated weights during the high-curvature transient and automatically switches to the standard accelerated schedule once this additional restriction is no longer active. The remaining parameter choices and clipping levels for the two procedures are specified in their respective convergence theorems.
Assumptions
We first assume convexity.
Assumption 1 (Convexity). The function f f f is differentiable and convex, i.e.,
f ( y ) ≥ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ ∀ x , y ∈ R d f(y) \ge f(x) + \langle\nabla f(x),y-x\rangle\quad\forall x,y\in\mathbb{R}^d f ( y ) ≥ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ ∀ x , y ∈ R d
We next assume ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness.
Assumption 2 (( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness). There exist constants L 0 , L 1 ≥ 0 L_0,L_1\ge0 L 0 , L 1 ≥ 0 such that, for all x , y ∈ R d x,y\in\mathbb{R}^d x , y ∈ R d ,
∥ ∇ f ( x ) − ∇ f ( y ) ∥ ≤ ( L 0 + L 1 sup u ∈ [ x , y ] ∥ ∇ f ( u ) ∥ ) ∥ x − y ∥ \lVert\nabla f(x)-\nabla f(y)\rVert\le\left(L_0+L_1\sup_{u\in[x,y]}\lVert\nabla f(u)\rVert\right)\lVert x-y\rVert ∥ ∇ f ( x ) − ∇ f ( y )∥ ≤ ( L 0 + L 1 u ∈ [ x , y ] sup ∥ ∇ f ( u )∥ ) ∥ x − y ∥
This assumption is strictly more general than the classical L L L -smoothness assumption, which requires ∇ f \nabla f ∇ f to be globally Lipschitz continuous: it reduces to the standard L L L -smoothness with L = L 0 L=L_0 L = L 0 if L 1 = 0 L_1=0 L 1 = 0 . Moreover, one can construct functions that satisfy Assumption 2 but not L L L -smoothness, e.g., exponent of norm f ( x ) = exp ( ∥ x ∥ ) + exp ( − ∥ x ∥ ) f(x)=\exp(\lVert x\rVert)+\exp(-\lVert x\rVert) f ( x ) = exp (∥ x ∥) + exp ( − ∥ x ∥) , power of norm f ( x ) = ∥ x ∥ n f(x)=\lVert x\rVert^n f ( x ) = ∥ x ∥ n for n > 2 n>2 n > 2 , and exponent of the linear function f ( x ) = exp ( ⟨ a , x ⟩ ) f(x)=\exp(\langle a,x\rangle) f ( x ) = exp (⟨ a , x ⟩) ([1 , 9 ] ).
Finally, we assume unbiasedness and boundedness of the central α \alpha α -th moment.
Assumption 3 (Stochastic oracle). There exist α ∈ ( 1 , 2 ] \alpha\in(1,2] α ∈ ( 1 , 2 ] and σ ≥ 0 \sigma\ge0 σ ≥ 0 such that, for every x ∈ R d x\in\mathbb{R}^d x ∈ R d ,
{ E ξ [ ∇ f ( x , ξ ) ] = ∇ f ( x ) , E ξ [ ∥ ∇ f ( x , ξ ) − ∇ f ( x ) ∥ α ] ≤ σ α . \begin{cases}
\mathbb{E}_{\xi}[\nabla f(x,\xi)]=\nabla f(x),\\
\mathbb{E}_{\xi}[\lVert\nabla f(x,\xi)-\nabla f(x)\rVert^\alpha]\le\sigma^\alpha.
\end{cases} { E ξ [ ∇ f ( x , ξ )] = ∇ f ( x ) , E ξ [∥ ∇ f ( x , ξ ) − ∇ f ( x ) ∥ α ] ≤ σ α .
This assumption has become relatively standard and has been considered in ([25 , 3 , 18 , 17 , 2 ] ). For α < 2 \alpha<2 α < 2 , examples include stable distributions with stability parameter β ∈ ( α , 2 ) \beta\in(\alpha,2) β ∈ ( α , 2 ) . When α = 2 \alpha=2 α = 2 , Assumption 3 reduces to the classical bounded-variance assumption ([16 , 7 , 19 ] ).
Main Results
Two-phase result
We first analyze the two-phase procedure. The following theorem gives the high-probability convergence guarantee for the accelerated stage initialized from a point satisfying the warm-start conditions.
Theorem 1. Suppose that Assumptions 1, 2, and 3 hold, where L 0 , L 1 > 0 L_0,L_1>0 L 0 , L 1 > 0 . Assume that ∥ x ˉ − x ⋆ ∥ ≤ R \lVert\bar{x}-x^\star\rVert\le R ∥ x ˉ − x ⋆ ∥ ≤ R and f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 f(\bar{x})-f^\star\le cL_0/L_1^2 f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 , where R > 0 R>0 R > 0 and 0 < c ≤ 1 / 32 0<c\le1/32 0 < c ≤ 1/32 . Let K ≥ 1 K\ge1 K ≥ 1 , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and set ℓ : = 8 log ( 4 K / δ ) \ell:=8\log(4K/\delta) ℓ := 8 log ( 4 K / δ ) and q : = 1024 1 / α ℓ ( α − 1 ) / α K 1 / α q:=1024^{1/\alpha}\ell^{(\alpha-1)/\alpha}K^{1/\alpha} q := 102 4 1/ α ℓ ( α − 1 ) / α K 1/ α . Let a : = max { 192 ℓ 2 c , 2 q 2 c ( σ L 1 L 0 ) 2 , 2 q σ K L 0 R } a:=\max\left\{\frac{192\ell^2}{c},\frac{2q^2}{c}\left(\frac{\sigma L_1}{L_0}\right)^2,\frac{2q\sigma K}{L_0R}\right\} a := max { c 192 ℓ 2 , c 2 q 2 ( L 0 σ L 1 ) 2 , L 0 R 2 q σ K } . Run Clip-SSTM with starting point v 0 = x ˉ v_0=\bar{x} v 0 = x ˉ , initialization A 0 = 0 A_0=0 A 0 = 0 and y 0 = z 0 = x ˉ y_0=z_0=\bar{x} y 0 = z 0 = x ˉ , and, for every k = 0 , … , K − 1 k=0,\ldots,K-1 k = 0 , … , K − 1 , let α k + 1 : = k + 2 a L 0 \alpha_{k+1}:=\frac{k+2}{aL_0} α k + 1 := a L 0 k + 2 . For i = 1 , … , K i=1,\ldots,K i = 1 , … , K , let λ i : = 1 α i ℓ min { c A i L 0 L 1 , R } \lambda_i:=\frac{1}{\alpha_i\ell}\min\left\{\sqrt{\frac{cA_iL_0}{L_1}},R\right\} λ i := α i ℓ 1 min { L 1 c A i L 0 , R } . Then, with probability at least 1 − δ 1-\delta 1 − δ , the output y K y_K y K of Clip-SSTM satisfies
f ( y K ) − f ⋆ = O ~ ( L 0 R 2 K 2 + σ R K ( α − 1 ) / α ) , f(y_K)-f^\star=\widetilde{O}\left(\frac{L_0R^2}{K^2}+\frac{\sigma R}{K^{(\alpha-1)/\alpha}}\right), f ( y K ) − f ⋆ = O ( K 2 L 0 R 2 + K ( α − 1 ) / α σ R ) , where O ~ \widetilde{O} O suppresses logarithmic factors in K K K and 1 / δ 1/\delta 1/ δ .
A direct warm-start bound. The warm-start condition required in Theorem 1 can be obtained using Clip-SGD. Let x 0 x_0 x 0 satisfy ∥ x 0 − x ⋆ ∥ ≤ R 0 \lVert x_0-x^\star\rVert\le R_0 ∥ x 0 − x ⋆ ∥ ≤ R 0 , let Δ 0 : = f ( x 0 ) − f ⋆ \Delta_0:=f(x_0)-f^\star Δ 0 := f ( x 0 ) − f ⋆ , and allocate failure probability δ / 2 \delta/2 δ /2 to Phase I. Define M : = max { L 1 R 0 , log 24 δ } M:=\max\left\{L_1R_0,\log\frac{24}{\delta}\right\} M := max { L 1 R 0 , log δ 24 } . By El Bakkali El Kadi et al. [5 ] , under Assumptions 1, 2, and 3, Clip-SGD produces, after K 1 K_1 K 1 stochastic gradient evaluations, a computable output x ˉ \bar{x} x ˉ satisfying, with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 ,
f ( x ˉ ) − f ⋆ = O ( max { L 0 R 0 2 K 1 , R 0 σ ( M K 1 ) α − 1 α } ) . f(\bar{x})-f^\star=O\left(\max\left\{\frac{L_0R_0^2}{K_1},R_0\sigma\left(\frac{M}{K_1}\right)^{\frac{\alpha-1}{\alpha}}\right\}\right). f ( x ˉ ) − f ⋆ = O ( max { K 1 L 0 R 0 2 , R 0 σ ( K 1 M ) α α − 1 } ) .
An explicit sufficient iteration requirement, as stated in El Bakkali El Kadi et al. (2026), is
K 1 = Ω ( log 2 24 δ + L 1 R 0 log [ 24 e δ ( 1 + L 1 Δ 0 L 0 R 0 ) ] ) . K_1=\Omega\left(\log^2\frac{24}{\delta}+L_1R_0\log\left[\frac{24e}{\delta}\left(1+\frac{L_1\Delta_0}{L_0R_0}\right)\right]\right). K 1 = Ω ( log 2 δ 24 + L 1 R 0 log [ δ 24 e ( 1 + L 0 R 0 L 1 Δ 0 ) ] ) .
To reach the switching accuracy f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 f(\bar{x})-f^\star\le cL_0/L_1^2 f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 , the two terms in the convergence bound require, up to universal constants, K 1 = Ω ( ( L 1 R 0 ) 2 ) K_1=\Omega\left((L_1R_0)^2\right) K 1 = Ω ( ( L 1 R 0 ) 2 ) and K 1 = Ω ( M ( σ R 0 L 1 2 L 0 ) α α − 1 ) K_1=\Omega\left(M\left(\frac{\sigma R_0L_1^2}{L_0}\right)^{\frac{\alpha}{\alpha-1}}\right) K 1 = Ω ( M ( L 0 σ R 0 L 1 2 ) α − 1 α ) , respectively. Moreover, the additional iteration requirement above is already covered, up to logarithmic factors, by 1 + ( L 1 R 0 ) 2 1+(L_1R_0)^2 1 + ( L 1 R 0 ) 2 . Consequently, Phase I reaches a point satisfying f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 f(\bar{x})-f^\star\le cL_0/L_1^2 f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 after
K 1 = O ~ ( 1 + ( L 1 R 0 ) 2 + M ( σ R 0 L 1 2 L 0 ) α α − 1 ) K_1=\widetilde{O}\left(1+(L_1R_0)^2+M\left(\frac{\sigma R_0L_1^2}{L_0}\right)^{\frac{\alpha}{\alpha-1}}\right) K 1 = O ( 1 + ( L 1 R 0 ) 2 + M ( L 0 σ R 0 L 1 2 ) α − 1 α )
stochastic gradient evaluations.
Moreover, on the same event, the localization argument in the proof of El Bakkali El Kadi et al. (2026, Theorem 2) gives ∥ x k − x ⋆ ∥ ≤ 2 R 0 \lVert x_k-x^\star\rVert\le2R_0 ∥ x k − x ⋆ ∥ ≤ 2 R 0 for all Phase I iterates. Since x ˉ \bar{x} x ˉ is a convex combination of a subset of these iterates, it follows that ∥ x ˉ − x ⋆ ∥ ≤ 2 R 0 \lVert\bar{x}-x^\star\rVert\le2R_0 ∥ x ˉ − x ⋆ ∥ ≤ 2 R 0 . Hence Theorem 1 can be applied in Phase II with R = 2 R 0 R=2R_0 R = 2 R 0 and failure probability δ / 2 \delta/2 δ /2 . Therefore, to obtain f ( y K ) − f ⋆ ≤ ε f(y_K)-f^\star\le\varepsilon f ( y K ) − f ⋆ ≤ ε , Phase II requires
K = O ~ ( L 0 R 0 2 ε + ( σ R 0 ε ) α α − 1 ) . K=\widetilde{O}\left(\sqrt{\frac{L_0R_0^2}{\varepsilon}}+\left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}\right). K = O ( ε L 0 R 0 2 + ( ε σ R 0 ) α − 1 α ) .
By a union bound, the complete two-phase procedure succeeds with probability at least 1 − δ 1-\delta 1 − δ , and its total stochastic-gradient complexity is
O ~ ( 1 + ( L 1 R 0 ) 2 + M ( σ R 0 L 1 2 L 0 ) α α − 1 + L 0 R 0 2 ε + ( σ R 0 ε ) α α − 1 ) . \widetilde{O}\left(1+(L_1R_0)^2+M\left(\frac{\sigma R_0L_1^2}{L_0}\right)^{\frac{\alpha}{\alpha-1}}+\sqrt{\frac{L_0R_0^2}{\varepsilon}}+\left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}\right). O ( 1 + ( L 1 R 0 ) 2 + M ( L 0 σ R 0 L 1 2 ) α − 1 α + ε L 0 R 0 2 + ( ε σ R 0 ) α − 1 α ) .
In particular, all explicit L 1 L_1 L 1 -dependence is confined to the accuracy-independent Phase-I cost.
We can further sharpen the transient term and remove its explicit dependence on σ \sigma σ . The key observation is that ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness also implies ( L ˉ 0 , L 1 ) (\bar{L}_0,L_1) ( L ˉ 0 , L 1 ) -smoothness for any L ˉ 0 ≥ L 0 \bar{L}_0\ge L_0 L ˉ 0 ≥ L 0 . By choosing a larger L ˉ 0 \bar{L}_0 L ˉ 0 , we can switch to Phase II earlier, while the additional cost incurred by using L ˉ 0 \bar{L}_0 L ˉ 0 in the accelerated phase is absorbed into the stochastic accuracy term. This yields the following corollary.
Corollary 1. Suppose that Assumptions 1, 2, and 3 hold, with L 0 , L 1 > 0 L_0, L_1 > 0 L 0 , L 1 > 0 . Let ∥ x 0 − x ⋆ ∥ ≤ R 0 \lVert x_0 - x^\star\rVert\le R_0 ∥ x 0 − x ⋆ ∥ ≤ R 0 , where R 0 > 0 R_0 > 0 R 0 > 0 . For any ε > 0 \varepsilon> 0 ε > 0 and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , a two-phase procedure consisting of Clip-SGD, with parameters chosen as in El Bakkali El Kadi et al. (2026, Theorem 2), followed by a single run of Clip-SSTM with the calibration specified in the proof, returns a point y y y satisfying f ( y ) − f ⋆ ≤ ε f(y) - f^\star\le\varepsilon f ( y ) − f ⋆ ≤ ε with probability at least 1 − δ 1-\delta 1 − δ , using O ~ ( 1 + ( L 1 R 0 ) 2 + L 0 R 0 2 ε + ( σ R 0 ε ) α α − 1 ) \widetilde{O}\left(1 + (L_1R_0)^2 + \sqrt{\frac{L_0R_0^2}{\varepsilon}} + \left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}\right) O ( 1 + ( L 1 R 0 ) 2 + ε L 0 R 0 2 + ( ε σ R 0 ) α − 1 α ) stochastic-gradient evaluations.
Proof overview of Theorem 1.
1) A simplified accelerated potential. Let g i : = clip ( ∇ f ( x i , ξ i ) , λ i ) g_i := \operatorname{clip}(\nabla f(x_i,\xi_i),\lambda_i) g i := clip ( ∇ f ( x i , ξ i ) , λ i ) and θ i : = g i − ∇ f ( x i ) \theta_i := g_i - \nabla f(x_i) θ i := g i − ∇ f ( x i ) , so that θ i \theta_i θ i is the error induced by replacing the true gradient with the clipped stochastic gradient. For any comparator u ∈ R d u \in\mathbb{R}^d u ∈ R d , define
R t ( u ) : = ∑ i = 1 t α i ⟨ θ i , u − z i − 1 ⟩ + ∑ i = 1 t α i 2 ∥ θ i ∥ 2 . \mathcal{R}_t(u) := \sum_{i=1}^{t}\alpha_i\langle\theta_i,u-z_{i-1}\rangle+\sum_{i=1}^{t}\alpha_i^2\lVert\theta_i\rVert^2. R t ( u ) := i = 1 ∑ t α i ⟨ θ i , u − z i − 1 ⟩ + i = 1 ∑ t α i 2 ∥ θ i ∥ 2 .
Compared with the stochastic error term in the comparator inequality of Sadiev et al. [18 ] , our residual R t ( u ) \mathcal{R}_t(u) R t ( u ) has a simpler form: it contains only the clipping error θ i \theta_i θ i and no additional term involving ∇ f ( x i ) \nabla f(x_i) ∇ f ( x i ) .
More importantly, the accelerated potential argument does not require a global classical smoothness bound. It is enough that, at every iteration, the particular pair ( x i , y i ) (x_i,y_i) ( x i , y i ) satisfies
f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 . (1) f(y_i) \le f(x_i) + \langle\nabla f(x_i),y_i-x_i\rangle+ 4L_0\lVert y_i-x_i\rVert^2. \tag*{(1)} f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 . (1)
Whenever Eq. (1) holds up to time t t t , Lemma 4, together with the weight condition A i ≥ 16 L 0 α i 2 A_i \ge16L_0\alpha_i^2 A i ≥ 16 L 0 α i 2 satisfied by our standard Clip-SSTM weights, gives, for every u ∈ R d u \in\mathbb{R}^d u ∈ R d ,
A t ( f ( y t ) − f ( u ) ) + 1 2 ∥ z t − u ∥ 2 ≤ 1 2 ∥ x ˉ − u ∥ 2 + R t ( u ) . (2) A_t(f(y_t)-f(u))+\frac{1}{2}\lVert z_t-u\rVert^2\le\frac{1}{2}\lVert\bar{x}-u\rVert^2+\mathcal{R}_t(u). \tag*{(2)} A t ( f ( y t ) − f ( u )) + 2 1 ∥ z t − u ∥ 2 ≤ 2 1 ∥ x ˉ − u ∥ 2 + R t ( u ) . (2)
Thus, rather than replacing ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness by a classical smoothness constant on an entire localization ball, our proof only needs to maintain Eq. (1) along the accelerated trajectory. This naturally leads to an induction: assuming the descent inequality holds up to time t t t , we use Eq. (2) to obtain bounds on the objective gap and the auxiliary iterates, and then use these bounds to establish the descent inequality at iteration t + 1 t+1 t + 1 . The main difficulty is therefore to control the accumulated stochastic error strongly enough to close this induction.
2) Why two potentials are needed. Recall that ∥ x ˉ − x ⋆ ∥ ≤ R \lVert\bar{x}-x^\star\rVert\le R ∥ x ˉ − x ⋆ ∥ ≤ R . A natural first attempt is to apply Eq. (2) only with u = x ⋆ u=x^\star u = x ⋆ . If R t ( x ⋆ ) ≤ R 2 / 2 \mathcal{R}_t(x^\star)\le R^2/2 R t ( x ⋆ ) ≤ R 2 /2 , then A t ( f ( y t ) − f ⋆ ) + 1 2 ∥ z t − x ⋆ ∥ 2 ≤ R 2 A_t(f(y_t)-f^\star)+\frac{1}{2}\lVert z_t-x^\star\rVert^2\le R^2 A t ( f ( y t ) − f ⋆ ) + 2 1 ∥ z t − x ⋆ ∥ 2 ≤ R 2 . Hence,
f ( y t ) − f ⋆ ≤ R 2 A t and ∥ z t − x ⋆ ∥ ≤ 2 R . f(y_t)-f^\star\le\frac{R^2}{A_t}
\quad\text{and}\quad
\lVert z_t-x^\star\rVert\le\sqrt{2}\,R. f ( y t ) − f ⋆ ≤ A t R 2 and ∥ z t − x ⋆ ∥ ≤ 2 R .
Since A t A_t A t grows quadratically, the first bound already has the desired accelerated form, while the second controls the distance ∥ x ⋆ − z i − 1 ∥ \lVert x^\star-z_{i-1}\rVert ∥ x ⋆ − z i − 1 ∥ appearing in the stochastic error R t ( x ⋆ ) \mathcal{R}_t(x^\star) R t ( x ⋆ ) .
However, these bounds alone are not sufficient to close the induction described above. To recover Eq. (1) at iteration t + 1 t+1 t + 1 , we first need to control ∥ ∇ f ( y t ) ∥ \lVert\nabla f(y_t)\rVert ∥ ∇ f ( y t )∥ and then transfer this control from y t y_t y t to the next query point x t + 1 x_{t+1} x t + 1 . In our argument, this gradient control is obtained by keeping the objective gap f ( y t ) − f ⋆ f(y_t)-f^\star f ( y t ) − f ⋆ at the scale L 0 / L 1 2 L_0/L_1^2 L 0 / L 1 2 . When A t A_t A t is small, however, the bound R 2 / A t R^2/A_t R 2 / A t can be much larger than this scale.
This issue is already visible at initialization. Phase I guarantees f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 f(\bar{x})-f^\star\le cL_0/L_1^2 f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 , but since A 0 = 0 A_0=0 A 0 = 0 , the x ⋆ x^\star x ⋆ -potential starts from 1 2 ∥ x ˉ − x ⋆ ∥ 2 \frac{1}{2}\lVert\bar{x}-x^\star\rVert^2 2 1 ∥ x ˉ − x ⋆ ∥ 2 and does not use this small objective gap.
To address this difficulty, we introduce a second potential centered at the switching point x ˉ \bar{x} x ˉ . Applying Eq. (2) with u = x ˉ u=\bar{x} u = x ˉ , and using z 0 = x ˉ z_0=\bar{x} z 0 = x ˉ , gives
A t ( f ( y t ) − f ( x ˉ ) ) + 1 2 ∥ z t − x ˉ ∥ 2 ≤ R t ( x ˉ ) . A_t(f(y_t)-f(\bar{x}))+\frac{1}{2}\lVert z_t-\bar{x}\rVert^2\le\mathcal{R}_t(\bar{x}). A t ( f ( y t ) − f ( x ˉ )) + 2 1 ∥ z t − x ˉ ∥ 2 ≤ R t ( x ˉ ) .
Therefore, if R t ( x ˉ ) ≤ c A t L 0 / L 1 2 \mathcal{R}_{t}(\bar{x}) \le cA_{t}L_{0}/L_{1}^{2} R t ( x ˉ ) ≤ c A t L 0 / L 1 2 , then
{ f ( y t ) − f ⋆ ≤ f ( x ˉ ) − f ⋆ + R t ( x ˉ ) A t ≤ 2 c L 0 L 1 2 , ∥ z t − x ˉ ∥ ≤ 2 c A t L 0 L 1 . \begin{cases}
f(y_{t})-f^{\star} \le f(\bar{x})-f^{\star}+\dfrac{\mathcal{R}_{t}(\bar{x})}{A_{t}} \le\dfrac{2cL_{0}}{L_{1}^{2}},\\
\lVert z_{t}-\bar{x}\rVert\le\dfrac{2\sqrt{cA_{t}L_{0}}}{L_{1}}.
\end{cases} ⎩ ⎨ ⎧ f ( y t ) − f ⋆ ≤ f ( x ˉ ) − f ⋆ + A t R t ( x ˉ ) ≤ L 1 2 2 c L 0 , ∥ z t − x ˉ ∥ ≤ L 1 2 c A t L 0 .
The first bound keeps y t y_{t} y t in the small-gap regime and therefore controls ∥ ∇ f ( y t ) ∥ \lVert\nabla f(y_{t})\rVert ∥ ∇ f ( y t )∥ at the scale L 0 / L 1 L_{0}/L_{1} L 0 / L 1 . Applying the second bound up to time t t t controls the auxiliary iterates relative to x ˉ \bar{x} x ˉ . Since y t = A t − 1 ∑ j = 1 t α j z j y_{t}=A_{t}^{-1}\sum_{j=1}^{t}\alpha_{j}z_{j} y t = A t − 1 ∑ j = 1 t α j z j , this gives ∥ z t − y t ∥ ≤ 4 c A t L 0 / L 1 \lVert z_{t}-y_{t}\rVert\le4\sqrt{cA_{t}L_{0}}/L_{1} ∥ z t − y t ∥ ≤ 4 c A t L 0 / L 1 . Moreover, x t + 1 − y t = α t + 1 A t + 1 ( z t − y t ) x_{t+1}-y_{t}=\frac{\alpha_{t+1}}{A_{t+1}}(z_{t}-y_{t}) x t + 1 − y t = A t + 1 α t + 1 ( z t − y t ) , so the weight conditions ensure that x t + 1 x_{t+1} x t + 1 remains sufficiently close to y t y_{t} y t . This allows the gradient control to be transferred from y t y_{t} y t to x t + 1 x_{t+1} x t + 1 ; in particular, ∥ ∇ f ( x t + 1 ) ∥ ≤ L 0 / L 1 \lVert\nabla f(x_{t+1})\rVert\le L_{0}/L_{1} ∥ ∇ f ( x t + 1 )∥ ≤ L 0 / L 1 .
Since the clipped update also keeps y t + 1 y_{t+1} y t + 1 sufficiently close to x t + 1 x_{t+1} x t + 1 , ( L 0 , L 1 ) (L_{0},L_{1}) ( L 0 , L 1 ) -smoothness then gives Eq. (1) at iteration t + 1 t+1 t + 1 . This closes the deterministic part of the induction.
The two potentials therefore play complementary roles. The x ˉ \bar{x} x ˉ -potential propagates the small-gap condition obtained in Phase I, while the x ⋆ x^{\star} x ⋆ -potential provides the accelerated bound and controls the distance to the minimizer. Together, they give
f ( y t ) − f ⋆ ≤ min { 2 c L 0 L 1 2 , R 2 A t } , f(y_{t})-f^{\star} \le\min\left\{\dfrac{2cL_{0}}{L_{1}^{2}},\dfrac{R^{2}}{A_{t}}\right\}, f ( y t ) − f ⋆ ≤ min { L 1 2 2 c L 0 , A t R 2 } ,
as well as bounds on both ∥ z t − x ˉ ∥ \lVert z_{t}-\bar{x}\rVert ∥ z t − x ˉ ∥ and ∥ z t − x ⋆ ∥ \lVert z_{t}-x^{\star}\rVert ∥ z t − x ⋆ ∥ .
Combining these bounds with ( L 0 , L 1 ) (L_{0},L_{1}) ( L 0 , L 1 ) -smoothness and the weight conditions gives two complementary bounds on the gradient at x t + 1 x_{t+1} x t + 1 , where the next stochastic gradient is evaluated. The x ˉ \bar{x} x ˉ -potential gives a bound of order c L 0 / L 1 \sqrt{c}L_{0}/L_{1} c L 0 / L 1 , while the x ⋆ x^{\star} x ⋆ -potential gives a bound that decays as A t + 1 − 1 / 2 A_{t+1}^{-1/2} A t + 1 − 1/2 . More precisely, up to numerical constants,
{ ∥ ∇ f ( x t + 1 ) ∥ ≲ c L 0 L 1 , ∥ ∇ f ( x t + 1 ) ∥ ≲ R L 0 A t + 1 . \begin{cases}
\lVert\nabla f(x_{t+1})\rVert\lesssim\dfrac{\sqrt{c}L_{0}}{L_{1}},\\
\lVert\nabla f(x_{t+1})\rVert\lesssim R\sqrt{\dfrac{L_{0}}{A_{t+1}}}.
\end{cases} ⎩ ⎨ ⎧ ∥ ∇ f ( x t + 1 )∥ ≲ L 1 c L 0 , ∥ ∇ f ( x t + 1 )∥ ≲ R A t + 1 L 0 .
Since both bounds hold simultaneously, we can use the smaller one. Together with A t + 1 ≳ ℓ 2 L 0 α t + 1 2 A_{t+1}\gtrsim\ell^{2}L_{0}\alpha_{t+1}^{2} A t + 1 ≳ ℓ 2 L 0 α t + 1 2 , this suggests the clipping scale 1 α t + 1 ℓ min { c A t + 1 L 0 L 1 , R } \frac{1}{\alpha_{t+1}\ell}\min\left\{\frac{\sqrt{cA_{t+1}L_{0}}}{L_{1}},R\right\} α t + 1 ℓ 1 min { L 1 c A t + 1 L 0 , R } . Accordingly, for every i = 1 , … , K i=1,\ldots,K i = 1 , … , K , we choose
λ i : = 1 α i ℓ min { c A i L 0 L 1 , R } . \lambda_{i}:=\frac{1}{\alpha_{i}\ell}\min\left\{\frac{\sqrt{cA_{i}L_{0}}}{L_{1}},R\right\}. λ i := α i ℓ 1 min { L 1 c A i L 0 , R } .
With the exact constants in Lemma 6, whenever the induction conditions hold up to time t t t , this choice gives
∥ ∇ f ( x t + 1 ) ∥ ≤ λ t + 1 2 . \lVert\nabla f(x_{t+1})\rVert\le\frac{\lambda_{t+1}}{2}. ∥ ∇ f ( x t + 1 )∥ ≤ 2 λ t + 1 .
Thus, before evaluating the stochastic gradient at x t + 1 x_{t+1} x t + 1 , the true gradient is guaranteed to lie below half the clipping threshold. This is precisely the condition needed for the clipping-bias and second-moment bounds used in the stochastic analysis.
3) Controlling the accumulated stochastic error. It remains to show that the residual bounds used above hold simultaneously throughout the run. The difficulty is that the clipping estimates require ∥ ∇ f ( x i ) ∥ ≤ λ i / 2 \lVert\nabla f(x_{i})\rVert\le\lambda_{i}/2 ∥ ∇ f ( x i )∥ ≤ λ i /2 , while this gradient bound is itself obtained only as long as the preceding descent and residual bounds remain valid. We handle this dependence with a stopped process.
For t = 0 , … , K t=0,\ldots,K t = 0 , … , K , let E t \mathcal{E}_{t} E t be the event that, for every i ≤ t i\le t i ≤ t , the descent inequality Eq. (1) holds and
R i ( x ˉ ) ≤ c A i L 0 L 1 2 and R i ( x ⋆ ) ≤ R 2 2 . \mathcal{R}_{i}(\bar{x})\le\frac{cA_{i}L_{0}}{L_{1}^{2}}
\qquad\text{and}\qquad
\mathcal{R}_{i}(x^{\star})\le\frac{R^{2}}{2}. R i ( x ˉ ) ≤ L 1 2 c A i L 0 and R i ( x ⋆ ) ≤ 2 R 2 .
For t = 0 t=0 t = 0 there are no conditions, so 1 E 0 = 1 \mathbf{1}_{\mathcal{E}_0}=1 1 E 0 = 1 . For u ∈ { x ˉ , x ⋆ } u\in\{\bar{x},x^\star\} u ∈ { x ˉ , x ⋆ } , define the stopped residual
R ^ t ( u ) : = ∑ i = 1 t 1 E i − 1 ( α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 ) . \widehat{R}_t(u):=\sum_{i=1}^{t}\mathbf{1}_{\mathcal{E}_{i-1}}\left(\alpha_i\langle\theta_i,u-z_{i-1}\rangle+\alpha_i^2\|\theta_i\|^2\right). R t ( u ) := i = 1 ∑ t 1 E i − 1 ( α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 ) .
The indicator ensures that each summand is analyzed only when the previous induction conditions hold. In particular, on E i − 1 \mathcal{E}_{i-1} E i − 1 , Lemma 6 gives ∥ ∇ f ( x i ) ∥ ≤ λ i / 2 \|\nabla f(x_i)\|\leq\lambda_i/2 ∥∇ f ( x i ) ∥ ≤ λ i /2 , and hence the following clipping estimates hold:
{ ∥ E [ θ i ∣ F i − 1 ] ∥ ≤ 2 α α σ α λ i α − 1 , E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 2 σ α λ i 2 − α . \begin{cases}
\|\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}]\|\leq\dfrac{2\alpha^\alpha\sigma^\alpha}{\lambda_i^{\alpha-1}},\\
\mathbb{E}[\|\theta_i\|^2\mid\mathcal{F}_{i-1}]\leq2\sigma^\alpha\lambda_i^{2-\alpha}.
\end{cases} ⎩ ⎨ ⎧ ∥ E [ θ i ∣ F i − 1 ] ∥ ≤ λ i α − 1 2 α α σ α , E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 2 σ α λ i 2 − α .
We then subtract the conditional means and write R ^ t ( u ) = M t ( u ) + B t ( u ) \widehat{R}_t(u)=M_t(u)+B_t(u) R t ( u ) = M t ( u ) + B t ( u ) , where M t ( u ) M_t(u) M t ( u ) is a martingale and B t ( u ) B_t(u) B t ( u ) collects the clipping bias and conditional second-moment terms. The localization bounds obtained from the two potentials control the comparator distances appearing in both terms. In particular,
{ ∥ x ˉ − z i − 1 ∥ ≲ s i , ∥ x ⋆ − z i − 1 ∥ ≲ R , s i : = min { c A i L 0 L 1 , R } . \begin{cases}
\|\bar{x}-z_{i-1}\|\lesssim s_i,\\
\|x^\star-z_{i-1}\|\lesssim R,
\end{cases}
\qquad
s_i:=\min\left\{\dfrac{\sqrt{cA_i}L_0}{L_1},R\right\}. { ∥ x ˉ − z i − 1 ∥ ≲ s i , ∥ x ⋆ − z i − 1 ∥ ≲ R , s i := min { L 1 c A i L 0 , R } .
Since α i λ i = s i / ℓ \alpha_i\lambda_i=s_i/\ell α i λ i = s i / ℓ , both the predictable terms and the conditional variances can be controlled by the same quantity S t : = σ α ∑ i = 1 t α i s i λ i α − 1 S_t:=\sigma^\alpha\sum_{i=1}^{t}\dfrac{\alpha_i}{s_i\lambda_i^{\alpha-1}} S t := σ α ∑ i = 1 t s i λ i α − 1 α i .
Thus, if S K ≤ 1 / 512 S_K\leq1/512 S K ≤ 1/512 , the predictable part is sufficiently small, while the martingale has controlled conditional variance and bounded increments. Applying Freedman’s inequality and taking a union bound over t = 1 , … , K t=1,\ldots,K t = 1 , … , K and u ∈ { x ˉ , x ⋆ } u\in\{\bar{x},x^\star\} u ∈ { x ˉ , x ⋆ } yields, with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every t t t ,
{ R ^ t ( x ˉ ) ≤ c A t L 0 L 1 2 , R ^ t ( x ⋆ ) ≤ R 2 2 . \begin{cases}
\widehat{R}_t(\bar{x})\leq\dfrac{cA_tL_0}{L_1^2},\\
\widehat{R}_t(x^\star)\leq\dfrac{R^2}{2}.
\end{cases} ⎩ ⎨ ⎧ R t ( x ˉ ) ≤ L 1 2 c A t L 0 , R t ( x ⋆ ) ≤ 2 R 2 .
It remains to transfer these bounds from the stopped process to the original residuals. Suppose that E t − 1 \mathcal{E}_{t-1} E t − 1 holds. Then R ^ t ( u ) = R t ( u ) \widehat{R}_t(u)=R_t(u) R t ( u ) = R t ( u ) for u ∈ { x ˉ , x ⋆ } u\in\{\bar{x},x^\star\} u ∈ { x ˉ , x ⋆ } . Moreover, the induction hypotheses together with the weight and clipping-level conditions satisfy the assumptions of Lemma 5 with index t − 1 t-1 t − 1 . Hence, Eq. (1) holds at iteration t t t . The two residual bounds above therefore imply that E t \mathcal{E}_t E t holds. Starting from 1 E 0 = 1 \mathbf{1}_{\mathcal{E}_0}=1 1 E 0 = 1 and proceeding by induction, we obtain E K \mathcal{E}_K E K with probability at least 1 − δ 1-\delta 1 − δ . Therefore, all the descent and localization bounds established above hold simultaneously throughout the run.
4) Verifying the parameter choice and obtaining the rate. It remains to check that the parameters in Theorem 1 satisfy S K ≤ 1 / 512 S_K\leq1/512 S K ≤ 1/512 . Using λ i = s i / ( α i ℓ ) \lambda_i=s_i/(\alpha_i\ell) λ i = s i / ( α i ℓ ) and the explicit accelerated weights, S K S_K S K splits into two contributions, corresponding to the two terms in the definition of s i s_i s i . By the choice of a a a , each contribution is at most 1 / 1024 1/1024 1/1024 . Hence S K ≤ 1 / 512 S_K\leq1/512 S K ≤ 1/512 , and therefore E K \mathcal{E}_K E K holds with probability at least 1 − δ 1-\delta 1 − δ .
On this event, the two potentials give
f ( y K ) − f ⋆ ≤ min { 2 c L 0 L 1 2 , R 2 A K } . f(y_K)-f^\star\leq\min\left\{\dfrac{2cL_0}{L_1^2},\dfrac{R^2}{A_K}\right\}. f ( y K ) − f ⋆ ≤ min { L 1 2 2 c L 0 , A K R 2 } .
Since A K = K ( K + 3 ) / ( 2 a L 0 ) A_K=K(K+3)/(2aL_0) A K = K ( K + 3 ) / ( 2 a L 0 ) , substituting the definition of a a a gives, up to numerical constants,
R 2 A K ≲ L 0 R 2 ℓ 2 K 2 + σ 2 L 1 2 R 2 c L 0 ℓ 2 ( α − 1 ) / α K 2 ( α − 1 ) / α + σ R ℓ ( α − 1 ) / α K ( α − 1 ) / α . \dfrac{R^2}{A_K}\lesssim\dfrac{L_0R^2\ell^2}{K^2}+\dfrac{\sigma^2L_1^2R^2}{cL_0}\dfrac{\ell^{2(\alpha-1)/\alpha}}{K^{2(\alpha-1)/\alpha}}+\dfrac{\sigma R\ell^{(\alpha-1)/\alpha}}{K^{(\alpha-1)/\alpha}}. A K R 2 ≲ K 2 L 0 R 2 ℓ 2 + c L 0 σ 2 L 1 2 R 2 K 2 ( α − 1 ) / α ℓ 2 ( α − 1 ) / α + K ( α − 1 ) / α σ R ℓ ( α − 1 ) / α .
The middle term contains L 1 L_1 L 1 , but combining it with the low-gap bound 2 c L 0 / L 1 2 2cL_0/L_1^2 2 c L 0 / L 1 2 and using min { x , y } ≤ x y \min\{x,y\}\le\sqrt{xy} min { x , y } ≤ x y gives another term of order σ R ℓ ( α − 1 ) / α K ( α − 1 ) / α \frac{\sigma R\ell^{(\alpha-1)/\alpha}}{K^{(\alpha-1)/\alpha}} K ( α − 1 ) / α σ R ℓ ( α − 1 ) / α . Therefore,
f ( y K ) − f ⋆ = O ~ ( L 0 R 2 K 2 + σ R K ( α − 1 ) / α ) . f(y_K)-f^\star=\widetilde{O}\left(\frac{L_0R^2}{K^2}+\frac{\sigma R}{K^{(\alpha-1)/\alpha}}\right). f ( y K ) − f ⋆ = O ( K 2 L 0 R 2 + K ( α − 1 ) / α σ R ) .
Thus, after the warm-start condition f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 f(\bar{x})-f^\star\le cL_0/L_1^2 f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 has been reached, the accelerated Phase II rate no longer depends on L 1 L_1 L 1 . The L 1 L_1 L 1 -dependence of the two-phase method comes only from the Phase I cost required to reach this warm-start regime.
One-phase result
We next analyze the one-phase procedure introduced in § 2 . In contrast to the two-phase procedure, the method starts directly from x 0 x_0 x 0 and does not require a small initial objective gap. The modified weight schedule first controls the high-curvature transient and then recovers the usual accelerated regime.
Theorem 2. Suppose that Assumptions 1, 2, and 3 hold, where L 0 , L 1 > 0 L_0,L_1>0 L 0 , L 1 > 0 . Assume that ∥ x 0 − x ⋆ ∥ ≤ R 0 \lVert x_0-x^\star\rVert\le R_0 ∥ x 0 − x ⋆ ∥ ≤ R 0 , where R 0 > 0 R_0>0 R 0 > 0 , and let Δ ˉ 0 ≥ f ( x 0 ) − f ⋆ \bar{\Delta}_0\ge f(x_0)-f^\star Δ ˉ 0 ≥ f ( x 0 ) − f ⋆ be a deterministic upper bound.
Let K ≥ 1 K\ge1 K ≥ 1 , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and set ℓ : = 8 log 4 K δ \ell:=8\log\frac{4K}{\delta} ℓ := 8 log δ 4 K , and a : = max { 128 ℓ 2 , 512 1 / α σ L 0 R 0 ℓ ( α − 1 ) / α K ( α + 1 ) / α } a:=\max\left\{128\ell^2,512^{1/\alpha}\frac{\sigma}{L_0R_0}\ell^{(\alpha-1)/\alpha}K^{(\alpha+1)/\alpha}\right\} a := max { 128 ℓ 2 , 51 2 1/ α L 0 R 0 σ ℓ ( α − 1 ) / α K ( α + 1 ) / α } . Choose A 0 > 0 A_0>0 A 0 > 0 satisfying A 0 ≤ 1 a L 0 A_0\le\frac{1}{aL_0} A 0 ≤ a L 0 1 and A 0 Δ ˉ 0 ≤ R 0 2 64 A_0\bar{\Delta}_0\le\frac{R_0^2}{64} A 0 Δ ˉ 0 ≤ 64 R 0 2 . For every k ≥ 0 k\ge0 k ≥ 0 , let
α k + 1 : = min { A k 64 ℓ ( 1 + L 1 R 0 ) , A k a L 0 } . \alpha_{k+1}:=\min\left\{\frac{A_k}{64\ell(1+L_1R_0)},\sqrt{\frac{A_k}{aL_0}}\right\}. α k + 1 := min { 64 ℓ ( 1 + L 1 R 0 ) A k , a L 0 A k } . and, for i = 1 , … , K i=1,\ldots,K i = 1 , … , K , let λ i : = R 0 α i ℓ \lambda_i:=\frac{R_0}{\alpha_i\ell} λ i := α i ℓ R 0 . Define K 0 : = ⌈ 128 ℓ ( 1 + L 1 R 0 ) log ( [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 A 0 ) ⌉ K_0:=\left\lceil128\ell(1+L_1R_0)\log\left(\frac{[64\ell(1+L_1R_0)]^2}{aL_0A_0}\right)\right\rceil K 0 := ⌈ 128 ℓ ( 1 + L 1 R 0 ) log ( a L 0 A 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 ) ⌉ . Then, if K ≥ 2 K 0 K\ge2K_0 K ≥ 2 K 0 , with probability at least 1 − δ 1-\delta 1 − δ , the output y K y_K y K of Clip-SSTM satisfies
f ( y K ) − f ⋆ = O ~ ( L 0 R 0 2 K 2 + σ R 0 K ( α − 1 ) / α ) , f(y_K)-f^\star=\widetilde{O}\left(\frac{L_0R_0^2}{K^2}+\frac{\sigma R_0}{K^{(\alpha-1)/\alpha}}\right), f ( y K ) − f ⋆ = O ( K 2 L 0 R 0 2 + K ( α − 1 ) / α σ R 0 ) , where O ~ \widetilde{O} O suppresses logarithmic factors in K K K and 1 / δ 1/\delta 1/ δ . With the initialization below, the dependence on Δ ˉ 0 \bar{\Delta}_0 Δ ˉ 0 and L 1 L_1 L 1 enters only through K 0 K_0 K 0 , which is accuracy-independent up to logarithmic factors.
Remark 1. The initialization parameter A 0 A_0 A 0 can be chosen explicitly as A 0 : = min { 1 a L 0 , R 0 2 64 Δ ˉ 0 } A_0:=\min\left\{\frac{1}{aL_0},\frac{R_0^2}{64\bar{\Delta}_0}\right\} A 0 := min { a L 0 1 , 64 Δ ˉ 0 R 0 2 } , with the convention that R 0 2 / ( 64 Δ ˉ 0 ) = + ∞ R_0^2/(64\bar{\Delta}_0)=+\infty R 0 2 / ( 64 Δ ˉ 0 ) = + ∞ when Δ ˉ 0 = 0 \bar{\Delta}_0=0 Δ ˉ 0 = 0 . This choice guarantees A 0 ≤ 1 / ( a L 0 ) A_0\le1/(aL_0) A 0 ≤ 1/ ( a L 0 ) and A 0 Δ ˉ 0 ≤ R 0 2 / 64 A_0\bar{\Delta}_0\le R_0^2/64 A 0 Δ ˉ 0 ≤ R 0 2 /64 . Since 1 a L 0 A 0 = max { 1 , 64 Δ ˉ 0 a L 0 R 0 2 } \frac{1}{aL_0A_0}=\max\left\{1,\frac{64\bar{\Delta}_0}{aL_0R_0^2}\right\} a L 0 A 0 1 = max { 1 , a L 0 R 0 2 64 Δ ˉ 0 } , the transient becomes
K 0 = ⌈ 128 ℓ ( 1 + L 1 R 0 ) log ( [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 max { 1 , 64 Δ ˉ 0 a L 0 R 0 2 } ) ⌉ . K_0=\left\lceil128\ell(1+L_1R_0)\log\left([64\ell(1+L_1R_0)]^2\max\left\{1,\frac{64\bar{\Delta}_0}{aL_0R_0^2}\right\}\right)\right\rceil. K 0 = ⌈ 128 ℓ ( 1 + L 1 R 0 ) log ( [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 max { 1 , a L 0 R 0 2 64 Δ ˉ 0 } ) ⌉ .
Consequently, K 0 = O ~ ( 1 + L 1 R 0 ) K_0=\widetilde{O}(1+L_1R_0) K 0 = O ( 1 + L 1 R 0 ) , where the notation suppresses logarithmic factors in K K K , 1 / δ 1/\delta 1/ δ , 1 + L 1 R 0 1+L_1R_0 1 + L 1 R 0 , and 1 + Δ ˉ 0 / ( L 0 R 0 2 ) 1+\bar{\Delta}_0/(L_0R_0^2) 1 + Δ ˉ 0 / ( L 0 R 0 2 ) . The transient is therefore accuracy-independent up to logarithmic factors: its dependence on the target accuracy through the horizon K K K is at most logarithmic.
Remark 2. In the worst case, the logarithmic term involving Δ ˉ 0 \bar{\Delta}_0 Δ ˉ 0 can add a factor of order L 1 R 0 L_1R_0 L 1 R 0 to the transient. Indeed, integrating the gradient bound in Lemma 3 along the segment from x ⋆ x^\star x ⋆ to x 0 x_0 x 0 gives
Δ 0 ≤ L 0 L 1 2 ( e L 1 R 0 − 1 − L 1 R 0 ) . \Delta_0\le\frac{L_0}{L_1^2}\left(e^{L_1R_0}-1-L_1R_0\right). Δ 0 ≤ L 1 2 L 0 ( e L 1 R 0 − 1 − L 1 R 0 ) .
We can use this upper bound as Δ ˉ 0 \bar{\Delta}_0 Δ ˉ 0 . It depends only on L 0 L_0 L 0 , L 1 L_1 L 1 , and R 0 R_0 R 0 , so we do not need to know f ⋆ f^\star f ⋆ . With this choice, the initialization above gives K 0 = O ~ ( ( 1 + L 1 R 0 ) 2 ) K_0=\widetilde{O}((1+L_1R_0)^2) K 0 = O (( 1 + L 1 R 0 ) 2 ) , where the notation hides logarithmic factors in K K K , 1 / δ 1/\delta 1/ δ , and 1 + L 1 R 0 1+L_1R_0 1 + L 1 R 0 . If a tighter bound on the initial gap is available, the transient can be smaller.
Proof overview of Theorem 2. We prove by induction that the local descent inequality and a bound on the accumulated error hold throughout the run. Let Δ t : = f ( y t ) − f ⋆ \Delta_t := f(y_t) - f^\star Δ t := f ( y t ) − f ⋆ , θ i : = g i − ∇ f ( x i ) \theta_i := g_i - \nabla f(x_i) θ i := g i − ∇ f ( x i ) , and R t : = ∑ i = 1 t α i ⟨ θ i , x ⋆ − z i − 1 ⟩ + ∑ i = 1 t α i 2 ∥ θ i ∥ 2 R_t := \sum_{i=1}^t \alpha_i \langle\theta_i, x^\star- z_{i-1}\rangle+ \sum_{i=1}^t \alpha_i^2 \lVert\theta_i \rVert^2 R t := ∑ i = 1 t α i ⟨ θ i , x ⋆ − z i − 1 ⟩ + ∑ i = 1 t α i 2 ∥ θ i ∥ 2 . For t = 0 , … , K t = 0, \ldots, K t = 0 , … , K , define
E t : = ⋂ i = 1 t { f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + A i 4 α i 2 ∥ y i − x i ∥ 2 , R i ≤ R 0 2 2 } . \mathcal{E}_t := \bigcap_{i=1}^t \left\{ \begin{aligned} f(y_i) &\le f(x_i) + \langle\nabla f(x_i), y_i - x_i\rangle+ \frac{A_i}{4\alpha_i^2}\lVert y_i - x_i\rVert^2, \\ R_i &\le\frac{R_0^2}{2} \end{aligned} \right\}. E t := i = 1 ⋂ t ⎩ ⎨ ⎧ f ( y i ) R i ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 α i 2 A i ∥ y i − x i ∥ 2 , ≤ 2 R 0 2 ⎭ ⎬ ⎫ .
with E 0 \mathcal{E}_0 E 0 the whole sample space. Our goal is to prove P ( E K ) ≥ 1 − δ \mathbb{P}(\mathcal{E}_K) \ge1 - \delta P ( E K ) ≥ 1 − δ .
Fix 0 ≤ t < K 0 \le t < K 0 ≤ t < K and suppose E t \mathcal{E}_t E t holds. By Lemma 8 and the initialization A 0 Δ 0 ≤ R 0 2 / 64 A_0\Delta_0 \le R_0^2/64 A 0 Δ 0 ≤ R 0 2 /64 , for every i = 1 , … , t i = 1, \ldots, t i = 1 , … , t ,
A i Δ i + 1 2 ∥ z i − x ⋆ ∥ 2 ≤ A 0 Δ 0 + 1 2 ∥ x 0 − x ⋆ ∥ 2 + R i ≤ 65 64 R 0 2 . A_i\Delta_i + \frac{1}{2}\lVert z_i - x^\star\rVert^2 \le A_0\Delta_0 + \frac{1}{2}\lVert x_0 - x^\star\rVert^2 + R_i \le\frac{65}{64}R_0^2. A i Δ i + 2 1 ∥ z i − x ⋆ ∥ 2 ≤ A 0 Δ 0 + 2 1 ∥ x 0 − x ⋆ ∥ 2 + R i ≤ 64 65 R 0 2 .
Thus Δ i ≤ 2 R 0 2 / A i \Delta_i \le2R_0^2/A_i Δ i ≤ 2 R 0 2 / A i and ∥ z i − x ⋆ ∥ ≤ 3 R 0 / 2 \lVert z_i - x^\star\rVert\le3R_0/2 ∥ z i − x ⋆ ∥ ≤ 3 R 0 /2 . These bounds also hold at i = 0 i = 0 i = 0 by initialization. Since y t y_t y t is a convex combination of x 0 , z 1 , … , z t x_0, z_1, \ldots, z_t x 0 , z 1 , … , z t , we obtain ∥ z t − y t ∥ ≤ 3 R 0 \lVert z_t - y_t\rVert\le3R_0 ∥ z t − y t ∥ ≤ 3 R 0 .
We next establish the descent inequality at iteration t + 1 t + 1 t + 1 . The gradient–gap bound in Lemma 2 controls ∥ ∇ f ( y t ) ∥ \lVert\nabla f(y_t)\rVert ∥ ∇ f ( y t )∥ using Δ t ≤ 2 R 0 2 / A t \Delta_t \le2R_0^2/A_t Δ t ≤ 2 R 0 2 / A t . The identity x t + 1 − y t = ( α t + 1 / A t + 1 ) ( z t − y t ) x_{t+1} - y_t = (\alpha_{t+1}/A_{t+1})(z_t - y_t) x t + 1 − y t = ( α t + 1 / A t + 1 ) ( z t − y t ) and the weight restriction α t + 1 / A t ≤ [ 64 ℓ ( 1 + L 1 R 0 ) ] − 1 \alpha_{t+1}/A_t \le[64\ell(1+L_1R_0)]^{-1} α t + 1 / A t ≤ [ 64 ℓ ( 1 + L 1 R 0 ) ] − 1 keep x t + 1 x_{t+1} x t + 1 close to y t y_t y t . Hence Lemma 3, together with A t ≥ a L 0 α t + 1 2 A_t \ge aL_0\alpha_{t+1}^2 A t ≥ a L 0 α t + 1 2 , gives L 0 + L 1 ∥ ∇ f ( x t + 1 ) ∥ ≤ A t + 1 / ( 8 α t + 1 2 ) L_0 + L_1\lVert\nabla f(x_{t+1})\rVert\le A_{t+1}/(8\alpha_{t+1}^2) L 0 + L 1 ∥ ∇ f ( x t + 1 )∥ ≤ A t + 1 / ( 8 α t + 1 2 ) . Clipping also bounds the distance from x t + 1 x_{t+1} x t + 1 to y t + 1 y_{t+1} y t + 1 . These two bounds give the next descent inequality by Lemma 9 . Under a ≥ 128 ℓ 2 a \ge128\ell^2 a ≥ 128 ℓ 2 , Lemma 10 further gives ∥ ∇ f ( x t + 1 ) ∥ ≤ λ t + 1 / 2 \lVert\nabla f(x_{t+1})\rVert\le\lambda_{t+1}/2 ∥ ∇ f ( x t + 1 )∥ ≤ λ t + 1 /2 , where λ i = R 0 / ( α i ℓ ) \lambda_i = R_0/(\alpha_i\ell) λ i = R 0 / ( α i ℓ ) .
To control the accumulated error, define R ^ t : = ∑ i = 1 t 1 E i − 1 ( α i ⟨ θ i , x ⋆ − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 ) \widehat{R}_t := \sum_{i=1}^t \mathbf{1}_{\mathcal{E}_{i-1}}\left(\alpha_i\langle\theta_i, x^\star- z_{i-1}\rangle+ \alpha_i^2\lVert\theta_i\rVert^2\right) R t := ∑ i = 1 t 1 E i − 1 ( α i ⟨ θ i , x ⋆ − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 ) . Each term is included only when the induction conditions hold through iteration i − 1 i - 1 i − 1 . On E i − 1 \mathcal{E}_{i-1} E i − 1 , the preceding argument gives ∥ ∇ f ( x i ) ∥ ≤ λ i / 2 \lVert\nabla f(x_i)\rVert\le\lambda_i/2 ∥ ∇ f ( x i )∥ ≤ λ i /2 , so the conditional clipping-bias and second-moment bounds apply. Using Freedman’s inequality (Proposition 2), Lemma 11 gives an event G \mathcal{G} G with P ( G ) ≥ 1 − δ \mathbb{P}(\mathcal{G}) \ge1 - \delta P ( G ) ≥ 1 − δ on which R ^ t ≤ R 0 2 / 2 \widehat{R}_t \le R_0^2/2 R t ≤ R 0 2 /2 for every t ≤ K t \le K t ≤ K , provided σ α ℓ α − 1 R 0 − α ∑ i = 1 K α i α ≤ 1 / 512 \sigma^\alpha\ell^{\alpha-1}R_0^{-\alpha}\sum_{i=1}^K\alpha_i^\alpha\le1/512 σ α ℓ α − 1 R 0 − α ∑ i = 1 K α i α ≤ 1/512 . By Lemma 12 , ∑ i = 1 K α i α ≤ K α + 1 / ( a L 0 ) α \sum_{i=1}^K\alpha_i^\alpha\le K^{\alpha+1}/(aL_0)^\alpha ∑ i = 1 K α i α ≤ K α + 1 / ( a L 0 ) α , and the chosen value of a a a ensures this condition.
We now complete the induction on G \mathcal{G} G . The event E 0 \mathcal{E}_0 E 0 holds by definition. Suppose E t \mathcal{E}_t E t holds for some 0 ≤ t < K 0 \le t < K 0 ≤ t < K . Since the events E i \mathcal{E}_i E i are nested, every indicator in R ^ t + 1 \widehat{R}_{t+1} R t + 1 equals one. Therefore, on G \mathcal{G} G , R t + 1 = R ^ t + 1 ≤ R 0 2 / 2 R_{t+1} = \widehat{R}_{t+1} \le R_0^2/2 R t + 1 = R t + 1 ≤ R 0 2 /2 . The preceding deterministic argument also gives the descent inequality at iteration t + 1 t + 1 t + 1 , so E t + 1 \mathcal{E}_{t+1} E t + 1 holds. Thus G ⊆ E K \mathcal{G} \subseteq\mathcal{E}_K G ⊆ E K , proving P ( E K ) ≥ 1 − δ \mathbb{P}(\mathcal{E}_K) \ge1 - \delta P ( E K ) ≥ 1 − δ .
Finally, set b : = 64 ℓ ( 1 + L 1 R 0 ) b := 64\ell(1 + L_1R_0) b := 64 ℓ ( 1 + L 1 R 0 ) . While A t < b 2 / ( a L 0 ) A_t < b^2/(aL_0) A t < b 2 / ( a L 0 ) , the weights give A t + 1 = ( 1 + 1 / b ) A t A_{t+1} = (1 + 1/b)A_t A t + 1 = ( 1 + 1/ b ) A t . After this threshold is reached, the square-root branch remains active and A t A_t A t grows quadratically. By Lemma 12 , A K ≥ 4 ( K − K 0 ) 2 / ( 25 a L 0 ) A_K \ge4(K - K_0)^2/(25aL_0) A K ≥ 4 ( K − K 0 ) 2 / ( 25 a L 0 ) for K ≥ K 0 K \ge K_0 K ≥ K 0 . On E K \mathcal{E}_K E K , the potential bound gives Δ K ≤ 2 R 0 2 / A K \Delta_K \le2R_0^2/A_K Δ K ≤ 2 R 0 2 / A K . Hence, for K ≥ 2 K 0 K \ge2K_0 K ≥ 2 K 0 , the choice of a a a yields
f ( y K ) − f ⋆ ≲ a L 0 R 0 2 K 2 ≲ L 0 R 0 2 ℓ 2 K 2 + σ R 0 ℓ ( α − 1 ) / α K ( α − 1 ) / α , f(y_K) - f^\star\lesssim\frac{aL_0R_0^2}{K^2} \lesssim\frac{L_0R_0^2\ell^2}{K^2} + \frac{\sigma R_0\ell^{(\alpha-1)/\alpha}}{K^{(\alpha-1)/\alpha}}, f ( y K ) − f ⋆ ≲ K 2 a L 0 R 0 2 ≲ K 2 L 0 R 0 2 ℓ 2 + K ( α − 1 ) / α σ R 0 ℓ ( α − 1 ) / α ,
with probability at least 1 − δ 1 - \delta 1 − δ . With the initialization in Remark 1, L 1 L_1 L 1 appears only in the transient K 0 K_0 K 0 , which is accuracy-independent up to logarithmic factors.
Related Work
Optimization under generalized smoothness. The ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness model captures objectives whose local curvature can grow with the gradient norm, a dependence highlighted by Zhang et al. [24 ] . A first-order formulation was introduced by Zhang et al. [23 ] . For convex problems, clipping and related first-order methods have been studied in [10 , 9 , 21 , 14 ] . In particular, recent works [9 , 21 , 14 ] obtain nonaccelerated guarantees without exponential dependence on L 1 R L_1R L 1 R . Deterministic acceleration has also been investigated. Li et al. (2023a) established accelerated convergence under an ℓ \ell ℓ -smoothness condition with subquadratic ℓ \ell ℓ , with a rate coefficient depending on an effective smoothness bound determined by the initialization. Gorbunov et al. [9 ] proposed an accelerated Similar Triangles method whose guarantee retains an exponential dependence on L 1 R L_1R L 1 R , whereas Vankov et al. [21 ] removed this dependence using a one-dimensional line search with function values. Tyurin [20 ] obtained accelerated guarantees under ℓ \ell ℓ -smoothness with a leading high-accuracy term depending on ℓ ( 0 ) \ell(0) ℓ ( 0 ) and an accuracy-independent transient.
Stochastic optimization under generalized smoothness. Several works study stochastic methods beyond classical smoothness [25 , 1 ] (Li et al. 2023b). For convex ( L 0 , L 1 ) (L_0, L_1) ( L 0 , L 1 ) -smooth objectives, Gaash et al. [6 ] establish high-probability guarantees under sub-Gaussian noise, while El Bakkali El Kadi et al. [5 ] obtain high-probability guarantees for standard Clip-SGD under bounded central α \alpha α -th moment noise, α ∈ ( 1 , 2 ] \alpha\in(1, 2] α ∈ ( 1 , 2 ] , without exponential dependence on L 1 R L_1R L 1 R . Yu et al. (2025) analyze stochastic accelerated-gradient methods under relaxed affine variance assumptions, but their convex guarantee does not achieve an accelerated rate. Among stochastic accelerated results, Dvinskikh et al. [4 ] obtain high-probability acceleration under norm-sub-Gaussian noise without exponential dependence on L 1 R L_1R L 1 R . However, their convex oracle complexity retains multiplicative factors ( 1 + L 1 R ˉ ) 1 / 2 (1 + L_1\bar{R})^{1/2} ( 1 + L 1 R ˉ ) 1/2 and ( 1 + L 1 R ˉ ) 3 (1 + L_1\bar{R})^3 ( 1 + L 1 R ˉ ) 3 in the optimization and statistical terms, respectively, where R ˉ \bar{R} R ˉ is their localization radius. Our guarantees allow heavy-tailed noise and, up to logarithmic factors, have L 1 L_1 L 1 -independent leading high-accuracy terms.
High-probability methods under heavy-tailed noise. Gradient clipping has become a standard tool for high-probability optimization under heavy-tailed stochastic gradients. Early work includes truncated stochastic mirror descent [15 ] and high-probability analyses of Clip-SGD and stochastic Similar Triangles methods [8 ] . Subsequent work established high-probability guarantees for clipping-based stochastic methods under bounded central α \alpha α -th moment assumptions with α < 2 \alpha< 2 α < 2 [3 , 18 , 17 , 2 ] . In particular, under classical L L L -smoothness, accelerated high-probability rates under heavy-tailed noise have been established in [18 , 17 ] . Our results connect this line of work with generalized smoothness: under ( L 0 , L 1 ) (L_0, L_1) ( L 0 , L 1 ) -smoothness and bounded central α \alpha α -th moment noise, α ∈ ( 1 , 2 ] \alpha\in(1, 2] α ∈ ( 1 , 2 ] , we recover, up to logarithmic factors, the classical accelerated rate while avoiding exponential dependence on L 1 R L_1R L 1 R .
Concurrent work. Concurrently, Lobanov et al. [13 ] established high-probability accelerated guarantees under ( H 0 , H 1 ) (H_0, H_1) ( H 0 , H 1 ) -smoothness and a more general heavy-tailed noise model, allowing both gradient-dependent and objective-gap-dependent noise components. Their method combines clipping with projections and phase restarts. Our approaches use different mechanisms: a warm start followed by a single accelerated run, or a single run with modified deterministic weights. Our results show that, under ( L 0 , L 1 ) (L_0, L_1) ( L 0 , L 1 ) -smoothness and a uniform bounded central α \alpha α -th noise moment, neither projections nor repeated restarts are necessary to recover the classical accelerated rate up to logarithmic factors, with L 1 L_1 L 1 confined to an initial transient.
Conclusion
We established high-probability accelerated convergence guarantees for convex ( L 0 , L 1 ) (L_0, L_1) ( L 0 , L 1 ) -smooth optimization with unbiased stochastic gradients under heavy-tailed noise. Our results show that the exponential dependence on L 1 R L_1R L 1 R arising from direct localization is not inherent. In the two-phase approach, Clip-SGD provides a warm start in a low-curvature region, after which Clip-SSTM attains the classical accelerated rate up to logarithmic factors:
O ~ ( L 0 R 2 K 2 + σ R K ( α − 1 ) / α ) \widetilde{O}\left(\frac{L_0R^2}{K^2}+\frac{\sigma R}{K^{(\alpha-1)/\alpha}}\right) O ( K 2 L 0 R 2 + K ( α − 1 ) / α σ R )
A warm start is not necessary: a modified weight schedule allows Clip-SSTM to start directly from the initial point, control the high-curvature transient, and recover the same accelerated regime. With the prescribed initialization, both approaches confine the L 1 L_1 L 1 -dependence to an accuracy-independent transient, up to logarithmic factors.
These results show that acceleration is compatible with generalized smoothness and heavy-tailed noise, and that classical clipped accelerated schemes retain their high-accuracy behavior beyond global smoothness. A natural direction is to develop adaptive variants requiring less prior knowledge of problem parameters while preserving these high-probability guarantees.
References
Notation, Useful Identities, and Preliminary Results
Notation. Table 2 summarizes the main notation for the problem, initialization, algorithm, and stochastic analysis. Most notation is shared by the two-phase and one-phase methods. When definitions differ, as for the filtration F t \mathcal{F}_t F t , the table describes their role; precise definitions are given in the corresponding analysis.
Table 2. Main notation. Quantities whose definitions differ between the two procedures are specified in the corresponding analysis.
Useful identities for Clip-SSTM. We collect in Table 3 the identities used throughout the proofs. They follow directly from the updates of Clip-SSTM and hold for both weight schedules. Here y 0 = z 0 = v 0 y_0=z_0=v_0 y 0 = z 0 = v 0 and g t + 1 : = clip ( ∇ f ( x t + 1 , ξ t + 1 ) , λ t + 1 ) g_{t+1}:=\operatorname{clip}(\nabla f(x_{t+1},\xi_{t+1}),\lambda_{t+1}) g t + 1 := clip ( ∇ f ( x t + 1 , ξ t + 1 ) , λ t + 1 ) .
Table 3. Useful identities for Clip-SSTM.
Proposition 1 ((Gorbunov et al., [9 ] , Lemma 2.2)). Suppose Assumption 2 holds. Then, for all x ∈ R d x\in\mathbb{R}^d x ∈ R d ,
ν ∥ ∇ f ( x ) ∥ 2 ≤ 2 ( L 0 + L 1 ∥ ∇ f ( x ) ∥ ) Δ ( x ) , \nu\lVert\nabla f(x)\rVert^2\le2\left(L_0+L_1\lVert\nabla f(x)\rVert\right)\Delta(x), ν ∥ ∇ f ( x ) ∥ 2 ≤ 2 ( L 0 + L 1 ∥ ∇ f ( x )∥ ) Δ ( x ) , where ν \nu ν is the solution of x exp ( x ) = 1 x\exp(x)=1 x exp ( x ) = 1 .
Lemma 1 (Consequence of (Gorbunov et al., [9 ] , Lemma 2.1)). Assume that Assumption 2 holds. Let x , y ∈ R d x,y\in\mathbb{R}^d x , y ∈ R d , and set s : = y − x s:=y-x s := y − x . If L 1 ∥ s ∥ ≤ 1 L_1\lVert s\rVert\le1 L 1 ∥ s ∥ ≤ 1 , then
f ( y ) ≤ f ( x ) + ⟨ ∇ f ( x ) , s ⟩ + 2 ( L 0 + L 1 ∥ ∇ f ( x ) ∥ ) ∥ s ∥ 2 . f(y)\le f(x)+\langle\nabla f(x),s\rangle+2\left(L_0+L_1\lVert\nabla f(x)\rVert\right)\lVert s\rVert^2. f ( y ) ≤ f ( x ) + ⟨ ∇ f ( x ) , s ⟩ + 2 ( L 0 + L 1 ∥ ∇ f ( x )∥ ) ∥ s ∥ 2 .
Lemma 2. Assume that Assumption 2 holds. Then, for every x ∈ R d x\in\mathbb{R}^d x ∈ R d ,
∥ ∇ f ( x ) ∥ ≤ L 1 Δ ( x ) + L 1 2 Δ ( x ) 2 + 2 ν L 0 Δ ( x ) ν . \lVert\nabla f(x)\rVert\le\frac{L_1\Delta(x)+\sqrt{L_1^2\Delta(x)^2+2\nu L_0\Delta(x)}}{\nu}. ∥ ∇ f ( x )∥ ≤ ν L 1 Δ ( x ) + L 1 2 Δ ( x ) 2 + 2 ν L 0 Δ ( x ) .
Proof of Lemma 2. By Proposition 1,
ν ∥ ∇ f ( x ) ∥ 2 − 2 L 1 Δ ( x ) ∥ ∇ f ( x ) ∥ − 2 L 0 Δ ( x ) ≤ 0. \nu\lVert\nabla f(x)\rVert^2-2L_1\Delta(x)\lVert\nabla f(x)\rVert-2L_0\Delta(x)\le0. ν ∥ ∇ f ( x ) ∥ 2 − 2 L 1 Δ ( x ) ∥ ∇ f ( x )∥ − 2 L 0 Δ ( x ) ≤ 0.
Define the quadratic polynomial
p ( u ) : = ν u 2 − 2 L 1 Δ ( x ) u − 2 L 0 Δ ( x ) . p(u):=\nu u^2-2L_1\Delta(x)u-2L_0\Delta(x). p ( u ) := ν u 2 − 2 L 1 Δ ( x ) u − 2 L 0 Δ ( x ) .
The preceding inequality states that p ( ∥ ∇ f ( x ) ∥ ) ≤ 0 p(\lVert\nabla f(x)\rVert)\le0 p (∥ ∇ f ( x )∥) ≤ 0 . The roots of p p p are
u ± = L 1 Δ ( x ) ± L 1 2 Δ ( x ) 2 + 2 ν L 0 Δ ( x ) ν . u_{\pm}=\frac{L_1\Delta(x)\pm\sqrt{L_1^2\Delta(x)^2+2\nu L_0\Delta(x)}}{\nu}. u ± = ν L 1 Δ ( x ) ± L 1 2 Δ ( x ) 2 + 2 ν L 0 Δ ( x ) .
We have p ( u ) ≤ 0 p(u) \le0 p ( u ) ≤ 0 only for u ∈ [ u − , u + ] u \in[u_-,u_+] u ∈ [ u − , u + ] . Therefore,
∥ ∇ f ( x ) ∥ ≤ u + = L 1 Δ ( x ) + L 1 2 Δ ( x ) 2 + 2 ν L 0 Δ ( x ) ν . \|\nabla f(x)\| \le u_+ = \frac{L_1\Delta(x) + \sqrt{L_1^2\Delta(x)^2 + 2\nu L_0\Delta(x)}}{\nu}. ∥∇ f ( x ) ∥ ≤ u + = ν L 1 Δ ( x ) + L 1 2 Δ ( x ) 2 + 2 ν L 0 Δ ( x ) .
Lemma 3. Assume that Assumption 2 holds. Then, for all x , y ∈ R d x,y \in\mathbb{R}^d x , y ∈ R d ,
L 0 + L 1 ∥ ∇ f ( y ) ∥ ≤ e L 1 ∥ y − x ∥ ( L 0 + L 1 ∥ ∇ f ( x ) ∥ ) . L_0 + L_1\|\nabla f(y)\| \le e^{L_1\|y-x\|}(L_0 + L_1\|\nabla f(x)\|). L 0 + L 1 ∥∇ f ( y ) ∥ ≤ e L 1 ∥ y − x ∥ ( L 0 + L 1 ∥∇ f ( x ) ∥ ) .
Proof of Lemma 3. Let n ∈ N n \in\mathbb{N} n ∈ N satisfy n > L 1 ∥ y − x ∥ n > L_1\|y-x\| n > L 1 ∥ y − x ∥ and define x i = x + i n ( y − x ) x_i = x + \frac{i}{n}(y-x) x i = x + n i ( y − x ) for i = 0 , … , n i = 0,\ldots,n i = 0 , … , n , and M i = sup u ∈ [ x i , x i + 1 ] ∥ ∇ f ( u ) ∥ M_i = \sup_{u \in[x_i,x_{i+1}]} \|\nabla f(u)\| M i = sup u ∈ [ x i , x i + 1 ] ∥∇ f ( u ) ∥ . For every u ∈ [ x i , x i + 1 ] u \in[x_i,x_{i+1}] u ∈ [ x i , x i + 1 ] ,
∥ ∇ f ( u ) ∥ ≤ ∥ ∇ f ( x i ) ∥ + ∥ y − x ∥ n ( L 0 + L 1 M i ) , \|\nabla f(u)\| \le\|\nabla f(x_i)\| + \frac{\|y-x\|}{n}(L_0 + L_1M_i), ∥∇ f ( u ) ∥ ≤ ∥∇ f ( x i ) ∥ + n ∥ y − x ∥ ( L 0 + L 1 M i ) ,
hence
L 0 + L 1 M i ≤ L 0 + L 1 ∥ ∇ f ( x i ) ∥ + L 1 ∥ y − x ∥ n ( L 0 + L 1 M i ) . L_0 + L_1M_i \le L_0 + L_1\|\nabla f(x_i)\| + \frac{L_1\|y-x\|}{n}(L_0 + L_1M_i). L 0 + L 1 M i ≤ L 0 + L 1 ∥∇ f ( x i ) ∥ + n L 1 ∥ y − x ∥ ( L 0 + L 1 M i ) .
Therefore,
L 0 + L 1 ∥ ∇ f ( x i + 1 ) ∥ ≤ ( 1 − L 1 ∥ y − x ∥ n ) − 1 ( L 0 + L 1 ∥ ∇ f ( x i ) ∥ ) . L_0 + L_1\|\nabla f(x_{i+1})\| \le\left(1-\frac{L_1\|y-x\|}{n}\right)^{-1}(L_0 + L_1\|\nabla f(x_i)\|). L 0 + L 1 ∥∇ f ( x i + 1 ) ∥ ≤ ( 1 − n L 1 ∥ y − x ∥ ) − 1 ( L 0 + L 1 ∥∇ f ( x i ) ∥ ) .
Iterating for i = 0 , … , n − 1 i = 0,\ldots,n-1 i = 0 , … , n − 1 gives
L 0 + L 1 ∥ ∇ f ( y ) ∥ ≤ ( 1 − L 1 ∥ y − x ∥ n ) − n ( L 0 + L 1 ∥ ∇ f ( x ) ∥ ) . L_0 + L_1\|\nabla f(y)\| \le\left(1-\frac{L_1\|y-x\|}{n}\right)^{-n}(L_0 + L_1\|\nabla f(x)\|). L 0 + L 1 ∥∇ f ( y ) ∥ ≤ ( 1 − n L 1 ∥ y − x ∥ ) − n ( L 0 + L 1 ∥∇ f ( x ) ∥ ) .
Letting n → ∞ n \to\infty n → ∞ and using ( 1 − L 1 ∥ y − x ∥ n ) − n → e L 1 ∥ y − x ∥ \left(1-\frac{L_1\|y-x\|}{n}\right)^{-n} \to e^{L_1\|y-x\|} ( 1 − n L 1 ∥ y − x ∥ ) − n → e L 1 ∥ y − x ∥ yields the result.
Proposition 2 (((El Bakkali El Kadi et al., 2026, Corollary 1))). Let ( X t ) t ≥ 1 (X_t)_{t\ge1} ( X t ) t ≥ 1 be a martingale difference sequence with respect to a filtration ( G t ) t ≥ 0 (\mathcal{G}_t)_{t\ge0} ( G t ) t ≥ 0 such that
∣ X t ∣ ≤ c almost surely for all t ≥ 1. |X_t| \le c \quad\text{almost surely for all } t \ge1. ∣ X t ∣ ≤ c almost surely for all t ≥ 1. Define V T : = ∑ t = 1 T E [ X t 2 ∣ G t − 1 ] V_T := \sum_{t=1}^{T}\mathbb{E}[X_t^2 \mid\mathcal{G}_{t-1}] V T := ∑ t = 1 T E [ X t 2 ∣ G t − 1 ] . Assume that, for some deterministic F > 0 F > 0 F > 0 , V T ≤ F V_T \le F V T ≤ F almost surely. Then, for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ ,
∣ ∑ t = 1 T X t ∣ ≤ 2 F log 2 δ + 2 c 3 log 2 δ . \left|\sum_{t=1}^{T}X_t\right| \le\sqrt{2F\log\frac{2}{\delta}} + \frac{2c}{3}\log\frac{2}{\delta}. t = 1 ∑ T X t ≤ 2 F log δ 2 + 3 2 c log δ 2 .
High-Probability Analysis for the Two-Phase Result
Let F 0 \mathcal{F}_0 F 0 denote the Phase-I history and set
F i : = σ ( F 0 , ξ 1 , … , ξ i ) , i = 1 , … , K . \mathcal{F}_i := \sigma(\mathcal{F}_0,\xi_1,\ldots,\xi_i), \qquad i=1,\ldots,K. F i := σ ( F 0 , ξ 1 , … , ξ i ) , i = 1 , … , K .
Then x i x_i x i is F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable. Recall that
G i : = ∇ f ( x i , ξ i ) , g i : = clip ( G i , λ i ) , θ i : = g i − ∇ f ( x i ) . G_i := \nabla f(x_i,\xi_i), \qquad g_i := \operatorname{clip}(G_i,\lambda_i), \qquad\theta_i := g_i-\nabla f(x_i). G i := ∇ f ( x i , ξ i ) , g i := clip ( G i , λ i ) , θ i := g i − ∇ f ( x i ) .
Since ξ i \xi_i ξ i is independent of F i − 1 \mathcal{F}_{i-1} F i − 1 , the conditions in Assumption 3 hold conditionally on F i − 1 \mathcal{F}_{i-1} F i − 1 at x i x_i x i .
For every u ∈ R d u \in\mathbb{R}^{d} u ∈ R d , let
R t ( u ) : = ∑ i = 1 t α i ⟨ θ i , u − z i − 1 ⟩ + ∑ i = 1 t α i 2 ∥ θ i ∥ 2 . \mathcal{R}_{t}(u) := \sum_{i=1}^{t} \alpha_i \langle\theta_i, u-z_{i-1}\rangle+ \sum_{i=1}^{t} \alpha_i^2 \|\theta_i\|^2. R t ( u ) := i = 1 ∑ t α i ⟨ θ i , u − z i − 1 ⟩ + i = 1 ∑ t α i 2 ∥ θ i ∥ 2 .
The following comparator inequality is the deterministic backbone of the analysis.
Lemma 4. Suppose that Assumption 1 1 1 holds, where L 0 > 0 L_0 > 0 L 0 > 0 . Assume that A 0 = 0 A_0 = 0 A 0 = 0 and y 0 = z 0 = x ˉ y_0 = z_0 = \bar{x} y 0 = z 0 = x ˉ . Let K ≥ 1 K \ge1 K ≥ 1 and suppose that
A i ≥ 16 L 0 α i 2 , i = 1 , … , K . A_i \ge16L_0\alpha_i^2,\qquad i=1,\ldots,K. A i ≥ 16 L 0 α i 2 , i = 1 , … , K . Let t ∈ { 1 , … , K } t \in\{1,\ldots,K\} t ∈ { 1 , … , K } and assume that, for every i = 1 , … , t i=1,\ldots,t i = 1 , … , t ,
f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 . f(y_i) \le f(x_i) + \langle\nabla f(x_i),y_i-x_i\rangle+ 4L_0\|y_i-x_i\|^2. f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 . Then, for every u ∈ R d u \in\mathbb{R}^{d} u ∈ R d ,
A t ( f ( y t ) − f ( u ) ) + 1 2 ∥ z t − u ∥ 2 ≤ 1 2 ∥ x ˉ − u ∥ 2 + R t ( u ) . A_t(f(y_t)-f(u))+\frac{1}{2}\|z_t-u\|^2 \le\frac{1}{2}\|\bar{x}-u\|^2+\mathcal{R}_t(u). A t ( f ( y t ) − f ( u )) + 2 1 ∥ z t − u ∥ 2 ≤ 2 1 ∥ x ˉ − u ∥ 2 + R t ( u ) .
Lemma 5. Suppose that Assumptions 1 1 1 and 2 2 2 hold with L 0 , L 1 > 0 L_0,L_1>0 L 0 , L 1 > 0 . Assume that f ( x ˉ ) − f ⋆ ≤ c L 0 L 1 2 f(\bar{x})-f^\star\le\frac{cL_0}{L_1^2} f ( x ˉ ) − f ⋆ ≤ L 1 2 c L 0 for some 0 < c ≤ 1 32 0<c\le\frac{1}{32} 0 < c ≤ 32 1 . Assume moreover that A 0 = 0 A_0=0 A 0 = 0 and y 0 = z 0 = x ˉ y_0=z_0=\bar{x} y 0 = z 0 = x ˉ . Let t ∈ { 0 , … , K − 1 } t\in\{0,\ldots,K-1\} t ∈ { 0 , … , K − 1 } and suppose that for all i ∈ { 1 , … , t + 1 } i\in\{1,\ldots,t+1\} i ∈ { 1 , … , t + 1 } , we have A i ≥ 16 L 0 α i 2 A_i\ge16L_0\alpha_i^2 A i ≥ 16 L 0 α i 2 . Suppose that, for every i = 1 , … , t i=1,\ldots,t i = 1 , … , t ,
{ f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 , R i ( x ˉ ) ≤ c A i L 0 L 1 2 , \left\{
\begin{aligned}
f(y_i)&\le f(x_i)+\langle\nabla f(x_i),y_i-x_i\rangle+4L_0\|y_i-x_i\|^2,\\
\mathcal{R}_i(\bar{x})&\le\frac{cA_iL_0}{L_1^2},
\end{aligned}
\right. ⎩ ⎨ ⎧ f ( y i ) R i ( x ˉ ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 , ≤ L 1 2 c A i L 0 , and that
λ t + 1 ≤ A t + 1 L 1 α t + 1 2 . \lambda_{t+1}\le\frac{A_{t+1}}{L_1\alpha_{t+1}^2}. λ t + 1 ≤ L 1 α t + 1 2 A t + 1 . Then
{ ∥ ∇ f ( x t + 1 ) ∥ ≤ L 0 L 1 , L 1 ∥ y t + 1 − x t + 1 ∥ ≤ 1. \left\{
\begin{aligned}
\|\nabla f(x_{t+1})\|&\le\frac{L_0}{L_1},\\
L_1\|y_{t+1}-x_{t+1}\|&\le1.
\end{aligned}
\right. ⎩ ⎨ ⎧ ∥∇ f ( x t + 1 ) ∥ L 1 ∥ y t + 1 − x t + 1 ∥ ≤ L 1 L 0 , ≤ 1. Consequently,
f ( y t + 1 ) ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 4 L 0 ∥ y t + 1 − x t + 1 ∥ 2 . f(y_{t+1})\le f(x_{t+1})+\langle\nabla f(x_{t+1}),y_{t+1}-x_{t+1}\rangle+4L_0\|y_{t+1}-x_{t+1}\|^2. f ( y t + 1 ) ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 4 L 0 ∥ y t + 1 − x t + 1 ∥ 2 .
Lemma 6. Suppose that Assumptions 1 1 1 and 2 2 2 hold with L 0 , L 1 > 0 L_0,L_1>0 L 0 , L 1 > 0 , and assume that ∥ x ˉ − x ⋆ ∥ ≤ R \|\bar{x}-x^\star\|\le R ∥ x ˉ − x ⋆ ∥ ≤ R and f ( x ˉ ) − f ⋆ ≤ c L 0 L 1 2 f(\bar{x})-f^\star\le\frac{cL_0}{L_1^2} f ( x ˉ ) − f ⋆ ≤ L 1 2 c L 0 , where R > 0 R>0 R > 0 and 0 < c ≤ 1 / 32 0<c\le1/32 0 < c ≤ 1/32 . Let ℓ ≥ 4 \ell\ge4 ℓ ≥ 4 and assume that A 0 = 0 A_0=0 A 0 = 0 and y 0 = z 0 = x ˉ y_0=z_0=\bar{x} y 0 = z 0 = x ˉ . Suppose that
{ A i ≥ 96 ℓ 2 L 0 α i 2 , i = 1 , … , K , A i + 1 ≤ 5 2 A i , i = 1 , … , K − 1. \left\{
\begin{aligned}
A_i&\ge96\ell^2L_0\alpha_i^2,\qquad i=1,\ldots,K,\\
A_{i+1}&\le\frac{5}{2}A_i,\qquad i=1,\ldots,K-1.
\end{aligned}
\right. ⎩ ⎨ ⎧ A i A i + 1 ≥ 96 ℓ 2 L 0 α i 2 , i = 1 , … , K , ≤ 2 5 A i , i = 1 , … , K − 1. Define
λ i : = 1 α i ℓ min { c A i L 0 L 1 , R } , i = 1 , … , K . \lambda_i:=\frac{1}{\alpha_i\ell}\min\left\{\frac{\sqrt{cA_iL_0}}{L_1},R\right\},\qquad i=1,\ldots,K. λ i := α i ℓ 1 min { L 1 c A i L 0 , R } , i = 1 , … , K . Let t ∈ { 0 , … , K − 1 } t\in\{0,\ldots,K-1\} t ∈ { 0 , … , K − 1 } and suppose that
{ f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 , R i ( x ˉ ) ≤ c A i L 0 L 1 2 , R i ( x ⋆ ) ≤ R 2 2 , i = 1 , … , t . \left\{
\begin{aligned}
f(y_i)&\le f(x_i)+\langle\nabla f(x_i),y_i-x_i\rangle+4L_0\|y_i-x_i\|^2,\\
\mathcal{R}_i(\bar{x})&\le\frac{cA_iL_0}{L_1^2},\\
\mathcal{R}_i(x^\star)&\le\frac{R^2}{2},
\end{aligned}
\right.
\qquad i=1,\ldots,t. ⎩ ⎨ ⎧ f ( y i ) R i ( x ˉ ) R i ( x ⋆ ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 , ≤ L 1 2 c A i L 0 , ≤ 2 R 2 , i = 1 , … , t .
Then
∥ ∇ f ( x t + 1 ) ∥ ≤ λ t + 1 2 . \lVert\nabla f(x_{t+1}) \rVert\le\frac{\lambda_{t+1}}{2}. ∥ ∇ f ( x t + 1 )∥ ≤ 2 λ t + 1 .
In the following, suppose that Assumptions 1 to 3 hold and that the initialization and parameter conditions of Lemma 6 are satisfied.
For t = 0 , … , K t=0,\ldots,K t = 0 , … , K , let E t \mathcal{E}_t E t be the event that, for every i = 1 , … , t i=1,\ldots,t i = 1 , … , t ,
{ f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 , R i ( x ˉ ) ≤ c A i L 0 L 1 2 , R i ( x ⋆ ) ≤ R 2 2 . \begin{cases}
f(y_i) \le f(x_i) + \langle\nabla f(x_i),y_i-x_i\rangle+ 4L_0\lVert y_i-x_i\rVert^2,\\
\mathcal{R}_i(\bar{x}) \le\dfrac{cA_iL_0}{L_1^2},\\
\mathcal{R}_i(x^\star) \le\dfrac{R^2}{2}.
\end{cases} ⎩ ⎨ ⎧ f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 L 0 ∥ y i − x i ∥ 2 , R i ( x ˉ ) ≤ L 1 2 c A i L 0 , R i ( x ⋆ ) ≤ 2 R 2 .
For t = 0 t=0 t = 0 , there are no conditions, so 1 E 0 = 1 \mathbb{1}_{\mathcal{E}_0}=1 1 E 0 = 1 .
For u ∈ { x ˉ , x ⋆ } u\in\{\bar{x},x^\star\} u ∈ { x ˉ , x ⋆ } , define
R ^ t ( u ) : = ∑ i = 1 t 1 E i − 1 ( α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 ) . \widehat{\mathcal{R}}_t(u):=\sum_{i=1}^{t}\mathbb{1}_{\mathcal{E}_{i-1}}\left(\alpha_i\langle\theta_i,u-z_{i-1}\rangle+\alpha_i^2\lVert\theta_i\rVert^2\right). R t ( u ) := i = 1 ∑ t 1 E i − 1 ( α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 ) .
Since the events E t \mathcal{E}_t E t are nested, on E t − 1 \mathcal{E}_{t-1} E t − 1 we have
R ^ t ( u ) = R t ( u ) . \widehat{\mathcal{R}}_t(u)=\mathcal{R}_t(u). R t ( u ) = R t ( u ) .
On E i − 1 \mathcal{E}_{i-1} E i − 1 , Lemma 6 gives ∥ ∇ f ( x i ) ∥ ≤ λ i / 2 \lVert\nabla f(x_i)\rVert\le\lambda_i/2 ∥ ∇ f ( x i )∥ ≤ λ i /2 . Since E i − 1 ∈ F i − 1 \mathcal{E}_{i-1}\in\mathcal{F}_{i-1} E i − 1 ∈ F i − 1 , applying Sadiev et al. [18 ] conditionally on F i − 1 \mathcal{F}_{i-1} F i − 1 yields, on E i − 1 \mathcal{E}_{i-1} E i − 1 ,
∥ E [ θ i ∣ F i − 1 ] ∥ ≤ 2 α σ α λ i α − 1 . \lVert\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}]\rVert\le\frac{2^\alpha\sigma^\alpha}{\lambda_i^{\alpha-1}}. ∥ E [ θ i ∣ F i − 1 ]∥ ≤ λ i α − 1 2 α σ α .
Moreover, nonexpansiveness of clipping and ∥ g i ∥ ≤ λ i \lVert g_i\rVert\le\lambda_i ∥ g i ∥ ≤ λ i imply, on E i − 1 \mathcal{E}_{i-1} E i − 1 , ∥ θ i ∥ ≤ min { ∥ G i − ∇ f ( x i ) ∥ , 3 2 λ i } \lVert\theta_i\rVert\le\min\{\lVert G_i-\nabla f(x_i)\rVert,\frac{3}{2}\lambda_i\} ∥ θ i ∥ ≤ min {∥ G i − ∇ f ( x i )∥ , 2 3 λ i } . Therefore, on E i − 1 \mathcal{E}_{i-1} E i − 1 ,
E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ ( 3 2 λ i ) 2 − α E [ ∥ G i − ∇ f ( x i ) ∥ α ∣ F i − 1 ] ≤ 2 σ α λ i 2 − α . \mathbb{E}[\lVert\theta_i\rVert^2\mid\mathcal{F}_{i-1}]
\le\left(\frac{3}{2}\lambda_i\right)^{2-\alpha}\mathbb{E}[\lVert G_i-\nabla f(x_i)\rVert^\alpha\mid\mathcal{F}_{i-1}]
\le2\sigma^\alpha\lambda_i^{2-\alpha}. E [∥ θ i ∥ 2 ∣ F i − 1 ] ≤ ( 2 3 λ i ) 2 − α E [∥ G i − ∇ f ( x i ) ∥ α ∣ F i − 1 ] ≤ 2 σ α λ i 2 − α .
To control R ^ t ( u ) \widehat{\mathcal{R}}_t(u) R t ( u ) , we subtract the conditional mean from each summand. This separates the random fluctuations from the clipping bias and the conditional second-moment terms. For u ∈ { x ˉ , x ⋆ } u\in\{\bar{x},x^\star\} u ∈ { x ˉ , x ⋆ } , we write
R ^ t ( u ) = M t ( u ) + B t ( u ) , \widehat{\mathcal{R}}_t(u)=M_t(u)+B_t(u), R t ( u ) = M t ( u ) + B t ( u ) ,
where
{ M t ( u ) : = ∑ i = 1 t 1 E i − 1 ( α i ⟨ θ i − E [ θ i ∣ F i − 1 ] , u − z i − 1 ⟩ + α i 2 ( ∥ θ i ∥ 2 − E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ) ) , B t ( u ) : = ∑ i = 1 t 1 E i − 1 ( α i ⟨ E [ θ i ∣ F i − 1 ] , u − z i − 1 ⟩ + α i 2 E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ) . \begin{cases}
M_t(u):=\sum_{i=1}^{t}\mathbb{1}_{\mathcal{E}_{i-1}}\left(\alpha_i\langle\theta_i-\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}],u-z_{i-1}\rangle+\alpha_i^2\left(\lVert\theta_i\rVert^2-\mathbb{E}[\lVert\theta_i\rVert^2\mid\mathcal{F}_{i-1}]\right)\right),\\
B_t(u):=\sum_{i=1}^{t}\mathbb{1}_{\mathcal{E}_{i-1}}\left(\alpha_i\langle\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}],u-z_{i-1}\rangle+\alpha_i^2\mathbb{E}[\lVert\theta_i\rVert^2\mid\mathcal{F}_{i-1}]\right).
\end{cases} { M t ( u ) := ∑ i = 1 t 1 E i − 1 ( α i ⟨ θ i − E [ θ i ∣ F i − 1 ] , u − z i − 1 ⟩ + α i 2 ( ∥ θ i ∥ 2 − E [∥ θ i ∥ 2 ∣ F i − 1 ] ) ) , B t ( u ) := ∑ i = 1 t 1 E i − 1 ( α i ⟨ E [ θ i ∣ F i − 1 ] , u − z i − 1 ⟩ + α i 2 E [∥ θ i ∥ 2 ∣ F i − 1 ] ) .
Since 1 E i − 1 \mathbb{1}_{\mathcal{E}_{i-1}} 1 E i − 1 and u − z i − 1 u-z_{i-1} u − z i − 1 are F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable, the increments of M t ( u ) M_t(u) M t ( u ) have zero conditional mean. Thus, ( M t ( u ) ) t = 0 K (M_t(u))_{t=0}^K ( M t ( u ) ) t = 0 K is a martingale, with M 0 ( u ) = 0 M_0(u)=0 M 0 ( u ) = 0 .
The clipping bias and second-moment bounds give
B t ( u ) ≤ ∑ i = 1 t 1 E i − 1 ( 2 α σ α α i λ i α − 1 ∥ u − z i − 1 ∥ + 2 σ α α i 2 λ i 2 − α ) . B_t(u)\le\sum_{i=1}^{t}\mathbb{1}_{\mathcal{E}_{i-1}}\left(\frac{2^\alpha\sigma^\alpha\alpha_i}{\lambda_i^{\alpha-1}}\lVert u-z_{i-1}\rVert+2\sigma^\alpha\alpha_i^2\lambda_i^{2-\alpha}\right). B t ( u ) ≤ i = 1 ∑ t 1 E i − 1 ( λ i α − 1 2 α σ α α i ∥ u − z i − 1 ∥ + 2 σ α α i 2 λ i 2 − α ) .
For i ≥ 2 i \ge2 i ≥ 2 , on E i − 1 \mathcal{E}_{i-1} E i − 1 , applying Lemma 4 at iteration i − 1 i-1 i − 1 with u = x ˉ u=\bar{x} u = x ˉ and u = x ⋆ u=x^\star u = x ⋆ gives
{ ∥ x ˉ − z i − 1 ∥ ≤ 2 c A i − 1 L 0 L 1 , ∥ x ⋆ − z i − 1 ∥ ≤ 2 R . \begin{cases}
\lVert\bar{x}-z_{i-1}\rVert\le\dfrac{2\sqrt{cA_{i-1}L_0}}{L_1},\\
\lVert x^\star-z_{i-1}\rVert\le\sqrt{2}R.
\end{cases} ⎩ ⎨ ⎧ ∥ x ˉ − z i − 1 ∥ ≤ L 1 2 c A i − 1 L 0 , ∥ x ⋆ − z i − 1 ∥ ≤ 2 R .
For i = 1 i=1 i = 1 , these bounds follow from z 0 = x ˉ z_0=\bar{x} z 0 = x ˉ , A 0 = 0 A_0=0 A 0 = 0 , and ∥ x ˉ − x ⋆ ∥ ≤ R \lVert\bar{x}-x^\star\rVert\le R ∥ x ˉ − x ⋆ ∥ ≤ R . To control M t ( u ) M_t(u) M t ( u ) , we bound its increments and the sum of their conditional variances, then apply Freedman’s inequality. A union bound over t = 1 , … , K t=1,\ldots,K t = 1 , … , K and u ∈ { x ˉ , x ⋆ } u\in\{\bar{x},x^\star\} u ∈ { x ˉ , x ⋆ } gives the following result.
Lemma 7. Suppose that Assumptions 1,2 and 3 hold with L 0 , L 1 > 0 L_0,L_1>0 L 0 , L 1 > 0 . Assume that ∥ x ˉ − x ⋆ ∥ ≤ R \lVert\bar{x}-x^\star\rVert\le R ∥ x ˉ − x ⋆ ∥ ≤ R and f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 f(\bar{x})-f^\star\le cL_0/L_1^2 f ( x ˉ ) − f ⋆ ≤ c L 0 / L 1 2 , where R > 0 R>0 R > 0 and 0 < c ≤ 1 / 32 0<c\le1/32 0 < c ≤ 1/32 . Assume moreover that A 0 = 0 A_0=0 A 0 = 0 and y 0 = z 0 = x ˉ y_0=z_0=\bar{x} y 0 = z 0 = x ˉ . Let K ≥ 1 K\ge1 K ≥ 1 , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and ℓ ≥ 8 log ( 4 K / δ ) \ell\ge8\log(4K/\delta) ℓ ≥ 8 log ( 4 K / δ ) . Suppose that
{ A i ≥ 96 ℓ 2 L 0 α i 2 , i = 1 , … , K , A i + 1 ≤ 5 2 A i , i = 1 , … , K − 1. \begin{cases}
A_i \ge96\ell^2 L_0\alpha_i^2,\quad i=1,\ldots,K,\\
A_{i+1}\le\dfrac{5}{2}A_i,\quad i=1,\ldots,K-1.
\end{cases} ⎩ ⎨ ⎧ A i ≥ 96 ℓ 2 L 0 α i 2 , i = 1 , … , K , A i + 1 ≤ 2 5 A i , i = 1 , … , K − 1. For i = 1 , … , K i=1,\ldots,K i = 1 , … , K , define
{ s i : = min { c A i L 0 L 1 , R } , λ i : = s i α i ℓ . \begin{cases}
s_i:=\min\left\{\dfrac{\sqrt{cA_iL_0}}{L_1},R\right\},\\
\lambda_i:=\dfrac{s_i}{\alpha_i\ell}.
\end{cases} ⎩ ⎨ ⎧ s i := min { L 1 c A i L 0 , R } , λ i := α i ℓ s i . Suppose that σ α ∑ i = 1 K α i s i α i − 1 ≤ 1 512 \sigma^\alpha\sum_{i=1}^{K}\dfrac{\alpha_i}{s_i\alpha_i^{-1}}\le\dfrac{1}{512} σ α ∑ i = 1 K s i α i − 1 α i ≤ 512 1 . Then, with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every t = 1 , … , K t=1,\ldots,K t = 1 , … , K ,
{ R ^ t ( x ˉ ) ≤ c A t L 0 L 1 2 , R ^ t ( x ⋆ ) ≤ R 2 2 . \begin{cases}
\widehat{R}_t(\bar{x})\le\dfrac{cA_tL_0}{L_1^2},\\
\widehat{R}_t(x^\star)\le\dfrac{R^2}{2}.
\end{cases} ⎩ ⎨ ⎧ R t ( x ˉ ) ≤ L 1 2 c A t L 0 , R t ( x ⋆ ) ≤ 2 R 2 .
On this event, E K \mathcal{E}_K E K holds.
For the choice α i = i + 1 a L 0 \alpha_i=\dfrac{i+1}{aL_0} α i = a L 0 i + 1 , we have A i = i ( i + 3 ) 2 a L 0 ≥ a L 0 2 α i 2 A_i=\dfrac{i(i+3)}{2aL_0}\ge\dfrac{aL_0}{2}\alpha_i^2 A i = 2 a L 0 i ( i + 3 ) ≥ 2 a L 0 α i 2 . Hence, if a ≥ 192 ℓ 2 a\ge192\ell^2 a ≥ 192 ℓ 2 , we have A i ≥ 96 ℓ 2 L 0 α i 2 A_i\ge96\ell^2L_0\alpha_i^2 A i ≥ 96 ℓ 2 L 0 α i 2 . Moreover, for all i ≥ 1 i\ge1 i ≥ 1 , we have A i + 1 A i = ( i + 1 ) ( i + 4 ) i ( i + 3 ) ≤ 5 2 \dfrac{A_{i+1}}{A_i}=\dfrac{(i+1)(i+4)}{i(i+3)}\le\dfrac{5}{2} A i A i + 1 = i ( i + 3 ) ( i + 1 ) ( i + 4 ) ≤ 2 5 . Thus, the conditions on A i A_i A i in Lemma 7 hold whenever a ≥ 192 ℓ 2 a\ge192\ell^2 a ≥ 192 ℓ 2 .
Proofs
Proof of Lemma 4 . Fix u ∈ R d u\in\mathbb{R}^d u ∈ R d and define
Φ i ( u ) : = A i ( f ( y i ) − f ( u ) ) + 1 2 ∥ z i − u ∥ 2 . \Phi_i(u):=A_i\bigl(f(y_i)-f(u)\bigr)+\frac{1}{2}\lVert z_i-u\rVert^2. Φ i ( u ) := A i ( f ( y i ) − f ( u ) ) + 2 1 ∥ z i − u ∥ 2 .
By convexity, we have
{ f ( y i − 1 ) ≥ f ( x i ) + ⟨ ∇ f ( x i ) , y i − 1 − x i ⟩ , f ( u ) ≥ f ( x i ) + ⟨ ∇ f ( x i ) , u − x i ⟩ . \begin{cases}
f(y_{i-1})\ge f(x_i)+\langle\nabla f(x_i),y_{i-1}-x_i\rangle,\\
f(u)\ge f(x_i)+\langle\nabla f(x_i),u-x_i\rangle.
\end{cases} { f ( y i − 1 ) ≥ f ( x i ) + ⟨ ∇ f ( x i ) , y i − 1 − x i ⟩ , f ( u ) ≥ f ( x i ) + ⟨ ∇ f ( x i ) , u − x i ⟩ .
Multiplying these inequalities by A i − 1 A_{i-1} A i − 1 and α i \alpha_i α i , respectively, and adding them yields
A i − 1 f ( y i − 1 ) + α i f ( u ) ≥ A i f ( x i ) + ⟨ ∇ f ( x i ) , A i − 1 y i − 1 + α i u − A i x i ⟩ . (3) A_{i-1}f(y_{i-1})+\alpha_i f(u)\ge A_i f(x_i)+\langle\nabla f(x_i),A_{i-1}y_{i-1}+\alpha_i u-A_i x_i\rangle. \tag*{(3)} A i − 1 f ( y i − 1 ) + α i f ( u ) ≥ A i f ( x i ) + ⟨ ∇ f ( x i ) , A i − 1 y i − 1 + α i u − A i x i ⟩ . (3)
where we used A i = A i − 1 + α i A_i=A_{i-1}+\alpha_i A i = A i − 1 + α i . By the definition of x i x_i x i , A i x i = A i − 1 y i − 1 + α i z i − 1 A_i x_i=A_{i-1}y_{i-1}+\alpha_i z_{i-1} A i x i = A i − 1 y i − 1 + α i z i − 1 . Therefore
A i f ( x i ) − A i − 1 f ( y i − 1 ) − α i f ( u ) ≤ α i ⟨ ∇ f ( x i ) , z i − 1 − u ⟩ . A_i f(x_i)-A_{i-1}f(y_{i-1})-\alpha_i f(u)\leq\alpha_i\langle\nabla f(x_i),z_{i-1}-u\rangle. A i f ( x i ) − A i − 1 f ( y i − 1 ) − α i f ( u ) ≤ α i ⟨ ∇ f ( x i ) , z i − 1 − u ⟩ .
We have y i − x i = − α i 2 A i g i y_i-x_i=-\frac{\alpha_i^2}{A_i}g_i y i − x i = − A i α i 2 g i . Substituting this into the assumed descent inequality and multiplying by A i A_i A i , we obtain
A i ( f ( y i ) − f ( x i ) ) ≤ − α i 2 ⟨ ∇ f ( x i ) , g i ⟩ + 4 L 0 α i 4 A i ∥ g i ∥ 2 . A_i\bigl(f(y_i)-f(x_i)\bigr)\leq-\alpha_i^2\langle\nabla f(x_i),g_i\rangle+\frac{4L_0\alpha_i^4}{A_i}\lVert g_i\rVert^2. A i ( f ( y i ) − f ( x i ) ) ≤ − α i 2 ⟨ ∇ f ( x i ) , g i ⟩ + A i 4 L 0 α i 4 ∥ g i ∥ 2 .
It follows that
A i ( f ( y i ) − f ( u ) ) − A i − 1 ( f ( y i − 1 ) − f ( u ) ) ≤ α i ⟨ ∇ f ( x i ) , z i − 1 − u ⟩ − α i 2 ⟨ ∇ f ( x i ) , g i ⟩ + 4 L 0 α i 4 A i ∥ g i ∥ 2 . A_i\bigl(f(y_i)-f(u)\bigr)-A_{i-1}\bigl(f(y_{i-1})-f(u)\bigr)\leq\alpha_i\langle\nabla f(x_i),z_{i-1}-u\rangle-\alpha_i^2\langle\nabla f(x_i),g_i\rangle+\frac{4L_0\alpha_i^4}{A_i}\lVert g_i\rVert^2. A i ( f ( y i ) − f ( u ) ) − A i − 1 ( f ( y i − 1 ) − f ( u ) ) ≤ α i ⟨ ∇ f ( x i ) , z i − 1 − u ⟩ − α i 2 ⟨ ∇ f ( x i ) , g i ⟩ + A i 4 L 0 α i 4 ∥ g i ∥ 2 .
Since z i = z i − 1 − α i g i z_i=z_{i-1}-\alpha_i g_i z i = z i − 1 − α i g i , it holds that
1 2 ∥ z i − u ∥ 2 − 1 2 ∥ z i − 1 − u ∥ 2 = − α i ⟨ g i , z i − 1 − u ⟩ + α i 2 2 ∥ g i ∥ 2 . \frac{1}{2}\lVert z_i-u\rVert^2-\frac{1}{2}\lVert z_{i-1}-u\rVert^2=-\alpha_i\langle g_i,z_{i-1}-u\rangle+\frac{\alpha_i^2}{2}\lVert g_i\rVert^2. 2 1 ∥ z i − u ∥ 2 − 2 1 ∥ z i − 1 − u ∥ 2 = − α i ⟨ g i , z i − 1 − u ⟩ + 2 α i 2 ∥ g i ∥ 2 .
Using θ i = g i − ∇ f ( x i ) \theta_i=g_i-\nabla f(x_i) θ i = g i − ∇ f ( x i ) , we obtain
Φ i ( u ) − Φ i − 1 ( u ) ≤ α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 [ ⟨ θ i , g i ⟩ − ( 1 2 − 4 L 0 α i 2 A i ) ∥ g i ∥ 2 ] , \Phi_i(u)-\Phi_{i-1}(u)\leq\alpha_i\langle\theta_i,u-z_{i-1}\rangle+\alpha_i^2\left[\langle\theta_i,g_i\rangle-\left(\frac{1}{2}-\frac{4L_0\alpha_i^2}{A_i}\right)\lVert g_i\rVert^2\right], Φ i ( u ) − Φ i − 1 ( u ) ≤ α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 [ ⟨ θ i , g i ⟩ − ( 2 1 − A i 4 L 0 α i 2 ) ∥ g i ∥ 2 ] ,
where we used − ⟨ ∇ f ( x i ) , g i ⟩ = − ⟨ g i − θ i , g i ⟩ = ⟨ θ i , g i ⟩ − ∥ g i ∥ 2 -\langle\nabla f(x_i),g_i\rangle=-\langle g_i-\theta_i,g_i\rangle=\langle\theta_i,g_i\rangle-\lVert g_i\rVert^2 − ⟨ ∇ f ( x i ) , g i ⟩ = − ⟨ g i − θ i , g i ⟩ = ⟨ θ i , g i ⟩ − ∥ g i ∥ 2 .
By assumption, A i ≥ 16 L 0 α i 2 A_i\geq16L_0\alpha_i^2 A i ≥ 16 L 0 α i 2 , and therefore 4 L 0 α i 2 A i ≤ 1 4 \frac{4L_0\alpha_i^2}{A_i}\leq\frac{1}{4} A i 4 L 0 α i 2 ≤ 4 1 . Consequently,
⟨ θ i , g i ⟩ − ( 1 2 − 4 L 0 α i 2 A i ) ∥ g i ∥ 2 ≤ ⟨ θ i , g i ⟩ − 1 4 ∥ g i ∥ 2 = ∥ θ i ∥ 2 − 1 4 ∥ g i − 2 θ i ∥ 2 ≤ ∥ θ i ∥ 2 . \langle\theta_i,g_i\rangle-\left(\frac{1}{2}-\frac{4L_0\alpha_i^2}{A_i}\right)\lVert g_i\rVert^2\leq\langle\theta_i,g_i\rangle-\frac{1}{4}\lVert g_i\rVert^2=\lVert\theta_i\rVert^2-\frac{1}{4}\lVert g_i-2\theta_i\rVert^2\leq\lVert\theta_i\rVert^2. ⟨ θ i , g i ⟩ − ( 2 1 − A i 4 L 0 α i 2 ) ∥ g i ∥ 2 ≤ ⟨ θ i , g i ⟩ − 4 1 ∥ g i ∥ 2 = ∥ θ i ∥ 2 − 4 1 ∥ g i − 2 θ i ∥ 2 ≤ ∥ θ i ∥ 2 .
We deduce that
Φ i ( u ) − Φ i − 1 ( u ) ≤ α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 . \Phi_i(u)-\Phi_{i-1}(u)\leq\alpha_i\langle\theta_i,u-z_{i-1}\rangle+\alpha_i^2\lVert\theta_i\rVert^2. Φ i ( u ) − Φ i − 1 ( u ) ≤ α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 .
Summing over i = 1 , … , t i=1,\ldots,t i = 1 , … , t yields
Φ t ( u ) ≤ Φ 0 ( u ) + R t ( u ) . \Phi_t(u)\leq\Phi_0(u)+R_t(u). Φ t ( u ) ≤ Φ 0 ( u ) + R t ( u ) .
Finally, A 0 = 0 A_0=0 A 0 = 0 and z 0 = x ˉ z_0=\bar{x} z 0 = x ˉ imply Φ 0 ( u ) = 1 2 ∥ x ˉ − u ∥ 2 \Phi_0(u)=\frac{1}{2}\lVert\bar{x}-u\rVert^2 Φ 0 ( u ) = 2 1 ∥ x ˉ − u ∥ 2 , so
A t ( f ( y t ) − f ( u ) ) + 1 2 ∥ z t − u ∥ 2 ≤ 1 2 ∥ x ˉ − u ∥ 2 + R t ( u ) . A_t\bigl(f(y_t)-f(u)\bigr)+\frac{1}{2}\lVert z_t-u\rVert^2\leq\frac{1}{2}\lVert\bar{x}-u\rVert^2+R_t(u). A t ( f ( y t ) − f ( u ) ) + 2 1 ∥ z t − u ∥ 2 ≤ 2 1 ∥ x ˉ − u ∥ 2 + R t ( u ) .
Proof of Lemma 5 . If t = 0 t=0 t = 0 , then x 1 = x ˉ x_1=\bar{x} x 1 = x ˉ , and by Lemma 2, we have
∥ ∇ f ( x 1 ) ∥ ≤ c + c 2 + 2 ν c L 0 ν ≤ ( 2 c + 4 c ) L 0 L 1 ≤ L 0 L 1 , \lVert\nabla f(x_1)\rVert\leq\frac{c+\sqrt{c^2+2\nu c}\,L_0}{\nu}\leq(2\sqrt{c}+4c)\frac{L_0}{L_1}\leq\frac{L_0}{L_1}, ∥ ∇ f ( x 1 )∥ ≤ ν c + c 2 + 2 ν c L 0 ≤ ( 2 c + 4 c ) L 1 L 0 ≤ L 1 L 0 ,
where we used ν ≥ 1 / 2 \nu\geq1/2 ν ≥ 1/2 and c ≤ 1 / 32 c\leq1/32 c ≤ 1/32 .
Now suppose t ≥ 1 t\geq1 t ≥ 1 . Applying Lemma 4 with u = x ˉ u=\bar{x} u = x ˉ , we obtain, for every j = 1 , … , t j=1,\ldots,t j = 1 , … , t ,
A j ( f ( y j ) − f ∗ ) + 1 2 ∥ z j − x ˉ ∥ 2 ≤ A j ( f ( x ˉ ) − f ∗ ) + R j ( x ˉ ) ≤ 2 c A j L 0 L 1 2 . A_j\bigl(f(y_j)-f^*\bigr)+\frac{1}{2}\lVert z_j-\bar{x}\rVert^2\leq A_j\bigl(f(\bar{x})-f^*\bigr)+R_j(\bar{x})\leq\frac{2cA_jL_0}{L_1^2}. A j ( f ( y j ) − f ∗ ) + 2 1 ∥ z j − x ˉ ∥ 2 ≤ A j ( f ( x ˉ ) − f ∗ ) + R j ( x ˉ ) ≤ L 1 2 2 c A j L 0 .
Then
{ f ( y t ) − f ∗ ≤ 2 c L 0 L 1 2 , ∥ z j − x ˉ ∥ ≤ 2 A j ( f ( x ˉ ) − f ∗ ) + 2 R j ( x ˉ ) ≤ 4 c A j L 0 L 1 2 = 2 c A j L 0 L 1 . \begin{cases}
f(y_t)-f^* \le\dfrac{2cL_0}{L_1^2},\\
\lVert z_j-\bar{x}\rVert\le\sqrt{2A_j\left(f(\bar{x})-f^*\right)+2R_j(\bar{x})}\le\sqrt{\dfrac{4cA_jL_0}{L_1^2}}=\dfrac{2\sqrt{cA_jL_0}}{L_1}.
\end{cases} ⎩ ⎨ ⎧ f ( y t ) − f ∗ ≤ L 1 2 2 c L 0 , ∥ z j − x ˉ ∥ ≤ 2 A j ( f ( x ˉ ) − f ∗ ) + 2 R j ( x ˉ ) ≤ L 1 2 4 c A j L 0 = L 1 2 c A j L 0 .
By Lemma 2 and ν ≥ 1 / 2 \nu\ge1/2 ν ≥ 1/2 , we obtain
∥ ∇ f ( y t ) ∥ ≤ 2 c + 4 c 2 + 4 ν c ν L 0 L 1 ≤ ( 4 c + 16 c 2 + 8 c ) L 0 L 1 ≤ 1 + 17 8 L 0 L 1 . \begin{aligned}
\lVert\nabla f(y_t)\rVert\le\dfrac{2c+\sqrt{4c^2+4\nu c}}{\nu}\dfrac{L_0}{L_1} \\
&\le\left(4c+\sqrt{16c^2+8c}\right)\dfrac{L_0}{L_1} \\
&\le\dfrac{1+\sqrt{17}}{8}\dfrac{L_0}{L_1}.
\end{aligned} ∥ ∇ f ( y t )∥ ≤ ν 2 c + 4 c 2 + 4 ν c L 1 L 0 ≤ ( 4 c + 16 c 2 + 8 c ) L 1 L 0 ≤ 8 1 + 17 L 1 L 0 .
where the last inequality uses c ≤ 1 32 c\le\dfrac{1}{32} c ≤ 32 1 . Using y t = A t − 1 ∑ j = 1 t α j z j y_t=A_t^{-1}\sum_{j=1}^{t}\alpha_jz_j y t = A t − 1 ∑ j = 1 t α j z j and A j ≤ A t A_j\le A_t A j ≤ A t for all j ∈ { 1 , … , t } j\in\{1,\ldots,t\} j ∈ { 1 , … , t }
∥ y t − x ˉ ∥ ≤ 1 A t ∑ j = 1 t α j ∥ z j − x ˉ ∥ ≤ 2 c A t L 0 L 1 . \lVert y_t-\bar{x}\rVert\le\dfrac{1}{A_t}\sum_{j=1}^{t}\alpha_j\lVert z_j-\bar{x}\rVert\le\dfrac{2\sqrt{cA_tL_0}}{L_1}. ∥ y t − x ˉ ∥ ≤ A t 1 j = 1 ∑ t α j ∥ z j − x ˉ ∥ ≤ L 1 2 c A t L 0 .
Therefore, using the definition of x t + 1 x_{t+1} x t + 1 and the triangle inequality,
L 1 ∥ x t + 1 − y t ∥ = L 1 α t + 1 A t + 1 ∥ z t − y t ∥ ≤ 4 α t + 1 c A t L 0 A t + 1 \begin{aligned}
L_1\lVert x_{t+1}-y_t\rVert=\dfrac{L_1\alpha_{t+1}}{A_{t+1}}\lVert z_t-y_t\rVert\\
&\le\dfrac{4\alpha_{t+1}\sqrt{cA_tL_0}}{A_{t+1}}
\end{aligned} L 1 ∥ x t + 1 − y t ∥ = A t + 1 L 1 α t + 1 ∥ z t − y t ∥ ≤ A t + 1 4 α t + 1 c A t L 0
≤ 4 α t + 1 c L 0 A t + 1 ≤ c ≤ 1 4 2 , \begin{aligned}
\le4\alpha_{t+1}\sqrt{\dfrac{cL_0}{A_{t+1}}} \\
&\le\sqrt{c}\le\dfrac{1}{4\sqrt{2}},
\end{aligned} ≤ 4 α t + 1 A t + 1 c L 0 ≤ c ≤ 4 2 1 ,
where we used A t ≤ A t + 1 A_t\le A_{t+1} A t ≤ A t + 1 , A t + 1 ≥ 16 L 0 α t + 1 2 A_{t+1}\ge16L_0\alpha_{t+1}^2 A t + 1 ≥ 16 L 0 α t + 1 2 , and c ≤ 1 32 c\le\dfrac{1}{32} c ≤ 32 1 .
By Lemma 3, we obtain
L 0 + L 1 ∥ ∇ f ( x t + 1 ) ∥ ≤ e L 1 ∥ x t + 1 − y t ∥ ( L 0 + L 1 ∥ ∇ f ( y t ) ∥ ) ≤ e 1 / ( 4 2 ) 9 + 17 8 L 0 \begin{aligned}
L_0+L_1\lVert\nabla f(x_{t+1})\rVert\le e^{L_1\lVert x_{t+1}-y_t\rVert}\left(L_0+L_1\lVert\nabla f(y_t)\rVert\right) \\
&\le e^{1/(4\sqrt{2})}\dfrac{9+\sqrt{17}}{8}L_0
\end{aligned} L 0 + L 1 ∥ ∇ f ( x t + 1 )∥ ≤ e L 1 ∥ x t + 1 − y t ∥ ( L 0 + L 1 ∥ ∇ f ( y t )∥ ) ≤ e 1/ ( 4 2 ) 8 9 + 17 L 0
Therefore, ∥ ∇ f ( x t + 1 ) ∥ ≤ L 0 L 1 \lVert\nabla f(x_{t+1})\rVert\le\dfrac{L_0}{L_1} ∥ ∇ f ( x t + 1 )∥ ≤ L 1 L 0 .
Finally, for every t ≥ 0 t\ge0 t ≥ 0 , the update rule and clipping give
L 1 ∥ y t + 1 − x t + 1 ∥ = L 1 α t + 1 2 A t + 1 ∥ g t + 1 ∥ ≤ L 1 α t + 1 2 A t + 1 λ t + 1 ≤ 1 , L_1\lVert y_{t+1}-x_{t+1}\rVert=\dfrac{L_1\alpha_{t+1}^2}{A_{t+1}}\lVert g_{t+1}\rVert\le\dfrac{L_1\alpha_{t+1}^2}{A_{t+1}}\lambda_{t+1}\le1, L 1 ∥ y t + 1 − x t + 1 ∥ = A t + 1 L 1 α t + 1 2 ∥ g t + 1 ∥ ≤ A t + 1 L 1 α t + 1 2 λ t + 1 ≤ 1 ,
where we used λ t + 1 ≤ A t + 1 L 1 α t + 1 2 \lambda_{t+1}\le\dfrac{A_{t+1}}{L_1\alpha_{t+1}^2} λ t + 1 ≤ L 1 α t + 1 2 A t + 1 . Applying Lemma 1, we obtain
f ( y t + 1 ) ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 2 ( L 0 + L 1 ∥ ∇ f ( x t + 1 ) ∥ ) ∥ y t + 1 − x t + 1 ∥ 2 ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 4 L 0 ∥ y t + 1 − x t + 1 ∥ 2 . \begin{aligned}
f(y_{t+1})\le f(x_{t+1})+\langle\nabla f(x_{t+1}),y_{t+1}-x_{t+1}\rangle+2\left(L_0+L_1\lVert\nabla f(x_{t+1})\rVert\right)\lVert y_{t+1}-x_{t+1}\rVert^2 \\
&\le f(x_{t+1})+\langle\nabla f(x_{t+1}),y_{t+1}-x_{t+1}\rangle+4L_0\lVert y_{t+1}-x_{t+1}\rVert^2.
\end{aligned} f ( y t + 1 ) ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 2 ( L 0 + L 1 ∥ ∇ f ( x t + 1 )∥ ) ∥ y t + 1 − x t + 1 ∥ 2 ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 4 L 0 ∥ y t + 1 − x t + 1 ∥ 2 .
Proof of Lemma 6. By Lemma 2 and ν ≥ 1 / 2 \nu\ge1/2 ν ≥ 1/2 ,
f ( x ) − f ⋆ ≤ L 0 16 L 1 2 ⟹ ∥ ∇ f ( x ) ∥ ≤ 3 L 0 ( f ( x ) − f ⋆ ) . f(x)-f^\star\le\frac{L_0}{16L_1^2} \Longrightarrow\lVert\nabla f(x)\rVert\le3\sqrt{L_0\left(f(x)-f^\star\right)}. f ( x ) − f ⋆ ≤ 16 L 1 2 L 0 ⟹ ∥ ∇ f ( x )∥ ≤ 3 L 0 ( f ( x ) − f ⋆ ) .
If t = 0 t=0 t = 0 , then x 1 = x ˉ x_1=\bar{x} x 1 = x ˉ . By the preceding implication and the convexity bound f ( x ˉ ) − f ⋆ ≤ R ∥ ∇ f ( x ˉ ) ∥ f(\bar{x})-f^\star\le R\lVert\nabla f(\bar{x})\rVert f ( x ˉ ) − f ⋆ ≤ R ∥ ∇ f ( x ˉ )∥ , we obtain
∥ ∇ f ( x 1 ) ∥ ≤ min { 3 c L 0 L 1 , 9 L 0 R } . \lVert\nabla f(x_1)\rVert\le\min\left\{\frac{3\sqrt{c}L_0}{L_1},9L_0R\right\}. ∥ ∇ f ( x 1 )∥ ≤ min { L 1 3 c L 0 , 9 L 0 R } .
Since A 1 = A 0 + α 1 = α 1 A_1=A_0+\alpha_1=\alpha_1 A 1 = A 0 + α 1 = α 1 and A 1 ≥ 96 ℓ 2 L 0 α 1 2 A_1\ge96\ell^2L_0\alpha_1^2 A 1 ≥ 96 ℓ 2 L 0 α 1 2 , we have
α 1 ℓ L 0 A 1 ≤ 1 96 and α 1 ℓ L 0 ≤ 1 96 ℓ . \alpha_1\ell\sqrt{\frac{L_0}{A_1}}\le\frac{1}{\sqrt{96}}
\qquad\text{and}\qquad
\alpha_1\ell L_0\le\frac{1}{96\ell}. α 1 ℓ A 1 L 0 ≤ 96 1 and α 1 ℓ L 0 ≤ 96 ℓ 1 .
Therefore,
α 1 ℓ ∥ ∇ f ( x 1 ) ∥ ≤ min { 3 96 c A 1 L 0 L 1 , 9 96 ℓ R } ≤ 1 2 min { c A 1 L 0 L 1 , R } , \alpha_1\ell\lVert\nabla f(x_1)\rVert
\le\min\left\{\frac{3}{\sqrt{96}}\frac{\sqrt{cA_1L_0}}{L_1},\frac{9}{96\ell}R\right\}
\le\frac{1}{2}\min\left\{\frac{\sqrt{cA_1L_0}}{L_1},R\right\}, α 1 ℓ ∥ ∇ f ( x 1 )∥ ≤ min { 96 3 L 1 c A 1 L 0 , 96 ℓ 9 R } ≤ 2 1 min { L 1 c A 1 L 0 , R } ,
where we used ℓ ≥ 4 \ell\ge4 ℓ ≥ 4 . Hence ∥ ∇ f ( x 1 ) ∥ ≤ λ 1 / 2 \lVert\nabla f(x_1)\rVert\le\lambda_1/2 ∥ ∇ f ( x 1 )∥ ≤ λ 1 /2 .
Now suppose t ≥ 1 t\ge1 t ≥ 1 . Since A i ≥ 96 ℓ 2 L 0 α i 2 ≥ 16 L 0 α i 2 A_i\ge96\ell^2L_0\alpha_i^2\ge16L_0\alpha_i^2 A i ≥ 96 ℓ 2 L 0 α i 2 ≥ 16 L 0 α i 2 , applying Lemma 4 with u = x ˉ u=\bar{x} u = x ˉ and u = x ⋆ u=x^\star u = x ⋆ , we obtain
f ( y t ) − f ⋆ ≤ min { 2 c L 0 L 1 2 , R 2 A t } . f(y_t)-f^\star\le\min\left\{\frac{2cL_0}{L_1^2},\frac{R^2}{A_t}\right\}. f ( y t ) − f ⋆ ≤ min { L 1 2 2 c L 0 , A t R 2 } .
Similarly to the proof of Lemma 5 , we obtain, for every j = 1 , … , t j=1,\ldots,t j = 1 , … , t ,
{ ∥ z j − x ˉ ∥ ≤ 2 c A j L 0 L 1 , ∥ z j − x ⋆ ∥ ≤ 2 R . \begin{cases}
\lVert z_j-\bar{x}\rVert\le\dfrac{2\sqrt{cA_jL_0}}{L_1},\\
\lVert z_j-x^\star\rVert\le\sqrt{2}R.
\end{cases} ⎩ ⎨ ⎧ ∥ z j − x ˉ ∥ ≤ L 1 2 c A j L 0 , ∥ z j − x ⋆ ∥ ≤ 2 R .
Since y t = A t − 1 ∑ j = 1 t α j z j y_t=A_t^{-1}\sum_{j=1}^t\alpha_jz_j y t = A t − 1 ∑ j = 1 t α j z j , it follows that
∥ z t − y t ∥ ≤ min { 4 c A t L 0 L 1 , 2 2 R } . \lVert z_t-y_t\rVert\le\min\left\{\frac{4\sqrt{cA_tL_0}}{L_1},2\sqrt{2}R\right\}. ∥ z t − y t ∥ ≤ min { L 1 4 c A t L 0 , 2 2 R } .
As in the proof of Lemma 5 , we obtain ∥ ∇ f ( y t ) ∥ ≤ 3 L 0 4 L 1 \lVert\nabla f(y_t)\rVert\le\frac{3L_0}{4L_1} ∥ ∇ f ( y t )∥ ≤ 4 L 1 3 L 0 . Moreover,
L 1 ∥ x t + 1 − y t ∥ = L 1 α t + 1 A t + 1 ∥ z t − y t ∥ ≤ 4 α t + 1 c A t L 0 A t + 1 ≤ 4 c α t + 1 L 0 A t + 1 ≤ 4 c 96 ℓ ≤ 1 8 , \begin{aligned}
L_1\lVert x_{t+1}-y_t\rVert
&=\frac{L_1\alpha_{t+1}}{A_{t+1}}\lVert z_t-y_t\rVert\\
&\le\frac{4\alpha_{t+1}\sqrt{cA_tL_0}}{A_{t+1}}\\
&\le4\sqrt{c}\,\alpha_{t+1}\sqrt{\frac{L_0}{A_{t+1}}}\\
&\le\frac{4\sqrt{c}}{\sqrt{96}\ell}\le\frac{1}{8},
\end{aligned} L 1 ∥ x t + 1 − y t ∥ = A t + 1 L 1 α t + 1 ∥ z t − y t ∥ ≤ A t + 1 4 α t + 1 c A t L 0 ≤ 4 c α t + 1 A t + 1 L 0 ≤ 96 ℓ 4 c ≤ 8 1 ,
where we used A t ≤ A t + 1 A_t\le A_{t+1} A t ≤ A t + 1 , A t + 1 ≥ 96 ℓ 2 L 0 α t + 1 2 A_{t+1}\ge96\ell^2L_0\alpha_{t+1}^2 A t + 1 ≥ 96 ℓ 2 L 0 α t + 1 2 , c ≤ 1 / 32 c\le1/32 c ≤ 1/32 , and ℓ ≥ 4 \ell\ge4 ℓ ≥ 4 . By Lemma 3, every v ∈ [ y t , x t + 1 ] v \in[y_t,x_{t+1}] v ∈ [ y t , x t + 1 ] therefore satisfies L 0 + L 1 ∥ ∇ f ( v ) ∥ ≤ 7 4 e 1 / 8 L 0 ≤ 2 L 0 L_0+L_1\|\nabla f(v)\| \le\frac{7}{4}e^{1/8}L_0 \le2L_0 L 0 + L 1 ∥∇ f ( v ) ∥ ≤ 4 7 e 1/8 L 0 ≤ 2 L 0 . Hence, by Assumption 2,
∥ ∇ f ( x t + 1 ) ∥ ≤ ∥ ∇ f ( y t ) ∥ + ∥ ∇ f ( x t + 1 ) − ∇ f ( y t ) ∥ ≤ ∥ ∇ f ( y t ) ∥ + ( L 0 + L 1 sup v ∈ [ y t , x t + 1 ] ∥ ∇ f ( v ) ∥ ) ∥ x t + 1 − y t ∥ ≤ ∥ ∇ f ( y t ) ∥ + 2 L 0 ∥ x t + 1 − y t ∥ ≤ 3 L 0 ( f ( y t ) − f ⋆ ) + 2 L 0 α t + 1 A t + 1 ∥ z t − y t ∥ . \begin{aligned}
\|\nabla f(x_{t+1})\| &\le\|\nabla f(y_t)\|+\|\nabla f(x_{t+1})-\nabla f(y_t)\| \\
&\le\|\nabla f(y_t)\|+\left(L_0+L_1\sup_{v\in[y_t,x_{t+1}]}\|\nabla f(v)\|\right)\|x_{t+1}-y_t\| \\
&\le\|\nabla f(y_t)\|+2L_0\|x_{t+1}-y_t\| \\
&\le3\sqrt{L_0(f(y_t)-f^\star)}+\frac{2L_0\alpha_{t+1}}{A_{t+1}}\|z_t-y_t\|.
\end{aligned} ∥∇ f ( x t + 1 ) ∥ ≤ ∥∇ f ( y t ) ∥ + ∥∇ f ( x t + 1 ) − ∇ f ( y t ) ∥ ≤ ∥∇ f ( y t ) ∥ + ( L 0 + L 1 v ∈ [ y t , x t + 1 ] sup ∥∇ f ( v ) ∥ ) ∥ x t + 1 − y t ∥ ≤ ∥∇ f ( y t ) ∥ + 2 L 0 ∥ x t + 1 − y t ∥ ≤ 3 L 0 ( f ( y t ) − f ⋆ ) + A t + 1 2 L 0 α t + 1 ∥ z t − y t ∥.
Substituting the preceding bounds, we obtain
{ ∥ ∇ f ( x t + 1 ) ∥ ≤ ( 3 2 + 8 96 ℓ ) c L 0 L 1 , ∥ ∇ f ( x t + 1 ) ∥ ≤ ( 3 5 2 + 4 2 96 ℓ ) R L 0 A t + 1 . \begin{cases}
\|\nabla f(x_{t+1})\|\le\left(3\sqrt{2}+\frac{8}{\sqrt{96\ell}}\right)\frac{\sqrt{c}L_0}{L_1},\\
\|\nabla f(x_{t+1})\|\le\left(3\sqrt{\frac{5}{2}}+\frac{4\sqrt{2}}{\sqrt{96\ell}}\right)R\sqrt{\frac{L_0}{A_{t+1}}}.
\end{cases} ⎩ ⎨ ⎧ ∥∇ f ( x t + 1 ) ∥ ≤ ( 3 2 + 96 ℓ 8 ) L 1 c L 0 , ∥∇ f ( x t + 1 ) ∥ ≤ ( 3 2 5 + 96 ℓ 4 2 ) R A t + 1 L 0 .
where we used A t ≤ A t + 1 ≤ 5 2 A t A_t\le A_{t+1}\le\frac{5}{2}A_t A t ≤ A t + 1 ≤ 2 5 A t and A t + 1 ≥ 96 ℓ 2 L 0 α t + 1 2 A_{t+1}\ge96\ell^2L_0\alpha_{t+1}^2 A t + 1 ≥ 96 ℓ 2 L 0 α t + 1 2 .
We have λ t + 1 : = 1 α t + 1 ℓ min { c A t + 1 + L 0 L 1 , R } \lambda_{t+1}:=\frac{1}{\alpha_{t+1}\ell}\min\left\{\frac{\sqrt{cA_{t+1}+L_0}}{L_1},R\right\} λ t + 1 := α t + 1 ℓ 1 min { L 1 c A t + 1 + L 0 , R } . Moreover, A t + 1 ≥ 96 ℓ 2 L 0 α t + 1 2 A_{t+1}\ge96\ell^2L_0\alpha_{t+1}^2 A t + 1 ≥ 96 ℓ 2 L 0 α t + 1 2 implies
α t + 1 ℓ L 0 A t + 1 ≤ 1 96 . \alpha_{t+1}\ell\sqrt{\frac{L_0}{A_{t+1}}}\le\frac{1}{\sqrt{96}}. α t + 1 ℓ A t + 1 L 0 ≤ 96 1 .
Multiplying the two gradient bounds above by α t + 1 ℓ \alpha_{t+1}\ell α t + 1 ℓ , we obtain
{ α t + 1 ℓ ∥ ∇ f ( x t + 1 ) ∥ ≤ ( 3 4 + 1 12 ℓ ) c A t + 1 L 0 L 1 ≤ ( 3 4 + 1 48 ) c A t + 1 L 0 L 1 < 1 2 c A t + 1 L 0 L 1 , α t + 1 ℓ ∥ ∇ f ( x t + 1 ) ∥ ≤ ( 15 8 + 2 24 ℓ ) R ≤ ( 15 8 + 2 96 ) R < 1 2 R . \begin{cases}
\alpha_{t+1}\ell\|\nabla f(x_{t+1})\|\le\left(\frac{\sqrt{3}}{4}+\frac{1}{12\ell}\right)\frac{\sqrt{cA_{t+1}L_0}}{L_1}\le\left(\frac{\sqrt{3}}{4}+\frac{1}{48}\right)\frac{\sqrt{cA_{t+1}L_0}}{L_1}<\frac{1}{2}\frac{\sqrt{cA_{t+1}L_0}}{L_1},\\
\alpha_{t+1}\ell\|\nabla f(x_{t+1})\|\le\left(\frac{\sqrt{15}}{8}+\frac{\sqrt{2}}{24\ell}\right)R\le\left(\frac{\sqrt{15}}{8}+\frac{\sqrt{2}}{96}\right)R<\frac{1}{2}R.
\end{cases} ⎩ ⎨ ⎧ α t + 1 ℓ ∥∇ f ( x t + 1 ) ∥ ≤ ( 4 3 + 12 ℓ 1 ) L 1 c A t + 1 L 0 ≤ ( 4 3 + 48 1 ) L 1 c A t + 1 L 0 < 2 1 L 1 c A t + 1 L 0 , α t + 1 ℓ ∥∇ f ( x t + 1 ) ∥ ≤ ( 8 15 + 24 ℓ 2 ) R ≤ ( 8 15 + 96 2 ) R < 2 1 R .
Therefore
∥ ∇ f ( x t + 1 ) ∥ ≤ 1 2 α t + 1 ℓ min { c A t + 1 L 0 L 1 , R } = λ t + 1 2 . \|\nabla f(x_{t+1})\|\le\frac{1}{2\alpha_{t+1}\ell}\min\left\{\frac{\sqrt{cA_{t+1}L_0}}{L_1},R\right\}=\frac{\lambda_{t+1}}{2}. ∥∇ f ( x t + 1 ) ∥ ≤ 2 α t + 1 ℓ 1 min { L 1 c A t + 1 L 0 , R } = 2 λ t + 1 .
Proof of Lemma 7. For t = 1 , … , K t=1,\ldots,K t = 1 , … , K , let S t : = σ α ∑ i = 1 t α i s i α i − 1 S_t:=\sigma^\alpha\sum_{i=1}^{t}\frac{\alpha_i}{s_i\alpha_{i-1}} S t := σ α ∑ i = 1 t s i α i − 1 α i . By assumption, S t ≤ S K ≤ 1 / 512 S_t\le S_K\le1/512 S t ≤ S K ≤ 1/512 .
For i ≥ 2 i\ge2 i ≥ 2 , on E i − 1 \mathcal{E}_{i-1} E i − 1 , Lemma 4 gives
{ ∥ x ˉ − z i − 1 ∥ ≤ 2 c A i L 0 L 1 ≤ 2 c A i 0 L 0 L 1 , ∥ x ⋆ − z i − 1 ∥ ≤ 2 R . \begin{cases}
\|\bar{x}-z_{i-1}\|\le2\frac{\sqrt{cA_iL_0}}{L_1}\le2\frac{\sqrt{cA_i^0L_0}}{L_1},\\
\|x^\star-z_{i-1}\|\le\sqrt{2}R.
\end{cases} { ∥ x ˉ − z i − 1 ∥ ≤ 2 L 1 c A i L 0 ≤ 2 L 1 c A i 0 L 0 , ∥ x ⋆ − z i − 1 ∥ ≤ 2 R .
We also have ∥ x ˉ − z i − 1 ∥ ≤ ∥ x ˉ − x ⋆ ∥ + ∥ x ⋆ − z i − 1 ∥ ≤ ( 1 + 2 ) R \|\bar{x}-z_{i-1}\|\le\|\bar{x}-x^\star\|+\|x^\star-z_{i-1}\|\le(1+\sqrt{2})R ∥ x ˉ − z i − 1 ∥ ≤ ∥ x ˉ − x ⋆ ∥ + ∥ x ⋆ − z i − 1 ∥ ≤ ( 1 + 2 ) R .
By the definition of s i s_i s i , for every i i i , we have
{ ∥ x ˉ − z i − 1 ∥ ≤ ( 1 + 2 ) s i ≤ 5 2 s i , ∥ x ⋆ − z i − 1 ∥ ≤ 2 R . \begin{cases}
\|\bar{x}-z_{i-1}\|\le(1+\sqrt{2})s_i\le\frac{5}{2}s_i,\\
\|x^\star-z_{i-1}\|\le\sqrt{2}R.
\end{cases} { ∥ x ˉ − z i − 1 ∥ ≤ ( 1 + 2 ) s i ≤ 2 5 s i , ∥ x ⋆ − z i − 1 ∥ ≤ 2 R .
The same bounds hold for i = 1 i=1 i = 1 since z 0 = x ˉ z_0=\bar{x} z 0 = x ˉ .
We have
B t ( u ) ≤ ∑ i = 1 t 1 E i − 1 ( 2 σ α α i λ i α − 1 ∥ u − z i − 1 ∥ + 2 σ α α i 2 λ i 2 − α ) . B_t(u) \le\sum_{i=1}^{t} \mathbb{1}_{\mathcal{E}_{i-1}}\left(\frac{2\sigma^\alpha\alpha_i}{\lambda_i^{\alpha-1}}\lVert u-z_{i-1}\rVert+2\sigma^\alpha\alpha_i^2\lambda_i^{2-\alpha}\right). B t ( u ) ≤ i = 1 ∑ t 1 E i − 1 ( λ i α − 1 2 σ α α i ∥ u − z i − 1 ∥ + 2 σ α α i 2 λ i 2 − α ) .
Using α i λ i = s i / ℓ \alpha_i\lambda_i=s_i/\ell α i λ i = s i / ℓ , we have σ α α i 2 λ i 2 − α = σ α α i λ i α − 1 s i ℓ \sigma^\alpha\alpha_i^2\lambda_i^{2-\alpha}=\frac{\sigma^\alpha\alpha_i}{\lambda_i^{\alpha-1}}\frac{s_i}{\ell} σ α α i 2 λ i 2 − α = λ i α − 1 σ α α i ℓ s i . Hence, since 2 α ≤ 4 2^\alpha\le4 2 α ≤ 4 ,
B t ( x ˉ ) ≤ ( 10 + 2 ℓ ) ∑ i = 1 t σ α α i s i λ i α − 1 s i 2 . B_t(\bar{x})\le\left(10+\frac{2}{\ell}\right)\sum_{i=1}^{t}\frac{\sigma^\alpha\alpha_i}{s_i\lambda_i^{\alpha-1}}s_i^2. B t ( x ˉ ) ≤ ( 10 + ℓ 2 ) i = 1 ∑ t s i λ i α − 1 σ α α i s i 2 .
Since s i 2 ≤ c A i L 0 / L 1 2 ≤ c A t L 0 / L 1 2 s_i^2\le cA_iL_0/L_1^2\le cA_tL_0/L_1^2 s i 2 ≤ c A i L 0 / L 1 2 ≤ c A t L 0 / L 1 2 ,
B t ( x ˉ ) ≤ ( 10 + 2 ℓ ) c A t L 0 L 1 2 S t < 1 32 c A t L 0 L 1 2 , B_t(\bar{x})\le\left(10+\frac{2}{\ell}\right)\frac{cA_tL_0}{L_1^2}S_t<\frac{1}{32}\frac{cA_tL_0}{L_1^2}, B t ( x ˉ ) ≤ ( 10 + ℓ 2 ) L 1 2 c A t L 0 S t < 32 1 L 1 2 c A t L 0 ,
where we used ℓ ≥ 4 \ell\ge4 ℓ ≥ 4 and S t ≤ 1 / 512 S_t\le1/512 S t ≤ 1/512 .
Similarly,
B t ( x ⋆ ) ≤ ( 4 2 + 2 ℓ ) R 2 S t < R 2 64 . B_t(x^\star)\le\left(4\sqrt{2}+\frac{2}{\ell}\right)R^2S_t<\frac{R^2}{64}. B t ( x ⋆ ) ≤ ( 4 2 + ℓ 2 ) R 2 S t < 64 R 2 .
Recall that θ i : = g i − ∇ f ( x i ) \theta_i:=g_i-\nabla f(x_i) θ i := g i − ∇ f ( x i ) , where G i : = ∇ f ( x i , ξ i ) G_i:=\nabla f(x_i,\xi_i) G i := ∇ f ( x i , ξ i ) and g i : = clip ( G i , λ i ) g_i:=\operatorname{clip}(G_i,\lambda_i) g i := clip ( G i , λ i ) . We now control the martingale term M t ( u ) M_t(u) M t ( u ) . Its i i i -th summand is
1 E i − 1 ( α i ⟨ θ i − E [ θ i ∣ F i − 1 ] , u − z i − 1 ⟩ + α i 2 ( ∥ θ i ∥ 2 − E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ) ) . \mathbb{1}_{\mathcal{E}_{i-1}}\left(\alpha_i\langle\theta_i-\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}],u-z_{i-1}\rangle+\alpha_i^2\left(\lVert\theta_i\rVert^2-\mathbb{E}[\lVert\theta_i\rVert^2\mid\mathcal{F}_{i-1}]\right)\right). 1 E i − 1 ( α i ⟨ θ i − E [ θ i ∣ F i − 1 ] , u − z i − 1 ⟩ + α i 2 ( ∥ θ i ∥ 2 − E [∥ θ i ∥ 2 ∣ F i − 1 ] ) ) .
On E i − 1 \mathcal{E}_{i-1} E i − 1 , we have ∥ θ i ∥ ≤ 3 2 λ i \lVert\theta_i\rVert\le\frac{3}{2}\lambda_i ∥ θ i ∥ ≤ 2 3 λ i , and therefore
∥ θ i − E [ θ i ∣ F i − 1 ] ∥ ≤ 3 λ i , \lVert\theta_i-\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}]\rVert\le3\lambda_i, ∥ θ i − E [ θ i ∣ F i − 1 ]∥ ≤ 3 λ i ,
and
∣ ∥ θ i ∥ 2 − E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ∣ ≤ 9 4 λ i 2 . \left\lvert\lVert\theta_i\rVert^2-\mathbb{E}[\lVert\theta_i\rVert^2\mid\mathcal{F}_{i-1}]\right\rvert\le\frac{9}{4}\lambda_i^2. ∥ θ i ∥ 2 − E [∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 4 9 λ i 2 .
Hence, for u = x ˉ u=\bar{x} u = x ˉ , the absolute value of the i i i -th summand of M t ( x ˉ ) M_t(\bar{x}) M t ( x ˉ ) is bounded by
3 α i λ i ∥ x ˉ − z i − 1 ∥ + 9 4 α i 2 λ i 2 ≤ 15 2 α i λ i s i + 9 4 α i 2 λ i 2 . 3\alpha_i\lambda_i\lVert\bar{x}-z_{i-1}\rVert+\frac{9}{4}\alpha_i^2\lambda_i^2\le\frac{15}{2}\alpha_i\lambda_i s_i+\frac{9}{4}\alpha_i^2\lambda_i^2. 3 α i λ i ∥ x ˉ − z i − 1 ∥ + 4 9 α i 2 λ i 2 ≤ 2 15 α i λ i s i + 4 9 α i 2 λ i 2 .
Using α i λ i = s i / ℓ \alpha_i\lambda_i=s_i/\ell α i λ i = s i / ℓ , for every i ≤ t i\le t i ≤ t ,
15 2 α i λ i s i + 9 4 α i 2 λ i 2 = ( 15 2 ℓ + 9 4 ℓ 2 ) s i 2 ≤ 9 ℓ c A t L 0 L 1 2 . \frac{15}{2}\alpha_i\lambda_i s_i+\frac{9}{4}\alpha_i^2\lambda_i^2=\left(\frac{15}{2\ell}+\frac{9}{4\ell^2}\right)s_i^2\le\frac{9}{\ell}\frac{cA_tL_0}{L_1^2}. 2 15 α i λ i s i + 4 9 α i 2 λ i 2 = ( 2 ℓ 15 + 4 ℓ 2 9 ) s i 2 ≤ ℓ 9 L 1 2 c A t L 0 .
Similarly, for u = x ⋆ u=x^\star u = x ⋆ , the absolute value of the i i i -th summand is bounded by
3 2 α i λ i R + 9 4 α i 2 λ i 2 = 3 2 ℓ s i R + 9 4 ℓ 2 s i 2 ≤ 5 R 2 ℓ . 3\sqrt{2}\alpha_i\lambda_iR+\frac{9}{4}\alpha_i^2\lambda_i^2=\frac{3\sqrt{2}}{\ell}s_iR+\frac{9}{4\ell^2}s_i^2\le\frac{5R^2}{\ell}. 3 2 α i λ i R + 4 9 α i 2 λ i 2 = ℓ 3 2 s i R + 4 ℓ 2 9 s i 2 ≤ ℓ 5 R 2 .
We next bound the conditional variances. Since
E [ ∥ θ i − E [ θ i ∣ F i − 1 ] ∥ 2 ∣ F i − 1 ] ≤ E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 2 σ α λ i 2 − α , E [ ( ∥ θ i ∥ 2 − E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ) 2 ∣ F i − 1 ] ≤ E [ ∥ θ i ∥ 4 ∣ F i − 1 ] ≤ 9 2 σ α λ i 4 − α , \begin{aligned}
\mathbb{E}\left[\left\lVert\theta_i-\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}]\right\rVert^2\mid\mathcal{F}_{i-1}\right]&\le\mathbb{E}[\lVert\theta_i\rVert^2\mid\mathcal{F}_{i-1}]\le2\sigma^\alpha\lambda_i^{2-\alpha},\\
\mathbb{E}\left[\left(\lVert\theta_i\rVert^2-\mathbb{E}[\lVert\theta_i\rVert^2\mid\mathcal{F}_{i-1}]\right)^2\mid\mathcal{F}_{i-1}\right]&\le\mathbb{E}[\lVert\theta_i\rVert^4\mid\mathcal{F}_{i-1}]\le\frac{9}{2}\sigma^\alpha\lambda_i^{4-\alpha},
\end{aligned} E [ ∥ θ i − E [ θ i ∣ F i − 1 ] ∥ 2 ∣ F i − 1 ] E [ ( ∥ θ i ∥ 2 − E [∥ θ i ∥ 2 ∣ F i − 1 ] ) 2 ∣ F i − 1 ] ≤ E [∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 2 σ α λ i 2 − α , ≤ E [∥ θ i ∥ 4 ∣ F i − 1 ] ≤ 2 9 σ α λ i 4 − α ,
using ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^2 \le2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2 , the conditional variance of the i i i -th summand of M t ( u ) M_t(u) M t ( u ) is at most
4 σ α α i 2 λ i 2 − α ∥ u − z i − 1 ∥ 2 + 9 σ α α i 4 λ i 4 − α . 4\sigma^\alpha\alpha_i^2\lambda_i^{2-\alpha}\lVert u-z_{i-1}\rVert^2+9\sigma^\alpha\alpha_i^4\lambda_i^{4-\alpha}. 4 σ α α i 2 λ i 2 − α ∥ u − z i − 1 ∥ 2 + 9 σ α α i 4 λ i 4 − α .
Using again α i λ i = s i / ℓ \alpha_i\lambda_i=s_i/\ell α i λ i = s i / ℓ , we have
σ α α i 2 λ i 2 − α = σ α α i s i λ i α − 1 s i 2 ℓ , \sigma^\alpha\alpha_i^2\lambda_i^{2-\alpha}=\frac{\sigma^\alpha\alpha_i}{s_i\lambda_i^{\alpha-1}}\frac{s_i^2}{\ell}, σ α α i 2 λ i 2 − α = s i λ i α − 1 σ α α i ℓ s i 2 ,
and
σ α α i 4 λ i 4 − α = σ α α i s i λ i α − 1 s i 4 ℓ 3 . \sigma^\alpha\alpha_i^4\lambda_i^{4-\alpha}=\frac{\sigma^\alpha\alpha_i}{s_i\lambda_i^{\alpha-1}}\frac{s_i^4}{\ell^3}. σ α α i 4 λ i 4 − α = s i λ i α − 1 σ α α i ℓ 3 s i 4 .
Therefore,
∑ i = 1 t E [ [ i -th summand of M t ( x ˉ ) ) 2 ∣ F i − 1 ] ≤ ( 25 ℓ + 9 ℓ 3 ) ( c A t L 0 L 1 2 ) 2 S t ≤ 26 512 ℓ ( c A t L 0 L 1 2 ) 2 , \sum_{i=1}^{t}\mathbb{E}\left[\left[\mathit{i}\text{-th summand of }M_t(\bar{x})\right)^2\mid\mathcal{F}_{i-1}\right]\le\left(\frac{25}{\ell}+\frac{9}{\ell^3}\right)\left(\frac{cA_tL_0}{L_1^2}\right)^2S_t\le\frac{26}{512\ell}\left(\frac{cA_tL_0}{L_1^2}\right)^2, i = 1 ∑ t E [ [ i -th summand of M t ( x ˉ ) ) 2 ∣ F i − 1 ] ≤ ( ℓ 25 + ℓ 3 9 ) ( L 1 2 c A t L 0 ) 2 S t ≤ 512 ℓ 26 ( L 1 2 c A t L 0 ) 2 ,
where we used ℓ ≥ 4 \ell\ge4 ℓ ≥ 4 and S t ≤ 1 / 512 S_t\le1/512 S t ≤ 1/512 .
Similarly, we obtain
∑ i = 1 t E [ [ i -th summand of M t ( x ∗ ) ) 2 ∣ F i − 1 ] ≤ ( 8 ℓ + 9 ℓ 3 ) R 4 S t ≤ 9 R 4 512 ℓ . \sum_{i=1}^{t}\mathbb{E}\left[\left[\mathit{i}\text{-th summand of }M_t(x^*)\right)^2\mid\mathcal{F}_{i-1}\right]\le\left(\frac{8}{\ell}+\frac{9}{\ell^3}\right)R^4S_t\le\frac{9R^4}{512\ell}. i = 1 ∑ t E [ [ i -th summand of M t ( x ∗ ) ) 2 ∣ F i − 1 ] ≤ ( ℓ 8 + ℓ 3 9 ) R 4 S t ≤ 512 ℓ 9 R 4 .
Fix t ∈ { 1 , … , K } t\in\{1,\ldots,K\} t ∈ { 1 , … , K } . For u = x ˉ u=\bar{x} u = x ˉ , the martingale differences are bounded in absolute value by 9 ℓ c A t L 0 L 1 2 \frac{9}{\ell}\frac{cA_tL_0}{L_1^2} ℓ 9 L 1 2 c A t L 0 , while their conditional variances sum to at most 26 512 ℓ ( c A t L 0 L 1 2 ) 2 \frac{26}{512\ell}\left(\frac{cA_tL_0}{L_1^2}\right)^2 512 ℓ 26 ( L 1 2 c A t L 0 ) 2 . Hence, by Proposition 2, with probability at least 1 − δ / ( 2 K ) 1-\delta/(2K) 1 − δ / ( 2 K ) ,
∣ M t ( x ˉ ) ∣ ≤ c A t L 0 L 1 2 ( 52 512 ℓ log 4 K δ + 6 ℓ log 4 K δ ) . |M_t(\bar{x})|\le\frac{cA_tL_0}{L_1^2}\left(\sqrt{\frac{52}{512\ell}\log\frac{4K}{\delta}}+\frac{6}{\ell}\log\frac{4K}{\delta}\right). ∣ M t ( x ˉ ) ∣ ≤ L 1 2 c A t L 0 ( 512 ℓ 52 log δ 4 K + ℓ 6 log δ 4 K ) .
Since ℓ ≥ 8 log ( 4 K / δ ) \ell\ge8\log(4K/\delta) ℓ ≥ 8 log ( 4 K / δ ) , the term in parentheses is at most 13 32 + 3 4 < 7 8 \frac{\sqrt{13}}{32}+\frac{3}{4}<\frac{7}{8} 32 13 + 4 3 < 8 7 . Thus
∣ M t ( x ˉ ) ∣ < 7 8 c A t L 0 L 1 2 . |M_t(\bar{x})|<\frac{7}{8}\frac{cA_tL_0}{L_1^2}. ∣ M t ( x ˉ ) ∣ < 8 7 L 1 2 c A t L 0 .
Combining this with B t ( x ˉ ) < 1 32 c A t L 0 L 1 2 B_t(\bar{x})<\frac{1}{32}\frac{cA_tL_0}{L_1^2} B t ( x ˉ ) < 32 1 L 1 2 c A t L 0 , we obtain
R ^ t ( x ˉ ) = M t ( x ˉ ) + B t ( x ˉ ) < ( 7 8 + 1 32 ) c A t L 0 L 1 2 < c A t L 0 L 1 2 . \widehat{R}_t(\bar{x})=M_t(\bar{x})+B_t(\bar{x})<\left(\frac{7}{8}+\frac{1}{32}\right)\frac{cA_tL_0}{L_1^2}<\frac{cA_tL_0}{L_1^2}. R t ( x ˉ ) = M t ( x ˉ ) + B t ( x ˉ ) < ( 8 7 + 32 1 ) L 1 2 c A t L 0 < L 1 2 c A t L 0 .
Similarly, for u = x ∗ u=x^* u = x ∗ , the martingale differences are bounded in absolute value by 5 R 2 / ℓ 5R^2/\ell 5 R 2 / ℓ , while their conditional variances sum to at most 9 R 4 / ( 512 ℓ ) 9R^4/(512\ell) 9 R 4 / ( 512 ℓ ) . Therefore, with probability at least 1 − δ / ( 2 K ) 1-\delta/(2K) 1 − δ / ( 2 K ) ,
∣ M t ( x ∗ ) ∣ ≤ R 2 ( 18 512 ℓ log 4 K δ + 10 3 ℓ log 4 K δ ) . |M_t(x^*)|\le R^2\left(\sqrt{\frac{18}{512\ell}\log\frac{4K}{\delta}}+\frac{10}{3\ell}\log\frac{4K}{\delta}\right). ∣ M t ( x ∗ ) ∣ ≤ R 2 ( 512 ℓ 18 log δ 4 K + 3 ℓ 10 log δ 4 K ) .
Since ℓ ≥ 8 log ( 4 K / δ ) \ell\ge8\log(4K/\delta) ℓ ≥ 8 log ( 4 K / δ ) , the term in parentheses is at most 3 16 8 + 5 12 < 29 60 \frac{3}{16\sqrt{8}}+\frac{5}{12}<\frac{29}{60} 16 8 3 + 12 5 < 60 29 . Hence,
R ^ t ( x ∗ ) ≤ ∣ M t ( x ∗ ) ∣ + B t ( x ∗ ) < ( 29 60 + 1 64 ) R 2 < R 2 2 . \widehat{R}_t(x^*)\le|M_t(x^*)|+B_t(x^*)<\left(\frac{29}{60}+\frac{1}{64}\right)R^2<\frac{R^2}{2}. R t ( x ∗ ) ≤ ∣ M t ( x ∗ ) ∣ + B t ( x ∗ ) < ( 60 29 + 64 1 ) R 2 < 2 R 2 .
Taking a union bound over t = 1 , … , K t=1,\ldots,K t = 1 , … , K and u ∈ { x ˉ , x ⋆ } u\in\{\bar{x},x^\star\} u ∈ { x ˉ , x ⋆ } , with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every t = 1 , … , K t=1,\ldots,K t = 1 , … , K ,
{ R ^ t ( x ˉ ) ≤ c A t L 0 L 1 2 , R ^ t ( x ⋆ ) ≤ R 2 2 . \left\{
\begin{aligned}
\widehat{R}_t(\bar{x}) &\le\frac{cA_tL_0}{L_1^2},\\
\widehat{R}_t(x^\star) &\le\frac{R^2}{2}.
\end{aligned}
\right. ⎩ ⎨ ⎧ R t ( x ˉ ) R t ( x ⋆ ) ≤ L 1 2 c A t L 0 , ≤ 2 R 2 .
It remains to show that E K \mathcal{E}_K E K holds on this event. We argue by induction. For t = 0 t=0 t = 0 , there are no conditions, so 1 E 0 = 1 \mathbf{1}_{\mathcal{E}_0}=1 1 E 0 = 1 . Suppose that E t − 1 \mathcal{E}_{t-1} E t − 1 holds for some t ∈ { 1 , … , K } t\in\{1,\ldots,K\} t ∈ { 1 , … , K } . Since A t ≥ 96 ℓ 2 L 0 α t 2 A_t\ge96\ell^2L_0\alpha_t^2 A t ≥ 96 ℓ 2 L 0 α t 2 , we have A t ≥ 16 L 0 α t 2 A_t\ge16L_0\alpha_t^2 A t ≥ 16 L 0 α t 2 . Moreover, by the definition of λ t \lambda_t λ t ,
λ t ≤ c A t L 0 α t ℓ L 1 ≤ A t L 1 α t 2 . \lambda_t\le\frac{\sqrt{cA_tL_0}}{\alpha_t\ell L_1}\le\frac{A_t}{L_1\alpha_t^2}. λ t ≤ α t ℓ L 1 c A t L 0 ≤ L 1 α t 2 A t .
where the last inequality follows from A t ≥ 96 ℓ 2 L 0 α t 2 A_t\ge96\ell^2L_0\alpha_t^2 A t ≥ 96 ℓ 2 L 0 α t 2 . Hence, applying Lemma 5 with t − 1 t-1 t − 1 , we obtain
f ( y t ) ≤ f ( x t ) + ⟨ ∇ f ( x t ) , y t − x t ⟩ + 4 L 0 ∥ y t − x t ∥ 2 . f(y_t)\le f(x_t)+\langle\nabla f(x_t),y_t-x_t\rangle+4L_0\lVert y_t-x_t\rVert^2. f ( y t ) ≤ f ( x t ) + ⟨ ∇ f ( x t ) , y t − x t ⟩ + 4 L 0 ∥ y t − x t ∥ 2 .
Since E t − 1 \mathcal{E}_{t-1} E t − 1 holds, R ^ t ( u ) = R t ( u ) \widehat{R}_t(u)=R_t(u) R t ( u ) = R t ( u ) for u ∈ { x ˉ , x ⋆ } u\in\{\bar{x},x^\star\} u ∈ { x ˉ , x ⋆ } . Therefore,
{ R t ( x ˉ ) ≤ c A t L 0 L 1 2 , R t ( x ⋆ ) ≤ R 2 2 . \left\{
\begin{aligned}
R_t(\bar{x}) &\le\frac{cA_tL_0}{L_1^2},\\
R_t(x^\star) &\le\frac{R^2}{2}.
\end{aligned}
\right. ⎩ ⎨ ⎧ R t ( x ˉ ) R t ( x ⋆ ) ≤ L 1 2 c A t L 0 , ≤ 2 R 2 .
Thus all the conditions defining E t \mathcal{E}_t E t hold. By induction, E K \mathcal{E}_K E K holds.
Proof of Theorem 1. Let s i : = min { c A i L 0 / L 1 , R } s_i:=\min\{\sqrt{cA_iL_0}/L_1,R\} s i := min { c A i L 0 / L 1 , R } . We have λ i = s i / ( α i ℓ ) \lambda_i=s_i/(\alpha_i\ell) λ i = s i / ( α i ℓ ) , then
σ α ∑ i = 1 K α i s i λ i α − 1 = σ α α α − 1 ∑ i = 1 K ( α i s i ) α . \sigma^\alpha\sum_{i=1}^{K}\frac{\alpha_i}{s_i\lambda_i^{\alpha-1}}
=\sigma^\alpha\alpha^{\alpha-1}\sum_{i=1}^{K}\left(\frac{\alpha_i}{s_i}\right)^\alpha. σ α i = 1 ∑ K s i λ i α − 1 α i = σ α α α − 1 i = 1 ∑ K ( s i α i ) α .
Since s i = min { c A i L 0 / L 1 , R } s_i=\min\{\sqrt{cA_iL_0}/L_1,R\} s i = min { c A i L 0 / L 1 , R } , ( α i / s i ) α ≤ ( α i L 1 / c A i L 0 ) α + ( α i / R ) α (\alpha_i/s_i)^\alpha\le(\alpha_iL_1/\sqrt{cA_iL_0})^\alpha+(\alpha_i/R)^\alpha ( α i / s i ) α ≤ ( α i L 1 / c A i L 0 ) α + ( α i / R ) α . Using A i ≥ ( a L 0 / 2 ) α i 2 A_i\ge(aL_0/2)\alpha_i^2 A i ≥ ( a L 0 /2 ) α i 2 , α i = ( i + 1 ) / ( a L 0 ) \alpha_i=(i+1)/(aL_0) α i = ( i + 1 ) / ( a L 0 ) , and ∑ i = 1 K ( i + 1 ) α ≤ 2 α K α + 1 \sum_{i=1}^{K}(i+1)^\alpha\le2^\alpha K^{\alpha+1} ∑ i = 1 K ( i + 1 ) α ≤ 2 α K α + 1 , we obtain
σ α ∑ i = 1 K α i s i λ i α − 1 ≤ σ α α α − 1 [ K ( 2 c a ) α / 2 ( L 1 L 0 ) α + 2 α K α + 1 a α L 0 α R α ] . \sigma^\alpha\sum_{i=1}^{K}\frac{\alpha_i}{s_i\lambda_i^{\alpha-1}}
\le\sigma^\alpha\alpha^{\alpha-1}\left[K\left(\frac{2}{ca}\right)^{\alpha/2}\left(\frac{L_1}{L_0}\right)^\alpha+\frac{2^\alpha K^{\alpha+1}}{a^\alpha L_0^\alpha R^\alpha}\right]. σ α i = 1 ∑ K s i λ i α − 1 α i ≤ σ α α α − 1 [ K ( c a 2 ) α /2 ( L 0 L 1 ) α + a α L 0 α R α 2 α K α + 1 ] .
By the choice of a a a , together with q α = 1024 ℓ α − 1 K q^\alpha=1024\ell^{\alpha-1}K q α = 1024 ℓ α − 1 K , each of the two terms above is bounded by 1 / 1024 1/1024 1/1024 . Therefore their sum is at most 1 / 512 1/512 1/512 . Since a ≥ 192 ℓ 2 a\ge192\ell^2 a ≥ 192 ℓ 2 as well, all the conditions of Lemma 7 are satisfied. Hence, with probability at least 1 − δ 1-\delta 1 − δ , the event E K \mathcal{E}_K E K holds.
On E K \mathcal{E}_K E K , applying Lemma 4 with u = x ⋆ u=x^\star u = x ⋆ gives f ( y K ) − f ⋆ ≤ R 2 / A K = 2 a L 0 R 2 / [ K ( K + 3 ) ] f(y_K)-f^\star\le R^2/A_K=2aL_0R^2/[K(K+3)] f ( y K ) − f ⋆ ≤ R 2 / A K = 2 a L 0 R 2 / [ K ( K + 3 )] . Applying the same lemma with u = x ˉ u=\bar{x} u = x ˉ yields f ( y K ) − f ⋆ ≤ 2 c L 0 / L 1 2 f(y_K)-f^\star\le2cL_0/L_1^2 f ( y K ) − f ⋆ ≤ 2 c L 0 / L 1 2 . Combining the two bounds, we obtain
f ( y K ) − f ⋆ ≤ min { 2 c L 0 L 1 2 , 2 a L 0 R 2 K ( K + 3 ) } . f(y_K)-f^\star\le\min\left\{\frac{2cL_0}{L_1^2},\frac{2aL_0R^2}{K(K+3)}\right\}. f ( y K ) − f ⋆ ≤ min { L 1 2 2 c L 0 , K ( K + 3 ) 2 a L 0 R 2 } .
Using K ( K + 3 ) ≥ K 2 K(K+3)\ge K^2 K ( K + 3 ) ≥ K 2 , max { x , y , z } ≤ x + y + z \max\{x,y,z\}\le x+y+z max { x , y , z } ≤ x + y + z , and the definition of a a a , we get
2 a L 0 R 2 K ( K + 3 ) ≤ 384 L 0 R 2 ℓ 2 K 2 + 4 q 2 σ 2 L 1 2 R 2 c L 0 K 2 + 4 q σ R K . \frac{2aL_0R^2}{K(K+3)}
\le\frac{384L_0R^2\ell^2}{K^2}+\frac{4q^2\sigma^2L_1^2R^2}{cL_0K^2}+\frac{4q\sigma R}{K}. K ( K + 3 ) 2 a L 0 R 2 ≤ K 2 384 L 0 R 2 ℓ 2 + c L 0 K 2 4 q 2 σ 2 L 1 2 R 2 + K 4 q σ R .
Therefore, using min { x , r + s + t } ≤ r + t + min { x , s } \min\{x,r+s+t\}\le r+t+\min\{x,s\} min { x , r + s + t } ≤ r + t + min { x , s } for nonnegative quantities,
f ( y K ) − f ⋆ ≤ 384 L 0 R 2 ℓ 2 K 2 + 4 q σ R K + min { 2 c L 0 L 1 2 , 4 q 2 σ 2 L 1 2 R 2 c L 0 K 2 } . f(y_K)-f^\star\le\frac{384L_0R^2\ell^2}{K^2}+\frac{4q\sigma R}{K}+\min\left\{\frac{2cL_0}{L_1^2},\frac{4q^2\sigma^2L_1^2R^2}{cL_0K^2}\right\}. f ( y K ) − f ⋆ ≤ K 2 384 L 0 R 2 ℓ 2 + K 4 q σ R + min { L 1 2 2 c L 0 , c L 0 K 2 4 q 2 σ 2 L 1 2 R 2 } .
Finally, using min { x , y } ≤ x y \min\{x,y\}\le\sqrt{xy} min { x , y } ≤ x y ,
min { 2 c L 0 L 1 2 , 4 q 2 σ 2 L 1 2 R 2 c L 0 K 2 } ≤ 2 2 q σ R K . \min\left\{\frac{2cL_0}{L_1^2},\frac{4q^2\sigma^2L_1^2R^2}{cL_0K^2}\right\}\le\frac{2\sqrt{2}q\sigma R}{K}. min { L 1 2 2 c L 0 , c L 0 K 2 4 q 2 σ 2 L 1 2 R 2 } ≤ K 2 2 q σ R .
Hence,
f ( y K ) − f ⋆ ≤ 384 L 0 R 2 ℓ 2 K 2 + ( 4 + 2 2 ) q σ R K . f(y_K)-f^\star\le\frac{384L_0R^2\ell^2}{K^2}+(4+2\sqrt{2})\frac{q\sigma R}{K}. f ( y K ) − f ⋆ ≤ K 2 384 L 0 R 2 ℓ 2 + ( 4 + 2 2 ) K q σ R .
Proof of Corollary 1. Let M : = max { L 1 R 0 , log 24 δ } M:=\max\{L_1R_0,\log\frac{24}{\delta}\} M := max { L 1 R 0 , log δ 24 } . For Phase I, we run Clip-SGD with the parameters of Theorem 2 of El Bakkali El Kadi et al. (2026) for K 1 K_1 K 1 iterations, where K 1 K_1 K 1 is chosen large enough so that
K 1 ≥ ( L 1 R 0 ) 2 , K 1 ≥ L 1 R 0 M α − 1 2 α , K_1\ge(L_1R_0)^2,\qquad K_1\ge L_1R_0M^{\frac{\alpha-1}{2\alpha}}, K 1 ≥ ( L 1 R 0 ) 2 , K 1 ≥ L 1 R 0 M 2 α α − 1 ,
and the iteration requirement of the cited result is satisfied.
Since α ∈ ( 1 , 2 ] \alpha\in(1,2] α ∈ ( 1 , 2 ] , we have 0 < α − 1 2 α ≤ 1 4 0<\frac{\alpha-1}{2\alpha}\le\frac14 0 < 2 α α − 1 ≤ 4 1 . Then
L 1 R 0 M α − 1 2 α ≤ max { ( L 1 R 0 ) 1 + α − 1 2 α , L 1 R 0 ( log 24 δ ) α − 1 2 α } = O ~ ( 1 + ( L 1 R 0 ) 2 ) . L_1R_0M^{\frac{\alpha-1}{2\alpha}}\le\max\left\{(L_1R_0)^{1+\frac{\alpha-1}{2\alpha}},L_1R_0\left(\log\frac{24}{\delta}\right)^{\frac{\alpha-1}{2\alpha}}\right\}=\widetilde{O}\left(1+(L_1R_0)^2\right). L 1 R 0 M 2 α α − 1 ≤ max { ( L 1 R 0 ) 1 + 2 α α − 1 , L 1 R 0 ( log δ 24 ) 2 α α − 1 } = O ( 1 + ( L 1 R 0 ) 2 ) .
The additional iteration requirement in Theorem 2 of El Bakkali El Kadi et al. (2026) is also covered by the same bound up to logarithmic factors. Therefore, it is enough to take
K 1 = O ~ ( 1 + ( L 1 R 0 ) 2 ) . K_1=\widetilde{O}\left(1+(L_1R_0)^2\right). K 1 = O ( 1 + ( L 1 R 0 ) 2 ) .
By El Bakkali El Kadi et al. (2026), with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 , the output x ˉ \bar{x} x ˉ of Phase I satisfies ∥ x ˉ − x ⋆ ∥ ≤ 2 R 0 \|\bar{x}-x^\star\|\le2R_0 ∥ x ˉ − x ⋆ ∥ ≤ 2 R 0 and f ( x ˉ ) − f ⋆ ≤ B f(\bar{x})-f^\star\le B f ( x ˉ ) − f ⋆ ≤ B , where, for some universal constant C > 0 C>0 C > 0 ,
B : = C max { L 0 R 0 2 K 1 , σ R 0 ( M K 1 ) α − 1 α } . B:=C\max\left\{\frac{L_0R_0^2}{K_1},\sigma R_0\left(\frac{M}{K_1}\right)^{\frac{\alpha-1}{\alpha}}\right\}. B := C max { K 1 L 0 R 0 2 , σ R 0 ( K 1 M ) α α − 1 } .
Let L ˉ 0 : = L 0 + 32 L 1 2 B \bar{L}_0:=L_0+32L_1^2B L ˉ 0 := L 0 + 32 L 1 2 B . Since L ˉ 0 ≥ L 0 \bar{L}_0\ge L_0 L ˉ 0 ≥ L 0 , Assumption 2 also holds with ( L ˉ 0 , L 1 ) (\bar{L}_0,L_1) ( L ˉ 0 , L 1 ) in place of ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) . Moreover,
f ( x ˉ ) − f ⋆ ≤ B ≤ L ˉ 0 32 L 1 2 . f(\bar{x})-f^\star\le B\le\frac{\bar{L}_0}{32L_1^2}. f ( x ˉ ) − f ⋆ ≤ B ≤ 32 L 1 2 L ˉ 0 .
Hence Theorem 1 can be applied in Phase II with L 0 L_0 L 0 replaced by L ˉ 0 \bar{L}_0 L ˉ 0 , R = 2 R 0 R=2R_0 R = 2 R 0 , and failure probability δ / 2 \delta/2 δ /2 . Therefore, Phase II reaches accuracy ε \varepsilon ε after
O ~ ( L ˉ 0 R 0 2 ε + ( σ R 0 ε ) α α − 1 ) \widetilde{O}\left(\sqrt{\frac{\bar{L}_0R_0^2}{\varepsilon}}+\left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}\right) O ( ε L ˉ 0 R 0 2 + ( ε σ R 0 ) α − 1 α )
stochastic-gradient evaluations.
Now we bound the first term. Using a + b ≤ a + b \sqrt{a+b}\le\sqrt{a}+\sqrt{b} a + b ≤ a + b , we obtain
L ‾ 0 R 0 2 ε ≲ L 0 R 0 2 ε + L 1 R 0 B ε . \sqrt{\frac{\overline{L}_0R_0^2}{\varepsilon}} \lesssim\sqrt{\frac{L_0R_0^2}{\varepsilon}}+L_1R_0\sqrt{\frac{B}{\varepsilon}}. ε L 0 R 0 2 ≲ ε L 0 R 0 2 + L 1 R 0 ε B .
Using the definition of B B B , we have
L 1 R 0 B ε ≲ L 1 R 0 K 1 L 0 R 0 2 ε + L 1 R 0 σ R 0 ε ( M K 1 ) α − 1 2 α . L_1R_0\sqrt{\frac{B}{\varepsilon}} \lesssim\frac{L_1R_0}{\sqrt{K_1}}\sqrt{\frac{L_0R_0^2}{\varepsilon}}+L_1R_0\sqrt{\frac{\sigma R_0}{\varepsilon}}\left(\frac{M}{K_1}\right)^{\frac{\alpha-1}{2\alpha}}. L 1 R 0 ε B ≲ K 1 L 1 R 0 ε L 0 R 0 2 + L 1 R 0 ε σ R 0 ( K 1 M ) 2 α α − 1 .
since K 1 ≳ ( L 1 R 0 ) 2 K_1 \gtrsim(L_1R_0)^2 K 1 ≳ ( L 1 R 0 ) 2 , the first term is bounded by L 0 R 0 2 / ε \sqrt{L_0R_0^2/\varepsilon} L 0 R 0 2 / ε . For the second term, we have
L 1 R 0 σ R 0 ε ( M K 1 ) α − 1 2 α = L 1 R 0 M α − 1 2 α ( ( σ R 0 ε ) α α − 1 K 1 ) α − 1 2 α . L_1R_0\sqrt{\frac{\sigma R_0}{\varepsilon}}\left(\frac{M}{K_1}\right)^{\frac{\alpha-1}{2\alpha}}=L_1R_0M^{\frac{\alpha-1}{2\alpha}}\left(\frac{\left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}}{K_1}\right)^{\frac{\alpha-1}{2\alpha}}. L 1 R 0 ε σ R 0 ( K 1 M ) 2 α α − 1 = L 1 R 0 M 2 α α − 1 ( K 1 ( ε σ R 0 ) α − 1 α ) 2 α α − 1 .
Since L 1 R 0 M α − 1 2 α = O ~ ( K 1 ) L_1R_0M^{\frac{\alpha-1}{2\alpha}}=\widetilde{O}(K_1) L 1 R 0 M 2 α α − 1 = O ( K 1 ) and 0 < ( α − 1 ) / ( 2 α ) ≤ 1 0<(\alpha-1)/(2\alpha)\leq1 0 < ( α − 1 ) / ( 2 α ) ≤ 1 , the inequality x r ≤ 1 + x x^r\leq1+x x r ≤ 1 + x for x ≥ 0 x\geq0 x ≥ 0 and 0 < r ≤ 1 0<r\leq1 0 < r ≤ 1 gives
L 1 R 0 σ R 0 ε ( M K 1 ) α − 1 2 α = O ~ ( K 1 + ( σ R 0 ε ) α α − 1 ) . L_1R_0\sqrt{\frac{\sigma R_0}{\varepsilon}}\left(\frac{M}{K_1}\right)^{\frac{\alpha-1}{2\alpha}}=\widetilde{O}\left(K_1+\left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}\right). L 1 R 0 ε σ R 0 ( K 1 M ) 2 α α − 1 = O ( K 1 + ( ε σ R 0 ) α − 1 α ) .
Consequently,
L ‾ 0 R 0 2 ε = O ~ ( L 0 R 0 2 ε + K 1 + ( σ R 0 ε ) α α − 1 ) . \sqrt{\frac{\overline{L}_0R_0^2}{\varepsilon}}=\widetilde{O}\left(\sqrt{\frac{L_0R_0^2}{\varepsilon}}+K_1+\left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}\right). ε L 0 R 0 2 = O ( ε L 0 R 0 2 + K 1 + ( ε σ R 0 ) α − 1 α ) .
Combining the two phases and using
K 1 = O ~ ( 1 + ( L 1 R 0 ) 2 ) K_1=\widetilde{O}\left(1+(L_1R_0)^2\right) K 1 = O ( 1 + ( L 1 R 0 ) 2 )
we obtain a total complexity of
O ~ ( 1 + ( L 1 R 0 ) 2 + L 0 R 0 2 ε + ( σ R 0 ε ) α α − 1 ) . \widetilde{O}\left(1+(L_1R_0)^2+\sqrt{\frac{L_0R_0^2}{\varepsilon}}+\left(\frac{\sigma R_0}{\varepsilon}\right)^{\frac{\alpha}{\alpha-1}}\right). O ( 1 + ( L 1 R 0 ) 2 + ε L 0 R 0 2 + ( ε σ R 0 ) α − 1 α ) .
High-Probability Analysis for the One-Phase Result
Throughout this section, assume
{ ∥ x 0 − x ⋆ ∥ ≤ R 0 , Δ 0 : = f ( x 0 ) − f ⋆ ≤ Δ ‾ 0 . \begin{cases}
\|x_0-x^\star\|\leq R_0,\\
\Delta_0:=f(x_0)-f^\star\leq\overline{\Delta}_0.
\end{cases} { ∥ x 0 − x ⋆ ∥ ≤ R 0 , Δ 0 := f ( x 0 ) − f ⋆ ≤ Δ 0 .
We also use
R t ( u ) : = ∑ i = 1 t α i ⟨ θ i , u − z i − 1 ⟩ + ∑ i = 1 t α i 2 ∥ θ i ∥ 2 , u ∈ R d , \mathcal{R}_t(u):=\sum_{i=1}^{t}\alpha_i\langle\theta_i,u-z_{i-1}\rangle+\sum_{i=1}^{t}\alpha_i^2\|\theta_i\|^2,\qquad u\in\mathbb{R}^d, R t ( u ) := i = 1 ∑ t α i ⟨ θ i , u − z i − 1 ⟩ + i = 1 ∑ t α i 2 ∥ θ i ∥ 2 , u ∈ R d ,
as introduced in the two-phase analysis.
Lemma 8. Suppose that Assumption 1 holds. Let A 0 ≥ 0 A_0\geq0 A 0 ≥ 0 , y 0 = z 0 = v 0 y_0=z_0=v_0 y 0 = z 0 = v 0 , and let t ∈ { 1 , … , K } t\in\{1,\ldots,K\} t ∈ { 1 , … , K } . Assume that, for every i = 1 , … , t i=1,\ldots,t i = 1 , … , t ,
f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + A i 4 α i 2 ∥ y i − x i ∥ 2 . f(y_i)\leq f(x_i)+\langle\nabla f(x_i),y_i-x_i\rangle+\frac{A_i}{4\alpha_i^2}\|y_i-x_i\|^2. f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 α i 2 A i ∥ y i − x i ∥ 2 . Then, for every u ∈ R d u\in\mathbb{R}^d u ∈ R d ,
A t ( f ( y t ) − f ( u ) ) + 1 2 ∥ z t − u ∥ 2 ≤ A 0 ( f ( v 0 ) − f ( u ) ) + 1 2 ∥ v 0 − u ∥ 2 + R t ( u ) . A_t(f(y_t)-f(u))+\frac{1}{2}\|z_t-u\|^2\leq A_0(f(v_0)-f(u))+\frac{1}{2}\|v_0-u\|^2+\mathcal{R}_t(u). A t ( f ( y t ) − f ( u )) + 2 1 ∥ z t − u ∥ 2 ≤ A 0 ( f ( v 0 ) − f ( u )) + 2 1 ∥ v 0 − u ∥ 2 + R t ( u ) .
Lemma 9. Suppose that Assumptions 1 1 1 and 2 2 2 hold, where L 0 , L 1 > 0 L_0,L_1 > 0 L 0 , L 1 > 0 . Let v 0 = x 0 v_0=x_0 v 0 = x 0 , ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 , a ≥ 32 a \ge32 a ≥ 32 , and suppose that A 0 Δ 0 ≤ R 0 2 64 A_0\Delta_0 \le\frac{R_0^2}{64} A 0 Δ 0 ≤ 64 R 0 2 . Assume that, for every k = 0 , … , K − 1 k=0,\ldots,K-1 k = 0 , … , K − 1 ,
{ α k + 1 A k ≤ 1 64 ℓ ( 1 + L 1 R 0 ) , A k ≥ a L 0 α k + 1 2 . \begin{cases}
\frac{\alpha_{k+1}}{A_k} \le\frac{1}{64\ell(1+L_1R_0)},\\
A_k \ge aL_0\alpha_{k+1}^2.
\end{cases} { A k α k + 1 ≤ 64 ℓ ( 1 + L 1 R 0 ) 1 , A k ≥ a L 0 α k + 1 2 . For i = 1 , … , K i=1,\ldots,K i = 1 , … , K , let
λ i : = R 0 α i ℓ . \lambda_i:=\frac{R_0}{\alpha_i\ell}. λ i := α i ℓ R 0 . Let t ∈ { 0 , … , K − 1 } t\in\{0,\ldots,K-1\} t ∈ { 0 , … , K − 1 } and suppose that, for every i = 1 , … , t i=1,\ldots,t i = 1 , … , t ,
{ f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + A i 4 α i 2 ∥ y i − x i ∥ 2 , R i ( x ⋆ ) ≤ R 0 2 2 . \begin{cases}
f(y_i)\le f(x_i)+\langle\nabla f(x_i),y_i-x_i\rangle+\frac{A_i}{4\alpha_i^2}\lVert y_i-x_i\rVert^2,\\
\mathcal{R}_i(x^\star)\le\frac{R_0^2}{2}.
\end{cases} { f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 α i 2 A i ∥ y i − x i ∥ 2 , R i ( x ⋆ ) ≤ 2 R 0 2 . Then
f ( y t + 1 ) ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + A t + 1 4 α t + 1 2 ∥ y t + 1 − x t + 1 ∥ 2 . f(y_{t+1})\le f(x_{t+1})+\langle\nabla f(x_{t+1}),y_{t+1}-x_{t+1}\rangle+\frac{A_{t+1}}{4\alpha_{t+1}^2}\lVert y_{t+1}-x_{t+1}\rVert^2. f ( y t + 1 ) ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 4 α t + 1 2 A t + 1 ∥ y t + 1 − x t + 1 ∥ 2 .
Lemma 10. Suppose that Assumptions 1 1 1 and 2 2 2 hold, where L 0 , L 1 > 0 L_0,L_1>0 L 0 , L 1 > 0 . Consider the iterates of Algorithm 1 with v 0 = x 0 v_0=x_0 v 0 = x 0 , ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 , a ≥ 128 ℓ 2 a\ge128\ell^2 a ≥ 128 ℓ 2 , and suppose that A 0 Δ 0 ≤ R 0 2 64 A_0\Delta_0\le\frac{R_0^2}{64} A 0 Δ 0 ≤ 64 R 0 2 . Assume that, for every k = 0 , … , K − 1 k=0,\ldots,K-1 k = 0 , … , K − 1 ,
{ α k + 1 A k ≤ 1 64 ℓ ( 1 + L 1 R 0 ) , A k ≥ a L 0 α k + 1 2 . \begin{cases}
\frac{\alpha_{k+1}}{A_k}\le\frac{1}{64\ell(1+L_1R_0)},\\
A_k\ge aL_0\alpha_{k+1}^2.
\end{cases} { A k α k + 1 ≤ 64 ℓ ( 1 + L 1 R 0 ) 1 , A k ≥ a L 0 α k + 1 2 . For i = 1 , … , K i=1,\ldots,K i = 1 , … , K , let
λ i : = R 0 α i ℓ . \lambda_i:=\frac{R_0}{\alpha_i\ell}. λ i := α i ℓ R 0 . Let t ∈ { 0 , … , K − 1 } t\in\{0,\ldots,K-1\} t ∈ { 0 , … , K − 1 } and suppose that, for every i = 1 , … , t i=1,\ldots,t i = 1 , … , t ,
{ f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + A i 4 α i 2 ∥ y i − x i ∥ 2 , R i ( x ⋆ ) ≤ R 0 2 2 . \begin{cases}
f(y_i)\le f(x_i)+\langle\nabla f(x_i),y_i-x_i\rangle+\frac{A_i}{4\alpha_i^2}\lVert y_i-x_i\rVert^2,\\
\mathcal{R}_i(x^\star)\le\frac{R_0^2}{2}.
\end{cases} { f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 α i 2 A i ∥ y i − x i ∥ 2 , R i ( x ⋆ ) ≤ 2 R 0 2 . Then
∥ ∇ f ( x t + 1 ) ∥ ≤ λ t + 1 2 . \lVert\nabla f(x_{t+1})\rVert\le\frac{\lambda_{t+1}}{2}. ∥ ∇ f ( x t + 1 )∥ ≤ 2 λ t + 1 .
In the following, suppose that Assumptions 1 1 1 , 2 2 2 , and 3 3 3 hold, and that the initialization and parameter conditions of Lemma 10 10 10 are satisfied. Let F 0 \mathcal{F}_0 F 0 be the trivial σ \sigma σ -algebra, and for i = 1 , … , K i=1,\ldots,K i = 1 , … , K , define
F i : = σ ( ξ 1 , … , ξ i ) . \mathcal{F}_i:=\sigma(\xi_1,\ldots,\xi_i). F i := σ ( ξ 1 , … , ξ i ) .
We use the stochastic notation G i , g i , θ i G_i,g_i,\theta_i G i , g i , θ i introduced in the two-phase analysis.
For t = 0 , … , K t=0,\ldots,K t = 0 , … , K , let E t \mathcal{E}_t E t be the event that, for every i = 1 , … , t i=1,\ldots,t i = 1 , … , t ,
{ f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + A i 4 α i 2 ∥ y i − x i ∥ 2 , R i ( x ⋆ ) ≤ R 0 2 2 . \begin{cases}
f(y_i)\le f(x_i)+\langle\nabla f(x_i),y_i-x_i\rangle+\frac{A_i}{4\alpha_i^2}\lVert y_i-x_i\rVert^2,\\
\mathcal{R}_i(x^\star)\le\frac{R_0^2}{2}.
\end{cases} { f ( y i ) ≤ f ( x i ) + ⟨ ∇ f ( x i ) , y i − x i ⟩ + 4 α i 2 A i ∥ y i − x i ∥ 2 , R i ( x ⋆ ) ≤ 2 R 0 2 .
For t = 0 t=0 t = 0 , there are no conditions, so 1 E 0 = 1 \mathbf{1}_{\mathcal{E}_0}=1 1 E 0 = 1 . Define
R ^ t ( x ⋆ ) : = ∑ i = 1 t 1 E i − 1 ( α i ⟨ θ i , x ⋆ − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 ) . \widehat{R}_t(x^\star) := \sum_{i=1}^{t} \mathbf{1}_{\mathcal{E}_{i-1}}\left(\alpha_i\langle\theta_i,x^\star-z_{i-1}\rangle+\alpha_i^2\|\theta_i\|^2\right). R t ( x ⋆ ) := i = 1 ∑ t 1 E i − 1 ( α i ⟨ θ i , x ⋆ − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 ) .
Since the events E t \mathcal{E}_t E t are nested, on E t − 1 \mathcal{E}_{t-1} E t − 1 we have R ^ t ( x ⋆ ) = R t ( x ⋆ ) \widehat{R}_t(x^\star)=R_t(x^\star) R t ( x ⋆ ) = R t ( x ⋆ ) . Moreover, on E i − 1 \mathcal{E}_{i-1} E i − 1 , Lemma 10 gives
∥ ∇ f ( x i ) ∥ ≤ λ i 2 . \|\nabla f(x_i)\|\le\frac{\lambda_i}{2}. ∥∇ f ( x i ) ∥ ≤ 2 λ i .
Therefore, on E i − 1 \mathcal{E}_{i-1} E i − 1 , the same clipping estimates as in the two-phase analysis apply:
{ ∥ E [ θ i ∣ F i − 1 ] ∥ ≤ 2 α σ α λ i α − 1 , E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 2 σ α λ i 2 − α , ∥ θ i ∥ ≤ 3 2 λ i . \left\{
\begin{aligned}
\|\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}]\|&\le\frac{2^{\alpha}\sigma^\alpha}{\lambda_i^{\alpha-1}},\\
\mathbb{E}[\|\theta_i\|^2\mid\mathcal{F}_{i-1}]&\le2\sigma^{\alpha}\lambda_i^{2-\alpha},\\
\|\theta_i\|&\le\frac{3}{2}\lambda_i.
\end{aligned}
\right. ⎩ ⎨ ⎧ ∥ E [ θ i ∣ F i − 1 ] ∥ E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ∥ θ i ∥ ≤ λ i α − 1 2 α σ α , ≤ 2 σ α λ i 2 − α , ≤ 2 3 λ i .
To control R ^ t ( x ⋆ ) \widehat{R}_t(x^\star) R t ( x ⋆ ) , we separate the martingale fluctuations from the clipping bias and the conditional second-moment terms. We write
R ^ t ( x ⋆ ) = M t + B t , \widehat{R}_t(x^\star)=M_t+B_t, R t ( x ⋆ ) = M t + B t ,
where
{ M t : = ∑ i = 1 t 1 E i − 1 ( α i ⟨ θ i − E [ θ i ∣ F i − 1 ] , x ⋆ − z i − 1 ⟩ + α i 2 ( ∥ θ i ∥ 2 − E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ) ) , B t : = ∑ i = 1 t 1 E i − 1 ( α i ⟨ E [ θ i ∣ F i − 1 ] , x ⋆ − z i − 1 ⟩ + α i 2 E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ) . \left\{
\begin{aligned}
M_t&:=\sum_{i=1}^{t}\mathbf{1}_{\mathcal{E}_{i-1}}\left(\alpha_i\langle\theta_i-\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}],x^\star-z_{i-1}\rangle+\alpha_i^2\left(\|\theta_i\|^2-\mathbb{E}[\|\theta_i\|^2\mid\mathcal{F}_{i-1}]\right)\right),\\
B_t&:=\sum_{i=1}^{t}\mathbf{1}_{\mathcal{E}_{i-1}}\left(\alpha_i\langle\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}],x^\star-z_{i-1}\rangle+\alpha_i^2\mathbb{E}[\|\theta_i\|^2\mid\mathcal{F}_{i-1}]\right).
\end{aligned}
\right. ⎩ ⎨ ⎧ M t B t := i = 1 ∑ t 1 E i − 1 ( α i ⟨ θ i − E [ θ i ∣ F i − 1 ] , x ⋆ − z i − 1 ⟩ + α i 2 ( ∥ θ i ∥ 2 − E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ) ) , := i = 1 ∑ t 1 E i − 1 ( α i ⟨ E [ θ i ∣ F i − 1 ] , x ⋆ − z i − 1 ⟩ + α i 2 E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ) .
Since 1 E i − 1 \mathbf{1}_{\mathcal{E}_{i-1}} 1 E i − 1 and x ⋆ − z i − 1 x^\star-z_{i-1} x ⋆ − z i − 1 are F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable, the increments of M t M_t M t have zero conditional mean. Hence ( M t ) t = 0 K (M_t)_{t=0}^{K} ( M t ) t = 0 K is a martingale with M 0 = 0 M_0=0 M 0 = 0 .
Lemma 11. Suppose that Assumptions 1, 2, and 3 hold, where L 0 , L 1 > 0 L_0,L_1>0 L 0 , L 1 > 0 . Assume that ∥ x 0 − x ⋆ ∥ ≤ R 0 \|x_0-x^\star\|\le R_0 ∥ x 0 − x ⋆ ∥ ≤ R 0 and A 0 Δ 0 ≤ R 0 2 64 A_0\Delta_0\le\frac{R_0^2}{64} A 0 Δ 0 ≤ 64 R 0 2 . Let K ≥ 1 K\ge1 K ≥ 1 , δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , ℓ ≥ 8 log 4 K δ \ell\ge8\log\frac{4K}{\delta} ℓ ≥ 8 log δ 4 K , a ≥ 128 ℓ 2 a\ge128\ell^2 a ≥ 128 ℓ 2 , and suppose that, for every k = 0 , … , K − 1 k=0,\ldots,K-1 k = 0 , … , K − 1 ,
{ α k + 1 A k ≤ 1 64 ℓ ( 1 + L 1 R 0 ) , A k ≥ a L 0 α k + 1 2 . \left\{
\begin{aligned}
\frac{\alpha_{k+1}}{A_k}&\le\frac{1}{64\ell(1+L_1R_0)},\\
A_k&\ge aL_0\alpha_{k+1}^2.
\end{aligned}
\right. ⎩ ⎨ ⎧ A k α k + 1 A k ≤ 64 ℓ ( 1 + L 1 R 0 ) 1 , ≥ a L 0 α k + 1 2 . For i = 1 , … , K i=1,\ldots,K i = 1 , … , K , let λ i : = R 0 α i ℓ \lambda_i:=\frac{R_0}{\alpha_i\ell} λ i := α i ℓ R 0 . Suppose moreover that
σ α ℓ α − 1 R 0 α ∑ i = 1 K α i α ≤ 1 512 . \frac{\sigma^\alpha\ell^{\alpha-1}}{R_0^\alpha}\sum_{i=1}^{K}\alpha_i^\alpha\le\frac{1}{512}. R 0 α σ α ℓ α − 1 i = 1 ∑ K α i α ≤ 512 1 . Then, with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every t = 1 , … , K t=1,\ldots,K t = 1 , … , K ,
R ^ t ( x ⋆ ) ≤ R 0 2 2 . \widehat{R}_t(x^\star)\le\frac{R_0^2}{2}. R t ( x ⋆ ) ≤ 2 R 0 2 . On this event, E K \mathcal{E}_K E K holds.
Lemma 12. Let L 0 > 0 L_0 > 0 L 0 > 0 , L 1 ≥ 0 L_1 \ge0 L 1 ≥ 0 , R 0 > 0 R_0 > 0 R 0 > 0 , ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 , a > 0 a > 0 a > 0 , and α ∈ ( 1 , 2 ] \alpha\in(1,2] α ∈ ( 1 , 2 ] . Suppose that 0 < A 0 ≤ 1 / ( a L 0 ) 0 < A_0 \le1/(aL_0) 0 < A 0 ≤ 1/ ( a L 0 ) and, for every k ≥ 0 k \ge0 k ≥ 0 ,
{ α k + 1 = min { A k 64 ℓ ( 1 + L 1 R 0 ) , A k a L 0 } , A k + 1 = A k + α k + 1 . \begin{cases}
\alpha_{k+1}=\min\left\{\dfrac{A_k}{64\ell(1+L_1R_0)},\sqrt{\dfrac{A_k}{aL_0}}\right\},\\
A_{k+1}=A_k+\alpha_{k+1}.
\end{cases} ⎩ ⎨ ⎧ α k + 1 = min { 64 ℓ ( 1 + L 1 R 0 ) A k , a L 0 A k } , A k + 1 = A k + α k + 1 . Define
K 0 : = ⌈ 128 ℓ ( 1 + L 1 R 0 ) log ( [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 A 0 ) ⌉ . K_0:=\left\lceil128\ell(1+L_1R_0)\log\left(\dfrac{[64\ell(1+L_1R_0)]^2}{aL_0A_0}\right)\right\rceil. K 0 := ⌈ 128 ℓ ( 1 + L 1 R 0 ) log ( a L 0 A 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 ) ⌉ . Then, for every K ≥ K 0 K \ge K_0 K ≥ K 0 , A K ≥ 4 ( K − K 0 ) 2 25 a L 0 A_K \ge\dfrac{4(K-K_0)^2}{25aL_0} A K ≥ 25 a L 0 4 ( K − K 0 ) 2 . Moreover, for every i ≥ 1 i \ge1 i ≥ 1 ,
α i ≤ i + 1 2 a L 0 , \alpha_i\le\dfrac{i+1}{2aL_0}, α i ≤ 2 a L 0 i + 1 , and, for every K ≥ 1 K \ge1 K ≥ 1 ,
∑ i = 1 K α i α ≤ K ( K + 1 ) α 2 α ( a L 0 ) α . \sum_{i=1}^{K}\alpha_i^\alpha\le\dfrac{K(K+1)^\alpha}{2^\alpha(aL_0)^\alpha}. i = 1 ∑ K α i α ≤ 2 α ( a L 0 ) α K ( K + 1 ) α .
Proofs
Proof of Lemma 8 . Fix u ∈ R d u \in\mathbb{R}^d u ∈ R d and define
Φ i ( u ) : = A i ( f ( y i ) − f ( u ) ) + 1 2 ∥ z i − u ∥ 2 . \Phi_i(u):=A_i\bigl(f(y_i)-f(u)\bigr)+\frac{1}{2}\|z_i-u\|^2. Φ i ( u ) := A i ( f ( y i ) − f ( u ) ) + 2 1 ∥ z i − u ∥ 2 .
By convexity,
{ f ( y i − 1 ) ≥ f ( x i ) + ⟨ ∇ f ( x i ) , y i − 1 − x i ⟩ , f ( u ) ≥ f ( x i ) + ⟨ ∇ f ( x i ) , u − x i ⟩ . \begin{cases}
f(y_{i-1})\ge f(x_i)+\langle\nabla f(x_i),y_{i-1}-x_i\rangle,\\
f(u)\ge f(x_i)+\langle\nabla f(x_i),u-x_i\rangle.
\end{cases} { f ( y i − 1 ) ≥ f ( x i ) + ⟨ ∇ f ( x i ) , y i − 1 − x i ⟩ , f ( u ) ≥ f ( x i ) + ⟨ ∇ f ( x i ) , u − x i ⟩ .
Multiplying the first inequality by A i − 1 A_{i-1} A i − 1 and the second by α i \alpha_i α i , and using A i = A i − 1 + α i A_i=A_{i-1}+\alpha_i A i = A i − 1 + α i and A i x i = A i − 1 y i − 1 + α i z i − 1 A_ix_i=A_{i-1}y_{i-1}+\alpha_iz_{i-1} A i x i = A i − 1 y i − 1 + α i z i − 1 , we obtain
A i f ( x i ) − A i − 1 f ( y i − 1 ) − α i f ( u ) ≤ α i ⟨ ∇ f ( x i ) , z i − 1 − u ⟩ . A_if(x_i)-A_{i-1}f(y_{i-1})-\alpha_if(u)\le\alpha_i\langle\nabla f(x_i),z_{i-1}-u\rangle. A i f ( x i ) − A i − 1 f ( y i − 1 ) − α i f ( u ) ≤ α i ⟨ ∇ f ( x i ) , z i − 1 − u ⟩ .
We have y i − x i = − α i 2 A i g i y_i-x_i=-\dfrac{\alpha_i^2}{A_i}g_i y i − x i = − A i α i 2 g i . Then, by the assumed descent inequality,
A i ( f ( y i ) − f ( x i ) ) ≤ − α i 2 ⟨ ∇ f ( x i ) , g i ⟩ + α i 2 4 ∥ g i ∥ 2 . A_i\bigl(f(y_i)-f(x_i)\bigr)\le-\alpha_i^2\langle\nabla f(x_i),g_i\rangle+\frac{\alpha_i^2}{4}\|g_i\|^2. A i ( f ( y i ) − f ( x i ) ) ≤ − α i 2 ⟨ ∇ f ( x i ) , g i ⟩ + 4 α i 2 ∥ g i ∥ 2 .
Since z i = z i − 1 − α i g i z_i=z_{i-1}-\alpha_ig_i z i = z i − 1 − α i g i , we have
1 2 ∥ z i − u ∥ 2 − 1 2 ∥ z i − 1 − u ∥ 2 = − α i ⟨ g i , z i − 1 − u ⟩ + α i 2 2 ∥ g i ∥ 2 . \frac{1}{2}\|z_i-u\|^2-\frac{1}{2}\|z_{i-1}-u\|^2=-\alpha_i\langle g_i,z_{i-1}-u\rangle+\frac{\alpha_i^2}{2}\|g_i\|^2. 2 1 ∥ z i − u ∥ 2 − 2 1 ∥ z i − 1 − u ∥ 2 = − α i ⟨ g i , z i − 1 − u ⟩ + 2 α i 2 ∥ g i ∥ 2 .
We deduce that
Φ i ( u ) − Φ i − 1 ( u ) = A i ( f ( y i ) − f ( u ) ) − A i − 1 ( f ( y i − 1 ) − f ( u ) ) + 1 2 ∥ z i − u ∥ 2 − 1 2 ∥ z i − 1 − u ∥ 2 = A i f ( y i ) − A i − 1 f ( y i − 1 ) − α i f ( u ) + 1 2 ∥ z i − u ∥ 2 − 1 2 ∥ z i − 1 − u ∥ 2 ≤ α i ⟨ ∇ f ( x i ) , z i − 1 − u ⟩ − α i 2 ⟨ ∇ f ( x i ) , g i ⟩ + α i 2 4 ∥ g i ∥ 2 − α i ⟨ g i , z i − 1 − u ⟩ + α i 2 2 ∥ g i ∥ 2 = α i ⟨ ∇ f ( x i ) − g i , z i − 1 − u ⟩ + α i 2 ( − ⟨ ∇ f ( x i ) , g i ⟩ + 3 4 ∥ g i ∥ 2 ) = α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ( ⟨ θ i , g i ⟩ − 1 4 ∥ g i ∥ 2 ) . \begin{aligned}
\Phi_i(u)-\Phi_{i-1}(u)
&=A_i\bigl(f(y_i)-f(u)\bigr)-A_{i-1}\bigl(f(y_{i-1})-f(u)\bigr)+\frac{1}{2}\|z_i-u\|^2-\frac{1}{2}\|z_{i-1}-u\|^2\\
&=A_if(y_i)-A_{i-1}f(y_{i-1})-\alpha_if(u)+\frac{1}{2}\|z_i-u\|^2-\frac{1}{2}\|z_{i-1}-u\|^2\\
&\le\alpha_i\langle\nabla f(x_i),z_{i-1}-u\rangle-\alpha_i^2\langle\nabla f(x_i),g_i\rangle+\frac{\alpha_i^2}{4}\|g_i\|^2-\alpha_i\langle g_i,z_{i-1}-u\rangle+\frac{\alpha_i^2}{2}\|g_i\|^2\\
&=\alpha_i\langle\nabla f(x_i)-g_i,z_{i-1}-u\rangle+\alpha_i^2\left(-\langle\nabla f(x_i),g_i\rangle+\frac{3}{4}\|g_i\|^2\right)\\
&=\alpha_i\langle\theta_i,u-z_{i-1}\rangle+\alpha_i^2\left(\langle\theta_i,g_i\rangle-\frac{1}{4}\|g_i\|^2\right).
\end{aligned} Φ i ( u ) − Φ i − 1 ( u ) = A i ( f ( y i ) − f ( u ) ) − A i − 1 ( f ( y i − 1 ) − f ( u ) ) + 2 1 ∥ z i − u ∥ 2 − 2 1 ∥ z i − 1 − u ∥ 2 = A i f ( y i ) − A i − 1 f ( y i − 1 ) − α i f ( u ) + 2 1 ∥ z i − u ∥ 2 − 2 1 ∥ z i − 1 − u ∥ 2 ≤ α i ⟨ ∇ f ( x i ) , z i − 1 − u ⟩ − α i 2 ⟨ ∇ f ( x i ) , g i ⟩ + 4 α i 2 ∥ g i ∥ 2 − α i ⟨ g i , z i − 1 − u ⟩ + 2 α i 2 ∥ g i ∥ 2 = α i ⟨ ∇ f ( x i ) − g i , z i − 1 − u ⟩ + α i 2 ( − ⟨ ∇ f ( x i ) , g i ⟩ + 4 3 ∥ g i ∥ 2 ) = α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ( ⟨ θ i , g i ⟩ − 4 1 ∥ g i ∥ 2 ) .
Since ⟨ θ i , g i ⟩ − 1 4 ∥ g i ∥ 2 = ∥ θ i ∥ 2 − 1 4 ∥ g i − 2 θ i ∥ 2 ≤ ∥ θ i ∥ 2 \langle\theta_i,g_i\rangle-\frac{1}{4}\lVert g_i\rVert^2=\lVert\theta_i\rVert^2-\frac{1}{4}\lVert g_i-2\theta_i\rVert^2\le\lVert\theta_i\rVert^2 ⟨ θ i , g i ⟩ − 4 1 ∥ g i ∥ 2 = ∥ θ i ∥ 2 − 4 1 ∥ g i − 2 θ i ∥ 2 ≤ ∥ θ i ∥ 2 , it follows that
Φ i ( u ) − Φ i − 1 ( u ) ≤ α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 . \Phi_i(u)-\Phi_{i-1}(u)\le\alpha_i\langle\theta_i,u-z_{i-1}\rangle+\alpha_i^2\lVert\theta_i\rVert^2. Φ i ( u ) − Φ i − 1 ( u ) ≤ α i ⟨ θ i , u − z i − 1 ⟩ + α i 2 ∥ θ i ∥ 2 .
Summing over i = 1 , … , t i=1,\ldots,t i = 1 , … , t , we obtain
Φ t ( u ) ≤ Φ 0 ( u ) + R t ( u ) , \Phi_t(u)\le\Phi_0(u)+R_t(u), Φ t ( u ) ≤ Φ 0 ( u ) + R t ( u ) ,
where Φ 0 ( u ) = A 0 ( f ( v 0 ) − f ( u ) ) + 1 2 ∥ v 0 − u ∥ 2 \Phi_0(u)=A_0(f(v_0)-f(u))+\frac{1}{2}\lVert v_0-u\rVert^2 Φ 0 ( u ) = A 0 ( f ( v 0 ) − f ( u )) + 2 1 ∥ v 0 − u ∥ 2 .
Proof of Lemma 9 . Let j = 1 , … , t j=1,\ldots,t j = 1 , … , t . By Lemma 8 with u = x ⋆ u=x^\star u = x ⋆ , we have
A j ( f ( y j ) − f ⋆ ) + 1 2 ∥ z j − x ⋆ ∥ 2 ≤ A 0 Δ 0 + 1 2 ∥ x 0 − x ⋆ ∥ 2 + R j ( x ⋆ ) ≤ 65 64 R 0 2 . A_j(f(y_j)-f^\star)+\frac{1}{2}\lVert z_j-x^\star\rVert^2\le A_0\Delta_0+\frac{1}{2}\lVert x_0-x^\star\rVert^2+R_j(x^\star)\le\frac{65}{64}R_0^2. A j ( f ( y j ) − f ⋆ ) + 2 1 ∥ z j − x ⋆ ∥ 2 ≤ A 0 Δ 0 + 2 1 ∥ x 0 − x ⋆ ∥ 2 + R j ( x ⋆ ) ≤ 64 65 R 0 2 .
For j = 0 j=0 j = 0 , we have A 0 Δ 0 ≤ R 0 2 / 64 A_0\Delta_0\le R_0^2/64 A 0 Δ 0 ≤ R 0 2 /64 and ∥ z 0 − x ⋆ ∥ = ∥ x 0 − x ⋆ ∥ ≤ R 0 \lVert z_0-x^\star\rVert=\lVert x_0-x^\star\rVert\le R_0 ∥ z 0 − x ⋆ ∥ = ∥ x 0 − x ⋆ ∥ ≤ R 0 . Therefore, for all j = 0 , … , t j=0,\ldots,t j = 0 , … , t , we have
{ f ( y j ) − f ⋆ ≤ 2 R 0 2 A j , ∥ z j − x ⋆ ∥ ≤ 3 2 R 0 . \left\{
\begin{aligned}
f(y_j)-f^\star&\le\frac{2R_0^2}{A_j},\\
\lVert z_j-x^\star\rVert&\le\frac{3}{2}R_0.
\end{aligned}
\right. ⎩ ⎨ ⎧ f ( y j ) − f ⋆ ∥ z j − x ⋆ ∥ ≤ A j 2 R 0 2 , ≤ 2 3 R 0 .
Moreover, A t y t = A 0 x 0 + ∑ j = 1 t α j z j A_ty_t=A_0x_0+\sum_{j=1}^{t}\alpha_jz_j A t y t = A 0 x 0 + ∑ j = 1 t α j z j and A t = A 0 + ∑ j = 1 t α j A_t=A_0+\sum_{j=1}^{t}\alpha_j A t = A 0 + ∑ j = 1 t α j . Then
y t − x ⋆ = A 0 A t ( x 0 − x ⋆ ) + ∑ j = 1 t α j A t ( z j − x ⋆ ) . y_t-x^\star=\frac{A_0}{A_t}(x_0-x^\star)+\sum_{j=1}^{t}\frac{\alpha_j}{A_t}(z_j-x^\star). y t − x ⋆ = A t A 0 ( x 0 − x ⋆ ) + j = 1 ∑ t A t α j ( z j − x ⋆ ) .
This implies that ∥ y t − x ⋆ ∥ ≤ 3 2 R 0 \lVert y_t-x^\star\rVert\le\frac{3}{2}R_0 ∥ y t − x ⋆ ∥ ≤ 2 3 R 0 , and hence
∥ z t − y t ∥ ≤ 3 R 0 . \lVert z_t-y_t\rVert\le3R_0. ∥ z t − y t ∥ ≤ 3 R 0 .
It remains to show that the local smoothness around x t + 1 x_{t+1} x t + 1 is small enough compared with A t + 1 / α t + 1 2 A_{t+1}/\alpha_{t+1}^2 A t + 1 / α t + 1 2 .
By the assumptions on the weights,
α t + 1 A t ≤ 1 64 ℓ ( 1 + L 1 R 0 ) and A t α t + 1 2 ≥ a L 0 . \frac{\alpha_{t+1}}{A_t}\le\frac{1}{64\ell(1+L_1R_0)}
\qquad\text{and}\qquad
\frac{A_t}{\alpha_{t+1}^2}\ge aL_0. A t α t + 1 ≤ 64 ℓ ( 1 + L 1 R 0 ) 1 and α t + 1 2 A t ≥ a L 0 .
Moreover,
A t α t + 1 2 = 1 A t ( A t α t + 1 ) 2 ≥ 64 2 ℓ 2 ( 1 + L 1 R 0 ) 2 A t . \frac{A_t}{\alpha_{t+1}^2}=\frac{1}{A_t}\left(\frac{A_t}{\alpha_{t+1}}\right)^2\ge\frac{64^2\ell^2(1+L_1R_0)^2}{A_t}. α t + 1 2 A t = A t 1 ( α t + 1 A t ) 2 ≥ A t 6 4 2 ℓ 2 ( 1 + L 1 R 0 ) 2 .
We first control the gradient at y t y_t y t . By Lemma 2 and ν ≥ 1 / 2 \nu\ge1/2 ν ≥ 1/2 ,
∥ ∇ f ( y t ) ∥ ≤ 4 L 1 ( f ( y t ) − f ⋆ ) + 2 L 0 ( f ( y t ) − f ⋆ ) . \lVert\nabla f(y_t)\rVert\le4L_1(f(y_t)-f^\star)+2\sqrt{L_0(f(y_t)-f^\star)}. ∥ ∇ f ( y t )∥ ≤ 4 L 1 ( f ( y t ) − f ⋆ ) + 2 L 0 ( f ( y t ) − f ⋆ ) .
Using f ( y t ) − f ⋆ ≤ 2 R 0 2 / A t f(y_t)-f^\star\le2R_0^2/A_t f ( y t ) − f ⋆ ≤ 2 R 0 2 / A t , we obtain ∥ ∇ f ( y t ) ∥ ≤ 8 L 1 R 0 2 A t + 2 2 R 0 L 0 A t \lVert\nabla f(y_t)\rVert\le\frac{8L_1R_0^2}{A_t}+2\sqrt{2}R_0\sqrt{\frac{L_0}{A_t}} ∥ ∇ f ( y t )∥ ≤ A t 8 L 1 R 0 2 + 2 2 R 0 A t L 0 . Therefore,
L 0 + L 1 ∥ ∇ f ( y t ) ∥ ≤ L 0 + 8 L 1 2 R 0 2 A t + 2 2 L 1 R 0 L 0 A t ≤ ( 1 a + 1 512 ℓ 2 + 2 32 ℓ a ) A t α t + 1 2 ≤ 21 512 A t α t + 1 2 . \begin{aligned}
L_0+L_1\lVert\nabla f(y_t)\rVert
&\le L_0+\frac{8L_1^2R_0^2}{A_t}+2\sqrt{2}L_1R_0\sqrt{\frac{L_0}{A_t}}\\
&\le\left(\frac{1}{a}+\frac{1}{512\ell^2}+\frac{\sqrt{2}}{32\ell\sqrt{a}}\right)\frac{A_t}{\alpha_{t+1}^2}\\
&\le\frac{21}{512}\frac{A_t}{\alpha_{t+1}^2}.
\end{aligned} L 0 + L 1 ∥ ∇ f ( y t )∥ ≤ L 0 + A t 8 L 1 2 R 0 2 + 2 2 L 1 R 0 A t L 0 ≤ ( a 1 + 512 ℓ 2 1 + 32 ℓ a 2 ) α t + 1 2 A t ≤ 512 21 α t + 1 2 A t .
where the last inequality uses a ≥ 32 a \ge32 a ≥ 32 and ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 .
We next transfer this bound from y t y_t y t to x t + 1 x_{t+1} x t + 1 . We have x t + 1 − y t = α t + 1 A t + 1 ( z t − y t ) x_{t+1}-y_t=\frac{\alpha_{t+1}}{A_{t+1}}(z_t-y_t) x t + 1 − y t = A t + 1 α t + 1 ( z t − y t ) . Therefore, using ∥ z t − y t ∥ ≤ 3 R 0 \lVert z_t-y_t\rVert\le3R_0 ∥ z t − y t ∥ ≤ 3 R 0 , we obtain
L 1 ∥ x t + 1 − y t ∥ ≤ 3 L 1 R 0 α t + 1 A t + 1 ≤ 3 64 . L_1\lVert x_{t+1}-y_t\rVert\le3L_1R_0\frac{\alpha_{t+1}}{A_{t+1}}\le\frac{3}{64}. L 1 ∥ x t + 1 − y t ∥ ≤ 3 L 1 R 0 A t + 1 α t + 1 ≤ 64 3 .
Thus, by Lemma 3,
L 0 + L 1 ∥ ∇ f ( x t + 1 ) ∥ ≤ e L 1 ∥ x t + 1 − y t ∥ ( L 0 + L 1 ∥ ∇ f ( y t ) ∥ ) ≤ e 3 / 64 21 512 A t α t + 1 2 ≤ 1 8 A t α t + 1 2 ≤ 1 8 A t + 1 α t + 1 2 . \begin{aligned}
L_0+L_1\lVert\nabla f(x_{t+1})\rVert\le e^{L_1\lVert x_{t+1}-y_t\rVert}\left(L_0+L_1\lVert\nabla f(y_t)\rVert\right) \\
&\le e^{3/64}\frac{21}{512}\frac{A_t}{\alpha_{t+1}^2} \\
&\le\frac{1}{8}\frac{A_t}{\alpha_{t+1}^2} \\
&\le\frac{1}{8}\frac{A_{t+1}}{\alpha_{t+1}^2}.
\end{aligned} L 0 + L 1 ∥ ∇ f ( x t + 1 )∥ ≤ e L 1 ∥ x t + 1 − y t ∥ ( L 0 + L 1 ∥ ∇ f ( y t )∥ ) ≤ e 3/64 512 21 α t + 1 2 A t ≤ 8 1 α t + 1 2 A t ≤ 8 1 α t + 1 2 A t + 1 .
It holds that 2 ( L 0 + L 1 ∥ ∇ f ( x t + 1 ) ∥ ) ≤ A t + 1 4 α t + 1 2 2\left(L_0+L_1\lVert\nabla f(x_{t+1})\rVert\right)\le\frac{A_{t+1}}{4\alpha_{t+1}^2} 2 ( L 0 + L 1 ∥ ∇ f ( x t + 1 )∥ ) ≤ 4 α t + 1 2 A t + 1 .
We have y t + 1 − x t + 1 = − α t + 1 2 A t + 1 g t + 1 y_{t+1}-x_{t+1}=-\frac{\alpha_{t+1}^2}{A_{t+1}}g_{t+1} y t + 1 − x t + 1 = − A t + 1 α t + 1 2 g t + 1 and ∥ g t + 1 ∥ ≤ λ t + 1 \lVert g_{t+1}\rVert\le\lambda_{t+1} ∥ g t + 1 ∥ ≤ λ t + 1 , then
L 1 ∥ y t + 1 − x t + 1 ∥ ≤ L 1 α t + 1 2 A t + 1 λ t + 1 = L 1 R 0 ℓ α t + 1 A t + 1 ≤ L 1 R 0 64 ℓ 2 ( 1 + L 1 R 0 ) ≤ 1. \begin{aligned}
L_1\lVert y_{t+1}-x_{t+1}\rVert\le\frac{L_1\alpha_{t+1}^2}{A_{t+1}}\lambda_{t+1} \\
&=\frac{L_1R_0}{\ell}\frac{\alpha_{t+1}}{A_{t+1}} \\
&\le\frac{L_1R_0}{64\ell^2(1+L_1R_0)}\le1.
\end{aligned} L 1 ∥ y t + 1 − x t + 1 ∥ ≤ A t + 1 L 1 α t + 1 2 λ t + 1 = ℓ L 1 R 0 A t + 1 α t + 1 ≤ 64 ℓ 2 ( 1 + L 1 R 0 ) L 1 R 0 ≤ 1.
By Lemma 1, we obtain
f ( y t + 1 ) ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 2 ( L 0 + L 1 ∥ ∇ f ( x t + 1 ) ∥ ) ∥ y t + 1 − x t + 1 ∥ 2 ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + A t + 1 4 α t + 1 2 ∥ y t + 1 − x t + 1 ∥ 2 . \begin{aligned}
f(y_{t+1})\le f(x_{t+1})+\langle\nabla f(x_{t+1}),y_{t+1}-x_{t+1}\rangle+2\left(L_0+L_1\lVert\nabla f(x_{t+1})\rVert\right)\lVert y_{t+1}-x_{t+1}\rVert^2 \\
&\le f(x_{t+1})+\langle\nabla f(x_{t+1}),y_{t+1}-x_{t+1}\rangle+\frac{A_{t+1}}{4\alpha_{t+1}^2}\lVert y_{t+1}-x_{t+1}\rVert^2.
\end{aligned} f ( y t + 1 ) ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 2 ( L 0 + L 1 ∥ ∇ f ( x t + 1 )∥ ) ∥ y t + 1 − x t + 1 ∥ 2 ≤ f ( x t + 1 ) + ⟨ ∇ f ( x t + 1 ) , y t + 1 − x t + 1 ⟩ + 4 α t + 1 2 A t + 1 ∥ y t + 1 − x t + 1 ∥ 2 .
Proof of Lemma 10 . From the proof of Lemma 9 , we have ∥ z t − y t ∥ ≤ 3 R 0 \lVert z_t-y_t\rVert\le3R_0 ∥ z t − y t ∥ ≤ 3 R 0 and
∥ ∇ f ( y t ) ∥ ≤ 8 L 1 R 0 2 A t + 2 2 R 0 L 0 A t . \lVert\nabla f(y_t)\rVert\le\frac{8L_1R_0^2}{A_t}+2\sqrt{2}R_0\sqrt{\frac{L_0}{A_t}}. ∥ ∇ f ( y t )∥ ≤ A t 8 L 1 R 0 2 + 2 2 R 0 A t L 0 .
Therefore, using the assumptions on the weights,
α t + 1 ℓ R 0 ∥ ∇ f ( y t ) ∥ ≤ 8 ℓ L 1 R 0 α t + 1 A t + 2 2 ℓ α t + 1 L 0 A t ≤ 1 8 + 2 2 ℓ a ≤ 3 8 , \begin{aligned}
\frac{\alpha_{t+1}\ell}{R_0}\lVert\nabla f(y_t)\rVert\le8\ell L_1R_0\frac{\alpha_{t+1}}{A_t}+2\sqrt{2}\ell\alpha_{t+1}\sqrt{\frac{L_0}{A_t}} \\
&\le\frac{1}{8}+\frac{2\sqrt{2}\ell}{\sqrt{a}} \\
&\le\frac{3}{8},
\end{aligned} R 0 α t + 1 ℓ ∥ ∇ f ( y t )∥ ≤ 8 ℓ L 1 R 0 A t α t + 1 + 2 2 ℓ α t + 1 A t L 0 ≤ 8 1 + a 2 2 ℓ ≤ 8 3 ,
where the last inequality follows from a ≥ 128 ℓ 2 a\ge128\ell^2 a ≥ 128 ℓ 2 . We next transfer this bound from y t y_t y t to x t + 1 x_{t+1} x t + 1 . Since x t + 1 − y t = α t + 1 A t + 1 ( z t − y t ) x_{t+1}-y_t=\frac{\alpha_{t+1}}{A_{t+1}}(z_t-y_t) x t + 1 − y t = A t + 1 α t + 1 ( z t − y t ) , and ∥ z t − y t ∥ ≤ 3 R 0 \lVert z_t-y_t\rVert\le3R_0 ∥ z t − y t ∥ ≤ 3 R 0 , we have
L 1 ∥ x t + 1 − y t ∥ ≤ 3 L 1 R 0 α t + 1 A t ≤ 3 64 . L_1\lVert x_{t+1}-y_t\rVert\le3L_1R_0\frac{\alpha_{t+1}}{A_t}\le\frac{3}{64}. L 1 ∥ x t + 1 − y t ∥ ≤ 3 L 1 R 0 A t α t + 1 ≤ 64 3 .
Hence, by Lemma 3,
L 0 + L 1 ∥ ∇ f ( x t + 1 ) ∥ ≤ e L 1 ∥ x t + 1 − y t ∥ ( L 0 + L 1 ∥ ∇ f ( y t ) ∥ ) . L_0+L_1\lVert\nabla f(x_{t+1})\rVert\le e^{L_1\lVert x_{t+1}-y_t\rVert}\left(L_0+L_1\lVert\nabla f(y_t)\rVert\right). L 0 + L 1 ∥ ∇ f ( x t + 1 )∥ ≤ e L 1 ∥ x t + 1 − y t ∥ ( L 0 + L 1 ∥ ∇ f ( y t )∥ ) .
and therefore
∥ ∇ f ( x t + 1 ) ∥ ≤ e L 1 ∥ x t + 1 − y t ∥ ∥ ∇ f ( y t ) ∥ + e L 1 ∥ x t + 1 − y t ∥ − 1 L 1 L 0 . \lVert\nabla f(x_{t+1})\rVert\le e^{L_1\lVert x_{t+1}-y_t\rVert}\lVert\nabla f(y_t)\rVert+\frac{e^{L_1\lVert x_{t+1}-y_t\rVert}-1}{L_1}L_0. ∥ ∇ f ( x t + 1 )∥ ≤ e L 1 ∥ x t + 1 − y t ∥ ∥ ∇ f ( y t )∥ + L 1 e L 1 ∥ x t + 1 − y t ∥ − 1 L 0 .
Using L 1 ∥ x t + 1 − y t ∥ ≤ 3 / 64 L_1\lVert x_{t+1}-y_t\rVert\le3/64 L 1 ∥ x t + 1 − y t ∥ ≤ 3/64 and e x − 1 ≤ x e x e^x-1\le xe^x e x − 1 ≤ x e x for x ≥ 0 x\ge0 x ≥ 0 , we obtain
e L 1 ∥ x t + 1 − y t ∥ − 1 L 1 L 0 ≤ e 3 / 64 L 0 ∥ x t + 1 − y t ∥ ≤ 3 e 3 / 64 L 0 R 0 α t + 1 A t . \frac{e^{L_1\lVert x_{t+1}-y_t\rVert}-1}{L_1}L_0\le e^{3/64}L_0\lVert x_{t+1}-y_t\rVert\le3e^{3/64}L_0R_0\frac{\alpha_{t+1}}{A_t}. L 1 e L 1 ∥ x t + 1 − y t ∥ − 1 L 0 ≤ e 3/64 L 0 ∥ x t + 1 − y t ∥ ≤ 3 e 3/64 L 0 R 0 A t α t + 1 .
Thus,
α t + 1 ℓ R 0 ∥ ∇ f ( x t + 1 ) ∥ ≤ e 3 / 64 α t + 1 ℓ R 0 ∥ ∇ f ( y t ) ∥ + 3 e 3 / 64 ℓ L 0 α t + 1 2 A t ≤ e 3 / 64 ( 3 8 + 3 ℓ a ) < 1 2 , \begin{aligned}
\frac{\alpha_{t+1}\ell}{R_0}\lVert\nabla f(x_{t+1})\rVert\le e^{3/64}\frac{\alpha_{t+1}\ell}{R_0}\lVert\nabla f(y_t)\rVert+3e^{3/64}\ell\frac{L_0\alpha_{t+1}^2}{A_t} \\
&\le e^{3/64}\left(\frac{3}{8}+\frac{3\ell}{a}\right) \\
&<\frac{1}{2},
\end{aligned} R 0 α t + 1 ℓ ∥ ∇ f ( x t + 1 )∥ ≤ e 3/64 R 0 α t + 1 ℓ ∥ ∇ f ( y t )∥ + 3 e 3/64 ℓ A t L 0 α t + 1 2 ≤ e 3/64 ( 8 3 + a 3 ℓ ) < 2 1 ,
where the last inequality follows from a ≥ 128 ℓ 2 a\ge128\ell^2 a ≥ 128 ℓ 2 , ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 , and e 3 / 64 < 16 / 15 e^{3/64}<16/15 e 3/64 < 16/15 . Consequently,
∥ ∇ f ( x t + 1 ) ∥ ≤ R 0 2 α t + 1 ℓ = λ t + 1 2 . \lVert\nabla f(x_{t+1})\rVert\le\frac{R_0}{2\alpha_{t+1}\ell}=\frac{\lambda_{t+1}}{2}. ∥ ∇ f ( x t + 1 )∥ ≤ 2 α t + 1 ℓ R 0 = 2 λ t + 1 .
□ \square □
Proof of Lemma 11. For t = 1 , … , K t=1,\ldots,K t = 1 , … , K , define
S t : = α α α α − 1 R 0 α ∑ i = 1 t α i α . S_t:=\frac{\alpha^\alpha\alpha^{\alpha-1}}{R_0^\alpha}\sum_{i=1}^{t}\alpha_i^\alpha. S t := R 0 α α α α α − 1 i = 1 ∑ t α i α .
By assumption, we have S t ≤ 1 512 S_t\le\frac{1}{512} S t ≤ 512 1 .
Using the proof of Lemma 9, for every i = 1 , … , t i=1,\ldots,t i = 1 , … , t , on E i − 1 \mathcal{E}_{i-1} E i − 1 ,
∥ x ⋆ − z i − 1 ∥ ≤ 3 2 R 0 . \lVert x^\star-z_{i-1}\rVert\le\frac{3}{2}R_0. ∥ x ⋆ − z i − 1 ∥ ≤ 2 3 R 0 .
We first control the predictable term B t B_t B t . Using the clipping bias and conditional second-moment bounds,
B t ≤ ∑ i = 1 t 1 E i − 1 ( 3 2 α σ α R 0 α i λ i 1 − α + 2 σ α α i 2 λ i 2 − α ) . B_t\le\sum_{i=1}^{t}\mathbf{1}_{\mathcal{E}_{i-1}}\left(\frac{3}{2}\alpha^\sigma\alpha R_0\alpha_i\lambda_i^{1-\alpha}+2\sigma^\alpha\alpha_i^2\lambda_i^{2-\alpha}\right). B t ≤ i = 1 ∑ t 1 E i − 1 ( 2 3 α σ α R 0 α i λ i 1 − α + 2 σ α α i 2 λ i 2 − α ) .
Since λ i = R 0 / ( α i ℓ ) \lambda_i=R_0/(\alpha_i\ell) λ i = R 0 / ( α i ℓ ) ,
{ α i λ i 1 − α = α i α R 0 1 − α ℓ α − 1 , α i 2 λ i 2 − α = α i α R 0 2 − α ℓ α − 2 . \begin{cases}
\alpha_i\lambda_i^{1-\alpha}=\alpha_i^\alpha R_0^{1-\alpha}\ell^{\alpha-1},\\
\alpha_i^2\lambda_i^{2-\alpha}=\alpha_i^\alpha R_0^{2-\alpha}\ell^{\alpha-2}.
\end{cases} { α i λ i 1 − α = α i α R 0 1 − α ℓ α − 1 , α i 2 λ i 2 − α = α i α R 0 2 − α ℓ α − 2 .
Therefore,
B t ≤ ( 3 ⋅ 2 α − 1 + 2 ℓ ) R 0 2 S t ≤ 8 R 0 2 S t ≤ R 0 2 64 . B_t \le\left(3 \cdot2^{\alpha-1}+\frac{2}{\ell}\right)R_0^2S_t \le8R_0^2S_t \le\frac{R_0^2}{64}. B t ≤ ( 3 ⋅ 2 α − 1 + ℓ 2 ) R 0 2 S t ≤ 8 R 0 2 S t ≤ 64 R 0 2 .
where we used 2 α ≤ 4 2^\alpha\le4 2 α ≤ 4 and ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 .
We next control the martingale term M t M_t M t . Let X i X_i X i denote its i i i -th increment. On E i − 1 \mathcal{E}_{i-1} E i − 1 ,
{ ∥ θ i − E [ θ i ∣ F i − 1 ] ∥ ≤ 3 λ i , ∣ ∥ θ i ∥ 2 − E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ∣ ≤ 9 4 λ i 2 . \begin{cases}
\left\|\theta_i-\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}]\right\| \le3\lambda_i,\\
\left|\|\theta_i\|^2-\mathbb{E}[\|\theta_i\|^2\mid\mathcal{F}_{i-1}]\right| \le\frac{9}{4}\lambda_i^2.
\end{cases} { ∥ θ i − E [ θ i ∣ F i − 1 ] ∥ ≤ 3 λ i , ∥ θ i ∥ 2 − E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 4 9 λ i 2 .
Therefore,
∣ X i ∣ ≤ 3 α i λ i ∥ x ⋆ − z i − 1 ∥ + 9 4 α i 2 λ i 2 ≤ 9 2 α i λ i R 0 + 9 4 α i 2 λ i 2 = ( 9 2 ℓ + 9 4 ℓ 2 ) R 0 2 ≤ 19 4 ℓ R 0 2 , \begin{aligned}
|X_i| &\le3\alpha_i\lambda_i\|x^\star-z_{i-1}\|+\frac{9}{4}\alpha_i^2\lambda_i^2\\
&\le\frac{9}{2}\alpha_i\lambda_iR_0+\frac{9}{4}\alpha_i^2\lambda_i^2\\
&=\left(\frac{9}{2\ell}+\frac{9}{4\ell^2}\right)R_0^2\le\frac{19}{4\ell}R_0^2,
\end{aligned} ∣ X i ∣ ≤ 3 α i λ i ∥ x ⋆ − z i − 1 ∥ + 4 9 α i 2 λ i 2 ≤ 2 9 α i λ i R 0 + 4 9 α i 2 λ i 2 = ( 2 ℓ 9 + 4 ℓ 2 9 ) R 0 2 ≤ 4 ℓ 19 R 0 2 ,
where the last inequality uses ℓ ≥ 8 log ( 4 K / δ ) > 9 \ell\ge8\log(4K/\delta)>9 ℓ ≥ 8 log ( 4 K / δ ) > 9 .
We now bound the conditional variances. On E i − 1 \mathcal{E}_{i-1} E i − 1 ,
{ E [ ∥ θ i − E [ θ i ∣ F i − 1 ] ∥ 2 ∣ F i − 1 ] ≤ E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 2 σ α λ i 2 − α , E [ ∥ θ i ∥ 4 ∣ F i − 1 ] ≤ 9 4 λ i 2 E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 9 2 σ α λ i 4 − α . \begin{cases}
\mathbb{E}\left[\left\|\theta_i-\mathbb{E}[\theta_i\mid\mathcal{F}_{i-1}]\right\|^2\mid\mathcal{F}_{i-1}\right]\le\mathbb{E}[\|\theta_i\|^2\mid\mathcal{F}_{i-1}]\le2\sigma^\alpha\lambda_i^{2-\alpha},\\
\mathbb{E}[\|\theta_i\|^4\mid\mathcal{F}_{i-1}]\le\frac{9}{4}\lambda_i^2\mathbb{E}[\|\theta_i\|^2\mid\mathcal{F}_{i-1}]\le\frac{9}{2}\sigma^\alpha\lambda_i^{4-\alpha}.
\end{cases} { E [ ∥ θ i − E [ θ i ∣ F i − 1 ] ∥ 2 ∣ F i − 1 ] ≤ E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 2 σ α λ i 2 − α , E [ ∥ θ i ∥ 4 ∣ F i − 1 ] ≤ 4 9 λ i 2 E [ ∥ θ i ∥ 2 ∣ F i − 1 ] ≤ 2 9 σ α λ i 4 − α .
Since 1 E i − 1 \mathbf{1}_{\mathcal{E}_{i-1}} 1 E i − 1 is F i − 1 \mathcal{F}_{i-1} F i − 1 -measurable, using ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^2\le2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2 and ∥ x ⋆ − z i − 1 ∥ ≤ 3 2 R 0 \|x^\star-z_{i-1}\|\le\frac{3}{2}R_0 ∥ x ⋆ − z i − 1 ∥ ≤ 2 3 R 0 , we obtain
E [ X i 2 ∣ F i − 1 ] ≤ 9 σ α α i 2 R 0 2 λ i 2 − α + 9 σ α α i 4 λ i 4 − α . \mathbb{E}[X_i^2\mid\mathcal{F}_{i-1}]\le9\sigma^\alpha\alpha_i^2R_0^2\lambda_i^{2-\alpha}+9\sigma^\alpha\alpha_i^4\lambda_i^{4-\alpha}. E [ X i 2 ∣ F i − 1 ] ≤ 9 σ α α i 2 R 0 2 λ i 2 − α + 9 σ α α i 4 λ i 4 − α .
Since λ i = R 0 / ( α i ℓ ) \lambda_i=R_0/(\alpha_i\ell) λ i = R 0 / ( α i ℓ ) , summing over i = 1 , … , t i=1,\ldots,t i = 1 , … , t , we obtain
∑ i = 1 t E [ X i 2 ∣ F i − 1 ] ≤ 9 R 0 4 ( 1 ℓ + 1 ℓ 3 ) S t ≤ 10 R 0 4 512 ℓ , \sum_{i=1}^{t}\mathbb{E}[X_i^2\mid\mathcal{F}_{i-1}]
\le9R_0^4\left(\frac{1}{\ell}+\frac{1}{\ell^3}\right)S_t
\le\frac{10R_0^4}{512\ell}, i = 1 ∑ t E [ X i 2 ∣ F i − 1 ] ≤ 9 R 0 4 ( ℓ 1 + ℓ 3 1 ) S t ≤ 512 ℓ 10 R 0 4 ,
where we again used ℓ > 9 \ell>9 ℓ > 9 and S t ≤ 1 / 512 S_t\le1/512 S t ≤ 1/512 .
Fix t ∈ { 1 , … , K } t\in\{1,\ldots,K\} t ∈ { 1 , … , K } . Applying Proposition 2 with failure probability δ / K \delta/K δ / K , we obtain, with probability at least 1 − δ / K 1-\delta/K 1 − δ / K ,
∣ M t ∣ ≤ R 0 2 ( 20 512 ℓ log 2 K δ + 19 6 ℓ log 2 K δ ) ≤ R 0 2 ( 5 32 + 19 48 ) < 31 64 R 0 2 , \begin{aligned}
|M_t|\le R_0^2\left(\sqrt{\frac{20}{512\ell}\log\frac{2K}{\delta}}+\frac{19}{6\ell}\log\frac{2K}{\delta}\right) \\
&\le R_0^2\left(\frac{\sqrt{5}}{32}+\frac{19}{48}\right) \\
&<\frac{31}{64}R_0^2,
\end{aligned} ∣ M t ∣ ≤ R 0 2 ( 512 ℓ 20 log δ 2 K + 6 ℓ 19 log δ 2 K ) ≤ R 0 2 ( 32 5 + 48 19 ) < 64 31 R 0 2 ,
where we used ℓ ≥ 8 log 4 K δ ≥ 8 log 2 K δ \ell\ge8\log\frac{4K}{\delta}\ge8\log\frac{2K}{\delta} ℓ ≥ 8 log δ 4 K ≥ 8 log δ 2 K .
Taking a union bound over t = 1 , … , K t=1,\ldots,K t = 1 , … , K , with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every t = 1 , … , K t=1,\ldots,K t = 1 , … , K ,
∣ M t ∣ < 31 64 R 0 2 . |M_t|<\frac{31}{64}R_0^2. ∣ M t ∣ < 64 31 R 0 2 .
On this event,
R ^ t ( x ∗ ) = M t + B t ≤ ∣ M t ∣ + B t < 31 64 R 0 2 + 1 64 R 0 2 = R 0 2 2 . \widehat{R}_t(x^*) = M_t + B_t \leq|M_t| + B_t < \frac{31}{64}R_0^2 + \frac{1}{64}R_0^2 = \frac{R_0^2}{2}. R t ( x ∗ ) = M t + B t ≤ ∣ M t ∣ + B t < 64 31 R 0 2 + 64 1 R 0 2 = 2 R 0 2 .
It remains to show that E K \mathcal{E}_K E K holds on this event. We argue by induction. For t = 0 t = 0 t = 0 , there are no conditions, so 1 E 0 = 1 \mathbf{1}_{\mathcal{E}_0} = 1 1 E 0 = 1 . Suppose that E t − 1 \mathcal{E}_{t-1} E t − 1 holds for some t ∈ { 1 , … , K } t \in\{1,\ldots,K\} t ∈ { 1 , … , K } . Applying Lemma 9 with t − 1 t-1 t − 1 gives
f ( y t ) ≤ f ( x t ) + ⟨ ∇ f ( x t ) , y t − x t ⟩ + A t 4 α t 2 ∥ y t − x t ∥ 2 . f(y_t) \leq f(x_t) + \langle\nabla f(x_t), y_t - x_t \rangle+ \frac{A_t}{4\alpha_t^2}\lVert y_t-x_t\rVert^2. f ( y t ) ≤ f ( x t ) + ⟨ ∇ f ( x t ) , y t − x t ⟩ + 4 α t 2 A t ∥ y t − x t ∥ 2 .
Moreover, since E t − 1 \mathcal{E}_{t-1} E t − 1 holds,
R ^ t ( x ∗ ) = R t ( x ∗ ) ≤ R 0 2 2 . \widehat{R}_t(x^*) = R_t(x^*) \leq\frac{R_0^2}{2}. R t ( x ∗ ) = R t ( x ∗ ) ≤ 2 R 0 2 .
Thus all the conditions defining E t \mathcal{E}_t E t hold. By induction, E K \mathcal{E}_K E K holds.
Proof of Lemma 12. We first show that, for every k ≥ 0 k \geq0 k ≥ 0 ,
a L 0 A k + 1 − a L 0 A k ≤ 1 2 . \sqrt{aL_0A_{k+1}}-\sqrt{aL_0A_k}\leq\frac{1}{2}. a L 0 A k + 1 − a L 0 A k ≤ 2 1 .
If A k ≤ [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 A_k \leq\frac{[64\ell(1+L_1R_0)]^2}{aL_0} A k ≤ a L 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 , then A k + 1 = ( 1 + 1 64 ℓ ( 1 + L 1 R 0 ) ) A k A_{k+1}=\left(1+\frac{1}{64\ell(1+L_1R_0)}\right)A_k A k + 1 = ( 1 + 64 ℓ ( 1 + L 1 R 0 ) 1 ) A k . Therefore,
a L 0 A k + 1 − a L 0 A k = a L 0 A k / [ 64 ℓ ( 1 + L 1 R 0 ) ] 1 + 1 / [ 64 ℓ ( 1 + L 1 R 0 ) ] + 1 ≤ 1 2 . \sqrt{aL_0A_{k+1}}-\sqrt{aL_0A_k}
=
\frac{\sqrt{aL_0A_k}/[64\ell(1+L_1R_0)]}{\sqrt{1+1/[64\ell(1+L_1R_0)]}+1}
\leq\frac{1}{2}. a L 0 A k + 1 − a L 0 A k = 1 + 1/ [ 64 ℓ ( 1 + L 1 R 0 )] + 1 a L 0 A k / [ 64 ℓ ( 1 + L 1 R 0 )] ≤ 2 1 .
If A k ≥ [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 A_k \geq\frac{[64\ell(1+L_1R_0)]^2}{aL_0} A k ≥ a L 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 , then A k + 1 = A k + A k a L 0 A_{k+1}=A_k+\sqrt{\frac{A_k}{aL_0}} A k + 1 = A k + a L 0 A k . Therefore
a L 0 A k + 1 − a L 0 A k = a L 0 ( A k + 1 − A k ) a L 0 A k + 1 + a L 0 A k = a L 0 A k a L 0 A k + 1 + a L 0 A k ≤ 1 2 . \begin{aligned}
\sqrt{aL_0A_{k+1}}-\sqrt{aL_0A_k}
&=\frac{aL_0(A_{k+1}-A_k)}{\sqrt{aL_0A_{k+1}}+\sqrt{aL_0A_k}}\\
&=\frac{\sqrt{aL_0A_k}}{\sqrt{aL_0A_{k+1}}+\sqrt{aL_0A_k}}\leq\frac{1}{2}.
\end{aligned} a L 0 A k + 1 − a L 0 A k = a L 0 A k + 1 + a L 0 A k a L 0 ( A k + 1 − A k ) = a L 0 A k + 1 + a L 0 A k a L 0 A k ≤ 2 1 .
Since A 0 ≤ 1 / ( a L 0 ) A_0\leq1/(aL_0) A 0 ≤ 1/ ( a L 0 ) , it follows that a L 0 A k ≤ 1 + k 2 = k + 2 2 \sqrt{aL_0A_k}\leq1+\frac{k}{2}=\frac{k+2}{2} a L 0 A k ≤ 1 + 2 k = 2 k + 2 . Using α k + 1 ≤ A k a L 0 = a L 0 A k a L 0 \alpha_{k+1}\leq\sqrt{\frac{A_k}{aL_0}}=\frac{\sqrt{aL_0A_k}}{aL_0} α k + 1 ≤ a L 0 A k = a L 0 a L 0 A k , we obtain
α k + 1 ≤ k + 2 2 a L 0 . \alpha_{k+1}\leq\frac{k+2}{2aL_0}. α k + 1 ≤ 2 a L 0 k + 2 .
Thus, for every i ≥ 1 i\geq1 i ≥ 1 , we have α i ≤ i + 1 2 a L 0 \alpha_i\leq\frac{i+1}{2aL_0} α i ≤ 2 a L 0 i + 1 .
Consequently,
∑ i = 1 K α i α ≤ 1 2 α ( a L 0 ) α ∑ i = 1 K ( i + 1 ) α ≤ K ( K + 1 ) α 2 α ( a L 0 ) α . \sum_{i=1}^{K}\alpha_i^\alpha\leq\frac{1}{2^\alpha(aL_0)^\alpha}\sum_{i=1}^{K}(i+1)^\alpha\leq\frac{K(K+1)^\alpha}{2^\alpha(aL_0)^\alpha}. i = 1 ∑ K α i α ≤ 2 α ( a L 0 ) α 1 i = 1 ∑ K ( i + 1 ) α ≤ 2 α ( a L 0 ) α K ( K + 1 ) α .
It remains to lower-bound A K A_K A K . Define
m : = min { k ≥ 0 : A k ≥ [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 } . m:=\min\left\{k\geq0:A_k\geq\frac{[64\ell(1+L_1R_0)]^2}{aL_0}\right\}. m := min { k ≥ 0 : A k ≥ a L 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 } .
Since A 0 > 0 A_0>0 A 0 > 0 , if A k < [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 A_k<\frac{[64\ell(1+L_1R_0)]^2}{aL_0} A k < a L 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 for every k ≥ 0 k\geq0 k ≥ 0 , then
A k = ( 1 + 1 64 ℓ ( 1 + L 1 R 0 ) ) k A 0 → ∞ , A_k=\left(1+\frac{1}{64\ell(1+L_1R_0)}\right)^kA_0\to\infty, A k = ( 1 + 64 ℓ ( 1 + L 1 R 0 ) 1 ) k A 0 → ∞ ,
which is impossible. Hence m m m is well-defined.
We first show that m ≤ K 0 m \le K_0 m ≤ K 0 . Suppose, by contradiction, that m > K 0 m > K_0 m > K 0 . Then, for every k = 0 , … , K 0 − 1 k = 0,\ldots,K_0 - 1 k = 0 , … , K 0 − 1 , we have A k < [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 A_k < \frac{[64\ell(1+L_1R_0)]^2}{aL_0} A k < a L 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 , and therefore
A K 0 = ( 1 + 1 64 ℓ ( 1 + L 1 R 0 ) ) K 0 A 0 . A_{K_0}=\left(1+\frac{1}{64\ell(1+L_1R_0)}\right)^{K_0}A_0. A K 0 = ( 1 + 64 ℓ ( 1 + L 1 R 0 ) 1 ) K 0 A 0 .
Since log ( 1 + x ) ≥ x / 2 \log(1+x)\ge x/2 log ( 1 + x ) ≥ x /2 for all x ∈ [ 0 , 1 ] x\in[0,1] x ∈ [ 0 , 1 ] , it holds that
log ( 1 + 1 64 ℓ ( 1 + L 1 R 0 ) ) ≥ 1 128 ℓ ( 1 + L 1 R 0 ) . \log\left(1+\frac{1}{64\ell(1+L_1R_0)}\right)\ge\frac{1}{128\ell(1+L_1R_0)}. log ( 1 + 64 ℓ ( 1 + L 1 R 0 ) 1 ) ≥ 128 ℓ ( 1 + L 1 R 0 ) 1 .
Thus,
A K 0 = A 0 exp ( K 0 log ( 1 + 1 64 ℓ ( 1 + L 1 R 0 ) ) ) ≥ A 0 exp ( K 0 128 ℓ ( 1 + L 1 R 0 ) ) ≥ A 0 exp ( log ( [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 A 0 ) ) = [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 a L 0 , \begin{aligned}
A_{K_0}
&=A_0\exp\left(K_0\log\left(1+\frac{1}{64\ell(1+L_1R_0)}\right)\right)\\
&\ge A_0\exp\left(\frac{K_0}{128\ell(1+L_1R_0)}\right)\\
&\ge A_0\exp\left(\log\left(\frac{[64\ell(1+L_1R_0)]^2}{aL_0A_0}\right)\right)\\
&=\frac{[64\ell(1+L_1R_0)]^2}{aL_0},
\end{aligned} A K 0 = A 0 exp ( K 0 log ( 1 + 64 ℓ ( 1 + L 1 R 0 ) 1 ) ) ≥ A 0 exp ( 128 ℓ ( 1 + L 1 R 0 ) K 0 ) ≥ A 0 exp ( log ( a L 0 A 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 ) ) = a L 0 [ 64 ℓ ( 1 + L 1 R 0 ) ] 2 ,
which contradicts m > K 0 m>K_0 m > K 0 . Thus m ≤ K 0 m\le K_0 m ≤ K 0 . For every k ≥ m k\ge m k ≥ m , we have
A k + 1 = A k + A k a L 0 . A_{k+1}=A_k+\sqrt{\frac{A_k}{aL_0}}. A k + 1 = A k + a L 0 A k .
Moreover, a L 0 A k ≥ 64 ℓ ( 1 + L 1 R 0 ) ≥ 1 \sqrt{aL_0A_k}\ge64\ell(1+L_1R_0)\ge1 a L 0 A k ≥ 64 ℓ ( 1 + L 1 R 0 ) ≥ 1 . Therefore,
a L 0 A k + 1 − a L 0 A k = 1 1 + 1 / a L 0 A k + 1 ≥ 1 2 + 1 > 2 5 . \sqrt{aL_0A_{k+1}}-\sqrt{aL_0A_k}
=\frac{1}{\sqrt{1+1/\sqrt{aL_0A_k}+1}}
\ge\frac{1}{\sqrt{2}+1}>\frac{2}{5}. a L 0 A k + 1 − a L 0 A k = 1 + 1/ a L 0 A k + 1 1 ≥ 2 + 1 1 > 5 2 .
Thus, for every K ≥ K 0 K\ge K_0 K ≥ K 0 ,
a L 0 A K ≥ 2 5 ( K − m ) ≥ 2 5 ( K − K 0 ) . \sqrt{aL_0A_K}\ge\frac{2}{5}(K-m)\ge\frac{2}{5}(K-K_0). a L 0 A K ≥ 5 2 ( K − m ) ≥ 5 2 ( K − K 0 ) .
We deduce that
A K ≥ 4 ( K − K 0 ) 2 25 a L 0 . A_K\ge\frac{4(K-K_0)^2}{25aL_0}. A K ≥ 25 a L 0 4 ( K − K 0 ) 2 .
Proof of Theorem 2. By the definition of the weights, for every k = 0 , … , K − 1 k=0,\ldots,K-1 k = 0 , … , K − 1 ,
{ α k + 1 A k ≤ 1 64 ℓ ( 1 + L 1 R 0 ) , A k ≥ a L 0 α k + 1 2 . \left\{
\begin{aligned}
\frac{\alpha_{k+1}}{A_k}&\le\frac{1}{64\ell(1+L_1R_0)},\\
A_k&\ge aL_0\alpha_{k+1}^2.
\end{aligned}
\right. ⎩ ⎨ ⎧ A k α k + 1 A k ≤ 64 ℓ ( 1 + L 1 R 0 ) 1 , ≥ a L 0 α k + 1 2 .
Moreover, A 0 Δ 0 ≤ A 0 Δ ‾ 0 ≤ R 0 2 / 64 A_0\Delta_0\le A_0\overline{\Delta}_0\le R_0^2/64 A 0 Δ 0 ≤ A 0 Δ 0 ≤ R 0 2 /64 , ℓ = 8 log ( 4 K / δ ) \ell=8\log(4K/\delta) ℓ = 8 log ( 4 K / δ ) , and a ≥ 128 ℓ 2 a\ge128\ell^2 a ≥ 128 ℓ 2 . It remains to verify the last condition of Lemma 11. By Lemma 12 and K + 1 ≤ 2 K K+1\le2K K + 1 ≤ 2 K ,
∑ i = 1 K α i α ≤ K ( K + 1 ) α 2 α ( a L 0 ) α ≤ K α + 1 ( a L 0 ) α . \sum_{i=1}^{K}\alpha_i^\alpha\le\frac{K(K+1)^\alpha}{2^\alpha(aL_0)^\alpha}\le\frac{K^{\alpha+1}}{(aL_0)^\alpha}. i = 1 ∑ K α i α ≤ 2 α ( a L 0 ) α K ( K + 1 ) α ≤ ( a L 0 ) α K α + 1 .
By definition of a a a , it holds that
σ α α ℓ − 1 R 0 α ∑ i = 1 K α i α ≤ σ α α ℓ − 1 K α + 1 R 0 α ( a L 0 ) α ≤ 1 512 . \frac{\sigma^{\alpha}\alpha^{\ell-1}}{R_0^{\alpha}}\sum_{i=1}^{K}\alpha_i^{\alpha}\le\frac{\sigma^{\alpha}\alpha^{\ell-1}K^{\alpha+1}}{R_0^{\alpha}(aL_0)^{\alpha}}\le\frac{1}{512}. R 0 α σ α α ℓ − 1 i = 1 ∑ K α i α ≤ R 0 α ( a L 0 ) α σ α α ℓ − 1 K α + 1 ≤ 512 1 .
Therefore, all the conditions of Lemma 11 are satisfied. Then, with probability at least 1 − δ 1-\delta 1 − δ , the event E K \mathcal{E}_K E K holds and
R K ( x ⋆ ) = R ^ K ( x ⋆ ) ≤ R 0 2 2 . \mathcal{R}_K(x^\star)=\widehat{\mathcal{R}}_K(x^\star)\le\frac{R_0^2}{2}. R K ( x ⋆ ) = R K ( x ⋆ ) ≤ 2 R 0 2 .
On this event, by applying Lemma 8 with u = x ⋆ u=x^\star u = x ⋆ , we obtain
A K ( f ( y K ) − f ⋆ ) + 1 2 ∥ z K − x ⋆ ∥ 2 ≤ A 0 Δ 0 + 1 2 ∥ x 0 − x ⋆ ∥ 2 + R K ( x ⋆ ) ≤ 65 64 R 0 2 . A_K(f(y_K)-f^\star)+\frac{1}{2}\lVert z_K-x^\star\rVert^2\le A_0\Delta_0+\frac{1}{2}\lVert x_0-x^\star\rVert^2+\mathcal{R}_K(x^\star)\le\frac{65}{64}R_0^2. A K ( f ( y K ) − f ⋆ ) + 2 1 ∥ z K − x ⋆ ∥ 2 ≤ A 0 Δ 0 + 2 1 ∥ x 0 − x ⋆ ∥ 2 + R K ( x ⋆ ) ≤ 64 65 R 0 2 .
Consequently,
f ( y K ) − f ⋆ ≤ 65 R 0 2 64 A K . f(y_K)-f^\star\le\frac{65R_0^2}{64A_K}. f ( y K ) − f ⋆ ≤ 64 A K 65 R 0 2 .
Since K ≥ 2 K 0 K\ge2K_0 K ≥ 2 K 0 , Lemma 12 yields
A K ≥ 4 ( K − K 0 ) 2 25 a L 0 ≥ K 2 25 a L 0 , A_K\ge\frac{4(K-K_0)^2}{25aL_0}\ge\frac{K^2}{25aL_0}, A K ≥ 25 a L 0 4 ( K − K 0 ) 2 ≥ 25 a L 0 K 2 ,
and therefore
f ( y K ) − f ⋆ ≤ 1625 64 a L 0 R 0 2 K 2 . f(y_K)-f^\star\le\frac{1625}{64}\frac{aL_0R_0^2}{K^2}. f ( y K ) − f ⋆ ≤ 64 1625 K 2 a L 0 R 0 2 .
Using the definition of a a a and max { x , y } ≤ x + y \max\{x,y\}\le x+y max { x , y } ≤ x + y , we obtain
f ( y K ) − f ⋆ ≤ 3250 L 0 R 0 2 ℓ 2 K 2 + 1625 64 512 1 / α σ R 0 ℓ ( α − 1 ) / α K ( α − 1 ) / α . f(y_K)-f^\star\le3250\frac{L_0R_0^2\ell^2}{K^2}+\frac{1625}{64}\frac{512^{1/\alpha}\sigma R_0\ell^{(\alpha-1)/\alpha}}{K^{(\alpha-1)/\alpha}}. f ( y K ) − f ⋆ ≤ 3250 K 2 L 0 R 0 2 ℓ 2 + 64 1625 K ( α − 1 ) / α 51 2 1/ α σ R 0 ℓ ( α − 1 ) / α .