Introduction
The length of a training run is rarely fixed before optimization begins. Early stopping (Aolaritei and Jordan, [2 ] ) and checkpoint selection pick a stopping point from the observed trajectory, large-scale runs increasingly avoid committing to a fixed duration (Hägele et al., [23 ] ), and schedule-free methods are designed so that no stopping time has to be specified (Defazio et al., [13 ] ). The model that is kept is the state of the run at the chosen time rather than an average of past iterates. In such runs, the stochastic gradients are often heavy-tailed ([61 ] (Zhang et al. 2020)), and gradient clipping (Pascanu et al., [53 ] ) is a standard remedy used routinely in training large language models (Touvron et al., [66 ] ). We ask what an unknown horizon costs: how fast can the last iterate of clipped stochastic gradient descent (SGD; Robbins and Monro [56 ] ) converge with high probability when the run may stop at a data-dependent time?
Fixed-time guarantees do not answer this question. A bound that holds with probability at least 1 − δ 1-\delta 1 − δ at each deterministic step t t t relies on an event that may change with t t t , so the bound can fail when the stopping time is chosen from the observed trajectory, just as continuously monitoring an A/B test with a fixed-sample rule inflates Type I error [25 ] . A guarantee holds at every bounded stopping time if and only if it is time-uniform , meaning that a single event of probability at least 1 − δ 1-\delta 1 − δ covers all steps t ≥ 1 t \ge1 t ≥ 1 simultaneously (Section 2). Under sub-Gaussian noise, time-uniform guarantees pay only logarithmic factors beyond the fixed-horizon exponent 1 / 2 1/2 1/2 (Li et al., 2026).
To study heavy-tailed noise, we consider a differentiable convex L L L -smooth objective f : R d → R f:\mathbb{R}^{d}\to\mathbb{R} f : R d → R with a minimizer x ⋆ x^{\star} x ⋆ satisfying ∥ x 1 − x ⋆ ∥ ≤ R \lVert x_{1}-x^{\star}\rVert\le R ∥ x 1 − x ⋆ ∥ ≤ R , and conditionally unbiased stochastic gradients g t g_t g t queried at x t x_t x t that satisfy
E [ ∥ g t − ∇ f ( x t ) ∥ p ∣ F t − 1 ] ≤ σ p , 1 < p ≤ 2. \mathbb{E}\left[\lVert g_t-\nabla f(x_t)\rVert^p \mid\mathcal{F}_{t-1}\right]\le\sigma^p,\qquad1<p\le2. E [ ∥ g t − ∇ f ( x t ) ∥ p ∣ F t − 1 ] ≤ σ p , 1 < p ≤ 2.
Clipped SGD updates its iterate according to
x t + 1 = x t − η t clip ( g t , λ t ) , clip ( g , λ ) = min { 1 , λ ∥ g ∥ } g , (1) \begin{aligned}
x_{t+1}&=x_t-\eta_t\operatorname{clip}(g_t,\lambda_t),\\
\operatorname{clip}(g,\lambda)&=\min\left\{1,\frac{\lambda}{\lVert g\rVert}\right\}g, \tag*{(1)}
\end{aligned} x t + 1 clip ( g , λ ) = x t − η t clip ( g t , λ t ) , = min { 1 , ∥ g ∥ λ } g , (1)
where η t \eta_t η t is the step size and λ t \lambda_t λ t is the clipping threshold. Without clipping, SGD with predefined step sizes attains the minimax optimal rate in expectation on convex problems whose stochastic gradients have bounded p p p th moments [18 ] , but its high-probability guarantees can require polynomial dependence on 1 / δ 1/\delta 1/ δ [58 , 18 ] . Clipping achieves logarithmic dependence on 1 / δ 1/\delta 1/ δ [43 , 21 , 10 , 58 , 48 ] , as in robust mean estimation [5 , 14 , 41 ] . Most of these high-probability guarantees apply to an averaged or auxiliary output rather than the last iterate. For an averaged output of clipped stochastic mirror descent, Nguyen et al. [48 ] proved that T T T oracle calls suffice to reach, with probability at least 1 − δ 1-\delta 1 − δ , the statistical scale
σ R ( log ( 1 / δ ) T ) ( p − 1 ) / p (2) \sigma R\left(\frac{\log(1/\delta)}{T}\right)^{(p-1)/p} \tag*{(2)} σ R ( T log ( 1/ δ ) ) ( p − 1 ) / p (2)
plus a deterministic term of order L R 2 log ( 1 / δ ) / T LR^2\log(1/\delta)/T L R 2 log ( 1/ δ ) / T . We write α p = ( p − 1 ) / p \alpha_p=(p-1)/p α p = ( p − 1 ) / p for the statistical exponent in (2) . For δ ≤ 1 / 8 \delta\le1/8 δ ≤ 1/8 and T ≥ log ( 1 / δ ) T\ge\log(1/\delta) T ≥ log ( 1/ δ ) , Jiang et al. [28 ] proved that no algorithm using T T T stochastic gradients improves on the minimum of L R 2 LR^2 L R 2 and the scale (2) by more than a constant factor.
For the last iterate x T + 1 x_{T+1} x T + 1 itself, existing bounds depend on whether the horizon is given in advance. With a known horizon T T T , Liu [37 ] and Jiang et al. [28 ] proved that clipped SGD with linearly decaying step sizes attains the optimal scale (2) at x T + 1 x_{T+1} x T + 1 ; we restate the explicit bound of Jiang et al. [28 ] as Theorem 1. Without a known horizon, Chezhegov et al. [8 ] obtained the slower rate O ~ ( t − 2 ( p − 1 ) / ( 3 p ) ) \widetilde{O}(t^{-2(p-1)/(3p)}) O ( t − 2 ( p − 1 ) / ( 3 p ) ) using a schedule that takes δ \delta δ as input; because their proof controls nested events across all steps, their guarantee is time-uniform. Guided by numerical experiments, Chezhegov et al. [8 ] conjectured that a better schedule reaches the optimal rate, while Liu [37 ] left the unknown-horizon case open. Since linear decay closes the last-iterate gap for a known horizon and time uniformity costs only logarithmic factors under sub-Gaussian noise, one would expect the gap between the exponents 2 ( p − 1 ) / ( 3 p ) 2(p-1)/(3p) 2 ( p − 1 ) / ( 3 p ) and α p \alpha_p α p to be an artifact of the analysis.
Main Results
We show that under heavy-tailed noise, part of this gap is inherent. Across the two regimes, clipped SGD attains its optimal last-iterate errors
O ~ ( T − α p ) \widetilde{O}(T^{-\alpha_p}) O ( T − α p )
with a known horizon T T T ,
O ~ ( t − β p ) \widetilde{O}(t^{-\beta_p}) O ( t − β p )
at all times t t t simultaneously,
where β p = 2 ( p − 1 ) / ( 3 p − 1 ) \beta_p = 2(p-1)/(3p-1) β p = 2 ( p − 1 ) / ( 3 p − 1 ) lies strictly between the exponents 2 ( p − 1 ) / ( 3 p ) 2(p-1)/(3p) 2 ( p − 1 ) / ( 3 p ) and α p \alpha_p α p , and O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) hides constants and factors polylogarithmic in t / δ t/\delta t / δ . Under finite variance (p = 2 p = 2 p = 2 ), the known-horizon, optimal time-uniform, and previous time-uniform exponents are 1 / 2 1/2 1/2 , 2 / 5 2/5 2/5 , and 1 / 3 1/3 1/3 . The conjecture of Chezhegov et al. [8 ] holds with a known horizon, as the results of Liu [37 ] and Jiang et al. [28 ] show, but fails without one, because β p < α p \beta_p < \alpha_p β p < α p . On the upper-bound side, we give a deterministic schedule with η t ∝ t − ( p + 1 ) / ( 3 p − 1 ) / log 2 ( 100 t ) \eta_t \propto t^{-(p+1)/(3p-1)}/\log^2(100t) η t ∝ t − ( p + 1 ) / ( 3 p − 1 ) / log 2 ( 100 t ) and λ t ∝ t 2 / ( 3 p − 1 ) \lambda_t \propto t^{2/(3p-1)} λ t ∝ t 2/ ( 3 p − 1 ) that uses neither the horizon nor the confidence level, and we prove that clipped SGD with this schedule attains error O ~ ( t − β p ) \widetilde{O}(t^{-\beta_p}) O ( t − β p ) at all times simultaneously (Theorem 2). Because a single trajectory serves every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , the rate O ~ ( t − β p ) \widetilde{O}(t^{-\beta_p}) O ( t − β p ) also holds almost surely. At p = 2 p = 2 p = 2 , this upper bound separates clipped SGD from unclipped SGD with deterministic step sizes: in concurrent work, Anonymous [1 ] prove that deterministic unclipped schedules attain no almost sure rate O ~ ( t − r ) \widetilde{O}(t^{-r}) O ( t − r ) for r > 1 / 3 r > 1/3 r > 1/3 , and we show that their construction also excludes time-uniform exponents above 1 / 3 1/3 1/3 on our problem class (Appendix § 15 , Remark 16).
A lower bound with exponent r r r takes the following form: for every r ′ > r r' > r r ′ > r , any claimed guarantee O ~ ( t − r ′ ) \widetilde{O}(t^{-r'}) O ( t − r ′ ) fails on some instance with probability greater than δ \delta δ , so no exponent above r r r is attainable. We begin with horizon-free gradient-only rules (Definition 3 ), which at each round t t t query one fresh stochastic gradient g t g_t g t at the current iterate x t x_t x t and update x t + 1 = x t − c t g t x_{t+1} = x_t - c_tg_t x t + 1 = x t − c t g t with a scalar c t ≥ 0 c_t \ge0 c t ≥ 0 that may depend on the observed history, on g t g_t g t , and on independent randomness. Every clipped SGD method with adaptive, randomized step sizes and thresholds belongs to this class. Even on one-dimensional instances with i.i.d. noise, we prove that no gradient-only rule attains a time-uniform guarantee O ~ ( t − r ) \widetilde{O}(t^{-r}) O ( t − r ) for any r > β p r > \beta_p r > β p (Theorem 4 ). We then extend this lower bound to every horizon-free rule whose iterate does not advance on rounds that bring no new gradient information, meaning that on one-dimensional instances a round with g t = 0 g_t = 0 g t = 0 does not move the iterate past its earlier maximum in a direction fixed before the run (Proposition 21 ). This broader class includes updates with fixed anchors, contracting weight decay toward the initialization, clipped KT coin betting (Orabona and Pál [49 ] ), and the conic-lifting methods CBA, CBA+ ^+ + , and conic FTRL of Grand-Clément et al. [22 ] in one dimension (Appendix § 17 ).
Why can these rules not reach the exponent α p \alpha_p α p without a known horizon? Our hard instances are translated one-dimensional smooth convex objectives whose stochastic gradients contain rare Bernoulli spikes; we index them by their population minimizers, or targets, y ∈ [ μ , 1 ] y \in[\mu,1] y ∈ [ μ , 1 ] , where μ ∈ ( 0 , 1 / 2 ] \mu\in(0,1/2] μ ∈ ( 0 , 1/2 ] is a parameter. Before the iterate crosses y − μ y-\mu y − μ , the oracle of every target in this family returns zero gradient on quiet rounds and the same negative spike on rare rounds, which occur with probability π ≍ μ p / ( p − 1 ) \pi\asymp\mu^{p/(p-1)} π ≍ μ p / ( p − 1 ) . Because the oracles of all uncrossed targets produce the same observation transcript, any rule whose iterate does not advance on zero gradients can move forward toward distant targets only on the rare spike rounds. Suppose that the rule promises an objective gap below an envelope b t b_t b t at every round t t t . If the iterate first crosses y − μ y-\mu y − μ at round t t t and lands more than μ + 2 b t \mu+\sqrt{2b_t} μ + 2 b t beyond this boundary, it overshoots y y y and violates that target’s envelope. When we integrate the probabilities of failing at the first crossing over the targets y y y , we find that if a rule keeps the gap below b t b_t b t at all rounds t ≤ N t \le N t ≤ N with probability at least 1 − ε 1-\varepsilon 1 − ε on every target, then the envelope values satisfy the progress budget π ∑ t = 1 N ( μ + 2 b t ) ≥ ( 1 − 2 ε ) / 2 \pi\sum_{t=1}^{N}(\mu+\sqrt{2b_t}) \ge(1-2\varepsilon)/2 π ∑ t = 1 N ( μ + 2 b t ) ≥ ( 1 − 2 ε ) /2 for every μ ∈ ( 4 θ N , 1 / 2 ) \mu\in(4\theta_N,1/2) μ ∈ ( 4 θ N , 1/2 ) (Proposition 17 ). For an envelope that decays like t − u t^{-u} t − u with u > β p u > \beta_p u > β p and μ \mu μ of order b N b_N b N , the left-hand side tends to zero as N → ∞ N \to\infty N → ∞ (§ 4.2 ), a contradiction when ε < 1 / 2 \varepsilon< 1/2 ε < 1/2 . A method that queries stochastic gradients at the running average is not bound by this budget: even on rounds with zero gradient, its iterate keeps moving toward a base sequence that accumulates past clipped gradient steps. Using a δ \delta δ -dependent schedule, we prove that a clipped running-average update attains O ~ ( t − α p ) \widetilde{O}(t^{-\alpha_p}) O ( t − α p ) at all times simultaneously (Theorem 9 ). The three-sequence accelerated clipped method of Nguyen et al. [48 ] also attains α p \alpha_p α p without a known horizon; their theorem is stated at each fixed time, and their proof gives the time-uniform form (Appendix § 21 ). Reaching α p \alpha_p α p without a known horizon therefore requires an output that can advance without new gradient information; rescaling individual gradients is not enough.
When a guarantee is required only at one deterministic time t t t rather than at all times simultaneously, the cost of an unknown horizon moves from the time exponent into the confidence factor (§ 5 ). For every r ∈ ( β p , α p ] r \in(\beta_p, \alpha_p] r ∈ ( β p , α p ] , we give horizon-free clipped SGD schedules that attain the fixed-time error O ~ ( δ − c t − r ) \widetilde{O}(\delta^{-c}t^{-r}) O ( δ − c t − r ) with c = ( 3 p − 1 ) ( r − β p ) / ( p − 1 ) c = (3p-1)(r-\beta_p)/(p-1) c = ( 3 p − 1 ) ( r − β p ) / ( p − 1 ) when δ \delta δ is supplied as input, and with a larger exponent c c c without a confidence input; at r = α p r = \alpha_p r = α p , the confidence factors are δ − α p \delta^{-\alpha_p} δ − α p with a confidence input and δ − 1 \delta^{-1} δ − 1 without one (Theorem 5 ). Conversely, by applying our progress budget across finite prefixes via a union bound, we prove that any family of gradient-only rules attaining O ~ ( δ − c t − r ) \widetilde{O}(\delta^{-c}t^{-r}) O ( δ − c t − r ) at each fixed time with δ \delta δ -independent constants must satisfy c ≥ r − β p c \ge r-\beta_p c ≥ r − β p (Theorem 6 ). Thus the optimal fixed-time confidence exponent c ∗ ( r ) c^*(r) c ∗ ( r ) satisfies r − β p ≤ c ∗ ( r ) ≤ ( 3 p − 1 ) ( r − β p ) / ( p − 1 ) r-\beta_p \le c^*(r) \le(3p-1)(r-\beta_p)/(p-1) r − β p ≤ c ∗ ( r ) ≤ ( 3 p − 1 ) ( r − β p ) / ( p − 1 ) ; its exact value remains open (Appendix § 11 ). With polylogarithmic dependence on 1 / δ 1/\delta 1/ δ , gradient-only rules are again limited to the rate O ~ ( t − β p ) \widetilde{O}(t^{-\beta_p}) O ( t − β p ) .
Expressed as heavy-ball momentum ([54 ] ), the running-average update of Theorem 9 has momentum coefficients β t \beta_t β t with 1 − β t 1-\beta_t 1 − β t of order 1 / t 1/t 1/ t , so its effective memory grows over time (§ 6 ). We prove that memory that does not grow cannot improve on the exponent β p \beta_p β p when step sizes are polynomially bounded. Starting from zero velocity, clipped heavy-ball momentum with geometric forgetting (β t ≤ β ˉ < 1 \beta_t \le\bar{\beta} < 1 β t ≤ β ˉ < 1 ) attains no time-uniform bound O ~ ( t − r ) \widetilde{O}(t^{-r}) O ( t − r ) for r > β p r > \beta_p r > β p even in one dimension; under polynomial forgetting 1 − β t ≥ ρ ( 1 + t ) − a 1-\beta_t \ge\rho(1+t)^{-a} 1 − β t ≥ ρ ( 1 + t ) − a with a ∈ [ 0 , 1 ) a \in[0,1) a ∈ [ 0 , 1 ) , we prove the exponent barriers ( 1 + a ) β p (1+a)\beta_p ( 1 + a ) β p in one dimension (Theorem 7 ) and max { β p , a } \max\{\beta_p,a\} max { β p , a } when the control and rate constants are shared across dimensions (Theorem 8 ). For learning-rate-free optimization, we also prove that the zero-initialized conic Blackwell algorithms CBA and CBA+ ^+ + in the form of Grand-Clément et al. [22 ] admit no time-uniform last-iterate bound that tends to zero at all, even under i.i.d. additive noise (Proposition 22 , Appendix § 17 ). We summarize the optimal last-iterate rates at p = 2 p=2 p = 2 in Table 1 .
Table 1. Best last-iterate error of clipped methods under finite variance (p = 2 p = 2 p = 2 ), up to constants and logarithmic factors. Shaded cells mark rates that are new in this paper; the arrow starts from the previous time-uniform rate of Chezhegov et al. [8 ] , and the known-horizon rate is due to Liu [37 ] and Jiang et al. [28 ] . Down the current-iterate column, knowing less about when the run stops costs a polynomial factor in t t t or 1 / δ 1/\delta 1/ δ ; the running-average column shows that querying gradients at the running average removes this cost (Theorem 9 ). For the guarantees in the bottom two rows, no horizon-free gradient-only rule, with any adaptive step sizes and thresholds, exceeds the exponent 2 / 5 2/5 2/5 , and for time-uniform guarantees neither does heavy-ball momentum with geometric forgetting and polynomially bounded step sizes (Theorems 7 and 8 ). For 1 < p ≤ 2 1 < p \le2 1 < p ≤ 2 , replace 1 / 2 1/2 1/2 by α p \alpha_p α p , 2 / 5 2/5 2/5 by β p \beta_p β p , 1 / 3 1/3 1/3 by 2 ( p − 1 ) / ( 3 p ) 2(p - 1)/(3p) 2 ( p − 1 ) / ( 3 p ) , and δ − 1 / 2 \delta^{-1/2} δ − 1/2 by δ − α p \delta^{-\alpha_p} δ − α p .
Techniques and Organization
Our upper bounds for clipped SGD rest on a single potential (Lemma 11 ) that combines a recursive average of past objective gaps, weighted using inverse tail sums of remaining step sizes following the comparators of Zamani and Glineur (2025) and Liu and Zhou [39 ] , with the distance from the current iterate to a matching average of the iterates. When the horizon T T T is known, these weights place full mass on the final iterate at step T T T , which yields Theorem 1 directly (Jiang et al. [28 ] ). Without a known horizon, we add a positive constant to the step-size tail sums so that the weights remain bounded for all t t t . Smoothness then converts the potential into a last-iterate error bound, and the concentration events for all deterministic horizons can be intersected into one event. We obtain each of these events from a single stopped Freedman inequality (Lemma 15 ) rather than from the step-by-step induction and union bound that Nguyen et al. [48 ] identify as the source of extra log T \log T log T factors in earlier analyses. To achieve fixed-time exponents r > β p r > \beta_p r > β p , we instead truncate late clipping errors at levels set by the target time and absorb the truncation probability into the polynomial δ \delta δ -factor. Both this bounded-weight potential and our pre-crossing progress budget for translated objectives (Proposition 17 ) may be useful in other time-uniform last-iterate analyses.
Appendix § 9 discusses related work, Appendix § 10 gives numerical illustrations at p = 2 p = 2 p = 2 , Appendix § 11 lists open problems, and the remaining appendices contain the proofs. We also provide machine-checked proofs in Lean 4 with Mathlib of Theorems 2, 4 , and 8 , together with the known-horizon bound of Theorem 1 (§ 7 ).
Problem Setting
Throughout, ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ and ⟨ ⋅ , ⋅ ⟩ \langle\cdot,\cdot\rangle ⟨ ⋅ , ⋅ ⟩ denote the Euclidean norm and inner product, and all logarithms are natural.
Assumption 1 (Objective). The function f : R d → R f : \mathbb{R}^{d} \to\mathbb{R} f : R d → R is differentiable and convex, and
∥ ∇ f ( x ) − ∇ f ( y ) ∥ ≤ L ∥ x − y ∥ for all x , y ∈ R d . \lVert\nabla f(x) - \nabla f(y)\rVert\le L\lVert x-y\rVert\quad\text{for all } x,y \in\mathbb{R}^{d}. ∥ ∇ f ( x ) − ∇ f ( y )∥ ≤ L ∥ x − y ∥ for all x , y ∈ R d . There is a minimizer x ⋆ x^\star x ⋆ , with f ⋆ = f ( x ⋆ ) f^\star= f(x^\star) f ⋆ = f ( x ⋆ ) , and a known bound ∥ x 1 − x ⋆ ∥ ≤ R \lVert x_1 - x^\star\rVert\le R ∥ x 1 − x ⋆ ∥ ≤ R , where L , R > 0 L, R > 0 L , R > 0 .
Assumption 2 (Oracle). Let F t − 1 \mathcal{F}_{t-1} F t − 1 contain the initial state, any independent algorithmic randomness, and all observations up to time t − 1 t-1 t − 1 , and write E t − 1 [ ⋅ ] = E [ ⋅ ∣ F t − 1 ] \mathbb{E}_{t-1}[\cdot] = \mathbb{E}[\cdot\mid\mathcal{F}_{t-1}] E t − 1 [ ⋅ ] = E [ ⋅ ∣ F t − 1 ] . The query x t x_t x t is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, and the oracle returns one fresh observation g t g_t g t with, for some σ ≥ 0 \sigma\ge0 σ ≥ 0 ,
E t − 1 [ g t ] = ∇ f ( x t ) , \mathbb{E}_{t-1}[g_t] = \nabla f(x_t), E t − 1 [ g t ] = ∇ f ( x t ) , E t − 1 [ ∥ g t − ∇ f ( x t ) ∥ p ] ≤ σ p , 1 < p ≤ 2. \mathbb{E}_{t-1}\left[\lVert g_t - \nabla f(x_t)\rVert^p\right] \le\sigma^p, \qquad1 < p \le2. E t − 1 [ ∥ g t − ∇ f ( x t ) ∥ p ] ≤ σ p , 1 < p ≤ 2. Assumption 2 covers both i.i.d. samples from a fixed oracle with uniform moment bounds and conditionally unbiased martingale difference noise. Bounding only the central p p p th moment permits
asymmetric noise and infinite variance when p < 2 p < 2 p < 2 , without requiring a uniform bound on E t − 1 ∥ g t ∥ p \mathbb{E}_{t-1}\lVert g_t\rVert^p E t − 1 ∥ g t ∥ p ; individual sample losses need not be smooth or convex. Following prior analyses ([58 , 48 , 8 ] ), our schedules may use the parameters L L L , R R R , σ \sigma σ , p p p and, when noted, a confidence parameter δ \delta δ , whereas the time-uniform schedule of § 4.1 requires no δ \delta δ .
At each step, clipped SGD applies Euclidean clipping,
clip ( g , λ ) = { min { 1 , λ / ∥ g ∥ } g , g ≠ 0 , 0 , g = 0 , \operatorname{clip}(g,\lambda) =
\begin{cases}
\min\{1,\lambda/\lVert g\rVert\}\,g, & g \ne0,\\
0, & g = 0,
\end{cases} clip ( g , λ ) = { min { 1 , λ / ∥ g ∥} g , 0 , g = 0 , g = 0 ,
which projects the stochastic gradient onto the ball of radius λ \lambda λ centered at zero; the iterates themselves are never projected. After t t t oracle calls, the algorithm outputs the last iterate x t + 1 x_{t+1} x t + 1 directly, without averaging, checkpoint selection, or restarts.
Fixed-time and time-uniform guarantees. What does high-probability convergence mean when the stopping time is not fixed before the run? Let ε t ( δ ) \varepsilon_t(\delta) ε t ( δ ) be a deterministic error envelope. A fixed-time guarantee requires that for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and every deterministic t ≥ 1 t \ge1 t ≥ 1 ,
P { f ( x t + 1 ) − f ⋆ ≤ ε t ( δ ) } ≥ 1 − δ , \mathbb{P}\{f(x_{t+1}) - f^\star\le\varepsilon_t(\delta)\} \ge1-\delta, P { f ( x t + 1 ) − f ⋆ ≤ ε t ( δ )} ≥ 1 − δ ,
where the high-probability event may vary with t t t . Such a bound covers only times chosen before the run begins, while a known-horizon method requires the bound only at its terminal step t = T t=T t = T . By contrast, a time-uniform guarantee, analogous to a confidence sequence (Howard et al. [25 ] ), requires the stronger simultaneous bound
P { f ( x t + 1 ) − f ⋆ ≤ ε t ( δ ) for all t ≥ 1 } ≥ 1 − δ . \mathbb{P}\{f(x_{t+1}) - f^\star\le\varepsilon_t(\delta)\ \text{for all } t \ge1\} \ge1-\delta. P { f ( x t + 1 ) − f ⋆ ≤ ε t ( δ ) for all t ≥ 1 } ≥ 1 − δ .
Time uniformity is equivalent to validity at every data-dependent stopping time. On the one hand, the simultaneous event implies f ( x τ + 1 ) − f ⋆ ≤ ε τ ( δ ) f(x_{\tau+1}) - f^\star\le\varepsilon_\tau(\delta) f ( x τ + 1 ) − f ⋆ ≤ ε τ ( δ ) at any random time τ \tau τ , such as one chosen by early stopping or checkpoint selection. On the other hand, any guarantee that holds at every bounded stopping time is time-uniform: apply it to min { τ , N } \min\{\tau,N\} min { τ , N } , where τ \tau τ is the first time the envelope fails, and let N → ∞ N \to\infty N → ∞ . A union-bound conversion from fixed-time to time-uniform guarantees splits the failure budget δ \delta δ across steps, and under heavy-tailed noise that cost cannot always be absorbed into logarithmic factors (§ 5 ).
Known Horizon: An Optimal Last-Iterate Rate
We first review the known-horizon baseline. When the total number of steps T T T is fixed in advance, clipped SGD with linearly decaying step sizes attains the statistical scale (2) at its last iterate ([37 , 28 ] ); Algorithm 1 is the version of Jiang et al. [28 ] . We then turn to unknown horizons in § 4 .
Algorithm 1 makes T T T oracle calls using O ( d ) O(d) O ( d ) memory and no knowledge of x ⋆ x^\star x ⋆ . It keeps the clipping threshold fixed for the entire run and decays the step size linearly from η \eta η to η / T \eta/T η / T ; linear decay is the worst-case optimal schedule in the analysis of Defazio et al. [12 ] . Jiang et al. [28 ] prove that this schedule attains the optimal rate O ( T − α p ) O(T^{-\alpha_p}) O ( T − α p ) at its last iterate, with explicit constants; we restate their result below.
Theorem 1 (Known horizon; Jiang et al. [28 ] ). Let Assumptions 1 and 2 hold, and let T ≥ 1 T \ge1 T ≥ 1 and 0 < δ < 1 0 < \delta< 1 0 < δ < 1 . With probability at least 1 − δ 1-\delta 1 − δ , the output of Algorithm 1 satisfies, with ℓ = log ( 8 / δ ) \ell= \log(8/\delta) ℓ = log ( 8/ δ ) ,
f ( x T + 1 ) − f ⋆ ≤ 1024 L R 2 ℓ T + 256 σ R ( ℓ T ) ( p − 1 ) / p . (3) f(x_{T+1}) - f^\star\le\frac{1024LR^2\ell}{T} + 256\sigma R\left(\frac{\ell}{T}\right)^{(p-1)/p}. \tag*{(3)} f ( x T + 1 ) − f ⋆ ≤ T 1024 L R 2 ℓ + 256 σ R ( T ℓ ) ( p − 1 ) / p . (3)
Algorithm 1 Clipped SGD for a known horizon
1: Input: $x_1$, $L$, $R$, $\sigma$, $p$, $T$, $\delta$
2: $\ell \leftarrow \log(8/\delta)$
3: $\lambda \leftarrow \max\{4LR,\sigma(T/\ell)^{1/p}\}$
4: $\eta \leftarrow R/(128\lambda\ell)$
5: for $t=1,\ldots,T$ do
6: Draw one fresh stochastic gradient $g_t$ at $x_t$
7: $\eta_t \leftarrow \eta(T-t+1)/T$
8: $x_{t+1} \leftarrow x_t-\eta_t\operatorname{clip}(g_t,\lambda)$
9: end for
10: Return $x_{T+1}$
Liu [37 ] obtain a bound of the same order using a state-dependent clipping threshold (Appendix § 9 .§ 2 ).
Unknown Horizon: Time-Uniform Guarantees
When the horizon is unknown, Algorithm 1 cannot tune its threshold or linear decay to T T T . We now ask what last-iterate rate clipped SGD can achieve at all times simultaneously. In § 4.1 , we construct a deterministic schedule that attains O ~ ( t − β p ) \widetilde{O}(t^{-\beta_p}) O ( t − β p ) uniformly over time without using a horizon or a confidence input. In § 4.2 , we prove that no gradient-only rule can attain a larger exponent.
A Horizon-Free Upper Bound
Define the exponents
β = β p = 2 ( p − 1 ) 3 p − 1 , a = p + 1 3 p − 1 = 1 − β , b = 2 3 p − 1 . (4) \begin{aligned}
\beta=\beta_p&=\frac{2(p-1)}{3p-1},\\
a&=\frac{p+1}{3p-1}=1-\beta,\qquad b=\frac{2}{3p-1}. \tag*{(4)}
\end{aligned} β = β p a = 3 p − 1 2 ( p − 1 ) , = 3 p − 1 p + 1 = 1 − β , b = 3 p − 1 2 . (4)
Let Q = max { L R , σ } Q=\max\{LR,\sigma\} Q = max { L R , σ } and ℓ t = log ( 100 t 2 ) \ell_t=\log(100t^2) ℓ t = log ( 100 t 2 ) . We run the recursion (1) for all t ≥ 1 t\ge1 t ≥ 1 with
η t = R Q c p t − a ℓ t 2 , λ t = 4 Q t b , (5) \eta_t=\frac{R}{Q}\frac{c_p t^{-a}}{\ell_t^2},\qquad\lambda_t=4Qt^b, \tag*{(5)} η t = Q R ℓ t 2 c p t − a , λ t = 4 Q t b , (5)
where c p > 0 c_p>0 c p > 0 depends only on p p p . At p = 2 p=2 p = 2 , the step size decays as η t ∝ t − 3 / 5 / ℓ t 2 \eta_t\propto t^{-3/5}/\ell_t^2 η t ∝ t − 3/5 / ℓ t 2 and the threshold grows as λ t ∝ t 2 / 5 \lambda_t\propto t^{2/5} λ t ∝ t 2/5 . For the analysis, write ι t ( δ ) = log ( 100 t 2 / δ ) \iota_t(\delta)=\log(100t^2/\delta) ι t ( δ ) = log ( 100 t 2 / δ ) .
Theorem 2 (Time-uniform upper bound). Let Assumptions 1 and 2 hold, and run (1) with the schedule (5). There is a finite positive constant c p c_p c p depending only on p p p , and we may take D p = 4 / c p D_p=4/c_p D p = 4/ c p (see Appendix § 18 .§ 1 ), such that for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , with probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every integer T ≥ 1 T\ge1 T ≥ 1 ,
f ( x T + 1 ) − f ⋆ ≤ D p R Q T − β ι T ( δ ) 4 , (6) f(x_{T+1})-f^\star\le D_pRQ T^{-\beta}\iota_T(\delta)^4, \tag*{(6)} f ( x T + 1 ) − f ⋆ ≤ D p R Q T − β ι T ( δ ) 4 , (6) ∥ x T + 1 − x ⋆ ∥ 2 ≤ R 2 [ log ( 100 / δ ) ] 2 . (7) \lVert x_{T+1}-x^\star\rVert^2\le R^2[\log(100/\delta)]^2. \tag*{(7)} ∥ x T + 1 − x ⋆ ∥ 2 ≤ R 2 [ log ( 100/ δ ) ] 2 . (7)
Because the schedule (5) does not depend on δ \delta δ , the algorithm produces a single trajectory that serves every confidence level simultaneously; only the high-probability event and its bounds vary with δ \delta δ . Hence the rate also holds almost surely (Appendix § 18 .§ 6 ).
Our proof produces conservative numerical constants, giving c 2 < 10 − 25 c_2 < 10^{-25} c 2 < 1 0 − 25 and D 2 > 10 25 D_2 > 10^{25} D 2 > 1 0 25 at p = 2 p = 2 p = 2 (Appendix § 18 .§ 1 ). The exponents of the schedule and of the rate do not depend on these constants, however, and the numerical illustration in Appendix § 10 uses step-size constants of order one. Chezhegov et al. [8 ] obtain the exponent 2 ( p − 1 ) / ( 3 p ) 2(p - 1)/(3p) 2 ( p − 1 ) / ( 3 p ) (equal to 1 / 3 1/3 1/3 at p = 2 p = 2 p = 2 ) with a δ \delta δ -dependent schedule; their guarantee is also time-uniform, because their proof controls nested events over all steps. Our improvement to β p \beta_p β p comes from the modified potential in Appendix § 12 .§ 2 , whose weights stay bounded without a horizon, so that, for each analysis horizon, a single stopped Freedman inequality controls the potential. We next show that β p \beta_p β p is optimal.
A Matching Lower Bound for Horizon-Free Rules
To show that β p \beta_p β p cannot be improved, we formalize the class of algorithms that query a stochastic gradient at their current iterate and move along that gradient. We refer to algorithms of this form as gradient-only rules ; Definition 3 fixes their update direction and oracle access, excluding momentum, extra queries, restarts, output selection, and averaging.
Definition 3 (Gradient-only rules). Let p ∈ ( 1 , 2 ] p \in(1, 2] p ∈ ( 1 , 2 ] , L = R = σ = 1 L = R = \sigma= 1 L = R = σ = 1 , and x 1 = 0 x_1 = 0 x 1 = 0 . A horizon-free gradient-only rule queries one fresh stochastic gradient g t g_t g t at the current iterate x t x_t x t and updates
x t + 1 = x t − c t g t , x_{t+1} = x_t - c_t g_t, x t + 1 = x t − c t g t , where the finite step factor c t ≥ 0 c_t \ge0 c t ≥ 0 is a measurable function of t t t , δ \delta δ , p p p , the bounds L , R , σ L, R, \sigma L , R , σ , the gradients g 1 , … , g t g_1, \ldots, g_t g 1 , … , g t , and independent randomness. Equivalently, c t g t = η t clip ( g t , λ t ) c_tg_t = \eta_t \operatorname{clip}(g_t, \lambda_t) c t g t = η t clip ( g t , λ t ) for some finite η t ≥ 0 \eta_t \ge0 η t ≥ 0 and λ t > 0 \lambda_t > 0 λ t > 0 chosen in the same way.
Theorem 4 (Time-uniform lower bound). Fix p ∈ ( 1 , 2 ] p \in(1,2] p ∈ ( 1 , 2 ] and δ ∈ ( 0 , 1 / 2 ) \delta\in(0,1/2) δ ∈ ( 0 , 1/2 ) . For every rule of Definition 3 , every C > 0 C > 0 C > 0 , m ≥ 0 m \ge0 m ≥ 0 , and
r > β p = 2 ( p − 1 ) 3 p − 1 , (8) r > \beta_p = \frac{2(p - 1)}{3p - 1}, \tag*{(8)} r > β p = 3 p − 1 2 ( p − 1 ) , (8) there is an instance satisfying Assumptions 1 and 2 with L = R = σ = 1 L = R = \sigma= 1 L = R = σ = 1 and x 1 = 0 x_1 = 0 x 1 = 0 such that
P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > C t − r [ log ( e t ) ] m } > δ . (9) \mathbb{P}\left\{\exists t \ge1 : f(x_{t+1}) - f^\star> Ct^{-r}[\log(et)]^m\right\} > \delta. \tag*{(9)} P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > C t − r [ log ( e t ) ] m } > δ . (9) The rule, C C C , and m m m may depend arbitrarily on δ \delta δ and p p p . The instance is one-dimensional, with i.i.d. additive noise and convex, globally 1-smooth sample losses that attain their minima.
Together, Theorems 2 and 4 establish that β p \beta_p β p is the optimal time-uniform exponent for gradient-only rules, strictly below the known-horizon exponent α p \alpha_p α p . Proposition 21 extends Theorems 4 and 6 to every horizon-free rule whose iterate never moves past its earlier maximum in one fixed direction on zero-gradient rounds, such as updates with fixed anchors or contracting weight decay toward the initialization, clipped KT coin betting (Orabona and Pál [49 ] ), and the conic-lifting methods of Grand-Clément et al. [22 ] in one dimension (Appendix § 17 ). Querying gradients at the running average escapes this barrier (§ 6 ).
Idea of the proof. We make the argument sketched in § 1.1 quantitative. The hard instances use the sample losses F μ , y ( x ; Y ) = 1 2 ( x − y + μ ) 2 − A Y x F_{\mu,y}(x;Y) = \frac{1}{2}(x - y + \mu)^2 - AYx F μ , y ( x ; Y ) = 2 1 ( x − y + μ ) 2 − A Y x , where Y ∼ Bernoulli ( μ / A ) Y \sim\operatorname{Bernoulli}(\mu/A) Y ∼ Bernoulli ( μ / A ) and A = ( 2 μ ) − 1 / ( p − 1 ) A = (2\mu)^{-1/(p-1)} A = ( 2 μ ) − 1/ ( p − 1 ) , so that the expected objective is a translated one-sided Huber function minimized at y ∈ [ μ , 1 ] y \in[\mu,1] y ∈ [ μ , 1 ] and the spikes Y = 1 Y = 1 Y = 1 arrive with probability π = μ / A \pi= \mu/A π = μ / A . Proposition 17 turns the first-crossing argument into a progress budget: if a deterministic envelope b 1 , … , b N b_1,\ldots,b_N b 1 , … , b N holds on every instance with probability at least 1 − ε 1-\varepsilon 1 − ε , then for every μ ∈ ( 4 b N , 1 / 2 ] \mu\in(4b_N,1/2] μ ∈ ( 4 b N , 1/2 ] ,
π ∑ t = 1 N ( μ + 2 b t ) ≥ 1 − 2 ε 2 . \pi\sum_{t=1}^{N}(\mu+\sqrt{2b_t}) \ge\frac{1-2\varepsilon}{2}. π t = 1 ∑ N ( μ + 2 b t ) ≥ 2 1 − 2 ε .
The budget holds for arbitrary adaptive jump sizes, and the targets y y y are fixed before the run rather than chosen after observing the path. Given a claimed exponent r > β p r>\beta_p r > β p , we pick β p < u < min { r , 1 } \beta_p<u<\min\{r,1\} β p < u < min { r , 1 } and dominate the first N N N values of the claimed envelope by b N , t = C log m ( e N ) t − u b_{N,t}=C\log^m(eN)t^{-u} b N , t = C log m ( e N ) t − u . With μ = 8 b N , N \mu=8b_{N,N} μ = 8 b N , N and π ≍ μ p / ( p − 1 ) \pi\asymp\mu^{p/(p-1)} π ≍ μ p / ( p − 1 ) , the left-hand side scales, up to logarithmic factors, as N 1 − u ( p / ( p − 1 ) + 1 / 2 ) N^{1-u(p/(p-1)+1/2)} N 1 − u ( p / ( p − 1 ) + 1/2 ) . Because u > β p = 1 / ( p / ( p − 1 ) + 1 / 2 ) u>\beta_p=1/(p/(p-1)+1/2) u > β p = 1/ ( p / ( p − 1 ) + 1/2 ) , this sum tends to zero as N → ∞ N\to\infty N → ∞ , contradicting the lower bound ( 1 − 2 ε ) / 2 (1-2\varepsilon)/2 ( 1 − 2 ε ) /2 when ε < 1 / 2 \varepsilon<1/2 ε < 1/2 . Appendix H carries out this calculation, which also yields Theorems 6 and 7 .
Unknown Horizon: Fixed-Time Guarantees
§ 4 showed that no gradient-only rule can exceed the exponent β p \beta_p β p uniformly over all times. We now ask what changes if the guarantee is required only at a single deterministic time T T T , while the horizon remains unknown to the algorithm. We show that every exponent in ( β p , α p ] (\beta_p,\alpha_p] ( β p , α p ] becomes attainable, but the cost of not knowing the horizon shifts into a polynomial dependence on 1 / δ 1/\delta 1/ δ .
Theorem 5 (Fixed-time confidence–rate tradeoff). Let Assumptions 1 and 2 hold, and let Q Q Q , c p c_p c p , D p D_p D p , ℓ t \ell_t ℓ t , and ι T ( δ ) \iota_T(\delta) ι T ( δ ) be as in Theorem 2. Fix r ∈ ( β p , α p ] r\in(\beta_p,\alpha_p] r ∈ ( β p , α p ] and u ≥ 0 u\ge0 u ≥ 0 , and use the horizon-free schedule
η t = R Q c p δ u t − ( 1 − r ) ℓ t 2 , λ t = 4 Q max { 1 , δ u / ( p − 1 ) t r / ( p − 1 ) } . (10) \eta_t=\frac{R}{Q}\frac{c_p\delta^u t^{-(1-r)}}{\ell_t^2},\qquad
\lambda_t=4Q\max\{1,\delta^{u/(p-1)}t^{r/(p-1)}\}. \tag*{(10)} η t = Q R ℓ t 2 c p δ u t − ( 1 − r ) , λ t = 4 Q max { 1 , δ u / ( p − 1 ) t r / ( p − 1 ) } . (10) Put
c i n ( r ) = 3 p − 1 p − 1 ( r − β p ) , c_{\mathrm{in}}(r)=\frac{3p-1}{p-1}(r-\beta_p), c in ( r ) = p − 1 3 p − 1 ( r − β p ) , c f r e e ( r ) = ( 3 p − 1 ) r − 2 ( p − 1 ) ( 2 p − 1 ) r − ( p − 1 ) . c_{\mathrm{free}}(r)=\frac{(3p-1)r-2(p-1)}{(2p-1)r-(p-1)}. c free ( r ) = ( 2 p − 1 ) r − ( p − 1 ) ( 3 p − 1 ) r − 2 ( p − 1 ) . If u = c i n ( r ) u=c_{\mathrm{in}}(r) u = c in ( r ) , let c = c i n ( r ) c=c_{\mathrm{in}}(r) c = c in ( r ) ; if u = 0 u=0 u = 0 , let c = c f r e e ( r ) c=c_{\mathrm{free}}(r) c = c free ( r ) . Then for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) and every deterministic T ≥ 1 T\ge1 T ≥ 1 , with probability at least 1 − δ 1-\delta 1 − δ ,
f ( x T + 1 ) − f ⋆ ≤ D p R Q T − r δ − c ι T ( δ ) 4 . (11) f(x_{T+1})-f^\star\le D_pRQ\,T^{-r}\delta^{-c}\iota_T(\delta)^4. \tag*{(11)} f ( x T + 1 ) − f ⋆ ≤ D p R Q T − r δ − c ι T ( δ ) 4 . (11)
Clipped SGD with the schedule (10) is horizon-free and steps along the clipped current stochastic gradient, so it is a gradient-only rule in the sense of Definition 3 . When u = 0 u=0 u = 0 , the schedule uses neither the horizon nor the confidence parameter, so the same trajectory serves every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) : for each δ \delta δ , (11) holds with probability at least 1 − δ 1-\delta 1 − δ , although the success event depends on both δ \delta δ and T T T . At the statistical exponent r = α p r=\alpha_p r = α p , the confidence factor is δ − ( p − 1 ) / p \delta^{-(p-1)/p} δ − ( p − 1 ) / p when the schedule takes δ \delta δ as an input and δ − 1 \delta^{-1} δ − 1 when it does not. Both confidence exponents c i n ( r ) c_{\mathrm{in}}(r) c in ( r ) and c f r e e ( r ) c_{\mathrm{free}}(r) c free ( r ) approach zero as r ↓ β p r\downarrow\beta_p r ↓ β p , matching the polylogarithmic confidence dependence of Theorem 2.
Can any horizon-free gradient-only rule attain a fixed-time exponent above β p \beta_p β p with only polylogarithmic dependence on 1 / δ 1/\delta 1/ δ ? We prove that it cannot: the progress budget of Proposition 17 constrains fixed-time guarantees as well, via a union bound over finite prefixes.
Theorem 6 (Fixed-time lower bound with polynomial confidence cost). Let ( r u l e δ ) δ ∈ ( 0 , 1 ) (\mathrm{rule}_{\delta})_{\delta\in(0,1)} ( rule δ ) δ ∈ ( 0 , 1 ) be any family of rules of Definition 3 , and let C > 0 C>0 C > 0 , m ≥ 0 m\ge0 m ≥ 0 , c ∈ [ 0 , 1 − β p ) c\in[0,1-\beta_p) c ∈ [ 0 , 1 − β p ) , and r > β p + c r>\beta_p+c r > β p + c . There exist δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , a deterministic integer t ≥ 1 t\ge1 t ≥ 1 , and an instance in the class of Theorem 4 such that r u l e δ \mathrm{rule}_{\delta} rule δ satisfies
P { f ( x t + 1 ) − f ⋆ > C t − r δ − c [ log ( e t / δ ) ] m } > δ . (12) \mathbb{P}\left\{f(x_{t+1})-f^\star> Ct^{-r}\delta^{-c}[\log(et/\delta)]^m\right\}>\delta. \tag*{(12)} P { f ( x t + 1 ) − f ⋆ > C t − r δ − c [ log ( e t / δ ) ] m } > δ . (12)
Here C C C , c c c , and m m m do not depend on δ \delta δ , t t t , or the instance, while each rule may depend arbitrarily on δ \delta δ , the history, independent randomness, and the current stochastic gradient.
Setting c = 0 c=0 c = 0 in Theorem 6 shows that gradient-only rules cannot attain any fixed-time exponent above β p \beta_p β p with uniformly polylogarithmic dependence on 1 / δ 1/\delta 1/ δ . Appendix § 16 .§ 4 gives the proof.
For each r ∈ ( β p , α p ] r\in(\beta_p,\alpha_p] r ∈ ( β p , α p ] , let c ⋆ ( r ) c^\star(r) c ⋆ ( r ) be the infimum of exponents c c c for which some family of gradient-only rules achieves a fixed-time bound C t − r δ − c [ log ( e t / δ ) ] m Ct^{-r}\delta^{-c}[\log(et/\delta)]^m C t − r δ − c [ log ( e t / δ ) ] m across all instances. Because ι T ( δ ) ≤ 5 log ( e T / δ ) \iota_T(\delta)\le5\log(eT/\delta) ι T ( δ ) ≤ 5 log ( e T / δ ) , combining Theorems 5 and 6 sandwiches c ⋆ ( r ) c^\star(r) c ⋆ ( r ) between two linear functions of r − β p r-\beta_p r − β p :
r − β p ≤ c ⋆ ( r ) ≤ 3 p − 1 p − 1 ( r − β p ) . (13) r-\beta_p\le c^\star(r)\le\frac{3p-1}{p-1}(r-\beta_p). \tag*{(13)} r − β p ≤ c ⋆ ( r ) ≤ p − 1 3 p − 1 ( r − β p ) . (13)
The exact value of c ⋆ ( r ) c^\star(r) c ⋆ ( r ) remains open (Appendix § 11 ).
Memory and the Momentum Boundary
Gradient-only rules cannot surpass the time-uniform exponent β p \beta_p β p without a known horizon (Theorem 4 ). Can momentum break this barrier? For clipped heavy-ball momentum with polynomially bounded step sizes, we prove that momentum helps only if its memory grows over time. To establish this boundary, we compare the running-average update of § 6.3 with two classes of clipped heavy-ball rules. Each method we study queries one stochastic gradient at its current iterate and outputs the updated state.
Forgetting in One Dimension
We first consider the stochastic heavy-ball recursion ([54 ] ) in velocity form:
v t + 1 = β t v t + clip ( g t , λ t ) , x t + 1 = x t − η t v t + 1 , x 1 = v 1 = 0 , (14) \begin{aligned}
v_{t+1} &= \beta_t v_t+\operatorname{clip}(g_t,\lambda_t),\\
x_{t+1} &= x_t-\eta_t v_{t+1},\qquad x_1=v_1=0, \tag*{(14)}
\end{aligned} v t + 1 x t + 1 = β t v t + clip ( g t , λ t ) , = x t − η t v t + 1 , x 1 = v 1 = 0 , (14)
where η t \eta_t η t is the step size and β t ∈ [ 0 , 1 ) \beta_t\in[0,1) β t ∈ [ 0 , 1 ) is the momentum coefficient. We allow a horizon-free rule to choose its scalar controls β t \beta_t β t , η t \eta_t η t , and λ t \lambda_t λ t measurably from the entire past history, an independent random seed, and the current raw gradient g t g_t g t , provided that on every trajectory they satisfy
0 ≤ β t < 1 , 1 − β t ≥ ρ ( 1 + t ) − a ( t ≥ t 0 ) , 0 ≤ η t ≤ H ( 1 + t ) h , 0 ≤ λ t < ∞ , (15) \begin{aligned}
0&\le\beta_t<1,\qquad1-\beta_t\ge\rho(1+t)^{-a}\quad(t\ge t_0),\\
0&\le\eta_t\le H(1+t)^h,\qquad0\le\lambda_t<\infty, \tag*{(15)}
\end{aligned} 0 0 ≤ β t < 1 , 1 − β t ≥ ρ ( 1 + t ) − a ( t ≥ t 0 ) , ≤ η t ≤ H ( 1 + t ) h , 0 ≤ λ t < ∞ , (15)
for deterministic constants a ∈ [ 0 , 1 ) a\in[0,1) a ∈ [ 0 , 1 ) , ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] , t 0 ∈ N t_0\in\mathbb{N} t 0 ∈ N , and H , h ≥ 0 H,h\ge0 H , h ≥ 0 that depend only on the rule and its confidence input δ \delta δ , not on the instance, the samples, or the random seed. Geometric forgetting, β t ≤ β ˉ < 1 \beta_t\le\bar{\beta}<1 β t ≤ β ˉ < 1 , corresponds to the case a = 0 a=0 a = 0 with ρ = 1 − β ˉ \rho=1-\bar{\beta} ρ = 1 − β ˉ and t 0 = 1 t_0=1 t 0 = 1 .
Theorem 7 (One-dimensional momentum lower bound). Fix p ∈ ( 1 , 2 ] p\in(1,2] p ∈ ( 1 , 2 ] , a ∈ [ 0 , 1 ) a\in[0,1) a ∈ [ 0 , 1 ) , and a confidence 0 < δ < 1 / 2 0<\delta<1/2 0 < δ < 1/2 , and let a horizon-free rule satisfy (14) and (15) . For every finite C > 0 C>0 C > 0 , m ≥ 0 m\ge0 m ≥ 0 , and r > ( 1 + a ) β p r>(1+a)\beta_p r > ( 1 + a ) β p , there is a fixed one-dimensional instance with L = R = σ = 1 L=R=\sigma=1 L = R = σ = 1 such that
P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > C t − r log m ( e t ) } > δ . (16) \mathbb{P}\left\{\exists t\ge1:f(x_{t+1})-f^\star>Ct^{-r}\log^m(et)\right\}>\delta. \tag*{(16)} P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > C t − r log m ( e t ) } > δ . (16)
The instance satisfies Assumptions 1 and 2 with i.i.d. additive noise and convex, globally 1-smooth sample losses that attain their minima, and it is fixed before the run.
Under geometric forgetting (a = 0 a = 0 a = 0 ), no rule satisfying (14) and (15) attains a time-uniform exponent above β p \beta_p β p , matching the barrier for gradient-only rules in Theorem 4 . In particular, Theorem 7 covers any fixed coefficient β ∈ [ 0 , 1 ) \beta\in[0,1) β ∈ [ 0 , 1 ) paired with arbitrary adaptive step sizes bounded by a constant. Even under slower polynomial forgetting with 0 ≤ a < ( p − 1 ) / ( 2 p ) 0 \le a < (p-1)/(2p) 0 ≤ a < ( p − 1 ) / ( 2 p ) , we have ( 1 + a ) β p < α p (1+a)\beta_p < \alpha_p ( 1 + a ) β p < α p , so the statistical exponent α p \alpha_p α p remains unattainable. To prove Theorem 7 , we bound the progress that rare gradient spikes can produce after passing through the decaying momentum kernel (Appendix § 19 ).
Polynomial Forgetting with Constants Independent of Dimension
Why does the one-dimensional lower bound in Theorem 7 stop at ( 1 + a ) β p (1+a)\beta_p ( 1 + a ) β p rather than β p \beta_p β p ? Under polynomial forgetting, the momentum kernel needs O ~ ( N a ) \widetilde{O}(N^a) O ( N a ) steps to decay on a horizon N N N , so in the one-dimensional budget argument each rare gradient spike can move the iterate for that many steps. If the dimension may grow and the control and rate constants are shared across dimensions, this extra factor disappears and the exponent barrier drops to max { β p , a } \max\{\beta_p,a\} max { β p , a } .
To state this bound, fix p ∈ ( 1 , 2 ] p \in(1,2] p ∈ ( 1 , 2 ] and a ∈ [ 0 , 1 ) a \in[0,1) a ∈ [ 0 , 1 ) . The class A u n i f ( a ) \mathcal{A}_{\mathrm{unif}}(a) A unif ( a ) consists of all rules defined in every finite dimension that run (14) from x 1 = 0 x_1 = 0 x 1 = 0 and v 1 = 0 v_1 = 0 v 1 = 0 , observe only stochastic gradients, and choose their controls as measurable functions of the dimension, time, the confidence level δ \delta δ , the bounds L = R = σ = 1 L = R = \sigma= 1 L = R = σ = 1 , the gradients g 1 , … , g t g_1,\ldots,g_t g 1 , … , g t , and an independent seed. For each rule in A u n i f ( a ) \mathcal{A}_{\mathrm{unif}}(a) A unif ( a ) , there must exist deterministic constants ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] , t 0 ∈ N t_0 \in\mathbb{N} t 0 ∈ N , and H , h ≥ 0 H,h \ge0 H , h ≥ 0 such that in every dimension and on every history,
0 ≤ β t < 1 , 0 ≤ λ t < ∞ , 0 ≤ η t ≤ H ( 1 + t ) h , 1 − β t ≥ ρ ( 1 + t ) − a ( t ≥ t 0 ) . (17) \begin{aligned}
0 &\le\beta_t < 1, \qquad0 \le\lambda_t < \infty, \qquad0 \le\eta_t \le H(1+t)^h,\\
1-\beta_t &\ge\rho(1+t)^{-a} \qquad(t \ge t_0). \tag*{(17)}
\end{aligned} 0 1 − β t ≤ β t < 1 , 0 ≤ λ t < ∞ , 0 ≤ η t ≤ H ( 1 + t ) h , ≥ ρ ( 1 + t ) − a ( t ≥ t 0 ) . (17)
These constants may depend on the rule, p p p , a a a , and δ \delta δ , but not on the dimension, the instance, or the seed.
Theorem 8 (Dimension-uniform lower bound under polynomial forgetting). Let C > 0 C > 0 C > 0 and k ≥ 0 k \ge0 k ≥ 0 be finite, let r > max { β p , a } r > \max\{\beta_p,a\} r > max { β p , a } , and fix a confidence δ ∈ ( 0 , 1 / 2 ) \delta\in(0,1/2) δ ∈ ( 0 , 1/2 ) . For any rule in A u n i f ( a ) \mathcal{A}_{\mathrm{unif}}(a) A unif ( a ) , there is a finite-dimensional instance with L = R = σ = 1 L = R = \sigma= 1 L = R = σ = 1 for which
P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > C t − r log k ( e t ) } > δ . (18) \mathbb{P}\{\exists t \ge1 : f(x_{t+1}) - f^\star> Ct^{-r}\log^k(et)\} > \delta. \tag*{(18)} P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > C t − r log k ( e t )} > δ . (18) Failure occurs on a deterministic finite prefix. The instance satisfies Assumptions 1 and 2 with i.i.d. additive noise and convex, globally 1-smooth sample losses that attain their minima, and it is fixed before the samples and the seed are drawn; C C C and k k k are shared across dimensions.
For 0 ≤ a ≤ β p 0 \le a \le\beta_p 0 ≤ a ≤ β p , the optimal dimension-uniform exponent in A u n i f ( a ) \mathcal{A}_{\mathrm{unif}}(a) A unif ( a ) is therefore β p \beta_p β p : our horizon-free schedule in Theorem 2 attains this exponent with β t = 0 \beta_t = 0 β t = 0 and bounded step sizes, which satisfy (17) for every such a a a . When 0 < a < α p 0 < a < \alpha_p 0 < a < α p , the barrier max { β p , a } \max\{\beta_p,a\} max { β p , a } lies strictly below min { ( 1 + a ) β p , α p } \min\{(1+a)\beta_p,\alpha_p\} min {( 1 + a ) β p , α p } . Theorem 8 is proved in Appendix § 20 . Propositions 26 and 28 in the appendix give fixed-time versions of Theorems 7 and 8 when ρ \rho ρ , t 0 t_0 t 0 , and h h h do not depend on δ \delta δ and H H H grows at most polynomially in 1 / δ 1/\delta 1/ δ .
Querying Gradients at the Running Average
Both lower bounds above require 1 − β t ≥ ρ ( 1 + t ) − a 1-\beta_t \ge\rho(1+t)^{-a} 1 − β t ≥ ρ ( 1 + t ) − a for some a < 1 a < 1 a < 1 . When the memory grows faster, so that 1 − β t 1-\beta_t 1 − β t is of order 1 / t 1/t 1/ t , a clipped method that queries gradients at the running average of a base sequence attains the statistical exponent α p \alpha_p α p at all times simultaneously (Theorem 9 ).
Theorem 9 (Time-uniform averaging momentum). Let Assumptions 1 and 2 hold. For δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , define deterministic horizon-free parameters
c 0 = 1 128 , h δ = log 4 δ + 4 p p − 1 , ℓ ˉ t = h δ + log t , c_0 = \frac{1}{128},\qquad h_\delta= \log\frac{4}{\delta}+\frac{4p}{p-1},\qquad\bar{\ell}_t = h_\delta+\log t, c 0 = 128 1 , h δ = log δ 4 + p − 1 4 p , ℓ ˉ t = h δ + log t , λ t = max { 4 L R , σ t 1 / p } , γ t = c 0 R λ t ℓ ˉ t 2 . (19) \lambda_t = \max\{4LR,\sigma t^{1/p}\},\qquad\gamma_t = \frac{c_0R}{\lambda_t\bar{\ell}_t^2}. \tag*{(19)} λ t = max { 4 L R , σ t 1/ p } , γ t = λ t ℓ ˉ t 2 c 0 R . (19) Starting from z 1 = x 1 z_1=x_1 z 1 = x 1 , make one oracle call at x t x_t x t and update
d t = clip ( g t , λ t ) , z t + 1 = z t − γ t d t , d_t=\operatorname{clip}(g_t,\lambda_t),\qquad z_{t+1}=z_t-\gamma_td_t, d t = clip ( g t , λ t ) , z t + 1 = z t − γ t d t , x t + 1 = t t + 1 x t + 1 t + 1 z t + 1 . (20) x_{t+1}=\frac{t}{t+1}x_t+\frac{1}{t+1}z_{t+1}. \tag*{(20)} x t + 1 = t + 1 t x t + t + 1 1 z t + 1 . (20) With probability at least 1 − δ 1-\delta 1 − δ , simultaneously for every integer T ≥ 1 T\ge1 T ≥ 1 , one has ∥ z T + 1 − x ⋆ ∥ < 2 R \lVert z_{T+1}-x^\star\rVert<\sqrt{2}R ∥ z T + 1 − x ⋆ ∥ < 2 R and
f ( x T + 1 ) − f ⋆ ≤ 128 R λ T ℓ ˉ T 2 + 2 L R 2 T + 1 (21) f(x_{T+1})-f^\star\le\frac{128R\lambda_T\bar{\ell}_T^2+2LR^2}{T+1} \tag*{(21)} f ( x T + 1 ) − f ⋆ ≤ T + 1 128 R λ T ℓ ˉ T 2 + 2 L R 2 (21) ≤ 514 ℓ ˉ T 2 ( L R 2 T + σ R T ( p − 1 ) / p ) . (22) \le514\bar{\ell}_T^2\left(\frac{LR^2}{T}+\frac{\sigma R}{T^{(p-1)/p}}\right). \tag*{(22)} ≤ 514 ℓ ˉ T 2 ( T L R 2 + T ( p − 1 ) / p σ R ) . (22) The point x T + 1 x_{T+1} x T + 1 is the actual updated state after exactly T T T calls.
The update (20) adapts the anytime online-to-batch conversion of Cutkosky [9 ] to clipped stochastic gradients on an unbounded domain. This conversion builds on classical online-to-batch conversion (Cesa-Bianchi et al. [7 ] ), and Defazio [11 ] shows that updates of this form are equivalent to heavy-ball momentum. Unlike the schedule in Theorem 2, the schedule (19) takes δ \delta δ as an input, though it remains independent of the horizon. The trajectory of (20) can be written in the heavy-ball velocity form (14) with bounded step sizes and momentum coefficients satisfying α p / ( 2 t ) ≤ 1 − β t ≤ 1 / t \alpha_p/(2t)\le1-\beta_t\le1/t α p / ( 2 t ) ≤ 1 − β t ≤ 1/ t for all t ≥ 2 t\ge2 t ≥ 2 (Proposition 29 ), so this representation violates the forgetting condition in (15) for every a < 1 a<1 a < 1 . The same trajectory can also be realized with a constant momentum coefficient β t = b ∈ ( 0 , 1 ) \beta_t=b\in(0,1) β t = b ∈ ( 0 , 1 ) and exponentially growing step sizes (Proposition 30 ), so Theorem 7 needs some restriction on the growth of the step sizes. Unlike the three-sequence accelerated method of Nguyen et al. [48 ] , which also attains α p \alpha_p α p without a known horizon (Appendix § 21 ), the update (20) stays within the two-sequence heavy-ball form (14) used in our momentum lower bounds.
Discussion
Implications. Knowing the horizon T T T in advance allows an algorithm to tune its step sizes and clipping threshold to T T T , whereas a guarantee that covers adaptive checkpoint selection or early stopping must be time-uniform (§ 2 ). Under heavy-tailed noise, our results show that an unknown horizon has a polynomial cost for every rule whose iterate does not advance on rounds without new gradient information, including clipped SGD with any adaptive step sizes and thresholds: for these rules, the optimal exponent of a time-uniform last-iterate guarantee drops from α p \alpha_p α p to β p \beta_p β p (Theorems 2 and 4 and Proposition 21 ). Recovering α p \alpha_p α p without a known horizon therefore requires a rule outside this class, such as the running-average update of Theorem 9 , whose iterate x t x_t x t keeps moving toward the sequence z t z_t z t of (20) when the gradient is zero. Learning-rate-free methods do not avoid this cost either. Proposition 21 covers, in one dimension, clipped KT coin betting and the conic-lifting methods of Grand-Clément et al. [22 ] , and for their zero-initialized conic Blackwell algorithms CBA and CBA+ we prove more: the last iterate admits no time-uniform bound that tends to zero, even under i.i.d. additive noise (Proposition 22 ).
Formalization. We provide machine-checked proofs in Lean 4 with Mathlib of Theorems 2, 4 , and 8 and of the known-horizon bound of Theorem 1, including their proof dependencies. The code is provided as ancillary files of this arXiv entry and at https://github.com/Thefootofhappiness/clipped-sgd-last-iterate-lean . The formalized arguments include the recursive potential and the two martingale tail bounds, the integration of the probabilities of failing at the first crossing over a continuum of targets, and the orthogonal-projection argument behind the dimension-uniform momentum bound.
AI Usage Statement
During the preparation of this manuscript, the authors used AI tools, including OpenAI’s GPT models and Anthropic’s Claude, for mathematical derivations and their verification, and to assist with writing. The authors take full responsibility for the content of this paper.
Related Work
Heavy-Tailed Noise and Gradient Clipping
Under the classical model with bounded p p p th moments ([45 , 69 ] ), truncated mirror descent ([43 ] ), smooth clipped methods ([21 , 58 , 48 ] ), and nonsmooth methods ([51 , 52 ] ) achieve high-probability statistical rates, typically for averaged or auxiliary outputs. Nekrasova et al. [44 ] relax the moment bound to a weak-L p L_p L p tail condition, with a noise scale that may grow with the gradient norm, and obtain high-probability rates for clipped SGD at its averaged iterate and for its accelerated variant at its last iterate, with clipping levels tuned to the horizon. Clipping is not always beneficial for averages: Suvorikova et al. [63 ] compare clipped and unclipped Polyak–Ruppert averaged SGD ([57 , 55 ] ) under finite p p p th moments with p ≥ 2 p \ge2 p ≥ 2 and show that, for p > 2 p > 2 p > 2 , clipping need not improve the leading term. Directional moments sharpen rates for structured instances ([38 ] (Zhang et al. 2026)).
In decentralized smooth nonconvex optimization, Armacki et al. [3 ] analyze clipped DSGD under bounded central p p p th noise moments and establish stationarity bounds in high probability and in expectation, with linear speedup in the leading term. They bound averages over time of squared gradient norms across agents; we bound convex objective error at the last iterate and distinguish fixed-time from time-uniform behavior.
Last-Iterate Convergence
Schedules for a known horizon remove the logarithmic last-iterate factor of standard nonsmooth SGD ([60 , 24 , 27 ] (Zamani and Glineur 2025)); for one-dimensional Lipschitz objectives with i.i.d. noise, constant step sizes tuned to the horizon already suffice, which fails without the i.i.d. assumption ([4 ] ). Smooth analyses based on moving comparators ([39 ] ) and online-to-batch regret reductions ([12 ] ) require stronger tail assumptions for high-probability guarantees. The bound in expectation under finite p p p th moments of Liu and Zhou (2026, Section 5) uses a p / ( p − 1 ) p/(p - 1) p / ( p − 1 ) -uniformly convex mirror map, excluding Euclidean SGD at p < 2 p < 2 p < 2 . Parletta et al. [52 ] treat the last iterate of clipped SGD on Lipschitz objectives in expectation. For AdaGrad ([16 ] ) and AdaGrad-Norm ([62 , 70 ] ) with fixed parameters, Ou and Wang [50 ] show that the expected last-iterate error can decay arbitrarily slowly under an oracle whose noise depends on time and history. In our smooth setting, Chezhegov et al. [8 ] obtain the horizon-free exponent 2 ( p − 1 ) / ( 3 p ) 2(p - 1)/(3p) 2 ( p − 1 ) / ( 3 p ) ; for clipped SGD with an unknown horizon, we establish β p \beta_p β p for time-uniform guarantees and α p \alpha_p α p for fixed-time guarantees with polynomial dependence on 1 / δ 1/\delta 1/ δ .
With a known horizon, recent work attains α p \alpha_p α p at the last iterate. Liu [37 ] treats Lipschitz and smooth convex objectives, strongly star-convex objectives, and composite and constrained problems under a fine-grained heavy-tailed noise model; Theorem 2 there covers smooth convex objectives with a state-dependent threshold and a refined noise dependence. Theorem 1 of Jiang et al. [28 ] is the bound we restate as Theorem 1; it avoids any log T \log T log T factor and carries constants independent of p p p and the dimension. They also treat composite objectives with proximal updates and oracles under adversarial sample corruption, and establish two-point statistical lower bounds ([33 , 68 ] ). Both papers assume a known horizon; Liu [37 ] does so to avoid a potential logarithmic dependence on T T T . For gradient-only rules (Definition 3 ) with polylogarithmic dependence on 1 / δ 1/\delta 1/ δ , Theorems 2, 4 , and 6 show that when the horizon is unknown the optimal exponent drops from α p \alpha_p α p to β p \beta_p β p . Liu [37 ] lists matching lower bounds for smooth objectives as open; Jiang et al. [28 ] matches the stochastic term when the stochastic error dominates.
The proof of Jiang et al. [28 ] combines the tail-sum potential (Appendix § 12 .§ 2 ) with a stopping-time localization of the iterates. When the stochastic term dominates, setting λ = σ ( T / ℓ ) 1 / p \lambda= \sigma(T/\ell)^{1/p} λ = σ ( T / ℓ ) 1/ p balances the cumulative clipping bias T σ p λ 1 − p T\sigma^p\lambda^{1-p} T σ p λ 1 − p against the deviation scale λ ℓ \lambda\ell λ ℓ , while each update moves the iterate by at most R / ( 128 ℓ ) R/(128\ell) R / ( 128 ℓ ) ; the final state x T + 1 x_{T+1} x T + 1 then attains the statistical scale (2) , which is optimal up to constants in this regime (Jiang et al., [28 ] ). The deterministic term L R 2 ℓ / T LR^2\ell/T L R 2 ℓ / T matches unaccelerated gradient descent, whereas acceleration attains L R 2 / T 2 LR^2/T^2 L R 2 / T 2 without noise (Nesterov, [47 ] ; Lan [32 ] ).
Unknown Horizons and Time-Uniform Guarantees
For the subgradient method, no single sequence of step sizes attains the optimal last-iterate rate at every iteration (Zamani and Glineur, 2025). Deterministic anytime lower bounds use flat objectives and overshoots (Kornowski and Shamir, [30 ] ; Tsai et al. [67 ] ; Kornowski and Shamir [31 ] ). Our stochastic lower bound additionally handles adaptive responses to a shared i.i.d. transcript. Under conditional norm-sub-Gaussian noise, Li et al. (2026, Theorems 2.3, 3.4, and 4.1) characterize exactly which overhead factors over the rate t − 1 / 2 t^{-1/2} t − 1/2 are attainable at all times; by their Corollary 2.4 and the example after it, ( log t ) ( 1 + ε ) / 2 (\log t)^{(1+\varepsilon)/2} ( log t ) ( 1 + ε ) /2 is attainable for every ε > 0 \varepsilon> 0 ε > 0 , whereas ( log t ) 1 / 2 (\log t)^{1/2} ( log t ) 1/2 is not. Finite second moments already permit the polynomial separation studied here.
Almost sure convergence. Under the ABC condition on conditional second moments (Khaled and Richtárik, [29 ] ), Liu and Yuan [35 ] and Liu and Yuan (2024, Theorem 13) obtain almost sure last-iterate rates O ( t − 1 / 3 + ε ) O(t^{-1/3+\varepsilon}) O ( t − 1/3 + ε ) on smooth convex objectives for unclipped SGD, heavy-ball (Polyak, [54 ] ), and Nesterov accelerated methods with deterministic step sizes of order t − 2 / 3 − ε t^{-2/3-\varepsilon} t − 2/3 − ε , 0 < ε < 1 / 3 0 < \varepsilon< 1/3 0 < ε < 1/3 . Sebbouh et al. [59 ] obtain o ( 1 / ∑ j < k η j ) o(1/\sum_{j<k}\eta_j) o ( 1/ ∑ j < k η j ) for weighted SGD averages and moving-average heavy-ball iterates under smooth convex sample losses, finite residual variance at the minimizer, and suitable step sizes and averaging weights. Concurrent and independent work by Anonymous [1 ] shows that, for every positive deterministic schedule of unclipped step sizes and every r > 1 / 3 r > 1/3 r > 1/3 , a one-dimensional smooth convex instance with a fixed i.i.d. finite-variance oracle satisfies lim sup t → ∞ ( t + 1 ) r ( f ( x t ) − f ⋆ ) = ∞ \limsup_{t\to\infty}(t+1)^r(f(x_t)-f^\star)=\infty lim sup t → ∞ ( t + 1 ) r ( f ( x t ) − f ⋆ ) = ∞ almost surely. Their Theorem 2.1 and Corollary 2.4 identify 1 / 3 1/3 1/3 as the supremal almost sure exponent under the ABC condition, and Proposition 2.5 establishes a matching rate up to logarithmic factors. Remark 16 adapts their lower bound to the time-uniform setting at a fixed confidence level δ \delta δ and compares it with clipping.
Anytime online-to-batch conversion (Cutkosky, [9 ] ), primal averaging (Defazio, [11 ] ; Defazio et al. [13 ] ), and accelerated clipped methods (Nguyen et al., [48 ] ) query coupled or averaged states. Schedule-free SGD (Defazio et al., [13 ] ) evaluates gradients between a running average and a base sequence. When it evaluates them at the average, its update with clipped gradients has the form (20) , except that its main form uses a constant step size, whereas γ t \gamma_t γ t in (20) decreases. In overparameterized linear regression, Meterez et al. [42 ] show that horizon-free, polynomially decaying step sizes combined with weight averaging attain the minimax expected rate of SGD up to logarithmic factors under source and capacity conditions, and in language-model pretraining they find such schedules with averaging comparable to a tuned cosine schedule. Section 6 compares such memory with geometric forgetting. The upper proofs use the maximal concentration inequalities in Appendix § 14 ; for broader time-uniform martingale methods, see Howard et al. [25 ] .
Momentum lower bounds on quadratics also appear in Ganesh et al. [20 ] , for constant parameters and expected error. Drori and Shamir [15 ] use orthogonal noise for nonconvex stationarity, with coordinates indexed by time and step sizes and weights that may depend on the norms and inner products of all past iterates and stochastic gradients. Our momentum
lower bound uses a fixed i.i.d. coordinate distribution and time-uniform convex objective error; its scalar controls may depend arbitrarily on signs, coordinates, and an independent seed, with constants shared across dimensions.
A Numerical Illustration
We illustrate the separation between α p \alpha_p α p and β p \beta_p β p numerically. Figure 1 compares the methods on the hard instances of Appendix § 16 .§ 1 at p = 2 p = 2 p = 2 . For each horizon N N N we use the targets y ∈ { 1 / 4 , 1 / 2 , 3 / 4 , 1 } y \in\{1/4, 1/2, 3/4, 1\} y ∈ { 1/4 , 1/2 , 3/4 , 1 } , the slopes μ = min { 0.2 , 1.3 N − α p } \mu= \min\{0.2, 1.3N^{-\alpha_p}\} μ = min { 0.2 , 1.3 N − α p } and μ = min { 0.2 , 1.3 N − β p } \mu= \min\{0.2, 1.3N^{-\beta_p}\} μ = min { 0.2 , 1.3 N − β p } , and the one-sided spike noise of (29) ; every method sees the same instances and the same spikes. Every method starts at x 1 = 0 x_1 = 0 x 1 = 0 , with z 1 = 0 z_1 = 0 z 1 = 0 for averaging and zero initial velocity for heavy-ball momentum, and N N N ranges over 500 ⋅ 2 k 500 \cdot2^k 500 ⋅ 2 k for k = 0 , … , 6 k = 0, \ldots, 6 k = 0 , … , 6 . Clipped SGD uses the time-uniform schedule η t = t − ( p + 1 ) / ( 3 p − 1 ) \eta_t = t^{-(p+1)/(3p-1)} η t = t − ( p + 1 ) / ( 3 p − 1 ) , λ t = 2 t 2 / ( 3 p − 1 ) \lambda_t = 2t^{2/(3p-1)} λ t = 2 t 2/ ( 3 p − 1 ) ; heavy-ball momentum uses β t = 0.8 \beta_t = 0.8 β t = 0.8 , the same thresholds, and step sizes 0.2 t − ( p + 1 ) / ( 3 p − 1 ) 0.2t^{-(p+1)/(3p-1)} 0.2 t − ( p + 1 ) / ( 3 p − 1 ) ; averaging uses (20) with γ t = 0.5 t − 1 / p \gamma_t = 0.5t^{-1/p} γ t = 0.5 t − 1/ p and λ t = 2 t 1 / p \lambda_t = 2t^{1/p} λ t = 2 t 1/ p ; and Algorithm 1 uses λ = 2 N 1 / p \lambda= 2N^{1/p} λ = 2 N 1/ p and η t = 1.5 ( N − t + 1 ) / ( N λ ) \eta_t = 1.5(N - t + 1)/(N\lambda) η t = 1.5 ( N − t + 1 ) / ( N λ ) . These constants are of order one and do not change with N N N ; the theorems use smaller step sizes, different thresholds, and logarithmic factors. The script that produces the figure and its outputs are provided as ancillary files of this arXiv entry. For each method we take, over 1000 runs, the 0.9-quantile of the largest gap f ( x t + 1 ) − f ⋆ f(x_{t+1}) - f^\star f ( x t + 1 ) − f ⋆ over N / 2 ≤ t ≤ N N/2 \le t \le N N /2 ≤ t ≤ N , or of the terminal gap for Algorithm 1, and report its maximum over the eight instances.
Figure 1. Algorithm 1 for a known horizon, averaging (Theorem 9 ), time-uniform clipped SGD (Theorem 2), and heavy-ball momentum with β t = 0.8 \beta_t = 0.8 β t = 0.8 , with the exponents of the theorems and illustrative constants, on the hard instances of Appendix § 16 .§ 1 at p = 2 p = 2 p = 2 : worst case over eight instances of the 0.9-quantile, over 1000 runs, of the largest gap over N / 2 ≤ t ≤ N N/2 \le t \le N N /2 ≤ t ≤ N (terminal gap for Algorithm 1). Slopes are least-squares fits in log N \log N log N ; their 95% bootstrap intervals over the runs (200 resamples) have half-width below 0.02.
Algorithm 1 and averaging decay with fitted slopes − 0.50 -0.50 − 0.50 and − 0.55 -0.55 − 0.55 , close to − α 2 = − 1 / 2 -\alpha_2 = -1/2 − α 2 = − 1/2 , whereas clipped SGD and heavy-ball momentum decay with slopes − 0.35 -0.35 − 0.35 and − 0.30 -0.30 − 0.30 , and at N = 3.2 × 10 4 N = 3.2 \times10^4 N = 3.2 × 1 0 4 their worst gaps are 2.5 to 3 times those of the first two. Slopes fitted over this range of N N N also reflect constants and logarithmic factors. At p = 1.5 p = 1.5 p = 1.5 the experiment does not separate averaging from the other methods. There the exponents α p = 1 / 3 \alpha_p = 1/3 α p = 1/3 and β p = 2 / 7 \beta_p = 2/7 β p = 2/7 differ by less than 0.05. The fitted slopes are − 0.31 -0.31 − 0.31 for Algorithm 1 and for averaging and − 0.24 -0.24 − 0.24 and − 0.22 -0.22 − 0.22 for the other two methods, but at every N N N in this range the worst gap of averaging stays above those of clipped SGD and heavy-ball momentum; only the worst gap of Algorithm 1 falls below them, at N = 3.2 × 10 4 N = 3.2 \times10^4 N = 3.2 × 1 0 4 .
Open Problems
Our schedules use L L L , R R R , σ \sigma σ , p p p and, except for the time-uniform schedule and the schedule of Theorem 5 with u = 0 u = 0 u = 0 , a confidence input; adaptation to unknown problem parameters is a natural next step. The matching results concern polynomial exponents, so sharp logarithmic factors and constants, parameter interpolation, and the optimal confidence exponent c ⋆ ( r ) c^{\star}(r) c ⋆ ( r ) of fixed-time guarantees, bracketed in (13) , are left for future work. At r = α p r = \alpha_p r = α p , the upper end of this bracket equals α p \alpha_p α p , whereas the lower end equals the gap between the statistical and time-uniform exponents,
α p − β p = ( p − 1 ) 2 p ( 3 p − 1 ) > 0 , \alpha_p - \beta_p = \frac{(p-1)^2}{p(3p-1)} > 0, α p − β p = p ( 3 p − 1 ) ( p − 1 ) 2 > 0 ,
which is 1 / 10 1/10 1/10 at p = 2 p = 2 p = 2 . The two ends of this bracket come from different proof mechanisms: the upper bound truncates late clipping errors at levels set by the analysis horizon and bounds the truncation probability via the p p p th moment, whereas the lower bound applies a union bound over the first N N N steps without using the fact that iterates on our hard instance move only at rare gradient spikes before crossing. The two momentum lower bounds concern updates with zero initial velocity, queries and outputs at the current state, gradients as the only oracle information, and polynomially bounded nonnegative step sizes; the dimension-uniform theorem also uses control and rate constants shared across dimensions. Within this class, three questions remain: how the gap between the one-dimensional barrier ( 1 + a ) β p (1+a)\beta_p ( 1 + a ) β p and the dimension-uniform barrier max { β p , a } \max\{\beta_p,a\} max { β p , a } closes, which rates are attainable for a > β p a > \beta_p a > β p , also under a two-sided condition 1 − β t ≍ t − a 1-\beta_t \asymp t^{-a} 1 − β t ≍ t − a , and how the confidence factor of a fixed-time exponent above max { β p , a } \max\{\beta_p,a\} max { β p , a } grows under shared constants. Lower bounds for general update directions are also open. Concurrent analyses for a known horizon cover Lipschitz, strongly star-convex, composite, and constrained problems (Liu, [37 ] ; Jiang et al., [28 ] ); horizon-free counterparts of these settings, and of non-Euclidean or ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smooth objectives (El Bakkali El Kadi et al., [17 ] ), are further directions. For nonconvex stationarity, the bound on the average squared gradient norm over the trajectory in Nguyen et al. [48 ] , Theorem B.2 retains the exponent for a known horizon up to logarithmic factors; obtaining a guarantee for a single selected iterate requires a separate selection argument, for example stationarity tests whose oracle calls are counted in the budget, combined with the doubling trick (Cesa-Bianchi and Lugosi, [6 ] ).
Deterministic Last-Iterate Calculations
This appendix proves the one-step inequality and the recursive potential (Lemma 11 ).
One Inexact Gradient Step
Lemma 10 (Inexact smooth convex step). Let f f f be convex and L L L -smooth. Fix x , z , e ∈ R d x,z,e \in\mathbb{R}^d x , z , e ∈ R d , 0 < γ ≤ 1 / ( 2 L ) 0 < \gamma\le1/(2L) 0 < γ ≤ 1/ ( 2 L ) , and y = x − γ ( ∇ f ( x ) + e ) y = x - \gamma(\nabla f(x) + e) y = x − γ ( ∇ f ( x ) + e ) . Then
γ [ f ( y ) − f ( z ) ] + 1 2 ∥ y − z ∥ 2 ≤ 1 2 ∥ x − z ∥ 2 + γ ⟨ e , z − x ⟩ + γ 2 ∥ e ∥ 2 . (23) \gamma[f(y)-f(z)] + \frac{1}{2}\lVert y-z\rVert^2 \le\frac{1}{2}\lVert x-z\rVert^2 + \gamma\langle e,z-x\rangle+ \gamma^2\lVert e\rVert^2. \tag*{(23)} γ [ f ( y ) − f ( z )] + 2 1 ∥ y − z ∥ 2 ≤ 2 1 ∥ x − z ∥ 2 + γ ⟨ e , z − x ⟩ + γ 2 ∥ e ∥ 2 . (23)
Proof Smoothness, convexity and the three-point norm identity give
γ [ f ( y ) − f ( z ) ] + 1 2 ∥ y − z ∥ 2 ≤ 1 2 ∥ x − z ∥ 2 + γ ⟨ e , z − x ⟩ − 1 − γ L 2 ∥ x − y ∥ 2 + γ ⟨ e , x − y ⟩ . \gamma[f(y)-f(z)]+\frac{1}{2}\lVert y-z\rVert^2 \le\frac{1}{2}\lVert x-z\rVert^2+\gamma\langle e,z-x\rangle-\frac{1-\gamma L}{2}\lVert x-y\rVert^2+\gamma\langle e,x-y\rangle. γ [ f ( y ) − f ( z )] + 2 1 ∥ y − z ∥ 2 ≤ 2 1 ∥ x − z ∥ 2 + γ ⟨ e , z − x ⟩ − 2 1 − γ L ∥ x − y ∥ 2 + γ ⟨ e , x − y ⟩ .
Use γ ⟨ e , x − y ⟩ ≤ γ 2 ∥ e ∥ 2 + ∥ x − y ∥ 2 / 4 \gamma\langle e,x-y\rangle\le\gamma^2\lVert e\rVert^2+\lVert x-y\rVert^2/4 γ ⟨ e , x − y ⟩ ≤ γ 2 ∥ e ∥ 2 + ∥ x − y ∥ 2 /4 and γ L ≤ 1 / 2 \gamma L\le1/2 γ L ≤ 1/2 .
A Recursively Averaged Potential
Potential functions are a standard tool for analyzing first-order methods Taylor and Bach [64 ] . Following the comparator weighted by inverse tail sums in Zamani and Glineur (2025) and Liu and Zhou (2024), we express the potential recursively. With boundary mass ζ = 0 \zeta=0 ζ = 0 below, it is the tail-sum potential that Jiang et al. [28 ] use to prove Theorem 1; the boundary mass ζ > 0 \zeta>0 ζ > 0 , which keeps the weights bounded without a horizon, is what our unknown-horizon results add. For any positive steps η t ≤ 1 / ( 2 L ) \eta_t\le1/(2L) η t ≤ 1/ ( 2 L ) and arbitrary realized errors, consider
x t + 1 = x t − η t ( ∇ f ( x t ) + e t ) . x_{t+1}=x_t-\eta_t(\nabla f(x_t)+e_t). x t + 1 = x t − η t ( ∇ f ( x t ) + e t ) .
For a deterministic analysis horizon T T T and boundary mass ζ ≥ 0 \zeta\ge0 ζ ≥ 0 , set
G t = f ( x t ) − f ⋆ , S t = ∑ r = t T η r , U t = ζ + S t , W = U 1 , θ t = η t / U t , G_t=f(x_t)-f^\star,\qquad S_t=\sum_{r=t}^{T}\eta_r,\qquad U_t=\zeta+S_t,\qquad W=U_1,\qquad\theta_t=\eta_t/U_t, G t = f ( x t ) − f ⋆ , S t = r = t ∑ T η r , U t = ζ + S t , W = U 1 , θ t = η t / U t ,
with S 0 = S 1 S_0=S_1 S 0 = S 1 , S T + 1 = 0 S_{T+1}=0 S T + 1 = 0 and U 0 = U 1 U_0=U_1 U 0 = U 1 . All denominators below involve only U 0 , … , U T > 0 U_0,\ldots,U_T>0 U 0 , … , U T > 0 , so ζ = 0 \zeta=0 ζ = 0 is allowed. Starting from z 0 = z 1 = x ⋆ z_0=z_1=x^\star z 0 = z 1 = x ⋆ and A 0 = 0 A_0=0 A 0 = 0 , define only for the analysis
z t + 1 = ( 1 − θ t ) z t + θ t x t + 1 , A t = ( 1 − θ t ) A t − 1 + θ t G t + 1 . z_{t+1}=(1-\theta_t)z_t+\theta_t x_{t+1},\qquad A_t=(1-\theta_t)A_{t-1}+\theta_t G_{t+1}. z t + 1 = ( 1 − θ t ) z t + θ t x t + 1 , A t = ( 1 − θ t ) A t − 1 + θ t G t + 1 .
Let
d t = z t − 1 − x t , H 0 = 1 2 ∥ d 1 ∥ 2 , H t = W 2 U t ∥ z t − x t + 1 ∥ 2 , d_t=z_{t-1}-x_t,\qquad H_0=\frac{1}{2}\lVert d_1\rVert^2,\qquad H_t=\frac{W}{2U_t}\lVert z_t-x_{t+1}\rVert^2, d t = z t − 1 − x t , H 0 = 2 1 ∥ d 1 ∥ 2 , H t = 2 U t W ∥ z t − x t + 1 ∥ 2 ,
q t = 2 W η t 2 U t , h t = W η t U t − 1 d t . q_t=\frac{2W\eta_t^2}{U_t},\qquad h_t=\frac{W\eta_t}{U_{t-1}}d_t. q t = U t 2 W η t 2 , h t = U t − 1 W η t d t .
In particular H T H_T H T uses z T − x T + 1 z_T-x_{T+1} z T − x T + 1 , not z T + 1 − x T + 1 z_{T+1}-x_{T+1} z T + 1 − x T + 1 .
Lemma 11 (Unified recursive potential). For every m ≤ T m\le T m ≤ T , A m , H m ≥ 0 A_m,H_m\ge0 A m , H m ≥ 0 and
W A m + H m ≤ R 2 2 + ∑ t ≤ m q t ∥ e t ∥ 2 + ∑ t ≤ m ⟨ e t , h t ⟩ . (24) W A_m+H_m\le\frac{R^2}{2}+\sum_{t\le m}q_t\lVert e_t\rVert^2+\sum_{t\le m}\langle e_t,h_t\rangle. \tag*{(24)} W A m + H m ≤ 2 R 2 + t ≤ m ∑ q t ∥ e t ∥ 2 + t ≤ m ∑ ⟨ e t , h t ⟩ . (24)
The full terminal potential also gives
G T + 1 ≤ ( 1 + L ζ 2 ζ + η T ) W A T + H T W ≤ ( 1 + L ζ ) W A T + H T W . (25) \begin{aligned}
G_{T+1}\le\left(1+\frac{L\zeta^2}{\zeta+\eta_T}\right)\frac{W A_T+H_T}{W} \\
&\le(1+L\zeta)\frac{W A_T+H_T}{W}. \tag*{(25)}
\end{aligned} G T + 1 ≤ ( 1 + ζ + η T L ζ 2 ) W W A T + H T ≤ ( 1 + L ζ ) W W A T + H T . (25)
In particular, ζ = 0 \zeta=0 ζ = 0 gives A T = G T + 1 A_T=G_{T+1} A T = G T + 1 , while ζ = 1 \zeta=1 ζ = 1 keeps the inverse tail weights bounded.
Proof. Convexity and the synchronized recursions give f ( z t ) − f ⋆ ≤ A t − 1 f(z_t)-f^\star\le A_{t-1} f ( z t ) − f ⋆ ≤ A t − 1 by induction. Thus A t − A t − 1 ≤ θ t [ f ( x t + 1 ) − f ( z t ) ] A_t-A_{t-1}\le\theta_t[f(x_{t+1})-f(z_t)] A t − A t − 1 ≤ θ t [ f ( x t + 1 ) − f ( z t )] . Also z t − x t = ( U t / U t − 1 ) d t z_t-x_t=(U_t/U_{t-1})d_t z t − x t = ( U t / U t − 1 ) d t , including t = 1 t=1 t = 1 by convention. Multiply (23) of Lemma 10, with z = z t z=z_t z = z t and γ = η t \gamma=\eta_t γ = η t , by W / U t W/U_t W / U t to obtain
W A t + H t ≤ W A t − 1 + U t U t − 1 H t − 1 + q t 2 ∥ e t ∥ 2 + ⟨ e t , h t ⟩ . W A_t+H_t\le W A_{t-1}+\frac{U_t}{U_{t-1}}H_{t-1}+\frac{q_t}{2}\lVert e_t\rVert^2+\langle e_t,h_t\rangle. W A t + H t ≤ W A t − 1 + U t − 1 U t H t − 1 + 2 q t ∥ e t ∥ 2 + ⟨ e t , h t ⟩ .
Since U t ≤ U t − 1 U_t\le U_{t-1} U t ≤ U t − 1 , summation proves (24) , using q t / 2 ≤ q t q_t/2\le q_t q t /2 ≤ q t . Nonnegativity follows from the definitions. The final synchronized update gives f ( z T + 1 ) − f ⋆ ≤ A T f(z_{T+1})-f^\star\le A_T f ( z T + 1 ) − f ⋆ ≤ A T and
∥ z T + 1 − x T + 1 ∥ 2 = 2 ζ 2 H T W ( ζ + η T ) . \lVert z_{T+1}-x_{T+1}\rVert^2=\frac{2\zeta^2 H_T}{W(\zeta+\eta_T)}. ∥ z T + 1 − x T + 1 ∥ 2 = W ( ζ + η T ) 2 ζ 2 H T .
By L L L -smoothness evaluated at z − ∇ f ( z ) / L z-\nabla f(z)/L z − ∇ f ( z ) / L , we have ∥ ∇ f ( z ) ∥ 2 ≤ 2 L [ f ( z ) − f ⋆ ] \lVert\nabla f(z)\rVert^2\le2L[f(z)-f^\star] ∥ ∇ f ( z ) ∥ 2 ≤ 2 L [ f ( z ) − f ⋆ ] . Thus, writing G ( x ) = f ( x ) − f ⋆ G(x)=f(x)-f^\star G ( x ) = f ( x ) − f ⋆ and ρ = L ζ 2 / ( ζ + η T ) \rho=L\zeta^2/(\zeta+\eta_T) ρ = L ζ 2 / ( ζ + η T ) , smoothness and Cauchy–Schwarz give
G ( x T + 1 ) ≤ ( G ( z T + 1 ) + L / 2 ∥ x T + 1 − z T + 1 ∥ ) 2 ≤ ( 1 + ρ ) ( A T + H T / W ) . G(x_{T+1})\le\left(\sqrt{G(z_{T+1})}+\sqrt{L/2}\lVert x_{T+1}-z_{T+1}\rVert\right)^2\le(1+\rho)(A_T+H_T/W). G ( x T + 1 ) ≤ ( G ( z T + 1 ) + L /2 ∥ x T + 1 − z T + 1 ∥ ) 2 ≤ ( 1 + ρ ) ( A T + H T / W ) .
Since ρ ≤ L ζ \rho\le L\zeta ρ ≤ L ζ , this proves (25) . If ζ = 0 \zeta=0 ζ = 0 , θ T = 1 \theta_T=1 θ T = 1 also gives A T = G T + 1 A_T=G_{T+1} A T = G T + 1 directly. ■
The proofs of Theorems 2 and 5 take ζ = 1 \zeta=1 ζ = 1 ; ζ = 0 \zeta=0 ζ = 0 gives W = S 1 W=S_1 W = S 1 and U t = S t U_t=S_t U t = S t .
Conditional Clipping Estimates
Lemma 12 (Bias and second moment after clipping). *Conditionally on a σ \sigma σ -field G \mathcal{G} G , suppose μ \mu μ is measurable, E [ g ∣ G ] = μ \mathbb{E}[g\mid\mathcal{G}]=\mu E [ g ∣ G ] = μ , and E [ ∥ g − μ ∥ p ∣ G ] ≤ σ p \mathbb{E}[\lVert g-\mu\rVert^p\mid\mathcal{G}]\le\sigma^p E [∥ g − μ ∥ p ∣ G ] ≤ σ p , for 1 < p ≤ 2 1<p\le2 1 < p ≤ 2 . If ∥ μ ∥ ≤ λ / 2 \lVert\mu\rVert\le\lambda/2 ∥ μ ∥ ≤ λ /2 , set
c = clip ( g , λ ) , e = c − μ , b = E [ e ∣ G ] , u = e − b . c=\operatorname{clip}(g,\lambda),\qquad e=c-\mu,\qquad b=\mathbb{E}[e\mid\mathcal{G}],\qquad u=e-b. c = clip ( g , λ ) , e = c − μ , b = E [ e ∣ G ] , u = e − b . Then
∥ e ∥ ≤ min { ∥ g − μ ∥ , 3 λ / 2 } , ∥ u ∥ ≤ 2 λ , \lVert e\rVert\le\min\{\lVert g-\mu\rVert,3\lambda/2\},\qquad\lVert u\rVert\le2\lambda, ∥ e ∥ ≤ min {∥ g − μ ∥ , 3 λ /2 } , ∥ u ∥ ≤ 2 λ , ∥ b ∥ ≤ 2 p − 1 σ p λ 1 − p ≤ 2 σ p λ 1 − p \lVert b\rVert\le2^{p-1}\sigma^p\lambda^{1-p}\le2\sigma^p\lambda^{1-p} ∥ b ∥ ≤ 2 p − 1 σ p λ 1 − p ≤ 2 σ p λ 1 − p E [ ∥ e ∥ 2 ∣ G ] ≤ ( 3 λ / 2 ) 2 − p σ p ≤ 2 σ p λ 2 − p , E [ ∥ u ∥ 2 ∣ G ] ≤ E [ ∥ e ∥ 2 ∣ G ] . \mathbb{E}[\lVert e\rVert^2\mid\mathcal{G}]\le(3\lambda/2)^{2-p}\sigma^p\le2\sigma^p\lambda^{2-p},\qquad\mathbb{E}[\lVert u\rVert^2\mid\mathcal{G}]\le\mathbb{E}[\lVert e\rVert^2\mid\mathcal{G}]. E [∥ e ∥ 2 ∣ G ] ≤ ( 3 λ /2 ) 2 − p σ p ≤ 2 σ p λ 2 − p , E [∥ u ∥ 2 ∣ G ] ≤ E [∥ e ∥ 2 ∣ G ] .
Proof. These are the standard clipping bounds (cf. [58 Lemma 5.1] ); the following Euclidean calculation records the sharper constants used here. Write P = clip ( ⋅ , λ ) P=\operatorname{clip}(\mathord{\cdot},\lambda) P = clip ( ⋅ , λ ) and ξ = g − μ \xi=g-\mu ξ = g − μ . Since P μ = μ P\mu=\mu P μ = μ , nonexpansiveness and the clipping radius give
∥ e ∥ ≤ min { ∥ ξ ∥ , 3 λ / 2 } , ∥ u ∥ = ∥ P g − E [ P g ∣ G ] ∥ ≤ 2 λ . \lVert e\rVert\le\min\{\lVert\xi\rVert,3\lambda/2\},\qquad\lVert u\rVert=\lVert Pg-\mathbb{E}[Pg\mid\mathcal{G}]\rVert\le2\lambda. ∥ e ∥ ≤ min {∥ ξ ∥ , 3 λ /2 } , ∥ u ∥ = ∥ P g − E [ P g ∣ G ]∥ ≤ 2 λ .
Also ∥ P g − g ∥ ≤ ∥ ξ ∥ 1 { ∥ ξ ∥ > λ / 2 } \lVert Pg-g\rVert\le\lVert\xi\rVert\mathbf{1}_{\{\lVert\xi\rVert>\lambda/2\}} ∥ P g − g ∥ ≤ ∥ ξ ∥ 1 {∥ ξ ∥ > λ /2 } , so
∥ b ∥ = ∥ E [ P g − g ∣ G ] ∥ ≤ ( λ / 2 ) 1 − p E [ ∥ ξ ∥ p ∣ G ] . \lVert b\rVert=\lVert\mathbb{E}[Pg-g\mid\mathcal{G}]\rVert\le(\lambda/2)^{1-p}\mathbb{E}[\lVert\xi\rVert^p\mid\mathcal{G}]. ∥ b ∥ = ∥ E [ P g − g ∣ G ]∥ ≤ ( λ /2 ) 1 − p E [∥ ξ ∥ p ∣ G ] .
Finally,
E [ ∥ u ∥ 2 ∣ G ] ≤ E [ ∥ e ∥ 2 ∣ G ] ≤ ( 3 λ / 2 ) 2 − p E [ ∥ ξ ∥ p ∣ G ] , \mathbb{E}[\lVert u\rVert^2\mid\mathcal{G}]\le\mathbb{E}[\lVert e\rVert^2\mid\mathcal{G}]\le(3\lambda/2)^{2-p}\mathbb{E}[\lVert\xi\rVert^p\mid\mathcal{G}], E [∥ u ∥ 2 ∣ G ] ≤ E [∥ e ∥ 2 ∣ G ] ≤ ( 3 λ /2 ) 2 − p E [∥ ξ ∥ p ∣ G ] ,
where the first inequality is conditional variance reduction by centering. ■ The next decomposition imposes no bound on the true gradient. Related Lipschitz estimates for the clipping factor appear in Tian et al. [65 ] , Appendix C.1.2, Eqs. (13)–(14), under bounded noise. We prove the conditional p p p -moment bounds used here.
Lemma 13 (Conditional mean and variance after clipping). Conditionally on a σ \sigma σ -field G \mathcal{G} G , let μ \mu μ be measurable, E [ g ∣ G ] = μ \mathbb{E}[g \mid\mathcal{G}] = \mu E [ g ∣ G ] = μ , and E [ ∥ g − μ ∥ p ∣ G ] ≤ σ p \mathbb{E}[\lVert g-\mu\rVert^{p} \mid\mathcal{G}] \le\sigma^{p} E [∥ g − μ ∥ p ∣ G ] ≤ σ p . For deterministic or G \mathcal{G} G -measurable λ > 0 \lambda> 0 λ > 0 , put
α ( v ) = { min { 1 , λ / ∥ v ∥ } , v ≠ 0 , 1 , v = 0 , C = clip ( g , λ ) , A = E [ α ( g ) ∣ G ] . \alpha(v)=
\begin{cases}
\min\{1,\lambda/\lVert v\rVert\}, & v\ne0,\\
1, & v=0,
\end{cases}
\qquad C=\operatorname{clip}(g,\lambda),\qquad A=\mathbb{E}[\alpha(g)\mid\mathcal{G}]. α ( v ) = { min { 1 , λ / ∥ v ∥} , 1 , v = 0 , v = 0 , C = clip ( g , λ ) , A = E [ α ( g ) ∣ G ] . Then 0 ≤ A ≤ 1 0\le A\le1 0 ≤ A ≤ 1 and
E [ C ∣ G ] = A μ + ε , ∥ ε ∥ ≤ σ p λ 1 − p . \mathbb{E}[C\mid\mathcal{G}]=A\mu+\varepsilon,\qquad\lVert\varepsilon\rVert\le\sigma^{p}\lambda^{1-p}. E [ C ∣ G ] = A μ + ε , ∥ ε ∥ ≤ σ p λ 1 − p . Moreover, u = C − E [ C ∣ G ] u=C-\mathbb{E}[C\mid\mathcal{G}] u = C − E [ C ∣ G ] satisfies
∥ u ∥ ≤ 2 λ , E [ ∥ u ∥ 2 ∣ G ] ≤ ( 2 λ ) 2 − p σ p . \lVert u\rVert\le2\lambda,\qquad\mathbb{E}[\lVert u\rVert^{2}\mid\mathcal{G}]\le(2\lambda)^{2-p}\sigma^{p}. ∥ u ∥ ≤ 2 λ , E [∥ u ∥ 2 ∣ G ] ≤ ( 2 λ ) 2 − p σ p . In particular, if σ ≤ 1 \sigma\le1 σ ≤ 1 and λ ≥ 4 \lambda\ge4 λ ≥ 4 , then d = C − A μ = ε + u d=C-A\mu=\varepsilon+u d = C − A μ = ε + u satisfies
∥ d ∥ ≤ 3 λ , E [ ∥ d ∥ 2 ∣ G ] ≤ 4 λ 2 − p . \lVert d\rVert\le3\lambda,\qquad\mathbb{E}[\lVert d\rVert^{2}\mid\mathcal{G}]\le4\lambda^{2-p}. ∥ d ∥ ≤ 3 λ , E [∥ d ∥ 2 ∣ G ] ≤ 4 λ 2 − p .
Proof. The scalar function r ↦ min { 1 , λ / r } r\mapsto\min\{1,\lambda/r\} r ↦ min { 1 , λ / r } , extended by 1 1 1 at zero, is 1 / λ 1/\lambda 1/ λ -Lipschitz on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) and takes values in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Writing ξ = g − μ \xi=g-\mu ξ = g − μ and using its conditional mean zero gives
ε = E [ α ( g ) ξ ∣ G ] = E [ ( α ( g ) − α ( μ ) ) ξ ∣ G ] , \varepsilon=\mathbb{E}[\alpha(g)\xi\mid\mathcal{G}]
=\mathbb{E}[(\alpha(g)-\alpha(\mu))\xi\mid\mathcal{G}], ε = E [ α ( g ) ξ ∣ G ] = E [( α ( g ) − α ( μ )) ξ ∣ G ] ,
∥ ε ∥ ≤ E [ min { ∥ ξ ∥ / λ , 1 } ∥ ξ ∥ ∣ G ] ≤ λ 1 − p E [ ∥ ξ ∥ p ∣ G ] . \lVert\varepsilon\rVert\le\mathbb{E}[\min\{\lVert\xi\rVert/\lambda,1\}\lVert\xi\rVert\mid\mathcal{G}]
\le\lambda^{1-p}\mathbb{E}[\lVert\xi\rVert^{p}\mid\mathcal{G}]. ∥ ε ∥ ≤ E [ min {∥ ξ ∥ / λ , 1 } ∥ ξ ∥ ∣ G ] ≤ λ 1 − p E [∥ ξ ∥ p ∣ G ] .
The final pointwise inequality follows separately for ∥ ξ ∥ ≤ λ \lVert\xi\rVert\le\lambda ∥ ξ ∥ ≤ λ and ∥ ξ ∥ > λ \lVert\xi\rVert>\lambda ∥ ξ ∥ > λ , using p ≤ 2 p\le2 p ≤ 2 and p > 1 p>1 p > 1 , respectively. For the projection P = clip ( ⋅ , λ ) P=\operatorname{clip}(\mathord{\cdot},\lambda) P = clip ( ⋅ , λ ) , variance reduction and nonexpansiveness give
E [ ∥ u ∥ 2 ∣ G ] ≤ E [ ∥ P g − P μ ∥ 2 ∣ G ] ≤ E [ min { ∥ ξ ∥ , 2 λ } 2 ∣ G ] ≤ ( 2 λ ) 2 − p σ p . \mathbb{E}[\lVert u\rVert^{2}\mid\mathcal{G}]
\le\mathbb{E}[\lVert Pg-P\mu\rVert^{2}\mid\mathcal{G}]
\le\mathbb{E}[\min\{\lVert\xi\rVert,2\lambda\}^{2}\mid\mathcal{G}]
\le(2\lambda)^{2-p}\sigma^{p}. E [∥ u ∥ 2 ∣ G ] ≤ E [∥ P g − P μ ∥ 2 ∣ G ] ≤ E [ min {∥ ξ ∥ , 2 λ } 2 ∣ G ] ≤ ( 2 λ ) 2 − p σ p .
Both C C C and its mean belong to the radius-λ \lambda λ ball, proving the range. Under the last assumptions, ∥ ε ∥ ≤ λ 1 − p ≤ λ \lVert\varepsilon\rVert\le\lambda^{1-p}\le\lambda ∥ ε ∥ ≤ λ 1 − p ≤ λ , so ∥ d ∥ ≤ 3 λ \lVert d\rVert\le3\lambda ∥ d ∥ ≤ 3 λ ; and
E [ ∥ d ∥ 2 ∣ G ] = E [ ∥ u ∥ 2 ∣ G ] + ∥ ε ∥ 2 ≤ 2 λ 2 − p + λ 2 − 2 p ≤ 4 λ 2 − p . \mathbb{E}[\lVert d\rVert^{2}\mid\mathcal{G}]
=\mathbb{E}[\lVert u\rVert^{2}\mid\mathcal{G}]+\lVert\varepsilon\rVert^{2}
\le2\lambda^{2-p}+\lambda^{2-2p}\le4\lambda^{2-p}. E [∥ d ∥ 2 ∣ G ] = E [∥ u ∥ 2 ∣ G ] + ∥ ε ∥ 2 ≤ 2 λ 2 − p + λ 2 − 2 p ≤ 4 λ 2 − p .
Elementary Maximal Concentration Bounds
We record a standard scalar concentration bound and one energy estimate used throughout the high-probability proofs. Both use only predictable local bounds.
Lemma 14 (Maximal inequality for bounded martingale increments). Let X t X_t X t be adapted to ( F t ) t = 0 T (\mathcal{F}_t)_{t=0}^{T} ( F t ) t = 0 T , with E t − 1 X t = 0 \mathbb{E}_{t-1}X_t=0 E t − 1 X t = 0 , ∣ X t ∣ ≤ b \lvert X_t\rvert\le b ∣ X t ∣ ≤ b almost surely, and
∑ t = 1 T E t − 1 X t 2 ≤ v almost surely , \sum_{t=1}^{T}\mathbb{E}_{t-1}X_t^{2}\le v\quad\text{almost surely}, t = 1 ∑ T E t − 1 X t 2 ≤ v almost surely , where b , v ≥ 0 b,v\ge0 b , v ≥ 0 are deterministic. For any ℓ > 0 \ell>0 ℓ > 0 ,
P { max 1 ≤ m ≤ T ∑ t = 1 m X t > 2 v ℓ + 2 b ℓ } ≤ e − ℓ . \mathbb{P}\left\{\max_{1\le m\le T}\sum_{t=1}^{m}X_t>2\sqrt{v\ell}+2b\ell\right\}\le e^{-\ell}. P { 1 ≤ m ≤ T max t = 1 ∑ m X t > 2 v ℓ + 2 b ℓ } ≤ e − ℓ .
Proof. The maximal form of Freedman’s inequality [19 Theorem 1.6] , after scaling the increments, gives
P { max m ≤ T ∑ t ≤ m X t > s } ≤ exp { − s 2 2 ( v + b s ) } . \mathbb{P}\left\{\max_{m\leq T}\sum_{t\leq m}X_t>s\right\}\leq\exp\left\{-\frac{s^2}{2(v+bs)}\right\}. P { m ≤ T max t ≤ m ∑ X t > s } ≤ exp { − 2 ( v + b s ) s 2 } .
Set s = 2 v ℓ + 2 b ℓ s=2\sqrt{v\ell}+2b\ell s = 2 v ℓ + 2 b ℓ ; then s 2 ≥ 2 ℓ ( v + b s ) s^2\geq2\ell(v+bs) s 2 ≥ 2 ℓ ( v + b s ) . Zero budgets follow by continuity. ■
Lemma 15 (A common energy bound). Let e t e_t e t be integrable and adapted, P 0 , … , P N ≥ 0 P_0,\ldots,P_N\geq0 P 0 , … , P N ≥ 0 adapted, J t ∈ { 0 , 1 } J_t\in\{0,1\} J t ∈ { 0 , 1 } predictable and nonincreasing, and E \mathcal{E} E an event, for instance E = Ω \mathcal{E}=\Omega E = Ω . On E ∩ { J m = 1 } \mathcal{E}\cap\{J_m=1\} E ∩ { J m = 1 } , suppose
P m ≤ P 0 + ∑ t ≤ m { a t ∥ e t ∥ 2 + ⟨ e t , w t ⟩ } . P_m\leq P_0+\sum_{t\leq m}\{a_t\lVert e_t\rVert^2+\langle e_t,w_t\rangle\}. P m ≤ P 0 + t ≤ m ∑ { a t ∥ e t ∥ 2 + ⟨ e t , w t ⟩} . where w t w_t w t is predictable with ∥ w t ∥ ≤ s t P t − 1 \lVert w_t\rVert\leq s_t\sqrt{P_{t-1}} ∥ w t ∥ ≤ s t P t − 1 on J t = 1 J_t=1 J t = 1 , and a t , s t ≥ 0 a_t,s_t\geq0 a t , s t ≥ 0 are deterministic. On J t = 1 J_t=1 J t = 1 , assume ∥ E t − 1 e t ∥ ≤ B t \lVert\mathbb{E}_{t-1}e_t\rVert\leq B_t ∥ E t − 1 e t ∥ ≤ B t , E t − 1 ∥ e t ∥ 2 ≤ V t \mathbb{E}_{t-1}\lVert e_t\rVert^2\leq V_t E t − 1 ∥ e t ∥ 2 ≤ V t , ∥ e t ∥ ≤ M t \lVert e_t\rVert\leq M_t ∥ e t ∥ ≤ M t , and ∥ e t − E t − 1 e t ∥ ≤ U t \lVert e_t-\mathbb{E}_{t-1}e_t\rVert\leq U_t ∥ e t − E t − 1 e t ∥ ≤ U t , with deterministic bounds. Put
A = ∑ t a t V t , K 0 = sup t a t M t 2 , B 0 = ∑ t s t B t , V 0 = ∑ t s t 2 V t , J 0 = sup t s t U t . A=\sum_t a_tV_t,\qquad K_0=\sup_t a_tM_t^2,\qquad B_0=\sum_t s_tB_t,\qquad V_0=\sum_t s_t^2V_t,\qquad J_0=\sup_t s_tU_t. A = t ∑ a t V t , K 0 = t sup a t M t 2 , B 0 = t ∑ s t B t , V 0 = t ∑ s t 2 V t , J 0 = t sup s t U t . For p 0 ≥ 0 p_0\geq0 p 0 ≥ 0 and q > 0 q>0 q > 0 , define
C = p 0 + A + 2 K 0 A q + 2 K 0 q , D = B 0 + 2 V 0 q + 2 J 0 q . C=p_0+A+2\sqrt{K_0Aq}+2K_0q,\qquad D=B_0+2\sqrt{V_0q}+2J_0q. C = p 0 + A + 2 K 0 A q + 2 K 0 q , D = B 0 + 2 V 0 q + 2 J 0 q . Except on probability e − q e^{-q} e − q , on E \mathcal{E} E , simultaneously for every m m m with J m = 1 J_m=1 J m = 1 and P 0 ≤ p 0 P_0\leq p_0 P 0 ≤ p 0 ,
P m ≤ ( C + D ) 2 . P_m\leq(\sqrt{C}+D)^2. P m ≤ ( C + D ) 2 . The initial state may be random; the conclusion extends to N = ∞ N=\infty N = ∞ if the budgets are finite.
Proof. Set H = ( C + D ) 2 H=(\sqrt{C}+D)^2 H = ( C + D ) 2 and I t = J t 1 { P 0 ≤ p 0 , max j < t P j ≤ H } I_t=J_t\mathbf{1}_{\{P_0\leq p_0,\ \max_{j<t}P_j\leq H\}} I t = J t 1 { P 0 ≤ p 0 , m a x j < t P j ≤ H } , a predictable indicator that does not involve E \mathcal{E} E . Center only stopped quantities: Y t = I t ∥ e t ∥ 2 Y_t=I_t\lVert e_t\rVert^2 Y t = I t ∥ e t ∥ 2 , m t = E t − 1 Y t m_t=\mathbb{E}_{t-1}Y_t m t = E t − 1 Y t , and U t ′ = I t e t − E t − 1 ( I t e t ) U'_t=I_te_t-\mathbb{E}_{t-1}(I_te_t) U t ′ = I t e t − E t − 1 ( I t e t ) . The martingale increments X t = a t ( Y t − m t ) + ⟨ U t ′ , w t ⟩ X_t=a_t(Y_t-m_t)+\langle U'_t,w_t\rangle X t = a t ( Y t − m t ) + ⟨ U t ′ , w t ⟩ satisfy
∣ X t ∣ ≤ K 0 + H J 0 , ∑ t E t − 1 X t 2 ≤ K 0 A + H V 0 . \lvert X_t\rvert\leq K_0+\sqrt{H}J_0,\qquad\sqrt{\sum_t\mathbb{E}_{t-1}X_t^2}\leq\sqrt{K_0A}+\sqrt{HV_0}. ∣ X t ∣ ≤ K 0 + H J 0 , t ∑ E t − 1 X t 2 ≤ K 0 A + H V 0 . Indeed, conditional variance reduction and the conditional L 2 L^2 L 2 triangle inequality give E t − 1 X t 2 ≤ I t V t ( a t M t + s t H ) \sqrt{\mathbb{E}_{t-1}X_t^2}\leq I_t\sqrt{V_t}(a_tM_t+s_t\sqrt{H}) E t − 1 X t 2 ≤ I t V t ( a t M t + s t H ) ; then use the triangle inequality for the deterministic sequence of bounds. By Lemma 14, except on probability e − q e^{-q} e − q , every partial sum of the X t X_t X t is at most 2 K 0 A q + 2 K 0 q + H ( 2 V 0 q + 2 J 0 q ) 2\sqrt{K_0Aq}+2K_0q+\sqrt{H}(2\sqrt{V_0q}+2J_0q) 2 K 0 A q + 2 K 0 q + H ( 2 V 0 q + 2 J 0 q ) . On this event and on P 0 ≤ p 0 P_0\leq p_0 P 0 ≤ p 0 , the mean and bias budgets give, for every m m m ,
P 0 + ∑ t ≤ m I t { a t ∥ e t ∥ 2 + ⟨ e t , w t ⟩ } ≤ C + H D ≤ H , P_0+\sum_{t\leq m}I_t\{a_t\lVert e_t\rVert^2+\langle e_t,w_t\rangle\}\leq C+\sqrt{H}D\leq H, P 0 + t ≤ m ∑ I t { a t ∥ e t ∥ 2 + ⟨ e t , w t ⟩} ≤ C + H D ≤ H , since H − C − H D = C D H-C-\sqrt{H}D=\sqrt{C}D H − C − H D = C D . Suppose that, on this event and on E \mathcal{E} E , some m m m with J m = 1 J_m=1 J m = 1 and P 0 ≤ p 0 P_0\leq p_0 P 0 ≤ p 0 has P m > H P_m>H P m > H , and take the first such m m m . Then I t = 1 I_t=1 I t = 1 for every t ≤ m t\leq m t ≤ m , so the assumed recursion bounds P m P_m P m by the left side above, a contradiction. Taking the union over finite prefixes yields the claim for N = ∞ N=\infty N = ∞ . ■
Clipped versus Unclipped SGD
Remark 16 (Clipping and the cube-root barrier). In concurrent and independent work, Anonymous [1 ] prove that at p = 2 p = 2 p = 2 , for every positive deterministic schedule of unclipped step sizes and every r > 1 / 3 r > 1/3 r > 1/3 , some finite-variance instance makes ( t + 1 ) r ( f ( x t ) − f ⋆ ) (t + 1)^r(f(x_t) - f^\star) ( t + 1 ) r ( f ( x t ) − f ⋆ ) unbounded almost surely. We first check that their theorem also bounds the time-uniform exponent of unclipped SGD on our class, and then compare it with clipping. Fix any positive deterministic schedule of unclipped step sizes, r ′ > 1 / 3 r' > 1/3 r ′ > 1/3 , and finite C > 0 C > 0 C > 0 , m ≥ 0 m \ge0 m ≥ 0 . Choose 1 / 3 < r < r ′ 1/3 < r < r' 1/3 < r < r ′ . Their theorem supplies a fixed one-dimensional instance with L = 1 L = 1 L = 1 , initial distance 1 / 2 1/2 1/2 , and an i.i.d. unbiased oracle with variance at most one, on which lim sup t → ∞ ( t + 1 ) r ( f ( x t ) − f ⋆ ) = ∞ \limsup_{t \to\infty}(t + 1)^r(f(x_t) - f^\star) = \infty lim sup t → ∞ ( t + 1 ) r ( f ( x t ) − f ⋆ ) = ∞ almost surely. Their construction also satisfies our conditions on the sample losses: the shock branch has sample loss x 2 / 2 + ξ x x^2/2 + \xi x x 2 /2 + ξ x , whose minimum is attained at − ξ -\xi − ξ , while the noiseless branch uses a convex smooth function with minimum at zero. Translating their initial point x 0 = 1 / 2 x_0 = 1/2 x 0 = 1/2 to our x 1 = 0 x_1 = 0 x 1 = 0 and shifting the time index preserves these properties, so both branches belong to our normalized p = 2 p = 2 p = 2 class. On the event
f ( x t + 1 ) − f ⋆ ≤ C t − r ′ log m ( e t ) for all t ≥ 1 , f(x_{t+1}) - f^\star\le Ct^{-r'}\log^m(et) \qquad\text{for all } t \ge1, f ( x t + 1 ) − f ⋆ ≤ C t − r ′ log m ( e t ) for all t ≥ 1 ,
the gap multiplied by t r t^r t r tends to zero, since t r − r ′ log m ( e t ) → 0 t^{r-r'}\log^m(et) \to0 t r − r ′ log m ( e t ) → 0 . The event therefore has probability zero. Thus, by their theorem, deterministic unclipped step sizes attain no time-uniform exponent above 1 / 3 1/3 1/3 on our class. In contrast, Theorem 2 attains β 2 = 2 / 5 \beta_2 = 2/5 β 2 = 2/5 with clipping, and Theorem 4 excludes every larger exponent for all gradient-only rules (Definition 3 ), including adaptive step sizes and clipping thresholds. Since clipping is a scalar response to the observed stochastic gradient, this resolves the time-uniform counterpart (at a fixed confidence level) of their question regarding adaptive step sizes: the exponent 2 / 5 2/5 2/5 is both attainable and optimal on our finite-variance class. Moreover, the almost sure consequence of Theorem 2 gives f ( x t + 1 ) − f ⋆ = O ( t − 2 / 5 log 4 t ) f(x_{t+1}) - f^\star= O(t^{-2/5}\log^4 t) f ( x t + 1 ) − f ⋆ = O ( t − 2/5 log 4 t ) almost surely on every instance of our class, including theirs, so clipping also exceeds the almost sure exponent 1 / 3 1/3 1/3 of unclipped deterministic schedules.
Proofs for the Time-Uniform Lower Bound and Fixed-Time Guarantees
This appendix proves Proposition 17 and Theorems 4 and 6 , then extends both lower bounds to rules whose record cannot advance on zero feedback (Proposition 21 ). Theorem 5 is proved in Appendix § 18 .§ 7 . For the lower bounds we fix p ∈ ( 1 , 2 ] p \in(1, 2] p ∈ ( 1 , 2 ] , L = R = σ = 1 L = R = \sigma= 1 L = R = σ = 1 , and x 1 = 0 x_1 = 0 x 1 = 0 . In every comparison the same rule is used on every instance.
Section 4.2 outlines the argument. Its core is the following progress bound.
Proposition 17 (Progress bound under time-uniform guarantees). Fix ε ∈ ( 0 , 1 / 2 ) \varepsilon\in(0, 1/2) ε ∈ ( 0 , 1/2 ) , N ≥ 1 N \ge1 N ≥ 1 , a rule of Definition 3 , and positive numbers b 1 , … , b N b_1,\ldots,b_N b 1 , … , b N ; the rule may be tuned to a confidence level different from ε \varepsilon ε . Suppose that every instance in the class of Theorem 4 satisfies
P { f ( x t + 1 ) − f ⋆ ≤ b t for all 1 ≤ t ≤ N } ≥ 1 − ε . (26) \mathbb{P}\{f(x_{t+1}) - f^\star\le b_t \text{ for all } 1 \le t \le N\} \ge1 - \varepsilon. \tag*{(26)} P { f ( x t + 1 ) − f ⋆ ≤ b t for all 1 ≤ t ≤ N } ≥ 1 − ε . (26) Then for every μ ∈ ( 4 b N , 1 / 2 ] \mu\in(4b_N, 1/2] μ ∈ ( 4 b N , 1/2 ] , with A = ( 2 μ ) − 1 / ( p − 1 ) A = (2\mu)^{-1/(p-1)} A = ( 2 μ ) − 1/ ( p − 1 ) and π = μ / A \pi= \mu/A π = μ / A ,
π ∑ t = 1 N ( μ + 2 b t ) ≥ 1 − 2 ε 2 . (27) \pi\sum_{t=1}^{N}(\mu+ \sqrt{2b_t}) \ge\frac{1 - 2\varepsilon}{2}. \tag*{(27)} π t = 1 ∑ N ( μ + 2 b t ) ≥ 2 1 − 2 ε . (27)
Translated Targets with a Common I.I.D. Transcript
For 0 < μ ≤ 1 / 2 0 < \mu\le1/2 0 < μ ≤ 1/2 and y ∈ [ μ , 1 ] y \in[\mu,1] y ∈ [ μ , 1 ] , define
f μ , y ( x ) = 1 2 ( x − y + μ ) + 2 − μ x , f μ , y ⋆ = − μ y + μ 2 / 2. f_{\mu,y}(x)=\frac{1}{2}(x-y+\mu)_{+}^{2}-\mu x,\qquad f_{\mu,y}^{\star}=-\mu y+\mu^{2}/2. f μ , y ( x ) = 2 1 ( x − y + μ ) + 2 − μx , f μ , y ⋆ = − μ y + μ 2 /2.
Its derivative is − μ -\mu − μ on x ≤ y − μ x \le y-\mu x ≤ y − μ and x − y x-y x − y on x ≥ y − μ x \ge y-\mu x ≥ y − μ . These branches agree at the transition. Thus f μ , y f_{\mu,y} f μ , y is convex and globally 1-smooth, with unique minimizer y y y and initial distance at most one. Its gap is
f μ , y ( x ) − f μ , y ⋆ = { μ ( y − x ) − μ 2 / 2 , x ≤ y − μ , ( x − y ) 2 / 2 , x ≥ y − μ . (28) f_{\mu,y}(x)-f_{\mu,y}^{\star}=
\begin{cases}
\mu(y-x)-\mu^{2}/2, & x\le y-\mu,\\
(x-y)^{2}/2, & x\ge y-\mu.
\end{cases} \tag*{(28)} f μ , y ( x ) − f μ , y ⋆ = { μ ( y − x ) − μ 2 /2 , ( x − y ) 2 /2 , x ≤ y − μ , x ≥ y − μ . (28)
This objective is a reflected and translated one-sided Huber loss [26 ] , following the construction of Kornowski and Shamir [30 ] through Theorem 2 and Section 3.2; Tsai et al. [67 ] use a related overshoot at an arbitrary iteration through Lemma 3.1. Those lower bounds concern exact gradients and prescribed step sizes. Against adaptive rules with i.i.d. noise, choosing the translation after observing a random trajectory would select the instance from the realized path, so we fix the translated family before the run and integrate, over the target location, the probabilities of failing at the first crossing. This accommodates adaptive step sizes and algorithmic randomness.
Set
A = ( 2 μ ) − 1 / ( p − 1 ) ≥ 1 , π = μ / A = 1 / ( 2 A p ) , Y t i.i.d. Bernoulli ( π ) . (29) A=(2\mu)^{-1/(p-1)}\ge1,\qquad\pi=\mu/A=1/(2A^{p}),\qquad Y_t\text{ i.i.d. Bernoulli}(\pi). \tag*{(29)} A = ( 2 μ ) − 1/ ( p − 1 ) ≥ 1 , π = μ / A = 1/ ( 2 A p ) , Y t i.i.d. Bernoulli ( π ) . (29)
Independently of the algorithm’s seed, each call draws a fresh Y t Y_t Y t . Use the sample loss
F μ , y ( x ; Y ) = 1 2 ( x − y + μ ) + 2 − A Y x = f μ , y ( x ) + ( μ − A Y ) x . (30) F_{\mu,y}(x;Y)=\frac{1}{2}(x-y+\mu)_{+}^{2}-AYx=f_{\mu,y}(x)+(\mu-AY)x. \tag*{(30)} F μ , y ( x ; Y ) = 2 1 ( x − y + μ ) + 2 − A Y x = f μ , y ( x ) + ( μ − A Y ) x . (30)
The additive noise ζ = μ − A Y \zeta=\mu-AY ζ = μ − A Y is centered, independent of the query and time, and has the same law for every y y y . Since μ ≤ A \mu\le A μ ≤ A ,
E ∣ ζ ∣ p = ( 1 − π ) μ p + π ( A − μ ) p ≤ μ p + μ A p − 1 ≤ 2 μ A p − 1 = 1. \mathbb{E}|\zeta|^{p}=(1-\pi)\mu^{p}+\pi(A-\mu)^{p}\le\mu^{p}+\mu A^{p-1}\le2\mu A^{p-1}=1. E ∣ ζ ∣ p = ( 1 − π ) μ p + π ( A − μ ) p ≤ μ p + μ A p − 1 ≤ 2 μ A p − 1 = 1.
Every sample loss is convex and globally 1-smooth, bounded below, and attains its minimum: for Y = 0 Y=0 Y = 0 , every x ≤ y − μ x\le y-\mu x ≤ y − μ is a minimizer; for Y = 1 Y=1 Y = 1 , the minimizer is y − μ + A y-\mu+A y − μ + A . On the common left branch the raw gradient is exactly
g μ , y ( x ; Y ) = − A Y ( x ≤ y − μ ) . g_{\mu,y}(x;Y)=-AY\qquad(x\le y-\mu). g μ , y ( x ; Y ) = − A Y ( x ≤ y − μ ) .
The algorithm observes only these gradients and does not know y y y or f ⋆ f^{\star} f ⋆ .
Proof of Proposition 17
We first isolate two lemmas used both here and for the momentum lower bound in Appendix § 19 : a crossing bound for monotone processes, and its application to the translated targets through a coupling at the first crossing.
Lemma 18 (Crossing bound for monotone processes). Let N ≥ 1 N\ge1 N ≥ 1 be an integer, let X 1 = 0 ≤ ⋯ ≤ X N + 1 X_1=0\le\cdots\le X_{N+1} X 1 = 0 ≤ ⋯ ≤ X N + 1 be finite random variables, and put D t = X t + 1 − X t ≥ 0 D_t=X_{t+1}-X_t\ge0 D t = X t + 1 − X t ≥ 0 . For deterministic finite c > 0 c>0 c > 0 , s 1 , … , s N ≥ 0 s_1,\ldots,s_N\ge0 s 1 , … , s N ≥ 0 , and 0 ≤ ε < 1 / 2 0\le\varepsilon<1/2 0 ≤ ε < 1/2 , suppose P ( X N + 1 > c ) ≥ 1 − ε \mathbb{P}(X_{N+1}>c)\ge1-\varepsilon P ( X N + 1 > c ) ≥ 1 − ε and, for each fixed z ∈ [ 0 , c ) z\in[0,c) z ∈ [ 0 , c ) ,
P { ∃ t ≤ N : X t ≤ z < X t + 1 , X t + 1 > z + s t } ≤ ε . \mathbb{P}\{\exists t\le N:X_t\le z<X_{t+1},\ X_{t+1}>z+s_t\}\le\varepsilon. P { ∃ t ≤ N : X t ≤ z < X t + 1 , X t + 1 > z + s t } ≤ ε . Then c ( 1 − 2 ε ) ≤ E ∑ t = 1 N min { s t , D t } c(1-2\varepsilon)\le\mathbb{E}\sum_{t=1}^{N}\min\{s_t,D_t\} c ( 1 − 2 ε ) ≤ E ∑ t = 1 N min { s t , D t } .
Proof. Let B = [ 0 , c ) ∩ ⋃ t = 1 N [ X t , X t + 1 − s t ) B=[0,c)\cap\bigcup_{t=1}^{N}[X_t,X_{t+1}-s_t) B = [ 0 , c ) ∩ ⋃ t = 1 N [ X t , X t + 1 − s t ) be the badly crossed boundaries, with reversed intervals empty. Measurable endpoints make 1 { z ∈ B } \mathbf{1}_{\{z\in B\}} 1 { z ∈ B } jointly measurable. The fixed-z z z hypothesis and Tonelli’s theorem give E ∣ B ∣ = ∫ 0 c P ( z ∈ B ) d z ≤ c ε \mathbb{E}|B|=\int_0^c\mathbb{P}(z\in B)\,dz\le c\varepsilon E ∣ B ∣ = ∫ 0 c P ( z ∈ B ) d z ≤ c ε , where ∣ B ∣ |B| ∣ B ∣ denotes Lebesgue measure. At update t t t , the safely crossed boundaries have total length at most min { s t , D t } \min\{s_t,D_t\} min { s t , D t } . On { X N + 1 > c } \{X_{N+1}>c\} { X N + 1 > c } every boundary in [ 0 , c ) [0,c) [ 0 , c ) has been crossed, so
c 1 { X N + 1 > c } ≤ ∣ B ∣ + ∑ t = 1 N min { s t , D t } . c\mathbf{1}_{\{X_{N+1}>c\}}\le|B|+\sum_{t=1}^{N}\min\{s_t,D_t\}. c 1 { X N + 1 > c } ≤ ∣ B ∣ + t = 1 ∑ N min { s t , D t } .
Taking expectations proves the claim. ■
Lemma 19. (Fixed targets coupled at the first crossing). *Fix 0 < ε < 1 / 2 0<\varepsilon<1/2 0 < ε < 1/2 , an integer N ≥ 1 N\ge1 N ≥ 1 , positive numbers b 1 , … , b N b_1,\ldots,b_N b 1 , … , b N , and μ ∈ ( 4 b N , 1 / 2 ] \mu\in(4b_N,1/2] μ ∈ ( 4 b N , 1/2 ] . Let A A A , π \pi π , and Y t Y_t Y t be as in (29) , and put c = 1 / 2 c=1/2 c = 1/2 and s t = μ + 2 b t s_t=\mu+\sqrt{2b_t} s t = μ + 2 b t . Consider a method that starts at x 1 = 0 x_1=0 x 1 = 0 , queries one stochastic gradient at its current iterate x t x_t x t , and computes x t + 1 ∈ R x_{t+1}\in\mathbb{R} x t + 1 ∈ R , together with any auxiliary state, from the gradients observed so far and an independent seed, by the same rule on every instance. Suppose every instance f μ , y f_{\mu,y} f μ , y , y ∈ [ μ , 1 ] y\in[\mu,1] y ∈ [ μ , 1 ] , with sample loss (30) satisfies (26) . Let the reference run X t X_t X t be the states produced from the observations − A Y t -AY_t − A Y t and the seed through the first update with X t + 1 > c X_{t+1}>c X t + 1 > c , held fixed thereafter without further oracle calls. Define M t = max { 0 , X 1 , … , X t } M_t=\max\{0,X_1,\ldots,X_t\} M t = max { 0 , X 1 , … , X t } and D t = M t + 1 − M t D_t=M_{t+1}-M_t D t = M t + 1 − M t . Then
c ( 1 − 2 ε ) ≤ E ∑ t = 1 N min { s t , D t } . c(1-2\varepsilon)\le\mathbb{E}\sum_{t=1}^{N}\min\{s_t,D_t\}. c ( 1 − 2 ε ) ≤ E t = 1 ∑ N min { s t , D t } .
Proof. Since c ≤ 1 − μ c\le1-\mu c ≤ 1 − μ , every query before stopping lies where target y = 1 y=1 y = 1 returns − A Y t -AY_t − A Y t , so the reference agrees with the run on that target until it stops. On { M N + 1 ≤ c } \{M_{N+1}\le c\} { M N + 1 ≤ c } it has not stopped by time N N N , so X N + 1 ≤ c X_{N+1}\le c X N + 1 ≤ c , and (28) gives
f μ , 1 ( X N + 1 ) − f μ , 1 ⋆ ≥ μ ( 1 − c ) − μ 2 / 2 ≥ μ / 4 > b N . f_{\mu,1}(X_{N+1})-f_{\mu,1}^{\star}\ge\mu(1-c)-\mu^2/2\ge\mu/4>b_N. f μ , 1 ( X N + 1 ) − f μ , 1 ⋆ ≥ μ ( 1 − c ) − μ 2 /2 ≥ μ /4 > b N .
Hence P ( M N + 1 > c ) ≥ 1 − ε \mathbb{P}(M_{N+1}>c)\ge1-\varepsilon P ( M N + 1 > c ) ≥ 1 − ε .
For any fixed z ∈ [ 0 , c ) z\in[0,c) z ∈ [ 0 , c ) , couple target y = z + μ y=z+\mu y = z + μ using the same Bernoulli sequence and the same seed. Its run agrees with the reference in all states, including auxiliary ones, through the first update strictly crossing z z z : all queries up to that update lie at or below z = y − μ z=y-\mu z = y − μ and have the same observations − A Y t -AY_t − A Y t . Each target is coupled separately, before the run. On { M t ≤ z < M t + 1 } \{M_t\le z<M_{t+1}\} { M t ≤ z < M t + 1 } this is the first crossing and M t + 1 = X t + 1 M_{t+1}=X_{t+1} M t + 1 = X t + 1 . If also M t + 1 > z + s t M_{t+1}>z+s_t M t + 1 > z + s t , the coupled target has gap ( X t + 1 − z − μ ) 2 / 2 > b t (X_{t+1}-z-\mu)^2/2>b_t ( X t + 1 − z − μ ) 2 /2 > b t . Its prefix guarantee verifies the crossing hypothesis of Lemma 18 for the nondecreasing process M t M_t M t , for each fixed z z z . Applying that lemma proves the claim. When X t X_t X t is nondecreasing, M t = X t M_t=X_t M t = X t and D t = X t + 1 − X t D_t=X_{t+1}-X_t D t = X t + 1 − X t . ■
Proof of Proposition 17. Let ε \varepsilon ε , N N N , the rule, and b 1 , … , b N b_1,\ldots,b_N b 1 , … , b N be as in its statement, and fix μ ∈ ( 4 b N , 1 / 2 ] \mu\in(4b_N,1/2] μ ∈ ( 4 b N , 1/2 ] . The instances of Appendix H.1 belong to the class of Theorem 4, and the rule meets the conditions of Lemma 19. Let X t X_t X t be its reference run, with c = 1 / 2 c=1/2 c = 1/2 and s t = μ + 2 b t s_t=\mu+\sqrt{2b_t} s t = μ + 2 b t . All observations prior to stopping equal − A Y t -AY_t − A Y t , so
X 1 = 0 , X t + 1 = X t + Y t q t ≥ X t ≥ 0 , X_1=0,\qquad X_{t+1}=X_t+Y_tq_t\ge X_t\ge0, X 1 = 0 , X t + 1 = X t + Y t q t ≥ X t ≥ 0 ,
where q t = A c t ≥ 0 q_t=Ac_t\ge0 q t = A c t ≥ 0 before stopping and q t = 0 q_t=0 q t = 0 afterwards. Here M t = X t M_t=X_t M t = X t , so the record increments equal the state increments. The responses are finite but need not be integrable or independent of Y t Y_t Y t . Since D t = Y t q t D_t=Y_tq_t D t = Y t q t , we have min { s t , D t } ≤ Y t s t \min\{s_t,D_t\}\le Y_ts_t min { s t , D t } ≤ Y t s t . Since c = 1 / 2 c=1/2 c = 1/2 , Lemma 19 and E Y t = π \mathbb{E}Y_t=\pi E Y t = π give (27) :
c ( 1 − 2 ε ) ≤ E ∑ t = 1 N min { s t , D t } ≤ π ∑ t = 1 N s t . c(1-2\varepsilon)\le\mathbb{E}\sum_{t=1}^{N}\min\{s_t,D_t\}\le\pi\sum_{t=1}^{N}s_t. c ( 1 − 2 ε ) ≤ E t = 1 ∑ N min { s t , D t } ≤ π t = 1 ∑ N s t .
Proof of the Time-Uniform Lower Bound
We first record the exponent calculation shared by Theorems 4 and 7 ; the proof of Theorem 6 repeats it with a confidence factor. Put q = p / ( p − 1 ) q = p/(p-1) q = p / ( p − 1 ) and choose β p < u < min { r , 1 } \beta_p < u < \min\{r,1\} β p < u < min { r , 1 } . For any positive subpolynomial factors D N , K N = N o ( 1 ) D_N,K_N = N^{o(1)} D N , K N = N o ( 1 ) , let
b N , t = D N t − u , μ = 8 D N N − u , π = 2 1 / ( p − 1 ) μ q . (31) b_{N,t}=D_Nt^{-u}, \qquad\mu=8D_NN^{-u}, \qquad\pi=2^{1/(p-1)}\mu^q. \tag*{(31)} b N , t = D N t − u , μ = 8 D N N − u , π = 2 1/ ( p − 1 ) μ q . (31)
Then μ / 2 b N , N = 4 2 D N N − u / 2 → 0 \mu/\sqrt{2b_{N,N}}=4\sqrt{2}\sqrt{D_N}N^{-u/2}\to0 μ / 2 b N , N = 4 2 D N N − u /2 → 0 . Since b N , t ≥ b N , N b_{N,t}\ge b_{N,N} b N , t ≥ b N , N , u < 1 u<1 u < 1 , and β p = 1 / ( q + 1 / 2 ) \beta_p=1/(q+1/2) β p = 1/ ( q + 1/2 ) , summing t − u / 2 t^{-u/2} t − u /2 gives
K N π ∑ t = 1 N ( μ + 2 b N , t ) = O ( K N D N q + 1 / 2 N 1 − u ( q + 1 / 2 ) ) ⟶ 0. (32) K_N\pi\sum_{t=1}^{N}\left(\mu+\sqrt{2b_{N,t}}\right)=O\left(K_ND_N^{q+1/2}N^{1-u(q+1/2)}\right)\longrightarrow0. \tag*{(32)} K N π t = 1 ∑ N ( μ + 2 b N , t ) = O ( K N D N q + 1/2 N 1 − u ( q + 1/2 ) ) ⟶ 0. (32)
Proof of Theorem 4 . Suppose its envelope holds on every instance. Choose u u u as above and D N = C log m ( e N ) D_N=C\log^m(eN) D N = C log m ( e N ) , so C t − r log m ( e t ) ≤ b N , t Ct^{-r}\log^m(et)\le b_{N,t} C t − r log m ( e t ) ≤ b N , t for t ≤ N t\le N t ≤ N . For all sufficiently large N N N , 4 b N , N < μ ≤ 1 / 2 4b_{N,N}<\mu\le1/2 4 b N , N < μ ≤ 1/2 . Proposition 17 with ε = δ \varepsilon=\delta ε = δ contradicts (32) with K N = 1 K_N=1 K N = 1 . A failing finite prefix yields (9) . The resulting instance is fixed before its i.i.d. samples and the algorithm seed are drawn, and it may depend on N N N . ■
Proof of Theorem 6
Proof Suppose the displayed envelope holds on every instance at every time and confidence. Since c < 1 − β p c<1-\beta_p c < 1 − β p and r > β p + c r>\beta_p+c r > β p + c , we may choose β p + c < u < min { r , 1 } \beta_p+c<u<\min\{r,1\} β p + c < u < min { r , 1 } ; put v = u − c > β p v=u-c>\beta_p v = u − c > β p . For each deterministic N N N , set
δ N = 1 4 N c , D N = C 4 c log m ( 4 e N 2 ) , b N , t = D N N c t − u . \delta_N=\frac{1}{4N^c}, \qquad D_N=C4^c\log^m(4eN^2), \qquad b_{N,t}=D_NN^ct^{-u}. δ N = 4 N c 1 , D N = C 4 c log m ( 4 e N 2 ) , b N , t = D N N c t − u .
Since δ N − c = ( 4 N ) c \delta_N^{-c}=(4N)^c δ N − c = ( 4 N ) c and log ( e t / δ N ) ≤ log ( 4 e N 2 ) \log(et/\delta_N)\le\log(4eN^2) log ( e t / δ N ) ≤ log ( 4 e N 2 ) , the claimed fixed-time envelope is at most b N , t b_{N,t} b N , t for t ≤ N t\le N t ≤ N . Use the δ N \delta_N δ N -tuned rule on the prefix. A union bound gives failure probability at most N δ N = 1 / 4 N\delta_N=1/4 N δ N = 1/4 on every instance. Proposition 17 , with the separate budget ε 0 = 1 / 4 \varepsilon_0=1/4 ε 0 = 1/4 and μ = 8 b N , N = 8 D N N − v \mu=8b_{N,N}=8D_NN^{-v} μ = 8 b N , N = 8 D N N − v , which lies in ( 4 b N , N , 1 / 2 ] (4b_{N,N},1/2] ( 4 b N , N , 1/2 ] for large N N N , therefore requires ∑ t ≤ N ( μ + 2 b N , t ) ≥ 1 / 4 \sum_{t\le N}(\mu+\sqrt{2b_{N,t}})\ge1/4 ∑ t ≤ N ( μ + 2 b N , t ) ≥ 1/4 . With π = 2 1 / ( p − 1 ) μ q \pi=2^{1/(p-1)}\mu^q π = 2 1/ ( p − 1 ) μ q and ∑ t ≤ N t − u / 2 ≤ 2 N 1 − u / 2 \sum_{t\le N}t^{-u/2}\le2N^{1-u/2} ∑ t ≤ N t − u /2 ≤ 2 N 1 − u /2 ,
π ∑ t ≤ N ( μ + 2 b N , t ) ≤ 2 1 / ( p − 1 ) 8 q D N q ( 8 D N N − v / 2 + 2 2 D N ) N 1 − v ( q + 1 / 2 ) . \pi\sum_{t\le N}\left(\mu+\sqrt{2b_{N,t}}\right)\le2^{1/(p-1)}8^qD_N^q\left(8D_NN^{-v/2}+2\sqrt{2D_N}\right)N^{1-v(q+1/2)}. π t ≤ N ∑ ( μ + 2 b N , t ) ≤ 2 1/ ( p − 1 ) 8 q D N q ( 8 D N N − v /2 + 2 2 D N ) N 1 − v ( q + 1/2 ) .
The right side tends to zero, because D N = N o ( 1 ) D_N=N^{o(1)} D N = N o ( 1 ) and v ( q + 1 / 2 ) = v / β p > 1 v(q+1/2)=v/\beta_p>1 v ( q + 1/2 ) = v / β p > 1 ; this is a contradiction. By the union bound, a failure probability exceeding 1 / 4 1/4 1/4 on the prefix 1 ≤ t ≤ N 1\le t\le N 1 ≤ t ≤ N implies a fixed-time failure probability above δ N \delta_N δ N at some deterministic t ≤ N t\le N t ≤ N . ■
Rules Whose Record Does Not Advance on Zero Feedback
We now allow the iterate to move toward a fixed anchor and, more generally, to retreat or recover earlier positions after a zero observation. The quantity used in the crossing argument is the running maximum of the queries in a fixed direction.
Definition 20 (Rules with no record advance on zero feedback). A one-dimensional horizon-free rule starts at x 1 = 0 x_1=0 x 1 = 0 , queries one fresh stochastic gradient g t g_t g t at x t x_t x t , and outputs its updated state x t + 1 x_{t+1} x t + 1 . Its states are finite measurable functions of the gradients observed so far, t t t , p p p , δ \delta δ , the bounds L L L , R R R , σ \sigma σ , and an independent random seed; arbitrary auxiliary memory is allowed. The rule receives
no other information about the objective. There is a direction e ∈ { − 1 , 1 } e \in\{-1,1\} e ∈ { − 1 , 1 } , fixed before the run, such that for every A > 0 A > 0 A > 0 , every finite feedback history with g s ∈ { 0 , − e A } g_s \in\{0,-eA\} g s ∈ { 0 , − e A } , and every seed, the quantities
U t = e ( x t − x 1 ) , M t = max { 0 , U 1 , … , U t } U_t=e(x_t-x_1),\qquad M_t=\max\{0,U_1,\ldots,U_t\} U t = e ( x t − x 1 ) , M t = max { 0 , U 1 , … , U t }
satisfy
g t = 0 ⟹ U t + 1 ≤ M t . (33) g_t=0\quad\Longrightarrow\quad U_{t+1}\le M_t. \tag*{(33)} g t = 0 ⟹ U t + 1 ≤ M t . (33)
The direction may depend on the rule, its deterministic initialization, and its confidence input, but is independent of the unknown objective and the realized seed.
Proposition 21 (Lower bounds for rules with no record advance). Fix p ∈ ( 1 , 2 ] p \in(1,2] p ∈ ( 1 , 2 ] and L = R = σ = 1 L=R=\sigma=1 L = R = σ = 1 .
For every δ ∈ ( 0 , 1 / 2 ) \delta\in(0,1/2) δ ∈ ( 0 , 1/2 ) , every rule of Definition 20 , every C > 0 C>0 C > 0 , m ≥ 0 m\ge0 m ≥ 0 , and r > β p r>\beta_p r > β p , there is an instance for which
P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > C t − r [ log ( e t ) ] m } > δ . \mathbb{P}\{\exists t\ge1:f(x_{t+1})-f^\star>Ct^{-r}[\log(et)]^m\}>\delta. P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > C t − r [ log ( e t ) ] m } > δ . The rule, C C C , and m m m may depend on δ \delta δ and p p p .
For every family ( rule δ ) δ ∈ ( 0 , 1 ) (\operatorname{rule}_\delta)_{\delta\in(0,1)} ( rule δ ) δ ∈ ( 0 , 1 ) of rules of Definition 20 , every C > 0 C>0 C > 0 , m ≥ 0 m\ge0 m ≥ 0 , c ∈ [ 0 , 1 − β p ) c\in[0,1-\beta_p) c ∈ [ 0 , 1 − β p ) , and r > β p + c r>\beta_p+c r > β p + c , there are δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , a deterministic integer t ≥ 1 t\ge1 t ≥ 1 , and an instance for which
P { f ( x t + 1 ) − f ⋆ > C t − r δ − c [ log ( e t / δ ) ] m } > δ . \mathbb{P}\{f(x_{t+1})-f^\star>Ct^{-r}\delta^{-c}[\log(et/\delta)]^m\}>\delta. P { f ( x t + 1 ) − f ⋆ > C t − r δ − c [ log ( e t / δ ) ] m } > δ . Here C C C , c c c , m m m are independent of δ \delta δ , t t t , and the instance.
In both parts the instance satisfies Assumptions 1 and 2, is one-dimensional, and has i.i.d. additive noise and convex, globally 1-smooth sample losses attaining their minima.
Proof. We first establish (27) for the larger class. Fix a rule, N ≥ 1 N\ge1 N ≥ 1 , ε ∈ ( 0 , 1 / 2 ) \varepsilon\in(0,1/2) ε ∈ ( 0 , 1/2 ) , positive b 1 , … , b N b_1,\ldots,b_N b 1 , … , b N , and μ ∈ ( 4 b N , 1 / 2 ] \mu\in(4b_N,1/2] μ ∈ ( 4 b N , 1/2 ] , and suppose its prefix guarantee (26) holds on every instance. Choose its direction e e e and use the reflected sample losses F μ , y ( e x ; Y ) F_{\mu,y}(ex;Y) F μ , y ( e x ; Y ) from (30). Reflection preserves convexity, smoothness, the initial distance, and the central moment bound. In the coordinate u = e x u=ex u = e x , every query with u ≤ y − μ u\le y-\mu u ≤ y − μ receives raw feedback − e A Y t -eAY_t − e A Y t . Thus the same seed and these observations define a reference run U t U_t U t , held fixed after its first crossing of 1 / 2 1/2 1/2 . Lemma 19 , applied in this coordinate with M t = max { 0 , U 1 , … , U t } M_t=\max\{0,U_1,\ldots,U_t\} M t = max { 0 , U 1 , … , U t } and D t = M t + 1 − M t D_t=M_{t+1}-M_t D t = M t + 1 − M t , gives
1 − 2 ε 2 ≤ E ∑ t = 1 N min { s t , D t } , s t = μ + 2 b t . \frac{1-2\varepsilon}{2}\le\mathbb{E}\sum_{t=1}^{N}\min\{s_t,D_t\},\qquad s_t=\mu+\sqrt{2b_t}. 2 1 − 2 ε ≤ E t = 1 ∑ N min { s t , D t } , s t = μ + 2 b t .
Condition (33) implies D t = 0 D_t=0 D t = 0 when Y t = 0 Y_t=0 Y t = 0 ; after stopping D t = 0 D_t=0 D t = 0 as well. Consequently min { s t , D t } ≤ Y t s t \min\{s_t,D_t\}\le Y_ts_t min { s t , D t } ≤ Y t s t , and E Y t = π \mathbb{E}Y_t=\pi E Y t = π proves exactly (27) . The truncated increments are bounded by s t s_t s t , so this step requires no moment bound on the state increments.
For the first part, suppose the asserted envelope holds on every instance. Put q = p / ( p − 1 ) q=p/(p-1) q = p / ( p − 1 ) , choose β p < u < min { r , 1 } \beta_p<u<\min\{r,1\} β p < u < min { r , 1 } , and set
D N = C log m ( e N ) , b N , t = D N t − u , μ = 8 D N N − u . D_N=C\log^m(eN),\qquad b_{N,t}=D_Nt^{-u},\qquad\mu=8D_NN^{-u}. D N = C log m ( e N ) , b N , t = D N t − u , μ = 8 D N N − u .
These b N , t b_{N,t} b N , t dominate the asserted envelope for t ≤ N t\le N t ≤ N , and 4 b N , N < μ ≤ 1 / 2 4b_{N,N}<\mu\le1/2 4 b N , N < μ ≤ 1/2 for all sufficiently large N N N . The budget with ε = δ \varepsilon=\delta ε = δ requires its left side to be at least ( 1 − 2 δ ) / 2 (1-2\delta)/2 ( 1 − 2 δ ) /2 . But π = 2 1 / ( p − 1 ) μ q \pi=2^{1/(p-1)}\mu^q π = 2 1/ ( p − 1 ) μ q and ∑ t ≤ N t − u / 2 ≤ 2 N 1 − u / 2 \sum_{t\le N}t^{-u/2}\le2N^{1-u/2} ∑ t ≤ N t − u /2 ≤ 2 N 1 − u /2 give
π ∑ t = 1 N ( μ + 2 b N , t ) ≤ 2 1 / ( p − 1 ) 8 q D N q ( 8 D N N − u / 2 + 2 2 D N ) N 1 − u ( q + 1 / 2 ) ⟶ 0 , \pi\sum_{t=1}^{N}\left(\mu+\sqrt{2b_{N,t}}\right)\le2^{1/(p-1)}8^qD_N^q\left(8D_NN^{-u/2}+2\sqrt{2D_N}\right)N^{1-u(q+1/2)}\longrightarrow0, π t = 1 ∑ N ( μ + 2 b N , t ) ≤ 2 1/ ( p − 1 ) 8 q D N q ( 8 D N N − u /2 + 2 2 D N ) N 1 − u ( q + 1/2 ) ⟶ 0 ,
since D N = N o ( 1 ) D_N = N^{o(1)} D N = N o ( 1 ) and u ( q + 1 / 2 ) > 1 u(q + 1/2) > 1 u ( q + 1/2 ) > 1 . This contradiction yields a fixed instance with prefix failure probability greater than δ \delta δ .
For the second part, suppose every fixed-time envelope holds on every instance at every confidence level. Choose β p + c < u < min { r , 1 } \beta_p + c < u < \min\{r,1\} β p + c < u < min { r , 1 } and put v = u − c > β p v = u - c > \beta_p v = u − c > β p . For each N N N , use the rule tuned to δ N = 1 / ( 4 N ) \delta_N = 1/(4N) δ N = 1/ ( 4 N ) and set
D N = C 4 q log m ( 4 e N 2 ) , b N , t = D N N c t − u , μ = 8 D N N − v . D_N = C_4^q \log^m(4eN^2), \qquad b_{N,t} = D_N N^{c t-u}, \qquad\mu= 8D_N N^{-v}. D N = C 4 q log m ( 4 e N 2 ) , b N , t = D N N c t − u , μ = 8 D N N − v .
For t ≤ N t \le N t ≤ N , these bounds dominate the claimed envelope at confidence δ N \delta_N δ N . A union bound gives the prefix guarantee with the separate failure budget ϵ = 1 / 4 \epsilon= 1/4 ϵ = 1/4 . The direction e e e is chosen for this rule before the run. For large N N N , the budget therefore requires
1 4 ≤ π ∑ t = 1 N ( μ + 2 b N , t ) ≤ 2 1 / ( p − 1 ) 8 q D N q ( 8 D N N − v / 2 + 2 2 D N ) N 1 − v ( q + 1 / 2 ) ⟶ 0 , \frac{1}{4} \le\pi\sum_{t=1}^{N} \left(\mu+ \sqrt{2b_{N,t}}\right) \le2^{1/(p-1)}8^qD_N^q \left(8D_NN^{-v/2}+2\sqrt{2D_N}\right)N^{1-v(q+1/2)} \longrightarrow0, 4 1 ≤ π t = 1 ∑ N ( μ + 2 b N , t ) ≤ 2 1/ ( p − 1 ) 8 q D N q ( 8 D N N − v /2 + 2 2 D N ) N 1 − v ( q + 1/2 ) ⟶ 0 ,
a contradiction. A prefix failure probability greater than 1 / 4 1/4 1/4 implies a failure probability greater than 1 / ( 4 N ) = δ N 1/(4N) = \delta_N 1/ ( 4 N ) = δ N at some deterministic t ≤ N t \le N t ≤ N , by the union bound. ■ \blacksquare ■
Learning-Rate-Free Methods
We apply Proposition 21 to fixed anchors, coin betting, and the one-dimensional conic algorithms of Grand-Clément et al. [22 ] . We then prove a stronger failure for the standard zero-initialized CBA and CBA+ iterates. Throughout this section the oracle is queried at x t x_t x t and the output is x t + 1 x_{t+1} x t + 1 . We write h t h_t h t for the feedback supplied to an online method, which can be the raw gradient g t g_t g t or its clipped value.
Fixed Anchors and Contracting Weight Decay
Rules of Definition 3 keep x t + 1 = x t x_{t+1} = x_t x t + 1 = x t whenever g t = 0 g_t = 0 g t = 0 , so they satisfy Definition 20 in either direction. More generally, fix a deterministic anchor a ∈ R a \in\mathbb{R} a ∈ R and consider
x t + 1 = ( 1 − ρ t ) x t + ρ t a − c t g t , 0 ≤ ρ t ≤ 1 , 0 ≤ c t < ∞ . (34) x_{t+1} = (1-\rho_t)x_t + \rho_t a - c_t g_t, \qquad0 \le\rho_t \le1,\qquad0 \le c_t < \infty. \tag*{(34)} x t + 1 = ( 1 − ρ t ) x t + ρ t a − c t g t , 0 ≤ ρ t ≤ 1 , 0 ≤ c t < ∞. (34)
The coefficients may depend measurably on the current and past gradients and the seed. Choose e ∈ { − 1 , 1 } e \in\{-1,1\} e ∈ { − 1 , 1 } so that e ( a − x 1 ) ≤ 0 e(a-x_1) \le0 e ( a − x 1 ) ≤ 0 . With U t = e ( x t − x 1 ) U_t = e(x_t-x_1) U t = e ( x t − x 1 ) and M t = max { 0 , U 1 , … , U t } M_t = \max\{0,U_1,\ldots,U_t\} M t = max { 0 , U 1 , … , U t } , a zero observation gives
U t + 1 = ( 1 − ρ t ) U t + ρ t e ( a − x 1 ) ≤ ( 1 − ρ t ) M t + ρ t M t = M t . U_{t+1} = (1-\rho_t)U_t + \rho_t e(a-x_1) \le(1-\rho_t)M_t + \rho_t M_t = M_t. U t + 1 = ( 1 − ρ t ) U t + ρ t e ( a − x 1 ) ≤ ( 1 − ρ t ) M t + ρ t M t = M t .
This proves the record condition for every fixed anchor, regardless of its distance from x 1 x_1 x 1 . When a = x 1 a=x_1 a = x 1 , either direction works. In particular, weight decay toward the initialization,
x t + 1 = x 1 + ( 1 − η t ω t ) ( x t − x 1 ) − η t clip ( g t , λ t ) , η t ≥ 0 , 0 ≤ η t ω t ≤ 1 , x_{t+1} = x_1 + (1-\eta_t\omega_t)(x_t-x_1) - \eta_t\operatorname{clip}(g_t,\lambda_t), \qquad\eta_t \ge0,\qquad0 \le\eta_t\omega_t \le1, x t + 1 = x 1 + ( 1 − η t ω t ) ( x t − x 1 ) − η t clip ( g t , λ t ) , η t ≥ 0 , 0 ≤ η t ω t ≤ 1 ,
has the form (34), with ρ t = η t ω t \rho_t = \eta_t\omega_t ρ t = η t ω t and the clipping multiplier absorbed into c t c_t c t . Thus both lower bounds of Proposition 21 hold for these rules.
One-Dimensional KT Coin Betting
We use the Krichevsky–Trofimov (KT) update of Orabona and Pál [49 ] , with normalized rewards − h t / G -h_t/G − h t / G and initial endowment ϵ / G \epsilon/G ϵ / G . Fix G , ϵ > 0 G,\epsilon> 0 G , ϵ > 0 , let ∣ h t ∣ ≤ G |h_t| \le G ∣ h t ∣ ≤ G , and define
S t − 1 = ∑ s < t h s , W t = ϵ − ∑ s < t h s x s , x t = − W t S t − 1 G 2 t . (35) S_{t-1} = \sum_{s<t} h_s,\qquad W_t = \epsilon- \sum_{s<t} h_s x_s,\qquad x_t = -\frac{W_t S_{t-1}}{G^2t}. \tag*{(35)} S t − 1 = s < t ∑ h s , W t = ϵ − s < t ∑ h s x s , x t = − G 2 t W t S t − 1 . (35)
Here x 1 = 0 x_1 = 0 x 1 = 0 . Since ∣ S t − 1 ∣ ≤ G ( t − 1 ) |S_{t-1}| \le G(t-1) ∣ S t − 1 ∣ ≤ G ( t − 1 ) ,
W t + 1 = W t − h t x t = W t ( 1 + h t S t − 1 G 2 t ) ≥ W t t > 0 , W_{t+1} = W_t - h_t x_t = W_t\left(1+\frac{h_tS_{t-1}}{G^2t}\right) \ge\frac{W_t}{t} > 0, W t + 1 = W t − h t x t = W t ( 1 + G 2 t h t S t − 1 ) ≥ t W t > 0 ,
starting from W 1 = ϵ W_1 = \epsilon W 1 = ϵ . Expanding the next iterate gives
x t + 1 = − W t G 2 ( t + 1 ) ( S t − 1 + h t + h t S t − 1 2 G 2 t + h t 2 S t − 1 G 2 t ) = ( 1 − 1 − h t 2 / G 2 t + 1 ) x t − W t ( 1 + S t − 1 2 / ( G 2 t ) ) G 2 ( t + 1 ) h t . \begin{aligned}
x_{t+1} = -\frac{W_t}{G^2(t+1)}\left(S_{t-1}+h_t+\frac{h_tS_{t-1}^2}{G^2t}+\frac{h_t^2S_{t-1}}{G^2t}\right) \\
&= \left(1-\frac{1-h_t^2/G^2}{t+1}\right)x_t-\frac{W_t(1+S_{t-1}^2/(G^2t))}{G^2(t+1)}h_t.
\end{aligned} x t + 1 = − G 2 ( t + 1 ) W t ( S t − 1 + h t + G 2 t h t S t − 1 2 + G 2 t h t 2 S t − 1 ) = ( 1 − t + 1 1 − h t 2 / G 2 ) x t − G 2 ( t + 1 ) W t ( 1 + S t − 1 2 / ( G 2 t )) h t .
The anchor coefficient lies in [ 0 , 1 / ( t + 1 ) ] [0,1/(t+1)] [ 0 , 1/ ( t + 1 )] and the step factor is positive. For raw gradients of arbitrary magnitude we may use any finite measurable preprocessing
h t = k t g t , k t ≥ 0 , ∣ h t ∣ ≤ G , h_t=k_tg_t,\qquad k_t\ge0,\qquad|h_t|\le G, h t = k t g t , k t ≥ 0 , ∣ h t ∣ ≤ G ,
with h t = 0 h_t=0 h t = 0 when g t = 0 g_t=0 g t = 0 . Examples are h t = clip ( g t , G ) h_t=\operatorname{clip}(g_t,G) h t = clip ( g t , G ) , or h t = clip ( g t , λ t ) / λ t h_t=\operatorname{clip}(g_t,\lambda_t)/\lambda_t h t = clip ( g t , λ t ) / λ t with G = 1 G=1 G = 1 and λ t > 0 \lambda_t>0 λ t > 0 . Multiplying the displayed step factor by k t k_t k t gives (34) with anchor zero, so Proposition 21 applies. In a zero-feedback round, x t + 1 = t x t / ( t + 1 ) x_{t+1}=tx_t/(t+1) x t + 1 = t x t / ( t + 1 ) ; after a nonzero first observation this differs from x t x_t x t . Thus KT belongs to the larger class even though it is not a rule of Definition 3 .
One-Dimensional Conic Algorithms
We first consider Algorithms 1 and 2 of Grand-Clément et al. [22 ] on [ − D , D ] [-D,D] [ − D , D ] , where D > 0 D>0 D > 0 is the decision radius. Their lifted state ( y t , v t ) (y_t,v_t) ( y t , v t ) belongs to
C D = { ( y , v ) : v ≥ 0 , ∣ y ∣ ≤ D v } , x t = { y t / v t , v t > 0 , 0 , ( y t , v t ) = ( 0 , 0 ) . \mathcal{C}_D=\{(y,v):v\ge0,\ |y|\le Dv\},\qquad
x_t=
\begin{cases}
y_t/v_t, & v_t>0,\\
0, & (y_t,v_t)=(0,0).
\end{cases} C D = {( y , v ) : v ≥ 0 , ∣ y ∣ ≤ D v } , x t = { y t / v t , 0 , v t > 0 , ( y t , v t ) = ( 0 , 0 ) .
Both start from ( y 1 , v 1 ) = ( 0 , 0 ) (y_1,v_1)=(0,0) ( y 1 , v 1 ) = ( 0 , 0 ) and use lifted feedback ( h t , − h t x t ) (h_t,-h_tx_t) ( h t , − h t x t ) . We allow h t = k t g t h_t=k_tg_t h t = k t g t with finite k t ≥ 0 k_t\ge0 k t ≥ 0 , including arbitrary positive clipping thresholds.
CBA. The fixed quadratic FTRL update (Algorithm 1) is
( y t + 1 , v t + 1 ) = arg min ( y , v ) ∈ C D { y 2 + D 2 v 2 2 + ∑ s = 1 t h s ( y − x s v ) } . (36) (y_{t+1},v_{t+1})=\underset{(y,v)\in\mathcal{C}_D}{\arg\min}\left\{\frac{y^2+D^2v^2}{2}+\sum_{s=1}^{t}h_s(y-x_sv)\right\}. \tag*{(36)} ( y t + 1 , v t + 1 ) = ( y , v ) ∈ C D arg min { 2 y 2 + D 2 v 2 + s = 1 ∑ t h s ( y − x s v ) } . (36)
The objective is coercive and strictly convex on a closed convex set, so its minimizer is unique. If g t = 0 g_t=0 g t = 0 , then h t = 0 h_t=0 h t = 0 , and the cumulative objective and its minimizer are unchanged. Hence x t + 1 = x t x_{t+1}=x_t x t + 1 = x t , proving the record condition in both directions for every history.
CBA+ and its projection. In the Hilbert specialization of Algorithm 2, the change of variables ( y , v ) ↦ ( y , D v ) (y,v) \mapsto(y,Dv) ( y , v ) ↦ ( y , D v ) makes the weighted projection Euclidean. Its unprojected point is
( y ~ , v ~ ) = ( y t − h t , D v t + h t x t / D ) , (\widetilde{y},\widetilde{v})=(y_t-h_t,Dv_t+h_tx_t/D), ( y , v ) = ( y t − h t , D v t + h t x t / D ) ,
and the cone in these coordinates is v ~ ≥ ∣ y ~ ∣ \widetilde{v}\geq\lvert\widetilde{y}\rvert v ≥ ∣ y ∣ . Set r t = D v t + y t r_t=Dv_t+y_t r t = D v t + y t and ℓ t = D v t − y t \ell_t=Dv_t-y_t ℓ t = D v t − y t . Squared Euclidean distance in ( y , D v ) (y,Dv) ( y , D v ) is one half the sum of squared distances in ( r , ℓ ) (r,\ell) ( r , ℓ ) , so projection onto the cone takes the positive part of each coordinate:
r t + 1 = [ r t − h t ( 1 − x t / D ) ] + , ℓ t + 1 = [ ℓ t + h t ( 1 + x t / D ) ] + . (37) \begin{aligned}
r_{t+1}&=[r_t-h_t(1-x_t/D)]_+,\\
\ell_{t+1}&=[\ell_t+h_t(1+x_t/D)]_+. \tag*{(37)}
\end{aligned} r t + 1 ℓ t + 1 = [ r t − h t ( 1 − x t / D ) ] + , = [ ℓ t + h t ( 1 + x t / D ) ] + . (37)
Away from the origin, x t = D ( r t − ℓ t ) / ( r t + ℓ t ) x_t=D(r_t-\ell_t)/(r_t+\ell_t) x t = D ( r t − ℓ t ) / ( r t + ℓ t ) . If h t > 0 h_t>0 h t > 0 , then r t + 1 ≤ r t r_{t+1}\leq r_t r t + 1 ≤ r t and ℓ t + 1 ≥ ℓ t \ell_{t+1}\geq\ell_t ℓ t + 1 ≥ ℓ t . A nonzero state remains nonzero: if ℓ t > 0 \ell_t>0 ℓ t > 0 , then ℓ t + 1 ≥ ℓ t > 0 \ell_{t+1}\geq\ell_t>0 ℓ t + 1 ≥ ℓ t > 0 ; if ℓ t = 0 \ell_t=0 ℓ t = 0 , then x t = D x_t=D x t = D and r t + 1 = r t > 0 r_{t+1}=r_t>0 r t + 1 = r t > 0 . Therefore
x t + 1 − x t = 2 D ( r t + 1 ℓ t − r t ℓ t + 1 ) ( r t + 1 + ℓ t + 1 ) ( r t + ℓ t ) ≤ 0. x_{t+1}-x_t=\frac{2D(r_{t+1}\ell_t-r_t\ell_{t+1})}{(r_{t+1}+\ell_{t+1})(r_t+\ell_t)}\leq0. x t + 1 − x t = ( r t + 1 + ℓ t + 1 ) ( r t + ℓ t ) 2 D ( r t + 1 ℓ t − r t ℓ t + 1 ) ≤ 0.
For h t < 0 h_t<0 h t < 0 , the two coordinate inequalities reverse. The denominator remains positive because either r t > 0 r_t>0 r t > 0 persists or, if r t = 0 r_t=0 r t = 0 , then x t = − D x_t=-D x t = − D and ℓ t + 1 = ℓ t > 0 \ell_{t+1}=\ell_t>0 ℓ t + 1 = ℓ t > 0 . Thus x t + 1 − x t ≥ 0 x_{t+1}-x_t\geq0 x t + 1 − x t ≥ 0 . For h t = 0 h_t=0 h t = 0 both coordinates stay fixed. At the origin, a nonzero feedback gives
r t + 1 = ( − h t ) + , ℓ t + 1 = ( h t ) + , x t + 1 = − D sign ( h t ) . (38) r_{t+1}=(-h_t)_+,\qquad\ell_{t+1}=(h_t)_+,\qquad x_{t+1}=-D\operatorname{sign}(h_t). \tag*{(38)} r t + 1 = ( − h t ) + , ℓ t + 1 = ( h t ) + , x t + 1 = − D sign ( h t ) . (38)
These cases show that, relative to the raw gradient, x t + 1 = x t − c t g t x_{t+1}=x_t-c_tg_t x t + 1 = x t − c t g t , where c t = 0 c_t=0 c t = 0 if g t = 0 g_t=0 g t = 0 and c t = ( x t − x t + 1 ) / g t c_t=(x_t-x_{t+1})/g_t c t = ( x t − x t + 1 ) / g t otherwise. This factor is nonnegative, measurable, and at most 2 D / ∣ g t ∣ 2D/\lvert g_t\rvert 2 D / ∣ g t ∣ when g t ≠ 0 g_t\ne0 g t = 0 . Thus one-dimensional CBA+ satisfies Definition 3 itself, including the projection.
Conic FTRL with a known feedback bound. For Algorithm 3 of Grand-Clément et al. [22 ] , take anchor zero, ∣ h t ∣ ≤ G \lvert h_t\rvert\leq G ∣ h t ∣ ≤ G , ϵ > 0 \epsilon>0 ϵ > 0 , and d t = ϑ G 2 t d_t=\vartheta G^2t d t = ϑ G 2 t with ϑ ≥ 1 \vartheta\geq1 ϑ ≥ 1 . The domain is [ − D , D ] [-D,D] [ − D , D ] or R \mathbb{R} R . With S t − 1 S_{t-1} S t − 1 and W t W_t W t as in (35) , the lifted objective is
Φ t ( y , v ) = d t 2 ϵ y 2 + S t − 1 y + W t v − ϵ log v , v > 0 , (39) \Phi_t(y,v)=\frac{d_t}{2\epsilon}y^2+S_{t-1}y+W_tv-\epsilon\log v,\qquad v>0, \tag*{(39)} Φ t ( y , v ) = 2 ϵ d t y 2 + S t − 1 y + W t v − ϵ log v , v > 0 , (39)
subject to ∣ y ∣ ≤ D v \lvert y\rvert\leq Dv ∣ y ∣ ≤ D v on the interval; the constraint is absent on R \mathbb{R} R . We minimize this objective and decode x t = y t / v t x_t=y_t/v_t x t = y t / v t . Suppose first that W t > 0 W_t>0 W t > 0 . Completing the square in y y y shows that sublevel sets stay away from v = 0 v=0 v = 0 and have bounded v v v and y y y , since W t v − ϵ log v W_tv-\epsilon\log v W t v − ϵ log v tends to infinity at both ends. Strict convexity therefore gives a unique minimizer. The unconstrained minimizer is
y t = − ϵ S t − 1 d t , v t = ϵ W t . y_t=-\frac{\epsilon S_{t-1}}{d_t},\qquad v_t=\frac{\epsilon}{W_t}. y t = − d t ϵ S t − 1 , v t = W t ϵ .
If it satisfies ∣ y t ∣ ≤ D v t \lvert y_t\rvert\leq Dv_t ∣ y t ∣ ≤ D v t , it is also the constrained minimizer. Otherwise the constrained minimizer lies on ∣ y ∣ = D v \lvert y\rvert=Dv ∣ y ∣ = D v : an interior minimizer would satisfy the same unconstrained first-order equations. For S t − 1 ≠ 0 S_{t-1}\ne0 S t − 1 = 0 , its y y y coordinate has the opposite sign, since reflection decreases the linear term if their signs agree, and y = 0 y=0 y = 0 has a feasible descent direction. When S t − 1 = 0 S_{t-1}=0 S t − 1 = 0 , the minimizing y y y is zero. We have proved
x t = clip ( − W t S t − 1 d t , D ) (40) x_t=\operatorname{clip}\left(-\frac{W_tS_{t-1}}{d_t},D\right) \tag*{(40)} x t = clip ( − d t W t S t − 1 , D ) (40)
on the interval, with no clipping on R \mathbb{R} R . In either case,
∣ x t ∣ ≤ W t ∣ S t − 1 ∣ d t ≤ W t ( t − 1 ) ϑ G t , W t + 1 ≥ W t − G ∣ x t ∣ ≥ W t ( 1 − t − 1 ϑ t ) > 0. \lvert x_t\rvert\leq\frac{W_t\lvert S_{t-1}\rvert}{d_t}\leq\frac{W_t(t-1)}{\vartheta Gt},\qquad W_{t+1}\geq W_t-G\lvert x_t\rvert\geq W_t\left(1-\frac{t-1}{\vartheta t}\right)>0. ∣ x t ∣ ≤ d t W t ∣ S t − 1 ∣ ≤ ϑGt W t ( t − 1 ) , W t + 1 ≥ W t − G ∣ x t ∣ ≥ W t ( 1 − ϑt t − 1 ) > 0.
Starting from W 1 = ϵ W_1 = \epsilon W 1 = ϵ , this proves existence and positivity at every round. On a history of nonpositive feedback, S t − 1 ≤ 0 S_{t-1} \le0 S t − 1 ≤ 0 and x t ≥ 0 x_t \ge0 x t ≥ 0 . A zero observation leaves S t − 1 S_{t-1} S t − 1 and W t W_t W t unchanged, while d t + 1 ≥ d t d_{t+1} \ge d_t d t + 1 ≥ d t , so (40) gives 0 ≤ x t + 1 ≤ x t 0 \le x_{t+1} \le x_t 0 ≤ x t + 1 ≤ x t . Reflection proves the corresponding statement for nonnegative histories. The bounded preprocessing used for KT preserves these signs and zeros, so Algorithm 3 also satisfies Definition 20 . On R \mathbb{R} R with ϑ = 1 \vartheta= 1 ϑ = 1 it is KT.
Why Querying at the Running Average Is Different
The algorithm of Theorem 9 queries the running average x t x_t x t and updates an auxiliary state z t z_t z t by (20) . Starting from z 1 = x 1 = 0 z_1 = x_1 = 0 z 1 = x 1 = 0 , take g 1 = − A g_1 = -A g 1 = − A and g 2 = 0 g_2 = 0 g 2 = 0 , and put d = γ 1 min { A , λ 1 } > 0 d = \gamma_1 \min\{A, \lambda_1\} > 0 d = γ 1 min { A , λ 1 } > 0 . Its first two updates give
z 2 = d , x 2 = d / 2 , z 3 = d , x 3 = 2 d / 3. z_2 = d,\qquad x_2 = d/2,\qquad z_3 = d,\qquad x_3 = 2d/3. z 2 = d , x 2 = d /2 , z 3 = d , x 3 = 2 d /3.
The second round advances the rightward query record despite zero feedback. Reflecting the history excludes the leftward direction as well. Thus neither direction satisfies (33) . The auxiliary state can move ahead of the queries, which then continue toward it on zero-feedback rounds. This is consistent with the α p \alpha_p α p time-uniform rate in Theorem 9 .
No Uniformly Vanishing Last-Iterate Rate for CBA and CBA+
For standard CBA and CBA+, a first nonzero feedback produces the same nonzero decision after any number of zero observations. We use this property to prove a stronger lower bound than Proposition 21 for these two methods.
Proposition 22 (No vanishing last-iterate rate for CBA and CBA+). Fix p ∈ ( 1 , 2 ] p \in(1, 2] p ∈ ( 1 , 2 ] . Run CBA (36) or CBA+ (37) on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] with zero lifted initialization and default decision zero, querying at the decoded state and outputting the next decoded state. The method receives either g t g_t g t or h t = clip ( g t , λ t ) h_t = \operatorname{clip}(g_t, \lambda_t) h t = clip ( g t , λ t ) with any finite strictly positive measurable thresholds λ t \lambda_t λ t .
For every deterministic integer t ≥ 1 t \ge1 t ≥ 1 , the objective f ( x ) = x 2 / 2 f(x) = x^2/2 f ( x ) = x 2 /2 admits an oracle satisfying Assumption 2 with σ = 1 \sigma= 1 σ = 1 for which
P { f ( x t + 1 ) − f ⋆ = 1 / 2 } = 1. \mathbb{P}\{f(x_{t+1}) - f^\star= 1/2\} = 1. P { f ( x t + 1 ) − f ⋆ = 1/2 } = 1. For every deterministic envelope b t → 0 b_t \to0 b t → 0 and every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , this same objective admits an i.i.d. additive-noise oracle with convex, globally 1 1 1 -smooth sample losses attaining their minima, such that
P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > b t } > δ . \mathbb{P}\{\exists t \ge1 : f(x_{t+1}) - f^\star> b_t\} > \delta. P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > b t } > δ . Both constructions satisfy Assumptions 1 and 2 with L = R = σ = 1 L = R = \sigma= 1 L = R = σ = 1 and x 1 = x ⋆ = 0 x_1 = x^\star= 0 x 1 = x ⋆ = 0 .
Proof We first compute the response from the initial state. For CBA+, a zero feedback leaves the lifted state at zero by (37) ; for CBA, its cumulative linear term remains zero, so (36) has the same zero minimizer. After any such prefix, let h ≠ 0 h \ne0 h = 0 be the first nonzero supplied feedback. For both methods the next lifted state is
argmin v ≥ ∣ y ∣ { y 2 + v 2 2 + h y } = ( − h 2 , ∣ h ∣ 2 ) . \underset{v \ge|y|}{\operatorname{argmin}}\left\{\frac{y^2+v^2}{2}+hy\right\} = \left(-\frac{h}{2},\frac{|h|}{2}\right). v ≥ ∣ y ∣ argmin { 2 y 2 + v 2 + h y } = ( − 2 h , 2 ∣ h ∣ ) .
Indeed, for each y y y the minimizing height is v = ∣ y ∣ v = |y| v = ∣ y ∣ , and the remaining objective is y 2 + h y y^2+hy y 2 + h y . Decoding gives x = − sign ( h ) x = -\operatorname{sign}(h) x = − sign ( h ) . Positive clipping preserves the nonzero sign of any pulse, independently of its time and the threshold's magnitude. For the first part, fix a time τ ≥ 1 \tau\ge1 τ ≥ 1 and define on the whole infinite run
g s = x s + ξ s , ξ s = { + 1 or − 1 with equal probability , s = τ , 0 , s ≠ τ . g_s = x_s + \xi_s,\qquad\xi_s =
\begin{cases}
+1\text{ or }-1\text{ with equal probability}, & s=\tau,\\
0, & s\ne\tau.
\end{cases} g s = x s + ξ s , ξ s = { + 1 or − 1 with equal probability , 0 , s = τ , s = τ .
The pulse sign is independent of the past, so E [ ξ s ∣ F s − 1 ] = 0 \mathbb{E}[\xi_s \mid\mathcal{F}_{s-1}] = 0 E [ ξ s ∣ F s − 1 ] = 0 and E [ ∣ ξ s ∣ p ∣ F s − 1 ] ≤ 1 \mathbb{E}[|\xi_s|^p \mid\mathcal{F}_{s-1}] \le1 E [ ∣ ξ s ∣ p ∣ F s − 1 ] ≤ 1 for every s s s . Before τ \tau τ , induction gives x s = 0 x_s = 0 x s = 0 and zero lifted state. At τ \tau τ , the first nonzero feedback therefore yields x τ + 1 = − sign ( ξ τ ) x_{\tau+1} = -\operatorname{sign}(\xi_\tau) x τ + 1 = − sign ( ξ τ ) and f ( x τ + 1 ) − f ⋆ = 1 / 2 f(x_{\tau+1}) - f^\star= 1/2 f ( x τ + 1 ) − f ⋆ = 1/2 surely. Since τ \tau τ was arbitrary, this proves the fixed-time statement. It also gives sup O E O [ f ( x t + 1 ) − f ⋆ ] ≥ 1 / 2 \sup_{\mathcal{O}} \mathbb{E}_{\mathcal{O}}[f(x_{t+1}) - f^\star] \ge1/2 sup O E O [ f ( x t + 1 ) − f ⋆ ] ≥ 1/2 for every t t t , where the supremum is over the admissible oracles for this objective. For any fixed confidence δ < 1 \delta< 1 δ < 1 , a uniformly vanishing fixed-time envelope eventually falls below 1 / 2 1/2 1/2 and hence fails with probability one.
For the second part, choose N ≥ 1 N \ge1 N ≥ 1 so that b t < 1 / 2 b_t < 1/2 b t < 1/2 for all t ≥ N t \ge N t ≥ N , and choose π ∈ ( 0 , 1 ) \pi\in(0,1) π ∈ ( 0 , 1 ) such that ( 1 − π ) N − 1 > δ (1-\pi)^{N-1} > \delta ( 1 − π ) N − 1 > δ . Let the additive noise be i.i.d. with
P ( ξ t = 1 ) = P ( ξ t = − 1 ) = π / 2 , P ( ξ t = 0 ) = 1 − π . \mathbb{P}(\xi_t = 1) = \mathbb{P}(\xi_t = -1) = \pi/2,\qquad\mathbb{P}(\xi_t = 0) = 1-\pi. P ( ξ t = 1 ) = P ( ξ t = − 1 ) = π /2 , P ( ξ t = 0 ) = 1 − π .
It is centered and has E t − 1 ∣ ξ t ∣ p = π ≤ 1 \mathbb{E}_{t-1}|\xi_t|^p = \pi\le1 E t − 1 ∣ ξ t ∣ p = π ≤ 1 . The first nonzero time τ \tau τ is finite almost surely and satisfies P ( τ ≥ N ) = ( 1 − π ) N − 1 > δ \mathbb{P}(\tau\ge N) = (1-\pi)^{N-1} > \delta P ( τ ≥ N ) = ( 1 − π ) N − 1 > δ . The state stays zero until this time, and the first-response calculation gives gap 1 / 2 1/2 1/2 at x τ + 1 x_{\tau+1} x τ + 1 . Consequently
P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > b t } ≥ P ( τ ≥ N ) > δ . \mathbb{P}\{\exists t \ge1 : f(x_{t+1}) - f^\star> b_t\} \ge\mathbb{P}(\tau\ge N) > \delta. P { ∃ t ≥ 1 : f ( x t + 1 ) − f ⋆ > b t } ≥ P ( τ ≥ N ) > δ .
The sample losses F ( x ; ξ ) = x 2 / 2 + ξ x F(x;\xi) = x^2/2 + \xi x F ( x ; ξ ) = x 2 /2 + ξ x are convex and globally 1 1 1 -smooth, and each attains its minimum at − ξ -\xi − ξ . ■
For f ( x ) = x 2 / 2 f(x) = x^2/2 f ( x ) = x 2 /2 , the delayed-pulse argument applies whenever the complete effective state has a time-homogeneous representation s t + 1 = Φ ( s t , h t ) s_{t+1} = \Phi(s_t,h_t) s t + 1 = Φ ( s t , h t ) , x t = ψ ( s t ) x_t = \psi(s_t) x t = ψ ( s t ) , the initial state satisfies Φ ( s 1 , 0 ) = s 1 \Phi(s_1,0) = s_1 Φ ( s 1 , 0 ) = s 1 with ψ ( s 1 ) = 0 \psi(s_1) = 0 ψ ( s 1 ) = 0 , and a fixed bounded symmetric pulse produces a positive objective gap for both signs. CBA and CBA+ with the fixed quadratic regularizer and zero initialization satisfy these conditions; moreover, their decoded first response is independent of every positive pulse magnitude. This last property allows arbitrary strictly positive clipping thresholds in Proposition 22 . A time counter is part of the effective state when it affects decisions. For example, after τ − 1 \tau- 1 τ − 1 zero observations, KT has S τ − 1 = 0 S_{\tau-1} = 0 S τ − 1 = 0 and W τ = ϵ W_\tau= \epsilon W τ = ϵ , so its first bounded feedback h τ h_\tau h τ gives
x τ + 1 = − ϵ h τ G 2 ( τ + 1 ) . x_{\tau+1} = -\frac{\epsilon h_\tau}{G^2(\tau+1)}. x τ + 1 = − G 2 ( τ + 1 ) ϵ h τ .
Its first-response magnitude decreases with the delay, while its last-iterate guarantees remain subject to Proposition 21 .
Proof of the Horizon-Free Time-Uniform Upper Bound
The proof first derives a time-uniform radius bound from a global decomposition of the conditional expectation of a clipped gradient. Once the clipping threshold exceeds the gradient norms allowed by this radius, the local clipping estimates apply on a deterministic suffix of the trajectory. For each analysis horizon, we use bounded inverse tail weights to control the full potential, which consists of the objective gap at a comparison point and its squared distance to the last iterate. Smoothness converts both terms into the endpoint gap by (25) . A summable allocation over horizons and a direct bound at early times give the same D p = 4 / c p D_p = 4/c_p D p = 4/ c p for all times. We first prove Theorem 2 under the normalized upper bounds L , R , σ ≤ 1 L,R,\sigma\le1 L , R , σ ≤ 1 . Thus ∥ x 1 − x ⋆ ∥ ≤ 1 \lVert x_1-x^\star\rVert\le1 ∥ x 1 − x ⋆ ∥ ≤ 1 , f f f is convex and at most 1-smooth, and the conditional central p p p th noise moment is at most one. Define β , a , b \beta,a,b β , a , b by (4) , and write
ℓ t = log ( 100 t 2 ) , η t = c t − a / ℓ t 2 , λ t = 4 t b . \ell_t=\log(100t^2),\qquad\eta_t=ct^{-a}/\ell_t^2,\qquad\lambda_t=4t^b. ℓ t = log ( 100 t 2 ) , η t = c t − a / ℓ t 2 , λ t = 4 t b .
Here c ∈ ( 0 , 1 ] c\in(0,1] c ∈ ( 0 , 1 ] is a deterministic constant depending only on p p p , chosen below. Set G t = f ( x t ) − f ⋆ G_t=f(x_t)-f^\star G t = f ( x t ) − f ⋆ , e t = clip ( g t , λ t ) − ∇ f ( x t ) e_t=\operatorname{clip}(g_t,\lambda_t)-\nabla f(x_t) e t = clip ( g t , λ t ) − ∇ f ( x t ) , b t = E t − 1 e t b_t=\mathbb{E}_{t-1}e_t b t = E t − 1 e t , and u t = e t − b t u_t=e_t-b_t u t = e t − b t . Whenever ∥ ∇ f ( x t ) ∥ ≤ λ t / 2 \lVert\nabla f(x_t)\rVert\le\lambda_t/2 ∥ ∇ f ( x t )∥ ≤ λ t /2 , Lemma 12 gives
∥ e t ∥ ≤ 3 2 λ t , ∥ u t ∥ ≤ 2 λ t , ∥ b t ∥ ≤ B t : = 2 λ t 1 − p , E t − 1 ∥ e t ∥ 2 , E t − 1 ∥ u t ∥ 2 ≤ V t : = 2 λ t 2 − p . (41) \lVert e_t\rVert\le\frac{3}{2}\lambda_t,\qquad\lVert u_t\rVert\le2\lambda_t,\qquad\lVert b_t\rVert\le B_t:=2\lambda_t^{1-p},\qquad\mathbb{E}_{t-1}\lVert e_t\rVert^2,\ \mathbb{E}_{t-1}\lVert u_t\rVert^2\le V_t:=2\lambda_t^{2-p}. \tag*{(41)} ∥ e t ∥ ≤ 2 3 λ t , ∥ u t ∥ ≤ 2 λ t , ∥ b t ∥ ≤ B t := 2 λ t 1 − p , E t − 1 ∥ e t ∥ 2 , E t − 1 ∥ u t ∥ 2 ≤ V t := 2 λ t 2 − p . (41)
The bounds B t , V t B_t,V_t B t , V t are deterministic. The concentration arguments below apply these conditional estimates on predictable events.
Explicit Constants and Deterministic Budgets
Put v = ( 2 − p ) b v=(2-p)b v = ( 2 − p ) b . The exponent identities needed throughout are
b ( p − 1 ) = β , a + β = 1 , 2 ( a − b ) = β , a − v = 3 2 β , 2 a − v = 1 + 1 2 β . (42) b(p-1)=\beta,\qquad a+\beta=1,\qquad2(a-b)=\beta,\qquad a-v=\frac{3}{2}\beta,\qquad2a-v=1+\frac{1}{2}\beta. \tag*{(42)} b ( p − 1 ) = β , a + β = 1 , 2 ( a − b ) = β , a − v = 2 3 β , 2 a − v = 1 + 2 1 β . (42)
Let q = 2 − β q=2^{-\beta} q = 2 − β and define the following finite constants:
J β = ∑ j = 0 ∞ q j ( j + 2 ) 2 = q ( 1 + q ) ( 1 − q ) 3 + 4 q ( 1 − q ) 2 + 4 1 − q , J_\beta=\sum_{j=0}^{\infty}q^j(j+2)^2=\frac{q(1+q)}{(1-q)^3}+\frac{4q}{(1-q)^2}+\frac{4}{1-q}, J β = j = 0 ∑ ∞ q j ( j + 2 ) 2 = ( 1 − q ) 3 q ( 1 + q ) + ( 1 − q ) 2 4 q + 1 − q 4 ,
H β = 64 ( 1 + β − 1 + J β ) , K = 2 20 H β 4 , (43) H_\beta=64(1+\beta^{-1}+J_\beta),\qquad K=2^{20}H_\beta^4, \tag*{(43)} H β = 64 ( 1 + β − 1 + J β ) , K = 2 20 H β 4 , (43)
c p = 1 1000 K , D p = 4 c p . c_p=\frac{1}{1000K},\qquad D_p=\frac{4}{c_p}. c p = 1000 K 1 , D p = c p 4 .
These constants depend only on p p p . The deterministic inequalities below hold for every 0 < c ≤ 1 0<c\le1 0 < c ≤ 1 ; we then take c = c p c=c_p c = c p .
For a deterministic proof horizon T T T , set S t = ∑ s = t T η s S_t=\sum_{s=t}^T\eta_s S t = ∑ s = t T η s , S 0 = S 1 S_0=S_1 S 0 = S 1 , S T + 1 = 0 S_{T+1}=0 S T + 1 = 0 , U = 1 + S 1 U=1+S_1 U = 1 + S 1 , and
Q t = U η t 2 1 + S t . (44) Q_t=\frac{U\eta_t^2}{1+S_t}. \tag*{(44)} Q t = 1 + S t U η t 2 . (44)
The following estimates treat early times, late times, and small cumulative step sums.
Lemma 23 (Deterministic anytime budgets). For all T ≥ 1 T\ge1 T ≥ 1 and 0 < c ≤ 1 0<c\le1 0 < c ≤ 1 ,
c T β 2 ℓ T 2 ≤ S 1 ≤ H β T β c T 2 , (45) \frac{cT^\beta}{2\ell_T^2}\le S_1\le\frac{H_\beta T^\beta}{c_T^2}, \tag*{(45)} 2 ℓ T 2 c T β ≤ S 1 ≤ c T 2 H β T β , (45) ∑ t ≥ 1 η t B t ≤ K c ℓ 1 , ∑ t ≥ 1 η t 2 V t ≤ K c 2 ℓ 1 4 , sup t ≥ 1 η t λ t ≤ 4 c ℓ 1 2 , (46) \sum_{t\ge1}\eta_tB_t\le\frac{Kc}{\ell_1},\qquad\sum_{t\ge1}\eta_t^2V_t\le\frac{Kc^2}{\ell_1^4},\qquad\sup_{t\ge1}\eta_t\lambda_t\le\frac{4c}{\ell_1^2}, \tag*{(46)} t ≥ 1 ∑ η t B t ≤ ℓ 1 K c , t ≥ 1 ∑ η t 2 V t ≤ ℓ 1 4 K c 2 , t ≥ 1 sup η t λ t ≤ ℓ 1 2 4 c , (46) ∑ t = 1 T Q t B t ≤ K c ℓ 1 , ∑ t = 1 T Q t V t ≤ K c 2 ℓ 1 3 , max t ≤ T Q t λ t 2 ≤ K c 2 ℓ 1 4 . (47) \sum_{t=1}^T\sqrt{Q_t}B_t\le\frac{Kc}{\ell_1},\qquad\sum_{t=1}^TQ_tV_t\le\frac{Kc^2}{\ell_1^3},\qquad\max_{t\le T}Q_t\lambda_t^2\le\frac{Kc^2}{\ell_1^4}. \tag*{(47)} t = 1 ∑ T Q t B t ≤ ℓ 1 K c , t = 1 ∑ T Q t V t ≤ ℓ 1 3 K c 2 , t ≤ T max Q t λ t 2 ≤ ℓ 1 4 K c 2 . (47)
Proof. From B t ≤ 2 t − β B_t\le2t^{-\beta} B t ≤ 2 t − β , V t ≤ 8 t v V_t\le8t^v V t ≤ 8 t v and (42) ,
η t B t ≤ 2 c t ℓ t 2 , η t 2 V t ≤ 8 c 2 t − 1 − β / 2 ℓ t 4 , η t 2 λ t 2 = 16 c 2 t − β ℓ t 4 . \eta_tB_t\le\frac{2c}{t\ell_t^2},\qquad\eta_t^2V_t\le\frac{8c^2t^{-1-\beta/2}}{\ell_t^4},\qquad\eta_t^2\lambda_t^2=\frac{16c^2t^{-\beta}}{\ell_t^4}. η t B t ≤ t ℓ t 2 2 c , η t 2 V t ≤ ℓ t 4 8 c 2 t − 1 − β /2 , η t 2 λ t 2 = ℓ t 4 16 c 2 t − β .
Integral comparison gives ∑ t ( t ℓ t 2 ) − 1 ≤ ℓ 1 − 1 \sum_t (t\ell_t^2)^{-1} \le\ell_1^{-1} ∑ t ( t ℓ t 2 ) − 1 ≤ ℓ 1 − 1 and ∑ t t − 1 − β / 2 ≤ 1 + 2 / β \sum_t t^{-1-\beta/2} \le1+2/\beta ∑ t t − 1 − β /2 ≤ 1 + 2/ β , proving (46) . The final ⌈ T / 2 ⌉ \lceil T/2\rceil ⌈ T /2 ⌉ indices give S 1 ≥ ( T / 2 ) η T S_1 \ge(T/2)\eta_T S 1 ≥ ( T /2 ) η T . On each nonempty dyadic block T / 2 j + 1 < t ≤ T / 2 j T/2^{j+1}<t\le T/2^j T / 2 j + 1 < t ≤ T / 2 j ,
ℓ T / ℓ t ≤ 2 ( j + 2 ) , ∑ T / 2 j + 1 < t ≤ T / 2 j t − a ≤ 2 T β 2 − β j . \ell_T/\ell_t \le2(j+2), \qquad\sum_{T/2^{j+1}<t\le T/2^j}t^{-a}\le2T^{\beta}2^{-\beta j}. ℓ T / ℓ t ≤ 2 ( j + 2 ) , T / 2 j + 1 < t ≤ T / 2 j ∑ t − a ≤ 2 T β 2 − β j .
Summation gives S 1 ≤ 8 J β c T β / ℓ T 2 S_1\le8J_\beta cT^\beta/\ell_T^2 S 1 ≤ 8 J β c T β / ℓ T 2 and hence (45) .
Split the sums of (47) at ⌊ T / 2 ⌋ \lfloor T/2\rfloor ⌊ T /2 ⌋ . On the early half, U / ( 1 + S t ) ≤ 4 H β U/(1+S_t)\le4H_\beta U / ( 1 + S t ) ≤ 4 H β , so its bias, variance and range contributions are at most
4 H β c ℓ 1 2 , 32 H β ( 1 + 2 / β ) c 2 ℓ 1 4 , 64 H β c 2 ℓ 1 4 . 4\sqrt{H_\beta\frac{c}{\ell_1^2}},\qquad32H_\beta(1+2/\beta)\frac{c^2}{\ell_1^4},\qquad64H_\beta\frac{c^2}{\ell_1^4}. 4 H β ℓ 1 2 c , 32 H β ( 1 + 2/ β ) ℓ 1 4 c 2 , 64 H β ℓ 1 4 c 2 .
On the late half, t ≥ T / 2 t\ge T/2 t ≥ T /2 and ℓ T / ℓ t ≤ 2 \ell_T/\ell_t\le2 ℓ T / ℓ t ≤ 2 imply
η t V t ≤ 64 c T − 3 β / 2 ℓ T 2 , η t 2 λ t 2 ≤ 512 c 2 T − β ℓ T 4 , B t ≤ 4 T − β . \eta_tV_t\le\frac{64cT^{-3\beta/2}}{\ell_T^2},\qquad\eta_t^2\lambda_t^2\le\frac{512c^2T^{-\beta}}{\ell_T^4},\qquad B_t\le4T^{-\beta}. η t V t ≤ ℓ T 2 64 c T − 3 β /2 , η t 2 λ t 2 ≤ ℓ T 4 512 c 2 T − β , B t ≤ 4 T − β .
Since η t = S t − S t + 1 \eta_t=S_t-S_{t+1} η t = S t − S t + 1 , comparing the sums with the corresponding integrals yields
∑ t η t 1 + S t ≤ log U , ∑ t η t 1 + S t ≤ 2 ( U − 1 ) . (48) \sum_t\frac{\eta_t}{1+S_t}\le\log U,\qquad\sum_t\frac{\eta_t}{\sqrt{1+S_t}}\le2(\sqrt{U}-1). \tag*{(48)} t ∑ 1 + S t η t ≤ log U , t ∑ 1 + S t η t ≤ 2 ( U − 1 ) . (48)
Consequently the late bias and range are bounded by
8 T − β U ( U − 1 ) ≤ 8 H β c ℓ T 2 , 512 U c 2 T − β ℓ T 4 ≤ 1024 H β c 2 ℓ T 4 . 8T^{-\beta}\sqrt{U}(\sqrt{U}-1)\le\frac{8H_\beta c}{\ell_T^2},\qquad\frac{512Uc^2T^{-\beta}}{\ell_T^4}\le\frac{1024H_\beta c^2}{\ell_T^4}. 8 T − β U ( U − 1 ) ≤ ℓ T 2 8 H β c , ℓ T 4 512 U c 2 T − β ≤ ℓ T 4 1024 H β c 2 .
For the late variance, use U log U ≤ S 1 ( 1 + log U ) U\log U\le S_1(1+\log U) U log U ≤ S 1 ( 1 + log U ) , valid also when S 1 ≤ 1 S_1\le1 S 1 ≤ 1 , and log U ≤ H β ℓ T \log U\le H_\beta\ell_T log U ≤ H β ℓ T to obtain
∑ t > T / 2 Q t V t ≤ 64 c T − 3 β / 2 ℓ T 2 U log U ≤ 128 H β 2 c 2 T − β / 2 ℓ T 3 . \sum_{t>T/2}Q_tV_t\le\frac{64cT^{-3\beta/2}}{\ell_T^2}U\log U\le\frac{128H_\beta^2c^2T^{-\beta/2}}{\ell_T^3}. t > T /2 ∑ Q t V t ≤ ℓ T 2 64 c T − 3 β /2 U log U ≤ ℓ T 3 128 H β 2 c 2 T − β /2 .
Here U ≤ 1 + H β T β U\le1+H_\beta T^\beta U ≤ 1 + H β T β gives the logarithmic bound. Adding the early contributions proves (47) with K = 2 20 H β 4 K=2^{20}H_\beta^4 K = 2 20 H β 4 .
A Radius Event for All Times
Use c = c p c=c_p c = c p and fix δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ; the schedule does not depend on δ \delta δ . Write h = log ( 100 / δ ) h=\log(100/\delta) h = log ( 100/ δ ) and ι T = ι T ( δ ) = log ( 100 T 2 / δ ) \iota_T=\iota_T(\delta)=\log(100T^2/\delta) ι T = ι T ( δ ) = log ( 100 T 2 / δ ) ; thus ι T ≥ h > 2 \iota_T\ge h>2 ι T ≥ h > 2 . Apply Lemma 13 conditionally at every step, with μ t = ∇ f ( x t ) \mu_t=\nabla f(x_t) μ t = ∇ f ( x t ) , and write
clip ( g t , λ t ) = A t s μ t + d t , d t = ε t + u t , E t − 1 u t = 0. \operatorname{clip}(g_t,\lambda_t)=A_t^s\mu_t+d_t,\qquad d_t=\varepsilon_t+u_t,\qquad\mathbb{E}_{t-1}u_t=0. clip ( g t , λ t ) = A t s μ t + d t , d t = ε t + u t , E t − 1 u t = 0.
The following bounds hold globally: 0 ≤ A t s ≤ 1 0\le A_t^s\le1 0 ≤ A t s ≤ 1 , ∥ ε t ∥ ≤ B t \|\varepsilon_t\|\le B_t ∥ ε t ∥ ≤ B t , E t − 1 ∥ u t ∥ 2 ≤ V t \mathbb{E}_{t-1}\|u_t\|^2\le V_t E t − 1 ∥ u t ∥ 2 ≤ V t , ∥ u t ∥ ≤ 2 λ t \|u_t\|\le2\lambda_t ∥ u t ∥ ≤ 2 λ t , ∥ d t ∥ ≤ 3 λ t \|d_t\|\le3\lambda_t ∥ d t ∥ ≤ 3 λ t , and E t − 1 ∥ d t ∥ 2 ≤ 2 V t \mathbb{E}_{t-1}\|d_t\|^2\le2V_t E t − 1 ∥ d t ∥ 2 ≤ 2 V t .
Let r t = ∥ x t − x ⋆ ∥ r_t=\|x_t-x^\star\| r t = ∥ x t − x ⋆ ∥ and G t = f ( x t ) − f ⋆ G_t=f(x_t)-f^\star G t = f ( x t ) − f ⋆ . Since η t A t s ≤ 1 / 2 \eta_tA_t^s\le1/2 η t A t s ≤ 1/2 , expansion and ∥ μ t ∥ 2 ≤ 2 G t \|\mu_t\|^2\le2G_t ∥ μ t ∥ 2 ≤ 2 G t , ⟨ μ t , x t − x ⋆ ⟩ ≥ G t \langle\mu_t,x_t-x^\star\rangle\ge G_t ⟨ μ t , x t − x ⋆ ⟩ ≥ G t give
r t + 1 2 ≤ r t 2 − 2 η t ⟨ d t , x t − x ⋆ ⟩ + 2 η t 2 ∥ d t ∥ 2 . r_{t+1}^2\le r_t^2-2\eta_t\langle d_t,x_t-x^\star\rangle+2\eta_t^2\|d_t\|^2. r t + 1 2 ≤ r t 2 − 2 η t ⟨ d t , x t − x ⋆ ⟩ + 2 η t 2 ∥ d t ∥ 2 .
The remaining term with μ t \mu_t μ t is at most 2 η t A t s ( 2 η t A t s − 1 ) G t ≤ 0 2\eta_tA_t^s(2\eta_tA_t^s-1)G_t\le0 2 η t A t s ( 2 η t A t s − 1 ) G t ≤ 0 , as 0 ≤ η t A t s ≤ 1 / 2 0\le\eta_tA_t^s\le1/2 0 ≤ η t A t s ≤ 1/2 .
Apply Lemma 15 to this recursion with P t − 1 = r t 2 P_{t-1}=r_t^2 P t − 1 = r t 2 , p 0 = 1 p_0=1 p 0 = 1 , a t = 2 η t a_t=2\eta_t a t = 2 η t , s t = 2 η t s_t=2\eta_t s t = 2 η t , and w t = − 2 η t ( x t − x ⋆ ) w_t=-2\eta_t(x_t-x^\star) w t = − 2 η t ( x t − x ⋆ ) , using confidence parameter h h h and J t ≡ 1 J_t\equiv1 J t ≡ 1 (no localization indicator). The error bounds above and (46) , using ℓ 1 > 1 \ell_1>1 ℓ 1 > 1 , give
C ≤ 1 + 650 K c 2 h , D ≤ 2 K c + 6 K c h + 32 c h . C\le1+650Kc^2h,\qquad D\le2Kc+6\sqrt{K}c\sqrt{h}+32ch. C ≤ 1 + 650 K c 2 h , D ≤ 2 K c + 6 K c h + 32 c h .
Since K ≥ 1 K \ge1 K ≥ 1 , h > 2 h > 2 h > 2 and K c = 1 / 1000 Kc = 1/1000 K c = 1/1000 , C + D ≤ 1 + 66 K c h < h \sqrt{C} + D \le1 + 66Kch < h C + D ≤ 1 + 66 K c h < h . Hence there is an event R δ \mathcal{R}_{\delta} R δ , with P ( R δ ) ≥ 1 − e − h \mathbb{P}(\mathcal{R}_{\delta}) \ge1 - e^{-h} P ( R δ ) ≥ 1 − e − h , on which
r t ≤ h , G t ≤ h 2 / 2 for every t ≥ 1. (49) r_t \le h,\qquad G_t \le h^2/2\qquad\text{for every }t \ge1. \tag*{(49)} r t ≤ h , G t ≤ h 2 /2 for every t ≥ 1. (49)
The gap bound uses smoothness at the minimizer.
A Potential on a Suffix of the Trajectory
Set k = ⌈ h 1 / b ⌉ k = \lceil h^{1/b}\rceil k = ⌈ h 1/ b ⌉ and split the analysis of the original recursion at time k k k . For t ≥ k t \ge k t ≥ k , λ t ≥ 4 h \lambda_t \ge4h λ t ≥ 4 h , so (41) holds whenever r t ≤ 2 h r_t \le2h r t ≤ 2 h , since ∥ ∇ f ( x t ) ∥ ≤ r t ≤ λ t / 2 \lVert\nabla f(x_t)\rVert\le r_t \le\lambda_t/2 ∥ ∇ f ( x t )∥ ≤ r t ≤ λ t /2 .
Fix a deterministic T ≥ 2 k T \ge2k T ≥ 2 k . Apply Lemma 11 , with ζ = 1 \zeta= 1 ζ = 1 , to the updates k , … , T k,\ldots,T k , … , T . Keep S t = ∑ s = t T η s S_t = \sum_{s=t}^{T}\eta_s S t = ∑ s = t T η s , and set U t = 1 + S t U_t = 1 + S_t U t = 1 + S t , W = U k W = U_k W = U k , U k − 1 = U k U_{k-1} = U_k U k − 1 = U k , and write A m A_m A m , H m H_m H m for the quantities A m A_m A m , H m H_m H m of Lemma 11 for these updates. Start its analysis recursions at z k − 1 = z k = x ⋆ z_{k-1} = z_k = x^\star z k − 1 = z k = x ⋆ and A k − 1 = 0 A_{k-1} = 0 A k − 1 = 0 , with H k − 1 = r k 2 / 2 H_{k-1} = r_k^2/2 H k − 1 = r k 2 /2 . Write
Q t ( k ) = W η t 2 U t , w t = W η t U t − 1 ( z t − 1 − x t ) , H t = W 2 U t ∥ z t − x t + 1 ∥ 2 . Q_t^{(k)} = \frac{W\eta_t^2}{U_t},\qquad w_t = \frac{W\eta_t}{U_{t-1}}(z_{t-1}-x_t),\qquad H_t = \frac{W}{2U_t}\lVert z_t-x_{t+1}\rVert^2. Q t ( k ) = U t W η t 2 , w t = U t − 1 W η t ( z t − 1 − x t ) , H t = 2 U t W ∥ z t − x t + 1 ∥ 2 .
The proof of Lemma 11 is pathwise, so here its initial distance term is r k 2 / 2 r_k^2/2 r k 2 /2 , also when x k x_k x k is random. Its inequality for the partial sums is
W A m + H m ≤ 1 2 r k 2 + 2 ∑ t = k m Q t ( k ) ∥ e t ∥ 2 + ∑ t = k m ⟨ e t , w t ⟩ . (50) WA_m + H_m \le\frac{1}{2}r_k^2 + 2\sum_{t=k}^{m}Q_t^{(k)}\lVert e_t\rVert^2 + \sum_{t=k}^{m}\langle e_t,w_t\rangle. \tag*{(50)} W A m + H m ≤ 2 1 r k 2 + 2 t = k ∑ m Q t ( k ) ∥ e t ∥ 2 + t = k ∑ m ⟨ e t , w t ⟩ . (50)
The denominator of Q t ( k ) Q_t^{(k)} Q t ( k ) agrees with that of Q t Q_t Q t in (44) , and its numerator is smaller. Thus Q t ( k ) ≤ Q t Q_t^{(k)} \le Q_t Q t ( k ) ≤ Q t , so all three budgets (47) apply on the suffix.
Apply Lemma 15 with the predictable indicators J t = 1 { r s ≤ 2 h for k ≤ s ≤ t } J_t = 1_{\{r_s\le2h\text{ for }k\le s\le t\}} J t = 1 { r s ≤ 2 h for k ≤ s ≤ t } , with initial event r k ≤ h r_k \le h r k ≤ h and p 0 = h 2 / 2 p_0 = h^2/2 p 0 = h 2 /2 . Set
P m = W A m + H m , a t = 2 Q t ( k ) , s t = 2 Q t ( k ) . P_m = WA_m + H_m,\qquad a_t = 2Q_t^{(k)},\qquad s_t = \sqrt{2Q_t^{(k)}}. P m = W A m + H m , a t = 2 Q t ( k ) , s t = 2 Q t ( k ) .
Indeed ∥ w t ∥ 2 ≤ 2 Q t ( k ) H t − 1 ≤ s t 2 P t − 1 \lVert w_t\rVert^2 \le2Q_t^{(k)}H_{t-1} \le s_t^2P_{t-1} ∥ w t ∥ 2 ≤ 2 Q t ( k ) H t − 1 ≤ s t 2 P t − 1 , including t = k t = k t = k . With exponent q = ι T q = \iota_T q = ι T , the budgets (47) and ℓ 1 > 1 \ell_1 > 1 ℓ 1 > 1 give C ≤ h 2 / 2 + 17 K c 2 q C \le h^2/2 + 17Kc^2q C ≤ h 2 /2 + 17 K c 2 q and D ≤ 11 K c q D \le11Kcq D ≤ 11 K c q . Hence C + D ≤ ( 1 / 2 + 16 K c ) q < q \sqrt{C} + D \le(1/\sqrt{2} + 16Kc)q < q C + D ≤ ( 1/ 2 + 16 K c ) q < q . Except on an unconditional event of probability e − ι T e^{-\iota_T} e − ι T , the lemma therefore gives
W A m + H m ≤ ι T 2 ( k ≤ m ≤ T ) . WA_m + H_m \le\iota_T^2\qquad(k \le m \le T). W A m + H m ≤ ι T 2 ( k ≤ m ≤ T ) .
on R δ \mathcal{R}_{\delta} R δ , where the initial event holds and every localization indicator equals one. Lemma 15 allows a random initial state, so the concentration bound holds unconditionally before the intersection with R δ \mathcal{R}_{\delta} R δ . Intersecting these concentration events over deterministic T ≥ 2 k T \ge2k T ≥ 2 k defines P δ \mathcal{P}_{\delta} P δ with total failure at most ∑ T ≥ 2 k e − ι T \sum_{T\ge2k}e^{-\iota_T} ∑ T ≥ 2 k e − ι T .
Direct Last-Iterate Control
On R δ ∩ P δ \mathcal{R}_{\delta}\cap\mathcal{P}_{\delta} R δ ∩ P δ , for every T ≥ 2 k T \ge2k T ≥ 2 k , the full endpoint estimate (25) gives
G T + 1 ≤ 2 ( W A T + H T ) W ≤ 2 ι T 2 W ≤ 4 c p T − β ι T 2 ≤ 4 c p T − β ι T 4 . G_{T+1} \le\frac{2(WA_T+H_T)}{W} \le\frac{2\iota_T^2}{W}
\le\frac{4}{c_p}T^{-\beta}\iota_T^2 \le\frac{4}{c_p}T^{-\beta}\iota_T^4. G T + 1 ≤ W 2 ( W A T + H T ) ≤ W 2 ι T 2 ≤ c p 4 T − β ι T 2 ≤ c p 4 T − β ι T 4 .
Indeed, the suffix contains the last half of the updates, so W ≥ ∑ t = k T η t ≥ ( T / 2 ) η T = ( c p / 2 ) T β / ℓ T 2 W \ge\sum_{t=k}^{T}\eta_t \ge(T/2)\eta_T = (c_p/2)T^\beta/\ell_T^2 W ≥ ∑ t = k T η t ≥ ( T /2 ) η T = ( c p /2 ) T β / ℓ T 2 . For 1 ≤ T < 2 k 1 \le T < 2^k 1 ≤ T < 2 k , use k ≤ 2 h 1 / b k \le2h^{1/b} k ≤ 2 h 1/ b , 0 < β ≤ 2 / 5 0 < \beta\le2/5 0 < β ≤ 2/5 and β / b = p − 1 \beta/b = p - 1 β / b = p − 1 . On R δ \mathcal{R}_\delta R δ ,
T β G T + 1 ≤ 1 2 h 2 ( 4 h 1 / b ) β ≤ h p + 1 ≤ h 3 ≤ ℓ T 4 . T^\beta G_{T+1} \le\frac{1}{2}h^2(4h^{1/b})^\beta\le h^{p+1} \le h^3 \le\ell_T^4. T β G T + 1 ≤ 2 1 h 2 ( 4 h 1/ b ) β ≤ h p + 1 ≤ h 3 ≤ ℓ T 4 .
Since 4 / c p ≥ 1 4/c_p \ge1 4/ c p ≥ 1 , the same rate thus covers the early times as well. The total failure probability is at most
e − h + ∑ T ≥ 2 k e − ℓ T ≤ δ 100 ( 1 + ∑ T ≥ 1 T − 2 ) < δ . e^{-h} + \sum_{T \ge2^k} e^{-\ell_T} \le\frac{\delta}{100}\left(1+\sum_{T \ge1}T^{-2}\right)<\delta. e − h + T ≥ 2 k ∑ e − ℓ T ≤ 100 δ ( 1 + T ≥ 1 ∑ T − 2 ) < δ .
Rescaling to the Original Parameters
For L , R > 0 L,R>0 L , R > 0 and σ ≥ 0 \sigma\ge0 σ ≥ 0 , put Q = max { L R , σ } Q=\max\{LR,\sigma\} Q = max { L R , σ } and use the normalized coordinate y = ( x − x 1 ) / R y=(x-x_1)/R y = ( x − x 1 ) / R with objective
F ( y ) = f ( x 1 + R y ) − f ⋆ R Q . F(y)=\frac{f(x_1+Ry)-f^\star}{RQ}. F ( y ) = R Q f ( x 1 + R y ) − f ⋆ .
Its minimizer is within distance one of zero, its smoothness is L R / Q ≤ 1 LR/Q \le1 L R / Q ≤ 1 , and the normalized oracle is g / Q g/Q g / Q , with conditional central p p p th moment at most ( σ / Q ) p ≤ 1 (\sigma/Q)^p \le1 ( σ / Q ) p ≤ 1 . Applying the normalized theorem with step c p t − a / ℓ t 2 c_p t^{-a}/\ell_t^2 c p t − a / ℓ t 2 and threshold 4 t b 4t^b 4 t b gives, after multiplying coordinates by R R R , exactly (5) . Multiplying the objective bound by R Q RQ R Q yields (6) , and (49) becomes ∥ x t − x ⋆ ∥ 2 ≤ R 2 [ log ( 100 / δ ) ] 2 \lVert x_t-x^\star\rVert^2 \le R^2[\log(100/\delta)]^2 ∥ x t − x ⋆ ∥ 2 ≤ R 2 [ log ( 100/ δ ) ] 2 . This scaling also covers σ = 0 \sigma=0 σ = 0 since Q = L R > 0 Q=LR>0 Q = L R > 0 .
Almost Sure Consequence
The schedule (5) does not depend on δ \delta δ , so Theorem 2 applies to the same trajectory for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Setting δ = 1 / n \delta=1/n δ = 1/ n yields events of probability at least 1 − 1 / n 1-1/n 1 − 1/ n on which f ( x T + 1 ) − f ⋆ ≤ D p R Q T − β p ℓ T 4 ( 1 + log n / ℓ T ) 4 f(x_{T+1})-f^\star\le D_pRQ T^{-\beta_p}\ell_T^4(1+\log n/\ell_T)^4 f ( x T + 1 ) − f ⋆ ≤ D p R Q T − β p ℓ T 4 ( 1 + log n / ℓ T ) 4 for every T ≥ 1 T \ge1 T ≥ 1 . Consequently, the rate also holds almost surely: with probability one,
lim sup T → ∞ f ( x T + 1 ) − f ⋆ T − β p ℓ T 4 ≤ D p R Q . \limsup_{T\to\infty}\frac{f(x_{T+1})-f^\star}{T^{-\beta_p}\ell_T^4}\le D_pRQ. T → ∞ lim sup T − β p ℓ T 4 f ( x T + 1 ) − f ⋆ ≤ D p R Q .
Proof of Theorem 5
The proof reuses the radius event and the suffix potential of Theorem 2, with the fixed time as the analysis horizon. For r > β p r>\beta_p r > β p , the range of the late clipping errors, weighted by the potential, grows with the horizon, so Lemma 15 does not apply directly. We truncate these errors at deterministic levels that depend on T T T and δ \delta δ , apply the lemma on the event that no truncation occurs, and bound the probability of truncation through the p p p th moment. The factor δ u \delta^u δ u shortens the steps and lowers the thresholds, which postpones truncation and makes it less likely; the shorter steps cost the factor δ − u \delta^{-u} δ − u .
We keep the normalized setting and the constants (43) , fix r ∈ ( β p , α p ] r\in(\beta_p,\alpha_p] r ∈ ( β p , α p ] and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and write h = log ( 100 / δ ) h=\log(100/\delta) h = log ( 100/ δ ) and ℓ T = log ( 100 T 2 / δ ) \ell_T=\log(100T^2/\delta) ℓ T = log ( 100 T 2 / δ ) . Put
a r = 1 − r , b r = r p − 1 , d r = a r − b r = p p − 1 ( α p − r ) ≥ 0 , a_r=1-r,\qquad b_r=\frac{r}{p-1},\qquad d_r=a_r-b_r=\frac{p}{p-1}(\alpha_p-r)\ge0, a r = 1 − r , b r = p − 1 r , d r = a r − b r = p − 1 p ( α p − r ) ≥ 0 ,
κ = 3 p − 1 p − 1 , E = κ ( r − β p ) > 0. \kappa=\frac{3p-1}{p-1},\qquad E=\kappa(r-\beta_p)>0. κ = p − 1 3 p − 1 , E = κ ( r − β p ) > 0.
Since κ β p = 2 \kappa\beta_p=2 κ β p = 2 , also E = 2 b r + r − 2 a r E=2b_r+r-2a_r E = 2 b r + r − 2 a r , and
p p − 1 E + κ d r = p κ p − 1 ( α p − β p ) = 1. (51) \frac{p}{p-1}E+\kappa d_r=\frac{p\kappa}{p-1}(\alpha_p-\beta_p)=1. \tag*{(51)} p − 1 p E + κ d r = p − 1 p κ ( α p − β p ) = 1. (51)
For u , w ≥ 0 u,w \ge0 u , w ≥ 0 let
Ξ ( u , w ) = p u p − 1 + 2 w + d r E ( κ u + 2 w ) . \Xi(u,w)=\frac{pu}{p-1}+2w+\frac{d_r}{E}(\kappa u+2w). Ξ ( u , w ) = p − 1 p u + 2 w + E d r ( κ u + 2 w ) .
We prove the following claim: if Ξ ( u , w ) ≥ 1 \Xi(u,w)\ge1 Ξ ( u , w ) ≥ 1 , then the schedule (10) with this u u u satisfies, for every deterministic T ≥ 1 T\ge1 T ≥ 1 ,
P { f ( x T + 1 ) − f ⋆ ≤ D p T − r δ − ( u + 2 w ) ℓ T 4 } ≥ 1 − δ . (52) \mathbb{P}\left\{f(x_{T+1})-f^\star\le D_pT^{-r}\delta^{-(u+2w)}\ell_T^4\right\}\ge1-\delta. \tag*{(52)} P { f ( x T + 1 ) − f ⋆ ≤ D p T − r δ − ( u + 2 w ) ℓ T 4 } ≥ 1 − δ . (52)
The parameter w w w enters only the analysis. For u = E = c i n ( r ) u=E=c_{\mathrm{in}}(r) u = E = c in ( r ) and w = 0 w=0 w = 0 , (51) gives Ξ = 1 \Xi=1 Ξ = 1 . For u = 0 u=0 u = 0 and 2 w = E / ( E + d r ) 2w=E/(E+d_r) 2 w = E / ( E + d r ) , Ξ = 2 w ( 1 + d r / E ) = 1 \Xi=2w(1+d_r/E)=1 Ξ = 2 w ( 1 + d r / E ) = 1 , and
E E + d r = ( 3 p − 1 ) r − 2 ( p − 1 ) ( 2 p − 1 ) r − ( p − 1 ) = c f r e e ( r ) . \frac{E}{E+d_r}=\frac{(3p-1)r-2(p-1)}{(2p-1)r-(p-1)}=c_{\mathrm{free}}(r). E + d r E = ( 2 p − 1 ) r − ( p − 1 ) ( 3 p − 1 ) r − 2 ( p − 1 ) = c free ( r ) .
Thus the claim proves Theorem 5 in the normalized setting, and the rescaling of Appendix § 18 .§ 5 multiplies η t \eta_t η t by R / Q R/Q R / Q , λ t \lambda_t λ t by Q Q Q , and the bound by R Q RQ R Q .
Lemma 24 (Budgets for the fixed-time schedules). Let u ≥ 0 u\ge0 u ≥ 0 , 0 < c ≤ 1 0<c\le1 0 < c ≤ 1 , and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , and let η t = c δ u t − a r / ℓ t 2 \eta_t=c\delta^u t^{-a_r}/\ell_t^2 η t = c δ u t − a r / ℓ t 2 and λ t = 4 max { 1 , δ u / ( p − 1 ) t b r } \lambda_t=4\max\{1,\delta^{u/(p-1)}t^{b_r}\} λ t = 4 max { 1 , δ u / ( p − 1 ) t b r } , with B t B_t B t and V t V_t V t as in (41) . Then (46) holds. Fix T ≥ 2 T\ge2 T ≥ 2 and an integer 1 ≤ k ≤ T / 2 1\le k\le T/2 1 ≤ k ≤ T /2 such that λ t = 4 δ u / ( p − 1 ) t b r \lambda_t=4\delta^{u/(p-1)}t^{b_r} λ t = 4 δ u / ( p − 1 ) t b r for all t ≥ k t\ge k t ≥ k , and put S t = ∑ s = t T η s S_t=\sum_{s=t}^T\eta_s S t = ∑ s = t T η s , W = 1 + S k W=1+S_k W = 1 + S k , and Q t = W η t 2 / ( 1 + S t ) Q_t=W\eta_t^2/(1+S_t) Q t = W η t 2 / ( 1 + S t ) . Then
c 2 δ u T r ℓ T 2 ≤ S k ≤ S 1 ≤ H β c δ u T r ℓ T 2 , (53) \frac{c}{2}\frac{\delta^{uT^r}}{\ell_T^2}\le S_k\le S_1\le H_\beta c\frac{\delta^{uT^r}}{\ell_T^2}, \tag*{(53)} 2 c ℓ T 2 δ u T r ≤ S k ≤ S 1 ≤ H β c ℓ T 2 δ u T r , (53) ∑ t = k T Q t B t ≤ K c 2 ℓ 1 , ∑ t = k T Q t V t ≤ K c 2 ℓ 1 3 , max k ≤ t ≤ T / 2 Q t λ t 2 ≤ K c 2 ℓ 1 4 , (54) \sum_{t=k}^{T}\sqrt{Q_t}B_t\le\frac{Kc}{2\ell_1},\qquad
\sum_{t=k}^{T}Q_tV_t\le\frac{Kc^2}{\ell_1^3},\qquad
\max_{k\le t\le T/2}Q_t\lambda_t^2\le\frac{Kc^2}{\ell_1^4}, \tag*{(54)} t = k ∑ T Q t B t ≤ 2 ℓ 1 K c , t = k ∑ T Q t V t ≤ ℓ 1 3 K c 2 , k ≤ t ≤ T /2 max Q t λ t 2 ≤ ℓ 1 4 K c 2 , (54) ∑ T / 2 < t ≤ T Q t V t ≤ 128 H β 2 c 2 ℓ T 3 δ p u / ( p − 1 ) T − d r . (55) \sum_{T/2<t\le T}Q_tV_t\le\frac{128H_\beta^2c^2}{\ell_T^3}\delta^{pu/(p-1)}T^{-d_r}. \tag*{(55)} T /2 < t ≤ T ∑ Q t V t ≤ ℓ T 3 128 H β 2 c 2 δ p u / ( p − 1 ) T − d r . (55)
Proof. In both branches of λ t \lambda_t λ t , B t ≤ 2 δ − u t − r B_t\le2\delta^{-u}t^{-r} B t ≤ 2 δ − u t − r and V t ≤ 8 max { 1 , δ ( 2 − p ) u / ( p − 1 ) t ( 2 − p ) b r } V_t\le8\max\{1,\delta^{(2-p)u/(p-1)}t^{(2-p)b_r}\} V t ≤ 8 max { 1 , δ ( 2 − p ) u / ( p − 1 ) t ( 2 − p ) b r } . Since a r + r = 1 a_r+r=1 a r + r = 1 , d r ≥ 0 d_r\ge0 d r ≥ 0 , 2 a r ≥ 1 2a_r\ge1 2 a r ≥ 1 , and 2 a r − ( 2 − p ) b r = 1 + d r 2a_r-(2-p)b_r=1+d_r 2 a r − ( 2 − p ) b r = 1 + d r ,
η t B t ≤ 2 c t ℓ t 2 , η t 2 V t ≤ 8 c 2 t ℓ t 4 , η t λ t ≤ 4 c ℓ 1 2 , \eta_tB_t\le\frac{2c}{t\ell_t^2},\qquad
\eta_t^2V_t\le\frac{8c^2}{t\ell_t^4},\qquad
\eta_t\lambda_t\le\frac{4c}{\ell_1^2}, η t B t ≤ t ℓ t 2 2 c , η t 2 V t ≤ t ℓ t 4 8 c 2 , η t λ t ≤ ℓ 1 2 4 c ,
and integral comparison, as in Lemma 23, gives (46) . The proof of (45) gives (53) : S k ≥ ( T / 2 ) η T S_k\ge(T/2)\eta_T S k ≥ ( T /2 ) η T , and every dyadic block has ∑ t − a r ≤ 2 T r 2 − r j ≤ 2 T r 2 − β p j \sum t^{-a_r}\le2T^r2^{-rj}\le2T^r2^{-\beta pj} ∑ t − a r ≤ 2 T r 2 − r j ≤ 2 T r 2 − β p j . As there, W / ( 1 + S t ) ≤ 4 H β W/(1+S_t)\le4H_\beta W / ( 1 + S t ) ≤ 4 H β for t ≤ T / 2 t\le T/2 t ≤ T /2 , so the global budgets bound the early parts of the bias and variance sums in (54) by 4 H β c / ℓ 1 4\sqrt{H_\beta}c/\ell_1 4 H β c / ℓ 1 and 64 H β c 2 / ℓ 1 4 64H_\beta c^2/\ell_1^4 64 H β c 2 / ℓ 1 4 , and the early range by 64 H β c 2 / ℓ 1 4 64H_\beta c^2/\ell_1^4 64 H β c 2 / ℓ 1 4 . For T / 2 < t ≤ T T/2<t\le T T /2 < t ≤ T , the branch assumption, ℓ T / ℓ t ≤ 2 \ell_T/\ell_t\le2 ℓ T / ℓ t ≤ 2 , and − a r + ( 2 − p ) b r = − r − d r -a_r+(2-p)b_r=-r-d_r − a r + ( 2 − p ) b r = − r − d r give
B t ≤ 4 δ − u T − r , η t V t ≤ 64 c δ u / ( p − 1 ) T − r − d r ℓ T 2 . B_t\le4\delta^{-u}T^{-r},\qquad
\eta_tV_t\le\frac{64c\delta^{u/(p-1)}T^{-r-d_r}}{\ell_T^2}. B t ≤ 4 δ − u T − r , η t V t ≤ ℓ T 2 64 c δ u / ( p − 1 ) T − r − d r .
By (48) , with W W W in place of U U U and the sums starting at k k k , the late bias sum is at most 8 δ − u T − r S k ≤ 8 H β c / ℓ T 2 8\delta^{-u}T^{-r}S_k\le8H_\beta c/\ell_T^2 8 δ − u T − r S k ≤ 8 H β c / ℓ T 2 . For the late variance, W log W ≤ S k ( 1 + log W ) W\log W\le S_k(1+\log W) W log W ≤ S k ( 1 + log W ) and 1 + log W ≤ H β ℓ T 1+\log W\le H_\beta\ell_T 1 + log W ≤ H β ℓ T give
∑ T / 2 < t ≤ T Q t V t ≤ 64 c δ u / ( p − 1 ) T − r − d r ℓ T 2 W log W ≤ 64 H β 2 c 2 ℓ T 3 δ p u / ( p − 1 ) T − d r . \sum_{T/2<t\le T}Q_tV_t
\le\frac{64c\delta^{u/(p-1)}T^{-r-d_r}}{\ell_T^2}W\log W
\le\frac{64H_\beta^2c^2}{\ell_T^3}\delta^{pu/(p-1)}T^{-d_r}. T /2 < t ≤ T ∑ Q t V t ≤ ℓ T 2 64 c δ u / ( p − 1 ) T − r − d r W log W ≤ ℓ T 3 64 H β 2 c 2 δ p u / ( p − 1 ) T − d r .
Adding both halves and using K = 2 20 H β 4 K=2^{20}H_\beta^4 K = 2 20 H β 4 proves (54) and (55) . ■
Radius and early times. Use c = c p c=c_p c = c p . Since λ t ≥ 4 \lambda_t \ge4 λ t ≥ 4 and η t ≤ c p \eta_t \le c_p η t ≤ c p , the argument of Appendix § 18 .§ 2 , with the global budgets of Lemma 24 , gives an event R δ \mathcal{R}_\delta R δ with P ( R δ c ) ≤ δ / 100 \mathbb{P}(\mathcal{R}_\delta^c) \le\delta/100 P ( R δ c ) ≤ δ /100 on which ∥ x t − x ⋆ ∥ ≤ h \lVert x_t-x^\star\rVert\le h ∥ x t − x ⋆ ∥ ≤ h and G t ≤ h 2 / 2 G_t \le h^2/2 G t ≤ h 2 /2 for all t ≥ 1 t \ge1 t ≥ 1 . Let k = ⌈ δ − u / r h 1 / b r ⌉ k=\lceil\delta^{-u/r}h^{1/b_r}\rceil k = ⌈ δ − u / r h 1/ b r ⌉ . Since b r / r = 1 / ( p − 1 ) b_r/r=1/(p-1) b r / r = 1/ ( p − 1 ) , every t ≥ k t\ge k t ≥ k has δ u / ( p − 1 ) t b r ≥ h ≥ 1 \delta^{u/(p-1)t^{b_r}}\ge h\ge1 δ u / ( p − 1 ) t b r ≥ h ≥ 1 , hence λ t = 4 δ u / ( p − 1 ) t b r ≥ 4 h \lambda_t=4\delta^{u/(p-1)t^{b_r}}\ge4h λ t = 4 δ u / ( p − 1 ) t b r ≥ 4 h . If T < 2 k T<2k T < 2 k , then T < 4 δ − u / r h 1 / b r T<4\delta^{-u/r}h^{1/b_r} T < 4 δ − u / r h 1/ b r , and on R δ \mathcal{R}_\delta R δ , using r ≤ 1 / 2 r\le1/2 r ≤ 1/2 , r / b r = p − 1 r/b_r=p-1 r / b r = p − 1 , and p ≤ 2 p\le2 p ≤ 2 ,
δ u T r G T + 1 ≤ 1 2 T r h p + 1 ≤ h 3 ≤ ι T 4 \delta^{u}T^{r}G_{T+1}\le\frac{1}{2}T^{r}h^{p+1}\le h^3\le\iota_T^4 δ u T r G T + 1 ≤ 2 1 T r h p + 1 ≤ h 3 ≤ ι T 4
which proves (52) , since D p ≥ 1 D_p\ge1 D p ≥ 1 and δ − 2 w ≥ 1 \delta^{-2w}\ge1 δ − 2 w ≥ 1 .
Truncation. Fix a deterministic T ≥ 2 k T\ge2k T ≥ 2 k , and apply Lemma 11 with ζ = 1 \zeta=1 ζ = 1 to the updates k , … , T k,\ldots,T k , … , T as in Appendix § 18 .§ 3 . This gives P ~ m = W A m + H m ≥ 0 \widetilde{P}_m=W A_m+H_m\ge0 P m = W A m + H m ≥ 0 with W = 1 + S k W=1+S_k W = 1 + S k , the recursion (50) , and ∥ w t ∥ ≤ 2 Q t P t − 1 \lVert w_t\rVert\le\sqrt{2Q_t}\sqrt{P_{t-1}} ∥ w t ∥ ≤ 2 Q t P t − 1 , where Q t = Q t ( k ) Q_t=Q_t^{(k)} Q t = Q t ( k ) is the quantity of Lemma 24 . Let J t J_t J t indicate that ∥ x s − x ⋆ ∥ ≤ 2 h \lVert x_s-x^\star\rVert\le2h ∥ x s − x ⋆ ∥ ≤ 2 h for k ≤ s ≤ t k\le s\le t k ≤ s ≤ t ; it is predictable and nonincreasing. On J t = 1 J_t=1 J t = 1 , ∥ ∇ f ( x t ) ∥ ≤ 2 h ≤ λ t / 2 \lVert\nabla f(x_t)\rVert\le2h\le\lambda_t/2 ∥ ∇ f ( x t )∥ ≤ 2 h ≤ λ t /2 , so Lemma 12 gives (41) and ∥ e t ∥ ≤ ∥ g t − ∇ f ( x t ) ∥ \lVert e_t\rVert\le\lVert g_t-\nabla f(x_t)\rVert ∥ e t ∥ ≤ ∥ g t − ∇ f ( x t )∥ . Set M 0 = K c p / ℓ 1 2 M_0=\sqrt{K}c_p/\ell_1^2 M 0 = K c p / ℓ 1 2 , and define deterministic levels and truncated errors by
m t = ∞ ( k ≤ t ≤ T / 2 ) , m t = M 0 δ − w Q t ( T / 2 < t ≤ T ) , e ^ t = e t 1 { ∥ e t ∥ ≤ m t } . m_t=\infty\quad(k\le t\le T/2),\qquad m_t=\frac{M_0\delta^{-w}}{\sqrt{Q_t}}\quad(T/2<t\le T),\qquad\widehat e_t=e_t\mathbf{1}_{\{\lVert e_t\rVert\le m_t\}}. m t = ∞ ( k ≤ t ≤ T /2 ) , m t = Q t M 0 δ − w ( T /2 < t ≤ T ) , e t = e t 1 {∥ e t ∥ ≤ m t } .
Let T = { t : m t < 3 λ t / 2 } \mathcal{T}=\{t:m_t<3\lambda_t/2\} T = { t : m t < 3 λ t /2 } and E T = { J t e t = J t e ^ t for k ≤ t ≤ T } \mathcal{E}_{T}=\{J_t e_t=J_t\widehat e_t\text{ for }k\le t\le T\} E T = { J t e t = J t e t for k ≤ t ≤ T } . Only times in T \mathcal{T} T can violate E T \mathcal{E}_T E T , and the conditional Markov inequality with E t − 1 ∥ g t − ∇ f ( x t ) ∥ p ≤ 1 \mathbb{E}_{t-1}\lVert g_t-\nabla f(x_t)\rVert^p\le1 E t − 1 ∥ g t − ∇ f ( x t ) ∥ p ≤ 1 gives
P ( E T c ) ≤ Σ T : = ∑ t ∈ T m t − p . \mathbb{P}(\mathcal{E}_T^c)\le\Sigma_T:=\sum_{t\in\mathcal{T}}m_t^{-p}. P ( E T c ) ≤ Σ T := t ∈ T ∑ m t − p .
Suppose T ≠ ∅ \mathcal{T}\ne\varnothing T = ∅ . Since δ u T r ≥ δ u k r ≥ 1 \delta^uT^r\ge\delta^u k^r\ge1 δ u T r ≥ δ u k r ≥ 1 and H β c p ≤ 1 H_\beta c_p\le1 H β c p ≤ 1 , (53) gives W ≤ 2 δ u T r W\le2\delta^uT^r W ≤ 2 δ u T r . For T / 2 < t ≤ T T/2<t\le T T /2 < t ≤ T , also η t ≤ 8 c p δ u T − a r / ℓ T 2 \eta_t\le8c_p\delta^{uT^{-a_r}}/\ell_T^2 η t ≤ 8 c p δ u T − a r / ℓ T 2 and λ t ≤ 4 δ u / ( p − 1 ) T b r \lambda_t\le4\delta^{u/(p-1)T^{b_r}} λ t ≤ 4 δ u / ( p − 1 ) T b r , so
Q t λ t 2 ≤ W η t 2 λ t 2 ≤ 2048 c p 2 ℓ T 4 δ κ u T E . Q_t\lambda_t^2\le W\eta_t^2\lambda_t^2\le\frac{2048c_p^2}{\ell_T^4}\delta^{\kappa uT}E. Q t λ t 2 ≤ W η t 2 λ t 2 ≤ ℓ T 4 2048 c p 2 δ κ u T E .
A time t ∈ T t\in\mathcal{T} t ∈ T has M 0 2 δ − 2 w < 9 4 Q t λ t 2 M_0^2\delta^{-2w}<\frac94Q_t\lambda_t^2 M 0 2 δ − 2 w < 4 9 Q t λ t 2 , so K ≥ 4608 K\ge4608 K ≥ 4608 gives
T E > δ − ( κ u + 2 w ) (56) T^E>\delta^{-(\kappa u+2w)} \tag*{(56)} T E > δ − ( κ u + 2 w ) (56)
For t ∈ T t\in\mathcal{T} t ∈ T , m t < 3 λ t / 2 m_t<3\lambda_t/2 m t < 3 λ t /2 and 0 ≤ 2 − p < 1 0\le2-p<1 0 ≤ 2 − p < 1 give m t 2 − p ≤ 3 4 V t m_t^{2-p}\le\frac34V_t m t 2 − p ≤ 4 3 V t , hence m t − p = δ 2 w Q t m t − p / M 0 2 ≤ 3 4 δ 2 w Q t V t / M 0 2 m_t^{-p}=\delta^{2w}Q_tm_t^{-p}/M_0^2\le\frac34\delta^{2w}Q_tV_t/M_0^2 m t − p = δ 2 w Q t m t − p / M 0 2 ≤ 4 3 δ 2 w Q t V t / M 0 2 . By (55) , d r ≥ 0 d_r\ge0 d r ≥ 0 , and (56) ,
Σ T ≤ 96 H β 2 ℓ 1 2 K δ p u p − 1 + 2 w T − d r ≤ 96 H β 2 ℓ 1 2 K δ Ξ ( u , w ) ≤ δ 2 . \Sigma_T\le\frac{96H_\beta^2\ell_1^2}{K}\delta^{\frac{pu}{p-1}+2wT-d_r}\le\frac{96H_\beta^2\ell_1^2}{K}\delta^{\Xi(u,w)}\le\frac{\delta}{2}. Σ T ≤ K 96 H β 2 ℓ 1 2 δ p − 1 p u + 2 w T − d r ≤ K 96 H β 2 ℓ 1 2 δ Ξ ( u , w ) ≤ 2 δ .
If T = ∅ \mathcal{T}=\varnothing T = ∅ , then Σ T = 0 \Sigma_T=0 Σ T = 0 .
Energy bound without truncation. On J t = 1 J_t=1 J t = 1 , the truncated errors satisfy
∥ E t − 1 e ^ t ∥ ≤ B t + 1 { t ∈ T } m t 1 − p , E t − 1 ∥ e ^ t ∥ 2 ≤ V t , \lVert\mathbb{E}_{t-1}\widehat e_t\rVert\le B_t+\mathbf{1}_{\{t\in\mathcal{T}\}}m_t^{1-p},\qquad\mathbb{E}_{t-1}\lVert\widehat e_t\rVert^2\le V_t, ∥ E t − 1 e t ∥ ≤ B t + 1 { t ∈ T } m t 1 − p , E t − 1 ∥ e t ∥ 2 ≤ V t ,
∥ e ^ t ∥ ≤ min { λ t 2 , m t } , ∥ e ^ t − E t − 1 e ^ t ∥ ≤ 2 min { λ t , m t } . \lVert\widehat e_t\rVert\le\min\left\{\frac{\lambda_t}{2},m_t\right\},\qquad\lVert\widehat e_t-\mathbb{E}_{t-1}\widehat e_t\rVert\le2\min\{\lambda_t,m_t\}. ∥ e t ∥ ≤ min { 2 λ t , m t } , ∥ e t − E t − 1 e t ∥ ≤ 2 min { λ t , m t } .
The first bound uses ∥ e t ∥ 1 { ∥ e t ∥ > m t } ≤ ∥ g t − ∇ f ( x t ) ∥ p m t 1 − p \lVert e_t\rVert\mathbf{1}_{\{\lVert e_t\rVert>m_t\}}\le\lVert g_t-\nabla f(x_t)\rVert^p m_t^{1-p} ∥ e t ∥ 1 {∥ e t ∥ > m t } ≤ ∥ g t − ∇ f ( x t ) ∥ p m t 1 − p . For the last, e ^ t + ∇ f ( x t ) \widehat e_t+\nabla f(x_t) e t + ∇ f ( x t ) equals clip ( g t , λ t ) \operatorname{clip}(g_t,\lambda_t) clip ( g t , λ t ) or ∇ f ( x t ) \nabla f(x_t) ∇ f ( x t ) , both in the ball of radius λ t \lambda_t λ t , and ∥ e ^ t ∥ ≤ m t \lVert\widehat e_t\rVert\le m_t ∥ e t ∥ ≤ m t . Put P ~ m = δ 2 w P m \widetilde P_m=\delta^{2w}P_m P m = δ 2 w P m . On E T ∩ { J m = 1 } \mathcal{E}_T\cap\{J_m=1\} E T ∩ { J m = 1 } , the recursion (50) holds with e ^ t \widehat e_t e t in place of e t e_t e t , so
P ~ m ≤ P ~ k − 1 + ∑ t = k m ( 2 δ 2 w Q t ∥ e ^ t ∥ 2 + ⟨ e ^ t , δ 2 w w t ⟩ ) , ∥ δ 2 w w t ∥ ≤ δ w 2 Q t P ~ t − 1 . \widetilde P_m\le\widetilde P_{k-1}+\sum_{t=k}^{m}\left(2\delta^{2w}Q_t\lVert\widehat e_t\rVert^2+\langle\widehat e_t,\delta^{2w}w_t\rangle\right),\qquad\lVert\delta^{2w}w_t\rVert\le\delta^w\sqrt{2Q_t}\sqrt{\widetilde P_{t-1}}. P m ≤ P k − 1 + t = k ∑ m ( 2 δ 2 w Q t ∥ e t ∥ 2 + ⟨ e t , δ 2 w w t ⟩ ) , ∥ δ 2 w w t ∥ ≤ δ w 2 Q t P t − 1 .
Apply Lemma 15 with the event E T \mathcal{E}_{T} E T , the errors e ^ t \hat{e}_{t} e ^ t , a t = 2 δ 2 w Q t a_{t}=2\delta^{2w}Q_{t} a t = 2 δ 2 w Q t , s t = δ w 2 Q t s_{t}=\delta^{w}\sqrt{2Q_{t}} s t = δ w 2 Q t , p 0 = h 2 / 2 p_{0}=h^{2}/2 p 0 = h 2 /2 , and exponent q = ι T q=\iota_{T} q = ι T . Lemma 24, Σ T ≤ 1 / 2 \Sigma_{T}\leq1/2 Σ T ≤ 1/2 , and the identities δ w Q t m t 1 − p = M 0 m t − p \delta^{w}\sqrt{Q_{t}}m_{t}^{1-p}=M_{0}m_{t}^{-p} δ w Q t m t 1 − p = M 0 m t − p and δ 2 w Q t m t 2 = M 0 2 \delta^{2w}Q_{t}m_{t}^{2}=M_{0}^{2} δ 2 w Q t m t 2 = M 0 2 for t > T / 2 t>T/2 t > T /2 give
A ≤ 2 K c p 2 ℓ 1 3 , K 0 ≤ 9 K c p 2 2 ℓ 1 4 , B 0 ≤ 2 K c p ℓ 1 , V 0 ≤ 2 K c p 2 ℓ 1 3 , J 0 ≤ 2 2 K c p ℓ 1 2 . A\leq\frac{2Kc_{p}^{2}}{\ell_{1}^{3}},\qquad K_{0}\leq\frac{9Kc_{p}^{2}}{2\ell_{1}^{4}},\qquad B_{0}\leq\frac{\sqrt{2}Kc_{p}}{\ell_{1}},\qquad V_{0}\leq\frac{2Kc_{p}^{2}}{\ell_{1}^{3}},\qquad J_{0}\leq\frac{2\sqrt{2}Kc_{p}}{\ell_{1}^{2}}. A ≤ ℓ 1 3 2 K c p 2 , K 0 ≤ 2 ℓ 1 4 9 K c p 2 , B 0 ≤ ℓ 1 2 K c p , V 0 ≤ ℓ 1 3 2 K c p 2 , J 0 ≤ ℓ 1 2 2 2 K c p .
Hence C ≤ h 2 / 2 + 17 K c p 2 q C\leq h^{2}/2+17Kc_{p}^{2}q C ≤ h 2 /2 + 17 K c p 2 q and D ≤ 11 K c p q D\leq11Kc_{p}q D ≤ 11 K c p q , so C + D ≤ ( 1 / 2 + 16 K c p ) q < q \sqrt{C}+D\leq(1/\sqrt{2}+16Kc_{p})q<q C + D ≤ ( 1/ 2 + 16 K c p ) q < q , as in Appendix J.3. On R δ \mathcal{R}_{\delta} R δ , P ~ k − 1 ≤ h 2 / 2 \widetilde{P}_{k-1}\leq h^{2}/2 P k − 1 ≤ h 2 /2 and every J t = 1 J_{t}=1 J t = 1 . Thus, except on probability e − ι T ≤ δ / 100 e^{-\iota_{T}}\leq\delta/100 e − ι T ≤ δ /100 , every outcome in R δ ∩ E T \mathcal{R}_{\delta}\cap\mathcal{E}_{T} R δ ∩ E T has P ~ T ≤ ι T \widetilde{P}_{T}\leq\iota_{T} P T ≤ ι T .
Conclusion. On R δ ∩ E T ∩ { P ~ T ≤ ι T } \mathcal{R}_{\delta}\cap\mathcal{E}_{T}\cap\{\widetilde{P}_{T}\leq\iota_{T}\} R δ ∩ E T ∩ { P T ≤ ι T } , the endpoint estimate (25) , W ≥ S k W\geq S_{k} W ≥ S k , and (53) give
G T + 1 ≤ 2 P T W ≤ 2 δ 2 − 2 w ι T 2 ( c p / 2 ) δ u T r / ℓ T 2 ≤ D p T − r δ − ( u + 2 w ) ι T 4 , G_{T+1}\leq\frac{2P_{T}}{W}\leq\frac{2\delta^{2-2w}\iota_{T}^{2}}{(c_{p}/2)\delta^{u}Tr/\ell_{T}^{2}}\leq D_{p}T^{-r}\delta^{-(u+2w)}\iota_{T}^{4}, G T + 1 ≤ W 2 P T ≤ ( c p /2 ) δ u T r / ℓ T 2 2 δ 2 − 2 w ι T 2 ≤ D p T − r δ − ( u + 2 w ) ι T 4 ,
since ℓ T ≤ ι T \ell_{T}\leq\iota_{T} ℓ T ≤ ι T and D p = 4 / c p D_{p}=4/c_{p} D p = 4/ c p . The three exceptional events have total probability at most δ / 100 + δ / 2 + δ / 100 < δ \delta/100+\delta/2+\delta/100<\delta δ /100 + δ /2 + δ /100 < δ , which proves (52) .
Proof of the One-Dimensional Momentum Lower Bound
We prove Theorem 7 for rules that follow the heavy-ball recursion (14) under the bounds (15) , with clip ( g , 0 ) = 0 \operatorname{clip}(g,0)=0 clip ( g , 0 ) = 0 . Such rules observe only stochastic gradients and output their current iterate; their clipping thresholds, momentum coefficients (including β t = 0 \beta_{t}=0 β t = 0 ), and step sizes may all adapt. Under (14) , the clipped gradient clip ( g j , λ j ) \operatorname{clip}(g_{j},\lambda_{j}) clip ( g j , λ j ) computed at step j j j enters the velocity v t + 1 v_{t+1} v t + 1 with weight ∏ i = j + 1 t β i \prod_{i=j+1}^{t}\beta_{i} ∏ i = j + 1 t β i and the iterate x t + 1 x_{t+1} x t + 1 with weight − ∑ s = j t η s ∏ i = s + 1 t β i -\sum_{s=j}^{t}\eta_{s}\prod_{i=s+1}^{t}\beta_{i} − ∑ s = j t η s ∏ i = s + 1 t β i . For an integer N ≥ 1 N\geq1 N ≥ 1 and k ≥ 0 k\geq0 k ≥ 0 , put
κ N , k = exp ( − ρ ( k − t 0 ) + ( 1 + N ) − α ) . \kappa_{N,k}=\exp\left(-\rho(k-t_{0})_{+}(1+N)^{-\alpha}\right). κ N , k = exp ( − ρ ( k − t 0 ) + ( 1 + N ) − α ) .
Proposition 25 (Necessary momentum progress budget). Fix such a rule and a failure budget 0 < ε < 1 / 2 0<\varepsilon<1/2 0 < ε < 1/2 . Let N ≥ 1 N\geq1 N ≥ 1 and let b 1 , … , b N b_{1},\ldots,b_{N} b 1 , … , b N be positive. Suppose every instance in the class of Theorem 7 satisfies
P { f ( x t + 1 ) − f ⋆ ≤ b t for all 1 ≤ t ≤ N } ≥ 1 − ε . \mathbb{P}\{f(x_{t+1})-f^{\star}\leq b_{t}\text{ for all }1\leq t\leq N\}\geq1-\varepsilon. P { f ( x t + 1 ) − f ⋆ ≤ b t for all 1 ≤ t ≤ N } ≥ 1 − ε . For every 4 b N < μ ≤ 1 / 2 4b_{N}<\mu\leq1/2 4 b N < μ ≤ 1/2 , put A = ( 2 μ ) − 1 / ( p − 1 ) A=(2\mu)^{-1/(p-1)} A = ( 2 μ ) − 1/ ( p − 1 ) , π = μ / A \pi=\mu/A π = μ / A , s t = μ + 2 b t s_{t}=\mu+\sqrt{2b_{t}} s t = μ + 2 b t , and H N = H ( 1 + N ) h H_{N}=H(1+N)^{h} H N = H ( 1 + N ) h . Then
1 − 2 ε 2 ≤ π ∑ t = 1 N ∑ k = 0 t − 1 min { s t , H N A κ N , k } . (57) \frac{1-2\varepsilon}{2}\leq\pi\sum_{t=1}^{N}\sum_{k=0}^{t-1}\min\{s_{t},H_{N}A\kappa_{N,k}\}. \tag*{(57)} 2 1 − 2 ε ≤ π t = 1 ∑ N k = 0 ∑ t − 1 min { s t , H N A κ N , k } . (57)
Proof. Use the translated family of Appendix H.1, fixed before the run:
F μ , y ( x ; Y ) = 1 2 ( x − y + μ ) + 2 − A Y x , y ∈ [ μ , 1 ] , Y ∼ Bernoulli ( π ) . F_{\mu,y}(x;Y)=\frac{1}{2}(x-y+\mu)_{+}^{2}-AYx,\qquad y\in[\mu,1],\qquad Y\sim\operatorname{Bernoulli}(\pi). F μ , y ( x ; Y ) = 2 1 ( x − y + μ ) + 2 − A Y x , y ∈ [ μ , 1 ] , Y ∼ Bernoulli ( π ) .
The population objective gap is given by (28) , and for x ≤ y − μ x\leq y-\mu x ≤ y − μ the stochastic gradient is − A Y -AY − A Y . In particular, all sample and population properties stated in the theorem hold, with E ∣ μ − A Y ∣ p ≤ μ p + μ A p − 1 ≤ 1 \mathbb{E}|\mu-AY|^{p}\leq\mu^{p}+\mu A^{p-1}\leq1 E ∣ μ − A Y ∣ p ≤ μ p + μ A p − 1 ≤ 1 . Thus these instances satisfy the assumed prefix guarantee, and the rule meets the conditions of Lemma 19, with the velocity as auxiliary state. Let X t X_{t} X t and V t V_{t} V t be the position and velocity of its reference run, with c = 1 / 2 c=1/2 c = 1/2 . Before stopping, let R t = − V t R_{t}=-V_{t} R t = − V t and write
R t + 1 = β t R t + Y t a t , a t = min { A , λ t } ∈ [ 0 , A ] , X t + 1 = X t + D t , D t = η t R t + 1 , R_{t+1}=\beta_{t}R_{t}+Y_{t}a_{t},\qquad a_{t}=\min\{A,\lambda_{t}\}\in[0,A],\qquad X_{t+1}=X_{t}+D_{t},\qquad D_{t}=\eta_{t}R_{t+1}, R t + 1 = β t R t + Y t a t , a t = min { A , λ t } ∈ [ 0 , A ] , X t + 1 = X t + D t , D t = η t R t + 1 ,
with X 1 = R 1 = 0 X_1 = R_1 = 0 X 1 = R 1 = 0 . Set D t = 0 D_t = 0 D t = 0 after stopping. Unrolling the recursion gives R t + 1 = ∑ j ≤ t ( ∏ i = j + 1 t β i ) Y j a j R_{t+1} = \sum_{j \le t} \left(\prod_{i=j+1}^{t}\beta_i\right)Y_j a_j R t + 1 = ∑ j ≤ t ( ∏ i = j + 1 t β i ) Y j a j . For t ≤ N t \le N t ≤ N , each index i ≥ t 0 i \ge t_0 i ≥ t 0 in the product satisfies β i ≤ 1 − ρ ( 1 + N ) − a ≤ exp ( − ρ ( 1 + N ) − a ) \beta_i \le1-\rho(1+N)^{-a} \le\exp(-\rho(1+N)^{-a}) β i ≤ 1 − ρ ( 1 + N ) − a ≤ exp ( − ρ ( 1 + N ) − a ) , and at least ( t − j − t 0 ) + (t-j-t_0)_+ ( t − j − t 0 ) + of its indices are at least t 0 t_0 t 0 , so ∏ i = j + 1 t β i ≤ κ N , t − j \prod_{i=j+1}^{t}\beta_i \le\kappa_{N,t-j} ∏ i = j + 1 t β i ≤ κ N , t − j . Hence
0 ≤ D t ≤ H N A ∑ j = 1 t κ N , t − j Y j ( t ≤ N ) . (58) 0 \le D_t \le H_N A \sum_{j=1}^{t}\kappa_{N,t-j}Y_j \quad(t \le N). \tag*{(58)} 0 ≤ D t ≤ H N A j = 1 ∑ t κ N , t − j Y j ( t ≤ N ) . (58)
This induction uses only the signs and pathwise bounds, so it holds for controls that depend on the current gradient and on internal randomness. In particular, the reference run is nondecreasing, so M t = X t M_t = X_t M t = X t and its record increments are D t D_t D t . Thus Lemma 19 and (58) give
c ( 1 − 2 ε ) ≤ E ∑ t = 1 N min { s t , D t } ≤ π ∑ t = 1 N ∑ k = 0 t − 1 min { s t , H N A κ N , k } . c(1-2\varepsilon) \le\mathbb{E}\sum_{t=1}^{N}\min\{s_t,D_t\} \le\pi\sum_{t=1}^{N}\sum_{k=0}^{t-1}\min\{s_t,H_NA\kappa_{N,k}\}. c ( 1 − 2 ε ) ≤ E t = 1 ∑ N min { s t , D t } ≤ π t = 1 ∑ N k = 0 ∑ t − 1 min { s t , H N A κ N , k } .
For the second inequality, use min { s , ∑ j w j } ≤ ∑ j min { s , w j } \min\{s,\sum_j w_j\}\le\sum_j\min\{s,w_j\} min { s , ∑ j w j } ≤ ∑ j min { s , w j } for w j ≥ 0 w_j\ge0 w j ≥ 0 and E Y j = π \mathbb{E}Y_j=\pi E Y j = π . ■
Proof of Theorem 7 . Suppose the asserted envelope holds on every instance. Since a < 1 a<1 a < 1 and β p ≤ 2 / 5 \beta_p\le2/5 β p ≤ 2/5 , we have ( 1 + a ) β p < 1 (1+a)\beta_p<1 ( 1 + a ) β p < 1 ; choose ( 1 + a ) β p < u < min { r , 1 } (1+a)\beta_p<u<\min\{r,1\} ( 1 + a ) β p < u < min { r , 1 } and put b t = b N , t b_t=b_{N,t} b t = b N , t from (31) , with D N = C log m ( e N ) D_N=C\log^m(eN) D N = C log m ( e N ) . This dominates the asserted envelope for t ≤ N t\le N t ≤ N . For sufficiently large N N N , 4 b N < μ ≤ 1 / 2 4b_N<\mu\le1/2 4 b N < μ ≤ 1/2 , and Proposition 25 applies with ε = δ \varepsilon=\delta ε = δ . If H = 0 H=0 H = 0 , the required progress is already impossible. Otherwise split the inner sum at
K N = t 0 + ⌈ ( 1 + N ) a ρ log + ( H N A μ ) ⌉ , log + ( v ) = max { 0 , log v } . K_N=t_0+\left\lceil\frac{(1+N)^a}{\rho}\log_+\left(\frac{H_NA}{\mu}\right)\right\rceil,\qquad\log_+(v)=\max\{0,\log v\}. K N = t 0 + ⌈ ρ ( 1 + N ) a log + ( μ H N A ) ⌉ , log + ( v ) = max { 0 , log v } .
For k ≥ K N k\ge K_N k ≥ K N , H N A κ N , k ≤ μ exp ( − ρ ( k − K N ) ( 1 + N ) − a ) H_NA\kappa_{N,k}\le\mu\exp(-\rho(k-K_N)(1+N)^{-a}) H N A κ N , k ≤ μ exp ( − ρ ( k − K N ) ( 1 + N ) − a ) , and ( 1 − e − x ) − 1 ≤ 1 + x − 1 (1-e^{-x})^{-1}\le1+x^{-1} ( 1 − e − x ) − 1 ≤ 1 + x − 1 for x > 0 x>0 x > 0 , so
∑ k = 0 t − 1 min { s t , H N A κ N , k } ≤ s t ( K N + 1 + ( 1 + N ) a ρ ) . (59) \sum_{k=0}^{t-1}\min\{s_t,H_NA\kappa_{N,k}\}\le s_t\left(K_N+1+\frac{(1+N)^a}{\rho}\right). \tag*{(59)} k = 0 ∑ t − 1 min { s t , H N A κ N , k } ≤ s t ( K N + 1 + ρ ( 1 + N ) a ) . (59)
The factor is O ( N a log ( e N ) ) O(N^a\log(eN)) O ( N a log ( e N )) , with constants depending on H , h , ρ , t 0 , a H,h,\rho,t_0,a H , h , ρ , t 0 , a , since log H N = log H + h log ( 1 + N ) \log H_N=\log H+h\log(1+N) log H N = log H + h log ( 1 + N ) and
A / μ = 2 − 1 / ( p − 1 ) μ − p / ( p − 1 ) . A/\mu=2^{-1/(p-1)}\mu^{-p/(p-1)}. A / μ = 2 − 1/ ( p − 1 ) μ − p / ( p − 1 ) .
By (32) with K N K_N K N replaced by this factor, the right side of (57) is
O ( D N q + 1 / 2 log ( e N ) N 1 + a − u / β p ) , O\left(D_N^{q+1/2}\log(eN)N^{1+a-u/\beta_p}\right), O ( D N q + 1/2 log ( e N ) N 1 + a − u / β p ) ,
which tends to zero because u > ( 1 + a ) β p u>(1+a)\beta_p u > ( 1 + a ) β p ; this contradicts ( 1 − 2 δ ) / 2 > 0 (1-2\delta)/2>0 ( 1 − 2 δ ) /2 > 0 . The contradiction at one finite N N N yields a fixed instance of this class on which (16) holds. ■
The same budget gives the fixed-time analogue, in the sense of Theorem 6 .
Proposition 26 (Fixed-time momentum lower bound). Fix p ∈ ( 1 , 2 ] p\in(1,2] p ∈ ( 1 , 2 ] and a ∈ [ 0 , 1 ) a\in[0,1) a ∈ [ 0 , 1 ) . Let ( r u l e δ ) δ ∈ ( 0 , 1 ) (\mathrm{rule}_\delta)_{\delta\in(0,1)} ( rule δ ) δ ∈ ( 0 , 1 ) satisfy (14) and (15) with ρ \rho ρ , t 0 t_0 t 0 , and h h h independent of δ \delta δ , and with H H H replaced by H δ − h 0 H\delta^{-h_0} H δ − h 0 for constants H , h 0 ≥ 0 H,h_0\ge0 H , h 0 ≥ 0 independent of δ \delta δ . For every finite C > 0 C>0 C > 0 , m ≥ 0 m\ge0 m ≥ 0 , c ∈ [ 0 , 1 − ( 1 + a ) β p ) c\in[0,1-(1+a)\beta_p) c ∈ [ 0 , 1 − ( 1 + a ) β p ) , and r > ( 1 + a ) β p + c r>(1+a)\beta_p+c r > ( 1 + a ) β p + c , there are δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , a deterministic integer t ≥ 1 t\ge1 t ≥ 1 , and a one-dimensional instance with L = R = σ = 1 L=R=\sigma=1 L = R = σ = 1 , fixed before the run and satisfying the conditions of Theorem 7 , such that r u l e δ \mathrm{rule}_\delta rule δ satisfies
P { f ( x t + 1 ) − f ⋆ > C t − r δ − c [ log ( e t / δ ) ] m } > δ . \mathbb{P}\{f(x_{t+1})-f^\star>Ct^{-r}\delta^{-c}[\log(et/\delta)]^m\}>\delta. P { f ( x t + 1 ) − f ⋆ > C t − r δ − c [ log ( e t / δ ) ] m } > δ .
An exponent above ( 1 + a ) β p (1+a)\beta_p ( 1 + a ) β p at a fixed time thus costs a polynomial confidence factor, and with uniformly polylogarithmic dependence on 1 / δ 1/\delta 1/ δ it is unattainable.
Proof of Proposition 26 . Suppose the envelope holds for every δ \delta δ , every deterministic time, and every instance. Since c < 1 − ( 1 + a ) β p c < 1-(1+a)\beta_p c < 1 − ( 1 + a ) β p and r > ( 1 + a ) β p + c r > (1+a)\beta_p+c r > ( 1 + a ) β p + c , choose ( 1 + a ) β p + c < u < min { r , 1 } (1+a)\beta_p+c < u < \min\{r,1\} ( 1 + a ) β p + c < u < min { r , 1 } and put v = u − c > ( 1 + a ) β p v=u-c>(1+a)\beta_p v = u − c > ( 1 + a ) β p . For each deterministic N N N , let δ N \delta_N δ N , D N D_N D N , and b N , t = D N N c t − u b_{N,t}=D_NN^{ct-u} b N , t = D N N c t − u be as in the proof of Theorem 6 ; the envelope of ruleδ N _{\delta_N} δ N is at most b N , t b_{N,t} b N , t for t ≤ N t\le N t ≤ N , and a union bound gives the prefix guarantee of Proposition 25 with failure budget 1 / 4 1/4 1/4 on every instance. On this prefix, η t ≤ H N : = H ( 4 N ) h 0 ( 1 + N ) h \eta_t\le H_N:=H(4N)^{h_0}(1+N)^h η t ≤ H N := H ( 4 N ) h 0 ( 1 + N ) h , and the proof of that proposition uses the step sizes only through this bound, so it applies with μ = 8 b N , N = 8 D N N − v \mu=8b_{N,N}=8D_NN^{-v} μ = 8 b N , N = 8 D N N − v , which lies in ( 4 b N , N , 1 / 2 ] (4b_{N,N},1/2] ( 4 b N , N , 1/2 ] for large N N N . Splitting the inner sums as in (59) , whose factor is again O ( N a log ( e N ) ) O(N^a\log(eN)) O ( N a log ( e N )) because log H N = O ( log ( e N ) ) \log H_N=O(\log(eN)) log H N = O ( log ( e N )) , and summing s t s_t s t as in the proof of Theorem 6 , the right side of (57) is
O ( D N q + 1 / 2 log ( e N ) N 1 + a − v / β p ) , O\left(D_N^{q+1/2}\log(eN)N^{1+a-v/\beta_p}\right), O ( D N q + 1/2 log ( e N ) N 1 + a − v / β p ) ,
which tends to zero because v > ( 1 + a ) β p v>(1+a)\beta_p v > ( 1 + a ) β p ; this contradicts 1 / 4 1/4 1/4 . At such an N N N , some instance fixed before the run fails the prefix guarantee with probability above 1 / 4 = N δ N 1/4=N\delta_N 1/4 = N δ N , and the union bound gives a deterministic t ≤ N t\le N t ≤ N whose fixed-time failure probability exceeds δ N \delta_N δ N . ■
A Dimension-Uniform Momentum Lower Bound
We prove Theorem 8 . Throughout, the control constants are common over dimensions, as required by A u n i f ( a ) \mathcal{A}_{\mathrm{unif}}(a) A unif ( a ) , and so are the rate constants C C C and k k k , as in Theorem 8 . Shared constants match our upper bounds, whose constants are likewise dimension-free, and allow the proof to choose the dimension after fixing the horizon. The proof augments the objective with auxiliary quadratic coordinates, the nuisance block, that carry i.i.d. signed coordinate noise at every round. On a finite prefix with distinct coordinates, an orthogonal test vector bounds the useful displacement that each rare observation accumulates over its whole forgetting window by the allowed nuisance radius, even when the controls depend on the current noisy gradient. This removes the factor O ~ ( N a ) \widetilde{O}(N^a) O ( N a ) of the one-dimensional bound, and polynomial forgetting makes the remaining tail of each spike’s effect negligible.
A Pathwise Projection Identity
Fix the numerical scalar controls on a realized finite path; they may depend on the observations. Write clip ( g s , λ s ) = c s g s \operatorname{clip}(g_s,\lambda_s)=c_sg_s clip ( g s , λ s ) = c s g s , with c s ∈ [ 0 , 1 ] c_s\in[0,1] c s ∈ [ 0 , 1 ] and c s = 0 c_s=0 c s = 0 when g s = 0 g_s=0 g s = 0 . On the nuisance block, where g s g_s g s has component y s + ζ s y_s+\zeta_s y s + ζ s with nuisance noise ζ s \zeta_s ζ s , the components y s y_s y s of the state and v s v_s v s of the velocity satisfy v s + 1 = β s v s + c s ( y s + ζ s ) v_{s+1}=\beta_sv_s+c_s(y_s+\zeta_s) v s + 1 = β s v s + c s ( y s + ζ s ) and y s + 1 = y s − η s v s + 1 y_{s+1}=y_s-\eta_sv_{s+1} y s + 1 = y s − η s v s + 1 , with y 1 = v 1 = 0 y_1=v_1=0 y 1 = v 1 = 0 . An impulse of size A A A at time τ \tau τ has flat response
h t = A c τ ∑ s = τ t − 1 η s ∏ i = τ + 1 s β i ( t > τ ) , h t = 0 ( t ≤ τ ) . (60) h_t=A c_\tau\sum_{s=\tau}^{t-1}\eta_s\prod_{i=\tau+1}^{s}\beta_i\quad(t>\tau),\qquad h_t=0\quad(t\le\tau). \tag*{(60)} h t = A c τ s = τ ∑ t − 1 η s i = τ + 1 ∏ s β i ( t > τ ) , h t = 0 ( t ≤ τ ) . (60)
Nonnegative controls make h t h_t h t nonnegative and nondecreasing.
Lemma 27 (Orthogonal pulse budget). Suppose all ζ j \zeta_j ζ j , 1 ≤ j < t 1\le j<t 1 ≤ j < t , are nonzero and mutually orthogonal, with ζ j = R j ϵ j e J j \zeta_j=R_j\epsilon_j e_{J_j} ζ j = R j ϵ j e J j , R j ≥ B > 0 R_j\ge B>0 R j ≥ B > 0 , signs ϵ j ∈ { − 1 , 1 } \epsilon_j\in\{-1,1\} ϵ j ∈ { − 1 , 1 } , and distinct J j J_j J j . For τ < t \tau<t τ < t with R τ = A R_\tau=A R τ = A , let h t h_t h t be (60) . If ∥ y t ∥ ≤ R \lVert y_t\rVert\le R ∥ y t ∥ ≤ R and R t − τ − 1 ≤ B / 2 R\sqrt{t-\tau-1}\le B/2 R t − τ − 1 ≤ B /2 , then h t ≤ 2 R h_t\le2R h t ≤ 2 R .
Proof. Define the test vector
w = ζ τ A + ∑ j = τ + 1 t − 1 h j R j 2 ζ j . w = \frac{\zeta_\tau}{A} + \sum_{j=\tau+1}^{t-1} \frac{h_j}{R_j^2}\zeta_j. w = A ζ τ + j = τ + 1 ∑ t − 1 R j 2 h j ζ j .
Orthogonality gives ⟨ ζ s , w ⟩ = h s + A 1 { s = τ } \langle\zeta_s,w\rangle=h_s+A\mathbf{1}_{\{s=\tau\}} ⟨ ζ s , w ⟩ = h s + A 1 { s = τ } for s < t s<t s < t . Let k 1 = 0 k_1=0 k 1 = 0 and k s + 1 = β s k s + A c s 1 { s = τ } k_{s+1}=\beta_s k_s+Ac_s\mathbf{1}_{\{s=\tau\}} k s + 1 = β s k s + A c s 1 { s = τ } ; then (60) gives h s + 1 = h s + η s k s + 1 h_{s+1}=h_s+\eta_s k_{s+1} h s + 1 = h s + η s k s + 1 . Induction in the nuisance recurrence yields ⟨ v s , w ⟩ = k s \langle v_s,w\rangle=k_s ⟨ v s , w ⟩ = k s and ⟨ y s , w ⟩ = − h s \langle y_s,w\rangle=-h_s ⟨ y s , w ⟩ = − h s for s ≤ t s\le t s ≤ t : the terms − h s + h s -h_s+h_s − h s + h s cancel at every update. Therefore
h t 2 ≤ ∥ y t ∥ 2 ∥ w ∥ 2 = ∥ y t ∥ 2 ( 1 + ∑ j = τ + 1 t − 1 h j 2 R j 2 ) ≤ R 2 + 1 4 h t 2 . h_t^2\le\|y_t\|^2\|w\|^2=\|y_t\|^2\left(1+\sum_{j=\tau+1}^{t-1}\frac{h_j^2}{R_j^2}\right)\le R^2+\frac{1}{4}h_t^2. h t 2 ≤ ∥ y t ∥ 2 ∥ w ∥ 2 = ∥ y t ∥ 2 ( 1 + j = τ + 1 ∑ t − 1 R j 2 h j 2 ) ≤ R 2 + 4 1 h t 2 .
Thus h t ≤ 2 R / 3 ≤ 2 R h_t\le2R/\sqrt{3}\le2R h t ≤ 2 R / 3 ≤ 2 R ; as the identity is pathwise, w w w may be chosen after the path. ■
A Fixed I.I.D. Witness and the Rate Contradiction
Proof of Theorem 8 . Choose max { β p , a } < u < min { r , 1 } \max\{\beta_p,a\}<u<\min\{r,1\} max { β p , a } < u < min { r , 1 } . A finite common D > 0 D>0 D > 0 makes b t = D t − u b_t=Dt^{-u} b t = D t − u dominate the proposed logarithmic envelope. Suppose this weaker envelope holds with probability at least 1 − δ 1-\delta 1 − δ on every finite-dimensional instance. For a large deterministic N N N , set
μ = 8 D N − u , κ = 1 / 128 , B = 1 / 16 , A = ( κ / μ ) 1 / ( p − 1 ) , π = μ / A . \mu=8DN^{-u},\qquad\kappa=1/128,\qquad B=1/16,\qquad A=(\kappa/\mu)^{1/(p-1)},\qquad\pi=\mu/A. μ = 8 D N − u , κ = 1/128 , B = 1/16 , A = ( κ / μ ) 1/ ( p − 1 ) , π = μ / A .
Choose an integer d N ≥ 2 N 2 / ( 1 − δ ) d_N\ge2N^2/(1-\delta) d N ≥ 2 N 2 / ( 1 − δ ) . Take N N N large enough that μ ≤ 1 / 2 \mu\le1/2 μ ≤ 1/2 , A ≥ 1 A\ge1 A ≥ 1 , and π ≤ 1 / 2 \pi\le1/2 π ≤ 1/2 . Independently each round, draw Y ∼ Bernoulli ( π ) Y\sim\operatorname{Bernoulli}(\pi) Y ∼ Bernoulli ( π ) , a uniform J ∈ { 1 , … , d N } J\in\{1,\ldots,d_N\} J ∈ { 1 , … , d N } , and a uniform sign ϵ \epsilon ϵ , independent within a sample as well. Let R 0 = B R_0=B R 0 = B , R 1 = A R_1=A R 1 = A , and define
F ( x , y ; Y , J , ϵ ) = 1 2 ( x − 1 + μ ) + 2 − A Y x + 1 2 ∥ y ∥ 2 + R Y ϵ y J . F(x,y;Y,J,\epsilon)=\frac{1}{2}(x-1+\mu)_+^2-AYx+\frac{1}{2}\|y\|^2+R_Y\epsilon y_J. F ( x , y ; Y , J , ϵ ) = 2 1 ( x − 1 + μ ) + 2 − A Y x + 2 1 ∥ y ∥ 2 + R Y ϵ y J .
Admissibility. Each sample is convex and globally 1-smooth: the two orthogonal Hessian blocks are bounded by one. In the first coordinate x x x , the sample loss attains its minimum at any x ≤ 1 − μ x\le1-\mu x ≤ 1 − μ when Y = 0 Y=0 Y = 0 , and at x = 1 − μ + A x=1-\mu+A x = 1 − μ + A when Y = 1 Y=1 Y = 1 ; in the nuisance block y y y , its minimizer is − R Y ϵ e J -R_Y\epsilon e_J − R Y ϵ e J . Thus every sample minimum is finite and attained. The population is
f ( x , y ) = 1 2 ( x − 1 + μ ) + 2 − μ x + 1 2 ∥ y ∥ 2 , f(x,y)=\frac{1}{2}(x-1+\mu)_+^2-\mu x+\frac{1}{2}\|y\|^2, f ( x , y ) = 2 1 ( x − 1 + μ ) + 2 − μx + 2 1 ∥ y ∥ 2 ,
with minimizer ( 1 , 0 ) (1,0) ( 1 , 0 ) at initial distance one. The fixed additive noise ξ = ( μ − A Y , R Y ϵ e J ) \xi=(\mu-AY,R_Y\epsilon e_J) ξ = ( μ − A Y , R Y ϵ e J ) is centered. Since π A p = κ \pi A^p=\kappa π A p = κ and μ = π A \mu=\pi A μ = π A ,
E ∣ μ − A Y ∣ p ≤ 2 κ , E ∥ ξ ∥ p ≤ 2 p − 1 ( 2 κ + B p + κ ) ≤ 6 / 128 + 2 / 16 < 1. \mathbb{E}|\mu-AY|^p\le2\kappa,\qquad\mathbb{E}\|\xi\|^p\le2^{p-1}(2\kappa+B^p+\kappa)\le6/128+2/16<1. E ∣ μ − A Y ∣ p ≤ 2 κ , E ∥ ξ ∥ p ≤ 2 p − 1 ( 2 κ + B p + κ ) ≤ 6/128 + 2/16 < 1.
For every state the gap is at least ∥ y ∥ 2 / 2 \|y\|^2/2 ∥ y ∥ 2 /2 ; when x ≤ 1 / 2 x\le1/2 x ≤ 1/2 it is also at least μ ( 1 − x ) − μ 2 / 2 ≥ μ / 4 \mu(1-x)-\mu^2/2\ge\mu/4 μ ( 1 − x ) − μ 2 /2 ≥ μ /4 .
Coupling and useful events. Feed the rule reference gradients ( − A Y t , y t + R Y t ϵ t e J t ) (-AY_t,y_t+R_{Y_t}\epsilon_t e_{J_t}) ( − A Y t , y t + R Y t ϵ t e J t ) until the first update whose first coordinate X t + 1 X_{t+1} X t + 1 exceeds 1 / 2 1/2 1/2 , or until round N N N . Denote that last retained round by S ≤ N S\le N S ≤ N . The first velocity is nonpositive, so X t X_t X t is nondecreasing. Under the shared random draws and seed, the reference and actual trajectories coincide through step S S S : prior to step S S S , every queried first coordinate satisfies X t ≤ 1 / 2 ≤ 1 − μ X_t\le1/2\le1-\mu X t ≤ 1/2 ≤ 1 − μ , so all stochastic gradients, histories, and controls coincide. Let G G G be the actual instance’s success event for all t ≤ N t\le N t ≤ N . Then P ( G ) ≥ 1 − δ \mathbb{P}(G)\ge1-\delta P ( G ) ≥ 1 − δ . On G G G a crossing occurs by N N N , since otherwise the final gap is at least μ / 4 = 2 b N \mu/4=2b_N μ /4 = 2 b N ; also
∥ y t ∥ ≤ 2 b t − 1 ( 2 ≤ t ≤ S + 1 ) . (61) \|y_t\|\le\sqrt{2b_{t-1}}\qquad(2\le t\le S+1). \tag*{(61)} ∥ y t ∥ ≤ 2 b t − 1 ( 2 ≤ t ≤ S + 1 ) . (61)
Let C C C mean that J 1 , … , J N J_1,\ldots,J_N J 1 , … , J N are distinct. The union bound gives P ( C c ) ≤ N ( N − 1 ) / ( 2 d N ) ≤ ( 1 − δ ) / 4 \mathbb{P}(C^c)\le N(N-1)/(2d_N)\le(1-\delta)/4 P ( C c ) ≤ N ( N − 1 ) / ( 2 d N ) ≤ ( 1 − δ ) /4 . For Q p = p / ( p − 1 ) Q_p=p/(p-1) Q p = p / ( p − 1 ) choose 0 < θ < min { 1 , u Q p } 0<\theta<\min\{1,uQ_p\} 0 < θ < min { 1 , u Q p } and set n 0 = ⌊ N θ ⌋ n_0=\lfloor N^\theta\rfloor n 0 = ⌊ N θ ⌋ . Let E \mathcal{E} E mean Y t = 0 Y_t=0 Y t = 0 for t < n 0 t<n_0 t < n 0 . Then P ( E c ) ≤ n 0 π = O ( N θ − u Q p ) → 0 \mathbb{P}(\mathcal{E}^c)\le n_0\pi=O(N^{\theta-uQ_p})\to0 P ( E c ) ≤ n 0 π = O ( N θ − u Q p ) → 0 ; the nuisance noise still acts.
Each pulse has a short useful window. Set P = h + u / ( p − 1 ) + 5 P = h + u/(p - 1) + 5 P = h + u / ( p − 1 ) + 5 , Λ = 3 a P / ρ \Lambda= 3^{a}P/\rho Λ = 3 a P / ρ , and w j = ⌈ Λ ( 1 + j ) a log N ⌉ w_j = \lceil\Lambda(1 + j)^a \log N\rceil w j = ⌈ Λ ( 1 + j ) a log N ⌉ . Uniformly for n 0 ≤ j ≤ N n_0 \le j \le N n 0 ≤ j ≤ N , large N N N gives j ≥ t 0 j \ge t_0 j ≥ t 0 , w j ≤ j w_j \le j w j ≤ j , and
2 b j w j B 2 = O ( N θ ( a − u ) log N + N − θ u ) ⟶ 0. (62) \frac{2b_jw_j}{B^2} = O\left(N^{\theta(a-u)}\log N + N^{-\theta u}\right) \longrightarrow0. \tag*{(62)} B 2 2 b j w j = O ( N θ ( a − u ) log N + N − θ u ) ⟶ 0. (62)
Fix the realized controls through S S S . For a rare arrival j ≤ S j \le S j ≤ S , let h t ( j ) h_t^{(j)} h t ( j ) be the impulse response (60) with τ = j \tau=j τ = j , and put t j = min { S + 1 , j + w j + 1 } t_j = \min\{S+1,j+w_j+1\} t j = min { S + 1 , j + w j + 1 } . On G ∩ C G \cap C G ∩ C , the nuisance observations ζ 1 , … , ζ t j − 1 \zeta_1,\ldots,\zeta_{t_j-1} ζ 1 , … , ζ t j − 1 are nonzero and orthogonal, each has norm at least B B B , and the vector at j j j has norm A A A . Equation (61) gives ∥ y t j ∥ ≤ 2 b j \lVert y_{t_j}\rVert\le\sqrt{2b_j} ∥ y t j ∥ ≤ 2 b j ; thus Lemma 27 and (62) give h t j ( j ) ≤ 2 2 b j h_{t_j}^{(j)} \le2\sqrt{2b_j} h t j ( j ) ≤ 2 2 b j for every j ≥ n 0 j \ge n_0 j ≥ n 0 .
If t j < S + 1 t_j < S+1 t j < S + 1 , the first w j w_j w j subsequent momentum coefficients satisfy
∏ i = j + 1 j + w j β i ≤ exp ( − ρ w j 3 a ( 1 + j ) a ) ≤ N − P . \prod_{i=j+1}^{j+w_j}\beta_i \le\exp\left(-\frac{\rho w_j}{3^a(1+j)^a}\right) \le N^{-P}. i = j + 1 ∏ j + w j β i ≤ exp ( − 3 a ( 1 + j ) a ρ w j ) ≤ N − P .
Every subsequent factor is at most one. Since c j ≤ 1 c_j \le1 c j ≤ 1 , the polynomial bound on the step sizes in (17) implies pathwise that
0 ≤ h S + 1 ( j ) − h t j ( j ) ≤ A H N ( 1 + N ) h N − P ≤ C 0 N − 4 , (63) 0 \le h_{S+1}^{(j)} - h_{t_j}^{(j)} \le AHN(1+N)^hN^{-P} \le C_0N^{-4}, \tag*{(63)} 0 ≤ h S + 1 ( j ) − h t j ( j ) ≤ A H N ( 1 + N ) h N − P ≤ C 0 N − 4 , (63)
with C 0 C_0 C 0 depending only on common fixed parameters; indeed A = ( κ / ( 8 D ) ) 1 / ( p − 1 ) N u / ( p − 1 ) A = \left(\kappa/(8D)\right)^{1/(p-1)}N^{u/(p-1)} A = ( κ / ( 8 D ) ) 1/ ( p − 1 ) N u / ( p − 1 ) . For t j = S + 1 t_j=S+1 t j = S + 1 the tail is zero.
Contradiction on total progress. The displacement along the first coordinate is X S + 1 = ∑ j ≤ S Y j h S + 1 ( j ) X_{S+1} = \sum_{j\le S}Y_jh_{S+1}^{(j)} X S + 1 = ∑ j ≤ S Y j h S + 1 ( j ) . On G ∩ C ∩ E G \cap C \cap\mathcal{E} G ∩ C ∩ E , it must exceed 1 / 2 1/2 1/2 , but the preceding bounds give
1 2 < X S + 1 ≤ 2 2 D ∑ j = n 0 N Y j j − u / 2 + C 0 N − 3 . \frac{1}{2} < X_{S+1} \le2\sqrt{2D}\sum_{j=n_0}^{N}Y_jj^{-u/2} + C_0N^{-3}. 2 1 < X S + 1 ≤ 2 2 D j = n 0 ∑ N Y j j − u /2 + C 0 N − 3 .
Multiply by that event’s indicator and take expectations. Dropping the indicator on the nonnegative right side and using E Y j = π \mathbb{E}Y_j=\pi E Y j = π yields
1 2 P ( G ∩ C ∩ E ) ≤ 2 2 D π ∑ j = 1 N j − u / 2 + C 0 N − 3 . (64) \frac{1}{2}\mathbb{P}(G \cap C \cap\mathcal{E}) \le2\sqrt{2D}\,\pi\sum_{j=1}^{N}j^{-u/2} + C_0N^{-3}. \tag*{(64)} 2 1 P ( G ∩ C ∩ E ) ≤ 2 2 D π j = 1 ∑ N j − u /2 + C 0 N − 3 . (64)
The left side has liminf at least 3 ( 1 − δ ) / 8 3(1-\delta)/8 3 ( 1 − δ ) /8 . The right side is O ( N 1 − u ( Q p + 1 / 2 ) ) + O ( N − 3 ) → 0 O(N^{1-u(Q_p+1/2)})+O(N^{-3}) \to0 O ( N 1 − u ( Q p + 1/2 ) ) + O ( N − 3 ) → 0 because u > β p = 1 / ( Q p + 1 / 2 ) u > \beta_p = 1/(Q_p+1/2) u > β p = 1/ ( Q p + 1/2 ) and u < 1 u < 1 u < 1 . This contradiction gives prefix failure strictly greater than δ \delta δ for one fixed law and finite dimension. ■
The same argument gives the fixed-time analogue.
Proposition 28 (Fixed-time dimension-uniform lower bound). Fix p ∈ ( 1 , 2 ] p \in(1,2] p ∈ ( 1 , 2 ] and a ∈ [ 0 , 1 ) a \in[0,1) a ∈ [ 0 , 1 ) , let C > 0 C > 0 C > 0 and k ≥ 0 k \ge0 k ≥ 0 be finite, and let r > max { β p , a } r > \max\{\beta_p,a\} r > max { β p , a } . Let ( r u l e δ ) δ ∈ ( 0 , 1 ) (\mathrm{rule}_\delta)_{\delta\in(0,1)} ( rule δ ) δ ∈ ( 0 , 1 ) satisfy the conditions of A u n i f ( a ) \mathcal{A}_{\mathrm{unif}}(a) A unif ( a ) with ρ \rho ρ , t 0 t_0 t 0 , and h h h independent of δ \delta δ , and with H H H replaced by H δ − h 0 H\delta^{-h_0} H δ − h 0 for constants H H H , h 0 ≥ 0 h_0 \ge0 h 0 ≥ 0 independent of δ \delta δ . There are δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , a deterministic integer t ≥ 1 t \ge1 t ≥ 1 , and a finite-dimensional instance with L = R = σ = 1 L = R = \sigma= 1 L = R = σ = 1 , fixed before the samples and the seed are drawn and satisfying the conditions of Theorem 8 , such that r u l e δ \mathrm{rule}_\delta rule δ satisfies P { f ( x t + 1 ) − f ⋆ > C t − r [ log ( e t / δ ) ] k } > δ \mathbb{P}\{f(x_{t+1}) - f^\star> Ct^{-r}[\log(et/\delta)]^k\} > \delta P { f ( x t + 1 ) − f ⋆ > C t − r [ log ( e t / δ ) ] k } > δ . Here C C C and k k k are shared across dimensions and do not depend on δ \delta δ or t t t .
With polylogarithmic dependence on 1 / δ 1/\delta 1/ δ , no fixed-time exponent above max { β p , a } \max\{\beta_p,a\} max { β p , a } is therefore attainable either.
Proof of Proposition 28 . Suppose the envelope holds for every δ \delta δ , every deterministic time, and every finite-dimensional instance. Choose max { β p , a } < u < min { r , 1 } \max\{\beta_p,a\}<u<\min\{r,1\} max { β p , a } < u < min { r , 1 } . For each deterministic N N N , put
δ N = 1 4 N , D N = C log k ( 4 e N 2 ) , b N , t = D N t − u . \delta_N=\frac{1}{4N},\qquad D_N=C\log^k(4eN^2),\qquad b_{N,t}=D_Nt^{-u}. δ N = 4 N 1 , D N = C log k ( 4 e N 2 ) , b N , t = D N t − u .
Since t − r ≤ t − u t^{-r}\le t^{-u} t − r ≤ t − u and log ( e t / δ N ) ≤ log ( 4 e N 2 ) \log(et/\delta_N)\le\log(4eN^2) log ( e t / δ N ) ≤ log ( 4 e N 2 ) for 1 ≤ t ≤ N 1\le t\le N 1 ≤ t ≤ N , the envelope of ruleδ N _{\delta_N} δ N is at most b N , t b_{N,t} b N , t on this prefix, and a union bound shows that, on every instance, f ( x t + 1 ) − f ⋆ ≤ b N , t f(x_{t+1})-f^\star\le b_{N,t} f ( x t + 1 ) − f ⋆ ≤ b N , t for all t ≤ N t\le N t ≤ N with probability at least 3 / 4 3/4 3/4 ; there η t ≤ H ( 4 N ) h 0 ( 1 + N ) h \eta_t\le H(4N)^{h_0}(1+N)^h η t ≤ H ( 4 N ) h 0 ( 1 + N ) h .
Run the proof of Theorem 8 for ruleδ N _{\delta_N} δ N , with D D D replaced by D N D_N D N , 1 − δ 1-\delta 1 − δ by 3 / 4 3/4 3/4 , and P P P by h + h 0 + u / ( p − 1 ) + 5 h+h_0+u/(p-1)+5 h + h 0 + u / ( p − 1 ) + 5 . Its estimates depend on the rule only through the common constants ρ , t 0 , H , h , h 0 \rho,t_0,H,h,h_0 ρ , t 0 , H , h , h 0 , and C ≤ D N = N o ( 1 ) C\le D_N=N^{o(1)} C ≤ D N = N o ( 1 ) , so they hold for all large N N N , with constants independent of N N N and of the rule. Explicitly, μ = 8 D N N − u → 0 \mu=8D_NN^{-u}\to0 μ = 8 D N N − u → 0 , d N ≥ 8 N 2 / 3 d_N\ge8N^{2/3} d N ≥ 8 N 2/3 gives P ( C c ) ≤ 3 / 16 \mathbb{P}(\mathcal{C}^c)\le3/16 P ( C c ) ≤ 3/16 , and, uniformly for n 0 ≤ j ≤ N n_0\le j\le N n 0 ≤ j ≤ N ,
π = κ − 1 / ( p − 1 ) μ Q p = O ( D N Q p N − u Q p ) , P ( E c ) ≤ n 0 π = O ( D N Q p N θ − u Q p ) , \pi=\kappa^{-1/(p-1)}\mu Q_p=O(D_N^{Q_p}N^{-uQ_p}),\qquad\mathbb{P}(\mathcal{E}^c)\le n_0\pi=O(D_N^{Q_p}N^{\theta-uQ_p}), π = κ − 1/ ( p − 1 ) μ Q p = O ( D N Q p N − u Q p ) , P ( E c ) ≤ n 0 π = O ( D N Q p N θ − u Q p ) ,
2 b N , j w j D N 2 = O ( D N [ N θ ( a − u ) log N + N − θ u ] ) , \frac{2b_{N,j}w_j}{D_N^2}=O\left(D_N\left[N^{\theta(a-u)}\log N+N^{-\theta u}\right]\right), D N 2 2 b N , j w j = O ( D N [ N θ ( a − u ) log N + N − θ u ] ) ,
all of which tend to zero. In (63), A ≤ ( κ / ( 8 C ) ) 1 / ( p − 1 ) N u / ( p − 1 ) A\le(\kappa/(8C))^{1/(p-1)}N^{u/(p-1)} A ≤ ( κ / ( 8 C ) ) 1/ ( p − 1 ) N u / ( p − 1 ) and η s ≤ H ( 4 N ) h 0 ( 1 + N ) h \eta_s\le H(4N)^{h_0}(1+N)^h η s ≤ H ( 4 N ) h 0 ( 1 + N ) h ; since 1 + h + h 0 + u / ( p − 1 ) − P = − 4 1+h+h_0+u/(p-1)-P=-4 1 + h + h 0 + u / ( p − 1 ) − P = − 4 , the tail is still O ( N − 4 ) O(N^{-4}) O ( N − 4 ) . The left side of (64) is at least ( 3 / 4 − 3 / 16 − o ( 1 ) ) / 2 (3/4-3/16-o(1))/2 ( 3/4 − 3/16 − o ( 1 )) /2 , with liminf at least 9 / 32 9/32 9/32 , whereas its right side is O ( D N Q p + 1 / 2 N 1 − u ( Q p + 1 / 2 ) ) + O ( N − 3 ) → 0 O(D_N^{Q_p+1/2}N^{1-u(Q_p+1/2)})+O(N^{-3})\to0 O ( D N Q p + 1/2 N 1 − u ( Q p + 1/2 ) ) + O ( N − 3 ) → 0 as u > β p = 1 / ( Q p + 1 / 2 ) u>\beta_p=1/(Q_p+1/2) u > β p = 1/ ( Q p + 1/2 ) . Hence, for some N N N , the instance of dimension d N d_N d N fails the prefix guarantee with probability above 1 / 4 = N δ N 1/4=N\delta_N 1/4 = N δ N , and the union bound gives a deterministic t ≤ N t\le N t ≤ N with fixed-time failure probability above δ N \delta_N δ N . ■
Time-Uniform Averaging Momentum
This appendix proves Theorem 9 and Propositions 29 and 30 .
Proof of Theorem 9 . Write F t = f ( x t ) − f ⋆ F_t=f(x_t)-f^\star F t = f ( x t ) − f ⋆ , μ t = ∇ f ( x t ) \mu_t=\nabla f(x_t) μ t = ∇ f ( x t ) , and D t = ∥ z t − x ⋆ ∥ D_t=\lVert z_t-x^\star\rVert D t = ∥ z t − x ⋆ ∥ . The averaging update gives
x t = 1 t ∑ s = 1 t z s , z t = t x t − ( t − 1 ) x t − 1 . (65) x_t=\frac{1}{t}\sum_{s=1}^{t}z_s,\qquad z_t=tx_t-(t-1)x_{t-1}. \tag*{(65)} x t = t 1 s = 1 ∑ t z s , z t = t x t − ( t − 1 ) x t − 1 . (65)
At t = 1 t=1 t = 1 the second identity is independent of x 0 x_0 x 0 .
A smooth online-to-batch inequality. By the standard inequality for smooth convex functions (Nesterov [46 ] , Theorem 2.1.5) and ∇ f ( x ⋆ ) = 0 \nabla f(x^\star)=0 ∇ f ( x ⋆ ) = 0 , we have
⟨ ∇ f ( x ) , x − x ⋆ ⟩ ≥ f ( x ) − f ⋆ + ∥ ∇ f ( x ) ∥ 2 2 L . \langle\nabla f(x),x-x^\star\rangle\ge f(x)-f^\star+\frac{\lVert\nabla f(x)\rVert^2}{2L}. ⟨ ∇ f ( x ) , x − x ⋆ ⟩ ≥ f ( x ) − f ⋆ + 2 L ∥ ∇ f ( x ) ∥ 2 .
Combining it with (65) and ⟨ μ t , x t − x t − 1 ⟩ ≥ F t − F t − 1 \langle\mu_t,x_t-x_{t-1}\rangle\ge F_t-F_{t-1} ⟨ μ t , x t − x t − 1 ⟩ ≥ F t − F t − 1 gives
⟨ μ t , z t − x ⋆ ⟩ ≥ t F t − ( t − 1 ) F t − 1 + ∥ μ t ∥ 2 2 L . (66) \langle\mu_t,z_t-x^\star\rangle\ge tF_t-(t-1)F_{t-1}+\frac{\lVert\mu_t\rVert^2}{2L}. \tag*{(66)} ⟨ μ t , z t − x ⋆ ⟩ ≥ t F t − ( t − 1 ) F t − 1 + 2 L ∥ μ t ∥ 2 . (66)
This is the smooth form of the anytime online-to-batch inequality ([9 ] ).
A pathwise energy recursion. Let e t = d t − μ t e_t = d_t - \mu_t e t = d t − μ t . Expanding the update and using (66) and ∥ μ t + e t ∥ 2 ≤ 2 ∥ μ t ∥ 2 + 2 ∥ e t ∥ 2 \lVert\mu_t + e_t\rVert^2 \le2\lVert\mu_t\rVert^2 + 2\lVert e_t\rVert^2 ∥ μ t + e t ∥ 2 ≤ 2 ∥ μ t ∥ 2 + 2 ∥ e t ∥ 2 gives
D t + 1 2 ≤ D t 2 − 2 γ t { t F t − ( t − 1 ) F t − 1 } + Δ t , Δ t = 2 γ t 2 ∥ e t ∥ 2 − 2 γ t ⟨ e t , z t − x ⋆ ⟩ . D_{t+1}^2 \le D_t^2 - 2\gamma_t\{tF_t - (t-1)F_{t-1}\} + \Delta_t,\qquad\Delta_t = 2\gamma_t^2\lVert e_t\rVert^2 - 2\gamma_t\langle e_t,z_t-x^\star\rangle. D t + 1 2 ≤ D t 2 − 2 γ t { t F t − ( t − 1 ) F t − 1 } + Δ t , Δ t = 2 γ t 2 ∥ e t ∥ 2 − 2 γ t ⟨ e t , z t − x ⋆ ⟩ .
Here the omitted term involving ∥ μ t ∥ 2 \lVert\mu_t\rVert^2 ∥ μ t ∥ 2 is nonpositive since γ t ≤ c 0 / ( 4 L ℓ ˉ t 2 ) ≤ 1 / ( 2 L ) \gamma_t \le c_0/(4L\bar{\ell}_t^2) \le1/(2L) γ t ≤ c 0 / ( 4 L ℓ ˉ t 2 ) ≤ 1/ ( 2 L ) . Define W 1 = D 1 2 W_1=D_1^2 W 1 = D 1 2 and
W t + 1 = D t + 1 2 + 2 t γ t F t + 2 ∑ s = 1 t − 1 s ( γ s − γ s + 1 ) F s . W_{t+1}=D_{t+1}^2+2t\gamma_tF_t+2\sum_{s=1}^{t-1}s(\gamma_s-\gamma_{s+1})F_s. W t + 1 = D t + 1 2 + 2 t γ t F t + 2 s = 1 ∑ t − 1 s ( γ s − γ s + 1 ) F s .
Since γ t \gamma_t γ t is nonincreasing, W t ≥ D t 2 ≥ 0 W_t\ge D_t^2\ge0 W t ≥ D t 2 ≥ 0 . The exact difference W t + 1 − W t = D t + 1 2 − D t 2 + 2 γ t { t F t − ( t − 1 ) F t − 1 } W_{t+1}-W_t=D_{t+1}^2-D_t^2+2\gamma_t\{tF_t-(t-1)F_{t-1}\} W t + 1 − W t = D t + 1 2 − D t 2 + 2 γ t { t F t − ( t − 1 ) F t − 1 } therefore gives W t + 1 ≤ W t + Δ t W_{t+1}\le W_t+\Delta_t W t + 1 ≤ W t + Δ t , also at t = 1 t=1 t = 1 without an F 0 F_0 F 0 term.
Predictable localization and conditional clipping. Let τ = inf { t ≥ 1 : W t ≥ 2 R 2 } \tau=\inf\{t\ge1:W_t\ge2R^2\} τ = inf { t ≥ 1 : W t ≥ 2 R 2 } and I t = 1 { t < τ } I_t=\mathbf{1}_{\{t<\tau\}} I t = 1 { t < τ } . Each W s W_s W s is F s − 1 \mathcal{F}_{s-1} F s − 1 -measurable, so I t I_t I t is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, that is, predictable, and nonincreasing in t t t . Since ∥ d s ∥ ≤ λ s \lVert d_s\rVert\le\lambda_s ∥ d s ∥ ≤ λ s , the states z t z_t z t , x t x_t x t and the errors e t e_t e t are bounded at each fixed t t t , so e t e_t e t is integrable. On t < τ t<\tau t < τ , all s ≤ t s\le t s ≤ t satisfy D s < 2 R D_s<\sqrt{2}R D s < 2 R . Thus (65) gives ∥ x t − x ⋆ ∥ < 2 R \lVert x_t-x^\star\rVert<2R ∥ x t − x ⋆ ∥ < 2 R and ∥ μ t ∥ ≤ 2 L R ≤ λ t / 2 \lVert\mu_t\rVert\le2LR\le\lambda_t/2 ∥ μ t ∥ ≤ 2 L R ≤ λ t /2 . Lemma 12, applied conditionally on this predictable event, yields
∥ e t ∥ ≤ 3 2 λ t , ∥ E t − 1 e t ∥ ≤ B t : = 2 σ p λ t 1 − p , E t − 1 ∥ e t ∥ 2 ≤ V t : = 2 σ 2 p λ t 2 − 2 p . (67) \lVert e_t\rVert\le\frac{3}{2}\lambda_t,\qquad\lVert\mathbb{E}_{t-1}e_t\rVert\le B_t:=2\sigma^p\lambda_t^{1-p},\qquad\mathbb{E}_{t-1}\lVert e_t\rVert^2\le V_t:=2\sigma^{2p}\lambda_t^{2-2p}. \tag*{(67)} ∥ e t ∥ ≤ 2 3 λ t , ∥ E t − 1 e t ∥ ≤ B t := 2 σ p λ t 1 − p , E t − 1 ∥ e t ∥ 2 ≤ V t := 2 σ 2 p λ t 2 − 2 p . (67)
For q > 1 q>1 q > 1 , integral comparison gives
∑ t ≥ 1 1 t ( h δ + log t ) q ≤ 1 h δ q + 1 ( q − 1 ) h δ q − 1 . \sum_{t\ge1}\frac{1}{t(h_\delta+\log t)^q}\le\frac{1}{h_\delta^q}+\frac{1}{(q-1)h_\delta^{q-1}}. t ≥ 1 ∑ t ( h δ + log t ) q 1 ≤ h δ q 1 + ( q − 1 ) h δ q − 1 1 .
Since λ t p ≥ σ p t \lambda_t^p\ge\sigma^{p}t λ t p ≥ σ p t , the deterministic budgets are
∑ t γ t B t ≤ 4 c 0 R h δ , ∑ t γ t 2 V t ≤ 4 c 0 2 R 2 h δ 3 , γ t λ t = c 0 R ℓ ˉ t 2 . (68) \sum_t\gamma_tB_t\le\frac{4c_0R}{h_\delta},\qquad\sum_t\gamma_t^2V_t\le\frac{4c_0^2R^2}{h_\delta^3},\qquad\gamma_t\lambda_t=\frac{c_0R}{\bar{\ell}_t^2}. \tag*{(68)} t ∑ γ t B t ≤ h δ 4 c 0 R , t ∑ γ t 2 V t ≤ h δ 3 4 c 0 2 R 2 , γ t λ t = ℓ ˉ t 2 c 0 R . (68)
These include σ = 0 \sigma=0 σ = 0 , when B t = V t = 0 B_t=V_t=0 B t = V t = 0 .
One application of the common energy bound. Apply Lemma 15 with P m = W m + 1 P_m=W_{m+1} P m = W m + 1 for m ≥ 0 m\ge0 m ≥ 0 , the predictable indicators J t = I t J_t=I_t J t = I t , E = Ω \mathcal{E}=\Omega E = Ω , p 0 = R 2 p_0=R^2 p 0 = R 2 , q = h δ q=h_\delta q = h δ , and
a t = 2 γ t 2 , w t = − 2 γ t ( z t − x ⋆ ) , s t = 2 γ t , M t = 3 2 λ t , U t = 2 λ t . a_t=2\gamma_t^2,\qquad w_t=-2\gamma_t(z_t-x^\star),\qquad s_t=2\gamma_t,\qquad M_t=\frac{3}{2}\lambda_t,\qquad U_t=2\lambda_t. a t = 2 γ t 2 , w t = − 2 γ t ( z t − x ⋆ ) , s t = 2 γ t , M t = 2 3 λ t , U t = 2 λ t .
The pathwise recursion W t + 1 ≤ W t + Δ t W_{t+1}\le W_t+\Delta_t W t + 1 ≤ W t + Δ t is the required one, P 0 = D 1 2 ≤ R 2 P_0=D_1^2\le R^2 P 0 = D 1 2 ≤ R 2 , and ∥ w t ∥ ≤ s t P t − 1 \lVert w_t\rVert\le s_t\sqrt{P_{t-1}} ∥ w t ∥ ≤ s t P t − 1 because D t 2 ≤ W t D_t^2\le W_t D t 2 ≤ W t . On I t = 1 I_t=1 I t = 1 , (67) gives B t B_t B t , V t V_t V t , and M t M_t M t , and Lemma 12 also gives ∥ e t − E t − 1 e t ∥ ≤ U t \lVert e_t-\mathbb{E}_{t-1}e_t\rVert\le U_t ∥ e t − E t − 1 e t ∥ ≤ U t . By (68) and γ t λ t = c 0 R / ℓ ˉ t 2 ≤ c 0 R / h δ 2 \gamma_t\lambda_t=c_0R/\bar{\ell}_t^2\le c_0R/h_\delta^2 γ t λ t = c 0 R / ℓ ˉ t 2 ≤ c 0 R / h δ 2 , the budgets of the lemma satisfy
A ≤ 8 c 0 2 R 2 h δ 3 , K 0 ≤ 9 c 0 2 R 2 2 h δ 4 , B 0 ≤ 8 c 0 R h δ , V 0 ≤ 16 c 0 2 R 2 h δ 3 , J 0 ≤ 4 c 0 R h δ 2 . A\le\frac{8c_0^2R^2}{h_\delta^3},\qquad K_0\le\frac{9c_0^2R^2}{2h_\delta^4},\qquad B_0\le\frac{8c_0R}{h_\delta},\qquad V_0\le\frac{16c_0^2R^2}{h_\delta^3},\qquad J_0\le\frac{4c_0R}{h_\delta^2}. A ≤ h δ 3 8 c 0 2 R 2 , K 0 ≤ 2 h δ 4 9 c 0 2 R 2 , B 0 ≤ h δ 8 c 0 R , V 0 ≤ h δ 3 16 c 0 2 R 2 , J 0 ≤ h δ 2 4 c 0 R .
so C ≤ R 2 ( 1 + 29 c 0 2 / h δ 3 ) C\le R^2(1+29c_0^2/h_\delta^3) C ≤ R 2 ( 1 + 29 c 0 2 / h δ 3 ) and D ≤ 24 c 0 R / h δ D\le24c_0R/h_\delta D ≤ 24 c 0 R / h δ . Since c 0 = 1 / 128 c_0=1/128 c 0 = 1/128 and h δ ≥ 8 h_\delta\ge8 h δ ≥ 8 , ( C + D ) 2 ≤ R 2 ( 1 + 29 c 0 2 / h δ 3 + 24 c 0 / h δ ) 2 < 2 R 2 (\sqrt{C}+D)^2\le R^2(1+29c_0^2/h_\delta^3+24c_0/h_\delta)^2<2R^2 ( C + D ) 2 ≤ R 2 ( 1 + 29 c 0 2 / h δ 3 + 24 c 0 / h δ ) 2 < 2 R 2 . Except on probability e − h δ < δ e^{-h_\delta}<\delta e − h δ < δ , the lemma therefore gives W m + 1 < 2 R 2 W_{m+1}<2R^2 W m + 1 < 2 R 2 whenever I m = 1 I_m=1 I m = 1 . If τ < ∞ \tau<\infty τ < ∞ , then τ ≥ 2 \tau\ge2 τ ≥ 2 because W 1 ≤ R 2 W_1\le R^2 W 1 ≤ R 2 , and m = τ − 1 m=\tau-1 m = τ − 1 has I m = 1 I_m=1 I m = 1 but W m + 1 = W τ ≥ 2 R 2 W_{m+1}=W_\tau\ge2R^2 W m + 1 = W τ ≥ 2 R 2 . Hence τ = ∞ \tau=\infty τ = ∞ on this event, and simultaneously for every T ≥ 1 T\ge1 T ≥ 1 ,
D T + 1 2 + 2 T γ T F T ≤ W T + 1 < 2 R 2 . D_{T+1}^2+2T\gamma_TF_T\le W_{T+1}<2R^2. D T + 1 2 + 2 T γ T F T ≤ W T + 1 < 2 R 2 .
In particular, D T + 1 < 2 R D_{T+1}<\sqrt{2}R D T + 1 < 2 R and F T ≤ R 2 / ( T γ T ) = 128 R λ T ℓ ˉ T 2 / T F_T\le R^2/(T\gamma_T)=128R\lambda_T\bar{\ell}_T^2/T F T ≤ R 2 / ( T γ T ) = 128 R λ T ℓ ˉ T 2 / T . The last iterate. Convexity of the implemented update and smoothness at the minimizer give
f ( x T + 1 ) − f ⋆ ≤ T T + 1 F T + f ( z T + 1 ) − f ⋆ T + 1 ≤ 128 R λ T ℓ ˉ T 2 + 2 L R 2 T + 1 . \begin{aligned}
f(x_{T+1}) - f^\star\le\frac{T}{T+1}F_T + \frac{f(z_{T+1}) - f^\star}{T+1} \\
&\le\frac{128R\lambda_T\bar{\ell}_T^2 + 2LR^2}{T+1}.
\end{aligned} f ( x T + 1 ) − f ⋆ ≤ T + 1 T F T + T + 1 f ( z T + 1 ) − f ⋆ ≤ T + 1 128 R λ T ℓ ˉ T 2 + 2 L R 2 .
Finally, λ T ≤ 4 L R + σ T 1 / p \lambda_T \le4LR + \sigma T^{1/p} λ T ≤ 4 L R + σ T 1/ p , ℓ ˉ T ≥ 1 \bar{\ell}_T \ge1 ℓ ˉ T ≥ 1 , and ( T + 1 ) − 1 ≤ T − 1 (T+1)^{-1} \le T^{-1} ( T + 1 ) − 1 ≤ T − 1 prove (22) . ■
Proposition 29 (Exact momentum conventions). The states in (20) obey
x t + 1 = x t − η t d t + β ~ t ( x t − x t − 1 ) , η t = γ t t + 1 , β ~ t = t − 1 t + 1 . x_{t+1} = x_t - \eta_t d_t + \widetilde{\beta}_t(x_t-x_{t-1}), \qquad\eta_t = \frac{\gamma_t}{t+1}, \qquad\widetilde{\beta}_t = \frac{t-1}{t+1}. x t + 1 = x t − η t d t + β t ( x t − x t − 1 ) , η t = t + 1 γ t , β t = t + 1 t − 1 . with x 0 = x 1 x_0=x_1 x 0 = x 1 . In the velocity convention v t + 1 = β t v t + d t v_{t+1}=\beta_t v_t+d_t v t + 1 = β t v t + d t , x t + 1 = x t − η t v t + 1 x_{t+1}=x_t-\eta_t v_{t+1} x t + 1 = x t − η t v t + 1 , v 1 = 0 v_1=0 v 1 = 0 , the equivalent parameters are β 1 = 0 \beta_1=0 β 1 = 0 and
β t = t − 1 t γ t − 1 γ t , α p 2 ≤ 1 − β t ≤ 1 t ( t ≥ 2 ) , α p = p − 1 p . \beta_t = \frac{t-1}{t}\frac{\gamma_{t-1}}{\gamma_t}, \qquad\frac{\alpha_p}{2} \le1-\beta_t \le\frac{1}{t} \quad(t\ge2), \qquad\alpha_p = \frac{p-1}{p}. β t = t t − 1 γ t γ t − 1 , 2 α p ≤ 1 − β t ≤ t 1 ( t ≥ 2 ) , α p = p p − 1 . The step sizes are uniformly bounded by c 0 / ( 8 L h δ 2 ) c_0/(8Lh_\delta^2) c 0 / ( 8 L h δ 2 ) .
Proof. Use z t − x t = ( t − 1 ) ( x t − x t − 1 ) z_t-x_t=(t-1)(x_t-x_{t-1}) z t − x t = ( t − 1 ) ( x t − x t − 1 ) in the averaging update. Eliminating velocity then gives β t = β ~ t η t − 1 / η t \beta_t=\widetilde{\beta}_t\eta_{t-1}/\eta_t β t = β t η t − 1 / η t . For r ( s ) = max { 4 L R , σ s 1 / p } ( h δ + log s ) 2 r(s)=\max\{4LR,\sigma s^{1/p}\}(h_\delta+\log s)^2 r ( s ) = max { 4 L R , σ s 1/ p } ( h δ + log s ) 2 , γ t = c 0 R / r ( t ) \gamma_t=c_0R/r(t) γ t = c 0 R / r ( t ) , and almost everywhere
0 ≤ d log r ( s ) d log s ≤ 1 p + 2 h δ ≤ 1 − α p 2 . 0 \le\frac{d\log r(s)}{d\log s} \le\frac{1}{p}+\frac{2}{h_\delta} \le1-\frac{\alpha_p}{2}. 0 ≤ d log s d log r ( s ) ≤ p 1 + h δ 2 ≤ 1 − 2 α p .
Integrating over [ t − 1 , t ] [t-1,t] [ t − 1 , t ] proves
t − 1 t ≤ β t ≤ ( t − 1 t ) α p / 2 < 1. \frac{t-1}{t} \le\beta_t \le\left(\frac{t-1}{t}\right)^{\alpha_p/2} < 1. t t − 1 ≤ β t ≤ ( t t − 1 ) α p /2 < 1.
Concavity of u ↦ u α p / 2 u\mapsto u^{\alpha_p/2} u ↦ u α p /2 yields the asserted lower bound on 1 − β t 1-\beta_t 1 − β t ; the upper bound is immediate. The bound on the step sizes follows from λ t ≥ 4 L R \lambda_t\ge4LR λ t ≥ 4 L R and t + 1 ≥ 2 t+1\ge2 t + 1 ≥ 2 . ■
Proposition 30 (Fixed momentum coefficient with unbounded step sizes). For every fixed b ∈ ( 0 , 1 ) b\in(0,1) b ∈ ( 0 , 1 ) , the trajectory in Theorem 9 admits the exact representation
w t + 1 = b w t + clip ( g t , Λ t ) , x t + 1 = x t − H t w t + 1 , w 1 = 0. w_{t+1}=bw_t+\operatorname{clip}(g_t,\Lambda_t), \qquad x_{t+1}=x_t-H_tw_{t+1}, \qquad w_1=0. w t + 1 = b w t + clip ( g t , Λ t ) , x t + 1 = x t − H t w t + 1 , w 1 = 0. with deterministic finite step sizes H t → ∞ H_t\to\infty H t → ∞ and clipping thresholds Λ t \Lambda_t Λ t that may depend on the current stochastic gradient. In particular, a constant momentum coefficient b ∈ ( 0 , 1 ) b\in(0,1) b ∈ ( 0 , 1 ) does not by itself force the β p \beta_p β p barrier if exponentially growing step sizes H t H_t H t and clipping thresholds depending on the current gradient are permitted. Because H t H_t H t grows exponentially, it violates the polynomial bound on the step sizes in Theorem 7 , so some restriction on the growth of the step sizes is needed there.
Proof. Set
H 1 = γ 1 2 ( 1 − b ) 2 , H t = H 1 2 b − ( t − 1 ) t ( t + 1 ) , s t = t γ t b t − 1 2 H 1 , Λ t = s t min { ∥ g t ∥ , λ t } . H_1=\frac{\gamma_1}{2(1-b)^2}, \qquad H_t=H_1\frac{2b^{-(t-1)}}{t(t+1)}, \qquad s_t=\frac{t\gamma_tb^{t-1}}{2H_1}, \qquad\Lambda_t=s_t\min\{\lVert g_t\rVert,\lambda_t\}. H 1 = 2 ( 1 − b ) 2 γ 1 , H t = H 1 t ( t + 1 ) 2 b − ( t − 1 ) , s t = 2 H 1 t γ t b t − 1 , Λ t = s t min {∥ g t ∥ , λ t } .
Since γ t ≤ γ 1 \gamma_t\le\gamma_1 γ t ≤ γ 1 and ∑ t ≥ 1 t b t − 1 = ( 1 − b ) − 2 \sum_{t\ge1}tb^{t-1}=(1-b)^{-2} ∑ t ≥ 1 t b t − 1 = ( 1 − b ) − 2 , 0 < s t ≤ 1 0<s_t\le1 0 < s t ≤ 1 . Therefore clip ( g t , Λ t ) = s t d t \operatorname{clip}(g_t,\Lambda_t)=s_td_t clip ( g t , Λ t ) = s t d t . At t = 1 t=1 t = 1 the displacement is − γ 1 d 1 / 2 -\gamma_1d_1/2 − γ 1 d 1 /2 . For t ≥ 2 t\ge2 t ≥ 2 , eliminating w t w_t w t gives
x t + 1 − x t = b H t H t − 1 ( x t − x t − 1 ) − H t s t d t = t − 1 t + 1 ( x t − x t − 1 ) − γ t t + 1 d t . (69) x_{t+1}-x_t=b\frac{H_t}{H_{t-1}}(x_t-x_{t-1})-H_ts_td_t=\frac{t-1}{t+1}(x_t-x_{t-1})-\frac{\gamma_t}{t+1}d_t. \tag*{(69)} x t + 1 − x t = b H t − 1 H t ( x t − x t − 1 ) − H t s t d t = t + 1 t − 1 ( x t − x t − 1 ) − t + 1 γ t d t . (69)
This is exactly Proposition 29 , path by path. Also H t → ∞ H_t \to\infty H t → ∞ exponentially. When g t = 0 g_t = 0 g t = 0 , take clipping at zero to be zero; if strictly positive thresholds are required, assign any positive threshold on that event without changing the update. ■
Comparison with existing averaging methods. Nguyen et al. [48 ] obtain the statistical exponent with horizon-free averaged methods. For their Algorithm 3, the proof of their Theorem D.1, via Lemma D.4 and Proposition D.5, gives ∥ z t − x ⋆ ∥ ≤ 2 R \lVert z_t - x^\star\rVert\le2R ∥ z t − x ⋆ ∥ ≤ 2 R on nested events over each finite prefix. Combining their query relation x t = ( 1 − a t ) y t + a t z t x_t = (1-a_t)y_t + a_tz_t x t = ( 1 − a t ) y t + a t z t (a t = 2 / ( t + 1 ) a_t = 2/(t+1) a t = 2/ ( t + 1 ) ) with convexity and L L L -smoothness at x ⋆ x^\star x ⋆ gives
f ( x T + 1 ) − f ⋆ ≤ f ( y T + 1 ) − f ⋆ + 4 L R 2 T + 2 . f(x_{T+1}) - f^\star\le f(y_{T+1}) - f^\star+ \frac{4LR^2}{T+2}. f ( x T + 1 ) − f ⋆ ≤ f ( y T + 1 ) − f ⋆ + T + 2 4 L R 2 .
Intersecting these nested events yields a time-uniform guarantee at the query state x T + 1 x_{T+1} x T + 1 as well, although their three-sequence accelerated scheme differs from the two-sequence recursion (20) . The high-probability regret bounds of Zhang and Cutkosky (2022, Theorems 4 and 5) assume bounded expected gradients and a horizon-dependent learner.