Introduction
We consider min x ∈ R n f ( x ) \min_{x\in\mathbb{R}^n} f(x) min x ∈ R n f ( x ) for a μ \mu μ -strongly convex f f f with L L L -Lipschitz gradient and condition number κ = L / μ \kappa= L/\mu κ = L / μ . Optimal methods need of order κ log ( 1 / ε ) \sqrt{\kappa}\log(1/\varepsilon) κ log ( 1/ ε ) gradient evaluations to reduce the gap by the factor ε \varepsilon ε , and gradient descent needs κ log ( 1 / ε ) \kappa\log(1/\varepsilon) κ log ( 1/ ε ) [1 , 2 ] . For κ = 10 6 \kappa= 10^6 κ = 1 0 6 this is a factor of a thousand. Nesterov’s method [2 , 3 ] attains the optimal order, but its momentum depends on μ \mu μ . Unlike L L L , the constant μ \mu μ cannot be found by backtracking: finitely many values and gradients never certify a positive lower bound on the curvature. The usual remedy is restarting [4 , 5 , 6 ] .
Conjugate gradients avoid μ \mu μ on quadratics. They minimize f f f exactly over the plane of the gradient and the previous step, and the minimization picks the very coefficients that Nesterov’s method computes from μ \mu μ and L L L [7 ] , so no constant has to be known. Exact subspace methods x k + 1 = arg min { f ( x ) : x ∈ x k + S k } x_{k+1}=\arg\min\{f(x):x\in x_k+S_k\} x k + 1 = arg min { f ( x ) : x ∈ x k + S k } extend this idea. Nemirovski’s orth-method [8 ] , known as SESOP [9 ] , uses
S k = span { ∇ f ( x k ) , x k − x k − 1 , x k − x 0 , ∑ i = 0 k w i ∇ f ( x i ) } S_k=\operatorname{span}\left\{\nabla f(x_k),\ x_k-x_{k-1},\ x_k-x_0,\ \sum_{i=0}^{k}w_i\nabla f(x_i)\right\} S k = span { ∇ f ( x k ) , x k − x k − 1 , x k − x 0 , i = 0 ∑ k w i ∇ f ( x i ) }
and attains the rate O ( L R 2 / k 2 ) O(LR^2/k^2) O ( L R 2 / k 2 ) on smooth convex functions without any parameter. It performs well on strongly convex problems [10 ] , in our experiments far better than Nesterov’s method with true constants. To the best of our knowledge, it is open whether unrestarted SESOP is accelerated uniformly over F μ , L \mathcal{F}_{\mu,L} F μ , L , in the sense of (2) . In this paper we continue the study of subspace methods started in [11 ] . Oracle lower bounds do not separate SESOP from accelerated methods, and on quadratics it coincides with conjugate gradients, so a hard instance must carry an infinite trajectory consistent with SESOP’s anchor and aggregate. For nonlinear conjugate gradients worst-case limitations are known [1 , 12 , 13 ] .
Our approach starts from the fact that exact minimization over x k + S k x_k+S_k x k + S k inherits the rate of an accelerated method as soon as S k S_k S k contains its momentum. This momentum is a geometric average of past gradients and steps with a horizon of order κ \sqrt{\kappa} κ , so, roughly speaking, acceleration without μ \mu μ comes down to memory: which subspaces built from the history contain such an average when κ \kappa κ is unknown? SESOP keeps everything, stale directions included, while a memory that forgets too fast cannot look far enough back. We show that forgetting on the right time scale is both necessary and sufficient, and that this scale can be found without κ \kappa κ .
Contributions
Remembering too much (§ 4.2 ). We measure a memory fixed in advance by its window m m m and the forgetting index δ \delta δ of its cumulative sums. For κ ≥ 100 ( m + 2 ) 2 \kappa\ge100(m+2)^2 κ ≥ 100 ( m + 2 ) 2 and slowly forgetting sums we construct functions in F μ , L \mathcal{F}_{\mu,L} F μ , L on which every exact subspace method with this memory needs Ω ( ( κ / m ) log ( 1 / ε ) ) \Omega((\kappa/m)\log(1/\varepsilon)) Ω (( κ / m ) log ( 1/ ε )) iterations as ε → 0 \varepsilon\to0 ε → 0 (Theorem 4.2 ). SESOP and the plane form of conjugate gradients are such methods and need Ω ( κ log ( 1 / ε ) ) \Omega(\kappa\log(1/\varepsilon)) Ω ( κ log ( 1/ ε )) iterations (Corollary 4.3 ).
Forgetting too fast and the right time scale (Sections 4.3 to 4.5). A filter pair that forgets much faster than at rate κ − 1 / 2 \kappa^{-1/2} κ − 1/2 leaves a method with a bounded window as
slow as a short window (Theorem 4.9 ). So uniform acceleration needs a window of length Ω ( κ ) \Omega(\sqrt{\kappa}) Ω ( κ ) or δ = Ω ( κ − 1 / 2 ) \delta= \Omega(\kappa^{-1/2}) δ = Ω ( κ − 1/2 ) , and with a bounded window and one pair exactly 1 − ρ = Θ ( κ − 1 / 2 ) 1-\rho= \Theta(\kappa^{-1/2}) 1 − ρ = Θ ( κ − 1/2 ) (Corollary 4.10). SESOP misses this time scale. Like the plane form, it stays slow with one aggregate on C ∞ \mathcal{C}^{\infty} C ∞ functions that are quadratic near the minimizer (Propositions 4.13, 4.14).
Windows (§ 5.1 ). A window of length 1.39 κ 1.39\sqrt{\kappa} 1.39 κ is accelerated, so the bound Ω ( κ ) \Omega(\sqrt{\kappa}) Ω ( κ ) is sharp (Proposition 5.1 ). A short window converges quadratically over blocks near a minimizer where the Lipschitz Hessian is a multiple of the identity plus a low-rank term (Theorem 5.2 ). Near such a minimizer this is faster than any linear rate.
Reaching the time scale without κ \kappa κ (§ 5.2 and § 5.3 ). Pairs of geometric filters store a horizon of order κ \sqrt{\kappa} κ in two vectors (Lemma 5.3). Created at times 2 j 2^j 2 j , they give a uniformly accelerated method with memory O ( log k ) O(\log k) O ( log k ) (Theorem 5.4 ). One pair with a smoothly adapted horizon and no resets is accelerated up to a polynomial prefactor, and with the SESOP directions it keeps the convex rate and is conjugate gradients on quadratics (Theorem 5.5 ). The schedule also allows inexact subproblems (Proposition 5.7).
All upper bounds follow from one estimate-sequence certificate. Restart schemes are nearly optimal [6 , 14 ] , and our rates are no better, but our methods need neither μ \mu μ nor L L L , are monotone and never reset their memory (Table 1 ).
Table 1. Methods of this paper, none of which uses μ \mu μ , L L L or a bound on κ \kappa κ . Storage counts vectors besides x k x_k x k and ∇ f ( x k ) \nabla f(x_k) ∇ f ( x k ) , and a filter with factor ρ \rho ρ has horizon 1 / ( 1 − ρ ) 1/(1-\rho) 1/ ( 1 − ρ ) . “Up to poly( κ ) (\kappa) ( κ ) ”: an accelerated rate with a prefactor polynomial in κ \kappa κ . The horizon bound needs κ ≥ 4 \kappa\geq4 κ ≥ 4 .
The rest of the paper is organized as follows. Sections 2 and 3 cover related work and the setting, Sections 4 and 5 the lower and upper bounds, and Section 6 experiments.
Related work
Exact minimization over low-dimensional subspaces is as old as conjugate gradients [7 ] and their nonlinear extensions [15 ] . A subspace containing the momentum of an accelerated method [2 ] gives an exact step at least as good as the accelerated one, which underlies the orth-method [8 ] , methods with a small-dimensional relaxation oracle [16 , 17 ] and the span search of Drori and Taylor [18 ] . Narkiss and Zibulevsky [9 ] named the approach SESOP and solved the subproblems by Newton’s method, so that for φ ( A x ) \varphi(Ax) φ ( A x ) a step costs as many products with A A A as a gradient step [19 ] . SESOP has been extended to weakly quasi-convex functions [10 ] , inexact gradients and restarted conjugate gradients under quadratic growth [11 , 20 ] . CGSO [12 ] attains O ( κ log ( 1 / ε ) ) O(\sqrt{\kappa}\log(1/\varepsilon)) O ( κ log ( 1/ ε )) without constants of the function with a subspace of dimension O ( log k ) O(\log k) O ( log k ) , using a progress test, the anchor and gradient sums restarted at every dyadic block. Other methods alternate conjugate gradient and accelerated steps [21 ] or minimize a model built from past gradients [22 ] . The limited-memory common-directions method [23 ] is accelerated when the subspace contains Nesterov’s momentum, which requires μ \mu μ and L L L , whereas our filter pairs contain such a momentum without the constants. Tupitsa [24 ] replaces the exact subproblem of SESOP by a cubic model and attains a local accelerated rate with adaptive restarts.
Restarting a convex accelerated method with period of order κ \sqrt{\kappa} κ gives an accelerated linear rate [25 ] . Adaptive schemes decrease an estimate of μ \mu μ when progress is insufficient [26 , 27 ] , test the progress directly [4 ] or search over a grid of constants [5 , 6 ] . Each restart throws the momentum away. The online estimate of μ \mu μ in [28 ] has no guarantee. The survey [29 ] listed the estimation of μ \mu μ without restarts as an open problem, which NAG-free [30 ] later addressed with a global rate of gradient descent type and acceleration near the minimizer. Our smooth pair estimates no constant: it adapts the forgetting horizon of the memory and is globally accelerated, up to a prefactor polynomial in κ \kappa κ . Lan, Ouyang and Zhang [31 ] reach ∥ ∇ f ∥ ≤ ε \lVert\nabla f\rVert\le\varepsilon ∥ ∇ f ∥ ≤ ε with O ( κ ( log ( μ 0 / μ ) + log ( ∥ ∇ f ( x 0 ) ∥ / ε ) ) ) O(\sqrt{\kappa}(\log(\mu_0/\mu)+\log(\lVert\nabla f(x_0)\rVert/\varepsilon))) O ( κ ( log ( μ 0 / μ ) + log (∥ ∇ f ( x 0 )∥ / ε ))) gradient evaluations from a guess μ 0 ≥ μ \mu_0 \ge\mu μ 0 ≥ μ . These methods count gradient evaluations, whereas we count subspace steps and bound the memory from below.
Oracle lower bounds [1 , 2 ] hold for all first-order methods and are attained when μ \mu μ is known. Nemirovski [32 ] extended them to methods that query the oracle on an affine manifold of fixed dimension at every step, covering exact subspace steps. In large dimension they still need of order κ log ( 1 / ε ) \sqrt{\kappa}\log(1/\varepsilon) κ log ( 1/ ε ) steps on F μ , L \mathcal{F}_{\mu,L} F μ , L , so no subspace method beats acceleration. The bounds of § 4 hold in a fixed dimension as ε → 0 \varepsilon\to0 ε → 0 and decide which memories reach it. Oblivious methods, whose coefficients do not depend on the data, cannot be accelerated without μ \mu μ [33 ] . Exact minimization is not oblivious, and our bounds cover it for memories fixed in advance. They rest on the interpolation theorem of Taylor, Hendrickx and Glineur [34 ] , which underlies performance estimation [35 ] and the exact rate of GD-ELS [36 ] , applied to infinite trajectories whose residuals keep a uniform margin (Lemma 4.7 ).
Setting and two tools
Inner products and norms are Euclidean. The class F μ , L ( R n ) \mathcal{F}_{\mu,L}(\mathbb{R}^{n}) F μ , L ( R n ) consists of differentiable μ \mu μ -strongly convex functions with L L L -Lipschitz gradients, 0 < μ ≤ L 0 < \mu\le L 0 < μ ≤ L , and κ = L / μ \kappa= L/\mu κ = L / μ , where the constants are valid bounds rather than the best ones. For a method generating points x k x_k x k we write s k = x k − x k − 1 s_k = x_k - x_{k-1} s k = x k − x k − 1 for k ≥ 1 k \ge1 k ≥ 1 and s 0 = 0 s_0 = 0 s 0 = 0 , and f ⋆ f^\star f ⋆ and x ⋆ x^\star x ⋆ are the minimum value and the minimizer. For 0 < ε ≤ 1 0 < \varepsilon\le1 0 < ε ≤ 1 the complexity N ( ε ) N(\varepsilon) N ( ε ) is the first k k k with f ( x k ) − f ⋆ ≤ ε ( f ( x 0 ) − f ⋆ ) f(x_k) - f^\star\le\varepsilon(f(x_0) - f^\star) f ( x k ) − f ⋆ ≤ ε ( f ( x 0 ) − f ⋆ ) .
Unless stated otherwise, the methods perform the exact subspace step
x k + 1 = arg min x ∈ x k + S k f ( x ) , ∇ f ( x k ) ∈ S k (1) x_{k+1} = \underset{x \in x_k+S_k}{\operatorname{arg\,min}}\ f(x), \qquad\nabla f(x_k) \in S_k \tag*{(1)} x k + 1 = x ∈ x k + S k arg min f ( x ) , ∇ f ( x k ) ∈ S k (1)
where the linear subspace S k S_k S k is built from the history by some rule. One iteration means one exact solution of (1) , whatever the inner solver. The window method of length m m m searches over the last m m m gradients and steps, $S_k = \operatorname{span}{\nabla f(x_i), s_i : k-m < i ≤ k i \le k i ≤ k . GD-ELS, gradient descent with exact line search, uses S k = span { ∇ f ( x k ) } S_k = \operatorname{span}\{\nabla f(x_k)\} S k = span { ∇ f ( x k )} , and the plane form of conjugate gradients uses S k = span { ∇ f ( x k ) , s k } S_k = \operatorname{span}\{\nabla f(x_k),s_k\} S k = span { ∇ f ( x k ) , s k } . SESOP with weights w 1 , … , w d w_1,\ldots,w_d w 1 , … , w d uses S k = span { ∇ f ( x k ) , s k , x k − x 0 , σ k 1 , … , σ k d } S_k = \operatorname{span}\{\nabla f(x_k),s_k,x_k-x_0,\sigma^1_k,\ldots,\sigma^d_k\} S k = span { ∇ f ( x k ) , s k , x k − x 0 , σ k 1 , … , σ k d } with the aggregates σ k a = ∑ i = 0 k w a ( i ) ∇ f ( x i ) \sigma^a_k = \sum_{i=0}^{k} w_a(i)\nabla f(x_i) σ k a = ∑ i = 0 k w a ( i ) ∇ f ( x i ) . Common choices are Nemirovski’s weights, w ( k ) = 1 2 + 1 4 + w ( k − 1 ) 2 w(k) = \frac{1}{2} + \sqrt{\frac{1}{4} + w(k-1)^2} w ( k ) = 2 1 + 4 1 + w ( k − 1 ) 2 with w ( 0 ) ∈ { 0 , 1 } w(0) \in\{0,1\} w ( 0 ) ∈ { 0 , 1 } , and the linear weights w ( i ) = i w(i)=i w ( i ) = i .
Our upper bounds use only the following properties of (1) .
Lemma 3.1 (exact subspace descent) Let f ∈ F μ , L ( R n ) f \in\mathcal{F}_{\mu,L}(\mathbb{R}^n) f ∈ F μ , L ( R n ) and let x k + 1 x_{k+1} x k + 1 be given by (1) . Then x k + 1 x_{k+1} x k + 1 is well defined, s k + 1 ∈ S k s_{k+1} \in S_k s k + 1 ∈ S k , ∇ f ( x k + 1 ) ⊥ S k \nabla f(x_{k+1}) \perp S_k ∇ f ( x k + 1 ) ⊥ S k , f ( x k + 1 ) ≤ f ( x k ) − ∥ ∇ f ( x k ) ∥ 2 2 L f(x_{k+1}) \le f(x_k) - \frac{\|\nabla f(x_k)\|^2}{2L} f ( x k + 1 ) ≤ f ( x k ) − 2 L ∥∇ f ( x k ) ∥ 2 , f ( x k + 1 ) − f ⋆ ≤ ( 1 − κ − 1 ) ( f ( x k ) − f ⋆ ) f(x_{k+1}) - f^\star\le(1-\kappa^{-1})(f(x_k)-f^\star) f ( x k + 1 ) − f ⋆ ≤ ( 1 − κ − 1 ) ( f ( x k ) − f ⋆ ) and 2 μ ( f ( x k ) − f ⋆ ) ≤ ∥ ∇ f ( x k ) ∥ 2 ≤ 2 L ( f ( x k ) − f ⋆ ) 2\mu(f(x_k)-f^\star) \le\|\nabla f(x_k)\|^2 \le2L(f(x_k)-f^\star) 2 μ ( f ( x k ) − f ⋆ ) ≤ ∥∇ f ( x k ) ∥ 2 ≤ 2 L ( f ( x k ) − f ⋆ ) .
Proof. Strong convexity gives a unique minimizer x k + 1 x_{k+1} x k + 1 , so s k + 1 ∈ S k s_{k+1} \in S_k s k + 1 ∈ S k , and optimality on x k + S k x_k+S_k x k + S k gives ∇ f ( x k + 1 ) ⊥ S k \nabla f(x_{k+1}) \perp S_k ∇ f ( x k + 1 ) ⊥ S k . Since x k − ∇ f ( x k ) / L ∈ x k + S k x_k-\nabla f(x_k)/L \in x_k+S_k x k − ∇ f ( x k ) / L ∈ x k + S k , the descent lemma gives f ( x k + 1 ) ≤ f ( x k − ∇ f ( x k ) / L ) ≤ f ( x k ) − ∥ ∇ f ( x k ) ∥ 2 / ( 2 L ) f(x_{k+1}) \le f(x_k-\nabla f(x_k)/L) \le f(x_k)-\|\nabla f(x_k)\|^2/(2L) f ( x k + 1 ) ≤ f ( x k − ∇ f ( x k ) / L ) ≤ f ( x k ) − ∥∇ f ( x k ) ∥ 2 / ( 2 L ) . As f ⋆ ≤ f ( x k + 1 ) f^\star\le f(x_{k+1}) f ⋆ ≤ f ( x k + 1 ) , this also yields ∥ ∇ f ( x k ) ∥ 2 ≤ 2 L ( f ( x k ) − f ⋆ ) \|\nabla f(x_k)\|^2 \le2L(f(x_k)-f^\star) ∥∇ f ( x k ) ∥ 2 ≤ 2 L ( f ( x k ) − f ⋆ ) . Minimizing the strong convexity inequality at x k x_k x k over the whole space gives 2 μ ( f ( x k ) − f ⋆ ) ≤ ∥ ∇ f ( x k ) ∥ 2 2\mu(f(x_k)-f^\star) \le\|\nabla f(x_k)\|^2 2 μ ( f ( x k ) − f ⋆ ) ≤ ∥∇ f ( x k ) ∥ 2 . Inserting this lower bound into the descent bound gives the contraction factor 1 − κ − 1 1-\kappa^{-1} 1 − κ − 1 .
The orthogonality ∇ f ( x k + 1 ) ⊥ S k \nabla f(x_{k+1}) \perp S_k ∇ f ( x k + 1 ) ⊥ S k plays the role of the coupling step of Nesterov’s method, and every lower bound must respect it. We call a method, or a family of methods indexed by κ \kappa κ , uniformly accelerated if there are absolute constants C , c > 0 C,c>0 C , c > 0 with
f ( x k ) − f ⋆ ≤ C exp ( − c k / κ ) ( f ( x 0 ) − f ⋆ ) (2) f(x_k)-f^\star\le C\exp(-ck/\sqrt{\kappa})(f(x_0)-f^\star) \tag*{(2)} f ( x k ) − f ⋆ ≤ C exp ( − c k / κ ) ( f ( x 0 ) − f ⋆ ) (2)
for every κ \kappa κ , every f ∈ F μ , L f \in\mathcal{F}_{\mu,L} f ∈ F μ , L with L / μ = κ L/\mu=\kappa L / μ = κ , every starting point and every k k k . A failure of (2) is a worst-case statement. It is compatible with the convex rate O ( 1 / k 2 ) O(1/k^2) O ( 1/ k 2 ) .
SESOP attains this rate and is conjugate gradients on quadratics, and the methods of § 5.2 that keep its directions inherit both known properties.
Proposition 3.2 (orth-method and conjugate gradients [7 , 8 , 9 ] ) Let the iterates satisfy (1) .
If S k S_k S k contains the anchor x k − x 0 x_k-x_0 x k − x 0 and the aggregate σ k = ∑ i = 0 k w i ∇ f ( x i ) \sigma_k=\sum_{i=0}^{k}w_i\nabla f(x_i) σ k = ∑ i = 0 k w i ∇ f ( x i ) with Nemirovski’s weights and w 0 = 1 w_0=1 w 0 = 1 , if f f f is convex with L L L -Lipschitz gradient and a minimizer x ⋆ x^\star x ⋆ , and if the subspace minimizers exist, then f ( x k ) − f ⋆ ≤ 2 L ∥ x 0 − x ⋆ ∥ 2 / ( k + 1 ) 2 f(x_k)-f^\star\le2L\|x_0-x^\star\|^2/(k+1)^2 f ( x k ) − f ⋆ ≤ 2 L ∥ x 0 − x ⋆ ∥ 2 / ( k + 1 ) 2 for k ≥ 1 k\ge1 k ≥ 1 .
If f ( x ) = 1 2 ( x − x ⋆ ) T A ( x − x ⋆ ) f(x)=\frac{1}{2}(x-x^\star)^\mathsf{T}A(x-x^\star) f ( x ) = 2 1 ( x − x ⋆ ) T A ( x − x ⋆ ) with A ≻ 0 A\succ0 A ≻ 0 and span { ∇ f ( x k ) , s k } ⊆ S k ⊆ span { ∇ f ( x 0 ) , … , ∇ f ( x k ) , s 1 , … , s k } \operatorname{span}\{\nabla f(x_k),s_k\}\subseteq S_k\subseteq\operatorname{span}\{\nabla f(x_0),\ldots,\nabla f(x_k),s_1,\ldots,s_k\} span { ∇ f ( x k ) , s k } ⊆ S k ⊆ span { ∇ f ( x 0 ) , … , ∇ f ( x k ) , s 1 , … , s k } for every k k k , then the iterates coincide with those of conjugate gradients started at x 0 x_0 x 0 , and f ( x k ) − f ⋆ ≤ 4 ( κ − 1 κ + 1 ) 2 k ( f ( x 0 ) − f ⋆ ) f(x_k)-f^\star\le4\left(\frac{\sqrt{\kappa}-1}{\sqrt{\kappa}+1}\right)^{2k}(f(x_0)-f^\star) f ( x k ) − f ⋆ ≤ 4 ( κ + 1 κ − 1 ) 2 k ( f ( x 0 ) − f ⋆ ) .
Proof. Part 1 is Nemirovski’s convex rate for the orth-method [8 ] . Its proof for SESOP [9 Theorem 1] uses three facts: exact minimization makes each gradient ∇ f ( x i ) \nabla f(x_i) ∇ f ( x i ) orthogonal to the anchor and to the previous aggregate, each step decreases the last iterate as much as the gradient step of length 1 / L 1/L 1/ L , and Nemirovski’s weights satisfy w k 2 = ∑ i ≤ k w i ≥ ( k + 2 ) 2 / 4 w_k^2=\sum_{i\le k}w_i\ge(k+2)^2/4 w k 2 = ∑ i ≤ k w i ≥ ( k + 2 ) 2 /4 . With convexity they give w k 2 ( f ( x k + 1 ) − f ⋆ ) ≤ L 2 ∥ x 0 − x ⋆ ∥ 2 w_k^2(f(x_{k+1})-f^\star)\le\frac{L}{2}\|x_0-x^\star\|^2 w k 2 ( f ( x k + 1 ) − f ⋆ ) ≤ 2 L ∥ x 0 − x ⋆ ∥ 2 , which is the stated bound. For part 2, the k k k th point of conjugate gradients minimizes f f f over x 0 x_0 x 0 plus the Krylov space of order k k k , and its step lies in span { ∇ f ( x k ) , s k } \operatorname{span}\{\nabla f(x_k),s_k\} span { ∇ f ( x k ) , s k } . If the iterates agree with conjugate gradients up to k k k , then x k + S k x_k+S_k x k + S k lies in x 0 x_0 x 0 plus the next Krylov space and contains the next point of conjugate gradients, which is therefore its minimizer. The rate is the standard bound for conjugate gradients [37 Chapter 5] . □
The first of our two tools is an estimate sequence in the spirit of Nesterov [2 ] , with the lower model taken at the new point. It shows that an exact step is accelerated whenever a virtual momentum π k \pi_k π k , which is never computed, lies in the search space, and all our upper bounds rest on it. Fix an origin b b b , a number 0 < λ ≤ μ 0<\lambda\le\mu 0 < λ ≤ μ that plays the role of μ \mu μ , L ′ ≥ L L'\ge L L ′ ≥ L and numbers a k ∈ ( 0 , 1 ] a_k\in(0,1] a k ∈ ( 0 , 1 ] with a k 2 / λ ≤ 1 / L ′ a_k^2/\lambda\le1/L' a k 2 / λ ≤ 1/ L ′ for k ≥ b k\ge b k ≥ b that play the role of μ / L \sqrt{\mu/L} μ / L . With ρ k = 1 − a k \rho_k=1-a_k ρ k = 1 − a k and some π b ∈ R n \pi_b\in\mathbb{R}^n π b ∈ R n define
π k + 1 = ρ k ( π k + s k + 1 ) + a k λ ∇ f ( x k + 1 ) ( k ≥ b ) . (3) \pi_{k+1}=\rho_k(\pi_k+s_{k+1})+\frac{a_k}{\lambda}\nabla f(x_{k+1}) \qquad(k\ge b). \tag*{(3)} π k + 1 = ρ k ( π k + s k + 1 ) + λ a k ∇ f ( x k + 1 ) ( k ≥ b ) . (3)
Inexact steps are measured by the defect
E k = [ ⟨ ∇ f ( x k + 1 ) , π k + s k + 1 ⟩ − λ 2 ∥ π k + s k + 1 ∥ 2 ] + , e b = 0 , e k + 1 = ρ k ( e k + a k E k ) . E_k=\left[\left\langle\nabla f(x_{k+1}),\pi_k+s_{k+1}\right\rangle-\frac{\lambda}{2}\|\pi_k+s_{k+1}\|^2\right]_+,\qquad e_b=0,\qquad e_{k+1}=\rho_k(e_k+a_kE_k). E k = [ ⟨ ∇ f ( x k + 1 ) , π k + s k + 1 ⟩ − 2 λ ∥ π k + s k + 1 ∥ 2 ] + , e b = 0 , e k + 1 = ρ k ( e k + a k E k ) .
Lemma 3.3 (estimate-sequence certificate) Let f ∈ F μ , L f\in\mathcal{F}_{\mu,L} f ∈ F μ , L and let f ( x k + 1 ) ≤ f ( x k ) − ∥ ∇ f ( x k ) ∥ 2 / ( 2 L ′ ) f(x_{k+1})\le f(x_k)-\|\nabla f(x_k)\|^2/(2L') f ( x k + 1 ) ≤ f ( x k ) − ∥∇ f ( x k ) ∥ 2 / ( 2 L ′ ) for k ≥ b k\ge b k ≥ b . Then for j ≥ 0 j\ge0 j ≥ 0
f ( x b + j + 1 ) − f ⋆ ≤ ( ∏ i = b b + j − 1 ρ i ) C b + e b + j , C b = f ( x b ) − f ⋆ + λ 2 ∥ x b − π b − x ⋆ ∥ 2 . f(x_{b+j+1})-f^\star\le\left(\prod_{i=b}^{b+j-1}\rho_i\right)C_b+e_{b+j},\qquad C_b=f(x_b)-f^\star+\frac{\lambda}{2}\|x_b-\pi_b-x^\star\|^2. f ( x b + j + 1 ) − f ⋆ ≤ ( i = b ∏ b + j − 1 ρ i ) C b + e b + j , C b = f ( x b ) − f ⋆ + 2 λ ∥ x b − π b − x ⋆ ∥ 2 . If π k + s k + 1 \pi_k+s_{k+1} π k + s k + 1 lies in a subspace S S S , then E k ≤ ∥ P S ∇ f ( x k + 1 ) ∥ 2 / ( 2 λ ) E_k\le\|P_S\nabla f(x_{k+1})\|^2/(2\lambda) E k ≤ ∥ P S ∇ f ( x k + 1 ) ∥ 2 / ( 2 λ ) , where P S P_S P S is the orthogonal projector onto S S S . For iterates of (1) with L ′ = L L'=L L ′ = L and π k ∈ S k \pi_k\in S_k π k ∈ S k for b ≤ k < b + j b\le k<b+j b ≤ k < b + j the defect e b + j e_{b+j} e b + j vanishes. For π b = 0 \pi_b=0 π b = 0 , C b ≤ 2 ( f ( x b ) − f ⋆ ) C_b\le2(f(x_b)-f^\star) C b ≤ 2 ( f ( x b ) − f ⋆ ) .
Proof. Let ϕ b ( x ) = f ( x b ) + λ 2 ∥ x − x b + π b ∥ 2 \phi_b(x)=f(x_b)+\frac{\lambda}{2}\|x-x_b+\pi_b\|^2 ϕ b ( x ) = f ( x b ) + 2 λ ∥ x − x b + π b ∥ 2 and, for k ≥ b k\ge b k ≥ b ,
ϕ k + 1 ( x ) = ρ k ϕ k ( x ) + a k ( f ( x k + 1 ) + ⟨ ∇ f ( x k + 1 ) , x − x k + 1 ⟩ + λ 2 ∥ x − x k + 1 ∥ 2 ) . \phi_{k+1}(x)=\rho_k\phi_k(x)+a_k\left(f(x_{k+1})+\langle\nabla f(x_{k+1}),x-x_{k+1}\rangle+\frac{\lambda}{2}\|x-x_{k+1}\|^2\right). ϕ k + 1 ( x ) = ρ k ϕ k ( x ) + a k ( f ( x k + 1 ) + ⟨ ∇ f ( x k + 1 ) , x − x k + 1 ⟩ + 2 λ ∥ x − x k + 1 ∥ 2 ) .
Each ϕ k \phi_k ϕ k equals ϕ k ⋆ + λ 2 ∥ x − v k ∥ 2 \phi_k^\star+\frac{\lambda}{2}\|x-v_k\|^2 ϕ k ⋆ + 2 λ ∥ x − v k ∥ 2 with v b = x b − π b v_b=x_b-\pi_b v b = x b − π b . The minimizers satisfy v k + 1 = ρ k v k + a k x k + 1 − a k λ ∇ f ( x k + 1 ) v_{k+1}=\rho_kv_k+a_kx_{k+1}-\frac{a_k}{\lambda}\nabla f(x_{k+1}) v k + 1 = ρ k v k + a k x k + 1 − λ a k ∇ f ( x k + 1 ) , so x k − v k x_k-v_k x k − v k satisfies (3) and equals π k \pi_k π k . Completing the square gives
ϕ k + 1 ⋆ = ρ k ϕ k ⋆ + a k f ( x k + 1 ) − a k 2 2 λ ∥ ∇ f ( x k + 1 ) ∥ 2 − a k ρ k ( ⟨ ∇ f ( x k + 1 ) , π k + s k + 1 ⟩ − λ 2 ∥ π k + s k + 1 ∥ 2 ) . \phi_{k+1}^\star=\rho_k\phi_k^\star+a_kf(x_{k+1})-\frac{a_k^2}{2\lambda}\|\nabla f(x_{k+1})\|^2-a_k\rho_k\left(\left\langle\nabla f(x_{k+1}),\pi_k+s_{k+1}\right\rangle-\frac{\lambda}{2}\|\pi_k+s_{k+1}\|^2\right). ϕ k + 1 ⋆ = ρ k ϕ k ⋆ + a k f ( x k + 1 ) − 2 λ a k 2 ∥∇ f ( x k + 1 ) ∥ 2 − a k ρ k ( ⟨ ∇ f ( x k + 1 ) , π k + s k + 1 ⟩ − 2 λ ∥ π k + s k + 1 ∥ 2 ) .
Let us show ϕ k ⋆ ≥ f ( x k + 1 ) − e k \phi_k^\star\ge f(x_{k+1})-e_k ϕ k ⋆ ≥ f ( x k + 1 ) − e k by induction. At k = b k=b k = b this is f ( x b ) ≥ f ( x b + 1 ) f(x_b)\ge f(x_{b+1}) f ( x b ) ≥ f ( x b + 1 ) . The step uses the definition of E k E_k E k , a k 2 / λ ≤ 1 / L ′ a_k^2/\lambda\le1/L' a k 2 / λ ≤ 1/ L ′ and the descent assumption:
ϕ k + 1 ⋆ ≥ ρ k ( f ( x k + 1 ) − e k ) + a k f ( x k + 1 ) − ∥ ∇ f ( x k + 1 ) ∥ 2 2 L ′ − a k ρ k E k ≥ f ( x k + 2 ) − e k + 1 . \phi_{k+1}^\star\ge\rho_k\bigl(f(x_{k+1})-e_k\bigr)+a_kf(x_{k+1})-\frac{\|\nabla f(x_{k+1})\|^2}{2L'}-a_k\rho_kE_k\ge f(x_{k+2})-e_{k+1}. ϕ k + 1 ⋆ ≥ ρ k ( f ( x k + 1 ) − e k ) + a k f ( x k + 1 ) − 2 L ′ ∥∇ f ( x k + 1 ) ∥ 2 − a k ρ k E k ≥ f ( x k + 2 ) − e k + 1 .
Since λ ≤ μ \lambda\le\mu λ ≤ μ , the added quadratics minorize f f f , so ϕ k + 1 ( x ⋆ ) − f ⋆ ≤ ρ k ( ϕ k ( x ⋆ ) − f ⋆ ) \phi_{k+1}(x^\star)-f^\star\le\rho_k(\phi_k(x^\star)-f^\star) ϕ k + 1 ( x ⋆ ) − f ⋆ ≤ ρ k ( ϕ k ( x ⋆ ) − f ⋆ ) and ϕ b ( x ⋆ ) − f ⋆ = C b \phi_b(x^\star)-f^\star=C_b ϕ b ( x ⋆ ) − f ⋆ = C b . Together with the induction this gives
f ( x b + j + 1 ) − f ⋆ − e b + j ≤ ϕ b + j ⋆ − f ⋆ ≤ ϕ b + j ( x ⋆ ) − f ⋆ ≤ ( ∏ i = b b + j − 1 ρ i ) C b . f(x_{b+j+1})-f^\star-e_{b+j}\le\phi_{b+j}^\star-f^\star\le\phi_{b+j}(x^\star)-f^\star\le\left(\prod_{i=b}^{b+j-1}\rho_i\right)C_b. f ( x b + j + 1 ) − f ⋆ − e b + j ≤ ϕ b + j ⋆ − f ⋆ ≤ ϕ b + j ( x ⋆ ) − f ⋆ ≤ ( i = b ∏ b + j − 1 ρ i ) C b .
If π k + s k + 1 ∈ S \pi_k+s_{k+1}\in S π k + s k + 1 ∈ S , the bracket in E k E_k E k is at most max u ∈ S { ⟨ P S ∇ f ( x k + 1 ) , u ⟩ − λ 2 ∥ u ∥ 2 } = ∥ P S ∇ f ( x k + 1 ) ∥ 2 2 λ \max_{u\in S}\{\langle P_S\nabla f(x_{k+1}),u\rangle-\frac{\lambda}{2}\lVert u\rVert^2\}=\frac{\lVert P_S\nabla f(x_{k+1})\rVert^2}{2\lambda} max u ∈ S {⟨ P S ∇ f ( x k + 1 ) , u ⟩ − 2 λ ∥ u ∥ 2 } = 2 λ ∥ P S ∇ f ( x k + 1 ) ∥ 2 . For iterates of (1) Lemma 3.1 gives the descent assumption with L ′ = L L'=L L ′ = L , s k + 1 ∈ S k s_{k+1}\in S_k s k + 1 ∈ S k and ∇ f ( x k + 1 ) ⊥ S k \nabla f(x_{k+1})\perp S_k ∇ f ( x k + 1 ) ⊥ S k , so π k ∈ S k \pi_k\in S_k π k ∈ S k gives ⟨ ∇ f ( x k + 1 ) , π k + s k + 1 ⟩ = 0 \langle\nabla f(x_{k+1}),\pi_k+s_{k+1}\rangle=0 ⟨ ∇ f ( x k + 1 ) , π k + s k + 1 ⟩ = 0 and E k = 0 E_k=0 E k = 0 . For π b = 0 \pi_b=0 π b = 0 , λ 2 ∥ x b − x ⋆ ∥ 2 ≤ μ 2 ∥ x b − x ⋆ ∥ 2 ≤ f ( x b ) − f ⋆ \frac{\lambda}{2}\lVert x_b-x^\star\rVert^2\leq\frac{\mu}{2}\lVert x_b-x^\star\rVert^2\leq f(x_b)-f^\star 2 λ ∥ x b − x ⋆ ∥ 2 ≤ 2 μ ∥ x b − x ⋆ ∥ 2 ≤ f ( x b ) − f ⋆ . □
The second tool builds the hard functions. The lower bounds prescribe points, gradients and values and extend them to a function of the class. For data ( x i , g i , f i ) (x_i,g_i,f_i) ( x i , g i , f i ) the ordered interpolation residual is
C i j = f i − f j − ⟨ g j , x i − x j ⟩ − 1 L ∥ g i − g j ∥ 2 + μ ∥ x i − x j ∥ 2 − 2 μ L ⟨ g i − g j , x i − x j ⟩ 2 ( 1 − μ / L ) . (4) C_{ij}=f_i-f_j-\langle g_j,x_i-x_j\rangle-\frac{1}{L}\lVert g_i-g_j\rVert^2+\mu\lVert x_i-x_j\rVert^2-\frac{\frac{2\mu}{L}\langle g_i-g_j,x_i-x_j\rangle}{2(1-\mu/L)}. \tag*{(4)} C ij = f i − f j − ⟨ g j , x i − x j ⟩ − L 1 ∥ g i − g j ∥ 2 + μ ∥ x i − x j ∥ 2 − 2 ( 1 − μ / L ) L 2 μ ⟨ g i − g j , x i − x j ⟩ . (4)
For finite data, C i j ≥ 0 C_{ij}\geq0 C ij ≥ 0 is the interpolation condition of [34 Theorem 4] . The extension to countable bounded data is routine, and we include its proof for completeness.
Lemma 3.4 (bounded countable interpolation) Let 0 < μ < L 0<\mu<L 0 < μ < L and let { ( x i , g i , f i ) } i ∈ I \{(x_i,g_i,f_i)\}_{i\in I} {( x i , g i , f i ) } i ∈ I be a nonempty finite or countable bounded subset of R n × R n × R \mathbb{R}^n\times\mathbb{R}^n\times\mathbb{R} R n × R n × R with C i j ≥ 0 C_{ij}\geq0 C ij ≥ 0 for all i , j ∈ I i,j\in I i , j ∈ I . Then there is F ∈ F μ , L ( R n ) F\in\mathcal{F}_{\mu,L}(\mathbb{R}^n) F ∈ F μ , L ( R n ) with F ( x i ) = f i F(x_i)=f_i F ( x i ) = f i and ∇ F ( x i ) = g i \nabla F(x_i)=g_i ∇ F ( x i ) = g i for every i ∈ I i\in I i ∈ I . If the triple ( 0 , 0 , 0 ) (0,0,0) ( 0 , 0 , 0 ) belongs to the data, then 0 0 0 is the minimizer of F F F and F ⋆ = 0 F^\star=0 F ⋆ = 0 .
Proof. As in [34 ] , we build the function through its conjugate. Put a = L − μ > 0 a=L-\mu>0 a = L − μ > 0 , h i = f i − μ ∥ x i ∥ 2 / 2 h_i=f_i-\mu\lVert x_i\rVert^2/2 h i = f i − μ ∥ x i ∥ 2 /2 , q i = g i − μ x i q_i=g_i-\mu x_i q i = g i − μ x i and u i = ⟨ q i , x i ⟩ − h i u_i=\langle q_i,x_i\rangle-h_i u i = ⟨ q i , x i ⟩ − h i . Expanding ∥ q i − q j ∥ 2 \lVert q_i-q_j\rVert^2 ∥ q i − q j ∥ 2 shows that C i j C_{ij} C ij is the left-hand side minus the right-hand side of
h i ≥ h j + ⟨ q j , x i − x j ⟩ + 1 2 a ∥ q i − q j ∥ 2 , (5) h_i\geq h_j+\langle q_j,x_i-x_j\rangle+\frac{1}{2a}\lVert q_i-q_j\rVert^2, \tag*{(5)} h i ≥ h j + ⟨ q j , x i − x j ⟩ + 2 a 1 ∥ q i − q j ∥ 2 , (5)
so the assumption C i j ≥ 0 C_{ij}\geq0 C ij ≥ 0 is exactly (5) . Define
Ψ ( y ) = sup i ∈ I { u i + ⟨ x i , y − q i ⟩ + 1 2 a ∥ y − q i ∥ 2 } . \Psi(y)=\sup_{i\in I}\left\{u_i+\langle x_i,y-q_i\rangle+\frac{1}{2a}\lVert y-q_i\rVert^2\right\}. Ψ ( y ) = i ∈ I sup { u i + ⟨ x i , y − q i ⟩ + 2 a 1 ∥ y − q i ∥ 2 } .
After subtracting ∥ y ∥ 2 / ( 2 a ) \lVert y\rVert^2/(2a) ∥ y ∥ 2 / ( 2 a ) , every function in the supremum is affine with slope and intercept bounded uniformly in i i i , because the data are bounded. Hence Ψ \Psi Ψ is finite, continuous and 1 / a 1/a 1/ a -strongly convex. At y = q j y=q_j y = q j the i i i th term minus the j j j th term u j u_j u j equals h j − h i + ⟨ q j , x i − x j ⟩ + ∥ q j − q i ∥ 2 / ( 2 a ) ≤ 0 h_j-h_i+\langle q_j,x_i-x_j\rangle+\lVert q_j-q_i\rVert^2/(2a)\leq0 h j − h i + ⟨ q j , x i − x j ⟩ + ∥ q j − q i ∥ 2 / ( 2 a ) ≤ 0 , by (5) . Therefore Ψ ( q j ) = u j \Psi(q_j)=u_j Ψ ( q j ) = u j , the j j j th quadratic touches Ψ \Psi Ψ from below at q j q_j q j , and x j ∈ ∂ Ψ ( q j ) x_j\in\partial\Psi(q_j) x j ∈ ∂ Ψ ( q j ) . By [38 Proposition 12.60] the conjugate h = Ψ ∗ h=\Psi^\ast h = Ψ ∗ is convex and differentiable with an a a a -Lipschitz gradient. The inclusion x j ∈ ∂ Ψ ( q j ) x_j\in\partial\Psi(q_j) x j ∈ ∂ Ψ ( q j ) is equivalent to ∇ h ( x j ) = q j \nabla h(x_j)=q_j ∇ h ( x j ) = q j , and the Fenchel–Young equality gives h ( x j ) = ⟨ x j , q j ⟩ − u j = h j h(x_j)=\langle x_j,q_j\rangle-u_j=h_j h ( x j ) = ⟨ x j , q j ⟩ − u j = h j . Hence F ( x ) = h ( x ) + μ 2 ∥ x ∥ 2 F(x)=h(x)+\frac{\mu}{2}\lVert x\rVert^2 F ( x ) = h ( x ) + 2 μ ∥ x ∥ 2 belongs to F μ , L ( R n ) \mathcal{F}_{\mu,L}(\mathbb{R}^n) F μ , L ( R n ) and satisfies F ( x i ) = f i F(x_i)=f_i F ( x i ) = f i and ∇ F ( x i ) = g i \nabla F(x_i)=g_i ∇ F ( x i ) = g i . If ( 0 , 0 , 0 ) (0,0,0) ( 0 , 0 , 0 ) is in the data, then ∇ F ( 0 ) = 0 \nabla F(0)=0 ∇ F ( 0 ) = 0 and F ⋆ = F ( 0 ) = 0 F^\star=F(0)=0 F ⋆ = F ( 0 ) = 0 . □
When memory blocks acceleration
By Lemma 3.3 a method is accelerated once its subspaces contain the virtual momentum π k \pi_k π k , an average over roughly the last κ \sqrt{\kappa} κ iterations. For memories fixed in advance this time scale is also necessary. A memory whose weights do not decay on it keeps directions far older than the horizon (§ 4.2 ), and one that forgets much faster sees only its last few gradients (§ 4.3 ). § 4.4 combines the two bounds, and § 4.5 shows that smoothness does not help.
Memories fixed in advance
SESOP combines a window of length one with the anchor x k − x 0 x_k-x_0 x k − x 0 , which sums all past steps, and with weighted sums of all past gradients, so it never forgets anything.
Definition 4.1 (windows with aggregates) A memory with window m m m and aggregates consists of the last m m m gradients and steps, finitely many gradient sums σ k a = ∑ i = 0 k w a ( i ) ∇ f ( x i ) \sigma_k^a=\sum_{i=0}^k w_a(i)\nabla f(x_i) σ k a = ∑ i = 0 k w a ( i ) ∇ f ( x i ) and finitely many step sums θ k b = ∑ i = 0 k v b ( i ) s i \theta_k^b=\sum_{i=0}^k v_b(i)s_i θ k b = ∑ i = 0 k v b ( i ) s i . The weight sequences are fixed before the run and do not depend on the objective. An exact subspace method uses this memory if for every k ≥ 0 k\ge0 k ≥ 0
∇ f ( x k ) ∈ S k ⊆ span { ∇ f ( x i ) , s i : k − m < i ≤ k } ∪ { σ k 1 , … , σ k q , θ k 1 , … , θ k q ′ } . \nabla f(x_k)\in S_k\subseteq\operatorname{span}\{\nabla f(x_i),s_i:k-m<i\leq k\}\cup\{\sigma_k^1,\ldots,\sigma_k^q,\theta_k^1,\ldots,\theta_k^{q'}\}. ∇ f ( x k ) ∈ S k ⊆ span { ∇ f ( x i ) , s i : k − m < i ≤ k } ∪ { σ k 1 , … , σ k q , θ k 1 , … , θ k q ′ } . The choice of S k S_k S k inside this space may depend on the points, values and gradients observed at x 0 , … , x k x_0,\ldots,x_k x 0 , … , x k .
Multiplying a register by a nonzero scalar does not change its direction, so forgetting is measured on normalized weights. A linear register Z k = Z k − 1 A k + [ ∇ f ( x k ) , s k ] B k Z_k=Z_{k-1}A_k+[\nabla f(x_k),s_k]B_k Z k = Z k − 1 A k + [ ∇ f ( x k ) , s k ] B k with invertible matrices A k A_k A k fixed in advance and Z − 1 = 0 Z_{-1}=0 Z − 1 = 0 satisfies
Z k Φ k − 1 = ∑ i = 0 k [ ∇ f ( x i ) , s i ] B i Φ i − 1 Z_k\Phi_k^{-1}=\sum_{i=0}^k[\nabla f(x_i),s_i]B_i\Phi_i^{-1} Z k Φ k − 1 = i = 0 ∑ k [ ∇ f ( x i ) , s i ] B i Φ i − 1
with Φ k = A 0 ⋯ A k \Phi_k=A_0\cdots A_k Φ k = A 0 ⋯ A k , so its span lies in the span of gradient and step sums whose normalized weights are the entries of B i Φ i − 1 B_i\Phi_i^{-1} B i Φ i − 1 . The raw weights of the geometric filter z k = ρ z k − 1 + ∇ f ( x k ) z_k=\rho z_{k-1}+\nabla f(x_k) z k = ρ z k − 1 + ∇ f ( x k ) decay, while its normalized weights ρ − i \rho^{-i} ρ − i grow.
The growth rate Γ \Gamma Γ and the forgetting index δ \delta δ of the memory are
Γ = max { 1 , max a lim sup i → ∞ ∣ w a ( i ) ∣ 1 / i , max b lim sup i → ∞ ∣ v b ( i ) ∣ 1 / i } , δ = 1 − 1 Γ . \Gamma=\max\left\{1,\max_a\limsup_{i\to\infty}|w_a(i)|^{1/i},\max_b\limsup_{i\to\infty}|v_b(i)|^{1/i}\right\},\qquad\delta=1-\frac{1}{\Gamma}. Γ = max { 1 , a max i → ∞ lim sup ∣ w a ( i ) ∣ 1/ i , b max i → ∞ lim sup ∣ v b ( i ) ∣ 1/ i } , δ = 1 − Γ 1 .
A maximum over an empty family is omitted, so a pure window has Γ = 1 \Gamma=1 Γ = 1 and δ = 0 \delta=0 δ = 0 . A geometric filter with factor ρ ∈ ( 0 , 1 ] \rho\in(0,1] ρ ∈ ( 0 , 1 ] has δ = 1 − ρ \delta=1-\rho δ = 1 − ρ . We call the memory slowly forgetting if Γ = 1 \Gamma=1 Γ = 1 , that is, if all its weights grow subexponentially. GD-ELS, the plane rule, windows and SESOP with bounded, polynomial or Nemirovski’s weights are slowly forgetting. A filter with factor 1 − 1 / κ 1-1/\kappa 1 − 1/ κ forgets exponentially, but δ = 1 / κ \delta=1/\kappa δ = 1/ κ is small on the scale of acceleration.
Remembering too much
Our first lower bound concerns memories with a window of length m ≤ p − 2 m\leq p-2 m ≤ p − 2 and weights that grow more slowly than 1 / r 1/r 1/ r , where r = 1 − p / κ r=1-p/\kappa r = 1 − p / κ . For such a memory one function makes every method that uses it follow the same trajectory, on which the gap shrinks only by the factor r 2 r^2 r 2 per step.
Theorem 4.2 (slowly forgetting memories) Fix a memory of Definition 4.1 with window m m m and forgetting index δ \delta δ . Let p ≥ max ( 8 , m + 2 ) p\geq\max(8,m+2) p ≥ max ( 8 , m + 2 ) be an integer, κ ≥ 100 p 2 \kappa\geq100p^2 κ ≥ 100 p 2 and δ < p / κ \delta<p/\kappa δ < p / κ . Then there are a dimension n n n , a function F ∈ F 1 / κ , 1 ( R n ) F\in\mathcal{F}_{1/\kappa,1}(\mathbb{R}^n) F ∈ F 1/ κ , 1 ( R n ) and a starting point such that every exact subspace method (1) using this memory generates the same trajectory on F F F , and F ( x k + 1 ) − F ⋆ = ( 1 − p / κ ) 2 ( F ( x k ) − F ⋆ ) F(x_{k+1})-F^\star=(1-p/\kappa)^2(F(x_k)-F^\star) F ( x k + 1 ) − F ⋆ = ( 1 − p / κ ) 2 ( F ( x k ) − F ⋆ ) for all large κ \kappa κ . Hence lim ε ↓ 0 N ( ε ) / log ( 1 / ε ) ≥ κ / ( 4 p ) \lim_{\varepsilon\downarrow0}N(\varepsilon)/\log(1/\varepsilon)\geq\kappa/(4p) lim ε ↓ 0 N ( ε ) / log ( 1/ ε ) ≥ κ / ( 4 p ) . The same holds for m = 1 m=1 m = 1 , p = 3 p=3 p = 3 and every κ ≥ 12 \kappa\geq12 κ ≥ 12 .
The proof of Theorem 4.2 is given at the end of this subsection.
Corollary 4.3 (SESOP and conjugate gradients) Let SESOP use any finite number of aggregates with subexponential weights, in particular Nemirovski’s or the linear weights, with or without the previous step and the anchor. For every κ ≥ 12 \kappa\ge12 κ ≥ 12 there is F ∈ F 1 / κ , 1 F \in\mathcal{F}_{1/\kappa,1} F ∈ F 1/ κ , 1 on which lim inf ε ↓ 0 N ( ε ) / log ( 1 / ε ) ≥ κ / 12 \liminf_{\varepsilon\downarrow0} N(\varepsilon)/\log(1/\varepsilon) \ge\kappa/12 lim inf ε ↓ 0 N ( ε ) / log ( 1/ ε ) ≥ κ /12 . The plane form of conjugate gradients and GD-ELS follow the same trajectory. None of these methods is uniformly accelerated.
Proof. These memories have the window m = 1 m=1 m = 1 and subexponential weights: the anchor is the step sum with weights 1 1 1 , and Nemirovski’s weights satisfy w ( k − 1 ) + 1 2 ≤ w ( k ) ≤ w ( k − 1 ) + 1 w(k-1)+\frac{1}{2} \le w(k) \le w(k-1)+1 w ( k − 1 ) + 2 1 ≤ w ( k ) ≤ w ( k − 1 ) + 1 . Hence Γ = 1 \Gamma=1 Γ = 1 , and Theorem 4.2 with m = 1 m=1 m = 1 and p = 3 p=3 p = 3 applies to every rule using this memory, which includes all methods of the corollary. By (2) a uniformly accelerated method has lim sup ε ↓ 0 N ( ε ) / log ( 1 / ε ) ≤ κ / c \limsup_{\varepsilon\downarrow0} N(\varepsilon)/\log(1/\varepsilon) \le\kappa/c lim sup ε ↓ 0 N ( ε ) / log ( 1/ ε ) ≤ κ / c , which is below κ / 12 \kappa/12 κ /12 for large κ \kappa κ .
The constant 1 / 12 1/12 1/12 is not sharp. The trajectory contracts by ( 1 − 3 / κ ) 2 (1-3/\kappa)^2 ( 1 − 3/ κ ) 2 per step, so the limit is about κ / 6 \kappa/6 κ /6 for large κ \kappa κ , and the proof loses a factor two in − log ( 1 − u ) ≤ 2 u -\log(1-u) \le2u − log ( 1 − u ) ≤ 2 u .
For the proof we prescribe points x k x_k x k , gradients g k g_k g k (which become ∇ F ( x k ) \nabla F(x_k) ∇ F ( x k ) ) and values f k f_k f k along an infinite trajectory together with the minimizer 0 0 0 , and Lemma 3.4 extends the data to a function of the class if all residuals are nonnegative. If every step is a negative multiple of the current gradient and every next gradient is orthogonal to the whole memory, the method with the largest admissible spaces generates the trajectory, and by the next lemma so does every method using the memory. The trajectory consists of a cyclic tail (Lemma 4.5 ) and an orthogonal initial segment (Lemma 4.6 ), which Lemma 4.7 joins, and Lemma 4.8 chooses the step lengths of the segment.
Lemma 4.4 (transfer) Let f f f be differentiable and strongly convex, and let an exact subspace method with spaces S k S_k S k generate a trajectory with s k + 1 ∈ span { ∇ f ( x k ) } s_{k+1} \in\operatorname{span}\{\nabla f(x_k)\} s k + 1 ∈ span { ∇ f ( x k )} for all k k k . Let another rule produce, along the same history, spaces with span { ∇ f ( x k ) } ⊆ S ~ k ⊆ S k \operatorname{span}\{\nabla f(x_k)\} \subseteq\widetilde{S}_k \subseteq S_k span { ∇ f ( x k )} ⊆ S k ⊆ S k . Then the second rule, started at the same x 0 x_0 x 0 , generates the same trajectory.
Proof. Assume that both methods agree up to iteration k k k , so that the second method forms S ~ k \widetilde{S}_k S k on the same history. The first inclusion gives x k + 1 ∈ x k + S ~ k x_{k+1} \in x_k+\widetilde{S}_k x k + 1 ∈ x k + S k , and optimality of the original step gives ∇ f ( x k + 1 ) ⊥ S k ⊇ S ~ k \nabla f(x_{k+1}) \perp S_k \supseteq\widetilde{S}_k ∇ f ( x k + 1 ) ⊥ S k ⊇ S k . Hence for z ∈ x k + S ~ k z \in x_k+\widetilde{S}_k z ∈ x k + S k strong convexity yields
f ( z ) ≥ f ( x k + 1 ) + ⟨ ∇ f ( x k + 1 ) , z − x k + 1 ⟩ + μ 2 ∥ z − x k + 1 ∥ 2 = f ( x k + 1 ) + μ 2 ∥ z − x k + 1 ∥ 2 . f(z) \ge f(x_{k+1})+\langle\nabla f(x_{k+1}),z-x_{k+1}\rangle+\frac{\mu}{2}\lVert z-x_{k+1}\rVert^2=f(x_{k+1})+\frac{\mu}{2}\lVert z-x_{k+1}\rVert^2. f ( z ) ≥ f ( x k + 1 ) + ⟨ ∇ f ( x k + 1 ) , z − x k + 1 ⟩ + 2 μ ∥ z − x k + 1 ∥ 2 = f ( x k + 1 ) + 2 μ ∥ z − x k + 1 ∥ 2 .
so x k + 1 x_{k+1} x k + 1 is the unique minimizer over x k + S ~ k x_k+\widetilde{S}_k x k + S k . Induction completes the proof.
Throughout this subsection and the next L = 1 L=1 L = 1 , μ ˉ = 1 − μ \bar{\mu}=1-\mu μ ˉ = 1 − μ , C i j C_{ij} C ij is the residual (4) , and the index k k k refers to the minimizer ( 0 , 0 ) (0,0) ( 0 , 0 ) . The slow part of the trajectory is a cyclic tail. Its gradient directions run through p p p orthonormal vectors, so that gradients at the lags 1 , … , p − 1 1,\ldots,p-1 1 , … , p − 1 are orthogonal, and their lengths decay by the factor r r r per step. With the right step length τ \tau τ and values f k f_k f k all residuals are bounded below by a fixed multiple of r 2 min ( i , j ) r^{2\min(i,j)} r 2 m i n ( i , j ) , a margin that later absorbs perturbations.
Both tails below are orbits of an orthogonal map Q Q Q with a factor r ∈ ( 0 , 1 ) r \in(0,1) r ∈ ( 0 , 1 ) ,
( x n , g n , f n ) = ( r n Q n X , r n Q n G , c 2 r 2 n ) ( n ≥ 0 ) , ⟨ G , X ⟩ = c . (6) (x_n,g_n,f_n) = (r^nQ^nX,r^nQ^nG,\frac{c}{2}r^{2n}) \quad(n \ge0), \qquad\langle G,X\rangle= c. \tag*{(6)} ( x n , g n , f n ) = ( r n Q n X , r n Q n G , 2 c r 2 n ) ( n ≥ 0 ) , ⟨ G , X ⟩ = c . (6)
A common shift of indices by i i i multiplies all their residuals, including those with the minimizer, by r 2 i r^{2i} r 2 i , and ⟨ G , X ⟩ = c \langle G,X\rangle= c ⟨ G , X ⟩ = c makes both residuals of ( X , G , c 2 ) (X,G,\frac{c}{2}) ( X , G , 2 c ) with the minimizer equal to one number C ⋆ C_\star C ⋆ . For ℓ > 0 \ell> 0 ℓ > 0 the residuals between ( X , G , c 2 ) (X,G,\frac{c}{2}) ( X , G , 2 c ) and ( t Q ℓ X , t Q ℓ G , c 2 t 2 ) (tQ^\ell X,tQ^\ell G,\frac{c}{2}t^2) ( t Q ℓ X , t Q ℓ G , 2 c t 2 ) are quadratic in t t t , equal C ⋆ C_\star C ⋆ at t = 0 t=0 t = 0 and, again by ⟨ G , X ⟩ = c \langle G,X\rangle= c ⟨ G , X ⟩ = c , have the coefficient C ⋆ C_\star C ⋆ at t 2 t^2 t 2 . With t = r ℓ t = r^\ell t = r ℓ this gives
C 0 ℓ = C ⋆ ( 1 − t ) 2 + t K ℓ + , C ℓ 0 = C ⋆ ( 1 − t ) 2 + t K ℓ − , (7) C_{0\ell}=C_\star(1-t)^2+tK_\ell^+,\qquad C_{\ell0}=C_\star(1-t)^2+tK_\ell^-, \tag*{(7)} C 0 ℓ = C ⋆ ( 1 − t ) 2 + t K ℓ + , C ℓ 0 = C ⋆ ( 1 − t ) 2 + t K ℓ − , (7)
where K ℓ + K_\ell^+ K ℓ + and K ℓ − K_\ell^- K ℓ − are the residuals between ( X , G , c 2 ) (X,G,\frac{c}{2}) ( X , G , 2 c ) and ( Q ℓ X , Q ℓ G , c 2 ) (Q^\ell X,Q^\ell G,\frac{c}{2}) ( Q ℓ X , Q ℓ G , 2 c ) in the same order.
Lemma 4.5 (cyclic tails) Let either (i) p = 3 p=3 p = 3 , 0 < μ ≤ 1 12 0<\mu\le\frac{1}{12} 0 < μ ≤ 12 1 , r = 1 − 3 μ r=1-3\mu r = 1 − 3 μ and c = [ 3 μ ( 1 − 2 μ ) ] − 1 c=[3\mu(1-2\mu)]^{-1} c = [ 3 μ ( 1 − 2 μ ) ] − 1 , or (ii) p ≥ 8 p\ge8 p ≥ 8 , κ ≥ 100 p 2 \kappa\ge100p^2 κ ≥ 100 p 2 , μ = 1 / κ \mu=1/\kappa μ = 1/ κ , r = 1 − p / κ r=1-p/\kappa r = 1 − p / κ and c = κ / p + 3 p / 4 c=\kappa/p+3p/4 c = κ / p + 3 p /4 . Put τ = c ( 1 − r p ) \tau=c(1-r^p) τ = c ( 1 − r p ) , ζ n = e 1 + ( n m o d p ) ∈ R p \zeta_n=e_{1+(n\bmod p)}\in\mathbb{R}^p ζ n = e 1 + ( n mod p ) ∈ R p and
g n = r n ζ n , x n = τ ∑ j = n ∞ g j , f n = c 2 r 2 n ( n ≥ 0 ) . g_n=r^n\zeta_n,\qquad x_n=\tau\sum_{j=n}^{\infty}g_j,\qquad f_n=\frac{c}{2}r^{2n}\quad(n\ge0). g n = r n ζ n , x n = τ j = n ∑ ∞ g j , f n = 2 c r 2 n ( n ≥ 0 ) . Then x n + 1 − x n = − τ g n x_{n+1}-x_n=-\tau g_n x n + 1 − x n = − τ g n , C i j ≥ η r 2 min ( i , j ) C_{ij}\ge\eta r^{2\min(i,j)} C ij ≥ η r 2 m i n ( i , j ) for i ≠ j i\ne j i = j and C i ⋆ , C ⋆ i ≥ η r 2 i C_{i\star},C_{\star i}\ge\eta r^{2i} C i ⋆ , C ⋆ i ≥ η r 2 i for some η > 0 \eta>0 η > 0 .
Proof. All residuals reduce to the scalars A \mathcal{A} A and E q E_q E q defined below, and the lemma follows once they are positive.
Step 1: reduction. Let R R R be the cyclic shift of R p \mathbb{R}^p R p . Grouping the series by residues gives x n = r n R n x 0 x_n=r^nR^nx_0 x n = r n R n x 0 with x 0 = c ( 1 , r , … , r p − 1 ) ⊤ x_0=c(1,r,\ldots,r^{p-1})^\top x 0 = c ( 1 , r , … , r p − 1 ) ⊤ , and the first coordinate of r R x 0 − x 0 rRx_0-x_0 r R x 0 − x 0 is c ( r p − 1 ) c(r^p-1) c ( r p − 1 ) while the others vanish, which gives the step identity. Put
S = ∑ i = 0 p − 1 r 2 i , T d = ∑ i = 0 p − 1 r i r ( i − d ) m o d p , A = ( 1 + μ ) c − 1 − μ c 2 S S=\sum_{i=0}^{p-1}r^{2i},\qquad T_d=\sum_{i=0}^{p-1}r^ir^{(i-d)\bmod p},\qquad\mathcal{A}=(1+\mu)c-1-\mu c^2S S = i = 0 ∑ p − 1 r 2 i , T d = i = 0 ∑ p − 1 r i r ( i − d ) mod p , A = ( 1 + μ ) c − 1 − μ c 2 S
and, for 1 ≤ q ≤ ⌊ p / 2 ⌋ 1\le q\le\lfloor p/2\rfloor 1 ≤ q ≤ ⌊ p /2 ⌋ ,
E q = c [ ( 1 − r q ) + μ ( 1 − r p − q ) ] − 1 − μ c 2 ( S − T q ) . E_q=c\left[(1-r^q)+\mu(1-r^{p-q})\right]-1-\mu c^2(S-T_q). E q = c [ ( 1 − r q ) + μ ( 1 − r p − q ) ] − 1 − μ c 2 ( S − T q ) .
The data form an orbit (6) with Q = R Q=R Q = R , X = x 0 X=x_0 X = x 0 and G = e 1 G=e_1 G = e 1 , and C ⋆ = A / ( 2 μ ˉ ) C_\star=\mathcal{A}/(2\bar{\mu}) C ⋆ = A / ( 2 μ ˉ ) . Fix a lag j > 0 j>0 j > 0 , t = r j t=r^j t = r j and d = j m o d p d=j\bmod p d = j mod p . If d = 0 d=0 d = 0 , then K j + = 0 K_j^+=0 K j + = 0 . If d ≠ 0 d\ne0 d = 0 and q = min ( d , p − d ) q=\min(d,p-d) q = min ( d , p − d ) , the identities ∥ e 1 − R d e 1 ∥ 2 = 2 \lVert e_1-R^de_1\rVert^2=2 ∥ e 1 − R d e 1 ∥ 2 = 2 , ∥ x 0 − R d x 0 ∥ 2 = 2 c 2 ( S − T d ) \lVert x_0-R^dx_0\rVert^2=2c^2(S-T_d) ∥ x 0 − R d x 0 ∥ 2 = 2 c 2 ( S − T d ) and ⟨ e 1 − R d e 1 , x 0 − R d x 0 ⟩ = c ( 2 − r d − r p − d ) \langle e_1-R^de_1,x_0-R^dx_0\rangle=c(2-r^d-r^{p-d}) ⟨ e 1 − R d e 1 , x 0 − R d x 0 ⟩ = c ( 2 − r d − r p − d ) give min K j ± = E q / μ ˉ \min K_j^\pm=E_q/\bar{\mu} min K j ± = E q / μ ˉ . Since t ≤ r p t\le r^p t ≤ r p for d = 0 d=0 d = 0 and A ( 1 − t ) 2 + 2 E q t ≥ min { A , E q } ( 1 + t 2 ) \mathcal{A}(1-t)^2+2E_qt\ge\min\{\mathcal{A},E_q\}(1+t^2) A ( 1 − t ) 2 + 2 E q t ≥ min { A , E q } ( 1 + t 2 ) , (7) shows that η = min { A ( 1 − r p ) 2 , A , min q E q } / ( 2 μ ˉ ) \eta=\min\{\mathcal{A}(1-r^p)^2,\mathcal{A},\min_qE_q\}/(2\bar{\mu}) η = min { A ( 1 − r p ) 2 , A , min q E q } / ( 2 μ ˉ ) works once A \mathcal{A} A and all E q E_q E q are positive.
Step 2: case (i). Inserting r r r and c c c gives
A = ( 1 − 3 μ ) ( 9 μ 2 − 5 μ + 2 ) 3 ( 1 − 2 μ ) 2 > 0 , E 1 = μ ( 1 + μ ) ( 1 − 3 μ ) ( 1 − 2 μ ) 2 > 0. \mathcal{A}=\frac{(1-3\mu)(9\mu^2-5\mu+2)}{3(1-2\mu)^2}>0,\qquad E_1=\frac{\mu(1+\mu)(1-3\mu)}{(1-2\mu)^2}>0. A = 3 ( 1 − 2 μ ) 2 ( 1 − 3 μ ) ( 9 μ 2 − 5 μ + 2 ) > 0 , E 1 = ( 1 − 2 μ ) 2 μ ( 1 + μ ) ( 1 − 3 μ ) > 0.
Step 3: positivity of A \mathcal{A} A in case (ii). Here 1 − r = p μ 1-r=p\mu 1 − r = p μ and c = 1 / ( p μ ) + b c=1/(p\mu)+b c = 1/ ( p μ ) + b with b = 3 p / 4 b=3p/4 b = 3 p /4 , and κ ≥ 100 p 2 \kappa\ge100p^2 κ ≥ 100 p 2 gives p 2 μ ≤ 1 100 p^2\mu\le\frac{1}{100} p 2 μ ≤ 100 1 . From r n ≤ 1 − n p μ + n ( n − 1 ) 2 p 2 μ 2 r^n\le1-np\mu+\frac{n(n-1)}{2}p^2\mu^2 r n ≤ 1 − n p μ + 2 n ( n − 1 ) p 2 μ 2 we get S ≤ p − p 2 ( p − 1 ) μ + 2 3 p 5 μ 2 S\le p-p^2(p-1)\mu+\frac{2}{3}p^5\mu^2 S ≤ p − p 2 ( p − 1 ) μ + 3 2 p 5 μ 2 and S ≤ p S \le p S ≤ p . Writing μ c 2 S = S p 2 μ + 2 b S p + b 2 μ S \mu c^{2}S = \frac{S}{p^{2}\mu} + \frac{2bS}{p} + b^{2}\mu S μ c 2 S = p 2 μ S + p 2 b S + b 2 μ S and using the first bound only in the first term gives
A ≥ p 4 − 2 + 1 p − 59 48 p 3 μ ≥ 19 p 80 − 2 + 1 p ≥ 1 40 . A \ge\frac{p}{4} - 2 + \frac{1}{p} - \frac{59}{48}p^{3}\mu\ge\frac{19p}{80} - 2 + \frac{1}{p} \ge\frac{1}{40}. A ≥ 4 p − 2 + p 1 − 48 59 p 3 μ ≥ 80 19 p − 2 + p 1 ≥ 40 1 .
Step 4: positivity of E q E_q E q in case (ii). For q = 1 q = 1 q = 1 the geometric sums give S − T 1 = ( 1 − r p − 1 ) ( 1 − r p ) / ( 1 + r ) S - T_1 = (1 - r^{p-1})(1 - r^p)/(1 + r) S − T 1 = ( 1 − r p − 1 ) ( 1 − r p ) / ( 1 + r ) . With V = ( 1 − r p − 1 ) / ( p μ ) ≤ p − 1 V = (1 - r^{p-1})/(p\mu) \le p - 1 V = ( 1 − r p − 1 ) / ( p μ ) ≤ p − 1 and W = ( 1 − r p ) / ( p μ ) ≤ p W = (1 - r^p)/(p\mu) \le p W = ( 1 − r p ) / ( p μ ) ≤ p ,
E 1 = p μ [ b + ( 1 + b p μ ) V p − ( 1 + b p μ ) 2 V W p ( 1 + r ) ] ≥ p μ ( 3 p 2 − 3 ( p − 1 ) 5 ) > 0 , \mathcal{E}_1 = p\mu\left[b + \frac{(1 + bp\mu)V}{p} - \frac{(1 + bp\mu)^2VW}{p(1 + r)}\right] \ge p\mu\left(\frac{3p}{2} - \frac{3(p - 1)}{5}\right) > 0, E 1 = p μ [ b + p ( 1 + b p μ ) V − p ( 1 + r ) ( 1 + b p μ ) 2 V W ] ≥ p μ ( 2 3 p − 5 3 ( p − 1 ) ) > 0 ,
because ( 1 + b p μ ) 2 / ( 1 + r ) ≤ 3 5 (1 + bp\mu)^2/(1 + r) \le\frac{3}{5} ( 1 + b p μ ) 2 / ( 1 + r ) ≤ 5 3 . For q ≥ 2 q \ge2 q ≥ 2 , the identity S − T q = 1 2 ∑ i = 0 p − 1 ( r i − r ( i − q ) m o d p ) 2 S - T_q = \frac{1}{2}\sum_{i=0}^{p-1}(r^i - r^{(i-q)\bmod p})^2 S − T q = 2 1 ∑ i = 0 p − 1 ( r i − r ( i − q ) mod p ) 2 and ∣ r i − r j ∣ ≤ p μ ∣ i − j ∣ |r^i - r^j| \le p\mu|i-j| ∣ r i − r j ∣ ≤ p μ ∣ i − j ∣ give S − T q ≤ 1 2 p 3 μ 2 q ( p − q ) S - T_q \le\frac{1}{2}p^3\mu^2q(p-q) S − T q ≤ 2 1 p 3 μ 2 q ( p − q ) . With u = p 2 μ ≤ 1 100 u = p^2\mu\le\frac{1}{100} u = p 2 μ ≤ 100 1 one obtains c ( 1 − r q ) ≥ q − q u / 4 c(1 - r^q) \ge q - qu/4 c ( 1 − r q ) ≥ q − q u /4 and μ c 2 ( S − T q ) ≤ u 2 ( 1 + 3 u 4 ) q 2 \mu c^2(S - T_q) \le\frac{u}{2}(1 + \frac{3u}{4})q^2 μ c 2 ( S − T q ) ≤ 2 u ( 1 + 4 3 u ) q 2 , hence E q ≥ 124 125 q − 1 > 0 E_q \ge\frac{124}{125}q - 1 > 0 E q ≥ 125 124 q − 1 > 0 .
The constant 100 in κ ≥ 100 p 2 \kappa\ge100p^2 κ ≥ 100 p 2 is not optimized. It only keeps the expansions in case (ii) short, and any bound that makes p 2 μ p^2\mu p 2 μ small enough would do.
Steps are multiples of gradients, so the window at iteration k k k is spanned by g k − m , … , g k g_{k-m},\ldots,g_k g k − m , … , g k , whose lags to k + 1 k+1 k + 1 lie between 1 and m + 1 ≤ p − 1 m+1 \le p-1 m + 1 ≤ p − 1 . The next tail gradient is therefore orthogonal to the window, and Lemmas 3.4 , 4.4 and 4.5 already settle pure windows. Sums are harder, since a gradient sum ∑ i ≤ k w ( i ) g i \sum_{i\le k}w(i)g_i ∑ i ≤ k w ( i ) g i is not orthogonal to the next tail gradient. If the weights grow more slowly than 1 / r 1/r 1/ r , its tail is geometrically small, and a small perturbation of the tail directions makes every tail gradient orthogonal to the sums of the trajectory if the sums start from suitable values. An initial segment of orthogonal gradient steps with a margin of its own provides them.
Lemma 4.6 (orthogonal initial segment) Let 0 < μ ≤ 1 / 12 0 < \mu\le1/12 0 < μ ≤ 1/12 and B ≥ 1 B \ge1 B ≥ 1 . Let G 0 , … , G B − 1 G_0,\ldots,G_{B-1} G 0 , … , G B − 1 be orthogonal with a i = ∥ G i ∥ 2 = 10 − i a_i = \lVert G_i\rVert^2 = 10^{-i} a i = ∥ G i ∥ 2 = 1 0 − i , and let t i ∈ [ 19 10 , 21 10 ] t_i \in[\frac{19}{10},\frac{21}{10}] t i ∈ [ 10 19 , 10 21 ] . Define X i = ∑ h = i B − 1 t h G h X_i = \sum_{h=i}^{B-1}t_hG_h X i = ∑ h = i B − 1 t h G h and F i = ∑ h = i B − 1 a h F_i = \sum_{h=i}^{B-1}a_h F i = ∑ h = i B − 1 a h for i < B i < B i < B , and ( X B , G B , F B ) = ( 0 , 0 , 0 ) (X_B,G_B,F_B) = (0,0,0) ( X B , G B , F B ) = ( 0 , 0 , 0 ) . Then C i j , C j i ≥ a i / 7 C_{ij},C_{ji} \ge a_i/7 C ij , C j i ≥ a i /7 for i < j ≤ B i < j \le B i < j ≤ B .
Proof. Put a B = 0 a_B = 0 a B = 0 and U i j = ∑ h = i j − 1 a h U_{ij} = \sum_{h=i}^{j-1}a_h U ij = ∑ h = i j − 1 a h . Orthogonality gives
∥ G i − G j ∥ 2 = a i + a j , ∥ X i − X j ∥ 2 = ∑ h = i j − 1 t h 2 a h , ⟨ G i − G j , X i − X j ⟩ = t i a i , \lVert G_i-G_j\rVert^2=a_i+a_j,\qquad\lVert X_i-X_j\rVert^2=\sum_{h=i}^{j-1}t_h^2a_h,\qquad\langle G_i-G_j,X_i-X_j\rangle=t_ia_i, ∥ G i − G j ∥ 2 = a i + a j , ∥ X i − X j ∥ 2 = h = i ∑ j − 1 t h 2 a h , ⟨ G i − G j , X i − X j ⟩ = t i a i ,
and G j ⊥ X i − X j G_j \perp X_i-X_j G j ⊥ X i − X j . Inserting this into (4) with L = 1 L=1 L = 1 gives C i j = U i j − Q i j ( t ) C_{ij}=U_{ij}-Q_{ij}(t) C ij = U ij − Q ij ( t ) and C j i = t i a i − U i j − Q i j ( t ) C_{ji}=t_ia_i-U_{ij}-Q_{ij}(t) C j i = t i a i − U ij − Q ij ( t ) , where
2 ( 1 − μ ) Q i j ( t ) = a i + a j + μ ∑ h = i j − 1 t h 2 a h − 2 μ t i a i . 2(1-\mu)Q_{ij}(t)=a_i+a_j+\mu\sum_{h=i}^{j-1}t_h^2a_h-2\mu t_ia_i. 2 ( 1 − μ ) Q ij ( t ) = a i + a j + μ h = i ∑ j − 1 t h 2 a h − 2 μ t i a i .
As μ ≤ 1 12 \mu\le\frac{1}{12} μ ≤ 12 1 , t i ( t i − 2 ) ≤ 1 4 t_i(t_i-2)\le\frac{1}{4} t i ( t i − 2 ) ≤ 4 1 and ∑ h = i + 1 j − 1 t h 2 a h ≤ a i \sum_{h=i+1}^{j-1}t_h^2a_h \le a_i ∑ h = i + 1 j − 1 t h 2 a h ≤ a i , the μ \mu μ -terms are at most a i / 14 a_i/14 a i /14 . With a j ≤ a i / 10 a_j \le a_i/10 a j ≤ a i /10 and 2 ( 1 − μ ) ≥ 11 6 2(1-\mu)\ge\frac{11}{6} 2 ( 1 − μ ) ≥ 6 11 this gives Q i j ( t ) ≤ 41 35 ⋅ 6 11 a i < 2 3 a i Q_{ij}(t)\le\frac{41}{35}\cdot\frac{6}{11}a_i < \frac{2}{3}a_i Q ij ( t ) ≤ 35 41 ⋅ 11 6 a i < 3 2 a i . Using a i ≤ U i j ≤ 10 9 a i a_i\le U_{ij}\le\frac{10}{9}a_i a i ≤ U ij ≤ 9 10 a i and t i ≥ 19 10 t_i\ge\frac{19}{10} t i ≥ 10 19 we get
C i j ≥ U i j − 2 3 a i ≥ 1 3 a i , C j i ≥ ( 19 10 − 10 9 − 41 35 ⋅ 6 11 ) a i ≥ a i 7 . C_{ij}\ge U_{ij}-\frac{2}{3}a_i\ge\frac{1}{3}a_i,\qquad C_{ji}\ge\left(\frac{19}{10}-\frac{10}{9}-\frac{41}{35}\cdot\frac{6}{11}\right)a_i\ge\frac{a_i}{7}. C ij ≥ U ij − 3 2 a i ≥ 3 1 a i , C j i ≥ ( 10 19 − 9 10 − 35 41 ⋅ 11 6 ) a i ≥ 7 a i .
The next lemma joins such a segment to a perturbed tail, and its proof is the technical core of this section. The unknowns are a small perturbation h n h_n h n of every tail direction and finitely many coordinates of the initial gradients, and every tail gradient has to be orthogonal to the anchor and to each aggregate. These are infinitely many nonlinear conditions, but at the unperturbed point they decouple. The lag conditions form a cyclic linear system, which can be inverted explicitly, and the conditions for the anchor and the aggregates fix the components of the perturbation along the initial segment through an invertible matrix A A A . The implicit function theorem then applies in a space of geometrically weighted sequences, and this is where the growth of the weights enters. It needs a small parameter ϑ \vartheta ϑ , and since the initial segment has size of order ϑ − 1 \vartheta^{-1} ϑ − 1 , the slow tail starts only after the gap has decreased by a large factor, which we did not try to control.
Lemma 4.7 (cyclic bridge). Let the data of Lemma 4.5 be given and 1 ≤ m ≤ p − 2 1 \le m \le p-2 1 ≤ m ≤ p − 2 . Let B > 1 B > 1 B > 1 , t ∈ ( 19 10 , 21 10 ) B t \in\left(\frac{19}{10},\frac{21}{10}\right)B t ∈ ( 10 19 , 10 21 ) B , and let w 1 , … , w B − 1 w_1,\ldots,w_{B-1} w 1 , … , w B − 1 be real sequences with ω = max a lim sup i → ∞ ∣ w a ( i ) ∣ 1 / i < 1 / r \omega=\max_{a}\limsup_{i\to\infty}|w_a(i)|^{1/i}<1/r ω = max a lim sup i → ∞ ∣ w a ( i ) ∣ 1/ i < 1/ r such that the B × B B \times B B × B matrix A A A with the rows t , w 1 ∣ [ 0 , B ) , … , w B − 1 ∣ [ 0 , B ) t,w_1|_{[0,B)},\ldots,w_{B-1}|_{[0,B)} t , w 1 ∣ [ 0 , B ) , … , w B − 1 ∣ [ 0 , B ) is invertible. Then there are F ∈ F μ , 1 ( R p + B ) F \in\mathcal{F}_{\mu,1}(\mathbb{R}^{p+B}) F ∈ F μ , 1 ( R p + B ) and a trajectory of the exact method with the spaces span { g i , s i : k − m < i ≤ k } + span { x k − x 0 , σ 1 k , … , σ B − 1 k } \operatorname{span}\{g_i,s_i:k-m<i\le k\}+\operatorname{span}\{x_k-x_0,\sigma_1^k,\ldots,\sigma_{B-1}^k\} span { g i , s i : k − m < i ≤ k } + span { x k − x 0 , σ 1 k , … , σ B − 1 k } , σ a k = ∑ i = 0 k w a ( i ) g i \sigma_a^k=\sum_{i=0}^{k}w_a(i)g_i σ a k = ∑ i = 0 k w a ( i ) g i , whose steps are − t k g k -t_kg_k − t k g k for k < B k<B k < B and − τ g k -\tau g_k − τ g k for k ≥ B k\ge B k ≥ B , and F ( x k ) − F ∗ = 1 2 r 2 k F(x_k)-F^*=\frac{1}{2}r^{2k} F ( x k ) − F ∗ = 2 1 r 2 k for k ≥ B k\ge B k ≥ B .
Proof. Joint system. Choose γ \gamma γ with max { 1 , ω } < γ < 1 / r \max\{1,\omega\}<\gamma<1/r max { 1 , ω } < γ < 1/ r and β ∈ ( r γ , 1 ) \beta\in(r\gamma,1) β ∈ ( r γ , 1 ) , so that ∣ w a ( j ) ∣ ≤ C a γ j |w_a(j)|\le C_a\gamma^j ∣ w a ( j ) ∣ ≤ C a γ j for all a a a after enlarging the constants. The tail sums ∑ j = n ∞ w a ( j ) g j \sum_{j=n}^{\infty}w_a(j)g_j ∑ j = n ∞ w a ( j ) g j are then O ( ( r γ ) n ) = O ( β n ) O((r\gamma)^n)=O(\beta^n) O (( r γ ) n ) = O ( β n ) . In the space E = R B E=\mathbb{R}^B E = R B orthogonal to R p \mathbb{R}^p R p take the vectors G i G_i G i of Lemma 4.6 . For triangular parameters λ i j \lambda_{ij} λ ij , j < i < B j<i<B j < i < B , put
Z i ( λ ) = G i + ∑ j = 0 i − 1 λ i j G j , U ( λ ) = ∑ i = 0 B − 1 t i Z i ( λ ) , V a ( λ ) = ∑ i = 0 B − 1 w a ( i ) Z i ( λ ) ( 1 ≤ a < B ) . Z_i(\lambda)=G_i+\sum_{j=0}^{i-1}\lambda_{ij}G_j,\qquad U(\lambda)=\sum_{i=0}^{B-1}t_iZ_i(\lambda),\qquad V_a(\lambda)=\sum_{i=0}^{B-1}w_a(i)Z_i(\lambda)\quad(1\le a<B). Z i ( λ ) = G i + j = 0 ∑ i − 1 λ ij G j , U ( λ ) = i = 0 ∑ B − 1 t i Z i ( λ ) , V a ( λ ) = i = 0 ∑ B − 1 w a ( i ) Z i ( λ ) ( 1 ≤ a < B ) .
For n ≥ B n\ge B n ≥ B seek v n = ζ n + h n v_n=\zeta_n+h_n v n = ζ n + h n and set g n = r n v n g_n=r^nv_n g n = r n v n and x n = τ ∑ j = n ∞ g j x_n=\tau\sum_{j=n}^{\infty}g_j x n = τ ∑ j = n ∞ g j . For every n ≥ B n\ge B n ≥ B we impose the p + B p+B p + B equations
∥ v n ∥ = 1 , ⟨ v n , v n + t ⟩ = 0 ( 1 ≤ t ≤ p − 1 ) (norm and lag conditions), \|v_n\|=1,\qquad\langle v_n,v_{n+t}\rangle=0\quad(1\le t\le p-1)\qquad\text{(norm and lag conditions),} ∥ v n ∥ = 1 , ⟨ v n , v n + t ⟩ = 0 ( 1 ≤ t ≤ p − 1 ) (norm and lag conditions),
⟨ v n , U ( λ ) ⟩ = ϑ ⟨ v n , x n ⟩ (anchor condition), \langle v_n,U(\lambda)\rangle=\vartheta\langle v_n,x_n\rangle\qquad\text{(anchor condition),} ⟨ v n , U ( λ )⟩ = ϑ ⟨ v n , x n ⟩ (anchor condition),
⟨ v n , V a ( λ ) ⟩ = ϑ ⟨ v n , ∑ j ≥ n w a ( j ) g j ⟩ ( 1 ≤ a < B ) (aggregate conditions). \langle v_n,V_a(\lambda)\rangle=\vartheta\left\langle v_n,\sum_{j\ge n}w_a(j)g_j\right\rangle\quad(1\le a<B)\qquad\text{(aggregate conditions).} ⟨ v n , V a ( λ )⟩ = ϑ ⟨ v n , j ≥ n ∑ w a ( j ) g j ⟩ ( 1 ≤ a < B ) (aggregate conditions).
Define correction vectors by ∑ i A ℓ i T i = c ℓ \sum_i A_{\ell i}T_i=c_\ell ∑ i A ℓ i T i = c ℓ for 0 ≤ ℓ < B 0\le\ell<B 0 ≤ ℓ < B , where c 0 = x B c_0=x_B c 0 = x B and c a = ∑ n ≥ B w a ( n ) g n c_a=\sum_{n\ge B}w_a(n)g_n c a = ∑ n ≥ B w a ( n ) g n for a ≥ 1 a\ge1 a ≥ 1 , and put P i = Z i ( λ ) − ϑ T i P_i=Z_i(\lambda)-\vartheta T_i P i = Z i ( λ ) − ϑ T i . Finally impose the finite Gram equations ⟨ P i , P j ⟩ = 0 \langle P_i,P_j\rangle=0 ⟨ P i , P j ⟩ = 0 for 0 ≤ j < i < B 0\le j<i<B 0 ≤ j < i < B .
Function spaces. The unknowns are λ ∈ R B ( B − 1 ) / 2 \lambda\in\mathbb{R}^{B(B-1)/2} λ ∈ R B ( B − 1 ) /2 and h = ( h n ) n ≥ B h=(h_n)_{n\ge B} h = ( h n ) n ≥ B in the space ℓ β ∞ \ell^\infty_\beta ℓ β ∞ of sequences in R p + B \mathbb{R}^{p+B} R p + B with the norm ∥ h ∥ β = sup n β − n ∥ h n ∥ \|h\|_\beta=\sup_n\beta^{-n}\|h_n\| ∥ h ∥ β = sup n β − n ∥ h n ∥ . The residual of the tail equations with index n n n is O ( β n ) O(\beta^n) O ( β n ) , so they define a map into ℓ β ∞ \ell^\infty_\beta ℓ β ∞ , and the Gram equations a map into R B ( B − 1 ) / 2 \mathbb{R}^{B(B-1)/2} R B ( B − 1 ) /2 . Since ∣ w a ( n ) ∣ r n ≤ C ( γ r ) n |w_a(n)|r^n\le C(\gamma r)^n ∣ w a ( n ) ∣ r n ≤ C ( γ r ) n and γ r < β \gamma r<\beta γ r < β , the vector x B x_B x B and the sums ∑ n ≥ B w a ( n ) g n \sum_{n\ge B}w_a(n)g_n ∑ n ≥ B w a ( n ) g n depend affinely and boundedly on h h h , and so do the corrections T i T_i T i . The whole map is therefore continuously differentiable near ( 0 , 0 , 0 ) (0,0,0) ( 0 , 0 , 0 ) .
Invertibility. The system holds at ( h , λ , ϑ ) = ( 0 , 0 , 0 ) (h,\lambda,\vartheta)=(0,0,0) ( h , λ , ϑ ) = ( 0 , 0 , 0 ) . At this point the derivative with respect to ( h , λ ) (h,\lambda) ( h , λ ) is block diagonal. The Gram equations do not depend on h h h at ϑ = 0 \vartheta=0 ϑ = 0 , because then P i = Z i ( λ ) P_i=Z_i(\lambda) P i = Z i ( λ ) . The tail equations have zero derivative with respect to λ \lambda λ at h = 0 h=0 h = 0 , because U ( λ ) U(\lambda) U ( λ ) and V a ( λ ) V_a(\lambda) V a ( λ ) lie in E E E , which is orthogonal to every ζ n \zeta_n ζ n . In the cyclic block the linearized lag equations read a n t + a n + t t = q n t a_n^t+a_{n+t}^t=q_n^t a n t + a n + t t = q n t with a n t = ⟨ h n , ζ n + t ⟩ a_n^t=\langle h_n,\zeta_{n+t}\rangle a n t = ⟨ h n , ζ n + t ⟩ and right-hand sides q n t q_n^t q n t , and their inverse is
a n t = ∑ ℓ = 0 ∞ ( q n + ℓ p t − q n + t + ℓ p p − t ) . (8) a_n^t=\sum_{\ell=0}^{\infty}\left(q_{n+\ell p}^t-q_{n+t+\ell p}^{p-t}\right). \tag*{(8)} a n t = ℓ = 0 ∑ ∞ ( q n + ℓ p t − q n + t + ℓ p p − t ) . (8)
which telescopes, converges in ℓ β ∞ \ell^\infty_\beta ℓ β ∞ because β p < 1 \beta^p < 1 β p < 1 , and is unique among decaying sequences. The block of the extra coordinates consists of the B B B scalar products with U ( 0 ) , V 1 ( 0 ) , … , V B − 1 ( 0 ) U(0), V_1(0), \ldots, V_{B-1}(0) U ( 0 ) , V 1 ( 0 ) , … , V B − 1 ( 0 ) . In the orthonormal basis G i / a i G_i/\sqrt{a_i} G i / a i of E E E its matrix is A diag ( a i ) A\operatorname{diag}(\sqrt{a_i}) A diag ( a i ) , which is invertible. The finite block is diagonal with entries a j > 0 a_j > 0 a j > 0 . The implicit function theorem gives a solution with ∥ h ∥ β + ∥ λ ∥ = O ( ϑ ) \lVert h\rVert_\beta+ \lVert\lambda\rVert= O(\vartheta) ∥ h ∥ β + ∥ λ ∥ = O ( ϑ ) .
Actual initial segment. For i < B i < B i < B put g i = ϑ − 1 P i g_i = \vartheta^{-1}P_i g i = ϑ − 1 P i , x i = x B + ϑ − 1 ∑ h = i B − 1 t h P h x_i = x_B + \vartheta^{-1}\sum_{h=i}^{B-1}t_hP_h x i = x B + ϑ − 1 ∑ h = i B − 1 t h P h and f i = ϑ − 2 F i f_i = \vartheta^{-2}F_i f i = ϑ − 2 F i . The corrections give x 0 = ϑ − 1 U ( λ ) x_0 = \vartheta^{-1}U(\lambda) x 0 = ϑ − 1 U ( λ ) and ∑ i = 0 ∞ w a ( i ) g i = ϑ − 1 V a ( λ ) \sum_{i=0}^{\infty}w_a(i)g_i = \vartheta^{-1}V_a(\lambda) ∑ i = 0 ∞ w a ( i ) g i = ϑ − 1 V a ( λ ) for every row 1 ≤ a < B 1 \le a < B 1 ≤ a < B . The initial gradients are mutually orthogonal and each initial step is − t i g i -t_ig_i − t i g i . The tail equations then give ⟨ g n , x n − x 0 ⟩ = 0 \langle g_n,x_n-x_0\rangle= 0 ⟨ g n , x n − x 0 ⟩ = 0 and ⟨ g n , ∑ i = 0 n − 1 w a ( i ) g i ⟩ = 0 \langle g_n,\sum_{i=0}^{n-1}w_a(i)g_i\rangle= 0 ⟨ g n , ∑ i = 0 n − 1 w a ( i ) g i ⟩ = 0 for n ≥ B n \ge B n ≥ B and all 1 ≤ a < B 1 \le a < B 1 ≤ a < B . For B ≤ n ≤ B + p − 2 B \le n \le B+p-2 B ≤ n ≤ B + p − 2 the lag conditions make g n g_n g n orthogonal to the tail gradients with indices from B B B to n − 1 n-1 n − 1 , which leaves the B B B relations ⟨ g n , ∑ i = 0 B − 1 A ℓ i g i ⟩ = 0 \langle g_n,\sum_{i=0}^{B-1}A_{\ell i}g_i\rangle= 0 ⟨ g n , ∑ i = 0 B − 1 A ℓ i g i ⟩ = 0 , one for each row ℓ \ell ℓ of A A A . Since A A A is invertible, g n ⊥ g i g_n \perp g_i g n ⊥ g i for all i < B i < B i < B . This range suffices, because for m ≤ p − 2 m \le p-2 m ≤ p − 2 the window at iteration n − 1 n-1 n − 1 contains an initial gradient only if n ≤ B + m n \le B+m n ≤ B + m .
Optimality. For k + 1 < B k+1 < B k + 1 < B every generator of S k S_k S k is a combination of g 0 , … , g k g_0,\ldots,g_k g 0 , … , g k , and g k + 1 g_{k+1} g k + 1 is orthogonal to them. For k + 1 ≥ B k+1 \ge B k + 1 ≥ B the lag conditions and the relations above make g k + 1 g_{k+1} g k + 1 orthogonal to the window. The anchor relation gives ⟨ g k + 1 , x k − x 0 ⟩ = ⟨ g k + 1 , x k + 1 − x 0 ⟩ − ⟨ g k + 1 , s k + 1 ⟩ = 0 \langle g_{k+1},x_k-x_0\rangle= \langle g_{k+1},x_{k+1}-x_0\rangle-\langle g_{k+1},s_{k+1}\rangle= 0 ⟨ g k + 1 , x k − x 0 ⟩ = ⟨ g k + 1 , x k + 1 − x 0 ⟩ − ⟨ g k + 1 , s k + 1 ⟩ = 0 , and the aggregate relations give g k + 1 ⊥ σ a g_{k+1}\perp\sigma^a g k + 1 ⊥ σ a . Thus s k + 1 ∈ S k s_{k+1}\in S_k s k + 1 ∈ S k and g k + 1 ⊥ S k g_{k+1}\perp S_k g k + 1 ⊥ S k , and by strong convexity x k + 1 x_{k+1} x k + 1 is the unique minimizer over x k + S k x_k+S_k x k + S k .
Interpolation. The solution satisfies ∥ g n − g n 0 ∥ ≤ K ϑ ( r β ) n \lVert g_n-g_n^0\rVert\le K\vartheta(r\beta)^n ∥ g n − g n 0 ∥ ≤ K ϑ ( r β ) n and ∥ x n − x n 0 ∥ ≤ K ′ ϑ ( r β ) n \lVert x_n-x_n^0\rVert\le K'\vartheta(r\beta)^n ∥ x n − x n 0 ∥ ≤ K ′ ϑ ( r β ) n for n ≥ B n \ge B n ≥ B . A residual C i j C_{ij} C ij is the unperturbed difference f i − f j f_i-f_j f i − f j plus a quadratic form in the data at i i i and j j j , which for tail indices have size O ( r min ( i , j ) ) O(r^{\min(i,j)}) O ( r m i n ( i , j ) ) and perturbations of size O ( ϑ r min ( i , j ) ) O(\vartheta r^{\min(i,j)}) O ( ϑ r m i n ( i , j ) ) . Hence ∣ C i j − C i j 0 ∣ ≤ M ϑ r 2 min ( i , j ) |C_{ij}-C_{ij}^0| \le M\vartheta r^{2\min(i,j)} ∣ C ij − C ij 0 ∣ ≤ M ϑ r 2 m i n ( i , j ) for i , j ≥ B i,j \ge B i , j ≥ B and for pairs with the minimizer, with M M M independent of i , j i,j i , j and ϑ \vartheta ϑ , and for M ϑ < η / 2 M\vartheta< \eta/2 M ϑ < η /2 the margin η r 2 min ( i , j ) \eta r^{2\min(i,j)} η r 2 m i n ( i , j ) of Lemma 4.5 keeps these residuals positive. As the perturbation and the margin have the same order, this single comparison covers the infinitely many constraints. For pairs that involve the initial segment we scale the data by ( x , g , f ) ↦ ( ϑ x , ϑ g , ϑ 2 f ) (x,g,f)\mapsto(\vartheta x,\vartheta g,\vartheta^2f) ( x , g , f ) ↦ ( ϑ x , ϑ g , ϑ 2 f ) , which multiplies all residuals by ϑ 2 \vartheta^2 ϑ 2 . The scaled initial data converge to ( X i , G i , F i ) (X_i,G_i,F_i) ( X i , G i , F i ) of Lemma 4.6 and the scaled tail data uniformly to ( 0 , 0 , 0 ) = ( X B , G B , F B ) (0,0,0)=(X_B,G_B,F_B) ( 0 , 0 , 0 ) = ( X B , G B , F B ) , so the bound a i / 7 a_i/7 a i /7 of Lemma 4.6 and uniform continuity of the residual keep these residuals positive for small ϑ \vartheta ϑ . Lemma 3.4 completes the construction. □ \square □
Lemma 4.7 handles the anchor and gradient sums. Along a trajectory with steps s i + 1 = − t i g i s_{i+1}=-t_ig_i s i + 1 = − t i g i a step sum θ k b = ∑ i ≤ k v b ( i ) s i \theta_k^b=\sum_{i\le k}v_b(i)s_i θ k b = ∑ i ≤ k v b ( i ) s i equals − ∑ j = 0 k − 1 t j v ~ b ( j ) g j -\sum_{j=0}^{k-1}t_j\widetilde v_b(j)g_j − ∑ j = 0 k − 1 t j v b ( j ) g j with v ~ b = v b ( ⋅ + 1 ) \widetilde v_b=v_b(\cdot+1) v b = v b ( ⋅ + 1 ) , a gradient sum without its last term, and the anchor is the case v b = 1 v_b=1 v b = 1 . Its initial weights depend on the lengths t j t_j t j , chosen so that the initial segment fixes these sums.
Lemma 4.8 (generic step lengths) Let w 1 , … , w e , v ~ 0 ≡ 1 , v ~ 1 , … , v ~ f w_1,\ldots,w_e,\tilde v_0\equiv1,\tilde v_1,\ldots,\tilde v_f w 1 , … , w e , v ~ 0 ≡ 1 , v ~ 1 , … , v ~ f be sequences and τ > 0 \tau>0 τ > 0 . There are B ≥ 1 B\ge1 B ≥ 1 and t 0 , … , t B − 1 ∈ I = ( 19 10 , 21 10 ) t_0,\ldots,t_{B-1}\in I=\left(\frac{19}{10},\frac{21}{10}\right) t 0 , … , t B − 1 ∈ I = ( 10 19 , 10 21 ) such that, with t i = τ t_i=\tau t i = τ for i ≥ B i\ge B i ≥ B and ( t v ~ ) ( i ) = t i v ~ ( i ) (t\tilde v)(i)=t_i\tilde v(i) ( t v ~ ) ( i ) = t i v ~ ( i ) , the restriction to [ 0 , B ) [0,B) [ 0 , B ) is injective on U = span { w 1 , … , w e , t v ~ 0 , … , t v ~ f } U=\operatorname{span}\{w_1,\ldots,w_e,t\tilde v_0,\ldots,t\tilde v_f\} U = span { w 1 , … , w e , t v ~ 0 , … , t v ~ f } .
Proof For a positive sequence t t t the dimension of U U U is at most e + f + 1 e+f+1 e + f + 1 . Let t ∗ t^* t ∗ attain its maximum r r r , choose r r r linearly independent members of the family for t ∗ t^* t ∗ and indices i 1 , … , i r i_1,\ldots,i_r i 1 , … , i r at which their values form a nonsingular matrix. Its determinant is a polynomial in t i 1 , … , t i r t_{i_1},\ldots,t_{i_r} t i 1 , … , t i r that does not vanish at t ∗ t^* t ∗ , so it does not vanish at some point of I r I^r I r . Take B > max i j B>\max i_j B > max i j , these values at i 1 , … , i r i_1,\ldots,i_r i 1 , … , i r , arbitrary values in I I I at the other indices below B B B , and t i = τ t_i=\tau t i = τ for i ≥ B i\ge B i ≥ B . Then dim U ≤ r \dim U\le r dim U ≤ r by maximality, while the restrictions to [ 0 , B ) [0,B) [ 0 , B ) have rank at least r r r , so the restriction is injective. □ \square □
Proof of Theorem 4.2 . Without aggregates Lemma 3.4 applied to the data of Lemma 4.5 gives the instance, as explained after Lemma 4.5 . With aggregates we proceed in three steps.
The weights. Apply Lemma 4.8 to the weights w a w_a w a , to v ~ b \widetilde{v}_b v b and to v ~ 0 ≡ 1 \widetilde{v}_0 \equiv1 v 0 ≡ 1 , and choose a basis t , u 1 , … , u q t,u_1,\ldots,u_q t , u 1 , … , u q of U U U . Its restrictions to [ 0 , B ) [0,B) [ 0 , B ) are linearly independent, and completing them by rows supported on [ 0 , B ) [0,B) [ 0 , B ) gives an invertible matrix A A A as in Lemma 4.7 . For i ≥ B i \ge B i ≥ B every u j ( i ) u_j(i) u j ( i ) is a fixed combination of w a ( i ) w_a(i) w a ( i ) , τ v ~ b ( i ) \tau\widetilde{v}_b(i) τ v b ( i ) and τ \tau τ , so δ < p / κ \delta< p/\kappa δ < p / κ , that is Γ < 1 / ( 1 − p / κ ) \Gamma< 1/(1-p/\kappa) Γ < 1/ ( 1 − p / κ ) , gives the growth condition of Lemma 4.7 .
The trajectory. Take the data of Lemma 4.5 , case (i) for m = 1 m=1 m = 1 , p = 3 p=3 p = 3 and case (ii) otherwise, and apply Lemma 4.7 . On its trajectory g k + 1 g_{k+1} g k + 1 is orthogonal to the window, to the anchor and to the sums with the weights u j u_j u j . The weights w a w_a w a and τ v ~ b \tau\widetilde{v}_b τ v b are combinations of t , u 1 , … , u q t,u_1,\ldots,u_q t , u 1 , … , u q as sequences, so g k + 1 g_{k+1} g k + 1 is orthogonal to every gradient sum and, since g k + 1 ⊥ g k g_{k+1} \perp g_k g k + 1 ⊥ g k , to every step sum. Every step is a multiple of the current gradient, so by Lemma 4.4 every rule using the memory generates this trajectory, and F ( x k ) − F ⋆ = c 2 r 2 k F(x_k)-F^\star= \frac{c}{2}r^{2k} F ( x k ) − F ⋆ = 2 c r 2 k for k ≥ B k \ge B k ≥ B .
The rate. The gap contracts by r 2 r^2 r 2 per step, so − log ( 1 − u ) ≤ 2 u -\log(1-u) \le2u − log ( 1 − u ) ≤ 2 u for u ≤ 1 2 u \le\frac{1}{2} u ≤ 2 1 gives
lim ε ↓ 0 N ( ε ) log ( 1 / ε ) = 1 − 2 log r ≥ κ 4 p . \lim_{\varepsilon\downarrow0} \frac{N(\varepsilon)}{\log(1/\varepsilon)}=\frac{1}{-2\log r}\ge\frac{\kappa}{4p}. ε ↓ 0 lim log ( 1/ ε ) N ( ε ) = − 2 log r 1 ≥ 4 p κ .
Forgetting too fast
A pair of filters with a fixed factor ρ \rho ρ has the normalized weights ρ − i \rho^{-i} ρ − i , and the construction of § 4.2 covers it only if 1 − ρ < p / κ 1-\rho< p/\kappa 1 − ρ < p / κ . Forgetting too fast is just as bad, because such a pair is no better than a short window: with θ = 1 − ρ m + 2 \theta=1-\rho^{m+2} θ = 1 − ρ m + 2 , some function costs every method using the window and the pair about κ θ / m \kappa\theta/m κ θ / m iterations per unit of log ( 1 / ε ) \log(1/\varepsilon) log ( 1/ ε ) . The perturbation of § 4.2 fails here, because the normalized weights grow faster than the tail decays. We therefore choose the tail with exact orthogonality to the pair, and only (7) and Lemmas 3.4 and 4.4 carry over.
Theorem 4.9 (a fast pair). Let the memory consist of the window of length m ≥ 1 m \ge1 m ≥ 1 and one pair z k + 1 = ρ z k + ∇ f ( x k + 1 ) z_{k+1}=\rho z_k+\nabla f(x_{k+1}) z k + 1 = ρ z k + ∇ f ( x k + 1 ) , y k + 1 = ρ ( y k + s k + 1 ) y_{k+1}=\rho(y_k+s_{k+1}) y k + 1 = ρ ( y k + s k + 1 ) , z 0 = y 0 = 0 z_0=y_0=0 z 0 = y 0 = 0 , with 0 < ρ < 1 0<\rho<1 0 < ρ < 1 . Let d = m + 2 d=m+2 d = m + 2 and θ = 1 − ρ d \theta=1-\rho^d θ = 1 − ρ d , and assume κ θ 2 ≥ 1024 d 4 \kappa\theta^2\ge1024d^4 κ θ 2 ≥ 1024 d 4 . Then the conclusion of Theorem 4.2 holds with the factor ( 1 − 8 d / ( κ θ ) ) 2 (1-8d/(\kappa\theta))^2 ( 1 − 8 d / ( κ θ ) ) 2 , and lim inf ε ↓ 0 N ( ε ) / log ( 1 / ε ) ≥ κ θ / ( 32 d ) \liminf_{\varepsilon\downarrow0}N(\varepsilon)/\log(1/\varepsilon)\ge\kappa\theta/(32d) lim inf ε ↓ 0 N ( ε ) / log ( 1/ ε ) ≥ κ θ / ( 32 d ) .
Proof of Theorem 4.9 . The tail directions run through 2 d 2d 2 d orthonormal vectors, and one component of each direction changes its sign after d d d steps. In the product of the next gradient with the filter only the lags that are multiples of d d d survive, and they come with alternating signs. The weight α \alpha α of the two components is chosen so that the even and the odd multiples of d d d cancel exactly, see (11) below. The window sees only the lags 1 , … , d − 1 1,\ldots,d-1 1 , … , d − 1 , at which the tail directions are orthogonal anyway, so d = m + 2 d=m+2 d = m + 2 is the smallest admissible period.
Parameters. Put μ = 1 / κ \mu=1/\kappa μ = 1/ κ , η = 8 d / κ \eta=8d/\kappa η = 8 d / κ , r = 1 − η r=1-\eta r = 1 − η , ν = r d \nu=r^d ν = r d , ν = ( ρ / r ) d \nu=(\rho/r)^d ν = ( ρ / r ) d ,
α = 1 − ν 2 , τ = 4 d θ , ψ = θ 1 − ν 2 , c = τ ψ . \alpha=\frac{1-\nu}{2},\qquad\tau=\frac{4d}{\theta},\qquad\psi=\frac{\theta}{1-\nu^2},\qquad c=\tau\psi. α = 2 1 − ν , τ = θ 4 d , ψ = 1 − ν 2 θ , c = τ ψ .
Since θ ≤ d ( 1 − ρ ) \theta\le d(1-\rho) θ ≤ d ( 1 − ρ ) and κ θ 2 ≥ 1024 d 4 \kappa\theta^2\ge1024d^4 κ θ 2 ≥ 1024 d 4 , we have η ≤ θ / ( 128 d 3 ) ≤ ( 1 − ρ ) / ( 128 d 2 ) \eta\le\theta/(128d^3)\le(1-\rho)/(128d^2) η ≤ θ / ( 128 d 3 ) ≤ ( 1 − ρ ) / ( 128 d 2 ) , so ρ < r < 1 \rho<r<1 ρ < r < 1 , 0 < ν < 1 0<\nu<1 0 < ν < 1 and μ < 1 / 100 \mu<1/100 μ < 1/100 . From d η ≤ 1 − r 2 d ≤ 2 d η d\eta\le1-r^{2d}\le2d\eta d η ≤ 1 − r 2 d ≤ 2 d η we get
ψ ≥ κ θ 2 16 d 2 ≥ 64 d 2 , η ψ ≤ 1 d , τ ψ η ≥ 2 , μ τ = η 2 . (9) \psi\ge\frac{\kappa\theta^2}{16d^2}\ge64d^2,\qquad\eta\psi\le\frac{1}{d},\qquad\tau\psi\eta\ge2,\qquad\mu\tau=\frac{\eta}{2}. \tag*{(9)} ψ ≥ 16 d 2 κ θ 2 ≥ 64 d 2 , η ψ ≤ d 1 , τ ψ η ≥ 2 , μτ = 2 η . (9)
The cyclic tail. Let e 0 , … , e d − 1 , b 0 , … , b d − 1 e_0,\ldots,e_{d-1},b_0,\ldots,b_{d-1} e 0 , … , e d − 1 , b 0 , … , b d − 1 be an orthonormal basis of R 2 d \mathbb{R}^{2d} R 2 d , let Q Q Q be the orthogonal map with Q e j = e j + 1 Qe_j=e_{j+1} Q e j = e j + 1 , Q b j = b j + 1 Qb_j=b_{j+1} Q b j = b j + 1 for j < d − 1 j<d-1 j < d − 1 , Q e d − 1 = e 0 Qe_{d-1}=e_0 Q e d − 1 = e 0 and Q b d − 1 = − b 0 Qb_{d-1}=-b_0 Q b d − 1 = − b 0 , and let v = α e 0 + 1 − α b 0 v=\sqrt{\alpha}e_0+\sqrt{1-\alpha}b_0 v = α e 0 + 1 − α b 0 . Then
⟨ v , Q ℓ v ⟩ = 0 if d ∤ ℓ , ⟨ v , Q s d v ⟩ = α + ( 1 − α ) ( − 1 ) s . (10) \langle v,Q^\ell v\rangle=0\quad\text{if }d\nmid\ell,\qquad\langle v,Q^{sd}v\rangle=\alpha+(1-\alpha)(-1)^s. \tag*{(10)} ⟨ v , Q ℓ v ⟩ = 0 if d ∤ ℓ , ⟨ v , Q s d v ⟩ = α + ( 1 − α ) ( − 1 ) s . (10)
The tail gradients g ^ j = r j Q j v \hat{g}_j=r^jQ^jv g ^ j = r j Q j v , j ∈ Z j\in\mathbb{Z} j ∈ Z , are therefore orthogonal at the lags 1 , … , d − 1 1,\ldots,d-1 1 , … , d − 1 . The stationary filter Z k = ∑ i ≤ k g ^ i Z_k=\sum_{i\leq k}\hat{g}_i Z k = ∑ i ≤ k g ^ i converges because ρ < r \rho<r ρ < r , and in ⟨ g ^ k + 1 , Z k ⟩ \langle\hat{g}_{k+1},Z_k\rangle ⟨ g ^ k + 1 , Z k ⟩ only the indices i = k + 1 − s d i=k+1-sd i = k + 1 − s d , s ≥ 1 s\geq1 s ≥ 1 , contribute. By the choice of α \alpha α ,
⟨ g ^ k + 1 , Z k ⟩ = r 2 k + 2 ρ ∑ s ≥ 1 ν s ( α + ( 1 − α ) ( − 1 ) s ) = r 2 k + 2 ρ ( α ν 1 − ν − ( 1 − α ) ν 1 + ν ) = 0. (11) \langle\hat{g}_{k+1},Z_k\rangle=\frac{r^{2k+2}}{\rho}\sum_{s\geq1}\nu^s\left(\alpha+(1-\alpha)(-1)^s\right)=\frac{r^{2k+2}}{\rho}\left(\frac{\alpha\nu}{1-\nu}-\frac{(1-\alpha)\nu}{1+\nu}\right)=0. \tag*{(11)} ⟨ g ^ k + 1 , Z k ⟩ = ρ r 2 k + 2 s ≥ 1 ∑ ν s ( α + ( 1 − α ) ( − 1 ) s ) = ρ r 2 k + 2 ( 1 − ν α ν − 1 + ν ( 1 − α ) ν ) = 0. (11)
Let w = ( I − r Q ) − 1 v = ∑ j ≥ 0 r j Q j v w=(I-rQ)^{-1}v=\sum_{j\geq0}r^jQ^jv w = ( I − r Q ) − 1 v = ∑ j ≥ 0 r j Q j v , x ^ k = τ r k Q k w = τ ∑ j ≥ k g ^ j \hat{x}_k=\tau r^kQ^kw=\tau\sum_{j\geq k}\hat{g}_j x ^ k = τ r k Q k w = τ ∑ j ≥ k g ^ j and f ^ k = c 2 r 2 k \hat{f}_k=\frac{c}{2}r^{2k} f ^ k = 2 c r 2 k . Then x ^ k + 1 − x ^ k = − τ g ^ k \hat{x}_{k+1}-\hat{x}_k=-\tau\hat{g}_k x ^ k + 1 − x ^ k = − τ g ^ k . By (10) and ν u = ρ d = 1 − θ \nu u=\rho^d=1-\theta ν u = ρ d = 1 − θ ,
⟨ v , w ⟩ = α 1 − u + 1 − α 1 + u = 1 − ν u 1 − u 2 = ψ , ∥ w ∥ 2 = 2 ψ − 1 1 − r 2 , \langle v,w\rangle=\frac{\alpha}{1-u}+\frac{1-\alpha}{1+u}=\frac{1-\nu u}{1-u^2}=\psi,\qquad\|w\|^2=\frac{2\psi-1}{1-r^2}, ⟨ v , w ⟩ = 1 − u α + 1 + u 1 − α = 1 − u 2 1 − ν u = ψ , ∥ w ∥ 2 = 1 − r 2 2 ψ − 1 ,
where the second identity follows by squaring w = v + r Q w w=v+rQw w = v + r Qw . Hence ⟨ g ^ k , x ^ k ⟩ = c r 2 k = 2 f ^ k \langle\hat{g}_k,\hat{x}_k\rangle=cr^{2k}=2\hat{f}_k ⟨ g ^ k , x ^ k ⟩ = c r 2 k = 2 f ^ k .
Residuals of the tail. The tail is an orbit (6) with X = τ w X=\tau w X = τ w and G = v G=v G = v , and Q 2 d = I Q^{2d}=I Q 2 d = I , so K ℓ ± K^\pm_\ell K ℓ ± depends only on j = ℓ m o d 2 d j=\ell\bmod2d j = ℓ mod 2 d , and we write K j ± K^\pm_j K j ± . Since ( 1 − r ℓ ) 2 > 0 (1-r^\ell)^2>0 ( 1 − r ℓ ) 2 > 0 , by (7) all residuals among the tail triples and the minimizer are positive once C ⋆ > 0 C_\star>0 C ⋆ > 0 and K j ± > 0 K^\pm_j>0 K j ± > 0 for 1 ≤ j < 2 d 1\leq j<2d 1 ≤ j < 2 d .
Here C ⋆ = c 2 − ( 1 + μ τ 2 ∥ w ∥ 2 − 2 μ c ) / ( 2 ( 1 − μ ) ) C_\star=\frac{c}{2}-(1+\mu\tau^2\|w\|^2-2\mu c)/(2(1-\mu)) C ⋆ = 2 c − ( 1 + μ τ 2 ∥ w ∥ 2 − 2 μ c ) / ( 2 ( 1 − μ )) . Since μ τ 2 ∥ w ∥ 2 = τ ( 2 ψ − 1 ) / ( 2 ( 2 − η ) ) < 3 5 c \mu\tau^2\|w\|^2=\tau(2\psi-1)/(2(2-\eta))<\frac{3}{5}c μ τ 2 ∥ w ∥ 2 = τ ( 2 ψ − 1 ) / ( 2 ( 2 − η )) < 5 3 c , μ < 1 100 \mu<\frac{1}{100} μ < 100 1 and c ≥ 12 ⋅ 64 d 2 c\geq12\cdot64d^2 c ≥ 12 ⋅ 64 d 2 , we get C ⋆ ≥ c / 8 C_\star\geq c/8 C ⋆ ≥ c /8 . The quadratic part of K j ± K^\pm_j K j ± is
P j = ∥ v − Q j v ∥ 2 + μ τ 2 ∥ w − Q j w ∥ 2 − 2 μ τ ⟨ v − Q j v , w − Q j w ⟩ 2 ( 1 − μ ) . P_j=\frac{\|v-Q^jv\|^2+\mu\tau^2\|w-Q^jw\|^2-2\mu\tau\langle v-Q^jv,w-Q^jw\rangle}{2(1-\mu)}. P j = 2 ( 1 − μ ) ∥ v − Q j v ∥ 2 + μ τ 2 ∥ w − Q j w ∥ 2 − 2 μτ ⟨ v − Q j v , w − Q j w ⟩ .
Since ( I − Q ) w = v − η Q w (I-Q)w=v-\eta Qw ( I − Q ) w = v − η Qw and η 2 ∥ w ∥ 2 ≤ η ψ ≤ 1 / d \eta^2\|w\|^2\leq\eta\psi\leq1/d η 2 ∥ w ∥ 2 ≤ η ψ ≤ 1/ d by (9) , ∥ w − Q j w ∥ ≤ j ( 1 + η ∥ w ∥ ) ≤ 4 d \|w-Q^jw\|\leq j(1+\eta\|w\|)\leq4d ∥ w − Q j w ∥ ≤ j ( 1 + η ∥ w ∥ ) ≤ 4 d . Hence μ τ 2 ∥ w − Q j w ∥ 2 ≤ 8 d 2 η τ = 256 d 4 / ( κ θ 2 ) ≤ 1 4 \mu\tau^2\|w-Q^jw\|^2\leq8d^2\eta\tau=256d^4/(\kappa\theta^2)\leq\frac14 μ τ 2 ∥ w − Q j w ∥ 2 ≤ 8 d 2 η τ = 256 d 4 / ( κ θ 2 ) ≤ 4 1 by μ τ = η / 2 \mu\tau=\eta/2 μτ = η /2 , and the mixed term is at most 8 d η ≤ 1 / ( 16 d 2 ) 8d\eta\leq1/(16d^2) 8 d η ≤ 1/ ( 16 d 2 ) . With ∥ v − Q j v ∥ 2 = 2 \|v-Q^jv\|^2=2 ∥ v − Q j v ∥ 2 = 2 for j ≠ d j\ne d j = d and ∥ v − Q d v ∥ 2 ≤ 4 \|v-Q^dv\|^2\leq4 ∥ v − Q d v ∥ 2 ≤ 4 this gives P j < 3 2 P_j<\frac32 P j < 2 3 for j ≠ d j\ne d j = d and P d < 9 4 P_d<\frac94 P d < 4 9 . Moreover
K j + = τ ( ψ − ⟨ Q j v , w ⟩ ) − P j , K j − = τ ( ψ − ⟨ v , Q j w ⟩ ) − P j . K^+_j=\tau(\psi-\langle Q^jv,w\rangle)-P_j,\qquad K^-_j=\tau(\psi-\langle v,Q^jw\rangle)-P_j. K j + = τ ( ψ − ⟨ Q j v , w ⟩) − P j , K j − = τ ( ψ − ⟨ v , Q j w ⟩) − P j .
Since w = v + r Q w w=v+rQw w = v + r Qw , the numbers a j = ⟨ Q j v , w ⟩ a_j=\langle Q^jv,w\rangle a j = ⟨ Q j v , w ⟩ satisfy a 0 = ψ a_0=\psi a 0 = ψ and a j = ⟨ Q j v , v ⟩ + r a j − 1 a_j=\langle Q^jv,v\rangle+ra_{j-1} a j = ⟨ Q j v , v ⟩ + r a j − 1 . By (10) and ⟨ v , Q d v ⟩ = − ν \langle v,Q^dv\rangle=-\nu ⟨ v , Q d v ⟩ = − ν this gives a j = r j ψ a_j=r^j\psi a j = r j ψ for j < d j<d j < d and a j = r j ψ − r j − d ν a_j=r^j\psi-r^{j-d}\nu a j = r j ψ − r j − d ν for d ≤ j < 2 d d\leq j<2d d ≤ j < 2 d , so a j ≤ r j ψ a_j\leq r^j\psi a j ≤ r j ψ , and ⟨ v , Q j w ⟩ = a 2 d − j \langle v,Q^jw\rangle=a_{2d-j} ⟨ v , Q j w ⟩ = a 2 d − j . Both inner products are therefore at most r ψ r\psi r ψ for j ≠ d j\ne d j = d and at most u ψ u\psi u ψ for j = d j=d j = d , and (9) and ψ ( 1 − u ) = θ / ( 1 + u ) \psi(1-u)=\theta/(1+u) ψ ( 1 − u ) = θ / ( 1 + u ) give
K j ± ≥ τ ψ η − P j > 1 2 ( j ≠ d ) , K d ± ≥ τ θ 1 + u − P d ≥ 2 d − 9 4 > 0. K^\pm_j\geq\tau\psi\eta-P_j>\frac12\quad(j\ne d),\qquad K^\pm_d\geq\frac{\tau\theta}{1+u}-P_d\geq2d-\frac94>0. K j ± ≥ τ ψ η − P j > 2 1 ( j = d ) , K d ± ≥ 1 + u τ θ − P d ≥ 2 d − 4 9 > 0.
Cold start. Let E 0 , E 1 , E 2 ⊥ R 2 d E_0,E_1,E_2\perp\mathbb{R}^{2d} E 0 , E 1 , E 2 ⊥ R 2 d be orthonormal, and G i = 10 − i / 2 E i G_i=10^{-i/2}E_i G i = 1 0 − i /2 E i . For ϑ > 0 \vartheta>0 ϑ > 0 put
g 0 = ϑ − 1 G 0 , g 1 = Z 1 + ϑ − 1 G 1 , g 2 = g ^ 2 + ϑ − 1 G 2 , g k = g ^ k ( k ≥ 3 ) , g_0=\vartheta^{-1}G_0,\qquad g_1=Z_1+\vartheta^{-1}G_1,\qquad g_2=\hat{g}_2+\vartheta^{-1}G_2,\qquad g_k=\hat{g}_k\quad(k\geq3), g 0 = ϑ − 1 G 0 , g 1 = Z 1 + ϑ − 1 G 1 , g 2 = g ^ 2 + ϑ − 1 G 2 , g k = g ^ k ( k ≥ 3 ) ,
x k = x ^ k x_k=\hat{x}_k x k = x ^ k and f k = f ^ k f_k=\hat{f}_k f k = f ^ k for k ≥ 3 k\geq3 k ≥ 3 , and x i = x i + 1 + τ g i x_i=x_{i+1}+\tau g_i x i = x i + 1 + τ g i and f i = f ^ 3 + τ 2 ϑ 2 ∑ h = i 2 10 − h f_i=\hat{f}_3+\frac{\tau}{2\vartheta^2}\sum_{h=i}^{2}10^{-h} f i = f ^ 3 + 2 ϑ 2 τ ∑ h = i 2 1 0 − h for i = 2 , 1 , 0 i=2,1,0 i = 2 , 1 , 0 . Then x k + 1 − x k = − τ g k x_{k+1}-x_k=-\tau g_k x k + 1 − x k = − τ g k for every k ≥ 0 k\geq0 k ≥ 0 , and the data are bounded. The map ( x , g , f ) ↦ ( ϑ x , ϑ g , ϑ 2 f ) (x,g,f)\mapsto(\vartheta x,\vartheta g,\vartheta^2f) ( x , g , f ) ↦ ( ϑ x , ϑ g , ϑ 2 f ) multiplies all residuals by ϑ 2 \vartheta^2 ϑ 2 . As ϑ → 0 \vartheta\to0 ϑ → 0 it sends the tail triples to ( 0 , 0 , 0 ) (0,0,0) ( 0 , 0 , 0 ) uniformly in k k k and the first three triples to X i = τ ∑ h = i 2 G h X_i=\tau\sum_{h=i}^{2}G_h X i = τ ∑ h = i 2 G h , G i G_i G i , F i = τ 2 ∑ h = i 2 10 − h F_i=\frac{\tau}{2}\sum_{h=i}^{2}10^{-h} F i = 2 τ ∑ h = i 2 1 0 − h . Put ( X 3 , G 3 , F 3 ) = ( 0 , 0 , 0 ) (X_3,G_3,F_3)=(0,0,0) ( X 3 , G 3 , F 3 ) = ( 0 , 0 , 0 ) , let 0 ≤ i < j ≤ 3 0\leq i<j\leq3 0 ≤ i < j ≤ 3 and U = ∑ h = j − 1 2 10 − h ∈ [ 10 − i , 10 9 10 − i ] U=\sum_{h=j-1}^{2}10^{-h}\in[10^{-i},\frac{10}{9}10^{-i}] U = ∑ h = j − 1 2 1 0 − h ∈ [ 1 0 − i , 9 10 1 0 − i ] . Then ∥ X i − X j ∥ 2 = τ 2 U \|X_i-X_j\|^2=\tau^2U ∥ X i − X j ∥ 2 = τ 2 U , ⟨ G i − G j , X i − X j ⟩ = τ 10 − i \langle G_i-G_j,X_i-X_j\rangle=\tau10^{-i} ⟨ G i − G j , X i − X j ⟩ = τ 1 0 − i , F i − F j = τ 2 U F_i-F_j=\frac{\tau}{2}U F i − F j = 2 τ U and ∥ G i − G j ∥ 2 ≤ 1.1 ⋅ 10 − i \|G_i-G_j\|^2\leq1.1\cdot10^{-i} ∥ G i − G j ∥ 2 ≤ 1.1 ⋅ 1 0 − i .
With τ ≥ 12 \tau\ge12 τ ≥ 12 , μ τ ≤ 1 / ( 256 d 3 ) \mu\tau\le1/(256d^3) μτ ≤ 1/ ( 256 d 3 ) and μ < 1 100 \mu< \frac{1}{100} μ < 100 1 , the quadratic part of the residual is at most τ 10 − i / 10 \tau10^{-i}/10 τ 1 0 − i /10 , so both residuals are at least min { 1 2 U , τ 10 − i − 1 2 U } − τ 10 − i 10 ≥ τ 10 − i 3 \min\{\frac{1}{2}U,\tau10^{-i}-\frac{1}{2}U\}-\frac{\tau10^{-i}}{10} \ge\frac{\tau10^{-i}}{3} min { 2 1 U , τ 1 0 − i − 2 1 U } − 10 τ 1 0 − i ≥ 3 τ 1 0 − i . By continuity and uniform convergence, for small ϑ \vartheta ϑ all residuals that involve one of the first three triples are positive. The residuals among the tail triples and the minimizer do not depend on ϑ \vartheta ϑ . Lemma 3.4 gives F ∈ F 1 / κ , 1 ( R 2 d + 3 ) F \in\mathcal{F}_{1/\kappa,1}(\mathbb{R}^{2d+3}) F ∈ F 1/ κ , 1 ( R 2 d + 3 ) with minimizer 0 0 0 that interpolates the data.
The trajectory. Every step is a multiple of the current gradient, so the window at iteration k k k is spanned by g k − m , … , g k g_{k-m},\ldots,g_k g k − m , … , g k , whose lags to k + 1 k+1 k + 1 lie between 1 1 1 and d − 1 d-1 d − 1 . By Lemma 4.4 it suffices to show that g k + 1 g_{k+1} g k + 1 is orthogonal to these gradients and to z k z_k z k and y k y_k y k . Among the tail gradients this is (10) . The cold gradients are mutually orthogonal, since ⟨ Z 1 , g ^ 2 ⟩ = 0 \langle Z_1,\hat{g}_2\rangle=0 ⟨ Z 1 , g ^ 2 ⟩ = 0 by (11) . For 3 ≤ j ≤ d 3 \le j \le d 3 ≤ j ≤ d we have g ^ j ⊥ Z 1 \hat{g}_j \perp Z_1 g ^ j ⊥ Z 1 , because Z j − 1 = ρ j − 2 Z 1 + ∑ i = 2 j − 1 ρ j − 1 − i g ^ i Z_{j-1}=\rho^{j-2}Z_1+\sum_{i=2}^{j-1}\rho^{j-1-i}\hat{g}_i Z j − 1 = ρ j − 2 Z 1 + ∑ i = 2 j − 1 ρ j − 1 − i g ^ i , g ^ j ⊥ Z j − 1 \hat{g}_j \perp Z_{j-1} g ^ j ⊥ Z j − 1 and g ^ j ⊥ g ^ i \hat{g}_j \perp\hat{g}_i g ^ j ⊥ g ^ i for these lags. The registers satisfy z 1 = g 1 z_1=g_1 z 1 = g 1 and y 1 = − ρ τ g 0 y_1=-\rho\tau g_0 y 1 = − ρ τ g 0 , and by induction their components in R 2 d \mathbb{R}^{2d} R 2 d are Z k Z_k Z k and − τ ρ Z k − 1 -\tau\rho Z_{k-1} − τ ρ Z k − 1 for k ≥ 2 k \ge2 k ≥ 2 . For k ≥ 2 k \ge2 k ≥ 2 , g ^ k + 1 \hat{g}_{k+1} g ^ k + 1 is orthogonal to Z k Z_k Z k by (11) and to Z k − 1 = ( Z k − g ^ k ) / ρ Z_{k-1}=(Z_k-\hat{g}_k)/\rho Z k − 1 = ( Z k − g ^ k ) / ρ . The components of the registers along E 0 , E 1 , E 2 E_0,E_1,E_2 E 0 , E 1 , E 2 are orthogonal to the tail gradients, and for k ≤ 1 k \le1 k ≤ 1 the registers lie in the span of g 0 g_0 g 0 and g 1 g_1 g 1 . Hence g k + 1 ⊥ S k g_{k+1}\perp S_k g k + 1 ⊥ S k for every admissible rule, and every such rule generates the trajectory. Since F ⋆ = 0 F^\star=0 F ⋆ = 0 , f ( x k ) − f ⋆ = c 2 r 2 k f(x_k)-f^\star=\frac{c}{2}r^{2k} f ( x k ) − f ⋆ = 2 c r 2 k for k ≥ 3 k \ge3 k ≥ 3 , and − log ( 1 − η ) ≤ 2 η -\log(1-\eta)\le2\eta − log ( 1 − η ) ≤ 2 η gives
lim ϵ ↓ 0 N ( ϵ ) log ( 1 / ϵ ) = 1 − 2 log r ≥ 1 4 η = κ θ 32 d . \lim_{\epsilon\downarrow0}\frac{N(\epsilon)}{\log(1/\epsilon)}=\frac{1}{-2\log r}\ge\frac{1}{4\eta}=\frac{\kappa\theta}{32d}. ϵ ↓ 0 lim log ( 1/ ϵ ) N ( ϵ ) = − 2 log r 1 ≥ 4 η 1 = 32 d κ θ .
The right time scale
Theorem 4.2 excludes memories that forget too slowly and Theorem 4.9 a pair that forgets too fast, which leaves a single time scale.
Corollary 4.10 (long memory or fast forgetting) For every κ \kappa κ let a method use one memory of Definition 4.1 with window m = m ( κ ) m=m(\kappa) m = m ( κ ) and forgetting index δ = δ ( κ ) \delta=\delta(\kappa) δ = δ ( κ ) , and let lim sup ϵ ↓ 0 N ( ϵ ) / log ( 1 / ϵ ) ≤ C κ \limsup_{\epsilon\downarrow0}N(\epsilon)/\log(1/\epsilon)\le C\sqrt{\kappa} lim sup ϵ ↓ 0 N ( ϵ ) / log ( 1/ ϵ ) ≤ C κ on all of F 1 / κ , 1 \mathcal{F}_{1/\kappa,1} F 1/ κ , 1 in every dimension, with C C C independent of κ \kappa κ . Then, with constants depending only on C C C and for all large κ \kappa κ :
(a) m ≥ c κ m\ge c\sqrt{\kappa} m ≥ c κ or δ ≥ c / κ \delta\ge c/\sqrt{\kappa} δ ≥ c / κ .
(b) If the memory consists of the window and one pair as in Theorem 4.9 and m = o ( κ 1 / 6 ) m=o(\kappa^{1/6}) m = o ( κ 1/6 ) , then Ω ( κ − 1 / 2 ) ≤ 1 − ρ ≤ O ( ( m + 1 ) κ − 1 / 2 ) \Omega(\kappa^{-1/2})\le1-\rho\le O((m+1)\kappa^{-1/2}) Ω ( κ − 1/2 ) ≤ 1 − ρ ≤ O (( m + 1 ) κ − 1/2 ) . In particular 1 − ρ = Θ ( κ − 1 / 2 ) 1-\rho=\Theta(\kappa^{-1/2}) 1 − ρ = Θ ( κ − 1/2 ) for a bounded window.
Proof (a) Let p = max { 8 , m + 2 , ⌊ κ δ ⌋ + 1 } p=\max\{8,m+2,\lfloor\kappa\delta\rfloor+1\} p = max { 8 , m + 2 , ⌊ κ δ ⌋ + 1 } , so that p ≤ max { m + 8 , κ δ + 1 } p\le\max\{m+8,\kappa\delta+1\} p ≤ max { m + 8 , κ δ + 1 } and δ < p / κ \delta<p/\kappa δ < p / κ . If κ ≥ 100 p 2 \kappa\ge100p^2 κ ≥ 100 p 2 , Theorem 4.2 gives κ / ( 4 p ) ≤ C κ \kappa/(4p)\le C\sqrt{\kappa} κ / ( 4 p ) ≤ C κ . Hence p ≥ c 0 κ p\ge c_0\sqrt{\kappa} p ≥ c 0 κ with c 0 = min { 1 10 , 1 4 C } c_0=\min\{\frac{1}{10},\frac{1}{4C}\} c 0 = min { 10 1 , 4 C 1 } , and (a) follows with c = c 0 / 2 c=c_0/2 c = c 0 /2 once $c_0\sqrt{\kappa}\ge 16.
(b) The pair is a gradient sum and a step sum with the normalized weights ρ − i \rho^{-i} ρ − i , so δ = 1 − ρ \delta=1-\rho δ = 1 − ρ , and part (a) gives δ = O ( κ − 1 / 2 ) \delta=O(\kappa^{-1/2}) δ = O ( κ − 1/2 ) because m = o ( κ ) m=o(\sqrt{\kappa}) m = o ( κ ) . A window of length 0 is contained in one of length 1, so let m ≥ 1 m\ge1 m ≥ 1 and d = m + 2 d=m+2 d = m + 2 and θ = 1 − ρ d \theta=1-\rho^d θ = 1 − ρ d . Either κ θ 2 < 1024 d 4 \kappa\theta^2<1024d^4 κ θ 2 < 1024 d 4 , or Theorem 4.9 gives κ θ / ( 32 d ) ≤ C κ \kappa\theta/(32d)\le C\sqrt{\kappa} κ θ / ( 32 d ) ≤ C κ . In both cases θ ≤ 32 d ( d + C ) / κ \theta\le32d(d+C)/\sqrt{\kappa} θ ≤ 32 d ( d + C ) / κ , so d θ → 0 d\theta\to0 d θ → 0 because m = o ( κ 1 / 6 ) m=o(\kappa^{1/6}) m = o ( κ 1/6 ) . For large κ \kappa κ this gives δ d ≤ d θ ≤ 1 \delta d\le d\theta\le1 δ d ≤ d θ ≤ 1 , hence θ = 1 − ( 1 − δ ) d ≥ 1 2 δ d \theta=1-(1-\delta)^d\ge\frac{1}{2}\delta d θ = 1 − ( 1 − δ ) d ≥ 2 1 δ d and δ ≤ 2 θ / d ≤ 64 ( d + C ) / κ \delta\le2\theta/d\le64(d+C)/\sqrt{\kappa} δ ≤ 2 θ / d ≤ 64 ( d + C ) / κ . \qed
By (2) every uniformly accelerated method satisfies the assumption with C = 1 / c C=1/c C = 1/ c , so the corollary applies to it. Either condition suffices: a window of length 1.39 κ 1.39\sqrt{\kappa} 1.39 κ is accelerated (Proposition 5.1), and so is a filter pair with 1 − ρ ∈ [ 1 / ( 2 κ ) , 1 / κ ] 1-\rho\in[1/(2\sqrt{\kappa}),1/\sqrt{\kappa}] 1 − ρ ∈ [ 1/ ( 2 κ ) , 1/ κ ] (Lemma 5.3, whose proof uses only 1 − ρ ≤ 1 / κ 1-\rho\le1/\sqrt{\kappa} 1 − ρ ≤ 1/ κ ). A register that the method never uses increases δ \delta δ without changing the iterates, so part (a) is only a necessary condition. The activation schedule and the smooth pair of § 5.2 lie outside Definition 4.1 : the first has unboundedly many registers, and the second adapts its weights to the run. For memories whose factors change in time, δ \delta δ gives no upper bound like part (b).
In our lower bounds the memory, including the number of aggregates, is fixed before the instance. Since n n n gradient sums with weights i a i^a i a , a < n a < n a < n , terminate on R n \mathbb{R}^n R n , families whose number of aggregates grows with the dimension are not covered. One function serves all accuracies, so no prefactor depending on κ \kappa κ compensates the bound. The convex rate of SESOP (Proposition 3.2 ) does not exclude such a slow tail, and the contraction ( 1 − 3 / κ ) 2 (1-3/\kappa)^2 ( 1 − 3/ κ ) 2 on the instances of Corollary 4.3 is of the order of the worst case 1 − 4 / κ + O ( κ − 2 ) 1 - 4/\kappa+ O(\kappa^{-2}) 1 − 4/ κ + O ( κ − 2 ) of GD-ELS [36 ] . Like that worst case, our bounds concern the asymptotic rate. The constructions first reduce the gap by a factor that depends on κ \kappa κ , so they say nothing about moderate accuracy. They are not twice differentiable, but § 4.5 shows that smoothness does not help.
Smoothness does not help
The instances of Theorems 4.2 and 4.9 have no second derivatives. Near a minimizer with a continuous Hessian a function looks like a quadratic, on which SESOP coincides with conjugate gradients, but the memory accumulated before the quadratic region keeps SESOP slow. Both instances below rest on one exact plane step on a quadratic with two eigenvalues.
Lemma 4.12 (plane rule on two eigenvalues) Let B = I + b v v T B = I + bvv^{\mathsf{T}} B = I + b v v T on R 3 \mathbb{R}^3 R 3 with ∥ v ∥ = 1 \lVert v\rVert= 1 ∥ v ∥ = 1 and b > 0 b > 0 b > 0 , and q ( x ) = 1 2 x T B x q(x) = \frac{1}{2}x^{\mathsf{T}}Bx q ( x ) = 2 1 x T B x . Let the gradient g = B x g = Bx g = B x and the inherited step s s s be nonzero and orthogonal, put γ = ⟨ v , g ⟩ 2 / ∥ g ∥ 2 \gamma= \langle v,g\rangle^2/\lVert g\rVert^2 γ = ⟨ v , g ⟩ 2 / ∥ g ∥ 2 , ζ = ⟨ v , s ⟩ 2 / ∥ s ∥ 2 \zeta= \langle v,s\rangle^2/\lVert s\rVert^2 ζ = ⟨ v , s ⟩ 2 / ∥ s ∥ 2 and z = 1 − γ − ζ z = 1 - \gamma- \zeta z = 1 − γ − ζ , and assume γ , ζ , z > 0 \gamma,\zeta,z > 0 γ , ζ , z > 0 . Let x + x^+ x + minimize q q q over x + span { g , s } x + \operatorname{span}\{g,s\} x + span { g , s } , with gradient g + g^+ g + and step s + = x + − x s^+ = x^+ - x s + = x + − x . Then
γ + = z , ζ + = γ ( 1 + b ζ ) 2 + b 2 γ ζ , J + = J , where J = b γ ζ z ( 1 + b ζ ) 2 , \gamma^+ = z,\qquad\zeta^+ = \frac{\gamma}{(1+b\zeta)^2+b^2\gamma\zeta},\qquad J^+ = J,\qquad\text{where } J = \frac{b\gamma\zeta z}{(1+b\zeta)^2}, γ + = z , ζ + = ( 1 + b ζ ) 2 + b 2 γ ζ γ , J + = J , where J = ( 1 + b ζ ) 2 bγ ζ z , and
q ( x + ) q ( x ) = b 2 γ z ( 1 + b − b γ ) ( 1 + b − b z ) ≥ b J 1 + b J . \frac{q(x^+)}{q(x)} = \frac{b^2\gamma z}{(1+b-b\gamma)(1+b-bz)} \ge\frac{bJ}{1+bJ}. q ( x ) q ( x + ) = ( 1 + b − bγ ) ( 1 + b − b z ) b 2 γ z ≥ 1 + b J b J . If 1 + b = K > 12 1+b = K > 12 1 + b = K > 12 and r ∈ ( 0 , 1 ) r \in(0,1) r ∈ ( 0 , 1 ) solves K ( 1 − r ) = ( 1 + r ) 2 K(1-r) = (1+r)^2 K ( 1 − r ) = ( 1 + r ) 2 , then γ = z = 1 + r 3 + r \gamma= z = \frac{1+r}{3+r} γ = z = 3 + r 1 + r , ζ = 1 − r 3 + r \zeta= \frac{1-r}{3+r} ζ = 3 + r 1 − r is a fixed point, at which q ( x + ) / q ( x ) = r 2 q(x^+)/q(x) = r^2 q ( x + ) / q ( x ) = r 2 and J = r / ( 3 + r ) 2 > 1 32 J = r/(3+r)^2 > \frac{1}{32} J = r / ( 3 + r ) 2 > 32 1 .
Proof. The gradient g + g^+ g + is orthogonal to g g g and s s s , so it is parallel to the unit normal h h h of the plane, and γ + = ⟨ v , h ⟩ 2 = 1 − γ − ζ \gamma^+ = \langle v,h\rangle^2 = 1-\gamma-\zeta γ + = ⟨ v , h ⟩ 2 = 1 − γ − ζ . Write s + = − α g + β s s^+ = -\alpha g+\beta s s + = − α g + β s . Then g + = g + B s + ⊥ s g^+ = g+Bs^+ \perp s g + = g + B s + ⊥ s gives
β ∥ s ∥ 2 ( 1 + b ζ ) = α b ⟨ v , g ⟩ ⟨ v , s ⟩ , \beta\lVert s\rVert^2(1+b\zeta)=\alpha b\langle v,g\rangle\langle v,s\rangle, β ∥ s ∥ 2 ( 1 + b ζ ) = α b ⟨ v , g ⟩ ⟨ v , s ⟩ ,
hence ⟨ v , s + ⟩ = − α ⟨ v , g ⟩ 1 + b ζ \langle v,s^+\rangle=-\frac{\alpha\langle v,g\rangle}{1+b\zeta} ⟨ v , s + ⟩ = − 1 + b ζ α ⟨ v , g ⟩ ,
and ∥ s + ∥ 2 = α 2 ∥ g ∥ 2 + β 2 ∥ s ∥ 2 \lVert s^+\rVert^2=\alpha^2\lVert g\rVert^2+\beta^2\lVert s\rVert^2 ∥ s + ∥ 2 = α 2 ∥ g ∥ 2 + β 2 ∥ s ∥ 2 gives ζ + \zeta^+ ζ + . With D ˉ = ( 1 + b ζ ) 2 + b 2 γ ζ \bar D=(1+b\zeta)^2+b^2\gamma\zeta D ˉ = ( 1 + b ζ ) 2 + b 2 γ ζ a direct computation gives
ζ + = ζ ( 1 + b γ + b ζ ) 2 D ˉ , 1 + b ζ + = ( 1 + b ζ ) ( 1 + b γ + b ζ ) D ˉ . \zeta^+=\frac{\zeta(1+b\gamma+b\zeta)^2}{\bar D},\qquad1+b\zeta^+=\frac{(1+b\zeta)(1+b\gamma+b\zeta)}{\bar D}. ζ + = D ˉ ζ ( 1 + bγ + b ζ ) 2 , 1 + b ζ + = D ˉ ( 1 + b ζ ) ( 1 + bγ + b ζ ) .
hence J + = J J^+=J J + = J . Since B − 1 = I − b 1 + b v v ⊤ B^{-1}=I-\frac{b}{1+b}vv^\top B − 1 = I − 1 + b b v v ⊤ and B − 1 ( g + − g ) = s + B^{-1}(g^+-g)=s^+ B − 1 ( g + − g ) = s + is orthogonal to h h h , we get
g + = − b ⟨ v , g ⟩ ⟨ v , h ⟩ 1 + b − b ⟨ v , h ⟩ 2 h , g^+=-\frac{b\langle v,g\rangle\langle v,h\rangle}{1+b-b\langle v,h\rangle^2}h, g + = − 1 + b − b ⟨ v , h ⟩ 2 b ⟨ v , g ⟩ ⟨ v , h ⟩ h ,
and q = 1 2 g ⊤ B − 1 g q=\frac{1}{2}g^\top B^{-1}g q = 2 1 g ⊤ B − 1 g gives the ratio. The identity ( 1 + b − b γ ) ( 1 + b − b ζ ) = b 2 γ ζ + ( 1 + b ) ( 1 + b ζ ) (1+b-b\gamma)(1+b-b\zeta)=b^2\gamma\zeta+(1+b)(1+b\zeta) ( 1 + b − bγ ) ( 1 + b − b ζ ) = b 2 γ ζ + ( 1 + b ) ( 1 + b ζ ) and ( 1 + b ) ζ ≤ 1 + b ζ (1+b)\zeta\leq1+b\zeta ( 1 + b ) ζ ≤ 1 + b ζ give the lower bound. For the fixed point, b ( 1 − r ) = r ( 3 + r ) b(1-r)=r(3+r) b ( 1 − r ) = r ( 3 + r ) gives b ζ = r b\zeta=r b ζ = r and b γ = r ( 1 + r ) / ( 1 − r ) b\gamma=r(1+r)/(1-r) bγ = r ( 1 + r ) / ( 1 − r ) , and the formulas follow. The function r / ( 3 + r ) 2 r/(3+r)^2 r / ( 3 + r ) 2 increases on ( 0 , 1 ) (0,1) ( 0 , 1 ) , and K ≥ 12 K\geq12 K ≥ 12 gives r > 1 2 r>\frac{1}{2} r > 2 1 . □ \square □
With b = κ − 1 b=\kappa-1 b = κ − 1 the tail of the following explicit instance for the plane rule sits at this fixed point, and its local and global condition numbers are comparable.
Proposition 4.13 (plane rule in R 3 \mathbb{R}^3 R 3 ) Let κ ≥ 100 \kappa\geq100 κ ≥ 100 , q ( x ) = 1 2 x ⊤ D x q(x)=\frac{1}{2}x^\top Dx q ( x ) = 2 1 x ⊤ D x with D = diag ( 1 , 1 , κ ) D=\operatorname{diag}(1,1,\kappa) D = diag ( 1 , 1 , κ ) , and let r ∈ ( 0 , 1 ) r\in(0,1) r ∈ ( 0 , 1 ) solve ( 1 + r ) 2 = κ ( 1 − r ) (1+r)^2=\kappa(1-r) ( 1 + r ) 2 = κ ( 1 − r ) . There are a convex function ϕ ∈ C ∞ ( R 3 ) \phi\in C^\infty(\mathbb{R}^3) ϕ ∈ C ∞ ( R 3 ) , ϕ ≥ 0 \phi\geq0 ϕ ≥ 0 , that vanishes on a neighbourhood of the unit ball, and a point x 0 x_0 x 0 such that f = q + ϕ f=q+\phi f = q + ϕ is 1 1 1 -strongly convex with a ( κ + 20 κ ) (\kappa+20\sqrt{\kappa}) ( κ + 20 κ ) -Lipschitz gradient and a 34 κ 34\sqrt{\kappa} 34 κ -Lipschitz Hessian, and the plane rule on f f f started at x 0 x_0 x 0 satisfies f ( x k + 1 ) = r 2 f ( x k ) f(x_{k+1})=r^2f(x_k) f ( x k + 1 ) = r 2 f ( x k ) for k ≥ 1 k\geq1 k ≥ 1 . Here r 2 = 1 − 8 / κ + O ( κ − 2 ) r^2=1-8/\kappa+O(\kappa^{-2}) r 2 = 1 − 8/ κ + O ( κ − 2 ) and lim inf ε ↓ 0 N ( ε ) / log ( 1 / ε ) ≥ ( κ − 4 ) / 8 \liminf_{\varepsilon\downarrow0}N(\varepsilon)/\log(1/\varepsilon)\geq(\kappa-4)/8 lim inf ε ↓ 0 N ( ε ) / log ( 1/ ε ) ≥ ( κ − 4 ) /8 .
Proof. Let cos α = 1 + r 2 \cos\alpha=\frac{1+r}{2} cos α = 2 1 + r , η = cos α κ \eta=\frac{\sqrt{\cos\alpha}}{\kappa} η = κ c o s α and x ^ k = ( r k cos k α , r k sin k α , ( − r ) k η ) \hat x_k=(r^k\cos k\alpha,r^k\sin k\alpha,(-r)^k\eta) x ^ k = ( r k cos k α , r k sin k α , ( − r ) k η ) for k ∈ Z k\in\mathbb{Z} k ∈ Z .
The tail. Put g ^ k = D x ^ k \hat g_k=D\hat x_k g ^ k = D x ^ k and s ^ k = x ^ k − x ^ k − 1 \hat s_k=\hat x_k-\hat x_{k-1} s ^ k = x ^ k − x ^ k − 1 . Identify the first two coordinates with C \mathbb{C} C , so that they equal r k e i k α r^k e^{ik\alpha} r k e ik α . Since e i α + e − i α = 1 + r e^{i\alpha}+e^{-i\alpha}=1+r e i α + e − i α = 1 + r and κ ( 1 − r ) = ( 1 + r ) 2 \kappa(1-r)=(1+r)^2 κ ( 1 − r ) = ( 1 + r ) 2 , both blocks satisfy
s ^ k + 1 = − ( 1 − r ) g ^ k + r 2 s ^ k . \hat s_{k+1}=-(1-r)\hat g_k+r^2\hat s_k. s ^ k + 1 = − ( 1 − r ) g ^ k + r 2 s ^ k .
With κ 2 η 2 = cos α \kappa^2\eta^2=\cos\alpha κ 2 η 2 = cos α and ( 1 + r ) κ η 2 = cos α ( 1 − r ) / ( 1 + r ) (1+r)\kappa\eta^2=\cos\alpha(1-r)/(1+r) ( 1 + r ) κ η 2 = cos α ( 1 − r ) / ( 1 + r ) we get
⟨ g ^ k + 1 , g ^ k ⟩ = r 2 k + 1 ( cos α − κ 2 η 2 ) = 0 , \langle\hat g_{k+1},\hat g_k\rangle=r^{2k+1}(\cos\alpha-\kappa^2\eta^2)=0, ⟨ g ^ k + 1 , g ^ k ⟩ = r 2 k + 1 ( cos α − κ 2 η 2 ) = 0 ,
⟨ g ^ k + 1 , s ^ k + 1 ⟩ = r 2 k + 1 ( r − cos α + ( 1 + r ) κ η 2 ) = 0 , \langle\hat g_{k+1},\hat s_{k+1}\rangle=r^{2k+1}(r-\cos\alpha+(1+r)\kappa\eta^2)=0, ⟨ g ^ k + 1 , s ^ k + 1 ⟩ = r 2 k + 1 ( r − cos α + ( 1 + r ) κ η 2 ) = 0 ,
and the recursion gives ⟨ g ^ k + 1 , s ^ k ⟩ = 0 \langle\hat g_{k+1},\hat s_k\rangle=0 ⟨ g ^ k + 1 , s ^ k ⟩ = 0 . Hence x ^ k + 1 \hat x_{k+1} x ^ k + 1 minimizes q q q over x ^ k + span { g ^ k , s ^ k } \hat x_k+\operatorname{span}\{\hat g_k,\hat s_k\} x ^ k + span { g ^ k , s ^ k } , and q ( x ^ k + 1 ) = r 2 q ( x ^ k ) q(\hat x_{k+1})=r^2q(\hat x_k) q ( x ^ k + 1 ) = r 2 q ( x ^ k ) .
The cold start. Let d = s ^ 1 / ∥ s ^ 1 ∥ d=\hat s_1/\lVert\hat s_1\rVert d = s ^ 1 / ∥ s ^ 1 ∥ , λ = 2 + κ \lambda=2+\sqrt{\kappa} λ = 2 + κ , x 0 = x ^ 1 − 8 d x_0=\hat x_1-8d x 0 = x ^ 1 − 8 d and w 0 = − 8 λ d − D x 0 w_0=-8\lambda d-Dx_0 w 0 = − 8 λ d − D x 0 . The formulas for x ^ 0 , x ^ 1 \hat x_0,\hat x_1 x ^ 0 , x ^ 1 and 1 − r = ( 1 + r ) 2 / κ ∈ [ 1 / κ , 4 / κ ] 1-r=(1+r)^2/\kappa\in[1/\kappa,4/\kappa] 1 − r = ( 1 + r ) 2 / κ ∈ [ 1/ κ , 4/ κ ] give
∥ x ^ 1 ∥ ≤ 1 , x ^ 1 ⊤ D x ^ 1 ≤ 1 , ∥ D x ^ 1 ∥ 2 ≤ 2 , \lVert\hat x_1\rVert\leq1,\qquad\hat x_1^\top D\hat x_1\leq1,\qquad\lVert D\hat x_1\rVert^2\leq2, ∥ x ^ 1 ∥ ≤ 1 , x ^ 1 ⊤ D x ^ 1 ≤ 1 , ∥ D x ^ 1 ∥ 2 ≤ 2 ,
d ⊤ D d = 3 + r 2 + ( 1 + r ) / κ ≤ 2 , ∥ D d ∥ 2 ≤ κ + 1 , ∣ ⟨ d , x ^ 1 ⟩ ∣ ≤ 1 5 . d^\top Dd=\frac{3+r}{2+(1+r)/\kappa}\leq2,\qquad\lVert Dd\rVert^2\leq\kappa+1,\qquad|\langle d,\hat x_1\rangle|\leq\frac{1}{5}. d ⊤ D d = 2 + ( 1 + r ) / κ 3 + r ≤ 2 , ∥ D d ∥ 2 ≤ κ + 1 , ∣ ⟨ d , x ^ 1 ⟩ ∣ ≤ 5 1 .
Since ⟨ D x ^ 1 , d ⟩ = 0 \langle D\hat x_1,d\rangle=0 ⟨ D x ^ 1 , d ⟩ = 0 by the tail identities,
⟨ w 0 , x 0 ⟩ = 64 ( λ − d ⊤ D d ) − 8 λ ⟨ d , x ^ 1 ⟩ − x ^ 1 ⊤ D x ^ 1 ≥ 62 κ − 5 , \langle w_0,x_0\rangle=64(\lambda-d^\top Dd)-8\lambda\langle d,\hat x_1\rangle-\hat x_1^\top D\hat x_1\geq62\sqrt{\kappa}-5, ⟨ w 0 , x 0 ⟩ = 64 ( λ − d ⊤ D d ) − 8 λ ⟨ d , x ^ 1 ⟩ − x ^ 1 ⊤ D x ^ 1 ≥ 62 κ − 5 ,
∥ w 0 ∥ ≤ 8 λ + ∥ D x ^ 1 ∥ + 8 ∥ D d ∥ ≤ 16 κ + 18 , \lVert w_0\rVert\leq8\lambda+\lVert D\hat x_1\rVert+8\lVert Dd\rVert\leq16\sqrt{\kappa}+18, ∥ w 0 ∥ ≤ 8 λ + ∥ D x ^ 1 ∥ + 8 ∥ D d ∥ ≤ 16 κ + 18 ,
so that u 0 = ⟨ w 0 , x 0 ⟩ / ∥ w 0 ∥ ≥ 3.4 u_0=\langle w_0,x_0\rangle/\lVert w_0\rVert\geq3.4 u 0 = ⟨ w 0 , x 0 ⟩ / ∥ w 0 ∥ ≥ 3.4 . Let τ = ( u 0 − 1 ) / 2 ≥ 5 2 \tau=(u_0-1)/2\geq\frac{5}{2} τ = ( u 0 − 1 ) /2 ≥ 2 5 , let ψ ( u ) = u + 3 \psi(u)=u_+^3 ψ ( u ) = u + 3 for u ≤ τ / 2 u\leq\tau/2 u ≤ τ /2 , continued by its second-order Taylor polynomial at τ / 2 \tau/2 τ /2 , and
ϕ ( x ) = ∥ w 0 ∥ ψ ′ ( τ ) ψ ( ⟨ w 0 , x ⟩ ∥ w 0 ∥ − u 0 + τ ) . \phi(x)=\frac{\lVert w_0\rVert}{\psi'(\tau)}\psi\left(\frac{\langle w_0,x\rangle}{\lVert w_0\rVert}-u_0+\tau\right). ϕ ( x ) = ψ ′ ( τ ) ∥ w 0 ∥ ψ ( ∥ w 0 ∥ ⟨ w 0 , x ⟩ − u 0 + τ ) .
Then ϕ \phi ϕ is convex, vanishes where ⟨ w 0 , x ⟩ / ∥ w 0 ∥ < ( u 0 + 1 ) / 2 \langle w_0,x\rangle/\lVert w_0\rVert< (u_0+1)/2 ⟨ w 0 , x ⟩ / ∥ w 0 ∥ < ( u 0 + 1 ) /2 , so near the unit ball, and ∇ ϕ ( x 0 ) = w 0 \nabla\phi(x_0)=w_0 ∇ ϕ ( x 0 ) = w 0 , so ∇ f ( x 0 ) = − 8 λ d \nabla f(x_0)=-8\lambda d ∇ f ( x 0 ) = − 8 λ d . Since 0 ≤ ψ ′ ′ ≤ 3 τ 0\leq\psi^{\prime\prime}\leq3\tau 0 ≤ ψ ′′ ≤ 3 τ , ∣ ψ ′ ′ ′ ∣ ≤ 6 |\psi^{\prime\prime\prime}|\leq6 ∣ ψ ′′′ ∣ ≤ 6 and ψ ′ ( τ ) = 9 4 τ 2 \psi^\prime(\tau)=\frac{9}{4}\tau^2 ψ ′ ( τ ) = 4 9 τ 2 , we have ∥ ∇ 2 ϕ ∥ ≤ 4 ∥ w 0 ∥ / ( 3 τ ) ≤ 20 κ \lVert\nabla^2\phi\rVert\leq4\lVert w_0\rVert/(3\tau)\leq20\sqrt{\kappa} ∥ ∇ 2 ϕ ∥ ≤ 4 ∥ w 0 ∥ / ( 3 τ ) ≤ 20 κ , and ∇ 2 ϕ \nabla^2\phi ∇ 2 ϕ is Lipschitz with constant 8 ∥ w 0 ∥ / ( 3 τ 2 ) ≤ 34 κ 8\lVert w_0\rVert/(3\tau^2)\leq34\sqrt{\kappa} 8 ∥ w 0 ∥ / ( 3 τ 2 ) ≤ 34 κ . Near x 0 x_0 x 0 the function ϕ \phi ϕ is quadratic, because τ − 1 2 ≥ τ 2 \tau-\frac{1}{2}\geq\frac{\tau}{2} τ − 2 1 ≥ 2 τ , so its convolution with an even mollifier supported in the ball of radius 1 2 \frac{1}{2} 2 1 keeps all these properties, vanishes on the open ball of radius u 0 / 2 > 1 u_0/2>1 u 0 /2 > 1 and is C ∞ C^\infty C ∞ . We replace ϕ \phi ϕ by this convolution.
The trajectory. The first step is the exact line search along d d d . The point x ^ 1 = x 0 + 8 d \hat{x}_1=x_0+8d x ^ 1 = x 0 + 8 d minimizes q q q on this line, because ⟨ ∇ q ( x ^ 1 ) , d ⟩ = 0 \langle\nabla q(\hat{x}_1),d\rangle=0 ⟨ ∇ q ( x ^ 1 ) , d ⟩ = 0 , and f ≥ q f\geq q f ≥ q with equality at x ^ 1 \hat{x}_1 x ^ 1 , so x 1 = x ^ 1 x_1=\hat{x}_1 x 1 = x ^ 1 . Then s 1 s_1 s 1 is parallel to s ^ 1 \hat{s}_1 s ^ 1 , and inductively every x k + 1 = x ^ k + 1 x_{k+1}=\hat{x}_{k+1} x k + 1 = x ^ k + 1 minimizes q q q , and hence f f f , over x k + span { g ^ k , s ^ k } x_k+\operatorname{span}\{\hat{g}_k,\hat{s}_k\} x k + span { g ^ k , s ^ k } , where ∇ f ( x k ) = g ^ k \nabla f(x_k)=\hat{g}_k ∇ f ( x k ) = g ^ k because ϕ \phi ϕ vanishes near x ^ k \hat{x}_k x ^ k . Finally f ⋆ = 0 f^\star=0 f ⋆ = 0 and − log r ≤ ( 1 − r ) / r ≤ 4 / ( κ − 4 ) -\log r\leq(1-r)/r\leq4/(\kappa-4) − log r ≤ ( 1 − r ) / r ≤ 4/ ( κ − 4 ) , so lim ε ↓ 0 N ( ε ) / log ( 1 / ε ) = 1 / ( − 2 log r ) ≥ ( κ − 4 ) / 8 \lim_{\varepsilon\downarrow0}N(\varepsilon)/\log(1/\varepsilon)=1/(-2\log r)\geq(\kappa-4)/8 lim ε ↓ 0 N ( ε ) / log ( 1/ ε ) = 1/ ( − 2 log r ) ≥ ( κ − 4 ) /8 . □
For SESOP, three steps bring the state near the fixed point of Lemma 4.12 .
Proposition 4.14 (smooth instances for SESOP) Let SESOP use the previous step, the anchor and one aggregate with slowly forgetting weights such that w ( 0 ) ≥ 0 w(0)\geq0 w ( 0 ) ≥ 0 and w ( 1 ) , w ( 2 ) > 0 w(1),w(2)>0 w ( 1 ) , w ( 2 ) > 0 . There is μ w > 0 \mu_w>0 μ w > 0 depending only on w ( 0 ) , w ( 1 ) , w ( 2 ) w(0),w(1),w(2) w ( 0 ) , w ( 1 ) , w ( 2 ) such that for every K ≥ 12 K\geq12 K ≥ 12 there are G ∈ C ∞ ( R 5 ) G\in C^\infty(\mathbb{R}^5) G ∈ C ∞ ( R 5 ) and a starting point with the following properties. The Hessian satisfies μ w I ⪯ ∇ 2 G ⪯ 2 K I \mu_w I\preceq\nabla^2G\preceq2KI μ w I ⪯ ∇ 2 G ⪯ 2 K I , all derivatives of order at least two are bounded, and G ( x ) = 1 2 x T H x G(x)=\frac{1}{2}x^\mathsf{T}Hx G ( x ) = 2 1 x T H x near the minimizer 0 0 0 with H = I + ( K − 1 ) e e T H=I+(K-1)ee^\mathsf{T} H = I + ( K − 1 ) e e T , ∥ e ∥ = 1 \lVert e\rVert=1 ∥ e ∥ = 1 . From x 3 x_3 x 3 on SESOP stays in this region, and G ( x k + 1 ) ≥ K − 1 K + 127 G ( x k ) G(x_{k+1})\geq\frac{K-1}{K+127}G(x_k) G ( x k + 1 ) ≥ K + 127 K − 1 G ( x k ) for k ≥ 3 k\geq3 k ≥ 3 .
Proof. Near the minimizer we reduce SESOP to the plane step of Lemma 4.12 and show that the slow regime persists (Steps 1 and 2). Then three explicit SESOP steps on a smooth function reach this regime (Steps 3 to 5).
Step 1: SESOP near the minimizer. From now on SESOP has one aggregate σ k = ∑ i ≤ k w ( i ) g i \sigma_k=\sum_{i\leq k}w(i)g_i σ k = ∑ i ≤ k w ( i ) g i , so S k = span { g k , s k , x k − x 0 , σ k − 1 } S_k=\operatorname{span}\{g_k,s_k,x_k-x_0,\sigma_{k-1}\} S k = span { g k , s k , x k − x 0 , σ k − 1 } and g k ⊥ s k , x k − x 0 , σ k − 1 g_k\perp s_k,x_k-x_0,\sigma_{k-1} g k ⊥ s k , x k − x 0 , σ k − 1 . Let H = I + a e e T H=I+ae e^\mathsf{T} H = I + a e e T on R 5 \mathbb{R}^5 R 5 with a = K − 1 a=K-1 a = K − 1 , and f ( x ) = 1 2 x T H x = 1 2 ∥ y ∥ 2 f(x)=\frac{1}{2}x^\mathsf{T}Hx=\frac{1}{2}\lVert y\rVert^2 f ( x ) = 2 1 x T H x = 2 1 ∥ y ∥ 2 in energy coordinates y = H 1 / 2 x y=H^{1/2}x y = H 1/2 x . Put
C k = [ y k − H 1 / 2 x 0 , H 1 / 2 σ k − 1 ] , W k = ( range C k ) ⊥ , C_k=\left[y_k-H^{1/2}x_0,\ H^{1/2}\sigma_{k-1}\right],\qquad W_k=(\operatorname{range}C_k)^\perp, C k = [ y k − H 1/2 x 0 , H 1/2 σ k − 1 ] , W k = ( range C k ) ⊥ ,
P k = P W k , λ k = ∥ P k e ∥ 2 , t k = P k ( y k − y k − 1 ) . P_k=P_{W_k},\qquad\lambda_k=\lVert P_ke\rVert^2,\qquad t_k=P_k(y_k-y_{k-1}). P k = P W k , λ k = ∥ P k e ∥ 2 , t k = P k ( y k − y k − 1 ) .
While the objective equals f f f , the orthogonality relations give y k ∈ W k y_k\in W_k y k ∈ W k and y k ⊥ t k y_k\perp t_k y k ⊥ t k . The search directions in energy coordinates are H y k Hy_k H y k , y k − y k − 1 y_k-y_{k-1} y k − y k − 1 and the columns of C k C_k C k , so y k + 1 y_{k+1} y k + 1 minimizes 1 2 ∥ y ∥ 2 \frac{1}{2}\lVert y\rVert^2 2 1 ∥ y ∥ 2 over y k + span { B k y k , t k } y_k+\operatorname{span}\{B_ky_k,t_k\} y k + span { B k y k , t k } with B k = P k H ∣ W k = I + a ( P k e ) ( P k e ) T B_k=P_kH|_{W_k}=I+a(P_ke)(P_ke)^\mathsf{T} B k = P k H ∣ W k = I + a ( P k e ) ( P k e ) T . If C k C_k C k has rank two, then dim W k = 3 \dim W_k=3 dim W k = 3 , and in the coordinates B k − 1 / 2 y B_k^{-1/2}y B k − 1/2 y this is the plane step of Lemma 4.12 with b = a λ k b=a\lambda_k b = a λ k , v = P k e / ∥ P k e ∥ v=P_ke/\lVert P_ke\rVert v = P k e / ∥ P k e ∥ , gradient B k 1 / 2 y k B_k^{1/2}y_k B k 1/2 y k and inherited step B k − 1 / 2 t k B_k^{-1/2}t_k B k − 1/2 t k , whose quantities are
γ = ( 1 + b ) ⟨ v , y ⟩ 2 ∥ y ∥ 2 + b ⟨ v , y ⟩ 2 , ζ = ⟨ v , t ⟩ 2 ( 1 + b ) ∥ t ∥ 2 − b ⟨ v , t ⟩ 2 , z = 1 − γ − ζ . \gamma=\frac{(1+b)\langle v,y\rangle^2}{\lVert y\rVert^2+b\langle v,y\rangle^2},\qquad
\zeta=\frac{\langle v,t\rangle^2}{(1+b)\lVert t\rVert^2-b\langle v,t\rangle^2},\qquad
z=1-\gamma-\zeta. γ = ∥ y ∥ 2 + b ⟨ v , y ⟩ 2 ( 1 + b ) ⟨ v , y ⟩ 2 , ζ = ( 1 + b ) ∥ t ∥ 2 − b ⟨ v , t ⟩ 2 ⟨ v , t ⟩ 2 , z = 1 − γ − ζ .
Let J k = J ( P k , y k , t k ) J_k=J(P_k,y_k,t_k) J k = J ( P k , y k , t k ) . The step conserves J ( P k , ⋅ , ⋅ ) J(P_k,\cdot,\cdot) J ( P k , ⋅ , ⋅ ) , and J k + 1 J_{k+1} J k + 1 differs from it only through the change of the projector. With a matrix M M M in place of P k P_k P k , b = a ∥ M e ∥ 2 b=a\lVert Me\rVert^2 b = a ∥ M e ∥ 2 , v = M e / ∥ M e ∥ v=Me/\lVert Me\rVert v = M e / ∥ M e ∥ and M t Mt M t in place of t t t , the formulas are smooth and homogeneous of degree zero in y y y and t t t where ∥ M e ∥ ≥ 1 4 \lVert Me\rVert\geq\frac{1}{4} ∥ M e ∥ ≥ 4 1 and ∥ M t ∥ ≥ 3 4 ∥ t ∥ \lVert Mt\rVert\geq\frac{3}{4}\lVert t\rVert ∥ M t ∥ ≥ 4 3 ∥ t ∥ . These conditions hold for t ∈ range P t\in\operatorname{range}P t ∈ range P on the segment between projectors P P P , P ′ P' P ′ with ∥ P ′ − P ∥ ≤ 1 4 \lVert P'-P\rVert\le\frac{1}{4} ∥ P ′ − P ∥ ≤ 4 1 and ∥ P e ∥ 2 ≥ 1 4 \lVert Pe\rVert^2 \ge\frac{1}{4} ∥ P e ∥ 2 ≥ 4 1 , so by compactness there is L K L_K L K , depending only on K K K , such that
∣ J ( P ′ , y , P ′ t ) − J ( P , y , t ) ∣ ≤ L K ∥ P ′ − P ∥ (12) \lvert J(P',y,P't)-J(P,y,t)\rvert\le L_K\lVert P'-P\rVert \tag*{(12)} ∣ J ( P ′ , y , P ′ t ) − J ( P , y , t )∣ ≤ L K ∥ P ′ − P ∥ (12)
for such P P P , P ′ P' P ′ , y ∈ range P ∩ range P ′ y \in\operatorname{range} P\cap\operatorname{range} P' y ∈ range P ∩ range P ′ and t ∈ range P t\in\operatorname{range} P t ∈ range P with t ⊥ y t\perp y t ⊥ y .
Step 2: persistence of the slow regime. Let q K = 1 − 1 / K q_K=\sqrt{1-1/K} q K = 1 − 1/ K and E = ∑ j ≥ 0 ( 2 + K ∣ w ( 3 + j ) ∣ ) q K j E=\sum_{j\ge0}(2+K\lvert w(3+j)\rvert)q_K^j E = ∑ j ≥ 0 ( 2 + K ∣ w ( 3 + j )∣) q K j , which is finite for slowly forgetting weights. Let the objective be convex and equal to f f f near the ball { ∥ H 1 / 2 x ∥ ≤ R } \{\lVert H^{1/2}x\rVert\le R\} {∥ H 1/2 x ∥ ≤ R } , where R = ∥ y 3 ∥ R=\lVert y_3\rVert R = ∥ y 3 ∥ , let g 3 ⊥ s 3 g_3\perp s_3 g 3 ⊥ s 3 , x 3 − x 0 x_3-x_0 x 3 − x 0 , σ 2 \sigma_2 σ 2 and t 3 ≠ 0 t_3\ne0 t 3 = 0 , and assume that for some m > 0 m>0 m > 0
σ min ( C 3 ) > m , λ 3 > 3 4 , J 3 > 1 32 , R E < min { m / 8 , m / ( 128 L K ) } . (13) \sigma_{\min}(C_3)>m,\qquad\lambda_3>\frac{3}{4},\qquad J_3>\frac{1}{32},\qquad RE<\min\{m/8,m/(128L_K)\}. \tag*{(13)} σ m i n ( C 3 ) > m , λ 3 > 4 3 , J 3 > 32 1 , R E < min { m /8 , m / ( 128 L K )} . (13)
We show that then SESOP never terminates and f ( x k + 1 ) ≥ K − 1 K + 127 f ( x k ) f(x_{k+1})\ge\frac{K-1}{K+127}f(x_k) f ( x k + 1 ) ≥ K + 127 K − 1 f ( x k ) for k ≥ 3 k\ge3 k ≥ 3 . The minimizer of f f f over x k + S k x_k+S_k x k + S k is not worse than the gradient step x k − g k / K x_k-g_k/K x k − g k / K , so its energy norm is at most q K ∥ y k ∥ q_K\lVert y_k\rVert q K ∥ y k ∥ . Hence it lies where the objective equals f f f , the gradient of the objective there is orthogonal to S k S_k S k , and by convexity it is the SESOP iterate. By induction ∥ y 3 + j ∥ ≤ q K j R \lVert y_{3+j}\rVert\le q_K^jR ∥ y 3 + j ∥ ≤ q K j R . The columns of C k C_k C k change by y k + 1 − y k y_{k+1}-y_k y k + 1 − y k and w ( k ) H y k w(k)Hy_k w ( k ) H y k , so ∑ k ≥ 3 ∥ C k + 1 − C k ∥ ≤ R E < m / 8 \sum_{k\ge3}\lVert C_{k+1}-C_k\rVert\le RE<m/8 ∑ k ≥ 3 ∥ C k + 1 − C k ∥ ≤ R E < m /8 , and σ min ( C k ) > m / 2 \sigma_{\min}(C_k)>m/2 σ m i n ( C k ) > m /2 for all k k k . The identity ( I − P range C ′ ) P range C = ( I − P range C ′ ) ( C − C ′ ) C + (I-P_{\operatorname{range} C'} )P_{\operatorname{range} C}=(I-P_{\operatorname{range} C'})(C-C')C^+ ( I − P range C ′ ) P range C = ( I − P range C ′ ) ( C − C ′ ) C + gives ∥ P k + 1 − P k ∥ ≤ 2 ∥ C k + 1 − C k ∥ / m \lVert P_{k+1}-P_k\rVert\le2\lVert C_{k+1}-C_k\rVert/m ∥ P k + 1 − P k ∥ ≤ 2 ∥ C k + 1 − C k ∥ / m , hence ∑ k ≥ 3 ∥ P k + 1 − P k ∥ < 1 4 \sum_{k\ge3}\lVert P_{k+1}-P_k\rVert<\frac{1}{4} ∑ k ≥ 3 ∥ P k + 1 − P k ∥ < 4 1 and λ k > λ 3 − 1 4 > 1 2 \lambda_k>\lambda_3-\frac{1}{4}>\frac{1}{2} λ k > λ 3 − 4 1 > 2 1 . By Lemma 4.12 and (11) the invariant changes by less than L K ⋅ 2 m R E < 1 64 L_K\cdot\frac{2}{m}RE<\frac{1}{64} L K ⋅ m 2 R E < 64 1 in total, so J k > 1 64 J_k>\frac{1}{64} J k > 64 1 . A positive invariant excludes a vanishing next point, and a projector change below 1 4 \frac{1}{4} 4 1 excludes a vanishing projected step, which closes the induction. Finally b = a λ k ≥ a / 2 b=a\lambda_k\ge a/2 b = a λ k ≥ a /2 and Lemma 4.12 give
f ( x k + 1 ) f ( x k ) ≥ b J k 1 + b J k ≥ a / 128 1 + a / 128 = K − 1 K + 127 . \frac{f(x_{k+1})}{f(x_k)}\ge\frac{bJ_k}{1+bJ_k}\ge\frac{a/128}{1+a/128}=\frac{K-1}{K+127}. f ( x k ) f ( x k + 1 ) ≥ 1 + b J k b J k ≥ 1 + a /128 a /128 = K + 127 K − 1 .
Step 3: the quadratic. Write w i = w ( i ) w_i=w(i) w i = w ( i ) , c = ( w 0 + w 2 ) / w 1 c=(w_0+w_2)/w_1 c = ( w 0 + w 2 ) / w 1 , m 0 = min { 1 , c } m_0=\min\{1,c\} m 0 = min { 1 , c } and M c = max { 1 , c } M_c=\max\{1,c\} M c = max { 1 , c } , and put ϵ g = m 0 / ( 100 M c 2 ) \epsilon_g=m_0/(100M_c^2) ϵ g = m 0 / ( 100 M c 2 ) , ρ w = ϵ g m 0 / 100 \rho_w=\epsilon_gm_0/100 ρ w = ϵ g m 0 /100 and μ w = ρ w / 100 \mu_w=\rho_w/100 μ w = ρ w /100 . Let e 0 , e 1 , e 2 , v 1 , v 2 e_0,e_1,e_2,v_1,v_2 e 0 , e 1 , e 2 , v 1 , v 2 be an orthonormal basis of R 5 \mathbb{R}^5 R 5 , d ∗ = ( − e 0 + e 1 − e 2 ) / 3 d^*=(-e_0+e_1-e_2)/\sqrt{3} d ∗ = ( − e 0 + e 1 − e 2 ) / 3 , d = 2 d=2 d = 2 if w 0 ≠ 2 w 2 w_0\ne2w_2 w 0 = 2 w 2 and d = 3 d=3 d = 3 otherwise, and U = ( d e 0 + ( d + 1 ) e 1 + e 2 ) / 3 U=(de_0+(d+1)e_1+e_2)/\sqrt{3} U = ( d e 0 + ( d + 1 ) e 1 + e 2 ) / 3 . With the fixed point ( γ ∗ , ζ ∗ ) (\gamma^*,\zeta^*) ( γ ∗ , ζ ∗ ) of Lemma 4.12 for b = a b=a b = a let e = γ ∗ v 1 + ζ ∗ v 2 e=\gamma^*v_1+\zeta^*v_2 e = γ ∗ v 1 + ζ ∗ v 2 and H = I + a e e ⊤ H=I+aee^\top H = I + a e e ⊤ . The vectors U U U and V = w 0 e 0 + ( w 0 + w 2 ) e 1 + w 2 e 2 V=w_0e_0+(w_0+w_2)e_1+w_2e_2 V = w 0 e 0 + ( w 0 + w 2 ) e 1 + w 2 e 2 are orthogonal to d ∗ d^* d ∗ and e e e , and independent since d ≠ w 0 / w 2 d\ne w_0/w_2 d = w 0 / w 2 .
Step 4: three SESOP steps. For small θ > 0 \theta>0 θ > 0 let n = U / ∥ U ∥ n=U/\lVert U\rVert n = U / ∥ U ∥ , D ∗ = ⟨ v 1 , H − 1 v 1 ⟩ D^*=\langle v_1,H^{-1}v_1\rangle D ∗ = ⟨ v 1 , H − 1 v 1 ⟩ and
ν = 1 2 ( ∥ U ∥ − ∥ U ∥ 2 − 4 θ 2 D ∗ ) , x 3 = θ H − 1 v 1 + ν n , g 3 = H x 3 = θ v 1 + ν n , \nu=\frac{1}{2}\left(\lVert U\rVert-\sqrt{\lVert U\rVert^2-4\theta^2D^*}\right),\qquad x_3=\theta H^{-1}v_1+\nu n,\qquad g_3=Hx_3=\theta v_1+\nu n, ν = 2 1 ( ∥ U ∥ − ∥ U ∥ 2 − 4 θ 2 D ∗ ) , x 3 = θ H − 1 v 1 + ν n , g 3 = H x 3 = θ v 1 + ν n ,
so that ⟨ g 3 , d ∗ ⟩ = 0 \langle g_3,d^*\rangle=0 ⟨ g 3 , d ∗ ⟩ = 0 and ⟨ g 3 , x 3 − U ⟩ = θ 2 D ∗ + ν 2 − ν ∥ U ∥ = 0 \langle g_3,x_3-U\rangle=\theta^2D^*+\nu^2-\nu\lVert U\rVert=0 ⟨ g 3 , x 3 − U ⟩ = θ 2 D ∗ + ν 2 − ν ∥ U ∥ = 0 . Put p = x 3 − d ∗ − U p=x_3-d^*-U p = x 3 − d ∗ − U , let t θ t_\theta t θ be the projection of e 2 e_2 e 2 onto span { g 3 , p , d ∗ } ⊥ \operatorname{span}\{g_3,p,d^*\}^{\perp} span { g 3 , p , d ∗ } ⊥ , E θ = span { p , d ∗ , t θ } E_\theta=\operatorname{span}\{p,d^*,t_\theta\} E θ = span { p , d ∗ , t θ } , and
g 2 = P E θ ∩ p ⊥ ( ϵ g e 2 ) , g 0 = ϵ g P E θ ∩ g 2 ⊥ e 0 ∥ P E θ ∩ g 2 ⊥ e 0 ∥ , g 1 = ϵ g c P E θ ∩ { g 0 , g 2 } ⊥ e 1 ∥ P E θ ∩ { g 0 , g 2 } ⊥ e 1 ∥ , g_2=P_{E_\theta\cap p^\perp}(\epsilon_g e_2),\qquad
g_0=\epsilon_g\frac{P_{E_\theta\cap g_2^\perp}e_0}{\lVert P_{E_\theta\cap g_2^\perp}e_0\rVert},\qquad
g_1=\epsilon_gc\frac{P_{E_\theta\cap\{g_0,g_2\}^\perp}e_1}{\lVert P_{E_\theta\cap\{g_0,g_2\}^\perp}e_1\rVert}, g 2 = P E θ ∩ p ⊥ ( ϵ g e 2 ) , g 0 = ϵ g ∥ P E θ ∩ g 2 ⊥ e 0 ∥ P E θ ∩ g 2 ⊥ e 0 , g 1 = ϵ g c ∥ P E θ ∩ { g 0 , g 2 } ⊥ e 1 ∥ P E θ ∩ { g 0 , g 2 } ⊥ e 1 ,
and x 0 = U x_0=U x 0 = U , x 1 = U − d − 1 3 ϵ g g 0 x_1=U-\frac{d-1}{\sqrt{3}\epsilon_g}g_0 x 1 = U − 3 ϵ g d − 1 g 0 , x 2 = x 3 − d ∗ x_2=x_3-d^* x 2 = x 3 − d ∗ . As θ ↓ 0 \theta\downarrow0 θ ↓ 0 , x 3 → 0 x_3\to0 x 3 → 0 , E θ → span { e 0 , e 1 , e 2 } E_\theta\to\operatorname{span}\{e_0,e_1,e_2\} E θ → span { e 0 , e 1 , e 2 } , ( g 0 , g 1 , g 2 ) → ϵ g ( e 0 , c e 1 , e 2 ) (g_0,g_1,g_2)\to\epsilon_g(e_0,ce_1,e_2) ( g 0 , g 1 , g 2 ) → ϵ g ( e 0 , c e 1 , e 2 ) , x 1 → X 1 = ( e 0 + ( d + 1 ) e 1 + e 2 ) / 3 x_1\to X_1=(e_0+(d+1)e_1+e_2)/\sqrt{3} x 1 → X 1 = ( e 0 + ( d + 1 ) e 1 + e 2 ) / 3 and x 2 → X 2 = − d ∗ x_2\to X_2=-d^* x 2 → X 2 = − d ∗ , and for small θ \theta θ the vectors g 0 , g 1 g_0,g_1 g 0 , g 1 are independent and so are p , x 2 − x 1 , g 2 p,x_2-x_1,g_2 p , x 2 − x 1 , g 2 . Let a strongly convex function have the gradients g i g_i g i at x i x_i x i for i ≤ 2 i\le2 i ≤ 2 and equal f f f near x 3 x_3 x 3 . In each of the first three SESOP steps the new point lies in the search space and its gradient is orthogonal to it:
S 0 = span { g 0 } , g 1 ⊥ g 0 , S_0=\operatorname{span}\{g_0\},\qquad g_1\perp g_0, S 0 = span { g 0 } , g 1 ⊥ g 0 ,
S 1 = span { g 0 , g 1 } ∋ x 2 − x 1 = p + d − 1 3 ϵ g g 0 , g 2 ⊥ g 0 , g 1 , (14) S_1=\operatorname{span}\{g_0,g_1\}\ni x_2-x_1=p+\frac{d-1}{\sqrt{3}\epsilon_g}g_0,\qquad g_2\perp g_0,g_1, \tag*{(14)} S 1 = span { g 0 , g 1 } ∋ x 2 − x 1 = p + 3 ϵ g d − 1 g 0 , g 2 ⊥ g 0 , g 1 , (14)
S 2 = span { g 2 , x 2 − x 1 , p , σ 1 } = E θ ∋ d ∗ , g 3 ⊥ d ∗ , p , t θ . S_{2}=\operatorname{span}\{g_{2},x_{2}-x_{1},p,\sigma_{1}\}=E_{\theta}\ni d^{*}, \qquad g_{3}\perp d^{*},p,t_{\theta}. S 2 = span { g 2 , x 2 − x 1 , p , σ 1 } = E θ ∋ d ∗ , g 3 ⊥ d ∗ , p , t θ .
Hence SESOP generates x 1 , x 2 , x 3 x_{1},x_{2},x_{3} x 1 , x 2 , x 3 . As θ ↓ 0 \theta\downarrow0 θ ↓ 0 , C 3 → [ − U , ϵ g V ] C_{3}\to[-U,\epsilon_{g}V] C 3 → [ − U , ϵ g V ] , whose range is orthogonal to e e e , so λ 3 → 1 \lambda_{3}\to1 λ 3 → 1 . The compressed gradient and inherited step at k = 3 k=3 k = 3 become parallel to v 1 v_{1} v 1 and d ∗ d^{*} d ∗ , whose coordinates are ( γ ∗ , ζ ∗ ) (\gamma^{*},\zeta^{*}) ( γ ∗ , ζ ∗ ) , so J 3 → r / ( 3 + r ) 32 J_{3}\to r/(3+r)^{32} J 3 → r / ( 3 + r ) 32 , and R = ∥ y 3 ∥ → 0 R=\lVert y_{3}\rVert\to0 R = ∥ y 3 ∥ → 0 . With m = 1 4 σ min [ − U , ϵ g V ] m=\frac{1}{4}\sigma_{\min}[-U,\epsilon_{g}V] m = 4 1 σ m i n [ − U , ϵ g V ] all conditions of Step 2 hold for small θ \theta θ , provided that the objective equals f f f on a ball of fixed radius around 0 0 0 .
Step 5: the function. Let L = 2 K L=2K L = 2 K , Λ = L − μ w \Lambda=L-\mu_{w} Λ = L − μ w , ( F 0 , F 1 , F 2 ) = ( 3 , 2 , 1 ) ρ w (F_{0},F_{1},F_{2})=(3,2,1)\rho_{w} ( F 0 , F 1 , F 2 ) = ( 3 , 2 , 1 ) ρ w , X 0 = U X_{0}=U X 0 = U and G i = ϵ g ( e 0 , c e 1 , e 2 ) i G_{i}=\epsilon_{g}(e_{0},ce_{1},e_{2})_{i} G i = ϵ g ( e 0 , c e 1 , e 2 ) i . With q i = g i − μ w x i q_{i}=g_{i}-\mu_{w}x_{i} q i = g i − μ w x i and u i = ⟨ x i , q i ⟩ − F i + μ w 2 ∥ x i ∥ 2 u_{i}=\langle x_{i},q_{i}\rangle-F_{i}+\frac{\mu_{w}}{2}\lVert x_{i}\rVert^{2} u i = ⟨ x i , q i ⟩ − F i + 2 μ w ∥ x i ∥ 2 put
ψ ( ξ ) = max { 1 2 ξ T ( H − μ w I ) − 1 ξ , max i ≤ 2 [ u i + ⟨ x i , ξ − q i ⟩ + ∥ ξ − q i ∥ 2 2 Λ ] } . \psi(\xi)=\max\left\{\frac{1}{2}\xi^{\mathsf T}(H-\mu_{w}I)^{-1}\xi,\max_{i\leq2}\left[u_{i}+\langle x_{i},\xi-q_{i}\rangle+\frac{\lVert\xi-q_{i}\rVert^{2}}{2\Lambda}\right]\right\}. ψ ( ξ ) = max { 2 1 ξ T ( H − μ w I ) − 1 ξ , i ≤ 2 max [ u i + ⟨ x i , ξ − q i ⟩ + 2Λ ∥ ξ − q i ∥ 2 ] } .
and F ( x ) = ψ ∗ ( x ) + μ w 2 ∥ x ∥ 2 F(x)=\psi^{*}(x)+\frac{\mu_{w}}{2}\lVert x\rVert^{2} F ( x ) = ψ ∗ ( x ) + 2 μ w ∥ x ∥ 2 . The limit data satisfy ∥ X i ∥ ≤ 3 \lVert X_{i}\rVert\leq3 ∥ X i ∥ ≤ 3 , ∥ G i ∥ 2 ≤ ρ w \lVert G_{i}\rVert^{2}\leq\rho_{w} ∥ G i ∥ 2 ≤ ρ w , ⟨ X i , G i ⟩ ≥ ϵ g m 0 / 3 \langle X_{i},G_{i}\rangle\geq\epsilon_{g}m_{0}/\sqrt{3} ⟨ X i , G i ⟩ ≥ ϵ g m 0 / 3 and ⟨ G j , X i − X j ⟩ ≤ 0 \langle G_{j},X_{i}-X_{j}\rangle\leq0 ⟨ G j , X i − X j ⟩ ≤ 0 for i ≠ j i\ne j i = j . One can check directly that the residuals (4) with the constants μ w \mu_{w} μ w and L L L are at least ρ w / 2 \rho_{w}/2 ρ w /2 , that
⟨ X i , G i ⟩ − F i − μ w 2 ∥ X i ∥ 2 − ∥ G i − μ w X i ∥ 2 2 ( 1 − μ w ) > ρ w , \langle X_{i},G_{i}\rangle-F_{i}-\frac{\mu_{w}}{2}\lVert X_{i}\rVert^{2}-\frac{\lVert G_{i}-\mu_{w}X_{i}\rVert^{2}}{2(1-\mu_{w})}>\rho_{w}, ⟨ X i , G i ⟩ − F i − 2 μ w ∥ X i ∥ 2 − 2 ( 1 − μ w ) ∥ G i − μ w X i ∥ 2 > ρ w ,
− F i + μ w 2 ∥ X i ∥ 2 + ∥ G i − μ w X i ∥ 2 2 Λ < − ρ w 2 , -F_{i}+\frac{\mu_{w}}{2}\lVert X_{i}\rVert^{2}+\frac{\lVert G_{i}-\mu_{w}X_{i}\rVert^{2}}{2\Lambda}<-\frac{\rho_{w}}{2}, − F i + 2 μ w ∥ X i ∥ 2 + 2Λ ∥ G i − μ w X i ∥ 2 < − 2 ρ w ,
and the same holds with halved margins for the actual data and small θ \theta θ . Every branch of ψ \psi ψ is Λ − 1 \Lambda^{-1} Λ − 1 -strongly convex, so F F F is μ w \mu_{w} μ w -strongly convex with an L L L -Lipschitz gradient, as in Lemma 3.4. By the margins, and since the gradients of the branches are bounded, only the i i i th data branch is active near q i q_{i} q i and only the quadratic branch near 0 0 0 , so there is an absolute constant c 1 > 0 c_{1}>0 c 1 > 0 such that F ( x i + h ) = F i + ⟨ g i , h ⟩ + L 2 ∥ h ∥ 2 F(x_{i}+h)=F_{i}+\langle g_{i},h\rangle+\frac{L}{2}\lVert h\rVert^{2} F ( x i + h ) = F i + ⟨ g i , h ⟩ + 2 L ∥ h ∥ 2 for ∥ h ∥ < c 1 ρ w / Λ \lVert h\rVert<c_{1}\rho_{w}/\Lambda ∥ h ∥ < c 1 ρ w /Λ and F = f F=f F = f for ∥ x ∥ < c 1 ρ w / K \lVert x\rVert<c_{1}\rho_{w}/K ∥ x ∥ < c 1 ρ w / K . With an even mollifier η \eta η supported in the unit ball and ϵ s = c 1 ρ w / ( 4 Λ ) \epsilon_{s}=c_{1}\rho_{w}/(4\Lambda) ϵ s = c 1 ρ w / ( 4Λ ) let G = F ∗ η ϵ s − 1 2 ∫ ξ T H ξ η ϵ s ( ξ ) d ξ G=F*\eta_{\epsilon_{s}}-\frac{1}{2}\int\xi^{\mathsf T}H\xi\,\eta_{\epsilon_{s}}(\xi)\,d\xi G = F ∗ η ϵ s − 2 1 ∫ ξ T H ξ η ϵ s ( ξ ) d ξ . Then ∇ G ( x i ) = g i \nabla G(x_{i})=g_{i} ∇ G ( x i ) = g i , G = f G=f G = f for ∥ x ∥ < c 1 ρ w / ( 2 K ) \lVert x\rVert<c_{1}\rho_{w}/(2K) ∥ x ∥ < c 1 ρ w / ( 2 K ) , μ w I ⪯ ∇ 2 G ⪯ L I \mu_{w}I\preceq\nabla^{2}G\preceq LI μ w I ⪯ ∇ 2 G ⪯ L I , and the derivatives of order three and higher are bounded by those of the kernel times L L L . Choose θ \theta θ so small that all the above holds and R < c 1 ρ w / ( 2 K ) R<c_{1}\rho_{w}/(2K) R < c 1 ρ w / ( 2 K ) . Then SESOP generates x 1 , x 2 , x 3 x_{1},x_{2},x_{3} x 1 , x 2 , x 3 , and Step 2 gives the tail, which completes the proof. □
By Proposition 4.14, lim inf ϵ ↓ 0 N ( ϵ ) / log ( 1 / ϵ ) ≥ ( K − 1 ) / 128 \liminf_{\epsilon\downarrow0}N(\epsilon)/\log(1/\epsilon)\geq(K-1)/128 lim inf ϵ ↓ 0 N ( ϵ ) / log ( 1/ ϵ ) ≥ ( K − 1 ) /128 , which for fixed weights is of the order of the condition number κ ≤ 2 K / μ w \kappa\leq2K/\mu_{w} κ ≤ 2 K / μ w of G G G . As K K K is the condition number near the minimizer, unrestarted SESOP has no local accelerated rate either, unlike the method of [24 ] with adaptive restarts. The conditions (13) are strict inequalities, and an exact step restores the orthogonality relations, so this slow regime survives small perturbations of the state at x 3 x_{3} x 3 , which the cyclic tails do not.
Memories that accelerate
We now give memories that hold the virtual momentum of Lemma 3.3. A window of length of order κ \sqrt{\kappa} κ holds it over blocks of steps (§ 5.1 ). A pair of geometric filters holds it in two vectors once its factor matches κ \kappa κ , and § 5.2 finds this factor by a fixed schedule and by an adapted horizon, also with inexact subproblems (§ 5.3 ).
Windows
Corollary 4.10(a) requires a window of length Ω ( κ ) \Omega(\sqrt{\kappa}) Ω ( κ ) from a slowly forgetting memory. With π b = 0 \pi_{b}=0 π b = 0 and a constant a k = a a_{k}=a a k = a , the virtual momentum (3) is a combination of the gradients and steps after the origin b b b , so a window that covers a block of steps contains it, and the certificate of Lemma 3.3 can be restarted block by block. Only the length of the window depends on a bound K ≥ κ K \ge\kappa K ≥ κ . We say that S k S_k S k covers the last m m m gradients and n n n steps after b b b if it contains ∇ f ( x i ) \nabla f(x_i) ∇ f ( x i ) for max ( b , k − m + 1 ) ≤ i ≤ k \max(b,k-m+1) \le i \le k max ( b , k − m + 1 ) ≤ i ≤ k and s i s_i s i for max ( b + 1 , k − n + 1 ) ≤ i ≤ k \max(b+1,k-n+1) \le i \le k max ( b + 1 , k − n + 1 ) ≤ i ≤ k .
Proposition 5.1 (long windows suffice) Let the iterates satisfy (1) , and for some b ≥ 0 b \ge0 b ≥ 0 , m ≥ 1 m \ge1 m ≥ 1 and every k ≥ b k \ge b k ≥ b let S k S_k S k cover the last m m m gradients and m m m steps after b b b . If K ≥ κ K \ge\kappa K ≥ κ and m ≥ ⌈ 2 log 2 K ⌉ m \ge\lceil2 \log2 \sqrt{K} \rceil m ≥ ⌈ 2 log 2 K ⌉ , then f ( x b + h ) − f ⋆ ≤ 2 ρ K h ( f ( x b ) − f ⋆ ) f(x_{b+h})-f^\star\le2\rho_K^{h}(f(x_b)-f^\star) f ( x b + h ) − f ⋆ ≤ 2 ρ K h ( f ( x b ) − f ⋆ ) for h ≥ 0 h \ge0 h ≥ 0 , with ρ K = 1 − 1 / ( 2 K ) \rho_K=1-1/(2\sqrt{K}) ρ K = 1 − 1/ ( 2 K ) .
Proof. For K = 1 K=1 K = 1 the claim is trivial. Otherwise apply Lemma 3.3 with π b = 0 \pi_b=0 π b = 0 , λ = L / K ≤ μ \lambda=L/K\le\mu λ = L / K ≤ μ and constant a k = a = K − 1 / 2 a_k=a=K^{-1/2} a k = a = K − 1/2 , so that a 2 / λ = 1 / L a^2/\lambda=1/L a 2 / λ = 1/ L . By (3) , π b + j \pi_{b+j} π b + j is a combination of ∇ f ( x b + i ) \nabla f(x_{b+i}) ∇ f ( x b + i ) and s b + i s_{b+i} s b + i , 1 ≤ i ≤ j 1\le i\le j 1 ≤ i ≤ j , and S b + j S_{b+j} S b + j covers them for j ≤ m j\le m j ≤ m . Hence Lemmas 3.1 and 3.3 give, with B = m + 2 B=m+2 B = m + 2 ,
f ( x b + h ) − f ⋆ ≤ min { 1 , 2 ( 1 − a ) h − 1 } ( f ( x b ) − f ⋆ ) ( 1 ≤ h ≤ B ) . f(x_{b+h})-f^\star\le\min\{1,2(1-a)^{h-1}\}(f(x_b)-f^\star) \qquad(1\le h\le B). f ( x b + h ) − f ⋆ ≤ min { 1 , 2 ( 1 − a ) h − 1 } ( f ( x b ) − f ⋆ ) ( 1 ≤ h ≤ B ) .
With ρ = ρ K \rho=\rho_K ρ = ρ K we have ρ 2 ≥ 1 − a \rho^2\ge1-a ρ 2 ≥ 1 − a and 2 ρ m ≤ 1 2\rho^m\le1 2 ρ m ≤ 1 , so f ( x b + B ) − f ⋆ ≤ 2 ρ 2 B − 2 ( f ( x b ) − f ⋆ ) ≤ ρ B ( f ( x b ) − f ⋆ ) f(x_{b+B})-f^\star\le2\rho^{2B-2}(f(x_b)-f^\star)\le\rho^B(f(x_b)-f^\star) f ( x b + B ) − f ⋆ ≤ 2 ρ 2 B − 2 ( f ( x b ) − f ⋆ ) ≤ ρ B ( f ( x b ) − f ⋆ ) , and f ( x b + h ) − f ⋆ ≤ 2 ρ h ( f ( x b ) − f ⋆ ) f(x_{b+h})-f^\star\le2\rho^h(f(x_b)-f^\star) f ( x b + h ) − f ⋆ ≤ 2 ρ h ( f ( x b ) − f ⋆ ) for h < B h<B h < B by ρ 2 ≥ 1 − a \rho^2\ge1-a ρ 2 ≥ 1 − a if h ≥ 2 h\ge2 h ≥ 2 and ρ ≥ 1 2 \rho\ge\frac12 ρ ≥ 2 1 if h = 1 h=1 h = 1 . Covering after b ′ ≥ b b'\ge b b ′ ≥ b follows from covering after b b b , so chaining blocks of length B B B gives the claim. □
A window of length 1.39 κ 1.39\sqrt{\kappa} 1.39 κ therefore suffices (the constant 2 log 2 ≈ 1.39 2\log2\approx1.39 2 log 2 ≈ 1.39 comes from the crude requirement 2 ρ K m ≤ 1 2\rho_K^m\le1 2 ρ K m ≤ 1 ), and the order Θ ( κ ) \Theta(\sqrt{\kappa}) Θ ( κ ) in Corollary 4.10 is sharp for slowly forgetting memories. If the Hessian at the minimizer is a multiple of the identity plus a term of low rank, even a short window is fast near the minimizer, because orthogonal gradients over a block of steps force a quadratic rate over blocks.
Theorem 5.2 (local window bound) Let f ∈ F μ , L f\in\mathcal{F}_{\mu,L} f ∈ F μ , L , let the iterates satisfy (1) , and for some b ≥ 0 b\ge0 b ≥ 0 , m ≥ 1 m\ge1 m ≥ 1 and every k ≥ b k\ge b k ≥ b let S k S_k S k cover the last m m m gradients and m − 1 m-1 m − 1 steps after b b b . Let ∇ 2 f \nabla^2 f ∇ 2 f be M M M -Lipschitz on the ball B = { x : ∥ x − x ⋆ ∥ < ω } B=\{x:\lVert x-x^\star\rVert<\omega\} B = { x : ∥ x − x ⋆ ∥ < ω } , and let ∇ 2 f ( x ⋆ ) = χ I + R \nabla^2f(x^\star)=\chi I+R ∇ 2 f ( x ⋆ ) = χ I + R with rank R ≤ m − 1 R\le m-1 R ≤ m − 1 . Then there is a constant c c c , depending only on m m m , L L L , μ \mu μ , M M M , ∥ R ∥ \lVert R\rVert ∥ R ∥ and ω \omega ω , such that f ( x k + m ) − f ⋆ ≤ c ( f ( x k ) − f ⋆ ) 2 f(x_{k+m})-f^\star\le c(f(x_k)-f^\star)^2 f ( x k + m ) − f ⋆ ≤ c ( f ( x k ) − f ⋆ ) 2 for k ≥ k 0 k\ge k_0 k ≥ k 0 , where k 0 k_0 k 0 is the first k ≥ b k\ge b k ≥ b with f ( x k ) − f ⋆ ≤ 1 2 μ ω 2 f(x_k)-f^\star\le\frac12\mu\omega^2 f ( x k ) − f ⋆ ≤ 2 1 μ ω 2 .
Proof. By Lemma 3.1 , f ( x k ) f(x_k) f ( x k ) decreases to f ⋆ f^\star f ⋆ , so k 0 k_0 k 0 exists and f ( x k ) − f ⋆ ≤ 1 2 μ ω 2 f(x_k)-f^\star\le\frac12\mu\omega^2 f ( x k ) − f ⋆ ≤ 2 1 μ ω 2 for k ≥ k 0 k\ge k_0 k ≥ k 0 . Fix such a k k k and shift the indices so that k = 0 k=0 k = 0 . If some ∇ f ( x i ) \nabla f(x_i) ∇ f ( x i ) , 0 ≤ i ≤ m 0\le i\le m 0 ≤ i ≤ m , vanishes, the claim is trivial, so let them be nonzero. Strong convexity and f ( x i ) ≤ f ( x 0 ) f(x_i)\le f(x_0) f ( x i ) ≤ f ( x 0 ) give ∥ x i − x ⋆ ∥ 2 ≤ 2 ( f ( x 0 ) − f ⋆ ) / μ < ω 2 \lVert x_i-x^\star\rVert^2\le2(f(x_0)-f^\star)/\mu<\omega^2 ∥ x i − x ⋆ ∥ 2 ≤ 2 ( f ( x 0 ) − f ⋆ ) / μ < ω 2 , so all segments between x 0 , … , x m , x ⋆ x_0,\ldots,x_m,x^\star x 0 , … , x m , x ⋆ lie in B B B .
By Lemma 3.1 , ∇ f ( x i ) ⊥ ∇ f ( x i − 1 ) \nabla f(x_i)\perp\nabla f(x_{i-1}) ∇ f ( x i ) ⊥ ∇ f ( x i − 1 ) , so strong convexity gives μ ∥ s i ∥ 2 ≤ ⟨ ∇ f ( x i ) − ∇ f ( x i − 1 ) , s i ⟩ ≤ ∥ ∇ f ( x i − 1 ) ∥ ∥ s i ∥ \mu\lVert s_i\rVert^2\le\langle\nabla f(x_i)-\nabla f(x_{i-1}),s_i\rangle\le\lVert\nabla f(x_{i-1})\rVert\lVert s_i\rVert μ ∥ s i ∥ 2 ≤ ⟨ ∇ f ( x i ) − ∇ f ( x i − 1 ) , s i ⟩ ≤ ∥ ∇ f ( x i − 1 )∥ ∥ s i ∥ . Put z i = ∇ f ( x i ) − ∇ f ( x i − 1 ) − χ s i z_i=\nabla f(x_i)-\nabla f(x_{i-1})-\chi s_i z i = ∇ f ( x i ) − ∇ f ( x i − 1 ) − χ s i for 1 ≤ i ≤ m 1\le i\le m 1 ≤ i ≤ m . Then z i = R s i + ϵ i z_i=Rs_i+\epsilon_i z i = R s i + ϵ i with
ϵ i = ∫ 0 1 ( ∇ 2 f ( x i − 1 + t s i ) − ∇ 2 f ( x ⋆ ) ) s i d t , ∥ ϵ i ∥ 2 ≤ 2 M 2 μ ( f ( x 0 ) − f ⋆ ) ∥ s i ∥ 2 . \epsilon_i=\int_0^1\left(\nabla^2f(x_{i-1}+ts_i)-\nabla^2f(x^\star)\right)s_i\,dt,\qquad\lVert\epsilon_i\rVert^2\le\frac{2M^2}{\mu}(f(x_0)-f^\star)\lVert s_i\rVert^2. ϵ i = ∫ 0 1 ( ∇ 2 f ( x i − 1 + t s i ) − ∇ 2 f ( x ⋆ ) ) s i d t , ∥ ϵ i ∥ 2 ≤ μ 2 M 2 ( f ( x 0 ) − f ⋆ ) ∥ s i ∥ 2 .
For 1 ≤ i < j ≤ m 1\le i<j\le m 1 ≤ i < j ≤ m the space S j − 1 S_{j-1} S j − 1 covers ∇ f ( x i − 1 ) \nabla f(x_{i-1}) ∇ f ( x i − 1 ) , ∇ f ( x i ) \nabla f(x_i) ∇ f ( x i ) and s i s_i s i , so ∇ f ( x i ) ⊥ z i \nabla f(x_i)\perp z_i ∇ f ( x i ) ⊥ z i , and ∇ f ( x 1 ) , … , ∇ f ( x m ) \nabla f(x_1),\ldots,\nabla f(x_m) ∇ f ( x 1 ) , … , ∇ f ( x m ) are mutually orthogonal. Let Q Q Q have the columns ∇ f ( x j ) / ∥ ∇ f ( x j ) ∥ \nabla f(x_j)/\lVert\nabla f(x_j)\rVert ∇ f ( x j ) / ∥ ∇ f ( x j )∥ and X X X the columns Q ⊤ z j / ∥ ∇ f ( x j − 1 ) ∥ Q^{\top}z_j/\|\nabla f(x_{j-1})\| Q ⊤ z j /∥∇ f ( x j − 1 ) ∥ , 1 ≤ j ≤ m 1 \le j \le m 1 ≤ j ≤ m . This matrix is upper triangular with the diagonal entries ∥ ∇ f ( x j ) ∥ / ∥ ∇ f ( x j − 1 ) ∥ \|\nabla f(x_j)\|/\|\nabla f(x_{j-1})\| ∥∇ f ( x j ) ∥/∥∇ f ( x j − 1 ) ∥ , so det X = ∥ ∇ f ( x m ) ∥ / ∥ ∇ f ( x 0 ) ∥ \det X = \|\nabla f(x_m)\|/\|\nabla f(x_0)\| det X = ∥∇ f ( x m ) ∥/∥∇ f ( x 0 ) ∥ . Its columns are a j + e j a_j+e_j a j + e j with a j = Q ⊤ R s j / ∥ ∇ f ( x j − 1 ) ∥ a_j=Q^{\top}Rs_j/\|\nabla f(x_{j-1})\| a j = Q ⊤ R s j /∥∇ f ( x j − 1 ) ∥ and e j = Q ⊤ ϵ j / ∥ ∇ f ( x j − 1 ) ∥ e_j=Q^{\top}\epsilon_j/\|\nabla f(x_{j-1})\| e j = Q ⊤ ϵ j /∥∇ f ( x j − 1 ) ∥ . The vectors a j a_j a j lie in the image of Q ⊤ R Q^{\top}R Q ⊤ R , whose dimension is at most m − 1 m-1 m − 1 , so expanding det X \det X det X by multilinearity in the columns leaves only terms with at least one column e j e_j e j . Hadamard’s inequality bounds each term by the product of its column norms, and ∥ a j ∥ ≤ ∥ R ∥ / μ \|a_j\| \le\|R\|/\mu ∥ a j ∥ ≤ ∥ R ∥/ μ and ∥ e j ∥ ≤ ( 2 M 2 ( f ( x 0 ) − f ⋆ ) / μ 3 ) 1 / 2 < M ω / μ \|e_j\| \le(2M^2(f(x_0)-f^\star)/\mu^3)^{1/2} < M\omega/\mu ∥ e j ∥ ≤ ( 2 M 2 ( f ( x 0 ) − f ⋆ ) / μ 3 ) 1/2 < M ω / μ give
∥ ∇ f ( x m ) ∥ ∥ ∇ f ( x 0 ) ∥ ≤ ∏ j = 1 m ( ∥ a j ∥ + ∥ e j ∥ ) − ∏ j = 1 m ∥ a j ∥ ≤ m ( 2 M 2 ( f ( x 0 ) − f ⋆ ) μ 3 ) 1 / 2 ( ∥ R ∥ + M ω μ ) m − 1 . \frac{\|\nabla f(x_m)\|}{\|\nabla f(x_0)\|} \le\prod_{j=1}^{m}(\|a_j\|+\|e_j\|)-\prod_{j=1}^{m}\|a_j\| \le m\left(\frac{2M^2(f(x_0)-f^\star)}{\mu^3}\right)^{1/2}\left(\frac{\|R\|+M\omega}{\mu}\right)^{m-1}. ∥∇ f ( x 0 ) ∥ ∥∇ f ( x m ) ∥ ≤ j = 1 ∏ m ( ∥ a j ∥ + ∥ e j ∥ ) − j = 1 ∏ m ∥ a j ∥ ≤ m ( μ 3 2 M 2 ( f ( x 0 ) − f ⋆ ) ) 1/2 ( μ ∥ R ∥ + M ω ) m − 1 .
Squaring and using f ( x m ) − f ⋆ ≤ ∥ ∇ f ( x m ) ∥ 2 / ( 2 μ ) f(x_m)-f^\star\le\|\nabla f(x_m)\|^2/(2\mu) f ( x m ) − f ⋆ ≤ ∥∇ f ( x m ) ∥ 2 / ( 2 μ ) and ∥ ∇ f ( x 0 ) ∥ 2 ≤ 2 L ( f ( x 0 ) − f ⋆ ) \|\nabla f(x_0)\|^2 \le2L(f(x_0)-f^\star) ∥∇ f ( x 0 ) ∥ 2 ≤ 2 L ( f ( x 0 ) − f ⋆ ) gives the claim with
c = 2 m 2 L M 2 μ 4 ( ∥ R ∥ + M ω μ ) 2 ( m − 1 ) . c=\frac{2m^2LM^2}{\mu^4}\left(\frac{\|R\|+M\omega}{\mu}\right)^{2(m-1)}. c = μ 4 2 m 2 L M 2 ( μ ∥ R ∥ + M ω ) 2 ( m − 1 ) .
On a quadratic with rank ( ∇ 2 f − χ I ) ≤ m − 1 \operatorname{rank}(\nabla^2 f-\chi I)\le m-1 rank ( ∇ 2 f − χ I ) ≤ m − 1 , M = 0 M=0 M = 0 and every method of the theorem terminates within m m m steps. The value of c c c from the proof is far from optimal and grows exponentially with m m m , so the theorem concerns the order of convergence, and the quadratic regime may start only close to the minimizer. Near the minimizer of the instances of Proposition 4.14, ∇ 2 G = I + ( K − 1 ) e e ⊤ \nabla^2G=I+(K-1)ee^{\top} ∇ 2 G = I + ( K − 1 ) e e ⊤ , and covering with m = 2 m=2 m = 2 needs only ∇ f ( x k ) \nabla f(x_k) ∇ f ( x k ) , ∇ f ( x k − 1 ) \nabla f(x_{k-1}) ∇ f ( x k − 1 ) and s k s_k s k . SESOP with the previous gradient added therefore reaches the minimizer two steps after entering a ball in the quadratic region, while SESOP itself keeps G ( x k + 1 ) ≥ ( K − 1 ) G ( x k ) / ( K + 127 ) G(x_{k+1})\ge(K-1)G(x_k)/(K+127) G ( x k + 1 ) ≥ ( K − 1 ) G ( x k ) / ( K + 127 ) , so adding one vector turns the rate of gradient descent into finite termination.
In a window of length m m m the last m + 1 m+1 m + 1 gradients are orthogonal, so dim S k ≥ m \dim S_k\ge m dim S k ≥ m before termination, and with m ≍ κ m\asymp\sqrt{\kappa} m ≍ κ a Newton step costs about κ \sqrt{\kappa} κ Hessian-vector products, κ log ( 1 / ε ) \kappa\log(1/\varepsilon) κ log ( 1/ ε ) in total. § 5.2 stores such a horizon in two vectors.
Filters on the right time scale
Two geometric filters keep the virtual momentum of Lemma 3.3 for a constant step. Any a ≤ 1 / κ a\le1/\sqrt{\kappa} a ≤ 1/ κ is admissible there, with the rate 1 − a 1-a 1 − a , but the best choice a = 1 / κ a=1/\sqrt{\kappa} a = 1/ κ is unknown. With π b = 0 \pi_b=0 π b = 0 and a k = a a_k=a a k = a , unrolling (3) gives
π k = y k + a λ z k , z k = ∑ i = b + 1 k ρ k − i ∇ f ( x i ) , y k = ∑ i = b + 1 k ρ k − i + 1 s i , \pi_k=y_k+\frac{a}{\lambda}z_k,\qquad z_k=\sum_{i=b+1}^{k}\rho^{k-i}\nabla f(x_i),\qquad y_k=\sum_{i=b+1}^{k}\rho^{k-i+1}s_i, π k = y k + λ a z k , z k = i = b + 1 ∑ k ρ k − i ∇ f ( x i ) , y k = i = b + 1 ∑ k ρ k − i + 1 s i ,
with the factor ρ = 1 − a \rho=1-a ρ = 1 − a . A method that stores z k z_k z k and y k y_k y k separately never needs the coefficient a / λ a/\lambda a / λ , which involves the unknown constants, and has to choose only ρ \rho ρ .
We restrict a a a to the dyadic values α J = 2 − J \alpha_J=2^{-J} α J = 2 − J and take λ = 4 − J L \lambda=4^{-J}L λ = 4 − J L . A level J ≥ 1 J\ge1 J ≥ 1 is admissible exactly when 4 J ≥ κ 4^J\ge\kappa 4 J ≥ κ , and it is nearly optimal when also 4 J ≤ 4 κ 4^J\le4\kappa 4 J ≤ 4 κ . With ρ J = 1 − α J \rho_J=1-\alpha_J ρ J = 1 − α J , the filter pair of level J J J started at b b b is
z b = y b = 0 , z k + 1 = ρ J z k + ∇ f ( x k + 1 ) , y k + 1 = ρ J ( y k + s k + 1 ) ( k ≥ b ) . (15) z_b=y_b=0,\qquad z_{k+1}=\rho_Jz_k+\nabla f(x_{k+1}),\qquad y_{k+1}=\rho_J(y_k+s_{k+1})\quad(k\ge b). \tag*{(15)} z b = y b = 0 , z k + 1 = ρ J z k + ∇ f ( x k + 1 ) , y k + 1 = ρ J ( y k + s k + 1 ) ( k ≥ b ) . (15)
Algorithm 1 Activation schedule
Require: starting point x 0 x_0 x 0
Compute ∇ f ( x 0 ) \nabla f(x_0) ∇ f ( x 0 ) .
for k = 0 , 1 , 2 , … k = 0, 1, 2, \ldots k = 0 , 1 , 2 , … do
If k = 2 j k = 2^j k = 2 j with j ≥ 1 j \ge1 j ≥ 1 , create the pair z k ( j ) = y k ( j ) = 0 z_k^{(j)} = y_k^{(j)} = 0 z k ( j ) = y k ( j ) = 0 .
Set S k = span { ∇ f ( x k ) } ∪ { z k ( j ) , y k ( j ) : 2 ≤ 2 j ≤ k } S_k = \operatorname{span}\{\nabla f(x_k)\} \cup\{z_k^{(j)}, y_k^{(j)} : 2 \le2^j \le k\} S k = span { ∇ f ( x k )} ∪ { z k ( j ) , y k ( j ) : 2 ≤ 2 j ≤ k } .
Compute x k + 1 = arg min x k + S k f x_{k+1} = \arg\min_{x_k+S_k} f x k + 1 = arg min x k + S k f , s k + 1 = x k + 1 − x k s_{k+1} = x_{k+1} - x_k s k + 1 = x k + 1 − x k and ∇ f ( x k + 1 ) \nabla f(x_{k+1}) ∇ f ( x k + 1 ) .
Update every pair by (13) with its own factor ρ j \rho_j ρ j .
end for
Lemma 5.3 (certificate of a filter pair) Let f ∈ F μ , L f \in\mathcal{F}_{\mu,L} f ∈ F μ , L and κ ≤ 4 J \kappa\le4^J κ ≤ 4 J , and let the iterates satisfy (1) with any rule for S k S_k S k . If the pair (13) of level J J J started at b b b lies in S k S_k S k for b ≤ k < b + h b \le k < b+h b ≤ k < b + h , then f ( x b + h + 1 ) − f ⋆ ≤ 2 ρ J h ( f ( x b ) − f ⋆ ) f(x_{b+h+1}) - f^\star\le2\rho_J^h(f(x_b) - f^\star) f ( x b + h + 1 ) − f ⋆ ≤ 2 ρ J h ( f ( x b ) − f ⋆ ) .
Proof With α = 2 − J \alpha= 2^{-J} α = 2 − J and λ = α 2 L ≤ μ \lambda= \alpha^2 L \le\mu λ = α 2 L ≤ μ we have α 2 / λ = 1 / L \alpha^2/\lambda= 1/L α 2 / λ = 1/ L . The vectors π k = y k + α z k \pi_k = y_k + \alpha z_k π k = y k + α z k satisfy π b = 0 \pi_b = 0 π b = 0 and, with a k = α a_k = \alpha a k = α ,
π k + 1 = ρ J ( y k + s k + 1 ) + α ρ J ( ρ J z k + ∇ f ( x k + 1 ) ) = ρ J ( π k + s k + 1 ) + α ρ J ∇ f ( x k + 1 ) , \pi_{k+1} = \rho_J(y_k+s_{k+1})+\frac{\alpha}{\rho_J}(\rho_J z_k+\nabla f(x_{k+1}))=\rho_J(\pi_k+s_{k+1})+\frac{\alpha}{\rho_J}\nabla f(x_{k+1}), π k + 1 = ρ J ( y k + s k + 1 ) + ρ J α ( ρ J z k + ∇ f ( x k + 1 )) = ρ J ( π k + s k + 1 ) + ρ J α ∇ f ( x k + 1 ) ,
which is (3) . They lie in S k S_k S k , so Lemma 3.3 with C b ≤ 2 ( f ( x b ) − f ⋆ ) C_b \le2(f(x_b)-f^\star) C b ≤ 2 ( f ( x b ) − f ⋆ ) gives the claim. □
The pair has δ = 2 − J ≥ 1 / ( 2 κ ) \delta= 2^{-J} \ge1/(2\sqrt{\kappa}) δ = 2 − J ≥ 1/ ( 2 κ ) , as Corollary 4.10 requires for a bounded window. For κ ≤ 4 J ≤ 4 κ \kappa\le4^J \le4\kappa κ ≤ 4 J ≤ 4 κ the rate is exp ( − 1 / ( 2 κ ) ) \exp(-1/(2\sqrt{\kappa})) exp ( − 1/ ( 2 κ )) per step with two stored vectors, where the window of Proposition 5.1 needs about 2.8 κ 2.8\sqrt{\kappa} 2.8 κ , that is thousands of vectors for κ = 10 6 \kappa= 10^6 κ = 1 0 6 .
Activation schedule.
Algorithm 1 creates the pair of level j j j at iteration 2 j 2^j 2 j , when the run is as long as its horizon 1 / α j 1/\alpha_j 1/ α j , and never discards it. It uses no constant of the function and no test.
Theorem 5.4 (activation schedule) Let f ∈ F μ , L f \in\mathcal{F}_{\mu,L} f ∈ F μ , L . Algorithm 1 satisfies f ( x k + 1 ) − f ⋆ ≤ 2 e exp ( − k / ( 2 κ ) ) ( f ( x 0 ) − f ⋆ ) f(x_{k+1}) - f^\star\le2e\exp(-k/(2\sqrt{\kappa}))(f(x_0)-f^\star) f ( x k + 1 ) − f ⋆ ≤ 2 e exp ( − k / ( 2 κ )) ( f ( x 0 ) − f ⋆ ) for k ≥ 0 k \ge0 k ≥ 0 , and at iteration k ≥ 2 k \ge2 k ≥ 2 it stores 2 ⌊ log 2 k ⌋ 2\lfloor\log_2 k\rfloor 2 ⌊ log 2 k ⌋ vectors. Hence N ( ε ) ≤ 1 + ⌈ 2 κ log ( 2 e / ε ) ⌉ N(\varepsilon) \le1+\lceil2\sqrt{\kappa}\log(2e/\varepsilon)\rceil N ( ε ) ≤ 1 + ⌈ 2 κ log ( 2 e / ε )⌉ , with O ( 1 + log κ + log ( 1 + log ( 1 / ε ) ) ) O(1+\log\kappa+\log(1+\log(1/\varepsilon))) O ( 1 + log κ + log ( 1 + log ( 1/ ε ))) vectors stored by then.
Proof Let J = max { 1 , ⌈ log 4 κ ⌉ } J=\max\{1,\lceil\log_4\kappa\rceil\} J = max { 1 , ⌈ log 4 κ ⌉} , so that κ ≤ 4 J ≤ 4 κ \kappa\le4^J \le4\kappa κ ≤ 4 J ≤ 4 κ and α = 2 − J \alpha=2^{-J} α = 2 − J satisfies 1 2 κ ≤ α ≤ 1 κ \frac{1}{2\sqrt{\kappa}}\le\alpha\le\frac{1}{\sqrt{\kappa}} 2 κ 1 ≤ α ≤ κ 1 .
The pair of level J J J is activated at τ = 2 J = 1 / α ≤ 2 κ \tau=2^J=1/\alpha\le2\sqrt{\kappa} τ = 2 J = 1/ α ≤ 2 κ with zero state. Lemma 5.3 with origin b = τ b=\tau b = τ and monotonicity give, for k ≥ τ k\ge\tau k ≥ τ ,
f ( x k + 1 ) − f ⋆ ≤ 2 ( 1 − α ) k − τ ( f ( x τ ) − f ⋆ ) ≤ 2 e 1 − α k ( f ( x 0 ) − f ⋆ ) ≤ 2 e e − k / ( 2 κ ) ( f ( x 0 ) − f ⋆ ) . f(x_{k+1})-f^\star\le2(1-\alpha)^{k-\tau}(f(x_\tau)-f^\star)\le2e^{1-\alpha k}(f(x_0)-f^\star)\le2e e^{-k/(2\sqrt{\kappa})}(f(x_0)-f^\star). f ( x k + 1 ) − f ⋆ ≤ 2 ( 1 − α ) k − τ ( f ( x τ ) − f ⋆ ) ≤ 2 e 1 − α k ( f ( x 0 ) − f ⋆ ) ≤ 2 e e − k / ( 2 κ ) ( f ( x 0 ) − f ⋆ ) .
For k < τ k<\tau k < τ the right-hand side exceeds 2 ( f ( x 0 ) − f ⋆ ) 2(f(x_0)-f^\star) 2 ( f ( x 0 ) − f ⋆ ) , and the bound holds by monotonicity. At iteration k ≥ 2 k\ge2 k ≥ 2 the active pairs are those with 2 ≤ 2 j ≤ k 2\le2^j\le k 2 ≤ 2 j ≤ k , which gives the storage, and solving the rate for k k k gives N ( ε ) N(\varepsilon) N ( ε ) . □
The order κ log ( 1 / ε ) \sqrt{\kappa}\log(1/\varepsilon) κ log ( 1/ ε ) matches the oracle lower bound [1 ] . Not knowing κ \kappa κ costs the schedule a factor two in the exponent and a memory that grows like log k \log k log k , and no restart, test or estimate of a constant is involved.
Algorithm 2 Smooth adaptive pair
Require: starting point x 0 x_0 x 0
Compute ∇ f ( x 0 ) \nabla f(x_0) ∇ f ( x 0 ) and set H 0 = 2 H_0=2 H 0 = 2 , R 0 = ∥ ∇ f ( x 0 ) ∥ 2 R_0=\lVert\nabla f(x_0)\rVert^2 R 0 = ∥ ∇ f ( x 0 ) ∥ 2 and z 0 = y 0 = 0 z_0=y_0=0 z 0 = y 0 = 0 .
for k = 0 , 1 , 2 , … k=0,1,2,\ldots k = 0 , 1 , 2 , … do
Set a k = 1 / H k a_k=1/H_k a k = 1/ H k , ρ k = 1 − a k \rho_k=1-a_k ρ k = 1 − a k and S k = span { ∇ f ( x k ) , z k , y k } S_k=\operatorname{span}\{\nabla f(x_k),z_k,y_k\} S k = span { ∇ f ( x k ) , z k , y k } , optionally with further directions.
Compute x k + 1 = arg min x k + S k f x_{k+1}=\operatorname*{arg\,min}_{x_k+S_k}f x k + 1 = arg min x k + S k f , s k + 1 = x k + 1 − x k s_{k+1}=x_{k+1}-x_k s k + 1 = x k + 1 − x k and ∇ f ( x k + 1 ) \nabla f(x_{k+1}) ∇ f ( x k + 1 ) .
Update z k + 1 = ρ k z k + a k ∇ f ( x k + 1 ) z_{k+1}=\rho_kz_k+a_k\nabla f(x_{k+1}) z k + 1 = ρ k z k + a k ∇ f ( x k + 1 ) and y k + 1 = ρ k ( y k + s k + 1 ) y_{k+1}=\rho_k(y_k+s_{k+1}) y k + 1 = ρ k ( y k + s k + 1 ) .
Update R k + 1 = ( 1 − a k 2 ) R k + a k 2 ∥ ∇ f ( x k + 1 ) ∥ 2 R_{k+1}=\left(1-\frac{a_k}{2}\right)R_k+\frac{a_k}{2}\lVert\nabla f(x_{k+1})\rVert^2 R k + 1 = ( 1 − 2 a k ) R k + 2 a k ∥ ∇ f ( x k + 1 ) ∥ 2 and ℓ k = log ( 1 + a k ∥ ∇ f ( x k + 1 ) ∥ 2 ( 2 − a k ) R k ) \ell_k=\log\left(1+\frac{a_k\lVert\nabla f(x_{k+1})\rVert^2}{(2-a_k)R_k}\right) ℓ k = log ( 1 + ( 2 − a k ) R k a k ∥ ∇ f ( x k + 1 ) ∥ 2 ) .
Update H k + 1 = exp ( ( log H k ) 2 + ℓ k / 4 ) H_{k+1}=\exp\left(\sqrt{(\log H_k)^2+\ell_k/4}\right) H k + 1 = exp ( ( log H k ) 2 + ℓ k /4 ) .
end for
Smooth adaptive pair.
Algorithm 2 updates one pair with the factor ρ k = 1 − 1 / H k \rho_k=1-1/H_k ρ k = 1 − 1/ H k . The scalar R k R_k R k averages ∥ ∇ f ( x k ) ∥ 2 \lVert\nabla f(x_k)\rVert^2 ∥ ∇ f ( x k ) ∥ 2 with the factor 1 − 1 / ( 2 H k ) 1-1/(2H_k) 1 − 1/ ( 2 H k ) , and ℓ k = log ( R k + 1 / ( ( 1 − 1 2 H k ) R k ) ) \ell_k=\log(R_{k+1}/((1-\frac{1}{2H_k})R_k)) ℓ k = log ( R k + 1 / (( 1 − 2 H k 1 ) R k )) measures how much the gradient energy fails to decay on the current horizon. The horizon never decreases, and ( log H k ) 2 (\log H_k)^2 ( log H k ) 2 grows by ℓ k / 4 \ell_k/4 ℓ k /4 , so H k H_k H k grows quickly only while this happens.
Theorem 5.5 (smooth adaptive pair) Let f ∈ F μ , L f\in\mathcal{F}_{\mu,L} f ∈ F μ , L with f ( x 0 ) − f ⋆ > 0 f(x_0)-f^\star>0 f ( x 0 ) − f ⋆ > 0 , and let the iterates of Algorithm 2 satisfy (1) , with or without further directions in S k S_k S k . The method stores two vectors and two scalars, never resets them, and satisfies
f ( x k ) − f ⋆ ≤ ( 5 2 ) 8 κ 5 e − k / ( 5 κ ) ( f ( x 0 ) − f ⋆ ) , N ( ε ) ≤ ⌈ 5 κ [ log 1 ε + 5 log κ + 8 log 5 2 ] ⌉ . f(x_k)-f^\star\le\left(\frac{5}{2}\right)^8\kappa^5e^{-k/(5\sqrt{\kappa})}(f(x_0)-f^\star),\qquad N(\varepsilon)\le\left\lceil5\sqrt{\kappa}\left[\log\frac{1}{\varepsilon}+5\log\kappa+8\log\frac{5}{2}\right]\right\rceil. f ( x k ) − f ⋆ ≤ ( 2 5 ) 8 κ 5 e − k / ( 5 κ ) ( f ( x 0 ) − f ⋆ ) , N ( ε ) ≤ ⌈ 5 κ [ log ε 1 + 5 log κ + 8 log 2 5 ] ⌉ . If S k S_k S k is spanned by the pair, ∇ f ( x k ) \nabla f(x_k) ∇ f ( x k ) , s k s_k s k , the anchor x k − x 0 x_k-x_0 x k − x 0 and Nemirovski’s aggregate with w 0 = 1 w_0=1 w 0 = 1 , Proposition 3.2 applies as well.
Proof. The proof has three steps. The stored vectors are bounded by the gradient energy R k R_k R k , the horizon never exceeds 5 2 κ \frac{5}{2}\sqrt{\kappa} 2 5 κ , and a summation by parts turns the growth of H k H_k H k into the rate. Put q k = 1 − a k / 2 q_k=1-a_k/2 q k = 1 − a k /2 . If some ∇ f ( x k ) \nabla f(x_k) ∇ f ( x k ) vanishes, the minimizer is reached and all claims are trivial, so let ∇ f ( x k ) ≠ 0 \nabla f(x_k)\ne0 ∇ f ( x k ) = 0 . By the update of R k R_k R k , R k + 1 = q k R k e ℓ k R_{k+1}=q_kR_ke^{\ell_k} R k + 1 = q k R k e ℓ k , and the update of H k H_k H k gives
4 ( log H k + 1 ) 2 − 4 ( log H k ) 2 = ℓ k ≥ 0. (16) 4(\log H_{k+1})^2-4(\log H_k)^2=\ell_k\ge0. \tag*{(16)} 4 ( log H k + 1 ) 2 − 4 ( log H k ) 2 = ℓ k ≥ 0. (16)
Hence H k H_k H k is nondecreasing, H k ≥ 2 H_k\ge2 H k ≥ 2 , a k ≤ 1 2 a_k\le\frac12 a k ≤ 2 1 and q k ≥ 3 4 q_k\ge\frac34 q k ≥ 4 3 .
Step 1: the stored states. We show that for every k ≥ 0 k\ge0 k ≥ 0
∥ z k ∥ 2 ≤ R k , ∥ y k ∥ 2 ≤ 2 H k 2 μ 2 R k , ∥ ∇ f ( x k ) ∥ 2 ≤ 2 H k R k , 2 μ ( f ( x k ) − f ⋆ ) ≤ R k ≤ 2 L ( f ( x 0 ) − f ⋆ ) . (17) \lVert z_k\rVert^2\le R_k,\qquad\lVert y_k\rVert^2\le\frac{2H_k^2}{\mu^2}R_k,\qquad\lVert\nabla f(x_k)\rVert^2\le2H_kR_k,\qquad2\mu(f(x_k)-f^\star)\le R_k\le2L(f(x_0)-f^\star). \tag*{(17)} ∥ z k ∥ 2 ≤ R k , ∥ y k ∥ 2 ≤ μ 2 2 H k 2 R k , ∥ ∇ f ( x k ) ∥ 2 ≤ 2 H k R k , 2 μ ( f ( x k ) − f ⋆ ) ≤ R k ≤ 2 L ( f ( x 0 ) − f ⋆ ) . (17)
Since z k ∈ S k z_k\in S_k z k ∈ S k and ∇ f ( x k + 1 ) ⊥ S k \nabla f(x_{k+1})\perp S_k ∇ f ( x k + 1 ) ⊥ S k , ∥ z k + 1 ∥ 2 = ρ k 2 ∥ z k ∥ 2 + a k 2 ∥ ∇ f ( x k + 1 ) ∥ 2 ≤ R k + 1 \lVert z_{k+1}\rVert^2=\rho_k^2\lVert z_k\rVert^2+a_k^2\lVert\nabla f(x_{k+1})\rVert^2\le R_{k+1} ∥ z k + 1 ∥ 2 = ρ k 2 ∥ z k ∥ 2 + a k 2 ∥ ∇ f ( x k + 1 ) ∥ 2 ≤ R k + 1 by induction, because ρ k 2 ≤ q k \rho_k^2\le q_k ρ k 2 ≤ q k and a k 2 ≤ a k / 2 a_k^2\le a_k/2 a k 2 ≤ a k /2 . For y k y_k y k put c j = ∏ t = j k − 1 ρ t c_j=\prod_{t=j}^{k-1}\rho_t c j = ∏ t = j k − 1 ρ t , so that y k = ∑ j = 0 k − 1 c j s j + 1 y_k=\sum_{j=0}^{k-1}c_js_{j+1} y k = ∑ j = 0 k − 1 c j s j + 1 . Strong convexity and ∇ f ( x j + 1 ) ⊥ s j + 1 \nabla f(x_{j+1})\perp s_{j+1} ∇ f ( x j + 1 ) ⊥ s j + 1 give μ ∥ s j + 1 ∥ 2 ≤ ⟨ ∇ f ( x j + 1 ) − ∇ f ( x j ) , s j + 1 ⟩ ≤ ∥ ∇ f ( x j ) ∥ ∥ s j + 1 ∥ \mu\lVert s_{j+1}\rVert^2\le\langle\nabla f(x_{j+1})-\nabla f(x_j),s_{j+1}\rangle\le\lVert\nabla f(x_j)\rVert\lVert s_{j+1}\rVert μ ∥ s j + 1 ∥ 2 ≤ ⟨ ∇ f ( x j + 1 ) − ∇ f ( x j ) , s j + 1 ⟩ ≤ ∥ ∇ f ( x j )∥ ∥ s j + 1 ∥ . Since H t ≤ H k H_t\le H_k H t ≤ H k for t ≤ k t\le k t ≤ k , c j ≤ ( 1 − 1 / H k ) k − j c_j\le(1-1/H_k)^{k-j} c j ≤ ( 1 − 1/ H k ) k − j and ∑ j c j ≤ H k \sum_jc_j\le H_k ∑ j c j ≤ H k . Expanding the recursion of R k R_k R k , the coefficient of ∥ ∇ f ( x 0 ) ∥ 2 \lVert\nabla f(x_0)\rVert^2 ∥ ∇ f ( x 0 ) ∥ 2 is ∏ t < k q t ≥ c 0 \prod_{t<k}q_t\ge c_0 ∏ t < k q t ≥ c 0 , and that of ∥ ∇ f ( x j ) ∥ 2 \lVert\nabla f(x_j)\rVert^2 ∥ ∇ f ( x j ) ∥ 2 , 1 ≤ j < k 1 \le j < k 1 ≤ j < k , is a j − 1 2 ∏ t = j k − 1 q t ≥ c j / ( 2 H k ) \frac{a_{j-1}}{2}\prod_{t=j}^{k-1}q_t \ge c_j/(2H_k) 2 a j − 1 ∏ t = j k − 1 q t ≥ c j / ( 2 H k ) , because q t ≥ ρ t q_t \ge\rho_t q t ≥ ρ t and a j − 1 ≥ 1 / H k a_{j-1} \ge1/H_k a j − 1 ≥ 1/ H k . Hence 2 H k R k ≥ ∑ j < k c j ∥ ∇ f ( x j ) ∥ 2 2H_kR_k \ge\sum_{j<k}c_j\lVert\nabla f(x_j)\rVert^2 2 H k R k ≥ ∑ j < k c j ∥ ∇ f ( x j ) ∥ 2 , and the Cauchy–Schwarz inequality gives
∥ y k ∥ ≤ 1 μ ∑ j = 0 k − 1 c j ∥ ∇ f ( x j ) ∥ ≤ 1 μ ( ∑ c j ) 1 / 2 ( ∑ c j ∥ ∇ f ( x j ) ∥ 2 ) 1 / 2 ≤ 2 H k μ R k . \lVert y_k\rVert\le\frac{1}{\mu}\sum_{j=0}^{k-1}c_j\lVert\nabla f(x_j)\rVert\le\frac{1}{\mu}\left(\sum c_j\right)^{1/2}\left(\sum c_j\lVert\nabla f(x_j)\rVert^2\right)^{1/2} \le\frac{\sqrt{2H_k}}{\mu}\sqrt{R_k}. ∥ y k ∥ ≤ μ 1 j = 0 ∑ k − 1 c j ∥ ∇ f ( x j )∥ ≤ μ 1 ( ∑ c j ) 1/2 ( ∑ c j ∥ ∇ f ( x j ) ∥ 2 ) 1/2 ≤ μ 2 H k R k .
For k ≥ 1 k \ge1 k ≥ 1 , R k ≥ a k − 1 2 ∥ ∇ f ( x k ) ∥ 2 ≥ ∥ ∇ f ( x k ) ∥ 2 / ( 2 H k ) R_k \ge\frac{a_{k-1}}{2}\lVert\nabla f(x_k)\rVert^2 \ge\lVert\nabla f(x_k)\rVert^2/(2H_k) R k ≥ 2 a k − 1 ∥ ∇ f ( x k ) ∥ 2 ≥ ∥ ∇ f ( x k ) ∥ 2 / ( 2 H k ) , and R 0 = ∥ ∇ f ( x 0 ) ∥ 2 R_0=\lVert\nabla f(x_0)\rVert^2 R 0 = ∥ ∇ f ( x 0 ) ∥ 2 , which gives the third bound. Finally R k R_k R k is a convex combination of ∥ ∇ f ( x j ) ∥ 2 \lVert\nabla f(x_j)\rVert^2 ∥ ∇ f ( x j ) ∥ 2 , j ≤ k j \le k j ≤ k , and 2 μ ( f ( x k ) − f ⋆ ) ≤ ∥ ∇ f ( x j ) ∥ 2 ≤ 2 L ( f ( x 0 ) − f ⋆ ) 2\mu(f(x_k)-f^\star) \le\lVert\nabla f(x_j)\rVert^2 \le2L(f(x_0)-f^\star) 2 μ ( f ( x k ) − f ⋆ ) ≤ ∥ ∇ f ( x j ) ∥ 2 ≤ 2 L ( f ( x 0 ) − f ⋆ ) for j ≤ k j \le k j ≤ k by Lemma 3.1 .
Step 2: the horizon. Let κ ≥ 4 \kappa\ge4 κ ≥ 4 . Lemma 3.1 and (17) give ∥ ∇ f ( x k + 1 ) ∥ 2 ≤ κ ∥ ∇ f ( x k ) ∥ 2 ≤ 2 κ H k R k \lVert\nabla f(x_{k+1})\rVert^2 \le\kappa\lVert\nabla f(x_k)\rVert^2 \le2\kappa H_kR_k ∥ ∇ f ( x k + 1 ) ∥ 2 ≤ κ ∥ ∇ f ( x k ) ∥ 2 ≤ 2 κ H k R k , so the update of ℓ k \ell_k ℓ k and q k ≥ 4 3 q_k \ge\frac{4}{3} q k ≥ 3 4 give ℓ k ≤ log ( 1 + 4 κ ) \ell_k \le\log(1+\frac{4}{\kappa}) ℓ k ≤ log ( 1 + κ 4 ) . If H k < κ H_k < \sqrt{\kappa} H k < κ for all k k k , there is nothing to prove. Otherwise let b b b be the first index with H b ≥ κ H_b \ge\sqrt{\kappa} H b ≥ κ . If b = 0 b=0 b = 0 , then κ = 4 \kappa=4 κ = 4 and H b = κ H_b=\sqrt{\kappa} H b = κ . If b ≥ 1 b\ge1 b ≥ 1 , then (16) gives 4 ( log H b ) 2 < ( log κ ) 2 + log ( 1 + 4 κ ) ≤ 4 ( log 7 5 κ ) 2 4(\log H_b)^2 < (\log\kappa)^2+\log(1+\frac{4}{\kappa}) \le4(\log\frac{7}{5}\sqrt{\kappa})^2 4 ( log H b ) 2 < ( log κ ) 2 + log ( 1 + κ 4 ) ≤ 4 ( log 5 7 κ ) 2 , where the last inequality holds at κ = 4 \kappa=4 κ = 4 and the derivative in log κ \log\kappa log κ of its right side exceeds that of its left side by at least 4 log 7 5 − 1 > 0 4\log\frac{7}{5}-1>0 4 log 5 7 − 1 > 0 . Hence H b ≤ 7 5 κ H_b\le\frac{7}{5}\sqrt{\kappa} H b ≤ 5 7 κ .
For t ≥ b t\ge b t ≥ b we have a t ≤ 1 / κ a_t\le1/\sqrt{\kappa} a t ≤ 1/ κ , so Lemma 3.3 applies with λ = μ \lambda=\mu λ = μ , the variable a t a_t a t and π b = y b + z b / μ \pi_b=y_b+z_b/\mu π b = y b + z b / μ , because the pair recursions give (3) for π t = y t + z t / μ \pi_t=y_t+z_t/\mu π t = y t + z t / μ , which lies in S t S_t S t . By (17) and strong convexity the constant C b C_b C b of Lemma 3.3 satisfies
f ( x b ) − f ⋆ + μ 2 ∥ x b − π b − x ⋆ ∥ 2 ≤ 3 ( f ( x b ) − f ⋆ ) + μ ∥ π b ∥ 2 ≤ ( 3 2 + 4 H b 2 + 2 ) R b μ ≤ 6 H b 2 R b μ , f(x_b)-f^\star+\frac{\mu}{2}\lVert x_b-\pi_b-x^\star\rVert^2 \le3(f(x_b)-f^\star)+\mu\lVert\pi_b\rVert^2 \le\left(\frac{3}{2}+4H_b^2+2\right)\frac{R_b}{\mu} \le6H_b^2\frac{R_b}{\mu}, f ( x b ) − f ⋆ + 2 μ ∥ x b − π b − x ⋆ ∥ 2 ≤ 3 ( f ( x b ) − f ⋆ ) + μ ∥ π b ∥ 2 ≤ ( 2 3 + 4 H b 2 + 2 ) μ R b ≤ 6 H b 2 μ R b ,
where H b ≥ 2 H_b\ge2 H b ≥ 2 was used in the last step. Hence ∥ ∇ f ( x t + 1 ) ∥ 2 ≤ 2 L ( f ( x t + 1 ) − f ⋆ ) ≤ 12 κ H b 2 R b ∏ i = b t ρ i \lVert\nabla f(x_{t+1})\rVert^2\le2L(f(x_{t+1})-f^\star)\le12\kappa H_b^2R_b\prod_{i=b}^{t}\rho_i ∥ ∇ f ( x t + 1 ) ∥ 2 ≤ 2 L ( f ( x t + 1 ) − f ⋆ ) ≤ 12 κ H b 2 R b ∏ i = b t ρ i . Put P t = exp ( ∑ i = b t − 1 ℓ i ) P_t=\exp(\sum_{i=b}^{t-1}\ell_i) P t = exp ( ∑ i = b t − 1 ℓ i ) and V t = ∏ i = b t − 1 ( ρ i / q i ) V_t=\prod_{i=b}^{t-1}(\rho_i/q_i) V t = ∏ i = b t − 1 ( ρ i / q i ) . Then R t = R b P t ∏ i = b t − 1 q i R_t=R_bP_t\prod_{i=b}^{t-1}q_i R t = R b P t ∏ i = b t − 1 q i , and
P t + 1 − P t = a t ∥ ∇ f ( x t + 1 ) ∥ 2 2 q t R b ∏ i = b t − 1 q i ≤ 12 κ H b 2 ( 1 − ρ t q t ) V t = 12 κ H b 2 ( V t − V t + 1 ) , P_{t+1}-P_t=\frac{a_t\lVert\nabla f(x_{t+1})\rVert^2}{2q_tR_b\prod_{i=b}^{t-1}q_i}\le12\kappa H_b^2\left(1-\frac{\rho_t}{q_t}\right)V_t=12\kappa H_b^2(V_t-V_{t+1}), P t + 1 − P t = 2 q t R b ∏ i = b t − 1 q i a t ∥ ∇ f ( x t + 1 ) ∥ 2 ≤ 12 κ H b 2 ( 1 − q t ρ t ) V t = 12 κ H b 2 ( V t − V t + 1 ) ,
because 1 − ρ t / q t = a t / ( 2 q t ) 1-\rho_t/q_t=a_t/(2q_t) 1 − ρ t / q t = a t / ( 2 q t ) . Summation gives ∑ i = b t − 1 ℓ i = log P t ≤ log ( 1 + 12 κ H b 2 ) ≤ log ( 1 + 588 25 κ 2 ) \sum_{i=b}^{t-1}\ell_i=\log P_t\le\log(1+12\kappa H_b^2)\le\log(1+\frac{588}{25}\kappa^2) ∑ i = b t − 1 ℓ i = log P t ≤ log ( 1 + 12 κ H b 2 ) ≤ log ( 1 + 25 588 κ 2 ) . Since 4 ( log 5 2 κ ) 2 − 4 ( log 7 5 κ ) 2 = 4 log 25 14 ( log κ + log 7 2 ) 4(\log\frac{5}{2}\sqrt{\kappa})^2-4(\log\frac{7}{5}\sqrt{\kappa})^2=4\log\frac{25}{14}(\log\kappa+\log\frac{7}{2}) 4 ( log 2 5 κ ) 2 − 4 ( log 5 7 κ ) 2 = 4 log 14 25 ( log κ + log 2 7 ) exceeds log ( 1 + 588 25 κ 2 ) \log(1+\frac{588}{25}\kappa^2) log ( 1 + 25 588 κ 2 ) at κ = 4 \kappa=4 κ = 4 and grows in log κ \log\kappa log κ with slope 4 log 25 14 > 2 4\log\frac{25}{14}>2 4 log 14 25 > 2 , (16) gives H t ≤ 5 2 κ H_t\le\frac{5}{2}\sqrt{\kappa} H t ≤ 2 5 κ .
Step 3: the rate. If κ < 4 \kappa<4 κ < 4 , Lemma 3.1 gives f ( x k ) − f ⋆ ≤ e − k / κ ( f ( x 0 ) − f ⋆ ) ≤ e − k / ( 5 κ ) ( f ( x 0 ) − f ⋆ ) f(x_k)-f^\star\le e^{-k/\kappa}(f(x_0)-f^\star)\le e^{-k/(5\sqrt{\kappa})}(f(x_0)-f^\star) f ( x k ) − f ⋆ ≤ e − k / κ ( f ( x 0 ) − f ⋆ ) ≤ e − k / ( 5 κ ) ( f ( x 0 ) − f ⋆ ) . Let κ ≥ 4 \kappa\ge4 κ ≥ 4 and k ≥ 1 k\ge1 k ≥ 1 . Then log ( R i + 1 / R i ) = log q i + ℓ i \log(R_{i+1}/R_i)=\log q_i+\ell_i log ( R i + 1 / R i ) = log q i + ℓ i and H i log q i ≤ − 1 2 H_i\log q_i\le-\frac{1}{2} H i log q i ≤ − 2 1 . By (16) and H i ≤ t H_i\le t H i ≤ t on [ H i , H i + 1 ] [H_i,H_{i+1}] [ H i , H i + 1 ] ,
∑ i = 0 k − 1 H i ℓ i = ∑ i = 0 k − 1 ∫ H i H i + 1 8 H i log t t d t ≤ ∫ H 0 H k 8 log t d t ≤ 8 H k log H k . \sum_{i=0}^{k-1}H_i\ell_i=\sum_{i=0}^{k-1}\int_{H_i}^{H_{i+1}}\frac{8H_i\log t}{t}\,dt\le\int_{H_0}^{H_k}8\log t\,dt\le8H_k\log H_k. i = 0 ∑ k − 1 H i ℓ i = i = 0 ∑ k − 1 ∫ H i H i + 1 t 8 H i log t d t ≤ ∫ H 0 H k 8 log t d t ≤ 8 H k log H k .
Summation by parts, H i ≥ H i − 1 H_i\ge H_{i-1} H i ≥ H i − 1 , R i ≤ 2 L ( f ( x 0 ) − f ⋆ ) R_i\le2L(f(x_0)-f^\star) R i ≤ 2 L ( f ( x 0 ) − f ⋆ ) and R k ≥ 2 μ ( f ( x k ) − f ⋆ ) R_k\ge2\mu(f(x_k)-f^\star) R k ≥ 2 μ ( f ( x k ) − f ⋆ ) give
8 H k log H k − k 2 ≥ ∑ i = 0 k − 1 H i log R i + 1 R i ≥ H k log R k 2 L ( f ( x 0 ) − f ⋆ ) ≥ H k log f ( x k ) − f ⋆ κ ( f ( x 0 ) − f ⋆ ) , 8H_k\log H_k-\frac{k}{2}\ge\sum_{i=0}^{k-1}H_i\log\frac{R_{i+1}}{R_i}\ge H_k\log\frac{R_k}{2L(f(x_0)-f^\star)}\ge H_k\log\frac{f(x_k)-f^\star}{\kappa(f(x_0)-f^\star)}, 8 H k log H k − 2 k ≥ i = 0 ∑ k − 1 H i log R i R i + 1 ≥ H k log 2 L ( f ( x 0 ) − f ⋆ ) R k ≥ H k log κ ( f ( x 0 ) − f ⋆ ) f ( x k ) − f ⋆ ,
that is ( f ( x k ) − f ⋆ ) / ( f ( x 0 ) − f ⋆ ) ≤ κ H k 8 e − k / ( 2 H k ) (f(x_k)-f^\star)/(f(x_0)-f^\star)\le\kappa H_k^8e^{-k/(2H_k)} ( f ( x k ) − f ⋆ ) / ( f ( x 0 ) − f ⋆ ) ≤ κ H k 8 e − k / ( 2 H k ) , and H k ≤ 5 2 κ H_k\le\frac{5}{2}\sqrt{\kappa} H k ≤ 2 5 κ gives the claimed bound. Solving it for k k k gives N ( ϵ ) N(\epsilon) N ( ϵ ) . Only the pair enters the proof, so further directions are allowed, and as the pair consists of combinations of past gradients and steps, Proposition 3.2 applies to the SESOP variant. □
The prefactor is polynomial in κ \kappa κ , so (2) is not established, and we do not know if it can be removed. The exponent is 2.5 times smaller than for the schedule, and unlike μ \mu μ -estimates in restart schemes [26 , 27 ] , H k H_k H k only sets the forgetting factor, and Step 2 of the proof gives H k ≤ 5 2 κ H_k\le\frac{5}{2}\sqrt{\kappa} H k ≤ 2 5 κ for κ ≥ 4 \kappa\ge4 κ ≥ 4 .
Proof of Remark 4.11 (periodic attenuation) Put a = 1 / h a=1/h a = 1/ h and λ = L / h 2 ≤ μ \lambda=L/h^2\le\mu λ = L / h 2 ≤ μ , so that a 2 / λ = 1 / L a^2/\lambda=1/L a 2 / λ = 1/ L , and W b = f ( x b ) − f ⋆ + λ ∥ y b ∥ 2 + L − 1 ∥ z b ∥ 2 W_b=f(x_b)-f^\star+\lambda\lVert y_b\rVert^2+L^{-1}\lVert z_b\rVert^2 W b = f ( x b ) − f ⋆ + λ ∥ y b ∥ 2 + L − 1 ∥ z b ∥ 2 at the start b b b of a block. In the block π k = y k + 1 L z k \pi_k=y_k+\frac{1}{L}z_k π k = y k + L 1 z k lies in S k S_k S k and satisfies (3) , and ∥ x b − x ⋆ ∥ 2 ≤ 2 ( f ( x b ) − f ⋆ ) / μ \lVert x_b-x^\star\rVert^2\le2(f(x_b)-f^\star)/\mu ∥ x b − x ⋆ ∥ 2 ≤ 2 ( f ( x b ) − f ⋆ ) / μ , so Lemma 3.3 gives
f ( x b + T ) − f ⋆ ≤ ρ T − 1 ( f ( x b ) − f ⋆ + λ 2 ∥ x b − π b − x ⋆ ∥ 2 ) ≤ 4 e − 4 W b . f(x_{b+T})-f^\star\le\rho^{T-1}\left(f(x_b)-f^\star+\frac{\lambda}{2}\lVert x_b-\pi_b-x^\star\rVert^2\right)\le4e^{-4}W_b. f ( x b + T ) − f ⋆ ≤ ρ T − 1 ( f ( x b ) − f ⋆ + 2 λ ∥ x b − π b − x ⋆ ∥ 2 ) ≤ 4 e − 4 W b .
Since x k + 1 − y k + 1 = ( 1 − ρ ) x k + 1 + ρ ( x k − y k ) x_{k+1}-y_{k+1}=(1-\rho)x_{k+1}+\rho(x_k-y_k) x k + 1 − y k + 1 = ( 1 − ρ ) x k + 1 + ρ ( x k − y k ) , the vector x b + T − y b + T x_{b+T}-y_{b+T} x b + T − y b + T is a convex combination of x b − y b x_b-y_b x b − y b and x b + 1 , … , x b + T x_{b+1},\ldots,x_{b+T} x b + 1 , … , x b + T . All iterates of the block lie within 2 ( f ( x b ) − f ⋆ ) / μ \sqrt{2(f(x_b)-f^\star)/\mu} 2 ( f ( x b ) − f ⋆ ) / μ of x ⋆ x^\star x ⋆ , ∥ ∇ f ( x i ) ∥ 2 ≤ 2 L ( f ( x b ) − f ⋆ ) \lVert\nabla f(x_i)\rVert^2\le2L(f(x_b)-f^\star) ∥ ∇ f ( x i ) ∥ 2 ≤ 2 L ( f ( x b ) − f ⋆ ) and ∑ j ≥ 0 ρ j = h \sum_{j\ge0}\rho^j=h ∑ j ≥ 0 ρ j = h , so before the attenuation λ ∥ y b + T ∥ 2 ≤ 18 W b \lambda\lVert y_{b+T}\rVert^2\le18W_b λ ∥ y b + T ∥ 2 ≤ 18 W b and L − 1 ∥ z b + T ∥ 2 ≤ ( 2 + 4 h 2 ) W b L^{-1}\lVert z_{b+T}\rVert^2\le(2+4h^2)W_b L − 1 ∥ z b + T ∥ 2 ≤ ( 2 + 4 h 2 ) W b , and after it W b + T ≤ ( 4 e − 4 + 20 + 4 h 2 64 h 2 ) W b < 1 2 W b W_{b+T}\le\left(4e^{-4}+\frac{20+4h^2}{64h^2}\right)W_b<\frac12W_b W b + T ≤ ( 4 e − 4 + 64 h 2 20 + 4 h 2 ) W b < 2 1 W b . Since W 0 = f ( x 0 ) − f ⋆ W_0=f(x_0)-f^\star W 0 = f ( x 0 ) − f ⋆ and f ( x k ) f(x_k) f ( x k ) is nonincreasing, f ( x k ) − f ⋆ ≤ 2 − ⌊ k / T ⌋ ( f ( x 0 ) − f ⋆ ) f(x_k)-f^\star\le2^{-\lfloor k/T\rfloor}(f(x_0)-f^\star) f ( x k ) − f ⋆ ≤ 2 − ⌊ k / T ⌋ ( f ( x 0 ) − f ⋆ ) , and T ≤ 9 κ T\le9\sqrt{\kappa} T ≤ 9 κ gives the rate. Normalized weights grow by ρ − 1 \rho^{-1} ρ − 1 per step and β − 1 \beta^{-1} β − 1 per block, so Γ = ρ − 1 β − 1 / T \Gamma=\rho^{-1}\beta^{-1/T} Γ = ρ − 1 β − 1/ T . □
Inexact subproblems
Let the gradients and the filter updates be exact, fix γ ∈ [ 0 , 1 ] \gamma\in[0,1] γ ∈ [ 0 , 1 ] , put a k = 2 − ⌊ log 2 k ⌋ a_k=2^{-\lfloor\log_2 k\rfloor} a k = 2 − ⌊ l o g 2 k ⌋ for k ≥ 2 k\ge2 k ≥ 2 and a 0 = a 1 = 1 2 a_0=a_1=\frac12 a 0 = a 1 = 2 1 , write ν k = ∥ P S k ∇ f ( x k + 1 ) ∥ \nu_k=\lVert P_{S_k}\nabla f(x_{k+1})\rVert ν k = ∥ P S k ∇ f ( x k + 1 )∥ for the reduced gradient at the computed point, and replace (1) by
x k + 1 ∈ x k + S k , f ( x k ) − f ( x k + 1 ) ≥ ∥ ∇ f ( x k ) ∥ 2 2 L ′ , ν k 2 ≤ γ a k ∥ ∇ f ( x k ) ∥ 2 , (18) x_{k+1}\in x_k+S_k,\qquad f(x_k)-f(x_{k+1})\ge\frac{\lVert\nabla f(x_k)\rVert^2}{2L'},\qquad\nu_k^2\le\gamma a_k\lVert\nabla f(x_k)\rVert^2, \tag*{(18)} x k + 1 ∈ x k + S k , f ( x k ) − f ( x k + 1 ) ≥ 2 L ′ ∥ ∇ f ( x k ) ∥ 2 , ν k 2 ≤ γ a k ∥ ∇ f ( x k ) ∥ 2 , (18)
with ∇ f ( x k ) ∈ S k \nabla f(x_k)\in S_k ∇ f ( x k ) ∈ S k and a constant L ′ ≥ L L'\ge L L ′ ≥ L that enters only the analysis, and put κ ′ = L ′ / μ \kappa'=L'/\mu κ ′ = L ′ / μ . A monotone inner solver started at a gradient step with sufficient decrease satisfies the second condition, and the third is observable.
Paying the defect E k E_k E k of Lemma 3.3 from the decrease of f f f slows a pair only slightly.
Lemma 5.6 (decrease budget) In the setting of Lemma 3.3 let a k = a ∈ ( 0 , 1 ) a_k=a\in(0,1) a k = a ∈ ( 0 , 1 ) , ρ = 1 − a \rho=1-a ρ = 1 − a , and E k ≤ q ( f ( x k ) − f ( x k + 1 ) ) E_k\le q(f(x_k)-f(x_{k+1})) E k ≤ q ( f ( x k ) − f ( x k + 1 )) for k ≥ b k\ge b k ≥ b with q ≥ 0 q\ge0 q ≥ 0 . Put r q = 1 − a / ( 1 + a q ) r_q=1-a/(1+aq) r q = 1 − a / ( 1 + a q ) and β q = a ρ q / r q \beta_q=a\rho^q/r_q β q = a ρ q / r q . Then β q ≤ a q \beta_q\le aq β q ≤ a q and f ( x b + h ) − f ⋆ ≤ r q h − 1 ( C b + β q ( f ( x b ) − f ⋆ ) ) f(x_{b+h})-f^\star\le r_q^{h-1}(C_b+\beta_q(f(x_b)-f^\star)) f ( x b + h ) − f ⋆ ≤ r q h − 1 ( C b + β q ( f ( x b ) − f ⋆ )) for h ≥ 1 h\ge1 h ≥ 1 . *
Proof Put t = a ρ q t=a\rho^q t = a ρ q and U j = ρ j C b + e b + j U_j=\rho^jC_b+e_{b+j} U j = ρ j C b + e b + j . Lemma 3.3 gives f ( x b + j + 1 ) − f ⋆ ≤ U j f(x_{b+j+1})-f^\star\le U_j f ( x b + j + 1 ) − f ⋆ ≤ U j , and the recursion of e k e_k e k gives
U j + 1 = ρ U j + a ρ E b + j ≤ ρ U j + t ( f ( x b + j ) − f ( x b + j + 1 ) ) . U_{j+1}=\rho U_j+a\rho E_{b+j}\le\rho U_j+t(f(x_{b+j})-f(x_{b+j+1})). U j + 1 = ρ U j + a ρ E b + j ≤ ρ U j + t ( f ( x b + j ) − f ( x b + j + 1 )) .
Since ρ ≤ r q ≤ 1 \rho\le r_q\le1 ρ ≤ r q ≤ 1 , we have t = r q β q ≤ β q ≤ t / ρ = a q t=r_q\beta_q\le\beta_q\le t/\rho=aq t = r q β q ≤ β q ≤ t / ρ = a q , and β q − t = β q ( 1 − r q ) ≤ a q 2 / ( 1 + a q ) = r q − ρ \beta_q-t=\beta_q(1-r_q)\le aq^2/(1+aq)=r_q-\rho β q − t = β q ( 1 − r q ) ≤ a q 2 / ( 1 + a q ) = r q − ρ . For V j = U j + β q ( f ( x b + j ) − f ⋆ ) V_j=U_j+\beta_q(f(x_{b+j})-f^\star) V j = U j + β q ( f ( x b + j ) − f ⋆ ) these relations and f ( x b + j + 1 ) − f ⋆ ≤ U j f(x_{b+j+1})-f^\star\le U_j f ( x b + j + 1 ) − f ⋆ ≤ U j give
V j + 1 ≤ ρ U j + t ( f ( x b + j ) − f ⋆ ) + ( β q − t ) ( f ( x b + j + 1 ) − f ⋆ ) ≤ ( ρ + β q − t ) U j + t ( f ( x b + j ) − f ⋆ ) ≤ r q V j . \begin{aligned}
V_{j+1}&\le\rho U_j+t(f(x_{b+j})-f^\star)+(\beta_q-t)(f(x_{b+j+1})-f^\star)\\
&\le(\rho+\beta_q-t)U_j+t(f(x_{b+j})-f^\star)\le r_qV_j.
\end{aligned} V j + 1 ≤ ρ U j + t ( f ( x b + j ) − f ⋆ ) + ( β q − t ) ( f ( x b + j + 1 ) − f ⋆ ) ≤ ( ρ + β q − t ) U j + t ( f ( x b + j ) − f ⋆ ) ≤ r q V j .
With f ( x b + h ) − f ⋆ ≤ U h − 1 ≤ V h − 1 f(x_{b+h})-f^\star\le U_{h-1}\le V_{h-1} f ( x b + h ) − f ⋆ ≤ U h − 1 ≤ V h − 1 and V 0 = C b + β q ( f ( x b ) − f ⋆ ) V_0=C_b+\beta_q(f(x_b)-f^\star) V 0 = C b + β q ( f ( x b ) − f ⋆ ) this gives the bound. □
Proposition 5.7 (inexact activation schedule) Let f ∈ F μ , L f\in\mathcal{F}_{\mu,L} f ∈ F μ , L with f ( x 0 ) − f ⋆ > 0 f(x_0)-f^\star>0 f ( x 0 ) − f ⋆ > 0 , and let the iterates of Algorithm 1 satisfy (18) . Then f ( x k + 1 ) − f ⋆ ≤ ( 2 + γ ) e 1 / ( 1 + γ ) exp ( − k / ( 2 ( 1 + γ ) κ ′ ) ) ( f ( x 0 ) − f ⋆ ) f(x_{k+1})-f^\star\le(2+\gamma)e^{1/(1+\gamma)}\exp\left(-k/(2(1+\gamma)\sqrt{\kappa'})\right)(f(x_0)-f^\star) f ( x k + 1 ) − f ⋆ ≤ ( 2 + γ ) e 1/ ( 1 + γ ) exp ( − k / ( 2 ( 1 + γ ) κ ′ ) ) ( f ( x 0 ) − f ⋆ ) for k ≥ 0 k\ge0 k ≥ 0 , also with additional directions in S k S_k S k .
Proof Let a pair of level J J J with a = 2 − J a = 2^{-J} a = 2 − J and λ = a 2 L ′ ≤ μ \lambda= a^2L' \le\mu λ = a 2 L ′ ≤ μ lie in S k S_k S k . As in Lemma 5.3, its virtual momentum π k \pi_k π k lies in S k S_k S k , and s k + 1 ∈ S k s_{k+1} \in S_k s k + 1 ∈ S k by (18) , so π k + s k + 1 ∈ S k \pi_k + s_{k+1} \in S_k π k + s k + 1 ∈ S k . Lemma 3.3 and (18) give
E k ≤ ν k 2 2 λ ≤ γ a k ∥ ∇ f ( x k ) ∥ 2 2 a 2 L ′ ≤ γ a k a 2 ( f ( x k ) − f ( x k + 1 ) ) . E_k \le\frac{\nu_k^2}{2\lambda} \le\frac{\gamma a_k\lVert\nabla f(x_k)\rVert^2}{2a^2L'} \le\frac{\gamma a_k}{a^2}\left(f(x_k)-f(x_{k+1})\right). E k ≤ 2 λ ν k 2 ≤ 2 a 2 L ′ γ a k ∥ ∇ f ( x k ) ∥ 2 ≤ a 2 γ a k ( f ( x k ) − f ( x k + 1 ) ) .
If a k ≤ a a_k \le a a k ≤ a , Lemma 5.6 applies with q = γ / a q = \gamma/a q = γ / a , so that a q = γ aq = \gamma a q = γ . A pair created at b b b has π b = 0 \pi_b = 0 π b = 0 and C b ≤ 2 ( f ( x b ) − f ⋆ ) C_b \le2(f(x_b)-f^\star) C b ≤ 2 ( f ( x b ) − f ⋆ ) , hence
f ( x b + h ) − f ⋆ ≤ ( 2 + γ ) ( 1 − a 1 + γ ) h − 1 ( f ( x b ) − f ⋆ ) ( h ≥ 1 ) . (19) f(x_{b+h}) - f^\star\le(2+\gamma)\left(1-\frac{a}{1+\gamma}\right)^{h-1}(f(x_b)-f^\star) \qquad(h \ge1). \tag*{(19)} f ( x b + h ) − f ⋆ ≤ ( 2 + γ ) ( 1 − 1 + γ a ) h − 1 ( f ( x b ) − f ⋆ ) ( h ≥ 1 ) . (19)
Now let J = max { 1 , ⌈ log 4 κ ′ ⌉ } J = \max\{1,\lceil\log_4\kappa'\rceil\} J = max { 1 , ⌈ log 4 κ ′ ⌉} , so that λ = a 2 L ′ ≤ μ \lambda= a^2L' \le\mu λ = a 2 L ′ ≤ μ and a ≥ 1 / ( 2 κ ′ ) a \ge1/(2\sqrt{\kappa'}) a ≥ 1/ ( 2 κ ′ ) . The pair is created at b = 2 J = 1 / a b = 2^J = 1/a b = 2 J = 1/ a , and a k ≤ a a_k \le a a k ≤ a for k ≥ b k \ge b k ≥ b . For k ≥ b k \ge b k ≥ b , (19) , f ( x b ) ≤ f ( x 0 ) f(x_b) \le f(x_0) f ( x b ) ≤ f ( x 0 ) and a b = 1 ab = 1 ab = 1 give
f ( x k + 1 ) − f ⋆ ≤ ( 2 + γ ) e ( 1 − a k ) / ( 1 + γ ) ( f ( x 0 ) − f ⋆ ) , f(x_{k+1}) - f^\star\le(2+\gamma)e^{(1-a_k)/(1+\gamma)}(f(x_0)-f^\star), f ( x k + 1 ) − f ⋆ ≤ ( 2 + γ ) e ( 1 − a k ) / ( 1 + γ ) ( f ( x 0 ) − f ⋆ ) ,
and a ≥ 1 / ( 2 κ ′ ) a \ge1/(2\sqrt{\kappa'}) a ≥ 1/ ( 2 κ ′ ) gives the claim. For k < b k < b k < b the right-hand side exceeds ( 2 + γ ) ( f ( x 0 ) − f ⋆ ) (2+\gamma)(f(x_0)-f^\star) ( 2 + γ ) ( f ( x 0 ) − f ⋆ ) , and monotonicity suffices.
Unlike [11 ] , where the gradients are inexact, here the defect of Lemma 3.3 is charged to the actual decrease, so the stopping rule is observable: at the matching level the relative residual may be of order κ ′ − 1 / 4 \kappa'^{-1/4} κ ′ − 1/4 , and the tolerance decreases like k − 1 / 2 k^{-1/2} k − 1/2 . Sufficient decrease alone certifies only the rate 1 − 1 / κ ′ 1-1/\kappa' 1 − 1/ κ ′ of the gradient step. For γ = 0 \gamma= 0 γ = 0 and L ′ = L L' = L L ′ = L the proposition reduces to Theorem 5.4 .
Numerical experiments
The experiments address two questions. Does SESOP, run as a black box, follow the slow trajectories of § 4 , and what do the guarantees of § 5 cost outside the worst case? All experiments use Python 3.13 with NumPy 2.5 and SciPy 1.18. Times are single runs with one BLAS thread, so differences of a few percent are insignificant.
Hard instances
For SESOP with one aggregate, the linear weights and κ ∈ [ 25 , 3000 ] \kappa\in[25, 3000] κ ∈ [ 25 , 3000 ] we computed a five-dimensional variant of the construction, with two explicit initial points instead of the orthogonal segment, and the methods evaluate its interpolant (Lemma 3.4 ) by a quadratic program over the simplex. The conditions of the proof of Lemma 4.7 are solved by backward substitution for h n h_n h n , 2 ≤ n ≤ N 2 \le n \le N 2 ≤ n ≤ N , with N N N from 150 to 8000, beyond every run, and ϑ \vartheta ϑ from 10 − 9 10^{-9} 1 0 − 9 to 10 − 3 10^{-3} 1 0 − 3 , with residual below 10 − 14 10^{-14} 1 0 − 14 . All interpolation residuals are positive, the smallest relative one between 6 ⋅ 10 − 9 6 \cdot10^{-9} 6 ⋅ 1 0 − 9 and 1.3 ⋅ 10 − 2 1.3 \cdot10^{-2} 1.3 ⋅ 1 0 − 2 . The data are not certified in exact arithmetic, so the runs only illustrate Theorem 4.2 .
For κ ≥ 50 \kappa\ge50 κ ≥ 50 double precision cannot resolve the first line search from f 0 = 12 ϑ − 2 > 10 8 f_0 = 12\vartheta^{-2} > 10^8 f 0 = 12 ϑ − 2 > 1 0 8 , so the runs in Figure 1 start at x 2 x_2 x 2 , SESOP with its state after two iterations and the filter methods with empty memory. SESOP follows the certified trajectory with relative deviation below 4 ⋅ 10 − 9 4 \cdot10^{-9} 4 ⋅ 1 0 − 9 and the rate − 2 log ( 1 − 3 / κ ) -2\log(1-3/\kappa) − 2 log ( 1 − 3/ κ ) to thirteen digits. Its counts grow like κ 1.01 \kappa^{1.01} κ 1.01 , those of gradient descent like κ 1.02 \kappa^{1.02} κ 1.02 and those of Nesterov’s method like κ 0.55 \kappa^{0.55} κ 0.55 . The activation schedule and the smooth pair leave the certified path at once and need 6 and 4 iterations, which says nothing about their worst case.
Figure 1. Black-box runs on the SESOP instances from x 2 x_2 x 2 . Left: κ = 200 \kappa= 200 κ = 200 , SESOP covers the certified slow trajectory (dashed). Right: iterations to reduce the gap by 10 6 10^6 1 0 6 against κ \kappa κ . Slopes fit iterations ≈ c κ s \approx c\kappa^s ≈ c κ s by least squares. Gradient descent uses the step 1 / L 1/L 1/ L , Nesterov’s method the true μ \mu μ and L L L .
Lemma 4.12 predicts the factor r 2 r^2 r 2 per step. The plane rule of Proposition 4.13 keeps this factor to twelve digits for 3000 iterations, and SESOP on the instances of Proposition 4.14 with Nemirovski’s and the linear weights, K ∈ { 30 , 100 , 300 , 1000 } K \in\{30, 100, 300, 1000\} K ∈ { 30 , 100 , 300 , 1000 } and θ = 10 − 8 \theta= 10^{-8} θ = 1 0 − 8 , keeps it to six digits for 1500 iterations after the cold start. So SESOP and the plane rule stay on the slow trajectories in floating point.
Logistic regression and capillary energies
We use regularized logistic regression f ( x ) = 1 ℓ ∑ i = 1 ℓ log ( 1 + e − y i ⟨ a i , x ⟩ ) + μ 2 ∥ x ∥ 2 f(x) = \frac{1}{\ell}\sum_{i=1}^{\ell}\log(1+e^{-y_i\langle a_i,x\rangle})+\frac{\mu}{2}\lVert x\rVert^2 f ( x ) = ℓ 1 ∑ i = 1 ℓ log ( 1 + e − y i ⟨ a i , x ⟩ ) + 2 μ ∥ x ∥ 2 with ℓ = 2000 \ell= 2000 ℓ = 2000 , n = 1000 n = 1000 n = 1000 , a Gaussian data matrix whose singular values decay log-uniformly over two decades, labels from a noisy linear model, and L = 1 L = 1 L = 1 . A harder test is the capillary-surface problem [39 ] : a liquid surface u u u over Ω = ( 0 , 1 ) d \Omega= (0,1)^d Ω = ( 0 , 1 ) d minimizes its area plus the energy of gravity and of a load q q q , E ( u ) = ∫ Ω ( 1 + ∥ ∇ u ∥ 2 − q u ) d x E(u) = \int_{\Omega}(\sqrt{1+\lVert\nabla u\rVert^2}-qu)\,dx E ( u ) = ∫ Ω ( 1 + ∥ ∇ u ∥ 2 − q u ) d x over u ∈ H 0 1 ( Ω ) u \in H^1_0(\Omega) u ∈ H 0 1 ( Ω ) . The gravity term makes E E E strongly convex, and the curvature ( 1 + t 2 ) − 3 / 2 (1+t^2)^{-3/2} ( 1 + t 2 ) − 3/2 of the area integrand degenerates where the surface is steep, so under a strong load the Hessian varies by orders of magnitude. For d = 1 d = 1 d = 1 we use finite differences on n n n interior points, so that κ ≤ 4 ( n + 1 ) 2 + 1 \kappa\le4(n+1)^2+1 κ ≤ 4 ( n + 1 ) 2 + 1 , and q ( x ) = Q ( sin 3 π x + 1 2 sign sin 7 π x + 3 10 1 { x > 3 / 10 } ) q(x) = Q(\sin3\pi x+\frac{1}{2}\operatorname{sign}\sin7\pi x+\frac{3}{10}\mathbf{1}_{\{x>3/10\}}) q ( x ) = Q ( sin 3 π x + 2 1 sign sin 7 π x + 10 3 1 { x > 3/10 } ) . All methods start at zero, and f ⋆ f^\star f ⋆ is computed by Newton’s method.
Nesterov’s method (AGD) uses μ \mu μ and L L L , its restarted versions [4 ] needed between 0.65 and 1.9 times its iterations, and L-BFGS is the SciPy code with memory 10 [37 , 40 ] . SESOP uses one aggregate with Nemirovski’s weights, and the window holds the 8 most recent pairs ( ∇ f ( x i ) , s i + 1 ) (\nabla f(x_i),s_{i+1}) ( ∇ f ( x i ) , s i + 1 ) and the current gradient. All subspace methods share one damped Newton solver, here run to reduced gradient 10 − 10 ∥ ∇ f ( x k ) ∥ 10^{-10}\lVert\nabla f(x_k)\rVert 1 0 − 10 ∥ ∇ f ( x k )∥ .
On logistic regression all subspace methods behave like accelerated methods, and their counts grow much more slowly than κ \sqrt{\kappa} κ , since the condition number near the solution is far below the nominal one (Table 2 ). The methods with guarantees need at most 10 percent more iterations than the best one. The horizon of the smooth pair stays below 5 there, and on the capillary energy with n = 200 n = 200 n = 200 below 0.09 κ 0.09\sqrt{\kappa} 0.09 κ , with κ \kappa κ of the Hessian at the solution. On the capillary energy the activation schedule is the fastest method or within 0.2 percent of it for every load, with a search space of dimension about 20 against 4 for SESOP, and the SESOP directions remove most of the loss of the smooth pair, which alone needs up to 2.3 times more iterations. On these problems the guarantee of the schedule thus costs almost nothing.
Table 2. Iterations to reduce f ( x k ) − f ⋆ f(x_k)-f^\star f ( x k ) − f ⋆ by 10 10 10^{10} 1 0 10 (logistic) and 10 8 10^{8} 1 0 8 (capillary, d = 1 d=1 d = 1 , n = 400 n=400 n = 400 ).
Cost of an iteration and inexact subproblems
For f ( x ) = ∑ i = 1 ℓ φ i ( ⟨ a i , x ⟩ ) + μ 2 ∥ x ∥ 2 f(x)=\sum_{i=1}^{\ell}\varphi_i(\langle a_i,x\rangle)+\frac{\mu}{2}\lVert x\rVert^2 f ( x ) = ∑ i = 1 ℓ φ i (⟨ a i , x ⟩) + 2 μ ∥ x ∥ 2 the reduced Hessian needs only the images A D k AD_k A D k of the stored directions, and if they follow the same recursions, a step costs one product with A A A and one with A ⊤ A^\top A ⊤ [9 , 19 ] . We timed logistic regression with ℓ = 10000 \ell=10000 ℓ = 10000 , n = 2000 n=2000 n = 2000 , κ = 10 5 \kappa=10^5 κ = 1 0 5 and inner tolerance 10 − 4 ∥ ∇ f ( x k ) ∥ 10^{-4}\lVert\nabla f(x_k)\rVert 1 0 − 4 ∥ ∇ f ( x k )∥ , and the capillary energy on ( 0 , 1 ) 2 (0,1)^2 ( 0 , 1 ) 2 with Q = 40 Q=40 Q = 40 in 64 2 64^2 6 4 2 Galerkin sine modes, that is Φ ( A c ) \Phi(Ac) Φ ( A c ) with a dense 19522 × 4096 19522\times4096 19522 × 4096 matrix A A A and κ ≈ 2.4 ⋅ 10 4 \kappa\approx2.4\cdot10^4 κ ≈ 2.4 ⋅ 1 0 4 . To reduce the gap by 10 10 10^{10} 1 0 10 on logistic regression the subspace methods needed 264 to 280 iterations and 6.4 to 7.9 seconds, L-BFGS 305 iterations and 10.1 seconds, and AGD with or without gradient restart about 3700 iterations and 90 seconds. On the capillary energy the subspace methods took 668 to 863 iterations and 67 to 82 seconds, L-BFGS 121 seconds and the AGD variants 133 to 201 seconds. The subspace methods were thus the fastest in both tests.
With the rule of Proposition 5.7 and γ = 1 \gamma=1 γ = 1 , Newton steps from the dyadic step until ν k 2 ≤ a k ∥ ∇ f ( x k ) ∥ 2 \nu_k^2\leq a_k\lVert\nabla f(x_k)\rVert^2 ν k 2 ≤ a k ∥ ∇ f ( x k ) ∥ 2 , the activation schedule needs 1 to 1.5 Newton steps per iteration on logistic regression (κ = 10 4 , 10 6 \kappa=10^4,10^6 κ = 1 0 4 , 1 0 6 ) and the capillary energy (n = 200 n=200 n = 200 , Q = 40 Q=40 Q = 40 ), 5 to 9 times fewer than with exact subproblems, and never more outer iterations.
Conclusion
We asked which subspaces built from the history contain the momentum of an accelerated method when κ \kappa κ is unknown. For memories fixed in advance the answer is a time scale. They need a window of length Ω ( κ ) \Omega(\sqrt{\kappa}) Ω ( κ ) or a forgetting index Ω ( κ − 1 / 2 ) \Omega(\kappa^{-1/2}) Ω ( κ − 1/2 ) , which excludes SESOP, the plane form of conjugate gradients and GD-ELS, also on smooth functions, and with one pair and a bounded window the scale is 1 − ρ = Θ ( κ − 1 / 2 ) 1-\rho=\Theta(\kappa^{-1/2}) 1 − ρ = Θ ( κ − 1/2 ) . This scale can be reached without κ \kappa κ : filter pairs created at fixed times are uniformly accelerated, and a single adaptive pair is accelerated up to a polynomial prefactor. We do not know whether SESOP is slow at moderate accuracy or whether an adapted constant memory can be uniformly accelerated. On our test problems the slow regime did not appear, as expected: the tails start after a κ \kappa κ -dependent reduction of the gap, and for logistic regression the local condition number is far below the nominal one.
Declarations
Funding. The research was supported by Russian Science Foundation (project No. 21-71-30005-П), https://rscf.ru/en/project/21-71-30005/ . Competing interests. The authors declare no competing interests. Data availability. No external data were used. All test problems are generated by the code. Code availability. The code for all experiments, the computed results and instructions are available on Zenodo, https://doi.org/10.5281/zenodo.23916709 . Author contributions. I. Kuruzov and A. Gasnikov conceived and formulated the problem. I. Kuruzov obtained the main theoretical results, wrote the manuscript and prepared the presentation of the results. I. Dorofeev and A. Vyguzov carried out the numerical experiments and contributed to the auxiliary results. All authors read and approved the final manuscript. Use of AI tools. The authors verified every proof independently. An AI assistant (Claude, Anthropic) was also used to write the code for the experiments, to recheck the proofs, and to help with writing. The authors are fully responsible for the paper.