Use of language models. The models Codex Astra 6 and Claude Opus 5.5 were used in obtaining the theoretical results of this work. They were also used to prepare and typeset the manuscript and to translate the original text into English. Nevertheless, every line of the paper, including every statement, every proof, and every computation, has been personally edited and checked by the authors. The authors bear sole responsibility for the correctness of all mathematical statements, proofs, and conclusions.
Introduction
We consider the problem of minimizing a finite sum
f ( x ) = 1 n ∑ i = 1 n f i ( x ) → min x ∈ R d . f(x)=\frac{1}{n}\sum_{i=1}^{n}f_i(x)\to\min_{x\in\mathbb{R}^d}. f ( x ) = n 1 i = 1 ∑ n f i ( x ) → x ∈ R d min .
Problems of this form arise in empirical risk minimization in machine learning, where f i f_i f i is the loss on the i i i -th example of the training set [31 , 6 ] . In federated and distributed learning, the component f i f_i f i is the loss on the data of the i i i -th device or computing node [24 ] . The gradient of a single component is often much cheaper to compute than the full gradient. Therefore, we measure the complexity of methods by the number of calls to an oracle that returns the value and the gradient of a single component.
For a sum of L L L -smooth functions, variance reduction methods [16 , 38 ] make it possible to reduce the number of oracle calls. They periodically compute the full gradient of f f f , and at the remaining steps they use the gradients of individual components f i f_i f i . Lower bounds for arbitrary randomized methods show which dependencies on the number of components and on the accuracy cannot be improved [37 ] .
However, the L L L -smoothness condition fails, for example, for exponential loss functions. The ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness condition was introduced by Zhang et al. [41 ] to explain the success of methods with gradient clipping. When training an LSTM-based language model, the authors observed that the local smoothness constant grows together with the gradient norm. A similar dependence was also found when training transformers [10 ] . The ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness condition admits precisely this kind of curvature growth. Its generalizations and first-order methods for such functions were studied in [19 , 14 , 36 , 34 ] . For finite sums, one has to account simultaneously for the number of components and for the growth of curvature. For example, in Poisson regression with the log link (Stan Development Team) f i ( x ) = exp ( a i T x ) − b i a i T x f_i(x) = \exp(a_i^\mathsf{T}x) - b_i a_i^\mathsf{T}x f i ( x ) = exp ( a i T x ) − b i a i T x , where b i ≥ 0 b_i \ge0 b i ≥ 0 , the Hessian is unbounded on the whole space, but
∥ ∇ 2 f i ( x ) ∥ ≤ b i ∥ a i ∥ 2 + ∥ a i ∥ ∥ ∇ f i ( x ) ∥ . \lVert\nabla^2 f_i(x)\rVert\le b_i\lVert a_i\rVert^2 + \lVert a_i\rVert\lVert\nabla f_i(x)\rVert. ∥ ∇ 2 f i ( x )∥ ≤ b i ∥ a i ∥ 2 + ∥ a i ∥ ∥ ∇ f i ( x )∥ .
In phase retrieval from intensity measurements, one uses the losses f i ( x ) = ( ⟨ a i , x ⟩ 2 − b i ) 2 f_i(x) = (\langle a_i,x\rangle^2 - b_i)^2 f i ( x ) = (⟨ a i , x ⟩ 2 − b i ) 2 [7 ] . Here, too, one has to minimize a finite sum of components with unbounded curvature. Another example is the exponential loss in classification [14 Example 1.5] . In all these problems, when the number of observations is large, computing the full gradient may be expensive, and the gradient itself is not globally Lipschitz. Therefore, one needs methods that account for the growth of curvature and do not compute the gradients of all components at every step. Below we assume that f f f attains its minimum, and that the distance to the minimizer, the initial optimality gap, and the heterogeneity of the components are bounded.
The peculiarity of finite sums. In the problem under consideration, each component satisfies the inequality
0 ⪯ ∇ 2 f i ( x ) ⪯ ( L 0 , i + L 1 ∥ ∇ f i ( x ) ∥ ) I . 0 \preceq\nabla^2 f_i(x) \preceq(L_{0,i} + L_1\lVert\nabla f_i(x)\rVert)I. 0 ⪯ ∇ 2 f i ( x ) ⪯ ( L 0 , i + L 1 ∥ ∇ f i ( x )∥) I .
Even at a minimizer of the average, the gradients of individual components can be large: the equality ∇ f ( x ∗ ) = 0 \nabla f(x^*) = 0 ∇ f ( x ∗ ) = 0 only means that their sum is zero. Therefore, a small norm of the full gradient does not imply small smoothness constants of the components, on which the bounds of variance reduction methods depend. What is the oracle complexity of minimizing a finite sum in this situation? We answer this question in this work.
Notation. We denote by ε > 0 \varepsilon> 0 ε > 0 the required accuracy of the solution in function value: the method must output a point x ^ \hat{x} x ^ such that f ( x ^ ) − f ∗ ≤ ε f(\hat{x}) - f^* \le\varepsilon f ( x ^ ) − f ∗ ≤ ε . Throughout, ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ denotes the Euclidean norm of a vector and the spectral norm of a matrix, [ x , y ] [x,y] [ x , y ] is the segment with endpoints x , y ∈ R d x, y \in\mathbb{R}^{d} x , y ∈ R d , and B ( x , r ) B(x,r) B ( x , r ) is the closed ball of radius r r r centered at x x x . We denote the sum of the first k k k terms of a sequence by the corresponding capital letter: for example, A k = a 1 + ⋯ + a k A_k = a_1 + \cdots+ a_k A k = a 1 + ⋯ + a k and A 0 = 0 A_0 = 0 A 0 = 0 .
For randomized methods, the stated accuracy is required in expectation. All projections are Euclidean; Π C \Pi_C Π C denotes the projection onto a nonempty closed convex set C C C .
Main result. Let the components be convex, twice continuously differentiable, and bounded from below, and let n − 1 ∑ i L 0 , i ≤ L 0 n^{-1}\sum_i L_{0,i} \le L_0 n − 1 ∑ i L 0 , i ≤ L 0 . We assume that the starting point x 0 x_0 x 0 is zero: this can be achieved by a shift, with respect to which the problem class is invariant. We assume that f f f attains its minimum at a point x ∗ x^* x ∗ with ∥ x ∗ ∥ ≤ R \lVert x^* \rVert\le R ∥ x ∗ ∥ ≤ R and f ( 0 ) − f ∗ ≤ Δ f(0)-f^* \le\Delta f ( 0 ) − f ∗ ≤ Δ . Under the conditions of Theorem 2.9 , the upper and lower complexity bounds are, up to logarithmic factors, of order
n ( 1 + ( L 1 R ) 2 / 3 ) + n ( L 0 + L 1 2 δ ) R 2 ε + n ( L 1 R ) 2 . n(1+(L_1R)^{2/3})+\sqrt{\frac{n(L_0+L_1^2\delta)R^2}{\varepsilon}+n(L_1R)^2}. n ( 1 + ( L 1 R ) 2/3 ) + ε n ( L 0 + L 1 2 δ ) R 2 + n ( L 1 R ) 2 .
Here, complexity means the worst-case expected number of queries of values and gradients of individual components that suffices to guarantee E [ f ( x ^ ) − f ∗ ] ≤ ε \mathbb{E}[f(\hat{x})-f^*] \le\varepsilon E [ f ( x ^ ) − f ∗ ] ≤ ε . The lower bound holds for arbitrary randomized methods; the dimension of the problem is not bounded. The precise definition of the class and the logarithmic factors are given in § 2 .
Note on the logarithmic factors. In the upper bound, the hidden factor is log 2 N \log^2 N log 2 N , where N N N from (5) grows linearly in Δ / ε \Delta/\varepsilon Δ/ ε . In the worst case, when Δ \Delta Δ is exponentially large in L 1 R L_1R L 1 R , this logarithm can be of order L 1 R L_1R L 1 R , so the factor log 2 N \log^2 N log 2 N can be of order ( L 1 R ) 2 (L_1R)^2 ( L 1 R ) 2 , that is, it can give an additional polynomial factor.
The term n L 0 R 2 / ε \sqrt{nL_0R^2/\varepsilon} n L 0 R 2 / ε is known from the theory of minimizing a sum of L L L -smooth functions ([37 ] ). The term n ( L 1 R ) 2 / 3 n(L_1R)^{2/3} n ( L 1 R ) 2/3 reflects the number of oracle calls required to approach the minimum from the region where the gradients are exponentially large. The term
n L 1 2 R 2 δ ε + n ( L 1 R ) 2 \sqrt{\frac{nL_1^2R^2\delta}{\varepsilon}+n(L_1R)^2} ε n L 1 2 R 2 δ + n ( L 1 R ) 2
arises because the curvature of the components near the minimum is determined by their gradients, which can remain large even at the minimizer of the average.
Method and experiments. The upper bound is attained by the method YETI. When the local curvature is small, it performs an accelerated step, solving the auxiliary subproblem by SVRG, and when it is large, it projects the auxiliary point onto a halfspace containing the minimizer. In addition, YETI uses a heuristic with a bounded number of oracle queries: it makes it possible to take advantage of good proposed points without worsening the theoretical bound. In the experiments, YETI is compared with other methods, and its stopping criterion is tested.
Relation to known methods. YETI uses the accelerated proximal scheme of Monteiro–Svaiter ([25 ] ) and the idea of adaptive choice of regularization without solving an implicit equation at every step ([9 ] ). The local subproblem is solved by a projected variant of SVRG. Acceleration via approximate solution of regularized subproblems was also studied in the Catalyst scheme ([20 , 21 ] ) and in Frostig et al. ((2015)). In YETI, the choice of the step depends on the computed curvature estimate of the sum: if this estimate is too large, the method constructs a separating halfspace instead of a step. The complexity bound accounts for the queries of both the local solver and the heuristic.
The lower bound relies on Yao’s minimax principle Yao, [39 ] : a hard instance is chosen at random from a finite family, after which we average over the random variables of the algorithm. Hidden coordinates and the distribution of information across different components continue the known approach to constructing lower bounds for finite-sum minimization Woodworth and Srebro, [37 ] . For the class under consideration, we apply exponential transformations of convex chains and choose their parameters so that the constraints on L 1 L_1 L 1 , δ \delta δ , and the initial optimality gap are satisfied.
The present paper continues our previous work Rudomanov et al. [30 ] , where the oracle complexity of this class was found for a single function (n = 1 n = 1 n = 1 ): its order is L 0 R 2 / ε + L 1 R \sqrt{L_0R^2/\varepsilon} + L_1R L 0 R 2 / ε + L 1 R , and for a small initial optimality gap it is L 0 R 2 / ε + ( L 1 R ) 2 / 3 \sqrt{L_0R^2/\varepsilon} + (L_1R)^{2/3} L 0 R 2 / ε + ( L 1 R ) 2/3 . Here we consider a finite sum, and the oracle returns the value and the gradient of only one component; therefore, to obtain the upper bound, we use variance reduction. The results for a single function Vankov et al., [36 ] ; Tyurin [35 ] concern an oracle that computes the full gradient. In particular, the optimality of the method of Tyurin [35 ] for sufficiently small ε \varepsilon ε is based on the classical lower bound for the smooth subclass and concerns only the leading term L 0 R 2 / ε \sqrt{L_0R^2/\varepsilon} L 0 R 2 / ε . The optimality of the term n ( L 1 R ) 2 / 3 n(L_1R)^{2/3} n ( L 1 R ) 2/3 for an oracle returning the gradient of a single component does not follow from it. Similarly, the bounds for stochastic methods in the case of a common minimizer of all components Gorbunov et al., [14 ] do not cover an arbitrary positive heterogeneity δ \delta δ .
Organization of the paper. The problem setting and the main result are given in § 2 , the method YETI and its analysis in § 3 –§ 6 , the main experiments in § 7 (further experiments and their details are in Appendix § 18 ), and the lower bound in § 8 . The main text contains the results and a proof sketch of the lower bound, while auxiliary proofs and variants of the method are deferred to the appendices. A summary table of known bounds is given in Appendix § 19 .
Problem setting and main bounds
We consider the function f = n − 1 ∑ i = 1 n f i f = n^{-1}\sum_{i=1}^{n} f_i f = n − 1 ∑ i = 1 n f i on R d \mathbb{R}^d R d . We are given n ≥ 2 n \ge2 n ≥ 2 , L 0 , δ ≥ 0 L_0,\delta\ge0 L 0 , δ ≥ 0 , and L 1 , R , Δ , ε > 0 L_1, R, \Delta, \varepsilon> 0 L 1 , R , Δ , ε > 0 . We assume that the starting point is zero; this can be achieved by a shift of coordinates. All general assumptions are collected below.
Assumption 2.1 (components).
The functions f i : R d → R f_i:\mathbb{R}^d \to\mathbb{R} f i : R d → R , 1 ≤ i ≤ n 1 \le i \le n 1 ≤ i ≤ n , are convex, twice continuously differentiable, and bounded from below.
Assumption 2.2 (generalized smoothness).
Numbers L 0 , i ≥ 0 L_{0,i} \ge0 L 0 , i ≥ 0 are known such that
∥ ∇ 2 f i ( x ) ∥ ≤ L 0 , i + L 1 ∥ ∇ f i ( x ) ∥ ( x ∈ R d , 1 ≤ i ≤ n ) , \lVert\nabla^2 f_i(x) \rVert\le L_{0,i} + L_1 \lVert\nabla f_i(x) \rVert\qquad(x \in\mathbb{R}^d,\ 1 \le i \le n), ∥ ∇ 2 f i ( x )∥ ≤ L 0 , i + L 1 ∥ ∇ f i ( x )∥ ( x ∈ R d , 1 ≤ i ≤ n ) , 1 n ∑ i = 1 n L 0 , i ≤ L 0 . (1) \frac{1}{n}\sum_{i=1}^{n} L_{0,i} \le L_0. \tag*{(1)} n 1 i = 1 ∑ n L 0 , i ≤ L 0 . (1)
Assumption 2.3 (minimizer).
There exists a minimizer x ∗ ∈ arg min f x^* \in\operatorname*{arg\,min} f x ∗ ∈ arg min f such that ∥ x ∗ ∥ ≤ R \lVert x^* \rVert\le R ∥ x ∗ ∥ ≤ R . We denote f ∗ = f ( x ∗ ) f^* = f(x^*) f ∗ = f ( x ∗ ) .
Assumption 2.4 (initial optimality gap). We have f ( 0 ) − f ∗ ≤ Δ f(0)-f^* \le\Delta f ( 0 ) − f ∗ ≤ Δ , and Δ ≥ 2 ε \Delta\ge2\varepsilon Δ ≥ 2 ε .
Assumption 2.5 (heterogeneity).
f ∗ − 1 n ∑ i = 1 n inf x ∈ R d f i ( x ) ≤ δ . (2) f^*-\frac{1}{n}\sum_{i=1}^{n}\inf_{x\in\mathbb{R}^d} f_i(x)\le\delta. \tag*{(2)} f ∗ − n 1 i = 1 ∑ n x ∈ R d inf f i ( x ) ≤ δ . (2)
Assumption 2.6 (range of accuracies).
L 1 2 R 2 δ ε ≥ 3. (3) \frac{L_1^2R^2\delta}{\varepsilon}\ge3. \tag*{(3)} ε L 1 2 R 2 δ ≥ 3. (3)
Definition 2.7 (problem class). We denote by P d \mathcal{P}_d P d the set of collections of functions f 1 , … , f n : R d → R f_1,\ldots,f_n:\mathbb{R}^d\to\mathbb{R} f 1 , … , f n : R d → R and known numbers L 0 , 1 , … , L 0 , n L_{0,1},\ldots,L_{0,n} L 0 , 1 , … , L 0 , n satisfying Assumptions 2.1–2.5.
Assumption 2.2 bounds only the average of the numbers L 0 , i L_{0,i} L 0 , i ; individual L 0 , i L_{0,i} L 0 , i may exceed L 0 L_0 L 0 . It suffices to construct the lower bound on the subclass L 0 , i = L 0 L_{0,i}=L_0 L 0 , i = L 0 . For C 2 C^2 C 2 functions, the Hessian bound from Assumption 2.2 is equivalent to symmetric ( L 0 , i , L 1 ) (L_{0,i},L_1) ( L 0 , i , L 1 ) -smoothness Gorbunov et al., [14 ] (Assumption 1.3, Eq. (5)):
∥ ∇ f i ( x ) − ∇ f i ( y ) ∥ ≤ ( L 0 , i + L 1 sup z ∈ [ x , y ] ∥ ∇ f i ( z ) ∥ ) ∥ x − y ∥ . \left\|\nabla f_i(x)-\nabla f_i(y)\right\|\le\left(L_{0,i}+L_1\sup_{z\in[x,y]}\left\|\nabla f_i(z)\right\|\right)\left\|x-y\right\|. ∥ ∇ f i ( x ) − ∇ f i ( y ) ∥ ≤ ( L 0 , i + L 1 z ∈ [ x , y ] sup ∥ ∇ f i ( z ) ∥ ) ∥ x − y ∥ .
Assumption 2.5 requires knowing neither f ∗ f^* f ∗ nor the infima of the components. Assumption 2.6 guarantees that in the hard instance with a hidden pair of functions from § 8 , the initial optimality gap exceeds the required accuracy (Lemma D.5).
Oracle model. The algorithm knows the problem parameters and the numbers L 0 , i L_{0,i} L 0 , i . It obtains information about the functions only from an oracle that returns ( f i ( x ) , ∇ f i ( x ) ) (f_i(x),\nabla f_i(x)) ( f i ( x ) , ∇ f i ( x )) for a query ( i , x ) (i,x) ( i , x ) . Queries are allowed anywhere in R d \mathbb{R}^d R d . The query points, the stopping time, and the output are measurable functions of the received oracle answers and the random variables of the algorithm. Complexity is measured by the number of oracle calls Nesterov, [26 ] (Section 1.1). Computing the function value and the full gradient at one point requires n n n queries.
Definition 2.8 (oracle complexity). Let τ M ( p ) \tau_M(p) τ M ( p ) be the number of oracle calls made by the algorithm M on the input p p p , and let x ^ M ( p ) \hat{x}_M(p) x ^ M ( p ) be its output. The algorithm is called valid if for any d d d and p ∈ P d p\in\mathcal{P}_d p ∈ P d it stops almost surely and
E [ f ( x ^ M ( p ) ) − f ∗ ] ≤ ε . \mathbb{E}\left[f(\hat{x}_M(p))-f^*\right]\le\varepsilon. E [ f ( x ^ M ( p )) − f ∗ ] ≤ ε . The complexity of an algorithm on the class and the minimax complexity are defined as
T ( M ) = sup d ≥ 1 sup p ∈ P d E τ M ( p ) , \mathcal{T}(M)=\sup_{d\ge1}\sup_{p\in\mathcal{P}_d}\mathbb{E}\tau_M(p), T ( M ) = d ≥ 1 sup p ∈ P d sup E τ M ( p ) , T ∗ = inf M v a l i d T ( M ) . (4) \mathcal{T}^*=\inf_{M\ \mathrm{valid}}\mathcal{T}(M). \tag*{(4)} T ∗ = M valid inf T ( M ) . (4)
To bound the number of queries, we set
N 0 = 4 + n + ( L 0 + L 1 2 δ ) R 2 ε + ( L 1 R ) 2 , N = N 0 − 2 + Δ ε . (5) N_0 = 4 + n + \frac{(L_0 + L_1^2\delta)R^2}{\varepsilon} + (L_1R)^2,\qquad N = N_0 - 2 + \frac{\Delta}{\varepsilon}. \tag*{(5)} N 0 = 4 + n + ε ( L 0 + L 1 2 δ ) R 2 + ( L 1 R ) 2 , N = N 0 − 2 + ε Δ . (5)
For nonnegative quantities, the notation a ≲ b a \lesssim b a ≲ b means a ≤ C b a \le Cb a ≤ C b with an absolute constant C C C independent of the problem parameters, the dimension, and the algorithm. We write a ∼ b a \sim b a ∼ b if a ≲ b ≲ a a \lesssim b \lesssim a a ≲ b ≲ a . The notation O O O and Ω \Omega Ω is used in the same uniform sense, and Θ ~ \widetilde{\Theta} Θ also hides powers of log N \log N log N . The lower bound uses the quantity N 0 N_0 N 0 , which does not depend on Δ \Delta Δ , and the upper bound uses the quantity N N N , which by Assumption 2.4 is not smaller than N 0 N_0 N 0 . We set
T 0 = n ( 1 + ( L 1 R ) 2 / 3 ) + n ( L 0 + L 1 2 δ ) R 2 ε + n ( L 1 R ) 2 . (6) T_0 = n\left(1 + (L_1R)^{2/3}\right) + \sqrt{\frac{n(L_0 + L_1^2\delta)R^2}{\varepsilon} + n(L_1R)^2}. \tag*{(6)} T 0 = n ( 1 + ( L 1 R ) 2/3 ) + ε n ( L 0 + L 1 2 δ ) R 2 + n ( L 1 R ) 2 . (6)
Theorem 2.9 (two-sided complexity bound). Under Assumptions 2.1 –2.6 , we have
T 0 log 2 N 0 ≲ T ∗ ≲ T 0 log 2 N . (7) \frac{T_0}{\log^2 N_0} \lesssim T^* \lesssim T_0\log^2 N. \tag*{(7)} log 2 N 0 T 0 ≲ T ∗ ≲ T 0 log 2 N . (7)
The upper bound is attained by the method YETI, including the case of different L 0 , i L_{0,i} L 0 , i . The lower bound holds already for L 0 , i = L 0 L_{0,i} = L_0 L 0 , i = L 0 .
The upper bound is proved in Theorem 5.1 , and the lower bound in Theorem 7.1 . The proofs are given in Appendices § 12 and § 14 . The question of closing the gap in the logarithmic factors remains open.
The method YETI
YETI combines an accelerated proximal method with a heuristic. The heuristic proposes a point z ∈ B ( 0 , R ) z \in B(0, R) z ∈ B ( 0 , R ) based on the query history. The method accepts it if f ( z ) ≤ f ( x ) f(z) \le f(x) f ( z ) ≤ f ( x ) . A single heuristic block is allowed at most n ⌈ log N ⌉ n\lceil\log N\rceil n ⌈ log N ⌉ oracle queries, including the verification of the proposed points. If the block has not terminated the method, the main iteration is performed (Figure 1 ).
Figure 1. Flowchart of the method YETI.
Accelerated step and cut. From the gradients of the components at the point y y y , we compute the curvature estimate
ℓ ( y ) = 1 n ∑ i ( L 0 , i + L 1 ∥ ∇ f i ( y ) ∥ ) . \ell(y) = \frac{1}{n}\sum_i (L_{0,i} + L_1\lVert\nabla f_i(y)\rVert). ℓ ( y ) = n 1 i ∑ ( L 0 , i + L 1 ∥ ∇ f i ( y )∥) .
By Lemma G.2, the following inequalities hold:
∥ ∇ 2 f ( y ) ∥ ≤ ℓ ( y ) ≤ 3 2 L 0 + 2 L 1 2 ( δ + f ( y ) − f ∗ ) , ℓ ( z ) ≤ e L 1 ∥ z − y ∥ ℓ ( y ) . \lVert\nabla^2 f(y)\rVert\le\ell(y) \le\frac{3}{2}L_0 + 2L_1^2(\delta+ f(y) - f^*), \qquad\ell(z) \le e^{L_1\lVert z-y\rVert}\ell(y). ∥ ∇ 2 f ( y )∥ ≤ ℓ ( y ) ≤ 2 3 L 0 + 2 L 1 2 ( δ + f ( y ) − f ∗ ) , ℓ ( z ) ≤ e L 1 ∥ z − y ∥ ℓ ( y ) .
If the quantity ℓ ( y ) \ell(y) ℓ ( y ) is small, then in the ball of radius r = min { R , 1 / ( 2 L 1 ) } r = \min\{R,1/(2L_1)\} r = min { R , 1/ ( 2 L 1 )} one can solve the regularized subproblem by SVRG and perform an accelerated step. If, however, the curvature check fails, then the upper bound on ℓ ( y ) \ell(y) ℓ ( y ) implies a lower bound on the optimality gap f ( y ) − f ∗ f(y)-f^* f ( y ) − f ∗ . By convexity (Assumption 2.1), ⟨ ∇ f ( y ) , x ∗ − y ⟩ ≤ − ( f ( y ) − f ∗ ) \langle\nabla f(y),x^*-y\rangle\le-(f(y)-f^*) ⟨ ∇ f ( y ) , x ∗ − y ⟩ ≤ − ( f ( y ) − f ∗ ) , so this bound defines a halfspace H y \mathcal{H}_y H y containing the minimizer. The projection of the auxiliary point v v v onto this halfspace brings it closer to x ∗ x^* x ∗ .
Stopping. By Assumptions 2.1 and 2.3, each computation of f ( p ) f(p) f ( p ) and ∇ f ( p ) \nabla f(p) ∇ f ( p ) yields the lower bound
f ~ ( p ) = f ( p ) − ⟨ ∇ f ( p ) , p ⟩ − R ∥ ∇ f ( p ) ∥ ≤ f ∗ . \widetilde{f}(p) = f(p) - \langle\nabla f(p),p\rangle- R\lVert\nabla f(p)\rVert\le f^*. f ( p ) = f ( p ) − ⟨ ∇ f ( p ) , p ⟩ − R ∥ ∇ f ( p )∥ ≤ f ∗ .
The subroutine EVALUATE stores the maximum f ~ b e s t \widetilde{f}_{\mathrm{best}} f best of such bounds and the best point found so far, w b e s t ∈ B ( 0 , R ) w_{\mathrm{best}}\in B(0,R) w best ∈ B ( 0 , R ) . If f ( w b e s t ) − f ~ b e s t ≤ ε / 2 f(w_{\mathrm{best}})-\widetilde{f}_{\mathrm{best}}\le\varepsilon/2 f ( w best ) − f best ≤ ε /2 , the method stops and returns w b e s t w_{\mathrm{best}} w best . Component answers that have already been obtained are not queried again.
The local subproblem
Algorithm 1 stores the points x , v x,v x , v , the weight A \mathcal{A} A , and the parameter h h h , from which the regularization is chosen. The bounds λ ↓ ( A ) \lambda_{\downarrow}(\mathcal{A}) λ ↓ ( A ) and λ ↑ \lambda_{\uparrow} λ ↑ are the lower and upper limits of the proximal parameter: at each iteration, the method takes λ = min { max { h , λ ↓ ( A ) } , λ ↑ } \lambda=\min\{\max\{h,\lambda_{\downarrow}(\mathcal{A})\},\lambda_{\uparrow}\} λ = min { max { h , λ ↓ ( A )} , λ ↑ } . The lower limit keeps the weight of the accelerated step bounded, makes the local subproblem well conditioned, and ensures stopping; it does not increase as the weight A \mathcal{A} A grows. The upper limit λ ↑ \lambda_{\uparrow} λ ↑ is the value of the lower limit at the initial weight A = R 2 / ( 2 Δ ) \mathcal{A}=R^2/(2\Delta) A = R 2 / ( 2Δ ) . Explicit formulas are given in (G.7). The subroutines EVALUATE and BLOCK, including the rules for counting queries, are given in Appendix § 16 . The requirements on the output of SOLVER are stated in the next section.
Algorithm 1 YETI
Input: n n n , R R R , Δ \Delta Δ , δ \delta δ , ε \varepsilon ε , L 0 L_0 L 0 , L 1 L_1 L 1 , constants L 0 , i L_{0,i} L 0 , i , a heuristic
set r r r , λ ↓ \lambda_{\downarrow} λ ↓ , and λ ↑ \lambda_{\uparrow} λ ↑ by (G.7)
x = v = 0 x=v=0 x = v = 0
A = R 2 / ( 2 Δ ) \mathcal{A}=R^2/(2\Delta) A = R 2 / ( 2Δ )
h = λ ↑ h=\lambda_{\uparrow} h = λ ↑
the query history is empty
loop
x ← BLOCK ( x ) x\leftarrow\operatorname{BLOCK}(x) x ← BLOCK ( x ) ▷ heuristic block
λ = min { max { h , λ ↓ ( A ) } , λ ↑ } \lambda=\min\{\max\{h,\lambda_{\downarrow}(\mathcal{A})\},\lambda_{\uparrow}\} λ = min { max { h , λ ↓ ( A )} , λ ↑ }
a λ = ( 1 + 1 + 4 λ A ) / ( 2 λ ) a_\lambda=(1+\sqrt{1+4\lambda\mathcal{A}})/(2\lambda) a λ = ( 1 + 1 + 4 λ A ) / ( 2 λ )
α = a λ / ( A + a λ ) \alpha=a_\lambda/(\mathcal{A}+a_\lambda) α = a λ / ( A + a λ )
y = ( 1 − α ) x + α v y=(1-\alpha)x+\alpha v y = ( 1 − α ) x + α v
EVALUATE ( y ) \operatorname{EVALUATE}(y) EVALUATE ( y )
ℓ ( y ) = 1 n ∑ i ( L 0 , i + L 1 ∥ ∇ f i ( y ) ∥ ) \ell(y)=\frac{1}{n}\sum_i(L_{0,i}+L_1\lVert\nabla f_i(y)\rVert) ℓ ( y ) = n 1 ∑ i ( L 0 , i + L 1 ∥ ∇ f i ( y )∥)
if ℓ ( y ) > e L 1 r ( 3 2 L 0 + 2 L 1 2 ( δ + R 2 / A ) ) \ell(y)>e^{L_1r}\left(\frac{3}{2}L_0+2L_1^2(\delta+R^2/\mathcal{A})\right) ℓ ( y ) > e L 1 r ( 2 3 L 0 + 2 L 1 2 ( δ + R 2 / A ) ) then ▷ cut
H y = { w : ⟨ ∇ f ( y ) , w − y ⟩ ≤ − ℓ ( y ) − 3 2 L 0 − 2 L 1 2 δ 2 L 1 2 } \mathcal{H}_y=\left\{w:\langle\nabla f(y),w-y\rangle\leq-\frac{\ell(y)-\frac{3}{2}L_0-2L_1^2\delta}{2L_1^2}\right\} H y = { w : ⟨ ∇ f ( y ) , w − y ⟩ ≤ − 2 L 1 2 ℓ ( y ) − 2 3 L 0 − 2 L 1 2 δ }
v ← Π B ( 0 , R ) Π H y v v\leftarrow\Pi_{B(0,R)}\Pi_{\mathcal{H}_y}v v ← Π B ( 0 , R ) Π H y v
h ← 2 λ h\leftarrow2\lambda h ← 2 λ
else ▷ step based on the solution of the local subproblem
( z , g , β ) ← SOLVER ( y , λ ) (z,g,\beta)\leftarrow\operatorname{SOLVER}(y,\lambda) ( z , g , β ) ← SOLVER ( y , λ )
if ∥ g ∥ ≤ ε / ( 4 R ) \lVert g\rVert\leq\varepsilon/(4R) ∥ g ∥ ≤ ε / ( 4 R ) then
return z z z
end if
γ = λ / ( λ + β ) \gamma=\lambda/(\lambda+\beta) γ = λ / ( λ + β )
x ← ( 1 − γ ) A x + γ ( A + a λ ) z A + γ a λ x\leftarrow\frac{(1-\gamma)\mathcal{A}x+\gamma(\mathcal{A}+a_\lambda)z}{\mathcal{A}+\gamma a_\lambda} x ← A + γ a λ ( 1 − γ ) A x + γ ( A + a λ ) z
v ← Π B ( 0 , R ) ( v − γ a λ g ) v\leftarrow\Pi_{B(0,R)}(v-\gamma a_\lambda g) v ← Π B ( 0 , R ) ( v − γ a λ g )
A ← A + γ a λ \mathcal{A}\leftarrow\mathcal{A}+\gamma a_\lambda A ← A + γ a λ
h ← { 2 λ , β > 0 , λ / 2 , β = 0 h\leftarrow\begin{cases}2\lambda,&\beta>0,\\\lambda/2,&\beta=0\end{cases} h ← { 2 λ , λ /2 , β > 0 , β = 0
if A ≥ 2 R 2 / ε \mathcal{A}\geq2R^2/\varepsilon A ≥ 2 R 2 / ε then
return x x x
end if
end if
end loop
The local subproblem
At the point y y y , we solve the subproblem
min z ∈ B ( 0 , R ) ∩ B ( y , r ) { f ( z ) + λ 2 ∥ z − y ∥ 2 } . \min_{z\in B(0,R)\cap B(y,r)}\left\{f(z)+\frac{\lambda}{2}\lVert z-y\rVert^{2}\right\}. z ∈ B ( 0 , R ) ∩ B ( y , r ) min { f ( z ) + 2 λ ∥ z − y ∥ 2 } .
The solver returns z ∈ B ( 0 , R ) ∩ B ( y , r ) z\in B(0,R)\cap B(y,r) z ∈ B ( 0 , R ) ∩ B ( y , r ) , g ∈ ∂ ( f + I B ( 0 , R ) ) ( z ) g\in\partial(f+I_{B(0,R)})(z) g ∈ ∂ ( f + I B ( 0 , R ) ) ( z ) , and β ≥ 0 \beta\ge0 β ≥ 0 , where β > 0 \beta>0 β > 0 is possible only if ∥ y − z ∥ = r \lVert y-z\rVert=r ∥ y − z ∥ = r . Here I B ( 0 , R ) I_{B(0,R)} I B ( 0 , R ) is the indicator function of the ball B ( 0 , R ) B(0,R) B ( 0 , R ) . The output of the solver must satisfy one of the two conditions
∥ g ∥ ≤ ε 4 R or ∥ g − ( λ + β ) ( y − z ) ∥ ≤ 4 5 ( λ + β ) ∥ y − z ∥ . \lVert g\rVert\le\frac{\varepsilon}{4R}
\qquad\text{or}\qquad
\lVert g-(\lambda+\beta)(y-z)\rVert\le\frac{4}{5}(\lambda+\beta)\lVert y-z\rVert. ∥ g ∥ ≤ 4 R ε or ∥ g − ( λ + β ) ( y − z )∥ ≤ 5 4 ( λ + β ) ∥ y − z ∥ .
The first one allows YETI to stop, and the second one ensures the accuracy of the accelerated update. If β = 0 \beta=0 β = 0 , the weight of this update does not change, and if β > 0 \beta>0 β > 0 , it is multiplied by the coefficient γ = λ / ( λ + β ) < 1 \gamma=\lambda/(\lambda+\beta)<1 γ = λ / ( λ + β ) < 1 .
To solve the subproblem, we use SVRG with component sampling probabilities proportional to their local smoothness constants. After each epoch, the above conditions are checked. The solver terminates almost surely (Lemma H.2 ), and the expected number of its queries per iteration of YETI with a fixed history is O ( n log ( n + 2 ) ) O(n\log(n+2)) O ( n log ( n + 2 )) (Appendix § 12 .§ 3 ). The full algorithm of the solver is given in Appendix § 17 .
Complexity bound for YETI
The analysis uses the Lyapunov function
Φ = A ( f ( x ) − f ∗ ) + 1 2 ∥ v − x ∗ ∥ 2 ≤ R 2 . \Phi=\mathcal{A}(f(x)-f^*)+\frac{1}{2}\lVert v-x^*\rVert^2\le R^2. Φ = A ( f ( x ) − f ∗ ) + 2 1 ∥ v − x ∗ ∥ 2 ≤ R 2 .
Accepting a point proposed by the heuristic does not increase the first term. The projection onto H y \mathcal{H}_y H y decreases the second one. Under the accelerated update, Φ \Phi Φ does not increase owing to the accuracy condition of the local solver. Hence f ( x ) − f ∗ ≤ R 2 / A f(x)-f^*\le R^2/\mathcal{A} f ( x ) − f ∗ ≤ R 2 / A .
The number of iterations at which the regularization increases is bounded owing to the decrease of Φ \Phi Φ . At the remaining iterations, the regularization is halved or the weight A \mathcal{A} A grows. This yields a bound on the number of iterations until the condition A ≥ 2 R 2 / ε \mathcal{A}\ge2R^2/\varepsilon A ≥ 2 R 2 / ε holds. The corresponding bounds are proved in Lemmas B.3 and B.4 of Appendix § 11 . Each iteration, together with the heuristic block preceding it, requires O ( n log N ) O(n\log N) O ( n log N ) queries in expectation for a fixed history of the previous iterations.
Theorem 5.1 (upper bound). Under Assumptions 2.1–2.6, Algorithm 1 stops almost surely and returns a point x ^ \hat{x} x ^ such that
E [ f ( x ^ ) − f ∗ ] ≤ ε , (8) \mathbb{E}[f(\hat{x})-f^*]\le\varepsilon, \tag*{(8)} E [ f ( x ^ ) − f ∗ ] ≤ ε , (8) and the expected total number T T T of oracle calls satisfies
E T ≲ T 0 log 2 N . (9) \mathbb{E}T\lesssim T_0\log^2 N. \tag*{(9)} E T ≲ T 0 log 2 N . (9) The count T T T includes all oracle calls in the heuristic blocks, the main iterations, and the checks.
The proof is given in Appendix § 12 .§ 3 . A variant in which the allowed number of queries of the heuristic depends on the work already done is considered in Appendix § 15 .§ 5 .
Numerical experiments
We compare the methods by the number of oracle calls on chains S and X (the hard families from § 8.1 ), on Poisson models, and on 65 instances from 13 families (five instances each), including sums of quartic functions and logistic regression. The number of queries is divided by n n n , that is, it is measured in passes over the data; the queries spent on rejected trial points during line search are also counted. The relative optimality gap equals ( f − f r e f ) / Δ ^ (f-f_{\mathrm{ref}})/\widehat{\Delta} ( f − f ref ) / Δ , where Δ ^ = f ( 0 ) − f r e f \widehat{\Delta}=f(0)-f_{\mathrm{ref}} Δ = f ( 0 ) − f ref and f r e f f_{\mathrm{ref}} f ref is the reference value of the minimum. On the chains, it coincides with f ∗ f^* f ∗ , which is known from the construction. On the other instances, f ∗ f^* f ∗ is unknown, and f r e f f_{\mathrm{ref}} f ref is its approximation computed numerically in advance; therefore, we denote it by f r e f f_{\mathrm{ref}} f ref rather than f ∗ f^* f ∗ . The queries spent on computing the reference value are not counted in the comparison. The problem formulations, method settings, and detailed results are given in Appendix § 18 .
On the chains, YETI performs approximately as well as L-BFGS. On five Poisson instances, it reaches the relative accuracy 10 − 4 10^{-4} 1 0 − 4 within 4–9 passes; on freMTPL2freq, SAGA requires fewer passes: 5 versus 7 (Table 2 , Table 4 ).
Table 2. Number of passes to relative accuracy 10 − 2 10^{-2} 1 0 − 2 on the chains. The minimum before rounding is shown in bold; “—” indicates that the tolerance was not reached within the allotted number of passes.
Table 4. Number of passes to relative accuracy 10 − 4 10^{-4} 1 0 − 4 on the Poisson models. Notation as in Table 2 .
Stopping. The history-based stopping criterion (G.2) does not use the reference value. For ε = 10 − 6 Δ ^ \varepsilon=10^{-6}\widehat{\Delta} ε = 1 0 − 6 Δ , it is triggered on all 65 instances, and all returned points satisfy the tolerance with respect to the reference value. The median number of passes until stopping is 15, and the maximum is 273. The criterion may be triggered later than the tolerance is actually reached (Figure 4 ).
Figure 4. Optimality gap relative to the reference value and its history-based upper bound at ε = 10 − 9 Δ ^ \varepsilon= 10^{-9}\widehat{\Delta} ε = 1 0 − 9 Δ . All quantities are divided by Δ ^ \widehat{\Delta} Δ ; the dotted line indicates the threshold ε / 2 \varepsilon/2 ε /2 . Values below 10 − 14 10^{-14} 1 0 − 14 are shown at the lower boundary of the axis.
Test problems. Chains S and X are numerical variants of the hard families from § 8.1 . The Poisson models are synthetic instances and instances on the Bike Sharing and freMTPL2freq data. The thirteen families from which the 65 instances are composed (their names are shown in Figure 2 and in Table 1) are constructed as follows.
Figure 2. Figure 2. Best relative optimality gap so far on the first instance of each family (log–log scale; YETI in red; κ \kappa κ is the condition number of the feature covariance).
Rotated, κ \kappa κ : Poisson regression in which the feature covariance matrix has condition number κ ∈ { 10 4 , 10 8 } \kappa\in\{10^4,10^8\} κ ∈ { 1 0 4 , 1 0 8 } and is not diagonal in the original basis (its eigenvectors are rotated with respect to the coordinate axes).
High dimension: Poisson regression with a large number of features: n = 600 n=600 n = 600 , d = 160 d=160 d = 160 , κ = 10 4 \kappa=10^4 κ = 1 0 4 .
Strong nonlinearity: Poisson regression in which the true parameter vector used to generate the responses is three times longer than in the other Poisson families; this increases the spread of the linear predictors a i T x a_i^{\mathsf T}x a i T x and of the values of the exponential.
Rare features and Rare, n = 5 ⋅ 10 4 n=5\cdot10^4 n = 5 ⋅ 1 0 4 : Poisson regression with sparse features; in the second family, the number of components equals 5 ⋅ 10 4 5\cdot10^4 5 ⋅ 1 0 4 .
n = 5 ⋅ 10 4 n=5\cdot10^4 n = 5 ⋅ 1 0 4 , κ = 10 8 \kappa=10^8 κ = 1 0 8 : ill-conditioned Poisson regression with 5 ⋅ 10 4 5\cdot10^4 5 ⋅ 1 0 4 components.
Quartics: sums of quartic functions f i ( x ) = 1 2 ( x − c i ) T H i ( x − c i ) + ν 4 ∥ x − c i ∥ 4 f_i(x)=\frac{1}{2}(x-c_i)^{\mathsf T}H_i(x-c_i)+\frac{\nu}{4}\lVert x-c_i\rVert^4 f i ( x ) = 2 1 ( x − c i ) T H i ( x − c i ) + 4 ν ∥ x − c i ∥ 4 , where H i ≻ 0 H_i\succ0 H i ≻ 0 and c i c_i c i are random centers. In the family names 32 × 48 and 256 × 32, the number of components n n n and the dimension d d d are indicated; ν \nu ν is the weight of the fourth-degree term (in these two families, ν = 10 − 6 \nu=10^{-6} ν = 1 0 − 6 ).
Logistic, rotated and Logistic, rare: regularized logistic regression with rotated and with sparse features, respectively.
Lower bound for arbitrary queries
We bound from below the number of queries required by any method that solves all instances of the class under consideration.
The origin of the four terms
We prove the lower bound separately for each of the four terms. Denote
B n = n , B S = n L 0 R 2 ε , B Q = n ( L 1 R ) 2 / 3 , B X = n L 1 2 R 2 δ ε + n ( L 1 R ) 2 . (10) \begin{aligned}
B_n &= n,\\
B_S &= \sqrt{\frac{nL_0R^2}{\varepsilon}},\\
B_Q &= n(L_1R)^{2/3},\\
B_X &= \sqrt{\frac{nL_1^2R^2\delta}{\varepsilon} + n(L_1R)^2}. \tag*{(10)}
\end{aligned} B n B S B Q B X = n , = ε n L 0 R 2 , = n ( L 1 R ) 2/3 , = ε n L 1 2 R 2 δ + n ( L 1 R ) 2 . (10)
Since a 2 + b 2 ∼ a + b \sqrt{a^2+b^2}\sim a+b a 2 + b 2 ∼ a + b for a , b ≥ 0 a,b\ge0 a , b ≥ 0 , we have T 0 ∼ B n + B S + B X + B Q T_0\sim B_n+B_S+B_X+B_Q T 0 ∼ B n + B S + B X + B Q .
Table 1.
For each term, we construct its own family of instances. Since the method must solve the instances of all four families, its complexity is not smaller than the maximum of the obtained bounds. Below, the dependencies are discussed up to numerical and logarithmic factors. The precise conditions of the constructions are given in the appendix; if a particular construction is not applicable, the corresponding term is bounded via the others.
Search for an informative component: B n = n B_n=n B n = n . Among the n n n components, two nonzero ones are chosen at random, and the rest are zero. Until the first call to one of them, the algorithm receives only zero answers. These answers do not allow it to distinguish two instances whose minimizers lie on opposite sides of the starting point. For the expected error to be small, one has to find the hidden pair of functions with sufficient probability, and this requires on the order of n n n oracle calls (Lemma D.5).
Independent smooth chains: B S B_S B S . Each component contains its own chain of coordinates. By a chain we mean a function in which the next coordinate affects the oracle answer only after sufficient progress along the previous coordinates. In the idealized model, a single query reveals at most one new link of one chain. How this property carries over to arbitrary queries is explained below.
In family S, the number of links of a single chain is of order L 0 R 2 n ε \sqrt{\frac{L_0R^2}{n\varepsilon}} n ε L 0 R 2 . To reach accuracy ε \varepsilon ε , one needs to reveal a sufficient fraction of the total number of links of all chains. Therefore, the number of queries is of order
n L 0 R 2 n ε = n L 0 R 2 ε = B S . n\sqrt{\frac{L_0R^2}{n\varepsilon}}=\sqrt{\frac{nL_0R^2}{\varepsilon}}=B_S. n n ε L 0 R 2 = ε n L 0 R 2 = B S .
The instances of this family are smooth: ∥ ∇ 2 f i ∥ ≤ L 0 \lVert\nabla^2 f_i\rVert\le L_0 ∥ ∇ 2 f i ∥ ≤ L 0 (Lemma D.6 ). If the conditions for constructing the chain are not satisfied, then B S ≲ B n B_S\lesssim B_n B S ≲ B n , and the bound for B n B_n B n suffices.
Exponential chains: B X B_X B X . Under generalized smoothness, the curvature of a component can grow together with its gradient. Even at a minimizer of the average, where ∇ f ( x ∗ ) = 0 \nabla f(x^*)=0 ∇ f ( x ∗ ) = 0 , the individual gradients ∇ f i ( x ∗ ) \nabla f_i(x^*) ∇ f i ( x ∗ ) can be nonzero and cancel each other. Therefore, a small norm of the gradient of the average does not imply small smoothness constants of the components. The parameter δ \delta δ bounds the discrepancy between the minimum of the average and the average of the infima of the components.
In family X, the chains enter the exponents of components grouped into pairs. The construction preserves the sequential revealing of links and sets the heterogeneity equal to δ \delta δ . The bound for this family, together with the bound for the hidden pair of functions, yields B X B_X B X (Lemmas D.8 and D.5). Since B X = n L 1 R 1 + δ / ϵ B_X=\sqrt{n}L_1R\sqrt{1+\delta/\epsilon} B X = n L 1 R 1 + δ / ϵ , we have
B X ∼ n L 1 R max { 1 , δ ϵ } . B_X\sim\sqrt{n}L_1R\max\left\{1,\sqrt{\frac{\delta}{\epsilon}}\right\}. B X ∼ n L 1 R max { 1 , ϵ δ } .
Search for the next link: B Q B_Q B Q . Family Q uses a single chain, but its links are randomly placed across pairs of components. After revealing the next coordinate, it is not enough to know in which direction to continue: one also has to find the pair containing the next link. A component that has already been queried may become informative only at the next level of the chain, so computing all components once does not eliminate these searches.
Under the conditions of Lemma D.10 , the length of the chain is of order ( L 1 R ) 2 / 3 (L_1R)^{2/3} ( L 1 R ) 2/3 , and the search for a pair at each level requires on the order of n n n queries in expectation. Hence, up to logarithmic factors,
( L 1 R ) 2 / 3 ⏟ number of levels n ⏟ search per level = B Q . \underbrace{(L_1R)^{2/3}}_{\text{number of levels}}\underbrace{n}_{\text{search per level}}=B_Q. number of levels ( L 1 R ) 2/3 search per level n = B Q .
The exponent 2 / 3 2/3 2/3 is explained by the choice of the chain length T T T . At its minimizer, the T T T active coordinates are equal to R / T R/\sqrt{T} R / T , so that the norm of this point equals R R R . The bound on the optimality gap in terms of the number of revealed links ((D.20)) shows that to reach the accuracy one has to pass on the order of L 1 R / T L_1R/\sqrt{T} L 1 R / T levels (up to logarithmic factors), while the total number of levels is T T T . The chain length is chosen so that the quantities T T T and L 1 R / T L_1R/\sqrt{T} L 1 R / T coincide up to absolute constants. From the equality T = L 1 R / T T=L_1R/\sqrt{T} T = L 1 R / T we obtain T 3 / 2 = L 1 R T^{3/2}=L_1R T 3/2 = L 1 R , that is, T = ( L 1 R ) 2 / 3 T=(L_1R)^{2/3} T = ( L 1 R ) 2/3 ; in Lemma D.10 , an integer T T T of this order is chosen. The precise conditions of this choice also take into account the initial optimality gap and the heterogeneity. The number of oracle calls required to find the next link is bounded in Lemmas D.9 and D.3 .
The lower bound
Let Assumptions 2.1 –2.6 hold. The number N 0 N_0 N 0 is defined in (5) .
Theorem 7.1. ▼ proof in Appendix E
For an oracle returning ( f i ( x ) , ∇ f i ( x ) ) (f_i(x),\nabla f_i(x)) ( f i ( x ) , ∇ f i ( x )) , we have
T ∗ ≳ T 0 log 2 N 0 . T^*\gtrsim\frac{T_0}{\log^2 N_0}. T ∗ ≳ log 2 N 0 T 0 .
The right-hand side does not depend on Δ \Delta Δ for Δ ≥ 2 ε \Delta\ge2\varepsilon Δ ≥ 2 ε . The bound refers to the worst instance of the class in arbitrary dimension. The method may be randomized and may choose query points anywhere in the space, not only in the linear span of the previously obtained gradients.
Together with the upper bound for YETI (Theorem 5.1 ), this theorem shows that in this setting the exponents of n n n , L 0 L_0 L 0 , L 1 L_1 L 1 , R R R , δ \delta δ , and ε \varepsilon ε cannot be improved (up to logarithmic factors).
The logarithmic gap between the upper and lower bounds is not closed here. Assumption 2.6 is essential for the scope of the result: this lower bound cannot be automatically transferred, for example, to the subclass with δ = 0 \delta= 0 δ = 0 . The equality δ = 0 \delta= 0 δ = 0 means that the point x ∗ x^* x ∗ minimizes each component, that is, all f i f_i f i have a common minimizer. For this case, convergence bounds for the stochastic methods ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -SGD and SGD-PS were obtained in Gorbunov et al. [14 ] .
Proof of the lower bound
Proof sketch of Theorem 7.1 . The instance with a hidden pair of functions gives the bound for B n B_n B n . Family S gives the bound for B S B_S B S , and if its conditions are violated, we use B S ≲ B n B_S \lesssim B_n B S ≲ B n . Family X, together with the hidden pair of functions, gives the bound for B X B_X B X up to a logarithmic factor. Family Q gives the bound for B Q B_Q B Q . When conditions (D.17) are not satisfied, this term is bounded via the bounds already obtained.
Consequently,
T 0 ∼ max { B n , B S , B X , B Q } ≲ T ∗ log 2 2 N 0 . T_0 \sim\max\{B_n, B_S, B_X, B_Q\} \lesssim T^* \log_2^2 N_0. T 0 ∼ max { B n , B S , B X , B Q } ≲ T ∗ log 2 2 N 0 .
Lemmas D.1–D.3 make it possible to apply these bounds to arbitrary randomized methods. The analysis of all parameter regimes is given in Appendix § 14 .
Conclusion
For minimizing a sum of convex ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smooth functions, the method YETI attains the upper bound O ( T 0 log 2 N ) O(T_0\log^2 N) O ( T 0 log 2 N ) , and for any valid method we prove the lower bound Ω ( T 0 / log 2 N 0 ) \Omega(T_0/\log^2 N_0) Ω ( T 0 / log 2 N 0 ) . If Δ / ε ≤ N 0 c \Delta/\varepsilon\le N_0^c Δ/ ε ≤ N 0 c for some fixed c ≥ 1 c \ge1 c ≥ 1 , then these bounds coincide up to powers of log N 0 \log N_0 log N 0 . Without a restriction on the growth of the initial optimality gap Δ \Delta Δ , the optimality of the upper bound has not been established.
On 65 instances with relative accuracy 10 − 4 10^{-4} 1 0 − 4 , the tested variant of YETI ranked first (including ties with other methods) in 46 cases, and its history-based stopping criterion was triggered on all 65 instances. In these experiments, the main gain comes from the heuristic (Appendix § 18 ). However, its matrix operations may be expensive, so a gain in the number of queries does not necessarily mean the same gain in running time.
Open questions
Large initial optimality gap. What is the optimal dependence of the complexity on Δ \Delta Δ ? If log ( Δ / ε ) \log(\Delta/\varepsilon) log ( Δ/ ε ) is of order L 1 R L_1R L 1 R , then the obtained upper and lower bounds may differ polynomially in L 1 R L_1R L 1 R . One needs either to improve the upper bound or to prove a lower bound that accounts for Δ \Delta Δ . For moderate Δ \Delta Δ , it remains to close the logarithmic gap.
Individual smoothness constants. Can one obtain a sharper complexity bound for a given collection ( L 0 , i , L 1 , i ) (L_{0,i},L_{1,i}) ( L 0 , i , L 1 , i ) without replacing all L 1 , i L_{1,i} L 1 , i by their maximum? Non-uniform sampling of components already makes it possible to use only the average value of L 0 , i L_{0,i} L 0 , i (Appendix § 15 ). To improve the general bound, one needs to account for the relation between L 1 , i L_{1,i} L 1 , i and the optimality gaps of individual components and to construct a matching lower bound: our lower bound is obtained on the subclass with equal L 0 , i L_{0,i} L 0 , i .
Constraints and non-Euclidean geometry. How can the result be extended to minimization over a convex set and to methods with a Bregman distance? Mirror descent under generalized smoothness was studied in Yu et al. [40 ] . Here, one will additionally need an analysis of variance reduction, a replacement of the Euclidean potential, and a bound on the number of cuts.
Nonsmooth components. Can one obtain similar two-sided bounds for finite sums whose subgradient norms are bounded in terms of the optimality gap? Such a nonsmooth
stochastic setting is considered in Kovalev [17 ] . For a finite sum, one needs to define the oracle model separately and to determine what gain variance reduction provides.
Second- and higher-order methods. Can variance reduction for finite sums [33 ] be combined with acceleration under generalized smoothness of the Hessian [1 ] ? This requires error bounds for stochastic derivatives and a new local solver. Conditions on higher-order derivatives do not follow from Assumption 2.2 .
The dual problem. Can YETI be interpreted as a coordinate method for the dual problem and related to the scheme of Podlipnova et al. [28 ] ? Besides transforming the problem, one will need to match the available oracles and the cost of a single step: a query of a component f i f_i f i and a query of a coordinate of the dual gradient are not equivalent in general.
Separately, it remains to determine the complexity outside the scope of Assumption 2.6 , in particular for δ = 0 \delta= 0 δ = 0 . This question concerns the heterogeneity of the components and is distinct from the question of a large initial optimality gap Δ \Delta Δ .
Analytic properties of the class
The auxiliary lemmas are stated for an arbitrary function φ \varphi φ . They apply to the components f i f_i f i by Assumptions 2.1 and 2.2, with the constant L 0 , i L_{0,i} L 0 , i in place of L 0 L_0 L 0 .
Local smoothness
See also Rudomanov et al. [30 ] and Vankov et al. [36 ] .
Lemma A.1 (growth of the gradient along a segment). Let φ ∈ C 2 ( R d ) \varphi\in C^2(\mathbb{R}^d) φ ∈ C 2 ( R d ) and ∥ ∇ 2 φ ( x ) ∥ ≤ L 0 + L 1 ∥ ∇ φ ( x ) ∥ \|\nabla^2\varphi(x)\| \le L_0 + L_1\|\nabla\varphi(x)\| ∥ ∇ 2 φ ( x ) ∥ ≤ L 0 + L 1 ∥∇ φ ( x ) ∥ for all x x x . Then for any x , y x,y x , y ,
L 0 + L 1 ∥ ∇ φ ( x ) ∥ ≤ e L 1 ∥ x − y ∥ ( L 0 + L 1 ∥ ∇ φ ( y ) ∥ ) . (11) L_0 + L_1\|\nabla\varphi(x)\| \le e^{L_1\|x-y\|}\left(L_0 + L_1\|\nabla\varphi(y)\|\right). \tag*{(11)} L 0 + L 1 ∥∇ φ ( x ) ∥ ≤ e L 1 ∥ x − y ∥ ( L 0 + L 1 ∥∇ φ ( y ) ∥ ) . (11) In particular, on the ball B ( y , r ) B(y,r) B ( y , r ) the gradient of φ \varphi φ is Lipschitz with constant e L 1 r ( L 0 + L 1 ∥ ∇ φ ( y ) ∥ ) e^{L_1r}(L_0+L_1\|\nabla\varphi(y)\|) e L 1 r ( L 0 + L 1 ∥∇ φ ( y ) ∥ ) .
Proof. By the Newton–Leibniz formula,
∇ φ ( y + t ( x − y ) ) − ∇ φ ( y ) = ∫ 0 t ∇ 2 φ ( y + u ( x − y ) ) ( x − y ) d u . \nabla\varphi(y+t(x-y))-\nabla\varphi(y)=\int_0^t\nabla^2\varphi(y+u(x-y))(x-y)\,\mathrm{d}u. ∇ φ ( y + t ( x − y )) − ∇ φ ( y ) = ∫ 0 t ∇ 2 φ ( y + u ( x − y )) ( x − y ) d u .
Taking norms and applying the Hessian condition, for t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] we obtain
L 0 + L 1 ∥ ∇ φ ( y + t ( x − y ) ) ∥ ≤ L 0 + L 1 ∥ ∇ φ ( y ) ∥ + L 1 ∥ x − y ∥ ∫ 0 t ( L 0 + L 1 ∥ ∇ φ ( y + u ( x − y ) ) ∥ ) d u . L_0+L_1\|\nabla\varphi(y+t(x-y))\|\le L_0+L_1\|\nabla\varphi(y)\|+L_1\|x-y\|\int_0^t\left(L_0+L_1\|\nabla\varphi(y+u(x-y))\|\right)\,\mathrm{d}u. L 0 + L 1 ∥∇ φ ( y + t ( x − y )) ∥ ≤ L 0 + L 1 ∥∇ φ ( y ) ∥ + L 1 ∥ x − y ∥ ∫ 0 t ( L 0 + L 1 ∥∇ φ ( y + u ( x − y )) ∥ ) d u .
By Grönwall’s inequality,
L 0 + L 1 ∥ ∇ φ ( y + t ( x − y ) ) ∥ ≤ e L 1 ∥ x − y ∥ t ( L 0 + L 1 ∥ ∇ φ ( y ) ∥ ) . L_0+L_1\|\nabla\varphi(y+t(x-y))\|\le e^{L_1\|x-y\|t}\left(L_0+L_1\|\nabla\varphi(y)\|\right). L 0 + L 1 ∥∇ φ ( y + t ( x − y )) ∥ ≤ e L 1 ∥ x − y ∥ t ( L 0 + L 1 ∥∇ φ ( y ) ∥ ) .
Setting t = 1 t=1 t = 1 gives the first claim. For x ∈ B ( y , r ) x\in B(y,r) x ∈ B ( y , r ) it yields a bound on the Hessian norm over the whole ball, and integrating the Hessian along segments inside the convex ball proves that the gradient is Lipschitz.
( (Baillon–Haddad theorem, Bauschke and Combettes, 2010) )
Lemma A.2 (cocoercivity on a convex domain). Let φ \varphi φ be convex and twice continuously differentiable in a neighborhood of a convex set D D D . If ∥ ∇ 2 φ ∥ ≤ L \|\nabla^2\varphi\|\le L ∥ ∇ 2 φ ∥ ≤ L on D D D , where L > 0 L>0 L > 0 , then for all x , z ∈ D x,z\in D x , z ∈ D ,
∥ ∇ φ ( x ) − ∇ φ ( z ) ∥ 2 ≤ L ⟨ ∇ φ ( x ) − ∇ φ ( z ) , x − z ⟩ . \|\nabla\varphi(x)-\nabla\varphi(z)\|^2\le L\langle\nabla\varphi(x)-\nabla\varphi(z),x-z\rangle. ∥∇ φ ( x ) − ∇ φ ( z ) ∥ 2 ≤ L ⟨ ∇ φ ( x ) − ∇ φ ( z ) , x − z ⟩ .
Proof. The averaged Hessian
H = ∫ 0 1 ∇ 2 φ ( z + t ( x − z ) ) d t H=\int_0^1\nabla^2\varphi(z+t(x-z))\,\mathrm{d}t H = ∫ 0 1 ∇ 2 φ ( z + t ( x − z )) d t
satisfies 0 ⪯ H ⪯ L I 0\preceq H\preceq LI 0 ⪯ H ⪯ L I by convexity; hence H 2 ⪯ L H H^2\preceq LH H 2 ⪯ L H and
∥ ∇ φ ( x ) − ∇ φ ( z ) ∥ 2 = ⟨ x − z , H 2 ( x − z ) ⟩ ≤ L ⟨ x − z , H ( x − z ) ⟩ = L ⟨ ∇ φ ( x ) − ∇ φ ( z ) , x − z ⟩ . \|\nabla\varphi(x)-\nabla\varphi(z)\|^2=\langle x-z,H^2(x-z)\rangle\le L\langle x-z,H(x-z)\rangle=L\langle\nabla\varphi(x)-\nabla\varphi(z),x-z\rangle. ∥∇ φ ( x ) − ∇ φ ( z ) ∥ 2 = ⟨ x − z , H 2 ( x − z )⟩ ≤ L ⟨ x − z , H ( x − z )⟩ = L ⟨ ∇ φ ( x ) − ∇ φ ( z ) , x − z ⟩ .
Lemma A.3 (bound on the gradient difference). Let φ \varphi φ be convex, φ ∈ C 2 ( R d ) \varphi\in C^2(\mathbb{R}^d) φ ∈ C 2 ( R d ) , and ∥ ∇ 2 φ ∥ ≤ L \|\nabla^2\varphi\|\le L ∥ ∇ 2 φ ∥ ≤ L on B ( y , 3 r ) B(y,3r) B ( y , 3 r ) , where L > 0 L>0 L > 0 . Then for x , w ∈ B ( y , r ) x,w\in B(y,r) x , w ∈ B ( y , r ) ,
∥ ∇ φ ( x ) − ∇ φ ( w ) ∥ 2 ≤ 2 L ( φ ( x ) − φ ( w ) − ⟨ ∇ φ ( w ) , x − w ⟩ ) . \|\nabla\varphi(x)-\nabla\varphi(w)\|^2\le2L\left(\varphi(x)-\varphi(w)-\langle\nabla\varphi(w),x-w\rangle\right). ∥∇ φ ( x ) − ∇ φ ( w ) ∥ 2 ≤ 2 L ( φ ( x ) − φ ( w ) − ⟨ ∇ φ ( w ) , x − w ⟩ ) .
Proof. Since the gradient of φ \varphi φ is L L L -Lipschitz on B ( y , r ) B(y,r) B ( y , r ) and x , w ∈ B ( y , r ) x,w \in B(y,r) x , w ∈ B ( y , r ) , we have
∥ ∇ φ ( x ) − ∇ φ ( w ) ∥ ≤ L ∥ x − w ∥ ≤ 2 L r . \lVert\nabla\varphi(x)-\nabla\varphi(w)\rVert\le L\lVert x-w\rVert\le2Lr. ∥ ∇ φ ( x ) − ∇ φ ( w )∥ ≤ L ∥ x − w ∥ ≤ 2 L r .
Therefore, the point
q = x − 1 L ( ∇ φ ( x ) − ∇ φ ( w ) ) q=x-\frac{1}{L}\left(\nabla\varphi(x)-\nabla\varphi(w)\right) q = x − L 1 ( ∇ φ ( x ) − ∇ φ ( w ) )
lies in B ( y , 3 r ) B(y,3r) B ( y , 3 r ) , and hence the whole segment [ x , q ] [x,q] [ x , q ] is contained in B ( y , 3 r ) B(y,3r) B ( y , 3 r ) . The function
z ↦ φ ( z ) − φ ( w ) − ⟨ ∇ φ ( w ) , z − w ⟩ z\mapsto\varphi(z)-\varphi(w)-\langle\nabla\varphi(w),z-w\rangle z ↦ φ ( z ) − φ ( w ) − ⟨ ∇ φ ( w ) , z − w ⟩
is convex, nonnegative on the whole space, and has on B ( y , 3 r ) B(y,3r) B ( y , 3 r ) the L L L -Lipschitz gradient ∇ φ ( z ) − ∇ φ ( w ) \nabla\varphi(z)-\nabla\varphi(w) ∇ φ ( z ) − ∇ φ ( w ) . The quadratic upper bound on the segment [ x , q ] [x,q] [ x , q ] gives
0 ≤ φ ( q ) − φ ( w ) − ⟨ ∇ φ ( w ) , q − w ⟩ ≤ φ ( x ) − φ ( w ) − ⟨ ∇ φ ( w ) , x − w ⟩ + ⟨ ∇ φ ( x ) − ∇ φ ( w ) , q − x ⟩ + L 2 ∥ q − x ∥ 2 = φ ( x ) − φ ( w ) − ⟨ ∇ φ ( w ) , x − w ⟩ − 1 2 L ∥ ∇ φ ( x ) − ∇ φ ( w ) ∥ 2 . \begin{aligned}
0 &\le\varphi(q)-\varphi(w)-\langle\nabla\varphi(w),q-w\rangle\\
&\le\varphi(x)-\varphi(w)-\langle\nabla\varphi(w),x-w\rangle+\langle\nabla\varphi(x)-\nabla\varphi(w),q-x\rangle+\frac{L}{2}\lVert q-x\rVert^2 \\
&=\varphi(x)-\varphi(w)-\langle\nabla\varphi(w),x-w\rangle-\frac{1}{2L}\lVert\nabla\varphi(x)-\nabla\varphi(w)\rVert^2.
\end{aligned} 0 ≤ φ ( q ) − φ ( w ) − ⟨ ∇ φ ( w ) , q − w ⟩ ≤ φ ( x ) − φ ( w ) − ⟨ ∇ φ ( w ) , x − w ⟩ + ⟨ ∇ φ ( x ) − ∇ φ ( w ) , q − x ⟩ + 2 L ∥ q − x ∥ 2 = φ ( x ) − φ ( w ) − ⟨ ∇ φ ( w ) , x − w ⟩ − 2 L 1 ∥ ∇ φ ( x ) − ∇ φ ( w ) ∥ 2 .
It remains to move the last term to the left-hand side.
Bounding the gradient norm by the optimality gap
Cf. Li et al. [19 ] and Yu et al. [40 ] .
Lemma A.4 (integral bound on the gradient norm). Let φ \varphi φ be convex, bounded from below, belong to C 2 ( R d ) C^2(\mathbb{R}^d) C 2 ( R d ) , and satisfy the condition of Lemma A.1. Then for any x x x ,
∫ 0 ∥ ∇ φ ( x ) ∥ t L 0 + L 1 t d t ≤ φ ( x ) − inf φ , \int_{0}^{\lVert\nabla\varphi(x)\rVert}\frac{t}{L_0+L_1t}\,dt\le\varphi(x)-\inf\varphi, ∫ 0 ∥ ∇ φ ( x )∥ L 0 + L 1 t t d t ≤ φ ( x ) − inf φ , ∥ ∇ φ ( x ) ∥ ≤ L 1 ( φ ( x ) − inf φ ) + 2 L 0 ( φ ( x ) − inf φ ) . (12) \lVert\nabla\varphi(x)\rVert\le L_1\bigl(\varphi(x)-\inf\varphi\bigr)+\sqrt{2L_0\bigl(\varphi(x)-\inf\varphi\bigr)}. \tag*{(12)} ∥ ∇ φ ( x )∥ ≤ L 1 ( φ ( x ) − inf φ ) + 2 L 0 ( φ ( x ) − inf φ ) . (12) For L 0 = 0 L_0=0 L 0 = 0 , the integrand is extended to zero by continuity. The infimum is not assumed to be attained.
Proof. If ∇ φ ( x ) = 0 \nabla\varphi(x)=0 ∇ φ ( x ) = 0 , the claim is obvious. Otherwise, choose h > 0 h>0 h > 0 such that
L 1 h ∥ ∇ φ ( x ) ∥ ≤ 1 2 , L_1h\lVert\nabla\varphi(x)\rVert\le\frac{1}{2}, L 1 h ∥ ∇ φ ( x )∥ ≤ 2 1 ,
h ( L 0 + L 1 ∥ ∇ φ ( x ) ∥ ) ≤ 1 2 , h\left(L_0+L_1\lVert\nabla\varphi(x)\rVert\right)\le\frac{1}{2}, h ( L 0 + L 1 ∥ ∇ φ ( x )∥ ) ≤ 2 1 ,
and consider gradient descent started from x 0 = x x_0=x x 0 = x :
x j + 1 = x j − h ∇ φ ( x j ) . x_{j+1}=x_j-h\nabla\varphi(x_j). x j + 1 = x j − h ∇ φ ( x j ) .
▷ The gradient norm does not increase. Suppose ∥ ∇ φ ( x j ) ∥ ≤ ∥ ∇ φ ( x ) ∥ \lVert\nabla\varphi(x_j)\rVert\le\lVert\nabla\varphi(x)\rVert ∥ ∇ φ ( x j )∥ ≤ ∥ ∇ φ ( x )∥ . The segment [ x j , x j + 1 ] [x_j,x_{j+1}] [ x j , x j + 1 ] has length h ∥ ∇ φ ( x j ) ∥ h\lVert\nabla\varphi(x_j)\rVert h ∥ ∇ φ ( x j )∥ ; hence, by Lemma A.1, the Hessian norm on it does not exceed
e L 1 h ∥ ∇ φ ( x j ) ∥ ( L 0 + L 1 ∥ ∇ φ ( x j ) ∥ ) ≤ 2 ( L 0 + L 1 ∥ ∇ φ ( x ) ∥ ) ≤ 1 h . (13) e^{L_1h\lVert\nabla\varphi(x_j)\rVert}\left(L_0+L_1\lVert\nabla\varphi(x_j)\rVert\right)\le2\left(L_0+L_1\lVert\nabla\varphi(x)\rVert\right)\le\frac{1}{h}. \tag*{(13)} e L 1 h ∥ ∇ φ ( x j )∥ ( L 0 + L 1 ∥ ∇ φ ( x j )∥ ) ≤ 2 ( L 0 + L 1 ∥ ∇ φ ( x )∥ ) ≤ h 1 . (13)
Thus, the averaged Hessian H j H_j H j on this segment satisfies 0 ⪯ h H j ⪯ I 0\preceq hH_j\preceq I 0 ⪯ h H j ⪯ I , and
∇ φ ( x j + 1 ) = ( I − h H j ) ∇ φ ( x j ) , \nabla\varphi(x_{j+1})=(I-hH_j)\nabla\varphi(x_j), ∇ φ ( x j + 1 ) = ( I − h H j ) ∇ φ ( x j ) ,
∥ ∇ φ ( x j + 1 ) ∥ ≤ ( a ) ∥ ∇ φ ( x j ) ∥ . \lVert\nabla\varphi(x_{j+1})\rVert\underset{(a)}{\le}\lVert\nabla\varphi(x_j)\rVert. ∥ ∇ φ ( x j + 1 )∥ ( a ) ≤ ∥ ∇ φ ( x j )∥ .
Here (a) follows from 0 ⪯ I − h H j ⪯ I 0\preceq I-hH_j\preceq I 0 ⪯ I − h H j ⪯ I . By induction, these bounds hold for all j j j .
▷ We bound two telescoping sums. The quadratic upper bound together with the curvature estimate (A.3) gives
φ ( x j ) − φ ( x j + 1 ) ≥ h ∥ ∇ φ ( x j ) ∥ 2 − h 2 2 e L 1 h ∥ ∇ φ ( x j ) ∥ ( L 0 + L 1 ∥ ∇ φ ( x j ) ∥ ) ∥ ∇ φ ( x j ) ∥ 2 ≥ h 2 ∥ ∇ φ ( x j ) ∥ 2 . \varphi(x_j)-\varphi(x_{j+1}) \ge h\|\nabla\varphi(x_j)\|^2-\frac{h^2}{2}e^{L_1h\|\nabla\varphi(x_j)\|}(L_0+L_1\|\nabla\varphi(x_j)\|)\|\nabla\varphi(x_j)\|^2 \ge\frac{h}{2}\|\nabla\varphi(x_j)\|^2. φ ( x j ) − φ ( x j + 1 ) ≥ h ∥∇ φ ( x j ) ∥ 2 − 2 h 2 e L 1 h ∥∇ φ ( x j ) ∥ ( L 0 + L 1 ∥∇ φ ( x j ) ∥ ) ∥∇ φ ( x j ) ∥ 2 ≥ 2 h ∥∇ φ ( x j ) ∥ 2 .
The integrand t / ( L 0 + L 1 t ) t/(L_0+L_1t) t / ( L 0 + L 1 t ) is nondecreasing, and
∥ ∇ φ ( x j ) ∥ − ∥ ∇ φ ( x j + 1 ) ∥ ≤ ∥ h H j ∇ φ ( x j ) ∥ . \|\nabla\varphi(x_j)\|-\|\nabla\varphi(x_{j+1})\| \le\|hH_j\nabla\varphi(x_j)\|. ∥∇ φ ( x j ) ∥ − ∥∇ φ ( x j + 1 ) ∥ ≤ ∥ h H j ∇ φ ( x j ) ∥.
Therefore,
∫ ∥ ∇ φ ( x j + 1 ) ∥ ∥ ∇ φ ( x j ) ∥ t d t L 0 + L 1 t ≤ ∥ ∇ φ ( x j ) ∥ L 0 + L 1 ∥ ∇ φ ( x j ) ∥ ∥ h H j ∇ φ ( x j ) ∥ ≤ ( a ) h e L 1 h ∥ ∇ φ ( x j ) ∥ ∥ ∇ φ ( x j ) ∥ 2 . \int_{\|\nabla\varphi(x_{j+1})\|}^{\|\nabla\varphi(x_j)\|}\frac{t\,dt}{L_0+L_1t}
\le\frac{\|\nabla\varphi(x_j)\|}{L_0+L_1\|\nabla\varphi(x_j)\|}\|hH_j\nabla\varphi(x_j)\|
\overset{(a)}{\le} he^{L_1h\|\nabla\varphi(x_j)\|}\|\nabla\varphi(x_j)\|^2. ∫ ∥∇ φ ( x j + 1 ) ∥ ∥∇ φ ( x j ) ∥ L 0 + L 1 t t d t ≤ L 0 + L 1 ∥∇ φ ( x j ) ∥ ∥∇ φ ( x j ) ∥ ∥ h H j ∇ φ ( x j ) ∥ ≤ ( a ) h e L 1 h ∥∇ φ ( x j ) ∥ ∥∇ φ ( x j ) ∥ 2 .
Here (a) follows from the bound (A.3) on ∥ H j ∥ \|H_j\| ∥ H j ∥ . Since e s − 1 ≤ 2 s e^s-1\le2s e s − 1 ≤ 2 s and e s ≤ 2 e^s\le2 e s ≤ 2 for 0 ≤ s ≤ 1 / 2 0\le s\le1/2 0 ≤ s ≤ 1/2 , the function decrease bound and the last bound on the integral yield
∫ ∥ ∇ φ ( x j + 1 ) ∥ ∥ ∇ φ ( x j ) ∥ t d t L 0 + L 1 t ≤ φ ( x j ) − φ ( x j + 1 ) + h 2 ( L 0 + 3 L 1 ∥ ∇ φ ( x ) ∥ ) ∥ ∇ φ ( x j ) ∥ 2 . \int_{\|\nabla\varphi(x_{j+1})\|}^{\|\nabla\varphi(x_j)\|}\frac{t\,dt}{L_0+L_1t}
\le\varphi(x_j)-\varphi(x_{j+1})+h^2(L_0+3L_1\|\nabla\varphi(x)\|)\|\nabla\varphi(x_j)\|^2. ∫ ∥∇ φ ( x j + 1 ) ∥ ∥∇ φ ( x j ) ∥ L 0 + L 1 t t d t ≤ φ ( x j ) − φ ( x j + 1 ) + h 2 ( L 0 + 3 L 1 ∥∇ φ ( x ) ∥ ) ∥∇ φ ( x j ) ∥ 2 .
Summing the inequality
φ ( x i ) − φ ( x i + 1 ) ≥ h 2 ∥ ∇ φ ( x i ) ∥ 2 \varphi(x_i)-\varphi(x_{i+1})\ge\frac{h}{2}\|\nabla\varphi(x_i)\|^2 φ ( x i ) − φ ( x i + 1 ) ≥ 2 h ∥∇ φ ( x i ) ∥ 2
over i = 0 , … , m − 1 i=0,\ldots,m-1 i = 0 , … , m − 1 , we obtain
h ∑ i < m ∥ ∇ φ ( x i ) ∥ 2 ≤ 2 ( φ ( x ) − inf φ ) . h\sum_{i<m}\|\nabla\varphi(x_i)\|^2\le2(\varphi(x)-\inf\varphi). h i < m ∑ ∥∇ φ ( x i ) ∥ 2 ≤ 2 ( φ ( x ) − inf φ ) .
Summing the bound on the difference of integrals over i = 0 , … , j − 1 i=0,\ldots,j-1 i = 0 , … , j − 1 , where 0 ≤ j < m 0\le j<m 0 ≤ j < m , and using the resulting bound on the sum of squared gradient norms, we obtain
∫ 0 ∥ ∇ φ ( x ) ∥ t d t L 0 + L 1 t ≤ ( φ ( x ) − inf φ ) ( 1 + 2 h ( L 0 + 3 L 1 ∥ ∇ φ ( x ) ∥ ) ) + ∫ 0 ∥ ∇ φ ( x j ) ∥ t d t L 0 + L 1 t . \int_0^{\|\nabla\varphi(x)\|}\frac{t\,dt}{L_0+L_1t}
\le(\varphi(x)-\inf\varphi)\left(1+2h(L_0+3L_1\|\nabla\varphi(x)\|)\right)
+\int_0^{\|\nabla\varphi(x_j)\|}\frac{t\,dt}{L_0+L_1t}. ∫ 0 ∥∇ φ ( x ) ∥ L 0 + L 1 t t d t ≤ ( φ ( x ) − inf φ ) ( 1 + 2 h ( L 0 + 3 L 1 ∥∇ φ ( x ) ∥ ) ) + ∫ 0 ∥∇ φ ( x j ) ∥ L 0 + L 1 t t d t .
Take m = ⌈ h − 2 ⌉ m=\lceil h^{-2}\rceil m = ⌈ h − 2 ⌉ and an index j < m j<m j < m with the smallest norm ∥ ∇ φ ( x j ) ∥ \|\nabla\varphi(x_j)\| ∥∇ φ ( x j ) ∥ . Then
∥ ∇ φ ( x j ) ∥ 2 ≤ 2 ( φ ( x ) − inf φ ) m h → 0 as h ↓ 0. \|\nabla\varphi(x_j)\|^2\le\frac{2(\varphi(x)-\inf\varphi)}{mh}\to0\qquad\text{as }h\downarrow0. ∥∇ φ ( x j ) ∥ 2 ≤ mh 2 ( φ ( x ) − inf φ ) → 0 as h ↓ 0.
The last integral does not exceed ∥ ∇ φ ( x j ) ∥ / L 1 \|\nabla\varphi(x_j)\|/L_1 ∥∇ φ ( x j ) ∥/ L 1 ; hence, passing to the limit, we obtain the first inequality in (A.2).
▷ For L 0 = 0 L_0=0 L 0 = 0 , the integral equals ∥ ∇ φ ( x ) ∥ / L 1 \|\nabla\varphi(x)\|/L_1 ∥∇ φ ( x ) ∥/ L 1 , which immediately gives the second inequality. For L 0 > 0 L_0>0 L 0 > 0 , after the substitution t = L 0 s / L 1 t=L_0s/L_1 t = L 0 s / L 1 , the integral bound takes the form
L 1 ∥ ∇ φ ( x ) ∥ L 0 − log ( 1 + L 1 ∥ ∇ φ ( x ) ∥ L 0 ) ≤ L 1 2 ( φ ( x ) − inf φ ) L 0 . \frac{L_1\|\nabla\varphi(x)\|}{L_0}-\log\left(1+\frac{L_1\|\nabla\varphi(x)\|}{L_0}\right)\le\frac{L_1^2(\varphi(x)-\inf\varphi)}{L_0}. L 0 L 1 ∥∇ φ ( x ) ∥ − log ( 1 + L 0 L 1 ∥∇ φ ( x ) ∥ ) ≤ L 0 L 1 2 ( φ ( x ) − inf φ ) .
The function s ↦ s − log ( 1 + s ) s\mapsto s-\log(1+s) s ↦ s − log ( 1 + s ) is increasing for s ≥ 0 s\ge0 s ≥ 0 . For v ≥ 0 v\ge0 v ≥ 0 , the power series expansion of the exponential gives e 2 v ≥ 1 + 2 v + v e^{\sqrt{2v}}\ge1+\sqrt{2v}+v e 2 v ≥ 1 + 2 v + v ; hence
( v + 2 v ) − log ( 1 + v + 2 v ) ≥ v . (v+\sqrt{2v})-\log(1+v+\sqrt{2v})\ge v. ( v + 2 v ) − log ( 1 + v + 2 v ) ≥ v .
Applying this inequality with v = L 1 2 ( φ ( x ) − inf φ ) / L 0 v=L_1^2(\varphi(x)-\inf\varphi)/L_0 v = L 1 2 ( φ ( x ) − inf φ ) / L 0 , we obtain
L 1 ∥ ∇ φ ( x ) ∥ L 0 ≤ L 1 2 ( φ ( x ) − inf φ ) L 0 + 2 L 1 2 ( φ ( x ) − inf φ ) L 0 , \frac{L_1\|\nabla\varphi(x)\|}{L_0}\le\frac{L_1^2(\varphi(x)-\inf\varphi)}{L_0}+\sqrt{\frac{2L_1^2(\varphi(x)-\inf\varphi)}{L_0}}, L 0 L 1 ∥∇ φ ( x ) ∥ ≤ L 0 L 1 2 ( φ ( x ) − inf φ ) + L 0 2 L 1 2 ( φ ( x ) − inf φ ) ,
as required.
Bounding the local curvature of the sum
Proof of Lemma G.2 ▲. For the average of the nonnegative quantities f i ( y ) − inf f i f_i(y)-\inf f_i f i ( y ) − inf f i we have
1 n ∑ i ( f i ( y ) − inf f i ) = f ( y ) − f ∗ + f ∗ − 1 n ∑ i inf f i ≤ δ + f ( y ) − f ∗ . \frac{1}{n}\sum_i\left(f_i(y)-\inf f_i\right)=f(y)-f^*+f^*-\frac{1}{n}\sum_i\inf f_i\le\delta+f(y)-f^*. n 1 i ∑ ( f i ( y ) − inf f i ) = f ( y ) − f ∗ + f ∗ − n 1 i ∑ inf f i ≤ δ + f ( y ) − f ∗ .
By Lemma A.4 , which applies by Assumptions 2.1 and 2.2,
1 n ∑ i ∥ ∇ f i ( y ) ∥ ≤ L 1 n ∑ i ( f i ( y ) − inf f i ) + 1 n ∑ i 2 L 0 , i ( f i ( y ) − inf f i ) ≤ L 1 ( δ + f ( y ) − f ∗ ) + 2 L 0 ( δ + f ( y ) − f ∗ ) , \begin{aligned}
\frac{1}{n}\sum_i\lVert\nabla f_i(y)\rVert\le\frac{L_1}{n}\sum_i\left(f_i(y)-\inf f_i\right)+\frac{1}{n}\sum_i\sqrt{2L_{0,i}\left(f_i(y)-\inf f_i\right)} \\
&\le L_1\left(\delta+f(y)-f^*\right)+\sqrt{2L_0\left(\delta+f(y)-f^*\right)},
\end{aligned} n 1 i ∑ ∥ ∇ f i ( y )∥ ≤ n L 1 i ∑ ( f i ( y ) − inf f i ) + n 1 i ∑ 2 L 0 , i ( f i ( y ) − inf f i ) ≤ L 1 ( δ + f ( y ) − f ∗ ) + 2 L 0 ( δ + f ( y ) − f ∗ ) ,
where (a) follows from the Cauchy–Schwarz inequality and the constraint on the average value of L 0 , i L_{0,i} L 0 , i from Assumption 2.2; (b) follows from 2 a b ≤ a 2 + b 2 2ab\le a^2+b^2 2 ab ≤ a 2 + b 2 . The triangle inequality gives both bounds for ∥ ∇ f ( y ) ∥ \lVert\nabla f(y)\rVert ∥ ∇ f ( y )∥ . The remaining claims follow by summing the bounds of Lemma A.1 and taking Assumption 2.2 into account.
Degenerate class
Proposition A.5. Under Assumptions 2.1–2.3 and 2.5, if L 0 = δ = 0 L_0=\delta=0 L 0 = δ = 0 , then all components are constant. Consequently, the minimax oracle complexity of this class is zero.
Proof. By Assumption 2.2, L 0 = 0 L_0=0 L 0 = 0 implies L 0 , i = 0 L_{0,i}=0 L 0 , i = 0 . At the point x ∗ x^* x ∗ from Assumption 2.3 we have
0 ≤ 1 n ∑ i ( f i ( x ∗ ) − inf f i ) = f ∗ − 1 n ∑ i inf f i ≤ δ = 0. 0\le\frac{1}{n}\sum_i\left(f_i(x^*)-\inf f_i\right)=f^*-\frac{1}{n}\sum_i\inf f_i\le\delta=0. 0 ≤ n 1 i ∑ ( f i ( x ∗ ) − inf f i ) = f ∗ − n 1 i ∑ inf f i ≤ δ = 0.
Hence each summand is zero, whence ∇ f i ( x ∗ ) = 0 \nabla f_i(x^*)=0 ∇ f i ( x ∗ ) = 0 . Lemma A.1 with L 0 , i = 0 L_{0,i}=0 L 0 , i = 0 gives
∥ ∇ f i ( x ) ∥ ≤ e L 1 ∥ x − x ∗ ∥ ∥ ∇ f i ( x ∗ ) ∥ = 0 \lVert\nabla f_i(x)\rVert\le e^{L_1\lVert x-x^*\rVert}\lVert\nabla f_i(x^*)\rVert=0 ∥ ∇ f i ( x )∥ ≤ e L 1 ∥ x − x ∗ ∥ ∥ ∇ f i ( x ∗ )∥ = 0
for any x x x . Consequently, all components are constant, and the answer 0 can be output without a single oracle call.
Proofs of the results on YETI
History lower bounds
Proof of Lemma G.1 ▲. Choose a point w ∈ B ( 0 , R ) w\in B(0,R) w ∈ B ( 0 , R ) minimizing the linear function u ↦ ⟨ ∇ f ( z ) , u ⟩ u\mapsto\langle\nabla f(z),u\rangle u ↦ ⟨ ∇ f ( z ) , u ⟩ . By the definition of the normal cone, ⟨ v z , w − z ⟩ ≤ 0 \langle v_z,w-z\rangle\le0 ⟨ v z , w − z ⟩ ≤ 0 . Therefore,
f ( z ) − f ~ ( z ) = ⟨ ∇ f ( z ) , z − w ⟩ ≤ ⟨ g , z − w ⟩ ≤ ∥ g ∥ ∥ z − w ∥ ≤ 2 R ∥ g ∥ , f(z)-\widetilde f(z)=\langle\nabla f(z),z-w\rangle\le\langle g,z-w\rangle\le\lVert g\rVert\lVert z-w\rVert\le2R\lVert g\rVert, f ( z ) − f ( z ) = ⟨ ∇ f ( z ) , z − w ⟩ ≤ ⟨ g , z − w ⟩ ≤ ∥ g ∥ ∥ z − w ∥ ≤ 2 R ∥ g ∥ ,
where (a) follows from the Cauchy–Schwarz inequality. After z z z is recorded in the history, we have f ( w b e s t ) ≤ f ( z ) f(w_{\mathrm{best}})\le f(z) f ( w best ) ≤ f ( z ) and f ~ b e s t ≥ f ~ ( z ) \widetilde f_{\mathrm{best}}\ge\widetilde f(z) f best ≥ f ( z ) , which completes the proof.
A lower bound from the responses of individual components
The stopping criterion (G.2) uses the lower bounds (G.1), each of which requires the responses of all n n n components at the same point. The heuristic in the experiments (Appendix § 18 ) queries individual components at different points. We show that such responses can also be combined into a lower bound on f ∗ f^* f ∗ without making new queries. This subsection is needed only for the experiments: it is not used in the proof of the complexity bound for YETI.
For each index i i i , choose one of the responses of this component already obtained; let it be obtained at the point p i p_i p i . Set
f ~ m i x = 1 n ∑ i = 1 n ( f i ( p i ) − ⟨ ∇ f i ( p i ) , p i ⟩ ) − R ∥ 1 n ∑ i = 1 n ∇ f i ( p i ) ∥ . (14) \widetilde{f}_{\mathrm{mix}} = \frac{1}{n}\sum_{i=1}^{n}\left(f_i(p_i)-\langle\nabla f_i(p_i),p_i\rangle\right)-R\left\lVert\frac{1}{n}\sum_{i=1}^{n}\nabla f_i(p_i)\right\rVert. \tag*{(14)} f mix = n 1 i = 1 ∑ n ( f i ( p i ) − ⟨ ∇ f i ( p i ) , p i ⟩ ) − R n 1 i = 1 ∑ n ∇ f i ( p i ) . (14)
If all points p i p_i p i coincide with some point p p p , then f ~ m i x = f ~ ( p ) \widetilde{f}_{\mathrm{mix}}=\widetilde{f}(p) f mix = f ( p ) ; that is, (B.1) generalizes (G.1).
Proposition B.1.
We have f ~ m i x ≤ f ∗ \widetilde{f}_{\mathrm{mix}}\le f^* f mix ≤ f ∗ . Therefore, the quantity f ~ m i x \widetilde{f}_{\mathrm{mix}} f mix can be included in the maximum f ~ b e s t \widetilde{f}_{\mathrm{best}} f best on a par with the bounds (G.1). When one selected response is replaced, recomputing both sums in (B.1) requires O ( d ) O(d) O ( d ) arithmetic operations and no new oracle calls.
Proof. By the convexity of the components (Assumption 2.1), for any w w w ,
f ( w ) ≥ 1 n ∑ i ( f i ( p i ) + ⟨ ∇ f i ( p i ) , w − p i ⟩ ) . f(w)\ge\frac{1}{n}\sum_i\left(f_i(p_i)+\langle\nabla f_i(p_i),w-p_i\rangle\right). f ( w ) ≥ n 1 i ∑ ( f i ( p i ) + ⟨ ∇ f i ( p i ) , w − p i ⟩ ) .
For w = x ∗ w=x^* w = x ∗ , Assumption 2.3 and the Cauchy–Schwarz inequality give f ∗ ≥ f ~ m i x f^*\ge\widetilde{f}_{\mathrm{mix}} f ∗ ≥ f mix . Replacing the response with index i i i changes only one term of the scalar sum and one term of the vector sum.
The bound (B.1) can be computed only after at least one response has been obtained for each component. No conditions on the points p i p_i p i or on the order in which they are obtained are required. Under subsequent replacements of responses, the value f ~ m i x \widetilde{f}_{\mathrm{mix}} f mix may decrease; therefore, f ~ b e s t \widetilde{f}_{\mathrm{best}} f best stores the maximum of all lower bounds obtained so far. To recompute f ~ m i x \widetilde{f}_{\mathrm{mix}} f mix , one needs to store one selected response per component; this is additional memory compared with storing only the best lower bound and the best point.
Local solver
Proof of Lemma H.1 ▲.
z ζ = Π B ( 0 , R ) ( b + ζ y 1 + ζ ) . z_\zeta=\Pi_{B(0,R)}\left(\frac{b+\zeta y}{1+\zeta}\right). z ζ = Π B ( 0 , R ) ( 1 + ζ b + ζ y ) .
It depends continuously on ζ \zeta ζ and tends to y y y as ζ → ∞ \zeta\to\infty ζ → ∞ . If ∥ z 0 − y ∥ ≤ r \lVert z_0-y\rVert\le r ∥ z 0 − y ∥ ≤ r , it suffices to take ζ = 0 \zeta=0 ζ = 0 . Otherwise, there exists ζ > 0 \zeta>0 ζ > 0 for which ∥ z ζ − y ∥ = r \lVert z_\zeta-y\rVert=r ∥ z ζ − y ∥ = r . The form of the normal cone to the ball B ( 0 , R ) B(0,R) B ( 0 , R ) gives (H.3). For any t ∈ K t\in K t ∈ K we obtain
⟨ b − z , t − z ⟩ = ξ ⟨ z , t − z ⟩ + ζ ⟨ z − y , t − z ⟩ ≤ 0. \langle b-z,t-z\rangle=\xi\langle z,t-z\rangle+\zeta\langle z-y,t-z\rangle\le0. ⟨ b − z , t − z ⟩ = ξ ⟨ z , t − z ⟩ + ζ ⟨ z − y , t − z ⟩ ≤ 0.
Here (a) follows from the fact that ξ z \xi z ξ z and ζ ( z − y ) \zeta(z-y) ζ ( z − y ) are normals to the balls B ( 0 , R ) B(0,R) B ( 0 , R ) and B ( y , r ) B(y,r) B ( y , r ) at the point z z z . This inequality characterizes the projection onto K K K and proves the claim, including the cases when the balls coincide or one of them is contained in the other.
Inequality (H.4) follows from Lemma A.3 with φ = P i \varphi=P_i φ = P i and L = L i L=L_i L = L i . The auxiliary point from the proof of this lemma also lies in the ball of radius 3 r 3r 3 r , so the smoothness bound applies to it. The convexity of P i P_i P i , which follows from Assumption 2.1, gives the nonnegativity of its deviation from the tangent. The cocoercivity of P P P follows from Lemma A.2: on the convex ball B ( y , r ) B(y,r) B ( y , r ) , the averaged Hessian of P P P lies between 0 0 0 and L ‾ I \overline{L}I L I . This is a local version of the classical cocoercivity property [3 ] .
For the local subproblem and the SVRG epoch, we keep the notation u u u , w w w , s s s , h s h_s h s , M M M , and L ‾ \overline{L} L from Section 4.
Lemma B.2 (decrease of the Lyapunov function over an epoch). Set
Ψ ( w , s ) = ∥ w − u ∥ 2 + 4 h s 2 ∑ j = 0 M − 1 ( 1 − h s λ ) j ∑ i = 1 n ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2 n L i / L ‾ . (15) \Psi(w,s) = \lVert w-u\rVert^2 + 4h_s^2 \sum_{j=0}^{M-1}(1-h_s\lambda)^j \sum_{i=1}^n \frac{\lVert\nabla P_i(s)-\nabla P_i(u)\rVert^2}{nL_i/\overline{L}}. \tag*{(15)} Ψ ( w , s ) = ∥ w − u ∥ 2 + 4 h s 2 j = 0 ∑ M − 1 ( 1 − h s λ ) j i = 1 ∑ n n L i / L ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2 . (15) Then, conditionally on the entire history before the start of the epoch,
E Ψ ( w + , s + ) ≤ 1 2 Ψ ( w , s ) , (16) \mathbb{E}\Psi(w^+,s^+) \le\frac{1}{2}\Psi(w,s), \tag*{(16)} E Ψ ( w + , s + ) ≤ 2 1 Ψ ( w , s ) , (16) Ψ ( y , y ) ≤ ( 1 + 2 L ‾ 3 λ ) ∥ y − u ∥ 2 . \Psi(y,y) \le\left(1+\frac{2\overline{L}}{3\lambda}\right)\lVert y-u\rVert^2. Ψ ( y , y ) ≤ ( 1 + 3 λ 2 L ) ∥ y − u ∥ 2 .
Proof.
▷ Let the random index I I I take the value i i i with probability L i / ( n L ‾ ) L_i/(n\overline{L}) L i / ( n L ) . Then
v I − ∇ P ( u ) = ∇ P I ( x ) − ∇ P I ( u ) L I / L ‾ + ( ∇ P ( s ) − ∇ P ( u ) − ∇ P I ( s ) − ∇ P I ( u ) L I / L ‾ ) . v_I-\nabla P(u)=\frac{\nabla P_I(x)-\nabla P_I(u)}{L_I/\overline{L}}+\left(\nabla P(s)-\nabla P(u)-\frac{\nabla P_I(s)-\nabla P_I(u)}{L_I/\overline{L}}\right). v I − ∇ P ( u ) = L I / L ∇ P I ( x ) − ∇ P I ( u ) + ( ∇ P ( s ) − ∇ P ( u ) − L I / L ∇ P I ( s ) − ∇ P I ( u ) ) .
The second term is centered, and the expectation of its squared norm does not exceed the sum from (B.2). For any point t ∈ B ( y , r ) t\in B(y,r) t ∈ B ( y , r ) ,
∑ i = 1 n ∥ ∇ P i ( t ) − ∇ P i ( u ) ∥ 2 n L i / L ‾ ≤ 2 L ‾ ( P ( t ) − P ( u ) − ⟨ ∇ P ( u ) , t − u ⟩ ) . (17) \sum_{i=1}^n\frac{\lVert\nabla P_i(t)-\nabla P_i(u)\rVert^2}{nL_i/\overline{L}}\le2\overline{L}\left(P(t)-P(u)-\langle\nabla P(u),t-u\rangle\right). \tag*{(17)} i = 1 ∑ n n L i / L ∥ ∇ P i ( t ) − ∇ P i ( u ) ∥ 2 ≤ 2 L ( P ( t ) − P ( u ) − ⟨ ∇ P ( u ) , t − u ⟩ ) . (17)
where (a) follows from the choice of the probabilities L i / ( n L ‾ ) L_i/(n\overline{L}) L i / ( n L ) , the bound (H.4), and the identity P = 1 n ∑ i P i P=\frac{1}{n}\sum_i P_i P = n 1 ∑ i P i . Therefore,
E ∥ v I − ∇ P ( u ) ∥ 2 ≤ ( a ) 2 ∑ i = 1 n ∥ ∇ P i ( x ) − ∇ P i ( u ) ∥ 2 n L i / L ‾ + 2 ∑ i = 1 n ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2 n L i / L ‾ \mathbb{E}\lVert v_I-\nabla P(u)\rVert^2 \underset{(a)}{\le} 2\sum_{i=1}^n\frac{\lVert\nabla P_i(x)-\nabla P_i(u)\rVert^2}{nL_i/\overline{L}}+2\sum_{i=1}^n\frac{\lVert\nabla P_i(s)-\nabla P_i(u)\rVert^2}{nL_i/\overline{L}} E ∥ v I − ∇ P ( u ) ∥ 2 ( a ) ≤ 2 i = 1 ∑ n n L i / L ∥ ∇ P i ( x ) − ∇ P i ( u ) ∥ 2 + 2 i = 1 ∑ n n L i / L ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2
≤ ( b ) 4 L ‾ ( P ( x ) − P ( u ) − ⟨ ∇ P ( u ) , x − u ⟩ ) + 2 ∑ i = 1 n ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2 n L i / L ‾ . \underset{(b)}{\le}4\overline{L}\left(P(x)-P(u)-\langle\nabla P(u),x-u\rangle\right)+2\sum_{i=1}^n\frac{\lVert\nabla P_i(s)-\nabla P_i(u)\rVert^2}{nL_i/\overline{L}}. ( b ) ≤ 4 L ( P ( x ) − P ( u ) − ⟨ ∇ P ( u ) , x − u ⟩ ) + 2 i = 1 ∑ n n L i / L ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2 .
Here (a) follows from the inequality
∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 , \lVert a+b\rVert^2\le2\lVert a\rVert^2+2\lVert b\rVert^2, ∥ a + b ∥ 2 ≤ 2 ∥ a ∥ 2 + 2 ∥ b ∥ 2 ,
and (b) from (B.4) with t = x t=x t = x .
▷ The optimality of u u u means that u = Π K ( u − h s ∇ P ( u ) ) u=\Pi_K(u-h_s\nabla P(u)) u = Π K ( u − h s ∇ P ( u )) . Therefore,
E ∥ x + − u ∥ 2 ≤ ( a ) ∥ x − u ∥ 2 − 2 h s ⟨ ∇ P ( x ) − ∇ P ( u ) , x − u ⟩ + h s 2 E ∥ v I − ∇ P ( u ) ∥ 2 \mathbb{E}\lVert x^+-u\rVert^2 \underset{(a)}{\le}\lVert x-u\rVert^2-2h_s\langle\nabla P(x)-\nabla P(u),x-u\rangle+h_s^2\mathbb{E}\lVert v_I-\nabla P(u)\rVert^2 E ∥ x + − u ∥ 2 ( a ) ≤ ∥ x − u ∥ 2 − 2 h s ⟨ ∇ P ( x ) − ∇ P ( u ) , x − u ⟩ + h s 2 E ∥ v I − ∇ P ( u ) ∥ 2
≤ ( b ) ( 1 − h s λ ) ∥ x − u ∥ 2 − 4 3 h s ( P ( x ) − P ( u ) − ⟨ ∇ P ( u ) , x − u ⟩ ) + 2 h s 2 ∑ i = 1 n ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2 n L i / L ‾ . \underset{(b)}{\le}(1-h_s\lambda)\lVert x-u\rVert^2-\frac{4}{3}h_s\left(P(x)-P(u)-\langle\nabla P(u),x-u\rangle\right)+2h_s^2\sum_{i=1}^n\frac{\lVert\nabla P_i(s)-\nabla P_i(u)\rVert^2}{nL_i/\overline{L}}. ( b ) ≤ ( 1 − h s λ ) ∥ x − u ∥ 2 − 3 4 h s ( P ( x ) − P ( u ) − ⟨ ∇ P ( u ) , x − u ⟩ ) + 2 h s 2 i = 1 ∑ n n L i / L ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2 .
Here (a) follows from the nonexpansiveness of the projection and the unbiasedness of v I v_I v I , and (b) from the strong convexity of P P P , the previous item, and the identity 2 h s − 4 h s 2 L ‾ = 4 3 h s 2h_s-4h_s^2\overline{L}=\frac{4}{3}h_s 2 h s − 4 h s 2 L = 3 4 h s .
▷ By (H.5), we have 0 < h s λ ≤ 1 / 6 0<h_s\lambda\le1/6 0 < h s λ ≤ 1/6 and h s λ M ≥ 7 / 10 h_s\lambda M\ge7/10 h s λ M ≥ 7/10 . We unroll the obtained recursive inequality over the whole epoch. Since 8 h s 2 L ‾ = 4 3 h s 8h_s^2\overline{L}=\frac{4}{3}h_s 8 h s 2 L = 3 4 h s , the terms with the linearization remainders at the points x j x_j x j cancel, and
E Ψ ( w + , s + ) ≤ ( a ) ( 1 − h s λ ) M ∥ w − u ∥ 2 + 2 h s 2 ∑ j = 0 M − 1 ( 1 − h s λ ) j ∑ i = 1 n ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2 n L i / L ‾ ≤ ( b ) 1 2 Ψ ( w , s ) . \mathbb{E}\Psi(w^+,s^+) \underset{(a)}{\le}(1-h_s\lambda)^M\lVert w-u\rVert^2+2h_s^2\sum_{j=0}^{M-1}(1-h_s\lambda)^j\sum_{i=1}^n\frac{\lVert\nabla P_i(s)-\nabla P_i(u)\rVert^2}{nL_i/\overline{L}}\underset{(b)}{\le}\frac{1}{2}\Psi(w,s). E Ψ ( w + , s + ) ( a ) ≤ ( 1 − h s λ ) M ∥ w − u ∥ 2 + 2 h s 2 j = 0 ∑ M − 1 ( 1 − h s λ ) j i = 1 ∑ n n L i / L ∥ ∇ P i ( s ) − ∇ P i ( u ) ∥ 2 ( b ) ≤ 2 1 Ψ ( w , s ) .
Here (a) follows from the recursive inequality of the previous item, the bound (B.4) with t = s + t=s^{+} t = s + , and Jensen’s inequality for the convex function t ↦ P ( t ) − P ( u ) − ⟨ ∇ P ( u ) , t − u ⟩ t\mapsto P(t)-P(u)-\langle\nabla P(u),t-u\rangle t ↦ P ( t ) − P ( u ) − ⟨ ∇ P ( u ) , t − u ⟩ , and (b) from ( 1 − h s λ ) M ≤ e − 7 / 10 < 1 / 2 (1-h_s\lambda)^M\le e^{-7/10}<1/2 ( 1 − h s λ ) M ≤ e − 7/10 < 1/2 . Finally, local smoothness gives
P ( y ) − P ( u ) − ⟨ ∇ P ( u ) , y − u ⟩ ≤ L ˉ 2 ∥ y − u ∥ 2 P(y)-P(u)-\langle\nabla P(u),y-u\rangle\le\frac{\bar{L}}{2}\lVert y-u\rVert^2 P ( y ) − P ( u ) − ⟨ ∇ P ( u ) , y − u ⟩ ≤ 2 L ˉ ∥ y − u ∥ 2
so, by (B.4), the sum from (B.2) with s = y s=y s = y does not exceed L ˉ 2 ∥ y − u ∥ 2 \bar{L}^2\lVert y-u\rVert^2 L ˉ 2 ∥ y − u ∥ 2 . Together with
∑ j = 0 M − 1 ( 1 − h s λ ) j ≤ 1 h s λ \sum_{j=0}^{M-1}(1-h_s\lambda)^j\le\frac{1}{h_s\lambda} j = 0 ∑ M − 1 ( 1 − h s λ ) j ≤ h s λ 1
this gives
Ψ ( y , y ) ≤ ( 1 + 4 h s L ˉ 2 λ ) ∥ y − u ∥ 2 = ( 1 + 2 L ˉ 3 λ ) ∥ y − u ∥ 2 . \Psi(y,y)\le\left(1+\frac{4h_s\bar{L}^2}{\lambda}\right)\lVert y-u\rVert^2=\left(1+\frac{2\bar{L}}{3\lambda}\right)\lVert y-u\rVert^2. Ψ ( y , y ) ≤ ( 1 + λ 4 h s L ˉ 2 ) ∥ y − u ∥ 2 = ( 1 + 3 λ 2 L ˉ ) ∥ y − u ∥ 2 .
Proof of Lemma H.2 ▲.
▷ By Lemma H.1, the map I − ∇ P / L ˉ I-\nabla P/\bar{L} I − ∇ P / L ˉ is nonexpansive on K \mathcal{K} K . Comparing the projections of the points w − ∇ P ( w ) / L ˉ w-\nabla P(w)/\bar{L} w − ∇ P ( w ) / L ˉ and u − ∇ P ( u ) / L ˉ u-\nabla P(u)/\bar{L} u − ∇ P ( u ) / L ˉ , we obtain the first bound in (H.11). For the second bound, we have
∥ e ∥ 2 = L ˉ 2 ∥ w − z ∥ 2 − 2 L ˉ ⟨ ∇ P ( w ) − ∇ P ( z ) , w − z ⟩ + ∥ ∇ P ( w ) − ∇ P ( z ) ∥ 2 ≤ L ˉ ⟨ e , w − z ⟩ ≤ ( a ) L ˉ ⟨ e , w − u ⟩ ≤ ( b ) L ˉ ∥ e ∥ ∥ w − u ∥ ≤ ( c ) L ˉ ∥ e ∥ ∥ w − u ∥ . \begin{aligned}
\lVert e\rVert^2=\bar{L}^2\lVert w-z\rVert^2-2\bar{L}\langle\nabla P(w)-\nabla P(z),w-z\rangle+\lVert\nabla P(w)-\nabla P(z)\rVert^2 \\
&\leq\bar{L}\langle e,w-z\rangle\underset{(a)}{\leq}\bar{L}\langle e,w-u\rangle\underset{(b)}{\leq}\bar{L}\lVert e\rVert\lVert w-u\rVert\underset{(c)}{\leq}\bar{L}\lVert e\rVert\lVert w-u\rVert.
\end{aligned} ∥ e ∥ 2 = L ˉ 2 ∥ w − z ∥ 2 − 2 L ˉ ⟨ ∇ P ( w ) − ∇ P ( z ) , w − z ⟩ + ∥ ∇ P ( w ) − ∇ P ( z ) ∥ 2 ≤ L ˉ ⟨ e , w − z ⟩ ( a ) ≤ L ˉ ⟨ e , w − u ⟩ ( b ) ≤ L ˉ ∥ e ∥ ∥ w − u ∥ ( c ) ≤ L ˉ ∥ e ∥ ∥ w − u ∥ .
Here (a) follows from the cocoercivity of ∇ P \nabla P ∇ P , (b) from the inequality
⟨ e , z − u ⟩ ≥ P ( z ) − P ( u ) ≥ 0 , \langle e,z-u\rangle\geq P(z)-P(u)\geq0, ⟨ e , z − u ⟩ ≥ P ( z ) − P ( u ) ≥ 0 ,
which holds since e ∈ ∇ P ( z ) + N K ( z ) e\in\nabla P(z)+N_{\mathcal{K}}(z) e ∈ ∇ P ( z ) + N K ( z ) by the definition of the projection, and (c) from the Cauchy–Schwarz inequality.
▷ If ∥ w − u ∥ \lVert w-u\rVert ∥ w − u ∥ satisfies the condition of the lemma, then
∥ e ∥ ≤ L ˉ ∥ w − u ∥ ≤ 4 5 λ ( ∥ y − u ∥ − ∥ w − u ∥ ) ≤ 4 5 λ ∥ y − z ∥ ≤ 4 5 ( λ + β ) ∥ y − z ∥ . \lVert e\rVert\leq\bar{L}\lVert w-u\rVert\leq\frac{4}{5}\lambda\bigl(\lVert y-u\rVert-\lVert w-u\rVert\bigr)\leq\frac{4}{5}\lambda\lVert y-z\rVert\leq\frac{4}{5}(\lambda+\beta)\lVert y-z\rVert. ∥ e ∥ ≤ L ˉ ∥ w − u ∥ ≤ 5 4 λ ( ∥ y − u ∥ − ∥ w − u ∥ ) ≤ 5 4 λ ∥ y − z ∥ ≤ 5 4 ( λ + β ) ∥ y − z ∥ .
The second-to-last step follows from ∥ z − u ∥ ≤ ∥ w − u ∥ \lVert z-u\rVert\leq\lVert w-u\rVert ∥ z − u ∥ ≤ ∥ w − u ∥ and the triangle inequality. Hence the second condition in (H.10) holds. If y = u y=u y = u , then Ψ ( y , y ) = 0 \Psi(y,y)=0 Ψ ( y , y ) = 0 , after the first epoch w = u w=u w = u almost surely, and the triple is accepted. If y ≠ u y\ne u y = u , then after j j j epochs Lemma B.2 and Markov’s inequality give
Pr ( T e p > j ) ≤ Pr ( ∥ w j − u ∥ > 4 λ 5 L ˉ + 4 λ ∥ y − u ∥ ) ≤ ( 1 + 5 L ˉ 4 λ ) 2 E Ψ ( w j , s j ) ∥ y − u ∥ 2 ≤ Γ 2 − j . \begin{aligned}
\Pr(T_{\mathrm{ep}}>j)\leq\Pr\left(\lVert w_j-u\rVert>\frac{4\lambda}{5\bar{L}+4\lambda}\lVert y-u\rVert\right) \\
&\leq\left(1+\frac{5\bar{L}}{4\lambda}\right)^2\frac{\mathbb{E}\Psi(w_j,s_j)}{\lVert y-u\rVert^2}\leq\Gamma2^{-j}.
\end{aligned} Pr ( T ep > j ) ≤ Pr ( ∥ w j − u ∥ > 5 L ˉ + 4 λ 4 λ ∥ y − u ∥ ) ≤ ( 1 + 4 λ 5 L ˉ ) 2 ∥ y − u ∥ 2 E Ψ ( w j , s j ) ≤ Γ 2 − j .
In deriving this bound, the epochs can be conceptually continued even after the first acceptance, using independent samples. The sum of the right-hand sides is finite, which proves both claims about the stopping time.
Decrease of the Lyapunov function
Lemma B.3 (decrease of the Lyapunov function). In all states of the method, the following conditions hold:
x , v ∈ B ( 0 , R ) , A ≥ R 2 / ( 2 Δ ) , h ≥ 1 2 λ ↓ ( A ) , A ( f ( x ) − f ∗ ) + 1 2 ∥ v − x ∗ ∥ 2 ≤ R 2 . (18) \begin{aligned}
x,v &\in B(0,R),\\
\mathcal{A} &\ge R^2/(2\Delta),\\
h &\ge\tfrac{1}{2}\lambda_{\downarrow}(\mathcal{A}),\\
\mathcal{A}(f(x)-f^*)+\tfrac{1}{2}\lVert v-x^*\rVert^2 &\le R^2. \tag*{(18)}
\end{aligned} x , v A h A ( f ( x ) − f ∗ ) + 2 1 ∥ v − x ∗ ∥ 2 ∈ B ( 0 , R ) , ≥ R 2 / ( 2Δ ) , ≥ 2 1 λ ↓ ( A ) , ≤ R 2 . (18) Denote the left-hand side of the last inequality by Φ \Phi Φ (the Lyapunov function). We always have α ≤ 1 2 ( r / R ) 2 / 3 ≤ 1 2 \alpha\le\tfrac{1}{2}(r/R)^{2/3} \le\tfrac{1}{2} α ≤ 2 1 ( r / R ) 2/3 ≤ 2 1 and λ ≤ 2 h \lambda\le2h λ ≤ 2 h . Accepting a point proposed by the heuristic block does not change v v v , A \mathcal{A} A , h h h and does not increase Φ \Phi Φ . If (G.9) is violated, the halfspace H y \mathcal{H}_y H y contains x ∗ x^* x ∗ and
Φ + ≤ Φ − 9 r 2 128 α 2 . (19) \Phi^+ \le\Phi-\frac{9r^2}{128\alpha^2}. \tag*{(19)} Φ + ≤ Φ − 128 α 2 9 r 2 . (19) Let y = ( 1 − α ) x + α v y=(1-\alpha)x+\alpha v y = ( 1 − α ) x + α v be the center of the local subproblem from (G.8), and let z z z be the point returned by the solver. If the triple ( z , g , β ) (z,g,\beta) ( z , g , β ) is accepted by the second condition in (H.10), then
Φ + ≤ Φ − 9 50 ∥ y − z ∥ 2 α 2 . (20) \Phi^+ \le\Phi-\frac{9}{50}\frac{\lVert y-z\rVert^2}{\alpha^2}. \tag*{(20)} Φ + ≤ Φ − 50 9 α 2 ∥ y − z ∥ 2 . (20) Under any mode of stopping, the optimality gap of the output does not exceed ε / 2 \varepsilon/2 ε /2 .
Proof.
▷ At the initial moment,
Φ 0 = R 2 2 Δ ( f ( 0 ) − f ∗ ) + 1 2 ∥ x ∗ ∥ 2 ≤ R 2 . \Phi_0=\frac{R^2}{2\Delta}(f(0)-f^*)+\frac{1}{2}\lVert x^*\rVert^2\le R^2. Φ 0 = 2Δ R 2 ( f ( 0 ) − f ∗ ) + 2 1 ∥ x ∗ ∥ 2 ≤ R 2 .
The function λ ↓ \lambda_{\downarrow} λ ↓ is nonincreasing; hence λ ↓ ( A ) ≤ λ ↑ \lambda_{\downarrow}(\mathcal{A})\le\lambda_{\uparrow} λ ↓ ( A ) ≤ λ ↑ and λ ≥ λ ↓ ( A ) \lambda\ge\lambda_{\downarrow}(\mathcal{A}) λ ≥ λ ↓ ( A ) . The identity λ A α 2 = 1 − α \lambda\mathcal{A}\alpha^2=1-\alpha λ A α 2 = 1 − α implies α ≤ 1 2 ( r / R ) 2 / 3 \alpha\le\tfrac{1}{2}(r/R)^{2/3} α ≤ 2 1 ( r / R ) 2/3 , since the function t ↦ ( 1 − t ) / t 2 t\mapsto(1-t)/t^2 t ↦ ( 1 − t ) / t 2 is decreasing. The inequality h ≥ λ ↓ ( A ) / 2 h\ge\lambda_{\downarrow}(\mathcal{A})/2 h ≥ λ ↓ ( A ) /2 implies λ ≤ 2 h \lambda\le2h λ ≤ 2 h .
▷ By the rule for accepting a heuristic point, z ∈ B ( 0 , R ) z\in B(0,R) z ∈ B ( 0 , R ) and f ( z ) ≤ f ( x ) f(z)\le f(x) f ( z ) ≤ f ( x ) . Replacing x x x by z z z changes the Lyapunov function by A ( f ( z ) − f ( x ) ) ≤ 0 \mathcal{A}(f(z)-f(x))\le0 A ( f ( z ) − f ( x )) ≤ 0 , while v v v , A \mathcal{A} A , h h h do not change. Therefore, conditions (B.5) are preserved.
▷ From Assumption 2.1 and Corollary G.3 we obtain
⟨ ∇ f ( y ) , x ∗ − y ⟩ ≤ f ∗ − f ( y ) ≤ − ℓ ( y ) − 3 2 L 0 − 2 L 1 2 δ 2 L 1 2 , \langle\nabla f(y),x^*-y\rangle\le f^*-f(y)\le-\frac{\ell(y)-\frac{3}{2}L_0-2L_1^2\delta}{2L_1^2}, ⟨ ∇ f ( y ) , x ∗ − y ⟩ ≤ f ∗ − f ( y ) ≤ − 2 L 1 2 ℓ ( y ) − 2 3 L 0 − 2 L 1 2 δ ,
that is, x ∗ ∈ H y x^*\in\mathcal{H}_y x ∗ ∈ H y . Suppose condition (G.9) is violated. Since L 1 r ≤ 1 / 2 L_1r\le1/2 L 1 r ≤ 1/2 ,
ℓ ( y ) − 3 2 L 0 − 2 L 1 2 ( δ + R 2 / A ) 2 L 1 2 > ( a ) ( 1 − e − L 1 r ) ℓ ( y ) 2 L 1 2 ≥ ( b ) 3 r ℓ ( y ) 8 L 1 > 0. \frac{\ell(y)-\frac{3}{2}L_0-2L_1^2(\delta+R^2/\mathcal{A})}{2L_1^2}
\mathrel{\overset{(a)}{>}}
\frac{(1-e^{-L_1r})\ell(y)}{2L_1^2}
\mathrel{\overset{(b)}{\ge}}
\frac{3r\ell(y)}{8L_1}>0. 2 L 1 2 ℓ ( y ) − 2 3 L 0 − 2 L 1 2 ( δ + R 2 / A ) > ( a ) 2 L 1 2 ( 1 − e − L 1 r ) ℓ ( y ) ≥ ( b ) 8 L 1 3 r ℓ ( y ) > 0.
Here (a) follows from the violation of (G.9), and (b) from 1 − e − s ≥ 3 s / 4 1-e^{-s}\ge3s/4 1 − e − s ≥ 3 s /4 for 0 ≤ s ≤ 1 / 2 0\le s\le1/2 0 ≤ s ≤ 1/2 . Since f ( x ) − f ∗ ≤ Φ / A ≤ R 2 / A f(x)-f^*\le\Phi/\mathcal{A}\le R^2/\mathcal{A} f ( x ) − f ∗ ≤ Φ/ A ≤ R 2 / A ,
⟨ ∇ f ( y ) , y − x ⟩ ≥ f ( y ) − f ( x ) ≥ f ( y ) − f ∗ − R 2 A ≥ ℓ ( y ) − 3 2 L 0 − 2 L 1 2 ( δ + R 2 / A ) 2 L 1 2 . \langle\nabla f(y),y-x\rangle\ge f(y)-f(x)\ge f(y)-f^*-\frac{R^2}{\mathcal{A}}\ge\frac{\ell(y)-\frac{3}{2}L_0-2L_1^2(\delta+R^2/\mathcal{A})}{2L_1^2}. ⟨ ∇ f ( y ) , y − x ⟩ ≥ f ( y ) − f ( x ) ≥ f ( y ) − f ∗ − A R 2 ≥ 2 L 1 2 ℓ ( y ) − 2 3 L 0 − 2 L 1 2 ( δ + R 2 / A ) .
Step (b) follows from Corollary G.3. The right-hand side is positive; consequently, ∇ f ( y ) ≠ 0 \nabla f(y)\ne0 ∇ f ( y ) = 0 . Using v − y = ( 1 − α ) ( y − x ) / α v-y=(1-\alpha)(y-x)/\alpha v − y = ( 1 − α ) ( y − x ) / α and ∥ ∇ f ( y ) ∥ ≤ ℓ ( y ) / L 1 \lVert\nabla f(y)\rVert\le\ell(y)/L_1 ∥ ∇ f ( y )∥ ≤ ℓ ( y ) / L 1 , we obtain
dist ( v , H y ) = 1 ∥ ∇ f ( y ) ∥ ( ⟨ ∇ f ( y ) , v − y ⟩ + ℓ ( y ) − 3 2 L 0 − 2 L 1 2 δ 2 L 1 2 ) ≥ ℓ ( y ) − 3 2 L 0 − 2 L 1 2 ( δ + R 2 / A ) 2 L 1 2 α ∥ ∇ f ( y ) ∥ ≥ 3 r 8 α . \begin{aligned}
\operatorname{dist}(v,\mathcal{H}_y)
&=\frac{1}{\lVert\nabla f(y)\rVert}\left(\langle\nabla f(y),v-y\rangle+\frac{\ell(y)-\frac{3}{2}L_0-2L_1^2\delta}{2L_1^2}\right)\\
&\ge\frac{\ell(y)-\frac{3}{2}L_0-2L_1^2(\delta+R^2/\mathcal{A})}{2L_1^2\alpha\lVert\nabla f(y)\rVert}
\ge\frac{3r}{8\alpha}.
\end{aligned} dist ( v , H y ) = ∥ ∇ f ( y )∥ 1 ( ⟨ ∇ f ( y ) , v − y ⟩ + 2 L 1 2 ℓ ( y ) − 2 3 L 0 − 2 L 1 2 δ ) ≥ 2 L 1 2 α ∥ ∇ f ( y )∥ ℓ ( y ) − 2 3 L 0 − 2 L 1 2 ( δ + R 2 / A ) ≥ 8 α 3 r .
The projection onto H y \mathcal{H}_y H y decreases the squared distance to x ∗ x^* x ∗ by at least the squared distance from v v v to H y \mathcal{H}_y H y , and the projection onto B ( 0 , R ) B(0,R) B ( 0 , R ) does not increase it, by Assumption 2.3. This gives (B.6). The argument does not require x x x to belong to the halfspace.
▷ Suppose the triple is accepted by the second condition in (H.10). Then g g g is a subgradient of f + I B ( 0 , R ) f+I_{B(0,R)} f + I B ( 0 , R ) at the point z z z , and
Φ + − Φ ≤ (a) γ A ⟨ g , z − x ⟩ + γ α λ ⟨ g , z − v ⟩ + 1 2 γ 2 α λ 2 ∥ g ∥ 2 = − γ ( A + α λ ) ⟨ g , y − z ⟩ + 1 2 γ 2 α λ 2 ∥ g ∥ 2 = 1 2 γ 2 α λ 2 ( ∥ e ∥ 2 − ( λ + β ) 2 ∥ y − z ∥ 2 ) ≤ (b) − 9 50 γ 2 α λ 2 ( λ + β ) 2 ∥ y − z ∥ 2 = − 9 50 α 2 ∥ y − z ∥ 2 . \begin{aligned}
\Phi^+-\Phi\underset{\text{(a)}}{\leq} \gamma\mathcal{A}\langle g,z-x\rangle+\gamma\alpha_\lambda\langle g,z-v\rangle+\frac{1}{2}\gamma^2\alpha_\lambda^2\lVert g\rVert^2=-\gamma(\mathcal{A}+\alpha_\lambda)\langle g,y-z\rangle+\frac{1}{2}\gamma^2\alpha_\lambda^2\lVert g\rVert^2 \\
&=\frac{1}{2}\gamma^2\alpha_\lambda^2\left(\lVert e\rVert^2-(\lambda+\beta)^2\lVert y-z\rVert^2\right)\underset{\text{(b)}}{\leq}-\frac{9}{50}\gamma^2\alpha_\lambda^2(\lambda+\beta)^2\lVert y-z\rVert^2=-\frac{9}{50\alpha^2}\lVert y-z\rVert^2.
\end{aligned} Φ + − Φ (a) ≤ γ A ⟨ g , z − x ⟩ + γ α λ ⟨ g , z − v ⟩ + 2 1 γ 2 α λ 2 ∥ g ∥ 2 = − γ ( A + α λ ) ⟨ g , y − z ⟩ + 2 1 γ 2 α λ 2 ∥ g ∥ 2 = 2 1 γ 2 α λ 2 ( ∥ e ∥ 2 − ( λ + β ) 2 ∥ y − z ∥ 2 ) (b) ≤ − 50 9 γ 2 α λ 2 ( λ + β ) 2 ∥ y − z ∥ 2 = − 50 α 2 9 ∥ y − z ∥ 2 .
In (a) we used the convexity of f f f (Assumption 2.1), the combination (G.11), and the nonexpansiveness of the projection, which preserves x ∗ x^* x ∗ by Assumption 2.3; (b) follows from ∥ e ∥ ≤ 5 4 ( λ + β ) ∥ y − z ∥ \lVert e\rVert\leq\frac{5}{4}(\lambda+\beta)\lVert y-z\rVert ∥ e ∥ ≤ 4 5 ( λ + β ) ∥ y − z ∥ . In the last equality we used the identities γ ( λ + β ) = λ \gamma(\lambda+\beta)=\lambda γ ( λ + β ) = λ and λ α λ = 1 / α \lambda\alpha_\lambda=1/\alpha λ α λ = 1/ α . If β > 0 \beta>0 β > 0 , the displacement equals r r r . All new points belong to B ( 0 , R ) B(0,R) B ( 0 , R ) , and A \mathcal{A} A does not decrease. On an up-step, h + = 2 λ ≥ λ ↓ ( A + ) / 2 h^+=2\lambda\geq\lambda_{\downarrow}(\mathcal{A}^+)/2 h + = 2 λ ≥ λ ↓ ( A + ) /2 ; on a down-step, h + = λ / 2 ≥ λ ↓ ( A + ) / 2 h^+=\lambda/2\geq\lambda_{\downarrow}(\mathcal{A}^+)/2 h + = λ /2 ≥ λ ↓ ( A + ) /2 . Therefore, conditions (B.5) are preserved.
▷ If a triple with ∥ g ∥ ≤ ε / ( 4 R ) \lVert g\rVert\leq\varepsilon/(4R) ∥ g ∥ ≤ ε / ( 4 R ) is accepted, then for z , x ∗ ∈ B ( 0 , R ) z,x^*\in B(0,R) z , x ∗ ∈ B ( 0 , R ) ,
f ( z ) − f ∗ ≤ (a) ⟨ g , z − x ∗ ⟩ ≤ (b) 2 R ∥ g ∥ ≤ ε / 2. f(z)-f^*\underset{\text{(a)}}{\leq}\langle g,z-x^*\rangle\underset{\text{(b)}}{\leq}2R\lVert g\rVert\leq\varepsilon/2. f ( z ) − f ∗ (a) ≤ ⟨ g , z − x ∗ ⟩ (b) ≤ 2 R ∥ g ∥ ≤ ε /2.
where (a) follows from g ∈ ∂ ( f + I B ( 0 , R ) ) ( z ) g\in\partial(f+I_{B(0,R)})(z) g ∈ ∂ ( f + I B ( 0 , R ) ) ( z ) , and (b) from the Cauchy–Schwarz inequality and Assumption 2.3. Stopping due to the growth of A \mathcal{A} A gives f ( x ) − f ∗ ≤ R 2 / A ≤ ε / 2 f(x)-f^*\leq R^2/\mathcal{A}\leq\varepsilon/2 f ( x ) − f ∗ ≤ R 2 / A ≤ ε /2 . By (G.1) and Proposition B.1 , we have f ~ b e s t ≤ f ∗ \widetilde{f}_{\mathrm{best}}\leq f^* f best ≤ f ∗ . Therefore, when stopping by criterion (G.2),
f ( w b e s t ) − f ∗ ≤ f ( w b e s t ) − f ~ b e s t ≤ ε / 2. f(w_{\mathrm{best}})-f^*\leq f(w_{\mathrm{best}})-\widetilde{f}_{\mathrm{best}}\leq\varepsilon/2. f ( w best ) − f ∗ ≤ f ( w best ) − f best ≤ ε /2.
Finally, a triple accepted by the second condition with β = 0 \beta=0 β = 0 and λ = ε / ( 8 r R ) \lambda=\varepsilon/(8rR) λ = ε / ( 8 r R ) already satisfies
∥ g ∥ ≤ 9 5 ⋅ ε 8 r R r = 9 ε 40 R < ε 4 R . \lVert g\rVert\leq\frac{9}{5}\cdot\frac{\varepsilon}{8rR}r=\frac{9\varepsilon}{40R}<\frac{\varepsilon}{4R}. ∥ g ∥ ≤ 5 9 ⋅ 8 r R ε r = 40 R 9 ε < 4 R ε .
Therefore, no separate stopping rule is needed for λ = ε / ( 8 r R ) \lambda=\varepsilon/(8rR) λ = ε / ( 8 r R ) .
Number of main iterations
Lemma B.4 (number of main iterations). For any choice of the heuristic in Algorithm 1, the method almost surely stops after at most K o u t K_{\mathrm{out}} K out main iterations, where K o u t K_{\mathrm{out}} K out is a deterministic quantity satisfying the bound
K o u t ≲ ( R r ) 2 / 3 ( 1 + log Δ ε ) + log ( 2 + r R λ ↑ ε ) + L 1 R n log Δ ε + ( L 0 + L 1 2 δ ) R 2 n ε . (21) K_{\mathrm{out}}\lesssim\left(\frac{R}{r}\right)^{2/3}\left(1+\log\frac{\Delta}{\varepsilon}\right)+\log\left(2+\frac{rR\lambda_{\uparrow}}{\varepsilon}\right)+\frac{L_1R}{\sqrt{n}}\log\frac{\Delta}{\varepsilon}+\sqrt{\frac{(L_0+L_1^2\delta)R^2}{n\varepsilon}}. \tag*{(21)} K out ≲ ( r R ) 2/3 ( 1 + log ε Δ ) + log ( 2 + ε r R λ ↑ ) + n L 1 R log ε Δ + n ε ( L 0 + L 1 2 δ ) R 2 . (21)
Proof. We count only the iterations after which the method has not yet stopped; on them, A < 2 R 2 / ε \mathcal{A}<2R^2/\varepsilon A < 2 R 2 / ε . Accepting the output of the heuristic does not change A \mathcal{A} A , h h h and does not increase the Lyapunov function Φ \Phi Φ .
▷ Let n U n_U n U be the number of iterations that increase the regularization. By Lemma B.3 , on each of them
Φ − Φ + ≥ 9 r 2 128 α 2 ≥ 9 32 R 2 ( r R ) 2 / 3 , α ≤ 1 2 ( r R ) 2 / 3 . \Phi-\Phi^+\geq\frac{9r^2}{128\alpha^2}\geq\frac{9}{32}R^2\left(\frac{r}{R}\right)^{2/3},\qquad\alpha\leq\frac{1}{2}\left(\frac{r}{R}\right)^{2/3}. Φ − Φ + ≥ 128 α 2 9 r 2 ≥ 32 9 R 2 ( R r ) 2/3 , α ≤ 2 1 ( R r ) 2/3 .
Since 0 ≤ Φ ≤ R 2 0\leq\Phi\leq R^2 0 ≤ Φ ≤ R 2 , we have n U ≤ 32 9 ( R / r ) 2 / 3 n_U\leq\frac{32}{9}(R/r)^{2/3} n U ≤ 9 32 ( R / r ) 2/3 . Each such iteration increases h h h by at most a factor of four. An iteration with β = 0 \beta=0 β = 0 and λ > λ ↓ ( A ) \lambda>\lambda_{\downarrow}(\mathcal{A}) λ > λ ↓ ( A ) decreases h h h by at least a factor of two, and for λ = λ ↓ ( A ) \lambda= \lambda_{\downarrow}(\mathcal{A}) λ = λ ↓ ( A ) it does not increase h h h . From h ≥ λ ↓ ( A ) / 2 ≥ ε / ( 16 r R ) h \ge\lambda_{\downarrow}(\mathcal{A})/2 \ge\varepsilon/(16rR) h ≥ λ ↓ ( A ) /2 ≥ ε / ( 16 r R ) it follows that the number n D n_{\mathrm{D}} n D of iterations with β = 0 \beta= 0 β = 0 and λ > λ ↓ ( A ) \lambda> \lambda_{\downarrow}(\mathcal{A}) λ > λ ↓ ( A ) satisfies
h 2 n D ≤ λ ↑ 4 n U , n D ≤ 2 n U + log 2 16 r R λ ↑ ε . h2^{n_{\mathrm{D}}} \le\lambda_{\uparrow}4^{n_{\mathrm{U}}}, \qquad n_{\mathrm{D}} \le2n_{\mathrm{U}} + \log_{2}\frac{16rR\lambda_{\uparrow}}{\varepsilon}. h 2 n D ≤ λ ↑ 4 n U , n D ≤ 2 n U + log 2 ε 16 r R λ ↑ .
▷ Consider the remaining iterations with β = 0 \beta= 0 β = 0 and λ = λ ↓ ( A ) \lambda= \lambda_{\downarrow}(\mathcal{A}) λ = λ ↓ ( A ) . On them, A + = A + a λ \mathcal{A}^{+} = \mathcal{A} + a_{\lambda} A + = A + a λ . If the maximum in (G.7) is attained at the first term, then α = 1 2 ( r / R ) 2 / 3 \alpha= \frac{1}{2}(r/R)^{2/3} α = 2 1 ( r / R ) 2/3 and
log A + A = − log ( 1 − α ) ≥ α = 1 2 ( r / R ) 2 / 3 . \log\frac{\mathcal{A}^{+}}{\mathcal{A}} = -\log(1-\alpha) \ge\alpha= \frac{1}{2}(r/R)^{2/3}. log A A + = − log ( 1 − α ) ≥ α = 2 1 ( r / R ) 2/3 .
Since A 0 = R 2 / ( 2 Δ ) \mathcal{A}_{0} = R^{2}/(2\Delta) A 0 = R 2 / ( 2Δ ) and, before stopping, A < 2 R 2 / ε \mathcal{A} < 2R^{2}/\varepsilon A < 2 R 2 / ε , the total growth of log A \log\mathcal{A} log A is less than log ( 4 Δ / ε ) \log(4\Delta/\varepsilon) log ( 4Δ/ ε ) . Therefore, there are at most 2 ( R / r ) 2 / 3 log ( 4 Δ / ε ) 2(R/r)^{2/3}\log(4\Delta/\varepsilon) 2 ( R / r ) 2/3 log ( 4Δ/ ε ) such iterations.
▷ If the maximum is attained at the second term, then
λ = 1 n ( 3 2 L 0 + 2 L 1 2 δ + 2 L 1 2 R 2 A ) . \lambda= \frac{1}{n}\left(\frac{3}{2}L_{0}+2L_{1}^{2}\delta+\frac{2L_{1}^{2}R^{2}}{\mathcal{A}}\right). λ = n 1 ( 2 3 L 0 + 2 L 1 2 δ + A 2 L 1 2 R 2 ) .
If 3 2 L 0 + 2 L 1 2 δ ≥ 2 L 1 2 R 2 / A \frac{3}{2}L_{0}+2L_{1}^{2}\delta\ge2L_{1}^{2}R^{2}/\mathcal{A} 2 3 L 0 + 2 L 1 2 δ ≥ 2 L 1 2 R 2 / A , then λ ≤ ( 3 L 0 + 4 L 1 2 δ ) / n \lambda\le(3L_{0}+4L_{1}^{2}\delta)/n λ ≤ ( 3 L 0 + 4 L 1 2 δ ) / n . From a λ ≥ A / λ a_{\lambda} \ge\sqrt{\mathcal{A}/\lambda} a λ ≥ A / λ and A + ≤ 2 A \mathcal{A}^{+} \le2\mathcal{A} A + ≤ 2 A we obtain
A + − A = a λ A + a λ + A ≥ 1 ( 1 + 2 ) λ ≥ n ( 1 + 2 ) 3 L 0 + 4 L 1 2 δ . \sqrt{\mathcal{A}^{+}}-\sqrt{\mathcal{A}} = \frac{a_{\lambda}}{\sqrt{\mathcal{A}+a_{\lambda}}+\sqrt{\mathcal{A}}}
\ge\frac{1}{(1+\sqrt{2})\sqrt{\lambda}} \ge\frac{\sqrt{n}}{(1+\sqrt{2})\sqrt{3L_{0}+4L_{1}^{2}\delta}}. A + − A = A + a λ + A a λ ≥ ( 1 + 2 ) λ 1 ≥ ( 1 + 2 ) 3 L 0 + 4 L 1 2 δ n .
Since A < 2 R 2 / ε \sqrt{\mathcal{A}} < \sqrt{2R^{2}/\varepsilon} A < 2 R 2 / ε , the number of these iterations does not exceed
( 1 + 2 ) 2 ( 3 L 0 + 4 L 1 2 δ ) R 2 n ε . (1+\sqrt{2})\sqrt{\frac{2(3L_{0}+4L_{1}^{2}\delta)R^{2}}{n\varepsilon}}. ( 1 + 2 ) n ε 2 ( 3 L 0 + 4 L 1 2 δ ) R 2 .
In the opposite case, λ A < 4 ( L 1 R ) 2 / n \lambda\mathcal{A} < 4(L_{1}R)^{2}/n λ A < 4 ( L 1 R ) 2 / n . Since α ≤ 1 / 2 \alpha\le1/2 α ≤ 1/2 , we have
α 2 = 1 − α λ A ≥ n 8 ( L 1 R ) 2 , log A + A ≥ α ≥ n 2 2 L 1 R . \alpha^{2} = \frac{1-\alpha}{\lambda\mathcal{A}} \ge\frac{n}{8(L_{1}R)^{2}}, \qquad\log\frac{\mathcal{A}^{+}}{\mathcal{A}} \ge\alpha\ge\frac{\sqrt{n}}{2\sqrt{2}L_{1}R}. α 2 = λ A 1 − α ≥ 8 ( L 1 R ) 2 n , log A A + ≥ α ≥ 2 2 L 1 R n .
Therefore, the number of such iterations is O ( ( L 1 R / n ) log ( Δ / ε ) ) O((L_{1}R/\sqrt{n})\log(\Delta/\varepsilon)) O (( L 1 R / n ) log ( Δ/ ε )) .
▷ If the maximum is attained at the last term, that is, λ = ε / ( 8 r R ) \lambda= \varepsilon/(8rR) λ = ε / ( 8 r R ) , then the method stops by Lemma B.3 . If several terms are equal, the iteration is assigned to any one of them.
Summing the obtained bounds and accounting for the last iteration, we obtain (B.8). Each solver call almost surely terminates after finitely many steps, and each heuristic block is finite by definition (Section G.5). Consequently, the method almost surely stops.
Number of heuristic queries
Proof of Theorem F.3 ▲.
▷ Initially, G = H = 0 G = H = 0 G = H = 0 . The inequality H ≤ B 0 + ω G H \le B_{0}+\omega G H ≤ B 0 + ω G implies that the number of allowed queries in (F.4) is nonnegative. A single block spends at most B 0 + ω G − H B_{0}+\omega G-H B 0 + ω G − H queries, so after the block the inequality is preserved. During a main iteration, only G G G increases, so the inequality is preserved as well. The limit of each block is at most B 0 B_{0} B 0 , which proves the second bound H ≤ k B 0 H \le kB_{0} H ≤ k B 0 .
▷ Blocks with limit B k ≤ B 0 B_{k} \le B_{0} B k ≤ B 0 satisfy the requirements of Section G.5; therefore, Lemmas B.3 and B.4 apply to them. All conditional bounds for the main iterations and the bound on the number of oracle calls in each heuristic block remain valid. Consequently, the bound of Theorem 5.1 is preserved. Finally, for every run,
T = G + H ≤ ( a ) B 0 + ( 1 + ω ) G , T = G + H \underset{(a)}{\le} B_{0} + (1+\omega)G, T = G + H ( a ) ≤ B 0 + ( 1 + ω ) G ,
where (a) follows from (F.5). Taking expectations, we obtain (F.6).
Proof of the upper bound
We use N N N and T 0 T_0 T 0 from § 2 . All bounds below are uniform in the problem parameters and in the history of the method.
Number of main iterations
By the choice of the radius r r r , we have
R r = max { 1 , 2 L 1 R } ≤ 2 N . \frac{R}{r}=\max\{1,2L_1R\}\le2\sqrt{N}. r R = max { 1 , 2 L 1 R } ≤ 2 N .
From (G.7) it follows that
λ ↑ r R ε ≲ Δ ε ( R r ) 1 / 3 + ( L 0 + L 1 2 δ ) R 2 n ε + Δ L 1 R ε n + 1 ≲ N 3 / 2 . (22) \frac{\lambda_{\uparrow}rR}{\varepsilon}\lesssim\frac{\Delta}{\varepsilon}\left(\frac{R}{r}\right)^{1/3}+\frac{(L_0+L_1^2\delta)R^2}{n\varepsilon}+\frac{\Delta L_1R}{\varepsilon n}+1\lesssim N^{3/2}. \tag*{(22)} ε λ ↑ r R ≲ ε Δ ( r R ) 1/3 + n ε ( L 0 + L 1 2 δ ) R 2 + ε n Δ L 1 R + 1 ≲ N 3/2 . (22)
Therefore, the logarithms in (B.8) are of order O ( log N ) O(\log N) O ( log N ) .
Lemma C.1. The number of main iterations satisfies
n K o u t ≲ T 0 log N . (23) nK_{\mathrm{out}}\lesssim T_0\log N. \tag*{(23)} n K out ≲ T 0 log N . (23)
Proof. Substituting (C.1) into (B.8), we obtain
n K o u t ≲ log N [ n ( R r ) 2 / 3 + n L 1 R + n ( L 0 + L 1 2 δ ) R 2 ε ] . nK_{\mathrm{out}}\lesssim\log N\left[n\left(\frac{R}{r}\right)^{2/3}+\sqrt{n}L_1R+\sqrt{\frac{n(L_0+L_1^2\delta)R^2}{\varepsilon}}\right]. n K out ≲ log N [ n ( r R ) 2/3 + n L 1 R + ε n ( L 0 + L 1 2 δ ) R 2 ] .
It remains to use ( R / r ) 2 / 3 ∼ 1 + ( L 1 R ) 2 / 3 (R/r)^{2/3}\sim1+(L_1R)^{2/3} ( R / r ) 2/3 ∼ 1 + ( L 1 R ) 2/3 and a + b ∼ a 2 + b 2 a+b\sim\sqrt{a^2+b^2} a + b ∼ a 2 + b 2 for a , b ≥ 0 a,b\ge0 a , b ≥ 0 .
Number of local solver queries
The curvature test (G.9) and the lower bound on λ \lambda λ give
κ : = L ‾ λ = 1 + e 3 L 1 r ℓ ( y ) λ ≤ 1 + n e 4 L 1 r ≤ 1 + n e 2 ≤ 8 n . (24) \kappa:=\frac{\overline{L}}{\lambda}=1+\frac{e^{3L_1r}\ell(y)}{\lambda}\le1+ne^{4L_1r}\le1+ne^2\le8n. \tag*{(24)} κ := λ L = 1 + λ e 3 L 1 r ℓ ( y ) ≤ 1 + n e 4 L 1 r ≤ 1 + n e 2 ≤ 8 n . (24)
since L 1 r ≤ 1 / 2 L_1r\le1/2 L 1 r ≤ 1/2 and n ≥ 2 n\ge2 n ≥ 2 . Here we used the fact that the same quantity 3 2 L 0 + 2 L 1 2 ( δ + R 2 / A ) \frac{3}{2}L_0+2L_1^2(\delta+R^2/A) 2 3 L 0 + 2 L 1 2 ( δ + R 2 / A ) enters both the curvature test and the lower bound on n λ n\lambda nλ .
Lemma C.2. For any history preceding a local call,
E [ T e p ∣ h i s t o r y ] ≤ 2 + ⌈ log 2 Γ ⌉ , n + M ≤ 36 n , (25) \mathbb{E}[T_{\mathrm{ep}}\mid\mathrm{history}]\le2+\lceil\log_2\Gamma\rceil,\qquad n+M\le36n, \tag*{(25)} E [ T ep ∣ history ] ≤ 2 + ⌈ log 2 Γ ⌉ , n + M ≤ 36 n , (25) where Γ \Gamma Γ is defined in Lemma H.2 and Γ ≤ 9 ( 8 n ) 3 \Gamma\le9(8n)^3 Γ ≤ 9 ( 8 n ) 3 . In particular, the conditional expectation of the number of local solver queries is of order O ( n log ( n + 2 ) ) O(n\log(n+2)) O ( n log ( n + 2 )) .
Proof. By (C.3), the epoch length is M = ⌈ 21 κ / 5 ⌉ ≤ 1 + 168 n / 5 ≤ 35 n M=\lceil21\kappa/5\rceil\le1+168n/5\le35n M = ⌈ 21 κ /5 ⌉ ≤ 1 + 168 n /5 ≤ 35 n . For the quantity Γ \Gamma Γ from (H.12) we have
Γ = ( 1 + 2 κ 3 ) ( 1 + 5 κ 4 ) 2 ≤ 5 κ 3 ( 9 κ 4 ) 2 ≤ 9 κ 3 ≤ 9 ( 8 n ) 3 , \Gamma=\left(1+\frac{2\kappa}{3}\right)\left(1+\frac{5\kappa}{4}\right)^2\le\frac{5\kappa}{3}\left(\frac{9\kappa}{4}\right)^2\le9\kappa^3\le9(8n)^3, Γ = ( 1 + 3 2 κ ) ( 1 + 4 5 κ ) 2 ≤ 3 5 κ ( 4 9 κ ) 2 ≤ 9 κ 3 ≤ 9 ( 8 n ) 3 ,
since κ ≥ 1 \kappa\ge1 κ ≥ 1 . Set j 0 = ⌈ log 2 Γ ⌉ j_0=\lceil\log_2\Gamma\rceil j 0 = ⌈ log 2 Γ ⌉ . Summing the tail bound (H.12), we obtain
E [ T e p ∣ h i s t o r y ] ≤ 1 + ∑ j ≥ 1 min { 1 , Γ 2 − j } ≤ 1 + j 0 + ∑ j > j 0 Γ 2 − j ≤ j 0 + 2. \mathbb{E}[T_{\mathrm{ep}}\mid\mathrm{history}]\le1+\sum_{j\ge1}\min\{1,\Gamma2^{-j}\}\le1+j_0+\sum_{j>j_0}\Gamma2^{-j}\le j_0+2. E [ T ep ∣ history ] ≤ 1 + j ≥ 1 ∑ min { 1 , Γ 2 − j } ≤ 1 + j 0 + j > j 0 ∑ Γ 2 − j ≤ j 0 + 2.
Each epoch, together with the verification of its result, requires O ( n + M ) = O ( n ) O(n+M)=O(n) O ( n + M ) = O ( n ) queries.
Total number of queries
Proof of Theorem 5.1 . By Lemmas B.3 and B.4, the method almost surely stops with an optimality gap of at most ε / 2 \varepsilon/2 ε /2 . This proves (8) .
If the curvature test fails, a main iteration requires n n n queries. Otherwise, by Lemma C.2 , the conditional expected number of queries is O ( n log ( n + 2 ) ) O(n\log(n+2)) O ( n log ( n + 2 )) . This bound includes the evaluations at the center, at the anchor points, and in the verification of the solver’s output. The history-based stopping can only decrease the number of queries: for comparison, the solver can be continued with fresh random indices until its own stopping criterion is met. By the definition in Section G.5, each heuristic block makes O ( n log N ) O(n\log N) O ( n log N ) queries. Since n + 2 ≤ N n+2\le N n + 2 ≤ N , the same bound applies to a main iteration as well.
The number of main iterations and of the blocks preceding them does not exceed K o u t K_{\mathrm{out}} K out . Summing the conditional bounds and using (C.2), we obtain
E T ≲ n K o u t log N ≲ T 0 log 2 N . \mathbb{E}T \lesssim nK_{\mathrm{out}}\log N \lesssim T_0\log^2 N. E T ≲ n K out log N ≲ T 0 log 2 N .
The bound is uniform over all heuristics admissible in the block from Section G.5.
Constructions for the lower bound
Throughout this appendix, Assumptions 2.1–2.6 are assumed to hold.
Extension to randomized methods
Lemma D.1 (averaging over a finite family). Let F \mathcal{F} F be a nonempty finite family of admissible inputs, and let c τ > 0 c_\tau>0 c τ > 0 and c E ≥ 0 c_{\mathcal{E}}\ge0 c E ≥ 0 be weight coefficients. Denote by τ F \tau_F τ F the number of oracle calls made by the algorithm on input F F F , and by E F \mathcal{E}_F E F the optimality gap of its output on this input, that is, f F ( x ^ F ) − f F ∗ f_F(\hat{x}_F)-f_F^* f F ( x ^ F ) − f F ∗ . If for every deterministic algorithm
1 ∣ F ∣ ∑ F ∈ F ( c τ τ F + c E E F ) ≥ c , \frac{1}{|\mathcal{F}|}\sum_{F\in\mathcal{F}}(c_\tau\tau_F+c_{\mathcal{E}}\mathcal{E}_F)\ge c, ∣ F ∣ 1 F ∈ F ∑ ( c τ τ F + c E E F ) ≥ c , then T ∗ ≥ ( c − c E ε ) / c τ T^*\ge(c-c_{\mathcal{E}}\varepsilon)/c_\tau T ∗ ≥ ( c − c E ε ) / c τ . Inputs in the family may repeat: the sum is taken over a finite set of parameters with equal weights.
Proof of Lemma D.1 . If all random variables used by the algorithm are fixed, it becomes deterministic. Measurability of the query selection rules in the adopted oracle model allows us to average the bound over the random variables of the algorithm. Let M be a valid algorithm (that is, its expected optimality gap on every input does not exceed ε \varepsilon ε ) with T ( M ) < + ∞ \mathcal{T}(\mathrm{M})<+\infty T ( M ) < + ∞ . Then
c ≤ E [ 1 ∣ F ∣ ∑ F ∈ F ( c τ τ F + c E E F ) ] = 1 ∣ F ∣ ∑ F ∈ F ( c τ E τ F + c E E E F ) ≤ c τ T ( M ) + c E ε . c\le\mathbb{E}\left[\frac{1}{|\mathcal{F}|}\sum_{F\in\mathcal{F}}(c_\tau\tau_F+c_{\mathcal{E}}\mathcal{E}_F)\right]
=\frac{1}{|\mathcal{F}|}\sum_{F\in\mathcal{F}}(c_\tau\mathbb{E}\tau_F+c_{\mathcal{E}}\mathbb{E}\mathcal{E}_F)
\le c_\tau\mathcal{T}(\mathrm{M})+c_{\mathcal{E}}\varepsilon. c ≤ E [ ∣ F ∣ 1 F ∈ F ∑ ( c τ τ F + c E E F ) ] = ∣ F ∣ 1 F ∈ F ∑ ( c τ E τ F + c E E E F ) ≤ c τ T ( M ) + c E ε .
Here inequality (a) follows from the hypothesis of the lemma applied for every value of the random variables of the algorithm, and (b) uses the inequalities E τ F ≤ T ( M ) \mathbb{E}\tau_F\le\mathcal{T}(\mathrm{M}) E τ F ≤ T ( M ) and E E F ≤ ε \mathbb{E}\mathcal{E}_F\le\varepsilon E E F ≤ ε . Hence T ( M ) ≥ ( c − c E ε ) / c τ \mathcal{T}(\mathrm{M})\ge(c-c_{\mathcal{E}}\varepsilon)/c_\tau T ( M ) ≥ ( c − c E ε ) / c τ ; for T ( M ) = + ∞ \mathcal{T}(\mathrm{M})=+\infty T ( M ) = + ∞ this inequality is obvious. Taking the infimum over valid algorithms, we obtain the claim for T ∗ T^* T ∗ . This is a version of Yao’s minimax principle [39 ] for a finite family.
Let c : { 1 , … , s } → { 1 , … , d } c:\{1,\ldots,s\}\to\{1,\ldots,d\} c : { 1 , … , s } → { 1 , … , d } be chosen uniformly, and let coordinate 0 in R d + 1 \mathbb{R}^{d+1} R d + 1 be distinguished. The revelation of coordinates and the oracle answers depend only on the current query and on the already revealed values, while the queries and the returned point depend on the history of answers. When bounding the probability of revealing coordinates, we also take into account the returned point; the number of queries does not increase in doing so.
Lemma D.2 (averaging over a hidden coordinate). Let α , L > 0 \alpha, L > 0 α , L > 0 , and let N q r y ≥ 1 N_{\mathrm{qry}} \ge1 N qry ≥ 1 be an arbitrary integer. The fraction of maps c c c for which among the first N q r y N_{\mathrm{qry}} N qry query points and the point returned before this deadline there is a point x x x with ∥ x ∥ ≤ L \lVert x\rVert\le L ∥ x ∥ ≤ L and ∣ x c ( j ) ∣ ≥ α \lvert x_{c(j)}\rvert\ge\alpha ∣ x c ( j ) ∣ ≥ α for some not yet revealed coordinate c ( j ) c(j) c ( j ) does not exceed ( N q r y + 1 ) s L 2 / ( d α 2 ) (N_{\mathrm{qry}}+1)sL^2/(d\alpha^2) ( N qry + 1 ) s L 2 / ( d α 2 ) . Taking non-injective maps into account, the fraction of exceptions does not exceed
s 2 + ( N q r y + 1 ) s L 2 / α 2 d . (26) \frac{s^2+(N_{\mathrm{qry}}+1)sL^2/\alpha^2}{d}. \tag*{(26)} d s 2 + ( N qry + 1 ) s L 2 / α 2 . (26)
Bound (D.1) refers to the uniform distribution on the set of all maps, not to the conditional distribution of injections. To each non-injective map we assign the input corresponding to one fixed injection. Thus, every input of the finite distribution is admissible. On the exceptions we use only the nonnegativity of the number of queries and of the optimality gap; no properties of the inadmissible construction are required there. The dimension is chosen after N q r y N_{\mathrm{qry}} N qry , L L L , α \alpha α , but before the choice of the algorithm. Therefore, one and the same dimension yields the required bound for every algorithm and every value of the random variables it uses.
Proof of Lemma D.2. Fix a position j j j and all values of c c c except c ( j ) c(j) c ( j ) . Until j j j is revealed, all corresponding runs have the same history: this is verified by induction on time, since the answers before the revelation do not depend on c ( j ) c(j) c ( j ) . Therefore, at any step before the revelation, the query point or the returned point is the same for all d d d values of c ( j ) c(j) c ( j ) . If its norm is at most L L L , then
α 2 # { v : ∣ x v ∣ ≥ α } ≤ ∑ v : ∣ x v ∣ ≥ α ∣ x v ∣ 2 ≤ ∥ x ∥ 2 ≤ L 2 . \alpha^2\#\{v:\lvert x_v\rvert\ge\alpha\}\le\sum_{v:\lvert x_v\rvert\ge\alpha}\lvert x_v\rvert^2\le\lVert x\rVert^2\le L^2. α 2 # { v : ∣ x v ∣ ≥ α } ≤ v : ∣ x v ∣ ≥ α ∑ ∣ x v ∣ 2 ≤ ∥ x ∥ 2 ≤ L 2 .
The fraction of values c ( j ) c(j) c ( j ) for which ∣ x c ( j ) ∣ ≥ α \lvert x_{c(j)}\rvert\ge\alpha ∣ x c ( j ) ∣ ≥ α is at most L 2 / ( d α 2 ) L^2/(d\alpha^2) L 2 / ( d α 2 ) . It remains to average over the values at the other positions and to sum over the s s s positions and the N q r y + 1 N_{\mathrm{qry}}+1 N qry + 1 points. The probability that c c c takes equal values at two distinct positions is at most ( s 2 ) / d ≤ s 2 / d \binom{s}{2}/d\le s^2/d ( 2 s ) / d ≤ s 2 / d . □ \square □
Lemma D.3 (lower bound via the number of revealed links). Suppose that for a finite distribution of admissible inputs an idealized process of revealing s s s links is given. For every deterministic algorithm and every deadline N q r y N_{\mathrm{qry}} N qry , denote by Q Q Q the number of queries in this process before stopping or before the given deadline, and by P P P the number of revealed links. Assume that E Q ≥ c τ E P \mathbb{E}Q\ge c_{\tau}\mathbb{E}P E Q ≥ c τ E P for some c τ > 0 c_{\tau}>0 c τ > 0 .
Assume, moreover, that outside the exceptional set the real and idealized histories coincide, and that the optimality gap of the point returned before this deadline is at least A − B P A-BP A − B P , where
A ≥ 7 ε / 4 , B > 0 , s B ≥ A . A\ge7\varepsilon/4,\qquad B>0,\qquad sB\ge A. A ≥ 7 ε /4 , B > 0 , s B ≥ A . If the fraction of exceptions is bounded by (D.1) with fixed L , α > 0 L,\alpha>0 L , α > 0 independent of the algorithm and of the dimension, then in some finite dimension
T ∗ ≥ c τ A 4 B . \mathcal{T}^*\ge\frac{c_{\tau}A}{4B}. T ∗ ≥ 4 B c τ A .
Proof. Set N q r y = ⌊ 16 A c τ / B ⌋ N_{\mathrm{qry}}=\lfloor16Ac_{\tau}/B\rfloor N qry = ⌊ 16 A c τ / B ⌋ , q = B N q r y / ( A c τ ) ≥ 16 q=BN_{\mathrm{qry}}/(Ac_{\tau})\ge16 q = B N qry / ( A c τ ) ≥ 16 and
Z = B Q A c τ + 1 − B A P . Z=\frac{BQ}{Ac_{\tau}}+1-\frac{B}{A}P. Z = A c τ B Q + 1 − A B P .
By assumption, E Z ≥ 1 \mathbb{E}Z\ge1 E Z ≥ 1 , and Q ≤ N q r y Q\le N_{\mathrm{qry}} Q ≤ N qry implies Z ≤ q + 1 Z\le q+1 Z ≤ q + 1 . By (D.1), the choice
d ≥ 16 q ( s 2 + ( N q r y + 1 ) s L 2 / α 2 ) d\ge16q\left(s^2+(N_{\mathrm{qry}}+1)sL^2/\alpha^2\right) d ≥ 16 q ( s 2 + ( N qry + 1 ) s L 2 / α 2 )
ensures q Pr ( b a d ) ≤ 1 / 16 q\Pr(\mathrm{bad})\le1/16 q Pr ( bad ) ≤ 1/16 simultaneously for all algorithms. Let τ \tau τ be the actual number of queries and E \mathcal{E} E the optimality gap. Outside the exceptions, for τ < N q r y \tau< N_{\mathrm{qry}} τ < N qry we have Q ≤ τ Q \le\tau Q ≤ τ and E ≥ max { 0 , A − B P } \mathcal{E} \ge\max\{0,A-BP\} E ≥ max { 0 , A − B P } ; for τ ≥ N q r y \tau\ge N_{\mathrm{qry}} τ ≥ N qry we use Z ≤ q + 1 Z \le q+1 Z ≤ q + 1 . In all cases,
B τ A c τ + E A ≥ q q + 1 Z − q 1 b a d . \frac{B\tau}{A c_{\tau}}+\frac{\mathcal{E}}{A}\ge\frac{q}{q+1}Z-q1_{\mathrm{bad}}. A c τ B τ + A E ≥ q + 1 q Z − q 1 bad .
Indeed, for Z < 0 Z<0 Z < 0 the nonnegativity of the left-hand side suffices, and on the exceptions the right-hand side is nonpositive. Averaging over the inputs and over the random variables of a valid algorithm, as in Lemma D.1 , we obtain
B T ( M ) A c τ ≥ q q + 1 − 1 16 − E A ≥ 16 17 − 1 16 − 4 7 > 1 4 . \frac{BT(\mathsf{M})}{A c_{\tau}}\ge\frac{q}{q+1}-\frac{1}{16}-\frac{\mathcal{E}}{A}\ge\frac{16}{17}-\frac{1}{16}-\frac{4}{7}>\frac{1}{4}. A c τ B T ( M ) ≥ q + 1 q − 16 1 − A E ≥ 17 16 − 16 1 − 7 4 > 4 1 .
It remains to take the infimum over valid algorithms.
Construction of the chain
Define the symmetric C 1 C^{1} C 1 transition function
h ( t ) = { 0 , t ≤ 0 , 3 t 2 − 2 t 3 , 0 ≤ t ≤ 1 , 1 , t ≥ 1. h(t)=
\begin{cases}
0, & t\le0,\\
3t^{2}-2t^{3}, & 0\le t\le1,\\
1, & t\ge1.
\end{cases} h ( t ) = ⎩ ⎨ ⎧ 0 , 3 t 2 − 2 t 3 , 1 , t ≤ 0 , 0 ≤ t ≤ 1 , t ≥ 1.
It belongs to C 1 C^{1} C 1 , is nondecreasing, and satisfies h ( 1 − t ) = 1 − h ( t ) h(1-t)=1-h(t) h ( 1 − t ) = 1 − h ( t ) . On [ 0 , 1 ] [0,1] [ 0 , 1 ] we have h ′ ( t ) = 6 t ( 1 − t ) h'(t)=6t(1-t) h ′ ( t ) = 6 t ( 1 − t ) ; hence 0 ≤ h ′ ≤ M = 3 / 2 0\le h'\le M=3/2 0 ≤ h ′ ≤ M = 3/2 . Set
p ( t ) = h ( 4 ∣ t ∣ − 1 ) ( 1 − h ( ∣ t ∣ − 2 ) ) , ψ ^ ′ ( t ) = ∫ 0 t p ( u ) d u , ψ ^ ( t ) = ∫ 0 t ψ ^ ′ ( u ) d u , G H = ψ ^ ′ ( 1 ) , c H = G H − ψ ^ ( 1 ) . (27) \begin{aligned}
p(t)&=h(4|t|-1)(1-h(|t|-2)),\\
\widehat{\psi}'(t)&=\int_{0}^{t}p(u)\,\mathrm{d}u,\\
\widehat{\psi}(t)&=\int_{0}^{t}\widehat{\psi}'(u)\,\mathrm{d}u,\\
G_H&=\widehat{\psi}'(1),\\
c_H&=G_H-\widehat{\psi}(1). \tag*{(27)}
\end{aligned} p ( t ) ψ ′ ( t ) ψ ( t ) G H c H = h ( 4∣ t ∣ − 1 ) ( 1 − h ( ∣ t ∣ − 2 )) , = ∫ 0 t p ( u ) d u , = ∫ 0 t ψ ′ ( u ) d u , = ψ ′ ( 1 ) , = G H − ψ ( 1 ) . (27)
On [ 1 / 4 , 1 / 2 ] [1/4,1/2] [ 1/4 , 1/2 ] , the substitution u = 4 t − 1 u=4t-1 u = 4 t − 1 and integration of the polynomial h h h give
G H = 5 8 , ψ ^ ( 1 ) = 63 320 , c H = 137 320 . (28) G_H=\frac{5}{8},\qquad\widehat{\psi}(1)=\frac{63}{320},\qquad c_H=\frac{137}{320}. \tag*{(28)} G H = 8 5 , ψ ( 1 ) = 320 63 , c H = 320 137 . (28)
The function p p p is even, 0 ≤ p ≤ 1 0\le p\le1 0 ≤ p ≤ 1 , vanishes for ∣ t ∣ ≤ 1 / 4 |t|\le1/4 ∣ t ∣ ≤ 1/4 and ∣ t ∣ ≥ 3 |t|\ge3 ∣ t ∣ ≥ 3 , and equals one on [ 1 / 2 , 2 ] [1/2,2] [ 1/2 , 2 ] . Consequently, the function ψ ^ \widehat{\psi} ψ belongs to C 2 C^{2} C 2 , is convex and nonnegative, vanishes on [ − 1 / 4 , 1 / 4 ] [-1/4,1/4] [ − 1/4 , 1/4 ] , and satisfies the bounds
∣ ψ ^ ′ ∣ ≤ 3 , \left|\widehat{\psi}'\right|\le3, ψ ′ ≤ 3 ,
0 ≤ ψ ^ ′ ′ ≤ 1 , 0\le\widehat{\psi}''\le1, 0 ≤ ψ ′′ ≤ 1 ,
ψ ^ ( v ) ≥ 3 2 ( v − 2 ) . (29) \widehat{\psi}(v)\ge\frac{3}{2}(v-2). \tag*{(29)} ψ ( v ) ≥ 2 3 ( v − 2 ) . (29)
For v ≥ 2 v\ge2 v ≥ 2 , the last bound follows from the tangent at the point 2 2 2 :
ψ ^ ′ ( 2 ) ≥ ∫ 1 / 2 2 p = 3 2 . \widehat{\psi}'(2)\ge\int_{1/2}^{2}p=\frac{3}{2}. ψ ′ ( 2 ) ≥ ∫ 1/2 2 p = 2 3 .
For arbitrary c ψ , σ > 0 c_{\psi},\sigma>0 c ψ , σ > 0 , consider the family of one-dimensional functions
ψ ( u ) = c ψ ψ ^ ( u / σ ) . \psi(u)=c_{\psi}\widehat{\psi}(u/\sigma). ψ ( u ) = c ψ ψ ( u / σ ) .
The parameters c ψ c_{\psi} c ψ and σ \sigma σ will be chosen separately for each family. For w > 0 w>0 w > 0 , to smooth the maximum exactly, we define
S w ′ ( u ) = 2 h ( u + w 2 w ) − 1 , S'_w(u)=2h\left(\frac{u+w}{2w}\right)-1, S w ′ ( u ) = 2 h ( 2 w u + w ) − 1 ,
S w ( u ) = w − ∫ 0 w S w ′ ( v ) d v + ∫ 0 u S w ′ ( v ) d v , S_w(u)=w-\int_{0}^{w}S'_w(v)\,\mathrm{d}v+\int_{0}^{u}S'_w(v)\,\mathrm{d}v, S w ( u ) = w − ∫ 0 w S w ′ ( v ) d v + ∫ 0 u S w ′ ( v ) d v ,
smax w ( a , b ) = a + b 2 + S w ( a − b 2 ) . \operatorname{smax}_w(a,b)=\frac{a+b}{2}+S_w\left(\frac{a-b}{2}\right). smax w ( a , b ) = 2 a + b + S w ( 2 a − b ) .
The symmetry of h h h implies that S w S_w S w is even; moreover, S w ( w ) = w S_w(w)=w S w ( w ) = w . From ∣ S w ′ ∣ ≤ 1 |S'_w|\le1 ∣ S w ′ ∣ ≤ 1 and 0 ≤ S w ′ ′ ≤ M / w 0\le S''_w\le M/w 0 ≤ S w ′′ ≤ M / w it follows that S w ≥ ∣ ⋅ ∣ S_w\ge|\cdot| S w ≥ ∣ ⋅ ∣ , with equality for ∣ u ∣ ≥ w |u|\ge w ∣ u ∣ ≥ w . Therefore, the function smax w \operatorname{smax}_w smax w belongs to C 2 C^2 C 2 , is convex, nondecreasing in each argument, not smaller than the maximum of its arguments, and coincides with it for ∣ a − b ∣ ≥ 2 w |a-b|\ge2w ∣ a − b ∣ ≥ 2 w . For convex C 2 C^2 C 2 functions a ( x ) , b ( x ) a(x),b(x) a ( x ) , b ( x ) ,
∇ smax w ( a , b ) = ∇ a + ∇ b 2 + 1 2 S w ′ ( ( a − b ) / 2 ) ( ∇ a − ∇ b ) , \nabla\operatorname{smax}_w(a,b)=\frac{\nabla a+\nabla b}{2}+\frac{1}{2}S'_w((a-b)/2)(\nabla a-\nabla b), ∇ smax w ( a , b ) = 2 ∇ a + ∇ b + 2 1 S w ′ (( a − b ) /2 ) ( ∇ a − ∇ b ) ,
∥ ∇ 2 smax w ( a , b ) ∥ ≤ ∥ ∇ 2 a ∥ + ∥ ∇ 2 b ∥ + M 4 w ∥ ∇ a − ∇ b ∥ 2 . (30) \lVert\nabla^2\operatorname{smax}_w(a,b)\rVert\le\lVert\nabla^2a\rVert+\lVert\nabla^2b\rVert+\frac{M}{4w}\lVert\nabla a-\nabla b\rVert^2. \tag*{(30)} ∥ ∇ 2 smax w ( a , b )∥ ≤ ∥ ∇ 2 a ∥ + ∥ ∇ 2 b ∥ + 4 w M ∥ ∇ a − ∇ b ∥ 2 . (30)
These formulas are obtained by differentiating twice; the additional term in the Hessian equals 1 4 S w ′ ′ ( ( a − b ) / 2 ) ( ∇ a − ∇ b ) ( ∇ a − ∇ b ) T \frac{1}{4}S''_w((a-b)/2)(\nabla a-\nabla b)(\nabla a-\nabla b)^\mathsf{T} 4 1 S w ′′ (( a − b ) /2 ) ( ∇ a − ∇ b ) ( ∇ a − ∇ b ) T .
For an arbitrary integer T ≥ 1 T\ge1 T ≥ 1 , set y j = x c ( j ) y_j=x_{c(j)} y j = x c ( j ) for j < T j<T j < T and y T = 0 y_T=0 y T = 0 . Define the chain
C ( x ) = − c ψ G H σ y 0 + ∑ j = 0 T − 1 ψ ( y j − y j + 1 ) . (31) C(x)=-\frac{c_\psi G_H}{\sigma}y_0+\sum_{j=0}^{T-1}\psi(y_j-y_{j+1}). \tag*{(31)} C ( x ) = − σ c ψ G H y 0 + j = 0 ∑ T − 1 ψ ( y j − y j + 1 ) . (31)
The convexity of ψ \psi ψ and the tangent at σ \sigma σ give, for 0 ≤ p ≤ T 0\le p\le T 0 ≤ p ≤ T ,
C ( x ) ≥ − c ψ G H σ y 0 + ∑ j = 0 p − 1 ( c ψ G H σ ( y j − y j + 1 ) − c ψ c H ) = (b) − c ψ G H σ y p − p c ψ c H . (32) C(x)\ge-\frac{c_\psi G_H}{\sigma}y_0+\sum_{j=0}^{p-1}\left(\frac{c_\psi G_H}{\sigma}(y_j-y_{j+1})-c_\psi c_H\right)\underset{\text{(b)}}{=} -\frac{c_\psi G_H}{\sigma}y_p-pc_\psi c_H. \tag*{(32)} C ( x ) ≥ − σ c ψ G H y 0 + j = 0 ∑ p − 1 ( σ c ψ G H ( y j − y j + 1 ) − c ψ c H ) (b) = − σ c ψ G H y p − p c ψ c H . (32)
Here (a) uses the tangent to the function ψ \psi ψ at the point σ \sigma σ , and (b) uses the cancellation of adjacent terms.
For p = T p=T p = T we obtain C ≥ − T c ψ c H C\ge-Tc_\psi c_H C ≥ − T c ψ c H , with equality at y j = ( T − j ) σ y_j=(T-j)\sigma y j = ( T − j ) σ . For injective c c c , the operator y ↦ ( y j − y j + 1 ) j < T y\mapsto(y_j-y_{j+1})_{j<T} y ↦ ( y j − y j + 1 ) j < T has norm at most 2:
∑ j < T ( y j − y j + 1 ) 2 ≤ 2 ∑ j < T ( y j 2 + y j + 1 2 ) ≤ 4 ∑ j < T y j 2 . \sum_{j<T}(y_j-y_{j+1})^2\le2\sum_{j<T}(y_j^2+y_{j+1}^2)\le4\sum_{j<T}y_j^2. j < T ∑ ( y j − y j + 1 ) 2 ≤ 2 j < T ∑ ( y j 2 + y j + 1 2 ) ≤ 4 j < T ∑ y j 2 .
Hence, for the full chain and for any sum of its links,
∥ ∇ C ∥ ≤ c ψ σ ( G H + 6 T ) , \lVert\nabla C\rVert\le\frac{c_\psi}{\sigma}(G_H+6\sqrt{T}), ∥ ∇ C ∥ ≤ σ c ψ ( G H + 6 T ) ,
∥ ∇ 2 C ∥ ≤ 4 c ψ σ 2 . (33) \lVert\nabla^2C\rVert\le\frac{4c_\psi}{\sigma^2}. \tag*{(33)} ∥ ∇ 2 C ∥ ≤ σ 2 4 c ψ . (33)
If ∣ y j ∣ < α ≤ σ / 8 |y_j|<\alpha\le\sigma/8 ∣ y j ∣ < α ≤ σ /8 for all j ≥ p j\ge p j ≥ p , then the links with j ≥ p j\ge p j ≥ p vanish in some neighborhood of x x x . Therefore, the value and the gradient of the chain at the point x x x coincide with the answer for the function
C [ p ] ( x ) = − c ψ G H σ y 0 + ∑ 0 ≤ j < min ( p , T ) ψ ( y j − y j + 1 ) . C^{[p]}(x)=-\frac{c_\psi G_H}{\sigma}y_0+\sum_{0\le j<\min(p,T)}\psi(y_j-y_{j+1}). C [ p ] ( x ) = − σ c ψ G H y 0 + 0 ≤ j < m i n ( p , T ) ∑ ψ ( y j − y j + 1 ) .
The answer for C [ p ] C^{[p]} C [ p ] depends only on the coordinates with indices at most p p p ; before the query some of them may still be hidden, but after this query at most one new coordinate is revealed.
To handle distant queries, we introduce a radial extension. It does not change the function in a ball containing its minimizer, and it hides the chain coordinates outside a larger ball.
Lemma D.4 (radial extension). Let r , G , L a u x > 0 r,G,L_{\mathrm{aux}}>0 r , G , L aux > 0 , and let a convex C 2 C^2 C 2 function C C C attain its minimum C ∗ C_* C ∗ in B ( 0 , r ) B(0,r) B ( 0 , r ) and satisfy
∥ ∇ C ∥ ≤ G , ∥ ∇ 2 C ∥ ≤ L a u x / 3 , 4 G / r ≤ L a u x / 3. \lVert\nabla C\rVert\le G,\qquad\lVert\nabla^2C\rVert\le L_{\mathrm{aux}}/3,\qquad4G/r\le L_{\mathrm{aux}}/3. ∥ ∇ C ∥ ≤ G , ∥ ∇ 2 C ∥ ≤ L aux /3 , 4 G / r ≤ L aux /3. Set
w = 27 M G 2 4 L a u x , w=\frac{27MG^2}{4L_{\mathrm{aux}}}, w = 4 L aux 27 M G 2 , ρ ( x ) = 2 G ∥ x ∥ 2 + r 2 − 4 G r + C ∗ − 2 w , \rho(x)=2G\sqrt{\lVert x\rVert^2+r^2}-4Gr+C_*-2w, ρ ( x ) = 2 G ∥ x ∥ 2 + r 2 − 4 G r + C ∗ − 2 w , H ( x ) = smax w ( C ( x ) , ρ ( x ) ) . (34) H(x)=\operatorname{smax}_w(C(x),\rho(x)). \tag*{(34)} H ( x ) = smax w ( C ( x ) , ρ ( x )) . (34)
Then H H H is convex, belongs to C 2 C^2 C 2 , coincides with C C C on B ( 0 , r ) B(0,r) B ( 0 , r ) , has the same minimum, and satisfies
∥ ∇ H ∥ ≤ 2 G , ∥ ∇ 2 H ∥ ≤ L a u x . \lVert\nabla H\rVert\le2G,\qquad\lVert\nabla^2 H\rVert\le L_{\mathrm{aux}}. ∥ ∇ H ∥ ≤ 2 G , ∥ ∇ 2 H ∥ ≤ L aux .
For ∥ x ∥ ≥ 4 r + ( C ( 0 ) − C ∗ + 4 w ) / G \lVert x\rVert\ge4r+(C(0)-C_*+4w)/G ∥ x ∥ ≥ 4 r + ( C ( 0 ) − C ∗ + 4 w ) / G we have H ( x ) = ρ ( x ) ≥ C ( 0 ) H(x)=\rho(x)\ge C(0) H ( x ) = ρ ( x ) ≥ C ( 0 ) .
Proof. On B ( 0 , r ) B(0,r) B ( 0 , r ) we have ρ ≤ C ∗ − 2 w ≤ C − 2 w \rho\le C_*-2w\le C-2w ρ ≤ C ∗ − 2 w ≤ C − 2 w ; hence H = C H=C H = C . Everywhere H ≥ C ≥ C ∗ H\ge C\ge C_* H ≥ C ≥ C ∗ , so the minimum is preserved. For the points x x x in the last statement, we use C ( x ) ≤ C ( 0 ) + G ∥ x ∥ C(x)\le C(0)+G\lVert x\rVert C ( x ) ≤ C ( 0 ) + G ∥ x ∥ :
ρ ( x ) ≥ 2 G ∥ x ∥ − 4 G r + C ∗ − 2 w ≥ C ( 0 ) + G ∥ x ∥ + 2 w ≥ C ( x ) + 2 w . \rho(x)\ge2G\lVert x\rVert-4Gr+C_*-2w\ge C(0)+G\lVert x\rVert+2w\ge C(x)+2w. ρ ( x ) ≥ 2 G ∥ x ∥ − 4 G r + C ∗ − 2 w ≥ C ( 0 ) + G ∥ x ∥ + 2 w ≥ C ( x ) + 2 w .
Hence, there H = ρ ≥ C ( 0 ) H=\rho\ge C(0) H = ρ ≥ C ( 0 ) .
We have ∥ ∇ ρ ∥ ≤ 2 G \lVert\nabla\rho\rVert\le2G ∥ ∇ ρ ∥ ≤ 2 G and ∥ ∇ 2 ρ ∥ ≤ 2 G / r \lVert\nabla^2\rho\rVert\le2G/r ∥ ∇ 2 ρ ∥ ≤ 2 G / r . The gradient of H H H is a convex combination of the gradients of C C C and ρ \rho ρ . By (D.5),
∥ ∇ 2 H ∥ ≤ L a u x 3 + 2 G r + 9 M G 2 4 w ≤ L a u x . \lVert\nabla^2H\rVert\le\frac{L_{\mathrm{aux}}}{3}+\frac{2G}{r}+\frac{9MG^2}{4w}\le L_{\mathrm{aux}}. ∥ ∇ 2 H ∥ ≤ 3 L aux + r 2 G + 4 w 9 M G 2 ≤ L aux .
For the chain (D.6) we take G = c ψ ( G H + 6 T ) / σ G=c_\psi(G_H+6\sqrt{T})/\sigma G = c ψ ( G H + 6 T ) / σ and C ∗ = − T c ψ c H C_*=-Tc_\psi c_H C ∗ = − T c ψ c H . The conditions of Lemma D.4 take the form
4 c ψ σ 2 ≤ L a u x 3 , 4 c ψ ( G H + 6 T ) σ τ ≤ L a u x 3 . (35) \frac{4c_\psi}{\sigma^2}\le\frac{L_{\mathrm{aux}}}{3},\qquad
\frac{4c_\psi(G_H+6\sqrt{T})}{\sigma\tau}\le\frac{L_{\mathrm{aux}}}{3}. \tag*{(35)} σ 2 4 c ψ ≤ 3 L aux , σ τ 4 c ψ ( G H + 6 T ) ≤ 3 L aux . (35)
The truncated chains C [ p ] C^{[p]} C [ p ] satisfy the same gradient bound and vanish at the origin. We extend them using the same function ρ \rho ρ : outside the indicated ball, all answers coincide with ρ \rho ρ . If the full and truncated chains coincide in a neighborhood of a point, applying the same smoothed maximum preserves this property. The idea of hiding information from distant queries by means of a radial extension is used in lower bounds on oracle complexity [8 ] .
Searching for a hidden pair: the n n n term
Choose distinct indices a , b a,b a , b and a sign s ∈ { − 1 , 1 } s\in\{-1,1\} s ∈ { − 1 , 1 } uniformly among all ordered triples. In the one-dimensional space, define
f a ( u ) = n min { δ , Δ cosh ( L 1 R ) − 1 } 2 e L 1 ( u − s R ) , f_a(u)=\frac{n\min\left\{\delta,\frac{\Delta}{\cosh(L_1R)-1}\right\}}{2}e^{L_1(u-sR)}, f a ( u ) = 2 n min { δ , c o s h ( L 1 R ) − 1 Δ } e L 1 ( u − s R ) ,
f b ( u ) = n min { δ , Δ cosh ( L 1 R ) − 1 } 2 e − L 1 ( u − s R ) , f_b(u)=\frac{n\min\left\{\delta,\frac{\Delta}{\cosh(L_1R)-1}\right\}}{2}e^{-L_1(u-sR)}, f b ( u ) = 2 n min { δ , c o s h ( L 1 R ) − 1 Δ } e − L 1 ( u − s R ) ,
f i = 0 ( i ≠ a , b ) . f_i=0\quad(i\ne a,b). f i = 0 ( i = a , b ) .
Lemma D.5. Under Assumption 2.6, T ∗ ≥ ( n + 1 ) / 9 T^*\ge(n+1)/9 T ∗ ≥ ( n + 1 ) /9 .
Proof of Lemma D.5.
The components of the hidden pair satisfy Assumptions 2.1 and 2.2: they are convex, smooth, bounded below, and ∣ f i ′ ′ ∣ = L 1 ∣ f i ′ ∣ \lvert f_i''\rvert=L_1\lvert f_i'\rvert ∣ f i ′′ ∣ = L 1 ∣ f i ′ ∣ . Their average equals
f ( u ) = min { δ , Δ cosh ( L 1 R ) − 1 } cosh ( L 1 ( u − s R ) ) , f(u)=\min\left\{\delta,\frac{\Delta}{\cosh(L_1R)-1}\right\}\cosh(L_1(u-sR)), f ( u ) = min { δ , cosh ( L 1 R ) − 1 Δ } cosh ( L 1 ( u − s R )) ,
and the minimum is attained at s R sR s R , so Assumption 2.3 holds. The initial optimality gap equals min { Δ , ( cosh ( L 1 R ) − 1 ) δ } ≤ Δ \min\{\Delta,(\cosh(L_1R)-1)\delta\}\le\Delta min { Δ , ( cosh ( L 1 R ) − 1 ) δ } ≤ Δ (Assumption 2.4), and Assumption 2.5 follows from
f ∗ = min { δ , Δ cosh ( L 1 R ) − 1 } , f^*=\min\left\{\delta,\frac{\Delta}{\cosh(L_1R)-1}\right\}, f ∗ = min { δ , cosh ( L 1 R ) − 1 Δ } ,
f ∗ − n − 1 ∑ i inf f i = min { δ , Δ cosh ( L 1 R ) − 1 } ≤ δ . f^*-n^{-1}\sum_i\inf f_i=\min\left\{\delta,\frac{\Delta}{\cosh(L_1R)-1}\right\}\le\delta. f ∗ − n − 1 i ∑ inf f i = min { δ , cosh ( L 1 R ) − 1 Δ } ≤ δ .
▷ Fix an algorithm and trace its run on the zero input. Until the first query to a a a or b b b , all answers coincide with the answers on the zero input. If this pair is not touched before stopping, the returned point u u u does not depend on s s s , and
1 2 ∑ s = ± 1 ( f ( u ) − f ∗ ) = min { δ , Δ cosh ( L 1 R ) − 1 } ( cosh ( L 1 u ) cosh ( L 1 R ) − 1 ) ≥ min { Δ , ( cosh ( L 1 R ) − 1 ) δ } . \begin{aligned}
\frac{1}{2}\sum_{s=\pm1}(f(u)-f^\ast)=\min\left\{\delta,\frac{\Delta}{\cosh(L_1R)-1}\right\}\left(\cosh(L_1u)\cosh(L_1R)-1\right) \\
&\geq\min\{\Delta,(\cosh(L_1R)-1)\delta\}.
\end{aligned} 2 1 s = ± 1 ∑ ( f ( u ) − f ∗ ) = min { δ , cosh ( L 1 R ) − 1 Δ } ( cosh ( L 1 u ) cosh ( L 1 R ) − 1 ) ≥ min { Δ , ( cosh ( L 1 R ) − 1 ) δ } .
Here (a) uses the inequality cosh ( L 1 u ) ≥ 1 \cosh(L_1u)\geq1 cosh ( L 1 u ) ≥ 1 .
▷ Suppose that the run on the zero input examines r ≤ n − 1 r\leq n-1 r ≤ n − 1 distinct indices (after n − 1 n-1 n − 1 examined indices, every pair has already been touched). Repeated queries only increase the number of queries. The probability of touching the pair equals
1 − ( n − r ) ( n − r − 1 ) n ( n − 1 ) . 1-\frac{(n-r)(n-r-1)}{n(n-1)}. 1 − n ( n − 1 ) ( n − r ) ( n − r − 1 ) .
The expected number of queries before the first hit or stopping is at least
∑ j = 0 r − 1 ( n − j ) ( n − j − 1 ) n ( n − 1 ) = n ( n 2 − 1 ) − ( n − r ) ( ( n − r ) 2 − 1 ) 3 n ( n − 1 ) . \sum_{j=0}^{r-1}\frac{(n-j)(n-j-1)}{n(n-1)}
=\frac{n(n^2-1)-(n-r)((n-r)^2-1)}{3n(n-1)}. j = 0 ∑ r − 1 n ( n − 1 ) ( n − j ) ( n − j − 1 ) = 3 n ( n − 1 ) n ( n 2 − 1 ) − ( n − r ) (( n − r ) 2 − 1 ) .
∑ j = 0 r − 1 ( n − j ) ( n − j − 1 ) n ( n − 1 ) − n + 1 3 ( 1 − ( n − r ) ( n − r − 1 ) n ( n − 1 ) ) = ( n − r ) ( n − r − 1 ) r 3 n ( n − 1 ) ≥ 0. \sum_{j=0}^{r-1}\frac{(n-j)(n-j-1)}{n(n-1)}
-\frac{n+1}{3}\left(1-\frac{(n-r)(n-r-1)}{n(n-1)}\right)
=\frac{(n-r)(n-r-1)r}{3n(n-1)}\geq0. j = 0 ∑ r − 1 n ( n − 1 ) ( n − j ) ( n − j − 1 ) − 3 n + 1 ( 1 − n ( n − 1 ) ( n − r ) ( n − r − 1 ) ) = 3 n ( n − 1 ) ( n − r ) ( n − r − 1 ) r ≥ 0.
The first equality uses the identity ∑ u = 1 v u ( u − 1 ) = v ( v 2 − 1 ) / 3 \sum_{u=1}^{v}u(u-1)=v(v^2-1)/3 ∑ u = 1 v u ( u − 1 ) = v ( v 2 − 1 ) /3 .
If on the zero input the algorithm makes infinitely many queries but accesses fewer than n − 1 n-1 n − 1 distinct indices, then on some input of the family the number of queries is infinite, and the required averaged bound also holds. Denote the number of queries on the chosen input by τ \tau τ , and the optimality gap of the output by E \mathcal{E} E . Then
E a , b , s τ ( n + 1 ) / 3 + E a , b , s E min { Δ , ( cosh ( L 1 R ) − 1 ) δ } ≥ 1. \frac{\mathbb{E}_{a,b,s}\tau}{(n+1)/3}
+\frac{\mathbb{E}_{a,b,s}\mathcal{E}}{\min\{\Delta,(\cosh(L_1R)-1)\delta\}}
\geq1. ( n + 1 ) /3 E a , b , s τ + min { Δ , ( cosh ( L 1 R ) − 1 ) δ } E a , b , s E ≥ 1.
▷ By Lemma D.1,
τ ∗ ≥ n + 1 3 ( 1 − ε min { Δ , ( cosh ( L 1 R ) − 1 ) δ } ) . \tau^\ast\geq\frac{n+1}{3}\left(1-\frac{\varepsilon}{\min\{\Delta,(\cosh(L_1R)-1)\delta\}}\right). τ ∗ ≥ 3 n + 1 ( 1 − min { Δ , ( cosh ( L 1 R ) − 1 ) δ } ε ) .
From cosh u − 1 ≥ u 2 / 2 \cosh u-1\geq u^2/2 cosh u − 1 ≥ u 2 /2 and Assumption 2.6 it follows that ( cosh ( L 1 R ) − 1 ) δ ≥ 3 ε / 2 (\cosh(L_1R)-1)\delta\geq3\varepsilon/2 ( cosh ( L 1 R ) − 1 ) δ ≥ 3 ε /2 . By Assumption 2.4, Δ ≥ 2 ε \Delta\geq2\varepsilon Δ ≥ 2 ε ; hence the expression in parentheses is at least 1 / 3 1/3 1/3 . Consequently,
τ ∗ ≥ ( n + 1 ) / 9. \tau^\ast\geq(n+1)/9. τ ∗ ≥ ( n + 1 ) /9.
Family S: independent smooth chains
Under the conditions of the next lemma, we define n n n chains (D.6) in disjoint coordinates c ( i , j ) c(i,j) c ( i , j ) , 1 ≤ i ≤ n 1\leq i\leq n 1 ≤ i ≤ n , 0 ≤ j < T 0\leq j<T 0 ≤ j < T , where
T = ⌈ c H L 0 R 2 24 n ε ⌉ , c ψ = 2 ε T c H , σ = 24 ε T c H L 0 . T=\left\lceil\sqrt{\frac{c_HL_0R^2}{24n\varepsilon}}\right\rceil,\qquad
c_\psi=\frac{2\varepsilon}{Tc_H},\qquad
\sigma=\sqrt{\frac{24\varepsilon}{Tc_HL_0}}. T = ⌈ 24 n ε c H L 0 R 2 ⌉ , c ψ = T c H 2 ε , σ = T c H L 0 24 ε .
Set f i = H i f_i=H_i f i = H i , where H i H_i H i is the extension of the chain C i C_i C i by Lemma D.4 with radius R R R and Hessian bound L 0 L_0 L 0 . The common minimizer has coordinates x c ( i , j ) ∗ = ( T − j ) σ x^\ast_{c(i,j)}=(T-j)\sigma x c ( i , j ) ∗ = ( T − j ) σ .
After q i q_i q i queries to component i i i , we regard p i = min { q i , T } p_i=\min\{q_i,T\} p i = min { q i , T } positions of its chain as revealed. The next idealized answer is given by the extension of C i [ p i ] C_i^{[p_i]} C i [ p i ] and reveals at most one position. For the hidden coordinates we use the threshold
α = σ min { 1 / 8 , c H / ( 8 G H ) } . \alpha=\sigma\min\{1/8,c_H/(8G_H)\}. α = σ min { 1/8 , c H / ( 8 G H )} .
The dimension is chosen by Lemma D.3 with s = n T s=nT s = n T .
Lemma D.6. If
L 0 R 2 ε ≥ 24 ( G H + 6 ) 2 c H n , (36) \frac{L_0R^2}{\varepsilon} \ge\frac{24(G_H+6)^2}{c_H}n, \tag*{(36)} ε L 0 R 2 ≥ c H 24 ( G H + 6 ) 2 n , (36) then
T ∗ ≳ n L 0 R 2 ε . (37) \mathcal{T}^* \gtrsim\sqrt{\frac{nL_0R^2}{\varepsilon}}. \tag*{(37)} T ∗ ≳ ε n L 0 R 2 . (37)
Proof. Condition (D.11) ensures T ≥ 1 T \ge1 T ≥ 1 and leaves a sufficient margin of smoothness for the radial extension. We verify Assumption 2.3:
∥ x ∗ ∥ 2 = n σ 2 ∑ j = 1 T j 2 ≤ 24 n ε c H L 0 T 2 ≤ R 2 . \lVert x^* \rVert^2 = n\sigma^2\sum_{j=1}^{T}j^2 \le\frac{24n\varepsilon}{c_HL_0}T^2 \le R^2. ∥ x ∗ ∥ 2 = n σ 2 j = 1 ∑ T j 2 ≤ c H L 0 24 n ε T 2 ≤ R 2 .
By (D.8) we have
∥ ∇ 2 C i ∥ ≤ 4 c ψ σ 2 = L 0 3 , G 2 = L 0 ε 6 c H ( G H T + 6 ) 2 ≤ L 0 ε ( G H + 6 ) 2 6 c H ≤ ( L 0 R 12 ) 2 . \lVert\nabla^2 C_i\rVert\le\frac{4c_\psi}{\sigma^2}=\frac{L_0}{3},\qquad G^2=\frac{L_0\varepsilon}{6c_H}\left(\frac{G_H}{\sqrt{T}}+6\right)^2\le\frac{L_0\varepsilon(G_H+6)^2}{6c_H}\le\left(\frac{L_0R}{12}\right)^2. ∥ ∇ 2 C i ∥ ≤ σ 2 4 c ψ = 3 L 0 , G 2 = 6 c H L 0 ε ( T G H + 6 ) 2 ≤ 6 c H L 0 ε ( G H + 6 ) 2 ≤ ( 12 L 0 R ) 2 .
The last inequality follows from (D.11). Hence 4 G / R ≤ L 0 / 3 4G/R\le L_0/3 4 G / R ≤ L 0 /3 , and Lemma D.4 yields convex, bounded below, L 0 L_0 L 0 -smooth components, that is, Assumptions 2.1 and 2.2 hold. We have f ( 0 ) = 0 f(0)=0 f ( 0 ) = 0 and f ∗ = − 2 ε f^*=-2\varepsilon f ∗ = − 2 ε ; the initial optimality gap equals 2 ε ≤ Δ 2\varepsilon\le\Delta 2 ε ≤ Δ , and the heterogeneity is zero. Thus, Assumptions 2.4 and 2.5 hold.
Outside the exceptional set, the real and idealized histories coincide: for interior queries this follows from the presence of a flat region of ψ \psi ψ , and for exterior ones from Lemma D.4. The same holds for the returned point x ^ \hat{x} x ^ . Inside the extension ball, by (D.7),
f i ( x ^ ) ≥ − 2 ε T ( p i + 1 8 ) . f_i(\hat{x})\ge-\frac{2\varepsilon}{T}\left(p_i+\frac{1}{8}\right). f i ( x ^ ) ≥ − T 2 ε ( p i + 8 1 ) .
For p i = T p_i=T p i = T it suffices that f i ≥ − 2 ε f_i\ge-2\varepsilon f i ≥ − 2 ε . Therefore, with P = ∑ i p i P=\sum_i p_i P = ∑ i p i , the optimality gap satisfies
E ≥ 2 ε − ε 4 T − 2 ε n T P ≥ 7 ε 4 − 2 ε n T P . \mathcal{E}\ge2\varepsilon-\frac{\varepsilon}{4T}-\frac{2\varepsilon}{nT}P\ge\frac{7\varepsilon}{4}-\frac{2\varepsilon}{nT}P. E ≥ 2 ε − 4 T ε − n T 2 ε P ≥ 4 7 ε − n T 2 ε P .
Outside the ball it is at least 2 ε 2\varepsilon 2 ε . Each query reveals at most one link. We apply Lemma D.3 with c τ = 1 c_\tau=1 c τ = 1 , A = 7 ε / 4 A=7\varepsilon/4 A = 7 ε /4 , B = 2 ε / ( n T ) B=2\varepsilon/(nT) B = 2 ε / ( n T ) . The condition s B ≥ A sB\ge A s B ≥ A holds, and
T ∗ ≥ 7 n T 32 ≥ 7 64 c H n L 0 R 2 24 ε . \mathcal{T}^*\ge\frac{7nT}{32}\ge\frac{7}{64}\sqrt{\frac{c_HnL_0R^2}{24\varepsilon}}. T ∗ ≥ 32 7 n T ≥ 64 7 24 ε c H n L 0 R 2 .
Here we used the inequality ⌊ u ⌋ ≥ u / 2 \lfloor u\rfloor\ge u/2 ⌊ u ⌋ ≥ u /2 for u ≥ 1 u\ge1 u ≥ 1 . □
Lemma D.7 (exponential transformation). Let a function m ≥ 0 m\ge0 m ≥ 0 be convex, belong to C 2 C^2 C 2 , and satisfy ∥ ∇ m ∥ ≤ L 1 / 7 \lVert\nabla m\rVert\le L_1/7 ∥ ∇ m ∥ ≤ L 1 /7 , ∥ ∇ 2 m ∥ ≤ 4 L 1 2 / 49 \lVert\nabla^2m\rVert\le4L_1^2/49 ∥ ∇ 2 m ∥ ≤ 4 L 1 2 /49 . Then, for δ ′ > 0 \delta'>0 δ ′ > 0 , both functions
v ( x ) = δ ′ ( e ± 2 L 1 x 0 / 7 + m ( x ) − 1 ) v(x)=\delta'\left(e^{\pm2L_1x_0/7+m(x)}-1\right) v ( x ) = δ ′ ( e ± 2 L 1 x 0 /7 + m ( x ) − 1 ) are convex, belong to C 2 C^2 C 2 , have infimum − δ ′ -\delta' − δ ′ , and satisfy ∥ ∇ 2 v ∥ ≤ L 1 ∥ ∇ v ∥ \lVert\nabla^2v\rVert\le L_1\lVert\nabla v\rVert ∥ ∇ 2 v ∥ ≤ L 1 ∥ ∇ v ∥ .
Proof. Set u = ± 2 L 1 x 0 / 7 + m ( x ) u = \pm2L_1x_0/7 + m(x) u = ± 2 L 1 x 0 /7 + m ( x ) . The function m m m may depend on x 0 x_0 x 0 . We have
L 1 / 7 ≤ ∥ ∇ u ∥ ≤ 3 L 1 / 7 , L_1/7 \le\lVert\nabla u\rVert\le3L_1/7, L 1 /7 ≤ ∥ ∇ u ∥ ≤ 3 L 1 /7 ,
∥ ∇ 2 u ∥ + ∥ ∇ u ∥ 2 ≤ ( 4 L 1 7 + 3 L 1 7 ) ∥ ∇ u ∥ = L 1 ∥ ∇ u ∥ , \lVert\nabla^2u\rVert+ \lVert\nabla u\rVert^2 \le\left(\frac{4L_1}{7}+\frac{3L_1}{7}\right)\lVert\nabla u\rVert= L_1\lVert\nabla u\rVert, ∥ ∇ 2 u ∥ + ∥ ∇ u ∥ 2 ≤ ( 7 4 L 1 + 7 3 L 1 ) ∥ ∇ u ∥ = L 1 ∥ ∇ u ∥ ,
∥ ∇ 2 v ∥ ≤ δ ′ e u ( ∥ ∇ 2 u ∥ + ∥ ∇ u ∥ 2 ) ≤ L 1 ∥ ∇ v ∥ . (38) \lVert\nabla^2v\rVert\le\delta' e^u\left(\lVert\nabla^2u\rVert+\lVert\nabla u\rVert^2\right) \le L_1\lVert\nabla v\rVert. \tag*{(38)} ∥ ∇ 2 v ∥ ≤ δ ′ e u ( ∥ ∇ 2 u ∥ + ∥ ∇ u ∥ 2 ) ≤ L 1 ∥ ∇ v ∥ . (38)
The convexity of v v v follows from the convexity of u u u . Along x = ∓ t e 0 x=\mp te_0 x = ∓ t e 0 we have u ( x ) ≤ m ( 0 ) − L 1 t / 7 → − ∞ u(x)\le m(0)-L_1t/7\to-\infty u ( x ) ≤ m ( 0 ) − L 1 t /7 → − ∞ ; hence inf v = − δ ′ \inf v=-\delta' inf v = − δ ′ .
Thus, v v v belongs to the required class for every L 0 ≥ 0 L_0\ge0 L 0 ≥ 0 . Verifying membership in the class via a Hessian bound corresponds to the characterization from the first part (Lemma 2.2 of Rudomanov et al., [30 ] ).
Set
η = log ( 1 + 2 ε δ ) , ρ = ( 1 + δ 2 ε ) η , \eta=\log\left(1+\frac{2\varepsilon}{\delta}\right),\qquad\rho=\left(1+\frac{\delta}{2\varepsilon}\right)\eta, η = log ( 1 + δ 2 ε ) , ρ = ( 1 + 2 ε δ ) η ,
D δ = max { 12 η c H , 4 ( G H + 6 ) η c H } , U = 2 L 1 R 7 D δ . (39) D_\delta=\max\left\{\sqrt{\frac{12\eta}{c_H}},\frac{4(G_H+6)\eta}{c_H}\right\},\qquad U=\frac{2L_1R}{7D_\delta}. \tag*{(39)} D δ = max { c H 12 η , c H 4 ( G H + 6 ) η } , U = 7 D δ 2 L 1 R . (39)
Here D δ ∼ max { η , η } D_\delta\sim\max\{\sqrt{\eta},\eta\} D δ ∼ max { η , η } . First consider arbitrary integers g , T ≥ 1 g,T\ge1 g , T ≥ 1 with 2 g ≤ n 2g\le n 2 g ≤ n . We construct g g g chains ((D.6)) in disjoint coordinates c ( b , j ) c(b,j) c ( b , j ) , 1 ≤ b ≤ g 1\le b\le g 1 ≤ b ≤ g , 0 ≤ j < T 0\le j<T 0 ≤ j < T , with parameters
c ψ = η T c H , σ = 7 2 L 1 max { 12 η T c H , 4 ( G H + 6 T ) η T c H } . c_\psi=\frac{\eta}{Tc_H},\qquad\sigma=\frac{7}{2L_1}\max\left\{\sqrt{\frac{12\eta}{Tc_H}},\frac{4(G_H+6\sqrt{T})\eta}{Tc_H}\right\}. c ψ = T c H η , σ = 2 L 1 7 max { T c H 12 η , T c H 4 ( G H + 6 T ) η } .
Denote by σ 1 \sigma_1 σ 1 the value of σ \sigma σ for T = 1 T=1 T = 1 . Then σ ≤ σ 1 / T \sigma\le\sigma_1/\sqrt{T} σ ≤ σ 1 / T and U = R / σ 1 U=R/\sigma_1 U = R / σ 1 . We extend each chain by Lemma D.4 with radius r X = max { R , 21 / ( 2 L 1 ) } r_X=\max\{R,21/(2L_1)\} r X = max { R , 21/ ( 2 L 1 )} and Hessian bound 4 L 1 2 / 49 4L_1^2/49 4 L 1 2 /49 . The resulting functions are denoted by H b H_b H b . Their common minimum is attained at the point x c ( b , j ) ∗ = ( T − j ) σ x^*_{c(b,j)}=(T-j)\sigma x c ( b , j ) ∗ = ( T − j ) σ .
We define 2 g 2g 2 g active components:
m b ( x ) = η + H b ( x ) , m_b(x)=\eta+H_b(x), m b ( x ) = η + H b ( x ) ,
f b , ± ( x ) = n δ 2 g ( e ± 2 L 1 x 0 / 7 + m b ( x ) − 1 ) . f_{b,\pm}(x)=\frac{n\delta}{2g}\left(e^{\pm2L_1x_0/7+m_b(x)}-1\right). f b , ± ( x ) = 2 g n δ ( e ± 2 L 1 x 0 /7 + m b ( x ) − 1 ) .
The remaining components are zero. The hidden coordinates are different from 0 0 0 ; hence x 0 ∗ = 0 x_0^*=0 x 0 ∗ = 0 . A query to either component of a pair reveals at most one link of its chain. For the hidden coordinates we use the threshold
α = σ min { 1 / 8 , c H / ( 8 G H ρ ) } . \alpha=\sigma\min\{1/8,c_H/(8G_H\rho)\}. α = σ min { 1/8 , c H / ( 8 G H ρ )} .
The dimension is chosen by Lemma D.3 with s = g T s=gT s = g T . For U ≥ 1 U\ge1 U ≥ 1 we take
g = min { ⌊ n / 2 ⌋ , ⌊ U 2 ⌋ } , T = ⌊ U / g ⌋ . g=\min\{\lfloor n/2\rfloor,\lfloor U^2\rfloor\},\qquad T=\lfloor U/\sqrt{g}\rfloor. g = min {⌊ n /2 ⌋ , ⌊ U 2 ⌋} , T = ⌊ U / g ⌋ .
Both numbers are positive, and 2 g ≤ n 2g\le n 2 g ≤ n .
Lemma D.8. If U ≥ 1 U\ge1 U ≥ 1 , then
T ∗ ≳ U min { n , U } ρ . (40) \mathcal{T}^*\gtrsim\frac{U\min\{\sqrt{n},U\}}{\rho}. \tag*{(40)} T ∗ ≳ ρ U min { n , U } . (40)
Proof. From the choice of σ \sigma σ and (D.8) we obtain
∥ ∇ C b ∥ ≤ L 1 / 14 , ∥ ∇ 2 C b ∥ ≤ 4 L 1 2 / 147. \lVert\nabla C_b\rVert\le L_1/14,\qquad\lVert\nabla^2C_b\rVert\le4L_1^2/147. ∥ ∇ C b ∥ ≤ L 1 /14 , ∥ ∇ 2 C b ∥ ≤ 4 L 1 2 /147.
Together with r X ≥ 21 / ( 2 L 1 ) r_X\ge21/(2L_1) r X ≥ 21/ ( 2 L 1 ) , this means that the conditions of Lemma D.4 hold. Moreover,
∥ x ∗ ∥ 2 = g σ 2 ∑ j = 1 T j 2 ≤ g σ 1 2 T 2 ≤ R 2 . \lVert x^*\rVert^2=g\sigma^2\sum_{j=1}^{T}j^2\le g\sigma_1^2T^2\le R^2. ∥ x ∗ ∥ 2 = g σ 2 j = 1 ∑ T j 2 ≤ g σ 1 2 T 2 ≤ R 2 .
Consequently, m b ≥ 0 m_b \ge0 m b ≥ 0 , ∥ ∇ m b ∥ ≤ L 1 / 7 \lVert\nabla m_b\rVert\le L_1/7 ∥ ∇ m b ∥ ≤ L 1 /7 and ∥ ∇ 2 m b ∥ ≤ 4 L 1 2 / 49 \lVert\nabla^2m_b\rVert\le4L_1^2/49 ∥ ∇ 2 m b ∥ ≤ 4 L 1 2 /49 . By Lemma D.7 , Assumptions 2.1 and 2.2 hold, and
f ( x ) = δ g ∑ b = 1 g ( cosh ( 2 L 1 x 0 / 7 ) e m b ( x ) − 1 ) ≥ 0. f(x)=\frac{\delta}{g}\sum_{b=1}^{g}\left(\cosh(2L_1x_0/7)e^{m_b(x)}-1\right)\ge0. f ( x ) = g δ b = 1 ∑ g ( cosh ( 2 L 1 x 0 /7 ) e m b ( x ) − 1 ) ≥ 0.
At the point x ∗ x^* x ∗ all m b = 0 m_b=0 m b = 0 ; hence f ( x ∗ ) = f ∗ = 0 f(x^*)=f^*=0 f ( x ∗ ) = f ∗ = 0 , and Assumption 2.3 holds. At zero m b = η m_b=\eta m b = η , whence f ( 0 ) = 2 ε ≤ Δ f(0)=2\varepsilon\le\Delta f ( 0 ) = 2 ε ≤ Δ (Assumption 2.4). Since inf f b , ± = − n δ / ( 2 g ) \inf f_{b,\pm}=-n\delta/(2g) inf f b , ± = − n δ / ( 2 g ) , the heterogeneity equals δ \delta δ (Assumption 2.5).
The coincidence of the real and idealized histories is proved in the same way as for family S: the exponential transformation preserves local equality of functions. Let p b ≤ T p_b\le T p b ≤ T be the number of revealed links of pair b b b , and let P = ∑ b p b P=\sum_b p_b P = ∑ b p b . Inside the extension ball and outside the exceptional set, bound (D.7) gives
m b ( x ^ ) ≥ η ( 1 − p b T ) − η 8 T ρ . m_b(\hat{x})\ge\eta\left(1-\frac{p_b}{T}\right)-\frac{\eta}{8T\rho}. m b ( x ^ ) ≥ η ( 1 − T p b ) − 8 T ρ η .
For p b = T p_b=T p b = T it suffices that m b ≥ 0 m_b\ge0 m b ≥ 0 . The tangent to the exponential at η \eta η and the equality e η η δ = 2 ε ρ e^{\eta}\eta\delta=2\varepsilon\rho e η η δ = 2 ερ give
ε 2 ε ≥ 1 − ρ g T P − 1 8 T . \frac{\varepsilon}{2\varepsilon}\ge1-\frac{\rho}{gT}P-\frac{1}{8T}. 2 ε ε ≥ 1 − g T ρ P − 8 T 1 .
Outside the ball, the optimality gap is at least 2 ε 2\varepsilon 2 ε . We apply Lemma D.3 with c τ = 1 c_\tau=1 c τ = 1 , A = 7 ε / 4 A=7\varepsilon/4 A = 7 ε /4 , B = 2 ε ρ / ( g T ) B=2\varepsilon\rho/(gT) B = 2 ερ / ( g T ) . The inequality ( 1 + 1 / z ) log ( 1 + z ) ≥ 1 (1+1/z)\log(1+z)\ge1 ( 1 + 1/ z ) log ( 1 + z ) ≥ 1 for z > 0 z>0 z > 0 gives ρ ≥ 1 \rho\ge1 ρ ≥ 1 ; hence s B ≥ A sB\ge A s B ≥ A . We obtain
T ∗ ≥ 7 g T 32 ρ . T^*\ge\frac{7gT}{32\rho}. T ∗ ≥ 32 ρ 7 g T .
From n ≥ 2 n\ge2 n ≥ 2 , U ≥ 1 U\ge1 U ≥ 1 and the definition of g , T g,T g , T it follows that
g ≥ 1 3 min { n , U 2 } , T ≥ U 2 g , g T ≥ U min { n , U } 2 3 . g\ge\frac{1}{3}\min\{n,U^2\},\qquad T\ge\frac{U}{2\sqrt{g}},\qquad gT\ge\frac{U\min\{\sqrt{n},U\}}{2\sqrt{3}}. g ≥ 3 1 min { n , U 2 } , T ≥ 2 g U , g T ≥ 2 3 U min { n , U } .
These inequalities imply (D.15).
Number of queries needed to find a link
Lemma D.9 (number of queries needed to find a link). Let T T T links be placed by an injection π : { 0 , … , T − 1 } → { 1 , … , m } \pi:\{0,\ldots,T-1\}\to\{1,\ldots,m\} π : { 0 , … , T − 1 } → { 1 , … , m } , T ≤ m T\le m T ≤ m . At level j j j , the idealized answer contains only the links with indices at most j j j , and the level increases by one only upon a query to the pair π ( j ) \pi(j) π ( j ) . Let Q j Q_j Q j be the number of queries at level j j j before stopping or before time N q r y N_{\mathrm{qry}} N qry , and let R j R_j R j be the indicator that this level has been passed by time N q r y N_{\mathrm{qry}} N qry . Then, for every fixed coordinate map,
E π Q j ≥ m − T + 2 2 E π R j . (41) \mathbb{E}_{\pi}Q_j\ge\frac{m-T+2}{2}\mathbb{E}_{\pi}R_j. \tag*{(41)} E π Q j ≥ 2 m − T + 2 E π R j . (41)
Proof of Lemma D.9.
Fix the placement of all links except j j j . For the placement of link j j j , there remain m − T + 1 m-T+1 m − T + 1 free pairs. Before level j j j , the history does not depend on the placement of this link. At level j j j itself, we compare it with the virtual history in which link j j j is placed in no pair. The runs coincide until the first query to the chosen free pair. Suppose that in the virtual history r r r distinct free pairs are queried before stopping or before time N q r y N_{\mathrm{qry}} N qry .
Exactly r r r placement options lead to passing the level; for the k k k -th of them at least k k k queries are made at this level, and for each of the remaining ( m − T + 1 ) − r (m-T+1)-r ( m − T + 1 ) − r at least r r r . Therefore,
∑ v = 1 m − T + 1 Q j ( v ) ≥ ( a ) ∑ k = 1 r k + ( ( m − T + 1 ) − r ) r = r ( ( m − T + 1 ) + 1 − r 2 ) \sum_{v=1}^{m-T+1}Q_j(v)\underset{(a)}{\ge}\sum_{k=1}^{r}k+\bigl((m-T+1)-r\bigr)r
=r\left((m-T+1)+\frac{1-r}{2}\right) v = 1 ∑ m − T + 1 Q j ( v ) ( a ) ≥ k = 1 ∑ r k + ( ( m − T + 1 ) − r ) r = r ( ( m − T + 1 ) + 2 1 − r )
≥ ( b ) ( m − T + 1 ) + 1 2 ∑ v = 1 m − T + 1 R j ( v ) . \underset{(b)}{\ge}\frac{(m-T+1)+1}{2}\sum_{v=1}^{m-T+1}R_j(v). ( b ) ≥ 2 ( m − T + 1 ) + 1 v = 1 ∑ m − T + 1 R j ( v ) .
Here (a) takes into account the order of the first visits to the free pairs, and (b) uses the relations r ≤ ( m − T + 1 ) r\le(m-T+1) r ≤ ( m − T + 1 ) , ∑ v R j ( v ) = r \sum_vR_j(v)=r ∑ v R j ( v ) = r . If the level is not reached at all, both right-hand quantities are zero. Repeated queries and queries to occupied pairs only increase the left-hand side. Summing over all fixed placements of the other links, we obtain (D.16). This is precisely averaging over all injections: each of them enters exactly one layer with weight 1 / ( m − T + 1 ) 1/(m-T+1) 1/ ( m − T + 1 ) .
Family Q: searching for the pair with the next link
The exponent 2 / 3 2/3 2/3 already appeared in the first part in the lower bound for a single function with bounded initial optimality gap ([30 Theorem B.3] . The family Q used here is defined by exponentials of auxiliary functions and by a random placement of links over pairs of components.
First we construct the family for arbitrary integers m , T ≥ 1 m,T \ge1 m , T ≥ 1 such that 2 m ≤ n 2m \le n 2 m ≤ n , T ≤ m / 2 T \le m/2 T ≤ m /2 and
2 L 1 R / 7 ≥ 3 , 2L_1R/7 \ge3, 2 L 1 R /7 ≥ 3 ,
( 2 L 1 R / 7 ) 2 / 3 ≤ T ≤ 2 ( 2 L 1 R / 7 ) 2 / 3 , (2L_1R/7)^{2/3} \le T \le2(2L_1R/7)^{2/3}, ( 2 L 1 R /7 ) 2/3 ≤ T ≤ 2 ( 2 L 1 R /7 ) 2/3 ,
24 log ( 1 + 2 ε m δ ) ≤ ψ ^ ( 1 ) ( 2 L 1 R / 7 ) 2 / 3 . 24\log\left(1+\frac{2\varepsilon m}{\delta}\right) \le\widehat{\psi}(1)(2L_1R/7)^{2/3}. 24 log ( 1 + δ 2 ε m ) ≤ ψ ( 1 ) ( 2 L 1 R /7 ) 2/3 .
The specific numbers of pairs and links will be set below.
We define the auxiliary functions
Ψ 0 ( u ) = log ( 1 + 2 ε m δ ) ψ ^ ( 1 ) ψ ^ ( u T / R ) , \Psi_0(u)=\frac{\log\left(1+\frac{2\varepsilon m}{\delta}\right)}{\widehat{\psi}(1)}\widehat{\psi}(u\sqrt{T}/R), Ψ 0 ( u ) = ψ ( 1 ) log ( 1 + δ 2 ε m ) ψ ( u T / R ) ,
Ψ ( u ) = 1 48 ψ ^ ( 4 L 1 u / 7 ) . \Psi(u)=\frac{1}{48}\widehat{\psi}(4L_1u/7). Ψ ( u ) = 48 1 ψ ( 4 L 1 u /7 ) .
The links have the form Ψ 0 ( R / T − x c ( 0 ) ) \Psi_0(R/\sqrt{T}-x_{c(0)}) Ψ 0 ( R / T − x c ( 0 ) ) and Ψ ( x c ( j − 1 ) − x c ( j ) ) \Psi(x_{c(j-1)}-x_{c(j)}) Ψ ( x c ( j − 1 ) − x c ( j ) ) , 1 ≤ j < T 1 \le j<T 1 ≤ j < T . The coordinates c ( j ) c(j) c ( j ) are pairwise distinct and different from 0 0 0 . The links are placed over m m m pairs by a uniformly chosen injection π \pi π , so that each pair contains at most one link. Denote by Ψ b \Psi_b Ψ b the sum of the links of pair b b b , and set
ρ ( x ) = L 1 7 ∥ x ∥ 2 + R 2 − 2 L 1 R 7 − 27 M 32 , \rho(x)=\frac{L_1}{7}\sqrt{\lVert x\rVert^2+R^2}-\frac{2L_1R}{7}-\frac{27M}{32}, ρ ( x ) = 7 L 1 ∥ x ∥ 2 + R 2 − 7 2 L 1 R − 32 27 M ,
m b ( x ) = max 27 M / 64 ( Ψ b ( x ) , ρ ( x ) ) . m_b(x)=\max_{27M/64}(\Psi_b(x),\rho(x)). m b ( x ) = 27 M /64 max ( Ψ b ( x ) , ρ ( x )) .
Define the point
x ∗ = R T ∑ j < T e c ( j ) . x^*=\frac{R}{\sqrt{T}}\sum_{j<T}e_{c(j)}. x ∗ = T R j < T ∑ e c ( j ) .
Define the components
f b , ± ( x ) = n δ 2 m ( e ± 2 L 1 x 0 / 7 + m b ( x ) − 1 ) . f_{b,\pm}(x)=\frac{n\delta}{2m}\left(e^{\pm2L_1x_0/7+m_b(x)}-1\right). f b , ± ( x ) = 2 m n δ ( e ± 2 L 1 x 0 /7 + m b ( x ) − 1 ) .
The remaining components are zero. Define the functions
ϕ 0 ( u ) = e Ψ 0 ( u ) − 1 , \phi_0(u)=e^{\Psi_0(u)}-1, ϕ 0 ( u ) = e Ψ 0 ( u ) − 1 ,
ϕ ( u ) = e Ψ ( u ) − 1. \phi(u)=e^{\Psi(u)}-1. ϕ ( u ) = e Ψ ( u ) − 1.
At level p p p , the idealized oracle keeps only the links j ≤ p j \le p j ≤ p and increases the level after a query to the pair π ( p ) \pi(p) π ( p ) . Before the query, the coordinates with indices j < p j<p j < p are revealed, and after it at most one new coordinate is revealed. For the hidden coordinates we use the threshold
min { 7 32 L 1 , ε m 4 ϕ 0 ( R / T ) δ } . \min\left\{\frac{7}{32L_1},\frac{\varepsilon m}{4\phi_0(R/\sqrt{T})\delta}\right\}. min { 32 L 1 7 , 4 ϕ 0 ( R / T ) δ ε m } .
For the next lemma, set
t = ( 2 L 1 R / 7 ) 2 / 3 , m = ⌊ n / 2 ⌋ , T = ⌈ t ⌉ , η m = log ( 1 + 2 ε m / δ ) . t=(2L_1R/7)^{2/3},\qquad m=\lfloor n/2\rfloor,\qquad T=\lceil t\rceil,\qquad\eta_m=\log(1+2\varepsilon m/\delta). t = ( 2 L 1 R /7 ) 2/3 , m = ⌊ n /2 ⌋ , T = ⌈ t ⌉ , η m = log ( 1 + 2 ε m / δ ) .
Lemma D.10. If
L 1 R ≥ 21 2 , 4 t ≤ m , 24 η m ≤ ψ ^ ( 1 ) t , (42) L_1R \ge\frac{21}{2}, \qquad4t \le m, \qquad24\eta_m \le\widehat{\psi}(1)t, \tag*{(42)} L 1 R ≥ 2 21 , 4 t ≤ m , 24 η m ≤ ψ ( 1 ) t , (42) then
T ∗ ≳ n ( L 1 R ) 2 / 3 ( 1 + η m ) 2 . (43) \mathcal{T}^* \gtrsim\frac{n(L_1R)^{2/3}}{(1+\eta_m)^2}. \tag*{(43)} T ∗ ≳ ( 1 + η m ) 2 n ( L 1 R ) 2/3 . (43)
Proof of Lemma D.10. The first condition in (D.17) allows us to extend the links beyond the ball B ( 0 , R ) B(0,R) B ( 0 , R ) . The second gives t ≤ T ≤ 2 t ≤ m / 2 t \le T \le2t \le m/2 t ≤ T ≤ 2 t ≤ m /2 : when searching for the next link, at least half of the pairs remain free. The third bounds the curvature of the first link.
▷ For an ordinary link, (D.4) implies the bounds L 1 / 14 L_1/14 L 1 /14 for the gradient norm and 4 L 1 2 / 147 4L_1^2/147 4 L 1 2 /147 for the Hessian norm. For the first link, the same bounds follow from
3 T η m ψ ^ ( 1 ) R ≤ 6 t η m ψ ^ ( 1 ) R ≤ t 3 / 2 4 R = L 1 14 , \frac{3\sqrt{T}\eta_m}{\widehat{\psi}(1)R} \le\frac{6\sqrt{t}\eta_m}{\widehat{\psi}(1)R} \le\frac{t^{3/2}}{4R} = \frac{L_1}{14}, ψ ( 1 ) R 3 T η m ≤ ψ ( 1 ) R 6 t η m ≤ 4 R t 3/2 = 14 L 1 ,
T η m ψ ^ ( 1 ) R 2 ≤ t 2 12 R 2 ≤ 4 L 1 2 147 . \frac{T\eta_m}{\widehat{\psi}(1)R^2} \le\frac{t^2}{12R^2} \le\frac{4L_1^2}{147}. ψ ( 1 ) R 2 T η m ≤ 12 R 2 t 2 ≤ 147 4 L 1 2 .
For 2 L 1 R / 7 ≥ 3 2L_1R/7 \ge3 2 L 1 R /7 ≥ 3 , Lemma D.4 applies with G = L 1 / 14 G=L_1/14 G = L 1 /14 , L a u x = 4 L 1 2 / 49 L_{\mathrm{aux}}=4L_1^2/49 L aux = 4 L 1 2 /49 , r = R r=R r = R , C ∗ = 0 C_* = 0 C ∗ = 0 . Therefore,
∥ ∇ m b ∥ ≤ L 1 / 7 , \lVert\nabla m_b\rVert\le L_1/7, ∥ ∇ m b ∥ ≤ L 1 /7 ,
∥ ∇ 2 m b ∥ ≤ 4 L 1 2 / 49. \lVert\nabla^2m_b\rVert\le4L_1^2/49. ∥ ∇ 2 m b ∥ ≤ 4 L 1 2 /49.
All potentials are nonnegative. The point x ∗ x^* x ∗ has norm R R R , and all links and potentials vanish at it. At zero, only the potential of the pair containing link 0 is nonzero; it equals η m \eta_m η m . As for family X X X , the minimum of the average function equals zero, the heterogeneity equals δ \delta δ , and
f ( 0 ) = 2 ε ≤ Δ . f(0)=2\varepsilon\le\Delta. f ( 0 ) = 2 ε ≤ Δ.
Thus, Assumptions 2.3–2.5 hold. Assumptions 2.1 and 2.2 follow from Lemma D.7.
▷ We prove a lower bound on the optimality gap. We have
ϕ 0 ( R / T ) = e η m − 1 = 2 ε m δ , \phi_0(R/\sqrt{T})=e^{\eta_m}-1=\frac{2\varepsilon m}{\delta}, ϕ 0 ( R / T ) = e η m − 1 = δ 2 ε m ,
ϕ 0 ′ ( R / T ) = G H T R ψ ^ ( 1 ) e η m η m . \phi'_0(R/\sqrt{T})=\frac{G_H\sqrt{T}}{R\widehat{\psi}(1)}e^{\eta_m}\eta_m. ϕ 0 ′ ( R / T ) = R ψ ( 1 ) G H T e η m η m .
Denote s 0 = ϕ 0 ′ ( R / T ) > 0 s_0=\phi'_0(R/\sqrt{T})>0 s 0 = ϕ 0 ′ ( R / T ) > 0 , log + u = max { 0 , log u } \log_+u=\max\{0,\log u\} log + u = max { 0 , log u } and
J = 1 + 16 log + ( 56 s 0 / L 1 ) , K = 7 s 0 2 L 1 J . J=1+16\log_+(56s_0/L_1), \qquad K=\frac{7s_0}{2L_1}J. J = 1 + 16 log + ( 56 s 0 / L 1 ) , K = 2 L 1 7 s 0 J .
The quantity K K K bounds the loss in the lower bound when one link is added:
sup u ∈ R { s 0 u − ϕ ( u ) } ≤ K . (44) \sup_{u\in\mathbb{R}}\{s_0u-\phi(u)\}\le K. \tag*{(44)} u ∈ R sup { s 0 u − ϕ ( u )} ≤ K . (44)
For u ≤ 7 / ( 2 L 1 ) u\le7/(2L_1) u ≤ 7/ ( 2 L 1 ) it suffices that ϕ ≥ 0 \phi\ge0 ϕ ≥ 0 . For u > 7 / ( 2 L 1 ) u>7/(2L_1) u > 7/ ( 2 L 1 ) , set v = 4 L 1 u / 7 − 2 > 0 v=4L_1u/7-2>0 v = 4 L 1 u /7 − 2 > 0 . By (D.4), Ψ ( u ) ≥ v / 32 \Psi(u)\ge v/32 Ψ ( u ) ≥ v /32 . For arbitrary μ ≥ 0 \mu\ge0 μ ≥ 0 , a > 0 a>0 a > 0 , v ≥ 0 v\ge0 v ≥ 0 , the tangent to the exponential at log ( μ / a ) \log(\mu/a) log ( μ / a ) for μ > a \mu>a μ > a and e a v ≥ 1 + a v e^{av}\ge1+av e a v ≥ 1 + a v for μ ≤ a \mu\le a μ ≤ a give
μ v − ( e a v − 1 ) ≤ μ a max { 0 , log ( μ / a ) } . \mu v-(e^{av}-1)\le\frac{\mu}{a}\max\{0,\log(\mu/a)\}. μv − ( e a v − 1 ) ≤ a μ max { 0 , log ( μ / a )} .
Substituting μ = 7 s 0 / ( 4 L 1 ) \mu=7s_0/(4L_1) μ = 7 s 0 / ( 4 L 1 ) and a = 1 / 32 a=1/32 a = 1/32 , we obtain
s 0 u − ϕ ( u ) ≤ 7 s 0 2 L 1 + 56 s 0 L 1 log + ( 56 s 0 / L 1 ) = K . s_0u-\phi(u)\le\frac{7s_0}{2L_1}+\frac{56s_0}{L_1}\log_+(56s_0/L_1)=K. s 0 u − ϕ ( u ) ≤ 2 L 1 7 s 0 + L 1 56 s 0 log + ( 56 s 0 / L 1 ) = K .
If the first p < T p<T p < T coordinates are revealed and the absolute value of the next coordinate is less than the chosen threshold, then the sum of the differences R / T − x c ( 0 ) R/\sqrt{T}-x_{c(0)} R / T − x c ( 0 ) and x c ( j − 1 ) − x c ( j ) x_{c(j-1)}-x_{c(j)} x c ( j − 1 ) − x c ( j ) , 1 ≤ j ≤ p 1\le j\le p 1 ≤ j ≤ p , equals R / T − x c ( p ) R/\sqrt{T}-x_{c(p)} R / T − x c ( p ) . The tangent to ϕ 0 \phi_0 ϕ 0 and (D.19) give
ϕ 0 ( R / T − x c ( 0 ) ) + ∑ j = 1 p ϕ ( x c ( j − 1 ) − x c ( j ) ) ≥ 2 ε m δ − s 0 x c ( p ) − p K . (45) \phi_0\left(R/\sqrt{T}-x_{c(0)}\right)+\sum_{j=1}^{p}\phi\left(x_{c(j-1)}-x_{c(j)}\right)\ge\frac{2\varepsilon m}{\delta}-s_0x_{c(p)}-pK. \tag*{(45)} ϕ 0 ( R / T − x c ( 0 ) ) + j = 1 ∑ p ϕ ( x c ( j − 1 ) − x c ( j ) ) ≥ δ 2 ε m − s 0 x c ( p ) − p K . (45)
The injectivity of π \pi π allows us to bound the average function from below by the sum of these exponential links: an empty pair makes a nonnegative contribution, and a pair with link j j j contributes at least ( δ / m ) ϕ ( x c ( j − 1 ) − x c ( j ) ) (\delta/m)\phi(x_{c(j-1)}-x_{c(j)}) ( δ / m ) ϕ ( x c ( j − 1 ) − x c ( j ) ) .
▷ The threshold for the hidden coordinates chosen in the construction of Q does not exceed 7 / ( 32 L 1 ) 7/(32L_1) 7/ ( 32 L 1 ) ; therefore, the distant links vanish in a neighborhood of the query point. For
∥ x ∥ ≥ 4 R + 14 L 1 log ( 1 + 2 ε m δ ) + 189 M 8 L 1 , \lVert x\rVert\ge4R+\frac{14}{L_1}\log\left(1+\frac{2\varepsilon m}{\delta}\right)+\frac{189M}{8L_1}, ∥ x ∥ ≥ 4 R + L 1 14 log ( 1 + δ 2 ε m ) + 8 L 1 189 M ,
by Lemma D.4, the full and truncated components are determined only by the radial extension and the coordinate x 0 x_0 x 0 . All potentials are at least log ( 1 + 2 ε m / δ ) \log(1+2\varepsilon m/\delta) log ( 1 + 2 ε m / δ ) , and the optimality gap is at least the initial one. Thus, the histories coincide for arbitrary distant queries as well. By (D.20) and the choice of the threshold, the optimality gap after passing p p p levels is at least A − B p A-Bp A − B p , where one can take
A = 7 ε / 4 , B = K δ m = 7 s 0 δ 2 m L 1 J . A=7\varepsilon/4,\qquad B=\frac{K\delta}{m}=\frac{7s_0\delta}{2mL_1}J. A = 7 ε /4 , B = m K δ = 2 m L 1 7 s 0 δ J .
Indeed, after multiplication by δ / m \delta/m δ / m , the term with the hidden coordinate in (D.20) causes a loss of at most ε / 4 \varepsilon/4 ε /4 . To apply Lemma D.3 , we verify two relations. From
s 0 = G H T ψ ^ ( 1 ) R e η m η m , δ m = 2 ε e η m − 1 s_0=\frac{G_H\sqrt{T}}{\widehat{\psi}(1)R}e^{\eta_m}\eta_m,\qquad\frac{\delta}{m}=\frac{2\varepsilon}{e^{\eta_m}-1} s 0 = ψ ( 1 ) R G H T e η m η m , m δ = e η m − 1 2 ε
and T ≤ 2 t \sqrt{T}\le2\sqrt{t} T ≤ 2 t it follows that
A B = ψ ^ ( 1 ) L 1 R 4 G H T 1 − e − η m η m J ≥ 7 ψ ^ ( 1 ) 16 G H t ( 1 + η m ) J . (46) \frac{A}{B}=\frac{\widehat{\psi}(1)L_1R}{4G_H\sqrt{T}}\frac{1-e^{-\eta_m}}{\eta_mJ}\ge\frac{7\widehat{\psi}(1)}{16G_H}\frac{t}{(1+\eta_m)J}. \tag*{(46)} B A = 4 G H T ψ ( 1 ) L 1 R η m J 1 − e − η m ≥ 16 G H 7 ψ ( 1 ) ( 1 + η m ) J t . (46)
Here we used the equality L 1 R = 7 t 3 / 2 / 2 L_1R=7t^{3/2}/2 L 1 R = 7 t 3/2 /2 and the inequality 1 − e − u ≥ u / ( 1 + u ) 1-e^{-u}\ge u/(1+u) 1 − e − u ≥ u / ( 1 + u ) for u ≥ 0 u\ge0 u ≥ 0 . Moreover, T 3 / 2 ≥ 2 L 1 R / 7 T^{3/2}\ge2L_1R/7 T 3/2 ≥ 2 L 1 R /7 , J ≥ 1 J\ge1 J ≥ 1 and G H / ψ ^ ( 1 ) ≥ 1 G_H/\widehat{\psi}(1)\ge1 G H / ψ ( 1 ) ≥ 1 give
T B = 7 δ 2 m L 1 G H T 3 / 2 R e η m η m J ≥ δ m e η m η m ≥ 2 ε > A . TB=\frac{7\delta}{2mL_1}\frac{G_HT^{3/2}}{R}e^{\eta_m}\eta_mJ\ge\frac{\delta}{m}e^{\eta_m}\eta_m\ge2\varepsilon>A. T B = 2 m L 1 7 δ R G H T 3/2 e η m η m J ≥ m δ e η m η m ≥ 2 ε > A .
For p = T p=T p = T , the bound on the optimality gap follows from f ≥ 0 f\ge0 f ≥ 0 .
▷ Summing (D.16) over the levels, we obtain E Q ≥ c τ E P \mathrm{EQ}\ge c_\tau\mathrm{EP} EQ ≥ c τ EP with c τ = ( m − T + 2 ) / 2 ≥ m / 4 c_\tau=(m-T+2)/2\ge m/4 c τ = ( m − T + 2 ) /2 ≥ m /4 . Lemma D.3 with s = T s=T s = T gives T ∗ ≥ m A / ( 16 B ) T^*\ge mA/(16B) T ∗ ≥ m A / ( 16 B ) . It remains to bound J J J . By the third condition in (D.17),
56 s 0 L 1 ≤ 32 G H ψ ^ ( 1 ) e η m η m t ≤ 4 G H 3 e η m = 5 6 e η m . \frac{56s_0}{L_1}\le\frac{32G_H}{\widehat{\psi}(1)}\frac{e^{\eta_m}\eta_m}{t}\le\frac{4G_H}{3}e^{\eta_m}=\frac{5}{6}e^{\eta_m}. L 1 56 s 0 ≤ ψ ( 1 ) 32 G H t e η m η m ≤ 3 4 G H e η m = 6 5 e η m .
Consequently, J < 1 + 16 η m J<1+16\eta_m J < 1 + 16 η m . Substituting this bound into (D.21) and using m ≥ n / 3 m\ge n/3 m ≥ n /3 and t = ( 2 / 7 ) 2 / 3 ( L 1 R ) 2 / 3 t=(2/7)^{2/3}(L_1R)^{2/3} t = ( 2/7 ) 2/3 ( L 1 R ) 2/3 , we obtain (D.18).
Proof of the lower bound
Proof of Theorem 7.1 ▲. It suffices to bound the four terms B n B_n B n , B S B_S B S , B X B_X B X , B Q B_Q B Q from (10) : their sum is of order T 0 T_0 T 0 . Set a = L 1 R a=L_1R a = L 1 R ; we use the quantity N 0 N_0 N 0 from (5) . The numbers G H G_H G H , ψ ^ ( 1 ) \widehat{\psi}(1) ψ ( 1 ) and c H c_H c H are computed in (D.3).
The L 0 L_0 L 0 term
By Lemma D.5,
B n = n ≤ 9 T ∗ . (47) B_n=n\le9T^*. \tag*{(47)} B n = n ≤ 9 T ∗ . (47)
If (D.11) holds, then Lemma D.6 applies. Otherwise,
B S < 24 c H ( G H + 6 ) n < 50 n . B_S<\sqrt{\frac{24}{c_H}}(G_H+6)n<50n. B S < c H 24 ( G H + 6 ) n < 50 n .
In both cases,
B S = n L 0 R 2 ε ≲ T ∗ . (48) B_S=\sqrt{\frac{nL_0R^2}{\varepsilon}}\lesssim T^*. \tag*{(48)} B S = ε n L 0 R 2 ≲ T ∗ . (48)
Terms obtained from family X
We use the notation η \eta η , ρ \rho ρ , D δ D_\delta D δ , U U U from (D.14). By Assumption 2.6 , ε / δ ≤ a 2 / 3 \varepsilon/\delta\le a^{2/3} ε / δ ≤ a 2/3 , whence
η ≤ log ( 1 + 2 a 2 / 3 ) ≤ log N 0 , 1 ≤ ρ ≤ 1 + η ≤ 1 + log N 0 . (49) \eta\le\log(1+2a^{2/3})\le\log N_0,\qquad1\le\rho\le1+\eta\le1+\log N_0. \tag*{(49)} η ≤ log ( 1 + 2 a 2/3 ) ≤ log N 0 , 1 ≤ ρ ≤ 1 + η ≤ 1 + log N 0 . (49)
For the second bound, the inequalities z / ( 1 + z ) ≤ log ( 1 + z ) ≤ z z/(1+z)\le\log(1+z)\le z z / ( 1 + z ) ≤ log ( 1 + z ) ≤ z with z = 2 ε / δ z=2\varepsilon/\delta z = 2 ε / δ suffice. Moreover,
D δ 1 + δ / ε ≲ log N 0 . (50) D_\delta\sqrt{1+\delta/\varepsilon}\lesssim\log N_0. \tag*{(50)} D δ 1 + δ / ε ≲ log N 0 . (50)
Indeed, η δ / ε ≤ 2 \eta\delta/\varepsilon\le2 η δ / ε ≤ 2 by the inequality log ( 1 + z ) ≤ z \log(1+z)\le z log ( 1 + z ) ≤ z , and 1 + δ / ε ≤ 1 + δ / ( 2 ε ) \sqrt{1+\delta/\varepsilon}\le1+\delta/(2\varepsilon) 1 + δ / ε ≤ 1 + δ / ( 2 ε ) . Therefore,
D δ 1 + δ / ε ≤ max { 12 ( η + 2 ) c H , 4 ( G H + 6 ) c H ρ } ≲ log N 0 . D_\delta\sqrt{1+\delta/\varepsilon}\le\max\left\{\sqrt{\frac{12(\eta+2)}{c_H}},\frac{4(G_H+6)}{c_H}\rho\right\}\lesssim\log N_0. D δ 1 + δ / ε ≤ max ⎩ ⎨ ⎧ c H 12 ( η + 2 ) , c H 4 ( G H + 6 ) ρ ⎭ ⎬ ⎫ ≲ log N 0 .
We show that
n U ≲ log N 0 T ∗ . (51) \sqrt{n}U\lesssim\log N_0T^*. \tag*{(51)} n U ≲ log N 0 T ∗ . (51)
For U ≥ n U\ge\sqrt{n} U ≥ n , this follows from Lemma D.8 and (E.3). For 1 ≤ U < n 1\le U<\sqrt{n} 1 ≤ U < n , the same lemma gives U 2 ≲ ρ T ∗ U^2\lesssim\rho T^* U 2 ≲ ρ T ∗ . Multiplying this inequality and (E.1) side by side, we obtain n U ≲ ρ T ∗ ≲ log N 0 T ∗ \sqrt{n}U\lesssim\sqrt{\rho}T^*\lesssim\log N_0T^* n U ≲ ρ T ∗ ≲ log N 0 T ∗ . Finally, for U < 1 U<1 U < 1 it suffices that n U ≤ n \sqrt{n}U\le n n U ≤ n . From a = 7 U D δ / 2 a=7UD_\delta/2 a = 7 U D δ /2 , (E.4) and (E.5) it follows that
B X = n a 1 + δ / ε ≲ log 2 N 0 T ∗ . (52) B_X=\sqrt{n}a\sqrt{1+\delta/\varepsilon}\lesssim\log^2N_0T^*. \tag*{(52)} B X = n a 1 + δ / ε ≲ log 2 N 0 T ∗ . (52)
The term with exponent 2 / 3 2/3 2/3
Set t = ( 2 a / 7 ) 2 / 3 t=(2a/7)^{2/3} t = ( 2 a /7 ) 2/3 , m = ⌊ n / 2 ⌋ m=\lfloor n/2\rfloor m = ⌊ n /2 ⌋ and η m = log ( 1 + 2 ε m / δ ) \eta_m=\log(1+2\varepsilon_m/\delta) η m = log ( 1 + 2 ε m / δ ) . By Assumption 2.6 ,
η m ≤ log ( 1 + n a 2 / 3 ) ≤ 2 log N 0 . \eta_m\le\log(1+na^{2/3})\le2\log N_0. η m ≤ log ( 1 + n a 2/3 ) ≤ 2 log N 0 .
If conditions (D.17) hold, Lemma D.10 gives
B Q = n a 2 / 3 ≲ ( 1 + η m ) 2 T ∗ ≲ log 2 N 0 T ∗ . (53) B_Q=na^{2/3}\lesssim(1+\eta_m)^2T^*\lesssim\log^2N_0T^*. \tag*{(53)} B Q = n a 2/3 ≲ ( 1 + η m ) 2 T ∗ ≲ log 2 N 0 T ∗ . (53)
If at least one of the conditions is violated, the bound follows from those already proved:
If a < 2 1 / 2 a<2^{1/2} a < 2 1/2 , then B Q < 5 n B_Q<5n B Q < 5 n .
If a ≥ 2 1 / 2 a\ge2^{1/2} a ≥ 2 1/2 but m < 4 t m<4t m < 4 t , then t > 1 t>1 t > 1 and n ≤ 2 m + 1 < 9 t n\le2m+1<9t n ≤ 2 m + 1 < 9 t . Therefore,
B Q n a = n a 1 / 3 < 3 ( 2 / 7 ) 1 / 3 < 2 , B Q < 2 B X . \frac{B_Q}{\sqrt{n}a}=\frac{\sqrt{n}}{a^{1/3}}<3(2/7)^{1/3}<2,\qquad B_Q<2B_X. n a B Q = a 1/3 n < 3 ( 2/7 ) 1/3 < 2 , B Q < 2 B X .
Thus, bound (E.7) holds in all cases. Adding the bounds for the four terms, we obtain
T 0 ∼ B n + B S + B X + B Q ≲ log 2 N 0 T ∗ . T_0\sim B_n+B_S+B_X+B_Q\lesssim\log^2N_0T^*. T 0 ∼ B n + B S + B X + B Q ≲ log 2 N 0 T ∗ .
Variants of YETI
Why the local curvature must be bounded
The condition ∥ y − x ∗ ∥ ≤ R \lVert y-x^* \rVert\le R ∥ y − x ∗ ∥ ≤ R by itself does not yield a bound on the ratio of local smoothness constants that is polynomial in L 1 R L_1R L 1 R . Consider the one-dimensional function
φ ( t ) = cosh ( L 1 t ) . \varphi(t)=\cosh(L_1t). φ ( t ) = cosh ( L 1 t ) .
It is convex, attains its minimum at zero, and satisfies
φ ′ ′ ( t ) = L 1 2 cosh ( L 1 t ) ≤ L 1 2 + L 1 ∣ φ ′ ( t ) ∣ . \varphi''(t)=L_1^2\cosh(L_1t)\le L_1^2+L_1|\varphi'(t)|. φ ′′ ( t ) = L 1 2 cosh ( L 1 t ) ≤ L 1 2 + L 1 ∣ φ ′ ( t ) ∣.
However,
φ ′ ′ ( R ) φ ′ ′ ( 0 ) = cosh ( L 1 R ) \frac{\varphi''(R)}{\varphi''(0)}=\cosh(L_1R) φ ′′ ( 0 ) φ ′′ ( R ) = cosh ( L 1 R )
grows exponentially. Therefore, the mere fact that the centers lie in the ball is not sufficient to drop the curvature test while preserving the same bound on the number of oracle calls per local call. This example concerns only the solver used here and does not rule out the existence of other algorithms.
Choice of the stochastic gradient
Let the functions φ i \varphi_i φ i have local smoothness constants L i > 0 L_i>0 L i > 0 on the enlarged ball, and let the points x , w x,w x , w lie in the original ball. For a random index I I I with probabilities q i > 0 q_i>0 q i > 0 , Lemma A.3 gives
E ∥ ∇ φ I ( x ) − ∇ φ I ( w ) n q I ∥ 2 = 1 n 2 ∑ i ∥ ∇ φ i ( x ) − ∇ φ i ( w ) ∥ 2 q i ≤ 2 n 2 ∑ i L i q i ( φ i ( x ) − φ i ( w ) − ⟨ ∇ φ i ( w ) , x − w ⟩ ) . (54) \begin{aligned}
\mathbb{E}\left\lVert\frac{\nabla\varphi_I(x)-\nabla\varphi_I(w)}{nq_I}\right\rVert^2
=\frac{1}{n^2}\sum_i\frac{\lVert\nabla\varphi_i(x)-\nabla\varphi_i(w)\rVert^2}{q_i} \\
&\le\frac{2}{n^2}\sum_i\frac{L_i}{q_i}\left(\varphi_i(x)-\varphi_i(w)-\langle\nabla\varphi_i(w),x-w\rangle\right). \tag*{(54)}
\end{aligned} E n q I ∇ φ I ( x ) − ∇ φ I ( w ) 2 = n 2 1 i ∑ q i ∥ ∇ φ i ( x ) − ∇ φ i ( w ) ∥ 2 ≤ n 2 2 i ∑ q i L i ( φ i ( x ) − φ i ( w ) − ⟨ ∇ φ i ( w ) , x − w ⟩ ) . (54)
If
q i = L i ∑ j L j , q_i=\frac{L_i}{\sum_j L_j}, q i = ∑ j L j L i ,
then the last line equals
( 2 n ∑ i L i ) 1 n ∑ i ( φ i ( x ) − φ i ( w ) − ⟨ ∇ φ i ( w ) , x − w ⟩ ) . \left(\frac{2}{n}\sum_i L_i\right)\frac{1}{n}\sum_i\left(\varphi_i(x)-\varphi_i(w)-\langle\nabla\varphi_i(w),x-w\rangle\right). ( n 2 i ∑ L i ) n 1 i ∑ ( φ i ( x ) − φ i ( w ) − ⟨ ∇ φ i ( w ) , x − w ⟩ ) .
Under uniform sampling, the standard upper bound contains 2 max i L i 2\max_i L_i 2 max i L i instead of 2 n − 1 ∑ i L i 2n^{-1}\sum_i L_i 2 n − 1 ∑ i L i . The ratio of these quantities can be of order n n n even within the class under consideration. In the one-dimensional case with L 0 > 0 L_0>0 L 0 > 0 , take
f 1 ( t ) = L 0 2 ( t − R ) 2 , f_1(t)=\frac{L_0}{2}(t-R)^2, f 1 ( t ) = 2 L 0 ( t − R ) 2 ,
f i ( t ) = 0 ( i > 1 ) . f_i(t)=0\quad(i>1). f i ( t ) = 0 ( i > 1 ) .
Set the known upper bounds L 0 , i = L 0 L_{0,i}=L_0 L 0 , i = L 0 for all i i i . Choose the parameters
r = 1 2 L 1 , r=\frac{1}{2L_1}, r = 2 L 1 1 ,
Then the constants from (H.2), that is,
L i = e 3 L 1 r ( L 0 , i + L 1 ∥ ∇ f i ( y ) ∥ ) + λ , L_i=e^{3L_1r}\left(L_{0,i}+L_1\lVert\nabla f_i(y)\rVert\right)+\lambda, L i = e 3 L 1 r ( L 0 , i + L 1 ∥ ∇ f i ( y )∥ ) + λ ,
satisfy
max i L i n − 1 ∑ i L i = 1 + e 3 / 2 ( 1 + n ) 1 + 2 e 3 / 2 . \frac{\max_i L_i}{n^{-1}\sum_i L_i}
=\frac{1+e^{3/2}(1+n)}{1+2e^{3/2}}. n − 1 ∑ i L i max i L i = 1 + 2 e 3/2 1 + e 3/2 ( 1 + n ) .
This ratio grows linearly in n n n . Hence, for this bound of the inner solver, non-uniform sampling is essential. It does not follow that every method with uniform sampling is necessarily worse.
The YETI solver uses the probabilities
q i = L i ∑ j L j . q_i = \frac{L_i}{\sum_j L_j}. q i = ∑ j L j L i .
If another distribution guarantees only
L i n q i ≤ c n ∑ j L j , \frac{L_i}{nq_i} \le\frac{c}{n}\sum_j L_j, n q i L i ≤ n c j ∑ L j ,
then a factor c c c appears on the right-hand side of bound (F.1). Its effect on the complexity bound must be taken into account together with the choice of the epoch parameters.
Storing the anchor gradient is also essential. For a > 0 a > 0 a > 0 and the functions
f 1 ( t ) = ( t − a ) 2 2 , f_1(t) = \frac{(t-a)^2}{2}, f 1 ( t ) = 2 ( t − a ) 2 ,
f 2 ( t ) = ( t + a ) 2 2 , f_2(t) = \frac{(t+a)^2}{2}, f 2 ( t ) = 2 ( t + a ) 2 ,
the minimum of their average is attained at zero, but
1 2 ∑ i = 1 2 ∣ f i ′ ( 0 ) ∣ 2 = a 2 > 0. \frac{1}{2}\sum_{i=1}^{2} \lvert f_i'(0)\rvert^2 = a^2 > 0. 2 1 i = 1 ∑ 2 ∣ f i ′ ( 0 ) ∣ 2 = a 2 > 0.
Therefore, the gradient of a randomly chosen component has nonzero variance at the solution. In bound (F.1), by contrast, the gradient differences vanish at x = w x = w x = w . Without the correction based on the stored gradients, the variance at the solution may remain positive, and the above proof of geometric convergence does not apply.
Componentwise secants for generalized linear models
Let
f i ( x ) = ϕ i ( a i T x ) , f_i(x) = \phi_i(a_i^{\mathsf{T}}x), f i ( x ) = ϕ i ( a i T x ) ,
where ϕ i \phi_i ϕ i is convex and twice continuously differentiable. Using the already computed gradients at the points x 0 , x 1 x_0, x_1 x 0 , x 1 , define
s = x 1 − x 0 , s = x_1 - x_0, s = x 1 − x 0 ,
Δ g i = ∇ f i ( x 1 ) − ∇ f i ( x 0 ) . \Delta g_i = \nabla f_i(x_1) - \nabla f_i(x_0). Δ g i = ∇ f i ( x 1 ) − ∇ f i ( x 0 ) .
Proposition F.1 (recovering the average Hessian). If ⟨ Δ g i , s ⟩ > 0 \langle\Delta g_i, s\rangle> 0 ⟨ Δ g i , s ⟩ > 0 , then
Δ g i Δ g i T ⟨ Δ g i , s ⟩ = ∫ 0 1 ∇ 2 f i ( x 0 + t s ) d t . (55) \frac{\Delta g_i \Delta g_i^{\mathsf{T}}}{\langle\Delta g_i, s\rangle}
=
\int_0^1 \nabla^2 f_i(x_0 + ts)\,\mathrm{d}t. \tag*{(55)} ⟨ Δ g i , s ⟩ Δ g i Δ g i T = ∫ 0 1 ∇ 2 f i ( x 0 + t s ) d t . (55) If a i T s = 0 a_i^{\mathsf{T}}s = 0 a i T s = 0 , then a zero secant may correspond to a nonzero matrix on the right-hand side. If, however, the average scalar curvature is zero and the secant is explicitly defined as the zero matrix (without division by zero), then both sides of the equality are zero.
Proof. By the Newton–Leibniz formula,
Δ g i = ( a i T s ) a i ∫ 0 1 ϕ i ′ ′ ( a i T ( x 0 + t s ) ) d t , \Delta g_i = (a_i^{\mathsf{T}}s)a_i \int_0^1 \phi_i''(a_i^{\mathsf{T}}(x_0 + ts))\,\mathrm{d}t, Δ g i = ( a i T s ) a i ∫ 0 1 ϕ i ′′ ( a i T ( x 0 + t s )) d t ,
⟨ Δ g i , s ⟩ = ( a i T s ) 2 ∫ 0 1 ϕ i ′ ′ ( a i T ( x 0 + t s ) ) d t , \langle\Delta g_i, s\rangle= (a_i^{\mathsf{T}}s)^2 \int_0^1 \phi_i''(a_i^{\mathsf{T}}(x_0 + ts))\,\mathrm{d}t, ⟨ Δ g i , s ⟩ = ( a i T s ) 2 ∫ 0 1 ϕ i ′′ ( a i T ( x 0 + t s )) d t ,
∫ 0 1 ∇ 2 f i ( x 0 + t s ) d t = a i a i T ∫ 0 1 ϕ i ′ ′ ( a i T ( x 0 + t s ) ) d t . \int_0^1 \nabla^2 f_i(x_0 + ts)\,\mathrm{d}t
=
a_i a_i^{\mathsf{T}} \int_0^1 \phi_i''(a_i^{\mathsf{T}}(x_0 + ts))\,\mathrm{d}t. ∫ 0 1 ∇ 2 f i ( x 0 + t s ) d t = a i a i T ∫ 0 1 ϕ i ′′ ( a i T ( x 0 + t s )) d t .
When the denominator is positive, cancelling the scalar factors gives (F.2). When a i T s = 0 a_i^{\mathsf{T}}s=0 a i T s = 0 , the equality Δ g i = 0 \Delta g_i=0 Δ g i = 0 does not determine the integral of ϕ i ′ ′ \phi_i^{\prime\prime} ϕ i ′′ . If, however, this integral is zero, then the average Hessian and the gradient difference are zero.
Thus, a positive componentwise secant recovers the average Hessian along the segment. Obtaining the Hessian at the endpoint of the segment requires an additional assumption.
Proposition F.2 (approximating the Hessian of the Poisson model). Let
f i ( x ) = e a i T x − y i a i T x , f_i(x)=e^{a_i^{\mathsf{T}}x}-y_i a_i^{\mathsf{T}}x, f i ( x ) = e a i T x − y i a i T x , where y i ≥ 0 y_i \ge0 y i ≥ 0 and ∥ a i ∥ ≤ L 1 \lVert a_i\rVert\le L_1 ∥ a i ∥ ≤ L 1 . These components satisfy the condition of the class with
L 0 , i = y i ∥ a i ∥ 2 . L_{0,i}=y_i\lVert a_i\rVert^2. L 0 , i = y i ∥ a i ∥ 2 . If a i T s ≠ 0 a_i^{\mathsf{T}}s\ne0 a i T s = 0 for every nonzero a i a_i a i , then the matrix M M M obtained by averaging the secants (F.2) satisfies
e − L 1 ∥ s ∥ ∇ 2 f ( x 1 ) ⪯ M ⪯ e L 1 ∥ s ∥ ∇ 2 f ( x 1 ) . (56) e^{-L_1\lVert s\rVert}\nabla^2 f(x_1)\preceq M\preceq e^{L_1\lVert s\rVert}\nabla^2 f(x_1). \tag*{(56)} e − L 1 ∥ s ∥ ∇ 2 f ( x 1 ) ⪯ M ⪯ e L 1 ∥ s ∥ ∇ 2 f ( x 1 ) . (56) Zero a i a_i a i contribute zero matrices to the sum.
Proof. Directly from the form of the component we obtain
∥ ∇ 2 f i ( x ) ∥ = e a i T x ∥ a i ∥ 2 ≤ y i ∥ a i ∥ 2 + ∣ e a i T x − y i ∣ ∥ a i ∥ 2 ≤ L 0 , i + L 1 ∥ ∇ f i ( x ) ∥ . \lVert\nabla^2 f_i(x)\rVert=e^{a_i^{\mathsf{T}}x}\lVert a_i\rVert^2\le y_i\lVert a_i\rVert^2+\lvert e^{a_i^{\mathsf{T}}x}-y_i\rvert\lVert a_i\rVert^2\le L_{0,i}+L_1\lVert\nabla f_i(x)\rVert. ∥ ∇ 2 f i ( x )∥ = e a i T x ∥ a i ∥ 2 ≤ y i ∥ a i ∥ 2 + ∣ e a i T x − y i ∣ ∥ a i ∥ 2 ≤ L 0 , i + L 1 ∥ ∇ f i ( x )∥ .
Moreover, for 0 ≤ t ≤ 1 0\le t\le1 0 ≤ t ≤ 1 , by the Cauchy–Schwarz inequality,
∣ a i T ( x 0 + t s − x 1 ) ∣ ≤ L 1 ∥ s ∥ . \lvert a_i^{\mathsf{T}}(x_0+ts-x_1)\rvert\le L_1\lVert s\rVert. ∣ a i T ( x 0 + t s − x 1 )∣ ≤ L 1 ∥ s ∥ .
Therefore,
e − L 1 ∥ s ∥ e a i T x 1 ≤ e a i T ( x 0 + t s ) ≤ e L 1 ∥ s ∥ e a i T x 1 . e^{-L_1\lVert s\rVert}e^{a_i^{\mathsf{T}}x_1}\le e^{a_i^{\mathsf{T}}(x_0+ts)}\le e^{L_1\lVert s\rVert}e^{a_i^{\mathsf{T}}x_1}. e − L 1 ∥ s ∥ e a i T x 1 ≤ e a i T ( x 0 + t s ) ≤ e L 1 ∥ s ∥ e a i T x 1 .
Multiplying by the positive semidefinite matrix a i a i T a_i a_i^{\mathsf{T}} a i a i T , integrating, and averaging over i i i , we obtain (F.3).
Applying the main theorem also requires Assumptions 2.3 –2.5 . The condition that no nonzero directions are missed is essential: if a i T s = 0 a_i^{\mathsf{T}}s=0 a i T s = 0 with a i ≠ 0 a_i\ne0 a i = 0 , then the gradient secant is zero, whereas the Hessian of the Poisson component is nonzero. Moreover, two observed parallel derivatives do not prove that the component is globally representable in the form ϕ i ( a i T x ) \phi_i(a_i^{\mathsf{T}}x) ϕ i ( a i T x ) . Therefore, the recovered matrix can be used to choose the heuristic proposal of YETI: the checks and the main iterations of the method preserve its complexity bound. Replacing the full sum of secants by a sample of a fixed number of components would require a separate theorem on spectral approximation.
Limiting the number of oracle queries made by the heuristic
Let B 0 = n ⌈ log N ⌉ B_0=n\lceil\log N\rceil B 0 = n ⌈ log N ⌉ , let G G G be the total number of oracle queries in the main iterations, and let H H H be the total number in the heuristic blocks. Choose ω ≥ 0 \omega\ge0 ω ≥ 0 . Before the next heuristic block, we allow
B k = n ⌊ min { B 0 , B 0 + ω G − H } n ⌋ (57) B_k=n\left\lfloor\frac{\min\{B_0,B_0+\omega G-H\}}{n}\right\rfloor \tag*{(57)} B k = n ⌊ n min { B 0 , B 0 + ω G − H } ⌋ (57)
queries. If B k < n B_k<n B k < n , the block is skipped. Before each evaluation of the function and the full gradient, n n n queries are reserved. All oracle calls made by the heuristic, including checks of the proposed points, are counted in H H H .
The proof of the following theorem reduces to a step-by-step verification of an invariant and is given in Appendix § 11 .§ 7 .
Theorem F.3 (number of heuristic queries). ▼ proof in Appendix § 11 .§ 7
If k k k heuristic blocks have been started by some moment of the run, then at this moment
H ≤ min { k B 0 , B 0 + ω G } . (58) H \le\min\{kB_0, B_0 + \omega G\}. \tag*{(58)} H ≤ min { k B 0 , B 0 + ω G } . (58) Rule (F.4) preserves the upper bound of Theorem 5.1 . Moreover,
E T ≤ B 0 + ( 1 + ω ) E G . (59) \mathbb{E}T \le B_0 + (1+\omega)\mathbb{E}G. \tag*{(59)} E T ≤ B 0 + ( 1 + ω ) E G . (59) Here all final evaluations of the function and the full gradient are included in G G G or H H H .
For ω = 1 \omega= 1 ω = 1 , beyond the first B 0 B_0 B 0 queries, the heuristic cannot make more oracle calls than the main iterations have already made. The first block has all B 0 B_0 B 0 queries available, so for the same values of the random variables it makes the same queries as a block with the constant limit B 0 B_0 B 0 . This does not require any assumption on the usefulness of the heuristic. Limiting the number of queries is essential: repeating the same point without stopping and without passing to a main iteration can make the number of oracle calls infinite, even if the function value does not increase.
Detailed description of YETI
History-based stopping criterion
Evaluating the function and the full gradient at a point p ∈ R d p \in\mathbb{R}^{d} p ∈ R d yields the affine minorant u ↦ f ( p ) + ⟨ ∇ f ( p ) , u − p ⟩ u \mapsto f(p) + \langle\nabla f(p),u-p\rangle u ↦ f ( p ) + ⟨ ∇ f ( p ) , u − p ⟩ . Its minimum over u ∈ B ( 0 , R ) u \in B(0,R) u ∈ B ( 0 , R ) equals
f ~ ( p ) = f ( p ) − ⟨ ∇ f ( p ) , p ⟩ − R ∥ ∇ f ( p ) ∥ . (60) \widetilde{f}(p)=f(p)-\langle\nabla f(p),p\rangle-R\lVert\nabla f(p)\rVert. \tag*{(60)} f ( p ) = f ( p ) − ⟨ ∇ f ( p ) , p ⟩ − R ∥ ∇ f ( p )∥ . (60)
This is a lower bound on the optimal value:
f ∗ ≥ f ( p ) + ⟨ ∇ f ( p ) , x ∗ − p ⟩ ≥ f ( p ) − ⟨ ∇ f ( p ) , p ⟩ − R ∥ ∇ f ( p ) ∥ . f^{*}\geq f(p)+\langle\nabla f(p),x^{*}-p\rangle\geq f(p)-\langle\nabla f(p),p\rangle-R\lVert\nabla f(p)\rVert. f ∗ ≥ f ( p ) + ⟨ ∇ f ( p ) , x ∗ − p ⟩ ≥ f ( p ) − ⟨ ∇ f ( p ) , p ⟩ − R ∥ ∇ f ( p )∥ .
The second step also uses the Cauchy–Schwarz inequality. If p ∈ B ( 0 , R ) p\in B(0,R) p ∈ B ( 0 , R ) , then the difference f ( p ) − f ~ ( p ) f(p)-\widetilde{f}(p) f ( p ) − f ( p ) is the Frank–Wolfe gap [15 ] ; the bound f ~ ( p ) ≤ f ∗ \widetilde{f}(p)\leq f^{*} f ( p ) ≤ f ∗ also holds for p ∉ B ( 0 , R ) p\notin B(0,R) p ∈ / B ( 0 , R ) . Denote by f ~ b e s t \widetilde{f}_{\mathrm{best}} f best the maximum of all stored lower bounds, and by w b e s t w_{\mathrm{best}} w best the best of the points of the ball B ( 0 , R ) B(0,R) B ( 0 , R ) at which the function value is known. The method stops and returns w b e s t w_{\mathrm{best}} w best if
f ( w b e s t ) − f ~ b e s t ≤ ε / 2. (61) f(w_{\mathrm{best}})-\widetilde{f}_{\mathrm{best}}\leq\varepsilon/2. \tag*{(61)} f ( w best ) − f best ≤ ε /2. (61)
The point w b e s t w_{\mathrm{best}} w best is stored separately from x x x : under the accelerated update of x x x , the function value does not necessarily decrease monotonically. All evaluations of the function and the full gradient are performed by the subroutine EVALUATE (Algorithm 2): at the points proposed by the heuristic, at the centers, and at the anchor and test points of the solver. If the values and gradients of all components at a given point are already known, repeated queries are not needed. To check (G.2), it suffices to store f ~ b e s t \widetilde{f}_{\mathrm{best}} f best , w b e s t w_{\mathrm{best}} w best , and f ( w b e s t ) f(w_{\mathrm{best}}) f ( w best ) ; the heuristic may also store additional data for choosing points.
Algorithm 2 EVALUATE( p ) (p) ( p ) : evaluation of the function and gradient, and accuracy check
Input: point p p p
query ( f i ( p ) , ∇ f i ( p ) ) (f_i(p),\nabla f_i(p)) ( f i ( p ) , ∇ f i ( p )) for all i i i whose answers at p p p have not yet been obtained
f ~ b e s t ← max { f ~ b e s t , f ( p ) − ⟨ ∇ f ( p ) , p ⟩ − R ∥ ∇ f ( p ) ∥ } \widetilde{f}_{\mathrm{best}}\leftarrow\max\{\widetilde{f}_{\mathrm{best}},f(p)-\langle\nabla f(p),p\rangle-R\lVert\nabla f(p)\rVert\} f best ← max { f best , f ( p ) − ⟨ ∇ f ( p ) , p ⟩ − R ∥ ∇ f ( p )∥}
if ∥ p ∥ ≤ R \lVert p\rVert\leq R ∥ p ∥ ≤ R and f ( p ) < f ( w b e s t ) f(p)<f(w_{\mathrm{best}}) f ( p ) < f ( w best ) then ▹ f ( w b e s t ) = + ∞ \triangleright f(w_{\mathrm{best}})=+\infty ▹ f ( w best ) = + ∞ until the first such point
w b e s t ← p w_{\mathrm{best}}\leftarrow p w best ← p
end if
if f ( w b e s t ) − f ~ b e s t ≤ ε / 2 f(w_{\mathrm{best}})-\widetilde{f}_{\mathrm{best}}\leq\varepsilon/2 f ( w best ) − f best ≤ ε /2 then
terminate YETI with output w b e s t w_{\mathrm{best}} w best
end if
Lemma G.1 (stopping at a small subgradient). ▼ proof in Appendix § 11 .§ 1
Let z ∈ B ( 0 , R ) z\in B(0,R) z ∈ B ( 0 , R ) , let the function and the full gradient at z z z be already computed, and let g = ∇ f ( z ) + v z g=\nabla f(z)+v_z g = ∇ f ( z ) + v z , where v z ∈ N B ( 0 , R ) ( z ) v_z\in N_{B(0,R)}(z) v z ∈ N B ( 0 , R ) ( z ) . Then
f ( z ) − f ~ ( z ) ≤ 2 R ∥ g ∥ . f(z)-\widetilde{f}(z)\leq2R\lVert g\rVert. f ( z ) − f ( z ) ≤ 2 R ∥ g ∥ . In particular, if ∥ g ∥ ≤ ε / ( 4 R ) \lVert g\rVert\leq\varepsilon/(4R) ∥ g ∥ ≤ ε / ( 4 R ) , then after z z z is recorded in the history, (G.2) holds.
The stopping test can also use component answers obtained at different points. The corresponding lower bound and the way to update it without new queries are given in Appendix § 11 .§ 2 . This option is used by the heuristic in the experiments, but it is not needed for the complexity bound of YETI.
Estimating the local curvature
After the function and the full gradient at y y y are evaluated, the quantity
ℓ ( y ) = 1 n ∑ i = 1 n ( L 0 , i + L 1 ∥ ∇ f i ( y ) ∥ ) (62) \ell(y)=\frac{1}{n}\sum_{i=1}^{n}\left(L_{0,i}+L_{1}\lVert\nabla f_i(y)\rVert\right) \tag*{(62)} ℓ ( y ) = n 1 i = 1 ∑ n ( L 0 , i + L 1 ∥ ∇ f i ( y )∥ ) (62)
is known.
Lemma G.2 (curvature estimate via the optimality gap). ▼ proof in Appendix § 10 .§ 3 Under Assumptions 2.1–2.3 and 2.5, for every point y y y ,
∥ ∇ f ( y ) ∥ ≤ L 1 ( δ + f ( y ) − f ∗ ) + 2 L 0 ( δ + f ( y ) − f ∗ ) , \lVert\nabla f(y)\rVert\le L_1(\delta+f(y)-f^*)+\sqrt{2L_0(\delta+f(y)-f^*)}, ∥ ∇ f ( y )∥ ≤ L 1 ( δ + f ( y ) − f ∗ ) + 2 L 0 ( δ + f ( y ) − f ∗ ) , L 1 ∥ ∇ f ( y ) ∥ ≤ ℓ ( y ) , L_1\lVert\nabla f(y)\rVert\le\ell(y), L 1 ∥ ∇ f ( y )∥ ≤ ℓ ( y ) , ℓ ( y ) ≤ 3 2 L 0 + 2 L 1 2 ( δ + f ( y ) − f ∗ ) . (63) \ell(y)\le\frac{3}{2}L_0+2L_1^2(\delta+f(y)-f^*). \tag*{(63)} ℓ ( y ) ≤ 2 3 L 0 + 2 L 1 2 ( δ + f ( y ) − f ∗ ) . (63) Moreover, for all x , y x,y x , y ,
ℓ ( x ) ≤ e L 1 ∥ x − y ∥ ℓ ( y ) , \ell(x)\le e^{L_1\lVert x-y\rVert}\ell(y), ℓ ( x ) ≤ e L 1 ∥ x − y ∥ ℓ ( y ) , ∥ ∇ 2 f ( x ) ∥ ≤ ℓ ( x ) . (64) \lVert\nabla^2 f(x)\rVert\le\ell(x). \tag*{(64)} ∥ ∇ 2 f ( x )∥ ≤ ℓ ( x ) . (64) In particular, for every r > 0 r>0 r > 0 , the norm of the Hessian of the component f i f_i f i on B ( y , 3 r ) B(y,3r) B ( y , 3 r ) does not exceed
e 3 L 1 r ( L 0 , i + L 1 ∥ ∇ f i ( y ) ∥ ) . (65) e^{3L_1r}\left(L_{0,i}+L_1\lVert\nabla f_i(y)\rVert\right). \tag*{(65)} e 3 L 1 r ( L 0 , i + L 1 ∥ ∇ f i ( y )∥ ) . (65)
Corollary G.3 (lower bound on the optimality gap via curvature). Evaluating the function and the full gradient at y y y yields the lower bound on the optimality gap
f ( y ) − f ∗ ≥ ℓ ( y ) − 3 2 L 0 − 2 L 1 2 δ 2 L 1 2 . f(y)-f^*\ge\frac{\ell(y)-\frac{3}{2}L_0-2L_1^2\delta}{2L_1^2}. f ( y ) − f ∗ ≥ 2 L 1 2 ℓ ( y ) − 2 3 L 0 − 2 L 1 2 δ . If the right-hand side is negative, the bound is trivial.
Parameters of YETI
The state of the method consists of points x , v ∈ B ( 0 , R ) x,v\in B(0,R) x , v ∈ B ( 0 , R ) , a number A > 0 A>0 A > 0 , and a parameter h > 0 h>0 h > 0 . The number A A A plays the same role as the sum of the coefficients A k A_k A k in accelerated methods with estimate sequences: we prove below that always f ( x ) − f ∗ ≤ R 2 / A f(x)-f^*\le R^2/A f ( x ) − f ∗ ≤ R 2 / A , and that A A A does not decrease and increases under the update (G.11). Unlike the sum A k A_k A k introduced in the notation, the initial value of A A A is positive. The radius r r r is chosen so that on the small ball the curvature changes by at most a constant factor. The proximal parameter is bounded from below for three reasons, related to the weight of the accelerated step, the conditioning of the local subproblem, and stopping. We use the following values:
r = min { R , 1 2 L 1 } , r=\min\left\{R,\frac{1}{2L_1}\right\}, r = min { R , 2 L 1 1 } ,
λ ↓ ( A ) = max { 1 2 ( r / R ) 2 / 3 1 4 ( r / R ) 4 / 3 A , 1 n ( 3 2 L 0 + 2 L 1 2 ( δ + R 2 A ) ) , ε 8 r R } , (66) \lambda_{\downarrow}(A)=\max\left\{\frac{\frac{1}{2}(r/R)^{2/3}}{\frac{1}{4}(r/R)^{4/3}A},\frac{1}{n}\left(\frac{3}{2}L_0+2L_1^2\left(\delta+\frac{R^2}{A}\right)\right),\frac{\varepsilon}{8rR}\right\}, \tag*{(66)} λ ↓ ( A ) = max { 4 1 ( r / R ) 4/3 A 2 1 ( r / R ) 2/3 , n 1 ( 2 3 L 0 + 2 L 1 2 ( δ + A R 2 ) ) , 8 r R ε } , (66)
λ ↑ = λ ↓ ( R 2 2 Δ ) . \lambda_{\uparrow}=\lambda_{\downarrow}\left(\frac{R^2}{2\Delta}\right). λ ↑ = λ ↓ ( 2Δ R 2 ) .
Here r r r is the radius of the small ball, λ ↓ ( A ) \lambda_{\downarrow}(A) λ ↓ ( A ) is the lower limit of the proximal parameter, and λ ↑ \lambda_{\uparrow} λ ↑ is the upper one. The three expressions under the maximum correspond to these reasons: the first prevents the weight α \alpha α of the accelerated step from exceeding 1 2 ( r / R ) 2 / 3 \frac{1}{2}(r/R)^{2/3} 2 1 ( r / R ) 2/3 , the second makes the local subproblem well conditioned, and the third is needed for stopping.
The method starts with x = v = 0 x=v=0 x = v = 0 , A = R 2 / ( 2 Δ ) A=R^2/(2\Delta) A = R 2 / ( 2Δ ) , h = λ ↑ h=\lambda_{\uparrow} h = λ ↑ , and an empty query history.
Main iteration of YETI
In the state ( x , v , A , h ) (x, v, \mathcal{A}, h) ( x , v , A , h ) , we compute
λ = min { max { h , λ ↓ ( A ) } , λ ↑ } , \lambda= \min\{\max\{h, \lambda_{\downarrow}(\mathcal{A})\}, \lambda_{\uparrow}\}, λ = min { max { h , λ ↓ ( A )} , λ ↑ } ,
a λ = 1 + 1 + 4 λ A 2 λ , a_{\lambda} = \frac{1+\sqrt{1+4\lambda\mathcal{A}}}{2\lambda}, a λ = 2 λ 1 + 1 + 4 λ A ,
α = a λ A + a λ , \alpha= \frac{a_{\lambda}}{\mathcal{A}+a_{\lambda}}, α = A + a λ a λ ,
y = ( 1 − α ) x + α v . (67) y = (1-\alpha)x+\alpha v. \tag*{(67)} y = ( 1 − α ) x + α v . (67)
The number a λ a_{\lambda} a λ is the positive root of the equation λ a λ 2 = A + a λ \lambda a_{\lambda}^{2}=\mathcal{A}+a_{\lambda} λ a λ 2 = A + a λ ; therefore,
λ A α 2 = 1 − α , \lambda\mathcal{A}\alpha^{2}=1-\alpha, λ A α 2 = 1 − α ,
λ a λ = 1 α . \lambda a_{\lambda}=\frac{1}{\alpha}. λ a λ = α 1 .
Using the values of the component gradients at y y y , we check the curvature condition
ℓ ( y ) ≤ e L 1 r ( 3 2 L 0 + 2 L 1 2 ( δ + R 2 A ) ) . (68) \ell(y)\le e^{L_{1}r}\left(\frac{3}{2}L_{0}+2L_{1}^{2}\left(\delta+\frac{R^{2}}{\mathcal{A}}\right)\right). \tag*{(68)} ℓ ( y ) ≤ e L 1 r ( 2 3 L 0 + 2 L 1 2 ( δ + A R 2 ) ) . (68)
If this condition is violated, then by Corollary G.3 the optimality gap at y y y is large, and the halfspace
H y = { w : ⟨ ∇ f ( y ) , w − y ⟩ ≤ − ℓ ( y ) − 3 2 L 0 − 2 L 1 2 δ 2 L 1 2 } (69) \mathcal{H}_{y}=\left\{w:\langle\nabla f(y),w-y\rangle\le-\frac{\ell(y)-\frac{3}{2}L_{0}-2L_{1}^{2}\delta}{2L_{1}^{2}}\right\} \tag*{(69)} H y = { w : ⟨ ∇ f ( y ) , w − y ⟩ ≤ − 2 L 1 2 ℓ ( y ) − 2 3 L 0 − 2 L 1 2 δ } (69)
contains the minimizer. In this case, we perform the update
v + = Π B ( 0 , R ) Π H y v , v^{+}=\Pi_{B(0,R)}\Pi_{\mathcal{H}_{y}}v, v + = Π B ( 0 , R ) Π H y v ,
h + = 2 λ , h^{+}=2\lambda, h + = 2 λ ,
under which x x x and A \mathcal{A} A do not change. Such a step is called an up-step.
If condition (G.9) holds, we run the local solver (Algorithm 5), which uses the already obtained values and gradients of the components at y y y . It returns a point z z z , a subgradient g g g of the problem on the ball B ( 0 , R ) B(0,R) B ( 0 , R ) at z z z , and a number β ≥ 0 \beta\ge0 β ≥ 0 . If g g g is small, the method returns z z z . Otherwise, setting γ = λ / ( λ + β ) \gamma=\lambda/(\lambda+\beta) γ = λ / ( λ + β ) , we reduce the weight of the accelerated update by multiplying it by γ \gamma γ , following the scheme of Carmon et al. [9 ] :
A + = A + γ a λ , \mathcal{A}^{+}=\mathcal{A}+\gamma a_{\lambda}, A + = A + γ a λ ,
x + = ( 1 − γ ) A x + γ ( A + a λ ) z A + , x^{+}=\frac{(1-\gamma)\mathcal{A}x+\gamma(\mathcal{A}+a_{\lambda})z}{\mathcal{A}^{+}}, x + = A + ( 1 − γ ) A x + γ ( A + a λ ) z ,
v + = Π B ( 0 , R ) ( v − γ a λ g ) , v^{+}=\Pi_{B(0,R)}(v-\gamma a_{\lambda}g), v + = Π B ( 0 , R ) ( v − γ a λ g ) ,
h + = { 2 λ , β > 0 , λ / 2 , β = 0. (70) h^{+}=
\begin{cases}
2\lambda, & \beta>0,\\
\lambda/2, & \beta=0.
\end{cases} \tag*{(70)} h + = { 2 λ , λ /2 , β > 0 , β = 0. (70)
The step with β > 0 \beta>0 β > 0 is also called an up-step, and the step with β = 0 \beta=0 β = 0 a down-step. If A + ≥ 2 R 2 / ε \mathcal{A}^{+}\ge2R^{2}/\varepsilon A + ≥ 2 R 2 / ε , the method returns x + x^{+} x + .
Heuristic block
The heuristic uses the known parameters of the problem, the history of oracle answers, and its own random variables. Each block returns control after finitely many computational steps. The limit n ⌈ log N ⌉ n\lceil\log N\rceil n ⌈ log N ⌉ applies to all queries, including queries at rejected points and the initial evaluation of the function and the full gradient at x x x . Before the heuristic is called, n n n queries are reserved for checking the proposal.
A point z z z is accepted if z ∈ B ( 0 , R ) z\in B(0,R) z ∈ B ( 0 , R ) and f ( z ) ≤ f ( x ) f(z)\le f(x) f ( z ) ≤ f ( x ) ; in this case only x x x changes. Stopping is possible by criterion (G.2). The decisions made by the heuristic are measurable. The indices of the local solver use fresh independent random variables; given the history, the indices have the distribution from § 5 .
Algorithm 3 BLOCK(x x x ): heuristic block
Input: point x ∈ B ( 0 , R ) x \in B(0,R) x ∈ B ( 0 , R ) , heuristic
b = n ⌈ log N ⌉ b = n\lceil\log N\rceil b = n ⌈ log N ⌉ ▷ each query decreases b b b by one
EVALUATE(x x x )
while b ≥ n b \ge n b ≥ n and the heuristic has not terminated the block do
the heuristic makes at most b − n b-n b − n queries and proposes z ∈ B ( 0 , R ) z \in B(0,R) z ∈ B ( 0 , R )
EVALUATE(z z z )
if f ( z ) ≤ f ( x ) f(z) \le f(x) f ( z ) ≤ f ( x ) then
x ← z x \gets z x ← z
end if
end while
return x x x
Local solution by SVRG
Local subproblem
Let y ∈ B ( 0 , R ) y \in B(0,R) y ∈ B ( 0 , R ) , λ > 0 \lambda> 0 λ > 0 , and r > 0 r > 0 r > 0 . Consider the functions
P ( z ) = f ( z ) + λ 2 ∥ z − y ∥ 2 , P i ( z ) = f i ( z ) + λ 2 ∥ z − y ∥ 2 , (71) \begin{aligned}
P(z) &= f(z) + \frac{\lambda}{2}\lVert z-y\rVert^2,\\
P_i(z) &= f_i(z) + \frac{\lambda}{2}\lVert z-y\rVert^2, \tag*{(71)}
\end{aligned} P ( z ) P i ( z ) = f ( z ) + 2 λ ∥ z − y ∥ 2 , = f i ( z ) + 2 λ ∥ z − y ∥ 2 , (71)
the set K = B ( 0 , R ) ∩ B ( y , r ) \mathcal{K} = B(0,R) \cap B(y,r) K = B ( 0 , R ) ∩ B ( y , r ) , and the point
u = arg min z ∈ K P ( z ) . u = \underset{z\in\mathcal{K}}{\arg\min}\,P(z). u = z ∈ K arg min P ( z ) .
The set K \mathcal{K} K is compact and convex, and P P P is strongly convex with parameter λ \lambda λ ; hence u u u exists and is unique. Evaluating the function and the full gradient at y y y allows us to find the numbers
L i = e 3 L 1 r ( L 0 , i + L 1 ∥ ∇ f i ( y ) ∥ ) + λ (72) L_i = e^{3L_1r}\left(L_{0,i}+L_1\lVert\nabla f_i(y)\rVert\right)+\lambda \tag*{(72)} L i = e 3 L 1 r ( L 0 , i + L 1 ∥ ∇ f i ( y )∥ ) + λ (72)
and their average
L ‾ = 1 n ∑ i L i = e 3 L 1 r ℓ ( y ) + λ . \overline{L} = \frac{1}{n}\sum_i L_i = e^{3L_1r}\ell(y)+\lambda. L = n 1 i ∑ L i = e 3 L 1 r ℓ ( y ) + λ .
By Lemma G.2 , the norm of the Hessian of P i P_i P i on B ( y , 3 r ) B(y,3r) B ( y , 3 r ) does not exceed L i L_i L i .
The following lemma describes the projection onto K \mathcal{K} K and the local cocoercivity of the components; its proof is given in Appendix § 11 .§ 3 .
Lemma H.1 (projection onto the intersection of balls). ▼ proof in Appendix § 11 .§ 3
If z = Π K ( b ) z=\Pi_{\mathcal{K}}(b) z = Π K ( b ) , then there exist ξ , ζ ≥ 0 \xi,\zeta\ge0 ξ , ζ ≥ 0 such that
b − z = ξ z + ζ ( z − y ) , ζ > 0 ⟹ ∥ z ∥ = R , ζ > 0 ⟹ ∥ z − y ∥ = r . (73) \begin{aligned}
b-z &= \xi z+\zeta(z-y),\\
\zeta>0 &\Longrightarrow\lVert z\rVert=R,\\
\zeta>0 &\Longrightarrow\lVert z-y\rVert=r. \tag*{(73)}
\end{aligned} b − z ζ > 0 ζ > 0 = ξ z + ζ ( z − y ) , ⟹ ∥ z ∥ = R , ⟹ ∥ z − y ∥ = r . (73) For x , t ∈ B ( y , r ) x,t\in B(y,r) x , t ∈ B ( y , r ) ,
∥ ∇ P i ( x ) − ∇ P i ( t ) ∥ 2 ≤ 2 L i ( P i ( x ) − P i ( t ) − ⟨ ∇ P i ( t ) , x − t ⟩ ) . (74) \lVert\nabla P_i(x)-\nabla P_i(t)\rVert^2
\le2L_i\left(P_i(x)-P_i(t)-\langle\nabla P_i(t),x-t\rangle\right). \tag*{(74)} ∥ ∇ P i ( x ) − ∇ P i ( t ) ∥ 2 ≤ 2 L i ( P i ( x ) − P i ( t ) − ⟨ ∇ P i ( t ) , x − t ⟩ ) . (74) In particular, the gradient of P P P on B ( y , r ) B(y,r) B ( y , r ) is 1 / L ‾ 1/\overline{L} 1/ L -cocoercive.
SVRG epoch
Within a single call of the solver, the quantities λ \lambda λ , y y y , and L i L_i L i do not change. The index i i i is chosen with probability L i / ( n L ‾ ) L_i/(n\overline{L}) L i / ( n L ) . The step size and the epoch length are
h s = 1 6 L ‾ , M = ⌈ 2 n L ‾ 5 λ ⌉ . (75) h_s = \frac{1}{6\overline{L}}, \qquad
M = \left\lceil\frac{2n\overline{L}}{5\lambda} \right\rceil. \tag*{(75)} h s = 6 L 1 , M = ⌈ 5 λ 2 n L ⌉ . (75)
Since L i ≥ λ > 0 L_i \ge\lambda> 0 L i ≥ λ > 0 , all probabilities L i / ( n L ‾ ) L_i/(n\overline{L}) L i / ( n L ) are positive and sum to one. At the anchor point s ∈ K s \in\mathcal{K} s ∈ K , the values and gradients of all components are known. From the current point x j x_j x j , a step is made with a new index i j i_j i j , conditionally independent of the previous ones:
v i j = ∇ P ( s ) + ∇ P i j ( x j ) − ∇ P i j ( s ) L i j / L ‾ , v_{i_j} = \nabla P(s) + \frac{\nabla P_{i_j}(x_j) - \nabla P_{i_j}(s)}{L_{i_j}/\overline{L}}, v i j = ∇ P ( s ) + L i j / L ∇ P i j ( x j ) − ∇ P i j ( s ) ,
x j + 1 = Π K ( x j − h s v i j ) . (76) x_{j+1} = \Pi_{\mathcal{K}}(x_j - h_s v_{i_j}). \tag*{(76)} x j + 1 = Π K ( x j − h s v i j ) . (76)
The next epoch starts at the point w + = x M w^+ = x_M w + = x M , and the new anchor point is
s + = ∑ j = 0 M − 1 ( 1 − h s λ ) M − 1 − j x j ∑ j = 0 M − 1 ( 1 − h s λ ) j . (77) s^+ =
\frac{\sum_{j=0}^{M-1}(1-h_s\lambda)^{M-1-j}x_j}
{\sum_{j=0}^{M-1}(1-h_s\lambda)^j}. \tag*{(77)} s + = ∑ j = 0 M − 1 ( 1 − h s λ ) j ∑ j = 0 M − 1 ( 1 − h s λ ) M − 1 − j x j . (77)
The first epoch of a call starts with w = s = y w=s=y w = s = y . The component gradients at the anchor point are stored, so one inner step requires one query.
Algorithm 4 EPOCH( w , s ) (w,s) ( w , s ) : SVRG epoch for the local subproblem
Input: points w , s ∈ K w,s \in\mathcal{K} w , s ∈ K , answers of all components at s s s , P i P_i P i , K \mathcal{K} K , L i L_i L i , L ‾ \overline{L} L , h s h_s h s , M M M from Algorithm 5
x 0 = w x_0 = w x 0 = w
for j = 0 , … , M − 1 j = 0,\ldots,M-1 j = 0 , … , M − 1 do
choose the index i j = i i_j=i i j = i with probability L i / ( n L ‾ ) L_i/(n\overline{L}) L i / ( n L ) independently of the past
query ∇ f i j ( x j ) \nabla f_{i_j}(x_j) ∇ f i j ( x j )
v j = ∇ P ( s ) + L ‾ L i j ( ∇ P i j ( x j ) − ∇ P i j ( s ) ) v_j = \nabla P(s) + \dfrac{\overline{L}}{L_{i_j}}\left(\nabla P_{i_j}(x_j)-\nabla P_{i_j}(s)\right) v j = ∇ P ( s ) + L i j L ( ∇ P i j ( x j ) − ∇ P i j ( s ) )
x j + 1 = Π K ( x j − h s v j ) x_{j+1} = \Pi_{\mathcal{K}}(x_j-h_sv_j) x j + 1 = Π K ( x j − h s v j )
end for
return ( x M , ∑ j = 0 M − 1 ( 1 − h s λ ) M − 1 − j x j ∑ j = 0 M − 1 ( 1 − h s λ ) j ) \left(x_M,\dfrac{\sum_{j=0}^{M-1}(1-h_s\lambda)^{M-1-j}x_j}{\sum_{j=0}^{M-1}(1-h_s\lambda)^j}\right) ( x M , ∑ j = 0 M − 1 ( 1 − h s λ ) j ∑ j = 0 M − 1 ( 1 − h s λ ) M − 1 − j x j )
Stopping the local solver
After an epoch, the full gradient at w w w is computed. First, the tuple
z ~ = w , β = 0 , g = ∇ f ( w ) , e = ∇ P ( w ) \tilde{z}=w,\qquad
\beta=0,\qquad
g=\nabla f(w),\qquad
e=\nabla P(w) z ~ = w , β = 0 , g = ∇ f ( w ) , e = ∇ P ( w )
is checked. If it is not accepted, we compute the point and the vectors
z = Π K ( w − ∇ P ( w ) L ‾ ) , z=\Pi_{\mathcal{K}}\left(w-\frac{\nabla P(w)}{\overline{L}}\right), z = Π K ( w − L ∇ P ( w ) ) ,
e = ∇ P ( z ) − ∇ P ( w ) + L ‾ ( w − z ) , e=\nabla P(z)-\nabla P(w)+\overline{L}(w-z), e = ∇ P ( z ) − ∇ P ( w ) + L ( w − z ) ,
g = ∇ f ( z ) + L ‾ ξ z , (78) g=\nabla f(z)+\overline{L}\xi_z, \tag*{(78)} g = ∇ f ( z ) + L ξ z , (78)
and the number
β = L ‾ ζ , \beta=\overline{L}\zeta, β = L ζ ,
where ξ , ζ \xi,\zeta ξ , ζ are the coefficients from (H.3) for precisely the projection in the first line. Computing e e e and g g g requires the full gradient at z z z . In both cases,
g ∈ ∂ ( f + I B ( 0 , R ) ) ( z ) , g \in\partial(f+I_{B(0,R)})(z), g ∈ ∂ ( f + I B ( 0 , R ) ) ( z ) ,
e = g − ( λ + β ) ( y − z ) , e = g-(\lambda+\beta)(y-z), e = g − ( λ + β ) ( y − z ) ,
β > 0 ⟹ ∥ y − z ∥ = r , (79) \beta>0 \Longrightarrow\lVert y-z\rVert=r, \tag*{(79)} β > 0 ⟹ ∥ y − z ∥ = r , (79)
where I B ( 0 , R ) I_{B(0,R)} I B ( 0 , R ) is the indicator function of the convex set; its subdifferential is the normal cone. The tuple is accepted if at least one of the conditions
∥ g ∥ ≤ ε 4 R , \lVert g\rVert\le\frac{\varepsilon}{4R}, ∥ g ∥ ≤ 4 R ε ,
∥ e ∥ ≤ 4 5 ( λ + β ) ∥ y − z ∥ (80) \lVert e\rVert\le\frac{4}{5}(\lambda+\beta)\lVert y-z\rVert \tag*{(80)} ∥ e ∥ ≤ 5 4 ( λ + β ) ∥ y − z ∥ (80)
holds. The first condition is checked before the second.
The complete solver is given in Algorithm 5. If no tuple is accepted after an epoch, the function and the full gradient are evaluated at the new anchor point, and the next epoch begins.
Algorithm 5 SOLVER( y , λ ) (y,\lambda) ( y , λ ) : approximate solution of the local subproblem
Input: center y ∈ B ( 0 , R ) y \in B(0,R) y ∈ B ( 0 , R ) at which the answers of all components are known, parameter λ > 0 \lambda>0 λ > 0
P i ( z ) = f i ( z ) + λ 2 ∥ z − y ∥ 2 P_i(z)=f_i(z)+\frac{\lambda}{2}\lVert z-y\rVert^2 P i ( z ) = f i ( z ) + 2 λ ∥ z − y ∥ 2 , i = 1 , … , n i=1,\ldots,n i = 1 , … , n
P ( z ) = f ( z ) + λ 2 ∥ z − y ∥ 2 P(z)=f(z)+\frac{\lambda}{2}\lVert z-y\rVert^2 P ( z ) = f ( z ) + 2 λ ∥ z − y ∥ 2
K = B ( 0 , R ) ∩ B ( y , r ) \mathcal{K}=B(0,R)\cap B(y,r) K = B ( 0 , R ) ∩ B ( y , r )
L i = e 3 L 1 r ( L 0 , i + L 1 ∥ ∇ f i ( y ) ∥ ) + λ L_i=e^{3L_1r}(L_{0,i}+L_1\lVert\nabla f_i(y)\rVert)+\lambda L i = e 3 L 1 r ( L 0 , i + L 1 ∥ ∇ f i ( y )∥) + λ , i = 1 , … , n i=1,\ldots,n i = 1 , … , n
L ‾ = 1 n ∑ i L i \overline{L}=\frac{1}{n}\sum_i L_i L = n 1 ∑ i L i
h s = 1 / ( 6 L ‾ ) h_s=1/(6\overline{L}) h s = 1/ ( 6 L )
M = ⌈ 21 L ‾ / ( 5 λ ) ⌉ M=\left\lceil21\overline{L}/(5\lambda)\right\rceil M = ⌈ 21 L / ( 5 λ ) ⌉
a tuple ( z , g , β ) (z,g,\beta) ( z , g , β ) is accepted iff ∥ g ∥ ≤ ε 4 R \lVert g\rVert\le\frac{\varepsilon}{4R} ∥ g ∥ ≤ 4 R ε or ∥ g − ( λ + β ) ( y − z ) ∥ ≤ 4 5 ( λ + β ) ∥ y − z ∥ \lVert g-(\lambda+\beta)(y-z)\rVert\le\frac{4}{5}(\lambda+\beta)\lVert y-z\rVert ∥ g − ( λ + β ) ( y − z )∥ ≤ 5 4 ( λ + β ) ∥ y − z ∥
w = s = y w=s=y w = s = y
loop
( w , s ) ← EPOCH ( w , s ) (w,s)\leftarrow\operatorname{EPOCH}(w,s) ( w , s ) ← EPOCH ( w , s )
EVALUATE ( w ) \operatorname{EVALUATE}(w) EVALUATE ( w )
if the tuple ( w , ∇ f ( w ) , 0 ) (w,\nabla f(w),0) ( w , ∇ f ( w ) , 0 ) is accepted then
return ( w , ∇ f ( w ) , 0 ) (w,\nabla f(w),0) ( w , ∇ f ( w ) , 0 )
end if
z = Π K ( w − ∇ P ( w ) / L ‾ ) z=\Pi_{\mathcal{K}}(w-\nabla P(w)/\overline{L}) z = Π K ( w − ∇ P ( w ) / L )
ξ , ζ ≥ 0 \xi,\zeta\ge0 ξ , ζ ≥ 0 are the multipliers of the normals to the two balls at z z z : w − ∇ P ( w ) / L ‾ − z = ξ z + ζ ( z − y ) , w-\nabla P(w)/\overline{L}-z=\xi z+\zeta(z-y), w − ∇ P ( w ) / L − z = ξ z + ζ ( z − y ) , ξ > 0 ⟹ ∥ z ∥ = R , \xi>0\Longrightarrow\lVert z\rVert=R, ξ > 0 ⟹ ∥ z ∥ = R , ζ > 0 ⟹ ∥ z − y ∥ = r \zeta>0\Longrightarrow\lVert z-y\rVert=r ζ > 0 ⟹ ∥ z − y ∥ = r
EVALUATE ( z ) \operatorname{EVALUATE}(z) EVALUATE ( z )
if the tuple ( z , ∇ f ( z ) + L ‾ ξ z , L ‾ ζ ) (z,\nabla f(z)+\overline{L}\xi z,\overline{L}\zeta) ( z , ∇ f ( z ) + L ξ z , L ζ ) is accepted then
return ( z , ∇ f ( z ) + L ‾ ξ z , L ‾ ζ ) (z,\nabla f(z)+\overline{L}\xi z,\overline{L}\zeta) ( z , ∇ f ( z ) + L ξ z , L ζ )
end if
EVALUATE ( s ) \operatorname{EVALUATE}(s) EVALUATE ( s )
end loop
Lemma H.2 (number of epochs until the solver stops). ▼ proof in Appendix § 11 .§ 3 For the tuple (H.8), the bounds
∥ z − u ∥ ≤ ∥ w − u ∥ , \lVert z-u\rVert\le\lVert w-u\rVert, ∥ z − u ∥ ≤ ∥ w − u ∥ , ∥ e ∥ ≤ L ‾ ∥ w − u ∥ (81) \lVert e\rVert\le\overline{L}\lVert w-u\rVert \tag*{(81)} ∥ e ∥ ≤ L ∥ w − u ∥ (81) hold. Therefore, if ∥ w − u ∥ ≤ 4 λ ‾ 5 L ‾ + 4 λ ‾ ∥ y − u ∥ \lVert w-u\rVert\le\frac{4\overline{\lambda}}{5\overline{L}+4\overline{\lambda}}\lVert y-u\rVert ∥ w − u ∥ ≤ 5 L + 4 λ 4 λ ∥ y − u ∥ , the tuple is accepted. Let T e p T_{\mathrm{ep}} T ep be the number of epochs until the first accepted tuple. Then for j ≥ 1 j \ge1 j ≥ 1 ,
Pr ( T e p > j ∣ history of the call ) ≤ min { 1 , Γ 2 − j } , (82) \Pr(T_{\mathrm{ep}}>j \mid\text{history of the call}) \le\min\{1,\Gamma2^{-j}\}, \tag*{(82)} Pr ( T ep > j ∣ history of the call ) ≤ min { 1 , Γ 2 − j } , (82) where
Γ = ( 1 + 2 L ‾ 3 λ ‾ ) ( 1 + 5 L ‾ 4 λ ‾ ) 2 . \Gamma= \left(1+\frac{2\overline{L}}{3\overline{\lambda}}\right)\left(1+\frac{5\overline{L}}{4\overline{\lambda}}\right)^2. Γ = ( 1 + 3 λ 2 L ) ( 1 + 4 λ 5 L ) 2 . In particular, the solver terminates almost surely and makes a finite expected number of oracle calls.
Experimental details
Experimental setup
Problems. We use three groups of problems:
Chains S and X: n = 16 n = 16 n = 16 , T ∈ { 8 , 16 , 32 , 64 } T \in\{8, 16, 32, 64\} T ∈ { 8 , 16 , 32 , 64 } . These are numerical versions of the families from § 8 with tabulated integrals, without hidden coordinates and without radial extension. They allow us to compare methods as T T T increases; these experiments do not test the lower bound for arbitrary methods.
Poisson models f i ( x ) = w i e a i T x − y i a i T x f_i(x) = w_i e^{a_i^{\mathsf{T}}x} - y_i a_i^{\mathsf{T}}x f i ( x ) = w i e a i T x − y i a i T x , where w i > 0 w_i > 0 w i > 0 , y i ≥ 0 y_i \ge0 y i ≥ 0 , L 0 , i = y i ∥ a i ∥ 2 L_{0,i} = y_i \lVert a_i \rVert^2 L 0 , i = y i ∥ a i ∥ 2 , L 1 = max i ∥ a i ∥ L_1 = \max_i \lVert a_i \rVert L 1 = max i ∥ a i ∥ . In the synthetic problems, d = 50 d = 50 d = 50 , n ∈ { 2000 , 20000 , 100000 } n \in\{2000, 20000, 100000\} n ∈ { 2000 , 20000 , 100000 } , and the condition number of the feature covariance is 10 4 10^4 1 0 4 . On the Bike Sharing data, n = 17379 n = 17379 n = 17379 , d = 48 d = 48 d = 48 ; on freMTPL2freq, n = 678013 n = 678013 n = 678013 , d = 68 d = 68 d = 68 .
65 problems: 13 families with five random instances each. This series includes rotated, ill-conditioned (up to 10 8 10^8 1 0 8 ), nonlinear, and sparse Poisson models, high-dimensional problems, sums of quartic functions, and regularized logistic regression. The implementation settings were fixed before these experiments were run; the problems used during its development are not included in the series.
Metric. We count all queries of a method, including queries for constructing the initial metric, computing full gradients, stochastic corrections, and line search. One pass equals n n n queries. The minimum on the chains is known from the construction; on the other problems, f r e f f_{\mathrm{ref}} f ref is found numerically. The reference solution is also used to compute the bounds R R R , Δ \Delta Δ , δ \delta δ passed to YETI as problem parameters. Queries for computing the reference and for external diagnostics are not counted. The number of queries to reach the tolerance is determined by the first checked point satisfying the tolerance; on the 65 problems, we consider all points with a complete set of component answers, including trial points of the line search. The plots show the smallest optimality gap attained up to a given moment. The triggering of a method’s own stopping criterion is measured separately.
Methods. On the 65 problems, we compare YETI, L-BFGS with memory 50, Cov-LBFGS 1/4 and 1/2, BFGS, Nesterov’s method with line search and restart, GRAAL with stabilization of the curvature estimate, and SVRG [22 , 27 , 5 , 16 ] . Cov-LBFGS uses the coordinate transformation C − 1 / 4 C^{-1/4} C − 1/4 or C − 1 / 2 C^{-1/2} C − 1/2 , where C C C is the centered covariance of the initial component gradients; on the cut-off subspace, the pseudoinverse is zero. On the main test set, we additionally consider nonlinear CG, AdGD, clipped gradient descent, SAGA, and Katyusha [23 , 41 , 11 , 2 ] . For the variance reduction methods, the local smoothness estimates are updated using the available component gradients.
Run limits. On the chains, the baseline methods are limited to 20000 passes, on the Poisson models to 1000, and on the 65 problems to 3000 passes and 1800 seconds. The main tables report results for one instance of each problem configuration. In the extended series, all runs are counted, including failed ones; the family median is reported only if all five instances succeed.
Tested variant of YETI. For collinear component gradients, the heuristic uses the secants from Proposition F.1 ; in the remaining cases, it uses L-BFGS in the covariance metric with the power chosen from the observed secant. When the history is insufficient, standard L-BFGS is used. On large samples, the initial secant matrix is built from a subsample of up to 4096 rows. After 32 evaluations of the full gradient or after an unsuccessful block, a switch to the full history of component answers is possible. The dense covariance requires O ( n d 2 ) O(nd^2) O ( n d 2 ) operations, and its spectral decomposition O ( d 3 ) O(d^3) O ( d 3 ) . The implementation limits the fraction of the accelerated step and the number of inner calls, and changes the settings of the local solver. Theorem 2.9 is proved for Algorithm 1; these modifications require a separate analysis.
Results by family
Figure 3 compares the methods on all 65 problems, and Table 1 shows the medians over the five instances of each family. In particular, on the quartic problems with n < d n < d n < d , Cov-LBFGS does not reach the tolerance on all instances because directions are lost when inverting the degenerate covariance matrix.
Figure 3. Fraction of the 65 problems on which the number of queries does not exceed the best result by more than a factor of τ \tau τ (Dolan and Moré [12 ] ). At τ = 1 \tau= 1 τ = 1 , all first places are counted, including shared ones; failed runs remain in the denominator.
Table 1 (PDF p. 54). Median number of passes to relative accuracy 10 − 6 10^{-6} 1 0 − 6 over five instances. “—” indicates failure on at least one instance; the minimum before rounding is shown in bold. κ \kappa κ is the condition number of the feature covariance, and n × d n \times d n × d are the number of components and the dimension.
Results on the main test set are given in Table 2 and Table 4 . In additional runs of YETI on five synthetic Poisson instances, the median numbers of passes are 4, 5, and 5 for n = 2000 n = 2000 n = 2000 , 20000, and 100000; the ranges are 4–5, 5–7, and 5–8, respectively.
Stopping test
On the 65 problems with ε = 10 − 6 Δ ^ \varepsilon= 10^{-6}\widehat{\Delta} ε = 1 0 − 6 Δ , at least 90% of the runs terminate within 37 passes. In 63 of the 65 runs, YETI stops already in the first heuristic block. The main proximal step was needed only on two instances with rare features at n = 50000 n = 50000 n = 50000 : it was performed 4 and 8 times. On the main test set, all 45 runs (24 on the chains and 21 on regression) terminated before the first such step. Figure 4 shows two additional runs with a stricter tolerance.
Table 3 compares the standard heuristic, a block consisting of a single checking pass without proposals, and a block that proposes random points of the feasible ball. In the last variant, each block spends the entire allowed limit: 243–264 passes. The overall limit is 60000 passes in the mode without proposals and 200000 with random proposals.
Table 3. Number of passes for three variants of the heuristic. The method’s own stopping criterion uses ε = 10 − 3 Δ ^ \varepsilon= 10^{-3}\widehat{\Delta} ε = 1 0 − 3 Δ . “—” indicates that it was not satisfied within 200000 passes.
The tolerance 10 − 2 Δ ^ 10^{-2}\widehat{\Delta} 1 0 − 2 Δ is reached in all runs. Disabling the heuristic proposals increases the number of passes to the tolerance by a factor of about 41–58, and random proposals increase it by a further factor of about 19–54 relative to the mode without proposals. Thus, in these experiments, the main gain comes from the heuristic. However, its matrix operations can be expensive, so a gain in the number of queries does not necessarily imply the same gain in time. On X with T = 16 T = 16 T = 16 , the method’s own stopping criterion is not satisfied within the allotted number of passes. Limiting the block length does not eliminate the cost of useless proposals.
Summary of known bounds
Table 5. Finite-sum minimization: number of calls to the component oracle.
Notation. R = ∥ x 0 − x ∗ ∥ R=\lVert x_0-x^*\rVert R = ∥ x 0 − x ∗ ∥ , F 0 = f ( x 0 ) − f ∗ F_0=f(x_0)-f^* F 0 = f ( x 0 ) − f ∗ , log + = max { log , 0 } \log_+=\max\{\log,0\} log + = max { log , 0 } ; O ( ⋅ ) O(\cdot) O ( ⋅ ) and Ω ( ⋅ ) \Omega(\cdot) Ω ( ⋅ ) hide absolute constants, and Θ ~ ( ⋅ ) \widetilde{\Theta}(\cdot) Θ ( ⋅ ) also hides logarithmic factors. Column “Convex”: ✓ — the result is obtained for convex f i f_i f i , ✗ — convexity is not assumed.
Smoothness. L L L : the gradient of each f i f_i f i is Lipschitz with constant L L L . sym.: symmetric ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness [14 Assumption 1.3] , ∥ ∇ g ( x ) − ∇ g ( y ) ∥ ≤ ( L 0 + L 1 sup u ∈ [ x , y ] ∥ ∇ g ( u ) ∥ ) ∥ x − y ∥ \lVert\nabla g(x)-\nabla g(y)\rVert\leq(L_0+L_1\sup_{u\in[x,y]}\lVert\nabla g(u)\rVert)\lVert x-y\rVert ∥ ∇ g ( x ) − ∇ g ( y )∥ ≤ ( L 0 + L 1 sup u ∈ [ x , y ] ∥ ∇ g ( u )∥) ∥ x − y ∥ for all x , y x,y x , y ; for C 2 C^2 C 2 functions this is equivalent to ∥ ∇ 2 g ( x ) ∥ ≤ L 0 + L 1 ∥ ∇ g ( x ) ∥ \lVert\nabla^2g(x)\rVert\leq L_0+L_1\lVert\nabla g(x)\rVert ∥ ∇ 2 g ( x )∥ ≤ L 0 + L 1 ∥ ∇ g ( x )∥ . Avg.: averaged ( L 0 , L 1 ) (L_0,L_1) ( L 0 , L 1 ) -smoothness [29 Assumption 1(ii] , ( 1 n ∑ i = 1 n ∥ ∇ f i ( x ) − ∇ f i ( y ) ∥ 2 ) 1 / 2 ≤ ( L 0 + L 1 ∥ ∇ f ( x ) ∥ ) ∥ x − y ∥ \left(\frac{1}{n}\sum_{i=1}^{n}\lVert\nabla f_i(x)-\nabla f_i(y)\rVert^2\right)^{1/2}\leq(L_0+L_1\lVert\nabla f(x)\rVert)\lVert x-y\rVert ( n 1 ∑ i = 1 n ∥ ∇ f i ( x ) − ∇ f i ( y ) ∥ 2 ) 1/2 ≤ ( L 0 + L 1 ∥ ∇ f ( x )∥) ∥ x − y ∥ for ∥ x − y ∥ ≤ 1 / L 1 \lVert x-y\rVert\leq1/L_1 ∥ x − y ∥ ≤ 1/ L 1 : the components are bounded only in the mean-square sense, and the right-hand side contains the gradient of the sum f f f rather than the gradients of the components.