Introduction
Modern machine learning models are overparameterized and interpolate the training data: a single parameter vector minimizes the loss of every example. In this regime the noise of stochastic gradients vanishes at the solution, and stochastic methods attain faster rates than in the classical setting. SGD with a constant step size converges at the rate of gradient descent [9 , 13 , 17 ] , momentum methods attain accelerated rates [7 , 17 ] , and minibatch accelerated methods achieve a linear speedup in the minibatch size [19 ] . These results concern a single machine.
A natural question is how interpolation affects distributed training, where communication between machines is the bottleneck, as in federated learning [4 , 10 ] . The standard model of this regime is intermittent communication [20 ] : m m m workers run for N N N rounds, each worker makes K K K sequential stochastic gradient queries in every round, and the workers communicate only at the end of rounds. For convex objectives with L L L -Lipschitz gradients and stochastic gradients whose variance is uniformly bounded by σ 2 \sigma^2 σ 2 , Woodworth et al. [23 ] determined the minimax complexity of this model up to logarithmic factors: the optimal method is the better of minibatch accelerated SGD [6 ] , with N N N updates and batch size m K mK m K , and accelerated SGD on a single worker. Under interpolation, the noise vanishes at the solution and scales with the smoothness H s H_s H s of the individual losses, and the minimax complexity of intermittent communication in this regime has remained open.
Besides interpolation itself, the answer depends on what the oracle reveals. Discussing how their lower bound might be circumvented, Woodworth et al. [23, Sections 5.2 and 5.3] pointed to two features of statistical learning. First, stochastic gradients are usually gradients ∇ f ( x ; z ) \nabla f(x; z) ∇ f ( x ; z ) of smooth convex losses at random examples z z z . The hard oracle of their lower bound is not of this form, and they asked whether this structure yields better guarantees or whether the lower bound persists. Second, an algorithm can often evaluate the gradient of the same example at several points, as variance-reduced methods do, and distributed variants of SVRG improve on their lower bound. We study intermittent communication under interpolation in three oracle models that isolate these features, ordered by the information that a query provides.
Weak growth (Assumption 1): the oracle is unbiased, and its second moment is at most 2 H s 2H_s 2 H s times the suboptimality of the query. Nothing else is assumed. Interpolation implies this condition for sample gradients (Lemma 4.1). The upper bounds in Sections 3 and 4 rely only on this condition.
Fresh sample gradients (Assumption 2): the oracle returns the gradient of an H s H_s H s -smooth convex sample loss at a fresh example, and all sample losses are minimized at a common point. This is the setting of Woodworth et al. [23, Section 5.2] under interpolation.
Sample reuse (Assumption 3): the examples form a stored training set, and the gradient of the same example can be computed at several points. This is the setting of Woodworth et al. [23, Section 5.3] under interpolation.
Here L ≤ H s L \le H_s L ≤ H s is the smoothness of the population objective. The resulting minimax bounds are summarized in Table 1.
Contributions
Weak growth (Section 3). We determine the minimax rate up to the factor log 2 ( e m ) \log^2(em) log 2 ( e m ) , for general convex and for quadratic populations. For general convex populations, the lower bound extends the construction of Woodworth et al. [23 ] to relative noise, and the optimal method is the best of three baselines: returning x 0 x_0 x 0 , serial SGD on one worker, and minibatch AC-SA. For quadratic populations, local SGD steps reduce the effective smoothness to min { L , H s / K } \min\{L, H_s/K\} min { L , H s / K } , and an accelerated method on the resulting spectral surrogate is optimal.
Fresh sample gradients (Section 4). We prove a lower bound with a new hard instance, since the construction of Woodworth et al. [23 ] is not a sample-gradient oracle. This answers the question of Woodworth et al. [23, Section 5.2] for quadratic populations under interpolation: the structure of sample gradients gives the same minimax rate as the weak growth condition. For general convex populations a gap remains, and we show that closing it from above requires structure beyond the weak growth condition.
Sample reuse (Section 5). In the setting of Woodworth et al. [23, Section 5.3] under interpolation, we determine the minimax rate up to the factor J log 2 ( e m ) J\log^2(em) J log 2 ( e m ) with J = O ( log ( K N ) ) J = O(\log(KN)) J = O ( log ( K N )) . The upper bound is attained by an accelerated method on empirical objectives, which each worker computes by SVRG on its own batch. The lower bound holds for arbitrary reuse of samples, and for an infinite pool it coincides with the lower bound for fresh sample gradients.
All lower bounds hold for arbitrary randomized algorithms in finite dimension, without linear-span assumptions.
Related work
Intermittent communication and local updates. The intermittent communication model was studied by Woodworth et al. [20 ] within a graph-oracle framework for parallel stochastic optimization, together with lower bounds for it. Local SGD was analyzed by Stich [16 ] and compared with minibatch SGD by Woodworth et al. [22 ] , who showed that local SGD strictly dominates minibatch SGD for quadratic objectives, that its accelerated variant is minimax optimal for quadratics, and that it does not dominate minibatch SGD for general convex objectives; heterogeneous objectives were studied by Woodworth et al. [21 ] . The min-max complexity under bounded variance was determined by Woodworth et al. [23 ] . We refer to Kairouz et al. [4 ] for a survey of federated learning. Our results show that the advantage of local updates on quadratics persists under weak growth, and that sample reuse extends it to general convex populations under interpolation.
Interpolation and growth conditions. Fast convergence of SGD under the strong growth condition, where the second moment of the stochastic gradient is bounded by a multiple of ∥ ∇ F ∥ 2 \lVert\nabla F\rVert^2 ∥ ∇ F ∥ 2 , was shown by Schmidt and Le Roux [13 ] . The benefits of interpolation for SGD and its minibatch speedup were explained by Ma et al. [9 ] . Vaswani et al. [17 ] introduced the weak growth condition, which interpolation implies for smooth finite sums, and showed that SGD with Nesterov acceleration attains the deterministic accelerated rate under strong growth. For least squares, Liu and Belkin [7 ] accelerated SGD in the interpolation regime. Rates that adapt to the minimal loss for smooth nonnegative losses go back to Srebro et al. [15 ] , and accelerated minibatch methods with such rates were given by Cotter et al. [2 ] and, with the optimal dependence on the minibatch size, by Woodworth and Srebro [19 ] . We use the accelerated scheme of the latter in Appendix C. These works concern a single machine; we study the intermittent communication model.
Table 1. Bounds on the minimax risk in the intermittent communication model with m m m workers, N N N rounds, and K K K queries per worker per round, for convex objectives with L L L -Lipschitz gradients and ∥ x 0 − x ∗ ∥ ≤ R \lVert x_0-x^*\rVert\le R ∥ x 0 − x ∗ ∥ ≤ R . Numerical constants are omitted. The result of Woodworth et al. [23 ] holds up to logarithmic factors. Here H s H_s H s is the constant of the weak growth condition (the smoothness of the sample functions), σ 2 \sigma^2 σ 2 the variance, and S S S the pool size.
Local subproblems. Solving a proximal subproblem on a minibatch is the idea of minibatch-prox [18 ] . Our method with sample reuse computes such proximal points locally by SVRG [3 ] and uses them as gradients of empirical Moreau envelopes inside an accelerated method; interpolation guarantees that the envelopes share the minimizer of the population.
Lower bounds. Our lower bounds use hidden chains, which go back to Nemirovsky and Yudin [11 ] , combined with random rotations to avoid linear-span assumptions, as in Woodworth et al. [23 ] . For the weak growth condition, we extend the rare-progress oracle of Woodworth et al. [23 ] to relative noise.
For quadratic populations, we use a random-rotation representation of adaptive matrix-vector algorithms, building on techniques developed by Braverman et al. [1 ] and closely related to those recently employed by Shamir [14 ] in the centralized setting. Our analysis additionally accounts for the depth of batched adaptive queries and uses a Chebyshev-type certificate tailored to smooth convex quadratic objectives. We combine these techniques with Bernoulli masking of entire sample losses, an approach already used in our previous work on federated variational inequalities [12 ] , to establish lower bounds for interpolating sample-gradient oracles under intermittent communication.
Setting
Throughout, 0 < L ≤ H s 0<L\le H_s 0 < L ≤ H s and R > 0 R>0 R > 0 are constants, m m m , K K K , N ≥ 1 N\ge1 N ≥ 1 are integers, d ≥ 1 d\ge1 d ≥ 1 is the dimension, and x 0 ∈ R d x_0\in\mathbb{R}^d x 0 ∈ R d is the initial point. All norms and inner products are Euclidean, and all logarithms are natural.
Intermittent communication.
Definition 2.1 (Intermittent communication). An algorithm in the class I C ( m , K , N ) \mathrm{IC}(m,K,N) IC ( m , K , N ) runs m m m workers for N N N rounds. In every round, each worker makes at most K K K sequential oracle queries. Each query of a worker is a measurable function of the public parameters, the random seed of the algorithm, all messages exchanged before the current round, and the queries and replies of this worker in the current round. At the end of every round the workers exchange arbitrary messages. The output x ^ ∈ R d \hat{x}\in\mathbb{R}^d x ^ ∈ R d is a measurable function of the seed and of all messages. The seed is independent of the instance and of the oracle randomness. No restriction is imposed on memory, local computation, message size, or the norms of queries and outputs; in particular, queries need not lie in the span of previous replies. The public parameters are m m m , K K K , N N N , L L L , H s H_s H s , R R R , d d d , x 0 x_0 x 0 , and the pool size S S S in Assumption 3. The objective, its minimizer, and the oracle law are unknown to the algorithm.
Oracle assumptions. We consider three oracle models, ordered by the information that a query provides.
Assumption 1 (Weak growth condition). The objective F : R d → R F : \mathbb{R}^d \to\mathbb{R} F : R d → R is convex, has an L L L -Lipschitz gradient, and has a minimizer x ∗ x^* x ∗ with ∥ x 0 − x ∗ ∥ ≤ R \lVert x_0-x^*\rVert\le R ∥ x 0 − x ∗ ∥ ≤ R . There are a probability distribution D \mathcal{D} D on a measurable space Ξ \Xi Ξ and a jointly measurable map g : R d × Ξ → R d g : \mathbb{R}^d \times\Xi\to\mathbb{R}^d g : R d × Ξ → R d such that, for every x ∈ R d x \in\mathbb{R}^d x ∈ R d ,
E ξ ∼ D g ( x , ξ ) = ∇ F ( x ) , E ξ ∼ D ∥ g ( x , ξ ) ∥ 2 ≤ 2 H s ( F ( x ) − F ( x ∗ ) ) . (1) \mathbb{E}_{\xi\sim\mathcal{D}} g(x,\xi)=\nabla F(x), \qquad\mathbb{E}_{\xi\sim\mathcal{D}}\lVert g(x,\xi)\rVert^2 \le2H_s\bigl(F(x)-F(x^*)\bigr). \tag*{(1)} E ξ ∼ D g ( x , ξ ) = ∇ F ( x ) , E ξ ∼ D ∥ g ( x , ξ ) ∥ 2 ≤ 2 H s ( F ( x ) − F ( x ∗ ) ) . (1)
On a query x x x , the oracle draws a fresh sample ξ ∼ D \xi\sim\mathcal{D} ξ ∼ D and returns only g ( x , ξ ) g(x,\xi) g ( x , ξ ) .
The map g ( ⋅ , ξ ) g(\cdot,\xi) g ( ⋅ , ξ ) in Assumption 1 need not be a gradient field or continuous.
Assumption 2 (Sample gradients under interpolation). There are a probability distribution D \mathcal{D} D on a measurable space Ξ \Xi Ξ and differentiable convex functions f ξ : R d → R f_\xi: \mathbb{R}^d \to\mathbb{R} f ξ : R d → R , ξ ∈ Ξ \xi\in\Xi ξ ∈ Ξ , such that ( x , ξ ) ↦ f ξ ( x ) (x,\xi) \mapsto f_\xi(x) ( x , ξ ) ↦ f ξ ( x ) and ( x , ξ ) ↦ ∇ f ξ ( x ) (x,\xi) \mapsto\nabla f_\xi(x) ( x , ξ ) ↦ ∇ f ξ ( x ) are jointly measurable and, for some x ∗ ∈ R d x^* \in\mathbb{R}^d x ∗ ∈ R d with ∥ x 0 − x ∗ ∥ ≤ R \lVert x_0-x^*\rVert\le R ∥ x 0 − x ∗ ∥ ≤ R and D \mathcal{D} D -almost every ξ \xi ξ ,
∥ ∇ f ξ ( x ) − ∇ f ξ ( y ) ∥ ≤ H s ∥ x − y ∥ for all x , y ∈ R d , ∇ f ξ ( x ∗ ) = 0. (2) \lVert\nabla f_\xi(x)-\nabla f_\xi(y)\rVert\le H_s\lVert x-y\rVert\quad\text{for all }x,y\in\mathbb{R}^d,\qquad\nabla f_\xi(x^*)=0. \tag*{(2)} ∥ ∇ f ξ ( x ) − ∇ f ξ ( y )∥ ≤ H s ∥ x − y ∥ for all x , y ∈ R d , ∇ f ξ ( x ∗ ) = 0. (2)
The objective
F ( x ) = E ξ ∼ D [ f ξ ( x ) − f ξ ( x ∗ ) ] F(x)=\mathbb{E}_{\xi\sim\mathcal{D}}\bigl[f_\xi(x)-f_\xi(x^*)\bigr] F ( x ) = E ξ ∼ D [ f ξ ( x ) − f ξ ( x ∗ ) ]
has an L L L -Lipschitz gradient. On a query x x x , the oracle draws a fresh sample ξ ∼ D \xi\sim\mathcal{D} ξ ∼ D , independent of everything else, and returns only the vector g ( x , ξ ) = ∇ f ξ ( x ) g(x,\xi)=\nabla f_\xi(x) g ( x , ξ ) = ∇ f ξ ( x ) .
The second condition in (2.2) is the interpolation assumption: x ∗ x^* x ∗ minimizes almost every sample function. In particular, F ( x ∗ ) = 0 = min F F(x^*)=0=\min F F ( x ∗ ) = 0 = min F .
Assumption 3 (Sample reuse). The functions f ξ f_\xi f ξ and F F F satisfy the conditions of Assumption 2, and S ∈ N ∪ { ∞ } S \in\mathbb{N}\cup\{\infty\} S ∈ N ∪ { ∞ } , S ≥ 1 S\ge1 S ≥ 1 . Before the run, samples ξ 1 , ξ 2 , … ∼ i i d D \xi_1,\xi_2,\ldots\mathrel{\stackrel{\mathrm{iid}}{\sim}} \mathcal{D} ξ 1 , ξ 2 , … ∼ iid D are drawn independently of the seed of the algorithm; only ξ 1 , … , ξ S \xi_1,\ldots,\xi_S ξ 1 , … , ξ S are available. A query is a pair ( i , x ) (i,x) ( i , x ) with i ∈ N i\in\mathbb{N} i ∈ N and x ∈ R d x\in\mathbb{R}^d x ∈ R d ; the oracle returns ∇ f ξ i ( x ) \nabla f_{\xi_i}(x) ∇ f ξ i ( x ) if i ≤ S i\le S i ≤ S and a failure symbol otherwise. Every query, including a repeated one, is one of the at most K K K queries of the worker in the current round. The label i i i carries no information about ξ i \xi_i ξ i , and labels may be included in messages.
An instance is a choice of the objects in an assumption. We denote by W d \mathcal{W}_d W d and S d \mathcal{S}_d S d the sets of instances satisfying Assumptions 1 and 2, and by W d q u a d \mathcal{W}^{\mathrm{quad}}_d W d quad and S d q u a d \mathcal{S}^{\mathrm{quad}}_d S d quad their subsets with a quadratic objective
F ( x ) = 1 2 ⟨ x − x ∗ , A ( x − x ∗ ) ⟩ , 0 ⪯ A ⪯ L I ; F(x)=\frac{1}{2}\langle x-x^*,A(x-x^*)\rangle,\qquad0\preceq A\preceq LI; F ( x ) = 2 1 ⟨ x − x ∗ , A ( x − x ∗ )⟩ , 0 ⪯ A ⪯ L I ;
the sample functions f ξ f_\xi f ξ need not be quadratic.
Minimax risk. For a set C \mathcal{C} C of instances and a model of oracle access, the minimax risk is
M d ( C ) = inf A ∈ A l g ( m , K , N ) sup instances in C E [ F ( x ^ ) − F ( x ∗ ) ] , \mathcal{M}_d(\mathcal{C})=
\inf_{A\in\mathrm{Alg}(m,K,N)}
\sup_{\text{instances in }\mathcal{C}}
\mathbb{E}\bigl[F(\widehat{x})-F(x^*)\bigr], M d ( C ) = A ∈ Alg ( m , K , N ) inf instances in C sup E [ F ( x ) − F ( x ∗ ) ] ,
where the expectation is over the oracle randomness and the seed of A A A ; the instance is fixed before both are drawn. We write
M d w g = M d ( W d ) , M d w g , q u a d = M d ( W d q u a d ) , M d s g = M d ( S d ) , M d s g , q u a d = M d ( S d q u a d ) \mathcal{M}^{\mathrm{wg}}_d=\mathcal{M}_d(\mathcal{W}_d),\qquad
\mathcal{M}^{\mathrm{wg},\mathrm{quad}}_d=\mathcal{M}_d(\mathcal{W}^{\mathrm{quad}}_d),\qquad
\mathcal{M}^{\mathrm{sg}}_d=\mathcal{M}_d(\mathcal{S}_d),\qquad
\mathcal{M}^{\mathrm{sg},\mathrm{quad}}_d=\mathcal{M}_d(\mathcal{S}^{\mathrm{quad}}_d) M d wg = M d ( W d ) , M d wg , quad = M d ( W d quad ) , M d sg = M d ( S d ) , M d sg , quad = M d ( S d quad )
for the fresh oracles of Assumptions 1 and 2, and M d p o o l ( S ) \mathcal{M}^{\mathrm{pool}}_d(S) M d pool ( S ) for M d ( S d ) \mathcal{M}_d(\mathcal{S}_d) M d ( S d ) under Assumption 3. The fresh oracle of Assumption 2 is the special case S = ∞ S=\infty S = ∞ of Assumption 3 in which every label is queried at most once; hence
M d p o o l ( ∞ ) ≤ M d s g . (3) \mathcal{M}^{\mathrm{pool}}_d(\infty)\le\mathcal{M}^{\mathrm{sg}}_d. \tag*{(3)} M d pool ( ∞ ) ≤ M d sg . (3)
see Appendix B.1.
Weak growth: tight minimax rates
We begin by considering the stochastic oracle satisfying Assumption 1, that satisfy the weak growth condition (WGC) 2.1. The WGC has been studied in the context of stochastic optimization by Waswani et al. [17 ] , who demonstrated its relevance to statistical learning under interpolation. In particular, they established the following result, showing that the WGC arises naturally for smooth convex finite-sum objectives whose component functions share a common minimizer.
Proposition 3.1 [17 ] . Let
f ( w ) = 1 n ∑ i = 1 n f i ( w ) f(w) = \frac{1}{n}\sum_{i=1}^{n} f_i(w) f ( w ) = n 1 i = 1 ∑ n f i ( w ) be a convex finite-sum objective, where each component f i f_i f i is convex and L i L_i L i -smooth. Suppose that the model interpolates the data, i.e., there exists a common minimizer w ∗ w^* w ∗ of all component functions. Then, for every w w w ,
E i [ ∥ ∇ f i ( w ) ∥ 2 ] ≤ 2 L max ( f ( w ) − f ( w ∗ ) ) , (4) \mathbb{E}_i\left[\left\|\nabla f_i(w)\right\|^2\right] \le2L_{\max}\left(f(w)-f(w^*)\right), \tag*{(4)} E i [ ∥ ∇ f i ( w ) ∥ 2 ] ≤ 2 L m a x ( f ( w ) − f ( w ∗ ) ) , (4) where L max = max i = 1 , … , n L i L_{\max} = \max_{i=1,\ldots,n} L_i L m a x = max i = 1 , … , n L i .
The proof can be found in Vaswani et al. [17 ] .
As we will show in the subsequent sections, the WGC follows from the stronger structural assumptions introduced in Assumptions 2 and 3. Thus, while the WGC is imposed explicitly in the present section, it holds automatically under the oracle models considered in § 4 and § 5 . Importantly, the upper bounds derived in this section rely only on the WGC 2.1, without exploiting the additional structure of these oracle models. They therefore provide a natural baseline for assessing whether the stronger assumptions lead to improved convergence guarantees.
Theorem 3.2 (General convex populations under weak growth). Let Assumption 1 hold. Then, in every dimension d ≥ 1 d \ge1 d ≥ 1 ,
M d w g ≤ 288 R 2 min { L , H s K N , L N 2 + H s m K N } , \mathfrak{M}^{\mathrm{wg}}_d \le288R^2 \min\left\{L,\frac{H_s}{KN},\frac{L}{N^2}+\frac{H_s}{mKN}\right\}, M d wg ≤ 288 R 2 min { L , K N H s , N 2 L + m K N H s } , and, for every d ≥ d w g d \ge d_{\mathrm{wg}} d ≥ d wg ,
M d w g ≥ R 2 2 24 log 2 ( e m ) min { L , H s K N , L N 2 + H s m K N } , \mathfrak{M}^{\mathrm{wg}}_d \ge\frac{R^2}{2^{24}\log^2(em)}\min\left\{L,\frac{H_s}{KN},\frac{L}{N^2}+\frac{H_s}{mKN}\right\}, M d wg ≥ 2 24 log 2 ( e m ) R 2 min { L , K N H s , N 2 L + m K N H s } , where d w g < ∞ d_{\mathrm{wg}} < \infty d wg < ∞ is defined in (A.10) and depends only on m m m , K K K , N N N , and H s / L H_s/L H s / L .
Theorem 3.3 (Quadratic populations under weak growth). Let Assumption 1 hold with a quadratic objective F F F . Then, in every dimension d ≥ 1 d \ge1 d ≥ 1 ,
M d w g , q u a d ≤ 8 R 2 min { L , min { L , H s / K } N 2 + H s m K N } , \mathfrak{M}^{\mathrm{wg},\mathrm{quad}}_d \le8R^2 \min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}, M d wg , quad ≤ 8 R 2 min { L , N 2 min { L , H s / K } + m K N H s } , and, for every d ≥ 2048 m ( K N ) 2 d \ge2048m(KN)^2 d ≥ 2048 m ( K N ) 2 ,
M d w g , q u a d ≥ R 2 73728 log 2 ( e m ) min { L , min { L , H s / K } N 2 + H s m K N } . \mathfrak{M}^{\mathrm{wg},\mathrm{quad}}_d \ge\frac{R^2}{73728\log^2(em)}\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}. M d wg , quad ≥ 73728 log 2 ( e m ) R 2 min { L , N 2 min { L , H s / K } + m K N H s } .
The upper bound in Theorem 3.2 is achieved by a parameter-based selection among returning x 0 x_0 x 0 , serial averaged SGD on one worker, and common-query minibatch AC-SA [6 ] with N N N accelerated updates and batch size m K mK m K . The upper bound in Theorem 3.3 is achieved by a parameter-based selection between returning x 0 x_0 x 0 and Algorithm 1, in which every worker runs K K K local SGD steps from a common point and the server performs an accelerated update with the averaged block displacements. The upper bounds are proved in Appendices § 8 .§ 3 and § 8 .§ 2 , the lower bounds in Appendices § 7 .§ 4 and § 7 .§ 3 , and the proofs are completed in Appendix § 10 .
General convex populations. The lower bound in Theorem 3.2 extends the lower bound of Woodworth et al. [23 ] from bounded variance to relative noise. As in their construction, the hard oracle reveals the next direction of a hidden chain only with a small probability p p p , so that the progress of the whole system within a round is the maximum of m m m binomial counts, which is of order log m \log m log m .
Together with the statistical term, Theorem 3.2 shows that under weak growth the minimax rate is attained by the best of three baselines: returning x 0 x_0 x 0 , serial SGD on one worker, and minibatch AC-SA. This mirrors the conclusion of Woodworth et al. [23 ] for bounded variance, where the better of minibatch and single-machine accelerated SGD is optimal: for general convex populations, local updates beyond these baselines do not help.
Quadratic populations. For quadratic populations, local updates do help, because the gradients are linear. The expected displacement of K K K local SGD steps with step size 1 / H s 1/H_s 1/ H s from a common point y y y equals the gradient at y y y of the spectral surrogate
F K ( y ) = 1 2 ⟨ y − x ∗ , A K ( y − x ∗ ) ⟩ , A K = H s K [ I − ( I − A H s ) K ] , F_K(y) = \frac{1}{2}\langle y-x^*,A_K(y-x^*)\rangle,\qquad A_K = \frac{H_s}{K}\left[I-\left(I-\frac{A}{H_s}\right)^K\right], F K ( y ) = 2 1 ⟨ y − x ∗ , A K ( y − x ∗ )⟩ , A K = K H s [ I − ( I − H s A ) K ] ,
whose smoothness is min { L , H s / K } \min\{L,H_s/K\} min { L , H s / K } (Lemma B.5). An accelerated outer method on F K F_K F K therefore replaces L L L by min { L , H s / K } \min\{L,H_s/K\} min { L , H s / K } in the communication term, while the averages of the local iterates control the original objective F F F . For least squares, linearity of the gradients was used to accelerate SGD by Liu and Belkin [7 ] . The lower bound in Theorem 3.3 is attained by sample-gradient instances, which we describe in § 4 .
We also note that our lower-bound construction may be useful for establishing corresponding complexity lower bounds in the centralized setting, thereby complementing the study of stochastic optimization under the weak growth condition initiated by Vaswani et al. [17 ] .
Fresh sample gradients: an open gap
In this section we consider the oracle of Assumption 2. It is motivated by statistical learning, where F ( x ) = E z ℓ ( x ; z ) F(x) = \mathbb{E}_z\,\ell(x;z) F ( x ) = E z ℓ ( x ; z ) is the expected loss over the data distribution and a stochastic gradient is the gradient of the loss at a fresh example, g ( x , z ) = ∇ ℓ ( x ; z ) g(x,z) = \nabla\ell(x;z) g ( x , z ) = ∇ ℓ ( x ; z ) ; a minibatch gradient is the average of such sample gradients. The interpolation condition ∇ f ξ ( x ∗ ) = 0 \nabla f_{\xi}(x^*) = 0 ∇ f ξ ( x ∗ ) = 0 describes overparameterized models, in which one parameter vector fits every example [9 , 17 ] .
This assumption gives the oracle a nontrivial structure. Every reply is the gradient of a convex H s H_s H s -smooth random function that is minimized at the same point x ∗ x^* x ∗ . We ask whether this structure leads to faster rates than the weak growth condition alone, that is, whether the bounds of § 3 can be improved. The basic consequence of the structure is the weak growth condition itself; it is the workhorse of all our upper bounds.
Lemma 4.1 (Interpolation implies weak growth). Let Assumption 2 hold. Then, for every x ∈ R d x \in\mathbb{R}^d x ∈ R d ,
E ∇ f ξ ( x ) = ∇ F ( x ) , E ∥ ∇ f ξ ( x ) ∥ 2 ≤ 2 H s ( F ( x ) − F ( x ∗ ) ) , \mathbb{E}\,\nabla f_{\xi}(x) = \nabla F(x), \qquad\mathbb{E}\lVert\nabla f_{\xi}(x)\rVert^2 \le2H_s\left(F(x)-F(x^*)\right), E ∇ f ξ ( x ) = ∇ F ( x ) , E ∥ ∇ f ξ ( x ) ∥ 2 ≤ 2 H s ( F ( x ) − F ( x ∗ ) ) , that is, the oracle g ( x , ξ ) = ∇ f ξ ( x ) g(x,\xi) = \nabla f_{\xi}(x) g ( x , ξ ) = ∇ f ξ ( x ) satisfies Assumption 1 with the same F F F and x ∗ x^* x ∗ . Consequently, M d s g ≤ M d w g \mathcal{M}^{\mathrm{sg}}_d \le\mathcal{M}^{\mathrm{wg}}_d M d sg ≤ M d wg and M d s g , q u a d ≤ M d w g , q u a d \mathcal{M}^{\mathrm{sg,quad}}_d \le\mathcal{M}^{\mathrm{wg,quad}}_d M d sg , quad ≤ M d wg , quad .
The proof is given in Appendix B.1.
Theorem 4.2 (Quadratic populations with sample gradients). Let Assumption 2 hold with a quadratic objective F F F . Then, in every dimension d ≥ 1 d \ge1 d ≥ 1 ,
M d s g , q u a d ≤ 8 R 2 min { L , min { L , H s / K } N 2 + H s m K N } , \mathcal{M}^{\mathrm{sg,quad}}_d \le8R^2\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}, M d sg , quad ≤ 8 R 2 min { L , N 2 min { L , H s / K } + m K N H s } , and, for every d ≥ 2048 m ( K N ) 2 d \ge2048m(KN)^2 d ≥ 2048 m ( K N ) 2 ,
M d s g , q u a d ≥ R 2 73728 log 2 ( e m ) min { L , min { L , H s / K } N 2 + H s m K N } . \mathcal{M}^{\mathrm{sg,quad}}_d \ge\frac{R^2}{73728\log^2(em)}\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}. M d sg , quad ≥ 73728 log 2 ( e m ) R 2 min { L , N 2 min { L , H s / K } + m K N H s } .
Theorem 4.3 (General convex populations with sample gradients). Let Assumption 2 hold. Then, in every dimension d ≥ 1 d \ge1 d ≥ 1 ,
M d s g ≤ 288 R 2 min { L , H s K N + L N 2 + H s m K N } , \mathcal{M}^{\mathrm{sg}}_d \le288R^2\min\left\{L,\frac{H_s}{KN}+\frac{L}{N^2}+\frac{H_s}{mKN}\right\}, M d sg ≤ 288 R 2 min { L , K N H s + N 2 L + m K N H s } , and, for every d ≥ 2048 m ( K N ) 2 d \ge2048m(KN)^2 d ≥ 2048 m ( K N ) 2 ,
M d s g ≥ R 2 73728 log 2 ( e m ) min { L , min { L , H s / K } N 2 + H s m K N } . \mathcal{M}^{\mathrm{sg}}_d \ge\frac{R^2}{73728\log^2(em)}\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}. M d sg ≥ 73728 log 2 ( e m ) R 2 min { L , N 2 min { L , H s / K } + m K N H s } .
By Lemma 4.1, the upper bounds are those of Theorems 3.3 and 3.2, attained by the same algorithms. The lower bound is the same in both theorems; it is proved in Appendix A.3, and the proofs are completed in Appendix D.
The lower bound. The lower bound of Woodworth et al. [23 ] for intermittent communication does not apply under Assumption 2. It is proved for oracles whose variance is bounded by σ 2 \sigma^2 σ 2 , and its hard oracle returns a randomly truncated or rescaled gradient of a hidden chain, where the truncation depends on the query. Such replies are not gradients of fixed convex sample functions minimized at a common point, as Assumption 2 requires. The same applies to the hard oracle of Theorem 3.2. We therefore build a new hard instance. The idea of Bernoulli masking of entire sample losses was previously used in our work on composite variational inequalities [12 ] . Related random-rotation and polynomial approximation techniques were also employed by Shamir [14 ] to establish lower bounds for centralized matrix-vector oracle algorithms. The sample functions are whole quadratics f ξ = ( ξ / p ) F f_{\xi} = (\xi/p)F f ξ = ( ξ / p ) F masked by Bernoulli variables ξ ∼ Bernoulli ( p ) \xi\sim\operatorname{Bernoulli}(p) ξ ∼ Bernoulli ( p ) , so that a worker receives the exact gradient of F F F only at the queries with ξ = 1 \xi= 1 ξ = 1 , and F F F is a randomly rotated quadratic whose spectrum is placed at Chebyshev nodes. A depth argument for adaptive matrix queries, without any linear-span assumption, gives the term min { L , H s / K } / N 2 \min\{L,H_s/K\}/N^2 min { L , H s / K } / N 2 , and a statistical term H s / ( m K N ) H_s/(mKN) H s / ( m K N ) is a simple consequence of lower bound from [19 ] .
The gap. For quadratic populations, Theorems 3.3 and 4.2 give the same rate: the structure of sample gradients adds nothing to the weak growth condition, and the improvement over general convex populations comes from the linearity of the gradients. For general convex populations, Theorem 4.3 leaves a gap between the rates
R 2 min { L , H s K N , L N 2 + H s m K N } and R 2 min { L , min { L , H s / K } N 2 + H s m K N } . R^2 \min\left\{L,\frac{H_s}{KN},\frac{L}{N^2}+\frac{H_s}{mKN}\right\}
\quad\text{and}\quad
R^2 \min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}. R 2 min { L , K N H s , N 2 L + m K N H s } and R 2 min { L , N 2 min { L , H s / K } + m K N H s } .
Theorem 3.2 shows that the upper bound cannot be improved by an algorithm whose guarantee relies only on Lemma 4.1 : any improvement must use the structure of Assumption 2 beyond the weak growth condition. On the other hand, Theorem 5.1 shows that the lower bound is attained, up to logarithmic factors, when samples can be reused. The gap is therefore the question of whether fresh access to samples is strictly weaker than reuse for general convex populations.
Sample reuse: closing the gap
In statistical learning the training set is usually stored, and the loss of the same example can be differentiated at several points, as in multi-epoch training. Assumption 3 models this situation: the samples form a pool of size S S S , and a query may reuse any label. With fresh samples, each sample function is observed through a single gradient. With reuse, a worker can query its own batch of samples repeatedly within a round and therefore optimize the empirical objective of the batch, rather than only estimate its gradient.
This possibility gives a new upper bound. Within one round, a worker takes a batch S S S of n n n samples and approximately computes, by SVRG [3 ] on this batch, the proximal point
p S ( y ) = arg min u { f S ( u ) + τ 2 ∥ u − y ∥ 2 } , f S = 1 n ∑ ξ ∈ S f ξ , p_S(y)=\underset{u}{\operatorname{arg\,min}}\left\{f_S(u)+\frac{\tau}{2}\lVert u-y\rVert^2\right\},\qquad f_S=\frac{1}{n}\sum_{\xi\in S}f_\xi, p S ( y ) = u arg min { f S ( u ) + 2 τ ∥ u − y ∥ 2 } , f S = n 1 ξ ∈ S ∑ f ξ ,
at the common point y y y , using gradient queries only. The vector τ ( y − p S ( y ) ) \tau(y-p_S(y)) τ ( y − p S ( y )) is the gradient of the empirical Moreau envelope of the batch, which is τ \tau τ -smooth with τ = 8 H s / n \tau=8H_s/n τ = 8 H s / n and, by interpolation, is minimized at the same point x ∗ x^* x ∗ as every sample function. The accelerated method of Woodworth and Srebro [19 ] , applied to these envelopes, therefore sees the smoothness τ \tau τ , of order H s log ( K N ) / K H_s\log(KN)/K H s log ( K N ) / K , instead of L L L , and a final proximal step on a fresh batch transfers the guarantee from the envelope back to F F F . Thus reuse plays for general convex populations the role that the linearity of the gradients plays for quadratic populations in Section 3: it replaces L L L by roughly H s / K H_s/K H s / K in the communication term. The scheme is related to minibatch-prox [18 ] , where every machine solves a regularized subproblem on its minibatch. It is stated as Algorithm 3.
Theorem 5.1 (Minimax bounds with sample reuse). Let Assumption 3 hold, and let
J = ⌈ log 2 ( 625 K ( K + 8 ) ( N + 1 ) 4 ) ⌉ . J=\left\lceil\log_2\left(625K(K+8)(N+1)^4\right)\right\rceil. J = ⌈ log 2 ( 625 K ( K + 8 ) ( N + 1 ) 4 ) ⌉ . Then, in every dimension d ≥ 1 d\ge1 d ≥ 1 ,
M d p o o l ( S ) ≤ 10 7 J R 2 min { L , min { L , H s / K } N 2 + H s min { S , m K N } } . \mathcal{M}^{\mathrm{pool}}_d(S)\le10^7JR^2\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{\min\{S,mKN\}}\right\}. M d pool ( S ) ≤ 1 0 7 J R 2 min { L , N 2 min { L , H s / K } + min { S , m K N } H s } . and, for every d ≥ d p o o l d\ge d_{\mathrm{pool}} d ≥ d pool ,
M d p o o l ( S ) ≥ R 2 10 6 log 2 ( e m ) min { L , min { L , H s / K } N 2 + H s min { S , m K N } } , \mathcal{M}^{\mathrm{pool}}_d(S)\ge\frac{R^2}{10^6\log^2(em)}\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{\min\{S,mKN\}}\right\}, M d pool ( S ) ≥ 1 0 6 log 2 ( e m ) R 2 min { L , N 2 min { L , H s / K } + min { S , m K N } H s } , where d p o o l < ∞ d_{\mathrm{pool}}<\infty d pool < ∞ is defined in (A.21) and depends only on m m m , K K K , N N N , and H s / L H_s/L H s / L .
The upper bound is achieved by a parameter-based selection among returning x 0 x_0 x 0 , serial averaged SGD on one worker, common-query minibatch AC-SA, and Algorithm 3; it is proved in Appendix C. The proof is completed in Appendix D.
The lower bound. The lower bound holds for arbitrary adaptive reuse of labels. In the hard instance every sample function carries its own Bernoulli coins on the edges of a hidden chain, so repeated queries of a sample reveal nothing beyond its coins (Appendix A.5). The statistical term depends on the number min { S , m K N } \min\{S,mKN\} min { S , m K N } of distinct samples that can be inspected: with a small pool, more queries do not help.
For S = ∞ S=\infty S = ∞ , both bounds of Theorem 5.1 coincide with the lower bound of Theorem 4.3 up to the factor J log 2 ( e m ) J\log^2(em) J log 2 ( e m ) , while M d p o o l ( ∞ ) ≤ M d s g \mathcal{M}^{\mathrm{pool}}_d(\infty)\le\mathcal{M}^{\mathrm{sg}}_d M d pool ( ∞ ) ≤ M d sg by (2.3). The lower bound of Theorem 4.3 is therefore the minimax rate with reuse, and the open question of Section 4 is whether the same rate can be attained with fresh samples.
Discussion and Future work
In this work, we establish upper and lower complexity bounds for distributed stochastic convex optimization under several oracle models motivated by statistical learning. Our results demonstrate the importance of oracle assumptions in determining the minimax complexity of intermittent communication.
Under the weakest oracle model, which assumes only the weak growth condition, we show that for general convex objectives, local updates cannot improve the minimax rate beyond that achieved by simple baselines. However, for quadratic population objectives, local computation becomes beneficial and leads to an improved communication-dependent term. The sample-reuse oracle extends this improvement to general convex objectives, allowing an accelerated rate through repeated access to individual sample functions.
These findings highlight that the weak growth condition alone does not fully capture the potential benefits of local computation. Additional structure, either in the objective function or in the oracle’s access to individual samples, can fundamentally change the achievable convergence rates. Nevertheless, several important questions remain open.
The gap for fresh sample gradients. For general convex population objectives with fresh sample gradients, Theorem 4.3 leaves a gap between the upper and lower bounds. By Theorem 3.2 , any improvement of the upper bound must exploit properties of sample gradients beyond the weak growth condition. On the other hand, Theorem 5.1 shows that the rate appearing in the lower bound can be attained, up to logarithmic factors, when repeated access to samples is allowed. It remains open whether the gap can be closed by an algorithm using only fresh samples, or whether repeated sample access leads to a genuine separation between the two oracle models.
Beyond convexity. Convex optimization provides a natural starting point, but many overparameterized learning models are inherently nonconvex. Nevertheless, certain overparameterized nonlinear systems, including sufficiently wide neural networks, have been shown to satisfy variants of the Polyak–Łojasiewicz condition, such as P L ∗ PL^{*} P L ∗ , under suitable assumptions and in appropriate regions of the parameter space [8 ] ; see also Karimi et al. [5 ] .
This motivates studying the minimax complexity of intermittent communication for nonconvex interpolating problems satisfying PL-type conditions. In particular, it would be interesting to determine whether the advantages of local computation observed for quadratic objectives and under sample reuse persist beyond convexity, and how the choice of stochastic oracle affects the resulting minimax rates.
Lower bounds
Conventions. By translation, x 0 = 0 x_0 = 0 x 0 = 0 in all constructions. Expectations of nonnegative random variables are understood in the extended sense, so an algorithm with infinite expected risk satisfies every lower bound. We use two elementary reductions.
Averaging over instances. Let a random instance be drawn from a law that may depend on the algorithm but is independent of its seed and of the oracle randomness. If the expected risk, averaged also over the instance, is at least c c c , then some fixed instance has expected risk at least c c c .
Padding. An instance in R d 0 \mathbb{R}^{d_0} R d 0 is extended to R d \mathbb{R}^d R d , d ≥ d 0 d \ge d_0 d ≥ d 0 , by letting all functions and oracle maps depend only on the first d 0 d_0 d 0 coordinates, with zero replies in the remaining coordinates. The extension belongs to the same class with the same constants, and an algorithm in R d \mathbb{R}^d R d is simulated in R d 0 \mathbb{R}^{d_0} R d 0 by projecting its queries and its output onto the first d 0 d_0 d 0 coordinates. Hence a lower bound in dimension d 0 d_0 d 0 holds in every dimension d ≥ d 0 d \ge d_0 d ≥ d 0 .
Tools
Lemma A.1 (Spherical caps). Let u u u be uniformly distributed on the unit sphere of an n n n -dimensional Euclidean space, n ≥ 2 n \ge2 n ≥ 2 , and let z z z be a fixed vector with ∥ z ∥ ≤ r ˉ \lVert z\rVert\le\bar r ∥ z ∥ ≤ r ˉ . Then, for every b > 0 b > 0 b > 0 ,
P ( ∣ ⟨ u , z ⟩ ∣ > b ) ≤ 2 exp ( − ( n − 1 ) b 2 2 r ˉ 2 ) . \mathbb{P}\left(\lvert\langle u,z\rangle\rvert> b\right) \le2\exp\left(-\frac{(n-1)b^2}{2\bar r^2}\right). P ( ∣⟨ u , z ⟩∣ > b ) ≤ 2 exp ( − 2 r ˉ 2 ( n − 1 ) b 2 ) . The same bound holds conditionally if z z z is measurable with respect to a σ \sigma σ -algebra given which u u u is uniform on such a sphere.
Proof. If b ≥ ∥ z ∥ b \ge\lVert z\rVert b ≥ ∥ z ∥ , the probability is zero. Otherwise put c = b / ∥ z ∥ ∈ ( 0 , 1 ) c = b/\lVert z\rVert\in(0,1) c = b / ∥ z ∥ ∈ ( 0 , 1 ) ; by rotation invariance the probability equals P ( ∣ u 1 ∣ > c ) \mathbb{P}(\lvert u_1\rvert> c) P (∣ u 1 ∣ > c ) . Write u = Z / ∥ Z ∥ u = Z/\lVert Z\rVert u = Z / ∥ Z ∥ with a standard Gaussian vector Z ∈ R n Z \in\mathbb{R}^n Z ∈ R n , and let W = ∑ j = 2 n Z j 2 W = \sum_{j=2}^{n} Z_j^2 W = ∑ j = 2 n Z j 2 . Then ∣ u 1 ∣ > c \lvert u_1\rvert> c ∣ u 1 ∣ > c if and only if ∣ Z 1 ∣ > c 2 W / ( 1 − c 2 ) \lvert Z_1\rvert> \sqrt{c^2W/(1-c^2)} ∣ Z 1 ∣ > c 2 W / ( 1 − c 2 ) . Since Z 1 Z_1 Z 1 is independent of W W W and P ( ∣ Z 1 ∣ > a ) ≤ 2 e − a 2 / 2 \mathbb{P}(\lvert Z_1\rvert> a) \le2e^{-a^2/2} P (∣ Z 1 ∣ > a ) ≤ 2 e − a 2 /2 for a ≥ 0 a \ge0 a ≥ 0 ,
P ( ∣ u 1 ∣ > c ) ≤ 2 E exp ( − c 2 W 2 ( 1 − c 2 ) ) = 2 ( 1 − c 2 ) ( n − 1 ) / 2 ≤ 2 exp ( − ( n − 1 ) c 2 2 ) , \mathbb{P}(\lvert u_1\rvert> c) \le2\mathbb{E}\exp\left(-\frac{c^2W}{2(1-c^2)}\right) = 2(1-c^2)^{(n-1)/2} \le2\exp\left(-\frac{(n-1)c^2}{2}\right), P (∣ u 1 ∣ > c ) ≤ 2 E exp ( − 2 ( 1 − c 2 ) c 2 W ) = 2 ( 1 − c 2 ) ( n − 1 ) /2 ≤ 2 exp ( − 2 ( n − 1 ) c 2 ) , where the equality is the Laplace transform of the χ n − 1 2 \chi^2_{n-1} χ n − 1 2 distribution. It remains to use c ≥ b / r ˉ c \ge b/\bar r c ≥ b / r ˉ . The conditional version is the same computation for every fixed value of the conditioning information.
Lemma A.2 (Maximum of success counts). Let p ∈ [ 0 , 1 ] p \in[0,1] p ∈ [ 0 , 1 ] . For every w = 1 , … , m w = 1,\ldots,m w = 1 , … , m , let G ⊆ G w , 0 ⊆ G w , 1 ⊆ ⋯ ⊆ G w , K \mathcal{G} \subseteq\mathcal{G}_{w,0} \subseteq\mathcal{G}_{w,1} \subseteq\cdots\subseteq\mathcal{G}_{w,K} G ⊆ G w , 0 ⊆ G w , 1 ⊆ ⋯ ⊆ G w , K be σ \sigma σ -algebras and let I w , k ∈ { 0 , 1 } I_{w,k} \in\{0,1\} I w , k ∈ { 0 , 1 } be G w , k \mathcal{G}_{w,k} G w , k -measurable with
E [ e I w , k ∣ G w , k − 1 ] ≤ 1 − p + p e , k = 1 , … , K . \mathbb{E}\left[e^{I_{w,k}} \mid\mathcal{G}_{w,k-1}\right] \le1-p+pe,\qquad k=1,\ldots,K. E [ e I w , k ∣ G w , k − 1 ] ≤ 1 − p + p e , k = 1 , … , K . Let S w = ∑ k = 1 K I w , k S_w = \sum_{k=1}^{K} I_{w,k} S w = ∑ k = 1 K I w , k . Then
E [ e max w S w ∣ G ] ≤ m e ( e − 1 ) K p , E [ max w S w ∣ G ] ≤ log m + ( e − 1 ) K p . \mathbb{E}\left[e^{\max_w S_w}\mid\mathcal{G}\right] \le me^{(e-1)Kp},\qquad
\mathbb{E}\left[\max_w S_w\mid\mathcal{G}\right] \le\log m+(e-1)Kp. E [ e m a x w S w ∣ G ] ≤ m e ( e − 1 ) K p , E [ w max S w ∣ G ] ≤ log m + ( e − 1 ) K p . In particular, this holds for independent S 1 , … , S m ∼ Bin ( K , p ) S_1,\ldots,S_m \sim\operatorname{Bin}(K,p) S 1 , … , S m ∼ Bin ( K , p ) .
Proof. Conditioning successively on G w , K − 1 , … , G w , 0 \mathcal{G}_{w,K-1},\ldots,\mathcal{G}_{w,0} G w , K − 1 , … , G w , 0 and using 1 − p + p e ≤ e ( e − 1 ) p 1-p+pe \le e^{(e-1)p} 1 − p + p e ≤ e ( e − 1 ) p gives E [ e S w ∣ G w , 0 ] ≤ e ( e − 1 ) K p \mathbb{E}[e^{S_w}\mid\mathcal{G}_{w,0}] \le e^{(e-1)Kp} E [ e S w ∣ G w , 0 ] ≤ e ( e − 1 ) K p , and the same bound holds given G \mathcal{G} G . Since e max w S w ≤ ∑ w e S w e^{\max_w S_w} \le\sum_w e^{S_w} e m a x w S w ≤ ∑ w e S w , the first inequality follows. The second follows from the first by Jensen’s inequality, E [ max w S w ∣ G ] ≤ log E [ e max w S w ∣ G ] \mathbb{E}[\max_w S_w\mid\mathcal{G}] \le\log\mathbb{E}[e^{\max_w S_w}\mid\mathcal{G}] E [ max w S w ∣ G ] ≤ log E [ e m a x w S w ∣ G ] .
The next lemma turns a convex function with bounded gradients into a globally convex function whose values and gradients far from the origin do not depend on it. Let
Γ 0 ( t ) = { 0 , t ≤ 0 , 3 t 2 − 2 t 3 , 0 < t < 1 , 1 , t ≥ 1. \Gamma_0(t) =
\begin{cases}
0, & t \le0,\\
3t^2-2t^3, & 0 < t < 1,\\
1, & t \ge1.
\end{cases} Γ 0 ( t ) = ⎩ ⎨ ⎧ 0 , 3 t 2 − 2 t 3 , 1 , t ≤ 0 , 0 < t < 1 , t ≥ 1.
Lemma A.3 (Radial extension). Let φ : R d → [ 0 , ∞ ) \varphi:\mathbb{R}^{d}\to[0,\infty) φ : R d → [ 0 , ∞ ) be convex and differentiable, with an L φ L_{\varphi} L φ -Lipschitz gradient and ∥ ∇ φ ∥ ≤ M φ \lVert\nabla\varphi\rVert\le M_{\varphi} ∥ ∇ φ ∥ ≤ M φ everywhere, and let φ ( x ∗ ) = 0 \varphi(x^{*})=0 φ ( x ∗ ) = 0 for some ∥ x ∗ ∥ ≤ R \lVert x^{*}\rVert\le R ∥ x ∗ ∥ ≤ R . Let c > 0 c>0 c > 0 and ρ ≥ max { 16 R , 64 M φ / c } \rho\ge\max\{16R,64M_{\varphi}/c\} ρ ≥ max { 16 R , 64 M φ / c } , and define
Γ ( x ) = Γ 0 ( ∥ x ∥ − 2 R ρ ) , T φ ( x ) = ( 1 − Γ ( x ) ) φ ( x ) + 2 c ( ∥ x ∥ − R ) + 2 . (5) \Gamma(x)=\Gamma_{0}\left(\frac{\lVert x\rVert-2R}{\rho}\right),\qquad\mathcal{T}\varphi(x)=(1-\Gamma(x))\varphi(x)+2c(\lVert x\rVert-R)_{+}^{2}. \tag*{(5)} Γ ( x ) = Γ 0 ( ρ ∥ x ∥ − 2 R ) , T φ ( x ) = ( 1 − Γ ( x )) φ ( x ) + 2 c (∥ x ∥ − R ) + 2 . (5) Then:
T φ \mathcal{T}\varphi T φ is convex and has an ( L φ + 5 c ) (L_{\varphi}+5c) ( L φ + 5 c ) -Lipschitz gradient;
φ ≤ T φ \varphi\le\mathcal{T}\varphi φ ≤ T φ and T φ ( x ∗ ) = 0 \mathcal{T}\varphi(x^{*})=0 T φ ( x ∗ ) = 0 ;
∥ ∇ Γ ∥ ≤ 3 2 ρ \lVert\nabla\Gamma\rVert\le\frac{3}{2\rho} ∥ ∇Γ ∥ ≤ 2 ρ 3 and
∇ T φ = ( 1 − Γ ) ∇ φ − φ ∇ Γ + 4 c ( ∥ x ∥ − R ) + x ∥ x ∥ , \nabla\mathcal{T}\varphi=(1-\Gamma)\nabla\varphi-\varphi\nabla\Gamma+4c(\lVert x\rVert-R)_{+}\frac{x}{\lVert x\rVert}, ∇ T φ = ( 1 − Γ ) ∇ φ − φ ∇Γ + 4 c (∥ x ∥ − R ) + ∥ x ∥ x , with the last term equal to zero at x = 0 x=0 x = 0 ;
if ∥ x ∥ ≥ 2 R + ρ \lVert x\rVert\ge2R+\rho ∥ x ∥ ≥ 2 R + ρ , then T φ ( x ) = 2 c ( ∥ x ∥ − R ) 2 \mathcal{T}\varphi(x)=2c(\lVert x\rVert-R)^{2} T φ ( x ) = 2 c (∥ x ∥ − R ) 2 and ∇ T φ ( x ) = 4 c ( ∥ x ∥ − R ) x ∥ x ∥ \nabla\mathcal{T}\varphi(x)=4c(\lVert x\rVert-R)\frac{x}{\lVert x\rVert} ∇ T φ ( x ) = 4 c (∥ x ∥ − R ) ∥ x ∥ x ; these do not depend on φ \varphi φ .
Proof. Write r = ∥ x ∥ r=\lVert x\rVert r = ∥ x ∥ and π ( x ) = 2 c ( r − R ) + 2 \pi(x)=2c(r-R)_{+}^{2} π ( x ) = 2 c ( r − R ) + 2 . The function π \pi π is convex, being a nondecreasing convex function of the norm, and for r > R r>R r > R
∇ π ( x ) = 4 c ( r − R ) x r , ∇ 2 π ( x ) = 4 c x x T r 2 + 4 c ( 1 − R r ) ( I − x x T r 2 ) . \nabla\pi(x)=4c(r-R)\frac{x}{r},\qquad\nabla^{2}\pi(x)=4c\frac{xx^{\mathsf{T}}}{r^{2}}+4c\left(1-\frac{R}{r}\right)\left(I-\frac{xx^{\mathsf{T}}}{r^{2}}\right). ∇ π ( x ) = 4 c ( r − R ) r x , ∇ 2 π ( x ) = 4 c r 2 x x T + 4 c ( 1 − r R ) ( I − r 2 x x T ) .
Hence the eigenvalues of ∇ 2 π \nabla^{2}\pi ∇ 2 π lie in [ 0 , 4 c ] [0,4c] [ 0 , 4 c ] , and they are at least 2 c 2c 2 c when r ≥ 2 R r\ge2R r ≥ 2 R .
Bounds on the cutoff. On the annulus 2 R < r < 2 R + ρ 2R<r<2R+\rho 2 R < r < 2 R + ρ put t = ( r − 2 R ) / ρ t=(r-2R)/\rho t = ( r − 2 R ) / ρ . Then
∇ Γ ( x ) = Γ 0 ′ ( t ) ρ x r , ∇ 2 Γ ( x ) = Γ 0 ′ ′ ( t ) ρ 2 x x T r 2 + Γ 0 ′ ( t ) ρ r ( I − x x T r 2 ) . \nabla\Gamma(x)=\frac{\Gamma_{0}'(t)}{\rho}\frac{x}{r},\qquad\nabla^{2}\Gamma(x)=\frac{\Gamma_{0}''(t)}{\rho^{2}}\frac{xx^{\mathsf{T}}}{r^{2}}+\frac{\Gamma_{0}'(t)}{\rho r}\left(I-\frac{xx^{\mathsf{T}}}{r^{2}}\right). ∇Γ ( x ) = ρ Γ 0 ′ ( t ) r x , ∇ 2 Γ ( x ) = ρ 2 Γ 0 ′′ ( t ) r 2 x x T + ρ r Γ 0 ′ ( t ) ( I − r 2 x x T ) .
Since 0 ≤ Γ 0 ′ ( t ) = 6 t ( 1 − t ) ≤ min { 3 / 2 , 6 t } 0\le\Gamma_{0}'(t)=6t(1-t)\le\min\{3/2,6t\} 0 ≤ Γ 0 ′ ( t ) = 6 t ( 1 − t ) ≤ min { 3/2 , 6 t } and ∣ Γ 0 ′ ′ ( t ) ∣ ≤ 6 \lvert\Gamma_{0}''(t)\rvert\le6 ∣ Γ 0 ′′ ( t )∣ ≤ 6 , and since Γ 0 ′ ( t ) ρ r ≤ 6 ( r − 2 R ) ρ 2 r ≤ 6 ρ 2 \frac{\Gamma_{0}'(t)}{\rho r}\le\frac{6(r-2R)}{\rho^{2}r}\le\frac{6}{\rho^{2}} ρ r Γ 0 ′ ( t ) ≤ ρ 2 r 6 ( r − 2 R ) ≤ ρ 2 6 , we get
∥ ∇ Γ ∥ ≤ 3 2 ρ , ∥ ∇ 2 Γ ∥ o p ≤ 6 ρ 2 . \lVert\nabla\Gamma\rVert\le\frac{3}{2\rho},\qquad\lVert\nabla^{2}\Gamma\rVert_{\mathrm{op}}\le\frac{6}{\rho^{2}}. ∥ ∇Γ ∥ ≤ 2 ρ 3 , ∥ ∇ 2 Γ ∥ op ≤ ρ 2 6 .
Outside the annulus Γ \Gamma Γ is constant. This proves the bound in Part 3; the formula for ∇ T φ \nabla\mathcal{T}\varphi ∇ T φ is the product rule.
Hessian bounds. The gradient of φ \varphi φ is Lipschitz, hence differentiable almost everywhere, and at every such point 0 ⪯ ∇ 2 φ ⪯ L φ I 0\preceq\nabla^{2}\varphi\preceq L_{\varphi}I 0 ⪯ ∇ 2 φ ⪯ L φ I by convexity. Since φ ( x ∗ ) = 0 \varphi(x^{*})=0 φ ( x ∗ ) = 0 and ∥ ∇ φ ∥ ≤ M φ \lVert\nabla\varphi\rVert\le M_{\varphi} ∥ ∇ φ ∥ ≤ M φ , on the annulus φ ( x ) ≤ M φ ∥ x − x ∗ ∥ ≤ M φ ( 3 R + ρ ) \varphi(x)\le M_{\varphi}\lVert x-x^{*}\rVert\le M_{\varphi}(3R+\rho) φ ( x ) ≤ M φ ∥ x − x ∗ ∥ ≤ M φ ( 3 R + ρ ) . The product rule gives, almost everywhere on the annulus,
∇ 2 [ ( 1 − Γ ) φ ] = ( 1 − Γ ) ∇ 2 φ − ∇ Γ ∇ φ T − ∇ φ ∇ Γ T − φ ∇ 2 Γ , \nabla^{2}\left[(1-\Gamma)\varphi\right]=(1-\Gamma)\nabla^{2}\varphi-\nabla\Gamma\nabla\varphi^{\mathsf{T}}-\nabla\varphi\nabla\Gamma^{\mathsf{T}}-\varphi\nabla^{2}\Gamma, ∇ 2 [ ( 1 − Γ ) φ ] = ( 1 − Γ ) ∇ 2 φ − ∇Γ∇ φ T − ∇ φ ∇ Γ T − φ ∇ 2 Γ ,
and the last three terms have operator norm at most
2 ⋅ 3 2 ρ M φ + M φ ( 3 R + ρ ) 6 ρ 2 = ( 9 + 18 R ρ ) M φ ρ ≤ 81 8 ⋅ M φ 64 ≤ c 4 , 2\cdot\frac{3}{2\rho}M_{\varphi}+M_{\varphi}(3R+\rho)\frac{6}{\rho^{2}}=\left(9+\frac{18R}{\rho}\right)\frac{M_{\varphi}}{\rho}\le\frac{81}{8}\cdot\frac{M_{\varphi}}{64}\le\frac{c}{4}, 2 ⋅ 2 ρ 3 M φ + M φ ( 3 R + ρ ) ρ 2 6 = ( 9 + ρ 18 R ) ρ M φ ≤ 8 81 ⋅ 64 M φ ≤ 4 c ,
where we used ρ ≥ 16 R \rho\ge16R ρ ≥ 16 R and ρ ≥ 64 M φ / c \rho\ge64M_{\varphi}/c ρ ≥ 64 M φ / c . Combining this with the eigenvalue bounds for ∇ 2 π \nabla^{2}\pi ∇ 2 π , we obtain almost everywhere: for r < 2 R r<2R r < 2 R , where Γ = 0 \Gamma=0 Γ = 0 , 0 ⪯ ∇ 2 T φ ⪯ ( L φ + 4 c ) I 0\preceq\nabla^{2}\mathcal{T}\varphi\preceq(L_{\varphi}+4c)I 0 ⪯ ∇ 2 T φ ⪯ ( L φ + 4 c ) I ; on the annulus, ( 2 c − c 4 ) I ⪯ ∇ 2 T φ ⪯ ( L φ + c 4 + 4 c ) I (2c-\frac{c}{4})I\preceq\nabla^{2}\mathcal{T}\varphi\preceq(L_{\varphi}+\frac{c}{4}+4c)I ( 2 c − 4 c ) I ⪯ ∇ 2 T φ ⪯ ( L φ + 4 c + 4 c ) I ; for r > 2 R + ρ r>2R+\rho r > 2 R + ρ , where Γ = 1 \Gamma=1 Γ = 1 , 0 ⪯ ∇ 2 T φ ⪯ 4 c I 0\preceq\nabla^{2}\mathcal{T}\varphi\preceq4cI 0 ⪯ ∇ 2 T φ ⪯ 4 c I . Thus
0 ⪯ ∇ 2 T φ ⪯ ( L φ + 5 c ) I almost everywhere . (6) 0\preceq\nabla^{2}\mathcal{T}\varphi\preceq(L_{\varphi}+5c)I\qquad\text{almost everywhere}. \tag*{(6)} 0 ⪯ ∇ 2 T φ ⪯ ( L φ + 5 c ) I almost everywhere . (6)
Part 1. The function ψ = T φ \psi=\mathcal{T}\varphi ψ = T φ is continuously differentiable, and ∇ ψ \nabla\psi ∇ ψ is locally Lipschitz. Hence the distributional derivative of ∇ ψ \nabla\psi ∇ ψ is its almost everywhere derivative. For a standard mollifier κ ε \kappa_{\varepsilon} κ ε , the function ψ ε = ψ ∗ κ ε \psi_{\varepsilon}=\psi*\kappa_{\varepsilon} ψ ε = ψ ∗ κ ε is smooth with ∇ 2 ψ ε = ( ∇ 2 ψ ) ∗ κ ε \nabla^{2}\psi_{\varepsilon}=(\nabla^{2}\psi)*\kappa_{\varepsilon} ∇ 2 ψ ε = ( ∇ 2 ψ ) ∗ κ ε , so (A.2) gives 0 ⪯ ∇ 2 ψ ε ⪯ ( L φ + 5 c ) I 0\preceq\nabla^{2}\psi_{\varepsilon}\preceq(L_{\varphi}+5c)I 0 ⪯ ∇ 2 ψ ε ⪯ ( L φ + 5 c ) I everywhere. Thus ψ ε \psi_{\varepsilon} ψ ε is convex with an ( L φ + 5 c ) (L_{\varphi}+5c) ( L φ + 5 c ) -Lipschitz gradient. As ε → 0 \varepsilon\to0 ε → 0 , ψ ε → ψ \psi_{\varepsilon}\to\psi ψ ε → ψ and ∇ ψ ε → ∇ ψ \nabla\psi_{\varepsilon}\to\nabla\psi ∇ ψ ε → ∇ ψ pointwise, and both properties pass to the limit.
Part 2. We show Γ φ ≤ π \Gamma\varphi\le\pi Γ φ ≤ π . For r ≤ 2 R r\le2R r ≤ 2 R this holds since Γ = 0 \Gamma=0 Γ = 0 . On the annulus, Γ 0 ( t ) ≤ 3 t 2 \Gamma_{0}(t)\le3t^{2} Γ 0 ( t ) ≤ 3 t 2 and t ≤ ( r − R ) / ρ t\le(r-R)/\rho t ≤ ( r − R ) / ρ give Γ ≤ 3 ( r − R ) 2 / ρ 2 \Gamma\le3(r-R)^{2}/\rho^{2} Γ ≤ 3 ( r − R ) 2 / ρ 2 , hence
Γ φ ≤ ( 3 + 9 R ρ ) M φ ρ ( r − R ) 2 ≤ 57 16 ⋅ c 64 ( r − R ) 2 ≤ 2 c ( r − R ) 2 . \Gamma\varphi\le\left(3+\frac{9R}{\rho}\right)\frac{M_{\varphi}}{\rho}(r-R)^{2}\le\frac{57}{16}\cdot\frac{c}{64}(r-R)^{2}\le2c(r-R)^{2}. Γ φ ≤ ( 3 + ρ 9 R ) ρ M φ ( r − R ) 2 ≤ 16 57 ⋅ 64 c ( r − R ) 2 ≤ 2 c ( r − R ) 2 .
For r ≥ 2 R + ρ r\ge2R+\rho r ≥ 2 R + ρ , we have r − R ≥ r / 2 r-R\ge r/2 r − R ≥ r /2 , r + R ≤ 3 r / 2 r+R\le3r/2 r + R ≤ 3 r /2 , and r ≥ 64 M φ / c r\ge64M_{\varphi}/c r ≥ 64 M φ / c , so φ ( x ) ≤ M φ ( r + R ) ≤ 3 2 M φ r ≤ c 2 r 2 ≤ π ( x ) \varphi(x)\le M_{\varphi}(r+R)\le\frac{3}{2}M_{\varphi}r\le\frac{c}{2}r^{2}\le\pi(x) φ ( x ) ≤ M φ ( r + R ) ≤ 2 3 M φ r ≤ 2 c r 2 ≤ π ( x ) . Therefore T φ − φ = π − Γ φ ≥ 0 \mathcal{T}\varphi-\varphi=\pi-\Gamma\varphi\ge0 T φ − φ = π − Γ φ ≥ 0 . Since ∥ x ∗ ∥ ≤ R \lVert x^{*}\rVert\le R ∥ x ∗ ∥ ≤ R , both Γ \Gamma Γ and π \pi π vanish at x ∗ x^{*} x ∗ , and T φ ( x ∗ ) = φ ( x ∗ ) = 0 \mathcal{T}\varphi(x^{*})=\varphi(x^{*})=0 T φ ( x ∗ ) = φ ( x ∗ ) = 0 .
Part 4. This follows from Γ = 1 \Gamma=1 Γ = 1 for r ≥ 2 R + ρ r\ge2R+\rho r ≥ 2 R + ρ .
The map T \mathcal{T} T is affine in φ \varphi φ : if a measurable family φ ζ \varphi_{\zeta} φ ζ satisfies the hypotheses of Lemma A.3 with common x ∗ x^{*} x ∗ , L φ L_{\varphi} L φ , M φ M_{\varphi} M φ , then so does E φ ζ \mathbb{E}\varphi_{\zeta} E φ ζ , and E T φ ζ = T ( E φ ζ ) \mathbb{E}\,\mathcal{T}\varphi_{\zeta}=\mathcal{T}(\mathbb{E}\varphi_{\zeta}) E T φ ζ = T ( E φ ζ ) .
The statistical term
Lemma A.4 (Statistical lower bound). Let S ∈ N ∪ { ∞ } S \in\mathbb{N} \cup\{\infty\} S ∈ N ∪ { ∞ } , S ≥ 1 S \ge1 S ≥ 1 , and d ≥ 1 d \ge1 d ≥ 1 . For every algorithm in IC ( m , K , N ) \operatorname{IC}(m,K,N) IC ( m , K , N ) with access to a pool of size S S S there is an instance of S d q u a d \mathcal{S}^{\mathrm{quad}}_d S d quad such that
E F ( x ^ ) ≥ R 2 8 min { L , H s min { S , m K N } } . (7) \mathbb{E}F(\widehat{x}) \ge\frac{R^2}{8}\min\left\{L,\frac{H_s}{\min\{S,mKN\}}\right\}. \tag*{(7)} E F ( x ) ≥ 8 R 2 min { L , min { S , m K N } H s } . (7)
Consequently, M d p o o l ( S ) \mathcal{M}^{\mathrm{pool}}_d(S) M d pool ( S ) is at least the right-hand side of (A.3), and each of M d s g , q u a d \mathcal{M}^{\mathrm{sg,quad}}_d M d sg , quad , M d s g \mathcal{M}^{\mathrm{sg}}_d M d sg , M d w g , q u a d \mathcal{M}^{\mathrm{wg,quad}}_d M d wg , quad , M d w g \mathcal{M}^{\mathrm{wg}}_d M d wg is at least R 2 8 min { L , H s m K N } \frac{R^2}{8}\min\left\{L,\frac{H_s}{mKN}\right\} 8 R 2 min { L , m K N H s } .
Proof. Put n S = min { S , m K N } n_S=\min\{S,mKN\} n S = min { S , m K N } and p 0 = min { L / H s , 1 / ( 2 n S ) } p_0=\min\{L/H_s,1/(2n_S)\} p 0 = min { L / H s , 1/ ( 2 n S )} . Let θ ∈ { − 1 , 1 } \theta\in\{-1,1\} θ ∈ { − 1 , 1 } be a uniform sign, independent of the seed, and consider the sample functions
f ξ ( x ) = ξ H s 2 ( x 1 − θ R ) 2 , ξ ∼ Bernoulli ( p 0 ) . f_{\xi}(x)=\frac{\xi H_s}{2}(x_1-\theta R)^2,\qquad\xi\sim\operatorname{Bernoulli}(p_0). f ξ ( x ) = 2 ξ H s ( x 1 − θ R ) 2 , ξ ∼ Bernoulli ( p 0 ) .
They are convex, H s H_s H s -smooth, and minimized at x ∗ = θ R e 1 x^*=\theta R e_1 x ∗ = θ R e 1 , and F θ ( x ) = p 0 H s 2 ( x 1 − θ R ) 2 F_{\theta}(x)=\frac{p_0H_s}{2}(x_1-\theta R)^2 F θ ( x ) = 2 p 0 H s ( x 1 − θ R ) 2 is a quadratic with smoothness p 0 H s ≤ L p_0H_s\le L p 0 H s ≤ L . Hence, for each θ \theta θ , this is an instance of S d q u a d \mathcal{S}^{\mathrm{quad}}_d S d quad .
Consider the zero run: the run of the algorithm in which every query with a valid label receives the reply 0. It is a function of the seed only; let X ^ \widehat{X} X be its output and I \mathcal{I} I the set of valid labels it queries. Since there are at most m K N mKN m K N queries and at most S S S valid labels, ∣ I ∣ ≤ n S |\mathcal{I}|\le n_S ∣ I ∣ ≤ n S . On the event E 0 = { ξ i = 0 for all i ∈ I } E_0=\{\xi_i=0\text{ for all }i\in\mathcal{I}\} E 0 = { ξ i = 0 for all i ∈ I } , induction over the queries shows that the actual run coincides with the zero run: every query is the same, and its reply is 0. Since I \mathcal{I} I depends only on the seed, which is independent of the pool and of θ \theta θ ,
P ( E 0 ∣ s e e d , θ ) = ( 1 − p 0 ) ∣ I ∣ ≥ 1 − n S p 0 ≥ 1 2 . \mathbb{P}(E_0\mid\mathrm{seed},\theta)=(1-p_0)^{|\mathcal{I}|}\ge1-n_Sp_0\ge\frac{1}{2}. P ( E 0 ∣ seed , θ ) = ( 1 − p 0 ) ∣ I ∣ ≥ 1 − n S p 0 ≥ 2 1 .
Therefore, using x ^ = X ^ \widehat{x}=\widehat{X} x = X on E 0 E_0 E 0 ,
E F θ ( x ^ ) ≥ E [ F θ ( X ^ ) P ( E 0 ∣ s e e d , θ ) ] ≥ 1 2 E F θ ( X ^ ) . \mathbb{E}F_{\theta}(\widehat{x})\ge\mathbb{E}\left[F_{\theta}(\widehat{X})\mathbb{P}(E_0\mid\mathrm{seed},\theta)\right]\ge\frac{1}{2}\mathbb{E}F_{\theta}(\widehat{X}). E F θ ( x ) ≥ E [ F θ ( X ) P ( E 0 ∣ seed , θ ) ] ≥ 2 1 E F θ ( X ) .
Since X ^ \widehat{X} X is independent of θ \theta θ ,
E θ F θ ( X ^ ) = p 0 H s 4 [ ( X ^ 1 − R ) 2 + ( X ^ 1 + R ) 2 ] = p 0 H s 2 ( X ^ 1 2 + R 2 ) ≥ p 0 H s R 2 2 . \mathbb{E}_{\theta}F_{\theta}(\widehat{X})
=\frac{p_0H_s}{4}\left[(\widehat{X}_1-R)^2+(\widehat{X}_1+R)^2\right]
=\frac{p_0H_s}{2}\left(\widehat{X}_1^2+R^2\right)
\ge\frac{p_0H_sR^2}{2}. E θ F θ ( X ) = 4 p 0 H s [ ( X 1 − R ) 2 + ( X 1 + R ) 2 ] = 2 p 0 H s ( X 1 2 + R 2 ) ≥ 2 p 0 H s R 2 .
Averaged over θ \theta θ , the risk is at least p 0 H s R 2 4 = R 2 4 min { L , H s 2 n S } \frac{p_0H_sR^2}{4}=\frac{R^2}{4}\min\left\{L,\frac{H_s}{2n_S}\right\} 4 p 0 H s R 2 = 4 R 2 min { L , 2 n S H s } , which is at least the right-hand side of (A.3). One of the two signs attains the average.
Fresh access is the case S = ∞ S=\infty S = ∞ with distinct labels, see (3) , for which n S = m K N n_S=mKN n S = m K N . Since the hard instance belongs to S d q u a d \mathcal{S}^{\mathrm{quad}}_d S d quad , the bound holds for M d s g , q u a d \mathcal{M}^{\mathrm{sg,quad}}_d M d sg , quad , and Lemma 4.1 together with S d q u a d ⊆ S d \mathcal{S}^{\mathrm{quad}}_d\subseteq\mathcal{S}_d S d quad ⊆ S d and W d q u a d ⊆ W d \mathcal{W}^{\mathrm{quad}}_d\subseteq\mathcal{W}_d W d quad ⊆ W d transfers it to the other three minimax risks. ∎
Quadratic populations
The construction first gives one law of convex quadratics that is hard for every algorithm with a bounded number of batches of exact gradient queries, and then masks whole sample functions by Bernoulli variables to convert the number of successful local queries into that number of batches. No linear-span assumption is used.
Adaptive matrix queries.
Lemma A.5 (Batched Haar representation). Let Λ ∈ R d × d \Lambda\in\mathbb{R}^{d\times d} Λ ∈ R d × d be symmetric and b 0 ∈ R d b_0\in\mathbb{R}^d b 0 ∈ R d . Let U U U be Haar distributed on the orthogonal group O ( d ) O(d) O ( d ) , and put A = U Λ U T A=U\Lambda U^{\mathsf{T}} A = U Λ U T and b = U b 0 b=Ub_0 b = U b 0 . Consider a procedure that knows Λ \Lambda Λ and b 0 b_0 b 0 , receives b b b , then in D D D adaptive batches submits at most m m m vectors per batch and receives their images under A A A , and finally outputs a vector z z z ; its seed is independent of U U U and all its maps are measurable. Then on one probability space one can construct U U U , the seed, the transcript of the procedure, and independent standard Gaussian vectors g r , i ∈ R d g_{r,i}\in\mathbb{R}^d g r , i ∈ R d , 1 ≤ r ≤ D 1\le r\le D 1 ≤ r ≤ D , 1 ≤ i ≤ m 1\le i\le m 1 ≤ i ≤ m , and g 0 ∈ R d g_0\in\mathbb{R}^d g 0 ∈ R d , such that ( U , s e e d , t r a n s c r i p t ) (U,\mathrm{seed},\mathrm{transcript}) ( U , seed , transcript ) has the same law as in the original experiment, the Gaussian vectors are independent of the seed, and
U T z ∈ span { Λ j b 0 : 0 ≤ j ≤ D } + ∑ r = 1 D ∑ i = 1 m span { Λ j g r , i : 0 ≤ j ≤ D − r + 1 } + span { g 0 } . (8) U^{\mathsf{T}}z\in\operatorname{span}\{\Lambda^j b_0:0\le j\le D\}
+\sum_{r=1}^{D}\sum_{i=1}^{m}\operatorname{span}\{\Lambda^j g_{r,i}:0\le j\le D-r+1\}
+\operatorname{span}\{g_0\}. \tag*{(8)} U T z ∈ span { Λ j b 0 : 0 ≤ j ≤ D } + r = 1 ∑ D i = 1 ∑ m span { Λ j g r , i : 0 ≤ j ≤ D − r + 1 } + span { g 0 } . (8)
Proof. Uniform extensions. Let E ⊆ R d E \subseteq\mathbb{R}^{d} E ⊆ R d be a subspace and W : E → R d W:E \to\mathbb{R}^{d} W : E → R d a linear isometry, and let O O O be uniformly distributed on the set O ( W ) \mathcal{O}(W) O ( W ) of orthogonal maps that coincide with W W W on E E E ; equivalently, the restriction of O O O to E ⊥ E^{\perp} E ⊥ is a Haar distributed isometry of E ⊥ E^{\perp} E ⊥ onto ( W E ) ⊥ (WE)^{\perp} ( W E ) ⊥ . Invariance of this Haar measure under rotations of E ⊥ E^{\perp} E ⊥ and of ( W E ) ⊥ (WE)^{\perp} ( W E ) ⊥ gives the following two facts.
(i) If w 1 , … , w k w_{1},\ldots,w_{k} w 1 , … , w k is an orthonormal family in ( W E ) ⊥ (WE)^{\perp} ( W E ) ⊥ , then ( O T w 1 , … , O T w k ) (O^{\mathsf{T}}w_{1},\ldots,O^{\mathsf{T}}w_{k}) ( O T w 1 , … , O T w k ) is a uniformly distributed orthonormal k k k -frame in E ⊥ E^{\perp} E ⊥ , and conditional on this frame O O O is uniform on the set of orthogonal maps that extend W W W and map the frame to ( w 1 , … , w k ) (w_{1},\ldots,w_{k}) ( w 1 , … , w k ) . The same frame law is obtained by the Gram–Schmidt orthonormalization of Π E ⊥ γ 1 , … , Π E ⊥ γ k \Pi_{E^{\perp}}\gamma_{1},\ldots,\Pi_{E^{\perp}}\gamma_{k} Π E ⊥ γ 1 , … , Π E ⊥ γ k for independent standard Gaussian vectors γ l ∈ R d \gamma_{l}\in\mathbb{R}^{d} γ l ∈ R d ; every vector of that frame lies in E + span { γ 1 , … , γ k } E+\operatorname{span}\{\gamma_{1},\ldots,\gamma_{k}\} E + span { γ 1 , … , γ k } .
(ii) If e 1 , … , e k e_{1},\ldots,e_{k} e 1 , … , e k is an orthonormal family in E ⊥ E^{\perp} E ⊥ , then ( O e 1 , … , O e k ) (Oe_{1},\ldots,Oe_{k}) ( O e 1 , … , O e k ) is a uniformly distributed orthonormal k k k -frame in ( W E ) ⊥ (WE)^{\perp} ( W E ) ⊥ , and conditional on it O O O is uniform on the set of orthogonal maps that extend W W W and map ( e l ) (e_{l}) ( e l ) to this frame.
Construction. Fix the seed. We reveal U U U progressively. At every stage there is an exposed subspace E E E and the isometry W = U ∣ E W=U|_{E} W = U ∣ E , and the following invariant holds: conditional on everything constructed so far, U U U is uniform on O ( W ) \mathcal{O}(W) O ( W ) .
Initially, draw b b b uniformly on the sphere of radius ∥ b 0 ∥ \lVert b_{0}\rVert ∥ b 0 ∥ and expose E 0 = span { b 0 } E_{0}=\operatorname{span}\{b_{0}\} E 0 = span { b 0 } with U b 0 = b Ub_{0}=b U b 0 = b . This is the law of U b 0 Ub_{0} U b 0 , and conditional on U b 0 = b Ub_{0}=b U b 0 = b the map U U U is uniform on O ( W ) \mathcal{O}(W) O ( W ) .
Consider batch r ∈ { 1 , … , D } r\in\{1,\ldots,D\} r ∈ { 1 , … , D } , and let E r − 1 E_{r-1} E r − 1 be the exposed subspace before it. The queries v 1 , … , v m v_{1},\ldots,v_{m} v 1 , … , v m (padded by zeros) are functions of the seed and of the previous replies. Let w 1 , … , w k w_{1},\ldots,w_{k} w 1 , … , w k , k ≤ m k\leq m k ≤ m , be an orthonormal basis, chosen by a fixed measurable rule, of the orthogonal projection of span { v 1 , … , v m } \operatorname{span}\{v_{1},\ldots,v_{m}\} span { v 1 , … , v m } onto ( U E r − 1 ) ⊥ (UE_{r-1})^{\perp} ( U E r − 1 ) ⊥ . Draw standard Gaussian vectors g r , 1 , … , g r , m g_{r,1},\ldots,g_{r,m} g r , 1 , … , g r , m independently of everything constructed so far, let q 1 , … , q k q_{1},\ldots,q_{k} q 1 , … , q k be the Gram–Schmidt orthonormalization of Π E r − 1 ⊥ g r , 1 , … , Π E r − 1 ⊥ g r , k \Pi_{E_{r-1}^{\perp}}g_{r,1},\ldots,\Pi_{E_{r-1}^{\perp}}g_{r,k} Π E r − 1 ⊥ g r , 1 , … , Π E r − 1 ⊥ g r , k , and expose U q l = w l Uq_{l}=w_{l} U q l = w l . By (i), this has the correct conditional law and preserves the invariant with E ′ = E r − 1 + span { q 1 , … , q k } E'=E_{r-1}+\operatorname{span}\{q_{1},\ldots,q_{k}\} E ′ = E r − 1 + span { q 1 , … , q k } . Every v i v_{i} v i lies in U E ′ UE' U E ′ , so U T v i ∈ E ′ ⊆ E r − 1 + span { g r , 1 , … , g r , m } U^{\mathsf{T}}v_{i}\in E'\subseteq E_{r-1}+\operatorname{span}\{g_{r,1},\ldots,g_{r,m}\} U T v i ∈ E ′ ⊆ E r − 1 + span { g r , 1 , … , g r , m } . The replies are A v i = U ( Λ U T v i ) Av_{i}=U(\Lambda U^{\mathsf{T}}v_{i}) A v i = U ( Λ U T v i ) with Λ U T v i ∈ Λ E ′ \Lambda U^{\mathsf{T}}v_{i}\in\Lambda E' Λ U T v i ∈ Λ E ′ . Let e 1 , … , e k ′ e_{1},\ldots,e_{k'} e 1 , … , e k ′ be an orthonormal basis of the projection of Λ E ′ \Lambda E' Λ E ′ onto E ′ ⊥ E'^{\perp} E ′ ⊥ , and expose U e 1 , … , U e k ′ Ue_{1},\ldots,Ue_{k'} U e 1 , … , U e k ′ by (ii), using auxiliary randomness independent of everything else. The exposed subspace becomes E r = E ′ + Λ E ′ E_{r}=E'+\Lambda E' E r = E ′ + Λ E ′ , the isometry on E r E_{r} E r determines all replies of the batch, and
E r ⊆ E r − 1 + Λ E r − 1 + span { g r , i , Λ g r , i : 1 ≤ i ≤ m } . (9) E_{r}\subseteq E_{r-1}+\Lambda E_{r-1}+\operatorname{span}\{g_{r,i},\Lambda g_{r,i}:1\leq i\leq m\}. \tag*{(9)} E r ⊆ E r − 1 + Λ E r − 1 + span { g r , i , Λ g r , i : 1 ≤ i ≤ m } . (9)
After the last batch, the output z z z is a function of the seed and of all replies. Draw g 0 g_{0} g 0 independently of everything. If the projection of z z z onto ( U E D ) ⊥ (UE_{D})^{\perp} ( U E D ) ⊥ is nonzero, let w w w be its normalization, let q q q be the normalization of Π E D ⊥ g 0 \Pi_{E_{D}^{\perp}}g_{0} Π E D ⊥ g 0 , and expose U q = w Uq=w U q = w by (i). In both cases U T z ∈ E D + span { g 0 } U^{\mathsf{T}}z\in E_{D}+\operatorname{span}\{g_{0}\} U T z ∈ E D + span { g 0 } . Finally, draw U U U from the uniform law on O ( W ) \mathcal{O}(W) O ( W ) for the final exposed isometry W W W . On null events where a Gram–Schmidt step degenerates, any fixed measurable convention may be used.
Verification. By the invariant, every exposed value has the conditional law prescribed by the original experiment given the previous ones, and the final completion has the correct conditional law. Hence, for every fixed seed, ( U , transcript ) (U,\text{transcript}) ( U , transcript ) has the original joint law; in particular U U U is Haar distributed and independent of the seed. Each Gaussian vector is drawn independently of everything constructed before it, so the Gaussian vectors are independent standard Gaussian vectors, independent of the seed. Finally, (A.5) and E 0 = span { b 0 } E_{0}=\operatorname{span}\{b_{0}\} E 0 = span { b 0 } give by induction
E r ⊆ span { Λ j b 0 : 0 ≤ j ≤ r } + ∑ r ′ = 1 r ∑ i = 1 m span { Λ j g r ′ , i : 0 ≤ j ≤ r − r ′ + 1 } . E_{r}\subseteq\operatorname{span}\{\Lambda^{j}b_{0}:0\leq j\leq r\}+\sum_{r'=1}^{r}\sum_{i=1}^{m}\operatorname{span}\{\Lambda^{j}g_{r',i}:0\leq j\leq r-r'+1\}. E r ⊆ span { Λ j b 0 : 0 ≤ j ≤ r } + r ′ = 1 ∑ r i = 1 ∑ m span { Λ j g r ′ , i : 0 ≤ j ≤ r − r ′ + 1 } .
because multiplication by Λ \Lambda Λ raises every degree by one and the new terms of batch r r r have degrees 0 0 0 and 1 = r − r + 1 1=r-r+1 1 = r − r + 1 . Taking r = D r=D r = D and using U T z ∈ E D + span { g 0 } U^{\mathsf{T}}z\in E_{D}+\operatorname{span}\{g_{0}\} U T z ∈ E D + span { g 0 } proves (A.4). □ \square □
Two certificates.
Lemma A.6 (Chebyshev derivative certificate). Let n n n be odd, a > 0 a>0 a > 0 , and λ j = a cos ( j π / n ) \lambda_{j}=a\cos(j\pi/n) λ j = a cos ( j π / n ) , 0 ≤ j ≤ n 0\leq j\leq n 0 ≤ j ≤ n . There are positive weights w j w_{j} w j with ∑ j = 0 n w j = 1 \sum_{j=0}^{n}w_{j}=1 ∑ j = 0 n w j = 1 such that
∑ j = 0 n w j χ ( λ j ) 2 ≥ a 2 n 2 for every polynomial χ with deg χ ≤ n , χ ′ ( 0 ) = 1. \sum_{j=0}^{n}w_{j}\chi(\lambda_{j})^{2}\geq\frac{a^{2}}{n^{2}}\qquad\text{for every polynomial }\chi\text{ with }\deg\chi\leq n,\quad\chi'(0)=1. j = 0 ∑ n w j χ ( λ j ) 2 ≥ n 2 a 2 for every polynomial χ with deg χ ≤ n , χ ′ ( 0 ) = 1.
Proof. The nodes are distinct, strictly decreasing, nonzero because n n n is odd, and symmetric: λ n − j = − λ j \lambda_{n-j}=-\lambda_j λ n − j = − λ j . Let ℓ j \ell_j ℓ j be the Lagrange basis polynomials of degree n n n on these nodes, c j = ℓ j ′ ( 0 ) c_j=\ell_j'(0) c j = ℓ j ′ ( 0 ) , and P ( z ) = ∏ j = 0 n ( z − λ j ) P(z)=\prod_{j=0}^{n}(z-\lambda_j) P ( z ) = ∏ j = 0 n ( z − λ j ) . The polynomial P P P has n + 1 n+1 n + 1 roots symmetric about zero, so it is even and P ′ ( 0 ) = 0 P'(0)=0 P ′ ( 0 ) = 0 . Differentiating ℓ j ( z ) = P ( z ) / [ ( z − λ j ) P ′ ( λ j ) ] \ell_j(z)=P(z)/[(z-\lambda_j)P'(\lambda_j)] ℓ j ( z ) = P ( z ) / [( z − λ j ) P ′ ( λ j )] at zero gives
c j = − λ j P ′ ( 0 ) − P ( 0 ) λ j 2 P ′ ( λ j ) = − P ( 0 ) λ j 2 P ′ ( λ j ) . c_j=\frac{-\lambda_jP'(0)-P(0)}{\lambda_j^2P'(\lambda_j)}=-\frac{P(0)}{\lambda_j^2P'(\lambda_j)}. c j = λ j 2 P ′ ( λ j ) − λ j P ′ ( 0 ) − P ( 0 ) = − λ j 2 P ′ ( λ j ) P ( 0 ) .
Since the nodes decrease, P ′ ( λ j ) = ∏ i ≠ j ( λ j − λ i ) P'(\lambda_j)=\prod_{i\ne j}(\lambda_j-\lambda_i) P ′ ( λ j ) = ∏ i = j ( λ j − λ i ) has sign ( − 1 ) j (-1)^j ( − 1 ) j , so all products ( − 1 ) j c j (-1)^jc_j ( − 1 ) j c j have the same sign. The Chebyshev polynomial T n T_n T n , defined by T n ( cos ϑ ) = cos ( n ϑ ) T_n(\cos\vartheta)=\cos(n\vartheta) T n ( cos ϑ ) = cos ( n ϑ ) , satisfies T n ( λ j / a ) = ( − 1 ) j T_n(\lambda_j/a)=(-1)^j T n ( λ j / a ) = ( − 1 ) j . Interpolation of T n ( z / a ) T_n(z/a) T n ( z / a ) , a polynomial of degree n n n , is exact, hence
∑ j = 0 n ∣ c j ∣ = ∣ ∑ j = 0 n ( − 1 ) j c j ∣ = ∣ d d z T n ( z / a ) ∣ z = 0 ∣ = ∣ T n ′ ( 0 ) ∣ a = n a , \sum_{j=0}^{n}|c_j|=\left|\sum_{j=0}^{n}(-1)^jc_j\right|=\left|\left.\frac{d}{dz}T_n(z/a)\right|_{z=0}\right|=\frac{|T_n'(0)|}{a}=\frac{n}{a}, j = 0 ∑ n ∣ c j ∣ = j = 0 ∑ n ( − 1 ) j c j = d z d T n ( z / a ) z = 0 = a ∣ T n ′ ( 0 ) ∣ = a n ,
where ∣ T n ′ ( 0 ) ∣ = n ∣ sin ( n π / 2 ) ∣ = n |T_n'(0)|=n|\sin(n\pi/2)|=n ∣ T n ′ ( 0 ) ∣ = n ∣ sin ( nπ /2 ) ∣ = n follows by differentiating T n ( cos ϑ ) = cos ( n ϑ ) T_n(\cos\vartheta)=\cos(n\vartheta) T n ( cos ϑ ) = cos ( n ϑ ) at ϑ = π / 2 \vartheta=\pi/2 ϑ = π /2 . Put w j = ∣ c j ∣ a / n w_j=|c_j|a/n w j = ∣ c j ∣ a / n . For χ \chi χ as in the statement, exact interpolation and the Cauchy–Schwarz inequality give
1 = χ ′ ( 0 ) 2 = ( ∑ j = 0 n c j χ ( λ j ) ) 2 ≤ ( ∑ j = 0 n ∣ c j ∣ ) ∑ j = 0 n ∣ c j ∣ χ ( λ j ) 2 = n 2 a 2 ∑ j = 0 n w j χ ( λ j ) 2 . 1=\chi'(0)^2=\left(\sum_{j=0}^{n}c_j\chi(\lambda_j)\right)^2\leq\left(\sum_{j=0}^{n}|c_j|\right)\sum_{j=0}^{n}|c_j|\chi(\lambda_j)^2=\frac{n^2}{a^2}\sum_{j=0}^{n}w_j\chi(\lambda_j)^2. 1 = χ ′ ( 0 ) 2 = ( j = 0 ∑ n c j χ ( λ j ) ) 2 ≤ ( j = 0 ∑ n ∣ c j ∣ ) j = 0 ∑ n ∣ c j ∣ χ ( λ j ) 2 = a 2 n 2 j = 0 ∑ n w j χ ( λ j ) 2 .
Lemma A.7 (Gaussian projection loss). Let t t t , k k k , h ≥ 1 h\geq1 h ≥ 1 be integers with h ≥ max { 16 k , 32 log ( 8 t ) } h\geq\max\{16k,32\log(8t)\} h ≥ max { 16 k , 32 log ( 8 t )} . For j = 1 , … , t j=1,\ldots,t j = 1 , … , t , let S j S_j S j be the span of k k k independent standard Gaussian vectors in R h \mathbb{R}^h R h (dependence between different j j j is allowed), and let e j ∈ R h e_j\in\mathbb{R}^h e j ∈ R h be fixed unit vectors. Then
E 1 − max j ≤ t ∥ Π S j e j ∥ 2 ≥ 3 4 2 > 1 2 . \mathbb{E}\sqrt{1-\max_{j\leq t}\|\Pi_{S_j}e_j\|^2}\geq\frac{3}{4\sqrt{2}}>\frac{1}{2}. E 1 − j ≤ t max ∥ Π S j e j ∥ 2 ≥ 4 2 3 > 2 1 .
Proof. Fix j j j . By rotation invariance, ∥ Π S j e j ∥ 2 \|\Pi_{S_j}e_j\|^2 ∥ Π S j e j ∥ 2 has the law of X / Z X/Z X / Z , where X = ∑ i = 1 k G i 2 X=\sum_{i=1}^{k}G_i^2 X = ∑ i = 1 k G i 2 and Z = ∑ i = 1 h G i 2 Z=\sum_{i=1}^{h}G_i^2 Z = ∑ i = 1 h G i 2 for independent standard Gaussian G i G_i G i . The event { X / Z > 1 / 2 } \{X/Z>1/2\} { X / Z > 1/2 } is contained in { X > h / 4 } ∪ { Z < h / 2 } \{X>h/4\}\cup\{Z<h/2\} { X > h /4 } ∪ { Z < h /2 } . By the Chernoff bound with the Gaussian moment generating function, and using k ≤ h / 16 k\leq h/16 k ≤ h /16 and 2 3 ≤ log 2 ≤ 1 \frac{2}{3}\leq\log2\leq1 3 2 ≤ log 2 ≤ 1 ,
P ( X > h / 4 ) ≤ e − h / 16 2 k / 2 ≤ e − h / 16 + h / 32 = e − h / 32 , P ( Z < h / 2 ) ≤ e h / 4 2 − h / 2 ≤ e − h / 12 . \mathbb{P}(X>h/4)\leq e^{-h/16}2^{k/2}\leq e^{-h/16+h/32}=e^{-h/32},\qquad\mathbb{P}(Z<h/2)\leq e^{h/4}2^{-h/2}\leq e^{-h/12}. P ( X > h /4 ) ≤ e − h /16 2 k /2 ≤ e − h /16 + h /32 = e − h /32 , P ( Z < h /2 ) ≤ e h /4 2 − h /2 ≤ e − h /12 .
A union bound over j j j shows that some ∥ Π S j e j ∥ 2 \|\Pi_{S_j}e_j\|^2 ∥ Π S j e j ∥ 2 exceeds 1 / 2 1/2 1/2 with probability at most 2 t e − h / 32 ≤ 1 / 4 2te^{-h/32}\leq1/4 2 t e − h /32 ≤ 1/4 . On the complementary event the square root is at least 1 / 2 1/\sqrt{2} 1/ 2 .
Lemma A.8 (Residual after Gaussian polynomial information). Let n n n , a a a , λ j \lambda_j λ j , and w j w_j w j be as in Lemma A.6. Let R ( n + 1 ) h = V 0 ⊕ ⋯ ⊕ V n \mathbb{R}^{(n+1)h}=V_0\oplus\cdots\oplus V_n R ( n + 1 ) h = V 0 ⊕ ⋯ ⊕ V n be an orthogonal decomposition into blocks of dimension h h h , let Λ ∣ V j = λ j I \Lambda|_{V_j}=\lambda_j I Λ ∣ V j = λ j I , and fix unit vectors e j ∈ V j e_j\in V_j e j ∈ V j . Put
z 0 = R ∑ j = 0 n w j e j , b 0 = Λ z 0 , z_0=R\sum_{j=0}^{n}\sqrt{w_j}\,e_j,\qquad b_0=\Lambda z_0, z 0 = R j = 0 ∑ n w j e j , b 0 = Λ z 0 , let g 1 , … , g k g_1,\ldots,g_k g 1 , … , g k be independent standard Gaussian vectors in R ( n + 1 ) h \mathbb{R}^{(n+1)h} R ( n + 1 ) h , and assume h ≥ max { 16 k , 32 log ( 8 ( n + 1 ) ) } h\geq\max\{16k,32\log(8(n+1))\} h ≥ max { 16 k , 32 log ( 8 ( n + 1 ))} . Let
v = p ( Λ ) b 0 + ∑ i = 1 k p i ( Λ ) g i , v=p(\Lambda)b_0+\sum_{i=1}^{k}p_i(\Lambda)g_i, v = p ( Λ ) b 0 + i = 1 ∑ k p i ( Λ ) g i , where p p p and p i p_i p i are polynomials whose coefficients may depend arbitrarily on the Gaussian vectors and on further randomness, with deg p ≤ n − 2 \deg p\leq n-2 deg p ≤ n − 2 and arbitrary deg p i \deg p_i deg p i . Then E ∥ Λ v + b 0 ∥ ≥ a R 2 n \mathbb{E}\|\Lambda v+b_0\|\geq\frac{aR}{2n} E ∥Λ v + b 0 ∥ ≥ 2 n a R .
Proof. Let S j ⊆ V j S_j\subseteq V_j S j ⊆ V j be the span of the projections of g 1 , … , g k g_1,\ldots,g_k g 1 , … , g k onto V j V_j V j ; these projections are independent standard Gaussian vectors in V j V_j V j . Since Λ \Lambda Λ acts on V j V_j V j as the scalar λ j \lambda_j λ j , the V j V_j V j -component of Λ v + b 0 \Lambda v+b_0 Λ v + b 0 equals
R w j χ ( λ j ) e j + s j , s j ∈ S j , χ ( λ ) = λ + λ 2 p ( λ ) . R\sqrt{w_j}\chi(\lambda_j)e_j+s_j,\qquad s_j\in S_j,\qquad\chi(\lambda)=\lambda+\lambda^2p(\lambda). R w j χ ( λ j ) e j + s j , s j ∈ S j , χ ( λ ) = λ + λ 2 p ( λ ) .
For every realization, deg χ ≤ n \deg\chi\leq n deg χ ≤ n and χ ′ ( 0 ) = 1 \chi'(0)=1 χ ′ ( 0 ) = 1 . Projecting each component onto S j ⊥ ∩ V j S_j^\perp\cap V_j S j ⊥ ∩ V j and using Lemma A.6,
∥ Λ v + b 0 ∥ 2 ≥ R 2 ∑ j = 0 n w j χ ( λ j ) 2 ( 1 − ∥ Π S j e j ∥ 2 ) ≥ a 2 R 2 n 2 ( 1 − max j ∥ Π S j e j ∥ 2 ) . \|\Lambda v+b_0\|^2\geq R^2\sum_{j=0}^{n}w_j\chi(\lambda_j)^2\left(1-\|\Pi_{S_j}e_j\|^2\right)\geq\frac{a^2R^2}{n^2}\left(1-\max_j\|\Pi_{S_j}e_j\|^2\right). ∥Λ v + b 0 ∥ 2 ≥ R 2 j = 0 ∑ n w j χ ( λ j ) 2 ( 1 − ∥ Π S j e j ∥ 2 ) ≥ n 2 a 2 R 2 ( 1 − j max ∥ Π S j e j ∥ 2 ) .
Taking square roots and applying Lemma A.7 with t = n + 1 t=n+1 t = n + 1 proves the claim. The argument is pointwise in the polynomial coefficients, so their dependence on the Gaussian vectors is irrelevant.
A hard law of quadratics.
Lemma A.9 (Quadratic depth bound). Let m , s ≥ 1 m,s \ge1 m , s ≥ 1 be integers and L ′ , R > 0 L',R>0 L ′ , R > 0 . There is a distribution of convex quadratic functions in dimension at most 768 m s 2 768ms^2 768 m s 2 , each with an L ′ L' L ′ -Lipschitz gradient, minimum value zero, and a unique minimizer of norm R R R , such that every randomized algorithm that makes s s s adaptive batches of at most m m m simultaneous queries, each returning F ( x ) F(x) F ( x ) and ∇ F ( x ) \nabla F(x) ∇ F ( x ) , and then outputs x ^ \hat{x} x ^ , satisfies
E F ( x ^ ) ≥ L ′ R 2 288 s 2 . (10) \mathbb{E}F(\hat{x}) \ge\frac{L'R^2}{288s^2}. \tag*{(10)} E F ( x ^ ) ≥ 288 s 2 L ′ R 2 . (10) The distribution does not depend on the algorithm.
Proof. Set n = 2 s + 3 n=2s+3 n = 2 s + 3 , k = 2 m s + 1 k=2ms+1 k = 2 m s + 1 , h = 128 m s h=128ms h = 128 m s , and a = L ′ a=\sqrt{L'} a = L ′ , and let Λ \Lambda Λ , b 0 b_0 b 0 , z 0 z_0 z 0 be as in Lemma A.8; the dimension is ( n + 1 ) h = ( 2 s + 4 ) ⋅ 128 m s ≤ 768 m s 2 (n+1)h=(2s+4)\cdot128ms\le768ms^2 ( n + 1 ) h = ( 2 s + 4 ) ⋅ 128 m s ≤ 768 m s 2 . The assumptions of Lemma A.8 hold: 16 k ≤ 48 m s ≤ h 16k\le48ms\le h 16 k ≤ 48 m s ≤ h , and 32 log ( 8 ( n + 1 ) ) = 32 log ( 16 s + 32 ) ≤ 128 s ≤ h 32\log(8(n+1))=32\log(16s+32)\le128s\le h 32 log ( 8 ( n + 1 )) = 32 log ( 16 s + 32 ) ≤ 128 s ≤ h , because log ( 16 s + 32 ) ≤ 4 s \log(16s+32)\le4s log ( 16 s + 32 ) ≤ 4 s holds at s = 1 s=1 s = 1 and the difference of the two sides increases with s s s .
Let U U U be Haar distributed, independent of the algorithm, and define
A = U Λ U T , b = U b 0 , F ( x ) = 1 2 ∥ A x + b ∥ 2 . A=U\Lambda U^\mathsf{T},\qquad b=Ub_0,\qquad F(x)=\frac{1}{2}\lVert Ax+b\rVert^2. A = U Λ U T , b = U b 0 , F ( x ) = 2 1 ∥ A x + b ∥ 2 .
The Hessian A 2 = U Λ 2 U T A^2=U\Lambda^2U^\mathsf{T} A 2 = U Λ 2 U T is positive definite, because no node vanishes, and has norm a 2 = L ′ a^2=L' a 2 = L ′ . The unique minimizer − A − 1 b = − U z 0 -A^{-1}b=-Uz_0 − A − 1 b = − U z 0 has norm ∥ z 0 ∥ = R \lVert z_0\rVert=R ∥ z 0 ∥ = R , and the minimum value is zero.
Give b b b to the algorithm. A batch of m m m queries of F F F and ∇ F \nabla F ∇ F is answered by two batches of m m m matrix products, first A x Ax A x and then A ( A x + b ) A(Ax+b) A ( A x + b ) , which also give F ( x ) = 1 2 ∥ A x + b ∥ 2 F(x)=\frac{1}{2}\lVert Ax+b\rVert^2 F ( x ) = 2 1 ∥ A x + b ∥ 2 . Hence the algorithm is a procedure as in Lemma A.5 with D = 2 s D=2s D = 2 s batches of width m m m . By that lemma, after granting all polynomial powers of the 2 s m + 1 = k 2sm+1=k 2 s m + 1 = k Gaussian vectors, U T x ^ = p ( Λ ) b 0 + ∑ i = 1 k p i ( Λ ) g i U^\mathsf{T}\hat{x}=p(\Lambda)b_0+\sum_{i=1}^{k}p_i(\Lambda)g_i U T x ^ = p ( Λ ) b 0 + ∑ i = 1 k p i ( Λ ) g i with deg p ≤ 2 s = n − 3 \deg p\le2s=n-3 deg p ≤ 2 s = n − 3 . Lemma A.8 gives
E ∥ A x ^ + b ∥ = E ∥ Λ U T x ^ + b 0 ∥ ≥ L ′ R 2 n . \mathbb{E}\lVert A\hat{x}+b\rVert=\mathbb{E}\lVert\Lambda U^\mathsf{T}\hat{x}+b_0\rVert\ge\frac{\sqrt{L'}R}{2n}. E ∥ A x ^ + b ∥ = E ∥ Λ U T x ^ + b 0 ∥ ≥ 2 n L ′ R .
By Jensen’s inequality, E F ( x ^ ) ≥ 1 2 ( E ∥ A x ^ + b ∥ ) 2 ≥ L ′ R 2 8 n 2 \mathbb{E}F(\hat{x})\ge\frac{1}{2}(\mathbb{E}\lVert A\hat{x}+b\rVert)^2\ge\frac{L'R^2}{8n^2} E F ( x ^ ) ≥ 2 1 ( E ∥ A x ^ + b ∥ ) 2 ≥ 8 n 2 L ′ R 2 , and n = 2 s + 3 ≤ 6 s n=2s+3\le6s n = 2 s + 3 ≤ 6 s proves (A.6). The law of F F F is the pushforward of the Haar law of U U U , which by Lemma A.5 is the same for every algorithm. □ \square □
Masking whole sample functions.
Theorem A.10 (Quadratic communication bound). Let 0 < L ′ ≤ L 0<L'\le L 0 < L ′ ≤ L , p = L ′ / H s p=L'/H_s p = L ′ / H s , and
b p = E max w ≤ m Z w , Z 1 , … , Z m ∼ i i d Bin ( K , p ) . b_p=\mathbb{E}\max_{w\le m} Z_w,\qquad Z_1,\ldots,Z_m\mathrel{\overset{\mathrm{iid}}{\sim}}\operatorname{Bin}(K,p). b p = E w ≤ m max Z w , Z 1 , … , Z m ∼ iid Bin ( K , p ) . For every d ≥ 768 m ( K N ) 2 d\ge768m(KN)^2 d ≥ 768 m ( K N ) 2 and every algorithm in I C ( m , K , N ) \mathrm{IC}(m,K,N) IC ( m , K , N ) with fresh access, there is an instance of S d q u a d \mathcal{S}^{\mathrm{quad}}_d S d quad such that
E F ( x ^ ) ≥ L ′ R 2 2304 ( 1 + N b p ) 2 . (11) \mathbb{E}F(\hat{x})\ge\frac{L'R^2}{2304(1+Nb_p)^2}. \tag*{(11)} E F ( x ^ ) ≥ 2304 ( 1 + N b p ) 2 L ′ R 2 . (11)
Proof. Let s = min { K N , max { 1 , ⌈ 2 N b p ⌉ } } s=\min\{KN,\max\{1,\lceil2Nb_p\rceil\}\} s = min { K N , max { 1 , ⌈ 2 N b p ⌉}} and draw F F F from the law of Lemma A.9 with parameters ( m , s , L ′ , R ) (m,s,L',R) ( m , s , L ′ , R ) , padded to R d \mathbb{R}^d R d . The sample functions are
f ξ = ξ p F , ξ ∼ Bernoulli ( p ) , f_\xi=\frac{\xi}{p}F,\qquad\xi\sim\operatorname{Bernoulli}(p), f ξ = p ξ F , ξ ∼ Bernoulli ( p ) ,
with ξ \xi ξ independent of F F F . Each f ξ f_\xi f ξ is a nonnegative convex quadratic with an H s H_s H s -Lipschitz gradient, minimized at the minimizer of F F F , and E f ξ = F \mathbb{E}f_\xi=F E f ξ = F . Thus every F F F in the support defines an instance of S d q u a d \mathcal{S}^{\mathrm{quad}}_d S d quad .
With fresh access, the k k k -th query of worker w w w in round n n n uses an independent variable ξ w , n , k \xi_{w,n,k} ξ w , n , k . Give the whole array of these variables to the algorithm before the run; this can only help it. Let Z w , n = ∑ k = 1 K ξ w , n , k Z_{w,n}=\sum_{k=1}^{K}\xi_{w,n,k} Z w , n = ∑ k = 1 K ξ w , n , k and Δ = ∑ n = 1 N max w ≤ m Z w , n \Delta=\sum_{n=1}^{N}\max_{w\le m}Z_{w,n} Δ = ∑ n = 1 N max w ≤ m Z w , n .
For a fixed array, the run can be reproduced by an algorithm that makes Δ \Delta Δ batches of at most m m m exact queries of ( F , ∇ F ) (F,\nabla F) ( F , ∇ F ) . Indeed, a query with ξ = 0 \xi=0 ξ = 0 has the known reply 0 0 0 , and a query with ξ = 1 \xi=1 ξ = 1 has the reply ∇ F ( x ) / p \nabla F(x)/p ∇ F ( x ) / p . In round n n n , let the j j j -th batch contain the j j j -th query with ξ = 1 \xi=1 ξ = 1 of every worker that has at least j j j such queries in this round. This query depends only on the seed, on the messages before round n n n , and on the earlier replies of the same worker in round n n n , all of which are available after the first j − 1 j-1 j − 1 batches of the round. Hence round n n n needs max w Z w , n \max_w Z_{w,n} max w Z w , n batches, and the whole run needs Δ \Delta Δ batches.
On the event { Δ ≤ s } \{\Delta\le s\} { Δ ≤ s } , which is determined by the array, pad the reproducing algorithm to exactly s s s batches. For every fixed array in this event it is an s s s -batch algorithm, and the array is independent of F F F , so Lemma A.9 gives E [ F ( x ^ ) ∣ array ] ≥ L ′ R 2 288 s 2 \mathbb{E}[F(\hat{x})\mid\text{array}] \ge\frac{L'R^2}{288s^2} E [ F ( x ^ ) ∣ array ] ≥ 288 s 2 L ′ R 2 on { Δ ≤ s } \{\Delta\le s\} { Δ ≤ s } . We have E Δ = N b p \mathbb{E}\Delta= N b_p E Δ = N b p and Δ ≤ K N \Delta\le KN Δ ≤ K N . If s = K N s=KN s = K N , then Δ ≤ s \Delta\le s Δ ≤ s surely; otherwise s ≥ 2 N b p s\ge2Nb_p s ≥ 2 N b p , and Markov’s inequality gives P ( Δ > s ) ≤ N b p / s ≤ 1 / 2 \mathbb{P}(\Delta>s)\le Nb_p/s\le1/2 P ( Δ > s ) ≤ N b p / s ≤ 1/2 . Therefore E F ( x ^ ) ≥ L ′ R 2 576 s 2 \mathbb{E}F(\hat{x})\ge\frac{L'R^2}{576s^2} E F ( x ^ ) ≥ 576 s 2 L ′ R 2 , and s ≤ 1 + 2 N b p ≤ 2 ( 1 + N b p ) s\le1+2Nb_p\le2(1+Nb_p) s ≤ 1 + 2 N b p ≤ 2 ( 1 + N b p ) proves (A.7) for the random instance; averaging selects a fixed one. The dimension of the law is at most 768 m s 2 ≤ 768 m ( K N ) 2 768ms^2\le768m(KN)^2 768 m s 2 ≤ 768 m ( K N ) 2 .
Theorem A.11 (Lower bound for quadratic populations). For every d ≥ 2048 m ( K N ) 2 d\ge2048m(KN)^2 d ≥ 2048 m ( K N ) 2 ,
M d s g , q u a d ≥ R 2 73728 log 2 ( e m ) min { L , min { L , H s / K } N 2 + H s m K N } . \mathcal{M}^{\mathrm{sg,quad}}_d \ge\frac{R^2}{73728\log^2(em)}\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}. M d sg , quad ≥ 73728 log 2 ( e m ) R 2 min { L , N 2 min { L , H s / K } + m K N H s } .
Proof. We first show that
M d s g , q u a d ≥ R 2 min { L , H s / K } 36864 N 2 log 2 ( e m ) . (12) \mathcal{M}^{\mathrm{sg,quad}}_d \ge\frac{R^2\min\{L,H_s/K\}}{36864N^2\log^2(em)}. \tag*{(12)} M d sg , quad ≥ 36864 N 2 log 2 ( e m ) R 2 min { L , H s / K } . (12)
If log ( e m ) ≤ K \log(em)\le K log ( e m ) ≤ K , let p = min { L / H s , log ( e m ) / K } p=\min\{L/H_s,\log(em)/K\} p = min { L / H s , log ( e m ) / K } and L ′ = p H s ≤ L L'=pH_s\le L L ′ = p H s ≤ L . By Lemma A.2 and e − 1 ≤ 2 e-1\le2 e − 1 ≤ 2 , b p ≤ log m + 2 K p ≤ 3 log ( e m ) b_p\le\log m+2Kp\le3\log(em) b p ≤ log m + 2 K p ≤ 3 log ( e m ) , so 1 + N b p ≤ 4 N log ( e m ) 1+Nb_p\le4N\log(em) 1 + N b p ≤ 4 N log ( e m ) . Theorem A.10 gives the lower bound L ′ R 2 36864 N 2 log 2 ( e m ) \frac{L'R^2}{36864N^2\log^2(em)} 36864 N 2 l o g 2 ( e m ) L ′ R 2 , and L ′ = min { L , H s log ( e m ) / K } ≥ min { L , H s / K } L'=\min\{L,H_s\log(em)/K\}\ge\min\{L,H_s/K\} L ′ = min { L , H s log ( e m ) / K } ≥ min { L , H s / K } .
If log ( e m ) > K \log(em)>K log ( e m ) > K , take deterministic samples f ξ = F f_\xi=F f ξ = F , where F F F is drawn from the law of Lemma A.9 with L ′ = L L'=L L ′ = L and s = K N s=KN s = K N ; these are admissible because L ≤ H s L\le H_s L ≤ H s . An IC algorithm with exact gradients is reproduced by K N KN K N batches of width m m m , the ( n , k ) (n,k) ( n , k ) -th batch consisting of the k k k -th queries of all workers in round n n n . Lemma A.9 gives the lower bound L R 2 288 K 2 N 2 ≥ 288 N 2 log 2 ( e m ) \frac{LR^2}{288K^2N^2}\ge\frac{288N^2\log^2(em)}{} 288 K 2 N 2 L R 2 ≥ 288 N 2 l o g 2 ( e m ) , which implies (A.8).
By Lemma A.4, M d s g , q u a d ≥ R 2 8 min { L , H s / ( m K N ) } \mathcal{M}^{\mathrm{sg,quad}}_d\ge\frac{R^2}{8}\min\{L,H_s/(mKN)\} M d sg , quad ≥ 8 R 2 min { L , H s / ( m K N )} . The maximum of two lower bounds is at least their average, and 1 16 ≥ 1 73728 log 2 ( e m ) \frac{1}{16}\ge\frac{1}{73728\log^2(em)} 16 1 ≥ 73728 l o g 2 ( e m ) 1 , so
M d s g , q u a d ≥ R 2 73728 log 2 ( e m ) [ min { L , H s / K } N 2 + min { L , H s m K N } ] ≥ R 2 73728 log 2 ( e m ) min { L , min { L , H s / K } N 2 + H s m K N } . \begin{aligned}
\mathcal{M}^{\mathrm{sg,quad}}_d\ge\frac{R^2}{73728\log^2(em)}\left[\frac{\min\{L,H_s/K\}}{N^2}+\min\left\{L,\frac{H_s}{mKN}\right\}\right] \\
&\ge\frac{R^2}{73728\log^2(em)}\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}.
\end{aligned} M d sg , quad ≥ 73728 log 2 ( e m ) R 2 [ N 2 min { L , H s / K } + min { L , m K N H s } ] ≥ 73728 log 2 ( e m ) R 2 min { L , N 2 min { L , H s / K } + m K N H s } .
where the last inequality holds because the bracket equals min { L , H s / K } N 2 + H s m K N \frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN} N 2 m i n { L , H s / K } + m K N H s if H s m K N ≤ L \frac{H_s}{mKN}\le L m K N H s ≤ L and is at least L L L otherwise. All constructions have dimension at most 768 m ( K N ) 2 768m(KN)^2 768 m ( K N ) 2 , so by padding the bound holds for every d ≥ 768 m ( K N ) 2 d\ge768m(KN)^2 d ≥ 768 m ( K N ) 2 , in particular for every d ≥ 2048 m ( K N ) 2 d\ge2048m(KN)^2 d ≥ 2048 m ( K N ) 2 .
Weak growth condition
The hard oracle reveals the next direction of a hidden chain only with probability p p p per query, and corrects the remaining bias with a rare Bernoulli variable. Throughout this subsection,
ℓ = L 1024 , q = ⌈ max { 8 N log ( e m ) , 64 ℓ K N H s } ⌉ , δ = R ( ∑ j = 1 q j 2 ) − 1 / 2 , (13) \ell=\frac{L}{1024},\qquad q=\left\lceil\max\left\{8N\log(em),64\sqrt{\frac{\ell KN}{H_s}}\right\}\right\rceil,\qquad\delta=R\left(\sum_{j=1}^{q}j^2\right)^{-1/2}, \tag*{(13)} ℓ = 1024 L , q = ⌈ max { 8 N log ( e m ) , 64 H s ℓ K N } ⌉ , δ = R ( j = 1 ∑ q j 2 ) − 1/2 , (13)
p = 128 ℓ H s q , ε = 1 16 m K N , α = min { δ 8 , δ H s ε 800 ℓ } , p=\frac{128\ell}{H_sq},\qquad\varepsilon=\frac{1}{16mKN},\qquad\alpha=\min\left\{\frac{\delta}{8},\delta\sqrt{\frac{H_s\varepsilon}{800\ell}}\right\}, p = H s q 128 ℓ , ε = 16 m K N 1 , α = min { 8 δ , δ 800 ℓ H s ε } ,
and
d w g = q + 2 + ⌈ 2 ( 130 R ) 2 α 2 log ( 16 q ( m K N + 1 ) ) ⌉ . (14) d_{\mathrm{wg}}=q+2+\left\lceil\frac{2(130R)^2}{\alpha^2}\log\bigl(16q(mKN+1)\bigr)\right\rceil. \tag*{(14)} d wg = q + 2 + ⌈ α 2 2 ( 130 R ) 2 log ( 16 q ( m K N + 1 ) ) ⌉ . (14)
Since R / α = ( ∑ j ≤ q j 2 ) 1 / 2 max { 8 , 800 ℓ / ( H s ε ) } R/\alpha=\left(\sum_{j\le q}j^2\right)^{1/2}\max\left\{8,\sqrt{800\ell/(H_s\varepsilon)}\right\} R / α = ( ∑ j ≤ q j 2 ) 1/2 max { 8 , 800 ℓ / ( H s ε ) } , the threshold d w g d_{\mathrm{wg}} d wg depends only on m m m , K K K , N N N , and H s / L H_s/L H s / L . Note that q ≥ 8 q\ge8 q ≥ 8 , that p ≤ 1 8 q p\le\frac{1}{8q} p ≤ 8 q 1 because 1024 ℓ = L ≤ H s 1024\ell=L\le H_s 1024 ℓ = L ≤ H s , and that ∑ j ≤ q j 2 ∈ [ q , q 3 ] \sum_{j\le q}j^2\in[q,q^3] ∑ j ≤ q j 2 ∈ [ q , q 3 ] gives
δ q ≤ R , δ 2 q ≥ R 2 q 2 . (15) \delta\sqrt{q}\le R,\qquad\delta^2q\ge\frac{R^2}{q^2}. \tag*{(15)} δ q ≤ R , δ 2 q ≥ q 2 R 2 . (15)
The chain. Let h δ ( u ) = u 2 / 2 h_{\delta}(u) = u^{2}/2 h δ ( u ) = u 2 /2 for ∣ u ∣ ≤ δ |u| \le\delta ∣ u ∣ ≤ δ and h δ ( u ) = δ ∣ u ∣ − δ 2 / 2 h_{\delta}(u) = \delta|u| - \delta^{2}/2 h δ ( u ) = δ ∣ u ∣ − δ 2 /2 otherwise; its derivative h δ ′ h_{\delta}' h δ ′ is the clipping onto [ − δ , δ ] [-\delta,\delta] [ − δ , δ ] . For y ∈ R q y \in\mathbb{R}^{q} y ∈ R q put y q + 1 = 0 y_{q+1}=0 y q + 1 = 0 and
f ( y ) = ℓ ∑ i = 1 q h δ ( y i − y i + 1 − δ ) , prog ( y ) = max { i ≤ q : ∣ y i ∣ > α } ∪ { 0 } . (16) f(y)=\ell\sum_{i=1}^{q}h_{\delta}(y_i-y_{i+1}-\delta),\qquad\operatorname{prog}(y)=\max\{i\le q:|y_i|>\alpha\}\cup\{0\}. \tag*{(16)} f ( y ) = ℓ i = 1 ∑ q h δ ( y i − y i + 1 − δ ) , prog ( y ) = max { i ≤ q : ∣ y i ∣ > α } ∪ { 0 } . (16)
For 0 ≤ t ≤ q 0 \le t \le q 0 ≤ t ≤ q , let P t P_t P t keep the first t t t coordinates of a vector in R q \mathbb{R}^{q} R q and set the others to zero, and let e 1 , … , e q e_1,\ldots,e_q e 1 , … , e q be the standard basis.
Lemma A.12 (The chain). The function f f f is nonnegative and convex, has a 4 ℓ 4\ell 4 ℓ -Lipschitz gradient with ∥ ∇ f ∥ ≤ 2 ℓ R \|\nabla f\|\le2\ell R ∥∇ f ∥ ≤ 2 ℓ R , and vanishes at r 0 = δ ( q , q − 1 , … , 1 ) r_0=\delta(q,q-1,\ldots,1) r 0 = δ ( q , q − 1 , … , 1 ) , where ∥ r 0 ∥ = R \|r_0\|=R ∥ r 0 ∥ = R . If t = prog ( y ) ≤ q / 2 t=\operatorname{prog}(y)\le q/2 t = prog ( y ) ≤ q /2 , then
f ( y ) ≥ ℓ δ 2 q 16 , (17) f(y)\ge\frac{\ell\delta^{2}q}{16}, \tag*{(17)} f ( y ) ≥ 16 ℓ δ 2 q , (17) and v = P t y v=P_ty v = P t y satisfies
∇ f ( v ) = P t ∇ f ( v ) + [ ∇ f ( v ) ] t + 1 e t + 1 , ∣ [ ∇ f ( v ) ] t + 1 ∣ ≤ 2 ℓ δ , \nabla f(v)=P_t\nabla f(v)+[\nabla f(v)]_{t+1}e_{t+1},\qquad|[\nabla f(v)]_{t+1}|\le2\ell\delta, ∇ f ( v ) = P t ∇ f ( v ) + [ ∇ f ( v ) ] t + 1 e t + 1 , ∣ [ ∇ f ( v ) ] t + 1 ∣ ≤ 2 ℓ δ , ∥ ∇ f ( y ) − ∇ f ( v ) ∥ ≤ 4 ℓ q α , ∣ f ( y ) − f ( v ) ∣ ≤ 2 ℓ δ q α , (18) \|\nabla f(y)-\nabla f(v)\|\le4\ell\sqrt{q}\alpha,\qquad|f(y)-f(v)|\le2\ell\delta q\alpha, \tag*{(18)} ∥∇ f ( y ) − ∇ f ( v ) ∥ ≤ 4 ℓ q α , ∣ f ( y ) − f ( v ) ∣ ≤ 2 ℓ δ q α , (18) Moreover, f ( v ) f(v) f ( v ) and ∇ f ( v ) \nabla f(v) ∇ f ( v ) depend only on y 1 , … , y t y_1,\ldots,y_t y 1 , … , y t .
Proof. Let D D D be the matrix with ( D y ) i = y i − y i + 1 (Dy)_i=y_i-y_{i+1} ( D y ) i = y i − y i + 1 . Since ( y i − y i + 1 ) 2 ≤ 2 y i 2 + 2 y i + 1 2 (y_i-y_{i+1})^2\le2y_i^2+2y_{i+1}^2 ( y i − y i + 1 ) 2 ≤ 2 y i 2 + 2 y i + 1 2 , we have ∥ D ∥ ≤ 2 \|D\|\le2 ∥ D ∥ ≤ 2 . Then
∇ f ( y ) = ℓ D T h δ ′ ( D y − δ 1 ) , \nabla f(y)=\ell D^{\mathsf T}h_{\delta}'(Dy-\delta\mathbf{1}), ∇ f ( y ) = ℓ D T h δ ′ ( D y − δ 1 ) ,
where h δ ′ h_{\delta}' h δ ′ acts coordinatewise. As h δ h_{\delta} h δ is convex with a 1 1 1 -Lipschitz derivative bounded by δ \delta δ , the function f f f is convex with a 4 ℓ 4\ell 4 ℓ -Lipschitz gradient, and ∥ ∇ f ∥ ≤ 2 ℓ δ q ≤ 2 ℓ R \|\nabla f\|\le2\ell\delta\sqrt{q}\le2\ell R ∥∇ f ∥ ≤ 2 ℓ δ q ≤ 2 ℓ R by (A.11). Since D r 0 = δ 1 Dr_0=\delta\mathbf{1} D r 0 = δ 1 , f ( r 0 ) = 0 f(r_0)=0 f ( r 0 ) = 0 , and ∥ r 0 ∥ 2 = δ 2 ∑ j j 2 = R 2 \|r_0\|^2=\delta^2\sum_{j}j^2=R^2 ∥ r 0 ∥ 2 = δ 2 ∑ j j 2 = R 2 .
Let t = prog ( y ) ≤ q / 2 t=\operatorname{prog}(y)\le q/2 t = prog ( y ) ≤ q /2 . For i > t i>t i > t we have ∣ y i ∣ ≤ α |y_i|\le\alpha ∣ y i ∣ ≤ α and ∣ y i + 1 ∣ ≤ α |y_{i+1}|\le\alpha ∣ y i + 1 ∣ ≤ α , hence ∣ y i − y i + 1 − δ ∣ ≥ δ − 2 α ≥ δ / 2 |y_i-y_{i+1}-\delta|\ge\delta-2\alpha\ge\delta/2 ∣ y i − y i + 1 − δ ∣ ≥ δ − 2 α ≥ δ /2 and h δ ( y i − y i + 1 − δ ) ≥ h δ ( δ / 2 ) = δ 2 / 8 h_{\delta}(y_i-y_{i+1}-\delta)\ge h_{\delta}(\delta/2)=\delta^2/8 h δ ( y i − y i + 1 − δ ) ≥ h δ ( δ /2 ) = δ 2 /8 . There are q − t ≥ q / 2 q-t\ge q/2 q − t ≥ q /2 such indices, which gives (A.13).
Let w = h δ ′ ( D v − δ 1 ) w=h_{\delta}'(Dv-\delta\mathbf{1}) w = h δ ′ ( D v − δ 1 ) . Since v i = 0 v_i=0 v i = 0 for i > t i>t i > t , we have w i = − δ w_i=-\delta w i = − δ for i > t i>t i > t . With w 0 = 0 w_0=0 w 0 = 0 , [ ∇ f ( v ) ] j = ℓ ( w j − w j − 1 ) [\nabla f(v)]_j=\ell(w_j-w_{j-1}) [ ∇ f ( v ) ] j = ℓ ( w j − w j − 1 ) , which vanishes for j ≥ t + 2 j\ge t+2 j ≥ t + 2 and has absolute value at most 2 ℓ δ 2\ell\delta 2 ℓ δ for j = t + 1 j=t+1 j = t + 1 . This proves the identity and the first inequality in (A.14). The coordinates of y − v y-v y − v are the y i y_i y i with i > t i>t i > t , so ∥ y − v ∥ ≤ q α \|y-v\|\le\sqrt{q}\alpha ∥ y − v ∥ ≤ q α ; the last two inequalities follow from the Lipschitz property of ∇ f \nabla f ∇ f and the bound on ∥ ∇ f ∥ \|\nabla f\| ∥∇ f ∥ . Finally, v v v is determined by y 1 , … , y t y_1,\ldots,y_t y 1 , … , y t .
The instance. Let U = ( u 1 , … , u q ) ∈ R d × q U=(u_1,\ldots,u_q)\in\mathbb{R}^{d\times q} U = ( u 1 , … , u q ) ∈ R d × q have orthonormal columns, and put F U ( x ) = f ( U T x ) F_U(x)=f(U^{\mathsf T}x) F U ( x ) = f ( U T x ) . By Lemma A.12, F U F_U F U satisfies the hypotheses of Lemma A.3 with x ∗ = U r 0 x^*=Ur_0 x ∗ = U r 0 , L φ = 4 ℓ L_{\varphi}=4\ell L φ = 4 ℓ , and M φ = 2 ℓ R M_{\varphi}=2\ell R M φ = 2 ℓ R . Apply that lemma with c = ℓ c=\ell c = ℓ and ρ = 128 R \rho=128R ρ = 128 R , and let Γ \Gamma Γ be the corresponding cutoff and
Φ U = T F U . \Phi_U=\mathcal{T}F_U. Φ U = T F U .
Then Φ U \Phi_U Φ U is convex with a 9 ℓ 9\ell 9 ℓ -Lipschitz gradient, 9 ℓ ≤ L 9\ell\le L 9 ℓ ≤ L , Φ U ≥ F U \Phi_U\ge F_U Φ U ≥ F U , Φ U ( U r 0 ) = 0 \Phi_U(Ur_0)=0 Φ U ( U r 0 ) = 0 , and for ∥ x ∥ ≥ 130 R \|x\|\ge130R ∥ x ∥ ≥ 130 R , Φ U ( x ) = 2 ℓ ( ∥ x ∥ − R ) 2 \Phi_U(x)=2\ell(\|x\|-R)^2 Φ U ( x ) = 2 ℓ ( ∥ x ∥ − R ) 2 with a gradient that does not depend on U U U .
Each oracle call draws independent B ∼ Bernoulli ( p ) B\sim\operatorname{Bernoulli}(p) B ∼ Bernoulli ( p ) and C ∼ Bernoulli ( ε ) C\sim\operatorname{Bernoulli}(\varepsilon) C ∼ Bernoulli ( ε ) . For y ∈ R q y\in\mathbb{R}^{q} y ∈ R q with t = prog ( y ) ≤ q / 2 t=\operatorname{prog}(y)\le q/2 t = prog ( y ) ≤ q /2 and v = P t y v=P_ty v = P t y , put
∇ f ^ ( y ; B , C ) = P t ∇ f ( v ) + B p [ ∇ f ( v ) ] t + 1 e t + 1 + C ε ( ∇ f ( y ) − ∇ f ( v ) ) , \widehat{\nabla f}(y;B,C)=P_t\nabla f(v)+\frac{B}{p}[\nabla f(v)]_{t+1}e_{t+1}+\frac{C}{\varepsilon}(\nabla f(y)-\nabla f(v)), ∇ f ( y ; B , C ) = P t ∇ f ( v ) + p B [ ∇ f ( v ) ] t + 1 e t + 1 + ε C ( ∇ f ( y ) − ∇ f ( v )) ,
f ^ ( y ; C ) = f ( v ) + C ε ( f ( y ) − f ( v ) ) . \widehat{f}(y;C)=f(v)+\frac{C}{\varepsilon}(f(y)-f(v)). f ( y ; C ) = f ( v ) + ε C ( f ( y ) − f ( v )) .
The oracle G U ( x ; B , C ) G_U(x;B,C) G U ( x ; B , C ) is defined as follows. If ∥ x ∥ ≥ 130 R \|x\|\ge130R ∥ x ∥ ≥ 130 R , it returns 4 ℓ ( ∥ x ∥ − R ) x ∥ x ∥ 4\ell(\|x\|-R)\frac{x}{\|x\|} 4 ℓ ( ∥ x ∥ − R ) ∥ x ∥ x . If ∥ x ∥ < 130 R \|x\|<130R ∥ x ∥ < 130 R and prog ( U T x ) > q / 2 \operatorname{prog}(U^{\mathsf T}x)>q/2 prog ( U T x ) > q /2 , it returns ∇ Φ U ( x ) \nabla\Phi_U(x) ∇ Φ U ( x ) . Otherwise, it returns
G U ( x ; B , C ) = ( 1 − Γ ( x ) ) U ∇ f ^ ( U T x ; B , C ) − f ^ ( U T x ; C ) ∇ Γ ( x ) + 4 ℓ ( ∥ x ∥ − R ) + x ∥ x ∥ , (19) G_U(x;B,C)=(1-\Gamma(x))U\widehat{\nabla f}(U^{\mathsf T}x;B,C)-\widehat{f}(U^{\mathsf T}x;C)\nabla\Gamma(x)+4\ell(\|x\|-R)_+\frac{x}{\|x\|}, \tag*{(19)} G U ( x ; B , C ) = ( 1 − Γ ( x )) U ∇ f ( U T x ; B , C ) − f ( U T x ; C ) ∇Γ ( x ) + 4 ℓ ( ∥ x ∥ − R ) + ∥ x ∥ x , (19)
with the radial term equal to zero at x = 0 x=0 x = 0 . The oracle is jointly measurable.
Lemma A.13 (Admissibility). For every U U U , the pair ( Φ U , G U ) (\Phi_U,G_U) ( Φ U , G U ) is an instance of W d \mathcal{W}_d W d : for every x ∈ R d x\in\mathbb{R}^{d} x ∈ R d ,
E G U ( x ; B , C ) = ∇ Φ U ( x ) , E ∥ G U ( x ; B , C ) ∥ 2 ≤ 2 H s Φ U ( x ) . \mathbb{E}\,G_U(x;B,C)=\nabla\Phi_U(x),\qquad\mathbb{E}\|G_U(x;B,C)\|^2\le2H_s\Phi_U(x). E G U ( x ; B , C ) = ∇ Φ U ( x ) , E ∥ G U ( x ; B , C ) ∥ 2 ≤ 2 H s Φ U ( x ) .
Proof. The function Φ U \Phi_U Φ U is convex, has an L L L -Lipschitz gradient, and is minimized at U r 0 U r_0 U r 0 , which has norm R R R , with Φ U ( U r 0 ) = 0 \Phi_U(U r_0)=0 Φ U ( U r 0 ) = 0 . By Lemma B.1 , Part 2, ∥ ∇ Φ U ∥ 2 ≤ 18 ℓ Φ U ≤ H s Φ U \lVert\nabla\Phi_U\rVert^2\le18\ell\Phi_U\le H_s\Phi_U ∥ ∇ Φ U ∥ 2 ≤ 18 ℓ Φ U ≤ H s Φ U . In the first two cases of the definition of G U G_U G U the reply equals ∇ Φ U ( x ) \nabla\Phi_U(x) ∇ Φ U ( x ) , by Lemma A.3 , Part 4, and the claim follows.
In the remaining case, where G U G_U G U is given by (A.15), let y = U T x y=U^\mathsf{T}x y = U T x , t = prog ( y ) ≤ q / 2 t=\operatorname{prog}(y)\le q/2 t = prog ( y ) ≤ q /2 , and v = P t y v=P_t y v = P t y . By the identity in (A.14), E ∇ ^ f ( y ; B , C ) = ∇ f ( y ) \mathbb{E}\widehat{\nabla}f(y;B,C)=\nabla f(y) E ∇ f ( y ; B , C ) = ∇ f ( y ) , and E f ^ ( y ; C ) = f ( y ) \mathbb{E}\widehat{f}(y;C)=f(y) E f ( y ; C ) = f ( y ) . Since ∇ Φ U ( x ) = U ∇ f ( y ) \nabla\Phi_U(x)=U\nabla f(y) ∇ Φ U ( x ) = U ∇ f ( y ) , Lemma A.3 , Part 3, gives E G U ( x ; B , C ) = ∇ Φ U ( x ) \mathbb{E}G_U(x;B,C)=\nabla\Phi_U(x) E G U ( x ; B , C ) = ∇ Φ U ( x ) . The centered reply equals
( B p − 1 ) ( 1 − Γ ) [ ∇ f ( v ) ] t + 1 U e t + 1 + ( C ε − 1 ) [ ( 1 − Γ ) U ( ∇ f ( y ) − ∇ f ( v ) ) − ( f ( y ) − f ( v ) ) ∇ Γ ] . \left(\frac{B}{p}-1\right)(1-\Gamma)[\nabla f(v)]_{t+1}Ue_{t+1}+\left(\frac{C}{\varepsilon}-1\right)\left[(1-\Gamma)U\left(\nabla f(y)-\nabla f(v)\right)-(f(y)-f(v))\nabla\Gamma\right]. ( p B − 1 ) ( 1 − Γ ) [ ∇ f ( v ) ] t + 1 U e t + 1 + ( ε C − 1 ) [ ( 1 − Γ ) U ( ∇ f ( y ) − ∇ f ( v ) ) − ( f ( y ) − f ( v )) ∇Γ ] .
The two scalar factors are independent and centered, with variances at most 1 / p 1/p 1/ p and 1 / ε 1/\varepsilon 1/ ε . By (A.14), the bound ∥ ∇ Γ ∥ ≤ 3 2 ρ = 3 256 R \lVert\nabla\Gamma\rVert\le\frac{3}{2\rho}=\frac{3}{256R} ∥ ∇Γ ∥ ≤ 2 ρ 3 = 256 R 3 of Lemma A.3 , Part 3, and (A.11), the norm of the bracket is at most 4 ℓ q α + 2 ℓ δ q α 3 256 R ≤ 5 ℓ q α 4\ell\sqrt{q}\alpha+2\ell\delta q\alpha\frac{3}{256R}\le5\ell\sqrt{q}\alpha 4 ℓ q α + 2 ℓ δ q α 256 R 3 ≤ 5 ℓ q α .
Using the definitions of p p p and α \alpha α and then (A.13),
E ∥ G U ( x ; B , C ) − ∇ Φ U ( x ) ∥ 2 ≤ 4 ℓ 2 δ 2 p + 25 ℓ 2 q α 2 ε ≤ H s ℓ δ 2 q 32 + H s ℓ δ 2 q 32 ≤ H s F U ( x ) ≤ H s Φ U ( x ) . \mathbb{E}\lVert G_U(x;B,C)-\nabla\Phi_U(x)\rVert^2\le\frac{4\ell^2\delta^2}{p}+\frac{25\ell^2q\alpha^2}{\varepsilon}\le\frac{H_s\ell\delta^2q}{32}+\frac{H_s\ell\delta^2q}{32}\le H_sF_U(x)\le H_s\Phi_U(x). E ∥ G U ( x ; B , C ) − ∇ Φ U ( x ) ∥ 2 ≤ p 4 ℓ 2 δ 2 + ε 25 ℓ 2 q α 2 ≤ 32 H s ℓ δ 2 q + 32 H s ℓ δ 2 q ≤ H s F U ( x ) ≤ H s Φ U ( x ) .
Adding ∥ ∇ Φ U ( x ) ∥ 2 ≤ H s Φ U ( x ) \lVert\nabla\Phi_U(x)\rVert^2\le H_s\Phi_U(x) ∥ ∇ Φ U ( x ) ∥ 2 ≤ H s Φ U ( x ) proves the second claim.
Remark A.14 (Information revealed by one reply). Let ∥ x ∥ < 130 R \lVert x\rVert<130R ∥ x ∥ < 130 R , fix an integer t ≤ q / 2 t\le q/2 t ≤ q /2 , and evaluate the right-hand side of (A.15) with C = 0 C=0 C = 0 and with this t t t in place of prog ( U T x ) \operatorname{prog}(U^\mathsf{T}x) prog ( U T x ) in the definition of ∇ ^ f \widehat{\nabla}f ∇ f and f ^ \widehat{f} f , where v = P t U T x v=P_tU^\mathsf{T}x v = P t U T x . By the last assertion of Lemma A.12 , the numbers f ( v ) f(v) f ( v ) and [ ∇ f ( v ) ] j [\nabla f(v)]_j [ ∇ f ( v ) ] j are functions of ⟨ u 1 , x ⟩ , … , ⟨ u t , x ⟩ \langle u_1,x\rangle,\ldots,\langle u_t,x\rangle ⟨ u 1 , x ⟩ , … , ⟨ u t , x ⟩ , and U P t ∇ f ( v ) = ∑ j ≤ t [ ∇ f ( v ) ] j u j UP_t\nabla f(v)=\sum_{j\le t}[\nabla f(v)]_ju_j U P t ∇ f ( v ) = ∑ j ≤ t [ ∇ f ( v ) ] j u j , U e t + 1 = u t + 1 Ue_{t+1}=u_{t+1} U e t + 1 = u t + 1 . Hence the result is a function of x x x , B B B , and u 1 , … , u t u_1,\ldots,u_t u 1 , … , u t , and, if B = 1 B=1 B = 1 , also of u t + 1 u_{t+1} u t + 1 . For t = prog ( U T x ) t=\operatorname{prog}(U^\mathsf{T}x) t = prog ( U T x ) it equals G U ( x ; B , 0 ) G_U(x;B,0) G U ( x ; B , 0 ) .
The lower bound.
Theorem A.15 (Lower bound under weak growth). For every d ≥ d w g d\ge d_{\mathrm{wg}} d ≥ d wg ,
M d w g ≥ R 2 2 24 log 2 ( e m ) min { L , H s K N , L N 2 + H s m K N } . \mathfrak{M}^{\mathrm{wg}}_d\ge\frac{R^2}{2^{24}\log^2(em)}\min\left\{L,\frac{H_s}{KN},\frac{L}{N^2}+\frac{H_s}{mKN}\right\}. M d wg ≥ 2 24 log 2 ( e m ) R 2 min { L , K N H s , N 2 L + m K N H s } .
Proof. Fix an algorithm. Let U U U be uniformly distributed on the set of orthonormal q q q -frames in R d \mathbb{R}^d R d , independently of the seed. Index the oracle calls by slots ( w , n , k ) (w,n,k) ( w , n , k ) (worker, round, call) and predraw independent pairs ( B w , n , k , C w , n , k ) (B_{w,n,k},C_{w,n,k}) ( B w , n , k , C w , n , k ) for all m K N mKN m K N slots; a worker that makes fewer than K K K queries in a round leaves the remaining slots unused.
Step 1: a virtual run. Each worker carries a prefix length s w s_w s w , initially zero. After the slot ( w , n , k ) (w,n,k) ( w , n , k ) , whether or not it is used, s w s_w s w increases by B w , n , k B_{w,n,k} B w , n , k ; at the end of every round all s w s_w s w are replaced by their maximum. The prefix lengths are thus functions of the array of B B B 's only. The virtual run applies the maps of the algorithm to the following virtual replies. At a query x x x in a slot where the current prefix length is s s s :
if s > q / 2 s>q/2 s > q /2 , the reply is 0 0 0 ;
if s ≤ q / 2 s\le q/2 s ≤ q /2 and ∥ x ∥ ≥ 130 R \lVert x\rVert\ge130R ∥ x ∥ ≥ 130 R , the reply is 4 ℓ ( ∥ x ∥ − R ) x / ∥ x ∥ 4\ell(\lVert x\rVert-R)x/\lVert x\rVert 4 ℓ (∥ x ∥ − R ) x / ∥ x ∥ ;
if s ≤ q / 2 s\le q/2 s ≤ q /2 and ∥ x ∥ ≤ 130 R \lVert x\rVert\le130R ∥ x ∥ ≤ 130 R , the reply is given by (A.15) with C = 0 C=0 C = 0 , the B B B of the slot, and t = prog ( U T x ) t=\operatorname{prog}(U^\mathsf{T}x) t = prog ( U T x ) in the definition of ∇ ^ f \widehat{\nabla}f ∇ f and f ^ \widehat{f} f replaced by t ~ = max ( { j ≤ s : ∣ ⟨ u j , x ⟩ ∣ > α } ∪ { 0 } ) \widetilde{t}=\max(\{j\le s:\lvert\langle u_j,x\rangle\rvert>\alpha\}\cup\{0\}) t = max ({ j ≤ s : ∣⟨ u j , x ⟩∣ > α } ∪ { 0 }) .
By Remark A.14, a virtual reply at a slot with prefix length s ≤ q / 2 s\le q/2 s ≤ q /2 uses only u 1 , … , u s u_1,\ldots,u_s u 1 , … , u s , and also u s + 1 u_{s+1} u s + 1 if B = 1 B=1 B = 1 , in which case the prefix length after the slot is s + 1 s+1 s + 1 .
Step 2: the depth of the virtual run. Let Z w , n = ∑ k B w , n , k Z_{w,n}=\sum_k B_{w,n,k} Z w , n = ∑ k B w , n , k and Δ = ∑ n = 1 N max w Z w , n \Delta=\sum_{n=1}^{N}\max_w Z_{w,n} Δ = ∑ n = 1 N max w Z w , n . Every prefix length at any time is at most Δ \Delta Δ . The variables Z w , n Z_{w,n} Z w , n are independent Bin ( K , p ) \operatorname{Bin}(K,p) Bin ( K , p ) , so Lemma A.2 , independence over rounds, and Markov's inequality give
P ( Δ > q / 2 ) ≤ e − q / 2 E e Δ ≤ exp ( − q 2 + N log m + 2 K N p ) ≤ e − 5 q / 16 < 1 8 . (20) \mathbb{P}(\Delta>q/2)\le e^{-q/2}\mathbb{E}e^\Delta\le\exp\left(-\frac{q}{2}+N\log m+2KNp\right)\le e^{-5q/16}<\frac{1}{8}. \tag*{(20)} P ( Δ > q /2 ) ≤ e − q /2 E e Δ ≤ exp ( − 2 q + N log m + 2 K N p ) ≤ e − 5 q /16 < 8 1 . (20)
because N log m ≤ q / 8 N\log m\le q/8 N log m ≤ q /8 and 2 K N p = 256 K N H s q = q 16 2KNp=\frac{256KN}{H_s q}=\frac{q}{16} 2 K N p = H s q 256 K N = 16 q by (A.9), and q ≥ 8 q\ge8 q ≥ 8 .
Step 3: hidden projections. Fix the seed and the arrays of B B B 's and C C C 's. Consider a slot where the prefix length is s ≤ q / 2 s\le q/2 s ≤ q /2 . The virtual query x ~ \widetilde{x} x at this slot is a function of u 1 , … , u s u_1,\ldots,u_s u 1 , … , u s ; it depends on the messages of previous rounds and on the earlier virtual replies of the same worker in the current round, and each of these replies was given at a slot whose prefix length after the slot is at most s s s , so by Step 1 it uses only u 1 , … , u s u_1,\ldots,u_s u 1 , … , u s . In the same way, the virtual output x ~ o u t \tilde{x}_{\mathrm{out}} x ~ out is a function of u 1 , … , u Δ u_1,\ldots,u_{\Delta} u 1 , … , u Δ when Δ ≤ q / 2 \Delta\le q/2 Δ ≤ q /2 . Conditional on the seed, the arrays, and u 1 , … , u s u_1,\ldots,u_s u 1 , … , u s , each u j u_j u j with j > s j>s j > s is uniform on the unit sphere of the ( d − s ) (d-s) ( d − s ) -dimensional space span { u 1 , … , u s } ⊥ \operatorname{span}\{u_1,\ldots,u_s\}^{\perp} span { u 1 , … , u s } ⊥ . Lemma A.1, applied to x ~ 1 { ∥ x ~ ∥ < 130 R } \tilde{x}\mathbf{1}\{\|\tilde{x}\|<130R\} x ~ 1 { ∥ x ~ ∥ < 130 R } , gives
P ( ∣ ⟨ u j , x ~ ⟩ ∣ > α , ∥ x ~ ∥ < 130 R ∣ seed, arrays, u 1 , … , u s ) ≤ 2 exp ( − ( d − q − 1 ) α 2 2 ( 130 R ) 2 ) . \mathbb{P}\left(\left.|\langle u_j,\tilde{x}\rangle|>\alpha,\ \|\tilde{x}\|<130R\ \right|\ \text{seed, arrays, }u_1,\ldots,u_s\right)\le2\exp\left(-\frac{(d-q-1)\alpha^2}{2(130R)^2}\right). P ( ∣ ⟨ u j , x ~ ⟩ ∣ > α , ∥ x ~ ∥ < 130 R ∣ seed, arrays, u 1 , … , u s ) ≤ 2 exp ( − 2 ( 130 R ) 2 ( d − q − 1 ) α 2 ) .
and the same bound holds for x ~ o u t \tilde{x}_{\mathrm{out}} x ~ out with s = Δ ≤ q / 2 s=\Delta\le q/2 s = Δ ≤ q /2 . Let E p r o j E_{\mathrm{proj}} E proj be the event that ∣ ⟨ u j , x ~ ⟩ ∣ ≤ α |\langle u_j,\tilde{x}\rangle|\le\alpha ∣ ⟨ u j , x ~ ⟩ ∣ ≤ α for every virtual query x ~ \tilde{x} x ~ with ∥ x ~ ∥ < 130 R \|\tilde{x}\|<130R ∥ x ~ ∥ < 130 R at a slot with prefix length s ≤ q / 2 s\le q/2 s ≤ q /2 and every j > s j>s j > s , and the same for x ~ o u t \tilde{x}_{\mathrm{out}} x ~ out if Δ ≤ q / 2 \Delta\le q/2 Δ ≤ q /2 and ∥ x ~ o u t ∥ < 130 R \|\tilde{x}_{\mathrm{out}}\|<130R ∥ x ~ out ∥ < 130 R , with j > Δ j>\Delta j > Δ . A union bound over at most m K N + 1 mKN+1 m K N + 1 vectors and q q q indices and d ≥ d w g d\ge d_{\mathrm{wg}} d ≥ d wg gives
P ( E p r o j c ) ≤ 2 q ( m K N + 1 ) exp ( − ( d − q − 1 ) α 2 2 ( 130 R ) 2 ) ≤ 1 8 . (21) \mathbb{P}(E_{\mathrm{proj}}^{c})\le2q(mKN+1)\exp\left(-\frac{(d-q-1)\alpha^2}{2(130R)^2}\right)\le\frac{1}{8}. \tag*{(21)} P ( E proj c ) ≤ 2 q ( m K N + 1 ) exp ( − 2 ( 130 R ) 2 ( d − q − 1 ) α 2 ) ≤ 8 1 . (21)
Step 4: coupling. Let E E E be the intersection of { C w , n , k = 0 for all slots } \{C_{w,n,k}=0\text{ for all slots}\} { C w , n , k = 0 for all slots } , { Δ ≤ q / 2 } \{\Delta\le q/2\} { Δ ≤ q /2 } , and E p r o j E_{\mathrm{proj}} E proj . Since m K N ε = 1 16 mKN\varepsilon=\frac{1}{16} m K N ε = 16 1 , the bounds (A.16) and (A.17) give P ( E ) ≥ 1 − 1 16 − 1 8 − 1 8 ≥ 1 2 \mathbb{P}(E)\ge1-\frac{1}{16}-\frac{1}{8}-\frac{1}{8}\ge\frac{1}{2} P ( E ) ≥ 1 − 16 1 − 8 1 − 8 1 ≥ 2 1 . On E E E the actual run coincides with the virtual run. Indeed, suppose all earlier replies coincide, so that the current queries coincide, and let x x x be the query at a slot with prefix length s ≤ Δ ≤ q / 2 s\le\Delta\le q/2 s ≤ Δ ≤ q /2 . If ∥ x ∥ ≥ 130 R \|x\|\ge130R ∥ x ∥ ≥ 130 R , both replies equal the public vector. Otherwise, ∣ ⟨ u j , x ⟩ ∣ ≤ α |\langle u_j,x\rangle|\le\alpha ∣ ⟨ u j , x ⟩ ∣ ≤ α for all j > s j>s j > s by E p r o j E_{\mathrm{proj}} E proj , so prog ( U T x ) = t ~ ≤ q / 2 \operatorname{prog}(U^{\mathsf T}x)=\tilde{t}\le q/2 prog ( U T x ) = t ~ ≤ q /2 , and both replies are given by (A.15) with the same t t t , the same B B B , and C = 0 C=0 C = 0 . By induction over the slots, the two runs and their outputs coincide.
Step 5: the risk. On E E E , let x ^ \hat{x} x ^ be the common output. If ∥ x ^ ∥ < 130 R \|\hat{x}\|<130R ∥ x ^ ∥ < 130 R , then prog ( U T x ^ ) ≤ Δ ≤ q / 2 \operatorname{prog}(U^{\mathsf T}\hat{x})\le\Delta\le q/2 prog ( U T x ^ ) ≤ Δ ≤ q /2 by E p r o j E_{\mathrm{proj}} E proj , and (A.13) gives Φ U ( x ^ ) ≥ F U ( x ^ ) ≥ ℓ δ 2 q 16 \Phi_U(\hat{x})\ge F_U(\hat{x})\ge\frac{\ell\delta^2q}{16} Φ U ( x ^ ) ≥ F U ( x ^ ) ≥ 16 ℓ δ 2 q . If ∥ x ^ ∥ ≥ 130 R \|\hat{x}\|\ge130R ∥ x ^ ∥ ≥ 130 R , then Φ U ( x ^ ) = 2 ℓ ( ∥ x ^ ∥ − R ) 2 ≥ ℓ R 2 ≥ ℓ δ 2 q 16 \Phi_U(\hat{x})=2\ell(\|\hat{x}\|-R)^2\ge\ell R^2\ge\frac{\ell\delta^2q}{16} Φ U ( x ^ ) = 2 ℓ ( ∥ x ^ ∥ − R ) 2 ≥ ℓ R 2 ≥ 16 ℓ δ 2 q . Since Φ U ≥ 0 \Phi_U\ge0 Φ U ≥ 0 , (A.11) gives
E Φ U ( x ^ ) ≥ ℓ δ 2 q 32 ≥ ℓ R 2 32 q 2 , (22) \mathbb{E}\,\Phi_U(\hat{x})\ge\frac{\ell\delta^2q}{32}\ge\frac{\ell R^2}{32q^2}, \tag*{(22)} E Φ U ( x ^ ) ≥ 32 ℓ δ 2 q ≥ 32 q 2 ℓ R 2 , (22)
where the expectation is over U U U , the seed, and the oracle variables. Averaging over U U U selects a fixed frame, that is, a fixed instance of W d \mathcal{W}_d W d by Lemma A.13.
Step 6: the rate. The quantity inside the ceiling defining q q q is at least one, so q 2 ≤ 4 max { 64 N 2 log 2 ( e m ) , 4096 ℓ K N / H s } q^2\le4\max\{64N^2\log^2(em),4096\ell KN/H_s\} q 2 ≤ 4 max { 64 N 2 log 2 ( e m ) , 4096 ℓ K N / H s } . Substituting this and ℓ = L / 1024 \ell=L/1024 ℓ = L /1024 into (A.18),
M d w g ≥ R 2 min { L 2 23 N 2 log 2 ( e m ) , H s 2 19 K N } ≥ R 2 2 23 log 2 ( e m ) min { L N 2 , H s K N } . (23) \mathcal{M}^{\mathrm{wg}}_d\ge R^2\min\left\{\frac{L}{2^{23}N^2\log^2(em)},\frac{H_s}{2^{19}KN}\right\}\ge\frac{R^2}{2^{23}\log^2(em)}\min\left\{\frac{L}{N^2},\frac{H_s}{KN}\right\}. \tag*{(23)} M d wg ≥ R 2 min { 2 23 N 2 log 2 ( e m ) L , 2 19 K N H s } ≥ 2 23 log 2 ( e m ) R 2 min { N 2 L , K N H s } . (23)
Lemma A.4 gives M d w g ≥ R 2 8 min { L , H s m K N } \mathcal{M}^{\mathrm{wg}}_d\ge\frac{R^2}{8}\min\{L,\frac{H_s}{mKN}\} M d wg ≥ 8 R 2 min { L , m K N H s } , and 1 8 ≥ 2 − 23 log 2 ( e m ) \frac{1}{8}\ge\frac{2^{-23}}{\log^2(em)} 8 1 ≥ l o g 2 ( e m ) 2 − 23 . It remains to note that
min { L , H s K N , L N 2 + H s m K N } ≤ min { H s K N , L N 2 } + min { L , H s m K N } \min\left\{L,\frac{H_s}{KN},\frac{L}{N^2}+\frac{H_s}{mKN}\right\}\le\min\left\{\frac{H_s}{KN},\frac{L}{N^2}\right\}+\min\left\{L,\frac{H_s}{mKN}\right\} min { L , K N H s , N 2 L + m K N H s } ≤ min { K N H s , N 2 L } + min { L , m K N H s }
≤ 2 max { min { H s K N , L N 2 } , min { L , H s m K N } } . \le2\max\left\{\min\left\{\frac{H_s}{KN},\frac{L}{N^2}\right\},\min\left\{L,\frac{H_s}{mKN}\right\}\right\}. ≤ 2 max { min { K N H s , N 2 L } , min { L , m K N H s } } .
Indeed, if H s K N ≤ L N 2 \frac{H_s}{KN}\le\frac{L}{N^2} K N H s ≤ N 2 L , the left-hand side is at most H s K N \frac{H_s}{KN} K N H s ; otherwise it is at most min { L , L N 2 + H s m K N } ≤ L N 2 + min { L , H s m K N } \min\{L,\frac{L}{N^2}+\frac{H_s}{mKN}\}\le\frac{L}{N^2}+\min\{L,\frac{H_s}{mKN}\} min { L , N 2 L + m K N H s } ≤ N 2 L + min { L , m K N H s } . Hence the larger of the two lower bounds is at least R 2 2 24 log 2 ( e m ) min { L , H s K N , L N 2 + H s m K N } \frac{R^2}{2^{24}\log^2(em)}\min\{L,\frac{H_s}{KN},\frac{L}{N^2}+\frac{H_s}{mKN}\} 2 24 l o g 2 ( e m ) R 2 min { L , K N H s , N 2 L + m K N H s } .
Sample reuse
In this construction every sample function carries its own independent Bernoulli coins on the edges of a hidden chain. Repeated queries of the same label therefore reveal nothing beyond these coins. Throughout this subsection,
p = min { L H s , log ( e m ) K } , ν = N ⌈ log m + ( e − 1 ) K p ⌉ , s = 1 + min { K N , ⌈ 4 ν ⌉ } , p=\min\left\{\frac{L}{H_s},\frac{\log(em)}{K}\right\},\qquad\nu=N\left\lceil\log m+(e-1)Kp\right\rceil,\qquad s=1+\min\{KN,\lceil4\nu\rceil\}, p = min { H s L , K log ( e m ) } , ν = N ⌈ log m + ( e − 1 ) K p ⌉ , s = 1 + min { K N , ⌈ 4 ν ⌉} ,
r = 2 ( s + 1 ) , a = R r , α = a 4 ( s + 1 ) , ρ = 32 H s R L . (24) r=2(s+1),\qquad a=\frac{R}{\sqrt{r}},\qquad\alpha=\frac{a}{4(s+1)},\qquad\rho=\frac{32H_sR}{L}. \tag*{(24)} r = 2 ( s + 1 ) , a = r R , α = 4 ( s + 1 ) a , ρ = L 32 H s R . (24)
and
d p o o l = r + 2 + ⌈ 8 ( 2 R + ρ ) 2 α 2 log ( 8 ( m K N + 1 ) r ) ⌉ . (25) d_{\mathrm{pool}}=r+2+\left\lceil\frac{8(2R+\rho)^2}{\alpha^2}\log\bigl(8(mKN+1)r\bigr)\right\rceil. \tag*{(25)} d pool = r + 2 + ⌈ α 2 8 ( 2 R + ρ ) 2 log ( 8 ( m K N + 1 ) r ) ⌉ . (25)
Since ( 2 R + ρ ) / α = 4 ( s + 1 ) r ( 2 + 32 H s / L ) (2R+\rho)/\alpha=4(s+1)\sqrt{r}(2+32H_s/L) ( 2 R + ρ ) / α = 4 ( s + 1 ) r ( 2 + 32 H s / L ) , the threshold d p o o l d_{\mathrm{pool}} d pool depends only on m m m , K K K , N N N , and H s / L H_s/L H s / L .
The chain. For t ≥ 0 t \ge0 t ≥ 0 let η a ( t ) = t 2 \eta_a(t) = t^2 η a ( t ) = t 2 if t ≤ a t \le a t ≤ a and η a ( t ) = 2 a t − a 2 \eta_a(t) = 2at-a^2 η a ( t ) = 2 a t − a 2 if t > a t > a t > a , and for β ≥ 0 \beta\ge0 β ≥ 0 let ψ β ( t ) = η a ( ( ∣ t ∣ − β ) + ) \psi_\beta(t) = \eta_a((|t|-\beta)_+) ψ β ( t ) = η a (( ∣ t ∣ − β ) + ) . The function ψ β \psi_\beta ψ β is convex and continuously differentiable, ψ β ′ \psi'_\beta ψ β ′ is 2-Lipschitz, ∣ ψ β ′ ′ ∣ ≤ 2 a |\psi''_\beta| \le2a ∣ ψ β ′′ ∣ ≤ 2 a , and ψ β \psi_\beta ψ β and ψ β ′ \psi'_\beta ψ β ′ vanish on [ − β , β ] [-\beta,\beta] [ − β , β ] . Let V = ( v 1 , … , v r ) V=(v_1,\ldots,v_r) V = ( v 1 , … , v r ) be an orthonormal frame in R d \mathbb{R}^d R d . A sample is a vector of coins ξ = ( b 1 , … , b r − 1 ) \xi=(b_1,\ldots,b_{r-1}) ξ = ( b 1 , … , b r − 1 ) with independent Bernoulli(p p p ) coordinates, and
f V , ξ ( x ) = H s 256 [ p ψ 0 ( ⟨ v 1 , x ⟩ − a ) + ∑ l = 1 r − 1 b l ψ α ( ⟨ v l − v l + 1 , x ⟩ ) ] . (26) f_{V,\xi}(x)=\frac{H_s}{256}\left[p\psi_0(\langle v_1,x\rangle-a)+\sum_{l=1}^{r-1}b_l\psi_\alpha(\langle v_l-v_{l+1},x\rangle)\right]. \tag*{(26)} f V , ξ ( x ) = 256 H s [ p ψ 0 (⟨ v 1 , x ⟩ − a ) + l = 1 ∑ r − 1 b l ψ α (⟨ v l − v l + 1 , x ⟩) ] . (26)
Let f V f_V f V denote the same expression with p p p and all b l b_l b l replaced by one, so that E ξ f V , ξ = p f V \mathbb{E}_{\xi}f_{V,\xi}=pf_V E ξ f V , ξ = p f V .
Lemma A.16 (The chain with coins). Every function (A.22) is nonnegative and convex, has an H s 32 \frac{H_s}{32} 32 H s -Lipschitz gradient with ∥ ∇ f V , ξ ∥ ≤ H s R 64 \|\nabla f_{V,\xi}\|\le\frac{H_sR}{64} ∥∇ f V , ξ ∥ ≤ 64 H s R , and vanishes at x ∗ = a ∑ l = 1 r v l x^*=a\sum_{l=1}^r v_l x ∗ = a ∑ l = 1 r v l , where ∥ x ∗ ∥ = R \|x^*\|=R ∥ x ∗ ∥ = R .
Proof. Let W : R d → R r W:\mathbb{R}^d\to\mathbb{R}^r W : R d → R r map x x x to ( ⟨ v 1 , x ⟩ , ⟨ v 1 − v 2 , x ⟩ , … , ⟨ v r − 1 − v r , x ⟩ ) (\langle v_1,x\rangle,\langle v_1-v_2,x\rangle,\ldots,\langle v_{r-1}-v_r,x\rangle) (⟨ v 1 , x ⟩ , ⟨ v 1 − v 2 , x ⟩ , … , ⟨ v r − 1 − v r , x ⟩) . In the coordinates of the frame, w 1 2 + ∑ l < r ( w l − w l + 1 ) 2 ≤ 4 ∑ l w l 2 w_1^2+\sum_{l<r}(w_l-w_{l+1})^2\le4\sum_l w_l^2 w 1 2 + ∑ l < r ( w l − w l + 1 ) 2 ≤ 4 ∑ l w l 2 , so ∥ W ∥ ≤ 2 \|W\|\le2 ∥ W ∥ ≤ 2 . The gradient of (A.22) is H s 256 W ⊤ Ψ ( x ) \frac{H_s}{256}W^\top\Psi(x) 256 H s W ⊤ Ψ ( x ) , where Ψ ( x ) ∈ R r \Psi(x)\in\mathbb{R}^r Ψ ( x ) ∈ R r collects the derivatives of the scalar terms multiplied by their coefficients in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Each entry of Ψ \Psi Ψ is 2-Lipschitz as a function of the corresponding entry of W x Wx W x and is bounded by 2 a 2a 2 a , so ∥ ∇ f V , ξ ( x ) − ∇ f V , ξ ( y ) ∥ ≤ H s 256 ⋅ 2 ⋅ 2 ⋅ 2 ∥ x − y ∥ = H s 32 ∥ x − y ∥ \|\nabla f_{V,\xi}(x)-\nabla f_{V,\xi}(y)\|\le\frac{H_s}{256}\cdot2\cdot2\cdot2\|x-y\|=\frac{H_s}{32}\|x-y\| ∥∇ f V , ξ ( x ) − ∇ f V , ξ ( y ) ∥ ≤ 256 H s ⋅ 2 ⋅ 2 ⋅ 2∥ x − y ∥ = 32 H s ∥ x − y ∥ and ∥ ∇ f V , ξ ∥ ≤ H s 256 ⋅ 2 ⋅ 2 a r = H s R 64 \|\nabla f_{V,\xi}\|\le\frac{H_s}{256}\cdot2\cdot2a\sqrt{r}=\frac{H_sR}{64} ∥∇ f V , ξ ∥ ≤ 256 H s ⋅ 2 ⋅ 2 a r = 64 H s R . Convexity and nonnegativity follow from the scalar functions. Finally, ⟨ v 1 , x ∗ ⟩ = a \langle v_1,x^*\rangle=a ⟨ v 1 , x ∗ ⟩ = a and ⟨ v l − v l + 1 , x ∗ ⟩ = 0 \langle v_l-v_{l+1},x^*\rangle=0 ⟨ v l − v l + 1 , x ∗ ⟩ = 0 , and ∥ x ∗ ∥ = a r = R \|x^*\|=a\sqrt{r}=R ∥ x ∗ ∥ = a r = R .
The instance. Apply Lemma A.3 with c = L / 32 c=L/32 c = L /32 , L φ = H s / 32 L_\varphi=H_s/32 L φ = H s /32 , M φ = H s R / 64 M_\varphi=H_sR/64 M φ = H s R /64 , and ρ \rho ρ from (A.20); indeed 64 M φ / c = ρ ≥ 32 R 64M_\varphi/c=\rho\ge32R 64 M φ / c = ρ ≥ 32 R . The sample functions are f ~ V , ξ = T f V , ξ \widetilde f_{V,\xi}=T f_{V,\xi} f V , ξ = T f V , ξ . They are nonnegative, convex, minimized at x ∗ x^* x ∗ , and have 6 H s 32 \frac{6H_s}{32} 32 6 H s -Lipschitz gradients. Since T T T is affine, the population is F V = E ξ f ~ V , ξ = T ( p f V ) F_V=\mathbb{E}_{\xi}\widetilde f_{V,\xi}=T(pf_V) F V = E ξ f V , ξ = T ( p f V ) , and Lemma A.3 applied to p f V pf_V p f V (with L φ = p H s / 32 ≤ L / 32 L_\varphi=pH_s/32\le L/32 L φ = p H s /32 ≤ L /32 ) shows that F V F_V F V has an L L L -Lipschitz gradient and
F V ≥ p f V . (27) F_V\ge pf_V. \tag*{(27)} F V ≥ p f V . (27)
Thus, for every frame, ( f ~ V , ξ ) (\widetilde f_{V,\xi}) ( f V , ξ ) is an instance of S d S_d S d . For ∥ x ∥ ≥ 2 R + ρ \|x\|\ge2R+\rho ∥ x ∥ ≥ 2 R + ρ , all sample values and gradients equal those of L 16 ( ∥ x ∥ − R ) 2 \frac{L}{16}(\|x\|-R)^2 16 L ( ∥ x ∥ − R ) 2 .
A simulated run. Fix an algorithm, and let V V V be uniformly distributed on the orthonormal r r r -frames, independently of the pool and of the seed. We strengthen the algorithm in three ways, each of which can only reduce its risk: every reply also contains the sample value; complete local histories are exchanged at the end of every round; and the oracle makes the following disclosures. Each worker w w w has a prefix length j w j_w j w , meaning that it knows v 1 , … , v j w v_1,\ldots,v_{j_w} v 1 , … , v j w ; initially j w = 1 j_w=1 j w = 1 for all w w w , and at the end of every round all disclosures are shared and every j w j_w j w is replaced by max w j w \max_w j_w max w j w . At a query ( i , x ) (i,x) ( i , x ) of worker w w w with a valid label and pre-query prefix length j < r j<r j < r , the oracle discloses to w w w the coins b l b_l b l of ξ i \xi_i ξ i with l < j l<j l < j and inspects b j b_j b j ; if b j = 1 b_j=1 b j = 1 , it discloses v j + 1 v_{j+1} v j + 1 and sets j w = j + 1 j_w=j+1 j w = j + 1 . A query with a valid label and j = r j=r j = r discloses all coins of ξ i \xi_i ξ i , and a query with an invalid label discloses nothing.
The true strengthened oracle returns the value and gradient of f ~ V , ξ i \widetilde f_{V,\xi_i} f V , ξ i at x x x . The simulated oracle makes the same disclosures and returns: the public radial reply if ∥ x ∥ ≥ 2 R + ρ \|x\|\ge2R+\rho ∥ x ∥ ≥ 2 R + ρ ; the true reply if j = r j=r j = r ; and otherwise the value and gradient at x x x of T f V , ξ i [ j ] T f_{V,\xi_i}^{[j]} T f V , ξ i [ j ] , where
f V , ξ i [ j ] ( x ) = H s 256 [ p ψ 0 ( ⟨ v 1 , x ⟩ − a ) + ∑ l = 1 j b l ψ α ( ⟨ v l − v l + 1 , x ⟩ ) ] , f_{V,\xi_i}^{[j]}(x)=\frac{H_s}{256}\left[p\psi_0(\langle v_1,x\rangle-a)+\sum_{l=1}^{j}b_l\psi_\alpha(\langle v_l-v_{l+1},x\rangle)\right], f V , ξ i [ j ] ( x ) = 256 H s [ p ψ 0 (⟨ v 1 , x ⟩ − a ) + l = 1 ∑ j b l ψ α (⟨ v l − v l + 1 , x ⟩) ] ,
the term l = j l=j l = j being omitted when b j = 0 b_j=0 b j = 0 . A simulated reply uses only the frame vectors and coins disclosed to the worker after the query. If ∥ x ∥ ≥ 2 R + ρ \|x\|\ge2R+\rho ∥ x ∥ ≥ 2 R + ρ , or if
∣ ⟨ v l , x ⟩ ∣ ≤ α 2 for every l > j , (28) |\langle v_l,x\rangle|\le\frac{\alpha}{2}\quad\text{for every }l>j, \tag*{(28)} ∣ ⟨ v l , x ⟩ ∣ ≤ 2 α for every l > j , (28)
the simulated and true replies coincide. Indeed, under (A.24), ∣ ⟨ v l − v l + 1 , x ⟩ ∣ ≤ α |\langle v_l-v_{l+1},x\rangle|\le\alpha ∣ ⟨ v l − v l + 1 , x ⟩ ∣ ≤ α for l > j l>j l > j , so the terms with l > j l>j l > j in (A.22) vanish together with their gradients at x x x ; hence f V , ξ f_{V,\xi} f V , ξ and f V , ξ [ j ] f_{V,\xi}^{[j]} f V , ξ [ j ] have the same value and gradient at x x x , and by Lemma A.3, Part 3, so do their images under T T T . The simulated run is the run of the strengthened algorithm with the simulated oracle; it is defined for every frame, pool, and seed.
Lemma A.17 (Progress of the simulated run). Let Z Z Z be the total increase of the common prefix length over the N N N rounds of the simulated run. Then E Z ≤ ν \mathbb{E}Z\le\nu E Z ≤ ν and Z ≤ K N Z\le KN Z ≤ K N .
Proof. Known frontier coins are zero. Let j j j be the current prefix length of worker w w w , and suppose a coin b j b_j b j of some sample has been disclosed to w w w . It was disclosed either at a query with pre-query prefix length j ′ > j j' > j j ′ > j , which is impossible because prefix lengths do not decrease and the prefix length of w w w would exceed j j j , or as the inspected coin at a query with pre-query prefix length j j j ; in the latter case, if it were one, that prefix length would have become j + 1 j+1 j + 1 , and the prefix length of w w w would exceed j j j as well. Coins disclosed to other workers in the current round are not disclosed to w w w . Hence b j = 0 b_j=0 b j = 0 .
One round. Fix a round, let G \mathcal{G} G be the shared information at its start, and let G w , k \mathcal{G}_{w,k} G w , k be generated by G \mathcal{G} G and the first k k k queries of worker w w w in the round, with their replies and disclosures. Let I w , k ∈ { 0 , 1 } I_{w,k} \in\{0,1\} I w , k ∈ { 0 , 1 } indicate that the k k k -th query of w w w advances its prefix (I w , k = 0 I_{w,k}=0 I w , k = 0 if the query is not made). The information in G w , k − 1 \mathcal{G}_{w,k-1} G w , k − 1 is a function of the seed, of V V V , and of the coins disclosed to w w w , and the label i i i and the frontier j j j of the k k k -th query are G w , k − 1 \mathcal{G}_{w,k-1} G w , k − 1 -measurable. If the coin b j b_j b j of ξ i \xi_i ξ i has been disclosed to w w w , it is zero and I w , k = 0 I_{w,k}=0 I w , k = 0 . Otherwise, since the coins are independent of each other, of V V V , and of the seed, and the coins disclosed to w w w were chosen sequentially from previously disclosed information, b j b_j b j is Bernoulli(p p p ) conditional on G w , k − 1 \mathcal{G}_{w,k-1} G w , k − 1 . In both cases E [ e I w , k ∣ G w , k − 1 ] ≤ 1 − p + p e \mathbb{E}[e^{I_{w,k}}\mid\mathcal{G}_{w,k-1}] \le1-p+pe E [ e I w , k ∣ G w , k − 1 ] ≤ 1 − p + p e . All workers start the round with the same prefix length, so the increase of the common prefix length in the round is max w ∑ k I w , k \max_w \sum_k I_{w,k} max w ∑ k I w , k , and Lemma A.2 bounds its conditional expectation given G \mathcal{G} G by log m + ( e − 1 ) K p \log m+(e-1)Kp log m + ( e − 1 ) K p . Summing over the rounds gives E Z ≤ ν \mathbb{E}Z\le\nu E Z ≤ ν . Each round increases the common prefix length by at most K K K , so Z ≤ K N Z\le KN Z ≤ K N .
Lemma A.18 (Hidden projections). Let d ≥ d p o o l d\ge d_{\mathrm{pool}} d ≥ d pool . With probability at least 3 / 4 3/4 3/4 , every query x x x of the simulated run with ∥ x ∥ < 2 R + ρ \lVert x\rVert<2R+\rho ∥ x ∥ < 2 R + ρ satisfies (A.24) with its pre-query prefix length j j j , and its output x ^ \widehat{x} x , if ∥ x ^ ∥ < 2 R + ρ \lVert\widehat{x}\rVert<2R+\rho ∥ x ∥ < 2 R + ρ , satisfies ∣ ⟨ v l , x ^ ⟩ ∣ ≤ α / 2 \lvert\langle v_l,\widehat{x}\rangle\rvert\le\alpha/2 ∣⟨ v l , x ⟩∣ ≤ α /2 for every l l l larger than the final common prefix length. On this event, the simulated run and the run with the true strengthened oracle coincide.
Proof. Fix the seed and the pool. Consider a query slot, let j ˉ \bar{j} j ˉ be the pre-query prefix length of the querying worker in the simulated run, and let j ≤ r j\le r j ≤ r . We claim that the event { j ˉ ≤ j } \{\bar{j}\le j\} { j ˉ ≤ j } , and on this event the query, are determined by v 1 , … , v j v_1,\ldots,v_j v 1 , … , v j . To see this, reconstruct the simulated run from v 1 , … , v j v_1,\ldots,v_j v 1 , … , v j round by round, simulating the workers of a round separately, which is possible because they do not interact within a round. If in a round before the round of the slot some worker would need v j + 1 v_{j+1} v j + 1 , the common prefix length at the start of the round of the slot exceeds j j j , so j ˉ > j \bar{j}>j j ˉ > j . If in the round of the slot the querying worker would need v j + 1 v_{j+1} v j + 1 before the slot, again j ˉ > j \bar{j}>j j ˉ > j . Otherwise all replies that the query depends on are computable from v 1 , … , v j v_1,\ldots,v_j v 1 , … , v j , so is the query, and j ˉ ≤ j \bar{j}\le j j ˉ ≤ j . The same argument applies to the output and the final common prefix length.
For a uniformly distributed frame, conditional on v 1 , … , v j v_1,\ldots,v_j v 1 , … , v j every v l v_l v l with l > j l>j l > j is uniformly distributed on the unit sphere of the ( d − j ) (d-j) ( d − j ) -dimensional space span { v 1 , … , v j } ⊥ \operatorname{span}\{v_1,\ldots,v_j\}^{\perp} span { v 1 , … , v j } ⊥ . Since the event { j ˉ = j } \{\bar{j}=j\} { j ˉ = j } and the query on it are determined by v 1 , … , v j v_1,\ldots,v_j v 1 , … , v j , Lemma A.1, applied to the query capped to { ∥ x ∥ < 2 R + ρ } \{\lVert x\rVert<2R+\rho\} {∥ x ∥ < 2 R + ρ } with the threshold α / 2 \alpha/2 α /2 , gives for every l > j l>j l > j
P ( ∣ ⟨ v l , x ⟩ ∣ > α 2 , ∥ x ∥ < 2 R + ρ ) ≤ 2 exp ( − ( d − r − 1 ) α 2 8 ( 2 R + ρ ) 2 ) . \mathbb{P}\left(\lvert\langle v_l,x\rangle\rvert>\frac{\alpha}{2},\ \lVert x\rVert<2R+\rho\right)\le2\exp\left(-\frac{(d-r-1)\alpha^2}{8(2R+\rho)^2}\right). P ( ∣⟨ v l , x ⟩∣ > 2 α , ∥ x ∥ < 2 R + ρ ) ≤ 2 exp ( − 8 ( 2 R + ρ ) 2 ( d − r − 1 ) α 2 ) .
The same bound holds for the output. There are at most m K N mKN m K N queries, one output, and at most r r r indices l l l for each, so by a union bound and (A.21) the failure probability is at most 2 ( m K N + 1 ) r exp ( − ( d − r − 1 ) α 2 8 ( 2 R + ρ ) 2 ) ≤ 1 4 2(mKN+1)r\exp\left(-\frac{(d-r-1)\alpha^2}{8(2R+\rho)^2}\right)\le\frac14 2 ( m K N + 1 ) r exp ( − 8 ( 2 R + ρ ) 2 ( d − r − 1 ) α 2 ) ≤ 4 1 . Integrating over the seed and the pool gives the same bound.
On this event, suppose the two runs coincide before some query. Then the query is the same in both runs and, since either ∥ x ∥ ≥ 2 R + ρ \lVert x\rVert\ge2R+\rho ∥ x ∥ ≥ 2 R + ρ or (A.24) holds, the two replies coincide; the disclosures coincide because they follow the same rule. By induction, the two runs coincide.
Lemma A.19 (Loss gap). If ∣ ⟨ v s + 1 , x ⟩ ∣ ≤ α / 2 \lvert\langle v_{s+1},x\rangle\rvert\le\alpha/2 ∣⟨ v s + 1 , x ⟩∣ ≤ α /2 , then
f V ( x ) ≥ 9 H s R 2 8192 ( s + 1 ) 2 . f_V(x)\ge\frac{9H_sR^2}{8192(s+1)^2}. f V ( x ) ≥ 8192 ( s + 1 ) 2 9 H s R 2 . Proof. Put t 0 = ∣ a − ⟨ v 1 , x ⟩ ∣ t_0=\lvert a-\langle v_1,x\rangle\rvert t 0 = ∣ a − ⟨ v 1 , x ⟩∣ and t l = ( ∣ ⟨ v l − v l + 1 , x ⟩ ∣ − α ) + t_l=(\lvert\langle v_l-v_{l+1},x\rangle\rvert-\alpha)_+ t l = (∣⟨ v l − v l + 1 , x ⟩∣ − α ) + for 1 ≤ l ≤ s 1\le l\le s 1 ≤ l ≤ s ; note that s ≤ r − 1 s\le r-1 s ≤ r − 1 . Since a = ( a − ⟨ v 1 , x ⟩ ) + ∑ l = 1 s ⟨ v l − v l + 1 , x ⟩ + ⟨ v s + 1 , x ⟩ a=(a-\langle v_1,x\rangle)+\sum_{l=1}^s\langle v_l-v_{l+1},x\rangle+\langle v_{s+1},x\rangle a = ( a − ⟨ v 1 , x ⟩) + ∑ l = 1 s ⟨ v l − v l + 1 , x ⟩ + ⟨ v s + 1 , x ⟩ ,
a ≤ t 0 + ∑ l = 1 s ( t l + α ) + α 2 , ∑ l = 0 s t l ≥ a − ( s + 1 2 ) α ≥ 3 a 4 . a\le t_0+\sum_{l=1}^s(t_l+\alpha)+\frac{\alpha}{2},\qquad\sum_{l=0}^s t_l\ge a-\left(s+\frac12\right)\alpha\ge\frac{3a}{4}. a ≤ t 0 + l = 1 ∑ s ( t l + α ) + 2 α , l = 0 ∑ s t l ≥ a − ( s + 2 1 ) α ≥ 4 3 a . The anchor term of f V f_V f V equals η a ( t 0 ) \eta_a(t_0) η a ( t 0 ) , and the edge terms with 1 ≤ l ≤ s 1\le l\le s 1 ≤ l ≤ s equal η a ( t l ) \eta_a(t_l) η a ( t l ) . If some t l ≥ a t_l\ge a t l ≥ a , then η a ( t l ) ≥ a 2 16 ( s + 1 ) \eta_a(t_l)\ge\frac{a^2}{16(s+1)} η a ( t l ) ≥ 16 ( s + 1 ) a 2 . Otherwise η a ( t l ) = t l 2 \eta_a(t_l)=t_l^2 η a ( t l ) = t l 2 for all l ≤ s l\le s l ≤ s , and the Cauchy–Schwarz inequality gives ∑ l = 0 s t l 2 ≥ 9 a 2 16 ( s + 1 ) \sum_{l=0}^s t_l^2\ge\frac{9a^2}{16(s+1)} ∑ l = 0 s t l 2 ≥ 16 ( s + 1 ) 9 a 2 . In both cases f V ( x ) ≥ H s 256 ⋅ 9 a 2 16 ( s + 1 ) f_V(x)\ge\frac{H_s}{256}\cdot\frac{9a^2}{16(s+1)} f V ( x ) ≥ 256 H s ⋅ 16 ( s + 1 ) 9 a 2 , and a 2 = R 2 2 ( s + 1 ) a^2=\frac{R^2}{2(s+1)} a 2 = 2 ( s + 1 ) R 2 .
Theorem A.20 (Lower bound with sample reuse). Let S ∈ N ∪ { ∞ } S \in\mathbb{N} \cup\{\infty\} S ∈ N ∪ { ∞ } , S ≥ 1 S \ge1 S ≥ 1 . For every d ≥ d p o o l d \ge d_{\mathrm{pool}} d ≥ d pool ,
M d p o o l ( S ) ≥ R 2 10 6 log 2 ( e m ) min { L , min { L , H s / K } N 2 + H s min { S , m K N } } . \mathcal{M}^{\mathrm{pool}}_d(S) \ge\frac{R^2}{10^6\log^2(em)}\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{\min\{S,mKN\}}\right\}. M d pool ( S ) ≥ 1 0 6 log 2 ( e m ) R 2 min { L , N 2 min { L , H s / K } + min { S , m K N } H s } .
Proof. The chain term. By Lemma A.17, P ( Z ≤ s − 1 ) ≥ 3 / 4 \mathbb{P}(Z \le s-1) \ge3/4 P ( Z ≤ s − 1 ) ≥ 3/4 : if s − 1 = K N s-1=KN s − 1 = K N this holds surely, and otherwise s − 1 = ⌈ 4 ν ⌉ s-1=\lceil4\nu\rceil s − 1 = ⌈ 4 ν ⌉ and Markov’s inequality applies (ν > 0 \nu>0 ν > 0 because p > 0 p>0 p > 0 ). The final common prefix length is 1 + Z 1+Z 1 + Z . Intersect { Z ≤ s − 1 } \{Z\le s-1\} { Z ≤ s − 1 } with the event of Lemma A.18; the intersection has probability at least 1 / 2 1/2 1/2 . On it, the output x ^ \hat{x} x ^ of the strengthened algorithm equals the simulated output, and its final common prefix length is at most s s s . If ∥ x ^ ∥ < 2 R + ρ \|\hat{x}\|<2R+\rho ∥ x ^ ∥ < 2 R + ρ , then ∣ ⟨ v s + 1 , x ^ ⟩ ∣ ≤ α / 2 |\langle v_{s+1},\hat{x}\rangle|\le\alpha/2 ∣ ⟨ v s + 1 , x ^ ⟩ ∣ ≤ α /2 , and (A.23) and Lemma A.19 give F V ( x ^ ) ≥ 9 p H s R 2 8192 ( s + 1 ) 2 F_V(\hat{x})\ge\frac{9pH_sR^2}{8192(s+1)^2} F V ( x ^ ) ≥ 8192 ( s + 1 ) 2 9 p H s R 2 . If ∥ x ^ ∥ ≥ 2 R + ρ \|\hat{x}\|\ge2R+\rho ∥ x ^ ∥ ≥ 2 R + ρ , then F V ( x ^ ) = L 16 ( ∥ x ^ ∥ − R ) 2 ≥ L 16 ρ 2 = 64 H s 2 R 2 L ≥ 64 H s R 2 F_V(\hat{x})=\frac{L}{16}(\|\hat{x}\|-R)^2\ge\frac{L}{16}\rho^2=\frac{64H_s^2R^2}{L}\ge64H_sR^2 F V ( x ^ ) = 16 L ( ∥ x ^ ∥ − R ) 2 ≥ 16 L ρ 2 = L 64 H s 2 R 2 ≥ 64 H s R 2 , which is larger. Hence
E F V ( x ^ ) ≥ 9 p H s R 2 16384 ( s + 1 ) 2 , \mathbb{E}F_V(\hat{x})\ge\frac{9pH_sR^2}{16384(s+1)^2}, E F V ( x ^ ) ≥ 16384 ( s + 1 ) 2 9 p H s R 2 ,
and averaging over V V V selects a fixed instance. By (A.20), K p ≤ log ( e m ) Kp\le\log(em) K p ≤ log ( e m ) , so ν ≤ e N log ( e m ) < 3 N log ( e m ) \nu\le eN\log(em)<3N\log(em) ν ≤ e N log ( e m ) < 3 N log ( e m ) and s + 1 ≤ 3 + 4 ν ≤ 15 N log ( e m ) s+1\le3+4\nu\le15N\log(em) s + 1 ≤ 3 + 4 ν ≤ 15 N log ( e m ) . Also p H s = min { L , H s log ( e m ) / K } ≥ min { L , H s / K } pH_s=\min\{L,H_s\log(em)/K\}\ge\min\{L,H_s/K\} p H s = min { L , H s log ( e m ) / K } ≥ min { L , H s / K } . Therefore
M d p o o l ( S ) ≥ 9 R 2 min { L , H s / K } 16384 ⋅ 225 N 2 log 2 ( e m ) = R 2 min { L , H s / K } 409600 N 2 log 2 ( e m ) . \mathcal{M}^{\mathrm{pool}}_d(S)\ge\frac{9R^2\min\{L,H_s/K\}}{16384\cdot225N^2\log^2(em)}
=\frac{R^2\min\{L,H_s/K\}}{409600N^2\log^2(em)}. M d pool ( S ) ≥ 16384 ⋅ 225 N 2 log 2 ( e m ) 9 R 2 min { L , H s / K } = 409600 N 2 log 2 ( e m ) R 2 min { L , H s / K } .
The pool size did not enter this argument: a finite pool consists of S S S independent coin vectors, and invalid labels disclose nothing.
Combination. Lemma A.4 gives M d p o o l ( S ) ≥ R 2 8 min { L , H s min { S , m K N } } \mathcal{M}^{\mathrm{pool}}_d(S)\ge\frac{R^2}{8}\min\left\{L,\frac{H_s}{\min\{S,mKN\}}\right\} M d pool ( S ) ≥ 8 R 2 min { L , m i n { S , m K N } H s } . The larger of the two bounds is at least their average, and 1 16 ≥ 1 819200 log 2 ( e m ) \frac{1}{16}\ge\frac{1}{819200\log^2(em)} 16 1 ≥ 819200 l o g 2 ( e m ) 1 , so
M d p o o l ( S ) ≥ R 2 819200 log 2 ( e m ) [ min { L , H s / K } N 2 + min { L , H s min { S , m K N } } ] ≥ R 2 10 6 log 2 ( e m ) min { L , min { L , H s / K } N 2 + H s min { S , m K N } } , \begin{aligned}
\mathcal{M}^{\mathrm{pool}}_d(S)\ge\frac{R^2}{819200\log^2(em)}
\left[
\frac{\min\{L,H_s/K\}}{N^2}+\min\left\{L,\frac{H_s}{\min\{S,mKN\}}\right\}
\right] \\
&\ge\frac{R^2}{10^6\log^2(em)}
\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{\min\{S,mKN\}}\right\},
\end{aligned} M d pool ( S ) ≥ 819200 log 2 ( e m ) R 2 [ N 2 min { L , H s / K } + min { L , min { S , m K N } H s } ] ≥ 1 0 6 log 2 ( e m ) R 2 min { L , N 2 min { L , H s / K } + min { S , m K N } H s } ,
where the last inequality holds exactly as in the proof of Theorem A.11.
Upper bounds with fresh samples
In this appendix the oracle returns, at every query x x x , a vector g ( x , ξ ) g(x,\xi) g ( x , ξ ) with a fresh sample ξ \xi ξ and the two properties in (2.1), that is, Assumption 1 holds. By Lemma 4.1, every result below applies to the sample-gradient oracle of Assumption 2. We normalize F ( x ∗ ) = 0 F(x^*)=0 F ( x ∗ ) = 0 . If H \mathcal{H} H denotes the history before a query x x x , then x x x is H \mathcal{H} H -measurable and ξ \xi ξ is independent of H \mathcal{H} H , so that (2.1) holds conditionally:
E [ g ( x , ξ ) ∣ H ] = ∇ F ( x ) , E [ ∥ g ( x , ξ ) ∥ 2 ∣ H ] ≤ 2 H s F ( x ) . (29) \mathbb{E}[g(x,\xi)\mid\mathcal{H}]=\nabla F(x),\qquad
\mathbb{E}[\|g(x,\xi)\|^2\mid\mathcal{H}]\le2H_sF(x). \tag*{(29)} E [ g ( x , ξ ) ∣ H ] = ∇ F ( x ) , E [ ∥ g ( x , ξ ) ∥ 2 ∣ H ] ≤ 2 H s F ( x ) . (29)
Preliminaries
Lemma B.1 (Smooth functions). Let ψ : R d → R \psi:\mathbb{R}^d\to\mathbb{R} ψ : R d → R be differentiable with an M M M -Lipschitz gradient, M > 0 M>0 M > 0 , and let u , v ∈ R d u,v\in\mathbb{R}^d u , v ∈ R d .
ψ ( v ) ≤ ψ ( u ) + ⟨ ∇ ψ ( u ) , v − u ⟩ + M 2 ∥ v − u ∥ 2 \psi(v)\le\psi(u)+\langle\nabla\psi(u),v-u\rangle+\frac{M}{2}\|v-u\|^2 ψ ( v ) ≤ ψ ( u ) + ⟨ ∇ ψ ( u ) , v − u ⟩ + 2 M ∥ v − u ∥ 2 .
If ψ ≥ 0 \psi\ge0 ψ ≥ 0 , then ∥ ∇ ψ ( u ) ∥ 2 ≤ 2 M ψ ( u ) \|\nabla\psi(u)\|^2\le2M\psi(u) ∥∇ ψ ( u ) ∥ 2 ≤ 2 M ψ ( u ) and ψ ( v ) ≤ 2 ψ ( u ) + M ∥ v − u ∥ 2 \psi(v)\le2\psi(u)+M\|v-u\|^2 ψ ( v ) ≤ 2 ψ ( u ) + M ∥ v − u ∥ 2 .
If ψ \psi ψ is convex, then ψ ( v ) − ψ ( u ) − ⟨ ∇ ψ ( u ) , v − u ⟩ ≥ 1 2 M ∥ ∇ ψ ( v ) − ∇ ψ ( u ) ∥ 2 \psi(v)-\psi(u)-\langle\nabla\psi(u),v-u\rangle\ge\frac{1}{2M}\|\nabla\psi(v)-\nabla\psi(u)\|^2 ψ ( v ) − ψ ( u ) − ⟨ ∇ ψ ( u ) , v − u ⟩ ≥ 2 M 1 ∥∇ ψ ( v ) − ∇ ψ ( u ) ∥ 2 and ∥ ∇ ψ ( v ) − ∇ ψ ( u ) ∥ 2 ≤ M ⟨ ∇ ψ ( v ) − ∇ ψ ( u ) , v − u ⟩ \|\nabla\psi(v)-\nabla\psi(u)\|^2\le M\langle\nabla\psi(v)-\nabla\psi(u),v-u\rangle ∥∇ ψ ( v ) − ∇ ψ ( u ) ∥ 2 ≤ M ⟨ ∇ ψ ( v ) − ∇ ψ ( u ) , v − u ⟩ .
If x ∗ x^* x ∗ minimizes ψ \psi ψ , then ∇ ψ ( x ∗ ) = 0 \nabla\psi(x^*)=0 ∇ ψ ( x ∗ ) = 0 , 0 ≤ ψ ( u ) − ψ ( x ∗ ) ≤ M 2 ∥ u − x ∗ ∥ 2 0\le\psi(u)-\psi(x^*)\le\frac{M}{2}\|u-x^*\|^2 0 ≤ ψ ( u ) − ψ ( x ∗ ) ≤ 2 M ∥ u − x ∗ ∥ 2 , and ∥ ∇ ψ ( u ) ∥ ≤ M ∥ u − x ∗ ∥ \|\nabla\psi(u)\|\le M\|u-x^*\| ∥∇ ψ ( u ) ∥ ≤ M ∥ u − x ∗ ∥ .
Proof. Part 1. By the fundamental theorem of calculus and the Lipschitz property of ∇ ψ \nabla\psi ∇ ψ ,
ψ ( v ) − ψ ( u ) − ⟨ ∇ ψ ( u ) , v − u ⟩ = ∫ 0 1 ⟨ ∇ ψ ( u + s ( v − u ) ) − ∇ ψ ( u ) , v − u ⟩ d s ≤ ∫ 0 1 M s ∥ v − u ∥ 2 d s = M 2 ∥ v − u ∥ 2 . \psi(v)-\psi(u)-\langle\nabla\psi(u),v-u\rangle
=\int_0^1\langle\nabla\psi(u+s(v-u))-\nabla\psi(u),v-u\rangle\,ds
\le\int_0^1Ms\|v-u\|^2\,ds
=\frac{M}{2}\|v-u\|^2. ψ ( v ) − ψ ( u ) − ⟨ ∇ ψ ( u ) , v − u ⟩ = ∫ 0 1 ⟨ ∇ ψ ( u + s ( v − u )) − ∇ ψ ( u ) , v − u ⟩ d s ≤ ∫ 0 1 M s ∥ v − u ∥ 2 d s = 2 M ∥ v − u ∥ 2 .
Part 2. Apply Part 1 with v = u − 1 M ∇ ψ ( u ) v=u-\frac{1}{M}\nabla\psi(u) v = u − M 1 ∇ ψ ( u ) :
0 ≤ ψ ( u − 1 M ∇ ψ ( u ) ) ≤ ψ ( u ) − 1 M ∥ ∇ ψ ( u ) ∥ 2 + 1 2 M ∥ ∇ ψ ( u ) ∥ 2 = ψ ( u ) − 1 2 M ∥ ∇ ψ ( u ) ∥ 2 . 0\leq\psi\left(u-\frac{1}{M}\nabla\psi(u)\right)\leq\psi(u)-\frac{1}{M}\|\nabla\psi(u)\|^2+\frac{1}{2M}\|\nabla\psi(u)\|^2=\psi(u)-\frac{1}{2M}\|\nabla\psi(u)\|^2. 0 ≤ ψ ( u − M 1 ∇ ψ ( u ) ) ≤ ψ ( u ) − M 1 ∥∇ ψ ( u ) ∥ 2 + 2 M 1 ∥∇ ψ ( u ) ∥ 2 = ψ ( u ) − 2 M 1 ∥∇ ψ ( u ) ∥ 2 .
This proves the first inequality. For the second, Part 1, the Cauchy–Schwarz inequality, and the first inequality give
ψ ( v ) ≤ ψ ( u ) + 2 M ψ ( u ) ∥ v − u ∥ + M 2 ∥ v − u ∥ 2 . \psi(v)\leq\psi(u)+\sqrt{2M\psi(u)}\|v-u\|+\frac{M}{2}\|v-u\|^2. ψ ( v ) ≤ ψ ( u ) + 2 M ψ ( u ) ∥ v − u ∥ + 2 M ∥ v − u ∥ 2 .
Since 2 a b ≤ a + b 2\sqrt{ab}\leq a+b 2 ab ≤ a + b with a = ψ ( u ) a=\psi(u) a = ψ ( u ) and b = M 2 ∥ v − u ∥ 2 b=\frac{M}{2}\|v-u\|^2 b = 2 M ∥ v − u ∥ 2 , the middle term is at most ψ ( u ) + M 2 ∥ v − u ∥ 2 \psi(u)+\frac{M}{2}\|v-u\|^2 ψ ( u ) + 2 M ∥ v − u ∥ 2 .
Part 3. The function r u ( v ) = ψ ( v ) − ψ ( u ) − ⟨ ∇ ψ ( u ) , v − u ⟩ r_u(v)=\psi(v)-\psi(u)-\langle\nabla\psi(u),v-u\rangle r u ( v ) = ψ ( v ) − ψ ( u ) − ⟨ ∇ ψ ( u ) , v − u ⟩ is nonnegative by convexity and has the M M M -Lipschitz gradient ∇ r u ( v ) = ∇ ψ ( v ) − ∇ ψ ( u ) \nabla r_u(v)=\nabla\psi(v)-\nabla\psi(u) ∇ r u ( v ) = ∇ ψ ( v ) − ∇ ψ ( u ) . Part 2 applied to r u r_u r u at v v v gives the first inequality. Adding it to the same inequality with u u u and v v v interchanged, and using r u ( v ) + r v ( u ) = ⟨ ∇ ψ ( v ) − ∇ ψ ( u ) , v − u ⟩ r_u(v)+r_v(u)=\langle\nabla\psi(v)-\nabla\psi(u),v-u\rangle r u ( v ) + r v ( u ) = ⟨ ∇ ψ ( v ) − ∇ ψ ( u ) , v − u ⟩ , gives the second.
Part 4. The gradient of a differentiable function vanishes at a minimizer. The upper bound on ψ ( u ) − ψ ( x ∗ ) \psi(u)-\psi(x^*) ψ ( u ) − ψ ( x ∗ ) is Part 1 at the point x ∗ x^* x ∗ , and ∥ ∇ ψ ( u ) ∥ = ∥ ∇ ψ ( u ) − ∇ ψ ( x ∗ ) ∥ ≤ M ∥ u − x ∗ ∥ \|\nabla\psi(u)\|=\|\nabla\psi(u)-\nabla\psi(x^*)\|\leq M\|u-x^*\| ∥∇ ψ ( u ) ∥ = ∥∇ ψ ( u ) − ∇ ψ ( x ∗ ) ∥ ≤ M ∥ u − x ∗ ∥ .
Proof of Lemma 4.1 and of (3) . Fix an instance of S d \mathcal{S}_d S d . For almost every ξ \xi ξ , the function f ξ f_\xi f ξ is convex and ∇ f ξ ( x ∗ ) = 0 \nabla f_\xi(x^*)=0 ∇ f ξ ( x ∗ ) = 0 , so x ∗ x^* x ∗ minimizes f ξ f_\xi f ξ . Lemma B.1 , Parts 4 and 2, applied to the nonnegative function f ξ − f ξ ( x ∗ ) f_\xi-f_\xi(x^*) f ξ − f ξ ( x ∗ ) , gives for every x x x
0 ≤ f ξ ( x ) − f ξ ( x ∗ ) ≤ H s 2 ∥ x − x ∗ ∥ 2 , ∥ ∇ f ξ ( x ) ∥ ≤ H s ∥ x − x ∗ ∥ , (30) 0\leq f_\xi(x)-f_\xi(x^*)\leq\frac{H_s}{2}\|x-x^*\|^2,\qquad\|\nabla f_\xi(x)\|\leq H_s\|x-x^*\|, \tag*{(30)} 0 ≤ f ξ ( x ) − f ξ ( x ∗ ) ≤ 2 H s ∥ x − x ∗ ∥ 2 , ∥∇ f ξ ( x ) ∥ ≤ H s ∥ x − x ∗ ∥ , (30)
∥ ∇ f ξ ( x ) ∥ 2 ≤ 2 H s ( f ξ ( x ) − f ξ ( x ∗ ) ) . (31) \|\nabla f_\xi(x)\|^2\leq2H_s\bigl(f_\xi(x)-f_\xi(x^*)\bigr). \tag*{(31)} ∥∇ f ξ ( x ) ∥ 2 ≤ 2 H s ( f ξ ( x ) − f ξ ( x ∗ ) ) . (31)
Fix x , u ∈ R d x,u\in\mathbb{R}^d x , u ∈ R d . For 0 < ∣ s ∣ ≤ 1 0<|s|\leq1 0 < ∣ s ∣ ≤ 1 , the mean value theorem and (B.2) give
∣ f ξ ( x + s u ) − f ξ ( x ) ∣ ∣ s ∣ ≤ sup ∣ σ ∣ ≤ 1 ∥ ∇ f ξ ( x + σ u ) ∥ ∥ u ∥ ≤ H s ( ∥ x − x ∗ ∥ + ∥ u ∥ ) ∥ u ∥ , \frac{|f_\xi(x+su)-f_\xi(x)|}{|s|}\leq\sup_{|\sigma|\leq1}\|\nabla f_\xi(x+\sigma u)\|\|u\|\leq H_s\bigl(\|x-x^*\|+\|u\|\bigr)\|u\|, ∣ s ∣ ∣ f ξ ( x + s u ) − f ξ ( x ) ∣ ≤ ∣ σ ∣ ≤ 1 sup ∥∇ f ξ ( x + σ u ) ∥∥ u ∥ ≤ H s ( ∥ x − x ∗ ∥ + ∥ u ∥ ) ∥ u ∥ ,
a bound independent of ξ \xi ξ and s s s . By dominated convergence, the directional derivative of F F F at x x x along u u u equals E ⟨ ∇ f ξ ( x ) , u ⟩ \mathbb{E}\langle\nabla f_\xi(x),u\rangle E ⟨ ∇ f ξ ( x ) , u ⟩ . Since F F F is differentiable, ∇ F ( x ) = E ∇ f ξ ( x ) \nabla F(x)=\mathbb{E}\nabla f_\xi(x) ∇ F ( x ) = E ∇ f ξ ( x ) . Taking expectations in (B.3) gives E ∥ ∇ f ξ ( x ) ∥ 2 ≤ 2 H s F ( x ) = 2 H s ( F ( x ) − F ( x ∗ ) ) \mathbb{E}\|\nabla f_\xi(x)\|^2\leq2H_sF(x)=2H_s(F(x)-F(x^*)) E ∥∇ f ξ ( x ) ∥ 2 ≤ 2 H s F ( x ) = 2 H s ( F ( x ) − F ( x ∗ )) . The function F F F is convex, has an L L L -Lipschitz gradient, and is minimized at x ∗ x^* x ∗ . Hence the instance, with the oracle g ( x , ξ ) = ∇ f ξ ( x ) g(x,\xi)=\nabla f_\xi(x) g ( x , ξ ) = ∇ f ξ ( x ) , belongs to W d \mathcal{W}_d W d , and to W d q u a d \mathcal{W}^{\mathrm{quad}}_d W d quad if F F F is quadratic.
The algorithm observes the same vectors in both interpretations of the instance, so every algorithm has the same risk in both, which gives M d s g ≤ M d w g \mathcal{M}^{\mathrm{sg}}_d\leq\mathcal{M}^{\mathrm{wg}}_d M d sg ≤ M d wg and M d s g , q u a d ≤ M d w g , q u a d \mathcal{M}^{\mathrm{sg,quad}}_d\leq\mathcal{M}^{\mathrm{wg,quad}}_d M d sg , quad ≤ M d wg , quad . For (3) , an algorithm with fresh access is run with a pool of size S = ∞ S=\infty S = ∞ by assigning the label ( ( n − 1 ) K + k − 1 ) m + w ((n-1)K+k-1)m+w (( n − 1 ) K + k − 1 ) m + w to the k k k -th query of worker w w w in round n n n . These labels are distinct, so the replies are computed at independent fresh samples and the run has the same law as with fresh access.
Lemma B.2 (Moments). Consider an instance of W d \mathcal{W}_d W d with F ( x ∗ ) = 0 F(x^*)=0 F ( x ∗ ) = 0 .
F ( x ) ≤ L 2 ∥ x − x ∗ ∥ 2 F(x)\leq\frac{L}{2}\|x-x^*\|^2 F ( x ) ≤ 2 L ∥ x − x ∗ ∥ 2 and ∥ ∇ F ( x ) ∥ 2 ≤ 2 L F ( x ) \|\nabla F(x)\|^2\leq2LF(x) ∥∇ F ( x ) ∥ 2 ≤ 2 L F ( x ) .
If g ˉ b ( x ) \bar{g}_b(x) g ˉ b ( x ) is the average of b ≥ 1 b\geq1 b ≥ 1 independent fresh responses at the same point x x x that is measurable with respect to the history H \mathcal{H} H before these responses, then
E [ g ˉ b ( x ) ∣ H ] = ∇ F ( x ) , E [ ∥ g ˉ b ( x ) − ∇ F ( x ) ∥ 2 ∣ H ] ≤ 2 H s b F ( x ) . (32) \mathbb{E}[\bar{g}_b(x)\mid\mathcal{H}]=\nabla F(x),\qquad\mathbb{E}[\|\bar{g}_b(x)-\nabla F(x)\|^2\mid\mathcal{H}]\leq\frac{2H_s}{b}F(x). \tag*{(32)} E [ g ˉ b ( x ) ∣ H ] = ∇ F ( x ) , E [ ∥ g ˉ b ( x ) − ∇ F ( x ) ∥ 2 ∣ H ] ≤ b 2 H s F ( x ) . (32) Every finite recurrence built from deterministic finite linear combinations of previous states and fresh oracle responses has finite second moments, provided its initial states have finite second moments.
Proof. Part 1. This is Lemma B.1 , Parts 4 and 2, applied to ψ = F \psi=F ψ = F , which is nonnegative, has an L L L -Lipschitz gradient, and is minimized at x ∗ x^* x ∗ .
Part 2. Write
g i = g ( x , ξ i ) , g ˉ b ( x ) = 1 b ∑ i = 1 b g i , ζ i = g i − ∇ F ( x ) . g_i=g(x,\xi_i),\qquad\bar{g}_b(x)=\frac{1}{b}\sum_{i=1}^{b}g_i,\qquad\zeta_i=g_i-\nabla F(x). g i = g ( x , ξ i ) , g ˉ b ( x ) = b 1 i = 1 ∑ b g i , ζ i = g i − ∇ F ( x ) .
By (B.1),
E [ g i ∣ H ] = ∇ F ( x ) , E [ ζ i ∣ H ] = 0 , E [ g ˉ b ( x ) ∣ H ] = 1 b ∑ i = 1 b E [ g i ∣ H ] = ∇ F ( x ) . \mathbb{E}[g_i \mid\mathcal{H}] = \nabla F(x), \qquad\mathbb{E}[\zeta_i \mid\mathcal{H}] = 0, \qquad\mathbb{E}[\bar{g}_b(x) \mid\mathcal{H}] = \frac{1}{b}\sum_{i=1}^{b}\mathbb{E}[g_i \mid\mathcal{H}] = \nabla F(x). E [ g i ∣ H ] = ∇ F ( x ) , E [ ζ i ∣ H ] = 0 , E [ g ˉ b ( x ) ∣ H ] = b 1 i = 1 ∑ b E [ g i ∣ H ] = ∇ F ( x ) .
Conditional on H \mathcal{H} H , the vectors ζ i \zeta_i ζ i are independent. Thus, for i ≠ j i \ne j i = j ,
E [ ⟨ ζ i , ζ j ⟩ ∣ H ] = ⟨ E [ ζ i ∣ H ] , E [ ζ j ∣ H ] ⟩ = 0. \mathbb{E}[\langle\zeta_i,\zeta_j\rangle\mid\mathcal{H}] = \langle\mathbb{E}[\zeta_i \mid\mathcal{H}],\mathbb{E}[\zeta_j \mid\mathcal{H}]\rangle= 0. E [⟨ ζ i , ζ j ⟩ ∣ H ] = ⟨ E [ ζ i ∣ H ] , E [ ζ j ∣ H ]⟩ = 0.
Expanding the squared norm therefore gives
E [ ∥ g ˉ b ( x ) − ∇ F ( x ) ∥ 2 ∣ H ] = 1 b 2 ∑ i = 1 b E [ ∥ ζ i ∥ 2 ∣ H ] + 2 b 2 ∑ i < j E [ ⟨ ζ i , ζ j ⟩ ∣ H ] = 1 b E [ ∥ g ( x , ξ ) − ∇ F ( x ) ∥ 2 ∣ H ] , \mathbb{E}[\|\bar{g}_b(x)-\nabla F(x)\|^2 \mid\mathcal{H}] = \frac{1}{b^2}\sum_{i=1}^{b}\mathbb{E}[\|\zeta_i\|^2 \mid\mathcal{H}] + \frac{2}{b^2}\sum_{i<j}\mathbb{E}[\langle\zeta_i,\zeta_j\rangle\mid\mathcal{H}] = \frac{1}{b}\mathbb{E}[\|g(x,\xi)-\nabla F(x)\|^2 \mid\mathcal{H}], E [ ∥ g ˉ b ( x ) − ∇ F ( x ) ∥ 2 ∣ H ] = b 2 1 i = 1 ∑ b E [ ∥ ζ i ∥ 2 ∣ H ] + b 2 2 i < j ∑ E [⟨ ζ i , ζ j ⟩ ∣ H ] = b 1 E [ ∥ g ( x , ξ ) − ∇ F ( x ) ∥ 2 ∣ H ] ,
where the last equality also uses that the b b b responses have the same conditional distribution. Expanding one centered second moment and using (B.1) once more,
E [ ∥ g ( x , ξ ) − ∇ F ( x ) ∥ 2 ∣ H ] = E [ ∥ g ( x , ξ ) ∥ 2 ∣ H ] − 2 ⟨ E [ g ( x , ξ ) ∣ H ] , ∇ F ( x ) ⟩ + ∥ ∇ F ( x ) ∥ 2 = E [ ∥ g ( x , ξ ) ∥ 2 ∣ H ] − ∥ ∇ F ( x ) ∥ 2 ≤ 2 H s F ( x ) . \begin{aligned}
\mathbb{E}[\|g(x,\xi)-\nabla F(x)\|^2 \mid\mathcal{H}] = \mathbb{E}[\|g(x,\xi)\|^2 \mid\mathcal{H}] - 2\langle\mathbb{E}[g(x,\xi)\mid\mathcal{H}],\nabla F(x)\rangle+ \|\nabla F(x)\|^2 \\
&= \mathbb{E}[\|g(x,\xi)\|^2 \mid\mathcal{H}] - \|\nabla F(x)\|^2 \le2H_sF(x).
\end{aligned} E [ ∥ g ( x , ξ ) − ∇ F ( x ) ∥ 2 ∣ H ] = E [ ∥ g ( x , ξ ) ∥ 2 ∣ H ] − 2 ⟨ E [ g ( x , ξ ) ∣ H ] , ∇ F ( x )⟩ + ∥∇ F ( x ) ∥ 2 = E [ ∥ g ( x , ξ ) ∥ 2 ∣ H ] − ∥∇ F ( x ) ∥ 2 ≤ 2 H s F ( x ) .
Combining the two displays proves (B.4).
Part 3. First consider a query X X X that is measurable with respect to the history H \mathcal{H} H before it and satisfies E ∥ X ∥ 2 < ∞ \mathbb{E}\|X\|^2 < \infty E ∥ X ∥ 2 < ∞ , and let ξ \xi ξ be a fresh sample. By (B.1) and Part 1,
E ∥ g ( X , ξ ) ∥ 2 = E E [ ∥ g ( X , ξ ) ∥ 2 ∣ H ] ≤ 2 H s E F ( X ) ≤ H s L E ∥ X − x ∗ ∥ 2 ≤ 2 H s L ( E ∥ X ∥ 2 + ∥ x ∗ ∥ 2 ) < ∞ . \mathbb{E}\|g(X,\xi)\|^2 = \mathbb{E}\mathbb{E}[\|g(X,\xi)\|^2 \mid\mathcal{H}] \le2H_s\mathbb{E}F(X) \le H_sL\mathbb{E}\|X-x^*\|^2 \le2H_sL\left(\mathbb{E}\|X\|^2+\|x^*\|^2\right)<\infty. E ∥ g ( X , ξ ) ∥ 2 = EE [ ∥ g ( X , ξ ) ∥ 2 ∣ H ] ≤ 2 H s E F ( X ) ≤ H s L E ∥ X − x ∗ ∥ 2 ≤ 2 H s L ( E ∥ X ∥ 2 + ∥ x ∗ ∥ 2 ) < ∞.
Thus a fresh response at a square-integrable query is itself square-integrable. Next, let Z 1 , … , Z s Z_1,\ldots,Z_s Z 1 , … , Z s be square-integrable random vectors and let a 1 , … , a s a_1,\ldots,a_s a 1 , … , a s be deterministic coefficients. The triangle and Cauchy–Schwarz inequalities give
∥ ∑ j = 1 s a j Z j ∥ 2 ≤ ( ∑ j = 1 s ∣ a j ∣ ∥ Z j ∥ ) 2 ≤ s ∑ j = 1 s a j 2 ∥ Z j ∥ 2 , \left\|\sum_{j=1}^{s}a_jZ_j\right\|^2 \le\left(\sum_{j=1}^{s}|a_j|\|Z_j\|\right)^2 \le s\sum_{j=1}^{s}a_j^2\|Z_j\|^2, j = 1 ∑ s a j Z j 2 ≤ ( j = 1 ∑ s ∣ a j ∣∥ Z j ∥ ) 2 ≤ s j = 1 ∑ s a j 2 ∥ Z j ∥ 2 ,
so every such linear combination is square-integrable; independence of the Z j Z_j Z j is not required. Starting from the square-integrable initial states, forming a query, drawing a fresh response at it, and forming the next states each preserve square integrability. Induction over the finite sequence of these operations proves Part 3. ∎
Lemma B.3 (Serial SGD). Let s ≥ 1 s \ge1 s ≥ 1 . Set X 0 = x 0 X_0=x_0 X 0 = x 0 and
X k + 1 = X k − 1 2 H s g ( X k , ξ k ) , k = 0 , … , s − 1 , X_{k+1}=X_k-\frac{1}{2H_s}g(X_k,\xi_k), \qquad k=0,\ldots,s-1, X k + 1 = X k − 2 H s 1 g ( X k , ξ k ) , k = 0 , … , s − 1 , with independent fresh samples ξ k \xi_k ξ k , and let X ˉ = 1 s ∑ k = 0 s − 1 X k \bar{X}=\frac{1}{s}\sum_{k=0}^{s-1}X_k X ˉ = s 1 ∑ k = 0 s − 1 X k . Then E F ( X ˉ ) ≤ 2 H s ∥ x 0 − x ∗ ∥ 2 s \mathbb{E}F(\bar{X})\le\frac{2H_s\|x_0-x^*\|^2}{s} E F ( X ˉ ) ≤ s 2 H s ∥ x 0 − x ∗ ∥ 2 .
Proof. All iterates are square-integrable by Lemma B.2, Part 3. Conditional on X k X_k X k and the preceding history, (B.1) and the convexity inequality ⟨ ∇ F ( X k ) , X k − x ∗ ⟩ ≥ F ( X k ) \langle\nabla F(X_k),X_k-x^*\rangle\ge F(X_k) ⟨ ∇ F ( X k ) , X k − x ∗ ⟩ ≥ F ( X k ) give
E [ ∥ X k + 1 − x ∗ ∥ 2 ∣ X k ] = ∥ X k − x ∗ ∥ 2 − 1 H s ⟨ ∇ F ( X k ) , X k − x ∗ ⟩ + 1 4 H s 2 E [ ∥ g ( X k , ξ k ) ∥ 2 ∣ X k ] ≤ ∥ X k − x ∗ ∥ 2 − 1 H s F ( X k ) + 1 2 H s F ( X k ) = ∥ X k − x ∗ ∥ 2 − 1 2 H s F ( X k ) . \begin{aligned}
\mathbb{E}[\|X_{k+1}-x^*\|^2\mid X_k] = \|X_k-x^*\|^2-\frac{1}{H_s}\langle\nabla F(X_k),X_k-x^*\rangle+\frac{1}{4H_s^2}\mathbb{E}[\|g(X_k,\xi_k)\|^2\mid X_k] \\
&\le\|X_k-x^*\|^2-\frac{1}{H_s}F(X_k)+\frac{1}{2H_s}F(X_k)=\|X_k-x^*\|^2-\frac{1}{2H_s}F(X_k).
\end{aligned} E [ ∥ X k + 1 − x ∗ ∥ 2 ∣ X k ] = ∥ X k − x ∗ ∥ 2 − H s 1 ⟨ ∇ F ( X k ) , X k − x ∗ ⟩ + 4 H s 2 1 E [ ∥ g ( X k , ξ k ) ∥ 2 ∣ X k ] ≤ ∥ X k − x ∗ ∥ 2 − H s 1 F ( X k ) + 2 H s 1 F ( X k ) = ∥ X k − x ∗ ∥ 2 − 2 H s 1 F ( X k ) .
Taking expectations and summing over k = 0 , … , s − 1 k=0,\ldots,s-1 k = 0 , … , s − 1 gives ∑ k = 0 s − 1 E F ( X k ) ≤ 2 H s ∥ x 0 − x ∗ ∥ 2 \sum_{k=0}^{s-1}\mathbb{E}F(X_k)\le2H_s\|x_0-x^*\|^2 ∑ k = 0 s − 1 E F ( X k ) ≤ 2 H s ∥ x 0 − x ∗ ∥ 2 . Convexity of F F F gives E F ( X ˉ ) ≤ 1 s ∑ k = 0 s − 1 E F ( X k ) \mathbb{E}F(\bar{X})\le\frac{1}{s}\sum_{k=0}^{s-1}\mathbb{E}F(X_k) E F ( X ˉ ) ≤ s 1 ∑ k = 0 s − 1 E F ( X k ) . ∎
Quadratic populations
In this subsection
F ( x ) = 1 2 ⟨ x − x ∗ , A ( x − x ∗ ) ⟩ , 0 ⪯ A ⪯ L I , F(x)=\frac{1}{2}\langle x-x^*,A(x-x^*)\rangle,\qquad0\preceq A\preceq LI, F ( x ) = 2 1 ⟨ x − x ∗ , A ( x − x ∗ )⟩ , 0 ⪯ A ⪯ L I ,
so that ∇ F ( x ) = A ( x − x ∗ ) \nabla F(x)=A(x-x^*) ∇ F ( x ) = A ( x − x ∗ ) and ⟨ ∇ F ( x ) , x − x ∗ ⟩ = 2 F ( x ) \langle\nabla F(x),x-x^*\rangle=2F(x) ⟨ ∇ F ( x ) , x − x ∗ ⟩ = 2 F ( x ) .
One local block. Let y y y be a point that is measurable with respect to the history H \mathcal{H} H at the start of a round, and write E H [ ⋅ ] = E [ ⋅ ∣ H ] \mathbb{E}_{\mathcal{H}}[\cdot] = \mathbb{E}[\cdot\mid\mathcal{H}] E H [ ⋅ ] = E [ ⋅ ∣ H ] . A worker performs K K K fresh queries,
z 0 = y , z j + 1 = z j − 1 H s g ( z j , ξ j ) , 0 ≤ j < K , (33) z_0 = y,\qquad z_{j+1} = z_j - \frac{1}{H_s}g(z_j,\xi_j),\qquad0 \le j < K, \tag*{(33)} z 0 = y , z j + 1 = z j − H s 1 g ( z j , ξ j ) , 0 ≤ j < K , (33)
and forms
G = H s K ( y − z K ) , z ˉ = 1 K ∑ j = 0 K − 1 z j . G = \frac{H_s}{K}(y-z_K),\qquad\bar{z} = \frac{1}{K}\sum_{j=0}^{K-1}z_j. G = K H s ( y − z K ) , z ˉ = K 1 j = 0 ∑ K − 1 z j .
The worker sends G G G and z ˉ \bar{z} z ˉ . The quantity
Φ ( y ) = 1 K ∑ j = 0 K − 1 E H F ( z j ) \Phi(y) = \frac{1}{K}\sum_{j=0}^{K-1}\mathbb{E}_{\mathcal{H}}F(z_j) Φ ( y ) = K 1 j = 0 ∑ K − 1 E H F ( z j )
is used only in the analysis.
Lemma B.4 (Energy of one block). If E ∥ y ∥ 2 < ∞ \mathbb{E}\|y\|^2 < \infty E ∥ y ∥ 2 < ∞ , all states and responses in (B.5) are square-integrable, and
Φ ( y ) + K 2 H s E H ∥ G ∥ 2 ≤ ⟨ E H G , y − x ∗ ⟩ , E H F ( z ˉ ) ≤ Φ ( y ) . (34) \Phi(y) + \frac{K}{2H_s}\mathbb{E}_{\mathcal{H}}\|G\|^2 \le\langle\mathbb{E}_{\mathcal{H}}G,y-x^*\rangle,\qquad\mathbb{E}_{\mathcal{H}}F(\bar{z}) \le\Phi(y). \tag*{(34)} Φ ( y ) + 2 H s K E H ∥ G ∥ 2 ≤ ⟨ E H G , y − x ∗ ⟩ , E H F ( z ˉ ) ≤ Φ ( y ) . (34)
Proof. Square integrability follows from Lemma B.2, Part 3. Fix j < K j<K j < K and condition on the history up to z j z_j z j . By (B.1) and ⟨ A ( z j − x ∗ ) , z j − x ∗ ⟩ = 2 F ( z j ) \langle A(z_j-x^*),z_j-x^*\rangle= 2F(z_j) ⟨ A ( z j − x ∗ ) , z j − x ∗ ⟩ = 2 F ( z j ) ,
E [ ∥ z j + 1 − x ∗ ∥ 2 ∣ z j ] = ∥ z j − x ∗ ∥ 2 − 2 H s ⟨ A ( z j − x ∗ ) , z j − x ∗ ⟩ + 1 H s 2 E [ ∥ g ( z j , ξ j ) ∥ 2 ∣ z j ] ≤ ∥ z j − x ∗ ∥ 2 − 2 H s F ( z j ) + 2 H s F ( z j ) = ∥ z j − x ∗ ∥ 2 − 2 H s F ( z j ) . \begin{aligned}
\mathbb{E}\left[\|z_{j+1}-x^*\|^2\mid z_j\right]
= \|z_j-x^*\|^2-\frac{2}{H_s}\langle A(z_j-x^*),z_j-x^*\rangle+\frac{1}{H_s^2}\mathbb{E}\left[\|g(z_j,\xi_j)\|^2\mid z_j\right] \\
&\le\|z_j-x^*\|^2-\frac{2}{H_s}F(z_j)+\frac{2}{H_s}F(z_j)
= \|z_j-x^*\|^2-\frac{2}{H_s}F(z_j).
\end{aligned} E [ ∥ z j + 1 − x ∗ ∥ 2 ∣ z j ] = ∥ z j − x ∗ ∥ 2 − H s 2 ⟨ A ( z j − x ∗ ) , z j − x ∗ ⟩ + H s 2 1 E [ ∥ g ( z j , ξ j ) ∥ 2 ∣ z j ] ≤ ∥ z j − x ∗ ∥ 2 − H s 2 F ( z j ) + H s 2 F ( z j ) = ∥ z j − x ∗ ∥ 2 − H s 2 F ( z j ) .
Applying E H \mathbb{E}_{\mathcal{H}} E H and summing over j = 0 , … , K − 1 j=0,\ldots,K-1 j = 0 , … , K − 1 gives
Φ ( y ) ≤ H s 2 K ( ∥ y − x ∗ ∥ 2 − E H ∥ z K − x ∗ ∥ 2 ) . \Phi(y)\le\frac{H_s}{2K}\left(\|y-x^*\|^2-\mathbb{E}_{\mathcal{H}}\|z_K-x^*\|^2\right). Φ ( y ) ≤ 2 K H s ( ∥ y − x ∗ ∥ 2 − E H ∥ z K − x ∗ ∥ 2 ) .
Since z K = y − K H s G z_K=y-\frac{K}{H_s}G z K = y − H s K G ,
∥ z K − x ∗ ∥ 2 = ∥ y − x ∗ ∥ 2 − 2 K H s ⟨ G , y − x ∗ ⟩ + K 2 H s 2 ∥ G ∥ 2 , \|z_K-x^*\|^2=\|y-x^*\|^2-\frac{2K}{H_s}\langle G,y-x^*\rangle+\frac{K^2}{H_s^2}\|G\|^2, ∥ z K − x ∗ ∥ 2 = ∥ y − x ∗ ∥ 2 − H s 2 K ⟨ G , y − x ∗ ⟩ + H s 2 K 2 ∥ G ∥ 2 ,
and substituting this identity into the previous bound proves the first inequality in (B.6). The second follows from convexity of F F F .
The spectral surrogate.
Define
A K = H s K [ I − ( I − A H s ) K ] , F K ( y ) = 1 2 ⟨ y − x ∗ , A K ( y − x ∗ ) ⟩ , L K = min { L , H s K } . (35) A_K=\frac{H_s}{K}\left[I-\left(I-\frac{A}{H_s}\right)^K\right],\qquad
F_K(y)=\frac{1}{2}\langle y-x^*,A_K(y-x^*)\rangle,\qquad
L_K=\min\left\{L,\frac{H_s}{K}\right\}. \tag*{(35)} A K = K H s [ I − ( I − H s A ) K ] , F K ( y ) = 2 1 ⟨ y − x ∗ , A K ( y − x ∗ )⟩ , L K = min { L , K H s } . (35)
Neither A K A_K A K nor F K F_K F K is used by the algorithm.
Lemma B.5 (Spectral mean of a block). The function F K F_K F K is convex, has an L K L_K L K -Lipschitz gradient, and has the same minimizers as F F F . For the block (B.5),
E H G = ∇ F K ( y ) , Φ ( y ) + K 2 H s E H ∥ G ∥ 2 ≤ 2 F K ( y ) . (36) \mathbb{E}_{\mathcal{H}}G=\nabla F_K(y),\qquad
\Phi(y)+\frac{K}{2H_s}\mathbb{E}_{\mathcal{H}}\|G\|^2\le2F_K(y). \tag*{(36)} E H G = ∇ F K ( y ) , Φ ( y ) + 2 H s K E H ∥ G ∥ 2 ≤ 2 F K ( y ) . (36)
Proof. By (B.1), E [ z j + 1 − x ∗ ∣ z j ] = ( I − A / H s ) ( z j − x ∗ ) \mathbb{E}[z_{j+1}-x^*\mid z_j]=(I-A/H_s)(z_j-x^*) E [ z j + 1 − x ∗ ∣ z j ] = ( I − A / H s ) ( z j − x ∗ ) . By the tower property, E H ( z K − x ∗ ) = ( I − A / H s ) K ( y − x ∗ ) \mathbb{E}_{\mathcal{H}}(z_K-x^*)=(I-A/H_s)^K(y-x^*) E H ( z K − x ∗ ) = ( I − A / H s ) K ( y − x ∗ ) , and therefore
E H G = H s K [ ( y − x ∗ ) − ( I − A H s ) K ( y − x ∗ ) ] = A K ( y − x ∗ ) = ∇ F K ( y ) . \mathbb{E}_{\mathcal{H}}G
=\frac{H_s}{K}\left[(y-x^*)-\left(I-\frac{A}{H_s}\right)^K(y-x^*)\right]
=A_K(y-x^*)=\nabla F_K(y). E H G = K H s [ ( y − x ∗ ) − ( I − H s A ) K ( y − x ∗ ) ] = A K ( y − x ∗ ) = ∇ F K ( y ) .
The matrices A A A and A K A_K A K have common eigenvectors. If θ ∈ [ 0 , L ] \theta\in[0,L] θ ∈ [ 0 , L ] is an eigenvalue of A A A , the corresponding eigenvalue of A K A_K A K is
θ K = H s K [ 1 − ( 1 − θ H s ) K ] . \theta_K=\frac{H_s}{K}\left[1-\left(1-\frac{\theta}{H_s}\right)^K\right]. θ K = K H s [ 1 − ( 1 − H s θ ) K ] .
Since 0 ≤ θ / H s ≤ 1 0\le\theta/H_s\le1 0 ≤ θ / H s ≤ 1 and, by Bernoulli’s inequality, 0 ≤ 1 − ( 1 − c ) K ≤ min { K c , 1 } 0\le1-(1-c)^K\le\min\{Kc,1\} 0 ≤ 1 − ( 1 − c ) K ≤ min { K c , 1 } for c ∈ [ 0 , 1 ] c\in[0,1] c ∈ [ 0 , 1 ] , we get 0 ≤ θ K ≤ min { θ , H s / K } ≤ L K 0\le\theta_K\le\min\{\theta,H_s/K\}\le L_K 0 ≤ θ K ≤ min { θ , H s / K } ≤ L K ; moreover, θ K = 0 \theta_K=0 θ K = 0 if and only if θ = 0 \theta=0 θ = 0 . Hence 0 ⪯ A K ⪯ L K I 0\preceq A_K\preceq L_KI 0 ⪯ A K ⪯ L K I and ker A K = ker A \ker A_K=\ker A ker A K = ker A . This proves convexity and smoothness of F K F_K F K , and arg min F K = x ∗ + ker A = arg min F \arg\min F_K=x^*+\ker A=\arg\min F arg min F K = x ∗ + ker A = arg min F . Finally, ⟨ ∇ F K ( y ) , y − x ∗ ⟩ = 2 F K ( y ) \langle\nabla F_K(y),y-x^*\rangle=2F_K(y) ⟨ ∇ F K ( y ) , y − x ∗ ⟩ = 2 F K ( y ) , so the inequality in (B.8) follows from (B.6).
Table 1 (PDF p. 26).
The algorithm. Algorithm 1 uses exactly K K K fresh queries per worker in each of its N N N rounds; the weighted sum in the output is accumulated online.
Theorem B.6 (Upper bound for quadratic populations). Let ∥ x 0 − x ∗ ∥ ≤ R \lVert x_0-x^*\rVert\le R ∥ x 0 − x ∗ ∥ ≤ R . Algorithm 1 satisfies
E F ( x ^ ) ≤ R 2 A N ≤ R 2 [ 4 L K N 2 + 8 H s m K N ] . (37) \mathbb{E}F(\hat x)\le\frac{R^2}{A_N}\le R^2\left[\frac{4L_K}{N^2}+\frac{8H_s}{mKN}\right]. \tag*{(37)} E F ( x ^ ) ≤ A N R 2 ≤ R 2 [ N 2 4 L K + m K N 8 H s ] . (37)
Proof. All states have finite second moments by Lemma B.4 and induction over the rounds, since the server states are linear combinations of the received vectors with deterministic coefficients. Let H t \mathcal{H}_t H t be the history at the beginning of round t t t , after y t y_t y t is formed, and write E t [ ⋅ ] = E [ ⋅ ∣ H t ] \mathbb{E}_t[\cdot]=\mathbb{E}[\cdot\mid\mathcal{H}_t] E t [ ⋅ ] = E [ ⋅ ∣ H t ] . Conditional on H t \mathcal{H}_t H t , the pairs ( G i , t , z ˉ i , t ) (G_{i,t},\bar z_{i,t}) ( G i , t , z ˉ i , t ) , i = 1 , … , m i=1,\ldots,m i = 1 , … , m , are independent and identically distributed, and Lemmas B.4 and B.5 apply to each of them with y = y t y=y_t y = y t . Consider the nonnegative potential
P t = A t F K ( x t ) + 1 2 ∥ v t − x ∗ ∥ 2 , t ≥ 0. P_t=A_tF_K(x_t)+\frac{1}{2}\lVert v_t-x^*\rVert^2,\qquad t\ge0. P t = A t F K ( x t ) + 2 1 ∥ v t − x ∗ ∥ 2 , t ≥ 0.
One-step inequality. Lemma B.1 , Part 1, for F K F_K F K and E t G ˉ t = ∇ F K ( y t ) \mathbb{E}_t\bar G_t=\nabla F_K(y_t) E t G ˉ t = ∇ F K ( y t ) give
E t F K ( x t ) ≤ F K ( y t ) − γ ∥ ∇ F K ( y t ) ∥ 2 + L K γ 2 2 E t ∥ G ˉ t ∥ 2 . \mathbb{E}_tF_K(x_t)\le F_K(y_t)-\gamma\lVert\nabla F_K(y_t)\rVert^2+\frac{L_K\gamma^2}{2}\mathbb{E}_t\lVert\bar G_t\rVert^2. E t F K ( x t ) ≤ F K ( y t ) − γ ∥ ∇ F K ( y t ) ∥ 2 + 2 L K γ 2 E t ∥ G ˉ t ∥ 2 .
The update of v t v_t v t gives
1 2 E t ∥ v t − x ∗ ∥ 2 = 1 2 ∥ v t − 1 − x ∗ ∥ 2 − a t ⟨ ∇ F K ( y t ) , v t − 1 − x ∗ ⟩ + a t 2 2 E t ∥ G ˉ t ∥ 2 . \frac{1}{2}\mathbb{E}_t\lVert v_t-x^*\rVert^2=\frac{1}{2}\lVert v_{t-1}-x^*\rVert^2-a_t\langle\nabla F_K(y_t),v_{t-1}-x^*\rangle+\frac{a_t^2}{2}\mathbb{E}_t\lVert\bar G_t\rVert^2. 2 1 E t ∥ v t − x ∗ ∥ 2 = 2 1 ∥ v t − 1 − x ∗ ∥ 2 − a t ⟨ ∇ F K ( y t ) , v t − 1 − x ∗ ⟩ + 2 a t 2 E t ∥ G ˉ t ∥ 2 .
Multiplying the first inequality by A t A_t A t and adding the second, we obtain
E t P t − P t − 1 ≤ A t F K ( y t ) − A t − 1 F K ( x t − 1 ) − a t ⟨ ∇ F K ( y t ) , v t − 1 − x ∗ ⟩ − A t γ ∥ ∇ F K ( y t ) ∥ 2 + A t L K γ 2 + a t 2 2 E t ∥ G ˉ t ∥ 2 . (38) \begin{aligned}
\mathbb{E}_tP_t-P_{t-1}
&\le A_tF_K(y_t)-A_{t-1}F_K(x_{t-1})-a_t\langle\nabla F_K(y_t),v_{t-1}-x^*\rangle\\
&\quad-A_t\gamma\lVert\nabla F_K(y_t)\rVert^2+\frac{A_tL_K\gamma^2+a_t^2}{2}\mathbb{E}_t\lVert\bar G_t\rVert^2. \tag*{(38)}
\end{aligned} E t P t − P t − 1 ≤ A t F K ( y t ) − A t − 1 F K ( x t − 1 ) − a t ⟨ ∇ F K ( y t ) , v t − 1 − x ∗ ⟩ − A t γ ∥ ∇ F K ( y t ) ∥ 2 + 2 A t L K γ 2 + a t 2 E t ∥ G ˉ t ∥ 2 . (38)
Since A t = A t − 1 + a t A_t=A_{t-1}+a_t A t = A t − 1 + a t and A t y t = A t − 1 x t − 1 + a t v t − 1 A_ty_t=A_{t-1}x_{t-1}+a_tv_{t-1} A t y t = A t − 1 x t − 1 + a t v t − 1 , we have a t ( v t − 1 − x ∗ ) = A t − 1 ( y t − x t − 1 ) + a t ( y t − x ∗ ) a_t(v_{t-1}-x^*)=A_{t-1}(y_t-x_{t-1})+a_t(y_t-x^*) a t ( v t − 1 − x ∗ ) = A t − 1 ( y t − x t − 1 ) + a t ( y t − x ∗ ) . Together with ⟨ ∇ F K ( y t ) , y t − x ∗ ⟩ = 2 F K ( y t ) \langle\nabla F_K(y_t),y_t-x^*\rangle=2F_K(y_t) ⟨ ∇ F K ( y t ) , y t − x ∗ ⟩ = 2 F K ( y t ) , this shows that the first line of the right-hand side of (B.10) equals
A t − 1 [ F K ( y t ) − F K ( x t − 1 ) − ⟨ ∇ F K ( y t ) , y t − x t − 1 ⟩ ] + a t [ F K ( y t ) − ⟨ ∇ F K ( y t ) , y t − x ∗ ⟩ ] = − A t − 1 D F K ( x t − 1 , y t ) − a t F K ( y t ) , \begin{aligned}
&A_{t-1}\left[F_K(y_t)-F_K(x_{t-1})-\langle\nabla F_K(y_t),y_t-x_{t-1}\rangle\right]\\
&\quad+a_t\left[F_K(y_t)-\langle\nabla F_K(y_t),y_t-x^*\rangle\right]\\
&=-A_{t-1}D_{F_K}(x_{t-1},y_t)-a_tF_K(y_t),
\end{aligned} A t − 1 [ F K ( y t ) − F K ( x t − 1 ) − ⟨ ∇ F K ( y t ) , y t − x t − 1 ⟩ ] + a t [ F K ( y t ) − ⟨ ∇ F K ( y t ) , y t − x ∗ ⟩ ] = − A t − 1 D F K ( x t − 1 , y t ) − a t F K ( y t ) ,
where $D_{F_K}(u,v)=F_K(u)-F_K(v)-\langle\nabla F_K(v),u-v\rangle\ge 0.
Coefficients. For t ≥ 1 t \ge1 t ≥ 1 , ( t + 1 ) 2 ≤ t ( t + 3 ) (t+1)^2 \le t(t+3) ( t + 1 ) 2 ≤ t ( t + 3 ) , hence a t 2 ≤ γ A t a_t^2 \le\gamma A_t a t 2 ≤ γ A t . Since γ L K ≤ 1 \gamma L_K \le1 γ L K ≤ 1 , also A t L K γ 2 ≤ γ A t A_t L_K \gamma^2 \le\gamma A_t A t L K γ 2 ≤ γ A t , so the coefficient of E t ∥ G ˉ t ∥ 2 \mathbb{E}_t\lVert\bar{G}_t\rVert^2 E t ∥ G ˉ t ∥ 2 in (B.10) is at most γ A t \gamma A_t γ A t . The vectors G i , t − ∇ F K ( y t ) G_{i,t}-\nabla F_K(y_t) G i , t − ∇ F K ( y t ) are conditionally independent and centered, which gives the exact identity
E t ∥ G ˉ t ∥ 2 = ( 1 − 1 m ) ∥ ∇ F K ( y t ) ∥ 2 + 1 m E t ∥ G 1 , t ∥ 2 . (39) \mathbb{E}_t\lVert\bar{G}_t\rVert^2 = \left(1-\frac{1}{m}\right)\lVert\nabla F_K(y_t)\rVert^2+\frac{1}{m}\mathbb{E}_t\lVert G_{1,t}\rVert^2. \tag*{(39)} E t ∥ G ˉ t ∥ 2 = ( 1 − m 1 ) ∥ ∇ F K ( y t ) ∥ 2 + m 1 E t ∥ G 1 , t ∥ 2 . (39)
Spending the certificate. By (B.8) with y = y t y=y_t y = y t ,
− a t F K ( y t ) ≤ − a t 2 Φ ( y t ) − a t K 4 H s E t ∥ G 1 , t ∥ 2 . -a_tF_K(y_t)\le-\frac{a_t}{2}\Phi(y_t)-\frac{a_tK}{4H_s}\mathbb{E}_t\lVert G_{1,t}\rVert^2. − a t F K ( y t ) ≤ − 2 a t Φ ( y t ) − 4 H s a t K E t ∥ G 1 , t ∥ 2 .
Substituting this bound, the bound on the coefficient, and (B.11) into (B.10) yields
E t P t − P t − 1 ≤ − A t − 1 D F K ( x t − 1 , y t ) − a t 2 Φ ( y t ) − γ A t m ∥ ∇ F K ( y t ) ∥ 2 + [ γ A t m − a t K 4 H s ] E t ∥ G 1 , t ∥ 2 . \mathbb{E}_tP_t-P_{t-1}\le-A_{t-1}D_{F_K}(x_{t-1},y_t)-\frac{a_t}{2}\Phi(y_t)-\frac{\gamma A_t}{m}\lVert\nabla F_K(y_t)\rVert^2+\left[\frac{\gamma A_t}{m}-\frac{a_tK}{4H_s}\right]\mathbb{E}_t\lVert G_{1,t}\rVert^2. E t P t − P t − 1 ≤ − A t − 1 D F K ( x t − 1 , y t ) − 2 a t Φ ( y t ) − m γ A t ∥ ∇ F K ( y t ) ∥ 2 + [ m γ A t − 4 H s a t K ] E t ∥ G 1 , t ∥ 2 .
Since t ( t + 3 ) ≤ ( t + 1 ) ( t + 2 ) t(t+3)\le(t+1)(t+2) t ( t + 3 ) ≤ ( t + 1 ) ( t + 2 ) and t ≤ N t\le N t ≤ N ,
4 γ A t H s a t m K = 2 γ H s m K ⋅ t ( t + 3 ) t + 1 ≤ 2 γ H s ( N + 2 ) m K ≤ 1 , \frac{4\gamma A_tH_s}{a_tmK}=\frac{2\gamma H_s}{mK}\cdot\frac{t(t+3)}{t+1}\le\frac{2\gamma H_s(N+2)}{mK}\le1, a t m K 4 γ A t H s = m K 2 γ H s ⋅ t + 1 t ( t + 3 ) ≤ m K 2 γ H s ( N + 2 ) ≤ 1 ,
so the last bracket is nonpositive. All terms except − a t 2 Φ ( y t ) -\frac{a_t}{2}\Phi(y_t) − 2 a t Φ ( y t ) on the right-hand side are therefore nonpositive, and
E t P t + a t 2 Φ ( y t ) ≤ P t − 1 . (40) \mathbb{E}_tP_t+\frac{a_t}{2}\Phi(y_t)\le P_{t-1}. \tag*{(40)} E t P t + 2 a t Φ ( y t ) ≤ P t − 1 . (40)
The output. Taking expectations in (B.12), summing over t = 1 , … , N t=1,\ldots,N t = 1 , … , N , and using P N ≥ 0 P_N\ge0 P N ≥ 0 and P 0 = 1 2 ∥ x 0 − x ∗ ∥ 2 ≤ R 2 2 P_0=\frac{1}{2}\lVert x_0-x^*\rVert^2\le\frac{R^2}{2} P 0 = 2 1 ∥ x 0 − x ∗ ∥ 2 ≤ 2 R 2 , we get ∑ t = 1 N a t E Φ ( y t ) ≤ R 2 \sum_{t=1}^N a_t\mathbb{E}\Phi(y_t)\le R^2 ∑ t = 1 N a t E Φ ( y t ) ≤ R 2 . By convexity and Lemma B.4, E t F ( z ^ t ) ≤ 1 m ∑ i = 1 m E t F ( z ˉ i , t ) ≤ Φ ( y t ) \mathbb{E}_tF(\hat{z}_t)\le\frac{1}{m}\sum_{i=1}^m\mathbb{E}_tF(\bar{z}_{i,t})\le\Phi(y_t) E t F ( z ^ t ) ≤ m 1 ∑ i = 1 m E t F ( z ˉ i , t ) ≤ Φ ( y t ) . Since ∑ t = 1 N a t = A N \sum_{t=1}^N a_t=A_N ∑ t = 1 N a t = A N , convexity of F F F gives
E F ( x ^ ) ≤ 1 A N ∑ t = 1 N a t E F ( z ^ t ) ≤ R 2 A N . \mathbb{E}F(\hat{x})\le\frac{1}{A_N}\sum_{t=1}^N a_t\mathbb{E}F(\hat{z}_t)\le\frac{R^2}{A_N}. E F ( x ^ ) ≤ A N 1 t = 1 ∑ N a t E F ( z ^ t ) ≤ A N R 2 .
Finally,
1 A N = 4 N ( N + 3 ) max { L K , 2 H s ( N + 2 ) m K } ≤ 4 L K N 2 + 8 H s m K N . \frac{1}{A_N}=\frac{4}{N(N+3)}\max\left\{L_K,\frac{2H_s(N+2)}{mK}\right\}\le\frac{4L_K}{N^2}+\frac{8H_s}{mKN}. A N 1 = N ( N + 3 ) 4 max { L K , m K 2 H s ( N + 2 ) } ≤ N 2 4 L K + m K N 8 H s .
Corollary B.7 (Quadratic selector). Selecting in advance between Algorithm 1 and returning x 0 x_0 x 0 gives
E F ( x ^ ) ≤ R 2 min { L 2 , 4 min { L , H s / K } N 2 + 8 H s m K N } ≤ 8 R 2 min { L , min { L , H s / K } N 2 + H s m K N } . (41) \mathbb{E}F(\hat{x})\le R^2\min\left\{\frac{L}{2},\frac{4\min\{L,H_s/K\}}{N^2}+\frac{8H_s}{mKN}\right\}\le8R^2\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{mKN}\right\}. \tag*{(41)} E F ( x ^ ) ≤ R 2 min { 2 L , N 2 4 min { L , H s / K } + m K N 8 H s } ≤ 8 R 2 min { L , N 2 min { L , H s / K } + m K N H s } . (41)
Proof. By Lemma B.2, Part 1, F ( x 0 ) ≤ L 2 R 2 F(x_0)\le\frac{L}{2}R^2 F ( x 0 ) ≤ 2 L R 2 , and Theorem B.6 bounds the risk of the other branch. Both bounds depend only on the public parameters, so the selection is made before the run. The second inequality follows by increasing all coefficients to 8.
General convex populations
The upper bound uses the following accelerated method with a fresh batch at every update.
Algorithm 2 Fresh-batch two-state accelerated recurrence
Require: Initial point x 0 x_0 x 0 , parameters L L L , H s > 0 H_s > 0 H s > 0 , integers q q q , b ≥ 1 b \ge1 b ≥ 1 , and oracle g g g .
γ ← min { 1 48 L , b 48 H s q } \gamma\leftarrow\min\left\{\frac{1}{48L},\frac{b}{48H_s q}\right\} γ ← min { 48 L 1 , 48 H s q b }
z 0 ← x 0 z_0 \leftarrow x_0 z 0 ← x 0 , v 0 ← x 0 v_0 \leftarrow x_0 v 0 ← x 0
for t = 0 , … , q − 1 t = 0,\ldots,q-1 t = 0 , … , q − 1 do
α t ← 6 t + 6 \alpha_t \leftarrow\frac{6}{t+6} α t ← t + 6 6 , η t ← γ ( t + 1 ) \eta_t \leftarrow\gamma(t+1) η t ← γ ( t + 1 )
y t ← ( 1 − α t ) z t + α t v t y_t \leftarrow(1-\alpha_t)z_t+\alpha_t v_t y t ← ( 1 − α t ) z t + α t v t
Draw ξ t , 1 , … , ξ t , b \xi_{t,1},\ldots,\xi_{t,b} ξ t , 1 , … , ξ t , b independently from the oracle law, independently of the past.
g ˉ t ← 1 b ∑ i = 1 b g ( y t , ξ t , i ) \bar{g}_t \leftarrow\frac{1}{b}\sum_{i=1}^{b}g(y_t,\xi_{t,i}) g ˉ t ← b 1 ∑ i = 1 b g ( y t , ξ t , i )
v t + 1 ← v t − η t g ˉ t v_{t+1} \leftarrow v_t-\eta_t\bar{g}_t v t + 1 ← v t − η t g ˉ t
z t + 1 ← ( 1 − α t ) z t + α t v t + 1 z_{t+1} \leftarrow(1-\alpha_t)z_t+\alpha_t v_{t+1} z t + 1 ← ( 1 − α t ) z t + α t v t + 1
end for
return z q z_q z q
The algorithm uses exactly b q bq b q fresh oracle responses. It requires no loss values, projections, or exact population-gradient evaluations.
Theorem B.8 (Accelerated common-query weak-growth bound). Consider an instance of W d \mathcal{W}_d W d with F ( x ∗ ) = 0 F(x^*)=0 F ( x ∗ ) = 0 and ∥ x 0 − x ∗ ∥ ≤ R \lVert x_0-x^*\rVert\le R ∥ x 0 − x ∗ ∥ ≤ R . Then Algorithm 2 satisfies
E F ( z q ) ≤ 288 R 2 max { L q 2 , H s b q } . (42) \mathbb{E}F(z_q)\le288R^2\max\left\{\frac{L}{q^2},\frac{H_s}{bq}\right\}. \tag*{(42)} E F ( z q ) ≤ 288 R 2 max { q 2 L , b q H s } . (42)
Proof. By Lemma B.2, Part 3, all iterates and oracle responses have finite second moments. Let F t \mathcal{F}_t F t denote the history before the batch at iteration t t t is sampled, and write
E t [ ⋅ ] = E [ ⋅ ∣ F t ] , A t = η t α t = γ ( t + 1 ) ( t + 6 ) 6 . \mathbb{E}_t[\cdot]=\mathbb{E}[\cdot\mid\mathcal{F}_t],\qquad A_t=\frac{\eta_t}{\alpha_t}=\frac{\gamma(t+1)(t+6)}{6}. E t [ ⋅ ] = E [ ⋅ ∣ F t ] , A t = α t η t = 6 γ ( t + 1 ) ( t + 6 ) .
The vectors z t , v t , y t z_t,v_t,y_t z t , v t , y t are F t \mathcal{F}_t F t -measurable.
Conditional variance. Let σ t 2 = E t ∥ g ˉ t − ∇ F ( y t ) ∥ 2 \sigma_t^2=\mathbb{E}_t\lVert\bar{g}_t-\nabla F(y_t)\rVert^2 σ t 2 = E t ∥ g ˉ t − ∇ F ( y t ) ∥ 2 . By (B.4),
E t g ˉ t = ∇ F ( y t ) , σ t 2 ≤ 2 H s b F ( y t ) . \mathbb{E}_t\bar{g}_t=\nabla F(y_t),\qquad\sigma_t^2\le\frac{2H_s}{b}F(y_t). E t g ˉ t = ∇ F ( y t ) , σ t 2 ≤ b 2 H s F ( y t ) .
Since y t − z t = α t ( v t − z t ) y_t-z_t=\alpha_t(v_t-z_t) y t − z t = α t ( v t − z t ) , Lemma B.1, Part 2, applied to the nonnegative function F F F gives F ( y t ) ≤ 2 F ( z t ) + L α t 2 ∥ v t − z t ∥ 2 F(y_t)\le2F(z_t)+L\alpha_t^2\lVert v_t-z_t\rVert^2 F ( y t ) ≤ 2 F ( z t ) + L α t 2 ∥ v t − z t ∥ 2 , and hence
σ t 2 ≤ 4 H s b F ( z t ) + 2 H s L b α t 2 ∥ v t − z t ∥ 2 . (43) \sigma_t^2\le\frac{4H_s}{b}F(z_t)+\frac{2H_sL}{b}\alpha_t^2\lVert v_t-z_t\rVert^2. \tag*{(43)} σ t 2 ≤ b 4 H s F ( z t ) + b 2 H s L α t 2 ∥ v t − z t ∥ 2 . (43)
One-step potential inequality. The two updates give z t + 1 = ( 1 − α t ) z t + α t ( v t − η t g ˉ t ) = y t − α t η t g ˉ t z_{t+1}=(1-\alpha_t)z_t+\alpha_t(v_t-\eta_t\bar{g}_t)=y_t-\alpha_t\eta_t\bar{g}_t z t + 1 = ( 1 − α t ) z t + α t ( v t − η t g ˉ t ) = y t − α t η t g ˉ t . Lemma B.1, Part 1, together with E t g ˉ t = ∇ F ( y t ) \mathbb{E}_t\bar{g}_t=\nabla F(y_t) E t g ˉ t = ∇ F ( y t ) and E t ∥ g ˉ t ∥ 2 = ∥ ∇ F ( y t ) ∥ 2 + σ t 2 \mathbb{E}_t\lVert\bar{g}_t\rVert^2=\lVert\nabla F(y_t)\rVert^2+\sigma_t^2 E t ∥ g ˉ t ∥ 2 = ∥ ∇ F ( y t ) ∥ 2 + σ t 2 , implies
E t F ( z t + 1 ) ≤ F ( y t ) − α t η t ∥ ∇ F ( y t ) ∥ 2 + L α t 2 η t 2 2 ( ∥ ∇ F ( y t ) ∥ 2 + σ t 2 ) . (44) \mathbb{E}_tF(z_{t+1})\le F(y_t)-\alpha_t\eta_t\lVert\nabla F(y_t)\rVert^2+\frac{L\alpha_t^2\eta_t^2}{2}\left(\lVert\nabla F(y_t)\rVert^2+\sigma_t^2\right). \tag*{(44)} E t F ( z t + 1 ) ≤ F ( y t ) − α t η t ∥ ∇ F ( y t ) ∥ 2 + 2 L α t 2 η t 2 ( ∥ ∇ F ( y t ) ∥ 2 + σ t 2 ) . (44)
Convexity at y t y_t y t , applied to z t z_t z t and x ∗ x^* x ∗ with weights 1 − α t 1-\alpha_t 1 − α t and α t \alpha_t α t , together with F ( x ∗ ) = 0 F(x^*)=0 F ( x ∗ ) = 0 and ( 1 − α t ) z t + α t x ∗ − y t = α t ( x ∗ − v t ) (1-\alpha_t)z_t+\alpha_tx^*-y_t=\alpha_t(x^*-v_t) ( 1 − α t ) z t + α t x ∗ − y t = α t ( x ∗ − v t ) , gives
F ( y t ) ≤ ( 1 − α t ) F ( z t ) + α t ⟨ ∇ F ( y t ) , v t − x ∗ ⟩ . (45) F(y_t)\le(1-\alpha_t)F(z_t)+\alpha_t\langle\nabla F(y_t),v_t-x^*\rangle. \tag*{(45)} F ( y t ) ≤ ( 1 − α t ) F ( z t ) + α t ⟨ ∇ F ( y t ) , v t − x ∗ ⟩ . (45)
The update of the auxiliary sequence satisfies the exact identity
1 2 E t ∥ v t + 1 − x ∗ ∥ 2 = 1 2 ∥ v t − x ∗ ∥ 2 − η t ⟨ ∇ F ( y t ) , v t − x ∗ ⟩ + η t 2 2 ( ∥ ∇ F ( y t ) ∥ 2 + σ t 2 ) . (46) \frac{1}{2}\mathbb{E}_t\lVert v_{t+1}-x^*\rVert^2=\frac{1}{2}\lVert v_t-x^*\rVert^2-\eta_t\langle\nabla F(y_t),v_t-x^*\rangle+\frac{\eta_t^2}{2}\left(\lVert\nabla F(y_t)\rVert^2+\sigma_t^2\right). \tag*{(46)} 2 1 E t ∥ v t + 1 − x ∗ ∥ 2 = 2 1 ∥ v t − x ∗ ∥ 2 − η t ⟨ ∇ F ( y t ) , v t − x ∗ ⟩ + 2 η t 2 ( ∥ ∇ F ( y t ) ∥ 2 + σ t 2 ) . (46)
Substitute (B.17) into (B.16), multiply the result by A t = η t / α t A_t=\eta_t/\alpha_t A t = η t / α t , and add (B.18). The terms ± η t ⟨ ∇ F ( y t ) , v t − x ∗ ⟩ \pm\eta_t\langle\nabla F(y_t),v_t-x^*\rangle ± η t ⟨ ∇ F ( y t ) , v t − x ∗ ⟩ cancel, and A t α t η t = η t 2 A_t\alpha_t\eta_t=\eta_t^2 A t α t η t = η t 2 , A t L α t 2 η t = η t 2 L α t η t A_tL\alpha_t^2\eta_t=\eta_t^2L\alpha_t\eta_t A t L α t 2 η t = η t 2 L α t η t ; this gives
E t [ A t F ( z t + 1 ) + 1 2 ∥ v t + 1 − x ∗ ∥ 2 ] ≤ ( A t − η t ) F ( z t ) + 1 2 ∥ v t − x ∗ ∥ 2 − η t 2 2 ( 1 − L α t η t ) ∥ ∇ F ( y t ) ∥ 2 + η t 2 2 ( 1 + L α t η t ) σ t 2 . (47) \mathbb{E}_t\left[A_tF(z_{t+1})+\frac{1}{2}\lVert v_{t+1}-x^*\rVert^2\right]\leq(A_t-\eta_t)F(z_t)+\frac{1}{2}\lVert v_t-x^*\rVert^2-\frac{\eta_t^2}{2}(1-L\alpha_t\eta_t)\lVert\nabla F(y_t)\rVert^2+\frac{\eta_t^2}{2}(1+L\alpha_t\eta_t)\sigma_t^2. \tag*{(47)} E t [ A t F ( z t + 1 ) + 2 1 ∥ v t + 1 − x ∗ ∥ 2 ] ≤ ( A t − η t ) F ( z t ) + 2 1 ∥ v t − x ∗ ∥ 2 − 2 η t 2 ( 1 − L α t η t ) ∥ ∇ F ( y t ) ∥ 2 + 2 η t 2 ( 1 + L α t η t ) σ t 2 . (47)
Since
α t η t = 6 γ t + 1 t + 6 ≤ 6 γ , L γ ≤ 1 48 , \alpha_t\eta_t=6\gamma\frac{t+1}{t+6}\leq6\gamma,\qquad L\gamma\leq\frac{1}{48}, α t η t = 6 γ t + 6 t + 1 ≤ 6 γ , L γ ≤ 48 1 ,
we have L α t η t ≤ 1 / 8 L\alpha_t\eta_t\leq1/8 L α t η t ≤ 1/8 . Thus the coefficient of ∥ ∇ F ( y t ) ∥ 2 \lVert\nabla F(y_t)\rVert^2 ∥ ∇ F ( y t ) ∥ 2 is nonpositive, and the coefficient of σ t 2 \sigma_t^2 σ t 2 is at most η t 2 \eta_t^2 η t 2 . Substituting (B.15) into (B.19) yields
E t [ A t F ( z t + 1 ) + 1 2 ∥ v t + 1 − x ∗ ∥ 2 ] ≤ ( A t − η t + 4 H s η t 2 b ) F ( z t ) + 1 2 ∥ v t − x ∗ ∥ 2 + 2 H s L b ( α t η t ) 2 ∥ v t − z t ∥ 2 . (48) \mathbb{E}_t\left[A_tF(z_{t+1})+\frac{1}{2}\lVert v_{t+1}-x^*\rVert^2\right]\leq\left(A_t-\eta_t+\frac{4H_s\eta_t^2}{b}\right)F(z_t)+\frac{1}{2}\lVert v_t-x^*\rVert^2+\frac{2H_sL}{b}(\alpha_t\eta_t)^2\lVert v_t-z_t\rVert^2. \tag*{(48)} E t [ A t F ( z t + 1 ) + 2 1 ∥ v t + 1 − x ∗ ∥ 2 ] ≤ ( A t − η t + b 4 H s η t 2 ) F ( z t ) + 2 1 ∥ v t − x ∗ ∥ 2 + b 2 H s L ( α t η t ) 2 ∥ v t − z t ∥ 2 . (48)
For 1 ≤ t ≤ q − 1 1\leq t\leq q-1 1 ≤ t ≤ q − 1 , direct calculation gives
A t − 1 − ( A t − η t ) = 2 3 γ t . A_{t-1}-(A_t-\eta_t)=\frac{2}{3}\gamma t. A t − 1 − ( A t − η t ) = 3 2 γ t .
Moreover, ( t + 1 ) 2 ≤ 4 t 2 ≤ 4 t q (t+1)^2\leq4t^2\leq4tq ( t + 1 ) 2 ≤ 4 t 2 ≤ 4 tq and γ ≤ b / ( 48 H s q ) \gamma\leq b/(48H_sq) γ ≤ b / ( 48 H s q ) imply
4 H s η t 2 b = 4 H s γ 2 ( t + 1 ) 2 b ≤ 16 H s γ 2 t q b ≤ 1 3 γ t . \frac{4H_s\eta_t^2}{b}=\frac{4H_s\gamma^2(t+1)^2}{b}\leq\frac{16H_s\gamma^2tq}{b}\leq\frac{1}{3}\gamma t. b 4 H s η t 2 = b 4 H s γ 2 ( t + 1 ) 2 ≤ b 16 H s γ 2 tq ≤ 3 1 γ t .
Therefore
A t − η t + 4 H s η t 2 b ≤ A t − 1 , 1 ≤ t ≤ q − 1. (49) A_t-\eta_t+\frac{4H_s\eta_t^2}{b}\leq A_{t-1},\qquad1\leq t\leq q-1. \tag*{(49)} A t − η t + b 4 H s η t 2 ≤ A t − 1 , 1 ≤ t ≤ q − 1. (49)
Controlling the auxiliary states. For t ≥ 1 t\geq1 t ≥ 1 , define
E t = E [ A t − 1 F ( z t ) + 1 2 ∥ v t − x ∗ ∥ 2 ] , M t = max { R 2 , 2 max 1 ≤ s ≤ t E s } . \mathcal{E}_t=\mathbb{E}\left[A_{t-1}F(z_t)+\frac{1}{2}\lVert v_t-x^*\rVert^2\right],\qquad M_t=\max\left\{R^2,\ 2\max_{1\leq s\leq t}\mathcal{E}_s\right\}. E t = E [ A t − 1 F ( z t ) + 2 1 ∥ v t − x ∗ ∥ 2 ] , M t = max { R 2 , 2 1 ≤ s ≤ t max E s } .
Since F ≥ 0 F\geq0 F ≥ 0 , we have E ∥ v s − x ∗ ∥ 2 ≤ 2 E s ≤ M t \mathbb{E}\lVert v_s-x^*\rVert^2\leq2\mathcal{E}_s\leq M_t E ∥ v s − x ∗ ∥ 2 ≤ 2 E s ≤ M t for 1 ≤ s ≤ t 1\leq s\leq t 1 ≤ s ≤ t . Because 0 < α s ≤ 1 0<\alpha_s\leq1 0 < α s ≤ 1 , the recurrence for z s z_s z s expresses z t z_t z t as a convex combination, with deterministic weights, of x 0 , v 1 , … , v t x_0,v_1,\ldots,v_t x 0 , v 1 , … , v t . Jensen’s inequality therefore gives E ∥ z t − x ∗ ∥ 2 ≤ M t \mathbb{E}\lVert z_t-x^*\rVert^2\leq M_t E ∥ z t − x ∗ ∥ 2 ≤ M t , and consequently
E ∥ v t − z t ∥ 2 ≤ 2 E ∥ v t − x ∗ ∥ 2 + 2 E ∥ z t − x ∗ ∥ 2 ≤ 4 M t . \mathbb{E}\lVert v_t-z_t\rVert^2\leq2\mathbb{E}\lVert v_t-x^*\rVert^2+2\mathbb{E}\lVert z_t-x^*\rVert^2\leq4M_t. E ∥ v t − z t ∥ 2 ≤ 2 E ∥ v t − x ∗ ∥ 2 + 2 E ∥ z t − x ∗ ∥ 2 ≤ 4 M t .
Taking expectations in (B.20) and using (B.21) and α t η t ≤ 6 γ \alpha_t\eta_t\leq6\gamma α t η t ≤ 6 γ , we obtain
E t + 1 ≤ E t + 288 H s L γ 2 b M t , 1 ≤ t ≤ q − 1. \mathcal{E}_{t+1}\leq\mathcal{E}_t+\frac{288H_sL\gamma^2}{b}M_t,\qquad1\leq t\leq q-1. E t + 1 ≤ E t + b 288 H s L γ 2 M t , 1 ≤ t ≤ q − 1.
The two upper bounds defining γ \gamma γ imply
γ 2 ≤ 1 48 L ⋅ b 48 H s q , \gamma^2\leq\frac{1}{48L}\cdot\frac{b}{48H_sq}, γ 2 ≤ 48 L 1 ⋅ 48 H s q b ,
hence
288 H s L γ 2 b ≤ 1 8 q . \frac{288H_sL\gamma^2}{b}\leq\frac{1}{8q}. b 288 H s L γ 2 ≤ 8 q 1 .
It follows that 2 E t + 1 ≤ M t + M t 4 q 2\mathcal{E}_{t+1}\leq M_t+\frac{M_t}{4q} 2 E t + 1 ≤ M t + 4 q M t , that is,
M t + 1 ≤ ( 1 + 1 4 q ) M t , 1 ≤ t ≤ q − 1. M_{t+1}\leq\left(1+\frac{1}{4q}\right)M_t,\qquad1\leq t\leq q-1. M t + 1 ≤ ( 1 + 4 q 1 ) M t , 1 ≤ t ≤ q − 1.
For the initial step, α 0 = 1 \alpha_0=1 α 0 = 1 , A 0 = η 0 A_0=\eta_0 A 0 = η 0 , and z 0 = v 0 = y 0 = x 0 z_0=v_0=y_0=x_0 z 0 = v 0 = y 0 = x 0 . By (B.4) and Lemma B.2, Part 1,
σ 0 2 ≤ 2 H s b F ( x 0 ) ≤ H s L b R 2 . (50) \sigma_0^2\leq\frac{2H_s}{b}F(x_0)\leq\frac{H_sL}{b}R^2. \tag*{(50)} σ 0 2 ≤ b 2 H s F ( x 0 ) ≤ b H s L R 2 . (50)
Using (B.19) at t = 0 t = 0 t = 0 gives
E 1 ≤ ( 1 2 + H s L γ 2 b ) R 2 ≤ ( 1 2 + 1 8 q ) R 2 , hence M 1 ≤ ( 1 + 1 4 q ) R 2 . \mathcal{E}_1 \le\left(\frac{1}{2} + \frac{H_s L \gamma^2}{b}\right) R^2 \le\left(\frac{1}{2} + \frac{1}{8q}\right) R^2, \qquad\text{hence} \qquad M_1 \le\left(1 + \frac{1}{4q}\right) R^2. E 1 ≤ ( 2 1 + b H s L γ 2 ) R 2 ≤ ( 2 1 + 8 q 1 ) R 2 , hence M 1 ≤ ( 1 + 4 q 1 ) R 2 .
Consequently, including the case q = 1 q = 1 q = 1 ,
M q ≤ ( 1 + 1 4 q ) q R 2 ≤ e 1 / 4 R 2 < 2 R 2 , M_q \le\left(1 + \frac{1}{4q}\right)^q R^2 \le e^{1/4}R^2 < 2R^2, M q ≤ ( 1 + 4 q 1 ) q R 2 ≤ e 1/4 R 2 < 2 R 2 ,
and therefore E q < R 2 \mathcal{E}_q < R^2 E q < R 2 .
Final bound. By nonnegativity of the squared-distance term,
A q − 1 E F ( z q ) ≤ E q ≤ R 2 . A_{q-1}\mathbb{E}F(z_q) \le\mathcal{E}_q \le R^2. A q − 1 E F ( z q ) ≤ E q ≤ R 2 .
Finally,
A q − 1 = γ q ( q + 5 ) 6 ≥ γ q 2 6 , A_{q-1} = \frac{\gamma q(q+5)}{6} \ge\frac{\gamma q^2}{6}, A q − 1 = 6 γ q ( q + 5 ) ≥ 6 γ q 2 ,
and hence
E F ( z q ) ≤ 6 R 2 γ q 2 = 288 R 2 max { L q 2 , H s b q } . \mathbb{E}F(z_q) \le\frac{6R^2}{\gamma q^2} = 288R^2 \max\left\{\frac{L}{q^2},\frac{H_s}{bq}\right\}. E F ( z q ) ≤ γ q 2 6 R 2 = 288 R 2 max { q 2 L , b q H s } .
Corollary B.9 (Upper bound for general convex populations). There is an algorithm in IC ( m , K , N ) \operatorname{IC}(m,K,N) IC ( m , K , N ) such that, for every instance of W d \mathcal{W}_d W d ,
E F ( x ^ ) ≤ R 2 min { L 2 , 2 H s K N , 288 max { L N 2 , H s m K N } } \mathbb{E}F(\widehat{x}) \le R^2 \min\left\{\frac{L}{2},\frac{2H_s}{KN},288\max\left\{\frac{L}{N^2},\frac{H_s}{mKN}\right\}\right\} E F ( x ) ≤ R 2 min { 2 L , K N 2 H s , 288 max { N 2 L , m K N H s } } ≤ 288 R 2 min { L , H s K N , L N 2 + H s m K N } . (51) \le288R^2 \min\left\{L,\frac{H_s}{KN},\frac{L}{N^2}+\frac{H_s}{mKN}\right\}. \tag*{(51)} ≤ 288 R 2 min { L , K N H s , N 2 L + m K N H s } . (51) Proof. The algorithm selects, before the run, the branch with the smallest of the following three guarantees; the selection uses only the public parameters.
Zero queries. Returning x 0 x_0 x 0 gives F ( x 0 ) ≤ L 2 R 2 F(x_0) \le\frac{L}{2}R^2 F ( x 0 ) ≤ 2 L R 2 by Lemma B.2, Part 1.
Serial branch. One worker runs the recursion of Lemma B.3 with s = K N s = KN s = K N , making K K K queries in each round and keeping its state between rounds; the other workers are idle. Its risk is at most 2 H s R 2 / ( K N ) 2H_sR^2/(KN) 2 H s R 2 / ( K N ) .
Common-query branch. Run Algorithm 2 with q = N q = N q = N and b = m K b = mK b = m K . At the start of round t + 1 t+1 t + 1 the point y t y_t y t is broadcast, every worker makes its K K K queries at y t y_t y t , and at the end of the round the responses are averaged and the updates of Algorithm 2 are performed. Thus every iteration is one round, and Theorem B.8 bounds the risk by 288 R 2 max { L / N 2 , H s / ( m K N ) } 288R^2\max\{L/N^2,H_s/(mKN)\} 288 R 2 max { L / N 2 , H s / ( m K N )} .
The second inequality in (B.22) follows by increasing all coefficients to 288 and using max { a , b } ≤ a + b \max\{a,b\} \le a+b max { a , b } ≤ a + b .
Upper bound with sample reuse {#sec:C} Let Assumption 3 hold, and fix an instance of S d \mathcal{S}_d S d . Replacing f ξ f_\xi f ξ by f ξ − f ξ ( x ∗ ) f_\xi- f_\xi(x^*) f ξ − f ξ ( x ∗ ) does not change any gradient, so we assume that f ξ ≥ 0 f_\xi\ge0 f ξ ≥ 0 and f ξ ( x ∗ ) = 0 f_\xi(x^*) = 0 f ξ ( x ∗ ) = 0 for almost every ξ \xi ξ , and F = E f ξ F = \mathbb{E}f_\xi F = E f ξ . Let
C = { x ∈ R d : ∥ x − x 0 ∥ ≤ R } , C = \{x \in\mathbb{R}^d : \|x-x_0\| \le R\}, C = { x ∈ R d : ∥ x − x 0 ∥ ≤ R } , which is known to the algorithm and contains x ∗ x^* x ∗ , and let Π C \Pi_C Π C be the Euclidean projection onto C C C . A fresh batch of size n n n is a set of n n n labels that have not been queried before; since labels carry no information, the corresponding samples are independent of the history and of each other.
Acceleration with biased gradients
The following projected variant of Algorithm 2 is the accelerated scheme of Woodworth and Srebro [19 ] ; it is applied below both to the sample functions and to their empirical envelopes.
Lemma C.1 (Projected acceleration with biased gradients). Let q ζ q_\zeta q ζ be jointly measurable nonnegative convex functions of x ∈ R d x \in\mathbb{R}^d x ∈ R d , indexed by a random sample ζ \zeta ζ , with H ′ H' H ′ -Lipschitz gradients and a common zero x ∗ ∈ C x^* \in C x ∗ ∈ C , and let Q = E q ζ Q = \mathbb{E}q_\zeta Q = E q ζ have an L ′ L' L ′ -Lipschitz gradient, where L ′ , H ′ > 0 L', H' > 0 L ′ , H ′ > 0 . Fix integers b , T ≥ 1 b, T \ge1 b , T ≥ 1 and put
γ = min { 1 12 L ′ , b 24 H ′ ( T + 1 ) } , α t = 6 t + 6 , η t = γ ( t + 1 ) . \gamma= \min\left\{\frac{1}{12L'},\frac{b}{24H'(T+1)}\right\}, \qquad\alpha_t = \frac{6}{t+6}, \qquad\eta_t = \gamma(t+1). γ = min { 12 L ′ 1 , 24 H ′ ( T + 1 ) b } , α t = t + 6 6 , η t = γ ( t + 1 ) . Let z 0 = v 0 = x 0 z_0 = v_0 = x_0 z 0 = v 0 = x 0 and, for t = 0 , … , T − 1 t = 0,\ldots,T-1 t = 0 , … , T − 1 ,
y t = ( 1 − α t ) z t + α t v t , v t + 1 = Π C ( v t − η t g ~ t ) , z t + 1 = ( 1 − α t ) z t + α t v t + 1 , y_t = (1-\alpha_t)z_t + \alpha_t v_t, \qquad v_{t+1} = \Pi_C(v_t-\eta_t\widetilde{g}_t), \qquad z_{t+1} = (1-\alpha_t)z_t + \alpha_t v_{t+1}, y t = ( 1 − α t ) z t + α t v t , v t + 1 = Π C ( v t − η t g t ) , z t + 1 = ( 1 − α t ) z t + α t v t + 1 , where g ~ t = g t + e t \widetilde{g}_t = g_t + e_t g t = g t + e t , g t g_t g t is the average of ∇ Q ζ ( y t ) \nabla Q_\zeta(y_t) ∇ Q ζ ( y t ) over b b b samples drawn independently of the history F t \mathcal{F}_t F t before them, and the error e t e_t e t may depend on these samples and on additional randomness and satisfies
E [ ∥ e t ∥ ∣ F t ] ≤ ε . (52) \mathbb{E}[\|e_t\| \mid\mathcal{F}_t] \le\varepsilon. \tag*{(52)} E [ ∥ e t ∥ ∣ F t ] ≤ ε . (52) Then all y t , v t , z t y_t, v_t, z_t y t , v t , z t belong to C C C , and
E Q ( z T ) ≤ 36 L ′ R 2 T 2 + 288 H ′ R 2 b T + 12 R ε . (53) \mathbb{E}Q(z_T) \le\frac{36L'R^2}{T^2}+\frac{288H'R^2}{bT}+12R\varepsilon. \tag*{(53)} E Q ( z T ) ≤ T 2 36 L ′ R 2 + b T 288 H ′ R 2 + 12 R ε . (53)
Proof. The iterates belong to C C C by induction and convexity of C C C ; in particular, ∥ v t − z t ∥ ≤ 2 R \|v_t-z_t\| \le2R ∥ v t − z t ∥ ≤ 2 R and ∥ v t + 1 − x ∗ ∥ ≤ 2 R \|v_{t+1}-x^*\| \le2R ∥ v t + 1 − x ∗ ∥ ≤ 2 R . On C C C every ∇ q ζ \nabla q_\zeta ∇ q ζ is bounded by 2 H ′ R 2H'R 2 H ′ R (Lemma B.1, Part 4), so all quantities below are integrable, and, as in the proof of Lemma 4.1, the bound ∥ ∇ q ζ ( x ) ∥ ≤ H ′ ∥ x − x ∗ ∥ \|\nabla q_\zeta(x)\| \le H'\|x-x^*\| ∥∇ q ζ ( x ) ∥ ≤ H ′ ∥ x − x ∗ ∥ justifies ∇ Q = E ∇ q ζ \nabla Q = \mathbb{E}\nabla q_\zeta ∇ Q = E ∇ q ζ . Write E t [ ⋅ ] = E [ ⋅ ∣ F t ] \mathbb{E}_t[\cdot] = \mathbb{E}[\cdot\mid\mathcal{F}_t] E t [ ⋅ ] = E [ ⋅ ∣ F t ] , A t = η t / α t = γ ( t + 1 ) ( t + 6 ) / 6 A_t = \eta_t/\alpha_t = \gamma(t+1)(t+6)/6 A t = η t / α t = γ ( t + 1 ) ( t + 6 ) /6 , and ω t = g t − ∇ Q ( y t ) \omega_t = g_t-\nabla Q(y_t) ω t = g t − ∇ Q ( y t ) , so that E t ω t = 0 \mathbb{E}_t\omega_t = 0 E t ω t = 0 .
Variance. For fixed x , y x,y x , y , Lemma B.1, Part 3, applied to each q ζ q_\zeta q ζ , and the smoothness of Q Q Q give
E ∥ ∇ q ζ ( x ) − ∇ q ζ ( y ) ∥ 2 ≤ H ′ ⟨ ∇ Q ( x ) − ∇ Q ( y ) , x − y ⟩ ≤ H ′ L ′ ∥ x − y ∥ 2 , \mathbb{E}\|\nabla q_\zeta(x)-\nabla q_\zeta(y)\|^2 \le H'\langle\nabla Q(x)-\nabla Q(y),x-y\rangle\le H'L'\|x-y\|^2, E ∥∇ q ζ ( x ) − ∇ q ζ ( y ) ∥ 2 ≤ H ′ ⟨ ∇ Q ( x ) − ∇ Q ( y ) , x − y ⟩ ≤ H ′ L ′ ∥ x − y ∥ 2 ,
and Lemma B.1, Part 2, gives E ∥ ∇ q ζ ( x ) ∥ 2 ≤ 2 H ′ Q ( x ) \mathbb{E}\|\nabla q_\zeta(x)\|^2 \le2H'Q(x) E ∥∇ q ζ ( x ) ∥ 2 ≤ 2 H ′ Q ( x ) . As in the proof of Lemma B.2, Part 2, averaging b b b independent samples divides the variance by b b b , and the variance is at most the second moment. Using ∥ y t − z t ∥ = α t ∥ v t − z t ∥ ≤ 2 R α t \|y_t-z_t\| = \alpha_t\|v_t-z_t\| \le2R\alpha_t ∥ y t − z t ∥ = α t ∥ v t − z t ∥ ≤ 2 R α t , we obtain
E t ∥ ω t ∥ 2 ≤ 2 b E t ∥ ∇ q ζ ( y t ) − ∇ q ζ ( z t ) ∥ 2 + 2 b E t ∥ ∇ q ζ ( z t ) ∥ 2 ≤ 8 H ′ L ′ R 2 α t 2 b + 4 H ′ b Q ( z t ) . (54) \mathbb{E}_t\|\omega_t\|^2 \le\frac{2}{b}\mathbb{E}_t\|\nabla q_\zeta(y_t)-\nabla q_\zeta(z_t)\|^2+\frac{2}{b}\mathbb{E}_t\|\nabla q_\zeta(z_t)\|^2 \le\frac{8H'L'R^2\alpha_t^2}{b}+\frac{4H'}{b}Q(z_t). \tag*{(54)} E t ∥ ω t ∥ 2 ≤ b 2 E t ∥∇ q ζ ( y t ) − ∇ q ζ ( z t ) ∥ 2 + b 2 E t ∥∇ q ζ ( z t ) ∥ 2 ≤ b 8 H ′ L ′ R 2 α t 2 + b 4 H ′ Q ( z t ) . (54)
At t = 0 t=0 t = 0 we use instead y 0 = x 0 y_0=x_0 y 0 = x 0 and Lemma B.1, Part 4, for Q Q Q : E ∥ ω 0 ∥ 2 ≤ 2 H ′ b Q ( x 0 ) ≤ H ′ L ′ R 2 b \mathbb{E}\|\omega_0\|^2 \le\frac{2H'}{b}Q(x_0) \le\frac{H'L'R^2}{b} E ∥ ω 0 ∥ 2 ≤ b 2 H ′ Q ( x 0 ) ≤ b H ′ L ′ R 2 .
One step. As in (B.17), convexity gives Q ( y t ) ≤ ( 1 − α t ) Q ( z t ) + α t ⟨ ∇ Q ( y t ) , v t − x ∗ ⟩ Q(y_t) \le(1-\alpha_t)Q(z_t)+\alpha_t\langle\nabla Q(y_t),v_t-x^*\rangle Q ( y t ) ≤ ( 1 − α t ) Q ( z t ) + α t ⟨ ∇ Q ( y t ) , v t − x ∗ ⟩ . Since z t + 1 − y t = α t ( v t + 1 − v t ) z_{t+1}-y_t=\alpha_t(v_{t+1}-v_t) z t + 1 − y t = α t ( v t + 1 − v t ) , Lemma B.1, Part 1, gives
Q ( z t + 1 ) ≤ Q ( y t ) + α t ⟨ ∇ Q ( y t ) , v t + 1 − v t ⟩ + L ′ α t 2 2 ∥ v t + 1 − v t ∥ 2 . Q(z_{t+1}) \le Q(y_t)+\alpha_t\langle\nabla Q(y_t),v_{t+1}-v_t\rangle+\frac{L'\alpha_t^2}{2}\|v_{t+1}-v_t\|^2. Q ( z t + 1 ) ≤ Q ( y t ) + α t ⟨ ∇ Q ( y t ) , v t + 1 − v t ⟩ + 2 L ′ α t 2 ∥ v t + 1 − v t ∥ 2 .
Combining the two inequalities and multiplying by A t A_t A t ,
A t Q ( z t + 1 ) ≤ ( A t − η t ) Q ( z t ) + η t ⟨ ∇ Q ( y t ) , v t + 1 − x ∗ ⟩ + L ′ α t η t 2 ∥ v t + 1 − v t ∥ 2 . A_tQ(z_{t+1}) \le(A_t-\eta_t)Q(z_t)+\eta_t\langle\nabla Q(y_t),v_{t+1}-x^*\rangle+\frac{L'\alpha_t\eta_t}{2}\|v_{t+1}-v_t\|^2. A t Q ( z t + 1 ) ≤ ( A t − η t ) Q ( z t ) + η t ⟨ ∇ Q ( y t ) , v t + 1 − x ∗ ⟩ + 2 L ′ α t η t ∥ v t + 1 − v t ∥ 2 .
The optimality condition of the projection, ⟨ v t − η t g ~ t − v t + 1 , x ∗ − v t + 1 ⟩ ≤ 0 \langle v_t-\eta_t\widetilde{g}_t-v_{t+1},x^*-v_{t+1}\rangle\le0 ⟨ v t − η t g t − v t + 1 , x ∗ − v t + 1 ⟩ ≤ 0 , gives
η t ⟨ g ~ t , v t + 1 − x ∗ ⟩ ≤ 1 2 ( ∥ v t − x ∗ ∥ 2 − ∥ v t + 1 − x ∗ ∥ 2 − ∥ v t + 1 − v t ∥ 2 ) . \eta_t\langle\widetilde{g}_t,v_{t+1}-x^*\rangle\le\frac{1}{2}\left(\|v_t-x^*\|^2-\|v_{t+1}-x^*\|^2-\|v_{t+1}-v_t\|^2\right). η t ⟨ g t , v t + 1 − x ∗ ⟩ ≤ 2 1 ( ∥ v t − x ∗ ∥ 2 − ∥ v t + 1 − x ∗ ∥ 2 − ∥ v t + 1 − v t ∥ 2 ) .
Since ∇ Q ( y t ) = g ~ t − ω t − e t \nabla Q(y_t)=\widetilde{g}_t-\omega_t-e_t ∇ Q ( y t ) = g t − ω t − e t , the last two displays yield
A t Q ( z t + 1 ) + 1 2 ∥ v t + 1 − x ∗ ∥ 2 ≤ ( A t − η t ) Q ( z t ) + 1 2 ∥ v t − x ∗ ∥ 2 − 1 − L ′ α t η t 2 ∥ v t + 1 − v t ∥ 2 A_tQ(z_{t+1})+\frac{1}{2}\|v_{t+1}-x^*\|^2 \le(A_t-\eta_t)Q(z_t)+\frac{1}{2}\|v_t-x^*\|^2-\frac{1-L'\alpha_t\eta_t}{2}\|v_{t+1}-v_t\|^2 A t Q ( z t + 1 ) + 2 1 ∥ v t + 1 − x ∗ ∥ 2 ≤ ( A t − η t ) Q ( z t ) + 2 1 ∥ v t − x ∗ ∥ 2 − 2 1 − L ′ α t η t ∥ v t + 1 − v t ∥ 2
+ η t ⟨ ω t , x ∗ − v t + 1 ⟩ + η t ⟨ e t , x ∗ − v t + 1 ⟩ . (55) \qquad+\eta_t\langle\omega_t,x^*-v_{t+1}\rangle+\eta_t\langle e_t,x^*-v_{t+1}\rangle. \tag*{(55)} + η t ⟨ ω t , x ∗ − v t + 1 ⟩ + η t ⟨ e t , x ∗ − v t + 1 ⟩ . (55)
We have α t η t ≤ 6 γ ≤ 1 L ′ \alpha_t\eta_t \le6\gamma\le\frac{1}{L^{\prime}} α t η t ≤ 6 γ ≤ L ′ 1 , so the coefficient of ∥ v t + 1 − v t ∥ 2 \lVert v_{t+1}-v_t\rVert^2 ∥ v t + 1 − v t ∥ 2 is at most − 1 4 -\frac{1}{4} − 4 1 . Split the noise term at v t v_t v t : η t ⟨ ω t , x ∗ − v t ⟩ \eta_t\langle\omega_t,x^*-v_t\rangle η t ⟨ ω t , x ∗ − v t ⟩ has zero conditional expectation, and η t ⟨ ω t , v t − v t + 1 ⟩ ≤ η t 2 ∥ ω t ∥ 2 + 1 4 ∥ v t + 1 − v t ∥ 2 \eta_t\langle\omega_t,v_t-v_{t+1}\rangle\le\eta_t^2\lVert\omega_t\rVert^2+\frac{1}{4}\lVert v_{t+1}-v_t\rVert^2 η t ⟨ ω t , v t − v t + 1 ⟩ ≤ η t 2 ∥ ω t ∥ 2 + 4 1 ∥ v t + 1 − v t ∥ 2 . The bias term is at most 2 R η t ∥ e t ∥ 2R\eta_t\lVert e_t\rVert 2 R η t ∥ e t ∥ . Taking expectations and using (C.1),
E [ A t Q ( z t + 1 ) + 1 2 ∥ v t + 1 − x ∗ ∥ 2 ] ≤ E [ ( A t − η t ) Q ( z t ) + 1 2 ∥ v t − x ∗ ∥ 2 ] + η t 2 E ∥ ω t ∥ 2 + 2 R η t ε . (56) \mathbb{E}\left[A_tQ(z_{t+1})+\frac{1}{2}\lVert v_{t+1}-x^*\rVert^2\right]\le\mathbb{E}\left[(A_t-\eta_t)Q(z_t)+\frac{1}{2}\lVert v_t-x^*\rVert^2\right]+\eta_t^2\mathbb{E}\lVert\omega_t\rVert^2+2R\eta_t\varepsilon. \tag*{(56)} E [ A t Q ( z t + 1 ) + 2 1 ∥ v t + 1 − x ∗ ∥ 2 ] ≤ E [ ( A t − η t ) Q ( z t ) + 2 1 ∥ v t − x ∗ ∥ 2 ] + η t 2 E ∥ ω t ∥ 2 + 2 R η t ε . (56)
Summation. As in the proof of Theorem B.8 , A t − 1 − ( A t − η t ) = 2 3 γ t A_{t-1}-(A_t-\eta_t)=\frac{2}{3}\gamma t A t − 1 − ( A t − η t ) = 3 2 γ t for t ≥ 1 t\ge1 t ≥ 1 , and A 0 = η 0 A_0=\eta_0 A 0 = η 0 . By (C.3), the coefficient of E Q ( z t ) \mathbb{E}Q(z_t) E Q ( z t ) contributed by η t 2 E ∥ ω t ∥ 2 \eta_t^2\mathbb{E}\lVert\omega_t\rVert^2 η t 2 E ∥ ω t ∥ 2 is 4 H ′ γ 2 ( t + 1 ) 2 b ≤ γ ( t + 1 ) 6 ( T + 1 ) ≤ γ ( t + 1 ) 6 ≤ γ t 3 \frac{4H^{\prime}\gamma^2(t+1)^2}{b}\le\frac{\gamma(t+1)}{6(T+1)}\le\frac{\gamma(t+1)}{6}\le\frac{\gamma t}{3} b 4 H ′ γ 2 ( t + 1 ) 2 ≤ 6 ( T + 1 ) γ ( t + 1 ) ≤ 6 γ ( t + 1 ) ≤ 3 γ t for 1 ≤ t ≤ T − 1 1\le t\le T-1 1 ≤ t ≤ T − 1 , where we used γ ≤ b 24 H ′ ( T + 1 ) \gamma\le\frac{b}{24H^{\prime}(T+1)} γ ≤ 24 H ′ ( T + 1 ) b and t + 1 ≤ 2 t t+1\le2t t + 1 ≤ 2 t . Hence these terms are absorbed when (C.5) is summed over t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 , and, using the bound for t = 0 t=0 t = 0 and ∥ x 0 − x ∗ ∥ ≤ R \lVert x_0-x^*\rVert\le R ∥ x 0 − x ∗ ∥ ≤ R ,
A T − 1 E Q ( z T ) ≤ R 2 2 + 8 H ′ L ′ R 2 b ∑ t = 0 T − 1 ( α t η t ) 2 + 2 R ε ∑ t = 0 T − 1 η t . A_{T-1}\mathbb{E}Q(z_T)\le\frac{R^2}{2}+\frac{8H^{\prime}L^{\prime}R^2}{b}\sum_{t=0}^{T-1}(\alpha_t\eta_t)^2+2R\varepsilon\sum_{t=0}^{T-1}\eta_t. A T − 1 E Q ( z T ) ≤ 2 R 2 + b 8 H ′ L ′ R 2 t = 0 ∑ T − 1 ( α t η t ) 2 + 2 R ε t = 0 ∑ T − 1 η t .
Since A T − 1 ≥ γ T 2 6 A_{T-1}\ge\frac{\gamma T^2}{6} A T − 1 ≥ 6 γ T 2 , α t η t ≤ 6 γ \alpha_t\eta_t\le6\gamma α t η t ≤ 6 γ , and ∑ t < T η t = γ T ( T + 1 ) 2 \sum_{t<T}\eta_t=\frac{\gamma T(T+1)}{2} ∑ t < T η t = 2 γ T ( T + 1 ) ,
E Q ( z T ) ≤ 3 R 2 γ T 2 + 1728 H ′ L ′ R 2 γ b T + 6 R ε ( T + 1 ) T . \mathbb{E}Q(z_T)\le\frac{3R^2}{\gamma T^2}+\frac{1728H^{\prime}L^{\prime}R^2\gamma}{bT}+\frac{6R\varepsilon(T+1)}{T}. E Q ( z T ) ≤ γ T 2 3 R 2 + b T 1728 H ′ L ′ R 2 γ + T 6 R ε ( T + 1 ) .
Finally, γ − 1 ≤ 12 L ′ + 24 H ′ ( T + 1 ) b \gamma^{-1}\le12L^{\prime}+\frac{24H^{\prime}(T+1)}{b} γ − 1 ≤ 12 L ′ + b 24 H ′ ( T + 1 ) , γ ≤ 1 12 L ′ \gamma\le\frac{1}{12L^{\prime}} γ ≤ 12 L ′ 1 , and T + 1 ≤ 2 T T+1\le2T T + 1 ≤ 2 T give (C.2).
With q ξ = f ξ q_{\xi}=f_{\xi} q ξ = f ξ , Q = F Q=F Q = F , L ′ = L L^{\prime}=L L ′ = L , H ′ = H s H^{\prime}=H_s H ′ = H s , and e t = 0 e_t=0 e t = 0 , Lemma C.1 gives
E F ( z T ) ≤ 36 L R 2 T 2 + 288 H s R 2 b T . (57) \mathbb{E}F(z_T)\le\frac{36LR^2}{T^2}+\frac{288H_sR^2}{bT}. \tag*{(57)} E F ( z T ) ≤ T 2 36 L R 2 + b T 288 H s R 2 . (57)
For T = N T=N T = N and b ≤ m K b\le mK b ≤ m K , the b b b queries of every iteration are made at the common point y t y_t y t and distributed among the workers, at most K K K per worker; every iteration is one round.
Empirical envelopes
For a fresh batch with samples S = ( ξ 1 , … , ξ n ) S=(\xi_1,\ldots,\xi_n) S = ( ξ 1 , … , ξ n ) and τ > 0 \tau>0 τ > 0 , define
f S = 1 n ∑ i = 1 n f ξ i , h S , y ( u ) = f S ( u ) + τ 2 ∥ u − y ∥ 2 , p S ( y ) = arg min u h S , y ( u ) , f_S=\frac{1}{n}\sum_{i=1}^{n}f_{\xi_i},\qquad h_{S,y}(u)=f_S(u)+\frac{\tau}{2}\lVert u-y\rVert^2,\qquad p_S(y)=\mathop{\arg\min}_{u}h_{S,y}(u), f S = n 1 i = 1 ∑ n f ξ i , h S , y ( u ) = f S ( u ) + 2 τ ∥ u − y ∥ 2 , p S ( y ) = arg min u h S , y ( u ) ,
ϕ S ( y ) = min u h S , y ( u ) , ϕ ˉ ( y ) = E S ϕ S ( y ) . \phi_S(y)=\min_{u}h_{S,y}(u),\qquad\bar{\phi}(y)=\mathbb{E}_S\phi_S(y). ϕ S ( y ) = u min h S , y ( u ) , ϕ ˉ ( y ) = E S ϕ S ( y ) .
These functions are used only in the analysis; the algorithm has no proximal oracle.
Lemma C.2 (Envelopes). For every batch, ϕ S \phi_S ϕ S is nonnegative and convex, has the τ \tau τ -Lipschitz gradient
∇ ϕ S ( y ) = τ ( y − p S ( y ) ) , (58) \nabla\phi_S(y)=\tau(y-p_S(y)), \tag*{(58)} ∇ ϕ S ( y ) = τ ( y − p S ( y )) , (58) and satisfies ϕ S ( x ∗ ) = 0 \phi_S(x^*)=0 ϕ S ( x ∗ ) = 0 , ϕ S ( y ) ≤ τ 2 ∥ y − x ∗ ∥ 2 \phi_S(y)\le\frac{\tau}{2}\lVert y-x^*\rVert^2 ϕ S ( y ) ≤ 2 τ ∥ y − x ∗ ∥ 2 , and ∥ p S ( y ) − x ∗ ∥ ≤ ∥ y − x ∗ ∥ \lVert p_S(y)-x^*\rVert\le\lVert y-x^*\rVert ∥ p S ( y ) − x ∗ ∥ ≤ ∥ y − x ∗ ∥ . The map ( S , y ) ↦ p S ( y ) (S,y)\mapsto p_S(y) ( S , y ) ↦ p S ( y ) is jointly measurable, and ϕ ˉ \bar{\phi} ϕ ˉ is nonnegative and convex, has a τ \tau τ -Lipschitz gradient ∇ ϕ ˉ = E ∇ ϕ S \nabla\bar{\phi}=\mathbb{E}\nabla\phi_S ∇ ϕ ˉ = E ∇ ϕ S , and vanishes at x ∗ x^* x ∗ . For y ∈ C y\in C y ∈ C , h S , y ( y ) − ϕ S ( y ) ≤ 2 H s R 2 h_{S,y}(y)-\phi_S(y)\le2H_sR^2 h S , y ( y ) − ϕ S ( y ) ≤ 2 H s R 2 .
Proof. The function h S , y h_{S,y} h S , y is continuous and τ \tau τ -strongly convex, so p S ( y ) p_S(y) p S ( y ) is well defined. Write p = p S p=p_S p = p S . The optimality conditions at the centers y y y and y ′ y' y ′ , ∇ f S ( p ( y ) ) = τ ( y − p ( y ) ) \nabla f_S(p(y))=\tau(y-p(y)) ∇ f S ( p ( y )) = τ ( y − p ( y )) , and monotonicity of ∇ f S \nabla f_S ∇ f S give ⟨ p ( y ) − p ( y ′ ) , y − y ′ ⟩ ≥ ∥ p ( y ) − p ( y ′ ) ∥ 2 \langle p(y)-p(y'),y-y'\rangle\ge\lVert p(y)-p(y')\rVert^2 ⟨ p ( y ) − p ( y ′ ) , y − y ′ ⟩ ≥ ∥ p ( y ) − p ( y ′ ) ∥ 2 . Hence p p p is nonexpansive, and so is I − p I-p I − p : with Δ = y − y ′ \Delta=y-y' Δ = y − y ′ and Δ p = p ( y ) − p ( y ′ ) \Delta_p=p(y)-p(y') Δ p = p ( y ) − p ( y ′ ) ,
∥ Δ − Δ p ∥ 2 = ∥ Δ ∥ 2 − 2 ⟨ Δ , Δ p ⟩ + ∥ Δ p ∥ 2 ≤ ∥ Δ ∥ 2 − ∥ Δ p ∥ 2 ≤ ∥ Δ ∥ 2 . \lVert\Delta-\Delta_p\rVert^2=\lVert\Delta\rVert^2-2\langle\Delta,\Delta_p\rangle+\lVert\Delta_p\rVert^2\le\lVert\Delta\rVert^2-\lVert\Delta_p\rVert^2\le\lVert\Delta\rVert^2. ∥ Δ − Δ p ∥ 2 = ∥ Δ ∥ 2 − 2 ⟨ Δ , Δ p ⟩ + ∥ Δ p ∥ 2 ≤ ∥ Δ ∥ 2 − ∥ Δ p ∥ 2 ≤ ∥ Δ ∥ 2 .
Evaluating h S , y ′ h_{S,y'} h S , y ′ at p ( y ) p(y) p ( y ) and h S , y h_{S,y} h S , y at p ( y ′ ) p(y') p ( y ′ ) gives
ϕ S ( y ′ ) − ϕ S ( y ) ≤ τ ⟨ y − p ( y ) , y ′ − y ⟩ + τ 2 ∥ y ′ − y ∥ 2 , \phi_S(y')-\phi_S(y)\le\tau\langle y-p(y),y'-y\rangle+\frac{\tau}{2}\lVert y'-y\rVert^2, ϕ S ( y ′ ) − ϕ S ( y ) ≤ τ ⟨ y − p ( y ) , y ′ − y ⟩ + 2 τ ∥ y ′ − y ∥ 2 ,
ϕ S ( y ′ ) − ϕ S ( y ) ≥ τ ⟨ y ′ − p ( y ′ ) , y ′ − y ⟩ − τ 2 ∥ y ′ − y ∥ 2 . \phi_S(y')-\phi_S(y)\ge\tau\langle y'-p(y'),y'-y\rangle-\frac{\tau}{2}\lVert y'-y\rVert^2. ϕ S ( y ′ ) − ϕ S ( y ) ≥ τ ⟨ y ′ − p ( y ′ ) , y ′ − y ⟩ − 2 τ ∥ y ′ − y ∥ 2 .
Since I − p I-p I − p is nonexpansive, ∣ τ ⟨ ( y ′ − p ( y ′ ) ) − ( y − p ( y ) ) , y ′ − y ⟩ ∣ ≤ τ ∥ y ′ − y ∥ 2 \left|\tau\left\langle(y'-p(y'))-(y-p(y)),y'-y\right\rangle\right|\leq\tau\lVert y'-y\rVert^2 ∣ τ ⟨ ( y ′ − p ( y ′ )) − ( y − p ( y )) , y ′ − y ⟩ ∣ ≤ τ ∥ y ′ − y ∥ 2 , and therefore ∣ φ S ( y ′ ) − φ S ( y ) − τ ⟨ y − p ( y ) , y ′ − y ⟩ ∣ ≤ 3 τ 2 ∥ y ′ − y ∥ 2 \left|\varphi_S(y')-\varphi_S(y)-\tau\left\langle y-p(y),y'-y\right\rangle\right|\leq\frac{3\tau}{2}\lVert y'-y\rVert^2 ∣ φ S ( y ′ ) − φ S ( y ) − τ ⟨ y − p ( y ) , y ′ − y ⟩ ∣ ≤ 2 3 τ ∥ y ′ − y ∥ 2 . This proves differentiability and (C.7), and nonexpansiveness of I − p I-p I − p gives the τ \tau τ -Lipschitz property. Convexity holds because φ S \varphi_S φ S is a partial minimum of a jointly convex function, and nonnegativity because f S ≥ 0 f_S\geq0 f S ≥ 0 . Taking u = x ∗ u=x^* u = x ∗ in the minimum gives φ S ( y ) ≤ τ 2 ∥ y − x ∗ ∥ 2 \varphi_S(y)\leq\frac{\tau}{2}\lVert y-x^*\rVert^2 φ S ( y ) ≤ 2 τ ∥ y − x ∗ ∥ 2 and φ S ( x ∗ ) = 0 \varphi_S(x^*)=0 φ S ( x ∗ ) = 0 ; since x ∗ x^* x ∗ is then the unique minimizer of h S , x ∗ h_{S,x^*} h S , x ∗ , p ( x ∗ ) = x ∗ p(x^*)=x^* p ( x ∗ ) = x ∗ , and nonexpansiveness gives ∥ p ( y ) − x ∗ ∥ ≤ ∥ y − x ∗ ∥ \lVert p(y)-x^*\rVert\leq\lVert y-x^*\rVert ∥ p ( y ) − x ∗ ∥ ≤ ∥ y − x ∗ ∥ .
For measurability, enumerate a countable dense set { u k } ⊆ R d \{u_k\}\subseteq\mathbb{R}^d { u k } ⊆ R d . The minimum of the continuous function h S , y h_{S,y} h S , y equals its infimum over { u k } \{u_k\} { u k } and is measurable in ( S , y ) (S,y) ( S , y ) . The first u k u_k u k whose value is within 1 / j 1/j 1/ j of the minimum is a measurable function of ( S , y ) (S,y) ( S , y ) and, by strong convexity, lies within 2 / ( τ j ) \sqrt{2/(\tau j)} 2/ ( τ j ) of p ( y ) p(y) p ( y ) ; letting j → ∞ j\to\infty j → ∞ shows that p p p is jointly measurable. The bound ∥ ∇ φ S ( y ) ∥ = ∥ ∇ φ S ( y ) − ∇ φ S ( x ∗ ) ∥ ≤ τ ∥ y − x ∗ ∥ \lVert\nabla\varphi_S(y)\rVert=\lVert\nabla\varphi_S(y)-\nabla\varphi_S(x^*)\rVert\leq\tau\lVert y-x^*\rVert ∥ ∇ φ S ( y )∥ = ∥ ∇ φ S ( y ) − ∇ φ S ( x ∗ )∥ ≤ τ ∥ y − x ∗ ∥ justifies differentiation under the expectation, as in the proof of Lemma 4.1, and the properties of φ ˉ \bar{\varphi} φ ˉ follow. Finally, for y ∈ C y\in C y ∈ C , ∥ y − x ∗ ∥ ≤ 2 R \lVert y-x^*\rVert\leq2R ∥ y − x ∗ ∥ ≤ 2 R and h S , y ( y ) − φ S ( y ) ≤ f S ( y ) ≤ H s 2 ∥ y − x ∗ ∥ 2 ≤ 2 H s R 2 h_{S,y}(y)-\varphi_S(y)\leq f_S(y)\leq\frac{H_s}{2}\lVert y-x^*\rVert^2\leq2H_sR^2 h S , y ( y ) − φ S ( y ) ≤ f S ( y ) ≤ 2 H s ∥ y − x ∗ ∥ 2 ≤ 2 H s R 2 by (B.2).
Lemma C.3 (From the envelope to the population). Let τ = 8 H s / n \tau=8H_s/n τ = 8 H s / n . For every fixed y y y ,
E S F ( p S ( y ) ) ≤ 2 φ ˉ ( y ) . (59) \mathbb{E}_S F(p_S(y))\leq2\bar{\varphi}(y). \tag*{(59)} E S F ( p S ( y )) ≤ 2 φ ˉ ( y ) . (59)
If y y y is a random point independent of the batch and p ~ \tilde{p} p ~ satisfies E [ h S , y ( p ~ ) − φ S ( y ) ∣ y ] ≤ δ \mathbb{E}[h_{S,y}(\tilde{p})-\varphi_S(y)\mid y]\leq\delta E [ h S , y ( p ~ ) − φ S ( y ) ∣ y ] ≤ δ , then
E F ( p ~ ) ≤ 4 E φ ˉ ( y ) + 2 H s δ τ . (60) \mathbb{E}F(\tilde{p})\leq4\mathbb{E}\bar{\varphi}(y)+\frac{2H_s\delta}{\tau}. \tag*{(60)} E F ( p ~ ) ≤ 4 E φ ˉ ( y ) + τ 2 H s δ . (60)
Proof. Fix y y y and i ≤ n i\leq n i ≤ n , let ξ i ′ \xi_i' ξ i ′ be an independent copy of ξ i \xi_i ξ i , and let S ′ S' S ′ be the batch with ξ i \xi_i ξ i replaced by ξ i ′ \xi_i' ξ i ′ . Put u = p S ( y ) u=p_S(y) u = p S ( y ) , u ′ = p S ′ ( y ) u'=p_{S'}(y) u ′ = p S ′ ( y ) , and Δ = ∥ u − u ′ ∥ \Delta=\lVert u-u'\rVert Δ = ∥ u − u ′ ∥ . Strong convexity of h S , y h_{S,y} h S , y and h S ′ , y h_{S',y} h S ′ , y at their minimizers gives h S , y ( u ′ ) − h S , y ( u ) ≥ τ 2 Δ 2 h_{S,y}(u')-h_{S,y}(u)\geq\frac{\tau}{2}\Delta^2 h S , y ( u ′ ) − h S , y ( u ) ≥ 2 τ Δ 2 and h S ′ , y ( u ) − h S ′ , y ( u ′ ) ≥ τ 2 Δ 2 h_{S',y}(u)-h_{S',y}(u')\geq\frac{\tau}{2}\Delta^2 h S ′ , y ( u ) − h S ′ , y ( u ′ ) ≥ 2 τ Δ 2 . Adding these inequalities and using h S , y − h S ′ , y = 1 n ( f ξ i − f ξ i ′ ) h_{S,y}-h_{S',y}=\frac{1}{n}(f_{\xi_i}-f_{\xi_i'}) h S , y − h S ′ , y = n 1 ( f ξ i − f ξ i ′ ) and convexity of f ξ i f_{\xi_i} f ξ i and f ξ i ′ f_{\xi_i'} f ξ i ′ ,
τ Δ 2 ≤ 1 n ( f ξ i ( u ′ ) − f ξ i ( u ) + f ξ i ′ ( u ) − f ξ i ′ ( u ′ ) ) ≤ Δ n ( ∥ ∇ f ξ i ( u ′ ) ∥ + ∥ ∇ f ξ i ′ ( u ) ∥ ) . (61) \tau\Delta^2\leq\frac{1}{n}\left(f_{\xi_i}(u')-f_{\xi_i}(u)+f_{\xi_i'}(u)-f_{\xi_i'}(u')\right)\leq\frac{\Delta}{n}\left(\lVert\nabla f_{\xi_i}(u')\rVert+\lVert\nabla f_{\xi_i'}(u)\rVert\right). \tag*{(61)} τ Δ 2 ≤ n 1 ( f ξ i ( u ′ ) − f ξ i ( u ) + f ξ i ′ ( u ) − f ξ i ′ ( u ′ ) ) ≤ n Δ ( ∥ ∇ f ξ i ( u ′ )∥ + ∥ ∇ f ξ i ′ ( u )∥ ) . (61)
Let R y = E F ( u ) R_y=\mathbb{E}F(u) R y = E F ( u ) , which is finite by Lemma C.2. The point u ′ u' u ′ is independent of ξ i \xi_i ξ i , the point u u u is independent of ξ i ′ \xi_i' ξ i ′ , and u , u ′ u,u' u , u ′ have the same law. Hence, by (B.3), E ∥ ∇ f ξ i ( u ′ ) ∥ 2 ≤ 2 H s R y \mathbb{E}\lVert\nabla f_{\xi_i}(u')\rVert^2\leq2H_sR_y E ∥ ∇ f ξ i ( u ′ ) ∥ 2 ≤ 2 H s R y and E ∥ ∇ f ξ i ′ ( u ) ∥ 2 ≤ 2 H s R y \mathbb{E}\lVert\nabla f_{\xi_i'}(u)\rVert^2\leq2H_sR_y E ∥ ∇ f ξ i ′ ( u ) ∥ 2 ≤ 2 H s R y , and squaring (C.10) gives E Δ 2 ≤ 8 H s R y τ 2 n 2 \mathbb{E}\Delta^2\leq\frac{8H_sR_y}{\tau^2n^2} E Δ 2 ≤ τ 2 n 2 8 H s R y . Convexity and the Cauchy–Schwarz inequality then give
E [ f ξ i ( u ′ ) − f ξ i ( u ) ] ≤ E ∥ ∇ f ξ i ( u ′ ) ∥ 2 E Δ 2 ≤ 4 H s τ n R y = R y 2 . \mathbb{E}\left[f_{\xi_i}(u')-f_{\xi_i}(u)\right]\leq\sqrt{\mathbb{E}\lVert\nabla f_{\xi_i}(u')\rVert^2\mathbb{E}\Delta^2}\leq\frac{4H_s}{\tau n}R_y=\frac{R_y}{2}. E [ f ξ i ( u ′ ) − f ξ i ( u ) ] ≤ E ∥ ∇ f ξ i ( u ′ ) ∥ 2 E Δ 2 ≤ τ n 4 H s R y = 2 R y .
Since u ′ u' u ′ is independent of ξ i \xi_i ξ i and has the law of u u u , E f ξ i ( u ′ ) = R y \mathbb{E}f_{\xi_i}(u')=R_y E f ξ i ( u ′ ) = R y . Averaging over i i i gives R y − E f S ( u ) ≤ R y 2 R_y-\mathbb{E}f_S(u)\leq\frac{R_y}{2} R y − E f S ( u ) ≤ 2 R y , so R y ≤ 2 E f S ( u ) ≤ 2 E φ S ( y ) = 2 φ ˉ ( y ) R_y\leq2\mathbb{E}f_S(u)\leq2\mathbb{E}\varphi_S(y)=2\bar{\varphi}(y) R y ≤ 2 E f S ( u ) ≤ 2 E φ S ( y ) = 2 φ ˉ ( y ) .
For the second claim, strong convexity gives E [ ∥ p ~ − p S ( y ) ∥ 2 ∣ y ] ≤ 2 δ / τ \mathbb{E}[\lVert\tilde{p}-p_S(y)\rVert^2\mid y]\leq2\delta/\tau E [∥ p ~ − p S ( y ) ∥ 2 ∣ y ] ≤ 2 δ / τ . Lemma B.1, Part 2, applied to F F F , which is nonnegative and H s H_s H s -smooth, gives F ( p ~ ) ≤ 2 F ( p S ( y ) ) + H s ∥ p ~ − p S ( y ) ∥ 2 F(\tilde{p})\leq2F(p_S(y))+H_s\lVert\tilde{p}-p_S(y)\rVert^2 F ( p ~ ) ≤ 2 F ( p S ( y )) + H s ∥ p ~ − p S ( y ) ∥ 2 . Taking expectations conditional on y y y and using (C.8) proves (C.9).
The local solver
The approximate proximal points are computed by the variance-reduced method of Johnson and Zhang [3 ] ; it uses only gradients.
Lemma C.4 (One epoch of SVRG). Let h = 1 n ∑ i = 1 n h i h=\frac{1}{n}\sum_{i=1}^n h_i h = n 1 ∑ i = 1 n h i , where every h i h_i h i is convex with an M M M -Lipschitz gradient and h h h is μ \mu μ -strongly convex with minimizer u u u . Given a snapshot x ˉ \bar{x} x ˉ , store all ∇ h i ( x ˉ ) \nabla h_i(\bar{x}) ∇ h i ( x ˉ ) and their average ∇ h ( x ˉ ) \nabla h(\bar{x}) ∇ h ( x ˉ ) . Set w 0 = x ˉ w_0=\bar{x} w 0 = x ˉ and, for k = 0 , … , I − 1 k=0,\ldots,I-1 k = 0 , … , I − 1 , select i k i_k i k uniformly from { 1 , … , n } \{1,\ldots,n\} { 1 , … , n } , independently, and set
V k = ∇ h i k ( w k ) − ∇ h i k ( x ˉ ) + ∇ h ( x ˉ ) , w k + 1 = w k − β V k . V_k=\nabla h_{i_k}(w_k)-\nabla h_{i_k}(\bar{x})+\nabla h(\bar{x}),\qquad w_{k+1}=w_k-\beta V_k. V k = ∇ h i k ( w k ) − ∇ h i k ( x ˉ ) + ∇ h ( x ˉ ) , w k + 1 = w k − β V k .
Return w κ w_\kappa w κ with κ \kappa κ uniform on { 0 , … , I − 1 } \{0,\ldots,I-1\} { 0 , … , I − 1 } , independent of everything else. If 0 < 2 M β < 1 0<2M\beta<1 0 < 2 M β < 1 , then
E [ h ( w κ ) − h ( u ) ∣ x ˉ ] ≤ [ 1 μ β ( 1 − 2 M β ) I + 2 M β 1 − 2 M β ] ( h ( x ˉ ) − h ( u ) ) . (62) \mathbb{E}[h(w_\kappa)-h(u)\mid\bar{x}]\leq\left[\frac{1}{\mu\beta(1-2M\beta)I}+\frac{2M\beta}{1-2M\beta}\right](h(\bar{x})-h(u)). \tag*{(62)} E [ h ( w κ ) − h ( u ) ∣ x ˉ ] ≤ [ μ β ( 1 − 2 M β ) I 1 + 1 − 2 M β 2 M β ] ( h ( x ˉ ) − h ( u )) . (62)
Proof. By Lemma B.1, Part 3, applied to each h i h_i h i at u u u , and ∇ h ( u ) = 0 \nabla h(u)=0 ∇ h ( u ) = 0 ,
1 n ∑ i = 1 n ∥ ∇ h i ( x ) − ∇ h i ( u ) ∥ 2 ≤ 2 M ( h ( x ) − h ( u ) ) for every x . \frac{1}{n}\sum_{i=1}^n\lVert\nabla h_i(x)-\nabla h_i(u)\rVert^2\leq2M(h(x)-h(u))\qquad\text{for every }x. n 1 i = 1 ∑ n ∥ ∇ h i ( x ) − ∇ h i ( u ) ∥ 2 ≤ 2 M ( h ( x ) − h ( u )) for every x .
Fix the inner history up to w k w_k w k and write V k = a i k − ( c i k − c ˉ ) V_k=a_{i_k}-(c_{i_k}-\bar{c}) V k = a i k − ( c i k − c ˉ ) with a i = ∇ h i ( w k ) − ∇ h i ( u ) a_i=\nabla h_i(w_k)-\nabla h_i(u) a i = ∇ h i ( w k ) − ∇ h i ( u ) , c i = ∇ h i ( x ˉ ) − ∇ h i ( u ) c_i=\nabla h_i(\bar{x})-\nabla h_i(u) c i = ∇ h i ( x ˉ ) − ∇ h i ( u ) , and c ˉ = 1 n ∑ i c i = ∇ h ( x ˉ ) \bar{c}=\frac{1}{n}\sum_i c_i=\nabla h(\bar{x}) c ˉ = n 1 ∑ i c i = ∇ h ( x ˉ ) . Then E [ V k ∣ w k ] = ∇ h ( w k ) \mathbb{E}[V_k\mid w_k]=\nabla h(w_k) E [ V k ∣ w k ] = ∇ h ( w k ) , and since the variance of c i k c_{i_k} c i k is at most its second moment,
E [ ∥ V k ∥ 2 ∣ w k ] ≤ 2 E ∥ a i k ∥ 2 + 2 E ∥ c i k ∥ 2 ≤ 4 M ( h ( w k ) − h ( u ) ) + 4 M ( h ( x ˉ ) − h ( u ) ) . \mathbb{E}[\|V_k\|^2\mid w_k]\leq2\mathbb{E}\|a_{i_k}\|^2+2\mathbb{E}\|c_{i_k}\|^2\leq4M(h(w_k)-h(u))+4M(h(\bar{x})-h(u)). E [ ∥ V k ∥ 2 ∣ w k ] ≤ 2 E ∥ a i k ∥ 2 + 2 E ∥ c i k ∥ 2 ≤ 4 M ( h ( w k ) − h ( u )) + 4 M ( h ( x ˉ ) − h ( u )) .
Expanding the square and using ⟨ ∇ h ( w k ) , w k − u ⟩ ≥ h ( w k ) − h ( u ) \langle\nabla h(w_k),w_k-u\rangle\geq h(w_k)-h(u) ⟨ ∇ h ( w k ) , w k − u ⟩ ≥ h ( w k ) − h ( u ) ,
E [ ∥ w k + 1 − u ∥ 2 ∣ w k ] ≤ ∥ w k − u ∥ 2 − 2 β ( 1 − 2 M β ) ( h ( w k ) − h ( u ) ) + 4 M β 2 ( h ( x ˉ ) − h ( u ) ) . \mathbb{E}[\|w_{k+1}-u\|^2\mid w_k]\leq\|w_k-u\|^2-2\beta(1-2M\beta)(h(w_k)-h(u))+4M\beta^2(h(\bar{x})-h(u)). E [ ∥ w k + 1 − u ∥ 2 ∣ w k ] ≤ ∥ w k − u ∥ 2 − 2 β ( 1 − 2 M β ) ( h ( w k ) − h ( u )) + 4 M β 2 ( h ( x ˉ ) − h ( u )) .
Summing over k = 0 , … , I − 1 k=0,\ldots,I-1 k = 0 , … , I − 1 , discarding the final squared distance, and using ∥ x ˉ − u ∥ 2 ≤ 2 μ ( h ( x ˉ ) − h ( u ) ) \|\bar{x}-u\|^2\leq\frac{2}{\mu}(h(\bar{x})-h(u)) ∥ x ˉ − u ∥ 2 ≤ μ 2 ( h ( x ˉ ) − h ( u )) by strong convexity,
2 β ( 1 − 2 M β ) ∑ k = 0 I − 1 E [ h ( w k ) − h ( u ) ∣ x ˉ ] ≤ ( 2 μ + 4 M β 2 I ) ( h ( x ˉ ) − h ( u ) ) . 2\beta(1-2M\beta)\sum_{k=0}^{I-1}\mathbb{E}[h(w_k)-h(u)\mid\bar{x}]\leq\left(\frac{2}{\mu}+4M\beta^2I\right)(h(\bar{x})-h(u)). 2 β ( 1 − 2 M β ) k = 0 ∑ I − 1 E [ h ( w k ) − h ( u ) ∣ x ˉ ] ≤ ( μ 2 + 4 M β 2 I ) ( h ( x ˉ ) − h ( u )) .
Dividing by 2 β ( 1 − 2 M β ) I 2\beta(1-2M\beta)I 2 β ( 1 − 2 M β ) I proves (C.11), because w κ w_\kappa w κ is uniform over w 0 , … , w I − 1 w_0,\ldots,w_{I-1} w 0 , … , w I − 1 .
The local routine. Fix integers n ≥ 1 n\geq1 n ≥ 1 , T ≥ 0 T\geq0 T ≥ 0 , and j ≥ 1 j\geq1 j ≥ 1 . Given a point y ∈ C y\in\mathcal{C} y ∈ C and a fresh batch with samples S S S , the procedure LOCALPROX of Algorithm 3 runs j j j epochs of Lemma C.4 on h = h S , y h=h_{S,y} h = h S , y with h i ( x ) = f ξ i ( x ) + τ 2 ∥ x − y ∥ 2 h_i(x)=f_{\xi_i}(x)+\frac{\tau}{2}\|x-y\|^2 h i ( x ) = f ξ i ( x ) + 2 τ ∥ x − y ∥ 2 , the first snapshot y y y , every later snapshot equal to the output of the preceding epoch, and
τ = 8 H s n , β = 1 10 ( H s + τ ) , I = ⌈ 100 ( H s + τ ) τ ⌉ . (63) \tau=\frac{8H_s}{n},\qquad\beta=\frac{1}{10(H_s+\tau)},\qquad I=\left\lceil\frac{100(H_s+\tau)}{\tau}\right\rceil. \tag*{(63)} τ = n 8 H s , β = 10 ( H s + τ ) 1 , I = ⌈ τ 100 ( H s + τ ) ⌉ . (63)
It returns the output p ~ \tilde{p} p ~ of the last epoch and g ~ = τ ( y − p ~ ) \tilde{g}=\tau(y-\tilde{p}) g ~ = τ ( y − p ~ ) . The quadratic part of the gradient is computed locally, so V k = ∇ f ξ i k ( w k ) − ∇ f ξ i k ( x ˉ ) + 1 n ∑ i ∇ f ξ i ( x ˉ ) + τ ( w k − y ) V_k=\nabla f_{\xi_{i_k}}(w_k)-\nabla f_{\xi_{i_k}}(\bar{x})+\frac{1}{n}\sum_i\nabla f_{\xi_i}(\bar{x})+\tau(w_k-y) V k = ∇ f ξ i k ( w k ) − ∇ f ξ i k ( x ˉ ) + n 1 ∑ i ∇ f ξ i ( x ˉ ) + τ ( w k − y ) , and an epoch makes n n n queries at the snapshot and I I I further queries, all with labels of the batch.
Lemma C.5 (Local routine). If
j ≥ ⌈ log 2 ( 625 n ( n + 8 ) ( T + 1 ) 4 ) ⌉ , j\geq\left\lceil\log_2\left(625n(n+8)(T+1)^4\right)\right\rceil, j ≥ ⌈ log 2 ( 625 n ( n + 8 ) ( T + 1 ) 4 ) ⌉ ,
then the local routine makes j ( n + I ) ≤ 114 j n j(n+I)\leq114jn j ( n + I ) ≤ 114 j n queries with n n n distinct labels, and
E [ h S , y ( p ~ ) − φ S ( y ) ∣ S , y ] ≤ δ : = τ 2 R 2 20000 ( H s + τ ) ( T + 1 ) 4 , (64) \mathbb{E}[h_{S,y}(\tilde{p})-\varphi_S(y)\mid S,y]\leq\delta:=\frac{\tau^2R^2}{20000(H_s+\tau)(T+1)^4}, \tag*{(64)} E [ h S , y ( p ~ ) − φ S ( y ) ∣ S , y ] ≤ δ := 20000 ( H s + τ ) ( T + 1 ) 4 τ 2 R 2 , (64)
E [ ∥ g ~ − ∇ φ S ( y ) ∥ ∣ y ] ≤ τ R 100 ( T + 1 ) 2 , 2 H s δ τ ≤ τ R 2 10 4 ( T + 1 ) 4 . (65) \mathbb{E}[\|\tilde{g}-\nabla\varphi_S(y)\|\mid y]\leq\frac{\tau R}{100(T+1)^2},\qquad\frac{2H_s\delta}{\tau}\leq\frac{\tau R^2}{10^4(T+1)^4}. \tag*{(65)} E [ ∥ g ~ − ∇ φ S ( y ) ∥ ∣ y ] ≤ 100 ( T + 1 ) 2 τ R , τ 2 H s δ ≤ 1 0 4 ( T + 1 ) 4 τ R 2 . (65)
Proof. Each h i h_i h i is convex with an ( H s + τ ) (H_s+\tau) ( H s + τ ) -Lipschitz gradient, and h h h is τ \tau τ -strongly convex. With (C.12), 2 ( H s + τ ) β = 1 5 2(H_s+\tau)\beta=\frac{1}{5} 2 ( H s + τ ) β = 5 1 , and the factor in (C.11) is at most 12.5 ( H s + τ ) τ I + 1 4 ≤ 1 8 + 1 4 < 1 2 \frac{12.5(H_s+\tau)}{\tau I}+\frac{1}{4}\leq\frac{1}{8}+\frac{1}{4}<\frac{1}{2} τ I 12.5 ( H s + τ ) + 4 1 ≤ 8 1 + 4 1 < 2 1 . By Lemma C.2, the gap at the first snapshot is at most 2 H s R 2 2H_sR^2 2 H s R 2 , so iterating (C.11) over the epochs gives the bound 2 − j ⋅ 2 H s R 2 2^{-j}\cdot2H_sR^2 2 − j ⋅ 2 H s R 2 in (C.13). Since H s / τ = n / 8 H_s/\tau=n/8 H s / τ = n /8 and ( H s + τ ) / τ = ( n + 8 ) / 8 (H_s+\tau)/\tau=(n+8)/8 ( H s + τ ) / τ = ( n + 8 ) /8 ,
2 H s R 2 δ = 40000 H s ( H s + τ ) ( T + 1 ) 4 τ 2 = 625 n ( n + 8 ) ( T + 1 ) 4 ≤ 2 j , \frac{2H_sR^2}{\delta}=\frac{40000H_s(H_s+\tau)(T+1)^4}{\tau^2}=625n(n+8)(T+1)^4\leq2^j, δ 2 H s R 2 = τ 2 40000 H s ( H s + τ ) ( T + 1 ) 4 = 625 n ( n + 8 ) ( T + 1 ) 4 ≤ 2 j ,
which proves (C.13). By (C.7), ∥ g ~ − ∇ φ S ( y ) ∥ = τ ∥ p ~ − p S ( y ) ∥ \|\tilde{g}-\nabla\varphi_S(y)\|=\tau\|\tilde{p}-p_S(y)\| ∥ g ~ − ∇ φ S ( y ) ∥ = τ ∥ p ~ − p S ( y ) ∥ , and strong convexity and Jensen’s inequality give the bound τ 2 δ / τ = 2 τ δ ≤ τ R 100 ( T + 1 ) 2 \tau\sqrt{2\delta/\tau}=\sqrt{2\tau\delta}\leq\frac{\tau R}{100(T+1)^2} τ 2 δ / τ = 2 τ δ ≤ 100 ( T + 1 ) 2 τ R , using τ ≤ H s + τ \tau\leq H_s+\tau τ ≤ H s + τ . The last inequality in (C.14) follows from H s ≤ H s + τ H_s\leq H_s+\tau H s ≤ H s + τ . Finally, n + I = n + 100 + ⌈ 12.5 n ⌉ ≤ 114 n n+I=n+100+\lceil12.5n\rceil\leq114n n + I = n + 100 + ⌈ 12.5 n ⌉ ≤ 114 n .
All maps of the routine are measurable, and its iterates have finite moments of all orders: by (B.2), the norms of the inner iterates obey a finite linear recursion with deterministic coefficients.
The distributed envelope method
Algorithm 3 combines the method of Lemma C.1, applied to the empirical envelopes, with the local routine. Only m ′ ≤ m m'\leq m m ′ ≤ m workers are active; this is used for small pools in Theorem C.8.
Algorithm 3 Accelerated envelope method with local SVRG
Require: Initial point $x_0$, parameters $R$, $H_s > 0$, integers $1 \le m' \le m$ and $n$, $j$, $N \ge 1$, and a pool of labels.
1: $\tau \leftarrow \frac{8H_s}{n}$, $\beta \leftarrow \frac{1}{10(H_s+\tau)}$, $I \leftarrow \left\lceil\frac{100(H_s+\tau)}{\tau}\right\rceil$, $T \leftarrow N-1$
2: $\gamma \leftarrow \min\left\{\frac{1}{12\tau},\frac{m'}{24\tau(T+1)}\right\}$, $z_0 \leftarrow x_0$, $v_0 \leftarrow x_0$
3: for $t=0,\ldots,T-1$ do ▷ round $t+1$
4: $\alpha_t \leftarrow \frac{2}{t+6}$, $\eta_t \leftarrow \gamma(t+1)$, $y_t \leftarrow (1-\alpha_t)z_t+\alpha_tv_t$, and broadcast $y_t$
5: Each worker $i\le m'$ takes a fresh batch $S_{i,t}$ of $n$ labels and sends $\widetilde{g}_{i,t}\leftarrow\tau(y_t-\operatorname{LOCALPROX}(y_t,S_{i,t}))$
6: $\widetilde{g}_t\leftarrow\frac{1}{m'}\sum_{i=1}^{m'}\widetilde{g}_{i,t}$
7: $v_{t+1}\leftarrow\Pi_C(v_t-\eta_t\widetilde{g}_t)$, $z_{t+1}\leftarrow(1-\alpha_t)z_t+\alpha_tv_{t+1}$
8: end for
9: In round $N$, one worker takes a fresh batch $S$ and return $\widehat{x}\leftarrow\operatorname{LOCALPROX}(z_T,S)$
10: procedure LOCALPROX($y$, batch with labels $i_1,\ldots,i_n$)
11: $\bar{x}\leftarrow y$
12: for $e=1,\ldots,j$ do ▷ one SVRG epoch
13: Query and store $\nabla f_{\xi_{i_l}}(\bar{x})$, $l=1,\ldots,n$, and set $\bar{g}\leftarrow\frac{1}{n}\sum_{l=1}^n\nabla f_{\xi_{i_l}}(\bar{x})$
14: $w_0\leftarrow\bar{x}$
15: for $k=0,\ldots,I-1$ do
16: Draw $l$ uniformly from $\{1,\ldots,n\}$ and query $\nabla f_{\xi_{i_l}}(w_k)$
17: $V_k\leftarrow\nabla f_{\xi_{i_l}}(w_k)-\nabla f_{\xi_{i_l}}(\bar{x})+\bar{g}+\tau(w_k-y)$
18: $w_{k+1}\leftarrow w_k-\beta V_k$
19: end for
20: $\bar{x}\leftarrow w_\kappa$ with $\kappa$ uniform on $\{0,\ldots,I-1\}$
21: end for
22: return $\bar{x}$
23: end procedure
A fresh batch consists of labels that have not been queried before. For N = 1 N=1 N = 1 the loop is empty and the output is computed at z 0 = x 0 z_0=x_0 z 0 = x 0 . Every call of LOCALPROX makes j ( n + I ) j(n+I) j ( n + I ) queries.
Proposition C.6 (Envelope method). Let 1 ≤ m ′ ≤ m 1\le m'\le m 1 ≤ m ′ ≤ m , n ≥ 1 n\ge1 n ≥ 1 , and j ≥ ⌈ log 2 ( 625 n ( n + 8 ) N 4 ) ⌉ j\ge\lceil\log_2(625n(n+8)N^4)\rceil j ≥ ⌈ log 2 ( 625 n ( n + 8 ) N 4 )⌉ satisfy 114 j n ≤ K 114jn\le K 114 j n ≤ K , and let τ = 8 H s / n \tau=8H_s/n τ = 8 H s / n . Then Algorithm 3 belongs to IC ( m , K , N ) \operatorname{IC}(m,K,N) IC ( m , K , N ) , uses at most m ′ N n m'Nn m ′ N n distinct labels, and satisfies
EF ( x ^ ) ≤ 150 τ R 2 ( N − 1 ) 2 + 1152 τ R 2 m ′ ( N − 1 ) if N ≥ 2 , EF ( x ^ ) ≤ 3 τ R 2 if N = 1. (66) \operatorname{EF}(\widehat{x})\le\frac{150\tau R^2}{(N-1)^2}+\frac{1152\tau R^2}{m'(N-1)}\quad\text{if }N\ge2,\qquad\operatorname{EF}(\widehat{x})\le3\tau R^2\quad\text{if }N=1. \tag*{(66)} EF ( x ) ≤ ( N − 1 ) 2 150 τ R 2 + m ′ ( N − 1 ) 1152 τ R 2 if N ≥ 2 , EF ( x ) ≤ 3 τ R 2 if N = 1. (66)
Proof. Let N ≥ 2 N\ge2 N ≥ 2 and T = N − 1 T=N-1 T = N − 1 . The loop of Algorithm 3 is the method of Lemma C.1 for the envelope samples q S = φ S q_S=\varphi_S q S = φ S and their mean Q = φ ˉ Q=\bar{\varphi} Q = φ ˉ , with L ′ = H ′ = τ L'=H'=\tau L ′ = H ′ = τ , b = m ′ b=m' b = m ′ , and T T T iterations; the hypotheses hold by Lemma C.2. In iteration t t t , every active worker runs the local routine on its own fresh batch at the common point y t ∈ C y_t\in C y t ∈ C with horizon T T T , and the average of the returned vectors is g ~ t \widetilde{g}_t g t . The average of the exact envelope gradients ∇ φ S ( y t ) \nabla\varphi_S(y_t) ∇ φ S ( y t ) over the m ′ m' m ′ batches is the average g t g_t g t of Lemma C.1, and by the triangle inequality and (C.14) the error e t = g ~ t − g t e_t=\widetilde{g}_t-g_t e t = g t − g t satisfies (C.1) with ε = τ R 100 ( T + 1 ) 2 \varepsilon=\frac{\tau R}{100(T+1)^2} ε = 100 ( T + 1 ) 2 τ R . Hence
E φ ˉ ( z T ) ≤ 36 τ R 2 T 2 + 288 τ R 2 m ′ T + 12 τ R 2 100 ( T + 1 ) 2 ≤ 37 τ R 2 T 2 + 288 τ R 2 m ′ T . \mathbb{E}\bar{\varphi}(z_T)\le\frac{36\tau R^2}{T^2}+\frac{288\tau R^2}{m'T}+\frac{12\tau R^2}{100(T+1)^2}\le\frac{37\tau R^2}{T^2}+\frac{288\tau R^2}{m'T}. E φ ˉ ( z T ) ≤ T 2 36 τ R 2 + m ′ T 288 τ R 2 + 100 ( T + 1 ) 2 12 τ R 2 ≤ T 2 37 τ R 2 + m ′ T 288 τ R 2 .
In round N N N , the batch is fresh and hence independent of z T z_T z T , and x ^ \widehat{x} x is the output p ~ \widetilde{p} p of the local routine at z T ∈ C z_T\in C z T ∈ C . Lemma C.3 and (C.14) give
EF ( x ^ ) ≤ 4 E φ ˉ ( z T ) + τ R 2 10 4 ( T + 1 ) 4 ≤ 150 τ R 2 T 2 + 1152 τ R 2 m ′ T . (67) \operatorname{EF}(\widehat{x})\le4\mathbb{E}\bar{\varphi}(z_T)+\frac{\tau R^2}{10^4(T+1)^4}\le\frac{150\tau R^2}{T^2}+\frac{1152\tau R^2}{m'T}. \tag*{(67)} EF ( x ) ≤ 4 E φ ˉ ( z T ) + 1 0 4 ( T + 1 ) 4 τ R 2 ≤ T 2 150 τ R 2 + m ′ T 1152 τ R 2 . (67)
@page 35/37 starts_mid=0 ends_mid=1
Algorithm 3 Accelerated envelope method with local SVRG
Require: Initial point $x_0$, parameters $R$, $H_s > 0$, integers $1 \le m' \le m$ and $n$, $j$, $N \ge 1$, and a pool of labels.
1: $\tau \leftarrow \frac{8H_s}{n}$, $\beta \leftarrow \frac{1}{10(H_s+\tau)}$, $I \leftarrow \left\lceil\frac{100(H_s+\tau)}{\tau}\right\rceil$, $T \leftarrow N-1$
2: $\gamma \leftarrow \min\left\{\frac{1}{12\tau},\frac{m'}{24\tau(T+1)}\right\}$, $z_0 \leftarrow x_0$, $v_0 \leftarrow x_0$
3: for $t=0,\ldots,T-1$ do ▷ round $t+1$
4: $\alpha_t \leftarrow \frac{2}{t+6}$, $\eta_t \leftarrow \gamma(t+1)$, $y_t \leftarrow (1-\alpha_t)z_t+\alpha_tv_t$, and broadcast $y_t$
5: Each worker $i\le m'$ takes a fresh batch $S_{i,t}$ of $n$ labels and sends $\widetilde{g}_{i,t}\leftarrow\tau(y_t-\operatorname{LOCALPROX}(y_t,S_{i,t}))$
6: $\widetilde{g}_t\leftarrow\frac{1}{m'}\sum_{i=1}^{m'}\widetilde{g}_{i,t}$
7: $v_{t+1}\leftarrow\Pi_C(v_t-\eta_t\widetilde{g}_t)$, $z_{t+1}\leftarrow(1-\alpha_t)z_t+\alpha_tv_{t+1}$
8: end for
9: In round $N$, one worker takes a fresh batch $S$ and return $\widehat{x}\leftarrow\operatorname{LOCALPROX}(z_T,S)$
10: procedure LOCALPROX($y$, batch with labels $i_1,\ldots,i_n$)
11: $\bar{x}\leftarrow y$
12: for $e=1,\ldots,j$ do ▷ one SVRG epoch
13: Query and store $\nabla f_{\xi_{i_l}}(\bar{x})$, $l=1,\ldots,n$, and set $\bar{g}\leftarrow\frac{1}{n}\sum_{l=1}^n\nabla f_{\xi_{i_l}}(\bar{x})$
14: $w_0\leftarrow\bar{x}$
15: for $k=0,\ldots,I-1$ do
16: Draw $l$ uniformly from $\{1,\ldots,n\}$ and query $\nabla f_{\xi_{i_l}}(w_k)$
17: $V_k\leftarrow\nabla f_{\xi_{i_l}}(w_k)-\nabla f_{\xi_{i_l}}(\bar{x})+\bar{g}+\tau(w_k-y)$
18: $w_{k+1}\leftarrow w_k-\beta V_k$
19: end for
20: $\bar{x}\leftarrow w_\kappa$ with $\kappa$ uniform on $\{0,\ldots,I-1\}$
21: end for
22: return $\bar{x}$
23: end procedure
A fresh batch consists of labels that have not been queried before. For N = 1 N=1 N = 1 the loop is empty and the output is computed at z 0 = x 0 z_0=x_0 z 0 = x 0 . Every call of LOCALPROX makes j ( n + I ) j(n+I) j ( n + I ) queries.
Proposition C.6 (Envelope method). Let 1 ≤ m ′ ≤ m 1\le m'\le m 1 ≤ m ′ ≤ m , n ≥ 1 n\ge1 n ≥ 1 , and j ≥ ⌈ log 2 ( 625 n ( n + 8 ) N 4 ) ⌉ j\ge\lceil\log_2(625n(n+8)N^4)\rceil j ≥ ⌈ log 2 ( 625 n ( n + 8 ) N 4 )⌉ satisfy 114 j n ≤ K 114jn\le K 114 j n ≤ K , and let τ = 8 H s / n \tau=8H_s/n τ = 8 H s / n . Then Algorithm 3 belongs to IC ( m , K , N ) \operatorname{IC}(m,K,N) IC ( m , K , N ) , uses at most m ′ N n m'Nn m ′ N n distinct labels, and satisfies
EF ( x ^ ) ≤ 150 τ R 2 ( N − 1 ) 2 + 1152 τ R 2 m ′ ( N − 1 ) if N ≥ 2 , EF ( x ^ ) ≤ 3 τ R 2 if N = 1. (68) \operatorname{EF}(\widehat{x})\le\frac{150\tau R^2}{(N-1)^2}+\frac{1152\tau R^2}{m'(N-1)}\quad\text{if }N\ge2,\qquad\operatorname{EF}(\widehat{x})\le3\tau R^2\quad\text{if }N=1. \tag*{(68)} EF ( x ) ≤ ( N − 1 ) 2 150 τ R 2 + m ′ ( N − 1 ) 1152 τ R 2 if N ≥ 2 , EF ( x ) ≤ 3 τ R 2 if N = 1. (68)
Proof. Let N ≥ 2 N\ge2 N ≥ 2 and T = N − 1 T=N-1 T = N − 1 . The loop of Algorithm 3 is the method of Lemma C.1 for the envelope samples q S = φ S q_S=\varphi_S q S = φ S and their mean Q = φ ˉ Q=\bar{\varphi} Q = φ ˉ , with L ′ = H ′ = τ L'=H'=\tau L ′ = H ′ = τ , b = m ′ b=m' b = m ′ , and T T T iterations; the hypotheses hold by Lemma C.2. In iteration t t t , every active worker runs the local routine on its own fresh batch at the common point y t ∈ C y_t\in C y t ∈ C with horizon T T T , and the average of the returned vectors is g ~ t \widetilde{g}_t g t . The average of the exact envelope gradients ∇ φ S ( y t ) \nabla\varphi_S(y_t) ∇ φ S ( y t ) over the m ′ m' m ′ batches is the average g t g_t g t of Lemma C.1, and by the triangle inequality and (C.14) the error e t = g ~ t − g t e_t=\widetilde{g}_t-g_t e t = g t − g t satisfies (C.1) with ε = τ R 100 ( T + 1 ) 2 \varepsilon=\frac{\tau R}{100(T+1)^2} ε = 100 ( T + 1 ) 2 τ R . Hence
E φ ˉ ( z T ) ≤ 36 τ R 2 T 2 + 288 τ R 2 m ′ T + 12 τ R 2 100 ( T + 1 ) 2 ≤ 37 τ R 2 T 2 + 288 τ R 2 m ′ T . \mathbb{E}\bar{\varphi}(z_T)\le\frac{36\tau R^2}{T^2}+\frac{288\tau R^2}{m'T}+\frac{12\tau R^2}{100(T+1)^2}\le\frac{37\tau R^2}{T^2}+\frac{288\tau R^2}{m'T}. E φ ˉ ( z T ) ≤ T 2 36 τ R 2 + m ′ T 288 τ R 2 + 100 ( T + 1 ) 2 12 τ R 2 ≤ T 2 37 τ R 2 + m ′ T 288 τ R 2 .
In round N N N , the batch is fresh and hence independent of z T z_T z T , and x ^ \widehat{x} x is the output p ~ \widetilde{p} p of the local routine at z T ∈ C z_T\in C z T ∈ C . Lemma C.3 and (C.14) give
EF ( x ^ ) ≤ 4 E φ ˉ ( z T ) + τ R 2 10 4 ( T + 1 ) 4 ≤ 150 τ R 2 T 2 + 1152 τ R 2 m ′ T . (69) \operatorname{EF}(\widehat{x})\le4\mathbb{E}\bar{\varphi}(z_T)+\frac{\tau R^2}{10^4(T+1)^4}\le\frac{150\tau R^2}{T^2}+\frac{1152\tau R^2}{m'T}. \tag*{(69)} EF ( x ) ≤ 4 E φ ˉ ( z T ) + 1 0 4 ( T + 1 ) 4 τ R 2 ≤ T 2 150 τ R 2 + m ′ T 1152 τ R 2 . (69)
For N = 1 N=1 N = 1 , the output is computed by the local routine at x 0 x_0 x 0 with horizon T = 0 T=0 T = 0 ; since φ ˉ ( x 0 ) ≤ 1 2 R 2 \bar{\varphi}(x_0) \le\frac{1}{2}R^2 φ ˉ ( x 0 ) ≤ 2 1 R 2 , Lemma C.3 gives E F ( x ^ ) ≤ 2 τ R 2 + τ R 2 10 4 ≤ 3 τ R 2 EF(\hat{x}) \le2\tau R^2 + \frac{\tau R^2}{10^4} \le3\tau R^2 E F ( x ^ ) ≤ 2 τ R 2 + 1 0 4 τ R 2 ≤ 3 τ R 2 .
In every round each active worker makes j ( n + 1 ) ≤ 114 j n ≤ K j(n+1) \le114jn \le K j ( n + 1 ) ≤ 114 j n ≤ K queries, the condition on j j j in Lemma C.5 holds because T + 1 ≤ N T+1 \le N T + 1 ≤ N , and no communication within a round is needed. The number of distinct labels is at most m ′ ( N − 1 ) + 1 ) n ≤ m ′ N m'(N-1)+1)n \le m'N m ′ ( N − 1 ) + 1 ) n ≤ m ′ N .
Proposition C.7 (Integer parameters). Let J J J be as in Theorem 5.1 and 1 ≤ m ′ ≤ m 1 \le m' \le m 1 ≤ m ′ ≤ m . There is an algorithm in IC ( m , K , N ) \operatorname{IC}(m,K,N) IC ( m , K , N ) that uses at most m ′ K N m'KN m ′ K N distinct labels and satisfies
E F ( x ^ ) ≤ 5 ⋅ 10 6 J H s R 2 ( 1 K N 2 + 1 m ′ K N ) . (70) EF(\hat{x}) \le5 \cdot10^6 JH_sR^2 \left(\frac{1}{KN^2}+\frac{1}{m'KN}\right). \tag*{(70)} E F ( x ^ ) ≤ 5 ⋅ 1 0 6 J H s R 2 ( K N 2 1 + m ′ K N 1 ) . (70)
Proof. If K ≥ 114 J K \ge114J K ≥ 114 J , apply Proposition C.6 with n = ⌊ K / ( 114 J ) ⌋ n=\lfloor K/(114J)\rfloor n = ⌊ K / ( 114 J )⌋ and j = J j=J j = J . Then 1 ≤ n ≤ K 1 \le n \le K 1 ≤ n ≤ K , n ≥ K 228 J n \ge\frac{K}{228J} n ≥ 228 J K , 114 J n ≤ K 114Jn \le K 114 J n ≤ K , and J ≥ ⌈ log 2 ( 625 n ( n + 8 ) N 4 ) ⌉ J \ge\lceil\log_2(625n(n+8)N^4)\rceil J ≥ ⌈ log 2 ( 625 n ( n + 8 ) N 4 )⌉ , so the proposition applies, the number of labels is at most m ′ N n ≤ m ′ K N m'Nn \le m'KN m ′ N n ≤ m ′ K N , and τ ≤ 1824 J H s K \tau\le\frac{1824JH_s}{K} τ ≤ K 1824 J H s . For N ≥ 2 N \ge2 N ≥ 2 we have N − 1 ≥ N / 2 N-1 \ge N/2 N − 1 ≥ N /2 , and (C.15) gives
E F ( x ^ ) ≤ 600 τ R 2 N 2 + 2304 τ R 2 m ′ N ≤ 1 094 400 J H s R 2 K N 2 + 4 202 496 J H s R 2 m ′ K N . EF(\hat{x}) \le\frac{600\tau R^2}{N^2}+\frac{2304\tau R^2}{m'N} \le\frac{1\,094\,400JH_sR^2}{KN^2}+\frac{4\,202\,496JH_sR^2}{m'KN}. E F ( x ^ ) ≤ N 2 600 τ R 2 + m ′ N 2304 τ R 2 ≤ K N 2 1 094 400 J H s R 2 + m ′ K N 4 202 496 J H s R 2 .
For N = 1 N=1 N = 1 , (C.15) gives E F ( x ^ ) ≤ 3 τ R 2 ≤ 5472 J H s R 2 K EF(\hat{x}) \le3\tau R^2 \le\frac{5472JH_sR^2}{K} E F ( x ^ ) ≤ 3 τ R 2 ≤ K 5472 J H s R 2 .
If K < 114 J K<114J K < 114 J , use (C.6) with b = m ′ K b=m'K b = m ′ K and T = N T=N T = N , that is, m ′ m' m ′ workers query a fresh label at every query. It uses m ′ K N m'KN m ′ K N labels, and, since L ≤ H s L \le H_s L ≤ H s and 1 < 114 J K 1<\frac{114J}{K} 1 < K 114 J ,
E F ( x ^ ) ≤ 36 H s R 2 N 2 + 288 H s R 2 m ′ K N ≤ 4104 J H s R 2 K N 2 + 288 H s R 2 m ′ K N . EF(\hat{x}) \le\frac{36H_sR^2}{N^2}+\frac{288H_sR^2}{m'KN} \le\frac{4104JH_sR^2}{KN^2}+\frac{288H_sR^2}{m'KN}. E F ( x ^ ) ≤ N 2 36 H s R 2 + m ′ K N 288 H s R 2 ≤ K N 2 4104 J H s R 2 + m ′ K N 288 H s R 2 .
In all cases (C.16) holds.
The full bound
Theorem C.8 (Upper bound with sample reuse). Let S ∈ N ∪ { ∞ } S \in\mathbb{N} \cup\{\infty\} S ∈ N ∪ { ∞ } , S ≥ 1 S \ge1 S ≥ 1 . There is an algorithm in IC ( m , K , N ) \operatorname{IC}(m,K,N) IC ( m , K , N ) with access to a pool of size S S S such that, for every instance of S d S_d S d ,
E F ( x ^ ) ≤ 10 7 J R 2 min { L , min { L , H s / K } N 2 + H s min { S , m K N } } . EF(\hat{x}) \le10^7JR^2\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{\min\{S,mKN\}}\right\}. E F ( x ^ ) ≤ 1 0 7 J R 2 min { L , N 2 min { L , H s / K } + min { S , m K N } H s } .
Proof. Put n S = min { S , m K N } n_S=\min\{S,mKN\} n S = min { S , m K N } . In each branch below the algorithm assigns disjoint sets of previously unqueried labels in { 1 , … , S } \{1,\ldots,S\} { 1 , … , S } to the batches and queries, so that every batch is fresh; the guarantees of the branches depend only on the public parameters, and the algorithm selects the branch with the smallest guarantee before the run. Returning x 0 x_0 x 0 gives F ( x 0 ) ≤ 1 2 R 2 F(x_0) \le\frac{1}{2}R^2 F ( x 0 ) ≤ 2 1 R 2 in all cases.
Case S ≤ K N S \le KN S ≤ K N . Here n S = S n_S=S n S = S . One worker runs serial SGD (Lemma B.3) with s = S s=S s = S queries on distinct labels, at most K K K per round, which gives E F ( x ^ ) ≤ 2 H s R 2 S EF(\hat{x}) \le\frac{2H_sR^2}{S} E F ( x ^ ) ≤ S 2 H s R 2 . Together with x 0 x_0 x 0 , the guarantee is R 2 min { L 2 , 2 H s S } ≤ 2 R 2 min { L , H s n S } R^2\min\{\frac{L}{2},\frac{2H_s}{S}\} \le2R^2\min\{L,\frac{H_s}{n_S}\} R 2 min { 2 L , S 2 H s } ≤ 2 R 2 min { L , n S H s } , which is at most the bound of the theorem.
Case S > K N S>KN S > K N . Let m ′ = min { m , ⌊ S / ( K N ) ⌋ } m'=\min\{m,\lfloor S/(KN)\rfloor\} m ′ = min { m , ⌊ S / ( K N )⌋} , with m ′ = m m'=m m ′ = m if S = ∞ S=\infty S = ∞ . Then 1 ≤ m ′ ≤ m 1 \le m' \le m 1 ≤ m ′ ≤ m , m ′ K N ≤ S m'KN \le S m ′ K N ≤ S , and m ′ K N ≥ n S / 2 m'KN \ge n_S/2 m ′ K N ≥ n S /2 : if m ′ = m m'=m m ′ = m , then m ′ K N = m K N ≥ n S m'KN=mKN \ge n_S m ′ K N = m K N ≥ n S , and otherwise ⌊ S / ( K N ) ⌋ ≥ S 2 K N \lfloor S/(KN)\rfloor\ge\frac{S}{2KN} ⌊ S / ( K N )⌋ ≥ 2 K N S . Proposition C.7 gives
E F ( x ^ ) ≤ 5 ⋅ 10 6 J H s R 2 ( 1 K N 2 + 2 n S ) ≤ 10 7 J R 2 ( H s K N 2 + H s n S ) . EF(\hat{x}) \le5\cdot10^6JH_sR^2\left(\frac{1}{KN^2}+\frac{2}{n_S}\right) \le10^7JR^2\left(\frac{H_s}{KN^2}+\frac{H_s}{n_S}\right). E F ( x ^ ) ≤ 5 ⋅ 1 0 6 J H s R 2 ( K N 2 1 + n S 2 ) ≤ 1 0 7 J R 2 ( K N 2 H s + n S H s ) .
Next, let b = min { m K , ⌊ S / N ⌋ } b=\min\{mK,\lfloor S/N\rfloor\} b = min { m K , ⌊ S / N ⌋} , with b = m K b=mK b = m K if S = ∞ S=\infty S = ∞ . Then b ≥ 1 b \ge1 b ≥ 1 and b N ≤ S bN \le S b N ≤ S , and b N ≥ n S / 2 bN \ge n_S/2 b N ≥ n S /2 by the same argument. The method (C.6) with T = N T=N T = N and b b b queries per round at a common point, distributed among the workers with at most K K K each, gives
E F ( x ^ ) ≤ R 2 ( 36 L N 2 + 576 H s n S ) . EF(\hat{x}) \le R^2\left(\frac{36L}{N^2}+\frac{576H_s}{n_S}\right). E F ( x ^ ) ≤ R 2 ( N 2 36 L + n S 576 H s ) .
The smallest of the three guarantees is at most
10 7 J R 2 min { L , min { L , H s / K } N 2 + H s n S } , 10^7JR^2\min\left\{L,\frac{\min\{L,H_s/K\}}{N^2}+\frac{H_s}{n_S}\right\}, 1 0 7 J R 2 min { L , N 2 min { L , H s / K } + n S H s } ,
since J ≥ 1 J \ge1 J ≥ 1 . This proves the theorem in both cases.
Proofs of the main results
Proof of Theorem 3.2 . The upper bound is Corollary B.9 , and the lower bound is Theorem A.15 . □ \square □
Proof of Theorem 3.3 . The upper bound is Corollary B.7, whose proof uses only (1) and the quadratic form of F F F . By Lemma 4.1 , M d w g , q u a d ≥ M d s g , q u a d \mathcal{M}^{\mathrm{wg},\mathrm{quad}}_d \ge\mathcal{M}^{\mathrm{sg},\mathrm{quad}}_d M d wg , quad ≥ M d sg , quad , and Theorem A.11 gives the lower bound for d ≥ 2048 m ( K N ) 2 d \ge2048m(KN)^2 d ≥ 2048 m ( K N ) 2 . □ \square □
Proof of Theorem 4.2 . By Lemma 4.1 , every instance of S d q u a d \mathcal{S}^{\mathrm{quad}}_d S d quad is an instance of W d q u a d \mathcal{W}^{\mathrm{quad}}_d W d quad , so Corollary B.7 gives the upper bound in every dimension. The lower bound is Theorem A.11 . □ \square □
Proof of Theorem 4.3 . By Lemma 4.1 , Corollary B.9 applies to every instance of S d \mathcal{S}_d S d and gives the upper bound in every dimension. Since S d q u a d ⊆ S d \mathcal{S}^{\mathrm{quad}}_d \subseteq\mathcal{S}_d S d quad ⊆ S d , we have M d s g ≥ M d s g , q u a d \mathcal{M}^{\mathrm{sg}}_d \ge\mathcal{M}^{\mathrm{sg},\mathrm{quad}}_d M d sg ≥ M d sg , quad , and Theorem A.11 gives the lower bound for d ≥ 2048 m ( K N ) 2 d \ge2048m(KN)^2 d ≥ 2048 m ( K N ) 2 . □ \square □
The lower bound of Theorem 4.3 is also implied, with the constant 10 6 10^6 1 0 6 and for d ≥ d p o o l d \ge d_{\mathrm{pool}} d ≥ d pool , by Theorem A.20 with S = ∞ S = \infty S = ∞ and (3) . That construction has a non-quadratic population and remains hard under arbitrary reuse of samples.
Proof of Theorem 5.1 . The upper bound is Theorem C.8, and the lower bound is Theorem A.20 . □ \square □