Introduction
Stochastic methods are an essential part of modern optimization, with applications spanning machine learning, statistics, and theoretical computer science. The foundational work of Nemirovski and Yudin [38 ] introduced the oracle model for convex optimization, whereby the goal is to understand, for a particular oracle class, how many queries are needed to minimize a function to sufficient accuracy. Often considered in the case of stochastic convex optimization (SCO) is the stochastic first-order oracle, which for a convex function f : R d → R f : \mathbb{R}^{d} \to\mathbb{R} f : R d → R queried at a point x x x , provides an unbiased estimate of the subgradient with bounded q q q -th moment (q ∈ ( 1 , 2 ] q \in(1,2] q ∈ ( 1 , 2 ] ), that is, ∇ ^ f ( x ) \widehat{\nabla} f(x) ∇ f ( x ) such that E [ ∇ ^ f ( x ) ] ∈ ∂ f ( x ) \mathbb{E}[\widehat{\nabla} f(x)] \in\partial f(x) E [ ∇ f ( x )] ∈ ∂ f ( x ) and E [ ∥ ∇ ^ f ( x ) ∥ q q ] ≤ σ q \mathbb{E}[\lVert\widehat{\nabla} f(x)\rVert_{q}^{q}] \le\sigma^{q} E [∥ ∇ f ( x ) ∥ q q ] ≤ σ q for σ > 0 \sigma> 0 σ > 0 .
An additional line of work, initiated by Nemirovski [36 ] , has further considered the setting of parallel convex optimization, in which we may query an oracle multiple times in parallel for each round of interaction, and where we denote the query depth as the total number of such rounds. While several works have studied first-order methods for parallel settings in the Euclidean (ℓ 2 \ell_{2} ℓ 2 -norm) case [BJL${}^{+}$19, CJJ${}^{+}$23, DBW12], meaning f f f is Lipschitz continuous with respect to the ℓ 2 \ell_{2} ℓ 2 -norm over an ℓ 2 \ell_{2} ℓ 2 -bounded feasible set, parallel SCO has been less explored in non-Euclidean settings that deal with ℓ p \ell_{p} ℓ p geometries for 2 ≤ p < ∞ 2 \le p < \infty 2 ≤ p < ∞ . For the p = ∞ p = \infty p = ∞ case, Nemirovski [36 ] established Ω ( ( d ln ( 2 K d ) ) 1 / 3 ln ( 1 / ε ) ) \Omega\left(\left(\frac{d}{\ln(2Kd)}\right)^{1/3}\ln(1/\varepsilon)\right) Ω ( ( l n ( 2 K d ) d ) 1/3 ln ( 1/ ε ) ) lower bounds on the number of rounds required to achieve an ε \varepsilon ε -accurate solution, where K K K denotes the number of parallel queries in each round, while Diakonikolas and Guzmán [18 ] provided lower bounds of the form Ω ( min { ε − p , ε 2 / 3 d 1 / 3 } ) \Omega(\min\{\varepsilon^{-p},\varepsilon^{2/3}d^{1/3}\}) Ω ( min { ε − p , ε 2/3 d 1/3 }) for p ≥ 2 p \ge2 p ≥ 2 .{^1} Meanwhile, work by Chakrabarty, Graur, Jiang, and Sidford [12 ] presented algorithms that achieve O ( d 1 / 3 ε − 2 / 3 ) O(d^{1/3}\varepsilon^{-2/3}) O ( d 1/3 ε − 2/3 ) parallel stochastic query depth for ℓ ∞ \ell_{\infty} ℓ ∞ -Lipschitz convex problems, based on leveraging additional Gaussian smoothing for this setting, thereby matching the lower bounds for constant accuracy, though their results rely on stronger bounded second-moment (i.e., variance) assumptions that are standard when analyzing stochastic first-order methods. Moreover, ℓ p \ell_{p} ℓ p -based optimization methods have been shown to be broadly useful across a variety of contexts including training deep neural networks [LBL${}^{+}$25, MFS26, Pet26], p p p -norm diffusion and flow problems [5 , 21 , 29 , 32 ] , and stochastic optimization under heavy-tailed noise assumptions [20 SBZ${}^{+}$26, VYB${}^{+}$22] .
In this work, we study parallel stochastic ℓ p \ell_{p} ℓ p -Lipschitz convex optimization under bounded q q q -th moment assumptions (for q = p p − 1 q = \frac{p}{p-1} q = p − 1 p ), for which we provide new query depth complexities that improve upon previous sequential results for certain ranges of solution accuracy. Underlying our results are two key technical contributions:
First, we introduce a class of ℓ p \ell_{p} ℓ p -optimal-distance-convergence (ODCp _{p} p ) algorithms, which we then show may be used to efficiently estimate the optimum of uniformly convex objectives. Although initially inspired by ideas on stochastic bias-reduced gradient methods [ACJ${}^{+}$21], we show how extending this type of approach to generalized ℓ p \ell_{p} ℓ p -based Moreau envelopes leads to several technical complications that need to be addressed, including the fact that the gradient function for the generalized Moreau envelope is a nonlinear mapping.
Second, we develop and analyze a new ℓ p \ell_{p} ℓ p -generalized Gaussian reweighting scheme for gradient estimation, which broadly extends previous approaches based on standard Gaussian convolution. As a result of these tools, we are able to handle several basic challenges that arise when attempting to estimate the gradients of these ℓ p \ell_{p} ℓ p -based Moreau envelopes that are central to our techniques.
By combining these techniques with recent algorithms for ℓ p \ell_{p} ℓ p ball-oracle acceleration [ABJS26, [13 ] , we arrive at our improved stochastic query depth complexities for parallel ℓ p \ell_p ℓ p -Lipschitz SCO. In addition, we establish new dimension-dependent lower bounds that improve upon the previous results of [18 ] , under certain ℓ p \ell_p ℓ p domain restrictions.
ℓ p \ell_p ℓ p Stochastic Bias-Reduced Gradient Methods
We begin by discussing the setting of bias-reduced gradient methods. When considering the problem of minimizing a function f : X → R f : \mathcal{X} \to\mathbb{R} f : X → R given access to a stochastic subgradient estimator ∇ ^ f \widehat{\nabla} f ∇ f that is unbiased and has bounded variance, a natural question is to ask whether this estimator can be transformed into a nearly unbiased estimator of the true minimizer x ∗ = argmin x ∈ X f ( x ) x_* = \operatorname*{argmin}_{x \in\mathcal{X}} f(x) x ∗ = argmin x ∈ X f ( x ) . For strongly convex f f f , [2 ] showed how to do so by proposing an estimator x ^ ∗ \widehat{x}_* x ∗ such that for any δ > 0 \delta> 0 δ > 0 , it holds that
∥ E x ^ ∗ − x ∗ ∥ 2 ≤ δ and E ∥ x ^ ∗ − E x ^ ∗ ∥ 2 2 = O ( G 2 μ 2 log ( G μ δ ) ) . \left\lVert\mathbb{E}\widehat{x}_* - x_* \right\rVert_2 \le\delta\quad\text{and} \quad\mathbb{E}\left\lVert\widehat{x}_* - \mathbb{E}\widehat{x}_* \right\rVert_2^2 = O\left(\frac{G^2}{\mu^2}\log\left(\frac{G}{\mu\delta}\right)\right). ∥ E x ∗ − x ∗ ∥ 2 ≤ δ and E ∥ x ∗ − E x ∗ ∥ 2 2 = O ( μ 2 G 2 log ( μ δ G ) ) .
where E ∥ ∇ ^ f ( x ) ∥ 2 2 ≤ G 2 \mathbb{E}\lVert\widehat{\nabla} f(x)\rVert_2^2 \le G^2 E ∥ ∇ f ( x ) ∥ 2 2 ≤ G 2 . Given the importance of these estimators and the extensive number of applications that these are used for such as estimating proximal points [39 , 35 ] , minimizing the maximal loss [16 ] , and gradient-efficient composite optimization [30 ] and the prevalence of uniformly convex optimization contexts such as ℓ p \ell_p ℓ p regression [11 , 1 , 4 , 26 ] and online learning [33 ] , it is natural to ask whether such estimators can be developed for the ℓ p \ell_p ℓ p setting.
In this work, we first answer this in the affirmative, and provide a class of ℓ p \ell_p ℓ p -optimal-distance-convergence (ODCp _p p ) algorithms that, when combined with multilevel Monte Carlo methods [22 , 7 ] , efficiently estimate the optimum of a uniformly convex function with low bias and bounded q q q -th moment in the dual ℓ q \ell_q ℓ q norm, where q = p p − 1 q = \frac{p}{p-1} q = p − 1 p . Further, we use these estimators to yield improved rates for projection-efficient ℓ p \ell_p ℓ p -Lipschitz optimization, derive complementary rates to [2 ] for composite minimization, and develop efficient methods for estimating proximal points, where the proximal point of y y y is P f , λ ( y ) = argmin x ∈ R d { f ( x ) + λ p ∥ x − y ∥ p p } P_{f,\lambda}(y) = \operatorname*{argmin}_{x \in\mathbb{R}^d}\left\{f(x) + \frac{\lambda}{p}\lVert x-y\rVert_p^p\right\} P f , λ ( y ) = argmin x ∈ R d { f ( x ) + p λ ∥ x − y ∥ p p } .
In Section 3, we describe our ODCp _p p algorithms, which are part of a larger set of ODC algorithms that take as input an unbiased subgradient estimator which satisfies a given moment bound, and provides a (hopefully) efficient convergence guarantee for convergence to x ∗ x_* x ∗ . In the Euclidean setting, [3 ] defines an ODC algorithm to minimize a strongly convex function F = f + ψ F = f + \psi F = f + ψ , where f f f is convex and ψ \psi ψ is simple such that the algorithm’s output x = ODC ( ∇ ^ f , ψ , T ) x = \operatorname{ODC}(\widehat{\nabla} f,\psi,T) x = ODC ( ∇ f , ψ , T ) satisfies E ∥ x − x ∗ ∥ 2 2 ≤ c G 2 T \mathbb{E}\lVert x-x_*\rVert_2^2 \le c\frac{G^2}{T} E ∥ x − x ∗ ∥ 2 2 ≤ c T G 2 for some constant c c c . Moreover, such algorithms are widely found in the literature [24 , 41 , 3 ] . Given the advantages that these algorithms have provided, we develop analogous ODCp _p p algorithms for p ≥ 2 p \ge2 p ≥ 2 , for minimizing a μ \mu μ -uniformly convex function F = f + ψ F = f + \psi F = f + ψ , where f f f is convex. As in [2 ] , we leverage the multilevel Monte Carlo (MLMC) technique of [7 ] to construct a low-bias, bounded-dual-moment estimator of the ℓ p \ell_p ℓ p -Moreau gradient. However, unlike in the Euclidean-case, when considering p > 2 p > 2 p > 2 , the map from the proximal point to the Moreau gradient is nonlinear. Therefore, we depart from previous analyses, by applying the MLMC construction directly to approximate the Moreau gradient, rather than first constructing a low-bias estimator of the proximal point and then using this to compute the Moreau gradient estimation. We prove that for any λ \lambda λ -uniformly convex function F = f + ψ F = f + \psi F = f + ψ where f f f is convex and iteration budget T max T_{\max} T m a x , our ODCp _p p algorithms provide good estimators g ′ g' g ′ of the true gradient g ∗ g_* g ∗ in a reasonably fast manner (see Lemma 3 for more details).
EpochDA (§ 3.1 ). Next, we provide a concrete ODCp _p p algorithm, EpochDA (Algorithm 3), which extends Algorithm 3 of [37 ] to a composite setting using an ℓ p \ell_p ℓ p -based uniformly convex prox function, and will be useful in achieving efficient ℓ p \ell_p ℓ p parallel rates (see § 4 for more details). Within each epoch, it applies a dual-averaging solver to the regularized objective, incorporating stochastic gradient estimates into successive updates while controlling the iterates through the global prox function centered at x ˉ \bar{x} x ˉ . It uses geometrically increasing epoch lengths and decreasing step sizes, averaging the iterates at the end of each epoch to initialize the next.
Projection-Efficient ℓ p \ell_p ℓ p -Lipschitz Optimization (§ 3.2 ). In stochastic nonsmooth optimization, constrained proximal or projection computations may be substantially more expensive than stochastic subgradient evaluations [2 ] . In § 3.2 , we study this tradeoff in the ℓ p \ell_p ℓ p setting for p > 2 p > 2 p > 2 , where f f f is convex and accessed through an unbiased stochastic subgradient oracle ∇ ^ f \widehat{\nabla} f ∇ f . In the Euclidean setting, [2 ] combine Moreau envelope smoothing with accelerated gradient descent to obtain O ( G R / ϵ ) O(GR/\epsilon) O ( GR / ϵ ) expensive projections while retaining near-optimal stochastic subgradient complexity. For p > 2 p > 2 p > 2 , we combine our ℓ p \ell_p ℓ p -Moreau envelope gradient estimator with accelerated updates using p p p -power regularization, inspired by [1 ] . Our analysis uses the smoothness of the envelope with respect to ℓ p \ell_p ℓ p , while all constrained updates use p p p -power regularization. The resulting algorithm requires O p ( ( G R / ϵ ) 2 p / ( p + 2 ) ) O_p((GR/\epsilon)^{2p/(p+2)}) O p (( GR / ϵ ) 2 p / ( p + 2 ) ) expensive constrained updates and O ~ p ( ( G R / ϵ ) p ( p − 1 ) ) \widetilde{O}_p((GR/\epsilon)^{p(p-1)}) O p (( GR / ϵ ) p ( p − 1 ) ) stochastic subgradient evaluations. Thus, compared with direct stochastic mirror descent, our method reduces the number of expensive constrained updates at the cost of a larger stochastic subgradient bound.
Convex Composite Minimization (§ 3.4 ). We consider the convex composite optimization problem
min x ∈ X Ψ ( x ) : = Λ ( x ) + f ( x ) , \min_{x\in\mathcal{X}} \Psi(x) := \Lambda(x) + f(x), x ∈ X min Ψ ( x ) := Λ ( x ) + f ( x ) ,
where Λ \Lambda Λ is ( L , κ ) (L,\kappa) ( L , κ ) -weakly smooth with respect to ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p , f f f satisfies Assumption 1, and there exists a minimizer x ∗ x^* x ∗ such that ∥ x 0 − x ∗ ∥ p ≤ R \|x_0-x^*\|_p \le R ∥ x 0 − x ∗ ∥ p ≤ R . A closely related setting was studied in [19 ] , who considered composite minimization with a weakly smooth component and a uniformly convex component. Their generalized AGD+ framework yields an iteration complexity of O ( ( L R κ / ϵ ) p / ( κ p − p + κ ) ) O((LR^\kappa/\epsilon)^{p/(\kappa p-p+\kappa)}) O (( L R κ / ϵ ) p / ( κ p − p + κ ) ) when specialized to ℓ p \ell_p ℓ p geometry without exploiting uniform convexity. Building on the stochastic composite AGD framework of [2 ] , we develop an ℓ p \ell_p ℓ p extension using Bregman proximal steps and our reduced-bias gradient estimators. Our algorithm (Algorithm 5) achieves the same dependence on ϵ \epsilon ϵ in the number of ∇ Λ \nabla\Lambda ∇Λ evaluations, up to logarithmic factors, while allowing f f f to be nonsmooth and accessed through a stochastic subgradient oracle.
ℓ p \ell_p ℓ p Accelerated Proximal Point Methods (Section 3.5). Here, we develop Monteiro-Svaiter-type ℓ p \ell_p ℓ p accelerated proximal point methods which leverage our reduced-bias Moreau envelope gradient estimators. Inspired by [2 ] , we present a stochastic accelerated ℓ p \ell_p ℓ p proximal point method (Algorithm 6), that computes adaptive proximal regularization with an accelerated first-order update. At each iteration, the algorithm selects a regularization parameter ( λ k + 1 ) (\lambda_{k+1}) ( λ k + 1 ) , from a designed NextLambda algorithm (Algorithm 7), and computes an acceleration weight ( a k + 1 ) (a_{k+1}) ( a k + 1 ) , which determines the cumulative weight ( A k + 1 ) (A_{k+1}) ( A k + 1 ) and the interpolation point ( y k ) (y_k) ( y k ) between the current iterate and the auxiliary sequence. We obtain the next iterate ( x k + 1 ) (x_{k+1}) ( x k + 1 ) by approximately solving a regularized problem centered at y k y_k y k , with the algorithm terminating when the cumulative accelerated weight reaches a prescribed threshold or the iteration budget is exhausted.
Parallel Optimization in p p p Geometries
Parallel Upper Bounds. In § 4 , we consider the following problem, simplified for the purposes of the introduction. We assume that there is a convex function f : R d → R f:\mathbb{R}^{d}\to\mathbb{R} f : R d → R whose minimizer lies in an ℓ p \ell_p ℓ p ball of radius R R R , which can be queried through a stochastic gradient oracle g g g , where E g ∈ ∂ f \mathbb{E}g\in\partial f E g ∈ ∂ f and E ∥ g ∥ q q ≤ L q \mathbb{E}\lVert g\rVert_q^q\leq L^q E ∥ g ∥ q q ≤ L q , for some L > 0 L>0 L > 0 . Moreover, throughout the text, unless otherwise specified, q = p p − 1 q=\frac{p}{p-1} q = p − 1 p . Our goal is to minimize f f f up to an expected additive error ϵ o p t \epsilon_{\mathrm{opt}} ϵ opt . Furthermore, we assume a setting where multiple queries can be submitted simultaneously. Along with the query depth, we also compute the computational depth and total number of queries to the stochastic gradient oracle to characterize the efficiency of our algorithms. As in [15 ] , the computational depth measures the parallel depth of the algorithm, assuming that each oracle query has O ( 1 ) O(1) O ( 1 ) depth, and the total number of queries refers to the total number of queries made to the stochastic gradient oracle over all rounds.
Generalized Gaussian. A key technical tool that we introduce to achieve the parallel rates for our EpochDA and ℓ p \ell_p ℓ p -AC-SA solvers is an ℓ p \ell_p ℓ p -Gaussian Reweighting for Gradient Estimation (p p p -GREG) estimator. Beginning with a generalized ℓ p \ell_p ℓ p -norm Gaussian as studied by [8 ] , our estimator computes an unbiased estimate of a convolution of our objective f f f with this ℓ p \ell_p ℓ p Gaussian, which furthermore offers the potential for parallelism.
EpochDA Based Solver. Our first algorithm (Theorem 14) is based on an epoch-based stochastic solver using our p p p -GREG estimators within the ball acceleration framework. By implementing the stochastic gradient queries in parallel, our algorithm achieves an oracle query depth of O ~ ( d p 2 ( p + 1 ) ϵ o p t − p p + 1 ) \widetilde{O}\left(d^{\frac{p}{2(p+1)}}\epsilon_{\mathrm{opt}}^{-\frac{p}{p+1}}\right) O ( d 2 ( p + 1 ) p ϵ opt − p + 1 p ) when L p = R p = 1 L_p=R_p=1 L p = R p = 1 . This improves upon the O ( ϵ o p t − p ) O(\epsilon_{\mathrm{opt}}^{-p}) O ( ϵ opt − p ) oracle query depth of stochastic ℓ p \ell_p ℓ p mirror descent in the regime ϵ o p t ≤ d − 1 2 p \epsilon_{\mathrm{opt}}\leq d^{-\frac{1}{2p}} ϵ opt ≤ d − 2 p 1 , up to logarithmic factors.
ℓ p \ell_p ℓ p -AC-SA Based Solver. Our second algorithm (Theorem 17) replaces EpochDA with an accelerated counterpart based on ℓ p \ell_p ℓ p -AC-SA, achieving a computational depth of O ~ ( d p 2 ( p + 1 ) ϵ o p t − p p + 1 + d p 2 ( p + 2 ) ϵ o p t − 2 p p + 1 ) \widetilde{O}\left(d^{\frac{p}{2(p+1)}}\epsilon_{\mathrm{opt}}^{-\frac{p}{p+1}}+d^{\frac{p}{2(p+2)}}\epsilon_{\mathrm{opt}}^{-\frac{2p}{p+1}}\right) O ( d 2 ( p + 1 ) p ϵ opt − p + 1 p + d 2 ( p + 2 ) p ϵ opt − p + 1 2 p ) when L p = R p = 1 L_p=R_p=1 L p = R p = 1 . This improves upon both the oracle query depth and computational depth of stochastic ℓ p \ell_p ℓ p mirror descent in the regime ϵ o p t < d − 1 2 p \epsilon_{\mathrm{opt}}<d^{-\frac{1}{2p}} ϵ opt < d − 2 p 1 , up to logarithmic factors. Compared with our EpochDA-based solver, our ℓ p \ell_p ℓ p AC-SA variant further reduces computational depth while maintaining the same oracle query depth and total query complexity up to logarithmic factors.
Parallel Lower Bounds. Several prior works have established fundamental limitations on parallelizing nonsmooth convex optimization. [10 ] developed an adaptivity-based lower-bound framework, showing that polynomially many parallel queries cannot improve upon sequential convergence rates for up to Ω ~ ( d 1 / 3 ) \widetilde{\Omega}(d^{1/3}) Ω ( d 1/3 ) rounds in Euclidean geometry. Subsequently, [9 ] strengthened the Euclidean lower bound to Ω ~ ( d 1 / 2 ) \widetilde{\Omega}(d^{1/2}) Ω ( d 1/2 ) by introducing a wall construction. For general ℓ p \ell_p ℓ p geometries with p ≥ 2 p\geq2 p ≥ 2 , [18 ] established a lower bound of Ω ~ ( min { ϵ − p , ϵ 2 / 3 d 1 / 3 } ) \widetilde{\Omega}\left(\min\{\epsilon^{-p},\epsilon^{2/3}d^{1/3}\}\right) Ω ( min { ϵ − p , ϵ 2/3 d 1/3 } ) . Building on the proof framework of [10 ] and incorporating the wall function adapted to ℓ p \ell_p ℓ p geometry, we establish an improved lower bound of
Ω ~ p ( min { ϵ − p , d p 3 p − 2 } ) \widetilde{\Omega}_{p}\left(\min\left\{\epsilon^{-p},d^{\frac{p}{3p-2}}\right\}\right) Ω p ( min { ϵ − p , d 3 p − 2 p } )
for nonsmooth convex optimization over a subset of the unit ℓ p \ell_p ℓ p ball, with polynomially many parallel local-oracle queries per round. Our result improves the dimension-dependent rate obtained by adapting the construction of [18 ] to the same feasible set, while recovering the known d 1 / 2 d^{1/2} d 1/2 exponent when p = 2 p = 2 p = 2 . However, our current upper bound retains a dimension exponent of 1 / 2 1/2 1/2 , whereas our lower bound has exponent p / ( 3 p − 2 ) < 1 / 2 p/(3p - 2) < 1/2 p / ( 3 p − 2 ) < 1/2 . Closing this gap and determining the optimal parallel complexity in general ℓ p \ell_p ℓ p geometry remains an open question.
Table 1. Query and computational depths to achieve ε o p t \varepsilon_{\mathrm{opt}} ε opt -accurate solution, for general L 2 L_2 L 2 , R 2 R_2 R 2 , L p L_p L p , R p > 0 R_p > 0 R p > 0 , where L 2 L_2 L 2 and L p L_p L p represent ℓ 2 \ell_2 ℓ 2 and ℓ p \ell_p ℓ p Lipschitz constants respectively and R 2 R_2 R 2 and R p R_p R p represent the ℓ 2 \ell_2 ℓ 2 and ℓ p \ell_p ℓ p norm bounds on the feasible set (omitting additional polylogarithmic factors).
Paper Organization and Notation
In Section 2, we introduce an ℓ p \ell_p ℓ p -generalized Gaussian distribution and present the p p p -GREG estimator, which estimates the gradient of a convolution of f f f with this ℓ p \ell_p ℓ p -generalized Gaussian. In Section 3, we introduce our ODCp _p p algorithms, and discuss their applications to projection-efficient (Section 3.2) and convex composite (Section 3.4) optimization, in addition to efficient ℓ p \ell_p ℓ p -proximal point estimation (Section 3.5).
In Section 4, we present two algorithms, EpochDA and ℓ p \ell_p ℓ p -AC-SA and present upper bounds on their query depths, computational depths, and total number of gradient queries. In Section 5, we present our lower bounds for the number of adaptive rounds needed to achieve at most ε \varepsilon ε error.
Throughout the paper, we use O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) to hide polylogarithmic factors in the parameters and will refer to p p p and q q q as Hölder conjugate exponents (i.e., 1 p + 1 q = 1 \frac{1}{p}+\frac{1}{q}=1 p 1 + q 1 = 1 ). Moreover, unless otherwise specified, we consider p ∈ [ 2 , ∞ ) p \in[2,\infty) p ∈ [ 2 , ∞ ) . We use B p ( x ˉ , r ) B_p(\bar{x},r) B p ( x ˉ , r ) to denote an ℓ p \ell_p ℓ p ball around x ˉ \bar{x} x ˉ of radius r r r , i.e., the set
Table 2. Total query complexities to achieve ε o p t \varepsilon_{\mathrm{opt}} ε opt -accurate solution, for general L 2 L_2 L 2 , R 2 R_2 R 2 , L p L_p L p , R p > 0 R_p > 0 R p > 0 , where L 2 L_2 L 2 and L p L_p L p represent ℓ 2 \ell_2 ℓ 2 and ℓ p \ell_p ℓ p Lipschitz constants respectively and R 2 R_2 R 2 and R p R_p R p represent the ℓ 2 \ell_2 ℓ 2 and ℓ p \ell_p ℓ p norm bounds on the feasible set (omitting additional polylogarithmic factors).
{ x ′ ∈ R d : ∥ x ′ − x ∥ p ≤ r } \{x' \in\mathbb{R}^d : \lVert x' - x\rVert_p \le r\} { x ′ ∈ R d : ∥ x ′ − x ∥ p ≤ r } . Furthermore, for a ∈ ( 0 , 1 ] a \in(0,1] a ∈ ( 0 , 1 ] , let Geom ( a ) \operatorname{Geom}(a) Geom ( a ) denote the geometric distribution with parameter a a a . We also refer to N ( μ , σ 2 I ) \mathcal{N}(\mu,\sigma^2 I) N ( μ , σ 2 I ) as a multivariate Gaussian distribution with mean μ ∈ R d \mu\in\mathbb{R}^d μ ∈ R d and covariance σ 2 I \sigma^2 I σ 2 I , where I I I denotes the identity matrix in R d × d \mathbb{R}^{d \times d} R d × d .
We say that a function f : R d → R f : \mathbb{R}^d \to\mathbb{R} f : R d → R is L L L -Lipschitz with respect to the ℓ p \ell_p ℓ p -norm if, for all x , y ∈ R d x,y \in\mathbb{R}^d x , y ∈ R d , it holds that ∣ f ( x ) − f ( y ) ∣ ≤ L ∥ x − y ∥ p |f(x)-f(y)| \le L\lVert x-y\rVert_p ∣ f ( x ) − f ( y ) ∣ ≤ L ∥ x − y ∥ p . We say that f f f is ( L , κ ) (L,\kappa) ( L , κ ) -weakly smooth with respect to ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p for L > 0 L>0 L > 0 and κ ∈ ( 1 , 2 ] \kappa\in(1,2] κ ∈ ( 1 , 2 ] , if its gradient is ( L , κ − 1 ) (L,\kappa-1) ( L , κ − 1 ) -Hölder continuous, i.e.,
∀ x , y ∈ R d , ∥ ∇ f ( x ) − ∇ f ( y ) ∥ q ≤ L ∥ x − y ∥ p κ − 1 . \forall x,y \in\mathbb{R}^d,\ \lVert\nabla f(x)-\nabla f(y)\rVert_q \le L\lVert x-y\rVert_p^{\kappa-1}. ∀ x , y ∈ R d , ∥ ∇ f ( x ) − ∇ f ( y ) ∥ q ≤ L ∥ x − y ∥ p κ − 1 .
We say that f f f is convex if for all x , y ∈ R d x,y \in\mathbb{R}^d x , y ∈ R d and λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] , f ( λ x + ( 1 − λ ) y ) ≤ λ f ( x ) + ( 1 − λ ) f ( y ) f(\lambda x+(1-\lambda)y) \le\lambda f(x)+(1-\lambda)f(y) f ( λ x + ( 1 − λ ) y ) ≤ λ f ( x ) + ( 1 − λ ) f ( y ) . Moreover, we say that f f f is uniformly convex with convexity parameter μ \mu μ and order p p p if for all x , y ∈ R d x,y \in\mathbb{R}^d x , y ∈ R d , it holds that
f ( y ) ≥ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ + μ p ∥ y − x ∥ p p . f(y) \ge f(x)+\langle\nabla f(x),y-x\rangle+\frac{\mu}{p}\lVert y-x\rVert_p^p. f ( y ) ≥ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ + p μ ∥ y − x ∥ p p .
Framework
Generalized ℓ p \ell_p ℓ p Gaussian
First, we state the density of our Generalized ℓ p \ell_p ℓ p Gaussian (G G p \mathcal{GG}_p GG p ) [8 ] . For any X ∼ G G p ( μ , σ 2 ) X \sim\mathcal{GG}_p(\mu,\sigma^2) X ∼ GG p ( μ , σ 2 ) , its density is represented as
( ( 2 Γ ( 1 + 1 / p ) ) d Γ ( 1 + d / p ) ( 2 σ 2 ) d / 2 Γ ( 1 + d / 2 ) ) − 1 exp ( − ∥ x − μ ∥ p 2 2 σ 2 ) . \left(\frac{(2\Gamma(1+1/p))^d}{\Gamma(1+d/p)}(2\sigma^2)^{d/2}\Gamma(1+d/2)\right)^{-1}\exp\left(-\frac{\lVert x-\mu\rVert_p^2}{2\sigma^2}\right). ( Γ ( 1 + d / p ) ( 2Γ ( 1 + 1/ p ) ) d ( 2 σ 2 ) d /2 Γ ( 1 + d /2 ) ) − 1 exp ( − 2 σ 2 ∥ x − μ ∥ p 2 ) .
We will use γ ρ , p \gamma_{\rho,p} γ ρ , p to denote the probability density function of G G p ( 0 , ρ 2 ) \mathcal{GG}_p(0,\rho^2) GG p ( 0 , ρ 2 ) . Now, we define the convolution operation for our distribution.
Definition 1. (*G G p \mathcal{GG}_p GG p -Convolution) For a function f : R d → R f : \mathbb{R}^d \to\mathbb{R} f : R d → R , we denote its convolution with a G G p \mathcal{GG}_p GG p random variable of mean 0 and variance ρ 2 \rho^2 ρ 2 by
f ^ ρ , p ( x ) : = E y ∼ G G p ( 0 , ρ 2 ) f ( x + y ) = ∫ y ∈ R d f ( x − y ) γ ρ , p ( y ) d y . \hat{f}_{\rho,p}(x) := \mathbb{E}_{y\sim\mathcal{GG}_p(0,\rho^2)} f(x+y) = \int_{y\in\mathbb{R}^d} f(x-y)\gamma_{\rho,p}(y)dy. f ^ ρ , p ( x ) := E y ∼ GG p ( 0 , ρ 2 ) f ( x + y ) = ∫ y ∈ R d f ( x − y ) γ ρ , p ( y ) d y .
p-GREG Estimator
We now present our p p p -GREG (ℓ p \ell_p ℓ p -Generalized Gaussian Reweighting for Gradient Estimation) estimator for the ℓ p \ell_p ℓ p norm, which is related to the ReSQue estimator [15 ] , as well as the estimator in [9 ] .
Definition 2. (*p p p -GREG Estimator) Let x ˉ ∈ R d \bar{x}\in\mathbb{R}^d x ˉ ∈ R d and let f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R be convex. Suppose we have a gradient estimator g : R d → R d g:\mathbb{R}^d\to\mathbb{R}^d g : R d → R d such that E [ g ] ∈ ∂ f \mathbb{E}[g]\in\partial f E [ g ] ∈ ∂ f . We define the p p p -GREG estimator of radius ρ \rho ρ as the following:
∇ ~ x ˉ g f ^ ρ , p ( x ) : = γ ρ , p ( x − x ˉ − ξ ) γ ρ , p ( ξ ) g ( x ˉ + ξ ) , ξ ∼ G G p ( 0 , ρ 2 ) \widetilde{\nabla}_{\bar{x}}^{g}\hat{f}_{\rho,p}(x) := \frac{\gamma_{\rho,p}(x-\bar{x}-\xi)}{\gamma_{\rho,p}(\xi)}g(\bar{x}+\xi),\ \xi\sim\mathcal{GG}_p(0,\rho^2) ∇ x ˉ g f ^ ρ , p ( x ) := γ ρ , p ( ξ ) γ ρ , p ( x − x ˉ − ξ ) g ( x ˉ + ξ ) , ξ ∼ GG p ( 0 , ρ 2 ) where ξ \xi ξ is sampled first and g g g is independently queried at x ˉ + ξ \bar{x}+\xi x ˉ + ξ .
We next prove that if g g g is an unbiased estimate of ∂ f \partial f ∂ f and has a certain moment bound, then our estimator is also unbiased for the convolved function and enjoys a similar moment bound.
Lemma 1. The following statements hold about our estimator presented in Definition 2:
E [ ∇ ~ x ˉ g f ^ ρ , p ( x ) ] = ∇ f ^ ρ , p ( x ) \mathbb{E}[\widetilde{\nabla}_{\bar{x}}^{g}\hat{f}_{\rho,p}(x)] = \nabla\hat{f}_{\rho,p}(x) E [ ∇ x ˉ g f ^ ρ , p ( x )] = ∇ f ^ ρ , p ( x )
If E ∥ g ( z ) ∥ q q ≤ G q \mathbb{E}\lVert g(z)\rVert_q^q\leq G^q E ∥ g ( z ) ∥ q q ≤ G q for all z ∈ R d z\in\mathbb{R}^d z ∈ R d and x ∈ B p ( x ˉ , ρ ) x\in B_p(\bar{x},\rho) x ∈ B p ( x ˉ , ρ ) , then E ∥ ∇ ~ x ˉ g f ^ ρ , p ( x ) ∥ q q ≤ 3 G q \mathbb{E}\lVert\widetilde{\nabla}_{\bar{x}}^{g}\hat{f}_{\rho,p}(x)\rVert_q^q\leq3G^q E ∥ ∇ x ˉ g f ^ ρ , p ( x ) ∥ q q ≤ 3 G q
Proof. To prove the first statement, we note that
E [ ∇ ~ x ˉ g f ^ ρ , p ( x ) ] = ∫ γ ρ , p ( x − x ˉ − ξ ) γ ρ , p ( ξ ) E [ g ( x ˉ + ξ ) ] γ ρ , p ( ξ ) d ξ = ∫ γ ρ , p ( x − x ˉ − ξ ) E [ g ( x ˉ + ξ ) ] d ξ = ∫ E [ g ( x ˉ + ξ ) ] ⋅ γ ρ , p ( x − x ˉ − ξ ) d ξ = ∇ f ^ ρ , p ( x ) \begin{aligned}
\mathbb{E}[\widetilde{\nabla}_{\bar{x}}^{g}\hat{f}_{\rho,p}(x)] = \int\frac{\gamma_{\rho,p}(x-\bar{x}-\xi)}{\gamma_{\rho,p}(\xi)}\mathbb{E}[g(\bar{x}+\xi)]\gamma_{\rho,p}(\xi)d\xi= \int\gamma_{\rho,p}(x-\bar{x}-\xi)\mathbb{E}[g(\bar{x}+\xi)]d\xi\\
&= \int\mathbb{E}[g(\bar{x}+\xi)]\cdot\gamma_{\rho,p}(x-\bar{x}-\xi)d\xi= \nabla\hat{f}_{\rho,p}(x)
\end{aligned} E [ ∇ x ˉ g f ^ ρ , p ( x )] = ∫ γ ρ , p ( ξ ) γ ρ , p ( x − x ˉ − ξ ) E [ g ( x ˉ + ξ )] γ ρ , p ( ξ ) d ξ = ∫ γ ρ , p ( x − x ˉ − ξ ) E [ g ( x ˉ + ξ )] d ξ = ∫ E [ g ( x ˉ + ξ )] ⋅ γ ρ , p ( x − x ˉ − ξ ) d ξ = ∇ f ^ ρ , p ( x )
To show the second statement, let v = x − x ˉ v=x-\bar{x} v = x − x ˉ . By definition,
∥ ∇ ~ x ˉ g f ^ ρ , p ( x ) ∥ q q = ( γ ρ , p ( v − ξ ) γ ρ , p ( ξ ) ) q ∥ g ( x ˉ + ξ ) ∥ q q . \left\lVert\widetilde{\nabla}_{\bar{x}}^{g}\hat{f}_{\rho,p}(x)\right\rVert_q^q = \left(\frac{\gamma_{\rho,p}(v-\xi)}{\gamma_{\rho,p}(\xi)}\right)^q\lVert g(\bar{x}+\xi)\rVert_q^q. ∇ x ˉ g f ^ ρ , p ( x ) q q = ( γ ρ , p ( ξ ) γ ρ , p ( v − ξ ) ) q ∥ g ( x ˉ + ξ ) ∥ q q .
Taking expectation first with respect to the randomness of g g g and using E g ∥ g ( z ) ∥ q q ≤ G q \mathbb{E}_g\lVert g(z)\rVert_q^q\leq G^q E g ∥ g ( z ) ∥ q q ≤ G q , we get
E ∥ ∇ ~ x ˉ g f ^ ρ , p ( x ) ∥ q q ≤ G q E ξ ∼ γ ρ , p [ ( γ ρ , p ( v − ξ ) γ ρ , p ( ξ ) ) q ] = G q ∫ γ ρ , p ( v − ξ ) q γ ρ , p ( ξ ) q − 1 d ξ = G q ( 2 Γ ( 1 + 1 / p ) d Γ ( 1 + d / p ) ( 2 ρ 2 ) d / 2 Γ ( 1 + d / 2 ) ) q − 1 ∫ exp ( ∥ ξ ∥ p 2 − p ∥ v − ξ ∥ p 2 2 ( p − 1 ) ρ 2 ) d ξ (1) \begin{aligned}
\mathbb{E}\left\lVert\widetilde{\nabla}_{\bar{x}}^{g}\hat{f}_{\rho,p}(x)\right\rVert_q^q \leq G^q\mathbb{E}_{\xi\sim\gamma_{\rho,p}}\left[\left(\frac{\gamma_{\rho,p}(v-\xi)}{\gamma_{\rho,p}(\xi)}\right)^q\right] = G^q\int\frac{\gamma_{\rho,p}(v-\xi)^q}{\gamma_{\rho,p}(\xi)^{q-1}}d\xi\\
&=G^q\left(\frac{2\Gamma(1+1/p)^d}{\Gamma(1+d/p)}(2\rho^2)^{d/2}\Gamma(1+d/2)\right)^{q-1}\int\exp\left(\frac{\lVert\xi\rVert_p^2-p\lVert v-\xi\rVert_p^2}{2(p-1)\rho^2}\right)d\xi \tag*{(1)}
\end{aligned} E ∇ x ˉ g f ^ ρ , p ( x ) q q ≤ G q E ξ ∼ γ ρ , p [ ( γ ρ , p ( ξ ) γ ρ , p ( v − ξ ) ) q ] = G q ∫ γ ρ , p ( ξ ) q − 1 γ ρ , p ( v − ξ ) q d ξ = G q ( Γ ( 1 + d / p ) 2Γ ( 1 + 1/ p ) d ( 2 ρ 2 ) d /2 Γ ( 1 + d /2 ) ) q − 1 ∫ exp ( 2 ( p − 1 ) ρ 2 ∥ ξ ∥ p 2 − p ∥ v − ξ ∥ p 2 ) d ξ (1)
Use the fact that f ( x ) = 1 2 ∥ x ∥ p 2 f(x)=\frac{1}{2}\lVert x\rVert_p^2 f ( x ) = 2 1 ∥ x ∥ p 2 is ( p − 1 ) (p-1) ( p − 1 ) -smooth with respect to the p p p norm, by Theorem 5.8 of [6 ] , we have
f ( λ x + ( 1 − λ ) y ) ≥ λ f ( x ) + ( 1 − λ ) f ( y ) − p − 1 2 λ ( 1 − λ ) ∥ x − y ∥ p 2 , ∀ x , y ∈ R d , λ ∈ [ 0 , 1 ] . f(\lambda x+(1-\lambda)y)\geq\lambda f(x)+(1-\lambda)f(y)-\frac{p-1}{2}\lambda(1-\lambda)\lVert x-y\rVert_p^2,\qquad\forall x,y\in\mathbb{R}^d,\ \lambda\in[0,1]. f ( λ x + ( 1 − λ ) y ) ≥ λ f ( x ) + ( 1 − λ ) f ( y ) − 2 p − 1 λ ( 1 − λ ) ∥ x − y ∥ p 2 , ∀ x , y ∈ R d , λ ∈ [ 0 , 1 ] .
Let λ = ( p − 1 ) / p \lambda=(p-1)/p λ = ( p − 1 ) / p , x = ξ − 1 λ v x=\xi-\frac{1}{\lambda}v x = ξ − λ 1 v and y = ξ y=\xi y = ξ one can get
f ( ξ − v ) = f ( λ x + ( 1 − λ ) y ) ≥ λ f ( x ) + ( 1 − λ ) f ( y ) − p − 1 2 λ ( 1 − λ ) ∥ x − y ∥ p 2 = p − 1 p f ( ξ − 1 λ v ) + 1 p f ( ξ ) − ( p − 1 ) 2 2 p 2 ∥ p p − 1 v ∥ p 2 \begin{aligned}
f(\xi-v)=f(\lambda x+(1-\lambda)y)\geq\lambda f(x)+(1-\lambda)f(y)-\frac{p-1}{2}\lambda(1-\lambda)\lVert x-y\rVert_p^2 \\
&=\frac{p-1}{p}f\left(\xi-\frac{1}{\lambda}v\right)+\frac{1}{p}f(\xi)-\frac{(p-1)^2}{2p^2}\left\lVert\frac{p}{p-1}v\right\rVert_p^2
\end{aligned} f ( ξ − v ) = f ( λ x + ( 1 − λ ) y ) ≥ λ f ( x ) + ( 1 − λ ) f ( y ) − 2 p − 1 λ ( 1 − λ ) ∥ x − y ∥ p 2 = p p − 1 f ( ξ − λ 1 v ) + p 1 f ( ξ ) − 2 p 2 ( p − 1 ) 2 p − 1 p v p 2
By rearranging and multiplying both sides by p / ( p − 1 ) p/(p-1) p / ( p − 1 ) , this yields
∥ ξ ∥ p 2 2 ( p − 1 ) − p ∥ ξ − v ∥ p 2 2 ( p − 1 ) ≤ − ∥ ξ − 1 λ v ∥ p 2 2 + p ∥ v ∥ p 2 2 ( p − 1 ) \frac{\lVert\xi\rVert_p^2}{2(p-1)}-\frac{p\lVert\xi-v\rVert_p^2}{2(p-1)}\leq-\frac{\left\lVert\xi-\frac{1}{\lambda}v\right\rVert_p^2}{2}+\frac{p\lVert v\rVert_p^2}{2(p-1)} 2 ( p − 1 ) ∥ ξ ∥ p 2 − 2 ( p − 1 ) p ∥ ξ − v ∥ p 2 ≤ − 2 ξ − λ 1 v p 2 + 2 ( p − 1 ) p ∥ v ∥ p 2
Using this inequality we have
E [ ∥ ∇ ~ x f ρ , p ( x ) ∥ q q ] ≤ G q ( ( 2 Γ ( 1 + 1 / p ) ) d Γ ( 1 + d / p ) ( 2 ρ 2 ) d / 2 Γ ( 1 + d / 2 ) ) − 1 exp ( p ∥ v ∥ p 2 2 ( p − 1 ) ρ 2 ) ∫ exp ( − ∥ ξ − 1 λ v ∥ p 2 2 ρ 2 ) d ξ ≤ G q exp ( p ∥ v ∥ p 2 2 ( p − 1 ) ρ 2 ) ≤ 3 G q \begin{aligned}
\mathbb{E}\left[\left\lVert\widetilde{\nabla}_{x}f_{\rho,p}(x)\right\rVert_q^q\right]\leq G^q\left(\frac{(2\Gamma(1+1/p))^d}{\Gamma(1+d/p)}(2\rho^2)^{d/2}\Gamma(1+d/2)\right)^{-1}\exp\left(\frac{p\lVert v\rVert_p^2}{2(p-1)\rho^2}\right)\int\exp\left(-\frac{\left\lVert\xi-\frac{1}{\lambda}v\right\rVert_p^2}{2\rho^2}\right)d\xi\\
&\leq G^q\exp\left(\frac{p\lVert v\rVert_p^2}{2(p-1)\rho^2}\right)\leq3G^q
\end{aligned} E [ ∇ x f ρ , p ( x ) q q ] ≤ G q ( Γ ( 1 + d / p ) ( 2Γ ( 1 + 1/ p ) ) d ( 2 ρ 2 ) d /2 Γ ( 1 + d /2 ) ) − 1 exp ( 2 ( p − 1 ) ρ 2 p ∥ v ∥ p 2 ) ∫ exp − 2 ρ 2 ξ − λ 1 v p 2 d ξ ≤ G q exp ( 2 ( p − 1 ) ρ 2 p ∥ v ∥ p 2 ) ≤ 3 G q
where the last inequality uses that ∥ v ∥ p ≤ ρ \lVert v\rVert_p\leq\rho ∥ v ∥ p ≤ ρ .
We also prove the following bounds on higher-order moments for two important quantities.
Lemma 2. If x , x ′ ∈ B x ( ρ / m ) x,x'\in B_x(\rho/m) x , x ′ ∈ B x ( ρ / m ) for m ≥ 2 m\geq2 m ≥ 2 , then
E ξ ∼ G p ( 0 , ρ 2 ) [ ( γ p , p ( x − x ˉ − ξ ) γ p , p ( ξ ) ) m ] ≤ e p − 1 \mathbb{E}_{\xi\sim\mathcal{G}_p(0,\rho^2)}\left[\left(\frac{\gamma_{p,p}(x-\bar{x}-\xi)}{\gamma_{p,p}(\xi)}\right)^m\right]\leq e^{p-1} E ξ ∼ G p ( 0 , ρ 2 ) [ ( γ p , p ( ξ ) γ p , p ( x − x ˉ − ξ ) ) m ] ≤ e p − 1 E ξ ∼ G p ( 0 , ρ 2 ) [ ∣ γ p , p ( x − x ˉ − ξ ) − γ p , p ( x ′ − x ˉ − ξ ) γ p , p ( ξ ) ∣ m ] ≤ e p − 1 ( C p m ∥ x − x ′ ∥ p ρ ) m \mathbb{E}_{\xi\sim\mathcal{G}_p(0,\rho^2)}\left[\left\lvert\frac{\gamma_{p,p}(x-\bar{x}-\xi)-\gamma_{p,p}(x'-\bar{x}-\xi)}{\gamma_{p,p}(\xi)}\right\rvert^m\right]\leq e^{p-1}\left(\frac{C_p\sqrt{m}\lVert x-x'\rVert_p}{\rho}\right)^m E ξ ∼ G p ( 0 , ρ 2 ) [ γ p , p ( ξ ) γ p , p ( x − x ˉ − ξ ) − γ p , p ( x ′ − x ˉ − ξ ) m ] ≤ e p − 1 ( ρ C p m ∥ x − x ′ ∥ p ) m for some constant $C_p>0.
Proof. Denote v = x − x ˉ v=x-\bar{x} v = x − x ˉ and v ′ = x ′ − x ˉ v'=x'-\bar{x} v ′ = x ′ − x ˉ . By definition,
E ξ ∼ G p ( 0 , ρ 2 ) [ ( γ p , p ( x − x ˉ − ξ ) γ p , p ( ξ ) ) m ] = E ξ ∼ G p ( 0 , ρ 2 ) [ exp ( m ( ∥ ξ ∥ p 2 − ∥ v − ξ ∥ p 2 ) 2 ρ 2 ) ] ≤ exp ( m ( p − 1 ) ∥ v ∥ p 2 2 ρ 2 ) ⋅ E ξ ∼ G p ( 0 , ρ 2 ) [ exp ( m ρ 2 ⟨ ∇ 1 2 ∥ ξ ∥ p 2 , v ⟩ ) ] \begin{aligned}
\mathbb{E}_{\xi\sim\mathcal{G}_p(0,\rho^2)}\left[\left(\frac{\gamma_{p,p}(x-\bar{x}-\xi)}{\gamma_{p,p}(\xi)}\right)^m\right]=\mathbb{E}_{\xi\sim\mathcal{G}_p(0,\rho^2)}\left[\exp\left(\frac{m(\lVert\xi\rVert_p^2-\lVert v-\xi\rVert_p^2)}{2\rho^2}\right)\right] \\
&\leq\exp\left(\frac{m(p-1)\lVert v\rVert_p^2}{2\rho^2}\right)\cdot\mathbb{E}_{\xi\sim\mathcal{G}_p(0,\rho^2)}\left[\exp\left(\frac{m}{\rho^2}\left\langle\nabla\frac{1}{2}\lVert\xi\rVert_p^2,v\right\rangle\right)\right]
\end{aligned} E ξ ∼ G p ( 0 , ρ 2 ) [ ( γ p , p ( ξ ) γ p , p ( x − x ˉ − ξ ) ) m ] = E ξ ∼ G p ( 0 , ρ 2 ) [ exp ( 2 ρ 2 m (∥ ξ ∥ p 2 − ∥ v − ξ ∥ p 2 ) ) ] ≤ exp ( 2 ρ 2 m ( p − 1 ) ∥ v ∥ p 2 ) ⋅ E ξ ∼ G p ( 0 , ρ 2 ) [ exp ( ρ 2 m ⟨ ∇ 2 1 ∥ ξ ∥ p 2 , v ⟩ ) ]
where the inequality is by the smoothness of 1 2 ∥ ⋅ ∥ p 2 \frac{1}{2}\lVert\cdot\rVert_p^2 2 1 ∥ ⋅ ∥ p 2 . Since ∥ v ∥ p ≤ ρ / m \lVert v\rVert_p\leq\rho/m ∥ v ∥ p ≤ ρ / m , we have, with Fact 5,
E ξ ∼ G p ( 0 , ρ 2 ) [ ( γ p , p ( x − x ˉ − ξ ) γ p , p ( ξ ) ) m ] ≤ exp ( m ( p − 1 ) ∥ v ∥ p 2 2 ρ 2 ) ⋅ E ξ ∼ G p ( 0 , ρ 2 ) [ exp ( m ρ 2 ⟨ ∇ 1 2 ∥ ξ ∥ p 2 , v ⟩ ) ] ≤ exp ( p − 1 2 m ) exp ( ( p − 1 ) m 2 2 ρ 2 ∥ v ∥ p 2 ) ≤ exp ( p − 1 2 m + p − 1 2 ) ≤ e p − 1 \begin{aligned}
\mathbb{E}_{\xi\sim\mathcal{G}_p(0,\rho^2)}\left[\left(\frac{\gamma_{p,p}(x-\bar{x}-\xi)}{\gamma_{p,p}(\xi)}\right)^m\right]\leq\exp\left(\frac{m(p-1)\lVert v\rVert_p^2}{2\rho^2}\right)\cdot\mathbb{E}_{\xi\sim\mathcal{G}_p(0,\rho^2)}\left[\exp\left(\frac{m}{\rho^2}\left\langle\nabla\frac{1}{2}\lVert\xi\rVert_p^2,v\right\rangle\right)\right] \\
&\leq\exp\left(\frac{p-1}{2m}\right)\exp\left(\frac{(p-1)m^2}{2\rho^2}\lVert v\rVert_p^2\right)\leq\exp\left(\frac{p-1}{2m}+\frac{p-1}{2}\right) \\
&\leq e^{p-1}
\end{aligned} E ξ ∼ G p ( 0 , ρ 2 ) [ ( γ p , p ( ξ ) γ p , p ( x − x ˉ − ξ ) ) m ] ≤ exp ( 2 ρ 2 m ( p − 1 ) ∥ v ∥ p 2 ) ⋅ E ξ ∼ G p ( 0 , ρ 2 ) [ exp ( ρ 2 m ⟨ ∇ 2 1 ∥ ξ ∥ p 2 , v ⟩ ) ] ≤ exp ( 2 m p − 1 ) exp ( 2 ρ 2 ( p − 1 ) m 2 ∥ v ∥ p 2 ) ≤ exp ( 2 m p − 1 + 2 p − 1 ) ≤ e p − 1
For the second inequality, we define the line segment v t : = v ′ + t ( v − v ′ ) v_t := v' + t(v - v') v t := v ′ + t ( v − v ′ ) for t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] . By the fundamental theorem of calculus,
γ p , ρ ( v − ξ ) − γ p , ρ ( v ′ − ξ ) γ p , ρ ( ξ ) = ∫ 0 1 γ p , ρ ( v t − ξ ) γ p , ρ ( ξ ) ⟨ ∇ Φ ( ξ − v t ) , v − v ′ ⟩ ρ 2 d t , \frac{\gamma_{p,\rho}(v-\xi)-\gamma_{p,\rho}(v'-\xi)}{\gamma_{p,\rho}(\xi)}
= \int_0^1 \frac{\gamma_{p,\rho}(v_t-\xi)}{\gamma_{p,\rho}(\xi)}\frac{\langle\nabla\Phi(\xi-v_t),v-v'\rangle}{\rho^2}\,dt, γ p , ρ ( ξ ) γ p , ρ ( v − ξ ) − γ p , ρ ( v ′ − ξ ) = ∫ 0 1 γ p , ρ ( ξ ) γ p , ρ ( v t − ξ ) ρ 2 ⟨ ∇Φ ( ξ − v t ) , v − v ′ ⟩ d t ,
where Φ ( z ) : = 1 2 ∥ z ∥ p 2 \Phi(z) := \frac{1}{2}\|z\|_p^2 Φ ( z ) := 2 1 ∥ z ∥ p 2 . Taking absolute values and applying Jensen’s inequality (convexity of ∣ ⋅ ∣ m \lvert\cdot\rvert^m ∣ ⋅ ∣ m ) gives
∣ γ p , ρ ( v − ξ ) − γ p , ρ ( v ′ − ξ ) γ p , ρ ( ξ ) ∣ m ≤ ∫ 0 1 ( γ p , ρ ( v t − ξ ) γ p , ρ ( ξ ) ) m ( ∣ ⟨ ∇ Φ ( ξ − v t ) , v − v ′ ⟩ ∣ ρ 2 ) m d t \left\lvert\frac{\gamma_{p,\rho}(v-\xi)-\gamma_{p,\rho}(v'-\xi)}{\gamma_{p,\rho}(\xi)}\right\rvert^m
\le\int_0^1 \left(\frac{\gamma_{p,\rho}(v_t-\xi)}{\gamma_{p,\rho}(\xi)}\right)^m\left(\frac{\lvert\langle\nabla\Phi(\xi-v_t),v-v'\rangle\rvert}{\rho^2}\right)^m\,dt γ p , ρ ( ξ ) γ p , ρ ( v − ξ ) − γ p , ρ ( v ′ − ξ ) m ≤ ∫ 0 1 ( γ p , ρ ( ξ ) γ p , ρ ( v t − ξ ) ) m ( ρ 2 ∣⟨ ∇Φ ( ξ − v t ) , v − v ′ ⟩∣ ) m d t
Taking expectations with respect to ξ ∼ G G p ( 0 , ρ 2 ) \xi\sim\mathcal{GG}_p(0,\rho^2) ξ ∼ GG p ( 0 , ρ 2 ) and applying Fubini’s theorem (as the integral is nonnegative) we get
E ξ [ ∣ γ p , ρ ( v − ξ ) − γ p , ρ ( v ′ − ξ ) γ p , ρ ( ξ ) ∣ m ] ≤ ∫ 0 1 E ξ [ ( γ p , ρ ( v t − ξ ) γ p , ρ ( ξ ) ) m ( ∣ ⟨ ∇ Φ ( ξ − v t ) , v − v ′ ⟩ ∣ ρ 2 ) m ] d t . \mathbb{E}_{\xi}\left[\left\lvert\frac{\gamma_{p,\rho}(v-\xi)-\gamma_{p,\rho}(v'-\xi)}{\gamma_{p,\rho}(\xi)}\right\rvert^m\right]
\le\int_0^1 \mathbb{E}_{\xi}\left[\left(\frac{\gamma_{p,\rho}(v_t-\xi)}{\gamma_{p,\rho}(\xi)}\right)^m\left(\frac{\lvert\langle\nabla\Phi(\xi-v_t),v-v'\rangle\rvert}{\rho^2}\right)^m\right]\,dt. E ξ [ γ p , ρ ( ξ ) γ p , ρ ( v − ξ ) − γ p , ρ ( v ′ − ξ ) m ] ≤ ∫ 0 1 E ξ [ ( γ p , ρ ( ξ ) γ p , ρ ( v t − ξ ) ) m ( ρ 2 ∣⟨ ∇Φ ( ξ − v t ) , v − v ′ ⟩∣ ) m ] d t .
Now, let z = ξ − v t z=\xi-v_t z = ξ − v t , which implies that ξ = z + v t \xi=z+v_t ξ = z + v t . Since Φ \Phi Φ is even, it holds that γ p , ρ ( v t − ξ ) = γ p , ρ ( − z ) = γ p , ρ ( z ) \gamma_{p,\rho}(v_t-\xi)=\gamma_{p,\rho}(-z)=\gamma_{p,\rho}(z) γ p , ρ ( v t − ξ ) = γ p , ρ ( − z ) = γ p , ρ ( z ) . Therefore, letting w = v − v ′ = x − x ′ w=v-v'=x-x' w = v − v ′ = x − x ′ ,
E ξ [ ( γ p , ρ ( v t − ξ ) γ p , ρ ( ξ ) ) m ( ∣ ⟨ ∇ Φ ( ξ − v t ) , v − v ′ ⟩ ∣ ρ 2 ) m ] = ∫ R d γ p , ρ ( ξ ) γ p , ρ ( v t − ξ ) m γ p , ρ ( ξ ) m ( ∣ ⟨ ∇ Φ ( ξ − v t ) , w ⟩ ∣ ρ 2 ) m d ξ = ∫ R d γ p , ρ ( z ) ( γ p , ρ ( z ) γ p , ρ ( z + v t ) ) m − 1 ( ∣ ⟨ ∇ Φ ( z ) , w ⟩ ∣ ρ 2 ) m d z = E z ∼ G G p ( 0 , ρ 2 ) [ exp ( ( m − 1 ) ( Φ ( z + v t ) − Φ ( z ) ) ρ 2 ) ( ∣ ⟨ ∇ Φ ( z ) , w ⟩ ∣ ρ 2 ) m ] \begin{aligned}
\mathbb{E}_{\xi}\left[\left(\frac{\gamma_{p,\rho}(v_t-\xi)}{\gamma_{p,\rho}(\xi)}\right)^m\left(\frac{\lvert\langle\nabla\Phi(\xi-v_t),v-v'\rangle\rvert}{\rho^2}\right)^m\right]
= \int_{\mathbb{R}^d}\frac{\gamma_{p,\rho}(\xi)\gamma_{p,\rho}(v_t-\xi)^m}{\gamma_{p,\rho}(\xi)^m}\left(\frac{\lvert\langle\nabla\Phi(\xi-v_t),w\rangle\rvert}{\rho^2}\right)^m\,d\xi\\
&= \int_{\mathbb{R}^d}\gamma_{p,\rho}(z)\left(\frac{\gamma_{p,\rho}(z)}{\gamma_{p,\rho}(z+v_t)}\right)^{m-1}\left(\frac{\lvert\langle\nabla\Phi(z),w\rangle\rvert}{\rho^2}\right)^m\,dz \\
&= \mathbb{E}_{z\sim\mathcal{GG}_p(0,\rho^2)}\left[\exp\left(\frac{(m-1)(\Phi(z+v_t)-\Phi(z))}{\rho^2}\right)\left(\frac{\lvert\langle\nabla\Phi(z),w\rangle\rvert}{\rho^2}\right)^m\right]
\end{aligned} E ξ [ ( γ p , ρ ( ξ ) γ p , ρ ( v t − ξ ) ) m ( ρ 2 ∣⟨ ∇Φ ( ξ − v t ) , v − v ′ ⟩∣ ) m ] = ∫ R d γ p , ρ ( ξ ) m γ p , ρ ( ξ ) γ p , ρ ( v t − ξ ) m ( ρ 2 ∣⟨ ∇Φ ( ξ − v t ) , w ⟩∣ ) m d ξ = ∫ R d γ p , ρ ( z ) ( γ p , ρ ( z + v t ) γ p , ρ ( z ) ) m − 1 ( ρ 2 ∣⟨ ∇Φ ( z ) , w ⟩∣ ) m d z = E z ∼ GG p ( 0 , ρ 2 ) [ exp ( ρ 2 ( m − 1 ) ( Φ ( z + v t ) − Φ ( z )) ) ( ρ 2 ∣⟨ ∇Φ ( z ) , w ⟩∣ ) m ]
By the ( p − 1 ) (p-1) ( p − 1 ) -smoothness of Φ \Phi Φ , it holds that
Φ ( z + v t ) − Φ ( z ) ≤ ⟨ ∇ Φ ( z ) , v t ⟩ + p − 1 2 ∥ v t ∥ p 2 . \Phi(z+v_t)-\Phi(z)\le\langle\nabla\Phi(z),v_t\rangle+\frac{p-1}{2}\|v_t\|_p^2. Φ ( z + v t ) − Φ ( z ) ≤ ⟨ ∇Φ ( z ) , v t ⟩ + 2 p − 1 ∥ v t ∥ p 2 .
Therefore
E z ∼ G G p ( 0 , ρ 2 ) [ exp ( ( m − 1 ) ( Φ ( z + v t ) − Φ ( z ) ) ρ 2 ) ( ∣ ⟨ ∇ Φ ( z ) , w ⟩ ∣ ρ 2 ) m ] ≤ exp ( ( m − 1 ) ( p − 1 ) 2 ρ 2 ∥ v t ∥ p 2 ) ⋅ E z [ exp ( m − 1 ρ 2 ⟨ ∇ Φ ( z ) , v t ⟩ ) ( ∣ ⟨ ∇ Φ ( z ) , w ⟩ ∣ ρ 2 ) m ] \begin{aligned}
\mathbb{E}_{z\sim\mathcal{GG}_p(0,\rho^2)}\left[\exp\left(\frac{(m-1)(\Phi(z+v_t)-\Phi(z))}{\rho^2}\right)\left(\frac{\lvert\langle\nabla\Phi(z),w\rangle\rvert}{\rho^2}\right)^m\right] \\
&\le\exp\left(\frac{(m-1)(p-1)}{2\rho^2}\|v_t\|_p^2\right)\cdot\mathbb{E}_z\left[\exp\left(\frac{m-1}{\rho^2}\langle\nabla\Phi(z),v_t\rangle\right)\left(\frac{\lvert\langle\nabla\Phi(z),w\rangle\rvert}{\rho^2}\right)^m\right]
\end{aligned} E z ∼ GG p ( 0 , ρ 2 ) [ exp ( ρ 2 ( m − 1 ) ( Φ ( z + v t ) − Φ ( z )) ) ( ρ 2 ∣⟨ ∇Φ ( z ) , w ⟩∣ ) m ] ≤ exp ( 2 ρ 2 ( m − 1 ) ( p − 1 ) ∥ v t ∥ p 2 ) ⋅ E z [ exp ( ρ 2 m − 1 ⟨ ∇Φ ( z ) , v t ⟩ ) ( ρ 2 ∣⟨ ∇Φ ( z ) , w ⟩∣ ) m ]
Letting
A = m − 1 ρ 2 ⟨ ∇ Φ ( z ) , v t ⟩ , X = ⟨ ∇ Φ ( z ) , w ⟩ ρ 2 A=\frac{m-1}{\rho^2}\langle\nabla\Phi(z),v_t\rangle,\quad X=\frac{\langle\nabla\Phi(z),w\rangle}{\rho^2} A = ρ 2 m − 1 ⟨ ∇Φ ( z ) , v t ⟩ , X = ρ 2 ⟨ ∇Φ ( z ) , w ⟩
we have that by Cauchy-Schwarz, E [ e A ∣ X ∣ m ] ≤ E [ e 2 A ] E [ ∣ X ∣ 2 m ] \mathbb{E}[e^A\lvert X\rvert^m]\le\sqrt{\mathbb{E}[e^{2A}]}\sqrt{\mathbb{E}[\lvert X\rvert^{2m}]} E [ e A ∣ X ∣ m ] ≤ E [ e 2 A ] E [∣ X ∣ 2 m ] . We bound the two terms separately. For the first term, using Fact 5, it holds that
E [ e 2 A ] = E z [ exp ( ⟨ ∇ Φ ( z ) , 2 ( m − 1 ) v t ⟩ ρ 2 ) ] ≤ exp ( 2 ( p − 1 ) ( m − 1 ) 2 ρ 2 ∥ v t ∥ p 2 ) ≤ exp ( 2 ( p − 1 ) ( m − 1 ) 2 m 2 ) \mathbb{E}[e^{2A}]=\mathbb{E}_z\left[\exp\left(\frac{\langle\nabla\Phi(z),2(m-1)v_t\rangle}{\rho^2}\right)\right]\le\exp\left(\frac{2(p-1)(m-1)^2}{\rho^2}\|v_t\|_p^2\right)\le\exp\left(\frac{2(p-1)(m-1)^2}{m^2}\right) E [ e 2 A ] = E z [ exp ( ρ 2 ⟨ ∇Φ ( z ) , 2 ( m − 1 ) v t ⟩ ) ] ≤ exp ( ρ 2 2 ( p − 1 ) ( m − 1 ) 2 ∥ v t ∥ p 2 ) ≤ exp ( m 2 2 ( p − 1 ) ( m − 1 ) 2 )
where we used ∥ v t ∥ p ≤ ( 1 − t ) ∥ v ′ ∥ p + t ∥ v ∥ p ≤ ρ / m \lVert v_t\rVert_p \le(1-t)\lVert v'\rVert_p+t\lVert v\rVert_p \le\rho/m ∥ v t ∥ p ≤ ( 1 − t ) ∥ v ′ ∥ p + t ∥ v ∥ p ≤ ρ / m . Therefore,
E [ e 2 A ] ≤ exp ( ( p − 1 ) ( m − 1 ) 2 m 2 ) \sqrt{\mathbb{E}[e^{2A}]} \le\exp\left(\frac{(p-1)(m-1)^2}{m^2}\right) E [ e 2 A ] ≤ exp ( m 2 ( p − 1 ) ( m − 1 ) 2 )
We now bound the second term. First, we show that X X X is sub-Gaussian. To prove this, we first prove that Y ( ξ ) = ⟨ ∇ Φ ( ξ ) , v − v ′ ⟩ Y(\xi)=\langle\nabla\Phi(\xi),v-v'\rangle Y ( ξ ) = ⟨ ∇Φ ( ξ ) , v − v ′ ⟩ is sub-Gaussian. First, since γ ρ , p \gamma_{\rho,p} γ ρ , p is even and ∇ Φ \nabla\Phi ∇Φ is odd, it can be shown that Y ( − ξ ) = − Y ( ξ ) Y(-\xi)=-Y(\xi) Y ( − ξ ) = − Y ( ξ ) , which means that E [ Y ( ξ ) ] = 0 \mathbb{E}[Y(\xi)]=0 E [ Y ( ξ )] = 0 . Again, letting w = v − v ′ w=v-v' w = v − v ′ , and using Fact 5 with u = λ ρ 2 w u=\lambda\rho^2w u = λ ρ 2 w for λ ∈ R \lambda\in\mathbb{R} λ ∈ R , it holds that
E ξ [ exp ( λ Y ) ] = E ξ [ exp ( 1 ρ 2 ⟨ ∇ Φ ( ξ ) , λ ρ 2 w ⟩ ) ] ≤ exp ( p − 1 2 ρ 2 λ 2 ρ 4 ∥ w ∥ p 2 ) \mathbb{E}_{\xi}[\exp(\lambda Y)]=\mathbb{E}_{\xi}\left[\exp\left(\frac{1}{\rho^2}\langle\nabla\Phi(\xi),\lambda\rho^2w\rangle\right)\right]\le\exp\left(\frac{p-1}{2\rho^2}\lambda^2\rho^4\lVert w\rVert_p^2\right) E ξ [ exp ( λY )] = E ξ [ exp ( ρ 2 1 ⟨ ∇Φ ( ξ ) , λ ρ 2 w ⟩ ) ] ≤ exp ( 2 ρ 2 p − 1 λ 2 ρ 4 ∥ w ∥ p 2 )
where we use the fact that ∥ w ∥ p ≤ 2 ρ / m \lVert w\rVert_p\le2\rho/m ∥ w ∥ p ≤ 2 ρ / m . By point 4 of proposition 2.5.2 of [43 ] , we have that Y Y Y is sub-gaussian with K 4 2 : = ( p − 1 ) ρ 2 ∥ w ∥ p 2 2 K_4^2:=\frac{(p-1)\rho^2\lVert w\rVert_p^2}{2} K 4 2 := 2 ( p − 1 ) ρ 2 ∥ w ∥ p 2 . By Markov’s inequality, we have that
Pr ( Y ≥ t ) = Pr ( e λ Y ≥ e λ t ) ≤ e − λ t E [ e λ Y ] ≤ exp ( K 4 2 λ 2 − λ t ) \Pr(Y\ge t)=\Pr(e^{\lambda Y}\ge e^{\lambda t})\le e^{-\lambda t}\mathbb{E}[e^{\lambda Y}]\le\exp(K_4^2\lambda^2-\lambda t) Pr ( Y ≥ t ) = Pr ( e λY ≥ e λ t ) ≤ e − λ t E [ e λY ] ≤ exp ( K 4 2 λ 2 − λ t )
Minimizing the exponent over λ ≥ 0 \lambda\ge0 λ ≥ 0 gives that Pr ( Y ≥ t ) ≤ exp ( − t 2 / 4 K 4 2 ) \Pr(Y\ge t)\le\exp(-t^2/4K_4^2) Pr ( Y ≥ t ) ≤ exp ( − t 2 /4 K 4 2 ) . Applying the same argument with − Y -Y − Y instead of Y Y Y , it holds that
Pr ( ∣ Y ∣ ≥ t ) ≤ 2 exp ( − t 2 4 K 4 2 ) : = 2 exp ( − t 2 K 1 2 ) \Pr(|Y|\ge t)\le2\exp\left(-\frac{t^2}{4K_4^2}\right):=2\exp\left(-\frac{t^2}{K_1^2}\right) Pr ( ∣ Y ∣ ≥ t ) ≤ 2 exp ( − 4 K 4 2 t 2 ) := 2 exp ( − K 1 2 t 2 )
so property 1 of proposition 2.5.2 of [43 ] holds with K 1 : = 2 K 4 K_1:=2K_4 K 1 := 2 K 4 . Now, it also holds that for all k ≥ 2 k\ge2 k ≥ 2 ,
E ∣ Y ∣ k ≤ ∫ 0 ∞ k t k − 1 Pr ( ∣ Y ∣ ≥ t ) d t ≤ ∫ 0 ∞ 2 k t k − 1 exp ( − t 2 K 1 2 ) d t = k K 1 k Γ ( k 2 ) ≤ k K 1 k ( k 2 ) k 2 \mathbb{E}|Y|^k\le\int_0^\infty kt^{k-1}\Pr(|Y|\ge t)\,dt\le\int_0^\infty2kt^{k-1}\exp\left(-\frac{t^2}{K_1^2}\right)dt=kK_1^k\Gamma\left(\frac{k}{2}\right)\le kK_1^k\left(\frac{k}{2}\right)^{\frac{k}{2}} E ∣ Y ∣ k ≤ ∫ 0 ∞ k t k − 1 Pr ( ∣ Y ∣ ≥ t ) d t ≤ ∫ 0 ∞ 2 k t k − 1 exp ( − K 1 2 t 2 ) d t = k K 1 k Γ ( 2 k ) ≤ k K 1 k ( 2 k ) 2 k
which implies that ( E ∣ Y ∣ k ) 1 / k ≤ K 1 2 k : = K 2 k (\mathbb{E}|Y|^k)^{1/k}\le K_1\sqrt{2k}:=K_2\sqrt{k} ( E ∣ Y ∣ k ) 1/ k ≤ K 1 2 k := K 2 k by property 2 of proposition 2.5.2 of [43 ] , so K 2 = K 1 2 = 2 K 4 2 K_2=K_1\sqrt{2}=2K_4\sqrt{2} K 2 = K 1 2 = 2 K 4 2 . By this property, it holds that
E ∣ X ∣ 2 m = E ∣ Y ∣ 2 m ρ 2 m ≤ ( K 2 2 m ) m ρ 2 m = ( 4 K 4 m ) m ρ 2 m ≤ ( 4 p − 1 ∥ v − v ′ ∥ p 2 m ) m 1 ρ 2 m = ( C p m ∥ v − v ′ ∥ p ρ ) m \sqrt{\mathbb{E}|X|^{2m}}=\frac{\sqrt{\mathbb{E}|Y|^{2m}}}{\rho^{2m}}\le\frac{(K_2\sqrt{2m})^m}{\rho^{2m}}=\frac{(4K_4\sqrt{m})^m}{\rho^{2m}}\le\left(\frac{4\sqrt{p-1}\lVert v-v'\rVert_p}{\sqrt{2}}\sqrt{m}\right)^m\frac{1}{\rho^{2m}}=\left(\frac{C_p\sqrt{m}\lVert v-v'\rVert_p}{\rho}\right)^m E ∣ X ∣ 2 m = ρ 2 m E ∣ Y ∣ 2 m ≤ ρ 2 m ( K 2 2 m ) m = ρ 2 m ( 4 K 4 m ) m ≤ ( 2 4 p − 1 ∥ v − v ′ ∥ p m ) m ρ 2 m 1 = ( ρ C p m ∥ v − v ′ ∥ p ) m
where C p = 2 2 ( p − 1 ) C_p=2\sqrt{2(p-1)} C p = 2 2 ( p − 1 ) . Therefore,
E [ e A ∣ X ∣ m ] ≤ exp ( ( p − 1 ) ( m − 1 ) 2 m 2 ) ( C p m ∥ w ∥ p ρ ) m . \mathbb{E}[e^{A}|X|^m]\le\exp\left(\frac{(p-1)(m-1)^2}{m^2}\right)\left(\frac{C_p\sqrt{m}\lVert w\rVert_p}{\rho}\right)^m. E [ e A ∣ X ∣ m ] ≤ exp ( m 2 ( p − 1 ) ( m − 1 ) 2 ) ( ρ C p m ∥ w ∥ p ) m .
Thus,
E z ∼ G p ( 0 , ρ 2 ) [ exp ( ( m − 1 ) ( Φ ( z + v t ) − Φ ( z ) ) ρ 2 ) ( ∣ ⟨ ∇ Φ ( z ) , w ⟩ ∣ ρ 2 ) m ] ≤ exp ( ( m − 1 ) ( p − 1 ) 2 ρ 2 ∥ v t ∥ p 2 ) ⋅ exp ( ( p − 1 ) ( m − 1 ) 2 m 2 ) ( C p m ∥ w ∥ p ρ ) m ≤ exp ( ( m − 1 ) ( p − 1 ) 2 m 2 ) exp ( ( p − 1 ) ( m − 1 ) 2 m 2 ) ( C p m ∥ w ∥ p ρ ) m = exp ( ( p − 1 ) ( m − 1 ) ( m − 1 2 ) m 2 ) ( C p m ∥ w ∥ p ρ ) m ≤ e p − 1 ( C p m ∥ v − v ′ ∥ p ρ ) m . \begin{aligned}
\mathbb{E}_{z\sim\mathcal{G}_p(0,\rho^2)}\left[\exp\left(\frac{(m-1)(\Phi(z+v_t)-\Phi(z))}{\rho^2}\right)\left(\frac{|\langle\nabla\Phi(z),w\rangle|}{\rho^2}\right)^m\right]
&\le\exp\left(\frac{(m-1)(p-1)}{2\rho^2}\lVert v_t\rVert_p^2\right)\cdot\exp\left(\frac{(p-1)(m-1)^2}{m^2}\right)\left(\frac{C_p\sqrt{m}\lVert w\rVert_p}{\rho}\right)^m\\
&\le\exp\left(\frac{(m-1)(p-1)}{2m^2}\right)\exp\left(\frac{(p-1)(m-1)^2}{m^2}\right)\left(\frac{C_p\sqrt{m}\lVert w\rVert_p}{\rho}\right)^m\\
&=\exp\left((p-1)\frac{(m-1)(m-\frac{1}{2})}{m^2}\right)\left(\frac{C_p\sqrt{m}\lVert w\rVert_p}{\rho}\right)^m\\
&\le e^{p-1}\left(\frac{C_p\sqrt{m}\lVert v-v'\rVert_p}{\rho}\right)^m.
\end{aligned} E z ∼ G p ( 0 , ρ 2 ) [ exp ( ρ 2 ( m − 1 ) ( Φ ( z + v t ) − Φ ( z )) ) ( ρ 2 ∣ ⟨ ∇Φ ( z ) , w ⟩ ∣ ) m ] ≤ exp ( 2 ρ 2 ( m − 1 ) ( p − 1 ) ∥ v t ∥ p 2 ) ⋅ exp ( m 2 ( p − 1 ) ( m − 1 ) 2 ) ( ρ C p m ∥ w ∥ p ) m ≤ exp ( 2 m 2 ( m − 1 ) ( p − 1 ) ) exp ( m 2 ( p − 1 ) ( m − 1 ) 2 ) ( ρ C p m ∥ w ∥ p ) m = exp ( ( p − 1 ) m 2 ( m − 1 ) ( m − 2 1 ) ) ( ρ C p m ∥ w ∥ p ) m ≤ e p − 1 ( ρ C p m ∥ v − v ′ ∥ p ) m .
Integrating over t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] completes the proof.
ℓ p \ell_p ℓ p Stochastic Bias-Reduced Methods
In this section, we introduce a class of ℓ p \ell_p ℓ p -optimal-distance-convergence (ODCp _p p ) algorithms, detailing the required convergence guarantee and giving a concrete algorithm for computing a low-bias estimator of an order-p p p uniformly convex function F F F . We then use these algorithms in conjunction with multilevel Monte Carlo (MLMC) methods to use them for various applications. We assume that F F F can be represented in the form F = f + ψ F=f+\psi F = f + ψ , where ψ \psi ψ is a simple function and f f f satisfies Assumption 1.
Assumption 1. The function f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R is convex, where X \mathcal{X} X is a closed convex set. We assume that there is an unbiased subgradient estimator ∇ ^ f \widehat{\nabla}f ∇ f that has bounded q q q th-moment E [ ∥ ∇ ^ f ( x ) ∥ q q ] ≤ G q \mathbb{E}[\|\widehat{\nabla}f(x)\|_q^q]\leq G^q E [ ∥ ∇ f ( x ) ∥ q q ] ≤ G q for all x ∈ X x\in\mathcal{X} x ∈ X .
ODCp _p p algorithms
Our estimator uses an arbitrary optimization routine for F F F as a black-box component, provided that it achieves a sufficiently fast convergence rate toward argmin x ∈ X F ( x ) \operatorname*{argmin}_{x\in\mathcal{X}}F(x) argmin x ∈ X F ( x ) . We describe the required convergence property below.
Definition 3 (ODCp _p p algorithm). For p ≥ 2 p\geq2 p ≥ 2 , an ℓ p \ell_p ℓ p * -optimal-distance-convergence (ODCp _p p ) algorithm takes as input an unbiased stochastic subgradient oracle* ∇ ^ f \widehat{\nabla}f ∇ f for a function f f f satisfying Assumption 1, a simple function ψ \psi ψ , and an oracle budget T ≥ 1 T\geq1 T ≥ 1 . Suppose that F = f + ψ F=f+\psi F = f + ψ is μ \mu μ * -uniformly convex of order* p p p with minimizer x ∗ x^* x ∗ . The algorithm returns a point x x x using at most T T T oracle calls, satisfying
E [ ∥ x − x ∗ ∥ p p ] ≤ c G q μ q T 1 / ( p − 1 ) . \mathbb{E}\left[\|x-x^*\|_p^p\right]\leq\frac{cG^q}{\mu^qT^{1/(p-1)}}. E [ ∥ x − x ∗ ∥ p p ] ≤ μ q T 1/ ( p − 1 ) c G q .
where c > 0 c>0 c > 0 is a constant.
As in the work of [2 ] , we leverage the multilevel Monte Carlo (MLMC) technique of Blanchet and Glynn [7 ] to construct a low-bias, bounded-dual-moment estimator of the ℓ p \ell_p ℓ p -Moreau gradient from an ODCp _p p algorithm, where we define the ℓ p \ell_p ℓ p -Moreau envelope to be
f p , λ ( y ) : = min x { f ( x ) + λ p ∥ x − y ∥ p p } . f_{p,\lambda}(y):=\min_x\left\{f(x)+\frac{\lambda}{p}\|x-y\|_p^p\right\}. f p , λ ( y ) := x min { f ( x ) + p λ ∥ x − y ∥ p p } .
For p > 2 p>2 p > 2 , however, the map from the proximal point to the Moreau gradient is nonlinear. We therefore apply the MLMC construction directly to the Moreau-gradient approximations, rather than first constructing a low-bias estimator of the proximal point. Fix y ∈ X y\in\mathcal{X} y ∈ X and λ > 0 \lambda>0 λ > 0 , and define ψ ( x ) : = λ p ∥ x − y ∥ p p \psi(x):=\frac{\lambda}{p}\|x-y\|_p^p ψ ( x ) := p λ ∥ x − y ∥ p p . For every j ≥ 0 j\geq0 j ≥ 0 , let x j : = ODC p ( ∇ ^ f , ψ , 2 j ) x_j:=\operatorname{ODC}_p(\widehat{\nabla}f,\psi,2^j) x j := ODC p ( ∇ f , ψ , 2 j ) , and define the corresponding approximate Moreau gradient
g j : = λ J p ( y − x j ) , J p ( z ) : = ( ∣ z k ∣ p − 2 z k ) k = 1 d . g_j:=\lambda J_p(y-x_j),\qquad J_p(z):=\left(|z_k|^{p-2}z_k\right)_{k=1}^d. g j := λ J p ( y − x j ) , J p ( z ) := ( ∣ z k ∣ p − 2 z k ) k = 1 d .
Let P ( J = j ) = 2 − j \mathbb{P}(J=j)=2^{-j} P ( J = j ) = 2 − j for j ∈ { 1 , 2 , … } j\in\{1,2,\ldots\} j ∈ { 1 , 2 , … } , and define
g ′ : = g 0 + { 2 J ( g J − g J − 1 ) , J ≤ j max , 0 , J > j max . (2) g':=g_0+
\begin{cases}
2^J(g_J-g_{J-1}), & J\leq j_{\max},\\
0, & J>j_{\max}.
\end{cases} \tag*{(2)} g ′ := g 0 + { 2 J ( g J − g J − 1 ) , 0 , J ≤ j m a x , J > j m a x . (2)
where j max = ⌊ log 2 T max ⌋ j_{\max}=\lfloor\log_2 T_{\max}\rfloor j m a x = ⌊ log 2 T m a x ⌋ .
Algorithm 1 ℓ p \ell_p ℓ p -MorGradEst
Input: ∇ ^ f , ψ , λ , δ , σ q , X \widehat{\nabla} f,\psi,\lambda,\delta,\sigma^q,\mathcal{X} ∇ f , ψ , λ , δ , σ q , X
Parameters: δ \delta δ : desired bias, σ q \sigma^q σ q : desired q q q -th moment error
T max = ⌈ ( 2 C 1 G min { δ , σ q } ) p ( p − 1 ) ⌉ T_{\max}=\left\lceil\left(\frac{2C_1G}{\min\{\delta,\sigma^q\}}\right)^{p(p-1)}\right\rceil T m a x = ⌈ ( m i n { δ , σ q } 2 C 1 G ) p ( p − 1 ) ⌉ (see (5) ) and N = ⌈ 2 p C 3 q G p σ q p T max p − 2 p − 1 ⌉ N=\left\lceil2^p C_3^q\frac{G^p}{\sigma^p_q}T_{\max}^{\frac{p-2}{p-1}}\right\rceil N = ⌈ 2 p C 3 q σ q p G p T m a x p − 1 p − 2 ⌉ (see (6) )
for i = 1 , … , N i=1,\ldots,N i = 1 , … , N do
g ^ ( i ) ← \widehat{g}^{(i)}\leftarrow g ( i ) ← draw of estimator (1)
end for
return ∇ ^ f p , λ ( y ) = 1 N ∑ i = 1 N g ^ ( i ) \widehat{\nabla}f_{p,\lambda}(y)=\frac{1}{N}\sum_{i=1}^{N}\widehat{g}^{(i)} ∇ f p , λ ( y ) = N 1 ∑ i = 1 N g ( i )
Lemma 3. Let f f f and ∇ ^ f \widehat{\nabla}f ∇ f satisfy Assumption 1, let y ∈ X y\in\mathcal{X} y ∈ X , p > 2 p>2 p > 2 , and let λ > 0 \lambda>0 λ > 0 . Let ψ ( x ) = λ p ∥ x − y ∥ p p \psi(x)=\frac{\lambda}{p}\lVert x-y\rVert_p^p ψ ( x ) = p λ ∥ x − y ∥ p p , suppose F = f + ψ F=f+\psi F = f + ψ has minimizer x ∗ x_* x ∗ over X \mathcal{X} X , and let T max ∈ N T_{\max}\in\mathbb{N} T m a x ∈ N with T max ≥ 1 T_{\max}\geq1 T m a x ≥ 1 . For any ODCp _p p algorithm on X \mathcal{X} X with constant c c c , let g ′ g' g ′ be the estimator in (1), where J J J is independent of all oracle randomness. The following statements hold, for some constant C p > 0 C_p>0 C p > 0 depending only on p p p and c c c :
∥ E g ′ − g ∗ ∥ q ≤ C p G T max − 1 p ( p − 1 ) and E ∥ g ′ − E g ′ ∥ q q ≤ C p G q T max − p − 2 p − 1 , \left\lVert\mathbb{E}g'-g_*\right\rVert_q\leq C_pGT_{\max}^{-\frac{1}{p(p-1)}}\qquad\text{and}\qquad\mathbb{E}\left\lVert g'-\mathbb{E}g'\right\rVert_q^q\leq C_pG^qT_{\max}^{-\frac{p-2}{p-1}}, ∥ E g ′ − g ∗ ∥ q ≤ C p G T m a x − p ( p − 1 ) 1 and E ∥ g ′ − E g ′ ∥ q q ≤ C p G q T m a x − p − 1 p − 2 ,
where g ∗ = λ J p ( y − x ∗ ) = ∇ f p , λ ( y ) g_*=\lambda J_p(y-x_*)=\nabla f_{p,\lambda}(y) g ∗ = λ J p ( y − x ∗ ) = ∇ f p , λ ( y ) . Moreover, the expected number of stochastic-gradient evaluations required to compute g ′ g' g ′ is O ( 1 + log T max ) O(1+\log T_{\max}) O ( 1 + log T m a x ) .
Proof. Since ψ ( y ) = 0 \psi(y)=0 ψ ( y ) = 0 and x ∗ x_* x ∗ is a minimizer over X \mathcal{X} X ,
f ( x ∗ ) + λ p ∥ x ∗ − y ∥ p p ≤ f ( y ) . f(x_*)+\frac{\lambda}{p}\lVert x_*-y\rVert_p^p\leq f(y). f ( x ∗ ) + p λ ∥ x ∗ − y ∥ p p ≤ f ( y ) .
By Jensen’s inequality, it holds that
∥ E ∇ ^ f ( x ) ∥ q q ≤ E ∥ ∇ ^ f ( x ) ∥ q q ≤ G q . \left\lVert\mathbb{E}\widehat{\nabla}f(x)\right\rVert_q^q\leq\mathbb{E}\left\lVert\widehat{\nabla}f(x)\right\rVert_q^q\leq G^q. E ∇ f ( x ) q q ≤ E ∇ f ( x ) q q ≤ G q .
Since E ∇ ^ f ( x ) ∈ ∂ f ( x ) \mathbb{E}\widehat{\nabla}f(x)\in\partial f(x) E ∇ f ( x ) ∈ ∂ f ( x ) , this implies that f f f is G G G -Lipschitz with respect to ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p on X \mathcal{X} X . Moreover, by convexity and Hölder’s inequality,
f ( y ) − f ( x ∗ ) ≤ ⟨ E ∇ ^ f ( y ) , y − x ∗ ⟩ ≤ ∥ E ∇ ^ f ( y ) ∥ q ∥ y − x ∗ ∥ p ≤ G ∥ y − x ∗ ∥ p . f(y)-f(x_*)\leq\left\langle\mathbb{E}\widehat{\nabla}f(y),y-x_*\right\rangle\leq\left\lVert\mathbb{E}\widehat{\nabla}f(y)\right\rVert_q\lVert y-x_*\rVert_p\leq G\lVert y-x_*\rVert_p. f ( y ) − f ( x ∗ ) ≤ ⟨ E ∇ f ( y ) , y − x ∗ ⟩ ≤ E ∇ f ( y ) q ∥ y − x ∗ ∥ p ≤ G ∥ y − x ∗ ∥ p .
Therefore,
λ p ∥ x ∗ − y ∥ p p ≤ G ∥ x ∗ − y ∥ p , \frac{\lambda}{p}\lVert x_*-y\rVert_p^p\leq G\lVert x_*-y\rVert_p, p λ ∥ x ∗ − y ∥ p p ≤ G ∥ x ∗ − y ∥ p ,
which implies that
∥ y − x ∗ ∥ p ≤ p 1 p − 1 ( G λ ) 1 p − 1 . \lVert y-x_*\rVert_p\leq p^{\frac{1}{p-1}}\left(\frac{G}{\lambda}\right)^{\frac{1}{p-1}}. ∥ y − x ∗ ∥ p ≤ p p − 1 1 ( λ G ) p − 1 1 .
Since F F F is 2 2 − p λ 2^{2-p}\lambda 2 2 − p λ -uniformly convex of order p p p , the ODC guarantee gives
E ∥ x j − x ∗ ∥ p p ≤ c ( G 2 2 − p λ ) q 2 − j p − 1 = c 2 ( p − 2 ) q ( G λ ) q 2 − j p − 1 . \mathbb{E}\lVert x_j-x_*\rVert_p^p\leq c\left(\frac{G}{2^{2-p}\lambda}\right)^q2^{-\frac{j}{p-1}}=c2^{(p-2)q}\left(\frac{G}{\lambda}\right)^q2^{-\frac{j}{p-1}}. E ∥ x j − x ∗ ∥ p p ≤ c ( 2 2 − p λ G ) q 2 − p − 1 j = c 2 ( p − 2 ) q ( λ G ) q 2 − p − 1 j .
By Lyapunov’s inequality, for 0 < s ≤ p 0<s\leq p 0 < s ≤ p , it holds that
E ∥ x j − x ∗ ∥ p s ≤ ( E ∥ x j − x ∗ ∥ p p ) s / p ≤ C p ( G λ ) s p − 1 2 − j s p ( p − 1 ) . \mathbb{E}\lVert x_j-x_*\rVert_p^s\leq\left(\mathbb{E}\lVert x_j-x_*\rVert_p^p\right)^{s/p}\leq C_p\left(\frac{G}{\lambda}\right)^{\frac{s}{p-1}}2^{-\frac{js}{p(p-1)}}. E ∥ x j − x ∗ ∥ p s ≤ ( E ∥ x j − x ∗ ∥ p p ) s / p ≤ C p ( λ G ) p − 1 s 2 − p ( p − 1 ) j s .
Moreover, it holds that
∥ J p ( u ) − J p ( v ) ∥ q ≤ ( p − 1 ) ( ∥ u ∥ p + ∥ v ∥ p ) p − 2 ∥ u − v ∥ p \lVert J_p(u)-J_p(v)\rVert_q \le(p-1)(\lVert u\rVert_p+\lVert v\rVert_p)^{p-2}\lVert u-v\rVert_p ∥ J p ( u ) − J p ( v ) ∥ q ≤ ( p − 1 ) (∥ u ∥ p + ∥ v ∥ p ) p − 2 ∥ u − v ∥ p
for all u , v ∈ R d u,v\in\mathbb{R}^d u , v ∈ R d . We now apply this inequality with u = y − x j u=y-x_j u = y − x j and v = y − x ∗ v=y-x_* v = y − x ∗ . Then,
∥ y − x ∗ ∥ p + ∥ y − x j ∥ p ≤ 2 ∥ y − x ∗ ∥ p + ∥ x j − x ∗ ∥ p \lVert y-x_*\rVert_p+\lVert y-x_j\rVert_p\le2\lVert y-x_*\rVert_p+\lVert x_j-x_*\rVert_p ∥ y − x ∗ ∥ p + ∥ y − x j ∥ p ≤ 2 ∥ y − x ∗ ∥ p + ∥ x j − x ∗ ∥ p
≤ 2 p 1 / ( p − 1 ) ( G λ ) 1 / ( p − 1 ) + ∥ x j − x ∗ ∥ p . \le2p^{1/(p-1)}\left(\frac{G}{\lambda}\right)^{1/(p-1)}+\lVert x_j-x_*\rVert_p. ≤ 2 p 1/ ( p − 1 ) ( λ G ) 1/ ( p − 1 ) + ∥ x j − x ∗ ∥ p .
Using that ( s + t ) p − 2 ≤ C p ( s p − 2 + t p − 2 ) (s+t)^{p-2}\le C_p(s^{p-2}+t^{p-2}) ( s + t ) p − 2 ≤ C p ( s p − 2 + t p − 2 ) for nonnegative s , t s,t s , t and some constant C p C_p C p , we have
∥ g j − g ∗ ∥ q ≤ C p λ [ ( G λ ) p − 2 p − 1 ∥ x j − x ∗ ∥ p + ∥ x j − x ∗ ∥ p p − 1 ] . (3) \lVert g_j-g_*\rVert_q\le C_p\lambda\left[\left(\frac{G}{\lambda}\right)^{\frac{p-2}{p-1}}\lVert x_j-x_*\rVert_p+\lVert x_j-x_*\rVert_p^{p-1}\right]. \tag*{(3)} ∥ g j − g ∗ ∥ q ≤ C p λ [ ( λ G ) p − 1 p − 2 ∥ x j − x ∗ ∥ p + ∥ x j − x ∗ ∥ p p − 1 ] . (3)
Taking expectations in (3) and using the Lyapunov bound with s = 1 s=1 s = 1 and s = p − 1 s=p-1 s = p − 1 yields
∥ E g j − g ∗ ∥ q ≤ E ∥ g j − g ∗ ∥ q ≤ C p G ( 2 − j p ( p − 1 ) + 2 − j p ) ≤ C p G 2 − j p ( p − 1 ) , (4) \lVert\mathbb{E}g_j-g_*\rVert_q\le\mathbb{E}\lVert g_j-g_*\rVert_q\le C_pG\left(2^{-\frac{j}{p(p-1)}}+2^{-\frac{j}{p}}\right)\le C_pG2^{-\frac{j}{p(p-1)}}, \tag*{(4)} ∥ E g j − g ∗ ∥ q ≤ E ∥ g j − g ∗ ∥ q ≤ C p G ( 2 − p ( p − 1 ) j + 2 − p j ) ≤ C p G 2 − p ( p − 1 ) j , (4)
where we use 2 − j / p ≤ 2 − j / [ p ( p − 1 ) ] 2^{-j/p}\le2^{-j/[p(p-1)]} 2 − j / p ≤ 2 − j / [ p ( p − 1 )] for j ≥ 0 j\ge0 j ≥ 0 and p > 2 p>2 p > 2 . Moreover, it holds that
∥ g j − g ∗ ∥ q q ≤ C p λ q [ ( G λ ) ( p − 2 ) q p − 1 ∥ x j − x ∗ ∥ p q + ∥ x j − x ∗ ∥ p p ] , \lVert g_j-g_*\rVert_q^q\le C_p\lambda^q\left[\left(\frac{G}{\lambda}\right)^{\frac{(p-2)q}{p-1}}\lVert x_j-x_*\rVert_p^q+\lVert x_j-x_*\rVert_p^{p}\right], ∥ g j − g ∗ ∥ q q ≤ C p λ q ( λ G ) p − 1 ( p − 2 ) q ∥ x j − x ∗ ∥ p q + ∥ x j − x ∗ ∥ p p ,
where we used ( p − 1 ) q = p (p-1)q=p ( p − 1 ) q = p . Absorbing constants into C p C_p C p and taking expectations yields
E ∥ g j − g ∗ ∥ q q ≤ C p G q ( 2 − j ( p − 1 ) 2 + 2 − j p − 1 ) ≤ C p G q 2 − j ( p − 1 ) 2 , \mathbb{E}\lVert g_j-g_*\rVert_q^q\le C_pG^q\left(2^{-\frac{j}{(p-1)^2}}+2^{-\frac{j}{p-1}}\right)\le C_pG^q2^{-\frac{j}{(p-1)^2}}, E ∥ g j − g ∗ ∥ q q ≤ C p G q ( 2 − ( p − 1 ) 2 j + 2 − p − 1 j ) ≤ C p G q 2 − ( p − 1 ) 2 j ,
because p > 2 p>2 p > 2 implies 2 − j / ( p − 1 ) ≤ 2 − j / ( p − 1 ) 2 2^{-j/(p-1)}\le2^{-j/(p-1)^2} 2 − j / ( p − 1 ) ≤ 2 − j / ( p − 1 ) 2 for j ≥ 0 j\ge0 j ≥ 0 . Therefore, for j ≥ 1 j\ge1 j ≥ 1 ,
E ∥ g j − g j − 1 ∥ q q ≤ 2 q − 1 ( E ∥ g j − g ∗ ∥ q q + E ∥ g j − 1 − g ∗ ∥ q q ) ≤ C p G q 2 − j ( p − 1 ) 2 . \mathbb{E}\lVert g_j-g_{j-1}\rVert_q^q\le2^{q-1}\left(\mathbb{E}\lVert g_j-g_*\rVert_q^q+\mathbb{E}\lVert g_{j-1}-g_*\rVert_q^q\right)\le C_pG^q2^{-\frac{j}{(p-1)^2}}. E ∥ g j − g j − 1 ∥ q q ≤ 2 q − 1 ( E ∥ g j − g ∗ ∥ q q + E ∥ g j − 1 − g ∗ ∥ q q ) ≤ C p G q 2 − ( p − 1 ) 2 j .
On the other hand,
E g ′ = E g 0 + ∑ j = 1 j max 2 − j E ( g j − g j − 1 ) = E g j max . \mathbb{E}g'=\mathbb{E}g_0+\sum_{j=1}^{j_{\max}}2^{-j}\mathbb{E}(g_j-g_{j-1})=\mathbb{E}g_{j_{\max}}. E g ′ = E g 0 + j = 1 ∑ j m a x 2 − j E ( g j − g j − 1 ) = E g j m a x .
Since 2 j max > T max / 2 2^{j_{\max}}>T_{\max}/2 2 j m a x > T m a x /2 , combining this with (4) implies that
∥ E g ′ − g ∗ ∥ q ≤ C p G T max − 1 p ( p − 1 ) . \lVert\mathbb{E}g'-g_*\rVert_q\le C_pGT_{\max}^{-\frac{1}{p(p-1)}}. ∥ E g ′ − g ∗ ∥ q ≤ C p G T m a x − p ( p − 1 ) 1 .
Since
g ′ − g ∗ = ( g 0 − g ∗ ) + 1 [ J ≤ j max ] 2 J ( g J − g J − 1 ) , g'-g_*=(g_0-g_*)+\mathbf{1}[J\le j_{\max}]2^J(g_J-g_{J-1}), g ′ − g ∗ = ( g 0 − g ∗ ) + 1 [ J ≤ j m a x ] 2 J ( g J − g J − 1 ) ,
it holds that
E ∥ g ′ − g ∗ ∥ q q ≤ 2 q − 1 E ∥ g 0 − g ∗ ∥ q q + 2 q − 1 E ∥ 1 [ J ≤ j max ] 2 J ( g J − g J − 1 ) ∥ q q \mathbb{E}\lVert g'-g_*\rVert_q^q\le2^{q-1}\mathbb{E}\lVert g_0-g_*\rVert_q^q+2^{q-1}\mathbb{E}\left\lVert\mathbf{1}[J\le j_{\max}]2^J(g_J-g_{J-1})\right\rVert_q^q E ∥ g ′ − g ∗ ∥ q q ≤ 2 q − 1 E ∥ g 0 − g ∗ ∥ q q + 2 q − 1 E 1 [ J ≤ j m a x ] 2 J ( g J − g J − 1 ) q q
≤ 2 q − 1 E ∥ g 0 − g ∗ ∥ q q + 2 q − 1 ∑ j = 1 j max 2 − j 2 q j E ∥ g j − g j − 1 ∥ q q ≤ C p G q ( 1 + ∑ j = 1 j max 2 ( 1 p − 1 − 1 ( p − 1 ) 2 ) j ) = C p G q ( 1 + ∑ j = 1 j max 2 ( p − 2 ) j ( p − 1 ) 2 ) \begin{aligned}
\le2^{q-1}\mathbb{E}\lVert g_0-g_*\rVert_q^q+2^{q-1}\sum_{j=1}^{j_{\max}}2^{-j}2^{qj}\mathbb{E}\lVert g_j-g_{j-1}\rVert_q^q \\
&\le C_pG^q\left(1+\sum_{j=1}^{j_{\max}}2^{\left(\frac{1}{p-1}-\frac{1}{(p-1)^2}\right)j}\right)=C_pG^q\left(1+\sum_{j=1}^{j_{\max}}2^{\frac{(p-2)j}{(p-1)^2}}\right)
\end{aligned} ≤ 2 q − 1 E ∥ g 0 − g ∗ ∥ q q + 2 q − 1 j = 1 ∑ j m a x 2 − j 2 q j E ∥ g j − g j − 1 ∥ q q ≤ C p G q ( 1 + j = 1 ∑ j m a x 2 ( p − 1 1 − ( p − 1 ) 2 1 ) j ) = C p G q ( 1 + j = 1 ∑ j m a x 2 ( p − 1 ) 2 ( p − 2 ) j )
where the second inequality follows by the law of total expectation and the independence of J J J from the oracle randomness. Since p > 2 p > 2 p > 2 , the geometric sum gives
E ∥ g ′ − g ∗ ∥ q q ≤ C p G q T max p − 2 ( p − 1 ) 2 . \mathbb{E}\|g' - g_*\|_q^q \le C_pG^qT_{\max}^{\frac{p-2}{(p-1)^2}}. E ∥ g ′ − g ∗ ∥ q q ≤ C p G q T m a x ( p − 1 ) 2 p − 2 .
Therefore,
E ∥ g ′ − E g ′ ∥ q q ≤ 2 q − 1 E ∥ g ′ − g ∗ ∥ q q + 2 q − 1 ∥ E g ′ − g ∗ ∥ q q ≤ C p G q ( T max p − 2 ( p − 1 ) 2 + T max q p ( p − 1 ) ) ≤ C p G q T max p − 2 ( p − 1 ) 2 . \begin{aligned}
\mathbb{E}\|g' - \mathbb{E}g'\|_q^q \le2^{q-1}\mathbb{E}\|g' - g_*\|_q^q + 2^{q-1}\|\mathbb{E}g' - g_*\|_q^q \le C_pG^q\left(T_{\max}^{\frac{p-2}{(p-1)^2}}+T_{\max}^{\frac{q}{p(p-1)}}\right) \\
&\le C_pG^qT_{\max}^{\frac{p-2}{(p-1)^2}}.
\end{aligned} E ∥ g ′ − E g ′ ∥ q q ≤ 2 q − 1 E ∥ g ′ − g ∗ ∥ q q + 2 q − 1 ∥ E g ′ − g ∗ ∥ q q ≤ C p G q ( T m a x ( p − 1 ) 2 p − 2 + T m a x p ( p − 1 ) q ) ≤ C p G q T m a x ( p − 1 ) 2 p − 2 .
Moreover, the expected number of stochastic-gradient evaluations is at most
1 + ∑ j = 1 j max 2 − j ( 2 j + 2 j − 1 ) = 1 + 3 2 j max = O ( 1 + log T max ) , 1+\sum_{j=1}^{j_{\max}}2^{-j}(2^j+2^{j-1})=1+\frac{3}{2}j_{\max}=O(1+\log T_{\max}), 1 + j = 1 ∑ j m a x 2 − j ( 2 j + 2 j − 1 ) = 1 + 2 3 j m a x = O ( 1 + log T m a x ) ,
since level j j j is drawn with probability 2 − j 2^{-j} 2 − j and requires at most 2 j + 2 j − 1 2^j+2^{j-1} 2 j + 2 j − 1 oracle calls, while the base term g 0 g_0 g 0 requires at most one oracle call. This finishes the proof.
Theorem 1. Let f f f and ∇ ^ f \widehat{\nabla}f ∇ f satisfy Assumption 1, let y ∈ X y \in\mathcal{X} y ∈ X , and let λ , δ , σ > 0 \lambda,\delta,\sigma> 0 λ , δ , σ > 0 and p > 2 p > 2 p > 2 . The procedure
ℓ p -MorGradEst ( ∇ ^ f , y , λ , δ , σ q , X ) \ell_p\text{-MorGradEst}(\widehat{\nabla}f,y,\lambda,\delta,\sigma^q,\mathcal{X}) ℓ p -MorGradEst ( ∇ f , y , λ , δ , σ q , X ) outputs an estimator ∇ ^ f p , λ ( y ) \widehat{\nabla}f_{p,\lambda}(y) ∇ f p , λ ( y ) satisfying
∥ E ∇ ^ f p , λ ( y ) − ∇ f p , λ ( y ) ∥ q ≤ δ and E ∥ ∇ ^ f p , λ ( y ) − ∇ f p , λ ( y ) ∥ q q ≤ σ q , \|\mathbb{E}\widehat{\nabla}f_{p,\lambda}(y)-\nabla f_{p,\lambda}(y)\|_q \le\delta\quad\text{and} \quad\mathbb{E}\|\widehat{\nabla}f_{p,\lambda}(y)-\nabla f_{p,\lambda}(y)\|_q^q \le\sigma^q, ∥ E ∇ f p , λ ( y ) − ∇ f p , λ ( y ) ∥ q ≤ δ and E ∥ ∇ f p , λ ( y ) − ∇ f p , λ ( y ) ∥ q q ≤ σ q , using N ∇ ^ f N_{\widehat{\nabla}f} N ∇ f stochastic-gradient computations, where
E N ∇ ^ f = O ( [ 1 + G p σ p ( max { 1 , G min { δ , σ } } ) p ( p − 2 ) ] [ 1 + log max { 1 , G min { δ , σ } } ] ) . \mathbb{E}N_{\widehat{\nabla}f}=O\left(\left[1+\frac{G^p}{\sigma^p}\left(\max\left\{1,\frac{G}{\min\{\delta,\sigma\}}\right\}\right)^{p(p-2)}\right]\left[1+\log\max\left\{1,\frac{G}{\min\{\delta,\sigma\}}\right\}\right]\right). E N ∇ f = O ( [ 1 + σ p G p ( max { 1 , min { δ , σ } G } ) p ( p − 2 ) ] [ 1 + log max { 1 , min { δ , σ } G } ] ) .
Proof. Write the algorithm’s output as ∇ ^ f p , λ ( y ) = 1 N ∑ i = 1 N g ′ ( i ) \widehat{\nabla}f_{p,\lambda}(y)=\frac{1}{N}\sum_{i=1}^{N}g'^{(i)} ∇ f p , λ ( y ) = N 1 ∑ i = 1 N g ′ ( i ) , with g ′ ( i ) g'^{(i)} g ′ ( i ) i.i.d. copies of the MLMC estimator. Note that from Lemma 3, we have
∥ E g ′ − ∇ f p , λ ( y ) ∥ q ≤ C 1 G T max − 1 p ( p − 1 ) , E ∥ g ′ − E g ′ ∥ q q ≤ C 2 G q T max p − 2 ( p − 1 ) 2 . (5) \|\mathbb{E}g'-\nabla f_{p,\lambda}(y)\|_q\le C_1GT_{\max}^{-\frac{1}{p(p-1)}},\qquad\mathbb{E}\|g'-\mathbb{E}g'\|_q^q\le C_2G^qT_{\max}^{\frac{p-2}{(p-1)^2}}. \tag*{(5)} ∥ E g ′ − ∇ f p , λ ( y ) ∥ q ≤ C 1 G T m a x − p ( p − 1 ) 1 , E ∥ g ′ − E g ′ ∥ q q ≤ C 2 G q T m a x ( p − 1 ) 2 p − 2 . (5)
Since T max ≥ ( 2 C 1 G min { δ , σ } ) p ( p − 1 ) T_{\max}\ge\left(\frac{2C_1G}{\min\{\delta,\sigma\}}\right)^{p(p-1)} T m a x ≥ ( m i n { δ , σ } 2 C 1 G ) p ( p − 1 ) , it holds that
∥ E g ′ − ∇ f p , λ ( y ) ∥ q ≤ min { δ , σ } 2 . \|\mathbb{E}g'-\nabla f_{p,\lambda}(y)\|_q\le\frac{\min\{\delta,\sigma\}}{2}. ∥ E g ′ − ∇ f p , λ ( y ) ∥ q ≤ 2 min { δ , σ } .
Since E ∇ ^ f p , λ ( y ) = E g ′ \mathbb{E}\widehat{\nabla}f_{p,\lambda}(y)=\mathbb{E}g' E ∇ f p , λ ( y ) = E g ′ , it holds that
∥ E ∇ ^ f p , λ ( y ) − ∇ f p , λ ( y ) ∥ q ≤ min { δ , σ } 2 ≤ δ 2 ≤ δ . \|\mathbb{E}\widehat{\nabla}f_{p,\lambda}(y)-\nabla f_{p,\lambda}(y)\|_q\le\frac{\min\{\delta,\sigma\}}{2}\le\frac{\delta}{2}\le\delta. ∥ E ∇ f p , λ ( y ) − ∇ f p , λ ( y ) ∥ q ≤ 2 min { δ , σ } ≤ 2 δ ≤ δ .
The random vectors g ( i ) − E g ( i ) g^{(i)}-\mathbb{E}g^{(i)} g ( i ) − E g ( i ) are i.i.d. and centered. Since 1 < q ≤ 2 1<q\le2 1 < q ≤ 2 , the von Bahr–Esseen inequality applied coordinatewise, together with the lemma, gives, for some constant C v B E C_{\mathrm{vBE}} C vBE ,
E ∥ ∥ 1 N ∑ i = 1 N ( g ( i ) − E g ( i ) ) ∥ ∥ q q ≤ C v B E N q − 1 E ∥ g ′ − E g ′ ∥ q q ≤ C v B E C 2 G q T max p − 2 ( p − 1 ) 2 N q − 1 . \mathbb{E}\left\|\left\|\frac{1}{N}\sum_{i=1}^{N}(g^{(i)}-\mathbb{E}g^{(i)})\right\|\right\|_{q}^{q}\le\frac{C_{\mathrm{vBE}}}{N^{q-1}}\mathbb{E}\|g'-\mathbb{E}g'\|_{q}^{q}\le\frac{C_{\mathrm{vBE}}C_{2}G^{q}T_{\max}^{\frac{p-2}{(p-1)^{2}}}}{N^{q-1}}. E N 1 i = 1 ∑ N ( g ( i ) − E g ( i ) ) q q ≤ N q − 1 C vBE E ∥ g ′ − E g ′ ∥ q q ≤ N q − 1 C vBE C 2 G q T m a x ( p − 1 ) 2 p − 2 .
Since
N ≥ 2 p ( C v B E C 2 ) p − 1 G p σ p T max p − 2 p − 1 = 2 p C 3 G p σ p T max p − 2 p − 1 , C 3 : = ( C v B E C 2 ) p − 1 , (6) N\ge2^{p}(C_{\mathrm{vBE}}C_{2})^{p-1}\frac{G^{p}}{\sigma^{p}}T_{\max}^{\frac{p-2}{p-1}}=2^{p}C_{3}\frac{G^{p}}{\sigma^{p}}T_{\max}^{\frac{p-2}{p-1}},\qquad C_{3}:=(C_{\mathrm{vBE}}C_{2})^{p-1}, \tag*{(6)} N ≥ 2 p ( C vBE C 2 ) p − 1 σ p G p T m a x p − 1 p − 2 = 2 p C 3 σ p G p T m a x p − 1 p − 2 , C 3 := ( C vBE C 2 ) p − 1 , (6)
it holds that
C v B E C 2 G q T max p − 2 ( p − 1 ) 2 N q − 1 ≤ ( σ 2 ) q . \frac{C_{\mathrm{vBE}}C_{2}G^{q}T_{\max}^{\frac{p-2}{(p-1)^{2}}}}{N^{q-1}}\le\left(\frac{\sigma}{2}\right)^{q}. N q − 1 C vBE C 2 G q T m a x ( p − 1 ) 2 p − 2 ≤ ( 2 σ ) q .
Therefore,
E ∥ ∇ ^ f p , λ ( y ) − E ∇ ^ f p , λ ( y ) ∥ q q ≤ σ q 2 q . \mathbb{E}\left\|\widehat{\nabla}f_{p,\lambda}(y)-\mathbb{E}\widehat{\nabla}f_{p,\lambda}(y)\right\|_{q}^{q}\le\frac{\sigma^{q}}{2^{q}}. E ∇ f p , λ ( y ) − E ∇ f p , λ ( y ) q q ≤ 2 q σ q .
Thus,
E ∥ ∇ ^ f p , λ ( y ) − ∇ f p , λ ( y ) ∥ q q ≤ 2 q − 1 E ∥ ∇ ^ f p , λ ( y ) − E ∇ ^ f p , λ ( y ) ∥ q q + 2 q − 1 ∥ E ∇ ^ f p , λ ( y ) − ∇ f p , λ ( y ) ∥ q q ≤ σ q 2 + 2 q − 1 ( min { δ , σ } 2 ) q ≤ σ q . \begin{aligned}
\mathbb{E}\left\|\widehat{\nabla}f_{p,\lambda}(y)-\nabla f_{p,\lambda}(y)\right\|_{q}^{q}\le2^{q-1}\mathbb{E}\left\|\widehat{\nabla}f_{p,\lambda}(y)-\mathbb{E}\widehat{\nabla}f_{p,\lambda}(y)\right\|_{q}^{q}+2^{q-1}\left\|\mathbb{E}\widehat{\nabla}f_{p,\lambda}(y)-\nabla f_{p,\lambda}(y)\right\|_{q}^{q} \\
&\le\frac{\sigma^{q}}{2}+2^{q-1}\left(\frac{\min\{\delta,\sigma\}}{2}\right)^{q}\le\sigma^{q}.
\end{aligned} E ∇ f p , λ ( y ) − ∇ f p , λ ( y ) q q ≤ 2 q − 1 E ∇ f p , λ ( y ) − E ∇ f p , λ ( y ) q q + 2 q − 1 E ∇ f p , λ ( y ) − ∇ f p , λ ( y ) q q ≤ 2 σ q + 2 q − 1 ( 2 min { δ , σ } ) q ≤ σ q .
Using ∣ x ∣ ≤ 1 + x |x|\le1+x ∣ x ∣ ≤ 1 + x , it holds that
T max ≤ 1 + ( 2 C 1 G min { δ , σ } ) p ( p − 1 ) ≤ ( 1 + ( 2 C 1 ) p ( p − 1 ) ) ( max { 1 , G min { δ , σ } } ) p ( p − 1 ) = C p ( max { 1 , G min { δ , σ } } ) p ( p − 1 ) . \begin{aligned}
T_{\max}\le1+\left(\frac{2C_{1}G}{\min\{\delta,\sigma\}}\right)^{p(p-1)}\le\left(1+(2C_{1})^{p(p-1)}\right)\left(\max\left\{1,\frac{G}{\min\{\delta,\sigma\}}\right\}\right)^{p(p-1)} \\
&=C_{p}\left(\max\left\{1,\frac{G}{\min\{\delta,\sigma\}}\right\}\right)^{p(p-1)}.
\end{aligned} T m a x ≤ 1 + ( min { δ , σ } 2 C 1 G ) p ( p − 1 ) ≤ ( 1 + ( 2 C 1 ) p ( p − 1 ) ) ( max { 1 , min { δ , σ } G } ) p ( p − 1 ) = C p ( max { 1 , min { δ , σ } G } ) p ( p − 1 ) .
for some constant C p ≥ 1 C_{p}\ge1 C p ≥ 1 . Consequently,
1 + log T max ≤ ( 1 + log C p ) + p ( p − 1 ) log max { 1 , G min { δ , σ } } ≤ C p ′ ( 1 + log max { 1 , G min { δ , σ } } ) . \begin{aligned}
1+\log T_{\max}\le(1+\log C_{p})+p(p-1)\log\max\left\{1,\frac{G}{\min\{\delta,\sigma\}}\right\} \\
&\le C'_{p}\left(1+\log\max\left\{1,\frac{G}{\min\{\delta,\sigma\}}\right\}\right).
\end{aligned} 1 + log T m a x ≤ ( 1 + log C p ) + p ( p − 1 ) log max { 1 , min { δ , σ } G } ≤ C p ′ ( 1 + log max { 1 , min { δ , σ } G } ) .
for some constant C p ′ C'_{p} C p ′ . Since N N N is deterministic, the expected total number of stochastic-gradient computations satisfies
E N ∇ ^ f ≤ N ( 1 + 3 2 j max ) = O ( N ( 1 + log T max ) ) . \mathbb{E}N_{\widehat{\nabla}f}\le N\left(1+\frac{3}{2}j_{\max}\right)=O(N(1+\log T_{\max})). E N ∇ f ≤ N ( 1 + 2 3 j m a x ) = O ( N ( 1 + log T m a x )) .
Moreover,
N ≤ 1 + 2 p C 3 G p σ p T max p − 2 p − 1 ≤ 1 + C p ′ ′ G p σ p ( max { 1 , G min { δ , σ } } ) p ( p − 2 ) N\le1+2^{p}C_{3}\frac{G^{p}}{\sigma^{p}}T_{\max}^{\frac{p-2}{p-1}}\le1+C''_{p}\frac{G^{p}}{\sigma^{p}}\left(\max\left\{1,\frac{G}{\min\{\delta,\sigma\}}\right\}\right)^{p(p-2)} N ≤ 1 + 2 p C 3 σ p G p T m a x p − 1 p − 2 ≤ 1 + C p ′′ σ p G p ( max { 1 , min { δ , σ } G } ) p ( p − 2 )
for some constant C p ′ ′ C''_{p} C p ′′ . Combining these bounds gives
E N ∇ ^ f = O ( [ 1 + G p σ p ( max { 1 , G min { δ , σ } } ) p ( p − 2 ) ] [ 1 + log max { 1 , G min { δ , σ } } ] ) . \mathbb{E}N_{\widehat{\nabla}f}=O\left(\left[1+\frac{G^{p}}{\sigma^{p}}\left(\max\left\{1,\frac{G}{\min\{\delta,\sigma\}}\right\}\right)^{p(p-2)}\right]\left[1+\log\max\left\{1,\frac{G}{\min\{\delta,\sigma\}}\right\}\right]\right). E N ∇ f = O ( [ 1 + σ p G p ( max { 1 , min { δ , σ } G } ) p ( p − 2 ) ] [ 1 + log max { 1 , min { δ , σ } G } ] ) .
which finishes the proof.
We now provide a concrete O D C p \mathrm{ODC}_{p} ODC p algorithm. Our Algorithm 3 extends Algorithm 3 of [37 ] to a composite setting. Following [2 ] , to show that Algorithm 3 is an O D C p \mathrm{ODC}_{p} ODC p algorithm, we first establish a convergence guarantee for the single-epoch Algorithm 2.
Throughout this section, we use the global prox function
d x ˉ , R ( x ) : = 1 p R p ∥ x − x ˉ ∥ p p , x ∈ X . d_{\bar{x},R}(x):=\frac{1}{pR^{p}}\lVert x-\bar{x}\rVert_{p}^{p},\qquad x\in\mathcal{X}. d x ˉ , R ( x ) := p R p 1 ∥ x − x ˉ ∥ p p , x ∈ X .
where R > 0 R>0 R > 0 . Let μ p > 0 \mu_{p}>0 μ p > 0 denote the p p p -uniform convexity constant of u ↦ 1 p ∥ u ∥ p p u\mapsto\frac{1}{p}\lVert u\rVert_{p}^{p} u ↦ p 1 ∥ u ∥ p p with respect to ∥ ⋅ ∥ p \lVert\cdot\rVert_{p} ∥ ⋅ ∥ p . Consequently, d x ˉ , R d_{\bar{x},R} d x ˉ , R is μ p / R p \mu_{p}/R^{p} μ p / R p -uniformly convex on all of X \mathcal{X} X .
V z , R , β , c ( s ) = max x ∈ X { ⟨ s , x − z ⟩ − c ψ ( x ) − β d z , R ( x ) } . (7) V_{z,R,\beta,c}(s)=\max_{x\in\mathcal{X}}\{\langle s,x-z\rangle-c\psi(x)-\beta d_{z,R}(x)\}. \tag*{(7)} V z , R , β , c ( s ) = x ∈ X max {⟨ s , x − z ⟩ − c ψ ( x ) − β d z , R ( x )} . (7)
π z , R , β , c ( s ) = arg max x ∈ X { ⟨ s , x − z ⟩ − c ψ ( x ) − β d z , R ( x ) } . (8) \pi_{z,R,\beta,c}(s)=\mathop{\arg\max}_{x\in\mathcal{X}}\{\langle s,x-z\rangle-c\psi(x)-\beta d_{z,R}(x)\}. \tag*{(8)} π z , R , β , c ( s ) = arg max x ∈ X {⟨ s , x − z ⟩ − c ψ ( x ) − β d z , R ( x )} . (8)
where ψ : X → R \psi:\mathcal{X}\to\mathbb{R} ψ : X → R is μ \mu μ -uniformly convex of order p p p , and c , β > 0 c,\beta>0 c , β > 0 . We also assume z ∈ X z\in\mathcal{X} z ∈ X .
Algorithm 2 Dual Averaging Method D A M ( ∇ ^ f , ψ , X , x ˉ , γ , R , N ) DAM(\widehat{\nabla}f,\psi,\mathcal{X},\bar{x},\gamma,R,N) D A M ( ∇ f , ψ , X , x ˉ , γ , R , N )
Set x 0 = x ˉ x_{0}=\bar{x} x 0 = x ˉ , s 0 = 0 ∈ E ⋆ s_{0}=0\in E^{\star} s 0 = 0 ∈ E ⋆ , c 0 = 0 c_{0}=0 c 0 = 0 and β = γ N 1 / q \beta=\gamma N^{1/q} β = γ N 1/ q
for i = 0 , 1 , … , N − 1 i=0,1,\ldots,N-1 i = 0 , 1 , … , N − 1 do
c i + 1 = c i + 1 c_{i+1}=c_{i}+1 c i + 1 = c i + 1
x i = π x ˉ , R , β , c i + 1 ( − s i ) x_{i}=\pi_{\bar{x},R,\beta,c_{i+1}}(-s_{i}) x i = π x ˉ , R , β , c i + 1 ( − s i )
Query a stochastic gradient ∇ ^ f ( x i ) \widehat{\nabla}f(x_{i}) ∇ f ( x i ) , and set s i + 1 = s i + ∇ ^ f ( x i ) s_{i+1}=s_{i}+\widehat{\nabla}f(x_{i}) s i + 1 = s i + ∇ f ( x i )
end for
return x ˉ N ( x ˉ , R ) = 1 N ∑ i = 0 N − 1 x i \bar{x}_{N}(\bar{x},R)=\frac{1}{N}\sum_{i=0}^{N-1}x_{i} x ˉ N ( x ˉ , R ) = N 1 ∑ i = 0 N − 1 x i
Algorithm 3 EpochDA( ∇ ^ f , ψ , X , x 0 , R 0 , N ) (\widehat{\nabla}f,\psi,\mathcal{X},x_{0},R_{0},N) ( ∇ f , ψ , X , x 0 , R 0 , N )
Set y 0 = x 0 y_{0}=x_{0} y 0 = x 0 , τ = p − 1 \tau=p-1 τ = p − 1
For each k ≥ 1 k\ge1 k ≥ 1 , define N k = ⌈ 2 p p p G p μ p μ p R k − 1 p ( p − 1 ) ⌉ N_{k}=\left\lceil\frac{2^{p}p^{p}G^{p}}{\mu_{p}\mu_{p}R_{k-1}^{p(p-1)}}\right\rceil N k = ⌈ μ p μ p R k − 1 p ( p − 1 ) 2 p p p G p ⌉ . Define m ( N ) = max { k ≥ 0 : ∑ j = 1 k N j ≤ N } m(N)=\max\left\{k\ge0:\sum_{j=1}^{k}N_{j}\le N\right\} m ( N ) = max { k ≥ 0 : ∑ j = 1 k N j ≤ N } .
for k = 1 , … , m ( N ) k=1,\ldots,m(N) k = 1 , … , m ( N ) do
R k p = 2 − k R 0 p R_{k}^{p}=2^{-k}R_{0}^{p} R k p = 2 − k R 0 p , γ k = G R k − 1 μ p 1 / p \gamma_{k}=\frac{GR_{k-1}}{\mu_{p}^{1/p}} γ k = μ p 1/ p G R k − 1
y k = D A M ( ∇ ^ f , ψ , X , y k − 1 , γ k , R k − 1 , N k ) y_{k}=DAM(\widehat{\nabla}f,\psi,\mathcal{X},y_{k-1},\gamma_{k},R_{k-1},N_{k}) y k = D A M ( ∇ f , ψ , X , y k − 1 , γ k , R k − 1 , N k )
end for
return x ^ N = y m ( N ) \hat{x}_{N}=y_{m(N)} x ^ N = y m ( N ) .
Lemma 4. (Lemma 1 of [Nes09], Lemma 1 of [IN14]) Let E ⋆ E^{\star} E ⋆ be the dual space of E E E . The following statements hold:
V z , R , β 1 , c ( s ) ≥ V z , R , β 2 , c ( s ) V_{z,R,\beta_1,c}(s) \ge V_{z,R,\beta_2,c}(s) V z , R , β 1 , c ( s ) ≥ V z , R , β 2 , c ( s )
The function V z , R , β , c V_{z,R,\beta,c} V z , R , β , c is convex and differentiable on E ∗ E^* E ∗ . Moreover,
∥ V z , R , β , c ′ ( s 1 ) − V z , R , β , c ′ ( s 2 ) ∥ p ≤ ( R p β μ p ) q − 1 ∥ s 1 − s 2 ∥ q q − 1 , ∀ s 1 , s 2 ∈ E ∗ . \left\|V'_{z,R,\beta,c}(s_1)-V'_{z,R,\beta,c}(s_2)\right\|_p \le\left(\frac{R^p}{\beta\mu_p}\right)^{q-1}\left\|s_1-s_2\right\|_q^{q-1}, \qquad\forall s_1,s_2 \in E^*. V z , R , β , c ′ ( s 1 ) − V z , R , β , c ′ ( s 2 ) p ≤ ( β μ p R p ) q − 1 ∥ s 1 − s 2 ∥ q q − 1 , ∀ s 1 , s 2 ∈ E ∗ .
V z , R , β , c ′ ( s ) + z = π z , R , β , c ( s ) ∈ X V'_{z,R,\beta,c}(s)+z=\pi_{z,R,\beta,c}(s)\in\mathcal{X} V z , R , β , c ′ ( s ) + z = π z , R , β , c ( s ) ∈ X
One might notice that V z , R , β , c ( s ) V_{z,R,\beta,c}(s) V z , R , β , c ( s ) , π z , R , β , c ( s ) \pi_{z,R,\beta,c}(s) π z , R , β , c ( s ) are defined without the term ψ ( x ) \psi(x) ψ ( x ) in [IN14]. However, the same result will be achieved by completely following the proof in [37 ] .
Lemma 5. Let f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R and ∇ ^ f \widehat{\nabla}f ∇ f satisfy Assumption 1. For any k ≥ 1 k\ge1 k ≥ 1 , N ≥ 1 N\ge1 N ≥ 1 and x ∈ X x\in\mathcal{X} x ∈ X , Algorithm 2 guarantees that
∑ i = 0 N − 1 [ ⟨ ∇ ^ f ( x i ) , x i − x ⟩ + ψ ( x i ) − ψ ( x ) ] ≤ β d x ˉ , R ( x ) + R q q μ p q − 1 β q − 1 ∑ i = 0 N − 1 ∥ ∇ ^ f ( x i ) ∥ q q . (9) \sum_{i=0}^{N-1}\left[\left\langle\widehat{\nabla}f(x_i),x_i-x\right\rangle+\psi(x_i)-\psi(x)\right]\le\beta d_{\bar{x},R}(x)+\frac{R^q}{q\mu_p^{q-1}\beta^{q-1}}\sum_{i=0}^{N-1}\left\|\widehat{\nabla}f(x_i)\right\|_q^q. \tag*{(9)} i = 0 ∑ N − 1 [ ⟨ ∇ f ( x i ) , x i − x ⟩ + ψ ( x i ) − ψ ( x ) ] ≤ β d x ˉ , R ( x ) + q μ p q − 1 β q − 1 R q i = 0 ∑ N − 1 ∇ f ( x i ) q q . (9)
Proof. Because d x ˉ , R ( x ) = 1 p R p ∥ x − x ˉ ∥ p p d_{\bar{x},R}(x)=\frac{1}{pR^p}\left\|x-\bar{x}\right\|_p^p d x ˉ , R ( x ) = p R p 1 ∥ x − x ˉ ∥ p p is μ p / R p \mu_p/R^p μ p / R p -uniformly convex, c ψ ( x ) + β d x ˉ , R ( x ) c\psi(x)+\beta d_{\bar{x},R}(x) c ψ ( x ) + β d x ˉ , R ( x ) is at least β μ p / R p \beta\mu_p/R^p β μ p / R p -uniformly convex. For each i = 0 , … , N − 1 i=0,\ldots,N-1 i = 0 , … , N − 1 , since s i + 1 = s i + ∇ ^ f ( x i ) s_{i+1}=s_i+\widehat{\nabla}f(x_i) s i + 1 = s i + ∇ f ( x i ) , the standard smoothness inequality for V x ˉ , R , β , c i + 1 V_{\bar{x},R,\beta,c_{i+1}} V x ˉ , R , β , c i + 1 gives
V x ˉ , R , β , c i + 1 ( − s i + 1 ) ≤ V x ˉ , R , β , c i + 1 ( − s i ) − ⟨ ∇ ^ f ( x i ) , V x ˉ , R , β , c i + 1 ′ ( − s i ) ⟩ + R q q μ p q − 1 β q − 1 ∥ ∇ ^ f ( x i ) ∥ q q . (10) V_{\bar{x},R,\beta,c_{i+1}}(-s_{i+1})\le V_{\bar{x},R,\beta,c_{i+1}}(-s_i)-\left\langle\widehat{\nabla}f(x_i),V'_{\bar{x},R,\beta,c_{i+1}}(-s_i)\right\rangle+\frac{R^q}{q\mu_p^{q-1}\beta^{q-1}}\left\|\widehat{\nabla}f(x_i)\right\|_q^q. \tag*{(10)} V x ˉ , R , β , c i + 1 ( − s i + 1 ) ≤ V x ˉ , R , β , c i + 1 ( − s i ) − ⟨ ∇ f ( x i ) , V x ˉ , R , β , c i + 1 ′ ( − s i ) ⟩ + q μ p q − 1 β q − 1 R q ∇ f ( x i ) q q . (10)
By the definition of x i x_i x i , V x ˉ , R , β , c i + 1 ′ ( − s i ) + x ˉ = x i V'_{\bar{x},R,\beta,c_{i+1}}(-s_i)+\bar{x}=x_i V x ˉ , R , β , c i + 1 ′ ( − s i ) + x ˉ = x i . Therefore,
V x ˉ , R , β , c i + 1 ( − s i + 1 ) ≤ V x ˉ , R , β , c i + 1 ( − s i ) − ⟨ ∇ ^ f ( x i ) , x i − x ˉ ⟩ + R q q μ p q − 1 β q − 1 ∥ ∇ ^ f ( x i ) ∥ q q . (11) V_{\bar{x},R,\beta,c_{i+1}}(-s_{i+1})\le V_{\bar{x},R,\beta,c_{i+1}}(-s_i)-\left\langle\widehat{\nabla}f(x_i),x_i-\bar{x}\right\rangle+\frac{R^q}{q\mu_p^{q-1}\beta^{q-1}}\left\|\widehat{\nabla}f(x_i)\right\|_q^q. \tag*{(11)} V x ˉ , R , β , c i + 1 ( − s i + 1 ) ≤ V x ˉ , R , β , c i + 1 ( − s i ) − ⟨ ∇ f ( x i ) , x i − x ˉ ⟩ + q μ p q − 1 β q − 1 R q ∇ f ( x i ) q q . (11)
Since c i + 1 = c i + 1 c_{i+1}=c_i+1 c i + 1 = c i + 1 , and x i x_i x i maximizes the objective defining V x ˉ , R , β , c i + 1 ( − s i ) V_{\bar{x},R,\beta,c_{i+1}}(-s_i) V x ˉ , R , β , c i + 1 ( − s i ) , we also have
V x ˉ , R , β , c i + 1 ( − s i ) ≤ V x ˉ , R , β , c i ( − s i ) − ψ ( x i ) . V_{\bar{x},R,\beta,c_{i+1}}(-s_i)\le V_{\bar{x},R,\beta,c_i}(-s_i)-\psi(x_i). V x ˉ , R , β , c i + 1 ( − s i ) ≤ V x ˉ , R , β , c i ( − s i ) − ψ ( x i ) .
Combining the previous two inequalities yields
V x ˉ , R , β , c i + 1 ( − s i + 1 ) ≤ V x ˉ , R , β , c i ( − s i ) − ⟨ ∇ ^ f ( x i ) , x i − x ˉ ⟩ − ψ ( x i ) + R q q μ p q − 1 β q − 1 ∥ ∇ ^ f ( x i ) ∥ q q (12) V_{\bar{x},R,\beta,c_{i+1}}(-s_{i+1})\le V_{\bar{x},R,\beta,c_i}(-s_i)-\left\langle\widehat{\nabla}f(x_i),x_i-\bar{x}\right\rangle-\psi(x_i)+\frac{R^q}{q\mu_p^{q-1}\beta^{q-1}}\left\|\widehat{\nabla}f(x_i)\right\|_q^q \tag*{(12)} V x ˉ , R , β , c i + 1 ( − s i + 1 ) ≤ V x ˉ , R , β , c i ( − s i ) − ⟨ ∇ f ( x i ) , x i − x ˉ ⟩ − ψ ( x i ) + q μ p q − 1 β q − 1 R q ∇ f ( x i ) q q (12)
Summing over i = 0 , … , N − 1 i=0,\ldots,N-1 i = 0 , … , N − 1 , we obtain
∑ i = 0 N − 1 [ ⟨ ∇ ^ f ( x i ) , x i − x ˉ ⟩ + ψ ( x i ) ] ≤ V x ˉ , R , β , c 0 ( − s 0 ) − V x ˉ , R , β , c N ( − s N ) + R q q μ p q − 1 β q − 1 ∑ i = 0 N − 1 ∥ ∇ ^ f ( x i ) ∥ q q . (13) \sum_{i=0}^{N-1}\left[\left\langle\widehat{\nabla}f(x_i),x_i-\bar{x}\right\rangle+\psi(x_i)\right]\le V_{\bar{x},R,\beta,c_0}(-s_0)-V_{\bar{x},R,\beta,c_N}(-s_N)+\frac{R^q}{q\mu_p^{q-1}\beta^{q-1}}\sum_{i=0}^{N-1}\left\|\widehat{\nabla}f(x_i)\right\|_q^q. \tag*{(13)} i = 0 ∑ N − 1 [ ⟨ ∇ f ( x i ) , x i − x ˉ ⟩ + ψ ( x i ) ] ≤ V x ˉ , R , β , c 0 ( − s 0 ) − V x ˉ , R , β , c N ( − s N ) + q μ p q − 1 β q − 1 R q i = 0 ∑ N − 1 ∇ f ( x i ) q q . (13)
Since s 0 = 0 s_0=0 s 0 = 0 , c 0 = 0 c_0=0 c 0 = 0 and
d x ˉ , R ( x ) ≥ 0 , d x ˉ , R ( x ˉ ) = 0 , x ˉ ∈ X , d_{\bar{x},R}(x)\ge0,\qquad d_{\bar{x},R}(\bar{x})=0,\qquad\bar{x}\in\mathcal{X}, d x ˉ , R ( x ) ≥ 0 , d x ˉ , R ( x ˉ ) = 0 , x ˉ ∈ X ,
we have
V x , R , β , c 0 ( − s 0 ) = max x ∈ X { − β d x ˉ , R ( x ) } = 0 V_{x,R,\beta,c_0}(-s_0)=\max_{x\in\mathcal{X}}\{-\beta d_{\bar{x},R}(x)\}=0 V x , R , β , c 0 ( − s 0 ) = x ∈ X max { − β d x ˉ , R ( x )} = 0
For any fixed x ∈ X x\in\mathcal{X} x ∈ X , the definition of V V V gives
− V x ˉ , R , β , c N ( − s N ) ≤ ⟨ s N , x − x ˉ ⟩ + N ψ ( x ) + β d x ˉ , R ( x ) = ∑ i = 0 N − 1 ⟨ ∇ ^ f ( x i ) , x − x ˉ ⟩ + N ψ ( x ) + β d x ˉ , R ( x ) . (14) -V_{\bar{x},R,\beta,c_N}(-s_N)\le\langle s_N,x-\bar{x}\rangle+N\psi(x)+\beta d_{\bar{x},R}(x)
=\sum_{i=0}^{N-1}\langle\widehat{\nabla}f(x_i),x-\bar{x}\rangle+N\psi(x)+\beta d_{\bar{x},R}(x). \tag*{(14)} − V x ˉ , R , β , c N ( − s N ) ≤ ⟨ s N , x − x ˉ ⟩ + N ψ ( x ) + β d x ˉ , R ( x ) = i = 0 ∑ N − 1 ⟨ ∇ f ( x i ) , x − x ˉ ⟩ + N ψ ( x ) + β d x ˉ , R ( x ) . (14)
Substituting this into (13) and rearranging proves (9) .
Lemma 6. Let F = f + ψ F=f+\psi F = f + ψ , and let x ⋆ ∈ a r g m i n x ∈ X F ( x ) x^\star\in\mathop{\rm arg\,min}_{x\in\mathcal{X}}F(x) x ⋆ ∈ arg min x ∈ X F ( x ) . Assume that the stochastic oracle satisfies
E [ ∇ ^ f ( x i ) ∣ F i ] = ∇ f ( x i ) and E [ ∥ ∇ ^ f ( x i ) ∥ q q ∣ F i ] ≤ G q , \mathbb{E}[\widehat{\nabla}f(x_i)\mid\mathcal{F}_i]=\nabla f(x_i)
\qquad\text{and}\qquad
\mathbb{E}[\|\widehat{\nabla}f(x_i)\|_q^q\mid\mathcal{F}_i]\le G^q, E [ ∇ f ( x i ) ∣ F i ] = ∇ f ( x i ) and E [ ∥ ∇ f ( x i ) ∥ q q ∣ F i ] ≤ G q , where F i \mathcal{F}_i F i denotes the history before the fresh oracle call at iteration i i i . Suppose that x ˉ \bar{x} x ˉ satisfies E ∥ x ˉ − x ⋆ ∥ p p ≤ R p \mathbb{E}\|\bar{x}-x^\star\|_p^p\le R^p E ∥ x ˉ − x ⋆ ∥ p p ≤ R p . Then the output of Algorithm 2 satisfies
E [ F ( x ˉ N ( x ˉ , R ) ) − F ( x ⋆ ) ] ≤ 1 N 1 / p [ γ p + R q G q q γ q − 1 μ p q − 1 ] . (15) \mathbb{E}[F(\bar{x}_N(\bar{x},R))-F(x^\star)]
\le\frac{1}{N^{1/p}}\left[\frac{\gamma}{p}+\frac{R^qG^q}{q\gamma^{q-1}\mu_p^{q-1}}\right]. \tag*{(15)} E [ F ( x ˉ N ( x ˉ , R )) − F ( x ⋆ )] ≤ N 1/ p 1 [ p γ + q γ q − 1 μ p q − 1 R q G q ] . (15) E ∥ x ˉ N ( x ˉ , R ) − x ⋆ ∥ p p ≤ p μ N 1 / p [ γ p + R q G q q γ q − 1 μ p q − 1 ] . (16) \mathbb{E}\|\bar{x}_N(\bar{x},R)-x^\star\|_p^p
\le\frac{p}{\mu N^{1/p}}\left[\frac{\gamma}{p}+\frac{R^qG^q}{q\gamma^{q-1}\mu_p^{q-1}}\right]. \tag*{(16)} E ∥ x ˉ N ( x ˉ , R ) − x ⋆ ∥ p p ≤ μ N 1/ p p [ p γ + q γ q − 1 μ p q − 1 R q G q ] . (16)
Proof. Apply Lemma 5 with x = x ⋆ x=x^\star x = x ⋆ . Since d x ˉ , R ( x ⋆ ) = ∥ x ˉ − x ⋆ ∥ p p p R p d_{\bar{x},R}(x^\star)=\frac{\|\bar{x}-x^\star\|_p^p}{pR^p} d x ˉ , R ( x ⋆ ) = p R p ∥ x ˉ − x ⋆ ∥ p p , we have E [ d x ˉ , R ( x ⋆ ) ] ≤ 1 p \mathbb{E}[d_{\bar{x},R}(x^\star)]\le\frac{1}{p} E [ d x ˉ , R ( x ⋆ )] ≤ p 1 . Moreover, conditional unbiasedness of the stochastic oracle gives
E [ ⟨ ∇ ^ f ( x i ) , x i − x ⋆ ⟩ ] = E [ ⟨ ∇ f ( x i ) , x i − x ⋆ ⟩ ] ≥ E [ f ( x i ) − f ( x ⋆ ) ] . (17) \mathbb{E}\left[\left\langle\widehat{\nabla}f(x_i),x_i-x^\star\right\rangle\right]
=\mathbb{E}\left[\left\langle\nabla f(x_i),x_i-x^\star\right\rangle\right]
\ge\mathbb{E}[f(x_i)-f(x^\star)]. \tag*{(17)} E [ ⟨ ∇ f ( x i ) , x i − x ⋆ ⟩ ] = E [ ⟨ ∇ f ( x i ) , x i − x ⋆ ⟩ ] ≥ E [ f ( x i ) − f ( x ⋆ )] . (17)
where the last inequality follows from convexity of f f f . Therefore,
E [ ⟨ ∇ ^ f ( x i ) , x i − x ⋆ ⟩ + ψ ( x i ) − ψ ( x ⋆ ) ] ≥ E [ F ( x i ) − F ( x ⋆ ) ] . (18) \mathbb{E}\left[\left\langle\widehat{\nabla}f(x_i),x_i-x^\star\right\rangle+\psi(x_i)-\psi(x^\star)\right]
\ge\mathbb{E}[F(x_i)-F(x^\star)]. \tag*{(18)} E [ ⟨ ∇ f ( x i ) , x i − x ⋆ ⟩ + ψ ( x i ) − ψ ( x ⋆ ) ] ≥ E [ F ( x i ) − F ( x ⋆ )] . (18)
Taking expectations in (9) , using β = γ N 1 / q \beta=\gamma N^{1/q} β = γ N 1/ q , and the moment bound E ∥ ∇ ^ f ( x i ) ∥ q q ≤ G q \mathbb{E}\|\widehat{\nabla}f(x_i)\|_q^q\le G^q E ∥ ∇ f ( x i ) ∥ q q ≤ G q , gives
1 N ∑ i = 0 N − 1 E [ F ( x i ) − F ( x ⋆ ) ] ≤ 1 N 1 / p [ γ p + R q G q q γ q − 1 μ p q − 1 ] . (19) \frac{1}{N}\sum_{i=0}^{N-1}\mathbb{E}[F(x_i)-F(x^\star)]
\le\frac{1}{N^{1/p}}\left[\frac{\gamma}{p}+\frac{R^qG^q}{q\gamma^{q-1}\mu_p^{q-1}}\right]. \tag*{(19)} N 1 i = 0 ∑ N − 1 E [ F ( x i ) − F ( x ⋆ )] ≤ N 1/ p 1 [ p γ + q γ q − 1 μ p q − 1 R q G q ] . (19)
By convexity of F F F ,
F ( x ˉ N ( x ˉ , R ) ) = F ( 1 N ∑ i = 0 N − 1 x i ) ≤ 1 N ∑ i = 0 N − 1 F ( x i ) . F(\bar{x}_N(\bar{x},R))=F\left(\frac{1}{N}\sum_{i=0}^{N-1}x_i\right)\le\frac{1}{N}\sum_{i=0}^{N-1}F(x_i). F ( x ˉ N ( x ˉ , R )) = F ( N 1 i = 0 ∑ N − 1 x i ) ≤ N 1 i = 0 ∑ N − 1 F ( x i ) .
Plugging this into (19) proves (15) . Finally, since F = f + ψ F=f+\psi F = f + ψ is μ \mu μ -uniformly convex of order p p p ,
F ( x ) − F ( x ⋆ ) ≥ μ p ∥ x − x ⋆ ∥ p p . F(x)-F(x^\star)\ge\frac{\mu}{p}\|x-x^\star\|_p^p. F ( x ) − F ( x ⋆ ) ≥ p μ ∥ x − x ⋆ ∥ p p .
Hence
E ∥ x ˉ N ( x ˉ , R ) − x ⋆ ∥ p p ≤ p μ E [ F ( x ˉ N ( x ˉ , R ) ) − F ( x ⋆ ) ] , \mathbb{E}\|\bar{x}_N(\bar{x},R)-x^\star\|_p^p
\le\frac{p}{\mu}\mathbb{E}[F(\bar{x}_N(\bar{x},R))-F(x^\star)], E ∥ x ˉ N ( x ˉ , R ) − x ⋆ ∥ p p ≤ μ p E [ F ( x ˉ N ( x ˉ , R )) − F ( x ⋆ )] ,
which proves (16) .
Corollary 2. Under the assumptions of Lemma 6, choose γ = G R μ p 1 / p \gamma= \frac{GR}{\mu_p^{1/p}} γ = μ p 1/ p GR . Then
E [ F ( x ˉ N ( x ˉ , R ) ) − F ( x ∗ ) ] ≤ G R μ p 1 / p N 1 / p , (20) \mathbb{E}[F(\bar{x}_N(\bar{x},R))-F(x^*)] \le\frac{GR}{\mu_p^{1/p}N^{1/p}}, \tag*{(20)} E [ F ( x ˉ N ( x ˉ , R )) − F ( x ∗ )] ≤ μ p 1/ p N 1/ p GR , (20) E ∥ x ˉ N ( x ˉ , R ) − x ∗ ∥ p p ≤ G R p μ μ p 1 / p N 1 / p . (21) \mathbb{E}\|\bar{x}_N(\bar{x},R)-x^*\|_p^p \le\frac{GRp}{\mu\mu_p^{1/p}N^{1/p}}. \tag*{(21)} E ∥ x ˉ N ( x ˉ , R ) − x ∗ ∥ p p ≤ μ μ p 1/ p N 1/ p GR p . (21)
Theorem 3. Assume ∥ y 0 − x ∗ ∥ p p ≤ R 0 p \|y_0-x^*\|_p^p \le R_0^p ∥ y 0 − x ∗ ∥ p p ≤ R 0 p . Then the sequence { y k } k = 0 m ( N ) \{y_k\}_{k=0}^{m(N)} { y k } k = 0 m ( N ) generated by Algorithm 3 satisfies
E ∥ y k − x ∗ ∥ p p ≤ R k p = 2 − k R 0 p , k = 0 , 1 , … , m ( N ) . \mathbb{E}\|y_k-x^*\|_p^p \le R_k^p=2^{-k}R_0^p,\qquad k=0,1,\ldots,m(N). E ∥ y k − x ∗ ∥ p p ≤ R k p = 2 − k R 0 p , k = 0 , 1 , … , m ( N ) . Moreover, for every completed epoch k ≥ 1 k\ge1 k ≥ 1 ,
E [ F ( y k ) − F ( x ∗ ) ] ≤ μ p R k p . \mathbb{E}[F(y_k)-F(x^*)] \le\frac{\mu}{p}R_k^p. E [ F ( y k ) − F ( x ∗ )] ≤ p μ R k p .
Proof. We first prove by induction that E ∥ y k − x ∗ ∥ p p ≤ R k p \mathbb{E}\|y_k-x^*\|_p^p\le R_k^p E ∥ y k − x ∗ ∥ p p ≤ R k p . For k = 0 k=0 k = 0 , this follows from the assumption ∥ y 0 − x ∗ ∥ p p ≤ R 0 p \|y_0-x^*\|_p^p\le R_0^p ∥ y 0 − x ∗ ∥ p p ≤ R 0 p . Now assume that for some k ≥ 1 k\ge1 k ≥ 1 , E ∥ y k − 1 − x ∗ ∥ p p ≤ R k − 1 p \mathbb{E}\|y_{k-1}-x^*\|_p^p\le R_{k-1}^p E ∥ y k − 1 − x ∗ ∥ p p ≤ R k − 1 p . By Corollary 2, this gives
E [ F ( y k ) − F ( x ∗ ) ] ≤ G R k − 1 μ p 1 / p N k 1 / p . \mathbb{E}[F(y_k)-F(x^*)]\le\frac{GR_{k-1}}{\mu_p^{1/p}N_k^{1/p}}. E [ F ( y k ) − F ( x ∗ )] ≤ μ p 1/ p N k 1/ p G R k − 1 .
By the definition of N k N_k N k , N k ≥ 2 p p p G p μ p μ p R k − 1 p ( p − 1 ) N_k\ge\frac{2^pp^pG^p}{\mu^p\mu_pR_{k-1}^{p(p-1)}} N k ≥ μ p μ p R k − 1 p ( p − 1 ) 2 p p p G p . Therefore, N k 1 / p ≥ 2 p G μ μ p 1 / p R k − 1 p − 1 N_k^{1/p}\ge\frac{2pG}{\mu\mu_p^{1/p}R_{k-1}^{p-1}} N k 1/ p ≥ μ μ p 1/ p R k − 1 p − 1 2 pG . Substituting this bound yields
E [ F ( y k ) − F ( x ∗ ) ] ≤ G R k − 1 μ p 1 / p ⋅ μ μ p 1 / p R k − 1 p − 1 2 p G = μ 2 p R k − 1 p = μ p R k p . (22) \mathbb{E}[F(y_k)-F(x^*)]\le\frac{GR_{k-1}}{\mu_p^{1/p}}\cdot\frac{\mu\mu_p^{1/p}R_{k-1}^{p-1}}{2pG}=\frac{\mu}{2p}R_{k-1}^p=\frac{\mu}{p}R_k^p. \tag*{(22)} E [ F ( y k ) − F ( x ∗ )] ≤ μ p 1/ p G R k − 1 ⋅ 2 pG μ μ p 1/ p R k − 1 p − 1 = 2 p μ R k − 1 p = p μ R k p . (22)
Since F F F is μ \mu μ -uniformly convex of order p p p ,
F ( y k ) − F ( x ∗ ) ≥ μ p ∥ y k − x ∗ ∥ p p . F(y_k)-F(x^*)\ge\frac{\mu}{p}\|y_k-x^*\|_p^p. F ( y k ) − F ( x ∗ ) ≥ p μ ∥ y k − x ∗ ∥ p p .
Taking expectations gives
E ∥ y k − x ∗ ∥ p p ≤ R k p . \mathbb{E}\|y_k-x^*\|_p^p\le R_k^p. E ∥ y k − x ∗ ∥ p p ≤ R k p .
which completes the induction. Since τ = p − 1 \tau=p-1 τ = p − 1 , then N k = ⌈ 2 p p p G p μ p μ p R 0 p ( p − 1 ) 2 ( k − 1 ) τ ⌉ N_k=\left\lceil\frac{2^pp^pG^p}{\mu^p\mu_pR_0^{p(p-1)}}2^{(k-1)\tau}\right\rceil N k = ⌈ μ p μ p R 0 p ( p − 1 ) 2 p p p G p 2 ( k − 1 ) τ ⌉ . Let m = m ( N ) m=m(N) m = m ( N ) . By maximality of m m m , N < ∑ k = 1 m + 1 N k N<\sum_{k=1}^{m+1}N_k N < ∑ k = 1 m + 1 N k . Then,
N < 2 p p p G p μ p μ p R 0 p ( p − 1 ) ∑ k = 0 m 2 k τ + ( m + 1 ) ≤ 2 p p p G p μ p μ p R 0 p ( p − 1 ) 2 ( m + 1 ) τ 2 τ − 1 + ( m + 1 ) . (23) N<\frac{2^pp^pG^p}{\mu^p\mu_pR_0^{p(p-1)}}\sum_{k=0}^{m}2^{k\tau}+(m+1)\le\frac{2^pp^pG^p}{\mu^p\mu_pR_0^{p(p-1)}}\frac{2^{(m+1)\tau}}{2^\tau-1}+(m+1). \tag*{(23)} N < μ p μ p R 0 p ( p − 1 ) 2 p p p G p k = 0 ∑ m 2 k τ + ( m + 1 ) ≤ μ p μ p R 0 p ( p − 1 ) 2 p p p G p 2 τ − 1 2 ( m + 1 ) τ + ( m + 1 ) . (23)
Since τ ≥ 1 \tau\ge1 τ ≥ 1 and m ≥ 1 m\ge1 m ≥ 1 , m + 1 ≤ 2 m ≤ 2 m τ m+1\le2^m\le2^{m\tau} m + 1 ≤ 2 m ≤ 2 m τ . Hence
N ≤ [ 2 p p p G p μ p μ p R 0 p ( p − 1 ) 2 τ 2 τ − 1 + 1 ] 2 m τ . N\le\left[\frac{2^pp^pG^p}{\mu^p\mu_pR_0^{p(p-1)}}\frac{2^\tau}{2^\tau-1}+1\right]2^{m\tau}. N ≤ [ μ p μ p R 0 p ( p − 1 ) 2 p p p G p 2 τ − 1 2 τ + 1 ] 2 m τ .
Therefore,
2 − m ≤ [ 2 p p p G p μ p μ p R 0 p ( p − 1 ) 2 τ ( 2 τ − 1 ) N + 1 N ] 1 / τ . 2^{-m}\le\left[\frac{2^pp^pG^p}{\mu^p\mu_pR_0^{p(p-1)}}\frac{2^\tau}{(2^\tau-1)N}+\frac{1}{N}\right]^{1/\tau}. 2 − m ≤ [ μ p μ p R 0 p ( p − 1 ) 2 p p p G p ( 2 τ − 1 ) N 2 τ + N 1 ] 1/ τ .
From the epoch guarantee,
E [ F ( x ^ N ) − F ( x ⋆ ) ] ≤ μ p R m p = μ p R 0 p 2 − m . \mathbb{E}[F(\widehat{x}_N)-F(x^\star)] \leq\frac{\mu}{p}R_m^p = \frac{\mu}{p}R_0^p2^{-m}. E [ F ( x N ) − F ( x ⋆ )] ≤ p μ R m p = p μ R 0 p 2 − m .
Therefore,
E [ F ( x ^ N ) − F ( x ⋆ ) ] ≤ μ p R 0 p [ 2 p p p G p μ p μ p p R 0 p ( p − 1 ) 2 τ ( 2 τ − 1 ) N + 1 N ] 1 / τ . \mathbb{E}[F(\widehat{x}_N)-F(x^\star)] \leq\frac{\mu}{p}R_0^p\left[\frac{2^p p^p G^p}{\mu^p \mu_p^p R_0^{p(p-1)}}\frac{2^\tau}{(2^\tau-1)N}+\frac{1}{N}\right]^{1/\tau}. E [ F ( x N ) − F ( x ⋆ )] ≤ p μ R 0 p [ μ p μ p p R 0 p ( p − 1 ) 2 p p p G p ( 2 τ − 1 ) N 2 τ + N 1 ] 1/ τ .
Using R 0 ≤ p q − 1 ( G μ ) 1 / ( p − 1 ) R_0 \leq p^{q-1}\left(\frac{G}{\mu}\right)^{1/(p-1)} R 0 ≤ p q − 1 ( μ G ) 1/ ( p − 1 ) , both terms are bounded by C p G q μ 1 / ( p − 1 ) N 1 / ( p − 1 ) C_p\frac{G^q}{\mu^{1/(p-1)}N^{1/(p-1)}} C p μ 1/ ( p − 1 ) N 1/ ( p − 1 ) G q , for a constant C p C_p C p depending only on p p p . Thus,
E [ F ( x ^ N ) − F ( x ⋆ ) ] ≤ C p G q μ 1 / ( p − 1 ) N 1 / ( p − 1 ) . \mathbb{E}[F(\widehat{x}_N)-F(x^\star)] \leq C_p\frac{G^q}{\mu^{1/(p-1)}N^{1/(p-1)}}. E [ F ( x N ) − F ( x ⋆ )] ≤ C p μ 1/ ( p − 1 ) N 1/ ( p − 1 ) G q .
Finally, uniform convexity gives
E ∥ x ^ N − x ⋆ ∥ p p ≤ 1 μ E [ F ( x ^ N ) − F ( x ⋆ ) ] , \mathbb{E}\|\widehat{x}_N-x^\star\|_p^p \leq\frac{1}{\mu}\mathbb{E}[F(\widehat{x}_N)-F(x^\star)], E ∥ x N − x ⋆ ∥ p p ≤ μ 1 E [ F ( x N ) − F ( x ⋆ )] ,
and hence
E ∥ x ^ N − x ⋆ ∥ p p ≤ C p G q μ q N 1 / ( p − 1 ) . \mathbb{E}\|\widehat{x}_N-x^\star\|_p^p \leq C_p\frac{G^q}{\mu^qN^{1/(p-1)}}. E ∥ x N − x ⋆ ∥ p p ≤ C p μ q N 1/ ( p − 1 ) G q .
Corollary 4. Algorithm 3 is an O D C p ODC_p O D C p algorithm with a constant C p C_p C p depending only on $p.
Projection-Efficient ℓ p \ell_p ℓ p -Lipschitz Optimization
We combine our ℓ p \ell_p ℓ p -Moreau envelope gradient estimator with an accelerated gradient method to reduce the number of expensive constrained updates over X \mathcal{X} X . We assume that f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R and its stochastic subgradient oracle satisfy Assumption 1 on R d \mathbb{R}^d R d , and that the initial point x 0 ∈ X x_0\in\mathcal{X} x 0 ∈ X satisfies ∥ x 0 − x ⋆ ∥ p ≤ R \|x_0-x^\star\|_p\leq R ∥ x 0 − x ⋆ ∥ p ≤ R . The Moreau envelope is defined over R d \mathbb{R}^d R d , and its gradient is estimated using O D C p ODC_p O D C p over ℓ p \ell_p ℓ p balls. Only the outer updates involve the complicated set X \mathcal{X} X .
Our approach is inspired by that of Algorithm 3 in [2 ] , which involves applying a variant of Nesterov’s accelerated gradient descent method on the Moreau envelope f λ ( y ) = min x { f ( x ) + λ 2 ∥ x − y ∥ 2 2 } f_\lambda(y)=\min_x\{f(x)+\frac{\lambda}{2}\|x-y\|_2^2\} f λ ( y ) = min x { f ( x ) + 2 λ ∥ x − y ∥ 2 2 } in the Euclidean setting. Importantly, it may be seen that projections onto X \mathcal{X} X are not needed for computing the Euclidean Moreau envelope itself, so that accurate approximations of ∇ f λ \nabla f_\lambda ∇ f λ allow the outer accelerated method to find an O ( ϵ ) O(\epsilon) O ( ϵ ) -suboptimal point of f λ f_\lambda f λ using T = O ( λ R 2 / ϵ ) T=O\left(\sqrt{\lambda R^2/\epsilon}\right) T = O ( λ R 2 / ϵ ) projections onto X \mathcal{X} X . To ensure that this point is also O ( ϵ ) O(\epsilon) O ( ϵ ) -suboptimal for the original objective f f f , one must choose λ = G 2 / ϵ \lambda=G^2/\epsilon λ = G 2 / ϵ , which yields the projection complexity O ( G R / ϵ ) O(GR/\epsilon) O ( GR / ϵ ) .
Algorithm 4 Stochastic AGD with ℓ p \ell_p ℓ p Moreau envelope
Parameters λ , M > 0 \lambda, M > 0 λ , M > 0 and T ≥ 1 T \ge1 T ≥ 1
r λ = 2 ( G / λ ) 1 / ( p − 1 ) r_\lambda= 2(G/\lambda)^{1/(p-1)} r λ = 2 ( G / λ ) 1/ ( p − 1 )
A 0 = 0 A_0 = 0 A 0 = 0 , x 0 = z 0 = y 0 = x i n i t x_0 = z_0 = y_0 = x_{\mathrm{init}} x 0 = z 0 = y 0 = x init
Bias tolerances { δ t } t = 1 T \{\delta_t\}_{t=1}^{T} { δ t } t = 1 T and moment bounds { σ t q } t = 1 T \{\sigma_t^q\}_{t=1}^{T} { σ t q } t = 1 T
for t = 0 , … , T − 1 t = 0, \ldots, T - 1 t = 0 , … , T − 1 do
A t + 1 = 2 p + q − 1 ( p − 1 ) a t + 1 q A_{t+1} = 2^{p+q-1}(p-1)a_{t+1}^q A t + 1 = 2 p + q − 1 ( p − 1 ) a t + 1 q , A t + 1 = A t + a t + 1 \quad A_{t+1} = A_t + a_{t+1} A t + 1 = A t + a t + 1
y t + 1 ← A t A t + 1 x t + a t + 1 A t + 1 z t y_{t+1} \leftarrow\frac{A_t}{A_{t+1}}x_t + \frac{a_{t+1}}{A_{t+1}}z_t y t + 1 ← A t + 1 A t x t + A t + 1 a t + 1 z t
∇ f p , λ ′ ( y t + 1 ) ← ℓ p \nabla f_{p,\lambda}'(y_{t+1}) \leftarrow\ell_p ∇ f p , λ ′ ( y t + 1 ) ← ℓ p -MorGradEst( ∇ ^ f , y t + 1 , λ , δ t + 1 , σ t + 1 q , B r λ ( y t + 1 ) ) (\widehat{\nabla} f, y_{t+1}, \lambda, \delta_{t+1}, \sigma_{t+1}^q, B_{r_\lambda}(y_{t+1})) ( ∇ f , y t + 1 , λ , δ t + 1 , σ t + 1 q , B r λ ( y t + 1 ))
x t + 1 ← a r g m i n x ∈ X { ⟨ ∇ f p , λ ′ ( y t + 1 ) , x − y t + 1 ⟩ + M p ∥ x − y t + 1 ∥ p p } x_{t+1} \leftarrow\mathop{\rm arg\,min}_{x\in\mathcal{X}} \left\{ \left\langle\nabla f_{p,\lambda}'(y_{t+1}), x-y_{t+1}\right\rangle+ \frac{M}{p}\|x-y_{t+1}\|_p^p \right\} x t + 1 ← arg min x ∈ X { ⟨ ∇ f p , λ ′ ( y t + 1 ) , x − y t + 1 ⟩ + p M ∥ x − y t + 1 ∥ p p }
z t + 1 ← a r g m i n z ∈ X { a t + 1 M ⟨ ∇ f p , λ ′ ( y t + 1 ) , z ⟩ + ω p ( z , z t ) } z_{t+1} \leftarrow\mathop{\rm arg\,min}_{z\in\mathcal{X}} \left\{ \frac{a_{t+1}}{M}\left\langle\nabla f_{p,\lambda}'(y_{t+1}),z\right\rangle+ \omega_p(z,z_t) \right\} z t + 1 ← arg min z ∈ X { M a t + 1 ⟨ ∇ f p , λ ′ ( y t + 1 ) , z ⟩ + ω p ( z , z t ) }
end for
return x T x_T x T
Throughout this section, we define
ψ ( u ) = 1 p ∥ u − x i n i t ∥ p p , ω p ( u , v ) = ψ ( u ) − ψ ( v ) − ⟨ ∇ ψ ( v ) , u − v ⟩ . \psi(u)=\frac{1}{p}\|u-x_{\mathrm{init}}\|_p^p,\qquad\omega_p(u,v)=\psi(u)-\psi(v)-\langle\nabla\psi(v),u-v\rangle. ψ ( u ) = p 1 ∥ u − x init ∥ p p , ω p ( u , v ) = ψ ( u ) − ψ ( v ) − ⟨ ∇ ψ ( v ) , u − v ⟩ .
Lemma 7. Let X ⊆ B R ( 0 ) \mathcal{X}\subseteq B_{\mathbb{R}}(0) X ⊆ B R ( 0 ) be a nonempty closed convex set. Let f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R and ∇ ^ f \widehat{\nabla}f ∇ f satisfy Assumption 1 . Let x 0 = x i n i t ∈ X x_0=x_{\mathrm{init}}\in\mathcal{X} x 0 = x init ∈ X satisfy ∥ x 0 − x ⋆ ∥ p ≤ R \|x_0-x^\star\|_p\le R ∥ x 0 − x ⋆ ∥ p ≤ R for some x ⋆ ∈ X x^\star\in\mathcal{X} x ⋆ ∈ X . For any ϵ > 0 \epsilon>0 ϵ > 0 and integer T ≥ 1 T\ge1 T ≥ 1 , choose λ , M > 0 \lambda,M>0 λ , M > 0 such that
L λ = ( p − 1 ) λ q − 1 G 2 − q , M ≥ 2 L λ p / 2 ( T ϵ ) ( p − 2 ) / 2 . L_\lambda=(p-1)\lambda^{q-1}G^{2-q},\qquad M\ge2L_\lambda^{p/2}\left(\frac{T}{\epsilon}\right)^{(p-2)/2}. L λ = ( p − 1 ) λ q − 1 G 2 − q , M ≥ 2 L λ p /2 ( ϵ T ) ( p − 2 ) /2 .
For t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 , set
δ t + 1 = ϵ 2 R , σ t + 1 q = q ϵ M 1 / ( p − 1 ) a t + 1 2 q − 1 A t + 1 . (24) \delta_{t+1}=\frac{\epsilon}{2R},\qquad\sigma_{t+1}^q=\frac{q\epsilon M^{1/(p-1)}a_{t+1}}{2^{q-1}A_{t+1}}. \tag*{(24)} δ t + 1 = 2 R ϵ , σ t + 1 q = 2 q − 1 A t + 1 q ϵ M 1/ ( p − 1 ) a t + 1 . (24)
Then the iterates of Algorithm 4 satisfy, for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 ,
A t + 1 M E [ f p , λ ( x t + 1 ) − f p , λ ( x ⋆ ) ] + E [ ω p ( x ⋆ , z t + 1 ) ] ≤ A t M E [ f p , λ ( x t ) − f p , λ ( x ⋆ ) ] + E [ ω p ( x ⋆ , z t ) ] + 3 a t + 1 M ϵ . (25) \frac{A_{t+1}}{M}\mathbb{E}[f_{p,\lambda}(x_{t+1})-f_{p,\lambda}(x^\star)]+\mathbb{E}[\omega_p(x^\star,z_{t+1})]
\le\frac{A_t}{M}\mathbb{E}[f_{p,\lambda}(x_t)-f_{p,\lambda}(x^\star)]+\mathbb{E}[\omega_p(x^\star,z_t)]+\frac{3a_{t+1}}{M}\epsilon. \tag*{(25)} M A t + 1 E [ f p , λ ( x t + 1 ) − f p , λ ( x ⋆ )] + E [ ω p ( x ⋆ , z t + 1 )] ≤ M A t E [ f p , λ ( x t ) − f p , λ ( x ⋆ )] + E [ ω p ( x ⋆ , z t )] + M 3 a t + 1 ϵ . (25)
Lemma 8 ([45 ] ). For any x , y ∈ R n x,y\in\mathbb{R}^n x , y ∈ R n and p ≥ 2 p\ge2 p ≥ 2 , ω p ( y , x ) ≥ 1 p 2 p + 1 ∥ y − x ∥ p p \omega_p(y,x)\ge\frac{1}{p2^{p+1}}\|y-x\|_p^p ω p ( y , x ) ≥ p 2 p + 1 1 ∥ y − x ∥ p p .
Lemma 9. [1 ] With A 0 = 0 A_0=0 A 0 = 0 , suppose that a t + 1 > 0 a_{t+1}>0 a t + 1 > 0 , A t + 1 = A t + a t + 1 A_{t+1}=A_t+a_{t+1} A t + 1 = A t + a t + 1 , and A t + 1 = 2 p + q − 1 ( p − 1 ) a t + 1 q A_{t+1}=2^{p+q-1}(p-1)a_{t+1}^q A t + 1 = 2 p + q − 1 ( p − 1 ) a t + 1 q . Then, for all t ≥ 0 t\ge0 t ≥ 0 ,
A t + 1 1 / p ≥ A t 1 / p + 1 p [ 2 p + q − 1 ( p − 1 ) ] 1 / q . A_{t+1}^{1/p}\ge A_t^{1/p}+\frac{1}{p[2^{p+q-1}(p-1)]^{1/q}}. A t + 1 1/ p ≥ A t 1/ p + p [ 2 p + q − 1 ( p − 1 ) ] 1/ q 1 .
Consequently, for every T ≥ 1 T\ge1 T ≥ 1 ,
A T 1 / p ≥ A 0 1 / p + T p [ 2 p + q − 1 ( p − 1 ) ] 1 / q . A_T^{1/p}\ge A_0^{1/p}+\frac{T}{p[2^{p+q-1}(p-1)]^{1/q}}. A T 1/ p ≥ A 0 1/ p + p [ 2 p + q − 1 ( p − 1 ) ] 1/ q T .
Proof of the convergence lemma
Proof. The proof is generally inspired by those of [1 , 2 ] . First, x t , z t , y t + 1 ∈ X x_t,z_t,y_{t+1}\in\mathcal{X} x t , z t , y t + 1 ∈ X by construction and the convexity of X \mathcal{X} X . By the convexity of f p , λ f_{p,\lambda} f p , λ and the identity a t + 1 ( y t + 1 − z t ) = A t ( x t − y t + 1 ) a_{t+1}(y_{t+1}-z_t)=A_t(x_t-y_{t+1}) a t + 1 ( y t + 1 − z t ) = A t ( x t − y t + 1 ) , we have
A t + 1 M ( f p , λ ( y t + 1 ) − f p , λ ( x ⋆ ) ) ≤ A t M ( f p , λ ( x t ) − f p , λ ( x ⋆ ) ) + a t + 1 M ⟨ ∇ f p , λ ( y t + 1 ) , z t − x ⋆ ⟩ . (26) \frac{A_{t+1}}{M}\left(f_{p,\lambda}(y_{t+1})-f_{p,\lambda}(x^\star)\right)\leq\frac{A_t}{M}\left(f_{p,\lambda}(x_t)-f_{p,\lambda}(x^\star)\right)+\frac{a_{t+1}}{M}\left\langle\nabla f_{p,\lambda}(y_{t+1}),z_t-x^\star\right\rangle. \tag*{(26)} M A t + 1 ( f p , λ ( y t + 1 ) − f p , λ ( x ⋆ ) ) ≤ M A t ( f p , λ ( x t ) − f p , λ ( x ⋆ ) ) + M a t + 1 ⟨ ∇ f p , λ ( y t + 1 ) , z t − x ⋆ ⟩ . (26)
The optimality condition for z t + 1 z_{t+1} z t + 1 and the three-point identity give
a t + 1 M ⟨ ∇ f p , λ ′ ( y t + 1 ) , z t + 1 − x ⋆ ⟩ ≤ ω p ( x ⋆ , z t ) − ω p ( x ⋆ , z t + 1 ) − ω p ( z t + 1 , z t ) . (27) \frac{a_{t+1}}{M}\left\langle\nabla f'_{p,\lambda}(y_{t+1}),z_{t+1}-x^\star\right\rangle\leq\omega_p(x^\star,z_t)-\omega_p(x^\star,z_{t+1})-\omega_p(z_{t+1},z_t). \tag*{(27)} M a t + 1 ⟨ ∇ f p , λ ′ ( y t + 1 ) , z t + 1 − x ⋆ ⟩ ≤ ω p ( x ⋆ , z t ) − ω p ( x ⋆ , z t + 1 ) − ω p ( z t + 1 , z t ) . (27)
Proximal optimality bounds the minimizing displacement by r λ / 2 r_\lambda/2 r λ /2 , so the smoothness argument of [19 ] gives L λ L_\lambda L λ . By the radius bound in the proof of Lemma 3 with X = R d \mathcal{X}=\mathbb{R}^d X = R d , the ball in Algorithm 4 contains the unconstrained proximal minimizer. Hence,
f p , λ ( u ) ≤ f p , λ ( v ) + ⟨ ∇ f p , λ ( v ) , u − v ⟩ + L λ 2 ∥ u − v ∥ p 2 ≤ f p , λ ( v ) + ⟨ ∇ f p , λ ( v ) , u − v ⟩ + M 2 p ∥ u − v ∥ p p + ϵ T . (28) \begin{aligned}
f_{p,\lambda}(u)\leq f_{p,\lambda}(v)+\left\langle\nabla f_{p,\lambda}(v),u-v\right\rangle+\frac{L_\lambda}{2}\lVert u-v\rVert_p^2 \\
&\leq f_{p,\lambda}(v)+\left\langle\nabla f_{p,\lambda}(v),u-v\right\rangle+\frac{M}{2p}\lVert u-v\rVert_p^p+\frac{\epsilon}{T}. \tag*{(28)}
\end{aligned} f p , λ ( u ) ≤ f p , λ ( v ) + ⟨ ∇ f p , λ ( v ) , u − v ⟩ + 2 L λ ∥ u − v ∥ p 2 ≤ f p , λ ( v ) + ⟨ ∇ f p , λ ( v ) , u − v ⟩ + 2 p M ∥ u − v ∥ p p + T ϵ . (28)
On the other hand, by Young’s inequality we have,
⟨ ∇ f p , λ ( y t + 1 ) − ∇ f p , λ ′ ( y t + 1 ) , x t + 1 − y t + 1 ⟩ ≤ M 2 p ∥ x t + 1 − y t + 1 ∥ p p + 2 q − 1 q M q − 1 ∥ ∇ f p , λ ( y t + 1 ) − ∇ f p , λ ′ ( y t + 1 ) ∥ q q . \left\langle\nabla f_{p,\lambda}(y_{t+1})-\nabla f'_{p,\lambda}(y_{t+1}),x_{t+1}-y_{t+1}\right\rangle
\leq\frac{M}{2p}\lVert x_{t+1}-y_{t+1}\rVert_p^p+\frac{2^{q-1}}{qM^{q-1}}\lVert\nabla f_{p,\lambda}(y_{t+1})-\nabla f'_{p,\lambda}(y_{t+1})\rVert_q^q. ⟨ ∇ f p , λ ( y t + 1 ) − ∇ f p , λ ′ ( y t + 1 ) , x t + 1 − y t + 1 ⟩ ≤ 2 p M ∥ x t + 1 − y t + 1 ∥ p p + q M q − 1 2 q − 1 ∥ ∇ f p , λ ( y t + 1 ) − ∇ f p , λ ′ ( y t + 1 ) ∥ q q .
By the convexity of X \mathcal{X} X ,
A t A t + 1 x t + a t + 1 A t + 1 z t + 1 ∈ X . \frac{A_t}{A_{t+1}}x_t+\frac{a_{t+1}}{A_{t+1}}z_{t+1}\in\mathcal{X}. A t + 1 A t x t + A t + 1 a t + 1 z t + 1 ∈ X .
With the optimality of x t + 1 x_{t+1} x t + 1 and (27), we obtain
A t + 1 M ( f p , λ ( x t + 1 ) − f p , λ ( y t + 1 ) ) ≤ a t + 1 M ⟨ ∇ f p , λ ′ ( y t + 1 ) , z t + 1 − z ~ t ⟩ + a t + 1 p p A t + 1 p − 1 ∥ z t + 1 − z ~ t ∥ p p + A t + 1 ϵ M T + 2 q − 1 A t + 1 q M q ∥ ∇ f p , λ ( y t + 1 ) − ∇ f p , λ ′ ( y t + 1 ) ∥ q q . (29) \begin{aligned}
\frac{A_{t+1}}{M}\left(f_{p,\lambda}(x_{t+1})-f_{p,\lambda}(y_{t+1})\right)\leq\frac{a_{t+1}}{M}\left\langle\nabla f'_{p,\lambda}(y_{t+1}),z_{t+1}-\widetilde{z}_t\right\rangle+\frac{a_{t+1}^p}{pA_{t+1}^{p-1}}\lVert z_{t+1}-\widetilde{z}_t\rVert_p^p \\
&+\frac{A_{t+1}\epsilon}{MT}+\frac{2^{q-1}A_{t+1}}{qM^q}\lVert\nabla f_{p,\lambda}(y_{t+1})-\nabla f'_{p,\lambda}(y_{t+1})\rVert_q^q. \tag*{(29)}
\end{aligned} M A t + 1 ( f p , λ ( x t + 1 ) − f p , λ ( y t + 1 ) ) ≤ M a t + 1 ⟨ ∇ f p , λ ′ ( y t + 1 ) , z t + 1 − z t ⟩ + p A t + 1 p − 1 a t + 1 p ∥ z t + 1 − z t ∥ p p + M T A t + 1 ϵ + q M q 2 q − 1 A t + 1 ∥ ∇ f p , λ ( y t + 1 ) − ∇ f p , λ ′ ( y t + 1 ) ∥ q q . (29)
By Lemma 8,
ω p ( z t + 1 , z t ) ≥ 1 p 2 p + 1 ∥ z t + 1 − z t ∥ p p . \omega_p(z_{t+1},z_t)\geq\frac{1}{p2^{p+1}}\lVert z_{t+1}-z_t\rVert_p^p. ω p ( z t + 1 , z t ) ≥ p 2 p + 1 1 ∥ z t + 1 − z t ∥ p p .
Using A t + 1 = 2 p + q − 1 ( p − 1 ) a t + 1 q A_{t+1}=2^{p+q-1}(p-1)a_{t+1}^q A t + 1 = 2 p + q − 1 ( p − 1 ) a t + 1 q and q ( p − 1 ) = p q(p-1)=p q ( p − 1 ) = p , we obtain
a t + 1 p p A t + 1 p − 1 = 1 p [ 2 p + q − 1 ( p − 1 ) ] p − 1 = 1 p ( p − 1 ) p − 1 2 p ( p − 1 ) + 1 ≤ 1 p 2 p + 1 , \frac{a_{t+1}^p}{pA_{t+1}^{p-1}}=\frac{1}{p[2^{p+q-1}(p-1)]^{p-1}}=\frac{1}{p(p-1)^{p-1}2^{p(p-1)+1}}\leq\frac{1}{p2^{p+1}}, p A t + 1 p − 1 a t + 1 p = p [ 2 p + q − 1 ( p − 1 ) ] p − 1 1 = p ( p − 1 ) p − 1 2 p ( p − 1 ) + 1 1 ≤ p 2 p + 1 1 ,
where the last inequality follows from p ≥ 2 p\geq2 p ≥ 2 , which gives ( p − 1 ) p − 1 ≥ 1 (p-1)^{p-1}\geq1 ( p − 1 ) p − 1 ≥ 1 and p ( p − 1 ) + 1 ≥ p + 1 p(p-1)+1\geq p+1 p ( p − 1 ) + 1 ≥ p + 1 . Consequently,
a t + 1 p p A t + 1 p − 1 ∥ z t + 1 − z t ∥ p p ≤ ω p ( z t + 1 , z t ) . \frac{a_{t+1}^p}{pA_{t+1}^{p-1}}\lVert z_{t+1}-z_t\rVert_p^p\leq\omega_p(z_{t+1},z_t). p A t + 1 p − 1 a t + 1 p ∥ z t + 1 − z t ∥ p p ≤ ω p ( z t + 1 , z t ) .
Combining (26) , (27) , and (29) , the terms involving ⟨ ∇ f p , λ ( y t + 1 ) , z t + 1 − z t ⟩ \langle\nabla f_{p,\lambda}(y_{t+1}),z_{t+1}-z_t\rangle ⟨ ∇ f p , λ ( y t + 1 ) , z t + 1 − z t ⟩ cancel. Taking conditional expectations, applying Theorem 1, and using ∥ z t − x ⋆ ∥ p ≤ 2 R \|z_t-x^\star\|_p\le2R ∥ z t − x ⋆ ∥ p ≤ 2 R , we obtain
A t + 1 M E [ f p , λ ( x t + 1 ) − f p , λ ( x ⋆ ) ] + E [ ω p ( x ⋆ , z t + 1 ) ] ≤ A t M E [ f p , λ ( x t ) − f p , λ ( x ⋆ ) ] + E [ ω p ( x ⋆ , z t ) ] + 2 R a t + 1 M δ t + 1 + A t + 1 ϵ M T + 2 q − 1 A t + 1 q M q σ t + 1 q . (30) \frac{A_{t+1}}{M}\mathbb{E}[f_{p,\lambda}(x_{t+1})-f_{p,\lambda}(x^\star)]+\mathbb{E}[\omega_p(x^\star,z_{t+1})]\le\frac{A_t}{M}\mathbb{E}[f_{p,\lambda}(x_t)-f_{p,\lambda}(x^\star)]+\mathbb{E}[\omega_p(x^\star,z_t)]
+\frac{2Ra_{t+1}}{M}\delta_{t+1}+\frac{A_{t+1}\epsilon}{MT}+\frac{2^{q-1}A_{t+1}}{qM^q}\sigma_{t+1}^q. \tag*{(30)} M A t + 1 E [ f p , λ ( x t + 1 ) − f p , λ ( x ⋆ )] + E [ ω p ( x ⋆ , z t + 1 )] ≤ M A t E [ f p , λ ( x t ) − f p , λ ( x ⋆ )] + E [ ω p ( x ⋆ , z t )] + M 2 R a t + 1 δ t + 1 + M T A t + 1 ϵ + q M q 2 q − 1 A t + 1 σ t + 1 q . (30)
To bound the last three terms, observe that
A k 1 / p − A k − 1 1 / p ≤ a k A k 1 / q − 1 [ 2 p + q − 1 ( p − 1 ) ] 1 / q . A_k^{1/p}-A_{k-1}^{1/p}\le\frac{a_k}{A_k^{1/q}}-\frac{1}{[2^{p+q-1}(p-1)]^{1/q}}. A k 1/ p − A k − 1 1/ p ≤ A k 1/ q a k − [ 2 p + q − 1 ( p − 1 ) ] 1/ q 1 .
Since A 0 = 0 A_0=0 A 0 = 0 , summing this inequality gives
A t + 1 a t + 1 = [ 2 p + q − 1 ( p − 1 ) ] 1 / q A t + 1 1 / p ≤ t + 1 ≤ T . \frac{A_{t+1}}{a_{t+1}}=[2^{p+q-1}(p-1)]^{1/q}A_{t+1}^{1/p}\le t+1\le T. a t + 1 A t + 1 = [ 2 p + q − 1 ( p − 1 ) ] 1/ q A t + 1 1/ p ≤ t + 1 ≤ T .
Substituting (24) , we therefore have
2 R a t + 1 M δ t + 1 + A t + 1 ϵ M T + 2 q − 1 A t + 1 q M q σ t + 1 q = 2 a t + 1 ϵ M + A t + 1 ϵ M T ≤ 3 a t + 1 ϵ M . \frac{2Ra_{t+1}}{M}\delta_{t+1}+\frac{A_{t+1}\epsilon}{MT}+\frac{2^{q-1}A_{t+1}}{qM^q}\sigma_{t+1}^q=\frac{2a_{t+1}\epsilon}{M}+\frac{A_{t+1}\epsilon}{MT}\le\frac{3a_{t+1}\epsilon}{M}. M 2 R a t + 1 δ t + 1 + M T A t + 1 ϵ + q M q 2 q − 1 A t + 1 σ t + 1 q = M 2 a t + 1 ϵ + M T A t + 1 ϵ ≤ M 3 a t + 1 ϵ .
Combining this with (30) proves (25) .
Theorem 5. Under the conditions of Lemma 7, let x ⋆ ∈ arg min x ∈ X f ( x ) x^\star\in\arg\min_{x\in\mathcal{X}}f(x) x ⋆ ∈ arg min x ∈ X f ( x ) . For any 0 < ϵ ≤ G R 0<\epsilon\le GR 0 < ϵ ≤ GR , with the parameter choices specified below, Algorithm 4 computes x T ∈ X x_T\in\mathcal{X} x T ∈ X such that E [ f ( x T ) ] ≤ f ( x ⋆ ) + ϵ \mathbb{E}[f(x_T)]\le f(x^\star)+\epsilon E [ f ( x T )] ≤ f ( x ⋆ ) + ϵ , using
T = O ( ( G R ϵ ) 2 p / ( p + 2 ) ) T=O\left(\left(\frac{GR}{\epsilon}\right)^{2p/(p+2)}\right) T = O ( ( ϵ GR ) 2 p / ( p + 2 ) ) outer iterations and N ∇ ^ f N_{\widehat{\nabla}f} N ∇ f stochastic subgradient computations, where
E [ N ∇ ^ f ] = O ( ( G R ϵ ) p ( p − 1 ) ( 1 + log G R ϵ ) ) . \mathbb{E}[N_{\widehat{\nabla}f}]=O\left(\left(\frac{GR}{\epsilon}\right)^{p(p-1)}\left(1+\log\frac{GR}{\epsilon}\right)\right). E [ N ∇ f ] = O ( ( ϵ GR ) p ( p − 1 ) ( 1 + log ϵ GR ) ) .
Proof. Set ϵ = ε / 5 \epsilon=\varepsilon/5 ϵ = ε /5 . Since A 0 = 0 A_0=0 A 0 = 0 , summing Lemma 7 from t = 0 t=0 t = 0 to T − 1 T-1 T − 1 gives
A T M E [ f p , λ ( x T ) − f p , λ ( x ⋆ ) ] + E [ ω p ( x ⋆ , z T ) ] ≤ ω p ( x ⋆ , z 0 ) + ∑ t = 0 T − 1 3 a t + 1 M ϵ = ω p ( x ⋆ , z 0 ) + 3 A T M ϵ . \frac{A_T}{M}\mathbb{E}[f_{p,\lambda}(x_T)-f_{p,\lambda}(x^\star)]+\mathbb{E}[\omega_p(x^\star,z_T)]\le\omega_p(x^\star,z_0)+\sum_{t=0}^{T-1}\frac{3a_{t+1}}{M}\epsilon=\omega_p(x^\star,z_0)+\frac{3A_T}{M}\epsilon. M A T E [ f p , λ ( x T ) − f p , λ ( x ⋆ )] + E [ ω p ( x ⋆ , z T )] ≤ ω p ( x ⋆ , z 0 ) + t = 0 ∑ T − 1 M 3 a t + 1 ϵ = ω p ( x ⋆ , z 0 ) + M 3 A T ϵ .
Using ω p ( x ⋆ , z T ) ≥ 0 \omega_p(x^\star,z_T)\ge0 ω p ( x ⋆ , z T ) ≥ 0 , we obtain
E [ f p , λ ( x T ) − f p , λ ( x ⋆ ) ] ≤ M A T ω p ( x ⋆ , z 0 ) + 3 ϵ . \mathbb{E}[f_{p,\lambda}(x_T)-f_{p,\lambda}(x^\star)]\le\frac{M}{A_T}\omega_p(x^\star,z_0)+3\epsilon. E [ f p , λ ( x T ) − f p , λ ( x ⋆ )] ≤ A T M ω p ( x ⋆ , z 0 ) + 3 ϵ .
By the G G G -Lipschitz property of f f f , for any y ∈ R d y\in\mathbb{R}^d y ∈ R d ,
0 ≤ f ( y ) − f p , λ ( y ) ≤ sup r ≥ 0 { G r − λ p r p } = G q q λ q − 1 . 0\le f(y)-f_{p,\lambda}(y)\le\sup_{r\ge0}\left\{Gr-\frac{\lambda}{p}r^p\right\}=\frac{G^q}{q\lambda^{q-1}}. 0 ≤ f ( y ) − f p , λ ( y ) ≤ r ≥ 0 sup { G r − p λ r p } = q λ q − 1 G q .
e n d s m i d = 1 {ends_mid=1} e n d s m i d = 1
Since f p , λ ( x ⋆ ) ≤ f ( x ⋆ ) f_{p,\lambda}(x^\star) \le f(x^\star) f p , λ ( x ⋆ ) ≤ f ( x ⋆ ) ,
E [ f ( x T ) ] − f ( x ⋆ ) ≤ M A T ω p ( x ⋆ , z 0 ) + G q q λ q − 1 + 3 ϵ . \mathbb{E}[f(x_T)] - f(x^\star) \le\frac{M}{A_T}\omega_p(x^\star,z_0) + \frac{G^q}{q\lambda^{q-1}} + 3\epsilon. E [ f ( x T )] − f ( x ⋆ ) ≤ A T M ω p ( x ⋆ , z 0 ) + q λ q − 1 G q + 3 ϵ .
Choose λ = ( G q q ϵ ) p − 1 \lambda= \left(\frac{G^q}{q\epsilon}\right)^{p-1} λ = ( q ϵ G q ) p − 1 . Then
G q q λ q − 1 = ϵ , L λ = ( p − 1 ) λ q − 1 G 2 − q = p − 1 q G 2 ϵ . \frac{G^q}{q\lambda^{q-1}} = \epsilon,\qquad L_\lambda= (p-1)\lambda^{q-1}G^{2-q} = \frac{p-1}{q}\frac{G^2}{\epsilon}. q λ q − 1 G q = ϵ , L λ = ( p − 1 ) λ q − 1 G 2 − q = q p − 1 ϵ G 2 .
By Lemma 9, we have
A T ≥ T p p p [ 2 p + q − 1 ( p − 1 ) ] p − 1 . A_T \ge\frac{T^p}{p^p[2^{p+q-1}(p-1)]^{p-1}}. A T ≥ p p [ 2 p + q − 1 ( p − 1 ) ] p − 1 T p .
Moreover, since z 0 = x i n i t z_0=x_{\mathrm{init}} z 0 = x init ,
ω p ( x ⋆ , z 0 ) = 1 p ∥ x ⋆ − x i n i t ∥ p p ≤ R p p . \omega_p(x^\star,z_0)=\frac{1}{p}\lVert x^\star-x_{\mathrm{init}}\rVert_p^p\le\frac{R^p}{p}. ω p ( x ⋆ , z 0 ) = p 1 ∥ x ⋆ − x init ∥ p p ≤ p R p .
Taking
M = 2 L λ p / 2 ( T ϵ ) ( p − 2 ) / 2 , M=2L_\lambda^{p/2}\left(\frac{T}{\epsilon}\right)^{(p-2)/2}, M = 2 L λ p /2 ( ϵ T ) ( p − 2 ) /2 ,
we therefore have
M A T ω p ( x ⋆ , z 0 ) ≤ 2 p p − 1 [ 2 p + q − 1 ( p − 1 ) ] p − 1 L λ p / 2 R p ϵ ( p − 2 ) / 2 T ( p + 2 ) / 2 . \frac{M}{A_T}\omega_p(x^\star,z_0)\le\frac{2p^{p-1}[2^{p+q-1}(p-1)]^{p-1}L_\lambda^{p/2}R^p}{\epsilon^{(p-2)/2}T^{(p+2)/2}}. A T M ω p ( x ⋆ , z 0 ) ≤ ϵ ( p − 2 ) /2 T ( p + 2 ) /2 2 p p − 1 [ 2 p + q − 1 ( p − 1 ) ] p − 1 L λ p /2 R p .
Thus it suffices to choose
T = ⌈ ( 2 p p − 1 [ 2 p + q − 1 ( p − 1 ) ] p − 1 ) 2 / ( p + 2 ) ( L λ R 2 ϵ ) p / ( p + 2 ) ⌉ . T=\left\lceil\left(2p^{p-1}[2^{p+q-1}(p-1)]^{p-1}\right)^{2/(p+2)}
\left(\frac{L_\lambda R^2}{\epsilon}\right)^{p/(p+2)}\right\rceil. T = ⌈ ( 2 p p − 1 [ 2 p + q − 1 ( p − 1 ) ] p − 1 ) 2/ ( p + 2 ) ( ϵ L λ R 2 ) p / ( p + 2 ) ⌉ .
Since L λ = Θ ( G 2 / ϵ ) L_\lambda=\Theta(G^2/\epsilon) L λ = Θ ( G 2 / ϵ ) , this gives
T = Θ ( ( G R ϵ ) 2 p / ( p + 2 ) ) , M A T ω p ( x ⋆ , z 0 ) ≤ ϵ . T=\Theta\left(\left(\frac{GR}{\epsilon}\right)^{2p/(p+2)}\right),\qquad
\frac{M}{A_T}\omega_p(x^\star,z_0)\le\epsilon. T = Θ ( ( ϵ GR ) 2 p / ( p + 2 ) ) , A T M ω p ( x ⋆ , z 0 ) ≤ ϵ .
Consequently,
E [ f ( x T ) ] − f ( x ⋆ ) ≤ ϵ + ϵ + 3 ϵ = 5 ϵ = ε . \mathbb{E}[f(x_T)]-f(x^\star)\le\epsilon+\epsilon+3\epsilon=5\epsilon=\varepsilon. E [ f ( x T )] − f ( x ⋆ ) ≤ ϵ + ϵ + 3 ϵ = 5 ϵ = ε .
To finish, we bound the expected total number of stochastic subgradient oracle calls. Recall that
δ t = ϵ 2 R , σ t q = q ϵ M 1 / ( p − 1 ) a t 2 q − 1 A t , t = 1 , … , T . \delta_t=\frac{\epsilon}{2R},\qquad
\sigma_t^q=\frac{q\epsilon M^{1/(p-1)}a_t}{2^{q-1}A_t},\qquad t=1,\ldots,T. δ t = 2 R ϵ , σ t q = 2 q − 1 A t q ϵ M 1/ ( p − 1 ) a t , t = 1 , … , T .
The choices of T T T and M M M above imply M = Θ ( ϵ T p R p ) M=\Theta\left(\frac{\epsilon T^p}{R^p}\right) M = Θ ( R p ϵ T p ) . Since p / q = p − 1 p/q=p-1 p / q = p − 1 and ( q − 1 ) ( p − 1 ) = 1 (q-1)(p-1)=1 ( q − 1 ) ( p − 1 ) = 1 ,
σ t p = q p − 1 2 ϵ p − 1 M ( a t A t ) p − 1 . \sigma_t^p=\frac{q^{p-1}}{2}\epsilon^{p-1}M\left(\frac{a_t}{A_t}\right)^{p-1}. σ t p = 2 q p − 1 ϵ p − 1 M ( A t a t ) p − 1 .
As shown in the proof of Lemma 7, A t / a t ≤ t ≤ T A_t/a_t\le t\le T A t / a t ≤ t ≤ T . Hence
σ t p ≥ q p − 1 ϵ p − 1 M 2 T p − 1 = Θ ( ϵ p T R p ) . \sigma_t^p\ge\frac{q^{p-1}\epsilon^{p-1}M}{2T^{p-1}}
=\Theta\left(\frac{\epsilon^pT}{R^p}\right). σ t p ≥ 2 T p − 1 q p − 1 ϵ p − 1 M = Θ ( R p ϵ p T ) .
Together with δ t = ϵ / ( 2 R ) \delta_t = \epsilon/(2R) δ t = ϵ / ( 2 R ) and T ≥ 1 T \ge1 T ≥ 1 , this implies, uniformly over t = 1 , … , T t = 1, \ldots, T t = 1 , … , T ,
min { δ t , σ t } = Θ ( ϵ / R ) , 1 + log G min { δ t , σ t } = O ( 1 + log G R ϵ ) . \min\{\delta_t,\sigma_t\} = \Theta(\epsilon/R), \qquad1 + \log\frac{G}{\min\{\delta_t,\sigma_t\}} = O\left(1 + \log\frac{GR}{\epsilon}\right). min { δ t , σ t } = Θ ( ϵ / R ) , 1 + log min { δ t , σ t } G = O ( 1 + log ϵ GR ) .
Also,
∑ t = 1 T G p σ t p = 2 G p q p − 1 ϵ p − 1 M ∑ t = 1 T ( A t a t ) p − 1 ≤ 2 G p T p q p − 1 ϵ p − 1 M = O ( ( G R ϵ ) p ) . \sum_{t=1}^{T} \frac{G^p}{\sigma_t^p} = \frac{2G^p}{q^{p-1}\epsilon^{p-1}M}\sum_{t=1}^{T}\left(\frac{A_t}{a_t}\right)^{p-1} \le\frac{2G^pT^p}{q^{p-1}\epsilon^{p-1}M} = O\left(\left(\frac{GR}{\epsilon}\right)^p\right). t = 1 ∑ T σ t p G p = q p − 1 ϵ p − 1 M 2 G p t = 1 ∑ T ( a t A t ) p − 1 ≤ q p − 1 ϵ p − 1 M 2 G p T p = O ( ( ϵ GR ) p ) .
Thus, by Theorem 1, the summation of stochastic gradient oracle is
E [ N ∇ ^ f ] = O ( T + ∑ t = 1 T G p σ t p ( G min { δ t , σ t } ) p ( p − 2 ) ( 1 + log G min { δ t , σ t } ) ) = O ( T + ( G R ϵ ) p ( p − 1 ) ( 1 + log G R ϵ ) ) = O ( ( G R ϵ ) p ( p − 1 ) ( 1 + log G R ϵ ) ) , \begin{aligned}
\mathbb{E}[N_{\widehat{\nabla} f}] = O\left(T + \sum_{t=1}^{T}\frac{G^p}{\sigma_t^p}\left(\frac{G}{\min\{\delta_t,\sigma_t\}}\right)^{p(p-2)}\left(1+\log\frac{G}{\min\{\delta_t,\sigma_t\}}\right)\right) \\
&= O\left(T+\left(\frac{GR}{\epsilon}\right)^{p(p-1)}\left(1+\log\frac{GR}{\epsilon}\right)\right) \\
&= O\left(\left(\frac{GR}{\epsilon}\right)^{p(p-1)}\left(1+\log\frac{GR}{\epsilon}\right)\right),
\end{aligned} E [ N ∇ f ] = O ( T + t = 1 ∑ T σ t p G p ( min { δ t , σ t } G ) p ( p − 2 ) ( 1 + log min { δ t , σ t } G ) ) = O ( T + ( ϵ GR ) p ( p − 1 ) ( 1 + log ϵ GR ) ) = O ( ( ϵ GR ) p ( p − 1 ) ( 1 + log ϵ GR ) ) ,
where the last equality uses ϵ = ε / 5 \epsilon= \varepsilon/5 ϵ = ε /5 and the bound on T T T .
Convex Composite Minimization
Consider the convex composite optimization problem
min x ∈ X Ψ ( x ) : = Λ ( x ) + f ( x ) , (31) \min_{x\in\mathcal{X}} \Psi(x) := \Lambda(x) + f(x), \tag*{(31)} x ∈ X min Ψ ( x ) := Λ ( x ) + f ( x ) , (31)
where Λ \Lambda Λ is ( L , κ ) (L,\kappa) ( L , κ ) -weakly smooth with respect to ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p and f f f satisfies Assumption 1. We assume that there exists x ⋆ ∈ arg min x ∈ X Ψ ( x ) x^\star\in\arg\min_{x\in\mathcal{X}} \Psi(x) x ⋆ ∈ arg min x ∈ X Ψ ( x ) such that ∥ x 0 − x ⋆ ∥ p ≤ R \|x_0-x^\star\|_p \le R ∥ x 0 − x ⋆ ∥ p ≤ R . A closely related setting was studied in [19 ] , who considered complementary composite minimization with Λ \Lambda Λ being ( L , κ ) (L,\kappa) ( L , κ ) -weakly smooth and f f f being uniformly convex. Their generalized AGD+ framework admits an ϵ \epsilon ϵ -additive optimality gap in O ( ( L R κ / ϵ ) p κ p − p + κ ) O\left((LR^\kappa/\epsilon)^{\frac{p}{\kappa p-p+\kappa}}\right) O ( ( L R κ / ϵ ) κ p − p + κ p ) iterations when specialized to ℓ p \ell_p ℓ p geometry.
In this section, we extend the stochastic composite AGD framework of [2 ] to the ℓ p \ell_p ℓ p setting using Bregman proximal steps and our reduced-bias gradient estimator. Our method handles a nonsmooth stochastic component f f f without requiring uniform convexity, while preserving the accelerated outer-loop structure.
Lemma 10. (Lemma 1 of [DG24]) Suppose Λ \Lambda Λ is ( L , κ ) (L,\kappa) ( L , κ ) -weakly smooth with respect to ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p for some κ ∈ ( 1 , 2 ] \kappa\in(1,2] κ ∈ ( 1 , 2 ] . Then, for δ > 0 \delta> 0 δ > 0 , if M ≥ ( 2 ( p − κ ) p κ δ ) p − κ κ L κ p κ M \ge\left(\frac{2(p-\kappa)}{p\kappa\delta}\right)^{\frac{p-\kappa}{\kappa}}L_\kappa^{\frac{p}{\kappa}} M ≥ ( p κ δ 2 ( p − κ ) ) κ p − κ L κ κ p , then for all x , y ∈ X x,y \in\mathcal{X} x , y ∈ X , it holds that
Λ ( y ) ≤ Λ ( x ) + ⟨ ∇ Λ ( x ) , y − x ⟩ + M p ∥ y − x ∥ p p + δ 2 . \Lambda(y) \le\Lambda(x) + \langle\nabla\Lambda(x),y-x\rangle+ \frac{M}{p}\|y-x\|_p^p + \frac{\delta}{2}. Λ ( y ) ≤ Λ ( x ) + ⟨ ∇Λ ( x ) , y − x ⟩ + p M ∥ y − x ∥ p p + 2 δ .
Definition 4. Let D h ( z , y ) : = h ( z ) − h ( y ) − ⟨ ∇ h ( y ) , z − y ⟩ D_h(z,y) := h(z) - h(y) - \langle\nabla h(y),z-y\rangle D h ( z , y ) := h ( z ) − h ( y ) − ⟨ ∇ h ( y ) , z − y ⟩ . For F ( z ) : = f ( z ) + λ D h ( z , y ) F(z) := f(z) + \lambda D_h(z,y) F ( z ) := f ( z ) + λ D h ( z , y ) , define
P f , λ ϵ , h ( y ) : = any x ∈ X ‾ such that E [ F ( x ) ] ≤ min z ∈ X ‾ F ( z ) + ϵ . (32) P_{f,\lambda}^{\epsilon,h}(y) := \text{any } x \in\overline{\mathcal{X}} \text{ such that } \mathbb{E}[F(x)] \le\min_{z\in\overline{\mathcal{X}}} F(z) + \epsilon. \tag*{(32)} P f , λ ϵ , h ( y ) := any x ∈ X such that E [ F ( x )] ≤ z ∈ X min F ( z ) + ϵ . (32)
where X ‾ = X ∩ B p ( v 0 , R ) \overline{\mathcal{X}} = \mathcal{X} \cap B_p(v_0,R) X = X ∩ B p ( v 0 , R )
Algorithm 5 Stochastic composite accelerated gradient descent
Input: A problem of the form (31) with Λ \Lambda Λ , f f f , ∇ Λ \nabla\Lambda ∇Λ , ∇ ^ f \widehat{\nabla}f ∇ f .
Parameters: Step size parameters β k = M γ k p − 1 \beta_k=M\gamma_k^{p-1} β k = M γ k p − 1 and γ k = p k + p − 1 \gamma_k=\frac{p}{k+p-1} γ k = k + p − 1 p , iteration number N N N , approximation parameters { ϵ k , δ k , σ k q } \{\epsilon_k,\delta_k,\sigma_k^q\} { ϵ k , δ k , σ k q } , and x 0 = v 0 x_0=v_0 x 0 = v 0 where x 0 ∈ X x_0\in X x 0 ∈ X satisfying ∥ x 0 − x ∗ ∥ p ≤ R \lVert x_0-x^\ast\rVert_p\le R ∥ x 0 − x ∗ ∥ p ≤ R .
Set h ( z ) = 2 p − 2 p ∥ z − v 0 ∥ p p h(z)=\frac{2^{p-2}}{p}\lVert z-v_0\rVert_p^p h ( z ) = p 2 p − 2 ∥ z − v 0 ∥ p p .
for k = 1 , 2 , ⋯ , N k=1,2,\cdots,N k = 1 , 2 , ⋯ , N do
y k = ( 1 − γ k ) x k − 1 + γ k v k − 1 y_k=(1-\gamma_k)x_{k-1}+\gamma_kv_{k-1} y k = ( 1 − γ k ) x k − 1 + γ k v k − 1
v ˉ k = P Λ ˉ k + f , β k ϵ k , h ( v k − 1 ) \bar{v}_k=P_{\bar{\Lambda}_k+f,\beta_k}^{\epsilon_k,h}(v_{k-1}) v ˉ k = P Λ ˉ k + f , β k ϵ k , h ( v k − 1 ) for Λ ˉ k ( v ) : = Λ ( y k ) + ⟨ ∇ Λ ( y k ) , v − y k ⟩ \bar{\Lambda}_k(v):=\Lambda(y_k)+\langle\nabla\Lambda(y_k),v-y_k\rangle Λ ˉ k ( v ) := Λ ( y k ) + ⟨ ∇Λ ( y k ) , v − y k ⟩
ψ k ( z ) : = β k D h ( z , v k − 1 ) + Λ ˉ k ( z ) \psi_k(z):=\beta_kD_h(z,v_{k-1})+\bar{\Lambda}_k(z) ψ k ( z ) := β k D h ( z , v k − 1 ) + Λ ˉ k ( z )
g ^ k = − 1 β k ℓ p -MorGradEst ( ∇ ^ f , ψ k , β k , β k δ k , β k q σ k q , X ˉ ) \widehat{g}_k=-\frac{1}{\beta_k}\ell_p\text{-MorGradEst}\left(\widehat{\nabla}f,\psi_k, \beta_k,\beta_k\delta_k,\beta_k^q\sigma_k^q,\bar{X}\right) g k = − β k 1 ℓ p -MorGradEst ( ∇ f , ψ k , β k , β k δ k , β k q σ k q , X ˉ )
v k = arg min v ∈ X ˉ { h ( v ) − ⟨ g ^ k , v ⟩ } v_k=\underset{v\in\bar{X}}{\arg\min}\{h(v)-\langle\widehat{g}_k,v\rangle\} v k = v ∈ X ˉ arg min { h ( v ) − ⟨ g k , v ⟩}
x k = ( 1 − γ k ) x k − 1 + γ k v ˉ k x_k=(1-\gamma_k)x_{k-1}+\gamma_k\bar{v}_k x k = ( 1 − γ k ) x k − 1 + γ k v ˉ k
end for
return x N x_N x N
Lemma 11. Let f f f and ∇ ^ f \widehat{\nabla}f ∇ f satisfy Assumption 1, p > 2 p>2 p > 2 , and let
v k ⋆ = arg min z ∈ X ˉ { f ( z ) + ψ k ( z ) } . v_k^\star=\underset{z\in\bar{X}}{\arg\min}\{f(z)+\psi_k(z)\}. v k ⋆ = z ∈ X ˉ arg min { f ( z ) + ψ k ( z )} . The procedure
g ^ k = − 1 β k ℓ p -MorGradEst ( ∇ ^ f , ψ k , 2 p − 2 β k , β k δ k , β k q σ k q , X ˉ ) \widehat{g}_k=-\frac{1}{\beta_k}\ell_p\text{-MorGradEst}\left(\widehat{\nabla}f,\psi_k,2^{p-2}\beta_k,\beta_k\delta_k,\beta_k^q\sigma_k^q,\bar{X}\right) g k = − β k 1 ℓ p -MorGradEst ( ∇ f , ψ k , 2 p − 2 β k , β k δ k , β k q σ k q , X ˉ ) outputs an estimator satisfying
∥ E g ^ k − ∇ h ( v k ⋆ ) ∥ q ≤ δ k and E ∥ g ^ k − ∇ h ( v k ⋆ ) ∥ q q ≤ σ k q , \left\lVert\mathbb{E}\widehat{g}_k-\nabla h(v_k^\star)\right\rVert_q\le\delta_k
\quad\text{and}\quad
\mathbb{E}\left\lVert\widehat{g}_k-\nabla h(v_k^\star)\right\rVert_q^q\le\sigma_k^q, ∥ E g k − ∇ h ( v k ⋆ ) ∥ q ≤ δ k and E ∥ g k − ∇ h ( v k ⋆ ) ∥ q q ≤ σ k q , using N ∇ ^ f N_{\widehat{\nabla}f} N ∇ f stochastic-gradient computations, where
E N ∇ ^ f ≤ N ( 1 + 3 2 ⌊ log 2 T max ⌋ ) . \mathbb{E}N_{\widehat{\nabla}f}\le N\left(1+\frac{3}{2}\left\lfloor\log_2 T_{\max}\right\rfloor\right). E N ∇ f ≤ N ( 1 + 2 3 ⌊ log 2 T m a x ⌋ ) . The parameters are
T max = ⌈ ( 2 p − 1 ( p − 1 ) c 1 / p R p p − 2 ( G / β k ) 1 / ( p − 1 ) min { δ k , σ k } ) p ( p − 1 ) ⌉ T_{\max}=\left\lceil\left(\frac{2^{p-1}(p-1)c^{1/p}Rp^{p-2}(G/\beta_k)^{1/(p-1)}}{\min\{\delta_k,\sigma_k\}}\right)^{p(p-1)}\right\rceil T m a x = ⌈ ( min { δ k , σ k } 2 p − 1 ( p − 1 ) c 1/ p R p p − 2 ( G / β k ) 1/ ( p − 1 ) ) p ( p − 1 ) ⌉ and
N = ⌈ 2 p 2 + p + 1 ( p − 1 ) p c σ k p ( 1 + 2 1 / ( p − 1 ) 2 1 − 2 − ( p − 2 ) / ( p − 1 ) 2 ) p − 1 R p ( p − 2 ) ( G β k ) q T max ( p − 2 ) / ( p − 1 ) ⌉ , N=\left\lceil\frac{2^{p^2+p+1}(p-1)^pc}{\sigma_k^p}
\left(\frac{1+2^{1/(p-1)^2}}{1-2^{-(p-2)/(p-1)^2}}\right)^{p-1}
R^{p(p-2)}
\left(\frac{G}{\beta_k}\right)^q
T_{\max}^{(p-2)/(p-1)}\right\rceil, N = σ k p 2 p 2 + p + 1 ( p − 1 ) p c ( 1 − 2 − ( p − 2 ) / ( p − 1 ) 2 1 + 2 1/ ( p − 1 ) 2 ) p − 1 R p ( p − 2 ) ( β k G ) q T m a x ( p − 2 ) / ( p − 1 ) , where c c c is the constant in the ODCp _p p guarantee.
Proof. The proof follows the MLMC argument in Lemma 3 and the averaging argument in Theorem 1. Since f + ψ k f+\psi_k f + ψ k is β k \beta_k β k -uniformly convex, the ODC iterates satisfy
E ∥ x j − v k ⋆ ∥ p p ≤ c ( G / β k ) q 2 − j / ( p − 1 ) . \mathbb{E}\lVert x_j-v_k^\star\rVert_p^p\le c(G/\beta_k)^q2^{-j/(p-1)}. E ∥ x j − v k ⋆ ∥ p p ≤ c ( G / β k ) q 2 − j / ( p − 1 ) .
Moreover, on B p ( v 0 , R ) B_p(v_0,R) B p ( v 0 , R ) ,
∥ ∇ h ( u ) − ∇ h ( v ) ∥ q ≤ 2 p − 2 ( p − 1 ) R p − 2 ∥ u − v ∥ p . \|\nabla h(u)-\nabla h(v)\|_q \le2^{p-2}(p-1)R^{p-2}\|u-v\|_p. ∥∇ h ( u ) − ∇ h ( v ) ∥ q ≤ 2 p − 2 ( p − 1 ) R p − 2 ∥ u − v ∥ p .
Consequently, the original MLMC estimator g ′ g' g ′ , after normalization Z = − g ′ / β k Z=-g'/\beta_k Z = − g ′ / β k , satisfies
∥ E Z − ∇ h ( v k ⋆ ) ∥ q ≤ 2 p − 2 + 1 / [ p ( p − 1 ) ] ( p − 1 ) c 1 / p R p − 2 ( G / β k ) 1 / ( p − 1 ) T max − 1 / [ p ( p − 1 ) ] . \|\mathbb{E}Z-\nabla h(v_k^\star)\|_q \le2^{p-2+1/[p(p-1)]}(p-1)c^{1/p}R^{p-2}(G/\beta_k)^{1/(p-1)}T_{\max}^{-1/[p(p-1)]}. ∥ E Z − ∇ h ( v k ⋆ ) ∥ q ≤ 2 p − 2 + 1/ [ p ( p − 1 )] ( p − 1 ) c 1/ p R p − 2 ( G / β k ) 1/ ( p − 1 ) T m a x − 1/ [ p ( p − 1 )] .
Evaluating the geometric series in the same moment calculation gives
E ∥ Z − E Z ∥ q q ≤ 2 3 q − 2 ( 1 + 2 1 / ( p − 1 ) 2 ) 1 − 2 − ( p − 2 ) / ( p − 1 ) 2 [ 2 p − 2 ( p − 1 ) c 1 / p R p − 2 ( G / β k ) 1 / ( p − 1 ) ] q T max ( p − 2 ) / ( p − 1 ) 2 . \mathbb{E}\|Z-\mathbb{E}Z\|_q^q \le\frac{2^{3q-2}(1+2^{1/(p-1)^2})}{1-2^{-(p-2)/(p-1)^2}}\left[2^{p-2}(p-1)c^{1/p}R^{p-2}(G/\beta_k)^{1/(p-1)}\right]^q T_{\max}^{(p-2)/(p-1)^2}. E ∥ Z − E Z ∥ q q ≤ 1 − 2 − ( p − 2 ) / ( p − 1 ) 2 2 3 q − 2 ( 1 + 2 1/ ( p − 1 ) 2 ) [ 2 p − 2 ( p − 1 ) c 1/ p R p − 2 ( G / β k ) 1/ ( p − 1 ) ] q T m a x ( p − 2 ) / ( p − 1 ) 2 .
The choice of T max T_{\max} T m a x makes the bias at most min { δ k , σ k } / 2 \min\{\delta_k,\sigma_k\}/2 min { δ k , σ k } /2 , and the choice of N N N makes the centered q q q th moment of the averaged estimator at most ( σ k / 2 ) q (\sigma_k/2)^q ( σ k /2 ) q . Combining these bounds as in Theorem 1 proves the guarantees. Finally, each MLMC sample uses at most
1 + ∑ j = 1 ⌊ log 2 T max ⌋ 2 − j ( 2 j + 2 j − 1 ) = 1 + 3 2 ⌊ log 2 T max ⌋ 1+\sum_{j=1}^{\lfloor\log_2 T_{\max}\rfloor}2^{-j}(2^j+2^{j-1})=1+\frac{3}{2}\lfloor\log_2 T_{\max}\rfloor 1 + j = 1 ∑ ⌊ l o g 2 T m a x ⌋ 2 − j ( 2 j + 2 j − 1 ) = 1 + 2 3 ⌊ log 2 T m a x ⌋
oracle calls in expectation, yielding the stated cost.
Proposition 6. Suppose x ⋆ x^\star x ⋆ minimizes Ψ \Psi Ψ such that ∥ x 0 − x ⋆ ∥ p ≤ R \|x_0-x^\star\|_p\le R ∥ x 0 − x ⋆ ∥ p ≤ R , and let the iterates be generated by Algorithm 5. Suppose M M M satisfies Lemma 10 with approximation error η > 0 \eta>0 η > 0 . Choose
σ k q = q R p 2 N , δ k = R p − 1 4 N , ϵ k = p p − 2 M R p ( k + p − 1 ) p − 1 N , \sigma_k^q=\frac{qR^p}{2N},\qquad\delta_k=\frac{R^{p-1}}{4N},\qquad\epsilon_k=\frac{p^{p-2}MR^p}{(k+p-1)^{p-1}N}, σ k q = 2 N q R p , δ k = 4 N R p − 1 , ϵ k = ( k + p − 1 ) p − 1 N p p − 2 M R p , where also γ k = p / ( k + p − 1 ) \gamma_k=p/(k+p-1) γ k = p / ( k + p − 1 ) and β k = M γ k p − 1 \beta_k=M\gamma_k^{p-1} β k = M γ k p − 1 . Further, for all k ≥ 2 k\ge2 k ≥ 2 , let Γ k = ( 1 − γ k ) Γ k − 1 \Gamma_k=(1-\gamma_k)\Gamma_{k-1} Γ k = ( 1 − γ k ) Γ k − 1 , where Γ 1 = 1 \Gamma_1=1 Γ 1 = 1 . Then
E [ Ψ ( x N ) − Ψ ( x ⋆ ) ] ≤ ( 2 p − 1 + 2 + 1 p ) M p p R p ( N + p − 1 ) p + Γ N ∑ k = 1 N η 2 Γ k \mathbb{E}[\Psi(x_N)-\Psi(x^\star)]\le\left(2^{p-1}+2+\frac{1}{p}\right)\frac{Mp^pR^p}{(N+p-1)^p}+\Gamma_N\sum_{k=1}^{N}\frac{\eta}{2\Gamma_k} E [ Ψ ( x N ) − Ψ ( x ⋆ )] ≤ ( 2 p − 1 + 2 + p 1 ) ( N + p − 1 ) p M p p R p + Γ N k = 1 ∑ N 2 Γ k η = ( 2 p − 1 + 2 + 1 p ) M p p R p ( N + p − 1 ) p + η ( N + p ) 2 ( p + 1 ) . =\left(2^{p-1}+2+\frac{1}{p}\right)\frac{Mp^pR^p}{(N+p-1)^p}+\frac{\eta(N+p)}{2(p+1)}. = ( 2 p − 1 + 2 + p 1 ) ( N + p − 1 ) p M p p R p + 2 ( p + 1 ) η ( N + p ) .
Proof. Let
Φ k ( v ) = Λ ˉ k ( v ) + f ( v ) + β k D h ( v , v k − 1 ) , v k ⋆ = arg min v ∈ X ˉ Φ k ( v ) . \Phi_k(v)=\bar{\Lambda}_k(v)+f(v)+\beta_kD_h(v,v_{k-1}),\qquad v_k^\star=\mathop{\arg\min}_{v\in\bar{X}}\Phi_k(v). Φ k ( v ) = Λ ˉ k ( v ) + f ( v ) + β k D h ( v , v k − 1 ) , v k ⋆ = arg min v ∈ X ˉ Φ k ( v ) .
Recall that
D h ( u , v ) ≥ 1 p ∥ u − v ∥ p p . D_h(u,v)\ge\frac{1}{p}\|u-v\|_p^p. D h ( u , v ) ≥ p 1 ∥ u − v ∥ p p .
By Lemma 10, the definitions of x k , y k x_k,y_k x k , y k , and β k = M γ k p − 1 \beta_k=M\gamma_k^{p-1} β k = M γ k p − 1 , we obtain
Λ ( x k ) ≤ ( 1 − γ k ) Λ ( x k − 1 ) + γ k Λ ˉ k ( v ˉ k ) + M γ k p p ∥ v ˉ k − v k − 1 ∥ p p + η 2 ≤ ( 1 − γ k ) Λ ( x k − 1 ) + γ k [ Λ ˉ k ( v ˉ k ) + β k D h ( v ˉ k , v k − 1 ) ] + η 2 . \begin{aligned}
\Lambda(x_k)\le(1-\gamma_k)\Lambda(x_{k-1})+\gamma_k\bar{\Lambda}_k(\bar{v}_k)+\frac{M\gamma_k^p}{p}\|\bar{v}_k-v_{k-1}\|_p^p+\frac{\eta}{2} \\
&\le(1-\gamma_k)\Lambda(x_{k-1})+\gamma_k\left[\bar{\Lambda}_k(\bar{v}_k)+\beta_kD_h(\bar{v}_k,v_{k-1})\right]+\frac{\eta}{2}.
\end{aligned} Λ ( x k ) ≤ ( 1 − γ k ) Λ ( x k − 1 ) + γ k Λ ˉ k ( v ˉ k ) + p M γ k p ∥ v ˉ k − v k − 1 ∥ p p + 2 η ≤ ( 1 − γ k ) Λ ( x k − 1 ) + γ k [ Λ ˉ k ( v ˉ k ) + β k D h ( v ˉ k , v k − 1 ) ] + 2 η .
Adding the convexity inequality for f f f gives
Ψ ( x k ) ≤ ( 1 − γ k ) Ψ ( x k − 1 ) + γ k Φ k ( v ˉ k ) + η 2 . \Psi(x_k)\le(1-\gamma_k)\Psi(x_{k-1})+\gamma_k\Phi_k(\bar{v}_k)+\frac{\eta}{2}. Ψ ( x k ) ≤ ( 1 − γ k ) Ψ ( x k − 1 ) + γ k Φ k ( v ˉ k ) + 2 η .
For any fixed u ∈ X ˉ u \in\bar{X} u ∈ X ˉ , convexity of Λ \Lambda Λ and optimality of v k ⋆ v_k^\star v k ⋆ give
Ψ ( u ) ≥ Φ k ( u ) − β k D h ( u , v k − 1 ) , Φ k ( u ) ≥ Φ k ( v k ⋆ ) + β k D h ( u , v k ⋆ ) . \Psi(u) \ge\Phi_k(u)-\beta_k D_h(u,v_{k-1}), \qquad\Phi_k(u) \ge\Phi_k(v_k^\star)+\beta_k D_h(u,v_k^\star). Ψ ( u ) ≥ Φ k ( u ) − β k D h ( u , v k − 1 ) , Φ k ( u ) ≥ Φ k ( v k ⋆ ) + β k D h ( u , v k ⋆ ) .
Consequently,
Ψ ( x k ) − Ψ ( u ) ≤ ( 1 − γ k ) [ Ψ ( x k − 1 ) − Ψ ( u ) ] + γ k [ Φ k ( v ˉ k ) − Φ k ( v k ⋆ ) ] \Psi(x_k)-\Psi(u) \le(1-\gamma_k)[\Psi(x_{k-1})-\Psi(u)]+\gamma_k[\Phi_k(\bar{v}_k)-\Phi_k(v_k^\star)] Ψ ( x k ) − Ψ ( u ) ≤ ( 1 − γ k ) [ Ψ ( x k − 1 ) − Ψ ( u )] + γ k [ Φ k ( v ˉ k ) − Φ k ( v k ⋆ )]
+ γ k β k [ D h ( u , v k − 1 ) − D h ( u , v k ⋆ ) ] + η 2 . \qquad+\gamma_k\beta_k[D_h(u,v_{k-1})-D_h(u,v_k^\star)]+\frac{\eta}{2}. + γ k β k [ D h ( u , v k − 1 ) − D h ( u , v k ⋆ )] + 2 η .
Now, define
D 0 = D h ( u , v 0 ) , D k = E [ h ( u ) − h ( v k ) − ⟨ g ^ k , u − v k ⟩ ] , k ≥ 1. D_0=D_h(u,v_0), \qquad D_k=\mathbb{E}\left[h(u)-h(v_k)-\langle\hat{g}_k,u-v_k\rangle\right], \qquad k\ge1. D 0 = D h ( u , v 0 ) , D k = E [ h ( u ) − h ( v k ) − ⟨ g ^ k , u − v k ⟩ ] , k ≥ 1.
Optimality of v k v_k v k in the mirror update implies
h ( u ) − h ( v k ) − ⟨ g ^ k , u − v k ⟩ ≥ D h ( u , v k ) ≥ 0. h(u)-h(v_k)-\langle\hat{g}_k,u-v_k\rangle\ge D_h(u,v_k)\ge0. h ( u ) − h ( v k ) − ⟨ g ^ k , u − v k ⟩ ≥ D h ( u , v k ) ≥ 0.
Moreover, Young’s inequality and D h ( v k , v k ⋆ ) ≥ ∥ v k − v k ⋆ ∥ p p / p D_h(v_k,v_k^\star)\ge\lVert v_k-v_k^\star\rVert_p^p/p D h ( v k , v k ⋆ ) ≥ ∥ v k − v k ⋆ ∥ p p / p imply
h ( u ) − h ( v k ) − ⟨ g ^ k , u − v k ⟩ ≤ D h ( u , v k ⋆ ) + ⟨ g ^ k − ∇ h ( v k ⋆ ) , v k ⋆ − u ⟩ + 1 q ∥ g ^ k − ∇ h ( v k ⋆ ) ∥ q q . \begin{aligned}
h(u)-h(v_k)-\langle\hat{g}_k,u-v_k\rangle\\
&\le D_h(u,v_k^\star)+\langle\hat{g}_k-\nabla h(v_k^\star),v_k^\star-u\rangle+\frac{1}{q}\lVert\hat{g}_k-\nabla h(v_k^\star)\rVert_q^q.
\end{aligned} h ( u ) − h ( v k ) − ⟨ g ^ k , u − v k ⟩ ≤ D h ( u , v k ⋆ ) + ⟨ g ^ k − ∇ h ( v k ⋆ ) , v k ⋆ − u ⟩ + q 1 ∥ g ^ k − ∇ h ( v k ⋆ ) ∥ q q .
Conditioning on F k − 1 \mathcal{F}_{k-1} F k − 1 and applying Lemma 11, together with ∥ v k ⋆ − u ∥ p ≤ 2 R \lVert v_k^\star-u\rVert_p\le2R ∥ v k ⋆ − u ∥ p ≤ 2 R , therefore gives
D k ≤ E D h ( u , v k ⋆ ) + 2 R δ k + σ k q q = E D h ( u , v k ⋆ ) + R p N . D_k\le\mathbb{E}D_h(u,v_k^\star)+2R\delta_k+\frac{\sigma_k^q}{q}=\mathbb{E}D_h(u,v_k^\star)+\frac{R^p}{N}. D k ≤ E D h ( u , v k ⋆ ) + 2 R δ k + q σ k q = E D h ( u , v k ⋆ ) + N R p .
Also, E D h ( u , v k − 1 ) ≤ D k − 1 \mathbb{E}D_h(u,v_{k-1})\le D_{k-1} E D h ( u , v k − 1 ) ≤ D k − 1 . By the definition of the approximate proximal step,
E [ Φ k ( v ˉ k ) − Φ k ( v k ⋆ ) ] ≤ ϵ k . \mathbb{E}[\Phi_k(\bar{v}_k)-\Phi_k(v_k^\star)]\le\epsilon_k. E [ Φ k ( v ˉ k ) − Φ k ( v k ⋆ )] ≤ ϵ k .
Taking expectations in the preceding recursion yields
E [ Ψ ( x k ) − Ψ ( u ) ] ≤ ( 1 − γ k ) E [ Ψ ( x k − 1 ) − Ψ ( u ) ] + γ k ϵ k \mathbb{E}[\Psi(x_k)-\Psi(u)]\le(1-\gamma_k)\mathbb{E}[\Psi(x_{k-1})-\Psi(u)]+\gamma_k\epsilon_k E [ Ψ ( x k ) − Ψ ( u )] ≤ ( 1 − γ k ) E [ Ψ ( x k − 1 ) − Ψ ( u )] + γ k ϵ k
+ γ k β k ( D k − 1 − D k ) + γ k β k R p N + η 2 . \qquad+\gamma_k\beta_k(D_{k-1}-D_k)+\frac{\gamma_k\beta_k R^p}{N}+\frac{\eta}{2}. + γ k β k ( D k − 1 − D k ) + N γ k β k R p + 2 η .
For u , v ∈ X ˉ u,v\in\bar{X} u , v ∈ X ˉ , Hölder’s inequality gives
D h ( u , v ) ≤ 2 p − 2 p R p + 2 p − 2 ( p − 1 ) p R p + 2 p − 2 R p = 2 p − 1 R p . D_h(u,v)\le\frac{2^{p-2}}{p}R^p+\frac{2^{p-2}(p-1)}{p}R^p+2^{p-2}R^p=2^{p-1}R^p. D h ( u , v ) ≤ p 2 p − 2 R p + p 2 p − 2 ( p − 1 ) R p + 2 p − 2 R p = 2 p − 1 R p .
Hence, including k = 0 k=0 k = 0 ,
0 ≤ D k ≤ ( 2 p − 1 + 1 ) R p . 0\le D_k\le(2^{p-1}+1)R^p. 0 ≤ D k ≤ ( 2 p − 1 + 1 ) R p .
Since γ 1 = 1 \gamma_1=1 γ 1 = 1 , dividing the recursion by Γ k \Gamma_k Γ k and summing gives, with a k = γ k β k / Γ k a_k=\gamma_k\beta_k/\Gamma_k a k = γ k β k / Γ k ,
E [ Ψ ( x N ) − Ψ ( u ) ] ≤ Γ N ∑ k = 1 N a k ( D k − 1 − D k ) \mathbb{E}[\Psi(x_N)-\Psi(u)]\le\Gamma_N\sum_{k=1}^{N}a_k(D_{k-1}-D_k) E [ Ψ ( x N ) − Ψ ( u )] ≤ Γ N k = 1 ∑ N a k ( D k − 1 − D k )
+ ( 1 + 1 / p ) R p Γ N N ∑ k = 1 N a k + Γ N ∑ k = 1 N η 2 Γ k , \qquad+\frac{(1+1/p)R^p\Gamma_N}{N}\sum_{k=1}^{N}a_k+\Gamma_N\sum_{k=1}^{N}\frac{\eta}{2\Gamma_k}, + N ( 1 + 1/ p ) R p Γ N k = 1 ∑ N a k + Γ N k = 1 ∑ N 2 Γ k η ,
where we used ϵ k = β k R p / ( p N ) \epsilon_k = \beta_k R^p/(pN) ϵ k = β k R p / ( pN ) . By Bernoulli’s inequality,
a k + 1 a k = k + p k ( 1 − 1 k + p ) p ≥ 1. \frac{a_{k+1}}{a_k} = \frac{k+p}{k}\left(1-\frac{1}{k+p}\right)^p \ge1. a k a k + 1 = k k + p ( 1 − k + p 1 ) p ≥ 1.
Therefore,
∑ k = 1 N a k ( D k − 1 − D k ) = a 1 D 0 + ∑ k = 1 N − 1 ( a k + 1 − a k ) D k − a N D N ≤ ( 2 p − 1 + 1 ) R p a N , \sum_{k=1}^{N} a_k(D_{k-1}-D_k) = a_1D_0+\sum_{k=1}^{N-1}(a_{k+1}-a_k)D_k-a_ND_N
\le(2^{p-1}+1)R^p a_N, k = 1 ∑ N a k ( D k − 1 − D k ) = a 1 D 0 + k = 1 ∑ N − 1 ( a k + 1 − a k ) D k − a N D N ≤ ( 2 p − 1 + 1 ) R p a N ,
and ∑ k = 1 N a k ≤ N a N \sum_{k=1}^{N}a_k\le Na_N ∑ k = 1 N a k ≤ N a N . Using
Γ N a N = γ N β N = M γ N p = M p p ( N + p − 1 ) p , \Gamma_N a_N=\gamma_N\beta_N=M\gamma_N^p=\frac{Mp^p}{(N+p-1)^p}, Γ N a N = γ N β N = M γ N p = ( N + p − 1 ) p M p p ,
we conclude that
E [ Ψ ( x N ) − Ψ ( u ) ] ≤ ( 2 p − 1 + 2 + 1 p ) M p p R p ( N + p − 1 ) p + Γ N ∑ k = 1 N η 2 Γ k . \mathbb{E}[\Psi(x_N)-\Psi(u)]\le\left(2^{p-1}+2+\frac{1}{p}\right)\frac{Mp^pR^p}{(N+p-1)^p}+\Gamma_N\sum_{k=1}^{N}\frac{\eta}{2\Gamma_k}. E [ Ψ ( x N ) − Ψ ( u )] ≤ ( 2 p − 1 + 2 + p 1 ) ( N + p − 1 ) p M p p R p + Γ N k = 1 ∑ N 2 Γ k η .
Finally, the recurrence for Γ k \Gamma_k Γ k gives
Γ N ∑ k = 1 N 1 Γ k = N + p p + 1 . \Gamma_N\sum_{k=1}^{N}\frac{1}{\Gamma_k}=\frac{N+p}{p+1}. Γ N k = 1 ∑ N Γ k 1 = p + 1 N + p .
Setting u = x ∗ u=x^* u = x ∗ finishes the proof.
Theorem 7. Suppose the conditions of Proposition 6 hold and let Λ \Lambda Λ be ( L , κ ) (L,\kappa) ( L , κ ) -weakly smooth with κ ∈ ( 1 , 2 ] \kappa\in(1,2] κ ∈ ( 1 , 2 ] . For a target accuracy 0 < ϵ ≤ L R κ 0<\epsilon\le LR^\kappa 0 < ϵ ≤ L R κ , choose
η = ϵ N , M = [ 2 ( p − κ ) p κ η ] p − κ κ p − p + κ L κ p κ . \eta=\frac{\epsilon}{N},\qquad M=\left[\frac{2(p-\kappa)}{p\kappa\eta}\right]^{\frac{p-\kappa}{\kappa p-p+\kappa}}L_\kappa^{\frac{p}{\kappa}}. η = N ϵ , M = [ p κ η 2 ( p − κ ) ] κ p − p + κ p − κ L κ κ p . With N N N chosen sufficiently large such that
N = Θ ( ( L R κ ϵ ) p κ p − p + κ ) , N=\Theta\left(\left(\frac{LR^\kappa}{\epsilon}\right)^{\frac{p}{\kappa p-p+\kappa}}\right), N = Θ ( ( ϵ L R κ ) κ p − p + κ p ) , Algorithm 5 , using EpochDA for the inner computations, returns an iterate satisfying E [ Ψ ( x N ) ] ≤ Ψ ( x ∗ ) + ϵ \mathbb{E}[\Psi(x_N)]\le\Psi(x^*)+\epsilon E [ Ψ ( x N )] ≤ Ψ ( x ∗ ) + ϵ . The number of evaluations of ∇ Λ \nabla\Lambda ∇Λ is
N ∇ Λ = N = O ( ( L R κ ϵ ) p κ p − p + κ ) , N_{\nabla\Lambda}=N=O\left(\left(\frac{LR^\kappa}{\epsilon}\right)^{\frac{p}{\kappa p-p+\kappa}}\right), N ∇Λ = N = O ( ( ϵ L R κ ) κ p − p + κ p ) , and the expected number of stochastic-gradient evaluations, including both inner computations, is
E N ∇ f = O ~ ( N + ( G R ϵ ) p N p ( p − 2 ) ) . \mathbb{E}N_{\nabla f}=\widetilde{O}\left(N+\left(\frac{GR}{\epsilon}\right)^pN^{p(p-2)}\right). E N ∇ f = O ( N + ( ϵ GR ) p N p ( p − 2 ) ) .
Proof. By Proposition 6,
E [ Ψ ( x N ) − Ψ ( x ∗ ) ] ≤ ( 2 p − 1 + 2 + 1 p ) M p p R p ( N + p − 1 ) p + η ( N + p ) 2 ( p + 1 ) . \mathbb{E}[\Psi(x_N)-\Psi(x^*)]\leq\left(2^{p-1}+2+\frac{1}{p}\right)\frac{Mp^pR^p}{(N+p-1)^p}+\frac{\eta(N+p)}{2(p+1)}. E [ Ψ ( x N ) − Ψ ( x ∗ )] ≤ ( 2 p − 1 + 2 + p 1 ) ( N + p − 1 ) p M p p R p + 2 ( p + 1 ) η ( N + p ) .
Choose η = ε / N \eta=\varepsilon/N η = ε / N . Since N + p ≤ ( p + 1 ) N N+p\leq(p+1)N N + p ≤ ( p + 1 ) N , the second term is at most ε / 2 \varepsilon/2 ε /2 . Substituting this choice into the definition of M M M gives
M = [ 2 ( p − κ ) N p κ ε ] p − κ κ L κ p κ . M=\left[\frac{2(p-\kappa)N}{p\kappa\varepsilon}\right]^{\frac{p-\kappa}{\kappa}}L_{\kappa}^{\frac{p}{\kappa}}. M = [ p κ ε 2 ( p − κ ) N ] κ p − κ L κ κ p .
Now choose N N N sufficiently large such that
N ≥ [ 2 p p ( 2 p − 1 + 2 + 1 p ) ( 2 ( p − κ ) p κ ) p − κ κ ] κ κ p − p + κ ( L R κ ε ) p κ p − p + κ . N\geq\left[2p^p\left(2^{p-1}+2+\frac{1}{p}\right)\left(\frac{2(p-\kappa)}{p\kappa}\right)^{\frac{p-\kappa}{\kappa}}\right]^{\frac{\kappa}{\kappa p-p+\kappa}}\left(\frac{LR^{\kappa}}{\varepsilon}\right)^{\frac{p}{\kappa p-p+\kappa}}. N ≥ [ 2 p p ( 2 p − 1 + 2 + p 1 ) ( p κ 2 ( p − κ ) ) κ p − κ ] κ p − p + κ κ ( ε L R κ ) κ p − p + κ p .
Using N + p − 1 ≥ N N+p-1\geq N N + p − 1 ≥ N , this ensures
( 2 p − 1 + 2 + 1 p ) M p p R p ( N + p − 1 ) p ≤ ε 2 . \left(2^{p-1}+2+\frac{1}{p}\right)\frac{Mp^pR^p}{(N+p-1)^p}\leq\frac{\varepsilon}{2}. ( 2 p − 1 + 2 + p 1 ) ( N + p − 1 ) p M p p R p ≤ 2 ε .
Therefore, E [ Ψ ( x N ) − Ψ ( x ∗ ) ] ≤ ε \mathbb{E}[\Psi(x_N)-\Psi(x^*)]\leq\varepsilon E [ Ψ ( x N ) − Ψ ( x ∗ )] ≤ ε . Each outer iteration evaluates ∇ Λ \nabla\Lambda ∇Λ once, so N ∇ Λ = N N_{\nabla\Lambda}=N N ∇Λ = N . For the stochastic-gradient complexity, at each outer iteration k k k , substitute
δ k = R p − 1 4 N , σ k = R p − 1 ( q 2 N ) 1 / q \delta_k=\frac{R^{p-1}}{4N},\qquad\sigma_k=R^{p-1}\left(\frac{q}{2N}\right)^{1/q} δ k = 4 N R p − 1 , σ k = R p − 1 ( 2 N q ) 1/ q
into Lemma 11. For fixed p > 2 p>2 p > 2 , δ k \delta_k δ k and σ k \sigma_k σ k scale as N − 1 N^{-1} N − 1 and N − 1 / q N^{-1/q} N − 1/ q , respectively, and min { δ k , σ k } = δ k \min\{\delta_k,\sigma_k\}=\delta_k min { δ k , σ k } = δ k . Consequently, the MLMC cost at iteration k k k is
O ~ ( 1 + ( G β k R p − 1 ) p N p 2 − p − 1 ) . \widetilde{O}\left(1+\left(\frac{G}{\beta_kR^{p-1}}\right)^pN^{p^2-p-1}\right). O ( 1 + ( β k R p − 1 G ) p N p 2 − p − 1 ) .
To compute v ˉ k \bar v_k v ˉ k , we choose z 0 ∈ arg min z ∈ X ˉ ψ k ( z ) z_0\in\arg\min_{z\in\bar X}\psi_k(z) z 0 ∈ arg min z ∈ X ˉ ψ k ( z ) and set R 0 = ( p G / β k ) 1 / ( p − 1 ) R_0=(pG/\beta_k)^{1/(p-1)} R 0 = ( pG / β k ) 1/ ( p − 1 ) . An expected suboptimality of at most ϵ k \epsilon_k ϵ k can then be achieved using O ( 1 + G p / ( β k ϵ k p − 1 ) ) O(1+G^p/(\beta_k\epsilon_k^{p-1})) O ( 1 + G p / ( β k ϵ k p − 1 )) stochastic subgradient calls, where ϵ k = β k R p / ( p N ) \epsilon_k=\beta_kR^p/(pN) ϵ k = β k R p / ( pN ) . The total cost at iteration k k k therefore satisfies
E N k = O ~ ( 1 + ( G β k R p − 1 ) p N p 2 − p − 1 ) . \mathbb{E}N_k=\widetilde{O}\left(1+\left(\frac{G}{\beta_kR^{p-1}}\right)^pN^{p^2-p-1}\right). E N k = O ( 1 + ( β k R p − 1 G ) p N p 2 − p − 1 ) .
Since β k = M γ k p − 1 \beta_k=M\gamma_k^{p-1} β k = M γ k p − 1 and γ k ≥ 1 / N \gamma_k\geq1/N γ k ≥ 1/ N , summing over the N N N outer iterations gives
E N ∇ f = O ~ ( N + ( G M R p − 1 ) p N 2 p ( p − 1 ) ) . \mathbb{E}N_{\nabla f}=\widetilde{O}\left(N+\left(\frac{G}{MR^{p-1}}\right)^pN^{2p(p-1)}\right). E N ∇ f = O ( N + ( M R p − 1 G ) p N 2 p ( p − 1 ) ) .
Finally, the choices of M M M and N N N imply M = Θ ( ε N p / R p ) M=\Theta(\varepsilon N^p/R^p) M = Θ ( ε N p / R p ) . Substituting this relation yields
E N ∇ f = O ~ ( N + ( G R ε ) p N p ( p − 2 ) ) , \mathbb{E}N_{\nabla f}=\widetilde{O}\left(N+\left(\frac{GR}{\varepsilon}\right)^pN^{p(p-2)}\right), E N ∇ f = O ( N + ( ε GR ) p N p ( p − 2 ) ) ,
which finishes the proof.
ℓ p \ell_p ℓ p Accelerated Proximal Point Methods
In this section, we establish the guarantees of our stochastic accelerated ℓ p \ell_p ℓ p proximal point method. We first analyze the NextLambda procedure, showing that it controls the proximal point movement while requiring only logarithmically many BROO calls. We then establish a potential decrease bound using the ℓ p \ell_p ℓ p Bregman divergence and our reduced-bias Moreau gradient estimator. Combining these results, we obtain convergence and stochastic gradient query complexity guarantees for the accelerated proximal point method. We define
Φ ~ f , λ ( y ) ∈ { x ∈ X : E [ F ( x ) ] ≤ F ( P f , λ ( y ) ) + φ , F ( x ) = f ( x ) + λ p ∥ x − y ∥ p p } \widetilde{\Phi}_{f,\lambda}(y) \in\left\{x \in X : \mathbb{E}[F(x)] \le F(P_{f,\lambda}(y)) + \varphi, F(x) = f(x) + \frac{\lambda}{p}\lVert x-y\rVert_p^p\right\} Φ f , λ ( y ) ∈ { x ∈ X : E [ F ( x )] ≤ F ( P f , λ ( y )) + φ , F ( x ) = f ( x ) + p λ ∥ x − y ∥ p p }
Algorithm 6 Stochastic accelerated proximal point method
Initialize x 0 = v 0 = 0 x_0 = v_0 = 0 x 0 = v 0 = 0 , A 0 ≥ 0 A_0 \ge0 A 0 ≥ 0 , gradient estimator ∇ ^ f \widehat{\nabla} f ∇ f , function NextLambda
for k = 0 , 1 , … k = 0,1,\ldots k = 0 , 1 , … , do
λ k + 1 = NextLambda ( x k , v k , A k ) \lambda_{k+1} = \operatorname{NextLambda}(x_k,v_k,A_k) λ k + 1 = NextLambda ( x k , v k , A k )
Solve ( A k + a k + 1 ) p − 1 = 2 p − 1 λ k + 1 a k + 1 p (A_k+a_{k+1})^{p-1} = 2^{p-1}\lambda_{k+1}a_{k+1}^p ( A k + a k + 1 ) p − 1 = 2 p − 1 λ k + 1 a k + 1 p for a k + 1 a_{k+1} a k + 1
A k + 1 = A k + a k + 1 A_{k+1} = A_k+a_{k+1} A k + 1 = A k + a k + 1
y k = A k A k + 1 x k + a k + 1 A k + 1 v k y_k = \frac{A_k}{A_{k+1}}x_k+\frac{a_{k+1}}{A_{k+1}}v_k y k = A k + 1 A k x k + A k + 1 a k + 1 v k
x k + 1 = P ~ f , λ k + 1 φ k + 1 ( y k ) x_{k+1} = \widetilde{P}_{f,\lambda_{k+1}}^{\varphi_{k+1}}(y_k) x k + 1 = P f , λ k + 1 φ k + 1 ( y k )
g k + 1 = MORGRADEST ( ∇ ^ f , y k , λ k + 1 , δ k + 1 , σ k + 1 q , X ∩ B r ( y k ) ) g_{k+1} = \operatorname{MORGRADEST}(\widehat{\nabla} f,y_k,\lambda_{k+1},\delta_{k+1},\sigma_{k+1}^{q},X\cap\mathbb{B}_r(y_k)) g k + 1 = MORGRADEST ( ∇ f , y k , λ k + 1 , δ k + 1 , σ k + 1 q , X ∩ B r ( y k ))
v k + 1 = arg min v ∈ X { 1 2 a k + 1 ⟨ g k + 1 , v ⟩ + D Φ ( v , v k ) } v_{k+1} = \arg\min_{v\in X}\left\{\frac{1}{2}a_{k+1}\langle g_{k+1},v\rangle+D_{\Phi}(v,v_k)\right\} v k + 1 = arg min v ∈ X { 2 1 a k + 1 ⟨ g k + 1 , v ⟩ + D Φ ( v , v k ) } , Φ ( x ) = 1 p ∥ x ∥ p p \Phi(x)=\frac{1}{p}\lVert x\rVert_p^p Φ ( x ) = p 1 ∥ x ∥ p p
return x k + 1 x_{k+1} x k + 1 if A k + 1 ≥ A max A_{k+1}\ge A_{\max} A k + 1 ≥ A m a x or k + 1 = K max k+1=K_{\max} k + 1 = K m a x
end for
NextLambda Algorithm
Algorithm 7 NextLambda(x x x , v v v , A A A )
Input: x , v ∈ R d x, v \in\mathbb{R}^{d} x , v ∈ R d , A ≥ 0 A \ge0 A ≥ 0 , λ min \lambda_{\min} λ m i n , λ max \lambda_{\max} λ m a x , G G G , R R R , r r r
For λ ′ > 0 \lambda' > 0 λ ′ > 0 , let a λ ′ > 0 a_{\lambda'} > 0 a λ ′ > 0 satisfy ( A + a λ ′ ) p − 1 = 2 p − 1 λ ′ a λ ′ p (A+a_{\lambda'})^{p-1}=2^{p-1}\lambda'a_{\lambda'}^{p} ( A + a λ ′ ) p − 1 = 2 p − 1 λ ′ a λ ′ p , where A λ ′ = A + a λ ′ A_{\lambda'}=A+a_{\lambda'} A λ ′ = A + a λ ′ and y λ ′ = A A λ ′ x + a λ ′ A λ ′ v y_{\lambda'}=\frac{A}{A_{\lambda'}}x+\frac{a_{\lambda'}}{A_{\lambda'}}v y λ ′ = A λ ′ A x + A λ ′ a λ ′ v
Define Δ ( λ ) = ∥ O λ , r / 17 ( y λ ) − y λ ∥ p \Delta(\lambda)=\left\|\mathcal{O}_{\lambda,r/17}(y_{\lambda})-y_{\lambda}\right\|_{p} Δ ( λ ) = O λ , r /17 ( y λ ) − y λ p .
λ ← λ max \lambda\leftarrow\lambda_{\max} λ ← λ m a x
while λ ≥ λ min \lambda\ge\lambda_{\min} λ ≥ λ m i n and Δ ( λ ) ≤ 13 r / 16 \Delta(\lambda) \le13r/16 Δ ( λ ) ≤ 13 r /16 do λ ← λ / 2 \lambda\leftarrow\lambda/2 λ ← λ /2
end while
if λ < λ min \lambda< \lambda_{\min} λ < λ m i n then return 2 λ 2\lambda 2 λ
end if
λ u = 2 λ \lambda_u=2\lambda λ u = 2 λ , λ ℓ = λ \lambda_\ell=\lambda λ ℓ = λ , λ m = λ u λ ℓ \lambda_m=\sqrt{\lambda_u\lambda_\ell} λ m = λ u λ ℓ
if Δ ( λ ℓ ) ≤ 15 r / 16 \Delta(\lambda_\ell) \le15r/16 Δ ( λ ℓ ) ≤ 15 r /16 then return λ ℓ \lambda_\ell λ ℓ
end if
while Δ ( λ m ) ∉ [ 13 r / 16 , 15 r / 16 ] \Delta(\lambda_m)\notin[13r/16,15r/16] Δ ( λ m ) ∈ / [ 13 r /16 , 15 r /16 ] and log λ u λ ℓ > r 8 ( R + 1 p − 1 ( G λ ℓ ) 1 / ( p − 1 ) ) \log\frac{\lambda_u}{\lambda_\ell}>\frac{r}{8\left(R+\frac{1}{p-1}\left(\frac{G}{\lambda_\ell}\right)^{1/(p-1)}\right)} log λ ℓ λ u > 8 ( R + p − 1 1 ( λ ℓ G ) 1/ ( p − 1 ) ) r do
if Δ ( λ m ) < 13 r / 16 \Delta(\lambda_m)<13r/16 Δ ( λ m ) < 13 r /16 then λ u = λ m \lambda_u=\lambda_m λ u = λ m
else λ ℓ = λ m \lambda_\ell=\lambda_m λ ℓ = λ m
end if
λ m = λ u λ ℓ \lambda_m=\sqrt{\lambda_u\lambda_\ell} λ m = λ u λ ℓ
end while
return λ m \lambda_m λ m
Here, we discuss our implementation of the subroutine NextLambda (inspired by [2 ] and [16 ] ), such that for x ^ k + 1 = P f , λ ( y k ) \hat{x}_{k+1}=P_{f,\lambda}(y_k) x ^ k + 1 = P f , λ ( y k ) , it holds that ∥ x ^ k + 1 − y k ∥ p ≤ r \|\hat{x}_{k+1}-y_k\|_p\le r ∥ x ^ k + 1 − y k ∥ p ≤ r and either ∥ x ^ k + 1 − y k ∥ p ≥ 3 r / 4 \|\hat{x}_{k+1}-y_k\|_p\ge3r/4 ∥ x ^ k + 1 − y k ∥ p ≥ 3 r /4 or λ < 2 λ min \lambda<2\lambda_{\min} λ < 2 λ m i n . We let J p ( z ) = ∇ ( ∥ z ∥ p p p ) = ( ∣ z i ∣ p − 2 z i ) i J_p(z)=\nabla\left(\frac{\|z\|_p^p}{p}\right)=(|z_i|^{p-2}z_i)_i J p ( z ) = ∇ ( p ∥ z ∥ p p ) = ( ∣ z i ∣ p − 2 z i ) i and x ^ λ = arg min x { f ( x ) + λ p ∥ x − y λ ∥ p p } \hat{x}_{\lambda}=\mathop{\arg\min}_{x}\{f(x)+\frac{\lambda}{p}\|x-y_{\lambda}\|_p^p\} x ^ λ = arg min x { f ( x ) + p λ ∥ x − y λ ∥ p p } . Assume that f f f is G G G -Lipschitz with respect to the ℓ p \ell_p ℓ p norm.
Definition 5. We say that a mapping O λ , ρ ( ⋅ ) : X → R d \mathcal{O}_{\lambda,\rho}(\cdot):\mathcal{X}\to\mathbb{R}^d O λ , ρ ( ⋅ ) : X → R d is a Ball Regularized Optimization Oracle of radius r r r for f f f , if for every query point x ˉ \bar{x} x ˉ , regularization parameter λ > 0 \lambda>0 λ > 0 , and desired accuracy ρ > 0 \rho>0 ρ > 0 , it returns x ~ = O λ , ρ ( x ˉ ) ∈ X ∩ B r ( x ˉ ) \tilde{x}=\mathcal{O}_{\lambda,\rho}(\bar{x})\in\mathcal{X}\cap B_r(\bar{x}) x ~ = O λ , ρ ( x ˉ ) ∈ X ∩ B r ( x ˉ ) such that
f ( x ~ ) + λ p ∥ x ~ − x ˉ ∥ p p ≤ min x ∈ B r ( x ˉ ) { f ( x ) + λ p ∥ x − x ˉ ∥ p p } + λ 2 2 − p p ρ p . f(\tilde{x})+\frac{\lambda}{p}\|\tilde{x}-\bar{x}\|_p^p\le\min_{x\in B_r(\bar{x})}\left\{f(x)+\frac{\lambda}{p}\|x-\bar{x}\|_p^p\right\}+\frac{\lambda2^{2-p}}{p}\rho^p. f ( x ~ ) + p λ ∥ x ~ − x ˉ ∥ p p ≤ x ∈ B r ( x ˉ ) min { f ( x ) + p λ ∥ x − x ˉ ∥ p p } + p λ 2 2 − p ρ p .
Lemma 12. Let x ^ λ = arg min z { f ( z ) + λ p ∥ z − y λ ∥ p p } \hat{x}_{\lambda}=\mathop{\arg\min}_{z}\{f(z)+\frac{\lambda}{p}\|z-y_{\lambda}\|_p^p\} x ^ λ = arg min z { f ( z ) + p λ ∥ z − y λ ∥ p p } . For all λ > 0 \lambda>0 λ > 0 , it holds that
∥ x ^ λ − y λ ∥ p ≤ ( G λ ) 1 / ( p − 1 ) . \|\hat{x}_{\lambda}-y_{\lambda}\|_p\le\left(\frac{G}{\lambda}\right)^{1/(p-1)}. ∥ x ^ λ − y λ ∥ p ≤ ( λ G ) 1/ ( p − 1 ) .
Proof. Let δ λ = x ^ λ − y λ \delta_{\lambda}=\hat{x}_{\lambda}-y_{\lambda} δ λ = x ^ λ − y λ . We know that for some g λ ∈ ∂ f ( x ^ λ ) g_{\lambda}\in\partial f(\hat{x}_{\lambda}) g λ ∈ ∂ f ( x ^ λ ) , we have that g λ + λ J p ( δ λ ) = 0 g_{\lambda}+\lambda J_p(\delta_{\lambda})=0 g λ + λ J p ( δ λ ) = 0 . Since f f f is G G G -Lipschitz, it holds that ∥ g λ ∥ q ≤ G \|g_{\lambda}\|_q\le G ∥ g λ ∥ q ≤ G , which implies that λ ∥ J p ( δ λ ) ∥ q ≤ G \lambda\|J_p(\delta_{\lambda})\|_q\le G λ ∥ J p ( δ λ ) ∥ q ≤ G . Since ∥ J p ( z ) ∥ q = ∥ z ∥ p p − 1 \|J_p(z)\|_q=\|z\|_p^{p-1} ∥ J p ( z ) ∥ q = ∥ z ∥ p p − 1 , it holds that λ ∥ δ λ ∥ p p − 1 ≤ G \lambda\|\delta_{\lambda}\|_p^{p-1}\le G λ ∥ δ λ ∥ p p − 1 ≤ G from which the stated claim follows. □ \square □
Lemma 13. Let A ≥ 0 A\ge0 A ≥ 0 and x , v ∈ R d x,v\in\mathbb{R}^d x , v ∈ R d satisfy ∥ x − v ∥ p ≤ 2 R \|x-v\|_p\le2R ∥ x − v ∥ p ≤ 2 R . For every λ > 0 \lambda>0 λ > 0 , let a λ > 0 a_{\lambda}>0 a λ > 0 be the unique solution of ( A + a λ ) p − 1 = 2 p − 1 λ a λ p (A+a_{\lambda})^{p-1}=2^{p-1}\lambda a_{\lambda}^{p} ( A + a λ ) p − 1 = 2 p − 1 λ a λ p , and define y λ = A x + a λ v A + a λ y_{\lambda}=\frac{Ax+a_{\lambda}v}{A+a_{\lambda}} y λ = A + a λ A x + a λ v . Then
∥ d d λ y λ ∥ p ≤ R λ . \left\|\frac{d}{d\lambda}y_{\lambda}\right\|_p\le\frac{R}{\lambda}. d λ d y λ p ≤ λ R .
Proof. Denote α λ = a λ / ( A + a λ ) \alpha_\lambda= a_\lambda/(A+a_\lambda) α λ = a λ / ( A + a λ ) . Taking logarithmic derivatives on both sides of ( A + a λ ) p − 1 = 2 p − 1 λ a λ p (A+a_\lambda)^{p-1}=2^{p-1}\lambda a_\lambda^p ( A + a λ ) p − 1 = 2 p − 1 λ a λ p , we get that
( p − 1 ) a λ ′ A + a λ = 1 λ + p a λ ′ a λ ⟹ a λ ′ = − a λ ( A + a λ ) λ ( a λ + p A ) (p-1)\frac{a'_\lambda}{A+a_\lambda}=\frac{1}{\lambda}+p\frac{a'_\lambda}{a_\lambda}\Longrightarrow a'_\lambda=-\frac{a_\lambda(A+a_\lambda)}{\lambda(a_\lambda+pA)} ( p − 1 ) A + a λ a λ ′ = λ 1 + p a λ a λ ′ ⟹ a λ ′ = − λ ( a λ + p A ) a λ ( A + a λ )
Since a λ + p A ≥ A + a λ a_\lambda+pA\ge A+a_\lambda a λ + p A ≥ A + a λ , this means that ∣ a λ ′ ∣ ≤ a λ / λ |a'_\lambda|\le a_\lambda/\lambda ∣ a λ ′ ∣ ≤ a λ / λ . Now, notice that
∣ α λ ′ ∣ = ∣ A a λ ′ ( A + a λ ) 2 ∣ ≤ A a λ λ ( A + a λ ) 2 ≤ 1 2 λ |\alpha'_\lambda|=\left|\frac{Aa'_\lambda}{(A+a_\lambda)^2}\right|\le\frac{Aa_\lambda}{\lambda(A+a_\lambda)^2}\le\frac{1}{2\lambda} ∣ α λ ′ ∣ = ( A + a λ ) 2 A a λ ′ ≤ λ ( A + a λ ) 2 A a λ ≤ 2 λ 1
and since y λ = ( 1 − α λ ) x + α λ v y_\lambda=(1-\alpha_\lambda)x+\alpha_\lambda v y λ = ( 1 − α λ ) x + α λ v , this implies that
∥ d d λ y λ ∥ p = ∣ α λ ′ ∣ ⋅ ∥ v − x ∥ p ≤ R λ \left\|\frac{d}{d\lambda}y_\lambda\right\|_p=|\alpha'_\lambda|\cdot\|v-x\|_p\le\frac{R}{\lambda} d λ d y λ p = ∣ α λ ′ ∣ ⋅ ∥ v − x ∥ p ≤ λ R
as ∥ v − x ∥ p ≤ 2 R \|v-x\|_p\le2R ∥ v − x ∥ p ≤ 2 R .
Lemma 14. Let f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R be convex and G G G -Lipschitz, let y λ y_\lambda y λ be continuously differentiable for λ > 0 \lambda>0 λ > 0 . For every 0 < λ 1 ≤ λ 2 0<\lambda_1\le\lambda_2 0 < λ 1 ≤ λ 2 , it holds that
∣ ∥ δ λ 2 ∥ p − ∥ δ λ 1 ∥ p ∣ ≤ ∫ λ 1 λ 2 ∥ d d λ y λ ∥ p d λ + 1 p − 1 ∫ λ 1 λ 2 ∥ δ λ ∥ p λ d λ . \left|\|\delta_{\lambda_2}\|_p-\|\delta_{\lambda_1}\|_p\right|\le\int_{\lambda_1}^{\lambda_2}\left\|\frac{d}{d\lambda}y_\lambda\right\|_p\,d\lambda+\frac{1}{p-1}\int_{\lambda_1}^{\lambda_2}\frac{\|\delta_\lambda\|_p}{\lambda}\,d\lambda. ∣ ∥ δ λ 2 ∥ p − ∥ δ λ 1 ∥ p ∣ ≤ ∫ λ 1 λ 2 d λ d y λ p d λ + p − 1 1 ∫ λ 1 λ 2 λ ∥ δ λ ∥ p d λ .
Proof. We first smooth f f f and use a nondegenerate approximation of the p p p -power regularizer. Let
f σ ( x ) = E [ f ( x + ν σ ) ] , ν σ ∼ N ( 0 , σ 2 I ) , f_\sigma(x)=\mathbb{E}[f(x+\nu_\sigma)],\qquad\nu_\sigma\sim\mathcal{N}(0,\sigma^2I), f σ ( x ) = E [ f ( x + ν σ )] , ν σ ∼ N ( 0 , σ 2 I ) ,
and, for ϵ > 0 \epsilon>0 ϵ > 0 , define ϕ ϵ ( z ) = 1 p ∑ i = 1 d [ ( z i 2 + ϵ 2 ) p / 2 − ϵ p ] \phi_\epsilon(z)=\frac{1}{p}\sum_{i=1}^d[(z_i^2+\epsilon^2)^{p/2}-\epsilon^p] ϕ ϵ ( z ) = p 1 ∑ i = 1 d [( z i 2 + ϵ 2 ) p /2 − ϵ p ] . Let
x ^ λ σ , ϵ = argmin x { f σ ( x ) + λ ϕ ϵ ( x − y λ ) } , δ λ σ , ϵ = x ^ λ σ , ϵ − y λ , \hat{x}^{\sigma,\epsilon}_\lambda=\underset{x}{\operatorname{argmin}}\left\{f_\sigma(x)+\lambda\phi_\epsilon(x-y_\lambda)\right\},\qquad\delta^{\sigma,\epsilon}_\lambda=\hat{x}^{\sigma,\epsilon}_\lambda-y_\lambda, x ^ λ σ , ϵ = x argmin { f σ ( x ) + λ ϕ ϵ ( x − y λ ) } , δ λ σ , ϵ = x ^ λ σ , ϵ − y λ ,
and define ρ ϵ ( z ) = ( ∑ i = 1 d ( z i 2 + ϵ 2 ) p / 2 ) 1 / p \rho_\epsilon(z)=\left(\sum_{i=1}^d(z_i^2+\epsilon^2)^{p/2}\right)^{1/p} ρ ϵ ( z ) = ( ∑ i = 1 d ( z i 2 + ϵ 2 ) p /2 ) 1/ p . Notice that ϕ ϵ ( z ) = 1 p ( ρ ϵ ( z ) p − d ϵ p ) \phi_\epsilon(z)=\frac{1}{p}(\rho_\epsilon(z)^p-d\epsilon^p) ϕ ϵ ( z ) = p 1 ( ρ ϵ ( z ) p − d ϵ p ) and ρ ϵ ( z ) → ∥ z ∥ p \rho_\epsilon(z)\to\|z\|_p ρ ϵ ( z ) → ∥ z ∥ p as ϵ → 0 \epsilon\to0 ϵ → 0 . Since f σ f_\sigma f σ is smooth and convex and ϕ ϵ \phi_\epsilon ϕ ϵ has a positive definite Hessian, x ^ λ σ , ϵ \hat{x}^{\sigma,\epsilon}_\lambda x ^ λ σ , ϵ is uniquely defined and differentiable in λ \lambda λ . Let J p , ϵ ( z ) = ∇ ϕ ϵ ( z ) J_{p,\epsilon}(z)=\nabla\phi_\epsilon(z) J p , ϵ ( z ) = ∇ ϕ ϵ ( z ) , so that
[ J p , ϵ ( z ) ] i = z i ( z i 2 + ϵ 2 ) p / 2 − 1 . [J_{p,\epsilon}(z)]_i=z_i(z_i^2+\epsilon^2)^{p/2-1}. [ J p , ϵ ( z ) ] i = z i ( z i 2 + ϵ 2 ) p /2 − 1 .
The Hessian of ϕ ϵ \phi_\epsilon ϕ ϵ is diagonal, with [ ∇ 2 ϕ ϵ ( z ) ] i i = ( z i 2 + ϵ 2 ) p / 2 − 2 [ ϵ 2 + ( p − 1 ) z i 2 ] > 0 [\nabla^2\phi_\epsilon(z)]_{ii}=(z_i^2+\epsilon^2)^{p/2-2}[\epsilon^2+(p-1)z_i^2]>0 [ ∇ 2 ϕ ϵ ( z ) ] ii = ( z i 2 + ϵ 2 ) p /2 − 2 [ ϵ 2 + ( p − 1 ) z i 2 ] > 0 . Now, the first-order optimality gives
∇ f σ ( x ^ λ σ , ϵ ) + λ J p , ϵ ( δ λ σ , ϵ ) = 0. \nabla f_\sigma(\hat{x}^{\sigma,\epsilon}_\lambda)+\lambda J_{p,\epsilon}(\delta^{\sigma,\epsilon}_\lambda)=0. ∇ f σ ( x ^ λ σ , ϵ ) + λ J p , ϵ ( δ λ σ , ϵ ) = 0.
Differentiating with respect to λ \lambda λ gives
[ ∇ 2 f σ ( x ^ λ σ , ϵ ) + λ ∇ 2 ϕ ϵ ( δ λ σ , ϵ ) ] d d λ δ λ σ , ϵ = − J p , ϵ ( δ λ σ , ϵ ) − ∇ 2 f σ ( x ^ λ σ , ϵ ) d d λ y λ . \left[\nabla^2f_\sigma(\hat{x}^{\sigma,\epsilon}_\lambda)+\lambda\nabla^2\phi_\epsilon(\delta^{\sigma,\epsilon}_\lambda)\right]\frac{d}{d\lambda}\delta^{\sigma,\epsilon}_\lambda=-J_{p,\epsilon}(\delta^{\sigma,\epsilon}_\lambda)-\nabla^2f_\sigma(\hat{x}^{\sigma,\epsilon}_\lambda)\frac{d}{d\lambda}y_\lambda. [ ∇ 2 f σ ( x ^ λ σ , ϵ ) + λ ∇ 2 ϕ ϵ ( δ λ σ , ϵ ) ] d λ d δ λ σ , ϵ = − J p , ϵ ( δ λ σ , ϵ ) − ∇ 2 f σ ( x ^ λ σ , ϵ ) d λ d y λ .
For brevity, write
H = ∇ 2 f σ ( x ^ λ σ , ϵ ) , D = ∇ 2 ϕ ϵ ( δ λ σ , ϵ ) . H=\nabla^2f_\sigma(\hat{x}^{\sigma,\epsilon}_\lambda),\qquad D=\nabla^2\phi_\epsilon(\delta^{\sigma,\epsilon}_\lambda). H = ∇ 2 f σ ( x ^ λ σ , ϵ ) , D = ∇ 2 ϕ ϵ ( δ λ σ , ϵ ) .
By construction we have H ⪰ 0 H\succeq0 H ⪰ 0 and D ≻ 0 D\succ0 D ≻ 0 , and
( H + λ D ) d d λ δ λ σ , ϵ = − J p , ϵ ( δ λ σ , ϵ ) − H d d λ y λ . (H+\lambda D)\frac{d}{d\lambda}\delta^{\sigma,\epsilon}_\lambda=-J_{p,\epsilon}(\delta^{\sigma,\epsilon}_\lambda)-H\frac{d}{d\lambda}y_\lambda. ( H + λ D ) d λ d δ λ σ , ϵ = − J p , ϵ ( δ λ σ , ϵ ) − H d λ d y λ .
Since ∇ ρ ϵ ( z ) = J p , ϵ ( z ) ρ ϵ ( z ) p − 1 \nabla\rho_{\epsilon}(z)=\frac{J_{p,\epsilon}(z)}{\rho_{\epsilon}(z)^{p-1}} ∇ ρ ϵ ( z ) = ρ ϵ ( z ) p − 1 J p , ϵ ( z ) , we have d d λ ρ ϵ ( δ λ σ , ϵ ) = J p , ϵ ( δ λ σ , ϵ ) T ρ ϵ ( δ λ σ , ϵ ) p − 1 d d λ δ λ σ , ϵ \frac{d}{d\lambda}\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})=\frac{J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}}{\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{p-1}}\frac{d}{d\lambda}\delta_{\lambda}^{\sigma,\epsilon} d λ d ρ ϵ ( δ λ σ , ϵ ) = ρ ϵ ( δ λ σ , ϵ ) p − 1 J p , ϵ ( δ λ σ , ϵ ) T d λ d δ λ σ , ϵ . Therefore,
∣ d d λ ρ ϵ ( δ λ σ , ϵ ) ∣ ≤ J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) ρ ϵ ( δ λ σ , ϵ ) p − 1 + ∣ J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 H d d λ y λ ∣ ρ ϵ ( δ λ σ , ϵ ) p − 1 . \left|\frac{d}{d\lambda}\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\right|\leq\frac{J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}(H+\lambda D)^{-1}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})}{\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{p-1}}+\frac{\left|J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}(H+\lambda D)^{-1}H\frac{d}{d\lambda}y_{\lambda}\right|}{\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{p-1}}. d λ d ρ ϵ ( δ λ σ , ϵ ) ≤ ρ ϵ ( δ λ σ , ϵ ) p − 1 J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) + ρ ϵ ( δ λ σ , ϵ ) p − 1 J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 H d λ d y λ .
We first bound the first term. Since H + λ D ⪰ λ D H+\lambda D\succeq\lambda D H + λ D ⪰ λ D and D ⪰ 0 D\succeq0 D ⪰ 0 , we have ( H + λ D ) − 1 ⪯ 1 λ D − 1 (H+\lambda D)^{-1}\preceq\frac{1}{\lambda}D^{-1} ( H + λ D ) − 1 ⪯ λ 1 D − 1 . Hence
J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) ≤ 1 λ J p , ϵ ( δ λ σ , ϵ ) T D − 1 J p , ϵ ( δ λ σ , ϵ ) . J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}(H+\lambda D)^{-1}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\leq\frac{1}{\lambda}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}D^{-1}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon}). J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) ≤ λ 1 J p , ϵ ( δ λ σ , ϵ ) T D − 1 J p , ϵ ( δ λ σ , ϵ ) .
For every coordinate,
[ J p , ϵ ( δ λ σ , ϵ ) i ] 2 D i i = ∣ δ λ , i σ , ϵ ∣ 2 ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) p / 2 ϵ 2 + ( p − 1 ) ∣ δ λ , i σ , ϵ ∣ 2 ≤ 1 p − 1 ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) p / 2 . \frac{[J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})_{i}]^{2}}{D_{ii}}=\frac{|\delta_{\lambda,i}^{\sigma,\epsilon}|^{2}(|\delta_{\lambda,i}^{\sigma,\epsilon}|^{2}+\epsilon^{2})^{p/2}}{\epsilon^{2}+(p-1)|\delta_{\lambda,i}^{\sigma,\epsilon}|^{2}}\leq\frac{1}{p-1}\left(|\delta_{\lambda,i}^{\sigma,\epsilon}|^{2}+\epsilon^{2}\right)^{p/2}. D ii [ J p , ϵ ( δ λ σ , ϵ ) i ] 2 = ϵ 2 + ( p − 1 ) ∣ δ λ , i σ , ϵ ∣ 2 ∣ δ λ , i σ , ϵ ∣ 2 ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) p /2 ≤ p − 1 1 ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) p /2 .
Therefore,
J p , ϵ ( δ λ σ , ϵ ) T D − 1 J p , ϵ ( δ λ σ , ϵ ) ≤ ρ ϵ ( δ λ σ , ϵ ) p p − 1 , J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}D^{-1}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\leq\frac{\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{p}}{p-1}, J p , ϵ ( δ λ σ , ϵ ) T D − 1 J p , ϵ ( δ λ σ , ϵ ) ≤ p − 1 ρ ϵ ( δ λ σ , ϵ ) p ,
and thus
J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) ≤ ρ ϵ ( δ λ σ , ϵ ) p ( p − 1 ) λ . J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}(H+\lambda D)^{-1}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\leq\frac{\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{p}}{(p-1)\lambda}. J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) ≤ ( p − 1 ) λ ρ ϵ ( δ λ σ , ϵ ) p .
We next bound the second term. Let S = D − 1 / 2 H D − 1 / 2 S=D^{-1/2}HD^{-1/2} S = D − 1/2 H D − 1/2 . Since H ⪰ 0 H\succeq0 H ⪰ 0 , we have S ⪰ 0 S\succeq0 S ⪰ 0 , and
D − 1 / 2 H ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) = S ( S + λ I ) − 1 D − 1 / 2 J p , ϵ ( δ λ σ , ϵ ) . D^{-1/2}H(H+\lambda D)^{-1}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})=S(S+\lambda I)^{-1}D^{-1/2}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon}). D − 1/2 H ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) = S ( S + λ I ) − 1 D − 1/2 J p , ϵ ( δ λ σ , ϵ ) .
Every eigenvalue of S ( S + λ I ) − 1 S(S+\lambda I)^{-1} S ( S + λ I ) − 1 is in [ 0 , 1 ] [0,1] [ 0 , 1 ] . Therefore,
∥ D − 1 / 2 H ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) ∥ 2 ≤ ∥ D − 1 / 2 J p , ϵ ( δ λ σ , ϵ ) ∥ 2 . \left\|D^{-1/2}H(H+\lambda D)^{-1}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\right\|_{2}\leq\left\|D^{-1/2}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\right\|_{2}. D − 1/2 H ( H + λ D ) − 1 J p , ϵ ( δ λ σ , ϵ ) 2 ≤ D − 1/2 J p , ϵ ( δ λ σ , ϵ ) 2 .
It follows from Cauchy–Schwarz that
∣ J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 H d d λ y λ ∣ ≤ [ J p , ϵ ( δ λ σ , ϵ ) T D − 1 J p , ϵ ( δ λ σ , ϵ ) ] 1 / 2 [ ( d d λ y λ ) T D ( d d λ y λ ) ] 1 / 2 . \left|J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}(H+\lambda D)^{-1}H\frac{d}{d\lambda}y_{\lambda}\right|\leq\left[J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}D^{-1}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\right]^{1/2}\left[\left(\frac{d}{d\lambda}y_{\lambda}\right)^{\mathsf{T}}D\left(\frac{d}{d\lambda}y_{\lambda}\right)\right]^{1/2}. J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 H d λ d y λ ≤ [ J p , ϵ ( δ λ σ , ϵ ) T D − 1 J p , ϵ ( δ λ σ , ϵ ) ] 1/2 [ ( d λ d y λ ) T D ( d λ d y λ ) ] 1/2 .
We already showed that J p , ϵ ( δ λ σ , ϵ ) T D − 1 J p , ϵ ( δ λ σ , ϵ ) ≤ ρ ϵ ( δ λ σ , ϵ ) p p − 1 J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}D^{-1}J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\leq\frac{\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{p}}{p-1} J p , ϵ ( δ λ σ , ϵ ) T D − 1 J p , ϵ ( δ λ σ , ϵ ) ≤ p − 1 ρ ϵ ( δ λ σ , ϵ ) p . Moreover,
D i i = ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) p / 2 − 2 [ ϵ 2 + ( p − 1 ) ∣ δ λ , i σ , ϵ ∣ 2 ] ≤ ( p − 1 ) ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) ( p − 2 ) / 2 . D_{ii}=\left(|\delta_{\lambda,i}^{\sigma,\epsilon}|^{2}+\epsilon^{2}\right)^{p/2-2}\left[\epsilon^{2}+(p-1)|\delta_{\lambda,i}^{\sigma,\epsilon}|^{2}\right]\leq(p-1)\left(|\delta_{\lambda,i}^{\sigma,\epsilon}|^{2}+\epsilon^{2}\right)^{(p-2)/2}. D ii = ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) p /2 − 2 [ ϵ 2 + ( p − 1 ) ∣ δ λ , i σ , ϵ ∣ 2 ] ≤ ( p − 1 ) ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) ( p − 2 ) /2 .
Hence, with Hölder’s inequality, we have
( d d λ y λ ) T D ( d d λ y λ ) ≤ ( p − 1 ) ∑ i ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) ( p − 2 ) / 2 ∣ d d λ y λ , i ∣ 2 ≤ ( p − 1 ) ρ ϵ ( δ λ σ , ϵ ) p − 2 ∥ d d λ y λ ∥ p 2 . \left(\frac{d}{d\lambda}y_{\lambda}\right)^{\mathsf{T}}D\left(\frac{d}{d\lambda}y_{\lambda}\right)\leq(p-1)\sum_{i}\left(|\delta_{\lambda,i}^{\sigma,\epsilon}|^{2}+\epsilon^{2}\right)^{(p-2)/2}\left|\frac{d}{d\lambda}y_{\lambda,i}\right|^{2}\leq(p-1)\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{p-2}\left\|\frac{d}{d\lambda}y_{\lambda}\right\|_{p}^{2}. ( d λ d y λ ) T D ( d λ d y λ ) ≤ ( p − 1 ) i ∑ ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) ( p − 2 ) /2 d λ d y λ , i 2 ≤ ( p − 1 ) ρ ϵ ( δ λ σ , ϵ ) p − 2 d λ d y λ p 2 .
Combining the preceding two inequalities gives
∣ J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 H d d λ y λ ∣ ≤ ρ ϵ ( δ λ σ , ϵ ) p − 1 ∥ d d λ y λ ∥ p . \left|J_{p,\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{\mathsf{T}}(H+\lambda D)^{-1}H\frac{d}{d\lambda}y_{\lambda}\right|\leq\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^{p-1}\left\|\frac{d}{d\lambda}y_{\lambda}\right\|_{p}. J p , ϵ ( δ λ σ , ϵ ) T ( H + λ D ) − 1 H d λ d y λ ≤ ρ ϵ ( δ λ σ , ϵ ) p − 1 d λ d y λ p .
Consequently,
∣ d d λ ρ ϵ ( δ λ σ , ϵ ) ∣ ≤ ∥ d d λ y λ ∥ p + ρ ϵ ( δ λ σ , ϵ ) ( p − 1 ) λ . \left|\frac{d}{d\lambda}\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\right| \le\left\|\frac{d}{d\lambda}y_{\lambda}\right\|_{p}+\frac{\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})}{(p-1)\lambda}. d λ d ρ ϵ ( δ λ σ , ϵ ) ≤ d λ d y λ p + ( p − 1 ) λ ρ ϵ ( δ λ σ , ϵ ) .
Integrating from λ 1 \lambda_1 λ 1 to λ 2 \lambda_2 λ 2 gives
∣ ρ ϵ ( δ λ 2 σ , ϵ ) − ρ ϵ ( δ λ 1 σ , ϵ ) ∣ ≤ ∫ λ 1 λ 2 ∥ d d λ y λ ∥ p d λ + 1 p − 1 ∫ λ 1 λ 2 ρ ϵ ( δ λ σ , ϵ ) λ d λ . \left|\rho_{\epsilon}(\delta_{\lambda_2}^{\sigma,\epsilon})-\rho_{\epsilon}(\delta_{\lambda_1}^{\sigma,\epsilon})\right| \le\int_{\lambda_1}^{\lambda_2}\left\|\frac{d}{d\lambda}y_{\lambda}\right\|_{p}\,d\lambda+\frac{1}{p-1}\int_{\lambda_1}^{\lambda_2}\frac{\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})}{\lambda}\,d\lambda. ρ ϵ ( δ λ 2 σ , ϵ ) − ρ ϵ ( δ λ 1 σ , ϵ ) ≤ ∫ λ 1 λ 2 d λ d y λ p d λ + p − 1 1 ∫ λ 1 λ 2 λ ρ ϵ ( δ λ σ , ϵ ) d λ .
It remains to let σ , ϵ → 0 \sigma,\epsilon\to0 σ , ϵ → 0 . Since f f f is G G G -Lipschitz, f σ f_{\sigma} f σ is also G G G -Lipschitz and
∣ f σ ( x ) − f ( x ) ∣ ≤ G ϵ ∥ ν σ ∥ p ⟶ 0 |f_{\sigma}(x)-f(x)|\leq G\epsilon\|\nu_{\sigma}\|_{p}\longrightarrow0 ∣ f σ ( x ) − f ( x ) ∣ ≤ G ϵ ∥ ν σ ∥ p ⟶ 0
uniformly in x x x . Now, by the definition of x ^ λ σ , ϵ \hat{x}_{\lambda}^{\sigma,\epsilon} x ^ λ σ , ϵ ,
f σ ( x ^ λ σ , ϵ ) + λ ϕ ϵ ( δ λ σ , ϵ ) = f σ ( x ^ λ σ , ϵ ) + λ ϕ ϵ ( x ^ λ σ , ϵ − y λ ) ≤ f σ ( y λ ) + λ ϕ ϵ ( y λ − y λ ) = f σ ( y λ ) + λ ϕ ϵ ( 0 ) f_{\sigma}(\hat{x}_{\lambda}^{\sigma,\epsilon})+\lambda\phi_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})=f_{\sigma}(\hat{x}_{\lambda}^{\sigma,\epsilon})+\lambda\phi_{\epsilon}(\hat{x}_{\lambda}^{\sigma,\epsilon}-y_{\lambda})\leq f_{\sigma}(y_{\lambda})+\lambda\phi_{\epsilon}(y_{\lambda}-y_{\lambda})=f_{\sigma}(y_{\lambda})+\lambda\phi_{\epsilon}(0) f σ ( x ^ λ σ , ϵ ) + λ ϕ ϵ ( δ λ σ , ϵ ) = f σ ( x ^ λ σ , ϵ ) + λ ϕ ϵ ( x ^ λ σ , ϵ − y λ ) ≤ f σ ( y λ ) + λ ϕ ϵ ( y λ − y λ ) = f σ ( y λ ) + λ ϕ ϵ ( 0 )
We now compute ϕ ϵ ( 0 ) \phi_{\epsilon}(0) ϕ ϵ ( 0 ) . Since ρ ϵ ( 0 ) p = ∑ i = 1 d ( 0 2 + ϵ 2 ) p / 2 = ∑ i = 1 d ϵ p = d ϵ p \rho_{\epsilon}(0)^p=\sum_{i=1}^{d}(0^2+\epsilon^2)^{p/2}=\sum_{i=1}^{d}\epsilon^p=d\epsilon^p ρ ϵ ( 0 ) p = ∑ i = 1 d ( 0 2 + ϵ 2 ) p /2 = ∑ i = 1 d ϵ p = d ϵ p , we have ϕ ϵ ( 0 ) = 1 p [ ρ ϵ ( 0 ) p − d ϵ p ] = 1 p [ d ϵ p − d ϵ p ] = 0 \phi_{\epsilon}(0)=\frac{1}{p}[\rho_{\epsilon}(0)^p-d\epsilon^p]=\frac{1}{p}[d\epsilon^p-d\epsilon^p]=0 ϕ ϵ ( 0 ) = p 1 [ ρ ϵ ( 0 ) p − d ϵ p ] = p 1 [ d ϵ p − d ϵ p ] = 0 . Therefore,
f σ ( x ^ λ σ , ϵ ) + λ ϕ ϵ ( δ λ σ , ϵ ) ≤ f σ ( y λ ) f_{\sigma}(\hat{x}_{\lambda}^{\sigma,\epsilon})+\lambda\phi_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\leq f_{\sigma}(y_{\lambda}) f σ ( x ^ λ σ , ϵ ) + λ ϕ ϵ ( δ λ σ , ϵ ) ≤ f σ ( y λ )
Using ϕ ϵ ( z ) = 1 p [ ρ ϵ ( z ) p − d ϵ p ] \phi_{\epsilon}(z)=\frac{1}{p}[\rho_{\epsilon}(z)^p-d\epsilon^p] ϕ ϵ ( z ) = p 1 [ ρ ϵ ( z ) p − d ϵ p ] again, we obtain
f σ ( x ^ λ σ , ϵ ) + λ p [ ρ ϵ ( δ λ σ , ϵ ) p − d ϵ p ] ≤ f σ ( y λ ) ⇒ λ p [ ρ ϵ ( δ λ σ , ϵ ) p − d ϵ p ] ≤ f σ ( y λ ) − f σ ( x ^ λ σ , ϵ ) f_{\sigma}(\hat{x}_{\lambda}^{\sigma,\epsilon})+\frac{\lambda}{p}[\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^p-d\epsilon^p]\leq f_{\sigma}(y_{\lambda})\Rightarrow\frac{\lambda}{p}[\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^p-d\epsilon^p]\leq f_{\sigma}(y_{\lambda})-f_{\sigma}(\hat{x}_{\lambda}^{\sigma,\epsilon}) f σ ( x ^ λ σ , ϵ ) + p λ [ ρ ϵ ( δ λ σ , ϵ ) p − d ϵ p ] ≤ f σ ( y λ ) ⇒ p λ [ ρ ϵ ( δ λ σ , ϵ ) p − d ϵ p ] ≤ f σ ( y λ ) − f σ ( x ^ λ σ , ϵ )
Since f σ f_{\sigma} f σ is G G G -Lipschitz, f σ ( y λ ) − f σ ( x ^ λ σ , ϵ ) ≤ G ∥ y λ − x ^ λ σ , ϵ ∥ p ≤ G ∥ δ λ σ , ϵ ∥ p f_{\sigma}(y_{\lambda})-f_{\sigma}(\hat{x}_{\lambda}^{\sigma,\epsilon})\leq G\|y_{\lambda}-\hat{x}_{\lambda}^{\sigma,\epsilon}\|_{p}\leq G\|\delta_{\lambda}^{\sigma,\epsilon}\|_{p} f σ ( y λ ) − f σ ( x ^ λ σ , ϵ ) ≤ G ∥ y λ − x ^ λ σ , ϵ ∥ p ≤ G ∥ δ λ σ , ϵ ∥ p . Moreover,
ρ ϵ ( δ λ σ , ϵ ) p = ∑ i = 1 d ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) p / 2 ≥ ∑ i = 1 d ∣ δ λ , i σ , ϵ ∣ p = ∥ δ λ σ , ϵ ∥ p p . \rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^p=\sum_{i=1}^{d}(|\delta_{\lambda,i}^{\sigma,\epsilon}|^2+\epsilon^2)^{p/2}\geq\sum_{i=1}^{d}|\delta_{\lambda,i}^{\sigma,\epsilon}|^p=\|\delta_{\lambda}^{\sigma,\epsilon}\|_{p}^p. ρ ϵ ( δ λ σ , ϵ ) p = i = 1 ∑ d ( ∣ δ λ , i σ , ϵ ∣ 2 + ϵ 2 ) p /2 ≥ i = 1 ∑ d ∣ δ λ , i σ , ϵ ∣ p = ∥ δ λ σ , ϵ ∥ p p .
Hence ∥ δ λ σ , ϵ ∥ p ≤ ρ ϵ ( δ λ σ , ϵ ) \|\delta_{\lambda}^{\sigma,\epsilon}\|_{p}\leq\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon}) ∥ δ λ σ , ϵ ∥ p ≤ ρ ϵ ( δ λ σ , ϵ ) . Combining the preceding inequalities gives
λ p [ ρ ϵ ( δ λ σ , ϵ ) p − d ϵ p ] ≤ G ρ ϵ ( δ λ σ , ϵ ) ⇒ λ p ρ ϵ ( δ λ σ , ϵ ) p ≤ G ρ ϵ ( δ λ σ , ϵ ) + λ d p ϵ p \frac{\lambda}{p}[\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^p-d\epsilon^p]\leq G\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\Rightarrow\frac{\lambda}{p}\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})^p\leq G\rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})+\frac{\lambda d}{p}\epsilon^p p λ [ ρ ϵ ( δ λ σ , ϵ ) p − d ϵ p ] ≤ G ρ ϵ ( δ λ σ , ϵ ) ⇒ p λ ρ ϵ ( δ λ σ , ϵ ) p ≤ G ρ ϵ ( δ λ σ , ϵ ) + p λ d ϵ p
Since λ ∈ [ λ 1 , λ 2 ] \lambda\in[\lambda_1,\lambda_2] λ ∈ [ λ 1 , λ 2 ] with λ 1 > 0 \lambda_1>0 λ 1 > 0 , this implies that ρ ϵ ( δ λ σ , ϵ ) \rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon}) ρ ϵ ( δ λ σ , ϵ ) is uniformly bounded over λ ∈ [ λ 1 , λ 2 ] \lambda\in[\lambda_1,\lambda_2] λ ∈ [ λ 1 , λ 2 ] for sufficiently small σ , ϵ \sigma,\epsilon σ , ϵ . Therefore the corresponding minimizers remain in a common bounded set.
On every bounded set,
f σ ( x ) + λ ϕ ϵ ( x − y λ ) ⟶ f ( x ) + λ p ∥ x − y λ ∥ p p f_{\sigma}(x)+\lambda\phi_{\epsilon}(x-y_{\lambda})\longrightarrow f(x)+\frac{\lambda}{p}\|x-y_{\lambda}\|_{p}^{p} f σ ( x ) + λ ϕ ϵ ( x − y λ ) ⟶ f ( x ) + p λ ∥ x − y λ ∥ p p
uniformly as σ , ϵ → 0 \sigma,\epsilon\to0 σ , ϵ → 0 . Since the limiting objective is strictly convex, its minimizer is unique. Also, x ^ λ σ , ϵ ⟶ x ^ λ \hat{x}_{\lambda}^{\sigma,\epsilon}\longrightarrow\hat{x}_{\lambda} x ^ λ σ , ϵ ⟶ x ^ λ and δ λ σ , ϵ ⟶ δ λ \delta_{\lambda}^{\sigma,\epsilon}\longrightarrow\delta_{\lambda} δ λ σ , ϵ ⟶ δ λ for every λ ∈ [ λ 1 , λ 2 ] \lambda\in[\lambda_1,\lambda_2] λ ∈ [ λ 1 , λ 2 ] . Moreover, ρ ϵ ( δ λ σ , ϵ ) ⟶ ∥ δ λ ∥ p \rho_{\epsilon}(\delta_{\lambda}^{\sigma,\epsilon})\longrightarrow\|\delta_{\lambda}\|_{p} ρ ϵ ( δ λ σ , ϵ ) ⟶ ∥ δ λ ∥ p . The uniform bound above and λ ≥ λ 1 > 0 \lambda\geq\lambda_1>0 λ ≥ λ 1 > 0 allow us to apply dominated convergence to the integral. Taking σ , ϵ → 0 \sigma,\epsilon\to0 σ , ϵ → 0 yields
∣ ∥ δ λ 2 ∥ p − ∥ δ λ 1 ∥ p ∣ ≤ ∫ λ 1 λ 2 ∥ d d λ y λ ∥ p d λ + 1 p − 1 ∫ λ 1 λ 2 ∥ δ λ ∥ p λ d λ , \left|\|\delta_{\lambda_2}\|_{p}-\|\delta_{\lambda_1}\|_{p}\right|\leq\int_{\lambda_1}^{\lambda_2}\left\|\frac{d}{d\lambda}y_{\lambda}\right\|_{p}\,d\lambda+\frac{1}{p-1}\int_{\lambda_1}^{\lambda_2}\frac{\|\delta_{\lambda}\|_{p}}{\lambda}\,d\lambda, ∣ ∥ δ λ 2 ∥ p − ∥ δ λ 1 ∥ p ∣ ≤ ∫ λ 1 λ 2 d λ d y λ p d λ + p − 1 1 ∫ λ 1 λ 2 λ ∥ δ λ ∥ p d λ ,
which proves the claim.
Lemma 15. Under the same context as Lemma 14, let Δ ^ ( λ ) = ∥ x ^ λ − y λ ∥ p \widehat{\Delta}(\lambda)=\|\widehat{x}_{\lambda}-y_{\lambda}\|_{p} Δ ( λ ) = ∥ x λ − y λ ∥ p . If ∥ x − v ∥ p ≤ 2 R \|x-v\|_{p}\le2R ∥ x − v ∥ p ≤ 2 R , then for every 0 < λ 1 ≤ λ 2 0<\lambda_{1}\le\lambda_{2} 0 < λ 1 ≤ λ 2 ,
∣ Δ ^ ( λ 1 ) − Δ ^ ( λ 2 ) ∣ ≤ [ R + 1 p − 1 ( G λ 1 ) 1 / ( p − 1 ) ] log λ 2 λ 1 . \left|\widehat{\Delta}(\lambda_{1})-\widehat{\Delta}(\lambda_{2})\right|\le\left[R+\frac{1}{p-1}\left(\frac{G}{\lambda_{1}}\right)^{1/(p-1)}\right]\log\frac{\lambda_{2}}{\lambda_{1}}. Δ ( λ 1 ) − Δ ( λ 2 ) ≤ [ R + p − 1 1 ( λ 1 G ) 1/ ( p − 1 ) ] log λ 1 λ 2 . Proof. By Lemma 14,
∣ Δ ^ ( λ 1 ) − Δ ^ ( λ 2 ) ∣ ≤ ∫ λ 1 λ 2 ∥ d d λ y λ ∥ p d λ + 1 p − 1 ∫ λ 1 λ 2 Δ ^ ( λ ) λ d λ . \left|\widehat{\Delta}(\lambda_{1})-\widehat{\Delta}(\lambda_{2})\right|\le\int_{\lambda_{1}}^{\lambda_{2}}\left\|\frac{d}{d\lambda}y_{\lambda}\right\|_{p}\,d\lambda+\frac{1}{p-1}\int_{\lambda_{1}}^{\lambda_{2}}\frac{\widehat{\Delta}(\lambda)}{\lambda}\,d\lambda. Δ ( λ 1 ) − Δ ( λ 2 ) ≤ ∫ λ 1 λ 2 d λ d y λ p d λ + p − 1 1 ∫ λ 1 λ 2 λ Δ ( λ ) d λ . Lemma 13 gives ∥ d d λ y λ ∥ p ≤ R λ \left\|\frac{d}{d\lambda}y_{\lambda}\right\|_{p}\le\frac{R}{\lambda} d λ d y λ p ≤ λ R , and Lemma 12 gives
Δ ^ ( λ ) ≤ ( G λ ) 1 / ( p − 1 ) ≤ ( G λ 1 ) 1 / ( p − 1 ) \widehat{\Delta}(\lambda)\le\left(\frac{G}{\lambda}\right)^{1/(p-1)}\le\left(\frac{G}{\lambda_{1}}\right)^{1/(p-1)} Δ ( λ ) ≤ ( λ G ) 1/ ( p − 1 ) ≤ ( λ 1 G ) 1/ ( p − 1 ) for every λ ∈ [ λ 1 , λ 2 ] \lambda\in[\lambda_{1},\lambda_{2}] λ ∈ [ λ 1 , λ 2 ] . Therefore,
∣ Δ ^ ( λ 1 ) − Δ ^ ( λ 2 ) ∣ ≤ [ R + 1 p − 1 ( G λ 1 ) 1 / ( p − 1 ) ] ∫ λ 1 λ 2 d λ λ , \left|\widehat{\Delta}(\lambda_{1})-\widehat{\Delta}(\lambda_{2})\right|\le\left[R+\frac{1}{p-1}\left(\frac{G}{\lambda_{1}}\right)^{1/(p-1)}\right]\int_{\lambda_{1}}^{\lambda_{2}}\frac{d\lambda}{\lambda}, Δ ( λ 1 ) − Δ ( λ 2 ) ≤ [ R + p − 1 1 ( λ 1 G ) 1/ ( p − 1 ) ] ∫ λ 1 λ 2 λ d λ , which proves the claim.
Lemma 16. Let x ^ λ = argmin x { f ( x ) + λ p ∥ x − y λ ∥ p p } \widehat{x}_{\lambda}=\operatorname*{argmin}_{x}\left\{f(x)+\frac{\lambda}{p}\|x-y_{\lambda}\|_{p}^{p}\right\} x λ = argmin x { f ( x ) + p λ ∥ x − y λ ∥ p p } and Δ ^ ( λ ) = ∥ x ^ λ − y λ ∥ p \widehat{\Delta}(\lambda)=\|\widehat{x}_{\lambda}-y_{\lambda}\|_{p} Δ ( λ ) = ∥ x λ − y λ ∥ p . Define
Δ ( λ ) = ∥ O λ , ρ ( y λ ) − y λ ∥ p . \Delta(\lambda)=\|\mathcal{O}_{\lambda,\rho}(y_{\lambda})-y_{\lambda}\|_{p}. Δ ( λ ) = ∥ O λ , ρ ( y λ ) − y λ ∥ p . If either Δ ^ ( λ ) ≤ r − ρ \widehat{\Delta}(\lambda)\le r-\rho Δ ( λ ) ≤ r − ρ or Δ ( λ ) < r − ρ \Delta(\lambda)<r-\rho Δ ( λ ) < r − ρ , then
bprox λ , r f ( y λ ) = prox λ f ( y λ ) and ∣ Δ ( λ ) − Δ ^ ( λ ) ∣ ≤ ρ . \operatorname{bprox}^{f}_{\lambda,r}(y_{\lambda})=\operatorname{prox}^{f}_{\lambda}(y_{\lambda})\quad\text{and}\quad|\Delta(\lambda)-\widehat{\Delta}(\lambda)|\le\rho. bprox λ , r f ( y λ ) = prox λ f ( y λ ) and ∣Δ ( λ ) − Δ ( λ ) ∣ ≤ ρ . where bprox λ , r f ( y λ ) = x λ , r ∗ \operatorname{bprox}^{f}_{\lambda,r}(y_{\lambda})=x^{*}_{\lambda,r} bprox λ , r f ( y λ ) = x λ , r ∗ and prox λ f ( y λ ) = x ^ λ \operatorname{prox}^{f}_{\lambda}(y_{\lambda})=\widehat{x}_{\lambda} prox λ f ( y λ ) = x λ
Proof. Let
F λ ( x ) = f ( x ) + λ p ∥ x − y λ ∥ p p , x λ , r ∗ = argmin x ∈ B r ( y λ ) F λ ( x ) . F_{\lambda}(x)=f(x)+\frac{\lambda}{p}\|x-y_{\lambda}\|_{p}^{p},\qquad x^{*}_{\lambda,r}=\operatorname*{argmin}_{x\in B_{r}(y_{\lambda})}F_{\lambda}(x). F λ ( x ) = f ( x ) + p λ ∥ x − y λ ∥ p p , x λ , r ∗ = x ∈ B r ( y λ ) argmin F λ ( x ) . By Definition 5,
F λ ( O λ , ρ ( y λ ) ) ≤ F λ ( x λ , r ∗ ) + λ 2 2 − p p ρ p . (33) F_{\lambda}(\mathcal{O}_{\lambda,\rho}(y_{\lambda}))\le F_{\lambda}(x^{*}_{\lambda,r})+\frac{\lambda2^{2-p}}{p}\rho^{p}. \tag*{(33)} F λ ( O λ , ρ ( y λ )) ≤ F λ ( x λ , r ∗ ) + p λ 2 2 − p ρ p . (33) Case 1: Δ ^ ( λ ) ≤ r − ρ \widehat{\Delta}(\lambda)\le r-\rho Δ ( λ ) ≤ r − ρ . Since x ^ λ \widehat{x}_{\lambda} x λ belongs to B r ( y λ ) B_{r}(y_{\lambda}) B r ( y λ ) and minimizes F λ F_{\lambda} F λ globally, we have x λ , r ∗ = x ^ λ x^{*}_{\lambda,r}=\widehat{x}_{\lambda} x λ , r ∗ = x λ . By the p p p -uniform convexity of 1 p ∥ ⋅ ∥ p p \frac{1}{p}\|\cdot\|_{p}^{p} p 1 ∥ ⋅ ∥ p p and the global optimality of x ^ λ \widehat{x}_{\lambda} x λ ,
F λ ( O λ , ρ ( y λ ) ) − F λ ( x ^ λ ) ≥ λ 2 2 − p p ∥ O λ , ρ ( y λ ) − x ^ λ ∥ p p . F_{\lambda}(\mathcal{O}_{\lambda,\rho}(y_{\lambda}))-F_{\lambda}(\widehat{x}_{\lambda})\ge\frac{\lambda2^{2-p}}{p}\|\mathcal{O}_{\lambda,\rho}(y_{\lambda})-\widehat{x}_{\lambda}\|_{p}^{p}. F λ ( O λ , ρ ( y λ )) − F λ ( x λ ) ≥ p λ 2 2 − p ∥ O λ , ρ ( y λ ) − x λ ∥ p p . Combining this with (33) gives
∥ O λ , ρ ( y λ ) − x ^ λ ∥ p ≤ ρ . (34) \|\mathcal{O}_{\lambda,\rho}(y_{\lambda})-\widehat{x}_{\lambda}\|_{p}\le\rho. \tag*{(34)} ∥ O λ , ρ ( y λ ) − x λ ∥ p ≤ ρ . (34)
Case 2: Δ ( λ ) < r − ρ \Delta(\lambda)<r-\rho Δ ( λ ) < r − ρ . By the definition of Δ ( λ ) \Delta(\lambda) Δ ( λ ) , O λ , ρ ( y λ ) ∈ B r ( y λ ) O_{\lambda,\rho}(y_\lambda)\in B_r(y_\lambda) O λ , ρ ( y λ ) ∈ B r ( y λ ) . Since x λ , r ∗ x^*_{\lambda,r} x λ , r ∗ minimizes F λ F_\lambda F λ over this convex ball, the constrained first-order optimality condition and p p p -uniform convexity imply
F λ ( O λ , ρ ( y λ ) ) − F λ ( x λ , r ∗ ) ≥ λ 2 2 − p p ∥ O λ , ρ ( y λ ) − x λ , r ∗ ∥ p p . F_\lambda(O_{\lambda,\rho}(y_\lambda))-F_\lambda(x^*_{\lambda,r})\ge\frac{\lambda2^{2-p}}{p}\left\|O_{\lambda,\rho}(y_\lambda)-x^*_{\lambda,r}\right\|_p^p. F λ ( O λ , ρ ( y λ )) − F λ ( x λ , r ∗ ) ≥ p λ 2 2 − p O λ , ρ ( y λ ) − x λ , r ∗ p p .
Together with (33) , this yields
∥ O λ , ρ ( y λ ) − x λ , r ∗ ∥ p ≤ ρ . \left\|O_{\lambda,\rho}(y_\lambda)-x^*_{\lambda,r}\right\|_p\le\rho. O λ , ρ ( y λ ) − x λ , r ∗ p ≤ ρ .
Consequently,
∥ x λ , r ∗ − y λ ∥ p ≤ ∥ x λ , r ∗ − O λ , ρ ( y λ ) ∥ p + Δ ( λ ) ≤ ρ + Δ ( λ ) < r . \left\|x^*_{\lambda,r}-y_\lambda\right\|_p\le\left\|x^*_{\lambda,r}-O_{\lambda,\rho}(y_\lambda)\right\|_p+\Delta(\lambda)\le\rho+\Delta(\lambda)<r. x λ , r ∗ − y λ p ≤ x λ , r ∗ − O λ , ρ ( y λ ) p + Δ ( λ ) ≤ ρ + Δ ( λ ) < r .
Thus x λ , r ∗ x^*_{\lambda,r} x λ , r ∗ lies in the interior of B r ( y λ ) B_r(y_\lambda) B r ( y λ ) . By convexity, an interior minimizer over the ball is also a global minimizer of F λ F_\lambda F λ . Hence x λ , r ∗ = x ^ λ x^*_{\lambda,r}=\hat{x}_\lambda x λ , r ∗ = x ^ λ , and (34) holds in this case as well. Therefore, in both cases,
bprox λ , r f ( y λ ) = prox λ f ( y λ ) . \operatorname{bprox}^{f}_{\lambda,r}(y_\lambda)=\operatorname{prox}^{f}_{\lambda}(y_\lambda). bprox λ , r f ( y λ ) = prox λ f ( y λ ) .
Finally, the reverse triangle inequality gives
∣ Δ ( λ ) − Δ ^ ( λ ) ∣ ≤ ∥ O λ , ρ ( y λ ) − x ^ λ ∥ p ≤ ρ . \left|\Delta(\lambda)-\widehat{\Delta}(\lambda)\right|\le\left\|O_{\lambda,\rho}(y_\lambda)-\hat{x}_\lambda\right\|_p\le\rho. Δ ( λ ) − Δ ( λ ) ≤ ∥ O λ , ρ ( y λ ) − x ^ λ ∥ p ≤ ρ .
Theorem 8. Let f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R be convex and G G G -Lipschitz, let x , v ∈ R d x,v\in\mathbb{R}^d x , v ∈ R d satisfy ∥ x − v ∥ p ≤ 2 R \left\|x-v\right\|_p\le2R ∥ x − v ∥ p ≤ 2 R , 0 < r ≤ R 0<r\le R 0 < r ≤ R , and let
x ^ λ = arg min x { f ( x ) + λ p ∥ x − y λ ∥ p p } , Δ ^ ( λ ) = ∥ x ^ λ − y λ ∥ p . \hat{x}_\lambda=\underset{x}{\arg\min}\left\{f(x)+\frac{\lambda}{p}\left\|x-y_\lambda\right\|_p^p\right\},\qquad\widehat{\Delta}(\lambda)=\left\|\hat{x}_\lambda-y_\lambda\right\|_p. x ^ λ = x arg min { f ( x ) + p λ ∥ x − y λ ∥ p p } , Δ ( λ ) = ∥ x ^ λ − y λ ∥ p . Given that λ max ≥ 2 p − 1 G r p − 1 \lambda_{\max}\ge\frac{2^{p-1}G}{r^{p-1}} λ m a x ≥ r p − 1 2 p − 1 G and λ min ∈ ( 0 , λ max ) \lambda_{\min}\in(0,\lambda_{\max}) λ m i n ∈ ( 0 , λ m a x ) , Algorithm 7 returns λ ∈ [ λ min , λ max ] \lambda\in[\lambda_{\min},\lambda_{\max}] λ ∈ [ λ m i n , λ m a x ] such that Δ ^ ( λ ) ≤ r \widehat{\Delta}(\lambda)\le r Δ ( λ ) ≤ r . Moreover, either
λ < 2 λ min \lambda<2\lambda_{\min} λ < 2 λ m i n ,
λ ∈ [ 2 λ min , λ max ] \lambda\in[2\lambda_{\min},\lambda_{\max}] λ ∈ [ 2 λ m i n , λ m a x ] and Δ ^ ( λ ) ∈ [ 3 r 4 , r ) \widehat{\Delta}(\lambda)\in\left[\frac{3r}{4},r\right) Δ ( λ ) ∈ [ 4 3 r , r ) .
Furthermore, the overall BROO complexity is
O ( log λ max λ min + log ( 1 + R + 1 p − 1 ( G λ min ) 1 / ( p − 1 ) r ) ) . O\left(\log\frac{\lambda_{\max}}{\lambda_{\min}}+\log\left(1+\frac{R+\frac{1}{p-1}\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)}}{r}\right)\right). O log λ m i n λ m a x + log 1 + r R + p − 1 1 ( λ m i n G ) 1/ ( p − 1 ) . Proof. First, by Lemma 12,
Δ ^ ( λ max ) ≤ ( G λ max ) 1 / ( p − 1 ) ≤ ( G 2 p − 1 G / r p − 1 ) 1 / ( p − 1 ) = r 2 . \widehat{\Delta}(\lambda_{\max})\le\left(\frac{G}{\lambda_{\max}}\right)^{1/(p-1)}\le\left(\frac{G}{2^{p-1}G/r^{p-1}}\right)^{1/(p-1)}=\frac{r}{2}. Δ ( λ m a x ) ≤ ( λ m a x G ) 1/ ( p − 1 ) ≤ ( 2 p − 1 G / r p − 1 G ) 1/ ( p − 1 ) = 2 r . Since r 2 < r − r 17 \frac{r}{2}<r-\frac{r}{17} 2 r < r − 17 r , Lemma 16 with ρ = r / 17 \rho=r/17 ρ = r /17 gives
Δ ( λ max ) ≤ Δ ^ ( λ max ) + r 17 ≤ r 2 + r 17 < 13 r 16 . \Delta(\lambda_{\max})\le\widehat{\Delta}(\lambda_{\max})+\frac{r}{17}\le\frac{r}{2}+\frac{r}{17}<\frac{13r}{16}. Δ ( λ m a x ) ≤ Δ ( λ m a x ) + 17 r ≤ 2 r + 17 r < 16 13 r .
Therefore, the first halving loop executes at least once. Suppose first that the halving loop terminates because λ < λ min \lambda< \lambda_{\min} λ < λ m i n . Let λ k \lambda_k λ k denote this value and λ k − 1 = 2 λ k \lambda_{k-1} = 2\lambda_k λ k − 1 = 2 λ k the preceding iterate. Since the loop body executed at λ k − 1 \lambda_{k-1} λ k − 1 ,
λ k − 1 ≥ λ min , Δ ( λ k − 1 ) ≤ 13 r 16 . \lambda_{k-1} \ge\lambda_{\min}, \qquad\Delta(\lambda_{k-1}) \le\frac{13r}{16}. λ k − 1 ≥ λ m i n , Δ ( λ k − 1 ) ≤ 16 13 r .
Because 13 r 16 < r − r 17 \frac{13r}{16} < r - \frac{r}{17} 16 13 r < r − 17 r , Lemma 16 gives Δ ^ ( λ k − 1 ) ≤ 13 r 16 + r 17 < r \widehat{\Delta}(\lambda_{k-1}) \le\frac{13r}{16} + \frac{r}{17} < r Δ ( λ k − 1 ) ≤ 16 13 r + 17 r < r . The algorithm returns
λ k − 1 = 2 λ k < 2 λ min , \lambda_{k-1} = 2\lambda_k < 2\lambda_{\min}, λ k − 1 = 2 λ k < 2 λ m i n ,
so the first case holds. For the remainder of the proof, the halving loop terminates because Δ ( λ ℓ ) > 13 r 16 \Delta(\lambda_\ell) > \frac{13r}{16} Δ ( λ ℓ ) > 16 13 r while λ ℓ ≥ λ min \lambda_\ell\ge\lambda_{\min} λ ℓ ≥ λ m i n . Since the loop executed at the previous iterate λ u = 2 λ ℓ \lambda_u = 2\lambda_\ell λ u = 2 λ ℓ , Δ ( λ u ) ≤ 13 r 16 \Delta(\lambda_u) \le\frac{13r}{16} Δ ( λ u ) ≤ 16 13 r . If Δ ( λ ℓ ) ≤ 15 r 16 \Delta(\lambda_\ell) \le\frac{15r}{16} Δ ( λ ℓ ) ≤ 16 15 r , then Δ ( λ ℓ ) ∈ ( 13 r 16 , 15 r 16 ] \Delta(\lambda_\ell) \in\left(\frac{13r}{16}, \frac{15r}{16}\right] Δ ( λ ℓ ) ∈ ( 16 13 r , 16 15 r ] . Since 15 r 16 < r − r 17 \frac{15r}{16} < r - \frac{r}{17} 16 15 r < r − 17 r , Lemma 16 gives
Δ ^ ( λ ℓ ) ∈ ( 13 r 16 − r 17 , 15 r 16 + r 17 ] ⊂ [ 3 r 4 , r ) . \widehat{\Delta}(\lambda_\ell) \in\left(\frac{13r}{16} - \frac{r}{17}, \frac{15r}{16} + \frac{r}{17}\right] \subset\left[\frac{3r}{4},r\right). Δ ( λ ℓ ) ∈ ( 16 13 r − 17 r , 16 15 r + 17 r ] ⊂ [ 4 3 r , r ) .
Thus Δ ^ ( λ ℓ ) < r \widehat{\Delta}(\lambda_\ell) < r Δ ( λ ℓ ) < r . It remains to consider the binary-search phase. At the beginning of this phase,
Δ ( λ ℓ ) > 15 r 16 , Δ ( λ u ) ≤ 13 r 16 . \Delta(\lambda_\ell) > \frac{15r}{16}, \qquad\Delta(\lambda_u) \le\frac{13r}{16}. Δ ( λ ℓ ) > 16 15 r , Δ ( λ u ) ≤ 16 13 r .
The update rule preserves these two inequalities throughout the binary search. If the binary search terminates because
Δ ( λ m ) ∈ [ 13 r 16 , 15 r 16 ] , \Delta(\lambda_m) \in\left[\frac{13r}{16},\frac{15r}{16}\right], Δ ( λ m ) ∈ [ 16 13 r , 16 15 r ] ,
then, exactly as above, Lemma 16 gives Δ ^ ( λ m ) ∈ [ 3 r 4 , r ) \widehat{\Delta}(\lambda_m) \in\left[\frac{3r}{4},r\right) Δ ( λ m ) ∈ [ 4 3 r , r ) . Finally, suppose the binary search terminates because
log λ u λ ℓ ≤ r 8 ( R + 1 p − 1 ( G λ ℓ ) 1 / ( p − 1 ) ) . \log\frac{\lambda_u}{\lambda_\ell} \le\frac{r}{8\left(R+\frac{1}{p-1}\left(\frac{G}{\lambda_\ell}\right)^{1/(p-1)}\right)}. log λ ℓ λ u ≤ 8 ( R + p − 1 1 ( λ ℓ G ) 1/ ( p − 1 ) ) r .
Then, Lemma 16 gives Δ ^ ( λ u ) ≤ 13 r 16 + r 17 < 7 r 8 \widehat{\Delta}(\lambda_u) \le\frac{13r}{16} + \frac{r}{17} < \frac{7r}{8} Δ ( λ u ) ≤ 16 13 r + 17 r < 8 7 r . We next show that Δ ^ ( λ ℓ ) > 7 r 8 \widehat{\Delta}(\lambda_\ell) > \frac{7r}{8} Δ ( λ ℓ ) > 8 7 r : Suppose instead that Δ ^ ( λ ℓ ) ≤ 7 r 8 \widehat{\Delta}(\lambda_\ell) \le\frac{7r}{8} Δ ( λ ℓ ) ≤ 8 7 r . Since 7 r 8 < r − r 17 \frac{7r}{8} < r - \frac{r}{17} 8 7 r < r − 17 r , Lemma 16 would imply
Δ ( λ ℓ ) ≤ 7 r 8 + r 17 < 15 r 16 , \Delta(\lambda_\ell) \le\frac{7r}{8} + \frac{r}{17} < \frac{15r}{16}, Δ ( λ ℓ ) ≤ 8 7 r + 17 r < 16 15 r ,
contradicting the binary-search invariant. Therefore, Δ ^ ( λ u ) < 7 r 8 < Δ ^ ( λ ℓ ) \widehat{\Delta}(\lambda_u) < \frac{7r}{8} < \widehat{\Delta}(\lambda_\ell) Δ ( λ u ) < 8 7 r < Δ ( λ ℓ ) . By Lemma 15, Δ ^ \widehat{\Delta} Δ is continuous. It follows by the Intermediate Value that there exists λ ∗ ∈ ( λ ℓ , λ u ) \lambda^* \in(\lambda_\ell,\lambda_u) λ ∗ ∈ ( λ ℓ , λ u ) such that Δ ^ ( λ ∗ ) = 7 r 8 \widehat{\Delta}(\lambda^*) = \frac{7r}{8} Δ ( λ ∗ ) = 8 7 r . Since λ m = λ u λ ℓ \lambda_m = \sqrt{\lambda_u\lambda_\ell} λ m = λ u λ ℓ , we have ∣ log λ m λ ∗ ∣ ≤ 1 2 log λ u λ ℓ \left|\log\frac{\lambda_m}{\lambda^*}\right| \le\frac{1}{2}\log\frac{\lambda_u}{\lambda_\ell} log λ ∗ λ m ≤ 2 1 log λ ℓ λ u . Applying Lemma 15 to λ m \lambda_m λ m and λ ∗ \lambda^* λ ∗ , and using min { λ m , λ ∗ } ≥ λ ℓ \min\{\lambda_m,\lambda^*\} \ge\lambda_\ell min { λ m , λ ∗ } ≥ λ ℓ , gives
∣ Δ ^ ( λ m ) − Δ ^ ( λ ∗ ) ∣ ≤ [ R + 1 p − 1 ( G λ ℓ ) 1 / ( p − 1 ) ] ∣ log λ m λ ∗ ∣ ≤ 1 2 [ R + 1 p − 1 ( G λ ℓ ) 1 / ( p − 1 ) ] log λ u λ ℓ ≤ r 16 . \begin{aligned}
\left|\widehat{\Delta}(\lambda_m)-\widehat{\Delta}(\lambda^*)\right|
\le\left[R+\frac{1}{p-1}\left(\frac{G}{\lambda_\ell}\right)^{1/(p-1)}\right]\left|\log\frac{\lambda_m}{\lambda^*}\right| \\
&\le\frac{1}{2}\left[R+\frac{1}{p-1}\left(\frac{G}{\lambda_\ell}\right)^{1/(p-1)}\right]\log\frac{\lambda_u}{\lambda_\ell}\le\frac{r}{16}.
\end{aligned} Δ ( λ m ) − Δ ( λ ∗ ) ≤ [ R + p − 1 1 ( λ ℓ G ) 1/ ( p − 1 ) ] log λ ∗ λ m ≤ 2 1 [ R + p − 1 1 ( λ ℓ G ) 1/ ( p − 1 ) ] log λ ℓ λ u ≤ 16 r .
Since Δ ^ ( λ ∗ ) = 7 r 8 \widehat{\Delta}(\lambda^*) = \frac{7r}{8} Δ ( λ ∗ ) = 8 7 r , it follows that Δ ^ ( λ m ) ∈ [ 13 r 16 , 15 r 16 ] ⊂ [ 3 r 4 , r ) \widehat{\Delta}(\lambda_m) \in\left[\frac{13r}{16},\frac{15r}{16}\right] \subset\left[\frac{3r}{4},r\right) Δ ( λ m ) ∈ [ 16 13 r , 16 15 r ] ⊂ [ 4 3 r , r ) . It remains to bound the number of BROO calls. The halving phase clearly requires at most O ( log λ max λ min ) O\left(\log\frac{\lambda_{\max}}{\lambda_{\min}}\right) O ( log λ m i n λ m a x ) calls. For the binary-search phase, initially log λ u λ ℓ = log 2 \log\frac{\lambda_u}{\lambda_\ell}=\log2 log λ ℓ λ u = log 2 . Since λ m = λ u λ ℓ \lambda_m=\sqrt{\lambda_u\lambda_\ell} λ m = λ u λ ℓ , each bisection halves the logarithmic interval width. Therefore, after k k k iterations, log λ u λ ℓ = log 2 2 k \log\frac{\lambda_u}{\lambda_\ell}=\frac{\log2}{2^k} log λ ℓ λ u = 2 k l o g 2 . The binary search terminates once
log λ u λ ℓ ≤ r 8 [ R + 1 p − 1 ( G λ ℓ ) 1 / ( p − 1 ) ] \log\frac{\lambda_u}{\lambda_\ell}\leq\frac{r}{8\left[R+\frac{1}{p-1}\left(\frac{G}{\lambda_\ell}\right)^{1/(p-1)}\right]} log λ ℓ λ u ≤ 8 [ R + p − 1 1 ( λ ℓ G ) 1/ ( p − 1 ) ] r
Throughout the binary search λ ℓ ≥ λ min \lambda_\ell\geq\lambda_{\min} λ ℓ ≥ λ m i n . Therefore, ( G λ ℓ ) 1 / ( p − 1 ) ≤ ( G λ min ) 1 / ( p − 1 ) \left(\frac{G}{\lambda_\ell}\right)^{1/(p-1)}\leq\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)} ( λ ℓ G ) 1/ ( p − 1 ) ≤ ( λ m i n G ) 1/ ( p − 1 ) . Consequently
r 8 [ R + 1 p − 1 ( G λ ℓ ) 1 / ( p − 1 ) ] ≥ r 8 [ R + 1 p − 1 ( G λ min ) 1 / ( p − 1 ) ] \frac{r}{8\left[R+\frac{1}{p-1}\left(\frac{G}{\lambda_\ell}\right)^{1/(p-1)}\right]}\geq\frac{r}{8\left[R+\frac{1}{p-1}\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)}\right]} 8 [ R + p − 1 1 ( λ ℓ G ) 1/ ( p − 1 ) ] r ≥ 8 [ R + p − 1 1 ( λ m i n G ) 1/ ( p − 1 ) ] r
Thus, it is sufficient to make the logarithmic interval width smaller than the right-hand side above. In particular, the binary search is guaranteed to have terminated once
log 2 2 k ≤ r 8 [ R + 1 p − 1 ( G λ min ) 1 / ( p − 1 ) ] ⇒ 2 k ≥ 8 log 2 r [ R + 1 p − 1 ( G λ min ) 1 / ( p − 1 ) ] \frac{\log2}{2^k}\leq\frac{r}{8\left[R+\frac{1}{p-1}\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)}\right]}\Rightarrow2^k\geq\frac{8\log2}{r}\left[R+\frac{1}{p-1}\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)}\right] 2 k log 2 ≤ 8 [ R + p − 1 1 ( λ m i n G ) 1/ ( p − 1 ) ] r ⇒ 2 k ≥ r 8 log 2 [ R + p − 1 1 ( λ m i n G ) 1/ ( p − 1 ) ]
Therefore, the number of binary-search iterations is at most
O ( log ( 1 + R + 1 p − 1 ( G λ min ) 1 / ( p − 1 ) r ) ) O\left(\log\left(1+\frac{R+\frac{1}{p-1}\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)}}{r}\right)\right) O log 1 + r R + p − 1 1 ( λ m i n G ) 1/ ( p − 1 )
Combining the halving and binary-search phases, the overall BROO complexity is
O ( log λ max λ min + log ( 1 + R + 1 p − 1 ( G λ min ) 1 / ( p − 1 ) r ) ) . O\left(\log\frac{\lambda_{\max}}{\lambda_{\min}}+\log\left(1+\frac{R+\frac{1}{p-1}\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)}}{r}\right)\right). O log λ m i n λ m a x + log 1 + r R + p − 1 1 ( λ m i n G ) 1/ ( p − 1 ) .
Proximal Point Method Guarantees
In the analysis, below, we define the filtration F k = σ ( x 1 , v 1 , A 1 , ζ 1 , … , x k , v k , A k , ζ k ) \mathcal{F}_k=\sigma(x_1,v_1,A_1,\zeta_1,\ldots,x_k,v_k,A_k,\zeta_k) F k = σ ( x 1 , v 1 , A 1 , ζ 1 , … , x k , v k , A k , ζ k ) , where ζ i \zeta_i ζ i represents the internal randomness in NextLambda( x i , v i , A i ) (x_i,v_i,A_i) ( x i , v i , A i ) . Furthermore, we let x ~ k = P f , λ k ( y k − 1 ) \tilde{x}_k=P_{f,\lambda_k}(y_{k-1}) x ~ k = P f , λ k ( y k − 1 ) , i.e the exact proximal mapping for which x k x_k x k approximates.
Lemma 17. Let f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R be convex. If X ⊆ B R ( x 0 ) \mathcal{X}\subseteq B_R(x_0) X ⊆ B R ( x 0 ) , it holds that
E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k + 1 ) ] \mathbb{E}[A_{k+1}(f(x_{k+1})-f(x^*))+2D_{\Phi}(x^*,v_{k+1})] E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ )) + 2 D Φ ( x ∗ , v k + 1 )] ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) + A k + 1 φ k + 1 − 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ~ k + 1 − y k ∥ p p \leq A_k(f(x_k)-f(x^*))+2D_{\Phi}(x^*,v_k)+A_{k+1}\varphi_{k+1}-\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\lVert\tilde{x}_{k+1}-y_k\rVert_p^p ≤ A k ( f ( x k ) − f ( x ∗ )) + 2 D Φ ( x ∗ , v k ) + A k + 1 φ k + 1 − q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ~ k + 1 − y k ∥ p p + 2 2 − q a k + 1 q q σ k + 1 q + 2 a k + 1 δ k + 1 R +\frac{2^{2-q}a_{k+1}^q}{q}\sigma_{k+1}^q+2a_{k+1}\delta_{k+1}R + q 2 2 − q a k + 1 q σ k + 1 q + 2 a k + 1 δ k + 1 R
Proof. Following a similar approach to [2 ] , we let g ^ k = ∇ f λ k ( y k − 1 ) \hat{g}_k=\nabla f_{\lambda_k}(y_{k-1}) g ^ k = ∇ f λ k ( y k − 1 ) and we bound from above and below the quantity a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ a_{k+1}\langle\hat{g}_{k+1},v_k-x^*\rangle a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ . First, observe that
v k − x ∗ = x ^ k + 1 − x ∗ + A k a k + 1 ( x ^ k + 1 − x k ) − A k + 1 a k + 1 ( x ^ k + 1 − y k ) v_k-x^*=\hat{x}_{k+1}-x^*+\frac{A_k}{a_{k+1}}(\hat{x}_{k+1}-x_k)-\frac{A_{k+1}}{a_{k+1}}(\hat{x}_{k+1}-y_k) v k − x ∗ = x ^ k + 1 − x ∗ + a k + 1 A k ( x ^ k + 1 − x k ) − a k + 1 A k + 1 ( x ^ k + 1 − y k )
which implies that
⟨ g ^ k + 1 , v k − x ∗ ⟩ = ⟨ g ^ k + 1 , x ^ k + 1 − x ∗ ⟩ + A k a k + 1 ⟨ g ^ k + 1 , x ^ k + 1 − x k ⟩ − A k + 1 a k + 1 ⟨ g ^ k + 1 , x ^ k + 1 − y k ⟩ ≥ f ( x ^ k + 1 ) − f ( x ∗ ) + A k a k + 1 ( f ( x ^ k + 1 ) − f ( x k ) ) + λ k + 1 A k + 1 a k + 1 ∥ x ^ k + 1 − y k ∥ p p = f ( x ^ k + 1 ) − f ( x ∗ ) + A k a k + 1 ( f ( x ^ k + 1 ) − f ( x ∗ ) − ( f ( x k ) − f ( x ∗ ) ) ) + λ k + 1 A k + 1 a k + 1 ∥ x ^ k + 1 − y k ∥ p p = A k + 1 a k + 1 ( f ( x ^ k + 1 ) − f ( x ∗ ) ) − A k a k + 1 ( f ( x k ) − f ( x ∗ ) ) + λ k + 1 A k + 1 a k + 1 ∥ x ^ k + 1 − y k ∥ p p . \begin{aligned}
\langle\hat{g}_{k+1},v_k-x^*\rangle
={}&\langle\hat{g}_{k+1},\hat{x}_{k+1}-x^*\rangle+\frac{A_k}{a_{k+1}}\langle\hat{g}_{k+1},\hat{x}_{k+1}-x_k\rangle\\
&-\frac{A_{k+1}}{a_{k+1}}\langle\hat{g}_{k+1},\hat{x}_{k+1}-y_k\rangle\\
\ge{}&f(\hat{x}_{k+1})-f(x^*)+\frac{A_k}{a_{k+1}}(f(\hat{x}_{k+1})-f(x_k))+\frac{\lambda_{k+1}A_{k+1}}{a_{k+1}}\lVert\hat{x}_{k+1}-y_k\rVert_p^p\\
={}&f(\hat{x}_{k+1})-f(x^*)+\frac{A_k}{a_{k+1}}\bigl(f(\hat{x}_{k+1})-f(x^*)-(f(x_k)-f(x^*))\bigr)\\
&+\frac{\lambda_{k+1}A_{k+1}}{a_{k+1}}\lVert\hat{x}_{k+1}-y_k\rVert_p^p\\
={}&\frac{A_{k+1}}{a_{k+1}}(f(\hat{x}_{k+1})-f(x^*))-\frac{A_k}{a_{k+1}}(f(x_k)-f(x^*))\\
&+\frac{\lambda_{k+1}A_{k+1}}{a_{k+1}}\lVert\hat{x}_{k+1}-y_k\rVert_p^p.
\end{aligned} ⟨ g ^ k + 1 , v k − x ∗ ⟩ = ≥ = = ⟨ g ^ k + 1 , x ^ k + 1 − x ∗ ⟩ + a k + 1 A k ⟨ g ^ k + 1 , x ^ k + 1 − x k ⟩ − a k + 1 A k + 1 ⟨ g ^ k + 1 , x ^ k + 1 − y k ⟩ f ( x ^ k + 1 ) − f ( x ∗ ) + a k + 1 A k ( f ( x ^ k + 1 ) − f ( x k )) + a k + 1 λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p f ( x ^ k + 1 ) − f ( x ∗ ) + a k + 1 A k ( f ( x ^ k + 1 ) − f ( x ∗ ) − ( f ( x k ) − f ( x ∗ )) ) + a k + 1 λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p a k + 1 A k + 1 ( f ( x ^ k + 1 ) − f ( x ∗ )) − a k + 1 A k ( f ( x k ) − f ( x ∗ )) + a k + 1 λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p .
where we used the fact that f f f is convex and that ⟨ g ^ k + 1 , x ^ k + 1 − y k ⟩ = − λ k + 1 ∥ x ^ k + 1 − y k ∥ p p \langle\hat{g}_{k+1},\hat{x}_{k+1}-y_k\rangle=-\lambda_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p ⟨ g ^ k + 1 , x ^ k + 1 − y k ⟩ = − λ k + 1 ∥ x ^ k + 1 − y k ∥ p p . We also have that
E [ f ( x k + 1 ) ∣ F k ] ≤ E [ f ( x k + 1 ) + λ k + 1 p ∥ x k + 1 − y k ∥ p p ] ≤ f ( x ^ k + 1 ) + λ k + 1 p ∥ x ^ k + 1 − y k ∥ p p + φ k + 1 \mathbb{E}[f(x_{k+1})\mid\mathcal{F}_k]\leq\mathbb{E}\left[f(x_{k+1})+\frac{\lambda_{k+1}}{p}\lVert x_{k+1}-y_k\rVert_p^p\right]\leq f(\hat{x}_{k+1})+\frac{\lambda_{k+1}}{p}\lVert\hat{x}_{k+1}-y_k\rVert_p^p+\varphi_{k+1} E [ f ( x k + 1 ) ∣ F k ] ≤ E [ f ( x k + 1 ) + p λ k + 1 ∥ x k + 1 − y k ∥ p p ] ≤ f ( x ^ k + 1 ) + p λ k + 1 ∥ x ^ k + 1 − y k ∥ p p + φ k + 1
which implies that
a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ ≥ A k + 1 ( E [ f ( x k + 1 ) ∣ F k ] − λ k + 1 p ∥ x ^ k + 1 − y k ∥ p p − φ k + 1 − f ( x ∗ ) ) − A k ( f ( x k ) − f ( x ∗ ) ) + λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p = A k + 1 ( E [ f ( x k + 1 ) ∣ F k ] − f ( x ∗ ) ) − A k ( f ( x k ) − f ( x ∗ ) ) + ( 1 − 1 p ) λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p − A k + 1 φ k + 1 . \begin{aligned}
a_{k+1}\langle\hat{g}_{k+1},v_k-x^*\rangle
\geq{}&A_{k+1}\left(\mathbb{E}[f(x_{k+1})\mid\mathcal{F}_k]-\frac{\lambda_{k+1}}{p}\lVert\hat{x}_{k+1}-y_k\rVert_p^p-\varphi_{k+1}-f(x^*)\right)\\
&-A_k(f(x_k)-f(x^*))+\lambda_{k+1}A_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p\\
={}&A_{k+1}(\mathbb{E}[f(x_{k+1})\mid\mathcal{F}_k]-f(x^*))-A_k(f(x_k)-f(x^*))\\
&+\left(1-\frac{1}{p}\right)\lambda_{k+1}A_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p-A_{k+1}\varphi_{k+1}.
\end{aligned} a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ ≥ = A k + 1 ( E [ f ( x k + 1 ) ∣ F k ] − p λ k + 1 ∥ x ^ k + 1 − y k ∥ p p − φ k + 1 − f ( x ∗ ) ) − A k ( f ( x k ) − f ( x ∗ )) + λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p A k + 1 ( E [ f ( x k + 1 ) ∣ F k ] − f ( x ∗ )) − A k ( f ( x k ) − f ( x ∗ )) + ( 1 − p 1 ) λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p − A k + 1 φ k + 1 .
We now upper bound a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ a_{k+1}\langle\hat{g}_{k+1},v_k-x^*\rangle a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ . We have that
v k + 1 = argmin v ∈ X { 1 2 a k + 1 ⟨ g k + 1 , v ⟩ + D Φ ( v , v k ) } = argmin v ∈ X { 1 2 a k + 1 ⟨ g k + 1 , v ⟩ + Φ ( v ) − ⟨ ∇ Φ ( v k ) , v ⟩ } v_{k+1}=\underset{v\in X}{\operatorname{argmin}}\left\{\frac{1}{2}a_{k+1}\langle g_{k+1},v\rangle+D_{\Phi}(v,v_k)\right\}=\underset{v\in X}{\operatorname{argmin}}\left\{\frac{1}{2}a_{k+1}\langle g_{k+1},v\rangle+\Phi(v)-\langle\nabla\Phi(v_k),v\rangle\right\} v k + 1 = v ∈ X argmin { 2 1 a k + 1 ⟨ g k + 1 , v ⟩ + D Φ ( v , v k ) } = v ∈ X argmin { 2 1 a k + 1 ⟨ g k + 1 , v ⟩ + Φ ( v ) − ⟨ ∇Φ ( v k ) , v ⟩ }
By first-order optimality conditions, it holds that 0 ∈ 1 2 a k + 1 g k + 1 + ∇ Φ ( v k + 1 ) − ∇ Φ ( v k ) + N X ( v k + 1 ) 0\in\frac{1}{2}a_{k+1}g_{k+1}+\nabla\Phi(v_{k+1})-\nabla\Phi(v_k)+N_X(v_{k+1}) 0 ∈ 2 1 a k + 1 g k + 1 + ∇Φ ( v k + 1 ) − ∇Φ ( v k ) + N X ( v k + 1 ) , where N X N_X N X denotes the normal cone, which implies that for all u ∈ X u\in X u ∈ X ,
⟨ 1 2 a k + 1 g k + 1 + ∇ Φ ( v k + 1 ) − ∇ Φ ( v k ) , u − v k + 1 ⟩ ≥ 0. \left\langle\frac{1}{2}a_{k+1}g_{k+1}+\nabla\Phi(v_{k+1})-\nabla\Phi(v_k),u-v_{k+1}\right\rangle\geq0. ⟨ 2 1 a k + 1 g k + 1 + ∇Φ ( v k + 1 ) − ∇Φ ( v k ) , u − v k + 1 ⟩ ≥ 0.
Therefore, it holds that
1 2 a k + 1 ⟨ g k + 1 , v k + 1 − x ∗ ⟩ ≤ ⟨ ∇ Φ ( v k + 1 ) − ∇ Φ ( v k ) , x ∗ − v k + 1 ⟩ \frac{1}{2}a_{k+1}\langle g_{k+1},v_{k+1}-x^*\rangle\leq\langle\nabla\Phi(v_{k+1})-\nabla\Phi(v_k),x^*-v_{k+1}\rangle 2 1 a k + 1 ⟨ g k + 1 , v k + 1 − x ∗ ⟩ ≤ ⟨ ∇Φ ( v k + 1 ) − ∇Φ ( v k ) , x ∗ − v k + 1 ⟩
Since ⟨ ∇ Φ ( v k + 1 ) − ∇ Φ ( v k ) , x ∗ − v k + 1 ⟩ = D Φ ( x ∗ , v k ) − D Φ ( x ∗ , v k + 1 ) − D Φ ( v k + 1 , v k ) \langle\nabla\Phi(v_{k+1})-\nabla\Phi(v_k),x^*-v_{k+1}\rangle=D_{\Phi}(x^*,v_k)-D_{\Phi}(x^*,v_{k+1})-D_{\Phi}(v_{k+1},v_k) ⟨ ∇Φ ( v k + 1 ) − ∇Φ ( v k ) , x ∗ − v k + 1 ⟩ = D Φ ( x ∗ , v k ) − D Φ ( x ∗ , v k + 1 ) − D Φ ( v k + 1 , v k ) , it holds that
1 2 a k + 1 ⟨ g k + 1 , v k + 1 − x ∗ ⟩ ≤ D Φ ( x ∗ , v k ) − D Φ ( x ∗ , v k + 1 ) − D Φ ( v k + 1 , v k ) \frac{1}{2}a_{k+1}\langle g_{k+1},v_{k+1}-x^*\rangle\leq D_{\Phi}(x^*,v_k)-D_{\Phi}(x^*,v_{k+1})-D_{\Phi}(v_{k+1},v_k) 2 1 a k + 1 ⟨ g k + 1 , v k + 1 − x ∗ ⟩ ≤ D Φ ( x ∗ , v k ) − D Φ ( x ∗ , v k + 1 ) − D Φ ( v k + 1 , v k )
Noting that v k − x ∗ = ( v k − v k + 1 ) + ( v k + 1 − x ∗ ) v_k-x^*=(v_k-v_{k+1})+(v_{k+1}-x^*) v k − x ∗ = ( v k − v k + 1 ) + ( v k + 1 − x ∗ ) , we have that
a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ = a k + 1 ⟨ g k + 1 , v k + 1 − x ∗ ⟩ + a k + 1 ⟨ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] , v k − v k + 1 ⟩ + a k + 1 ⟨ E [ g k + 1 ∣ F k ] − g ^ k + 1 , x ∗ − v k + 1 ⟩ − a k + 1 ⟨ g k + 1 − E [ g k + 1 ∣ F k ] , v k − x ∗ ⟩ . \begin{aligned}
a_{k+1}\langle\hat{g}_{k+1},v_k-x^*\rangle={}&a_{k+1}\langle g_{k+1},v_{k+1}-x^*\rangle\\
&+a_{k+1}\langle\hat{g}_{k+1}+g_{k+1}-\mathbb{E}[g_{k+1}\mid\mathcal{F}_k],v_k-v_{k+1}\rangle\\
&+a_{k+1}\langle\mathbb{E}[g_{k+1}\mid\mathcal{F}_k]-\hat{g}_{k+1},x^*-v_{k+1}\rangle\\
&-a_{k+1}\langle g_{k+1}-\mathbb{E}[g_{k+1}\mid\mathcal{F}_k],v_k-x^*\rangle.
\end{aligned} a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ = a k + 1 ⟨ g k + 1 , v k + 1 − x ∗ ⟩ + a k + 1 ⟨ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] , v k − v k + 1 ⟩ + a k + 1 ⟨ E [ g k + 1 ∣ F k ] − g ^ k + 1 , x ∗ − v k + 1 ⟩ − a k + 1 ⟨ g k + 1 − E [ g k + 1 ∣ F k ] , v k − x ∗ ⟩ .
By Hölder’s inequality, we have that for all a , b ∈ R d a,b\in\mathbb{R}^d a , b ∈ R d and η > 0 \eta>0 η > 0 , ⟨ a , b ⟩ ≤ η 1 − q q ∥ a ∥ q q + η p ∥ b ∥ p p \langle a,b\rangle\leq\frac{\eta^{1-q}}{q}\|a\|_q^q+\frac{\eta}{p}\|b\|_p^p ⟨ a , b ⟩ ≤ q η 1 − q ∥ a ∥ q q + p η ∥ b ∥ p p . By choosing η = 2 3 − p \eta=2^{3-p} η = 2 3 − p , it holds that
a k + 1 ⟨ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] , v k − v k + 1 ⟩ ≤ 2 3 − 2 q a k + 1 q q ∥ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] ∥ q q + 2 3 − p p ∥ v k − v k + 1 ∥ p p . \begin{aligned}
a_{k+1}\langle\hat{g}_{k+1}+g_{k+1}-\mathbb{E}[g_{k+1}\mid\mathcal{F}_k],v_k-v_{k+1}\rangle
\leq{}&\frac{2^{3-2q}a_{k+1}^q}{q}\|\hat{g}_{k+1}+g_{k+1}-\mathbb{E}[g_{k+1}\mid\mathcal{F}_k]\|_q^q\\
&+\frac{2^{3-p}}{p}\|v_k-v_{k+1}\|_p^p.
\end{aligned} a k + 1 ⟨ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] , v k − v k + 1 ⟩ ≤ q 2 3 − 2 q a k + 1 q ∥ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] ∥ q q + p 2 3 − p ∥ v k − v k + 1 ∥ p p .
By Fact 2, we have that − 2 D Φ ( v k + 1 , v k ) + 2 3 − p p ∥ v k − v k + 1 ∥ p p ≤ 0 -2D_\Phi(v_{k+1},v_k)+\frac{2^{3-p}}{p}\|v_k-v_{k+1}\|_p^p\leq0 − 2 D Φ ( v k + 1 , v k ) + p 2 3 − p ∥ v k − v k + 1 ∥ p p ≤ 0 , which implies that
a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ ≤ 2 D Φ ( x ∗ , v k ) − 2 E [ D Φ ( x ∗ , v k + 1 ) ∣ F k ] + 2 3 − 2 q a k + 1 q q E [ ∥ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] ∥ q q ∣ F k ] + a k + 1 E [ ⟨ E [ g k + 1 ∣ F k ] − g ^ k + 1 , x ∗ − v k + 1 ⟩ ∣ F k ] , \begin{aligned}
a_{k+1}\langle\hat{g}_{k+1},v_k-x^*\rangle\leq{}&2D_\Phi(x^*,v_k)-2\mathbb{E}[D_\Phi(x^*,v_{k+1})\mid\mathcal{F}_k]\\
&+\frac{2^{3-2q}a_{k+1}^q}{q}\mathbb{E}[\|\hat{g}_{k+1}+g_{k+1}-\mathbb{E}[g_{k+1}\mid\mathcal{F}_k]\|_q^q\mid\mathcal{F}_k]\\
&+a_{k+1}\mathbb{E}[\langle\mathbb{E}[g_{k+1}\mid\mathcal{F}_k]-\hat{g}_{k+1},x^*-v_{k+1}\rangle\mid\mathcal{F}_k],
\end{aligned} a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ ≤ 2 D Φ ( x ∗ , v k ) − 2 E [ D Φ ( x ∗ , v k + 1 ) ∣ F k ] + q 2 3 − 2 q a k + 1 q E [ ∥ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] ∥ q q ∣ F k ] + a k + 1 E [⟨ E [ g k + 1 ∣ F k ] − g ^ k + 1 , x ∗ − v k + 1 ⟩ ∣ F k ] ,
where we used the fact that v k v_k v k and x ∗ x^* x ∗ are F k \mathcal{F}_k F k -measurable and therefore
E [ ⟨ g k + 1 − E [ g k + 1 ∣ F k ] , v k − x ∗ ⟩ ∣ F k ] = 0. \mathbb{E}[\langle g_{k+1}-\mathbb{E}[g_{k+1}\mid\mathcal{F}_k],v_k-x^*\rangle\mid\mathcal{F}_k]=0. E [⟨ g k + 1 − E [ g k + 1 ∣ F k ] , v k − x ∗ ⟩ ∣ F k ] = 0.
It also holds that
E [ ∥ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] ∥ q q ∣ F k ] ≤ 2 q − 1 ∥ g ^ k + 1 ∥ q q + 2 q − 1 E [ ∥ g k + 1 − E [ g k + 1 ∣ F k ] ∥ q q ∣ F k ] ≤ 2 q − 1 σ k + 1 q + 2 q − 1 ∥ g ^ k + 1 ∥ q q . \begin{aligned}
\mathbb{E}[\|\hat{g}_{k+1}+g_{k+1}-\mathbb{E}[g_{k+1}\mid\mathcal{F}_k]\|_q^q\mid\mathcal{F}_k]
&\leq2^{q-1}\|\hat{g}_{k+1}\|_q^q+2^{q-1}\mathbb{E}[\|g_{k+1}-\mathbb{E}[g_{k+1}\mid\mathcal{F}_k]\|_q^q\mid\mathcal{F}_k]\\
&\leq2^{q-1}\sigma_{k+1}^q+2^{q-1}\|\hat{g}_{k+1}\|_q^q.
\end{aligned} E [ ∥ g ^ k + 1 + g k + 1 − E [ g k + 1 ∣ F k ] ∥ q q ∣ F k ] ≤ 2 q − 1 ∥ g ^ k + 1 ∥ q q + 2 q − 1 E [ ∥ g k + 1 − E [ g k + 1 ∣ F k ] ∥ q q ∣ F k ] ≤ 2 q − 1 σ k + 1 q + 2 q − 1 ∥ g ^ k + 1 ∥ q q .
Moreover, by the bias guarantee and the fact that X ⊆ B R ( x 0 ) \mathcal{X}\subseteq\mathbb{B}_R(x_0) X ⊆ B R ( x 0 ) , we have
E [ ⟨ E [ g k + 1 ∣ F k ] − g ^ k + 1 , x ∗ − v k + 1 ⟩ ∣ F k ] ≤ ∥ E [ g k + 1 ∣ F k ] − g ^ k + 1 ∥ q E [ ∥ x ∗ − v k + 1 ∥ p ∣ F k ] ≤ 2 δ k + 1 R . \begin{aligned}
\mathbb{E}[\langle\mathbb{E}[g_{k+1}\mid\mathcal{F}_k]-\hat{g}_{k+1},x^*-v_{k+1}\rangle\mid\mathcal{F}_k]
&\leq\|\mathbb{E}[g_{k+1}\mid\mathcal{F}_k]-\hat{g}_{k+1}\|_q\mathbb{E}[\|x^*-v_{k+1}\|_p\mid\mathcal{F}_k]\\
&\leq2\delta_{k+1}R.
\end{aligned} E [⟨ E [ g k + 1 ∣ F k ] − g ^ k + 1 , x ∗ − v k + 1 ⟩ ∣ F k ] ≤ ∥ E [ g k + 1 ∣ F k ] − g ^ k + 1 ∥ q E [ ∥ x ∗ − v k + 1 ∥ p ∣ F k ] ≤ 2 δ k + 1 R .
Therefore, we have that
a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ ≤ 2 D Φ ( x ∗ , v k ) − 2 E [ D Φ ( x ∗ , v k + 1 ) ∣ F k ] + 2 2 − q a k + 1 q q ( σ k + 1 q + ∥ g ^ k + 1 ∥ q q ) + 2 a k + 1 δ k + 1 R = 2 D Φ ( x ∗ , v k ) − 2 E [ D Φ ( x ∗ , v k + 1 ) ∣ F k ] + 2 2 − q a k + 1 q q ( σ k + 1 q + λ k + 1 q ∥ x ^ k + 1 − y k ∥ p q ) + 2 a k + 1 δ k + 1 R . \begin{aligned}
a_{k+1}\langle\hat{g}_{k+1},v_k-x^*\rangle\leq{}&2D_\Phi(x^*,v_k)-2\mathbb{E}[D_\Phi(x^*,v_{k+1})\mid\mathcal{F}_k]\\
&+\frac{2^{2-q}a_{k+1}^q}{q}\left(\sigma_{k+1}^q+\|\hat{g}_{k+1}\|_q^q\right)+2a_{k+1}\delta_{k+1}R\\
={}&2D_\Phi(x^*,v_k)-2\mathbb{E}[D_\Phi(x^*,v_{k+1})\mid\mathcal{F}_k]\\
&+\frac{2^{2-q}a_{k+1}^q}{q}\left(\sigma_{k+1}^q+\lambda_{k+1}^q\|\hat{x}_{k+1}-y_k\|_p^q\right)+2a_{k+1}\delta_{k+1}R.
\end{aligned} a k + 1 ⟨ g ^ k + 1 , v k − x ∗ ⟩ ≤ = 2 D Φ ( x ∗ , v k ) − 2 E [ D Φ ( x ∗ , v k + 1 ) ∣ F k ] + q 2 2 − q a k + 1 q ( σ k + 1 q + ∥ g ^ k + 1 ∥ q q ) + 2 a k + 1 δ k + 1 R 2 D Φ ( x ∗ , v k ) − 2 E [ D Φ ( x ∗ , v k + 1 ) ∣ F k ] + q 2 2 − q a k + 1 q ( σ k + 1 q + λ k + 1 q ∥ x ^ k + 1 − y k ∥ p q ) + 2 a k + 1 δ k + 1 R .
This implies that
A k + 1 ( E [ f ( x k + 1 ) ∣ F k ] − f ( x ∗ ) ) − A k ( f ( x k ) − f ( x ∗ ) ) + 1 q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p − A k + 1 φ k + 1 ≤ 2 D Φ ( x ∗ , v k ) − 2 E [ D Φ ( x ∗ , v k + 1 ) ∣ F k ] + 2 2 − q a k + 1 q q ( σ k + 1 q + λ k + 1 q ∥ x ^ k + 1 − y k ∥ p q ) + 2 a k + 1 δ k + 1 R . \begin{aligned}
A_{k+1}(\mathbb{E}[f(x_{k+1})\mid\mathcal{F}_k]-f(x^*))-A_k(f(x_k)-f(x^*))+\frac{1}{q}\lambda_{k+1}A_{k+1}\|\hat{x}_{k+1}-y_k\|_p^p-A_{k+1}\varphi_{k+1}\\
\leq2D_\Phi(x^*,v_k)-2\mathbb{E}[D_\Phi(x^*,v_{k+1})\mid\mathcal{F}_k]+\frac{2^{2-q}a_{k+1}^q}{q}\left(\sigma_{k+1}^q+\lambda_{k+1}^q\|\hat{x}_{k+1}-y_k\|_p^q\right)+2a_{k+1}\delta_{k+1}R.
\end{aligned} A k + 1 ( E [ f ( x k + 1 ) ∣ F k ] − f ( x ∗ )) − A k ( f ( x k ) − f ( x ∗ )) + q 1 λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p − A k + 1 φ k + 1 ≤ 2 D Φ ( x ∗ , v k ) − 2 E [ D Φ ( x ∗ , v k + 1 ) ∣ F k ] + q 2 2 − q a k + 1 q ( σ k + 1 q + λ k + 1 q ∥ x ^ k + 1 − y k ∥ p q ) + 2 a k + 1 δ k + 1 R .
This implies that
E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k + 1 ) ∣ F k ] ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) − 1 q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p \begin{aligned}
\mathbb{E}\left[A_{k+1}\left(f(x_{k+1})-f(x^*)\right)+2D_{\Phi}(x^*,v_{k+1})\mid\mathcal{F}_k\right] \\
&\le A_k\left(f(x_k)-f(x^*)\right)+2D_{\Phi}(x^*,v_k)-\frac{1}{q}\lambda_{k+1}A_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p
\end{aligned} E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k + 1 ) ∣ F k ] ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) − q 1 λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p
+ A k + 1 φ k + 1 + 2 2 − q a k + 1 q q ( σ k + 1 q + λ k + 1 q ∥ x ^ k + 1 − y k ∥ p q ) + 2 a k + 1 δ k + 1 R ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) − 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p + A k + 1 φ k + 1 \begin{aligned}
\quad+A_{k+1}\varphi_{k+1}+\frac{2^{2-q}a_{k+1}^q}{q}\left(\sigma_{k+1}^q+\lambda_{k+1}^q\lVert\hat{x}_{k+1}-y_k\rVert_p^q\right)+2a_{k+1}\delta_{k+1}R \\
&\le A_k\left(f(x_k)-f(x^*)\right)+2D_{\Phi}(x^*,v_k)-\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p+A_{k+1}\varphi_{k+1}
\end{aligned} + A k + 1 φ k + 1 + q 2 2 − q a k + 1 q ( σ k + 1 q + λ k + 1 q ∥ x ^ k + 1 − y k ∥ p q ) + 2 a k + 1 δ k + 1 R ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) − q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p + A k + 1 φ k + 1
+ 2 2 − q a k + 1 q q σ k + 1 q + 2 a k + 1 δ k + 1 R , \quad+\frac{2^{2-q}a_{k+1}^q}{q}\sigma_{k+1}^q+2a_{k+1}\delta_{k+1}R, + q 2 2 − q a k + 1 q σ k + 1 q + 2 a k + 1 δ k + 1 R ,
where we used A k + 1 p − 1 = 2 p − 1 λ k + 1 a k + 1 p A_{k+1}^{p-1}=2^{p-1}\lambda_{k+1}a_{k+1}^p A k + 1 p − 1 = 2 p − 1 λ k + 1 a k + 1 p in the last inequality.
Lemma 18. Let K ≤ K max K\le K_{\max} K ≤ K m a x be the iteration in which the algorithm stops and let
ε ˉ ≥ max k < K max { A k + 1 φ k + 1 a k + 1 + 2 2 − q a k + 1 q − 1 σ k + 1 q q + 2 δ k + 1 R } \bar{\varepsilon}\ge\max_{k<K_{\max}}\left\{\frac{A_{k+1}\varphi_{k+1}}{a_{k+1}}+\frac{2^{2-q}a_{k+1}^{q-1}\sigma_{k+1}^q}{q}+2\delta_{k+1}R\right\} ε ˉ ≥ k < K m a x max { a k + 1 A k + 1 φ k + 1 + q 2 2 − q a k + 1 q − 1 σ k + 1 q + 2 δ k + 1 R } We then have that
E [ A K ( f ( x K ) − f ( x ∗ ) − ε ˉ ) + 2 D Φ ( x ∗ , v K ) + 1 − 2 1 − q q ∑ i = 1 K λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ A 0 ( f ( x 0 ) − f ( x ∗ ) ) + 3 p − 2 ( p − 1 ) R p \mathbb{E}\left[A_K\left(f(x_K)-f(x^*)-\bar{\varepsilon}\right)+2D_{\Phi}(x^*,v_K)+\frac{1-2^{1-q}}{q}\sum_{i=1}^{K}\lambda_iA_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p\right]
\le A_0\left(f(x_0)-f(x^*)\right)+3^{p-2}(p-1)R^p E [ A K ( f ( x K ) − f ( x ∗ ) − ε ˉ ) + 2 D Φ ( x ∗ , v K ) + q 1 − 2 1 − q i = 1 ∑ K λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ A 0 ( f ( x 0 ) − f ( x ∗ ) ) + 3 p − 2 ( p − 1 ) R p
Proof. Let
T k : = − 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p + A k + 1 φ k + 1 + 2 2 − q a k + 1 q q σ k + 1 q + 2 a k + 1 δ k + 1 R . T_k:=-\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p+A_{k+1}\varphi_{k+1}+\frac{2^{2-q}a_{k+1}^q}{q}\sigma_{k+1}^q+2a_{k+1}\delta_{k+1}R. T k := − q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p + A k + 1 φ k + 1 + q 2 2 − q a k + 1 q σ k + 1 q + 2 a k + 1 δ k + 1 R .
First, we have from Lemma 17 that
E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k + 1 ) ] ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) + T k . \mathbb{E}\left[A_{k+1}\left(f(x_{k+1})-f(x^*)\right)+2D_{\Phi}(x^*,v_{k+1})\right]\le A_k\left(f(x_k)-f(x^*)\right)+2D_{\Phi}(x^*,v_k)+T_k. E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k + 1 ) ] ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) + T k .
Define
M k = A k ( f ( x k ) − f ( x ∗ ) − ε ˉ ) + 2 D Φ ( x ∗ , v k ) + 1 − 2 1 − q q ∑ i = 1 k λ i A i ∥ x ^ i − y i − 1 ∥ p p . M_k=A_k\left(f(x_k)-f(x^*)-\bar{\varepsilon}\right)+2D_{\Phi}(x^*,v_k)+\frac{1-2^{1-q}}{q}\sum_{i=1}^{k}\lambda_iA_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p. M k = A k ( f ( x k ) − f ( x ∗ ) − ε ˉ ) + 2 D Φ ( x ∗ , v k ) + q 1 − 2 1 − q i = 1 ∑ k λ i A i ∥ x ^ i − y i − 1 ∥ p p .
We will now prove that M k M_k M k is a supermartingale. It holds that
E [ M k + 1 ∣ F k ] = E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ ) − ε ˉ ) + 2 D Φ ( x ∗ , v k + 1 ) ] + 1 − 2 1 − q q ∑ i = 1 k λ i A i ∥ x ^ i − y i − 1 ∥ p p + 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) + T k − A k + 1 ε ˉ + 1 − 2 1 − q q ∑ i = 1 k λ i A i ∥ x ^ i − y i − 1 ∥ p p + 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) + T k − ( A k + a k + 1 ) ε ˉ + 1 − 2 1 − q q ∑ i = 1 k λ i A i ∥ x ^ i − y i − 1 ∥ p p + 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p ≤ M k + T k + 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p − a k + 1 ε ˉ = M k + A k + 1 φ k + 1 + 2 2 − q a k + 1 q σ k + 1 q q + 2 a k + 1 δ k + 1 R − a k + 1 ε ˉ . \begin{aligned}
\mathbb{E}[M_{k+1}\mid\mathcal{F}_k]
&= \mathbb{E}\left[A_{k+1}(f(x_{k+1})-f(x^*)-\bar{\varepsilon})+2D_{\Phi}(x^*,v_{k+1})\right]
+\frac{1-2^{1-q}}{q}\sum_{i=1}^{k}\lambda_i A_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p \\
&\quad+\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p \\
&\le A_k(f(x_k)-f(x^*))+2D_{\Phi}(x^*,v_k)+T_k-A_{k+1}\bar{\varepsilon}
+\frac{1-2^{1-q}}{q}\sum_{i=1}^{k}\lambda_i A_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p \\
&\quad+\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p \\
&\le A_k(f(x_k)-f(x^*))+2D_{\Phi}(x^*,v_k)+T_k-(A_k+a_{k+1})\bar{\varepsilon}
+\frac{1-2^{1-q}}{q}\sum_{i=1}^{k}\lambda_i A_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p \\
&\quad+\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p \\
&\le M_k+T_k+\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\lVert\hat{x}_{k+1}-y_k\rVert_p^p-a_{k+1}\bar{\varepsilon} \\
&=M_k+A_{k+1}\varphi_{k+1}+\frac{2^{2-q}a_{k+1}^q\sigma_{k+1}^q}{q}+2a_{k+1}\delta_{k+1}R-a_{k+1}\bar{\varepsilon}.
\end{aligned} E [ M k + 1 ∣ F k ] = E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ ) − ε ˉ ) + 2 D Φ ( x ∗ , v k + 1 ) ] + q 1 − 2 1 − q i = 1 ∑ k λ i A i ∥ x ^ i − y i − 1 ∥ p p + q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p ≤ A k ( f ( x k ) − f ( x ∗ )) + 2 D Φ ( x ∗ , v k ) + T k − A k + 1 ε ˉ + q 1 − 2 1 − q i = 1 ∑ k λ i A i ∥ x ^ i − y i − 1 ∥ p p + q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p ≤ A k ( f ( x k ) − f ( x ∗ )) + 2 D Φ ( x ∗ , v k ) + T k − ( A k + a k + 1 ) ε ˉ + q 1 − 2 1 − q i = 1 ∑ k λ i A i ∥ x ^ i − y i − 1 ∥ p p + q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p ≤ M k + T k + q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p − a k + 1 ε ˉ = M k + A k + 1 φ k + 1 + q 2 2 − q a k + 1 q σ k + 1 q + 2 a k + 1 δ k + 1 R − a k + 1 ε ˉ .
By our setting of ε ˉ \bar{\varepsilon} ε ˉ , we have that
M k + A k + 1 φ k + 1 + 2 2 − q a k + 1 q q σ k + 1 q + 2 a k + 1 δ k + 1 R − a k + 1 ε ˉ ≤ M k M_k+A_{k+1}\varphi_{k+1}+\frac{2^{2-q}a_{k+1}^q}{q}\sigma_{k+1}^q+2a_{k+1}\delta_{k+1}R-a_{k+1}\bar{\varepsilon}\le M_k M k + A k + 1 φ k + 1 + q 2 2 − q a k + 1 q σ k + 1 q + 2 a k + 1 δ k + 1 R − a k + 1 ε ˉ ≤ M k
so M k M_k M k is a supermartingale. Therefore, by the optional stopping theorem, it holds that
E [ M K ] ≤ M 0 = A 0 ( f ( x 0 ) − f ( x ∗ ) − ε ˉ ) + 2 D Φ ( x ∗ , v 0 ) . \mathbb{E}[M_K]\le M_0=A_0(f(x_0)-f(x^*)-\bar{\varepsilon})+2D_{\Phi}(x^*,v_0). E [ M K ] ≤ M 0 = A 0 ( f ( x 0 ) − f ( x ∗ ) − ε ˉ ) + 2 D Φ ( x ∗ , v 0 ) .
Using that v 0 = x 0 v_0=x_0 v 0 = x 0 , we have that
D Φ ( x ∗ , v 0 ) = D Φ ( x ∗ , x 0 ) ≤ p − 1 2 ( ∥ x ∗ ∥ p + 2 ∥ x 0 ∥ p ) p − 2 ∥ x ∗ − x 0 ∥ 2 2 ≤ p − 1 2 ( 3 ∥ x 0 ∥ p + ∥ x ∗ − x 0 ∥ p ) p − 2 R 2 ≤ 3 p − 2 ( p − 1 ) 2 R p \begin{aligned}
D_{\Phi}(x^*,v_0)=D_{\Phi}(x^*,x_0)
&\le\frac{p-1}{2}(\lVert x^*\rVert_p+2\lVert x_0\rVert_p)^{p-2}\lVert x^*-x_0\rVert_2^2 \\
&\le\frac{p-1}{2}(3\lVert x_0\rVert_p+\lVert x^*-x_0\rVert_p)^{p-2}R^2 \\
&\le\frac{3^{p-2}(p-1)}{2}R^p
\end{aligned} D Φ ( x ∗ , v 0 ) = D Φ ( x ∗ , x 0 ) ≤ 2 p − 1 (∥ x ∗ ∥ p + 2 ∥ x 0 ∥ p ) p − 2 ∥ x ∗ − x 0 ∥ 2 2 ≤ 2 p − 1 ( 3 ∥ x 0 ∥ p + ∥ x ∗ − x 0 ∥ p ) p − 2 R 2 ≤ 2 3 p − 2 ( p − 1 ) R p
which finishes the proof. □
Fact 1. (Lemma 32 of [CJJ+20]) Let { B k } k ∈ N \{B_k\}_{k\in\mathbb{N}} { B k } k ∈ N be a nonnegative, nondecreasing sequence such that B k ≥ ∑ i ∈ [ k ] α B i B_k\ge\sum_{i\in[k]}\alpha B_i B k ≥ ∑ i ∈ [ k ] α B i for some α ∈ [ 0 , 1 ) \alpha\in[0,1) α ∈ [ 0 , 1 ) and all k k k . Then for all k k k , B k ≥ exp ( α ( k − 1 ) ) B 1 B_k\ge\exp(\alpha(k-1))B_1 B k ≥ exp ( α ( k − 1 )) B 1 .
Lemma 19. Fix any k > 0 k>0 k > 0 . Define
T λ : = ∑ i ≤ k 1 { λ i < 2 λ min } , T r : = ∑ i ≤ k 1 { ∥ x ^ i − y i − 1 ∥ ≥ 3 r / 4 } T_{\lambda}:=\sum_{i\le k}\mathbf{1}_{\{\lambda_i<2\lambda_{\min}\}},\qquad
T_r:=\sum_{i\le k}\mathbf{1}_{\{\lVert\hat{x}_i-y_{i-1}\rVert\ge3r/4\}} T λ := i ≤ k ∑ 1 { λ i < 2 λ m i n } , T r := i ≤ k ∑ 1 {∥ x ^ i − y i − 1 ∥ ≥ 3 r /4 }
Then, with probability one, for ε < 2 q − 1 − 1 3 ⋅ 2 q − 1 − 1 \varepsilon< \frac{2^{q-1}-1}{3\cdot2^{q-1}-1} ε < 3 ⋅ 2 q − 1 − 1 2 q − 1 − 1 ,
1 A k ( 3 C p R p − 1 − 2 1 − q q ∑ i ≤ k λ i A i ∥ x ^ i − y i − 1 ∥ p p ) ≤ O ( min { λ min R p T λ p , R p A 0 exp ( − Ω ( 1 ) r p / ( p + 1 ) R p / ( p + 1 ) T r ) } ) . \frac{1}{A_k}\left(3C_pR^p-\frac{1-2^{1-q}}{q}\sum_{i\le k}\lambda_iA_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p\right)\le O\left(\min\left\{\frac{\lambda_{\min}R^p}{T_\lambda^p},\frac{R^p}{A_0}\exp\left(-\Omega(1)\frac{r^{p/(p+1)}}{R^{p/(p+1)}}T_r\right)\right\}\right). A k 1 ( 3 C p R p − q 1 − 2 1 − q i ≤ k ∑ λ i A i ∥ x ^ i − y i − 1 ∥ p p ) ≤ O ( min { T λ p λ m i n R p , A 0 R p exp ( − Ω ( 1 ) R p / ( p + 1 ) r p / ( p + 1 ) T r ) } ) .
See (40) for a definition of C p C_p C p .
Proof. If 3 C p R p − 1 − 2 1 − q q ∑ i ≤ k λ i A i ∥ x ^ i − y i − 1 ∥ p p ≤ 0 3C_pR^p-\frac{1-2^{1-q}}{q}\sum_{i\le k}\lambda_iA_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p\le0 3 C p R p − q 1 − 2 1 − q ∑ i ≤ k λ i A i ∥ x ^ i − y i − 1 ∥ p p ≤ 0 , then the desired inequality trivially holds. Therefore, we only consider the case where 1 − 2 1 − q q ∑ i ≤ k λ i A i ∥ x ^ i − y i − 1 ∥ p p ≤ 3 C p R p \frac{1-2^{1-q}}{q}\sum_{i\le k}\lambda_iA_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p\le3C_pR^p q 1 − 2 1 − q ∑ i ≤ k λ i A i ∥ x ^ i − y i − 1 ∥ p p ≤ 3 C p R p . We first prove that
A t 1 / p ≥ 1 2 p ∑ i = 1 t λ i − 1 / p . (35) A_t^{1/p}\ge\frac{1}{2p}\sum_{i=1}^{t}\lambda_i^{-1/p}. \tag*{(35)} A t 1/ p ≥ 2 p 1 i = 1 ∑ t λ i − 1/ p . (35)
Since A i p − 1 = 2 p − 1 λ i a i p A_i^{p-1}=2^{p-1}\lambda_i a_i^p A i p − 1 = 2 p − 1 λ i a i p , it holds that a i = A i ( p − 1 ) / p 2 ( p − 1 ) / p λ i 1 / p a_i=\frac{A_i^{(p-1)/p}}{2^{(p-1)/p}\lambda_i^{1/p}} a i = 2 ( p − 1 ) / p λ i 1/ p A i ( p − 1 ) / p . Therefore,
a i ≥ 1 2 ( p − 1 ) / p A i ( p − 1 ) / p λ i − 1 / p ≥ 1 2 A i ( p − 1 ) / p λ i − 1 / p . a_i\ge\frac{1}{2^{(p-1)/p}}A_i^{(p-1)/p}\lambda_i^{-1/p}\ge\frac{1}{2}A_i^{(p-1)/p}\lambda_i^{-1/p}. a i ≥ 2 ( p − 1 ) / p 1 A i ( p − 1 ) / p λ i − 1/ p ≥ 2 1 A i ( p − 1 ) / p λ i − 1/ p .
Since f ( x ) = x 1 / p f(x)=x^{1/p} f ( x ) = x 1/ p is concave for p ≥ 2 p\ge2 p ≥ 2 , it holds that
A i 1 / p − A i − 1 1 / p ≥ 1 p A i 1 / p − 1 ( A i − A i − 1 ) = a i p A i ( p − 1 ) / p ≥ λ i − 1 / p 2 p . (36) A_i^{1/p}-A_{i-1}^{1/p}\ge\frac{1}{p}A_i^{1/p-1}(A_i-A_{i-1})=\frac{a_i}{pA_i^{(p-1)/p}}\ge\frac{\lambda_i^{-1/p}}{2p}. \tag*{(36)} A i 1/ p − A i − 1 1/ p ≥ p 1 A i 1/ p − 1 ( A i − A i − 1 ) = p A i ( p − 1 ) / p a i ≥ 2 p λ i − 1/ p . (36)
Telescoping (36) from i = 1 i=1 i = 1 to t t t and using that A 0 ≥ 0 A_0\ge0 A 0 ≥ 0 yields (35) . For t ≤ k t\le k t ≤ k , define
I r , t = { i ≤ t : ∥ x ^ i − y i − 1 ∥ p ≥ 3 r 4 } . I_{r,t}=\left\{i\le t:\lVert\hat{x}_i-y_{i-1}\rVert_p\ge\frac{3r}{4}\right\}. I r , t = { i ≤ t : ∥ x ^ i − y i − 1 ∥ p ≥ 4 3 r } .
It holds that
1 − 2 1 − q q ∑ i ∈ I r , t λ i A i ( 3 r 4 ) p ≤ 1 − 2 1 − q q ∑ i ≤ t λ i A i ∥ x ^ i − y i − 1 ∥ p p ≤ 3 C p R p ⟹ ∑ i ∈ I r , t λ i A i ≤ 3 C p ( R r ) p ( 4 3 ) p ⋅ q 1 − 2 1 − q . \frac{1-2^{1-q}}{q}\sum_{i\in I_{r,t}}\lambda_iA_i\left(\frac{3r}{4}\right)^p\le\frac{1-2^{1-q}}{q}\sum_{i\le t}\lambda_iA_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p\le3C_pR^p\Longrightarrow\sum_{i\in I_{r,t}}\lambda_iA_i\le3C_p\left(\frac{R}{r}\right)^p\left(\frac{4}{3}\right)^p\cdot\frac{q}{1-2^{1-q}}. q 1 − 2 1 − q i ∈ I r , t ∑ λ i A i ( 4 3 r ) p ≤ q 1 − 2 1 − q i ≤ t ∑ λ i A i ∥ x ^ i − y i − 1 ∥ p p ≤ 3 C p R p ⟹ i ∈ I r , t ∑ λ i A i ≤ 3 C p ( r R ) p ( 3 4 ) p ⋅ 1 − 2 1 − q q .
Now, by a reverse Hölder inequality, it holds that for all θ ∈ ( 0 , 1 ) \theta\in(0,1) θ ∈ ( 0 , 1 ) ,
∑ i u i v i ≥ ( ∑ i u i θ ) 1 / θ ⋅ ( ∑ i v i − θ / ( 1 − θ ) ) − ( 1 − θ ) / θ . \sum_i u_iv_i\ge\left(\sum_i u_i^\theta\right)^{1/\theta}\cdot\left(\sum_i v_i^{-\theta/(1-\theta)}\right)^{-(1-\theta)/\theta}. i ∑ u i v i ≥ ( i ∑ u i θ ) 1/ θ ⋅ ( i ∑ v i − θ / ( 1 − θ ) ) − ( 1 − θ ) / θ .
Taking u i = A i 1 / p u_i=A_i^{1/p} u i = A i 1/ p , v i = 1 λ i 1 / p A i 1 / p v_i=\frac{1}{\lambda_i^{1/p}A_i^{1/p}} v i = λ i 1/ p A i 1/ p 1 , θ = p / ( p + 1 ) \theta=p/(p+1) θ = p / ( p + 1 ) , it holds that
∑ i ∈ I r , t 1 λ i 1 / p ≥ ( ∑ i ∈ I r , t A i 1 / ( p + 1 ) ) ( p + 1 ) / p ⋅ ( ∑ i ∈ I r , t λ i A i ) − 1 / p ≥ ( 3 C p ⋅ q 1 − 2 1 − q ⋅ ( 4 3 ) p ) − 1 / p ⋅ r R ⋅ ( ∑ i ∈ I r , t A i 1 / ( p + 1 ) ) ( p + 1 ) / p ≥ C p ′ ⋅ r R ⋅ ( ∑ i ∈ I r , t A i 1 / ( p + 1 ) ) ( p + 1 ) / p , \begin{aligned}
\sum_{i\in I_{r,t}}\frac{1}{\lambda_i^{1/p}}\ge\left(\sum_{i\in I_{r,t}}A_i^{1/(p+1)}\right)^{(p+1)/p}\cdot\left(\sum_{i\in I_{r,t}}\lambda_iA_i\right)^{-1/p}\ge\left(3C_p\cdot\frac{q}{1-2^{1-q}}\cdot\left(\frac{4}{3}\right)^p\right)^{-1/p}\cdot\frac{r}{R}\cdot\left(\sum_{i\in I_{r,t}}A_i^{1/(p+1)}\right)^{(p+1)/p} \\
&\ge C'_p\cdot\frac{r}{R}\cdot\left(\sum_{i\in I_{r,t}}A_i^{1/(p+1)}\right)^{(p+1)/p},
\end{aligned} i ∈ I r , t ∑ λ i 1/ p 1 ≥ i ∈ I r , t ∑ A i 1/ ( p + 1 ) ( p + 1 ) / p ⋅ i ∈ I r , t ∑ λ i A i − 1/ p ≥ ( 3 C p ⋅ 1 − 2 1 − q q ⋅ ( 3 4 ) p ) − 1/ p ⋅ R r ⋅ i ∈ I r , t ∑ A i 1/ ( p + 1 ) ( p + 1 ) / p ≥ C p ′ ⋅ R r ⋅ i ∈ I r , t ∑ A i 1/ ( p + 1 ) ( p + 1 ) / p ,
where we let C p ′ = ( 3 C p ⋅ q 1 − 2 1 − q ⋅ ( 4 3 ) p ) − 1 / p C'_p=\left(3C_p\cdot\frac{q}{1-2^{1-q}}\cdot\left(\frac{4}{3}\right)^p\right)^{-1/p} C p ′ = ( 3 C p ⋅ 1 − 2 1 − q q ⋅ ( 3 4 ) p ) − 1/ p . Combining with (35) gives that
A t 1 / ( p + 1 ) ≥ ( C p ′ 2 p ) p / ( p + 1 ) ⋅ ( r R ) p / ( p + 1 ) ∑ i ∈ I r , t A i 1 / ( p + 1 ) . A_t^{1/(p+1)}\ge\left(\frac{C'_p}{2p}\right)^{p/(p+1)}\cdot\left(\frac{r}{R}\right)^{p/(p+1)}\sum_{i\in I_{r,t}}A_i^{1/(p+1)}. A t 1/ ( p + 1 ) ≥ ( 2 p C p ′ ) p / ( p + 1 ) ⋅ ( R r ) p / ( p + 1 ) i ∈ I r , t ∑ A i 1/ ( p + 1 ) .
By Fact 1, it holds that
A k 1 / ( p + 1 ) ≥ A 0 1 / ( p + 1 ) exp ( Ω ( 1 ) ( r R ) p p + 1 T r ) ⟹ 1 A k ≤ 1 A 0 exp ( − Ω ( 1 ) ( r R ) p p + 1 T r ) (37) A_k^{1/(p+1)} \ge A_0^{1/(p+1)}\exp\left(\Omega(1)\left(\frac{r}{R}\right)^{\frac{p}{p+1}}T_r\right)\quad\Longrightarrow\quad\frac{1}{A_k}\le\frac{1}{A_0}\exp\left(-\Omega(1)\left(\frac{r}{R}\right)^{\frac{p}{p+1}}T_r\right) \tag*{(37)} A k 1/ ( p + 1 ) ≥ A 0 1/ ( p + 1 ) exp ( Ω ( 1 ) ( R r ) p + 1 p T r ) ⟹ A k 1 ≤ A 0 1 exp ( − Ω ( 1 ) ( R r ) p + 1 p T r ) (37)
Therefore,
1 A k ( 3 C p R p − 1 − 2 1 − q q ∑ i ≤ k λ i A i ∥ x ^ i − y i − 1 ∥ p p ) ≤ 3 C p R p A k ≤ O ( R p A 0 exp ( − Ω ( 1 ) ( r R ) p p + 1 T r ) ) (38) \frac{1}{A_k}\left(3C_pR^p-\frac{1-2^{1-q}}{q}\sum_{i\le k}\lambda_iA_i\lVert\widehat{x}_i-y_{i-1}\rVert_p^p\right)\le\frac{3C_pR^p}{A_k}\le O\left(\frac{R^p}{A_0}\exp\left(-\Omega(1)\left(\frac{r}{R}\right)^{\frac{p}{p+1}}T_r\right)\right) \tag*{(38)} A k 1 ( 3 C p R p − q 1 − 2 1 − q i ≤ k ∑ λ i A i ∥ x i − y i − 1 ∥ p p ) ≤ A k 3 C p R p ≤ O ( A 0 R p exp ( − Ω ( 1 ) ( R r ) p + 1 p T r ) ) (38)
Let I λ = { i ≤ k : λ i < 2 λ min } I_\lambda=\{i\le k:\lambda_i<2\lambda_{\min}\} I λ = { i ≤ k : λ i < 2 λ m i n } . Combining (35) with λ i < 2 λ min \lambda_i<2\lambda_{\min} λ i < 2 λ m i n yields
A k 1 / p ≥ 1 2 p ⋅ T λ ( 2 λ min ) 1 / p ⟹ 1 A k ≤ O ( λ min T λ ) ⟹ 3 C p R p A k ≤ O ( R p λ min T λ ) (39) A_k^{1/p}\ge\frac{1}{2p}\cdot\frac{T_\lambda}{(2\lambda_{\min})^{1/p}}\quad\Longrightarrow\quad\frac{1}{A_k}\le O\left(\frac{\lambda_{\min}}{T_\lambda}\right)\quad\Longrightarrow\quad\frac{3C_pR^p}{A_k}\le O\left(\frac{R^p\lambda_{\min}}{T_\lambda}\right) \tag*{(39)} A k 1/ p ≥ 2 p 1 ⋅ ( 2 λ m i n ) 1/ p T λ ⟹ A k 1 ≤ O ( T λ λ m i n ) ⟹ A k 3 C p R p ≤ O ( T λ R p λ m i n ) (39)
Combining (38) and (39) finishes the proof.
Proposition 9. Let f : X → R f:\mathcal{X}\to\mathbb{R} f : X → R and ∇ ^ f \widehat{\nabla}f ∇ f satisfy Assumption 1, and let X ⊆ B R ( x 0 ) \mathcal{X}\subseteq B_R(x_0) X ⊆ B R ( x 0 ) . For a target accuracy ε ≤ G R \varepsilon\le GR ε ≤ GR let
φ k = ε 48 C p λ k q − 1 a k q − 1 , δ k = ε 96 C p R , σ k q = q ε 48 C p 2 2 − q a k q − 1 , A 0 = R p − 1 G , A max = 3 C p R p ε . \varphi_k=\frac{\varepsilon}{48C_p\lambda_k^{q-1}a_k^{q-1}},\qquad\delta_k=\frac{\varepsilon}{96C_pR},\qquad\sigma_k^q=\frac{q\varepsilon}{48C_p2^{2-q}a_k^{q-1}},\qquad A_0=\frac{R^{p-1}}{G},\qquad A_{\max}=3C_p\frac{R^p}{\varepsilon}. φ k = 48 C p λ k q − 1 a k q − 1 ε , δ k = 96 C p R ε , σ k q = 48 C p 2 2 − q a k q − 1 q ε , A 0 = G R p − 1 , A m a x = 3 C p ε R p . see (40) . If λ k ≥ λ min ≥ 1 A max = Ω ( ε R p ) \lambda_k\ge\lambda_{\min}\ge\frac{1}{A_{\max}}=\Omega\left(\frac{\varepsilon}{R^p}\right) λ k ≥ λ m i n ≥ A m a x 1 = Ω ( R p ε ) for all k ≤ K max k\le K_{\max} k ≤ K m a x , then lines 7 and 9 of Algorithm 6 have total complexity
E N ∇ ^ f = O ( K max log ( G R ε ) + ( G R ε ) p ( p − 1 ) log ( G R ε ) ) . \mathbb{E}N_{\widehat{\nabla}f}=O\left(K_{\max}\log\left(\frac{GR}{\varepsilon}\right)+\left(\frac{GR}{\varepsilon}\right)^{p(p-1)}\log\left(\frac{GR}{\varepsilon}\right)\right). E N ∇ f = O ( K m a x log ( ε GR ) + ( ε GR ) p ( p − 1 ) log ( ε GR ) ) . If in addition ∥ P f , λ k ( y k − 1 ) − y k − 1 ∥ p ≥ 3 r / 4 \lVert P_{f,\lambda_k}(y_{k-1})-y_{k-1}\rVert_p\ge3r/4 ∥ P f , λ k ( y k − 1 ) − y k − 1 ∥ p ≥ 3 r /4 whenever λ k ≥ 2 λ min \lambda_k\ge2\lambda_{\min} λ k ≥ 2 λ m i n then for
K max = O ( ( R r ) p p + 1 log ( G R ε ) + ( λ min R p ε ) 1 / p ) K_{\max}=O\left(\left(\frac{R}{r}\right)^{\frac{p}{p+1}}\log\left(\frac{GR}{\varepsilon}\right)+\left(\frac{\lambda_{\min}R^p}{\varepsilon}\right)^{1/p}\right) K m a x = O ( ( r R ) p + 1 p log ( ε GR ) + ( ε λ m i n R p ) 1/ p ) the algorithm’s output x K x_K x K satisfies f ( x K ) − f ( x ⋆ ) ≤ ε f(x_K)-f(x^\star)\le\varepsilon f ( x K ) − f ( x ⋆ ) ≤ ε with probability at least 2 / 3 2/3 2/3 .
Proof. First, let us prove correctness of the algorithm. The settings of φ k \varphi_k φ k , δ k \delta_k δ k and σ k \sigma_k σ k in the proposition guarantee that
max k < K max { A k + 1 φ k + 1 a k + 1 + 2 2 − q a k + 1 q − 1 σ k + 1 q q + 2 δ k + 1 R } = ε 12 C p . \max_{k<K_{\max}}\left\{\frac{A_{k+1}\varphi_{k+1}}{a_{k+1}}+\frac{2^{2-q}a_{k+1}^{q-1}\sigma_{k+1}^q}{q}+2\delta_{k+1}R\right\}=\frac{\varepsilon}{12C_p}. k < K m a x max { a k + 1 A k + 1 φ k + 1 + q 2 2 − q a k + 1 q − 1 σ k + 1 q + 2 δ k + 1 R } = 12 C p ε . Therefore, applying Lemma 18 with ε ˉ = ε / ( 12 C p ) ≤ ε / ( 6 C p ) ≤ R p / ( 2 A max ) \bar{\varepsilon}=\varepsilon/(12C_p)\le\varepsilon/(6C_p)\le R^p/(2A_{\max}) ε ˉ = ε / ( 12 C p ) ≤ ε / ( 6 C p ) ≤ R p / ( 2 A m a x ) yields
E [ A K ( f ( x K ) − f ( x ⋆ ) ) + 1 − 2 1 − q q ∑ i ≤ K λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ 3 p − 2 R p ( p − 1 ) + ε ˉ ⋅ E A K + A 0 ( f ( x 0 ) − f ( x ⋆ ) ) . \mathbb{E}\left[A_K(f(x_K)-f(x^\star))+\frac{1-2^{1-q}}{q}\sum_{i\le K}\lambda_iA_i\lVert\widehat{x}_i-y_{i-1}\rVert_p^p\right]\le3^{p-2}R^p(p-1)+\bar{\varepsilon}\cdot\mathbb{E}A_K+A_0(f(x_0)-f(x^\star)). E [ A K ( f ( x K ) − f ( x ⋆ )) + q 1 − 2 1 − q i ≤ K ∑ λ i A i ∥ x i − y i − 1 ∥ p p ] ≤ 3 p − 2 R p ( p − 1 ) + ε ˉ ⋅ E A K + A 0 ( f ( x 0 ) − f ( x ⋆ )) . We now upper bound a K a_K a K in terms of A K − 1 A_{K-1} A K − 1 and λ K \lambda_K λ K . It holds that ( A K − 1 + a K ) p − 1 = 2 p − 1 λ K a K p (A_{K-1}+a_K)^{p-1}=2^{p-1}\lambda_Ka_K^p ( A K − 1 + a K ) p − 1 = 2 p − 1 λ K a K p . If a K > A K − 1 a_K>A_{K-1} a K > A K − 1 , then A K − 1 + a K < 2 a K A_{K-1}+a_K<2a_K A K − 1 + a K < 2 a K , meaning that a K < 1 / λ K a_K<1/\lambda_K a K < 1/ λ K because 2 p − 1 λ K a K p = ( A K − 1 + a K ) p − 1 < ( 2 a K ) p − 1 2^{p-1}\lambda_Ka_K^p=(A_{K-1}+a_K)^{p-1}<(2a_K)^{p-1} 2 p − 1 λ K a K p = ( A K − 1 + a K ) p − 1 < ( 2 a K ) p − 1 . Combining this with the a K < A K − 1 a_K<A_{K-1} a K < A K − 1 case, it holds that
a K ≤ max { A K − 1 , 1 λ K } ⟹ a K ≤ A K − 1 + 1 λ K ⟹ A K ≤ 2 A K − 1 + 1 λ K a_K\le\max\left\{A_{K-1},\frac{1}{\lambda_K}\right\}\Longrightarrow a_K\le A_{K-1}+\frac{1}{\lambda_K}\Longrightarrow A_K\le2A_{K-1}+\frac{1}{\lambda_K} a K ≤ max { A K − 1 , λ K 1 } ⟹ a K ≤ A K − 1 + λ K 1 ⟹ A K ≤ 2 A K − 1 + λ K 1
Since A K − 1 ≤ A max A_{K-1} \le A_{\max} A K − 1 ≤ A m a x and λ K ≥ 1 / A max \lambda_K \ge1/A_{\max} λ K ≥ 1/ A m a x , it holds that A K ≤ 3 A max A_K \le3A_{\max} A K ≤ 3 A m a x with probability 1. Since A 0 = R p − 1 / G A_0 = R^{p-1}/G A 0 = R p − 1 / G and f f f is G G G -Lipschitz, it holds that A 0 ( f ( x 0 ) − f ( x ∗ ) ) ≤ A 0 G ⋅ ∥ x 0 − x ∗ ∥ p ≤ A 0 G R = R p A_0(f(x_0)-f(x^*)) \le A_0G\cdot\lVert x_0-x^*\rVert_p \le A_0GR = R^p A 0 ( f ( x 0 ) − f ( x ∗ )) ≤ A 0 G ⋅ ∥ x 0 − x ∗ ∥ p ≤ A 0 GR = R p . Moreover, using the definition of A max A_{\max} A m a x , we have that ε A K ≤ 3 12 C p ⋅ 3 ( C p ε p ) = 3 4 R p \varepsilon A_K \le\frac{3}{12C_p}\cdot3(C_p\frac{\varepsilon}{p}) = \frac{3}{4}R^p ε A K ≤ 12 C p 3 ⋅ 3 ( C p p ε ) = 4 3 R p . Thus,
E [ A K ( f ( x K ) − f ( x ∗ ) ) + 1 − 2 1 − q q ∑ i ≤ K λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ A 0 ( f ( x 0 ) − f ( x ∗ ) ) + 3 p − 2 R p ( p − 1 ) + ε E [ A K ] ≤ [ 3 p − 2 ( p − 1 ) + 1 + 3 4 ] R p \begin{aligned}
\mathbb{E}\left[A_K(f(x_K)-f(x^*))+\frac{1-2^{1-q}}{q}\sum_{i\le K}\lambda_iA_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p\right]\le A_0(f(x_0)-f(x^*))+3^{p-2}R^p(p-1)+\varepsilon\mathbb{E}[A_K] \\
&\le[3^{p-2}(p-1)+1+\frac{3}{4}]R^p
\end{aligned} E [ A K ( f ( x K ) − f ( x ∗ )) + q 1 − 2 1 − q i ≤ K ∑ λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ A 0 ( f ( x 0 ) − f ( x ∗ )) + 3 p − 2 R p ( p − 1 ) + ε E [ A K ] ≤ [ 3 p − 2 ( p − 1 ) + 1 + 4 3 ] R p
Denote
C p = 3 p − 2 ( p − 1 ) + 7 4 (40) C_p=3^{p-2}(p-1)+\frac{7}{4} \tag*{(40)} C p = 3 p − 2 ( p − 1 ) + 4 7 (40)
Therefore, with probability at least 2 / 3 2/3 2/3 ,
E [ A K ( f ( x K ) − f ( x ∗ ) ) + 1 − 2 1 − q q ∑ i ≤ K λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ 3 C p R p ⟹ f ( x K ) − f ( x ∗ ) ≤ 3 C p R p A K \mathbb{E}\left[A_K(f(x_K)-f(x^*))+\frac{1-2^{1-q}}{q}\sum_{i\le K}\lambda_iA_i\lVert\hat{x}_i-y_{i-1}\rVert_p^p\right]\le3C_pR^p\ \Longrightarrow\ f(x_K)-f(x^*)\le\frac{3C_pR^p}{A_K} E [ A K ( f ( x K ) − f ( x ∗ )) + q 1 − 2 1 − q i ≤ K ∑ λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ 3 C p R p ⟹ f ( x K ) − f ( x ∗ ) ≤ A K 3 C p R p
If A K ≥ A max A_K\ge A_{\max} A K ≥ A m a x , then we are done. Otherwise, K = K max K=K_{\max} K = K m a x and by the assumption on NextLambda, it holds that T λ + T r ≥ K max T_\lambda+T_r\ge K_{\max} T λ + T r ≥ K m a x , which means that either T λ ≥ K max / 2 T_\lambda\ge K_{\max}/2 T λ ≥ K m a x /2 or T r ≥ K max / 2 T_r\ge K_{\max}/2 T r ≥ K m a x /2 . If T λ ≥ K max / 2 T_\lambda\ge K_{\max}/2 T λ ≥ K m a x /2 , using (35) for the T λ T_\lambda T λ iterations where λ i < 2 λ min \lambda_i<2\lambda_{\min} λ i < 2 λ m i n , we get that
A K 1 / p ≥ T λ 2 p ( 2 λ min ) 1 / p ⟹ 3 C p R p A K ≤ O ( λ min R p T λ p ) ≤ ε A_K^{1/p}\ge\frac{T_\lambda}{2p(2\lambda_{\min})^{1/p}}\ \Longrightarrow\ \frac{3C_pR^p}{A_K}\le O\left(\frac{\lambda_{\min}R^p}{T_\lambda^p}\right)\le\varepsilon A K 1/ p ≥ 2 p ( 2 λ m i n ) 1/ p T λ ⟹ A K 3 C p R p ≤ O ( T λ p λ m i n R p ) ≤ ε
as long as T λ ≥ Ω ( ( λ min R p ε ) 1 / p ) T_\lambda\ge\Omega((\frac{\lambda_{\min}R^p}{\varepsilon})^{1/p}) T λ ≥ Ω (( ε λ m i n R p ) 1/ p ) . This implies that it suffices to let K max ≥ Ω ( ( λ min R p ε ) 1 / p ) K_{\max}\ge\Omega((\frac{\lambda_{\min}R^p}{\varepsilon})^{1/p}) K m a x ≥ Ω (( ε λ m i n R p ) 1/ p ) . If T r ≥ K max / 2 T_r\ge K_{\max}/2 T r ≥ K m a x /2 , then we can use that A K ≥ A 0 exp ( Ω ( 1 ) ( r R ) p / ( p + 1 ) T r ) A_K\ge A_0\exp(\Omega(1)(\frac{r}{R})^{p/(p+1)}T_r) A K ≥ A 0 exp ( Ω ( 1 ) ( R r ) p / ( p + 1 ) T r ) (from (37) ) and that R p / A 0 = G R R^p/A_0=GR R p / A 0 = GR to say that
3 C p R p A K ≤ 3 C p G R exp ( − Ω ( 1 ) ( r R ) p / ( p + 1 ) T r ) \frac{3C_pR^p}{A_K}\le3C_pGR\exp(-\Omega(1)(\frac{r}{R})^{p/(p+1)}T_r) A K 3 C p R p ≤ 3 C p GR exp ( − Ω ( 1 ) ( R r ) p / ( p + 1 ) T r )
which is at most ε \varepsilon ε provided that T r = Ω ( ( R r ) p / ( p + 1 ) log ( G R ε ) ) T_r=\Omega((\frac{R}{r})^{p/(p+1)}\log(\frac{GR}{\varepsilon})) T r = Ω (( r R ) p / ( p + 1 ) log ( ε GR )) . Therefore, taking
K max = O ( ( R r ) p / ( p + 1 ) log ( G R ε ) + ( λ min R p ε ) 1 / p ) K_{\max}=O\left(\left(\frac{R}{r}\right)^{p/(p+1)}\log\left(\frac{GR}{\varepsilon}\right)+\left(\frac{\lambda_{\min}R^p}{\varepsilon}\right)^{1/p}\right) K m a x = O ( ( r R ) p / ( p + 1 ) log ( ε GR ) + ( ε λ m i n R p ) 1/ p )
suffices to conclude that f ( x K ) − f ( x ∗ ) ≤ ε f(x_K)-f(x^*)\le\varepsilon f ( x K ) − f ( x ∗ ) ≤ ε with probability at least 2 / 3 2/3 2/3 . Now, we prove the complexity guarantee. Each step of computing x k + 1 x_{k+1} x k + 1 on line 7 of Algorithm 6 requires O ( G q λ k ϕ k p − 1 ) O(\frac{G^q}{\lambda_k\phi_k^{p-1}}) O ( λ k ϕ k p − 1 G q ) queries by Definition 3 and the definition of an approximate proximal mapping. Moreover, by Theorem 1 it requires
O ( G p σ p ( G min { δ , σ } ) p ( p − 2 ) ( 1 + log G min { δ , σ } ) ) O\left(\frac{G^p}{\sigma^p}\left(\frac{G}{\min\{\delta,\sigma\}}\right)^{p(p-2)}\left(1+\log\frac{G}{\min\{\delta,\sigma\}}\right)\right) O ( σ p G p ( min { δ , σ } G ) p ( p − 2 ) ( 1 + log min { δ , σ } G ) )
in expectation to compute g k + 1 g_{k+1} g k + 1 in line 8 of Algorithm 6. Observe that
σ k p = ( σ k q ) p − 1 = O ( ε p − 1 a k ( q − 1 ) ( p − 1 ) ) = O ( ε p − 1 a k ) ⟹ G p σ k p = O ( G p a k ε p − 1 ) . \sigma_k^p=(\sigma_k^q)^{p-1}=O\left(\frac{\varepsilon^{p-1}}{a_k^{(q-1)(p-1)}}\right)=O\left(\frac{\varepsilon^{p-1}}{a_k}\right)\ \Longrightarrow\ \frac{G^p}{\sigma_k^p}=O\left(\frac{G^pa_k}{\varepsilon^{p-1}}\right). σ k p = ( σ k q ) p − 1 = O ( a k ( q − 1 ) ( p − 1 ) ε p − 1 ) = O ( a k ε p − 1 ) ⟹ σ k p G p = O ( ε p − 1 G p a k ) .
Therefore, plugging in ϕ k , δ k \phi_k,\delta_k ϕ k , δ k and σ k \sigma_k σ k , we have
E N ∇ f ~ = ∑ k ∈ [ K ] O ( G p λ k ϕ k p − 1 ) + ∑ k ∈ [ K ] O ( [ 1 + G p σ k p ( G min { δ , σ k } ) p ] p ( p − 2 ) ) ⋅ log ( G min { δ , σ k } ) ≤ O ( G p ε p − 1 ∑ k ∈ [ K ] a k ) + O ( ∑ k ∈ [ K ] log G min { δ , σ k } + G p σ k p ( G min { δ , σ k } ) p ( p − 2 ) log G min { δ , σ k } ) ≤ O ( G p ε p − 1 ∑ k ∈ [ K ] a k ) + O ( ∑ k ∈ [ K ] log G R ε + G p a k ε p − 1 ( G R ε ) p ( p − 2 ) log G R ε ) ≤ O ( G p ε p − 1 A max ) + O ( K max log G R ε ) + O ( G p A max ε p − 1 ( G R ε ) p ( p − 2 ) log G R ε ) ≤ O ( ( G R ε ) p ) + O ( K max log G R ε ) + O ( ( G R ε ) p ( p − 1 ) log G R ε ) \begin{aligned}
\mathbb{E}N_{\nabla\tilde f} = \sum_{k\in[K]} O\left(\frac{G^p}{\lambda_k\phi_k^{p-1}}\right) + \sum_{k\in[K]} O\left(\left[1+\frac{G^p}{\sigma_k^p}\left(\frac{G}{\min\{\delta,\sigma_k\}}\right)^p\right]^{p(p-2)}\right)\cdot\log\left(\frac{G}{\min\{\delta,\sigma_k\}}\right) \\
&\le O\left(\frac{G^p}{\varepsilon^{p-1}}\sum_{k\in[K]}a_k\right)+O\left(\sum_{k\in[K]}\log\frac{G}{\min\{\delta,\sigma_k\}}+\frac{G^p}{\sigma_k^p}\left(\frac{G}{\min\{\delta,\sigma_k\}}\right)^{p(p-2)}\log\frac{G}{\min\{\delta,\sigma_k\}}\right) \\
&\le O\left(\frac{G^p}{\varepsilon^{p-1}}\sum_{k\in[K]}a_k\right)+O\left(\sum_{k\in[K]}\log\frac{GR}{\varepsilon}+\frac{G^pa_k}{\varepsilon^{p-1}}\left(\frac{GR}{\varepsilon}\right)^{p(p-2)}\log\frac{GR}{\varepsilon}\right) \\
&\le O\left(\frac{G^p}{\varepsilon^{p-1}}A_{\max}\right)+O\left(K_{\max}\log\frac{GR}{\varepsilon}\right)+O\left(\frac{G^pA_{\max}}{\varepsilon^{p-1}}\left(\frac{GR}{\varepsilon}\right)^{p(p-2)}\log\frac{GR}{\varepsilon}\right) \\
&\le O\left(\left(\frac{GR}{\varepsilon}\right)^p\right)+O\left(K_{\max}\log\frac{GR}{\varepsilon}\right)+O\left(\left(\frac{GR}{\varepsilon}\right)^{p(p-1)}\log\frac{GR}{\varepsilon}\right)
\end{aligned} E N ∇ f ~ = k ∈ [ K ] ∑ O ( λ k ϕ k p − 1 G p ) + k ∈ [ K ] ∑ O ( [ 1 + σ k p G p ( min { δ , σ k } G ) p ] p ( p − 2 ) ) ⋅ log ( min { δ , σ k } G ) ≤ O ε p − 1 G p k ∈ [ K ] ∑ a k + O k ∈ [ K ] ∑ log min { δ , σ k } G + σ k p G p ( min { δ , σ k } G ) p ( p − 2 ) log min { δ , σ k } G ≤ O ε p − 1 G p k ∈ [ K ] ∑ a k + O k ∈ [ K ] ∑ log ε GR + ε p − 1 G p a k ( ε GR ) p ( p − 2 ) log ε GR ≤ O ( ε p − 1 G p A m a x ) + O ( K m a x log ε GR ) + O ( ε p − 1 G p A m a x ( ε GR ) p ( p − 2 ) log ε GR ) ≤ O ( ( ε GR ) p ) + O ( K m a x log ε GR ) + O ( ( ε GR ) p ( p − 1 ) log ε GR )
since a k ≤ A K ≤ 3 A max ≤ O ( R p / ε ) a_k\le A_K\le3A_{\max}\le O(R^p/\varepsilon) a k ≤ A K ≤ 3 A m a x ≤ O ( R p / ε ) , min { δ k , σ k } = Ω ( ε / R ) \min\{\delta_k,\sigma_k\}=\Omega(\varepsilon/R) min { δ k , σ k } = Ω ( ε / R ) , and ∑ k ∈ [ K ] a k p − 1 ≤ ( ∑ k ∈ [ K ] a k ) p − 1 ≤ O ( A K p − 1 ) \sum_{k\in[K]}a_k^{p-1}\le\left(\sum_{k\in[K]}a_k\right)^{p-1}\le O(A_K^{p-1}) ∑ k ∈ [ K ] a k p − 1 ≤ ( ∑ k ∈ [ K ] a k ) p − 1 ≤ O ( A K p − 1 ) and A k = O ( A max ) A_k=O(A_{\max}) A k = O ( A m a x ) . This finishes the proof.
ℓ p \ell_p ℓ p Parallel Stochastic Convex Optimization: Upper Bounds
Preliminaries
We study the following parallel SCO problem, where access to the objective is only allowed through the stochastic gradient oracle.
Problem 1. Let f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R be convex. Assume that there exists a stochastic gradient oracle g g g such that for all x ∈ R d x\in\mathbb{R}^d x ∈ R d , E [ g ( x ) ] ∈ ∂ f ( x ) \mathbb{E}[g(x)]\in\partial f(x) E [ g ( x )] ∈ ∂ f ( x ) and E ∥ g ( x ) ∥ q q ≤ L q \mathbb{E}\|g(x)\|_q^q\le L^q E ∥ g ( x ) ∥ q q ≤ L q . Our goal is to produce x ∈ R d x\in\mathbb{R}^d x ∈ R d such that
E [ f ( x ) ] ≤ min x ∗ ∈ B ( R ) f ( x ∗ ) + ϵ o p t \mathbb{E}[f(x)]\le\min_{x^*\in B(R)} f(x^*)+\epsilon_{\mathrm{opt}} E [ f ( x )] ≤ x ∗ ∈ B ( R ) min f ( x ∗ ) + ϵ opt for some $\epsilon_{\mathrm{opt}}>0.
We will use the query depth, computational depth, and the total number of stochastic gradient oracle queries as metrics for characterizing the efficiency of our algorithms. Following [15 ] , we present a few useful oracle definitions which will be useful in our analysis.
Definition 6 (Line search oracle). We say O l s \mathcal{O}_{\mathrm{ls}} O ls is a ( Δ , λ ) (\Delta,\lambda) ( Δ , λ ) -line search oracle for F : R d → R F:\mathbb{R}^d\to\mathbb{R} F : R d → R if given x ˉ ∈ R d \bar{x}\in\mathbb{R}^d x ˉ ∈ R d , O l s \mathcal{O}_{\mathrm{ls}} O ls returns x ∈ R d x\in\mathbb{R}^d x ∈ R d with
∥ x − x x ˉ , λ ∗ ∥ p ≤ Δ . \|x-x^*_{\bar{x},\lambda}\|_p\le\Delta. ∥ x − x x ˉ , λ ∗ ∥ p ≤ Δ.
Definition 7 (Ball optimization oracle). We say O b o \mathcal{O}_{\mathrm{bo}} O bo is a ( ϕ , λ ) (\phi,\lambda) ( ϕ , λ ) -ball optimization oracle for F : R d → R F:\mathbb{R}^d\to\mathbb{R} F : R d → R if given x ˉ ∈ R d \bar{x}\in\mathbb{R}^d x ˉ ∈ R d , O b o \mathcal{O}_{\mathrm{bo}} O bo returns x ∈ R d x\in\mathbb{R}^d x ∈ R d with
E [ F ( x ) + λ p ∥ x − x ˉ ∥ p p ] ≤ F ( x x ˉ , λ ∗ ) + λ p ∥ x x ˉ , λ ∗ − x ˉ ∥ p p + ϕ . \mathbb{E}\left[F(x)+\frac{\lambda}{p}\|x-\bar{x}\|_p^p\right]\le F(x^*_{\bar{x},\lambda})+\frac{\lambda}{p}\|x^*_{\bar{x},\lambda}-\bar{x}\|_p^p+\phi. E [ F ( x ) + p λ ∥ x − x ˉ ∥ p p ] ≤ F ( x x ˉ , λ ∗ ) + p λ ∥ x x ˉ , λ ∗ − x ˉ ∥ p p + ϕ .
Definition 8 (Stochastic proximal oracle). We say O s p \mathcal{O}_{\mathrm{sp}} O sp is a ( Δ , σ , λ ) (\Delta,\sigma,\lambda) ( Δ , σ , λ ) -stochastic proximal oracle for F : R d → R F:\mathbb{R}^d\to\mathbb{R} F : R d → R if, given x ˉ ∈ R d \bar{x}\in\mathbb{R}^d x ˉ ∈ R d , O s p \mathcal{O}_{\mathrm{sp}} O sp returns a random vector g ∈ R d g\in\mathbb{R}^d g ∈ R d such that
∥ E g − g x ˉ , λ ∗ ∥ q ≤ Δ , E ∥ g − E g ∥ q q ≤ σ q , \|\mathbb{E}g-g^*_{\bar{x},\lambda}\|_q\le\Delta,\qquad\mathbb{E}\|g-\mathbb{E}g\|_q^q\le\sigma^q, ∥ E g − g x ˉ , λ ∗ ∥ q ≤ Δ , E ∥ g − E g ∥ q q ≤ σ q ,
Proposition 10. Let F : R d → R F:\mathbb{R}^{d}\to\mathbb{R} F : R d → R be L L L -Lipschitz and convex, and let x ⋆ ∈ B p ( R ) x^\star\in\mathbb{B}_{p}(R) x ⋆ ∈ B p ( R ) . There is an algorithm B a l l A c c e l \mathop{\mathrm{BallAccel}} BallAccel (Algorithm 6) taking parameters r ∈ [ 0 , R ] r\in[0,R] r ∈ [ 0 , R ] and ϵ o p t ∈ ( 0 , L R ] \epsilon_{\mathrm{opt}}\in(0,LR] ϵ opt ∈ ( 0 , L R ] with the following guarantee. Define
κ : = L R ϵ o p t , K : = ( R r ) p p + 1 , λ ⋆ : = ϵ o p t K p R p log p κ . \kappa:=\frac{LR}{\epsilon_{\mathrm{opt}}},\qquad K:=\left(\frac{R}{r}\right)^{\frac{p}{p+1}},\qquad\lambda^\star:=\frac{\epsilon_{\mathrm{opt}}K^{p}}{R^{p}}\log^{p}\kappa. κ := ϵ opt L R , K := ( r R ) p + 1 p , λ ⋆ := R p ϵ opt K p log p κ . For a universal constant C b a > 0 C_{\mathrm{ba}}>0 C ba > 0 , B a l l A c c e l \mathop{\mathrm{BallAccel}} BallAccel runs in at most C b a K log κ C_{\mathrm{ba}}K\log\kappa C ba K log κ iterations and produces x ∈ R d x\in\mathbb{R}^{d} x ∈ R d such that
E F ( x ) ≤ F ( x ⋆ ) + ϵ o p t . \mathbb{E}F(x)\leq F(x^\star)+\epsilon_{\mathrm{opt}}. E F ( x ) ≤ F ( x ⋆ ) + ϵ opt . Moreover, in each iteration B a l l A c c e l \mathop{\mathrm{BallAccel}} BallAccel requires the following oracle calls (all for F F F ).
At most C b a log ( R κ r ) C_{\mathrm{ba}}\log\left(\frac{R\kappa}{r}\right) C ba log ( r R κ ) calls to a ( r C b a , λ ) \left(\frac{r}{C_{\mathrm{ba}}},\lambda\right) ( C ba r , λ ) -line search oracle with values of λ ∈ [ λ ⋆ C b a , C b a L r p − 1 ] \lambda\in\left[\frac{\lambda^\star}{C_{\mathrm{ba}}},\frac{C_{\mathrm{ba}}L}{rp-1}\right] λ ∈ [ C ba λ ⋆ , r p − 1 C ba L ] .
A single call to a ( λ r p C b a log p + 1 κ , λ ) \left(\frac{\lambda r^{p}}{C_{\mathrm{ba}}\log^{p+1}\kappa},\lambda\right) ( C ba l o g p + 1 κ λ r p , λ ) -ball optimization oracle with λ ∈ [ λ ⋆ C b a , C b a L r p − 1 ] \lambda\in\left[\frac{\lambda^\star}{C_{\mathrm{ba}}},\frac{C_{\mathrm{ba}}L}{rp-1}\right] λ ∈ [ C ba λ ⋆ , r p − 1 C ba L ] .
A single call to a ( ϵ o p t C b a R , ϵ o p t K 1 / p C b a R , λ ) \left(\frac{\epsilon_{\mathrm{opt}}}{C_{\mathrm{ba}}R},\frac{\epsilon_{\mathrm{opt}}K^{1/p}}{C_{\mathrm{ba}}R},\lambda\right) ( C ba R ϵ opt , C ba R ϵ opt K 1/ p , λ ) -stochastic proximal oracle with λ ∈ [ λ ⋆ C b a , C b a L r p − 1 ] \lambda\in\left[\frac{\lambda^\star}{C_{\mathrm{ba}}},\frac{C_{\mathrm{ba}}L}{rp-1}\right] λ ∈ [ C ba λ ⋆ , r p − 1 C ba L ] .
Proof.
Iteration Count: Having λ min = O ( λ ⋆ ) \lambda_{\min}=O(\lambda^\star) λ m i n = O ( λ ⋆ ) and ϵ = ϵ o p t \epsilon=\epsilon_{\mathrm{opt}} ϵ = ϵ opt , our analogous of B a l l A c c e l \mathop{\mathrm{BallAccel}} BallAccel runs in at most C b a K log κ C_{\mathrm{ba}}K\log\kappa C ba K log κ iterations, where the value of K max K_{\max} K m a x from Proposition 9 is O ( K log κ ) O(K\log\kappa) O ( K log κ ) .
O ( log λ max λ min + log ( 1 + R + 1 p − 1 ( G λ min ) 1 / ( p − 1 ) r ) ) . O\left(\log\frac{\lambda_{\max}}{\lambda_{\min}}+\log\left(1+\frac{R+\frac{1}{p-1}\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)}}{r}\right)\right). O log λ m i n λ m a x + log 1 + r R + p − 1 1 ( λ m i n G ) 1/ ( p − 1 ) .
With λ max = C b a L r p − 1 \lambda_{\max}=\frac{C_{\mathrm{ba}}L}{r^{p-1}} λ m a x = r p − 1 C ba L and λ min = λ ⋆ C b a \lambda_{\min}=\frac{\lambda^\star}{C_{\mathrm{ba}}} λ m i n = C ba λ ⋆ , we have, as K ≥ 1 K\geq1 K ≥ 1 ,
λ max λ min = C b a L r p − 1 × C b a λ ⋆ = C b a 2 L R p r p − 1 ϵ o p t K p log p κ = C b a 2 K κ R p − 1 r p − 1 K p log p κ = C b a 2 κ K 1 / p log p κ ≤ C b a 2 κ log p κ . \frac{\lambda_{\max}}{\lambda_{\min}}=\frac{C_{\mathrm{ba}}L}{r^{p-1}}\times\frac{C_{\mathrm{ba}}}{\lambda^\star}=\frac{C_{\mathrm{ba}}^{2}LR^{p}}{r^{p-1}\epsilon_{\mathrm{opt}}K^{p}\log^{p}\kappa}=\frac{C_{\mathrm{ba}}^{2}K\kappa R^{p-1}}{r^{p-1}K^{p}\log^{p}\kappa}=\frac{C_{\mathrm{ba}}^{2}\kappa}{K^{1/p}\log^{p}\kappa}\leq\frac{C_{\mathrm{ba}}^{2}\kappa}{\log^{p}\kappa}. λ m i n λ m a x = r p − 1 C ba L × λ ⋆ C ba = r p − 1 ϵ opt K p log p κ C ba 2 L R p = r p − 1 K p log p κ C ba 2 K κ R p − 1 = K 1/ p log p κ C ba 2 κ ≤ log p κ C ba 2 κ .
Now, note that
1 r ( G λ min ) 1 p − 1 = R r κ 1 / ( p − 1 ) K p / ( p − 1 ) log p / ( p − 1 ) κ = κ 1 / ( p − 1 ) ( R / r ) 1 / ( p 2 − 1 ) log p / ( p − 1 ) κ ≤ κ 1 / ( p − 1 ) ≤ κ , \frac{1}{r}\left(\frac{G}{\lambda_{\min}}\right)^{\frac{1}{p-1}}=\frac{R}{r}\frac{\kappa^{1/(p-1)}}{K^{p/(p-1)}\log^{p/(p-1)}\kappa}=\frac{\kappa^{1/(p-1)}}{(R/r)^{1/(p^{2}-1)}\log^{p/(p-1)}\kappa}\leq\kappa^{1/(p-1)}\leq\kappa, r 1 ( λ m i n G ) p − 1 1 = r R K p / ( p − 1 ) log p / ( p − 1 ) κ κ 1/ ( p − 1 ) = ( R / r ) 1/ ( p 2 − 1 ) log p / ( p − 1 ) κ κ 1/ ( p − 1 ) ≤ κ 1/ ( p − 1 ) ≤ κ ,
Therefore,
log ( 1 + R + 1 p − 1 ( G λ min ) 1 / ( p − 1 ) r ) ≤ log ( 1 + R r + κ ) = O ( log ( R κ r ) ) , \log\left(1+\frac{R+\frac{1}{p-1}\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)}}{r}\right)\leq\log\left(1+\frac{R}{r}+\kappa\right)=O\left(\log\left(\frac{R\kappa}{r}\right)\right), log 1 + r R + p − 1 1 ( λ m i n G ) 1/ ( p − 1 ) ≤ log ( 1 + r R + κ ) = O ( log ( r R κ ) ) ,
Together,
O ( log λ max λ min + log ( 1 + R + 1 p − 1 ( G λ min ) 1 / ( p − 1 ) r ) ) ≤ O ( log R κ r ) O\left(\log\frac{\lambda_{\max}}{\lambda_{\min}}+\log\left(1+\frac{R+\frac{1}{p-1}\left(\frac{G}{\lambda_{\min}}\right)^{1/(p-1)}}{r}\right)\right)\leq O\left(\log\frac{R\kappa}{r}\right) O log λ m i n λ m a x + log 1 + r R + p − 1 1 ( λ m i n G ) 1/ ( p − 1 ) ≤ O ( log r R κ )
E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k + 1 ) ] ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) + A k + 1 φ k + 1 − 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p + 2 2 − q a k + 1 q q σ k + 1 q + 2 a k + 1 δ k + 1 R ≤ A k ( f ( x k ) − f ( x ∗ ) ) + 2 D Φ ( x ∗ , v k ) + A k + 1 φ k + 1 − 1 q ( 1 − 2 1 − q ) λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p + 2 2 − q a k + 1 q q σ k + 1 q + 2 a k + 1 δ k + 1 R . \begin{aligned}
\mathbb{E}\left[A_{k+1}(f(x_{k+1})-f(x^*))+2D_{\Phi}(x^*,v_{k+1})\right] \\
&\leq A_k(f(x_k)-f(x^*))+2D_{\Phi}(x^*,v_k)+A_{k+1}\varphi_{k+1}-\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\|\widehat{x}_{k+1}-y_k\|_p^p
+\frac{2^{2-q}a_{k+1}^q}{q}\sigma_{k+1}^q+2a_{k+1}\delta_{k+1}R \\
&\leq A_k(f(x_k)-f(x^*))+2D_{\Phi}(x^*,v_k)+A_{k+1}\varphi_{k+1}-\frac{1}{q}\left(1-2^{1-q}\right)\lambda_{k+1}A_{k+1}\|\widehat{x}_{k+1}-y_k\|_p^p
+\frac{2^{2-q}a_{k+1}^q}{q}\sigma_{k+1}^q+2a_{k+1}\delta_{k+1}R.
\end{aligned} E [ A k + 1 ( f ( x k + 1 ) − f ( x ∗ )) + 2 D Φ ( x ∗ , v k + 1 ) ] ≤ A k ( f ( x k ) − f ( x ∗ )) + 2 D Φ ( x ∗ , v k ) + A k + 1 φ k + 1 − q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x k + 1 − y k ∥ p p + q 2 2 − q a k + 1 q σ k + 1 q + 2 a k + 1 δ k + 1 R ≤ A k ( f ( x k ) − f ( x ∗ )) + 2 D Φ ( x ∗ , v k ) + A k + 1 φ k + 1 − q 1 ( 1 − 2 1 − q ) λ k + 1 A k + 1 ∥ x k + 1 − y k ∥ p p + q 2 2 − q a k + 1 q σ k + 1 q + 2 a k + 1 δ k + 1 R .
If our algorithm does not terminate, one of the following occurs:
∥ x ^ k + 1 − y k ∥ p p = Ω ( r p ) \|\widehat{x}_{k+1}-y_k\|_p^p=\Omega(r^p) ∥ x k + 1 − y k ∥ p p = Ω ( r p )
λ k + 1 = O ( λ ∗ ) \lambda_{k+1}=O(\lambda^*) λ k + 1 = O ( λ ∗ )
If (1) occurs, then − 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p = − Ω ( λ k + 1 A k + 1 r p ) -\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\|\widehat{x}_{k+1}-y_k\|_p^p=-\Omega(\lambda_{k+1}A_{k+1}r^p) − q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x k + 1 − y k ∥ p p = − Ω ( λ k + 1 A k + 1 r p ) . If we choose φ k + 1 ≤ λ k + 1 r p C b a log p + 1 κ \varphi_{k+1}\leq\frac{\lambda_{k+1}r^p}{C_{\mathrm{ba}}\log^{p+1}\kappa} φ k + 1 ≤ C ba l o g p + 1 κ λ k + 1 r p , then A k + 1 φ k + 1 A_{k+1}\varphi_{k+1} A k + 1 φ k + 1 can be bounded by the first case. If (2) occurs, since Proposition 9 requires A max ≤ O ( R p / ϵ ) A_{\max}\leq O(R^p/\epsilon) A m a x ≤ O ( R p / ϵ ) , with the fact that the total iteration is K log κ K\log\kappa K log κ and φ k + 1 ≤ λ k + 1 r p C b a log p + 1 κ \varphi_{k+1}\leq\frac{\lambda_{k+1}r^p}{C_{\mathrm{ba}}\log^{p+1}\kappa} φ k + 1 ≤ C ba l o g p + 1 κ λ k + 1 r p by design, we have
∑ k + 1 ≤ K A k + 1 φ k + 1 ≤ O ( K log κ ⋅ R p ϵ o p t ⋅ λ ∗ r p C b a log p + 1 κ ) ≤ O ( R p ) . \sum_{k+1\leq K}A_{k+1}\varphi_{k+1}\leq O\left(K\log\kappa\cdot\frac{R^p}{\epsilon_{\mathrm{opt}}}\cdot\frac{\lambda^*r^p}{C_{\mathrm{ba}}\log^{p+1}\kappa}\right)\leq O(R^p). k + 1 ≤ K ∑ A k + 1 φ k + 1 ≤ O ( K log κ ⋅ ϵ opt R p ⋅ C ba log p + 1 κ λ ∗ r p ) ≤ O ( R p ) .
Here, the first term is the iteration count, the second term is due to an upper bound on A k + 1 A_{k+1} A k + 1 , and the third term is bounded since λ k + 1 = O ( λ ∗ ) \lambda_{k+1}=O(\lambda^*) λ k + 1 = O ( λ ∗ ) .
Stochastic proximal oracle parameters: In Proposition 9 we require σ q = O ( ϵ o p t / a k + 1 q − 1 ) \sigma^q=O(\epsilon_{\mathrm{opt}}/a_{k+1}^{q-1}) σ q = O ( ϵ opt / a k + 1 q − 1 ) , using the relation that A k + 1 p = 2 p − 1 λ k + 1 a k + 1 p A_{k+1}^p=2^{p-1}\lambda_{k+1}a_{k+1}^p A k + 1 p = 2 p − 1 λ k + 1 a k + 1 p , we have
σ q = ϵ o p t a k + 1 q − 1 = ϵ o p t λ k + 1 1 / p ( q − 1 ) 2 1 / p A k + 1 1 / p = Ω ( ϵ o p t 1 + 1 / p ⋅ ( λ ∗ ) 1 / p ( q − 1 ) R ) = Ω ( ϵ o p t q / ( q − 1 ) ⋅ K 1 / ( q − 1 ) log 1 / ( q − 1 ) κ R p / ( q − 1 ) ) , \sigma^q=\frac{\epsilon_{\mathrm{opt}}}{a_{k+1}^{q-1}}=\frac{\epsilon_{\mathrm{opt}}\lambda_{k+1}^{1/p(q-1)}2^{1/p}}{A_{k+1}^{1/p}}=\Omega\left(\frac{\epsilon_{\mathrm{opt}}^{1+1/p}\cdot(\lambda^*)^{1/p(q-1)}}{R}\right)=\Omega\left(\frac{\epsilon_{\mathrm{opt}}^{q/(q-1)}\cdot K^{1/(q-1)}\log^{1/(q-1)}\kappa}{R^{p/(q-1)}}\right), σ q = a k + 1 q − 1 ϵ opt = A k + 1 1/ p ϵ opt λ k + 1 1/ p ( q − 1 ) 2 1/ p = Ω ( R ϵ opt 1 + 1/ p ⋅ ( λ ∗ ) 1/ p ( q − 1 ) ) = Ω ( R p / ( q − 1 ) ϵ opt q / ( q − 1 ) ⋅ K 1/ ( q − 1 ) log 1/ ( q − 1 ) κ ) ,
where the second equality uses A k + 1 ≤ O ( R p / ϵ o p t ) A_{k+1}\leq O(R^p/\epsilon_{\mathrm{opt}}) A k + 1 ≤ O ( R p / ϵ opt ) . Hence, σ = Ω ~ ( ϵ o p t K 1 / p / R ) \sigma=\widetilde{\Omega}(\epsilon_{\mathrm{opt}}K^{1/p}/R) σ = Ω ( ϵ opt K 1/ p / R ) .
Lemma 20. Let x ˉ ∈ X \bar{x}\in\mathcal{X} x ˉ ∈ X , λ > 0 \lambda>0 λ > 0 . Suppose a ( 2 2 − p λ r p p p + 1 C b a p , λ ) \left(\frac{2^{2-p}\lambda r^p}{p^{p+1}C_{\mathrm{ba}}^p},\lambda\right) ( p p + 1 C ba p 2 2 − p λ r p , λ ) -ball optimization oracle returns x ∈ X ∩ B p ( x ˉ , r ) x\in\mathcal{X}\cap B_p(\bar{x},r) x ∈ X ∩ B p ( x ˉ , r ) , then
Pr ( ∥ x − x x ˉ , λ ∗ ∥ p ≤ r 3 C b a ) ≥ 2 3 . (41) \Pr\left(\|x-x^*_{\bar{x},\lambda}\|_p\leq\frac{r}{3C_{\mathrm{ba}}}\right)\geq\frac{2}{3}. \tag*{(41)} Pr ( ∥ x − x x ˉ , λ ∗ ∥ p ≤ 3 C ba r ) ≥ 3 2 . (41) Moreover, let x 1 , … , x m x_1,\ldots,x_m x 1 , … , x m be m m m independent outputs of the ball optimization oracle. If
m = ⌈ 20 log ( 1 / ζ ) ⌉ , m=\lceil20\log(1/\zeta)\rceil, m = ⌈ 20 log ( 1/ ζ )⌉ , then, with probability at least 1 − ζ 1-\zeta 1 − ζ , at least 0.51 m 0.51m 0.51 m of the points x 1 , … , x m x_1,\ldots,x_m x 1 , … , x m satisfy
∥ x i − x x ˉ , λ ∗ ∥ p ≤ r 3 C b a . \|x_i-x^*_{\bar{x},\lambda}\|_p\leq\frac{r}{3C_{\mathrm{ba}}}. ∥ x i − x x ˉ , λ ∗ ∥ p ≤ 3 C ba r .
Consequently, applying Aggregate (Algorithm 11) to x 1 , … , x m x_1,\ldots,x_m x 1 , … , x m returns a point x ^ \widehat{x} x satisfying
∥ x ^ − x x ˉ , λ ⋆ ∥ p ≤ r C b a \lVert\widehat{x}-x^\star_{\bar{x},\lambda}\rVert_p \le\frac{r}{C_{\mathrm{ba}}} ∥ x − x x ˉ , λ ⋆ ∥ p ≤ C ba r
with probability at least 1 − ζ 1-\zeta 1 − ζ .
Proof. By uniform convexity, the function Φ ( x ) : = 1 p ∥ x ∥ p p \Phi(x):=\frac{1}{p}\lVert x\rVert_p^p Φ ( x ) := p 1 ∥ x ∥ p p satisfies, for all u , v ∈ R d u,v\in\mathbb{R}^d u , v ∈ R d ,
Φ ( u ) ≥ Φ ( v ) + ⟨ J p ( v ) , u − v ⟩ + 2 2 − p p ∥ u − v ∥ p p (42) \Phi(u)\ge\Phi(v)+\langle J_p(v),u-v\rangle+\frac{2^{2-p}}{p}\lVert u-v\rVert_p^p \tag*{(42)} Φ ( u ) ≥ Φ ( v ) + ⟨ J p ( v ) , u − v ⟩ + p 2 2 − p ∥ u − v ∥ p p (42)
where J p ( v ) = ( ∣ v i ∣ p − 2 v i ) i = 1 d J_p(v)=\left(|v_i|^{p-2}v_i\right)_{i=1}^d J p ( v ) = ( ∣ v i ∣ p − 2 v i ) i = 1 d . Therefore,
λ p ∥ x − x ˉ ∥ p p ≥ λ p ∥ y − x ˉ ∥ p p + λ ⟨ J p ( y − x ˉ ) , x − y ⟩ + 2 2 − p λ p ∥ x − y ∥ p p . \frac{\lambda}{p}\lVert x-\bar{x}\rVert_p^p\ge\frac{\lambda}{p}\lVert y-\bar{x}\rVert_p^p+\lambda\langle J_p(y-\bar{x}),x-y\rangle+\frac{2^{2-p}\lambda}{p}\lVert x-y\rVert_p^p. p λ ∥ x − x ˉ ∥ p p ≥ p λ ∥ y − x ˉ ∥ p p + λ ⟨ J p ( y − x ˉ ) , x − y ⟩ + p 2 2 − p λ ∥ x − y ∥ p p .
Since F F F is convex, for any g ∈ ∂ F ( y ) g\in\partial F(y) g ∈ ∂ F ( y ) , F ( x ) ≥ F ( y ) + ⟨ g , x − y ⟩ F(x)\ge F(y)+\langle g,x-y\rangle F ( x ) ≥ F ( y ) + ⟨ g , x − y ⟩ . Combining two inequalities gives
F ( x ) + λ p ∥ x − x ˉ ∥ p p ≥ F ( y ) + λ p ∥ y − x ˉ ∥ p p + ⟨ g + λ J p ( y − x ˉ ) , x − y ⟩ + 2 2 − p λ p ∥ x − y ∥ p p . (43) F(x)+\frac{\lambda}{p}\lVert x-\bar{x}\rVert_p^p\ge F(y)+\frac{\lambda}{p}\lVert y-\bar{x}\rVert_p^p+\langle g+\lambda J_p(y-\bar{x}),x-y\rangle+\frac{2^{2-p}\lambda}{p}\lVert x-y\rVert_p^p. \tag*{(43)} F ( x ) + p λ ∥ x − x ˉ ∥ p p ≥ F ( y ) + p λ ∥ y − x ˉ ∥ p p + ⟨ g + λ J p ( y − x ˉ ) , x − y ⟩ + p 2 2 − p λ ∥ x − y ∥ p p . (43)
Taking y = x x ˉ , λ ⋆ y=x^\star_{\bar{x},\lambda} y = x x ˉ , λ ⋆ and using the first-order optimality condition for x x ˉ , λ ⋆ x^\star_{\bar{x},\lambda} x x ˉ , λ ⋆ gives
F ( x ) + λ p ∥ x − x ˉ ∥ p p ≥ F ( y ) + λ p ∥ y − x ˉ ∥ p p + 2 2 − p λ p ∥ x − y ∥ p p . (44) F(x)+\frac{\lambda}{p}\lVert x-\bar{x}\rVert_p^p\ge F(y)+\frac{\lambda}{p}\lVert y-\bar{x}\rVert_p^p+\frac{2^{2-p}\lambda}{p}\lVert x-y\rVert_p^p. \tag*{(44)} F ( x ) + p λ ∥ x − x ˉ ∥ p p ≥ F ( y ) + p λ ∥ y − x ˉ ∥ p p + p 2 2 − p λ ∥ x − y ∥ p p . (44)
Combining (44) with the definition of ( λ r p C b a p , λ ) \left(\frac{\lambda r^p}{C_{\mathrm{ba}}^p},\lambda\right) ( C ba p λ r p , λ ) -ball optimization oracle gives
E ∥ x − x x ˉ , λ ⋆ ∥ p p ≤ p 2 p − 2 λ E [ F ( x ) + λ p ∥ x − x ˉ ∥ p p − F ( x x ˉ , λ ⋆ ) − λ p ∥ x x ˉ , λ ⋆ − x ˉ ∥ p p ] ≤ p 2 p − 2 λ ⋅ 2 2 − p λ r p p 3 p + 1 C b a p = r p 3 p + 1 C b a p . \begin{aligned}
\mathbb{E}\lVert x-x^\star_{\bar{x},\lambda}\rVert_p^p\le\frac{p2^{p-2}}{\lambda}\mathbb{E}\left[F(x)+\frac{\lambda}{p}\lVert x-\bar{x}\rVert_p^p-F(x^\star_{\bar{x},\lambda})-\frac{\lambda}{p}\lVert x^\star_{\bar{x},\lambda}-\bar{x}\rVert_p^p\right] \\
&\le\frac{p2^{p-2}}{\lambda}\cdot\frac{2^{2-p}\lambda r^p}{p3^{p+1}C_{\mathrm{ba}}^p}=\frac{r^p}{3^{p+1}C_{\mathrm{ba}}^p}.
\end{aligned} E ∥ x − x x ˉ , λ ⋆ ∥ p p ≤ λ p 2 p − 2 E [ F ( x ) + p λ ∥ x − x ˉ ∥ p p − F ( x x ˉ , λ ⋆ ) − p λ ∥ x x ˉ , λ ⋆ − x ˉ ∥ p p ] ≤ λ p 2 p − 2 ⋅ p 3 p + 1 C ba p 2 2 − p λ r p = 3 p + 1 C ba p r p .
Therefore, by Markov’s inequality,
Pr ( ∥ x − x x ˉ , λ ⋆ ∥ p > r 3 C b a ) = Pr ( ∥ x − x x ˉ , λ ⋆ ∥ p p > r p 3 p C b a p ) ≤ E ∥ x − x x ˉ , λ ⋆ ∥ p p r p / ( 3 p C b a p ) ≤ 1 3 . \Pr\left(\lVert x-x^\star_{\bar{x},\lambda}\rVert_p>\frac{r}{3C_{\mathrm{ba}}}\right)=\Pr\left(\lVert x-x^\star_{\bar{x},\lambda}\rVert_p^p>\frac{r^p}{3^pC_{\mathrm{ba}}^p}\right)\le\frac{\mathbb{E}\lVert x-x^\star_{\bar{x},\lambda}\rVert_p^p}{r^p/(3^pC_{\mathrm{ba}}^p)}\le\frac{1}{3}. Pr ( ∥ x − x x ˉ , λ ⋆ ∥ p > 3 C ba r ) = Pr ( ∥ x − x x ˉ , λ ⋆ ∥ p p > 3 p C ba p r p ) ≤ r p / ( 3 p C ba p ) E ∥ x − x x ˉ , λ ⋆ ∥ p p ≤ 3 1 .
Hence
Pr ( ∥ x − x x ˉ , λ ⋆ ∥ p ≤ r 3 C b a ) ≥ 2 3 . \Pr\left(\lVert x-x^\star_{\bar{x},\lambda}\rVert_p\le\frac{r}{3C_{\mathrm{ba}}}\right)\ge\frac{2}{3}. Pr ( ∥ x − x x ˉ , λ ⋆ ∥ p ≤ 3 C ba r ) ≥ 3 2 .
For i = 1 , … , m i=1,\ldots,m i = 1 , … , m , define
Z i : = 1 { ∥ x i − x x ˉ , λ ⋆ ∥ p ≤ r 3 C b a } . Z_i:=\mathbf{1}\left\{\lVert x_i-x^\star_{\bar{x},\lambda}\rVert_p\le\frac{r}{3C_{\mathrm{ba}}}\right\}. Z i := 1 { ∥ x i − x x ˉ , λ ⋆ ∥ p ≤ 3 C ba r } .
Then Z 1 , … , Z m Z_1,\ldots,Z_m Z 1 , … , Z m are independent Bernoulli random variables with E Z i ≥ 2 3 \mathbb{E}Z_i\ge\frac{2}{3} E Z i ≥ 3 2 . By a Chernoff bound, with probability at least 1 − ζ 1-\zeta 1 − ζ , at least 0.51 m 0.51m 0.51 m of the points x 1 , … , x m x_1,\ldots,x_m x 1 , … , x m satisfy
∥ x i − x x ˉ , λ ⋆ ∥ p ≤ r 3 C b a . \lVert x_i-x^\star_{\bar{x},\lambda}\rVert_p\le\frac{r}{3C_{\mathrm{ba}}}. ∥ x i − x x ˉ , λ ⋆ ∥ p ≤ 3 C ba r .
Therefore, the preconditions of Fact 6 are satisfied, and Aggregate (Algorithm 11) returns x ^ \widehat{x} x such that
∥ x ^ − x x ˉ , λ ⋆ ∥ p ≤ r C b a . \lVert\widehat{x}-x^\star_{\bar{x},\lambda}\rVert_p\le\frac{r}{C_{\mathrm{ba}}}. ∥ x − x x ˉ , λ ⋆ ∥ p ≤ C ba r .
Algorithm 8 p p p -MorGradEst
Input: r r r -BROO O λ , δ ( ⋅ ) \mathcal{O}_{\lambda,\delta}(\cdot) O λ , δ ( ⋅ ) of radius r r r , query point y y y , regularization λ > 0 \lambda> 0 λ > 0 , bias β > 0 \beta> 0 β > 0 , σ > 0 \sigma> 0 σ > 0
Set T max = max { 2 , C p λ r p ( p − 1 ) β p } T_{\max} = \max\left\{2,\frac{C_p\lambda r^p(p-1)}{\beta^p}\right\} T m a x = max { 2 , β p C p λ r p ( p − 1 ) } , T 0 = max { 1 , C p λ p r p ( p − 1 ) log p − 1 ( 2 T max ) σ p } T_0 = \max\left\{1,\frac{C_p\lambda^p r^{p(p-1)}\log^{p-1}(2T_{\max})}{\sigma^p}\right\} T 0 = max { 1 , σ p C p λ p r p ( p − 1 ) l o g p − 1 ( 2 T m a x ) }
j max = ⌊ log 2 T max ⌋ j_{\max} = \lfloor\log_2 T_{\max}\rfloor j m a x = ⌊ log 2 T m a x ⌋ , δ j = r ( 2 j T 0 ) 1 / p \delta_j = \frac{r}{(2^jT_0)^{1/p}} δ j = ( 2 j T 0 ) 1/ p r
x 0 = O λ , δ 0 ( y ) x_0 = \mathcal{O}_{\lambda,\delta_0}(y) x 0 = O λ , δ 0 ( y ) , g 0 = λ J p ( y − x 0 ) g_0 = \lambda J_p(y-x_0) g 0 = λ J p ( y − x 0 )
Sample J ∼ Geom ( 1 / 2 ) J \sim\operatorname{Geom}(1/2) J ∼ Geom ( 1/2 )
if J ≤ j max J \le j_{\max} J ≤ j m a x then
x J = O λ , δ J ( y ) x_J = \mathcal{O}_{\lambda,\delta_J}(y) x J = O λ , δ J ( y ) , x J − 1 = O λ , δ J − 1 ( y ) x_{J-1} = \mathcal{O}_{\lambda,\delta_{J-1}}(y) x J − 1 = O λ , δ J − 1 ( y )
g J = λ J p ( y − x J ) g_J = \lambda J_p(y-x_J) g J = λ J p ( y − x J ) , g J − 1 = λ J p ( y − x J − 1 ) g_{J-1} = \lambda J_p(y-x_{J-1}) g J − 1 = λ J p ( y − x J − 1 )
g ^ = g 0 + 2 J ( g J − g J − 1 ) \widehat{g} = g_0 + 2^J(g_J-g_{J-1}) g = g 0 + 2 J ( g J − g J − 1 )
else
g ^ = g 0 \widehat{g} = g_0 g = g 0
end if
return g ^ \widehat{g} g
Lemma 21. Let F : X → R F:\mathcal{X}\to\mathbb{R} F : X → R satisfy Assumption 1. For a query point y ∈ X y\in\mathcal{X} y ∈ X and λ , r > 0 \lambda,r>0 λ , r > 0 , define
x ⋆ ∈ arg min x ∈ X { F ( x ) + λ p ∥ x − y ∥ p p } , g = λ J p ( y − x ⋆ ) , x^\star\in\arg\min_{x\in\mathcal{X}}\left\{F(x)+\frac{\lambda}{p}\|x-y\|_p^p\right\},\qquad g=\lambda J_p(y-x^\star), x ⋆ ∈ arg x ∈ X min { F ( x ) + p λ ∥ x − y ∥ p p } , g = λ J p ( y − x ⋆ ) , where J p ( z ) = ( ∣ z i ∣ p − 2 z i ) i = 1 d J_p(z)=(|z_i|^{p-2}z_i)_{i=1}^d J p ( z ) = ( ∣ z i ∣ p − 2 z i ) i = 1 d . Suppose that ∥ x ⋆ − y ∥ p ≤ r \|x^\star-y\|_p\le r ∥ x ⋆ − y ∥ p ≤ r . Let g ^ \widehat{g} g be the geometric estimator in Algorithm 8, run with these values of T 0 T_0 T 0 and j max j_{\max} j m a x and with
δ j = r ( 2 j T 0 ) 1 / p , 0 ≤ j ≤ j max , \delta_j=\frac{r}{(2^jT_0)^{1/p}},\qquad0\le j\le j_{\max}, δ j = ( 2 j T 0 ) 1/ p r , 0 ≤ j ≤ j m a x , Then, for a constant C p > 0 C_p>0 C p > 0 depending only on p p p ,
∥ E g ^ − g ∥ q ≤ C p λ r p − 1 2 j max T 0 , (45) \|\mathbb{E}\widehat{g}-g\|_q\le\frac{C_p\lambda r^{p-1}}{\sqrt{2^{j_{\max}}T_0}}, \tag*{(45)} ∥ E g − g ∥ q ≤ 2 j m a x T 0 C p λ r p − 1 , (45) E ∥ g ^ − E g ^ ∥ q q ≤ C p λ q r p T 0 q / 2 ( 1 + ∑ j = 1 j max 2 − j ( 1 − q / 2 ) ) . (46) \mathbb{E}\|\widehat{g}-\mathbb{E}\widehat{g}\|_q^q\le\frac{C_p\lambda^q r^p}{T_0^{q/2}}\left(1+\sum_{j=1}^{j_{\max}}2^{-j(1-q/2)}\right). \tag*{(46)} E ∥ g − E g ∥ q q ≤ T 0 q /2 C p λ q r p ( 1 + j = 1 ∑ j m a x 2 − j ( 1 − q /2 ) ) . (46)
Proof. Let Φ ( u ) = ∥ u ∥ p p / p \Phi(u)=\|u\|_p^p/p Φ ( u ) = ∥ u ∥ p p / p , so that
D Φ ( u , v ) = Φ ( u ) − Φ ( v ) − ⟨ J p ( v ) , u − v ⟩ . D_\Phi(u,v)=\Phi(u)-\Phi(v)-\langle J_p(v),u-v\rangle. D Φ ( u , v ) = Φ ( u ) − Φ ( v ) − ⟨ J p ( v ) , u − v ⟩ .
We first show that
∥ J p ( u ) − J p ( v ) ∥ q 2 ≤ C p ( ∥ u ∥ p + ∥ v ∥ p ) p − 2 D Φ ( u , v ) . (47) \|J_p(u)-J_p(v)\|_q^2\le C_p(\|u\|_p+\|v\|_p)^{p-2}D_\Phi(u,v). \tag*{(47)} ∥ J p ( u ) − J p ( v ) ∥ q 2 ≤ C p ( ∥ u ∥ p + ∥ v ∥ p ) p − 2 D Φ ( u , v ) . (47)
For p = 2 p=2 p = 2 , this follows from ∥ u − v ∥ 2 2 = 2 D Φ ( u , v ) \|u-v\|_2^2=2D_\Phi(u,v) ∥ u − v ∥ 2 2 = 2 D Φ ( u , v ) . For p > 2 p>2 p > 2 , integrating the second derivative of ∣ a ∣ p / p |a|^p/p ∣ a ∣ p / p gives
D Φ ( u , v ) = ( p − 1 ) ∑ i = 1 d ( u i − v i ) 2 ∫ 0 1 ( 1 − t ) ∣ v i + t ( u i − v i ) ∣ p − 2 d t ≥ c p ∑ i = 1 d ( ∣ u i ∣ + ∣ v i ∣ ) p − 2 ( u i − v i ) 2 D_\Phi(u,v)=(p-1)\sum_{i=1}^d(u_i-v_i)^2\int_0^1(1-t)|v_i+t(u_i-v_i)|^{p-2}\,\mathrm{d}t\ge c_p\sum_{i=1}^d(|u_i|+|v_i|)^{p-2}(u_i-v_i)^2 D Φ ( u , v ) = ( p − 1 ) i = 1 ∑ d ( u i − v i ) 2 ∫ 0 1 ( 1 − t ) ∣ v i + t ( u i − v i ) ∣ p − 2 d t ≥ c p i = 1 ∑ d ( ∣ u i ∣ + ∣ v i ∣ ) p − 2 ( u i − v i ) 2
for a constant c p > 0 c_p>0 c p > 0 depending only on p p p . Indeed, after rescaling so that ∣ a ∣ + ∣ b ∣ = 1 |a|+|b|=1 ∣ a ∣ + ∣ b ∣ = 1 , the integral
∫ 0 1 ( 1 − t ) ∣ b + t ( a − b ) ∣ p − 2 d t \int_0^1(1-t)|b+t(a-b)|^{p-2}\,\mathrm{d}t ∫ 0 1 ( 1 − t ) ∣ b + t ( a − b ) ∣ p − 2 d t
is continuous and strictly positive on this compact set, and therefore has a positive minimum. Also,
∣ ∣ u i ∣ p − 2 u i − ∣ v i ∣ p − 2 v i ∣ ≤ ( p − 1 ) ( ∣ u i ∣ + ∣ v i ∣ ) p − 2 ∣ u i − v i ∣ . \left||u_i|^{p-2}u_i-|v_i|^{p-2}v_i\right|\le(p-1)(|u_i|+|v_i|)^{p-2}|u_i-v_i|. ∣ u i ∣ p − 2 u i − ∣ v i ∣ p − 2 v i ≤ ( p − 1 ) ( ∣ u i ∣ + ∣ v i ∣ ) p − 2 ∣ u i − v i ∣.
Hölder’s inequality therefore yields
∥ J p ( u ) − J p ( v ) ∥ q 2 ≤ ( p − 1 ) 2 ( ∑ i = 1 d ( ∣ u i ∣ + ∣ v i ∣ ) p ) ( p − 2 ) / p ∑ i = 1 d ( ∣ u i ∣ + ∣ v i ∣ ) p − 2 ( u i − v i ) 2 ≤ C p ( ∥ u ∥ p + ∥ v ∥ p ) p − 2 D Φ ( u , v ) , \begin{aligned}
\lVert J_p(u)-J_p(v)\rVert_q^2 \le(p-1)^2\left(\sum_{i=1}^{d}(|u_i|+|v_i|)^p\right)^{(p-2)/p}\sum_{i=1}^{d}(|u_i|+|v_i|)^{p-2}(u_i-v_i)^2 \\
&\le C_p(\lVert u\rVert_p+\lVert v\rVert_p)^{p-2}D_\Phi(u,v),
\end{aligned} ∥ J p ( u ) − J p ( v ) ∥ q 2 ≤ ( p − 1 ) 2 ( i = 1 ∑ d ( ∣ u i ∣ + ∣ v i ∣ ) p ) ( p − 2 ) / p i = 1 ∑ d ( ∣ u i ∣ + ∣ v i ∣ ) p − 2 ( u i − v i ) 2 ≤ C p (∥ u ∥ p + ∥ v ∥ p ) p − 2 D Φ ( u , v ) ,
which proves (47) . By the convexity of F F F and optimality of x ⋆ x^\star x ⋆ , for every x ∈ X x\in\mathcal{X} x ∈ X ,
F ( x ) + λ p ∥ x − y ∥ p p − F ( x ⋆ ) − λ p ∥ x ⋆ − y ∥ p p ≥ λ D Φ ( x − y , x ⋆ − y ) . F(x)+\frac{\lambda}{p}\lVert x-y\rVert_p^p-F(x^\star)-\frac{\lambda}{p}\lVert x^\star-y\rVert_p^p\ge\lambda D_\Phi(x-y,x^\star-y). F ( x ) + p λ ∥ x − y ∥ p p − F ( x ⋆ ) − p λ ∥ x ⋆ − y ∥ p p ≥ λ D Φ ( x − y , x ⋆ − y ) .
Since ∥ x j − y ∥ p ≤ r \lVert x_j-y\rVert_p\le r ∥ x j − y ∥ p ≤ r and ∥ x ⋆ − y ∥ p ≤ r \lVert x^\star-y\rVert_p\le r ∥ x ⋆ − y ∥ p ≤ r , applying (47) with u = x j − y u=x_j-y u = x j − y and v = x ⋆ − y v=x^\star-y v = x ⋆ − y gives
E ∥ g j − g ∥ q 2 ≤ C p λ r p − 2 E [ F ( x j ) + λ p ∥ x j − y ∥ p p − F ( x ⋆ ) − λ p ∥ x ⋆ − y ∥ p p ] ≤ C p λ 2 r 2 p − 2 2 j T 0 . (48) \begin{aligned}
\mathbb{E}\lVert g_j-g\rVert_q^2\le C_p\lambda r^{p-2}\mathbb{E}\left[F(x_j)+\frac{\lambda}{p}\lVert x_j-y\rVert_p^p-F(x^\star)-\frac{\lambda}{p}\lVert x^\star-y\rVert_p^p\right] \\
&\le\frac{C_p\lambda^2r^{2p-2}}{2^jT_0}. \tag*{(48)}
\end{aligned} E ∥ g j − g ∥ q 2 ≤ C p λ r p − 2 E [ F ( x j ) + p λ ∥ x j − y ∥ p p − F ( x ⋆ ) − p λ ∥ x ⋆ − y ∥ p p ] ≤ 2 j T 0 C p λ 2 r 2 p − 2 . (48)
Here we used g j = λ J p ( y − x j ) g_j=\lambda J_p(y-x_j) g j = λ J p ( y − x j ) , g = λ J p ( y − x ⋆ ) g=\lambda J_p(y-x^\star) g = λ J p ( y − x ⋆ ) , and J p ( − u ) = − J p ( u ) J_p(-u)=-J_p(u) J p ( − u ) = − J p ( u ) . By the definition of g ^ \widehat{g} g ,
E g ^ = E g 0 + ∑ j = 1 j max ( E g j − E g j − 1 ) = E g j max . \mathbb{E}\widehat{g}=\mathbb{E}g_0+\sum_{j=1}^{j_{\max}}(\mathbb{E}g_j-\mathbb{E}g_{j-1})=\mathbb{E}g_{j_{\max}}. E g = E g 0 + j = 1 ∑ j m a x ( E g j − E g j − 1 ) = E g j m a x .
Consequently,
∥ E g ^ − g ∥ q ≤ ( E ∥ g j max − g ∥ q 2 ) 1 / 2 ≤ C p λ r p − 1 2 j max T 0 , \lVert\mathbb{E}\widehat{g}-g\rVert_q\le\left(\mathbb{E}\lVert g_{j_{\max}}-g\rVert_q^2\right)^{1/2}\le\frac{C_p\lambda r^{p-1}}{\sqrt{2^{j_{\max}}T_0}}, ∥ E g − g ∥ q ≤ ( E ∥ g j m a x − g ∥ q 2 ) 1/2 ≤ 2 j m a x T 0 C p λ r p − 1 ,
which proves (45) . Since 1 < q ≤ 2 1<q\le2 1 < q ≤ 2 , Jensen’s inequality and ( p − 1 ) q = p (p-1)q=p ( p − 1 ) q = p give
E ∥ g j − g ∥ q q ≤ ( E ∥ g j − g ∥ q 2 ) q / 2 ≤ C p λ q r p ( 2 j T 0 ) q / 2 . \mathbb{E}\lVert g_j-g\rVert_q^q\le\left(\mathbb{E}\lVert g_j-g\rVert_q^2\right)^{q/2}\le\frac{C_p\lambda^qr^p}{(2^jT_0)^{q/2}}. E ∥ g j − g ∥ q q ≤ ( E ∥ g j − g ∥ q 2 ) q /2 ≤ ( 2 j T 0 ) q /2 C p λ q r p .
It follows that
E ∥ g j − g j − 1 ∥ q q ≤ 2 q − 1 ( E ∥ g j − g ∥ q q + E ∥ g j − 1 − g ∥ q q ) ≤ C p λ q r p ( 2 j − 1 T 0 ) q / 2 . \mathbb{E}\lVert g_j-g_{j-1}\rVert_q^q\le2^{q-1}\left(\mathbb{E}\lVert g_j-g\rVert_q^q+\mathbb{E}\lVert g_{j-1}-g\rVert_q^q\right)\le\frac{C_p\lambda^qr^p}{(2^{j-1}T_0)^{q/2}}. E ∥ g j − g j − 1 ∥ q q ≤ 2 q − 1 ( E ∥ g j − g ∥ q q + E ∥ g j − 1 − g ∥ q q ) ≤ ( 2 j − 1 T 0 ) q /2 C p λ q r p .
Using the definition of g ^ \widehat{g} g once more,
E ∥ g ^ − E g ^ ∥ q q ≤ 2 q E ∥ g ^ − g ∥ q q ≤ C p ( E ∥ g 0 − g ∥ q q + ∑ j = 1 j max 2 j ( q − 1 ) E ∥ g j − g j − 1 ∥ q q ) ≤ C p λ q r p T 0 q / 2 ( 1 + ∑ j = 1 j max 2 − j ( 1 − q / 2 ) ) . \begin{aligned}
\mathbb{E}\lVert\widehat{g}-\mathbb{E}\widehat{g}\rVert_q^q\le2^q\mathbb{E}\lVert\widehat{g}-g\rVert_q^q \\
&\le C_p\left(\mathbb{E}\lVert g_0-g\rVert_q^q+\sum_{j=1}^{j_{\max}}2^{j(q-1)}\mathbb{E}\lVert g_j-g_{j-1}\rVert_q^q\right) \\
&\le\frac{C_p\lambda^qr^p}{T_0^{q/2}}\left(1+\sum_{j=1}^{j_{\max}}2^{-j(1-q/2)}\right).
\end{aligned} E ∥ g − E g ∥ q q ≤ 2 q E ∥ g − g ∥ q q ≤ C p ( E ∥ g 0 − g ∥ q q + j = 1 ∑ j m a x 2 j ( q − 1 ) E ∥ g j − g j − 1 ∥ q q ) ≤ T 0 q /2 C p λ q r p ( 1 + j = 1 ∑ j m a x 2 − j ( 1 − q /2 ) ) .
This proves (46) . The argument also applies after conditioning on the history before the call.
Lemma 22. Let F F F satisfy Assumption 1 with a minimizer x ⋆ x^\star x ⋆ . Let ℓ = log ( G R 2 r ϵ o p t ) \ell= \log\left(\frac{GR^2}{r\epsilon_{\mathrm{opt}}}\right) ℓ = log ( r ϵ opt G R 2 ) . Let
φ k = λ k r p C p ℓ p + 1 , σ k q = λ k q r p C p ℓ p + 1 , β k = ϵ o p t C p R , A 0 = R p − 1 G and A max = C p R p ϵ o p t . \varphi_k = \frac{\lambda_k r^p}{C_p\ell^{p+1}}, \qquad\sigma_k^q = \frac{\lambda_k^q r^p}{C_p\ell^{p+1}}, \qquad\beta_k = \frac{\epsilon_{\mathrm{opt}}}{C_pR}, \qquad A_0 = \frac{R^{p-1}}{G} \quad\text{and}\quad A_{\max} = \frac{C_pR^p}{\epsilon_{\mathrm{opt}}}. φ k = C p ℓ p + 1 λ k r p , σ k q = C p ℓ p + 1 λ k q r p , β k = C p R ϵ opt , A 0 = G R p − 1 and A m a x = ϵ opt C p R p . Define x ^ k : = arg min x ∈ X { F ( x ) + λ k p ∥ x − y k − 1 ∥ p p } \widehat{x}_k := \mathop{\arg\min}_{x\in\mathcal{X}}\left\{F(x)+\frac{\lambda_k}{p}\lVert x-y_{k-1}\rVert_p^p\right\} x k := arg min x ∈ X { F ( x ) + p λ k ∥ x − y k − 1 ∥ p p } , and assume that for each k k k we have ∥ x ^ k − y k − 1 ∥ p ≤ r \lVert\widehat{x}_k-y_{k-1}\rVert_p\le r ∥ x k − y k − 1 ∥ p ≤ r and that one of the following must occur:
λ k < 2 λ m : = 2 ϵ o p t r p + 1 R p / ( p + 1 ) ℓ p \lambda_k<2\lambda_m:=\frac{2\epsilon_{\mathrm{opt}}}{r^{p+1}R^{p/(p+1)}}\ell^p λ k < 2 λ m := r p + 1 R p / ( p + 1 ) 2 ϵ opt ℓ p , or
∥ x ^ k − y k − 1 ∥ p ≥ 3 4 r \lVert\widehat{x}_k-y_{k-1}\rVert_p\ge\frac{3}{4}r ∥ x k − y k − 1 ∥ p ≥ 4 3 r .
Suppose λ k ≥ λ m \lambda_k\ge\lambda_m λ k ≥ λ m for every k k k and
A k + 1 p − 1 = 2 p − 1 λ k + 1 a k + 1 p , A k + 1 = A k + a k + 1 . A_{k+1}^{p-1}=2^{p-1}\lambda_{k+1}a_{k+1}^p,\qquad A_{k+1}=A_k+a_{k+1}. A k + 1 p − 1 = 2 p − 1 λ k + 1 a k + 1 p , A k + 1 = A k + a k + 1 . Let K ≤ K max K\le K_{\max} K ≤ K m a x be the stopping time of the algorithm, which stops when A k ≥ A max A_k\ge A_{\max} A k ≥ A m a x or k = K max k=K_{\max} k = K m a x . Then, for sufficiently large C p C_p C p , there exist constants c 1 , p , c 2 , p > 0 c_{1,p},c_{2,p}>0 c 1 , p , c 2 , p > 0 depending only on p p p such that
E [ A K ( F ( x K ) − F ( x ⋆ ) − c 1 , p ϵ o p t ) + c 2 , p ∑ i ≤ K λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ A 0 ( F ( x 0 ) − F ( x ⋆ ) ) + C p R p . \mathbb{E}\left[A_K\left(F(x_K)-F(x^\star)-c_{1,p}\epsilon_{\mathrm{opt}}\right)+c_{2,p}\sum_{i\le K}\lambda_iA_i\lVert\widehat{x}_i-y_{i-1}\rVert_p^p\right]\le A_0\left(F(x_0)-F(x^\star)\right)+C_pR^p. E [ A K ( F ( x K ) − F ( x ⋆ ) − c 1 , p ϵ opt ) + c 2 , p i ≤ K ∑ λ i A i ∥ x i − y i − 1 ∥ p p ] ≤ A 0 ( F ( x 0 ) − F ( x ⋆ ) ) + C p R p .
Proof. Define the filtration
F k = σ ( x 1 , v 1 , A 1 , ζ 1 , … , x k , v k , A k , ζ k ) , \mathcal{F}_k=\sigma(x_1,v_1,A_1,\zeta_1,\ldots,x_k,v_k,A_k,\zeta_k), F k = σ ( x 1 , v 1 , A 1 , ζ 1 , … , x k , v k , A k , ζ k ) ,
where ζ i \zeta_i ζ i is the internal randomness of NextLambda, and note that A k + 1 A_{k+1} A k + 1 , y k y_k y k , and x ^ k + 1 \widehat{x}_{k+1} x k + 1 are deterministic when conditioned on x k , v k , A k , ζ k x_k,v_k,A_k,\zeta_k x k , v k , A k , ζ k . Following the proof of Lemma 18, we define
M k = A k ( F ( x k ) − F ( x ⋆ ) − ϵ o p t 20 ) + 2 D Φ ( x ⋆ , v k ) + 1 − 2 1 − q 2 q ∑ i ≤ k λ i A i ∥ x ^ i − y i − 1 ∥ p p M_k=A_k\left(F(x_k)-F(x^\star)-\frac{\epsilon_{\mathrm{opt}}}{20}\right)+2D_\Phi(x^\star,v_k)+\frac{1-2^{1-q}}{2q}\sum_{i\le k}\lambda_iA_i\lVert\widehat{x}_i-y_{i-1}\rVert_p^p M k = A k ( F ( x k ) − F ( x ⋆ ) − 20 ϵ opt ) + 2 D Φ ( x ⋆ , v k ) + 2 q 1 − 2 1 − q i ≤ k ∑ λ i A i ∥ x i − y i − 1 ∥ p p
and show it is a supermartingale adapted to the filtration F k \mathcal{F}_k F k . From Lemma 17, we have
E [ M k + 1 ∣ F k ] ≤ A k ( F ( x k ) − F ( x ⋆ ) ) + 2 D Φ ( x ⋆ , v k ) − 1 − 2 1 − q q λ k + 1 A k + 1 ∥ x ^ k + 1 − y k ∥ p p + μ k + 1 − A k + 1 ϵ o p t 20 + 1 − 2 1 − q 2 q ∑ i ≤ k + 1 λ i A i ∥ x ^ i − y i − 1 ∥ p p , (49) \begin{aligned}
\mathbb{E}[M_{k+1}\mid\mathcal{F}_k]\le{}& A_k\left(F(x_k)-F(x^\star)\right)+2D_\Phi(x^\star,v_k)-\frac{1-2^{1-q}}{q}\lambda_{k+1}A_{k+1}\lVert\widehat{x}_{k+1}-y_k\rVert_p^p\\
&+\mu_{k+1}-A_{k+1}\frac{\epsilon_{\mathrm{opt}}}{20}+\frac{1-2^{1-q}}{2q}\sum_{i\le k+1}\lambda_iA_i\lVert\widehat{x}_i-y_{i-1}\rVert_p^p, \tag*{(49)}
\end{aligned} E [ M k + 1 ∣ F k ] ≤ A k ( F ( x k ) − F ( x ⋆ ) ) + 2 D Φ ( x ⋆ , v k ) − q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x k + 1 − y k ∥ p p + μ k + 1 − A k + 1 20 ϵ opt + 2 q 1 − 2 1 − q i ≤ k + 1 ∑ λ i A i ∥ x i − y i − 1 ∥ p p , (49)
where
μ k + 1 : = A k + 1 φ k + 1 + 2 2 − q a k + 1 q q σ k + 1 q + 2 R a k + 1 β k + 1 . \mu_{k+1}:=A_{k+1}\varphi_{k+1}+\frac{2^{2-q}a_{k+1}^q}{q}\sigma_{k+1}^q+2Ra_{k+1}\beta_{k+1}. μ k + 1 := A k + 1 φ k + 1 + q 2 2 − q a k + 1 q σ k + 1 q + 2 R a k + 1 β k + 1 .
Substituting the values of φ k + 1 \varphi_{k+1} φ k + 1 , σ k + 1 q \sigma_{k+1}^q σ k + 1 q , and β k + 1 \beta_{k+1} β k + 1 into the definition of μ k + 1 \mu_{k+1} μ k + 1 gives
μ k + 1 = λ k + 1 A k + 1 r p C p ℓ p + 1 + 2 2 − q a k + 1 q λ k + 1 q r p q C p ℓ p + 1 + 2 a k + 1 ϵ o p t C p . \mu_{k+1}=\frac{\lambda_{k+1}A_{k+1}r^p}{C_p\ell^{p+1}}+\frac{2^{2-q}a_{k+1}^q\lambda_{k+1}^qr^p}{qC_p\ell^{p+1}}+\frac{2a_{k+1}\epsilon_{\mathrm{opt}}}{C_p}. μ k + 1 = C p ℓ p + 1 λ k + 1 A k + 1 r p + q C p ℓ p + 1 2 2 − q a k + 1 q λ k + 1 q r p + C p 2 a k + 1 ϵ opt .
Recall that A k + 1 p − 1 = 2 p − 1 λ k + 1 a k + 1 p A_{k+1}^{p-1}=2^{p-1}\lambda_{k+1}a_{k+1}^p A k + 1 p − 1 = 2 p − 1 λ k + 1 a k + 1 p , therefore a k + 1 q λ k + 1 q = λ k + 1 A k + 1 / 2 a_{k+1}^q\lambda_{k+1}^q=\lambda_{k+1}A_{k+1}/2 a k + 1 q λ k + 1 q = λ k + 1 A k + 1 /2 and
μ k + 1 = 1 + 2 1 − q / q C p ℓ p + 1 λ k + 1 A k + 1 r p + 2 a k + 1 ϵ o p t C p = 2 ( p − 1 ) / p ( 1 + 2 1 − q / q ) a k + 1 λ k + 1 1 / p A k + 1 1 / p r p C p ℓ p + 1 + 2 a k + 1 ϵ o p t C p . \mu_{k+1}=\frac{1+2^{1-q}/q}{C_p\ell^{p+1}}\lambda_{k+1}A_{k+1}r^p+\frac{2a_{k+1}\epsilon_{\mathrm{opt}}}{C_p}=\frac{2^{(p-1)/p}(1+2^{1-q}/q)a_{k+1}\lambda_{k+1}^{1/p}A_{k+1}^{1/p}r^p}{C_p\ell^{p+1}}+\frac{2a_{k+1}\epsilon_{\mathrm{opt}}}{C_p}. μ k + 1 = C p ℓ p + 1 1 + 2 1 − q / q λ k + 1 A k + 1 r p + C p 2 a k + 1 ϵ opt = C p ℓ p + 1 2 ( p − 1 ) / p ( 1 + 2 1 − q / q ) a k + 1 λ k + 1 1/ p A k + 1 1/ p r p + C p 2 a k + 1 ϵ opt .
If ∥ x ~ k + 1 − y k ∥ p ≥ 3 r / 4 \lVert\tilde{x}_{k+1}-y_k\rVert_p \ge3r/4 ∥ x ~ k + 1 − y k ∥ p ≥ 3 r /4 , then for sufficiently large C p C_p C p and ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 ,
μ k + 1 ≤ 1 − 2 1 − q 2 q λ k + 1 A k + 1 ∥ x ~ k + 1 − y k ∥ p p + a k + 1 ϵ o p t 20 . \mu_{k+1} \le\frac{1-2^{1-q}}{2q}\lambda_{k+1}A_{k+1}\lVert\tilde{x}_{k+1}-y_k\rVert_p^p+\frac{a_{k+1}\epsilon_{\mathrm{opt}}}{20}. μ k + 1 ≤ 2 q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ~ k + 1 − y k ∥ p p + 20 a k + 1 ϵ opt .
Else, if λ k + 1 < 2 λ m \lambda_{k+1}<2\lambda_m λ k + 1 < 2 λ m , we have
μ k + 1 < 2 ( p − 1 ) / p ( 1 + 2 1 − q / q ) a k + 1 ( 2 λ m ) 1 + 1 / p A k + 1 1 / p r p C p ℓ p + 1 + 2 a k + 1 ϵ o p t C p . \mu_{k+1}<\frac{2^{(p-1)/p}(1+2^{1-q}/q)a_{k+1}(2\lambda_m)^{1+1/p}A_{k+1}^{1/p}r^p}{C_p\ell^{p+1}}+\frac{2a_{k+1}\epsilon_{\mathrm{opt}}}{C_p}. μ k + 1 < C p ℓ p + 1 2 ( p − 1 ) / p ( 1 + 2 1 − q / q ) a k + 1 ( 2 λ m ) 1 + 1/ p A k + 1 1/ p r p + C p 2 a k + 1 ϵ opt .
Now, note that λ m A max = C p ( R / r ) 2 p / ( p + 1 ) ℓ p ≥ 1 \lambda_m A_{\max}=C_p(R/r)^{2p/(p+1)}\ell^p\ge1 λ m A m a x = C p ( R / r ) 2 p / ( p + 1 ) ℓ p ≥ 1 for sufficiently large C p C_p C p when 0 < r ≤ R 0<r\le R 0 < r ≤ R and ℓ ≥ 1 \ell\ge1 ℓ ≥ 1 , so the NextLambda lower bound λ k ≥ λ m \lambda_k\ge\lambda_m λ k ≥ λ m implies λ k ≥ 1 / A max \lambda_k\ge1/A_{\max} λ k ≥ 1/ A m a x . From the definition of the stopping time, A K − 1 ≤ A max A_{K-1}\le A_{\max} A K − 1 ≤ A m a x . If a K > A max a_K>A_{\max} a K > A m a x , then A K = A K − 1 + a K ≤ 2 a K A_K=A_{K-1}+a_K\le2a_K A K = A K − 1 + a K ≤ 2 a K , and the weight equation gives
a K p = A K p − 1 2 p − 1 λ K ≤ ( 2 a K ) p − 1 A max 2 p − 1 = a K p − 1 A max , a_K^p=\frac{A_K^{p-1}}{2^{p-1}\lambda_K}\le\frac{(2a_K)^{p-1}A_{\max}}{2^{p-1}}=a_K^{p-1}A_{\max}, a K p = 2 p − 1 λ K A K p − 1 ≤ 2 p − 1 ( 2 a K ) p − 1 A m a x = a K p − 1 A m a x ,
a contradiction. Therefore a K ≤ A max a_K\le A_{\max} a K ≤ A m a x , and A k ≤ A K ≤ 2 A max A_k\le A_K\le2A_{\max} A k ≤ A K ≤ 2 A m a x for every k ≤ K k\le K k ≤ K . Thus, when λ k + 1 < 2 λ m \lambda_{k+1}<2\lambda_m λ k + 1 < 2 λ m , the bound on μ k + 1 \mu_{k+1} μ k + 1 becomes
μ k + 1 < 2 ( p − 1 ) / p ( 1 + 2 1 − q / q ) a k + 1 ( 2 λ m ) 1 + 1 / p ( 2 A max ) 1 / p r p C p ℓ p + 1 + 2 a k + 1 ϵ o p t C p = a k + 1 ϵ o p t ( 4 ⋅ 2 1 / p ( 1 + 2 1 − q / q ) C p ( p − 1 ) / p + 2 C p ) ≤ a k + 1 ϵ o p t 20 ≤ 1 − 2 1 − q 2 q λ k + 1 A k + 1 ∥ x ~ k + 1 − y k ∥ p p + a k + 1 ϵ o p t 20 , \begin{aligned}
\mu_{k+1}<\frac{2^{(p-1)/p}(1+2^{1-q}/q)a_{k+1}(2\lambda_m)^{1+1/p}(2A_{\max})^{1/p}r^p}{C_p\ell^{p+1}}+\frac{2a_{k+1}\epsilon_{\mathrm{opt}}}{C_p} \\
&=a_{k+1}\epsilon_{\mathrm{opt}}\left(\frac{4\cdot2^{1/p}(1+2^{1-q}/q)}{C_p^{(p-1)/p}}+\frac{2}{C_p}\right) \\
&\le a_{k+1}\frac{\epsilon_{\mathrm{opt}}}{20}\le\frac{1-2^{1-q}}{2q}\lambda_{k+1}A_{k+1}\lVert\tilde{x}_{k+1}-y_k\rVert_p^p+a_{k+1}\frac{\epsilon_{\mathrm{opt}}}{20},
\end{aligned} μ k + 1 < C p ℓ p + 1 2 ( p − 1 ) / p ( 1 + 2 1 − q / q ) a k + 1 ( 2 λ m ) 1 + 1/ p ( 2 A m a x ) 1/ p r p + C p 2 a k + 1 ϵ opt = a k + 1 ϵ opt ( C p ( p − 1 ) / p 4 ⋅ 2 1/ p ( 1 + 2 1 − q / q ) + C p 2 ) ≤ a k + 1 20 ϵ opt ≤ 2 q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ~ k + 1 − y k ∥ p p + a k + 1 20 ϵ opt ,
where the penultimate inequality holds for sufficiently large C p C_p C p . Therefore, for every k < K k<K k < K , we have
μ k + 1 ≤ 1 − 2 1 − q 2 q λ k + 1 A k + 1 ∥ x ~ k + 1 − y k ∥ p p + a k + 1 ϵ o p t 20 . \mu_{k+1}\le\frac{1-2^{1-q}}{2q}\lambda_{k+1}A_{k+1}\lVert\tilde{x}_{k+1}-y_k\rVert_p^p+a_{k+1}\frac{\epsilon_{\mathrm{opt}}}{20}. μ k + 1 ≤ 2 q 1 − 2 1 − q λ k + 1 A k + 1 ∥ x ~ k + 1 − y k ∥ p p + a k + 1 20 ϵ opt .
Noting that E ∣ M k ∣ < ∞ \mathbb{E}|M_k|<\infty E ∣ M k ∣ < ∞ and substituting the bound on μ k + 1 \mu_{k+1} μ k + 1 into (22) , we get
E [ M k + 1 ∣ F k ] ≤ A k ( F ( x k ) − F ( x ⋆ ) − ϵ o p t 20 ) + 2 D Φ ( x ⋆ , v k ) + 1 − 2 1 − q 2 q ∑ i ≤ k λ i A i ∥ x ~ i − y i − 1 ∥ p p = M k . \mathbb{E}[M_{k+1}\mid\mathcal{F}_k]\le A_k\left(F(x_k)-F(x^\star)-\frac{\epsilon_{\mathrm{opt}}}{20}\right)+2D_\Phi(x^\star,v_k)+\frac{1-2^{1-q}}{2q}\sum_{i\le k}\lambda_iA_i\lVert\tilde{x}_i-y_{i-1}\rVert_p^p=M_k. E [ M k + 1 ∣ F k ] ≤ A k ( F ( x k ) − F ( x ⋆ ) − 20 ϵ opt ) + 2 D Φ ( x ⋆ , v k ) + 2 q 1 − 2 1 − q i ≤ k ∑ λ i A i ∥ x ~ i − y i − 1 ∥ p p = M k .
Therefore M k M_k M k is a supermartingale adapted to F k \mathcal{F}_k F k . Since K ≤ K max K\le K_{\max} K ≤ K m a x is a bounded stopping time adapted to F k \mathcal{F}_k F k , by the optional stopping theorem for supermartingales we have
E M K ≤ M 0 = A 0 ( F ( x 0 ) − F ( x ⋆ ) − ϵ o p t 20 ) + 2 D Φ ( x ⋆ , v 0 ) ≤ A 0 ( F ( x 0 ) − F ( x ⋆ ) ) + C p R p , \begin{aligned}
\mathbb{E}M_K\le M_0=A_0\left(F(x_0)-F(x^\star)-\frac{\epsilon_{\mathrm{opt}}}{20}\right)+2D_\Phi(x^\star,v_0) \\
&\le A_0(F(x_0)-F(x^\star))+C_pR^p,
\end{aligned} E M K ≤ M 0 = A 0 ( F ( x 0 ) − F ( x ⋆ ) − 20 ϵ opt ) + 2 D Φ ( x ⋆ , v 0 ) ≤ A 0 ( F ( x 0 ) − F ( x ⋆ )) + C p R p ,
where v 0 = x 0 v_0=x_0 v 0 = x 0 , and 2 D Φ ( x ⋆ , x 0 ) ≤ C p R p 2D_\Phi(x^\star,x_0)\le C_pR^p 2 D Φ ( x ⋆ , x 0 ) ≤ C p R p holds when the mirror potential is centered at x 0 x_0 x 0 (or equivalently after translating x 0 x_0 x 0 to the origin). This completes the proof.
Proposition 11. Let F : R d → R F:\mathbb{R}^d\to\mathbb{R} F : R d → R be convex and L L L -Lipschitz with respect to ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p , and let x ⋆ ∈ B p ( R ) x^\star\in B_p(R) x ⋆ ∈ B p ( R ) . There is an implementation of BallAccel (see Proposition 10) taking parameters r ∈ ( 0 , R ] r\in(0,R] r ∈ ( 0 , R ] and ϵ o p t ∈ ( 0 , L R / e ] \epsilon_{\mathrm{opt}}\in(0,LR/e] ϵ opt ∈ ( 0 , L R / e ] with the following guarantee. Define κ \kappa κ , K K K , and λ ⋆ \lambda^\star λ ⋆ as in Proposition 10. For a sufficiently large constant C b a > 0 C_{\mathrm{ba}}>0 C ba > 0 depending only on p p p , BallAccel runs in at most C b a K log κ log ( R κ r ) C_{\mathrm{ba}}K\log\kappa\log\left(\frac{R\kappa}{r}\right) C ba K log κ log ( r R κ ) iterations and produces x ∈ R d x \in\mathbb{R}^{d} x ∈ R d such that E F ( x ) ≤ F ( x ⋆ ) + ϵ o p t \mathbb{E}F(x) \leq F(x^\star) + \epsilon_{\mathrm{opt}} E F ( x ) ≤ F ( x ⋆ ) + ϵ opt . For general F F F , the ball optimization oracles below are applied to
x ↦ F ( x ) + L ( ∥ x ∥ p − R ) + . x \mapsto F(x) + L(\lVert x\rVert_p - R)_+. x ↦ F ( x ) + L (∥ x ∥ p − R ) + .
Their calls satisfy:
Each iteration makes at most C b a log 2 ( R κ r ) C_{\mathrm{ba}}\log^2\left(\frac{R\kappa}{r}\right) C ba log 2 ( r R κ ) calls to a ( λ r p C b a , λ ) \left(\frac{\lambda r^p}{C_{\mathrm{ba}}},\lambda\right) ( C ba λ r p , λ ) -ball optimization oracle, with λ ∈ [ λ ⋆ C b a , C b a L r p − 1 ] \lambda\in\left[\frac{\lambda^\star}{C_{\mathrm{ba}}},\frac{C_{\mathrm{ba}}L}{r^{p-1}}\right] λ ∈ [ C ba λ ⋆ , r p − 1 C ba L ] .
For each j ∈ [ ⌈ log 2 K + C b a ⌉ ] j \in\left[\lceil\log_2 K+C_{\mathrm{ba}}\rceil\right] j ∈ [ ⌈ log 2 K + C ba ⌉ ] , at most C b a 2 2 − j K log κ log ( R κ r ) C_{\mathrm{ba}}^2 2^{-j}K\log\kappa\log\left(\frac{R\kappa}{r}\right) C ba 2 2 − j K log κ log ( r R κ ) iterations query a ( λ r p C p 2 j log 2 p ( R κ / r ) , λ ) \left(\frac{\lambda r^p}{C_p2^j\log^{2p}(R\kappa/r)},\lambda\right) ( C p 2 j l o g 2 p ( R κ / r ) λ r p , λ ) -ball optimization oracle, for some λ ∈ [ λ ⋆ C b a , C b a L r p − 1 ] \lambda\in\left[\frac{\lambda^\star}{C_{\mathrm{ba}}},\frac{C_{\mathrm{ba}}L}{r^{p-1}}\right] λ ∈ [ C ba λ ⋆ , r p − 1 C ba L ] .
Moreover, if F F F admits a global minimizer in B p ( R ) \mathbb{B}_p(R) B p ( R ) , the same guarantees and oracle-call bounds hold for an implementation whose ball optimization oracles are applied directly to F F F , without adding a radial penalty and without projecting the final output.
Proof. We follow the organization of Appendix C of [15 ] . For the general implementation, it suffices to take
x ⋆ ∈ arg min x ∈ B p ( R ) F ( x ) . x^\star\in\mathop{\arg\min}_{x\in\mathbb{B}_p(R)} F(x). x ⋆ ∈ arg min x ∈ B p ( R ) F ( x ) .
Apply the method to x ↦ F ( x ) + L ( ∥ x ∥ p − R ) + x \mapsto F(x) + L(\lVert x\rVert_p - R)_+ x ↦ F ( x ) + L (∥ x ∥ p − R ) + . This function is convex and 2 L 2L 2 L -Lipschitz, and
F ( x ) + L ( ∥ x ∥ p − R ) + ≥ F ( x max { 1 , ∥ x ∥ p / R } ) ≥ F ( x ⋆ ) . F(x) + L(\lVert x\rVert_p - R)_+ \geq F\left(\frac{x}{\max\{1,\lVert x\rVert_p/R\}}\right) \geq F(x^\star). F ( x ) + L (∥ x ∥ p − R ) + ≥ F ( max { 1 , ∥ x ∥ p / R } x ) ≥ F ( x ⋆ ) .
Thus its objective error is nonnegative everywhere, and projecting the final output onto B p ( R ) \mathbb{B}_p(R) B p ( R ) gives the required guarantee for F F F . Replacing L L L by 2 L 2L 2 L changes the bounds only by constant factors. For a stochastic-subgradient implementation, add a subgradient of L ( ∥ x ∥ p − R ) + L(\lVert x\rVert_p - R)_+ L (∥ x ∥ p − R ) + to the original oracle. Its dual norm is at most L L L , so Minkowski’s inequality bounds the resulting q q q -th moment by ( 2 L ) q (2L)^q ( 2 L ) q . Initialize x 0 = v 0 = 0 x_0=v_0=0 x 0 = v 0 = 0 , using Lipschitz bound 2 L 2L 2 L in the initialization of A 0 A_0 A 0 , and retain
A i = A i − 1 + a i , A i p − 1 = 2 p − 1 λ i a i p . A_i=A_{i-1}+a_i,\qquad A_i^{p-1}=2^{p-1}\lambda_i a_i^p. A i = A i − 1 + a i , A i p − 1 = 2 p − 1 λ i a i p .
Use the mirror update over B p ( R ) \mathbb{B}_p(R) B p ( R ) and define
x ^ i = arg min x ∈ R d { F ( x ) + L ( ∥ x ∥ p − R ) + + λ i p ∥ x − y i − 1 ∥ p p } . \widehat{x}_i=\mathop{\arg\min}_{x\in\mathbb{R}^d}\left\{F(x)+L(\lVert x\rVert_p-R)_+ + \frac{\lambda_i}{p}\lVert x-y_{i-1}\rVert_p^p\right\}. x i = arg min x ∈ R d { F ( x ) + L (∥ x ∥ p − R ) + + p λ i ∥ x − y i − 1 ∥ p p } .
Lemma 17 applies with the same 2 R a i β i 2Ra_i\beta_i 2 R a i β i term: its radius bound is needed only for x ⋆ x^\star x ⋆ and the mirror iterates, while x ^ i \widehat{x}_i x i minimizes the objective over R d \mathbb{R}^d R d . Use oracle implementations that return points in their queried radius-r r r balls for the primal, line-search, and MorGradEst calls. Each execution stops at the first completed iteration with A i ≥ A max A_i\geq A_{\max} A i ≥ A m a x , or after ⌈ C p K log ( R κ / r ) ⌉ \lceil C_pK\log(R\kappa/r)\rceil ⌈ C p K log ( R κ / r )⌉ iterations, where
A max = C b a R p ϵ o p t . A_{\max}=\frac{C_{\mathrm{ba}}R^p}{\epsilon_{\mathrm{opt}}}. A m a x = ϵ opt C ba R p .
Here C b a C_{\mathrm{ba}} C ba is sufficiently large and depends only on p p p . Write τ \tau τ for this stopping index. An execution is accepted only if A τ ≥ A max A_\tau\geq A_{\max} A τ ≥ A m a x . We first bound the calls required in each iteration. The primal updates and the mirror constraint give
∥ x i ∥ p ≤ R + i r , ∥ x i − v i ∥ p ≤ 2 R + i r . \lVert x_i\rVert_p\leq R+ir,\qquad\lVert x_i-v_i\rVert_p\leq2R+ir. ∥ x i ∥ p ≤ R + i r , ∥ x i − v i ∥ p ≤ 2 R + i r .
Each execution has O ( K log ( R κ / r ) ) O(K\log(R\kappa/r)) O ( K log ( R κ / r )) iterations. Since ∥ x i − v i ∥ p ≤ 2 R + i r \lVert x_i-v_i\rVert_p\le2R+ir ∥ x i − v i ∥ p ≤ 2 R + i r and K r ≤ R Kr\le R K r ≤ R , we have
∥ x i − v i ∥ p ≤ 2 R + O ( K r log ( R κ / r ) ) = O ( R log ( R κ / r ) ) . \lVert x_i-v_i\rVert_p\le2R+O(Kr\log(R\kappa/r))=O(R\log(R\kappa/r)). ∥ x i − v i ∥ p ≤ 2 R + O ( K r log ( R κ / r )) = O ( R log ( R κ / r )) .
Using this bound in Theorem 8 gives O ( log ( R κ / r ) ) O(\log(R\kappa/r)) O ( log ( R κ / r )) distance tests per iteration. The mirror iterates remain in B p ( R ) B_p(R) B p ( R ) .
For each test, use independent calls with expected objective error λ r p / C p \lambda r^p/C_p λ r p / C p . By Lemma 20 , O ( log ( R κ / r ) ) O(\log(R\kappa/r)) O ( log ( R κ / r )) repetitions followed by Aggregate make the conditional failure probability at most
1 C p κ K ⌈ log 2 ( 128 κ ) ⌉ log 2 ( R κ / r ) . \frac{1}{C_p\kappa K\lceil\log_2(128\kappa)\rceil\log^2(R\kappa/r)}. C p κ K ⌈ log 2 ( 128 κ )⌉ log 2 ( R κ / r ) 1 .
A union bound over all tests and executions gives total failure probability at most 1 / ( C p κ ) 1/(C_p\kappa) 1/ ( C p κ ) . Abort an execution before exceeding the stated test budget. When all tests are accurate, this budget is sufficient. This proves item 1 without function-value queries.
For the analysis, consider an execution without the caps on oracle calls, in which inaccurate distance tests are replaced by their exact outcomes. Fresh primal and MorGradEst randomness retains its conditional guarantees; no conditioning on future test accuracy or future caps is used. Theorem 8 then gives
∥ x ^ i − y i − 1 ∥ p ≤ r , λ i < 2 λ min or ∥ x ^ i − y i − 1 ∥ p ≥ 3 r 4 . \lVert\widehat{x}_i-y_{i-1}\rVert_p\le r,\qquad\lambda_i<2\lambda_{\min}\quad\text{or}\quad\lVert\widehat{x}_i-y_{i-1}\rVert_p\ge\frac{3r}{4}. ∥ x i − y i − 1 ∥ p ≤ r , λ i < 2 λ m i n or ∥ x i − y i − 1 ∥ p ≥ 4 3 r .
where λ min \lambda_{\min} λ m i n differs from λ ⋆ \lambda^\star λ ⋆ by a constant factor. For the additional calls, use the construction of Lemma 21 with
j max = ⌈ log 2 K ⌉ , T 0 = C b a 4 p log 2 p ( R κ / r ) , δ j = r ( 2 j T 0 ) 1 / p . j_{\max}=\lceil\log_2 K\rceil,\qquad T_0=C_{\mathrm{ba}}^{4p}\log^{2p}(R\kappa/r),\qquad\delta_j=\frac{r}{(2^jT_0)^{1/p}}. j m a x = ⌈ log 2 K ⌉ , T 0 = C ba 4 p log 2 p ( R κ / r ) , δ j = ( 2 j T 0 ) 1/ p r .
That lemma and an independent primal call give
β i ≤ C p λ i r p − 1 K C b a 4 p log p ( R κ / r ) , \beta_i\le\frac{C_p\lambda_i r^{p-1}}{\sqrt{K C_{\mathrm{ba}}^{4p}\log^p(R\kappa/r)}}, β i ≤ K C ba 4 p log p ( R κ / r ) C p λ i r p − 1 ,
σ i q ≤ C p λ i q r p C b a 2 p q log p q ( R κ / r ) ( 1 + ∑ j = 1 ⌈ log 2 K ⌉ 2 − j ( 1 − q / 2 ) ) , \sigma_i^q\le\frac{C_p\lambda_i^q r^p}{C_{\mathrm{ba}}^{2pq}\log^{pq}(R\kappa/r)}
\left(1+\sum_{j=1}^{\lceil\log_2 K\rceil}2^{-j(1-q/2)}\right), σ i q ≤ C ba 2 pq log pq ( R κ / r ) C p λ i q r p 1 + j = 1 ∑ ⌈ l o g 2 K ⌉ 2 − j ( 1 − q /2 ) ,
φ i ≤ C p λ i r p C b a 4 p log 2 p ( R κ / r ) . \varphi_i\le\frac{C_p\lambda_i r^p}{C_{\mathrm{ba}}^{4p}\log^{2p}(R\kappa/r)}. φ i ≤ C ba 4 p log 2 p ( R κ / r ) C p λ i r p .
The sum in the second bound is bounded by a constant depending only on p p p when p > 2 p>2 p > 2 , and is O ( log ( R κ / r ) ) O(\log(R\kappa/r)) O ( log ( R κ / r )) when p = 2 p=2 p = 2 . Using a i q λ i q = λ i A i / 2 a_i^q\lambda_i^q=\lambda_iA_i/2 a i q λ i q = λ i A i /2 and Young’s inequality, we obtain, for any sufficiently small fixed c p > 0 c_p>0 c p > 0 and sufficiently large C b a C_{\mathrm{ba}} C ba ,
A i φ i + 2 2 − q a i q q σ i q + 2 R a i β i ≤ c p λ i A i r p log p + 1 ( R κ / r ) + C p R p C b a 2 p K p / 2 log ( R κ / r ) . A_i\varphi_i+\frac{2^{2-q}a_i^q}{q}\sigma_i^q+2Ra_i\beta_i\le\frac{c_p\lambda_iA_i r^p}{\log^{p+1}(R\kappa/r)}+\frac{C_pR^p}{C_{\mathrm{ba}}^{2p}K^{p/2}\log(R\kappa/r)}. A i φ i + q 2 2 − q a i q σ i q + 2 R a i β i ≤ log p + 1 ( R κ / r ) c p λ i A i r p + C ba 2 p K p /2 log ( R κ / r ) C p R p .
If ∥ x ^ i − y i − 1 ∥ p ≥ 3 r / 4 \lVert\widehat{x}_i-y_{i-1}\rVert_p\ge3r/4 ∥ x i − y i − 1 ∥ p ≥ 3 r /4 , the first term is absorbed by a fixed fraction of the negative displacement term. Otherwise λ i < 2 λ min \lambda_i<2\lambda_{\min} λ i < 2 λ m i n . The final-increment argument gives A τ ≤ 2 A max A_\tau\le2A_{\max} A τ ≤ 2 A m a x , and K p + 1 r p = R p K^{p+1}r^p=R^p K p + 1 r p = R p implies
λ i A i r p a i log p + 1 ( R κ / r ) ≤ C p C b a 1 / p ϵ o p t ( log κ log ( R κ / r ) ) p + 1 ≤ C p C b a 1 / p ϵ o p t . \frac{\lambda_iA_i r^p}{a_i\log^{p+1}(R\kappa/r)}
\le C_pC_{\mathrm{ba}}^{1/p}\epsilon_{\mathrm{opt}}
\left(\frac{\log\kappa}{\log(R\kappa/r)}\right)^{p+1}
\le C_pC_{\mathrm{ba}}^{1/p}\epsilon_{\mathrm{opt}}. a i log p + 1 ( R κ / r ) λ i A i r p ≤ C p C ba 1/ p ϵ opt ( log ( R κ / r ) log κ ) p + 1 ≤ C p C ba 1/ p ϵ opt .
Choose c p c_p c p sufficiently small, using ∑ i ≤ τ a i ≤ 2 A max \sum_{i\leq\tau} a_i \leq2A_{\max} ∑ i ≤ τ a i ≤ 2 A m a x , and then choose C b a C_{\mathrm{ba}} C ba sufficiently large, using τ ≤ ⌈ C p K log ( R κ / r ) ⌉ \tau\leq\lceil C_pK\log(R\kappa/r)\rceil τ ≤ ⌈ C p K log ( R κ / r )⌉ and K 1 − p / 2 ≤ 1 K^{1-p/2}\leq1 K 1 − p /2 ≤ 1 . The accumulated errors are then bounded by a fixed multiple of R p R^p R p . Applying bounded stopping to the conditional potential inequality gives
E [ A τ ( F ( x τ ) + L ( ∥ x τ ∥ p − R ) + − F ( x ∗ ) ) + 1 − 2 1 − q 2 q ∑ i = 1 τ λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ C 0 , p R p . (50) \mathbb{E}\left[A_\tau\left(F(x_\tau)+L(\lVert x_\tau\rVert_p-R)_+-F(x^*)\right)+\frac{1-2^{1-q}}{2q}\sum_{i=1}^{\tau}\lambda_iA_i\lVert\widehat{x}_i-y_{i-1}\rVert_p^p\right]\leq C_{0,p}R^p. \tag*{(50)} E [ A τ ( F ( x τ ) + L (∥ x τ ∥ p − R ) + − F ( x ∗ ) ) + 2 q 1 − 2 1 − q i = 1 ∑ τ λ i A i ∥ x i − y i − 1 ∥ p p ] ≤ C 0 , p R p . (50)
Both terms on the left are nonnegative. Since the error constants are chosen after C b a C_{\mathrm{ba}} C ba and the iteration budget, C 0 , p C_{0,p} C 0 , p can be fixed independently of those choices.
We next show that an execution is accepted with constant probability. On { A τ < A max } \{A_\tau<A_{\max}\} { A τ < A m a x } , τ = ⌈ C p K log ( R κ / r ) ⌉ \tau=\lceil C_pK\log(R\kappa/r)\rceil τ = ⌈ C p K log ( R κ / r )⌉ . If at least half the iterations have λ i < 2 λ min \lambda_i<2\lambda_{\min} λ i < 2 λ m i n , the weight identity gives
A τ 1 / p − A 0 1 / p ≥ 1 p 2 ( p − 1 ) / p ∑ i = 1 τ λ i − 1 / p ≥ τ C p λ min 1 / p . A_\tau^{1/p}-A_0^{1/p}\geq\frac{1}{p2^{(p-1)/p}}\sum_{i=1}^{\tau}\lambda_i^{-1/p}\geq\frac{\tau}{C_p\lambda_{\min}^{1/p}}. A τ 1/ p − A 0 1/ p ≥ p 2 ( p − 1 ) / p 1 i = 1 ∑ τ λ i − 1/ p ≥ C p λ m i n 1/ p τ .
For a sufficiently large iteration budget, this implies A τ ≥ A max A_\tau\geq A_{\max} A τ ≥ A m a x , a contradiction. Thus at least half the iterations have displacement at least 3 r / 4 3r/4 3 r /4 . Using
λ i A i = 1 2 p − 1 ( A i a i ) p , ∑ i = 1 τ a i A i ≤ log A τ A 0 ≤ C p log ( R κ / r ) , \lambda_iA_i=\frac{1}{2^{p-1}}\left(\frac{A_i}{a_i}\right)^p,\qquad\sum_{i=1}^{\tau}\frac{a_i}{A_i}\leq\log\frac{A_\tau}{A_0}\leq C_p\log(R\kappa/r), λ i A i = 2 p − 1 1 ( a i A i ) p , i = 1 ∑ τ A i a i ≤ log A 0 A τ ≤ C p log ( R κ / r ) ,
Hölder’s inequality yields
∑ i = 1 τ λ i A i ∥ x ^ i − y i − 1 ∥ p p ≥ ( 3 r / 4 ) p 2 p − 1 ( τ / 2 ) p + 1 ( ∑ i = 1 τ a i / A i ) p ≥ c p r p τ p + 1 log p ( R κ / r ) . \sum_{i=1}^{\tau}\lambda_iA_i\lVert\widehat{x}_i-y_{i-1}\rVert_p^p\geq\frac{(3r/4)^p}{2^{p-1}}\frac{(\tau/2)^{p+1}}{\left(\sum_{i=1}^{\tau}a_i/A_i\right)^p}\geq\frac{c_pr^p\tau^{p+1}}{\log^p(R\kappa/r)}. i = 1 ∑ τ λ i A i ∥ x i − y i − 1 ∥ p p ≥ 2 p − 1 ( 3 r /4 ) p ( ∑ i = 1 τ a i / A i ) p ( τ /2 ) p + 1 ≥ log p ( R κ / r ) c p r p τ p + 1 .
Increasing the iteration budget makes the displacement term in (50) exceed 4 C 0 , p R p 4C_{0,p}R^p 4 C 0 , p R p on { A τ < A max } \{A_\tau<A_{\max}\} { A τ < A m a x } . Markov’s inequality therefore gives
Pr ( A τ ≥ A max ) ≥ 3 4 . \Pr(A_\tau\geq A_{\max})\geq\frac{3}{4}. Pr ( A τ ≥ A m a x ) ≥ 4 3 .
The same potential bound gives
E [ ( F ( x τ ) + L ( ∥ x τ ∥ p − R ) + − F ( x ∗ ) ) 1 { A τ ≥ A max } ] ≤ C 0 , p R p A max ≤ ϵ o p t 64 , \mathbb{E}\left[(F(x_\tau)+L(\lVert x_\tau\rVert_p-R)_+-F(x^*))\mathbf{1}_{\{A_\tau\geq A_{\max}\}}\right]\leq\frac{C_{0,p}R^p}{A_{\max}}\leq\frac{\epsilon_{\mathrm{opt}}}{64}, E [ ( F ( x τ ) + L (∥ x τ ∥ p − R ) + − F ( x ∗ )) 1 { A τ ≥ A m a x } ] ≤ A m a x C 0 , p R p ≤ 64 ϵ opt ,
where C b a ≥ 64 C 0 , p C_{\mathrm{ba}}\geq64C_{0,p} C ba ≥ 64 C 0 , p .
To enforce deterministic bounds on the additional calls, pre-sample independent geometric levels for all potential iterations, revealing each only when its iteration is reached. Level 0 is always used. For t ≥ 1 t\geq1 t ≥ 1 , a level-t t t call occurs only when the sampled level is t t t or t + 1 t+1 t + 1 , with probability O ( 2 − t ) O(2^{-t}) O ( 2 − t ) . A Chernoff bound and a union bound over all levels and executions give the per-execution bounds
C p 2 − t K log ( R κ / r ) , 0 ≤ t ≤ ⌈ log 2 K ⌉ , C_p2^{-t}K\log(R\kappa/r),\qquad0\leq t\leq\lceil\log_2K\rceil, C p 2 − t K log ( R κ / r ) , 0 ≤ t ≤ ⌈ log 2 K ⌉ ,
except with total probability at most 1 / ( C p κ ) 1/(C_p\kappa) 1/ ( C p κ ) . Abort before exceeding a bound and treat aborted executions as unsuccessful. Combining this event with the distance-test events, the implemented and analyzed executions agree except with probability at most 1 / ( C p κ ) 1/(C_p\kappa) 1/ ( C p κ ) , after increasing constants.
Project each output onto B p ( R ) \mathbb{B}_p(R) B p ( R ) . Its excess loss lies in [ 0 , 2 L R ] [0,2LR] [ 0 , 2 L R ] , so this comparison adds at most 2 L R / ( C p κ ) 2LR/(C_p\kappa) 2 L R / ( C p κ ) to the expected excess loss. Consequently, each implemented execution satisfies
Pr ( A τ ≥ A max ) ≥ 1 2 , \Pr(A_\tau\ge A_{\max}) \ge\frac{1}{2}, Pr ( A τ ≥ A m a x ) ≥ 2 1 ,
E [ ( F ( x τ max { 1 , ∥ x τ ∥ p / R } ) − F ( x ⋆ ) ) 1 { A τ ≥ A max } ] ≤ ϵ o p t 32 . \mathbb{E}\left[\left(F\left(\frac{x_\tau}{\max\{1,\lVert x_\tau\rVert_p/R\}}\right)-F(x^\star)\right)\mathbf{1}_{\{A_\tau\ge A_{\max}\}}\right]\le\frac{\epsilon_{\mathrm{opt}}}{32}. E [ ( F ( max { 1 , ∥ x τ ∥ p / R } x τ ) − F ( x ⋆ ) ) 1 { A τ ≥ A m a x } ] ≤ 32 ϵ opt .
Independently restart from 0 0 0 until the first accepted execution and return its projected output. After ⌈ log 2 ( 128 κ ) ⌉ \lceil\log_2(128\kappa)\rceil ⌈ log 2 ( 128 κ )⌉ unsuccessful executions, return 0 0 0 . This selection uses no objective values, and
E [ F ( x ) − F ( x ⋆ ) ] ≤ ∑ b = 1 ⌈ log 2 ( 128 κ ) ⌉ 2 − ( b − 1 ) ϵ o p t 32 + L R 2 − ⌈ log 2 ( 128 κ ) ⌉ < ϵ o p t . \mathbb{E}[F(x)-F(x^\star)]\le\sum_{b=1}^{\lceil\log_2(128\kappa)\rceil}2^{-(b-1)}\frac{\epsilon_{\mathrm{opt}}}{32}+LR\,2^{-\lceil\log_2(128\kappa)\rceil}<\epsilon_{\mathrm{opt}}. E [ F ( x ) − F ( x ⋆ )] ≤ b = 1 ∑ ⌈ l o g 2 ( 128 κ )⌉ 2 − ( b − 1 ) 32 ϵ opt + L R 2 − ⌈ l o g 2 ( 128 κ )⌉ < ϵ opt .
This proves the guarantee for every comparator in B p ( R ) \mathbb{B}_p(R) B p ( R ) . Finally, the total iteration budget is
⌈ log 2 ( 128 κ ) ⌉ ⌈ C p K log ( R κ / r ) ⌉ = O ( K log κ log ( R κ / r ) ) . \lceil\log_2(128\kappa)\rceil\lceil C_pK\log(R\kappa/r)\rceil=O(K\log\kappa\log(R\kappa/r)). ⌈ log 2 ( 128 κ )⌉ ⌈ C p K log ( R κ / r )⌉ = O ( K log κ log ( R κ / r )) .
Index the primal and level-0 calls by j = 1 j=1 j = 1 , and level t ≥ 1 t\ge1 t ≥ 1 by j = t + 1 j=t+1 j = t + 1 . The total number of iterations querying level j j j is
O ( 2 − j K log κ log ( R κ / r ) ) , O\left(2^{-j}K\log\kappa\log(R\kappa/r)\right), O ( 2 − j K log κ log ( R κ / r ) ) ,
with objective accuracy
λ p C p 2 j log 2 p ( R κ / r ) . \frac{\lambda^p}{C_p^2j\log^{2p}(R\kappa/r)}. C p 2 j log 2 p ( R κ / r ) λ p .
We now prove the additional claim. Choose
x ⋆ ∈ arg min x ∈ R d F ( x ) ∩ B p ( R ) . x^\star\in\mathop{\arg\min}_{x\in\mathbb{R}^d}F(x)\cap\mathbb{B}_p(R). x ⋆ ∈ arg min x ∈ R d F ( x ) ∩ B p ( R ) .
Run the same construction directly on F F F , initialize A 0 A_0 A 0 using its Lipschitz bound L L L , and define
x ^ i = arg min x ∈ R d { F ( x ) + λ i p ∥ x − y i − 1 ∥ p p } . \widehat{x}_i=\mathop{\arg\min}_{x\in\mathbb{R}^d}\left\{F(x)+\frac{\lambda_i}{p}\lVert x-y_{i-1}\rVert_p^p\right\}. x i = arg min x ∈ R d { F ( x ) + p λ i ∥ x − y i − 1 ∥ p p } .
All local oracle calls are now for F F F itself. The oracle-call estimates and the conditional potential analysis remain unchanged up to constants depending only on p p p . Since x ⋆ x^\star x ⋆ is a global minimizer, F ( x ) − F ( x ⋆ ) ≥ 0 F(x)-F(x^\star)\ge0 F ( x ) − F ( x ⋆ ) ≥ 0 for every x ∈ R d x\in\mathbb{R}^d x ∈ R d . Let
N = ⌈ C p K log ( R κ / r ) ⌉ . N=\lceil C_pK\log(R\kappa/r)\rceil. N = ⌈ C p K log ( R κ / r )⌉ .
For the analyzed execution with exact distance tests, the same bounded-stopping argument gives
E [ A τ ( F ( x τ ) − F ( x ⋆ ) ) + 1 − 2 1 − q 2 q ∑ i = 1 τ λ i A i ∥ x ^ i − y i − 1 ∥ p p ] ≤ C 0 , p R p . \mathbb{E}\left[A_\tau\left(F(x_\tau)-F(x^\star)\right)+\frac{1-2^{1-q}}{2q}\sum_{i=1}^{\tau}\lambda_iA_i\lVert\widehat{x}_i-y_{i-1}\rVert_p^p\right]\le C_{0,p}R^p. E [ A τ ( F ( x τ ) − F ( x ⋆ ) ) + 2 q 1 − 2 1 − q i = 1 ∑ τ λ i A i ∥ x i − y i − 1 ∥ p p ] ≤ C 0 , p R p .
Both terms are nonnegative. The preceding acceptance argument therefore gives
Pr ( A τ ≥ A max ) ≥ 3 4 \Pr(A_\tau\ge A_{\max})\ge\frac{3}{4} Pr ( A τ ≥ A m a x ) ≥ 4 3
and, for sufficiently large C b a C_{\mathrm{ba}} C ba ,
E [ ( F ( x τ ) − F ( x ⋆ ) ) 1 { A τ ≥ A max } ] ≤ ϵ o p t 64 . \mathbb{E}\left[(F(x_{\tau})-F(x^\star))\mathbf{1}_{\{A_{\tau}\ge A_{\max}\}}\right]\le\frac{\epsilon_{\mathrm{opt}}}{64}. E [ ( F ( x τ ) − F ( x ⋆ )) 1 { A τ ≥ A m a x } ] ≤ 64 ϵ opt .
For both the implemented and analyzed executions, ball-feasibility of the primal oracle outputs implies ∥ x i ∥ p ≤ R + i r \|x_i\|_p\le R+ir ∥ x i ∥ p ≤ R + i r for every completed iteration. Consequently, for every possible output with τ ≤ N \tau\le N τ ≤ N ,
0 ≤ F ( x τ ) − F ( x ⋆ ) ≤ L ∥ x τ − x ⋆ ∥ p ≤ L ( 2 R + N r ) . 0\le F(x_{\tau})-F(x^\star)\le L\|x_{\tau}-x^\star\|_p\le L(2R+Nr). 0 ≤ F ( x τ ) − F ( x ⋆ ) ≤ L ∥ x τ − x ⋆ ∥ p ≤ L ( 2 R + N r ) .
Strengthen the coupling failure probability over all planned executions to at most
1 C p κ ( 2 + N r / R ) . \frac{1}{C_p\kappa(2+Nr/R)}. C p κ ( 2 + N r / R ) 1 .
This preserves the stated asymptotic oracle-call budgets. Indeed, K r ≤ R Kr\le R K r ≤ R implies
2 + N r / R = O ( log ( R κ / r ) ) , 2+Nr/R=O(\log(R\kappa/r)), 2 + N r / R = O ( log ( R κ / r )) ,
so the additional logarithm in the amplification requirement is absorbed by the existing O ( log ( R κ / r ) ) O(\log(R\kappa/r)) O ( log ( R κ / r )) repetitions. The geometric-level caps are adjusted by constant factors in the same way. The contribution of the coupling failures is now at most
L ( 2 R + N r ) 1 C p κ ( 2 + N r / R ) = L R C p κ = ϵ o p t C p . L(2R+Nr)\frac{1}{C_p\kappa(2+Nr/R)}=\frac{LR}{C_p\kappa}=\frac{\epsilon_{\mathrm{opt}}}{C_p}. L ( 2 R + N r ) C p κ ( 2 + N r / R ) 1 = C p κ L R = C p ϵ opt .
Thus, after increasing constants, each implemented execution satisfies
Pr ( A τ ≥ A max ) ≥ 1 2 \Pr(A_{\tau}\ge A_{\max})\ge\frac{1}{2} Pr ( A τ ≥ A m a x ) ≥ 2 1
and
E [ ( F ( x τ ) − F ( x ⋆ ) ) 1 { A τ ≥ A max } ] ≤ ϵ o p t 32 . \mathbb{E}\left[(F(x_{\tau})-F(x^\star))\mathbf{1}_{\{A_{\tau}\ge A_{\max}\}}\right]\le\frac{\epsilon_{\mathrm{opt}}}{32}. E [ ( F ( x τ ) − F ( x ⋆ )) 1 { A τ ≥ A m a x } ] ≤ 32 ϵ opt .
Independently restart until the first accepted execution, returning its unprojected output. After ⌈ log 2 ( 128 κ ) ⌉ \lceil\log_2(128\kappa)\rceil ⌈ log 2 ( 128 κ )⌉ unsuccessful executions, return 0 0 0 . Since F ( 0 ) − F ( x ⋆ ) ≤ L R F(0)-F(x^\star)\le LR F ( 0 ) − F ( x ⋆ ) ≤ L R , the same restart calculation proves
E [ F ( x ) − F ( x ⋆ ) ] < ϵ o p t . \mathbb{E}[F(x)-F(x^\star)]<\epsilon_{\mathrm{opt}}. E [ F ( x ) − F ( x ⋆ )] < ϵ opt .
The total iteration and oracle-call bounds are unchanged up to constants depending only on p p p . Since a global minimizer has objective value no larger than any comparator in B p ( R ) \mathbb{B}_p(R) B p ( R ) , the guarantee also holds for every such comparator.
Proposition 12 (based on results in [23 ] ). Let f , g f,g f , g satisfy the assumptions of Problem 1. For x ˉ ∈ X \bar{x}\in\mathcal{X} x ˉ ∈ X , interpret local minimization as being over X ∩ B p ( x ˉ , r ) \mathcal{X}\cap\mathbb{B}_p(\bar{x},r) X ∩ B p ( x ˉ , r ) , as in Proposition 11. When ρ = r \rho=r ρ = r , Algorithm 10 is a ( ϕ , λ ) (\phi,\lambda) ( ϕ , λ ) -ball optimization oracle for f ^ ρ \hat{f}_{\rho} f ^ ρ for 0 < ϕ ≤ λ r p / 2 0<\phi\le\lambda r^p/2 0 < ϕ ≤ λ r p /2 . It makes
O ( 1 + log λ r p ϕ + ( ( C p L / ρ + ( p − 1 ) λ r p − 2 ) p λ 2 ϕ p − 2 ) 1 p + 2 + L p λ ϕ p − 1 ) O\left(1+\log\frac{\lambda r^p}{\phi}+\left(\frac{(C_pL/\rho+(p-1)\lambda r^{p-2})^p}{\lambda^2\phi^{p-2}}\right)^{\frac{1}{p+2}}+\frac{Lp}{\lambda\phi^{p-1}}\right) O ( 1 + log ϕ λ r p + ( λ 2 ϕ p − 2 ( C p L / ρ + ( p − 1 ) λ r p − 2 ) p ) p + 2 1 + λ ϕ p − 1 L p ) total queries to g g g with constant query depth, and has an additional computational cost of
O ( 1 + log λ r p ϕ + ( ( C p L / ρ + ( p − 1 ) λ r p − 2 ) p λ 2 ϕ p − 2 ) 1 p + 2 ) O\left(1+\log\frac{\lambda r^p}{\phi}+\left(\frac{(C_pL/\rho+(p-1)\lambda r^{p-2})^p}{\lambda^2\phi^{p-2}}\right)^{\frac{1}{p+2}}\right) O ( 1 + log ϕ λ r p + ( λ 2 ϕ p − 2 ( C p L / ρ + ( p − 1 ) λ r p − 2 ) p ) p + 2 1 )
sequential proximal-update depth and
O ( 1 + log λ r p ϕ + ( C p L / ρ + ( p − 1 ) λ r p − 2 λ 2 ϕ p − 2 ) p 1 p + 2 + L p λ ϕ p − 1 ) O\left(1+\log\frac{\lambda r^p}{\phi}+\left(\frac{C_pL/\rho+(p-1)\lambda r^{p-2}}{\lambda^2\phi^{p-2}}\right)^p{}^{\frac{1}{p+2}}+\frac{L^p}{\lambda\phi^{p-1}}\right) O ( 1 + log ϕ λ r p + ( λ 2 ϕ p − 2 C p L / ρ + ( p − 1 ) λ r p − 2 ) p p + 2 1 + λ ϕ p − 1 L p )
vector-arithmetic work.
Proof. We follow the proof of Proposition 4 in Appendix D of [15 ] . Let
L Ψ = C p L ρ + ( p − 1 ) λ r p − 2 , Ψ ( x ) = f ^ ρ ( x ) + λ p ∥ x − x ˉ ∥ p p . L_{\Psi}=\frac{C_pL}{\rho}+(p-1)\lambda r^{p-2},\qquad\Psi(x)=\widehat{f}_{\rho}(x)+\frac{\lambda}{p}\|x-\bar{x}\|_p^p. L Ψ = ρ C p L + ( p − 1 ) λ r p − 2 , Ψ ( x ) = f ρ ( x ) + p λ ∥ x − x ˉ ∥ p p .
The function Ψ \Psi Ψ is L Ψ L_{\Psi} L Ψ -smooth on X ∩ B p ( x ˉ , r ) \mathcal{X}\cap B_p(\bar{x},r) X ∩ B p ( x ˉ , r ) and λ \lambda λ -uniformly convex relative to
V ( x , y ) = D ∥ ⋅ − x ˉ ∥ p p / p ( y , x ) , V(x,y)=D_{\|\cdot-\bar{x}\|_p^p/p}(y,x), V ( x , y ) = D ∥ ⋅− x ˉ ∥ p p / p ( y , x ) ,
with V ( x , y ) ≥ ν ∥ y − x ∥ p p / p V(x,y)\geq\nu\|y-x\|_p^p/p V ( x , y ) ≥ ν ∥ y − x ∥ p p / p and ν = 2 2 − p \nu=2^{2-p} ν = 2 2 − p . Let
x ⋆ ∈ arg min x ∈ X ∩ B p ( x ˉ , r ) Ψ ( x ) , ϕ k = λ r p 2 k . x^\star\in\arg\min_{x\in\mathcal{X}\cap B_p(\bar{x},r)}\Psi(x),\qquad\phi_k=\frac{\lambda r^p}{2^k}. x ⋆ ∈ arg x ∈ X ∩ B p ( x ˉ , r ) min Ψ ( x ) , ϕ k = 2 k λ r p .
To justify the minibatch moment term below, condition on the past before a minibatch is used, and let x x x be its current iterate. Since ρ = r \rho=r ρ = r and x ∈ X ∩ B p ( x ˉ , r ) x\in\mathcal{X}\cap B_p(\bar{x},r) x ∈ X ∩ B p ( x ˉ , r ) , Lemma 1 implies that the N k N_k N k independent p p p -GREG estimates ∇ ~ x ˉ g , j f ^ ρ ( x ) \widetilde{\nabla}_{\bar{x}}^{g,j}\widehat{f}_{\rho}(x) ∇ x ˉ g , j f ρ ( x ) , j = 1 , … , N k j=1,\ldots,N_k j = 1 , … , N k , are unbiased and have q q q -th moments at most 3 L q 3L^q 3 L q . Since q = p / ( p − 1 ) ∈ ( 1 , 2 ] q=p/(p-1)\in(1,2] q = p / ( p − 1 ) ∈ ( 1 , 2 ] , applying the von Bahr–Esseen inequality coordinatewise and summing over coordinates gives
E ∥ 1 N k ∑ j = 1 N k ( ∇ ~ x ˉ g , j f ^ ρ , p ( x ) − ∇ ^ f ^ ρ , p ( x ) ) ∥ q q ≤ C p N k q ∑ j = 1 N k E ∥ ∇ ~ x ˉ g , j f ^ ρ , p ( x ) − ∇ f ^ ρ , p ( x ) ∥ q q ≤ C p L q N k q − 1 . \mathbb{E}\left\|\frac{1}{N_k}\sum_{j=1}^{N_k}\left(\widetilde{\nabla}_{\bar{x}}^{g,j}\widehat{f}_{\rho,p}(x)-\widehat{\nabla}\widehat{f}_{\rho,p}(x)\right)\right\|_q^q
\leq\frac{C_p}{N_k^q}\sum_{j=1}^{N_k}\mathbb{E}\left\|\widetilde{\nabla}_{\bar{x}}^{g,j}\widehat{f}_{\rho,p}(x)-\nabla\widehat{f}_{\rho,p}(x)\right\|_q^q
\leq\frac{C_pL^q}{N_k^{q-1}}. E N k 1 j = 1 ∑ N k ( ∇ x ˉ g , j f ρ , p ( x ) − ∇ f ρ , p ( x ) ) q q ≤ N k q C p j = 1 ∑ N k E ∇ x ˉ g , j f ρ , p ( x ) − ∇ f ρ , p ( x ) q q ≤ N k q − 1 C p L q .
Applying Theorem 16 (Equation (73) ) conditionally on the initial point of epoch k k k , with μ = λ \mu=\lambda μ = λ and the parameters of Algorithm 10, gives
E [ Ψ ( x T k a g ) − Ψ ( x ⋆ ) ] ≤ C p [ L Ψ p − 1 E V ( x 0 , x ⋆ ) ν M f , k p − 2 T k p + M f , k q + L q / N k q − 1 ( ν λ ) q − 1 T k q − 1 ] . (51) \mathbb{E}\left[\Psi\left(x_{T_k}^{\mathrm{ag}}\right)-\Psi(x^\star)\right]\leq C_p\left[\frac{L_{\Psi}^{p-1}\mathbb{E}V(x_0,x^\star)}{\nu M_{f,k}^{p-2}T_k^p}+\frac{M_{f,k}^q+L^q/N_k^{q-1}}{(\nu\lambda)^{q-1}T_k^{q-1}}\right]. \tag*{(51)} E [ Ψ ( x T k ag ) − Ψ ( x ⋆ ) ] ≤ C p [ ν M f , k p − 2 T k p L Ψ p − 1 E V ( x 0 , x ⋆ ) + ( ν λ ) q − 1 T k q − 1 M f , k q + L q / N k q − 1 ] . (51)
We prove by induction that
E [ Ψ ( x T k a g ) − Ψ ( x ⋆ ) ] ≤ ϕ k . \mathbb{E}\left[\Psi\left(x_{T_k}^{\mathrm{ag}}\right)-\Psi(x^\star)\right]\leq\phi_k. E [ Ψ ( x T k ag ) − Ψ ( x ⋆ ) ] ≤ ϕ k .
For the first epoch, x 0 = x ˉ x_0=\bar{x} x 0 = x ˉ , and hence V ( x 0 , x ⋆ ) ≤ r p / p V(x_0,x^\star)\leq r^p/p V ( x 0 , x ⋆ ) ≤ r p / p . For subsequent epochs, first-order optimality at the constrained minimizer x ⋆ x^\star x ⋆ and relative uniform convexity give
Ψ ( x 0 ) − Ψ ( x ⋆ ) ≥ ⟨ ∇ Ψ ( x ⋆ ) , x 0 − x ⋆ ⟩ + λ V ( x ⋆ , x 0 ) ≥ λ V ( x ⋆ , x 0 ) . \Psi(x_0)-\Psi(x^\star)\geq\langle\nabla\Psi(x^\star),x_0-x^\star\rangle+\lambda V(x^\star,x_0)\geq\lambda V(x^\star,x_0). Ψ ( x 0 ) − Ψ ( x ⋆ ) ≥ ⟨ ∇Ψ ( x ⋆ ) , x 0 − x ⋆ ⟩ + λV ( x ⋆ , x 0 ) ≥ λV ( x ⋆ , x 0 ) .
By proof of Lemma 22 and Fact 3, we have, for all a , b ∈ R a,b\in\mathbb{R} a , b ∈ R ,
D ∣ ⋅ ∣ p / p ( a , b ) ≤ C p D ∣ ⋅ ∣ p / p ( b , a ) , D_{|\cdot|^p/p}(a,b)\leq C_pD_{|\cdot|^p/p}(b,a), D ∣ ⋅ ∣ p / p ( a , b ) ≤ C p D ∣ ⋅ ∣ p / p ( b , a ) ,
where C p C_p C p depends only on p p p . Translating by x ˉ \bar{x} x ˉ and summing coordinatewise therefore gives
V ( x 0 , x ⋆ ) = D ∥ ⋅ − x ˉ ∥ p p / p ( x ⋆ , x 0 ) ≤ C p D ∥ ⋅ − x ˉ ∥ p p / p ( x 0 , x ⋆ ) = C p V ( x ⋆ , x 0 ) . V(x_0,x^\star)=D_{\|\cdot-\bar{x}\|_p^p/p}(x^\star,x_0)\leq C_pD_{\|\cdot-\bar{x}\|_p^p/p}(x_0,x^\star)=C_pV(x^\star,x_0). V ( x 0 , x ⋆ ) = D ∥ ⋅− x ˉ ∥ p p / p ( x ⋆ , x 0 ) ≤ C p D ∥ ⋅− x ˉ ∥ p p / p ( x 0 , x ⋆ ) = C p V ( x ⋆ , x 0 ) .
Consequently,
EV ( x 0 , x ⋆ ) ≤ C p λ E [ Ψ ( x 0 ) − Ψ ( x ⋆ ) ] ≤ C p ϕ k − 1 λ . \operatorname{E V}(x_0,x^\star) \leq\frac{C_p}{\lambda}\mathbb{E}[\Psi(x_0)-\Psi(x^\star)] \leq\frac{C_p\phi_{k-1}}{\lambda}. EV ( x 0 , x ⋆ ) ≤ λ C p E [ Ψ ( x 0 ) − Ψ ( x ⋆ )] ≤ λ C p ϕ k − 1 .
Substituting the choices of M f , k , T k , N k M_{f,k},T_k,N_k M f , k , T k , N k from Algorithm Algorithm 10 into (51) , and choosing the p p p -dependent constants sufficiently large, yields
E [ Ψ ( x T k a g ) − Ψ ( x ⋆ ) ] ≤ ϕ k − 1 2 = ϕ k . \mathbb{E}[\Psi(x_{T_k}^{\mathrm{ag}})-\Psi(x^\star)] \leq\frac{\phi_{k-1}}{2}=\phi_k. E [ Ψ ( x T k ag ) − Ψ ( x ⋆ )] ≤ 2 ϕ k − 1 = ϕ k .
Since K = ⌈ log 2 ( λ r p / ϕ ) ⌉ K=\lceil\log_2(\lambda r^p/\phi)\rceil K = ⌈ log 2 ( λ r p / ϕ )⌉ , the final output satisfies
E [ Ψ ( x T K a g ) − Ψ ( x ⋆ ) ] ≤ λ r p 2 K ≤ ϕ . \mathbb{E}[\Psi(x_{T_K}^{\mathrm{ag}})-\Psi(x^\star)] \leq\frac{\lambda r^p}{2^K}\leq\phi. E [ Ψ ( x T K ag ) − Ψ ( x ⋆ )] ≤ 2 K λ r p ≤ ϕ .
All iterates lie in X ∩ B p ( x ˉ , r ) \mathcal{X}\cap B_p(\bar{x},r) X ∩ B p ( x ˉ , r ) by convexity of the feasible set. Finally, the geometric decrease of ϕ k \phi_k ϕ k and p > 2 p>2 p > 2 give
∑ k = 1 K T k = O ( 1 + log λ r p ϕ + ( L Ψ p λ 2 ϕ p − 2 ) 1 p + 2 ) \sum_{k=1}^{K}T_k=O\left(1+\log\frac{\lambda r^p}{\phi}+\left(\frac{L_\Psi^p}{\lambda^2\phi^{p-2}}\right)^{\frac{1}{p+2}}\right) k = 1 ∑ K T k = O ( 1 + log ϕ λ r p + ( λ 2 ϕ p − 2 L Ψ p ) p + 2 1 )
and
∑ k = 1 K T k N k = O ( 1 + log λ r p ϕ + ( L Ψ p λ 2 ϕ p − 2 ) 1 p + 2 + L p λ ϕ p − 1 ) . \sum_{k=1}^{K}T_kN_k=O\left(1+\log\frac{\lambda r^p}{\phi}+\left(\frac{L_\Psi^p}{\lambda^2\phi^{p-2}}\right)^{\frac{1}{p+2}}+\frac{L^p}{\lambda\phi^{p-1}}\right). k = 1 ∑ K T k N k = O ( 1 + log ϕ λ r p + ( λ 2 ϕ p − 2 L Ψ p ) p + 2 1 + λ ϕ p − 1 L p ) .
All original stochastic-gradient queries are issued in parallel, since the anchor x ˉ \bar{x} x ˉ and the batch sizes are fixed in advance. This proves the claimed oracle-query and proximal-update bounds.
EpochDA Solver
Table 9.
Throughout, let f , g f,g f , g satisfy Problem 1, and let
x ⋆ = argmin x ∈ B p ( x ˉ , r ) { f ^ ρ ( x ) + λ p ∥ x − x ˉ ∥ p p } . x^\star=\underset{x\in B_p(\bar{x},r)}{\operatorname{argmin}}\left\{\widehat{f}_{\rho}(x)+\frac{\lambda}{p}\|x-\bar{x}\|_p^p\right\}. x ⋆ = x ∈ B p ( x ˉ , r ) argmin { f ρ ( x ) + p λ ∥ x − x ˉ ∥ p p } .
Let C r e s , p ≥ 1 C_{\mathrm{res},p}\ge1 C res , p ≥ 1 be the constant in the conditional q q q -moment bound for ρ = r \rho=r ρ = r .
Lemma 23. In view of Problem 1, Each epoch k k k of Algorithm 9 satisfies
E [ f ^ ρ ( x k + 1 0 ) + λ p ∥ x k + 1 0 − x ˉ ∥ p p ] − f ^ ρ ( x ⋆ ) − λ p ∥ x ⋆ − x ˉ ∥ p p ≤ E ∥ x ⋆ − x k 0 ∥ p p p η k T k + ( 4 L ) q q ( η k 2 2 − p ) q − 1 . (52) \mathbb{E}\left[\widehat{f}_{\rho}(x_{k+1}^{0})+\frac{\lambda}{p}\|x_{k+1}^{0}-\bar{x}\|_p^p\right]-\widehat{f}_{\rho}(x^\star)-\frac{\lambda}{p}\|x^\star-\bar{x}\|_p^p\leq\frac{\mathbb{E}\|x^\star-x_k^0\|_p^p}{p\eta_kT_k}+\frac{(4L)^q}{q}\left(\frac{\eta_k}{2^{2-p}}\right)^{q-1}. \tag*{(52)} E [ f ρ ( x k + 1 0 ) + p λ ∥ x k + 1 0 − x ˉ ∥ p p ] − f ρ ( x ⋆ ) − p λ ∥ x ⋆ − x ˉ ∥ p p ≤ p η k T k E ∥ x ⋆ − x k 0 ∥ p p + q ( 4 L ) q ( 2 2 − p η k ) q − 1 . (52)
Proof. Problem 1 gives ∥ E g ( x ) ∥ q ≤ E ∥ g ( x ) ∥ q ≤ L \|\mathbb{E}g(x)\|_q\leq\mathbb{E}\|g(x)\|_q\leq L ∥ E g ( x ) ∥ q ≤ E ∥ g ( x ) ∥ q ≤ L ; hence f f f and f ^ ρ \widehat{f}_{\rho} f ρ are convex and L L L -Lipschitz. Let F k , t − 1 \mathcal{F}_{k,t-1} F k , t − 1 be the history of the samples already used, including x k t x_k^t x k t . By Lemma 1,
E [ ∇ ~ x ˉ g f ^ ρ ( x k t ) ∣ F k , t − 1 ] = ∇ f ^ ρ ( x k t ) , E [ ∥ ∇ ~ x ˉ g f ^ ρ ( x k t ) ∥ q q ∣ F k , t − 1 ] ≤ 3 L q . (53) \mathbb{E}\left[\widetilde{\nabla}_{\bar{x}}^g\widehat{f}_{\rho}(x_k^t)\mid\mathcal{F}_{k,t-1}\right]=\nabla\widehat{f}_{\rho}(x_k^t),\qquad\mathbb{E}\left[\left\|\widetilde{\nabla}_{\bar{x}}^g\widehat{f}_{\rho}(x_k^t)\right\|_q^q\mid\mathcal{F}_{k,t-1}\right]\leq3L^q. \tag*{(53)} E [ ∇ x ˉ g f ρ ( x k t ) ∣ F k , t − 1 ] = ∇ f ρ ( x k t ) , E [ ∇ x ˉ g f ρ ( x k t ) q q ∣ F k , t − 1 ] ≤ 3 L q . (53)
The optimality condition of the update, convexity of the regularizer, and the Bregman three-point identity imply
⟨ ∇ ~ x ˉ g f ^ ρ ( x k t ) , x k t + 1 − x ⋆ ⟩ + λ p ∥ x k t + 1 − x ˉ ∥ p p − λ p ∥ x ⋆ − x ˉ ∥ p p ≤ 1 η k [ D Φ k ( x ⋆ , x k t ) − D Φ k ( x ⋆ , x k t + 1 ) − D Φ k ( x k t + 1 , x k t ) ] . (54) \left\langle\widetilde{\nabla}_{\bar{x}}^g\widehat{f}_{\rho}(x_k^t),x_k^{t+1}-x^\star\right\rangle+\frac{\lambda}{p}\|x_k^{t+1}-\bar{x}\|_p^p-\frac{\lambda}{p}\|x^\star-\bar{x}\|_p^p
\leq\frac{1}{\eta_k}\left[D_{\Phi_k}(x^\star,x_k^t)-D_{\Phi_k}(x^\star,x_k^{t+1})-D_{\Phi_k}(x_k^{t+1},x_k^t)\right]. \tag*{(54)} ⟨ ∇ x ˉ g f ρ ( x k t ) , x k t + 1 − x ⋆ ⟩ + p λ ∥ x k t + 1 − x ˉ ∥ p p − p λ ∥ x ⋆ − x ˉ ∥ p p ≤ η k 1 [ D Φ k ( x ⋆ , x k t ) − D Φ k ( x ⋆ , x k t + 1 ) − D Φ k ( x k t + 1 , x k t ) ] . (54)
By Lipschitz continuity, convexity, and conditional unbiasedness,
E [ f ^ ρ ( x k t + 1 ) − f ^ ρ ( x ⋆ ) ] ≤ E [ ⟨ ∇ ~ x ˉ g f ^ ρ ( x k t ) , x k t − x ⋆ ⟩ + L ∥ x k t + 1 − x k t ∥ p ] ≤ E [ ⟨ ∇ ~ x ˉ g f ^ ρ ( x k t ) , x k t + 1 − x ⋆ ⟩ + ( ∥ ∇ ~ x ˉ g f ^ ρ ( x k t ) ∥ q + L ) ∥ x k t + 1 − x k t ∥ p ] . \begin{aligned}
\mathbb{E}\left[\widehat{f}_{\rho}(x_k^{t+1})-\widehat{f}_{\rho}(x^\star)\right]\leq\mathbb{E}\left[\left\langle\widetilde{\nabla}_{\bar{x}}^g\widehat{f}_{\rho}(x_k^t),x_k^t-x^\star\right\rangle+L\|x_k^{t+1}-x_k^t\|_p\right] \\
&\leq\mathbb{E}\left[\left\langle\widetilde{\nabla}_{\bar{x}}^g\widehat{f}_{\rho}(x_k^t),x_k^{t+1}-x^\star\right\rangle+\left(\left\|\widetilde{\nabla}_{\bar{x}}^g\widehat{f}_{\rho}(x_k^t)\right\|_q+L\right)\|x_k^{t+1}-x_k^t\|_p\right].
\end{aligned} E [ f ρ ( x k t + 1 ) − f ρ ( x ⋆ ) ] ≤ E [ ⟨ ∇ x ˉ g f ρ ( x k t ) , x k t − x ⋆ ⟩ + L ∥ x k t + 1 − x k t ∥ p ] ≤ E [ ⟨ ∇ x ˉ g f ρ ( x k t ) , x k t + 1 − x ⋆ ⟩ + ( ∇ x ˉ g f ρ ( x k t ) q + L ) ∥ x k t + 1 − x k t ∥ p ] .
Since D Φ k ( u , v ) ≥ 2 2 − p ∥ u − v ∥ p p / p D_{\Phi_k}(u,v)\geq2^{2-p}\|u-v\|_p^p/p D Φ k ( u , v ) ≥ 2 2 − p ∥ u − v ∥ p p / p , Young’s inequality gives
( ∥ ∇ ~ x ˉ g f ^ ρ ( x k t ) ∥ q + L ) ∥ x k t + 1 − x k t ∥ p − D Φ k ( x k t + 1 , x k t ) η k ≤ 1 q ( η k 2 2 − p ) q − 1 ( ∥ ∇ ~ x ˉ g f ^ ρ ( x k t ) ∥ q + L ) q . \left(\left\|\widetilde{\nabla}_{\bar{x}}^g\widehat{f}_{\rho}(x_k^t)\right\|_q+L\right)\|x_k^{t+1}-x_k^t\|_p-\frac{D_{\Phi_k}(x_k^{t+1},x_k^t)}{\eta_k}
\leq\frac{1}{q}\left(\frac{\eta_k}{2^{2-p}}\right)^{q-1}\left(\left\|\widetilde{\nabla}_{\bar{x}}^g\widehat{f}_{\rho}(x_k^t)\right\|_q+L\right)^q. ( ∇ x ˉ g f ρ ( x k t ) q + L ) ∥ x k t + 1 − x k t ∥ p − η k D Φ k ( x k t + 1 , x k t ) ≤ q 1 ( 2 2 − p η k ) q − 1 ( ∇ x ˉ g f ρ ( x k t ) q + L ) q .
Moreover, conditional Minkowski’s inequality and (53) yield
E [ ( ∥ ∇ ~ x ˉ g f ^ ρ ( x k t ) ∥ q + L ) q ∣ F k , t − 1 ] ≤ ( 4 L ) q . \mathbb{E}\left[\left(\left\|\widetilde{\nabla}_{\bar{x}}^g\widehat{f}_{\rho}(x_k^t)\right\|_q+L\right)^q\mid\mathcal{F}_{k,t-1}\right]\leq(4L)^q. E [ ( ∇ x ˉ g f ρ ( x k t ) q + L ) q ∣ F k , t − 1 ] ≤ ( 4 L ) q .
Combining these estimates with (54) gives
E [ f ^ ρ ( x k t + 1 ) + λ p ∥ x k t + 1 − x ˉ ∥ p p − f ^ ρ ( x ⋆ ) − λ p ∥ x ⋆ − x ˉ ∥ p p ] ≤ E D Φ k ( x ⋆ , x k t ) − E D Φ k ( x ⋆ , x k t + 1 ) η k + ( 4 L ) q q ( η k 2 2 − p ) q − 1 . \mathbb{E}\left[\widehat{f}_{\rho}(x_k^{t+1})+\frac{\lambda}{p}\|x_k^{t+1}-\bar{x}\|_p^p-\widehat{f}_{\rho}(x^\star)-\frac{\lambda}{p}\|x^\star-\bar{x}\|_p^p\right]
\leq\frac{\mathbb{E}D_{\Phi_k}(x^\star,x_k^t)-\mathbb{E}D_{\Phi_k}(x^\star,x_k^{t+1})}{\eta_k}+\frac{(4L)^q}{q}\left(\frac{\eta_k}{2^{2-p}}\right)^{q-1}. E [ f ρ ( x k t + 1 ) + p λ ∥ x k t + 1 − x ˉ ∥ p p − f ρ ( x ⋆ ) − p λ ∥ x ⋆ − x ˉ ∥ p p ] ≤ η k E D Φ k ( x ⋆ , x k t ) − E D Φ k ( x ⋆ , x k t + 1 ) + q ( 4 L ) q ( 2 2 − p η k ) q − 1 .
Summing over t = 1 , … , T k t=1,\ldots,T_k t = 1 , … , T k telescopes the Bregman terms because Φ k \Phi_k Φ k is fixed within the epoch. As x k 1 = x k 0 x_k^1=x_k^0 x k 1 = x k 0 ,
D Φ k ( x ⋆ , x k 1 ) = 1 p ∥ x ⋆ − x k 0 ∥ p p . D_{\Phi_k}(x^\star,x_k^1)=\frac{1}{p}\|x^\star-x_k^0\|_p^p. D Φ k ( x ⋆ , x k 1 ) = p 1 ∥ x ⋆ − x k 0 ∥ p p .
Dropping the nonnegative final divergence, dividing by T k T_k T k , and using convexity at x k + 1 0 = T k − 1 ∑ t = 1 T k x k t + 1 x_{k+1}^0=T_k^{-1}\sum_{t=1}^{T_k}x_k^{t+1} x k + 1 0 = T k − 1 ∑ t = 1 T k x k t + 1 proves (52) .
Lemma 24. After each completed epoch k k k of Algorithm 9,
E [ f ^ ρ ( x k + 1 0 ) + λ p ∥ x k + 1 0 − x ˉ ∥ p p ] − f ^ ρ ( x ⋆ ) − λ p ∥ x ⋆ − x ˉ ∥ p p ≤ ( 4 L ) q λ q − 1 2 k . (55) \mathbb{E}\left[\widehat{f}_{\rho}(x_{k+1}^{0})+\frac{\lambda}{p}\left\|x_{k+1}^{0}-\bar{x}\right\|_{p}^{p}\right]-\widehat{f}_{\rho}(x^{\star})-\frac{\lambda}{p}\left\|x^{\star}-\bar{x}\right\|_{p}^{p}\leq\frac{(4L)^{q}}{\lambda^{q-1}2^{k}}. \tag*{(55)} E [ f ρ ( x k + 1 0 ) + p λ x k + 1 0 − x ˉ p p ] − f ρ ( x ⋆ ) − p λ ∥ x ⋆ − x ˉ ∥ p p ≤ λ q − 1 2 k ( 4 L ) q . (55)
Proof. Convexity of f ^ ρ \widehat{f}_{\rho} f ρ , p p p -uniform convexity of the regularizer, and constrained optimality of x ⋆ x^{\star} x ⋆ imply, for every w ∈ B p ( x ˉ , r ) w\in\mathbb{B}_{p}(\bar{x},r) w ∈ B p ( x ˉ , r ) ,
f ^ ρ ( w ) + λ p ∥ w − x ˉ ∥ p p − f ^ ρ ( x ⋆ ) − λ p ∥ x ⋆ − x ˉ ∥ p p ≥ 2 2 − p λ p ∥ w − x ⋆ ∥ p p . (56) \widehat{f}_{\rho}(w)+\frac{\lambda}{p}\left\|w-\bar{x}\right\|_{p}^{p}-\widehat{f}_{\rho}(x^{\star})-\frac{\lambda}{p}\left\|x^{\star}-\bar{x}\right\|_{p}^{p}\geq\frac{2^{2-p}\lambda}{p}\left\|w-x^{\star}\right\|_{p}^{p}. \tag*{(56)} f ρ ( w ) + p λ ∥ w − x ˉ ∥ p p − f ρ ( x ⋆ ) − p λ ∥ x ⋆ − x ˉ ∥ p p ≥ p 2 2 − p λ ∥ w − x ⋆ ∥ p p . (56)
Since x 1 0 = x ˉ x_{1}^{0}=\bar{x} x 1 0 = x ˉ and f ^ ρ \widehat{f}_{\rho} f ρ is L L L -Lipschitz,
f ^ ρ ( x ˉ ) − f ^ ρ ( x ⋆ ) − λ p ∥ x ⋆ − x ˉ ∥ p p ≤ sup a ≥ 0 { L a − λ p a p } = L q q λ q − 1 ≤ ( 4 L ) q λ q − 1 . \widehat{f}_{\rho}(\bar{x})-\widehat{f}_{\rho}(x^{\star})-\frac{\lambda}{p}\left\|x^{\star}-\bar{x}\right\|_{p}^{p}\leq\sup_{a\geq0}\left\{La-\frac{\lambda}{p}a^{p}\right\}=\frac{L^{q}}{q\lambda^{q-1}}\leq\frac{(4L)^{q}}{\lambda^{q-1}}. f ρ ( x ˉ ) − f ρ ( x ⋆ ) − p λ ∥ x ⋆ − x ˉ ∥ p p ≤ a ≥ 0 sup { L a − p λ a p } = q λ q − 1 L q ≤ λ q − 1 ( 4 L ) q .
The parameter choices in Algorithm 9 give
η k = 2 2 − p 4 p − 1 λ 2 ( p − 1 ) ( k − 1 ) , T k ≥ 4 p ( 2 2 − p ) 2 2 ( p − 1 ) ( k − 1 ) , \eta_{k}=\frac{2^{2-p}}{4^{p-1}\lambda2^{(p-1)(k-1)}},\qquad T_{k}\geq\frac{4^{p}}{(2^{2-p})^{2}}2^{(p-1)(k-1)}, η k = 4 p − 1 λ 2 ( p − 1 ) ( k − 1 ) 2 2 − p , T k ≥ ( 2 2 − p ) 2 4 p 2 ( p − 1 ) ( k − 1 ) ,
and hence η k T k ≥ 4 / ( 2 2 − p λ ) \eta_{k}T_{k}\geq4/(2^{2-p}\lambda) η k T k ≥ 4/ ( 2 2 − p λ ) . Suppose by induction that the expected objective error at the start of epoch k k k is at most ( 4 L ) q / ( λ q − 1 2 k − 1 ) (4L)^{q}/(\lambda^{q-1}2^{k-1}) ( 4 L ) q / ( λ q − 1 2 k − 1 ) , the base case k = 1 k=1 k = 1 is verified above. By (56) ,
E ∥ x ⋆ − x k 0 ∥ p p p η k T k ≤ 1 4 ( 4 L ) q λ q − 1 2 k − 1 . \frac{\mathbb{E}\left\|x^{\star}-x_{k}^{0}\right\|_{p}^{p}}{p\eta_{k}T_{k}}\leq\frac{1}{4}\frac{(4L)^{q}}{\lambda^{q-1}2^{k-1}}. p η k T k E x ⋆ − x k 0 p p ≤ 4 1 λ q − 1 2 k − 1 ( 4 L ) q .
Using ( p − 1 ) ( q − 1 ) = 1 (p-1)(q-1)=1 ( p − 1 ) ( q − 1 ) = 1 , the other term in Lemma 23 is
( 4 L ) q q ( η k 2 2 − p ) q − 1 = ( 4 L ) q 4 q λ q − 1 2 k − 1 . \frac{(4L)^{q}}{q}\left(\frac{\eta_{k}}{2^{2-p}}\right)^{q-1}=\frac{(4L)^{q}}{4q\lambda^{q-1}2^{k-1}}. q ( 4 L ) q ( 2 2 − p η k ) q − 1 = 4 q λ q − 1 2 k − 1 ( 4 L ) q .
Applying Lemma 23 therefore gives
E [ f ^ ρ ( x k + 1 0 ) + λ p ∥ x k + 1 0 − x ˉ ∥ p p ] − f ^ ρ ( x ⋆ ) − λ p ∥ x ⋆ − x ˉ ∥ p p ≤ ( 1 4 + 1 4 q ) ( 4 L ) q λ q − 1 2 k − 1 ≤ ( 4 L ) q λ q − 1 2 k . \mathbb{E}\left[\widehat{f}_{\rho}(x_{k+1}^{0})+\frac{\lambda}{p}\left\|x_{k+1}^{0}-\bar{x}\right\|_{p}^{p}\right]-\widehat{f}_{\rho}(x^{\star})-\frac{\lambda}{p}\left\|x^{\star}-\bar{x}\right\|_{p}^{p}\leq\left(\frac{1}{4}+\frac{1}{4q}\right)\frac{(4L)^{q}}{\lambda^{q-1}2^{k-1}}\leq\frac{(4L)^{q}}{\lambda^{q-1}2^{k}}. E [ f ρ ( x k + 1 0 ) + p λ x k + 1 0 − x ˉ p p ] − f ρ ( x ⋆ ) − p λ ∥ x ⋆ − x ˉ ∥ p p ≤ ( 4 1 + 4 q 1 ) λ q − 1 2 k − 1 ( 4 L ) q ≤ λ q − 1 2 k ( 4 L ) q .
This is the required bound for epoch k k k and the induction hypothesis for the start of epoch k + 1 k+1 k + 1 . □ \square □
:::
Proposition 13 (Proposition 3, [15 ] ). Let f , g f,g f , g satisfy the assumptions of Problem 1. When ρ = r \rho=r ρ = r , Algorithm 9 is a ( ϕ , λ ) (\phi,\lambda) ( ϕ , λ ) -ball optimization oracle for f ^ ρ \widehat{f}_{\rho} f ρ which makes O ( L p λ ϕ p − 1 ) O\left(\frac{L^{p}}{\lambda\phi^{p-1}}\right) O ( λ ϕ p − 1 L p ) total queries to g g g with constant query depth, and an additional computational cost of O ( L p λ ϕ p − 1 ) O\left(\frac{L^{p}}{\lambda\phi^{p-1}}\right) O ( λ ϕ p − 1 L p ) depth and work.
Proof. Let k k k be the epoch index when Algorithm 9 terminates, so that the returned point is x k 0 x_{k}^{0} x k 0 . The recurrence T j + 1 = ⌈ 2 p − 1 T j ⌉ T_{j+1}=\lceil2^{p-1}T_{j}\rceil T j + 1 = ⌈ 2 p − 1 T j ⌉ implies
T j ≤ 2 ( p − 1 ) ( j − 1 ) ( T 1 + 1 2 p − 1 − 1 ) ≤ 2 ⋅ 4 p ( 2 2 − p ) 2 2 ( p − 1 ) ( j − 1 ) . T_{j}\leq2^{(p-1)(j-1)}\left(T_{1}+\frac{1}{2^{p-1}-1}\right)\leq\frac{2\cdot4^{p}}{(2^{2-p})^{2}}2^{(p-1)(j-1)}. T j ≤ 2 ( p − 1 ) ( j − 1 ) ( T 1 + 2 p − 1 − 1 1 ) ≤ ( 2 2 − p ) 2 2 ⋅ 4 p 2 ( p − 1 ) ( j − 1 ) . By the stopping condition,
T < ∑ j = 1 k T j ≤ 2 ⋅ 4 p ( 2 2 − p ) 2 2 ( p − 1 ) k − 1 2 p − 1 − 1 ≤ 2 ⋅ 4 p 2 p − 1 ( 2 2 − p ) 2 ( 2 p − 1 − 1 ) 2 ( p − 1 ) ( k − 1 ) . T<\sum_{j=1}^{k}T_{j}\leq\frac{2\cdot4^{p}}{(2^{2-p})^{2}}\frac{2^{(p-1)k}-1}{2^{p-1}-1}\leq\frac{2\cdot4^{p}2^{p-1}}{(2^{2-p})^{2}(2^{p-1}-1)}2^{(p-1)(k-1)}. T < j = 1 ∑ k T j ≤ ( 2 2 − p ) 2 2 ⋅ 4 p 2 p − 1 − 1 2 ( p − 1 ) k − 1 ≤ ( 2 2 − p ) 2 ( 2 p − 1 − 1 ) 2 ⋅ 4 p 2 p − 1 2 ( p − 1 ) ( k − 1 ) .
The initial-error bound and Lemma 24 therefore give
E [ f ^ ρ ( x k 0 ) + λ p ∥ x k 0 − x ˉ ∥ p p ] − min x ∈ B p ( x ˉ , r ) { f ^ ρ ( x ) + λ p ∥ x − x ˉ ∥ p p } ≤ ( 4 L ) q λ q − 1 2 k − 1 ≤ [ 2 ⋅ 4 p 2 p − 1 3 p ( 2 2 − p ) ( 2 p − 1 − 1 ) λ T L p λ T ] 1 / ( p − 1 ) ≤ ϕ , \mathbb{E}\left[\widehat{f}_{\rho}(x_k^0)+\frac{\lambda}{p}\lVert x_k^0-\bar{x}\rVert_p^p\right]-\min_{x\in B_p(\bar{x},r)}\left\{\widehat{f}_{\rho}(x)+\frac{\lambda}{p}\lVert x-\bar{x}\rVert_p^p\right\}
\le\frac{(4L)^q}{\lambda q^{-1}2^{k-1}}\le\left[\frac{2\cdot4^p2^{p-1}3^p}{(2^{2-p})(2^{p-1}-1)\lambda T}\frac{L^p}{\lambda T}\right]^{1/(p-1)}\le\phi, E [ f ρ ( x k 0 ) + p λ ∥ x k 0 − x ˉ ∥ p p ] − x ∈ B p ( x ˉ , r ) min { f ρ ( x ) + p λ ∥ x − x ˉ ∥ p p } ≤ λ q − 1 2 k − 1 ( 4 L ) q ≤ [ ( 2 2 − p ) ( 2 p − 1 − 1 ) λ T 2 ⋅ 4 p 2 p − 1 3 p λ T L p ] 1/ ( p − 1 ) ≤ ϕ ,
where the last inequality follows from T = ⌈ C p L p / ( λ ϕ p − 1 ) ⌉ T=\lceil CpL^p/(\lambda\phi^{p-1})\rceil T = ⌈ C p L p / ( λ ϕ p − 1 )⌉ . The output lies in B p ( x ˉ , r ) B_p(\bar{x},r) B p ( x ˉ , r ) by construction and convexity, so the preceding error bound establishes the required guarantee. All 2 T = O ( 1 + L p / ( λ ϕ p − 1 ) ) 2T=O(1+L^p/(\lambda\phi^{p-1})) 2 T = O ( 1 + L p / ( λ ϕ p − 1 )) queries can be made in one parallel round. The algorithm performs at most T T T Bregman proximal updates and O ( T ) O(T) O ( T ) vector operations, giving O ( T ) O(T) O ( T ) additional computational depth and work if each update and vector operation has constant depth and work.
Lemma 25 (Lemma 2.3, [12 ] ). Let f , g f,g f , g satisfy the assumptions of Problem 1, with L > 0 L>0 L > 0 . For any c ∈ R d c\in\mathbb{R}^d c ∈ R d and R > 0 R>0 R > 0 , the function
x ↦ f ( x ) + 2 L ( ∥ x − c ∥ p − R ) + x\mapsto f(x)+2L(\lVert x-c\rVert_p-R)_+ x ↦ f ( x ) + 2 L (∥ x − c ∥ p − R ) + is convex and 3 L 3L 3 L -Lipschitz with respect to ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p . For any y y y with ∥ y − c ∥ p ≥ R \lVert y-c\rVert_p\ge R ∥ y − c ∥ p ≥ R , define
y c , R : = c + R ∥ y − c ∥ p ( y − c ) . y^{c,R}:=c+\frac{R}{\lVert y-c\rVert_p}(y-c). y c , R := c + ∥ y − c ∥ p R ( y − c ) . Then ∥ y c , R − c ∥ p = R \lVert y^{c,R}-c\rVert_p=R ∥ y c , R − c ∥ p = R and
f ( y c , R ) ≤ f ( y ) + 2 L ( ∥ y − c ∥ p − R ) + − L ( ∥ y − c ∥ p − R ) . f(y^{c,R})\le f(y)+2L(\lVert y-c\rVert_p-R)_+-L(\lVert y-c\rVert_p-R). f ( y c , R ) ≤ f ( y ) + 2 L (∥ y − c ∥ p − R ) + − L (∥ y − c ∥ p − R ) . The penalized function attains a global minimum, and every global minimizer x ⋆ x^\star x ⋆ satisfies
∥ x ⋆ − c ∥ p ≤ R , f ( x ⋆ ) + 2 L ( ∥ x ⋆ − c ∥ p − R ) + = f ( x ⋆ ) = min x ∈ B p ( c , R ) f ( x ) . \lVert x^\star-c\rVert_p\le R,\qquad f(x^\star)+2L(\lVert x^\star-c\rVert_p-R)_+=f(x^\star)=\min_{x\in B_p(c,R)}f(x). ∥ x ⋆ − c ∥ p ≤ R , f ( x ⋆ ) + 2 L (∥ x ⋆ − c ∥ p − R ) + = f ( x ⋆ ) = x ∈ B p ( c , R ) min f ( x ) . Moreover, adding a deterministic subgradient of 2 L ( ∥ x − c ∥ p − R ) + 2L(\lVert x-c\rVert_p-R)_+ 2 L (∥ x − c ∥ p − R ) + to g ( x ) g(x) g ( x ) gives a stochastic subgradient oracle for the penalized function with q q q -th moment at most ( 4 L ) q (4L)^q ( 4 L ) q , using one query to g g g .
Proof. The proof follows the argument of Lemma 2.3 in [12 ] . We first show that f ( x ) + 2 L ( ∥ x − c ∥ p − R ) + f(x)+2L(\lVert x-c\rVert_p-R)_+ f ( x ) + 2 L (∥ x − c ∥ p − R ) + is convex and 3 L 3L 3 L -Lipschitz. The assumptions of Problem 1 give
∥ E g ( x ) ∥ q ≤ ( E ∥ g ( x ) ∥ q q ) 1 / q ≤ L . \lVert\mathbb{E}g(x)\rVert_q\le\left(\mathbb{E}\lVert g(x)\rVert_q^q\right)^{1/q}\le L. ∥ E g ( x ) ∥ q ≤ ( E ∥ g ( x ) ∥ q q ) 1/ q ≤ L .
Since E g ( x ) ∈ ∂ f ( x ) \mathbb{E}g(x)\in\partial f(x) E g ( x ) ∈ ∂ f ( x ) , the subgradient inequality implies that f f f is L L L -Lipschitz with respect to ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p . The function x ↦ 2 L ( ∥ x − c ∥ p − R ) + x\mapsto2L(\lVert x-c\rVert_p-R)_+ x ↦ 2 L (∥ x − c ∥ p − R ) + is convex and 2 L 2L 2 L -Lipschitz. Adding f f f therefore gives a convex and 3 L 3L 3 L -Lipschitz function. Next, consider y ∈ R d y\in\mathbb{R}^d y ∈ R d with ∥ y − c ∥ p ≥ R \lVert y-c\rVert_p\ge R ∥ y − c ∥ p ≥ R . By the definition of y c , R y^{c,R} y c , R ,
∥ y c , R − c ∥ p = R , ∥ y c , R − y ∥ p = ∥ y − c ∥ p − R . \lVert y^{c,R}-c\rVert_p=R,\qquad\lVert y^{c,R}-y\rVert_p=\lVert y-c\rVert_p-R. ∥ y c , R − c ∥ p = R , ∥ y c , R − y ∥ p = ∥ y − c ∥ p − R .
In particular, 2 L ( ∥ y c , R − c ∥ p − R ) + = 0 2L(\lVert y^{c,R}-c\rVert_p-R)_+=0 2 L (∥ y c , R − c ∥ p − R ) + = 0 . Moreover,
f ( y ) + 2 L ( ∥ y − c ∥ p − R ) + = f ( y ) + 2 L ( ∥ y − c ∥ p − R ) ≥ f ( y c , R ) − L ∥ y c , R − y ∥ p + 2 L ( ∥ y − c ∥ p − R ) = f ( y c , R ) + L ( ∥ y − c ∥ p − R ) , \begin{aligned}
f(y)+2L(\lVert y-c\rVert_p-R)_+&=f(y)+2L(\lVert y-c\rVert_p-R)\\
&\ge f(y^{c,R})-L\lVert y^{c,R}-y\rVert_p+2L(\lVert y-c\rVert_p-R)\\
&=f(y^{c,R})+L(\lVert y-c\rVert_p-R),
\end{aligned} f ( y ) + 2 L (∥ y − c ∥ p − R ) + = f ( y ) + 2 L (∥ y − c ∥ p − R ) ≥ f ( y c , R ) − L ∥ y c , R − y ∥ p + 2 L (∥ y − c ∥ p − R ) = f ( y c , R ) + L (∥ y − c ∥ p − R ) ,
where the inequality follows from the L L L -Lipschitz continuity of f f f . This yields the desired result, as it implies that the value of f ( x ) + 2 L ( ∥ x − c ∥ p − R ) + f(x) + 2L(\lVert x-c\rVert_p-R)_+ f ( x ) + 2 L (∥ x − c ∥ p − R ) + at any point outside the ball is larger than its value at some point satisfying ∥ x − c ∥ p ≤ R \lVert x-c\rVert_p \le R ∥ x − c ∥ p ≤ R , where f ( x ) + 2 L ( ∥ x − c ∥ p − R ) + = f ( x ) f(x) + 2L(\lVert x-c\rVert_p-R)_+ = f(x) f ( x ) + 2 L (∥ x − c ∥ p − R ) + = f ( x ) . Further, since the ball is compact and f f f is continuous, the minimum of f f f over the ball is attained. For the stochastic oracle, define
v ( x ) : = { 0 , ∥ x − c ∥ p ≤ R , ( ∣ x i − c i ∣ p − 2 ( x i − c i ) ) i = 1 d ∥ x − c ∥ p p − 1 , ∥ x − c ∥ p > R . v(x) :=
\begin{cases}
0, & \lVert x-c\rVert_p \le R,\\
\frac{\left(|x_i-c_i|^{p-2}(x_i-c_i)\right)_{i=1}^{d}}{\lVert x-c\rVert_p^{p-1}}, & \lVert x-c\rVert_p > R.
\end{cases} v ( x ) := ⎩ ⎨ ⎧ 0 , ∥ x − c ∥ p p − 1 ( ∣ x i − c i ∣ p − 2 ( x i − c i ) ) i = 1 d , ∥ x − c ∥ p ≤ R , ∥ x − c ∥ p > R .
Then v ( x ) ∈ ∂ [ ( ∥ ⋅ − c ∥ p − R ) + ] ( x ) v(x) \in\partial[(\lVert\cdot-c\rVert_p-R)_+](x) v ( x ) ∈ ∂ [(∥ ⋅ − c ∥ p − R ) + ] ( x ) and ∥ v ( x ) ∥ q ≤ 1 \lVert v(x)\rVert_q \le1 ∥ v ( x ) ∥ q ≤ 1 , so
E [ g ( x ) + 2 L v ( x ) ] ∈ ∂ [ f + 2 L ( ∥ ⋅ − c ∥ p − R ) + ] ( x ) . \mathbb{E}[g(x)+2Lv(x)] \in\partial\left[f+2L(\lVert\cdot-c\rVert_p-R)_+\right](x). E [ g ( x ) + 2 Lv ( x )] ∈ ∂ [ f + 2 L (∥ ⋅ − c ∥ p − R ) + ] ( x ) .
By Minkowski’s inequality,
( E ∥ g ( x ) + 2 L v ( x ) ∥ q q ) 1 / q ≤ ( E ∥ g ( x ) ∥ q q ) 1 / q + 2 L ∥ v ( x ) ∥ q ≤ 3 L . \left(\mathbb{E}\lVert g(x)+2Lv(x)\rVert_q^q\right)^{1/q} \le\left(\mathbb{E}\lVert g(x)\rVert_q^q\right)^{1/q}+2L\lVert v(x)\rVert_q \le3L. ( E ∥ g ( x ) + 2 Lv ( x ) ∥ q q ) 1/ q ≤ ( E ∥ g ( x ) ∥ q q ) 1/ q + 2 L ∥ v ( x ) ∥ q ≤ 3 L .
Computing g ( x ) + 2 L v ( x ) g(x)+2Lv(x) g ( x ) + 2 Lv ( x ) requires only one query to g g g .
Theorem 14 (Parallel EpochDA-based solver). BallAccel (Proposition 11) using parallel EpochDA (Algorithm 9) as a ball optimization oracle solves Problem 1 with expected error ϵ o p t \epsilon_{\mathrm{opt}} ϵ opt , with
O ( d p 2 ( p + 1 ) κ p p + 1 log 4 ( d κ ) ) query depth O\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\log^4(d\kappa)\right)\ \text{query depth} O ( d 2 ( p + 1 ) p κ p + 1 p log 4 ( d κ ) ) query depth and
O ( d p 2 ( p + 1 ) κ p p + 1 log 4 ( d κ ) + d p ( p − 2 ) 2 ( p + 1 ) κ p ( 2 p − 1 ) p + 1 log 2 p ( p − 1 ) + 2 ( d κ ) ) total queries , O\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\log^4(d\kappa)+d^{\frac{p(p-2)}{2(p+1)}}\kappa^{\frac{p(2p-1)}{p+1}}\log^{2p(p-1)+2}(d\kappa)\right)\ \text{total queries}, O ( d 2 ( p + 1 ) p κ p + 1 p log 4 ( d κ ) + d 2 ( p + 1 ) p ( p − 2 ) κ p + 1 p ( 2 p − 1 ) log 2 p ( p − 1 ) + 2 ( d κ ) ) total queries , and an additional computational cost of
O ( d p 2 ( p + 1 ) κ p p + 1 log 4 ( d κ ) + d p ( p − 2 ) 2 ( p + 1 ) κ p ( 2 p − 1 ) p + 1 log 2 p ( p − 1 ) + 2 ( d κ ) ) depth O\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\log^4(d\kappa)+d^{\frac{p(p-2)}{2(p+1)}}\kappa^{\frac{p(2p-1)}{p+1}}\log^{2p(p-1)+2}(d\kappa)\right)\ \text{depth} O ( d 2 ( p + 1 ) p κ p + 1 p log 4 ( d κ ) + d 2 ( p + 1 ) p ( p − 2 ) κ p + 1 p ( 2 p − 1 ) log 2 p ( p − 1 ) + 2 ( d κ ) ) depth and
O ( ( d p 2 ( p + 1 ) κ p p + 1 log 4 ( d κ ) + d p ( p − 2 ) 2 ( p + 1 ) κ p ( 2 p − 1 ) p + 1 log 2 p ( p − 1 ) + 2 ( d κ ) ) ⋅ d ) work . O\left(\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\log^4(d\kappa)+d^{\frac{p(p-2)}{2(p+1)}}\kappa^{\frac{p(2p-1)}{p+1}}\log^{2p(p-1)+2}(d\kappa)\right)\cdot d\right)\ \text{work}. O ( ( d 2 ( p + 1 ) p κ p + 1 p log 4 ( d κ ) + d 2 ( p + 1 ) p ( p − 2 ) κ p + 1 p ( 2 p − 1 ) log 2 p ( p − 1 ) + 2 ( d κ ) ) ⋅ d ) work .
Proof. Let x ⋆ ∈ arg min x ∈ B p ( R ) f ( x ) x^\star\in\mathop{\arg\min}_{x\in B_p(R)} f(x) x ⋆ ∈ arg min x ∈ B p ( R ) f ( x ) , x ^ \hat{x} x ^ be the point returned by BallAccel and define
f p e n ( x ) = f ( x ) + 2 L ( ∥ x ∥ p − R ) + . f_{\mathrm{pen}}(x)=f(x)+2L(\lVert x\rVert_p-R)_+. f pen ( x ) = f ( x ) + 2 L (∥ x ∥ p − R ) + .
Let F = ( f p e n ^ ) ρ F=(\widehat{f_{\mathrm{pen}}})_\rho F = ( f pen ) ρ and choose r = ρ = ϵ o p t C p L d r=\rho=\frac{\epsilon_{\mathrm{opt}}}{C_pL\sqrt{d}} r = ρ = C p L d ϵ opt , where C p C_p C p is a sufficiently large constant depending only on p p p . Since f p e n f_{\mathrm{pen}} f pen is 3 L 3L 3 L -Lipschitz and E ∥ ξ ρ ∥ p ≤ ρ d \mathbb{E}\lVert\xi_\rho\rVert_p\le\rho\sqrt{d} E ∥ ξ ρ ∥ p ≤ ρ d , this choice gives
∣ F ( x ) − f p e n ( x ) ∣ ≤ 3 L E ∥ ξ ρ ∥ p ≤ ϵ o p t 4 \lvert F(x)-f_{\mathrm{pen}}(x)\rvert\le3L\mathbb{E}\lVert\xi_\rho\rVert_p \le\frac{\epsilon_{\mathrm{opt}}}{4} ∣ F ( x ) − f pen ( x )∣ ≤ 3 L E ∥ ξ ρ ∥ p ≤ 4 ϵ opt
for every x ∈ R d x\in\mathbb{R}^d x ∈ R d . We now show that F F F has a minimizer in B p ( 3 R ) B_p(3R) B p ( 3 R ) . To see this, let y ∈ R d y\in\mathbb{R}^d y ∈ R d satisfy ∥ y ∥ p > 3 R \lVert y\rVert_p>3R ∥ y ∥ p > 3 R , and set y ~ = R ∥ y ∥ p y \tilde{y}=\frac{R}{\lVert y\rVert_p}y y ~ = ∥ y ∥ p R y . By Lemma 25 ,
F ( y ) ≥ f p e n ( y ) − ϵ o p t 4 ≥ f p e n ( y ~ ) + L ( ∥ y ∥ p − R ) − ϵ o p t 4 ≥ F ( y ~ ) + L ( ∥ y ∥ p − R ) − ϵ o p t 2 > F ( y ~ ) , \begin{aligned}
F(y)\ge f_{\mathrm{pen}}(y)-\frac{\epsilon_{\mathrm{opt}}}{4}\ge f_{\mathrm{pen}}(\tilde{y})+L(\lVert y\rVert_p-R)-\frac{\epsilon_{\mathrm{opt}}}{4} \\
&\ge F(\tilde{y})+L(\lVert y\rVert_p-R)-\frac{\epsilon_{\mathrm{opt}}}{2}>F(\tilde{y}),
\end{aligned} F ( y ) ≥ f pen ( y ) − 4 ϵ opt ≥ f pen ( y ~ ) + L (∥ y ∥ p − R ) − 4 ϵ opt ≥ F ( y ~ ) + L (∥ y ∥ p − R ) − 2 ϵ opt > F ( y ~ ) ,
where the last inequality uses ∥ y ∥ p > 3 R \lVert y\rVert_p > 3R ∥ y ∥ p > 3 R and ϵ o p t ≤ L R \epsilon_{\mathrm{opt}} \le LR ϵ opt ≤ L R . Since F F F is continuous, it attains its minimum over B p ( 3 R ) B_p(3R) B p ( 3 R ) , and the inequality shows that this is also a global minimum. Since F F F has a global minimizer in B p ( 3 R ) B_p(3R) B p ( 3 R ) , we use the unpenalized implementation in Proposition 11 with radius 3 R 3R 3 R , Lipschitz bound 3 L 3L 3 L , and target accuracy ϵ o p t / 2 \epsilon_{\mathrm{opt}}/2 ϵ opt /2 . Let g p e n g_{\mathrm{pen}} g pen be the stochastic subgradient oracle for f p e n f_{\mathrm{pen}} f pen from Lemma 25 . Each evaluation of g p e n g_{\mathrm{pen}} g pen uses one query to g g g , and E ∥ g p e n ( z ) ∥ q q ≤ ( 4 L ) q \mathbb{E}\lVert g_{\mathrm{pen}}(z)\rVert_q^q \le(4L)^q E ∥ g pen ( z ) ∥ q q ≤ ( 4 L ) q . Thus, with L p e n = 4 L L_{\mathrm{pen}}=4L L pen = 4 L , Proposition 13 implements each ball optimization call for F F F using Algorithm 9 on ( f p e n , g p e n ) (f_{\mathrm{pen}},g_{\mathrm{pen}}) ( f pen , g pen ) . The local subproblem has objective
x ↦ F ( x ) + λ p ∥ x − x ˉ ∥ p p , x ∈ B p ( x ˉ , r ) . x \mapsto F(x) + \frac{\lambda}{p}\lVert x-\bar{x}\rVert_p^p,\qquad x\in B_p(\bar{x},r). x ↦ F ( x ) + p λ ∥ x − x ˉ ∥ p p , x ∈ B p ( x ˉ , r ) .
Rescaling the Lipschitz parameter, radius, and target accuracy changes the corresponding BallAccel parameters only by constant factors depending on p p p . Hence, we retain the notation
κ = L R ϵ o p t , K = ( R r ) p p + 1 = Θ ( d p 2 ( p + 1 ) κ p p + 1 ) , λ ⋆ = ϵ o p t K p R p log p κ \kappa=\frac{LR}{\epsilon_{\mathrm{opt}}},\qquad K=\left(\frac{R}{r}\right)^{\frac{p}{p+1}}=\Theta\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\right),\qquad\lambda^\star=\frac{\epsilon_{\mathrm{opt}}K^p}{Rp}\log^p\kappa κ = ϵ opt L R , K = ( r R ) p + 1 p = Θ ( d 2 ( p + 1 ) p κ p + 1 p ) , λ ⋆ = R p ϵ opt K p log p κ
in the following complexity bounds, with constant-factor changes absorbed into the O ( ⋅ ) O(\cdot) O ( ⋅ ) notation. For the query depth, each ball optimization oracle implemented by Algorithm 9 has constant query depth. Proposition 11 requires at most O ( K log κ log ( R κ / r ) ) O(K\log\kappa\log(R\kappa/r)) O ( K log κ log ( R κ / r )) iterations, with at most O ( log 2 ( R κ / r ) ) O(\log^2(R\kappa/r)) O ( log 2 ( R κ / r )) ball optimization oracle calls per iteration. Since log ( R κ / r ) = O ( log ( d κ ) ) \log(R\kappa/r)=O(\log(d\kappa)) log ( R κ / r ) = O ( log ( d κ )) , the query depth is
O ( K log κ log 3 ( R κ r ) ) = O ( d p 2 ( p + 1 ) κ p p + 1 log 4 ( d κ ) ) . O\left(K\log\kappa\log^3\left(\frac{R\kappa}{r}\right)\right)=O\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\log^4(d\kappa)\right). O ( K log κ log 3 ( r R κ ) ) = O ( d 2 ( p + 1 ) p κ p + 1 p log 4 ( d κ ) ) .
By Proposition 13 with L p e n = 4 L L_{\mathrm{pen}}=4L L pen = 4 L , each ( ϕ , λ ) (\phi,\lambda) ( ϕ , λ ) -BROO uses O ( 1 + L p λ ϕ p − 1 ) O\left(1+\frac{L^p}{\lambda\phi^{p-1}}\right) O ( 1 + λ ϕ p − 1 L p ) queries to g g g . We may upper bound every query cost using λ = Ω p ( λ ⋆ ) \lambda=\Omega_p(\lambda^\star) λ = Ω p ( λ ⋆ ) . Writing ℓ : = log ( R κ / r ) \ell:=\log(R\kappa/r) ℓ := log ( R κ / r ) and j max : = ⌈ log 2 K + C b a ⌉ j_{\max}:=\lceil\log_2K+C_{\mathrm{ba}}\rceil j m a x := ⌈ log 2 K + C ba ⌉ , and applying both requirements of Proposition 11, the total number of stochastic-gradient queries is at most
O ( K log κ ℓ 3 + K log κ ℓ 3 L p ( λ ⋆ ) p r p ( p − 1 ) + K log κ ℓ ∑ j = 1 j max 2 − j L p ( 2 j ℓ 2 p ) p − 1 ( λ ⋆ ) p r p ( p − 1 ) ) = O ( K log κ ℓ 3 + K log κ ℓ 3 L p ( λ ⋆ ) p r p ( p − 1 ) + K log κ ℓ 2 p ( p − 1 ) + 1 L p ( λ ⋆ ) p r p ( p − 1 ) ∑ j = 1 j max 2 j ( p − 2 ) ) . \begin{aligned}
O\left(K\log\kappa\,\ell^3+\frac{K\log\kappa\,\ell^3L^p}{(\lambda^\star)^pr^p(p-1)}+K\log\kappa\,\ell\sum_{j=1}^{j_{\max}}2^{-j}\frac{L^p(2^j\ell^{2p})^{p-1}}{(\lambda^\star)^pr^p(p-1)}\right) \\
&=O\left(K\log\kappa\,\ell^3+\frac{K\log\kappa\,\ell^3L^p}{(\lambda^\star)^pr^p(p-1)}+\frac{K\log\kappa\,\ell^{2p(p-1)+1}L^p}{(\lambda^\star)^pr^p(p-1)}\sum_{j=1}^{j_{\max}}2^{j(p-2)}\right).
\end{aligned} O ( K log κ ℓ 3 + ( λ ⋆ ) p r p ( p − 1 ) K log κ ℓ 3 L p + K log κ ℓ j = 1 ∑ j m a x 2 − j ( λ ⋆ ) p r p ( p − 1 ) L p ( 2 j ℓ 2 p ) p − 1 ) = O ( K log κ ℓ 3 + ( λ ⋆ ) p r p ( p − 1 ) K log κ ℓ 3 L p + ( λ ⋆ ) p r p ( p − 1 ) K log κ ℓ 2 p ( p − 1 ) + 1 L p j = 1 ∑ j m a x 2 j ( p − 2 ) ) .
For p ≥ 2 p\ge2 p ≥ 2 , the sum contains O ( ℓ ) O(\ell) O ( ℓ ) terms, each bounded by O ( K p − 2 ) O(K^{p-2}) O ( K p − 2 ) , and hence
∑ j = 1 j max 2 j ( p − 2 ) = O ( K p − 2 ℓ ) . \sum_{j=1}^{j_{\max}}2^{j(p-2)}=O(K^{p-2}\ell). j = 1 ∑ j m a x 2 j ( p − 2 ) = O ( K p − 2 ℓ ) .
Moreover, by K p + 1 r p = R p K^{p+1}r^p=R^p K p + 1 r p = R p ,
K L p ( λ ⋆ ) p r p ( p − 1 ) = κ p log p 2 κ . \frac{KL^p}{(\lambda^\star)^pr^p(p-1)}=\frac{\kappa^p}{\log^{p^2}\kappa}. ( λ ⋆ ) p r p ( p − 1 ) K L p = log p 2 κ κ p .
Therefore, the total number of queries is bounded by
O ( K log κ ℓ 3 + κ p log κ ℓ 3 log p 2 κ + κ p K p − 2 log κ ℓ 2 p ( p − 1 ) + 2 log p 2 κ ) = O ( K log 4 ( d κ ) + κ p K p − 2 log 2 p ( p − 1 ) + 2 ( d κ ) ) = O ( d p 2 ( p + 1 ) κ p p + 1 log 4 ( d κ ) + d p ( p − 2 ) 2 ( p + 1 ) κ p ( 2 p − 1 ) p + 1 log 2 p ( p − 1 ) + 2 ( d κ ) ) , \begin{aligned}
O\left(K\log\kappa\,\ell^3+\frac{\kappa^p\log\kappa\,\ell^3}{\log^{p^2}\kappa}+\frac{\kappa^pK^{p-2}\log\kappa\,\ell^{2p(p-1)+2}}{\log^{p^2}\kappa}\right) \\
&=O\left(K\log^4(d\kappa)+\kappa^pK^{p-2}\log^{2p(p-1)+2}(d\kappa)\right) \\
&=O\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\log^4(d\kappa)+d^{\frac{p(p-2)}{2(p+1)}}\kappa^{\frac{p(2p-1)}{p+1}}\log^{2p(p-1)+2}(d\kappa)\right),
\end{aligned} O ( K log κ ℓ 3 + log p 2 κ κ p log κ ℓ 3 + log p 2 κ κ p K p − 2 log κ ℓ 2 p ( p − 1 ) + 2 ) = O ( K log 4 ( d κ ) + κ p K p − 2 log 2 p ( p − 1 ) + 2 ( d κ ) ) = O ( d 2 ( p + 1 ) p κ p + 1 p log 4 ( d κ ) + d 2 ( p + 1 ) p ( p − 2 ) κ p + 1 p ( 2 p − 1 ) log 2 p ( p − 1 ) + 2 ( d κ ) ) ,
where we used log κ / log p 2 ≤ 1 \log\kappa/ \log p^2 \le1 log κ / log p 2 ≤ 1 and κ p K p − 2 = Θ ( d p ( p − 2 ) 2 ( p + 1 ) κ p ( 2 p − 1 ) p + 1 ) \kappa^p K^{p-2} = \Theta\left(d^{\frac{p(p-2)}{2(p+1)}}\kappa^{\frac{p(2p-1)}{p+1}}\right) κ p K p − 2 = Θ ( d 2 ( p + 1 ) p ( p − 2 ) κ p + 1 p ( 2 p − 1 ) ) . Finally, Proposition 11 gives
E F ( x ^ ) ≤ min x ∈ R d F ( x ) + ϵ o p t 2 ≤ F ( x ⋆ ) + ϵ o p t 2 . \mathbb{E}F(\widehat{x}) \le\min_{x\in\mathbb{R}^d} F(x) + \frac{\epsilon_{\mathrm{opt}}}{2} \le F(x^\star) + \frac{\epsilon_{\mathrm{opt}}}{2}. E F ( x ) ≤ x ∈ R d min F ( x ) + 2 ϵ opt ≤ F ( x ⋆ ) + 2 ϵ opt .
Since f ( x ) ≤ f p e n ( x ) f(x) \le f_{\mathrm{pen}}(x) f ( x ) ≤ f pen ( x ) and f p e n f_{\mathrm{pen}} f pen is 3 L 3L 3 L -Lipschitz,
f ( x ) ≤ F ( x ) + 3 L E ∥ ξ ρ ∥ p f(x) \le F(x) + 3L\mathbb{E}\|\xi_\rho\|_p f ( x ) ≤ F ( x ) + 3 L E ∥ ξ ρ ∥ p
for all x ∈ R d x \in\mathbb{R}^d x ∈ R d . Therefore,
E f ( x ^ ) ≤ E F ( x ^ ) + 3 L E ∥ ξ ρ ∥ p ≤ F ( x ⋆ ) + ϵ o p t 2 + 3 L E ∥ ξ ρ ∥ p ≤ f ( x ⋆ ) + ϵ o p t 2 + 6 L E ∥ ξ ρ ∥ p ≤ f ( x ⋆ ) + ϵ o p t . \begin{aligned}
\mathbb{E}f(\widehat{x}) &\le\mathbb{E}F(\widehat{x}) + 3L\mathbb{E}\|\xi_\rho\|_p \\
&\le F(x^\star) + \frac{\epsilon_{\mathrm{opt}}}{2} + 3L\mathbb{E}\|\xi_\rho\|_p \\
&\le f(x^\star) + \frac{\epsilon_{\mathrm{opt}}}{2} + 6L\mathbb{E}\|\xi_\rho\|_p \\
&\le f(x^\star) + \epsilon_{\mathrm{opt}}.
\end{aligned} E f ( x ) ≤ E F ( x ) + 3 L E ∥ ξ ρ ∥ p ≤ F ( x ⋆ ) + 2 ϵ opt + 3 L E ∥ ξ ρ ∥ p ≤ f ( x ⋆ ) + 2 ϵ opt + 6 L E ∥ ξ ρ ∥ p ≤ f ( x ⋆ ) + ϵ opt .
p-AC-SA Solver
Algorithm 10 ℓ p \ell_p ℓ p -AC-SA(f , x ˉ , r , ρ , λ , ϕ ˉ f,\bar{x},r,\rho,\lambda,\bar{\phi} f , x ˉ , r , ρ , λ , ϕ ˉ )
Require: f : R d → R f:\mathbb{R}^{d}\to\mathbb{R} f : R d → R and g : R d → R d g:\mathbb{R}^{d}\to\mathbb{R}^{d} g : R d → R d satisfying the assumptions of Problem 1, x ˉ ∈ X \bar{x}\in\mathcal{X} x ˉ ∈ X , r , ρ , λ > 0 r,\rho,\lambda>0 r , ρ , λ > 0 , 0 < ϕ ˉ ≤ λ r p / 2 0<\bar{\phi}\leq\lambda r^{p}/2 0 < ϕ ˉ ≤ λ r p /2 , V ( x , y ) = D ∥ ⋅ − x ˉ ∥ p p / p ( y , x ) V(x,y)=D_{\lVert\cdot-\bar{x}\rVert_p^p/p}(y,x) V ( x , y ) = D ∥ ⋅− x ˉ ∥ p p / p ( y , x ) and ν = 2 2 − p \nu=2^{2-p} ν = 2 2 − p .
K ← ⌈ log 2 ( λ r p / ϕ ˉ ) ⌉ K\leftarrow\lceil\log_{2}(\lambda r^{p}/\bar{\phi})\rceil K ← ⌈ log 2 ( λ r p / ϕ ˉ )⌉ .
For k ∈ [ K ] k\in[K] k ∈ [ K ] , set
M f , k ← ( ( C p L / ρ + ( p − 1 ) λ r p − 2 ) ν λ ( λ r p / 2 k − 1 ) p ) 1 / ( p + 2 ) , M_{f,k}\leftarrow\left((C_pL/\rho+(p-1)\lambda r^{p-2})\nu\lambda\left(\lambda r^p/2^{k-1}\right)^p\right)^{1/(p+2)}, M f , k ← ( ( C p L / ρ + ( p − 1 ) λ r p − 2 ) ν λ ( λ r p / 2 k − 1 ) p ) 1/ ( p + 2 ) ,
T k ← ⌈ C p [ 1 + ( ( C p L / ρ + ( p − 1 ) λ r p − 2 ) p ( ν λ ) 2 ( λ r p / 2 k − 1 ) p − 2 ) 1 / ( p + 2 ) ] ⌉ , T_k\leftarrow\left\lceil C_p\left[1+\left(\frac{(C_pL/\rho+(p-1)\lambda r^{p-2})^p}{(\nu\lambda)^2(\lambda r^p/2^{k-1})^{p-2}}\right)^{1/(p+2)}\right]\right\rceil, T k ← ⌈ C p [ 1 + ( ( ν λ ) 2 ( λ r p / 2 k − 1 ) p − 2 ( C p L / ρ + ( p − 1 ) λ r p − 2 ) p ) 1/ ( p + 2 ) ] ⌉ ,
N k ← ⌈ C p max { 1 , L p ν λ T k ( λ r p / 2 k − 1 ) p − 1 } ⌉ . N_k\leftarrow\left\lceil C_p\max\left\{1,\frac{L^p}{\nu\lambda T_k(\lambda r^p/2^{k-1})^{p-1}}\right\}\right\rceil. N k ← ⌈ C p max { 1 , ν λ T k ( λ r p / 2 k − 1 ) p − 1 L p } ⌉ .
Sample ξ i ∼ G G p ( 0 , ρ 2 I d ) \xi_i\sim\mathcal{GG}_p(0,\rho^2I_d) ξ i ∼ GG p ( 0 , ρ 2 I d ) , i ∈ [ N ] i\in[N] i ∈ [ N ] independently, for N = ∑ k ∈ [ K ] T k N k N=\sum_{k\in[K]}T_kN_k N = ∑ k ∈ [ K ] T k N k .
Query g ( x ˉ + ξ i ) g(\bar{x}+\xi_i) g ( x ˉ + ξ i ) for all i ∈ [ N ] i\in[N] i ∈ [ N ] in parallel.
x 0 a g ← x ˉ x^{\mathrm{ag}}_0\leftarrow\bar{x} x 0 ag ← x ˉ , x 0 ← x ˉ x_0\leftarrow\bar{x} x 0 ← x ˉ .
for k ∈ [ K ] k\in[K] k ∈ [ K ] do
for t ∈ [ T k ] t\in[T_k] t ∈ [ T k ] do
α t ← p / ( t + p − 1 ) \alpha_t\leftarrow p/(t+p-1) α t ← p / ( t + p − 1 ) , Γ t ← Γ ( p + 1 ) Γ ( t ) / Γ ( t + p ) \Gamma_t\leftarrow\Gamma(p+1)\Gamma(t)/\Gamma(t+p) Γ t ← Γ ( p + 1 ) Γ ( t ) /Γ ( t + p ) .
γ t , k ← 2 ( sup s ≥ 1 α s p / Γ s ) ( C p L / ρ + ( p − 1 ) λ r p − 2 ) p − 1 ν M f , k p − 2 Γ t . \gamma_{t,k}\leftarrow\frac{2\left(\sup_{s\geq1}\alpha_s^p/\Gamma_s\right)(C_pL/\rho+(p-1)\lambda r^{p-2})^{p-1}}{\nu M_{f,k}^{p-2}}\Gamma_t. γ t , k ← ν M f , k p − 2 2 ( sup s ≥ 1 α s p / Γ s ) ( C p L / ρ + ( p − 1 ) λ r p − 2 ) p − 1 Γ t .
x t m d ← ( 1 − α t ) ( λ + γ t , k ) γ t , k + ( 1 − α t ) 2 λ x t − 1 a g + α t [ ( 1 − α t ) λ + γ t , k ] γ t , k + ( 1 − α t ) 2 λ x t − 1 . x_t^{\mathrm{md}}\leftarrow\frac{(1-\alpha_t)(\lambda+\gamma_{t,k})}{\gamma_{t,k}+(1-\alpha_t)^2\lambda}x^{\mathrm{ag}}_{t-1}+\frac{\alpha_t[(1-\alpha_t)\lambda+\gamma_{t,k}]}{\gamma_{t,k}+(1-\alpha_t)^2\lambda}x_{t-1}. x t md ← γ t , k + ( 1 − α t ) 2 λ ( 1 − α t ) ( λ + γ t , k ) x t − 1 ag + γ t , k + ( 1 − α t ) 2 λ α t [( 1 − α t ) λ + γ t , k ] x t − 1 .
N T , [ k − 1 ] ← ∑ k ′ ∈ [ k − 1 ] T k ′ N k ′ N_{T,[k-1]}\leftarrow\sum_{k'\in[k-1]}T_{k'}N_{k'} N T , [ k − 1 ] ← ∑ k ′ ∈ [ k − 1 ] T k ′ N k ′ .
∇ ^ f ( x t m d ) ← 1 N k ∑ n ∈ [ N k ] γ ρ , p ( x t m d − x ˉ − ξ N T , [ k − 1 ] + ( t − 1 ) N k + n ) γ ρ , p ( ξ N T , [ k − 1 ] + ( t − 1 ) N k + n ) ⋅ g ( x ˉ + ξ N T , [ k − 1 ] + ( t − 1 ) N k + n ) . \widehat{\nabla}f(x_t^{\mathrm{md}})\leftarrow\frac{1}{N_k}\sum_{n\in[N_k]}\frac{\gamma_{\rho,p}\left(x_t^{\mathrm{md}}-\bar{x}-\xi_{N_{T,[k-1]}+(t-1)N_k+n}\right)}{\gamma_{\rho,p}\left(\xi_{N_{T,[k-1]}+(t-1)N_k+n}\right)}\cdot g\left(\bar{x}+\xi_{N_{T,[k-1]}+(t-1)N_k+n}\right). ∇ f ( x t md ) ← N k 1 n ∈ [ N k ] ∑ γ ρ , p ( ξ N T , [ k − 1 ] + ( t − 1 ) N k + n ) γ ρ , p ( x t md − x ˉ − ξ N T , [ k − 1 ] + ( t − 1 ) N k + n ) ⋅ g ( x ˉ + ξ N T , [ k − 1 ] + ( t − 1 ) N k + n ) .
x t ← arg min x ∈ X ∩ B p ( x ˉ , r ) { ⟨ α t [ ∇ ^ f ( x t m d ) + λ J p ( x t m d − x ˉ ) ] , x ⟩ + α t λ V ( x t m d , x ) + [ ( 1 − α t ) λ + γ t , k ] V ( x t − 1 , x ) } . x_t\leftarrow\arg\min_{x\in\mathcal{X}\cap B_p(\bar{x},r)}
\left\{\left\langle\alpha_t[\widehat{\nabla}f(x_t^{\mathrm{md}})+\lambda J_p(x_t^{\mathrm{md}}-\bar{x})],x\right\rangle+\alpha_t\lambda V(x_t^{\mathrm{md}},x)+[(1-\alpha_t)\lambda+\gamma_{t,k}]V(x_{t-1},x)\right\}. x t ← arg x ∈ X ∩ B p ( x ˉ , r ) min { ⟨ α t [ ∇ f ( x t md ) + λ J p ( x t md − x ˉ )] , x ⟩ + α t λV ( x t md , x ) + [( 1 − α t ) λ + γ t , k ] V ( x t − 1 , x ) } .
x t a g ← α t x t + ( 1 − α t ) x t − 1 a g x_t^{\mathrm{ag}}\leftarrow\alpha_tx_t+(1-\alpha_t)x^{\mathrm{ag}}_{t-1} x t ag ← α t x t + ( 1 − α t ) x t − 1 ag .
end for
x 0 a g ← x T k x_0^{\mathrm{ag}}\leftarrow x_{T_k} x 0 ag ← x T k , x 0 ← x T k a g x_0\leftarrow x_{T_k}^{\mathrm{ag}} x 0 ← x T k ag .
end for
return x T K a g x_{T_K}^{\mathrm{ag}} x T K ag
Definition 9. Consider the class of stochastic convex optimization (SCO) problems
Ψ ∗ : = min x ∈ X { Ψ ( x ) : = f ( x ) + χ ( x ) } , (57) \Psi^*:=\min_{x\in\mathcal{X}}\{\Psi(x):=f(x)+\chi(x)\}, \tag*{(57)} Ψ ∗ := x ∈ X min { Ψ ( x ) := f ( x ) + χ ( x )} , (57)
where X ⊆ R d X \subseteq\mathbb{R}^{d} X ⊆ R d is a nonempty closed convex set, X : X → R \mathcal{X}: X \to\mathbb{R} X : X → R is a simple convex function, and f : X → R f: X \to\mathbb{R} f : X → R is convex. Suppose ω : X → R \omega: X \to\mathbb{R} ω : X → R is differentiable and its Bregman divergence
V ( x , y ) : = ω ( y ) − ω ( x ) − ⟨ ∇ ω ( x ) , y − x ⟩ V(x,y) := \omega(y) - \omega(x) - \langle\nabla\omega(x), y-x\rangle V ( x , y ) := ω ( y ) − ω ( x ) − ⟨ ∇ ω ( x ) , y − x ⟩
satisfies V ( x , y ) ≥ ν ∥ y − x ∥ p p / p V(x,y) \ge\nu\lVert y-x\rVert_{p}^{p}/p V ( x , y ) ≥ ν ∥ y − x ∥ p p / p for some ν > 0 \nu> 0 ν > 0 . Assume that λ ≥ 0 \lambda\ge0 λ ≥ 0 , μ ≥ 0 \mu\ge0 μ ≥ 0 , L ≥ 0 L \ge0 L ≥ 0 , and M f > 0 M_f > 0 M f > 0 , and that, for every x , y ∈ X x,y \in X x , y ∈ X and f ′ ( x ) ∈ ∂ f ( x ) f'(x) \in\partial f(x) f ′ ( x ) ∈ ∂ f ( x ) ,
μ V ( x , y ) ≤ f ( y ) − f ( x ) − ⟨ f ′ ( x ) , y − x ⟩ ≤ L 2 ∥ y − x ∥ p 2 + M f ∥ y − x ∥ p . (58) \mu V(x,y) \le f(y) - f(x) - \langle f'(x), y-x\rangle\le\frac{L}{2}\lVert y-x\rVert_{p}^{2} + M_f\lVert y-x\rVert_{p}. \tag*{(58)} μ V ( x , y ) ≤ f ( y ) − f ( x ) − ⟨ f ′ ( x ) , y − x ⟩ ≤ 2 L ∥ y − x ∥ p 2 + M f ∥ y − x ∥ p . (58)
Define the lower model
l Ψ ( z , x ) : = f ( z ) + ⟨ f ′ ( z ) , x − z ⟩ + μ V ( z , x ) + X ( x ) . (59) l_{\Psi}(z,x) := f(z) + \langle f'(z),x-z\rangle+ \mu V(z,x) + \mathcal{X}(x). \tag*{(59)} l Ψ ( z , x ) := f ( z ) + ⟨ f ′ ( z ) , x − z ⟩ + μ V ( z , x ) + X ( x ) . (59)
Lemma 26. Let the convex function p : X → R p: X \to\mathbb{R} p : X → R , the points x ~ , y ~ ∈ X \tilde{x},\tilde{y} \in X x ~ , y ~ ∈ X and the scalars μ 1 , μ 2 ≥ 0 \mu_{1},\mu_{2} \ge0 μ 1 , μ 2 ≥ 0 be given. Let ω : X → R \omega: X \to\mathbb{R} ω : X → R be a differentiable convex function and V ( x , z ) V(x,z) V ( x , z ) be defined in Definition 9 (Bregman divergence). If
u ∗ ∈ Argmin { p ( u ) + μ 1 V ( x ~ , u ) + μ 2 V ( y ~ , u ) : u ∈ X } , u^{*} \in\operatorname{Argmin}\{p(u) + \mu_{1}V(\tilde{x},u) + \mu_{2}V(\tilde{y},u): u \in X\}, u ∗ ∈ Argmin { p ( u ) + μ 1 V ( x ~ , u ) + μ 2 V ( y ~ , u ) : u ∈ X } , then for any u ∈ X u \in X u ∈ X , we have
p ( u ∗ ) + μ 1 V ( x ~ , u ∗ ) + μ 2 V ( y ~ , u ∗ ) ≤ p ( u ) + μ 1 V ( x ~ , u ) + μ 2 V ( y ~ , u ) − ( μ 1 + μ 2 ) V ( u ∗ , u ) . p(u^{*}) + \mu_{1}V(\tilde{x},u^{*}) + \mu_{2}V(\tilde{y},u^{*}) \le p(u) + \mu_{1}V(\tilde{x},u) + \mu_{2}V(\tilde{y},u) - (\mu_{1}+\mu_{2})V(u^{*},u). p ( u ∗ ) + μ 1 V ( x ~ , u ∗ ) + μ 2 V ( y ~ , u ∗ ) ≤ p ( u ) + μ 1 V ( x ~ , u ) + μ 2 V ( y ~ , u ) − ( μ 1 + μ 2 ) V ( u ∗ , u ) .
Proof. The proof is the same as in [23 ] , it holds for any differentiable convex function ω \omega ω .
Lemma 27. Let x t a g : = ( 1 − α t ) x t − 1 a g + α t x t x_{t}^{\mathrm{ag}} := (1-\alpha_t)x_{t-1}^{\mathrm{ag}} + \alpha_t x_t x t ag := ( 1 − α t ) x t − 1 ag + α t x t for some α t ∈ [ 0 , 1 ] \alpha_t \in[0,1] α t ∈ [ 0 , 1 ] and ( x t − 1 a g , x t ) ∈ X × X (x_{t-1}^{\mathrm{ag}},x_t) \in X \times X ( x t − 1 ag , x t ) ∈ X × X . Suppose that f f f satisfies the upper-model bound in (58) . Then, for any z ∈ X z \in X z ∈ X and any f ′ ( z ) ∈ ∂ f ( z ) f'(z) \in\partial f(z) f ′ ( z ) ∈ ∂ f ( z ) , we have
Ψ ( x t a g ) ≤ ( 1 − α t ) Ψ ( x t − 1 a g ) + α t [ f ( z ) + ⟨ f ′ ( z ) , x t − z ⟩ + X ( x t ) ] + L 2 ∥ x t a g − z ∥ p 2 + M f ∥ x t a g − z ∥ p . \Psi(x_{t}^{\mathrm{ag}}) \le(1-\alpha_t)\Psi(x_{t-1}^{\mathrm{ag}}) + \alpha_t\left[f(z) + \langle f'(z),x_t-z\rangle+ \mathcal{X}(x_t)\right] + \frac{L}{2}\lVert x_{t}^{\mathrm{ag}}-z\rVert_{p}^{2} + M_f\lVert x_{t}^{\mathrm{ag}}-z\rVert_{p}. Ψ ( x t ag ) ≤ ( 1 − α t ) Ψ ( x t − 1 ag ) + α t [ f ( z ) + ⟨ f ′ ( z ) , x t − z ⟩ + X ( x t ) ] + 2 L ∥ x t ag − z ∥ p 2 + M f ∥ x t ag − z ∥ p .
Proof. The proof is the same as in [23 ] .
Let F t − 1 : = σ ( ξ 1 , … , ξ t − 1 ) \mathcal{F}_{t-1} := \sigma(\xi_1,\ldots,\xi_{t-1}) F t − 1 := σ ( ξ 1 , … , ξ t − 1 ) denote the oracle history, with F 0 \mathcal{F}_0 F 0 trivial. Assume that x t m d x_{t}^{\mathrm{md}} x t md is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable and that the stochastic oracle has a selected subgradient g ( x ) ∈ ∂ f ( x ) g(x) \in\partial f(x) g ( x ) ∈ ∂ f ( x ) satisfying
E [ G ( x t m d , ξ t ) ∣ F t − 1 ] = g ( x t m d ) , E [ ∥ G ( x t m d , ξ t ) − g ( x t m d ) ∥ q q ∣ F t − 1 ] ≤ σ q . (60) \mathbb{E}[G(x_{t}^{\mathrm{md}},\xi_t)\mid\mathcal{F}_{t-1}] = g(x_{t}^{\mathrm{md}}), \qquad\mathbb{E}[\lVert G(x_{t}^{\mathrm{md}},\xi_t)-g(x_{t}^{\mathrm{md}})\rVert_{q}^{q}\mid\mathcal{F}_{t-1}] \le\sigma^{q}. \tag*{(60)} E [ G ( x t md , ξ t ) ∣ F t − 1 ] = g ( x t md ) , E [∥ G ( x t md , ξ t ) − g ( x t md ) ∥ q q ∣ F t − 1 ] ≤ σ q . (60)
Throughout the proof use f ′ ( x t m d ) = g ( x t m d ) f'(x_{t}^{\mathrm{md}})=g(x_{t}^{\mathrm{md}}) f ′ ( x t md ) = g ( x t md ) and write
G t : = G ( x t m d , ξ t ) , δ t : = G t − g ( x t m d ) . (61) G_t := G(x_{t}^{\mathrm{md}},\xi_t), \qquad\delta_t := G_t-g(x_{t}^{\mathrm{md}}). \tag*{(61)} G t := G ( x t md , ξ t ) , δ t := G t − g ( x t md ) . (61)
The following proposition establishes a basic recursion for the generic AC-SA algorithm.
Proposition 15. Let ( x t − 1 , x t − 1 a g ) ∈ X × X (x_{t-1},x_{t-1}^{\mathrm{ag}}) \in X \times X ( x t − 1 , x t − 1 ag ) ∈ X × X be given. Also let ( x t m d , x t , x t a g ) ∈ X × X × X (x_{t}^{\mathrm{md}},x_t,x_{t}^{\mathrm{ag}}) \in X \times X \times X ( x t md , x t , x t ag ) ∈ X × X × X be computed according to (3.1), (3.2) and (3.3) in [23 ] . If K t : = ν ( μ + γ t ) − ( L p − 1 / M f p − 2 ) α t q > 0 K_t := \nu(\mu+\gamma_t) - (L^{p-1}/M_f^{p-2})\alpha_t^{q} > 0 K t := ν ( μ + γ t ) − ( L p − 1 / M f p − 2 ) α t q > 0 , then for any x ∈ X x \in X x ∈ X , we have
Ψ ( x t a g ) + μ V ( x t , x ) ≤ ( 1 − α t ) [ Ψ ( x t − 1 a g ) + μ V ( x t − 1 , x ) ] + α t l Ψ ( x t m d , x ) + γ t [ V ( x t − 1 , x ) − V ( x t , x ) ] + Δ t ( x ) , (62) \Psi(x_{t}^{\mathrm{ag}}) + \mu V(x_t,x) \le(1-\alpha_t)\left[\Psi(x_{t-1}^{\mathrm{ag}}) + \mu V(x_{t-1},x)\right] + \alpha_t l_{\Psi}(x_{t}^{\mathrm{md}},x) + \gamma_t\left[V(x_{t-1},x)-V(x_t,x)\right] + \Delta_t(x), \tag*{(62)} Ψ ( x t ag ) + μ V ( x t , x ) ≤ ( 1 − α t ) [ Ψ ( x t − 1 ag ) + μ V ( x t − 1 , x ) ] + α t l Ψ ( x t md , x ) + γ t [ V ( x t − 1 , x ) − V ( x t , x ) ] + Δ t ( x ) , (62) where l Ψ ( z , x ) l_{\Psi}(z,x) l Ψ ( z , x ) is defined in (59) ,
Δ t ( x ) : = α t ⟨ δ t , x − x t − 1 + ⟩ + α t q ( 2 M f + ∥ δ t ∥ q ) q q K t q − 1 , (63) \Delta_t(x) := \alpha_t\langle\delta_t,x-x_{t-1}^{+}\rangle+ \frac{\alpha_t^{q}(2M_f+\lVert\delta_t\rVert_q)^{q}}{qK_t^{q-1}}, \tag*{(63)} Δ t ( x ) := α t ⟨ δ t , x − x t − 1 + ⟩ + q K t q − 1 α t q ( 2 M f + ∥ δ t ∥ q ) q , (63) x t − 1 + : = α t μ μ + γ t x t m d + ( 1 − α t ) μ + γ t μ + γ t x t − 1 . (64) x_{t-1}^{+} := \frac{\alpha_t\mu}{\mu+\gamma_t}x_{t}^{\mathrm{md}} + \frac{(1-\alpha_t)\mu+\gamma_t}{\mu+\gamma_t}x_{t-1}. \tag*{(64)} x t − 1 + := μ + γ t α t μ x t md + μ + γ t ( 1 − α t ) μ + γ t x t − 1 . (64)
Proof. We first establish some basic relations among the search points x t a g x_t^{ag} x t a g , x t m d x_t^{md} x t m d and x t x_t x t . Denote d t : = x t a g − x t m d d_t := x_t^{ag} - x_t^{md} d t := x t a g − x t m d . It follows from (3.1) and (3.3) in [23 ] that
d t = α t x t + ( 1 − α t ) x t − 1 a g − x t m d = α t ( x t − α t μ μ + γ t x t m d − ( 1 − α t ) μ + γ t μ + γ t x t − 1 ) = α t ( x t − x t − 1 + ) . (65) \begin{aligned}
d_t &= \alpha_t x_t + (1-\alpha_t)x_{t-1}^{ag} - x_t^{md}\\
&= \alpha_t\left(x_t-\frac{\alpha_t\mu}{\mu+\gamma_t}x_t^{md}-\frac{(1-\alpha_t)\mu+\gamma_t}{\mu+\gamma_t}x_{t-1}\right)=\alpha_t(x_t-x_{t-1}^{+}). \tag*{(65)}
\end{aligned} d t = α t x t + ( 1 − α t ) x t − 1 a g − x t m d = α t ( x t − μ + γ t α t μ x t m d − μ + γ t ( 1 − α t ) μ + γ t x t − 1 ) = α t ( x t − x t − 1 + ) . (65)
which, in view of the convexity of ∥ ⋅ ∥ p p \lVert\cdot\rVert_p^p ∥ ⋅ ∥ p p and the uniform convexity of ω \omega ω , implies that
ν ( μ + γ t ) p ρ α t p ∥ d t ∥ p p ≤ ν ( μ + γ t ) p ( μ + γ t ) [ α t μ ∥ x t − x t m d ∥ p p + ( ( 1 − α t ) μ + γ t ) ∥ x t − x t − 1 ∥ p p ] ≤ α t μ V ( x t m d , x t ) + ( ( 1 − α t ) μ + γ t ) V ( x t − 1 , x t ) . (66) \frac{\nu(\mu+\gamma_t)}{p\rho\alpha_t^p}\lVert d_t\rVert_p^p\leq\frac{\nu(\mu+\gamma_t)}{p(\mu+\gamma_t)}\left[\alpha_t\mu\lVert x_t-x_t^{md}\rVert_p^p+((1-\alpha_t)\mu+\gamma_t)\lVert x_t-x_{t-1}\rVert_p^p\right]
\leq\alpha_t\mu V(x_t^{md},x_t)+((1-\alpha_t)\mu+\gamma_t)V(x_{t-1},x_t). \tag*{(66)} pρ α t p ν ( μ + γ t ) ∥ d t ∥ p p ≤ p ( μ + γ t ) ν ( μ + γ t ) [ α t μ ∥ x t − x t m d ∥ p p + (( 1 − α t ) μ + γ t ) ∥ x t − x t − 1 ∥ p p ] ≤ α t μ V ( x t m d , x t ) + (( 1 − α t ) μ + γ t ) V ( x t − 1 , x t ) . (66)
Using the above result and Lemma 27 (with z = x t m d z=x_t^{md} z = x t m d ), we have
Ψ ( x t a g ) ≤ ( 1 − α t ) Ψ ( x t − 1 a g ) + α t [ f ( x t m d ) + ⟨ f ′ ( x t m d ) , x t − x t m d ⟩ + X ( x t ) ] + L 2 ∥ d t ∥ p 2 + M f ∥ d t ∥ p = ( 1 − α t ) Ψ ( x t − 1 a g ) + α t [ f ( x t m d ) + ⟨ f ′ ( x t m d ) , x t − x t m d ⟩ + X ( x t ) ] + ν ( μ + γ t ) p ρ α t p ∥ d t ∥ p p − ν ( μ + γ t ) p ρ α t p ∥ d t ∥ p p + L 2 ∥ d t ∥ p 2 + M f ∥ d t ∥ p ≤ ( 1 − α t ) Ψ ( x t − 1 a g ) + α t [ f ( x t m d ) + ⟨ f ′ ( x t m d ) , x t − x t m d ⟩ + X ( x t ) + μ V ( x t m d , x t ) ] + ( ( 1 − α t ) μ + γ t ) V ( x t − 1 , x t ) − ν ( μ + γ t ) p ρ α t p ∥ d t ∥ p p + L 2 ∥ d t ∥ p 2 + M f ∥ d t ∥ p . (67) \begin{aligned}
\Psi(x_t^{ag})\leq{}&(1-\alpha_t)\Psi(x_{t-1}^{ag})+\alpha_t\left[f(x_t^{md})+\langle f'(x_t^{md}),x_t-x_t^{md}\rangle+\mathcal{X}(x_t)\right]+\frac{L}{2}\lVert d_t\rVert_p^2+M_f\lVert d_t\rVert_p\\
={}&(1-\alpha_t)\Psi(x_{t-1}^{ag})+\alpha_t\left[f(x_t^{md})+\langle f'(x_t^{md}),x_t-x_t^{md}\rangle+\mathcal{X}(x_t)\right]+\frac{\nu(\mu+\gamma_t)}{p\rho\alpha_t^p}\lVert d_t\rVert_p^p\\
&-\frac{\nu(\mu+\gamma_t)}{p\rho\alpha_t^p}\lVert d_t\rVert_p^p+\frac{L}{2}\lVert d_t\rVert_p^2+M_f\lVert d_t\rVert_p\\
\leq{}&(1-\alpha_t)\Psi(x_{t-1}^{ag})+\alpha_t\left[f(x_t^{md})+\langle f'(x_t^{md}),x_t-x_t^{md}\rangle+\mathcal{X}(x_t)+\mu V(x_t^{md},x_t)\right]\\
&+((1-\alpha_t)\mu+\gamma_t)V(x_{t-1},x_t)-\frac{\nu(\mu+\gamma_t)}{p\rho\alpha_t^p}\lVert d_t\rVert_p^p+\frac{L}{2}\lVert d_t\rVert_p^2+M_f\lVert d_t\rVert_p. \tag*{(67)}
\end{aligned} Ψ ( x t a g ) ≤ = ≤ ( 1 − α t ) Ψ ( x t − 1 a g ) + α t [ f ( x t m d ) + ⟨ f ′ ( x t m d ) , x t − x t m d ⟩ + X ( x t ) ] + 2 L ∥ d t ∥ p 2 + M f ∥ d t ∥ p ( 1 − α t ) Ψ ( x t − 1 a g ) + α t [ f ( x t m d ) + ⟨ f ′ ( x t m d ) , x t − x t m d ⟩ + X ( x t ) ] + pρ α t p ν ( μ + γ t ) ∥ d t ∥ p p − pρ α t p ν ( μ + γ t ) ∥ d t ∥ p p + 2 L ∥ d t ∥ p 2 + M f ∥ d t ∥ p ( 1 − α t ) Ψ ( x t − 1 a g ) + α t [ f ( x t m d ) + ⟨ f ′ ( x t m d ) , x t − x t m d ⟩ + X ( x t ) + μ V ( x t m d , x t ) ] + (( 1 − α t ) μ + γ t ) V ( x t − 1 , x t ) − pρ α t p ν ( μ + γ t ) ∥ d t ∥ p p + 2 L ∥ d t ∥ p 2 + M f ∥ d t ∥ p . (67)
By using Lemma 26 with p ( u ) = α t [ ⟨ G t , u ⟩ + X ( u ) ] p(u)=\alpha_t[\langle G_t,u\rangle+\mathcal{X}(u)] p ( u ) = α t [⟨ G t , u ⟩ + X ( u )] , μ 1 = α t μ \mu_1=\alpha_t\mu μ 1 = α t μ , μ 2 = ( 1 − α t ) μ + γ t \mu_2=(1-\alpha_t)\mu+\gamma_t μ 2 = ( 1 − α t ) μ + γ t , x ~ = x t m d \tilde{x}=x_t^{md} x ~ = x t m d and y ~ = x t − 1 \tilde{y}=x_{t-1} y ~ = x t − 1 , we have
α t [ f ( x t m d ) + ⟨ G t , x t − x t m d ⟩ + X ( x t ) + μ V ( x t m d , x t ) ] + ( ( 1 − α t ) μ + γ t ) V ( x t − 1 , x t ) ≤ α t [ f ( x t m d ) + ⟨ G t , x − x t m d ⟩ + X ( x ) + μ V ( x t m d , x ) ] + ( ( 1 − α t ) μ + γ t ) V ( x t − 1 , x ) − ( μ + γ t ) V ( x t , x ) = α t l Ψ ( x t m d , x ) + α t ⟨ δ t , x − x t m d ⟩ + ( ( 1 − α t ) μ + γ t ) V ( x t − 1 , x ) − ( μ + γ t ) V ( x t , x ) , (68) \begin{aligned}
\alpha_t\left[f(x_t^{md})+\langle G_t,x_t-x_t^{md}\rangle+\mathcal{X}(x_t)+\mu V(x_t^{md},x_t)\right]+((1-\alpha_t)\mu+\gamma_t)V(x_{t-1},x_t)\\
\leq\alpha_t\left[f(x_t^{md})+\langle G_t,x-x_t^{md}\rangle+\mathcal{X}(x)+\mu V(x_t^{md},x)\right]+((1-\alpha_t)\mu+\gamma_t)V(x_{t-1},x)-(\mu+\gamma_t)V(x_t,x)\\
=\alpha_t l\Psi(x_t^{md},x)+\alpha_t\langle\delta_t,x-x_t^{md}\rangle+((1-\alpha_t)\mu+\gamma_t)V(x_{t-1},x)-(\mu+\gamma_t)V(x_t,x), \tag*{(68)}
\end{aligned} α t [ f ( x t m d ) + ⟨ G t , x t − x t m d ⟩ + X ( x t ) + μ V ( x t m d , x t ) ] + (( 1 − α t ) μ + γ t ) V ( x t − 1 , x t ) ≤ α t [ f ( x t m d ) + ⟨ G t , x − x t m d ⟩ + X ( x ) + μ V ( x t m d , x ) ] + (( 1 − α t ) μ + γ t ) V ( x t − 1 , x ) − ( μ + γ t ) V ( x t , x ) = α t l Ψ ( x t m d , x ) + α t ⟨ δ t , x − x t m d ⟩ + (( 1 − α t ) μ + γ t ) V ( x t − 1 , x ) − ( μ + γ t ) V ( x t , x ) , (68)
for any x ∈ X x\in X x ∈ X , where the last equality follows from (59) and (61) . Combining (67) and (68) , and using the fact that f ′ ( x t m d ) = G t − δ t f'(x_t^{md})=G_t-\delta_t f ′ ( x t m d ) = G t − δ t due to (61) , we obtain
Ψ ( x t a g ) ≤ ( 1 − α t ) [ Ψ ( x t − 1 a g ) + μ V ( x t − 1 , x ) ] + α t l Ψ ( x t m d , x ) + γ t [ V ( x t − 1 , x ) − V ( x t , x ) ] − μ V ( x t , x ) − ν ( μ + γ t ) p ρ α t p ∥ d t ∥ p p + L 2 ∥ d t ∥ p 2 + M f ∥ d t ∥ p + α t ⟨ δ t , x − x t ⟩ , ∀ x ∈ X . (69) \begin{aligned}
\Psi(x_t^{ag})\leq{}&(1-\alpha_t)\left[\Psi(x_{t-1}^{ag})+\mu V(x_{t-1},x)\right]+\alpha_t l\Psi(x_t^{md},x)+\gamma_t[V(x_{t-1},x)-V(x_t,x)]\\
&-\mu V(x_t,x)-\frac{\nu(\mu+\gamma_t)}{p\rho\alpha_t^p}\lVert d_t\rVert_p^p+\frac{L}{2}\lVert d_t\rVert_p^2+M_f\lVert d_t\rVert_p+\alpha_t\langle\delta_t,x-x_t\rangle,\qquad\forall x\in X. \tag*{(69)}
\end{aligned} Ψ ( x t a g ) ≤ ( 1 − α t ) [ Ψ ( x t − 1 a g ) + μ V ( x t − 1 , x ) ] + α t l Ψ ( x t m d , x ) + γ t [ V ( x t − 1 , x ) − V ( x t , x )] − μ V ( x t , x ) − pρ α t p ν ( μ + γ t ) ∥ d t ∥ p p + 2 L ∥ d t ∥ p 2 + M f ∥ d t ∥ p + α t ⟨ δ t , x − x t ⟩ , ∀ x ∈ X . (69)
It remains to show that the term U t U_t U t defined above is bounded by Δ t ( x ) \Delta_t(x) Δ t ( x ) as defined in the proposition.
Indeed, we have, by the last identity in (65) ,
U t = − ν ( μ + γ t ) p α t p ∥ d t ∥ p p + L 2 ∥ d t ∥ p 2 + M f ∥ d t ∥ p − ⟨ δ t , d t ⟩ + ⟨ δ t , d t + α t ( x − x t ) ⟩ ≤ − ν ( μ + γ t ) p α t p ∥ d t ∥ p p + L p − 1 p M f p − 2 ∥ d t ∥ p p + 2 M f ∥ d t ∥ p − ⟨ δ t , d t ⟩ + α t ⟨ δ t , x − x t − 1 + ⟩ ≤ − ν ( μ + γ t ) − ( L p − 1 / M f p − 2 ) α t p p α t p ∥ d t ∥ p p + ( 2 M f + ∥ δ t ∥ q ) ∥ d t ∥ p + α t ⟨ δ t , x − x t − 1 + ⟩ ≤ α t q ( 2 M f + ∥ δ t ∥ q ) q q K t q − 1 + α t ⟨ δ t , x − x t − 1 + ⟩ = Δ t ( x ) , (70) \begin{aligned}
U_t&=-\frac{\nu(\mu+\gamma_t)}{p\alpha_t^p}\lVert d_t\rVert_p^p+\frac{L}{2}\lVert d_t\rVert_p^2+M_f\lVert d_t\rVert_p-\langle\delta_t,d_t\rangle+\langle\delta_t,d_t+\alpha_t(x-x_t)\rangle\\
&\le-\frac{\nu(\mu+\gamma_t)}{p\alpha_t^p}\lVert d_t\rVert_p^p+\frac{L^{p-1}}{pM_f^{p-2}}\lVert d_t\rVert_p^p+2M_f\lVert d_t\rVert_p-\langle\delta_t,d_t\rangle+\alpha_t\langle\delta_t,x-x_{t-1}^+\rangle\\
&\le-\frac{\nu(\mu+\gamma_t)-(L^{p-1}/M_f^{p-2})\alpha_t^p}{p\alpha_t^p}\lVert d_t\rVert_p^p+(2M_f+\lVert\delta_t\rVert_q)\lVert d_t\rVert_p+\alpha_t\langle\delta_t,x-x_{t-1}^+\rangle\\
&\le\frac{\alpha_t^q(2M_f+\lVert\delta_t\rVert_q)^q}{qK_t^{q-1}}+\alpha_t\langle\delta_t,x-x_{t-1}^+\rangle=\Delta_t(x), \tag*{(70)}
\end{aligned} U t = − p α t p ν ( μ + γ t ) ∥ d t ∥ p p + 2 L ∥ d t ∥ p 2 + M f ∥ d t ∥ p − ⟨ δ t , d t ⟩ + ⟨ δ t , d t + α t ( x − x t )⟩ ≤ − p α t p ν ( μ + γ t ) ∥ d t ∥ p p + p M f p − 2 L p − 1 ∥ d t ∥ p p + 2 M f ∥ d t ∥ p − ⟨ δ t , d t ⟩ + α t ⟨ δ t , x − x t − 1 + ⟩ ≤ − p α t p ν ( μ + γ t ) − ( L p − 1 / M f p − 2 ) α t p ∥ d t ∥ p p + ( 2 M f + ∥ δ t ∥ q ) ∥ d t ∥ p + α t ⟨ δ t , x − x t − 1 + ⟩ ≤ q K t q − 1 α t q ( 2 M f + ∥ δ t ∥ q ) q + α t ⟨ δ t , x − x t − 1 + ⟩ = Δ t ( x ) , (70)
where the last inequality follows by the Young’s inequality w.r.t. ∥ d t ∥ \lVert d_t\rVert ∥ d t ∥ .
Lemma 28 below follows from Proposition 15 by taking summation of the relations in (62) .
Lemma 28. Let { x t a g } t ≥ 1 \{x_t^{ag}\}_{t\ge1} { x t a g } t ≥ 1 be computed by the generic AC-SA algorithm for solving F X , X ( L , M f , σ , μ ) F_{X,X}(L,M_f,\sigma,\mu) F X , X ( L , M f , σ , μ ) . Also assume that { α t } t ≥ 1 \{\alpha_t\}_{t\ge1} { α t } t ≥ 1 and { γ t } t ≥ 1 \{\gamma_t\}_{t\ge1} { γ t } t ≥ 1 are chosen so that K t > 0 K_t>0 K t > 0 for all t ≥ 1 t\ge1 t ≥ 1 . We have
Ψ ( x t a g ) + μ V ( x t , x ) − Γ t ∑ τ = 1 t [ α τ Γ τ l Ψ ( x τ m d , x ) ] ≤ Γ t ∑ τ = 1 t γ τ Γ τ [ V ( x τ − 1 , x ) − V ( x τ , x ) ] + Γ t ∑ τ = 1 t Δ τ ( x ) Γ τ , (71) \Psi(x_t^{ag})+\mu V(x_t,x)-\Gamma_t\sum_{\tau=1}^{t}\left[\frac{\alpha_\tau}{\Gamma_\tau}l_\Psi(x_\tau^{md},x)\right]\le\Gamma_t\sum_{\tau=1}^{t}\frac{\gamma_\tau}{\Gamma_\tau}\left[V(x_{\tau-1},x)-V(x_\tau,x)\right]+\Gamma_t\sum_{\tau=1}^{t}\frac{\Delta_\tau(x)}{\Gamma_\tau}, \tag*{(71)} Ψ ( x t a g ) + μ V ( x t , x ) − Γ t τ = 1 ∑ t [ Γ τ α τ l Ψ ( x τ m d , x ) ] ≤ Γ t τ = 1 ∑ t Γ τ γ τ [ V ( x τ − 1 , x ) − V ( x τ , x ) ] + Γ t τ = 1 ∑ t Γ τ Δ τ ( x ) , (71) for any x ∈ X x\in X x ∈ X and any t ≥ 1 t\ge1 t ≥ 1 , where l Ψ ( z , x ) l_\Psi(z,x) l Ψ ( z , x ) and Δ τ ( x ) \Delta_\tau(x) Δ τ ( x ) are defined in (59) and (63) , respectively.
Proof. Dividing both sides of relation (62) by Γ t \Gamma_t Γ t , and using the definition of Γ t \Gamma_t Γ t in (74) and the fact that α 1 = 1 \alpha_1=1 α 1 = 1 , we have
1 Γ t [ Ψ ( x t a g ) + μ V ( x t , x ) ] ≤ 1 Γ t − 1 [ Ψ ( x t − 1 a g ) + μ V ( x t − 1 , x ) ] + α t Γ t l Ψ ( x t m d , x ) + γ t Γ t [ V ( x t − 1 , x ) − V ( x t , x ) ] + Δ t ( x ) Γ t , ∀ t ≥ 2 , \frac{1}{\Gamma_t}\left[\Psi(x_t^{ag})+\mu V(x_t,x)\right]\le\frac{1}{\Gamma_{t-1}}\left[\Psi(x_{t-1}^{ag})+\mu V(x_{t-1},x)\right]+\frac{\alpha_t}{\Gamma_t}l_\Psi(x_t^{md},x)+\frac{\gamma_t}{\Gamma_t}\left[V(x_{t-1},x)-V(x_t,x)\right]+\frac{\Delta_t(x)}{\Gamma_t},\qquad\forall t\ge2, Γ t 1 [ Ψ ( x t a g ) + μ V ( x t , x ) ] ≤ Γ t − 1 1 [ Ψ ( x t − 1 a g ) + μ V ( x t − 1 , x ) ] + Γ t α t l Ψ ( x t m d , x ) + Γ t γ t [ V ( x t − 1 , x ) − V ( x t , x ) ] + Γ t Δ t ( x ) , ∀ t ≥ 2 ,
and
1 Γ 1 [ Ψ ( x 1 a g ) + μ V ( x 1 , x ) ] ≤ α 1 Γ 1 l Ψ ( x 1 m d , x ) + γ 1 Γ 1 [ V ( x 0 , x ) − V ( x 1 , x ) ] + Δ 1 ( x ) Γ 1 . \frac{1}{\Gamma_1}\left[\Psi(x_1^{ag})+\mu V(x_1,x)\right]\le\frac{\alpha_1}{\Gamma_1}l_\Psi(x_1^{md},x)+\frac{\gamma_1}{\Gamma_1}\left[V(x_0,x)-V(x_1,x)\right]+\frac{\Delta_1(x)}{\Gamma_1}. Γ 1 1 [ Ψ ( x 1 a g ) + μ V ( x 1 , x ) ] ≤ Γ 1 α 1 l Ψ ( x 1 m d , x ) + Γ 1 γ 1 [ V ( x 0 , x ) − V ( x 1 , x ) ] + Γ 1 Δ 1 ( x ) .
Summing up the above inequalities, we have
1 Γ t [ Ψ ( x t a g ) + μ V ( x t , x ) ] ≤ ∑ τ = 1 t α τ Γ τ l Ψ ( x τ m d , x ) + ∑ τ = 1 t γ τ Γ τ [ V ( x τ − 1 , x ) − V ( x τ , x ) ] + ∑ τ = 1 t Δ τ ( x ) Γ τ , \frac{1}{\Gamma_t}\left[\Psi(x_t^{ag})+\mu V(x_t,x)\right]\le\sum_{\tau=1}^{t}\frac{\alpha_\tau}{\Gamma_\tau}l_\Psi(x_\tau^{md},x)+\sum_{\tau=1}^{t}\frac{\gamma_\tau}{\Gamma_\tau}\left[V(x_{\tau-1},x)-V(x_\tau,x)\right]+\sum_{\tau=1}^{t}\frac{\Delta_\tau(x)}{\Gamma_\tau}, Γ t 1 [ Ψ ( x t a g ) + μ V ( x t , x ) ] ≤ τ = 1 ∑ t Γ τ α τ l Ψ ( x τ m d , x ) + τ = 1 ∑ t Γ τ γ τ [ V ( x τ − 1 , x ) − V ( x τ , x ) ] + τ = 1 ∑ t Γ τ Δ τ ( x ) ,
which clearly implies (71) .
Theorem 16. Consider the generic AC-SA algorithm for F X , X ( L , M f , σ , μ ) F_{X,X}(L,M_f,\sigma,\mu) F X , X ( L , M f , σ , μ ) . Assume α 1 = 1 \alpha_1=1 α 1 = 1 , 0 < α t < 1 0<\alpha_t<1 0 < α t < 1 for t ≥ 2 t\ge2 t ≥ 2 , and that { α t } t ≥ 1 \{\alpha_t\}_{t\ge1} { α t } t ≥ 1 and { γ t } t ≥ 1 \{\gamma_t\}_{t\ge1} { γ t } t ≥ 1 are chosen such that
K t : = ν ( μ + γ t ) − L p − 1 M f p − 2 α t p > 0 , (72) K_t:=\nu(\mu+\gamma_t)-\frac{L^{p-1}}{M_f^{p-2}}\alpha_t^p>0, \tag*{(72)} K t := ν ( μ + γ t ) − M f p − 2 L p − 1 α t p > 0 , (72)
γ 1 Γ 1 = γ 2 Γ 2 = ⋯ , (73) \frac{\gamma_1}{\Gamma_1}=\frac{\gamma_2}{\Gamma_2}=\cdots, \tag*{(73)} Γ 1 γ 1 = Γ 2 γ 2 = ⋯ , (73)
where
Γ t : = { 1 , t = 1 , ( 1 − α t ) Γ t − 1 , t ≥ 2. (74) \Gamma_t :=
\begin{cases}
1, & t=1,\\
(1-\alpha_t)\Gamma_{t-1}, & t\ge2.
\end{cases} \tag*{(74)} Γ t := { 1 , ( 1 − α t ) Γ t − 1 , t = 1 , t ≥ 2. (74)
Then,
a) under the stochastic-oracle conditions (60) , we have
E [ Ψ ( x t a g ) − Ψ ∗ ] ≤ B e ( t ) : = Γ t γ 1 V ( x 0 , x ∗ ) + Γ t ∑ τ = 1 t 2 q − 1 α τ q [ ( 2 M f ) q + σ q ] q Γ τ K τ q − 1 (75) \mathbb{E}[\Psi(x_t^{ag})-\Psi^*]\le B_e(t):=\Gamma_t\gamma_1 V(x_0,x^*)+\Gamma_t\sum_{\tau=1}^{t}\frac{2^{q-1}\alpha_\tau^q[(2M_f)^q+\sigma^q]}{q\Gamma_\tau K_\tau^{q-1}} \tag*{(75)} E [ Ψ ( x t a g ) − Ψ ∗ ] ≤ B e ( t ) := Γ t γ 1 V ( x 0 , x ∗ ) + Γ t τ = 1 ∑ t q Γ τ K τ q − 1 2 q − 1 α τ q [( 2 M f ) q + σ q ] (75)
for any t ≥ 1 t\ge1 t ≥ 1 , where x ∗ x^* x ∗ is an arbitrary optimal solution of (57) .
Proof. We show this following the original proof of Theorem 1 in [23 ] . Observe that by the definition of Γ t \Gamma_t Γ t and the fact that α 1 = 1 \alpha_1=1 α 1 = 1 , we have
∑ τ = 1 t α τ Γ τ = α 1 Γ 1 + ∑ τ = 2 t 1 Γ τ ( 1 − Γ τ Γ τ − 1 ) = 1 Γ 1 + ∑ τ = 2 t ( 1 Γ τ − 1 Γ τ − 1 ) = 1 Γ t . (76) \sum_{\tau=1}^{t}\frac{\alpha_\tau}{\Gamma_\tau}
=\frac{\alpha_1}{\Gamma_1}+\sum_{\tau=2}^{t}\frac{1}{\Gamma_\tau}\left(1-\frac{\Gamma_\tau}{\Gamma_{\tau-1}}\right)
=\frac{1}{\Gamma_1}+\sum_{\tau=2}^{t}\left(\frac{1}{\Gamma_\tau}-\frac{1}{\Gamma_{\tau-1}}\right)
=\frac{1}{\Gamma_t}. \tag*{(76)} τ = 1 ∑ t Γ τ α τ = Γ 1 α 1 + τ = 2 ∑ t Γ τ 1 ( 1 − Γ τ − 1 Γ τ ) = Γ 1 1 + τ = 2 ∑ t ( Γ τ 1 − Γ τ − 1 1 ) = Γ t 1 . (76)
The lower-model inequality follows directly from (58) : for every x , z ∈ X x,z\in X x , z ∈ X ,
l Ψ ( z , x ) = f ( z ) + ⟨ f ′ ( z ) , x − z ⟩ + μ V ( z , x ) + X ( x ) ≤ f ( x ) + X ( x ) = Ψ ( x ) . l_\Psi(z,x)=f(z)+\langle f'(z),x-z\rangle+\mu V(z,x)+\mathcal{X}(x)\le f(x)+\mathcal{X}(x)=\Psi(x). l Ψ ( z , x ) = f ( z ) + ⟨ f ′ ( z ) , x − z ⟩ + μ V ( z , x ) + X ( x ) ≤ f ( x ) + X ( x ) = Ψ ( x ) .
Using this inequality and the previous observation, we obtain
Γ t ∑ τ = 1 t [ α τ Γ τ l Ψ ( x τ m d , x ) ] ≤ Γ t ∑ τ = 1 t [ α τ Γ τ Ψ ( x ) ] = Ψ ( x ) , ∀ x ∈ X . (77) \Gamma_t\sum_{\tau=1}^{t}\left[\frac{\alpha_\tau}{\Gamma_\tau}l_\Psi(x_\tau^{md},x)\right]\le\Gamma_t\sum_{\tau=1}^{t}\left[\frac{\alpha_\tau}{\Gamma_\tau}\Psi(x)\right]=\Psi(x),\qquad\forall x\in X. \tag*{(77)} Γ t τ = 1 ∑ t [ Γ τ α τ l Ψ ( x τ m d , x ) ] ≤ Γ t τ = 1 ∑ t [ Γ τ α τ Ψ ( x ) ] = Ψ ( x ) , ∀ x ∈ X . (77)
Moreover, by the condition γ 1 / Γ 1 = γ 2 / Γ 2 = ⋯ \gamma_1/\Gamma_1=\gamma_2/\Gamma_2=\cdots γ 1 / Γ 1 = γ 2 / Γ 2 = ⋯ , we have
Γ t ∑ τ = 1 t γ τ Γ τ [ V ( x τ − 1 , x ) − V ( x τ , x ) ] = Γ t γ 1 Γ 1 [ V ( x 0 , x ) − V ( x t , x ) ] ≤ Γ t γ 1 V ( x 0 , x ) , (78) \Gamma_t\sum_{\tau=1}^{t}\frac{\gamma_\tau}{\Gamma_\tau}[V(x_{\tau-1},x)-V(x_\tau,x)]
=\Gamma_t\frac{\gamma_1}{\Gamma_1}[V(x_0,x)-V(x_t,x)]
\le\Gamma_t\gamma_1V(x_0,x), \tag*{(78)} Γ t τ = 1 ∑ t Γ τ γ τ [ V ( x τ − 1 , x ) − V ( x τ , x )] = Γ t Γ 1 γ 1 [ V ( x 0 , x ) − V ( x t , x )] ≤ Γ t γ 1 V ( x 0 , x ) , (78)
where the last inequality follows from Γ 1 = 1 \Gamma_1=1 Γ 1 = 1 and V ( x t , x ) ≥ 0 V(x_t,x)\ge0 V ( x t , x ) ≥ 0 . Substituting (77) and (78) into the cumulative recursion (71) , and using μ V ( x t , x ) ≥ 0 \mu V(x_t,x)\ge0 μ V ( x t , x ) ≥ 0 , gives
Ψ ( x t a g ) − Ψ ( x ) ≤ Ψ ( x t a g ) + μ V ( x t , x ) − Ψ ( x ) ≤ Γ t γ 1 V ( x 0 , x ) + Γ t ∑ τ = 1 t Δ τ ( x ) Γ τ , ∀ x ∈ X . (79) \Psi(x_t^{ag})-\Psi(x)\le\Psi(x_t^{ag})+\mu V(x_t,x)-\Psi(x)\le\Gamma_t\gamma_1V(x_0,x)+\Gamma_t\sum_{\tau=1}^{t}\frac{\Delta_\tau(x)}{\Gamma_\tau},\qquad\forall x\in X. \tag*{(79)} Ψ ( x t a g ) − Ψ ( x ) ≤ Ψ ( x t a g ) + μ V ( x t , x ) − Ψ ( x ) ≤ Γ t γ 1 V ( x 0 , x ) + Γ t τ = 1 ∑ t Γ τ Δ τ ( x ) , ∀ x ∈ X . (79)
Note that ( x τ − 1 m d , x τ − 1 , x τ − 1 a g ) (x_{\tau-1}^{md},x_{\tau-1},x_{\tau-1}^{ag}) ( x τ − 1 m d , x τ − 1 , x τ − 1 a g ) is determined by the history F τ − 1 = σ ( ξ 1 , … , ξ τ − 1 ) \mathcal{F}_{\tau-1}=\sigma(\xi_1,\ldots,\xi_{\tau-1}) F τ − 1 = σ ( ξ 1 , … , ξ τ − 1 ) . Consequently, x τ − 1 + x_{\tau-1}^{+} x τ − 1 + is F τ − 1 \mathcal{F}_{\tau-1} F τ − 1 -measurable. Taking x = x ∗ x=x^* x = x ∗ and using conditional unbiasedness of the stochastic oracle, we obtain
E [ ⟨ δ τ , x ∗ − x τ − 1 + ⟩ ∣ F τ − 1 ] = ⟨ E [ δ τ ∣ F τ − 1 ] , x ∗ − x τ − 1 + ⟩ = 0. (80) \mathbb{E}[\langle\delta_\tau,x^*-x_{\tau-1}^{+}\rangle\mid\mathcal{F}_{\tau-1}]
=\langle\mathbb{E}[\delta_\tau\mid\mathcal{F}_{\tau-1}],x^*-x_{\tau-1}^{+}\rangle=0. \tag*{(80)} E [⟨ δ τ , x ∗ − x τ − 1 + ⟩ ∣ F τ − 1 ] = ⟨ E [ δ τ ∣ F τ − 1 ] , x ∗ − x τ − 1 + ⟩ = 0. (80)
By the conditional q q q -moment assumption in (60) ,
E [ ∥ δ τ ∥ ∗ q ∣ F τ − 1 ] ≤ σ q . \mathbb{E}[\|\delta_\tau\|_*^q\mid\mathcal{F}_{\tau-1}]\le\sigma^q. E [ ∥ δ τ ∥ ∗ q ∣ F τ − 1 ] ≤ σ q .
Furthermore, the elementary inequality ( a + b ) q ≤ 2 q − 1 ( a q + b q ) (a+b)^q \le2^{q-1}(a^q+b^q) ( a + b ) q ≤ 2 q − 1 ( a q + b q ) , valid for a , b ≥ 0 a,b\ge0 a , b ≥ 0 , implies
E [ ( 2 M f + ∥ δ τ ∥ q ) q ] ≤ 2 q − 1 ( ( 2 M f ) q + E ∥ δ τ ∥ q q ) ≤ 2 q − 1 ( ( 2 M f ) q + σ q ) . \mathbb{E}[(2M_f+\|\delta_\tau\|_q)^q]\le2^{q-1}((2M_f)^q+\mathbb{E}\|\delta_\tau\|_q^q)
\le2^{q-1}((2M_f)^q+\sigma^q). E [( 2 M f + ∥ δ τ ∥ q ) q ] ≤ 2 q − 1 (( 2 M f ) q + E ∥ δ τ ∥ q q ) ≤ 2 q − 1 (( 2 M f ) q + σ q ) .
Recalling the definition of Δ τ ( x ) \Delta_\tau(x) Δ τ ( x ) in the preceding proposition and observing that q − 1 = 1 / ( p − 1 ) q-1=1/(p-1) q − 1 = 1/ ( p − 1 ) , we conclude that
E [ Δ τ ( x ⋆ ) ] = α τ E [ ⟨ δ τ , x ⋆ − x τ − 1 ⋆ ⟩ ] + α τ q q K τ q − 1 E [ ( 2 M f + ∥ δ τ ∥ q ) q ] ≤ 2 q − 1 α τ q ( ( 2 M f ) q + σ q ) q K τ q − 1 . \mathbb{E}[\Delta_\tau(x^\star)]=\alpha_\tau\mathbb{E}[\langle\delta_\tau,x^\star-x_{\tau-1}^\star\rangle]+\frac{\alpha_\tau^q}{qK_\tau^{q-1}}\mathbb{E}[(2M_f+\|\delta_\tau\|_q)^q]
\le\frac{2^{q-1}\alpha_\tau^q((2M_f)^q+\sigma^q)}{qK_\tau^{q-1}}. E [ Δ τ ( x ⋆ )] = α τ E [⟨ δ τ , x ⋆ − x τ − 1 ⋆ ⟩] + q K τ q − 1 α τ q E [( 2 M f + ∥ δ τ ∥ q ) q ] ≤ q K τ q − 1 2 q − 1 α τ q (( 2 M f ) q + σ q ) .
Finally, taking expectations on both sides of (79) with x = x ⋆ x=x^\star x = x ⋆ , and applying the previous estimate, we have
E [ Ψ ( x t a g ) − Ψ ⋆ ] ≤ Γ t γ 1 V ( x 0 , x ⋆ ) + Γ t ∑ τ = 1 t E [ Δ τ ( x ⋆ ) ] Γ τ ≤ Γ t γ 1 V ( x 0 , x ⋆ ) + 2 q − 1 Γ t q ∑ τ = 1 t α τ q ( ( 2 M f ) q + σ q ) Γ τ K τ q − 1 , \begin{aligned}
\mathbb{E}[\Psi(x_t^{ag})-\Psi^\star]\le\Gamma_t\gamma_1V(x_0,x^\star)+\Gamma_t\sum_{\tau=1}^{t}\frac{\mathbb{E}[\Delta_\tau(x^\star)]}{\Gamma_\tau} \\
&\le\Gamma_t\gamma_1V(x_0,x^\star)+\frac{2^{q-1}\Gamma_t}{q}\sum_{\tau=1}^{t}\frac{\alpha_\tau^q((2M_f)^q+\sigma^q)}{\Gamma_\tau K_\tau^{q-1}},
\end{aligned} E [ Ψ ( x t a g ) − Ψ ⋆ ] ≤ Γ t γ 1 V ( x 0 , x ⋆ ) + Γ t τ = 1 ∑ t Γ τ E [ Δ τ ( x ⋆ )] ≤ Γ t γ 1 V ( x 0 , x ⋆ ) + q 2 q − 1 Γ t τ = 1 ∑ t Γ τ K τ q − 1 α τ q (( 2 M f ) q + σ q ) ,
as claimed.
Theorem 17. Assume that the conditions of Propositions 11 and 12 hold, then BallAccel (Proposition 11), using parallel ℓ p \ell_p ℓ p -AC-SA (Algorithm 10) as a ball optimization oracle, solves Problem 1 with expected error ϵ o p t \epsilon_{\mathrm{opt}} ϵ opt , with
O ( d p 2 ( p + 1 ) κ p p + 1 log 4 ( d κ ) ) query depth O\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\log^4(d\kappa)\right)\quad\text{query depth} O ( d 2 ( p + 1 ) p κ p + 1 p log 4 ( d κ ) ) query depth and
O ( ( d p 2 ( p + 1 ) κ p p + 1 + d p 2 ( p + 2 ) κ 2 p p + 2 ) log 2 p ( d κ ) + d p ( p − 2 ) 2 ( p + 1 ) κ p ( 2 p − 1 ) p + 1 log 2 p ( p − 1 ) + 2 ( d κ ) ) total queries , O\left(\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}+d^{\frac{p}{2(p+2)}}\kappa^{\frac{2p}{p+2}}\right)\log^{2p}(d\kappa)+d^{\frac{p(p-2)}{2(p+1)}}\kappa^{\frac{p(2p-1)}{p+1}}\log^{2p(p-1)+2}(d\kappa)\right)\quad\text{total queries}, O ( ( d 2 ( p + 1 ) p κ p + 1 p + d 2 ( p + 2 ) p κ p + 2 2 p ) log 2 p ( d κ ) + d 2 ( p + 1 ) p ( p − 2 ) κ p + 1 p ( 2 p − 1 ) log 2 p ( p − 1 ) + 2 ( d κ ) ) total queries , and an additional computational cost of
O ( ( d p 2 ( p + 1 ) κ p p + 1 + d p 2 ( p + 2 ) κ 2 p p + 2 ) log 2 p ( d κ ) ) depth O\left(\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}+d^{\frac{p}{2(p+2)}}\kappa^{\frac{2p}{p+2}}\right)\log^{2p}(d\kappa)\right)\quad\text{depth} O ( ( d 2 ( p + 1 ) p κ p + 1 p + d 2 ( p + 2 ) p κ p + 2 2 p ) log 2 p ( d κ ) ) depth and
O ( [ ( d p 2 ( p + 1 ) κ p p + 1 + d p 2 ( p + 2 ) κ 2 p p + 2 ) log 2 p ( d κ ) + d p ( p − 2 ) 2 ( p + 1 ) κ p ( 2 p − 1 ) p + 1 log 2 p ( p − 1 ) + 2 ( d κ ) ] d ) work O\left(\left[\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}+d^{\frac{p}{2(p+2)}}\kappa^{\frac{2p}{p+2}}\right)\log^{2p}(d\kappa)+d^{\frac{p(p-2)}{2(p+1)}}\kappa^{\frac{p(2p-1)}{p+1}}\log^{2p(p-1)+2}(d\kappa)\right]d\right)\quad\text{work} O ( [ ( d 2 ( p + 1 ) p κ p + 1 p + d 2 ( p + 2 ) p κ p + 2 2 p ) log 2 p ( d κ ) + d 2 ( p + 1 ) p ( p − 2 ) κ p + 1 p ( 2 p − 1 ) log 2 p ( p − 1 ) + 2 ( d κ ) ] d ) work The computational bounds use the vector-operation cost convention of Problem 1 and Proposition 12.
Proof. Let x ⋆ ∈ arg min x ∈ B p ( R ) f ( x ) x^\star\in\arg\min_{x\in B_p(R)}f(x) x ⋆ ∈ arg min x ∈ B p ( R ) f ( x ) , and define
f p e n ( x ) = f ( x ) + 2 L ( ∥ x ∥ p − R ) + , F = f p e n ~ ρ , r = ρ = ϵ o p t C p ′ L d . f_{\mathrm{pen}}(x)=f(x)+2L(\|x\|_p-R)_+,\qquad F=\widetilde{f_{\mathrm{pen}}}_\rho,\qquad r=\rho=\frac{\epsilon_{\mathrm{opt}}}{C'_pL\sqrt{d}}. f pen ( x ) = f ( x ) + 2 L ( ∥ x ∥ p − R ) + , F = f pen ρ , r = ρ = C p ′ L d ϵ opt .
By Lemma 25, f p e n f_{\mathrm{pen}} f pen is convex and 3 L 3L 3 L -Lipschitz, and its stochastic subgradient oracle has dual q q q -th moment at most ( 4 L ) q (4L)^q ( 4 L ) q . Convexity and smoothing give F ( x ) ≥ f p e n ( x ) F(x) \ge f_{\mathrm{pen}}(x) F ( x ) ≥ f pen ( x ) . Choosing C p ′ C'_p C p ′ sufficiently large, the smoothing estimate yields
0 ≤ F ( x ) − f p e n ( x ) ≤ 3 L E ∥ ξ ρ ∥ p ≤ ϵ o p t 4 , x ∈ R d . 0 \le F(x) - f_{\mathrm{pen}}(x) \le3L\mathbb{E}\|\xi_\rho\|_p \le\frac{\epsilon_{\mathrm{opt}}}{4}, \qquad x \in\mathbb{R}^d. 0 ≤ F ( x ) − f pen ( x ) ≤ 3 L E ∥ ξ ρ ∥ p ≤ 4 ϵ opt , x ∈ R d .
By the Lipschitz continuity of f f f and projection onto B p ( R ) \mathbb{B}_p(R) B p ( R ) ,
f p e n ( x ) ≥ f ( x ⋆ ) + L ( ∥ x ∥ p − R ) + , x ∈ R d . f_{\mathrm{pen}}(x) \ge f(x^\star) + L(\|x\|_p - R)_+, \qquad x \in\mathbb{R}^d. f pen ( x ) ≥ f ( x ⋆ ) + L ( ∥ x ∥ p − R ) + , x ∈ R d .
Consequently, F ≥ f p e n F \ge f_{\mathrm{pen}} F ≥ f pen is coercive and attains its global minimum. For any global minimizer z z z of F F F ,
L ( ∥ z ∥ p − R ) + ≤ F ( z ) − f ( x ⋆ ) ≤ F ( x ⋆ ) − f ( x ⋆ ) ≤ ϵ o p t 4 . L(\|z\|_p - R)_+ \le F(z) - f(x^\star) \le F(x^\star) - f(x^\star) \le\frac{\epsilon_{\mathrm{opt}}}{4}. L ( ∥ z ∥ p − R ) + ≤ F ( z ) − f ( x ⋆ ) ≤ F ( x ⋆ ) − f ( x ⋆ ) ≤ 4 ϵ opt .
Since ϵ o p t ≤ L R \epsilon_{\mathrm{opt}} \le LR ϵ opt ≤ L R , this implies
∥ z ∥ p ≤ R + ϵ o p t 4 L ≤ 2 R . \|z\|_p \le R + \frac{\epsilon_{\mathrm{opt}}}{4L} \le2R. ∥ z ∥ p ≤ R + 4 L ϵ opt ≤ 2 R .
Thus F F F has a global minimizer in B p ( 2 R ) \mathbb{B}_p(2R) B p ( 2 R ) . Apply the implementation in Proposition 11 to F F F , with comparison point x ⋆ x^\star x ⋆ , comparison radius 2 R 2R 2 R , Lipschitz parameter 3 L 3L 3 L , and target accuracy ϵ o p t / 2 \epsilon_{\mathrm{opt}}/2 ϵ opt /2 . All ball optimization oracles are applied directly to F F F . Each required regularized subproblem therefore has objective
x ↦ F ( x ) + λ p ∥ x − x ˉ ∥ p p . x \mapsto F(x) + \frac{\lambda}{p}\|x-\bar{x}\|_p^p. x ↦ F ( x ) + p λ ∥ x − x ˉ ∥ p p .
Proposition 12, applied to f p e n f_{\mathrm{pen}} f pen and its stochastic subgradient oracle, implements these calls using Algorithm 10, with L L L replaced by 3 L 3L 3 L . The BallAccel parameters corresponding to this invocation differ from the original κ \kappa κ , K K K , and λ ⋆ \lambda^\star λ ⋆ only by constant factors depending on p p p . In the complexity estimates below, we retain their definitions from Proposition 10. In particular,
K = Θ ( d p 2 ( p + 1 ) κ p p + 1 ) , log ( R κ / r ) = O ( log ( d κ ) ) . K = \Theta\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\right), \qquad\log(R\kappa/r) = O(\log(d\kappa)). K = Θ ( d 2 ( p + 1 ) p κ p + 1 p ) , log ( R κ / r ) = O ( log ( d κ )) .
Each ℓ p \ell_p ℓ p -AC-SA ball optimization call has constant query depth. Thus Proposition 11 gives total query depth
O ( K log κ log 3 ( R κ / r ) ) = O ( d p 2 ( p + 1 ) κ p p + 1 log 4 ( d κ ) ) . O\left(K\log\kappa\log^3(R\kappa/r)\right)
= O\left(d^{\frac{p}{2(p+1)}}\kappa^{\frac{p}{p+1}}\log^4(d\kappa)\right). O ( K log κ log 3 ( R κ / r ) ) = O ( d 2 ( p + 1 ) p κ p + 1 p log 4 ( d κ ) ) .
For the computational depth, all calls use λ = Ω p ( λ ⋆ ) \lambda= \Omega_p(\lambda^\star) λ = Ω p ( λ ⋆ ) , and hence
L λ r p − 1 ≤ C p κ K 1 / p log p κ ≤ C p κ K 1 / p . \frac{L}{\lambda r^{p-1}} \le\frac{C_p\kappa}{K^{1/p}\log^p\kappa} \le\frac{C_p\kappa}{K^{1/p}}. λ r p − 1 L ≤ K 1/ p log p κ C p κ ≤ K 1/ p C p κ .
By Proposition 12, each call in the first requirement of Proposition 11 requires
O ( 1 + ( κ K 1 / p ) p p + 2 ) O\left(1+\left(\frac{\kappa}{K^{1/p}}\right)^{\frac{p}{p+2}}\right) O ( 1 + ( K 1/ p κ ) p + 2 p )
sequential updates. A level-j j j call in the second requirement requires
O ( 1 + j + log log ( R κ / r ) + ( 1 + κ K 1 / p ) p p + 2 2 j ( p − 2 ) p + 2 log 2 p ( p − 2 ) p + 2 ( R κ / r ) ) O\left(1+j+\log\log(R\kappa/r)+\left(1+\frac{\kappa}{K^{1/p}}\right)^{\frac{p}{p+2}}2^{\frac{j(p-2)}{p+2}}\log^{\frac{2p(p-2)}{p+2}}(R\kappa/r)\right) O ( 1 + j + log log ( R κ / r ) + ( 1 + K 1/ p κ ) p + 2 p 2 p + 2 j ( p − 2 ) log p + 2 2 p ( p − 2 ) ( R κ / r ) )
updates. Using the call counts in the two requirements and
∑ j ≥ 1 2 − j ( 1 + j ) = O ( 1 ) , ∑ j ≥ 1 2 − 4 j p + 2 = O ( 1 ) , \sum_{j\ge1}2^{-j}(1+j)=O(1), \qquad\sum_{j\ge1}2^{-\frac{4j}{p+2}}=O(1), j ≥ 1 ∑ 2 − j ( 1 + j ) = O ( 1 ) , j ≥ 1 ∑ 2 − p + 2 4 j = O ( 1 ) ,
the combined computational depth is
O ( [ K + κ p p + 2 K p + 1 p + 2 ] log 2 p ( d κ ) ) O\left(\left[K+\kappa^{\frac{p}{p+2}}K^{\frac{p+1}{p+2}}\right]\log^{2p}(d\kappa)\right) O ( [ K + κ p + 2 p K p + 2 p + 1 ] log 2 p ( d κ ) )
Here we used
4 ≤ 2 p , 2 + 2 p ( p − 2 ) p + 2 ≤ 2 p . 4\le2p,\qquad2+\frac{2p(p-2)}{p+2}\le2p. 4 ≤ 2 p , 2 + p + 2 2 p ( p − 2 ) ≤ 2 p .
Substituting the expression for K K K gives the stated computational-depth bound. The contribution of the L p / ( λ ϕ p − 1 ) L^{p}/(\lambda\phi^{p-1}) L p / ( λ ϕ p − 1 ) term in Proposition 12 is summed exactly as in the proof of Theorem 14, and is at most
O ( κ p K p − 2 log 2 p ( p − 1 ) + 2 ( d κ ) ) . O\left(\kappa^{p}K^{p-2}\log^{2p(p-1)+2}(d\kappa)\right). O ( κ p K p − 2 log 2 p ( p − 1 ) + 2 ( d κ ) ) .
Adding the update bound above gives total queries
O ( [ K + κ p p + 2 K p + 1 p + 2 ] log 2 p ( d κ ) + κ p K p − 2 log 2 p ( p − 1 ) + 2 ( d κ ) ) . O\left(\left[K+\kappa^{\frac{p}{p+2}}K^{\frac{p+1}{p+2}}\right]\log^{2p}(d\kappa)+\kappa^{p}K^{p-2}\log^{2p(p-1)+2}(d\kappa)\right). O ( [ K + κ p + 2 p K p + 2 p + 1 ] log 2 p ( d κ ) + κ p K p − 2 log 2 p ( p − 1 ) + 2 ( d κ ) ) .
The identities
κ p p + 2 K p + 1 p + 2 = Θ ( d p 2 ( p + 2 ) κ 2 p p + 2 ) \kappa^{\frac{p}{p+2}}K^{\frac{p+1}{p+2}}=\Theta\left(d^{\frac{p}{2(p+2)}}\kappa^{\frac{2p}{p+2}}\right) κ p + 2 p K p + 2 p + 1 = Θ ( d 2 ( p + 2 ) p κ p + 2 2 p )
and
κ p K p − 2 = Θ ( d p ( p − 2 ) 2 ( p + 1 ) κ p ( 2 p − 1 ) p + 1 ) \kappa^{p}K^{p-2}=\Theta\left(d^{\frac{p(p-2)}{2(p+1)}}\kappa^{\frac{p(2p-1)}{p+1}}\right) κ p K p − 2 = Θ ( d 2 ( p + 1 ) p ( p − 2 ) κ p + 1 p ( 2 p − 1 ) )
give the stated query bound. Under the stated computational cost convention, multiplication by d d d gives the work bound. Finally, the unpenalized implementation of Proposition 11 returns x ^ \hat{x} x ^ satisfying
E F ( x ^ ) ≤ F ( x ⋆ ) + ϵ o p t 2 . \mathbb{E}F(\hat{x})\le F(x^\star)+\frac{\epsilon_{\mathrm{opt}}}{2}. E F ( x ^ ) ≤ F ( x ⋆ ) + 2 ϵ opt .
Since f ≤ f p e n ≤ F f\le f_{\mathrm{pen}}\le F f ≤ f pen ≤ F and f p e n ( x ⋆ ) = f ( x ⋆ ) f_{\mathrm{pen}}(x^\star)=f(x^\star) f pen ( x ⋆ ) = f ( x ⋆ ) ,
E f ( x ^ ) ≤ E F ( x ^ ) ≤ F ( x ⋆ ) + ϵ o p t 2 ≤ f ( x ⋆ ) + 3 ϵ o p t 4 ≤ f ( x ⋆ ) + ϵ o p t . \mathbb{E}f(\hat{x})\le\mathbb{E}F(\hat{x})\le F(x^\star)+\frac{\epsilon_{\mathrm{opt}}}{2}\le f(x^\star)+\frac{3\epsilon_{\mathrm{opt}}}{4}\le f(x^\star)+\epsilon_{\mathrm{opt}}. E f ( x ^ ) ≤ E F ( x ^ ) ≤ F ( x ⋆ ) + 2 ϵ opt ≤ f ( x ⋆ ) + 4 3 ϵ opt ≤ f ( x ⋆ ) + ϵ opt .
This completes the proof.
ℓ p \ell_p ℓ p Parallel Stochastic Convex Optimization: Lower Bounds
We consider minimizing a convex function f : R d → R f:\mathbb{R}^{d}\to\mathbb{R} f : R d → R that is 1-Lipschitz with respect to ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p . To fix notation, let
B p d = { x ∈ R d : ∥ x ∥ p ≤ 1 } , B q d = { s ∈ R d : ∥ s ∥ q ≤ 1 } . B_p^d=\{x\in\mathbb{R}^{d}:\lVert x\rVert_p\le1\},\qquad B_q^d=\{s\in\mathbb{R}^{d}:\lVert s\rVert_q\le1\}. B p d = { x ∈ R d : ∥ x ∥ p ≤ 1 } , B q d = { s ∈ R d : ∥ s ∥ q ≤ 1 } .
Then f f f is 1-Lipschitz with respect to ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p if and only if all its subgradients lie in B q d B_q^d B q d . An algorithm accesses f f f only through a local oracle , meaning that the answer at a point x x x depends only on the restriction of f f f to an arbitrarily small neighborhood of x x x . This class contains the usual (sub)gradient oracle as well as its noisy variants, and our lower bound holds for any fixed local oracle.
We consider K K K -parallel algorithms, whereby in each round the algorithm makes up to K K K queries in parallel, chosen based on the answers from previous rounds but not on the answers to other queries in the same round. Following [10 ] , we measure complexity through adaptivity , i.e., the number of sequential rounds M M M . Throughout the section, we focus on K ≤ poly ( d ) K\le\operatorname{poly}(d) K ≤ poly ( d ) . Otherwise, as noted by [18 ] , with exponentially many queries per round, one can query an ϵ \epsilon ϵ -net of the feasible set and return the best point found.
Our hard instance is constructed on a restricted region of the unit ball B p d B_p^d B p d in which every coordinate is of order d − 1 / p d^{-1/p} d − 1/ p . Specifically, we define the feasible region
X : = { x ∈ R d : ∥ x ∥ p ≤ 1 , ∥ x ∥ ∞ ≤ 4 d − 1 / p } . X := \left\{x \in\mathbb{R}^d : \lVert x\rVert_p \le1,\quad\lVert x\rVert_\infty\le4d^{-1/p}\right\}. X := { x ∈ R d : ∥ x ∥ p ≤ 1 , ∥ x ∥ ∞ ≤ 4 d − 1/ p } .
and require that all queries and the final output of the algorithm belong to X X X .
Theorem 18. There are universal constants c , C > 0 c, C > 0 c , C > 0 with the following property. Let M ≥ 2 M \ge2 M ≥ 2 , K ≥ 1 K \ge1 K ≥ 1 , ρ ∈ ( 0 , 1 / 4 ) \rho\in(0,1/4) ρ ∈ ( 0 , 1/4 ) , and suppose d ≥ 16 M d \ge16M d ≥ 16 M . Further, assume that
M − 1 / p ≥ C ε , (81) M^{-1/p} \ge C\varepsilon, \tag*{(81)} M − 1/ p ≥ C ε , (81) M 3 2 − 1 p log ( 16 K M 2 / ρ ) d ≤ c . (82) M^{\frac{3}{2}-\frac{1}{p}}\sqrt{\frac{\log(16KM^2/\rho)}{d}} \le c. \tag*{(82)} M 2 3 − p 1 d log ( 16 K M 2 / ρ ) ≤ c . (82) Then for every randomized K K K -parallel algorithm whose queries and output belong to X X X , and every fixed local oracle, there exists a convex function f : R d → R f : \mathbb{R}^d \to\mathbb{R} f : R d → R , 1 1 1 -Lipschitz with respect to ℓ p \ell_p ℓ p , such that the algorithm returns an ε \varepsilon ε -minimizer of f f f over X X X in fewer than M M M rounds with probability at most ρ \rho ρ . Consequently, up to logarithmic factors,
M = Ω ~ ( min { ε − p , d p 3 p − 2 } ) . M = \widetilde{\Omega}\left(\min\left\{\varepsilon^{-p},d^{\frac{p}{3p-2}}\right\}\right). M = Ω ( min { ε − p , d 3 p − 2 p } ) .
Random block directions
The directions below follow the disjoint-support construction of [18 ] for p ≥ 2 p \ge2 p ≥ 2 , with two key differences: the supports are drawn at random and each z i z_i z i comes with a direction h i h_i h i that is used to construct the wall function. For the rest of the section let L = ⌊ d 8 M ⌋ L = \left\lfloor\frac{d}{8M}\right\rfloor L = ⌊ 8 M d ⌋ and choose pairwise disjoint sets J 1 , … , J M ⊂ [ d ] J_1,\ldots,J_M \subset[d] J 1 , … , J M ⊂ [ d ] , each of size L L L , uniformly among all ordered disjoint collections. Independently choose Rademacher signs r i j ∈ { − 1 , + 1 } r_{ij} \in\{-1,+1\} r ij ∈ { − 1 , + 1 } and set
z i = L − 1 / q ∑ j ∈ J i r i j e j , h i = L − 1 / p ∑ j ∈ J i r i j e j , x ∗ = − M − 1 / p ∑ i = 1 M h i . z_i = L^{-1/q}\sum_{j\in J_i}r_{ij}e_j,\qquad h_i = L^{-1/p}\sum_{j\in J_i}r_{ij}e_j,\qquad x^* = -M^{-1/p}\sum_{i=1}^{M}h_i. z i = L − 1/ q j ∈ J i ∑ r ij e j , h i = L − 1/ p j ∈ J i ∑ r ij e j , x ∗ = − M − 1/ p i = 1 ∑ M h i .
Since the blocks are disjoint, ∥ z i ∥ q = ∥ h i ∥ p = ∥ x ∗ ∥ p = 1 \lVert z_i\rVert_q = \lVert h_i\rVert_p = \lVert x^*\rVert_p = 1 ∥ z i ∥ q = ∥ h i ∥ p = ∥ x ∗ ∥ p = 1 and ⟨ z i , x ∗ ⟩ = − M − 1 / p \langle z_i,x^*\rangle= -M^{-1/p} ⟨ z i , x ∗ ⟩ = − M − 1/ p for every i ∈ [ M ] i \in[M] i ∈ [ M ] . Moreover, x ∗ ∈ X x^* \in X x ∗ ∈ X : d ≥ 16 M d \ge16M d ≥ 16 M implies L ≥ d 16 M L \ge\frac{d}{16M} L ≥ 16 M d , and therefore
∥ x ∗ ∥ ∞ = ( M L ) − 1 / p ≤ ( 16 d ) 1 / p ≤ 4 d − 1 / p . \lVert x^*\rVert_\infty= (ML)^{-1/p} \le\left(\frac{16}{d}\right)^{1/p} \le4d^{-1/p}. ∥ x ∗ ∥ ∞ = ( M L ) − 1/ p ≤ ( d 16 ) 1/ p ≤ 4 d − 1/ p .
The wall function
As in [9 ] , who introduced a wall in the ℓ 2 \ell_2 ℓ 2 setting, our objective also contains a wall W W W . For t ∈ [ M ] t \in[M] t ∈ [ M ] , let R t : = [ d ] ∖ ⋃ i < t J i R_t := [d]\setminus\bigcup_{i<t}J_i R t := [ d ] ∖ ⋃ i < t J i denote the coordinates outside the first t − 1 t-1 t − 1 blocks. The wall is a convex function that is small at x ∗ x^* x ∗ and, at round t t t , dominates all future affine pieces at every query x ∈ X x \in X x ∈ X with ∥ x R t ∥ p > 4 M − 1 / p \lVert x_{R_t}\rVert_p > 4M^{-1/p} ∥ x R t ∥ p > 4 M − 1/ p (Lemmas 29 and 31 ). At such queries the objective carries no information about the future directions, so concentration is needed only for the remaining queries. Fix a parameter θ > 0 \theta> 0 θ > 0 , to be chosen in Lemma 33 . For t ∈ [ M ] t \in[M] t ∈ [ M ] let
C t : = { s ∈ B q d : ∥ s ∥ ∞ ≤ L − 1 / q , ∣ ⟨ h i , s ⟩ ∣ ≤ θ for all i < t } , (83) C_t := \left\{s \in B_q^d : \lVert s\rVert_\infty\le L^{-1/q},\quad\lvert\langle h_i,s\rangle\rvert\le\theta\ \text{for all } i<t\right\}, \tag*{(83)} C t := { s ∈ B q d : ∥ s ∥ ∞ ≤ L − 1/ q , ∣⟨ h i , s ⟩∣ ≤ θ for all i < t } , (83)
and let
C : = { s ∈ B q d : ∥ s ∥ ∞ ≤ L − 1 / q , ∣ ⟨ h i , s ⟩ ∣ ≤ θ for all i ≤ M } . (84) C := \left\{s \in B_q^d : \|s\|_\infty\le L^{-1/q},\ |\langle h_i,s\rangle| \le\theta\ \text{for all } i \le M\right\}. \tag*{(84)} C := { s ∈ B q d : ∥ s ∥ ∞ ≤ L − 1/ q , ∣ ⟨ h i , s ⟩ ∣ ≤ θ for all i ≤ M } . (84)
The caps ∣ ⟨ h i , s ⟩ ∣ ≤ θ |\langle h_i,s\rangle| \le\theta ∣ ⟨ h i , s ⟩ ∣ ≤ θ keep the slope s s s nearly orthogonal to the block directions, which makes the wall small at x ∗ x^* x ∗ . We define the truncated wall
W t ( x ) : = max s ∈ C t ( ⟨ x , s ⟩ − M − 1 / p 32 ∥ s ∥ 2 2 ) − 33 32 M − 1 / p , (85) W_t(x) := \max_{s \in C_t} \left(\langle x,s\rangle- \frac{M^{-1/p}}{32}\|s\|_2^2\right) - \frac{33}{32}M^{-1/p}, \tag*{(85)} W t ( x ) := s ∈ C t max ( ⟨ x , s ⟩ − 32 M − 1/ p ∥ s ∥ 2 2 ) − 32 33 M − 1/ p , (85)
which depends only on h 1 , … , h t − 1 h_1,\ldots,h_{t-1} h 1 , … , h t − 1 , and the full wall
W ( x ) : = max s ∈ C ( ⟨ x , s ⟩ − M − 1 / p 32 ∥ s ∥ 2 2 ) − 33 32 M − 1 / p . (86) W(x) := \max_{s \in C} \left(\langle x,s\rangle- \frac{M^{-1/p}}{32}\|s\|_2^2\right) - \frac{33}{32}M^{-1/p}. \tag*{(86)} W ( x ) := s ∈ C max ( ⟨ x , s ⟩ − 32 M − 1/ p ∥ s ∥ 2 2 ) − 32 33 M − 1/ p . (86)
The quadratic term is used only to make the maximizing slope unique and continuous, so that W t = W W_t = W W t = W on a neighborhood of each query (Lemma 30 ).
Now, we establish the two properties of the wall that we need: it is large on queries with large residual norm on R t R_t R t , and small at x ∗ x^* x ∗ .
Lemma 29. Let R t : = [ d ] ∖ ⋃ i < t J i R_t := [d] \setminus\bigcup_{i<t} J_i R t := [ d ] ∖ ⋃ i < t J i . For every x ∈ X x \in X x ∈ X satisfying ∥ x R t ∥ p > 4 M − 1 / p \|x_{R_t}\|_p > 4M^{-1/p} ∥ x R t ∥ p > 4 M − 1/ p , we have
W t ( x ) ≥ ∥ x R t ∥ p − 17 16 M − 1 / p . (87) W_t(x) \ge\|x_{R_t}\|_p - \frac{17}{16}M^{-1/p}. \tag*{(87)} W t ( x ) ≥ ∥ x R t ∥ p − 16 17 M − 1/ p . (87) Moreover, if M θ ≤ 1 M\theta\le1 M θ ≤ 1 , then
W ( x ∗ ) ≤ − 1 32 M − 1 / p . W(x^*) \le-\frac{1}{32}M^{-1/p}. W ( x ∗ ) ≤ − 32 1 M − 1/ p . Proof. Fix x ∈ X x \in X x ∈ X with ∥ x R t ∥ p > 4 M − 1 / p \|x_{R_t}\|_p > 4M^{-1/p} ∥ x R t ∥ p > 4 M − 1/ p and define the ℓ p \ell_p ℓ p -norming functional of x R t x_{R_t} x R t by
s j : = sgn ( x j ) ∣ x j ∣ p − 1 ∥ x R t ∥ p p − 1 1 { j ∈ R t } . s_j := \frac{\operatorname{sgn}(x_j)|x_j|^{p-1}}{\|x_{R_t}\|_p^{p-1}}\mathbf{1}_{\{j\in R_t\}}. s j := ∥ x R t ∥ p p − 1 sgn ( x j ) ∣ x j ∣ p − 1 1 { j ∈ R t } . Then ∥ s ∥ q = 1 \|s\|_q = 1 ∥ s ∥ q = 1 , ⟨ x , s ⟩ = ∥ x R t ∥ p \langle x,s\rangle= \|x_{R_t}\|_p ⟨ x , s ⟩ = ∥ x R t ∥ p . We claim that s ∈ C t s \in C_t s ∈ C t . Since x ∈ X x \in X x ∈ X , ∥ x ∥ ∞ ≤ 4 d − 1 / p \|x\|_\infty\le4d^{-1/p} ∥ x ∥ ∞ ≤ 4 d − 1/ p . Hence
∥ s ∥ ∞ = ( ∥ x R t ∥ ∞ ∥ x R t ∥ p ) p − 1 < ( 4 d − 1 / p 4 M − 1 / p ) p − 1 = ( M d ) 1 / q . \|s\|_\infty= \left(\frac{\|x_{R_t}\|_\infty}{\|x_{R_t}\|_p}\right)^{p-1} < \left(\frac{4d^{-1/p}}{4M^{-1/p}}\right)^{p-1} = \left(\frac{M}{d}\right)^{1/q}. ∥ s ∥ ∞ = ( ∥ x R t ∥ p ∥ x R t ∥ ∞ ) p − 1 < ( 4 M − 1/ p 4 d − 1/ p ) p − 1 = ( d M ) 1/ q . Since L ≤ d / ( 8 M ) L \le d/(8M) L ≤ d / ( 8 M ) ,
L − 1 / q ≥ ( 8 M d ) 1 / q > ( M d ) 1 / q , L^{-1/q} \ge\left(\frac{8M}{d}\right)^{1/q} > \left(\frac{M}{d}\right)^{1/q}, L − 1/ q ≥ ( d 8 M ) 1/ q > ( d M ) 1/ q , so ∥ s ∥ ∞ < L − 1 / q \|s\|_\infty< L^{-1/q} ∥ s ∥ ∞ < L − 1/ q . Moreover, s s s is supported on R t R_t R t , whereas every h i h_i h i with i < t i<t i < t is supported on J i J_i J i . Therefore, ⟨ h i , s ⟩ = 0 \langle h_i,s\rangle= 0 ⟨ h i , s ⟩ = 0 for i < t i<t i < t . Thus s ∈ C t s \in C_t s ∈ C t . Since q ≤ 2 q \le2 q ≤ 2 and ∥ s ∥ q = 1 \|s\|_q = 1 ∥ s ∥ q = 1 , ∥ s ∥ 2 ≤ 1 \|s\|_2 \le1 ∥ s ∥ 2 ≤ 1 . Using s s s as a feasible candidate in (85) ,
W t ( x ) ≥ ⟨ x , s ⟩ − M − 1 / p 32 ∥ s ∥ 2 2 − 33 32 M − 1 / p ≥ ∥ x R t ∥ p − 34 32 M − 1 / p = ∥ x R t ∥ p − 17 16 M − 1 / p , W_t(x) \ge\langle x,s\rangle- \frac{M^{-1/p}}{32}\|s\|_2^2 - \frac{33}{32}M^{-1/p} \ge\|x_{R_t}\|_p - \frac{34}{32}M^{-1/p} = \|x_{R_t}\|_p - \frac{17}{16}M^{-1/p}, W t ( x ) ≥ ⟨ x , s ⟩ − 32 M − 1/ p ∥ s ∥ 2 2 − 32 33 M − 1/ p ≥ ∥ x R t ∥ p − 32 34 M − 1/ p = ∥ x R t ∥ p − 16 17 M − 1/ p , which proves (87) . We next bound the full wall at x ∗ x^* x ∗ . Let s ∈ C s \in C s ∈ C . Since
x ∗ = − M − 1 / p ∑ i = 1 M h i , x^* = -M^{-1/p}\sum_{i=1}^{M} h_i, x ∗ = − M − 1/ p i = 1 ∑ M h i ,
the cap constraints give
⟨ x ∗ , s ⟩ = − M − 1 / p ∑ i = 1 M ⟨ h i , s ⟩ ≤ M − 1 / p ∑ i = 1 M ∣ ⟨ h i , s ⟩ ∣ ≤ M 1 − 1 / p θ . \langle x^*,s\rangle=-M^{-1/p}\sum_{i=1}^{M}\langle h_i,s\rangle\le M^{-1/p}\sum_{i=1}^{M}|\langle h_i,s\rangle|\le M^{1-1/p}\theta. ⟨ x ∗ , s ⟩ = − M − 1/ p i = 1 ∑ M ⟨ h i , s ⟩ ≤ M − 1/ p i = 1 ∑ M ∣ ⟨ h i , s ⟩ ∣ ≤ M 1 − 1/ p θ .
If M θ ≤ 1 M\theta\le1 M θ ≤ 1 , then M 1 − 1 / p θ = M − 1 / p ( M θ ) ≤ M − 1 / p M^{1-1/p}\theta=M^{-1/p}(M\theta)\le M^{-1/p} M 1 − 1/ p θ = M − 1/ p ( M θ ) ≤ M − 1/ p . The quadratic term is nonpositive, so for all s ∈ C s\in C s ∈ C , it satisfies
⟨ x ∗ , s ⟩ − M − 1 / p 32 ∥ s ∥ 2 2 ≤ M − 1 / p . \langle x^*,s\rangle-\frac{M^{-1/p}}{32}\lVert s\rVert_2^2\le M^{-1/p}. ⟨ x ∗ , s ⟩ − 32 M − 1/ p ∥ s ∥ 2 2 ≤ M − 1/ p .
Taking the maximum over s ∈ C s\in C s ∈ C and subtracting 33 M − 1 / p / 32 33M^{-1/p}/32 33 M − 1/ p /32 yields
W ( x ∗ ) ≤ − 1 32 M − 1 / p . W(x^*)\le-\frac{1}{32}M^{-1/p}. W ( x ∗ ) ≤ − 32 1 M − 1/ p .
Lemma 30. For every t ∈ [ M ] t\in[M] t ∈ [ M ] , the truncated wall W t W_t W t and the full wall W W W are convex and 1-Lipschitz with respect to ℓ p \ell_p ℓ p . Let
s t ( x ) : = arg max s ∈ C t { ⟨ x , s ⟩ − M − 1 / p 32 ∥ s ∥ 2 2 } . s_t(x):=\underset{s\in C_t}{\operatorname{arg\,max}}\left\{\langle x,s\rangle-\frac{M^{-1/p}}{32}\lVert s\rVert_2^2\right\}. s t ( x ) := s ∈ C t arg max { ⟨ x , s ⟩ − 32 M − 1/ p ∥ s ∥ 2 2 } . The maximizer is unique. Moreover, if
∣ ⟨ h i , s t ( x ) ⟩ ∣ ≤ θ 2 for every i = t , … , M , (88) \lvert\langle h_i,s_t(x)\rangle\rvert\le\frac{\theta}{2}\qquad\text{for every }i=t,\ldots,M, \tag*{(88)} ∣⟨ h i , s t ( x )⟩∣ ≤ 2 θ for every i = t , … , M , (88) then there exists an open neighborhood U U U of x x x such that
W t ( y ) = W ( y ) for every y ∈ U . W_t(y)=W(y)\qquad\text{for every }y\in U. W t ( y ) = W ( y ) for every y ∈ U . Proof. For every fixed s ∈ C t s\in C_t s ∈ C t , the map
x ↦ ⟨ x , s ⟩ − M − 1 / p 32 ∥ s ∥ 2 2 x\mapsto\langle x,s\rangle-\frac{M^{-1/p}}{32}\lVert s\rVert_2^2 x ↦ ⟨ x , s ⟩ − 32 M − 1/ p ∥ s ∥ 2 2 is affine with slope s ∈ B q d s\in B_q^d s ∈ B q d . Hence the maximum over s ∈ C t s\in C_t s ∈ C t is convex and 1-Lipschitz with respect to ℓ p \ell_p ℓ p . The same argument applies to W W W . Subtracting a constant would not affect both properties. Completing the square gives s t ( x ) = Π C t ( 16 M 1 / p x ) s_t(x)=\Pi_{C_t}(16M^{1/p}x) s t ( x ) = Π C t ( 16 M 1/ p x ) , so the maximizer is unique and, as the Euclidean projection is 1-Lipschitz,
∥ s t ( y ) − s t ( x ) ∥ 2 ≤ ∥ 16 M 1 / p y − 16 M 1 / p x ∥ 2 = 16 M 1 / p ∥ y − x ∥ 2 . (89) \lVert s_t(y)-s_t(x)\rVert_2\le\lVert16M^{1/p}y-16M^{1/p}x\rVert_2=16M^{1/p}\lVert y-x\rVert_2. \tag*{(89)} ∥ s t ( y ) − s t ( x ) ∥ 2 ≤ ∥ 16 M 1/ p y − 16 M 1/ p x ∥ 2 = 16 M 1/ p ∥ y − x ∥ 2 . (89) Suppose (88) holds. For every i = t , … , M i=t,\ldots,M i = t , … , M ,
∣ ⟨ h i , s t ( y ) ⟩ ∣ ≤ ∣ ⟨ h i , s t ( x ) ⟩ ∣ + ∣ ⟨ h i , s t ( y ) − s t ( x ) ⟩ ∣ ≤ θ 2 + ∥ h i ∥ 2 ∥ s t ( y ) − s t ( x ) ∥ 2 ≤ θ 2 + 16 M 1 / p ∥ h i ∥ 2 ∥ y − x ∥ 2 . \begin{aligned}
\lvert\langle h_i,s_t(y)\rangle\rvert
&\le\lvert\langle h_i,s_t(x)\rangle\rvert+\lvert\langle h_i,s_t(y)-s_t(x)\rangle\rvert\\
&\le\frac{\theta}{2}+\lVert h_i\rVert_2\lVert s_t(y)-s_t(x)\rVert_2\\
&\le\frac{\theta}{2}+16M^{1/p}\lVert h_i\rVert_2\lVert y-x\rVert_2.
\end{aligned} ∣⟨ h i , s t ( y )⟩∣ ≤ ∣⟨ h i , s t ( x )⟩∣ + ∣⟨ h i , s t ( y ) − s t ( x )⟩∣ ≤ 2 θ + ∥ h i ∥ 2 ∥ s t ( y ) − s t ( x ) ∥ 2 ≤ 2 θ + 16 M 1/ p ∥ h i ∥ 2 ∥ y − x ∥ 2 . Since there are finitely many indices, there exists δ > 0 \delta>0 δ > 0 such that, whenever ∥ y − x ∥ 2 < δ \lVert y-x\rVert_2<\delta ∥ y − x ∥ 2 < δ ,
∣ ⟨ h i , s t ( y ) ⟩ ∣ < θ for every i = t , … , M . \lvert\langle h_i,s_t(y)\rangle\rvert<\theta\qquad\text{for every }i=t,\ldots,M. ∣⟨ h i , s t ( y )⟩∣ < θ for every i = t , … , M . The vector s t ( y ) s_t(y) s t ( y ) already satisfies the caps corresponding to i < t i<t i < t because s t ( y ) ∈ C t s_t(y)\in C_t s t ( y ) ∈ C t . Thus s t ( y ) ∈ C s_t(y)\in C s t ( y ) ∈ C . Since C ⊆ C t C\subseteq C_t C ⊆ C t , the full and truncated optimization problems have the same optimal value at such y y y . Therefore W t ( y ) = W ( y ) W_t(y)=W(y) W t ( y ) = W ( y ) throughout a neighborhood of x x x .
Hard objective and hiding future pieces
For τ > 0 \tau> 0 τ > 0 , define
f ( x ) : = max { max 1 ≤ i ≤ M ( 1 32 ⟨ z i , x ⟩ − i τ 8 ) , W ( x ) } . (90) f(x) := \max\left\{\max_{1 \le i \le M}\left(\frac{1}{32}\langle z_i,x\rangle-\frac{i\tau}{8}\right), W(x)\right\}. \tag*{(90)} f ( x ) := max { 1 ≤ i ≤ M max ( 32 1 ⟨ z i , x ⟩ − 8 i τ ) , W ( x ) } . (90)
f t ( x ) : = max { max 1 ≤ i ≤ t ( 1 32 ⟨ z i , x ⟩ − i τ 8 ) , W t ( x ) } . (91) f_t(x) := \max\left\{\max_{1 \le i \le t}\left(\frac{1}{32}\langle z_i,x\rangle-\frac{i\tau}{8}\right), W_t(x)\right\}. \tag*{(91)} f t ( x ) := max { 1 ≤ i ≤ t max ( 32 1 ⟨ z i , x ⟩ − 8 i τ ) , W t ( x ) } . (91)
Lemma 31. Fix a round t < M t < M t < M and a query x ∈ X x \in X x ∈ X . Assume (88) . If ∥ x R t ∥ p ≤ 4 M − 1 / p \lVert x_{R_t}\rVert_p \le4M^{-1/p} ∥ x R t ∥ p ≤ 4 M − 1/ p , also assume
∣ ⟨ z i , x ⟩ ∣ ≤ τ for every i ≥ t . (92) \lvert\langle z_i,x\rangle\rvert\le\tau\qquad\text{for every } i \ge t. \tag*{(92)} ∣⟨ z i , x ⟩∣ ≤ τ for every i ≥ t . (92) Then f = f t f = f_t f = f t on a neighborhood of x x x .
Proof. Lemma 30 gives W = W t W = W_t W = W t on a neighborhood of x x x . First suppose ∥ x R t ∥ p > 4 M − 1 / p \lVert x_{R_t}\rVert_p > 4M^{-1/p} ∥ x R t ∥ p > 4 M − 1/ p . Every future z i z_i z i , i ≥ t i \ge t i ≥ t , is supported on R t R_t R t , and ∥ z i ∥ q = 1 \lVert z_i\rVert_q = 1 ∥ z i ∥ q = 1 , so
⟨ z i , x ⟩ = ⟨ z i , x R t ⟩ ≤ ∥ x R t ∥ p . \langle z_i,x\rangle= \langle z_i,x_{R_t}\rangle\le\lVert x_{R_t}\rVert_p. ⟨ z i , x ⟩ = ⟨ z i , x R t ⟩ ≤ ∥ x R t ∥ p .
By Lemma 29 ,
W t ( x ) ≥ ∥ x R t ∥ p − 17 16 M − 1 / p . W_t(x) \ge\lVert x_{R_t}\rVert_p - \frac{17}{16}M^{-1/p}. W t ( x ) ≥ ∥ x R t ∥ p − 16 17 M − 1/ p .
Therefore, for every i ≥ t i \ge t i ≥ t ,
W t ( x ) − 1 32 ⟨ z i , x ⟩ + i τ 8 ≥ 31 32 ∥ x R t ∥ p − 17 16 M − 1 / p > 45 16 M − 1 / p . W_t(x)-\frac{1}{32}\langle z_i,x\rangle+\frac{i\tau}{8}
\ge\frac{31}{32}\lVert x_{R_t}\rVert_p-\frac{17}{16}M^{-1/p}
> \frac{45}{16}M^{-1/p}. W t ( x ) − 32 1 ⟨ z i , x ⟩ + 8 i τ ≥ 32 31 ∥ x R t ∥ p − 16 17 M − 1/ p > 16 45 M − 1/ p .
Thus the wall strictly dominates every future affine piece, and the strict inequality holds locally. Now suppose ∥ x R t ∥ p ≤ 4 M − 1 / p \lVert x_{R_t}\rVert_p \le4M^{-1/p} ∥ x R t ∥ p ≤ 4 M − 1/ p . For every i > t i > t i > t , (92) gives
( 1 32 ⟨ z t , x ⟩ − t τ 8 ) − ( 1 32 ⟨ z i , x ⟩ − i τ 8 ) ≥ − τ 32 − t τ 8 − τ 32 + ( t + 1 ) τ 8 = τ 16 > 0. \left(\frac{1}{32}\langle z_t,x\rangle-\frac{t\tau}{8}\right)-\left(\frac{1}{32}\langle z_i,x\rangle-\frac{i\tau}{8}\right)
\ge-\frac{\tau}{32}-\frac{t\tau}{8}-\frac{\tau}{32}+\frac{(t+1)\tau}{8}
=\frac{\tau}{16}>0. ( 32 1 ⟨ z t , x ⟩ − 8 t τ ) − ( 32 1 ⟨ z i , x ⟩ − 8 i τ ) ≥ − 32 τ − 8 t τ − 32 τ + 8 ( t + 1 ) τ = 16 τ > 0.
Hence every future affine piece lies strictly below the t t t -th piece. Combining the affine and wall parts proves the claim. □ \square □
Random-block concentration
Lemma 32. Let R ⊆ [ d ] R \subseteq[d] R ⊆ [ d ] have cardinality n ≥ 7 d / 8 n \ge7d/8 n ≥ 7 d /8 , and let J J J be a uniformly random L L L -subset of R R R , where L = ⌊ d 8 M ⌋ L = \left\lfloor\frac{d}{8M}\right\rfloor L = ⌊ 8 M d ⌋ . Let ( r j ) j ∈ J (r_j)_{j\in J} ( r j ) j ∈ J be independent Rademacher signs and define
z = L − 1 / q ∑ j ∈ J r j e j , h = L − 1 / p ∑ j ∈ J r j e j . z = L^{-1/q}\sum_{j\in J}r_j e_j,\qquad h = L^{-1/p}\sum_{j\in J}r_j e_j. z = L − 1/ q j ∈ J ∑ r j e j , h = L − 1/ p j ∈ J ∑ r j e j . Then there exists a universal constant C > 0 C > 0 C > 0 such that, for every u ≥ 1 u \ge1 u ≥ 1 , the following hold.
(i) For every deterministic x ∈ X x \in X x ∈ X satisfying ∥ x R ∥ p ≤ 4 M − 1 / p \lVert x_R\rVert_p \le4M^{-1/p} ∥ x R ∥ p ≤ 4 M − 1/ p , with probability at least 1 − 2 e − u 1-2e^{-u} 1 − 2 e − u ,
∣ ⟨ z , x ⟩ ∣ ≤ C M − 1 / p [ M 1 2 − 1 p u d + M u d ] . (93) \lvert\langle z,x\rangle\rvert\le CM^{-1/p}\left[M^{\frac{1}{2}-\frac{1}{p}}\sqrt{\frac{u}{d}}+\frac{Mu}{d}\right]. \tag*{(93)} ∣⟨ z , x ⟩∣ ≤ C M − 1/ p [ M 2 1 − p 1 d u + d M u ] . (93)
(ii) For every deterministic s ∈ B q d s \in B_q^d s ∈ B q d satisfying ∥ s ∥ ∞ ≤ L − 1 / q \lVert s\rVert_\infty\le L^{-1/q} ∥ s ∥ ∞ ≤ L − 1/ q , with probability at least 1 − 2 e − u 1-2e^{-u} 1 − 2 e − u ,
∣ ⟨ h , s ⟩ ∣ ≤ C [ u d + M u d ] . (94) \lvert\langle h,s\rangle\rvert\le C\left[\sqrt{\frac{u}{d}}+\frac{Mu}{d}\right]. \tag*{(94)} ∣⟨ h , s ⟩∣ ≤ C [ d u + d M u ] . (94)
Proof. Fix a deterministic vector a ∈ R R a \in\mathbb{R}^R a ∈ R R and write
Q J ( a ) : = ∑ j ∈ J a j 2 . Q_J(a) := \sum_{j\in J} a_j^2. Q J ( a ) := j ∈ J ∑ a j 2 .
Since J J J is a uniform sample without replacement from R R R , Hoeffding’s comparison theorem allows us to compare the moment generating function of Q J ( a ) Q_J(a) Q J ( a ) with that of a with-replacement sample. Let Y 1 , … , Y L Y_1,\ldots,Y_L Y 1 , … , Y L be independent and uniformly distributed over the finite population { a j 2 : j ∈ R } \{a_j^2:j\in R\} { a j 2 : j ∈ R } . Then
E Y ℓ = ∥ a ∥ 2 2 n , 0 ≤ Y ℓ ≤ ∥ a ∥ ∞ 2 , \mathbb{E}Y_\ell= \frac{\lVert a\rVert_2^2}{n}, \qquad0 \le Y_\ell\le\lVert a\rVert_\infty^2, E Y ℓ = n ∥ a ∥ 2 2 , 0 ≤ Y ℓ ≤ ∥ a ∥ ∞ 2 ,
and
Var ( Y ℓ ) ≤ ∥ a ∥ ∞ 2 ∥ a ∥ 2 2 n . \operatorname{Var}(Y_\ell) \le\frac{\lVert a\rVert_\infty^2\lVert a\rVert_2^2}{n}. Var ( Y ℓ ) ≤ n ∥ a ∥ ∞ 2 ∥ a ∥ 2 2 .
The standard Bernstein bound for the independent sample gives: for every t > 0 t>0 t > 0 , with probability at least 1 − e − t 1-e^{-t} 1 − e − t ,
Q J ( a ) ≤ 2 [ L n ∥ a ∥ 2 2 + ∥ a ∥ ∞ 2 t ] . (95) Q_J(a) \le2\left[\frac{L}{n}\lVert a\rVert_2^2+\lVert a\rVert_\infty^2t\right]. \tag*{(95)} Q J ( a ) ≤ 2 [ n L ∥ a ∥ 2 2 + ∥ a ∥ ∞ 2 t ] . (95)
Conditioning on J J J , since the signs are independent Rademachers, we have
Pr ( ∣ ∑ j ∈ J r j a j ∣ ≥ r ∣ J ) ≤ 2 exp ( − r 2 2 Q J ( a ) ) . \Pr\left(\left.\left\lvert\sum_{j\in J}r_ja_j\right\rvert\ge r\right|J\right)\le2\exp\left(-\frac{r^2}{2Q_J(a)}\right). Pr j ∈ J ∑ r j a j ≥ r J ≤ 2 exp ( − 2 Q J ( a ) r 2 ) .
Applying (95) with t = 2 u t=2u t = 2 u and then the Rademacher tail bound, we obtain, with probability at least 1 − 2 e − u 1-2e^{-u} 1 − 2 e − u ,
∣ ∑ j ∈ J r j a j ∣ ≤ C [ L n ∥ a ∥ 2 u + ∥ a ∥ ∞ u ] . (96) \left\lvert\sum_{j\in J}r_ja_j\right\rvert\le C\left[\sqrt{\frac{L}{n}}\lVert a\rVert_2\sqrt{u}+\lVert a\rVert_\infty u\right]. \tag*{(96)} j ∈ J ∑ r j a j ≤ C [ n L ∥ a ∥ 2 u + ∥ a ∥ ∞ u ] . (96)
For part (i), apply (96) with a = x R a=x_R a = x R and multiply by L − 1 / q L^{-1/q} L − 1/ q . Since p ≥ 2 p\ge2 p ≥ 2 ,
∥ x R ∥ 2 ≤ n 1 2 − 1 p ∥ x R ∥ p ≤ 4 n 1 2 − 1 p M − 1 / p . \lVert x_R\rVert_2\le n^{\frac{1}{2}-\frac{1}{p}}\lVert x_R\rVert_p\le4n^{\frac{1}{2}-\frac{1}{p}}M^{-1/p}. ∥ x R ∥ 2 ≤ n 2 1 − p 1 ∥ x R ∥ p ≤ 4 n 2 1 − p 1 M − 1/ p .
Hence the first term satisfies
L − 1 / q L n ∥ x R ∥ 2 u ≤ C M − 1 / p L 1 p − 1 2 n − 1 / p u ≤ C M − 1 / p M 1 2 − 1 p u d , L^{-1/q}\sqrt{\frac{L}{n}}\lVert x_R\rVert_2\sqrt{u}\le CM^{-1/p}L^{\frac{1}{p}-\frac{1}{2}}n^{-1/p}\sqrt{u}\le CM^{-1/p}M^{\frac{1}{2}-\frac{1}{p}}\sqrt{\frac{u}{d}}, L − 1/ q n L ∥ x R ∥ 2 u ≤ C M − 1/ p L p 1 − 2 1 n − 1/ p u ≤ C M − 1/ p M 2 1 − p 1 d u ,
where we used
d 16 M ≤ L ≤ d 8 M , n ≥ 7 d 8 . \frac{d}{16M}\le L\le\frac{d}{8M},\qquad n\ge\frac{7d}{8}. 16 M d ≤ L ≤ 8 M d , n ≥ 8 7 d .
For the second term, because x ∈ X x\in X x ∈ X , ∥ x R ∥ ∞ ≤ 4 d − 1 / p \lVert x_R\rVert_\infty\le4d^{-1/p} ∥ x R ∥ ∞ ≤ 4 d − 1/ p , and therefore
L − 1 / q ∥ x R ∥ ∞ u ≤ C ( M d ) 1 / q d − 1 / p u = C M 1 / q u d = C M − 1 / p M u d . L^{-1/q}\lVert x_R\rVert_\infty u\le C\left(\frac{M}{d}\right)^{1/q}d^{-1/p}u=CM^{1/q}\frac{u}{d}=CM^{-1/p}\frac{Mu}{d}. L − 1/ q ∥ x R ∥ ∞ u ≤ C ( d M ) 1/ q d − 1/ p u = C M 1/ q d u = C M − 1/ p d M u .
This proves (93) . For part (ii), apply (96) with a = s R a=s_R a = s R and multiply by L − 1 / p L^{-1/p} L − 1/ p . Since q ≤ 2 q\le2 q ≤ 2 ,
∥ s R ∥ 2 2 = ∑ j ∈ R ∣ s j ∣ 2 = ∑ j ∈ R ∣ s j ∣ 2 − q ∣ s j ∣ q ≤ ∥ s ∥ ∞ 2 − q ∥ s ∥ q q ≤ L − ( 2 − q ) / q , \|s_R\|_2^2=\sum_{j\in R}|s_j|^2=\sum_{j\in R}|s_j|^{2-q}|s_j|^q\le\|s\|_\infty^{2-q}\|s\|_q^q\le L^{-(2-q)/q}, ∥ s R ∥ 2 2 = j ∈ R ∑ ∣ s j ∣ 2 = j ∈ R ∑ ∣ s j ∣ 2 − q ∣ s j ∣ q ≤ ∥ s ∥ ∞ 2 − q ∥ s ∥ q q ≤ L − ( 2 − q ) / q ,
so ∥ s R ∥ 2 ≤ L 1 p − 1 2 \|s_R\|_2\le L^{\frac{1}{p}-\frac{1}{2}} ∥ s R ∥ 2 ≤ L p 1 − 2 1 . Thus L − 1 / p L n ∥ s R ∥ 2 u ≤ C u d L^{-1/p}\sqrt{\frac{L}{n}}\|s_R\|_2\sqrt{u}\le C\sqrt{\frac{u}{d}} L − 1/ p n L ∥ s R ∥ 2 u ≤ C d u , while
L − 1 / p ∥ s R ∥ ∞ u ≤ L − 1 / p L − 1 / q u = u L ≤ C M u d . L^{-1/p}\|s_R\|_\infty u\le L^{-1/p}L^{-1/q}u=\frac{u}{L}\le C\frac{Mu}{d}. L − 1/ p ∥ s R ∥ ∞ u ≤ L − 1/ p L − 1/ q u = L u ≤ C d M u .
which proves (94) .
The high-probability event
Lemma 33. With the choices
τ = C M − 1 / p [ M 1 2 − 1 p log ( 16 K M 2 / ρ ) d + M log ( 16 K M 2 / ρ ) d ] (97) \tau=CM^{-1/p}\left[M^{\frac{1}{2}-\frac{1}{p}}\sqrt{\frac{\log(16KM^2/\rho)}{d}}+\frac{M\log(16KM^2/\rho)}{d}\right] \tag*{(97)} τ = C M − 1/ p [ M 2 1 − p 1 d log ( 16 K M 2 / ρ ) + d M log ( 16 K M 2 / ρ ) ] (97) and
θ = C [ log ( 16 K M 2 / ρ ) d + M log ( 16 K M 2 / ρ ) d ] , (98) \theta=C\left[\sqrt{\frac{\log(16KM^2/\rho)}{d}}+\frac{M\log(16KM^2/\rho)}{d}\right], \tag*{(98)} θ = C [ d log ( 16 K M 2 / ρ ) + d M log ( 16 K M 2 / ρ ) ] , (98) and with C C C sufficiently large, the following statements hold simultaneously with probability at least 1 − ρ 1-\rho 1 − ρ :
(i) for every round t < M t<M t < M , every query x x x in that round satisfying ∥ x R t ∥ p ≤ 4 M − 1 / p \|x_{R_t}\|_p\le4M^{-1/p} ∥ x R t ∥ p ≤ 4 M − 1/ p , and every i ≥ t i\ge t i ≥ t ,
∣ ⟨ z i , x ⟩ ∣ ≤ τ ; |\langle z_i,x\rangle|\le\tau; ∣ ⟨ z i , x ⟩ ∣ ≤ τ ; (ii) for every round t < M t<M t < M , every query x x x in that round, and every i ≥ t i\ge t i ≥ t ,
∣ ⟨ h i , s t ( x ) ⟩ ∣ ≤ θ / 2 ; |\langle h_i,s_t(x)\rangle|\le\theta/2; ∣ ⟨ h i , s t ( x )⟩ ∣ ≤ θ /2 ; (iii) after round M − 1 M-1 M − 1 , the final output satisfies the analogous two bounds with the fresh last block J M J_M J M .
Proof. For each t ∈ [ M ] t\in[M] t ∈ [ M ] , let F t − 1 \mathcal{F}_{t-1} F t − 1 denote the σ \sigma σ -field generated by all randomness revealed before round t t t . In particular, J 1 , … , J t − 1 J_1,\ldots,J_{t-1} J 1 , … , J t − 1 and their signs are F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, and therefore
R t = [ d ] ∖ ⋃ j < t J j R_t=[d]\setminus\bigcup_{j<t}J_j R t = [ d ] ∖ j < t ⋃ J j
is fixed under this conditioning. Since
∣ R t ∣ = d − ( t − 1 ) L ≥ d − M L ≥ 7 d 8 , |R_t|=d-(t-1)L\ge d-ML\ge\frac{7d}{8}, ∣ R t ∣ = d − ( t − 1 ) L ≥ d − M L ≥ 8 7 d ,
Lemma 32 applies on R t R_t R t . Let
x t , 1 , … , x t , K t , K t ≤ K , x_{t,1},\ldots,x_{t,K_t},\qquad K_t\le K, x t , 1 , … , x t , K t , K t ≤ K ,
be the queries in round t t t . Conditioned on F t − 1 \mathcal{F}_{t-1} F t − 1 , these queries are fixed. Moreover, every query belongs to X X X . For every i ≥ t i \ge t i ≥ t , the conditional marginal distribution of J i J_i J i is uniform over the L L L -subsets of R t R_t R t , and its signs are fresh. Thus Lemma 32 applies to each fixed triple ( t , k , i ) (t,k,i) ( t , k , i ) . Set
u : = log ( 16 K M 2 ρ ) . u := \log\left(\frac{16KM^2}{\rho}\right). u := log ( ρ 16 K M 2 ) .
If ∥ ( x t , k ) R t ∥ p ≤ 4 M − 1 / p \|(x_{t,k})_{R_t}\|_p \le4M^{-1/p} ∥ ( x t , k ) R t ∥ p ≤ 4 M − 1/ p , part (i) of Lemma 32 , together with the choice of τ \tau τ , gives
Pr ( ∣ ⟨ z i , x t , k ⟩ ∣ > τ ∣ F t − 1 ) ≤ 4 e − u = ρ 4 K M 2 . \Pr\left(\left|\langle z_i,x_{t,k}\rangle\right|>\tau\mid\mathcal{F}_{t-1}\right) \le4e^{-u} = \frac{\rho}{4KM^2}. Pr ( ∣ ⟨ z i , x t , k ⟩ ∣ > τ ∣ F t − 1 ) ≤ 4 e − u = 4 K M 2 ρ .
If the residual norm is larger, the corresponding hard-direction event is irrelevant. For the wall bounds, conditioned on F t − 1 \mathcal{F}_{t-1} F t − 1 , the set C t C_t C t and the query x t , k x_{t,k} x t , k are fixed. By Lemma 30 , s t ( x t , k ) s_t(x_{t,k}) s t ( x t , k ) is uniquely determined, and
∥ s t ( x t , k ) ∥ q ≤ 1 , ∥ s t ( x t , k ) ∥ ∞ ≤ L − 1 / q . \|s_t(x_{t,k})\|_q \le1,\qquad\|s_t(x_{t,k})\|_\infty\le L^{-1/q}. ∥ s t ( x t , k ) ∥ q ≤ 1 , ∥ s t ( x t , k ) ∥ ∞ ≤ L − 1/ q .
Part (ii) of Lemma 32 , together with the choice of θ \theta θ , gives
Pr ( ∣ ⟨ h i , s t ( x t , k ) ⟩ ∣ > θ / 2 ∣ F t − 1 ) ≤ ρ 4 K M 2 . \Pr\left(\left|\langle h_i,s_t(x_{t,k})\rangle\right|>\theta/2 \mid\mathcal{F}_{t-1}\right) \le\frac{\rho}{4KM^2}. Pr ( ∣ ⟨ h i , s t ( x t , k )⟩ ∣ > θ /2 ∣ F t − 1 ) ≤ 4 K M 2 ρ .
The same argument applies to the final output x ^ \widehat{x} x after round M − 1 M-1 M − 1 . By assumption x ^ ∈ X \widehat{x}\in X x ∈ X , and conditioned on F M − 1 \mathcal{F}_{M-1} F M − 1 the block J M J_M J M and its signs are still fresh. Therefore the analogous hard-direction and wall bounds each fail with probability at most ρ / ( 4 K M 2 ) \rho/(4KM^2) ρ / ( 4 K M 2 ) . There are fewer than K M 2 KM^2 K M 2 hard-direction events and fewer than K M 2 KM^2 K M 2 wall events, together with two final-output events. Since M ≥ 2 M\ge2 M ≥ 2 and K ≥ 1 K\ge1 K ≥ 1 ,
2 K M 2 + 2 ≤ 4 K M 2 . 2KM^2+2\le4KM^2. 2 K M 2 + 2 ≤ 4 K M 2 .
A union bound therefore gives total failure probability at most ρ \rho ρ .
Proof of Theorem 18
Proof of Theorem 18. First consider a deterministic algorithm. The function f f f is convex and 1-Lipschitz with respect to ℓ p \ell_p ℓ p : every affine hard piece has dual norm 1 / 32 1/32 1/32 , while Lemma 30 gives that the wall is 1-Lipschitz. On the event in Lemma 33 , Lemma 31 shows that f f f and f t f_t f t agree on a neighborhood of every round-t t t query. Induction over the rounds therefore shows that the complete transcripts coincide through round M − 1 M-1 M − 1 . Let
u = log ( 16 K M 2 ρ ) . u=\log\left(\frac{16KM^2}{\rho}\right). u = log ( ρ 16 K M 2 ) .
The assumption (82) , with c c c sufficiently small relative to the constant in (97) , implies
M τ ≤ 1 64 M − 1 / p . (99) M\tau\le\frac{1}{64}M^{-1/p}. \tag*{(99)} M τ ≤ 64 1 M − 1/ p . (99)
Thus (99) follows by choosing c c c sufficiently small. The same assumption also implies M θ ≤ 1 M\theta\le1 M θ ≤ 1 . We next bound the optimum. By Lemma 29 ,
W ( x ∗ ) ≤ − 1 32 M − 1 / p . W(x^*)\le-\frac{1}{32}M^{-1/p}. W ( x ∗ ) ≤ − 32 1 M − 1/ p .
Also, for every i i i ,
1 32 ⟨ z i , x ∗ ⟩ − i τ 8 = − 1 32 M − 1 / p − i τ 8 ≤ − 1 32 M − 1 / p . \frac{1}{32}\langle z_i,x^*\rangle-\frac{i\tau}{8}=-\frac{1}{32}M^{-1/p}-\frac{i\tau}{8}\leq-\frac{1}{32}M^{-1/p}. 32 1 ⟨ z i , x ∗ ⟩ − 8 i τ = − 32 1 M − 1/ p − 8 i τ ≤ − 32 1 M − 1/ p .
Since x ∗ ∈ X x^*\in X x ∗ ∈ X ,
f ∗ : = min x ∈ X f ( x ) ≤ f ( x ∗ ) ≤ − 1 32 M − 1 / p . (100) f^*:=\min_{x\in X}f(x)\leq f(x^*)\leq-\frac{1}{32}M^{-1/p}. \tag*{(100)} f ∗ := x ∈ X min f ( x ) ≤ f ( x ∗ ) ≤ − 32 1 M − 1/ p . (100)
Consider any query x x x made in round t < M t<M t < M . If ∥ x R t ∥ p > 4 M − 1 / p \|x_{R_t}\|_p>4M^{-1/p} ∥ x R t ∥ p > 4 M − 1/ p , then Lemmas 31 and 29 give
f ( x ) = f t ( x ) ≥ W t ( x ) ≥ ∥ x R t ∥ p − 17 16 M − 1 / p > 47 16 M − 1 / p . f(x)=f_t(x)\geq W_t(x)\geq\|x_{R_t}\|_p-\frac{17}{16}M^{-1/p}>\frac{47}{16}M^{-1/p}. f ( x ) = f t ( x ) ≥ W t ( x ) ≥ ∥ x R t ∥ p − 16 17 M − 1/ p > 16 47 M − 1/ p .
If instead ∥ x R t ∥ p ≤ 4 M − 1 / p \|x_{R_t}\|_p\leq4M^{-1/p} ∥ x R t ∥ p ≤ 4 M − 1/ p , Lemma 33 gives ∣ ⟨ z t , x ⟩ ∣ ≤ τ |\langle z_t,x\rangle|\leq\tau ∣ ⟨ z t , x ⟩ ∣ ≤ τ , and therefore
f ( x ) = f t ( x ) ≥ 1 32 ⟨ z t , x ⟩ − t τ 8 ≥ − τ 32 − M τ 8 . f(x)=f_t(x)\geq\frac{1}{32}\langle z_t,x\rangle-\frac{t\tau}{8}\geq-\frac{\tau}{32}-\frac{M\tau}{8}. f ( x ) = f t ( x ) ≥ 32 1 ⟨ z t , x ⟩ − 8 t τ ≥ − 32 τ − 8 M τ .
Combining this with (99) and (100) , in either case,
f ( x ) − f ∗ ≥ 1 64 M − 1 / p . (101) f(x)-f^*\geq\frac{1}{64}M^{-1/p}. \tag*{(101)} f ( x ) − f ∗ ≥ 64 1 M − 1/ p . (101)
It remains to treat the final output x ^ \widehat{x} x after round M − 1 M-1 M − 1 . Since x ^ \widehat{x} x is determined by the first M − 1 M-1 M − 1 blocks, J M J_M J M is still uniform over the L L L -subsets of R M R_M R M , with independent signs. We now apply part (iii) of Lemma 33 . If ∥ x ^ R M ∥ p > 4 M − 1 / p \|\widehat{x}_{R_M}\|_p>4M^{-1/p} ∥ x R M ∥ p > 4 M − 1/ p , then part (iii) of Lemma 33 gives
∣ ⟨ h M , s M ( x ^ ) ⟩ ∣ ≤ θ / 2. |\langle h_M,s_M(\widehat{x})\rangle|\leq\theta/2. ∣ ⟨ h M , s M ( x )⟩ ∣ ≤ θ /2.
Hence Lemma 30 gives W ( x ^ ) = W M ( x ^ ) W(\widehat{x})=W_M(\widehat{x}) W ( x ) = W M ( x ) , and Lemma 29 yields
f ( x ^ ) ≥ W ( x ^ ) = W M ( x ^ ) > 47 16 M − 1 / p . f(\widehat{x})\geq W(\widehat{x})=W_M(\widehat{x})>\frac{47}{16}M^{-1/p}. f ( x ) ≥ W ( x ) = W M ( x ) > 16 47 M − 1/ p .
If instead ∥ x ^ R M ∥ p ≤ 4 M − 1 / p \|\widehat{x}_{R_M}\|_p\leq4M^{-1/p} ∥ x R M ∥ p ≤ 4 M − 1/ p , part (iii) gives ∣ ⟨ z M , x ^ ⟩ ∣ ≤ τ |\langle z_M,\widehat{x}\rangle|\leq\tau ∣ ⟨ z M , x ⟩ ∣ ≤ τ , so the M M M -th affine piece gives
f ( x ^ ) ≥ − τ 32 − M τ 8 . f(\widehat{x})\geq-\frac{\tau}{32}-\frac{M\tau}{8}. f ( x ) ≥ − 32 τ − 8 M τ .
Thus (101) also holds for the final output. Choose the constant C C C in (81) sufficiently large that
1 64 M − 1 / p > ε . \frac{1}{64}M^{-1/p}>\varepsilon. 64 1 M − 1/ p > ε .
Then, on the event of Lemma 33 , no output produced in fewer than M M M rounds is an ε \varepsilon ε -minimizer. This event has probability at least 1 − ρ 1-\rho 1 − ρ . Finally, let the algorithm be randomized, and let ω \omega ω denote its random choices, which are independent of the blocks and signs. For every fixed ω \omega ω the algorithm is deterministic, so the argument above shows that it returns an ε \varepsilon ε -minimizer in fewer than M M M rounds with probability at most ρ \rho ρ over the blocks and signs. Averaging over ω \omega ω , the same bound holds over ω \omega ω , the blocks and the signs jointly. Hence, for some fixed choice of blocks and signs, the probability over ω \omega ω alone is at most ρ \rho ρ , and the corresponding f f f is the function in the theorem. The condition (81) gives
M ≲ ε − p , M\lesssim\varepsilon^{-p}, M ≲ ε − p ,
while (82) gives, up to logarithmic factors,
M ≲ d p 3 p − 2 . M\lesssim d^{\frac{p}{3p-2}}. M ≲ d 3 p − 2 p .
This proves the theorem.
Conclusion
In this work, we studied parallel SCO in ℓ p \ell_p ℓ p geometries for p ∈ [ 2 , ∞ ) p \in[2,\infty) p ∈ [ 2 , ∞ ) . We first introduced a class of ODCp _p p algorithms, and presented various applications of these tools. We then showed how to use these algorithms to develop two efficient ℓ p \ell_p ℓ p parallel algorithms, which in favorable geometries, improves upon the corresponding sequential stochastic query depth complexity. We finally present a dimension-dependent lower bound on the number of adaptive rounds required for parallel ℓ p \ell_p ℓ p nonsmooth convex optimization, improving upon that of [18 ] . One interesting extension to this work would be to consider how these tools could be used in a private SCO setting, where the goal is to produce an iterate that has at most ε o p t \varepsilon_{\mathrm{opt}} ε opt expected population risk more than the optimal risk. One work that has studied private SCO in ℓ p \ell_p ℓ p geometries is [8 ] , and it would be interesting to improve upon the upper and lower bounds presented here.
AI Methodology Statement.
We used ChatGPT 5.6 and Claude to assist with checking and aiding some steps of our proofs. In particular, the addition of the quadratic term in the wall function was suggested by ChatGPT. Generative AI tools were not used for devising the main ideas and conclusions of the paper. The authors have reviewed all AI-assisted work and take full responsibility for the contents of the paper.
Helper Facts
We present a few useful properties and facts about Bregman divergences, which we use in our analysis.
Property 1. (Three Point Property ). For all a , b , c ∈ R d a,b,c \in\mathbb{R}^{d} a , b , c ∈ R d and convex Φ \Phi Φ , it holds that
⟨ ∇ Φ ( a ) − ∇ Φ ( b ) , c − a ⟩ = D Φ ( c , b ) − D Φ ( c , a ) − D Φ ( a , b ) . \langle\nabla\Phi(a)-\nabla\Phi(b),c-a\rangle= D_{\Phi}(c,b)-D_{\Phi}(c,a)-D_{\Phi}(a,b). ⟨ ∇Φ ( a ) − ∇Φ ( b ) , c − a ⟩ = D Φ ( c , b ) − D Φ ( c , a ) − D Φ ( a , b ) .
Fact 2. Let Φ ( x ) = 1 p ∥ x ∥ p p \Phi(x)=\frac{1}{p}\lVert x\rVert_{p}^{p} Φ ( x ) = p 1 ∥ x ∥ p p for p ≥ 2 p\ge2 p ≥ 2 . For all x , y ∈ R d x,y\in\mathbb{R}^{d} x , y ∈ R d , it holds that
D Φ ( x , y ) ≥ 2 2 − p p ∥ x − y ∥ p p . D_{\Phi}(x,y)\ge\frac{2^{2-p}}{p}\lVert x-y\rVert_{p}^{p}. D Φ ( x , y ) ≥ p 2 2 − p ∥ x − y ∥ p p .
Proof. Fix x , y ∈ R d x,y\in\mathbb{R}^{d} x , y ∈ R d . It holds that
D Φ ( x , y ) = ∫ 0 1 ⟨ ∇ Φ ( y + t ( x − y ) ) − ∇ Φ ( y ) , x − y ⟩ d t D_{\Phi}(x,y)=\int_{0}^{1}\langle\nabla\Phi(y+t(x-y))-\nabla\Phi(y),x-y\rangle\,dt D Φ ( x , y ) = ∫ 0 1 ⟨ ∇Φ ( y + t ( x − y )) − ∇Φ ( y ) , x − y ⟩ d t
Since Φ ( x ) = 1 p ∑ i = 1 d ∣ x i ∣ p \Phi(x)=\frac{1}{p}\sum_{i=1}^{d}|x_i|^{p} Φ ( x ) = p 1 ∑ i = 1 d ∣ x i ∣ p , we analyze (slightly overloading notation here) for ψ ( t ) = ∣ t ∣ p − 2 t \psi(t)=|t|^{p-2}t ψ ( t ) = ∣ t ∣ p − 2 t :
I ψ ( x , y ) = ∫ 0 1 ( ψ ( y + t ( x − y ) ) − ψ ( y ) ) ( x − y ) d t I_{\psi}(x,y)=\int_{0}^{1}\bigl(\psi(y+t(x-y))-\psi(y)\bigr)(x-y)\,dt I ψ ( x , y ) = ∫ 0 1 ( ψ ( y + t ( x − y )) − ψ ( y ) ) ( x − y ) d t
We now show that for all a , b ∈ R a,b\in\mathbb{R} a , b ∈ R , it holds that ( ψ ( a ) − ψ ( b ) ) ( a − b ) ≥ 2 2 − p ∣ a − b ∣ p (\psi(a)-\psi(b))(a-b)\ge2^{2-p}|a-b|^{p} ( ψ ( a ) − ψ ( b )) ( a − b ) ≥ 2 2 − p ∣ a − b ∣ p . If a = b a=b a = b , the result trivially holds, so we only consider the setting in which a ≠ b a\ne b a = b . Since ψ ′ ( t ) = ( p − 1 ) ∣ t ∣ p − 2 \psi'(t)=(p-1)|t|^{p-2} ψ ′ ( t ) = ( p − 1 ) ∣ t ∣ p − 2 , it holds that
( ψ ( a ) − ψ ( b ) ) ( a − b ) = ( a − b ) 2 ( p − 1 ) ∫ 0 1 ∣ b + s ( a − b ) ∣ p − 2 d s (\psi(a)-\psi(b))(a-b)=(a-b)^{2}(p-1)\int_{0}^{1}|b+s(a-b)|^{p-2}\,ds ( ψ ( a ) − ψ ( b )) ( a − b ) = ( a − b ) 2 ( p − 1 ) ∫ 0 1 ∣ b + s ( a − b ) ∣ p − 2 d s
Letting u = b / ( a − b ) u=b/(a-b) u = b / ( a − b ) so b + ( a − b ) s = ( a − b ) ( u + s ) b+(a-b)s=(a-b)(u+s) b + ( a − b ) s = ( a − b ) ( u + s ) , we write
∫ 0 1 ∣ b + s ( a − b ) ∣ p − 2 d s = ∣ a − b ∣ p − 2 ∫ 0 1 ∣ u + s ∣ p − 2 d s \int_{0}^{1}|b+s(a-b)|^{p-2}\,ds=|a-b|^{p-2}\int_{0}^{1}|u+s|^{p-2}\,ds ∫ 0 1 ∣ b + s ( a − b ) ∣ p − 2 d s = ∣ a − b ∣ p − 2 ∫ 0 1 ∣ u + s ∣ p − 2 d s
Letting F ( u ) = ∫ 0 1 ∣ u + s ∣ p − 2 d s F(u)=\int_{0}^{1}|u+s|^{p-2}\,ds F ( u ) = ∫ 0 1 ∣ u + s ∣ p − 2 d s , observe that the minimum of F F F occurs at u = − 1 2 u=-\frac{1}{2} u = − 2 1 , since ∣ x ∣ p − 2 |x|^{p-2} ∣ x ∣ p − 2 is convex for p ≥ 2 p\ge2 p ≥ 2 . Therefore,
∣ a − b ∣ p − 2 ∫ 0 1 ∣ u + s ∣ p − 2 d s ≥ ∣ a − b ∣ p − 2 ∫ 0 1 ∣ s − 1 2 ∣ p − 2 d s = ∣ a − b ∣ p − 2 ∫ − 1 / 2 1 / 2 ∣ u ∣ p − 2 d u = ∣ a − b ∣ p − 2 ⋅ 2 2 − p p − 1 |a-b|^{p-2}\int_{0}^{1}|u+s|^{p-2}\,ds\ge|a-b|^{p-2}\int_{0}^{1}\left|s-\frac{1}{2}\right|^{p-2}\,ds=|a-b|^{p-2}\int_{-1/2}^{1/2}|u|^{p-2}\,du=|a-b|^{p-2}\cdot\frac{2^{2-p}}{p-1} ∣ a − b ∣ p − 2 ∫ 0 1 ∣ u + s ∣ p − 2 d s ≥ ∣ a − b ∣ p − 2 ∫ 0 1 s − 2 1 p − 2 d s = ∣ a − b ∣ p − 2 ∫ − 1/2 1/2 ∣ u ∣ p − 2 d u = ∣ a − b ∣ p − 2 ⋅ p − 1 2 2 − p
which implies that ( ψ ( a ) − ψ ( b ) ) ( a − b ) ≥ 2 2 − p ∣ a − b ∣ p (\psi(a)-\psi(b))(a-b)\ge2^{2-p}|a-b|^{p} ( ψ ( a ) − ψ ( b )) ( a − b ) ≥ 2 2 − p ∣ a − b ∣ p . Further, it holds that
( ψ ( y + t ( x − y ) ) − ψ ( y ) ) ( x − y ) = 1 t ( ψ ( y + t ( x − y ) ) − ψ ( y ) ) ( t x − t y ) ≥ 2 2 − p ∣ x − y ∣ p t p − 1 \bigl(\psi(y+t(x-y))-\psi(y)\bigr)(x-y)=\frac{1}{t}\bigl(\psi(y+t(x-y))-\psi(y)\bigr)(tx-ty)\ge2^{2-p}|x-y|^{p}t^{p-1} ( ψ ( y + t ( x − y )) − ψ ( y ) ) ( x − y ) = t 1 ( ψ ( y + t ( x − y )) − ψ ( y ) ) ( t x − t y ) ≥ 2 2 − p ∣ x − y ∣ p t p − 1
which means that I ψ ( x , y ) ≥ 2 2 − p ∣ x − y ∣ p ∫ 0 1 t p − 1 d t = 2 2 − p p ∣ x − y ∣ p I_{\psi}(x,y)\ge2^{2-p}|x-y|^{p}\int_{0}^{1}t^{p-1}\,dt=\frac{2^{2-p}}{p}|x-y|^{p} I ψ ( x , y ) ≥ 2 2 − p ∣ x − y ∣ p ∫ 0 1 t p − 1 d t = p 2 2 − p ∣ x − y ∣ p . As D Φ = ∑ i = 1 d I ψ ( x i , y i ) D_{\Phi}=\sum_{i=1}^{d}I_{\psi}(x_i,y_i) D Φ = ∑ i = 1 d I ψ ( x i , y i ) , the claim follows.
Fact 3. Let Φ ( x ) = 1 p ∥ x ∥ p p \Phi(x)=\frac{1}{p}\lVert x\rVert_{p}^{p} Φ ( x ) = p 1 ∥ x ∥ p p for some p ≥ 2 p\ge2 p ≥ 2 . For all x , y ∈ R d x,y\in\mathbb{R}^{d} x , y ∈ R d , it holds that
D Φ ( x , y ) ≤ p − 1 2 ( ∥ x ∥ p + 2 ∥ y ∥ p ) p − 2 ∥ x − y ∥ p 2 . D_{\Phi}(x,y)\le\frac{p-1}{2}\bigl(\lVert x\rVert_{p}+2\lVert y\rVert_{p}\bigr)^{p-2}\lVert x-y\rVert_{p}^{2}. D Φ ( x , y ) ≤ 2 p − 1 ( ∥ x ∥ p + 2 ∥ y ∥ p ) p − 2 ∥ x − y ∥ p 2 .
Proof. We first prove a smoothness-like guarantee for Φ \Phi Φ : ∥ ∇ Φ ( x ) − ∇ Φ ( y ) ∥ q ≤ ( p − 1 ) ( ∥ x ∥ p + ∥ y ∥ p ) p − 2 ∥ x − y ∥ p \|\nabla\Phi(x)-\nabla\Phi(y)\|_q \le(p-1)(\|x\|_p+\|y\|_p)^{p-2}\|x-y\|_p ∥∇Φ ( x ) − ∇Φ ( y ) ∥ q ≤ ( p − 1 ) ( ∥ x ∥ p + ∥ y ∥ p ) p − 2 ∥ x − y ∥ p . Since ∇ Φ ( x ) i = ψ ( x i ) \nabla\Phi(x)_i=\psi(x_i) ∇Φ ( x ) i = ψ ( x i ) , ψ ( t ) = ∣ t ∣ p − 2 t \psi(t)=|t|^{p-2}t ψ ( t ) = ∣ t ∣ p − 2 t , we have that for all a , b ∈ R a,b\in\mathbb{R} a , b ∈ R :
∣ ψ ( a ) − ψ ( b ) ∣ = ( p − 1 ) ∣ a − b ∣ ∫ 0 1 ∣ b + s ( a − b ) ∣ p − 2 d s ≤ ( p − 1 ) ∣ a − b ∣ ∫ 0 1 ( ( 1 − s ) ∣ b ∣ + s ∣ a ∣ ) p − 2 d s ≤ ( p − 1 ) ∣ a − b ∣ ∫ 0 1 ( ∣ a ∣ + ∣ b ∣ ) p − 2 d s = ( p − 1 ) ∣ a − b ∣ ( ∣ a ∣ + ∣ b ∣ ) p − 2 \begin{aligned}
|\psi(a)-\psi(b)|=(p-1)|a-b|\int_0^1|b+s(a-b)|^{p-2}\,ds \\
&\le(p-1)|a-b|\int_0^1((1-s)|b|+s|a|)^{p-2}\,ds \\
&\le(p-1)|a-b|\int_0^1(|a|+|b|)^{p-2}\,ds=(p-1)|a-b|(|a|+|b|)^{p-2}
\end{aligned} ∣ ψ ( a ) − ψ ( b ) ∣ = ( p − 1 ) ∣ a − b ∣ ∫ 0 1 ∣ b + s ( a − b ) ∣ p − 2 d s ≤ ( p − 1 ) ∣ a − b ∣ ∫ 0 1 (( 1 − s ) ∣ b ∣ + s ∣ a ∣ ) p − 2 d s ≤ ( p − 1 ) ∣ a − b ∣ ∫ 0 1 ( ∣ a ∣ + ∣ b ∣ ) p − 2 d s = ( p − 1 ) ∣ a − b ∣ ( ∣ a ∣ + ∣ b ∣ ) p − 2
Thus, for each coordinate: ∣ ∇ Φ ( x i ) − ∇ Φ ( y i ) ∣ ≤ ( p − 1 ) ( ∣ x i ∣ + ∣ y i ∣ ) p − 2 ∣ x i − y i ∣ |\nabla\Phi(x_i)-\nabla\Phi(y_i)|\le(p-1)(|x_i|+|y_i|)^{p-2}|x_i-y_i| ∣∇Φ ( x i ) − ∇Φ ( y i ) ∣ ≤ ( p − 1 ) ( ∣ x i ∣ + ∣ y i ∣ ) p − 2 ∣ x i − y i ∣ so we have that
∥ ∇ Φ ( x ) − ∇ Φ ( y ) ∥ q = ( ∑ i ∣ ∇ Φ ( x i ) − ∇ Φ ( y i ) ∣ q ) 1 / q ≤ ( p − 1 ) ( ∑ i ( ∣ x i ∣ + ∣ y i ∣ ) ( p − 2 ) q ∣ x i − y i ∣ q ) 1 / q \|\nabla\Phi(x)-\nabla\Phi(y)\|_q=\left(\sum_i|\nabla\Phi(x_i)-\nabla\Phi(y_i)|^q\right)^{1/q}\le(p-1)\left(\sum_i(|x_i|+|y_i|)^{(p-2)q}|x_i-y_i|^q\right)^{1/q} ∥∇Φ ( x ) − ∇Φ ( y ) ∥ q = ( i ∑ ∣∇Φ ( x i ) − ∇Φ ( y i ) ∣ q ) 1/ q ≤ ( p − 1 ) ( i ∑ ( ∣ x i ∣ + ∣ y i ∣ ) ( p − 2 ) q ∣ x i − y i ∣ q ) 1/ q
Applying Hölder’s inequality, we have that
∑ i ( ∣ x i ∣ + ∣ y i ∣ ) ( p − 2 ) q ∣ x i − y i ∣ q ≤ ( ∑ i ( ∣ x i ∣ + ∣ y i ∣ ) ( p − 2 ) q ⋅ ( p − 1 ) p − 2 ) p − 2 p − 1 ⋅ ( ∑ i ∣ x i − y i ∣ q ( p − 1 ) ) 1 p − 1 ≤ ( ∑ i ( ∣ x i ∣ + ∣ y i ∣ ) p ) p − 2 p − 1 ⋅ ( ∑ i ∣ x i − y i ∣ p ) 1 p − 1 \begin{aligned}
\sum_i(|x_i|+|y_i|)^{(p-2)q}|x_i-y_i|^q\le\left(\sum_i(|x_i|+|y_i|)^{\frac{(p-2)q\cdot(p-1)}{p-2}}\right)^{\frac{p-2}{p-1}}\cdot\left(\sum_i|x_i-y_i|^{q(p-1)}\right)^{\frac{1}{p-1}} \\
&\le\left(\sum_i(|x_i|+|y_i|)^p\right)^{\frac{p-2}{p-1}}\cdot\left(\sum_i|x_i-y_i|^p\right)^{\frac{1}{p-1}}
\end{aligned} i ∑ ( ∣ x i ∣ + ∣ y i ∣ ) ( p − 2 ) q ∣ x i − y i ∣ q ≤ ( i ∑ ( ∣ x i ∣ + ∣ y i ∣ ) p − 2 ( p − 2 ) q ⋅ ( p − 1 ) ) p − 1 p − 2 ⋅ ( i ∑ ∣ x i − y i ∣ q ( p − 1 ) ) p − 1 1 ≤ ( i ∑ ( ∣ x i ∣ + ∣ y i ∣ ) p ) p − 1 p − 2 ⋅ ( i ∑ ∣ x i − y i ∣ p ) p − 1 1
which implies that
( ∑ i ( ∣ x i ∣ + ∣ y i ∣ ) ( p − 2 ) q ∣ x i − y i ∣ q ) 1 / q ≤ ( ∑ i ( ∣ x i ∣ + ∣ y i ∣ ) p ) p − 2 p ⋅ ( ∑ i ∣ x i − y i ∣ p ) 1 / p = ( ∑ i ( ∣ x i ∣ + ∣ y i ∣ ) p ) 1 / p ( p − 2 ) ⋅ ∥ x − y ∥ p = ( ( ∑ i ( ∣ x i ∣ + ∣ y i ∣ ) p ) 1 / p ) p − 2 ⋅ ∥ x − y ∥ p ≤ ( ( ∑ i ∣ x i ∣ p ) 1 / p + ( ∑ i ∣ y i ∣ p ) 1 / p ) p − 2 ⋅ ∥ x − y ∥ p = ( ∥ x ∥ p + ∥ y ∥ p ) p − 2 ⋅ ∥ x − y ∥ p \begin{aligned}
\left(\sum_i(|x_i|+|y_i|)^{(p-2)q}|x_i-y_i|^q\right)^{1/q}\le\left(\sum_i(|x_i|+|y_i|)^p\right)^{\frac{p-2}{p}}\cdot\left(\sum_i|x_i-y_i|^p\right)^{1/p} \\
&=\left(\sum_i(|x_i|+|y_i|)^p\right)^{1/p(p-2)}\cdot\|x-y\|_p \\
&=\left(\left(\sum_i(|x_i|+|y_i|)^p\right)^{1/p}\right)^{p-2}\cdot\|x-y\|_p \\
&\le\left(\left(\sum_i|x_i|^p\right)^{1/p}+\left(\sum_i|y_i|^p\right)^{1/p}\right)^{p-2}\cdot\|x-y\|_p \\
&=(\|x\|_p+\|y\|_p)^{p-2}\cdot\|x-y\|_p
\end{aligned} ( i ∑ ( ∣ x i ∣ + ∣ y i ∣ ) ( p − 2 ) q ∣ x i − y i ∣ q ) 1/ q ≤ ( i ∑ ( ∣ x i ∣ + ∣ y i ∣ ) p ) p p − 2 ⋅ ( i ∑ ∣ x i − y i ∣ p ) 1/ p = ( i ∑ ( ∣ x i ∣ + ∣ y i ∣ ) p ) 1/ p ( p − 2 ) ⋅ ∥ x − y ∥ p = ( i ∑ ( ∣ x i ∣ + ∣ y i ∣ ) p ) 1/ p p − 2 ⋅ ∥ x − y ∥ p ≤ ( i ∑ ∣ x i ∣ p ) 1/ p + ( i ∑ ∣ y i ∣ p ) 1/ p p − 2 ⋅ ∥ x − y ∥ p = ( ∥ x ∥ p + ∥ y ∥ p ) p − 2 ⋅ ∥ x − y ∥ p
where we used Minkowski’s inequality in the last inequality. Therefore, it holds that
D Φ ( x , y ) = ∫ 0 1 ⟨ ∇ Φ ( y + t ( x − y ) ) − ∇ Φ ( y ) , x − y ⟩ d t ≤ ∫ 0 1 ∥ ∇ Φ ( y + t ( x − y ) ) − ∇ Φ ( y ) ∥ q ⋅ ∥ x − y ∥ p d t ≤ ∫ 0 1 ( p − 1 ) ⋅ ( ∥ y + t ( x − y ) ∥ p + ∥ y ∥ p ) p − 2 ⋅ ∥ t ( x − y ) ∥ p ⋅ ∥ x − y ∥ p d t = ( p − 1 ) ∥ x − y ∥ p 2 ∫ 0 1 t ( ∥ y + t ( x − y ) ∥ p + ∥ y ∥ p ) p − 2 d t ≤ ( p − 1 ) ∥ x − y ∥ p 2 ⋅ ( ∥ x ∥ p + 2 ∥ y ∥ p ) p − 2 ∫ 0 1 t d t = p − 1 2 ( ∥ x ∥ p + 2 ∥ y ∥ p ) p − 2 ⋅ ∥ x − y ∥ p 2 \begin{aligned}
D_\Phi(x,y)=\int_0^1\langle\nabla\Phi(y+t(x-y))-\nabla\Phi(y),x-y\rangle dt \\
&\le\int_0^1\|\nabla\Phi(y+t(x-y))-\nabla\Phi(y)\|_q\cdot\|x-y\|_pdt \\
&\le\int_0^1(p-1)\cdot(\|y+t(x-y)\|_p+\|y\|_p)^{p-2}\cdot\|t(x-y)\|_p\cdot\|x-y\|_pdt \\
&=(p-1)\|x-y\|_p^2\int_0^1t(\|y+t(x-y)\|_p+\|y\|_p)^{p-2}dt \\
&\le(p-1)\|x-y\|_p^2\cdot(\|x\|_p+2\|y\|_p)^{p-2}\int_0^1t\,dt \\
&=\frac{p-1}{2}(\|x\|_p+2\|y\|_p)^{p-2}\cdot\|x-y\|_p^2
\end{aligned} D Φ ( x , y ) = ∫ 0 1 ⟨ ∇Φ ( y + t ( x − y )) − ∇Φ ( y ) , x − y ⟩ d t ≤ ∫ 0 1 ∥∇Φ ( y + t ( x − y )) − ∇Φ ( y ) ∥ q ⋅ ∥ x − y ∥ p d t ≤ ∫ 0 1 ( p − 1 ) ⋅ ( ∥ y + t ( x − y ) ∥ p + ∥ y ∥ p ) p − 2 ⋅ ∥ t ( x − y ) ∥ p ⋅ ∥ x − y ∥ p d t = ( p − 1 ) ∥ x − y ∥ p 2 ∫ 0 1 t ( ∥ y + t ( x − y ) ∥ p + ∥ y ∥ p ) p − 2 d t ≤ ( p − 1 ) ∥ x − y ∥ p 2 ⋅ ( ∥ x ∥ p + 2∥ y ∥ p ) p − 2 ∫ 0 1 t d t = 2 p − 1 ( ∥ x ∥ p + 2∥ y ∥ p ) p − 2 ⋅ ∥ x − y ∥ p 2
which finishes the proof.
Now, we present a few useful facts regarding our Generalized Gaussian (G G p \mathcal{GG}_p GG p ) distribution.
Fact 4. Suppose that f : R d → R f:\mathbb{R}^d\to\mathbb{R} f : R d → R is convex and L L L -Lipschitz with respect to ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p . Then its generalized Gaussian convolution f ^ ρ , p \hat{f}_{\rho,p} f ^ ρ , p (Definition 1) satisfies the following:
f ^ ρ , p \hat{f}_{\rho,p} f ^ ρ , p is ( C p ′ L / ρ ) (C'_pL/\rho) ( C p ′ L / ρ ) -smooth with respect to ∥ ⋅ ∥ p \|\cdot\|_p ∥ ⋅ ∥ p , for some constant C p ′ > 0 C'_p>0 C p ′ > 0 depending only on p p p .
For every x ∈ R d x\in\mathbb{R}^d x ∈ R d ,
f ( x ) ≤ f ^ ρ , p ( x ) ≤ f ( x ) + L ρ d . f(x)\leq\hat{f}_{\rho,p}(x)\leq f(x)+L\rho\sqrt{d}. f ( x ) ≤ f ^ ρ , p ( x ) ≤ f ( x ) + L ρ d . The dependence on d d d in this bound is asymptotically tight.
Proof. First consider x , x ′ ∈ R d x,x'\in\mathbb{R}^d x , x ′ ∈ R d with ∥ x − x ′ ∥ p ≤ ρ / 2 \|x-x'\|_p\leq\rho/2 ∥ x − x ′ ∥ p ≤ ρ /2 . Use the same center x ˉ = x \bar{x}=x x ˉ = x in both estimators from Definition 2. By the unbiasedness assertion in Lemma 1, for ξ ∼ G G p ( 0 , ρ 2 ) \xi\sim\mathcal{GG}_p(0,\rho^2) ξ ∼ GG p ( 0 , ρ 2 ) ,
∇ f ^ ρ , p ( x ) − ∇ f ^ ρ , p ( x ′ ) = E ξ [ γ ρ , p ( − ξ ) − γ ρ , p ( x ′ − x − ξ ) γ ρ , p ( ξ ) g ( x + ξ ) ] . \nabla\hat{f}_{\rho,p}(x)-\nabla\hat{f}_{\rho,p}(x')=\mathbb{E}_{\xi}\left[\frac{\gamma_{\rho,p}(-\xi)-\gamma_{\rho,p}(x'-x-\xi)}{\gamma_{\rho,p}(\xi)}g(x+\xi)\right]. ∇ f ^ ρ , p ( x ) − ∇ f ^ ρ , p ( x ′ ) = E ξ [ γ ρ , p ( ξ ) γ ρ , p ( − ξ ) − γ ρ , p ( x ′ − x − ξ ) g ( x + ξ ) ] .
The triangle inequality, the bound ∥ g ( x + ξ ) ∥ q ≤ L \|g(x+\xi)\|_q\leq L ∥ g ( x + ξ ) ∥ q ≤ L , and Cauchy–Schwarz give
∥ ∇ f ^ ρ , p ( x ) − ∇ f ^ ρ , p ( x ′ ) ∥ q ≤ L E ξ ∣ γ ρ , p ( − ξ ) − γ ρ , p ( x ′ − x − ξ ) γ ρ , p ( ξ ) ∣ ≤ L ( E ξ ∣ γ ρ , p ( − ξ ) − γ ρ , p ( x ′ − x − ξ ) γ ρ , p ( ξ ) ∣ 2 ) 1 / 2 ≤ 2 e ( p − 1 ) / 2 C p L ρ ∥ x − x ′ ∥ p . \begin{aligned}
\left\|\nabla\hat{f}_{\rho,p}(x)-\nabla\hat{f}_{\rho,p}(x')\right\|_q\leq L\mathbb{E}_{\xi}\left|\frac{\gamma_{\rho,p}(-\xi)-\gamma_{\rho,p}(x'-x-\xi)}{\gamma_{\rho,p}(\xi)}\right| \\
&\leq L\left(\mathbb{E}_{\xi}\left|\frac{\gamma_{\rho,p}(-\xi)-\gamma_{\rho,p}(x'-x-\xi)}{\gamma_{\rho,p}(\xi)}\right|^2\right)^{1/2} \\
&\leq\frac{\sqrt{2}e^{(p-1)/2}C_pL}{\rho}\|x-x'\|_p.
\end{aligned} ∇ f ^ ρ , p ( x ) − ∇ f ^ ρ , p ( x ′ ) q ≤ L E ξ γ ρ , p ( ξ ) γ ρ , p ( − ξ ) − γ ρ , p ( x ′ − x − ξ ) ≤ L ( E ξ γ ρ , p ( ξ ) γ ρ , p ( − ξ ) − γ ρ , p ( x ′ − x − ξ ) 2 ) 1/2 ≤ ρ 2 e ( p − 1 ) /2 C p L ∥ x − x ′ ∥ p .
The last inequality follows from the second bound in Lemma 2 with m = 2 m=2 m = 2 and x ˉ = x \bar{x}=x x ˉ = x . For arbitrary x , x ′ ∈ R d x,x'\in\mathbb{R}^d x , x ′ ∈ R d , choose an integer N ≥ 1 N\geq1 N ≥ 1 such that ∥ x − x ′ ∥ p / N ≤ ρ / 2 \|x-x'\|_p/N\leq\rho/2 ∥ x − x ′ ∥ p / N ≤ ρ /2 , and define
x i = x + i N ( x ′ − x ) , i = 0 , … , N . x_i=x+\frac{i}{N}(x'-x),\qquad i=0,\ldots,N. x i = x + N i ( x ′ − x ) , i = 0 , … , N .
Then x 0 = x x_0=x x 0 = x , x N = x ′ x_N=x' x N = x ′ , and ∥ x i − x i − 1 ∥ p = ∥ x − x ′ ∥ p / N ≤ ρ / 2 \|x_i-x_{i-1}\|_p=\|x-x'\|_p/N\leq\rho/2 ∥ x i − x i − 1 ∥ p = ∥ x − x ′ ∥ p / N ≤ ρ /2 . Applying the local bound to each consecutive pair yields
∥ ∇ f ^ ρ , p ( x ) − ∇ f ^ ρ , p ( x ′ ) ∥ q ≤ ∑ i = 1 N ∥ ∇ f ^ ρ , p ( x i ) − ∇ f ^ ρ , p ( x i − 1 ) ∥ q ≤ 2 e ( p − 1 ) / 2 C p L ρ ∑ i = 1 N ∥ x i − x i − 1 ∥ p = 2 e ( p − 1 ) / 2 C p L ρ ∥ x − x ′ ∥ p . \begin{aligned}
\left\|\nabla\hat{f}_{\rho,p}(x)-\nabla\hat{f}_{\rho,p}(x')\right\|_q\leq\sum_{i=1}^{N}\left\|\nabla\hat{f}_{\rho,p}(x_i)-\nabla\hat{f}_{\rho,p}(x_{i-1})\right\|_q \\
&\leq\frac{\sqrt{2}e^{(p-1)/2}C_pL}{\rho}\sum_{i=1}^{N}\|x_i-x_{i-1}\|_p \\
&=\frac{\sqrt{2}e^{(p-1)/2}C_pL}{\rho}\|x-x'\|_p.
\end{aligned} ∇ f ^ ρ , p ( x ) − ∇ f ^ ρ , p ( x ′ ) q ≤ i = 1 ∑ N ∇ f ^ ρ , p ( x i ) − ∇ f ^ ρ , p ( x i − 1 ) q ≤ ρ 2 e ( p − 1 ) /2 C p L i = 1 ∑ N ∥ x i − x i − 1 ∥ p = ρ 2 e ( p − 1 ) /2 C p L ∥ x − x ′ ∥ p .
Thus the first assertion holds with C p ′ = 2 e ( p − 1 ) / 2 C p C'_p=\sqrt{2}e^{(p-1)/2}C_p C p ′ = 2 e ( p − 1 ) /2 C p .
For the second property, we use the randomized smoothing argument in [17 ] . Let ξ ∼ G G p ( 0 , ρ 2 ) \xi\sim\mathcal{GG}_p(0,\rho^2) ξ ∼ GG p ( 0 , ρ 2 ) . By symmetry, E [ ξ ] = 0 \mathbb{E}[\xi]=0 E [ ξ ] = 0 , so convexity and Jensen’s inequality imply
f ( x ) = f ( x + E [ ξ ] ) ≤ E [ f ( x + ξ ) ] = f ^ ρ , p ( x ) . f(x)=f\left(x+\mathbb{E}[\xi]\right)\leq\mathbb{E}[f(x+\xi)]=\hat{f}_{\rho,p}(x). f ( x ) = f ( x + E [ ξ ] ) ≤ E [ f ( x + ξ )] = f ^ ρ , p ( x ) .
On the other hand, Lipschitz continuity and Cauchy–Schwarz give
f ^ ρ , p ( x ) ≤ f ( x ) + L E ∥ ξ ∥ p ≤ f ( x ) + L ( E ∥ ξ ∥ p 2 ) 1 / 2 \hat{f}_{\rho,p}(x) \le f(x) + L\mathbb{E}\lVert\xi\rVert_p \le f(x) + L\left(\mathbb{E}\lVert\xi\rVert_p^2\right)^{1/2} f ^ ρ , p ( x ) ≤ f ( x ) + L E ∥ ξ ∥ p ≤ f ( x ) + L ( E ∥ ξ ∥ p 2 ) 1/2
Since γ ρ , p ( z ) \gamma_{\rho,p}(z) γ ρ , p ( z ) is proportional to exp ( − ∥ z ∥ p 2 / ( 2 ρ 2 ) ) \exp(-\lVert z\rVert_p^2/(2\rho^2)) exp ( − ∥ z ∥ p 2 / ( 2 ρ 2 )) , radial integration gives
E ∥ ξ ∥ p 2 = ∫ 0 ∞ r d + 1 exp ( − r 2 2 ρ 2 ) d r ∫ 0 ∞ r d − 1 exp ( − r 2 2 ρ 2 ) d r = 2 ρ 2 Γ ( d / 2 + 1 ) Γ ( d / 2 ) = d ρ 2 . \mathbb{E}\lVert\xi\rVert_p^2 = \frac{\int_0^\infty r^{d+1}\exp\left(-\frac{r^2}{2\rho^2}\right)\,dr}{\int_0^\infty r^{d-1}\exp\left(-\frac{r^2}{2\rho^2}\right)\,dr} = 2\rho^2\frac{\Gamma(d/2+1)}{\Gamma(d/2)} = d\rho^2. E ∥ ξ ∥ p 2 = ∫ 0 ∞ r d − 1 exp ( − 2 ρ 2 r 2 ) d r ∫ 0 ∞ r d + 1 exp ( − 2 ρ 2 r 2 ) d r = 2 ρ 2 Γ ( d /2 ) Γ ( d /2 + 1 ) = d ρ 2 .
Consequently, f ^ ρ , p ( x ) ≤ f ( x ) + L ρ d \hat{f}_{\rho,p}(x) \le f(x) + L\rho\sqrt{d} f ^ ρ , p ( x ) ≤ f ( x ) + L ρ d . To see this bound is tight, we take f ( x ) = L ∥ x ∥ p f(x)=L\lVert x\rVert_p f ( x ) = L ∥ x ∥ p . Notice that this yields
f ^ ρ , p ( 0 ) − f ( 0 ) = L E ∥ ξ ∥ p = 2 L ρ Γ ( ( d + 1 ) / 2 ) Γ ( d / 2 ) ∼ L ρ d as d → ∞ . \hat{f}_{\rho,p}(0)-f(0)=L\mathbb{E}\lVert\xi\rVert_p=\sqrt{2}L\rho\frac{\Gamma((d+1)/2)}{\Gamma(d/2)}\sim L\rho\sqrt{d}\qquad\text{as }d\to\infty. f ^ ρ , p ( 0 ) − f ( 0 ) = L E ∥ ξ ∥ p = 2 L ρ Γ ( d /2 ) Γ (( d + 1 ) /2 ) ∼ L ρ d as d → ∞.
Fact 5. Let p ∈ [ 2 , ∞ ) p\in[2,\infty) p ∈ [ 2 , ∞ ) and let Φ ( x ) = 1 2 ∥ x ∥ p 2 \Phi(x)=\frac{1}{2}\lVert x\rVert_p^2 Φ ( x ) = 2 1 ∥ x ∥ p 2 . Furthermore, let γ ρ , p ( x ) = Z ρ , p − 1 exp ( − Φ ( x ) ρ 2 ) \gamma_{\rho,p}(x)=Z_{\rho,p}^{-1}\exp\left(-\frac{\Phi(x)}{\rho^2}\right) γ ρ , p ( x ) = Z ρ , p − 1 exp ( − ρ 2 Φ ( x ) ) where Z ρ , p − 1 Z_{\rho,p}^{-1} Z ρ , p − 1 is the appropriate normalization constant. Then, for every u ∈ R d u\in\mathbb{R}^d u ∈ R d ,
E ξ ∼ γ ρ , p [ exp ( ⟨ ∇ Φ ( ξ ) , u ⟩ ρ 2 ) ] ≤ exp ( p − 1 2 ρ 2 ∥ u ∥ p 2 ) \mathbb{E}_{\xi\sim\gamma_{\rho,p}}\left[\exp\left(\frac{\langle\nabla\Phi(\xi),u\rangle}{\rho^2}\right)\right]\le\exp\left(\frac{p-1}{2\rho^2}\lVert u\rVert_p^2\right) E ξ ∼ γ ρ , p [ exp ( ρ 2 ⟨ ∇Φ ( ξ ) , u ⟩ ) ] ≤ exp ( 2 ρ 2 p − 1 ∥ u ∥ p 2 )
Proof. Since Φ ( x ) = 1 2 ∥ x ∥ p 2 \Phi(x)=\frac{1}{2}\lVert x\rVert_p^2 Φ ( x ) = 2 1 ∥ x ∥ p 2 is ( p − 1 ) (p-1) ( p − 1 ) -smooth with respect to ∥ ⋅ ∥ p \lVert\cdot\rVert_p ∥ ⋅ ∥ p , for every x , h ∈ R d x,h\in\mathbb{R}^d x , h ∈ R d ,
Φ ( x + h ) ≤ Φ ( x ) + ⟨ ∇ Φ ( x ) , h ⟩ + p − 1 2 ∥ h ∥ p 2 \Phi(x+h)\le\Phi(x)+\langle\nabla\Phi(x),h\rangle+\frac{p-1}{2}\lVert h\rVert_p^2 Φ ( x + h ) ≤ Φ ( x ) + ⟨ ∇Φ ( x ) , h ⟩ + 2 p − 1 ∥ h ∥ p 2
Taking x = ξ x=\xi x = ξ and h = − u h=-u h = − u gives
Φ ( ξ − u ) ≤ Φ ( ξ ) − ⟨ ∇ Φ ( ξ ) , u ⟩ + p − 1 2 ∥ u ∥ p 2 , \Phi(\xi-u)\le\Phi(\xi)-\langle\nabla\Phi(\xi),u\rangle+\frac{p-1}{2}\lVert u\rVert_p^2, Φ ( ξ − u ) ≤ Φ ( ξ ) − ⟨ ∇Φ ( ξ ) , u ⟩ + 2 p − 1 ∥ u ∥ p 2 ,
and therefore
⟨ ∇ Φ ( ξ ) , u ⟩ ≤ Φ ( ξ ) − Φ ( ξ − u ) + p − 1 2 ∥ u ∥ p 2 \langle\nabla\Phi(\xi),u\rangle\le\Phi(\xi)-\Phi(\xi-u)+\frac{p-1}{2}\lVert u\rVert_p^2 ⟨ ∇Φ ( ξ ) , u ⟩ ≤ Φ ( ξ ) − Φ ( ξ − u ) + 2 p − 1 ∥ u ∥ p 2
Hence
E ξ ∼ γ ρ , p [ exp ( ⟨ ∇ Φ ( ξ ) , u ⟩ ρ 2 ) ] = 1 Z ρ , p ∫ R d exp ( ⟨ ∇ Φ ( ξ ) , u ⟩ − Φ ( ξ ) ρ 2 ) d ξ ≤ exp ( p − 1 2 ρ 2 ∥ u ∥ p 2 ) 1 Z ρ , p ∫ R d exp ( − Φ ( ξ − u ) ρ 2 ) d ξ . \begin{aligned}
\mathbb{E}_{\xi\sim\gamma_{\rho,p}}\left[\exp\left(\frac{\langle\nabla\Phi(\xi),u\rangle}{\rho^2}\right)\right]=\frac{1}{Z_{\rho,p}}\int_{\mathbb{R}^d}\exp\left(\frac{\langle\nabla\Phi(\xi),u\rangle-\Phi(\xi)}{\rho^2}\right)\,d\xi\\
&\le\exp\left(\frac{p-1}{2\rho^2}\lVert u\rVert_p^2\right)\frac{1}{Z_{\rho,p}}\int_{\mathbb{R}^d}\exp\left(-\frac{\Phi(\xi-u)}{\rho^2}\right)\,d\xi.
\end{aligned} E ξ ∼ γ ρ , p [ exp ( ρ 2 ⟨ ∇Φ ( ξ ) , u ⟩ ) ] = Z ρ , p 1 ∫ R d exp ( ρ 2 ⟨ ∇Φ ( ξ ) , u ⟩ − Φ ( ξ ) ) d ξ ≤ exp ( 2 ρ 2 p − 1 ∥ u ∥ p 2 ) Z ρ , p 1 ∫ R d exp ( − ρ 2 Φ ( ξ − u ) ) d ξ .
By translation invariance of Lebesgue measure,
∫ R d exp ( − Φ ( ξ − u ) ρ 2 ) d ξ = ∫ R d exp ( − Φ ( ξ ) ρ 2 ) d ξ = Z ρ , p \int_{\mathbb{R}^d}\exp\left(-\frac{\Phi(\xi-u)}{\rho^2}\right)\,d\xi=\int_{\mathbb{R}^d}\exp\left(-\frac{\Phi(\xi)}{\rho^2}\right)\,d\xi=Z_{\rho,p} ∫ R d exp ( − ρ 2 Φ ( ξ − u ) ) d ξ = ∫ R d exp ( − ρ 2 Φ ( ξ ) ) d ξ = Z ρ , p
Therefore,
E ξ ∼ γ ρ , p [ exp ( ⟨ ∇ Φ ( ξ ) , u ⟩ ρ 2 ) ] ≤ exp ( p − 1 2 ρ 2 ∥ u ∥ p 2 ) \mathbb{E}_{\xi\sim\gamma_{\rho,p}}\left[\exp\left(\frac{\langle\nabla\Phi(\xi),u\rangle}{\rho^2}\right)\right]\le\exp\left(\frac{p-1}{2\rho^2}\lVert u\rVert_p^2\right) E ξ ∼ γ ρ , p [ exp ( ρ 2 ⟨ ∇Φ ( ξ ) , u ⟩ ) ] ≤ exp ( 2 ρ 2 p − 1 ∥ u ∥ p 2 )
Algorithm 11 Aggregate
Input: S = { y i } i ∈ [ k ] ⊂ R d S = \{y_i\}_{i \in[k]} \subset\mathbb{R}^d S = { y i } i ∈ [ k ] ⊂ R d , R ≥ 0 R \ge0 R ≥ 0 such that for some z ∈ R d z \in\mathbb{R}^d z ∈ R d at least 0.51 k 0.51k 0.51 k points y i ∈ S y_i \in S y i ∈ S have ∥ y i − z ∥ p ≤ R / 3 \lVert y_i-z\rVert_p \le R/3 ∥ y i − z ∥ p ≤ R /3
for i ∈ [ k ] i \in[k] i ∈ [ k ] do
if at least 0.51 k 0.51k 0.51 k points y j ∈ S y_j \in S y j ∈ S satisfy ∥ y i − y j ∥ p ≤ 2 R / 3 \lVert y_i-y_j\rVert_p \le2R/3 ∥ y i − y j ∥ p ≤ 2 R /3 then return z ~ ← y i \tilde z \leftarrow y_i z ~ ← y i
end if
end for
Fact 6. Let S = ( y 1 , … , y k ) S=(y_1,\ldots,y_k) S = ( y 1 , … , y k ) with y i ∈ R d y_i \in\mathbb{R}^d y i ∈ R d , and let R > 0 R>0 R > 0 . Suppose that there exists z ∈ R d z \in\mathbb{R}^d z ∈ R d such that
∣ { i ∈ [ k ] : ∥ y i − z ∥ p ≤ R / 3 } ∣ ≥ 0.51 k . \left|\{i \in[k] : \lVert y_i-z\rVert_p \le R/3\}\right| \ge0.51k. ∣ { i ∈ [ k ] : ∥ y i − z ∥ p ≤ R /3 } ∣ ≥ 0.51 k . Then Aggregate( S , R ) (S,R) ( S , R ) runs in O ( k 2 d ) O(k^2d) O ( k 2 d ) time and returns z ~ ∈ S \tilde z \in S z ~ ∈ S satisfying ∥ z ~ − z ∥ p ≤ R \lVert\tilde z-z\rVert_p \le R ∥ z ~ − z ∥ p ≤ R .
Proof. Following [28 ] , let
T = { i ∈ [ k ] : ∥ y i − z ∥ p ≤ R / 3 } . T=\{i \in[k] : \lVert y_i-z\rVert_p \le R/3\}. T = { i ∈ [ k ] : ∥ y i − z ∥ p ≤ R /3 } .
For every i , j ∈ T i,j \in T i , j ∈ T , the triangle inequality gives ∥ y i − y j ∥ p ≤ 2 R / 3 \lVert y_i-y_j\rVert_p \le2R/3 ∥ y i − y j ∥ p ≤ 2 R /3 . Thus every i ∈ T i \in T i ∈ T has at least 0.51 k 0.51k 0.51 k points within distance 2 R / 3 2R/3 2 R /3 , so the algorithm has a valid point to return.
Suppose that the algorithm returns z ~ = y i \tilde z=y_i z ~ = y i , and let
Q = { j ∈ [ k ] : ∥ y i − y j ∥ p ≤ 2 R / 3 } . Q=\{j \in[k] : \lVert y_i-y_j\rVert_p \le2R/3\}. Q = { j ∈ [ k ] : ∥ y i − y j ∥ p ≤ 2 R /3 } .
By the selection rule, ∣ Q ∣ ≥ 0.51 k |Q| \ge0.51k ∣ Q ∣ ≥ 0.51 k . Since ∣ T ∣ ≥ 0.51 k |T| \ge0.51k ∣ T ∣ ≥ 0.51 k ,
∣ Q ∩ T ∣ ≥ ∣ Q ∣ + ∣ T ∣ − k ≥ 0.02 k > 0. |Q \cap T| \ge|Q|+|T|-k \ge0.02k>0. ∣ Q ∩ T ∣ ≥ ∣ Q ∣ + ∣ T ∣ − k ≥ 0.02 k > 0.
Choose j ⋆ ∈ Q ∩ T j^\star\in Q \cap T j ⋆ ∈ Q ∩ T . Then
∥ z ~ − z ∥ p ≤ ∥ y i − y j ⋆ ∥ p + ∥ y j ⋆ − z ∥ p ≤ 2 R 3 + R 3 = R . \lVert\tilde z-z\rVert_p \le\lVert y_i-y_{j^\star}\rVert_p+\lVert y_{j^\star}-z\rVert_p \le\frac{2R}{3}+\frac{R}{3}=R. ∥ z ~ − z ∥ p ≤ ∥ y i − y j ⋆ ∥ p + ∥ y j ⋆ − z ∥ p ≤ 3 2 R + 3 R = R .
The running time is O ( k 2 d ) O(k^2d) O ( k 2 d ) because the algorithm performs at most O ( k 2 ) O(k^2) O ( k 2 ) pairwise distance computations, each costing O ( d ) O(d) O ( d ) time.