Introduction
In zeroth-order optimization, an algorithm optimizes an objective function with access only to evaluations of its value. Such methods are the tool of choice whenever gradients are unavailable or too costly to compute, as when attacking or explaining a model only through its outputs [13 ] , tuning simulations, hyperparameters, and engineering designs [33 ] , or searching over policies in reinforcement learning [46 ] ; see Liu et al. [37 ] for a survey. Large language models have brought renewed interest to this setting. Because fine-tuning with forward passes alone requires about the same memory as inference, zeroth-order methods allow a single GPU to tune models that backpropagation cannot fit in memory [38 ] (Zhang et al. 2024), while seed-based zeroth-order updates reduce communication in federated fine-tuning [44 ] .
For an L L L -smooth nonconvex function f : R d → R f : \mathbb{R}^{d} \to\mathbb{R} f : R d → R with initial gap f ( 0 ) − inf f ≤ Δ f(0)-\inf f \le\Delta f ( 0 ) − inf f ≤ Δ , random finite differences find an ε \varepsilon ε -stationary point, a point x ^ \hat{x} x ^ with ∥ ∇ f ( x ^ ) ∥ ≤ ε \|\nabla f(\hat{x})\| \le\varepsilon ∥∇ f ( x ^ ) ∥ ≤ ε , and the classical analysis bounds their cost by O ( d T ) O(dT) O ( d T ) function values, where T = L Δ / ε 2 T = L\Delta/\varepsilon^{2} T = L Δ/ ε 2 (Nesterov and Spokoiny, 2017; Ghadimi
and Lan (2013). In high dimension, this rate cannot be improved on the full smooth class even with exact function values. Concurrent and independent work of Xia and Yuan [55 ] proves that randomized algorithms, and hence deterministic ones as well, require at least Ω ( d T ) \Omega(dT) Ω ( d T ) exact value calls when d ≳ T log T d\gtrsim T\log T d ≳ T log T . For deterministic algorithms, we prove the explicit lower bound d T / 500 dT/500 d T /500 for 1 ≤ T ≤ d / log ( e d ) 1\le T\le d/\log(ed) 1 ≤ T ≤ d / log ( e d ) , which matches coordinate finite differences up to constants (Theorem 3.4 ). In high-dimensional tasks such as language model fine-tuning, however, randomized zeroth-order methods succeed at far lower query cost. Effective rank analyses attribute this speedup to a low curvature trace ([38 , 57 ] ), consistent with the low intrinsic dimension of fine-tuning ([35 , 1 ] ). Does this saving come from problem structure alone, or does it require randomness?
This question also applies to practical implementations of randomized methods. Memory-efficient zeroth-order fine-tuning regenerates each perturbation from a stored random seed ([38 ] ). Once its seeds are fixed in advance, such a procedure is a deterministic algorithm, so deterministic lower bounds apply to it on functions chosen after the seeds are set; we defeat such a fixed-seed run experimentally in § 6 . Classical derivative-free optimizers, including coordinate and pattern search ([24 ] ), the Nelder–Mead method ([39 ] ), and Powell’s interpolation methods ([15 , 33 ] ), are deterministic as well. This raises the question of whether a deterministic algorithm can learn the active subspace of curvature along its trajectory and search within it, paying only for the curvature trace rather than for the ambient dimension d d d . Whereas Jordan et al. [28 ] and Kornowski [31 ] establish a price of determinism for nonsmooth Goldstein stationarity (§ 5 ), we study smooth first-order stationarity in high dimension and show that randomness is essential to fully exploit a low curvature trace.
To compare deterministic and randomized methods on the same functions, we consider an L L L -smooth objective f f f with initial gap Δ \Delta Δ whose positive curvature has bounded trace on the initial sublevel set { x ∈ R d : f ( x ) < f ( 0 ) } \{x \in\mathbb{R}^{d}: f(x) < f(0)\} { x ∈ R d : f ( x ) < f ( 0 )} , where descent methods operate. For almost every x x x with f ( x ) < f ( 0 ) f(x) < f(0) f ( x ) < f ( 0 ) , we assume that the Hessian ∇ 2 f ( x ) \nabla^{2}f(x) ∇ 2 f ( x ) is dominated by one fixed positive semidefinite matrix of operator norm at most L L L and trace at most L E LE L E . Because E E E bounds only the trace of this dominating matrix, positive curvature may spread across many directions. Since the matrix bounds the Hessian only from above, negative curvature is not charged at all. § 2.1 compares this class with prior curvature assumptions. Malladi et al. [38 ] assume such a dominating matrix near each iterate of a trajectory and derive an effective-rank upper bound for zeroth-order SGD, whereas our condition is fixed on the initial sublevel set before the run begins.
Suppose 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , T ≥ 1 T \ge1 T ≥ 1 , and E T ≤ d ET \le d E T ≤ d , and let N d e t N_{\mathrm{det}} N det and N r a n d N_{\mathrm{rand}} N rand denote the deterministic and randomized worst-case query complexities on this class (Section 2). Randomized methods pay only for the curvature trace. We show that the Gaussian two-point method attains O ( E T ) O(ET) O ( E T ) calls when it accepts only steps that decrease f f f , and that every randomized method requires at least Ω ( E T ) \Omega(ET) Ω ( E T ) calls. A deterministic method, by contrast, pays roughly d d d calls for each active direction it discovers, while the trace budget allows at most O ( E T ) O(\sqrt{ET}) O ( E T ) such directions. We show that adaptive subspace search attains O ( d min { T , E T } ) O(d\min\{T,\sqrt{ET}\}) O ( d min { T , E T }) calls, while our hidden-chain construction forces Ω ~ ( min { T , E T } ) \widetilde{\Omega}(\min\{T,\sqrt{ET}\}) Ω ( min { T , E T }) directions. We prove (Theorems 3.1 and 3.2 )
Ω ~ ( d min { T , E T } ) ≤ N d e t ≤ O ( d min { T , E T } ) , Ω ( E T ) ≤ N r a n d ≤ O ( E T ) , \begin{aligned}
\widetilde{\Omega}(d\min\{T,\sqrt{ET}\}) &\le N_{\mathrm{det}} \le O(d\min\{T,\sqrt{ET}\}),\\
\Omega(ET) &\le N_{\mathrm{rand}} \le O(ET),
\end{aligned} Ω ( d min { T , E T }) Ω ( E T ) ≤ N det ≤ O ( d min { T , E T }) , ≤ N rand ≤ O ( E T ) ,
where Ω ~ \widetilde{\Omega} Ω hides only a factor log ( e + T ) \sqrt{\log(e+T)} log ( e + T ) ; Section 3 states the bounds with this factor written out. For E T ≤ d ET \le d E T ≤ d , the price of determinism N d e t / N r a n d N_{\mathrm{det}}/N_{\mathrm{rand}} N det / N rand is therefore bounded below by Ω ~ ( d / ( E + E T ) ) \widetilde{\Omega}(d/(E+\sqrt{ET})) Ω ( d / ( E + E T )) and above by O ( d / ( E + E T ) ) O(d/(E+\sqrt{ET})) O ( d / ( E + E T )) , growing linearly with d d d when E E E and T T T are fixed. Once E ≥ T log ( e + T ) E \ge T\log(e+T) E ≥ T log ( e + T ) within this range, so that T log ( e + T ) ≤ E ≤ d / T T\log(e+T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T , deterministic methods require at least Ω ~ ( d T ) \widetilde{\Omega}(dT) Ω ( d T ) calls, just as without a trace budget. Our upper and lower bounds then match up to constant factors for both algorithm classes, and their costs differ by a factor of order d / E d/E d / E (Corollary 3.3). Table 1 compares these bounds with the closest prior ones.
Table 1. Query complexity of finding an ε \varepsilon ε -stationary point and the algorithms that attain the upper bounds. D and R denote deterministic and randomized algorithms, NC means nonconvex, and T = L Δ / ε 2 T=L\Delta/\varepsilon^{2} T = L Δ/ ε 2 . All results use exact function values unless the setting names another oracle. Bold entries are new in this paper. “Matched”: whether the two bounds agree up to constant factors (yes) or up to the stated factor; a dash means that only one side is known.
Why can exact real replies still force Ω ( d ) \Omega(d) Ω ( d ) calls per hidden direction? When each reply carries a bounded number of bits, lower bounds follow by counting the revealed information (Basu et al., [7 ] ). An exact real reply, however, can encode arbitrarily much discrete information, so counting returned scalars gives no lower bound. Against a deterministic algorithm, we instead combine the gated chains of Carmon et al. [12 ] with a resisting oracle (Nemirovsky and Yudin, [40 ] ), so that the replies agree exactly with those of a single function in the class along the adaptive transcript. The oracle fixes each hidden direction only when its stage closes, after Ω ( d ) \Omega(d) Ω ( d ) units of cost, and keeps an unfinished direction orthogonal to the output, even if the output was never queried. The resulting function produces every reply of the run, and its gradient at the output has norm above ε \varepsilon ε . Multiplying the chain by a perspective weight rules out small gradients far from the origin and keeps the Hessian dominated by one fixed matrix on all of R d \mathbb{R}^{d} R d (§ 4 ). Against randomized algorithms, the function must be chosen before the seed, and Yao’s principle (Yao, [56 ] ) applied to a quadratic with an unknown center gives Ω ( E T ) \Omega(ET) Ω ( E T ) .
Contributions.
Randomized complexity scales with the trace rather than the dimension. We show that for all 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d and T ≥ 1 T \ge1 T ≥ 1 , the Gaussian two-point method of Nesterov and Spokoiny [41 ] attains O ( E T ) O(ET) O ( E T ) calls when it accepts only steps that decrease f f f . Without this acceptance test, we prove that it attains O ( E T ) O(ET) O ( E T ) calls when the matrix bound holds on all of R d \mathbb{R}^{d} R d and O ( E T + T log ( E T ) ) O(ET + T \log(ET)) O ( E T + T log ( E T )) calls on our sublevel-set class. Conversely, we prove that when E T ≤ d ET \le d E T ≤ d , every randomized algorithm requires at least Ω ( E T ) \Omega(ET) Ω ( E T ) calls (Theorem 3.1 ).
Quantifying the deterministic cost under a low curvature trace. For E T ≤ d ET \le d E T ≤ d , we adapt the subspace search scheme of Cosson et al. [16 ] to find an ε \varepsilon ε -stationary point in O ( d min { T , E T } ) O(d\min\{T,\sqrt{ET}\}) O ( d min { T , E T }) calls, and we construct C ∞ C^\infty C ∞ hard functions satisfying the matrix bound on all of R d \mathbb{R}^{d} R d to prove that every deterministic algorithm requires at least Ω ~ ( d min { T , E T } ) \widetilde{\Omega}(d\min\{T,\sqrt{ET}\}) Ω ( d min { T , E T }) calls (Theorem 3.2 ). Across T log ( e + T ) ≤ E ≤ d / T T\log(e+T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T , we show that deterministic algorithms require at least Ω ( d T ) \Omega(dT) Ω ( d T ) calls, losing any savings from the trace budget, so determinism multiplies the worst-case query cost by a factor of order d / E d/E d / E (Corollary 3.3).
Optimality of coordinate finite differences without a trace budget. On the unrestricted L L L -smooth class, we prove that for 1 ≤ T ≤ d / log ( e d ) 1 \le T \le d/\log(ed) 1 ≤ T ≤ d / log ( e d ) , every deterministic algorithm requires at least d T / 500 dT/500 d T /500 calls, even when each call may also return projections of the gradient, charged by their rank (Theorem 3.4 ). For exact function values alone, the randomized lower bound of Xia and Yuan [55 ] already implies a deterministic Ω ( d T ) \Omega(dT) Ω ( d T ) bound when d ≳ T log T d \gtrsim T\log T d ≳ T log T ; our result adds an explicit constant and covers these gradient projections, which shows that coordinate finite differences are optimal up to constants under both oracles.
Setting
Function classes. For L , Δ > 0 L,\Delta> 0 L , Δ > 0 , we consider the smooth nonconvex class
F L , Δ d : = { f ∈ C 1 , 1 ( R d ) : Lip ( ∇ f ) ≤ L , f ( 0 ) − inf f ≤ Δ } . (1) \mathcal{F}_{L,\Delta}^{d} := \{f \in C^{1,1}(\mathbb{R}^{d}) : \operatorname{Lip}(\nabla f) \le L,\quad f(0) - \inf f \le\Delta\}. \tag*{(1)} F L , Δ d := { f ∈ C 1 , 1 ( R d ) : Lip ( ∇ f ) ≤ L , f ( 0 ) − inf f ≤ Δ } . (1)
The goal is a point x ^ ∈ R d \widehat{x} \in\mathbb{R}^{d} x ∈ R d with ∥ ∇ f ( x ^ ) ∥ ≤ ε \|\nabla f(\widehat{x})\| \le\varepsilon ∥∇ f ( x ) ∥ ≤ ε , and we write T : = L Δ / ε 2 T := L\Delta/\varepsilon^{2} T := L Δ/ ε 2 . For 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , the main class B L , Δ , E d \mathcal{B}_{L,\Delta,E}^{d} B L , Δ , E d consists of the functions f ∈ F L , Δ d f \in\mathcal{F}_{L,\Delta}^{d} f ∈ F L , Δ d for which some matrix H ⪰ 0 H \succeq0 H ⪰ 0 , unknown to the algorithm, with ∥ H ∥ o p ≤ L \|H\|_{\mathrm{op}} \le L ∥ H ∥ op ≤ L and tr H ≤ L E \operatorname{tr} H \le LE tr H ≤ L E satisfies
∇ 2 f ( x ) ⪯ H for almost every x with f ( x ) < f ( 0 ) . (2) \nabla^{2}f(x) \preceq H \qquad\text{for almost every }x\text{ with }f(x)<f(0). \tag*{(2)} ∇ 2 f ( x ) ⪯ H for almost every x with f ( x ) < f ( 0 ) . (2)
Matrix bounds of this form, with tr H / ∥ H ∥ o p \operatorname{tr} H/\|H\|_{\mathrm{op}} tr H /∥ H ∥ op called an effective rank, are used to explain why zeroth-order fine-tuning is cheap (Malladi et al. [38 ] ). We impose (2) only on the initial sublevel set U = { x : f ( x ) < f ( 0 ) } U = \{x : f(x) < f(0)\} U = { x : f ( x ) < f ( 0 )} , because this is where descent methods operate. Above f ( 0 ) f(0) f ( 0 ) the class asks only for L L L -smoothness, and since H H H bounds the Hessian only from above, negative curvature is not charged, so functions in the class may be strongly nonconvex.
The trace of H H H bounds the total positive curvature by L E LE L E , so E E E plays the role of a dimension: E = d E = d E = d adds nothing to L L L -smoothness, whereas a small E E E forces the curvature to concentrate in a few directions or to spread thinly over many. A single matrix for the whole sublevel set is a structure that a deterministic method can learn and reuse (§ 4 ), whereas randomized methods need only a bound at each point.
Our lower bounds hold on the smaller global fixed-matrix class F L , Δ , E d ⊆ B L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} \subseteq\mathcal{B}^{d}_{L,\Delta,E} F L , Δ , E d ⊆ B L , Δ , E d (Lemma G.1), which requires ∇ 2 f ( x ) ⪯ H \nabla^{2}f(x) \preceq H ∇ 2 f ( x ) ⪯ H for almost every x ∈ R d x \in\mathbb{R}^{d} x ∈ R d . For C 1 , 1 C^{1,1} C 1 , 1 functions, this is the quadratic upper model
f ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 1 2 v T H v ( x , v ∈ R d ) , (3) f(x+v) \leq f(x) + \langle\nabla f(x),v\rangle+ \frac{1}{2}v^{\mathsf{T}}Hv \qquad(x,v \in\mathbb{R}^{d}), \tag*{(3)} f ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 2 1 v T H v ( x , v ∈ R d ) , (3)
and we call H H H an upper model of f f f . For E = d E=d E = d , every function in F L , Δ d \mathcal{F}^{d}_{L,\Delta} F L , Δ d belongs to F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d . Our randomized upper bound, on the other hand, needs neither a common matrix nor smoothness above f ( 0 ) f(0) f ( 0 ) : it holds on the local pointwise class of Appendix § 15 .1, which asks for a locally L L L -Lipschitz gradient on U U U and for the positive eigenvalues of ∇ 2 f ( x ) \nabla^{2}f(x) ∇ 2 f ( x ) to sum to at most L E LE L E at almost every point of U U U .
Oracles and costs. At each call t t t , the algorithm chooses a point x t ∈ R d x_t \in\mathbb{R}^{d} x t ∈ R d based on the earlier replies, and the oracle returns the exact value f ( x t ) f(x_t) f ( x t ) . The output x ^ \hat{x} x ^ need not be one of the queried points. The deterministic complexity N d e t N_{\mathrm{det}} N det is the smallest worst-case number of calls of a deterministic algorithm that succeeds on every function in the class. A randomized algorithm must succeed with probability at least 2 / 3 2/3 2/3 on each function while making at most N N N calls on every run (a cost cap of N N N ), and N r a n d N_{\mathrm{rand}} N rand is the smallest such cap. Algorithms know d d d , L L L , Δ \Delta Δ , ε \varepsilon ε , and E E E ; computation is free and arithmetic is exact, and randomized policies are jointly measurable in their seed and the transcript (Appendix § 14 ).
All our lower bounds also hold for a stronger oracle that additionally returns projections of the gradient and charges for their rank: at call t t t , the algorithm may choose a matrix V t ∈ R d × s t V_t \in\mathbb{R}^{d \times s_t} V t ∈ R d × s t and receive
O f ( x t , V t ) = ( f ( x t ) , V t T ∇ f ( x t ) ) at cost 1 + rank V t . (4) \mathcal{O}_{f}(x_t,V_t) = \left(f(x_t), V_t^{\mathsf{T}}\nabla f(x_t)\right) \quad\text{at cost } 1+\operatorname{rank} V_t. \tag*{(4)} O f ( x t , V t ) = ( f ( x t ) , V t T ∇ f ( x t ) ) at cost 1 + rank V t . (4)
A rank-one call returns, besides f ( x t ) f(x_t) f ( x t ) , a forward-mode directional derivative (Baydin et al. [8 ] ), and V t = I d V_t=I_d V t = I d gives the full gradient. We write N v a l N^{\mathrm{val}} N val and N l i n N^{\mathrm{lin}} N lin when the oracle matters (N l i n ≤ N v a l N^{\mathrm{lin}} \leq N^{\mathrm{val}} N lin ≤ N val ). The lower bounds also hold for the worst-case expected cost.
Comparison with prior curvature assumptions
How does our condition relate to the conditions under which earlier zeroth-order methods improve their dependence on d d d ?
Sparse and latent structure ([52 , 21 ] ). Wang et al. [52 ] assume sparsity, and Golovin et al. [21 ] study objectives with a low-dimensional latent structure. A globally L L L -smooth objective with f ( 0 ) − inf f ≤ Δ f(0)-\inf f \leq\Delta f ( 0 ) − inf f ≤ Δ that depends only on a fixed k k k -dimensional subspace lies in F L , Δ , k d \mathcal{F}^{d}_{L,\Delta,k} F L , Δ , k d , and our trace bound also allows Hessians of full rank.
Local effective rank (Malladi et al. [38 ] ). Malladi et al. [38 ] assume that near each iterate x t x_t x t of a run, ∇ 2 f ⪯ H t \nabla^{2}f \preceq H_t ∇ 2 f ⪯ H t with tr H t ≤ r ∥ H t ∥ o p \operatorname{tr} H_t \leq r\lVert H_t\rVert_{\mathrm{op}} tr H t ≤ r ∥ H t ∥ op . Letting H t H_t H t change along the run is more general than our single H H H , but the condition then depends on the algorithm, whereas a lower bound needs a class that is fixed before the run. Every f ∈ F L , Δ , E d f \in\mathcal{F}^{d}_{L,\Delta,E} f ∈ F L , Δ , E d satisfies their condition with r = E + 1 r=E+1 r = E + 1 : take H t = H + ( L − ∥ H ∥ o p ) v v T H_t=H+(L-\lVert H\rVert_{\mathrm{op}})vv^{\mathsf{T}} H t = H + ( L − ∥ H ∥ op ) v v T for a top unit eigenvector v v v of H H H , so that ∥ H t ∥ o p = L \lVert H_t\rVert_{\mathrm{op}}=L ∥ H t ∥ op = L and tr H t ≤ L ( E + 1 ) \operatorname{tr} H_t \leq L(E+1) tr H t ≤ L ( E + 1 ) . Our lower bounds therefore apply in their setting.
Effective dimension and nuclear norm ([57 , 3 ] ). These conditions sum powers of the absolute Hessian eigenvalues, or bound ∥ ∇ 2 f ∥ ∗ ≤ L ∗ \lVert\nabla^{2}f \rVert_{*} \leq L_{*} ∥ ∇ 2 f ∥ ∗ ≤ L ∗ , at every point of R d \mathbb{R}^{d} R d . Our condition is weaker in that it charges only positive curvature and only below f ( 0 ) f(0) f ( 0 ) , so a strongly nonconvex function can have a small E E E but large values of these measures. It is stronger in that it asks for one common matrix, the structure that deterministic subspace search exploits.
Our randomized upper bound holds on the local pointwise class of Appendix § 15 .1, which for L = L ∗ L = L_{\ast} L = L ∗ and E = 1 E = 1 E = 1 contains the nuclear norm class of Anonymous [3 ] , and our lower bounds hold on functions of F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d with Hessian nuclear norm at most L E LE L E . The separation therefore holds on every class between these hard functions and the local pointwise class, including that nuclear norm class.
Main Results
We now state our bounds. Throughout, d ≥ 1 d \ge1 d ≥ 1 , 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , and T ≥ 1 T \ge1 T ≥ 1 , and c , C > 0 c, C > 0 c , C > 0 denote absolute constants. Unless another class is named, costs refer to the main class B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d , and all bounds hold for both oracles. We prove all lower bounds on the global fixed-matrix class F L , Δ , E d ⊆ B L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} \subseteq\mathcal{B}^{d}_{L,\Delta,E} F L , Δ , E d ⊆ B L , Δ , E d (Appendix § 15 .3), and the randomized upper bound holds even under the pointwise trace bound of Appendix § 15 .1.
Randomized complexity.
Theorem 3.1 (Randomized cost). For all the above parameters , N r a n d ≤ C E T N_{\mathrm{rand}} \le C ET N rand ≤ C E T . If E T ≤ d ET \le d E T ≤ d , then N r a n d ≥ c E T N_{\mathrm{rand}} \ge c ET N rand ≥ c E T .
Random directions replace the dimension d d d by the trace budget E E E . The upper bound is attained by the Gaussian two-point method (Nesterov and Spokoiny [41 ] ), whose estimator is used in zeroth-order fine-tuning (Malladi et al. [38 ] ), with step size 1 / ( L ( E + 2 ) ) 1/(L(E + 2)) 1/ ( L ( E + 2 )) . If the method accepts a step only when f f f decreases, it uses at most 12 E T + 26 T + 3 12ET + 26T + 3 12 E T + 26 T + 3 calls, still with two function values per step (Theorem B.10 ). Without this test, it uses at most 34 ( E + 2 ) T + 2 34(E + 2)T + 2 34 ( E + 2 ) T + 2 calls on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d (Theorem B.12 ). On B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d , a long step may leave the sublevel set, so the method needs a smaller step and uses O ( E T + T log ( E T ) ) O(ET + T\log(ET)) O ( E T + T log ( E T )) calls (Theorem B.15 ). The stochastic three-point method (Bergou et al. [9 ] ) attains O ( E T ) O(ET) O ( E T ) calls under the pointwise trace bound (Theorem G.3). Our lower bound holds for every randomized method, including all these variants, and uses quadratics whose unknown center lies in a subspace of dimension at most min { d , E T } \min\{d, ET\} min { d , E T } that is known to the algorithm.
The price of determinism.
Theorem 3.2 (Deterministic cost). If E T ≤ d ET \le d E T ≤ d , then
c d min { T , E T log ( e + T ) } ≤ N d e t ≤ C d min { T , E T } . cd\min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\} \le N_{\mathrm{det}} \le Cd\min\{T,\sqrt{ET}\}. c d min { T , log ( e + T ) E T } ≤ N det ≤ C d min { T , E T } .
In Ω ~ \widetilde{\Omega} Ω notation, the two bounds are Ω ~ ( d min { T , E T } ) \widetilde{\Omega}(d\min\{T,\sqrt{ET}\}) Ω ( d min { T , E T }) and O ( d min { T , E T } ) O(d\min\{T,\sqrt{ET}\}) O ( d min { T , E T }) , and they agree up to a factor log ( e + T ) \sqrt{\log(e + T)} log ( e + T ) . When E < T E < T E < T , adaptive subspace search beats coordinate finite differences, which use O ( d T ) O(dT) O ( d T ) calls, because it pays about d d d calls for a full gradient estimate only when its current subspace is exhausted. The common matrix H H H of B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d limits the number of these subspace expansions to O ( E T ) O(\sqrt{ET}) O ( E T ) , even though the search receives neither E E E nor H H H as input (§ 4 , Algorithm 1, and Theorem B.1 ). Our lower bound uses C ∞ C^{\infty} C ∞ hard functions whose Hessian nuclear norm is at most L E LE L E on all of R d \mathbb{R}^{d} R d and whose Hessian Lipschitz constant is finite (Appendix § 15 .3).
Corollary 3.3 (A window with matching bounds). If T log ( e + T ) ≤ E ≤ d / T T \log(e + T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T , then
c d T ≤ N d e t ≤ C d T , c\,dT \le N_{\mathrm{det}} \le C\,dT, c d T ≤ N det ≤ C d T , c E T ≤ N r a n d ≤ C E T . c\,ET \le N_{\mathrm{rand}} \le C\,ET. c E T ≤ N rand ≤ C E T . For the same oracle, the cost ratio is bounded below by Ω ( d / E ) \Omega(d/E) Ω ( d / E ) and above by O ( d / E ) O(d/E) O ( d / E ) .
Once E ≥ T log ( e + T ) E \ge T\log(e + T) E ≥ T log ( e + T ) , a threshold set by the accuracy alone, deterministic methods lose their savings from the trace and pay Ω ( d T ) \Omega(dT) Ω ( d T ) . The window of Corollary 3.3 is nonempty when T 2 log ( e + T ) ≤ d T^2\log(e + T) \le d T 2 log ( e + T ) ≤ d ; for d = 10 6 d = 10^6 d = 1 0 6 it allows T T T up to about 400. Over the whole range E T ≤ d ET \le d E T ≤ d , the cost ratio lies between Ω ~ ( d / ( E + E T ) ) \widetilde{\Omega}(d/(E + \sqrt{ET})) Ω ( d / ( E + E T )) and O ( d / ( E + E T ) ) O(d/(E + \sqrt{ET})) O ( d / ( E + E T )) (Corollary E.5 ); this ratio is at least Ω ~ ( d ) \widetilde{\Omega}(\sqrt{d}) Ω ( d ) whenever E ≤ T E \le T E ≤ T . Appendix § 9 gives the constants explicitly, and Theorems 3.1 and 3.2 and Corollary 3.3 are formalized in Lean 4 on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d (Appendix § 18 ).
Without a trace budget.
Theorem 3.4 (Unrestricted smooth class). On F L , Δ d \mathcal{F}^{d}_{L,\Delta} F L , Δ d , if 1 ≤ T ≤ d / log ( e d ) 1 \le T \le d/\log(ed) 1 ≤ T ≤ d / log ( e d ) , then
d T 500 ≤ N d e t ≤ 10 d T . \frac{dT}{500} \le N_{\mathrm{det}} \le10\,dT. 500 d T ≤ N det ≤ 10 d T . For exact values, the randomized lower bound of Xia and Yuan [55 ] already gives Ω ( d T ) \Omega(dT) Ω ( d T ) for deterministic methods when d ≳ T log T d \gtrsim T\log T d ≳ T log T . We add the explicit constant 1 / 500 1/500 1/500 , and our bound also holds when each call may return gradient projections, charged by their rank (Theorem E.8 ); for that oracle a randomized Ω ( d T ) \Omega(dT) Ω ( d T ) bound remains open. Coordinate finite differences are therefore optimal up to constant factors for 1 ≤ T ≤ d / log ( e d ) 1 \le T \le d/\log(ed) 1 ≤ T ≤ d / log ( e d ) under both oracles. The hard functions of Theorem 3.4 also separate deterministic from randomized methods: a randomized batching method minimizes each of them exactly with probability at least 5 / 6 5/6 5/6 using O ( d T log ( e + T ) ) O(d\sqrt{T}\log(e + T)) O ( d T log ( e + T )) function values (Proposition H.2 ).
Beyond these ranges. Figure 1 maps the parameter plane. For exact values, the lower bound of Xia and Yuan [55 ] widens the window of Corollary 3.3 to T log ( e + T ) ≤ E ≤ d T\log(e + T) \le E \le d T log ( e + T ) ≤ E ≤ d when T ≤ c d / log ( e d ) T \le cd/\log(ed) T ≤ c d / log ( e d ) (Corollary H.3 ).
Figure 1. Which of our bounds match? The curves E = T log ( e + T ) E = T\log(e + T) E = T log ( e + T ) and E T = d ET = d E T = d divide the parameter plane, drawn for d = 10 6 d = 10^{6} d = 1 0 6 . In each region, the figure gives the deterministic and randomized costs where our lower and upper bounds match up to constant factors, or up to log ( e + T ) \sqrt{\log(e + T)} log ( e + T ) where marked with ≈ \approx ≈ ; “not matched” marks regions where our lower and upper bounds differ. Theorem A.1 and Corollary G.10 give the conditions.
Proof Mechanisms
We explain the ideas behind our bounds here; Appendix § 9 .1 shows how the constructions combine into the results of Section 3.
Hiding one direction at a time
Why must a deterministic method pay Ω ( d ) \Omega(d) Ω ( d ) calls for each hidden direction, even though an exact real reply can carry infinitely many bits? We separate two parts: an adversary argument, which shows that one unfinished direction forces Ω ( d ) \Omega(d) Ω ( d ) units of cost, and an analytic construction, which packs as many directions as smoothness, the initial gap, and the trace budget allow.
The hard function hides orthonormal directions u 1 , u 2 , … u_1, u_2, \ldots u 1 , u 2 , … , one per stage. As in the chains of Carmon et al. [12 ] , later directions sit behind scalar gates that vanish identically near zero, so at a point whose coordinate along u j u_j u j is zero and whose coordinates along later directions are at most 1 / 2 1/2 1/2 in magnitude, the directions after u j u_j u j contribute nothing to f ( x ) f(x) f ( x ) or ∇ f ( x ) \nabla f(x) ∇ f ( x ) . At a point where some stage is unfinished, meaning that the point’s coordinate along that stage’s direction has magnitude below a fixed level, the gradient has norm above ε \varepsilon ε . A resisting oracle (Nemirovsky and Yudin, [40 ] ) (Lemma C.4) chooses each u j u_j u j only after the earlier queries are fixed. While the calls made so far in stage j j j have total cost below ⌊ d / 2 ⌋ \lfloor d/2\rfloor ⌊ d /2 ⌋ , any u j u_j u j orthogonal to their query points and projection subspaces leaves every reply equal to that of the prefix function, which involves only u 1 , … , u j − 1 u_1,\ldots,u_{j-1} u 1 , … , u j − 1 , as long as the later directions stay nearly orthogonal to these points. On the call that closes stage j j j , the oracle fixes u j u_j u j before returning the reply. Keeping each later direction nearly orthogonal to the queries of all earlier stages, so that the earlier replies remain valid, costs the factor log ( e + T ) \sqrt{\log(e + T)} log ( e + T ) (Lemma C.2 ). If the budget runs out first, the oracle also chooses an unfinished direction orthogonal to the output x ^ \hat{x} x ^ , even if x ^ \hat{x} x ^ was never queried. One fixed function in the class then produces every reply of the transcript and has ∥ ∇ f ( x ^ ) ∥ > ε \lVert\nabla f(\hat{x})\rVert> \varepsilon ∥ ∇ f ( x ^ )∥ > ε .
Fitting the directions into the class
Hiding m m m directions forces a cost of Ω ( d m ) \Omega(dm) Ω ( d m ) , and after rescaling, each direction uses up a constant amount of the initial gap. To allow queries anywhere in R d \mathbb{R}^{d} R d , we compose the chain with a radial compression of radius R R R , which keeps the hidden coordinates bounded, and add a radial potential that rules out small gradients far from the origin. For the lower bounds on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d , the potential is weakly curved inside a ball that contains { f < f ( 0 ) } \{f < f(0)\} { f < f ( 0 )} , and it grows steeply only outside that ball, where f ( x ) > f ( 0 ) f(x) > f(0) f ( x ) > f ( 0 ) . Inside the ball, one fixed matrix, a multiple of the projection onto the m m m hidden directions plus a small multiple of the identity, dominates the Hessian, so the trace budget pays only for the compressed chain and the weak inner curvature. When T log ( e + T ) ≤ E ≤ d T\log(e + T) \le E \le d T log ( e + T ) ≤ E ≤ d , every stage that the accuracy allows fits into this budget. To obtain the lower bounds on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , we multiply the chain, instead of adding a potential, by the perspective weight ( 1 + ∥ x ∥ 2 / R 2 ) 1 / 2 (1 + \lVert x\rVert^{2}/R^{2})^{1/2} ( 1 + ∥ x ∥ 2 / R 2 ) 1/2 (Boyd and Vandenberghe, [10 ] ), which cancels the decay of the gradient far away; one fixed matrix then dominates the Hessian on all of R d \mathbb{R}^d R d . Smoothing only the scalar gates, while keeping their flat zero intervals, makes the hard functions C ∞ C^\infty C ∞ , and the stage argument still applies (Appendix § 15 .3).
Both bounds count discovered directions
The number of discovered directions is the bottleneck on both sides. Our deterministic search, which adapts the scheme of Cosson et al. [16 ] , looks for a new unit direction u i u_i u i only when the span of the earlier directions is exhausted, decreases f f f by more than ε 2 / ( 10 u i ⊤ H u i ) \varepsilon^2/(10u_i^\top H u_i) ε 2 / ( 10 u i ⊤ H u i ) along u i u_i u i , and restarts with an empty span after r r r expansions (Algorithm 1). Orthonormal directions within a block satisfy ∑ i u i ⊤ H u i ≤ L E \sum_i u_i^\top H u_i \le LE ∑ i u i ⊤ H u i ≤ L E , so by Cauchy–Schwarz a full block decreases f f f by at least Ω ( r 2 ε 2 / ( L E ) ) \Omega(r^2\varepsilon^2/(LE)) Ω ( r 2 ε 2 / ( L E )) , while the total decrease is at most Δ \Delta Δ . With r r r of order E T \sqrt{ET} E T , this allows only O ( E T ) O(\sqrt{ET}) O ( E T ) expansions when E T ≤ d ET \le d E T ≤ d (Appendix § 10 ). Conversely, the hidden chain forces Ω ( min { T , E T / log ( e + T ) } ) \Omega(\min\{T,\sqrt{ET/\log(e+T)}\}) Ω ( min { T , E T / log ( e + T ) }) directions, so the two bounds of Theorem 3.2 agree up to log ( e + T ) \sqrt{\log(e+T)} log ( e + T ) .
The randomized lower bound
Against a randomized algorithm, the hard function cannot be chosen after the random seed is drawn. By Yao’s principle (Yao, [56 ] ), it suffices to fix, before the run begins, a random quadratic whose unknown center lies in a subspace known to the algorithm. Each call yields at most as many linear observations of the center as it costs, and rotational symmetry bounds the remaining error, even at an output that was never queried. The subspace has dimension of order E T ET E T , which fits in R d \mathbb{R}^d R d when E T ≤ d ET \le d E T ≤ d , and the argument then gives the lower bound Ω ( E T ) \Omega(ET) Ω ( E T ) .
Random directions need curvature control only below f ( 0 ) f(0) f ( 0 )
A Gaussian two-point step moves from x x x to x − η s a x-\eta sa x − η s a , where s s s is the finite-difference slope of f f f along a ∼ N ( 0 , I d ) a \sim N(0,I_d) a ∼ N ( 0 , I d ) . With g = ∇ f ( x ) g=\nabla f(x) g = ∇ f ( x ) , the quadratic upper model and the Gaussian moment
E [ ( a ⊤ g ) 2 a ⊤ H a ] = ∥ g ∥ 2 tr H + 2 g ⊤ H g ≤ L ( E + 2 ) ∥ g ∥ 2 \mathbb{E}[(a^\top g)^2a^\top Ha]=\lVert g\rVert^2\operatorname{tr}H+2g^\top Hg\le L(E+2)\lVert g\rVert^2 E [( a ⊤ g ) 2 a ⊤ H a ] = ∥ g ∥ 2 tr H + 2 g ⊤ H g ≤ L ( E + 2 ) ∥ g ∥ 2
involve the trace of H H H rather than d d d . With the exact slope a ⊤ g a^\top g a ⊤ g in place of s s s , a step with step size η = 1 / ( L ( E + 2 ) ) \eta=1/(L(E+2)) η = 1/ ( L ( E + 2 )) would therefore decrease the quadratic upper model on average by at least ∥ g ∥ 2 / ( 2 L ( E + 2 ) ) \lVert g\rVert^2/(2L(E+2)) ∥ g ∥ 2 / ( 2 L ( E + 2 )) ; while ∥ g ∥ > ε \lVert g\rVert>\varepsilon ∥ g ∥ > ε , a small finite-difference radius would keep most of this decrease. A step this long may land outside the sublevel set, where only L L L -smoothness holds. Even so, the capped value min { f ( x − η s a ) , f ( 0 ) } \min\{f(x-\eta sa),f(0)\} min { f ( x − η s a ) , f ( 0 )} satisfies the same quadratic bound (Lemma B.3 ), so accepting only the steps that lower f f f suffices (Theorem B.10 ). To analyze the method without this test, we compare its iterates with an auxiliary run that drops the rare directions with large a ⊤ H a a^\top Ha a ⊤ H a ; with a slightly smaller step and a small finite-difference radius, the two runs stay close, at the price of a T log ( E T ) T\log(ET) T log ( E T ) term (Theorem B.15 ).
Related Work
Zeroth-order upper bounds under curvature conditions. Classical analyses of Gaussian finite differences reduce derivative-free optimization to stochastic first-order optimization at the cost of a factor of d d d in the query complexity ([41 , 19 , 18 ] ). To remove most of this dimension dependence for randomized methods, several works impose structural curvature conditions, which we compare with our setting in § 2.1 . These conditions measure curvature differently from our sublevel trace bound. For example, one of our hard functions F F F , with m m m hidden directions and compression radius R R R , satisfies ∇ 2 F ( 0 ) = μ I \nabla^2F(0)=\mu I ∇ 2 F ( 0 ) = μ I for μ = 11 m / R 2 \mu=11m/R^2 μ = 11 m / R 2 (Theorem D.4 ), which gives an effective dimension of at least d μ d\sqrt{\mu} d μ in the sense of Yue et al. [57 ] , whereas its positive curvature trace is of order m + μ d m+\mu d m + μ d . Under a global nuclear-norm bound ∥ ∇ 2 f ∥ ∗ ≤ L ∗ \lVert\nabla^{2} f\rVert_{*} \le L_{*} ∥ ∇ 2 f ∥ ∗ ≤ L ∗ , Anonymous [3 ] show that the Gaussian two-point method finds an ε \varepsilon ε -stationary point using O ( L ∗ Δ / ε 2 ) O(L_{*}\Delta/\varepsilon^{2}) O ( L ∗ Δ/ ε 2 ) exact function values, and they also analyze noisy values at the cost of an extra noise-dependent factor. Their nuclear-norm class satisfies our pointwise trace bound with L = L ∗ L = L_{*} L = L ∗ and E = 1 E = 1 E = 1 , and on this class the three-point method attains the same O ( L ∗ Δ / ε 2 ) O(L_{*}\Delta/\varepsilon^{2}) O ( L ∗ Δ/ ε 2 ) rate. Because fixed or neighborhood matrix bounds can require a trace of order d d d on their Twisted Valley example while the pointwise bound remains constant, we prove our randomized upper bound under that pointwise condition (Appendix § 15 .1). Unlike our work, neither Anonymous [3 ] nor Anonymous [2 ] proves lower bounds. For stochastic convex problems on a compact domain, when the subgradients lie in the range of a fixed structural preconditioner, Sholokhov and Rogozin [48 ] use ellipsoidal sampling and obtain bounds that depend on the subspace rank, conditioning, and normalized trace. Random subspace methods ([32 , 43 ] ) exploit related low-dimensional structure under different assumptions, and local gradient smoothness has been studied for both first-order and derivative-free methods ([30 ] (Zhang and Hong 2024)). On the deterministic side, we adapt the subspace expansion scheme of Cosson et al. [16 ] from low-dimensional gradient subspaces to a fixed trace budget. Under higher-order smoothness, Zhou et al. (2026) develop deterministic value-based methods that find stationary points in any dimension through a trapping-or-descent scheme with query complexity polynomial in d d d , whereas our function classes require only a Lipschitz gradient together with a trace budget.
Lower bounds for stationarity and for exact values. Query lower bounds for finding stationary points in a fixed dimension on a box date back to Vavasis [51 ] , and Bubeck and Mikulincer [11 ] extended them to randomized algorithms in d = 2 d = 2 d = 2 . On R 2 \mathbb{R}^{2} R 2 , Hollen der and Zampetakis [23 ] show that deterministic algorithms require at least Ω ( T ) \Omega(\sqrt{T}) Ω ( T ) function-value or gradient calls for stationarity, and that O ( T ) O(\sqrt{T}) O ( T ) function-value calls suffice. In high dimension, first-order Ω ( T ) \Omega(T) Ω ( T ) lower bounds build on the hidden chains of Carmon et al. [12 ] , which also yield stochastic lower bounds (Arjevani et al., [4 ] ). When function values are noisy, the query complexity of convex optimization grows with the dimension d d d (Jamieson et al., [27 ] ; Shamir [47 ] ; Duchi et al. [18 ] ). Recent work studies exact-value oracles for convex optimization on a ball, where a single real-valued reply can encode many coordinates. For deterministic algorithms, Kerger [29 ] proves a nearly quadratic lower bound for Lipschitz convex minimization, and Wu et al. [53 ] use shielded chains and delayed rotations to prove lower bounds for smooth and strongly convex optimization that match upper bounds on specified accuracy ranges. For randomized algorithms, Zhang et al. [58 ] use a posterior mean energy method to prove a lower bound with an extra factor of d d d , up to logarithmic factors, in Lipschitz convex optimization. Under noisy evaluations on a ball, Zhang et al. [58 ] establish a joint dimension–accuracy lower bound for Goldstein stationarity. Those exact-value convex bounds concern the objective error on a ball, whereas ours concern the gradient norm on all of R d \mathbb{R}^{d} R d under a curvature trace budget and hold even when the oracle returns adaptive gradient projections charged by their rank. On the unrestricted smooth class with exact function values, concurrent and independent work of Xia and Yuan [55 ] proves a randomized lower bound of Ω ( d T ) \Omega(dT) Ω ( d T ) when T ≥ c T \ge c T ≥ c and d ≥ C T log ( 2 + T ) d \ge CT\log(2+T) d ≥ C T log ( 2 + T ) for absolute constants c , C > 0 c,C>0 c , C > 0 . We prove a deterministic lower bound with the explicit constant 1 / 500 1/500 1/500 that also covers these gradient projections (Theorem 3.4 ); for that oracle, a randomized Ω ( d T ) \Omega(dT) Ω ( d T ) lower bound remains open.
The price of determinism. For Lipschitz nonsmooth functions and Goldstein stationarity, Jordan et al. [28 ] prove that deterministic methods require at least Ω ( d ) \Omega(d) Ω ( d ) calls in a setting where randomized methods attain dimension-free rates, and Kornowski [31 ] shows that this deterministic overhead can be exponential. In one dimension, Chewi et al. [14 ] prove that randomness or function values help when finding stationary points of smooth functions. We establish the high-dimensional smooth counterpart under a low curvature trace (Theorems 3.1 and 3.2 ; Corollary 3.3). Specifically, when E T ≤ d ET \le d E T ≤ d , determinism multiplies the query complexity by a factor of order d / ( E + E T ) d/(E+\sqrt{ET}) d / ( E + E T ) up to log ( e + T ) \sqrt{\log(e+T)} log ( e + T ) , and across the window T log ( e + T ) ≤ E ≤ d / T T\log(e+T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T , the deterministic-to-randomized cost ratio is of order d / E d/E d / E up to constant factors.
Empirical Illustration
We test the algorithms from our upper bounds on fixed instances to illustrate the mechanisms behind Theorems 3.1 and 3.2 . The runs highlight four behaviors: (i) deterministic query counts scale linearly with d d d , whereas randomized query counts stay nearly constant in d d d and grow with E E E and T T T ; (ii) the Gaussian two-point method and its monotone variant, which accepts only steps that lower f f f (Theorems B.12 and B.10 ), achieve similar first-hit counts, roughly half those of the three-point method, even when curvature is controlled only below the starting value f ( 0 ) f(0) f ( 0 ) ; (iii) fixed-seed runs can be defeated; and (iv) subspace search saves calls by reusing directions. We run every method without access to the true gradient, count every function evaluation (including rejected proposals), and compute first-hit call counts offline from the logged iterates; full protocols and parameter grids appear in Appendix § 17 .
Randomized versus deterministic. We compare the Gaussian two-point method (Nesterov and Spokoiny, [41 ] ), its monotone variant, and the three-point method (Theorem G.3 ) against coordinate central differences, finite-difference L-BFGS (Nocedal, [42 ] ; Liu and Nocedal, [36 ] ), and restarted subspace search on rank-one quadratics and rotated separable nonconvex functions. Because deterministic methods spend about d d d calls per gradient estimate, their cost grows linearly with d d d , whereas that of the randomized methods stays nearly constant in d d d (Figure 2 (a)). At d = 10 5 d = 10^{5} d = 1 0 5 in panel (a), subspace search reaches the target after 200,009 calls and coordinate finite differences after 600,001 calls, whereas the three-point method uses 77 calls, the two-point method uses 38, and the monotone two-point method uses 39. Across the 215 instances with E ∈ { 1 , 4 , 16 } E \in\{1, 4, 16\} E ∈ { 1 , 4 , 16 } (five seeds per cell; Appendix § 17 ), the monotone two-point method has a median first-hit count of 31 calls against 30 for the plain two-point method. On rank-one quadratics whose direction is orthogonal to every query of a fixed-seed run, that run never reaches the target, whereas fresh seeds succeed in 17–20 of 20 runs.
Figure 2. Query costs on synthetic functions and local curvature of GPT-2 (GELU). (a) Concentrated-spectrum family with E = 4 E = 4 E = 4 and T = 8 T = 8 T = 8 : offline first-hit calls of the three-point method (STP), the Gaussian two-point method, the monotone two-point method, the subspace search, coordinate finite differences, and finite-difference L-BFGS, and actual total calls (dashed). (b) Actual total calls of the subspace search and coordinate finite differences on a tridiagonal quadratic chain with d = 10 4 d = 10^{4} d = 1 0 4 and E = 8 E = 8 E = 8 ; labels give the number of subspace expansions. In (a) and (b), points and bands are medians and interquartile ranges over five seeds. (c) Upper confidence bounds, at simultaneous level 0.95, for tr ( G + ) / ( d ∥ G ∥ o p ) \operatorname{tr}(G_{+})/(d \lVert G\rVert_{\mathrm{op}}) tr ( G + ) / ( d ∥ G ∥ op ) , where G G G is the Hessian on a fixed batch, during MeZO fine-tuning with three seeds per task. Protocols and further results are in Appendix § 17 .
Curvature controlled only below f ( 0 ) f(0) f ( 0 ) . On wall functions whose Hessian is small where f ( x ) < f ( 0 ) f(x) < f(0) f ( x ) < f ( 0 ) but of order L L L along almost every direction at points where f ( x ) > f ( 0 ) f(x) > f(0) f ( x ) > f ( 0 ) (Appendix § 17 ), the monotone two-point method again matches the plain two-point method, with median first-hit counts of 69 and 72 calls against 173 for the three-point method.
Subspace search. On a tridiagonal quadratic chain, gradients reveal new directions one at a time, so subspace search uses one expansion at T = 2 T = 2 T = 2 and ten at $T = 512 ({xref}Figure 2(b)`). By reusing discovered directions instead of re-estimating full gradients, at d = 10 4 d = 10^{4} d = 1 0 4 and T = 512 T = 512 T = 512 it uses about a quarter of the calls of coordinate finite differences.
Local curvature of a smooth model. Because GPT-2 uses a smooth activation, its Hessian is defined everywhere. During MeZO fine-tuning, a simultaneous upper confidence bound for tr ( G + ) / ( d ∥ G ∥ o p ) \operatorname{tr}(G_{+})/(d \lVert G\rVert_{\mathrm{op}}) tr ( G + ) / ( d ∥ G ∥ op ) , where G G G is the fixed-batch Hessian, stays between 6.7 × 10 − 5 6.7 \times10^{-5} 6.7 × 1 0 − 5 and 1.2 × 10 − 4 1.2 \times10^{-4} 1.2 × 1 0 − 4 (Figure 2 (c)). For 1.24 × 10 8 1.24 \times10^{8} 1.24 × 1 0 8 parameters, this places the upper confidence bound on the local trace budget tr ( G + ) / ∥ G ∥ o p \operatorname{tr}(G_{+})/\lVert G\rVert_{\mathrm{op}} tr ( G + ) / ∥ G ∥ op below 1.5 × 10 4 1.5 \times10^{4} 1.5 × 1 0 4 , comparable to d \sqrt{d} d and consistent with the few large Hessian eigenvalues reported for neural networks (Ghorbani et al., [20 ] ).
Discussion and Open Problems
Conclusion. We quantified the price of determinism for smooth nonconvex zeroth-order optimization under a low curvature trace. Even when the curvature is controlled only below f ( 0 ) f(0) f ( 0 ) , randomized methods pay only for the trace, and the Gaussian two-point method with an acceptance test matches our Ω ( E T ) \Omega(ET) Ω ( E T ) lower bound for E T ≤ d ET \le d E T ≤ d . Deterministic methods also exploit a low trace, but they pay about d d d calls for each discovered direction, so for E T ≤ d ET \le d E T ≤ d their cost exceeds the randomized cost by a factor of order d / ( E + E T ) d/(E + \sqrt{ET}) d / ( E + E T ) , up to log ( e + T ) \sqrt{\log(e + T)} log ( e + T ) (Theorems 3.1 and 3.2 ).
Limitations and open problems. We prove the separation for exact function values, whereas Anonymous [3 ] also allow noisy evaluations; because independent noise can supply random bits, our deterministic lower bounds do not transfer to that model. For the Gaussian two-point method on B L , Δ , E d \mathcal{B}_{L,\Delta,E}^{d} B L , Δ , E d without an acceptance test, our bound carries an extra T log ( E T ) T\log(ET) T log ( E T ) term and requires a very small finite-difference radius (Theorem B.15 ), so it is open whether the plain method with a step size of order 1 / ( L E ) 1/(LE) 1/ ( L E ) achieves O ( E T ) O(ET) O ( E T ) calls when E < log ( 1000 E T ) E < \log(1000ET) E < log ( 1000 E T ) . The methods behind our randomized upper bounds also take E E E as an input, leaving open whether a randomized algorithm can attain an O ( E T ) O(ET) O ( E T ) query cap without knowing E E E in advance. Other open problems are the remaining factor log ( e + T ) \sqrt{\log(e + T)} log ( e + T ) , the deterministic complexity on the local pointwise class of Appendix § 15 .1, and the complexities for E T > d ET > d E T > d .
AI Usage Statement
We used AI assistants, including OpenAI’s GPT models and Anthropic’s Claude, to help develop and check proofs, formalize them in Lean, implement the experiments, and edit the text. The authors take full responsibility for the content of this paper.
Main Theorem and Proof Overview
Write T = L Δ / ε 2 ≥ 1 T = L\Delta/\varepsilon^2 \ge1 T = L Δ/ ε 2 ≥ 1 and 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d . The class B \mathcal{B} B contains globally L L L -smooth C 1 , 1 C^{1,1} C 1 , 1 functions with f ( 0 ) − inf f ≤ Δ f(0)-\inf f \le\Delta f ( 0 ) − inf f ≤ Δ and one fixed unknown matrix 0 ⪯ H ⪯ L I 0 \preceq H \preceq LI 0 ⪯ H ⪯ L I , tr H ≤ L E \operatorname{tr} H \le LE tr H ≤ L E , dominating the Hessian almost everywhere on { f < f ( 0 ) } \{f < f(0)\} { f < f ( 0 )} . The subclass F \mathcal{F} F requires the matrix bound globally. An exact value costs one; feedback ( f ( x ) , V ⊤ ∇ f ( x ) ) (f(x), V^\top\nabla f(x)) ( f ( x ) , V ⊤ ∇ f ( x )) costs 1 + rank V 1+\operatorname{rank} V 1 + rank V . The output need not be queried; exact computation is uncharged, and the superscripts cap and exp mark the cost cap of Section 2 and the worst-case expected cost.
Theorem A.1 (Main bounds with explicit constants). For either oracle on B \mathcal{B} B .
N r a n d c a p ≤ 23 E T , N r a n d c a p ≥ min { d , E T } / 12 , N r a n d e x p ≥ min { d , E T } / 144 , N_{\mathrm{rand}}^{\mathrm{cap}} \le23ET,\qquad N_{\mathrm{rand}}^{\mathrm{cap}} \ge\min\{d,ET\}/12,\qquad N_{\mathrm{rand}}^{\mathrm{exp}} \ge\min\{d,ET\}/144, N rand cap ≤ 23 E T , N rand cap ≥ min { d , E T } /12 , N rand exp ≥ min { d , E T } /144 , N d e t ≤ min { 10 d T , 100 d E T ( d + T ) } , N d e t ≥ d . (5) N_{\mathrm{det}} \le\min\{10dT,100\sqrt{dET(d+T)}\},\qquad N_{\mathrm{det}} \ge d. \tag*{(5)} N det ≤ min { 10 d T , 100 d E T ( d + T ) } , N det ≥ d . (5) If E T ≤ d ET \le d E T ≤ d , then
N d e t ≥ d 672 min { T , E T / log ( e + T ) } . N_{\mathrm{det}} \ge\frac{d}{672}\min\{T,\sqrt{ET/\log(e+T)}\}. N det ≥ 672 d min { T , E T / log ( e + T ) } . Thus for T log ( e + T ) ≤ E ≤ d / T T\log(e+T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T the deterministic bounds are Ω ( d T ) \Omega(dT) Ω ( d T ) and O ( d T ) O(dT) O ( d T ) , and the randomized bounds are Ω ( E T ) \Omega(ET) Ω ( E T ) and O ( E T ) O(ET) O ( E T ) . On the unrestricted globally smooth class, N d e t ≥ d 500 min { T , d / log ( e d ) } N_{\mathrm{det}} \ge\frac{d}{500}\min\{T,d/\log(ed)\} N det ≥ 500 d min { T , d / log ( e d )} . Every bound above for B \mathcal{B} B also holds on F \mathcal{F} F : the upper bounds because F ⊆ B \mathcal{F} \subseteq\mathcal{B} F ⊆ B , the randomized lower bounds and N d e t ≥ d N_{\mathrm{det}} \ge d N det ≥ d because their quadratic witnesses lie in F \mathcal{F} F , and the bound for E T ≤ d ET \le d E T ≤ d by Corollary G.9 .
Mechanism and roadmap. For the upper bounds (Appendix § 10 ), stopping a segment at its first exit from the initial sublevel set gives a capped quadratic upper model, so comparison and acceptance steps pay only for the trace, and the deterministic search buys a full gradient estimate only when its current subspace is exhausted, with its orthogonal expansions limited by the trace budget. The deterministic lower bounds combine one scalar chain with a radial compression and two choices of its parameters; a transcript lemma handles pending and closing calls and unqueried outputs, finite partial coloring selects the hidden directions, and scaling places the witnesses in the class. A quadratic family with an unknown center gives the randomized lower bound.
Scope. The multistage witnesses of this proof chain are C 1 , 1 C^{1,1} C 1 , 1 , and the Lean formalization of the lower bounds (Appendix § 18 ) follows this chain. Appendix § 15 gives the three-point bound on the larger class A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d (the local pointwise class of Appendix § 15 .1) and, reusing the steps of this chain, C ∞ C^\infty C ∞ witnesses with a Lipschitz Hessian for the same deterministic lower bound on F \mathcal{F} F , where a perspective weight replaces the radial potential. Appendix § 16 shows that, for 500 < T ≤ d / log ( e d ) 500 < T \le d/\log(ed) 500 < T ≤ d / log ( e d ) , the hard functions of the unrestricted bound are minimized exactly, with probability at least 5 / 6 5/6 5/6 , by a randomized method using O ( d T log ( e + T ) ) O(d\sqrt{T\log(e+T)}) O ( d T log ( e + T ) ) values, and combines our bounds with the lower bound of Xia and Yuan [55 ] into a wider window with matching bounds for exact values. In the unrestricted bound the witnesses are coercive and nonconvex when min { T , d / log ( e d ) } > 500 \min\{T,d/\log(ed)\}>500 min { T , d / log ( e d )} > 500 ; otherwise the rank-one quadratic of Lemma E.2 is used.
From Theorem A.1 to Section 3
Theorem 3.1 consists of the bounds 23 E T 23ET 23 E T and min { d , E T } / 12 \min\{d,ET\}/12 min { d , E T } /12 ; the upper bound also holds on A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d with 43 E T 43ET 43 E T calls (Theorem G.3 ). The Gaussian two-point method attains it with 12 E T + 26 T + 3 12ET+26T+3 12 E T + 26 T + 3 calls when it accepts only steps that lower f f f (Theorem B.10 ); without this test it uses $34(E+2)T+2 calls on F \mathcal{F} F and O ( E T + T log ( E T ) ) O(ET+T\log(ET)) O ( E T + T log ( E T )) calls on B \mathcal{B} B (Theorems B.12 and B.15 ). In Theorem 3.2 , E T ≤ d ET\le d E T ≤ d and E ≥ 1 E\ge1 E ≥ 1 give T ≤ d T\le d T ≤ d , so 100 d E T ( d + T ) ≤ 100 2 d E T 100\sqrt{dET}(d+T)\le100\sqrt{2}d\sqrt{ET} 100 d E T ( d + T ) ≤ 100 2 d E T ; the lower bound is Corollary E.4 , and on F \mathcal{F} F it is Corollary G.9 . If T log ( e + T ) ≤ E ≤ d / T T\log(e+T)\le E\le d/T T log ( e + T ) ≤ E ≤ d / T , then E T ≤ d ET\le d E T ≤ d and E T / log ( e + T ) ≥ T \sqrt{ET}/\log(e+T)\ge T E T / log ( e + T ) ≥ T , which gives Corollary 3.3, with the explicit ratio bounds of Corollary E.5 . Theorem 3.4 combines Theorem E.8 with the coordinate method in the proof of Theorem B.1 , which uses only global L L L -smoothness. The remaining statements of Section 3 are Lemma E.2, Theorem B.1 , Corollaries E.4 , G.9 and G.10 , and, for exact values beyond E T ≤ d ET\le d E T ≤ d , Corollary H.3 .
Upper Bounds
Theorem B.1 (Restarted deterministic subspace search). For all the preceding parameters, including T > d T>d T > d , one has
N d e t l i n ( B ) ≤ N d e t v a l ( B ) ≤ min { 10 d T , 100 d E T ( d + T ) } . (6) N_{\mathrm{det}}^{\mathrm{lin}}(\mathcal{B})\le N_{\mathrm{det}}^{\mathrm{val}}(\mathcal{B})\le\min\{10dT,\ 100\sqrt{dET(d+T)}\}. \tag*{(6)} N det lin ( B ) ≤ N det val ( B ) ≤ min { 10 d T , 100 d E T ( d + T ) } . (6) The bounds hold for every function in the class with a single deterministic, finite algorithm that uses exact values, chosen from the public parameters. No convexity, Hessian rank bound, or Hessian Lipschitz assumption is added. The same result holds on the global fixed-matrix subclass F \mathcal{F} F . A single algorithm not given E E E or H H H has the hard bound
Q ≤ min { 20 d T , 560 d E T ( d + T ) } . (7) Q\le\min\{20dT,\ 560\sqrt{dET(d+T)}\}. \tag*{(7)} Q ≤ min { 20 d T , 560 d E T ( d + T ) } . (7) It is still given d d d , L L L , Δ \Delta Δ , ε \varepsilon ε .
Lemma B.2 (Hessians almost everywhere and on every segment). Let f ∈ C 1 , 1 ( R d ) f\in C^{1,1}(\mathbb{R}^{d}) f ∈ C 1 , 1 ( R d ) , U ⊆ R d U\subseteq\mathbb{R}^{d} U ⊆ R d be open, and H H H be symmetric. Then ∇ 2 f ⪯ H \nabla^{2}f\preceq H ∇ 2 f ⪯ H almost everywhere on U U U is equivalent to
f ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 1 2 v T H v for each [ x , x + v ] ⊂ U . (8) f(x+v)\le f(x)+\langle\nabla f(x),v\rangle+\frac{1}{2}v^{\mathsf{T}}Hv\quad\text{for each }[x,x+v]\subset U. \tag*{(8)} f ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 2 1 v T H v for each [ x , x + v ] ⊂ U . (8) Proof. Choose a measurable null set N ⊂ U N\subset U N ⊂ U outside which the Lipschitz map ∇ f \nabla f ∇ f is differentiable, and its derivative is symmetric and bounded above by H H H . Fix v ≠ 0 v\ne0 v = 0 and [ x , x + v ] ⊂ U [x,x+v]\subset U [ x , x + v ] ⊂ U . By compactness a closed tube of sufficiently small positive radius about this segment lies in U U U . Use orthonormal coordinates with first direction v / ∥ v ∥ v/\lVert v\rVert v / ∥ v ∥ . Fubini applied to 1 N 1_{N} 1 N on the tube implies that for almost every sufficiently small transverse displacement z ∈ v ⊥ z\in v^{\perp} z ∈ v ⊥ the set { t ∈ [ 0 , 1 ] : x + z + t v ∈ N } \{t\in[0,1]:x+z+tv\in N\} { t ∈ [ 0 , 1 ] : x + z + t v ∈ N } has one-dimensional measure zero. For each such z z z , G z ( t ) = ∇ f ( x + z + t v ) G_z(t)=\nabla f(x+z+tv) G z ( t ) = ∇ f ( x + z + t v ) is Lipschitz and hence absolutely continuous. For almost every t t t , G z ′ ( t ) = ∇ 2 f ( x + z + t v ) v G'_z(t)=\nabla^{2}f(x+z+tv)v G z ′ ( t ) = ∇ 2 f ( x + z + t v ) v , whence
⟨ G z ( t ) − G z ( 0 ) , v ⟩ = ∫ 0 t v T ∇ 2 f ( x + z + s v ) v d s ≤ t v T H v . \langle G_z(t)-G_z(0),v\rangle=\int_{0}^{t}v^{\mathsf{T}}\nabla^{2}f(x+z+sv)v\,\mathrm{d}s\le tv^{\mathsf{T}}Hv. ⟨ G z ( t ) − G z ( 0 ) , v ⟩ = ∫ 0 t v T ∇ 2 f ( x + z + s v ) v d s ≤ t v T H v . Integration over t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] proves (8) on the translated segment. The permitted transverse displacements have full measure in a ball, so choose z n → 0 z_n\to0 z n → 0 among them. Continuity of f f f and ∇ f \nabla f ∇ f proves the inequality for the given segment. When d = 1 d=1 d = 1 , the inverse image of a one-dimensional null set under an injective affine map is null; the same argument works without translation. The case v = 0 v=0 v = 0 is immediate. Conversely, at each point where ∇ f \nabla f ∇ f is differentiable with derivative B B B , continuity and integration of the gradient give
f ( x + t v ) = f ( x ) + t ⟨ ∇ f ( x ) , v ⟩ + t 2 2 v T B v + o ( t 2 ) . f(x+tv)=f(x)+t\langle\nabla f(x),v\rangle+\frac{t^{2}}{2}v^{\mathsf{T}}Bv+o(t^{2}). f ( x + t v ) = f ( x ) + t ⟨ ∇ f ( x ) , v ⟩ + 2 t 2 v T B v + o ( t 2 ) . Apply (8) for small positive t t t and divide by t 2 / 2 t^{2}/2 t 2 /2 ; then v T B v ≤ v T H v v^{\mathsf{T}}Bv\le v^{\mathsf{T}}Hv v T B v ≤ v T H v . Because the differentiability point was fixed, this holds for every v v v at once. Rademacher’s theorem for ∇ f \nabla f ∇ f provides such points almost everywhere.
The capped upper model and finite differences
Lemma B.3 (First exit and the capped upper model). Suppose f ∈ C 1 f \in C^1 f ∈ C 1 , H ⪰ 0 H \succeq0 H ⪰ 0 , and every segment [ x , y ] [x,y] [ x , y ] contained in { f < c } \{f<c\} { f < c } satisfies
f ( y ) ≤ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ + 1 2 ( y − x ) T H ( y − x ) . f(y) \le f(x) + \langle\nabla f(x),y-x\rangle+ \frac{1}{2}(y-x)^{\mathsf T}H(y-x). f ( y ) ≤ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ + 2 1 ( y − x ) T H ( y − x ) . For every f ( x ) ≤ c f(x) \le c f ( x ) ≤ c and every v v v ,
min { f ( x + v ) , c } ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 1 2 v T H v . (9) \min\{f(x+v),c\} \le f(x) + \langle\nabla f(x),v\rangle+ \frac{1}{2}v^{\mathsf T}Hv. \tag*{(9)} min { f ( x + v ) , c } ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 2 1 v T H v . (9) The hypothesis holds for B \mathcal{B} B with c = f ( 0 ) c=f(0) c = f ( 0 ) .
Proof. For f ( x ) < c f(x)<c f ( x ) < c , let q ( t ) = f ( x ) + t ⟨ ∇ f ( x ) , v ⟩ + t 2 v T H v / 2 q(t)=f(x)+t\langle\nabla f(x),v\rangle+t^2v^{\mathsf T}Hv/2 q ( t ) = f ( x ) + t ⟨ ∇ f ( x ) , v ⟩ + t 2 v T H v /2 . If the segment stays strictly below c c c , use the hypothesis. Otherwise let s ∈ ( 0 , 1 ] s\in(0,1] s ∈ ( 0 , 1 ] be its first exit time. The hypothesis at times t < s t<s t < s and continuity imply q ( s ) ≥ c q(s)\ge c q ( s ) ≥ c . Convexity gives q ( s ) ≤ ( 1 − s ) q ( 0 ) + s q ( 1 ) q(s)\le(1-s)q(0)+sq(1) q ( s ) ≤ ( 1 − s ) q ( 0 ) + s q ( 1 ) , hence q ( 1 ) ≥ c q(1)\ge c q ( 1 ) ≥ c . This proves the claim. At f ( x ) = c f(x)=c f ( x ) = c , a nonnegative directional derivative makes the assertion immediate. With a negative directional derivative, x + s v x+sv x + s v lies strictly below c c c for all sufficiently small positive s s s . Apply the proved claim there, with displacement ( 1 − s ) v (1-s)v ( 1 − s ) v , and take s ↓ 0 s\downarrow0 s ↓ 0 .
The full equivalence is in Lemma B.2. In particular, the hypothesis for B \mathcal{B} B is supplied by that lemma.
For an orthonormal d × k d\times k d × k matrix V V V , k ≥ 1 k\ge1 k ≥ 1 , define
b k = ε 8 L k , [ D V f ( x ) ] i = f ( x + b k v i ) − f ( x − b k v i ) 2 b k . (10) b_k=\frac{\varepsilon}{8L\sqrt{k}}, \qquad[D_Vf(x)]_i=\frac{f(x+b_kv_i)-f(x-b_kv_i)}{2b_k}. \tag*{(10)} b k = 8 L k ε , [ D V f ( x ) ] i = 2 b k f ( x + b k v i ) − f ( x − b k v i ) . (10)
It costs 2 k 2k 2 k values and has vector error at most ε / 16 \varepsilon/16 ε /16 . Indeed, the scalar error is bounded by ( 2 b k ) − 1 ∫ − b k b k L ∣ t ∣ d t = L b k / 2 (2b_k)^{-1}\int_{-b_k}^{b_k}L|t|\,dt=Lb_k/2 ( 2 b k ) − 1 ∫ − b k b k L ∣ t ∣ d t = L b k /2 ; summing its squares proves the assertion. For k = 0 k=0 k = 0 use the empty vector at zero cost. Global L L L -smoothness, not the sublevel matrix condition, controls these arbitrary finite probes.
Lemma B.4 (Restricted descent and a certified residual). Repeatedly compute z = D V f ( x ) z=D_Vf(x) z = D V f ( x ) , stop when ∥ z ∥ ≤ 7 ε / 16 \lVert z\rVert\le7\varepsilon/16 ∥ z ∥ ≤ 7 ε /16 , and otherwise set x ← x − V z / L x\leftarrow x-Vz/L x ← x − V z / L . Each nonterminal step decreases f f f by more than 35 ε 2 / ( 512 L ) 35\varepsilon^2/(512L) 35 ε 2 / ( 512 L ) . On return, ∥ V T ∇ f ( x ) ∥ ≤ ε / 2 \lVert V^{\mathsf T}\nabla f(x)\rVert\le\varepsilon/2 ∥ V T ∇ f ( x )∥ ≤ ε /2 . Next compute g = D I d f ( x ) g=D_{\mathrm{Id}}f(x) g = D Id f ( x ) . If ∥ g ∥ ≤ 15 ε / 16 \lVert g\rVert\le15\varepsilon/16 ∥ g ∥ ≤ 15 ε /16 , then ∥ ∇ f ( x ) ∥ ≤ ε \lVert\nabla f(x)\rVert\le\varepsilon ∥ ∇ f ( x )∥ ≤ ε . Otherwise
u = g − V V T g ∥ g − V V T g ∥ u=\frac{g-VV^{\mathsf T}g}{\lVert g-VV^{\mathsf T}g\rVert} u = ∥ g − V V T g ∥ g − V V T g is well defined, orthogonal to V V V , and ⟨ ∇ f ( x ) , u ⟩ > 11 ε / 16 \langle\nabla f(x),u\rangle>11\varepsilon/16 ⟨ ∇ f ( x ) , u ⟩ > 11 ε /16 .
Proof. Writing w = V T ∇ f ( x ) w=V^{\mathsf T}\nabla f(x) w = V T ∇ f ( x ) and e = z − w e=z-w e = z − w gives
f ( x − V z / L ) ≤ f ( x ) − ∥ w ∥ 2 2 L + ∥ e ∥ 2 2 L . f(x-Vz/L)\le f(x)-\frac{\lVert w\rVert^2}{2L}+\frac{\lVert e\rVert^2}{2L}. f ( x − V z / L ) ≤ f ( x ) − 2 L ∥ w ∥ 2 + 2 L ∥ e ∥ 2 .
At a nonterminal step ∥ w ∥ > 3 ε / 8 \lVert w\rVert>3\varepsilon/8 ∥ w ∥ > 3 ε /8 and ∥ e ∥ ≤ ε / 16 \lVert e\rVert\le\varepsilon/16 ∥ e ∥ ≤ ε /16 , giving 35 / 512 35/512 35/512 . The terminal assertion follows by the triangle inequality. At a failed full test, ∥ V T g ∥ ≤ 9 ε / 16 \lVert V^{\mathsf T}g\rVert\le9\varepsilon/16 ∥ V T g ∥ ≤ 9 ε /16 , whereas ∥ g ∥ > 15 ε / 16 \lVert g\rVert>15\varepsilon/16 ∥ g ∥ > 15 ε /16 . Pythagoras gives ∥ g − V V T g ∥ > 3 ε / 4 \lVert g-VV^{\mathsf T}g\rVert>3\varepsilon/4 ∥ g − V V T g ∥ > 3 ε /4 . Subtract the full estimation error ε / 16 \varepsilon/16 ε /16 to obtain the asserted slope. The successful full test also follows from that error bound.
A doubling line search
The search uses an Armijo-type sufficient decrease test ([5 ] ) with doubling trial steps.
Lemma B.5 (Decrease along a new direction and amortized query cost). Suppose f ( x ) ≤ f ( 0 ) f(x) \le f(0) f ( x ) ≤ f ( 0 ) and u u u is a unit vector satisfying ⟨ ∇ f ( x ) , u ⟩ > 11 ε / 16 \langle\nabla f(x),u\rangle> 11\varepsilon/16 ⟨ ∇ f ( x ) , u ⟩ > 11 ε /16 . Query f ( x ) f(x) f ( x ) , and test t j = 2 j ε / ( 2 L ) t_j = 2^j\varepsilon/(2L) t j = 2 j ε / ( 2 L ) for j = 0 , 1 , … j = 0,1,\ldots j = 0 , 1 , … . A test passes precisely when
f ( x − t j u ) ≤ f ( x ) − ε t j / 4. (11) f(x-t_j u) \le f(x)-\varepsilon t_j/4. \tag*{(11)} f ( x − t j u ) ≤ f ( x ) − ε t j /4. (11) Stop at the first failure and return the preceding accepted point x + x^+ x + . For λ = u ⊤ H u \lambda= u^\top H u λ = u ⊤ H u one has λ > 0 \lambda> 0 λ > 0 and
D x : = f ( x ) − f ( x + ) > 7 ε 2 64 λ > ε 2 10 λ . (12) D_x := f(x)-f(x^+) > \frac{7\varepsilon^2}{64\lambda} > \frac{\varepsilon^2}{10\lambda}. \tag*{(12)} D x := f ( x ) − f ( x + ) > 64 λ 7 ε 2 > 10 λ ε 2 . (12) The search terminates. Its number of proposal evaluations, including the failed proposal but excluding its one center evaluation, is at most 8 L D x / ε 2 + 1 8LD_x/\varepsilon^2+1 8 L D x / ε 2 + 1 .
Proof. Apply (9) to − t u -tu − t u . If λ = 0 \lambda= 0 λ = 0 , the capped upper bound tends to − ∞ -\infty − ∞ , contradicting boundedness below. If 0 < t ≤ 7 ε / ( 8 λ ) 0 < t \le7\varepsilon/(8\lambda) 0 < t ≤ 7 ε / ( 8 λ ) , the bound is strictly below f ( x ) − ε t / 4 < f ( 0 ) f(x)-\varepsilon t/4 < f(0) f ( x ) − εt /4 < f ( 0 ) , so the actual value equals its cap and passes the test. Since λ ≤ L \lambda\le L λ ≤ L , the initial test passes. An accepted test must have t ≤ 4 ( f ( x ) − inf f ) / ε t \le4(f(x)-\inf f)/\varepsilon t ≤ 4 ( f ( x ) − inf f ) / ε , so a failure occurs at a finite index. If s s s was the preceding accepted step, the failed step 2 s 2s 2 s exceeds 7 ε / ( 8 λ ) 7\varepsilon/(8\lambda) 7 ε / ( 8 λ ) . Therefore D x ≥ ε s / 4 > 7 ε 2 / ( 64 λ ) D_x \ge\varepsilon s/4 > 7\varepsilon^2/(64\lambda) D x ≥ ε s /4 > 7 ε 2 / ( 64 λ ) . No monotonicity of test outcomes after the guaranteed initial interval is assumed.
Write s = 2 j ε / ( 2 L ) s = 2^j\varepsilon/(2L) s = 2 j ε / ( 2 L ) . The search used j + 2 j+2 j + 2 proposals, and D x ≥ 2 j ε 2 / ( 8 L ) D_x \ge2^j\varepsilon^2/(8L) D x ≥ 2 j ε 2 / ( 8 L ) . The elementary induction j + 2 ≤ 2 j + 1 j+2 \le2^j+1 j + 2 ≤ 2 j + 1 gives the last assertion. ▫
Consequently, for n n n expansion moves along one monotone trajectory , all their proposal evaluations together number at most 8 T + n 8T+n 8 T + n . The n n n center evaluations are charged separately. There is no n log T n\log T n log T term.
The restarted algorithm and its cost
Algorithm 1 collects the steps of Lemmas B.4 and B.5 for a fixed capacity r ∈ { 1 , … , d } r \in\{1,\ldots,d\} r ∈ { 1 , … , d } . Within the current span V V V it descends by central differences; when no progress is certified there, it estimates the full gradient, returns if the estimate is small, and otherwise adds the normalized residual direction, moves along it by a doubling line search, and restarts with an empty span after r r r additions.
The test at t 0 t_0 t 0 always passes (Lemma B.5), so j ≥ 1 j \ge1 j ≥ 1 when the line search stops. Theorem B.1 uses r = ⌈ d E T / ( d + T ) ⌉ r = \lceil\sqrt{dET}/(d+T)\rceil r = ⌈ d E T / ( d + T )⌉ ; without E E E and H H H , Section B.4 runs copies with r = 1 , 2 , 4 , … r = 1,2,4,\ldots r = 1 , 2 , 4 , … under fixed budgets. Each branch uses exact function values and deterministic operations. A single fixed-r r r run does not need E E E , H H H , or Δ \Delta Δ . The basis dimension at the beginning of any restricted search is at most r − 1 r-1 r − 1 .
Lemma B.6 (A square budget across restarts). Suppose there are n = q r + s n=qr+s n = q r + s expansions in total, where q ≥ 0 q \ge0 q ≥ 0 and 0 ≤ s < r 0 \le s < r 0 ≤ s < r . Then
q r 2 + s 2 < 10 E T , n < 10 E T r + r 4 . (13) qr^2+s^2 < 10ET,\qquad n < \frac{10ET}{r}+\frac{r}{4}. \tag*{(13)} q r 2 + s 2 < 10 E T , n < r 10 E T + 4 r . (13) All restricted nonterminal updates together number at most J = ⌊ 512 T / 35 ⌋ J=\lfloor512T/35\rfloor J = ⌊ 512 T /35 ⌋ . The algorithm stops with a certified output.
Algorithm 1 Restarted subspace search with capacity r r r . For an orthonormal d × k d \times k d × k matrix V V V with columns v 1 , … , v k v_1,\ldots,v_k v 1 , … , v k , the vector D V f ( x ) D_V f(x) D V f ( x ) has entries [ f ( x + b k v i ) − f ( x − b k v i ) ] / ( 2 b k ) [f(x+b_kv_i)-f(x-b_kv_i)]/(2b_k) [ f ( x + b k v i ) − f ( x − b k v i )] / ( 2 b k ) with b k = ε / ( 8 L k ) b_k=\varepsilon/(8L\sqrt{k}) b k = ε / ( 8 L k ) , as in (10) , and is empty when k = 0 k=0 k = 0 ; D I d D_{I_d} D I d uses the coordinate vectors. Steps are t j = 2 j ε / ( 2 L ) t_j=2^j\varepsilon/(2L) t j = 2 j ε / ( 2 L ) .
Require: d, L, ε, a capacity r ∈ {1, …, d}, and exact values of f
1: x ← 0, and let V be the empty d × 0 matrix
2: loop
3: z ← D_V f(x)
4: while ‖z‖ > 7ε/16 do descent within the span
5: x ← x − Vz/L, z ← D_V f(x)
6: end while
7: g ← D_{I_d} f(x) full estimate, 2d values
8: if ‖g‖ ≤ 15ε/16 then
9: return x
10: end if
11: u ← (g − VVᵀg)/‖g − VVᵀg‖ new direction
12: j ← 0
13: while f(x − t_j u) ≤ f(x) − εt_j/4 do doubling line search
14: j ← j + 1
15: end while
16: x ← x − t_{j−1}u, V ← [V u]
17: if V has r columns then
18: let V be empty restart
19: end if
20: end loop
Proof. In a restart block containing k k k expansions, the appended vectors are orthonormal. For their positive curvatures λ i = u i T H u i \lambda_i=u_i^{\mathsf T}Hu_i λ i = u i T H u i ,
∑ i = 1 k λ i ≤ L E , ∑ i = 1 k D i > ε 2 10 ∑ i = 1 k λ i − 1 ≥ ε 2 k 2 10 L E . \sum_{i=1}^{k}\lambda_i \le LE,\qquad
\sum_{i=1}^{k}D_i>\frac{\varepsilon^2}{10}\sum_{i=1}^{k}\lambda_i^{-1}\ge\frac{\varepsilon^2k^2}{10LE}. i = 1 ∑ k λ i ≤ L E , i = 1 ∑ k D i > 10 ε 2 i = 1 ∑ k λ i − 1 ≥ 10 L E ε 2 k 2 .
The last inequality is Cauchy–Schwarz. Crucially, this trace sum is used within each block only ; vectors from different blocks need not be orthogonal. Their actual decreases are disjoint parts of one monotone trajectory, so summing the block inequalities gives q r 2 + s 2 < 10 E T qr^2+s^2<10ET q r 2 + s 2 < 10 E T . Now
n = q r 2 + s 2 r + s ( 1 − s / r ) < 10 E T r + r 4 , n=\frac{qr^2+s^2}{r}+s(1-s/r)<\frac{10ET}{r}+\frac{r}{4}, n = r q r 2 + s 2 + s ( 1 − s / r ) < r 10 E T + 4 r ,
since ( s − r / 2 ) 2 ≥ 0 (s-r/2)^2\ge0 ( s − r /2 ) 2 ≥ 0 .
Lemma B.4 and the same monotone gap budget bound the restricted updates by J J J . Every line search is finite. An infinite run would therefore require infinitely many expansions, hence infinitely many complete blocks, contradicting their positive uniform gap charge. The only return is the certified return after a full gradient estimate.
Algorithm 2 Coordinate finite differences. The estimate uses central differences along the coordinate vectors $e_1,\ldots,e_d$ with radius $b=\varepsilon/(4L\sqrt{d})$.
Require: $d$, $L$, $\varepsilon$, and exact values of $f$
1: $x\leftarrow 0$
2: loop
3: $g\leftarrow\left([f(x+be_a)-f(x-be_a)]/(2b)\right)_{a=1}^{d}$ \hfill $2d$ values
4: if $\lVert g\rVert\le 7\varepsilon/8$ then
5: return $x$
6: end if
7: $x\leftarrow x-g/L$
8: end loop
Theorem B.7 (Cost at a fixed capacity). For any r ∈ { 1 , … , d } r\in\{1,\ldots,d\} r ∈ { 1 , … , d } the preceding deterministic algorithm has the worst-case query count
Q r ≤ 2 ( d + r − 1 ) ( n + 1 ) + 2 ( r − 1 ) J + 8 T + 2 n , (14) Q_r\le2(d+r-1)(n+1)+2(r-1)J+8T+2n, \tag*{(14)} Q r ≤ 2 ( d + r − 1 ) ( n + 1 ) + 2 ( r − 1 ) J + 8 T + 2 n , (14) Q r < 40 d E T r + 30 ( d + T ) r . (15) Q_r<\frac{40dET}{r}+30(d+T)r. \tag*{(15)} Q r < r 40 d E T + 30 ( d + T ) r . (15) Its stored basis uses at most d r dr d r real entries.
Proof. There are n + 1 n+1 n + 1 full estimates at cost 2 d 2d 2 d each and at most n + 1 n+1 n + 1 terminal restricted estimates at cost 2 ( r − 1 ) 2(r-1) 2 ( r − 1 ) each. The J J J nonterminal restricted estimates also cost at most 2 ( r − 1 ) 2(r-1) 2 ( r − 1 ) each. The preceding line search amortization charges at most 8 T + n 8T+n 8 T + n proposals and n n n centers. This proves (14) ; in particular it charges the final tests and every rejected proposal.
Because r ≤ d r\le d r ≤ d , (14) and (13) yield
Q r < 40 d E T r + d r + 4 d + 1024 35 ( r − 1 ) T + 8 T . Q_r<\frac{40dET}{r}+dr+4d+\frac{1024}{35}(r-1)T+8T. Q r < r 40 d E T + d r + 4 d + 35 1024 ( r − 1 ) T + 8 T .
Use d r + 4 d ≤ 5 d r dr+4d\le5dr d r + 4 d ≤ 5 d r , and ( 1024 / 35 ) ( r − 1 ) T + 8 T ≤ ( 1024 / 35 ) r T < 30 r T (1024/35)(r-1)T+8T\le(1024/35)rT<30rT ( 1024/35 ) ( r − 1 ) T + 8 T ≤ ( 1024/35 ) r T < 30 r T . The resulting expression is smaller than (15) .
Proof of (6) . Set A = d E T / ( d + T ) A=dET/(d+T) A = d E T / ( d + T ) and Z = d E T ( d + T ) Z=\sqrt{dET(d+T)} Z = d E T ( d + T ) . One has A ≥ 1 / 2 A\ge1/2 A ≥ 1/2 and A < d 2 A<d^2 A < d 2 , so r = ⌈ A ⌉ r=\lceil\sqrt{A}\rceil r = ⌈ A ⌉ lies in { 1 , … , d } \{1,\ldots,d\} { 1 , … , d } and
A ≤ r ≤ 2 A . \sqrt{A}\le r\le2\sqrt{A}. A ≤ r ≤ 2 A .
For A ≤ 1 \sqrt{A}\le1 A ≤ 1 the latter follows from A ≥ 1 / 2 A\ge1/2 A ≥ 1/2 ; for A > 1 \sqrt{A}>1 A > 1 it follows from ⌈ A ⌉ ≤ A + 1 \lceil\sqrt{A}\rceil\le\sqrt{A}+1 ⌈ A ⌉ ≤ A + 1 . Equation (15) then gives Q r < 40 Z + 60 Z = 100 Z Q_r<40Z+60Z=100Z Q r < 40 Z + 60 Z = 100 Z .
For completeness, the ordinary coordinate algorithm (Algorithm 2) uses difference radius ε / ( 4 L d ) \varepsilon/(4L\sqrt{d}) ε / ( 4 L d ) , vector error ε / 8 \varepsilon/8 ε /8 , stopping threshold 7 ε / 8 7\varepsilon/8 7 ε /8 , and update x − g / L x-g/L x − g / L otherwise. Every nonterminal step decreases f f f by more than 35 ε 2 / ( 128 L ) 35\varepsilon^2/(128L) 35 ε 2 / ( 128 L ) . Including the final estimate costs at most 2 d ( 1 + ⌊ 128 T / 35 ⌋ ) ≤ 10 d T 2d(1+\lfloor128T/35\rfloor)\le10dT 2 d ( 1 + ⌊ 128 T /35 ⌋) ≤ 10 d T for T ≥ 1 T\ge1 T ≥ 1 . Selecting the algorithm with the smaller public bound proves the minimum. If an exact comparison of bounds is inconvenient, interleave the two correct procedures one charged query at a time; this loses at most a factor two, not correctness.
Algorithm 3 Subspace search without knowledge of E E E or H H H . Each trial is a fresh run of Algorithm 1 from the origin, stopped after its budget; an unfinished trial is discarded.
Require: d d d , L L L , Δ \Delta Δ , ε \varepsilon ε , and exact values of f f f ; T = L Δ / ε 2 T = L\Delta/\varepsilon^2 T = L Δ/ ε 2
procedure Trials
for j = 0 , 1 , … , ⌈ log 2 d ⌉ j = 0,1,\ldots,\lceil\log_2 d\rceil j = 0 , 1 , … , ⌈ log 2 d ⌉ do
r j ← min { 2 j , d } r_j \leftarrow\min\{2^j,d\} r j ← min { 2 j , d } , B j ← ⌊ 70 ( d + T ) r j ⌋ B_j \leftarrow\lfloor70(d+T)r_j\rfloor B j ← ⌊ 70 ( d + T ) r j ⌋
run Algorithm 1 with capacity r j r_j r j for at most B j B_j B j value calls
if that run returned a point x x x then
return x x x
end if
end for
end procedure
Run Trials and Algorithm 2 in alternation, one value call from each at a time with separate states, and return the first output.
Removing knowledge of the trace
Algorithm 3 combines restarted trials of Algorithm 1 with growing capacity and the coordinate algorithm; it reads neither E E E nor H H H .
Proof of (7) . Use capacities r j = min { 2 j , d } r_j = \min\{2^j,d\} r j = min { 2 j , d } for j = 0 , … , ⌈ log 2 d ⌉ j = 0,\ldots,\lceil\log_2 d\rceil j = 0 , … , ⌈ log 2 d ⌉ . Run a fresh copy of the fixed-r j r_j r j procedure from the origin, capped at
B j = ⌊ 70 ( d + T ) r j ⌋ B_j = \lfloor70(d+T)r_j\rfloor B j = ⌊ 70 ( d + T ) r j ⌋
actual value evaluations. A completed certification at the budget is valid; otherwise discard an unfinished computation, increase the capacity, and restart. No trial reads E E E or H H H .
Define A , Z A,Z A , Z as above only for the analysis, and let j ∗ j_* j ∗ be the first index with r j ∗ ≥ A r_{j_*} \ge\sqrt{A} r j ∗ ≥ A . It exists because A < d \sqrt{A}<d A < d . For that trial (15) gives the strict inequality Q r j ∗ < 70 ( d + T ) r j ∗ Q_{r_{j_*}} < 70(d+T)r_{j_*} Q r j ∗ < 70 ( d + T ) r j ∗ , and since Q r j ∗ Q_{r_{j_*}} Q r j ∗ is an integer we have Q r j ∗ ≤ B j ∗ Q_{r_{j_*}} \le B_{j_*} Q r j ∗ ≤ B j ∗ . It therefore completes. If j ∗ = 0 j_* = 0 j ∗ = 0 , then ∑ j = 0 j ∗ r j = 1 ≤ 2 A < 4 A \sum_{j=0}^{j_*}r_j = 1 \le\sqrt{2A}<4\sqrt{A} ∑ j = 0 j ∗ r j = 1 ≤ 2 A < 4 A ; if j ∗ ≥ 1 j_* \ge1 j ∗ ≥ 1 , then 2 j ∗ − 1 = r j ∗ − 1 < A 2^{j_*-1}=r_{j_*-1}<\sqrt{A} 2 j ∗ − 1 = r j ∗ − 1 < A , so ∑ j = 0 j ∗ − 1 r j = 2 j ∗ − 1 < 2 A \sum_{j=0}^{j_*-1}r_j=2^{j_*}-1<2\sqrt{A} ∑ j = 0 j ∗ − 1 r j = 2 j ∗ − 1 < 2 A and r j ∗ ≤ 2 j ∗ < 2 A r_{j_*}\le2^{j_*}<2\sqrt{A} r j ∗ ≤ 2 j ∗ < 2 A , giving ∑ j = 0 j ∗ r j < 4 A \sum_{j=0}^{j_*}r_j<4\sqrt{A} ∑ j = 0 j ∗ r j < 4 A even when the final capacity is shortened to d d d . The total is less than 280 ( d + T ) A = 280 Z 280(d+T)\sqrt{A}=280Z 280 ( d + T ) A = 280 Z .
Finally interleave this entire procedure, which needs no knowledge of the trace, and the coordinate algorithm, keeping their states separate, one actual value call from each in alternation. Stop when either returns. Both returns are certified, and the cost is at most twice the smaller individual cost. This gives (7) . The restarting trials use separate gap budgets; only their already bounded oracle costs are added.
The three-point method on the main class
Algorithm 4 states the three-point method with step h h h and N N N iterations; Theorem B.8 below and Theorem G.3 in Appendix G fix these parameters.
Theorem B.8 (An elementary three-point bound on B \mathcal{B} B ). For f ∈ B f \in\mathcal{B} f ∈ B , take h = 4 ε / ( 5 L E ) h = 4\varepsilon/(5LE) h = 4 ε / ( 5 L E ) and N = ⌈ 10 E T ⌉ N = \lceil10ET\rceil N = ⌈ 10 E T ⌉ . Query f ( 0 ) f(0) f ( 0 ) once. For t = 0 , … , N − 1 t = 0,\ldots,N-1 t = 0 , … , N − 1 , draw an independent a t ∼ N ( 0 , I d ) a_t \sim\mathcal{N}(0,I_d) a t ∼ N ( 0 , I d ) and keep a point of lowest value among x t x_t x t , x t + h a t x_t+ha_t x t + h a t , x t − h a t x_t-ha_t x t − h a t , caching its value and retaining the center on a tie. Return an independently uniform point from x 0 , … , x N − 1 x_0,\ldots,x_{N-1} x 0 , … , x N − 1 . Then
P ( ∥ ∇ f ( x ) ∥ > ε ) < 25 / 78 < 1 / 3 , Q = 1 + 2 ⌈ 10 E T ⌉ ≤ 23 E T . \mathbb{P}(\|\nabla f(x)\|>\varepsilon)<25/78<1/3,\qquad Q=1+2\lceil10ET\rceil\le23ET. P ( ∥∇ f ( x ) ∥ > ε ) < 25/78 < 1/3 , Q = 1 + 2 ⌈ 10 E T ⌉ ≤ 23 E T .
Figure 4. Algorithm 4 Three-point method with step h h h and N N N iterations. The comparison step is the update of Bergou et al. [9 ] . On B \mathcal{B} B , h = 4 ε / ( 5 L E ) h=4\varepsilon/(5LE) h = 4 ε / ( 5 L E ) and N = ⌈ 10 E T ⌉ N=\lceil10ET\rceil N = ⌈ 10 E T ⌉ (Theorem B.8 ); on A \mathcal{A} A , h = 4 ε / ( 5 L ( E + 1 ) ) h=4\varepsilon/(5L(E+1)) h = 4 ε / ( 5 L ( E + 1 )) and N = ⌈ 10 L ( E + 1 ) Δ / ε 2 ⌉ N=\lceil10L(E+1)\Delta/\varepsilon^2\rceil N = ⌈ 10 L ( E + 1 ) Δ/ ε 2 ⌉ (Theorem G.3 ). Require: h h h , N N N , and exact values of f f f . 1: x 0 ← 0 x_0\leftarrow0 x 0 ← 0 ; query f ( x 0 ) f(x_0) f ( x 0 ) . 2: for t = 0 , … , N − 1 t=0,\ldots,N-1 t = 0 , … , N − 1 do. 3: draw a t ∼ N ( 0 , I d ) a_t\sim N(0,I_d) a t ∼ N ( 0 , I d ) independently; query f ( x t + h a t ) f(x_t+ha_t) f ( x t + h a t ) and f ( x t − h a t ) f(x_t-ha_t) f ( x t − h a t ) . 4: x t + 1 ← x_{t+1}\leftarrow x t + 1 ← a point of lowest value among x t x_t x t , x t + h a t x_t+ha_t x t + h a t , x t − h a t x_t-ha_t x t − h a t , keeping x t x_t x t on a tie. 5: end for. 6: draw J J J uniformly from { 0 , … , N − 1 } \{0,\ldots,N-1\} { 0 , … , N − 1 } , independently, and return x J x_J x J .
This is the update of Bergou et al. [9 ] ; on B \mathcal{B} B its analysis needs only Lemma B.3 and two Gaussian moments.
Proof. At a base point x x x with f ( x ) ≤ f ( 0 ) f(x)\le f(0) f ( x ) ≤ f ( 0 ) , write g = ∇ f ( x ) g=\nabla f(x) g = ∇ f ( x ) . Of the two proposals, the sign − sign ( a ⊤ g ) -\operatorname{sign}(a^\top g) − sign ( a ⊤ g ) has capped quadratic bound f ( x ) − h ∣ a ⊤ g ∣ + ( h 2 / 2 ) a ⊤ H a f(x)-h|a^\top g|+(h^2/2)a^\top Ha f ( x ) − h ∣ a ⊤ g ∣ + ( h 2 /2 ) a ⊤ H a . The true three-point minimum is no larger than this expression: if the expression is below f ( x ) f(x) f ( x ) , Lemma B.3 forces an actual proposal below f ( x ) f(x) f ( x ) ; otherwise the retained center already suffices. Consequently
E [ f ( x ) − f ( x + ) ∣ x ] ≥ h 2 / π ∥ g ∥ − ( h 2 / 2 ) tr H . \mathbb{E}[f(x)-f(x^+)\mid x]\ge h\sqrt{2/\pi}\,\lVert g\rVert-(h^2/2)\operatorname{tr}H. E [ f ( x ) − f ( x + ) ∣ x ] ≥ h 2/ π ∥ g ∥ − ( h 2 /2 ) tr H .
It is also nonnegative because each realized update is monotone. For ∥ g ∥ > ε \lVert g\rVert>\varepsilon ∥ g ∥ > ε , use 2 / π > 79 / 100 \sqrt{2/\pi}>79/100 2/ π > 79/100 (for example from π < 22 / 7 \pi<22/7 π < 22/7 ) to lower bound the decrease by 39 ε 2 / ( 125 L E ) 39\varepsilon^2/(125LE) 39 ε 2 / ( 125 L E ) . Sum over all steps: the expected number of indices with gradient above ε \varepsilon ε is less than 125 E T / 39 125ET/39 125 E T /39 . Uniform output and N ≥ 10 E T N\ge10ET N ≥ 10 E T give 25 / 78 25/78 25/78 . Every proposal, accepted or rejected, costs one query, while center values are cached. Only Gaussian first and second moments have been used. □
A monotone two-point method on the main class
The three-point method moves by a fixed length proportional to ε \varepsilon ε . The Gaussian two-point method of Nesterov and Spokoiny [41 ] instead scales each step by an estimated directional derivative, but a step may leave the initial sublevel set, where B \mathcal{B} B controls only the gradient Lipschitz constant. Accepting a proposal only when it lowers f f f keeps every iterate below f ( 0 ) f(0) f ( 0 ) , so Lemma B.3 again controls every step. Algorithm 5 states the method with the parameters of Theorem B.10 .
Lemma B.9 (Gaussian moments). Let a ∼ N ( 0 , I d ) a\sim N(0,I_d) a ∼ N ( 0 , I d ) , g ∈ R d g\in\mathbb{R}^d g ∈ R d , and let H H H be symmetric. Then
E [ ( g ⊤ a ) 2 ( a ⊤ H a ) ] = ∥ g ∥ 2 tr H + 2 g ⊤ H g , \mathbb{E}[(g^\top a)^2(a^\top Ha)]=\lVert g\rVert^2\operatorname{tr}H+2g^\top Hg, E [( g ⊤ a ) 2 ( a ⊤ H a )] = ∥ g ∥ 2 tr H + 2 g ⊤ H g , and E ∥ a ∥ 2 k = d ( d + 2 ) ⋯ ( d + 2 k − 2 ) \mathbb{E}\lVert a\rVert^{2k}=d(d+2)\cdots(d+2k-2) E ∥ a ∥ 2 k = d ( d + 2 ) ⋯ ( d + 2 k − 2 ) for $k=1,2,3.
Proof. For g ≠ 0 g\ne0 g = 0 , take an orthonormal basis v 1 , … , v d v_1,\ldots,v_d v 1 , … , v d with v 1 = g / ∥ g ∥ v_1=g/\lVert g\rVert v 1 = g / ∥ g ∥ and let z i = v i ⊤ a z_i=v_i^\top a z i = v i ⊤ a , which are independent standard normal variables. Then ( g ⊤ a ) 2 = ∥ g ∥ 2 z 1 2 (g^\top a)^2=\lVert g\rVert^2z_1^2 ( g ⊤ a ) 2 = ∥ g ∥ 2 z 1 2 , and E [ z 1 4 ] = 3 \mathbb{E}[z_1^4]=3 E [ z 1 4 ] = 3 , E [ z i 2 ] = 1 \mathbb{E}[z_i^2]=1 E [ z i 2 ] = 1 for i ≥ 2 i\ge2 i ≥ 2 , and E [ z 1 2 z i z j ] = 0 \mathbb{E}[z_1^2z_iz_j]=0 E [ z 1 2 z i z j ] = 0 for i ≠ j i\ne j i = j give ∥ g ∥ 2 ( 3 v 1 ⊤ H v 1 + ∑ i ≥ 2 v i ⊤ H v i ) \lVert g\rVert^2(3v_1^\top Hv_1+\sum_{i\ge2}v_i^\top Hv_i) ∥ g ∥ 2 ( 3 v 1 ⊤ H v 1 + ∑ i ≥ 2 v i ⊤ H v i ) . The moments of ∥ a ∥ 2 ∼ χ d 2 \lVert a\rVert^2\sim\chi_d^2 ∥ a ∥ 2 ∼ χ d 2 are standard. □
Theorem B.10 (A monotone two-point bound on B \mathcal{B} B ). For f ∈ B f\in\mathcal{B} f ∈ B , set
η = 1 L ( E + 2 ) , N = ⌈ ( 6 E + 13 ) T ⌉ , μ = ε 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 ) . \eta=\frac{1}{L(E+2)},\qquad N=\lceil(6E+13)T\rceil,\qquad\mu=\frac{\varepsilon}{4L(6E+13)\sqrt{d(d+2)(d+4)}}. η = L ( E + 2 ) 1 , N = ⌈( 6 E + 13 ) T ⌉ , μ = 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 ) ε . Query f ( 0 ) f(0) f ( 0 ) once. For t = 0 , … , N − 1 t=0,\ldots,N-1 t = 0 , … , N − 1 , draw an independent a t ∼ N ( 0 , I d ) a_t\sim N(0,I_d) a t ∼ N ( 0 , I d ) , query f ( x t + μ a t ) f(x_t+\mu a_t) f ( x t + μ a t ) , and with the cached value f ( x t ) f(x_t) f ( x t ) set s t = [ f ( x t + μ a t ) − f ( x t ) ] / μ s_t=[f(x_t+\mu a_t)-f(x_t)]/\mu s t = [ f ( x t + μ a t ) − f ( x t )] / μ and y t = x t − η s t a t y_t=x_t-\eta s_ta_t y t = x t − η s t a t . Query f ( y t ) f(y_t) f ( y t ) , and
Algorithm 5 Monotone two-point method, with η = 1 / ( L ( E + 2 ) ) \eta= 1/(L(E + 2)) η = 1/ ( L ( E + 2 )) , N = ⌈ ( 6 E + 13 ) T ⌉ N = \lceil(6E + 13)T\rceil N = ⌈( 6 E + 13 ) T ⌉ , and μ = ε / ( 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 ) ) \mu= \varepsilon/(4L(6E + 13)\sqrt{d}(d + 2)(d + 4)) μ = ε / ( 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 )) as in Theorem B.10 .
Require: η \eta η , N N N , μ \mu μ , and exact values of f f f
x 0 ← 0 x_0 \leftarrow0 x 0 ← 0 ; query f ( x 0 ) f(x_0) f ( x 0 )
for t = 0 , … , N − 1 t = 0, \ldots, N - 1 t = 0 , … , N − 1 do
draw a t ∼ N ( 0 , I d ) a_t \sim N(0, I_d) a t ∼ N ( 0 , I d ) independently; query f ( x t + μ a t ) f(x_t + \mu a_t) f ( x t + μ a t )
s t ← [ f ( x t + μ a t ) − f ( x t ) ] / μ s_t \leftarrow[f(x_t + \mu a_t) - f(x_t)]/\mu s t ← [ f ( x t + μ a t ) − f ( x t )] / μ \hfill f ( x t ) f(x_t) f ( x t ) is cached
y t ← x t − η s t a t y_t \leftarrow x_t - \eta s_t a_t y t ← x t − η s t a t ; query f ( y t ) f(y_t) f ( y t )
if f ( y t ) < f ( x t ) f(y_t) < f(x_t) f ( y t ) < f ( x t ) then
x t + 1 ← y t \quad x_{t+1} \leftarrow y_t x t + 1 ← y t
else
x t + 1 ← x t \quad x_{t+1} \leftarrow x_t x t + 1 ← x t
end if
end for
draw J J J uniformly from { 0 , … , N − 1 } \{0, \ldots, N - 1\} { 0 , … , N − 1 } , independently, and return x J x_J x J
set x t + 1 = y t x_{t+1} = y_t x t + 1 = y t if f ( y t ) < f ( x t ) f(y_t) < f(x_t) f ( y t ) < f ( x t ) and x t + 1 = x t x_{t+1} = x_t x t + 1 = x t otherwise. Return an independently uniform point from x 0 , … , x N − 1 x_0, \ldots, x_{N-1} x 0 , … , x N − 1 . Then
P ( ∥ ∇ f ( x ^ ) ∥ > ε ) < 1 / 3 , Q = 1 + 2 ⌈ ( 6 E + 13 ) T ⌉ ≤ 12 E T + 26 T + 3. \mathbb{P}(\lVert\nabla f(\widehat{x})\rVert> \varepsilon) < 1/3,\qquad Q = 1 + 2\lceil(6E + 13)T\rceil\le12ET + 26T + 3. P (∥ ∇ f ( x )∥ > ε ) < 1/3 , Q = 1 + 2 ⌈( 6 E + 13 ) T ⌉ ≤ 12 E T + 26 T + 3.
Proof. The acceptance rule keeps f ( x t ) ≤ f ( 0 ) f(x_t) \le f(0) f ( x t ) ≤ f ( 0 ) . At a base point x x x with f ( x ) ≤ f ( 0 ) f(x) \le f(0) f ( x ) ≤ f ( 0 ) , write g = ∇ f ( x ) g = \nabla f(x) g = ∇ f ( x ) , a = a t a = a_t a = a t , s = s t s = s_t s = s t , M = L ( E + 2 ) = 1 / η M = L(E + 2) = 1/\eta M = L ( E + 2 ) = 1/ η , and x + x^+ x + for the next iterate. Since f ( x + ) = min { f ( x ) , f ( y t ) } ≤ min { f ( y t ) , f ( 0 ) } f(x^+) = \min\{f(x), f(y_t)\} \le\min\{f(y_t), f(0)\} f ( x + ) = min { f ( x ) , f ( y t )} ≤ min { f ( y t ) , f ( 0 )} , Lemma B.3 with displacement − η s a -\eta sa − η s a gives
f ( x ) − f ( x + ) ≥ η s ( g ⊤ a ) − η 2 2 s 2 ( a ⊤ H a ) . f(x) - f(x^+) \ge\eta s(g^\top a) - \frac{\eta^2}{2}s^2(a^\top H a). f ( x ) − f ( x + ) ≥ η s ( g ⊤ a ) − 2 η 2 s 2 ( a ⊤ H a ) .
Global L L L -smoothness bounds the difference error δ = s − g ⊤ a \delta= s - g^\top a δ = s − g ⊤ a by ∣ δ ∣ ≤ L μ ∥ a ∥ 2 / 2 |\delta| \le L\mu\lVert a\rVert^2/2 ∣ δ ∣ ≤ Lμ ∥ a ∥ 2 /2 . With γ = 1 / ( 4 ( 6 E + 13 ) ) \gamma= 1/(4(6E + 13)) γ = 1/ ( 4 ( 6 E + 13 )) , Young’s inequality gives s ( g ⊤ a ) ≥ ( 1 − γ / 2 ) ( g ⊤ a ) 2 − δ 2 / ( 2 γ ) s(g^\top a) \ge(1-\gamma/2)(g^\top a)^2-\delta^2/(2\gamma) s ( g ⊤ a ) ≥ ( 1 − γ /2 ) ( g ⊤ a ) 2 − δ 2 / ( 2 γ ) and s 2 ≤ ( 1 + γ ) ( g ⊤ a ) 2 + ( 1 + 1 / γ ) δ 2 s^2 \le(1+\gamma)(g^\top a)^2+(1+1/\gamma)\delta^2 s 2 ≤ ( 1 + γ ) ( g ⊤ a ) 2 + ( 1 + 1/ γ ) δ 2 . Taking expectations and using Lemma B.9 with tr H ≤ L E \operatorname{tr} H \le LE tr H ≤ L E and g ⊤ H g ≤ L ∥ g ∥ 2 g^\top H g \le L\lVert g\rVert^2 g ⊤ H g ≤ L ∥ g ∥ 2 ,
E [ f ( x ) − f ( x + ) ∣ x ] ≥ 1 − 2 γ 2 M ∥ g ∥ 2 − 1 2 γ M ( E [ δ 2 ] + 1 + γ M E [ δ 2 ( a ⊤ H a ) ] ) . \mathbb{E}[f(x)-f(x^+)\mid x] \ge\frac{1-2\gamma}{2M}\lVert g\rVert^2-\frac{1}{2\gamma M}\left(\mathbb{E}[\delta^2]+\frac{1+\gamma}{M}\mathbb{E}[\delta^2(a^\top H a)]\right). E [ f ( x ) − f ( x + ) ∣ x ] ≥ 2 M 1 − 2 γ ∥ g ∥ 2 − 2 γ M 1 ( E [ δ 2 ] + M 1 + γ E [ δ 2 ( a ⊤ H a )] ) .
By Lemma B.9 and a ⊤ H a ≤ L ∥ a ∥ 2 a^\top H a \le L\lVert a\rVert^2 a ⊤ H a ≤ L ∥ a ∥ 2 , E [ δ 2 ] ≤ L 2 μ 2 d ( d + 2 ) / 4 < ε 2 / ( 64 ( 6 E + 13 ) 2 ) \mathbb{E}[\delta^2] \le L^2\mu^2d(d+2)/4 < \varepsilon^2/(64(6E+13)^2) E [ δ 2 ] ≤ L 2 μ 2 d ( d + 2 ) /4 < ε 2 / ( 64 ( 6 E + 13 ) 2 ) and E [ δ 2 ( a ⊤ H a ) ] ≤ L 3 μ 2 d ( d + 2 ) ( d + 4 ) / 4 = L ε 2 / ( 64 ( 6 E + 13 ) 2 ) \mathbb{E}[\delta^2(a^\top H a)] \le L^3\mu^2d(d+2)(d+4)/4 = L\varepsilon^2/(64(6E+13)^2) E [ δ 2 ( a ⊤ H a )] ≤ L 3 μ 2 d ( d + 2 ) ( d + 4 ) /4 = L ε 2 / ( 64 ( 6 E + 13 ) 2 ) . Since L / M ≤ 1 / 3 L/M \le1/3 L / M ≤ 1/3 and γ ≤ 1 / 76 \gamma\le1/76 γ ≤ 1/76 , the term in parentheses is below 2 ε 2 / ( 64 ( 6 E + 13 ) 2 ) 2\varepsilon^2/(64(6E+13)^2) 2 ε 2 / ( 64 ( 6 E + 13 ) 2 ) , so the subtracted term is below ε 2 / ( 16 M ( 6 E + 13 ) ) \varepsilon^2/(16M(6E+13)) ε 2 / ( 16 M ( 6 E + 13 )) . For ∥ g ∥ > ε \lVert g\rVert> \varepsilon ∥ g ∥ > ε , as 2 γ = 1 / ( 2 ( 6 E + 13 ) ) 2\gamma= 1/(2(6E+13)) 2 γ = 1/ ( 2 ( 6 E + 13 )) , this gives
E [ f ( x ) − f ( x + ) ∣ x ] > ( 1 − 1 6 E + 13 ) ε 2 2 M = 3 ε 2 L ( 6 E + 13 ) . \mathbb{E}[f(x)-f(x^+)\mid x] > \left(1-\frac{1}{6E+13}\right)\frac{\varepsilon^2}{2M} = \frac{3\varepsilon^2}{L(6E+13)}. E [ f ( x ) − f ( x + ) ∣ x ] > ( 1 − 6 E + 13 1 ) 2 M ε 2 = L ( 6 E + 13 ) 3 ε 2 .
and every realized update is nonnegative. Summing over all steps and using f ( 0 ) − inf f ≤ Δ f(0)-\inf f \le\Delta f ( 0 ) − inf f ≤ Δ , the expected number of indices with gradient norm above ε \varepsilon ε is less than ( 6 E + 13 ) T / 3 ≤ N / 3 (6E+13)T/3 \le N/3 ( 6 E + 13 ) T /3 ≤ N /3 , which gives the failure bound under uniform output. Each step queries f ( x t + μ a t ) f(x_t+\mu a_t) f ( x t + μ a t ) and f ( y t ) f(y_t) f ( y t ) and reuses the cached value f ( x t ) f(x_t) f ( x t ) .
Gaussian two-point methods without an acceptance test
We now analyze the Gaussian two-point method of Nesterov and Spokoiny [41 ] , the estimator behind zeroth-order fine-tuning (Malladi et al., [38 ] ), with q q q directions per step and no acceptance Algorithm 6 Gaussian two-point method with q q q directions per step and no acceptance test. Theorem B.12 gives η , μ , N \eta,\mu,N η , μ , N on F \mathcal{F} F , and Theorem B.15 gives them on B \mathcal{B} B .
Require: q , η , μ , N q,\eta,\mu,N q , η , μ , N , the choice of forward or central differences, and exact values of f f f
x 0 ← 0 x_0 \leftarrow0 x 0 ← 0
for t = 0 , … , N − 1 t=0,\ldots,N-1 t = 0 , … , N − 1 do
draw a t , 1 , … , a t , q ∼ N ( 0 , I d ) a_{t,1},\ldots,a_{t,q} \sim N(0,I_d) a t , 1 , … , a t , q ∼ N ( 0 , I d ) independently
for i = 1 , … , q i=1,\ldots,q i = 1 , … , q do
s t , i ← [ f ( x t + μ a t , i ) − f ( x t ) ] / μ s_{t,i} \leftarrow[f(x_t+\mu a_{t,i})-f(x_t)]/\mu s t , i ← [ f ( x t + μ a t , i ) − f ( x t )] / μ (forward) or [ f ( x t + μ a t , i ) − f ( x t − μ a t , i ) ] / ( 2 μ ) [f(x_t+\mu a_{t,i})-f(x_t-\mu a_{t,i})]/(2\mu) [ f ( x t + μ a t , i ) − f ( x t − μ a t , i )] / ( 2 μ ) (central)
end for
x t + 1 ← x t − η q ∑ i = 1 q s t , i a t , i x_{t+1} \leftarrow x_t-\frac{\eta}{q}\sum_{i=1}^{q}s_{t,i}a_{t,i} x t + 1 ← x t − q η ∑ i = 1 q s t , i a t , i
end for
draw J J J uniformly from { 0 , … , N − 1 } \{0,\ldots,N-1\} { 0 , … , N − 1 } , independently, and return x J x_J x J
test. For q ≥ 1 q \ge1 q ≥ 1 , let a t , 1 , … , a t , q a_{t,1},\ldots,a_{t,q} a t , 1 , … , a t , q be independent standard Gaussian vectors in R d \mathbb{R}^d R d , fresh at each iteration. We consider either the forward differences
s t , i f = f ( x t + μ a t , i ) − f ( x t ) μ , (16) s_{t,i}^{\mathrm{f}}=\frac{f(x_t+\mu a_{t,i})-f(x_t)}{\mu}, \tag*{(16)} s t , i f = μ f ( x t + μ a t , i ) − f ( x t ) , (16)
or the central differences
s t , i c = f ( x t + μ a t , i ) − f ( x t − μ a t , i ) 2 μ . (17) s_{t,i}^{\mathrm{c}}=\frac{f(x_t+\mu a_{t,i})-f(x_t-\mu a_{t,i})}{2\mu}. \tag*{(17)} s t , i c = 2 μ f ( x t + μ a t , i ) − f ( x t − μ a t , i ) . (17)
In either case the update is
x t + 1 = x t − η q ∑ i = 1 q s t , i a t , i , x 0 = 0. (18) x_{t+1}=x_t-\frac{\eta}{q}\sum_{i=1}^{q}s_{t,i}a_{t,i},\qquad x_0=0. \tag*{(18)} x t + 1 = x t − q η i = 1 ∑ q s t , i a t , i , x 0 = 0. (18)
There is no test or rejection of x t + 1 x_{t+1} x t + 1 . The output is x J x_J x J for an independent J J J uniform on { 0 , … , N − 1 } \{0,\ldots,N-1\} { 0 , … , N − 1 } (Algorithm 6). For forward differences, f ( x t ) f(x_t) f ( x t ) is shared among the directions, so each iteration costs q + 1 q+1 q + 1 values; for central differences it costs 2 q 2q 2 q values. Both costs are at most 2 q 2q 2 q . In particular, no evaluation of f ( x t + 1 ) f(x_{t+1}) f ( x t + 1 ) is required in the current iteration.
Lemma B.11 (Two-point errors and batch Gaussian curvature). Suppose f f f is globally L L L -smooth. For either difference in (16)–(17), writing g = ∇ f ( x ) g=\nabla f(x) g = ∇ f ( x ) , σ i = a i ⊤ g \sigma_i=a_i^\top g σ i = a i ⊤ g , and s i = σ i + β i s_i=\sigma_i+\beta_i s i = σ i + β i ,
∣ β i ∣ ≤ L μ 2 ∥ a i ∥ 2 . (19) |\beta_i|\leq\frac{L\mu}{2}\|a_i\|^2. \tag*{(19)} ∣ β i ∣ ≤ 2 Lμ ∥ a i ∥ 2 . (19) Define v = q − 1 ∑ i σ i a i v=q^{-1}\sum_i\sigma_i a_i v = q − 1 ∑ i σ i a i and e = q − 1 ∑ i β i a i e=q^{-1}\sum_i\beta_i a_i e = q − 1 ∑ i β i a i , and write D 6 = d ( d + 2 ) ( d + 4 ) D_6=d(d+2)(d+4) D 6 = d ( d + 2 ) ( d + 4 ) . Then
E v = g , E ∥ e ∥ 2 ≤ L 2 μ 2 D 6 4 . (20) \mathbb{E}v=g,\qquad\mathbb{E}\|e\|^2\leq\frac{L^2\mu^2D_6}{4}. \tag*{(20)} E v = g , E ∥ e ∥ 2 ≤ 4 L 2 μ 2 D 6 . (20) If also H ⪰ 0 H\succeq0 H ⪰ 0 , ∥ H ∥ o p ≤ L \|H\|_{\mathrm{op}}\leq L ∥ H ∥ op ≤ L , and tr H ≤ L E \operatorname{tr}H\leq LE tr H ≤ L E , then
E [ v ⊤ H v ] ≤ L ( E + q + 1 ) q ∥ g ∥ 2 . (21) \mathbb{E}[v^\top Hv]\leq\frac{L(E+q+1)}{q}\|g\|^2. \tag*{(21)} E [ v ⊤ H v ] ≤ q L ( E + q + 1 ) ∥ g ∥ 2 . (21)
Proof. For forward differences integrate ( ∇ f ( x + t a i ) − g ) ⊤ a i (\nabla f(x+ta_i)-g)^\top a_i ( ∇ f ( x + t a i ) − g ) ⊤ a i over t ∈ [ 0 , μ ] t\in[0,\mu] t ∈ [ 0 , μ ] ; for central differences integrate over [ − μ , μ ] [-\mu,\mu] [ − μ , μ ] and divide by 2 μ 2\mu 2 μ . Global smoothness gives (19) in either case. Jensen’s inequality applied to the average defining e e e , followed by E ∥ a ∥ 6 = D 6 \mathbb{E}\|a\|^6=D_6 E ∥ a ∥ 6 = D 6 , proves (20). The identity E [ ( a ⊤ g ) a ] = g \mathbb{E}[(a^{\top}g)a] = g E [( a ⊤ g ) a ] = g proves the other part of (20) . For the last claim expand the quadratic form of v v v . The q q q diagonal contributions are bounded, using Lemma B.9, by ∥ g ∥ 2 tr H + 2 g ⊤ H g ≤ L ( E + 2 ) ∥ g ∥ 2 \lVert g\rVert^2\operatorname{tr} H + 2g^{\top}Hg \le L(E+2)\lVert g\rVert^2 ∥ g ∥ 2 tr H + 2 g ⊤ H g ≤ L ( E + 2 ) ∥ g ∥ 2 . By independence, each of the q ( q − 1 ) q(q-1) q ( q − 1 ) off-diagonal contributions is g ⊤ H g ≤ L ∥ g ∥ 2 g^{\top}Hg \le L\lVert g\rVert^2 g ⊤ H g ≤ L ∥ g ∥ 2 . Division by q 2 q^2 q 2 gives (21) .
Theorem B.12 (Gaussian two-point method on F \mathcal{F} F ). Let f ∈ F L , Δ , E d f \in\mathcal{F}^{d}_{L,\Delta,E} f ∈ F L , Δ , E d and 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , T = L Δ / ε 2 ≥ 1 T = L\Delta/\varepsilon^2 \ge1 T = L Δ/ ε 2 ≥ 1 . For any integer q ≥ 1 q \ge1 q ≥ 1 , run (18) with either (16) or (17) , using
η = q L ( E + q + 1 ) , μ = ε 8 L d ( d + 2 ) ( d + 4 ) , N = ⌈ 17 ( E + q + 1 ) T q ⌉ . (22) \eta= \frac{q}{L(E+q+1)}, \qquad\mu= \frac{\varepsilon}{8L\sqrt{d(d+2)(d+4)}}, \qquad N = \left\lceil\frac{17(E+q+1)T}{q} \right\rceil. \tag*{(22)} η = L ( E + q + 1 ) q , μ = 8 L d ( d + 2 ) ( d + 4 ) ε , N = ⌈ q 17 ( E + q + 1 ) T ⌉ . (22) Then on each function
P ( ∥ ∇ f ( x J ) ∥ > ε ) ≤ 11 34 < 1 3 , Q ≤ 2 q N ≤ 34 ( E + q + 1 ) T + 2 q . (23) \mathbb{P}(\lVert\nabla f(x_J)\rVert> \varepsilon) \le\frac{11}{34} < \frac{1}{3}, \qquad Q \le2qN \le34(E+q+1)T+2q. \tag*{(23)} P (∥ ∇ f ( x J )∥ > ε ) ≤ 34 11 < 3 1 , Q ≤ 2 q N ≤ 34 ( E + q + 1 ) T + 2 q . (23) In particular, with one direction per step (q = 1 q=1 q = 1 ) the method uses at most 34 ( E + 2 ) T + 2 34(E+2)T+2 34 ( E + 2 ) T + 2 value calls.
Proof. Let C = L ( E + q + 1 ) / q = 1 / η C = L(E+q+1)/q = 1/\eta C = L ( E + q + 1 ) / q = 1/ η . The global quadratic upper model (3) applied with the random displacement − η ( v + e ) -\eta(v+e) − η ( v + e ) implies, conditionally on x x x ,
E [ f ( x ) − f ( x − η ( v + e ) ) ] ≥ η ∥ g ∥ 2 − η ∥ g ∥ E ∥ e ∥ − η 2 2 E [ ( v + e ) ⊤ H ( v + e ) ] . \mathbb{E}[f(x)-f(x-\eta(v+e))] \ge\eta\lVert g\rVert^2-\eta\lVert g\rVert\mathbb{E}\lVert e\rVert-\frac{\eta^2}{2}\mathbb{E}[(v+e)^{\top}H(v+e)]. E [ f ( x ) − f ( x − η ( v + e ))] ≥ η ∥ g ∥ 2 − η ∥ g ∥ E ∥ e ∥ − 2 η 2 E [( v + e ) ⊤ H ( v + e )] .
Since H ⪰ 0 H \succeq0 H ⪰ 0 , the matrix Young inequality with parameter 1 / 8 1/8 1/8 gives
( v + e ) ⊤ H ( v + e ) ≤ 9 8 v ⊤ H v + 9 e ⊤ H e . (v+e)^{\top}H(v+e) \le\frac{9}{8}v^{\top}Hv+9e^{\top}He. ( v + e ) ⊤ H ( v + e ) ≤ 8 9 v ⊤ H v + 9 e ⊤ H e .
Put κ = L μ D 6 / 2 = ε / 16 \kappa= L\mu\sqrt{D_6}/2 = \varepsilon/16 κ = Lμ D 6 /2 = ε /16 . By Lemma B.11, Jensen and ∥ H ∥ o p ≤ L \lVert H\rVert_{\mathrm{op}} \le L ∥ H ∥ op ≤ L ,
E ∥ e ∥ ≤ κ , E [ e ⊤ H e ] ≤ L κ 2 , E [ v ⊤ H v ] ≤ C ∥ g ∥ 2 . \mathbb{E}\lVert e\rVert\le\kappa,\qquad\mathbb{E}[e^{\top}He] \le L\kappa^2,\qquad\mathbb{E}[v^{\top}Hv] \le C\lVert g\rVert^2. E ∥ e ∥ ≤ κ , E [ e ⊤ H e ] ≤ L κ 2 , E [ v ⊤ H v ] ≤ C ∥ g ∥ 2 .
Since κ ∥ g ∥ ≤ ∥ g ∥ 2 / 16 + 4 κ 2 \kappa\lVert g\rVert\le\lVert g\rVert^2/16+4\kappa^2 κ ∥ g ∥ ≤ ∥ g ∥ 2 /16 + 4 κ 2 and η L = q / ( E + q + 1 ) ≤ 1 \eta L=q/(E+q+1)\le1 η L = q / ( E + q + 1 ) ≤ 1 ,
E [ f ( x ) − f ( x t + 1 ) ∣ x t = x ] ≥ η ( 7 16 ∥ g ∥ 2 − κ ∥ g ∥ − 9 2 η L κ 2 ) ≥ η ( 3 8 ∥ g ∥ 2 − 17 2 κ 2 ) ≥ η ( 3 8 ∥ g ∥ 2 − 1 16 ε 2 ) . \begin{aligned}
\mathbb{E}[f(x)-f(x_{t+1})\mid x_t=x]
&\ge\eta\left(\frac{7}{16}\lVert g\rVert^2-\kappa\lVert g\rVert-\frac{9}{2}\eta L\kappa^2\right) \\
&\ge\eta\left(\frac{3}{8}\lVert g\rVert^2-\frac{17}{2}\kappa^2\right) \\
&\ge\eta\left(\frac{3}{8}\lVert g\rVert^2-\frac{1}{16}\varepsilon^2\right).
\end{aligned} E [ f ( x ) − f ( x t + 1 ) ∣ x t = x ] ≥ η ( 16 7 ∥ g ∥ 2 − κ ∥ g ∥ − 2 9 η L κ 2 ) ≥ η ( 8 3 ∥ g ∥ 2 − 2 17 κ 2 ) ≥ η ( 8 3 ∥ g ∥ 2 − 16 1 ε 2 ) .
where 17 / 512 ≤ 1 / 16 17/512 \le1/16 17/512 ≤ 1/16 is the last numerical inequality. Summing the conditional expectations and using E f ( x N ) ≥ inf f ≥ f ( 0 ) − Δ \mathbb{E}f(x_N) \ge\inf f \ge f(0)-\Delta E f ( x N ) ≥ inf f ≥ f ( 0 ) − Δ yields
1 N ∑ t = 0 N − 1 E ∥ ∇ f ( x t ) ∥ 2 ≤ 8 Δ 3 η N + ε 2 6 . \frac{1}{N}\sum_{t=0}^{N-1}\mathbb{E}\lVert\nabla f(x_t)\rVert^2 \le\frac{8\Delta}{3\eta N}+\frac{\varepsilon^2}{6}. N 1 t = 0 ∑ N − 1 E ∥ ∇ f ( x t ) ∥ 2 ≤ 3 η N 8Δ + 6 ε 2 .
By the independence and uniform distribution of J J J , Markov’s inequality and the choice of N N N imply
P ( ∥ ∇ f ( x J ) ∥ > ε ) ≤ 8 3 ( E + q + 1 ) T q N + 1 6 ≤ 8 51 + 1 6 = 11 34 . \mathbb{P}(\lVert\nabla f(x_J)\rVert>\varepsilon) \le\frac{8}{3}\frac{(E+q+1)T}{qN}+\frac{1}{6}\le\frac{8}{51}+\frac{1}{6}=\frac{11}{34}. P (∥ ∇ f ( x J )∥ > ε ) ≤ 3 8 q N ( E + q + 1 ) T + 6 1 ≤ 51 8 + 6 1 = 34 11 .
Every iteration makes the stated number of value calls whatever the directions, so the cap holds on every trajectory.
The first bound follows from Laurent and Massart [34 ] .
Lemma B.13 (Gaussian quadratic tail). Let a ∼ N ( 0 , I d ) a \sim N(0,I_d) a ∼ N ( 0 , I d ) and M ⪰ 0 M \succeq0 M ⪰ 0 , ∥ M ∥ op ≤ L \lVert M\rVert_{\operatorname{op}} \le L ∥ M ∥ op ≤ L , tr M ≤ L r \operatorname{tr} M \le Lr tr M ≤ L r , with r > 0 r>0 r > 0 . For u > 0 u>0 u > 0 ,
P ( a T M a > L ( r + 2 r u + 2 u ) ) ≤ e − u . (24) \mathbb{P}\left(a^{\mathsf{T}}Ma>L(r+2\sqrt{ru}+2u)\right)\le e^{-u}. \tag*{(24)} P ( a T M a > L ( r + 2 r u + 2 u ) ) ≤ e − u . (24) Consequently, if tr H ≤ L E \operatorname{tr} H\le LE tr H ≤ L E , then
P ( a T H a > L ( 2 E + 3 u ) ) ≤ e − u , P ( ∥ a ∥ 2 > 2 d + 3 u ) ≤ e − u . (25) \mathbb{P}\left(a^{\mathsf{T}}Ha>L(2E+3u)\right)\le e^{-u},\qquad\mathbb{P}\left(\lVert a\rVert^2>2d+3u\right)\le e^{-u}. \tag*{(25)} P ( a T H a > L ( 2 E + 3 u ) ) ≤ e − u , P ( ∥ a ∥ 2 > 2 d + 3 u ) ≤ e − u . (25)
Proof. Diagonalize M / L M/L M / L , with eigenvalues w j ∈ [ 0 , 1 ] w_j\in[0,1] w j ∈ [ 0 , 1 ] and ∑ j w j ≤ r \sum_j w_j\le r ∑ j w j ≤ r . For independent Z j ∼ N ( 0 , 1 ) Z_j\sim N(0,1) Z j ∼ N ( 0 , 1 ) and 0 < λ < 1 / 2 0<\lambda<1/2 0 < λ < 1/2 ,
log E exp ( λ ∑ j w j ( Z j 2 − 1 ) ) = ∑ j [ − λ w j − 1 2 log ( 1 − 2 λ w j ) ] ≤ λ 2 ∑ j w j 2 1 − 2 λ ≤ r λ 2 1 − 2 λ . \begin{aligned}
\log\mathbb{E}\exp\left(\lambda\sum_j w_j(Z_j^2-1)\right)=\sum_j\left[-\lambda w_j-\frac{1}{2}\log(1-2\lambda w_j)\right] \\
&\le\frac{\lambda^2\sum_j w_j^2}{1-2\lambda}\le\frac{r\lambda^2}{1-2\lambda}.
\end{aligned} log E exp ( λ j ∑ w j ( Z j 2 − 1 ) ) = j ∑ [ − λ w j − 2 1 log ( 1 − 2 λ w j ) ] ≤ 1 − 2 λ λ 2 ∑ j w j 2 ≤ 1 − 2 λ r λ 2 .
The first inequality follows by expanding − log ( 1 − z ) − z -\log(1-z)-z − log ( 1 − z ) − z into its nonnegative Taylor series and bounding it by z 2 / [ 2 ( 1 − z ) ] z^2/[2(1-z)] z 2 / [ 2 ( 1 − z )] , for 0 ≤ z < 1 0\le z<1 0 ≤ z < 1 . Apply the exponential Markov inequality at λ = u / ( r + 2 u ) \lambda=\sqrt{u}/(\sqrt{r}+2\sqrt{u}) λ = u / ( r + 2 u ) ; the resulting exponent at 2 r u + 2 u 2\sqrt{ru}+2u 2 r u + 2 u is − u -u − u . Adding the mean, at most r r r , proves (24) . For the first claim in (25) use 2 E u ≤ E + u 2\sqrt{Eu}\le E+u 2 E u ≤ E + u ; for the second set M = I d M=I_d M = I d , L = 1 L=1 L = 1 , r = d r=d r = d and apply the same inequality. □ \square □
Lemma B.14 (Capped descent for a truncated Gaussian batch). Let f ∈ B L , Δ , E d f\in\mathcal{B}^{d}_{L,\Delta,E} f ∈ B L , Δ , E d , let H H H be its fixed upper matrix, and let x x x satisfy f ( x ) ≤ f ( 0 ) f(x)\le f(0) f ( x ) ≤ f ( 0 ) . Fix Λ > 0 \Lambda>0 Λ > 0 and draw a 1 , … , a q a_1,\ldots,a_q a 1 , … , a q independently from N ( 0 , I d ) N(0,I_d) N ( 0 , I d ) . Set
v = 1 q ∑ i = 1 q ( a i T ∇ f ( x ) ) a i 1 { a i T H a i ≤ Λ } , η = Λ − 1 , (26) v=\frac{1}{q}\sum_{i=1}^{q}(a_i^{\mathsf{T}}\nabla f(x))a_i\mathbf{1}_{\{a_i^{\mathsf{T}}Ha_i\le\Lambda\}},\qquad\eta=\Lambda^{-1}, \tag*{(26)} v = q 1 i = 1 ∑ q ( a i T ∇ f ( x )) a i 1 { a i T H a i ≤ Λ } , η = Λ − 1 , (26) Then f ( x − η v ) ≤ f ( x ) f(x-\eta v)\le f(x) f ( x − η v ) ≤ f ( x ) and, pathwise,
f ( x ) − f ( x − η v ) ≥ η 2 q ∑ i = 1 q ( a i T ∇ f ( x ) ) 2 1 { a i T H a i ≤ Λ } . (27) f(x)-f(x-\eta v)\ge\frac{\eta}{2q}\sum_{i=1}^{q}(a_i^{\mathsf{T}}\nabla f(x))^2\mathbf{1}_{\{a_i^{\mathsf{T}}Ha_i\le\Lambda\}}. \tag*{(27)} f ( x ) − f ( x − η v ) ≥ 2 q η i = 1 ∑ q ( a i T ∇ f ( x ) ) 2 1 { a i T H a i ≤ Λ } . (27) If P ( a T H a > Λ ) ≤ 1 / 48 \mathbb{P}(a^{\mathsf{T}}Ha>\Lambda)\le1/48 P ( a T H a > Λ ) ≤ 1/48 , then
E [ f ( x ) − f ( x − η v ) ∣ x ] ≥ 3 η 8 ∥ ∇ f ( x ) ∥ 2 . (28) \mathbb{E}[f(x)-f(x-\eta v)\mid x]\ge\frac{3\eta}{8}\lVert\nabla f(x)\rVert^2. \tag*{(28)} E [ f ( x ) − f ( x − η v ) ∣ x ] ≥ 8 3 η ∥ ∇ f ( x ) ∥ 2 . (28)
Proof. Write g = ∇ f ( x ) g=\nabla f(x) g = ∇ f ( x ) , σ i = a i T g \sigma_i=a_i^{\mathsf{T}}g σ i = a i T g and I = { i : a i T H a i ≤ Λ } I=\{i:a_i^{\mathsf{T}}Ha_i\le\Lambda\} I = { i : a i T H a i ≤ Λ } . Cauchy–Schwarz in the H H H seminorm, followed by scalar Cauchy–Schwarz, yields
v T H v ≤ 1 q 2 ( ∑ i ∈ I ∣ σ i ∣ a i T H a i ) 2 ≤ Λ q ∑ i ∈ I σ i 2 = Λ g T v . v^{\mathsf{T}}Hv\le\frac{1}{q^2}\left(\sum_{i\in I}|\sigma_i|\sqrt{a_i^{\mathsf{T}}Ha_i}\right)^2\le\frac{\Lambda}{q}\sum_{i\in I}\sigma_i^2=\Lambda g^{\mathsf{T}}v. v T H v ≤ q 2 1 ( i ∈ I ∑ ∣ σ i ∣ a i T H a i ) 2 ≤ q Λ i ∈ I ∑ σ i 2 = Λ g T v .
Therefore, by the capped upper model in Lemma B.3 ,
min { f ( x − η v ) , f ( 0 ) } ≤ f ( x ) − η g T v + η 2 2 v T H v ≤ f ( x ) − η 2 g T v . \min\{f(x-\eta v),f(0)\}\le f(x)-\eta g^{\mathsf{T}}v+\frac{\eta^2}{2}v^{\mathsf{T}}Hv\le f(x)-\frac{\eta}{2}g^{\mathsf{T}}v. min { f ( x − η v ) , f ( 0 )} ≤ f ( x ) − η g T v + 2 η 2 v T H v ≤ f ( x ) − 2 η g T v .
If g T v > 0 g^{\mathsf{T}}v>0 g T v > 0 the right side is strictly below f ( 0 ) f(0) f ( 0 ) , so the minimum must be the actual value. If g T v = 0 g^{\mathsf{T}}v=0 g T v = 0 , then every retained σ i \sigma_i σ i vanishes, v = 0 v=0 v = 0 , and the assertion is immediate. This proves (27) . For σ = a T g \sigma=a^{\mathsf{T}}g σ = a T g and p = P ( a T H a > Λ ) p=\mathbb{P}(a^{\mathsf{T}}Ha>\Lambda) p = P ( a T H a > Λ ) , Cauchy–Schwarz and the normal fourth moment give
E [ σ 2 1 { a T H a > Λ } ] ≤ E σ 4 p = 3 p ∥ g ∥ 2 ≤ 1 4 ∥ g ∥ 2 . \mathbb{E}[\sigma^2\mathbf{1}_{\{a^{\mathsf{T}}Ha>\Lambda\}}]\le\sqrt{\mathbb{E}\sigma^4p}=\sqrt{3p}\lVert g\rVert^2\le\frac{1}{4}\lVert g\rVert^2. E [ σ 2 1 { a T H a > Λ } ] ≤ E σ 4 p = 3 p ∥ g ∥ 2 ≤ 4 1 ∥ g ∥ 2 .
Independence and the preceding pathwise inequality yield (28) . □ \square □
Theorem B.15 (Gaussian two-point method on B \mathcal{B} B ). Let f ∈ B L , Δ , E d f \in\mathcal{B}^{d}_{L,\Delta,E} f ∈ B L , Δ , E d , 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d and T = L Δ / ε 2 ≥ 1 T = L\Delta/\varepsilon^{2} \ge1 T = L Δ/ ε 2 ≥ 1 . For any integer q ≥ 1 q \ge1 q ≥ 1 define the public parameters
W = E + log ( 1000 q E T ) , N = ⌈ 256 T W ⌉ , u = log ( 48 q N ) , m = 2 E + 3 u , R = 2 d + 3 u , η = ( L m ) − 1 , μ = ε 4 L R ( 1 + η L R ) N . (29) \begin{aligned}
W &= E + \log(1000qET), & N &= \lceil256TW\rceil, \\
u &= \log(48qN), & m &= 2E + 3u, \\
R &= 2d + 3u, & \eta&= (Lm)^{-1}, \\
\mu&= \frac{\varepsilon}{4L\sqrt{R}(1+\eta LR)^{N}}. \tag*{(29)}
\end{aligned} W u R μ = E + log ( 1000 q E T ) , = log ( 48 q N ) , = 2 d + 3 u , = 4 L R ( 1 + η L R ) N ε . N m η = ⌈ 256 T W ⌉ , = 2 E + 3 u , = ( L m ) − 1 , (29) Run the update (18) , without an acceptance test, with this fixed η , μ , N \eta,\mu,N η , μ , N and either forward or central differences. Then, for every such f f f ,
P ( ∥ ∇ f ( x J ) ∥ > ε ) ≤ 13 56 < 1 3 , Q ≤ 2 q N ≤ 512 q T ( E + log ( 1000 q E T ) ) + 2 q . (30) \mathbb{P}\left(\lVert\nabla f(x_J)\rVert> \varepsilon\right) \le\frac{13}{56} < \frac{1}{3}, \qquad Q \le2qN \le512qT\left(E+\log(1000qET)\right)+2q. \tag*{(30)} P ( ∥ ∇ f ( x J )∥ > ε ) ≤ 56 13 < 3 1 , Q ≤ 2 q N ≤ 512 q T ( E + log ( 1000 q E T ) ) + 2 q . (30) In particular, for q = 1 q=1 q = 1 this is an O ( E T + T log ( E T ) ) O(ET+T\log(ET)) O ( E T + T log ( E T )) value bound. For q = 1 q=1 q = 1 and E ≥ log ( 1000 E T ) E\ge\log(1000ET) E ≥ log ( 1000 E T ) , the chosen step size has order 1 / ( L E ) 1/(LE) 1/ ( L E ) and the call bound is O ( E T ) O(ET) O ( E T ) , with absolute constants.
Proof. Let H H H be the unknown matrix witnessing membership in B \mathcal{B} B . Its use below is confined to the analysis; the algorithm’s parameters depend only on d , L , Δ , ε , E , q d,L,\Delta,\varepsilon,E,q d , L , Δ , ε , E , q . Since T W ≥ 1 TW\ge1 T W ≥ 1 , N ≤ 257 T W N\le257TW N ≤ 257 T W . By log ( 48 ⋅ 257 ) < 10 \log(48\cdot257)<10 log ( 48 ⋅ 257 ) < 10 , log ( q T ) ≤ log ( q E T ) ≤ W \log(qT)\le\log(qET)\le W log ( q T ) ≤ log ( q E T ) ≤ W , log W ≤ W \log W\le W log W ≤ W , and 10 ≤ 2 W 10\le2W 10 ≤ 2 W , we have
u ≤ 10 + log ( q T ) + log W ≤ 4 W , m = 2 E + 3 u ≤ 14 W . (31) u \le10+\log(qT)+\log W\le4W,\qquad m=2E+3u\le14W. \tag*{(31)} u ≤ 10 + log ( q T ) + log W ≤ 4 W , m = 2 E + 3 u ≤ 14 W . (31)
Lemma B.13 applied to H H H gives P ( a T H a > L m ) ≤ e − u = 1 / ( 48 q N ) \mathbb{P}(a^{\mathsf{T}}Ha>Lm)\le e^{-u}=1/(48qN) P ( a T H a > L m ) ≤ e − u = 1/ ( 48 q N ) .
Construct an auxiliary process z 0 = 0 z_0=0 z 0 = 0 , and, for t = 0 , … , N − 1 t=0,\ldots,N-1 t = 0 , … , N − 1 , define z t + 1 z_{t+1} z t + 1 by the truncated directional derivative step (26) with Λ = L m \Lambda=Lm Λ = L m and the same Gaussian directions as the actual algorithm. This process is only a proof device; the actual algorithm evaluates neither directional derivatives nor H H H . By Lemma B.14 , its iterates satisfy f ( z t ) ≤ f ( 0 ) f(z_t)\le f(0) f ( z t ) ≤ f ( 0 ) pathwise and
E [ f ( z t ) − f ( z t + 1 ) ∣ z t ] ≥ 3 η 8 ∥ ∇ f ( z t ) ∥ 2 . \mathbb{E}\left[f(z_t)-f(z_{t+1})\mid z_t\right]\ge\frac{3\eta}{8}\lVert\nabla f(z_t)\rVert^2. E [ f ( z t ) − f ( z t + 1 ) ∣ z t ] ≥ 8 3 η ∥ ∇ f ( z t ) ∥ 2 .
Telescoping and using the initial gap yield
P ( ∥ ∇ f ( z J ) ∥ > 7 ε 8 ) ≤ 512 m T 147 N ≤ 4 21 . (32) \mathbb{P}\left(\lVert\nabla f(z_J)\rVert>\frac{7\varepsilon}{8}\right)\le\frac{512mT}{147N}\le\frac{4}{21}. \tag*{(32)} P ( ∥ ∇ f ( z J )∥ > 8 7 ε ) ≤ 147 N 512 m T ≤ 21 4 . (32)
In the first inequality we used the second-moment bound N − 1 ∑ t E ∥ ∇ f ( z t ) ∥ 2 ≤ 8 Δ / ( 3 η N ) N^{-1}\sum_t\mathbb{E}\lVert\nabla f(z_t)\rVert^2\le8\Delta/(3\eta N) N − 1 ∑ t E ∥ ∇ f ( z t ) ∥ 2 ≤ 8Δ/ ( 3 η N ) and Markov’s inequality; the last inequality follows from (31) and N ≥ 256 T W N\ge256TW N ≥ 256 T W .
Let G \mathcal{G} G be the event that, for every one of the q N qN q N generated directions,
a t , i T H a t , i ≤ L m , ∥ a t , i ∥ 2 ≤ R . (33) a_{t,i}^{\mathsf{T}}Ha_{t,i}\le Lm,\qquad\lVert a_{t,i}\rVert^2\le R. \tag*{(33)} a t , i T H a t , i ≤ L m , ∥ a t , i ∥ 2 ≤ R . (33)
Both Gaussian bounds in (25) and a union bound imply P ( G c ) ≤ 1 / 24 \mathbb{P}(\mathcal{G}^{\mathsf{c}})\le1/24 P ( G c ) ≤ 1/24 . On G \mathcal{G} G , none of the auxiliary directions is removed. We compare its entire trajectory to that of the actual finite radius algorithm. Write D t = ∥ x t − z t ∥ D_t=\lVert x_t-z_t\rVert D t = ∥ x t − z t ∥ and A t = q − 1 ∑ i a t , i a t , i T A_t=q^{-1}\sum_i a_{t,i}a_{t,i}^{\mathsf{T}} A t = q − 1 ∑ i a t , i a t , i T . On G \mathcal{G} G , ∥ A t ∥ o p ≤ R \lVert A_t\rVert_{\mathrm{op}}\le R ∥ A t ∥ op ≤ R and
∥ q − 1 ∑ i ( s t , i − a t , i T ∇ f ( x t ) ) a t , i ∥ ≤ L μ R 3 / 2 2 \left\lVert q^{-1}\sum_i(s_{t,i}-a_{t,i}^{\mathsf{T}}\nabla f(x_t))a_{t,i}\right\rVert\le\frac{L\mu R^{3/2}}{2} q − 1 i ∑ ( s t , i − a t , i T ∇ f ( x t )) a t , i ≤ 2 Lμ R 3/2
by (19) . Global L L L -smoothness of f f f gives
D t + 1 ≤ ( 1 + η L R ) D t + η L μ R 3 / 2 2 , D 0 = 0. D_{t+1} \le(1+\eta LR)D_t+\frac{\eta L\mu R^{3/2}}{2},\qquad D_0=0. D t + 1 ≤ ( 1 + η L R ) D t + 2 η Lμ R 3/2 , D 0 = 0.
Summing the geometric recursion,
max 0 ≤ t ≤ N D t ≤ μ R 2 ( ( 1 + η L R ) N − 1 ) ≤ ε 8 L . (34) \max_{0\le t\le N}D_t\le\frac{\mu\sqrt{R}}{2}\left((1+\eta LR)^N-1\right)\le\frac{\varepsilon}{8L}. \tag*{(34)} 0 ≤ t ≤ N max D t ≤ 2 μ R ( ( 1 + η L R ) N − 1 ) ≤ 8 L ε . (34)
Thus, on G \mathcal{G} G , ∥ ∇ f ( x J ) ∥ ≤ ∥ ∇ f ( z J ) ∥ + ε / 8 \lVert\nabla f(x_J)\rVert\le\lVert\nabla f(z_J)\rVert+\varepsilon/8 ∥ ∇ f ( x J )∥ ≤ ∥ ∇ f ( z J )∥ + ε /8 . Combining this deterministic implication with (32) and the union bound,
P ( ∥ ∇ f ( x J ) ∥ > ε ) ≤ 4 21 + 1 24 = 13 56 < 1 3 . \mathbb{P}(\lVert\nabla f(x_J)\rVert>\varepsilon)\le\frac{4}{21}+\frac{1}{24}=\frac{13}{56}<\frac{1}{3}. P (∥ ∇ f ( x J )∥ > ε ) ≤ 21 4 + 24 1 = 56 13 < 3 1 .
Exactly N N N iterations are run, and each consumes at most 2 q 2q 2 q calls. Finally, N ≤ 256 T W + 1 N\le256TW+1 N ≤ 256 T W + 1 gives the asserted call cap.
The radius μ \mu μ in (29) is exponentially small in N N N . Exact values allow it, and the proof needs it: with a nonzero radius a step can raise f f f slightly even when a ⊤ H a a^\top Ha a ⊤ H a is small, so the iterates are compared with the auxiliary process along the whole run. For E < log ( 1000 E T ) E<\log(1000ET) E < log ( 1000 E T ) , whether a step of order 1 / ( L E ) 1/(LE) 1/ ( L E ) without an acceptance test gives O ( E T ) O(ET) O ( E T ) calls on B \mathcal{B} B remains open. With the acceptance test, Theorem B.10 gives O ( E T ) O(ET) O ( E T ) ; its comparison uses the value f ( y t ) f(y_t) f ( y t ) , which the forward method without the test also queries as the next base value f ( x t + 1 ) f(x_{t+1}) f ( x t + 1 ) .
Direction Selection and the Stage Argument
Direction selection by finite partial coloring
The selection lemma uses the classical finite partial coloring argument ([50 ] ): quantize linear forms of sign vectors, find a large common fiber, and subtract two distant members.
Lemma C.1 (Finite entropy facts; see Cover and Thomas, [17 ] ). For a finitely supported probability mass function p p p , write H ( p ) = ∑ j : p j > 0 p j log ( 1 / p j ) H(p)=\sum_{j:p_j>0}p_j\log(1/p_j) H ( p ) = ∑ j : p j > 0 p j log ( 1/ p j ) . If reference masses q j > 0 q_j>0 q j > 0 on the support of p p p have total mass at most one, then H ( p ) ≤ ∑ j p j log ( 1 / q j ) H(p)\le\sum_jp_j\log(1/q_j) H ( p ) ≤ ∑ j p j log ( 1/ q j ) . Joint entropy is at most the sum of marginal entropies, and some atom has mass ≥ exp [ − H ( p ) ] \ge\exp[-H(p)] ≥ exp [ − H ( p )] .
Proof. Use log t ≤ t − 1 \log t\le t-1 log t ≤ t − 1 in ∑ p j > 0 p j log ( q j / p j ) ≤ ∑ p j > 0 q j − 1 ≤ 0 \sum_{p_j>0}p_j\log(q_j/p_j)\le\sum_{p_j>0}q_j-1\le0 ∑ p j > 0 p j log ( q j / p j ) ≤ ∑ p j > 0 q j − 1 ≤ 0 . For subadditivity choose the product of the marginal laws as reference; it is positive on the joint support. Finally H ( p ) ≥ log ( 1 / max j p j ) H(p)\ge\log(1/\max_jp_j) H ( p ) ≥ log ( 1/ max j p j ) .
Lemma C.2 (Strip selection). If S ⊂ R d S\subset\mathbb{R}^d S ⊂ R d has dimension p ≥ 1 p\ge1 p ≥ 1 and ∥ a i ∥ ≤ R \lVert a_i\rVert\le R ∥ a i ∥ ≤ R for 1 ≤ i ≤ M 1\le i\le M 1 ≤ i ≤ M , there is a unit u ∈ S u\in S u ∈ S such that
max i ≤ M ∣ ⟨ a i , u ⟩ ∣ ≤ 13 3 R log ( e + M / p ) p . (35) \max_{i\le M}\lvert\langle a_i,u\rangle\rvert\le\frac{13}{3}R\sqrt{\frac{\log(e+M/p)}{p}}. \tag*{(35)} i ≤ M max ∣⟨ a i , u ⟩∣ ≤ 3 13 R p log ( e + M / p ) . (35)
Proof. If M < p M<p M < p or R = 0 R=0 R = 0 , choose a unit vector u ∈ S u\in S u ∈ S orthogonal to the projections of a 1 , … , a M a_1,\ldots,a_M a 1 , … , a M onto S S S , giving max i ≤ M ∣ ⟨ a i , u ⟩ ∣ = 0 \max_{i\le M}\lvert\langle a_i,u\rangle\rvert=0 max i ≤ M ∣⟨ a i , u ⟩∣ = 0 . Assume now that s : = M / p ≥ 1 s:=M/p\ge1 s := M / p ≥ 1 and R > 0 R>0 R > 0 . Use orthonormal coordinates on S S S and project the a i a_i a i to S S S . Let X X X be uniform on { − 1 , 1 } p \{-1,1\}^p { − 1 , 1 } p ; set a = 7 2 log ( e + s ) a=\frac{7}{2}\log(e+s) a = 2 7 log ( e + s ) , t = 2 a t=\sqrt{2a} t = 2 a , Y i = ⟨ a i , X ⟩ / R Y_i=\langle a_i,X\rangle/R Y i = ⟨ a i , X ⟩ / R , and K i = ⌊ Y i / ( 2 t ) + 1 / 2 ⌋ K_i=\lfloor Y_i/(2t)+1/2\rfloor K i = ⌊ Y i / ( 2 t ) + 1/2 ⌋ . Since ( e + 1 ) 7 > e 9 (e+1)^7>e^9 ( e + 1 ) 7 > e 9 , we have a ≥ 7 2 log ( e + 1 ) > 9 2 a\ge\frac{7}{2}\log(e+1)>\frac{9}{2} a ≥ 2 7 log ( e + 1 ) > 2 9 . The inequality log cosh z ≤ z 2 / 2 \log\cosh z\le z^2/2 log cosh z ≤ z 2 /2 follows by integrating tanh z ≤ z \tanh z\le z tanh z ≤ z for z ≥ 0 z\ge0 z ≥ 0 and using evenness. Independence gives E e λ Y i ≤ e λ 2 / 2 \mathbb{E}e^{\lambda Y_i}\le e^{\lambda^2/2} E e λ Y i ≤ e λ 2 /2 , hence P ( ∣ Y i ∣ ≥ b ) ≤ 2 e − b 2 / 2 \mathbb{P}(\lvert Y_i\rvert\ge b)\le2e^{-b^2/2} P (∣ Y i ∣ ≥ b ) ≤ 2 e − b 2 /2 , as in Hoeffding’s inequality ([22 ] ). Choose an integer N N N bounding every value ∣ K i ( X ) ∣ \lvert K_i(X)\rvert ∣ K i ( X )∣ . Since ( 2 k − 1 ) 2 ≥ k (2k-1)^2 \ge k ( 2 k − 1 ) 2 ≥ k for every integer k ≥ 1 k \ge1 k ≥ 1 ,
P ( ∣ K i ∣ ≥ k ) ≤ 2 e − ( 2 k − 1 ) 2 a ≤ 2 e − k a , E ∣ K i ∣ = ∑ k = 1 N P ( ∣ K i ∣ ≥ k ) < 2 e − a 1 − e − a . \mathbb{P}(\lvert K_i\rvert\ge k) \le2e^{-(2k-1)^2a} \le2e^{-ka}, \qquad\mathbb{E}\lvert K_i\rvert= \sum_{k=1}^{N}\mathbb{P}(\lvert K_i\rvert\ge k) < \frac{2e^{-a}}{1-e^{-a}}. P (∣ K i ∣ ≥ k ) ≤ 2 e − ( 2 k − 1 ) 2 a ≤ 2 e − k a , E ∣ K i ∣ = k = 1 ∑ N P (∣ K i ∣ ≥ k ) < 1 − e − a 2 e − a .
Take reference masses q k = Z − 1 e − a ∣ k ∣ q_k=Z^{-1}e^{-a\lvert k\rvert} q k = Z − 1 e − a ∣ k ∣ on { − N , … , N } \{-N,\ldots,N\} { − N , … , N } . Finite geometric summation gives log Z ≤ Z − 1 < 2 e − a / ( 1 − e − a ) \log Z \le Z-1 < 2e^{-a}/(1-e^{-a}) log Z ≤ Z − 1 < 2 e − a / ( 1 − e − a ) . Lemma C.1 implies
H ( K i ) ≤ log Z + a E ∣ K i ∣ < 2 ( a + 1 ) e − a 1 − e − a . (36) H(K_i) \le\log Z+a\mathbb{E}\lvert K_i\rvert< \frac{2(a+1)e^{-a}}{1-e^{-a}}. \tag*{(36)} H ( K i ) ≤ log Z + a E ∣ K i ∣ < 1 − e − a 2 ( a + 1 ) e − a . (36)
For s ≥ 1 s\ge1 s ≥ 1 and a = 7 2 log ( e + s ) > 9 2 a=\frac{7}{2}\log(e+s)>\frac{9}{2} a = 2 7 log ( e + s ) > 2 9 , the derivative of log s + log ( a + 1 ) − a \log s+\log(a+1)-a log s + log ( a + 1 ) − a with respect to s s s is at most 1 s − 63 22 ( e + s ) \frac{1}{s}-\frac{63}{22(e+s)} s 1 − 22 ( e + s ) 63 , which is positive only for s < 22 41 e < 3 2 s<\frac{22}{41}e<\frac{3}{2} s < 41 22 e < 2 3 , and across all s ≥ 1 s\ge1 s ≥ 1 one has s H ( K i ) < 3 25 sH(K_i)<\frac{3}{25} sH ( K i ) < 25 3 . By subadditivity, the joint entropy of K = ( K 1 , … , K M ) K=(K_1,\ldots,K_M) K = ( K 1 , … , K M ) satisfies H ( K ) ≤ ∑ i = 1 M H ( K i ) < 3 M 25 s = 3 25 p H(K)\le\sum_{i=1}^{M}H(K_i)<\frac{3M}{25s}=\frac{3}{25}p H ( K ) ≤ ∑ i = 1 M H ( K i ) < 25 s 3 M = 25 3 p . By Lemma C.1 , some common fiber F ⊆ { − 1 , 1 } p F\subseteq\{-1,1\}^{p} F ⊆ { − 1 , 1 } p of K K K has cardinality ∣ F ∣ ≥ 2 p e − H ( K ) \lvert F\rvert\ge2^p e^{-H(K)} ∣ F ∣ ≥ 2 p e − H ( K ) .
Let W W W be uniform on F F F with mean μ = E W ∈ [ − 1 , 1 ] p \mu=\mathbb{E}W\in[-1,1]^p μ = E W ∈ [ − 1 , 1 ] p . Applying Lemma C.1 to each coordinate W j W_j W j with reference masses e ± μ j / ( 2 cosh μ j ) e^{\pm\mu_j}/(2\cosh\mu_j) e ± μ j / ( 2 cosh μ j ) gives
H ( W j ) ≤ log ( 2 cosh μ j ) − μ j 2 ≤ log 2 − 1 2 μ j 2 . H(W_j)\le\log(2\cosh\mu_j)-\mu_j^2\le\log2-\frac{1}{2}\mu_j^2. H ( W j ) ≤ log ( 2 cosh μ j ) − μ j 2 ≤ log 2 − 2 1 μ j 2 .
Subadditivity of entropy across the p p p coordinates gives p log 2 − H ( K ) ≤ log ∣ F ∣ = H ( W ) ≤ p log 2 − 1 2 ∥ μ ∥ 2 p\log2-H(K)\le\log\lvert F\rvert=H(W)\le p\log2-\frac{1}{2}\lVert\mu\rVert^2 p log 2 − H ( K ) ≤ log ∣ F ∣ = H ( W ) ≤ p log 2 − 2 1 ∥ μ ∥ 2 , whence ∥ μ ∥ 2 ≤ 2 H ( K ) < 6 25 p \lVert\mu\rVert^2\le2H(K)<\frac{6}{25}p ∥ μ ∥ 2 ≤ 2 H ( K ) < 25 6 p . Two independent uniform draws X , X ′ X,X' X , X ′ from F F F satisfy
E ∥ X − X ′ ∥ 2 = 2 ( p − ∥ μ ∥ 2 ) > 38 25 p , \mathbb{E}\lVert X-X'\rVert^2=2(p-\lVert\mu\rVert^2)>\frac{38}{25}p, E ∥ X − X ′ ∥ 2 = 2 ( p − ∥ μ ∥ 2 ) > 25 38 p ,
so some pair X , X ′ ∈ F X,X'\in F X , X ′ ∈ F satisfies ∥ X − X ′ ∥ > ( 38 / 25 ) p \lVert X-X'\rVert>\sqrt{(38/25)p} ∥ X − X ′ ∥ > ( 38/25 ) p , while equal bins give ∣ ⟨ a i , X − X ′ ⟩ ∣ < 2 t R = 28 log ( e + s ) R \lvert\langle a_i,X-X'\rangle\rvert<2tR=\sqrt{28\log(e+s)}R ∣⟨ a i , X − X ′ ⟩∣ < 2 tR = 28 log ( e + s ) R . Normalizing X − X ′ X-X' X − X ′ yields a unit vector u ∈ S u\in S u ∈ S with squared projection bound at most 350 19 R 2 log ( e + s ) / p < ( 13 3 R 2 ) log ( e + s ) / p \frac{350}{19}R^2\log(e+s)/p<(\frac{13}{3}R^2)\log(e+s)/p 19 350 R 2 log ( e + s ) / p < ( 3 13 R 2 ) log ( e + s ) / p .
Lemma C.3 (Avoiding spherical caps by a union bound). Let R > 0 R>0 R > 0 and τ ≥ 0 \tau\ge0 τ ≥ 0 . Let S S S have dimension p ≥ 1 p\ge1 p ≥ 1 , and let ∥ a i ∥ ≤ R \lVert a_i\rVert\le R ∥ a i ∥ ≤ R . If 2 M exp ( − p τ 2 / ( 2 R 2 ) ) < 1 2M\exp(-p\tau^2/(2R^2))<1 2 M exp ( − p τ 2 / ( 2 R 2 )) < 1 , there is a unit u ∈ S u\in S u ∈ S with ∣ ⟨ u , a i ⟩ ∣ ≤ τ \lvert\langle u,a_i\rangle\rvert\le\tau ∣⟨ u , a i ⟩∣ ≤ τ for all i i i .
Proof. A uniform sphere direction u u u satisfies P ( ∣ ⟨ u , a ⟩ ∣ > τ ) ≤ 2 e − p τ 2 / ( 2 R 2 ) \mathbb{P}(\lvert\langle u,a\rangle\rvert>\tau)\le2e^{-p\tau^2/(2R^2)} P (∣⟨ u , a ⟩∣ > τ ) ≤ 2 e − p τ 2 / ( 2 R 2 ) . Indeed its odd moments vanish and its 2 k 2k 2 k -th projection moment is ∥ P S a ∥ 2 k ( 2 k − 1 ) ! ! / [ p ( p + 2 ) ⋯ ( p + 2 k − 2 ) ] \lVert P_Sa\rVert^{2k}(2k-1)!!/[p(p+2)\cdots(p+2k-2)] ∥ P S a ∥ 2 k ( 2 k − 1 )!! / [ p ( p + 2 ) ⋯ ( p + 2 k − 2 )] , where P S P_S P S is the orthogonal projection onto S S S . The moment-generating series is dominated by e λ 2 R 2 / ( 2 p ) e^{\lambda^2R^2/(2p)} e λ 2 R 2 / ( 2 p ) and exponential Markov proves the tail. For the moment identity, polar coordinates for an isotropic Gaussian give an independent uniform direction and radius; scalar Gaussian moments are ( 2 k − 1 ) ! ! (2k-1)!! ( 2 k − 1 )!! and the radius moments are p ( p + 2 ) ⋯ ( p + 2 k − 2 ) p(p+2)\cdots(p+2k-2) p ( p + 2 ) ⋯ ( p + 2 k − 2 ) by integration by parts, including the p = 1 p=1 p = 1 case. The union bound now leaves positive probability of avoiding all the caps.
One transcript argument
The following lemma handles the deterministic lower bounds. It handles values, rank-priced linear feedback, closing calls, unqueried outputs, and the identification of the simulated run with the run on one fixed function.
Lemma C.4 (Stage argument). Let F U F_U F U range over orthonormal m m m -frames U = ( u 1 , … , u m ) U=(u_1,\ldots,u_m) U = ( u 1 , … , u m ) in R d \mathbb{R}^d R d , and let z : R d → R d z:\mathbb{R}^d\to\mathbb{R}^d z : R d → R d satisfy ∥ z ( x ) ∥ ≤ R \lVert z(x)\rVert\le R ∥ z ( x )∥ ≤ R . Suppose that for each partial frame there are a prefix function r j r_j r j and a coefficient c j c_j c j , depending only on u 1 , … , u j − 1 u_1,\ldots,u_{j-1} u 1 , … , u j − 1 , such that
F U ( x ) = r j ( x ) , ∇ F U ( x ) = ∇ r j ( x ) − c j ( x ) u j F_U(x)=r_j(x),\qquad\nabla F_U(x)=\nabla r_j(x)-c_j(x)u_j F U ( x ) = r j ( x ) , ∇ F U ( x ) = ∇ r j ( x ) − c j ( x ) u j whenever ⟨ u j , z ( x ) ⟩ = 0 \langle u_j,z(x)\rangle=0 ⟨ u j , z ( x )⟩ = 0 and ∣ ⟨ u k , z ( x ) ⟩ ∣ ≤ τ \lvert\langle u_k,z(x)\rangle\rvert\le\tau ∣⟨ u k , z ( x )⟩∣ ≤ τ for all k > j k>j k > j . Assume also that ⟨ u j , z ( x ) ⟩ = 0 \langle u_j,z(x)\rangle=0 ⟨ u j , z ( x )⟩ = 0 for some j j j implies ∥ ∇ F U ( x ) ∥ > b \lVert\nabla F_U(x)\rVert>b ∥ ∇ F U ( x )∥ > b .
Let m , h ≥ 1 m,h\ge1 m , h ≥ 1 and Q ≥ 0 Q\ge0 Q ≥ 0 be integers and put p = d − m − h + 1 ≥ 1 p=d-m-h+1\ge1 p = d − m − h + 1 ≥ 1 . Assume that for every subspace S S S of dimension at least p p p and every list of at most Q Q Q points a i a_i a i of norm at most R R R there is a unit u ∈ S u\in S u ∈ S with ∣ ⟨ u , a i ⟩ ∣ ≤ τ \lvert\langle u,a_i\rangle\rvert\le\tau ∣⟨ u , a i ⟩∣ ≤ τ for all i i i . Then every deterministic algorithm whose calls ( x , V ) (x,V) ( x , V ) return ( F U ( x ) , V ⊤ ∇ F U ( x ) ) (F_U(x),V^\top\nabla F_U(x)) ( F U ( x ) , V ⊤ ∇ F U ( x )) at cost 1 + rank V 1+\operatorname{rank}V 1 + rank V , and which on every transcript makes at most Q Q Q calls of total cost below h m hm hm , outputs a point with gradient norm greater than b b b for at least one fixed F U F_U F U . Queries, matrices, and the output are unrestricted. The geometric assumption holds, for τ ≥ 0 \tau\ge0 τ ≥ 0 , under any one of the conditions
Q < p ; R > 0 and 2 Q exp ( − p τ 2 2 R 2 ) < 1 ; 13 3 R log ( e + Q / p ) p ≤ τ . Q<p;\qquad R>0\ \text{ and }\ 2Q\exp\left(-\frac{p\tau^2}{2R^2}\right)<1;\qquad\frac{13}{3}R\sqrt{\frac{\log(e+Q/p)}{p}}\le\tau. Q < p ; R > 0 and 2 Q exp ( − 2 R 2 p τ 2 ) < 1 ; 3 13 R p log ( e + Q / p ) ≤ τ .
Proof. Maintain the selected directions, the points z ( x ) z(x) z ( x ) of closed stages, and the pending calls of the current stage. If the pending calls have total cost ℓ < h \ell<h ℓ < h , their points z ( x ) z(x) z ( x ) and matrix column spaces together span at most ℓ \ell ℓ dimensions. For a new call ( x , V ) (x,V) ( x , V ) of cost w = 1 + rank V w=1+\operatorname{rank}V w = 1 + rank V : if ℓ + w < h \ell+w<h ℓ + w < h , return the prefix reply ( r j ( x ) , V ⊤ ∇ r j ( x ) ) (r_j(x),V^\top\nabla r_j(x)) ( r j ( x ) , V ⊤ ∇ r j ( x )) and record the call as pending. If ℓ + w ≥ h \ell+w\ge h ℓ + w ≥ h , select u j u_j u j before answering: perpendicular to the preceding frame, to all pending constraints, and to z ( x ) z(x) z ( x ) , and with inner products at most τ \tau τ in magnitude with all points of closed stages. These exact constraints have dimension at most ( m − 1 ) + ( h − 1 ) + 1 (m-1)+(h-1)+1 ( m − 1 ) + ( h − 1 ) + 1 , which leaves dimension at least p p p , so the geometric assumption supplies u j u_j u j . The current column space is not constrained. Return ( r j ( x ) , V ⊤ [ ∇ r j ( x ) − c j ( x ) u j ] ) (r_j(x),V^\top[\nabla r_j(x)-c_j(x)u_j]) ( r j ( x ) , V ⊤ [ ∇ r j ( x ) − c j ( x ) u j ]) , close the stage, move its points to the list of past points, and reset ℓ = 0 \ell=0 ℓ = 0 . Each closed stage consumes at least h h h units of cost, so at most m − 1 m-1 m − 1 stages close.
At the output x ^ \hat{x} x ^ , select the next direction in the same way, with z ( x ^ ) z(\hat{x}) z ( x ^ ) in place of a closing query point. Move the pending points to the list of past points and complete the frame with directions perpendicular to their predecessors and with inner products at most τ \tau τ at every past point; there are at least p p p residual dimensions and at most Q Q Q past points. For the completed frame, every call of stage j j j has zero j j j th coordinate and later coordinates bounded by τ \tau τ , so its reply is the prefix reply together with the term of the current direction. That term vanishes for pending calls, whose column spaces were among the orthogonality constraints, and closing calls received it explicitly. Induction over the deterministic transcript shows that the simulated run is the run on this single function. Its output has a zero hidden coordinate, so its gradient norm exceeds b b b .
Exact orthogonality to fewer than p p p past points is possible inside a p p p -dimensional subspace; the other two conditions are Lemmas C.3 and C.2, whose bounds only improve in larger subspaces. □ \square □
A budget asserted only on functions in the class reduces to this form for whole transcripts by truncating before any call that would exceed the budget: every finite simulated prefix extends to a fixed function in the class, on which the algorithm terminates within its budget, so the truncation never changes that function’s run. This applies to call caps and cost caps even when their worst cases occur on different functions.
The Chain and the Radial Compression
The deterministic witnesses share the form F U , R , Q ( x ) = g m ( U ⊤ ρ R ( x ) ) + Q ( ∥ x ∥ ) F_{U,R,Q}(x)=g_m(U^\top\rho_R(x))+Q(\lVert x\rVert) F U , R , Q ( x ) = g m ( U ⊤ ρ R ( x )) + Q (∥ x ∥) . Only the public radial parameters differ. All scalar gates are piecewise polynomial; no convolution is needed.
One scalar chain
Define
φ ( t ) = { 0 , t ≤ − 5 / 4 , 2 ( t + 5 / 4 ) 2 , − 5 / 4 < t < − 1 , t + 9 / 8 , − 1 ≤ t ≤ 1 , 9 / 4 − 2 ( 5 / 4 − t ) 2 , 1 < t < 5 / 4 , 9 / 4 , t ≥ 5 / 4 , \varphi(t)=
\begin{cases}
0, & t\le-5/4,\\
2(t+5/4)^2, & -5/4<t<-1,\\
t+9/8, & -1\le t\le1,\\
9/4-2(5/4-t)^2, & 1<t<5/4,\\
9/4, & t\ge5/4,
\end{cases} φ ( t ) = ⎩ ⎨ ⎧ 0 , 2 ( t + 5/4 ) 2 , t + 9/8 , 9/4 − 2 ( 5/4 − t ) 2 , 9/4 , t ≤ − 5/4 , − 5/4 < t < − 1 , − 1 ≤ t ≤ 1 , 1 < t < 5/4 , t ≥ 5/4 ,
and
ψ ( t ) = { 0 , t ≤ 1 / 2 , 8 ( t − 1 / 2 ) 2 , 1 / 2 < t ≤ 3 / 4 , 1 − 8 ( 1 − t ) 2 , 3 / 4 < t < 1 , 1 , t ≥ 1. \psi(t)=
\begin{cases}
0, & t\le1/2,\\
8(t-1/2)^2, & 1/2<t\le3/4,\\
1-8(1-t)^2, & 3/4<t<1,\\
1, & t\ge1.
\end{cases} ψ ( t ) = ⎩ ⎨ ⎧ 0 , 8 ( t − 1/2 ) 2 , 1 − 8 ( 1 − t ) 2 , 1 , t ≤ 1/2 , 1/2 < t ≤ 3/4 , 3/4 < t < 1 , t ≥ 1.
and, for m ≥ 1 m\ge1 m ≥ 1 ,
g m ( y ) = − φ ( y 1 ) + ∑ i = 2 m [ ψ ( − y i − 1 ) φ ( − y i ) − ψ ( y i − 1 ) φ ( y i ) ] . (37) g_m(y)=-\varphi(y_1)+\sum_{i=2}^{m}\left[\psi(-y_{i-1})\varphi(-y_i)-\psi(y_{i-1})\varphi(y_i)\right]. \tag*{(37)} g m ( y ) = − φ ( y 1 ) + i = 2 ∑ m [ ψ ( − y i − 1 ) φ ( − y i ) − ψ ( y i − 1 ) φ ( y i ) ] . (37)
The same chain is used for the lower bounds with and without a trace budget; only radial parameters change.
Lemma D.1 (The scalar chain). Let p = ∇ g m ( y ) p=\nabla g_m(y) p = ∇ g m ( y ) and I ( y ) = { i : ∣ y i ∣ ≤ 5 / 4 } I(y)=\{i:|y_i|\le5/4\} I ( y ) = { i : ∣ y i ∣ ≤ 5/4 } . The function g m g_m g m is C 1 , 1 C^{1,1} C 1 , 1 and bounded, and
∥ p ∥ ≤ 10 m , Lip ( ∇ g m ) ≤ 42 , g m ( 0 ) − inf g m ≤ 9 4 ( m − 1 2 ) . \lVert p\rVert\le10\sqrt{m},\qquad\operatorname{Lip}(\nabla g_m)\le42,\qquad g_m(0)-\inf g_m\le\frac{9}{4}\left(m-\frac{1}{2}\right). ∥ p ∥ ≤ 10 m , Lip ( ∇ g m ) ≤ 42 , g m ( 0 ) − inf g m ≤ 4 9 ( m − 2 1 ) . Every p i ≤ 0 p_i\le0 p i ≤ 0 , and ∣ y i ∣ ≥ 5 / 4 |y_i|\ge5/4 ∣ y i ∣ ≥ 5/4 implies p i = 0 p_i=0 p i = 0 . If j j j is the first index with ∣ y j ∣ < 1 |y_j|<1 ∣ y j ∣ < 1 , then p j ≤ − 1 p_j\le-1 p j ≤ − 1 . On each smooth piece, the Hessian B = ∇ 2 g m ( y ) B=\nabla^2g_m(y) B = ∇ 2 g m ( y ) is supported on I ( y ) × I ( y ) I(y)\times I(y) I ( y ) × I ( y ) and satisfies ∥ B ∥ op ≤ 42 \lVert B\rVert_{\operatorname{op}}\le42 ∥ B ∥ op ≤ 42 . Consequently,
∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m , ∣ y ⊤ p ∣ ≤ 31 4 m . (38) \lVert By\rVert\le44\sqrt{m},\qquad|y^\top By|\le44m,\qquad|y^\top p|\le\frac{31}{4}m. \tag*{(38)} ∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m , ∣ y ⊤ p ∣ ≤ 4 31 m . (38)
Proof. The formulas have matching values and first derivatives at every junction. They give
0 ≤ φ ≤ 9 / 4 , 0 ≤ φ ′ ≤ 1 , ∣ φ ′ ′ ∣ ≤ 4 , 0 ≤ ψ ≤ 1 , 0 ≤ ψ ′ ≤ 4 , ∣ ψ ′ ′ ∣ ≤ 16 0\le\varphi\le9/4,\qquad0\le\varphi'\le1,\qquad|\varphi''|\le4,\qquad0\le\psi\le1,\qquad0\le\psi'\le4,\qquad|\psi''|\le16 0 ≤ φ ≤ 9/4 , 0 ≤ φ ′ ≤ 1 , ∣ φ ′′ ∣ ≤ 4 , 0 ≤ ψ ≤ 1 , 0 ≤ ψ ′ ≤ 4 , ∣ ψ ′′ ∣ ≤ 16
on their smooth pieces. Moreover, φ ′ \varphi' φ ′ is even and equals one on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , so φ ′ ′ ( t ) \varphi''(t) φ ′′ ( t ) vanishes outside 1 ≤ ∣ t ∣ ≤ 5 / 4 1\le|t|\le5/4 1 ≤ ∣ t ∣ ≤ 5/4 , whereas ψ ′ ( ± t ) \psi'(\pm t) ψ ′ ( ± t ) and ψ ′ ′ ( ± t ) \psi''(\pm t) ψ ′′ ( ± t ) are supported on 1 / 2 ≤ ∣ t ∣ ≤ 1 1/2\le|t|\le1 1/2 ≤ ∣ t ∣ ≤ 1 . At most one of ψ ( t ) , ψ ( − t ) \psi(t),\psi(-t) ψ ( t ) , ψ ( − t ) is nonzero. With γ 0 = 1 \gamma_0=1 γ 0 = 1 and γ i − 1 = ψ ( − y i − 1 ) + ψ ( y i − 1 ) ∈ [ 0 , 1 ] \gamma_{i-1}=\psi(-y_{i-1})+\psi(y_{i-1})\in[0,1] γ i − 1 = ψ ( − y i − 1 ) + ψ ( y i − 1 ) ∈ [ 0 , 1 ] for i > 1 i>1 i > 1 ,
p i = − γ i − 1 φ ′ ( y i ) − 1 { i < m } [ ψ ′ ( − y i ) φ ( − y i + 1 ) + ψ ′ ( y i ) φ ( y i + 1 ) ] . (39) p_i=-\gamma_{i-1}\varphi'(y_i)-\mathbf{1}_{\{i<m\}}\left[\psi'(-y_i)\varphi(-y_{i+1})+\psi'(y_i)\varphi(y_{i+1})\right]. \tag*{(39)} p i = − γ i − 1 φ ′ ( y i ) − 1 { i < m } [ ψ ′ ( − y i ) φ ( − y i + 1 ) + ψ ′ ( y i ) φ ( y i + 1 ) ] . (39)
This proves the sign, the support, the coordinate bound ∣ p i ∣ ≤ 10 |p_i|\le10 ∣ p i ∣ ≤ 10 , and the barrier: at the first index with ∣ y j ∣ < 1 |y_j|<1 ∣ y j ∣ < 1 the preceding gate is saturated. Because ∣ s φ ′ ( s ) ∣ ≤ 1 |s\varphi'(s)|\le1 ∣ s φ ′ ( s ) ∣ ≤ 1 on R \mathbb{R} R (as 4 s ( 5 / 4 − s ) = 1 − 4 ( s − 1 ) ( s − 1 / 4 ) ≤ 1 4s(5/4-s)=1-4(s-1)(s-1/4)\le1 4 s ( 5/4 − s ) = 1 − 4 ( s − 1 ) ( s − 1/4 ) ≤ 1 on [ 1 , 5 / 4 ] [1,5/4] [ 1 , 5/4 ] ) and t ψ ′ ( t ) ≤ 3 t\psi'(t)\le3 t ψ ′ ( t ) ≤ 3 on [ 1 / 2 , 1 ] [1/2,1] [ 1/2 , 1 ] (as 16 t ( 1 − t ) = 3 − 16 ( t − 3 / 4 ) ( t − 1 / 4 ) ≤ 3 16t(1-t)=3-16(t-3/4)(t-1/4)\le3 16 t ( 1 − t ) = 3 − 16 ( t − 3/4 ) ( t − 1/4 ) ≤ 3 on [ 3 / 4 , 1 ] [3/4,1] [ 3/4 , 1 ] ), each coordinate satisfies ∣ y i p i ∣ ≤ 1 + 3 ( 9 / 4 ) = 31 / 4 |y_i p_i|\le1+3(9/4)=31/4 ∣ y i p i ∣ ≤ 1 + 3 ( 9/4 ) = 31/4 , which sums to ∣ y ⊤ p ∣ ≤ ( 31 / 4 ) m |y^\top p|\le(31/4)m ∣ y ⊤ p ∣ ≤ ( 31/4 ) m .
The Hessian B B B is symmetric tridiagonal with ∣ B i , i − 1 ∣ ≤ 4 |B_{i,i-1}|\le4 ∣ B i , i − 1 ∣ ≤ 4 , and B i , i − 1 ≠ 0 B_{i,i-1}\ne0 B i , i − 1 = 0 requires ∣ y i − 1 ∣ ≤ 1 |y_{i-1}|\le1 ∣ y i − 1 ∣ ≤ 1 . Since the supports of φ ′ ′ ( y i ) \varphi''(y_i) φ ′′ ( y i ) and ψ ′ ′ ( ± y i ) \psi''(\pm y_i) ψ ′′ ( ± y i ) meet only at ∣ y i ∣ = 1 |y_i|=1 ∣ y i ∣ = 1 , there are two cases for row i i i . If ∣ y i ∣ ≤ 1 |y_i|\le1 ∣ y i ∣ ≤ 1 , then φ ′ ′ ( y i ) = 0 \varphi''(y_i)=0 φ ′′ ( y i ) = 0 , and with s = ± y i + 1 s=\pm y_{i+1} s = ± y i + 1 one has
∣ B i , i − 1 ∣ + ∣ B i i ∣ + ∣ B i , i + 1 ∣ ≤ 4 + 16 φ ( s ) + 4 φ ′ ( s ) ≤ 42 , (40) |B_{i,i-1}|+|B_{ii}|+|B_{i,i+1}|\le4+16\varphi(s)+4\varphi'(s)\le42, \tag*{(40)} ∣ B i , i − 1 ∣ + ∣ B ii ∣ + ∣ B i , i + 1 ∣ ≤ 4 + 16 φ ( s ) + 4 φ ′ ( s ) ≤ 42 , (40)
because 4 + 16 φ ( s ) + 4 φ ′ ( s ) 4 + 16\varphi(s) + 4\varphi'(s) 4 + 16 φ ( s ) + 4 φ ′ ( s ) is at most 4 + 16 ( 17 / 8 ) + 4 = 42 4 + 16(17/8) + 4 = 42 4 + 16 ( 17/8 ) + 4 = 42 for s ≤ 1 s \le1 s ≤ 1 , equals 42 − 32 ( s − 1 ) 2 ≤ 42 42 - 32(s - 1)^2 \le42 42 − 32 ( s − 1 ) 2 ≤ 42 on [ 1 , 5 / 4 ] [1, 5/4] [ 1 , 5/4 ] , and equals 40 40 40 for s ≥ 5 / 4 s \ge5/4 s ≥ 5/4 . In the same case ∣ y i − 1 ∣ ≤ 1 |y_{i-1}| \le1 ∣ y i − 1 ∣ ≤ 1 , ∣ y i ∣ ≤ 1 |y_i| \le1 ∣ y i ∣ ≤ 1 , and ∣ y i + 1 ∣ φ ′ ( ± y i + 1 ) ≤ 1 |y_{i+1}|\varphi'(\pm y_{i+1}) \le1 ∣ y i + 1 ∣ φ ′ ( ± y i + 1 ) ≤ 1 , so ∣ ( B y ) i ∣ ≤ 4 ⋅ 1 + 36 ⋅ 1 + 4 ⋅ 1 = 44 |(By)_i| \le4 \cdot1 + 36 \cdot1 + 4 \cdot1 = 44 ∣ ( B y ) i ∣ ≤ 4 ⋅ 1 + 36 ⋅ 1 + 4 ⋅ 1 = 44 and ∣ y i ( B y ) i ∣ ≤ 44 |y_i(By)_i| \le44 ∣ y i ( B y ) i ∣ ≤ 44 . If 1 < ∣ y i ∣ ≤ 5 / 4 1 < |y_i| \le5/4 1 < ∣ y i ∣ ≤ 5/4 , then ψ ′ ( ± y i ) = ψ ′ ′ ( ± y i ) = 0 \psi'(\pm y_i) = \psi''(\pm y_i) = 0 ψ ′ ( ± y i ) = ψ ′′ ( ± y i ) = 0 , so B i , i + 1 = 0 B_{i,i+1} = 0 B i , i + 1 = 0 , ∣ B i i ∣ ≤ 4 |B_{ii}| \le4 ∣ B ii ∣ ≤ 4 , the absolute row sum is at most 8 8 8 , ∣ ( B y ) i ∣ ≤ 4 ( 5 / 4 ) + 4 ⋅ 1 = 9 |(By)_i| \le4(5/4) + 4 \cdot1 = 9 ∣ ( B y ) i ∣ ≤ 4 ( 5/4 ) + 4 ⋅ 1 = 9 , and ∣ y i ( B y ) i ∣ ≤ 45 / 4 < 44 |y_i(By)_i| \le45/4 < 44 ∣ y i ( B y ) i ∣ ≤ 45/4 < 44 . Thus ∥ B ∥ o p ≤ 42 \|B\|_{\mathrm{op}} \le42 ∥ B ∥ op ≤ 42 , and (38) holds. Matching derivatives extend ∥ B ∥ o p ≤ 42 \|B\|_{\mathrm{op}} \le42 ∥ B ∥ op ≤ 42 to Lip ( ∇ g m ) ≤ 42 \operatorname{Lip}(\nabla g_m) \le42 Lip ( ∇ g m ) ≤ 42 by integration over the finitely many polynomial pieces of a line segment. Finally g m ( 0 ) = − 9 / 8 g_m(0) = -9/8 g m ( 0 ) = − 9/8 , and each of the m m m summands of g m g_m g m is at least − 9 / 4 -9/4 − 9/4 .
Radial compression and shifted Huber potentials
For R > 0 R > 0 R > 0 write
a ( x ) = ( 1 + ∥ x ∥ 2 R 2 ) − 1 / 2 , ρ R ( x ) = a ( x ) x . (41) a(x) = \left(1 + \frac{\|x\|^2}{R^2}\right)^{-1/2}, \qquad\rho_R(x) = a(x)x. \tag*{(41)} a ( x ) = ( 1 + R 2 ∥ x ∥ 2 ) − 1/2 , ρ R ( x ) = a ( x ) x . (41)
For λ , s > 0 \lambda, s > 0 λ , s > 0 and r 0 ≥ 0 r_0 \ge0 r 0 ≥ 0 , define the shifted radial Huber potential ([25 ] ) through its derivative,
Q ′ ( r ) = λ min { ( r − r 0 ) + , s } , Q ( 0 ) = 0 , q ( x ) = Q ( ∥ x ∥ ) . (42) Q'(r) = \lambda\min\{(r-r_0)_+,s\}, \qquad Q(0) = 0, \qquad q(x) = Q(\|x\|). \tag*{(42)} Q ′ ( r ) = λ min {( r − r 0 ) + , s } , Q ( 0 ) = 0 , q ( x ) = Q ( ∥ x ∥ ) . (42)
It vanishes on the ball of radius r 0 r_0 r 0 , is quadratic across an annulus of width s s s , and is linear afterwards. The construction with a trace budget below uses a related sum of two such potentials; the unrestricted choice uses one flat on an inner ball.
Lemma D.2 (Radial derivatives). The potential (42) is nonnegative, convex, coercive, and C 1 , 1 C^{1,1} C 1 , 1 , with ∥ ∇ q ∥ ≤ λ s \|\nabla q\| \le\lambda s ∥∇ q ∥ ≤ λ s and Lip ( ∇ q ) ≤ λ \operatorname{Lip}(\nabla q) \le\lambda Lip ( ∇ q ) ≤ λ . For z = ρ R ( x ) z = \rho_R(x) z = ρ R ( x ) ,
D ρ R ( x ) = a ( x ) ( I − z z T R 2 ) , ∥ D ρ R ( x ) ∥ o p ≤ 1 , ∥ D 2 ρ R ( x ) [ v , w ] ∥ ≤ 2 3 R ∥ v ∥ ∥ w ∥ . (43) D\rho_R(x) = a(x)\left(I-\frac{zz^{\mathsf T}}{R^2}\right), \qquad\|D\rho_R(x)\|_{\mathrm{op}} \le1, \qquad\|D^2\rho_R(x)[v,w]\| \le\frac{2}{\sqrt{3}R}\|v\|\|w\|. \tag*{(43)} D ρ R ( x ) = a ( x ) ( I − R 2 z z T ) , ∥ D ρ R ( x ) ∥ op ≤ 1 , ∥ D 2 ρ R ( x ) [ v , w ] ∥ ≤ 3 R 2 ∥ v ∥∥ w ∥. (43) Proof. For r > 0 r > 0 r > 0 , ∇ q = Q ′ ( r ) x / r \nabla q = Q'(r)x/r ∇ q = Q ′ ( r ) x / r . On every smooth piece its radial and tangential Hessian eigenvalues are Q ′ ′ ( r ) Q''(r) Q ′′ ( r ) and Q ′ ( r ) / r Q'(r)/r Q ′ ( r ) / r , both in [ 0 , λ ] [0,\lambda] [ 0 , λ ] , because Q ′ Q' Q ′ is nondecreasing, Q ′ ′ ∈ { 0 , λ } Q'' \in\{0,\lambda\} Q ′′ ∈ { 0 , λ } , and Q ′ ( r ) ≤ λ r Q'(r) \le\lambda r Q ′ ( r ) ≤ λ r . The gradient is continuous across the junction spheres, and integrating along line segments gives the Lipschitz bound. The other properties follow from Q ′ ≥ 0 Q' \ge0 Q ′ ≥ 0 and Q ′ ( r ) = λ s Q'(r) = \lambda s Q ′ ( r ) = λ s for large r r r .
Differentiation gives the formula for D ρ R D\rho_R D ρ R , whose eigenvalues are a a a tangentially and a 3 a^3 a 3 radially. A second differentiation gives
D 2 ρ R ( x ) [ v , w ] = − a 3 R 2 [ ( x T w ) v + ( x T v ) w + ( v T w ) x ] + 3 a 5 R 4 ( x T v ) ( x T w ) x . D^2\rho_R(x)[v,w] = -\frac{a^3}{R^2}\left[(x^{\mathsf T}w)v + (x^{\mathsf T}v)w + (v^{\mathsf T}w)x\right] + \frac{3a^5}{R^4}(x^{\mathsf T}v)(x^{\mathsf T}w)x. D 2 ρ R ( x ) [ v , w ] = − R 2 a 3 [ ( x T w ) v + ( x T v ) w + ( v T w ) x ] + R 4 3 a 5 ( x T v ) ( x T w ) x . For a unit v v v and x ≠ 0 x \ne0 x = 0 , put t = ∥ x ∥ / R t = \|x\|/R t = ∥ x ∥/ R , C = ( x T v / ∥ x ∥ ) 2 C = (x^{\mathsf T}v/\|x\|)^2 C = ( x T v /∥ x ∥ ) 2 , and z = t 2 / ( 1 + t 2 ) z = t^2/(1+t^2) z = t 2 / ( 1 + t 2 ) . Then
∥ D 2 ρ R ( x ) [ v , v ] ∥ 2 = t 2 R 2 ( 1 + t 2 ) 3 [ 1 + ( 8 − 6 z ) C + ( 9 z 2 − 12 z ) C 2 ] . \|D^2\rho_R(x)[v,v]\|^2 = \frac{t^2}{R^2(1+t^2)^3}\left[1 + (8-6z)C + (9z^2-12z)C^2\right]. ∥ D 2 ρ R ( x ) [ v , v ] ∥ 2 = R 2 ( 1 + t 2 ) 3 t 2 [ 1 + ( 8 − 6 z ) C + ( 9 z 2 − 12 z ) C 2 ] . For fixed C ∈ [ 0 , 1 ] C \in[0,1] C ∈ [ 0 , 1 ] the bracket is convex in z ∈ [ 0 , 1 ] z \in[0,1] z ∈ [ 0 , 1 ] ; at the endpoints it equals 1 + 8 C ≤ 9 1 + 8C \le9 1 + 8 C ≤ 9 and 1 + 2 C − 3 C 2 ≤ 4 / 3 1 + 2C - 3C^2 \le4/3 1 + 2 C − 3 C 2 ≤ 4/3 . Hence the diagonal norm is at most 3 t / [ R ( 1 + t 2 ) 3 / 2 ] ≤ 2 / ( 3 R ) 3t/[R(1+t^2)^{3/2}] \le2/(\sqrt{3}R) 3 t / [ R ( 1 + t 2 ) 3/2 ] ≤ 2/ ( 3 R ) , and it vanishes at x = 0 x = 0 x = 0 . Polarization, D 2 ρ R [ v , w ] = 1 4 ( D 2 ρ R [ v + w , v + w ] − D 2 ρ R [ v − w , v − w ] ) D^2\rho_R[v,w] = \frac{1}{4}(D^2\rho_R[v+w,v+w] - D^2\rho_R[v-w,v-w]) D 2 ρ R [ v , w ] = 4 1 ( D 2 ρ R [ v + w , v + w ] − D 2 ρ R [ v − w , v − w ]) , and the parallelogram law give the same bound for unit v , w v,w v , w .
Integration on finitely many smooth pieces
Lemma D.3 (Integration over finitely many pieces). Let F : R d → R F:\mathbb{R}^{d}\to\mathbb{R} F : R d → R be continuously differentiable. Suppose that along every segment x + t v x+tv x + t v , 0 ≤ t ≤ 1 0\le t\le1 0 ≤ t ≤ 1 , its gradient is continuously differentiable except at finitely many points, and that on each remaining interval
d d t ∇ F ( x + t v ) = B ( t ) v , ∥ B ( t ) ∥ op ≤ L 0 , B ( t ) ⪯ H , \frac{d}{dt}\nabla F(x+tv)=B(t)v,\qquad\lVert B(t)\rVert_{\operatorname{op}}\le L_{0},\qquad B(t)\preceq H, d t d ∇ F ( x + t v ) = B ( t ) v , ∥ B ( t ) ∥ op ≤ L 0 , B ( t ) ⪯ H , where B ( t ) B(t) B ( t ) is symmetric and H H H is a fixed symmetric matrix. Then Lip ( ∇ F ) ≤ L 0 \operatorname{Lip}(\nabla F)\le L_{0} Lip ( ∇ F ) ≤ L 0 and F ( x + v ) ≤ F ( x ) + ∇ F ( x ) T v + 1 2 v T H v F(x+v)\le F(x)+\nabla F(x)^{\mathsf{T}}v+\frac{1}{2}v^{\mathsf{T}}Hv F ( x + v ) ≤ F ( x ) + ∇ F ( x ) T v + 2 1 v T H v .
Proof. Apply the fundamental theorem of calculus on each interval and add; the endpoint terms cancel by continuity of the gradient. This gives ∥ ∇ F ( x + t v ) − ∇ F ( x ) ∥ ≤ L 0 t ∥ v ∥ \lVert\nabla F(x+tv)-\nabla F(x)\rVert\le L_{0}t\lVert v\rVert ∥ ∇ F ( x + t v ) − ∇ F ( x )∥ ≤ L 0 t ∥ v ∥ and v T [ ∇ F ( x + t v ) − ∇ F ( x ) ] ≤ t v T H v v^{\mathsf{T}}[\nabla F(x+tv)-\nabla F(x)]\le tv^{\mathsf{T}}Hv v T [ ∇ F ( x + t v ) − ∇ F ( x )] ≤ t v T H v . Integrating the second inequality over t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] proves the upper model. □
The lemma applies to every function built from (37) , ρ R \rho_R ρ R , and (42) . On a segment, each compressed coordinate has the form ( α + β t ) / ( 1 + ∥ x + t v ∥ 2 / R 2 ) 1 / 2 (\alpha+\beta t)/(1+\lVert x+tv\rVert^{2}/R^{2})^{1/2} ( α + β t ) / ( 1 + ∥ x + t v ∥ 2 / R 2 ) 1/2 . Squaring shows that it equals a nonzero scalar junction value at most twice, unless it does so identically; in that case its derivative along the segment vanishes, and either adjacent one-sided second derivative gives the correct expression from the chain rule. Crossings of a Huber junction sphere are roots of a quadratic polynomial as well. The resulting finite partition makes the gradient smooth on each open interval.
Sublevel parameter choice
Theorem D.4 (Wrapper with a sublevel trace bound). Let m ≥ 1 m\ge1 m ≥ 1 and R 2 ≥ 64 m R^{2}\ge64m R 2 ≥ 64 m . Set r 0 = 2 3 R r_{0}=\frac{2}{3}R r 0 = 3 2 R , μ = 11 m / R 2 \mu=11m/R^{2} μ = 11 m / R 2 , and define a public radial potential q ( x ) = Q ( ∥ x ∥ ) q(x)=Q(\lVert x\rVert) q ( x ) = Q (∥ x ∥) by
Q ( 0 ) = 0 , Q ′ ( r ) = μ min { r , r 0 } + min { ( r − r 0 ) + , 1 } . (44) Q(0)=0,\qquad Q'(r)=\mu\min\{r,r_{0}\}+\min\{(r-r_{0})_{+},1\}. \tag*{(44)} Q ( 0 ) = 0 , Q ′ ( r ) = μ min { r , r 0 } + min {( r − r 0 ) + , 1 } . (44) For an orthonormal frame U U U set
F U ( x ) = g m ( U T ρ R ( x ) ) + q ( x ) . F_{U}(x)=g_{m}(U^{\mathsf{T}}\rho_{R}(x))+q(x). F U ( x ) = g m ( U T ρ R ( x )) + q ( x ) . Then F U F_{U} F U is coercive, nonconvex, and C 1 , 1 C^{1,1} C 1 , 1 , and
Lip ( ∇ F U ) < 45 , ∥ ∇ F U ∥ ≤ 12 m , (45) \operatorname{Lip}(\nabla F_{U})<45,\qquad\lVert\nabla F_{U}\rVert\le12\sqrt{m}, \tag*{(45)} Lip ( ∇ F U ) < 45 , ∥ ∇ F U ∥ ≤ 12 m , (45) F U ( 0 ) − inf F U ≤ 9 4 ( m − 1 2 ) , { F U ≤ F U ( 0 ) } ⊂ { ∥ x ∥ < 3 2 R } . (46) F_{U}(0)-\inf F_{U}\le\frac{9}{4}\left(m-\frac{1}{2}\right),\qquad\{F_{U}\le F_{U}(0)\}\subset\{\lVert x\rVert<\tfrac{3}{2}R\}. \tag*{(46)} F U ( 0 ) − inf F U ≤ 4 9 ( m − 2 1 ) , { F U ≤ F U ( 0 )} ⊂ {∥ x ∥ < 2 3 R } . (46) On the initial sublevel, almost everywhere, one fixed matrix satisfies
∇ 2 F U ⪯ K U : = 54 U U T + 80 m R 2 I , ∥ K U ∥ op < 56. (47) \nabla^{2}F_{U}\preceq K_{U}:=54UU^{\mathsf{T}}+\frac{80m}{R^{2}}I,\qquad\lVert K_{U}\rVert_{\operatorname{op}}<56. \tag*{(47)} ∇ 2 F U ⪯ K U := 54 U U T + R 2 80 m I , ∥ K U ∥ op < 56. (47) Consequently,
tr [ ( ∇ 2 F U ) + ] ≤ 54 m + 80 d m R 2 . (48) \operatorname{tr}[(\nabla^{2}F_{U})_{+}]\le54m+\frac{80dm}{R^{2}}. \tag*{(48)} tr [( ∇ 2 F U ) + ] ≤ 54 m + R 2 80 d m . (48) If any compressed coordinate has magnitude below one, then at any x ∈ R d x\in\mathbb{R}^{d} x ∈ R d ,
∥ ∇ F U ( x ) ∥ > 3 4 . (49) \lVert\nabla F_{U}(x)\rVert>\frac{3}{4}. \tag*{(49)} ∥ ∇ F U ( x )∥ > 4 3 . (49) Finally its exact prefix replies have the following form: if y = U T ρ R ( x ) y=U^{\mathsf{T}}\rho_{R}(x) y = U T ρ R ( x ) , y j = 0 y_{j}=0 y j = 0 , and ∣ y k ∣ ≤ 1 / 2 \lvert y_{k}\rvert\le1/2 ∣ y k ∣ ≤ 1/2 for k > j k>j k > j , then, with r j ( x ) = g m ( y 1 , … , y j − 1 , 0 , … , 0 ) + q ( x ) r_{j}(x)=g_{m}(y_{1},\ldots,y_{j-1},0,\ldots,0)+q(x) r j ( x ) = g m ( y 1 , … , y j − 1 , 0 , … , 0 ) + q ( x ) ,
F U ( x ) = r j ( x ) , ∇ F U ( x ) = ∇ r j ( x ) − a ( x ) γ j − 1 ( x ) u j . (50) F_{U}(x)=r_{j}(x),\qquad\nabla F_{U}(x)=\nabla r_{j}(x)-a(x)\gamma_{j-1}(x)u_{j}. \tag*{(50)} F U ( x ) = r j ( x ) , ∇ F U ( x ) = ∇ r j ( x ) − a ( x ) γ j − 1 ( x ) u j . (50)
Proof. We have R ≥ 8 m ≥ 8 R \ge8\sqrt{m} \ge8 R ≥ 8 m ≥ 8 , μ ≤ 11 / 64 \mu\le11/64 μ ≤ 11/64 , and μ r 0 = 22 m / ( 3 R ) ≤ ( 11 / 12 ) m < m \mu r_0 = 22m/(3R) \le(11/12)\sqrt{m} < \sqrt{m} μ r 0 = 22 m / ( 3 R ) ≤ ( 11/12 ) m < m . The derivative Q ′ Q' Q ′ is continuous, nonnegative and nondecreasing, with slopes μ \mu μ , 1 1 1 , and 0 0 0 on its successive intervals. Also Q ′ ( r ) ≤ r Q'(r) \le r Q ′ ( r ) ≤ r , and its final value is μ r 0 + 1 < 2 m \mu r_0 + 1 < 2\sqrt{m} μ r 0 + 1 < 2 m . The radial and tangential Hessian eigenvalues of q q q are Q ′ ′ ( r ) Q''(r) Q ′′ ( r ) and Q ′ ( r ) / r Q'(r)/r Q ′ ( r ) / r , so its gradient is globally 1 1 1 -Lipschitz and bounded by 2 m 2\sqrt{m} 2 m . Thus
Lip ( ∇ F U ) ≤ 42 + 20 m 3 R + 1 ≤ 43 + 5 2 3 < 45. \operatorname{Lip}(\nabla F_U) \le42 + \frac{20\sqrt{m}}{\sqrt{3}R} + 1 \le43 + \frac{5}{2\sqrt{3}} < 45. Lip ( ∇ F U ) ≤ 42 + 3 R 20 m + 1 ≤ 43 + 2 3 5 < 45.
The gradient bound, gap, and coercivity follow from the chain bounds, q ≥ 0 q \ge0 q ≥ 0 , and its positive linear growth. Moreover Q ( r 0 ) = μ r 0 2 / 2 = 22 m / 9 Q(r_0) = \mu r_0^2/2 = 22m/9 Q ( r 0 ) = μ r 0 2 /2 = 22 m /9 , so for r ≥ r 0 r \ge r_0 r ≥ r 0 , F U ( x ) ≥ − 9 m / 4 + 22 m / 9 = 7 m / 36 > 0 > F U ( 0 ) = − 9 / 8 F_U(x) \ge-9m/4 + 22m/9 = 7m/36 > 0 > F_U(0) = -9/8 F U ( x ) ≥ − 9 m /4 + 22 m /9 = 7 m /36 > 0 > F U ( 0 ) = − 9/8 . This proves the sublevel inclusion. Consequently throughout the indicated sublevel the penalty is μ ∥ x ∥ 2 / 2 \mu\lVert x\rVert^2/2 μ ∥ x ∥ 2 /2 .
For the sublevel Hessian bound write p = ∇ g m ( y ) p = \nabla g_m(y) p = ∇ g m ( y ) , B = ∇ 2 g m ( y ) B = \nabla^2 g_m(y) B = ∇ 2 g m ( y ) , α = U T v \alpha= U^\mathsf{T}v α = U T v , and β = a 2 x T v / R 2 \beta= a^2x^\mathsf{T}v/R^2 β = a 2 x T v / R 2 . Then U T D ρ R v = a α − β y U^\mathsf{T}D\rho_Rv = a\alpha- \beta y U T D ρ R v = a α − β y , and on { ∥ x ∥ < 2 3 R } \{\lVert x\rVert< \frac{2}{3}R\} {∥ x ∥ < 3 2 R } one has ∣ β ∣ ≤ 1 + 4 / 9 1 + 4 / 9 ∥ v ∥ R = 6 13 ∥ v ∥ R |\beta| \le\frac{1+4/9}{1+4/9}\frac{\lVert v\rVert}{R} = \frac{6}{13}\frac{\lVert v\rVert}{R} ∣ β ∣ ≤ 1 + 4/9 1 + 4/9 R ∥ v ∥ = 13 6 R ∥ v ∥ and a 2 ≥ 9 13 a^2 \ge\frac{9}{13} a 2 ≥ 13 9 . The second term of the chain rule for the Hessian equals
⟨ U p , D 2 ρ R [ v , v ] ⟩ = − 2 a β p T α + ( 3 β 2 − a 2 ∥ v ∥ 2 / R 2 ) y T p . \left\langle Up,D^2\rho_R[v,v]\right\rangle= -2a\beta p^\mathsf{T}\alpha+ (3\beta^2-a^2\lVert v\rVert^2/R^2)y^\mathsf{T}p. ⟨ U p , D 2 ρ R [ v , v ] ⟩ = − 2 a β p T α + ( 3 β 2 − a 2 ∥ v ∥ 2 / R 2 ) y T p .
Since 0 ≤ 3 β 2 ≤ 108 169 ∥ v ∥ 2 R 2 < 9 13 ∥ v ∥ 2 R 2 ≤ a 2 ∥ v ∥ 2 R 2 ≤ ∥ v ∥ 2 R 2 0 \le3\beta^2 \le\frac{108}{169}\frac{\lVert v\rVert^2}{R^2} < \frac{9}{13}\frac{\lVert v\rVert^2}{R^2} \le\frac{a^2\lVert v\rVert^2}{R^2} \le\frac{\lVert v\rVert^2}{R^2} 0 ≤ 3 β 2 ≤ 169 108 R 2 ∥ v ∥ 2 < 13 9 R 2 ∥ v ∥ 2 ≤ R 2 a 2 ∥ v ∥ 2 ≤ R 2 ∥ v ∥ 2 , we have ∣ 3 β 2 − a 2 ∥ v ∥ 2 / R 2 ∣ ≤ ∥ v ∥ 2 / R 2 |3\beta^2-a^2\lVert v\rVert^2/R^2| \le\lVert v\rVert^2/R^2 ∣3 β 2 − a 2 ∥ v ∥ 2 / R 2 ∣ ≤ ∥ v ∥ 2 / R 2 . Combining the cross terms − 2 a β α T ( B y + p ) -2a\beta\alpha^\mathsf{T}(By+p) − 2 a β α T ( B y + p ) via Young’s inequality with parameter γ = 2 / 9 \gamma=2/9 γ = 2/9 and using ∥ B ∥ o p ≤ 42 \lVert B\rVert_{\mathrm{op}}\le42 ∥ B ∥ op ≤ 42 , ∥ B y ∥ + ∥ p ∥ ≤ 54 m \lVert By\rVert+\lVert p\rVert\le54\sqrt{m} ∥ B y ∥ + ∥ p ∥ ≤ 54 m , ∣ y T B y ∣ ≤ 44 m |y^\mathsf{T}By|\le44m ∣ y T B y ∣ ≤ 44 m , and ∣ y T p ∣ ≤ 31 m / 4 |y^\mathsf{T}p|\le31m/4 ∣ y T p ∣ ≤ 31 m /4 gives
( a α − β y ) T B ( a α − β y ) + ⟨ U p , D 2 ρ R [ v , v ] ⟩ ≤ ( 42 + 54 γ ) ∥ α ∥ 2 + [ ( 54 γ + 44 ) 36 169 + 31 4 ] m R 2 ∥ v ∥ 2 . (a\alpha-\beta y)^\mathsf{T}B(a\alpha-\beta y)+\left\langle Up,D^2\rho_R[v,v]\right\rangle
\le(42+54\gamma)\lVert\alpha\rVert^2+\left[\left(\frac{54}{\gamma}+44\right)\frac{36}{169}+\frac{31}{4}\right]\frac{m}{R^2}\lVert v\rVert^2. ( a α − β y ) T B ( a α − β y ) + ⟨ U p , D 2 ρ R [ v , v ] ⟩ ≤ ( 42 + 54 γ ) ∥ α ∥ 2 + [ ( γ 54 + 44 ) 169 36 + 4 31 ] R 2 m ∥ v ∥ 2 .
Adding μ I = ( 11 m / R 2 ) I \mu I=(11m/R^2)I μ I = ( 11 m / R 2 ) I and noting that 42 + 54 ( 2 / 9 ) = 54 42+54(2/9)=54 42 + 54 ( 2/9 ) = 54 and 11 + 287 ( 36 / 169 ) + 31 / 4 = 54003 / 676 < 80 11+287(36/169)+31/4=54003/676<80 11 + 287 ( 36/169 ) + 31/4 = 54003/676 < 80 gives ∇ 2 F U ⪯ 54 U U T + ( 80 m / R 2 ) I \nabla^2F_U\preceq54UU^\mathsf{T}+(80m/R^2)I ∇ 2 F U ⪯ 54 U U T + ( 80 m / R 2 ) I on the initial sublevel, with operator norm at most 54 + 80 / 64 < 56 54+80/64<56 54 + 80/64 < 56 . The same calculation bounds the absolute value of the quadratic form. For symmetric A ⪯ H A\preceq H A ⪯ H , H ⪰ 0 H\succeq0 H ⪰ 0 , one has tr A + ≤ tr H \operatorname{tr}A_+\le\operatorname{tr}H tr A + ≤ tr H : trace against the positive spectral projection of A A A . This proves (48) . For global smoothness one may directly use ∇ ( g m ∘ U T ρ R ) = D ρ R T U ∇ g m ( U T ρ R ) \nabla(g_m\circ U^\mathsf{T}\rho_R)=D\rho_R^\mathsf{T}U\nabla g_m(U^\mathsf{T}\rho_R) ∇ ( g m ∘ U T ρ R ) = D ρ R T U ∇ g m ( U T ρ R ) : D ρ R D\rho_R D ρ R has norm at most one and Lipschitz constant 2 / ( 3 R ) 2/(\sqrt{3}R) 2/ ( 3 R ) , while ∇ g m \nabla g_m ∇ g m is bounded and 42 42 42 -Lipschitz. The product rule for differences gives the displayed global bound. For the trace estimate, the finitely many scalar junction hyperplanes have null preimages under U T ρ R U^\mathsf{T}\rho_R U T ρ R , a smooth submersion; the radial junction spheres are null as well.
For the barrier on the whole space, let j j j be the first unfinished coordinate, so p j ≤ − 1 p_j\le-1 p j ≤ − 1 and ∣ y j ∣ < 1 |y_j|<1 ∣ y j ∣ < 1 . Write r = ∥ x ∥ r=\lVert x\rVert r = ∥ x ∥ , w = ∥ ρ R ( x ) ∥ = a r w=\lVert\rho_R(x)\rVert=ar w = ∥ ρ R ( x )∥ = a r , and g = ∇ F U ( x ) = D ρ R ( x ) U p + Q ′ ( r ) r x g=\nabla F_U(x)=D\rho_R(x)Up+\frac{Q'(r)}{r}x g = ∇ F U ( x ) = D ρ R ( x ) U p + r Q ′ ( r ) x . If w 2 ≤ 13 4 m w^2\le\frac{13}{4}m w 2 ≤ 4 13 m , then since w ( r 0 ) 2 = 4 13 R 2 ≥ 256 13 m > 13 4 m w(r_0)^2=\frac{4}{13}R^2\ge\frac{256}{13}m>\frac{13}{4}m w ( r 0 ) 2 = 13 4 R 2 ≥ 13 256 m > 4 13 m , we have r < r 0 r<r_0 r < r 0 , Q ′ ( r ) / r = μ ≤ 11 / 64 Q'(r)/r=\mu\le11/64 Q ′ ( r ) / r = μ ≤ 11/64 , and a 2 = 1 − w 2 / R 2 ≥ 243 / 256 a^2=1-w^2/R^2\ge243/256 a 2 = 1 − w 2 / R 2 ≥ 243/256 . Since D ρ R ( x ) − 1 u j = a − 1 ( u j + a − 1 y j x / R 2 ) D\rho_R(x)^{-1}u_j=a^{-1}(u_j+a^{-1}y_jx/R^2) D ρ R ( x ) − 1 u j = a − 1 ( u j + a − 1 y j x / R 2 ) satisfies ⟨ D ρ R ( x ) − 1 u j , x ⟩ = a − 4 y j \langle D\rho_R(x)^{-1}u_j,x\rangle=a^{-4}y_j ⟨ D ρ R ( x ) − 1 u j , x ⟩ = a − 4 y j ,
∣ ⟨ D ρ R ( x ) − 1 u j , g ⟩ ∣ = ∣ p j + a − 4 μ y j ∣ ≥ 1 − 11 64 ( 256 243 ) 2 , (51) \left|\left\langle D\rho_R(x)^{-1}u_j,g\right\rangle\right|=\left|p_j+a^{-4}\mu y_j\right|\ge1-\frac{11}{64}\left(\frac{256}{243}\right)^2, \tag*{(51)} ⟨ D ρ R ( x ) − 1 u j , g ⟩ = p j + a − 4 μ y j ≥ 1 − 64 11 ( 243 256 ) 2 , (51)
while ∥ D ρ R ( x ) − 1 u j ∥ 2 = a − 2 [ 1 + ( 1 + a 2 ) a − 4 y j 2 / R 2 ] ≤ 256 243 [ 1 + 2 4 ( 256 243 ) 2 ] \lVert D\rho_R(x)^{-1}u_j\rVert^2=a^{-2}[1+(1+a^2)a^{-4}y_j^2/R^2]\le\frac{256}{243}[1+\frac{2}{4}(\frac{256}{243})^2] ∥ D ρ R ( x ) − 1 u j ∥ 2 = a − 2 [ 1 + ( 1 + a 2 ) a − 4 y j 2 / R 2 ] ≤ 243 256 [ 1 + 4 2 ( 243 256 ) 2 ] because 1 + a 2 ≤ 2 1+a^2\le2 1 + a 2 ≤ 2 and y j 2 / R 2 < 1 / 64 y_j^2/R^2<1/64 y j 2 / R 2 < 1/64 , giving ∥ g ∥ > 3 / 4 \lVert g\rVert>3/4 ∥ g ∥ > 3/4 . Next suppose w 2 ≥ 13 4 m w^2\ge\frac{13}{4}m w 2 ≥ 4 13 m and r ≤ r 0 + 1 r\le r_0+1 r ≤ r 0 + 1 . Let I = { i : ∣ y i ∣ ≤ 5 / 4 } I=\{i:|y_i|\le5/4\} I = { i : ∣ y i ∣ ≤ 5/4 } and W = span { u i : i ∈ I } W=\operatorname{span}\{u_i:i\in I\} W = span { u i : i ∈ I } . Since g = a U p + c x g=aUp+cx g = a U p + c x for a scalar c c c and D 2 : = ∥ P W ⊥ ρ R ( x ) ∥ 2 ≥ w 2 − 25 16 m D^2:=\lVert P_{W^\perp}\rho_R(x)\rVert^2\ge w^2-\frac{25}{16}m D 2 := ∥ P W ⊥ ρ R ( x ) ∥ 2 ≥ w 2 − 16 25 m , projecting g g g onto the orthogonal directions u j u_j u j and P W ⊥ x P_{W^\perp}x P W ⊥ x gives
∥ g ∥ 2 ≥ a 2 p j 2 D 2 D 2 + y j 2 ≥ ( 1 − w 2 R 2 ) w 2 / m − 25 / 16 w 2 / m − 9 / 16 . (52) \lVert g\rVert^2\ge\frac{a^2p_j^2D^2}{D^2+y_j^2}\ge\left(1-\frac{w^2}{R^2}\right)\frac{w^2/m-25/16}{w^2/m-9/16}. \tag*{(52)} ∥ g ∥ 2 ≥ D 2 + y j 2 a 2 p j 2 D 2 ≥ ( 1 − R 2 w 2 ) w 2 / m − 9/16 w 2 / m − 25/16 . (52)
The right-hand side is concave in w 2 w^2 w 2 on [ 13 4 m , w ( r 0 + 1 ) 2 ] [\frac{13}{4}m,w(r_0+1)^2] [ 4 13 m , w ( r 0 + 1 ) 2 ] . At w 2 = 13 4 m w^2=\frac{13}{4}m w 2 = 4 13 m it is at least 243 256 ⋅ 27 43 = 6561 11008 > 9 16 \frac{243}{256}\cdot\frac{27}{43}=\frac{6561}{11008}>\frac{9}{16} 256 243 ⋅ 43 27 = 11008 6561 > 16 9 ; at r = r 0 + 1 r=r_0+1 r = r 0 + 1 , since r / R ≤ 2 / 3 + 1 / 8 = 19 / 24 r/R\le2/3+1/8=19/24 r / R ≤ 2/3 + 1/8 = 19/24 and w ( r 0 + 1 ) 2 ≥ w ( r 0 ) 2 ≥ 13 4 m w(r_0+1)^2\ge w(r_0)^2\ge\frac{13}{4}m w ( r 0 + 1 ) 2 ≥ w ( r 0 ) 2 ≥ 4 13 m , it is at least 576 937 ⋅ 3771 3979 > 9 16 \frac{576}{937}\cdot\frac{3771}{3979}>\frac{9}{16} 937 576 ⋅ 3979 3771 > 16 9 . Finally, if r ≥ r 0 + 1 r\ge r_0+1 r ≥ r 0 + 1 , then Q ′ ( r ) ≥ 1 Q'(r)\ge1 Q ′ ( r ) ≥ 1 , w ≥ 16 13 m w\ge\frac{16}{\sqrt{13}}\sqrt{m} w ≥ 13 16 m , and v : = D ρ R ( x ) U p v:=D\rho_R(x)U p v := D ρ R ( x ) U p satisfies ∣ ⟨ x / r , v ⟩ ∣ ≤ ( 5 / 4 ) m w ∥ v ∥ \left|\left\langle x/r,v\right\rangle\right|\le\frac{(5/4)\sqrt{m}}{w}\|v\| ∣ ⟨ x / r , v ⟩ ∣ ≤ w ( 5/4 ) m ∥ v ∥ because D ρ R D\rho_R D ρ R shrinks the radial component relative to the tangential component; projecting g = v + Q ′ ( r ) x / r g=v+Q'(r)x/r g = v + Q ′ ( r ) x / r onto v ⊥ v^\perp v ⊥ gives ∥ g ∥ 2 ≥ 1 − 25 m 16 w 2 ≥ 3771 4096 > 9 16 \|g\|^2\ge1-\frac{25m}{16w^2}\ge\frac{3771}{4096}>\frac{9}{16} ∥ g ∥ 2 ≥ 1 − 16 w 2 25 m ≥ 4096 3771 > 16 9 .
To verify nonconvexity, choose the point on the negative u 1 u_1 u 1 axis with y 1 = − 9 / 8 y_1=-9/8 y 1 = − 9/8 . Since R ≥ 8 R\ge8 R ≥ 8 , it lies in { ∥ x ∥ < r 0 } \{\|x\|<r_0\} { ∥ x ∥ < r 0 } with a 2 ≥ 1 − 81 / 4096 > 31 / 32 a^2\ge1-81/4096>31/32 a 2 ≥ 1 − 81/4096 > 31/32 , where the second axis derivative is at most − 4 ( 31 / 32 ) 3 + 11 / 64 < 0 -4(31/32)^3+11/64<0 − 4 ( 31/32 ) 3 + 11/64 < 0 . Finally, after a zero coordinate every later gate and its derivative vanishes. The chain derivative at coordinate j j j is − γ j − 1 -\gamma_{j-1} − γ j − 1 . Since y j = 0 y_j=0 y j = 0 implies u j T x = 0 u_j^\mathsf{T}x=0 u j T x = 0 , D ρ R u j = a u j D\rho_Ru_j=au_j D ρ R u j = a u j . The radial term is public and cancels from prefix differences, proving (50) .
Scaling, Dimension Floor, and Deterministic Lower Bounds
Lemma E.1 (One scaling rule). Let Λ , b > 0 \Lambda,b>0 Λ , b > 0 , and let F F F have gradient Lipschitz constant at most Λ \Lambda Λ , initial gap at most D D D , and one fixed positive semidefinite Hessian upper matrix K K K of operator norm at most Λ \Lambda Λ , either globally or on its initial sublevel. Assume ∥ ∇ F ∥ > b \|\nabla F\|>b ∥∇ F ∥ > b at the points used in a lower bound. For 0 < L ′ ≤ L 0<L'\le L 0 < L ′ ≤ L set
σ = Λ ε L ′ b , α = Λ ε 2 L ′ b 2 , f ( x ) = α F ( x / σ ) . (53) \sigma=\frac{\Lambda\varepsilon}{L'b},\qquad\alpha=\frac{\Lambda\varepsilon^2}{L'b^2},\qquad f(x)=\alpha F(x/\sigma). \tag*{(53)} σ = L ′ b Λ ε , α = L ′ b 2 Λ ε 2 , f ( x ) = α F ( x / σ ) . (53) If Λ D / b 2 ≤ L ′ Δ / ε 2 \Lambda D/b^2\le L'\Delta/\varepsilon^2 Λ D / b 2 ≤ L ′ Δ/ ε 2 and L ′ tr K / Λ ≤ L E L'\operatorname{tr}K/\Lambda\le LE L ′ tr K /Λ ≤ L E , then f ∈ B f\in\mathcal{B} f ∈ B , or f ∈ F f\in\mathcal{F} f ∈ F when the matrix bound is global. The barrier becomes strictly greater than ε \varepsilon ε and all oracle costs are preserved.
Proof. The gradient multiplier is α / σ = ε / b \alpha/\sigma=\varepsilon/b α / σ = ε / b , the Hessian multiplier is α / σ 2 = L ′ / Λ \alpha/\sigma^2=L'/\Lambda α / σ 2 = L ′ /Λ , and the gap is at most α D ≤ Δ \alpha D\le\Delta α D ≤ Δ . The fixed upper matrix becomes ( L ′ / Λ ) K (L'/\Lambda)K ( L ′ /Λ ) K , whose norm is at most L ′ ≤ L L'\le L L ′ ≤ L and whose trace satisfies the hypothesis. Positive scaling maps initial sublevels exactly. A call ( x , V ) (x,V) ( x , V ) is simulated by ( x / σ , V ) (x/\sigma,V) ( x / σ , V ) on F F F , multiplying values by α \alpha α and derivative projections by α / σ \alpha/\sigma α / σ ; its rank and cost are unchanged.
Lemma E.2 (Dimension floor). For T > 1 / 2 T>1/2 T > 1/2 and 1 ≤ E ≤ d 1\le E\le d 1 ≤ E ≤ d , N d e t l i n ≥ d N_{\mathrm{det}}^{\mathrm{lin}}\ge d N det lin ≥ d , already on the global fixed-matrix class F \mathcal{F} F .
Proof. Let r = 2 Δ / L r=\sqrt{2\Delta/L} r = 2Δ/ L and f u ( x ) = L 2 ( ⟨ u , x ⟩ − r ) 2 f_u(x)=\frac{L}{2}(\langle u,x\rangle-r)^2 f u ( x ) = 2 L (⟨ u , x ⟩ − r ) 2 for a unit vector u u u . Simulate the reply ( Δ , 0 ) (\Delta,0) ( Δ , 0 ) against a deterministic algorithm truncated below cost d d d . The span of all query points and requested column spaces has dimension strictly less than d d d . Choose a perpendicular unit u u u , with its sign arranged so that ⟨ u , x ^ ⟩ ≤ 0 \langle u,\hat{x}\rangle\le0 ⟨ u , x ^ ⟩ ≤ 0 . The function f u f_u f u reproduces all replies, has upper matrix L u u T Luu^\mathsf{T} Lu u T , and its output gradient norm is at least L r = 2 T ε > ε Lr=\sqrt{2T}\,\varepsilon>\varepsilon L r = 2 T ε > ε . Truncation is justified as in Lemma C.4 .
Theorem E.3 (Multistage lower bound on the main class). If an integer m ≥ 2 m\ge2 m ≥ 2 satisfies
224 m ≤ T , 224 2 m 2 log ( e + m ) ≤ E T , 10000 m log ( e + m ) ≤ d (54) 224m\le T,\qquad224^2m^2\log(e+m)\le ET,\qquad10000m\log(e+m)\le d \tag*{(54)} 224 m ≤ T , 22 4 2 m 2 log ( e + m ) ≤ E T , 10000 m log ( e + m ) ≤ d (54) then both deterministic oracle costs on B \mathcal{B} B are at least ⌊ d / 2 ⌋ m \lfloor d/2\rfloor m ⌊ d /2 ⌋ m . The witnesses are C 1 , 1 C^{1,1} C 1 , 1 , globally L L L -smooth, coercive, nonconvex, and globally 2 L Δ 2\sqrt{L\Delta} 2 L Δ -Lipschitz.
Proof. Set ℓ = log ( e + m ) \ell=\log(e+m) ℓ = log ( e + m ) , R 2 = d / ( 156 ℓ ) R^2=d/(156\ell) R 2 = d / ( 156 ℓ ) , h = ⌊ d / 2 ⌋ h=\lfloor d/2\rfloor h = ⌊ d /2 ⌋ , p = d − m − h + 1 p=d-m-h+1 p = d − m − h + 1 , and use Theorem D.4 . Since 156 ⋅ 64 = 9984 < 10000 156\cdot64=9984<10000 156 ⋅ 64 = 9984 < 10000 , the last inequality of (54) implies R 2 ≥ 64 m R^2\ge64m R 2 ≥ 64 m , m ≤ d / 128 m\le d/128 m ≤ d /128 , and
p ≥ 63 d / 128 p \ge63d/128 p ≥ 63 d /128 . For a list of M ≤ h m M \le hm M ≤ hm past points, M / p ≤ 64 m / 63 M/p \le64m/63 M / p ≤ 64 m /63 . Since log ( e + c m ) ≤ c log ( e + m ) \log(e+cm) \le c\log(e+m) log ( e + c m ) ≤ c log ( e + m ) when c ≥ 1 c \ge1 c ≥ 1 , Lemma C.2 gives squared projection bound at most
( 13 / 3 ) 2 R 2 p log ( e + M / p ) ≤ 169 ⋅ 128 ⋅ 64 9 ⋅ 63 2 ⋅ 156 = 26624 107163 < 1 4 . \frac{(13/3)^2R^2}{p}\log(e+M/p) \le\frac{169\cdot128\cdot64}{9\cdot63^2\cdot156}=\frac{26624}{107163}<\frac{1}{4}. p ( 13/3 ) 2 R 2 log ( e + M / p ) ≤ 9 ⋅ 6 3 2 ⋅ 156 169 ⋅ 128 ⋅ 64 = 107163 26624 < 4 1 .
Thus the transcript lemma applies with τ = 1 / 2 \tau=1/2 τ = 1/2 and b = 3 / 4 b=3/4 b = 3/4 . For the fixed matrix (47) , since ℓ ≥ log ( e + 2 ) > 3 / 2 \ell\ge\log(e+2)>3/2 ℓ ≥ log ( e + 2 ) > 3/2 ,
tr K U 56 = 54 m + 80 ⋅ 156 m ℓ 56 < 3129 14 m ℓ < 224 m ℓ . \frac{\operatorname{tr}K_U}{56}=\frac{54m+80\cdot156m\ell}{56}<\frac{3129}{14}m\ell<224m\ell. 56 tr K U = 56 54 m + 80 ⋅ 156 m ℓ < 14 3129 m ℓ < 224 m ℓ .
Take L ′ = 224 m L / T L'=224mL/T L ′ = 224 m L / T , Λ = 56 \Lambda=56 Λ = 56 , and apply Lemma E.1. We have L ′ ≤ L L'\le L L ′ ≤ L , and 56 ( 9 / 4 ) / ( 3 / 4 ) 2 = 224 56(9/4)/(3/4)^2=224 56 ( 9/4 ) / ( 3/4 ) 2 = 224 , so the condition on the initial gap holds. The trace becomes at most L ′ 224 m ℓ ≤ L E L'\,224m\ell\le LE L ′ 224 m ℓ ≤ L E by the second inequality of (54) . The normalized matrix also has operator norm at most L ′ L' L ′ , so the instance lies in B \mathcal{B} B . The global gradient bound is ∥ ∇ f ∥ ≤ 16 m ε ≤ 2 L Δ \lVert\nabla f\rVert\le16\sqrt{m}\,\varepsilon\le2\sqrt{L\Delta} ∥ ∇ f ∥ ≤ 16 m ε ≤ 2 L Δ because T ≥ 224 m T\ge224m T ≥ 224 m . Positive scaling preserves coercivity and nonconvexity.
Corollary E.4 (Rounded main lower bound). If E T ≤ d ET\le d E T ≤ d , then
N d e t ≥ max { d , d 672 min { T , E T log ( e + T ) } } . (55) N_{\mathrm{det}}\ge\max\left\{d,\frac{d}{672}\min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\}\right\}. \tag*{(55)} N det ≥ max { d , 672 d min { T , log ( e + T ) E T } } . (55) Without the assumption E T ≤ d ET\le d E T ≤ d , E ≥ T log ( e + T ) E\ge T\log(e+T) E ≥ T log ( e + T ) implies N d e t ≥ d T / 30000 N_{\mathrm{det}}\ge dT/30000 N det ≥ d T /30000 .
Proof. Put M = min { T , E T / log ( e + T ) } M=\min\{T,\sqrt{ET/\log(e+T)}\} M = min { T , E T / log ( e + T ) } , z = M / 224 z=M/224 z = M /224 , and m = ⌊ z ⌋ m=\lfloor z\rfloor m = ⌊ z ⌋ . If z < 3 z<3 z < 3 , the dimension floor dominates d z / 3 dz/3 d z /3 . If z ≥ 3 z\ge3 z ≥ 3 , the first two stage conditions follow from 224 m ≤ M 224m\le M 224 m ≤ M and log ( e + m ) ≤ log ( e + T ) \log(e+m)\le\log(e+T) log ( e + m ) ≤ log ( e + T ) . The third follows from 10000 < 2 ⋅ 224 2 ≤ m 224 2 10000<2\cdot224^2\le m\,224^2 10000 < 2 ⋅ 22 4 2 ≤ m 22 4 2 and E T ≤ d ET\le d E T ≤ d . Moreover m / z ≥ 3 / 4 m/z\ge3/4 m / z ≥ 3/4 and h / d ≥ 15 / 32 h/d\ge15/32 h / d ≥ 15/32 , so h m ≥ ( 45 / 128 ) d z > d z / 3 = d M / 672 hm\ge(45/128)dz>dz/3=dM/672 hm ≥ ( 45/128 ) d z > d z /3 = d M /672 . For the final claim take m = ⌊ T / 10000 ⌋ m=\lfloor T/10000\rfloor m = ⌊ T /10000 ⌋ and apply the same rounding argument. Its stage conditions follow from T log ( e + T ) ≤ E ≤ d T\log(e+T)\le E\le d T log ( e + T ) ≤ E ≤ d and 224 2 < 10000 2 224^2<10000^2 22 4 2 < 1000 0 2 ; the dimension floor covers the case T / 10000 < 3 T/10000<3 T /10000 < 3 .
Corollary E.5 (Matched window and cost ratio). For E T ≤ d ET\le d E T ≤ d , write ℓ = log ( e + T ) \ell=\log(e+T) ℓ = log ( e + T ) . Then
d 15456 ( E + E T ) ℓ ≤ N d e t N r a n d c a p ≤ 1824 d E + E T . \frac{d}{15456(E+\sqrt{ET})\sqrt{\ell}}\le\frac{N_{\mathrm{det}}}{N_{\mathrm{rand}}^{\mathrm{cap}}}\le\frac{1824d}{E+\sqrt{ET}}. 15456 ( E + E T ) ℓ d ≤ N rand cap N det ≤ E + E T 1824 d . For T ℓ ≤ E ≤ d / T T\ell\le E\le d/T T ℓ ≤ E ≤ d / T , the ratio lies between d / ( 15456 E ) d/(15456E) d / ( 15456 E ) and 120 d / E 120d/E 120 d / E .
Proof. As E T ≤ d ET\le d E T ≤ d implies T ≤ d T\le d T ≤ d , the deterministic upper bound is at most d min { 10 T , 100 2 E T } d\min\{10T,100\sqrt{2}\sqrt{ET}\} d min { 10 T , 100 2 E T } , and
min { 10 T , 100 2 E T } ( E + E T ) ≤ 10 T E + 100 2 E T E T = ( 10 + 100 2 ) E T < 152 E T . \min\{10T,100\sqrt{2}\sqrt{ET}\}(E+\sqrt{ET})\le10TE+100\sqrt{2}\sqrt{ET}\sqrt{ET}=(10+100\sqrt{2})ET<152ET. min { 10 T , 100 2 E T } ( E + E T ) ≤ 10 T E + 100 2 E T E T = ( 10 + 100 2 ) E T < 152 E T . For the lower bound use a b / ( a + b ) ≤ min ( a , b ) ab/(a+b)\le\min(a,b) ab / ( a + b ) ≤ min ( a , b ) with a = T a=T a = T , b = E T b=\sqrt{ET} b = E T and min { T , E T / ℓ } ≥ min { T , E T } / ℓ \min\{T,\sqrt{ET}/\ell\}\ge\min\{T,\sqrt{ET}\}/\ell min { T , E T / ℓ } ≥ min { T , E T } / ℓ . Divide deterministic lower and upper bounds by 23 E T 23ET 23 E T and E T / 12 ET/12 E T /12 , respectively, noting that 672 ⋅ 23 = 15456 672\cdot23=15456 672 ⋅ 23 = 15456 and 12 ⋅ 152 = 1824 12\cdot152=1824 12 ⋅ 152 = 1824 . For the matched window use 10 d T 10dT 10 d T , d T / 672 dT/672 d T /672 and the random bounds directly.
Unrestricted parameter choice
Assume m ≥ 2 m \ge2 m ≥ 2 and put
R 2 = 5 m , r 0 2 = 4 m , r 1 2 = 9 2 m . R^2 = 5m, \qquad r_0^2 = 4m, \qquad r_1^2 = \frac{9}{2}m. R 2 = 5 m , r 0 2 = 4 m , r 1 2 = 2 9 m .
Let q q q be the shifted Huber potential (42) with these r 0 r_0 r 0 , with s = r 1 − r 0 s = r_1-r_0 s = r 1 − r 0 , and with λ = 1 / ( r 1 − r 0 ) \lambda= 1/(r_1-r_0) λ = 1/ ( r 1 − r 0 ) ; thus q q q vanishes for ∥ x ∥ ≤ r 0 \lVert x\rVert\le r_0 ∥ x ∥ ≤ r 0 , its radial derivative increases linearly from zero to one on [ r 0 , r 1 ] [r_0,r_1] [ r 0 , r 1 ] , and it equals one afterwards. For an orthonormal frame U = ( u 1 , … , u m ) U = (u_1,\ldots,u_m) U = ( u 1 , … , u m ) define
F ~ m , U ( x ) = g m ( U T ρ R ( x ) ) + q ( x ) . \widetilde{F}_{m,U}(x) = g_m(U^{\mathsf{T}}\rho_R(x)) + q(x). F m , U ( x ) = g m ( U T ρ R ( x )) + q ( x ) .
Lemma E.6 (Smoothness, gap, and nonconvexity). For every m ≥ 2 m \ge2 m ≥ 2 and every orthonormal U U U , the function F ~ m , U \widetilde{F}_{m,U} F m , U is coercive and nonconvex, and
Lip ( ∇ F ~ m , U ) < 44 , F ~ m , U ( 0 ) − inf F ~ m , U ≤ 9 4 ( m − 1 2 ) . \operatorname{Lip}(\nabla\widetilde{F}_{m,U}) < 44, \qquad\widetilde{F}_{m,U}(0) - \inf\widetilde{F}_{m,U} \le\frac{9}{4}\left(m-\frac{1}{2}\right). Lip ( ∇ F m , U ) < 44 , F m , U ( 0 ) − inf F m , U ≤ 4 9 ( m − 2 1 ) .
Proof. At m ≥ 2 m \ge2 m ≥ 2 one has r 1 − r 0 = ( 9 / 2 − 2 ) m ≥ 3 − 2 2 > 1 / 6 r_1-r_0 = (\sqrt{9/2}-2)\sqrt{m} \ge3-2\sqrt{2} > 1/6 r 1 − r 0 = ( 9/2 − 2 ) m ≥ 3 − 2 2 > 1/6 , so λ < 6 \lambda< 6 λ < 6 . Write t = ∥ x ∥ / R t = \lVert x\rVert/R t = ∥ x ∥ / R and a = ( 1 + t 2 ) − 1 / 2 a = (1+t^2)^{-1/2} a = ( 1 + t 2 ) − 1/2 . On every smooth piece of F ~ m , U \widetilde{F}_{m,U} F m , U , the chain Hessian has operator norm at most 42 a 2 + ∥ U p ∥ ∥ D 2 ρ R ( x ) ∥ ≤ 42 / ( 1 + t 2 ) + ( 30 / 5 ) t a 3 42a^2 + \lVert Up\rVert\lVert D^2\rho_R(x)\rVert\le42/(1+t^2) + (30/\sqrt{5})ta^3 42 a 2 + ∥ U p ∥ ∥ D 2 ρ R ( x )∥ ≤ 42/ ( 1 + t 2 ) + ( 30/ 5 ) t a 3 by Lemmas D.1 and D.2. For r < r 0 r<r_0 r < r 0 , ∇ 2 q ( x ) = 0 \nabla^2q(x)=0 ∇ 2 q ( x ) = 0 and
∥ ∇ 2 F ~ m , U ( x ) ∥ o p ≤ 42 + ( 30 / 5 ) t 1 + t 2 ≤ 42 + 225 42 ⋅ 5 = 603 14 < 44. \left\lVert\nabla^2\widetilde{F}_{m,U}(x)\right\rVert_{\mathrm{op}}
\le\frac{42+(30/\sqrt{5})t}{1+t^2}
\le42+\frac{225}{42\cdot5}
= \frac{603}{14}<44. ∇ 2 F m , U ( x ) op ≤ 1 + t 2 42 + ( 30/ 5 ) t ≤ 42 + 42 ⋅ 5 225 = 14 603 < 44.
For r > r 0 r>r_0 r > r 0 , t 2 ≥ 4 / 5 t^2\ge4/5 t 2 ≥ 4/5 , a 2 ≤ 5 / 9 a^2\le5/9 a 2 ≤ 5/9 , ∥ D 2 ρ R ∥ ≤ 2 / 15 m \lVert D^2\rho_R\rVert\le2/\sqrt{15m} ∥ D 2 ρ R ∥ ≤ 2/ 15 m , and ∥ ∇ 2 q ∥ o p ≤ λ < 6 \lVert\nabla^2q\rVert_{\mathrm{op}}\le\lambda<6 ∥ ∇ 2 q ∥ op ≤ λ < 6 , so
∥ ∇ 2 F ~ m , U ( x ) ∥ o p ≤ 42 ⋅ 5 9 + 20 15 + 6 < 44. \left\lVert\nabla^2\widetilde{F}_{m,U}(x)\right\rVert_{\mathrm{op}}
\le42\cdot\frac{5}{9}+\frac{20}{\sqrt{15}}+6<44. ∇ 2 F m , U ( x ) op ≤ 42 ⋅ 9 5 + 15 20 + 6 < 44.
Lemma D.3 integrates this piecewise Hessian bound to Lip ( ∇ F ~ m , U ) < 44 \operatorname{Lip}(\nabla\widetilde{F}_{m,U})<44 Lip ( ∇ F m , U ) < 44 . The bounded chain and the eventually linear growth of q q q give coercivity. Since q ≥ 0 q\ge0 q ≥ 0 and q ( 0 ) = 0 q(0)=0 q ( 0 ) = 0 , Lemma D.1 gives the gap.
For nonconvexity, take x x x on the negative u 1 u_1 u 1 axis with y 1 = − 9 / 8 y_1=-9/8 y 1 = − 9/8 and all other compressed coordinates zero. This point lies inside r 0 r_0 r 0 , where q = 0 q=0 q = 0 , and the gate ψ ( − y 1 ) \psi(-y_1) ψ ( − y 1 ) is saturated near it. Along the axis the only varying chain term is − φ ( y 1 ) -\varphi(y_1) − φ ( y 1 ) . There φ ′ ′ = 4 \varphi''=4 φ ′′ = 4 and φ ′ = 1 / 2 \varphi'=1/2 φ ′ = 1/2 , while the scalar compression s ↦ − s / 1 + s 2 / R 2 s\mapsto-s/\sqrt{1+s^2/R^2} s ↦ − s / 1 + s 2 / R 2 has positive second derivative for s > 0 s>0 s > 0 . The second derivative of the composite along the axis is therefore negative. □
Lemma E.7 (Gradient barrier on the whole space). If some coordinate of U T ρ R ( x ) U^{\mathsf{T}}\rho_R(x) U T ρ R ( x ) has magnitude less than one, then ∥ ∇ F ~ m , U ( x ) ∥ > 3 / 5 \lVert\nabla\widetilde{F}_{m,U}(x)\rVert>3/5 ∥ ∇ F m , U ( x )∥ > 3/5 .
Proof. Write r = ∥ x ∥ r=\lVert x\rVert r = ∥ x ∥ , w = ∥ ρ R ( x ) ∥ = a r w=\lVert\rho_R(x)\rVert=ar w = ∥ ρ R ( x )∥ = a r , y = U T ρ R ( x ) y=U^{\mathsf{T}}\rho_R(x) y = U T ρ R ( x ) , p = ∇ g m ( y ) p=\nabla g_m(y) p = ∇ g m ( y ) , and g = ∇ F ~ m , U ( x ) g=\nabla\widetilde{F}_{m,U}(x) g = ∇ F m , U ( x ) . Let j j j be the first index with ∣ y j ∣ < 1 \lvert y_j\rvert<1 ∣ y j ∣ < 1 , so that p j ≤ − 1 p_j\le-1 p j ≤ − 1 by Lemma D.1.
Inner ball, r ≤ r 0 r\le r_0 r ≤ r 0 . At x = 0 x=0 x = 0 one has g = U ∇ g m ( 0 ) = − u 1 g=U\nabla g_m(0)=-u_1 g = U ∇ g m ( 0 ) = − u 1 , of norm one, so assume x ≠ 0 x\ne0 x = 0 . Here g = D ρ R ( x ) U p g=D\rho_R(x)Up g = D ρ R ( x ) U p . With t = r / R t=r/R t = r / R , since t 2 ≤ 4 / 5 t^2\le4/5 t 2 ≤ 4/5 and R 2 ≥ 10 R^2\ge10 R 2 ≥ 10 ,
∥ D ρ R ( x ) − 1 u j ∥ 2 = a − 2 [ 1 + ( a − 4 − 1 ) y j 2 a 2 r 2 ] ≤ ( 1 + t 2 ) [ 1 + ( 2 + t 2 ) ( 1 + t 2 ) R 2 ] ≤ 1692 625 < 25 9 . \left\lVert D\rho_R(x)^{-1}u_j\right\rVert^2
= a^{-2}\left[1+(a^{-4}-1)\frac{y_j^2}{a^2r^2}\right]
\le(1+t^2)\left[1+\frac{(2+t^2)(1+t^2)}{R^2}\right]
\le\frac{1692}{625}<\frac{25}{9}. D ρ R ( x ) − 1 u j 2 = a − 2 [ 1 + ( a − 4 − 1 ) a 2 r 2 y j 2 ] ≤ ( 1 + t 2 ) [ 1 + R 2 ( 2 + t 2 ) ( 1 + t 2 ) ] ≤ 625 1692 < 9 25 .
Hence 1 ≤ ∣ p j ∣ = ∣ ⟨ D ρ R ( x ) − 1 u j , g ⟩ ∣ < ( 5 / 3 ) ∥ g ∥ 1\le\lvert p_j\rvert=\lvert\langle D\rho_R(x)^{-1}u_j,g\rangle\rvert<(5/3)\lVert g\rVert 1 ≤ ∣ p j ∣ = ∣⟨ D ρ R ( x ) − 1 u j , g ⟩∣ < ( 5/3 ) ∥ g ∥ , so ∥ g ∥ > 3 / 5 \lVert g\rVert>3/5 ∥ g ∥ > 3/5 . Transition annulus, r 0 ≤ r ≤ r 1 r_0 \le r \le r_1 r 0 ≤ r ≤ r 1 . Here w 2 / m ∈ [ 20 / 9 , 45 / 19 ] w^2/m \in[20/9,45/19] w 2 / m ∈ [ 20/9 , 45/19 ] and a 2 = 1 − w 2 / ( 5 m ) a^2 = 1-w^2/(5m) a 2 = 1 − w 2 / ( 5 m ) . Let I = { i : ∣ y i ∣ < 5 / 4 } I=\{i:|y_i|<5/4\} I = { i : ∣ y i ∣ < 5/4 } and W = span { u i : i ∈ I } W=\operatorname{span}\{u_i:i\in I\} W = span { u i : i ∈ I } . By Lemma D.1 , U p ∈ W Up\in W U p ∈ W and j ∈ I j\in I j ∈ I . Since ∣ y j ∣ < 1 |y_j|<1 ∣ y j ∣ < 1 and ∣ y i ∣ < 5 / 4 |y_i|<5/4 ∣ y i ∣ < 5/4 for the other indices in I I I , ∑ i ∈ I ∖ { j } y i 2 ≤ 25 16 ( m − 1 ) \sum_{i\in I\setminus\{j\}} y_i^2\le\frac{25}{16}(m-1) ∑ i ∈ I ∖ { j } y i 2 ≤ 16 25 ( m − 1 ) . With D = ∥ P W ⊥ ρ R ( x ) ∥ D=\lVert P_{W^\perp}\rho_R(x)\rVert D = ∥ P W ⊥ ρ R ( x )∥ , since m ≥ 2 m\ge2 m ≥ 2 and w 2 / m ≥ 20 / 9 > 25 / 16 w^2/m\ge20/9>25/16 w 2 / m ≥ 20/9 > 25/16 ,
D 2 + y j 2 = w 2 − ∑ i ∈ I ∖ { j } y i 2 ≥ w 2 − 25 16 ( m − 1 ) = 2 ( w 2 m − 25 32 ) + ( m − 2 ) ( w 2 m − 25 16 ) ≥ 2 ( w 2 m − 25 32 ) . D^2+y_j^2=w^2-\sum_{i\in I\setminus\{j\}}y_i^2\ge w^2-\frac{25}{16}(m-1)=2\left(\frac{w^2}{m}-\frac{25}{32}\right)+(m-2)\left(\frac{w^2}{m}-\frac{25}{16}\right)\ge2\left(\frac{w^2}{m}-\frac{25}{32}\right). D 2 + y j 2 = w 2 − i ∈ I ∖ { j } ∑ y i 2 ≥ w 2 − 16 25 ( m − 1 ) = 2 ( m w 2 − 32 25 ) + ( m − 2 ) ( m w 2 − 16 25 ) ≥ 2 ( m w 2 − 32 25 ) .
As y j 2 < 1 y_j^2<1 y j 2 < 1 , projecting g = a U p + c x g=aUp+cx g = a U p + c x onto u j u_j u j and P W ⊥ x P_{W^\perp}x P W ⊥ x gives
∥ g ∥ 2 ≥ a 2 p j 2 D 2 D 2 + y j 2 ≥ ( 1 − w 2 5 m ) ( 1 − 1 / 2 w 2 / m − 25 / 32 ) . (56) \lVert g\rVert^2\ge\frac{a^2p_j^2D^2}{D^2+y_j^2}\ge\left(1-\frac{w^2}{5m}\right)\left(1-\frac{1/2}{w^2/m-25/32}\right). \tag*{(56)} ∥ g ∥ 2 ≥ D 2 + y j 2 a 2 p j 2 D 2 ≥ ( 1 − 5 m w 2 ) ( 1 − w 2 / m − 25/32 1/2 ) . (56)
The right-hand side of (56) is concave in w 2 / m w^2/m w 2 / m on [ 20 / 9 , 45 / 19 ] [20/9,45/19] [ 20/9 , 45/19 ] , with endpoint values 271 / 747 > 9 / 25 271/747>9/25 271/747 > 9/25 and 1322 / 3667 > 9 / 25 1322/3667>9/25 1322/3667 > 9/25 , so ∥ g ∥ > 3 / 5 \lVert g\rVert>3/5 ∥ g ∥ > 3/5 .
Exterior, r ≥ r 1 r\ge r_1 r ≥ r 1 . Now ∇ q = n : = x / r \nabla q=n:=x/r ∇ q = n := x / r , w 2 ≥ 45 19 m w^2\ge\frac{45}{19}m w 2 ≥ 19 45 m , a 2 = 1 − w 2 / ( 5 m ) ≤ 10 19 a^2=1-w^2/(5m)\le\frac{10}{19} a 2 = 1 − w 2 / ( 5 m ) ≤ 19 10 , and ∥ P W ρ R ( x ) ∥ 2 = ∑ i ∈ I y i 2 < 25 16 m \lVert P_W\rho_R(x)\rVert^2=\sum_{i\in I}y_i^2<\frac{25}{16}m ∥ P W ρ R ( x ) ∥ 2 = ∑ i ∈ I y i 2 < 16 25 m . If p ≠ 0 p\ne0 p = 0 , put c = ∣ ⟨ n , U p ⟩ ∣ / ∥ U p ∥ ≤ ∥ P W n ∥ ≤ ( 25 / 16 ) m / w c=|\langle n,Up\rangle|/\lVert Up\rVert\le\lVert P_Wn\rVert\le\sqrt{(25/16)m}/w c = ∣ ⟨ n , U p ⟩ ∣/ ∥ U p ∥ ≤ ∥ P W n ∥ ≤ ( 25/16 ) m / w , so c 2 < 25 / 16 45 / 19 = 95 144 c^2<\frac{25/16}{45/19}=\frac{95}{144} c 2 < 45/19 25/16 = 144 95 . Since D ρ R ( x ) D\rho_R(x) D ρ R ( x ) has radial eigenvalue a 3 a^3 a 3 and tangential eigenvalue a a a , for v = D ρ R ( x ) U p v=D\rho_R(x)Up v = D ρ R ( x ) U p we have
∣ ⟨ n , v ⟩ ∣ 2 ∥ v ∥ 2 = a 4 c 2 1 − c 2 + a 4 c 2 ≤ 500 1431 , \frac{|\langle n,v\rangle|^2}{\lVert v\rVert^2}=\frac{a^4c^2}{1-c^2+a^4c^2}\le\frac{500}{1431}, ∥ v ∥ 2 ∣ ⟨ n , v ⟩ ∣ 2 = 1 − c 2 + a 4 c 2 a 4 c 2 ≤ 1431 500 ,
and projecting g = v + n g=v+n g = v + n onto v ⊥ v^\perp v ⊥ gives ∥ g ∥ 2 ≥ 1 − 500 1431 = 931 1431 > 9 25 \lVert g\rVert^2\ge1-\frac{500}{1431}=\frac{931}{1431}>\frac{9}{25} ∥ g ∥ 2 ≥ 1 − 1431 500 = 1431 931 > 25 9 . If p = 0 p=0 p = 0 , then ∥ g ∥ = 1 \lVert g\rVert=1 ∥ g ∥ = 1 .
When y j = 0 y_j=0 y j = 0 and ∣ y i ∣ ≤ 1 / 2 |y_i|\le1/2 ∣ y i ∣ ≤ 1/2 for all i > j i>j i > j , let
r ~ j ( x ) = g m ( u 1 ⊤ ρ R ( x ) , … , u j − 1 ⊤ ρ R ( x ) , 0 , … , 0 ) + q ( x ) . \widetilde r_j(x)=g_m(u_1^\top\rho_R(x),\ldots,u_{j-1}^\top\rho_R(x),0,\ldots,0)+q(x). r j ( x ) = g m ( u 1 ⊤ ρ R ( x ) , … , u j − 1 ⊤ ρ R ( x ) , 0 , … , 0 ) + q ( x ) .
Every later gate and its derivative vanish, p j = − γ j − 1 p_j=-\gamma_{j-1} p j = − γ j − 1 , and u j ⊤ x = 0 u_j^\top x=0 u j ⊤ x = 0 gives D ρ R ( x ) u j = a ( x ) u j D\rho_R(x)u_j=a(x)u_j D ρ R ( x ) u j = a ( x ) u j . Hence
F ~ m , U ( x ) = r ~ j ( x ) , ∇ F ~ m , U ( x ) = ∇ r ~ j ( x ) − a ( x ) γ j − 1 ( x ) u j . (57) \widetilde F_{m,U}(x)=\widetilde r_j(x),\qquad\nabla\widetilde F_{m,U}(x)=\nabla\widetilde r_j(x)-a(x)\gamma_{j-1}(x)u_j. \tag*{(57)} F m , U ( x ) = r j ( x ) , ∇ F m , U ( x ) = ∇ r j ( x ) − a ( x ) γ j − 1 ( x ) u j . (57)
Theorem E.8 (Lower bound on the unrestricted smooth class). For all d , T ≥ 1 d,T\ge1 d , T ≥ 1 on the unrestricted globally L L L -smooth class, N d e t l i n ≥ d 500 min { T , d / log ( e d ) } N_{\mathrm{det}}^{\mathrm{lin}}\ge\frac{d}{500}\min\{T,d/\log(ed)\} N det lin ≥ 500 d min { T , d / log ( e d )} ; the same bound holds for exact values.
Proof of Theorem E.8
Put b = 3 / 5 b=3/5 b = 3/5 , A = 44 ⋅ ( 9 / 4 ) / b 2 = 275 A=44\cdot(9/4)/b^2=275 A = 44 ⋅ ( 9/4 ) / b 2 = 275 , and S = min { T , d / log ( e d ) } S=\min\{T,d/\log(ed)\} S = min { T , d / log ( e d )} .
Case S > 500 S>500 S > 500 . Let m = ⌊ S / A + 1 / 2 ⌋ m=\lfloor S/A+1/2\rfloor m = ⌊ S / A + 1/2 ⌋ and h = ⌊ 7 d / 10 ⌋ h=\lfloor7d/10\rfloor h = ⌊ 7 d /10 ⌋ . Since 500 > 3 A / 2 = 825 / 2 500>3A/2=825/2 500 > 3 A /2 = 825/2 , we have m ≥ 2 m\ge2 m ≥ 2 , and m − 1 / 2 ≤ S / A ≤ d / ( A log ( e d ) ) m-1/2\le S/A\le d/(A\log(ed)) m − 1/2 ≤ S / A ≤ d / ( A log ( e d )) gives d / m ≥ ( A / 2 ) ( 2 − 1 / m ) log ( e d ) d/m\ge(A/2)(2-1/m)\log(ed) d / m ≥ ( A /2 ) ( 2 − 1/ m ) log ( e d ) and m < d / 1000 m<d/1000 m < d /1000 . Hence p 0 = d − m − h ≥ 3 d / 10 − m > 0 p_0=d-m-h\ge3d/10-m>0 p 0 = d − m − h ≥ 3 d /10 − m > 0 and, with τ = 1 / 2 \tau=1/2 τ = 1/2 and R 2 = 5 m R^2=5m R 2 = 5 m ,
p 0 τ 2 2 R 2 = p 0 40 m ≥ 3 d 400 m − 1 40 ≥ 33 32 ( 2 − 1 m ) log ( e d ) − 1 40 > ( 2 − 1 m ) log ( e d ) . \frac{p_0\tau^2}{2R^2}=\frac{p_0}{40m}\ge\frac{3d}{400m}-\frac{1}{40}\ge\frac{33}{32}\left(2-\frac{1}{m}\right)\log(ed)-\frac{1}{40}>\left(2-\frac{1}{m}\right)\log(ed). 2 R 2 p 0 τ 2 = 40 m p 0 ≥ 400 m 3 d − 40 1 ≥ 32 33 ( 2 − m 1 ) log ( e d ) − 40 1 > ( 2 − m 1 ) log ( e d ) .
Since d > 500 d>500 d > 500 and m < d / 1000 m<d/1000 m < d /1000 , we have 2 m ( e d ) 1 / m ≤ max { 4 e d , 2 e m } < e 2 d 2m(ed)^{1/m}\le\max\{4\sqrt{ed},2em\}<e^2d 2 m ( e d ) 1/ m ≤ max { 4 e d , 2 e m } < e 2 d , so 2 h m < 2 d m < ( e d ) 2 − 1 / m 2hm<2dm<(ed)^{2-1/m} 2 hm < 2 d m < ( e d ) 2 − 1/ m and
2 h m exp ( − p 0 τ 2 2 R 2 ) < ( e d ) 2 − 1 / m exp ( − ( 2 − 1 m ) log ( e d ) ) = 1 , 2hm\exp\left(-\frac{p_0\tau^2}{2R^2}\right)<(ed)^{2-1/m}\exp\left(-\left(2-\frac{1}{m}\right)\log(ed)\right)=1, 2 hm exp ( − 2 R 2 p 0 τ 2 ) < ( e d ) 2 − 1/ m exp ( − ( 2 − m 1 ) log ( e d ) ) = 1 ,
which is the cap condition of Lemma C.4 , whose residual dimension p 0 + 1 p_0+1 p 0 + 1 only improves it. Apply Lemma E.1 with L ′ = L L'=L L ′ = L , Λ = 44 \Lambda=44 Λ = 44 , K = 44 I K=44I K = 44 I , and D = 9 4 ( m − 1 2 ) D=\frac{9}{4}(m-\frac{1}{2}) D = 4 9 ( m − 2 1 ) ; the trace condition holds because E = d E=d E = d , and the accuracy condition is T ≥ 44 D / b 2 = A ( m − 1 2 ) T\ge44D/b^2=A(m-\frac{1}{2}) T ≥ 44 D / b 2 = A ( m − 2 1 ) , which follows from A ( m − 1 2 ) ≤ S ≤ T A(m-\frac{1}{2})\le S\le T A ( m − 2 1 ) ≤ S ≤ T . Lemmas E.6 and E.7 supply the smoothness, the gap, and a barrier larger than ε \varepsilon ε after scaling, and (57) verifies the prefix hypotheses of Lemma C.4 with z = ρ R z=\rho R z = ρR and c j = a γ j − 1 c_j=a\gamma^{j-1} c j = a γ j − 1 . Every deterministic algorithm with rank-priced cost below h m hm hm therefore fails on some fixed scaled F ~ m , U \widetilde{F}_{m,U} F m , U . Finally, S < A ( m + 1 / 2 ) S<A(m+1/2) S < A ( m + 1/2 ) and m ≥ 2 m\ge2 m ≥ 2 give m > 4 S / ( 5 A ) m>4S/(5A) m > 4 S / ( 5 A ) ; since d > 500 d>500 d > 500 ,
h m > d S A ( 14 25 − 4 5 d ) > d S 500 . hm>\frac{dS}{A}\left(\frac{14}{25}-\frac{4}{5d}\right)>\frac{dS}{500}. hm > A d S ( 25 14 − 5 d 4 ) > 500 d S .
using 14 / 25 − A / 500 = 1 / 100 > 4 / ( 5 d ) 14/25-A/500=1/100>4/(5d) 14/25 − A /500 = 1/100 > 4/ ( 5 d ) .
The witnesses in the chain regime are coercive and nonconvex by Lemma E.6 . They also have gradient norm at most 2 L Δ 2\sqrt{L\Delta} 2 L Δ : ∥ ∇ F ~ m , U ∥ ≤ 10 m + 1 ≤ 11 m \lVert\nabla\widetilde{F}_{m,U}\rVert\le10\sqrt{m}+1\le11\sqrt{m} ∥ ∇ F m , U ∥ ≤ 10 m + 1 ≤ 11 m , so the scaled gradient is at most ( 55 / 3 ) m ε (55/3)\sqrt{m}\,\varepsilon ( 55/3 ) m ε , while L Δ ≥ A ( m − 1 / 2 ) ε 2 ≥ ( A / 2 ) m ε 2 L\Delta\ge A(m-1/2)\varepsilon^2\ge(A/2)m\varepsilon^2 L Δ ≥ A ( m − 1/2 ) ε 2 ≥ ( A /2 ) m ε 2 and ( 55 / 3 ) 2 < 2 A (55/3)^2<2A ( 55/3 ) 2 < 2 A .
Case S ≤ 500 S\le500 S ≤ 500 . Lemma E.2 with E = d E=d E = d gives N d e t l i n ≥ d ≥ d S / 500 N_{\mathrm{det}}^{\mathrm{lin}}\ge d\ge dS/500 N det lin ≥ d ≥ d S /500 . Algorithms that use only values are a special case of rank-priced ones, so N d e t v a l ≥ N d e t l i n N_{\mathrm{det}}^{\mathrm{val}}\ge N_{\mathrm{det}}^{\mathrm{lin}} N det val ≥ N det lin .
Quadratic Randomized Lower Bound
Algorithms here are measurable functions of their independent seed and finite transcript; success is at least 2 / 3 2/3 2/3 on each fixed function in the class, not merely on average over the distribution used in the lower bound. A subscript E E E , as in N r a n d , E N_{\mathrm{rand},E} N rand , E , indicates the class F L , Δ , E \mathcal{F}_{L,\Delta,E} F L , Δ , E .
Lemma F.1 (Explicit spherical coordinate splitting). For q ≥ 2 q\ge2 q ≥ 2 , a uniform U ∈ S q − 1 U\in S^{q-1} U ∈ S q − 1 admits the product representation
U = ( C , 1 − C 2 W ) , W ∼ Unif ( S q − 2 ) , W independent of C , E C 2 = 1 / q . U=\left(C,\sqrt{1-C^2}\,W\right),\qquad W\sim\operatorname{Unif}(S^{q-2}),\qquad W\text{ independent of }C,\qquad\mathbb{E}C^2=1/q. U = ( C , 1 − C 2 W ) , W ∼ Unif ( S q − 2 ) , W independent of C , E C 2 = 1/ q . It is valid after any specified orthogonal change of coordinates. For q = 1 q=1 q = 1 the sphere is a fair sign.
Proof. Let ( G 1 , G ′ ) (G_1,G') ( G 1 , G ′ ) be standard Gaussian in R × R q − 1 \mathbb{R}\times\mathbb{R}^{q-1} R × R q − 1 . Its normalized direction is uniform by rotational invariance of Gaussian density. Polar coordinates for G ′ G' G ′ show that W = G ′ / ∥ G ′ ∥ W=G'/\lVert G'\rVert W = G ′ / ∥ G ′ ∥ is uniform and independent of ( G 1 , ∥ G ′ ∥ ) (G_1,\lVert G'\rVert) ( G 1 , ∥ G ′ ∥) ; for q = 2 q=2 q = 2 this is independence of the sign and magnitude of a scalar Gaussian. Set C = G 1 / G 1 2 + ∥ G ′ ∥ 2 C=G_1/\sqrt{G_1^2+\lVert G'\rVert^2} C = G 1 / G 1 2 + ∥ G ′ ∥ 2 . Coordinate exchangeability and ∑ i U i 2 = 1 \sum_i U_i^2=1 ∑ i U i 2 = 1 give E C 2 = 1 / q \mathbb{E}C^2=1/q E C 2 = 1/ q . For a specified unit direction v v v , a measurable alignment is the Householder matrix I − 2 ( e 1 − v ) ( e 1 − v ) ⊤ / ∥ e 1 − v ∥ 2 I-2(e_1-v)(e_1-v)^\top/\lVert e_1-v\rVert^2 I − 2 ( e 1 − v ) ( e 1 − v ) ⊤ / ∥ e 1 − v ∥ 2 if v ≠ e 1 v\ne e_1 v = e 1 and I I I otherwise; it preserves the uniform law.
Lemma F.2 (Adaptive projections: energy and antipodes). Let u u u be uniform on the unit sphere in an e e e -dimensional Euclidean space and independent of an algorithm’s seed. After at most w w w adaptively chosen exact linear observations, where 0 ≤ w ≤ e − 1 0\le w\le e-1 0 ≤ w ≤ e − 1 , any measurable estimate z z z satisfies
P ( ∥ z − u ∥ ≤ r ) ≤ 1 2 + w 2 e ( 1 − r 2 ) ( 0 ≤ r < 1 ) . (58) \mathbb{P}(\lVert z-u\rVert\le r)\le\frac{1}{2}+\frac{w}{2e(1-r^2)}\qquad(0\le r<1). \tag*{(58)} P (∥ z − u ∥ ≤ r ) ≤ 2 1 + 2 e ( 1 − r 2 ) w ( 0 ≤ r < 1 ) . (58)
Proof. Fix the seed. Orthogonalize independent queried directions, omitting dependent directions whose replies are already determined. After termination pad the span using Gram–Schmidt on a fixed basis until its dimension is exactly w w w ; the extra observations can be revealed to the analysis even if the estimator ignores them. The successive unit directions v j v_j v j are measurable functions of previous observations, and each is orthogonal to those preceding it.
We construct the joint law explicitly on a product space. Begin with residual squared norm 1 1 1 and an orthonormal basis. At step j j j , whose residual dimension is q = e − j q=e-j q = e − j , align the next chosen direction to the first residual axis by Lemma F.1 . Draw an independent scalar C j C_j C j with the law of the first coordinate in dimension q q q . Reveal α j = 1 − Z j C j \alpha_j=\sqrt{1-Z_j}C_j α j = 1 − Z j C j , define Z j + 1 = Z j + ( 1 − Z j ) C j 2 Z_{j+1}=Z_j+(1-Z_j)C_j^2 Z j + 1 = Z j + ( 1 − Z j ) C j 2 , and use the remaining aligned axes as a basis of the new residual subspace. Finally draw an independent uniform W W W in the residual dimension e − w e-w e − w and form
u = ∑ j = 0 w − 1 α j v j + 1 − Z w B w W , (59) u=\sum_{j=0}^{w-1}\alpha_jv_j+\sqrt{1-Z_w}B_wW, \tag*{(59)} u = j = 0 ∑ w − 1 α j v j + 1 − Z w B w W , (59)
where B w B_w B w is the residual isometry determined by the revealed history. This produces the same joint integrals as a uniform u u u together with the observations selected by the adaptive rule: at each step apply the product representation of Lemma F.1 to the innermost integral over the residual sphere, hold the outer history fixed, and then use Fubini. Induction over the finite w w w steps establishes this for every bounded measurable test function of ( u , h i s t o r y ) (u,\mathop{\rm history}) ( u , history ) . When the residual radius is zero, any residual basis can be used without changing the vector. No regular conditional probability is invoked.
Because C j C_j C j is independent of the preceding history and has second moment 1 / ( e − j ) 1/(e-j) 1/ ( e − j ) ,
E Z j + 1 = E Z j + 1 − E Z j e − j , E Z w = w e . \mathbb{E}Z_{j+1}=\mathbb{E}Z_j+\frac{1-\mathbb{E}Z_j}{e-j},\qquad\mathbb{E}Z_w=\frac{w}{e}. E Z j + 1 = E Z j + e − j 1 − E Z j , E Z w = e w .
Fix a history on the product space and hence the estimator z z z . If 1 − Z w > r 2 1-Z_w>r^2 1 − Z w > r 2 , the two vectors in (59) generated by W W W and − W -W − W have separation 2 1 − Z w > 2 r 2\sqrt{1-Z_w}>2r 2 1 − Z w > 2 r , so at most one lies within distance r r r of z z z . Paired integration over the final symmetric sphere bounds success by 1 / 2 1/2 1/2 on this event, and by one otherwise. Markov’s inequality gives P ( Z w ≥ 1 − r 2 ) ≤ w / [ e ( 1 − r 2 ) ] \mathbb{P}(Z_w\ge1-r^2)\le w/[e(1-r^2)] P ( Z w ≥ 1 − r 2 ) ≤ w / [ e ( 1 − r 2 )] . This proves (58) ; average over the independent seed.
Proposition F.3 (Quadratic randomized lower bound). Let W ⊆ R d W\subseteq\mathbb{R}^d W ⊆ R d be a public subspace of dimension e ≥ 1 e\ge1 e ≥ 1 , let ℓ , Δ , ε > 0 \ell,\Delta,\varepsilon>0 ℓ , Δ , ε > 0 , and suppose θ = ℓ Δ / ε 2 > 1 / 2 \theta=\ell\Delta/\varepsilon^2>1/2 θ = ℓ Δ/ ε 2 > 1/2 . Define
a θ = 1 − 1 2 θ , R 0 = 2 Δ ℓ , f u ( x ) = ℓ 2 ∥ P W x − R 0 u ∥ 2 , u ∼ Unif ( S ( W ) ) . a_\theta=1-\frac{1}{2\theta},\qquad R_0=\sqrt{\frac{2\Delta}{\ell}},\qquad f_u(x)=\frac{\ell}{2}\lVert P_Wx-R_0u\rVert^2,\qquad u\sim\operatorname{Unif}(S(W)). a θ = 1 − 2 θ 1 , R 0 = ℓ 2Δ , f u ( x ) = 2 ℓ ∥ P W x − R 0 u ∥ 2 , u ∼ Unif ( S ( W )) . Every f u f_u f u is convex and ℓ \ell ℓ -smooth, has initial gap exactly Δ \Delta Δ , and admits the global fixed upper matrix ℓ P W \ell P_W ℓ P W . Any randomized algorithm successful with probability at least 2 / 3 2/3 2/3 on every f u f_u f u has rank-priced cost cap at least e a θ / 3 ea_\theta/3 e a θ /3 and worst-case expected rank-priced cost at least e a θ / 36 ea_\theta/36 e a θ /36 . These also hold for exact values.
Proof. We have
f u ( x ) = ℓ 2 ∥ P W x ∥ 2 + Δ − ℓ R 0 ⟨ u , P W x ⟩ , ∇ f u ( x ) = ℓ ( P W x − R 0 u ) . f_u(x)=\frac{\ell}{2}\lVert P_Wx\rVert^2+\Delta-\ell R_0\langle u,P_Wx\rangle,\qquad\nabla f_u(x)=\ell(P_Wx-R_0u). f u ( x ) = 2 ℓ ∥ P W x ∥ 2 + Δ − ℓ R 0 ⟨ u , P W x ⟩ , ∇ f u ( x ) = ℓ ( P W x − R 0 u ) . A response to call ( x , V ) (x,V) ( x , V ) is determined by the projections of u u u on span { P W x } + col ( P W V ) \operatorname{span}\{P_Wx\}+\operatorname{col}(P_WV) span { P W x } + col ( P W V ) , whose dimension is at most 1 + rank V 1+\operatorname{rank}V 1 + rank V and is chosen before the reply. Revealing a basis of those projections simulates the response within its cost. Success means ∥ P W x ~ / R 0 − u ∥ ≤ ( 2 θ ) − 1 / 2 \lVert P_W\widetilde{x}/R_0-u\rVert\le(2\theta)^{-1/2} ∥ P W x / R 0 − u ∥ ≤ ( 2 θ ) − 1/2 . Costs are integers. If a cost cap w w w is below e e e , Lemma F.2 bounds average success over u u u by 1 / 2 + w / ( 2 e a θ ) 1/2+w/(2ea_\theta) 1/2 + w / ( 2 e a θ ) , so success probability 2 / 3 2/3 2/3 forces w ≥ e a θ / 3 w\ge ea_\theta/3 w ≥ e a θ /3 . Caps w ≥ e w\ge e w ≥ e satisfy this since a θ < 1 a_\theta<1 a θ < 1 .
For expected cost let W W W be the random cost. For every integer 0 ≤ k < e 0\le k<e 0 ≤ k < e , truncate before any call that would raise cost above k k k , returning a fixed point. It agrees with the original algorithm whenever W < k W<k W < k ; therefore, with probability averaged over u u u and the seed, P ( W > k ) ≥ ( 1 / 6 − k / ( 2 e a θ ) ) + \mathbb{P}(W>k)\ge(1/6-k/(2ea_\theta))_+ P ( W > k ) ≥ ( 1/6 − k / ( 2 e a θ ) ) + . Since e a θ / 3 < e ea_\theta/3<e e a θ /3 < e , tail summation gives
E W ≥ ∑ k ≥ 0 ( 1 6 − k 2 e a θ ) + ≥ ∫ 0 e a θ / 3 ( 1 6 − t 2 e a θ ) d t = e a θ 36 . \mathbb{E}W\ge\sum_{k\ge0}\left(\frac{1}{6}-\frac{k}{2ea_\theta}\right)_+\ge\int_0^{ea_\theta/3}\left(\frac{1}{6}-\frac{t}{2ea_\theta}\right)\,dt=\frac{ea_\theta}{36}. E W ≥ k ≥ 0 ∑ ( 6 1 − 2 e a θ k ) + ≥ ∫ 0 e a θ /3 ( 6 1 − 2 e a θ t ) d t = 36 e a θ .
as the integrand is decreasing and nonnegative. The supremum over u u u of conditional expected cost is at least this average. Exact value calls have rank zero.
Corollary F.4 (Randomized baseline on the trace class). For all d ≥ 1 d \ge1 d ≥ 1 , 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , and T ≥ 1 T \ge1 T ≥ 1 ,
N r a n d , E v a l ≥ N r a n d , E l i n ≥ min { d , E T } 12 , N^{\mathrm{val}}_{\mathrm{rand},E} \ge N^{\mathrm{lin}}_{\mathrm{rand},E} \ge\frac{\min\{d,ET\}}{12}, N rand , E val ≥ N rand , E lin ≥ 12 min { d , E T } , and both worst-case expected costs are at least min { d , E T } / 144 \min\{d,ET\}/144 min { d , E T } /144 .
Proof. Take e = min { d , ⌊ E T ⌋ } e = \min\{d,\lfloor ET\rfloor\} e = min { d , ⌊ E T ⌋} , a coordinate subspace W W W of dimension e e e , and ℓ = min { L , L E / e } \ell= \min\{L,LE/e\} ℓ = min { L , L E / e } . Since E T ≥ 1 ET \ge1 E T ≥ 1 , e ≥ min { d , E T } / 2 e \ge\min\{d,ET\}/2 e ≥ min { d , E T } /2 . Further, ℓ ≤ L \ell\le L ℓ ≤ L , ℓ e ≤ L E \ell e \le LE ℓ e ≤ L E , and θ = min { T , E T / e } ≥ 1 \theta= \min\{T,ET/e\} \ge1 θ = min { T , E T / e } ≥ 1 . Thus these quadratics lie in F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , a θ ≥ 1 / 2 a_\theta\ge1/2 a θ ≥ 1/2 , and the proposition yields cap at least e / 6 e/6 e /6 and expected cost at least e / 72 e/72 e /72 .
Further Results
Theorem A.1 on B \mathcal{B} B does not use this appendix; its extension to F \mathcal{F} F uses Section G.3. The appendix defines the pointwise classes, records the inclusions between the classes, gives the randomized upper bound on the larger class A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d , and C ∞ C^\infty C ∞ hard functions with a Lipschitz Hessian in the global fixed-matrix class F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , which give the deterministic lower bounds on every class that contains it.
Pointwise classes and inclusions
Our randomized upper bound holds on a larger class, the local pointwise class A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d , that requires neither a common matrix H H H nor smoothness above f ( 0 ) f(0) f ( 0 ) : it consists of the continuously differentiable f f f with f ( 0 ) − inf f ≤ Δ f(0)-\inf f \le\Delta f ( 0 ) − inf f ≤ Δ whose gradient is locally L L L -Lipschitz on U U U and which satisfy tr [ ( ∇ 2 f ( x ) ) + ] ≤ L E \operatorname{tr}[(\nabla^{2}f(x))_{+}] \le LE tr [( ∇ 2 f ( x ) ) + ] ≤ L E for almost every x ∈ U x \in U x ∈ U , where A + A_{+} A + keeps the positive eigenvalues of a symmetric matrix A A A (U U U need not be bounded or connected). Since A ⪯ H A \preceq H A ⪯ H and H ⪰ 0 H \succeq0 H ⪰ 0 imply tr A + ≤ tr H \operatorname{tr} A_{+} \le\operatorname{tr} H tr A + ≤ tr H , we have B L , Δ , E d ⊆ A L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} \subseteq\mathcal{A}^{d}_{L,\Delta,E} B L , Δ , E d ⊆ A L , Δ , E d . The global pointwise class P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d requires f ∈ F L , Δ d f \in\mathcal{F}^{d}_{L,\Delta} f ∈ F L , Δ d and tr [ ( ∇ 2 f ( x ) ) + ] ≤ L E \operatorname{tr}[(\nabla^{2}f(x))_{+}] \le LE tr [( ∇ 2 f ( x ) ) + ] ≤ L E for almost every x ∈ R d x \in\mathbb{R}^{d} x ∈ R d .
Lemma G.1 (Class inclusions). At each fixed choice of the public parameters,
F L , Δ , E d ⊆ P L , Δ , E d ⊆ A L , Δ , E d , F L , Δ , E d ⊆ B L , Δ , E d ⊆ A L , Δ , E d . \mathcal{F}^{d}_{L,\Delta,E} \subseteq\mathcal{P}^{d}_{L,\Delta,E} \subseteq\mathcal{A}^{d}_{L,\Delta,E}, \qquad\mathcal{F}^{d}_{L,\Delta,E} \subseteq\mathcal{B}^{d}_{L,\Delta,E} \subseteq\mathcal{A}^{d}_{L,\Delta,E}. F L , Δ , E d ⊆ P L , Δ , E d ⊆ A L , Δ , E d , F L , Δ , E d ⊆ B L , Δ , E d ⊆ A L , Δ , E d .
Proof. Let f ∈ F L , Δ , E d f \in\mathcal{F}^{d}_{L,\Delta,E} f ∈ F L , Δ , E d have upper model H H H . At a point where ∇ f \nabla f ∇ f is differentiable, the quadratic inequality (3) , applied to t v tv t v and letting t → 0 t \to0 t → 0 , gives v ⊤ ∇ 2 f ( x ) v ≤ v ⊤ H v v^{\top}\nabla^{2}f(x)v \le v^{\top}Hv v ⊤ ∇ 2 f ( x ) v ≤ v ⊤ H v for every v v v . By Rademacher’s theorem this holds almost everywhere on R d \mathbb{R}^{d} R d , so f ∈ B L , Δ , E d f \in\mathcal{B}^{d}_{L,\Delta,E} f ∈ B L , Δ , E d . For symmetric A ⪯ H A \preceq H A ⪯ H with H ⪰ 0 H \succeq0 H ⪰ 0 , let P + P_{+} P + be the spectral projection onto the positive eigenspaces of A A A . Then
tr A + = tr ( P + A ) ≤ tr ( P + H ) ≤ tr H ≤ L E . \operatorname{tr} A_{+} = \operatorname{tr}(P_{+}A) \le\operatorname{tr}(P_{+}H) \le\operatorname{tr} H \le LE. tr A + = tr ( P + A ) ≤ tr ( P + H ) ≤ tr H ≤ L E .
Applied almost everywhere on R d \mathbb{R}^{d} R d , this gives f ∈ P L , Δ , E d f \in\mathcal{P}^{d}_{L,\Delta,E} f ∈ P L , Δ , E d . Applied almost everywhere on the initial sublevel set, it gives the trace bound of A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d for every f ∈ B L , Δ , E d f \in\mathcal{B}^{d}_{L,\Delta,E} f ∈ B L , Δ , E d . Finally, a globally L L L -Lipschitz gradient is locally L L L -Lipschitz on the initial sublevel set, and a trace bound on R d \mathbb{R}^{d} R d holds in particular there; this gives both inclusions in A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d .
Consequently lower bounds proved on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d and upper bounds proved on A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d hold on all four classes.
The randomized upper bound on the local pointwise class
Recall that f ∈ A L , Δ , E d f \in A^{d}_{L,\Delta,E} f ∈ A L , Δ , E d means f ∈ C 1 ( R d ) f \in C^{1}(\mathbb{R}^{d}) f ∈ C 1 ( R d ) , f ( 0 ) − inf f ≤ Δ f(0)-\inf f \le\Delta f ( 0 ) − inf f ≤ Δ , and, on U = { f < f ( 0 ) } U=\{f<f(0)\} U = { f < f ( 0 )} , a gradient that is locally L L L -Lipschitz with one constant throughout U U U and tr ( ( ∇ 2 f ) + ) ≤ L E \operatorname{tr}((\nabla^{2}f)_{+}) \le LE tr (( ∇ 2 f ) + ) ≤ L E almost everywhere. No common matrix is assumed, so a capped function replaces the first exit Lemma B.3 .
Lemma G.2 (Capped function). Let f ∈ A L , Δ , E d f \in A^{d}_{L,\Delta,E} f ∈ A L , Δ , E d , c = f ( 0 ) c=f(0) c = f ( 0 ) , G = 2 L Δ G=\sqrt{2L\Delta} G = 2 L Δ , and F c = min { f , c } F_c=\min\{f,c\} F c = min { f , c } . Then F c F_c F c is globally G G G -Lipschitz and bounded. For f ( x ) ≤ c f(x)\le c f ( x ) ≤ c ,
F c ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + L 2 ∥ v ∥ 2 . (60) F_c(x+v) \le f(x)+\langle\nabla f(x),v\rangle+\frac{L}{2}\lVert v\rVert^{2}. \tag*{(60)} F c ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 2 L ∥ v ∥ 2 . (60) For any fixed orthogonal projection P P P , Z ∼ N ( 0 , P ) Z\sim N(0,P) Z ∼ N ( 0 , P ) , t ≥ 0 t\ge0 t ≥ 0 , and y ∈ R d y\in\mathbb{R}^{d} y ∈ R d ,
E F c ( y + t Z ) ≤ F c ( y ) + L E 2 t . (61) \mathbb{E}F_c(y+\sqrt{t}Z) \le F_c(y)+\frac{LE}{2}t. \tag*{(61)} E F c ( y + t Z ) ≤ F c ( y ) + 2 L E t . (61) Proof. For x ∈ U x\in U x ∈ U , put g = ∇ f ( x ) g=\nabla f(x) g = ∇ f ( x ) . Follow x − t g x-tg x − t g up to t = 1 / L t=1/L t = 1/ L . Before its first possible exit from U U U , local smoothness and integration along the segment give f ( x − t g ) ≤ f ( x ) − t ( 1 − L t / 2 ) ∥ g ∥ 2 f(x-tg)\le f(x)-t(1-Lt/2)\lVert g\rVert^{2} f ( x − t g ) ≤ f ( x ) − t ( 1 − L t /2 ) ∥ g ∥ 2 . A first exit before 1 / L 1/L 1/ L would contradict this strict decrease and continuity. Consequently ∥ g ∥ 2 ≤ 2 L ( f ( x ) − inf f ) ≤ G 2 \lVert g\rVert^{2}\le2L(f(x)-\inf f)\le G^{2} ∥ g ∥ 2 ≤ 2 L ( f ( x ) − inf f ) ≤ G 2 . The restriction of F c F_c F c to any compact line segment is absolutely continuous: f f f is C 1 C^{1} C 1 there and truncation is Lipschitz. Its derivative is bounded in magnitude by G G G almost everywhere, being either a derivative of f f f below c c c or zero. This proves the global Lipschitz bound. Boundedness follows from inf f ≤ F c ≤ c \inf f\le F_c\le c inf f ≤ F c ≤ c .
We argue by smoothing. For δ > 0 \delta>0 δ > 0 , choose a smooth concave nondecreasing function ϕ δ \phi_\delta ϕ δ with 0 ≤ ϕ δ ′ ≤ 1 0\le\phi'_\delta\le1 0 ≤ ϕ δ ′ ≤ 1 , derivative one below c − 2 δ c-2\delta c − 2 δ , derivative zero above c − δ c-\delta c − δ , and sup s ∣ ϕ δ ( s ) − min ( s , c ) ∣ ≤ 2 δ \sup_s|\phi_\delta(s)-\min(s,c)|\le2\delta sup s ∣ ϕ δ ( s ) − min ( s , c ) ∣ ≤ 2 δ . It is obtained by integrating a smooth decreasing transition on [ c − 2 δ , c − δ ] [c-2\delta,c-\delta] [ c − 2 δ , c − δ ] and fixing the additive constant. The function u δ = ϕ δ ∘ f u_\delta=\phi_\delta\circ f u δ = ϕ δ ∘ f is globally C 1 , 1 C^{1,1} C 1 , 1 : where its derivatives are nonzero one is in U U U , its Hessian has norm bounded by L + ∥ ϕ δ ′ ′ ∥ ∞ G 2 L+\lVert\phi''_\delta\rVert_\infty G^{2} L + ∥ ϕ δ ′′ ∥ ∞ G 2 , and near the remaining points the composition is constant. Local bounds with this common constant integrate to a global bound. Almost everywhere,
∇ 2 u δ = ϕ δ ′ ( f ) ∇ 2 f + ϕ δ ′ ′ ( f ) ∇ f ∇ f T ⪯ L I , tr ( P ∇ 2 u δ ) ≤ L E . \nabla^{2}u_\delta=\phi'_\delta(f)\nabla^{2}f+\phi''_\delta(f)\nabla f\nabla f^{\mathsf T}\preceq LI,\qquad\operatorname{tr}(P\nabla^{2}u_\delta)\le LE. ∇ 2 u δ = ϕ δ ′ ( f ) ∇ 2 f + ϕ δ ′′ ( f ) ∇ f ∇ f T ⪯ L I , tr ( P ∇ 2 u δ ) ≤ L E . The second inequality uses tr ( P B ) ≤ tr ( B + ) \operatorname{tr}(PB)\le\operatorname{tr}(B_{+}) tr ( P B ) ≤ tr ( B + ) for every symmetric B B B and 0 ⪯ P ⪯ I 0\preceq P\preceq I 0 ⪯ P ⪯ I ; the cutoff’s second term is nonpositive. Convolve in all d d d dimensions with a nonnegative, symmetric, compactly supported smooth approximate identity. Both inequalities survive. For the resulting smooth bounded function u u u , the quadratic upper model follows by integration. Gaussian integration by parts gives
d d t E u ( y + t Z ) = 1 2 E tr ( P ∇ 2 u ( y + t Z ) ) ≤ L E / 2. \frac{d}{dt}\mathbb{E}u(y+\sqrt{t}Z)=\frac{1}{2}\mathbb{E}\operatorname{tr}(P\nabla^{2}u(y+\sqrt{t}Z))\le LE/2. d t d E u ( y + t Z ) = 2 1 E tr ( P ∇ 2 u ( y + t Z )) ≤ L E /2. This is also valid for degenerate P P P by using coordinates on its range; it is applied only after full-dimensional smoothing. Bounded Hessians and Gaussian moments justify differentiation and integration by parts. Integrate from 0 0 0 to t t t , send the convolution radius to zero, and then send δ \delta δ to zero. The convergence of the clipped values is uniform, which proves (61) .
For x ∈ U x\in U x ∈ U , the same argument gives (60) , because u δ ( x ) → f ( x ) u_\delta(x)\to f(x) u δ ( x ) → f ( x ) and ∇ u δ ( x ) → ∇ f ( x ) \nabla u_\delta(x)\to\nabla f(x) ∇ u δ ( x ) → ∇ f ( x ) . If f ( x ) = c f(x)=c f ( x ) = c and ∇ f ( x ) ≠ 0 \nabla f(x)\ne0 ∇ f ( x ) = 0 , approach x x x from the local downhill side in U U U and use continuity. If ∇ f ( x ) = 0 \nabla f(x)=0 ∇ f ( x ) = 0 , use F c ≤ c F_c\le c F c ≤ c . □
In Lemma G.2 the concave scalar cutoff handles the negative curvature created by truncation, and the Gaussian identity is applied only after smoothing in all d d d dimensions, because an almost everywhere bound in R d \mathbb{R}^{d} R d may fail on a hyperplane.
The update below is the stochastic three-point method of Theorem B.8 (Algorithm 4), with the curvature parameter L ( E + 1 ) L(E+1) L ( E + 1 ) in place of L E LE L E .
Theorem G.3 (Three-point comparison on A \mathcal{A} A ). Let M = L ( E + 1 ) M = L(E + 1) M = L ( E + 1 ) , h = 4 ε / ( 5 M ) h = 4\varepsilon/(5M) h = 4 ε / ( 5 M ) , and N = ⌈ 10 M Δ / ε 2 ⌉ N = \lceil10M\Delta/\varepsilon^2 \rceil N = ⌈ 10 M Δ/ ε 2 ⌉ . Query f ( 0 ) f(0) f ( 0 ) once and cache the current value. At each iteration draw a ∼ N ( 0 , I d ) a \sim N(0,I_d) a ∼ N ( 0 , I d ) , query f ( x + h a ) f(x+ha) f ( x + ha ) and f ( x − h a ) f(x-ha) f ( x − ha ) , and retain a point of lowest value among { x , x + h a , x − h a } \{x,x+ha,x-ha\} { x , x + ha , x − ha } , keeping x x x on a tie. Return an independent uniformly selected iterate from before the updates. For every fixed f ∈ A f \in\mathcal{A} f ∈ A , the probability of failure is at most 25 / 78 < 1 / 3 25/78 < 1/3 25/78 < 1/3 , and the hard query cap is
Q = 1 + 2 ⌈ 10 ( E + 1 ) T ⌉ ≤ 43 E T ( E , T ≥ 1 ) . Q = 1 + 2\lceil10(E+1)T\rceil\le43ET \qquad(E,T \ge1). Q = 1 + 2 ⌈ 10 ( E + 1 ) T ⌉ ≤ 43 E T ( E , T ≥ 1 ) .
Proof. All centers satisfy f ( x ) ≤ c = f ( 0 ) f(x) \le c = f(0) f ( x ) ≤ c = f ( 0 ) . Fix such a center and write g = ∇ f ( x ) ≠ 0 g = \nabla f(x) \ne0 g = ∇ f ( x ) = 0 , u = g / ∥ g ∥ u = g/\lVert g\rVert u = g / ∥ g ∥ , and a = s u + z a = su + z a = s u + z , where s ∼ N ( 0 , 1 ) s \sim N(0,1) s ∼ N ( 0 , 1 ) and z ∼ N ( 0 , I − u u T ) z \sim N(0,I-uu^{\mathsf{T}}) z ∼ N ( 0 , I − u u T ) are independent. Of the two proposals, one is
x − h ∣ s ∣ u − h sign ( s ) z . x-h\lvert s\rvert u-h\operatorname{sign}(s)z. x − h ∣ s ∣ u − h sign ( s ) z .
Although the algorithm does not know this sign, it compares both proposals. Conditioning on s s s , apply (61) in the orthogonal subspace, then apply (60) at x x x along − h ∣ s ∣ u -h\lvert s\rvert u − h ∣ s ∣ u . Since the minimum of the three true values is at most the capped value of this particular proposal, this gives
E [ f ( x + ) ∣ x ] ≤ f ( x ) − α 0 h ∥ g ∥ + M 2 h 2 , α 0 = E ∣ s ∣ = 2 π > 79 / 100. (62) \mathbb{E}[f(x^+) \mid x] \le f(x)-\alpha_0h\lVert g\rVert+\frac{M}{2}h^2,\qquad\alpha_0=\mathbb{E}\lvert s\rvert=\sqrt{\frac{2}{\pi}}>79/100. \tag*{(62)} E [ f ( x + ) ∣ x ] ≤ f ( x ) − α 0 h ∥ g ∥ + 2 M h 2 , α 0 = E ∣ s ∣ = π 2 > 79/100. (62)
Only capped functions and the accepted value are integrated, so the value at a rejected proposal need not have a finite Gaussian expectation.
The accepted value never increases. Thus when ∥ g ∥ > ε \lVert g\rVert>\varepsilon ∥ g ∥ > ε the conditional expected decrease is at least
( 79 100 4 5 − 1 2 16 25 ) ε 2 M = 39 125 ε 2 M , \left(\frac{79}{100}\frac{4}{5}-\frac{1}{2}\frac{16}{25}\right)\frac{\varepsilon^2}{M}=\frac{39}{125}\frac{\varepsilon^2}{M}, ( 100 79 5 4 − 2 1 25 16 ) M ε 2 = 125 39 M ε 2 ,
and at every other center it is nonnegative. Summation and the initial gap yield
1 N ∑ t = 0 N − 1 P ( ∥ ∇ f ( x t ) ∥ > ε ) ≤ 125 M Δ 39 N ε 2 ≤ 25 78 . \frac{1}{N}\sum_{t=0}^{N-1}\mathbb{P}(\lVert\nabla f(x_t)\rVert>\varepsilon)\le\frac{125M\Delta}{39N\varepsilon^2}\le\frac{25}{78}. N 1 t = 0 ∑ N − 1 P (∥ ∇ f ( x t )∥ > ε ) ≤ 39 N ε 2 125 M Δ ≤ 78 25 .
This is exactly the failure probability of the independent uniform output. Every iteration uses two new values, including unsuccessful proposals. The final bound follows from 1 + 2 ⌈ 10 ( E + 1 ) T ⌉ ≤ 3 + 20 ( E + 1 ) T ≤ 43 E T 1+2\lceil10(E+1)T\rceil\le3+20(E+1)T\le43ET 1 + 2 ⌈ 10 ( E + 1 ) T ⌉ ≤ 3 + 20 ( E + 1 ) T ≤ 43 E T . □ \square □
Smooth lower bounds on the global classes
This subsection proves the deterministic lower bounds of Theorem 3.2 and Corollary 3.3 with C ∞ C^\infty C ∞ witnesses that have a Lipschitz Hessian and lie in the global fixed-matrix class F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , and hence in P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d and B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d . It changes the witnesses, not the definition of the optimization oracle. Smoothing an entire hard function need not keep the hidden directions invisible to exact values, so we smooth the one-dimensional building blocks of the chain of Appendix § 12 .1 and retain an open interval on which each gate is exactly zero. The smoothed chain is multiplied by a perspective weight instead of being added to a radial potential, so the Hessian bound holds on all of R d \mathbb{R}^d R d .
The constants of this subsection are derived with the following selection lemma. It uses the symmetric Gaussian rectangle inequality (Šidák, [49 ] ) instead of Lemma C.2 , and its constant 12 / 5 12/5 12/5 is smaller than the constant 13 / 3 13/3 13/3 there.
Lemma G.4 (Subspace strip avoidance). Let S ⊆ R d S\subseteq\mathbb{R}^d S ⊆ R d have dimension p ≥ 1 p\ge1 p ≥ 1 , and let a 1 , … , a M ∈ R d a_1,\ldots,a_M\in\mathbb{R}^d a 1 , … , a M ∈ R d satisfy ∥ a i ∥ ≤ R \lVert a_i\rVert\le R ∥ a i ∥ ≤ R . There is a unit u ∈ S u\in S u ∈ S such that
max i ≤ M ∣ ⟨ u , a i ⟩ ∣ ≤ 12 5 R log ( e + M / p ) p . (63) \max_{i\le M}\lvert\langle u,a_i\rangle\rvert\le\frac{12}{5}R\sqrt{\frac{\log(e+M/p)}{p}}. \tag*{(63)} i ≤ M max ∣⟨ u , a i ⟩∣ ≤ 5 12 R p log ( e + M / p ) . (63) :::`
Proof. If M < p M < p M < p or R = 0 R = 0 R = 0 , choose a unit vector u ∈ S u \in S u ∈ S orthogonal to the projections of a 1 , … , a M a_1,\ldots,a_M a 1 , … , a M onto S S S . Assume now that s : = M / p ≥ 1 s := M/p \ge1 s := M / p ≥ 1 and R > 0 R > 0 R > 0 . Draw a standard Gaussian g g g in S S S and put t = 9 5 log ( e + s ) > 2 t = \frac{9}{5}\sqrt{\log(e+s)} > 2 t = 5 9 log ( e + s ) > 2 . Each X i = ⟨ g , a i ⟩ X_i = \langle g,a_i\rangle X i = ⟨ g , a i ⟩ is centered Gaussian with variance at most R 2 R^2 R 2 , so
P { ∣ X i ∣ > R t } ≤ 2 π e − t 2 / 2 t < 4 5 t ( e + s ) − 81 / 50 < 1 16 s . \mathbb{P}\{|X_i| > Rt\} \le\sqrt{\frac{2}{\pi}}\frac{e^{-t^2/2}}{t} < \frac{4}{5t}(e+s)^{-81/50} < \frac{1}{16s}. P { ∣ X i ∣ > R t } ≤ π 2 t e − t 2 /2 < 5 t 4 ( e + s ) − 81/50 < 16 s 1 .
The symmetric Gaussian rectangle inequality gives
P { max i ∣ X i ∣ ≤ R t } ≥ ∏ i = 1 M P { ∣ X i ∣ ≤ R t } ≥ ( 1 − 1 16 s ) M ≥ e − p / 15 . \mathbb{P}\left\{\max_i |X_i| \le Rt\right\} \ge\prod_{i=1}^{M}\mathbb{P}\{|X_i| \le Rt\} \ge\left(1-\frac{1}{16s}\right)^M \ge e^{-p/15}. P { i max ∣ X i ∣ ≤ R t } ≥ i = 1 ∏ M P { ∣ X i ∣ ≤ R t } ≥ ( 1 − 16 s 1 ) M ≥ e − p /15 .
using log ( 1 − z ) ≥ − z / ( 1 − z ) \log(1-z) \ge-z/(1-z) log ( 1 − z ) ≥ − z / ( 1 − z ) and M / s = p M/s = p M / s = p . Since ∥ g ∥ 2 \lVert g\rVert^2 ∥ g ∥ 2 is chi-squared with p p p degrees of freedom, exponential Markov with parameter 7 / 18 7/18 7/18 gives P { ∥ g ∥ 2 ≤ 9 16 p } ≤ ( 3 4 e 7 / 32 ) p < e − p / 15 \mathbb{P}\{\lVert g\rVert^2 \le\frac{9}{16}p\} \le\left(\frac{3}{4}e^{7/32}\right)^p < e^{-p/15} P {∥ g ∥ 2 ≤ 16 9 p } ≤ ( 4 3 e 7/32 ) p < e − p /15 . Hence some g g g satisfies both max i ∣ X i ∣ ≤ R t \max_i |X_i| \le Rt max i ∣ X i ∣ ≤ R t and ∥ g ∥ > 3 4 p \lVert g\rVert> \frac{3}{4}\sqrt{p} ∥ g ∥ > 4 3 p , and u = g / ∥ g ∥ u = g/\lVert g\rVert u = g / ∥ g ∥ satisfies max i ∣ ⟨ u , a i ⟩ ∣ ≤ 4 3 R t / p = 12 5 R log ( e + s ) / p \max_i |\langle u,a_i\rangle| \le\frac{4}{3}Rt/\sqrt{p} = \frac{12}{5}R\sqrt{\log(e+s)/p} max i ∣ ⟨ u , a i ⟩ ∣ ≤ 3 4 R t / p = 5 12 R log ( e + s ) / p .
One-dimensional smoothing
Let ϕ , ψ \phi,\psi ϕ , ψ be the scalar functions of Appendix § 12 .1. An equivalent definition, useful for checking their bounds, is
ϕ ( t ) = ∫ − ∞ t ( min { 1 , 4 ( s + 5 / 4 ) , 4 ( 5 / 4 − s ) } ) + d s , ψ ( t ) = ∫ − ∞ t 16 min { ( s − 1 / 2 ) + , ( 1 − s ) + } d s . \phi(t) = \int_{-\infty}^{t}\left(\min\{1,4(s+5/4),4(5/4-s)\}\right)_{+}\,ds,\qquad\psi(t) = \int_{-\infty}^{t}16\min\{(s-1/2)_{+},(1-s)_{+}\}\,ds. ϕ ( t ) = ∫ − ∞ t ( min { 1 , 4 ( s + 5/4 ) , 4 ( 5/4 − s )} ) + d s , ψ ( t ) = ∫ − ∞ t 16 min {( s − 1/2 ) + , ( 1 − s ) + } d s .
Thus 0 ≤ ϕ ≤ 9 / 4 0 \le\phi\le9/4 0 ≤ ϕ ≤ 9/4 , ϕ ′ ( t ) = 1 \phi'(t)=1 ϕ ′ ( t ) = 1 on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , ∣ ϕ ′ ′ ∣ ≤ 4 |\phi''| \le4 ∣ ϕ ′′ ∣ ≤ 4 almost everywhere, 0 ≤ ψ ′ ≤ 4 0 \le\psi' \le4 0 ≤ ψ ′ ≤ 4 , and ∣ ψ ′ ′ ∣ ≤ 16 |\psi''| \le16 ∣ ψ ′′ ∣ ≤ 16 almost everywhere. Put η = 1 / 24 \eta=1/24 η = 1/24 and
w ( t ) = Z − 1 exp [ − 1 / ( 1 − t 2 ) ] 1 { ∣ t ∣ < 1 } , Z = ∫ − 1 1 exp [ − 1 / ( 1 − t 2 ) ] d t , w η ( t ) = η − 1 w ( t / η ) . w(t)=Z^{-1}\exp[-1/(1-t^2)]\mathbf{1}_{\{|t|<1\}},\qquad Z=\int_{-1}^{1}\exp[-1/(1-t^2)]\,dt,\qquad w_\eta(t)=\eta^{-1}w(t/\eta). w ( t ) = Z − 1 exp [ − 1/ ( 1 − t 2 )] 1 { ∣ t ∣ < 1 } , Z = ∫ − 1 1 exp [ − 1/ ( 1 − t 2 )] d t , w η ( t ) = η − 1 w ( t / η ) .
Define
ϕ ~ = ϕ ∗ w η , ψ ~ ( t ) = ( ψ ∗ w η ) ( t + 2 η ) . (64) \widetilde{\phi}=\phi*w_\eta,\qquad\widetilde{\psi}(t)=(\psi*w_\eta)(t+2\eta). \tag*{(64)} ϕ = ϕ ∗ w η , ψ ( t ) = ( ψ ∗ w η ) ( t + 2 η ) . (64)
The shift by 2 η 2\eta 2 η moves the transition of ψ ~ \widetilde{\psi} ψ into [ 3 / 8 , 23 / 24 ] [3/8,23/24] [ 3/8 , 23/24 ] , so ψ ~ = 1 \widetilde{\psi}=1 ψ = 1 on [ 1 , ∞ ) [1,\infty) [ 1 , ∞ ) , as is ψ \psi ψ .
Lemma G.5. (Smooth gates and a dimension-free chain estimate.) The gates in (64) are C ∞ C^\infty C ∞ . Write a 0 = 3 / 8 a_0=3/8 a 0 = 3/8 and a 1 = 23 / 24 a_1=23/24 a 1 = 23/24 . Then ψ ~ = 0 \widetilde{\psi}=0 ψ = 0 on ( − ∞ , a 0 ] (-\infty,a_0] ( − ∞ , a 0 ] , ψ ~ = 1 \widetilde{\psi}=1 ψ = 1 on [ a 1 , ∞ ) [a_1,\infty) [ a 1 , ∞ ) , 0 ≤ ψ ~ ′ ≤ 4 0\le\widetilde{\psi}'\le4 0 ≤ ψ ′ ≤ 4 , ∣ ψ ~ ′ ′ ∣ ≤ 16 |\widetilde{\psi}''|\le16 ∣ ψ ′′ ∣ ≤ 16 , and ∣ ψ ~ ′ ′ ′ ∣ < 788 |\widetilde{\psi}'''|<788 ∣ ψ ′′′ ∣ < 788 . The function ϕ ~ \widetilde{\phi} ϕ takes values in [ 0 , 9 / 4 ] [0,9/4] [ 0 , 9/4 ] , ϕ ~ ( 0 ) = 9 / 8 \widetilde{\phi}(0)=9/8 ϕ ( 0 ) = 9/8 , its derivative is even, is one on [ − a 1 , a 1 ] [-a_1,a_1] [ − a 1 , a 1 ] , and satisfies 0 ≤ ϕ ~ ′ ≤ 1 0\le\widetilde{\phi}'\le1 0 ≤ ϕ ′ ≤ 1 , ∣ ϕ ~ ′ ′ ∣ ≤ 4 |\widetilde{\phi}''|\le4 ∣ ϕ ′′ ∣ ≤ 4 , ∣ ϕ ~ ′ ′ ′ ∣ < 100 |\widetilde{\phi}'''|<100 ∣ ϕ ′′′ ∣ < 100 .
For the chain (37) with these gates, denoted g ~ m \widetilde{g}_m g m , put p = ∇ g ~ m ( y ) p=\nabla\widetilde{g}_m(y) p = ∇ g m ( y ) and B = ∇ 2 g ~ m ( y ) B=\nabla^2\widetilde{g}_m(y) B = ∇ 2 g m ( y ) . Then
p i ≤ 0 , ∥ p ∥ ≤ 10 m , ∥ B ∥ o p ≤ 42 , ∥ D 3 g ~ m ∥ o p ≤ 1872 , (65) p_i\le0,\qquad\lVert p\rVert\le10\sqrt{m},\qquad\lVert B\rVert_{\mathrm{op}}\le42,\qquad\lVert D^3\widetilde{g}_m\rVert_{\mathrm{op}}\le1872, \tag*{(65)} p i ≤ 0 , ∥ p ∥ ≤ 10 m , ∥ B ∥ op ≤ 42 , ∥ D 3 g m ∥ op ≤ 1872 , (65) ∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m , ∣ y ⊤ p ∣ ≤ 8 m , (66) \lVert By\rVert\le44\sqrt{m},\qquad|y^\top By|\le44m,\qquad|y^\top p|\le8m, \tag*{(66)} ∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m , ∣ y ⊤ p ∣ ≤ 8 m , (66) g ~ m ( 0 ) − inf g ~ m ≤ 9 4 ( m − 1 2 ) . (67) \widetilde{g}_m(0)-\inf\widetilde{g}_m\le\frac{9}{4}\left(m-\frac{1}{2}\right). \tag*{(67)} g m ( 0 ) − inf g m ≤ 4 9 ( m − 2 1 ) . (67) If some ∣ y i ∣ < a 1 |y_i|<a_1 ∣ y i ∣ < a 1 , the first such index j j j satisfies p j ≤ − 1 p_j\le-1 p j ≤ − 1 . If y j = 0 y_j=0 y j = 0 and ∣ y i ∣ ≤ a 0 |y_i|\le a_0 ∣ y i ∣ ≤ a 0 for i > j i>j i > j , the exact prefix identities of (39) hold; here ϕ ~ ( 0 ) = 1 \widetilde{\phi}(0)=1 ϕ ( 0 ) = 1 , and the current derivative is − γ j − 1 -\gamma_{j-1} − γ j − 1 , where γ 0 = 1 \gamma_0=1 γ 0 = 1 and γ j − 1 = ψ ~ ( − y j − 1 ) + ψ ~ ( y j − 1 ) ∈ [ 0 , 1 ] \gamma_{j-1}=\widetilde{\psi}(-y_{j-1})+\widetilde{\psi}(y_{j-1})\in[0,1] γ j − 1 = ψ ( − y j − 1 ) + ψ ( y j − 1 ) ∈ [ 0 , 1 ] for j > 1 j>1 j > 1 . Values at such a point may still depend on several earlier coordinates.
Proof. Convolution with a nonnegative probability kernel preserves all the stated bounds on the values and the first and second derivatives. The support and plateau claims follow by enlarging endpoints by η \eta η and applying the shift 2 η 2\eta 2 η . Evenness gives φ ~ ( 0 ) = 9 / 8 \widetilde{\varphi}(0)=9/8 φ ( 0 ) = 9/8 and an even derivative. On [ − 3 / 5 , 3 / 5 ] [-3/5,3/5] [ − 3/5 , 3/5 ] , 1 / ( 1 − t 2 ) ≤ 1 + ( 25 / 16 ) t 2 1/(1-t^2)\le1+(25/16)t^2 1/ ( 1 − t 2 ) ≤ 1 + ( 25/16 ) t 2 gives exp [ − 1 / ( 1 − t 2 ) ] ≥ e − 1 ( 1 − ( 25 / 16 ) t 2 ) \exp[-1/(1-t^2)]\ge e^{-1}(1-(25/16)t^2) exp [ − 1/ ( 1 − t 2 )] ≥ e − 1 ( 1 − ( 25/16 ) t 2 ) , so integrating over [ − 3 / 5 , 3 / 5 ] [-3/5,3/5] [ − 3/5 , 3/5 ] yields Z > ( 39 / 40 ) e − 1 Z>(39/40)e^{-1} Z > ( 39/40 ) e − 1 and w ( 0 ) = e − 1 / Z < 40 / 39 w(0)=e^{-1}/Z<40/39 w ( 0 ) = e − 1 / Z < 40/39 . Because φ ′ ′ \varphi^{\prime\prime} φ ′′ is a step function with jumps of magnitude 4 4 4 at { − 5 / 4 , − 1 , 1 , 5 / 4 } \{-5/4,-1,1,5/4\} { − 5/4 , − 1 , 1 , 5/4 } and ψ ′ ′ \psi^{\prime\prime} ψ ′′ has jumps of magnitude at most 32 32 32 at { 1 / 2 , 3 / 4 , 1 } \{1/2,3/4,1\} { 1/2 , 3/4 , 1 } , and the jumps in each set are separated by 1 / 4 > 2 η 1/4>2\eta 1/4 > 2 η , the bumps in φ ~ ′ ′ ′ = φ ′ ′ ′ ∗ w η \widetilde{\varphi}^{\prime\prime\prime}=\varphi^{\prime\prime\prime}*w_\eta φ ′′′ = φ ′′′ ∗ w η and ψ ~ ′ ′ ′ ( t ) = ( ψ ′ ′ ′ ∗ w η ) ( t + 2 η ) \widetilde{\psi}^{\prime\prime\prime}(t)=(\psi^{\prime\prime\prime}*w_\eta)(t+2\eta) ψ ′′′ ( t ) = ( ψ ′′′ ∗ w η ) ( t + 2 η ) have disjoint supports. Thus ∣ φ ~ ′ ′ ′ ∣ ≤ 4 w ( 0 ) / η < 100 |\widetilde{\varphi}^{\prime\prime\prime}|\le4w(0)/\eta<100 ∣ φ ′′′ ∣ ≤ 4 w ( 0 ) / η < 100 and ∣ ψ ~ ′ ′ ′ ∣ ≤ 32 w ( 0 ) / η < 788 |\widetilde{\psi}^{\prime\prime\prime}|\le32w(0)/\eta<788 ∣ ψ ′′′ ∣ ≤ 32 w ( 0 ) / η < 788 .
The coordinate gradient is the sum of a nonpositive predecessor term of magnitude at most 1 1 1 and a nonpositive successor term of magnitude at most 4 ( 9 / 4 ) = 9 4(9/4)=9 4 ( 9/4 ) = 9 . Since φ ~ ′ ′ \widetilde{\varphi}^{\prime\prime} φ ′′ and φ ~ ′ ′ ′ \widetilde{\varphi}^{\prime\prime\prime} φ ′′′ are supported on { ∣ t ∣ ∈ [ a 1 , 5 / 4 + η ] } \{|t|\in[a_1,5/4+\eta]\} { ∣ t ∣ ∈ [ a 1 , 5/4 + η ]} whereas ψ ~ ′ \widetilde{\psi}^{\prime} ψ ′ , ψ ~ ′ ′ \widetilde{\psi}^{\prime\prime} ψ ′′ , and ψ ~ ′ ′ ′ \widetilde{\psi}^{\prime\prime\prime} ψ ′′′ are supported on { ∣ t ∣ ∈ [ a 0 , a 1 ] } \{|t|\in[a_0,a_1]\} { ∣ t ∣ ∈ [ a 0 , a 1 ]} , their supports meet only at ∣ t ∣ = a 1 |t|=a_1 ∣ t ∣ = a 1 , where both higher derivatives vanish. Convolving (40) with w η ≥ 0 w_\eta\ge0 w η ≥ 0 gives 4 + 16 φ ~ ( s ) + 4 φ ~ ′ ( s ) ≤ 42 4+16\widetilde{\varphi}(s)+4\widetilde{\varphi}^{\prime}(s)\le42 4 + 16 φ ( s ) + 4 φ ′ ( s ) ≤ 42 for every s ∈ R s\in\mathbb{R} s ∈ R , so the Hessian row sum is at most 42 42 42 for ∣ y i ∣ ≤ a 1 |y_i|\le a_1 ∣ y i ∣ ≤ a 1 and at most 8 8 8 for ∣ y i ∣ > a 1 |y_i|>a_1 ∣ y i ∣ > a 1 , proving ∥ B ∥ op ≤ 42 \lVert B\rVert_{\operatorname{op}}\le42 ∥ B ∥ op ≤ 42 . Similarly, a diagonal third derivative has magnitude at most max { 100 , ( 9 / 4 ) 788 } = 1773 \max\{100,(9/4)788\}=1773 max { 100 , ( 9/4 ) 788 } = 1773 , and adding the six types of adjacent mixed trilinear terms (each bounded by 16 16 16 ) gives 1773 + 6 ⋅ 16 < 1872 1773+6\cdot16<1872 1773 + 6 ⋅ 16 < 1872 .
Every active coordinate has ∣ y i ∣ ≤ 5 / 4 + η = 31 / 24 < 4 / 3 |y_i|\le5/4+\eta=31/24<4/3 ∣ y i ∣ ≤ 5/4 + η = 31/24 < 4/3 . For ∣ y i ∣ ≤ a 1 |y_i|\le a_1 ∣ y i ∣ ≤ a 1 , ∣ ( B y ) i ∣ ≤ 36 a 1 + 4 a 1 + 4 ( 31 / 24 ) = 87 / 2 < 44 |(By)_i|\le36a_1+4a_1+4(31/24)=87/2<44 ∣ ( B y ) i ∣ ≤ 36 a 1 + 4 a 1 + 4 ( 31/24 ) = 87/2 < 44 and ∣ y i ( B y ) i ∣ < 44 |y_i(By)_i|<44 ∣ y i ( B y ) i ∣ < 44 ; for a 1 < ∣ y i ∣ ≤ 31 / 24 a_1<|y_i|\le31/24 a 1 < ∣ y i ∣ ≤ 31/24 , ∣ ( B y ) i ∣ ≤ 4 ( 31 / 24 ) + 4 a 1 < 10 |(By)_i|\le4(31/24)+4a_1<10 ∣ ( B y ) i ∣ ≤ 4 ( 31/24 ) + 4 a 1 < 10 . Also s ψ ′ ( s ) ≤ 3 s\psi'(s)\le3 s ψ ′ ( s ) ≤ 3 and ∣ s φ ′ ( s ) ∣ ≤ 1 |s\varphi'(s)|\le1 ∣ s φ ′ ( s ) ∣ ≤ 1 imply t ψ ~ ′ ( t ) ≤ 3 t\widetilde{\psi}'(t)\le3 t ψ ′ ( t ) ≤ 3 and ∣ t φ ~ ′ ( t ) ∣ ≤ 1 + η |t\widetilde{\varphi}'(t)|\le1+\eta ∣ t φ ′ ( t ) ∣ ≤ 1 + η , so ∣ y i p i ∣ ≤ 1 + η + 27 / 4 < 8 |y_ip_i|\le1+\eta+27/4<8 ∣ y i p i ∣ ≤ 1 + η + 27/4 < 8 . The barrier and prefix identities follow as in Lemma D.1.
Perspective scaling and exact derivatives
For R > 0 R>0 R > 0 , define the perspective weight, compression factor, and bounded radial compression map on R d \mathbb{R}^d R d by
w ( x ) = 1 + ∥ x ∥ 2 R 2 , a ( x ) = 1 w ( x ) , z ( x ) = ρ R ( x ) = x w ( x ) = a ( x ) x . (68) w(x)=\sqrt{1+\frac{\lVert x\rVert^2}{R^2}},\qquad a(x)=\frac{1}{w(x)},\qquad z(x)=\rho_R(x)=\frac{x}{w(x)}=a(x)x. \tag*{(68)} w ( x ) = 1 + R 2 ∥ x ∥ 2 , a ( x ) = w ( x ) 1 , z ( x ) = ρ R ( x ) = w ( x ) x = a ( x ) x . (68)
Then w ∈ C ∞ ( R d ) w\in C^\infty(\mathbb{R}^d) w ∈ C ∞ ( R d ) , w ( x ) ≥ 1 w(x)\ge1 w ( x ) ≥ 1 , 0 < a ( x ) ≤ 1 0<a(x)\le1 0 < a ( x ) ≤ 1 , and ∥ z ( x ) ∥ < R \lVert z(x)\rVert<R ∥ z ( x )∥ < R for all x ∈ R d x\in\mathbb{R}^d x ∈ R d . Multiplying a scalar chain by w ( x ) w(x) w ( x ) while evaluating it at the compressed coordinates U ⊤ z ( x ) = U ⊤ x / w ( x ) U^\top z(x)=U^\top x/w(x) U ⊤ z ( x ) = U ⊤ x / w ( x ) forms the perspective function ([10 Section 3.2.6] ) of G G G evaluated at ( U ⊤ x , w ( x ) ) (U^\top x,w(x)) ( U ⊤ x , w ( x )) , namely w ( x ) G ( U ⊤ x / w ( x ) ) w(x)G(U^\top x/w(x)) w ( x ) G ( U ⊤ x / w ( x )) . Because w ( x ) a ( x ) = 1 w(x)a(x)=1 w ( x ) a ( x ) = 1 identically, the weight w ( x ) w(x) w ( x ) cancels the radial attenuation a ( x ) a(x) a ( x ) in the leading chain gradient while keeping the second derivative bounded across all of R d \mathbb{R}^d R d .
Lemma G.6 (Exact derivatives under perspective scaling). Let U = ( u 1 , … , u m ) ∈ R d × m U=(u_1,\ldots,u_m)\in\mathbb{R}^{d\times m} U = ( u 1 , … , u m ) ∈ R d × m have orthonormal columns, let G : R m → R G:\mathbb{R}^m\to\mathbb{R} G : R m → R be continuously differentiable, and set
Φ U ( x ) = w ( x ) G ( U ⊤ ρ R ( x ) ) , y = U ⊤ z ( x ) . \Phi_U(x)=w(x)G(U^\top\rho_R(x)),\qquad y=U^\top z(x). Φ U ( x ) = w ( x ) G ( U ⊤ ρ R ( x )) , y = U ⊤ z ( x ) . At every point x ∈ R d x\in\mathbb{R}^d x ∈ R d , with p = ∇ G ( y ) p=\nabla G(y) p = ∇ G ( y ) ,
∇ Φ U ( x ) = U p + [ G ( y ) − y ⊤ p ] z ( x ) R 2 . (69) \nabla\Phi_U(x)=Up+[G(y)-y^\top p]\frac{z(x)}{R^2}. \tag*{(69)} ∇ Φ U ( x ) = U p + [ G ( y ) − y ⊤ p ] R 2 z ( x ) . (69) At every point x ∈ R d x\in\mathbb{R}^d x ∈ R d where G G G is twice differentiable at y y y with Hessian B = ∇ 2 G ( y ) B=\nabla^2G(y) B = ∇ 2 G ( y ) , for every v ∈ R d v\in\mathbb{R}^d v ∈ R d , writing α = U ⊤ v \alpha=U^\top v α = U ⊤ v and β = z ( x ) ⊤ v / R 2 \beta=z(x)^\top v/R^2 β = z ( x ) ⊤ v / R 2 ,
v ⊤ ∇ 2 Φ U ( x ) v = a ( x ) ( α − β y ) ⊤ B ( α − β y ) + a ( x ) [ G ( y ) − y ⊤ p ] ( ∥ v ∥ 2 R 2 − β 2 ) . (70) v^\top\nabla^2\Phi_U(x)v=a(x)(\alpha-\beta y)^\top B(\alpha-\beta y)+a(x)[G(y)-y^\top p]\left(\frac{\lVert v\rVert^2}{R^2}-\beta^2\right). \tag*{(70)} v ⊤ ∇ 2 Φ U ( x ) v = a ( x ) ( α − β y ) ⊤ B ( α − β y ) + a ( x ) [ G ( y ) − y ⊤ p ] ( R 2 ∥ v ∥ 2 − β 2 ) . (70)
If G G G is three times differentiable at y y y , then for every v ∈ R d v \in\mathbb{R}^{d} v ∈ R d , with u = α − β y u=\alpha-\beta y u = α − β y ,
D 3 Φ U ( x ) [ v , v , v ] = a ( x ) 2 D 3 G ( y ) [ u , u , u ] − 3 a ( x ) 2 β u T B u − 3 a ( x ) 2 ( ∥ v ∥ 2 R 2 − β 2 ) u T B y − 3 a ( x ) 2 β [ G ( y ) − y T p ] ( ∥ v ∥ 2 R 2 − β 2 ) . (71) D^{3}\Phi_{U}(x)[v,v,v]=a(x)^{2}D^{3}G(y)[u,u,u]-3a(x)^{2}\beta u^{\mathsf T}Bu
-3a(x)^{2}\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)u^{\mathsf T}By-3a(x)^{2}\beta\left[G(y)-y^{\mathsf T}p\right]\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right). \tag*{(71)} D 3 Φ U ( x ) [ v , v , v ] = a ( x ) 2 D 3 G ( y ) [ u , u , u ] − 3 a ( x ) 2 β u T B u − 3 a ( x ) 2 ( R 2 ∥ v ∥ 2 − β 2 ) u T B y − 3 a ( x ) 2 β [ G ( y ) − y T p ] ( R 2 ∥ v ∥ 2 − β 2 ) . (71)
Proof. Fix v ∈ R d v \in\mathbb{R}^{d} v ∈ R d and write α = U T v \alpha=U^{\mathsf T}v α = U T v and β = ⟨ z ( x ) / R 2 , v ⟩ \beta=\langle z(x)/R^{2},v\rangle β = ⟨ z ( x ) / R 2 , v ⟩ . Direct differentiation of w ( x ) = ( 1 + ∥ x ∥ 2 / R 2 ) 1 / 2 w(x)=(1+\lVert x\rVert^{2}/R^{2})^{1/2} w ( x ) = ( 1 + ∥ x ∥ 2 / R 2 ) 1/2 and z ( x ) = x / w ( x ) z(x)=x/w(x) z ( x ) = x / w ( x ) gives
D w ( x ) v = x T v R 2 w ( x ) = β , D z ( x ) v = v w ( x ) − x ( D w ( x ) v ) w ( x ) 2 = a ( x ) ( v − β z ( x ) ) , Dw(x)v=\frac{x^{\mathsf T}v}{R^{2}w(x)}=\beta,\qquad Dz(x)v=\frac{v}{w(x)}-\frac{x(Dw(x)v)}{w(x)^{2}}=a(x)(v-\beta z(x)), D w ( x ) v = R 2 w ( x ) x T v = β , D z ( x ) v = w ( x ) v − w ( x ) 2 x ( D w ( x ) v ) = a ( x ) ( v − β z ( x )) ,
and hence D y ( x ) v = U T D z ( x ) v = a ( x ) ( α − β y ) Dy(x)v=U^{\mathsf T}Dz(x)v=a(x)(\alpha-\beta y) D y ( x ) v = U T D z ( x ) v = a ( x ) ( α − β y ) with u : = α − β y u:=\alpha-\beta y u := α − β y . By the product and chain rules,
D Φ U ( x ) v = ( D w ( x ) v ) G ( y ) + w ( x ) p T D y ( x ) v = β G ( y ) + w ( x ) a ( x ) p T ( α − β y ) = p T α + [ G ( y ) − y T p ] β , D\Phi_{U}(x)v=(Dw(x)v)G(y)+w(x)p^{\mathsf T}Dy(x)v=\beta G(y)+w(x)a(x)p^{\mathsf T}(\alpha-\beta y)=p^{\mathsf T}\alpha+[G(y)-y^{\mathsf T}p]\beta, D Φ U ( x ) v = ( D w ( x ) v ) G ( y ) + w ( x ) p T D y ( x ) v = β G ( y ) + w ( x ) a ( x ) p T ( α − β y ) = p T α + [ G ( y ) − y T p ] β ,
since w ( x ) a ( x ) = 1 w(x)a(x)=1 w ( x ) a ( x ) = 1 . Because α = U T v \alpha=U^{\mathsf T}v α = U T v and β = ⟨ z ( x ) / R 2 , v ⟩ \beta=\langle z(x)/R^{2},v\rangle β = ⟨ z ( x ) / R 2 , v ⟩ , this is (69) .
Next differentiate D Φ U ( x ) v = p ( y ) T α + [ G ( y ) − y T p ( y ) ] β D\Phi_{U}(x)v=p(y)^{\mathsf T}\alpha+[G(y)-y^{\mathsf T}p(y)]\beta D Φ U ( x ) v = p ( y ) T α + [ G ( y ) − y T p ( y )] β along the constant vector v v v . Since ∇ y [ G ( y ) − y T p ( y ) ] = p ( y ) − ( p ( y ) + B ( y ) y ) = − B ( y ) y \nabla_{y}[G(y)-y^{\mathsf T}p(y)]=p(y)-(p(y)+B(y)y)=-B(y)y ∇ y [ G ( y ) − y T p ( y )] = p ( y ) − ( p ( y ) + B ( y ) y ) = − B ( y ) y , the chain rule yields
D [ p ( y ) T α ] v = a ( x ) α T B ( y ) u , D [ G ( y ) − y T p ( y ) ] v = − a ( x ) y T B ( y ) u , D[p(y)^{\mathsf T}\alpha]v=a(x)\alpha^{\mathsf T}B(y)u,\qquad D[G(y)-y^{\mathsf T}p(y)]v=-a(x)y^{\mathsf T}B(y)u, D [ p ( y ) T α ] v = a ( x ) α T B ( y ) u , D [ G ( y ) − y T p ( y )] v = − a ( x ) y T B ( y ) u ,
while D β ( x ) v = v T D z ( x ) v / R 2 = a ( x ) ( ∥ v ∥ 2 / R 2 − β 2 ) D\beta(x)v=v^{\mathsf T}Dz(x)v/R^{2}=a(x)(\lVert v\rVert^{2}/R^{2}-\beta^{2}) D β ( x ) v = v T D z ( x ) v / R 2 = a ( x ) (∥ v ∥ 2 / R 2 − β 2 ) . Combining the terms gives
D 2 Φ U ( x ) [ v , v ] = a ( x ) ( α T − β y T ) B ( y ) u + a ( x ) [ G ( y ) − y T p ( y ) ] ( ∥ v ∥ 2 R 2 − β 2 ) , D^{2}\Phi_{U}(x)[v,v]=a(x)(\alpha^{\mathsf T}-\beta y^{\mathsf T})B(y)u+a(x)[G(y)-y^{\mathsf T}p(y)]\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right), D 2 Φ U ( x ) [ v , v ] = a ( x ) ( α T − β y T ) B ( y ) u + a ( x ) [ G ( y ) − y T p ( y )] ( R 2 ∥ v ∥ 2 − β 2 ) ,
which is (70) .
Finally, when G G G is three times differentiable at y y y , differentiate (70) once more along v v v . Using
D a ( x ) v = − a ( x ) 2 D w ( x ) v = − a ( x ) 2 β Da(x)v=-a(x)^{2}Dw(x)v=-a(x)^{2}\beta D a ( x ) v = − a ( x ) 2 D w ( x ) v = − a ( x ) 2 β
and
D u ( x ) v = − ( D β ( x ) v ) y − β ( D y ( x ) v ) = − a ( x ) ( ∥ v ∥ 2 R 2 − β 2 ) y − a ( x ) β u , Du(x)v=-(D\beta(x)v)y-\beta(Dy(x)v)=-a(x)\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)y-a(x)\beta u, D u ( x ) v = − ( D β ( x ) v ) y − β ( D y ( x ) v ) = − a ( x ) ( R 2 ∥ v ∥ 2 − β 2 ) y − a ( x ) β u ,
the derivative of a ( x ) u T B ( y ) u a(x)u^{\mathsf T}B(y)u a ( x ) u T B ( y ) u along v v v equals
− a ( x ) 2 β u T B ( y ) u + a ( x ) 2 D 3 G ( y ) [ u , u , u ] − 2 a ( x ) 2 ( ∥ v ∥ 2 R 2 − β 2 ) u T B ( y ) y − 2 a ( x ) 2 β u T B ( y ) u . -a(x)^{2}\beta u^{\mathsf T}B(y)u+a(x)^{2}D^{3}G(y)[u,u,u]-2a(x)^{2}\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)u^{\mathsf T}B(y)y-2a(x)^{2}\beta u^{\mathsf T}B(y)u. − a ( x ) 2 β u T B ( y ) u + a ( x ) 2 D 3 G ( y ) [ u , u , u ] − 2 a ( x ) 2 ( R 2 ∥ v ∥ 2 − β 2 ) u T B ( y ) y − 2 a ( x ) 2 β u T B ( y ) u .
Similarly, since
D [ a ( x ) ( ∥ v ∥ 2 R 2 − β 2 ) ] v = − a ( x ) 2 β ( ∥ v ∥ 2 R 2 − β 2 ) + a ( x ) ( − 2 β a ( x ) ( ∥ v ∥ 2 R 2 − β 2 ) ) = − 3 a ( x ) 2 β ( ∥ v ∥ 2 R 2 − β 2 ) , D\left[a(x)\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)\right]v=-a(x)^{2}\beta\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)+a(x)\left(-2\beta a(x)\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)\right)=-3a(x)^{2}\beta\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right), D [ a ( x ) ( R 2 ∥ v ∥ 2 − β 2 ) ] v = − a ( x ) 2 β ( R 2 ∥ v ∥ 2 − β 2 ) + a ( x ) ( − 2 β a ( x ) ( R 2 ∥ v ∥ 2 − β 2 ) ) = − 3 a ( x ) 2 β ( R 2 ∥ v ∥ 2 − β 2 ) ,
the derivative of a ( x ) [ G ( y ) − y T p ( y ) ] ( ∥ v ∥ 2 / R 2 − β 2 ) a(x)[G(y)-y^{\mathsf T}p(y)](\lVert v\rVert^{2}/R^{2}-\beta^{2}) a ( x ) [ G ( y ) − y T p ( y )] (∥ v ∥ 2 / R 2 − β 2 ) along v v v equals
− a ( x ) 2 ( ∥ v ∥ 2 R 2 − β 2 ) u T B ( y ) y − 3 a ( x ) 2 β [ G ( y ) − y T p ( y ) ] ( ∥ v ∥ 2 R 2 − β 2 ) . -a(x)^{2}\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)u^{\mathsf T}B(y)y-3a(x)^{2}\beta[G(y)-y^{\mathsf T}p(y)]\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right). − a ( x ) 2 ( R 2 ∥ v ∥ 2 − β 2 ) u T B ( y ) y − 3 a ( x ) 2 β [ G ( y ) − y T p ( y )] ( R 2 ∥ v ∥ 2 − β 2 ) .
Adding the two expressions yields (71) .
C ∞ C^\infty C ∞ global perspective witnesses with a Lipschitz Hessian
Now let g ~ m : R m → R \widetilde{g}_m:\mathbb{R}^m \to\mathbb{R} g m : R m → R be the C ∞ C^\infty C ∞ chain of Lemma G.5 formed from the convolved gates φ ~ , ψ ~ \widetilde{\varphi},\widetilde{\psi} φ , ψ of (64) with η = 1 / 24 \eta=1/24 η = 1/24 , a 0 = 3 / 8 a_0=3/8 a 0 = 3/8 , and a 1 = 23 / 24 a_1=23/24 a 1 = 23/24 . Define
g ^ m ( y ) : = g ~ m ( y ) + 9 4 m + 1 4 , F ~ U g ( x ) : = w ( x ) g ^ m ( U ⊤ ρ R ( x ) ) − 1 4 . (72) \widehat{g}_m(y):=\widetilde{g}_m(y)+\frac{9}{4}m+\frac{1}{4},\qquad\widetilde{F}^{\mathrm{g}}_U(x):=w(x)\widehat{g}_m(U^\top\rho_R(x))-\frac{1}{4}. \tag*{(72)} g m ( y ) := g m ( y ) + 4 9 m + 4 1 , F U g ( x ) := w ( x ) g m ( U ⊤ ρ R ( x )) − 4 1 . (72)
Because w w w and ρ R \rho_R ρ R are C ∞ C^\infty C ∞ on all of R d \mathbb{R}^d R d and g ~ m \widetilde{g}_m g m is C ∞ C^\infty C ∞ on R m \mathbb{R}^m R m , F ~ U g \widetilde{F}^{\mathrm{g}}_U F U g is C ∞ C^\infty C ∞ on R d \mathbb{R}^d R d .
Lemma G.7 (Analytic bounds for the C ∞ C^\infty C ∞ global perspective wrapper). Let m ≥ 1 m\ge1 m ≥ 1 and R 2 ≥ 64 m R^2\ge64m R 2 ≥ 64 m . For every orthonormal m m m -frame U U U in R d \mathbb{R}^d R d , the function F ~ U g \widetilde{F}^{\mathrm{g}}_U F U g in (72) is C ∞ C^\infty C ∞ , coercive, nonconvex, globally 58 58 58 -smooth, and globally 12 m 12\sqrt{m} 12 m -Lipschitz, with initial gap F ~ U g ( 0 ) − inf x ∈ R d F ~ U g ( x ) ≤ 9 4 ( m − 1 2 ) \widetilde{F}^{\mathrm{g}}_U(0)-\inf_{x\in\mathbb{R}^d}\widetilde{F}^{\mathrm{g}}_U(x)\le\frac{9}{4}(m-\frac{1}{2}) F U g ( 0 ) − inf x ∈ R d F U g ( x ) ≤ 4 9 ( m − 2 1 ) and Lip ( ∇ 2 F ~ U g ) < 2016 \operatorname{Lip}(\nabla^2\widetilde{F}^{\mathrm{g}}_U)<2016 Lip ( ∇ 2 F U g ) < 2016 . At every point x ∈ R d x\in\mathbb{R}^d x ∈ R d , its Hessian satisfies the two-sided global matrix bound
− H U g ⪯ ∇ 2 F ~ U g ( x ) ⪯ H U g : = 56 U U ⊤ + 84 m R 2 I , ∥ H U g ∥ o p < 58 , (73) -H^{\mathrm{g}}_U\preceq\nabla^2\widetilde{F}^{\mathrm{g}}_U(x)\preceq H^{\mathrm{g}}_U:=56UU^\top+\frac{84m}{R^2}I,\qquad\lVert H^{\mathrm{g}}_U\rVert_{\mathrm{op}}<58, \tag*{(73)} − H U g ⪯ ∇ 2 F U g ( x ) ⪯ H U g := 56 U U ⊤ + R 2 84 m I , ∥ H U g ∥ op < 58 , (73) and hence at every point x ∈ R d x\in\mathbb{R}^d x ∈ R d ,
tr ( [ ∇ 2 F ~ U g ( x ) ] + ) ≤ ∥ ∇ 2 F ~ U g ( x ) ∥ ∗ ≤ tr H U g = 56 m + 84 d m R 2 . (74) \operatorname{tr}\left([\nabla^2\widetilde{F}^{\mathrm{g}}_U(x)]_+\right)\le\lVert\nabla^2\widetilde{F}^{\mathrm{g}}_U(x)\rVert_*\le\operatorname{tr}H^{\mathrm{g}}_U=56m+\frac{84dm}{R^2}. \tag*{(74)} tr ( [ ∇ 2 F U g ( x ) ] + ) ≤ ∥ ∇ 2 F U g ( x ) ∥ ∗ ≤ tr H U g = 56 m + R 2 84 d m . (74) If any coordinate of y = U ⊤ ρ R ( x ) y=U^\top\rho_R(x) y = U ⊤ ρ R ( x ) has magnitude below a 1 = 23 / 24 a_1=23/24 a 1 = 23/24 , then ∥ ∇ F ~ U g ( x ) ∥ > 4 / 5 \lVert\nabla\widetilde{F}^{\mathrm{g}}_U(x)\rVert>4/5 ∥ ∇ F U g ( x )∥ > 4/5 . The prefix hypotheses of Lemma C.4 hold on all of R d \mathbb{R}^d R d with z = ρ R z=\rho_R z = ρ R , τ = a 0 = 3 / 8 \tau=a_0=3/8 τ = a 0 = 3/8 , and c j = γ j − 1 c_j=\gamma_{j-1} c j = γ j − 1 , where γ 0 = 1 \gamma_0=1 γ 0 = 1 and γ j − 1 ( x ) = ψ ~ ( − y j − 1 ) + ψ ~ ( y j − 1 ) ∈ [ 0 , 1 ] \gamma_{j-1}(x)=\widetilde{\psi}(-y_{j-1})+\widetilde{\psi}(y_{j-1})\in[0,1] γ j − 1 ( x ) = ψ ( − y j − 1 ) + ψ ( y j − 1 ) ∈ [ 0 , 1 ] for j > 1 j>1 j > 1 .
Proof. In (37) with the gates φ ~ , ψ ~ \widetilde{\varphi},\widetilde{\psi} φ , ψ , one has 0 ≤ φ ~ ≤ 9 / 4 0\le\widetilde{\varphi}\le9/4 0 ≤ φ ≤ 9/4 , 0 ≤ ψ ~ ≤ 1 0\le\widetilde{\psi}\le1 0 ≤ ψ ≤ 1 , and at most one of ψ ~ ( − t ) , ψ ~ ( t ) \widetilde{\psi}(-t),\widetilde{\psi}(t) ψ ( − t ) , ψ ( t ) is nonzero, so each of the m m m summands of g ~ m \widetilde{g}_m g m lies in [ − 9 / 4 , 9 / 4 ] [-9/4,9/4] [ − 9/4 , 9/4 ] and the initial summand − φ ~ ( y 1 ) -\widetilde{\varphi}(y_1) − φ ( y 1 ) lies in [ − 9 / 4 , 0 ] [-9/4,0] [ − 9/4 , 0 ] . Hence 1 / 4 ≤ g ^ m ( y ) < 9 2 m 1/4\le\widehat{g}_m(y)<\frac{9}{2}m 1/4 ≤ g m ( y ) < 2 9 m for all y ∈ R m y\in\mathbb{R}^m y ∈ R m . Since w ( x ) ≥ 1 w(x)\ge1 w ( x ) ≥ 1 , we get F ~ U g ( x ) ≥ ( w ( x ) − 1 ) / 4 ≥ 0 \widetilde{F}^{\mathrm{g}}_U(x)\ge(w(x)-1)/4\ge0 F U g ( x ) ≥ ( w ( x ) − 1 ) /4 ≥ 0 , which proves coercivity and inf F ~ U g ≥ 0 \inf\widetilde{F}^{\mathrm{g}}_U\ge0 inf F U g ≥ 0 . Because w ( 0 ) = 1 w(0)=1 w ( 0 ) = 1 and g ~ m ( 0 ) = − φ ~ ( 0 ) = − 9 / 8 \widetilde{g}_m(0)=-\widetilde{\varphi}(0)=-9/8 g m ( 0 ) = − φ ( 0 ) = − 9/8 , we have F ~ U g ( 0 ) = 9 4 ( m − 1 2 ) \widetilde{F}^{\mathrm{g}}_U(0)=\frac{9}{4}(m-\frac{1}{2}) F U g ( 0 ) = 4 9 ( m − 2 1 ) .
Write p = ∇ g ~ m ( y ) p=\nabla\widetilde{g}_m(y) p = ∇ g m ( y ) and B = ∇ 2 g ~ m ( y ) B=\nabla^2\widetilde{g}_m(y) B = ∇ 2 g m ( y ) . By Lemma G.5, ∥ p ∥ ≤ 10 m \lVert p\rVert\le10\sqrt{m} ∥ p ∥ ≤ 10 m , ∥ B ∥ o p ≤ 42 \lVert B\rVert_{\mathrm{op}}\le42 ∥ B ∥ op ≤ 42 , ∥ D 3 g ~ m ∥ o p ≤ 1872 \lVert D^3\widetilde{g}_m\rVert_{\mathrm{op}}\le1872 ∥ D 3 g m ∥ op ≤ 1872 , ∥ B y ∥ ≤ 44 m \lVert By\rVert\le44\sqrt{m} ∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m \lvert y^\top By\rvert\le44m ∣ y ⊤ B y ∣ ≤ 44 m , and ∣ y ⊤ p ∣ ≤ 8 m \lvert y^\top p\rvert\le8m ∣ y ⊤ p ∣ ≤ 8 m . Therefore
∣ g ^ m ( y ) − y ⊤ p ∣ ≤ ( 9 2 + 8 ) m = 25 2 m . \left\lvert\widehat{g}_m(y)-y^\top p\right\rvert\le\left(\frac{9}{2}+8\right)m=\frac{25}{2}m. g m ( y ) − y ⊤ p ≤ ( 2 9 + 8 ) m = 2 25 m .
For R ≥ 8 m ≥ 8 R\ge8\sqrt{m}\ge8 R ≥ 8 m ≥ 8 , identity (69) gives
∥ ∇ F ~ U g ( x ) ∥ ≤ 10 m + 25 m 2 R ≤ ( 10 + 25 16 ) m < 12 m . \left\lVert\nabla\widetilde{F}^{\mathrm{g}}_U(x)\right\rVert\le10\sqrt{m}+\frac{25m}{2R}\le\left(10+\frac{25}{16}\right)\sqrt{m}<12\sqrt{m}. ∇ F U g ( x ) ≤ 10 m + 2 R 25 m ≤ ( 10 + 16 25 ) m < 12 m .
For any v ∈ R d v\in\mathbb{R}^d v ∈ R d , put α = U ⊤ v \alpha=U^\top v α = U ⊤ v and β = z ( x ) ⊤ v / R 2 \beta=z(x)^\top v/R^2 β = z ( x ) ⊤ v / R 2 . Since ∥ z ( x ) ∥ 2 / R 2 = 1 − a ( x ) 2 \lVert z(x)\rVert^2/R^2=1-a(x)^2 ∥ z ( x ) ∥ 2 / R 2 = 1 − a ( x ) 2 , Cauchy–Schwarz gives β 2 ≤ ( 1 − a ( x ) 2 ) ∥ v ∥ 2 / R 2 \beta^2\le(1-a(x)^2)\lVert v\rVert^2/R^2 β 2 ≤ ( 1 − a ( x ) 2 ) ∥ v ∥ 2 / R 2 , and hence
a ( x ) β 2 ≤ a ( x ) ( 1 − a ( x ) 2 ) ∥ v ∥ 2 R 2 ≤ 2 3 3 ∥ v ∥ 2 R 2 < 7 18 ∥ v ∥ 2 R 2 . (75) a(x)\beta^2\le a(x)(1-a(x)^2)\frac{\lVert v\rVert^2}{R^2}\le\frac{2}{3\sqrt{3}}\frac{\lVert v\rVert^2}{R^2}<\frac{7}{18}\frac{\lVert v\rVert^2}{R^2}. \tag*{(75)} a ( x ) β 2 ≤ a ( x ) ( 1 − a ( x ) 2 ) R 2 ∥ v ∥ 2 ≤ 3 3 2 R 2 ∥ v ∥ 2 < 18 7 R 2 ∥ v ∥ 2 . (75)
By the bounds on B B B above and Young’s inequality with parameter γ = 7 / 22 \gamma=7/22 γ = 7/22 ,
a ( x ) ∣ ( α − β y ) ⊤ B ( α − β y ) ∣ ≤ ( 42 + 44 γ ) ∥ α ∥ 2 + ( 44 γ + 44 ) m a ( x ) β 2 ≤ 56 ∥ U ⊤ v ∥ 2 + 638 m 9 R 2 ∥ v ∥ 2 . a(x)\left\lvert(\alpha-\beta y)^\top B(\alpha-\beta y)\right\rvert\le(42+44\gamma)\lVert\alpha\rVert^2+\left(\frac{44}{\gamma}+44\right)ma(x)\beta^2\le56\lVert U^\top v\rVert^2+\frac{638m}{9R^2}\lVert v\rVert^2. a ( x ) ( α − β y ) ⊤ B ( α − β y ) ≤ ( 42 + 44 γ ) ∥ α ∥ 2 + ( γ 44 + 44 ) ma ( x ) β 2 ≤ 56 ∥ U ⊤ v ∥ 2 + 9 R 2 638 m ∥ v ∥ 2 .
Adding the second term of (70) , which has magnitude at most 25 m 2 R 2 ∥ v ∥ 2 \frac{25m}{2R^{2}}\|v\|^{2} 2 R 2 25 m ∥ v ∥ 2 , and noting that 638 / 9 + 25 / 2 = 1501 / 18 < 84 638/9+25/2=1501/18<84 638/9 + 25/2 = 1501/18 < 84 , we obtain (73) and (74) at every x ∈ R d x\in\mathbb{R}^{d} x ∈ R d . Since R 2 ≥ 64 m R^{2}\ge64m R 2 ≥ 64 m , ∥ H ~ U g ~ ∥ o p ≤ 56 + 84 / 64 < 58 \|\widetilde{H}_{U}^{\widetilde{g}}\|_{\mathrm{op}}\le56+84/64<58 ∥ H U g ∥ op ≤ 56 + 84/64 < 58 .
If some ∣ y i ∣ < a 1 |y_i|<a_1 ∣ y i ∣ < a 1 , the smallest such index j j j satisfies p j ≤ − 1 p_j\le-1 p j ≤ − 1 by Lemma G.5. Projecting (69) onto u j u_j u j gives
u j T ∇ F ~ U g ~ ( x ) = p j + [ g ~ m ( y ) − y T p ] y j R 2 ≤ − 1 + 25 m 2 R 2 ≤ − 1 + 25 128 = − 103 128 < − 4 5 . u_j^{\mathsf T}\nabla\widetilde{F}_{U}^{\widetilde{g}}(x)=p_j+\left[\widetilde{g}_m(y)-y^{\mathsf T}p\right]\frac{y_j}{R^2}\le-1+\frac{25m}{2R^2}\le-1+\frac{25}{128}=-\frac{103}{128}<-\frac45. u j T ∇ F U g ( x ) = p j + [ g m ( y ) − y T p ] R 2 y j ≤ − 1 + 2 R 2 25 m ≤ − 1 + 128 25 = − 128 103 < − 5 4 .
For nonconvexity, choose x = t u 1 x=tu_1 x = t u 1 (t < 0 t<0 t < 0 ) so that y 1 = − 9 / 8 y_1=-9/8 y 1 = − 9/8 and y 2 = ⋯ = y m = 0 y_2=\cdots=y_m=0 y 2 = ⋯ = y m = 0 . In a neighborhood of this point, ψ ~ ( − y 1 ) = 1 \widetilde{\psi}(-y_1)=1 ψ ( − y 1 ) = 1 and ψ ~ ( y 1 ) = 0 \widetilde{\psi}(y_1)=0 ψ ( y 1 ) = 0 are constant, because 9 / 8 > a 1 9/8>a_1 9/8 > a 1 and − 9 / 8 < a 0 -9/8<a_0 − 9/8 < a 0 , so B 11 = − φ ~ ′ ′ ( − 9 / 8 ) = − 4 B_{11}=-\widetilde{\varphi}^{\prime\prime}(-9/8)=-4 B 11 = − φ ′′ ( − 9/8 ) = − 4 ; indeed w η w_\eta w η is supported in [ − η , η ] [-\eta,\eta] [ − η , η ] and φ ′ ′ = 4 \varphi^{\prime\prime}=4 φ ′′ = 4 on [ − 5 / 4 , − 1 ] [-5/4,-1] [ − 5/4 , − 1 ] , which contains [ − 9 / 8 − η , − 9 / 8 + η ] [-9/8-\eta,-9/8+\eta] [ − 9/8 − η , − 9/8 + η ] . Since R ≥ 8 R\ge8 R ≥ 8 , 1 − y 1 2 / R 2 ≥ 1 − 81 / 4096 > 31 / 32 1-y_1^2/R^2\ge1-81/4096>31/32 1 − y 1 2 / R 2 ≥ 1 − 81/4096 > 31/32 and a ( x ) = 1 − y 1 2 / R 2 > 31 / 32 a(x)=\sqrt{1-y_1^2/R^2}>31/32 a ( x ) = 1 − y 1 2 / R 2 > 31/32 . Applying (70) with v = u 1 v=u_1 v = u 1 (α = e 1 \alpha=e_1 α = e 1 , β = y 1 / R 2 \beta=y_1/R^2 β = y 1 / R 2 ) gives
u 1 T ∇ 2 F ~ U g ~ ( x ) u 1 = − 4 a ( x ) ( 1 − y 1 2 R 2 ) 2 + a ( x ) [ g ~ m ( y ) − y 1 p 1 ] 1 − y 1 2 / R 2 R 2 ≤ − 4 ( 31 32 ) 5 + 25 128 < − 3 < 0. u_1^{\mathsf T}\nabla^2\widetilde{F}_{U}^{\widetilde{g}}(x)u_1=-4a(x)\left(1-\frac{y_1^2}{R^2}\right)^2+a(x)\left[\widetilde{g}_m(y)-y_1p_1\right]\frac{1-y_1^2/R^2}{R^2}\le-4\left(\frac{31}{32}\right)^5+\frac{25}{128}<-3<0. u 1 T ∇ 2 F U g ( x ) u 1 = − 4 a ( x ) ( 1 − R 2 y 1 2 ) 2 + a ( x ) [ g m ( y ) − y 1 p 1 ] R 2 1 − y 1 2 / R 2 ≤ − 4 ( 32 31 ) 5 + 128 25 < − 3 < 0.
For the third derivative, fix a unit vector v ∈ R d v\in\mathbb{R}^{d} v ∈ R d and write u = α − β y u=\alpha-\beta y u = α − β y and s = ∥ z ( x ) ∥ / R = 1 − a ( x ) 2 ∈ [ 0 , 1 ] s=\|z(x)\|/R=\sqrt{1-a(x)^2}\in[0,1] s = ∥ z ( x ) ∥/ R = 1 − a ( x ) 2 ∈ [ 0 , 1 ] . Let I ( y ) = { i : ∣ y i ∣ ≤ 4 / 3 } I(y)=\{i:|y_i|\le4/3\} I ( y ) = { i : ∣ y i ∣ ≤ 4/3 } be the active coordinate set of Lemma G.5, and let u I = P I ( y ) u u_I=P_{I(y)}u u I = P I ( y ) u . Since ∥ P I ( y ) y ∥ ≤ ( 4 / 3 ) m \|P_{I(y)}y\|\le(4/3)\sqrt{m} ∥ P I ( y ) y ∥ ≤ ( 4/3 ) m and ∣ β ∣ ≤ s / R ≤ s / ( 8 m ) |\beta|\le s/R\le s/(8\sqrt{m}) ∣ β ∣ ≤ s / R ≤ s / ( 8 m ) , we have ∥ u I ∥ ≤ ∥ α ∥ + ∣ β ∣ ∥ P I ( y ) y ∥ ≤ 1 + s / 6 \|u_I\|\le\|\alpha\|+|\beta|\|P_{I(y)}y\|\le1+s/6 ∥ u I ∥ ≤ ∥ α ∥ + ∣ β ∣∥ P I ( y ) y ∥ ≤ 1 + s /6 , and hence
a ( x ) 2 ∥ u I ∥ 3 ≤ ( 1 − s 2 ) ( 1 + s 6 ) 3 < 17 16 , a(x)^2\|u_I\|^3\le(1-s^2)\left(1+\frac{s}{6}\right)^3<\frac{17}{16}, a ( x ) 2 ∥ u I ∥ 3 ≤ ( 1 − s 2 ) ( 1 + 6 s ) 3 < 16 17 ,
because on [ 0 , 1 ] [0,1] [ 0 , 1 ] the middle expression increases while 5 s 2 + 12 s < 3 5s^2+12s<3 5 s 2 + 12 s < 3 and decreases afterwards, and the root of 5 s 2 + 12 s = 3 5s^2+12s=3 5 s 2 + 12 s = 3 lies in [ 0.228 , 0.229 ] [0.228,0.229] [ 0.228 , 0.229 ] , where the expression is at most ( 1 − 0.228 2 ) ( 1 + 0.229 / 6 ) 3 < 1.061 (1-0.228^2)(1+0.229/6)^3<1.061 ( 1 − 0.22 8 2 ) ( 1 + 0.229/6 ) 3 < 1.061 . Because B B B and D 3 g ~ m ( y ) D^3\widetilde{g}_m(y) D 3 g m ( y ) are supported on I ( y ) I(y) I ( y ) , identity (71) gives
∣ D 3 F ~ U g ~ ( x ) [ v , v , v ] ∣ ≤ 1872 ⋅ 17 16 + 3 R ( 56 + 638 m 9 R 2 ) + 3 R 2 ⋅ 7 6 ⋅ 44 m + 3 R 25 m 2 R 2 < 2016 , \left|D^3\widetilde{F}_{U}^{\widetilde{g}}(x)[v,v,v]\right|\le1872\cdot\frac{17}{16}+\frac{3}{R}\left(56+\frac{638m}{9R^2}\right)+\frac{3}{R^2}\cdot\frac{7}{6}\cdot44\sqrt{m}+\frac{3}{R}\frac{25m}{2R^2}<2016, D 3 F U g ( x ) [ v , v , v ] ≤ 1872 ⋅ 16 17 + R 3 ( 56 + 9 R 2 638 m ) + R 2 3 ⋅ 6 7 ⋅ 44 m + R 3 2 R 2 25 m < 2016 ,
using R ≥ 8 m ≥ 8 R\ge8\sqrt{m}\ge8 R ≥ 8 m ≥ 8 . Since D 3 F ~ U g ~ ( x ) D^3\widetilde{F}_{U}^{\widetilde{g}}(x) D 3 F U g ( x ) is a symmetric trilinear form on R d \mathbb{R}^{d} R d , its operator norm equals sup ∥ v ∥ = 1 ∣ D 3 F ~ U g ~ ( x ) [ v , v , v ] ∣ \sup_{\|v\|=1}|D^3\widetilde{F}_{U}^{\widetilde{g}}(x)[v,v,v]| sup ∥ v ∥ = 1 ∣ D 3 F U g ( x ) [ v , v , v ] ∣ ([6 ] ), so ∥ D 3 F ~ U g ~ ( x ) ∥ o p < 2016 \|D^3\widetilde{F}_{U}^{\widetilde{g}}(x)\|_{\mathrm{op}}<2016 ∥ D 3 F U g ( x ) ∥ op < 2016 , which integrates to Lip ( ∇ 2 F ~ U g ~ ) < 2016 \operatorname{Lip}(\nabla^2\widetilde{F}_{U}^{\widetilde{g}})<2016 Lip ( ∇ 2 F U g ) < 2016 .
Finally, suppose y j = 0 y_j=0 y j = 0 and ∣ y k ∣ ≤ a 0 |y_k|\le a_0 ∣ y k ∣ ≤ a 0 for all k > j k>j k > j , and put
r j ( x ) = w ( x ) g ~ m ( y 1 , … , y j − 1 , 0 , … , 0 ) − 1 / 4. r_j(x)=w(x)\widetilde{g}_m(y_1,\ldots,y_{j-1},0,\ldots,0)-1/4. r j ( x ) = w ( x ) g m ( y 1 , … , y j − 1 , 0 , … , 0 ) − 1/4.
The gates ψ ~ ( ± y k ) \widetilde{\psi}(\pm y_k) ψ ( ± y k ) vanish with their derivatives for all k ≥ j k\ge j k ≥ j , so g ~ m ( y ) = g ~ m ( y 1 , … , y j − 1 , 0 , … , 0 ) \widetilde{g}_m(y)=\widetilde{g}_m(y_1,\ldots,y_{j-1},0,\ldots,0) g m ( y ) = g m ( y 1 , … , y j − 1 , 0 , … , 0 ) , F ~ U g ~ ( x ) = r j ( x ) \widetilde{F}_{U}^{\widetilde{g}}(x)=r_j(x) F U g ( x ) = r j ( x ) , p k = 0 p_k=0 p k = 0 for k > j k>j k > j , and p j = − γ j − 1 ( x ) p_j=-\gamma_{j-1}(x) p j = − γ j − 1 ( x ) . Because y j = 0 y_j=0 y j = 0 , the product y j p j y_jp_j y j p j vanishes, so y T p y^{\mathsf T}p y T p agrees with the corresponding inner product for ( y 1 , … , y j − 1 , 0 , … , 0 ) (y_1,\ldots,y_{j-1},0,\ldots,0) ( y 1 , … , y j − 1 , 0 , … , 0 ) . Subtracting (69) for r j r_j r j from (69) for F ~ U g ~ \widetilde{F}_{U}^{\widetilde{g}} F U g cancels the z ( x ) / R 2 z(x)/R^2 z ( x ) / R 2 term and leaves ∇ F ~ U g ~ ( x ) − ∇ r j ( x ) = − γ j − 1 ( x ) u j \nabla\widetilde{F}_{U}^{\widetilde{g}}(x)-\nabla r_j(x)=-\gamma_{j-1}(x)u_j ∇ F U g ( x ) − ∇ r j ( x ) = − γ j − 1 ( x ) u j . Both r j r_j r j and γ j − 1 \gamma_{j-1} γ j − 1 depend only on u 1 , … , u j − 1 u_1,\ldots,u_{j-1} u 1 , … , u j − 1 , and y j = 0 < a 1 y_j=0<a_1 y j = 0 < a 1 gives ∥ ∇ F ~ U g ~ ( x ) ∥ > 4 / 5 \|\nabla\widetilde{F}_{U}^{\widetilde{g}}(x)\|>4/5 ∥∇ F U g ( x ) ∥ > 4/5 by the barrier above, so the hypotheses of Lemma C.4 hold with z = ρ R z=\rho R z = ρR and τ = a 0 \tau=a_0 τ = a 0 .
Theorem G.8 (C ∞ C^\infty C ∞ multistage lower bound on the global fixed-matrix class). Let m ≥ 2 m\ge2 m ≥ 2 satisfy
204 m ≤ T , 28560 m 2 log ( e + m ) ≤ E T , 6144 m log ( e + m ) ≤ d . (76) 204m\le T,\qquad28560m^2\log(e+m)\le ET,\qquad6144m\log(e+m)\le d. \tag*{(76)} 204 m ≤ T , 28560 m 2 log ( e + m ) ≤ E T , 6144 m log ( e + m ) ≤ d . (76)
Then, for either oracle, there are C ∞ C^\infty C ∞ , globally L L L -smooth, coercive, nonconvex witnesses in the global fixed-matrix class F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d (and hence in P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d , B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d , and A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d ) giving N d e t v a l ≥ N d e t l i n ≥ ⌊ d / 2 ⌋ m N_{\mathrm{det}}^{\mathrm{val}} \ge N_{\mathrm{det}}^{\mathrm{lin}} \ge\lfloor d/2\rfloor m N det val ≥ N det lin ≥ ⌊ d /2 ⌋ m . The witnesses are globally 2 L Δ 2\sqrt{L\Delta} 2 L Δ -Lipschitz, satisfy the global nuclear norm bound ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E \|\nabla^2 f(x)\|_* \le LE ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E at every x ∈ R d x \in\mathbb{R}^d x ∈ R d , and have
Lip ( ∇ 2 f ) < 1 2 ( 204 m T ) 2 L 2 ε . (77) \operatorname{Lip}(\nabla^2 f) < \frac{1}{2}\left(\frac{204m}{T}\right)^2\frac{L^2}{\varepsilon}. \tag*{(77)} Lip ( ∇ 2 f ) < 2 1 ( T 204 m ) 2 ε L 2 . (77)
Proof. Set ℓ = log ( e + m ) \ell= \log(e+m) ℓ = log ( e + m ) , R 2 = d / ( 96 ℓ ) R^2 = d/(96\ell) R 2 = d / ( 96 ℓ ) , L ′ = 204 m L / T L' = 204mL/T L ′ = 204 m L / T , and E ′ = 140 m ℓ E' = 140m\ell E ′ = 140 m ℓ . Conditions (76) imply R 2 ≥ 64 m R^2 \ge64m R 2 ≥ 64 m , m ≤ d / 128 m \le d/128 m ≤ d /128 , L ′ ≤ L L' \le L L ′ ≤ L , and L ′ E ′ = 28560 m 2 ℓ L / T ≤ L E L'E' = 28560m^2\ell L/T \le LE L ′ E ′ = 28560 m 2 ℓ L / T ≤ L E . By (73) , since ℓ ≥ log ( e + 2 ) > 3 / 2 \ell\ge\log(e+2) > 3/2 ℓ ≥ log ( e + 2 ) > 3/2 ,
tr H U g 58 = 56 m + 84 ⋅ 96 m ℓ 58 < 12152 87 m ℓ < 140 m ℓ = E ′ . \frac{\operatorname{tr} H_U^g}{58} = \frac{56m+84\cdot96m\ell}{58} < \frac{12152}{87}m\ell< 140m\ell= E'. 58 tr H U g = 58 56 m + 84 ⋅ 96 m ℓ < 87 12152 m ℓ < 140 m ℓ = E ′ .
Put h = ⌊ d / 2 ⌋ h=\lfloor d/2\rfloor h = ⌊ d /2 ⌋ . Then p = d − m − h + 1 ≥ 63 d / 128 p=d-m-h+1\ge63d/128 p = d − m − h + 1 ≥ 63 d /128 and M / p ≤ 64 m / 63 M/p\le64m/63 M / p ≤ 64 m /63 for M ≤ h m M\le hm M ≤ hm . Since log ( e + 64 m / 63 ) ≤ ( 64 / 63 ) ℓ \log(e+64m/63)\le(64/63)\ell log ( e + 64 m /63 ) ≤ ( 64/63 ) ℓ , Lemma G.4 gives a squared projection bound at most ( 144 / 25 ) ( 128 / 63 ) ( 64 / 63 ) / 96 < ( 3 / 8 ) 2 = a 0 2 (144/25)(128/63)(64/63)/96 < (3/8)^2=a_0^2 ( 144/25 ) ( 128/63 ) ( 64/63 ) /96 < ( 3/8 ) 2 = a 0 2 . Lemma C.4 with z = ρ R z=\rho_R z = ρ R , τ = a 0 \tau=a_0 τ = a 0 , b = 4 / 5 b=4/5 b = 4/5 , and Q = h m Q=hm Q = hm therefore applies to F ~ U g \widetilde{F}_U^g F U g .
Scale F ~ U g \widetilde{F}_U^g F U g via Lemma E.1 with Λ = 58 \Lambda=58 Λ = 58 and b = 4 / 5 b=4/5 b = 4/5 :
σ = 58 ε L ′ b , α = 58 ε 2 L ′ b 2 , f ( x ) = α F ~ U g ( x / σ ) . \sigma=\frac{58\varepsilon}{L'b},\qquad\alpha=\frac{58\varepsilon^2}{L'b^2},\qquad f(x)=\alpha\widetilde{F}_U^g(x/\sigma). σ = L ′ b 58 ε , α = L ′ b 2 58 ε 2 , f ( x ) = α F U g ( x / σ ) .
Since 58 ( 9 / 4 ) / b 2 = 6525 / 32 < 204 58(9/4)/b^2=6525/32<204 58 ( 9/4 ) / b 2 = 6525/32 < 204 , the initial gap is at most 204 ( m − 1 / 2 ) ε 2 / L ′ ≤ Δ 204(m-1/2)\varepsilon^2/L'\le\Delta 204 ( m − 1/2 ) ε 2 / L ′ ≤ Δ . The global upper matrix ( L ′ / 58 ) H U g (L'/58)H_U^g ( L ′ /58 ) H U g has operator norm at most L ′ ≤ L L'\le L L ′ ≤ L and trace at most L ′ E ′ ≤ L E L'E'\le LE L ′ E ′ ≤ L E on all of R d \mathbb{R}^d R d , and (74) gives ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E \|\nabla^2 f(x)\|_*\le LE ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E for all x ∈ R d x\in\mathbb{R}^d x ∈ R d . Thus f ∈ F L , Δ , E d f\in\mathcal{F}^{d}_{L,\Delta,E} f ∈ F L , Δ , E d , and at the algorithm’s output ∥ ∇ f ( x ^ ) ∥ > ε \|\nabla f(\widehat{x})\|>\varepsilon ∥∇ f ( x ) ∥ > ε . The Hessian Lipschitz constant of f f f is at most
α σ 3 2016 = 2016 b ( L ′ ) 2 58 2 ε = 2016 4205 ( 204 m T ) 2 L 2 ε < 1 2 ( 204 m T ) 2 L 2 ε , \frac{\alpha}{\sigma^3}2016=\frac{2016b(L')^2}{58^2\varepsilon}=\frac{2016}{4205}\left(\frac{204m}{T}\right)^2\frac{L^2}{\varepsilon}<\frac{1}{2}\left(\frac{204m}{T}\right)^2\frac{L^2}{\varepsilon}, σ 3 α 2016 = 5 8 2 ε 2016 b ( L ′ ) 2 = 4205 2016 ( T 204 m ) 2 ε L 2 < 2 1 ( T 204 m ) 2 ε L 2 ,
which proves (77) . Finally, ∥ ∇ f ( x ) ∥ ≤ ( 12 / b ) m ε = 15 m ε ≤ 2 L Δ \|\nabla f(x)\|\le(12/b)\sqrt{m}\varepsilon=15\sqrt{m}\varepsilon\le2\sqrt{L\Delta} ∥∇ f ( x ) ∥ ≤ ( 12/ b ) m ε = 15 m ε ≤ 2 L Δ because T ≥ 204 m T\ge204m T ≥ 204 m . □
Corollary G.9 (Rate on the global class with a prescribed Hessian Lipschitz constant). For any ρ ≥ 0 \rho\ge0 ρ ≥ 0 , on the C ∞ C^\infty C ∞ members of F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d (and hence of P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d and B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d ) satisfying ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E \|\nabla^2f(x)\|_*\le LE ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E everywhere on R d \mathbb{R}^d R d and Lip ( ∇ 2 f ) ≤ ρ \operatorname{Lip}(\nabla^2f)\le\rho Lip ( ∇ 2 f ) ≤ ρ , if E T ≤ d ET\le d E T ≤ d , then for either oracle,
N d e t v a l ≥ N d e t l i n ≥ max { d , d 612 min { T , 2 ρ ε L , E T log ( e + T ) } } . (78) N_{\mathrm{det}}^{\mathrm{val}}\ge N_{\mathrm{det}}^{\mathrm{lin}}\ge\max\left\{d,\frac{d}{612}\min\left\{T,\frac{\sqrt{2\rho\varepsilon}}{L},\sqrt{\frac{ET}{\log(e+T)}}\right\}\right\}. \tag*{(78)} N det val ≥ N det lin ≥ max { d , 612 d min { T , L 2 ρε , log ( e + T ) E T } } . (78) In particular, if ρ ≥ L 2 / ( 2 ε ) \rho\ge L^2/(2\varepsilon) ρ ≥ L 2 / ( 2 ε ) , the middle term in the minimum is redundant and
N d e t ( F L , Δ , E d ) ≥ max { d , d 612 min { T , E T log ( e + T ) } } ≥ d 672 min { T , E T log ( e + T ) } . N_{\mathrm{det}}(\mathcal{F}^{d}_{L,\Delta,E})\ge\max\left\{d,\frac{d}{612}\min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\}\right\}\ge\frac{d}{672}\min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\}. N det ( F L , Δ , E d ) ≥ max { d , 612 d min { T , log ( e + T ) E T } } ≥ 672 d min { T , log ( e + T ) E T } .
Proof. Let M M M denote the minimum of three terms in (78) , put z = M / 204 z=M/204 z = M /204 , and set m = ⌊ z ⌋ m=\lfloor z\rfloor m = ⌊ z ⌋ . When z < 3 z<3 z < 3 , the rank-one quadratic dimension floor N d e t l i n ( F L , Δ , E d ) ≥ d N_{\mathrm{det}}^{\mathrm{lin}}(\mathcal{F}^{d}_{L,\Delta,E})\ge d N det lin ( F L , Δ , E d ) ≥ d of Lemma E.2 (whose witness is C ∞ C^\infty C ∞ with constant Hessian L u u ⊤ Luu^\top Lu u ⊤ , so Lip ( ∇ 2 f ) = 0 \operatorname{Lip}(\nabla^2f)=0 Lip ( ∇ 2 f ) = 0 and ∥ ∇ 2 f ∥ ∗ = L ≤ L E \|\nabla^2f\|_*=L\le LE ∥ ∇ 2 f ∥ ∗ = L ≤ L E ) exceeds d z / 3 = d M / 612 dz/3=dM/612 d z /3 = d M /612 . When z ≥ 3 z\ge3 z ≥ 3 , we have m ≥ 3 m\ge3 m ≥ 3 , 204 m ≤ M ≤ T 204m\le M\le T 204 m ≤ M ≤ T , 28560 m 2 log ( e + m ) ≤ 204 2 m 2 log ( e + T ) ≤ E T 28560m^2\log(e+m)\le204^2m^2\log(e+T)\le ET 28560 m 2 log ( e + m ) ≤ 20 4 2 m 2 log ( e + T ) ≤ E T , and 6144 m log ( e + m ) < 2 ⋅ 204 2 m log ( e + m ) ≤ m ⋅ 204 2 ⋅ m log ( e + T ) ≤ E T ≤ d 6144m\log(e+m)<2\cdot204^2m\log(e+m)\le m\cdot204^2\cdot m\log(e+T)\le ET\le d 6144 m log ( e + m ) < 2 ⋅ 20 4 2 m log ( e + m ) ≤ m ⋅ 20 4 2 ⋅ m log ( e + T ) ≤ E T ≤ d . Moreover, ( 204 m / T ) L ≤ 2 ρ ε (204m/T)L\le\sqrt{2\rho\varepsilon} ( 204 m / T ) L ≤ 2 ρε , so (77) gives Lip ( ∇ 2 f ) < ρ \operatorname{Lip}(\nabla^2f)<\rho Lip ( ∇ 2 f ) < ρ . Theorem G.8 yields N d e t l i n ≥ ⌊ d / 2 ⌋ m N_{\mathrm{det}}^{\mathrm{lin}}\ge\lfloor d/2\rfloor m N det lin ≥ ⌊ d /2 ⌋ m . Since m / z ≥ 3 / 4 m/z\ge3/4 m / z ≥ 3/4 and ⌊ d / 2 ⌋ / d ≥ 15 / 32 \lfloor d/2\rfloor/d\ge15/32 ⌊ d /2 ⌋ / d ≥ 15/32 for d ≥ 6144 d\ge6144 d ≥ 6144 , we have ⌊ d / 2 ⌋ m ≥ ( 45 / 128 ) d z > d z / 3 = d M / 612 \lfloor d/2\rfloor m\ge(45/128)dz>dz/3=dM/612 ⌊ d /2 ⌋ m ≥ ( 45/128 ) d z > d z /3 = d M /612 . □
Corollary G.10 (The Ω ( d T ) \Omega(dT) Ω ( d T ) threshold on the global classes). On the C ∞ C^\infty C ∞ subclass of Corollary G.9 with ρ ≥ L 2 / ( 2 ε ) \rho\ge L^2/(2\varepsilon) ρ ≥ L 2 / ( 2 ε ) , and hence on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d and P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d , whenever E ≥ T log ( e + T ) E \ge T\log(e+T) E ≥ T log ( e + T ) and E ≤ d E \le d E ≤ d (without requiring E T ≤ d ET \le d E T ≤ d ), both deterministic oracle costs satisfy
N d e t v a l ≥ N d e t l i n ≥ d T 18432 . (79) N_{\mathrm{det}}^{\mathrm{val}} \ge N_{\mathrm{det}}^{\mathrm{lin}} \ge\frac{dT}{18432}. \tag*{(79)} N det val ≥ N det lin ≥ 18432 d T . (79) If in addition T log ( e + T ) ≤ E ≤ d / T T\log(e+T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T , then on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d *,
d T 612 ≤ N d e t ≤ 10 d T , E T 12 ≤ N r a n d c a p ≤ 23 E T , (80) \frac{dT}{612} \le N_{\mathrm{det}} \le10dT,\qquad\frac{ET}{12} \le N_{\mathrm{rand}}^{\mathrm{cap}} \le23ET, \tag*{(80)} 612 d T ≤ N det ≤ 10 d T , 12 E T ≤ N rand cap ≤ 23 E T , (80) and the cost ratio for the same oracle N d e t / N r a n d c a p N_{\mathrm{det}}/N_{\mathrm{rand}}^{\mathrm{cap}} N det / N rand cap lies between d / ( 14076 E ) d/(14076E) d / ( 14076 E ) and 120 d / E 120d/E 120 d / E .
Proof. For (79) , set z = T / 6144 z=T/6144 z = T /6144 and m = ⌊ z ⌋ m=\lfloor z\rfloor m = ⌊ z ⌋ . If z < 3 z<3 z < 3 , Lemma E.2 gives N d e t ≥ d > d z / 3 = d T / 18432 N_{\mathrm{det}}\ge d>dz/3=dT/18432 N det ≥ d > d z /3 = d T /18432 . If z ≥ 3 z\ge3 z ≥ 3 , then m ≥ 3 m\ge3 m ≥ 3 , 204 m ≤ T 204m\le T 204 m ≤ T , 28560 m 2 log ( e + m ) ≤ T 2 log ( e + T ) ≤ E T 28560m^2\log(e+m)\le T^2\log(e+T)\le ET 28560 m 2 log ( e + m ) ≤ T 2 log ( e + T ) ≤ E T , and 6144 m log ( e + m ) ≤ T log ( e + T ) ≤ E ≤ d 6144m\log(e+m)\le T\log(e+T)\le E\le d 6144 m log ( e + m ) ≤ T log ( e + T ) ≤ E ≤ d , so Theorem G.8 gives N d e t ≥ ⌊ d / 2 ⌋ m ≥ d z / 3 = d T / 18432 N_{\mathrm{det}}\ge\lfloor d/2\rfloor m\ge dz/3=dT/18432 N det ≥ ⌊ d /2 ⌋ m ≥ d z /3 = d T /18432 . When T log ( e + T ) ≤ E ≤ d / T T\log(e+T)\le E\le d/T T log ( e + T ) ≤ E ≤ d / T , we also have E T ≤ d ET\le d E T ≤ d and min { T , E T / log ( e + T ) } = T \min\{T,\sqrt{ET/\log(e+T)}\}=T min { T , E T / log ( e + T ) } = T , so Corollary G.9 gives N d e t ≥ d T / 612 N_{\mathrm{det}}\ge dT/612 N det ≥ d T /612 , while coordinate differences, Theorem B.8 , and Corollary F.4 give the remaining bounds in (80) .
Further Results for Randomized Methods
This appendix shows that the hard functions of Theorem E.8 do not extend the deterministic bound Ω ( d T ) \Omega(dT) Ω ( d T ) to randomized methods (Appendix § 16 .1), and combines our bounds with the lower bound of Xia and Yuan [55 ] into a wider window with matching bounds for exact values (Appendix § 16 .2).
The unrestricted hard functions are easy for randomized methods
Theorem E.8 uses the functions F ~ m , U ( x ) = g m ( U ⊤ ρ R ( x ) ) + q ( x ) \widetilde{F}_{m,U}(x)=g_m(U^\top\rho_R(x))+q(x) F m , U ( x ) = g m ( U ⊤ ρ R ( x )) + q ( x ) with R 2 = 5 m R^2=5m R 2 = 5 m , where ρ R ( x ) = x / 1 + ∥ x ∥ 2 / R 2 \rho_R(x)=x/\sqrt{1+\|x\|^2/R^2} ρ R ( x ) = x / 1 + ∥ x ∥ 2 / R 2 and the public potential q q q vanishes on ∥ x ∥ ≤ r 0 = 2 m \|x\|\le r_0=2\sqrt{m} ∥ x ∥ ≤ r 0 = 2 m . A deterministic method must pay order d d d calls for each of the m m m hidden directions. A randomized method can do better on this family, because the minimizer depends only on the sum of the hidden directions, and this sum can be read in batches.
Lemma H.1 (Stages of one parity are read together). Let U = ( u 1 , … , u m ) U=(u_1,\ldots,u_m) U = ( u 1 , … , u m ) have orthonormal columns, and let B ⊂ { 1 , … , m − 1 } B\subset\{1,\ldots,m-1\} B ⊂ { 1 , … , m − 1 } be a nonempty set of indices of the same parity. Suppose z B ∈ R d z_B\in\mathbb{R}^d z B ∈ R d satisfies
⟨ u i , z B ⟩ ≥ 13 8 ( i ∈ B ) , ∣ ⟨ u k , z B ⟩ ∣ ≤ 3 8 ( k ∉ B ) . (81) \langle u_i,z_B\rangle\ge\frac{13}{8}\quad(i\in B),\qquad|\langle u_k,z_B\rangle|\le\frac{3}{8}\quad(k\notin B). \tag*{(81)} ⟨ u i , z B ⟩ ≥ 8 13 ( i ∈ B ) , ∣ ⟨ u k , z B ⟩ ∣ ≤ 8 3 ( k ∈ / B ) . (81) Then for every z z z with ∥ z − z B ∥ ≤ 1 / 16 \|z-z_B\|\le1/16 ∥ z − z B ∥ ≤ 1/16 ,
g m ( U ⊤ z ) = c B − ⟨ u 1 1 { 1 ∉ B } + ∑ i ∈ B u i + 1 , z ⟩ , g_m(U^\top z)=c_B-\left\langle u_1\mathbf{1}_{\{1\notin B\}}+\sum_{i\in B}u_{i+1},z\right\rangle, g m ( U ⊤ z ) = c B − ⟨ u 1 1 { 1 ∈ / B } + i ∈ B ∑ u i + 1 , z ⟩ , where c B = − 9 8 1 { 1 ∉ B } − 9 4 1 { 1 ∈ B } − 9 8 ∣ B ∣ c_B=-\frac{9}{8}\mathbf{1}_{\{1\notin B\}}-\frac{9}{4}\mathbf{1}_{\{1\in B\}}-\frac{9}{8}|B| c B = − 8 9 1 { 1 ∈ / B } − 4 9 1 { 1 ∈ B } − 8 9 ∣ B ∣ . Moreover, at z ⋆ = 5 4 ∑ j = 1 m u j z_\star=\frac{5}{4}\sum_{j=1}^{m}u_j z ⋆ = 4 5 ∑ j = 1 m u j the chain attains its minimum − 9 4 m -\frac{9}{4}m − 4 9 m , and its gradient in z z z vanishes.
Proof. Indices of the same parity are never consecutive, so B B B and { i + 1 : i ∈ B } \{i+1:i\in B\} { i + 1 : i ∈ B } are disjoint. For ∥ z − z B ∥ ≤ 1 / 16 \|z-z_B\|\le1/16 ∥ z − z B ∥ ≤ 1/16 , the coordinates y = U ⊤ z y=U^\top z y = U ⊤ z satisfy y i ≥ 25 / 16 > 5 / 4 y_i\ge25/16>5/4 y i ≥ 25/16 > 5/4 for i ∈ B i\in B i ∈ B and $|y_k|\le7/16<1/2 for k ∉ B k \notin B k ∈ / B . In (37) , the first term is − 9 / 4 -9/4 − 9/4 if 1 ∈ B 1 \in B 1 ∈ B and − ( y 1 + 9 / 8 ) -(y_1+9/8) − ( y 1 + 9/8 ) otherwise. For r ≥ 2 r \ge2 r ≥ 2 , the r r r th term vanishes when r − 1 ∉ B r-1 \notin B r − 1 ∈ / B , because then ψ ( ± y r − 1 ) = 0 \psi(\pm y_{r-1})=0 ψ ( ± y r − 1 ) = 0 . When r − 1 ∈ B r-1 \in B r − 1 ∈ B , we have r ∉ B r \notin B r ∈ / B , so ψ ( y r − 1 ) = 1 \psi(y_{r-1})=1 ψ ( y r − 1 ) = 1 , ψ ( − y r − 1 ) = 0 \psi(-y_{r-1})=0 ψ ( − y r − 1 ) = 0 , and φ ( y r ) = y r + 9 / 8 \varphi(y_r)=y_r+9/8 φ ( y r ) = y r + 9/8 . Summing gives the displayed affine function. At z ∗ z_\ast z ∗ every coordinate equals 5 / 4 5/4 5/4 , where φ = 9 / 4 \varphi=9/4 φ = 9/4 and ψ = 1 \psi=1 ψ = 1 and both have zero derivative, so each of the m m m terms equals its minimum − 9 / 4 -9/4 − 9/4 .
Proposition H.2 (A randomized method on the unrestricted hard functions). Let m ≥ 2 m \ge2 m ≥ 2 and d ≥ 2 + 80 m log ( 24 m 2 ) d \ge2+80m\log(24m^2) d ≥ 2 + 80 m log ( 24 m 2 ) , and consider the scaled functions f U ( x ) = α F ~ m , U ( x / σ ) f_U(x)=\alpha\widetilde F_{m,U}(x/\sigma) f U ( x ) = α F m , U ( x / σ ) of the proof of Theorem E.8 . There is a randomized method using exact values that, for every orthonormal frame U U U , uses at most
2 m + 17 ( d + 1 ) m log ( 24 m 2 ) 2m+17(d+1)\sqrt{m\log(24m^2)} 2 m + 17 ( d + 1 ) m log ( 24 m 2 ) values and, with probability at least 5 / 6 5/6 5/6 , outputs an unqueried exact global minimizer x ^ \widehat{x} x of f U f_U f U , so that ∇ f U ( x ^ ) = 0 \nabla f_U(\widehat{x})=0 ∇ f U ( x ) = 0 . If 500 < T ≤ d / log ( e d ) 500<T\le d/\log(ed) 500 < T ≤ d / log ( e d ) and m m m is the number of stages used in Theorem E.8 , these hypotheses hold and the bound is O ( d T log ( e + T ) ) O(d\sqrt{T}\log(e+T)) O ( d T log ( e + T )) , whereas every deterministic method needs Ω ( d T ) \Omega(dT) Ω ( d T ) values on the same functions.
Algorithm 7 collects the steps of the following proof.
Proof. The constants α , σ > 0 \alpha,\sigma>0 α , σ > 0 are public, so we work with F ~ m , U \widetilde F_{m,U} F m , U . At a query point x x x with ρ R ( x ) = z \rho_R(x)=z ρ R ( x ) = z , subtracting the public value q ( x ) q(x) q ( x ) returns g m ( U ⊤ z ) g_m(U^\top z) g m ( U ⊤ z ) , and ρ R − 1 ( z ) = z / 1 − ∥ z ∥ 2 / R 2 \rho_R^{-1}(z)=z/\sqrt{1-\lVert z\rVert^2/R^2} ρ R − 1 ( z ) = z / 1 − ∥ z ∥ 2 / R 2 for ∥ z ∥ < R \lVert z\rVert<R ∥ z ∥ < R . Put δ = 1 / 16 \delta=1/16 δ = 1/16 ,
R e f f = 3 4 R , K = ⌊ m 40 log ( 24 m 2 ) ⌋ , b = ⌈ 48 d K log ( 24 m 2 ) R e f f 2 ⌉ . R_{\mathrm{eff}}=\frac{3}{4}R,\qquad K=\left\lfloor\sqrt{\frac{m}{40\log(24m^2)}}\right\rfloor,\qquad b=\left\lceil\frac{48dK\log(24m^2)}{R_{\mathrm{eff}}^2}\right\rceil. R eff = 4 3 R , K = ⌊ 40 log ( 24 m 2 ) m ⌋ , b = ⌈ R eff 2 48 d K log ( 24 m 2 ) ⌉ .
The assumption on d d d is equivalent to R e f f ≤ 3 16 ( d − 2 ) / log ( 24 m 2 ) R_{\mathrm{eff}}\le\frac{3}{16}\sqrt{(d-2)/\log(24m^2)} R eff ≤ 16 3 ( d − 2 ) / log ( 24 m 2 ) . All points z z z used below satisfy ∥ z ∥ ≤ R e f f + δ < R \lVert z\rVert\le R_{\mathrm{eff}}+\delta<R ∥ z ∥ ≤ R eff + δ < R . If B B B and z B z_B z B satisfy the hypotheses of Lemma H.1 , the d + 1 d+1 d + 1 values at z B z_B z B and z B + δ e a z_B+\delta e_a z B + δ e a , a = 1 , … , d a=1,\ldots,d a = 1 , … , d , give the difference quotients δ − 1 [ g m ( U ⊤ z B ) − g m ( U ⊤ ( z B + δ e a ) ) ] \delta^{-1}[g_m(U^\top z_B)-g_m(U^\top(z_B+\delta e_a))] δ − 1 [ g m ( U ⊤ z B ) − g m ( U ⊤ ( z B + δ e a ))] , which are exactly the coordinates of u 1 1 { 1 ∉ B } + ∑ i ∈ B u i + 1 u_1\mathbf{1}_{\{1\notin B\}}+\sum_{i\in B}u_{i+1} u 1 1 { 1 ∈ / B } + ∑ i ∈ B u i + 1 .
Step 0. Near z = 0 z=0 z = 0 every ∣ y i ∣ ≤ δ < 1 / 2 |y_i|\le\delta<1/2 ∣ y i ∣ ≤ δ < 1/2 , so the chain equals − ( y 1 + 9 / 8 ) -(y_1+9/8) − ( y 1 + 9/8 ) there. The d + 1 d+1 d + 1 values at z = 0 z=0 z = 0 and z = δ e a z=\delta e_a z = δ e a , a = 1 , … , d a=1,\ldots,d a = 1 , … , d , give u 1 u_1 u 1 exactly.
Small m m m . If K = 0 K=0 K = 0 , then m < 40 log ( 24 m 2 ) m<40\log(24m^2) m < 40 log ( 24 m 2 ) . For j = 2 , … , m j=2,\ldots,m j = 2 , … , m in turn, the point 2 u j − 1 2u_{j-1} 2 u j − 1 satisfies (81) with B = { j − 1 } B=\{j-1\} B = { j − 1 } , so its d + 1 d+1 d + 1 values give u 1 1 { j ≠ 2 } + u j u_1\mathbf{1}_{\{j\ne2\}}+u_j u 1 1 { j = 2 } + u j , and hence u j u_j u j . This deterministic method finds ∑ j u j \sum_j u_j ∑ j u j exactly with m ( d + 1 ) < ( d + 1 ) 40 m log ( 24 m 2 ) m(d+1)<(d+1)\sqrt{40m\log(24m^2)} m ( d + 1 ) < ( d + 1 ) 40 m log ( 24 m 2 ) values and outputs x ^ \widehat{x} x as in Phase 2 below. Assume from now on that K ≥ 1 K\ge1 K ≥ 1 , so that K 2 ≤ m / ( 40 log ( 24 m 2 ) ) K^2\le m/(40\log(24m^2)) K 2 ≤ m / ( 40 log ( 24 m 2 )) and m ≥ 645 m\ge645 m ≥ 645 . Since R e f f 2 = 45 16 m R_{\mathrm{eff}}^2=\frac{45}{16}m R eff 2 = 16 45 m , we have K b ≤ 32 75 d + K ≤ ( d − 1 ) / 2 Kb\le\frac{32}{75}d+K\le(d-1)/2 K b ≤ 75 32 d + K ≤ ( d − 1 ) /2 .
Phase 1. Split the odd and the even indices of { 1 , … , m − 1 } \{1,\ldots,m-1\} { 1 , … , m − 1 } into batches of at most K K K indices each; there are at most ⌈ ( m − 1 ) / ( 2 K ) ⌉ + ⌈ ( m − 2 ) / ( 2 K ) ⌉ ≤ min { m − 1 , m / K + 2 } \lceil(m-1)/(2K)\rceil+\lceil(m-2)/(2K)\rceil\le\min\{m-1,m/K+2\} ⌈( m − 1 ) / ( 2 K )⌉ + ⌈( m − 2 ) / ( 2 K )⌉ ≤ min { m − 1 , m / K + 2 } batches. Within each batch, take mutually orthogonal random orthonormal bases of S j S_j S j of u 1 u_1 u 1 , one for each index i ≠ 1 i\ne1 i = 1 , as consecutive blocks of a uniformly random orthonormal basis of u 1 ⊥ u_1^\perp u 1 ⊥ , independently across batches; this is possible because K b ≤ ( d − 1 ) / 2 Kb\le(d-1)/2 K b ≤ ( d − 1 ) /2 . Put p ^ 1 = u 1 \widehat{p}_1=u_1 p 1 = u 1 . For j = 2 , … , m − 1 j=2,\ldots,m-1 j = 2 , … , m − 1 in turn, take z ( j ) = R e f f p ^ j − 1 z^{(j)}=R_{\mathrm{eff}}\widehat{p}_{j-1} z ( j ) = R eff p j − 1 , evaluate at z ( j ) z^{(j)} z ( j ) and at z ( j ) + δ v z^{(j)}+\delta v z ( j ) + δ v for an orthonormal basis v v v of S j S_j S j , put
g j = ∑ v δ − 1 [ g m ( U ⊤ z ( j ) ) − g m ( U ⊤ ( z ( j ) + δ v ) ) ] v ∈ S j , g_j=\sum_{v}\delta^{-1}\left[g_m(U^\top z^{(j)})-g_m\left(U^\top(z^{(j)}+\delta v)\right)\right]v\in S_j, g j = v ∑ δ − 1 [ g m ( U ⊤ z ( j ) ) − g m ( U ⊤ ( z ( j ) + δ v ) ) ] v ∈ S j ,
and set p ^ j = g j / ∥ g j ∥ \widehat{p}_j=g_j/\lVert g_j\rVert p j = g j / ∥ g j ∥ , or a fixed unit vector of S j S_j S j if g j = 0 g_j=0 g j = 0 .
Probability. Fix U U U , put η = 4 log ( 24 m 2 ) / ( d − 2 ) \eta=\sqrt{4\log(24m^2)/(d-2)} η = 4 log ( 24 m 2 ) / ( d − 2 ) , so that R e f f η ≤ 3 / 8 R_{\mathrm{eff}}\eta\le3/8 R eff η ≤ 3/8 , and for i ≥ 2 i\ge2 i ≥ 2 let p ^ i t r u e = P S i u i / ∥ P S i u i ∥ \widehat{p}_i^{\mathrm{true}}=P_{S_i}u_i/\lVert P_{S_i}u_i\rVert p i true = P S i u i / ∥ P S i u i ∥ . Consider three kinds of events.
Algorithm 7 Randomized method on the hard functions of Proposition H.2 . As in the proof, a query at a point z z z with ∥ z ∥ < R \lVert z\rVert< R ∥ z ∥ < R is made at ρ R − 1 ( z ) \rho_R^{-1}(z) ρ R − 1 ( z ) and returns g ( z ) = g m ( U ⊤ z ) g(z)=g_m(U^\top z) g ( z ) = g m ( U ⊤ z ) after the public value q q q is subtracted. For a point z z z and an orthonormal list W W W , Diff ( z , W ) = ∑ v ∈ W δ − 1 [ g ( z ) − g ( z + δ v ) ] v \operatorname{Diff}(z,W)=\sum_{v\in W}\delta^{-1}[g(z)-g(z+\delta v)]v Diff ( z , W ) = ∑ v ∈ W δ − 1 [ g ( z ) − g ( z + δ v )] v uses 1 + ∣ W ∣ 1+|W| 1 + ∣ W ∣ values; e 1 , … , e d e_1,\ldots,e_d e 1 , … , e d are the coordinate vectors.
Require: m m m , d d d , the public α \alpha α , σ \sigma σ , R R R , q q q , ρ R \rho_R ρ R , and exact values of f U f_U f U
1: δ ← 1 / 16 \delta\leftarrow1/16 δ ← 1/16 , R e f f ← 3 4 R R_{\mathrm{eff}}\leftarrow\frac{3}{4}R R eff ← 4 3 R , K ← ⌊ m / ( 40 log ( 24 m 2 ) ) ⌋ K\leftarrow\left\lfloor\sqrt{m/(40\log(24m^2))}\right\rfloor K ← ⌊ m / ( 40 log ( 24 m 2 )) ⌋ , b ← ⌈ 48 d K log ( 24 m 2 ) / R e f f 2 ⌉ b\leftarrow\left\lceil48dK\log(24m^2)/R_{\mathrm{eff}}^2\right\rceil b ← ⌈ 48 d K log ( 24 m 2 ) / R eff 2 ⌉
2: u 1 ← Diff ( 0 , ( e 1 , … , e d ) ) u_1\leftarrow\operatorname{Diff}(0,(e_1,\ldots,e_d)) u 1 ← Diff ( 0 , ( e 1 , … , e d )) \hfill Step 0
3: if K = 0 K=0 K = 0 then \hfill small m m m
4: \hspace{2em}for j = 2 , … , m j=2,\ldots,m j = 2 , … , m do
5: \hspace{4em}u j ← Diff ( 2 u j − 1 , ( e 1 , … , e d ) ) − u 1 1 { j ≠ 2 } u_j\leftarrow\operatorname{Diff}(2u_{j-1},(e_1,\ldots,e_d))-u_1\mathbf{1}_{\{j\ne2\}} u j ← Diff ( 2 u j − 1 , ( e 1 , … , e d )) − u 1 1 { j = 2 }
6: \hspace{2em}end for
7: \hspace{2em}s ← ∑ j = 1 m u j s\leftarrow\sum_{j=1}^{m}u_j s ← ∑ j = 1 m u j
8: else \hfill Phase 1
9: \hspace{2em}split the odd and the even indices of { 1 , … , m − 1 } \{1,\ldots,m-1\} { 1 , … , m − 1 } into batches of at most K K K indices
10: \hspace{2em}for each batch, independently, draw a uniformly random orthonormal basis of u 1 ⊥ u_1^\perp u 1 ⊥ and give each index i ≠ 1 i\ne1 i = 1 of the batch its own block S i S_i S i of b b b consecutive basis vectors
11: \hspace{2em}p ^ 1 ← u 1 \widehat p_1\leftarrow u_1 p 1 ← u 1
12: \hspace{2em}for j = 2 , … , m − 1 j=2,\ldots,m-1 j = 2 , … , m − 1 do
13: \hspace{4em}g j ← Diff ( R e f f p ^ j − 1 , basis of S j ) g_j\leftarrow\operatorname{Diff}(R_{\mathrm{eff}}\widehat p_{j-1},\text{ basis of }S_j) g j ← Diff ( R eff p j − 1 , basis of S j )
14: \hspace{4em}p ^ j ← g j / ∥ g j ∥ \widehat p_j\leftarrow g_j/\lVert g_j\rVert p j ← g j / ∥ g j ∥ , or a fixed unit vector of S j S_j S j if g j = 0 g_j=0 g j = 0
15: \hspace{2em}end for
16: \hspace{2em}s ← u 1 s\leftarrow u_1 s ← u 1 \hfill Phase 2
17: \hspace{2em}for each batch B B B do
18: \hspace{4em}z B ← ( R e f f / ∣ B ∣ ) ∑ i ∈ B p ^ i z_B\leftarrow(R_{\mathrm{eff}}/\sqrt{|B|})\sum_{i\in B}\widehat p_i z B ← ( R eff / ∣ B ∣ ) ∑ i ∈ B p i
19: \hspace{4em}s ← s + Diff ( z B , ( e 1 , … , e d ) ) − u 1 1 { 1 ∉ B } s\leftarrow s+\operatorname{Diff}(z_B,(e_1,\ldots,e_d))-u_1\mathbf{1}_{\{1\notin B\}} s ← s + Diff ( z B , ( e 1 , … , e d )) − u 1 1 { 1 ∈ / B }
20: \hspace{2em}end for
21: end if
22: z ∗ ← 5 4 s z_\ast\leftarrow\frac{5}{4}s z ∗ ← 4 5 s
23: return x ^ = σ ρ R − 1 ( z ∗ ) \widehat x=\sigma\rho_R^{-1}(z_\ast) x = σ ρ R − 1 ( z ∗ ) if ∥ z ∗ ∥ < R \lVert z_\ast\rVert<R ∥ z ∗ ∥ < R , and x ^ = 0 \widehat x=0 x = 0 otherwise
(i) Signal , for 2 ≤ i ≤ m − 1 2\le i\le m-1 2 ≤ i ≤ m − 1 : ∥ P S i u i ∥ 2 ≥ b / ( 3 d ) \lVert P_{S_i}u_i\rVert^2\ge b/(3d) ∥ P S i u i ∥ 2 ≥ b / ( 3 d ) . Each S i S_i S i is uniform among the b b b -dimensional subspaces of the ( d − 1 ) (d-1) ( d − 1 ) -dimensional space u 1 ⊥ u_1^\perp u 1 ⊥ , which contains u i u_i u i , so ∥ P S i u i ∥ 2 \lVert P_{S_i}u_i\rVert^2 ∥ P S i u i ∥ 2 has the law of X / ( X + Y ) X/(X+Y) X / ( X + Y ) for independent X ∼ χ b 2 X\sim\chi_b^2 X ∼ χ b 2 and Y ∼ χ d − 1 − b 2 Y\sim\chi_{d-1-b}^2 Y ∼ χ d − 1 − b 2 . The event holds if X ≥ b / 2 X\ge b/2 X ≥ b /2 and X + Y ≤ 3 2 ( d − 1 ) X+Y\le\frac{3}{2}(d-1) X + Y ≤ 2 3 ( d − 1 ) . By Chernoff bounds for χ 2 \chi^2 χ 2 variables, these fail with probability at most e − b / 11 + e − ( d − 1 ) / 22 ≤ 1 / ( 12 m ) e^{-b/11}+e^{-(d-1)/22}\le1/(12m) e − b /11 + e − ( d − 1 ) /22 ≤ 1/ ( 12 m ) , since b ≥ 11 log ( 24 m ) b\ge11\log(24m) b ≥ 11 log ( 24 m ) and d − 1 ≥ 22 log ( 24 m ) d-1\ge22\log(24m) d − 1 ≥ 22 log ( 24 m ) .
(ii) Triggers , for 2 ≤ i ≤ m − 2 2\le i\le m-2 2 ≤ i ≤ m − 2 and k ∉ { 1 , i } k\notin\{1,i\} k ∈ / { 1 , i } : ∣ ⟨ u k , p ^ i t r u e ⟩ ∣ ≤ η |\langle u_k,\widehat p_i^{\mathrm{true}}\rangle|\le\eta ∣ ⟨ u k , p i true ⟩ ∣ ≤ η . The law of S i S_i S i is invariant under the rotations of u 1 ⊥ u_1^\perp u 1 ⊥ that fix u i u_i u i , so the component of p ^ i t r u e \widehat p_i^{\mathrm{true}} p i true orthogonal to u i u_i u i has a uniform direction in the ( d − 2 ) (d-2) ( d − 2 ) -dimensional space { u 1 , u i } ⊥ \{u_1,u_i\}^{\perp} { u 1 , u i } ⊥ , which contains u k u_k u k . The tail bound in the proof of Lemma C.3 shows that the event fails with probability at most 2 e − ( d − 2 ) η 2 / 2 = 1 / ( 288 m 4 ) 2e^{-(d-2)\eta^2/2}=1/(288m^4) 2 e − ( d − 2 ) η 2 /2 = 1/ ( 288 m 4 ) .
(iii) Batches , for each batch B B B and each k ≥ 2 k\ge2 k ≥ 2 : ∣ ⟨ u k , w B , k ⟩ ∣ ≤ η ∣ B ∣ |\langle u_k,w_{B,k}\rangle|\le\eta\sqrt{|B|} ∣ ⟨ u k , w B , k ⟩ ∣ ≤ η ∣ B ∣ , where w B , k = ∑ i ∈ B ∖ { 1 , k } p ^ i t r u e w_{B,k}=\sum_{i\in B\setminus\{1,k\}}\widehat p_i^{\mathrm{true}} w B , k = ∑ i ∈ B ∖ { 1 , k } p i true . The joint law of the subspaces of B B B is invariant under the rotations of u 1 ⊥ u_1^\perp u 1 ⊥ that fix every u i u_i u i with i ∈ B ∖ { 1 , k } i\in B\setminus\{1,k\} i ∈ B ∖ { 1 , k } , and these rotations act on each such p ^ i t r u e \widehat p_i^{\mathrm{true}} p i true by rotation. Hence the component of w B , k w_{B,k} w B , k orthogonal to A = span { u 1 , u i : i ∈ B ∖ { 1 , k } } A=\operatorname{span}\{u_1,u_i:i\in B\setminus\{1,k\}\} A = span { u 1 , u i : i ∈ B ∖ { 1 , k }} has a uniform direction in A ⊥ A^\perp A ⊥ , which contains u k u_k u k and has dimension at least d − 1 − K ≥ ( d − 2 ) / 2 d-1-K\ge(d-2)/2 d − 1 − K ≥ ( d − 2 ) /2 . As ∥ w B , k ∥ ≤ ∣ B ∣ \lVert w_{B,k}\rVert\le\sqrt{|B|} ∥ w B , k ∥ ≤ ∣ B ∣ , the event fails with probability at most 2 e − ( d − 2 ) η 2 / 4 = 1 / ( 12 m 2 ) 2e^{-(d-2)\eta^2/4}=1/(12m^2) 2 e − ( d − 2 ) η 2 /4 = 1/ ( 12 m 2 ) .
There are m − 2 m-2 m − 2 signal events, fewer than m 2 m^2 m 2 trigger events, and at most m ( m − 1 ) m(m-1) m ( m − 1 ) batch events, so all of them hold with probability at least 1 − 2 m − 3 12 m − 288 m 2 m 2 > 5 6 1-\frac{2m^{-3}}{12m}-\frac{288m^2}{m^2}>\frac{5}{6} 1 − 12 m 2 m − 3 − m 2 288 m 2 > 6 5 .
On this event, p ^ j = p ^ j t r u e \hat{p}_j=\hat{p}^{\mathrm{true}}_j p ^ j = p ^ j true for 2 ≤ j ≤ m − 1 2\leq j\leq m-1 2 ≤ j ≤ m − 1 , by induction on j j j . Indeed, z ( j ) z^{(j)} z ( j ) satisfies (81) with B = { j − 1 } B=\{j-1\} B = { j − 1 } : its inner product with u j − 1 u_{j-1} u j − 1 is R e f f ≥ 2 R_{\mathrm{eff}}\geq2 R eff ≥ 2 if j = 2 j=2 j = 2 and otherwise R e f f ∥ P S j − 1 u j − 1 ∥ ≥ R e f f b / ( 3 d ) ≥ 4 K log ( 24 m 2 ) ≥ 2 R_{\mathrm{eff}}\lVert P_{S_{j-1}}u_{j-1}\rVert\geq R_{\mathrm{eff}}\sqrt{b/(3d)}\geq4\sqrt{K}\log(24m^2)\geq2 R eff ∥ P S j − 1 u j − 1 ∥ ≥ R eff b / ( 3 d ) ≥ 4 K log ( 24 m 2 ) ≥ 2 , and its other inner products vanish or, by the trigger events, are at most R e f f η ≤ 3 / 8 R_{\mathrm{eff}}\eta\leq3/8 R eff η ≤ 3/8 in absolute value. By Lemma H.1 the chain is affine near z ( j ) z^{(j)} z ( j ) with gradient − u 2 -u_2 − u 2 if j = 2 j=2 j = 2 and − ( u 1 + u j ) -(u_1+u_j) − ( u 1 + u j ) otherwise. Every v ∈ S j v\in S_j v ∈ S j is orthogonal to u 1 u_1 u 1 , so the difference quotients are exact, g j = P S j u j ≠ 0 g_j=P_{S_j}u_j\ne0 g j = P S j u j = 0 , and p ^ j = p ^ j t r u e \hat{p}_j=\hat{p}^{\mathrm{true}}_j p ^ j = p ^ j true . For each batch B B B put
z B = R e f f ∣ B ∣ ∑ i ∈ B p ^ i . z_B=\frac{R_{\mathrm{eff}}}{\sqrt{|B|}}\sum_{i\in B}\hat{p}_i. z B = ∣ B ∣ R eff i ∈ B ∑ p ^ i .
which has norm R e f f R_{\mathrm{eff}} R eff , because the p ^ i \hat{p}_i p ^ i of one batch are orthonormal. For i ∈ B i\in B i ∈ B with i ≥ 2 i\geq2 i ≥ 2 , the signal and batch events give ⟨ u i , z B ⟩ ≥ R e f f b / ( 3 d K ) − R e f f η ≥ 2 − 8 3 = 13 8 \langle u_i,z_B\rangle\geq R_{\mathrm{eff}}\sqrt{b/(3dK)}-R_{\mathrm{eff}}\eta\geq2-\frac{8}{3}=\frac{13}{8} ⟨ u i , z B ⟩ ≥ R eff b / ( 3 d K ) − R eff η ≥ 2 − 3 8 = 8 13 ; if 1 ∈ B 1\in B 1 ∈ B , then ⟨ u 1 , z B ⟩ = R e f f / ∣ B ∣ ≥ 2 \langle u_1,z_B\rangle=R_{\mathrm{eff}}/\sqrt{|B|}\geq2 ⟨ u 1 , z B ⟩ = R eff / ∣ B ∣ ≥ 2 ; and for k ∉ B k\notin B k ∈ / B , ⟨ u k , z B ⟩ \langle u_k,z_B\rangle ⟨ u k , z B ⟩ vanishes if k = 1 k=1 k = 1 and otherwise has absolute value at most R e f f η ≤ 3 8 R_{\mathrm{eff}}\eta\leq\frac{3}{8} R eff η ≤ 8 3 by the batch events.
Phase 2. For each batch B B B , the d + 1 d+1 d + 1 values at z B z_B z B and z B + δ e a z_B+\delta e_a z B + δ e a give u 1 1 { 1 ∉ B } + ∑ i ∈ B u i + 1 u_1\mathbf{1}_{\{1\notin B\}}+\sum_{i\in B}u_{i+1} u 1 1 { 1 ∈ / B } + ∑ i ∈ B u i + 1 , and hence ∑ i ∈ B u i + 1 \sum_{i\in B}u_{i+1} ∑ i ∈ B u i + 1 . The batches partition { 1 , … , m − 1 } \{1,\ldots,m-1\} { 1 , … , m − 1 } , so u 1 u_1 u 1 plus the sum of all batch vectors equals ∑ j = 1 m u j \sum_{j=1}^m u_j ∑ j = 1 m u j exactly. Let z ∗ z_\ast z ∗ be 5 4 \frac{5}{4} 4 5 times this computed sum, and output x ^ = σ ρ R − 1 ( z ∗ ) \hat{x}=\sigma\rho_R^{-1}(z_\ast) x ^ = σ ρ R − 1 ( z ∗ ) , or x ^ = 0 \hat{x}=0 x ^ = 0 if ∥ z ∗ ∥ ≥ R \lVert z_\ast\rVert\geq R ∥ z ∗ ∥ ≥ R . On the event above, z ∗ = 5 4 ∑ j u j z_\ast=\frac{5}{4}\sum_j u_j z ∗ = 4 5 ∑ j u j , ∥ z ∗ ∥ 2 = 25 16 m \lVert z_\ast\rVert^2=\frac{25}{16}m ∥ z ∗ ∥ 2 = 16 25 m , and
∥ ρ R − 1 ( z ∗ ) ∥ 2 = 25 m / 16 1 − 5 / 16 = 25 11 m < 4 m = r 0 2 , \left\|\rho_R^{-1}(z_\ast)\right\|^2=\frac{25m/16}{1-5/16}=\frac{25}{11}m<4m=r_0^2, ρ R − 1 ( z ∗ ) 2 = 1 − 5/16 25 m /16 = 11 25 m < 4 m = r 0 2 ,
so q q q and ∇ q \nabla q ∇ q vanish there, and Lemma H.1 shows that x ^ \hat{x} x ^ is a global minimizer with zero gradient.
Cost. The method uses at most d + 1 + ( m − 2 ) ( b + 1 ) + ( d + 1 ) ( m / K + 2 ) d+1+(m-2)(b+1)+(d+1)(m/K+2) d + 1 + ( m − 2 ) ( b + 1 ) + ( d + 1 ) ( m / K + 2 ) values. Here ( m − 2 ) ( b + 1 ) ≤ 256 15 d K log ( 24 m 2 ) + 2 m ≤ 256 15 d m log ( 24 m 2 ) + 2 m (m-2)(b+1)\leq\frac{256}{15}dK\log(24m^2)+2m\leq\frac{256}{15}d\sqrt{m}\log(24m^2)+2m ( m − 2 ) ( b + 1 ) ≤ 15 256 d K log ( 24 m 2 ) + 2 m ≤ 15 256 d m log ( 24 m 2 ) + 2 m and m / K ≤ 2 40 m log ( 24 m 2 ) m/K\leq2\sqrt{40m\log(24m^2)} m / K ≤ 2 40 m log ( 24 m 2 ) , which give the stated bound since m log ( 24 m 2 ) ≥ 100 \sqrt{m\log(24m^2)}\geq100 m log ( 24 m 2 ) ≥ 100 .
Range. If 500 < T ≤ d / log ( e d ) 500<T\leq d/\log(ed) 500 < T ≤ d / log ( e d ) , then Theorem E.8 uses m = ⌊ T / 275 + 1 / 2 ⌋ ≥ 2 m=\lfloor T/275+1/2\rfloor\geq2 m = ⌊ T /275 + 1/2 ⌋ ≥ 2 , so m ≤ 4 3 ( m − 1 2 ) ≤ d / ( 206 log ( e d ) ) m\leq\frac{4}{3}(m-\frac{1}{2})\leq d/(206\log(ed)) m ≤ 3 4 ( m − 2 1 ) ≤ d / ( 206 log ( e d )) . Hence 24 m 2 < d 2 24m^2<d^2 24 m 2 < d 2 and 80 m log ( 24 m 2 ) < 160 206 d ≤ d − 2 80m\log(24m^2)<\frac{160}{206}d\leq d-2 80 m log ( 24 m 2 ) < 206 160 d ≤ d − 2 , because d > 4700 d>4700 d > 4700 . Finally 2 m ≤ d 2m\leq d 2 m ≤ d and m log ( 24 m 2 ) = O ( T log ( e + T ) ) \sqrt{m}\log(24m^2)=O(\sqrt{T}\log(e+T)) m log ( 24 m 2 ) = O ( T log ( e + T )) , and Theorem E.8 gives the deterministic bound d T / 500 dT/500 d T /500 .
The hard functions of Theorem E.8, on which deterministic methods pay Ω ( d T ) \Omega(dT) Ω ( d T ) , are solved with O ( d T log ( e + T ) ) O(d\sqrt{T}\log(e+T)) O ( d T log ( e + T )) values by the proposition, so they cannot give a randomized lower bound of that order. For exact values, Xia and Yuan [55 ] prove a randomized lower bound of order d T dT d T with a different construction when T ≥ c T\geq c T ≥ c and d ≥ C T log ( 2 + T ) d\geq CT\log(2+T) d ≥ C T log ( 2 + T ) for absolute constants c , C > 0 c,C>0 c , C > 0 ; whether such a bound also holds under gradient projections is open.
Exact values: a wider window with matching bounds
For exact values, the concurrent and independent lower bound of Xia and Yuan [55 ] extends the randomized bound Ω ( E T ) \Omega(ET) Ω ( E T ) beyond E T ≤ d ET\leq d E T ≤ d . Combined with our deterministic bounds, it removes the condition E ≤ d / T E\leq d/T E ≤ d / T from Corollary 3.3 when T T T is at most of order d / log ( e d ) d/\log(ed) d / log ( e d ) . Their bound concerns exact values only, so the rank-priced oracle is not covered, and the constants are no longer explicit.
Corollary H.3 (A wider window for exact values). There are absolute constants c 1 , c 2 > 0 c_1,c_2 > 0 c 1 , c 2 > 0 such that, for exact values on every class between F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d and B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d , if E T ≤ c 1 d 2 / log ( e d ) ET \le c_1d^2/\log(ed) E T ≤ c 1 d 2 / log ( e d ) , then N r a n d ≥ c 2 E T N_{\mathrm{rand}} \ge c_2ET N rand ≥ c 2 E T . Consequently, if T log ( e + T ) ≤ E ≤ d T\log(e+T) \le E \le d T log ( e + T ) ≤ E ≤ d and T ≤ c 1 d / log ( e d ) T \le c_1d/\log(ed) T ≤ c 1 d / log ( e d ) , then
d T 18432 ≤ N d e t ≤ 10 d T , c 2 E T ≤ N r a n d ≤ 23 E T . \frac{dT}{18432} \le N_{\mathrm{det}} \le10dT,\qquad c_2ET \le N_{\mathrm{rand}} \le23ET. 18432 d T ≤ N det ≤ 10 d T , c 2 E T ≤ N rand ≤ 23 E T . Proof. The deterministic lower bound is Corollary G.10 , which requires only T log ( e + T ) ≤ E ≤ d T\log(e+T) \le E \le d T log ( e + T ) ≤ E ≤ d . The two upper bounds and the randomized lower bound min { d , E T } / 12 \min\{d,ET\}/12 min { d , E T } /12 are in Theorem A.1 . Lower bounds proved on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d and upper bounds proved on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d hold on every class between them. It remains to bound N r a n d N_{\mathrm{rand}} N rand from below when E T > d ET > d E T > d .
Put ℓ = L E / d ≤ L \ell= LE/d \le L ℓ = L E / d ≤ L and τ = ℓ Δ / ε 2 = E T / d > 1 \tau= \ell\Delta/\varepsilon^2 = ET/d > 1 τ = ℓ Δ/ ε 2 = E T / d > 1 . Every globally ℓ \ell ℓ -smooth function with f ( 0 ) − inf f ≤ Δ f(0)-\inf f \le\Delta f ( 0 ) − inf f ≤ Δ lies in F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , with the upper model ℓ I d \ell I_d ℓ I d , whose norm is ℓ ≤ L \ell\le L ℓ ≤ L and whose trace is L E LE L E . Theorem 2.3 of Xia and Yuan [55 ] uses the same starting point 0 0 0 , the same gap f ( 0 ) − inf f f(0)-\inf f f ( 0 ) − inf f , a cap on the number of exact values, and success probability at least 2 / 3 2/3 2/3 on every function. We apply it with smoothness ℓ \ell ℓ , gap Δ \Delta Δ , and a Lipschitz constant L 0 ≥ ℓ Δ L_0 \ge\sqrt{\ell\Delta} L 0 ≥ ℓ Δ , so that min { ℓ Δ , L 0 2 } = ℓ Δ \min\{\ell\Delta,L_0^2\}=\ell\Delta min { ℓ Δ , L 0 2 } = ℓ Δ . It provides absolute constants c , C , c 0 > 0 c,C,c_0 > 0 c , C , c 0 > 0 such that, if τ ≥ 1 / c 0 \tau\ge1/c_0 τ ≥ 1/ c 0 and d ≥ C τ log ( 2 + τ ) d \ge C\tau\log(2+\tau) d ≥ C τ log ( 2 + τ ) , every randomized algorithm needs at least c d τ = c E T cd\tau=cET c d τ = c E T exact values on its hard functions; these are globally ℓ \ell ℓ -smooth with gap at most Δ \Delta Δ , and hence lie in F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d . If 1 < τ < 1 / c 0 1 < \tau< 1/c_0 1 < τ < 1/ c 0 , the bound min { d , E T } / 12 \min\{d,ET\}/12 min { d , E T } /12 gives N r a n d ≥ d / 12 > c 0 E T / 12 N_{\mathrm{rand}} \ge d/12 > c_0ET/12 N rand ≥ d /12 > c 0 E T /12 . Finally, let c 1 = min { 1 , 1 / ( 2 C ) } c_1=\min\{1,1/(2C)\} c 1 = min { 1 , 1/ ( 2 C )} . Then τ = E T / d ≤ c 1 d / log ( e d ) ≤ d \tau=ET/d \le c_1d/\log(ed) \le d τ = E T / d ≤ c 1 d / log ( e d ) ≤ d , and 2 + d ≤ ( e d ) 2 2+d \le(ed)^2 2 + d ≤ ( e d ) 2 gives τ log ( 2 + τ ) ≤ c 1 d log ( 2 + d ) / log ( e d ) ≤ 2 c 1 d ≤ d / C \tau\log(2+\tau) \le c_1d\log(2+d)/\log(ed) \le2c_1d \le d/C τ log ( 2 + τ ) ≤ c 1 d log ( 2 + d ) / log ( e d ) ≤ 2 c 1 d ≤ d / C . Hence the corollary holds with c 2 = min { 1 / 12 , c 0 / 12 , c } c_2=\min\{1/12,c_0/12,c\} c 2 = min { 1/12 , c 0 /12 , c } , and the cost ratio N d e t / N r a n d N_{\mathrm{det}}/N_{\mathrm{rand}} N det / N rand is of order d / E d/E d / E .
Experimental Protocol
These experiments illustrate the mechanisms behind the main theorems and the plausibility of the curvature assumption. All synthetic runs use L = Δ = 1 L=\Delta=1 L = Δ = 1 , x 0 = 0 x_0=0 x 0 = 0 , and ε = T − 1 / 2 \varepsilon=T^{-1/2} ε = T − 1/2 , and they charge every function value, including difference probes, rejected proposals, line search centers, and final checks. Three metrics are reported separately: the offline first-hit cost, that is, the number of calls before the first retained iterate with ∥ ∇ f ∥ ≤ ε \lVert\nabla f\rVert\le\varepsilon ∥ ∇ f ∥ ≤ ε , computed by an observer that does not affect the run; whether the output is ε \varepsilon ε -stationary; and the total call count.
Randomized versus deterministic. The first family is the rank-one quadratic f u ( x ) = 1 2 ( ⟨ u , x ⟩ − 2 ) 2 f_u(x)=\frac{1}{2}(\langle u,x\rangle-\sqrt{2})^2 f u ( x ) = 2 1 (⟨ u , x ⟩ − 2 ) 2 , with E = 1 E=1 E = 1 . The second uses ϕ ( t ) = 0.1 t 2 + 0.8 ( 1 − cos t ) \phi(t)=0.1t^2+0.8(1-\cos t) ϕ ( t ) = 0.1 t 2 + 0.8 ( 1 − cos t ) and
f ( x ) = a 2 ∑ i = 1 d h i ϕ ( ( Q T x ) i / a + θ i ) , θ = ( 2 , 0 , … , 0 ) , a = 1 ϕ ( 2 ) . f(x)=a^2\sum_{i=1}^{d}h_i\phi((Q^\mathsf{T}x)_i/a+\theta_i),\qquad\theta=(2,0,\ldots,0),\qquad a=\frac{1}{\sqrt{\phi(2)}}. f ( x ) = a 2 i = 1 ∑ d h i ϕ (( Q T x ) i / a + θ i ) , θ = ( 2 , 0 , … , 0 ) , a = ϕ ( 2 ) 1 .
with max i h i = 1 \max_i h_i=1 max i h i = 1 and ∑ i h i = E \sum_i h_i=E ∑ i h i = E . Since ϕ ′ ′ ∈ [ − 0.6 , 1 ] \phi''\in[-0.6,1] ϕ ′′ ∈ [ − 0.6 , 1 ] , one has ∇ 2 f ⪯ Q diag ( h ) Q T \nabla^2f\preceq Q\operatorname{diag}(h)Q^\mathsf{T} ∇ 2 f ⪯ Q diag ( h ) Q T everywhere, so f f f lies in the fixed-max-trace class; it is nonconvex and coercive, and f ( 0 ) − inf f = 1 f(0)-\inf f=1 f ( 0 ) − inf f = 1 . At x 0 = 0 x_0=0 x 0 = 0 only the first rotated coordinate is away from its minimizer, since ϕ ′ ( 0 ) = 0 \phi'(0)=0 ϕ ′ ( 0 ) = 0 , so along the line through x 0 x_0 x 0 in direction Q e 1 Qe_1 Q e 1 the gradient stays parallel to Q e 1 Qe_1 Q e 1 ; random directions still pay the curvature of the other coordinates. The concentrated spectrum sets h 1 = 1 h_1=1 h 1 = 1 , assigns 0.1 / ( d − 1 ) 0.1/(d-1) 0.1/ ( d − 1 ) to each other eigenvalue, and distributes the remaining trace in order without exceeding one; the diffuse spectrum sets h 1 = 1 h_1=1 h 1 = 1 and h i = ( E − 1 ) / ( d − 1 ) h_i=(E-1)/(d-1) h i = ( E − 1 ) / ( d − 1 ) for i > 1 i>1 i > 1 . The rotation is the Householder reflection Q = I − 2 v v T Q=I-2vv^\mathsf{T} Q = I − 2 v v T with v = ( e 1 − u ) / ∥ e 1 − u ∥ v=(e_1-u)/\lVert e_1-u\rVert v = ( e 1 − u ) / ∥ e 1 − u ∥ and u u u uniform on the sphere. The grid uses d ∈ { 10 3 , 10 4 , 10 5 } d\in\{10^3,10^4,10^5\} d ∈ { 1 0 3 , 1 0 4 , 1 0 5 } , T ∈ { 2 , 8 , 32 } T\in\{2,8,32\} T ∈ { 2 , 8 , 32 } , and E ∈ { 2 , 4 , 8 , … , 128 } E\in\{2,4,8,\ldots,128\} E ∈ { 2 , 4 , 8 , … , 128 } for the nonconvex family, keeps the cells with E T ≤ d / 4 ET\le d/4 E T ≤ d /4 , and has five rotation seeds per cell. The three-point method uses h = 4 ε / ( 5 L ( E + 1 ) ) h=4\varepsilon/(5L(E+1)) h = 4 ε / ( 5 L ( E + 1 )) , N = ⌈ 10 ( E + 1 ) T ⌉ N=\lceil10(E+1)T\rceil N = ⌈ 10 ( E + 1 ) T ⌉ steps, an output drawn uniformly and independently from the iterates before the updates, and the hard cap 1 + 2 N 1 + 2N 1 + 2 N of Theorem G.3 ; it is also run with a fixed seed. The deterministic methods are coordinate central differences and finite-difference L-BFGS on both families, and Krylov conjugate gradients on the quadratic family.
On the regular grid (575 runs per method, 45 for Krylov), every method reached the target offline in every run. The returned output was ε \varepsilon ε -stationary in 524 of 575 runs of the three-point method, in 509 of 575 with a fixed seed, and in all runs of the deterministic methods. At d = 10 5 d = 10^{5} d = 1 0 5 , the log–log slopes of the first-hit cost of the three-point method range from 0.54 to 0.75 in T T T and from 0.90 to 0.95 in E E E . The 27 targeted runs comprise nine each of three-point method with a fixed seed, coordinate differences, and finite-difference L-BFGS. Each uses a rank-one quadratic chosen orthogonal to all of the run’s queries, a property certified in exact integer arithmetic (15 distinct functions). All 27 runs ended at gradient norm 2 \sqrt{2} 2 without an offline hit and are reported as right-censored, whereas 20 fresh seeds on each of these functions succeeded in 17–20 runs.
Stronger baselines. On the 43 cells of the regular grid with E ∈ { 1 , 4 , 16 } E \in\{1, 4, 16\} E ∈ { 1 , 4 , 16 } (215 instances, with the same functions and rotation seeds), we also ran two further methods. The Gaussian two-point method uses g k = [ f ( x k + μ a k ) − f ( x k − μ a k ) ] a k / ( 2 μ ) g_k = [f(x_k + \mu a_k) - f(x_k - \mu a_k)]a_k/(2\mu) g k = [ f ( x k + μ a k ) − f ( x k − μ a k )] a k / ( 2 μ ) with a k ∼ N ( 0 , I d ) a_k \sim N(0, I_d) a k ∼ N ( 0 , I d ) , step 1 / ( L ( E + 2 ) ) 1/(L(E + 2)) 1/ ( L ( E + 2 )) , N = ⌈ ( 6 E + 13 ) T ⌉ N = \lceil(6E + 13)T\rceil N = ⌈( 6 E + 13 ) T ⌉ steps, μ = ε / ( 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 ) ) \mu= \varepsilon/(4L(6E + 13)\sqrt{d}(d + 2)(d + 4)) μ = ε / ( 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 )) , and an output drawn uniformly from the iterates before the updates, for 2 N 2N 2 N calls in total. The restarted subspace search uses the capacity given below. The two-point method returned ε \varepsilon ε -stationary outputs in 203 of 215 runs and the subspace search in all 215; paired with the three-point method on the same functions and seeds, the subspace search reached the target later in all 43 cells, by a median factor between 8.7 and 1.8 × 10 4 1.8 \times10^{4} 1.8 × 1 0 4 . As in the targeted runs above, the subspace search with budgets 32, 64, and 128 below d d d ended at gradient norm 2 \sqrt{2} 2 on six rank-one quadratics orthogonal to all of its queries and its output, whereas 20 fresh seeds of the three-point method per function succeeded in 114 of 120 runs.
Monotone two-point method and wall functions. We also ran the monotone two-point method of Theorem B.10 on the same 215 instances, with the step, the number of updates, and the difference radius of the two-point method above, forward differences, and 2 N + 1 2N + 1 2 N + 1 calls. Its first hit came after a median of 31 calls, against 30 for the two-point method; in the cells of Figure 2 (a) it trailed the two-point method by about one call, its initial query. Its uniformly drawn output was ε \varepsilon ε -stationary in 203 of 215 runs, as for the two-point method, and its last iterate in all 215. We also built wall functions f ( x ) = h ( P x ) + β ( ∥ ( I − P ) x ∥ 2 ) f(x) = h(Px) + \beta(\lVert(I - P)x\rVert^{2}) f ( x ) = h ( P x ) + β (∥( I − P ) x ∥ 2 ) , where P P P projects onto a random subspace of dimension E 0 E_0 E 0 , h h h is a sum of smooth one-dimensional profile polynomials with ∣ h ′ ′ ∣ ≤ L \lvert h''\rvert\le L ∣ h ′′ ∣ ≤ L and h ( 0 ) = inf h = Δ / 2 h(0) = \inf h = \Delta/2 h ( 0 ) = inf h = Δ/2 , and β ( s ) = L E 0 s / ( 2 d ) \beta(s) = LE_0s/(2d) β ( s ) = L E 0 s / ( 2 d ) for s ≤ s 1 = d Δ / ( L E 0 ) s \le s_1 = d\Delta/(LE_0) s ≤ s 1 = d Δ/ ( L E 0 ) , beyond which β ′ \beta' β ′ rises smoothly to L / 4 L/4 L /4 with 2 β ′ + 4 s β ′ ′ ≤ L 2\beta' + 4s\beta'' \le L 2 β ′ + 4 s β ′′ ≤ L . Below f ( 0 ) f(0) f ( 0 ) one has s < s 1 s < s_1 s < s 1 , so these functions lie in B L , Δ , E 0 d \mathcal{B}^{d}_{L,\Delta,E_0} B L , Δ , E 0 d , whereas above f ( 0 ) f(0) f ( 0 ) the Hessian trace reaches about L d / 2 Ld/2 L d /2 . On 45 instances (9 cells with d ∈ { 10 3 , 10 4 , 10 5 } d \in\{10^{3}, 10^{4}, 10^{5}\} d ∈ { 1 0 3 , 1 0 4 , 1 0 5 } , T ∈ { 8 , 32 } T \in\{8, 32\} T ∈ { 8 , 32 } , E 0 ∈ { 4 , 16 } E_0 \in\{4, 16\} E 0 ∈ { 4 , 16 } , and 8 E 0 T ≤ d 8E_0T \le d 8 E 0 T ≤ d , five seeds each), the last iterate of the monotone two-point method was ε \varepsilon ε -stationary in all runs, and the uniformly drawn outputs of the two-point and three-point methods in 44 and 42; their median first hits were 69, 72, and 173 calls. No method exceeded f ( 0 ) f(0) f ( 0 ) . The two-point method with step 1 / ( L ( d + 2 ) ) 1/(L(d + 2)) 1/ ( L ( d + 2 )) , which is safe without an acceptance test, reached the target in none of the 45 runs within the same budget.
Subspace search. The restarted subspace search uses capacity r = ⌈ d E T / ( d + T ) ⌉ r = \lceil\sqrt{dET/(d + T)}\rceil r = ⌈ d E T / ( d + T ) ⌉ , a variant without restarts uses r = d r = d r = d , and coordinate central differences follow Appendix B. On the two families above with d ∈ { 10 3 , 10 4 } d \in\{10^{3}, 10^{4}\} d ∈ { 1 0 3 , 1 0 4 } , T ∈ { 2 , 8 , 32 , 128 , 512 } T \in\{2, 8, 32, 128, 512\} T ∈ { 2 , 8 , 32 , 128 , 512 } , E = 1 E = 1 E = 1 for the quadratic, and E ∈ { 2 , 4 , 8 , 16 } E \in\{2, 4, 8, 16\} E ∈ { 2 , 4 , 8 , 16 } for the nonconvex family (1,800 runs including the three-point method), all deterministic runs returned ε \varepsilon ε -stationary outputs and the three-point method did so in 425 of 450 runs. Every subspace run used exactly one expansion and no restart, consistent with the gradient staying along Q e 1 Qe_1 Q e 1 ; on the nonconvex family, the paired median ratio of total calls between the subspace search and coordinate differences is about 0.67 for T ≤ 128 T \le128 T ≤ 128 and 0.50 at T = 512 T = 512 T = 512 .
The chain has E ∈ { 8 , 16 } E \in\{8,16\} E ∈ { 8 , 16 } , m = 2 E m = 2E m = 2 E , a = 8 ( m + 1 ) / m a = \sqrt{8(m+1)/m} a = 8 ( m + 1 ) / m , a fixed Householder reflection R R R , z = ( R x ) 1 : m z = (Rx)_{1:m} z = ( R x ) 1 : m , and
f ( x ) = 1 8 [ ( z 1 − a ) 2 + ∑ i = 1 m − 1 ( z i − z i + 1 ) 2 + z m 2 ] − 1 m . f(x) = \frac{1}{8}\left[(z_1-a)^2+\sum_{i=1}^{m-1}(z_i-z_{i+1})^2+z_m^2\right]-\frac{1}{m}. f ( x ) = 8 1 [ ( z 1 − a ) 2 + i = 1 ∑ m − 1 ( z i − z i + 1 ) 2 + z m 2 ] − m 1 .
Its global upper model H = 1 4 R T diag ( K m , 0 ) R H = \frac{1}{4}R^{\mathsf{T}}\operatorname{diag}(K_m,0)R H = 4 1 R T diag ( K m , 0 ) R , where K m K_m K m is tridiagonal with diagonal 2 and off-diagonal − 1 -1 − 1 , satisfies ∥ H ∥ o p < 1 \lVert H\rVert_{\mathrm{op}} < 1 ∥ H ∥ op < 1 and tr H = E \operatorname{tr} H = E tr H = E ; moreover f ( 0 ) = 1 f(0) = 1 f ( 0 ) = 1 and inf f = 0 \inf f = 0 inf f = 0 . At a point supported on the first j j j active coordinates, the gradient adds only coordinate j + 1 j+1 j + 1 , so each expansion can add at most one direction. With d ∈ { 10 3 , 10 4 } d \in\{10^3,10^4\} d ∈ { 1 0 3 , 1 0 4 } , T ∈ { 2 , 8 , 32 , 128 , 512 } T \in\{2,8,32,128,512\} T ∈ { 2 , 8 , 32 , 128 , 512 } , and five seeds, all 300 runs returned ε \varepsilon ε -stationary outputs; 240 of them satisfy E T ≤ d ET \le d E T ≤ d . The number of expansions was 1, 2, 3, 6, 10 at the five accuracies, with no restart. At d = 10 4 d = 10^4 d = 1 0 4 , the paired median ratio of total calls was about 1.00 for T ≤ 8 T \le8 T ≤ 8 , 0.57 at T = 32 T = 32 T = 32 , 0.39 to 0.41 at T = 128 T = 128 T = 128 , and 0.25 to 0.26 at T = 512 T = 512 T = 512 . Over the five accuracies, the log–log slope of the total cost is about 0.31 for the subspace search and 0.57 for coordinate differences, both below the exponents 1 / 2 1/2 1/2 and 1 of the worst-case upper bounds.
Local curvature of GPT-2. GPT-2 uses the GELU activation, so its loss is smooth in the parameters. We ran MeZO on SST-2 and RTE with three seeds each and 1,024 updates, reaching final accuracies of 76.6–85.2% and 50.8–53.9%, and at updates 0, 256, 512, 1024 measured the Hessian G G G of a fixed training batch in all d = 124,439,808 d = 124{,}439{,}808 d = 124 , 439 , 808 parameters, in FP32. Since tr ( G + ) = ( tr G + ∥ G ∥ ∗ ) / 2 \operatorname{tr}(G_+) = (\operatorname{tr} G + \lVert G\rVert_*)/2 tr ( G + ) = ( tr G + ∥ G ∥ ∗ ) /2 and ∥ G ∥ ∗ ≤ d ∥ G ∥ F \lVert G\rVert_* \le\sqrt{d}\lVert G\rVert_F ∥ G ∥ ∗ ≤ d ∥ G ∥ F , any u 1 ≥ tr G u_1 \ge\operatorname{tr} G u 1 ≥ tr G , u 2 ≥ ∥ G ∥ F 2 u_2 \ge\lVert G\rVert_F^2 u 2 ≥ ∥ G ∥ F 2 and ℓ ∈ ( 0 , ∥ G ∥ o p ] \ell\in(0,\lVert G\rVert_{\mathrm{op}}] ℓ ∈ ( 0 , ∥ G ∥ op ] give
tr ( G + ) d ∥ G ∥ o p ≤ u 1 / d + u 2 / d 2 ℓ . \frac{\operatorname{tr}(G_+)}{d\lVert G\rVert_{\mathrm{op}}} \le\frac{u_1/d+\sqrt{u_2/d}}{2\ell}. d ∥ G ∥ op tr ( G + ) ≤ 2 ℓ u 1 / d + u 2 / d .
We take for ℓ \ell ℓ the absolute Rayleigh quotient after a 32-step power iteration. From m = 128 m = 128 m = 128 Rademacher probes, one Hessian-vector product each, we form the Hutchinson estimates σ ^ 1 \hat{\sigma}_1 σ ^ 1 of tr G \operatorname{tr} G tr G and σ ^ 2 \hat{\sigma}_2 σ ^ 2 of tr ( G 2 ) = ∥ G ∥ F 2 \operatorname{tr}(G^2) = \lVert G\rVert_F^2 tr ( G 2 ) = ∥ G ∥ F 2 [26 ] , and set u 2 = σ ^ 2 / ( 1 − ρ ) u_2 = \hat{\sigma}_2/(1-\rho) u 2 = σ ^ 2 / ( 1 − ρ ) with ρ = 6 log ( 2 / δ ) / m ≈ 0.595 \rho= \sqrt{6\log(2/\delta)/m} \approx0.595 ρ = 6 log ( 2/ δ ) / m ≈ 0.595 and u 1 = σ ^ 1 + 2 u 2 / ( m δ ) u_1 = \hat{\sigma}_1+\sqrt{2u_2/(m\delta)} u 1 = σ ^ 1 + 2 u 2 / ( m δ ) , where δ = 0.025 / 24 \delta= 0.025/24 δ = 0.025/24 . As in the proof of Theorem 1 of Roosta-Khorasani and Ascher [45 ] , Markov’s inequality, convexity and E e − t z 2 ≤ 1 − t + 3 t 2 / 2 \mathbb{E}e^{-tz^2} \le1-t+3t^2/2 E e − t z 2 ≤ 1 − t + 3 t 2 /2 for z = a T w z = a^{\mathsf{T}}w z = a T w with ∥ a ∥ = 1 \lVert a\rVert= 1 ∥ a ∥ = 1 and Rademacher w w w give P ( u 2 < ∥ G ∥ F 2 ) ≤ e m t ( 1 − ρ ) ( 1 − t + 3 t 2 / 2 ) m \mathbb{P}(u_2 < \lVert G\rVert_F^2) \le e^{mt(1-\rho)}(1-t+3t^2/2)^m P ( u 2 < ∥ G ∥ F 2 ) ≤ e m t ( 1 − ρ ) ( 1 − t + 3 t 2 /2 ) m , which is below 1.3 × 10 − 4 1.3 \times10^{-4} 1.3 × 1 0 − 4 at t = 0.22 t = 0.22 t = 0.22 . Each Rademacher sample of tr G \operatorname{tr} G tr G has variance at most 2 ∥ G ∥ F 2 2\lVert G\rVert_F^2 2 ∥ G ∥ F 2 , so by Chebyshev’s inequality u 2 ≥ ∥ G ∥ F 2 u_2 \ge\lVert G\rVert_F^2 u 2 ≥ ∥ G ∥ F 2 and u 1 < tr G u_1 < \operatorname{tr} G u 1 < tr G hold together with probability at most δ \delta δ . A union bound over the 24 points shows that, with probability at least 0.95, every plotted value bounds the ratio from above. These bounds range from 6.70 × 10 − 5 6.70 \times10^{-5} 6.70 × 1 0 − 5 to 9.19 × 10 − 5 9.19 \times10^{-5} 9.19 × 1 0 − 5 on SST-2 and from 6.83 × 10 − 5 6.83 \times10^{-5} 6.83 × 1 0 − 5 to 1.16 × 10 − 4 1.16 \times10^{-4} 1.16 × 1 0 − 4 on RTE, and decrease along every run.
Low-dimensional diagnostics. Comparisons on a related separable family at d ∈ { 16 , 64 , 128 } d \in\{16,64,128\} d ∈ { 16 , 64 , 128 } had E T > d ET > d E T > d in every cell, outside the regime where randomized costs are known to be of order E T ET E T . There, at the isotropic endpoint E = d E = d E = d the coordinate methods were faster, random directions were slower in most cells at d = 16 d = 16 d = 16 , and diffuse spectra gave heterogeneous results. Curvature measurements on OPT models use the ReLU activation, so their Hessian products concern one activation branch only.
Lean Formalization of the Main Bounds
Theorems 3.1 and 3.2 and Corollary 3.3 on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d , together with the randomized lower bound of Corollary F.4 on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , are formalized, in the form stated below, in Lean 4 (version 4.19.0) with the Mathlib library (revision c44e0c8) and checked by the Lean kernel. The formalization is available at https://github.com/Pyx9876/zo-lower-bounds-lean and in the ancillary files of this arXiv entry. Its README lists the Lean name of each statement below. Beyond propositional extensionality, the axiom of choice, and the soundness of quotients, on which Mathlib rests, the proofs use no axioms or unproved assumptions.
Model. Queries, outputs, and costs follow Section 2. A call ( x , V ) (x,V) ( x , V ) returns ( f ( x ) , V T ∇ f ( x ) ) (f(x),V^{\mathsf{T}}\nabla f(x)) ( f ( x ) , V T ∇ f ( x )) at cost 1 + rank V 1+\operatorname{rank} V 1 + rank V , and a call with exact values only has V = 0 V=0 V = 0 . A deterministic algorithm maps each finite transcript to its next call or to an output point; the output is unrestricted and uncharged. N det N_{\det} N d e t is the smallest worst-case cost of an algorithm that returns an ε \varepsilon ε -stationary point on every function of the class. A randomized algorithm draws its seed from a law fixed before the function and chooses each call by a policy that is jointly measurable in the seed and the transcript. It must return an ε \varepsilon ε -stationary point with probability at least 2 / 3 2/3 2/3 on each function of the class; the number of calls is not bounded in advance, and a run that does not terminate counts as a failure with infinite cost. N_{\rand} is the smallest cap on the cost of all runs, and N_{\rand}^{\exp} is the smallest worst-case expected cost. The class B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d is defined as in (2) : one fixed H ⪰ 0 H\succeq0 H ⪰ 0 with ∥ H ∥ o p ≤ L \lVert H\rVert_{\mathrm{op}}\le L ∥ H ∥ op ≤ L and tr H ≤ L E \operatorname{tr}H\le LE tr H ≤ L E dominates the derivative of ∇ f \nabla f ∇ f almost everywhere on { f < f ( 0 ) } \{f<f(0)\} { f < f ( 0 )} . Lemma B.2, which identifies this class with the segment form used in the proofs, is formalized as well. The class F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d is formalized through the quadratic upper model (3) on all of R d \mathbb{R}^{d} R d .
Formalized statements. Let d ≥ 1 d\ge1 d ≥ 1 , L , Δ , ε > 0 L,\Delta,\varepsilon>0 L , Δ , ε > 0 , 1 ≤ E ≤ d 1\le E\le d 1 ≤ E ≤ d , and T ≥ 1 T\ge1 T ≥ 1 . For the value oracle and for the rank-priced oracle, N det ( B L , Δ , E d ) N_{\det}(\mathcal{B}^{d}_{L,\Delta,E}) N d e t ( B L , Δ , E d ) is finite and satisfies
d ≤ N det ≤ min { 10 d T , 100 d E T ( d + T ) } . d\le N_{\det}\le\min\{10dT,\ 100\sqrt{dET(d+T)}\}. d ≤ N d e t ≤ min { 10 d T , 100 d E T ( d + T ) } .
E T ≤ d ⟹ d 2250 min { T , E T log ( e + T ) } ≤ N det ≤ 100 2 d E T , ET\le d\quad\Longrightarrow\quad\frac{d}{2250}\min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\}\le N_{\det}\le100\sqrt{2}\,d\sqrt{ET}, E T ≤ d ⟹ 2250 d min { T , log ( e + T ) E T } ≤ N d e t ≤ 100 2 d E T ,
E ≥ T log ( e + T ) ⟹ N det ≥ d T 2359296 , T log ( e + T ) ≤ E ≤ d T ⟹ N det ≥ d T 2250 . E\ge T\log(e+T)\quad\Longrightarrow\quad N_{\det}\ge\frac{dT}{2359296},\qquad T\log(e+T)\le E\le\frac{d}{T}\quad\Longrightarrow\quad N_{\det}\ge\frac{dT}{2250}. E ≥ T log ( e + T ) ⟹ N d e t ≥ 2359296 d T , T log ( e + T ) ≤ E ≤ T d ⟹ N d e t ≥ 2250 d T .
For randomized algorithms and both oracles, N_{\rand}(\mathcal{B}^{d}_{L,\Delta,E}) is finite, and
N_{\rand}(\mathcal{F}^{d}_{L,\Delta,E})\ge\frac{\min\{d,ET\}}{12},\qquad N_{\rand}^{\exp}(\mathcal{F}^{d}_{L,\Delta,E})\ge\frac{\min\{d,ET\}}{144},
\frac{\min\{d,ET\}}{12}\le N_{\rand}(\mathcal{B}^{d}_{L,\Delta,E})\le23ET,
where the upper bound is attained by the three-point method of Theorem B.8. If T log ( e + T ) ≤ E ≤ d / T T\log(e+T)\le E\le d/T T log ( e + T ) ≤ E ≤ d / T , then for the same oracle
\frac{d}{51750E}\le\frac{N_{\det}(\mathcal{B}^{d}_{L,\Delta,E})}{N_{\rand}(\mathcal{B}^{d}_{L,\Delta,E})}\le\frac{120d}{E}.
These statements imply Theorems 3.1 and 3.2 and Corollary 3.3 on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d , whose constants are not specified. The randomized lower bounds have the constants of Corollary F.4. The formalized deterministic lower bounds use more conservative constants than Theorem A.1 (1/2250 in place of 1/672), including the constant 9 in place of 13/3 in the selection Lemma C.2 and cruder chain and wrapper bounds.
Not formalized. The lower bound of Theorem E.8 , the variant (7) without E E E or H H H , the three-point bound on A L , Δ , E d A^{d}_{L,\Delta,E} A L , Δ , E d , the coercivity, nonconvexity, and Lipschitz properties of the hard functions, the deterministic lower bounds on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , and the results of Appendices G and H are not formalized.