Introduction
Zeroth-order methods optimize with function values alone. They are the tool of choice when gradients are unavailable or too costly: attacking or explaining a model only through its outputs [13 ] , tuning simulations, hyperparameters, and engineering designs [33 ] , and searching policies in reinforcement learning [46 ] ; see [37 ] for a survey. Interest has surged with large language models. Fine-tuning with forward passes alone needs about the memory of inference, so a single GPU can tune models that backpropagation cannot fit [38 , 62 ] , and seed-based zeroth-order updates reduce communication in federated fine-tuning [44 ] .
For an L L L -smooth nonconvex function on R d \mathbb{R}^{d} R d with initial gap Δ \Delta Δ , the classical analysis of random finite differences finds an ε \varepsilon ε -stationary point with O ( d T ) O(dT) O ( d T ) function values, where T = L Δ / ε 2 T = L\Delta/\varepsilon^{2} T = L Δ/ ε 2 [41 , 19 ] . On the whole smooth class this is optimal for exact
values: concurrent and independent work of Xia and Yuan [55 ] shows that randomized, and hence deterministic, methods need Ω ( d T ) \Omega(dT) Ω ( d T ) calls when d ≳ T log T d\gtrsim T\log T d ≳ T log T . For deterministic methods we give the explicit bound d T / 500 dT/500 d T /500 for 1 ≤ T ≤ d / log ( e d ) 1\le T\le d/\log(ed) 1 ≤ T ≤ d / log ( e d ) and extend it to rank-priced gradient projections (Theorem 3.4 ), matching coordinate finite differences. In high-dimensional tasks such as language model fine-tuning, however, randomized zeroth-order methods succeed at far lower cost, which effective rank analyses attribute to a low curvature trace (Malladi et al., [38 ] ; Yue et al. [57 ] ), in line with the low intrinsic dimension of fine-tuning (Li et al., [35 ] ; Aghajanyan et al. [1 ] ). Does this saving come from problem structure alone, or does it require randomness? The question concerns randomized methods in practice too: zeroth-order fine-tuning regenerates each perturbation from a stored random seed (Malladi et al., [38 ] ), and a run whose seeds are fixed in advance is a deterministic algorithm, so deterministic lower bounds apply to it on functions chosen after the seeds are fixed (§ 7 defeats such a run). Classical derivative-free methods—coordinate and pattern search (Hooke and Jeeves, [24 ] ), the Nelder–Mead method (Nelder and Mead, [39 ] ), and Powell’s interpolation methods (Conn et al., [15 ] ; Larson et al. [33 ] )—are deterministic as well. We ask: what is the price of determinism under a low curvature trace—can deterministic methods match random directions by learning active subspaces, or must they still pay for the ambient dimension?
We quantify the worst-case cost of both algorithm classes on the same functions. Whereas prior work establishes a price of determinism for nonsmooth Goldstein stationarity (§ 6 ), we establish the smooth first-order counterpart in high dimension: randomness is essential to fully exploit a low curvature trace. Our main class keeps L L L -smoothness and the initial gap Δ \Delta Δ and adds an effective rank condition on the initial sublevel set, where descent methods operate: wherever f ( x ) < f ( 0 ) f(x)<f(0) f ( x ) < f ( 0 ) , the Hessian is dominated by one positive semidefinite matrix of operator norm at most L L L and trace at most L E LE L E . Because E E E bounds the trace rather than the rank, curvature may spread across many directions while negative curvature is uncharged; § 2.1 compares this condition with prior curvature assumptions. Malladi et al. [38 ] assume such a matrix near each iterate of a run and show that the slowdown of zeroth-order SGD then depends on its effective rank rather than on d d d ; this is an upper bound for one randomized method. Our condition is fixed before the run and imposed only on the initial sublevel set. Under it, when E T ≤ d ET\le d E T ≤ d , we prove that the Gaussian two-point method attains O ( E T ) O(ET) O ( E T ) calls if it rejects steps that raise f f f , that no randomized method needs fewer, and that deterministic methods need more calls by a factor of order d / ( E + E T ) d/(E+\sqrt{ET}) d / ( E + E T ) , up to logarithmic factors.
The price of determinism. Suppose E , T ≥ 1 E,T\ge1 E , T ≥ 1 and E T ≤ d ET\le d E T ≤ d . Intuitively, random probes pay only for the curvature trace, whereas deterministic methods pay the dimension d d d for each active direction they discover, and a fixed trace budget allows at most O ( E T ) O(\sqrt{ET}) O ( E T ) such directions. We show that deterministic methods always need at least d d d calls, already far above O ( E T ) O(ET) O ( E T ) when E T ≪ d ET\ll d E T ≪ d ; adaptive subspace search uses O ( d min { T , E T } ) O(d\min\{T,\sqrt{ET}\}) O ( d min { T , E T }) calls, and a hidden chain forces Ω ~ ( min { T , E T } ) \widetilde{\Omega}(\min\{T,\sqrt{ET}\}) Ω ( min { T , E T }) directions. Hence, writing N d e t N_{\mathrm{det}} N det and N r a n d N_{\mathrm{rand}} N rand for the worst-case query complexities (Theorems 3.1 and 3.2 ),
Ω ~ ( d min { T , E T } ) ≤ N d e t ≤ O ( d min { T , E T } ) , Ω ( E T ) ≤ N r a n d ≤ O ( E T ) , \begin{aligned}
\widetilde{\Omega}(d\min\{T,\sqrt{ET}\}) &\le N_{\mathrm{det}} \le O(d\min\{T,\sqrt{ET}\}),\\
\Omega(ET) &\le N_{\mathrm{rand}} \le O(ET),
\end{aligned} Ω ( d min { T , E T }) Ω ( E T ) ≤ N det ≤ O ( d min { T , E T }) , ≤ N rand ≤ O ( E T ) ,
where Ω ~ \widetilde{\Omega} Ω hides only a factor log ( e + T ) \sqrt{\log(e+T)} log ( e + T ) ; Section 3 states the logarithm explicitly. The price of determinism, N d e t / N r a n d N_{\mathrm{det}}/N_{\mathrm{rand}} N det / N rand , thus lies between Ω ~ ( d / ( E + E T ) ) \widetilde{\Omega}(d/(E+\sqrt{ET})) Ω ( d / ( E + E T )) and O ( d / ( E + E T ) ) O(d/(E+\sqrt{ET})) O ( d / ( E + E T )) , growing linearly with d d d for fixed E E E and T T T . Once E ≥ T log ( e + T ) E\ge T\log(e+T) E ≥ T log ( e + T ) , deterministic cost reverts to Ω ( d T ) \Omega(dT) Ω ( d T ) , so both classes are matched up to constant factors with cost ratio of order d / E d/E d / E (Corollary 3.3).
Why exact feedback can still be hard. When each reply carries a bounded number of bits, counting information gives lower bounds (Basu et al. [7 ] ). An exact real reply can encode arbitrarily much discrete information, so counting returned scalars gives no lower bound. Recent lower bounds with exact function values for convex minimization on a ball overcome this obstacle for deterministic (Kerger [29 ] ; Wu et al. [53 ] ) and randomized (Zhang et al. [59 ] ) methods, bounding objective error on a compact domain. Our lower bounds concern the gradient norm on R d \mathbb{R}^{d} R d under unrestricted queries, unqueried outputs, and rank-priced gradient projections, by constructing functions whose replies agree exactly along the adaptive transcript. Against a deterministic algorithm, a resisting oracle (Nemirovsky and Yudin [40 ] ) fixes each hidden direction only when its stage closes, after Ω ( d ) \Omega(d) Ω ( d ) units of feedback: it is chosen orthogonal to the stage’s query points and to all its projection subspaces except possibly the closing one, and the next unfinished direction is chosen orthogonal to the output. One fixed function in the class realizes all replies and keeps the output gradient above ε \varepsilon ε . Multiplying the hidden chain by a perspective weight rules out small gradients far away and keeps the Hessian dominated by one fixed matrix on all of R d \mathbb{R}^{d} R d (Appendix § 16 .3). For randomized algorithms, Yao’s principle (Yao [56 ] ) with an unknown quadratic center gives Ω ( E T ) \Omega(ET) Ω ( E T ) .
Contributions.
Randomized cost scales with the trace, not the dimension. The Gaussian two-point method [41 ] uses O ( E T ) O(ET) O ( E T ) calls for all E , T ≥ 1 E,T \ge1 E , T ≥ 1 when it rejects steps that raise f f f ; without this test it uses O ( E T ) O(ET) O ( E T ) calls when the matrix bound holds on all of R d \mathbb{R}^d R d and O ( E T + T log ( E T ) ) O(ET+T\log(ET)) O ( E T + T log ( E T )) calls in general. Conversely, every randomized method needs Ω ( E T ) \Omega(ET) Ω ( E T ) calls when E T ≤ d ET \le d E T ≤ d (Theorem 3.1 ).
Quantifying the price of determinism under low trace. For E T ≤ d ET \le d E T ≤ d , adaptive subspace search achieves O ( d min { T , E T } ) O(d\min\{T,\sqrt{ET}\}) O ( d min { T , E T }) calls, while C ∞ C^\infty C ∞ hard functions force Ω ~ ( d min { T , E T } ) \widetilde{\Omega}(d\min\{T,\sqrt{ET}\}) Ω ( d min { T , E T }) calls even when the matrix bound holds on all of R d \mathbb{R}^d R d (Theorem 3.2 ). Across T log ( e + T ) ≤ E ≤ d / T T\log(e+T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T , the trace stops helping deterministic methods (Ω ( d T ) \Omega(dT) Ω ( d T ) calls), and determinism multiplies the cost by order d / E d/E d / E (Corollary 3.3).
Optimality of coordinate finite differences without a trace budget. On the unrestricted smooth class, deterministic methods need at least d T / 500 dT/500 d T /500 calls when 1 ≤ T ≤ d / log ( e d ) 1 \le T \le d/\log(ed) 1 ≤ T ≤ d / log ( e d ) , including under adaptive rank-priced gradient projections (Theorem 3.4 ). For exact values, the randomized bound of Xia and Yuan [55 ] already implies Ω ( d T ) \Omega(dT) Ω ( d T ) ; our bound adds the explicit constant and the gradient projections.
Table 1 places these results among the closest prior bounds.
Table 1. Query complexity of finding an ε \varepsilon ε -stationary point and the algorithms that attain the upper bounds. D and R denote deterministic and randomized algorithms, NC means nonconvex, and T = L Δ / ε 2 T=L\Delta/\varepsilon^{2} T = L Δ/ ε 2 . All results use exact function values unless the setting names another oracle. Bold entries are new in this paper. “Matched”: whether the two bounds agree up to constant factors (yes), up to the stated factor, or only on stated ranges; a dash means that only one side is known.
Section 2 defines the classes and oracles, Section 3 states the formal bounds, and Section 4 explains the proof mechanisms. Section 5 compares with prior work, Section 6 illustrates the mechanisms numerically, and Section 7 covers open problems and limitations. Appendices A, B, and G give explicit constants, algorithm analyses, and C ∞ C^\infty C ∞ constructions under global trace bounds.
Setting
Function classes. For L , Δ > 0 L,\Delta> 0 L , Δ > 0 , let
F L , Δ d : = { f ∈ C 1 , 1 ( R d ) : Lip ( ∇ f ) ≤ L , f ( 0 ) − inf f ≤ Δ } . (1) \mathcal{F}_{L,\Delta}^{d} := \left\{f \in C^{1,1}(\mathbb{R}^{d}) : \operatorname{Lip}(\nabla f) \le L,\ f(0)-\inf f \le\Delta\right\}. \tag*{(1)} F L , Δ d := { f ∈ C 1 , 1 ( R d ) : Lip ( ∇ f ) ≤ L , f ( 0 ) − inf f ≤ Δ } . (1)
The goal is a point x ^ ∈ R d \hat{x} \in\mathbb{R}^{d} x ^ ∈ R d with ∥ ∇ f ( x ^ ) ∥ ≤ ϵ \|\nabla f(\hat{x})\| \le\epsilon ∥∇ f ( x ^ ) ∥ ≤ ϵ , and we write T : = L Δ / ϵ 2 T := L\Delta/\epsilon^{2} T := L Δ/ ϵ 2 . For 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , the main class B L , Δ , E d \mathcal{B}_{L,\Delta,E}^{d} B L , Δ , E d consists of the f ∈ F L , Δ d f \in\mathcal{F}_{L,\Delta}^{d} f ∈ F L , Δ d for which some unknown matrix H ⪰ 0 H \succeq0 H ⪰ 0 with ∥ H ∥ o p ≤ L \|H\|_{\mathrm{op}} \le L ∥ H ∥ op ≤ L and tr H ≤ L E \operatorname{tr} H \le LE tr H ≤ L E satisfies
∇ 2 f ( x ) ⪯ H for almost every x with f ( x ) < f ( 0 ) . (2) \nabla^{2}f(x) \preceq H \qquad\text{for almost every }x\text{ with }f(x)<f(0). \tag*{(2)} ∇ 2 f ( x ) ⪯ H for almost every x with f ( x ) < f ( 0 ) . (2)
Matrix bounds of this form, with tr H / ∥ H ∥ o p \operatorname{tr} H/\|H\|_{\mathrm{op}} tr H /∥ H ∥ op called an effective rank, are used to explain the low cost of zeroth-order fine-tuning ([38 ] ). Requiring (2) only on the initial sublevel set U = { x : f ( x ) < f ( 0 ) } U=\{x:f(x)<f(0)\} U = { x : f ( x ) < f ( 0 )} matches where descent methods operate: above f ( 0 ) f(0) f ( 0 ) the class requires only L L L -smoothness, and negative curvature is uncharged, so functions may be strongly nonconvex.
The global fixed-matrix class F L , Δ , E d ⊆ B L , Δ , E d \mathcal{F}_{L,\Delta,E}^{d} \subseteq\mathcal{B}_{L,\Delta,E}^{d} F L , Δ , E d ⊆ B L , Δ , E d ((Lemma G.1)) requires (2) for almost every x ∈ R d x \in\mathbb{R}^{d} x ∈ R d ; for C 1 , 1 C^{1,1} C 1 , 1 functions this is the quadratic upper model
f ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 1 2 v T H v ( x , v ∈ R d ) , (3) f(x+v) \le f(x)+\langle\nabla f(x),v\rangle+\frac{1}{2}v^{\mathsf{T}}Hv \qquad(x,v\in\mathbb{R}^{d}), \tag*{(3)} f ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 2 1 v T H v ( x , v ∈ R d ) , (3)
and we call H H H an upper model of f f f . For E = d E=d E = d every function in F L , Δ d \mathcal{F}_{L,\Delta}^{d} F L , Δ d belongs to F L , Δ , E d \mathcal{F}_{L,\Delta,E}^{d} F L , Δ , E d . Our lower bounds hold on F L , Δ , E d \mathcal{F}_{L,\Delta,E}^{d} F L , Δ , E d and thus on B L , Δ , E d \mathcal{B}_{L,\Delta,E}^{d} B L , Δ , E d . Our randomized upper bound needs neither a common matrix nor smoothness above f ( 0 ) f(0) f ( 0 ) , only a locally L L L -Lipschitz gradient on the sublevel set and the pointwise bound tr ( [ ∇ 2 f ( x ) ] + ) ≤ L E \operatorname{tr}([\nabla^{2}f(x)]_{+}) \le LE tr ([ ∇ 2 f ( x ) ] + ) ≤ L E for almost every x x x with f ( x ) < f ( 0 ) f(x)<f(0) f ( x ) < f ( 0 ) , where A + A_{+} A + keeps the positive eigenvalues of a symmetric matrix A A A ; Appendix G.1 defines this local pointwise class.
What H H H and E E E mean. The matrix H H H is an upper model for the curvature that a descent method can meet. Its trace bounds the total positive curvature by L E LE L E , so E E E plays the role of a dimension: E = d E=d E = d adds nothing to L L L -smoothness, whereas a small E E E forces curvature to be concentrated in few directions or spread thinly over many. Only an upper bound is imposed, so negative curvature, and with it strong nonconvexity, does not count toward E E E . One matrix for the whole sublevel set is the structure that a deterministic method can learn and reuse (§ 5 ); randomized methods need only the pointwise bound.
Oracles and costs. At call t t t , an algorithm chooses x t ∈ R d x_t \in\mathbb{R}^d x t ∈ R d from past replies and receives the exact value f ( x t ) f(x_t) f ( x t ) ; its output x ^ ∈ R d \hat{x} \in\mathbb{R}^d x ^ ∈ R d need not be queried. The deterministic cost N d e t N_{\mathrm{det}} N det is the smallest worst-case call count over deterministic algorithms that succeed on every function in the class. A randomized algorithm must succeed with probability at least 2 / 3 2/3 2/3 on each function while making at most N N N calls on every trajectory (a cost cap of N N N ); N r a n d N_{\mathrm{rand}} N rand is the smallest such cap. Algorithms know d d d , L L L , Δ \Delta Δ , ε \varepsilon ε , and E E E , and internal computation and precision are free. Randomized policies are jointly measurable in their seed and finite transcript (Appendix § 15 ).
All lower bounds also hold for a stronger rank-priced oracle: at call t t t the algorithm may in addition choose V t ∈ R d × s t V_t \in\mathbb{R}^{d \times s_t} V t ∈ R d × s t and receive
O f ( x t , V t ) = ( f ( x t ) , V t ⊤ ∇ f ( x t ) ) at cost 1 + rank V t . (4) \mathcal{O}_f(x_t,V_t) = \left(f(x_t), V_t^\top\nabla f(x_t)\right) \quad\text{at cost } 1+\operatorname{rank} V_t. \tag*{(4)} O f ( x t , V t ) = ( f ( x t ) , V t ⊤ ∇ f ( x t ) ) at cost 1 + rank V t . (4)
A rank-one call is a forward-mode directional derivative ([8 ] ), and V t = I d V_t=I_d V t = I d is a full gradient. We write N v a l N^{\mathrm{val}} N val and N l i n N^{\mathrm{lin}} N lin when the oracle matters (N l i n ≤ N v a l N^{\mathrm{lin}} \le N^{\mathrm{val}} N lin ≤ N val ). The lower bounds also hold for worst-case expected cost.
Comparison with prior curvature assumptions
We compare F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d with the conditions under which prior zeroth-order rates improve on d d d .
Sparse and latent structure ([52 , 21 ] ). Wang et al. [52 ] assume sparsity, and Golovin et al. [21 ] study objectives with a low-dimensional latent structure. A globally L L L -smooth objective that depends only on a fixed k k k -dimensional subspace lies in F L , Δ , k d \mathcal{F}^{d}_{L,\Delta,k} F L , Δ , k d ; our trace bound also allows curvature of full rank.
Local effective rank ([38 ] ). Near each iterate x t x_t x t of a run, ∇ 2 f ⪯ H t \nabla^2 f \preceq H_t ∇ 2 f ⪯ H t with tr H t ≤ r ∥ H t ∥ o p \operatorname{tr} H_t \le r \lVert H_t \rVert_{\mathrm{op}} tr H t ≤ r ∥ H t ∥ op . Letting H t H_t H t change along the run is more general than our single H H H , but the condition depends on the algorithm, whereas a lower bound needs a class fixed before the run. Every f ∈ F L , Δ , E d f \in\mathcal{F}^{d}_{L,\Delta,E} f ∈ F L , Δ , E d satisfies it with r = E + 1 r=E+1 r = E + 1 : take H t = H + ( L − ∥ H ∥ o p ) v v ⊤ H_t=H+(L-\lVert H\rVert_{\mathrm{op}})vv^\top H t = H + ( L − ∥ H ∥ op ) v v ⊤ for a top unit eigenvector v v v of H H H , so that ∥ H t ∥ o p = L \lVert H_t\rVert_{\mathrm{op}}=L ∥ H t ∥ op = L and tr H t ≤ L ( E + 1 ) \operatorname{tr} H_t \le L(E+1) tr H t ≤ L ( E + 1 ) . Our lower bounds therefore apply in their setting.
Effective dimension and nuclear norm ([57 , 3 ] ). These sum powers of the absolute Hessian eigenvalues, or bound ∥ ∇ 2 f ∥ ∗ ≤ L ∗ \lVert\nabla^2 f\rVert_* \le L_* ∥ ∇ 2 f ∥ ∗ ≤ L ∗ , at every point of R d \mathbb{R}^d R d . Our condition is weaker in charging only positive curvature and only below f ( 0 ) f(0) f ( 0 ) , so a strongly nonconvex function can have a small E E E but large values of these measures; it is stronger in asking for one common matrix, the structure that deterministic subspace search exploits.
Since our randomized upper bound holds on the local pointwise class of Appendix § 16 .1, which contains the nuclear norm class with L E = L ∗ LE=L_* L E = L ∗ , and our lower bounds hold on functions of F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d with Hessian nuclear norm at most L E LE L E , the separation holds on every class between them, including the class of Anonymous [3 ] .
Main Results
Throughout, d ≥ 1 d \ge1 d ≥ 1 , 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , and T ≥ 1 T \ge1 T ≥ 1 , with absolute constants c , C > 0 c,C > 0 c , C > 0 . Unless another class is named, costs refer to the main class B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d . All bounds hold for both oracles and on the global fixed-matrix class F L , Δ , E d ⊆ B L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} \subseteq\mathcal{B}^{d}_{L,\Delta,E} F L , Δ , E d ⊆ B L , Δ , E d , where all lower bounds are proved (Appendix § 16 .3); the randomized upper bound holds even under the pointwise trace bound of Appendix § 16 .1.
Randomized complexity.
Theorem 3.1 (Randomized cost). For all the above parameters , N r a n d ≤ C E T N_{\mathrm{rand}} \le C\,ET N rand ≤ C E T . If E T ≤ d ET \le d E T ≤ d , then N r a n d ≥ c E T N_{\mathrm{rand}} \ge c\,ET N rand ≥ c E T .
Random directions replace d d d by the trace budget E E E . The upper bound is attained by the Gaussian two-point method ([41 ] ), the estimator of zeroth-order fine-tuning ([38 ] ), with step 1 / ( L ( E + 2 ) ) 1/(L(E+2)) 1/ ( L ( E + 2 )) : if it accepts a step only when f f f decreases, it uses at most 12 E T + 26 T + 3 12ET+26T+3 12 E T + 26 T + 3 calls, two values per step as without the test (Theorem B.10 ). Without the test, it uses at most 34 ( E + 2 ) T + 2 34(E+2)T+2 34 ( E + 2 ) T + 2 calls on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , and O ( E T + T log ( E T ) ) O(ET+T\log(ET)) O ( E T + T log ( E T )) calls on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d with a smaller step, because a long step may leave the sublevel set (Theorems B.12 and B.15 ). The stochastic three-point method ([9 ] ) attains O ( E T ) O(ET) O ( E T ) under that pointwise bound (Theorem G.3 ). The lower bound holds for every randomized method, including all these variants, and uses quadratics with an unknown center in a subspace of dimension at most d d d .
The price of determinism.
Theorem 3.2 (Deterministic cost). If E T ≤ d ET \le d E T ≤ d , then
c d min { T , E T log ( e + T ) } ≤ N d e t ≤ C d min { T , E T } . c d \min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\} \le N_{\mathrm{det}} \le C d \min\left\{T,\sqrt{ET}\right\}. c d min { T , log ( e + T ) E T } ≤ N det ≤ C d min { T , E T } .
In Ω ~ \widetilde{\Omega} Ω notation, the bounds are Ω ~ ( d min { T , E T } ) \widetilde{\Omega}(d\min\{T,\sqrt{ET}\}) Ω ( d min { T , E T }) and O ( d min { T , E T } ) O(d\min\{T,\sqrt{ET}\}) O ( d min { T , E T }) , agreeing up to log ( e + T ) \sqrt{\log(e+T)} log ( e + T ) . When E < T E<T E < T , adaptive subspace search beats coordinate finite differences (O ( d T ) O(dT) O ( d T ) ) by paying about d d d calls for a full gradient estimate only when its current subspace is exhausted; the common matrix H H H of B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d caps these expansions at O ( E T ) O(\sqrt{ET}) O ( E T ) , even without E E E or H H H as input (§ 5 , Algorithm 1, and Theorem B.1 ). The lower bound uses C ∞ C^\infty C ∞ hard functions with ∥ ∇ 2 f ∥ ∗ ≤ L E \|\nabla^2 f\|_* \le LE ∥ ∇ 2 f ∥ ∗ ≤ L E on all of R d \mathbb{R}^d R d and a prescribed Hessian Lipschitz constant (Appendix § 16 .3).
Corollary 3.3 (A window with matching bounds). If T log ( e + T ) ≤ E ≤ d / T T\log(e+T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T , then
c d T ≤ N d e t ≤ C d T , c E T ≤ N r a n d ≤ C E T . \begin{aligned}
c\,dT &\le N_{\mathrm{det}} \le C\,dT,\\
c\,ET &\le N_{\mathrm{rand}} \le C\,ET.
\end{aligned} c d T c E T ≤ N det ≤ C d T , ≤ N rand ≤ C E T .
For the same oracle , the cost ratio is bounded below by Ω ( d / E ) \Omega(d/E) Ω ( d / E ) and above by O ( d / E ) O(d/E) O ( d / E ) .
Once E ≥ T log ( e + T ) E \ge T\log(e+T) E ≥ T log ( e + T ) , a threshold set by accuracy alone, deterministic methods lose their trace savings and pay Ω ( d T ) \Omega(dT) Ω ( d T ) ; the window is nonempty when T 2 log ( e + T ) ≤ d T^2\log(e+T) \le d T 2 log ( e + T ) ≤ d , which for d = 10 6 d=10^6 d = 1 0 6 allows T T T up to about 400. The constants are explicit in the appendix, for example c = 1 / 672 c=1/672 c = 1/672 in Theorem 3.2 on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d (Theorem A.1 ) and 1 / 612 1/612 1/612 on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d (Corollary G.9 ); they come from the analysis, including stage conditions such as 28560 m 2 log ( e + m ) ≤ E T 28560\,m^2\log(e+m) \le ET 28560 m 2 log ( e + m ) ≤ E T in Theorem G.8 , and are not optimized. Across all E T ≤ d ET \le d E T ≤ d , the cost ratio lies between Ω ~ ( d / ( E + E T ) ) \widetilde{\Omega}(d/(E+\sqrt{ET})) Ω ( d / ( E + E T )) and O ( d / ( E + E T ) ) O(d/(E+\sqrt{ET})) O ( d / ( E + E T )) (Corollary E.5 ), which is at least Ω ~ ( d ) \widetilde{\Omega}(\sqrt{d}) Ω ( d ) whenever E ≤ T E \le T E ≤ T . Theorem 3.2 , the deterministic part of Corollary 3.3, and the three-point upper bound on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d are formalized in Lemma 4 (Appendix § 19 ).
Without a trace budget.
Theorem 3.4 (Unrestricted smooth class). On F L , Δ d \mathcal{F}^{d}_{L,\Delta} F L , Δ d , if 1 ≤ T ≤ d / log ( e d ) 1 \le T \le d/\log(ed) 1 ≤ T ≤ d / log ( e d ) , then
d T 500 ≤ N d e t ≤ 10 d T . \frac{dT}{500} \le N_{\mathrm{det}} \le10dT. 500 d T ≤ N det ≤ 10 d T . For exact values, the randomized lower bound of Xia and Yuan [55 ] already gives Ω ( d T ) \Omega(dT) Ω ( d T ) for deterministic methods when d ≳ T log T d \gtrsim T\log T d ≳ T log T ; Theorem 3.4 adds the explicit constant 1 / 500 1/500 1/500 and holds under rank-priced gradient projections (Theorem E.8 ), where a randomized Ω ( d T ) \Omega(dT) Ω ( d T ) bound remains open. Coordinate finite differences are thus optimal up to constants across 1 ≤ T ≤ d / log ( e d ) 1 \le T \le d/\log(ed) 1 ≤ T ≤ d / log ( e d ) for both oracles. The hard functions of Theorem 3.4 (which for T > 500 T > 500 T > 500 are coercive, nonconvex, and globally 2 L Δ 2\sqrt{L\Delta} 2 L Δ -Lipschitz) separate deterministic from randomized methods: a randomized batching method minimizes each of them exactly with probability at least 5 / 6 5/6 5/6 using O ( d T log ( e + T ) ) O(d\sqrt{T}\log(e + T)) O ( d T log ( e + T )) values (Proposition H.2 ).
Beyond these ranges. Figure 1 maps the parameter plane; Appendix § 10 gives explicit constants. For all E , T ≥ 1 E,T \ge1 E , T ≥ 1 , deterministic methods need at least d d d calls, satisfy N d e t ≤ C min { d T , d E T ( d + T ) } N_{\mathrm{det}} \le C\min\{dT,\sqrt{dET}(d+T)\} N det ≤ C min { d T , d E T ( d + T )} via subspace search, and need Ω ( d T ) \Omega(dT) Ω ( d T ) whenever E ≥ T log ( e + T ) E \ge T\log(e + T) E ≥ T log ( e + T ) (also when E T > d ET > d E T > d ). For exact values, the lower bound of Xia and Yuan [55 ] extends N r a n d ≥ c E T N_{\mathrm{rand}} \ge cET N rand ≥ c E T to E T ≤ c d 2 / log ( e d ) ET \le cd^{2}/\log(ed) E T ≤ c d 2 / log ( e d ) , so the matched window of Corollary 3.3 widens to T log ( e + T ) ≤ E ≤ d T\log(e + T) \le E \le d T log ( e + T ) ≤ E ≤ d when T ≤ c d / log ( e d ) T \le cd/\log(ed) T ≤ c d / log ( e d ) (Corollary H.3 ).
Figure 1. Which of our bounds match? The curves E = T log ( e + T ) E = T\log(e + T) E = T log ( e + T ) and E T = d ET = d E T = d divide the parameter plane, drawn for d = 10 6 d = 10^{6} d = 1 0 6 . Each region gives the cost of deterministic and randomized methods where our lower and upper bounds match up to constant factors; ≈ \approx ≈ also hides a factor log ( e + T ) \sqrt{\log(e + T)} log ( e + T ) , and “not matched” indicates differing bounds; see Appendix § 10 for the conditions.
Proof Mechanisms
Combining the constructions below across parameter regimes yields all statements in Section 3; Appendix § 10 .1 gives the short derivations.
Hiding one direction at a time
Why must a deterministic method pay Ω ( d ) \Omega(d) Ω ( d ) calls per hidden direction even when exact real replies carry infinitely many bits? Our lower bound separates an adversary argument, which shows that one unfinished direction forces Ω ( d ) \Omega(d) Ω ( d ) units of feedback, from an analytic construction that packs as many directions as smoothness, initial gap, and sublevel trace allow.
As in the chains of Carmon et al. [12 ] , later directions sit behind scalar gates that vanish identically near zero. At a completed query, later directions contribute nothing to f ( x ) f(x) f ( x ) or ∇ f ( x ) \nabla f(x) ∇ f ( x ) , while the active unfinished direction u j u_j u j contributes a gradient above ε \varepsilon ε . A resisting oracle (Nemirovsky and Yudin, [40 ] ) (Lemma C.4) chooses each u j u_j u j after earlier queries are fixed. While pending calls in stage j j j have total cost below ⌊ d / 2 ⌋ \lfloor d/2\rfloor ⌊ d /2 ⌋ , choosing u j u_j u j orthogonal to their query points and projection subspaces makes all replies match the prefix function; on the call closing stage j j j , u j u_j u j is fixed before its projection is returned. Keeping each later direction nearly orthogonal to all queries of earlier stages, so that their replies stay valid, costs the factor log ( e + T ) \sqrt{\log(\mathrm{e}+T)} log ( e + T ) (Lemma C.2 ). If the call budget runs out early, an unfinished direction is also chosen orthogonal to the final output x ^ \hat{x} x ^ , even if x ^ \hat{x} x ^ was never queried, so one fixed function in the class realizes the transcript with ∥ ∇ f ( x ^ ) ∥ > ε \lVert\nabla f(\hat{x})\rVert> \varepsilon ∥ ∇ f ( x ^ )∥ > ε .
Hiding m m m directions forces cost Ω ( d m ) \Omega(dm) Ω ( d m ) and spends a constant normalized initial gap per direction. To allow unrestricted queries across R d \mathbb{R}^{d} R d , a radial compression ρ R ( x ) \rho_R(x) ρ R ( x ) at a radius R R R bounds the hidden coordinates and a radial potential prevents small gradients far away. On B L , Δ , E d B^d_{L,\Delta,E} B L , Δ , E d , the potential is weakly curved inside a ball containing { f < f ( 0 ) } \{f<f(0)\} { f < f ( 0 )} and grows steeply only outside, where f ( x ) > f ( 0 ) f(x)>f(0) f ( x ) > f ( 0 ) . Inside the ball, the Hessian is dominated by one fixed matrix, a multiple of the projection onto the m m m hidden directions plus a small multiple of the identity, so the trace budget pays only for the weak inner curvature and compressed chain. When E ≥ T log ( e + T ) E\ge T\log(\mathrm{e}+T) E ≥ T log ( e + T ) and E ≤ d E\le d E ≤ d , all stages allowed by accuracy fit the trace budget and dimension. Multiplying the compressed chain by the perspective weight ( 1 + ∥ x ∥ 2 / R 2 ) 1 / 2 (1+\lVert x\rVert^2/R^2)^{1/2} ( 1 + ∥ x ∥ 2 / R 2 ) 1/2 , which forms a perspective function (Boyd and Vandenberghe, [10 ] ), instead of adding a potential cancels the decay of its gradient far away, so one fixed matrix dominates the Hessian on all of R d \mathbb{R}^{d} R d and every lower bound holds on F L , Δ , E d F^d_{L,\Delta,E} F L , Δ , E d (Appendix § 16 .3). Smoothing only the scalar gates while preserving their flat zero intervals makes the hard functions C ∞ C^\infty C ∞ without changing a single reply (Appendices § 13 .1 and § 16 .3).
The number of discovered directions is the bottleneck on both sides. A deterministic search that, as in Cosson et al. [16 ] , seeks a new unit direction u i u_i u i only when the span of earlier ones is exhausted gains about ε 2 / ( u i T H u i ) \varepsilon^2/(u_i^{\mathsf{T}}Hu_i) ε 2 / ( u i T H u i ) along u i u_i u i , and restarts with an empty span after r r r expansions (Algorithm 1). Within each block, orthonormal directions satisfy ∑ i u i T H u i ≤ L E \sum_i u_i^{\mathsf{T}}Hu_i\le LE ∑ i u i T H u i ≤ L E , so by Cauchy–Schwarz a full block decreases f f f by at least Ω ( r ε 2 / ( L E ) ) \Omega(r\varepsilon^2/(LE)) Ω ( r ε 2 / ( L E )) while total decrease is at most Δ \Delta Δ . Taking r r r of order E T \sqrt{ET} E T allows only O ( E T ) O(\sqrt{ET}) O ( E T ) expansions when E T ≤ d ET\le d E T ≤ d (Appendix § 11 ). Conversely, the hidden chain forces min { T , E T / log ( e + T ) } \min\{T,\sqrt{ET/\log(\mathrm{e}+T)}\} min { T , E T / log ( e + T ) } directions, so the deterministic bounds of Theorem 3.2 meet up to log ( e + T ) \sqrt{\log(\mathrm{e}+T)} log ( e + T ) .
The randomized obstruction is different
Against a randomized algorithm, the hard function cannot be chosen after observing the random seed. By Yao’s principle (Yao, [56 ] ), a distributional lower bound suffices: it fixes a random quadratic with an unknown center in a public subspace before the run begins. Each rank-priced query reduces to a counted number of linear observations of the center, and rotational symmetry bounds the remaining error, including at an unqueried output. Because the public subspace has dimension at most d d d , this argument gives Ω ( E T ) \Omega(ET) Ω ( E T ) up to E T = d ET=d E T = d .
Random directions need control only below f ( 0 ) f(0) f ( 0 )
A Gaussian two-point step moves from x x x to x − η s a x-\eta s a x − η s a , where s s s is the finite-difference slope of f f f along a ∼ N ( 0 , I d ) a \sim N(0,I_d) a ∼ N ( 0 , I d ) . With g = ∇ f ( x ) g=\nabla f(x) g = ∇ f ( x ) , the quadratic upper model and the Gaussian moment E [ ( a ⊤ g ) 2 a ⊤ H a ] = ∥ g ∥ 2 tr H + 2 g ⊤ H g ≤ L ( E + 2 ) ∥ g ∥ 2 \mathbb{E}[(a^\top g)^2a^\top Ha]=\lVert g\rVert^2\operatorname{tr}H+2g^\top Hg\le L(E+2)\lVert g\rVert^2 E [( a ⊤ g ) 2 a ⊤ H a ] = ∥ g ∥ 2 tr H + 2 g ⊤ H g ≤ L ( E + 2 ) ∥ g ∥ 2 charge the trace of H H H rather than d d d , so the step η = 1 / ( L ( E + 2 ) ) \eta=1/(L(E+2)) η = 1/ ( L ( E + 2 )) lowers f f f by order ∥ g ∥ 2 / ( L E ) \lVert g\rVert^2/(LE) ∥ g ∥ 2 / ( L E ) on average. A step this long may leave the sublevel set, where only L L L -smoothness holds. The capped model bounds min { f ( x − η s a ) , f ( 0 ) } \min\{f(x-\eta s a),f(0)\} min { f ( x − η s a ) , f ( 0 )} by the same quadratic (Lemma B.3 ), so accepting only steps that lower f f f suffices (Theorem B.10 ). Without the test, we compare the iterates with an auxiliary run that drops the rare directions with large a ⊤ H a a^\top Ha a ⊤ H a ; with a slightly smaller step and a small difference radius the two runs stay within ε / ( 8 L ) \varepsilon/(8L) ε / ( 8 L ) , at the price of a T log ( E T ) T\log(ET) T log ( E T ) term (Theorem B.15 ). Under the pointwise trace bound there is no common H H H ; there the three-point method keeps the lowest of f ( x ) f(x) f ( x ) and f ( x ± h a ) f(x\pm ha) f ( x ± ha ) , and averaging over the Gaussian component across the gradient charges the positive trace (Theorem G.3 ).
Related Work
Zeroth-order upper bounds under curvature conditions. The classical analysis of Gaussian finite differences reduces derivative-free optimization to stochastic first-order methods at the cost of a factor d d d ([41 , 19 , 18 ] ). The curvature conditions compared in § 2.1 remove most of this factor for randomized methods. They measure curvature differently from us: a hard function F F F of Theorem D.4 with compression radius R R R has ∇ 2 F ( 0 ) = μ I \nabla^2F(0)=\mu I ∇ 2 F ( 0 ) = μ I with μ = 11 m / R 2 \mu=11m/R^2 μ = 11 m / R 2 , giving effective dimension at least d μ d\sqrt{\mu} d μ in the sense of [57 ] , whereas its positive trace is of order m + μ d m+\mu d m + μ d . Anonymous [3 ] show that the Gaussian two-point method finds an ε \varepsilon ε -stationary point under ∥ ∇ 2 f ∥ ∗ ≤ L ∗ \lVert\nabla^2f\rVert_*\le L_* ∥ ∇ 2 f ∥ ∗ ≤ L ∗ in O ( L ∗ Δ / ε 2 ) O(L_*\Delta/\varepsilon^2) O ( L ∗ Δ/ ε 2 ) exact values (and with an extra factor depending on the noise under noisy values). Their class satisfies our pointwise trace bound with L E = L ∗ LE=L_* L E = L ∗ , where the three-point method attains the same rate, and their Twisted Valley example, in which fixed or neighborhood matrix bounds can require trace of order d d d while the pointwise bound is constant, is why our randomized upper bound uses the pointwise bound (Appendix § 16 .1). Neither Anonymous [3 ] nor Anonymous [2 ] proves lower bounds. For stochastic convex problems on a compact domain whose subgradients lie in the range of a fixed structural preconditioner, Sholokhov and Rogozin [48 ] use ellipsoidal sampling, with bounds depending on the subspace rank, conditioning, and normalized trace. The Hessian trace also governs the implicit regularization of two-point estimators ([61 ] ), random subspace methods ([32 , 43 ] ) use related structure under different assumptions, and local gradient smoothness has been studied for first-order and derivative-free methods ([30 ] (Zhang and Hong 2024)). On the deterministic side, our upper bound adapts the subspace expansion scheme of Cosson et al. [16 ] from low-dimensional gradient subspaces to a fixed trace budget. Under higher-order smoothness, Zhou et al. [63 ] give deterministic value-based methods for stationarity in every dimension through a trapping-or-descent scheme, with query counts polynomial in d d d ; our classes assume only a Lipschitz gradient, together with a trace budget.
Lower bounds for stationarity and for exact values. Stationarity lower bounds in fixed dimension on a box go back to Vavasis [51 ] and were extended to randomized algorithms in d = 2 d=2 d = 2 by Bubeck and Mikulincer [11 ] . On R 2 \mathbb{R}^2 R 2 , Hollender and Zampetakis [23 ] show that deterministic methods need Ω ( T ) \Omega(\sqrt{T}) Ω ( T ) value or gradient calls for stationarity and O ( T ) O(\sqrt{T}) O ( T ) value calls suffice; high-dimensional first-order Ω ( T ) \Omega(T) Ω ( T ) lower bounds use the hidden chains of Carmon et al. [12 ] , which also give stochastic lower bounds ([4 ] ). With noisy values, convex query complexity grows with d d d ([27 , 47 , 18 ] ). Recent work studies oracles returning exact values for convex optimization on a ball, where one real reply can encode many coordinates: for deterministic methods, Kerger [29 ] proves a nearly quadratic lower bound for Lipschitz convex minimization, and Wu et al. [53 ] prove smooth and strongly convex bounds via shielded chains and delayed rotations that match on specified accuracy ranges; for randomized methods, Zhang et al. [59 ] use a posterior mean energy method to prove an extra factor d d d up to logarithms in Lipschitz convex optimization. With noisy values on a ball, Zhang et al. [58 ] prove a joint dimension–accuracy lower bound for Goldstein stationarity. Whereas those convex bounds with exact values concern objective error on a ball, our bounds control the gradient norm on all of R d \mathbb{R}^{d} R d under a curvature trace budget and allow rank-priced adaptive gradient projections. On the unrestricted smooth class with exact values, concurrent and independent work of Xia and Yuan [55 ] proves a randomized Ω ( d T ) \Omega(dT) Ω ( d T ) lower bound when d ≥ C T log ( 2 + T ) d \ge CT \log(2 + T) d ≥ C T log ( 2 + T ) for an absolute constant C C C ; Theorem 3.4 gives an explicit constant and covers gradient projections, where a randomized Ω ( d T ) \Omega(dT) Ω ( d T ) lower bound remains open.
The price of determinism. For Lipschitz nonsmooth functions and Goldstein stationarity, Jordan et al. [28 ] prove a deterministic Ω ( d ) \Omega(d) Ω ( d ) lower bound against dimension-free randomized methods, and Kornowski [31 ] shows that the deterministic price can be exponential. In one dimension, Chewi et al. [14 ] show that randomness or function values provably help in finding stationary points of smooth functions. Our work establishes the high-dimensional smooth counterpart under a low curvature trace: for E T ≤ d ET \le d E T ≤ d , determinism multiplies the complexity by order d / ( E + E T ) d/(E + \sqrt{ET}) d / ( E + E T ) up to log ( e + T ) \sqrt{\log(e + T)} log ( e + T ) , and by order d / E d/E d / E up to constant factors across T log ( e + T ) ≤ E ≤ d / T T\log(e + T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T .
Empirical Illustration
We run the algorithms of our upper bounds on fixed instances to illustrate the mechanisms behind Theorems 3.1 and 3.2 . The runs show four things: (i) deterministic costs grow linearly with d d d , while randomized costs stay nearly constant in d d d and scale with E E E and T T T ; (ii) the Gaussian two-point method and its monotone variant, which accepts only steps that lower f f f (Theorems B.12 and B.10 ) have comparable first-hit counts and need about half the calls of the three-point method on these instances, also with curvature controlled only below f ( 0 ) f(0) f ( 0 ) ; (iii) runs with a fixed seed can be defeated, as in our lower bound; and (iv) subspace search saves calls by reusing directions, and the local curvature of GPT-2 is consistent with a small trace budget. All synthetic runs use L = Δ = 1 L = \Delta= 1 L = Δ = 1 and ε = T − 1 / 2 \varepsilon= T^{-1/2} ε = T − 1/2 and charge every function value, including rejected proposals and final checks. The first-hit count of a run is the number of calls before its first ε \varepsilon ε -stationary iterate, computed offline from the recorded iterates; no method uses the true gradient to stop or to select its output.
Randomized versus deterministic. We compare the Gaussian two-point method with central differences (Nesterov and Spokoiny, [41 ] ), its monotone variant, and the three-point method of Theorem G.3 against coordinate central differences, finite-difference L-BFGS (Nocedal, [42 ] ; Liu and Nocedal, [36 ] ), and our restarted subspace search on a rank-one quadratic and on rotated separable nonconvex functions with concentrated or diffuse spectra, for d ∈ { 10 3 , 10 4 , 10 5 } d \in\{10^3, 10^4, 10^5\} d ∈ { 1 0 3 , 1 0 4 , 1 0 5 } , T ∈ { 2 , 8 , 32 } T \in\{2, 8, 32\} T ∈ { 2 , 8 , 32 } , and E E E up to 128 with E T ≤ d / 4 ET \le d/4 E T ≤ d /4 . Because the deterministic methods spend about d d d calls per gradient estimate, their cost grows linearly with d d d , whereas the randomized methods stay nearly constant in d d d (Figure 2 (a)). In panel (a) at d = 10 5 d = 10^5 d = 1 0 5 , the subspace search first reaches the target after 200,009 calls and coordinate finite differences after 600,001, whereas the three-point method needs 77, the two-point method 38, and the monotone two-point method 39. Over the 215 instances with E ∈ { 1 , 4 , 16 } E \in\{1, 4, 16\} E ∈ { 1 , 4 , 16 } (five seeds per cell; Appendix § 18 ), the monotone two-point method first reached the target after a median of 31 calls, against 30 for the two-point method; its last iterate was ε \varepsilon ε -stationary in all 215 runs and its uniformly drawn output, which Theorem B.10 analyzes, in 203, as for the two-point method. These instances satisfy the upper model on all of R \mathbb{R} R , where the two-point method needs O ( E T ) O(ET) O ( E T ) calls (Theorem B.12 ). A run with a fixed seed is a deterministic algorithm and can be defeated: on rank-one quadratics chosen orthogonal to its query sequence it never reaches the target, while fresh seeds on the same functions succeed in 17–20 of 20 runs.
Figure 2. Query costs on synthetic functions and local curvature of GPT-2 (GELU). (a) Family with a concentrated spectrum with E = 4 E = 4 E = 4 and T = 8 T = 8 T = 8 : offline first-hit calls (solid) of the three-point method (STP), the Gaussian two-point method, the monotone two-point method, the subspace search, coordinate finite differences, and finite-difference L-BFGS, and actual total calls (dashed). (b) Actual total calls of the subspace search and coordinate finite differences on a tridiagonal quadratic chain with d = 10 4 d = 10^{4} d = 1 0 4 and E = 8 E = 8 E = 8 ; numbers give the subspace expansions. In (a) and (b), points and bands are medians and interquartile ranges over five seeds. (c) Upper confidence bounds, at simultaneous level 0.95, for tr ( G + ) / ( d ∥ G ∥ o p ) \operatorname{tr}(G_{+})/(d \lVert G \rVert_{\mathrm{op}}) tr ( G + ) / ( d ∥ G ∥ op ) , where G G G is the Hessian on a fixed batch, along MeZO fine-tuning, three seeds per task. Protocols and further results are in Appendix § 18 .
Curvature controlled only below f ( 0 ) f(0) f ( 0 ) . On wall functions whose Hessian is small below f ( 0 ) f(0) f ( 0 ) but of order L L L in almost every direction above it (Appendix § 18 ), the monotone two-point method again matched the two-point method, with median first hits of 69 and 72 calls against 173 for the three-point method. No method exceeded f ( 0 ) f(0) f ( 0 ) on these instances, so the plain two-point method was not hurt either, but shrinking its step to 1 / ( L ( d + 2 ) ) 1/(L(d+2)) 1/ ( L ( d + 2 )) , which makes it safe without an acceptance test, left all 45 runs short of the target within the same budget.
Subspace search. On a tridiagonal quadratic chain, gradients reveal new directions one at a time, so the subspace search needs one expansion at T = 2 T = 2 T = 2 and ten at T = 512 T = 512 T = 512 (Figure 2 (b)). Reusing the directions already found instead of estimating full gradients again, it uses about a quarter of the calls of coordinate finite differences at d = 10 4 d = 10^4 d = 1 0 4 and T = 512 T = 512 T = 512 , while matching their cost at the two lowest accuracies. On the separable family of panel (a), whose gradient stays along one hidden direction, a single expansion suffices, yet it still costs about d d d calls.
Local curvature of a smooth model. GPT-2 uses a smooth activation, so its loss Hessian is defined everywhere, unlike for ReLU networks. Along MeZO fine-tuning, an upper confidence bound for tr ( G + ) / ( d ∥ G ∥ o p ) \operatorname{tr}(G_+)/(d \Vert G \Vert_{\mathrm{op}}) tr ( G + ) / ( d ∥ G ∥ op ) , where G G G is the Hessian on a fixed batch, stays between 6.7 × 10 − 5 6.7 \times10^{-5} 6.7 × 1 0 − 5 and 1.2 × 10 − 4 1.2 \times10^{-4} 1.2 × 1 0 − 4 (Figure 2 (c)). With about 1.24 × 10 8 1.24 \times10^{8} 1.24 × 1 0 8 parameters, this keeps the local trace budget tr ( G + ) / ∥ G ∥ o p \operatorname{tr}(G_+)/\Vert G \Vert_{\mathrm{op}} tr ( G + ) /∥ G ∥ op below 1.5 × 10 4 1.5 \times10^{4} 1.5 × 1 0 4 , comparable to d \sqrt{d} d and consistent with the few large Hessian eigenvalues reported for neural networks ([20 ] ). Low-dimensional runs with E T > d ET > d E T > d , where coordinate differences can beat random directions, and measurements on ReLU models are reported in Appendix § 18 .
Discussion and Open Problems
Limitations. The separation is proved for exact replies, whereas Anonymous [3 ] also allow noise; independent noise can supply random bits, so deterministic lower bounds do not transfer to that model. Without an acceptance test, our bound for the two-point method on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d has an extra T log ( E T ) T \log(ET) T log ( E T ) term and uses a very small difference radius; whether a step of order 1 / ( L E ) 1/(LE) 1/ ( L E ) alone gives O ( E T ) O(ET) O ( E T ) calls is open. The randomized upper bounds take E E E as input, and a randomized cap of order E T ET E T without knowledge of E E E remains open.
Conclusion. We quantified the price of determinism for smooth nonconvex zeroth-order optimization under a low curvature trace. Below f ( 0 ) f(0) f ( 0 ) , the Gaussian two-point method with an acceptance test finds an ε \varepsilon ε -stationary point in O ( E T ) O(ET) O ( E T ) calls, matching our Ω ( E T ) \Omega(ET) Ω ( E T ) lower bound for E T ≤ d ET \le d E T ≤ d , whereas every deterministic method pays a factor of order d / ( E + E T ) d/(E+\sqrt{ET}) d / ( E + E T ) more, up to log ( e + T ) \sqrt{\log(e+T)} log ( e + T ) . Closing the remaining logarithm, determining deterministic costs on the local pointwise class, and resolving the remaining complexities for E T > d ET > d E T > d are the main open problems.
AI Usage Statement
We used AI assistants, including OpenAI’s GPT models and Anthropic’s Claude, to help develop and check proofs, formalize them in Lean, implement the experiments, and edit the text. The authors take full responsibility for the content of this paper.
Main Theorem and Proof Overview
Write T = L Δ / ε 2 ≥ 1 T = L\Delta/\varepsilon^2 \ge1 T = L Δ/ ε 2 ≥ 1 and 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d . The class B \mathcal{B} B contains globally L L L -smooth C 1 , 1 C^{1,1} C 1 , 1 functions with f ( 0 ) − inf f ≤ Δ f(0) - \inf f \le\Delta f ( 0 ) − inf f ≤ Δ and one fixed unknown matrix 0 ≺ H ⪯ L I 0 \prec H \preceq LI 0 ≺ H ⪯ L I , tr H ≤ L E \operatorname{tr} H \le LE tr H ≤ L E , dominating the Hessian almost everywhere on { f < f ( 0 ) } \{f < f(0)\} { f < f ( 0 )} . The subclass F \mathcal{F} F requires the matrix bound globally. An exact value costs one; feedback ( f ( x ) , V T ∇ f ( x ) ) (f(x), V^{\mathsf{T}}\nabla f(x)) ( f ( x ) , V T ∇ f ( x )) costs 1 + rank V 1 + \operatorname{rank} V 1 + rank V . The output need not be queried; exact computation is uncharged, and the superscripts cap and exp mark the cost cap of Section 2 and the worst-case expected cost.
Theorem A.1 (Main bounds with explicit constants). For either oracle on B \mathcal{B} B ,
N r a n d c a p ≤ 23 E T , N r a n d c a p ≥ min { d , E T } / 12 , N r a n d e x p ≥ min { d , E T } / 144 , N_{\mathrm{rand}}^{\mathrm{cap}} \le23ET,\qquad N_{\mathrm{rand}}^{\mathrm{cap}} \ge\min\{d,ET\}/12,\qquad N_{\mathrm{rand}}^{\mathrm{exp}} \ge\min\{d,ET\}/144, N rand cap ≤ 23 E T , N rand cap ≥ min { d , E T } /12 , N rand exp ≥ min { d , E T } /144 , N d e t ≤ min { 10 d T , 100 d E T ( d + T ) } , N d e t ≥ d . (5) N_{\mathrm{det}} \le\min\{10dT,100\sqrt{dET(d+T)}\},\qquad N_{\mathrm{det}} \ge d. \tag*{(5)} N det ≤ min { 10 d T , 100 d E T ( d + T ) } , N det ≥ d . (5) If E T ≤ d ET \le d E T ≤ d , then
N d e t ≥ d 672 min { T , E T / log ( e + T ) } . N_{\mathrm{det}} \ge\frac{d}{672}\min\{T,\sqrt{ET/\log(e+T)}\}. N det ≥ 672 d min { T , E T / log ( e + T ) } . Thus for T log ( e + T ) ≤ E ≤ d / T T\log(e+T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T the deterministic bounds are Ω ( d T ) \Omega(dT) Ω ( d T ) and O ( d T ) O(dT) O ( d T ) , and the randomized bounds are Ω ( E T ) \Omega(ET) Ω ( E T ) and O ( E T ) O(ET) O ( E T ) . On the unrestricted globally smooth class, N d e t ≥ d 500 min { T , d / log ( e d ) } N_{\mathrm{det}} \ge\frac{d}{500}\min\{T,d/\log(ed)\} N det ≥ 500 d min { T , d / log ( e d )} . Every bound above for B \mathcal{B} B also holds on F \mathcal{F} F : the upper bounds because F ⊆ B \mathcal{F} \subseteq\mathcal{B} F ⊆ B , the randomized lower bounds and N d e t ≥ d N_{\mathrm{det}} \ge d N det ≥ d because their quadratic witnesses lie in F \mathcal{F} F , and the bound for E T ≤ d ET \le d E T ≤ d by Corollary G.9.
Mechanism and roadmap. For the upper bounds (Appendix § 11 ), stopping a segment at its first exit from the initial sublevel set gives a capped quadratic upper model, so comparison and acceptance steps pay only for the trace, and the deterministic search buys a full gradient estimate only when its current subspace is exhausted, with its orthogonal expansions limited by the trace budget. The deterministic lower bounds combine one scalar chain with a radial compression and two choices of its parameters; a transcript lemma handles pending and closing calls and unqueried outputs, finite partial coloring selects the hidden directions, and scaling places the witnesses in the class. A quadratic family with an unknown center gives the randomized lower bound.
Scope. The multistage witnesses of this proof chain are C 1 , 1 C^{1,1} C 1 , 1 , and the Lean formalization of the lower bounds (Appendix § 19 ) follows this chain. Appendix § 16 gives the three-point bound on the larger class A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d (the local pointwise class of Appendix § 16 .1) and, reusing the steps of this chain, C ∞ C^\infty C ∞ witnesses with a Lipschitz Hessian for the same deterministic lower bound on F \mathcal{F} F , where a perspective weight replaces the radial potential. Appendix § 17 shows that, for 500 < T ≤ d / log ( e d ) 500 < T \le d/\log(ed) 500 < T ≤ d / log ( e d ) , the hard functions of the unrestricted bound are minimized exactly, with probability at least 5 / 6 5/6 5/6 , by a randomized method using O ( d T log ( e + T ) ) O(d\sqrt{T\log(e+T)}) O ( d T log ( e + T ) ) values, and combines our bounds with the lower bound of Xia and Yunan [55 ] into a wider window with matching bounds for exact values. In the unrestricted bound the witnesses are coercive and nonconvex when min { T , d / log ( e d ) } > 500 \min\{T,d/\log(ed)\} > 500 min { T , d / log ( e d )} > 500 ; otherwise the rank-one quadratic of Lemma E.2 is used.
From Theorem A.1 to Section 3
Theorem 3.1 consists of the bounds 23 E T 23ET 23 E T and min { d , E T } / 12 \min\{d,ET\}/12 min { d , E T } /12 ; the upper bound also holds on A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d with 43 E T 43ET 43 E T calls (Theorem G.3). The Gaussian two-point method attains it with 12 E T + 26 T + 3 12ET + 26T + 3 12 E T + 26 T + 3 calls when it accepts only steps that lower f f f (Theorem B.10); without this test it uses $34(E+2)T + 2 calls on F \mathcal{F} F and O ( E T + T log ( E T ) ) O(ET + T\log(ET)) O ( E T + T log ( E T )) calls on B \mathcal{B} B (Theorems B.12 and B.15 ). In Theorem 3.2 , E T ≤ d ET \le d E T ≤ d and E ≥ 1 E \ge1 E ≥ 1 give T ≤ d T \le d T ≤ d , so 100 d E T ( d + T ) ≤ 100 2 d E T 100\sqrt{dET}(d + T) \le100\sqrt{2}d\sqrt{ET} 100 d E T ( d + T ) ≤ 100 2 d E T ; the lower bound is Corollary E.4 , and on F \mathcal{F} F it is Corollary G.9 . If T log ( e + T ) ≤ E ≤ d / T T\log(e + T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T , then E T ≤ d ET \le d E T ≤ d and E T / log ( e + T ) ≥ T \sqrt{ET}/\log(e + T) \ge T E T / log ( e + T ) ≥ T , which gives Corollary 3.3, with the explicit ratio bounds of Corollary E.5 . Theorem 3.4 combines Theorem E.8 with the coordinate method in the proof of Theorem B.1 , which uses only global L L L -smoothness. The remaining statements of Section 3 are Lemma E.2, Theorem B.1 , Corollaries E.4 , G.9 and G.10, and, for exact values beyond E T ≤ d ET \le d E T ≤ d , Corollary H.3 .
Upper Bounds
Theorem B.1 (Restarted deterministic subspace search). For all the preceding parameters, including T > d T > d T > d , one has
N_{\det}^{\lin}(\mathcal{B}) \le N_{\det}^{\val}(\mathcal{B}) \le\min\{10dT,\ 100\sqrt{dET(d + T)}\}. \tag*{(6)}
The bounds hold for every function in the class with a single deterministic, finite algorithm that uses exact values, chosen from the public parameters. No convexity, Hessian rank bound, or Hessian Lipschitz assumption is added. The same result holds on the global fixed-matrix subclass F \mathcal{F} F . A single algorithm not given E E E or H H H has the hard bound
Q ≤ min { 20 d T , 560 d E T ( d + T ) } . (7) Q \le\min\{20dT,\ 560\sqrt{dET(d + T)}\}. \tag*{(7)} Q ≤ min { 20 d T , 560 d E T ( d + T ) } . (7) It is still given d d d , L L L , Δ \Delta Δ , ε \varepsilon ε .
Lemma B.2 (Hessians almost everywhere and on every segment). Let f ∈ C 1 , 1 ( R d ) f \in C^{1,1}(\mathbb{R}^d) f ∈ C 1 , 1 ( R d ) , U ⊆ R d U \subseteq\mathbb{R}^d U ⊆ R d be open, and H H H be symmetric. Then ∇ 2 f ⪯ H \nabla^2 f \preceq H ∇ 2 f ⪯ H almost everywhere on U U U is equivalent to
f ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 1 2 v T H v for each [ x , x + v ] ⊂ U . (8) f(x + v) \le f(x) + \langle\nabla f(x),v\rangle+ \frac{1}{2}v^{\mathsf{T}}Hv \quad\text{for each } [x,x + v] \subset U. \tag*{(8)} f ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 2 1 v T H v for each [ x , x + v ] ⊂ U . (8)
Proof. Choose a measurable null set N ⊂ U N \subset U N ⊂ U outside which the Lipschitz map ∇ f \nabla f ∇ f is differentiable, and its derivative is symmetric and bounded above by H H H . Fix v ≠ 0 v \ne0 v = 0 and [ x , x + v ] ⊂ U [x,x + v] \subset U [ x , x + v ] ⊂ U . By compactness a closed tube of sufficiently small positive radius about this segment lies in U U U . Use orthonormal coordinates with first direction v / ∥ v ∥ v/\|v\| v /∥ v ∥ . Fubini applied to 1 N \mathbf{1}_N 1 N on the tube implies that for almost every sufficiently small transverse displacement z ∈ v ⊥ z \in v^\perp z ∈ v ⊥ the set { t ∈ [ 0 , 1 ] : x + z + t v ∈ N } \{t \in[0,1] : x + z + tv \in N\} { t ∈ [ 0 , 1 ] : x + z + t v ∈ N } has one-dimensional measure zero. For each such z z z , G z ( t ) = ∇ f ( x + z + t v ) G_z(t) = \nabla f(x + z + tv) G z ( t ) = ∇ f ( x + z + t v ) is Lipschitz and hence absolutely continuous. For almost every t t t , G z ′ ( t ) = ∇ 2 f ( x + z + t v ) v G'_z(t) = \nabla^2 f(x + z + tv)v G z ′ ( t ) = ∇ 2 f ( x + z + t v ) v , whence
⟨ G z ( t ) − G z ( 0 ) , v ⟩ = ∫ 0 t v T ∇ 2 f ( x + z + s v ) v d s ≤ t v T H v . \langle G_z(t) - G_z(0),v\rangle= \int_0^t v^{\mathsf{T}}\nabla^2 f(x + z + sv)v\,\mathrm{d}s \le tv^{\mathsf{T}}Hv. ⟨ G z ( t ) − G z ( 0 ) , v ⟩ = ∫ 0 t v T ∇ 2 f ( x + z + s v ) v d s ≤ t v T H v .
Integration over t ∈ [ 0 , 1 ] t \in[0,1] t ∈ [ 0 , 1 ] proves (8) on the translated segment. The permitted transverse displacements have full measure in a ball, so choose z n → 0 z_n \to0 z n → 0 among them. Continuity of f f f and ∇ f \nabla f ∇ f proves the inequality for the given segment. When d = 1 d = 1 d = 1 , the inverse image of a one-dimensional null set under an injective affine map is null; the same argument works without translation. The case v = 0 v = 0 v = 0 is immediate. Conversely, at each point where ∇ f \nabla f ∇ f is differentiable with derivative B B B , continuity and integration of the gradient give
f ( x + t v ) = f ( x ) + t ⟨ ∇ f ( x ) , v ⟩ + t 2 2 v T B v + o ( t 2 ) . f(x + tv) = f(x) + t\langle\nabla f(x),v\rangle+ \frac{t^2}{2}v^{\mathsf{T}}Bv + o(t^2). f ( x + t v ) = f ( x ) + t ⟨ ∇ f ( x ) , v ⟩ + 2 t 2 v T B v + o ( t 2 ) .
Apply (8) for small positive t t t and divide by t 2 / 2 t^2/2 t 2 /2 ; then v T B v ≤ v T H v v^{\mathsf{T}}Bv \le v^{\mathsf{T}}Hv v T B v ≤ v T H v . Because the differentiability point was fixed, this holds for every v v v at once. Rademacher’s theorem for ∇ f \nabla f ∇ f provides such points almost everywhere.
The capped upper model and finite differences
Lemma B.3 (First exit and the capped upper model). Suppose f ∈ C 1 f \in C^{1} f ∈ C 1 , H ⪰ 0 H \succeq0 H ⪰ 0 , and every segment [ x , y ] [x,y] [ x , y ] contained in { f < c } \{f<c\} { f < c } satisfies
f ( y ) ≤ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ + 1 2 ( y − x ) T H ( y − x ) . f(y) \le f(x) + \langle\nabla f(x),y-x\rangle+ \frac{1}{2}(y-x)^{\mathsf{T}}H(y-x). f ( y ) ≤ f ( x ) + ⟨ ∇ f ( x ) , y − x ⟩ + 2 1 ( y − x ) T H ( y − x ) . For every f ( x ) ≤ c f(x) \le c f ( x ) ≤ c and every v v v ,
min { f ( x + v ) , c } ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 1 2 v T H v . (9) \min\{f(x+v),c\} \le f(x) + \langle\nabla f(x),v\rangle+ \frac{1}{2}v^{\mathsf{T}}Hv. \tag*{(9)} min { f ( x + v ) , c } ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 2 1 v T H v . (9) The hypothesis holds for B \mathcal{B} B with c = f ( 0 ) c=f(0) c = f ( 0 ) .
Proof. For f ( x ) < c f(x)<c f ( x ) < c , let q ( t ) = f ( x ) + t ⟨ ∇ f ( x ) , v ⟩ + t 2 v T H v / 2 q(t)=f(x)+t\langle\nabla f(x),v\rangle+t^{2}v^{\mathsf{T}}Hv/2 q ( t ) = f ( x ) + t ⟨ ∇ f ( x ) , v ⟩ + t 2 v T H v /2 . If the segment stays strictly below c c c , use the hypothesis. Otherwise let s ∈ ( 0 , 1 ] s\in(0,1] s ∈ ( 0 , 1 ] be its first exit time. The hypothesis at times t < s t<s t < s and continuity imply q ( s ) ≥ c q(s)\ge c q ( s ) ≥ c . Convexity gives q ( s ) ≤ ( 1 − s ) q ( 0 ) + s q ( 1 ) q(s)\le(1-s)q(0)+sq(1) q ( s ) ≤ ( 1 − s ) q ( 0 ) + s q ( 1 ) , hence q ( 1 ) ≥ c q(1)\ge c q ( 1 ) ≥ c . This proves the claim. At f ( x ) = c f(x)=c f ( x ) = c , a nonnegative directional derivative makes the assertion immediate. With a negative directional derivative, x + s v x+sv x + s v lies strictly below c c c for all sufficiently small positive s s s . Apply the proved claim there, with displacement ( 1 − s ) v (1-s)v ( 1 − s ) v , and take s ↓ 0 s\downarrow0 s ↓ 0 .
The full equivalence is in Lemma B.2. In particular, the hypothesis for B \mathcal{B} B is supplied by that lemma.
For an orthonormal d × k d\times k d × k matrix V V V , k ≥ 1 k\ge1 k ≥ 1 , define
b k = ε 8 L k , [ D V f ( x ) ] i = f ( x + b k v i ) − f ( x − b k v i ) 2 b k . (10) b_k=\frac{\varepsilon}{8L\sqrt{k}}, \qquad[D_Vf(x)]_i=\frac{f(x+b_kv_i)-f(x-b_kv_i)}{2b_k}. \tag*{(10)} b k = 8 L k ε , [ D V f ( x ) ] i = 2 b k f ( x + b k v i ) − f ( x − b k v i ) . (10)
It costs 2 k 2k 2 k values and has vector error at most ε / 16 \varepsilon/16 ε /16 . Indeed, the scalar error is bounded by ( 2 b k ) − 1 ∫ − b k b k L ∣ t ∣ d t = L b k / 2 (2b_k)^{-1}\int_{-b_k}^{b_k}L|t|\,dt=Lb_k/2 ( 2 b k ) − 1 ∫ − b k b k L ∣ t ∣ d t = L b k /2 ; summing its squares proves the assertion. For k = 0 k=0 k = 0 use the empty vector at zero cost. Global L L L -smoothness, not the sublevel matrix condition, controls these arbitrary finite probes.
Lemma B.4 (Restricted descent and a certified residual). Repeatedly compute z = D V f ( x ) z=D_Vf(x) z = D V f ( x ) , stop when ∥ z ∥ ≤ 7 ε / 16 \|z\|\le7\varepsilon/16 ∥ z ∥ ≤ 7 ε /16 , and otherwise set x ← x − V z / L x\leftarrow x-Vz/L x ← x − V z / L . Each nonterminal step decreases f f f by more than 35 ε 2 / ( 512 L ) 35\varepsilon^{2}/(512L) 35 ε 2 / ( 512 L ) . On return, ∥ V T ∇ f ( x ) ∥ ≤ ε / 2 \|V^{\mathsf{T}}\nabla f(x)\|\le\varepsilon/2 ∥ V T ∇ f ( x ) ∥ ≤ ε /2 . Next compute g = D I d f ( x ) g=D_{I_d}f(x) g = D I d f ( x ) . If ∥ g ∥ ≤ 15 ε / 16 \|g\|\le15\varepsilon/16 ∥ g ∥ ≤ 15 ε /16 , then ∥ ∇ f ( x ) ∥ ≤ ε \|\nabla f(x)\|\le\varepsilon ∥∇ f ( x ) ∥ ≤ ε . Otherwise
u = g − V V T g ∥ g − V V T g ∥ u=\frac{g-VV^{\mathsf{T}}g}{\|g-VV^{\mathsf{T}}g\|} u = ∥ g − V V T g ∥ g − V V T g is well defined, orthogonal to V V V , and ⟨ ∇ f ( x ) , u ⟩ > 11 ε / 16 \langle\nabla f(x),u\rangle>11\varepsilon/16 ⟨ ∇ f ( x ) , u ⟩ > 11 ε /16 .
Proof. Writing w = V T ∇ f ( x ) w=V^{\mathsf{T}}\nabla f(x) w = V T ∇ f ( x ) and e = z − w e=z-w e = z − w gives
f ( x − V z / L ) ≤ f ( x ) − ∥ w ∥ 2 2 L + ∥ e ∥ 2 2 L . f(x-Vz/L)\le f(x)-\frac{\|w\|^{2}}{2L}+\frac{\|e\|^{2}}{2L}. f ( x − V z / L ) ≤ f ( x ) − 2 L ∥ w ∥ 2 + 2 L ∥ e ∥ 2 .
At a nonterminal step ∥ w ∥ > 3 ε / 8 \|w\|>3\varepsilon/8 ∥ w ∥ > 3 ε /8 and ∥ e ∥ ≤ ε / 16 \|e\|\le\varepsilon/16 ∥ e ∥ ≤ ε /16 , giving 35 / 512 35/512 35/512 . The terminal assertion follows by the triangle inequality. At a failed full test, ∥ V T g ∥ ≤ 9 ε / 16 \|V^{\mathsf{T}}g\|\le9\varepsilon/16 ∥ V T g ∥ ≤ 9 ε /16 , whereas ∥ g ∥ > 15 ε / 16 \|g\|>15\varepsilon/16 ∥ g ∥ > 15 ε /16 . Pythagoras gives ∥ g − V V T g ∥ > 3 ε / 4 \|g-VV^{\mathsf{T}}g\|>3\varepsilon/4 ∥ g − V V T g ∥ > 3 ε /4 . Subtract the full estimation error ε / 16 \varepsilon/16 ε /16 to obtain the asserted slope. The successful full test also follows from that error bound.
A doubling line search
The search uses an Armijo-type sufficient decrease test [5 ] with doubling trial steps.
Lemma B.5 (Decrease along a new direction and amortized query cost). Suppose f ( x ) ≤ f ( 0 ) f(x) \le f(0) f ( x ) ≤ f ( 0 ) and u u u is a unit vector satisfying ⟨ ∇ f ( x ) , u ⟩ > 11 ε / 16 \langle\nabla f(x),u\rangle> 11\varepsilon/16 ⟨ ∇ f ( x ) , u ⟩ > 11 ε /16 . Query f ( x ) f(x) f ( x ) , and test t j = 2 j ε / ( 2 L ) t_j = 2^j\varepsilon/(2L) t j = 2 j ε / ( 2 L ) for j = 0 , 1 , … j = 0,1,\ldots j = 0 , 1 , … . A test passes precisely when
f ( x − t j u ) ≤ f ( x ) − ε t j / 4. (11) f(x-t_ju) \le f(x)-\varepsilon t_j/4. \tag*{(11)} f ( x − t j u ) ≤ f ( x ) − ε t j /4. (11) Stop at the first failure and return the preceding accepted point x + x^+ x + . For λ = u ⊤ H u \lambda= u^\top H u λ = u ⊤ H u one has λ > 0 \lambda> 0 λ > 0 and
D x : = f ( x ) − f ( x + ) > 7 ε 2 64 λ > ε 2 10 λ . (12) D_x := f(x)-f(x^+) > \frac{7\varepsilon^2}{64\lambda} > \frac{\varepsilon^2}{10\lambda}. \tag*{(12)} D x := f ( x ) − f ( x + ) > 64 λ 7 ε 2 > 10 λ ε 2 . (12) The search terminates. Its number of proposal evaluations, including the failed proposal but excluding its one center evaluation, is at most 8 L D x / ε 2 + 1 8LD_x/\varepsilon^2 + 1 8 L D x / ε 2 + 1 .
Proof. Apply (9) to − t u -tu − t u . If λ = 0 \lambda= 0 λ = 0 , the capped upper bound tends to − ∞ -\infty − ∞ , contradicting boundedness below. If 0 < t ≤ 7 ε / ( 8 λ ) 0 < t \le7\varepsilon/(8\lambda) 0 < t ≤ 7 ε / ( 8 λ ) , the bound is strictly below f ( x ) − ε t / 4 < f ( 0 ) f(x)-\varepsilon t/4 < f(0) f ( x ) − εt /4 < f ( 0 ) , so the actual value equals its cap and passes the test. Since λ ≤ L \lambda\le L λ ≤ L , the initial test passes. An accepted test must have t ≤ 4 ( f ( x ) − inf f ) / ε t \le4(f(x)-\inf f)/\varepsilon t ≤ 4 ( f ( x ) − inf f ) / ε , so a failure occurs at a finite index. If s s s was the preceding accepted step, the failed step 2 s 2s 2 s exceeds 7 ε / ( 8 λ ) 7\varepsilon/(8\lambda) 7 ε / ( 8 λ ) . Therefore D x ≥ ε s / 4 > 7 ε 2 / ( 64 λ ) D_x \ge\varepsilon s/4 > 7\varepsilon^2/(64\lambda) D x ≥ ε s /4 > 7 ε 2 / ( 64 λ ) . No monotonicity of test outcomes after the guaranteed initial interval is assumed.
Write s = 2 j ε / ( 2 L ) s = 2^j\varepsilon/(2L) s = 2 j ε / ( 2 L ) . The search used j + 2 j+2 j + 2 proposals, and D x ≥ 2 j ε 2 / ( 8 L ) D_x \ge2^j\varepsilon^2/(8L) D x ≥ 2 j ε 2 / ( 8 L ) . The elementary induction j + 2 ≤ 2 j + 1 j+2 \le2^j+1 j + 2 ≤ 2 j + 1 gives the last assertion.
Consequently, for n n n expansion moves along one monotone trajectory , all their proposal evaluations together number at most 8 T + n 8T+n 8 T + n . The n n n center evaluations are charged separately. There is no n log T n\log T n log T term.
The restarted algorithm and its cost
Algorithm 1 collects the steps of Lemmas B.4 and B.5 for a fixed capacity r ∈ { 1 , … , d } r \in\{1,\ldots,d\} r ∈ { 1 , … , d } . Within the current span V V V it descends by central differences; when no progress is certified there, it estimates the full gradient, returns if the estimate is small, and otherwise adds the normalized residual direction, moves along it by a doubling line search, and restarts with an empty span after r r r additions.
The test at t 0 t_0 t 0 always passes (Lemma B.5 ), so j ≥ 1 j \ge1 j ≥ 1 when the line search stops. Theorem B.1 uses r = ⌈ d E T / ( d + T ) ⌉ r = \lceil\sqrt{dET/(d+T)}\rceil r = ⌈ d E T / ( d + T ) ⌉ ; without E E E and H H H , Section B.4 runs copies with r = 1 , 2 , 4 , … r = 1,2,4,\ldots r = 1 , 2 , 4 , … under fixed budgets. Each branch uses exact function values and deterministic operations. A single fixed-r r r run does not need E E E , H H H , or Δ \Delta Δ . The basis dimension at the beginning of any restricted search is at most r − 1 r-1 r − 1 .
Lemma B.6 (A square budget across restarts). Suppose there are n = q r + s n = qr+s n = q r + s expansions in total, where q ≥ 0 q \ge0 q ≥ 0 and 0 ≤ s < r 0 \le s < r 0 ≤ s < r . Then
q r 2 + s 2 < 10 E T , n < 10 E T r + r 4 . (13) qr^2+s^2 < 10ET,\qquad n < \frac{10ET}{r}+\frac{r}{4}. \tag*{(13)} q r 2 + s 2 < 10 E T , n < r 10 E T + 4 r . (13) All restricted nonterminal updates together number at most J = ⌊ 512 T / 35 ⌋ J = \lfloor512T/35\rfloor J = ⌊ 512 T /35 ⌋ . The algorithm stops with a certified output.
Algorithm 1 Restarted subspace search with capacity r r r . For an orthonormal d × k d \times k d × k matrix V V V with columns v 1 , … , v k v_1,\ldots,v_k v 1 , … , v k , the vector D V f ( x ) D_V f(x) D V f ( x ) has entries [ f ( x + b k v i ) − f ( x − b k v i ) ] / ( 2 b k ) [f(x+b_kv_i)-f(x-b_kv_i)]/(2b_k) [ f ( x + b k v i ) − f ( x − b k v i )] / ( 2 b k ) with b k = ε / ( 8 L k ) b_k=\varepsilon/(8L\sqrt{k}) b k = ε / ( 8 L k ) , as in (10) , and is empty when k = 0 k=0 k = 0 ; D I d D_{I_d} D I d uses the coordinate vectors. Steps are t j = 2 j ε / ( 2 L ) t_j=2^j\varepsilon/(2L) t j = 2 j ε / ( 2 L ) .
Require: d d d , L L L , ε \varepsilon ε , a capacity r ∈ { 1 , … , d } r \in\{1,\ldots,d\} r ∈ { 1 , … , d } , and exact values of f f f
x ← 0 x \leftarrow0 x ← 0 , and let V V V be the empty d × 0 d \times0 d × 0 matrix
loop
z ← D V f ( x ) z \leftarrow D_V f(x) z ← D V f ( x )
while ∥ z ∥ > 7 ε / 16 \lVert z\rVert> 7\varepsilon/16 ∥ z ∥ > 7 ε /16 do descent within the span
x ← x − V z / L , z ← D V f ( x ) x \leftarrow x-Vz/L,\ z \leftarrow D_V f(x) x ← x − V z / L , z ← D V f ( x )
end while
g ← D I d f ( x ) g \leftarrow D_{I_d}f(x) g ← D I d f ( x ) full estimate, 2 d 2d 2 d values
if ∥ g ∥ ≤ 15 ε / 16 \lVert g\rVert\le15\varepsilon/16 ∥ g ∥ ≤ 15 ε /16 then
return x x x
end if
u ← ( g − V V T g ) / ∥ g − V V T g ∥ u \leftarrow(g-VV^{\mathsf T}g)/\lVert g-VV^{\mathsf T}g\rVert u ← ( g − V V T g ) / ∥ g − V V T g ∥ new direction
j ← 0 j \leftarrow0 j ← 0
while f ( x − t j u ) ≤ f ( x ) − ε t j / 4 f(x-t_ju)\le f(x)-\varepsilon t_j/4 f ( x − t j u ) ≤ f ( x ) − ε t j /4 do doubling line search
j ← j + 1 j \leftarrow j+1 j ← j + 1
end while
x ← x − t j − 1 u , V ← [ V u ] x \leftarrow x-t_{j-1}u,\ V \leftarrow[V\ u] x ← x − t j − 1 u , V ← [ V u ]
if V V V has r r r columns then
let V V V be empty restart
end if
end loop
Proof. In a restart block containing k k k expansions, the appended vectors are orthonormal. For their positive curvatures λ i = u i T H u i \lambda_i=u_i^{\mathsf T}Hu_i λ i = u i T H u i ,
∑ i = 1 k λ i ≤ L E , ∑ i = 1 k D i > ε 2 10 ∑ i = 1 k λ i − 1 ≥ ε 2 k 2 10 L E . \sum_{i=1}^{k}\lambda_i \le LE,\qquad\sum_{i=1}^{k}D_i>\frac{\varepsilon^2}{10}\sum_{i=1}^{k}\lambda_i^{-1}\ge\frac{\varepsilon^2k^2}{10LE}. i = 1 ∑ k λ i ≤ L E , i = 1 ∑ k D i > 10 ε 2 i = 1 ∑ k λ i − 1 ≥ 10 L E ε 2 k 2 .
The last inequality is Cauchy–Schwarz. Crucially, this trace sum is used within each block only ; vectors from different blocks need not be orthogonal. Their actual decreases are disjoint parts of one monotone trajectory, so summing the block inequalities gives q r 2 + s 2 < 10 E T qr^2+s^2<10ET q r 2 + s 2 < 10 E T . Now
n = q r 2 + s 2 r + s ( 1 − s / r ) < 10 E T r + r 4 , n=\frac{qr^2+s^2}{r}+s(1-s/r)<\frac{10ET}{r}+\frac{r}{4}, n = r q r 2 + s 2 + s ( 1 − s / r ) < r 10 E T + 4 r ,
since ( s − r / 2 ) 2 ≥ 0 (s-r/2)^2\ge0 ( s − r /2 ) 2 ≥ 0 .
Lemma B.4 and the same monotone gap budget bound the restricted updates by J J J . Every line search is finite. An infinite run would therefore require infinitely many expansions, hence infinitely many complete blocks, contradicting their positive uniform gap charge. The only return is the certified return after a full gradient estimate.
Algorithm 2 Coordinate finite differences. The estimate uses central differences along the coordinate vectors e 1 , … , e d e_1,\ldots,e_d e 1 , … , e d with radius b = ε / ( 4 L d ) b=\varepsilon/(4L\sqrt{d}) b = ε / ( 4 L d ) .
Require: d d d , L L L , ε \varepsilon ε , and exact values of f f f
x ← 0 x \leftarrow0 x ← 0
loop
g ← ( [ f ( x + b e a ) − f ( x − b e a ) ] / ( 2 b ) ) a = 1 d g \leftarrow\left(\left[f(x+be_a)-f(x-be_a)\right]/(2b)\right)_{a=1}^{d} g ← ( [ f ( x + b e a ) − f ( x − b e a ) ] / ( 2 b ) ) a = 1 d \hfill 2 d 2d 2 d values
if ∥ g ∥ ≤ 7 ε / 8 \lVert g\rVert\le7\varepsilon/8 ∥ g ∥ ≤ 7 ε /8 then
return x x x
end if
x ← x − g / L x \leftarrow x-g/L x ← x − g / L
end loop
Theorem B.7 (Cost at a fixed capacity). For any r ∈ { 1 , … , d } r\in\{1,\ldots,d\} r ∈ { 1 , … , d } the preceding deterministic algorithm has the worst-case query count
Q r ≤ 2 ( d + r − 1 ) ( n + 1 ) + 2 ( r − 1 ) J + 8 T + 2 n , (14) Q_r \le2(d+r-1)(n+1)+2(r-1)J+8T+2n, \tag*{(14)} Q r ≤ 2 ( d + r − 1 ) ( n + 1 ) + 2 ( r − 1 ) J + 8 T + 2 n , (14) Q r < 40 d E T r + 30 ( d + T ) r . (15) Q_r < \frac{40dET}{r}+30(d+T)r. \tag*{(15)} Q r < r 40 d E T + 30 ( d + T ) r . (15) Its stored basis uses at most d r dr d r real entries.
Proof. There are n + 1 n+1 n + 1 full estimates at cost 2 d 2d 2 d each and at most n + 1 n+1 n + 1 terminal restricted estimates at cost 2 ( r − 1 ) 2(r-1) 2 ( r − 1 ) each. The J J J nonterminal restricted estimates also cost at most 2 ( r − 1 ) 2(r-1) 2 ( r − 1 ) each. The preceding line search amortization charges at most 8 T + n 8T+n 8 T + n proposals and n n n centers. This proves (14) ; in particular it charges the final tests and every rejected proposal.
Because r ≤ d r\le d r ≤ d , (14) and (13) yield
Q r < 40 d E T r + d r + 4 d + 1024 35 ( r − 1 ) T + 8 T . Q_r < \frac{40dET}{r}+dr+4d+\frac{1024}{35}(r-1)T+8T. Q r < r 40 d E T + d r + 4 d + 35 1024 ( r − 1 ) T + 8 T .
Use d r + 4 d ≤ 5 d r dr+4d\le5dr d r + 4 d ≤ 5 d r , and ( 1024 / 35 ) ( r − 1 ) T + 8 T ≤ ( 1024 / 35 ) r T < 30 r T (1024/35)(r-1)T+8T\le(1024/35)rT<30rT ( 1024/35 ) ( r − 1 ) T + 8 T ≤ ( 1024/35 ) r T < 30 r T . The resulting expression is smaller than (15) . □
Proof of (6) . Set A = d E T / ( d + T ) A=dET/(d+T) A = d E T / ( d + T ) and Z = d E T ( d + T ) Z=\sqrt{dET(d+T)} Z = d E T ( d + T ) . One has A ≥ 1 / 2 A\ge1/2 A ≥ 1/2 and A < d 2 A<d^2 A < d 2 , so r = ⌈ A ⌉ r=\lceil\sqrt{A}\rceil r = ⌈ A ⌉ lies in { 1 , … , d } \{1,\ldots,d\} { 1 , … , d } and
A ≤ r ≤ 2 A . \sqrt{A}\le r\le2\sqrt{A}. A ≤ r ≤ 2 A .
For A ≤ 1 \sqrt{A}\le1 A ≤ 1 the latter follows from A ≥ 1 / 2 A\ge1/2 A ≥ 1/2 ; for A > 1 \sqrt{A}>1 A > 1 it follows from ⌈ A ⌉ ≤ A + 1 \lceil\sqrt{A}\rceil\le\sqrt{A}+1 ⌈ A ⌉ ≤ A + 1 . Equation (15) then gives Q r < 40 Z + 60 Z = 100 Z Q_r<40Z+60Z=100Z Q r < 40 Z + 60 Z = 100 Z .
For completeness, the ordinary coordinate algorithm (Algorithm 2) uses difference radius ε / ( 4 L d ) \varepsilon/(4L\sqrt{d}) ε / ( 4 L d ) , vector error ε / 8 \varepsilon/8 ε /8 , stopping threshold 7 ε / 8 7\varepsilon/8 7 ε /8 , and update x − g / L x-g/L x − g / L otherwise. Every nonterminal step decreases f f f by more than 35 ε 2 / ( 128 L ) 35\varepsilon^2/(128L) 35 ε 2 / ( 128 L ) . Including the final estimate costs at most 2 d ( 1 + ⌊ 128 T / 35 ⌋ ) ≤ 10 d T 2d(1+\lfloor128T/35\rfloor)\le10dT 2 d ( 1 + ⌊ 128 T /35 ⌋) ≤ 10 d T for T ≥ 1 T\ge1 T ≥ 1 . Selecting the algorithm with the smaller public bound proves the minimum. If an exact comparison of bounds is inconvenient, interleave the two correct procedures one charged query at a time; this loses at most a factor two, not correctness. □
Algorithm 3 Subspace search without knowledge of $E$ or $H$. Each trial is a fresh run of Algorithm 1 from the origin, stopped after its budget; an unfinished trial is discarded.
Require: $d$, $L$, $\Delta$, $\varepsilon$, and exact values of $f$; $T = L\Delta/\varepsilon^2$
1: procedure Trials
2: for $j = 0, 1, \ldots, \lceil\log_2 d\rceil$ do
3: $r_j \leftarrow \min\{2^j,d\}$, $B_j \leftarrow \lfloor 70(d+T)r_j\rfloor$
4: run Algorithm 1 with capacity $r_j$ for at most $B_j$ value calls
5: if that run returned a point $x$ then
6: return $x$
7: end if
8: end for
9: end procedure
10: Run Trials and Algorithm 2 in alternation, one value call from each at a time with separate states, and return the first output.
Removing knowledge of the trace
Algorithm 3 combines restarted trials of Algorithm 1 with growing capacity and the coordinate algorithm; it reads neither E E E nor H H H .
Proof of (7) . Use capacities r j = min { 2 j , d } r_j=\min\{2^j,d\} r j = min { 2 j , d } for j = 0 , … , ⌈ log 2 d ⌉ j=0,\ldots,\lceil\log_2 d\rceil j = 0 , … , ⌈ log 2 d ⌉ . Run a fresh copy of the fixed-r j r_j r j procedure from the origin, capped at
B j = ⌊ 70 ( d + T ) r j ⌋ B_j=\lfloor70(d+T)r_j\rfloor B j = ⌊ 70 ( d + T ) r j ⌋
actual value evaluations. A completed certification at the budget is valid; otherwise discard an unfinished computation, increase the capacity, and restart. No trial reads E E E or H H H .
Define A , Z A,Z A , Z as above only for the analysis, and let j ∗ j_* j ∗ be the first index with r j ∗ ≥ A r_{j_*}\ge\sqrt{A} r j ∗ ≥ A . It exists because A < d \sqrt{A}<d A < d . For that trial (15) gives the strict inequality Q r j ∗ < 70 ( d + T ) r j ∗ Q_{r_{j_*}}<70(d+T)r_{j_*} Q r j ∗ < 70 ( d + T ) r j ∗ , and since Q r j ∗ Q_{r_{j_*}} Q r j ∗ is an integer we have Q r j ∗ ≤ B j ∗ Q_{r_{j_*}}\le B_{j_*} Q r j ∗ ≤ B j ∗ . It therefore completes. If j ∗ = 0 j_*=0 j ∗ = 0 , then ∑ j = 0 j ∗ r j = 1 ≤ 2 A < 4 A \sum_{j=0}^{j_*}r_j=1\le\sqrt{2A}<4\sqrt{A} ∑ j = 0 j ∗ r j = 1 ≤ 2 A < 4 A ; if j ∗ ≥ 1 j_*\ge1 j ∗ ≥ 1 , then 2 j ∗ − 1 = r j ∗ − 1 < A 2^{j_*-1}=r_{j_*-1}<\sqrt{A} 2 j ∗ − 1 = r j ∗ − 1 < A , so ∑ j = 0 j ∗ − 1 r j = 2 j ∗ − 1 < 2 A \sum_{j=0}^{j_*-1}r_j=2^{j_*}-1<2\sqrt{A} ∑ j = 0 j ∗ − 1 r j = 2 j ∗ − 1 < 2 A and r j ∗ ≤ 2 j ∗ < 2 A r_{j_*}\le2^{j_*}<2\sqrt{A} r j ∗ ≤ 2 j ∗ < 2 A , giving ∑ j = 0 j ∗ r j < 4 A \sum_{j=0}^{j_*}r_j<4\sqrt{A} ∑ j = 0 j ∗ r j < 4 A even when the final capacity is shortened to d d d . The total is less than 280 ( d + T ) A = 280 Z 280(d+T)\sqrt{A}=280Z 280 ( d + T ) A = 280 Z .
Finally interleave this entire procedure, which needs no knowledge of the trace, and the coordinate algorithm, keeping their states separate, one actual value call from each in alternation. Stop when either returns. Both returns are certified, and the cost is at most twice the smaller individual cost. This gives (7) . The restarting trials use separate gap budgets; only their already bounded oracle costs are added.
The three-point method on the main class
Algorithm 4 states the three-point method with step h h h and N N N iterations; Theorem B.8 below and Theorem G.3 in Appendix G fix these parameters.
Theorem B.8 (An elementary three-point bound on B \mathcal{B} B ). For f ∈ B f\in\mathcal{B} f ∈ B , take h = 4 ε / ( 5 L E ) h=4\varepsilon/(5LE) h = 4 ε / ( 5 L E ) and N = ⌈ 10 E T ⌉ N=\lceil10ET\rceil N = ⌈ 10 E T ⌉ . Query f ( 0 ) f(0) f ( 0 ) once. For t = 0 , … , N − 1 t=0,\ldots,N-1 t = 0 , … , N − 1 , draw an independent a t ∼ N ( 0 , I d ) a_t\sim\mathcal{N}(0,I_d) a t ∼ N ( 0 , I d ) and keep a point of lowest value among x t x_t x t , x t + h a t x_t+ha_t x t + h a t , x t − h a t x_t-ha_t x t − h a t , caching its value and retaining the center on a tie. Return an independently uniform point from x 0 , … , x N − 1 x_0,\ldots,x_{N-1} x 0 , … , x N − 1 . Then
P ( ∥ ∇ f ( x ) ∥ > ε ) < 25 / 78 < 1 / 3 , Q = 1 + 2 ⌈ 10 E T ⌉ ≤ 23 E T . \mathbb{P}(\lVert\nabla f(x)\rVert>\varepsilon)<25/78<1/3,\qquad Q=1+2\lceil10ET\rceil\le23ET. P (∥ ∇ f ( x )∥ > ε ) < 25/78 < 1/3 , Q = 1 + 2 ⌈ 10 E T ⌉ ≤ 23 E T .
Algorithm 4 Three-point method with step h h h and N N N iterations. The comparison step is the update of Bergou et al. [9 ] . On B \mathcal{B} B , h = 4 ε / ( 5 L E ) h = 4\varepsilon/(5LE) h = 4 ε / ( 5 L E ) and N = ⌈ 10 E T ⌉ N = \lceil10ET\rceil N = ⌈ 10 E T ⌉ (Theorem B.8 ); on A \mathcal{A} A , h = 4 ε / ( 5 L ( E + 1 ) ) h = 4\varepsilon/(5L(E + 1)) h = 4 ε / ( 5 L ( E + 1 )) and N = ⌈ 10 L ( E + 1 ) Δ / ε 2 ⌉ N = \lceil10L(E + 1)\Delta/\varepsilon^2\rceil N = ⌈ 10 L ( E + 1 ) Δ/ ε 2 ⌉ (Theorem G.3 ).
Require: h h h , N N N , and exact values of f f f
x 0 ← 0 x_0 \leftarrow0 x 0 ← 0 ; query f ( x 0 ) f(x_0) f ( x 0 )
for t = 0 , … , N − 1 t = 0,\ldots,N-1 t = 0 , … , N − 1 do
draw a t ∼ N ( 0 , I d ) a_t \sim N(0,I_d) a t ∼ N ( 0 , I d ) independently; query f ( x t + h a t ) f(x_t+ha_t) f ( x t + h a t ) and f ( x t − h a t ) f(x_t-ha_t) f ( x t − h a t )
x t + 1 ← x_{t+1} \leftarrow x t + 1 ← a point of lowest value among x t x_t x t , x t + h a t x_t+ha_t x t + h a t , x t − h a t x_t-ha_t x t − h a t , keeping x t x_t x t on a tie
end for
draw J J J uniformly from { 0 , … , N − 1 } \{0,\ldots,N-1\} { 0 , … , N − 1 } , independently, and return x J x_J x J
This is the update of Bergou et al. [9 ] ; on B \mathcal{B} B its analysis needs only Lemma B.3 and two Gaussian moments.
Proof. At a base point x x x with f ( x ) ≤ f ( 0 ) f(x) \le f(0) f ( x ) ≤ f ( 0 ) , write g = ∇ f ( x ) g = \nabla f(x) g = ∇ f ( x ) . Of the two proposals, the sign − sign ( a ⊤ g ) -\operatorname{sign}(a^\top g) − sign ( a ⊤ g ) has capped quadratic bound f ( x ) − h ∣ a ⊤ g ∣ + ( h 2 / 2 ) a ⊤ H a f(x) - h|a^\top g| + (h^2/2)a^\top Ha f ( x ) − h ∣ a ⊤ g ∣ + ( h 2 /2 ) a ⊤ H a . The true three-point minimum is no larger than this expression: if the expression is below f ( x ) f(x) f ( x ) , Lemma B.3 forces an actual proposal below f ( x ) f(x) f ( x ) ; otherwise the retained center already suffices. Consequently
E [ f ( x ) − f ( x + ) ∣ x ] ≥ h 2 / π ∥ g ∥ − ( h 2 / 2 ) tr H . \mathbb{E}[f(x)-f(x^+)\mid x] \ge h\sqrt{2/\pi}\,\lVert g\rVert- (h^2/2)\operatorname{tr}H. E [ f ( x ) − f ( x + ) ∣ x ] ≥ h 2/ π ∥ g ∥ − ( h 2 /2 ) tr H .
It is also nonnegative because each realized update is monotone. For ∥ g ∥ > ε \lVert g\rVert> \varepsilon ∥ g ∥ > ε , use 2 / π > 79 / 100 \sqrt{2/\pi} > 79/100 2/ π > 79/100 (for example from π < 22 / 7 \pi< 22/7 π < 22/7 ) to lower bound the decrease by 39 ε 2 / ( 125 L E ) 39\varepsilon^2/(125LE) 39 ε 2 / ( 125 L E ) . Sum over all steps: the expected number of indices with gradient above ε \varepsilon ε is less than 125 E T / 39 125ET/39 125 E T /39 . Uniform output and N ≥ 10 E T N \ge10ET N ≥ 10 E T give 25 / 78 25/78 25/78 . Every proposal, accepted or rejected, costs one query, while center values are cached. Only Gaussian first and second moments have been used. □
A monotone two-point method on the main class
The three-point method moves by a fixed length proportional to ε \varepsilon ε . The Gaussian two-point method of Nesterov and Spokoiny [41 ] instead scales each step by an estimated directional derivative, but a step may leave the initial sublevel set, where B \mathcal{B} B controls only the gradient Lipschitz constant. Accepting a proposal only when it lowers f f f keeps every iterate below f ( 0 ) f(0) f ( 0 ) , so Lemma B.3 again controls every step. Algorithm 5 states the method with the parameters of Theorem B.10 .
Lemma B.9 (Gaussian moments). Let a ∼ N ( 0 , I d ) a \sim N(0,I_d) a ∼ N ( 0 , I d ) , g ∈ R d g \in\mathbb{R}^d g ∈ R d , and let H H H be symmetric. Then
E [ ( g ⊤ a ) 2 ( a ⊤ H a ) ] = ∥ g ∥ 2 tr H + 2 g ⊤ H g , \mathbb{E}[(g^\top a)^2(a^\top Ha)] = \lVert g\rVert^2\operatorname{tr}H + 2g^\top Hg, E [( g ⊤ a ) 2 ( a ⊤ H a )] = ∥ g ∥ 2 tr H + 2 g ⊤ H g , and
E ∥ a ∥ 2 k = d ( d + 2 ) ⋯ ( d + 2 k − 2 ) \mathbb{E}\lVert a\rVert^{2k} = d(d+2)\cdots(d+2k-2) E ∥ a ∥ 2 k = d ( d + 2 ) ⋯ ( d + 2 k − 2 ) for k = 1 , 2 , 3 k = 1,2,3 k = 1 , 2 , 3 .
Proof. For g ≠ 0 g \ne0 g = 0 , take an orthonormal basis v 1 , … , v d v_1,\ldots,v_d v 1 , … , v d with v 1 = g / ∥ g ∥ v_1 = g/\lVert g\rVert v 1 = g / ∥ g ∥ and let z i = v i ⊤ a z_i = v_i^\top a z i = v i ⊤ a , which are independent standard normal variables. Then ( g ⊤ a ) 2 = ∥ g ∥ 2 z 1 2 (g^\top a)^2 = \lVert g\rVert^2z_1^2 ( g ⊤ a ) 2 = ∥ g ∥ 2 z 1 2 , and E [ z 1 4 ] = 3 \mathbb{E}[z_1^4] = 3 E [ z 1 4 ] = 3 , E [ z i 2 ] = 1 \mathbb{E}[z_i^2] = 1 E [ z i 2 ] = 1 for i ≥ 2 i \ge2 i ≥ 2 , and E [ z i 2 z j 2 ] = 0 \mathbb{E}[z_i^2z_j^2] = 0 E [ z i 2 z j 2 ] = 0 for i ≠ j i \ne j i = j give ∥ g ∥ 2 ( 3 v 1 ⊤ H v 1 + ∑ i ≥ 2 v i ⊤ H v i ) \lVert g\rVert^2(3v_1^\top Hv_1 + \sum_{i\ge2}v_i^\top Hv_i) ∥ g ∥ 2 ( 3 v 1 ⊤ H v 1 + ∑ i ≥ 2 v i ⊤ H v i ) . The moments of ∥ a ∥ 2 ∼ χ d 2 \lVert a\rVert^2 \sim\chi_d^2 ∥ a ∥ 2 ∼ χ d 2 are standard. □
Theorem B.10 (A monotone two-point bound on B \mathcal{B} B ). For f ∈ B f \in\mathcal{B} f ∈ B , set
η = 1 L ( E + 2 ) , N = ⌈ ( 6 E + 13 ) T ⌉ , μ = ε 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 ) . \eta= \frac{1}{L(E+2)}, \qquad N = \lceil(6E+13)T\rceil, \qquad\mu= \frac{\varepsilon}{4L(6E+13)\sqrt{d(d+2)(d+4)}}. η = L ( E + 2 ) 1 , N = ⌈( 6 E + 13 ) T ⌉ , μ = 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 ) ε . Query f ( 0 ) f(0) f ( 0 ) once. For t = 0 , … , N − 1 t=0,\ldots,N-1 t = 0 , … , N − 1 , draw an independent a t ∼ N ( 0 , I d ) a_t \sim N(0,I_d) a t ∼ N ( 0 , I d ) , query f ( x t + μ a t ) f(x_t+\mu a_t) f ( x t + μ a t ) , and with the cached value f ( x t ) f(x_t) f ( x t ) set s t = [ f ( x t + μ a t ) − f ( x t ) ] / μ s_t=[f(x_t+\mu a_t)-f(x_t)]/\mu s t = [ f ( x t + μ a t ) − f ( x t )] / μ and y t = x t − η s t a t y_t=x_t-\eta s_ta_t y t = x t − η s t a t . Query f ( y t ) f(y_t) f ( y t ) , and
Algorithm 5 Monotone two-point method, with η = 1 / ( L ( E + 2 ) ) \eta= 1/(L(E + 2)) η = 1/ ( L ( E + 2 )) , N = ⌈ ( 6 E + 13 ) T ⌉ N = \lceil(6E + 13)T\rceil N = ⌈( 6 E + 13 ) T ⌉ , and μ = ε / ( 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 ) ) \mu= \varepsilon/(4L(6E + 13)\sqrt{d}(d + 2)(d + 4)) μ = ε / ( 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 )) as in Theorem B.10 .
Require: η \eta η , N N N , μ \mu μ , and exact values of f f f
x 0 ← 0 x_0 \leftarrow0 x 0 ← 0 ; query f ( x 0 ) f(x_0) f ( x 0 )
for t = 0 , … , N − 1 t = 0, \ldots, N - 1 t = 0 , … , N − 1 do
draw a t ∼ N ( 0 , I d ) a_t \sim N(0, I_d) a t ∼ N ( 0 , I d ) independently; query f ( x t + μ a t ) f(x_t + \mu a_t) f ( x t + μ a t )
s t ← [ f ( x t + μ a t ) − f ( x t ) ] / μ s_t \leftarrow[f(x_t + \mu a_t) - f(x_t)]/\mu s t ← [ f ( x t + μ a t ) − f ( x t )] / μ \hfill f ( x t ) f(x_t) f ( x t ) is cached
y t ← x t − η s t a t y_t \leftarrow x_t - \eta s_t a_t y t ← x t − η s t a t ; query f ( y t ) f(y_t) f ( y t )
if f ( y t ) < f ( x t ) f(y_t) < f(x_t) f ( y t ) < f ( x t ) then
x t + 1 ← y t x_{t+1} \leftarrow y_t x t + 1 ← y t
else
x t + 1 ← x t x_{t+1} \leftarrow x_t x t + 1 ← x t
end if
end for
draw J J J uniformly from { 0 , … , N − 1 } \{0, \ldots, N - 1\} { 0 , … , N − 1 } , independently, and return x J x_J x J
set x t + 1 = y t x_{t+1} = y_t x t + 1 = y t if f ( y t ) < f ( x t ) f(y_t) < f(x_t) f ( y t ) < f ( x t ) and x t + 1 = x t x_{t+1} = x_t x t + 1 = x t otherwise. Return an independently uniform point from x 0 , … , x N − 1 x_0, \ldots, x_{N-1} x 0 , … , x N − 1 . Then
P ( ∥ ∇ f ( x ^ ) ∥ > ε ) < 1 / 3 , Q = 1 + 2 ⌈ ( 6 E + 13 ) T ⌉ ≤ 12 E T + 26 T + 3. \mathbb{P}(\lVert\nabla f(\hat{x})\rVert> \varepsilon) < 1/3,\qquad Q = 1 + 2\lceil(6E + 13)T\rceil\le12ET + 26T + 3. P (∥ ∇ f ( x ^ )∥ > ε ) < 1/3 , Q = 1 + 2 ⌈( 6 E + 13 ) T ⌉ ≤ 12 E T + 26 T + 3.
Proof. The acceptance rule keeps f ( x t ) ≤ f ( 0 ) f(x_t) \le f(0) f ( x t ) ≤ f ( 0 ) . At a base point x x x with f ( x ) ≤ f ( 0 ) f(x) \le f(0) f ( x ) ≤ f ( 0 ) , write g = ∇ f ( x ) g = \nabla f(x) g = ∇ f ( x ) , a = a t a = a_t a = a t , s = s t s = s_t s = s t , M = L ( E + 2 ) = 1 / η M = L(E + 2) = 1/\eta M = L ( E + 2 ) = 1/ η , and x + x^+ x + for the next iterate. Since f ( x + ) = min { f ( x ) , f ( y t ) } ≤ min { f ( y t ) , f ( 0 ) } f(x^+) = \min\{f(x), f(y_t)\} \le\min\{f(y_t), f(0)\} f ( x + ) = min { f ( x ) , f ( y t )} ≤ min { f ( y t ) , f ( 0 )} , Lemma B.3 with displacement − η s a -\eta sa − η s a gives
f ( x ) − f ( x + ) ≥ η s ( g ⊤ a ) − η 2 2 s 2 ( a ⊤ H a ) . f(x) - f(x^+) \ge\eta s(g^\top a) - \frac{\eta^2}{2}s^2(a^\top Ha). f ( x ) − f ( x + ) ≥ η s ( g ⊤ a ) − 2 η 2 s 2 ( a ⊤ H a ) .
Global L L L -smoothness bounds the difference error δ = s − g ⊤ a \delta= s - g^\top a δ = s − g ⊤ a by ∣ δ ∣ ≤ L μ ∥ a ∥ 2 / 2 |\delta| \le L\mu\lVert a\rVert^2/2 ∣ δ ∣ ≤ Lμ ∥ a ∥ 2 /2 . With γ = 1 / ( 4 ( 6 E + 13 ) ) \gamma= 1/(4(6E + 13)) γ = 1/ ( 4 ( 6 E + 13 )) , Young’s inequality gives s ( g ⊤ a ) ≥ ( 1 − γ / 2 ) ( g ⊤ a ) 2 − δ 2 / ( 2 γ ) s(g^\top a) \ge(1-\gamma/2)(g^\top a)^2-\delta^2/(2\gamma) s ( g ⊤ a ) ≥ ( 1 − γ /2 ) ( g ⊤ a ) 2 − δ 2 / ( 2 γ ) and s 2 ≤ ( 1 + γ ) ( g ⊤ a ) 2 + ( 1 + 1 / γ ) δ 2 s^2 \le(1+\gamma)(g^\top a)^2+(1+1/\gamma)\delta^2 s 2 ≤ ( 1 + γ ) ( g ⊤ a ) 2 + ( 1 + 1/ γ ) δ 2 . Taking expectations and using Lemma B.9 with tr H ≤ L E \operatorname{tr} H \le LE tr H ≤ L E and g ⊤ H g ≤ L ∥ g ∥ 2 g^\top Hg \le L\lVert g\rVert^2 g ⊤ H g ≤ L ∥ g ∥ 2 ,
E [ f ( x ) − f ( x + ) ∣ x ] ≥ 1 − 2 γ 2 M ∥ g ∥ 2 − 1 2 γ M ( E [ δ 2 ] + 1 + γ M E [ δ 2 ( a ⊤ H a ) ] ) . \mathbb{E}[f(x) - f(x^+) \mid x] \ge\frac{1 - 2\gamma}{2M}\lVert g\rVert^2 - \frac{1}{2\gamma M}\left(\mathbb{E}[\delta^2] + \frac{1+\gamma}{M}\mathbb{E}[\delta^2(a^\top Ha)]\right). E [ f ( x ) − f ( x + ) ∣ x ] ≥ 2 M 1 − 2 γ ∥ g ∥ 2 − 2 γ M 1 ( E [ δ 2 ] + M 1 + γ E [ δ 2 ( a ⊤ H a )] ) .
By Lemma B.9 and a ⊤ H a ≤ L ∥ a ∥ 2 a^\top Ha \le L\lVert a\rVert^2 a ⊤ H a ≤ L ∥ a ∥ 2 , E [ δ 2 ] ≤ L 2 μ 2 d ( d + 2 ) / 4 < ε 2 / ( 64 ( 6 E + 13 ) 2 ) \mathbb{E}[\delta^2] \le L^2\mu^2d(d+2)/4 < \varepsilon^2/(64(6E+13)^2) E [ δ 2 ] ≤ L 2 μ 2 d ( d + 2 ) /4 < ε 2 / ( 64 ( 6 E + 13 ) 2 ) and E [ δ 2 ( a ⊤ H a ) ] ≤ L 3 μ 2 d ( d + 2 ) ( d + 4 ) / 4 = L ε 2 / ( 64 ( 6 E + 13 ) 2 ) \mathbb{E}[\delta^2(a^\top Ha)] \le L^3\mu^2d(d+2)(d+4)/4 = L\varepsilon^2/(64(6E+13)^2) E [ δ 2 ( a ⊤ H a )] ≤ L 3 μ 2 d ( d + 2 ) ( d + 4 ) /4 = L ε 2 / ( 64 ( 6 E + 13 ) 2 ) . Since L / M ≤ 1 / 3 L/M \le1/3 L / M ≤ 1/3 and γ ≤ 1 / 76 \gamma\le1/76 γ ≤ 1/76 , the term in parentheses is below 2 ε 2 / ( 64 ( 6 E + 13 ) 2 ) 2\varepsilon^2/(64(6E+13)^2) 2 ε 2 / ( 64 ( 6 E + 13 ) 2 ) , so the subtracted term is below ε 2 / ( 16 M ( 6 E + 13 ) ) \varepsilon^2/(16M(6E+13)) ε 2 / ( 16 M ( 6 E + 13 )) . For ∥ g ∥ > ε \lVert g\rVert> \varepsilon ∥ g ∥ > ε , as 2 γ = 1 / ( 2 ( 6 E + 13 ) ) 2\gamma= 1/(2(6E + 13)) 2 γ = 1/ ( 2 ( 6 E + 13 )) , this gives
E [ f ( x ) − f ( x + ) ∣ x ] > ( 1 − 1 6 E + 13 ) ε 2 2 M = 3 ε 2 L ( 6 E + 13 ) . \mathbb{E}[f(x) - f(x^+) \mid x] > \left(1-\frac{1}{6E+13}\right)\frac{\varepsilon^2}{2M} = \frac{3\varepsilon^2}{L(6E+13)}. E [ f ( x ) − f ( x + ) ∣ x ] > ( 1 − 6 E + 13 1 ) 2 M ε 2 = L ( 6 E + 13 ) 3 ε 2 .
and every realized update is nonnegative. Summing over all steps and using f ( 0 ) − inf f ≤ Δ f(0) - \inf f \le\Delta f ( 0 ) − inf f ≤ Δ , the expected number of indices with gradient norm above ε \varepsilon ε is less than ( 6 E + 13 ) T / 3 ≤ N / 3 (6E + 13)T/3 \le N/3 ( 6 E + 13 ) T /3 ≤ N /3 , which gives the failure bound under uniform output. Each step queries f ( x t + μ a t ) f(x_t + \mu a_t) f ( x t + μ a t ) and f ( y t ) f(y_t) f ( y t ) and reuses the cached value f ( x t ) f(x_t) f ( x t ) .
Gaussian two-point methods without an acceptance test
We now analyze the Gaussian two-point method of Nesterov and Spokoiny [41 ] , the estimator behind zeroth-order fine-tuning (Malladi et al., [38 ] ), with q q q directions per step and no acceptance Algorithm 6 Gaussian two-point method with q q q directions per step and no acceptance test. Theorem B.12 gives η , μ , N \eta,\mu,N η , μ , N on F \mathcal{F} F , and Theorem B.15 gives them on B \mathcal{B} B .
Require: q q q , η \eta η , μ \mu μ , N N N , the choice of forward or central differences, and exact values of f f f
x 0 ← 0 x_0 \leftarrow0 x 0 ← 0
for t = 0 , … , N − 1 t=0,\ldots,N-1 t = 0 , … , N − 1 do
draw a t , 1 , … , a t , q ∼ N ( 0 , I d ) a_{t,1},\ldots,a_{t,q}\sim N(0,I_d) a t , 1 , … , a t , q ∼ N ( 0 , I d ) independently
for i = 1 , … , q i=1,\ldots,q i = 1 , … , q do
s t , i ← [ f ( x t + μ a t , i ) − f ( x t ) ] / μ s_{t,i}\leftarrow[f(x_t+\mu a_{t,i})-f(x_t)]/\mu s t , i ← [ f ( x t + μ a t , i ) − f ( x t )] / μ (forward) or [ f ( x t + μ a t , i ) − f ( x t − μ a t , i ) ] / ( 2 μ ) [f(x_t+\mu a_{t,i})-f(x_t-\mu a_{t,i})]/(2\mu) [ f ( x t + μ a t , i ) − f ( x t − μ a t , i )] / ( 2 μ ) (central)
end for
x t + 1 ← x t − η q ∑ i = 1 q s t , i a t , i x_{t+1}\leftarrow x_t-\dfrac{\eta}{q}\sum_{i=1}^{q}s_{t,i}a_{t,i} x t + 1 ← x t − q η ∑ i = 1 q s t , i a t , i
end for
draw J J J uniformly from { 0 , … , N − 1 } \{0,\ldots,N-1\} { 0 , … , N − 1 } , independently, and return x J x_J x J
test. For q ≥ 1 q\ge1 q ≥ 1 , let a t , 1 , … , a t , q a_{t,1},\ldots,a_{t,q} a t , 1 , … , a t , q be independent standard Gaussian vectors in R d \mathbb{R}^d R d , fresh at each iteration. We consider either the forward differences
s t , i f = f ( x t + μ a t , i ) − f ( x t ) μ , (16) s_{t,i}^{\mathrm{f}}=\frac{f(x_t+\mu a_{t,i})-f(x_t)}{\mu}, \tag*{(16)} s t , i f = μ f ( x t + μ a t , i ) − f ( x t ) , (16)
or the central differences
s t , i c = f ( x t + μ a t , i ) − f ( x t − μ a t , i ) 2 μ . (17) s_{t,i}^{\mathrm{c}}=\frac{f(x_t+\mu a_{t,i})-f(x_t-\mu a_{t,i})}{2\mu}. \tag*{(17)} s t , i c = 2 μ f ( x t + μ a t , i ) − f ( x t − μ a t , i ) . (17)
In either case the update is
x t + 1 = x t − η q ∑ i = 1 q s t , i a t , i , x 0 = 0. (18) x_{t+1}=x_t-\frac{\eta}{q}\sum_{i=1}^{q}s_{t,i}a_{t,i},\qquad x_0=0. \tag*{(18)} x t + 1 = x t − q η i = 1 ∑ q s t , i a t , i , x 0 = 0. (18)
There is no test or rejection of x t + 1 x_{t+1} x t + 1 . The output is x J x_J x J for an independent J J J uniform on { 0 , … , N − 1 } \{0,\ldots,N-1\} { 0 , … , N − 1 } (Algorithm 6). For forward differences, f ( x t ) f(x_t) f ( x t ) is shared among the directions, so each iteration costs q + 1 q+1 q + 1 values; for central differences it costs 2 q 2q 2 q values. Both costs are at most 2 q 2q 2 q . In particular, no evaluation of f ( x t + 1 ) f(x_{t+1}) f ( x t + 1 ) is required in the current iteration.
Lemma B.11 (Two-point errors and batch Gaussian curvature). Suppose f f f is globally L L L -smooth. For either difference in (16) –(17) , writing g = ∇ f ( x ) g=\nabla f(x) g = ∇ f ( x ) , σ i = a i T g \sigma_i=a_i^\mathsf{T}g σ i = a i T g , and s i = σ i + β i s_i=\sigma_i+\beta_i s i = σ i + β i ,
∣ β i ∣ ≤ L μ 2 ∥ a i ∥ 2 . (19) |\beta_i|\le\frac{L\mu}{2}\lVert a_i\rVert^2. \tag*{(19)} ∣ β i ∣ ≤ 2 Lμ ∥ a i ∥ 2 . (19) Define v = q − 1 ∑ i σ i a i v=q^{-1}\sum_i\sigma_i a_i v = q − 1 ∑ i σ i a i and e = q − 1 ∑ i β i a i e=q^{-1}\sum_i\beta_i a_i e = q − 1 ∑ i β i a i , and write D 6 = d ( d + 2 ) ( d + 4 ) D_6=d(d+2)(d+4) D 6 = d ( d + 2 ) ( d + 4 ) . Then
E v = g , E ∥ e ∥ 2 ≤ L 2 μ 2 D 6 4 . (20) \mathbb{E}v=g,\qquad\mathbb{E}\lVert e\rVert^2\le\frac{L^2\mu^2D_6}{4}. \tag*{(20)} E v = g , E ∥ e ∥ 2 ≤ 4 L 2 μ 2 D 6 . (20) If also H ⪰ 0 H\succeq0 H ⪰ 0 , ∥ H ∥ o p ≤ L \lVert H\rVert_{\mathrm{op}}\le L ∥ H ∥ op ≤ L , and tr H ≤ L E \operatorname{tr}H\le LE tr H ≤ L E , then
E [ v T H v ] ≤ L ( E + q + 1 ) q ∥ g ∥ 2 . (21) \mathbb{E}[v^\mathsf{T}Hv]\le\frac{L(E+q+1)}{q}\lVert g\rVert^2. \tag*{(21)} E [ v T H v ] ≤ q L ( E + q + 1 ) ∥ g ∥ 2 . (21)
Proof. For forward differences integrate ( ∇ f ( x + t a i ) − g ) T a i (\nabla f(x+ta_i)-g)^\mathsf{T}a_i ( ∇ f ( x + t a i ) − g ) T a i over t ∈ [ 0 , μ ] t\in[0,\mu] t ∈ [ 0 , μ ] ; for central differences integrate over [ − μ , μ ] [-\mu,\mu] [ − μ , μ ] and divide by 2 μ 2\mu 2 μ . Global smoothness gives (19) in either case. Jensen’s inequality applied to the average defining e e e , followed by E ∥ a ∥ 6 = D 6 \mathbb{E}\lVert a\rVert^6=D_6 E ∥ a ∥ 6 = D 6 , proves (20) . The identity E [ ( a ⊤ g ) a ] = g \mathbb{E}[(a^{\top}g)a] = g E [( a ⊤ g ) a ] = g proves the other part of (20) . For the last claim expand the quadratic form of v v v . The q q q diagonal contributions are bounded, using Lemma B.9, by ∥ g ∥ 2 tr H + 2 g ⊤ H g ≤ L ( E + 2 ) ∥ g ∥ 2 \|g\|^{2}\operatorname{tr}H + 2g^{\top}Hg \le L(E + 2)\|g\|^{2} ∥ g ∥ 2 tr H + 2 g ⊤ H g ≤ L ( E + 2 ) ∥ g ∥ 2 . By independence, each of the q ( q − 1 ) q(q - 1) q ( q − 1 ) off-diagonal contributions is g ⊤ H g ≤ L ∥ g ∥ 2 g^{\top}Hg \le L\|g\|^{2} g ⊤ H g ≤ L ∥ g ∥ 2 . Division by q 2 q^{2} q 2 gives (21) .
Theorem B.12 (Gaussian two-point method on F \mathcal{F} F ). Let f ∈ F L , Δ , E d f \in\mathcal{F}^{d}_{L,\Delta,E} f ∈ F L , Δ , E d and 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , T = L Δ / ε 2 ≥ 1 T = L\Delta/\varepsilon^{2} \ge1 T = L Δ/ ε 2 ≥ 1 . For any integer q ≥ 1 q \ge1 q ≥ 1 , run (18) with either (16) or (17) , using
η = q L ( E + q + 1 ) , μ = ε 8 L d ( d + 2 ) ( d + 4 ) , N = ⌈ 17 ( E + q + 1 ) T q ⌉ . (22) \eta= \frac{q}{L(E+q+1)},\qquad\mu= \frac{\varepsilon}{8L\sqrt{d(d+2)(d+4)}},\qquad N = \left\lceil\frac{17(E+q+1)T}{q}\right\rceil. \tag*{(22)} η = L ( E + q + 1 ) q , μ = 8 L d ( d + 2 ) ( d + 4 ) ε , N = ⌈ q 17 ( E + q + 1 ) T ⌉ . (22) Then on each function
P ( ∥ ∇ f ( x J ) ∥ > ε ) ≤ 11 34 < 1 3 , Q ≤ 2 q N ≤ 34 ( E + q + 1 ) T + 2 q . (23) \mathbb{P}(\|\nabla f(x_J)\|>\varepsilon)\le\frac{11}{34}<\frac{1}{3},\qquad Q\le2qN\le34(E+q+1)T+2q. \tag*{(23)} P ( ∥∇ f ( x J ) ∥ > ε ) ≤ 34 11 < 3 1 , Q ≤ 2 q N ≤ 34 ( E + q + 1 ) T + 2 q . (23) In particular, with one direction per step (q = 1 q = 1 q = 1 ) the method uses at most 34 ( E + 2 ) T + 2 34(E+2)T+2 34 ( E + 2 ) T + 2 value calls.
Proof. Let C = L ( E + q + 1 ) / q = 1 / η C = L(E+q+1)/q = 1/\eta C = L ( E + q + 1 ) / q = 1/ η . The global quadratic upper model (3) applied with the random displacement − η ( v + e ) -\eta(v+e) − η ( v + e ) implies, conditionally on x x x ,
E [ f ( x ) − f ( x − η ( v + e ) ) ] ≥ η ∥ g ∥ 2 − η ∥ g ∥ E ∥ e ∥ − η 2 2 E [ ( v + e ) ⊤ H ( v + e ) ] . \mathbb{E}[f(x)-f(x-\eta(v+e))]\ge\eta\|g\|^{2}-\eta\|g\|\mathbb{E}\|e\|-\frac{\eta^{2}}{2}\mathbb{E}[(v+e)^{\top}H(v+e)]. E [ f ( x ) − f ( x − η ( v + e ))] ≥ η ∥ g ∥ 2 − η ∥ g ∥ E ∥ e ∥ − 2 η 2 E [( v + e ) ⊤ H ( v + e )] .
Since H ⪰ 0 H\succeq0 H ⪰ 0 , the matrix Young inequality with parameter 1 / 8 1/8 1/8 gives
( v + e ) ⊤ H ( v + e ) ≤ 9 8 v ⊤ H v + 9 e ⊤ H e . (v+e)^{\top}H(v+e)\le\frac{9}{8}v^{\top}Hv+9e^{\top}He. ( v + e ) ⊤ H ( v + e ) ≤ 8 9 v ⊤ H v + 9 e ⊤ H e .
Put κ = L μ D 6 / 2 = ε / 16 \kappa=L\mu\sqrt{D_{6}}/2=\varepsilon/16 κ = Lμ D 6 /2 = ε /16 . By Lemma B.11, Jensen and ∥ H ∥ o p ≤ L \|H\|_{\mathrm{op}}\le L ∥ H ∥ op ≤ L ,
E ∥ e ∥ ≤ κ , E [ e ⊤ H e ] ≤ L κ 2 , E [ v ⊤ H v ] ≤ C ∥ g ∥ 2 . \mathbb{E}\|e\|\le\kappa,\qquad\mathbb{E}[e^{\top}He]\le L\kappa^{2},\qquad\mathbb{E}[v^{\top}Hv]\le C\|g\|^{2}. E ∥ e ∥ ≤ κ , E [ e ⊤ H e ] ≤ L κ 2 , E [ v ⊤ H v ] ≤ C ∥ g ∥ 2 .
Since κ ∥ g ∥ ≤ ∥ g ∥ 2 / 16 + 4 κ 2 \kappa\|g\|\le\|g\|^{2}/16+4\kappa^{2} κ ∥ g ∥ ≤ ∥ g ∥ 2 /16 + 4 κ 2 and η L = q / ( E + q + 1 ) ≤ 1 \eta L=q/(E+q+1)\le1 η L = q / ( E + q + 1 ) ≤ 1 ,
E [ f ( x ) − f ( x t + 1 ) ∣ x t = x ] ≥ η ( 7 16 ∥ g ∥ 2 − κ ∥ g ∥ − 9 2 η L κ 2 ) ≥ η ( 3 8 ∥ g ∥ 2 − 17 2 κ 2 ) ≥ η ( 3 8 ∥ g ∥ 2 − 1 16 ε 2 ) . \begin{aligned}
\mathbb{E}[f(x)-f(x_{t+1})\mid x_t=x]
&\ge\eta\left(\frac{7}{16}\|g\|^{2}-\kappa\|g\|-\frac{9}{2}\eta L\kappa^{2}\right)\\
&\ge\eta\left(\frac{3}{8}\|g\|^{2}-\frac{17}{2}\kappa^{2}\right)\\
&\ge\eta\left(\frac{3}{8}\|g\|^{2}-\frac{1}{16}\varepsilon^{2}\right).
\end{aligned} E [ f ( x ) − f ( x t + 1 ) ∣ x t = x ] ≥ η ( 16 7 ∥ g ∥ 2 − κ ∥ g ∥ − 2 9 η L κ 2 ) ≥ η ( 8 3 ∥ g ∥ 2 − 2 17 κ 2 ) ≥ η ( 8 3 ∥ g ∥ 2 − 16 1 ε 2 ) .
where 17 / 512 ≤ 1 / 16 17/512\le1/16 17/512 ≤ 1/16 is the last numerical inequality. Summing the conditional expectations and using E f ( x N ) ≥ inf f ≥ f ( 0 ) − Δ \mathbb{E}f(x_N)\ge\inf f\ge f(0)-\Delta E f ( x N ) ≥ inf f ≥ f ( 0 ) − Δ yields
1 N ∑ t = 0 N − 1 E ∥ ∇ f ( x t ) ∥ 2 ≤ 8 Δ 3 η N + ε 2 6 . \frac{1}{N}\sum_{t=0}^{N-1}\mathbb{E}\|\nabla f(x_t)\|^{2}\le\frac{8\Delta}{3\eta N}+\frac{\varepsilon^{2}}{6}. N 1 t = 0 ∑ N − 1 E ∥∇ f ( x t ) ∥ 2 ≤ 3 η N 8Δ + 6 ε 2 .
By the independence and uniform distribution of J J J , Markov’s inequality and the choice of N N N imply
P ( ∥ ∇ f ( x J ) ∥ > ε ) ≤ 8 3 ( E + q + 1 ) T q N + 1 6 ≤ 8 51 + 1 6 = 11 34 . \mathbb{P}(\|\nabla f(x_J)\|>\varepsilon)\le\frac{8}{3}\frac{(E+q+1)T}{qN}+\frac{1}{6}\le\frac{8}{51}+\frac{1}{6}=\frac{11}{34}. P ( ∥∇ f ( x J ) ∥ > ε ) ≤ 3 8 q N ( E + q + 1 ) T + 6 1 ≤ 51 8 + 6 1 = 34 11 .
Every iteration makes the stated number of value calls whatever the directions, so the cap holds on every trajectory.
The first bound follows from Laurent and Massart [34 ] .
Lemma B.13 (Gaussian quadratic tail). Let a ∼ N ( 0 , I d ) a \sim N(0,I_d) a ∼ N ( 0 , I d ) and M ⪰ 0 M \succeq0 M ⪰ 0 , ∥ M ∥ o p ≤ L \lVert M\rVert_{\mathrm{op}} \le L ∥ M ∥ op ≤ L , tr M ≤ L r \operatorname{tr} M \le Lr tr M ≤ L r , with r > 0 r > 0 r > 0 . For u > 0 u > 0 u > 0 ,
P ( a T M a > L ( r + 2 r u + 2 u ) ) ≤ e − u . (24) \mathbb{P}\left(a^{\mathsf{T}}Ma > L\left(r+2\sqrt{ru}+2u\right)\right) \le e^{-u}. \tag*{(24)} P ( a T M a > L ( r + 2 r u + 2 u ) ) ≤ e − u . (24) Consequently, if tr H ≤ L E \operatorname{tr} H \le LE tr H ≤ L E , then
P ( a T H a > L ( 2 E + 3 u ) ) ≤ e − u , P ( ∥ a ∥ 2 > 2 d + 3 u ) ≤ e − u . (25) \mathbb{P}\left(a^{\mathsf{T}}Ha > L(2E+3u)\right) \le e^{-u}, \qquad\mathbb{P}\left(\lVert a\rVert^2 > 2d+3u\right) \le e^{-u}. \tag*{(25)} P ( a T H a > L ( 2 E + 3 u ) ) ≤ e − u , P ( ∥ a ∥ 2 > 2 d + 3 u ) ≤ e − u . (25) Proof. Diagonalize M / L M/L M / L , with eigenvalues w j ∈ [ 0 , 1 ] w_j \in[0,1] w j ∈ [ 0 , 1 ] and ∑ j w j ≤ r \sum_j w_j \le r ∑ j w j ≤ r . For independent Z j ∼ N ( 0 , 1 ) Z_j \sim N(0,1) Z j ∼ N ( 0 , 1 ) and 0 < λ < 1 / 2 0 < \lambda< 1/2 0 < λ < 1/2 ,
log E exp ( λ ∑ j w j ( Z j 2 − 1 ) ) = ∑ j [ − λ w j − 1 2 log ( 1 − 2 λ w j ) ] ≤ λ 2 ∑ j w j 2 1 − 2 λ ≤ r λ 2 1 − 2 λ . \log\mathbb{E}\exp\left(\lambda\sum_j w_j(Z_j^2-1)\right)
= \sum_j\left[-\lambda w_j-\frac{1}{2}\log(1-2\lambda w_j)\right]
\le\frac{\lambda^2\sum_j w_j^2}{1-2\lambda}
\le\frac{r\lambda^2}{1-2\lambda}. log E exp ( λ j ∑ w j ( Z j 2 − 1 ) ) = j ∑ [ − λ w j − 2 1 log ( 1 − 2 λ w j ) ] ≤ 1 − 2 λ λ 2 ∑ j w j 2 ≤ 1 − 2 λ r λ 2 . The first inequality follows by expanding − log ( 1 − z ) − z -\log(1-z)-z − log ( 1 − z ) − z into its nonnegative Taylor series and bounding it by z 2 / [ 2 ( 1 − z ) ] z^2/[2(1-z)] z 2 / [ 2 ( 1 − z )] , for 0 ≤ z < 1 0 \le z < 1 0 ≤ z < 1 . Apply the exponential Markov inequality at λ = u / ( r + 2 u ) \lambda=\sqrt{u}/(\sqrt{r}+2\sqrt{u}) λ = u / ( r + 2 u ) ; the resulting exponent at 2 r u + 2 u 2\sqrt{ru}+2u 2 r u + 2 u is − u -u − u . Adding the mean, at most r r r , proves (24) . For the first claim in (25) use 2 E u ≤ E + u 2\sqrt{Eu}\le E+u 2 E u ≤ E + u ; for the second set M = I d M=I_d M = I d , L = 1 L=1 L = 1 , r = d r=d r = d and apply the same inequality.
Lemma B.14 (Capped descent for a truncated Gaussian batch). Let f ∈ B L , Δ , E d f \in\mathcal{B}^{d}_{L,\Delta,E} f ∈ B L , Δ , E d , let H H H be its fixed upper matrix, and let x x x satisfy f ( x ) ≤ f ( 0 ) f(x) \le f(0) f ( x ) ≤ f ( 0 ) . Fix Λ > 0 \Lambda> 0 Λ > 0 and draw a 1 , … , a q a_1,\ldots,a_q a 1 , … , a q independently from N ( 0 , I d ) N(0,I_d) N ( 0 , I d ) . Set
v = 1 q ∑ i = 1 q ( a i T ∇ f ( x ) ) a i 1 { a i T H a i ≤ Λ } , η = Λ − 1 , (26) v=\frac{1}{q}\sum_{i=1}^{q}(a_i^{\mathsf{T}}\nabla f(x))a_i\mathbf{1}_{\{a_i^{\mathsf{T}}Ha_i\le\Lambda\}}, \qquad\eta=\Lambda^{-1}, \tag*{(26)} v = q 1 i = 1 ∑ q ( a i T ∇ f ( x )) a i 1 { a i T H a i ≤ Λ } , η = Λ − 1 , (26) Then f ( x − η v ) ≤ f ( x ) f(x-\eta v)\le f(x) f ( x − η v ) ≤ f ( x ) and, pathwise,
f ( x ) − f ( x − η v ) ≥ η 2 q ∑ i = 1 q ( a i T ∇ f ( x ) ) 2 1 { a i T H a i ≤ Λ } . (27) f(x)-f(x-\eta v)\ge\frac{\eta}{2q}\sum_{i=1}^{q}(a_i^{\mathsf{T}}\nabla f(x))^2\mathbf{1}_{\{a_i^{\mathsf{T}}Ha_i\le\Lambda\}}. \tag*{(27)} f ( x ) − f ( x − η v ) ≥ 2 q η i = 1 ∑ q ( a i T ∇ f ( x ) ) 2 1 { a i T H a i ≤ Λ } . (27) If P ( a T H a > Λ ) ≤ 1 / 48 \mathbb{P}(a^{\mathsf{T}}Ha>\Lambda)\le1/48 P ( a T H a > Λ ) ≤ 1/48 , then
E [ f ( x ) − f ( x − η v ) ∣ x ] ≥ 3 η 8 ∥ ∇ f ( x ) ∥ 2 . (28) \mathbb{E}[f(x)-f(x-\eta v)\mid x]\ge\frac{3\eta}{8}\lVert\nabla f(x)\rVert^2. \tag*{(28)} E [ f ( x ) − f ( x − η v ) ∣ x ] ≥ 8 3 η ∥ ∇ f ( x ) ∥ 2 . (28) Proof. Write g = ∇ f ( x ) g=\nabla f(x) g = ∇ f ( x ) , σ i = a i T g \sigma_i=a_i^{\mathsf{T}}g σ i = a i T g and I = { i : a i T H a i ≤ Λ } I=\{i:a_i^{\mathsf{T}}Ha_i\le\Lambda\} I = { i : a i T H a i ≤ Λ } . Cauchy–Schwarz in the H H H seminorm, followed by scalar Cauchy–Schwarz, yields
v T H v ≤ 1 q 2 ( ∑ i ∈ I ∣ σ i ∣ a i T H a i ) 2 ≤ Λ q ∑ i ∈ I σ i 2 = Λ g T v . v^{\mathsf{T}}Hv\le\frac{1}{q^2}\left(\sum_{i\in I}|\sigma_i|\sqrt{a_i^{\mathsf{T}}Ha_i}\right)^2\le\frac{\Lambda}{q}\sum_{i\in I}\sigma_i^2=\Lambda g^{\mathsf{T}}v. v T H v ≤ q 2 1 ( i ∈ I ∑ ∣ σ i ∣ a i T H a i ) 2 ≤ q Λ i ∈ I ∑ σ i 2 = Λ g T v . Therefore, by the capped upper model in Lemma B.3 ,
min { f ( x − η v ) , f ( 0 ) } ≤ f ( x ) − η g T v + η 2 2 v T H v ≤ f ( x ) − η 2 g T v . \min\{f(x-\eta v),f(0)\}\le f(x)-\eta g^{\mathsf{T}}v+\frac{\eta^2}{2}v^{\mathsf{T}}Hv\le f(x)-\frac{\eta}{2}g^{\mathsf{T}}v. min { f ( x − η v ) , f ( 0 )} ≤ f ( x ) − η g T v + 2 η 2 v T H v ≤ f ( x ) − 2 η g T v . If g T v > 0 g^{\mathsf{T}}v>0 g T v > 0 the right side is strictly below f ( 0 ) f(0) f ( 0 ) , so the minimum must be the actual value. If g T v = 0 g^{\mathsf{T}}v=0 g T v = 0 , then every retained σ i \sigma_i σ i vanishes, v = 0 v=0 v = 0 , and the assertion is immediate. This proves (27) . For σ = a T g \sigma=a^{\mathsf{T}}g σ = a T g and p = P ( a T H a > Λ ) p=\mathbb{P}(a^{\mathsf{T}}Ha>\Lambda) p = P ( a T H a > Λ ) , Cauchy–Schwarz and the normal fourth moment give
E [ σ 2 1 { a T H a > Λ } ] ≤ E σ 4 p = 3 p ∥ g ∥ 2 ≤ 1 4 ∥ g ∥ 2 . \mathbb{E}[\sigma^2\mathbf{1}_{\{a^{\mathsf{T}}Ha>\Lambda\}}]\le\sqrt{\mathbb{E}\sigma^4p}=\sqrt{3p}\lVert g\rVert^2\le\frac14\lVert g\rVert^2. E [ σ 2 1 { a T H a > Λ } ] ≤ E σ 4 p = 3 p ∥ g ∥ 2 ≤ 4 1 ∥ g ∥ 2 . Independence and the preceding pathwise inequality yield (28) .
Theorem B.15 (Gaussian two-point method on B \mathcal{B} B ). Let f ∈ B L , Δ , E d f \in\mathcal{B}^{d}_{L,\Delta,E} f ∈ B L , Δ , E d , 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d and T = L Δ / ε 2 ≥ 1 T = L\Delta/\varepsilon^{2} \ge1 T = L Δ/ ε 2 ≥ 1 . For any integer q ≥ 1 q \ge1 q ≥ 1 define the public parameters
W = E + log ( 1000 q E T ) , N = ⌈ 256 T W ⌉ , u = log ( 48 q N ) , m = 2 E + 3 u , R = 2 d + 3 u , η = ( L m ) − 1 , μ = ε 4 L R ( 1 + η L R ) N . (29) \begin{aligned}
W &= E + \log(1000qET), & N &= \lceil256TW\rceil,\\
u &= \log(48qN), & m &= 2E + 3u,\\
R &= 2d + 3u, & \eta&= (Lm)^{-1},\\
\mu&= \frac{\varepsilon}{4L\sqrt{R}(1+\eta LR)^{N}}. \tag*{(29)}
\end{aligned} W u R μ = E + log ( 1000 q E T ) , = log ( 48 q N ) , = 2 d + 3 u , = 4 L R ( 1 + η L R ) N ε . N m η = ⌈ 256 T W ⌉ , = 2 E + 3 u , = ( L m ) − 1 , (29) Run the update (18) , without an acceptance test, with this fixed η , μ , N \eta,\mu,N η , μ , N and either forward or central differences. Then, for every such f f f ,
P ( ∥ ∇ f ( x J ) ∥ > ε ) ≤ 13 56 < 1 3 , Q ≤ 2 q N ≤ 512 q T ( E + log ( 1000 q E T ) ) + 2 q . (30) \mathbb{P}\left(\lVert\nabla f(x_J)\rVert> \varepsilon\right) \le\frac{13}{56} < \frac{1}{3}, \qquad Q \le2qN \le512qT\left(E+\log(1000qET)\right)+2q. \tag*{(30)} P ( ∥ ∇ f ( x J )∥ > ε ) ≤ 56 13 < 3 1 , Q ≤ 2 q N ≤ 512 q T ( E + log ( 1000 q E T ) ) + 2 q . (30) In particular, for q = 1 q=1 q = 1 this is an O ( E T + T log ( E T ) ) O(ET+T\log(ET)) O ( E T + T log ( E T )) value bound. For q = 1 q=1 q = 1 and E ≥ log ( 1000 E T ) E \ge\log(1000ET) E ≥ log ( 1000 E T ) , the chosen step size has order 1 / ( L E ) 1/(LE) 1/ ( L E ) and the call bound is O ( E T ) O(ET) O ( E T ) , with absolute constants.
Proof. Let H H H be the unknown matrix witnessing membership in B \mathcal{B} B . Its use below is confined to the analysis; the algorithm’s parameters depend only on d , L , Δ , ε , E , q d,L,\Delta,\varepsilon,E,q d , L , Δ , ε , E , q . Since T W ≥ 1 TW \ge1 T W ≥ 1 , N ≤ 257 T W N \le257TW N ≤ 257 T W . By log ( 48 ⋅ 257 ) < 10 \log(48\cdot257)<10 log ( 48 ⋅ 257 ) < 10 , log ( q T ) ≤ log ( q E T ) ≤ W \log(qT) \le\log(qET) \le W log ( q T ) ≤ log ( q E T ) ≤ W , log W ≤ W \log W \le W log W ≤ W , and 10 ≤ 2 W 10 \le2W 10 ≤ 2 W , we have
u ≤ 10 + log ( q T ) + log W ≤ 4 W , m = 2 E + 3 u ≤ 14 W . (31) u \le10+\log(qT)+\log W \le4W,\qquad m=2E+3u\le14W. \tag*{(31)} u ≤ 10 + log ( q T ) + log W ≤ 4 W , m = 2 E + 3 u ≤ 14 W . (31)
Lemma B.13 applied to H H H gives P ( a T H a > L m ) ≤ e − u = 1 / ( 48 q N ) \mathbb{P}(a^{\mathsf{T}}Ha>Lm)\le e^{-u}=1/(48qN) P ( a T H a > L m ) ≤ e − u = 1/ ( 48 q N ) . Construct an auxiliary process z 0 = 0 z_{0}=0 z 0 = 0 , and, for t = 0 , … , N − 1 t=0,\ldots,N-1 t = 0 , … , N − 1 , define z t + 1 z_{t+1} z t + 1 by the truncated directional derivative step (26) with Λ = L m \Lambda=Lm Λ = L m and the same Gaussian directions as the actual algorithm. This process is only a proof device; the actual algorithm evaluates neither directional derivatives nor H H H . By Lemma B.14, its iterates satisfy f ( z t ) ≤ f ( 0 ) f(z_t)\le f(0) f ( z t ) ≤ f ( 0 ) pathwise and
E [ f ( z t ) − f ( z t + 1 ) ∣ z t ] ≥ 3 η 8 ∥ ∇ f ( z t ) ∥ 2 . \mathbb{E}\left[f(z_t)-f(z_{t+1})\mid z_t\right]\ge\frac{3\eta}{8}\lVert\nabla f(z_t)\rVert^{2}. E [ f ( z t ) − f ( z t + 1 ) ∣ z t ] ≥ 8 3 η ∥ ∇ f ( z t ) ∥ 2 .
Telescoping and using the initial gap yield
P ( ∥ ∇ f ( z J ) ∥ > 7 ε 8 ) ≤ 512 m T 147 N ≤ 4 21 . (32) \mathbb{P}\left(\lVert\nabla f(z_J)\rVert>\frac{7\varepsilon}{8}\right)\le\frac{512mT}{147N}\le\frac{4}{21}. \tag*{(32)} P ( ∥ ∇ f ( z J )∥ > 8 7 ε ) ≤ 147 N 512 m T ≤ 21 4 . (32)
In the first inequality we used the second-moment bound N − 1 ∑ t E ∥ ∇ f ( z t ) ∥ 2 ≤ 8 Δ / ( 3 η N ) N^{-1}\sum_t\mathbb{E}\lVert\nabla f(z_t)\rVert^{2}\le8\Delta/(3\eta N) N − 1 ∑ t E ∥ ∇ f ( z t ) ∥ 2 ≤ 8Δ/ ( 3 η N ) and Markov’s inequality; the last inequality follows from (31) and N ≥ 256 T W N\ge256TW N ≥ 256 T W .
Let G \mathcal{G} G be the event that, for every one of the q N qN q N generated directions,
a t , i T H a t , i ≤ L m , ∥ a t , i ∥ 2 ≤ R . (33) a_{t,i}^{\mathsf{T}}Ha_{t,i}\le Lm,\qquad\lVert a_{t,i}\rVert^{2}\le R. \tag*{(33)} a t , i T H a t , i ≤ L m , ∥ a t , i ∥ 2 ≤ R . (33)
Both Gaussian bounds in (25) and a union bound imply P ( G c ) ≤ 1 / 24 \mathbb{P}(\mathcal{G}^{\mathsf{c}})\le1/24 P ( G c ) ≤ 1/24 . On G \mathcal{G} G , none of the auxiliary directions is removed. We compare its entire trajectory to that of the actual finite radius algorithm. Write D t = ∥ x t − z t ∥ D_t=\lVert x_t-z_t\rVert D t = ∥ x t − z t ∥ and A t = q − 1 ∑ i a t , i a t , i T A_t=q^{-1}\sum_i a_{t,i}a_{t,i}^{\mathsf{T}} A t = q − 1 ∑ i a t , i a t , i T . On G \mathcal{G} G , ∥ A t ∥ o p ≤ R \lVert A_t\rVert_{\mathrm{op}}\le R ∥ A t ∥ op ≤ R and
∥ q − 1 ∑ i ( s t , i − a t , i T ∇ f ( x t ) ) a t , i ∥ ≤ L μ R 3 / 2 2 \left\lVert q^{-1}\sum_i\left(s_{t,i}-a_{t,i}^{\mathsf{T}}\nabla f(x_t)\right)a_{t,i}\right\rVert\le\frac{L\mu R^{3/2}}{2} q − 1 i ∑ ( s t , i − a t , i T ∇ f ( x t ) ) a t , i ≤ 2 Lμ R 3/2
by (19) . Global L L L -smoothness of f f f gives
D t + 1 ≤ ( 1 + η L R ) D t + η L μ R 3 / 2 2 , D 0 = 0. D_{t+1} \le(1+\eta LR)D_t+\frac{\eta L\mu R^{3/2}}{2}, \qquad D_0=0. D t + 1 ≤ ( 1 + η L R ) D t + 2 η Lμ R 3/2 , D 0 = 0.
Summing the geometric recursion,
max 0 ≤ t ≤ N D t ≤ μ R 2 ( ( 1 + η L R ) N − 1 ) ≤ ε 8 L . (34) \max_{0\le t\le N}D_t\le\frac{\mu\sqrt{R}}{2}\left((1+\eta LR)^N-1\right)\le\frac{\varepsilon}{8L}. \tag*{(34)} 0 ≤ t ≤ N max D t ≤ 2 μ R ( ( 1 + η L R ) N − 1 ) ≤ 8 L ε . (34)
Thus, on G \mathcal{G} G , ∥ ∇ f ( x J ) ∥ ≤ ∥ ∇ f ( z J ) ∥ + ε / 8 \lVert\nabla f(x_J)\rVert\le\lVert\nabla f(z_J)\rVert+\varepsilon/8 ∥ ∇ f ( x J )∥ ≤ ∥ ∇ f ( z J )∥ + ε /8 . Combining this deterministic implication with (32) and the union bound,
P ( ∥ ∇ f ( x J ) ∥ > ε ) ≤ 4 21 + 1 24 = 13 56 < 1 3 . \mathbb{P}(\lVert\nabla f(x_J)\rVert>\varepsilon)\le\frac{4}{21}+\frac{1}{24}=\frac{13}{56}<\frac{1}{3}. P (∥ ∇ f ( x J )∥ > ε ) ≤ 21 4 + 24 1 = 56 13 < 3 1 .
Exactly N N N iterations are run, and each consumes at most 2 q 2q 2 q calls. Finally, N ≤ 256 T W + 1 N\le256TW+1 N ≤ 256 T W + 1 gives the asserted call cap.
The radius μ \mu μ in (29) is exponentially small in N N N . Exact values allow it, and the proof needs it: with a nonzero radius a step can raise f f f slightly even when a ⊤ H a a^{\top}Ha a ⊤ H a is small, so the iterates are compared with the auxiliary process along the whole run. For E < log ( 1000 E T ) E<\log(1000ET) E < log ( 1000 E T ) , whether a step of order 1 / ( L E ) 1/(LE) 1/ ( L E ) without an acceptance test gives O ( E T ) O(ET) O ( E T ) calls on B \mathcal{B} B remains open. With the acceptance test, Theorem B.10 gives O ( E T ) O(ET) O ( E T ) ; its comparison uses the value f ( y t ) f(y_t) f ( y t ) , which the forward method without the test also queries as the next base value f ( x t + 1 ) f(x_{t+1}) f ( x t + 1 ) .
Direction Selection and the Stage Argument
Direction selection by finite partial coloring
The selection lemma uses the classical finite partial coloring argument (Spencer, [50 ] ): quantize linear forms of sign vectors, find a large common fiber, and subtract two distant members.
Lemma C.1 (Finite entropy facts; see Cover and Thomas, [17 ] ). For a finitely supported probability mass function p p p , write H ( p ) = ∑ j : p j > 0 p j log ( 1 / p j ) H(p)=\sum_{j:p_j>0}p_j\log(1/p_j) H ( p ) = ∑ j : p j > 0 p j log ( 1/ p j ) . If reference masses q j > 0 q_j>0 q j > 0 on the support of p p p have total mass at most one, then H ( p ) ≤ ∑ j p j log ( 1 / q j ) H(p)\le\sum_jp_j\log(1/q_j) H ( p ) ≤ ∑ j p j log ( 1/ q j ) . Joint entropy is at most the sum of marginal entropies, and some atom has mass ≥ exp [ − H ( p ) ] \ge\exp[-H(p)] ≥ exp [ − H ( p )] .
Proof. Use log t ≤ t − 1 \log t\le t-1 log t ≤ t − 1 in ∑ p j > 0 p j log ( q j / p j ) ≤ ∑ p j > 0 q j − 1 ≤ 0 \sum_{p_j>0}p_j\log(q_j/p_j)\le\sum_{p_j>0}q_j-1\le0 ∑ p j > 0 p j log ( q j / p j ) ≤ ∑ p j > 0 q j − 1 ≤ 0 . For subadditivity choose the product of the marginal laws as reference; it is positive on the joint support. Finally H ( p ) ≥ log ( 1 / max j p j ) H(p)\ge\log(1/\max_jp_j) H ( p ) ≥ log ( 1/ max j p j ) .
Lemma C.2 (Strip selection). If S ⊂ R d S\subset\mathbb{R}^d S ⊂ R d has dimension p ≥ 1 p\ge1 p ≥ 1 and ∥ a i ∥ ≤ R \lVert a_i\rVert\le R ∥ a i ∥ ≤ R for 1 ≤ i ≤ M 1\le i\le M 1 ≤ i ≤ M , there is a unit u ∈ S u\in S u ∈ S such that
max i ≤ M ∣ ⟨ a i , u ⟩ ∣ ≤ 13 3 R log ( e + M / p ) p . (35) \max_{i\le M}|\langle a_i,u\rangle|\le\frac{13}{3}R\sqrt{\frac{\log(e+M/p)}{p}}. \tag*{(35)} i ≤ M max ∣ ⟨ a i , u ⟩ ∣ ≤ 3 13 R p log ( e + M / p ) . (35)
Proof. If M < p M<p M < p or R = 0 R=0 R = 0 , choose a unit vector u ∈ S u\in S u ∈ S orthogonal to the projections of a 1 , … , a M a_1,\ldots,a_M a 1 , … , a M onto S S S , giving max i ≤ M ∣ ⟨ a i , u ⟩ ∣ = 0 \max_{i\le M}|\langle a_i,u\rangle|=0 max i ≤ M ∣ ⟨ a i , u ⟩ ∣ = 0 . Assume now that s : = M / p ≥ 1 s:=M/p\ge1 s := M / p ≥ 1 and R > 0 R>0 R > 0 . Use orthonormal coordinates on S S S and project the a i a_i a i to S S S . Let X X X be uniform on { − 1 , 1 } p \{-1,1\}^p { − 1 , 1 } p ; set a = 7 2 log ( e + s ) a=\frac{7}{2}\log(e+s) a = 2 7 log ( e + s ) , t = 2 a t=\sqrt{2a} t = 2 a , Y i = ⟨ a i , X ⟩ / R Y_i=\langle a_i,X\rangle/R Y i = ⟨ a i , X ⟩ / R , and K i = ⌊ Y i / ( 2 t ) + 1 / 2 ⌋ K_i=\lfloor Y_i/(2t)+1/2\rfloor K i = ⌊ Y i / ( 2 t ) + 1/2 ⌋ . Since ( e + 1 ) 7 > e 9 (e+1)^7>e^9 ( e + 1 ) 7 > e 9 , we have a ≥ 7 2 log ( e + 1 ) > 9 2 a\ge\frac{7}{2}\log(e+1)>\frac{9}{2} a ≥ 2 7 log ( e + 1 ) > 2 9 . The inequality log cosh z ≤ z 2 / 2 \log\cosh z\le z^2/2 log cosh z ≤ z 2 /2 follows by integrating tanh z ≤ z \tanh z\le z tanh z ≤ z for z ≥ 0 z\ge0 z ≥ 0 and using evenness. Independence gives E e λ Y i ≤ e λ 2 / 2 \mathbb{E}e^{\lambda Y_i}\le e^{\lambda^2/2} E e λ Y i ≤ e λ 2 /2 , hence P ( ∣ Y i ∣ ≥ b ) ≤ 2 e − b 2 / 2 \mathbb{P}(|Y_i|\ge b)\le2e^{-b^2/2} P ( ∣ Y i ∣ ≥ b ) ≤ 2 e − b 2 /2 , as in Hoeffding’s inequality (Hoeffding, [22 ] ). Choose an integer N N N bounding every value ∣ K i ( X ) ∣ |K_i(X)| ∣ K i ( X ) ∣ . Since ( 2 k − 1 ) 2 ≥ k (2k-1)^2 \ge k ( 2 k − 1 ) 2 ≥ k for every integer k ≥ 1 k \ge1 k ≥ 1 ,
P ( ∣ K i ∣ ≥ k ) ≤ 2 e − ( 2 k − 1 ) 2 a ≤ 2 e − k a , E ∣ K i ∣ = ∑ k = 1 N P ( ∣ K i ∣ ≥ k ) < 2 e − a 1 − e − a . \mathbb{P}(|K_i| \ge k) \le2e^{-(2k-1)^2a} \le2e^{-ka}, \qquad\mathbb{E}|K_i| = \sum_{k=1}^{N}\mathbb{P}(|K_i| \ge k) < \frac{2e^{-a}}{1-e^{-a}}. P ( ∣ K i ∣ ≥ k ) ≤ 2 e − ( 2 k − 1 ) 2 a ≤ 2 e − k a , E ∣ K i ∣ = k = 1 ∑ N P ( ∣ K i ∣ ≥ k ) < 1 − e − a 2 e − a .
Take reference masses q k = Z − 1 e − a ∣ k ∣ q_k = Z^{-1}e^{-a|k|} q k = Z − 1 e − a ∣ k ∣ on { − N , … , N } \{-N,\ldots,N\} { − N , … , N } . Finite geometric summation gives log Z ≤ Z − 1 < 2 e − a / ( 1 − e − a ) \log Z \le Z-1 < 2e^{-a}/(1-e^{-a}) log Z ≤ Z − 1 < 2 e − a / ( 1 − e − a ) . Lemma C.1 implies
H ( K i ) ≤ log Z + a E ∣ K i ∣ < 2 ( a + 1 ) e − a 1 − e − a . (36) H(K_i) \le\log Z + a\mathbb{E}|K_i| < \frac{2(a+1)e^{-a}}{1-e^{-a}}. \tag*{(36)} H ( K i ) ≤ log Z + a E ∣ K i ∣ < 1 − e − a 2 ( a + 1 ) e − a . (36)
For s ≥ 1 s \ge1 s ≥ 1 and a = 7 2 log ( e + s ) > 9 2 a = \frac{7}{2}\log(e+s) > \frac{9}{2} a = 2 7 log ( e + s ) > 2 9 , the derivative of log s + log ( a + 1 ) − a \log s+\log(a+1)-a log s + log ( a + 1 ) − a with respect to s s s is at most 1 s − 63 22 ( e + s ) \frac{1}{s}-\frac{63}{22(e+s)} s 1 − 22 ( e + s ) 63 , which is positive only for s < 22 41 e < 3 2 s < \frac{22}{41}e < \frac{3}{2} s < 41 22 e < 2 3 , and across all s ≥ 1 s \ge1 s ≥ 1 one has s H ( K i ) < 3 25 sH(K_i) < \frac{3}{25} sH ( K i ) < 25 3 . By subadditivity, the joint entropy of K = ( K 1 , … , K M ) K=(K_1,\ldots,K_M) K = ( K 1 , … , K M ) satisfies H ( K ) ≤ ∑ i = 1 M H ( K i ) < 3 M 25 s = 3 25 p H(K) \le\sum_{i=1}^{M}H(K_i) < \frac{3M}{25s}=\frac{3}{25}p H ( K ) ≤ ∑ i = 1 M H ( K i ) < 25 s 3 M = 25 3 p .
By Lemma C.1 , some common fiber F ⊆ { − 1 , 1 } p F \subseteq\{-1,1\}^{p} F ⊆ { − 1 , 1 } p of K K K has cardinality ∣ F ∣ ≥ 2 p e − H ( K ) |F| \ge2^p e^{-H(K)} ∣ F ∣ ≥ 2 p e − H ( K ) .
Let W W W be uniform on F F F with mean μ = E W ∈ [ − 1 , 1 ] p \mu=\mathbb{E}W \in[-1,1]^p μ = E W ∈ [ − 1 , 1 ] p . Applying Lemma C.1 to each coordinate W j W_j W j with reference masses e ± μ j / ( 2 cosh μ j ) e^{\pm\mu_j}/(2\cosh\mu_j) e ± μ j / ( 2 cosh μ j ) gives
H ( W j ) ≤ log ( 2 cosh μ j ) − μ j 2 ≤ log 2 − 1 2 μ j 2 . H(W_j) \le\log(2\cosh\mu_j)-\mu_j^2 \le\log2-\frac{1}{2}\mu_j^2. H ( W j ) ≤ log ( 2 cosh μ j ) − μ j 2 ≤ log 2 − 2 1 μ j 2 .
Subadditivity of entropy across the p p p coordinates gives p log 2 − H ( K ) ≤ log ∣ F ∣ = H ( W ) ≤ p log 2 − 1 2 ∥ μ ∥ 2 p\log2-H(K) \le\log|F|=H(W) \le p\log2-\frac{1}{2}\|\mu\|^2 p log 2 − H ( K ) ≤ log ∣ F ∣ = H ( W ) ≤ p log 2 − 2 1 ∥ μ ∥ 2 , whence ∥ μ ∥ 2 ≤ 2 H ( K ) < 6 25 p \|\mu\|^2 \le2H(K)<\frac{6}{25}p ∥ μ ∥ 2 ≤ 2 H ( K ) < 25 6 p . Two independent uniform draws X , X ′ X,X' X , X ′ from F F F satisfy
E ∥ X − X ′ ∥ 2 = 2 ( p − ∥ μ ∥ 2 ) > 38 25 p , \mathbb{E}\|X-X'\|^2=2(p-\|\mu\|^2)>\frac{38}{25}p, E ∥ X − X ′ ∥ 2 = 2 ( p − ∥ μ ∥ 2 ) > 25 38 p ,
so some pair X , X ′ ∈ F X,X'\in F X , X ′ ∈ F satisfies ∥ X − X ′ ∥ > ( 38 / 25 ) p \|X-X'\|>\sqrt{(38/25)p} ∥ X − X ′ ∥ > ( 38/25 ) p , while equal bins give ∣ ⟨ a i , X − X ′ ⟩ ∣ < 2 t R = 28 log ( e + s ) R |\langle a_i,X-X'\rangle|<2tR=\sqrt{28\log(e+s)}R ∣ ⟨ a i , X − X ′ ⟩ ∣ < 2 tR = 28 log ( e + s ) R . Normalizing X − X ′ X-X' X − X ′ yields a unit vector u ∈ S u\in S u ∈ S with squared projection bound at most 350 19 R 2 log ( e + s ) / p < ( 13 3 R 2 ) log ( e + s ) / p \frac{350}{19}R^2\log(e+s)/p<\left(\frac{13}{3}R^2\right)\log(e+s)/p 19 350 R 2 log ( e + s ) / p < ( 3 13 R 2 ) log ( e + s ) / p .
Lemma C.3 (Avoiding spherical caps by a union bound). Let R > 0 R>0 R > 0 and τ ≥ 0 \tau\ge0 τ ≥ 0 . Let S S S have dimension p ≥ 1 p\ge1 p ≥ 1 , and let ∥ a i ∥ ≤ R \|a_i\|\le R ∥ a i ∥ ≤ R . If 2 M exp ( − p τ 2 / ( 2 R 2 ) ) < 1 2M\exp(-p\tau^2/(2R^2))<1 2 M exp ( − p τ 2 / ( 2 R 2 )) < 1 , there is a unit u ∈ S u\in S u ∈ S with ∣ ⟨ u , a i ⟩ ∣ ≤ τ |\langle u,a_i\rangle|\le\tau ∣ ⟨ u , a i ⟩ ∣ ≤ τ for all i i i .
Proof. A uniform sphere direction u u u satisfies P ( ∣ ⟨ u , a ⟩ ∣ > τ ) ≤ 2 e − p τ 2 / ( 2 R 2 ) \mathbb{P}(|\langle u,a\rangle|>\tau)\le2e^{-p\tau^2/(2R^2)} P ( ∣ ⟨ u , a ⟩ ∣ > τ ) ≤ 2 e − p τ 2 / ( 2 R 2 ) . Indeed its odd moments vanish and its 2 k 2k 2 k -th projection moment is ∥ P S a ∥ 2 k ( 2 k − 1 ) ! ! / [ p ( p + 2 ) ⋯ ( p + 2 k − 2 ) ] \|P_Sa\|^{2k}(2k-1)!!/[p(p+2)\cdots(p+2k-2)] ∥ P S a ∥ 2 k ( 2 k − 1 )!! / [ p ( p + 2 ) ⋯ ( p + 2 k − 2 )] , where P S P_S P S is the orthogonal projection onto S S S . The moment-generating series is dominated by e λ 2 R 2 / ( 2 p ) e^{\lambda^2R^2/(2p)} e λ 2 R 2 / ( 2 p ) and exponential Markov proves the tail. For the moment identity, polar coordinates for an isotropic Gaussian give an independent uniform direction and radius; scalar Gaussian moments are ( 2 k − 1 ) ! ! (2k-1)!! ( 2 k − 1 )!! and the radius moments are p ( p + 2 ) ⋯ ( p + 2 k − 2 ) p(p+2)\cdots(p+2k-2) p ( p + 2 ) ⋯ ( p + 2 k − 2 ) by integration by parts, including the p = 1 p=1 p = 1 case. The union bound now leaves positive probability of avoiding all the caps.
One transcript argument
The following lemma handles the deterministic lower bounds. It handles values, rank-priced linear feedback, closing calls, unqueried outputs, and the identification of the simulated run with the run on one fixed function.
Lemma C.4 (Stage argument). Let F U F_U F U range over orthonormal m m m -frames U = ( u 1 , … , u m ) U=(u_1,\ldots,u_m) U = ( u 1 , … , u m ) in R d \mathbb{R}^d R d , and let z : R d → R d z:\mathbb{R}^d\to\mathbb{R}^d z : R d → R d satisfy ∥ z ( x ) ∥ ≤ R \lVert z(x)\rVert\le R ∥ z ( x )∥ ≤ R . Suppose that for each partial frame there are a prefix function r j r_j r j and a coefficient c j c_j c j , depending only on u 1 , … , u j − 1 u_1,\ldots,u_{j-1} u 1 , … , u j − 1 , such that
F U ( x ) = r j ( x ) , ∇ F U ( x ) = ∇ r j ( x ) − c j ( x ) u j F_U(x)=r_j(x),\qquad\nabla F_U(x)=\nabla r_j(x)-c_j(x)u_j F U ( x ) = r j ( x ) , ∇ F U ( x ) = ∇ r j ( x ) − c j ( x ) u j whenever ⟨ u j , z ( x ) ⟩ = 0 \langle u_j,z(x)\rangle=0 ⟨ u j , z ( x )⟩ = 0 and ∣ ⟨ u k , z ( x ) ⟩ ∣ ≤ τ \lvert\langle u_k,z(x)\rangle\rvert\le\tau ∣⟨ u k , z ( x )⟩∣ ≤ τ for all k > j k>j k > j . Assume also that ⟨ u j , z ( x ) ⟩ = 0 \langle u_j,z(x)\rangle=0 ⟨ u j , z ( x )⟩ = 0 for some j j j implies ∥ ∇ F U ( x ) ∥ > b \lVert\nabla F_U(x)\rVert>b ∥ ∇ F U ( x )∥ > b .
Let m , h ≥ 1 m,h\ge1 m , h ≥ 1 and Q ≥ 0 Q\ge0 Q ≥ 0 be integers and put p = d − m − h + 1 ≥ 1 p=d-m-h+1\ge1 p = d − m − h + 1 ≥ 1 . Assume that for every subspace S S S of dimension at least p p p and every list of at most Q Q Q points a i a_i a i of norm at most R R R there is a unit u ∈ S u\in S u ∈ S with ∣ ⟨ u , a i ⟩ ∣ ≤ τ \lvert\langle u,a_i\rangle\rvert\le\tau ∣⟨ u , a i ⟩∣ ≤ τ for all i i i . Then every deterministic algorithm whose calls ( x , V ) (x,V) ( x , V ) return ( F U ( x ) , V ⊤ ∇ F U ( x ) ) (F_U(x),V^\top\nabla F_U(x)) ( F U ( x ) , V ⊤ ∇ F U ( x )) at cost 1 + rank V 1+\operatorname{rank}V 1 + rank V , and which on every transcript makes at most Q Q Q calls of total cost below h m hm hm , outputs a point with gradient norm greater than b b b for at least one fixed F U F_U F U . Queries, matrices, and the output are unrestricted. The geometric assumption holds, for τ ≥ 0 \tau\ge0 τ ≥ 0 , under any one of the conditions
Q < p ; R > 0 and 2 Q exp ( − p τ 2 2 R 2 ) < 1 ; 13 3 R log ( e + Q / p ) p ≤ τ . Q<p;\qquad R>0\ \text{and}\ 2Q\exp\left(-\frac{p\tau^2}{2R^2}\right)<1;\qquad\frac{13}{3}R\sqrt{\frac{\log(e+Q/p)}{p}}\le\tau. Q < p ; R > 0 and 2 Q exp ( − 2 R 2 p τ 2 ) < 1 ; 3 13 R p log ( e + Q / p ) ≤ τ .
Proof. Maintain the selected directions, the points z ( x ) z(x) z ( x ) of closed stages, and the pending calls of the current stage. If the pending calls have total cost ℓ < h \ell<h ℓ < h , their points z ( x ) z(x) z ( x ) and matrix column spaces together span at most ℓ \ell ℓ dimensions. For a new call ( x , V ) (x,V) ( x , V ) of cost w = 1 + rank V w=1+\operatorname{rank}V w = 1 + rank V : if ℓ + w < h \ell+w<h ℓ + w < h , return the prefix reply ( r j ( x ) , V ⊤ ∇ r j ( x ) ) (r_j(x),V^\top\nabla r_j(x)) ( r j ( x ) , V ⊤ ∇ r j ( x )) and record the call as pending. If ℓ + w ≥ h \ell+w\ge h ℓ + w ≥ h , select u j u_j u j before answering: perpendicular to the preceding frame, to all pending constraints, and to z ( x ) z(x) z ( x ) , and with inner products at most τ \tau τ in magnitude with all points of closed stages. These exact constraints have dimension at most ( m − 1 ) + ( h − 1 ) + 1 (m-1)+(h-1)+1 ( m − 1 ) + ( h − 1 ) + 1 , which leaves dimension at least p p p , so the geometric assumption supplies u j u_j u j . The current column space is not constrained. Return ( r j ( x ) , V ⊤ [ ∇ r j ( x ) − c j ( x ) u j ] ) (r_j(x),V^\top[\nabla r_j(x)-c_j(x)u_j]) ( r j ( x ) , V ⊤ [ ∇ r j ( x ) − c j ( x ) u j ]) , close the stage, move its points to the list of past points, and reset ℓ = 0 \ell=0 ℓ = 0 . Each closed stage consumes at least h h h units of cost, so at most m − 1 m-1 m − 1 stages close.
At the output x ^ \hat{x} x ^ , select the next direction in the same way, with z ( x ^ ) z(\hat{x}) z ( x ^ ) in place of a closing query point. Move the pending points to the list of past points and complete the frame with directions perpendicular to their predecessors and with inner products at most τ \tau τ at every past point; there are at least p p p residual dimensions and at most Q Q Q past points. For the completed frame, every call of stage j j j has zero j j j th coordinate and later coordinates bounded by τ \tau τ , so its reply is the prefix reply together with the term of the current direction. That term vanishes for pending calls, whose column spaces were among the orthogonality constraints, and closing calls received it explicitly. Induction over the deterministic transcript shows that the simulated run is the run on this single function. Its output has a zero hidden coordinate, so its gradient norm exceeds b b b .
Exact orthogonality to fewer than p p p past points is possible inside a p p p -dimensional subspace; the other two conditions are Lemmas C.3 and C.2 , whose bounds only improve in larger subspaces. □ \square □
A budget asserted only on functions in the class reduces to this form for whole transcripts by truncating before any call that would exceed the budget: every finite simulated prefix extends to a fixed function in the class, on which the algorithm terminates within its budget, so the truncation never changes that function’s run. This applies to call caps and cost caps even when their worst cases occur on different functions.
The Chain and the Radial Compression
The deterministic witnesses share the form F U , R , Q ( x ) = g m ( U ⊤ ρ R ( x ) ) + Q ( ∥ x ∥ ) F_{U,R,Q}(x)=g_m(U^\top\rho_R(x))+Q(\lVert x\rVert) F U , R , Q ( x ) = g m ( U ⊤ ρ R ( x )) + Q (∥ x ∥) . Only the public radial parameters differ. All scalar gates are piecewise polynomial; no convolution is needed.
One scalar chain
Define
φ ( t ) = { 0 , t ≤ − 5 / 4 , 2 ( t + 5 / 4 ) 2 , − 5 / 4 < t < − 1 , t + 9 / 8 , − 1 ≤ t ≤ 1 , 9 / 4 − 2 ( 5 / 4 − t ) 2 , 1 < t < 5 / 4 , 9 / 4 , t ≥ 5 / 4 , ψ ( t ) = { 0 , t ≤ 1 / 2 , 8 ( t − 1 / 2 ) 2 , 1 / 2 < t ≤ 3 / 4 , 1 − 8 ( 1 − t ) 2 , 3 / 4 < t < 1 , 1 , t ≥ 1. \varphi(t)=
\begin{cases}
0, & t\le-5/4,\\
2(t+5/4)^2, & -5/4<t<-1,\\
t+9/8, & -1\le t\le1,\\
9/4-2(5/4-t)^2, & 1<t<5/4,\\
9/4, & t\ge5/4,
\end{cases}
\qquad
\psi(t)=
\begin{cases}
0, & t\le1/2,\\
8(t-1/2)^2, & 1/2<t\le3/4,\\
1-8(1-t)^2, & 3/4<t<1,\\
1, & t\ge1.
\end{cases} φ ( t ) = ⎩ ⎨ ⎧ 0 , 2 ( t + 5/4 ) 2 , t + 9/8 , 9/4 − 2 ( 5/4 − t ) 2 , 9/4 , t ≤ − 5/4 , − 5/4 < t < − 1 , − 1 ≤ t ≤ 1 , 1 < t < 5/4 , t ≥ 5/4 , ψ ( t ) = ⎩ ⎨ ⎧ 0 , 8 ( t − 1/2 ) 2 , 1 − 8 ( 1 − t ) 2 , 1 , t ≤ 1/2 , 1/2 < t ≤ 3/4 , 3/4 < t < 1 , t ≥ 1.
and, for m ≥ 1 m\ge1 m ≥ 1 ,
g m ( y ) = − φ ( y 1 ) + ∑ i = 2 m [ ψ ( − y i − 1 ) φ ( − y i ) − ψ ( y i − 1 ) φ ( y i ) ] . (37) g_m(y)=-\varphi(y_1)+\sum_{i=2}^{m}\left[\psi(-y_{i-1})\varphi(-y_i)-\psi(y_{i-1})\varphi(y_i)\right]. \tag*{(37)} g m ( y ) = − φ ( y 1 ) + i = 2 ∑ m [ ψ ( − y i − 1 ) φ ( − y i ) − ψ ( y i − 1 ) φ ( y i ) ] . (37)
The same chain is used for the lower bounds with and without a trace budget; only radial parameters change.
Lemma D.1 (The scalar chain). Let p = ∇ g m ( y ) p=\nabla g_m(y) p = ∇ g m ( y ) and I ( y ) = { i : ∣ y i ∣ ≤ 5 / 4 } I(y)=\{i:|y_i|\le5/4\} I ( y ) = { i : ∣ y i ∣ ≤ 5/4 } . The function g m g_m g m is C 1 , 1 C^{1,1} C 1 , 1 and bounded, and
∥ p ∥ ≤ 10 m , Lip ( ∇ g m ) ≤ 42 , g m ( 0 ) − inf g m ≤ 9 4 ( m − 1 2 ) . \lVert p\rVert\le10\sqrt{m},\qquad\operatorname{Lip}(\nabla g_m)\le42,\qquad g_m(0)-\inf g_m\le\frac{9}{4}\left(m-\frac{1}{2}\right). ∥ p ∥ ≤ 10 m , Lip ( ∇ g m ) ≤ 42 , g m ( 0 ) − inf g m ≤ 4 9 ( m − 2 1 ) . Every p i ≤ 0 p_i\le0 p i ≤ 0 , and ∣ y i ∣ ≥ 5 / 4 |y_i|\ge5/4 ∣ y i ∣ ≥ 5/4 implies p i = 0 p_i=0 p i = 0 . If j j j is the first index with ∣ y j ∣ < 1 |y_j|<1 ∣ y j ∣ < 1 , then p j ≤ − 1 p_j\le-1 p j ≤ − 1 . On each smooth piece, the Hessian B = ∇ 2 g m ( y ) B=\nabla^2g_m(y) B = ∇ 2 g m ( y ) is supported on I ( y ) × I ( y ) I(y)\times I(y) I ( y ) × I ( y ) and satisfies ∥ B ∥ o p ≤ 42 \lVert B\rVert_{\mathrm{op}}\le42 ∥ B ∥ op ≤ 42 . Consequently,
∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m , ∣ y ⊤ p ∣ ≤ 31 4 m . (38) \lVert By\rVert\le44\sqrt{m},\qquad|y^\top By|\le44m,\qquad|y^\top p|\le\frac{31}{4}m. \tag*{(38)} ∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m , ∣ y ⊤ p ∣ ≤ 4 31 m . (38)
Proof. The formulas have matching values and first derivatives at every junction. They give
0 ≤ φ ≤ 9 / 4 , 0 ≤ φ ′ ≤ 1 , ∣ φ ′ ′ ∣ ≤ 4 , 0 ≤ ψ ≤ 1 , 0 ≤ ψ ′ ≤ 4 , ∣ ψ ′ ′ ∣ ≤ 16 0\le\varphi\le9/4,\quad0\le\varphi'\le1,\quad|\varphi''|\le4,\quad0\le\psi\le1,\quad0\le\psi'\le4,\quad|\psi''|\le16 0 ≤ φ ≤ 9/4 , 0 ≤ φ ′ ≤ 1 , ∣ φ ′′ ∣ ≤ 4 , 0 ≤ ψ ≤ 1 , 0 ≤ ψ ′ ≤ 4 , ∣ ψ ′′ ∣ ≤ 16
on their smooth pieces. Moreover, φ ′ \varphi' φ ′ is even and equals one on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , so φ ′ ′ ( t ) \varphi''(t) φ ′′ ( t ) vanishes outside 1 ≤ ∣ t ∣ ≤ 5 / 4 1\le|t|\le5/4 1 ≤ ∣ t ∣ ≤ 5/4 , whereas ψ ′ ( ± t ) \psi'(\pm t) ψ ′ ( ± t ) and ψ ′ ′ ( ± t ) \psi''(\pm t) ψ ′′ ( ± t ) are supported on 1 / 2 ≤ ∣ t ∣ ≤ 1 1/2\le|t|\le1 1/2 ≤ ∣ t ∣ ≤ 1 . At most one of ψ ( t ) , ψ ( − t ) \psi(t),\psi(-t) ψ ( t ) , ψ ( − t ) is nonzero. With γ 0 = 1 \gamma_0=1 γ 0 = 1 and γ i − 1 = ψ ( − y i − 1 ) + ψ ( y i − 1 ) ∈ [ 0 , 1 ] \gamma_{i-1}=\psi(-y_{i-1})+\psi(y_{i-1})\in[0,1] γ i − 1 = ψ ( − y i − 1 ) + ψ ( y i − 1 ) ∈ [ 0 , 1 ] for i > 1 i>1 i > 1 ,
p i = − γ i − 1 φ ′ ( y i ) − 1 { i < m } [ ψ ′ ( − y i ) φ ( − y i + 1 ) + ψ ′ ( y i ) φ ( y i + 1 ) ] . (39) p_i=-\gamma_{i-1}\varphi'(y_i)-\mathbf{1}_{\{i<m\}}\left[\psi'(-y_i)\varphi(-y_{i+1})+\psi'(y_i)\varphi(y_{i+1})\right]. \tag*{(39)} p i = − γ i − 1 φ ′ ( y i ) − 1 { i < m } [ ψ ′ ( − y i ) φ ( − y i + 1 ) + ψ ′ ( y i ) φ ( y i + 1 ) ] . (39)
This proves the sign, the support, the coordinate bound ∣ p i ∣ ≤ 10 |p_i|\le10 ∣ p i ∣ ≤ 10 , and the barrier: at the first index with ∣ y j ∣ < 1 |y_j|<1 ∣ y j ∣ < 1 the preceding gate is saturated. Because ∣ s φ ′ ( s ) ∣ ≤ 1 |s\varphi'(s)|\le1 ∣ s φ ′ ( s ) ∣ ≤ 1 on R \mathbb{R} R (as 4 s ( 5 / 4 − s ) = 1 − 4 ( s − 1 ) ( s − 1 / 4 ) ≤ 1 4s(5/4-s)=1-4(s-1)(s-1/4)\le1 4 s ( 5/4 − s ) = 1 − 4 ( s − 1 ) ( s − 1/4 ) ≤ 1 on [ 1 , 5 / 4 ] [1,5/4] [ 1 , 5/4 ] ) and t ψ ′ ( t ) ≤ 3 t\psi'(t)\le3 t ψ ′ ( t ) ≤ 3 on [ 1 / 2 , 1 ] [1/2,1] [ 1/2 , 1 ] (as 16 t ( 1 − t ) = 3 − 16 ( t − 3 / 4 ) ( t − 1 / 4 ) ≤ 3 16t(1-t)=3-16(t-3/4)(t-1/4)\le3 16 t ( 1 − t ) = 3 − 16 ( t − 3/4 ) ( t − 1/4 ) ≤ 3 on [ 3 / 4 , 1 ] [3/4,1] [ 3/4 , 1 ] ), each coordinate satisfies ∣ y i p i ∣ ≤ 1 + 3 ( 9 / 4 ) = 31 / 4 |y_ip_i|\le1+3(9/4)=31/4 ∣ y i p i ∣ ≤ 1 + 3 ( 9/4 ) = 31/4 , which sums to ∣ y ⊤ p ∣ ≤ ( 31 / 4 ) m |y^\top p|\le(31/4)m ∣ y ⊤ p ∣ ≤ ( 31/4 ) m .
The Hessian B B B is symmetric tridiagonal with ∣ B i , i − 1 ∣ ≤ 4 |B_{i,i-1}|\le4 ∣ B i , i − 1 ∣ ≤ 4 , and B i , i − 1 ≠ 0 B_{i,i-1}\ne0 B i , i − 1 = 0 requires ∣ y i − 1 ∣ ≤ 1 |y_{i-1}|\le1 ∣ y i − 1 ∣ ≤ 1 . Since the supports of φ ′ ′ ( y i ) \varphi''(y_i) φ ′′ ( y i ) and ψ ′ ′ ( ± y i ) \psi''(\pm y_i) ψ ′′ ( ± y i ) meet only at ∣ y i ∣ = 1 |y_i|=1 ∣ y i ∣ = 1 , there are two cases for row i i i . If ∣ y i ∣ ≤ 1 |y_i|\le1 ∣ y i ∣ ≤ 1 , then φ ′ ′ ( y i ) = 0 \varphi''(y_i)=0 φ ′′ ( y i ) = 0 , and with s = ± y i + 1 s=\pm y_{i+1} s = ± y i + 1 one has
∣ B i , i − 1 ∣ + ∣ B i i ∣ + ∣ B i , i + 1 ∣ ≤ 4 + 16 φ ( s ) + 4 φ ′ ( s ) ≤ 42 , (40) |B_{i,i-1}|+|B_{ii}|+|B_{i,i+1}|\le4+16\varphi(s)+4\varphi'(s)\le42, \tag*{(40)} ∣ B i , i − 1 ∣ + ∣ B ii ∣ + ∣ B i , i + 1 ∣ ≤ 4 + 16 φ ( s ) + 4 φ ′ ( s ) ≤ 42 , (40)
because 4 + 16 φ ( s ) + 4 φ ′ ( s ) 4+16\varphi(s)+4\varphi'(s) 4 + 16 φ ( s ) + 4 φ ′ ( s ) is at most 4 + 16 ( 17 / 8 ) + 4 = 42 4+16(17/8)+4=42 4 + 16 ( 17/8 ) + 4 = 42 for s ≤ 1 s\le1 s ≤ 1 , equals 42 − 32 ( s − 1 ) 2 ≤ 42 42-32(s-1)^2\le42 42 − 32 ( s − 1 ) 2 ≤ 42 on [ 1 , 5 / 4 ] [1,5/4] [ 1 , 5/4 ] , and equals 40 40 40 for s ≥ 5 / 4 s\ge5/4 s ≥ 5/4 . In the same case ∣ y i − 1 ∣ ≤ 1 |y_{i-1}|\le1 ∣ y i − 1 ∣ ≤ 1 , ∣ y i ∣ ≤ 1 |y_i|\le1 ∣ y i ∣ ≤ 1 , and ∣ y i + 1 φ ′ ( ± y i + 1 ) ∣ ≤ 1 |y_{i+1}\varphi'(\pm y_{i+1})|\le1 ∣ y i + 1 φ ′ ( ± y i + 1 ) ∣ ≤ 1 , so ∣ ( B y ) i ∣ ≤ 4 ⋅ 1 + 36 ⋅ 1 + 4 ⋅ 1 = 44 |(By)_i|\le4\cdot1+36\cdot1+4\cdot1=44 ∣ ( B y ) i ∣ ≤ 4 ⋅ 1 + 36 ⋅ 1 + 4 ⋅ 1 = 44 and ∣ y i ( B y ) i ∣ ≤ 44 |y_i(By)_i|\le44 ∣ y i ( B y ) i ∣ ≤ 44 . If 1 < ∣ y i ∣ ≤ 5 / 4 1<|y_i|\le5/4 1 < ∣ y i ∣ ≤ 5/4 , then ψ ′ ( ± y i ) = ψ ′ ′ ( ± y i ) = 0 \psi'(\pm y_i)=\psi''(\pm y_i)=0 ψ ′ ( ± y i ) = ψ ′′ ( ± y i ) = 0 , so B i , i + 1 = 0 B_{i,i+1}=0 B i , i + 1 = 0 , ∣ B i i ∣ ≤ 4 |B_{ii}|\le4 ∣ B ii ∣ ≤ 4 , the absolute row sum is at most 8 8 8 , ∣ ( B y ) i ∣ ≤ 4 ( 5 / 4 ) + 4 ⋅ 1 = 9 |(By)_i|\le4(5/4)+4\cdot1=9 ∣ ( B y ) i ∣ ≤ 4 ( 5/4 ) + 4 ⋅ 1 = 9 , and ∣ y i ( B y ) i ∣ ≤ 45 / 4 < 44 |y_i(By)_i|\le45/4<44 ∣ y i ( B y ) i ∣ ≤ 45/4 < 44 . Thus ∥ B ∥ o p ≤ 42 \|B\|_{\mathrm{op}}\le42 ∥ B ∥ op ≤ 42 , and (38) holds. Matching derivatives extend ∥ B ∥ o p ≤ 42 \|B\|_{\mathrm{op}}\le42 ∥ B ∥ op ≤ 42 to Lip ( ∇ g m ) ≤ 42 \operatorname{Lip}(\nabla g_m)\le42 Lip ( ∇ g m ) ≤ 42 by integration over the finitely many polynomial pieces of a line segment. Finally g m ( 0 ) = − 9 / 8 g_m(0)=-9/8 g m ( 0 ) = − 9/8 , and each of the m m m summands of g m g_m g m is at least − 9 / 4 -9/4 − 9/4 .
Radial compression and shifted Huber potentials
For R > 0 R>0 R > 0 write
a ( x ) = ( 1 + ∥ x ∥ 2 / R 2 ) − 1 / 2 , ρ R ( x ) = a ( x ) x . (41) a(x)=(1+\|x\|^2/R^2)^{-1/2},\qquad\rho_R(x)=a(x)x. \tag*{(41)} a ( x ) = ( 1 + ∥ x ∥ 2 / R 2 ) − 1/2 , ρ R ( x ) = a ( x ) x . (41)
For λ , s > 0 \lambda,s>0 λ , s > 0 and r 0 ≥ 0 r_0\ge0 r 0 ≥ 0 , define the shifted radial Huber potential ([25 ] ) through its derivative,
Q ′ ( r ) = λ min { ( r − r 0 ) + , s } , Q ( 0 ) = 0 , q ( x ) = Q ( ∥ x ∥ ) . (42) Q'(r)=\lambda\min\{(r-r_0)_+,s\},\qquad Q(0)=0,\qquad q(x)=Q(\|x\|). \tag*{(42)} Q ′ ( r ) = λ min {( r − r 0 ) + , s } , Q ( 0 ) = 0 , q ( x ) = Q ( ∥ x ∥ ) . (42)
It vanishes on the ball of radius r 0 r_0 r 0 , is quadratic across an annulus of width s s s , and is linear afterwards. The construction with a trace budget below uses a related sum of two such potentials; the unrestricted choice uses one flat on an inner ball.
Lemma D.2 (Radial derivatives). The potential (42) is nonnegative, convex, coercive, and C 1 , 1 C^{1,1} C 1 , 1 , with ∥ ∇ q ∥ ≤ λ s \|\nabla q\|\le\lambda s ∥∇ q ∥ ≤ λ s and Lip ( ∇ q ) ≤ λ \operatorname{Lip}(\nabla q)\le\lambda Lip ( ∇ q ) ≤ λ . For z = ρ R ( x ) z=\rho_R(x) z = ρ R ( x ) ,
D ρ R ( x ) = a ( x ) ( I − z z T R 2 ) , ∥ D ρ R ( x ) ∥ o p ≤ 1 , ∥ D 2 ρ R ( x ) [ v , w ] ∥ ≤ 2 3 R ∥ v ∥ ∥ w ∥ . (43) D\rho_R(x)=a(x)\left(I-\frac{zz^{\mathsf T}}{R^2}\right),\qquad\|D\rho_R(x)\|_{\mathrm{op}}\le1,\qquad\|D^2\rho_R(x)[v,w]\|\le\frac{2}{\sqrt{3}R}\|v\|\|w\|. \tag*{(43)} D ρ R ( x ) = a ( x ) ( I − R 2 z z T ) , ∥ D ρ R ( x ) ∥ op ≤ 1 , ∥ D 2 ρ R ( x ) [ v , w ] ∥ ≤ 3 R 2 ∥ v ∥∥ w ∥. (43) Proof. For r > 0 r>0 r > 0 , ∇ q = Q ′ ( r ) x / r \nabla q=Q'(r)x/r ∇ q = Q ′ ( r ) x / r . On every smooth piece its radial and tangential Hessian eigenvalues are Q ′ ′ ( r ) Q''(r) Q ′′ ( r ) and Q ′ ( r ) / r Q'(r)/r Q ′ ( r ) / r , both in [ 0 , λ ] [0,\lambda] [ 0 , λ ] , because Q ′ Q' Q ′ is nondecreasing, Q ′ ′ ∈ { 0 , λ } Q''\in\{0,\lambda\} Q ′′ ∈ { 0 , λ } , and Q ′ ( r ) ≤ λ r Q'(r)\le\lambda r Q ′ ( r ) ≤ λ r . The gradient is continuous across the junction spheres, and integrating along line segments gives the Lipschitz bound. The other properties follow from Q ′ ≥ 0 Q'\ge0 Q ′ ≥ 0 and Q ′ ( r ) = λ s Q'(r)=\lambda s Q ′ ( r ) = λ s for large r r r .
Differentiation gives the formula for D ρ R D\rho_R D ρ R , whose eigenvalues are a a a tangentially and a 3 a^3 a 3 radially. A second differentiation gives
D 2 ρ R ( x ) [ v , w ] = − a 3 R 2 [ ( x T w ) v + ( x T v ) w + ( v T w ) x ] + 3 a 5 R 4 ( x T v ) ( x T w ) x . D^2\rho_R(x)[v,w]=-\frac{a^3}{R^2}\left[(x^{\mathsf T}w)v+(x^{\mathsf T}v)w+(v^{\mathsf T}w)x\right]+\frac{3a^5}{R^4}(x^{\mathsf T}v)(x^{\mathsf T}w)x. D 2 ρ R ( x ) [ v , w ] = − R 2 a 3 [ ( x T w ) v + ( x T v ) w + ( v T w ) x ] + R 4 3 a 5 ( x T v ) ( x T w ) x . For a unit v v v and x ≠ 0 x\ne0 x = 0 , put t = ∥ x ∥ / R t=\|x\|/R t = ∥ x ∥/ R , C = ( x T v / ∥ x ∥ ) 2 C=(x^{\mathsf T}v/\|x\|)^2 C = ( x T v /∥ x ∥ ) 2 , and z = t 2 / ( 1 + t 2 ) z=t^2/(1+t^2) z = t 2 / ( 1 + t 2 ) . Then
∥ D 2 ρ R ( x ) [ v , v ] ∥ 2 = t 2 R 2 ( 1 + t 2 ) 3 [ 1 + ( 8 − 6 z ) C + ( 9 z 2 − 12 z ) C 2 ] . \left\|D^2\rho_R(x)[v,v]\right\|^2=\frac{t^2}{R^2(1+t^2)^3}\left[1+(8-6z)C+(9z^2-12z)C^2\right]. D 2 ρ R ( x ) [ v , v ] 2 = R 2 ( 1 + t 2 ) 3 t 2 [ 1 + ( 8 − 6 z ) C + ( 9 z 2 − 12 z ) C 2 ] . For fixed C ∈ [ 0 , 1 ] C\in[0,1] C ∈ [ 0 , 1 ] the bracket is convex in z ∈ [ 0 , 1 ] z\in[0,1] z ∈ [ 0 , 1 ] ; at the endpoints it equals 1 + 8 C ≤ 9 1+8C\le9 1 + 8 C ≤ 9 and 1 + 2 C − 3 C 2 ≤ 4 / 3 1+2C-3C^2\le4/3 1 + 2 C − 3 C 2 ≤ 4/3 . Hence the diagonal norm is at most 3 t / [ R ( 1 + t 2 ) 3 / 2 ] ≤ 2 / ( 3 R ) 3t/[R(1+t^2)^{3/2}]\le2/(\sqrt{3}R) 3 t / [ R ( 1 + t 2 ) 3/2 ] ≤ 2/ ( 3 R ) , and it vanishes at x = 0 x=0 x = 0 . Polarization, D 2 ρ R [ v , w ] = 1 4 ( D 2 ρ R [ v + w , v + w ] − D 2 ρ R [ v − w , v − w ] ) D^2\rho_R[v,w]=\frac14(D^2\rho_R[v+w,v+w]-D^2\rho_R[v-w,v-w]) D 2 ρ R [ v , w ] = 4 1 ( D 2 ρ R [ v + w , v + w ] − D 2 ρ R [ v − w , v − w ]) , and the parallelogram law give the same bound for unit v , w v,w v , w .
Integration on finitely many smooth pieces
Lemma D.3 (Integration over finitely many pieces). Let F : R d → R F:\mathbb{R}^{d}\to\mathbb{R} F : R d → R be continuously differentiable. Suppose that along every segment x + t v x+tv x + t v , 0 ≤ t ≤ 1 0\le t\le1 0 ≤ t ≤ 1 , its gradient is continuously differentiable except at finitely many points, and that on each remaining interval
d d t ∇ F ( x + t v ) = B ( t ) v , ∥ B ( t ) ∥ op ≤ L 0 , B ( t ) ⪯ H , \frac{d}{dt}\nabla F(x+tv)=B(t)v,\qquad\lVert B(t)\rVert_{\operatorname{op}}\le L_0,\qquad B(t)\preceq H, d t d ∇ F ( x + t v ) = B ( t ) v , ∥ B ( t ) ∥ op ≤ L 0 , B ( t ) ⪯ H , where B ( t ) B(t) B ( t ) is symmetric and H H H is a fixed symmetric matrix. Then Lip ( ∇ F ) ≤ L 0 \operatorname{Lip}(\nabla F)\le L_0 Lip ( ∇ F ) ≤ L 0 and F ( x + v ) ≤ F ( x ) + ∇ F ( x ) T v + 1 2 v T H v F(x+v)\le F(x)+\nabla F(x)^{\mathsf{T}}v+\frac{1}{2}v^{\mathsf{T}}Hv F ( x + v ) ≤ F ( x ) + ∇ F ( x ) T v + 2 1 v T H v .
Proof. Apply the fundamental theorem of calculus on each interval and add; the endpoint terms cancel by continuity of the gradient. This gives ∥ ∇ F ( x + t v ) − ∇ F ( x ) ∥ ≤ L 0 t ∥ v ∥ \lVert\nabla F(x+tv)-\nabla F(x)\rVert\le L_0t\lVert v\rVert ∥ ∇ F ( x + t v ) − ∇ F ( x )∥ ≤ L 0 t ∥ v ∥ and v T [ ∇ F ( x + t v ) − ∇ F ( x ) ] ≤ t v T H v v^{\mathsf{T}}[\nabla F(x+tv)-\nabla F(x)]\le tv^{\mathsf{T}}Hv v T [ ∇ F ( x + t v ) − ∇ F ( x )] ≤ t v T H v . Integrating the second inequality over t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] proves the upper model.
The lemma applies to every function built from (37) , ρ R \rho_R ρ R , and (42) . On a segment, each compressed coordinate has the form ( α + β t ) / ( 1 + ∥ x + t v ∥ 2 / R 2 ) 1 / 2 (\alpha+\beta t)/(1+\lVert x+tv\rVert^2/R^2)^{1/2} ( α + β t ) / ( 1 + ∥ x + t v ∥ 2 / R 2 ) 1/2 . Squaring shows that it equals a nonzero scalar junction value at most twice, unless it does so identically; in that case its derivative along the segment vanishes, and either adjacent one-sided second derivative gives the correct expression from the chain rule. Crossings of a Huber junction sphere are roots of a quadratic polynomial as well. The resulting finite partition makes the gradient smooth on each open interval.
Sublevel parameter choice
Theorem D.4 (Wrapper with a sublevel trace bound). Let m ≥ 1 m\ge1 m ≥ 1 and R 2 ≥ 64 m R^2\ge64m R 2 ≥ 64 m . Set r 0 = 2 3 R r_0=\frac{2}{3}R r 0 = 3 2 R , μ = 11 m / R 2 \mu=11m/R^2 μ = 11 m / R 2 , and define a public radial potential q ( x ) = Q ( ∥ x ∥ ) q(x)=Q(\lVert x\rVert) q ( x ) = Q (∥ x ∥) by
Q ( 0 ) = 0 , Q ′ ( r ) = μ min { r , r 0 } + min { ( r − r 0 ) + , 1 } . (44) Q(0)=0,\qquad Q'(r)=\mu\min\{r,r_0\}+\min\{(r-r_0)_+,1\}. \tag*{(44)} Q ( 0 ) = 0 , Q ′ ( r ) = μ min { r , r 0 } + min {( r − r 0 ) + , 1 } . (44) For an orthonormal frame U U U set
F U ( x ) = g m ( U T ρ R ( x ) ) + q ( x ) . F_U(x)=g_m(U^{\mathsf{T}}\rho_R(x))+q(x). F U ( x ) = g m ( U T ρ R ( x )) + q ( x ) . Then F U F_U F U is coercive, nonconvex, and C 1 , 1 C^{1,1} C 1 , 1 , and
Lip ( ∇ F U ) < 45 , ∥ ∇ F U ∥ ≤ 12 m , (45) \operatorname{Lip}(\nabla F_U)<45,\qquad\lVert\nabla F_U\rVert\le12\sqrt{m}, \tag*{(45)} Lip ( ∇ F U ) < 45 , ∥ ∇ F U ∥ ≤ 12 m , (45) F U ( 0 ) − inf F U ≤ 9 4 ( m − 1 2 ) , { F U ≤ F U ( 0 ) } ⊂ { ∥ x ∥ < 2 3 R } . (46) F_U(0)-\inf F_U\le\frac{9}{4}\left(m-\frac{1}{2}\right),\qquad\{F_U\le F_U(0)\}\subset\{\lVert x\rVert<\tfrac{2}{3}R\}. \tag*{(46)} F U ( 0 ) − inf F U ≤ 4 9 ( m − 2 1 ) , { F U ≤ F U ( 0 )} ⊂ {∥ x ∥ < 3 2 R } . (46) On the initial sublevel, almost everywhere, one fixed matrix satisfies
∇ 2 F U ⪯ K U : = 54 U U T + 80 m R 2 I , ∥ K U ∥ op < 56. (47) \nabla^2F_U\preceq K_U:=54UU^{\mathsf{T}}+\frac{80m}{R^2}I,\qquad\lVert K_U\rVert_{\operatorname{op}}<56. \tag*{(47)} ∇ 2 F U ⪯ K U := 54 U U T + R 2 80 m I , ∥ K U ∥ op < 56. (47) Consequently,
tr [ ( ∇ 2 F U ) + ] ≤ 54 m + 80 d m R 2 . (48) \operatorname{tr}[(\nabla^2F_U)_+]\le54m+\frac{80dm}{R^2}. \tag*{(48)} tr [( ∇ 2 F U ) + ] ≤ 54 m + R 2 80 d m . (48) If any compressed coordinate has magnitude below one, then at any x ∈ R d x\in\mathbb{R}^{d} x ∈ R d ,
∥ ∇ F U ( x ) ∥ > 3 / 4. (49) \lVert\nabla F_U(x)\rVert>3/4. \tag*{(49)} ∥ ∇ F U ( x )∥ > 3/4. (49) Finally its exact prefix replies have the following form: if y = U T ρ R ( x ) y=U^{\mathsf{T}}\rho_R(x) y = U T ρ R ( x ) , y j = 0 y_j=0 y j = 0 , and ∣ y k ∣ ≤ 1 / 2 \lvert y_k\rvert\le1/2 ∣ y k ∣ ≤ 1/2 for k > j k>j k > j , then, with r j ( x ) = g m ( y 1 , … , y j − 1 , 0 , … , 0 ) + q ( x ) r_j(x)=g_m(y_1,\ldots,y_{j-1},0,\ldots,0)+q(x) r j ( x ) = g m ( y 1 , … , y j − 1 , 0 , … , 0 ) + q ( x ) ,
F U ( x ) = r j ( x ) , ∇ F U ( x ) = ∇ r j ( x ) − a ( x ) γ j − 1 ( x ) u j . (50) F_U(x)=r_j(x),\qquad\nabla F_U(x)=\nabla r_j(x)-a(x)\gamma_{j-1}(x)u_j. \tag*{(50)} F U ( x ) = r j ( x ) , ∇ F U ( x ) = ∇ r j ( x ) − a ( x ) γ j − 1 ( x ) u j . (50)
Proof. We have R ≥ 8 m ≥ 8 R \ge8\sqrt{m} \ge8 R ≥ 8 m ≥ 8 , μ ≤ 11 / 64 \mu\le11/64 μ ≤ 11/64 , and μ r 0 = 22 m / ( 3 R ) ≤ ( 11 / 12 ) m < m \mu r_0 = 22m/(3R) \le(11/12)\sqrt{m} < \sqrt{m} μ r 0 = 22 m / ( 3 R ) ≤ ( 11/12 ) m < m . The derivative Q ′ Q' Q ′ is continuous, nonnegative and nondecreasing, with slopes μ \mu μ , 1 1 1 , and 0 0 0 on its successive intervals. Also Q ′ ( r ) ≤ r Q'(r) \le r Q ′ ( r ) ≤ r , and its final value is μ r 0 + 1 < 2 m \mu r_0 + 1 < 2\sqrt{m} μ r 0 + 1 < 2 m . The radial and tangential Hessian eigenvalues of q q q are Q ′ ′ ( r ) Q''(r) Q ′′ ( r ) and Q ′ ( r ) / r Q'(r)/r Q ′ ( r ) / r , so its gradient is globally 1 1 1 -Lipschitz and bounded by 2 m 2\sqrt{m} 2 m . Thus
Lip ( ∇ F U ) ≤ 42 + 20 m 3 R + 1 ≤ 43 + 5 2 3 < 45. \operatorname{Lip}(\nabla F_U) \le42 + \frac{20\sqrt{m}}{\sqrt{3}R} + 1 \le43 + \frac{5}{2\sqrt{3}} < 45. Lip ( ∇ F U ) ≤ 42 + 3 R 20 m + 1 ≤ 43 + 2 3 5 < 45.
The gradient bound, gap, and coercivity follow from the chain bounds, q ≥ 0 q \ge0 q ≥ 0 , and its positive linear growth. Moreover Q ( r 0 ) = μ r 0 2 / 2 = 22 m / 9 Q(r_0) = \mu r_0^2/2 = 22m/9 Q ( r 0 ) = μ r 0 2 /2 = 22 m /9 , so for r ≥ r 0 r \ge r_0 r ≥ r 0 , F U ( x ) ≥ − 9 m / 4 + 22 m / 9 = 7 m / 36 > 0 > F U ( 0 ) = − 9 / 8 F_U(x) \ge-9m/4 + 22m/9 = 7m/36 > 0 > F_U(0) = -9/8 F U ( x ) ≥ − 9 m /4 + 22 m /9 = 7 m /36 > 0 > F U ( 0 ) = − 9/8 . This proves the sublevel inclusion. Consequently throughout the indicated sublevel the penalty is μ ∥ x ∥ 2 / 2 \mu\|x\|^2/2 μ ∥ x ∥ 2 /2 .
For the sublevel Hessian bound write p = ∇ g m ( y ) p = \nabla g_m(y) p = ∇ g m ( y ) , B = ∇ 2 g m ( y ) B = \nabla^2 g_m(y) B = ∇ 2 g m ( y ) , α = U ⊤ v \alpha= U^\top v α = U ⊤ v , and β = a 2 x ⊤ v / R 2 \beta= a^2x^\top v/R^2 β = a 2 x ⊤ v / R 2 . Then U ⊤ D ρ R v = a α − β y U^\top D\rho_Rv = a\alpha- \beta y U ⊤ D ρ R v = a α − β y , and on { ∥ x ∥ < 2 3 R } \{\|x\| < \frac{2}{3}R\} { ∥ x ∥ < 3 2 R } one has ∣ β ∣ ≤ 2 / 3 1 + 4 / 9 ∥ v ∥ R = 6 13 ∥ v ∥ R |\beta| \le\frac{2/3}{1+4/9}\frac{\|v\|}{R} = \frac{6}{13}\frac{\|v\|}{R} ∣ β ∣ ≤ 1 + 4/9 2/3 R ∥ v ∥ = 13 6 R ∥ v ∥ and a 2 ≥ 9 13 a^2 \ge\frac{9}{13} a 2 ≥ 13 9 . The second term of the chain rule for the Hessian equals
⟨ U p , D 2 ρ R [ v , v ] ⟩ = − 2 a β p ⊤ α + ( 3 β 2 − a 2 ∥ v ∥ 2 / R 2 ) y ⊤ p . \left\langle Up,D^2\rho_R[v,v]\right\rangle= -2a\beta p^\top\alpha+ (3\beta^2-a^2\|v\|^2/R^2)y^\top p. ⟨ U p , D 2 ρ R [ v , v ] ⟩ = − 2 a β p ⊤ α + ( 3 β 2 − a 2 ∥ v ∥ 2 / R 2 ) y ⊤ p .
Since 0 ≤ 3 β 2 ≤ 108 169 ∥ v ∥ 2 R 2 < 9 13 ∥ v ∥ 2 R 2 ≤ a 2 ∥ v ∥ 2 R 2 ≤ ∥ v ∥ 2 R 2 0 \le3\beta^2 \le\frac{108}{169}\frac{\|v\|^2}{R^2} < \frac{9}{13}\frac{\|v\|^2}{R^2} \le\frac{a^2\|v\|^2}{R^2} \le\frac{\|v\|^2}{R^2} 0 ≤ 3 β 2 ≤ 169 108 R 2 ∥ v ∥ 2 < 13 9 R 2 ∥ v ∥ 2 ≤ R 2 a 2 ∥ v ∥ 2 ≤ R 2 ∥ v ∥ 2 , we have ∣ 3 β 2 − a 2 ∥ v ∥ 2 / R 2 ∣ ≤ ∥ v ∥ 2 / R 2 |3\beta^2-a^2\|v\|^2/R^2| \le\|v\|^2/R^2 ∣3 β 2 − a 2 ∥ v ∥ 2 / R 2 ∣ ≤ ∥ v ∥ 2 / R 2 . Combining the cross terms − 2 a β α ⊤ ( B y + p ) -2a\beta\alpha^\top(By+p) − 2 a β α ⊤ ( B y + p ) via Young’s inequality with parameter γ = 2 / 9 \gamma= 2/9 γ = 2/9 and using ∥ B ∥ o p ≤ 42 \|B\|_{\mathrm{op}} \le42 ∥ B ∥ op ≤ 42 , ∥ B y ∥ + ∥ p ∥ ≤ 54 m \|By\|+\|p\| \le54\sqrt{m} ∥ B y ∥ + ∥ p ∥ ≤ 54 m , ∣ y ⊤ B y ∣ ≤ 44 m |y^\top By| \le44m ∣ y ⊤ B y ∣ ≤ 44 m , and ∣ y ⊤ p ∣ ≤ 31 m / 4 |y^\top p| \le31m/4 ∣ y ⊤ p ∣ ≤ 31 m /4 gives
( a α − β y ) ⊤ B ( a α − β y ) + ⟨ U p , D 2 ρ R [ v , v ] ⟩ ≤ ( 42 + 54 γ ) ∥ α ∥ 2 + [ ( 54 γ + 44 ) 36 169 + 31 4 ] m R 2 ∥ v ∥ 2 . (a\alpha-\beta y)^\top B(a\alpha-\beta y)+\left\langle Up,D^2\rho_R[v,v]\right\rangle\le(42+54\gamma)\|\alpha\|^2+\left[\left(\frac{54}{\gamma}+44\right)\frac{36}{169}+\frac{31}{4}\right]\frac{m}{R^2}\|v\|^2. ( a α − β y ) ⊤ B ( a α − β y ) + ⟨ U p , D 2 ρ R [ v , v ] ⟩ ≤ ( 42 + 54 γ ) ∥ α ∥ 2 + [ ( γ 54 + 44 ) 169 36 + 4 31 ] R 2 m ∥ v ∥ 2 .
Adding μ I = ( 11 m / R 2 ) I \mu I = (11m/R^2)I μ I = ( 11 m / R 2 ) I and noting that 42 + 54 ( 2 / 9 ) = 54 42+54(2/9)=54 42 + 54 ( 2/9 ) = 54 and 11 + 287 ( 36 / 169 ) + 31 / 4 = 54003 / 676 < 80 11+287(36/169)+31/4=54003/676<80 11 + 287 ( 36/169 ) + 31/4 = 54003/676 < 80 gives ∇ 2 F U ⪯ 54 U U ⊤ + ( 80 m / R 2 ) I \nabla^2F_U \preceq54UU^\top+(80m/R^2)I ∇ 2 F U ⪯ 54 U U ⊤ + ( 80 m / R 2 ) I on the initial sublevel, with operator norm at most 54 + 80 / 64 < 56 54+80/64<56 54 + 80/64 < 56 . The same calculation bounds the absolute value of the quadratic form. For symmetric A ⪯ H A\preceq H A ⪯ H , H ⪰ 0 H\succeq0 H ⪰ 0 , one has tr A + ≤ tr H \operatorname{tr} A_+ \le\operatorname{tr} H tr A + ≤ tr H : trace against the positive spectral projection of A A A . This proves (48) . For global smoothness one may directly use ∇ ( g m ∘ U ⊤ ρ R ) = D ρ R ⊤ U ∇ g m ( U ⊤ ρ R ) \nabla(g_m\circ U^\top\rho_R)=D\rho_R^\top U\nabla g_m(U^\top\rho_R) ∇ ( g m ∘ U ⊤ ρ R ) = D ρ R ⊤ U ∇ g m ( U ⊤ ρ R ) : D ρ R D\rho_R D ρ R has norm at most one and Lipschitz constant 2 / ( 3 R ) 2/(\sqrt{3}R) 2/ ( 3 R ) , while ∇ g m \nabla g_m ∇ g m is bounded and 42 42 42 -Lipschitz. The product rule for differences gives the displayed global bound. For the trace estimate, the finitely many scalar junction hyperplanes have null preimages under U ⊤ ρ R U^\top\rho_R U ⊤ ρ R , a smooth submersion; the radial junction spheres are null as well.
For the barrier on the whole space, let j j j be the first unfinished coordinate, so p j ≤ − 1 p_j\le-1 p j ≤ − 1 and ∣ y j ∣ < 1 |y_j|<1 ∣ y j ∣ < 1 . Write r = ∥ x ∥ r=\|x\| r = ∥ x ∥ , w = ∥ ρ R ( x ) ∥ = a r w=\|\rho_R(x)\|=ar w = ∥ ρ R ( x ) ∥ = a r , and g = ∇ F U ( x ) = D ρ R ( x ) U p + Q ′ ( r ) r x g=\nabla F_U(x)=D\rho_R(x)Up+\frac{Q'(r)}{r}x g = ∇ F U ( x ) = D ρ R ( x ) U p + r Q ′ ( r ) x . If w 2 ≤ 13 4 m w^2\le\frac{13}{4}m w 2 ≤ 4 13 m , then since w ( r 0 ) 2 = 4 13 R 2 ≥ 256 13 m > 13 4 m w(r_0)^2=\frac{4}{13}R^2\ge\frac{256}{13}m>\frac{13}{4}m w ( r 0 ) 2 = 13 4 R 2 ≥ 13 256 m > 4 13 m , we have r < r 0 r<r_0 r < r 0 , Q ′ ( r ) / r = μ ≤ 11 / 64 Q'(r)/r=\mu\le11/64 Q ′ ( r ) / r = μ ≤ 11/64 , and a 2 = 1 − w 2 / R 2 ≥ 243 / 256 a^2=1-w^2/R^2\ge243/256 a 2 = 1 − w 2 / R 2 ≥ 243/256 . Since D ρ R ( x ) − 1 u j = a − 1 ( u j + a − 1 y j x / R 2 ) D\rho_R(x)^{-1}u_j=a^{-1}(u_j+a^{-1}y_jx/R^2) D ρ R ( x ) − 1 u j = a − 1 ( u j + a − 1 y j x / R 2 ) satisfies ⟨ D ρ R ( x ) − 1 u j , x ⟩ = a − 4 y j \langle D\rho_R(x)^{-1}u_j,x\rangle=a^{-4}y_j ⟨ D ρ R ( x ) − 1 u j , x ⟩ = a − 4 y j ,
∣ ⟨ D ρ R ( x ) − 1 u j , g ⟩ ∣ = ∣ p j + a − 4 μ y j ∣ ≥ 1 − 11 64 ( 256 243 ) 2 , (51) \left|\left\langle D\rho_R(x)^{-1}u_j,g\right\rangle\right|=\left|p_j+a^{-4}\mu y_j\right|\ge1-\frac{11}{64}\left(\frac{256}{243}\right)^2, \tag*{(51)} ⟨ D ρ R ( x ) − 1 u j , g ⟩ = p j + a − 4 μ y j ≥ 1 − 64 11 ( 243 256 ) 2 , (51)
while ∥ D ρ R ( x ) − 1 u j ∥ 2 = a − 2 [ 1 + ( 2 − a 2 ) a − 4 y j 2 / R 2 ] ≤ 256 243 [ 1 + 259 256 ⋅ 4096 ( 256 243 ) 2 ] \|D\rho_R(x)^{-1}u_j\|^2=a^{-2}[1+(2-a^2)a^{-4}y_j^2/R^2]\le\frac{256}{243}[1+\frac{259}{256\cdot4096}(\frac{256}{243})^2] ∥ D ρ R ( x ) − 1 u j ∥ 2 = a − 2 [ 1 + ( 2 − a 2 ) a − 4 y j 2 / R 2 ] ≤ 243 256 [ 1 + 256 ⋅ 4096 259 ( 243 256 ) 2 ] , giving ∥ g ∥ > 3 / 4 \|g\|>3/4 ∥ g ∥ > 3/4 . Next suppose w 2 ≥ 13 4 m w^2\ge\frac{13}{4}m w 2 ≥ 4 13 m and r ≤ r 0 + 1 r\le r_0+1 r ≤ r 0 + 1 . Let I = { i : ∣ y i ∣ ≤ 5 / 4 } I=\{i:|y_i|\le5/4\} I = { i : ∣ y i ∣ ≤ 5/4 } and W = span { u i : i ∈ I } W=\operatorname{span}\{u_i:i\in I\} W = span { u i : i ∈ I } . Since g = a U p + c x g=aUp+cx g = a U p + c x for a scalar c c c and D 2 : = ∥ P W ⊥ ρ R ( x ) ∥ 2 ≥ w 2 − 25 16 m D^2:=\|P_{W^\perp}\rho_R(x)\|^2\ge w^2-\frac{25}{16}m D 2 := ∥ P W ⊥ ρ R ( x ) ∥ 2 ≥ w 2 − 16 25 m , projecting g g g onto the orthogonal directions u j u_j u j and P W ⊥ x P_{W^\perp}x P W ⊥ x gives
∥ g ∥ 2 ≥ a 2 p j 2 D 2 D 2 + y j 2 ≥ ( 1 − w 2 R 2 ) w 2 / m − 25 / 16 w 2 / m − 9 / 16 . (52) \|g\|^2\ge\frac{a^2p_j^2D^2}{D^2+y_j^2}\ge\left(1-\frac{w^2}{R^2}\right)\frac{w^2/m-25/16}{w^2/m-9/16}. \tag*{(52)} ∥ g ∥ 2 ≥ D 2 + y j 2 a 2 p j 2 D 2 ≥ ( 1 − R 2 w 2 ) w 2 / m − 9/16 w 2 / m − 25/16 . (52)
The right-hand side is concave in w 2 w^2 w 2 on [ 13 4 m , w ( r 0 + 1 ) 2 ] [\frac{13}{4}m,w(r_0+1)^2] [ 4 13 m , w ( r 0 + 1 ) 2 ] . At w 2 = 13 4 m w^2=\frac{13}{4}m w 2 = 4 13 m it is at least 243 256 ⋅ 27 43 = 6561 11008 > 9 16 \frac{243}{256}\cdot\frac{27}{43}=\frac{6561}{11008}>\frac{9}{16} 256 243 ⋅ 43 27 = 11008 6561 > 16 9 ; at r = r 0 + 1 r=r_0+1 r = r 0 + 1 , since r / R ≤ 2 / 3 + 1 / 8 = 19 / 24 r/R\le2/3+1/8=19/24 r / R ≤ 2/3 + 1/8 = 19/24 and w ( r 0 + 1 ) 2 ≥ w ( r 0 ) 2 ≥ 13 4 m w(r_0+1)^2\ge w(r_0)^2\ge\frac{13}{4}m w ( r 0 + 1 ) 2 ≥ w ( r 0 ) 2 ≥ 4 13 m , it is at least 576 937 ⋅ 3771 3979 > 9 16 \frac{576}{937}\cdot\frac{3771}{3979}>\frac{9}{16} 937 576 ⋅ 3979 3771 > 16 9 . Finally, if r ≥ r 0 + 1 r\ge r_0+1 r ≥ r 0 + 1 , then Q ′ ( r ) ≥ 1 Q'(r)\ge1 Q ′ ( r ) ≥ 1 , w ≥ 16 13 m w\ge\sqrt{\frac{16}{13}\sqrt{m}} w ≥ 13 16 m , and v : = D ρ R ( x ) U p v:=D\rho_R(x)Up v := D ρ R ( x ) U p satisfies ∣ ⟨ x / r , v ⟩ ∣ ≤ ( 5 / 4 ) m w ∥ v ∥ \lvert\langle x/r,v\rangle\rvert\le\frac{(5/4)\sqrt{m}}{w}\lVert v\rVert ∣⟨ x / r , v ⟩∣ ≤ w ( 5/4 ) m ∥ v ∥ because D ρ R D\rho_R D ρ R shrinks the radial component relative to the tangential component; projecting g = v + Q ′ ( r ) x / r g=v+Q'(r)x/r g = v + Q ′ ( r ) x / r onto v ⊥ v^\perp v ⊥ gives ∥ g ∥ 2 ≥ 1 − 25 m 16 w 2 ≥ 3771 4096 > 9 16 \lVert g\rVert^2\ge1-\frac{25m}{16w^2}\ge\frac{3771}{4096}>\frac{9}{16} ∥ g ∥ 2 ≥ 1 − 16 w 2 25 m ≥ 4096 3771 > 16 9 .
To verify nonconvexity, choose the point on the negative u 1 u_1 u 1 axis with y 1 = − 9 / 8 y_1=-9/8 y 1 = − 9/8 . Since R ≥ 8 R\ge8 R ≥ 8 , it lies in { ∥ x ∥ < r 0 } \{\lVert x\rVert<r_0\} {∥ x ∥ < r 0 } with a 2 ≥ 1 − 81 / 4096 > 31 / 32 a^2\ge1-81/4096>31/32 a 2 ≥ 1 − 81/4096 > 31/32 , where the second axis derivative is at most − 4 ( 31 / 32 ) 3 + 11 / 64 < 0 -4(31/32)^3+11/64<0 − 4 ( 31/32 ) 3 + 11/64 < 0 . Finally, after a zero coordinate every later gate and its derivative vanishes. The chain derivative at coordinate j j j is − γ j − 1 -\gamma_{j-1} − γ j − 1 . Since y j = 0 y_j=0 y j = 0 implies u j T x = 0 u_j^\mathsf{T}x=0 u j T x = 0 , D ρ R u j = a u j D\rho_Ru_j=au_j D ρ R u j = a u j . The radial term is public and cancels from prefix differences, proving (50) .
Scaling, Dimension Floor, and Deterministic Lower Bounds
Lemma E.1 (One scaling rule). Let Λ , b > 0 \Lambda,b>0 Λ , b > 0 , and let F F F have gradient Lipschitz constant at most Λ \Lambda Λ , initial gap at most D D D , and one fixed positive semidefinite Hessian upper matrix K K K of operator norm at most Λ \Lambda Λ , either globally or on its initial sublevel. Assume ∥ ∇ F ∥ > b \lVert\nabla F\rVert>b ∥ ∇ F ∥ > b at the points used in a lower bound. For 0 < L ′ ≤ L 0<L'\le L 0 < L ′ ≤ L set
σ = Λ ε L ′ b , α = Λ ε 2 L ′ b 2 , f ( x ) = α F ( x / σ ) . (53) \sigma=\frac{\Lambda\varepsilon}{L'b},\qquad\alpha=\frac{\Lambda\varepsilon^2}{L'b^2},\qquad f(x)=\alpha F(x/\sigma). \tag*{(53)} σ = L ′ b Λ ε , α = L ′ b 2 Λ ε 2 , f ( x ) = α F ( x / σ ) . (53)
If Λ D / b 2 ≤ L ′ Δ / ε 2 \Lambda D/b^2\le L'\Delta/\varepsilon^2 Λ D / b 2 ≤ L ′ Δ/ ε 2 and L ′ tr K / Λ ≤ L E L'\operatorname{tr}K/\Lambda\le LE L ′ tr K /Λ ≤ L E , then f ∈ B f\in\mathcal{B} f ∈ B , or f ∈ F f\in\mathcal{F} f ∈ F when the matrix bound is global. The barrier becomes strictly greater than ε \varepsilon ε and all oracle costs are preserved.
Proof. The gradient multiplier is α / σ = ε / b \alpha/\sigma=\varepsilon/b α / σ = ε / b , the Hessian multiplier is α / σ 2 = L ′ / Λ \alpha/\sigma^2=L'/\Lambda α / σ 2 = L ′ /Λ , and the gap is at most α D ≤ Δ \alpha D\le\Delta α D ≤ Δ . The fixed upper matrix becomes ( L ′ / Λ ) K (L'/\Lambda)K ( L ′ /Λ ) K , whose norm is at most L ′ ≤ L L'\le L L ′ ≤ L and whose trace satisfies the hypothesis. Positive scaling maps initial sublevels exactly. A call ( x , V ) (x,V) ( x , V ) is simulated by ( x / σ , V ) (x/\sigma,V) ( x / σ , V ) on F F F , multiplying values by α \alpha α and derivative projections by α / σ \alpha/\sigma α / σ ; its rank and cost are unchanged.
Lemma E.2 (Dimension floor). For T > 1 / 2 T>1/2 T > 1/2 and 1 ≤ E ≤ d 1\le E\le d 1 ≤ E ≤ d , N d e t l i n ≥ d N_{\mathrm{det}}^{\mathrm{lin}}\ge d N det lin ≥ d , already on the global fixed-matrix class F \mathcal{F} F .
Proof. Let r = 2 Δ / L r=\sqrt{2\Delta/L} r = 2Δ/ L and f u ( x ) = L 2 ( ⟨ u , x ⟩ − r ) 2 f_u(x)=\frac{L}{2}(\langle u,x\rangle-r)^2 f u ( x ) = 2 L (⟨ u , x ⟩ − r ) 2 for a unit vector u u u . Simulate the reply ( Δ , 0 ) (\Delta,0) ( Δ , 0 ) against a deterministic algorithm truncated below cost d d d . The span of all query points and requested column spaces has dimension strictly less than d d d . Choose a perpendicular unit u u u , with its sign arranged so that ⟨ u , x ^ ⟩ ≤ 0 \langle u,\hat{x}\rangle\le0 ⟨ u , x ^ ⟩ ≤ 0 . The function f u f_u f u reproduces all replies, has upper matrix L u u T Luu^\mathsf{T} Lu u T , and its output gradient norm is at least L r = 2 T ε > ε Lr=\sqrt{2T}\,\varepsilon>\varepsilon L r = 2 T ε > ε . Truncation is justified as in Lemma C.4 .
Theorem E.3 (Multistage lower bound on the main class). If an integer m ≥ 2 m\ge2 m ≥ 2 satisfies
224 m ≤ T , 224 2 m 2 log ( e + m ) ≤ E T , 10000 m log ( e + m ) ≤ d (54) 224m\le T,\qquad224^2m^2\log(e+m)\le ET,\qquad10000m\log(e+m)\le d \tag*{(54)} 224 m ≤ T , 22 4 2 m 2 log ( e + m ) ≤ E T , 10000 m log ( e + m ) ≤ d (54) then both deterministic oracle costs on B \mathcal{B} B are at least ⌊ d / 2 ⌋ m \lfloor d/2\rfloor m ⌊ d /2 ⌋ m . The witnesses are C 1 , 1 C^{1,1} C 1 , 1 , globally L L L -smooth, coercive, nonconvex, and globally 2 L Δ 2\sqrt{L\Delta} 2 L Δ -Lipschitz.
Proof. Set ℓ = log ( e + m ) \ell=\log(e+m) ℓ = log ( e + m ) , R 2 = d / ( 156 ℓ ) R^2=d/(156\ell) R 2 = d / ( 156 ℓ ) , h = ⌊ d / 2 ⌋ h=\lfloor d/2\rfloor h = ⌊ d /2 ⌋ , p = d − m − h + 1 p=d-m-h+1 p = d − m − h + 1 , and use Theorem D.4 . Since 156 ⋅ 64 = 9984 < 10000 156\cdot64=9984<10000 156 ⋅ 64 = 9984 < 10000 , the last inequality of (54) implies R 2 ≥ 64 m R^2\ge64m R 2 ≥ 64 m , m ≤ d / 128 m\le d/128 m ≤ d /128 , and
p ≥ 63 d / 128 p \ge63d/128 p ≥ 63 d /128 . For a list of M ≤ h m M \le hm M ≤ hm past points, M / p ≤ 64 m / 63 M/p \le64m/63 M / p ≤ 64 m /63 . Since log ( e + c m ) ≤ c log ( e + m ) \log(e+cm) \le c\log(e+m) log ( e + c m ) ≤ c log ( e + m ) when c ≥ 1 c \ge1 c ≥ 1 , Lemma C.2 gives squared projection bound at most
( 13 / 3 ) 2 R 2 p log ( e + M / p ) ≤ 169 ⋅ 128 ⋅ 64 9 ⋅ 63 2 ⋅ 156 = 26624 107163 < 1 4 . \frac{(13/3)^2R^2}{p}\log(e+M/p) \le\frac{169\cdot128\cdot64}{9\cdot63^2\cdot156}=\frac{26624}{107163}<\frac{1}{4}. p ( 13/3 ) 2 R 2 log ( e + M / p ) ≤ 9 ⋅ 6 3 2 ⋅ 156 169 ⋅ 128 ⋅ 64 = 107163 26624 < 4 1 .
Thus the transcript lemma applies with τ = 1 / 2 \tau=1/2 τ = 1/2 and b = 3 / 4 b=3/4 b = 3/4 . For the fixed matrix (47) , since ℓ ≥ log ( e + 2 ) > 3 / 2 \ell\ge\log(e+2)>3/2 ℓ ≥ log ( e + 2 ) > 3/2 ,
tr K U 56 = 54 m + 80 ⋅ 156 m ℓ 56 < 3129 14 m ℓ < 224 m ℓ . \frac{\operatorname{tr}K_U}{56}=\frac{54m+80\cdot156m\ell}{56}<\frac{3129}{14}m\ell<224m\ell. 56 tr K U = 56 54 m + 80 ⋅ 156 m ℓ < 14 3129 m ℓ < 224 m ℓ .
Take L ′ = 224 m L / T L'=224mL/T L ′ = 224 m L / T , Λ = 56 \Lambda=56 Λ = 56 , and apply Lemma E.1. We have L ′ < L L'<L L ′ < L , and 56 ( 9 / 4 ) / ( 3 / 4 ) 2 = 224 56(9/4)/(3/4)^2=224 56 ( 9/4 ) / ( 3/4 ) 2 = 224 , so the condition on the initial gap holds. The trace becomes at most L ′ 224 m ℓ ≤ L E L'\,224m\ell\le LE L ′ 224 m ℓ ≤ L E by the second inequality of (54) . The normalized matrix also has operator norm at most L ′ L' L ′ , so the instance lies in B \mathcal{B} B . The global gradient bound is ∥ ∇ f ∥ ≤ 16 m ε ≤ 2 L Δ \|\nabla f\|\le16\sqrt{m}\,\varepsilon\le2\sqrt{L\Delta} ∥∇ f ∥ ≤ 16 m ε ≤ 2 L Δ because T ≥ 224 m T\ge224m T ≥ 224 m . Positive scaling preserves coercivity and nonconvexity.
Corollary E.4 (Rounded main lower bound). If E T ≤ d ET\le d E T ≤ d , then
N d e t ≥ max { d , d 672 min { T , E T log ( e + T ) } } . (55) N_{\mathrm{det}}\ge\max\left\{d,\frac{d}{672}\min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\}\right\}. \tag*{(55)} N det ≥ max { d , 672 d min { T , log ( e + T ) E T } } . (55) Without the assumption E T ≤ d ET\le d E T ≤ d , E ≥ T log ( e + T ) E\ge T\log(e+T) E ≥ T log ( e + T ) implies N d e t ≥ d T / 30000 N_{\mathrm{det}}\ge dT/30000 N det ≥ d T /30000 .
Proof. Put M = min { T , E T / log ( e + T ) } M=\min\{T,\sqrt{ET/\log(e+T)}\} M = min { T , E T / log ( e + T ) } , z = M / 224 z=M/224 z = M /224 , and m = ⌊ z ⌋ m=\lfloor z\rfloor m = ⌊ z ⌋ . If z < 3 z<3 z < 3 , the dimension floor dominates d z / 3 dz/3 d z /3 . If z ≥ 3 z\ge3 z ≥ 3 , the first two stage conditions follow from 224 m ≤ M 224m\le M 224 m ≤ M and log ( e + m ) ≤ log ( e + T ) \log(e+m)\le\log(e+T) log ( e + m ) ≤ log ( e + T ) . The third follows from 10000 < 2 ⋅ 224 2 ≤ m 224 2 10000<2\cdot224^2\le m\,224^2 10000 < 2 ⋅ 22 4 2 ≤ m 22 4 2 and E T ≤ d ET\le d E T ≤ d . Moreover m / z ≥ 3 / 4 m/z\ge3/4 m / z ≥ 3/4 and h / d ≥ 15 / 32 h/d\ge15/32 h / d ≥ 15/32 , so h m ≥ ( 45 / 128 ) d z > d z / 3 = d M / 672 hm\ge(45/128)dz>dz/3=dM/672 hm ≥ ( 45/128 ) d z > d z /3 = d M /672 . For the final claim take m = ⌊ T / 10000 ⌋ m=\lfloor T/10000\rfloor m = ⌊ T /10000 ⌋ and apply the same rounding argument. Its stage conditions follow from T log ( e + T ) ≤ E ≤ d T\log(e+T)\le E\le d T log ( e + T ) ≤ E ≤ d and 224 2 < 10000 2 224^2<10000^2 22 4 2 < 1000 0 2 ; the dimension floor covers the case T / 10000 < 3 T/10000<3 T /10000 < 3 .
Corollary E.5 (Matched window and cost ratio). For E T ≤ d ET\le d E T ≤ d , write ℓ = log ( e + T ) \ell=\log(e+T) ℓ = log ( e + T ) . Then
d 15456 ( E + E T ) ℓ ≤ N d e t N r a n d c a p ≤ 1824 d E + E T . \frac{d}{15456(E+\sqrt{ET})\sqrt{\ell}}\le\frac{N_{\mathrm{det}}}{N_{\mathrm{rand}}^{\mathrm{cap}}}\le\frac{1824d}{E+\sqrt{ET}}. 15456 ( E + E T ) ℓ d ≤ N rand cap N det ≤ E + E T 1824 d . For T ℓ ≤ E ≤ d / T T\ell\le E\le d/T T ℓ ≤ E ≤ d / T , the ratio lies between d / ( 15456 E ) d/(15456E) d / ( 15456 E ) and 120 d / E 120d/E 120 d / E .
Proof. As E T ≤ d ET\le d E T ≤ d implies T ≤ d T\le d T ≤ d , the deterministic upper bound is at most d min { 10 T , 100 2 E T } d\min\{10T,100\sqrt{2}\sqrt{ET}\} d min { 10 T , 100 2 E T } , and
min { 10 T , 100 2 E T } ( E + E T ) ≤ 10 T E + 100 2 E T E T = ( 10 + 100 2 ) E T < 152 E T . \min\{10T,100\sqrt{2}\sqrt{ET}\}(E+\sqrt{ET})\le10TE+100\sqrt{2}\sqrt{ET}\sqrt{ET}=(10+100\sqrt{2})ET<152ET. min { 10 T , 100 2 E T } ( E + E T ) ≤ 10 T E + 100 2 E T E T = ( 10 + 100 2 ) E T < 152 E T .
For the lower bound use a b / ( a + b ) ≤ min ( a , b ) ab/(a+b)\le\min(a,b) ab / ( a + b ) ≤ min ( a , b ) with a = T a=T a = T , b = E T b=\sqrt{ET} b = E T and min { T , E T / ℓ } ≥ min { T , E T } / ℓ \min\{T,\sqrt{ET}/\ell\}\ge\min\{T,\sqrt{ET}\}/\sqrt{\ell} min { T , E T / ℓ } ≥ min { T , E T } / ℓ . Divide deterministic lower and upper bounds by 23 E T 23ET 23 E T and E T / 12 ET/12 E T /12 , respectively, noting that 672 ⋅ 23 = 15456 672\cdot23=15456 672 ⋅ 23 = 15456 and 12 ⋅ 152 = 1824 12\cdot152=1824 12 ⋅ 152 = 1824 . For the matched window use 10 d T 10dT 10 d T , d T / 672 dT/672 d T /672 and the random bounds directly.
Unrestricted parameter choice
Assume m ≥ 2 m \ge2 m ≥ 2 and put
R 2 = 5 m , r 0 2 = 4 m , r 1 2 = 9 2 m . R^2 = 5m,\qquad r_0^2 = 4m,\qquad r_1^2 = \frac{9}{2}m. R 2 = 5 m , r 0 2 = 4 m , r 1 2 = 2 9 m .
Let q q q be the shifted Huber potential (42) with these r 0 r_0 r 0 , with s = r 1 − r 0 s = r_1-r_0 s = r 1 − r 0 , and with λ = 1 / ( r 1 − r 0 ) \lambda= 1/(r_1-r_0) λ = 1/ ( r 1 − r 0 ) ; thus q q q vanishes for ∥ x ∥ ≤ r 0 \lVert x\rVert\le r_0 ∥ x ∥ ≤ r 0 , its radial derivative increases linearly from zero to one on [ r 0 , r 1 ] [r_0,r_1] [ r 0 , r 1 ] , and it equals one afterwards. For an orthonormal frame U = ( u 1 , … , u m ) U=(u_1,\ldots,u_m) U = ( u 1 , … , u m ) define
F ~ m , U ( x ) = g m ( U T ρ R ( x ) ) + q ( x ) . \widetilde{F}_{m,U}(x)=g_m(U^{\mathsf{T}}\rho_R(x))+q(x). F m , U ( x ) = g m ( U T ρ R ( x )) + q ( x ) .
Lemma E.6 (Smoothness, gap, and nonconvexity). For every m ≥ 2 m \ge2 m ≥ 2 and every orthonormal U U U , the function F ~ m , U \widetilde{F}_{m,U} F m , U is coercive and nonconvex, and
Lip ( ∇ F ~ m , U ) < 44 , F ~ m , U ( 0 ) − inf F ~ m , U ≤ 9 4 ( m − 1 2 ) . \operatorname{Lip}(\nabla\widetilde{F}_{m,U}) < 44,\qquad\widetilde{F}_{m,U}(0)-\inf\widetilde{F}_{m,U}\le\frac{9}{4}\left(m-\frac{1}{2}\right). Lip ( ∇ F m , U ) < 44 , F m , U ( 0 ) − inf F m , U ≤ 4 9 ( m − 2 1 ) .
Proof. At m ≥ 2 m \ge2 m ≥ 2 one has r 1 − r 0 = ( 9 / 2 − 2 ) m ≥ 3 − 2 2 > 1 / 6 r_1-r_0=(\sqrt{9/2}-2)\sqrt{m}\ge3-2\sqrt{2}>1/6 r 1 − r 0 = ( 9/2 − 2 ) m ≥ 3 − 2 2 > 1/6 , so λ < 6 \lambda<6 λ < 6 . Write t = ∥ x ∥ / R t=\lVert x\rVert/R t = ∥ x ∥ / R and a = ( 1 + t 2 ) − 1 / 2 a=(1+t^2)^{-1/2} a = ( 1 + t 2 ) − 1/2 . On every smooth piece of F ~ m , U \widetilde{F}_{m,U} F m , U , the chain Hessian has operator norm at most 42 a 2 + ∥ U p ∥ ∥ D 2 ρ R ( x ) ∥ ≤ 42 / ( 1 + t 2 ) + ( 30 / 5 ) t a 3 42a^2+\lVert Up\rVert\lVert D^2\rho_R(x)\rVert\le42/(1+t^2)+(30/\sqrt{5})ta^3 42 a 2 + ∥ U p ∥ ∥ D 2 ρ R ( x )∥ ≤ 42/ ( 1 + t 2 ) + ( 30/ 5 ) t a 3 by Lemmas D.1 and D.2 . For r < r 0 r<r_0 r < r 0 , ∇ 2 q ( x ) = 0 \nabla^2q(x)=0 ∇ 2 q ( x ) = 0 and
∥ ∇ 2 F ~ m , U ( x ) ∥ o p ≤ 42 + ( 30 / 5 ) t 1 + t 2 ≤ 42 + 225 42 ⋅ 5 = 603 14 < 44. \left\lVert\nabla^2\widetilde{F}_{m,U}(x)\right\rVert_{\mathrm{op}}
\le\frac{42+(30/\sqrt{5})t}{1+t^2}
\le42+\frac{225}{42\cdot5}
=\frac{603}{14}<44. ∇ 2 F m , U ( x ) op ≤ 1 + t 2 42 + ( 30/ 5 ) t ≤ 42 + 42 ⋅ 5 225 = 14 603 < 44.
For r > r 0 r>r_0 r > r 0 , t 2 ≥ 4 / 5 t^2\ge4/5 t 2 ≥ 4/5 , a 2 ≤ 5 / 9 a^2\le5/9 a 2 ≤ 5/9 , ∥ D 2 ρ R ∥ ≤ 2 / 15 m \lVert D^2\rho_R\rVert\le2/\sqrt{15m} ∥ D 2 ρ R ∥ ≤ 2/ 15 m , and ∥ ∇ 2 q ∥ o p ≤ λ < 6 \lVert\nabla^2q\rVert_{\mathrm{op}}\le\lambda<6 ∥ ∇ 2 q ∥ op ≤ λ < 6 , so
∥ ∇ 2 F ~ m , U ( x ) ∥ o p ≤ 42 ⋅ 5 9 + 20 15 + 6 < 44. \left\lVert\nabla^2\widetilde{F}_{m,U}(x)\right\rVert_{\mathrm{op}}
\le42\cdot\frac{5}{9}+\frac{20}{\sqrt{15}}+6<44. ∇ 2 F m , U ( x ) op ≤ 42 ⋅ 9 5 + 15 20 + 6 < 44.
Lemma D.3 integrates this piecewise Hessian bound to Lip ( ∇ F ~ m , U ) < 44 \operatorname{Lip}(\nabla\widetilde{F}_{m,U})<44 Lip ( ∇ F m , U ) < 44 . The bounded chain and the eventually linear growth of q q q give coercivity. Since q ≥ 0 q\ge0 q ≥ 0 and q ( 0 ) = 0 q(0)=0 q ( 0 ) = 0 , Lemma D.1 gives the gap.
For nonconvexity, take x x x on the negative u 1 u_1 u 1 axis with y 1 = − 9 / 8 y_1=-9/8 y 1 = − 9/8 and all other compressed coordinates zero. This point lies inside r 0 r_0 r 0 , where q = 0 q=0 q = 0 , and the gate ψ ( − y 1 ) \psi(-y_1) ψ ( − y 1 ) is saturated near it. Along the axis the only varying chain term is − φ ( y 1 ) -\varphi(y_1) − φ ( y 1 ) . There φ ′ ′ = 4 \varphi''=4 φ ′′ = 4 and φ ′ = 1 / 2 \varphi'=1/2 φ ′ = 1/2 , while the scalar compression s ↦ − s / 1 + s 2 / R 2 s\mapsto-s/\sqrt{1+s^2/R^2} s ↦ − s / 1 + s 2 / R 2 has positive second derivative for s > 0 s>0 s > 0 . The second derivative of the composite along the axis is therefore negative. □
Lemma E.7 (Gradient barrier on the whole space). If some coordinate of U T ρ R ( x ) U^{\mathsf{T}}\rho_R(x) U T ρ R ( x ) has magnitude less than one, then ∥ ∇ F ~ m , U ( x ) ∥ > 3 / 5 \lVert\nabla\widetilde{F}_{m,U}(x)\rVert>3/5 ∥ ∇ F m , U ( x )∥ > 3/5 .
Proof. Write r = ∥ x ∥ r=\lVert x\rVert r = ∥ x ∥ , w = ∥ ρ R ( x ) ∥ = a r w=\lVert\rho_R(x)\rVert=ar w = ∥ ρ R ( x )∥ = a r , y = U T ρ R ( x ) y=U^{\mathsf{T}}\rho_R(x) y = U T ρ R ( x ) , p = ∇ g m ( y ) p=\nabla g_m(y) p = ∇ g m ( y ) , and g = ∇ F ~ m , U ( x ) g=\nabla\widetilde{F}_{m,U}(x) g = ∇ F m , U ( x ) . Let j j j be the first index with ∣ y j ∣ < 1 \lvert y_j\rvert<1 ∣ y j ∣ < 1 , so that p j ≤ − 1 p_j\le-1 p j ≤ − 1 by Lemma D.1 .
Inner ball, r ≤ r 0 r\le r_0 r ≤ r 0 . At x = 0 x=0 x = 0 one has g = U ∇ g m ( 0 ) = − u 1 g=U\nabla g_m(0)=-u_1 g = U ∇ g m ( 0 ) = − u 1 , of norm one, so assume x ≠ 0 x\ne0 x = 0 . Here g = D ρ R ( x ) U p g=D\rho_R(x)Up g = D ρ R ( x ) U p . With t = r / R t=r/R t = r / R , since t 2 ≤ 4 / 5 t^2\le4/5 t 2 ≤ 4/5 and R 2 ≥ 10 R^2\ge10 R 2 ≥ 10 ,
∥ D ρ R ( x ) − 1 u j ∥ 2 = a − 2 [ 1 + ( a − 4 − 1 ) y j 2 a 2 r 2 ] ≤ ( 1 + t 2 ) [ 1 + ( 2 + t 2 ) ( 1 + t 2 ) R 2 ] ≤ 1692 625 < 25 9 . \left\lVert D\rho_R(x)^{-1}u_j\right\rVert^2
=a^{-2}\left[1+(a^{-4}-1)\frac{y_j^2}{a^2r^2}\right]
\le(1+t^2)\left[1+\frac{(2+t^2)(1+t^2)}{R^2}\right]
\le\frac{1692}{625}<\frac{25}{9}. D ρ R ( x ) − 1 u j 2 = a − 2 [ 1 + ( a − 4 − 1 ) a 2 r 2 y j 2 ] ≤ ( 1 + t 2 ) [ 1 + R 2 ( 2 + t 2 ) ( 1 + t 2 ) ] ≤ 625 1692 < 9 25 .
Hence 1 ≤ ∣ p j ∣ = ∣ ⟨ D ρ R ( x ) − 1 u j , g ⟩ ∣ < ( 5 / 3 ) ∥ g ∥ 1\le\lvert p_j\rvert=\lvert\langle D\rho_R(x)^{-1}u_j,g\rangle\rvert<(5/3)\lVert g\rVert 1 ≤ ∣ p j ∣ = ∣⟨ D ρ R ( x ) − 1 u j , g ⟩∣ < ( 5/3 ) ∥ g ∥ , so ∥ g ∥ > 3 / 5 \lVert g\rVert>3/5 ∥ g ∥ > 3/5 .
Transition annulus, r 0 ≤ r ≤ r 1 r_0 \le r \le r_1 r 0 ≤ r ≤ r 1 . Here w 2 / m ∈ [ 20 / 9 , 45 / 19 ] w^2/m \in[20/9,45/19] w 2 / m ∈ [ 20/9 , 45/19 ] and a 2 = 1 − w 2 / ( 5 m ) a^2 = 1-w^2/(5m) a 2 = 1 − w 2 / ( 5 m ) . Let I = { i : ∣ y i ∣ < 5 / 4 } I=\{i: |y_i|<5/4\} I = { i : ∣ y i ∣ < 5/4 } and W = span { u i : i ∈ I } W=\operatorname{span}\{u_i:i\in I\} W = span { u i : i ∈ I } . By Lemma D.1 , U p ∈ W Up\in W U p ∈ W and j ∈ I j\in I j ∈ I . Since ∣ y j ∣ < 1 |y_j|<1 ∣ y j ∣ < 1 and ∣ y i ∣ < 5 / 4 |y_i|<5/4 ∣ y i ∣ < 5/4 for the other indices in I I I , ∑ i ∈ I ∖ { j } y i 2 ≤ 25 16 ( m − 1 ) \sum_{i\in I\setminus\{j\}}y_i^2\le\frac{25}{16}(m-1) ∑ i ∈ I ∖ { j } y i 2 ≤ 16 25 ( m − 1 ) . With D = ∥ P W ⊥ ρ R ( x ) ∥ D=\|P_{W^\perp}\rho_R(x)\| D = ∥ P W ⊥ ρ R ( x ) ∥ , since m ≥ 2 m\ge2 m ≥ 2 and w 2 / m ≥ 20 / 9 > 25 / 16 w^2/m\ge20/9>25/16 w 2 / m ≥ 20/9 > 25/16 ,
D 2 + y j 2 = w 2 − ∑ i ∈ I ∖ { j } y i 2 ≥ w 2 − 25 16 ( m − 1 ) = 2 ( w 2 m − 25 32 ) + ( m − 2 ) ( w 2 m − 25 16 ) ≥ 2 ( w 2 m − 25 32 ) . D^2+y_j^2=w^2-\sum_{i\in I\setminus\{j\}}y_i^2\ge w^2-\frac{25}{16}(m-1)=2\left(\frac{w^2}{m}-\frac{25}{32}\right)+(m-2)\left(\frac{w^2}{m}-\frac{25}{16}\right)\ge2\left(\frac{w^2}{m}-\frac{25}{32}\right). D 2 + y j 2 = w 2 − i ∈ I ∖ { j } ∑ y i 2 ≥ w 2 − 16 25 ( m − 1 ) = 2 ( m w 2 − 32 25 ) + ( m − 2 ) ( m w 2 − 16 25 ) ≥ 2 ( m w 2 − 32 25 ) .
As y j 2 < 1 y_j^2<1 y j 2 < 1 , projecting g = a U p + c x g=aUp+cx g = a U p + c x onto u j u_j u j and P W ⊥ x P_{W^\perp}x P W ⊥ x gives
∥ g ∥ 2 ≥ a 2 p j 2 D 2 D 2 + y j 2 ≥ ( 1 − w 2 5 m ) ( 1 − 1 / 2 w 2 / m − 25 / 32 ) . (56) \|g\|^2\ge\frac{a^2p_j^2D^2}{D^2+y_j^2}\ge\left(1-\frac{w^2}{5m}\right)\left(1-\frac{1/2}{w^2/m-25/32}\right). \tag*{(56)} ∥ g ∥ 2 ≥ D 2 + y j 2 a 2 p j 2 D 2 ≥ ( 1 − 5 m w 2 ) ( 1 − w 2 / m − 25/32 1/2 ) . (56)
The right-hand side of (56) is concave in w 2 / m w^2/m w 2 / m on [ 20 / 9 , 45 / 19 ] [20/9,45/19] [ 20/9 , 45/19 ] , with endpoint values 271 / 747 > 9 / 25 271/747>9/25 271/747 > 9/25 and 1322 / 3667 > 9 / 25 1322/3667>9/25 1322/3667 > 9/25 , so ∥ g ∥ > 3 / 5 \|g\|>3/5 ∥ g ∥ > 3/5 .
Exterior, r ≥ r 1 r\ge r_1 r ≥ r 1 . Now ∇ q = n : = x / r \nabla q=n:=x/r ∇ q = n := x / r , w 2 ≥ 45 19 m w^2\ge\frac{45}{19}m w 2 ≥ 19 45 m , a 2 = 1 − w 2 / ( 5 m ) ≤ 10 / 19 a^2=1-w^2/(5m)\le10/19 a 2 = 1 − w 2 / ( 5 m ) ≤ 10/19 , and ∥ P W ρ R ( x ) ∥ 2 = ∑ i ∈ I y i 2 < 25 16 m \|P_W\rho_R(x)\|^2=\sum_{i\in I}y_i^2<\frac{25}{16}m ∥ P W ρ R ( x ) ∥ 2 = ∑ i ∈ I y i 2 < 16 25 m . If p ≠ 0 p\ne0 p = 0 , put c = ∣ ⟨ n , U p ⟩ ∣ / ∥ U p ∥ ≤ ∥ P W n ∥ ≤ ( 25 / 16 ) m / w c=|\langle n,Up\rangle|/\|Up\|\le\|P_Wn\|\le\sqrt{(25/16)m}/w c = ∣ ⟨ n , U p ⟩ ∣/∥ U p ∥ ≤ ∥ P W n ∥ ≤ ( 25/16 ) m / w , so c 2 < 25 / 16 45 / 19 = 95 144 c^2<\frac{25/16}{45/19}=\frac{95}{144} c 2 < 45/19 25/16 = 144 95 . Since D ρ R ( x ) D\rho_R(x) D ρ R ( x ) has radial eigenvalue a 3 a^3 a 3 and tangential eigenvalue a a a , for v = D ρ R ( x ) U p v=D\rho_R(x)Up v = D ρ R ( x ) U p we have
∣ ⟨ n , v ⟩ ∣ 2 ∥ v ∥ 2 = a 4 c 2 1 − c 2 + a 4 c 2 ≤ 500 1431 , \frac{|\langle n,v\rangle|^2}{\|v\|^2}=\frac{a^4c^2}{1-c^2+a^4c^2}\le\frac{500}{1431}, ∥ v ∥ 2 ∣ ⟨ n , v ⟩ ∣ 2 = 1 − c 2 + a 4 c 2 a 4 c 2 ≤ 1431 500 ,
and projecting g = v + n g=v+n g = v + n onto v ⊥ v^\perp v ⊥ gives ∥ g ∥ 2 ≥ 1 − 500 1431 = 931 1431 > 9 25 \|g\|^2\ge1-\frac{500}{1431}=\frac{931}{1431}>\frac{9}{25} ∥ g ∥ 2 ≥ 1 − 1431 500 = 1431 931 > 25 9 . If p = 0 p=0 p = 0 , then ∥ g ∥ = 1 \|g\|=1 ∥ g ∥ = 1 .
When y j = 0 y_j=0 y j = 0 and ∣ y i ∣ ≤ 1 / 2 |y_i|\le1/2 ∣ y i ∣ ≤ 1/2 for all i > j i>j i > j , let
r ~ j ( x ) = g m ( u 1 ⊤ ρ R ( x ) , … , u j − 1 ⊤ ρ R ( x ) , 0 , … , 0 ) + q ( x ) . \widetilde r_j(x)=g_m(u_1^\top\rho_R(x),\ldots,u_{j-1}^\top\rho_R(x),0,\ldots,0)+q(x). r j ( x ) = g m ( u 1 ⊤ ρ R ( x ) , … , u j − 1 ⊤ ρ R ( x ) , 0 , … , 0 ) + q ( x ) .
Every later gate and its derivative vanish, p j = − γ j − 1 p_j=-\gamma_{j-1} p j = − γ j − 1 , and u j ⊤ x = 0 u_j^\top x=0 u j ⊤ x = 0 gives D ρ R ( x ) u j = a ( x ) u j D\rho_R(x)u_j=a(x)u_j D ρ R ( x ) u j = a ( x ) u j . Hence
F ~ m , U ( x ) = r ~ j ( x ) , ∇ F ~ m , U ( x ) = ∇ r ~ j ( x ) − a ( x ) γ j − 1 ( x ) u j . (57) \widetilde F_{m,U}(x)=\widetilde r_j(x),\qquad\nabla\widetilde F_{m,U}(x)=\nabla\widetilde r_j(x)-a(x)\gamma_{j-1}(x)u_j. \tag*{(57)} F m , U ( x ) = r j ( x ) , ∇ F m , U ( x ) = ∇ r j ( x ) − a ( x ) γ j − 1 ( x ) u j . (57)
Theorem E.8 (Lower bound on the unrestricted smooth class). For all d , T ≥ 1 d,T\ge1 d , T ≥ 1 on the unrestricted globally L L L -smooth class, N d e t l i n ≥ d 500 min { T , d / log ( e d ) } N_{\mathrm{det}}^{\mathrm{lin}}\ge\frac{d}{500}\min\{T,d/\log(ed)\} N det lin ≥ 500 d min { T , d / log ( e d )} ; the same bound holds for exact values.
Proof of Theorem E.8
Put b = 3 / 5 b=3/5 b = 3/5 , A = 44 ⋅ ( 9 / 4 ) / b 2 = 275 A=44\cdot(9/4)/b^2=275 A = 44 ⋅ ( 9/4 ) / b 2 = 275 , and S = min { T , d / log ( e d ) } S=\min\{T,d/\log(ed)\} S = min { T , d / log ( e d )} .
Case S > 500 S>500 S > 500 . Let m = ⌊ S / A + 1 / 2 ⌋ m=\lfloor S/A+1/2\rfloor m = ⌊ S / A + 1/2 ⌋ and h = ⌊ 7 d / 10 ⌋ h=\lfloor7d/10\rfloor h = ⌊ 7 d /10 ⌋ . Since 500 > 3 A / 2 = 825 / 2 500>3A/2=825/2 500 > 3 A /2 = 825/2 , we have m ≥ 2 m\ge2 m ≥ 2 , and m − 1 / 2 ≤ S / A ≤ d / ( A log ( e d ) ) m-1/2\le S/A\le d/(A\log(ed)) m − 1/2 ≤ S / A ≤ d / ( A log ( e d )) gives d / m ≥ ( A / 2 ) ( 2 − 1 / m ) log ( e d ) d/m\ge(A/2)(2-1/m)\log(ed) d / m ≥ ( A /2 ) ( 2 − 1/ m ) log ( e d ) and m < d / 1000 m<d/1000 m < d /1000 . Hence p 0 = d − m − h ≥ 3 d / 10 − m > 0 p_0=d-m-h\ge3d/10-m>0 p 0 = d − m − h ≥ 3 d /10 − m > 0 and, with τ = 1 / 2 \tau=1/2 τ = 1/2 and R 2 = 5 m R^2=5m R 2 = 5 m ,
p 0 τ 2 2 R 2 = p 0 40 m ≥ 3 d 400 m − 1 40 ≥ 33 32 ( 2 − 1 m ) log ( e d ) − 1 40 > ( 2 − 1 m ) log ( e d ) . \frac{p_0\tau^2}{2R^2}=\frac{p_0}{40m}\ge\frac{3d}{400m}-\frac{1}{40}\ge\frac{33}{32}\left(2-\frac{1}{m}\right)\log(ed)-\frac{1}{40}>\left(2-\frac{1}{m}\right)\log(ed). 2 R 2 p 0 τ 2 = 40 m p 0 ≥ 400 m 3 d − 40 1 ≥ 32 33 ( 2 − m 1 ) log ( e d ) − 40 1 > ( 2 − m 1 ) log ( e d ) .
Since d > 500 d>500 d > 500 and m < d / 1000 m<d/1000 m < d /1000 , we have 2 m ( e d ) 1 / m ≤ max { 4 e d , 2 e m } < e 2 d 2m(ed)^{1/m}\le\max\{4\sqrt{ed},2em\}<e^2d 2 m ( e d ) 1/ m ≤ max { 4 e d , 2 e m } < e 2 d , so 2 h m < 2 d m < ( e d ) 2 − 1 / m 2hm<2dm<(ed)^{2-1/m} 2 hm < 2 d m < ( e d ) 2 − 1/ m and
2 h m exp ( − p 0 τ 2 2 R 2 ) < ( e d ) 2 − 1 / m exp ( − ( 2 − 1 m ) log ( e d ) ) = 1 , 2hm\exp\left(-\frac{p_0\tau^2}{2R^2}\right)<(ed)^{2-1/m}\exp\left(-\left(2-\frac{1}{m}\right)\log(ed)\right)=1, 2 hm exp ( − 2 R 2 p 0 τ 2 ) < ( e d ) 2 − 1/ m exp ( − ( 2 − m 1 ) log ( e d ) ) = 1 ,
which is the cap condition of Lemma C.4 , whose residual dimension p 0 + 1 p_0+1 p 0 + 1 only improves it. Apply Lemma E.1 with L ′ = L L'=L L ′ = L , Λ = 44 \Lambda=44 Λ = 44 , K = 44 I K=44I K = 44 I , and D = 9 4 ( m − 1 2 ) D=\frac{9}{4}(m-\frac{1}{2}) D = 4 9 ( m − 2 1 ) ; the trace condition holds because E = d E=d E = d , and the accuracy condition is T ≥ 44 D / b 2 = A ( m − 1 2 ) T\ge44D/b^2=A(m-\frac{1}{2}) T ≥ 44 D / b 2 = A ( m − 2 1 ) , which follows from A ( m − 1 2 ) ≤ S ≤ T A(m-\frac{1}{2})\le S\le T A ( m − 2 1 ) ≤ S ≤ T . Lemmas E.6 and E.7 supply the smoothness, the gap, and a barrier larger than ε \varepsilon ε after scaling, and (57) verifies the prefix hypotheses of Lemma C.4 with z = ρ R z=\rho R z = ρR and c j = a γ j − 1 c_j=a\gamma^{j-1} c j = a γ j − 1 . Every deterministic algorithm with rank-priced cost below h m hm hm therefore fails on some fixed scaled F ~ m , U \widetilde F_{m,U} F m , U . Finally, S < A ( m + 1 / 2 ) S<A(m+1/2) S < A ( m + 1/2 ) and m ≥ 2 m\ge2 m ≥ 2 give m > 4 S / ( 5 A ) m>4S/(5A) m > 4 S / ( 5 A ) ; since d > 500 d>500 d > 500 ,
h m > d S A ( 14 25 − 4 5 d ) > d S 500 . hm>\frac{dS}{A}\left(\frac{14}{25}-\frac{4}{5d}\right)>\frac{dS}{500}. hm > A d S ( 25 14 − 5 d 4 ) > 500 d S .
using 14 / 25 − A / 500 = 1 / 100 > 4 / ( 5 d ) 14/25-A/500=1/100>4/(5d) 14/25 − A /500 = 1/100 > 4/ ( 5 d ) .
The witnesses in the chain regime are coercive and nonconvex by Lemma E.6 . They also have gradient norm at most 2 L Δ 2\sqrt{L\Delta} 2 L Δ : ∥ ∇ F ~ m , U ∥ ≤ 10 m + 1 ≤ 11 m \lVert\nabla\widetilde F_{m,U}\rVert\le10\sqrt{m}+1\le11\sqrt{m} ∥ ∇ F m , U ∥ ≤ 10 m + 1 ≤ 11 m , so the scaled gradient is at most ( 55 / 3 ) m ε (55/3)\sqrt{m}\,\varepsilon ( 55/3 ) m ε , while L Δ ≥ A ( m − 1 / 2 ) ε 2 ≥ ( A / 2 ) m ε 2 L\Delta\ge A(m-1/2)\varepsilon^2\ge(A/2)m\varepsilon^2 L Δ ≥ A ( m − 1/2 ) ε 2 ≥ ( A /2 ) m ε 2 and ( 55 / 3 ) 2 < 2 A (55/3)^2<2A ( 55/3 ) 2 < 2 A .
Case S ≤ 500 S\le500 S ≤ 500 . Lemma E.2 with E = d E=d E = d gives N d e t l i n ≥ d ≥ d S / 500 N_{\mathrm{det}}^{\mathrm{lin}}\ge d\ge dS/500 N det lin ≥ d ≥ d S /500 . Algorithms that use only values are a special case of rank-priced ones, so N d e t v a l ≥ N d e t l i n N_{\mathrm{det}}^{\mathrm{val}}\ge N_{\mathrm{det}}^{\mathrm{lin}} N det val ≥ N det lin .
Quadratic Randomized Lower Bound
Algorithms here are measurable functions of their independent seed and finite transcript; success is at least 2 / 3 2/3 2/3 on each fixed function in the class, not merely on average over the distribution used in the lower bound. A subscript E E E , as in N r a n d , E N_{\mathrm{rand},E} N rand , E , indicates the class F L , Δ , E \mathcal{F}_{L,\Delta,E} F L , Δ , E .
Lemma F.1 (Explicit spherical coordinate splitting). For q ≥ 2 q\ge2 q ≥ 2 , a uniform U ∈ S q − 1 U\in S^{q-1} U ∈ S q − 1 admits the product representation
U = ( C , 1 − C 2 W ) , W ∼ Unif ( S q − 2 ) , W independent of C , E C 2 = 1 / q . U=\left(C,\sqrt{1-C^2}\,W\right),\qquad W\sim\operatorname{Unif}(S^{q-2}),\qquad W\text{ independent of }C,\qquad\mathbb{E}C^2=1/q. U = ( C , 1 − C 2 W ) , W ∼ Unif ( S q − 2 ) , W independent of C , E C 2 = 1/ q . It is valid after any specified orthogonal change of coordinates. For q = 1 q=1 q = 1 the sphere is a fair sign.
Proof. Let ( G 1 , G ′ ) (G_1,G') ( G 1 , G ′ ) be standard Gaussian in R × R q − 1 \mathbb{R}\times\mathbb{R}^{q-1} R × R q − 1 . Its normalized direction is uniform by rotational invariance of Gaussian density. Polar coordinates for G ′ G' G ′ show that W = G ′ / ∥ G ′ ∥ W=G'/\lVert G'\rVert W = G ′ / ∥ G ′ ∥ is uniform and independent of ( G 1 , ∥ G ′ ∥ ) (G_1,\lVert G'\rVert) ( G 1 , ∥ G ′ ∥) ; for q = 2 q=2 q = 2 this is independence of the sign and magnitude of a scalar Gaussian. Set C = G 1 / G 1 2 + ∥ G ′ ∥ 2 C=G_1/\sqrt{G_1^2+\lVert G'\rVert^2} C = G 1 / G 1 2 + ∥ G ′ ∥ 2 . Coordinate exchangeability and ∑ i U i 2 = 1 \sum_i U_i^2=1 ∑ i U i 2 = 1 give E C 2 = 1 / q \mathbb{E}C^2=1/q E C 2 = 1/ q . For a specified unit direction v v v , a measurable alignment is the Householder matrix I − 2 ( e 1 − v ) ( e 1 − v ) ⊤ / ∥ e 1 − v ∥ 2 I-2(e_1-v)(e_1-v)^\top/\lVert e_1-v\rVert^2 I − 2 ( e 1 − v ) ( e 1 − v ) ⊤ / ∥ e 1 − v ∥ 2 if v ≠ e 1 v\ne e_1 v = e 1 and I I I otherwise; it preserves the uniform law.
Lemma F.2 (Adaptive projections: energy and antipodes). Let u u u be uniform on the unit sphere in an e e e -dimensional Euclidean space and independent of an algorithm’s seed. After at most w w w adaptively chosen exact linear observations, where 0 ≤ w ≤ e − 1 0\le w\le e-1 0 ≤ w ≤ e − 1 , any measurable estimate z z z satisfies
P ( ∥ z − u ∥ ≤ r ) ≤ 1 2 + w 2 e ( 1 − r 2 ) ( 0 ≤ r < 1 ) . (58) \mathbb{P}(\lVert z-u\rVert\le r)\le\frac{1}{2}+\frac{w}{2e(1-r^2)}\qquad(0\le r<1). \tag*{(58)} P (∥ z − u ∥ ≤ r ) ≤ 2 1 + 2 e ( 1 − r 2 ) w ( 0 ≤ r < 1 ) . (58)
Proof. Fix the seed. Orthogonalize independent queried directions, omitting dependent directions whose replies are already determined. After termination pad the span using Gram–Schmidt on a fixed basis until its dimension is exactly w w w ; the extra observations can be revealed to the analysis even if the estimator ignores them. The successive unit directions v j v_j v j are measurable functions of previous observations, and each is orthogonal to those preceding it.
We construct the joint law explicitly on a product space. Begin with residual squared norm 1 1 1 and an orthonormal basis. At step j j j , whose residual dimension is q = e − j q=e-j q = e − j , align the next chosen direction to the first residual axis by Lemma F.1 . Draw an independent scalar C j C_j C j with the law of the first coordinate in dimension q q q . Reveal α j = 1 − Z j C j \alpha_j = \sqrt{1-Z_j}C_j α j = 1 − Z j C j , define Z j + 1 = Z j + ( 1 − Z j ) C j 2 Z_{j+1}=Z_j+(1-Z_j)C_j^2 Z j + 1 = Z j + ( 1 − Z j ) C j 2 , and use the remaining aligned axes as a basis of the new residual subspace. Finally draw an independent uniform W W W in the residual dimension e − w e-w e − w and form
u = ∑ j = 0 w − 1 α j v j + 1 − Z w B w W , (59) u=\sum_{j=0}^{w-1}\alpha_jv_j+\sqrt{1-Z_w}B_wW, \tag*{(59)} u = j = 0 ∑ w − 1 α j v j + 1 − Z w B w W , (59)
where B w B_w B w is the residual isometry determined by the revealed history. This produces the same joint integrals as a uniform u u u together with the observations selected by the adaptive rule: at each step apply the product representation of Lemma F.1 to the innermost integral over the residual sphere, hold the outer history fixed, and then use Fubini. Induction over the finite w w w steps establishes this for every bounded measurable test function of ( u , history ) (u,\text{history}) ( u , history ) . When the residual radius is zero, any residual basis can be used without changing the vector. No regular conditional probability is invoked.
Because C j C_j C j is independent of the preceding history and has second moment 1 / ( e − j ) 1/(e-j) 1/ ( e − j ) ,
E Z j + 1 = E Z j + 1 − E Z j e − j , E Z w = w / e . \mathbb{E}Z_{j+1}=\mathbb{E}Z_j+\frac{1-\mathbb{E}Z_j}{e-j},\qquad\mathbb{E}Z_w=w/e. E Z j + 1 = E Z j + e − j 1 − E Z j , E Z w = w / e .
Fix a history on the product space and hence the estimator z z z . If 1 − Z w > r 2 1-Z_w>r^2 1 − Z w > r 2 , the two vectors in (59) generated by W W W and − W -W − W have separation 2 1 − Z w > 2 r 2\sqrt{1-Z_w}>2r 2 1 − Z w > 2 r , so at most one lies within distance r r r of z z z . Paired integration over the final symmetric sphere bounds success by 1 / 2 1/2 1/2 on this event, and by one otherwise. Markov’s inequality gives P ( Z w ≥ 1 − r 2 ) ≤ w / [ e ( 1 − r 2 ) ] \mathbb{P}(Z_w\ge1-r^2)\le w/[e(1-r^2)] P ( Z w ≥ 1 − r 2 ) ≤ w / [ e ( 1 − r 2 )] . This proves (58) ; average over the independent seed.
Proposition F.3 (Quadratic randomized lower bound). Let W ⊆ R d W\subseteq\mathbb{R}^d W ⊆ R d be a public subspace of dimension e ≥ 1 e\ge1 e ≥ 1 , let ℓ , Δ , ε > 0 \ell,\Delta,\varepsilon>0 ℓ , Δ , ε > 0 , and suppose θ = ℓ Δ / ε 2 > 1 / 2 \theta=\ell\Delta/\varepsilon^2>1/2 θ = ℓ Δ/ ε 2 > 1/2 . Define
a θ = 1 − 1 2 θ , R 0 = 2 Δ ℓ , f u ( x ) = ℓ 2 ∥ P W x − R 0 u ∥ 2 , u ∼ Unif ( S ( W ) ) . a_\theta=1-\frac{1}{2\theta},\qquad R_0=\sqrt{\frac{2\Delta}{\ell}},\qquad f_u(x)=\frac{\ell}{2}\lVert P_Wx-R_0u\rVert^2,\qquad u\sim\operatorname{Unif}(S(W)). a θ = 1 − 2 θ 1 , R 0 = ℓ 2Δ , f u ( x ) = 2 ℓ ∥ P W x − R 0 u ∥ 2 , u ∼ Unif ( S ( W )) . Every f u f_u f u is convex and ℓ \ell ℓ -smooth, has initial gap exactly Δ \Delta Δ , and admits the global fixed upper matrix ℓ P W \ell P_W ℓ P W . Any randomized algorithm successful with probability at least 2 / 3 2/3 2/3 on every f u f_u f u has rank-priced cost cap at least e a θ / 3 ea_\theta/3 e a θ /3 and worst-case expected rank-priced cost at least e a θ / 36 ea_\theta/36 e a θ /36 . These also hold for exact values.
Proof. We have
f u ( x ) = ℓ 2 ∥ P W x ∥ 2 + Δ − ℓ R 0 ⟨ u , P W x ⟩ , ∇ f u ( x ) = ℓ ( P W x − R 0 u ) . f_u(x)=\frac{\ell}{2}\lVert P_Wx\rVert^2+\Delta-\ell R_0\langle u,P_Wx\rangle,\qquad\nabla f_u(x)=\ell(P_Wx-R_0u). f u ( x ) = 2 ℓ ∥ P W x ∥ 2 + Δ − ℓ R 0 ⟨ u , P W x ⟩ , ∇ f u ( x ) = ℓ ( P W x − R 0 u ) .
A response to call ( x , V ) (x,V) ( x , V ) is determined by the projections of u u u on span { P W x } + col ( P W V ) \operatorname{span}\{P_Wx\}+\operatorname{col}(P_WV) span { P W x } + col ( P W V ) , whose dimension is at most 1 + rank V 1+\operatorname{rank}V 1 + rank V and is chosen before the reply. Revealing a basis of those projections simulates the response within its cost. Success means ∥ P W x ~ / R 0 − u ∥ ≤ ( 2 θ ) − 1 / 2 \lVert P_W\widetilde{x}/R_0-u\rVert\le(2\theta)^{-1/2} ∥ P W x / R 0 − u ∥ ≤ ( 2 θ ) − 1/2 . Costs are integers. If a cost cap w w w is below e e e , Lemma F.2 bounds average success over u u u by 1 / 2 + w / ( 2 e a θ ) 1/2+w/(2ea_\theta) 1/2 + w / ( 2 e a θ ) , so success probability 2 / 3 2/3 2/3 forces w ≥ e a θ / 3 w\ge ea_\theta/3 w ≥ e a θ /3 . Caps w ≥ e w\ge e w ≥ e satisfy this since a θ < 1 a_\theta<1 a θ < 1 .
For expected cost let W W W be the random cost. For every integer 0 ≤ k < e 0\le k<e 0 ≤ k < e , truncate before any call that would raise cost above k k k , returning a fixed point. It agrees with the original whenever W < k W<k W < k ; therefore, with probability averaged over u u u and the seed, P ( W > k ) ≥ ( 1 / 6 − k / ( 2 e a θ ) ) + \mathbb{P}(W>k)\ge(1/6-k/(2ea_\theta))_+ P ( W > k ) ≥ ( 1/6 − k / ( 2 e a θ ) ) + . Since e a θ / 3 < e ea_\theta/3<e e a θ /3 < e , tail summation gives
E W ≥ ∑ k ≥ 0 ( 1 6 − k 2 e a θ ) + ≥ ∫ 0 e a θ / 3 ( 1 6 − t 2 e a θ ) d t = e a θ 36 . \mathbb{E}W\ge\sum_{k\ge0}\left(\frac{1}{6}-\frac{k}{2ea_\theta}\right)_+\ge\int_0^{ea_\theta/3}\left(\frac{1}{6}-\frac{t}{2ea_\theta}\right)\,dt=\frac{ea_\theta}{36}. E W ≥ k ≥ 0 ∑ ( 6 1 − 2 e a θ k ) + ≥ ∫ 0 e a θ /3 ( 6 1 − 2 e a θ t ) d t = 36 e a θ .
as the integrand is decreasing and nonnegative. The supremum over u u u of conditional expected cost is at least this average. Exact value calls have rank zero.
Corollary F.4 (Randomized baseline on the trace class). For all d ≥ 1 d \ge1 d ≥ 1 , 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , and T ≥ 1 T \ge1 T ≥ 1 ,
N r a n d , E v a l ≥ N r a n d , E l i n ≥ min { d , E T } 12 , N^{\mathrm{val}}_{\mathrm{rand},E} \ge N^{\mathrm{lin}}_{\mathrm{rand},E} \ge\frac{\min\{d,ET\}}{12}, N rand , E val ≥ N rand , E lin ≥ 12 min { d , E T } , and both worst-case expected costs are at least min { d , E T } / 144 \min\{d,ET\}/144 min { d , E T } /144 .
Proof. Take e = min { d , ⌊ E T ⌋ } e=\min\{d,\lfloor ET\rfloor\} e = min { d , ⌊ E T ⌋} , a coordinate subspace W W W of dimension e e e , and ℓ = min { L , L E / e } \ell=\min\{L,LE/e\} ℓ = min { L , L E / e } . Since E T ≥ 1 ET \ge1 E T ≥ 1 , e ≥ min { d , E T } / 2 e \ge\min\{d,ET\}/2 e ≥ min { d , E T } /2 . Further, ℓ ≤ L \ell\le L ℓ ≤ L , ℓ e ≤ L E \ell e \le LE ℓ e ≤ L E , and θ = min { T , E T / e } ≥ 1 \theta=\min\{T,ET/e\} \ge1 θ = min { T , E T / e } ≥ 1 . Thus these quadratics lie in F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , a θ ≥ 1 / 2 a_\theta\ge1/2 a θ ≥ 1/2 , and the proposition yields cap at least e / 6 e/6 e /6 and expected cost at least e / 72 e/72 e /72 .
Further Results
Theorem A.1 on B \mathcal{B} B does not use this appendix; its extension to F \mathcal{F} F uses Section G.3. The appendix defines the pointwise classes, records the inclusions between the classes, gives the randomized upper bound on the larger class A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d , and C ∞ C^\infty C ∞ hard functions with a Lipschitz Hessian in the global fixed-matrix class F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , which give the deterministic lower bounds on every class that contains it.
Pointwise classes and inclusions
Our randomized upper bound holds on a larger class, the local pointwise class A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d , that requires neither a common matrix H H H nor smoothness above f ( 0 ) f(0) f ( 0 ) : it consists of the continuously differentiable f f f with f ( 0 ) − inf f ≤ Δ f(0)-\inf f \le\Delta f ( 0 ) − inf f ≤ Δ whose gradient is locally L L L -Lipschitz on U U U and which satisfy tr [ ( ∇ 2 f ( x ) ) + ] ≤ L E \operatorname{tr}[(\nabla^{2}f(x))_{+}] \le LE tr [( ∇ 2 f ( x ) ) + ] ≤ L E for almost every x ∈ U x \in U x ∈ U , where A + A_{+} A + keeps the positive eigenvalues of a symmetric matrix A A A (U U U need not be bounded or connected). Since A ⪯ H A \preceq H A ⪯ H and H ⪰ 0 H \succeq0 H ⪰ 0 imply tr A + ≤ tr H \operatorname{tr} A_{+} \le\operatorname{tr} H tr A + ≤ tr H , we have B L , Δ , E d ⊆ A L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} \subseteq\mathcal{A}^{d}_{L,\Delta,E} B L , Δ , E d ⊆ A L , Δ , E d . The global pointwise class P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d requires f ∈ F L , Δ d f \in\mathcal{F}^{d}_{L,\Delta} f ∈ F L , Δ d and tr [ ( ∇ 2 f ( x ) ) + ] ≤ L E \operatorname{tr}[(\nabla^{2}f(x))_{+}] \le LE tr [( ∇ 2 f ( x ) ) + ] ≤ L E for almost every x ∈ R d x \in\mathbb{R}^{d} x ∈ R d .
Lemma G.1 (Class inclusions). At each fixed choice of the public parameters,
F L , Δ , E d ⊆ P L , Δ , E d ⊆ A L , Δ , E d , F L , Δ , E d ⊆ B L , Δ , E d ⊆ A L , Δ , E d . \mathcal{F}^{d}_{L,\Delta,E} \subseteq\mathcal{P}^{d}_{L,\Delta,E} \subseteq\mathcal{A}^{d}_{L,\Delta,E}, \qquad\mathcal{F}^{d}_{L,\Delta,E} \subseteq\mathcal{B}^{d}_{L,\Delta,E} \subseteq\mathcal{A}^{d}_{L,\Delta,E}. F L , Δ , E d ⊆ P L , Δ , E d ⊆ A L , Δ , E d , F L , Δ , E d ⊆ B L , Δ , E d ⊆ A L , Δ , E d .
Proof. Let f ∈ F L , Δ , E d f \in\mathcal{F}^{d}_{L,\Delta,E} f ∈ F L , Δ , E d have upper model H H H . At a point where ∇ f \nabla f ∇ f is differentiable, the quadratic inequality (3) , applied to t v tv t v and letting t → 0 t \to0 t → 0 , gives v ⊤ ∇ 2 f ( x ) v ≤ v ⊤ H v v^{\top}\nabla^{2}f(x)v \le v^{\top}Hv v ⊤ ∇ 2 f ( x ) v ≤ v ⊤ H v for every v v v . By Rademacher’s theorem this holds almost everywhere on R d \mathbb{R}^{d} R d , so f ∈ B L , Δ , E d f \in\mathcal{B}^{d}_{L,\Delta,E} f ∈ B L , Δ , E d . For symmetric A ⪯ H A \preceq H A ⪯ H with H ⪰ 0 H \succeq0 H ⪰ 0 , let P + P_{+} P + be the spectral projection onto the positive eigenspaces of A A A . Then
tr A + = tr ( P + A ) ≤ tr ( P + H ) ≤ tr H ≤ L E . \operatorname{tr} A_{+}=\operatorname{tr}(P_{+}A)\le\operatorname{tr}(P_{+}H)\le\operatorname{tr} H\le LE. tr A + = tr ( P + A ) ≤ tr ( P + H ) ≤ tr H ≤ L E .
Applied almost everywhere on R d \mathbb{R}^{d} R d , this gives f ∈ P L , Δ , E d f \in\mathcal{P}^{d}_{L,\Delta,E} f ∈ P L , Δ , E d . Applied almost everywhere on the initial sublevel set, it gives the trace bound of A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d for every f ∈ B L , Δ , E d f \in\mathcal{B}^{d}_{L,\Delta,E} f ∈ B L , Δ , E d . Finally, a globally L L L -Lipschitz gradient is locally L L L -Lipschitz on the initial sublevel set, and a trace bound on R d \mathbb{R}^{d} R d holds in particular there; this gives both inclusions in A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d .
Consequently lower bounds proved on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d and upper bounds proved on A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d hold on all four classes.
The randomized upper bound on the local pointwise class
Recall that f ∈ A L , Δ , E d f \in\mathcal{A}^{d}_{L,\Delta,E} f ∈ A L , Δ , E d means f ∈ C 1 ( R d ) f \in C^{1}(\mathbb{R}^{d}) f ∈ C 1 ( R d ) , f ( 0 ) − inf f ≤ Δ f(0)-\inf f \le\Delta f ( 0 ) − inf f ≤ Δ , and, on U = { f < f ( 0 ) } U=\{f<f(0)\} U = { f < f ( 0 )} , a gradient that is locally L L L -Lipschitz with one constant throughout U U U and tr ( ( ∇ 2 f ) + ) ≤ L E \operatorname{tr}((\nabla^{2}f)_{+})\le LE tr (( ∇ 2 f ) + ) ≤ L E almost everywhere. No common matrix is assumed, so a capped function replaces the first exit Lemma B.3 .
Lemma G.2 (Capped function). Let f ∈ A L , Δ , E d f \in\mathcal{A}^{d}_{L,\Delta,E} f ∈ A L , Δ , E d , c = f ( 0 ) c=f(0) c = f ( 0 ) , G = 2 L Δ G=\sqrt{2L\Delta} G = 2 L Δ , and F c = min { f , c } F_c=\min\{f,c\} F c = min { f , c } . Then F c F_c F c is globally G G G -Lipschitz and bounded. For f ( x ) ≤ c f(x)\le c f ( x ) ≤ c ,
F c ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + L 2 ∥ v ∥ 2 . (60) F_c(x+v)\le f(x)+\langle\nabla f(x),v\rangle+\frac{L}{2}\lVert v\rVert^{2}. \tag*{(60)} F c ( x + v ) ≤ f ( x ) + ⟨ ∇ f ( x ) , v ⟩ + 2 L ∥ v ∥ 2 . (60) For any fixed orthogonal projection P P P , Z ∼ N ( 0 , P ) Z\sim N(0,P) Z ∼ N ( 0 , P ) , t ≥ 0 t\ge0 t ≥ 0 , and y ∈ R d y\in\mathbb{R}^{d} y ∈ R d ,
E F c ( y + t Z ) ≤ F c ( y ) + L E 2 t . (61) \mathbb{E}F_c(y+\sqrt{t}Z)\le F_c(y)+\frac{LE}{2}t. \tag*{(61)} E F c ( y + t Z ) ≤ F c ( y ) + 2 L E t . (61) Proof. For x ∈ U x\in U x ∈ U , put g = ∇ f ( x ) g=\nabla f(x) g = ∇ f ( x ) . Follow x − t g x-tg x − t g up to t = 1 / L t=1/L t = 1/ L . Before its first possible exit from U U U , local smoothness and integration along the segment give f ( x − t g ) ≤ f ( x ) − t ( 1 − L t / 2 ) ∥ g ∥ 2 f(x-tg)\le f(x)-t(1-Lt/2)\lVert g\rVert^{2} f ( x − t g ) ≤ f ( x ) − t ( 1 − L t /2 ) ∥ g ∥ 2 . A first exit before 1 / L 1/L 1/ L would contradict this strict decrease and continuity. Consequently ∥ g ∥ 2 ≤ 2 L ( f ( x ) − inf f ) ≤ G 2 \lVert g\rVert^{2}\le2L(f(x)-\inf f)\le G^{2} ∥ g ∥ 2 ≤ 2 L ( f ( x ) − inf f ) ≤ G 2 . The restriction of F c F_c F c to any compact line segment is absolutely continuous: f f f is C 1 C^{1} C 1 there and truncation is Lipschitz. Its derivative is bounded in magnitude by G G G almost everywhere, being either a derivative of f f f below c c c or zero. This proves the global Lipschitz bound. Boundedness follows from inf f ≤ F c ≤ c \inf f\le F_c\le c inf f ≤ F c ≤ c .
We argue by smoothing. For δ > 0 \delta>0 δ > 0 , choose a smooth concave nondecreasing function ϕ δ \phi_\delta ϕ δ with 0 ≤ ϕ δ ′ ≤ 1 0\le\phi'_\delta\le1 0 ≤ ϕ δ ′ ≤ 1 , derivative one below c − 2 δ c-2\delta c − 2 δ , derivative zero above c − δ c-\delta c − δ , and sup s ∣ ϕ δ ( s ) − min ( s , c ) ∣ ≤ 2 δ \sup_s\lvert\phi_\delta(s)-\min(s,c)\rvert\le2\delta sup s ∣ ϕ δ ( s ) − min ( s , c )∣ ≤ 2 δ . It is obtained by integrating a smooth decreasing transition on [ c − 2 δ , c − δ ] [c-2\delta,c-\delta] [ c − 2 δ , c − δ ] and fixing the additive constant. The function u δ = ϕ δ ∘ f u_\delta=\phi_\delta\circ f u δ = ϕ δ ∘ f is globally C 1 , 1 C^{1,1} C 1 , 1 : where its derivatives are nonzero one is in U U U , its Hessian has norm bounded by L + ∥ ϕ δ ′ ′ ∥ ∞ G 2 L+\lVert\phi''_\delta\rVert_\infty G^{2} L + ∥ ϕ δ ′′ ∥ ∞ G 2 , and near the remaining points the composition is constant. Local bounds with this common constant integrate to a global bound. Almost everywhere,
∇ 2 u δ = ϕ δ ′ ( f ) ∇ 2 f + ϕ δ ′ ′ ( f ) ∇ f ∇ f T ⪯ L I , tr ( P ∇ 2 u δ ) ≤ L E . \nabla^{2}u_\delta=\phi'_\delta(f)\nabla^{2}f+\phi''_\delta(f)\nabla f\nabla f^{\mathsf{T}}\preceq LI,\qquad\operatorname{tr}(P\nabla^{2}u_\delta)\le LE. ∇ 2 u δ = ϕ δ ′ ( f ) ∇ 2 f + ϕ δ ′′ ( f ) ∇ f ∇ f T ⪯ L I , tr ( P ∇ 2 u δ ) ≤ L E . The second inequality uses tr ( P B ) ≤ tr ( B + ) \operatorname{tr}(PB)\le\operatorname{tr}(B_{+}) tr ( P B ) ≤ tr ( B + ) for every symmetric B B B and 0 ⪯ P ⪯ I 0\preceq P\preceq I 0 ⪯ P ⪯ I ; the cutoff’s second term is nonpositive. Convolve in all d d d dimensions with a nonnegative, symmetric, compactly supported smooth approximate identity. Both inequalities survive. For the resulting smooth bounded function u u u , the quadratic upper model follows by integration. Gaussian integration by parts gives
d d t E u ( y + t Z ) = 1 2 E tr ( P ∇ 2 u ( y + t Z ) ) ≤ L E / 2. \frac{d}{dt}\mathbb{E}u(y+\sqrt{t}Z)=\frac{1}{2}\mathbb{E}\operatorname{tr}(P\nabla^{2}u(y+\sqrt{t}Z))\le LE/2. d t d E u ( y + t Z ) = 2 1 E tr ( P ∇ 2 u ( y + t Z )) ≤ L E /2. This is also valid for degenerate P P P by using coordinates on its range; it is applied only after full-dimensional smoothing. Bounded Hessians and Gaussian moments justify differentiation and integration by parts. Integrate from 0 0 0 to t t t , send the convolution radius to zero, and then send δ \delta δ to zero. The convergence of the clipped values is uniform, which proves (61) .
For x ∈ U x\in U x ∈ U , the same argument gives (60) , because u δ ( x ) → f ( x ) u_\delta(x)\to f(x) u δ ( x ) → f ( x ) and ∇ u δ ( x ) → ∇ f ( x ) \nabla u_\delta(x)\to\nabla f(x) ∇ u δ ( x ) → ∇ f ( x ) . If f ( x ) = c f(x)=c f ( x ) = c and ∇ f ( x ) ≠ 0 \nabla f(x)\ne0 ∇ f ( x ) = 0 , approach x x x from the local downhill side in U U U and use continuity. If ∇ f ( x ) = 0 \nabla f(x)=0 ∇ f ( x ) = 0 , use F c ≤ c F_c\le c F c ≤ c . □ \square □
In Lemma G.2 the concave scalar cutoff handles the negative curvature created by truncation, and the Gaussian identity is applied only after smoothing in all d d d dimensions, because an almost everywhere bound in R d \mathbb{R}^{d} R d may fail on a hyperplane.
The update below is the stochastic three-point method of Theorem B.8 (Algorithm 4), with the curvature parameter L ( E + 1 ) L(E+1) L ( E + 1 ) in place of L E LE L E .
Theorem G.3 (Three-point comparison on A \mathcal{A} A ). Let M = L ( E + 1 ) M=L(E+1) M = L ( E + 1 ) , h = 4 ε / ( 5 M ) h=4\varepsilon/(5M) h = 4 ε / ( 5 M ) , and N = ⌈ 10 M Δ / ε 2 ⌉ N=\lceil10M\Delta/\varepsilon^2\rceil N = ⌈ 10 M Δ/ ε 2 ⌉ . Query f ( 0 ) f(0) f ( 0 ) once and cache the current value. At each iteration draw a ∼ N ( 0 , I d ) a\sim N(0,I_d) a ∼ N ( 0 , I d ) , query f ( x + h a ) f(x+ha) f ( x + ha ) and f ( x − h a ) f(x-ha) f ( x − ha ) , and retain a point of lowest value among { x , x + h a , x − h a } \{x,x+ha,x-ha\} { x , x + ha , x − ha } , keeping x x x on a tie. Return an independent uniformly selected iterate from before the updates. For every fixed f ∈ A f\in\mathcal{A} f ∈ A , the probability of failure is at most 25 / 78 < 1 / 3 25/78<1/3 25/78 < 1/3 , and the hard query cap is
Q = 1 + 2 ⌈ 10 ( E + 1 ) T ⌉ ≤ 43 E T ( E , T ≥ 1 ) . Q=1+2\lceil10(E+1)T\rceil\le43ET\qquad(E,T\ge1). Q = 1 + 2 ⌈ 10 ( E + 1 ) T ⌉ ≤ 43 E T ( E , T ≥ 1 ) .
Proof. All centers satisfy f ( x ) ≤ c = f ( 0 ) f(x)\le c=f(0) f ( x ) ≤ c = f ( 0 ) . Fix such a center and write g = ∇ f ( x ) ≠ 0 g=\nabla f(x)\ne0 g = ∇ f ( x ) = 0 , u = g / ∥ g ∥ u=g/\lVert g\rVert u = g / ∥ g ∥ , and a = s u + z a=su+z a = s u + z , where s ∼ N ( 0 , 1 ) s\sim N(0,1) s ∼ N ( 0 , 1 ) and z ∼ N ( 0 , I − u u ⊤ ) z\sim N(0,I-uu^\top) z ∼ N ( 0 , I − u u ⊤ ) are independent. Of the two proposals, one is
x − h ∣ s ∣ u − h sign ( s ) z . x-h|s|u-h\operatorname{sign}(s)z. x − h ∣ s ∣ u − h sign ( s ) z .
Although the algorithm does not know this sign, it compares both proposals. Conditioning on s s s , apply (61) in the orthogonal subspace, then apply (60) at x x x along − h ∣ s ∣ u -h|s|u − h ∣ s ∣ u . Since the minimum of the three true values is at most the capped value of this particular proposal, this gives
E [ f ( x + ) ∣ x ] ≤ f ( x ) − α 0 h ∥ g ∥ + M 2 h 2 , α 0 = E ∣ s ∣ = 2 π > 79 / 100. (62) \mathbb{E}[f(x^+)\mid x]\le f(x)-\alpha_0h\lVert g\rVert+\frac{M}{2}h^2,\qquad\alpha_0=\mathbb{E}|s|=\sqrt{\frac{2}{\pi}}>79/100. \tag*{(62)} E [ f ( x + ) ∣ x ] ≤ f ( x ) − α 0 h ∥ g ∥ + 2 M h 2 , α 0 = E ∣ s ∣ = π 2 > 79/100. (62)
Only capped functions and the accepted value are integrated, so the value at a rejected proposal need not have a finite Gaussian expectation.
The accepted value never increases. Thus when ∥ g ∥ > ε \lVert g\rVert>\varepsilon ∥ g ∥ > ε the conditional expected decrease is at least
( 79 100 4 5 − 1 2 16 25 ) ε 2 M = 39 125 ε 2 M , \left(\frac{79}{100}\frac{4}{5}-\frac{1}{2}\frac{16}{25}\right)\frac{\varepsilon^2}{M}
=\frac{39}{125}\frac{\varepsilon^2}{M}, ( 100 79 5 4 − 2 1 25 16 ) M ε 2 = 125 39 M ε 2 ,
and at every other center it is nonnegative. Summation and the initial gap yield
1 N ∑ t = 0 N − 1 P ( ∥ ∇ f ( x t ) ∥ > ε ) ≤ 125 M Δ 39 N ε 2 ≤ 25 78 . \frac{1}{N}\sum_{t=0}^{N-1}\mathbb{P}(\lVert\nabla f(x_t)\rVert>\varepsilon)
\le\frac{125M\Delta}{39N\varepsilon^2}\le\frac{25}{78}. N 1 t = 0 ∑ N − 1 P (∥ ∇ f ( x t )∥ > ε ) ≤ 39 N ε 2 125 M Δ ≤ 78 25 .
This is exactly the failure probability of the independent uniform output. Every iteration uses two new values, including unsuccessful proposals. The final bound follows from 1 + 2 ⌈ 10 ( E + 1 ) T ⌉ ≤ 3 + 20 ( E + 1 ) T ≤ 43 E T 1+2\lceil10(E+1)T\rceil\le3+20(E+1)T\le43ET 1 + 2 ⌈ 10 ( E + 1 ) T ⌉ ≤ 3 + 20 ( E + 1 ) T ≤ 43 E T .
:::
Smooth lower bounds on the global classes
This subsection proves the deterministic lower bounds of Theorem 3.2 and Corollary 3.3 with C ∞ C^\infty C ∞ witnesses that have a Lipschitz Hessian and lie in the global fixed-matrix class F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , and hence in P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d and B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d . It changes the witnesses, not the definition of the optimization oracle. Smoothing an entire hard function need not keep the hidden directions invisible to exact values, so we smooth the one-dimensional building blocks of the chain of Appendix § 13 .1 and retain an open interval on which each gate is exactly zero. The smoothed chain is multiplied by a perspective weight instead of being added to a radial potential, so the Hessian bound holds on all of R d \mathbb{R}^d R d .
The constants of this subsection are derived with the following selection lemma. It uses the symmetric Gaussian rectangle inequality ([49 ] ) instead of Lemma C.2 , and its constant 12 / 5 12/5 12/5 is smaller than the constant 13 / 3 13/3 13/3 there.
Lemma G.4 (Subspace strip avoidance). Let S ⊂ R d S\subset\mathbb{R}^d S ⊂ R d have dimension p ≥ 1 p\ge1 p ≥ 1 , and let a 1 , … , a M ∈ R d a_1,\ldots,a_M\in\mathbb{R}^d a 1 , … , a M ∈ R d satisfy ∥ a i ∥ ≤ R \lVert a_i\rVert\le R ∥ a i ∥ ≤ R . There is a unit u ∈ S u\in S u ∈ S such that
max i ≤ M ∣ ⟨ u , a i ⟩ ∣ ≤ 12 5 R log ( e + M / p ) p . (63) \max_{i\le M}|\langle u,a_i\rangle|\le\frac{12}{5}R\sqrt{\frac{\log(e+M/p)}{p}}. \tag*{(63)} i ≤ M max ∣ ⟨ u , a i ⟩ ∣ ≤ 5 12 R p log ( e + M / p ) . (63)
Proof. If M < p M < p M < p or R = 0 R = 0 R = 0 , choose a unit vector u ∈ S u \in S u ∈ S orthogonal to the projections of a 1 , … , a M a_1,\ldots,a_M a 1 , … , a M onto S S S . Assume now that s : = M / p ≥ 1 s := M/p \ge1 s := M / p ≥ 1 and R > 0 R > 0 R > 0 . Draw a standard Gaussian g g g in S S S and put t = 9 5 log ( e + s ) > 2 t = \frac{9}{5}\sqrt{\log(e+s)} > 2 t = 5 9 log ( e + s ) > 2 . Each X i = ⟨ g , a i ⟩ X_i = \langle g,a_i\rangle X i = ⟨ g , a i ⟩ is centered Gaussian with variance at most R 2 R^2 R 2 , so
P { ∣ X i ∣ > R t } ≤ 2 π e − t 2 / 2 t < 4 5 t ( e + s ) − 81 / 50 < 1 16 s . \mathbb{P}\{|X_i| > Rt\} \le\sqrt{\frac{2}{\pi}}\frac{e^{-t^2/2}}{t} < \frac{4}{5t}(e+s)^{-81/50} < \frac{1}{16s}. P { ∣ X i ∣ > R t } ≤ π 2 t e − t 2 /2 < 5 t 4 ( e + s ) − 81/50 < 16 s 1 .
The symmetric Gaussian rectangle inequality gives
P { max i ∣ X i ∣ ≤ R t } ≥ ∏ i = 1 M P { ∣ X i ∣ ≤ R t } ≥ ( 1 − 1 16 s ) M ≥ e − p / 15 . \mathbb{P}\left\{\max_i |X_i| \le Rt\right\} \ge\prod_{i=1}^{M}\mathbb{P}\{|X_i| \le Rt\} \ge\left(1-\frac{1}{16s}\right)^M \ge e^{-p/15}. P { i max ∣ X i ∣ ≤ R t } ≥ i = 1 ∏ M P { ∣ X i ∣ ≤ R t } ≥ ( 1 − 16 s 1 ) M ≥ e − p /15 .
using log ( 1 − z ) ≥ − z / ( 1 − z ) \log(1-z) \ge-z/(1-z) log ( 1 − z ) ≥ − z / ( 1 − z ) and M / s = p M/s = p M / s = p . Since ∥ g ∥ 2 \lVert g\rVert^2 ∥ g ∥ 2 is chi-squared with p p p degrees of freedom, exponential Markov with parameter 7 / 18 7/18 7/18 gives P { ∥ g ∥ 2 ≤ 9 16 p } ≤ ( 3 4 e 7 / 32 ) p < e − p / 15 \mathbb{P}\{\lVert g\rVert^2 \le\frac{9}{16}p\} \le\left(\frac{3}{4}e^{7/32}\right)^p < e^{-p/15} P {∥ g ∥ 2 ≤ 16 9 p } ≤ ( 4 3 e 7/32 ) p < e − p /15 . Hence some g g g satisfies both max i ∣ X i ∣ ≤ R t \max_i |X_i| \le Rt max i ∣ X i ∣ ≤ R t and ∥ g ∥ > 3 4 p \lVert g\rVert> \frac{3}{4}\sqrt{p} ∥ g ∥ > 4 3 p , and u = g / ∥ g ∥ u = g/\lVert g\rVert u = g / ∥ g ∥ satisfies max i ∣ ⟨ u , a i ⟩ ∣ ≤ 4 3 R t / p = 12 5 R log ( e + s ) / p \max_i |\langle u,a_i\rangle| \le\frac{4}{3}Rt/\sqrt{p} = \frac{12}{5}R\sqrt{\log(e+s)/p} max i ∣ ⟨ u , a i ⟩ ∣ ≤ 3 4 R t / p = 5 12 R log ( e + s ) / p .
One-dimensional smoothing
Let φ , ψ \varphi,\psi φ , ψ be the scalar functions of Appendix D.1. An equivalent definition, useful for checking their bounds, is
φ ( t ) = ∫ − ∞ t ( min { 1 , 4 ( s + 5 / 4 ) , 4 ( 5 / 4 − s ) } ) + d s , ψ ( t ) = ∫ − ∞ t 16 min { ( s − 1 / 2 ) + , ( 1 − s ) + } d s . \varphi(t) = \int_{-\infty}^{t} \bigl(\min\{1,4(s+5/4),4(5/4-s)\}\bigr)_+\,ds,\qquad\psi(t) = \int_{-\infty}^{t} 16\min\{(s-1/2)_+,(1-s)_+\}\,ds. φ ( t ) = ∫ − ∞ t ( min { 1 , 4 ( s + 5/4 ) , 4 ( 5/4 − s )} ) + d s , ψ ( t ) = ∫ − ∞ t 16 min {( s − 1/2 ) + , ( 1 − s ) + } d s .
Thus 0 ≤ φ ≤ 9 / 4 0 \le\varphi\le9/4 0 ≤ φ ≤ 9/4 , φ ′ ( t ) = 1 \varphi'(t)=1 φ ′ ( t ) = 1 on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] , ∣ φ ′ ′ ∣ ≤ 4 |\varphi''| \le4 ∣ φ ′′ ∣ ≤ 4 almost everywhere, 0 ≤ ψ ′ ≤ 4 0 \le\psi' \le4 0 ≤ ψ ′ ≤ 4 , and ∣ ψ ′ ′ ∣ ≤ 16 |\psi''| \le16 ∣ ψ ′′ ∣ ≤ 16 almost everywhere. Put η = 1 / 24 \eta= 1/24 η = 1/24 and
w ( t ) = Z − 1 exp [ − 1 / ( 1 − t 2 ) ] 1 { ∣ t ∣ < 1 } , Z = ∫ − 1 1 exp [ − 1 / ( 1 − t 2 ) ] d t , w η ( t ) = η − 1 w ( t / η ) . w(t) = Z^{-1}\exp[-1/(1-t^2)]\mathbf{1}_{\{|t|<1\}},\qquad Z = \int_{-1}^{1}\exp[-1/(1-t^2)]\,dt,\qquad w_\eta(t) = \eta^{-1}w(t/\eta). w ( t ) = Z − 1 exp [ − 1/ ( 1 − t 2 )] 1 { ∣ t ∣ < 1 } , Z = ∫ − 1 1 exp [ − 1/ ( 1 − t 2 )] d t , w η ( t ) = η − 1 w ( t / η ) .
Define
φ ~ = φ ∗ w η , ψ ~ ( t ) = ( ψ ∗ w η ) ( t + 2 η ) . (64) \widetilde{\varphi} = \varphi* w_\eta,\qquad\widetilde{\psi}(t) = (\psi* w_\eta)(t+2\eta). \tag*{(64)} φ = φ ∗ w η , ψ ( t ) = ( ψ ∗ w η ) ( t + 2 η ) . (64)
The shift by 2 η 2\eta 2 η moves the transition of ψ ~ \widetilde{\psi} ψ into [ 3 / 8 , 23 / 24 ] [3/8,23/24] [ 3/8 , 23/24 ] , so ψ ~ = 1 \widetilde{\psi}=1 ψ = 1 on [ 1 , ∞ ) [1,\infty) [ 1 , ∞ ) , as is ψ \psi ψ .
Lemma G.5 (Smooth gates and a dimension-free chain estimate). The gates in (64) are C ∞ C^\infty C ∞ . Write a 0 = 3 / 8 a_0=3/8 a 0 = 3/8 and a 1 = 23 / 24 a_1=23/24 a 1 = 23/24 . Then ψ ~ = 0 \widetilde{\psi}=0 ψ = 0 on ( − ∞ , a 0 ] (-\infty,a_0] ( − ∞ , a 0 ] , ψ ~ = 1 \widetilde{\psi}=1 ψ = 1 on [ a 1 , ∞ ) [a_1,\infty) [ a 1 , ∞ ) , 0 ≤ ψ ~ ′ ≤ 4 0\le\widetilde{\psi}'\le4 0 ≤ ψ ′ ≤ 4 , ∣ ψ ~ ′ ′ ∣ ≤ 16 |\widetilde{\psi}''|\le16 ∣ ψ ′′ ∣ ≤ 16 , and ∣ ψ ~ ′ ′ ′ ∣ < 788 |\widetilde{\psi}'''|<788 ∣ ψ ′′′ ∣ < 788 . The function φ ~ \widetilde{\varphi} φ takes values in [ 0 , 9 / 4 ] [0,9/4] [ 0 , 9/4 ] , φ ~ ( 0 ) = 9 / 8 \widetilde{\varphi}(0)=9/8 φ ( 0 ) = 9/8 , its derivative is even, is one on [ − a 1 , a 1 ] [-a_1,a_1] [ − a 1 , a 1 ] , and satisfies 0 ≤ φ ~ ′ ≤ 1 0\le\widetilde{\varphi}'\le1 0 ≤ φ ′ ≤ 1 , ∣ φ ~ ′ ′ ∣ ≤ 4 |\widetilde{\varphi}''|\le4 ∣ φ ′′ ∣ ≤ 4 , ∣ φ ~ ′ ′ ′ ∣ < 100 |\widetilde{\varphi}'''|<100 ∣ φ ′′′ ∣ < 100 .
For the chain (37) with these gates, denoted g ~ m \widetilde{g}_m g m , put p = ∇ g ~ m ( y ) p=\nabla\widetilde{g}_m(y) p = ∇ g m ( y ) and B = ∇ 2 g ~ m ( y ) B=\nabla^2\widetilde{g}_m(y) B = ∇ 2 g m ( y ) . Then
p i ≤ 0 , ∥ p ∥ ≤ 10 m , ∥ B ∥ op ≤ 42 , ∥ D 3 g ~ m ∥ op ≤ 1872 , (65) p_i\le0,\qquad\lVert p\rVert\le10\sqrt{m},\qquad\lVert B\rVert_{\operatorname{op}}\le42,\qquad\lVert D^3\widetilde{g}_m\rVert_{\operatorname{op}}\le1872, \tag*{(65)} p i ≤ 0 , ∥ p ∥ ≤ 10 m , ∥ B ∥ op ≤ 42 , ∥ D 3 g m ∥ op ≤ 1872 , (65) ∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m , ∣ y ⊤ p ∣ ≤ 8 m , (66) \lVert By\rVert\le44\sqrt{m},\qquad|y^\top By|\le44m,\qquad|y^\top p|\le8m, \tag*{(66)} ∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m , ∣ y ⊤ p ∣ ≤ 8 m , (66) g ~ m ( 0 ) − inf g ~ m ≤ 9 4 ( m − 1 2 ) . (67) \widetilde{g}_m(0)-\inf\widetilde{g}_m\le\frac{9}{4}\left(m-\frac{1}{2}\right). \tag*{(67)} g m ( 0 ) − inf g m ≤ 4 9 ( m − 2 1 ) . (67) If some ∣ y i ∣ < a 1 |y_i|<a_1 ∣ y i ∣ < a 1 , the first such index j j j satisfies p j ≤ − 1 p_j\le-1 p j ≤ − 1 . If y j = 0 y_j=0 y j = 0 and ∣ y i ∣ ≤ a 0 |y_i|\le a_0 ∣ y i ∣ ≤ a 0 for i > j i>j i > j , the exact prefix identities of (39) hold; here φ ~ ( 0 ) = 1 \widetilde{\varphi}(0)=1 φ ( 0 ) = 1 , and the current derivative is − γ j − 1 -\gamma_{j-1} − γ j − 1 , where γ 0 = 1 \gamma_0=1 γ 0 = 1 and γ j − 1 = ψ ~ ( − y j − 1 ) + ψ ~ ( y j − 1 ) ∈ [ 0 , 1 ] \gamma_{j-1}=\widetilde{\psi}(-y_{j-1})+\widetilde{\psi}(y_{j-1})\in[0,1] γ j − 1 = ψ ( − y j − 1 ) + ψ ( y j − 1 ) ∈ [ 0 , 1 ] for j > 1 j>1 j > 1 . Values at such a point may still depend on several earlier coordinates.
Proof. Convolution with a nonnegative probability kernel preserves all the stated bounds on the values and the first and second derivatives. The support and plateau claims follow by enlarging endpoints by η \eta η and applying the shift 2 η 2\eta 2 η . Evenness gives φ ~ ( 0 ) = 9 / 8 \widetilde{\varphi}(0)=9/8 φ ( 0 ) = 9/8 and an even derivative. On [ − 3 / 5 , 3 / 5 ] [-3/5,3/5] [ − 3/5 , 3/5 ] , 1 / ( 1 − t 2 ) ≤ 1 + ( 25 / 16 ) t 2 1/(1-t^2)\le1+(25/16)t^2 1/ ( 1 − t 2 ) ≤ 1 + ( 25/16 ) t 2 gives exp [ − 1 / ( 1 − t 2 ) ] ≥ e − 1 ( 1 − ( 25 / 16 ) t 2 ) \exp[-1/(1-t^2)]\ge e^{-1}(1-(25/16)t^2) exp [ − 1/ ( 1 − t 2 )] ≥ e − 1 ( 1 − ( 25/16 ) t 2 ) , so integrating over [ − 3 / 5 , 3 / 5 ] [-3/5,3/5] [ − 3/5 , 3/5 ] yields Z > ( 39 / 40 ) e − 1 Z>(39/40)e^{-1} Z > ( 39/40 ) e − 1 and w ( 0 ) = e − 1 / Z < 40 / 39 w(0)=e^{-1}/Z<40/39 w ( 0 ) = e − 1 / Z < 40/39 . Because φ ′ ′ \varphi^{\prime\prime} φ ′′ is a step function with jumps of magnitude 4 4 4 at { − 5 / 4 , − 1 , 1 , 5 / 4 } \{-5/4,-1,1,5/4\} { − 5/4 , − 1 , 1 , 5/4 } and ψ ′ ′ \psi^{\prime\prime} ψ ′′ has jumps of magnitude at most 32 32 32 at { 1 / 2 , 3 / 4 , 1 } \{1/2,3/4,1\} { 1/2 , 3/4 , 1 } , and the jumps in each set are separated by 1 / 4 > 2 η 1/4>2\eta 1/4 > 2 η , the bumps in φ ~ ′ ′ ′ = φ ′ ′ ′ ∗ w η \widetilde{\varphi}^{\prime\prime\prime}=\varphi^{\prime\prime\prime}*w_\eta φ ′′′ = φ ′′′ ∗ w η and ψ ~ ′ ′ ′ ( t ) = ( ψ ′ ′ ′ ∗ w η ) ( t + 2 η ) \widetilde{\psi}^{\prime\prime\prime}(t)=(\psi^{\prime\prime\prime}*w_\eta)(t+2\eta) ψ ′′′ ( t ) = ( ψ ′′′ ∗ w η ) ( t + 2 η ) have disjoint supports. Thus ∣ φ ~ ′ ′ ′ ∣ ≤ 4 w ( 0 ) / η < 100 |\widetilde{\varphi}^{\prime\prime\prime}|\le4w(0)/\eta<100 ∣ φ ′′′ ∣ ≤ 4 w ( 0 ) / η < 100 and ∣ ψ ~ ′ ′ ′ ∣ ≤ 32 w ( 0 ) / η < 788 |\widetilde{\psi}^{\prime\prime\prime}|\le32w(0)/\eta<788 ∣ ψ ′′′ ∣ ≤ 32 w ( 0 ) / η < 788 .
The coordinate gradient is the sum of a nonpositive predecessor term of magnitude at most 1 1 1 and a nonpositive successor term of magnitude at most 4 ( 9 / 4 ) = 9 4(9/4)=9 4 ( 9/4 ) = 9 . Since φ ~ ′ ′ \widetilde{\varphi}^{\prime\prime} φ ′′ and φ ~ ′ ′ ′ \widetilde{\varphi}^{\prime\prime\prime} φ ′′′ are supported on { ∣ t ∣ ∈ [ a 1 , 5 / 4 + η ] } \{|t|\in[a_1,5/4+\eta]\} { ∣ t ∣ ∈ [ a 1 , 5/4 + η ]} whereas ψ ~ ′ \widetilde{\psi}^{\prime} ψ ′ , ψ ~ ′ ′ \widetilde{\psi}^{\prime\prime} ψ ′′ , and ψ ~ ′ ′ ′ \widetilde{\psi}^{\prime\prime\prime} ψ ′′′ are supported on { ∣ t ∣ ∈ [ a 0 , a 1 ] } \{|t|\in[a_0,a_1]\} { ∣ t ∣ ∈ [ a 0 , a 1 ]} , their supports meet only at ∣ t ∣ = a 1 |t|=a_1 ∣ t ∣ = a 1 , where both higher derivatives vanish. Convolving (40) with w η ≥ 0 w_\eta\ge0 w η ≥ 0 gives 4 + 16 φ ~ ( s ) + 4 φ ~ ′ ( s ) ≤ 42 4+16\widetilde{\varphi}(s)+4\widetilde{\varphi}^{\prime}(s)\le42 4 + 16 φ ( s ) + 4 φ ′ ( s ) ≤ 42 for every s ∈ R s\in\mathbb{R} s ∈ R , so the Hessian row sum is at most 42 42 42 for ∣ y i ∣ ≤ a 1 |y_i|\le a_1 ∣ y i ∣ ≤ a 1 and at most 8 8 8 for ∣ y i ∣ > a 1 |y_i|>a_1 ∣ y i ∣ > a 1 , proving ∥ B ∥ op ≤ 42 \|B\|_{\operatorname{op}}\le42 ∥ B ∥ op ≤ 42 . Similarly, a diagonal third derivative has magnitude at most max { 100 , ( 9 / 4 ) 788 } = 1773 \max\{100,(9/4)788\}=1773 max { 100 , ( 9/4 ) 788 } = 1773 , and adding the six types of adjacent mixed trilinear terms (each bounded by 16 16 16 ) gives 1773 + 6 ⋅ 16 < 1872 1773+6\cdot16<1872 1773 + 6 ⋅ 16 < 1872 .
Every active coordinate has ∣ y i ∣ ≤ 5 / 4 + η = 31 / 24 < 4 / 3 |y_i|\le5/4+\eta=31/24<4/3 ∣ y i ∣ ≤ 5/4 + η = 31/24 < 4/3 . For ∣ y i ∣ ≤ a 1 |y_i|\le a_1 ∣ y i ∣ ≤ a 1 , ∣ ( B y ) i ∣ ≤ 36 a 1 + 4 a 1 + 4 ( 31 / 24 ) = 87 / 2 < 44 |(By)_i|\le36a_1+4a_1+4(31/24)=87/2<44 ∣ ( B y ) i ∣ ≤ 36 a 1 + 4 a 1 + 4 ( 31/24 ) = 87/2 < 44 and ∣ y i ( B y ) i ∣ < 44 |y_i(By)_i|<44 ∣ y i ( B y ) i ∣ < 44 ; for a 1 < ∣ y i ∣ ≤ 31 / 24 a_1<|y_i|\le31/24 a 1 < ∣ y i ∣ ≤ 31/24 , ∣ ( B y ) i ∣ ≤ 4 ( 31 / 24 ) + 4 a 1 < 10 |(By)_i|\le4(31/24)+4a_1<10 ∣ ( B y ) i ∣ ≤ 4 ( 31/24 ) + 4 a 1 < 10 . Also s ψ ′ ( s ) ≤ 3 s\psi'(s)\le3 s ψ ′ ( s ) ≤ 3 and ∣ s φ ′ ( s ) ∣ ≤ 1 |s\varphi'(s)|\le1 ∣ s φ ′ ( s ) ∣ ≤ 1 imply t ψ ~ ′ ( t ) ≤ 3 t\widetilde{\psi}'(t)\le3 t ψ ′ ( t ) ≤ 3 and ∣ t φ ~ ′ ( t ) ∣ ≤ 1 + η |t\widetilde{\varphi}'(t)|\le1+\eta ∣ t φ ′ ( t ) ∣ ≤ 1 + η , so ∣ y i p i ∣ ≤ 1 + η + 27 / 4 < 8 |y_i p_i|\le1+\eta+27/4<8 ∣ y i p i ∣ ≤ 1 + η + 27/4 < 8 . The barrier and prefix identities follow as in Lemma D.1. □ \square □
Perspective scaling and exact derivatives
For R > 0 R>0 R > 0 , define the perspective weight, compression factor, and bounded radial compression map on R d \mathbb{R}^d R d by
w ( x ) = 1 + ∥ x ∥ 2 R 2 , a ( x ) = 1 w ( x ) , z ( x ) = ρ R ( x ) = x w ( x ) = a ( x ) x . (68) w(x)=\sqrt{1+\frac{\lVert x\rVert^2}{R^2}},\qquad a(x)=\frac{1}{w(x)},\qquad z(x)=\rho_R(x)=\frac{x}{w(x)}=a(x)x. \tag*{(68)} w ( x ) = 1 + R 2 ∥ x ∥ 2 , a ( x ) = w ( x ) 1 , z ( x ) = ρ R ( x ) = w ( x ) x = a ( x ) x . (68)
Then w ∈ C ∞ ( R d ) w\in C^\infty(\mathbb{R}^d) w ∈ C ∞ ( R d ) , w ( x ) ≥ 1 w(x)\ge1 w ( x ) ≥ 1 , 0 < a ( x ) ≤ 1 0<a(x)\le1 0 < a ( x ) ≤ 1 , and ∥ z ( x ) ∥ < R \lVert z(x)\rVert<R ∥ z ( x )∥ < R for all x ∈ R d x\in\mathbb{R}^d x ∈ R d . Multiplying a scalar chain by w ( x ) w(x) w ( x ) while evaluating it at the compressed coordinates U ⊤ z ( x ) = U ⊤ x / w ( x ) U^\top z(x)=U^\top x/w(x) U ⊤ z ( x ) = U ⊤ x / w ( x ) forms the perspective function ([10 Section 3.2.6] ) of G G G evaluated at ( U ⊤ x , w ( x ) ) (U^\top x,w(x)) ( U ⊤ x , w ( x )) , namely w ( x ) G ( U ⊤ x / w ( x ) ) w(x)G(U^\top x/w(x)) w ( x ) G ( U ⊤ x / w ( x )) . Because w ( x ) a ( x ) = 1 w(x)a(x)=1 w ( x ) a ( x ) = 1 identically, the weight w ( x ) w(x) w ( x ) cancels the radial attenuation a ( x ) a(x) a ( x ) in the leading chain gradient while keeping the second derivative bounded across all of R d \mathbb{R}^d R d .
Lemma G.6 (Exact derivatives under perspective scaling). Let U = ( u 1 , … , u m ) ∈ R d × m U=(u_1,\ldots,u_m)\in\mathbb{R}^{d\times m} U = ( u 1 , … , u m ) ∈ R d × m have orthonormal columns, let G : R m → R G:\mathbb{R}^m\to\mathbb{R} G : R m → R be continuously differentiable, and set
Φ U ( x ) = w ( x ) G ( U ⊤ ρ R ( x ) ) , y = U ⊤ z ( x ) . \Phi_U(x)=w(x)G(U^\top\rho_R(x)),\qquad y=U^\top z(x). Φ U ( x ) = w ( x ) G ( U ⊤ ρ R ( x )) , y = U ⊤ z ( x ) . At every point x ∈ R d x\in\mathbb{R}^d x ∈ R d , with p = ∇ G ( y ) p=\nabla G(y) p = ∇ G ( y ) ,
∇ Φ U ( x ) = U p + [ G ( y ) − y ⊤ p ] z ( x ) R 2 . (69) \nabla\Phi_U(x)=Up+[G(y)-y^\top p]\frac{z(x)}{R^2}. \tag*{(69)} ∇ Φ U ( x ) = U p + [ G ( y ) − y ⊤ p ] R 2 z ( x ) . (69) At every point x ∈ R d ∗ w h e r e ∗ x\in\mathbb{R}^d *where* x ∈ R d ∗ w h er e ∗ G is twice differentiable at $y with Hessian B = ∇ 2 G ( y ) B=\nabla^2G(y) B = ∇ 2 G ( y ) , for every v ∈ R d , ∗ w r i t i n g ∗ v\in\mathbb{R}^d, *writing* v ∈ R d , ∗ w r i t in g ∗ \alpha=U^\top v$ and β = z ( x ) ⊤ v / R 2 \beta=z(x)^\top v/R^2 β = z ( x ) ⊤ v / R 2 ,
v ⊤ ∇ 2 Φ U ( x ) v = a ( x ) ( α − β y ) ⊤ B ( α − β y ) + a ( x ) [ G ( y ) − y ⊤ p ] ( ∥ v ∥ 2 R 2 − β 2 ) . (70) v^\top\nabla^2\Phi_U(x)v=a(x)(\alpha-\beta y)^\top B(\alpha-\beta y)+a(x)[G(y)-y^\top p]\left(\frac{\lVert v\rVert^2}{R^2}-\beta^2\right). \tag*{(70)} v ⊤ ∇ 2 Φ U ( x ) v = a ( x ) ( α − β y ) ⊤ B ( α − β y ) + a ( x ) [ G ( y ) − y ⊤ p ] ( R 2 ∥ v ∥ 2 − β 2 ) . (70)
If G G G is three times differentiable at y y y , then for every v ∈ R d v \in\mathbb{R}^{d} v ∈ R d , with u = α − β y u=\alpha-\beta y u = α − β y ,
D 3 Φ U ( x ) [ v , v , v ] = a ( x ) 2 D 3 G ( y ) [ u , u , u ] − 3 a ( x ) 2 β u T B u − 3 a ( x ) 2 ( ∥ v ∥ 2 R 2 − β 2 ) u T B y − 3 a ( x ) 2 β [ G ( y ) − y T p ] ( ∥ v ∥ 2 R 2 − β 2 ) . (71) D^{3}\Phi_{U}(x)[v,v,v]=a(x)^{2}D^{3}G(y)[u,u,u]-3a(x)^{2}\beta\,u^{\mathsf{T}}Bu
-3a(x)^{2}\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)u^{\mathsf{T}}By-3a(x)^{2}\beta\left[G(y)-y^{\mathsf{T}}p\right]\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right). \tag*{(71)} D 3 Φ U ( x ) [ v , v , v ] = a ( x ) 2 D 3 G ( y ) [ u , u , u ] − 3 a ( x ) 2 β u T B u − 3 a ( x ) 2 ( R 2 ∥ v ∥ 2 − β 2 ) u T B y − 3 a ( x ) 2 β [ G ( y ) − y T p ] ( R 2 ∥ v ∥ 2 − β 2 ) . (71)
Proof. Fix v ∈ R d v \in\mathbb{R}^{d} v ∈ R d and write α = U T v \alpha=U^{\mathsf{T}}v α = U T v and β = ⟨ z ( x ) / R 2 , v ⟩ \beta=\langle z(x)/R^{2},v\rangle β = ⟨ z ( x ) / R 2 , v ⟩ . Direct differentiation of w ( x ) = ( 1 + ∥ x ∥ 2 / R 2 ) 1 / 2 w(x)=(1+\lVert x\rVert^{2}/R^{2})^{1/2} w ( x ) = ( 1 + ∥ x ∥ 2 / R 2 ) 1/2 and z ( x ) = x / w ( x ) z(x)=x/w(x) z ( x ) = x / w ( x ) gives
D w ( x ) v = x T v R 2 w ( x ) = β , D z ( x ) v = v w ( x ) − x ( D w ( x ) v ) w ( x ) 2 = a ( x ) ( v − β z ( x ) ) , Dw(x)v=\frac{x^{\mathsf{T}}v}{R^{2}w(x)}=\beta,\qquad Dz(x)v=\frac{v}{w(x)}-\frac{x(Dw(x)v)}{w(x)^{2}}=a(x)(v-\beta z(x)), D w ( x ) v = R 2 w ( x ) x T v = β , D z ( x ) v = w ( x ) v − w ( x ) 2 x ( D w ( x ) v ) = a ( x ) ( v − β z ( x )) ,
and hence D y ( x ) v = U T D z ( x ) v = a ( x ) ( α − β y ) Dy(x)v=U^{\mathsf{T}}Dz(x)v=a(x)(\alpha-\beta y) D y ( x ) v = U T D z ( x ) v = a ( x ) ( α − β y ) with u : = α − β y u:=\alpha-\beta y u := α − β y . By the product and chain rules,
D Φ U ( x ) v = ( D w ( x ) v ) G ( y ) + w ( x ) p T D y ( x ) v = β G ( y ) + w ( x ) a ( x ) p T ( α − β y ) = p T α + [ G ( y ) − y T p ] β , D\Phi_{U}(x)v=(Dw(x)v)G(y)+w(x)p^{\mathsf{T}}Dy(x)v=\beta G(y)+w(x)a(x)p^{\mathsf{T}}(\alpha-\beta y)=p^{\mathsf{T}}\alpha+[G(y)-y^{\mathsf{T}}p]\beta, D Φ U ( x ) v = ( D w ( x ) v ) G ( y ) + w ( x ) p T D y ( x ) v = β G ( y ) + w ( x ) a ( x ) p T ( α − β y ) = p T α + [ G ( y ) − y T p ] β ,
since w ( x ) a ( x ) = 1 w(x)a(x)=1 w ( x ) a ( x ) = 1 . Because α = U T v \alpha=U^{\mathsf{T}}v α = U T v and β = ⟨ z ( x ) / R 2 , v ⟩ \beta=\langle z(x)/R^{2},v\rangle β = ⟨ z ( x ) / R 2 , v ⟩ , this is (69) .
Next differentiate D Φ U ( x ) v = p ( y ) T α + [ G ( y ) − y T p ( y ) ] β D\Phi_{U}(x)v=p(y)^{\mathsf{T}}\alpha+[G(y)-y^{\mathsf{T}}p(y)]\beta D Φ U ( x ) v = p ( y ) T α + [ G ( y ) − y T p ( y )] β along the constant vector v v v . Since ∇ y [ G ( y ) − y T p ( y ) ] = p ( y ) − ( p ( y ) + B ( y ) y ) = − B ( y ) y \nabla_{y}[G(y)-y^{\mathsf{T}}p(y)]=p(y)-(p(y)+B(y)y)=-B(y)y ∇ y [ G ( y ) − y T p ( y )] = p ( y ) − ( p ( y ) + B ( y ) y ) = − B ( y ) y , the chain rule yields
D [ p ( y ) T α ] v = a ( x ) α T B ( y ) u , D [ G ( y ) − y T p ( y ) ] v = − a ( x ) y T B ( y ) u , D[p(y)^{\mathsf{T}}\alpha]v=a(x)\alpha^{\mathsf{T}}B(y)u,\qquad D[G(y)-y^{\mathsf{T}}p(y)]v=-a(x)y^{\mathsf{T}}B(y)u, D [ p ( y ) T α ] v = a ( x ) α T B ( y ) u , D [ G ( y ) − y T p ( y )] v = − a ( x ) y T B ( y ) u ,
while D β ( x ) v = v T D z ( x ) v / R 2 = a ( x ) ( ∥ v ∥ 2 / R 2 − β 2 ) D\beta(x)v=v^{\mathsf{T}}Dz(x)v/R^{2}=a(x)(\lVert v\rVert^{2}/R^{2}-\beta^{2}) D β ( x ) v = v T D z ( x ) v / R 2 = a ( x ) (∥ v ∥ 2 / R 2 − β 2 ) . Combining the terms gives
D 2 Φ U ( x ) [ v , v ] = a ( x ) ( α T − β y T ) B ( y ) u + a ( x ) [ G ( y ) − y T p ( y ) ] ( ∥ v ∥ 2 R 2 − β 2 ) , D^{2}\Phi_{U}(x)[v,v]=a(x)(\alpha^{\mathsf{T}}-\beta y^{\mathsf{T}})B(y)u+a(x)[G(y)-y^{\mathsf{T}}p(y)]\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right), D 2 Φ U ( x ) [ v , v ] = a ( x ) ( α T − β y T ) B ( y ) u + a ( x ) [ G ( y ) − y T p ( y )] ( R 2 ∥ v ∥ 2 − β 2 ) ,
which is (70) .
Finally, when G G G is three times differentiable at y y y , differentiate (70) once more along v v v . Using
D a ( x ) v = − a ( x ) 2 D w ( x ) v = − a ( x ) 2 β Da(x)v=-a(x)^{2}Dw(x)v=-a(x)^{2}\beta D a ( x ) v = − a ( x ) 2 D w ( x ) v = − a ( x ) 2 β
and
D u ( x ) v = − ( D β ( x ) v ) y − β ( D y ( x ) v ) = − a ( x ) ( ∥ v ∥ 2 R 2 − β 2 ) y − a ( x ) β u , Du(x)v=-(D\beta(x)v)y-\beta(Dy(x)v)=-a(x)\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)y-a(x)\beta u, D u ( x ) v = − ( D β ( x ) v ) y − β ( D y ( x ) v ) = − a ( x ) ( R 2 ∥ v ∥ 2 − β 2 ) y − a ( x ) β u ,
the derivative of a ( x ) u T B ( y ) u a(x)u^{\mathsf{T}}B(y)u a ( x ) u T B ( y ) u along v v v equals
− a ( x ) 2 β u T B ( y ) u + a ( x ) 2 D 3 G ( y ) [ u , u , u ] − 2 a ( x ) 2 ( ∥ v ∥ 2 R 2 − β 2 ) u T B ( y ) y − 2 a ( x ) 2 β u T B ( y ) u . -a(x)^{2}\beta u^{\mathsf{T}}B(y)u+a(x)^{2}D^{3}G(y)[u,u,u]-2a(x)^{2}\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)u^{\mathsf{T}}B(y)y-2a(x)^{2}\beta u^{\mathsf{T}}B(y)u. − a ( x ) 2 β u T B ( y ) u + a ( x ) 2 D 3 G ( y ) [ u , u , u ] − 2 a ( x ) 2 ( R 2 ∥ v ∥ 2 − β 2 ) u T B ( y ) y − 2 a ( x ) 2 β u T B ( y ) u .
Similarly, since
D [ a ( x ) ( ∥ v ∥ 2 R 2 − β 2 ) ] v = − a ( x ) 2 β ( ∥ v ∥ 2 R 2 − β 2 ) + a ( x ) ( − 2 β a ( x ) ( ∥ v ∥ 2 R 2 − β 2 ) ) = − 3 a ( x ) 2 β ( ∥ v ∥ 2 R 2 − β 2 ) , D\left[a(x)\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)\right]v=-a(x)^{2}\beta\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)+a(x)\left(-2\beta a(x)\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)\right)=-3a(x)^{2}\beta\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right), D [ a ( x ) ( R 2 ∥ v ∥ 2 − β 2 ) ] v = − a ( x ) 2 β ( R 2 ∥ v ∥ 2 − β 2 ) + a ( x ) ( − 2 β a ( x ) ( R 2 ∥ v ∥ 2 − β 2 ) ) = − 3 a ( x ) 2 β ( R 2 ∥ v ∥ 2 − β 2 ) ,
the derivative of a ( x ) [ G ( y ) − y T p ( y ) ] ( ∥ v ∥ 2 / R 2 − β 2 ) a(x)[G(y)-y^{\mathsf{T}}p(y)](\lVert v\rVert^{2}/R^{2}-\beta^{2}) a ( x ) [ G ( y ) − y T p ( y )] (∥ v ∥ 2 / R 2 − β 2 ) along v v v equals
− a ( x ) 2 ( ∥ v ∥ 2 R 2 − β 2 ) u T B ( y ) y − 3 a ( x ) 2 β [ G ( y ) − y T p ( y ) ] ( ∥ v ∥ 2 R 2 − β 2 ) . -a(x)^{2}\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right)u^{\mathsf{T}}B(y)y-3a(x)^{2}\beta[G(y)-y^{\mathsf{T}}p(y)]\left(\frac{\lVert v\rVert^{2}}{R^{2}}-\beta^{2}\right). − a ( x ) 2 ( R 2 ∥ v ∥ 2 − β 2 ) u T B ( y ) y − 3 a ( x ) 2 β [ G ( y ) − y T p ( y )] ( R 2 ∥ v ∥ 2 − β 2 ) .
Adding the two expressions yields (71) .
C∞ global perspective witnesses with a Lipschitz Hessian
Now let g ~ m : R m → R \tilde{g}_m:\mathbb{R}^m \to\mathbb{R} g ~ m : R m → R be the C ∞ C^\infty C ∞ chain of Lemma G.5 formed from the convolved gates φ ~ , ψ ~ \tilde{\varphi},\tilde{\psi} φ ~ , ψ ~ of (64) with η = 1 / 24 \eta=1/24 η = 1/24 , a 0 = 3 / 8 a_0=3/8 a 0 = 3/8 , and a 1 = 23 / 24 a_1=23/24 a 1 = 23/24 . Define
g ^ m ( y ) : = g ~ m ( y ) + 9 4 m + 1 4 , F ~ U g ( x ) : = w ( x ) g ^ m ( U ⊤ ρ R ( x ) ) − 1 4 . (72) \widehat{g}_m(y):=\tilde{g}_m(y)+\frac{9}{4}m+\frac{1}{4},\qquad\widetilde{F}^{g}_{U}(x):=w(x)\widehat{g}_m(U^\top\rho_R(x))-\frac{1}{4}. \tag*{(72)} g m ( y ) := g ~ m ( y ) + 4 9 m + 4 1 , F U g ( x ) := w ( x ) g m ( U ⊤ ρ R ( x )) − 4 1 . (72)
Because w w w and ρ R \rho_R ρ R are C ∞ C^\infty C ∞ on all of R d \mathbb{R}^d R d and g ~ m \tilde{g}_m g ~ m is C ∞ C^\infty C ∞ on R m \mathbb{R}^m R m , F ~ U g \widetilde{F}^{g}_{U} F U g is C ∞ C^\infty C ∞ on R d \mathbb{R}^d R d .
Lemma G.7 (Analytic bounds for the C ∞ C^\infty C ∞ global perspective wrapper). Let m ≥ 1 m\ge1 m ≥ 1 and R 2 ≥ 64 m R^2\ge64m R 2 ≥ 64 m . For every orthonormal m m m -frame U U U in R d \mathbb{R}^d R d , the function F ~ U g \widetilde{F}^{g}_{U} F U g in (72) is C ∞ C^\infty C ∞ , coercive, nonconvex, globally 58 58 58 -smooth, and globally 12 m 12\sqrt{m} 12 m -Lipschitz, with initial gap F ~ U g ( 0 ) − inf x ∈ R d F ~ U g ( x ) ≤ 9 4 ( m − 1 2 ) \widetilde{F}^{g}_{U}(0)-\inf_{x\in\mathbb{R}^d}\widetilde{F}^{g}_{U}(x)\le\frac{9}{4}\left(m-\frac{1}{2}\right) F U g ( 0 ) − inf x ∈ R d F U g ( x ) ≤ 4 9 ( m − 2 1 ) and Lip ( ∇ 2 F ~ U g ) < 2016 \operatorname{Lip}(\nabla^2\widetilde{F}^{g}_{U})<2016 Lip ( ∇ 2 F U g ) < 2016 . At every point x ∈ R d x\in\mathbb{R}^d x ∈ R d , its Hessian satisfies the two-sided global matrix bound
− H U g ⪯ ∇ 2 F ~ U g ( x ) ⪯ H U g : = 56 U U ⊤ + 84 m R 2 I , ∥ H U g ∥ o p < 58 , (73) -H^{g}_{U}\preceq\nabla^2\widetilde{F}^{g}_{U}(x)\preceq H^{g}_{U}:=56UU^\top+\frac{84m}{R^2}I,\qquad\lVert H^{g}_{U}\rVert_{\mathrm{op}}<58, \tag*{(73)} − H U g ⪯ ∇ 2 F U g ( x ) ⪯ H U g := 56 U U ⊤ + R 2 84 m I , ∥ H U g ∥ op < 58 , (73) and hence at every point x ∈ R d x\in\mathbb{R}^d x ∈ R d ,
tr [ ( ∇ 2 F ~ U g ( x ) ) + ] ≤ ∥ ∇ 2 F ~ U g ( x ) ∥ ∗ ≤ tr H U g = 56 m + 84 d m R 2 . (74) \operatorname{tr}\left[(\nabla^2\widetilde{F}^{g}_{U}(x))_+\right]\le\lVert\nabla^2\widetilde{F}^{g}_{U}(x)\rVert_*\le\operatorname{tr}H^{g}_{U}=56m+\frac{84dm}{R^2}. \tag*{(74)} tr [ ( ∇ 2 F U g ( x ) ) + ] ≤ ∥ ∇ 2 F U g ( x ) ∥ ∗ ≤ tr H U g = 56 m + R 2 84 d m . (74) If any coordinate of y = U ⊤ ρ R ( x ) y=U^\top\rho_R(x) y = U ⊤ ρ R ( x ) has magnitude below a 1 = 23 / 24 a_1=23/24 a 1 = 23/24 , then ∥ ∇ F ~ U g ( x ) ∥ > 4 / 5 \lVert\nabla\widetilde{F}^{g}_{U}(x)\rVert>4/5 ∥ ∇ F U g ( x )∥ > 4/5 . The prefix hypotheses of Lemma C.4 hold on all of R d \mathbb{R}^d R d with z = ρ R z=\rho_R z = ρ R , τ = a 0 = 3 / 8 \tau=a_0=3/8 τ = a 0 = 3/8 , and c j = γ j − 1 c_j=\gamma_{j-1} c j = γ j − 1 , where γ 0 = 1 \gamma_0=1 γ 0 = 1 and γ j − 1 ( x ) = ψ ~ ( − y j − 1 ) + ψ ~ ( y j − 1 ) ∈ [ 0 , 1 ] \gamma_{j-1}(x)=\tilde{\psi}(-y_{j-1})+\tilde{\psi}(y_{j-1})\in[0,1] γ j − 1 ( x ) = ψ ~ ( − y j − 1 ) + ψ ~ ( y j − 1 ) ∈ [ 0 , 1 ] for j > 1 j>1 j > 1 .
Proof. In (37) with the gates φ ~ , ψ ~ \tilde{\varphi},\tilde{\psi} φ ~ , ψ ~ , one has 0 ≤ φ ~ ≤ 9 / 4 0\le\tilde{\varphi}\le9/4 0 ≤ φ ~ ≤ 9/4 , 0 ≤ ψ ~ ≤ 1 0\le\tilde{\psi}\le1 0 ≤ ψ ~ ≤ 1 , and at most one of ψ ~ ( − t ) , ψ ~ ( t ) \tilde{\psi}(-t),\tilde{\psi}(t) ψ ~ ( − t ) , ψ ~ ( t ) is nonzero, so each of the m m m summands of g ~ m \tilde{g}_m g ~ m lies in [ − 9 / 4 , 9 / 4 ] [-9/4,9/4] [ − 9/4 , 9/4 ] and the initial summand − φ ~ ( y 1 ) -\tilde{\varphi}(y_1) − φ ~ ( y 1 ) lies in [ − 9 / 4 , 0 ] [-9/4,0] [ − 9/4 , 0 ] . Hence 1 / 4 ≤ g ^ m ( y ) < 9 m / 2 1/4\le\widehat{g}_m(y)<9m/2 1/4 ≤ g m ( y ) < 9 m /2 for all y ∈ R m y\in\mathbb{R}^m y ∈ R m . Since w ( x ) ≥ 1 w(x)\ge1 w ( x ) ≥ 1 , we get F ~ U g ( x ) ≥ ( w ( x ) − 1 ) / 4 ≥ 0 \widetilde{F}^{g}_{U}(x)\ge(w(x)-1)/4\ge0 F U g ( x ) ≥ ( w ( x ) − 1 ) /4 ≥ 0 , which proves coercivity and inf F ~ U g ≥ 0 \inf\widetilde{F}^{g}_{U}\ge0 inf F U g ≥ 0 . Because w ( 0 ) = 1 w(0)=1 w ( 0 ) = 1 and g ~ m ( 0 ) = − φ ~ ( 0 ) = − 9 / 8 \tilde{g}_m(0)=-\tilde{\varphi}(0)=-9/8 g ~ m ( 0 ) = − φ ~ ( 0 ) = − 9/8 , we have F ~ U g ( 0 ) = 9 4 ( m − 1 2 ) \widetilde{F}^{g}_{U}(0)=\frac{9}{4}\left(m-\frac{1}{2}\right) F U g ( 0 ) = 4 9 ( m − 2 1 ) .
Write p = ∇ g ~ m ( y ) p=\nabla\tilde{g}_m(y) p = ∇ g ~ m ( y ) and B = ∇ 2 g ~ m ( y ) B=\nabla^2\tilde{g}_m(y) B = ∇ 2 g ~ m ( y ) . By Lemma G.5 , ∥ p ∥ ≤ 10 m \lVert p\rVert\le10\sqrt{m} ∥ p ∥ ≤ 10 m , ∥ B ∥ o p ≤ 42 \lVert B\rVert_{\mathrm{op}}\le42 ∥ B ∥ op ≤ 42 , ∥ D 3 g ~ m ∥ o p ≤ 1872 \lVert D^3\tilde{g}_m\rVert_{\mathrm{op}}\le1872 ∥ D 3 g ~ m ∥ op ≤ 1872 , ∥ B y ∥ ≤ 44 m \lVert By\rVert\le44\sqrt{m} ∥ B y ∥ ≤ 44 m , ∣ y ⊤ B y ∣ ≤ 44 m \lvert y^\top By\rvert\le44m ∣ y ⊤ B y ∣ ≤ 44 m , and ∣ y ⊤ p ∣ ≤ 8 m \lvert y^\top p\rvert\le8m ∣ y ⊤ p ∣ ≤ 8 m . Therefore
∣ g ^ m ( y ) − y ⊤ p ∣ ≤ ( 9 2 + 8 ) m = 25 2 m . \left\lvert\widehat{g}_m(y)-y^\top p\right\rvert\le\left(\frac{9}{2}+8\right)m=\frac{25}{2}m. g m ( y ) − y ⊤ p ≤ ( 2 9 + 8 ) m = 2 25 m .
For R ≥ 8 m ≥ 8 R\ge8\sqrt{m}\ge8 R ≥ 8 m ≥ 8 , identity (69) gives
∥ ∇ F ~ U g ( x ) ∥ ≤ 10 m + 25 m 2 R ≤ ( 10 + 25 16 ) m < 12 m . \left\lVert\nabla\widetilde{F}^{g}_{U}(x)\right\rVert\le10\sqrt{m}+\frac{25m}{2R}\le\left(10+\frac{25}{16}\right)\sqrt{m}<12\sqrt{m}. ∇ F U g ( x ) ≤ 10 m + 2 R 25 m ≤ ( 10 + 16 25 ) m < 12 m .
For any v ∈ R d v\in\mathbb{R}^d v ∈ R d , put α = U ⊤ v \alpha=U^\top v α = U ⊤ v and β = z ( x ) ⊤ v / R 2 \beta=z(x)^\top v/R^2 β = z ( x ) ⊤ v / R 2 . Since ∥ z ( x ) ∥ 2 / R 2 = 1 − a ( x ) 2 \lVert z(x)\rVert^2/R^2=1-a(x)^2 ∥ z ( x ) ∥ 2 / R 2 = 1 − a ( x ) 2 , Cauchy–Schwarz gives β 2 ≤ ( 1 − a ( x ) 2 ) ∥ v ∥ 2 / R 2 \beta^2\le(1-a(x)^2)\lVert v\rVert^2/R^2 β 2 ≤ ( 1 − a ( x ) 2 ) ∥ v ∥ 2 / R 2 , and hence
a ( x ) β 2 ≤ a ( x ) ( 1 − a ( x ) 2 ) ∥ v ∥ 2 R 2 ≤ 2 3 3 ∥ v ∥ 2 R 2 < 7 18 ∥ v ∥ 2 R 2 . (75) a(x)\beta^2\le a(x)(1-a(x)^2)\frac{\lVert v\rVert^2}{R^2}\le\frac{2}{3\sqrt{3}}\frac{\lVert v\rVert^2}{R^2}<\frac{7}{18}\frac{\lVert v\rVert^2}{R^2}. \tag*{(75)} a ( x ) β 2 ≤ a ( x ) ( 1 − a ( x ) 2 ) R 2 ∥ v ∥ 2 ≤ 3 3 2 R 2 ∥ v ∥ 2 < 18 7 R 2 ∥ v ∥ 2 . (75)
By the bounds on B B B above and Young’s inequality with parameter γ = 7 / 22 \gamma=7/22 γ = 7/22 ,
a ( x ) ∣ ( α − β y ) ⊤ B ( α − β y ) ∣ ≤ ( 42 + 44 γ ) ∥ α ∥ 2 + ( 44 γ + 44 ) m a ( x ) β 2 ≤ 56 ∥ U ⊤ v ∥ 2 + 638 m 9 R 2 ∥ v ∥ 2 . a(x)\left\lvert(\alpha-\beta y)^\top B(\alpha-\beta y)\right\rvert\le(42+44\gamma)\lVert\alpha\rVert^2+\left(\frac{44}{\gamma}+44\right)ma(x)\beta^2\le56\lVert U^\top v\rVert^2+\frac{638m}{9R^2}\lVert v\rVert^2. a ( x ) ( α − β y ) ⊤ B ( α − β y ) ≤ ( 42 + 44 γ ) ∥ α ∥ 2 + ( γ 44 + 44 ) ma ( x ) β 2 ≤ 56 ∥ U ⊤ v ∥ 2 + 9 R 2 638 m ∥ v ∥ 2 .
Adding the second term of (70) , which has magnitude at most 25 m 2 R 2 ∥ v ∥ 2 \frac{25m}{2R^{2}}\|v\|^{2} 2 R 2 25 m ∥ v ∥ 2 , and noting that 638 / 9 + 25 / 2 = 1501 / 18 < 84 638/9+25/2=1501/18<84 638/9 + 25/2 = 1501/18 < 84 , we obtain (73) and (74) at every x ∈ R d x\in\mathbb{R}^{d} x ∈ R d . Since R 2 ≥ 64 m R^{2}\ge64m R 2 ≥ 64 m , ∥ H U G ~ ∥ o p ≤ 56 + 84 / 64 < 58 \|H^{\widetilde{G}}_{U}\|_{\mathrm{op}}\le56+84/64<58 ∥ H U G ∥ op ≤ 56 + 84/64 < 58 .
If some ∣ y i ∣ < a 1 |y_i|<a_1 ∣ y i ∣ < a 1 , the smallest such index j j j satisfies p j ≤ − 1 p_j\le-1 p j ≤ − 1 by Lemma G.5 . Projecting (69) onto u j u_j u j gives
u j ⊤ ∇ F ~ U G ~ ( x ) = p j + [ g ~ m ( y ) − y ⊤ p ] y j R 2 ≤ − 1 + 25 m 2 R 2 ≤ − 1 + 25 128 = − 103 128 < − 4 5 . u_j^{\top}\nabla\widetilde{F}^{\widetilde{G}}_{U}(x)=p_j+\left[\widetilde{g}_m(y)-y^{\top}p\right]\frac{y_j}{R^2}\le-1+\frac{25m}{2R^2}\le-1+\frac{25}{128}=-\frac{103}{128}<-\frac{4}{5}. u j ⊤ ∇ F U G ( x ) = p j + [ g m ( y ) − y ⊤ p ] R 2 y j ≤ − 1 + 2 R 2 25 m ≤ − 1 + 128 25 = − 128 103 < − 5 4 .
For nonconvexity, choose x = t u 1 x=tu_1 x = t u 1 (t < 0 t<0 t < 0 ) so that y 1 = − 9 / 8 y_1=-9/8 y 1 = − 9/8 and y 2 = ⋯ = y m = 0 y_2=\cdots=y_m=0 y 2 = ⋯ = y m = 0 . In a neighborhood of this point, ψ ~ ( − y 1 ) = 1 \widetilde{\psi}(-y_1)=1 ψ ( − y 1 ) = 1 and ψ ~ ( y 1 ) = 0 \widetilde{\psi}(y_1)=0 ψ ( y 1 ) = 0 are constant, because 9 / 8 > a 1 9/8>a_1 9/8 > a 1 and − 9 / 8 < a 0 -9/8<a_0 − 9/8 < a 0 , so B 11 = − φ ~ ′ ′ ( − 9 / 8 ) = − 4 B_{11}=-\widetilde{\varphi}^{\prime\prime}(-9/8)=-4 B 11 = − φ ′′ ( − 9/8 ) = − 4 ; indeed w η w_\eta w η is supported in [ − η , η ] [-\eta,\eta] [ − η , η ] and φ ′ ′ = 4 \varphi^{\prime\prime}=4 φ ′′ = 4 on [ − 5 / 4 , − 1 ] [-5/4,-1] [ − 5/4 , − 1 ] , which contains [ − 9 / 8 − η , − 9 / 8 + η ] [-9/8-\eta,-9/8+\eta] [ − 9/8 − η , − 9/8 + η ] . Since R ≥ 8 R\ge8 R ≥ 8 , 1 − y 1 2 / R 2 ≥ 1 − 81 / 4096 > 31 / 32 1-y_1^2/R^2\ge1-81/4096>31/32 1 − y 1 2 / R 2 ≥ 1 − 81/4096 > 31/32 and a ( x ) = 1 − y 1 2 / R 2 > 31 / 32 a(x)=\sqrt{1-y_1^2/R^2}>31/32 a ( x ) = 1 − y 1 2 / R 2 > 31/32 . Applying (70) with v = u 1 v=u_1 v = u 1 (α = e 1 \alpha=e_1 α = e 1 , β = y 1 / R 2 \beta=y_1/R^2 β = y 1 / R 2 ) gives
u 1 ⊤ ∇ 2 F ~ U G ~ ( x ) u 1 = − 4 a ( x ) ( 1 − y 1 2 R 2 ) 2 + a ( x ) [ g ~ m ( y ) − y 1 p 1 ] 1 − y 1 2 / R 2 R 2 ≤ − 4 ( 31 32 ) 5 + 25 128 < − 3 < 0. u_1^{\top}\nabla^2\widetilde{F}^{\widetilde{G}}_{U}(x)u_1=-4a(x)\left(1-\frac{y_1^2}{R^2}\right)^2+a(x)\left[\widetilde{g}_m(y)-y_1p_1\right]\frac{1-y_1^2/R^2}{R^2}\le-4\left(\frac{31}{32}\right)^5+\frac{25}{128}<-3<0. u 1 ⊤ ∇ 2 F U G ( x ) u 1 = − 4 a ( x ) ( 1 − R 2 y 1 2 ) 2 + a ( x ) [ g m ( y ) − y 1 p 1 ] R 2 1 − y 1 2 / R 2 ≤ − 4 ( 32 31 ) 5 + 128 25 < − 3 < 0.
For the third derivative, fix a unit vector v ∈ R d v\in\mathbb{R}^{d} v ∈ R d and write u = α − β y u=\alpha-\beta y u = α − β y and s = ∥ z ( x ) ∥ / R = 1 − a ( x ) 2 ∈ [ 0 , 1 ] s=\|z(x)\|/R=\sqrt{1-a(x)^2}\in[0,1] s = ∥ z ( x ) ∥/ R = 1 − a ( x ) 2 ∈ [ 0 , 1 ] . Let I ( y ) = { i : ∣ y i ∣ ≤ 4 / 3 } I(y)=\{i:|y_i|\le4/3\} I ( y ) = { i : ∣ y i ∣ ≤ 4/3 } be the active coordinate set of Lemma G.5 , and let u I = P I ( y ) u u_I=P_{I(y)}u u I = P I ( y ) u . Since ∥ P I ( y ) y ∥ ≤ ( 4 / 3 ) m \|P_{I(y)}y\|\le(4/3)\sqrt{m} ∥ P I ( y ) y ∥ ≤ ( 4/3 ) m and ∣ β ∣ ≤ s / R ≤ s / ( 8 m ) |\beta|\le s/R\le s/(8\sqrt{m}) ∣ β ∣ ≤ s / R ≤ s / ( 8 m ) , we have ∥ u I ∥ ≤ ∥ α ∥ + ∣ β ∣ ∥ P I ( y ) y ∥ ≤ 1 + s / 6 \|u_I\|\le\|\alpha\|+|\beta|\|P_{I(y)}y\|\le1+s/6 ∥ u I ∥ ≤ ∥ α ∥ + ∣ β ∣∥ P I ( y ) y ∥ ≤ 1 + s /6 , and hence
a ( x ) 2 ∥ u I ∥ 3 ≤ ( 1 − s 2 ) ( 1 + s 6 ) 3 < 17 16 . a(x)^2\|u_I\|^3\le(1-s^2)\left(1+\frac{s}{6}\right)^3<\frac{17}{16}. a ( x ) 2 ∥ u I ∥ 3 ≤ ( 1 − s 2 ) ( 1 + 6 s ) 3 < 16 17 .
because on [ 0 , 1 ] [0,1] [ 0 , 1 ] the middle expression increases while 5 s 2 + 12 s < 3 5s^2+12s<3 5 s 2 + 12 s < 3 and decreases afterwards, and the root of 5 s 2 + 12 s = 3 5s^2+12s=3 5 s 2 + 12 s = 3 lies in [ 0.228 , 0.229 ] [0.228,0.229] [ 0.228 , 0.229 ] , where the expression is at most ( 1 − 0.228 2 ) ( 1 + 0.229 / 6 ) 3 < 1.061 (1-0.228^2)(1+0.229/6)^3<1.061 ( 1 − 0.22 8 2 ) ( 1 + 0.229/6 ) 3 < 1.061 . Because B B B and D 3 g ~ m ( y ) D^3\widetilde{g}_m(y) D 3 g m ( y ) are supported on I ( y ) I(y) I ( y ) , identity (71) gives
∣ D 3 F ~ U G ~ ( x ) [ v , v , v ] ∣ ≤ 1872 ⋅ 17 16 + 3 R ( 56 + 638 m 9 R 2 ) + 3 R 2 ⋅ 7 6 ⋅ 44 m + 3 R 25 m 2 R 2 < 2016 , \left|D^3\widetilde{F}^{\widetilde{G}}_{U}(x)[v,v,v]\right|\le1872\cdot\frac{17}{16}+\frac{3}{R}\left(56+\frac{638m}{9R^2}\right)+\frac{3}{R^2}\cdot\frac{7}{6}\cdot44\sqrt{m}+\frac{3}{R}\frac{25m}{2R^2}<2016, D 3 F U G ( x ) [ v , v , v ] ≤ 1872 ⋅ 16 17 + R 3 ( 56 + 9 R 2 638 m ) + R 2 3 ⋅ 6 7 ⋅ 44 m + R 3 2 R 2 25 m < 2016 ,
using R ≥ 8 m ≥ 8 R\ge8\sqrt{m}\ge8 R ≥ 8 m ≥ 8 . Since D 3 F ~ U G ~ ( x ) D^3\widetilde{F}^{\widetilde{G}}_{U}(x) D 3 F U G ( x ) is a symmetric trilinear form on R d \mathbb{R}^{d} R d , its operator norm equals sup ∥ v ∥ = 1 ∣ D 3 F ~ U G ~ ( x ) [ v , v , v ] ∣ \sup_{\|v\|=1}|D^3\widetilde{F}^{\widetilde{G}}_{U}(x)[v,v,v]| sup ∥ v ∥ = 1 ∣ D 3 F U G ( x ) [ v , v , v ] ∣ ([6 ] ), so ∥ D 3 F ~ U G ~ ( x ) ∥ o p < 2016 \|D^3\widetilde{F}^{\widetilde{G}}_{U}(x)\|_{\mathrm{op}}<2016 ∥ D 3 F U G ( x ) ∥ op < 2016 , which integrates to Lip ( ∇ 2 F ~ U G ~ ) < 2016 \operatorname{Lip}(\nabla^2\widetilde{F}^{\widetilde{G}}_{U})<2016 Lip ( ∇ 2 F U G ) < 2016 .
Finally, suppose y j = 0 y_j=0 y j = 0 and ∣ y k ∣ ≤ a 0 |y_k|\le a_0 ∣ y k ∣ ≤ a 0 for all k > j k>j k > j , and put
r j ( x ) = w ( x ) g ~ m ( y 1 , … , y j − 1 , 0 , … , 0 ) − 1 / 4. r_j(x)=w(x)\widetilde{g}_m(y_1,\ldots,y_{j-1},0,\ldots,0)-1/4. r j ( x ) = w ( x ) g m ( y 1 , … , y j − 1 , 0 , … , 0 ) − 1/4.
The gates ψ ~ ( ± y k ) \widetilde{\psi}(\pm y_k) ψ ( ± y k ) vanish with their derivatives for all k ≥ j k\ge j k ≥ j , so g ~ m ( y ) = g ~ m ( y 1 , … , y j − 1 , 0 , … , 0 ) \widetilde{g}_m(y)=\widetilde{g}_m(y_1,\ldots,y_{j-1},0,\ldots,0) g m ( y ) = g m ( y 1 , … , y j − 1 , 0 , … , 0 ) , F ~ U G ~ ( x ) = r j ( x ) \widetilde{F}^{\widetilde{G}}_{U}(x)=r_j(x) F U G ( x ) = r j ( x ) , p k = 0 p_k=0 p k = 0 for k > j k>j k > j , and p j = − γ j − 1 ( x ) p_j=-\gamma_{j-1}(x) p j = − γ j − 1 ( x ) . Because y j = 0 y_j=0 y j = 0 , the product y j p j y_jp_j y j p j vanishes, so y ⊤ p y^{\top}p y ⊤ p agrees with the corresponding inner product for ( y 1 , … , y j − 1 , 0 , … , 0 ) (y_1,\ldots,y_{j-1},0,\ldots,0) ( y 1 , … , y j − 1 , 0 , … , 0 ) . Subtracting (69) for r j r_j r j from (69) for F ~ U G ~ \widetilde{F}^{\widetilde{G}}_{U} F U G cancels the z ( x ) / R 2 z(x)/R^2 z ( x ) / R 2 term and leaves ∇ F ~ U G ~ ( x ) − ∇ r j ( x ) = − γ j − 1 ( x ) u j \nabla\widetilde{F}^{\widetilde{G}}_{U}(x)-\nabla r_j(x)=-\gamma_{j-1}(x)u_j ∇ F U G ( x ) − ∇ r j ( x ) = − γ j − 1 ( x ) u j . Both r j r_j r j and γ j − 1 \gamma_{j-1} γ j − 1 depend only on u 1 , … , u j − 1 u_1,\ldots,u_{j-1} u 1 , … , u j − 1 , and y j = 0 < a 1 y_j=0<a_1 y j = 0 < a 1 gives ∥ ∇ F ~ U G ~ ( x ) ∥ > 4 / 5 \|\nabla\widetilde{F}^{\widetilde{G}}_{U}(x)\|>4/5 ∥∇ F U G ( x ) ∥ > 4/5 by the barrier above, so the hypotheses of Lemma C.4 hold with z = ρ R z=\rho R z = ρR and τ = a 0 \tau=a_0 τ = a 0 .
Theorem G.8 (C ∞ C^\infty C ∞ multistage lower bound on the global fixed-matrix class). Let m ≥ 2 m\ge2 m ≥ 2 satisfy
204 m ≤ T , 28560 m 2 log ( e + m ) ≤ E T , 6144 m log ( e + m ) ≤ d . (76) 204m\le T,\qquad28560m^2\log(e+m)\le ET,\qquad6144m\log(e+m)\le d. \tag*{(76)} 204 m ≤ T , 28560 m 2 log ( e + m ) ≤ E T , 6144 m log ( e + m ) ≤ d . (76)
Then, for either oracle, there are C ∞ C^\infty C ∞ , globally L L L -smooth, coercive, nonconvex witnesses in the global fixed-matrix class F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d (and hence in P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d , B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d , and A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d ) giving N d e t v a l ≥ N d e t l i n ≥ ⌊ d / 2 ⌋ m N_{\mathrm{det}}^{\mathrm{val}} \ge N_{\mathrm{det}}^{\mathrm{lin}} \ge\lfloor d/2\rfloor m N det val ≥ N det lin ≥ ⌊ d /2 ⌋ m . The witnesses are globally 2 L Δ 2\sqrt{L\Delta} 2 L Δ -Lipschitz, satisfy the global nuclear norm bound ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E \lVert\nabla^2 f(x)\rVert_* \le LE ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E at every x ∈ R d x \in\mathbb{R}^d x ∈ R d , and have
Lip ( ∇ 2 f ) < 1 2 ( 204 m T ) 2 L 2 ε . (77) \operatorname{Lip}(\nabla^2 f) < \frac{1}{2}\left(\frac{204m}{T}\right)^2\frac{L^2}{\varepsilon}. \tag*{(77)} Lip ( ∇ 2 f ) < 2 1 ( T 204 m ) 2 ε L 2 . (77)
Proof. Set ℓ = log ( e + m ) \ell= \log(e+m) ℓ = log ( e + m ) , R 2 = d / ( 96 ℓ ) R^2 = d/(96\ell) R 2 = d / ( 96 ℓ ) , L ′ = 204 m L / T L' = 204mL/T L ′ = 204 m L / T , and E ′ = 140 m ℓ E' = 140m\ell E ′ = 140 m ℓ . Conditions (76) imply R 2 ≥ 64 m R^2 \ge64m R 2 ≥ 64 m , m ≤ d / 128 m \le d/128 m ≤ d /128 , L ′ ≤ L L' \le L L ′ ≤ L , and L ′ E ′ = 28560 m 2 ℓ L / T ≤ L E L'E' = 28560m^2\ell L/T \le LE L ′ E ′ = 28560 m 2 ℓ L / T ≤ L E . By (73) , since ℓ ≥ log ( e + 2 ) > 3 / 2 \ell\ge\log(e+2) > 3/2 ℓ ≥ log ( e + 2 ) > 3/2 ,
tr H U g 58 = 56 m + 84 ⋅ 96 m ℓ 58 < 12152 87 m ℓ < 140 m ℓ = E ′ . \frac{\operatorname{tr}H_U^g}{58}=\frac{56m+84\cdot96m\ell}{58}<\frac{12152}{87}m\ell<140m\ell=E'. 58 tr H U g = 58 56 m + 84 ⋅ 96 m ℓ < 87 12152 m ℓ < 140 m ℓ = E ′ .
Put h = ⌊ d / 2 ⌋ h=\lfloor d/2\rfloor h = ⌊ d /2 ⌋ . Then p = d − m − h + 1 ≥ 63 d / 128 p=d-m-h+1\ge63d/128 p = d − m − h + 1 ≥ 63 d /128 and M / p ≤ 64 m / 63 M/p\le64m/63 M / p ≤ 64 m /63 for M ≤ h m M\le hm M ≤ hm . Since log ( e + 64 m / 63 ) ≤ ( 64 / 63 ) ℓ \log(e+64m/63)\le(64/63)\ell log ( e + 64 m /63 ) ≤ ( 64/63 ) ℓ , Lemma G.4 gives a squared projection bound at most ( 144 / 25 ) ( 128 / 63 ) ( 64 / 63 ) / 96 < ( 3 / 8 ) 2 = a 0 2 (144/25)(128/63)(64/63)/96<(3/8)^2=a_0^2 ( 144/25 ) ( 128/63 ) ( 64/63 ) /96 < ( 3/8 ) 2 = a 0 2 . Lemma C.4 with z = ρ R z=\rho_R z = ρ R , τ = a 0 \tau=a_0 τ = a 0 , b = 4 / 5 b=4/5 b = 4/5 , and Q = h m Q=hm Q = hm therefore applies to F ~ U g \widetilde{F}_U^g F U g .
Scale F ~ U g \widetilde{F}_U^g F U g via Lemma E.1 with Λ = 58 \Lambda=58 Λ = 58 and b = 4 / 5 b=4/5 b = 4/5 :
σ = 58 ε L ′ b , α = 58 ε 2 L ′ b 2 , f ( x ) = α F ~ U g ( x / σ ) . \sigma=\frac{58\varepsilon}{L'b},\qquad\alpha=\frac{58\varepsilon^2}{L'b^2},\qquad f(x)=\alpha\widetilde{F}_U^g(x/\sigma). σ = L ′ b 58 ε , α = L ′ b 2 58 ε 2 , f ( x ) = α F U g ( x / σ ) .
Since 58 ( 9 / 4 ) / b 2 = 6525 / 32 < 204 58(9/4)/b^2=6525/32<204 58 ( 9/4 ) / b 2 = 6525/32 < 204 , the initial gap is at most 204 ( m − 1 / 2 ) ε 2 / L ′ ≤ Δ 204(m-1/2)\varepsilon^2/L'\le\Delta 204 ( m − 1/2 ) ε 2 / L ′ ≤ Δ . The global upper matrix ( L ′ / 58 ) H U g (L'/58)H_U^g ( L ′ /58 ) H U g has operator norm at most L ′ ≤ L L'\le L L ′ ≤ L and trace at most L ′ E ′ ≤ L E L'E'\le LE L ′ E ′ ≤ L E on all of R d \mathbb{R}^d R d , and (74) gives ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E \lVert\nabla^2 f(x)\rVert_* \le LE ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E for all x ∈ R d x\in\mathbb{R}^d x ∈ R d . Thus f ∈ F L , Δ , E d f\in\mathcal{F}^{d}_{L,\Delta,E} f ∈ F L , Δ , E d , and at the algorithm’s output ∥ ∇ f ( x ^ ) ∥ > ε \lVert\nabla f(\widehat{x})\rVert>\varepsilon ∥ ∇ f ( x )∥ > ε . The Hessian Lipschitz constant of f f f is at most
α σ 3 2016 = 2016 b ( L ′ ) 2 58 2 ε = 2016 4205 ( 204 m T ) 2 L 2 ε < 1 2 ( 204 m T ) 2 L 2 ε , \frac{\alpha}{\sigma^3}2016=\frac{2016b(L')^2}{58^2\varepsilon}=\frac{2016}{4205}\left(\frac{204m}{T}\right)^2\frac{L^2}{\varepsilon}<\frac{1}{2}\left(\frac{204m}{T}\right)^2\frac{L^2}{\varepsilon}, σ 3 α 2016 = 5 8 2 ε 2016 b ( L ′ ) 2 = 4205 2016 ( T 204 m ) 2 ε L 2 < 2 1 ( T 204 m ) 2 ε L 2 ,
which proves (77) . Finally, ∥ ∇ f ( x ) ∥ ≤ ( 12 / b ) m ε = 15 m ε ≤ 2 L Δ \lVert\nabla f(x)\rVert\le(12/b)\sqrt{m}\varepsilon=15\sqrt{m}\varepsilon\le2\sqrt{L\Delta} ∥ ∇ f ( x )∥ ≤ ( 12/ b ) m ε = 15 m ε ≤ 2 L Δ because T ≥ 204 m T\ge204m T ≥ 204 m . □
Corollary G.9 (Rate on the global class with a prescribed Hessian Lipschitz constant). For any ρ ≥ 0 \rho\ge0 ρ ≥ 0 , on the C ∞ C^\infty C ∞ members of F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d (and hence of P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d and B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d ) satisfying ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E \lVert\nabla^2f(x)\rVert_*\le LE ∥ ∇ 2 f ( x ) ∥ ∗ ≤ L E everywhere on R d \mathbb{R}^d R d and Lip ( ∇ 2 f ) ≤ ρ \operatorname{Lip}(\nabla^2f)\le\rho Lip ( ∇ 2 f ) ≤ ρ , if E T ≤ d ET\le d E T ≤ d , then for either oracle,
N d e t v a l ≥ N d e t l i n ≥ max { d , d 612 min { T , T 2 ρ ε L , E T log ( e + T ) } } . (78) N_{\mathrm{det}}^{\mathrm{val}}\ge N_{\mathrm{det}}^{\mathrm{lin}}\ge\max\left\{d,\frac{d}{612}\min\left\{T,\frac{T\sqrt{2\rho\varepsilon}}{L},\sqrt{\frac{ET}{\log(e+T)}}\right\}\right\}. \tag*{(78)} N det val ≥ N det lin ≥ max { d , 612 d min { T , L T 2 ρε , log ( e + T ) E T } } . (78) In particular, if ρ ≥ L 2 / ( 2 ε ) \rho\ge L^2/(2\varepsilon) ρ ≥ L 2 / ( 2 ε ) , the middle term in the minimum is redundant and
N d e t ( F L , Δ , E d ) ≥ max { d , d 612 min { T , E T log ( e + T ) } } ≥ d 672 min { T , E T log ( e + T ) } . N_{\mathrm{det}}(\mathcal{F}^{d}_{L,\Delta,E})\ge\max\left\{d,\frac{d}{612}\min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\}\right\}\ge\frac{d}{672}\min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\}. N det ( F L , Δ , E d ) ≥ max { d , 612 d min { T , log ( e + T ) E T } } ≥ 672 d min { T , log ( e + T ) E T } .
Proof. Let M M M denote the minimum of three terms in (78) , put z = M / 204 z=M/204 z = M /204 , and set m = ⌊ z ⌋ m=\lfloor z\rfloor m = ⌊ z ⌋ . When z < 3 z<3 z < 3 , the rank-one quadratic dimension floor N d e t l i n ( F L , Δ , E d ) ≥ d N_{\mathrm{det}}^{\mathrm{lin}}(\mathcal{F}^{d}_{L,\Delta,E})\ge d N det lin ( F L , Δ , E d ) ≥ d of Lemma E.2 (whose witness is C ∞ C^\infty C ∞ with constant Hessian L u u ⊤ Luu^\top Lu u ⊤ , so Lip ( ∇ 2 f ) = 0 \operatorname{Lip}(\nabla^2f)=0 Lip ( ∇ 2 f ) = 0 and ∥ ∇ 2 f ∥ ∗ = L ≤ L E \lVert\nabla^2f\rVert_*=L\le LE ∥ ∇ 2 f ∥ ∗ = L ≤ L E ) exceeds d z / 3 = d M / 612 dz/3=dM/612 d z /3 = d M /612 . When z ≥ 3 z\ge3 z ≥ 3 , we have m ≥ 3 m\ge3 m ≥ 3 , 204 m ≤ M ≤ T 204m\le M\le T 204 m ≤ M ≤ T , 28560 m 2 log ( e + m ) ≤ 204 2 m 2 log ( e + T ) ≤ E T 28560m^2\log(e+m)\le204^2m^2\log(e+T)\le ET 28560 m 2 log ( e + m ) ≤ 20 4 2 m 2 log ( e + T ) ≤ E T , and 6144 m log ( e + m ) < 2 ⋅ 204 2 m log ( e + m ) ≤ m ⋅ 204 2 ⋅ m log ( e + T ) ≤ E T ≤ d 6144m\log(e+m)<2\cdot204^2m\log(e+m)\le m\cdot204^2\cdot m\log(e+T)\le ET\le d 6144 m log ( e + m ) < 2 ⋅ 20 4 2 m log ( e + m ) ≤ m ⋅ 20 4 2 ⋅ m log ( e + T ) ≤ E T ≤ d . Moreover, ( 204 m / T ) L ≤ 2 ρ ε (204m/T)L\le\sqrt{2\rho\varepsilon} ( 204 m / T ) L ≤ 2 ρε , so (77) gives Lip ( ∇ 2 f ) < ρ \operatorname{Lip}(\nabla^2f)<\rho Lip ( ∇ 2 f ) < ρ . Theorem G.8 yields N d e t l i n ≥ ⌊ d / 2 ⌋ m N_{\mathrm{det}}^{\mathrm{lin}}\ge\lfloor d/2\rfloor m N det lin ≥ ⌊ d /2 ⌋ m . Since m / z ≥ 3 / 4 m/z\ge3/4 m / z ≥ 3/4 and ⌊ d / 2 ⌋ / d ≥ 15 / 32 \lfloor d/2\rfloor/d\ge15/32 ⌊ d /2 ⌋ / d ≥ 15/32 for d ≥ 6144 d\ge6144 d ≥ 6144 , we have ⌊ d / 2 ⌋ m ≥ ( 45 / 128 ) d z > d z / 3 = d M / 612 \lfloor d/2\rfloor m\ge(45/128)dz>dz/3=dM/612 ⌊ d /2 ⌋ m ≥ ( 45/128 ) d z > d z /3 = d M /612 . □
Corollary G.10 (The Ω ( d T ) \Omega(dT) Ω ( d T ) threshold on the global classes). On the C ∞ C^\infty C ∞ subclass of Corollary G.9 with ρ ≥ L 2 / ( 2 ε ) \rho\ge L^2/(2\varepsilon) ρ ≥ L 2 / ( 2 ε ) , and hence on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d and P L , Δ , E d \mathcal{P}^{d}_{L,\Delta,E} P L , Δ , E d , whenever E ≥ T log ( e + T ) E \ge T\log(e+T) E ≥ T log ( e + T ) and E ≤ d E \le d E ≤ d (without requiring E T ≤ d ET \le d E T ≤ d ), both deterministic oracle costs satisfy
N d e t v a l ≥ N d e t l i n ≥ d T 18432 . (79) N_{\mathrm{det}}^{\mathrm{val}} \ge N_{\mathrm{det}}^{\mathrm{lin}} \ge\frac{dT}{18432}. \tag*{(79)} N det val ≥ N det lin ≥ 18432 d T . (79) If in addition T log ( e + T ) ≤ E ≤ d / T T\log(e+T) \le E \le d/T T log ( e + T ) ≤ E ≤ d / T , then on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d ,
d T 612 ≤ N d e t ≤ 10 d T , E T 12 ≤ N r a n d c a p ≤ 23 E T , (80) \frac{dT}{612} \le N_{\mathrm{det}} \le10dT,\qquad\frac{ET}{12} \le N_{\mathrm{rand}}^{\mathrm{cap}} \le23ET, \tag*{(80)} 612 d T ≤ N det ≤ 10 d T , 12 E T ≤ N rand cap ≤ 23 E T , (80) and the cost ratio for the same oracle N d e t / N r a n d c a p N_{\mathrm{det}}/N_{\mathrm{rand}}^{\mathrm{cap}} N det / N rand cap lies between d / ( 14076 E ) d/(14076E) d / ( 14076 E ) and 120 d / E 120d/E 120 d / E .
Proof. For (79) , set z = T / 6144 z=T/6144 z = T /6144 and m = ⌊ z ⌋ m=\lfloor z\rfloor m = ⌊ z ⌋ . If z < 3 z<3 z < 3 , Lemma E.2 gives N d e t ≥ d > d z / 3 = d T / 18432 N_{\mathrm{det}}\ge d>dz/3=dT/18432 N det ≥ d > d z /3 = d T /18432 . If z ≥ 3 z\ge3 z ≥ 3 , then m ≥ 3 m\ge3 m ≥ 3 , 204 m ≤ T 204m\le T 204 m ≤ T , 28560 m 2 log ( e + m ) ≤ T 2 log ( e + T ) ≤ E T 28560m^2\log(e+m)\le T^2\log(e+T)\le ET 28560 m 2 log ( e + m ) ≤ T 2 log ( e + T ) ≤ E T , and 6144 m log ( e + m ) ≤ T log ( e + T ) ≤ E ≤ d 6144m\log(e+m)\le T\log(e+T)\le E\le d 6144 m log ( e + m ) ≤ T log ( e + T ) ≤ E ≤ d , so Theorem G.8 gives N d e t ≥ ⌊ d / 2 ⌋ m ≥ d z / 3 = d T / 18432 N_{\mathrm{det}}\ge\lfloor d/2\rfloor m\ge dz/3=dT/18432 N det ≥ ⌊ d /2 ⌋ m ≥ d z /3 = d T /18432 . When T log ( e + T ) ≤ E ≤ d / T T\log(e+T)\le E\le d/T T log ( e + T ) ≤ E ≤ d / T , we also have E T ≤ d ET\le d E T ≤ d and min { T , E T / log ( e + T ) } = T \min\{T,\sqrt{ET/\log(e+T)}\}=T min { T , E T / log ( e + T ) } = T , so Corollary G.9 gives N d e t ≥ d T / 612 N_{\mathrm{det}}\ge dT/612 N det ≥ d T /612 , while coordinate differences, Theorem B.8 , and Corollary F.4 give the remaining bounds in (80) .
Further Results for Randomized Methods
This appendix shows that the hard functions of Theorem E.8 do not extend the deterministic bound Ω ( d T ) \Omega(dT) Ω ( d T ) to randomized methods (Appendix § 17 .1), and combines our bounds with the lower bound of Xia and Yuan [55 ] into a wider window with matching bounds for exact values (Appendix § 17 .2).
The unrestricted hard functions are easy for randomized methods
Theorem E.8 uses the functions F ~ m , U ( x ) = g m ( U ⊤ ρ R ( x ) ) + q ( x ) \widetilde{F}_{m,U}(x)=g_m(U^\top\rho_R(x))+q(x) F m , U ( x ) = g m ( U ⊤ ρ R ( x )) + q ( x ) with R 2 = 5 m R^2=5m R 2 = 5 m , where ρ R ( x ) = x / 1 + ∥ x ∥ 2 / R 2 \rho_R(x)=x/\sqrt{1+\lVert x\rVert^2/R^2} ρ R ( x ) = x / 1 + ∥ x ∥ 2 / R 2 and the public potential q q q vanishes on ∥ x ∥ ≤ r 0 = 2 m \lVert x\rVert\le r_0=2\sqrt{m} ∥ x ∥ ≤ r 0 = 2 m . A deterministic method must pay order d d d calls for each of the m m m hidden directions. A randomized method can do better on this family, because the minimizer depends only on the sum of the hidden directions, and this sum can be read in batches.
Lemma H.1 (Stages of one parity are read together). Let U = ( u 1 , … , u m ) U=(u_1,\ldots,u_m) U = ( u 1 , … , u m ) have orthonormal columns, and let B ⊂ { 1 , … , m − 1 } B\subset\{1,\ldots,m-1\} B ⊂ { 1 , … , m − 1 } be a nonempty set of indices of the same parity. Suppose z B ∈ R d z_B\in\mathbb{R}^d z B ∈ R d satisfies
⟨ u i , z B ⟩ ≥ 13 8 ( i ∈ B ) , ∣ ⟨ u k , z B ⟩ ∣ ≤ 3 8 ( k ∉ B ) . (81) \langle u_i,z_B\rangle\ge\frac{13}{8}\quad(i\in B),\qquad\lvert\langle u_k,z_B\rangle\rvert\le\frac{3}{8}\quad(k\notin B). \tag*{(81)} ⟨ u i , z B ⟩ ≥ 8 13 ( i ∈ B ) , ∣⟨ u k , z B ⟩∣ ≤ 8 3 ( k ∈ / B ) . (81) Then for every z z z with ∥ z − z B ∥ ≤ 1 / 16 \lVert z-z_B\rVert\le1/16 ∥ z − z B ∥ ≤ 1/16 ,
g m ( U ⊤ z ) = c B − ⟨ u 1 1 { 1 ∉ B } + ∑ i ∈ B u i + 1 , z ⟩ , g_m(U^\top z)=c_B-\left\langle u_1\mathbf{1}_{\{1\notin B\}}+\sum_{i\in B}u_{i+1},z\right\rangle, g m ( U ⊤ z ) = c B − ⟨ u 1 1 { 1 ∈ / B } + i ∈ B ∑ u i + 1 , z ⟩ , where c B = − 9 8 1 { 1 ∉ B } − 9 4 1 { 1 ∈ B } − 9 8 ∣ B ∣ c_B=-\frac{9}{8}\mathbf{1}_{\{1\notin B\}}-\frac{9}{4}\mathbf{1}_{\{1\in B\}}-\frac{9}{8}\lvert B\rvert c B = − 8 9 1 { 1 ∈ / B } − 4 9 1 { 1 ∈ B } − 8 9 ∣ B ∣ . Moreover, at z ∗ = 5 4 ∑ j = 1 m u j z_*=\frac{5}{4}\sum_{j=1}^m u_j z ∗ = 4 5 ∑ j = 1 m u j the chain attains its minimum − 9 4 m -\frac{9}{4}m − 4 9 m , and its gradient in z z z vanishes.
Proof. Indices of the same parity are never consecutive, so B B B and { i + 1 : i ∈ B } \{i+1:i\in B\} { i + 1 : i ∈ B } are disjoint. For ∥ z − z B ∥ ≤ 1 / 16 \lVert z-z_B\rVert\le1/16 ∥ z − z B ∥ ≤ 1/16 , the coordinates y = U ⊤ z y=U^\top z y = U ⊤ z satisfy y i ≥ 25 / 16 > 5 / 4 y_i\ge25/16>5/4 y i ≥ 25/16 > 5/4 for i ∈ B i\in B i ∈ B and ∣ y k ∣ ≤ 7 / 16 < 1 / 2 \lvert y_k\rvert\le7/16<1/2 ∣ y k ∣ ≤ 7/16 < 1/2 for k ∉ B k \notin B k ∈ / B . In (37) , the first term is − 9 / 4 -9/4 − 9/4 if 1 ∈ B 1 \in B 1 ∈ B and − ( y 1 + 9 / 8 ) -(y_1+9/8) − ( y 1 + 9/8 ) otherwise. For r ≥ 2 r \ge2 r ≥ 2 , the r r r th term vanishes when r − 1 ∉ B r-1 \notin B r − 1 ∈ / B , because then ψ ( ± y r − 1 ) = 0 \psi(\pm y_{r-1})=0 ψ ( ± y r − 1 ) = 0 . When r − 1 ∈ B r-1 \in B r − 1 ∈ B , we have r ∉ B r \notin B r ∈ / B , so ψ ( y r − 1 ) = 1 \psi(y_{r-1})=1 ψ ( y r − 1 ) = 1 , ψ ( − y r − 1 ) = 0 \psi(-y_{r-1})=0 ψ ( − y r − 1 ) = 0 , and φ ( y r ) = y r + 9 / 8 \varphi(y_r)=y_r+9/8 φ ( y r ) = y r + 9/8 . Summing gives the displayed affine function. At z ∗ z_\ast z ∗ every coordinate equals 5 / 4 5/4 5/4 , where φ = 9 / 4 \varphi=9/4 φ = 9/4 and ψ = 1 \psi=1 ψ = 1 and both have zero derivative, so each of the m m m terms equals its minimum − 9 / 4 -9/4 − 9/4 .
Proposition H.2 (A randomized method on the unrestricted hard functions). Let m ≥ 2 m \ge2 m ≥ 2 and d ≥ 2 + 80 m log ( 24 m 2 ) d \ge2+80m\log(24m^2) d ≥ 2 + 80 m log ( 24 m 2 ) , and consider the scaled functions f U ( x ) = α F ~ m , U ( x / σ ) f_U(x)=\alpha\widetilde{F}_{m,U}(x/\sigma) f U ( x ) = α F m , U ( x / σ ) of the proof of Theorem E.8 . There is a randomized method using exact values that, for every orthonormal frame U U U , uses at most
2 m + 17 ( d + 1 ) m log ( 24 m 2 ) 2m+17(d+1)\sqrt{m\log(24m^2)} 2 m + 17 ( d + 1 ) m log ( 24 m 2 ) values and, with probability at least 5 / 6 5/6 5/6 , outputs an unqueried exact global minimizer x ^ \widehat{x} x of f U f_U f U , so that ∇ f U ( x ^ ) = 0 \nabla f_U(\widehat{x})=0 ∇ f U ( x ) = 0 . If 500 < T ≤ d / log ( e d ) 500<T\le d/\log(ed) 500 < T ≤ d / log ( e d ) and m m m is the number of stages used in Theorem E.8 , these hypotheses hold and the bound is O ( d T log ( e + T ) ) O(d\sqrt{T}\log(e+T)) O ( d T log ( e + T )) , whereas every deterministic method needs Ω ( d T ) \Omega(dT) Ω ( d T ) values on the same functions.
Algorithm 7 collects the steps of the following proof.
Proof. The constants α , σ > 0 \alpha,\sigma>0 α , σ > 0 are public, so we work with F ~ m , U \widetilde{F}_{m,U} F m , U . At a query point x x x with ρ R ( x ) = z \rho_R(x)=z ρ R ( x ) = z , subtracting the public value q ( x ) q(x) q ( x ) returns g m ( U T z ) g_m(U^\mathsf{T}z) g m ( U T z ) , and ρ R − 1 ( z ) = z / 1 − ∥ z ∥ 2 / R 2 \rho_R^{-1}(z)=z/\sqrt{1-\lVert z\rVert^2/R^2} ρ R − 1 ( z ) = z / 1 − ∥ z ∥ 2 / R 2 for ∥ z ∥ < R \lVert z\rVert<R ∥ z ∥ < R . Put δ = 1 / 16 \delta=1/16 δ = 1/16 ,
R e f f = 3 4 R , K = ⌊ m 40 log ( 24 m 2 ) ⌋ , b = ⌈ 48 d K log ( 24 m 2 ) R e f f 2 ⌉ . R_{\mathrm{eff}}=\frac{3}{4}R,\qquad K=\left\lfloor\sqrt{\frac{m}{40\log(24m^2)}}\right\rfloor,\qquad b=\left\lceil\frac{48dK\log(24m^2)}{R_{\mathrm{eff}}^2}\right\rceil. R eff = 4 3 R , K = ⌊ 40 log ( 24 m 2 ) m ⌋ , b = ⌈ R eff 2 48 d K log ( 24 m 2 ) ⌉ .
The assumption on d d d is equivalent to R e f f ≤ 3 16 ( d − 2 ) / log ( 24 m 2 ) R_{\mathrm{eff}}\le\frac{3}{16}\sqrt{(d-2)/\log(24m^2)} R eff ≤ 16 3 ( d − 2 ) / log ( 24 m 2 ) . All points z z z used below satisfy ∥ z ∥ ≤ R e f f + δ < R \lVert z\rVert\le R_{\mathrm{eff}}+\delta<R ∥ z ∥ ≤ R eff + δ < R . If B B B and z B z_B z B satisfy the hypotheses of Lemma H.1 , the d + 1 d+1 d + 1 values at z B z_B z B and z B + δ e a z_B+\delta e_a z B + δ e a , a = 1 , … , d a=1,\ldots,d a = 1 , … , d , give the difference quotients δ − 1 [ g m ( U T z B ) − g m ( U T ( z B + δ e a ) ) ] \delta^{-1}[g_m(U^\mathsf{T}z_B)-g_m(U^\mathsf{T}(z_B+\delta e_a))] δ − 1 [ g m ( U T z B ) − g m ( U T ( z B + δ e a ))] , which are exactly the coordinates of u 1 1 { 1 ∉ B } + ∑ i ∈ B u i + 1 u_1\mathbf{1}_{\{1\notin B\}}+\sum_{i\in B}u_{i+1} u 1 1 { 1 ∈ / B } + ∑ i ∈ B u i + 1 .
Step 0. Near z = 0 z=0 z = 0 every ∣ y i ∣ ≤ δ < 1 / 2 \lvert y_i\rvert\le\delta<1/2 ∣ y i ∣ ≤ δ < 1/2 , so the chain equals − ( y 1 + 9 / 8 ) -(y_1+9/8) − ( y 1 + 9/8 ) there. The d + 1 d+1 d + 1 values at z = 0 z=0 z = 0 and z = δ e a z=\delta e_a z = δ e a , a = 1 , … , d a=1,\ldots,d a = 1 , … , d , give u 1 u_1 u 1 exactly.
Small m m m . If K = 0 K=0 K = 0 , then m < 40 log ( 24 m 2 ) m<40\log(24m^2) m < 40 log ( 24 m 2 ) . For j = 2 , … , m j=2,\ldots,m j = 2 , … , m in turn, the point 2 u j − 1 2u_{j-1} 2 u j − 1 satisfies (81) with B = { j − 1 } B=\{j-1\} B = { j − 1 } , so its d + 1 d+1 d + 1 values give u 1 1 { j ≠ 2 } + u j u_1\mathbf{1}_{\{j\ne2\}}+u_j u 1 1 { j = 2 } + u j , and hence u j u_j u j . This deterministic method finds ∑ j u j \sum_j u_j ∑ j u j exactly with m ( d + 1 ) < ( d + 1 ) 40 m log ( 24 m 2 ) m(d+1)<(d+1)\sqrt{40m\log(24m^2)} m ( d + 1 ) < ( d + 1 ) 40 m log ( 24 m 2 ) values and outputs x ^ \widehat{x} x as in Phase 2 below. Assume from now on that K ≥ 1 K\ge1 K ≥ 1 , so that K 2 ≤ m / ( 40 log ( 24 m 2 ) ) K^2\le m/(40\log(24m^2)) K 2 ≤ m / ( 40 log ( 24 m 2 )) and m ≥ 645 m\ge645 m ≥ 645 . Since R e f f 2 = 45 16 m R_{\mathrm{eff}}^2=\frac{45}{16}m R eff 2 = 16 45 m , we have K b ≤ 32 75 d + K ≤ ( d − 1 ) / 2 Kb\le\frac{32}{75}d+K\le(d-1)/2 K b ≤ 75 32 d + K ≤ ( d − 1 ) /2 .
Phase 1. Split the odd and the even indices of { 1 , … , m − 1 } \{1,\ldots,m-1\} { 1 , … , m − 1 } into batches of at most K K K indices each; there are at most ⌈ ( m − 1 ) / ( 2 K ) ⌉ + ⌈ ( m − 2 ) / ( 2 K ) ⌉ ≤ min { m − 1 , m / K + 2 } \lceil(m-1)/(2K)\rceil+\lceil(m-2)/(2K)\rceil\le\min\{m-1,m/K+2\} ⌈( m − 1 ) / ( 2 K )⌉ + ⌈( m − 2 ) / ( 2 K )⌉ ≤ min { m − 1 , m / K + 2 } batches. Within each batch, take mutually orthogonal random orthonormal bases S j S_j S j of u j ⊥ u_j^\perp u j ⊥ , one for each index j ≠ 1 j\ne1 j = 1 , as consecutive blocks of a uniformly random orthonormal basis of u 1 ⊥ u_1^\perp u 1 ⊥ , independently across batches; this is possible because K b ≤ ( d − 1 ) / 2 Kb\le(d-1)/2 K b ≤ ( d − 1 ) /2 . Put p ^ 1 = u 1 \widehat{p}_1=u_1 p 1 = u 1 . For j = 2 , … , m − 1 j=2,\ldots,m-1 j = 2 , … , m − 1 in turn, take z ( j ) = R e f f p ^ j − 1 z^{(j)}=R_{\mathrm{eff}}\widehat{p}_{j-1} z ( j ) = R eff p j − 1 , evaluate at z ( j ) z^{(j)} z ( j ) and at z ( j ) + δ v z^{(j)}+\delta v z ( j ) + δ v for an orthonormal basis v v v of S j S_j S j , put
g j = ∑ v δ − 1 [ g m ( U T z ( j ) ) − g m ( U T ( z ( j ) + δ v ) ) ] v ∈ S j , g_j=\sum_v\delta^{-1}\left[g_m(U^\mathsf{T}z^{(j)})-g_m(U^\mathsf{T}(z^{(j)}+\delta v))\right]v\in S_j, g j = v ∑ δ − 1 [ g m ( U T z ( j ) ) − g m ( U T ( z ( j ) + δ v )) ] v ∈ S j ,
and set p ^ j = g j / ∥ g j ∥ \widehat{p}_j=g_j/\lVert g_j\rVert p j = g j / ∥ g j ∥ , or a fixed unit vector of S j S_j S j if g j = 0 g_j=0 g j = 0 .
Probability. Fix U U U , put η = 4 log ( 24 m 2 ) / ( d − 2 ) \eta=\sqrt{4\log(24m^2)/(d-2)} η = 4 log ( 24 m 2 ) / ( d − 2 ) , so that R e f f η ≤ 3 / 8 R_{\mathrm{eff}}\eta\le3/8 R eff η ≤ 3/8 , and for i ≥ 2 i\ge2 i ≥ 2 let p ^ i t r u e = P S i u i / ∥ P S i u i ∥ \widehat{p}_i^{\mathrm{true}}=P_{S_i}u_i/\lVert P_{S_i}u_i\rVert p i true = P S i u i / ∥ P S i u i ∥ . Consider three kinds of events.
Algorithm 7 Randomized method on the hard functions of Proposition H.2 . As in the proof, a query at a point z z z with ∥ z ∥ < R \lVert z\rVert< R ∥ z ∥ < R is made at ρ R − 1 ( z ) \rho_R^{-1}(z) ρ R − 1 ( z ) and returns g ( z ) = g m ( U ⊤ z ) g(z)=g_m(U^\top z) g ( z ) = g m ( U ⊤ z ) after the public value q q q is subtracted. For a point z z z and an orthonormal list W W W , Diff ( z , W ) = ∑ v ∈ W δ − 1 [ g ( z ) − g ( z + δ v ) ] v \operatorname{Diff}(z,W)=\sum_{v\in W}\delta^{-1}[g(z)-g(z+\delta v)]v Diff ( z , W ) = ∑ v ∈ W δ − 1 [ g ( z ) − g ( z + δ v )] v uses 1 + ∣ W ∣ 1+|W| 1 + ∣ W ∣ values; e 1 , … , e d e_1,\ldots,e_d e 1 , … , e d are the coordinate vectors.
Require: m m m , d d d , the public α \alpha α , σ \sigma σ , R R R , q q q , ρ R \rho_R ρ R , and exact values of f U f_U f U
1: δ ← 1 / 16 \delta\leftarrow1/16 δ ← 1/16 , R e f f ← 3 4 R R_{\mathrm{eff}}\leftarrow\frac{3}{4}R R eff ← 4 3 R , K ← ⌊ m / ( 40 log ( 24 m 2 ) ) ⌋ K\leftarrow\left\lfloor\sqrt{m/(40\log(24m^2))}\right\rfloor K ← ⌊ m / ( 40 log ( 24 m 2 )) ⌋ , b ← ⌈ 48 d K log ( 24 m 2 ) / R e f f 2 ⌉ b\leftarrow\left\lceil48dK\log(24m^2)/R_{\mathrm{eff}}^2\right\rceil b ← ⌈ 48 d K log ( 24 m 2 ) / R eff 2 ⌉
2: u 1 ← Diff ( 0 , ( e 1 , … , e d ) ) u_1\leftarrow\operatorname{Diff}(0,(e_1,\ldots,e_d)) u 1 ← Diff ( 0 , ( e 1 , … , e d )) — Step 0
3: if K = 0 K=0 K = 0 then — small m m m
4: for j = 2 , … , m j=2,\ldots,m j = 2 , … , m do
5: u j ← Diff ( 2 u j − 1 , ( e 1 , … , e d ) ) − u 1 1 { j ≠ 2 } u_j\leftarrow\operatorname{Diff}(2u_{j-1},(e_1,\ldots,e_d))-u_1\mathbf{1}_{\{j\ne2\}} u j ← Diff ( 2 u j − 1 , ( e 1 , … , e d )) − u 1 1 { j = 2 }
6: end for
7: s ← ∑ j = 1 m u j s\leftarrow\sum_{j=1}^{m}u_j s ← ∑ j = 1 m u j
8: else — Phase 1
9: split the odd and the even indices of { 1 , … , m − 1 } \{1,\ldots,m-1\} { 1 , … , m − 1 } into batches of at most K K K indices
10: for each batch, independently, draw a uniformly random orthonormal basis of u 1 ⊥ u_1^\perp u 1 ⊥ and give each index i ≠ 1 i\ne1 i = 1 of the batch its own block S i S_i S i of b b b consecutive basis vectors
11: p ^ 1 ← u 1 \hat p_1\leftarrow u_1 p ^ 1 ← u 1
12: for j = 2 , … , m − 1 j=2,\ldots,m-1 j = 2 , … , m − 1 do
13: g j ← Diff ( R e f f p ^ j − 1 , basis of S j ) g_j\leftarrow\operatorname{Diff}(R_{\mathrm{eff}}\hat p_{j-1},\text{basis of }S_j) g j ← Diff ( R eff p ^ j − 1 , basis of S j )
14: p ^ j ← g j / ∥ g j ∥ \hat p_j\leftarrow g_j/\lVert g_j\rVert p ^ j ← g j / ∥ g j ∥ , or a fixed unit vector of S j S_j S j if g j = 0 g_j=0 g j = 0
15: end for
16: s ← u 1 s\leftarrow u_1 s ← u 1 — Phase 2
17: for each batch B B B do
18: z B ← ( R e f f / ∣ B ∣ ) ∑ i ∈ B p ^ i z_B\leftarrow(R_{\mathrm{eff}}/\sqrt{|B|})\sum_{i\in B}\hat p_i z B ← ( R eff / ∣ B ∣ ) ∑ i ∈ B p ^ i
19: s ← s + Diff ( z B , ( e 1 , … , e d ) ) − u 1 1 { 1 ∉ B } s\leftarrow s+\operatorname{Diff}(z_B,(e_1,\ldots,e_d))-u_1\mathbf{1}_{\{1\notin B\}} s ← s + Diff ( z B , ( e 1 , … , e d )) − u 1 1 { 1 ∈ / B }
20: end for
21: end if
22: z ∗ ← 5 4 s z_*\leftarrow\frac{5}{4}s z ∗ ← 4 5 s
23: return x ^ = σ ρ R − 1 ( z ∗ ) \hat x=\sigma\rho_R^{-1}(z_*) x ^ = σ ρ R − 1 ( z ∗ ) if ∥ z ∗ ∥ < R \lVert z_*\rVert<R ∥ z ∗ ∥ < R , and x ^ = 0 \hat x=0 x ^ = 0 otherwise
(i) Signal , for 2 ≤ i ≤ m − 1 2\le i\le m-1 2 ≤ i ≤ m − 1 : ∥ P S i u i ∥ 2 ≥ b / ( 3 d ) \lVert P_{S_i}u_i\rVert^2\ge b/(3d) ∥ P S i u i ∥ 2 ≥ b / ( 3 d ) . Each S i S_i S i is uniform among the b b b -dimensional subspaces of the ( d − 1 ) (d-1) ( d − 1 ) -dimensional space u 1 ⊥ u_1^\perp u 1 ⊥ , which contains u i u_i u i , so ∥ P S i u i ∥ 2 \lVert P_{S_i}u_i\rVert^2 ∥ P S i u i ∥ 2 has the law of X / ( X + Y ) X/(X+Y) X / ( X + Y ) for independent X ∼ χ b 2 X\sim\chi_b^2 X ∼ χ b 2 and Y ∼ χ d − 1 − b 2 Y\sim\chi_{d-1-b}^2 Y ∼ χ d − 1 − b 2 . The event holds if X ≥ b / 2 X\ge b/2 X ≥ b /2 and X + Y ≤ 3 2 ( d − 1 ) X+Y\le\frac{3}{2}(d-1) X + Y ≤ 2 3 ( d − 1 ) . By Chernoff bounds for χ 2 \chi^2 χ 2 variables, these fail with probability at most e − b / 11 + e − ( d − 1 ) / 22 ≤ 1 / ( 12 m ) e^{-b/11}+e^{-(d-1)/22}\le1/(12m) e − b /11 + e − ( d − 1 ) /22 ≤ 1/ ( 12 m ) , since b ≥ 11 log ( 24 m ) b\ge11\log(24m) b ≥ 11 log ( 24 m ) and d − 1 ≥ 22 log ( 24 m ) d-1\ge22\log(24m) d − 1 ≥ 22 log ( 24 m ) .
(ii) Triggers , for 2 ≤ i ≤ m − 2 2\le i\le m-2 2 ≤ i ≤ m − 2 and k ∉ { 1 , i } k\notin\{1,i\} k ∈ / { 1 , i } : ∣ ⟨ u k , p ^ i t r u e ⟩ ∣ ≤ η |\langle u_k,\hat p_i^{\mathrm{true}}\rangle|\le\eta ∣ ⟨ u k , p ^ i true ⟩ ∣ ≤ η . The law of S i S_i S i is invariant under the rotations of u 1 ⊥ u_1^\perp u 1 ⊥ that fix u i u_i u i , so the component of p ^ i t r u e \hat p_i^{\mathrm{true}} p ^ i true orthogonal to u i u_i u i has a uniform direction in the ( d − 2 ) (d-2) ( d − 2 ) -dimensional space { u 1 , u i } ⊥ \{u_1,u_i\}^{\perp} { u 1 , u i } ⊥ , which contains u k u_k u k . The tail bound in the proof of Lemma C.3 shows that the event fails with probability at most 2 e − ( d − 2 ) η 2 / 2 = 1 / ( 288 m 4 ) 2e^{-(d-2)\eta^2/2}=1/(288m^4) 2 e − ( d − 2 ) η 2 /2 = 1/ ( 288 m 4 ) .
(iii) Batches , for each batch B B B and each k ≥ 2 k\ge2 k ≥ 2 : ∣ ⟨ u k , w B , k ⟩ ∣ ≤ η ∣ B ∣ |\langle u_k,w_{B,k}\rangle|\le\eta\sqrt{|B|} ∣ ⟨ u k , w B , k ⟩ ∣ ≤ η ∣ B ∣ , where w B , k = ∑ i ∈ B ∖ { 1 , k } p ^ i t r u e w_{B,k}=\sum_{i\in B\setminus\{1,k\}}\hat p_i^{\mathrm{true}} w B , k = ∑ i ∈ B ∖ { 1 , k } p ^ i true . The joint law of the subspaces of B B B is invariant under the rotations of u 1 ⊥ u_1^\perp u 1 ⊥ that fix every u i u_i u i with i ∈ B ∖ { 1 , k } i\in B\setminus\{1,k\} i ∈ B ∖ { 1 , k } , and these rotations act on each such p ^ i t r u e \hat p_i^{\mathrm{true}} p ^ i true by rotation. Hence the component of w B , k w_{B,k} w B , k orthogonal to A = span { u 1 , u i : i ∈ B ∖ { 1 , k } } A=\operatorname{span}\{u_1,u_i:i\in B\setminus\{1,k\}\} A = span { u 1 , u i : i ∈ B ∖ { 1 , k }} has a uniform direction in A ⊥ A^\perp A ⊥ , which contains u k u_k u k and has dimension at least d − 1 − K ≥ ( d − 2 ) / 2 d-1-K\ge(d-2)/2 d − 1 − K ≥ ( d − 2 ) /2 . As ∥ w B , k ∥ ≤ ∣ B ∣ \lVert w_{B,k}\rVert\le\sqrt{|B|} ∥ w B , k ∥ ≤ ∣ B ∣ , the event fails with probability at most 2 e − ( d − 2 ) η 2 / 4 = 1 / ( 12 m 2 ) 2e^{-(d-2)\eta^2/4}=1/(12m^2) 2 e − ( d − 2 ) η 2 /4 = 1/ ( 12 m 2 ) .
There are m − 2 m-2 m − 2 signal events, fewer than m 2 m^2 m 2 trigger events, and at most m ( m − 1 ) m(m-1) m ( m − 1 ) batch events, so all of them hold with probability at least 1 − 2 m − 3 12 m − 288 m 2 > 5 6 1-\frac{2m^{-3}}{12m}-\frac{288}{m^2}>\frac{5}{6} 1 − 12 m 2 m − 3 − m 2 288 > 6 5 .
On this event, p ^ j = p ^ j t r u e \hat{p}_j=\hat{p}^{\mathrm{true}}_j p ^ j = p ^ j true for 2 ≤ j ≤ m − 1 2\le j\le m-1 2 ≤ j ≤ m − 1 , by induction on j j j . Indeed, z ( j ) z^{(j)} z ( j ) satisfies (81) with B = { j − 1 } B=\{j-1\} B = { j − 1 } : its inner product with u j − 1 u_{j-1} u j − 1 is R e f f ≥ 2 R_{\mathrm{eff}}\ge2 R eff ≥ 2 if j = 2 j=2 j = 2 and otherwise R e f f ∥ P S j − 1 u j − 1 ∥ ≥ R e f f b / ( 3 d ) ≥ 4 K log ( 24 m 2 ) ≥ 2 R_{\mathrm{eff}}\lVert P_{S_{j-1}}u_{j-1}\rVert\ge R_{\mathrm{eff}}\sqrt{b/(3d)}\ge4\sqrt{K}\log(24m^2)\ge2 R eff ∥ P S j − 1 u j − 1 ∥ ≥ R eff b / ( 3 d ) ≥ 4 K log ( 24 m 2 ) ≥ 2 , and its other inner products vanish or, by the trigger events, are at most R e f f η ≤ 3 / 8 R_{\mathrm{eff}}\eta\le3/8 R eff η ≤ 3/8 in absolute value. By Lemma H.1 the chain is affine near z ( j ) z^{(j)} z ( j ) with gradient − u 2 -u_2 − u 2 if j = 2 j=2 j = 2 and − ( u 1 + u j ) -(u_1+u_j) − ( u 1 + u j ) otherwise. Every v ∈ S j v\in S_j v ∈ S j is orthogonal to u 1 u_1 u 1 , so the difference quotients are exact, g j = P S j u j ≠ 0 g_j=P_{S_j}u_j\ne0 g j = P S j u j = 0 , and p ^ j = p ^ j t r u e \hat{p}_j=\hat{p}^{\mathrm{true}}_j p ^ j = p ^ j true . For each batch B B B put
z B = R e f f ∣ B ∣ ∑ i ∈ B p ^ i . z_B=\frac{R_{\mathrm{eff}}}{\sqrt{\lvert B\rvert}}\sum_{i\in B}\hat{p}_i. z B = ∣ B ∣ R eff i ∈ B ∑ p ^ i .
which has norm R e f f R_{\mathrm{eff}} R eff , because the p ^ i \hat{p}_i p ^ i of one batch are orthonormal. For i ∈ B i\in B i ∈ B with i ≥ 2 i\ge2 i ≥ 2 , the signal and batch events give ⟨ u i , z B ⟩ ≥ R e f f b / ( 3 d K ) − R e f f η ≥ 2 − 3 8 = 13 8 \langle u_i,z_B\rangle\ge R_{\mathrm{eff}}\sqrt{b/(3dK)}-R_{\mathrm{eff}}\eta\ge2-\frac{3}{8}=\frac{13}{8} ⟨ u i , z B ⟩ ≥ R eff b / ( 3 d K ) − R eff η ≥ 2 − 8 3 = 8 13 ; if 1 ∈ B 1\in B 1 ∈ B , then ⟨ u 1 , z B ⟩ = R e f f / ∣ B ∣ ≥ 2 \langle u_1,z_B\rangle=R_{\mathrm{eff}}/\sqrt{\lvert B\rvert}\ge2 ⟨ u 1 , z B ⟩ = R eff / ∣ B ∣ ≥ 2 ; and for k ∉ B k\notin B k ∈ / B , ⟨ u k , z B ⟩ \langle u_k,z_B\rangle ⟨ u k , z B ⟩ vanishes if k = 1 k=1 k = 1 and otherwise has absolute value at most R e f f η ≤ 3 8 R_{\mathrm{eff}}\eta\le\frac{3}{8} R eff η ≤ 8 3 by the batch events.
Phase 2. For each batch B B B , the d + 1 d+1 d + 1 values at z B z_B z B and z B + δ e a z_B+\delta e_a z B + δ e a give u 1 1 { 1 ∉ B } + ∑ i ∈ B u i + 1 u_1\mathbf{1}_{\{1\notin B\}}+\sum_{i\in B}u_{i+1} u 1 1 { 1 ∈ / B } + ∑ i ∈ B u i + 1 , and hence ∑ i ∈ B u i + 1 \sum_{i\in B}u_{i+1} ∑ i ∈ B u i + 1 . The batches partition { 1 , … , m − 1 } \{1,\ldots,m-1\} { 1 , … , m − 1 } , so u 1 u_1 u 1 plus the sum of all batch vectors equals ∑ j = 1 m u j \sum_{j=1}^{m}u_j ∑ j = 1 m u j exactly. Let z ∗ z_* z ∗ be 5 4 \frac{5}{4} 4 5 times this computed sum, and output x ^ = σ ρ R − 1 ( z ∗ ) \hat{x}=\sigma\rho_R^{-1}(z_*) x ^ = σ ρ R − 1 ( z ∗ ) , or x ^ = 0 \hat{x}=0 x ^ = 0 if ∥ z ∗ ∥ ≥ R \lVert z_*\rVert\ge R ∥ z ∗ ∥ ≥ R . On the event above, z ∗ = 5 4 ∑ j u j z_*=\frac{5}{4}\sum_j u_j z ∗ = 4 5 ∑ j u j , ∥ z ∗ ∥ 2 = 25 16 m \lVert z_*\rVert^2=\frac{25}{16}m ∥ z ∗ ∥ 2 = 16 25 m , and
∥ ρ R − 1 ( z ∗ ) ∥ 2 = 25 m / 16 1 − 5 / 16 = 25 11 m < 4 m = r 0 2 . \left\lVert\rho_R^{-1}(z_*)\right\rVert^2=\frac{25m/16}{1-5/16}=\frac{25}{11}m<4m=r_0^2. ρ R − 1 ( z ∗ ) 2 = 1 − 5/16 25 m /16 = 11 25 m < 4 m = r 0 2 .
so q q q and ∇ q \nabla q ∇ q vanish there, and Lemma H.1 shows that x ^ \hat{x} x ^ is a global minimizer with zero gradient.
Cost. The method uses at most d + 1 + ( m − 2 ) ( b + 1 ) + ( d + 1 ) ( m / K + 2 ) d+1+(m-2)(b+1)+(d+1)(m/K+2) d + 1 + ( m − 2 ) ( b + 1 ) + ( d + 1 ) ( m / K + 2 ) values. Here ( m − 2 ) ( b + 1 ) ≤ 256 15 d K log ( 24 m 2 ) + 2 m ≤ 256 15 d m log ( 24 m 2 ) + 2 m (m-2)(b+1)\le\frac{256}{15}dK\log(24m^2)+2m\le\frac{256}{15}d\sqrt{m}\log(24m^2)+2m ( m − 2 ) ( b + 1 ) ≤ 15 256 d K log ( 24 m 2 ) + 2 m ≤ 15 256 d m log ( 24 m 2 ) + 2 m and m / K ≤ 2 40 m log ( 24 m 2 ) m/K\le2\sqrt{40m\log(24m^2)} m / K ≤ 2 40 m log ( 24 m 2 ) , which give the stated bound since m log ( 24 m 2 ) ≥ 100 \sqrt{m}\log(24m^2)\ge100 m log ( 24 m 2 ) ≥ 100 .
Range. If 500 < T ≤ d / log ( e d ) 500<T\le d/\log(ed) 500 < T ≤ d / log ( e d ) , then Theorem E.8 uses m = ⌊ T / 275 + 1 / 2 ⌋ ≥ 2 m=\lfloor T/275+1/2\rfloor\ge2 m = ⌊ T /275 + 1/2 ⌋ ≥ 2 , so m ≤ 4 3 ( m − 1 2 ) ≤ d / ( 206 log ( e d ) ) m\le\frac{4}{3}(m-\frac{1}{2})\le d/(206\log(ed)) m ≤ 3 4 ( m − 2 1 ) ≤ d / ( 206 log ( e d )) . Hence 24 m 2 < d 2 24m^2<d^2 24 m 2 < d 2 and 80 m log ( 24 m 2 ) < 160 206 d ≤ d − 2 80m\log(24m^2)<\frac{160}{206}d\le d-2 80 m log ( 24 m 2 ) < 206 160 d ≤ d − 2 , because d > 4700 d>4700 d > 4700 . Finally 2 m ≤ d 2m\le d 2 m ≤ d and m log ( 24 m 2 ) = O ( T log ( e + T ) ) \sqrt{m}\log(24m^2)=O(\sqrt{T}\log(e+T)) m log ( 24 m 2 ) = O ( T log ( e + T )) , and Theorem E.8 gives the deterministic bound d T / 500 dT/500 d T /500 .
The hard functions of Theorem E.8 , on which deterministic methods pay Ω ( d T ) \Omega(dT) Ω ( d T ) , are solved with O ( d T log ( e + T ) ) O(d\sqrt{T}\log(e+T)) O ( d T log ( e + T )) values by the proposition, so they cannot give a randomized lower bound of that order. For exact values, Xia and Yuan [55 ] prove a randomized lower bound of order d T dT d T with a different construction when d ≥ C T log ( 2 + T ) d\ge CT\log(2+T) d ≥ C T log ( 2 + T ) for an absolute constant C C C ; whether such a bound also holds under gradient projections is open.
Exact values: a wider window with matching bounds
For exact values, the concurrent and independent lower bound of Xia and Yuan [55 ] extends the randomized bound Ω ( E T ) \Omega(ET) Ω ( E T ) beyond E T ≤ d ET\le d E T ≤ d . Combined with our deterministic bounds, it removes the condition E ≤ d / T E\le d/T E ≤ d / T from Corollary 3.3 when T T T is at most of order d / log ( e d ) d/\log(ed) d / log ( e d ) . Their bound concerns exact values only, so the rank-priced oracle is not covered, and the constants are no longer explicit.
Corollary H.3 (A wider window for exact values). There are absolute constants c 1 , c 2 > 0 c_1,c_2>0 c 1 , c 2 > 0 such that, for exact values on every class between F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d and B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d , if E T ≤ c 1 d 2 / log ( e d ) ET\le c_1d^2/\log(ed) E T ≤ c 1 d 2 / log ( e d ) , then N r a n d ≥ c 2 E T N_{\mathrm{rand}}\ge c_2ET N rand ≥ c 2 E T . Consequently, if T log ( e + T ) ≤ E ≤ d T\log(e+T)\le E\le d T log ( e + T ) ≤ E ≤ d and T ≤ c 1 d / log ( e d ) T\le c_1d/\log(ed) T ≤ c 1 d / log ( e d ) , then
d T 18432 ≤ N d e t ≤ 10 d T , c 2 E T ≤ N r a n d ≤ 23 E T . \frac{dT}{18432}\le N_{\mathrm{det}}\le10dT,\qquad c_2ET\le N_{\mathrm{rand}}\le23ET. 18432 d T ≤ N det ≤ 10 d T , c 2 E T ≤ N rand ≤ 23 E T .
Proof. The deterministic lower bound is Corollary G.10, which requires only T log ( e + T ) ≤ E ≤ d T\log(e+T)\le E\le d T log ( e + T ) ≤ E ≤ d . The two upper bounds and the randomized lower bound min { d , E T } / 12 \min\{d,ET\}/12 min { d , E T } /12 are in Theorem A.1. Lower bounds proved on F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d and upper bounds proved on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d hold on every class between them. It remains to bound N r a n d N_{\mathrm{rand}} N rand from below when E T > d ET>d E T > d .
Put ℓ = L E / d ≤ L \ell=LE/d\le L ℓ = L E / d ≤ L and τ = ℓ Δ / ε 2 = E T / d > 1 \tau=\ell\Delta/\varepsilon^2=ET/d>1 τ = ℓ Δ/ ε 2 = E T / d > 1 . Every globally ℓ \ell ℓ -smooth function with f ( 0 ) − inf f ≤ Δ f(0)-\inf f\le\Delta f ( 0 ) − inf f ≤ Δ lies in F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , with the upper model ℓ I d \ell I_d ℓ I d , whose norm is ℓ ≤ L \ell\le L ℓ ≤ L and whose trace is L E LE L E . Theorem 10.5 of Xia and Yuan [55 ] uses the same starting point 0 0 0 , the same gap f ( 0 ) − inf f f(0)-\inf f f ( 0 ) − inf f , a cap on the number of exact values, and success probability at least 2 / 3 2/3 2/3 on every function. We apply it with smoothness ℓ \ell ℓ , gap Δ \Delta Δ , and a Lipschitz constant L 0 ≥ ℓ Δ L_0\ge\sqrt{\ell\Delta} L 0 ≥ ℓ Δ , so that min { ℓ Δ , L 0 2 } = ℓ Δ \min\{\ell\Delta,L_0^2\}=\ell\Delta min { ℓ Δ , L 0 2 } = ℓ Δ . It provides absolute constants c , C , c 0 > 0 c,C,c_0>0 c , C , c 0 > 0 such that, if τ ≥ 1 / c 0 \tau\ge1/c_0 τ ≥ 1/ c 0 and d ≥ C τ log ( 2 + τ ) d\ge C\tau\log(2+\tau) d ≥ C τ log ( 2 + τ ) , every randomized algorithm needs at least c d τ = c E T c\,d\tau=c\,ET c d τ = c E T exact values on its hard functions; these are globally ℓ \ell ℓ -smooth with gap at most Δ \Delta Δ , and hence lie in F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d . If 1 < τ < 1 / c 0 1<\tau<1/c_0 1 < τ < 1/ c 0 , the bound min { d , E T } / 12 \min\{d,ET\}/12 min { d , E T } /12 gives N r a n d ≥ d / 12 > c 0 E T / 12 N_{\mathrm{rand}}\ge d/12>c_0ET/12 N rand ≥ d /12 > c 0 E T /12 . Finally, let c 1 = min { 1 , 1 / ( 2 C ) } c_1=\min\{1,1/(2C)\} c 1 = min { 1 , 1/ ( 2 C )} . Then τ = E T / d ≤ c 1 d / log ( e d ) ≤ d \tau=ET/d\le c_1d/\log(ed)\le d τ = E T / d ≤ c 1 d / log ( e d ) ≤ d , and 2 + d ≤ ( e d ) 2 2+d\le(ed)^2 2 + d ≤ ( e d ) 2 gives τ log ( 2 + τ ) ≤ c 1 d log ( 2 + d ) / log ( e d ) ≤ 2 c 1 d ≤ d / C \tau\log(2+\tau)\le c_1d\log(2+d)/\log(ed)\le2c_1d\le d/C τ log ( 2 + τ ) ≤ c 1 d log ( 2 + d ) / log ( e d ) ≤ 2 c 1 d ≤ d / C . Hence the corollary holds with c 2 = min { 1 / 12 , c 0 / 12 , c } c_2=\min\{1/12,c_0/12,c\} c 2 = min { 1/12 , c 0 /12 , c } , and the cost ratio N d e t / N r a n d N_{\mathrm{det}}/N_{\mathrm{rand}} N det / N rand is of order d / E d/E d / E .
Experimental Protocol
These experiments illustrate the mechanisms behind the main theorems and the plausibility of the curvature assumption. All synthetic runs use L = Δ = 1 L=\Delta=1 L = Δ = 1 , x 0 = 0 x_0=0 x 0 = 0 , and ε = T − 1 / 2 \varepsilon=T^{-1/2} ε = T − 1/2 , and they charge every function value, including difference probes, rejected proposals, line search centers, and final checks. Three metrics are reported separately: the offline first-hit cost, that is, the number of calls before the first retained iterate with ∥ ∇ f ∥ ≤ ε \lVert\nabla f\rVert\le\varepsilon ∥ ∇ f ∥ ≤ ε , computed by an observer that does not affect the run; whether the output is ε \varepsilon ε -stationary; and the total call count.
Randomized versus deterministic.
The first family is the rank-one quadratic f u ( x ) = 1 2 ( ⟨ u , x ⟩ − 2 ) 2 f_u(x)=\frac{1}{2}(\langle u,x\rangle-\sqrt{2})^2 f u ( x ) = 2 1 (⟨ u , x ⟩ − 2 ) 2 , with E = 1 E=1 E = 1 . The second uses ϕ ( t ) = 0.1 t 2 + 0.8 ( 1 − cos t ) \phi(t)=0.1t^2+0.8(1-\cos t) ϕ ( t ) = 0.1 t 2 + 0.8 ( 1 − cos t ) and
f ( x ) = a 2 ∑ i = 1 d h i ϕ ( ( Q ⊤ x ) i / a + θ i ) , θ = ( 2 , 0 , … , 0 ) , a = 1 ϕ ( 2 ) . f(x)=a^2\sum_{i=1}^{d}h_i\phi\left((Q^\top x)_i/a+\theta_i\right),\qquad\theta=(2,0,\ldots,0),\qquad a=\frac{1}{\sqrt{\phi(2)}}. f ( x ) = a 2 i = 1 ∑ d h i ϕ ( ( Q ⊤ x ) i / a + θ i ) , θ = ( 2 , 0 , … , 0 ) , a = ϕ ( 2 ) 1 .
with max i h i = 1 \max_i h_i=1 max i h i = 1 and ∑ i h i = E \sum_i h_i=E ∑ i h i = E . Since ϕ ′ ′ ∈ [ − 0.6 , 1 ] \phi''\in[-0.6,1] ϕ ′′ ∈ [ − 0.6 , 1 ] , one has ∇ 2 f ⪯ Q diag ( h ) Q ⊤ \nabla^2f\preceq Q\operatorname{diag}(h)Q^\top ∇ 2 f ⪯ Q diag ( h ) Q ⊤ everywhere, so f f f lies in the fixed-matrix class; it is nonconvex and coercive, and f ( 0 ) − inf f = 1 f(0)-\inf f=1 f ( 0 ) − inf f = 1 . At x 0 = 0 x_0=0 x 0 = 0 only the first rotated coordinate is away from its minimizer, since ϕ ′ ( 0 ) = 0 \phi'(0)=0 ϕ ′ ( 0 ) = 0 , so along the line through x 0 x_0 x 0 in direction Q e 1 Qe_1 Q e 1 the gradient stays parallel to Q e 1 Qe_1 Q e 1 ; random directions still pay the curvature of the other coordinates. The concentrated spectrum sets h 1 = 1 h_1=1 h 1 = 1 , assigns $0.1/(d-1)$ to each other eigenvalue, and distributes the remaining trace in order without exceeding one; the diffuse spectrum sets h 1 = 1 h_1=1 h 1 = 1 and h i = ( E − 1 ) / ( d − 1 ) h_i=(E-1)/(d-1) h i = ( E − 1 ) / ( d − 1 ) for i > 1 i>1 i > 1 . The rotation is the Householder reflection Q = I − 2 v v ⊤ Q=I-2vv^\top Q = I − 2 v v ⊤ with v = ( e 1 − u ) / ∥ e 1 − u ∥ v=(e_1-u)/\lVert e_1-u\rVert v = ( e 1 − u ) / ∥ e 1 − u ∥ and u u u uniform on the sphere. The grid uses d ∈ { 10 3 , 10 4 , 10 5 } d\in\{10^3,10^4,10^5\} d ∈ { 1 0 3 , 1 0 4 , 1 0 5 } , T ∈ { 2 , 8 , 32 } T\in\{2,8,32\} T ∈ { 2 , 8 , 32 } , and E ∈ { 2 , 4 , 8 , … , 128 } E\in\{2,4,8,\ldots,128\} E ∈ { 2 , 4 , 8 , … , 128 } for the nonconvex family, keeps the cells with E T ≤ d / 4 ET\le d/4 E T ≤ d /4 , and has five rotation seeds per cell. The three-point method uses h = 4 ε / ( 5 L ( E + 1 ) ) h=4\varepsilon/(5L(E+1)) h = 4 ε / ( 5 L ( E + 1 )) , $N=\lceil 10(E+1)T\rceil steps, an output drawn uniformly and independently from the iterates before the updates, and the hard cap 1 + 2 N 1 + 2N 1 + 2 N of Theorem G.3 ; it is also run with a fixed seed. The deterministic methods are coordinate central differences and finite-difference L-BFGS on both families, and Krylov conjugate gradients on the quadratic family.
On the regular grid (575 runs per method, 45 for Krylov), every method reached the target offline in every run. The returned output was ε \varepsilon ε -stationary in 524 of 575 runs of the three-point method, in 509 of 575 with a fixed seed, and in all runs of the deterministic methods. At d = 10 5 d = 10^{5} d = 1 0 5 , the log–log slopes of the first-hit cost of the three-point method range from 0.54 to 0.75 in T T T and from 0.90 to 0.95 in E E E . The 27 targeted runs comprise nine each of three-point method with a fixed seed, coordinate differences, and finite-difference L-BFGS. Each uses a rank-one quadratic chosen orthogonal to all of the run’s queries, a property certified in exact integer arithmetic (15 distinct functions). All 27 runs ended at gradient norm 2 \sqrt{2} 2 without an offline hit and are reported as right-censored, whereas 20 fresh seeds on each of these functions succeeded in 17–20 runs.
Stronger baselines. On the 43 cells of the regular grid with E ∈ { 1 , 4 , 16 } E \in\{1, 4, 16\} E ∈ { 1 , 4 , 16 } (215 instances, with the same functions and rotation seeds), we also ran two further methods. The Gaussian two-point method uses g k = [ f ( x k + μ a k ) − f ( x k − μ a k ) ] a k / ( 2 μ ) g_k = [f(x_k + \mu a_k) - f(x_k - \mu a_k)]a_k/(2\mu) g k = [ f ( x k + μ a k ) − f ( x k − μ a k )] a k / ( 2 μ ) with a k ∼ N ( 0 , I d ) a_k \sim N(0, I_d) a k ∼ N ( 0 , I d ) , step 1 / ( L ( E + 2 ) ) 1/(L(E + 2)) 1/ ( L ( E + 2 )) , N = ⌈ ( 6 E + 13 ) T ⌉ N = \lceil(6E + 13)T\rceil N = ⌈( 6 E + 13 ) T ⌉ steps, μ = ε / ( 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 ) ) \mu= \varepsilon/(4L(6E + 13)\sqrt{d}(d + 2)(d + 4)) μ = ε / ( 4 L ( 6 E + 13 ) d ( d + 2 ) ( d + 4 )) , and an output drawn uniformly from the iterates before the updates, for 2 N 2N 2 N calls in total. The restarted subspace search uses the capacity given below. The two-point method returned ε \varepsilon ε -stationary outputs in 203 of 215 runs and the subspace search in all 215; paired with the three-point method on the same functions and seeds, the subspace search reached the target later in all 43 cells, by a median factor between 8.7 and 1.8 × 10 4 1.8 \times10^{4} 1.8 × 1 0 4 . As in the targeted runs above, the subspace search with budgets 32, 64, and 128 below d d d ended at gradient norm 2 \sqrt{2} 2 on six rank-one quadratics orthogonal to all of its queries and its output, whereas 20 fresh seeds of the three-point method per function succeeded in 114 of 120 runs.
Monotone two-point method and wall functions. We also ran the monotone two-point method of Theorem B.10 on the same 215 instances, with the step, the number of updates, and the difference radius of the two-point method above, forward differences, and 2 N + 1 2N + 1 2 N + 1 calls. Its first hit came after a median of 31 calls, against 30 for the two-point method; in the cells of Figure 2 (a) it trailed the two-point method by about one call, its initial query. Its uniformly drawn output was ε \varepsilon ε -stationary in 203 of 215 runs, as for the two-point method, and its last iterate in all 215. We also built wall functions f ( x ) = h ( P x ) + β ( ∥ ( I − P ) x ∥ 2 ) f(x) = h(Px) + \beta(\lVert(I - P)x\rVert^{2}) f ( x ) = h ( P x ) + β (∥( I − P ) x ∥ 2 ) , where P P P projects onto a random subspace of dimension E 0 E_0 E 0 , h h h is a sum of d d d -dimensional separable profiles with ∣ h ′ ′ ∣ ≤ L |h''| \leq L ∣ h ′′ ∣ ≤ L and h ( 0 ) = inf h = Δ / 2 h(0) = \inf h = \Delta/2 h ( 0 ) = inf h = Δ/2 , and β ( s ) = L E 0 s / ( 2 d ) \beta(s) = LE_0s/(2d) β ( s ) = L E 0 s / ( 2 d ) for s ≤ s 1 = d Δ / ( L E 0 ) s \leq s_1 = d\Delta/(LE_0) s ≤ s 1 = d Δ/ ( L E 0 ) , beyond which β ′ \beta' β ′ rises smoothly to L / 4 L/4 L /4 with 2 β ′ + 4 s β ′ ′ ≤ L 2\beta' + 4s\beta'' \leq L 2 β ′ + 4 s β ′′ ≤ L . Below f ( 0 ) f(0) f ( 0 ) one has s < s 1 s < s_1 s < s 1 , so these functions lie in B L , Δ , E 0 d \mathcal{B}^{d}_{L,\Delta,E_0} B L , Δ , E 0 d , whereas above f ( 0 ) f(0) f ( 0 ) the Hessian trace reaches about L d / 2 Ld/2 L d /2 . On 45 instances (9 cells with d ∈ { 10 3 , 10 4 , 10 5 } d \in\{10^{3}, 10^{4}, 10^{5}\} d ∈ { 1 0 3 , 1 0 4 , 1 0 5 } , T ∈ { 8 , 32 } T \in\{8, 32\} T ∈ { 8 , 32 } , E 0 ∈ { 4 , 16 } E_0 \in\{4, 16\} E 0 ∈ { 4 , 16 } , and 8 E 0 T ≤ d 8E_0T \leq d 8 E 0 T ≤ d , five seeds each), the last iterate of the monotone two-point method was ε \varepsilon ε -stationary in all runs, and the uniformly drawn outputs of the two-point and three-point methods in 44 and 42; their median first hits were 69, 72, and 173 calls. No method exceeded f ( 0 ) f(0) f ( 0 ) . The two-point method with step 1 / ( L ( d + 2 ) ) 1/(L(d + 2)) 1/ ( L ( d + 2 )) , which is safe without an acceptance test, reached the target in none of the 45 runs within the same budget.
Subspace search. The restarted subspace search uses capacity r = ⌈ d E T / ( d + T ) ⌉ r = \lceil\sqrt{dET/(d + T)}\rceil r = ⌈ d E T / ( d + T ) ⌉ , a variant without restarts uses r = d r = d r = d , and coordinate central differences follow Appendix § 11 . On the two families above with d ∈ { 10 3 , 10 4 } d \in\{10^{3}, 10^{4}\} d ∈ { 1 0 3 , 1 0 4 } , T ∈ { 2 , 8 , 32 , 128 , 512 } T \in\{2, 8, 32, 128, 512\} T ∈ { 2 , 8 , 32 , 128 , 512 } , E = 1 E = 1 E = 1 for the quadratic, and E ∈ { 2 , 4 , 8 , 16 } E \in\{2, 4, 8, 16\} E ∈ { 2 , 4 , 8 , 16 } for the nonconvex family (1,800 runs including the three-point method), all deterministic runs returned ε \varepsilon ε -stationary outputs and the three-point method did so in 425 of 450 runs. Every subspace run used exactly one expansion and no restart, consistent with the gradient staying along Q e 1 Qe_1 Q e 1 ; on the nonconvex family, the paired median ratio of total calls between the subspace search and coordinate differences is about 0.67 for T ≤ 128 T \le128 T ≤ 128 and 0.50 at T = 512 T = 512 T = 512 .
The chain has E ∈ { 8 , 16 } E \in\{8,16\} E ∈ { 8 , 16 } , m = 2 E m = 2E m = 2 E , a = 8 ( m + 1 ) / m a = \sqrt{8(m+1)/m} a = 8 ( m + 1 ) / m , a fixed Householder reflection R R R , z = ( R x ) 1 : m z = (Rx)_{1:m} z = ( R x ) 1 : m , and
f ( x ) = 1 8 [ ( z 1 − a ) 2 + ∑ i = 1 m − 1 ( z i − z i + 1 ) 2 + z m 2 ] − 1 m . f(x) = \frac{1}{8}\left[(z_1-a)^2+\sum_{i=1}^{m-1}(z_i-z_{i+1})^2+z_m^2\right]-\frac{1}{m}. f ( x ) = 8 1 [ ( z 1 − a ) 2 + i = 1 ∑ m − 1 ( z i − z i + 1 ) 2 + z m 2 ] − m 1 .
Its global upper model H = 1 4 R T diag ( K m , 0 ) R H = \frac{1}{4}R^\mathsf{T}\operatorname{diag}(K_m,0)R H = 4 1 R T diag ( K m , 0 ) R , where K m K_m K m is tridiagonal with diagonal 2 and off-diagonal − 1 -1 − 1 , satisfies ∥ H ∥ o p < 1 \lVert H\rVert_{\mathrm{op}} < 1 ∥ H ∥ op < 1 and tr H = E \operatorname{tr} H = E tr H = E ; moreover f ( 0 ) = 1 f(0) = 1 f ( 0 ) = 1 and inf f = 0 \inf f = 0 inf f = 0 . At a point supported on the first j j j active coordinates, the gradient adds only coordinate j + 1 j+1 j + 1 , so each expansion can add at most one direction. With d ∈ { 10 3 , 10 4 } d \in\{10^3,10^4\} d ∈ { 1 0 3 , 1 0 4 } , T ∈ { 2 , 8 , 32 , 128 , 512 } T \in\{2,8,32,128,512\} T ∈ { 2 , 8 , 32 , 128 , 512 } , and five seeds, all 300 runs returned ε \varepsilon ε -stationary outputs; 240 of them satisfy E T ≤ d ET \le d E T ≤ d . The number of expansions was 1, 2, 3, 6, 10 at the five accuracies, with no restart. At d = 10 4 d = 10^4 d = 1 0 4 , the paired median ratio of total calls was about 1.00 for T ≤ 8 T \le8 T ≤ 8 , 0.57 at T = 32 T = 32 T = 32 , 0.39 to 0.41 at T = 128 T = 128 T = 128 , and 0.25 to 0.26 at T = 512 T = 512 T = 512 . Over the five accuracies, the log–log slope of the total cost is about 0.31 for the subspace search and 0.57 for coordinate differences, both below the exponents 1 / 2 1/2 1/2 and 1 of the worst-case upper bounds.
Local curvature of GPT-2. GPT-2 uses the GELU activation, so its loss is smooth in the parameters. We ran MeZO on SST-2 and RTE with three seeds each and 1,024 updates, reaching final accuracies of 76.6–85.2% and 50.8–53.9%, and at updates 0, 256, 512, 1024 measured the Hessian G G G of a fixed training batch in all d = 124,439,808 d = 124{,}439{,}808 d = 124 , 439 , 808 parameters, in FP32. Since tr ( G + ) = ( tr G + ∥ G ∥ ∗ ) / 2 \operatorname{tr}(G_+) = (\operatorname{tr} G + \lVert G\rVert_*)/2 tr ( G + ) = ( tr G + ∥ G ∥ ∗ ) /2 and ∥ G ∥ ∗ ≤ d ∥ G ∥ F \lVert G\rVert_* \le\sqrt{d}\lVert G\rVert_F ∥ G ∥ ∗ ≤ d ∥ G ∥ F , any u 1 ≥ tr G u_1 \ge\operatorname{tr} G u 1 ≥ tr G , u 2 ≥ ∥ G ∥ F 2 u_2 \ge\lVert G\rVert_F^2 u 2 ≥ ∥ G ∥ F 2 and ℓ ∈ ( 0 , ∥ G ∥ o p ] \ell\in(0,\lVert G\rVert_{\mathrm{op}}] ℓ ∈ ( 0 , ∥ G ∥ op ] give
tr ( G + ) d ∥ G ∥ o p ≤ u 1 / d + u 2 / d 2 ℓ . \frac{\operatorname{tr}(G_+)}{d\lVert G\rVert_{\mathrm{op}}} \le\frac{u_1/d+\sqrt{u_2/d}}{2\ell}. d ∥ G ∥ op tr ( G + ) ≤ 2 ℓ u 1 / d + u 2 / d .
We take for ℓ \ell ℓ the absolute Rayleigh quotient after a 32-step power iteration. From m = 128 m = 128 m = 128 Rademacher probes, one Hessian-vector product each, we form the Hutchinson estimates σ ^ 1 \hat{\sigma}_1 σ ^ 1 of tr G \operatorname{tr} G tr G and σ ^ 2 \hat{\sigma}_2 σ ^ 2 of tr ( G 2 ) = ∥ G ∥ F 2 \operatorname{tr}(G^2)=\lVert G\rVert_F^2 tr ( G 2 ) = ∥ G ∥ F 2 [26 ] , and set u 2 = σ ^ 2 / ( 1 − ρ ) u_2 = \hat{\sigma}_2/(1-\rho) u 2 = σ ^ 2 / ( 1 − ρ ) with ρ = 6 log ( 2 / δ ) / m ≈ 0.595 \rho= \sqrt{6\log(2/\delta)/m} \approx0.595 ρ = 6 log ( 2/ δ ) / m ≈ 0.595 and u 1 = σ ^ 1 + 2 u 2 / ( m δ ) u_1 = \hat{\sigma}_1 + \sqrt{2u_2/(m\delta)} u 1 = σ ^ 1 + 2 u 2 / ( m δ ) , where δ = 0.025 / 24 \delta= 0.025/24 δ = 0.025/24 . As in the proof of Theorem 1 of Roosta-Khorasani and Ascher [45 ] , Markov’s inequality, convexity and E e − t z 2 ≤ 1 − t + 3 t 2 / 2 \mathbb{E}e^{-tz^2} \le1-t+3t^2/2 E e − t z 2 ≤ 1 − t + 3 t 2 /2 for z = a T w z = a^\mathsf{T}w z = a T w with ∥ a ∥ = 1 \lVert a\rVert= 1 ∥ a ∥ = 1 and Rademacher w w w give P ( u 2 < ∥ G ∥ F 2 ) ≤ e m t ( 1 − ρ ) ( 1 − t + 3 t 2 / 2 ) m \mathbb{P}(u_2 < \lVert G\rVert_F^2) \le e^{mt(1-\rho)}(1-t+3t^2/2)^m P ( u 2 < ∥ G ∥ F 2 ) ≤ e m t ( 1 − ρ ) ( 1 − t + 3 t 2 /2 ) m , which is below 1.3 × 10 − 4 1.3 \times10^{-4} 1.3 × 1 0 − 4 at t = 0.22 t = 0.22 t = 0.22 . Each Rademacher sample of tr G \operatorname{tr} G tr G has variance at most 2 ∥ G ∥ F 2 2\lVert G\rVert_F^2 2 ∥ G ∥ F 2 , so by Chebyshev’s inequality u 2 ≥ ∥ G ∥ F 2 u_2 \ge\lVert G\rVert_F^2 u 2 ≥ ∥ G ∥ F 2 and u 1 < tr G u_1 < \operatorname{tr} G u 1 < tr G hold together with probability at most δ \delta δ . A union bound over the 24 points shows that, with probability at least 0.95, every plotted value bounds the ratio from above. These bounds range from 6.70 × 10 − 5 6.70 \times10^{-5} 6.70 × 1 0 − 5 to 9.19 × 10 − 5 9.19 \times10^{-5} 9.19 × 1 0 − 5 on SST-2 and from 6.83 × 10 − 5 6.83 \times10^{-5} 6.83 × 1 0 − 5 to 1.16 × 10 − 4 1.16 \times10^{-4} 1.16 × 1 0 − 4 on RTE, and decrease along every run.
Low-dimensional diagnostics. Comparisons on a related separable family at d ∈ { 16 , 64 , 128 } d \in\{16,64,128\} d ∈ { 16 , 64 , 128 } had E T > d ET > d E T > d in every cell, outside the regime where randomized costs are known to be of order E T ET E T . There, at the isotropic endpoint E = d E = d E = d the coordinate methods were faster, random directions were slower in most cells at d = 16 d = 16 d = 16 , and diffuse spectra gave heterogeneous results. Curvature measurements on OPT models use the ReLU activation, so their Hessian products concern one activation branch only.
Lean Formalization of the Deterministic Bounds
Theorem 3.2 , the deterministic part of Corollary 3.3, and the three-point upper bound of Theorem 3.1 on B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d are formalized, in the form stated below, in Lean 4 (version 4.19.0) with the Mathlib library (revision c44e0c8) and checked by the Lean kernel. The formalization is available at https://github.com/Pyx9876/zo-lower-bounds-lean and in the ancillary files of this arXiv entry. Its README lists the Lean name of each statement below. Beyond propositional extensionality, the axiom of choice, and the soundness of quotients, on which Mathlib rests, the proofs use no axioms or unproved assumptions.
Model. Queries, outputs, and costs follow Section 2. A call ( x , V ) (x,V) ( x , V ) returns ( f ( x ) , V T ∇ f ( x ) ) (f(x),V^{\mathsf{T}}\nabla f(x)) ( f ( x ) , V T ∇ f ( x )) at cost 1 + rank V 1+\operatorname{rank} V 1 + rank V , and a call with exact values only has V = 0 V=0 V = 0 . A deterministic algorithm maps each finite transcript to its next call or to an output point; the output is unrestricted and uncharged. N d e t N_{\mathrm{det}} N det is the smallest worst-case cost of an algorithm that returns an ε \varepsilon ε -stationary point on every function of the class, and randomized algorithms are jointly measurable in their seed and transcript. The class B L , Δ , E d \mathcal{B}^{d}_{L,\Delta,E} B L , Δ , E d is defined as in (2) : one fixed H ⪰ 0 H \succeq0 H ⪰ 0 with ∥ H ∥ o p ≤ L \lVert H\rVert_{\mathrm{op}} \le L ∥ H ∥ op ≤ L and tr H ≤ L E \operatorname{tr} H \le LE tr H ≤ L E dominates the derivative of ∇ f \nabla f ∇ f almost everywhere on { f < f ( 0 ) } \{f < f(0)\} { f < f ( 0 )} . Lemma B.2, which identifies this class with the segment form used in the proofs, is formalized as well.
Formalized statements. Let d ≥ 1 d \ge1 d ≥ 1 , L , Δ , ε > 0 L,\Delta,\varepsilon> 0 L , Δ , ε > 0 , 1 ≤ E ≤ d 1 \le E \le d 1 ≤ E ≤ d , and T ≥ 1 T \ge1 T ≥ 1 . For the value oracle and for the rank-priced oracle, N d e t ( B L , Δ , E d ) N_{\mathrm{det}}(\mathcal{B}^{d}_{L,\Delta,E}) N det ( B L , Δ , E d ) is finite and satisfies
d ≤ N d e t ≤ min { 10 d T , 100 d E T ( d + T ) } , d \le N_{\mathrm{det}} \le\min\left\{10dT,\ 100\sqrt{dET(d+T)}\right\}, d ≤ N det ≤ min { 10 d T , 100 d E T ( d + T ) } ,
E T ≤ d ⟹ d 2250 min { T , E T log ( e + T ) } ≤ N d e t ≤ 100 2 d E T , ET \le d \Longrightarrow\frac{d}{2250}\min\left\{T,\sqrt{\frac{ET}{\log(e+T)}}\right\} \le N_{\mathrm{det}} \le100\sqrt{2}d\sqrt{ET}, E T ≤ d ⟹ 2250 d min { T , log ( e + T ) E T } ≤ N det ≤ 100 2 d E T ,
E ≥ T log ( e + T ) ⟹ N d e t ≥ d T 2359296 , T log ( e + T ) ≤ E ≤ d T ⟹ N d e t ≥ d T 2250 . E \ge T\log(e+T) \Longrightarrow N_{\mathrm{det}} \ge\frac{dT}{2359296},\qquad T\log(e+T) \le E \le\frac{d}{T} \Longrightarrow N_{\mathrm{det}} \ge\frac{dT}{2250}. E ≥ T log ( e + T ) ⟹ N det ≥ 2359296 d T , T log ( e + T ) ≤ E ≤ T d ⟹ N det ≥ 2250 d T .
and the three-point method of Theorem B.8 gives N r a n d ( B L , Δ , E d ) ≤ 23 E T N_{\mathrm{rand}}(\mathcal{B}^{d}_{L,\Delta,E}) \le23ET N rand ( B L , Δ , E d ) ≤ 23 E T . These statements imply Theorem 3.2 , the deterministic part of Corollary 3.3, and the upper bound of Theorem 3.1 , whose constants are not specified. The formalized lower bounds use more conservative constants than Theorem A.1 (1 / 2250 1/2250 1/2250 in place of 1 / 672 1/672 1/672 ), including the constant 9 9 9 in place of 13 / 3 13/3 13/3 in the selection Lemma C.2 and cruder chain and wrapper bounds.
Not formalized. The randomized lower bound of Corollary F.4 , the lower bound of Theorem E.8 , the variant (7) without E E E or H H H , the three-point bound on A L , Δ , E d \mathcal{A}^{d}_{L,\Delta,E} A L , Δ , E d , the coercivity, nonconvexity, and Lipschitz properties of the hard functions, and the results of Appendices G and H, including the lower bounds on the global class F L , Δ , E d \mathcal{F}^{d}_{L,\Delta,E} F L , Δ , E d , are not formalized.