Introduction
Finite-sum minimax problems arise when a learning objective is optimized against an adversary or against a worst-case weighting of the training examples, with one component per example, as in adversarial training ([13 ] ), distributionally robust optimization ([15 ] ), and agnostic federated learning ([14 ] ). When the adversary’s objective is concave and carries a strongly convex penalty, the inner maximization
©2026 Yuxing Peng, Zhiqing Tang, and Weijia Jia.
License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. is strongly concave, and we obtain the nonconvex–strongly-concave (NC–SC) problem
min p ∈ R d p V ( p ) , V ( p ) = max y ∈ R d y F ( p , y ) , F = 1 n ∑ i = 1 n F i (1) \min_{p\in\mathbb{R}^{d_p}} V(p),\qquad V(p)=\max_{y\in\mathbb{R}^{d_y}} F(p,y),\qquad F=\frac{1}{n}\sum_{i=1}^{n}F_i \tag*{(1)} p ∈ R d p min V ( p ) , V ( p ) = y ∈ R d y max F ( p , y ) , F = n 1 i = 1 ∑ n F i (1)
where F F F is nonconvex in p p p , μ \mu μ -strongly concave in y y y , and the component gradients have joint mean-squared smoothness L L L . The condition number κ = L / μ \kappa=L/\mu κ = L / μ measures the difficulty of the inner maximization. The goal is a point with a small gradient of the value function V V V , also called the primal function [25 , 27 ] . Gradient descent ascent and its variance-reduced variants are standard methods for this problem [10 , 11 ] , and the oracle complexity of the problem is the number of component queries that the best method requires in the worst case.
Known lower bounds for finite-sum NC–SC problems cover linear-span algorithms [25 ] , a restricted proximal protocol [7 ] , and zero-respecting algorithms [27 ] . None of them covers arbitrary randomized adaptive algorithms, which may choose components and query points based on the full transcript and a random seed. An exact component proximal query also carries more information than a gradient. It is available at every legal parameter, for which the regularized component problem is globally convex–concave and has a saddle point, and it returns a global saddle point, which can expose hidden directions that gradients do not reveal. Whether unrestricted adaptivity or exact proximal access can lower the complexity has therefore remained open.
We prove that neither of them changes the polynomial complexity. Write S = L Δ / ε 2 S=L\Delta/\varepsilon^2 S = L Δ/ ε 2 and h ( n , κ ) = min { n κ , n 3 / 4 κ } h(n,\kappa)=\min\{\sqrt{n\kappa},n^{3/4}\sqrt{\kappa}\} h ( n , κ ) = min { nκ , n 3/4 κ } . On the class of § 2 , whose instances start from a point with primal gap at most Δ \Delta Δ and joint gradient norm at most 2 L Δ \sqrt{2L\Delta} 2 L Δ , our lower and upper bounds on the oracle complexity are
Ω ( ( n + h ( n , κ ) S ) ) and O ( ( n + h ( n , κ ) S ) log 2 ( 2 + κ ) ) , \Omega\bigl((n+h(n,\kappa)S)\bigr)\quad\text{and}\quad O\bigl((n+h(n,\kappa)S)\log^2(2+\kappa)\bigr), Ω ( ( n + h ( n , κ ) S ) ) and O ( ( n + h ( n , κ ) S ) log 2 ( 2 + κ ) ) ,
for n ≥ 2 n\ge2 n ≥ 2 , every κ ≥ 1 \kappa\ge1 κ ≥ 1 , and ε 2 \varepsilon^2 ε 2 below a universal multiple of L Δ L\Delta L Δ , so they match up to the factor log 2 ( 2 + κ ) \log^2(2+\kappa) log 2 ( 2 + κ ) . Both bounds hold for the expected cost and for a deterministic query cap with success probability $2/3 ({xref}Theorems 1 and 2), and for $\kappa\ge 2^{18}\sqrt{n}$ the lower bound also holds under the protocol of {citet}Han et al. [7 ] ({xref}Corollary 4`).
Where does the factor n 3 / 4 κ n^{3/4}\sqrt{\kappa} n 3/4 κ come from at large condition numbers? Our hard instances chain M ≍ S M\asymp S M ≍ S nonlinear primal stages through dual paths with m m m nodes, and each internal edge of a path is assigned independently to one uniformly random component, its owner. A query to another component only rules that component out as the owner, so an algorithm is unlikely to find the owner with far fewer than order n n n queries. For the mean F F F to carry an edge with weight a a a , the owner must carry it with weight n a na na , which adds a term of order n − 1 ( n a ) 2 = n a \sqrt{n^{-1}(na)^2}=\sqrt{n}a n − 1 ( na ) 2 = n a to the mean-squared smoothness constant and limits a a a to at most order L / n L/\sqrt{n} L / n ; we take a a a of this order. A path with m m m nodes and edge weight a a a preserves the amplitude of its input from one end to the other up to a constant factor when its strong concavity is of order a / m 2 a/m^2 a / m 2 , and this strong concavity must be at least μ \mu μ . Paths can therefore have m ≍ a / μ ≍ κ / n 1 / 4 m\asymp\sqrt{a/\mu}\asymp\sqrt{\kappa}/n^{1/4} m ≍ a / μ ≍ κ / n 1/4 nodes. Each of the M ( m − 1 ) M(m-1) M ( m − 1 ) owner searches costs at least order n n n queries in expectation, so the total expected cost is at least of order
n ⋅ m ⋅ M ≍ n 3 / 4 κ L Δ ε 2 . n\cdot m\cdot M\asymp n^{3/4}\sqrt{\kappa}\,\frac{L\Delta}{\varepsilon^2}. n ⋅ m ⋅ M ≍ n 3/4 κ ε 2 L Δ .
Exact proximal responses threaten this count in two ways. If the legal parameters depended on the component, testing legality alone would reveal owners; and since a response is a global saddle point, one query could reveal edges far beyond the current one. A public coordinate, with the same curvature in every component, makes the legal parameters identical for all components. A reflection symmetry confines the saddle point of a censored problem, in which the unrevealed hidden directions are set to zero, to the spans of the current query centers and of the blocks already revealed. Edge interactions vanish while the hidden coordinates are small, so with high probability this saddle point is the actual response, and a query to a component reveals only its run of consecutive edges and stops at the first edge with another owner.
Concurrent work by Zhou [27 ] establishes the same polynomial rate for randomized zero-respecting incremental first-order methods, starting from an exact dual maximizer y ∗ ( 0 ) = 0 y^{*}(0)=0 y ∗ ( 0 ) = 0 . Our lower bound removes the zero-respecting restriction, adds exact values and proximal responses, and also holds under that start (§ 2.2 ). Our upper bound starts instead from a joint-gradient bound and pays for the resulting initialization. Wu et al. [22 ] prove lower bounds for arbitrary randomized algorithms in the related finite-sum nonconvex–concave (NC–C) problem over a bounded dual domain, with values and gradients only. Table 1 compares these bounds, and § 2.2 compares our assumptions with those of prior work.
Table 1. Bounds on component queries for finite-sum NC–SC problems with joint mean-squared smoothness, for small ε \varepsilon ε and every κ ≥ 1 \kappa\ge1 κ ≥ 1 unless stated otherwise; h = min { n κ , n 3 / 4 κ } h=\min\{\sqrt{n\kappa},n^{3/4}\sqrt{\kappa}\} h = min { nκ , n 3/4 κ } and S = L Δ / ε 2 S=L\Delta/\varepsilon^{2} S = L Δ/ ε 2 . Shaded: this paper. The first row combines Zhang et al. [25 ] , for linear-span methods, and Han et al. [7 ] , for their restricted proximal protocol with primal nonconvexity parameter L L L ; the second is the concurrent lower bound of Zhou [27 ] , whose Theorem 1 also gives a matching upper bound up to logarithmic factors from an exact dual maximizer. A dash marks a combination not treated separately. Our lower bound Ω ( n + h S ) \Omega(n+hS) Ω ( n + h S ) covers the largest algorithm class and the stronger oracle; a method using only component gradients attains this rate up to the factor log 2 ( 2 + κ ) \log^{2}(2+\kappa) log 2 ( 2 + κ ) .
Models and Main Results
Our lower and upper bounds are stated for the same class of finite sums and the same cost model. They differ in the oracle: the lower bounds allow exact proximal responses, whereas our upper bounds use only component gradients.
Strongly Concave Finite Sums and Exact Indexed Oracles
Write [ n ] = { 1 , … , n } [n]=\{1,\ldots,n\} [ n ] = { 1 , … , n } , w = ( p , y ) w=(p,y) w = ( p , y ) , and S = L Δ / ε 2 S=L\Delta/\varepsilon^{2} S = L Δ/ ε 2 . With the dual space Y = R d y \mathcal{Y}=\mathbb{R}^{d_y} Y = R d y , the class F ( n , L , μ , Δ ) \mathcal{F}(n,L,\mu,\Delta) F ( n , L , μ , Δ ) consists of the finite sums F = 1 n ∑ i = 1 n F i F=\frac{1}{n}\sum_{i=1}^{n}F_i F = n 1 ∑ i = 1 n F i with continuously differentiable components that satisfy, globally,
1 n ∑ i ∥ ∇ F i ( w ) − ∇ F i ( w ′ ) ∥ 2 ≤ L 2 ∥ w − w ′ ∥ 2 , (2) \frac{1}{n}\sum_i\left\|\nabla F_i(w)-\nabla F_i(w')\right\|^2\le L^2\left\|w-w'\right\|^2, \tag*{(2)} n 1 i ∑ ∥ ∇ F i ( w ) − ∇ F i ( w ′ ) ∥ 2 ≤ L 2 ∥ w − w ′ ∥ 2 , (2)
F ( p , ⋅ ) is μ -strongly concave , (3) F(p,\cdot)\text{ is }\mu\text{-strongly concave}, \tag*{(3)} F ( p , ⋅ ) is μ -strongly concave , (3)
V ( 0 ) − inf p V ( p ) ≤ Δ , (4) V(0)-\inf_p V(p)\le\Delta, \tag*{(4)} V ( 0 ) − p inf V ( p ) ≤ Δ , (4)
∥ ∇ F ( 0 , 0 ) ∥ 2 ≤ 2 L Δ . (5) \left\|\nabla F(0,0)\right\|^2\le2L\Delta. \tag*{(5)} ∥ ∇ F ( 0 , 0 ) ∥ 2 ≤ 2 L Δ. (5)
All norms are Euclidean; for matrices and multilinear maps, ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ is the induced operator norm. We call condition (2) joint mean-squared smoothness, or root-mean-square (RMS) smoothness for short. It bounds only an average over the components: it implies that the mean F F F is L L L -smooth and that each component is at most n L \sqrt{n}L n L -smooth. By the strong concavity (3), for every p p p the function F ( p , ⋅ ) F(p,\cdot) F ( p , ⋅ ) has a unique maximizer y ∗ ( p ) y^*(p) y ∗ ( p ) , and ∇ V ( p ) = ∇ p F ( p , y ∗ ( p ) ) \nabla V(p)=\nabla_pF(p,y^*(p)) ∇ V ( p ) = ∇ p F ( p , y ∗ ( p )) . We call conditions (4) and (5) together a normalized start. They describe the starting point by its primal gap and its joint gradient, without requiring that its dual part be an exact maximizer. We write G 0 = sup y F ( 0 , y ) − inf p F ( p , 0 ) G_0=\sup_y F(0,y)-\inf_p F(p,0) G 0 = sup y F ( 0 , y ) − inf p F ( p , 0 ) for the initial primal–dual gap; it differs from the primal gap in (4) , and the class places no bound on it. For a component index i i i and a point w w w , an exact indexed first-order query returns
O ( i , w ) = ( F i ( w ) , ∇ p F i ( w ) , ∇ y F i ( w ) ) . (6) \mathcal{O}(i,w)=(F_i(w),\nabla_pF_i(w),\nabla_yF_i(w)). \tag*{(6)} O ( i , w ) = ( F i ( w ) , ∇ p F i ( w ) , ∇ y F i ( w )) . (6)
The component saddle-proximal oracle performs a proximal point step [18 ] on one component. For a parameter γ > 0 \gamma>0 γ > 0 , it returns, together with the first-order information (6), a saddle pair of
H i , γ ( p , y ) = F i ( p , y ) + ∥ p − p ˉ ∥ 2 − ∥ y − y ˉ ∥ 2 2 γ (7) H_{i,\gamma}(p,y)=F_i(p,y)+\frac{\lVert p-\bar{p}\rVert^2-\lVert y-\bar{y}\rVert^2}{2\gamma} \tag*{(7)} H i , γ ( p , y ) = F i ( p , y ) + 2 γ ∥ p − p ˉ ∥ 2 − ∥ y − y ˉ ∥ 2 (7)
where ( p ˉ , y ˉ ) (\bar{p},\bar{y}) ( p ˉ , y ˉ ) is the queried center. Since a component may be nonconvex in p p p , the regularized problem (7) need not be convex–concave for every γ \gamma γ , so the oracle answers only at legal parameters. As in Equation (2) and Remark 12(iv) of Han et al. [7 ] , a finite γ > 0 \gamma>0 γ > 0 is legal when H i , γ H_{i,\gamma} H i , γ is globally convex in p p p and concave in y y y and has a saddle point. A saddle pair ( p + , y + ) (p^+,y^+) ( p + , y + ) satisfies H i , γ ( p + , y ) ≤ H i , γ ( p + , y + ) ≤ H i , γ ( p , y + ) H_{i,\gamma}(p^+,y)\le H_{i,\gamma}(p^+,y^+)\le H_{i,\gamma}(p,y^+) H i , γ ( p + , y ) ≤ H i , γ ( p + , y + ) ≤ H i , γ ( p , y + ) for all p p p and y y y .
One query to either oracle costs one unit, a full gradient costs n n n units, and Q Q Q denotes the total cost of a run. At each query, the algorithm chooses the component index, a finite query point and, for a proximal query, a legal parameter based on the entire exact transcript and a private random seed. Its computation and its primal output are unrestricted, and it may stop at a random time. Apart from the oracle answers, the algorithm is given only the dimensions, the target accuracy ε \varepsilon ε , and the class parameters, which are the same for every instance of the class.
Theorem 1 (Unrestricted lower bound for every condition number). There are universal constants c ⋆ ∈ ( 0 , 1 / 2 ] c_{\star} \in(0,1/2] c ⋆ ∈ ( 0 , 1/2 ] and c > 0 c > 0 c > 0 such that, for n ≥ 2 n \ge2 n ≥ 2 , L , μ , Δ > 0 L,\mu,\Delta> 0 L , μ , Δ > 0 , κ = L / μ ≥ 1 \kappa= L/\mu\ge1 κ = L / μ ≥ 1 , and
0 < ε 2 ≤ 2 − 52 c ⋆ 2 L Δ , (8) 0 < \varepsilon^2 \le2^{-52}c_{\star}^2L\Delta, \tag*{(8)} 0 < ε 2 ≤ 2 − 52 c ⋆ 2 L Δ , (8) every algorithm using either exact indexed oracle above, terminating almost surely and guaranteeing E ∥ ∇ V ( p o u t ) ∥ ≤ ε \mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert\le\varepsilon E ∥ ∇ V ( p out )∥ ≤ ε uniformly on F ( n , L , μ , Δ ) \mathcal{F}(n,L,\mu,\Delta) F ( n , L , μ , Δ ) has worst-case expected cost at least
c ( n + min { n κ , n 3 / 4 κ } L Δ ε 2 ) . (9) c\left(n+\min\left\{\sqrt{n}\kappa,n^{3/4}\sqrt{\kappa}\right\}\frac{L\Delta}{\varepsilon^2}\right). \tag*{(9)} c ( n + min { n κ , n 3/4 κ } ε 2 L Δ ) . (9) A lower bound of the same order holds for success probability 2 / 3 2/3 2/3 under a deterministic query cap. The hard instances have polynomial dimension, C 1 , 1 C^{1,1} C 1 , 1 components that are individually μ \mu μ -strongly concave in y y y , y ⋆ ( 0 ) = 0 y^{\star}(0)=0 y ⋆ ( 0 ) = 0 , finite initial primal–dual gap G 0 G_0 G 0 , and a unique response at every legal proximal parameter. Their exact legal domain is common to all components, all hidden realizations and all proximal centers.
The accuracy condition (8) asks only that ε 2 / ( L Δ ) \varepsilon^2/(L\Delta) ε 2 / ( L Δ ) be below a universal constant. We prove Theorem 1 in Appendix D, where we obtain c = c ⋆ 2 / 2 96 c=c_{\star}^2/2^{96} c = c ⋆ 2 / 2 96 .
Theorem 2 (Upper bound for all condition numbers and matching polynomial rate). For n ≥ 2 n \ge2 n ≥ 2 , κ ≥ 1 \kappa\ge1 κ ≥ 1 and S = L Δ / ε 2 ≥ 2 S=L\Delta/\varepsilon^2 \ge2 S = L Δ/ ε 2 ≥ 2 , there is an indexed component-gradient algorithm on F ( n , L , μ , Δ ) \mathcal{F}(n,L,\mu,\Delta) F ( n , L , μ , Δ ) that terminates almost surely and satisfies
E ∥ ∇ V ( p o u t ) ∥ 2 ≤ ε 2 , (10) \mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert^2 \le\varepsilon^2, \tag*{(10)} E ∥ ∇ V ( p out ) ∥ 2 ≤ ε 2 , (10) E Q ≤ C [ n + h ( n , κ ) ( 1 + S ) ] log 2 ( 2 + κ ) , (11) \mathbb{E}Q \le C\left[n+h(n,\kappa)(1+S)\right]\log^2(2+\kappa), \tag*{(11)} E Q ≤ C [ n + h ( n , κ ) ( 1 + S ) ] log 2 ( 2 + κ ) , (11) h ( n , κ ) = min { n κ , n 3 / 4 κ } . h(n,\kappa)=\min\left\{\sqrt{n}\kappa,n^{3/4}\sqrt{\kappa}\right\}. h ( n , κ ) = min { n κ , n 3/4 κ } . Here C > 0 C>0 C > 0 is universal, and all initialization, snapshots and final refinement calls are included. The same order holds for a deterministic query cap with success probability 2 / 3 2/3 2/3 . Under (8) , we have S ≥ 2 S \ge2 S ≥ 2 and E ∥ ∇ V ( p o u t ) ∥ ≤ ( E ∥ ∇ V ( p o u t ) ∥ 2 ) 1 / 2 ≤ ε \mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert\le\left(\mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert^2\right)^{1/2} \le\varepsilon E ∥ ∇ V ( p out )∥ ≤ ( E ∥ ∇ V ( p out ) ∥ 2 ) 1/2 ≤ ε , so this upper bound is at most a constant multiple of log 2 ( 2 + κ ) \log^2(2+\kappa) log 2 ( 2 + κ ) times the lower bound of Theorem 1 .
We prove Theorem 2 in Appendix § 11 .8 by combining two methods: the method of Appendix § 11 for κ ≤ 2 18 n \kappa\le2^{18}\sqrt{n} κ ≤ 2 18 n , and the method of the next theorem for larger κ \kappa κ . We state the regime of large condition numbers separately because there the construction with long paths gives a larger constant c c c , the bound G 0 ≤ 1025 Δ G_0 \le1025\Delta G 0 ≤ 1025Δ , the dimension estimate (16), and a single hard instance for all early horizons under the protocol of Han et al. [7 ] .
Theorem 3 (Special case of large condition numbers and matching upper bound). There are universal constants c ⋆ ∈ ( 0 , 1 / 2 ] c_{\star} \in(0,1/2] c ⋆ ∈ ( 0 , 1/2 ] and c , C > 0 c,C>0 c , C > 0 such that, for n ≥ 2 n \ge2 n ≥ 2 , L , μ , Δ > 0 L,\mu,\Delta>0 L , μ , Δ > 0 , and
κ = L / μ ≥ 2 18 n , 0 < ε 2 ≤ 2 − 52 c ⋆ 2 L Δ , (12) \kappa=L/\mu\ge2^{18}\sqrt{n}, \qquad0<\varepsilon^2\le2^{-52}c_{\star}^2L\Delta, \tag*{(12)} κ = L / μ ≥ 2 18 n , 0 < ε 2 ≤ 2 − 52 c ⋆ 2 L Δ , (12)
every algorithm using either oracle above, terminating almost surely and guaranteeing E ∥ ∇ V ( p o u t ) ∥ ≤ ε \mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert\le\varepsilon E ∥ ∇ V ( p out )∥ ≤ ε uniformly on F ( n , L , μ , Δ ) \mathcal{F}(n,L,\mu,\Delta) F ( n , L , μ , Δ ) , has worst-case expected query cost at least
c n 3 / 4 κ S . (13) cn^{3/4}\sqrt{\kappa}\,S. \tag*{(13)} c n 3/4 κ S . (13)
For each algorithm, the hard instance that witnesses this bound can be chosen with C 1 , 1 C^{1,1} C 1 , 1 components, each μ \mu μ -strongly concave in y y y , and with y ∗ ( 0 ) = 0 y^*(0)=0 y ∗ ( 0 ) = 0 and G 0 ≤ 1025 Δ G_0\le1025\Delta G 0 ≤ 1025Δ .
For every ε > 0 \varepsilon>0 ε > 0 and κ ≥ 2 18 n \kappa\ge2^{18}\sqrt{n} κ ≥ 2 18 n , without the accuracy condition in Equation (12) , our method of Appendix F, which combines existing first-order methods, achieves
E ∥ ∇ V ( p o u t ) ∥ 2 ≤ ε 2 , (14) \mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert^2\le\varepsilon^2, \tag*{(14)} E ∥ ∇ V ( p out ) ∥ 2 ≤ ε 2 , (14)
E Q ≤ C n 3 / 4 κ ( 1 + S ) log 2 ( 2 + κ ) . (15) \mathbb{E}Q\le Cn^{3/4}\sqrt{\kappa}(1+S)\log^2(2+\kappa). \tag*{(15)} E Q ≤ C n 3/4 κ ( 1 + S ) log 2 ( 2 + κ ) . (15)
In the respective regimes of the lower and upper bounds, the same orders hold for success probability 2 / 3 2/3 2/3 under a deterministic query cap.
Our proof gives c = 133 c ⋆ 2 / ( 3 ⋅ 2 73 ) c=133c_\star^2/(3\cdot2^{73}) c = 133 c ⋆ 2 / ( 3 ⋅ 2 73 ) , where c ⋆ c_\star c ⋆ is determined by the minima of continuous residuals over three fixed-dimensional compact sets (Lemma 10). The primal and dual dimensions d p d_p d p and d y d_y d y of the witnessing hard instances are fixed before the algorithm runs and satisfy
d p + d y = O ( ( S n 3 / 4 κ + S 2 κ 3 n − 3 / 2 ) log ( n κ S ) ) . (16) d_p+d_y=O\left(\left(Sn^{3/4}\sqrt{\kappa}+S^2\kappa^3n^{-3/2}\right)\log(n\kappa S)\right). \tag*{(16)} d p + d y = O ( ( S n 3/4 κ + S 2 κ 3 n − 3/2 ) log ( nκ S ) ) . (16)
Corollary 4 (The Han et al. proximal protocol). In the regime of Theorem 3 , a lower bound of the same order also holds for the proximal incremental first-order oracle (PIFO) algorithms of Han et al. [7 ] , with their complexity sup F inf t \sup_F\inf_t sup F inf t of Definition 13 measured by the expected gradient norm.
In the proof of Corollary 4 (Appendix C.§ 2 ), one fixed hard instance serves every early output horizon, and the cost counts both component calls and full-gradient refreshes. The upper bound in Theorem 3 is stated in the indexed oracle model.
Comparison with Prior Assumptions
Smoothness. The joint mean-squared smoothness condition (2) is Assumption 1 of Zhou [27 ] and the smoothness condition of the finite-sum lower bounds of Zhang et al. [25 ] and Han et al. [7 ] . It is weaker than L L L -smoothness of every component, which Wu et al. [22 ] use for their lower bounds, and stronger than L L L -smoothness of the mean alone. Our upper bounds need only (2). The components of our hard instances use the extra freedom that (2) allows: their individual smoothness constants grow proportionally to n L \sqrt{n}\,L n L .
Dual variable. As in Zhou [27 ] , the dual space is R d y \mathbb{R}^{d_y} R d y and the mean is μ \mu μ -strongly concave; the components of our hard instances are even individually μ \mu μ -strongly concave. Wu et al. [22 ] instead use a known compact convex dual set of diameter at most D y D_y D y with a concave mean, and they measure stationarity through the Moreau envelope of the value function.
Initialization. The normalized start (4)–(5) bounds the primal gap and the joint gradient at the origin. Zhou [27 ] assumes instead a bound on the primal gap and an exact dual maximizer y ∗ ( 0 ) = 0 y^*(0)=0 y ∗ ( 0 ) = 0 , and Wu et al. [22 ] assume only a bound on the primal gap; in their setting, the bounded dual domain controls the dual side. Neither our start nor the exact dual start of Zhou [27 ] implies the other: y ∗ ( 0 ) = 0 y^*(0)=0 y ∗ ( 0 ) = 0 does not imply the bound (5) , and a bounded joint gradient does not give y ∗ ( 0 ) = 0 y^*(0)=0 y ∗ ( 0 ) = 0 . Our hard instances satisfy both conditions, so Theorem 1 also holds under the start of Zhou [27 ] . Every hard instance also has a finite initial primal–dual gap (Appendix D.§ 4 ).
Accuracy and cost. Theorem 1 bounds the expected cost of algorithms with E ∥ ∇ V ( p o u t ) ∥ ≤ ε \mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert\leq\varepsilon E ∥ ∇ V ( p out )∥ ≤ ε , and the deterministic cap needed for confidence 2 / 3 2/3 2/3 ; Theorem 2 guarantees E ∥ ∇ V ( p o u t ) ∥ 2 ≤ ε 2 \mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert^2\leq\varepsilon^2 E ∥ ∇ V ( p out ) ∥ 2 ≤ ε 2 . Zhou [27 ] characterizes expected query counts under E ∥ ∇ V ∥ 2 ≤ ε 2 \mathbb{E}\lVert\nabla V\rVert^2\leq\varepsilon^2 E ∥ ∇ V ∥ 2 ≤ ε 2 , with a fixed budget for κ ≤ n \kappa\leq\sqrt{n} κ ≤ n , and that lower bound also covers the criterion E ∥ ∇ V ( p o u t ) ∥ ≤ ε \mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert\leq\varepsilon E ∥ ∇ V ( p out )∥ ≤ ε and success probability 2 / 3 2/3 2/3 under a fixed budget; Wu et al. [22 ] use fixed budgets and confidence 2 / 3 2/3 2/3 . The remaining gap between the upper and lower bounds is log 2 ( 2 + κ ) \log^2(2+\kappa) log 2 ( 2 + κ ) here and logarithmic for Zhou [27 ] ; for Wu et al. [22 ] there is no gap when L D y / ε ≤ n 1 / 4 LD_y/\varepsilon\leq n^{1/4} L D y / ε ≤ n 1/4 and a single logarithmic factor otherwise.
Technical Overview
Our lower bounds hide a chain of nonlinear primal stages that an algorithm must uncover one at a time, and for κ ≥ 64 \kappa\geq64 κ ≥ 64 the hard instances link consecutive stages through dual paths whose edges belong to random components. We first describe the hard instances for large condition numbers and then explain how they stay hidden from arbitrary queries and from exact proximal responses. The symbols of this section are local to the lower bounds.
Dual Paths with Component Assignments
As described in the introduction, each primal stage passes its progress to the next through a dual path with m m m nodes, and each internal edge of a path belongs to one uniformly random component, its owner, which carries the edge interaction with weight n a na na ; the edge weight a a a is of order L / n L/\sqrt{n} L / n . When the flat zones introduced below have zero width and the cutoffs are inactive, the quadratic dual term of the mean is − a 2 t ⊤ Q t -\frac{a}{2}t^\top Q t − 2 a t ⊤ Qt in the hidden coordinates t t t of a path, with Q = P m + m − 2 I Q=P_m+m^{-2}I Q = P m + m − 2 I , where P m P_m P m is the Laplacian of the path graph on m m m vertices. The ridge m − 2 m^{-2} m − 2 supplies the strong concavity a / m 2 ≥ μ a/m^2\geq\mu a / m 2 ≥ μ , and Lemma 5 gives the endpoint entries u = ( Q − 1 ) 11 ≍ m u=(Q^{-1})_{11}\asymp m u = ( Q − 1 ) 11 ≍ m and v = ( Q − 1 ) 1 m ≍ m v=(Q^{-1})_{1m}\asymp m v = ( Q − 1 ) 1 m ≍ m of the Green matrix, with 1 < u / v ≤ cosh 1 1<u/v\leq\cosh1 1 < u / v ≤ cosh 1 . A path with m ≍ a / μ ≍ κ / n 1 / 4 m\asymp\sqrt{a/\mu}\asymp\sqrt{\kappa}/n^{1/4} m ≍ a / μ ≍ κ / n 1/4 nodes therefore passes the forcing at one endpoint to the other up to a constant factor. Figure 1 shows how queries interact with the owners, and Appendix A fixes the constants in Equations 23 and 24.
Every node of every path has an independent hidden axis, and the owners are independent of the axes. The primal variable P = ( P 1 , … , P M ) P=(P_1,\ldots,P_M) P = ( P 1 , … , P M ) has one block P j = ( X j , Y j + 1 ) P_j=(X_j,Y_{j+1}) P j = ( X j , Y j + 1 ) per stage, and public identity maps pairs its two subblocks with the first and the last node of path j j j . Each component has the form
f i , η ( P , y ) = P ( P ) − ν 2 ∥ y ∥ 2 + b χ X ( Y ∂ ) ⟨ A ( P ) , Y ∂ ⟩ + c ∑ v χ X ( y v ) φ η ( t v ) − n a ∑ e : Σ e = i A η ( Y e ) + ( public radial terms ) ; f_{i,\eta}(P,y)=\mathcal{P}(P)-\frac{\nu}{2}\lVert y\rVert^2+b\chi_{\mathcal{X}}(Y_{\partial})\langle\mathcal{A}(P),Y_{\partial}\rangle+c\sum_v\chi_{\mathcal{X}}(y_v)\varphi_\eta(t_v)
-na\sum_{e:\,\Sigma_e=i}\mathcal{A}_\eta(Y_e)+(\text{public radial terms}); f i , η ( P , y ) = P ( P ) − 2 ν ∥ y ∥ 2 + b χ X ( Y ∂ ) ⟨ A ( P ) , Y ∂ ⟩ + c v ∑ χ X ( y v ) φ η ( t v ) − na e : Σ e = i ∑ A η ( Y e ) + ( public radial terms ) ;
see Equation (45) . Here P \mathcal{P} P collects a scaled source chain built from the scalar functions of Carmon et al. [4 ] , the compensation − d ∥ A ∥ 2 -d\lVert\mathcal{A}\rVert^2 − d ∥ A ∥ 2 , and a radial penalty. The forcing A ( P ) \mathcal{A}(P) A ( P ) stacks the gated and capped primal blocks and acts on the concatenated path endpoints Y ∂ Y_{\partial} Y ∂ . Each dual node v v v has a hidden coordinate t v t_v t v along its axis, ϕ η ( t ) = 1 2 ( ∣ t ∣ − η ) + 2 \phi_{\eta}(t)=\frac{1}{2}(|t|-\eta)_{+}^{2} ϕ η ( t ) = 2 1 ( ∣ t ∣ − η ) + 2 vanishes on the flat zone ∣ t ∣ ≤ η |t|\le\eta ∣ t ∣ ≤ η , and χ ∗ \chi_{*} χ ∗ is a cutoff. The owned term A η ( Y e ) = 1 2 ( ∣ t e ∣ − h η ( ∥ Y e ∥ ) ) + 2 A_{\eta}(Y_e)=\frac{1}{2}(|t_e|-h_{\eta}(\lVert Y_e\rVert))_{+}^{2} A η ( Y e ) = 2 1 ( ∣ t e ∣ − h η (∥ Y e ∥) ) + 2 acts on the difference t e t_e t e of the two hidden coordinates of an edge e e e with endpoint pair Y e Y_e Y e , and only the owner Σ e \Sigma_e Σ e of e e e contains it. The width η ≥ 0 \eta\ge0 η ≥ 0 of the flat zones is fixed in Appendix § 8 .§ 2 , and a , b , c , d , ν > 0 a,b,c,d,\nu>0 a , b , c , d , ν > 0 are constants.
Why do the paths not weaken the primal chain? With the primal stage scale λ \lambda λ of Equation (24) , we calibrate the ridge ν = a / u \nu=a/u ν = a / u , the coupling b 2 = 12 a λ / v b^{2}=12a\lambda/v b 2 = 12 aλ / v , and the compensation d = 6 λ ( u / v − 1 ) d=6\lambda(u/v-1) d = 6 λ ( u / v − 1 ) . At zero width, eliminating the dual variable then contributes, for each stage j j j , the squared gate h j 2 h_j^{2} h j 2 of that stage times the completed quadratic (Lemma 7 )
6 λ ( x j − z j + 1 / 2 ) 2 + 6 λ ∥ A j , ⊥ ∥ 2 + 3 2 λ ∥ B j + 1 , ⊥ ∥ 2 , (17) 6\lambda(x_j-z_{j+1/2})^{2}+6\lambda\lVert A_{j,\perp}\rVert^{2}+\frac{3}{2}\lambda\lVert B_{j+1,\perp}\rVert^{2}, \tag*{(17)} 6 λ ( x j − z j + 1/2 ) 2 + 6 λ ∥ A j , ⊥ ∥ 2 + 2 3 λ ∥ B j + 1 , ⊥ ∥ 2 , (17)
where x j x_j x j and z j + 1 z_{j+1} z j + 1 are the coordinates of the capped subblocks of P j P_j P j along the first and the last hidden axis of path j j j , and A j , ⊥ A_{j,\perp} A j , ⊥ and B j + 1 , ⊥ B_{j+1,\perp} B j + 1 , ⊥ are their orthogonal parts. Dual elimination thus preserves the same normalized primal quadratic, including in the orthogonal directions that unrestricted outputs can use. The path and source ingredients follow Li et al. [8 ] and Carmon et al. [4 ] .
Global Geometry and Exact Hiding
Against zero-respecting algorithms ([27 ] ), every query lies in the span of the coordinates that earlier gradients have touched, so the unexposed coordinates are exactly zero at every query. An arbitrary algorithm can instead query points with nonzero projections onto unexposed hidden axes and with arbitrary norms, and an exact proximal response is a global saddle point. What keeps the hidden structure invisible in this case?
Smooth radial maps (cf. [4 Section 5.2] ) compress the primal blocks to bounded images, so the forcing and its derivatives stay bounded at every query (Lemma 6 ). Dual cutoffs with bounded support control the primal Hessian of the coupling. An owned edge interaction vanishes on its flat zone (cf. [20 Appendix C] ), so a small projection onto unexposed axes leaves the edge invisible. A flat zone of fixed width would not hide an edge from queries of large norm, so the width grows with the radius. r = ∥ Y ℓ ∥ r = \lVert Y_{\ell}\rVert r = ∥ Y ℓ ∥ of the edge pair beyond a localization radius R ∗ R_{\ast} R ∗ :
h η ( r ) = η ϱ ( r / R ∗ ) , (18) h_{\eta}(r) = \eta\varrho(r/R_{\ast}), \tag*{(18)} h η ( r ) = η ϱ ( r / R ∗ ) , (18)
ϱ ( u ) = { 1 , u ≤ 1 , 1 + ( u − 1 ) 2 / 2 , 1 < u < 2 , u − 1 / 2 , u ≥ 2. (19) \varrho(u) =
\begin{cases}
1, & u \le1, \\
1 + (u - 1)^2/2, & 1 < u < 2, \\
u - 1/2, & u \ge2.
\end{cases} \tag*{(19)} ϱ ( u ) = ⎩ ⎨ ⎧ 1 , 1 + ( u − 1 ) 2 /2 , u − 1/2 , u ≤ 1 , 1 < u < 2 , u ≥ 2. (19)
Since ϱ ( u ) ≥ u / 2 \varrho(u) \ge u/2 ϱ ( u ) ≥ u /2 , flatness holds on a cone at every radius.
These devices perturb the instance, and the perturbation must not destroy the terminal barrier. Public radial penalties, common to all components, dominate the positive curvature of the cutoffs without revealing owners. For every primal point, the dual maximizer of the mean without cutoffs and radial corrections has every relevant node, edge pair, and endpoint tuple strictly inside radius 3 R ∗ / 16 3R_{\ast}/16 3 R ∗ /16 . There the cutoffs equal one and the radial corrections vanish, so strong concavity makes it the maximizer of the actual mean as well. Using the radial calculus of Lemma 8 , Lemma 9 shows that the components are smooth and componentwise strongly concave, and Lemma 11 completes class membership. With the primal length scale s = 8 ε / ( c ⋆ λ ) s = 8\varepsilon/(c_{\star}\lambda) s = 8 ε / ( c ⋆ λ ) , the barrier of the zero-width chain on the last stage coordinate z M + 1 z_{M+1} z M + 1 survives the perturbation (Lemmas 10 and 11 ):
∣ z M + 1 ∣ ≤ s / 8 ⟹ ∥ ∇ V ( P ) ∥ ≥ 6 ε . \lvert z_{M+1}\rvert\le s/8 \quad\Longrightarrow\quad\lVert\nabla V(P)\rVert\ge6\varepsilon. ∣ z M + 1 ∣ ≤ s /8 ⟹ ∥ ∇ V ( P )∥ ≥ 6 ε .
A point whose last stage coordinate is this small is therefore far from ε \varepsilon ε -stationary.
Exact Oracle Responses and Component Search
We now address the two threats from exact proximal responses described in the introduction: a legal domain that depends on the component, and a global saddle point that reveals edges far beyond the current one.
Against the first threat, we add one public primal coordinate ξ \xi ξ with the common term ( L / 8 ) ( 1 − cos ξ ) (L/8)(1 - \cos\xi) ( L /8 ) ( 1 − cos ξ ) to every component. This term preserves all initial budgets and gradient barriers. Its curvature reaches − L / 8 -L/8 − L /8 , while the hidden part of every component has primal curvature below L / 32 L/32 L /32 (Lemma 9), so the public term alone decides legality: the exact legal domain is τ ≥ L / 8 \tau\ge L/8 τ ≥ L /8 for the inverse step τ = 1 / γ \tau= 1/\gamma τ = 1/ γ , the same for every component (Lemma 12 ). In the regularized problem the objective is strongly convex–concave in ( P , y ) (P,y) ( P , y ) , and the separate scalar problem in ξ \xi ξ is strictly convex even at the boundary.
Against the second threat, consider a legal call to a component i i i . It can reveal the whole run of consecutive edges that i i i owns (Figure 1 ), and we show that, with high probability, it reveals nothing beyond that run except that the next edge, if any, is not owned by i i i . Beyond the run, we set the unrevealed hidden axes to zero in every term and keep all public terms; this censored problem on the full space has a unique saddle point. We reflect the unrevealed blocks through the spans of their current query centers, using matching reflections for paired primal and dual endpoints. Invariance and uniqueness then place the unrevealed blocks of the censored saddle point in these spans, each of dimension at most two: one for an interior node, and two at an endpoint, whose span also contains the paired primal center. This step uses no property of the unrevealed axes; their random directions enter only in the next step. With high probability, the unrevealed axes have small projections onto these spans, so the actual and the censored problems have the same values and full gradients wherever the unrevealed blocks lie in these spans, and the censored saddle point is the actual response (Lemma 13 ). A separate projection argument handles the arbitrary final output.
Figure 1. A dual path. In the ideal exposure process of Appendix C.§ 1 , a query to component i i i reveals the entire run of consecutive edges owned by i i i and stops at the first edge with another owner; the query reveals only that i i i does not own that edge, and no later owner or axis. There are of order M m Mm M m serial searches, and each costs at least order n n n queries in expectation.
Locating the owners is a sequence of E = M ( m − 1 ) E=M(m-1) E = M ( m − 1 ) serial searches. Given the history of these searches, the owner of the first unrevealed edge is uniform over the components that have not yet been excluded. A nonnegative posterior potential starts at E ( n − 1 ) / 2 E(n-1)/2 E ( n − 1 ) /2 and decreases in conditional expectation by at most one per call, including calls that reveal several consecutive edges (Lemma 14 ). Together with the terminal barrier, this bound on completing the searches shows that, even under random stopping, every algorithm with E ∥ ∇ V ( p o u t ) ∥ ≤ ε \mathbb{E}\lVert\nabla V(p_{\mathrm{out}})\rVert\le\varepsilon E ∥ ∇ V ( p out )∥ ≤ ε uniformly on the class makes at least order n M m nMm n M m queries in expectation over the random owners and axes. Some fixed choice of component functions therefore inherits this bound, as in Yao’s minimax principle (Yao, [23 ] .
The Remaining Condition Numbers
For 64 ≤ κ ≤ 2 18 n 64\le\kappa\le2^{18}\sqrt{n} 64 ≤ κ ≤ 2 18 n , we use paths with two nodes. With r = μ / a r=\mu/a r = μ / a , they keep the calibration λ = 2 − 39 L v / u = 2 − 39 L / ( 1 + r ) \lambda=2^{-39}Lv/u=2^{-39}L/(1+r) λ = 2 − 39 Lv / u = 2 − 39 L / ( 1 + r ) and preserve (17) , and their owner searches cost at least order n M nM n M queries in expectation, which is of order n κ S \sqrt{n}\kappa S n κ S (Corollary 16 in Appendix D.§ 1 ). We use paths only for κ ≥ 64 \kappa\ge64 κ ≥ 64 : their dual ridge and cutoff barriers add about 47 ν 47\nu 47 ν to our smoothness bound (Lemma 9), and ν ≥ μ = L / κ \nu\ge\mu=L/\kappa ν ≥ μ = L / κ .
For 1 ≤ κ < 64 1\le\kappa<64 1 ≤ κ < 64 , the hard family has no hidden dual path: the dual variable enters only through a decoupled strongly concave term, and each stage is linked to the next by a primal cross term C η ( t 1 , t 2 ) = ϕ η ( t 1 + t 2 ) − ϕ η ( t 1 ) − ϕ η ( t 2 ) C_\eta(t_1,t_2)=\phi_\eta(t_1+t_2)-\phi_\eta(t_1)-\phi_\eta(t_2) C η ( t 1 , t 2 ) = ϕ η ( t 1 + t 2 ) − ϕ η ( t 1 ) − ϕ η ( t 2 ) owned by one random component, where ϕ η ( t ) = 1 2 ( ∣ t ∣ − η ) + 2 \phi_\eta(t)=\frac{1}{2}(|t|-\eta)_+^2 ϕ η ( t ) = 2 1 ( ∣ t ∣ − η ) + 2 . This cross term equals t 1 t 2 t_1t_2 t 1 t 2 at zero width and is flat near the hidden origin. Independent owners cost only a factor n \sqrt{n} n in RMS smoothness, but a single component can then have primal curvature up to order n L \sqrt{n}L n L , so without a correction the legal parameters of a component would depend on what it owns. Each component i i i therefore has its own public coordinate ξ i \xi_i ξ i with the term B ( 1 − cos ξ i ) B(1-\cos\xi_i) B ( 1 − cos ξ i ) , where B = n L / 4 B=\sqrt{n}L/4 B = n L /4 . These orthogonal coordinates cost only B / n = L / 4 B/\sqrt{n}=L/4 B / n = L /4 in RMS smoothness and give every component the exact legal domain τ ≥ B \tau\ge B τ ≥ B . Appendices D.§ 2 and D.§ 3 treat this family (Lemmas 17 –20 ). Each family is used only within its accuracy window (Equation (25) , with its own parameter λ \lambda λ ). Outside this window, and for the term n n n , we use a separate instance in which one component, unknown to the algorithm, carries a random sign (Lemma 21 ).
Matching Upper Bound and Initialization Cost
Our upper bound uses only component gradients, and it needs strong concavity only of the mean. We match the two branches of h ( n , κ ) h(n,\kappa) h ( n , κ ) with two methods: a single-loop method for κ ≤ 2 18 n \kappa\le2^{18}\sqrt{n} κ ≤ 2 18 n and an accelerated method for larger κ \kappa κ . Both methods begin at the normalized start and pay for computing a good dual point themselves.
For κ ≤ 2 18 n \kappa\le2^{18}\sqrt{n} κ ≤ 2 18 n , Algorithm 1 in Appendix § 11 combines the stochastic projected damped extragradient (SPDE) updates of Zhang et al. [24 ] with recursive finite-sum gradient estimates (Nguyen et al., [16 ] ; Fang et al. [6 ] ) and loopless stochastic variance-reduced extragradient (L-SVRE) steps (Alacaoglu and Malitsky, [1 ] ; Luo et al. [12 ] ), which need strong monotonicity only of the mean operator. The analysis tracks an energy that combines the gap of the Moreau envelope M ( z ) = min p { V ( p ) + L ∥ p − z ∥ 2 } \mathcal{M}(z)=\min_p\{V(p)+L\lVert p-z\rVert^2\} M ( z ) = min p { V ( p ) + L ∥ p − z ∥ 2 } at a moving primal center z z z with a term that controls the distance to the saddle point of F ( p , y ) + L ∥ p − z ∥ 2 F(p,y) + L\lVert p-z\rVert^2 F ( p , y ) + L ∥ p − z ∥ 2 . The normalized start bounds this distance only by κ 2 Δ / L \kappa\sqrt{2\Delta/L} κ 2Δ/ L . We prove the L-SVRE contraction directly (Lemma 24 ), and running L-SVRE at the input center, at expected query cost O ( ( n + n κ ) log ( 2 κ ) ) O((n+\sqrt{n\kappa})\log(2\kappa)) O (( n + nκ ) log ( 2 κ )) , brings the expected energy below 2 Δ 2\Delta 2Δ . At the end, the same solver refines a sampled endpoint, whose distance to its proximal saddle point is already controlled by the SPDE dissipation. Zhang et al. [24 ] analyze SPDE with a compact dual domain; we apply their one-step inequalities on the whole space by placing each step in balls on which the projections are inactive (Lemma 23). Our cost count includes every full snapshot and both gradients evaluated in each recursive difference. With the pathwise drift and motion bounds of Lemma 23, Theorem 22 then gives the bound O ( n + n κ S + ( n + n κ ) log ( 2 κ ) ) O(n+\sqrt{n\kappa}S+(n+\sqrt{n\kappa})\log(2\kappa)) O ( n + nκ S + ( n + nκ ) log ( 2 κ )) , with no logarithmic factor on the accuracy term.
For larger κ \kappa κ , the method of Appendix F uses Catalyst acceleration. We normalize f i = F i / L f_i=F_i/L f i = F i / L , ν = μ / L \nu=\mu/L ν = μ / L , D 0 = Δ / L D_0=\Delta/L D 0 = Δ/ L , and e 0 = ε / L e_0=\varepsilon/L e 0 = ε / L , and set
s = n − 1 / 2 , τ = s − ν , q = ν / s , ρ = 0.9 q . (20) s=n^{-1/2},\qquad\tau=s-\nu,\qquad q=\nu/s,\qquad\rho=0.9\sqrt{q}. \tag*{(20)} s = n − 1/2 , τ = s − ν , q = ν / s , ρ = 0.9 q . (20)
Catalyst subtracts the dual regularizer τ 2 ∥ y − z k ∥ 2 \frac{\tau}{2}\lVert y-z_k\rVert^2 2 τ ∥ y − z k ∥ 2 at dual centers z k z_k z k , so each subproblem is s s s -strongly concave, and the accuracy targets decrease like ( 1 − ρ ) k (1-\rho)^k ( 1 − ρ ) k . The regime of Theorem 3 ensures τ > 0 \tau>0 τ > 0 and q ≤ 1 / 64 q\le1/64 q ≤ 1/64 . Since our start does not supply y ∗ ( 0 ) y^*(0) y ∗ ( 0 ) , the method first computes a dual warm start: the same L-SVRE solver with an exact residual stopping test (Lemma 25), combined with Catalyst for strongly convex problems (Lin et al. [9 ] ), returns a point satisfying
∥ y w − y ∗ ( 0 ) ∥ 2 ≤ D 0 (21) \lVert y_w-y^*(0)\rVert^2\le D_0 \tag*{(21)} ∥ y w − y ∗ ( 0 ) ∥ 2 ≤ D 0 (21)
on every terminating run, at expected cost O ( n q − 1 / 2 log 2 ( 2 + κ ) ) O(nq^{-1/2}\log^2(2+\kappa)) O ( n q − 1/2 log 2 ( 2 + κ )) .
The tolerances of the accelerated scheme of Zhang et al. [25 ] depend on the saddle gap of each outer subproblem, which the algorithm does not know. At an outer point w t w_t w t , let f ^ t ( p , y ) = f ( p , y ) + ∥ p − p t ∥ 2 \widehat{f}_t(p,y)=f(p,y)+\lVert p-p_t\rVert^2 f t ( p , y ) = f ( p , y ) + ∥ p − p t ∥ 2 , let G t G_t G t be its saddle gap, and let R t 2 = ∥ ∇ f ^ t ( w t ) ∥ 2 R_t^2=\lVert\nabla\widehat{f}_t(w_t)\rVert^2 R t 2 = ∥ ∇ f t ( w t ) ∥ 2 , which costs n n n queries to compute. If R t = 0 R_t=0 R t = 0 , the point is stationary. Otherwise
B t ≔ ν R t 2 72 ≤ G t , G t B t ≤ 36 ν 2 . (22) B_t\coloneqq\frac{\nu R_t^2}{72}\le G_t,\qquad\frac{G_t}{B_t}\le\frac{36}{\nu^2}. \tag*{(22)} B t : = 72 ν R t 2 ≤ G t , B t G t ≤ ν 2 36 . (22)
Thus B t B_t B t is a computable lower bound on G t G_t G t that is off by at most the factor 36 / ν 2 36/\nu^2 36/ ν 2 . Replacing the unavailable gap by B t B_t B t in the acceleration tolerances preserves the bounds of Zhang et al. [25 ] , Theorems 4.1–4.3, at the cost of a logarithmic factor, and the residual estimate also controls each auxiliary warm start directly. Each of the O ( 1 + S ) O(1+S) O ( 1 + S ) outer solves costs O ( n q − 1 / 2 log 2 ( 2 + κ ) ) O(nq^{-1/2}\log^2(2+\kappa)) O ( n q − 1/2 log 2 ( 2 + κ )) in expectation, and n q − 1 / 2 = n 3 / 4 κ nq^{-1/2}=n^{3/4}\sqrt{\kappa} n q − 1/2 = n 3/4 κ . Appendix F also counts the warm-start solves that start from y = 0 y=0 y = 0 , the exact residual checks, and the conversion to fixed confidence, and Appendix E.8 combines the two regimes.
Related Work
For finite-sum nonconvex–concave problems, the lower bounds of Wu et al. [22 ] , Theorems 3.1 and 3.3, differ from ours in the dual geometry and the stationarity measure. As their Section 4 notes, substituting the effective condition number of a regularized dual into the NC–SC rate is only a formal comparison, and converting their Moreau envelope guarantee into a bound on the gradient of the value function can lose a factor of order κ \kappa κ . For finite-sum monotone inclusions under mean-squared Lipschitz continuity, Zhou [26 ] gives an O ( n + n L R / ε ) O(n + \sqrt{n}LR/\varepsilon) O ( n + n L R / ε ) method and a matching Ω ( n + n L R / ε ) \Omega(n + \sqrt{n}LR/\varepsilon) Ω ( n + n L R / ε ) lower bound, where R R R is an initial distance to a solution. That lower bound concerns randomized linear-span algorithms with component operator evaluations, and the resolvent there is that of the common operator, not of the components.
Our construction builds on established precedents for the path and the nonlinear source. Li et al. [8 ] use dual paths and endpoint compensation with the source functions of Carmon et al. [4 ] , and Zhou [27 ] assigns internal dual edges to random component owners. Han et al. [7 ] develop component grouping and path bounds for convex finite sums. Our construction preserves the primal and dual scales while satisfying global smoothness and hiding the component assignments from arbitrary exact proximal queries.
Exact hiding also has distinct antecedents. Woodworth and Srebro [20 ] use flat edge penalties to limit what proximal responses reveal, Carmon et al. [4 ] bound the effect of unrestricted query norms by radial compaction, and Bubeck et al. [3 ] use a convex wall outside correlation cones. In randomized information arguments, Arjevani et al. [2 ] condition on preceding projected queries. Peng et al. [17 ] develop dense weak hiding for individually smooth nonconvex and PL finite sums with exact values and gradients. Our construction combines blockwise exact transcript coupling with radius-dependent flat edges and componentwise strong concavity, and it controls the entire saddle-proximal response. In the related NC–C setting with a bounded dual domain, Wu et al. [21 ] prove a lower bound for deterministic zero-respecting algorithms, under a different stationarity measure.
Discussion
Exact proximal access and unrestricted adaptivity do not change the polynomial complexity of the problem for any condition number. Although an exact component proximal step solves a saddle problem in each call, on this class it saves at most a factor O ( log 2 ( 2 + κ ) ) O(\log^{2}(2+\kappa)) O ( log 2 ( 2 + κ )) in worst-case component queries over component gradients, for ε \varepsilon ε in the range of Theorem 1 . Two parts of our upper bounds apply to other finite-sum minimax methods: the shared L-SVRE solver needs strong concavity only of the mean, not of each component, and the residual estimate B t B_t B t of § 4 makes the subproblem tolerances of the accelerated scheme of Zhang et al. [25 ] computable.
Our lower bounds use hard instances whose dimension grows with the parameters. As in the lower bound of Zhou [27 ] , our lower bound requires ε 2 / ( L Δ ) \varepsilon^{2}/(L\Delta) ε 2 / ( L Δ ) to be below a universal constant, and we have not optimized this constant or the other universal constants. Under (8) , for κ ≤ 2 18 n \kappa\le2^{18}\sqrt{n} κ ≤ 2 18 n and S ≥ log ( 2 κ ) S \ge\log(2\kappa) S ≥ log ( 2 κ ) , Theorem 22 gives O ( n log ( 2 κ ) + h ( n , κ ) S ) O(n\log(2\kappa)+h(n,\kappa)S) O ( n log ( 2 κ ) + h ( n , κ ) S ) , so the upper and lower bounds match in the accuracy term and differ by at most a factor log ( 2 κ ) \log(2\kappa) log ( 2 κ ) in the term n n n . For larger κ \kappa κ , the accelerated method of Appendix F leaves a gap of a factor log 2 ( 2 + κ ) \log^{2}(2+\kappa) log 2 ( 2 + κ ) .
Two questions remain open. Finite sums with individually L L L -smooth components form a smaller class than those with joint mean-squared smoothness L L L . On a bilinear subclass, Zhou (2026b, Theorem 4) proves the lower bound Ω ( n + min { κ , n κ } L Δ / ε 2 ) \Omega(n+\min\{\kappa,\sqrt{n\kappa}\}L\Delta/\varepsilon^2) Ω ( n + min { κ , nκ } L Δ/ ε 2 ) for zero-respecting methods; what is the complexity of the individually smooth class for unrestricted algorithms? For nonconvex–concave finite sums over bounded dual domains, Wu et al. [22 ] determine the complexity with exact values and gradients up to a single logarithmic factor; does exact component proximal access change that complexity?
Formal verification. We provide a Lean 4 formalization of the lower-bound assertions of Theorem 1 as supplementary material and as ancillary files of the arXiv version. It covers the expected query cost and the fixed-confidence complexity under a deterministic cap, in all three ranges of the condition number and for the baseline instance that gives the term n n n . The formal proof includes the path calibration of Lemma 5, the posterior owner cost of Lemma 14, and the public cosine terms in (74) and (91) together with their exact legal boundary. In place of the Haar-distributed axes, it uses a finite Rademacher construction, and it obtains a barrier constant that plays the role of the constant of Lemma 10 by the same compactness argument. The files CLAIMS and TRUSTED DEFINITIONS of the formalization state the formal theorems, the dimension bounds, and the remaining differences from the written proof. The Haar conditioning argument, the zero-axis extension, the equality of the two barrier constants, Corollary 4, the upper bounds, and the dimension estimate (16) are not formalized.
The Hard Family and Its Global Geometry
We use one path family for every κ ≥ 64 \kappa\ge64 κ ≥ 64 : long paths prove Theorem 3, and two-node paths cover the intermediate range in Appendix D. The scalar source functions are those of Carmon et al. [4 ] , and the dual paths with endpoint compensation follow Li et al. [8 ] . Symbols defined in Appendices A–D are local to the lower-bound proofs.
Let P m P_m P m be the Laplacian of the m m m -vertex path, with diagonal ( 1 , 2 , … , 2 , 1 ) (1,2,\ldots,2,1) ( 1 , 2 , … , 2 , 1 ) . Fix
a = L 2 16 n , t = a / μ , m = max { 2 , ⌊ t ⌋ } , a=\frac{L}{2^{16}\sqrt{n}},\qquad t=\sqrt{a/\mu},\qquad m=\max\{2,\lfloor t\rfloor\}, a = 2 16 n L , t = a / μ , m = max { 2 , ⌊ t ⌋} ,
r = { μ / a , t < 2 , m − 2 , t ≥ 2 , μ 0 = a r , Q = P m + r I . (23) r=\begin{cases}
\mu/a, & t<2,\\
m^{-2}, & t\ge2,
\end{cases}
\qquad\mu_0=ar,\qquad Q=P_m+rI. \tag*{(23)} r = { μ / a , m − 2 , t < 2 , t ≥ 2 , μ 0 = a r , Q = P m + r I . (23)
Write u = ( Q − 1 ) 11 u=(Q^{-1})_{11} u = ( Q − 1 ) 11 , v = ( Q − 1 ) 1 m v=(Q^{-1})_{1m} v = ( Q − 1 ) 1 m and g = u / v g=u/v g = u / v , and put
δ 0 = 2 − 39 , λ = δ 0 L g , ν = a u , c = ν − μ 0 , b 2 = 12 a λ v , d = 6 λ ( g − 1 ) . (24) \delta_0=2^{-39},\qquad\lambda=\frac{\delta_0L}{g},\qquad\nu=\frac{a}{u},\qquad c=\nu-\mu_0,\qquad b^2=\frac{12a\lambda}{v},\qquad d=6\lambda(g-1). \tag*{(24)} δ 0 = 2 − 39 , λ = g δ 0 L , ν = u a , c = ν − μ 0 , b 2 = v 12 aλ , d = 6 λ ( g − 1 ) . (24)
Instantiate the path family only when
X : = c ⋆ 2 λ Δ 2048 ε 2 ≥ 2 , equivalently ε 2 ≤ c ⋆ 2 λ Δ 4096 . (25) X:=\frac{c_\star^2\lambda\Delta}{2048\varepsilon^2}\ge2,\qquad\text{equivalently}\qquad\varepsilon^2\le\frac{c_\star^2\lambda\Delta}{4096}. \tag*{(25)} X := 2048 ε 2 c ⋆ 2 λ Δ ≥ 2 , equivalently ε 2 ≤ 4096 c ⋆ 2 λ Δ . (25)
Outside this window the indexed baseline will suffice. Within it set
s = 8 ε c ⋆ λ , M = ⌊ X ⌋ , K = M m , E = M ( m − 1 ) . (26) s=\frac{8\varepsilon}{c_\star\lambda},\qquad M=\lfloor X\rfloor,\qquad K=Mm,\qquad E=M(m-1). \tag*{(26)} s = c ⋆ λ 8 ε , M = ⌊ X ⌋ , K = M m , E = M ( m − 1 ) . (26)
Thus m , M ≥ 2 m, M \ge2 m , M ≥ 2 and
M ≥ c ⋆ 2 λ Δ 4096 ϵ 2 , M λ s 2 ≤ Δ / 32 , λ s 2 ≤ Δ / 64. (27) M \ge\frac{c_\star^2\lambda\Delta}{4096\epsilon^2}, \qquad M\lambda s^2 \le\Delta/32, \qquad\lambda s^2 \le\Delta/64. \tag*{(27)} M ≥ 4096 ϵ 2 c ⋆ 2 λ Δ , M λ s 2 ≤ Δ/32 , λ s 2 ≤ Δ/64. (27)
The width and hiding dimension are fixed by (66) and (80) . With A 1 = 2433 A_1=2433 A 1 = 2433 , the bound on ∥ D A ∥ \lVert D\mathcal{A}\rVert ∥ D A ∥ in Lemma 6 , and C 0 = 294912 A 1 / c ⋆ C_0=294912A_1/c_\star C 0 = 294912 A 1 / c ⋆ , their ratio is
R ⋆ η = max { C 0 g r K , 16 n } , R ⋆ η 0 = C 0 g r K > 128. (28) \frac{R_\star}{\eta}=\max\left\{C_0\frac{g}{r}\sqrt{K},16\sqrt{n}\right\}, \qquad\frac{R_\star}{\eta_0}=C_0\frac{g}{r}\sqrt{K}>128. \tag*{(28)} η R ⋆ = max { C 0 r g K , 16 n } , η 0 R ⋆ = C 0 r g K > 128. (28)
In particular H n = max { C 0 2 ( g / r ) 2 K , 256 n } H_n=\max\{C_0^2(g/r)^2K,256n\} H n = max { C 0 2 ( g / r ) 2 K , 256 n } , and the dimension is fixed before execution, independently of the stopping time.
Path Geometry, Hidden Axes, and the Scalar Components
There are M M M publicly identified dual paths. Every node y j , k ∈ R D y_{j,k}\in\mathbb{R}^D y j , k ∈ R D has an independent Haar-uniform hidden unit axis U j , k U_{j,k} U j , k . The primal coordinate block P j = ( X j , Z j + 1 ) ∈ R D × R D P_j=(X_j,Z_{j+1})\in\mathbb{R}^D\times\mathbb{R}^D P j = ( X j , Z j + 1 ) ∈ R D × R D is publicly identified with the two endpoint coordinate blocks ( j , 1 ) (j,1) ( j , 1 ) and ( j , m ) (j,m) ( j , m ) , respectively. Thus its source and outgoing connector axes are U j , 1 U_{j,1} U j , 1 and U j , m U_{j,m} U j , m . Interior axes are independent of all endpoint axes. For each edge ( j , k ) (j,k) ( j , k ) , k < m k<m k < m , draw a fixed independent owner Σ j , k \Sigma_{j,k} Σ j , k uniformly from [ n ] [n] [ n ] . These draws and the axes are independent of the algorithm’s private seed; they will be fixed before invoking its oracle.
Lemma 5 (Common path bounds and branch calibration). In both branches,
μ 0 ≥ μ , 0 ≤ c ≤ a , ν ≤ μ + a , 0 < v ≤ u , \mu_0\ge\mu,\qquad0\le c\le a,\qquad\nu\le\mu+a,\qquad0<v\le u, μ 0 ≥ μ , 0 ≤ c ≤ a , ν ≤ μ + a , 0 < v ≤ u , b 2 ν = 12 δ 0 L , 0 ≤ d = 6 δ 0 L ( 1 − 1 / g ) ≤ 6 δ 0 L . (29) \frac{b^2}{\nu}=12\delta_0L,\qquad0\le d=6\delta_0L(1-1/g)\le6\delta_0L. \tag*{(29)} ν b 2 = 12 δ 0 L , 0 ≤ d = 6 δ 0 L ( 1 − 1/ g ) ≤ 6 δ 0 L . (29) *Every entry in either endpoint inverse column is at most u u u . For t ≥ 2 t\ge2 t ≥ 2 , that is, κ ≥ 2 18 n , \kappa\ge2^{18}\sqrt{n}, κ ≥ 2 18 n ,
m ≤ u ≤ m 2 , m sinh 1 ≤ v ≤ m , 1 < g ≤ cosh 1 < 5 / 3. (30) m\le u\le m^2,\qquad\frac{m}{\sinh1}\le v\le m,\qquad1<g\le\cosh1<5/3. \tag*{(30)} m ≤ u ≤ m 2 , sinh 1 m ≤ v ≤ m , 1 < g ≤ cosh 1 < 5/3. (30) Then μ ≤ μ 0 ≤ 4 μ \mu\le\mu_0\le4\mu μ ≤ μ 0 ≤ 4 μ , ν , c ≤ a / 2 \nu,c\le a/2 ν , c ≤ a /2 , d < 4 λ d<4\lambda d < 4 λ and λ ≥ 2 − 40 L \lambda\ge2^{-40}L λ ≥ 2 − 40 L . Hence the window (12) of Theorem 3 implies (25) and
m ≥ κ 512 n 1 / 4 , M ≥ c ⋆ 2 S 2 52 . (31) m\ge\frac{\sqrt{\kappa}}{512n^{1/4}},\qquad M\ge\frac{c_\star^2S}{2^{52}}. \tag*{(31)} m ≥ 512 n 1/4 κ , M ≥ 2 52 c ⋆ 2 S . (31) For t < 2 t<2 t < 2 , m = 2 m=2 m = 2 , r = μ / a > 1 / 4 r=\mu/a>1/4 r = μ / a > 1/4 , μ 0 = μ \mu_0=\mu μ 0 = μ , and
Q − 1 = 1 r ( r + 2 ) ( r + 1 1 1 r + 1 ) , g = 1 + r , ν = μ + a r r + 1 , c = a r r + 1 . (32) Q^{-1}=\frac{1}{r(r+2)}
\begin{pmatrix}
r+1 & 1\\
1 & r+1
\end{pmatrix},
\qquad g=1+r,\qquad\nu=\mu+\frac{ar}{r+1},\qquad c=\frac{ar}{r+1}. \tag*{(32)} Q − 1 = r ( r + 2 ) 1 ( r + 1 1 1 r + 1 ) , g = 1 + r , ν = μ + r + 1 a r , c = r + 1 a r . (32) The two calibrations agree at t = 2 t=2 t = 2 .
Proof The two-node formula is direct. For the long path and any mass r > 0 r>0 r > 0 , put θ = arcosh ( 1 + r / 2 ) \theta=\operatorname{arcosh}(1+r/2) θ = arcosh ( 1 + r /2 ) . The leading-minor recurrence d j = ( 2 + r ) d j − 1 − d j − 2 d_j=(2+r)d_{j-1}-d_{j-2} d j = ( 2 + r ) d j − 1 − d j − 2 , with first endpoint entry 1 + r 1+r 1 + r , gives d j = cosh ( ( j + 1 / 2 ) θ ) / cosh ( θ / 2 ) d_j=\cosh((j+1/2)\theta)/\cosh(\theta/2) d j = cosh (( j + 1/2 ) θ ) / cosh ( θ /2 ) before the last endpoint, and det ( P m + r I ) = r sinh ( m θ ) / sinh θ \det(P_m+rI)=r\sinh(m\theta)/\sinh\theta det ( P m + r I ) = r sinh ( m θ ) / sinh θ . The endpoint cofactors therefore give
v = sinh θ r sinh ( m θ ) , g = cosh ( ( m − 1 / 2 ) θ ) cosh ( θ / 2 ) . v=\frac{\sinh\theta}{r\sinh(m\theta)},\qquad g=\frac{\cosh((m-1/2)\theta)}{\cosh(\theta/2)}. v = r sinh ( m θ ) sinh θ , g = cosh ( θ /2 ) cosh (( m − 1/2 ) θ ) .
For r = m − 2 r=m^{-2} r = m − 2 , θ = 2 asinh ( 1 / ( 2 m ) ) \theta=2\operatorname{asinh}(1/(2m)) θ = 2 asinh ( 1/ ( 2 m )) , m θ ≤ 1 m\theta\le1 m θ ≤ 1 , sinh θ ≥ 1 / m \sinh\theta\ge1/m sinh θ ≥ 1/ m and sinh ( m θ ) ≥ m sinh θ \sinh(m\theta)\ge m\sinh\theta sinh ( m θ ) ≥ m sinh θ , proving the bounds on v , g v,g v , g . The constant eigenvector contributes m m m to each diagonal entry of Q − 1 Q^{-1} Q − 1 , and ∥ Q − 1 ∥ = m 2 \lVert Q^{-1}\rVert=m^2 ∥ Q − 1 ∥ = m 2 , giving the bounds on u u u . The endpoint column maximum principle gives the entrywise claim in both branches. The remaining bounds follow by substitution and ⌊ t ⌋ ≥ t / 2 \lfloor t\rfloor\ge t/2 ⌊ t ⌋ ≥ t /2 for t ≥ 2 t\ge2 t ≥ 2 .
For the high branch, r = m − 2 r=m^{-2} r = m − 2 and K = M m K=Mm K = M m , so (28) specializes to
H n = max { C 0 2 g 2 M m 5 , 256 n } , H_n=\max\{C_0^2g^2Mm^5,256n\}, H n = max { C 0 2 g 2 M m 5 , 256 n } ,
D = O ( ( n + S κ 5 / 2 n − 5 / 4 ) log ( n κ S ) ) . D=O\left(\left(n+S\kappa^{5/2}n^{-5/4}\right)\log(n\kappa S)\right). D = O ( ( n + S κ 5/2 n − 5/4 ) log ( nκ S ) ) .
Indeed M = O ( S ) M=O(S) M = O ( S ) , m = O ( κ n − 1 / 4 ) m=O(\sqrt{\kappa}n^{-1/4}) m = O ( κ n − 1/4 ) and g ≤ cosh 1 g\le\cosh1 g ≤ cosh 1 . The primal dimension is 2 M D + 1 2MD+1 2 M D + 1 , including the public coordinate in (74) , and the dual dimension is K D KD K D , so the total dimension ( 2 M + K ) D + 1 (2M+K)D+1 ( 2 M + K ) D + 1 is (16) .
The source uses the scalar functions of Carmon et al. [4 ] and Li et al. [8 ] :
Ψ ( t ) = { 0 , t ≤ 1 / 2 , e 1 − ( 2 t − 1 ) − 2 , t > 1 / 2 , Φ ( t ) = e ∫ − ∞ t e − r 2 / 2 d r . (33) \Psi(t)=
\begin{cases}
0, & t\le1/2,\\
e^{1-(2t-1)^{-2}}, & t>1/2,
\end{cases}
\qquad
\Phi(t)=\sqrt{e}\int_{-\infty}^{t}e^{-r^2/2}\,\mathrm{d}r. \tag*{(33)} Ψ ( t ) = { 0 , e 1 − ( 2 t − 1 ) − 2 , t ≤ 1/2 , t > 1/2 , Φ ( t ) = e ∫ − ∞ t e − r 2 /2 d r . (33)
The cap and primal penalty are
w j = ρ ( P j ) = P j / ( 1 + ∥ P j ∥ 2 / R 2 ) 1 / 2 , R = 128 s , (34) w_j=\rho(P_j)=P_j/(1+\lVert P_j\rVert^2/R^2)^{1/2},\qquad R=128s, \tag*{(34)} w j = ρ ( P j ) = P j / ( 1 + ∥ P j ∥ 2 / R 2 ) 1/2 , R = 128 s , (34)
p ( P j ) = λ 1024 ( ∥ P j ∥ − R ) + 2 , x j = ⟨ U j , 1 , w j , X ⟩ , z j + 1 = ⟨ U j , m , w j , Z ⟩ . (35) p(P_j)=\frac{\lambda}{1024}(\lVert P_j\rVert-R)_+^2,\qquad x_j=\langle U_{j,1},w_{j,X}\rangle,\quad z_{j+1}=\langle U_{j,m},w_{j,Z}\rangle. \tag*{(35)} p ( P j ) = 1024 λ (∥ P j ∥ − R ) + 2 , x j = ⟨ U j , 1 , w j , X ⟩ , z j + 1 = ⟨ U j , m , w j , Z ⟩ . (35)
Write w j , X = x j U j , 1 + A j , ⊥ w_{j,X}=x_jU_{j,1}+A_{j,\perp} w j , X = x j U j , 1 + A j , ⊥ and w j , Z = z j + 1 U j , m + B j + 1 , ⊥ w_{j,Z}=z_{j+1}U_{j,m}+B_{j+1,\perp} w j , Z = z j + 1 U j , m + B j + 1 , ⊥ . Let
J M = − Φ ( x 1 / s ) + ∑ j = 2 M [ Ψ ( − z j / s ) Φ ( − x j / s ) − Ψ ( z j / s ) Φ ( x j / s ) ] . (36) J_M=-\Phi(x_1/s)+\sum_{j=2}^{M}\left[\Psi(-z_j/s)\Phi(-x_j/s)-\Psi(z_j/s)\Phi(x_j/s)\right]. \tag*{(36)} J M = − Φ ( x 1 / s ) + j = 2 ∑ M [ Ψ ( − z j / s ) Φ ( − x j / s ) − Ψ ( z j / s ) Φ ( x j / s ) ] . (36)
For 1 / 2 < t < 1 1/2<t<1 1/2 < t < 1 define
w ( t ) = Ψ ( t ) , ω ( w ) = 1 − ( 1 − w ) 3 , w(t)=\sqrt{\Psi(t)},\qquad\omega(w)=1-(1-w)^3, w ( t ) = Ψ ( t ) , ω ( w ) = 1 − ( 1 − w ) 3 ,
θ + ( t ) = π 2 ω ( w ( t ) ) , a + ( t ) = sin θ + ( t ) , b + ( t ) = cos θ + ( t ) . (37) \theta_+(t)=\frac{\pi}{2}\omega(w(t)),\qquad a_+(t)=\sin\theta_+(t),\qquad b_+(t)=\cos\theta_+(t). \tag*{(37)} θ + ( t ) = 2 π ω ( w ( t )) , a + ( t ) = sin θ + ( t ) , b + ( t ) = cos θ + ( t ) . (37)
extending θ + \theta_+ θ + by 0 0 0 for t ≤ 1 / 2 t\le1/2 t ≤ 1/2 and by π / 2 \pi/2 π /2 for t ≥ 1 t\ge1 t ≥ 1 . Set
α = a + 2 , q j = ∏ k = 2 j α ( z k / s ) , q 1 = 1 , (38) \alpha=a_+^2,\qquad q_j=\prod_{k=2}^{j}\alpha(z_k/s),\qquad q_1=1, \tag*{(38)} α = a + 2 , q j = k = 2 ∏ j α ( z k / s ) , q 1 = 1 , (38)
h j = ( ∏ k = 2 j a + ( z k / s ) ) b + ( z j + 1 / s ) , A j = h j Γ w j , Γ = diag ( I , − I / 2 ) . (39) h_j=\left(\prod_{k=2}^{j}a_+(z_k/s)\right)b_+(z_{j+1}/s),\qquad A_j=h_j\Gamma w_j,\qquad\Gamma=\operatorname{diag}(I,-I/2). \tag*{(39)} h j = ( k = 2 ∏ j a + ( z k / s ) ) b + ( z j + 1 / s ) , A j = h j Γ w j , Γ = diag ( I , − I /2 ) . (39)
The concatenated endpoint forcing obeys
h j 2 = q j [ 1 − α ( z j + 1 / s ) ] , ∑ j h j 2 ≤ 1 , ∥ A ∥ ≤ R . h_j^2=q_j[1-\alpha(z_{j+1}/s)],\qquad\sum_j h_j^2\le1,\qquad\lVert\mathcal{A}\rVert\le R. h j 2 = q j [ 1 − α ( z j + 1 / s )] , j ∑ h j 2 ≤ 1 , ∥ A ∥ ≤ R .
Define the common primal part
P ( P ) = λ s 2 J M ( P ) − d ∥ A ( P ) ∥ 2 + ∑ j p ( P j ) . (40) \mathcal{P}(P)=\lambda s^2J_M(P)-d\lVert\mathcal{A}(P)\rVert^2+\sum_j p(P_j). \tag*{(40)} P ( P ) = λ s 2 J M ( P ) − d ∥ A ( P ) ∥ 2 + j ∑ p ( P j ) . (40)
Let t j , k = ⟨ U j , k , y j , k ⟩ t_{j,k}=\langle U_{j,k},y_{j,k}\rangle t j , k = ⟨ U j , k , y j , k ⟩ , ϕ η ( t ) = 1 2 ( ∣ t ∣ − η ) + 2 \phi_\eta(t)=\frac{1}{2}(\lvert t\rvert-\eta)_+^2 ϕ η ( t ) = 2 1 (∣ t ∣ − η ) + 2 , and write Y ∂ Y_{\partial} Y ∂ for the concatenation of all endpoint coordinate blocks, ordered like A \mathcal{A} A . For an edge e = ( ( j , k ) , ( j , k + 1 ) ) e=((j,k),(j,k+1)) e = (( j , k ) , ( j , k + 1 )) , write Y e = ( y j , k , y j , k + 1 ) Y_e=(y_{j,k},y_{j,k+1}) Y e = ( y j , k , y j , k + 1 ) and t e = t j , k − t j , k + 1 t_e=t_{j,k}-t_{j,k+1} t e = t j , k − t j , k + 1 . For a node v v v write y v , t v y_v,t_v y v , t v .
The cutoff functions are
χ ( t ) = { 1 t ≤ 1 , 1 − 10 r 3 + 15 r 4 − 6 r 5 t = 1 + r , 0 < r < 1 , 0 t ≥ 2. (41) \chi(t)=
\begin{cases}
1 & t\le1,\\
1-10r^3+15r^4-6r^5 & t=1+r,\ 0<r<1,\\
0 & t\ge2.
\end{cases} \tag*{(41)} χ ( t ) = ⎩ ⎨ ⎧ 1 1 − 10 r 3 + 15 r 4 − 6 r 5 0 t ≤ 1 , t = 1 + r , 0 < r < 1 , t ≥ 2. (41)
ψ ( t ) = { 0 t ≤ 1 , ( t − 1 ) 3 / 6 1 < t < 2 , ( t − 2 ) 2 / 2 + ( t − 2 ) / 2 + 1 / 6 t ≥ 2. (42) \psi(t)=
\begin{cases}
0 & t\le1,\\
(t-1)^3/6 & 1<t<2,\\
(t-2)^2/2+(t-2)/2+1/6 & t\ge2.
\end{cases} \tag*{(42)} ψ ( t ) = ⎩ ⎨ ⎧ 0 ( t − 1 ) 3 /6 ( t − 2 ) 2 /2 + ( t − 2 ) /2 + 1/6 t ≤ 1 , 1 < t < 2 , t ≥ 2. (42)
We localize the cutoff χ \chi χ and the dual radial profile ψ \psi ψ at the radius R ∗ R_* R ∗ by setting
R ∗ = 16 b R ν , χ ∗ ( z ) = χ ( ∥ z ∥ / R ∗ ) , ψ ∗ ( z ) = R ∗ 2 ψ ( ∥ z ∥ / R ∗ ) . (43) R_*=\frac{16bR}{\nu},\qquad\chi_*(z)=\chi(\lVert z\rVert/R_*),\qquad\psi_*(z)=R_*^2\psi(\lVert z\rVert/R_*). \tag*{(43)} R ∗ = ν 16 b R , χ ∗ ( z ) = χ (∥ z ∥ / R ∗ ) , ψ ∗ ( z ) = R ∗ 2 ψ (∥ z ∥ / R ∗ ) . (43)
For the owned edges use the profile ϱ \varrho ϱ in (19) and define
h η ( r ) = η ϱ ( r / R ∗ ) , A η ( Y e ) = 1 2 ( ∣ t e ∣ − h η ( ∥ Y e ∥ ) ) + 2 , B T ( z ) = 1 2 ( ∥ z ∥ − T ) + 2 . (44) h_\eta(r)=\eta\varrho(r/R_*),\qquad A_\eta(Y_e)=\frac{1}{2}(\lvert t_e\rvert-h_\eta(\lVert Y_e\rVert))_+^2,\qquad B_T(z)=\frac{1}{2}(\lVert z\rVert-T)_+^2. \tag*{(44)} h η ( r ) = η ϱ ( r / R ∗ ) , A η ( Y e ) = 2 1 (∣ t e ∣ − h η (∥ Y e ∥) ) + 2 , B T ( z ) = 2 1 (∥ z ∥ − T ) + 2 . (44)
Node and endpoint cutoffs keep radius R ∗ R_* R ∗ ; we use the positive width (66) and width zero. The globally defined components are precisely
f i , η ( P , y ) = P ( P ) − ν 2 ∥ y ∥ 2 + b χ ∗ ( Y ∂ ) ⟨ A ( P ) , Y ∂ ⟩ − 45 ν ψ ∗ ( Y ∂ ) + ∑ v [ c χ ∗ ( y v ) ϕ η ( t v ) − 720 c ψ ∗ ( y v ) ] − n a ∑ e : ∑ e = i A η ( Y e ) − a ν ∑ e B R ∗ / 2 ( Y e ) . (45) \begin{aligned}
f_{i,\eta}(P,y)={}&\mathcal{P}(P)-\frac{\nu}{2}\lVert y\rVert^2+b\chi_*(Y_{\partial})\langle\mathcal{A}(P),Y_{\partial}\rangle-45\nu\psi_*(Y_{\partial})\\
&+\sum_v\left[c\chi_*(y_v)\phi_\eta(t_v)-720c\psi_*(y_v)\right]\\
&-n a\sum_{e:\sum e=i}A_\eta(Y_e)-a\sqrt{\nu}\sum_e B_{R_*/2}(Y_e). \tag*{(45)}
\end{aligned} f i , η ( P , y ) = P ( P ) − 2 ν ∥ y ∥ 2 + b χ ∗ ( Y ∂ ) ⟨ A ( P ) , Y ∂ ⟩ − 45 ν ψ ∗ ( Y ∂ ) + v ∑ [ c χ ∗ ( y v ) ϕ η ( t v ) − 720 c ψ ∗ ( y v ) ] − na e : ∑ e = i ∑ A η ( Y e ) − a ν e ∑ B R ∗ /2 ( Y e ) . (45)
Only the edge sum with a flat region depends on owners; every radial correction uses public norms and is common to all components.
The radii are nested as follows. Since ∥ A ∥ ≤ R \lVert\mathcal{A}\rVert\le R ∥ A ∥ ≤ R , every node, edge pair and endpoint tuple of the uncut dual maximizer has norm at most 2 b R / ν = R ∗ / 8 2bR/\nu=R_*/8 2 b R / ν = R ∗ /8 (Lemma 7), and after the width perturbation these stay inside 3 R ∗ / 16 3R_*/16 3 R ∗ /16 (Lemma 11). There the cutoffs equal one and the barriers ψ ∗ \psi_* ψ ∗ and B R ∗ / 2 B_{R_*/2} B R ∗ /2 vanish, so localization does not move the maximizer. The barriers act only beyond R ∗ / 2 R_*/2 R ∗ /2 , where B R ∗ / 2 B_{R_*/2} B R ∗ /2 dominates the curvature of the flat edges (Lemma 8).
Forcing Derivatives
Lemma 6 (Gate and forcing estimates). The nonnegative amplitudes a + , b + a_{+},b_{+} a + , b + are C 2 C^{2} C 2 , flat to second order at both joins, and satisfy a + 2 + b + 2 = 1 a_{+}^{2}+b_{+}^{2}=1 a + 2 + b + 2 = 1 . With C = 23 C=23 C = 23 ,
Ψ ( t ) ≤ α ( t ) ≤ C Ψ ( t ) , 1 2 < t < 1 , (46) \Psi(t)\leq\alpha(t)\leq C\Psi(t),\qquad\frac{1}{2}<t<1, \tag*{(46)} Ψ ( t ) ≤ α ( t ) ≤ C Ψ ( t ) , 2 1 < t < 1 , (46) α ( t ) / C ≤ Ψ ( t ) ≤ e α ( t ) , t ≥ 0 , (47) \alpha(t)/C\leq\Psi(t)\leq e\alpha(t),\qquad t\geq0, \tag*{(47)} α ( t ) / C ≤ Ψ ( t ) ≤ e α ( t ) , t ≥ 0 , (47) 0 ≤ α ′ ( t ) ≤ C Ψ ′ ( t ) , 1 2 < t < 1. (48) 0\leq\alpha'(t)\leq C\Psi'(t),\qquad\frac{1}{2}<t<1. \tag*{(48)} 0 ≤ α ′ ( t ) ≤ C Ψ ′ ( t ) , 2 1 < t < 1. (48) Outside the transition α ′ = 0 \alpha'=0 α ′ = 0 . Uniformly in length, dimension, owners, signs and raw radii,
∥ D A ∥ ≤ 2433 , ∥ D 2 A ∥ < 231000 / s , (49) \lVert D\mathcal{A}\rVert\leq2433,\qquad\lVert D^{2}\mathcal{A}\rVert<231000/s, \tag*{(49)} ∥ D A ∥ ≤ 2433 , ∥ D 2 A ∥ < 231000/ s , (49) where the second norm is bilinear-to-vector in Euclidean norms.
Proof. Gate. At onset w = e 1 / 2 − [ 2 ( 2 t − 1 ) 2 ] − 1 w=e^{1/2-[2(2t-1)^{2}]^{-1}} w = e 1/2 − [ 2 ( 2 t − 1 ) 2 ] − 1 is infinitely flat; at completion 1 − ω ( w ) = ( 1 − w ) 3 1-\omega(w)=(1-w)^{3} 1 − ω ( w ) = ( 1 − w ) 3 , so b + = O ( ( 1 − t ) 3 ) b_{+}=O((1-t)^{3}) b + = O (( 1 − t ) 3 ) and 1 − a + = O ( ( 1 − t ) 6 ) 1-a_{+}=O((1-t)^{6}) 1 − a + = O (( 1 − t ) 6 ) . For 0 ≤ w ≤ 1 0\leq w\leq1 0 ≤ w ≤ 1 , w ≤ ω ( w ) ≤ 3 w w\leq\omega(w)\leq3w w ≤ ω ( w ) ≤ 3 w and ω ( w ) ≤ sin ( π ω ( w ) / 2 ) ≤ π ω ( w ) / 2 \omega(w)\leq\sin(\pi\omega(w)/2)\leq\pi\omega(w)/2 ω ( w ) ≤ sin ( π ω ( w ) /2 ) ≤ π ω ( w ) /2 . Hence the first two comparisons hold, using 9 π 2 / 4 < C 9\pi^{2}/4<C 9 π 2 /4 < C , α = 1 \alpha=1 α = 1 and 1 ≤ Ψ < e 1\leq\Psi<e 1 ≤ Ψ < e on t ≥ 1 t\geq1 t ≥ 1 . Differentiating gives α ′ / Ψ ′ = π ω ′ ( w ) sin θ + cos θ + / ( 2 w ) < C \alpha'/\Psi'=\pi\omega'(w)\sin\theta_{+}\cos\theta_{+}/(2w)<C α ′ / Ψ ′ = π ω ′ ( w ) sin θ + cos θ + / ( 2 w ) < C .
Angle and radial maps. For v = ( 2 t − 1 ) − 2 ≥ 1 v=(2t-1)^{-2}\geq1 v = ( 2 t − 1 ) − 2 ≥ 1 , w ′ = 2 e ( 1 − v ) / 2 v − 3 / 2 w'=2e^{(1-v)/2}v^{-3/2} w ′ = 2 e ( 1 − v ) /2 v − 3/2 and w ′ ′ = e ( 1 − v ) / 2 v − 2 ( v − 3 ) w''=e^{(1-v)/2}v^{-2}(v-3) w ′′ = e ( 1 − v ) /2 v − 2 ( v − 3 ) . Maximizing v − v / 2 v^{-v/2} v − v /2 gives ∣ w ′ ∣ < 4 |w'|<4 ∣ w ′ ∣ < 4 , ∣ w ′ ′ ∣ < 120 |w''|<120 ∣ w ′′ ∣ < 120 and thus ∣ θ + ′ ∣ < 19 |\theta_{+}'|<19 ∣ θ + ′ ∣ < 19 , ∣ θ + ′ ′ ∣ < 720 |\theta_{+}''|<720 ∣ θ + ′′ ∣ < 720 . Writing c = ( 1 + ∥ P ∥ 2 / R 2 ) − 1 / 2 c=(1+\lVert P\rVert^{2}/R^{2})^{-1/2} c = ( 1 + ∥ P ∥ 2 / R 2 ) − 1/2 gives D ρ = c I − c 3 P P ⊤ / R 2 D\rho=cI-c^{3}PP^{\top}/R^{2} D ρ = c I − c 3 P P ⊤ / R 2 , with eigenvalues c , c 3 c,c^{3} c , c 3 . The three second-derivative terms linear in P P P total at most 3 r ( 1 + r 2 ) − 3 / 2 / R 3r(1+r^{2})^{-3/2}/R 3 r ( 1 + r 2 ) − 3/2 / R , and the cubic term is at most 3 r 3 ( 1 + r 2 ) − 5 / 2 / R 3r^{3}(1+r^{2})^{-5/2}/R 3 r 3 ( 1 + r 2 ) − 5/2 / R , where r = ∥ P ∥ / R r=\lVert P\rVert/R r = ∥ P ∥ / R . Each scalar factor is at most 1 / 2 1/2 1/2 , so ∥ D ρ ∥ ≤ 1 \lVert D\rho\rVert\leq1 ∥ D ρ ∥ ≤ 1 , ∥ D 2 ρ ∥ ≤ 3 / R \lVert D^{2}\rho\rVert\leq3/R ∥ D 2 ρ ∥ ≤ 3/ R . The disjoint-block angle map Θ j ( P ) = θ + ( z j + 1 / s ) \Theta_{j}(P)=\theta_{+}(z_{j+1}/s) Θ j ( P ) = θ + ( z j + 1 / s ) therefore obeys
∥ D Θ ∥ < 19 / s , ∥ D 2 Θ ∥ < 721 / s 2 . \lVert D\Theta\rVert<19/s,\qquad\lVert D^{2}\Theta\rVert<721/s^{2}. ∥ D Θ ∥ < 19/ s , ∥ D 2 Θ ∥ < 721/ s 2 .
For the second bound use ∑ j ∥ U j ∥ 2 ∥ W j ∥ 2 ≤ ∥ U ∥ 2 ∥ W ∥ 2 \sum_{j}\lVert U_{j}\rVert^{2}\lVert W_{j}\rVert^{2}\leq\lVert U\rVert^{2}\lVert W\rVert^{2} ∑ j ∥ U j ∥ 2 ∥ W j ∥ 2 ≤ ∥ U ∥ 2 ∥ W ∥ 2 .
Spherical coordinates. To avoid a length factor, put p 0 = 1 p_{0}=1 p 0 = 1 , p k = ∏ i ≤ k sin ϑ i p_{k}=\prod_{i\leq k}\sin\vartheta_{i} p k = ∏ i ≤ k sin ϑ i and
H ( ϑ ) = ( cos ϑ 1 , p 1 cos ϑ 2 , … , p M − 1 cos ϑ M , p M ) . H(\vartheta)=(\cos\vartheta_{1},p_{1}\cos\vartheta_{2},\ldots,p_{M-1}\cos\vartheta_{M},p_{M}). H ( ϑ ) = ( cos ϑ 1 , p 1 cos ϑ 2 , … , p M − 1 cos ϑ M , p M ) .
Let T k T_{k} T k be the unit tail starting with cos ϑ k \cos\vartheta_{k} cos ϑ k , padded by zeros, and Q k = ∂ k T k Q_{k}=\partial_{k}T_{k} Q k = ∂ k T k . The Q k Q_{k} Q k are orthonormal, since earlier derivatives are multiples of a later tail and tangents are orthogonal to that tail. Moreover,
∂ k H = p k − 1 Q k , ∥ ∇ p k − 1 ∥ 2 = ∑ i < k cos 2 ϑ i ∏ r < k r ≠ i sin 2 ϑ r ≤ 1. \partial_{k}H=p_{k-1}Q_{k},\qquad\lVert\nabla p_{k-1}\rVert^{2}=\sum_{i<k}\cos^{2}\vartheta_{i}\prod_{\substack{r<k\\r\ne i}}\sin^{2}\vartheta_{r}\leq1. ∂ k H = p k − 1 Q k , ∥ ∇ p k − 1 ∥ 2 = i < k ∑ cos 2 ϑ i r < k r = i ∏ sin 2 ϑ r ≤ 1.
The last inequality follows by expanding ∏ i < k ( sin 2 ϑ i + cos 2 ϑ i ) = 1 \prod_{i<k}(\sin^{2}\vartheta_{i}+\cos^{2}\vartheta_{i})=1 ∏ i < k ( sin 2 ϑ i + cos 2 ϑ i ) = 1 . Thus ∥ D H ∥ ≤ 1 \lVert DH\rVert\leq1 ∥ D H ∥ ≤ 1 , and
D 2 H [ U , W ] = − ∑ k U k W k p k − 1 T k + ∑ k [ W k ⟨ ∇ p k − 1 , U < k ⟩ + U k ⟨ ∇ p k − 1 , W < k ⟩ ] Q k . D^{2}H[U,W]=-\sum_{k}U_{k}W_{k}p_{k-1}T_{k}
+\sum_{k}[W_{k}\langle\nabla p_{k-1},U_{<k}\rangle+U_{k}\langle\nabla p_{k-1},W_{<k}\rangle]Q_{k}. D 2 H [ U , W ] = − k ∑ U k W k p k − 1 T k + k ∑ [ W k ⟨ ∇ p k − 1 , U < k ⟩ + U k ⟨ ∇ p k − 1 , W < k ⟩] Q k .
has norm at most 3 ∥ U ∥ ∥ W ∥ 3\lVert U\rVert\lVert W\rVert 3 ∥ U ∥ ∥ W ∥ , by Cauchy–Schwarz in the first sum and orthogonality in each of the other two.
Forcing map. Since h h h contains the first M M M coordinates of H ( Θ ) H(\Theta) H ( Θ ) , ∥ D h ∥ ≤ 19 / s \lVert Dh\rVert\le19/s ∥ D h ∥ ≤ 19/ s and ∥ D 2 h ∥ < 1804 / s 2 \lVert D^2h\rVert< 1804/s^2 ∥ D 2 h ∥ < 1804/ s 2 . Differentiating A j = h j Γ ρ ( P j ) A_j=h_j\Gamma\rho(P_j) A j = h j Γ ρ ( P j ) in orthogonal output blocks gives
∥ D A ∥ ≤ 1 + R ∥ D h ∥ ≤ 2433 , ∥ D 2 A ∥ ≤ 3 / R + 2 ∥ D h ∥ + R ∥ D 2 h ∥ < 231000 / s . \lVert DA\rVert\le1+R\lVert Dh\rVert\le2433,\qquad\lVert D^2A\rVert\le3/R+2\lVert Dh\rVert+R\lVert D^2h\rVert< 231000/s. ∥ D A ∥ ≤ 1 + R ∥ D h ∥ ≤ 2433 , ∥ D 2 A ∥ ≤ 3/ R + 2 ∥ D h ∥ + R ∥ D 2 h ∥ < 231000/ s .
Class Membership and Exact Zero-Width Elimination
Lemma 7 (Uncut energy and Schur identity). Define
E η ( y ) = ν 2 ∥ y ∥ 2 − c ∑ v ϕ η ( t v ) + a ∑ e ϕ η ( t e ) . (50) \mathcal{E}_{\eta}(y)=\frac{\nu}{2}\lVert y\rVert^2-c\sum_v\phi_{\eta}(t_v)+a\sum_e\phi_{\eta}(t_e). \tag*{(50)} E η ( y ) = 2 ν ∥ y ∥ 2 − c v ∑ ϕ η ( t v ) + a e ∑ ϕ η ( t e ) . (50) It is μ 0 \mu_0 μ 0 -strongly convex. At η = 0 \eta=0 η = 0 , its hidden-axis Hessian on every path is a Q aQ a Q , and its complementary Hessian is ν I \nu I ν I . Maximizing P ( P ) + b ⟨ A ( P ) , Y ∂ ⟩ − E 0 ( y ) P(P)+b\langle A(P),Y_{\partial}\rangle-\mathcal{E}_0(y) P ( P ) + b ⟨ A ( P ) , Y ∂ ⟩ − E 0 ( y ) gives exactly
V 0 ( P ) = λ s 2 J M ( P ) + ∑ j h j 2 Q j ( w j ) + ∑ j p ( P j ) , (51) V_0(P)=\lambda s^2J_M(P)+\sum_j h_j^2Q_j(w_j)+\sum_j p(P_j), \tag*{(51)} V 0 ( P ) = λ s 2 J M ( P ) + j ∑ h j 2 Q j ( w j ) + j ∑ p ( P j ) , (51) where
Q j ( w ) = 6 λ ( x j − z j + 1 / 2 ) 2 + 6 λ ∥ A j , ⊥ ∥ 2 + 3 2 λ ∥ B j + 1 , ⊥ ∥ 2 . (52) Q_j(w)=6\lambda(x_j-z_{j+1/2})^2+6\lambda\lVert A_{j,\perp}\rVert^2+\frac{3}{2}\lambda\lVert B_{j+1,\perp}\rVert^2. \tag*{(52)} Q j ( w ) = 6 λ ( x j − z j + 1/2 ) 2 + 6 λ ∥ A j , ⊥ ∥ 2 + 2 3 λ ∥ B j + 1 , ⊥ ∥ 2 . (52) Its unique uncut maximizer obeys
∥ Y ∂ , 0 ∥ ≤ 2 b R / ν , ∥ y v , 0 ∥ ≤ 2 b R / ν , ∥ Y e , 0 ∥ ≤ 2 b R / ν . (53) \lVert Y_{\partial,0}\rVert\le2bR/\nu,\qquad\lVert y_{v,0}\rVert\le2bR/\nu,\qquad\lVert Y_{e,0}\rVert\le2bR/\nu. \tag*{(53)} ∥ Y ∂ , 0 ∥ ≤ 2 b R / ν , ∥ y v , 0 ∥ ≤ 2 b R / ν , ∥ Y e , 0 ∥ ≤ 2 b R / ν . (53)
Proof Let U U U map the vector of hidden scalar coordinates to their orthogonal ambient node axes, and let D U D_U D U be the incidence map on those scalars; ∥ D U ∥ ≤ 2 \lVert D_U\rVert\le2 ∥ D U ∥ ≤ 2 . Almost everywhere,
∇ 2 E η = ν I − c U diag ( active nodes ) U T + a D U T diag ( active edges ) D U ⪰ ( ν − c ) I = μ 0 I . \nabla^2\mathcal{E}_{\eta}=\nu I-cU\operatorname{diag}(\text{active nodes})U^{\mathsf{T}}+aD_U^{\mathsf{T}}\operatorname{diag}(\text{active edges})D_U\succeq(\nu-c)I=\mu_0I. ∇ 2 E η = ν I − c U diag ( active nodes ) U T + a D U T diag ( active edges ) D U ⪰ ( ν − c ) I = μ 0 I .
The gradient is continuous at the hinges, so line integration gives strong convexity everywhere. At zero width the hidden Hessian is μ 0 I + a P m = a Q \mu_0I+aP_m=aQ μ 0 I + a P m = a Q , while the orthogonal complements have Hessian ν I \nu I ν I . For one path its eliminated contribution before compensation is
b 2 h j 2 2 a ( u x j 2 − v x j z j + 1 + u z j + 1 2 / 4 ) + b 2 h j 2 2 ν ( ∥ A j , ∥ ∥ 2 + ∥ B j + 1 , ∥ ∥ 2 / 4 ) . \frac{b^2h_j^2}{2a}\left(ux_j^2-vx_jz_{j+1}+uz_{j+1}^2/4\right)+\frac{b^2h_j^2}{2\nu}\left(\lVert A_{j,\parallel}\rVert^2+\lVert B_{j+1,\parallel}\rVert^2/4\right). 2 a b 2 h j 2 ( u x j 2 − v x j z j + 1 + u z j + 1 2 /4 ) + 2 ν b 2 h j 2 ( ∥ A j , ∥ ∥ 2 + ∥ B j + 1 , ∥ ∥ 2 /4 ) .
Both endpoint diagonal coefficients after subtracting d h j 2 ( ∥ w j , X ∥ 2 + ∥ w j , Z ∥ 2 / 4 ) dh_j^2(\lVert w_{j,X}\rVert^2+\lVert w_{j,Z}\rVert^2/4) d h j 2 (∥ w j , X ∥ 2 + ∥ w j , Z ∥ 2 /4 ) are fixed by b 2 u / ( 2 a ) − d = b 2 / ( 2 ν ) − d = 6 λ b^2u/(2a)-d=b^2/(2\nu)-d=6\lambda b 2 u / ( 2 a ) − d = b 2 / ( 2 ν ) − d = 6 λ . This proves the identity.
The hidden endpoint Green matrix has norm u + v ≤ 2 u u+v\le2u u + v ≤ 2 u and the complementary inverse is 1 / ν 1/\nu 1/ ν . Summing over the orthogonal bridges and using ∑ j ∥ A j ∥ 2 ≤ R 2 \sum_j\lVert A_j\rVert^2\le R^2 ∑ j ∥ A j ∥ 2 ≤ R 2 proves the first optimizer bound. Any two interior rows, such as the two nodes of an edge, restricted to the two source columns form a 2 × 2 2\times2 2 × 2 matrix with entries at most u u u , hence norm at most 2 u 2u 2 u . The complementary sources occur only at endpoints, where the inverse has norm 1 / ν 1/\nu 1/ ν . Taking the maximum of these orthogonal operators proves both local bounds.
Lemma 8 (Radial calculus). The following bounds hold globally, where Hessians are understood almost everywhere.
(i) If a C 1 , 1 C^{1,1} C 1 , 1 scalar A A A satisfies ∣ A ∣ / R ∗ 2 , ∥ ∇ A ∥ / R ∗ ≤ 4 A 0 \lvert A\rvert/R_*^2,\ \lVert\nabla A\rVert/R_* \le4A_0 ∣ A ∣ / R ∗ 2 , ∥ ∇ A ∥ / R ∗ ≤ 4 A 0 on R ∗ < ∥ z ∥ < 2 R ∗ R_* < \lVert z\rVert< 2R_* R ∗ < ∥ z ∥ < 2 R ∗ , then
∇ 2 ( χ ∗ A ) − χ ∗ ∇ 2 A ⪯ 720 A 0 ∇ 2 ψ ∗ . \nabla^2(\chi_* A)-\chi_*\nabla^2 A\preceq720A_0\nabla^2\psi_*. ∇ 2 ( χ ∗ A ) − χ ∗ ∇ 2 A ⪯ 720 A 0 ∇ 2 ψ ∗ . (ii) For every η ≥ 0 \eta\ge0 η ≥ 0 , the radial threshold h η ( ∥ z ∥ ) h_\eta(\lVert z\rVert) h η (∥ z ∥) is convex and globally C 1 , 1 C^{1,1} C 1 , 1 , constant on ∥ z ∥ ≤ R ∗ \lVert z\rVert\le R_* ∥ z ∥ ≤ R ∗ , and satisfies ∥ ∇ h η ∥ ≤ β : = η / R ∗ \lVert\nabla h_\eta\rVert\le\beta:=\eta/R_* ∥ ∇ h η ∥ ≤ β := η / R ∗ and h η ( r ) ≥ max { η , η r / ( 2 R ∗ ) } h_\eta(r)\ge\max\{\eta,\eta r/(2R_*)\} h η ( r ) ≥ max { η , η r / ( 2 R ∗ )} . For r = ∥ z ∥ > 0 r=\lVert z\rVert>0 r = ∥ z ∥ > 0 , u = z / r u=z/r u = z / r ,
∇ 2 h η = η R ∗ 2 ϱ ′ ′ ( r / R ∗ ) u u T + η R ∗ r ϱ ′ ( r / R ∗ ) ( I − u u T ) ⪰ 0. (54) \nabla^2h_\eta=\frac{\eta}{R_*^2}\varrho''(r/R_*)uu^\mathsf{T}+\frac{\eta}{R_*r}\varrho'(r/R_*)(I-uu^\mathsf{T})\succeq0. \tag*{(54)} ∇ 2 h η = R ∗ 2 η ϱ ′′ ( r / R ∗ ) u u T + R ∗ r η ϱ ′ ( r / R ∗ ) ( I − u u T ) ⪰ 0. (54) Here 0 ≤ ϱ ′ ≤ 1 0\le\varrho'\le1 0 ≤ ϱ ′ ≤ 1 and ϱ ′ ′ = 1 ( 1 , 2 ) \varrho''=\mathbb{1}_{(1,2)} ϱ ′′ = 1 ( 1 , 2 ) almost everywhere.
(iii) If n ≥ 2 n\ge2 n ≥ 2 , 0 ≤ η ≤ R ∗ / ( 16 n ) 0\le\eta\le R_*/(16\sqrt{n}) 0 ≤ η ≤ R ∗ / ( 16 n ) and a vector e U e_U e U satisfies ∥ e U ∥ ≤ 2 \lVert e_U\rVert\le\sqrt{2} ∥ e U ∥ ≤ 2 , the owned edge G ( z ) = − n a 2 ( ∣ ⟨ e U , z ⟩ ∣ − h η ( ∥ z ∥ ) ) + 2 G(z)=-\frac{na}{2}(\lvert\langle e_U,z\rangle\rvert-h_\eta(\lVert z\rVert))_+^2 G ( z ) = − 2 na (∣⟨ e U , z ⟩∣ − h η (∥ z ∥) ) + 2 is globally C 1 , 1 C^{1,1} C 1 , 1 , with
∥ ∇ 2 G ∥ ≤ 9 4 n a , G − a n B R ∗ / 2 globally concave . (55) \lVert\nabla^2G\rVert\le\frac{9}{4}na,\qquad G-a\sqrt{n}B_{R_*/2}\text{ globally concave}. \tag*{(55)} ∥ ∇ 2 G ∥ ≤ 4 9 na , G − a n B R ∗ /2 globally concave . (55)
Proof. On ∥ z ∥ / R ∗ = 1 + r \lVert z\rVert/R_*=1+r ∥ z ∥ / R ∗ = 1 + r , 0 < r < 1 0<r<1 0 < r < 1 , one has ∣ χ ′ ∣ ≤ 30 r 2 \lvert\chi'\rvert\le30r^2 ∣ χ ′ ∣ ≤ 30 r 2 , ∣ χ ′ ′ ∣ ≤ 60 r \lvert\chi''\rvert\le60r ∣ χ ′′ ∣ ≤ 60 r . In direction t e r + w te_r+w t e r + w , w ⊥ e r w\perp e_r w ⊥ e r , the two extra product rule terms are bounded above by
A 0 ( 480 r t 2 + 240 r 2 ∣ t ∥ w ∥ + 120 r 2 ∥ w ∥ 2 ) ≤ 720 A 0 ( r t 2 + r 2 ∥ w ∥ 2 / 4 ) . A_0(480rt^2+240r^2\lvert t\rVert w\rVert+120r^2\lVert w\rVert^2)\le720A_0(rt^2+r^2\lVert w\rVert^2/4). A 0 ( 480 r t 2 + 240 r 2 ∣ t ∥ w ∥ + 120 r 2 ∥ w ∥ 2 ) ≤ 720 A 0 ( r t 2 + r 2 ∥ w ∥ 2 /4 ) .
The difference is positive semidefinite, since its matrix has diagonals 240 r 240r 240 r , 60 r 2 60r^2 60 r 2 and off-diagonal − 120 r 2 -120r^2 − 120 r 2 , with determinant 14400 r 3 ( 1 − r ) ≥ 0 14400r^3(1-r)\ge0 14400 r 3 ( 1 − r ) ≥ 0 . The radial and tangential eigenvalues of ∇ 2 ψ ∗ \nabla^2\psi_* ∇ 2 ψ ∗ are r r r and r 2 / [ 2 ( 1 + r ) ] ≥ r 2 / 4 r^2/[2(1+r)]\ge r^2/4 r 2 / [ 2 ( 1 + r )] ≥ r 2 /4 . Outside this annulus the cutoff is constant, proving (i).
The derivative of ϱ \varrho ϱ is 0 0 0 , t − 1 t-1 t − 1 , and 1 1 1 on its three intervals, continuous and 1 1 1 -Lipschitz. The radial Hessian formula gives (ii), including regularity at the origin since the threshold is locally constant. The lower bound uses ϱ ≥ 1 ≥ t / 2 \varrho\ge1\ge t/2 ϱ ≥ 1 ≥ t /2 on [ 0 , 2 ] [0,2] [ 0 , 2 ] and ϱ ( t ) = t − 1 / 2 ≥ t / 2 \varrho(t)=t-1/2\ge t/2 ϱ ( t ) = t − 1/2 ≥ t /2 for every t ≥ 2 t\ge2 t ≥ 2 .
For (iii), write r = ∥ z ∥ r=\lVert z\rVert r = ∥ z ∥ and, on the active set, put q = ∣ ⟨ e U , z ⟩ ∣ − h η ( r ) > 0 q=\lvert\langle e_U,z\rangle\rvert-h_\eta(r)>0 q = ∣⟨ e U , z ⟩∣ − h η ( r ) > 0 and v = sign ( ⟨ e U , z ⟩ ) e U − ∇ h η v=\operatorname{sign}(\langle e_U,z\rangle)e_U-\nabla h_\eta v = sign (⟨ e U , z ⟩) e U − ∇ h η . Then
∇ G = − n a q v , ∇ 2 G = − n a v v T + n a q ∇ 2 h η . \nabla G=-naqv,\qquad\nabla^2G=-na\,vv^\mathsf{T}+naq\nabla^2h_\eta. ∇ G = − na q v , ∇ 2 G = − na v v T + na q ∇ 2 h η .
For η > 0 \eta>0 η > 0 , away from ⟨ e U , z ⟩ = 0 \langle e_U,z\rangle=0 ⟨ e U , z ⟩ = 0 this is the locally C 1 , 1 C^{1,1} C 1 , 1 composition − n a ( q + ) 2 / 2 -na(q_+)^2/2 − na ( q + ) 2 /2 ; the gradient vanishes continuously at activation. Near ⟨ e U , z ⟩ = 0 \langle e_U,z\rangle=0 ⟨ e U , z ⟩ = 0 the function is locally zero. For η = 0 \eta=0 η = 0 , G = − n a ⟨ e U , z ⟩ 2 / 2 G=-na\langle e_U,z\rangle^2/2 G = − na ⟨ e U , z ⟩ 2 /2 directly. Since q ≤ 2 r q\le\sqrt{2}r q ≤ 2 r , the radial and tangential eigenvalues of q ∇ 2 h η q\nabla^2h_\eta q ∇ 2 h η are at most 2 2 β 2\sqrt{2}\beta 2 2 β and 2 β \sqrt{2}\beta 2 β , respectively; the radial term is supported on R ∗ < r < 2 R ∗ R_*<r<2R_* R ∗ < r < 2 R ∗ . Thus
0 ⪯ n a q ∇ 2 h η ⪯ 2 2 n a β I ⪯ ( 2 / 8 ) a n I , (56) 0\preceq naq\nabla^2h_\eta\preceq2\sqrt{2}na\beta I\preceq(\sqrt{2}/8)a\sqrt{n}I, \tag*{(56)} 0 ⪯ na q ∇ 2 h η ⪯ 2 2 na β I ⪯ ( 2 /8 ) a n I , (56)
vanishing on r < R ∗ r<R_* r < R ∗ . Also, since β ≤ 1 / ( 16 n ) \beta\le1/(16\sqrt{n}) β ≤ 1/ ( 16 n ) and n ≥ 2 n\ge2 n ≥ 2 , ∥ v ∥ 2 ≤ ( 2 + β ) 2 ≤ ( 2 + 1 / ( 16 2 ) ) 2 = 1089 / 512 < 9 / 4 \lVert v\rVert^2\le(\sqrt{2}+\beta)^2\le(\sqrt{2}+1/(16\sqrt{2}))^2=1089/512<9/4 ∥ v ∥ 2 ≤ ( 2 + β ) 2 ≤ ( 2 + 1/ ( 16 2 ) ) 2 = 1089/512 < 9/4 and 2 2 β ≤ 1 / 8 2\sqrt{2}\beta\le1/8 2 2 β ≤ 1/8 . The norm of the difference of two positive semidefinite matrices is at most the larger norm, proving the edge bound. For r > R ∗ / 2 r > R_*/2 r > R ∗ /2 , ∇ 2 B R ∗ / 2 = u u ⊤ + ( 1 − R ∗ / ( 2 r ) ) ( I − u u ⊤ ) \nabla^2 B_{R_*/2} = uu^\top+ (1 - R_*/(2r))(I - uu^\top) ∇ 2 B R ∗ /2 = u u ⊤ + ( 1 − R ∗ / ( 2 r )) ( I − u u ⊤ ) ; it is nonnegative everywhere and at least I / 2 I/2 I /2 on r ≥ R ∗ r \ge R_* r ≥ R ∗ , which dominates (56) after multiplication by a n a\sqrt{n} a n . All gradients match at the profile, cutoff and activation joins. Bounded weak Hessians, mollification and line integration therefore give the stated global C 1 , 1 C^{1,1} C 1 , 1 and curvature claims. ■
Lemma 9 (Componentwise strong concavity and RMS smoothness). The components (45) are globally C 1 , 1 C^{1,1} C 1 , 1 , each component is μ 0 \mu_0 μ 0 -strongly concave in y y y , and their global joint RMS smoothness constant is at most
L r o b ≤ 601 λ + 10 8 d + 32 ⋅ 128 ⋅ 231000 b 2 / ν + 13000 b + 13 2 a n + 741 c + 47 ν < 4 L / 5. (57) L_{\mathrm{rob}} \le601\lambda+ 10^{8}d + 32 \cdot128 \cdot231000 b^2/\nu+ 13000b + \frac{13}{2}a\sqrt{n} + 741c + 47\nu< 4L/5. \tag*{(57)} L rob ≤ 601 λ + 1 0 8 d + 32 ⋅ 128 ⋅ 231000 b 2 / ν + 13000 b + 2 13 a n + 741 c + 47 ν < 4 L /5. (57) The pure primal Hessian also obeys, almost everywhere,
∥ ∇ P P 2 f i , η ∥ ≤ K P : = 601 λ + 10 8 d + 32 ⋅ 128 ⋅ 231000 b 2 / ν < L / 32. (58) \lVert\nabla^2_{PP} f_{i,\eta} \rVert\le K_P := 601\lambda+ 10^{8}d + 32 \cdot128 \cdot231000 b^2/\nu< L/32. \tag*{(58)} ∥ ∇ P P 2 f i , η ∥ ≤ K P := 601 λ + 1 0 8 d + 32 ⋅ 128 ⋅ 231000 b 2 / ν < L /32. (58) These bounds hold for every owner assignment, all dimensions and 0 ≤ η ≤ R ∗ / ( 16 n ) 0 \le\eta\le R_*/(16\sqrt{n}) 0 ≤ η ≤ R ∗ / ( 16 n ) , including when any subset of axes is replaced by zero consistently in its primal and dual occurrences.
Proof. Strong concavity. Lemma 8 (iii) makes each owned edge plus its shared correction concave; unowned corrections are concave. Apply part (i) to ( A , A 0 ) = ( c ϕ η ( ⟨ U , y v ⟩ ) , c ) (A,A_0)=(c\phi_\eta(\langle U,y_v\rangle),c) ( A , A 0 ) = ( c ϕ η (⟨ U , y v ⟩) , c ) at each node and ( b ⟨ A ( P ) , Y ∂ ⟩ , b R / R ∗ ) (b\langle A(P),Y_{\partial}\rangle,bR/R_*) ( b ⟨ A ( P ) , Y ∂ ⟩ , b R / R ∗ ) at the endpoints. Since b R / R ∗ = ν / 16 bR/R_*=\nu/16 b R / R ∗ = ν /16 , the barriers remove the extra curvature and leave ∇ y y 2 f i , η ⪯ − ν I + c U U ⊤ ⪯ − μ 0 I \nabla^2_{yy}f_{i,\eta}\preceq-\nu I+cUU^\top\preceq-\mu_0I ∇ y y 2 f i , η ⪯ − ν I + c U U ⊤ ⪯ − μ 0 I .
Common terms. For the norm estimates, put A 1 = 2433 A_1=2433 A 1 = 2433 , A 2 = 231000 A_2=231000 A 2 = 231000 , C 1 = 15 / 8 C_1=15/8 C 1 = 15/8 and C 2 = 10 / 3 C_2=10/3 C 2 = 10/3 . Direct differentiation gives ∥ ∇ χ ∗ ∥ ≤ C 1 / R ∗ \lVert\nabla\chi_*\rVert\le C_1/R_* ∥ ∇ χ ∗ ∥ ≤ C 1 / R ∗ , ∥ ∇ 2 χ ∗ ∥ ≤ C 2 / R ∗ 2 \lVert\nabla^2\chi_*\rVert\le C_2/R_*^2 ∥ ∇ 2 χ ∗ ∥ ≤ C 2 / R ∗ 2 and ∥ ∇ 2 ψ ∗ ∥ ≤ 1 \lVert\nabla^2\psi_*\rVert\le1 ∥ ∇ 2 ψ ∗ ∥ ≤ 1 . Also 0 ≤ Ψ < 3 0\le\Psi<3 0 ≤ Ψ < 3 , ∣ Ψ ′ ∣ < 5 |\Psi'|<5 ∣ Ψ ′ ∣ < 5 , ∣ Ψ ′ ′ ∣ < 100 |\Psi''|<100 ∣ Ψ ′′ ∣ < 100 and 0 < Φ < 5 0<\Phi<5 0 < Φ < 5 , ∣ Φ ′ ∣ < 2 |\Phi'|<2 ∣ Φ ′ ∣ < 2 , ∣ Φ ′ ′ ∣ ≤ 1 |\Phi''|\le1 ∣ Φ ′′ ∣ ≤ 1 . In particular, for v = ( 2 t − 1 ) − 2 v=(2t-1)^{-2} v = ( 2 t − 1 ) − 2 , Ψ ′ ′ = 8 e 1 − v v 2 ( 2 v − 3 ) \Psi''=8e^{1-v}v^2(2v-3) Ψ ′′ = 8 e 1 − v v 2 ( 2 v − 3 ) gives ∣ Ψ ′ ′ ∣ ≤ 432 / e 2 + 96 / e < 100 |\Psi''|\le432/e^2+96/e<100 ∣ Ψ ′′ ∣ ≤ 432/ e 2 + 96/ e < 100 by maximizing powers times exponentials. The source-pair Hessian entries are bounded by 500, 3, 10, hence its norm is below 510; composition with the caps adds less than ( 3 / 128 ) 6 2 + 25 2 < 1 (3/128)\sqrt{6^2+25^2}<1 ( 3/128 ) 6 2 + 2 5 2 < 1 and the penalty adds λ / 512 \lambda/512 λ /512 . Together with Lemma 6, this gives the following bounds for the common terms (the endpoint forcing includes its cutoff):
Table 2.
On node cutoff supports, ∣ ϕ n ∣ ≤ 2 R ∗ 2 |\phi_n| \le2R_*^2 ∣ ϕ n ∣ ≤ 2 R ∗ 2 and ∥ ∇ ϕ n ∥ ≤ 2 R ∗ \|\nabla\phi_n\| \le2R_* ∥∇ ϕ n ∥ ≤ 2 R ∗ ; node blocks are disjoint. The effective endpoint vector has norm at most 2 R ∗ 2R_* 2 R ∗ . The symmetric mixed block has norm equal to that of its rectangular P y Py P y block.
Owned edges. Write H e H_e H e for the unamplified pair Hessian, ∥ H e ∥ ≤ C a = 9 a / 4 \|H_e\| \le C_a = 9a/4 ∥ H e ∥ ≤ C a = 9 a /4 . Degree two and blockwise Cauchy–Schwarz give
∑ i ∥ ∑ e : ∑ e = i H e w e ∥ 2 ≤ 2 ∑ e ∥ H e w e ∥ 2 ≤ 4 C a 2 ∥ w ∥ 2 . \sum_i \left\|\sum_{e:\sum e=i} H_e w_e\right\|^2 \le2\sum_e \|H_e w_e\|^2 \le4C_a^2\|w\|^2. i ∑ e : ∑ e = i ∑ H e w e 2 ≤ 2 e ∑ ∥ H e w e ∥ 2 ≤ 4 C a 2 ∥ w ∥ 2 .
Amplification by n n n and RMS averaging cost 2 n C a 2\sqrt{n}C_a 2 n C a ; the shared pair barriers cost 2 a n 2a\sqrt{n} 2 a n by the same degree bound. Their sum is ( 13 / 2 ) a n (13/2)a\sqrt{n} ( 13/2 ) a n , uniformly over owner assignments and lengths. Minkowski’s inequality in the component-indexed Hilbert space now proves (57); the three P P PP P P rows give (58). All terms are locally C 1 , 1 C^{1,1} C 1 , 1 . Mollification preserves both component curvature and the stacked-gradient bound for n − 1 / 2 ( ∇ f 1 , … , ∇ f n ) n^{-1/2}(\nabla f_1,\ldots,\nabla f_n) n − 1/2 ( ∇ f 1 , … , ∇ f n ) ; line integration and passage to the limit give the global claims across every hinge and join.
Numerical constants. (29) and λ ≤ δ 0 L \lambda\le\delta_0L λ ≤ δ 0 L bound λ \lambda λ , d / 6 d/6 d /6 and b 2 / ( 12 ν ) b^2/(12\nu) b 2 / ( 12 ν ) by δ 0 L \delta_0L δ 0 L , so K P / L ≤ ( 601 + 6 ⋅ 10 8 + 12 ⋅ 32 ⋅ 128 ⋅ 231000 ) δ 0 < 2 34 δ 0 = 1 / 32 K_P/L \le(601+6\cdot10^8+12\cdot32\cdot128\cdot231000)\delta_0 < 2^{34}\delta_0 = 1/32 K P / L ≤ ( 601 + 6 ⋅ 1 0 8 + 12 ⋅ 32 ⋅ 128 ⋅ 231000 ) δ 0 < 2 34 δ 0 = 1/32 . Since κ ≥ 64 \kappa\ge64 κ ≥ 64 and ν / L ≤ 1 / 64 + 2 − 16 < 1 / 32 \nu/L \le1/64+2^{-16}<1/32 ν / L ≤ 1/64 + 2 − 16 < 1/32 , the joint bound is
L r o b L ≤ 1 32 + 13000 12 δ 0 / 32 + 47 64 + 788 + 13 / 2 2 16 < 0.79 < 4 5 . \frac{L_{\mathrm{rob}}}{L} \le\frac{1}{32}+13000\sqrt{12\delta_0}/32+\frac{47}{64}+\frac{788+13/2}{2^{16}}<0.79<\frac{4}{5}. L L rob ≤ 32 1 + 13000 12 δ 0 /32 + 64 47 + 2 16 788 + 13/2 < 0.79 < 5 4 .
This includes the public dual ridge and endpoint correction 47 ν 47\nu 47 ν .
Zeroed axes. Every estimate used only axis norms at most one; zeroing axes preserves the source, forcing and derivative bounds, while node curvature remains at most c I cI c I and edge vectors have norm at most 2 \sqrt{2} 2 . Retain all public radial corrections and the full ambient endpoint forcing. The same curvature and RMS bounds hold for these zero-axis formulas, used as censored objectives in Appendix C.
The Value Function and Stationarity Barrier
We first bound the gradient of the zero-width value function V 0 V_0 V 0 near the end of the stage chain (Lemma 10). A positive width then preserves this barrier, keeps the optimizer inside radius 3 R ∗ / 16 3R_*/16 3 R ∗ /16 , and gives the initial budgets (Lemma 11).
A Uniform Terminal Gradient Bound
Lemma 10 (Uniform terminal-band barrier). There is a universal c ⋆ > 0 c_\star>0 c ⋆ > 0 , depending only on the displayed scalar functions and normalized radius 128, such that, for every finite length, dimension, owner assignment, and finite uncompressed primal point,
∣ z M + 1 ∣ ≤ s / 8 ⟹ ∥ ∇ V 0 ( P ) ∥ ≥ c ⋆ λ s . (59) |z_{M+1}| \le s/8 \quad\Longrightarrow\quad\|\nabla V_0(P)\| \ge c_\star\lambda s. \tag*{(59)} ∣ z M + 1 ∣ ≤ s /8 ⟹ ∥∇ V 0 ( P ) ∥ ≥ c ⋆ λ s . (59)
The gradient is in the uncompressed primal coordinates, with arbitrary radii, signs, and orthogonal complements of the axes.
Proof.
Outline. After normalization the value is a chain in which block j j j carries a mass q j ∈ [ 0 , 1 ] q_j \in[0,1] q j ∈ [ 0 , 1 ] that can only decrease along the chain. We show that at an approximately stationary point the mass never falls below a fixed level ℓ \ell ℓ , and that a block of mass at least ℓ \ell ℓ cannot be approximately stationary when its last coordinate lies in the terminal band. Step 3 proves both facts for one or two blocks with exactly vanishing residuals; compactness makes them quantitative (Step 4), and induction along the chain concludes (Step 5).
Step 1: residuals in compressed coordinates. Normalize lengths by s s s and values by λ s 2 \lambda s^2 λ s 2 , keeping the same symbols for the normalized quantities, and write R 0 = 128 R_0=128 R 0 = 128 . The cap is a diffeomorphism onto the open radius-R 0 R_0 R 0 ball. In compressed coordinates its primal penalty is
π ( r ) = 1 1024 ( r 1 − r 2 / R 0 2 − R 0 ) 2 , 0 ≤ r < R 0 . \pi(r)=\frac{1}{1024}\left(\frac{r}{\sqrt{1-r^2/R_0^2}}-R_0\right)^2,\qquad0\le r<R_0. π ( r ) = 1024 1 ( 1 − r 2 / R 0 2 r − R 0 ) 2 , 0 ≤ r < R 0 .
Set τ ( r ) = π ′ ( r ) / r \tau(r)=\pi'(r)/r τ ( r ) = π ′ ( r ) / r , with τ ( 0 ) = 0 \tau(0)=0 τ ( 0 ) = 0 . It is continuous, nonnegative, and positive exactly for r > R 0 / 2 r>R_0/\sqrt{2} r > R 0 / 2 . The value is
V = J M + ∑ j q j [ 1 − α ( z j + 1 ) ] Q j + ∑ j π ( ∥ w j ∥ ) , V=J_M+\sum_j q_j[1-\alpha(z_{j+1})]Q_j+\sum_j\pi(\lVert w_j\rVert), V = J M + j ∑ q j [ 1 − α ( z j + 1 )] Q j + j ∑ π (∥ w j ∥) ,
Q j = 6 ( x j − z j + 1 / 2 ) 2 + 6 ∥ A j , ∥ ∥ 2 + 3 2 ∥ B j + 1 , ∥ ∥ 2 . Q_j=6(x_j-z_{j+1}/2)^2+6\lVert A_{j,\parallel}\rVert^2+\frac{3}{2}\lVert B_{j+1,\parallel}\rVert^2. Q j = 6 ( x j − z j + 1 /2 ) 2 + 6 ∥ A j , ∥ ∥ 2 + 2 3 ∥ B j + 1 , ∥ ∥ 2 .
Define conditional energies by
E j = [ 1 − α ( z j + 1 ) ] Q j + α ( z j + 1 ) E j + 1 , E M + 1 = 0. E_j=[1-\alpha(z_{j+1})]Q_j+\alpha(z_{j+1})E_{j+1},\qquad E_{M+1}=0. E j = [ 1 − α ( z j + 1 )] Q j + α ( z j + 1 ) E j + 1 , E M + 1 = 0.
The quadratic matrix has norm at most 12 12 12 , so telescoping gives 0 ≤ Q j , E j ≤ E max : = 12 R 0 2 0\le Q_j,E_j\le E_{\max}:=12R_0^2 0 ≤ Q j , E j ≤ E m a x := 12 R 0 2 , including arbitrary complements. The source coefficient in block j j j is β 1 = 1 \beta_1=1 β 1 = 1 and β j = Ψ ( z j ) + Ψ ( − z j ) \beta_j=\Psi(z_j)+\Psi(-z_j) β j = Ψ ( z j ) + Ψ ( − z j ) for j ≥ 2 j\ge2 j ≥ 2 . At a nonterminal connector t = z j + 1 t=z_{j+1} t = z j + 1 , with Y = x j + 1 Y=x_{j+1} Y = x j + 1 , its source derivative is − B ( t , Y ) -B(t,Y) − B ( t , Y ) , where
B ( t , Y ) = Ψ ′ ( − t ) Φ ( − Y ) + Ψ ′ ( t ) Φ ( Y ) ∈ [ 0 , 25 ] . B(t,Y)=\Psi'(-t)\Phi(-Y)+\Psi'(t)\Phi(Y)\in[0,25]. B ( t , Y ) = Ψ ′ ( − t ) Φ ( − Y ) + Ψ ′ ( t ) Φ ( Y ) ∈ [ 0 , 25 ] .
At the terminal connector it is zero. Put q 0 = 1 q_0=1 q 0 = 1 . If q j > 0 q_j>0 q j > 0 , every incoming factor is positive and its connector exceeds 1 / 2 1/2 1/2 ; hence, also for block one,
0 < q j ≤ q j − 1 ≤ 1 , q j C q j − 1 ≤ β j ≤ e q j q j − 1 , C = 23. 0<q_j\le q_{j-1}\le1,\qquad\frac{q_j}{Cq_{j-1}}\le\beta_j\le\frac{e q_j}{q_{j-1}},\qquad C=23. 0 < q j ≤ q j − 1 ≤ 1 , C q j − 1 q j ≤ β j ≤ q j − 1 e q j , C = 23.
At a current block abbreviate x = x j x=x_j x = x j , t = z j + 1 t=z_{j+1} t = z j + 1 , q = q j q=q_j q = q j , q p r e v = q j − 1 q_{\mathrm{prev}}=q_{j-1} q prev = q j − 1 , H = q [ 1 − α ( t ) ] H=q[1-\alpha(t)] H = q [ 1 − α ( t )] , D = x − t / 2 D=x-t/2 D = x − t /2 , E = E j + 1 E=E_{j+1} E = E j + 1 , and τ = τ ( ∥ w j ∥ ) \tau=\tau(\lVert w_j\rVert) τ = τ (∥ w j ∥) . All compressed residuals are as follows; the suffix enters only through the bounded scalar E E E :
R x = − β Φ ′ ( x ) + 12 H D + τ x , R_x=-\beta\Phi'(x)+12HD+\tau x, R x = − β Φ ′ ( x ) + 12 H D + τ x ,
R t = − B − 6 H D + q α ′ ( t ) ( E − Q ) + τ t , R_t=-B-6HD+q\alpha'(t)(E-Q)+\tau t, R t = − B − 6 H D + q α ′ ( t ) ( E − Q ) + τ t ,
∇ A ⊥ V = ( 12 H + τ ) A ⊥ , ∇ B ⊥ V = ( 3 H + τ ) B ⊥ . \nabla_{A_\perp}V=(12H+\tau)A_\perp,\qquad\nabla_{B_\perp}V=(3H+\tau)B_\perp. ∇ A ⊥ V = ( 12 H + τ ) A ⊥ , ∇ B ⊥ V = ( 3 H + τ ) B ⊥ .
Step 2: a priori bounds on the block states. Before the inverse-cap penalty, the gradient of each compressed block has norm at most B 0 = 26 + 24 R 0 + ( 23 ⋅ 5 ) ( 12 R 0 2 ) B_0=26+24R_0+(23\cdot5)(12R_0^2) B 0 = 26 + 24 R 0 + ( 23 ⋅ 5 ) ( 12 R 0 2 ) , the sum of bounds for the source, the direct quadratic and the gate derivative. If the full normalized uncompressed gradient has norm at most one, ∥ D ρ ∥ ≤ 1 \lVert D\rho\rVert\le1 ∥ D ρ ∥ ≤ 1 implies ρ ′ ( ∥ P j ∥ ) ≤ B 0 + 1 \rho'(\lVert P_j\rVert) \le B_0+1 ρ ′ (∥ P j ∥) ≤ B 0 + 1 , and therefore
∥ P j ∥ ≤ R ‾ : = R 0 + 512 ( B 0 + 1 ) , ∥ w j ∥ ≤ r : = R 0 R ‾ R 0 2 + R ‾ 2 < R 0 , \lVert P_j\rVert\le\overline{R}:=R_0+512(B_0+1),\qquad\lVert w_j\rVert\le r:=\frac{R_0\overline{R}}{\sqrt{R_0^2+\overline{R}^2}}<R_0, ∥ P j ∥ ≤ R := R 0 + 512 ( B 0 + 1 ) , ∥ w j ∥ ≤ r := R 0 2 + R 2 R 0 R < R 0 ,
σ min ( D ρ ( P j ) ) ≥ σ 0 : = ( 1 + R ‾ 2 / R 0 2 ) − 3 / 2 > 0. \sigma_{\min}(D\rho(P_j))\ge\sigma_0:=(1+\overline{R}^2/R_0^2)^{-3/2}>0. σ m i n ( D ρ ( P j )) ≥ σ 0 := ( 1 + R 2 / R 0 2 ) − 3/2 > 0.
Every compressed block gradient norm is consequently at most ∥ ∇ V ∥ / σ 0 \lVert\nabla V\rVert/\sigma_0 ∥ ∇ V ∥ / σ 0 .
Represent a compressed block by ( x , t , ξ , ζ ) (x,t,\xi,\zeta) ( x , t , ξ , ζ ) , with ξ = ∥ A ⊥ ∥ \xi=\lVert A_\perp\rVert ξ = ∥ A ⊥ ∥ and ζ = ∥ B ⊥ ∥ \zeta=\lVert B_\perp\rVert ζ = ∥ B ⊥ ∥ :
ξ , ζ ≥ 0 , r 2 = x 2 + t 2 + ξ 2 + ζ 2 ≤ r ‾ 2 , τ = π ′ ( r ) / r , Q = 6 ( x − t / 2 ) 2 + 6 ξ 2 + 3 2 ζ 2 . (60) \xi,\zeta\ge0,\qquad r^2=x^2+t^2+\xi^2+\zeta^2\le\overline{r}^2,\qquad\tau=\pi'(r)/r,\qquad Q=6(x-t/2)^2+6\xi^2+\frac{3}{2}\zeta^2. \tag*{(60)} ξ , ζ ≥ 0 , r 2 = x 2 + t 2 + ξ 2 + ζ 2 ≤ r 2 , τ = π ′ ( r ) / r , Q = 6 ( x − t /2 ) 2 + 6 ξ 2 + 2 3 ζ 2 . (60)
Let R = max { ∣ R x ∣ , ∣ R t ∣ , ( 12 H + τ ) ξ , ( 3 H + τ ) ζ } \mathcal{R}=\max\{|R_x|,|R_t|,(12H+\tau)\xi,(3H+\tau)\zeta\} R = max { ∣ R x ∣ , ∣ R t ∣ , ( 12 H + τ ) ξ , ( 3 H + τ ) ζ } . Allowing both complementary norms only enlarges this dimension-free set. The statements below use the relations
0 < q ≤ q p r e v ≤ 1 , q C q p r e v ≤ β ≤ e q q p r e v . (61) 0<q\le q_{\mathrm{prev}}\le1,\qquad\frac{q}{Cq_{\mathrm{prev}}}\le\beta\le\frac{eq}{q_{\mathrm{prev}}}. \tag*{(61)} 0 < q ≤ q prev ≤ 1 , C q prev q ≤ β ≤ q prev e q . (61)
Fix
q ∗ = ( 24 C R 0 2 ) − 1 , β ∗ = e − 99 , m ∗ = q ∗ β ∗ , ℓ = m ∗ / 4 , E R = e − 4095 , p e x t = e E R / 540. q_*=(24CR_0^2)^{-1},\qquad\beta_*=e^{-99},\qquad m_*=q_*\beta_*,\qquad\ell=m_*/4,\qquad E_R=e^{-4095},\qquad p_{\mathrm{ext}}=eE_R/540. q ∗ = ( 24 C R 0 2 ) − 1 , β ∗ = e − 99 , m ∗ = q ∗ β ∗ , ℓ = m ∗ /4 , E R = e − 4095 , p ext = e E R /540.
Here β ∗ = Ψ ( 11 / 20 ) \beta_*=\Psi(11/20) β ∗ = Ψ ( 11/20 ) , and E R E_R E R bounds Φ ′ ( x ) = e e − x 2 / 2 \Phi'(x)=\sqrt{e}\,e^{-x^2/2} Φ ′ ( x ) = e e − x 2 /2 when x 2 > R 0 2 / 2 − 1 x^2>R_0^2/2-1 x 2 > R 0 2 /2 − 1 , the region where an aligned block meets the active penalty. In particular 0 < p e x t < e − 4100 < ℓ < m ∗ / 2 < m ∗ < q ∗ < 1 0<p_{\mathrm{ext}}<e^{-4100}<\ell<m_*/2<m_*<q_*<1 0 < p ext < e − 4100 < ℓ < m ∗ /2 < m ∗ < q ∗ < 1 .
Step 3: exact local statements. We prove three statements about blocks whose residuals vanish exactly: (a) if ℓ ≤ q ≤ q ∗ \ell\le q\le q_* ℓ ≤ q ≤ q ∗ , q p r e v ≥ ℓ q_{\mathrm{prev}}\ge\ell q prev ≥ ℓ and the residuals of a nonterminal block and its successor vanish, then t > 1 t>1 t > 1 , so the mass does not drop; (b) if q , q p r e v ≥ q ∗ q,q_{\mathrm{prev}}\ge q_* q , q prev ≥ q ∗ and the residuals of a nonterminal block vanish, then q α ( t ) ≥ m ∗ q\alpha(t)\ge m_* q α ( t ) ≥ m ∗ ; (c) if q , q p r e v ≥ ℓ q,q_{\mathrm{prev}}\ge\ell q , q prev ≥ ℓ and ∣ t ∣ ≤ 1 / 8 |t|\le1/8 ∣ t ∣ ≤ 1/8 , the residuals of the terminal block do not all vanish. We begin with facts about a single block. Assume that all current residuals vanish and q , β > 0 q,\beta>0 q , β > 0 . For t < 1 t<1 t < 1 , H > 0 H>0 H > 0 eliminates the complements. For 0 < t < 1 0<t<1 0 < t < 1 the source equation forces x > t / 2 x>t/2 x > t /2 . Indeed, x ≤ 0 x\le0 x ≤ 0 is impossible, and 0 < x ≤ t / 2 0<x\le t/2 0 < x ≤ t /2 has radius below two, hence τ = 0 \tau=0 τ = 0 , also impossible. At t = 1 t=1 t = 1 , τ x = β Φ ′ ( x ) > 0 \tau x=\beta\Phi'(x)>0 τ x = β Φ ′ ( x ) > 0 forces x , τ > 0 x,\tau>0 x , τ > 0 and zero complements. For t ≤ 0 t\le0 t ≤ 0 , α = α ′ = 0 \alpha=\alpha'=0 α = α ′ = 0 and the source equation forces D > 0 D>0 D > 0 , whence
R t = − B − 6 q D + τ t < 0. R_t=-\mathcal{B}-6qD+\tau t<0. R t = − B − 6 q D + τ t < 0.
For 0 < t ≤ 1 / 2 0<t\le1/2 0 < t ≤ 1/2 , B = 0 \mathcal{B}=0 B = 0 and τ t = 6 q D \tau t=6qD τ t = 6 q D , so τ > 0 \tau>0 τ > 0 . Every block with 0 < t < 1 0<t<1 0 < t < 1 and τ > 0 \tau>0 τ > 0 satisfies
x > R 0 2 / 2 − 1 , D > x − 1 / 2 > 90 , Φ ′ ( x ) < E R . (62) x>\sqrt{R_0^2/2-1},\qquad D>x-1/2>90,\qquad\Phi'(x)<E_R. \tag*{(62)} x > R 0 2 /2 − 1 , D > x − 1/2 > 90 , Φ ′ ( x ) < E R . (62)
Substitute 12 q ( 1 − α ) D ≤ β Φ ′ ( x ) 12q(1-\alpha)D\le\beta\Phi'(x) 12 q ( 1 − α ) D ≤ β Φ ′ ( x ) and β ≤ e q / q p r e v \beta\le eq/q_{\mathrm{prev}} β ≤ e q / q prev to obtain
q p r e v [ 1 − α ( t ) ] < e E R / 1080 , α ( t ) ≤ 1 / 2 ⟹ q p r e v < p e x t . (63) q_{\mathrm{prev}}[1-\alpha(t)]<eE_R/1080,\qquad\alpha(t)\le1/2\Longrightarrow q_{\mathrm{prev}}<p_{\mathrm{ext}}. \tag*{(63)} q prev [ 1 − α ( t )] < e E R /1080 , α ( t ) ≤ 1/2 ⟹ q prev < p ext . (63)
At a nonterminal connector t > 1 / 2 t>1/2 t > 1/2 , the adjacent data are
q ′ = q α ( t ) > 0 , β ′ = Ψ ( t ) + Ψ ( − t ) > 0. (64) q'=q\alpha(t)>0,\qquad\beta'=\Psi(t)+\Psi(-t)>0. \tag*{(64)} q ′ = q α ( t ) > 0 , β ′ = Ψ ( t ) + Ψ ( − t ) > 0. (64)
Use ( Y , u , ξ ′ , ζ ′ ) (Y,u,\xi',\zeta') ( Y , u , ξ ′ , ζ ′ ) for the next block, with its actual τ ′ \tau' τ ′ , Q ′ Q' Q ′ , H ′ = q ′ [ 1 − α ( u ) ] H'=q'[1-\alpha(u)] H ′ = q ′ [ 1 − α ( u )] and arbitrary farther B ′ ∈ [ 0 , 25 ] \mathcal{B}'\in[0,25] B ′ ∈ [ 0 , 25 ] , E ′ ∈ [ 0 , E max ] E'\in[0,E_{\max}] E ′ ∈ [ 0 , E m a x ] . If its residuals vanish, then Y > 0 Y>0 Y > 0 . For Y ≤ 0 Y\leq0 Y ≤ 0 , u ≥ 0 u\geq0 u ≥ 0 its source residual cannot vanish; for Y ≤ 0 Y\leq0 Y ≤ 0 , u < 0 u<0 u < 0 that equation forces D ′ = Y − u / 2 > 0 D'=Y-u/2>0 D ′ = Y − u /2 > 0 , making the connector residual − B ′ − 6 q ′ D ′ + τ ′ u < 0 -\mathcal{B}'-6q'D'+\tau'u<0 − B ′ − 6 q ′ D ′ + τ ′ u < 0 .
Proof of (a). Suppose ℓ ≤ q ≤ q ∗ \ell\leq q\leq q_* ℓ ≤ q ≤ q ∗ , q p r e v ≥ ℓ q_{\mathrm{prev}}\geq\ell q prev ≥ ℓ , and both blocks have zero residuals. We show t > 1 t>1 t > 1 . Nonpositive t t t is excluded above; 0 < t ≤ 1 / 2 0<t\leq1/2 0 < t ≤ 1/2 forces q p r e v < p e x t < ℓ q_{\mathrm{prev}}<p_{\mathrm{ext}}<\ell q prev < p ext < ℓ . For 1 / 2 < t < 1 1/2<t<1 1/2 < t < 1 , Y > 0 Y>0 Y > 0 gives Φ ( Y ) > 1 \Phi(Y)>1 Φ ( Y ) > 1 , whereas q α ′ ( t ) ∣ E − Q ∣ ≤ q ∗ C Ψ ′ ( t ) E max = Ψ ′ ( t ) / 2 q\alpha'(t)|E-Q|\leq q_*C\Psi'(t)E_{\max}=\Psi'(t)/2 q α ′ ( t ) ∣ E − Q ∣ ≤ q ∗ C Ψ ′ ( t ) E m a x = Ψ ′ ( t ) /2 . The connector equation and D > 0 D>0 D > 0 imply τ t > Ψ ′ ( t ) / 2 \tau t>\Psi'(t)/2 τ t > Ψ ′ ( t ) /2 . Then (62) and the source equation imply τ ≤ e E R / 90 \tau\leq eE_R/90 τ ≤ e E R /90 and Ψ ′ ( t ) ≤ e E R / 45 \Psi'(t)\leq eE_R/45 Ψ ′ ( t ) ≤ e E R /45 . Since Ψ ′ = 4 Ψ / ( 2 t − 1 ) 3 \Psi'=4\Psi/(2t-1)^3 Ψ ′ = 4Ψ/ ( 2 t − 1 ) 3 , we have α ( t ) ≤ C Ψ ′ ( t ) / 4 < 1 / 2 \alpha(t)\leq C\Psi'(t)/4<1/2 α ( t ) ≤ C Ψ ′ ( t ) /4 < 1/2 , contradicting (63). At t = 1 t=1 t = 1 , zero complements and the source equation give x > 90 x>90 x > 90 and τ ≤ e E R / 90 < 1 \tau\leq eE_R/90<1 τ ≤ e E R /90 < 1 . But α ( 1 ) = 1 \alpha(1)=1 α ( 1 ) = 1 and α ′ ( 1 ) = 0 \alpha'(1)=0 α ′ ( 1 ) = 0 give H = 0 H=0 H = 0 , so the connector equation yields
τ = B ( 1 , Y ) = Ψ ′ ( 1 ) Φ ( Y ) = 4 Φ ( Y ) > 4. \tau=\mathcal{B}(1,Y)=\Psi'(1)\Phi(Y)=4\Phi(Y)>4. τ = B ( 1 , Y ) = Ψ ′ ( 1 ) Φ ( Y ) = 4Φ ( Y ) > 4.
Proof of (b). If q , q p r e v ≥ q ∗ q,q_{\mathrm{prev}}\geq q_* q , q prev ≥ q ∗ and only the current nonterminal residuals vanish (with actual B ( t , Y ) \mathcal{B}(t,Y) B ( t , Y ) ), then
q α ( t ) ≥ m ∗ . (65) q\alpha(t)\geq m_*. \tag*{(65)} q α ( t ) ≥ m ∗ . (65)
For α ≥ 1 / 2 \alpha\geq1/2 α ≥ 1/2 this is immediate. Otherwise the single-block facts above force 1 / 2 < t < 1 1/2<t<1 1/2 < t < 1 , τ = 0 \tau=0 τ = 0 , D > 0 D>0 D > 0 . The source equation with β / q ≥ 1 / C \beta/q\geq1/C β / q ≥ 1/ C gives D ≥ 1 / ( 24 C ) D\geq1/(24C) D ≥ 1/ ( 24 C ) , since 0 < x < 3 / 2 0<x<3/2 0 < x < 3/2 and Φ ′ ( x ) > 1 / 2 \Phi'(x)>1/2 Φ ′ ( x ) > 1/2 when D < 1 D<1 D < 1 . Since α ( t ) < 1 / 2 \alpha(t)<1/2 α ( t ) < 1/2 gives H > q / 2 H>q/2 H > q /2 and B ≥ 0 \mathcal{B}\geq0 B ≥ 0 , the connector equation with τ = 0 \tau=0 τ = 0 gives q α ′ ( t ) ( E − Q ) = B + 6 H D ≥ 3 q D q\alpha'(t)(E-Q)=\mathcal{B}+6HD\geq3qD q α ′ ( t ) ( E − Q ) = B + 6 H D ≥ 3 q D , hence α ′ ( t ) ≥ D / ( 4 R 0 2 ) ≥ 1 / ( 96 C R 0 2 ) \alpha'(t)\geq D/(4R_0^2)\geq1/(96CR_0^2) α ′ ( t ) ≥ D / ( 4 R 0 2 ) ≥ 1/ ( 96 C R 0 2 ) . But for t ≤ 11 / 20 t\leq11/20 t ≤ 11/20 , α ′ ( t ) ≤ 4000 C e − 99 \alpha'(t)\leq4000Ce^{-99} α ′ ( t ) ≤ 4000 C e − 99 is smaller. Thus t > 11 / 20 t>11/20 t > 11/20 and α ( t ) ≥ Ψ ( t ) > e − 99 \alpha(t)\geq\Psi(t)>e^{-99} α ( t ) ≥ Ψ ( t ) > e − 99 .
Proof of (c). Here B = E = 0 \mathcal{B}=E=0 B = E = 0 . For q , q p r e v ≥ ℓ q,q_{\mathrm{prev}}\geq\ell q , q prev ≥ ℓ and ∣ t ∣ ≤ 1 / 8 |t|\leq1/8 ∣ t ∣ ≤ 1/8 , zero residuals are impossible, since nonpositive t t t was excluded and positive t t t is flat and forces q p r e v < p e x t q_{\mathrm{prev}}<p_{\mathrm{ext}} q prev < p ext .
Step 4: quantitative versions by compactness. Every current block obeys (60) and (61), with Q , τ Q,\tau Q , τ the actual functions of that block. Define:
(i) K J \mathcal{K}_J K J : q ∗ ≤ q ≤ q p r e v ≤ 1 q_*\leq q\leq q_{\mathrm{prev}}\leq1 q ∗ ≤ q ≤ q prev ≤ 1 , q α ( t ) ≤ m ∗ / 2 q\alpha(t)\leq m_*/2 q α ( t ) ≤ m ∗ /2 , 0 ≤ E ≤ E max 0\leq E\leq E_{\max} 0 ≤ E ≤ E m a x , ∣ Y ∣ ≤ r ˉ |Y|\leq\bar r ∣ Y ∣ ≤ r ˉ , and B = B ( t , Y ) \mathcal{B}=\mathcal{B}(t,Y) B = B ( t , Y ) . Its residual is R \mathcal{R} R ; statement (b), in the form (65), excludes a zero.
(ii) K L \mathcal{K}_L K L : ℓ ≤ q ≤ q ∗ \ell\leq q\leq q_* ℓ ≤ q ≤ q ∗ , q ≤ q p r e v ≤ 1 q\leq q_{\mathrm{prev}}\leq1 q ≤ q prev ≤ 1 , t ≤ 1 t\leq1 t ≤ 1 , 0 ≤ E ≤ E max 0\leq E\leq E_{\max} 0 ≤ E ≤ E m a x , and a second block ( Y , u , ξ ′ , ζ ′ ) (Y,u,\xi',\zeta') ( Y , u , ξ ′ , ζ ′ ) satisfying the same compact radius condition. Retain B = B ( t , Y ) \mathcal{B}=\mathcal{B}(t,Y) B = B ( t , Y ) and (64); allow independent E ′ ∈ [ 0 , E max ] E'\in[0,E_{\max}] E ′ ∈ [ 0 , E m a x ] , B ′ ∈ [ 0 , 25 ] \mathcal{B}'\in[0,25] B ′ ∈ [ 0 , 25 ] . The next residuals are
R Y = − β ′ Φ ′ ( Y ) + 12 H ′ ( Y − u / 2 ) + τ ′ Y , R_Y=-\beta'\Phi'(Y)+12H'(Y-u/2)+\tau'Y, R Y = − β ′ Φ ′ ( Y ) + 12 H ′ ( Y − u /2 ) + τ ′ Y ,
R u = − B ′ − 6 H ′ ( Y − u / 2 ) + q ′ α ′ ( u ) ( E ′ − Q ′ ) + τ ′ u , R_u=-\mathcal{B}'-6H'(Y-u/2)+q'\alpha'(u)(E'-Q')+\tau'u, R u = − B ′ − 6 H ′ ( Y − u /2 ) + q ′ α ′ ( u ) ( E ′ − Q ′ ) + τ ′ u ,
R ξ ′ = ( 12 H ′ + τ ′ ) ξ ′ , R ζ ′ = ( 3 H ′ + τ ′ ) ζ ′ , H ′ = q ′ [ 1 − α ( u ) ] . R_{\xi'}=(12H'+\tau')\xi',\qquad R_{\zeta'}=(3H'+\tau')\zeta',\qquad H'=q'[1-\alpha(u)]. R ξ ′ = ( 12 H ′ + τ ′ ) ξ ′ , R ζ ′ = ( 3 H ′ + τ ′ ) ζ ′ , H ′ = q ′ [ 1 − α ( u )] .
Its residual is the maximum of R \mathcal{R} R and the absolute values of these four quantities. For t ≤ 1 / 2 t\leq1/2 t ≤ 1/2 the current flat exclusions apply; otherwise next-block positivity and statement (a), including t = 1 t=1 t = 1 , rule out a zero. The coefficient β ′ = Ψ ( t ) + Ψ ( − t ) \beta'=\Psi(t)+\Psi(-t) β ′ = Ψ ( t ) + Ψ ( − t ) is defined also at $q'=0.
(iii) K T \mathcal{K}_T K T : ℓ ≤ q ≤ q p r e v ≤ 1 \ell\leq q\leq q_{\mathrm{prev}}\leq1 ℓ ≤ q ≤ q prev ≤ 1 , ∣ t ∣ ≤ 1 / 8 |t|\leq1/8 ∣ t ∣ ≤ 1/8 and B = E = 0 \mathcal{B}=E=0 B = E = 0 ; by (c), R \mathcal{R} R has no zero.
These are fixed-dimensional compact sets. Masses in denominators are bounded away from zero, r ˉ < R 0 \bar r<R_0 r ˉ < R 0 , and residuals are continuous at gate joins, penalty onset and zero complements. They cover every actual selected window, including a terminal next block through B ′ = E ′ = 0 B'=E'=0 B ′ = E ′ = 0 ; independent farther data only enlarge the sets. Let ϵ J , ϵ L , ϵ T > 0 \epsilon_J,\epsilon_L,\epsilon_T>0 ϵ J , ϵ L , ϵ T > 0 be their residual minima, with min ∅ = 1 \min\varnothing=1 min ∅ = 1 ; they are positive by the no-zero arguments. Thus residual below ϵ J \epsilon_J ϵ J forces q α ( t ) > m ∗ / 2 q\alpha(t)>m_*/2 q α ( t ) > m ∗ /2 in the high-mass range, two-block residual below ϵ L \epsilon_L ϵ L forces t > 1 t>1 t > 1 in the low-mass range, and the terminal-band residual is at least ϵ T \epsilon_T ϵ T . Set
ϵ l o c = min { 1 , ϵ J , ϵ L , ϵ T } , c ⋆ = 1 2 min { 1 , σ 0 ϵ l o c } > 0. \epsilon_{\mathrm{loc}}=\min\{1,\epsilon_J,\epsilon_L,\epsilon_T\},\qquad c_\star=\frac{1}{2}\min\{1,\sigma_0\epsilon_{\mathrm{loc}}\}>0. ϵ loc = min { 1 , ϵ J , ϵ L , ϵ T } , c ⋆ = 2 1 min { 1 , σ 0 ϵ loc } > 0.
Step 5: induction along the chain. Suppose ∥ ∇ V ∥ < c ⋆ \lVert\nabla V\rVert<c_\star ∥ ∇ V ∥ < c ⋆ and ∣ z M + 1 ∣ ≤ 1 / 8 \lvert z_{M+1}\rvert\leq1/8 ∣ z M + 1 ∣ ≤ 1/8 . Radius control makes every compressed residual smaller than ϵ l o c \epsilon_{\mathrm{loc}} ϵ loc . We inductively prove q j ≥ ℓ q_j\geq\ell q j ≥ ℓ , starting from q 1 = 1 q_1=1 q 1 = 1 . For j < M j<M j < M , if q j > q ∗ q_j>q_\ast q j > q ∗ , separation of K J \mathcal{K}_J K J gives q j + 1 > m ∗ / 2 > ℓ q_{j+1}>m_\ast/2>\ell q j + 1 > m ∗ /2 > ℓ . Otherwise ℓ ≤ q j ≤ q ∗ \ell\leq q_j\leq q_\ast ℓ ≤ q j ≤ q ∗ ; since q j − 1 ≥ q j q_{j-1}\geq q_j q j − 1 ≥ q j , separation of K L \mathcal{K}_L K L gives z j + 1 > 1 z_{j+1}>1 z j + 1 > 1 , so q j + 1 = q j α ( z j + 1 ) = q j ≥ ℓ q_{j+1}=q_j\alpha(z_{j+1})=q_j\geq\ell q j + 1 = q j α ( z j + 1 ) = q j ≥ ℓ . Thus q M − 1 ≥ q M ≥ ℓ q_{M-1}\geq q_M\geq\ell q M − 1 ≥ q M ≥ ℓ (with q 0 = q 1 = 1 q_0=q_1=1 q 0 = q 1 = 1 for M = 1 M=1 M = 1 ), and the terminal band contradicts separation of K T \mathcal{K}_T K T . Restoring units multiplies the primal gradient by λ s \lambda_s λ s , proving the result. ■
Positive Width, Exact Optimizer Localization, and Initialization
Fix the constant c ⋆ c_\star c ⋆ of the preceding barrier proof. Take
η 0 = c ⋆ λ s μ 0 12 A 1 a b K , η = min { η 0 , R ∗ / ( 16 n ) } > 0 , A 1 = 2433 , K = M m . (66) \eta_0=\frac{c_\star\lambda_s\mu_0}{12A_1ab\sqrt{K}},\qquad\eta=\min\{\eta_0,R_\ast/(16\sqrt{n})\}>0,\qquad A_1=2433,\qquad K=Mm. \tag*{(66)} η 0 = 12 A 1 ab K c ⋆ λ s μ 0 , η = min { η 0 , R ∗ / ( 16 n )} > 0 , A 1 = 2433 , K = M m . (66)
The width meets both the perturbation budget and the curvature condition of Lemma 8(iii), without changing the number of stages.
Lemma 11 (Uniform perturbation and initial budgets). For all finite raw P P P , the uncut maximizer of P ( P ) + b ⟨ A ( P ) , Y ∂ ⟩ − E η ( y ) \mathcal{P}(P)+b\langle A(P),Y_\partial\rangle-\mathcal{E}_\eta(y) P ( P ) + b ⟨ A ( P ) , Y ∂ ⟩ − E η ( y ) has every relevant node, edge tuple, and complete endpoint tuple strictly inside radius 3 R ∗ / 16 3R_\ast/16 3 R ∗ /16 . The node and endpoint cutoffs χ ∗ \chi_\ast χ ∗ equal one, the radial edge width equals the same constant η \eta η , and every radial barrier vanishes on a neighborhood of this point. It is therefore the unique maximizer of the modified mean as well. Its value function V η V_\eta V η obeys
∥ ∇ V η ( P ) − ∇ V 0 ( P ) ∥ ≤ c ⋆ λ s / 4 , (67) \lVert\nabla V_\eta(P)-\nabla V_0(P)\rVert\leq c_\star\lambda_s/4, \tag*{(67)} ∥ ∇ V η ( P ) − ∇ V 0 ( P )∥ ≤ c ⋆ λ s /4 , (67) ∣ V η ( P ) − V 0 ( P ) ∣ ≤ ( 32 c ⋆ / A 1 ) λ s 2 < 0.014 λ s 2 . (68) \lvert V_\eta(P)-V_0(P)\rvert\leq(32c_\star/A_1)\lambda_s^2<0.014\lambda_s^2. \tag*{(68)} ∣ V η ( P ) − V 0 ( P )∣ ≤ ( 32 c ⋆ / A 1 ) λ s 2 < 0.014 λ s 2 . (68) Consequently
∣ z M + 1 ∣ ≤ s / 8 ⟹ ∥ ∇ V η ( P ) ∥ ≥ 3 4 c ⋆ λ s = 6 ϵ . (69) \lvert z_{M+1}\rvert\leq s/8\quad\Longrightarrow\quad\lVert\nabla V_\eta(P)\rVert\geq\frac{3}{4}c_\star\lambda_s=6\epsilon. \tag*{(69)} ∣ z M + 1 ∣ ≤ s /8 ⟹ ∥ ∇ V η ( P )∥ ≥ 4 3 c ⋆ λ s = 6 ϵ . (69) At the supplied origin y ( 0 ) = 0 y^{(0)}=0 y ( 0 ) = 0 ,
∥ ∇ F η ( 0 , 0 ) ∥ 2 = e λ 2 s 2 , (70) \lVert\nabla F_\eta(0,0)\rVert^2=e\lambda^2s^2, \tag*{(70)} ∥ ∇ F η ( 0 , 0 ) ∥ 2 = e λ 2 s 2 , (70) V η ( 0 ) − inf P V η ( P ) ≤ ( 5 e M + 0.014 ) λ s 2 , (71) V_\eta(0)-\inf_P V_\eta(P)\leq(5e^M+0.014)\lambda_s^2, \tag*{(71)} V η ( 0 ) − P inf V η ( P ) ≤ ( 5 e M + 0.014 ) λ s 2 , (71) G 0 : = V η ( 0 ) − inf P F η ( P , 0 ) ≤ 5 e M λ s 2 + d R 2 < ∞ . (72) G_0:=V_\eta(0)-\inf_P F_\eta(P,0)\leq5e^M\lambda_s^2+dR^2<\infty. \tag*{(72)} G 0 := V η ( 0 ) − P inf F η ( P , 0 ) ≤ 5 e M λ s 2 + d R 2 < ∞. (72) The primal–dual gap G 0 G_0 G 0 is independent of η \eta η .
Proof Localization. The clipping identity ϕ η ′ ( t ) = t − clip η ( t ) \phi_\eta'(t)=t-\operatorname{clip}_\eta(t) ϕ η ′ ( t ) = t − clip η ( t ) and ∥ D U ∥ ≤ 2 \lVert D_U\rVert\le2 ∥ D U ∥ ≤ 2 give uniformly on the entire dual space
∥ ∇ E η − ∇ E 0 ∥ ≤ ( c + 2 a ) η K ≤ 3 a η K . \lVert\nabla\mathcal{E}_\eta-\nabla\mathcal{E}_0\rVert\le(c+2a)\eta\sqrt{K}\le3a\eta\sqrt{K}. ∥ ∇ E η − ∇ E 0 ∥ ≤ ( c + 2 a ) η K ≤ 3 a η K .
Strong monotonicity of either uncut energy therefore yields
∥ y η u n c u t ( P ) − y 0 ( P ) ∥ ≤ 3 a η K μ 0 ≤ c ⋆ λ s 4 A 1 b . (73) \lVert y_\eta^{\mathrm{uncut}}(P)-y_0(P)\rVert\le\frac{3a\eta\sqrt{K}}{\mu_0}\le\frac{c_\star\lambda s}{4A_1b}. \tag*{(73)} ∥ y η uncut ( P ) − y 0 ( P )∥ ≤ μ 0 3 a η K ≤ 4 A 1 b c ⋆ λ s . (73)
The ratio of this error to b R / ν bR/\nu b R / ν is at most c ⋆ / ( 6144 A 1 ) < 1 c_\star/(6144A_1)<1 c ⋆ / ( 6144 A 1 ) < 1 , since R = 128 s R=128s R = 128 s and b 2 / ν ≥ 12 λ b^2/\nu\ge12\lambda b 2 / ν ≥ 12 λ . Lemma 7 now puts every relevant node, pair and complete endpoint tuple strictly within 3 b R / ν = 3 R ⋆ / 16 < R ⋆ / 2 3bR/\nu=3R_\star/16<R_\star/2 3 b R / ν = 3 R ⋆ /16 < R ⋆ /2 . All node and endpoint cutoffs χ ⋆ \chi_\star χ ⋆ are identically one, every edge threshold is exactly η \eta η , and all radial barriers vanish on a neighborhood of this point. Its uncut first-order condition is thus also the localized first-order condition. Global mean strong concavity makes it the unique global localized maximizer.
Gradients. Both maxima are unique and the objective is continuously differentiable, including at all hinges, so Danskin’s theorem gives the value-function gradients. Their exact difference is
b [ D A ( P ) ] ⊤ Π ∂ [ y η ( P ) − y 0 ( P ) ] , b[DA(P)]^\top\Pi_{\partial}[y_\eta(P)-y_0(P)], b [ D A ( P ) ] ⊤ Π ∂ [ y η ( P ) − y 0 ( P )] ,
where Π ∂ \Pi_{\partial} Π ∂ has norm one. The forcing bound and (73) prove (67), and the preceding barrier proves (69).
Values. Since ∣ ϕ η ( t ) − t 2 / 2 ∣ ≤ η ∣ t ∣ |\phi_\eta(t)-t^2/2|\le\eta|t| ∣ ϕ η ( t ) − t 2 /2∣ ≤ η ∣ t ∣ , we have ∣ E η ( y ) − E 0 ( y ) ∣ ≤ 3 a η K ∥ y ∥ |\mathcal{E}_\eta(y)-\mathcal{E}_0(y)|\le3a\eta\sqrt{K}\lVert y\rVert ∣ E η ( y ) − E 0 ( y ) ∣ ≤ 3 a η K ∥ y ∥ . The gradients of both energies vanish at zero; strong monotonicity therefore bounds each uncut optimizer’s norm by b R / μ 0 bR/\mu_0 b R / μ 0 . Evaluating either variational problem at the other’s optimizer proves (68).
Initial budgets. At the origin A = 0 \mathcal{A}=0 A = 0 and the unique maximizer is zero; the only nonzero initial derivative is − e λ s U 1 , 1 -\sqrt{e}\lambda sU_{1,1} − e λ s U 1 , 1 in the first primal subblock. The values with positive and zero width coincide there. Since J M ≥ − 5 e M J_M\ge-5eM J M ≥ − 5 e M , the nonnegative quadratic and penalty terms in (51), combined with (68), give (71). Finally
F η ( P , 0 ) = λ s 2 J M ( P ) − d ∥ A ( P ) ∥ 2 + ∑ j p ( P j ) , F_\eta(P,0)=\lambda s^2J_M(P)-d\lVert\mathcal{A}(P)\rVert^2+\sum_j p(P_j), F η ( P , 0 ) = λ s 2 J M ( P ) − d ∥ A ( P ) ∥ 2 + j ∑ p ( P j ) ,
which is width-independent and bounded below by − 5 e M λ s 2 − d R 2 -5eM\lambda s^2-dR^2 − 5 e M λ s 2 − d R 2 . Its value at the origin is negative. This proves (72).
Normalization and class membership. All realizations meet the global smoothness and mean strong concavity requirements by Lemma 9, since μ 0 ≥ μ \mu_0\ge\mu μ 0 ≥ μ . By (27), the primal gap is at most ( 5 e 32 + 0.896 4096 ) Δ < Δ / 2 (\frac{5e}{32}+\frac{0.896}{4096})\Delta<\Delta/2 ( 32 5 e + 4096 0.896 ) Δ < Δ/2 . The initial gradient satisfies e λ 2 s 2 = 64 e ε 2 / c ⋆ 2 ≤ e λ Δ / 64 < 2 L Δ e\lambda^2s^2=64e\varepsilon^2/c_\star^2\le e\lambda\Delta/64<2L\Delta e λ 2 s 2 = 64 e ε 2 / c ⋆ 2 ≤ e λ Δ/64 < 2 L Δ . Only in the high branch, d < 4 λ d<4\lambda d < 4 λ gives the additional check G 0 ≤ ( 5 e 32 + 1024 ) Δ < 1025 Δ G_0\le(\frac{5e}{32}+1024)\Delta<1025\Delta G 0 ≤ ( 32 5 e + 1024 ) Δ < 1025Δ . Thus the hard family also has finite initial primal–dual gap.
Unrestricted Oracle Lower Bounds
A public convexification coordinate. For exact oracle access, augment the components already constructed by one common public primal coordinate:
f ~ i , η ( P , ξ p u b , y ) = f i , η ( P , y ) + B ( 1 − cos ξ p u b ) , B = L / 8. (74) \widetilde{f}_{i,\eta}(P,\xi_{\mathrm{pub}},y)=f_{i,\eta}(P,y)+B(1-\cos\xi_{\mathrm{pub}}),\qquad B=L/8. \tag*{(74)} f i , η ( P , ξ pub , y ) = f i , η ( P , y ) + B ( 1 − cos ξ pub ) , B = L /8. (74)
We keep V η ( P ) V_{\eta}(P) V η ( P ) for the value studied above; the augmented value and gradient satisfy
V ~ η ( P , ξ p u b ) = V η ( P ) + B ( 1 − cos ξ p u b ) , ∥ ∇ V ~ η ( P , ξ p u b ) ∥ 2 = ∥ ∇ V η ( P ) ∥ 2 + B 2 sin 2 ξ p u b . \widetilde{V}_{\eta}(P,\xi_{\mathrm{pub}})=V_{\eta}(P)+B(1-\cos\xi_{\mathrm{pub}}),\qquad\left\lVert\nabla\widetilde{V}_{\eta}(P,\xi_{\mathrm{pub}})\right\rVert^{2}=\left\lVert\nabla V_{\eta}(P)\right\rVert^{2}+B^{2}\sin^{2}\xi_{\mathrm{pub}}. V η ( P , ξ pub ) = V η ( P ) + B ( 1 − cos ξ pub ) , ∇ V η ( P , ξ pub ) 2 = ∥ ∇ V η ( P ) ∥ 2 + B 2 sin 2 ξ pub .
The added term is nonnegative and vanishes with its gradient at zero. Both initial gaps, the initial gradient and the dual maximizer are unchanged. Orthogonality gives joint RMS constant max { L r o b , B } < 4 L / 5 \max\{L_{\mathrm{rob}},B\}<4L/5 max { L rob , B } < 4 L /5 , and one extra primal dimension. All preceding widths, constants and value-transfer differences are unchanged. For every finite ( P , ξ p u b ) (P,\xi_{\mathrm{pub}}) ( P , ξ pub ) ,
∣ z M + 1 ( P ) ∣ ≤ s / 8 ⟹ ∥ ∇ V ~ η ( P , ξ p u b ) ∥ ≥ 6 ε . (75) \lvert z_{M+1}(P)\rvert\le s/8\quad\Longrightarrow\quad\left\lVert\nabla\widetilde{V}_{\eta}(P,\xi_{\mathrm{pub}})\right\rVert\ge6\varepsilon. \tag*{(75)} ∣ z M + 1 ( P )∣ ≤ s /8 ⟹ ∇ V η ( P , ξ pub ) ≥ 6 ε . (75)
Proximal notation. At a finite center w ˉ = ( P ˉ , ξ ˉ p u b , y ˉ ) \bar{w}=(\bar{P},\bar{\xi}_{\mathrm{pub}},\bar{y}) w ˉ = ( P ˉ , ξ ˉ pub , y ˉ ) and inverse step τ = 1 / γ > 0 \tau=1/\gamma>0 τ = 1/ γ > 0 , the regularized objective is
H ~ i = f ~ i , η + τ 2 ( ∥ P − P ˉ ∥ 2 + ∣ ξ p u b − ξ ˉ p u b ∣ 2 − ∥ y − y ˉ ∥ 2 ) . (76) \widetilde{H}_{i}=\widetilde{f}_{i,\eta}+\frac{\tau}{2}\left(\lVert P-\bar{P}\rVert^{2}+\lvert\xi_{\mathrm{pub}}-\bar{\xi}_{\mathrm{pub}}\rvert^{2}-\lVert y-\bar{y}\rVert^{2}\right). \tag*{(76)} H i = f i , η + 2 τ ( ∥ P − P ˉ ∥ 2 + ∣ ξ pub − ξ ˉ pub ∣ 2 − ∥ y − y ˉ ∥ 2 ) . (76)
Exact Proximal Localization and Oracle Complexity
Lemma 12 (Public legality and unique responses). The exact legal domain of (76) is τ ∈ [ B , ∞ ) \tau\in[B,\infty) τ ∈ [ B , ∞ ) , common to all components, hidden realizations and centers. Every legal response exists and is unique. The same conclusions hold for every consistent zero-axis formula that keeps all public terms; its response is measurable in its coefficients and center.
Proof. The scalar curvature is τ + B cos ξ p u b \tau+B\cos\xi_{\mathrm{pub}} τ + B cos ξ pub , negative at ξ p u b = π \xi_{\mathrm{pub}}=\pi ξ pub = π when τ < B \tau<B τ < B . For τ ≥ B \tau\ge B τ ≥ B , its derivative τ ( ξ p u b − ξ ˉ p u b ) + B sin ξ p u b \tau(\xi_{\mathrm{pub}}-\bar{\xi}_{\mathrm{pub}})+B\sin\xi_{\mathrm{pub}} τ ( ξ pub − ξ ˉ pub ) + B sin ξ pub tends to ± ∞ \pm\infty ± ∞ and is strictly increasing. At τ = B \tau=B τ = B , its increment on [ ξ 1 , ξ 2 ] [\xi_{1},\xi_{2}] [ ξ 1 , ξ 2 ] , ξ 1 < ξ 2 \xi_{1}<\xi_{2} ξ 1 < ξ 2 , is
B ∫ ξ 1 ξ 2 ( 1 + cos t ) d t > 0. B\int_{\xi_{1}}^{\xi_{2}}(1+\cos t)\,\mathrm{d}t>0. B ∫ ξ 1 ξ 2 ( 1 + cos t ) d t > 0.
Thus the scalar has a unique minimizer, measurably given by the inverse of ξ p u b + ( B / τ ) sin ξ p u b = ξ ˉ p u b \xi_{\mathrm{pub}}+(B/\tau)\sin\xi_{\mathrm{pub}}=\bar{\xi}_{\mathrm{pub}} ξ pub + ( B / τ ) sin ξ pub = ξ ˉ pub . It is not strongly convex at τ = B \tau=B τ = B .
For the remaining variables z = ( P , y ) z=(P,y) z = ( P , y ) , let H H H be the corresponding regularized objective and put T = ( ∇ P H , − ∇ y H ) T=(\nabla_{P}H,-\nabla_{y}H) T = ( ∇ P H , − ∇ y H ) . Lemma 9 gives strong monotonicity with m 0 = min { τ − K P , μ 0 + τ } > 0 m_{0}=\min\{\tau-K_{P},\mu_{0}+\tau\}>0 m 0 = min { τ − K P , μ 0 + τ } > 0 and a Lipschitz bound M 0 = n L + τ M_{0}=\sqrt{n}L+\tau M 0 = n L + τ . Mixed Hessian terms cancel in the monotonicity quadratic form. For a 0 = m 0 / M 0 2 a_{0}=m_{0}/M_{0}^{2} a 0 = m 0 / M 0 2 ,
∥ ( I − a 0 T ) z − ( I − a 0 T ) z ′ ∥ 2 ≤ ( 1 − m 0 2 M 0 2 ) ∥ z − z ′ ∥ 2 . \left\lVert(I-a_{0}T)z-(I-a_{0}T)z'\right\rVert^{2}\le\left(1-\frac{m_{0}^{2}}{M_{0}^{2}}\right)\lVert z-z'\rVert^{2}. ∥ ( I − a 0 T ) z − ( I − a 0 T ) z ′ ∥ 2 ≤ ( 1 − M 0 2 m 0 2 ) ∥ z − z ′ ∥ 2 .
Contraction gives a unique zero, hence a unique global saddle by convexity–concavity. Iteration from zero also makes this root measurable. The same bounds hold for zeroed axes by Lemma 9. Combining this with the scalar minimizer proves all claims, including those at the legal boundary.
Full-Space Localization by Reflection
We grant the first source axis and maintain a serial frontier through E = M ( m − 1 ) E=M(m-1) E = M ( m − 1 ) fixed owned edges. A component-i i i call tests the frontier: a hit reveals its successor axis and continues with i i i ; completing a path also grants the next source axis. The first mismatch reveals only that the owner is not i i i and ends the run. Granting this entire run can only help the algorithm.
After exposure we retain full ambient blocks for revealed axes. For an unrevealed interior dual block retain its current-center span; for an unrevealed endpoint retain the span of its current dual and paired primal centers on both sides of the public identity. Their product is S = S P × R ξ p u b × S y S = S_P \times\mathbb{R}\xi_{\mathrm{pub}} \times S_y S = S P × R ξ pub × S y , with each unrevealed span S b S_b S b of dimension at most two. We rebuild it at each call and replace each unrevealed axis by zero in every primal and dual occurrence of the full-space component, retaining all public norms, corrections and identity maps. Let j j j be the stage whose path contains the frontier. Its outgoing axis is unrevealed, so its gate factor becomes b + ( 0 ) = 1 b_+(0) = 1 b + ( 0 ) = 1 , and we retain it. We call the resulting function f i 0 f_i^0 f i 0 . The first missing edge is absent because its owner is known to differ from i i i ; every later edge has two zero axes and is identically zero, regardless of its owner, and unrevealed node terms vanish. Later sources have Ψ ( ± 0 ) = 0 \Psi(\pm0) = 0 Ψ ( ± 0 ) = 0 , and the forcing of every stage after j j j has an incoming factor a + ( 0 ) = 0 a_+(0) = 0 a + ( 0 ) = 0 . The ambient forcing of stage j j j and its square are retained. Thus the censored full-space formula reads only the exposed prefix.
Lemma 13 (Exact localization of the unique proximal response). For every legal call, the unique full-space censored saddle lies in S S S . If the actual axes obey
∥ Proj S b U b ∥ ≤ δ , δ = min { s 8 R , η 8 R ∗ } , (77) \left\lVert\operatorname{Proj}_{S_b} U_b \right\rVert\le\delta,\qquad\delta= \min\left\{\frac{s}{8R},\frac{\eta}{8R_*}\right\}, \tag*{(77)} Proj S b U b ≤ δ , δ = min { 8 R s , 8 R ∗ η } , (77) then it is also the unique actual proximal response. Actual and censored center values and full joint gradients agree as well.
Proof. The full-space censored saddle exists uniquely by Lemma 12. Reflect each unrevealed block through S b S_b S b , using the same reflection on paired primal and dual endpoints; fix all revealed blocks and ξ p u b \xi_{\mathrm{pub}} ξ pub . Norms and radial terms are invariant, caps and the full ambient forcing are equivariant, the paired endpoint inner product is invariant, and zeroed projections stay zero. The center is fixed, so this reflection maps the censored saddle to another censored saddle. Uniqueness therefore places it in S S S .
Under (77), capped primal projections on the entire S S S have magnitude at most R δ ≤ s / 8 R\delta\le s/8 R δ ≤ s /8 . The profile satisfies ρ ( u ) ≥ u / 2 \rho(u) \ge u/2 ρ ( u ) ≥ u /2 for all u ≥ 0 u \ge0 u ≥ 0 , since on [ 1 , 2 ] [1,2] [ 1 , 2 ] this is equivalent to u 2 − 3 u + 3 ≥ 0 u^2 - 3u + 3 \ge0 u 2 − 3 u + 3 ≥ 0 and the other intervals are immediate. For a hidden–hidden edge with r = ∥ Y e ∥ > 0 r = \lVert Y_e\rVert> 0 r = ∥ Y e ∥ > 0 , every point of the retained subspace therefore satisfies
∣ t e ∣ ≤ δ ( ∥ y v ∥ + ∥ y w ∥ ) ≤ 2 δ r ≤ 2 η r 8 R ∗ < η r 2 R ∗ ≤ h η ( r ) . \lvert t_e\rvert\le\delta(\lVert y_v\rVert+ \lVert y_w\rVert) \le\sqrt{2}\delta r \le\frac{\sqrt{2}\eta r}{8R_*} < \frac{\eta r}{2R_*} \le h_\eta(r). ∣ t e ∣ ≤ δ (∥ y v ∥ + ∥ y w ∥) ≤ 2 δ r ≤ 8 R ∗ 2 η r < 2 R ∗ η r ≤ h η ( r ) .
At r = 0 r = 0 r = 0 the margin is η > 0 \eta> 0 η > 0 . Thus the edge and its full gradient vanish on an open neighborhood of every such point, at every radius. For an unrevealed node with ∥ y v ∥ ≤ 2 R ∗ \lVert y_v\rVert\le2R_* ∥ y v ∥ ≤ 2 R ∗ , its hidden projection is at most 2 R ∗ δ ≤ η / 4 2R_*\delta\le\eta/4 2 R ∗ δ ≤ η /4 ; outside that radius its cutoff vanishes. The strict margin of the flat region also covers the cutoff boundary. Hence all discarded terms are locally zero in the full space. The actual and censored functions have the same values and full gradients on S S S . The censored full-space saddle is therefore stationary for the actual convex–concave objective, hence is its unique saddle. The center itself belongs to S S S , proving exact scalar and joint-gradient equality there as well. ■
A Total Ideal Transcript and Uniform Subspace Hiding
Fix the finite comparison horizon
E = M ( m − 1 ) , H = E ( n − 1 ) , q P = ⌊ H / 16 ⌋ , p = 1 / 64. (78) E = M(m-1), \qquad H = E(n-1), \qquad q_P = \lfloor H/16 \rfloor, \qquad p = 1/64. \tag*{(78)} E = M ( m − 1 ) , H = E ( n − 1 ) , q P = ⌊ H /16 ⌋ , p = 1/64. (78)
We compare the algorithm with an ideal process that sees only what the owner runs have exposed. The instance and the private seed are sampled before execution. The algorithm’s query, stopping and output rules are defined on the histories the real oracle can produce. We fix a public extension of them, measurable and using the same seed, to every history of length at most q P q_P q P the ideal process can produce, with valid indices, finite centers and outputs, and finite positive inverse steps; the extension need not terminate. At each call the ideal process chooses its query from the past, exposes the owner run, and returns the full-space censored tuple; if τ < B \tau< B τ < B , it returns a fixed dummy instead. This public fallback reads no hidden data and reveals nothing about the real oracle. All other responses are unique and measurable by Lemma 12. The algorithm is legal almost surely on each fixed instance; averaging over the prior and taking a countable union over calls gives one null event over instances and seeds, so the owner and Haar laws are never conditioned on legality or on projection success.
Conditional on the ideal history and current run exposure, every unexposed axis is independent Haar. Its current-center span is measurable from the prefix and private seed and has dimension at most two; earlier projection successes are not conditioned upon [cf. 2 Appendix B.1.2] . For Haar U ∈ S D − 1 U \in S^{D-1} U ∈ S D − 1 , rotational symmetry and the beta integral give E ⟨ U , v ⟩ 2 r = ( 2 r − 1 ) ! ! ∥ v ∥ 2 r / [ D ( D + 2 ) ⋯ ( D + 2 r − 2 ) ] \mathbb{E}\langle U,v\rangle^{2r} = (2r-1)!!\|v\|^{2r}/[D(D+2)\cdots(D+2r-2)] E ⟨ U , v ⟩ 2 r = ( 2 r − 1 )!! ∥ v ∥ 2 r / [ D ( D + 2 ) ⋯ ( D + 2 r − 2 )] and zero odd moments. Bounding denominators by D r D^r D r in the exponential series yields E e θ ⟨ U , v ⟩ ≤ e θ 2 ∥ v ∥ 2 / ( 2 D ) \mathbb{E}e^{\theta\langle U,v\rangle} \le e^{\theta^2\|v\|^2/(2D)} E e θ ⟨ U , v ⟩ ≤ e θ 2 ∥ v ∥ 2 / ( 2 D ) , so Chernoff gives P { ∣ ⟨ U , v ⟩ ∣ > t } ≤ 2 e − D t 2 / ( 2 ∥ v ∥ 2 ) \mathbb{P}\{|\langle U,v\rangle| > t\} \le2e^{-Dt^2/(2\|v\|^2)} P { ∣ ⟨ U , v ⟩ ∣ > t } ≤ 2 e − D t 2 / ( 2∥ v ∥ 2 ) . A 1 / 2 1/2 1/2 -net of a d d d -dimensional unit sphere has at most 5 d 5^d 5 d points [19 Corollary 4.2.13] ; net comparison [19 Lemma 4.4.1] therefore gives, for any fixed subspace S S S ,
P { ∥ Proj S U ∥ > δ } ≤ 2 5 d exp ( − D δ 2 / 8 ) . (79) \mathbb{P}\{\|\operatorname{Proj}_S U\| > \delta\} \le2\,5^d\exp(-D\delta^2/8). \tag*{(79)} P { ∥ Proj S U ∥ > δ } ≤ 2 5 d exp ( − D δ 2 /8 ) . (79)
Conditioning on the ideal history and averaging extends this to the random current span. We include the actual proposed output when stopping by q P q_P q P , and a fixed virtual output otherwise. In each unrevealed endpoint coordinate block, its capped primal output subblock spans at most one dimension and is determined by the ideal history, and we test that span separately. There are at most K ( q P + 1 ) K(q_P+1) K ( q P + 1 ) coordinate block tests in total, with d ≤ 2 d \le2 d ≤ 2 , including the output when q P = 0 q_P=0 q P = 0 . With
H n = max { 1 , R 2 / s 2 , R ∗ 2 / η 2 } , δ − 2 ≤ 64 H n , H_n = \max\{1, R^2/s^2, R_*^2/\eta^2\}, \qquad\delta^{-2} \le64H_n, H n = max { 1 , R 2 / s 2 , R ∗ 2 / η 2 } , δ − 2 ≤ 64 H n ,
a union bound yields a common event G q P G_{q_P} G q P of probability at least 1 − p 1-p 1 − p whenever
D ≥ max { 3 , ⌈ 512 H n [ 2 log 5 + log 2 K ( q P + 1 ) p ] ⌉ } . (80) D \ge\max\left\{3,\ \left\lceil512H_n\left[2\log5+\log\frac{2K(q_P+1)}{p}\right]\right\rceil\right\}. \tag*{(80)} D ≥ max { 3 , ⌈ 512 H n [ 2 log 5 + log p 2 K ( q P + 1 ) ] ⌉ } . (80)
Localization uses only the current-center subspace; the transcript keeps all earlier responses. We choose D = ⌈ 1024 H n log ( 1024 K ( q P + 1 ) ) ⌉ D=\lceil1024H_n\log(1024K(q_P+1))\rceil D = ⌈ 1024 H n log ( 1024 K ( q P + 1 ))⌉ . For p = 1 / 64 p=1/64 p = 1/64 the logarithmic bracket in (80) is log ( 3200 K ( q P + 1 ) ) \log(3200K(q_P+1)) log ( 3200 K ( q P + 1 )) . Since K ≥ 4 K\ge4 K ≥ 4 ,
( 1024 K ( q P + 1 ) ) 2 ≥ 3200 K ( q P + 1 ) . (81) (1024K(q_P+1))^2 \ge3200K(q_P+1). \tag*{(81)} ( 1024 K ( q P + 1 ) ) 2 ≥ 3200 K ( q P + 1 ) . (81)
Taking logarithms proves (80), including D ≥ 3 D \ge3 D ≥ 3 .
On G q P G_{q_P} G q P , induction using Lemma 13 couples all exact real and ideal responses through q P q_P q P , since equal pasts give the same legal query, localization gives its same unique tuple, and fallback is unused. The output test gives (75) when the terminal axis is hidden. The proof covers all raw radii because primal vectors are capped, node cutoffs vanish outside radius 2 R ∗ 2R_\ast 2 R ∗ , and edge flatness holds on the entire retained cone.
Posterior Owner Cost and Integer Stopping
Keep the owners fixed before execution, independently of the axes and private seed. The augmented ideal history contains the seed, exposed axes, internal hits and misses, full censored responses, and stop/output decisions; it is not conditioned on geometric success or legality of the real oracle. Induction over internal tests leaves the frontier owner uniform among its unexcluded labels and all successors independent and uniform, since a miss removes only its label, a hit fixes the current owner, and a new axis is independent. Full values, gradients and proximal tuples read only the exposed prefix; the fallback reads none.
Lemma 14 (Posterior cost of serial owner exposure). For E ≥ 1 E \ge1 E ≥ 1 independent uniform owners on [ n ] [n] [ n ] , n ≥ 2 n \ge2 n ≥ 2 , assume the posterior interface above. A call traverses matching owners until its first mismatch. Freeze at termination and let C q C_q C q be complete chain exposure within q q q calls, for integer q ≥ 0 q \ge0 q ≥ 0 . Then
P ( C q ) ≤ 2 q H , H = E ( n − 1 ) , q ≥ 0. (82) \mathbb{P}(C_q) \le\frac{2q}{H}, \qquad H = E(n-1), \qquad q \ge0. \tag*{(82)} P ( C q ) ≤ H 2 q , H = E ( n − 1 ) , q ≥ 0. (82)
Proof. With r r r frontier candidates and s s s unreached successors, set W = ( r − 1 ) / 2 + s ( n − 1 ) / 2 W=(r-1)/2+s(n-1)/2 W = ( r − 1 ) /2 + s ( n − 1 ) /2 , and W = 0 W=0 W = 0 after completion. Repeated failed labels leave W W W unchanged. A new label misses with probability ( r − 1 ) / r (r-1)/r ( r − 1 ) / r , decreasing W W W by 1 / 2 1/2 1/2 , or hits with probability 1 / r 1/r 1/ r , removing ( r − 1 ) / 2 (r-1)/2 ( r − 1 ) /2 and continuing at an independent successor. The conditional expected decrease during the entire call therefore obeys
d ( r , 0 ) = r − 1 r , d ( r , s ) = r − 1 r + d ( n , s − 1 ) r ≤ 1 ( s ≥ 1 ) , d(r,0)=\frac{r-1}{r}, \qquad d(r,s)=\frac{r-1}{r}+\frac{d(n,s-1)}{r}\le1 \qquad(s\ge1), d ( r , 0 ) = r r − 1 , d ( r , s ) = r r − 1 + r d ( n , s − 1 ) ≤ 1 ( s ≥ 1 ) ,
by induction, including all consecutive hits in one call. The potential is nonnegative and nonincreasing, starts at W 0 = H / 2 W_0=H/2 W 0 = H /2 , and freezes at termination. Summing conditional expectations gives E [ W 0 − W q ] ≤ q \mathbb{E}[W_0-W_q]\le q E [ W 0 − W q ] ≤ q . Completion implies W q = 0 W_q=0 W q = 0 . Thus P ( C q ) W 0 ≤ E [ W 0 − W q ] ≤ q \mathbb{P}(C_q)W_0\le\mathbb{E}[W_0-W_q]\le q P ( C q ) W 0 ≤ E [ W 0 − W q ] ≤ q .
For q P = ⌊ H / 16 ⌋ q_P=\lfloor H/16\rfloor q P = ⌊ H /16 ⌋ , the lemma gives
P ( C q P c ) ≥ 7 8 . (83) \mathbb{P}(C_{q_P}^{\mathsf c})\ge\frac{7}{8}. \tag*{(83)} P ( C q P c ) ≥ 8 7 . (83)
On C q P c ∩ { N ≤ q P } ∩ G q P C_{q_P}^{\mathsf c}\cap\{N\le q_P\}\cap G_{q_P} C q P c ∩ { N ≤ q P } ∩ G q P the terminal axis remains unexposed, the actual and ideal outputs coincide, and the output barrier is at least 6 ε 6\varepsilon 6 ε . Uniform expected accuracy on every fixed instance, averaged over the hard prior, implies by Markov’s inequality that this event has probability at most 1 / 6 1/6 1/6 . Thus
P { N > q P } ≥ 7 8 − 1 64 − 1 6 = 133 192 . (84) \mathbb{P}\{N>q_P\}\ge\frac{7}{8}-\frac{1}{64}-\frac{1}{6}=\frac{133}{192}. \tag*{(84)} P { N > q P } ≥ 8 7 − 64 1 − 6 1 = 192 133 . (84)
This argument also covers q P = 0 q_P=0 q P = 0 , where the only required geometric test is the proposed zero-call output, already included above. Since the query count is integer,
E N ≥ ( q P + 1 ) P { N > q P } ≥ 133 H 3072 = 133 M ( m − 1 ) ( n − 1 ) 3072 ≥ 133 n M m 12288 . (85) \mathbb{E}N \ge(q_P+1)\mathbb{P}\{N>q_P\} \ge\frac{133H}{3072}
= \frac{133M(m-1)(n-1)}{3072} \ge\frac{133nMm}{12288}. \tag*{(85)} E N ≥ ( q P + 1 ) P { N > q P } ≥ 3072 133 H = 3072 133 M ( m − 1 ) ( n − 1 ) ≥ 12288 133 n M m . (85)
The last inequality uses m , n ≥ 2 m,n\ge2 m , n ≥ 2 ; infinite expected cost is trivial. Since the instance prior is seed-independent, averaging the real cost selects one fixed realization with this expected cost, while the assumed accuracy still holds there.
In the high branch, using (31) in (85) , with 12288 = 3 ⋅ 2 12 12288=3\cdot2^{12} 12288 = 3 ⋅ 2 12 and 512 = 2 9 512=2^9 512 = 2 9 , gives
E N ≥ 133 c x 2 3 ⋅ 2 73 n 3 / 4 κ L Δ ε 2 , \mathbb{E}N \ge\frac{133c_x^2}{3\cdot2^{73}}n^{3/4}\sqrt{\kappa}\frac{L\Delta}{\varepsilon^2}, E N ≥ 3 ⋅ 2 73 133 c x 2 n 3/4 κ ε 2 L Δ ,
proving Theorem 3’s lower bound.
Fixed confidence. For a deterministic cap q P q_P q P , the failure probability is at least 7 / 8 − 1 / 64 = 55 / 64 > 1 / 3 7/8-1/64=55/64>1/3 7/8 − 1/64 = 55/64 > 1/3 , proving the fixed-confidence lower bound.
Han’s Protocol and All Early Horizons
Corollary 15 (Han’s protocol and one fixed instance for all early horizons). In the window (12) , the same family gives the lower bound under Han et al. [7 ] , whose full-gradient refreshes occur with probability at most c 0 / n c_0/n c 0 / n . More precisely, put
E = M ( m − 1 ) , H = E ( n − 1 ) , C = 2 + c 0 , T 0 = ⌊ H 512 C ⌋ . E=M(m-1),\qquad H=E(n-1),\qquad C=2+c_0,\qquad T_0=\left\lfloor\frac{H}{512C}\right\rfloor. E = M ( m − 1 ) , H = E ( n − 1 ) , C = 2 + c 0 , T 0 = ⌊ 512 C H ⌋ . For every algorithm in that protocol, there is one fixed hard realization for which, simultaneously at every step 0 ≤ t ≤ T 0 0\le t\le T_0 0 ≤ t ≤ T 0 ,
E s e e d ∥ ∇ V ~ η ( P t , ξ p u b , t ) ∥ ≥ 159 32 ε > ε . (86) \mathbb{E}_{\mathrm{seed}}\left\|\nabla\widetilde{V}_{\eta}(P_t,\xi_{\mathrm{pub},t})\right\|\ge\frac{159}{32}\varepsilon>\varepsilon. \tag*{(86)} E seed ∇ V η ( P t , ξ pub , t ) ≥ 32 159 ε > ε . (86) Consequently its first accurate horizon, and hence the inf A sup f inf t \inf_A\sup_f\inf_t inf A sup f inf t complexity of Definition 13, is at least
H 512 ( 2 + c 0 ) ≥ n M m 2048 ( 2 + c 0 ) ≥ c x 2 2 72 ( 2 + c 0 ) n 3 / 4 κ L Δ ε 2 . (87) \frac{H}{512(2+c_0)}\ge\frac{nMm}{2048(2+c_0)}\ge\frac{c_x^2}{2^{72}(2+c_0)}n^{3/4}\sqrt{\kappa}\frac{L\Delta}{\varepsilon^2}. \tag*{(87)} 512 ( 2 + c 0 ) H ≥ 2048 ( 2 + c 0 ) n M m ≥ 2 72 ( 2 + c 0 ) c x 2 n 3/4 κ ε 2 L Δ . (87) The dimension D D D chosen in Appendix C.§ 1 suffices.
Proof Outline. We embed Han’s protocol in our oracle model, add a subspace test for every early iterate to the ideal simulator, and find one event of probability at least 53 / 64 53/64 53/64 on which the terminal axis stays unexposed at every early step; the terminal barrier then bounds every early gradient.
Embedding the protocol. Whole-space projections are identities, so Han’s fixed sampling and linear-span operations restrict our model. Supply the common component smoothness bound and grant the owner-independent initial full gradient, whose only nonzero block is − ϵ λ s U 1 , 1 -\sqrt{\epsilon}\lambda sU_{1,1} − ϵ λ s U 1 , 1 in the first primal subblock; the public coordinate has zero derivative. It therefore reveals only the already free source axis. Simulate a full gradient by n n n component calls, ignoring their proximal responses. A Han step has at most two component PIFO calls and an independent refresh coin A j A_j A j with mean π ≤ c 0 / n \pi\le c_0/n π ≤ c 0 / n . The predrawn budget B T = 2 T + n ∑ j ≤ T A j B_T = 2T + n\sum_{j\le T} A_j B T = 2 T + n ∑ j ≤ T A j dominates its cost, even under mid-step truncation, and satisfies E B T ≤ C T \mathbb{E}B_T \le CT E B T ≤ C T .
Tested early outputs. Write d = H / 16 d = H/16 d = H /16 and q P = ⌊ d ⌋ q_P = \lfloor d\rfloor q P = ⌊ d ⌋ . Use the exact ideal component simulator for its first q P q_P q P calls. In addition to its rank-at-most-two current-center tests, test the rank-one capped output span for each of the iterates P 0 , … , P T 0 P_0,\ldots,P_{T_0} P 0 , … , P T 0 produced within that call budget; assign a fixed virtual output to any step not completed within the budget. This keeps every test measurable from the ideal history. On B T 0 ≤ d B_{T_0} \le d B T 0 ≤ d , all T 0 T_0 T 0 steps have completed within q P q_P q P calls, so none of their outputs is virtual.
Subspace tests. Use the same δ \delta δ , H n H_n H n as in (80) . Each rank-at-most-two test fails with probability at most 50 e − D / ( 512 H n ) 50e^{-D/(512H_n)} 50 e − D / ( 512 H n ) . There are at most K ( q P + T 0 + 1 ) ≤ 2 K ( q P + 1 ) K(q_P+T_0+1) \le2K(q_P+1) K ( q P + T 0 + 1 ) ≤ 2 K ( q P + 1 ) tests since T 0 ≤ q P T_0 \le q_P T 0 ≤ q P . The same dimension therefore gives an event G G G with
P ( G c ) ≤ 100 K ( q P + 1 ) ( 1024 K ( q P + 1 ) ) 2 = 100 1024 2 K ( q P + 1 ) < 1 64 . \mathbb{P}(G^c) \le\frac{100K(q_P+1)}{(1024K(q_P+1))^2} = \frac{100}{1024^2K(q_P+1)} < \frac{1}{64}. P ( G c ) ≤ ( 1024 K ( q P + 1 ) ) 2 100 K ( q P + 1 ) = 102 4 2 K ( q P + 1 ) 100 < 64 1 .
This includes T 0 = q P = 0 T_0=q_P=0 T 0 = q P = 0 , since K ≥ 4 K\ge4 K ≥ 4 .
One common event. Lemma 14 gives P ( C q P c ) ≥ 7 / 8 \mathbb{P}(C_{q_P}^c) \ge7/8 P ( C q P c ) ≥ 7/8 without conditioning on G G G . Markov’s inequality and the definition of T 0 T_0 T 0 give P ( B T 0 > d ) ≤ C T 0 / d ≤ 1 / 32 \mathbb{P}(B_{T_0}>d) \le CT_0/d \le1/32 P ( B T 0 > d ) ≤ C T 0 / d ≤ 1/32 . Thus the single common event
E = C q P c ∩ { B T 0 ≤ d } ∩ G \mathcal{E} = C_{q_P}^c \cap\{B_{T_0}\le d\} \cap G E = C q P c ∩ { B T 0 ≤ d } ∩ G
has probability at least 7 / 8 − 1 / 32 − 1 / 64 = 53 / 64 7/8-1/32-1/64=53/64 7/8 − 1/32 − 1/64 = 53/64 . On this one event the terminal axis is still unexposed after all T 0 T_0 T 0 steps, and therefore also at every earlier step. Exact transcript coupling and the separately tested early outputs give ∥ ∇ V ~ η ( P t , ξ p u b , t ) ∥ ≥ 6 ϵ \lVert\nabla\widetilde{V}_{\eta}(P_t,\xi_{\mathrm{pub},t})\rVert\ge6\epsilon ∥ ∇ V η ( P t , ξ pub , t )∥ ≥ 6 ϵ simultaneously for all 0 ≤ t ≤ T 0 0\le t\le T_0 0 ≤ t ≤ T 0 .
A fixed realization. Let B \mathcal{B} B be the event, over real instances and seeds, that ∥ ∇ V ~ η ( P t , ξ p u b , t ) ∥ ≥ 6 ϵ \lVert\nabla\widetilde{V}_{\eta}(P_t,\xi_{\mathrm{pub},t})\rVert\ge6\epsilon ∥ ∇ V η ( P t , ξ pub , t )∥ ≥ 6 ϵ for all 0 ≤ t ≤ T 0 0\le t\le T_0 0 ≤ t ≤ T 0 . The coupling gives E ⊆ B \mathcal{E}\subseteq\mathcal{B} E ⊆ B , so P ( B ) ≥ 53 / 64 \mathbb{P}(\mathcal{B})\ge53/64 P ( B ) ≥ 53/64 . Averaging B \mathcal{B} B selects one fixed realization with this seed probability. Every early expected gradient there is therefore at least 6 ϵ ( 53 / 64 ) = 159 ϵ / 32 6\epsilon(53/64)=159\epsilon/32 6 ϵ ( 53/64 ) = 159 ϵ /32 . Definition 13 indexes the step-( T − 1 ) (T-1) ( T − 1 ) point by T T T ; thus the first successful horizon is at least T 0 + 1 ≥ H / ( 512 C ) T_0+1\ge H/(512C) T 0 + 1 ≥ H / ( 512 C ) . The floor bounds on m , M m,M m , M give (87) .
Membership in Han’s class. The mean is L L L -smooth by the RMS assumption, so its primal weak-convexity parameter is at most L L L . Together with the already proved gap and strong concavity, this places the witness in Han’s nonconvex–strongly-concave class with μ x = L \mu_x=L μ x = L . Because every legal proximal subproblem has a unique saddle, the response does not depend on a selection rule.
The Strongly Concave Lower Bound for Every Condition Number
To complete the proof of Theorem 1 , we specialize the common path family to 64 ≤ κ ≤ 2 18 n 64\le\kappa\le2^{18}\sqrt{n} 64 ≤ κ ≤ 2 18 n , and then add a primal family for 1 ≤ κ < 64 1\le\kappa<64 1 ≤ κ < 64 and an indexed-search baseline.
The Two-Node Specialization
The common parameters (23) –(24) give m = 2 m=2 m = 2 , r = μ / a ≥ 1 / 4 r=\mu/a\ge1/4 r = μ / a ≥ 1/4 , g = 1 + r g=1+r g = 1 + r , and
λ = 2 − 39 L κ κ + 2 16 n . (88) \lambda=\frac{2^{-39}L\kappa}{\kappa+2^{16}\sqrt{n}}. \tag*{(88)} λ = κ + 2 16 n 2 − 39 L κ . (88)
As before, use this family only in the path window (25) , with the components, widths and oracle simulator of Appendices § 7 –C.
Corollary 16 (Intermediate admissibility and oracle cost). The augmented two-node family has joint RMS smoothness below 4 L / 5 4L/5 4 L /5 , componentwise μ \mu μ -strong dual concavity, the 6 ε 6\varepsilon 6 ε terminal barrier, y ∗ ( 0 ) = 0 y^*(0)=0 y ∗ ( 0 ) = 0 , initial squared gradient norm at most 2 L Δ 2L\Delta 2 L Δ and primal gap less than Δ / 2 \Delta/2 Δ/2 . Its common exact legal domain is τ ≥ L / 8 \tau\ge L/8 τ ≥ L /8 , including a unique response at the boundary. A fixed polynomial dimension yields a fixed hard realization with
E N ≥ 133 M ( n − 1 ) 3072 ≥ 133 n M 6144 . (89) \mathbb{E}N\ge\frac{133M(n-1)}{3072}\ge\frac{133nM}{6144}. \tag*{(89)} E N ≥ 3072 133 M ( n − 1 ) ≥ 6144 133 n M . (89) and the same fixed-confidence lower bound. By (72) and d = 6 λ r d=6\lambda r d = 6 λ r , the initial primal–dual gap satisfies G 0 ≤ ( 5 e / 32 + 1536 μ / a ) Δ G_0\le(5e/32+1536\mu/a)\Delta G 0 ≤ ( 5 e /32 + 1536 μ / a ) Δ .
Proof Lemmas 9, 11 and 12 apply directly to this branch. The exact localization, total ideal transcript, posterior owner cost of Lemma 14, and arbitrary-output test in Appendix C.§ 1 then apply without changes. Here E = M E=M E = M , so (85) gives the sharper first inequality in (89) ; the capped failure probability is still 55 / 64 > 1 / 3 55/64>1/3 55/64 > 1/3 . Finally g / r = ( 1 + r ) / r ≤ 5 g/r=(1+r)/r\le5 g / r = ( 1 + r ) / r ≤ 5 in (28) , hence H n = O ( n + M ) H_n=O(n+M) H n = O ( n + M ) and D = O ( ( n + M ) log ( n M ) ) D=O((n+M)\log(nM)) D = O (( n + M ) log ( n M )) with D D D chosen as in Appendix C.§ 1 . Total dimension is 4 M D + 1 4MD+1 4 M D + 1 , fixed before execution.
A Primal Family with a Public Exact Convexification Threshold
The next family works for every κ ≥ 1 \kappa\ge1 κ ≥ 1 and will be used when κ < 64 \kappa<64 κ < 64 . It has no hidden dual path. Retain the primal blocks P j = ( X j , Z j + 1 ) P_j=(X_j,Z_{j+1}) P j = ( X j , Z j + 1 ) , independent Haar axes U j , X , U j , Z U_{j,X},U_{j,Z} U j , X , U j , Z , and the same w j , x j , z j + 1 , J M , h j , A j = h j Γ w j w_j,x_j,z_{j+1},J_M,h_j,A_j=h_j\Gamma w_j w j , x j , z j + 1 , J M , h j , A j = h j Γ w j . Define hidden forcing coordinates
t j , 1 = ⟨ U j , X , A j , X ⟩ = h j x j , t j , 2 = ⟨ U j , Z , A j , Z ⟩ = − h j z j + 1 / 2 , t_{j,1}=\langle U_{j,X},A_{j,X}\rangle=h_jx_j,\qquad t_{j,2}=\langle U_{j,Z},A_{j,Z}\rangle=-h_jz_{j+1}/2, t j , 1 = ⟨ U j , X , A j , X ⟩ = h j x j , t j , 2 = ⟨ U j , Z , A j , Z ⟩ = − h j z j + 1 /2 ,
and the flat cross term
C η ( t 1 , t 2 ) = ϕ η ( t 1 + t 2 ) − ϕ η ( t 1 ) − ϕ η ( t 2 ) , ϕ η ( t ) = 1 2 ( ∣ t ∣ − η ) + 2 . (90) C_\eta(t_1,t_2)=\phi_\eta(t_1+t_2)-\phi_\eta(t_1)-\phi_\eta(t_2),\qquad\phi_\eta(t)=\frac{1}{2}(|t|-\eta)_+^2. \tag*{(90)} C η ( t 1 , t 2 ) = ϕ η ( t 1 + t 2 ) − ϕ η ( t 1 ) − ϕ η ( t 2 ) , ϕ η ( t ) = 2 1 ( ∣ t ∣ − η ) + 2 . (90)
Choose independent fixed owners Σ j ∼ Unif [ n ] \Sigma_j\sim\operatorname{Unif}[n] Σ j ∼ Unif [ n ] . With public auxiliary primal coordinates ξ ∈ R n \xi\in\mathbb{R}^n ξ ∈ R n , set
g i ( P , ξ ) = λ s 2 J M ( P ) + ∑ j p ( P j ) + 6 λ ∥ A ( P ) ∥ 2 g_i(P,\xi)=\lambda s^2J_M(P)+\sum_j p(P_j)+6\lambda\|A(P)\|^2 g i ( P , ξ ) = λ s 2 J M ( P ) + j ∑ p ( P j ) + 6 λ ∥ A ( P ) ∥ 2
+ 12 n λ ∑ j : Σ j = i C η ( t j , 1 , t j , 2 ) + B ( 1 − cos ξ i ) , (91) \qquad{}+12n\lambda\sum_{j:\Sigma_j=i}C_\eta(t_{j,1},t_{j,2})+B(1-\cos\xi_i), \tag*{(91)} + 12 nλ j : Σ j = i ∑ C η ( t j , 1 , t j , 2 ) + B ( 1 − cos ξ i ) , (91)
F i ( P , ξ , y ) = g i ( P , ξ ) − μ y 2 / 2 , y ∈ R . (92) F_i(P,\xi,y)=g_i(P,\xi)-\mu y^2/2,\qquad y\in\mathbb{R}. \tag*{(92)} F i ( P , ξ , y ) = g i ( P , ξ ) − μ y 2 /2 , y ∈ R . (92)
The calibration is
λ = L 2 34 n , B = 2 32 n λ , s = 8 ε c ⋆ λ , R = 128 s , (93) \lambda= \frac{L}{2^{34}\sqrt{n}}, \qquad B = 2^{32}n\lambda, \qquad s = \frac{8\varepsilon}{c_{\star}\lambda}, \qquad R = 128s, \tag*{(93)} λ = 2 34 n L , B = 2 32 nλ , s = c ⋆ λ 8 ε , R = 128 s , (93)
M = ⌈ c ⋆ 2 Δ 2048 ε 2 ⌉ , η = c ⋆ s 96 2 M A 1 , A 1 = 2433 , A 2 = 231000. (94) M = \left\lceil\frac{c_{\star}^{2}\Delta}{2048\varepsilon^{2}} \right\rceil, \qquad\eta= \frac{c_{\star}s}{96\sqrt{2M}A_{1}}, \qquad A_{1}=2433, \qquad A_{2}=231000. \tag*{(94)} M = ⌈ 2048 ε 2 c ⋆ 2 Δ ⌉ , η = 96 2 M A 1 c ⋆ s , A 1 = 2433 , A 2 = 231000. (94)
Again assume (25), with this new λ \lambda λ , so M ≥ 2 M \ge2 M ≥ 2 .
Lemma 17 (Global smoothness and a common exact legal domain). The components (92) satisfy joint RMS smoothness L L L and componentwise μ \mu μ -strong dual concavity. Their exact legal domain of inverse step sizes is τ ∈ [ B , ∞ ) \tau\in[B,\infty) τ ∈ [ B , ∞ ) , independent of all hidden axes, owners, component indices and centers, and every legal response is unique. These statements also hold for censored formulas obtained by setting any unexposed scalar projections to zero.
Proof. Almost everywhere, ∇ 2 C η = a 0 ( 1 1 1 1 ) − diag ( b 0 , c 0 ) \nabla^{2}C_{\eta}=a_{0}\begin{pmatrix}1 & 1 \\ 1 & 1\end{pmatrix}-\operatorname{diag}(b_{0},c_{0}) ∇ 2 C η = a 0 ( 1 1 1 1 ) − diag ( b 0 , c 0 ) for some a 0 , b 0 , c 0 ∈ { 0 , 1 } a_{0},b_{0},c_{0}\in\{0,1\} a 0 , b 0 , c 0 ∈ { 0 , 1 } ; hence ∥ ∇ 2 C η ∥ ≤ 3 \lVert\nabla^{2}C_{\eta}\rVert\le3 ∥ ∇ 2 C η ∥ ≤ 3 and ∥ ∇ C η ( t ) ∥ ≤ 3 ∥ t ∥ \lVert\nabla C_{\eta}(t)\rVert\le3\lVert t\rVert ∥ ∇ C η ( t )∥ ≤ 3 ∥ t ∥ . Put K A = A 1 2 + 128 A 2 < 3.6 ⋅ 10 7 K_{A}=A_{1}^{2}+128A_{2}<3.6\cdot10^{7} K A = A 1 2 + 128 A 2 < 3.6 ⋅ 1 0 7 . The public forcing square costs 12 K A λ 12K_{A}\lambda 12 K A λ in Hessian norm; the source and penalty cost 601 λ 601\lambda 601 λ by Lemma 9. As functions of the forcing vector A = A ( P ) A=A(P) A = A ( P ) , the owned terms are T i ( A ) = 12 n λ ∑ j : Σ j = i C η ( ⟨ U j , X , A j , X ⟩ , ⟨ U j , Z , A j , Z ⟩ ) T_{i}(A)=12n\lambda\sum_{j:\Sigma_{j}=i}C_{\eta}(\langle U_{j,X},A_{j,X}\rangle,\langle U_{j,Z},A_{j,Z}\rangle) T i ( A ) = 12 nλ ∑ j : Σ j = i C η (⟨ U j , X , A j , X ⟩ , ⟨ U j , Z , A j , Z ⟩) . Before composition, the stage pairs have disjoint output coordinates, norm-one hidden maps, and exactly one owner. Thus, for every direction v v v ,
( n − 1 ∑ i ∥ D 2 T i ( A ) v ∥ 2 ) 1 / 2 ≤ 36 n λ ∥ v ∥ \left(n^{-1}\sum_{i}\lVert D^{2}T_{i}(A)v\rVert^{2}\right)^{1/2}\le36\sqrt{n}\lambda\lVert v\rVert ( n − 1 i ∑ ∥ D 2 T i ( A ) v ∥ 2 ) 1/2 ≤ 36 n λ ∥ v ∥ and
( n − 1 ∑ i ∥ ∇ T i ( A ) ∥ 2 ) 1 / 2 ≤ 36 n λ ∥ A ∥ . \left(n^{-1}\sum_{i}\lVert\nabla T_{i}(A)\rVert^{2}\right)^{1/2}\le36\sqrt{n}\lambda\lVert A\rVert. ( n − 1 i ∑ ∥ ∇ T i ( A ) ∥ 2 ) 1/2 ≤ 36 n λ ∥ A ∥ . The two chain-rule terms, with ∥ A ∥ ≤ R \lVert A\rVert\le R ∥ A ∥ ≤ R , cost 36 n λ ( A 1 2 + R A 2 / s ) = 36 K A n λ 36\sqrt{n}\lambda(A_{1}^{2}+RA_{2}/s)=36K_{A}\sqrt{n}\lambda 36 n λ ( A 1 2 + R A 2 / s ) = 36 K A n λ . Minkowski’s inequality gives
L P ≤ ( 601 + 48 K A ) n λ < 2 31 n λ . L_{P}\le(601+48K_{A})\sqrt{n}\lambda<2^{31}\sqrt{n}\lambda. L P ≤ ( 601 + 48 K A ) n λ < 2 31 n λ . Without averaging the same argument gives each P P P -Hessian norm below 2 31 n λ = B / 2 2^{31}n\lambda=B/2 2 31 nλ = B /2 . The C 1 , 1 C^{1,1} C 1 , 1 components therefore have these global gradient bounds. The auxiliary gradients occupy distinct coordinates, so their RMS constant is B / n = L / 4 B/\sqrt{n}=L/4 B / n = L /4 . Orthogonality of the P , ξ , y P,\xi,y P , ξ , y blocks gives total RMS constant max { L P , B / n , μ } ≤ L \max\{L_{P},B/\sqrt{n},\mu\}\le L max { L P , B / n , μ } ≤ L , including μ = L \mu=L μ = L .
For τ < B \tau<B τ < B , the i i i th auxiliary curvature at ξ i = π \xi_{i}=\pi ξ i = π is τ − B < 0 \tau-B<0 τ − B < 0 , ruling out global primal convexity at every center. For τ ≥ B \tau\ge B τ ≥ B the P P P part is ( τ − B / 2 ) (\tau-B/2) ( τ − B /2 ) -strongly convex. The i i i th auxiliary curvature τ + B cos ξ i \tau+B\cos\xi_{i} τ + B cos ξ i is nonnegative; at τ = B \tau=B τ = B its isolated zeros still give a positive integral on every nontrivial interval, hence strict convexity. The other coordinates have positive proximal quadratics. All capped and auxiliary terms are bounded, p ≥ 0 p\ge0 p ≥ 0 , and the proximal quadratic is positive, giving coercivity and a unique primal minimizer. The unique dual maximizer is
y + = τ y ˉ μ + τ . (95) y^{+}=\frac{\tau\bar{y}}{\mu+\tau}. \tag*{(95)} y + = μ + τ τ y ˉ . (95) Censoring replaces norm-one hidden maps by contractions; all estimates and the argument for the public auxiliary coordinates persist. This proves the exact domain and unique responses for both actual and censored formulas.
Lemma 18 (Value identity, perturbation and initialization). The value of (92) satisfies the terminal barrier ∥ ∇ ξ η ( P , ξ ) ∥ ≥ 6 ε \lVert\nabla_{\xi}\eta(P,\xi)\rVert\ge6\varepsilon ∥ ∇ ξ η ( P , ξ )∥ ≥ 6 ε whenever ∣ z M + 1 ∣ ≤ s / 8 \lvert z_{M+1}\rvert\le s/8 ∣ z M + 1 ∣ ≤ s /8 , and belongs to the stated class. Moreover y ∗ ( P , ξ ) = 0 y^{*}(P,\xi)=0 y ∗ ( P , ξ ) = 0 everywhere and G 0 = V ( 0 ) − inf V ≤ Δ G_{0}=V(0)-\inf V\le\Delta G 0 = V ( 0 ) − inf V ≤ Δ .
Proof. At width zero, C 0 ( t 1 , t 2 ) = t 1 t 2 C_{0}(t_{1},t_{2})=t_{1}t_{2} C 0 ( t 1 , t 2 ) = t 1 t 2 . Hence
6 λ ∥ A j ∥ 2 + 12 λ t j , 1 t j , 2 = h j 2 [ 6 λ ( x j − z j + 1 / 2 ) 2 + 6 λ ∥ A j , ⊥ ∥ 2 + 3 2 λ ∥ B j + 1 , ⊥ ∥ 2 ] . 6\lambda\|A_{j}\|^{2}+12\lambda t_{j,1}t_{j,2}=h_{j}^{2}\left[6\lambda\left(x_{j}-z_{j+1/2}\right)^{2}+6\lambda\|A_{j,\perp}\|^{2}+\frac{3}{2}\lambda\|B_{j+1,\perp}\|^{2}\right]. 6 λ ∥ A j ∥ 2 + 12 λ t j , 1 t j , 2 = h j 2 [ 6 λ ( x j − z j + 1/2 ) 2 + 6 λ ∥ A j , ⊥ ∥ 2 + 2 3 λ ∥ B j + 1 , ⊥ ∥ 2 ] .
Since averaging leaves exactly one copy of each bridge, the P P P value is exactly V 0 V_{0} V 0 in (51) . The auxiliary mean B ∑ i ( 1 − cos ξ i ) / n B\sum_{i}(1-\cos\xi_{i})/n B ∑ i ( 1 − cos ξ i ) / n is nonnegative and vanishes with its gradient at 0. Use ϕ η ′ ( t ) = t − clip η ( t ) \phi_{\eta}'(t)=t-\operatorname{clip}_{\eta}(t) ϕ η ′ ( t ) = t − clip η ( t ) and ∣ ϕ η ( t ) − t 2 / 2 ∣ ≤ η ∣ t ∣ |\phi_{\eta}(t)-t^{2}/2|\leq\eta|t| ∣ ϕ η ( t ) − t 2 /2∣ ≤ η ∣ t ∣ . The map taking each pair to its sum has norm 2 \sqrt{2} 2 . Let T η , m e a n = n − 1 ∑ i T i T_{\eta,\mathrm{mean}}=n^{-1}\sum_{i}T_{i} T η , mean = n − 1 ∑ i T i , with T i T_{i} T i from the proof of Lemma 17 , and let T 0 , m e a n T_{0,\mathrm{mean}} T 0 , mean be its value with C 0 C_{0} C 0 . The M M M pair-sum clips and the 2 M 2M 2 M individual clips give, before composition,
∥ ∇ A ( T η , m e a n − T 0 , m e a n ) ∥ ≤ 24 2 M λ η , \|\nabla_{A}(T_{\eta,\mathrm{mean}}-T_{0,\mathrm{mean}})\|\leq24\sqrt{2}M\lambda\eta, ∥ ∇ A ( T η , mean − T 0 , mean ) ∥ ≤ 24 2 M λ η ,
∣ T η , m e a n − T 0 , m e a n ∣ ≤ 24 2 M λ η ∥ A ∥ . |T_{\eta,\mathrm{mean}}-T_{0,\mathrm{mean}}|\leq24\sqrt{2}M\lambda\eta\|A\|. ∣ T η , mean − T 0 , mean ∣ ≤ 24 2 M λ η ∥ A ∥.
After composition and the choice of η \eta η ,
∥ ∇ P V η − ∇ V 0 ∥ ≤ c ⋆ λ s / 4 , \|\nabla_{P}V_{\eta}-\nabla V_{0}\|\leq c_{\star}\lambda s/4, ∥ ∇ P V η − ∇ V 0 ∥ ≤ c ⋆ λ s /4 ,
∣ V η ( P , 0 ) − V 0 ( P ) ∣ ≤ ( 32 c ⋆ / A 1 ) λ s 2 < 0.014 λ s 2 . |V_{\eta}(P,0)-V_{0}(P)|\leq(32c_{\star}/A_{1})\lambda s^{2}<0.014\lambda s^{2}. ∣ V η ( P , 0 ) − V 0 ( P ) ∣ ≤ ( 32 c ⋆ / A 1 ) λ s 2 < 0.014 λ s 2 .
Lemma 10 now gives the terminal bound; the orthogonal auxiliary gradient cannot cancel the P P P gradient. At the origin, the sole nonzero derivative has norm e λ s \sqrt{e}\lambda s e λ s . The source bound J M ≥ − 5 e M J_{M}\geq-5eM J M ≥ − 5 e M and (27) imply
V η ( 0 ) − inf V η ≤ ( 5 e M + 0.014 ) λ s 2 < Δ / 2 , e λ 2 s 2 ≤ e λ Δ / 64 < 2 L Δ . V_{\eta}(0)-\inf V_{\eta}\leq(5eM+0.014)\lambda s^{2}<\Delta/2,\qquad e\lambda^{2}s^{2}\leq e\lambda\Delta/64<2L\Delta. V η ( 0 ) − inf V η ≤ ( 5 e M + 0.014 ) λ s 2 < Δ/2 , e λ 2 s 2 ≤ e λ Δ/64 < 2 L Δ.
The dual is separable and maximized at zero, proving the identity for G 0 G_{0} G 0 .
Exact Proximal Hiding for the Primal Family
Lemma 19 (Localization of the complete exact response). The family (92) admits an exact owner-prefix simulation for arbitrary finite centers and arbitrary legal proximal parameters. Every unexposed primal subblock of its censored proximal response lies in the span of that subblock’s current center. If each unexposed axis has projection norm at most δ h = η / ( 4 R ) \delta_{h}=\eta/(4R) δ h = η / ( 4 R ) onto that span, the actual and censored value, full gradient, and unique proximal response agree.
Proof. Grant U 1 , X U_{1,X} U 1 , X and complete each queried component’s same-owner run: a hit reveals U j , Z U_{j,Z} U j , Z and the next source U j + 1 , X U_{j+1,X} U j + 1 , X ; the first mismatch reveals only failure of that label. Censor every unexposed scalar projection, retaining public norms and maps. For an unexposed subblock the censored regularized objective is invariant under all orthogonal transformations fixing its center. Its unique minimizer (Lemma 17 ) therefore lies in that center’s span, or is zero when the center is zero.
On the projection event, both center and censored minimizer have every unexposed capped coordinate bounded by R δ h = η / 4 < s / 8 R\delta_{h}=\eta/4<s/8 R δ h = η /4 < s /8 , independently of raw radii, so termwise equality follows. Earlier bridges use exposed axes. Let j j j be the first unexposed stage. Its
source and incoming connector are known, its outgoing gate factor is locally b + ( z j + 1 / s ) = 1 b_{+}(z_{j+1}/s)=1 b + ( z j + 1 / s ) = 1 , so its forcing square is radial, and its owned interaction is absent because its owner mismatched. The gate weights h k h_k h k of later stages contain the factor α ( z j + 1 / s ) \alpha(z_{j+1}/s) α ( z j + 1 / s ) , which vanishes locally. Each later source vanishes through its own unexposed connector, where both Ψ ( ± z k / s ) \Psi(\pm z_k/s) Ψ ( ± z k / s ) and their derivatives vanish. For any fully hidden pair, ∣ t 1 ∣ , ∣ t 2 ∣ ≤ η / 4 |t_1|, |t_2| \le\eta/4 ∣ t 1 ∣ , ∣ t 2 ∣ ≤ η /4 and ∣ t 1 + t 2 ∣ ≤ η / 2 |t_1+t_2| \le\eta/2 ∣ t 1 + t 2 ∣ ≤ η /2 , so all three terms of (90) vanish with their full gradients. These strict neighborhood margins establish equality of center tuples and of full gradients at the censored minimizer, including omitted directions. It is thus stationary for the actual globally convex regularized component and equals its unique minimizer. The dual response is the public (95) .
Lemma 20 (Primal-family information bound). Under (25), there is a fixed realization of (92) with polynomial dimension such that E N ≥ 133 n M / 6144 \mathbb{E}N \ge133nM/6144 E N ≥ 133 n M /6144 . The assertion includes arbitrary randomized query points, legal proximal parameters, stopping times and outputs. Up to the same smaller constant, it also holds for success probability 2 / 3 2/3 2/3 under a deterministic cap on the number of queries.
Proof. We run the argument of Appendix C.§ 1 with three changes. First, the localizer is Lemma 19 , the public legality test is τ < B \tau< B τ < B , and all admissible censored problems have unique responses by Lemma 17 ; fallback and exposed owner runs therefore read only the revealed prefix, and the null-set argument handles almost-sure legality as before. Second, each current-center span has rank at most one, there are K = 2 M K=2M K = 2 M blocks, and we put H = M ( n − 1 ) H=M(n-1) H = M ( n − 1 ) , q = ⌊ H / 16 ⌋ q=\lfloor H/16\rfloor q = ⌊ H /16 ⌋ and
D = ⌈ 1024 δ h − 2 log ( 1024 K ( q + 1 ) ) ⌉ . (96) D=\left\lceil1024\delta_h^{-2}\log(1024K(q+1))\right\rceil. \tag*{(96)} D = ⌈ 1024 δ h − 2 log ( 1024 K ( q + 1 )) ⌉ . (96)
The spherical bound (79) and a union bound over at most K ( q + 1 ) K(q+1) K ( q + 1 ) tests, including a stopped or virtual output, give geometric failure below 1 / 64 1/64 1/64 , because D δ h 2 / 8 ≥ 128 log ( 1024 K ( q + 1 ) ) ≥ log ( 640 K ( q + 1 ) ) D\delta_h^2/8 \ge128\log(1024K(q+1)) \ge\log(640K(q+1)) D δ h 2 /8 ≥ 128 log ( 1024 K ( q + 1 )) ≥ log ( 640 K ( q + 1 )) . Off this event an unexposed terminal axis has ∣ z M + 1 ∣ ≤ R δ h = η / 4 < s / 8 |z_{M+1}| \le R\delta_h=\eta/4 < s/8 ∣ z M + 1 ∣ ≤ R δ h = η /4 < s /8 , hence gradient at least 6 ϵ 6\epsilon 6 ϵ . Third, each same-owner run stops at its first miss, and each miss response reads only the prefix and the failed label, so the augmented ideal history satisfies the posterior interface of Lemma 14 with E = M E=M E = M , without conditioning on geometric success.
With these inputs, (83) , (84) and (85) give P ( N > q ) ≥ 133 / 192 \mathbb{P}(N>q) \ge133/192 P ( N > q ) ≥ 133/192 and E N ≥ 133 H / 3072 ≥ 133 n M / 6144 \mathbb{E}N \ge133H/3072 \ge133nM/6144 E N ≥ 133 H /3072 ≥ 133 n M /6144 , including q = 0 q=0 q = 0 . Averaging over the hard prior selects one realization, and under a deterministic cap q q q the failure probability is at least 7 / 8 − 1 / 64 = 55 / 64 > 1 / 3 7/8-1/64=55/64>1/3 7/8 − 1/64 = 55/64 > 1/3 . Since δ h − 2 = O ( M ) \delta_h^{-2}=O(M) δ h − 2 = O ( M ) , the dimension is 2 M D + n + 1 = O ( M 2 log ( n M ) + n ) 2MD+n+1=O(M^2\log(nM)+n) 2 M D + n + 1 = O ( M 2 log ( n M ) + n ) .
The Indexed Baseline and Completion of All Accuracy Windows
Lemma 21 (An indexed baseline with exact proximal responses). For n ≥ 2 n\ge2 n ≥ 2 , κ ≥ 1 \kappa\ge1 κ ≥ 1 , and ϵ 2 ≤ 2 L Δ / 9 \epsilon^2\le2L\Delta/9 ϵ 2 ≤ 2 L Δ/9 , the class contains fixed instances requiring worst-case expected cost at least n / 16 n/16 n /16 for expected-norm accuracy. The same lower bound holds for success probability 2 / 3 2/3 2/3 under a deterministic query cap.
Proof. Choose independent uniform I ∈ [ n ] I \in[n] I ∈ [ n ] and sign σ ∈ { − 1 , 1 } \sigma\in\{-1,1\} σ ∈ { − 1 , 1 } , and set (cf. the hidden-component quadratic of Zhou, [27 ] , Appendix D.3)
F i ( x , y ) = L x 2 / 2 − 3 n σ ε 1 { i = I } x − μ y 2 / 2. F_i(x,y) = Lx^2/2 - 3n\sigma\varepsilon\mathbf{1}_{\{i=I\}}x - \mu y^2/2. F i ( x , y ) = L x 2 /2 − 3 nσ ε 1 { i = I } x − μ y 2 /2.
The joint RMS constant is L L L , every component is μ \mu μ -strongly concave in y y y , and every τ > 0 \tau> 0 τ > 0 is legal with a unique response. The mean initial gap and squared joint gradient are 9 ε 2 / ( 2 L ) 9\varepsilon^2/(2L) 9 ε 2 / ( 2 L ) and 9 ε 2 9\varepsilon^2 9 ε 2 , respectively, within the budgets. Before index I I I is queried, center values, gradients, and proximal responses are all sign-independent. At any sign-independent output,
1 2 ∣ L x − 3 ε ∣ + 1 2 ∣ L x + 3 ε ∣ ≥ 3 ε ≥ 2 ε . \frac{1}{2}|Lx-3\varepsilon|+\frac{1}{2}|Lx+3\varepsilon|\geq3\varepsilon\geq2\varepsilon. 2 1 ∣ Lx − 3 ε ∣ + 2 1 ∣ Lx + 3 ε ∣ ≥ 3 ε ≥ 2 ε .
For q = ⌊ n / 4 ⌋ q=\lfloor n/4\rfloor q = ⌊ n /4 ⌋ , adaptive search hits I I I within q q q calls with probability at most q / n q/n q / n , since before the hit the label sequence depends only on misses. Thus expected accuracy implies
ε ≥ 2 ε [ 1 − P ( N > q ) − q / n ] , E N ≥ ( q + 1 ) / 4 ≥ n / 16. \varepsilon\geq2\varepsilon[1-\mathbb{P}(N>q)-q/n], \qquad\mathbb{E}N \geq(q+1)/4 \geq n/16. ε ≥ 2 ε [ 1 − P ( N > q ) − q / n ] , E N ≥ ( q + 1 ) /4 ≥ n /16.
For a deterministic cap q q q , the success intervals for the two signs are disjoint and success is at most 1 / 2 + q / ( 2 n ) ≤ 5 / 8 < 2 / 3 1/2+q/(2n)\leq5/8<2/3 1/2 + q / ( 2 n ) ≤ 5/8 < 2/3 . Averaging over the prior fixes a hard instance. ■
Proof of Theorem 1. Write S = L Δ / ε 2 S=L\Delta/\varepsilon^2 S = L Δ/ ε 2 and h ( n , κ ) = min { n κ , n 3 / 4 κ } h(n,\kappa)=\min\{\sqrt{n\kappa},n^{3/4}\sqrt{\kappa}\} h ( n , κ ) = min { nκ , n 3/4 κ } . The common accuracy window is stronger than that of Lemma 21 , so the n / 16 n/16 n /16 baseline is always available. If κ ≥ 2 18 n \kappa\geq2^{18}\sqrt{n} κ ≥ 2 18 n , Theorem 3 gives the accuracy term, which is the second branch of h h h .
For 64 ≤ κ ≤ 2 18 n 64\leq\kappa\leq2^{18}\sqrt{n} 64 ≤ κ ≤ 2 18 n , the two-node coefficient obeys
λ = 2 − 39 L κ κ + 2 16 n ≥ L κ 5 ⋅ 2 55 n . \lambda=\frac{2^{-39}L\kappa}{\kappa+2^{16}\sqrt{n}}\geq\frac{L\kappa}{5\cdot2^{55}\sqrt{n}}. λ = κ + 2 16 n 2 − 39 L κ ≥ 5 ⋅ 2 55 n L κ .
If its path window holds, Corollary 16 and the floor bound give
E N ≥ 133 c ⋆ 2 6144 ⋅ 4096 n λ ε 2 ≥ 133 c ⋆ 2 15 ⋅ 2 78 n κ S ≥ 133 c ⋆ 2 15 ⋅ 2 78 h ( n , κ ) S . \mathbb{E}N\geq\frac{133c_\star^2}{6144\cdot4096}\frac{n\lambda}{\varepsilon^2}\geq\frac{133c_\star^2}{15\cdot2^{78}}\sqrt{n\kappa}S\geq\frac{133c_\star^2}{15\cdot2^{78}}h(n,\kappa)S. E N ≥ 6144 ⋅ 4096 133 c ⋆ 2 ε 2 nλ ≥ 15 ⋅ 2 78 133 c ⋆ 2 nκ S ≥ 15 ⋅ 2 78 133 c ⋆ 2 h ( n , κ ) S .
If the path window fails, instead
h ( n , κ ) S ≤ n κ S < 5 ⋅ 2 67 c ⋆ 2 n . h(n,\kappa)S\leq\sqrt{n\kappa}S<\frac{5\cdot2^{67}}{c_\star^2}n. h ( n , κ ) S ≤ nκ S < c ⋆ 2 5 ⋅ 2 67 n .
The indexed baseline then supplies the requested sum up to a universal constant. For 1 ≤ κ < 64 1\leq\kappa<64 1 ≤ κ < 64 , use the primal family. In its path window, Lemma 20 gives
E N ≥ 133 c ⋆ 2 6144 ⋅ 4096 ⋅ 2 34 n S = 133 c ⋆ 2 3 ⋅ 2 57 n S . \mathbb{E}N\geq\frac{133c_\star^2}{6144\cdot4096\cdot2^{34}}\sqrt{n}S=\frac{133c_\star^2}{3\cdot2^{57}}\sqrt{n}S. E N ≥ 6144 ⋅ 4096 ⋅ 2 34 133 c ⋆ 2 n S = 3 ⋅ 2 57 133 c ⋆ 2 n S .
Since h ( n , κ ) ≤ 64 n h(n,\kappa)\leq64\sqrt{n} h ( n , κ ) ≤ 64 n , this bounds the required accuracy term. If its path window fails, n S < 2 46 n / c ⋆ 2 \sqrt{n}S<2^{46}n/c_\star^2 n S < 2 46 n / c ⋆ 2 , so h ( n , κ ) S < 2 52 n / c ⋆ 2 h(n,\kappa)S<2^{52}n/c_\star^2 h ( n , κ ) S < 2 52 n / c ⋆ 2 , again covered by the indexed baseline. In a path window, the maximum of the baseline and accuracy lower bounds is at least half their sum. Outside it, the accuracy term is at most the displayed universal multiple of n n n . The common constant c = c ⋆ 2 / 2 96 c=c_\star^2/2^{96} c = c ⋆ 2 / 2 96 is valid in all cases. Both lower bounds are worst cases over instances with the same public parameters.
Algorithm 1 Normalized-start method of Theorem 22
Require: L L L , μ \mu μ , Δ \Delta Δ , ε \varepsilon ε ; a start u i n = ( x i n , y i n ) u_{\mathrm{in}}=(x_{\mathrm{in}},y_{\mathrm{in}}) u in = ( x in , y in ) with V ( x i n ) − V ∗ ≤ Δ V(x_{\mathrm{in}})-V_* \le\Delta V ( x in ) − V ∗ ≤ Δ and ∥ ∇ F ( u i n ) ∥ 2 ≤ 2 L Δ \lVert\nabla F(u_{\mathrm{in}})\rVert^2 \le2L\Delta ∥ ∇ F ( u in ) ∥ 2 ≤ 2 L Δ
set the parameters (100), (112) and (114), and the budgets j w j_w j w of (110) and j p j_p j p of (116)
u w ← u_w \leftarrow u w ← endpoint of j w j_w j w L-SVRE iterations (108) for Q x i n Q_{x_{\mathrm{in}}} Q x in , from u i n u_{\mathrm{in}} u in
( x 0 , y 0 ) ← u w , z 0 ← x i n , v 0 ← 0 (x_0,y_0)\leftarrow u_w,\quad z_0\leftarrow x_{\mathrm{in}},\quad v_0\leftarrow0 ( x 0 , y 0 ) ← u w , z 0 ← x in , v 0 ← 0
for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 do
apply the updates (101), whose gradient requests j = 3 t , 3 t + 1 , 3 t + 2 j=3t,3t+1,3t+2 j = 3 t , 3 t + 1 , 3 t + 2 use g ^ j = ∇ F ( u j ) \widehat{g}_j=\nabla F(u_j) g j = ∇ F ( u j ) if j ≡ 0 ( m o d q ) j\equiv0\pmod q j ≡ 0 ( mod q ) ; otherwise (111) with b b b fresh uniform indices
end for
draw J J J uniformly from { 0 , … , T − 1 } \{0,\ldots,T-1\} { 0 , … , T − 1 }
u ^ ← \widehat{u}\leftarrow u ← endpoint of j p j_p j p L-SVRE iterations for Q z J Q_{z_J} Q z J with fresh randomness, from ( x J + 1 , y J + 1 ) (x_{J+1},y_{J+1}) ( x J + 1 , y J + 1 )
return the primal coordinate x ^ \widehat{x} x of u ^ \widehat{u} u
The confidence statements of the three families and the baseline give the same rate with this smaller common constant. All components are C 1 , 1 C^{1,1} C 1 , 1 and μ \mu μ -strongly concave, have y ∗ ( 0 ) = 0 y^*(0)=0 y ∗ ( 0 ) = 0 , and have unique responses on common exact legal domains. The long-path dimension is (16); the intermediate and primal dimensions were proved above and are polynomial since M = O ( S ) M=O(S) M = O ( S ) . For G 0 G_0 G 0 , the long-path family has G 0 ≤ 1025 Δ G_0\le1025\Delta G 0 ≤ 1025Δ , the primal family has G 0 ≤ Δ G_0\le\Delta G 0 ≤ Δ , and the intermediate family has the bound of Corollary 16.
A Normalized-Start Upper Bound for Every Condition Number
We use the stochastic projected damped extragradient (SPDE) update of Zhang et al. [24 ] and the solver for the mean operator of Alacaoglu and Malitsky [1 ] and Luo et al. [12 ] . Fixed differentiable components satisfy (2), with mean μ \mu μ -strong concavity, n ≥ 2 n\ge2 n ≥ 2 , κ ≥ 1 \kappa\ge1 κ ≥ 1 and S = L Δ / ε 2 ≥ 2 S=L\Delta/\varepsilon^2\ge2 S = L Δ/ ε 2 ≥ 2 . The finite supplied pair u i n = ( x i n , y i n ) u_{\mathrm{in}}=(x_{\mathrm{in}},y_{\mathrm{in}}) u in = ( x in , y in ) obeys V ( x i n ) − V ∗ ≤ Δ V(x_{\mathrm{in}})-V_*\le\Delta V ( x in ) − V ∗ ≤ Δ and ∥ ∇ F ( u i n ) ∥ 2 ≤ 2 L Δ \lVert\nabla F(u_{\mathrm{in}})\rVert^2\le2L\Delta ∥ ∇ F ( u in ) ∥ 2 ≤ 2 L Δ ; in this appendix we write x x x for the primal variable and M \mathcal{M} M for the Moreau envelope below; as in the lower-bound appendices, symbols defined in Appendices E and F are local. Proximal regularizers use exact arithmetic; each gradient call queries one component F i F_i F i .
Theorem 22 (Normalized-start algorithm for all condition numbers). Under these assumptions, a randomized component-gradient algorithm achieves E ∥ ∇ V ( x ^ ) ∥ 2 ≤ ε 2 \mathbb{E}\lVert\nabla V(\widehat{x})\rVert^2\le\varepsilon^2 E ∥ ∇ V ( x ) ∥ 2 ≤ ε 2 with expected cost at most
C N [ n + n κ S + ( n + n κ ) log ( 2 κ ) ] C_N\left[n+\sqrt{n\kappa}S+(n+\sqrt{n\kappa})\log(2\kappa)\right] C N [ n + nκ S + ( n + nκ ) log ( 2 κ ) ] for a universal constant C N C_N C N . The iteration budgets are finite and fixed from supplied parameters. This statement holds for every κ ≥ 1 \kappa\ge1 κ ≥ 1 .
We prove the theorem by initializing the proximal energy, running a fixed SPDE budget with recursive estimates, and refining one uniformly selected endpoint (Algorithm 1). All three stages use component gradients.
The component inequality is
n − 1 ∑ i ∥ ∇ F i ( u ) − ∇ F i ( u ′ ) ∥ 2 ≤ L 2 ∥ u − u ′ ∥ 2 . (97) n^{-1}\sum_i \lVert\nabla F_i(u)-\nabla F_i(u')\rVert^2\le L^2\lVert u-u'\rVert^2. \tag*{(97)} n − 1 i ∑ ∥ ∇ F i ( u ) − ∇ F i ( u ′ ) ∥ 2 ≤ L 2 ∥ u − u ′ ∥ 2 . (97)
Unconstrained Proximal Geometry
Strong concavity makes F ( x , ⋅ ) F(x,\cdot) F ( x , ⋅ ) coercive from above, so its maximum is attained uniquely at y ∗ ( x ) y^*(x) y ∗ ( x ) . Joint mean L L L -smoothness follows from Jensen’s inequality in (97) . The two dual optimality conditions imply ∥ y ∗ ( x ) − y ∗ ( x ′ ) ∥ ≤ κ ∥ x − x ′ ∥ \lVert y^*(x)-y^*(x')\rVert\leq\kappa\lVert x-x'\rVert ∥ y ∗ ( x ) − y ∗ ( x ′ )∥ ≤ κ ∥ x − x ′ ∥ . Consequently V V V is differentiable, L ( 1 + κ ) L(1+\kappa) L ( 1 + κ ) -smooth and L L L -weakly convex (cf. Lin et al., [10 ] , stated there for bounded dual sets). Its infimum V ∗ > − ∞ V_* >-\infty V ∗ > − ∞ by the supplied primal gap. Define the proximal objective Q z Q_z Q z , the Moreau envelope M M M of V V V (Davis and Drusvyatskiy, [5 ] ), and the proximal saddle u z ∗ u_z^* u z ∗ by
Q z ( x , y ) = F ( x , y ) + L ∥ x − z ∥ 2 , M ( z ) = min x { V ( x ) + L ∥ x − z ∥ 2 } , u z ∗ = ( x z ∗ , y z ∗ ) = ( argmin x { V ( x ) + L ∥ x − z ∥ 2 } , y ∗ ( x z ∗ ) ) , inf M = V ∗ , ∇ M ( z ) = 2 L ( z − x z ∗ ) = ∇ x F ( u z ∗ ) , ∇ y F ( u z ∗ ) = 0. (98) \begin{aligned}
Q_z(x,y)&=F(x,y)+L\lVert x-z\rVert^2,\qquad M(z)=\min_x\{V(x)+L\lVert x-z\rVert^2\},\\
u_z^*&=(x_z^*,y_z^*)=\left(\underset{x}{\operatorname{argmin}}\{V(x)+L\lVert x-z\rVert^2\},y^*(x_z^*)\right),\\
\inf M&=V_*,\qquad\nabla M(z)=2L(z-x_z^*)=\nabla_xF(u_z^*),\qquad\nabla_yF(u_z^*)=0. \tag*{(98)}
\end{aligned} Q z ( x , y ) u z ∗ inf M = F ( x , y ) + L ∥ x − z ∥ 2 , M ( z ) = x min { V ( x ) + L ∥ x − z ∥ 2 } , = ( x z ∗ , y z ∗ ) = ( x argmin { V ( x ) + L ∥ x − z ∥ 2 } , y ∗ ( x z ∗ ) ) , = V ∗ , ∇ M ( z ) = 2 L ( z − x z ∗ ) = ∇ x F ( u z ∗ ) , ∇ y F ( u z ∗ ) = 0. (98)
The proximal objective is coercive and L L L -strongly convex, hence has a unique minimizer. Its first-order condition and dual optimality make u z ∗ u_z^* u z ∗ the unique unconstrained saddle. Strong monotonicity of the mean proximal saddle operator gives
L ∥ x z ∗ − x z ′ ∗ ∥ 2 + μ ∥ y z ∗ − y z ′ ∗ ∥ 2 ≤ 2 L ⟨ z − z ′ , x z ∗ − x z ′ ∗ ⟩ . L\lVert x_z^*-x_{z'}^*\rVert^2+\mu\lVert y_z^*-y_{z'}^*\rVert^2\leq2L\langle z-z',x_z^*-x_{z'}^*\rangle. L ∥ x z ∗ − x z ′ ∗ ∥ 2 + μ ∥ y z ∗ − y z ′ ∗ ∥ 2 ≤ 2 L ⟨ z − z ′ , x z ∗ − x z ′ ∗ ⟩ .
Young’s inequality therefore yields
∥ x z ∗ − x z ′ ∗ ∥ ≤ 2 ∥ z − z ′ ∥ , ∥ y z ∗ − y z ′ ∗ ∥ ≤ 2 L μ ∥ z − z ′ ∥ , Lip ( ∇ M ) ≤ 6 L . (99) \lVert x_z^*-x_{z'}^*\rVert\leq2\lVert z-z'\rVert,\qquad\lVert y_z^*-y_{z'}^*\rVert\leq\sqrt{\frac{2L}{\mu}}\lVert z-z'\rVert,\qquad\operatorname{Lip}(\nabla M)\leq6L. \tag*{(99)} ∥ x z ∗ − x z ′ ∗ ∥ ≤ 2 ∥ z − z ′ ∥ , ∥ y z ∗ − y z ′ ∗ ∥ ≤ μ 2 L ∥ z − z ′ ∥ , Lip ( ∇ M ) ≤ 6 L . (99)
Pathwise SPDE Descent and Motion Bounds
Set
s = 2 L μ , h = ( 192 L ) − 1 , d = 1 / 64 , α = ( 1 + h s / 16 ) − 1 , s=\sqrt{2L\mu},\qquad h=(192L)^{-1},\qquad d=1/64,\qquad\alpha=(1+hs/16)^{-1}, s = 2 Lμ , h = ( 192 L ) − 1 , d = 1/64 , α = ( 1 + h s /16 ) − 1 ,
k = h ( 8 L − s ) 16 + h s , β = h s / 4096 , C = 140. (100) k=\frac{h(8L-s)}{16+hs},\qquad\beta=hs/4096,\qquad C=140. \tag*{(100)} k = 16 + h s h ( 8 L − s ) , β = h s /4096 , C = 140. (100)
From any initial state, use three gradient requests g ^ j \widehat{g}_j g j at points u j u_j u j per iteration:
u 3 t = ( x t , y t ) , x ~ t = x t − h [ g ^ x , 3 t + 2 L ( x t − z t ) ] , y ~ t = y t + h ( g ^ y , 3 t + v t ) , u 3 t + 1 = ( x ~ t , y ~ t ) , v ‾ t = α v t + k g ^ y , 3 t + 1 , x t + 1 = x t − h [ g ^ x , 3 t + 1 + 2 L ( x ~ t − z t ) ] , y t + 1 = y t + h ( g ^ y , 3 t + 1 + v ‾ t ) , u 3 t + 2 = ( x t + 1 , y t + 1 ) , v t + 1 = α v t + k g ^ y , 3 t + 2 , z t + 1 = z t + β ( x t + 1 − z t ) . (101) \begin{aligned}
u_{3t}&=(x_t,y_t),&\widetilde{x}_t&=x_t-h[\widehat{g}_{x,3t}+2L(x_t-z_t)],\\
\widetilde{y}_t&=y_t+h(\widehat{g}_{y,3t}+v_t),&u_{3t+1}&=(\widetilde{x}_t,\widetilde{y}_t),\\
\overline{v}_t&=\alpha v_t+k\widehat{g}_{y,3t+1},&x_{t+1}&=x_t-h[\widehat{g}_{x,3t+1}+2L(\widetilde{x}_t-z_t)],\\
y_{t+1}&=y_t+h(\widehat{g}_{y,3t+1}+\overline{v}_t),&u_{3t+2}&=(x_{t+1},y_{t+1}),\\
v_{t+1}&=\alpha v_t+k\widehat{g}_{y,3t+2},&z_{t+1}&=z_t+\beta(x_{t+1}-z_t). \tag*{(101)}
\end{aligned} u 3 t y t v t y t + 1 v t + 1 = ( x t , y t ) , = y t + h ( g y , 3 t + v t ) , = α v t + k g y , 3 t + 1 , = y t + h ( g y , 3 t + 1 + v t ) , = α v t + k g y , 3 t + 2 , x t u 3 t + 1 x t + 1 u 3 t + 2 z t + 1 = x t − h [ g x , 3 t + 2 L ( x t − z t )] , = ( x t , y t ) , = x t − h [ g x , 3 t + 1 + 2 L ( x t − z t )] , = ( x t + 1 , y t + 1 ) , = z t + β ( x t + 1 − z t ) . (101)
Always u 3 t = u 3 t − 1 u_{3t}=u_{3t-1} u 3 t = u 3 t − 1 for t ≥ 1 t\geq1 t ≥ 1 . Proximal gradients are evaluated by exact arithmetic, outside the gradient estimator.
For w = ( x , y , v ) w=(x,y,v) w = ( x , y , v ) at center z z z , write X = x − x z ∗ X=x-x_z^* X = x − x z ∗ , Y ′ = y − y z ∗ Y'=y-y_z^* Y ′ = y − y z ∗ , P = ∇ x Q z ( x , y ) P=\nabla_xQ_z(x,y) P = ∇ x Q z ( x , y ) , R = − ∇ y F ( x , y ) R=-\nabla_yF(x,y) R = − ∇ y F ( x , y ) and Q = R − v Q=R-v Q = R − v . The common potential is
E z ( w ) = − ( 1 − s 16 L ) ( Q z ( x , y ) − M ( z ) ) + ∥ P ∥ 2 + ∥ Q ∥ 2 L + s 16 L ⟨ v , Y ′ ⟩ + μ 256 ∥ Y ′ ∥ 2 , E_z(w)=-\left(1-\frac{s}{16L}\right)(Q_z(x,y)-M(z))+\frac{\lVert P\rVert^2+\lVert Q\rVert^2}{L}+\frac{s}{16L}\langle v,Y'\rangle+\frac{\mu}{256}\lVert Y'\rVert^2, E z ( w ) = − ( 1 − 16 L s ) ( Q z ( x , y ) − M ( z )) + L ∥ P ∥ 2 + ∥ Q ∥ 2 + 16 L s ⟨ v , Y ′ ⟩ + 256 μ ∥ Y ′ ∥ 2 ,
H z ( w ) = E z ( w ) + s 256 L 2 ∥ v ∥ 2 , V ( z , w ) = M ( z ) − V ∗ + H z ( w ) . (102) H_z(w)=E_z(w)+\frac{s}{256L^2}\lVert v\rVert^2,\qquad\mathcal{V}(z,w)=M(z)-V_*+H_z(w). \tag*{(102)} H z ( w ) = E z ( w ) + 256 L 2 s ∥ v ∥ 2 , V ( z , w ) = M ( z ) − V ∗ + H z ( w ) . (102)
Strong saddle inequalities give
Q z ( x , y ) − M ( z ) + ⟨ R , Y ′ ⟩ ≥ L 2 ∥ X ∥ 2 + μ 2 ∥ Y ′ ∥ 2 , Q z ( x , y ) − M ( z ) ≤ ∥ P ∥ 2 2 L − μ 2 ∥ Y ∥ 2 . Q_z(x,y)-\mathcal{M}(z)+\langle R,Y'\rangle\ge\frac{L}{2}\lVert X\rVert^2+\frac{\mu}{2}\lVert Y'\rVert^2,\qquad Q_z(x,y)-\mathcal{M}(z)\le\frac{\lVert P\rVert^2}{2L}-\frac{\mu}{2}\lVert Y\rVert^2. Q z ( x , y ) − M ( z ) + ⟨ R , Y ′ ⟩ ≥ 2 L ∥ X ∥ 2 + 2 μ ∥ Y ′ ∥ 2 , Q z ( x , y ) − M ( z ) ≤ 2 L ∥ P ∥ 2 − 2 μ ∥ Y ∥ 2 .
Completing the Q , Y ′ Q,Y' Q , Y ′ square yields the following bound; in particular V ≥ M − V ∗ ≥ 0 \mathcal{V}\ge\mathcal{M}-V_*\ge0 V ≥ M − V ∗ ≥ 0 .
E z ( w ) ≥ ∥ P ∥ 2 2 L + s 32 ∥ X ∥ 2 + μ 2 ∥ Y ′ ∥ 2 + 1 L ∥ Q − s Y ′ / 32 ∥ 2 ≥ 0. (103) E_z(w)\ge\frac{\lVert P\rVert^2}{2L}+\frac{s}{32}\lVert X\rVert^2+\frac{\mu}{2}\lVert Y'\rVert^2+\frac{1}{L}\lVert Q-sY'/32\rVert^2\ge0. \tag*{(103)} E z ( w ) ≥ 2 L ∥ P ∥ 2 + 32 s ∥ X ∥ 2 + 2 μ ∥ Y ′ ∥ 2 + L 1 ∥ Q − s Y ′ /32 ∥ 2 ≥ 0. (103)
Let η j = g ^ j − ∇ F ( u j ) \eta_j=\widehat{g}_j-\nabla F(u_j) η j = g j − ∇ F ( u j ) and ( a t , b t , c t ) = ( η 3 t , η 3 t + 1 , η 3 t + 2 ) (a_t,b_t,c_t)=(\eta_{3t},\eta_{3t+1},\eta_{3t+2}) ( a t , b t , c t ) = ( η 3 t , η 3 t + 1 , η 3 t + 2 ) . The proof uses
v t + 1 ∘ = α v t + k ∇ y F ( x t + 1 , y t + 1 ) , v t + 1 = v t + 1 ∘ + k c y , t . (104) v^\circ_{t+1}=\alpha v_t+k\nabla_yF(x_{t+1},y_{t+1}),\qquad v_{t+1}=v^\circ_{t+1}+kc_{y,t}. \tag*{(104)} v t + 1 ∘ = α v t + k ∇ y F ( x t + 1 , y t + 1 ) , v t + 1 = v t + 1 ∘ + k c y , t . (104)
This is the true-gradient momentum at the realized endpoint. Fix center z t z_t z t when defining P t , Q t P_t,Q_t P t , Q t as above, and set
P t + = ∇ x Q z t ( u 3 t + 2 ) , Q t + = − ∇ y F ( u 3 t + 2 ) − v t + 1 ∘ , X t + = x t + 1 − x z t ∗ , Y t + = y t + 1 − y z t ∗ , P_t^+=\nabla_xQ_{z_t}(u_{3t+2}),\qquad Q_t^+=-\nabla_yF(u_{3t+2})-v^\circ_{t+1},\qquad X_t^+=x_{t+1}-x_{z_t}^*,\qquad Y_t^+=y_{t+1}-y_{z_t}^*, P t + = ∇ x Q z t ( u 3 t + 2 ) , Q t + = − ∇ y F ( u 3 t + 2 ) − v t + 1 ∘ , X t + = x t + 1 − x z t ∗ , Y t + = y t + 1 − y z t ∗ ,
I t = ∥ P t + − P t ∥ 2 + ∥ Q t + − Q t ∥ 2 , \mathcal{I}_t=\lVert P_t^+-P_t\rVert^2+\lVert Q_t^+-Q_t\rVert^2, I t = ∥ P t + − P t ∥ 2 + ∥ Q t + − Q t ∥ 2 ,
D t = h 4 ∥ P t + ∥ 2 + 24 β L ∥ Q t + ∥ 2 + 7 β L ∥ v t + 1 ∘ ∥ 2 + 32 μ β ∥ Y t + ∥ 2 + 4 β ( 16 L − s ) ∥ X t + ∥ 2 + I t 4 L . (105) D_t=\frac{h}{4}\lVert P_t^+\rVert^2+\frac{24\beta}{L}\lVert Q_t^+\rVert^2+\frac{7\beta}{L}\lVert v^\circ_{t+1}\rVert^2+32\mu\beta\lVert Y_t^+\rVert^2+4\beta(16L-s)\lVert X_t^+\rVert^2+\frac{\mathcal{I}_t}{4L}. \tag*{(105)} D t = 4 h ∥ P t + ∥ 2 + L 24 β ∥ Q t + ∥ 2 + L 7 β ∥ v t + 1 ∘ ∥ 2 + 32 μ β ∥ Y t + ∥ 2 + 4 β ( 16 L − s ) ∥ X t + ∥ 2 + 4 L I t . (105)
Lemma 23 (Pathwise drift and joint motion). For arbitrary realized vector errors and every finite initial state, with V t = V ( z t , w t ) \mathcal{V}_t=\mathcal{V}(z_t,w_t) V t = V ( z t , w t ) , the updates satisfy
V t + 1 − V t ≤ − 1 4 D t − β 16 L ∥ ∇ M ( z t ) ∥ 2 + C L β ( ∥ a t ∥ 2 + ∥ b t ∥ 2 + ∥ c t ∥ 2 ) , (106) \mathcal{V}_{t+1}-\mathcal{V}_t\le-\frac{1}{4}D_t-\frac{\beta}{16L}\lVert\nabla\mathcal{M}(z_t)\rVert^2+\frac{C}{L\beta}\left(\lVert a_t\rVert^2+\lVert b_t\rVert^2+\lVert c_t\rVert^2\right), \tag*{(106)} V t + 1 − V t ≤ − 4 1 D t − 16 L β ∥ ∇ M ( z t ) ∥ 2 + L β C ( ∥ a t ∥ 2 + ∥ b t ∥ 2 + ∥ c t ∥ 2 ) , (106) ∥ u 3 t + 1 − u 3 t ∥ 2 + ∥ u 3 t + 2 − u 3 t + 1 ∥ 2 ≤ D t L β + ∥ a t ∥ 2 + ∥ b t ∥ 2 L 2 . (107) \lVert u_{3t+1}-u_{3t}\rVert^2+\lVert u_{3t+2}-u_{3t+1}\rVert^2\le\frac{D_t}{L\beta}+\frac{\lVert a_t\rVert^2+\lVert b_t\rVert^2}{L^2}. \tag*{(107)} ∥ u 3 t + 1 − u 3 t ∥ 2 + ∥ u 3 t + 2 − u 3 t + 1 ∥ 2 ≤ L β D t + L 2 ∥ a t ∥ 2 + ∥ b t ∥ 2 . (107)
Proof. The drift and joint-motion estimates are the deterministic inequalities of Zhang et al. [24 ] (Lemmas 11–12, Eqs. (117) and (121) ). Use their strongly concave substitution from the proof of Lemma 10, {xref}Eq. (100)}, as explicitly applied to these lemmas in the proof of Lemma 15: G s c = Q z G_{\mathrm{sc}}=Q_z G sc = Q z , p s c = M p_{\mathrm{sc}}=\mathcal{M} p sc = M , all normal-cone vectors are zero, the virtual momentum is v t + 1 ∘ v^\circ_{t+1} v t + 1 ∘ , and their D t ∘ D_t^\circ D t ∘ is (105) . Their numerical parameters are exactly (100) , including C = 140 C=140 C = 140 .
The cited lemmas are stated for a compact dual domain. Equations (98) –(99) and (103) supply the all-space saddles, envelope and nonnegative energy. For one finite realized step, choose compact primal and dual balls whose interiors contain its centers, its query points and the saddle path between the two centers; such balls exist by (99) . On these balls the projections are inactive, and the saddles and envelope values coincide with their all-space counterparts. The lower energy floor changes by an additive constant, which cancels in one-step differences. Their fixed-center Lemma 5 starts from arbitrary feasible states, and the subsequent center and endpoint estimates, including the endpoint expansion Eq. (88) and the Young bounds (118)–(121), are algebraic in the realized errors. Hence both estimates hold for arbitrary finite initial states and correlated errors.
A Shared Solver with Mean Strong Monotonicity
We use loopless stochastic variance-reduced extragradient (L-SVRE) [1 , 12 ] , with the following self-contained guarantee.
Lemma 24 (L-SVRE contraction under mean strong monotonicity). Let n ≥ 2 n \ge2 n ≥ 2 and A = n − 1 ∑ i A i A = n^{-1}\sum_i A_i A = n − 1 ∑ i A i on R d \mathbb{R}^d R d have a zero z ∗ z^* z ∗ , be α \alpha α -strongly monotone, and satisfy n − 1 ∑ i ∥ A i ( z ) − A i ( z ′ ) ∥ 2 ≤ M s 2 ∥ z − z ′ ∥ 2 n^{-1}\sum_i \lVert A_i(z)-A_i(z')\rVert^2 \le M_s^2\lVert z-z'\rVert^2 n − 1 ∑ i ∥ A i ( z ) − A i ( z ′ ) ∥ 2 ≤ M s 2 ∥ z − z ′ ∥ 2 . No component monotonicity is assumed. Starting from r 0 = w 0 = u r_0 = w_0 = u r 0 = w 0 = u , pay n n n calls for A ( w 0 ) A(w_0) A ( w 0 ) and, with a fresh uniform index i i i , perform
r ˉ = ( 1 − p s ) r + p s w , b = r ˉ − τ A ( w ) , v s = A ( w ) + A i ( b ) − A i ( w ) , r + = r ˉ − τ v s , p s = 1 2 n , τ = 1 4 n M s . (108) \begin{aligned}
\bar r &= (1-p_s)r+p_sw, & b &= \bar r-\tau A(w),\\
v_s &= A(w)+A_i(b)-A_i(w), & r^+ &= \bar r-\tau v_s,\\
p_s &= \frac{1}{2n}, & \tau&= \frac{1}{4\sqrt{n}M_s}. \tag*{(108)}
\end{aligned} r ˉ v s p s = ( 1 − p s ) r + p s w , = A ( w ) + A i ( b ) − A i ( w ) , = 2 n 1 , b r + τ = r ˉ − τ A ( w ) , = r ˉ − τ v s , = 4 n M s 1 . (108) Independently refresh w + = r + w^+ = r^+ w + = r + with probability p s p_s p s , paying n n n calls for its mean, and otherwise retain w w w . Then, for every j ≥ 0 j \ge0 j ≥ 0 ,
E ∥ r j − z ∗ ∥ 2 ≤ 4 ( 1 − θ ) j ∥ u − z ∗ ∥ 2 , θ − 1 = 4 n + 8 n M s / α . (109) \mathbb{E}\lVert r_j-z^*\rVert^2 \le4(1-\theta)^j\lVert u-z^*\rVert^2,\qquad\theta^{-1}=4n+8\sqrt{n}M_s/\alpha. \tag*{(109)} E ∥ r j − z ∗ ∥ 2 ≤ 4 ( 1 − θ ) j ∥ u − z ∗ ∥ 2 , θ − 1 = 4 n + 8 n M s / α . (109) Each iteration costs at most 2 + n p s = 5 / 2 2+np_s=5/2 2 + n p s = 5/2 calls in conditional expectation; the guarantee holds conditionally on preceding history, with fresh randomness.
Proof. Condition on ( r , w ) (r,w) ( r , w ) and write p = p s p=p_s p = p s , t = τ α t=\tau\alpha t = τ α , R = ∥ r − z ∗ ∥ 2 R=\lVert r-z^*\rVert^2 R = ∥ r − z ∗ ∥ 2 , W = ∥ w − z ∗ ∥ 2 W=\lVert w-z^*\rVert^2 W = ∥ w − z ∗ ∥ 2 . Unbiasedness gives E v s = A ( b ) \mathbb{E}v_s=A(b) E v s = A ( b ) and E ∥ r + − b ∥ 2 ≤ τ 2 M s 2 ∥ b − w ∥ 2 \mathbb{E}\lVert r^+-b\rVert^2\le\tau^2M_s^2\lVert b-w\rVert^2 E ∥ r + − b ∥ 2 ≤ τ 2 M s 2 ∥ b − w ∥ 2 . The identity
E R + = ∥ r ˉ − z ∗ ∥ 2 − 2 τ ⟨ A ( b ) , b − z ∗ ⟩ − ∥ r ˉ − b ∥ 2 + E ∥ r + − b ∥ 2 \mathbb{E}R^+=\lVert\bar r-z^*\rVert^2-2\tau\langle A(b),b-z^*\rangle-\lVert\bar r-b\rVert^2+\mathbb{E}\lVert r^+-b\rVert^2 E R + = ∥ r ˉ − z ∗ ∥ 2 − 2 τ ⟨ A ( b ) , b − z ∗ ⟩ − ∥ r ˉ − b ∥ 2 + E ∥ r + − b ∥ 2
and R + ≤ 2 ∥ b − z ∗ ∥ 2 + 2 ∥ r + − b ∥ 2 R^+\le2\lVert b-z^*\rVert^2+2\lVert r^+-b\rVert^2 R + ≤ 2 ∥ b − z ∗ ∥ 2 + 2 ∥ r + − b ∥ 2 imply
( 1 + t ) E R + ≤ ∥ r ˉ − z ∗ ∥ 2 − ∥ r ˉ − b ∥ 2 + ( 1 + 2 t ) τ 2 M s 2 ∥ b − w ∥ 2 . (1+t)\mathbb{E}R^+\le\lVert\bar r-z^*\rVert^2-\lVert\bar r-b\rVert^2+(1+2t)\tau^2M_s^2\lVert b-w\rVert^2. ( 1 + t ) E R + ≤ ∥ r ˉ − z ∗ ∥ 2 − ∥ r ˉ − b ∥ 2 + ( 1 + 2 t ) τ 2 M s 2 ∥ b − w ∥ 2 .
Mean Lipschitzness and strong monotonicity give α ≤ M s \alpha\le M_s α ≤ M s ; hence p , t ≤ 1 / 4 p,t\le1/4 p , t ≤ 1/4 and τ 2 M s 2 = p / 8 \tau^2M_s^2=p/8 τ 2 M s 2 = p /8 . Expand ∥ r ˉ − z ∗ ∥ 2 = ( 1 − p ) R + p W − p ( 1 − p ) ∥ r − w ∥ 2 \lVert\bar r-z^*\rVert^2=(1-p)R+pW-p(1-p)\lVert r-w\rVert^2 ∥ r ˉ − z ∗ ∥ 2 = ( 1 − p ) R + p W − p ( 1 − p ) ∥ r − w ∥ 2 and use ∥ b − w ∥ 2 ≤ 2 ∥ b − r ˉ ∥ 2 + 2 ( 1 − p ) 2 ∥ r − w ∥ 2 \lVert b-w\rVert^2\le2\lVert b-\bar r\rVert^2+2(1-p)^2\lVert r-w\rVert^2 ∥ b − w ∥ 2 ≤ 2 ∥ b − r ˉ ∥ 2 + 2 ( 1 − p ) 2 ∥ r − w ∥ 2 . Both motion coefficients are nonpositive, so
( 1 + t ) E R + ≤ ( 1 − p ) R + p W . (1+t)\mathbb{E}R^+\le(1-p)R+pW. ( 1 + t ) E R + ≤ ( 1 − p ) R + p W .
Set ω = 2 ( p + t ) / [ ( 2 p + t ) ( 1 − p ) ] \omega=2(p+t)/[(2p+t)(1-p)] ω = 2 ( p + t ) / [( 2 p + t ) ( 1 − p )] , Λ = R + ω W \Lambda=R+\omega W Λ = R + ω W and θ = p t / [ 2 ( p + t ) ] \theta=pt/[2(p+t)] θ = pt / [ 2 ( p + t )] . Since E W + = p E R + + ( 1 − p ) W \mathbb{E}W^+=p\mathbb{E}R^++(1-p)W E W + = p E R + + ( 1 − p ) W , the relative decreases of the R R R and ω W \omega W ω W coefficients are
t ( p + t ) ( 2 p + t ) ( 1 + t ) ≥ θ , p t ( 1 + p + 2 t ) 2 ( p + t ) ( 1 + t ) ≥ θ . \frac{t(p+t)}{(2p+t)(1+t)}\ge\theta,\qquad
\frac{pt(1+p+2t)}{2(p+t)(1+t)}\ge\theta. ( 2 p + t ) ( 1 + t ) t ( p + t ) ≥ θ , 2 ( p + t ) ( 1 + t ) pt ( 1 + p + 2 t ) ≥ θ .
The first inequality is equivalent to t [ 3 p − 2 p 2 + ( 2 − p ) t ] ≥ 0 t[3p-2p^2+(2-p)t]\ge0 t [ 3 p − 2 p 2 + ( 2 − p ) t ] ≥ 0 ; the second is immediate. Thus E Λ + ≤ ( 1 − θ ) Λ \mathbb{E}\Lambda^+\le(1-\theta)\Lambda E Λ + ≤ ( 1 − θ ) Λ . As 1 + ω ≤ 1 + 2 / ( 1 − p ) ≤ 11 / 3 < 4 1+\omega\le1+2/(1-p)\le11/3<4 1 + ω ≤ 1 + 2/ ( 1 − p ) ≤ 11/3 < 4 , iteration proves (109).
For SPDE, use A i = ( ∇ x F i + 2 L ( x − z ) , − ∇ y F i ) A_i=(\nabla_xF_i+2L(x-z),-\nabla_yF_i) A i = ( ∇ x F i + 2 L ( x − z ) , − ∇ y F i ) . Minkowski’s inequality gives RMS constant M s = 3 L M_s=3L M s = 3 L ; proximal geometry gives mean curvature α = μ \alpha=\mu α = μ , and the zero of the operator is u z ∗ u_z^* u z ∗ . Thus θ − 1 = 4 ( n + 6 n κ ) \theta^{-1}=4(n+6\sqrt{n\kappa}) θ − 1 = 4 ( n + 6 nκ ) in (109).
Initialization at the Input Center
For a normalized start, use an initial proximal solve for Q x i n = F + L ∥ x − x i n ∥ 2 Q_{x_{\mathrm{in}}}=F+L\lVert x-x_{\mathrm{in}}\rVert^{2} Q x in = F + L ∥ x − x in ∥ 2 . Its unique saddle u ∗ = ( x ∗ , y ∗ ) u_{*}=(x_{*},y_{*}) u ∗ = ( x ∗ , y ∗ ) exists by (98) using only V ∗ > − ∞ V_{*}>-\infty V ∗ > − ∞ , and ∇ Q x i n ( u ∗ ) = 0 \nabla Q_{x_{\mathrm{in}}}(u_{*})=0 ∇ Q x in ( u ∗ ) = 0 . At u i n u_{\mathrm{in}} u in the regularizer gradient vanishes, so strong monotonicity implies the deterministic bound
∥ u i n − u ∗ ∥ 2 ≤ ∥ ∇ F ( u i n ) ∥ 2 μ 2 ≤ 2 κ 2 Δ L . \lVert u_{\mathrm{in}}-u_{*}\rVert^{2}\leq\frac{\lVert\nabla F(u_{\mathrm{in}})\rVert^{2}}{\mu^{2}}\leq\frac{2\kappa^{2}\Delta}{L}. ∥ u in − u ∗ ∥ 2 ≤ μ 2 ∥ ∇ F ( u in ) ∥ 2 ≤ L 2 κ 2 Δ .
Run (108) on the whole space for
j w = ⌈ 4 ( n + 6 n κ ) log ( 512 κ 2 ) ⌉ , E ∥ u w − u ∗ ∥ 2 ≤ Δ / ( 64 L ) . (110) j_{w}=\left\lceil4(n+6\sqrt{n\kappa})\log(512\kappa^{2})\right\rceil,\qquad\mathbb{E}\lVert u_{w}-u_{*}\rVert^{2}\leq\Delta/(64L). \tag*{(110)} j w = ⌈ 4 ( n + 6 nκ ) log ( 512 κ 2 ) ⌉ , E ∥ u w − u ∗ ∥ 2 ≤ Δ/ ( 64 L ) . (110)
By (109) , the cost including the snapshot is O ( ( n + n κ ) log ( 2 κ ) ) O((n+\sqrt{n\kappa})\log(2\kappa)) O (( n + nκ ) log ( 2 κ )) , with all budgets supplied. Initialize at u w u_{w} u w , with
z 0 = x i n , v 0 = 0. z_{0}=x_{\mathrm{in}},\qquad v_{0}=0. z 0 = x in , v 0 = 0.
Since Q x i n ( u ∗ ) = M ( x i n ) Q_{x_{\mathrm{in}}}(u_{*})=\mathcal{M}(x_{\mathrm{in}}) Q x in ( u ∗ ) = M ( x in ) , joint 3 L 3L 3 L -smoothness and ∇ Q x i n ( u ∗ ) = 0 \nabla Q_{x_{\mathrm{in}}}(u_{*})=0 ∇ Q x in ( u ∗ ) = 0 imply
∣ Q x i n ( u w ) − M ( x i n ) ∣ ≤ ( 3 L / 2 ) ∥ u w − u ∗ ∥ 2 , ∥ ∇ Q x i n ( u w ) ∥ 2 ≤ 9 L 2 ∥ u w − u ∗ ∥ 2 . \lvert Q_{x_{\mathrm{in}}}(u_{w})-\mathcal{M}(x_{\mathrm{in}})\rvert\leq(3L/2)\lVert u_{w}-u_{*}\rVert^{2},\qquad\lVert\nabla Q_{x_{\mathrm{in}}}(u_{w})\rVert^{2}\leq9L^{2}\lVert u_{w}-u_{*}\rVert^{2}. ∣ Q x in ( u w ) − M ( x in )∣ ≤ ( 3 L /2 ) ∥ u w − u ∗ ∥ 2 , ∥ ∇ Q x in ( u w ) ∥ 2 ≤ 9 L 2 ∥ u w − u ∗ ∥ 2 .
Using (103) gives
0 ≤ H z 0 ( u w , 0 ) ≤ 11 L ∥ u w − u ∗ ∥ 2 , E V 0 ≤ Δ + 11 Δ / 64 < 2 Δ . 0\leq H_{z_{0}}(u_{w},0)\leq11L\lVert u_{w}-u_{*}\rVert^{2},\qquad\mathbb{E}V_{0}\leq\Delta+11\Delta/64<2\Delta. 0 ≤ H z 0 ( u w , 0 ) ≤ 11 L ∥ u w − u ∗ ∥ 2 , E V 0 ≤ Δ + 11Δ/64 < 2Δ.
Main-stage batches are fresh after this fixed-budget initialization.
Recursive Gradient Estimation with Full Gradient Refreshes
Write A m = ∑ t < m E D t A_{m}=\sum_{t<m}\mathbb{E}D_{t} A m = ∑ t < m E D t and E ( m ) = ∑ j < 3 m E ∥ η j ∥ 2 E(m)=\sum_{j<3m}\mathbb{E}\lVert\eta_{j}\rVert^{2} E ( m ) = ∑ j < 3 m E ∥ η j ∥ 2 . Let H j \mathcal{H}_{j} H j be the pre-query history, which includes the chosen point, all previous estimates and all past randomness, but excludes the fresh batches at that request. Fixed finite budgets, finite-support component draws and Bernoulli refreshes leave finitely many states at each finite horizon, including initialization, so all conditional expectations below exist.
Full-gradient refreshes and recursive updates. At j ≡ 0 ( m o d q ) j\equiv0\pmod q j ≡ 0 ( mod q ) , use g ^ j = ∇ F ( u j ) \widehat{g}_{j}=\nabla F(u_{j}) g j = ∇ F ( u j ) , at cost n n n . Otherwise, with b b b fresh uniform indices sampled with replacement, use the recursive estimate of Nguyen et al. [16 ] and Fang et al. [6 ] ,
g ^ j = g ^ j − 1 + 1 b ∑ i ∈ I j [ ∇ F i ( u j ) − ∇ F i ( u j − 1 ) ] . (111) \widehat{g}_{j}=\widehat{g}_{j-1}+\frac{1}{b}\sum_{i\in I_{j}}\left[\nabla F_{i}(u_{j})-\nabla F_{i}(u_{j-1})\right]. \tag*{(111)} g j = g j − 1 + b 1 i ∈ I j ∑ [ ∇ F i ( u j ) − ∇ F i ( u j − 1 ) ] . (111)
Although E [ η j ∣ H j ] = η j − 1 \mathbb{E}[\eta_{j}\mid\mathcal{H}_{j}]=\eta_{j-1} E [ η j ∣ H j ] = η j − 1 , the centered new increment and (97) give
E [ ∥ η j ∥ 2 ∣ H j ] ≤ ∥ η j − 1 ∥ 2 + ( L 2 / b ) ∥ u j − u j − 1 ∥ 2 ; \mathbb{E}\left[\lVert\eta_{j}\rVert^{2}\mid\mathcal{H}_{j}\right]\leq\lVert\eta_{j-1}\rVert^{2}+(L^{2}/b)\lVert u_{j}-u_{j-1}\rVert^{2}; E [ ∥ η j ∥ 2 ∣ H j ] ≤ ∥ η j − 1 ∥ 2 + ( L 2 / b ) ∥ u j − u j − 1 ∥ 2 ;
at refreshes η j = 0 \eta_{j}=0 η j = 0 . Summing within epochs counts each motion at most q q q times. Since consecutive iterations share their boundary points, (107) gives E ( m ) ≤ q L b β A m + q b E ( m ) E(m)\leq\frac{qL}{b\beta}A_{m}+\frac{q}{b}E(m) E ( m ) ≤ b β q L A m + b q E ( m ) . Set
A = 16 C / β 2 , q = max { 1 , ⌊ n / A ⌋ } , b = ⌈ A q ⌉ . (112) A=16C/\beta^{2},\qquad q=\max\{1,\lfloor\sqrt{n/A}\rfloor\},\qquad b=\lceil Aq\rceil. \tag*{(112)} A = 16 C / β 2 , q = max { 1 , ⌊ n / A ⌋} , b = ⌈ A q ⌉ . (112)
Since q / b ≤ 1 / A < 1 / 2 q/b \le1/A < 1/2 q / b ≤ 1/ A < 1/2 , the preceding recursion gives E ( m ) ≤ 2 q L A m / ( b β ) E(m) \le2qLAm/(b\beta) E ( m ) ≤ 2 q L A m / ( b β ) and C E ( m ) / ( L β ) ≤ 2 C A m / ( A β 2 ) = A m / 8 CE(m)/(L\beta) \le2CAm/(A\beta^2) = A_m/8 C E ( m ) / ( L β ) ≤ 2 C A m / ( A β 2 ) = A m /8 . When q = 1 q = 1 q = 1 , every request is exact, E ( m ) = 0 E(m) = 0 E ( m ) = 0 , and the difference batch is unused. Telescoping (106) from any initialization with E V 0 ≤ B 0 : = 2 Δ \mathbb{E}V_0 \le B_0 := 2\Delta E V 0 ≤ B 0 := 2Δ gives
E V m + 1 8 A m + β 16 L ∑ t < m E ∥ ∇ M ( z t ) ∥ 2 ≤ B 0 . (113) \mathbb{E}V_m + \frac{1}{8}A_m + \frac{\beta}{16L}\sum_{t<m}\mathbb{E}\lVert\nabla\mathcal{M}(z_t)\rVert^2 \le B_0. \tag*{(113)} E V m + 8 1 A m + 16 L β t < m ∑ E ∥ ∇ M ( z t ) ∥ 2 ≤ B 0 . (113)
Common Budgets, Dissipation, and Output Refinement
For the main stage choose
δ = ε / 16 , B 0 = 2 Δ , T = ⌈ 128 L B 0 β δ 2 ⌉ . (114) \delta= \varepsilon/16,\qquad B_0 = 2\Delta,\qquad T = \left\lceil\frac{128LB_0}{\beta\delta^2}\right\rceil. \tag*{(114)} δ = ε /16 , B 0 = 2Δ , T = ⌈ β δ 2 128 L B 0 ⌉ . (114)
Choose J J J independently and uniformly from { 0 , … , T − 1 } \{0,\ldots,T-1\} { 0 , … , T − 1 } . Then (113) gives
E ∥ ∇ M ( z J ) ∥ 2 ≤ δ 2 / 8 = ε 2 / 2048 , A T ≤ 16 Δ , E [ M ( z J ) − V ∗ ] ≤ 2 Δ . (115) \mathbb{E}\lVert\nabla\mathcal{M}(z_J)\rVert^2 \le\delta^2/8 = \varepsilon^2/2048,\qquad A_T \le16\Delta,\qquad\mathbb{E}[\mathcal{M}(z_J)-V_*] \le2\Delta. \tag*{(115)} E ∥ ∇ M ( z J ) ∥ 2 ≤ δ 2 /8 = ε 2 /2048 , A T ≤ 16Δ , E [ M ( z J ) − V ∗ ] ≤ 2Δ. (115)
The seed for the final proximal solve is u s e e d = ( x J + 1 , y J + 1 ) u_{\mathrm{seed}}=(x_{J+1},y_{J+1}) u seed = ( x J + 1 , y J + 1 ) at center z J z_J z J . The distance terms in (105) already give, since μ ≤ L \mu\le L μ ≤ L and s ≤ 2 L s \le2L s ≤ 2 L ,
D t ≥ 32 μ β ( ∥ X t + ∥ 2 + ∥ Y t + ∥ 2 ) = 32 μ β ∥ u 3 t + 2 − u z t ∗ ∥ 2 . D_t \ge32\mu\beta(\lVert X_t^+\rVert^2+\lVert Y_t^+\rVert^2)=32\mu\beta\lVert u_{3t+2}-u_{z_t}^*\rVert^2. D t ≥ 32 μ β (∥ X t + ∥ 2 + ∥ Y t + ∥ 2 ) = 32 μ β ∥ u 3 t + 2 − u z t ∗ ∥ 2 .
Here X t + , Y t + X_t^+,Y_t^+ X t + , Y t + are measured from the saddle at the center z t z_t z t . As A T / T ≤ β δ 2 / ( 16 L ) A_T/T \le\beta\delta^2/(16L) A T / T ≤ β δ 2 / ( 16 L ) , averaging over J J J yields
E ∥ u s e e d − u z J ∗ ∥ 2 ≤ A T 32 μ β T ≤ κ δ 2 512 L 2 ≤ κ 2 ε 2 L 2 . \mathbb{E}\lVert u_{\mathrm{seed}}-u_{z_J}^*\rVert^2 \le\frac{A_T}{32\mu\beta T}\le\frac{\kappa\delta^2}{512L^2}\le\frac{\kappa^2\varepsilon^2}{L^2}. E ∥ u seed − u z J ∗ ∥ 2 ≤ 32 μ β T A T ≤ 512 L 2 κ δ 2 ≤ L 2 κ 2 ε 2 .
Use fresh solver randomness at this center and seed, with the fixed budget
j p = ⌈ 4 ( n + 6 n κ ) log [ 1024 κ 2 ( 1 + κ ) 2 ] ⌉ , E ∥ u ^ − u z J ∗ ∥ 2 ≤ ε 2 256 L 2 ( 1 + κ ) 2 . (116) j_p=\left\lceil4(n+6\sqrt{n\kappa})\log[1024\kappa^2(1+\kappa)^2]\right\rceil,\qquad\mathbb{E}\lVert\widehat{u}-u_{z_J}^*\rVert^2\le\frac{\varepsilon^2}{256L^2(1+\kappa)^2}. \tag*{(116)} j p = ⌈ 4 ( n + 6 nκ ) log [ 1024 κ 2 ( 1 + κ ) 2 ] ⌉ , E ∥ u − u z J ∗ ∥ 2 ≤ 256 L 2 ( 1 + κ ) 2 ε 2 . (116)
Indeed, condition on the complete history of the main stage and J J J . Equation (109) bounds the conditional mean-square distance by 4 e − θ j p ∥ u s e e d − u z J ∗ ∥ 2 4e^{-\theta j_p}\lVert u_{\mathrm{seed}}-u_{z_J}^*\rVert^2 4 e − θ j p ∥ u seed − u z J ∗ ∥ 2 ; then use 4 e − θ j p ≤ [ 256 κ 2 ( 1 + κ ) 2 ] − 1 4e^{-\theta j_p}\le[256\kappa^2(1+\kappa)^2]^{-1} 4 e − θ j p ≤ [ 256 κ 2 ( 1 + κ ) 2 ] − 1 and average. This solve includes its own n n n -call snapshot.
Return the primal coordinate x ^ \widehat{x} x of u ^ \widehat{u} u . The geometry above gives ∇ V ( x z J ∗ ) = ∇ M ( z J ) \nabla V(x_{z_J}^*)=\nabla\mathcal{M}(z_J) ∇ V ( x z J ∗ ) = ∇ M ( z J ) and Lip ( ∇ V ) ≤ L ( 1 + κ ) \operatorname{Lip}(\nabla V)\le L(1+\kappa) Lip ( ∇ V ) ≤ L ( 1 + κ ) , so pathwise
∥ ∇ V ( x ^ ) ∥ ≤ ∥ ∇ M ( z J ) ∥ + L ( 1 + κ ) ∥ u ^ − u z J ∗ ∥ . \lVert\nabla V(\widehat{x})\rVert\le\lVert\nabla\mathcal{M}(z_J)\rVert+L(1+\kappa)\lVert\widehat{u}-u_{z_J}^*\rVert. ∥ ∇ V ( x )∥ ≤ ∥ ∇ M ( z J )∥ + L ( 1 + κ ) ∥ u − u z J ∗ ∥ .
Squaring, taking expectations, and using (115) and (116) proves
E ∥ ∇ V ( x ^ ) ∥ 2 ≤ ( 1 / 1024 + 1 / 128 ) ε 2 ≤ ε 2 . \mathbb{E}\lVert\nabla V(\widehat{x})\rVert^2\le(1/1024+1/128)\varepsilon^2\le\varepsilon^2. E ∥ ∇ V ( x ) ∥ 2 ≤ ( 1/1024 + 1/128 ) ε 2 ≤ ε 2 .
Oracle Costs and Completion of the Upper Bounds
With integer rounding, a recursive estimator with refresh cost M 0 ≥ 1 M_0 \ge1 M 0 ≥ 1 , cost two per paired difference, ratio parameter A ≥ 1 A \ge1 A ≥ 1 , q = max { 1 , ⌊ M 0 / A ⌋ } q = \max\{1, \lfloor\sqrt{M_0/A}\rfloor\} q = max { 1 , ⌊ M 0 / A ⌋} , and b = ⌈ A q ⌉ b = \lceil Aq\rceil b = ⌈ A q ⌉ has N r = ⌈ 3 T / q ⌉ N_r = \lceil3T/q\rceil N r = ⌈ 3 T / q ⌉ refreshes and cost
M 0 N r + 2 b ( 3 T − N r ) ≤ M 0 + 18 T min { M 0 , A M 0 } . (117) M_0N_r + 2b(3T-N_r) \le M_0 + 18T\min\{M_0,\sqrt{AM_0}\}. \tag*{(117)} M 0 N r + 2 b ( 3 T − N r ) ≤ M 0 + 18 T min { M 0 , A M 0 } . (117)
For q ≥ 2 q \ge2 q ≥ 2 , use q ≥ 1 2 M 0 / A q \ge\frac{1}{2}\sqrt{M_0/A} q ≥ 2 1 M 0 / A , b ≤ 2 A M 0 b \le2\sqrt{AM_0} b ≤ 2 A M 0 and A M 0 ≤ M 0 / 2 \sqrt{AM_0} \le M_0/2 A M 0 ≤ M 0 /2 . For q = 1 q = 1 q = 1 , the cost is 3 T M 0 3TM_0 3 T M 0 , while M 0 < 4 A M_0 < 4A M 0 < 4 A implies min { M 0 , A M 0 } ≥ M 0 / 2 \min\{M_0,\sqrt{AM_0}\} \ge M_0/2 min { M 0 , A M 0 } ≥ M 0 /2 .
Taking M 0 = n M_0 = n M 0 = n and A = 16 C / β 2 A = 16C/\beta^2 A = 16 C / β 2 in (117) bounds the main cost by n + 18 T min { n , A n } n+18T\min\{n,\sqrt{An}\} n + 18 T min { n , A n } . Since β − 1 = O ( κ ) \beta^{-1} = O(\sqrt{\kappa}) β − 1 = O ( κ ) , T = O ( κ S ) T = O(\sqrt{\kappa}S) T = O ( κ S ) and A = O ( κ ) A = O(\kappa) A = O ( κ ) , this is O ( n + n κ S ) O(n+\sqrt{n\kappa}S) O ( n + nκ S ) ; the ceiling contributes at most another O ( n ) O(n) O ( n ) term. Adding the initial proximal solve and the final refinement proves Theorem 22 . The bound holds for every κ ≥ 1 \kappa\ge1 κ ≥ 1 because min { n , A n } ≤ A n \min\{n,\sqrt{An}\} \le\sqrt{An} min { n , A n } ≤ A n ; S ≥ 2 S \ge2 S ≥ 2 absorbs the ceiling.
For the explicit coefficient, use S = L Δ / ε 2 S = L\Delta/\varepsilon^2 S = L Δ/ ε 2 and β − 1 = ( 3 ⋅ 2 18 / 2 ) κ \beta^{-1} = (3\cdot2^{18}/\sqrt{2})\sqrt{\kappa} β − 1 = ( 3 ⋅ 2 18 / 2 ) κ :
β − 1 < 2 20 κ , A < 2 52 κ , T = ⌈ 2 16 S / β ⌉ < 2 37 κ S . \beta^{-1} < 2^{20}\sqrt{\kappa}, \qquad A < 2^{52}\kappa, \qquad T = \lceil2^{16}S/\beta\rceil< 2^{37}\sqrt{\kappa}S. β − 1 < 2 20 κ , A < 2 52 κ , T = ⌈ 2 16 S / β ⌉ < 2 37 κ S .
Thus the cost of the main stage is at most n + 2 68 n κ S n+2^{68}\sqrt{n\kappa}S n + 2 68 nκ S . Each solver iteration costs at most 5 / 2 5/2 5/2 in expectation, including refreshes. Since
log ( 512 κ 2 ) ≤ 9 log ( 2 κ ) , log [ 1024 κ 2 ( 1 + κ ) 2 ] ≤ 12 log ( 2 κ ) , \log(512\kappa^2) \le9\log(2\kappa), \qquad\log[1024\kappa^2(1+\kappa)^2] \le12\log(2\kappa), log ( 512 κ 2 ) ≤ 9 log ( 2 κ ) , log [ 1024 κ 2 ( 1 + κ ) 2 ] ≤ 12 log ( 2 κ ) ,
the warm and final solves together cost at most 2 n + 2 10 ( n + 6 n κ ) log ( 2 κ ) + 5 ≤ 2 12 ( n + n κ ) log ( 2 κ ) 2n+2^{10}(n+6\sqrt{n\kappa})\log(2\kappa)+5 \le2^{12}(n+\sqrt{n\kappa})\log(2\kappa) 2 n + 2 10 ( n + 6 nκ ) log ( 2 κ ) + 5 ≤ 2 12 ( n + nκ ) log ( 2 κ ) , including both initial snapshots and both iteration ceilings. Consequently C N = 2 72 C_N=2^{72} C N = 2 72 is valid in Theorem 22 .
Assembly over All Condition Numbers
Write h ( n , κ ) = min { n κ , n 3 / 4 κ } h(n,\kappa)=\min\{\sqrt{n\kappa},n^{3/4}\sqrt{\kappa}\} h ( n , κ ) = min { nκ , n 3/4 κ } . For 1 ≤ κ ≤ n 1\le\kappa\le\sqrt{n} 1 ≤ κ ≤ n , h = n κ h=\sqrt{n\kappa} h = nκ . For n ≤ κ ≤ 2 18 n \sqrt{n}\le\kappa\le2^{18}\sqrt{n} n ≤ κ ≤ 2 18 n ,
n κ h ( n , κ ) = κ n ≤ 2 9 . \frac{\sqrt{n\kappa}}{h(n,\kappa)}=\sqrt{\frac{\kappa}{\sqrt{n}}}\le2^9. h ( n , κ ) nκ = n κ ≤ 2 9 .
Use Theorem 22 throughout 1 ≤ κ ≤ 2 18 n 1\le\kappa\le2^{18}\sqrt{n} 1 ≤ κ ≤ 2 18 n . As log 2 ( 2 + κ ) ≥ 1 \log^2(2+\kappa)\ge1 log 2 ( 2 + κ ) ≥ 1 and log ( 2 κ ) ≤ 2 log 2 ( 2 + κ ) \log(2\kappa)\le2\log^2(2+\kappa) log ( 2 κ ) ≤ 2 log 2 ( 2 + κ ) , its cost is at most 2 83 [ n + h ( n , κ ) ( 1 + S ) ] log 2 ( 2 + κ ) 2^{83}[n+h(n,\kappa)(1+S)]\log^2(2+\kappa) 2 83 [ n + h ( n , κ ) ( 1 + S )] log 2 ( 2 + κ ) . For κ > 2 18 n \kappa>2^{18}\sqrt{n} κ > 2 18 n , Appendix F proves the upper bound of Theorem 3 for every S > 0 S>0 S > 0 , with schedules that use only n n n , κ \kappa κ , S S S and T o u t = ⌈ 60 S ⌉ T_{\mathrm{out}}=\lceil60S\rceil T out = ⌈ 60 S ⌉ . Taking C = 2 83 + C H C=2^{83}+C_H C = 2 83 + C H , where C H C_H C H is the universal constant in (15), proves Theorem 2 . Both branches terminate almost surely with finite expected cost, and their output satisfies the expected squared-gradient criterion. For a deterministic cap with success probability 2 / 3 2/3 2/3 , use target ε 2 / 6 \varepsilon^2/6 ε 2 /6 , cap at six times the resulting bound on the expected cost, and apply Markov’s inequality separately to cost and squared gradient norm. The two failure probabilities are at most 1 / 6 1/6 1/6 each.
Matching Upper Bound Including All Initialization Costs
We combine the shared L-SVRE solver with Lin et al. [9 ] and Zhang et al. [25 ] to prove (14)–(15). Algorithm 2 lists the steps; Appendices F.2–F.4 analyze them.
Algorithm 2 Method of Theorem 3 for large condition numbers
Require: f i = F i / L f_i = F_i/L f i = F i / L , ν = μ / L \nu= \mu/L ν = μ / L , D 0 = Δ / L D_0 = \Delta/L D 0 = Δ/ L , e 0 = ϵ / L e_0 = \epsilon/L e 0 = ϵ / L ; the parameters (20) and β = ( 1 − q ) / ( 1 + q ) \beta= (1-\sqrt{q})/(1+\sqrt{q}) β = ( 1 − q ) / ( 1 + q ) ; J J J of (122); K K K and α \alpha α of Appendix F.3
y 0 ← 0 y_0 \leftarrow0 y 0 ← 0 , z 1 ← 0 z_1 \leftarrow0 z 1 ← 0
for k = 1 , … , J k = 1,\ldots,J k = 1 , … , J do ▷ dual warm start, Appendix F.2
from y = 0 y = 0 y = 0 , run certified L-SVRE epochs (Lemma 25) on g k g_k g k of (120) until ∥ ∇ g k ( y k ) ∥ 2 ≤ 2 s η k \lVert\nabla g_k(y_k)\rVert^2 \leq2s\eta_k ∥ ∇ g k ( y k ) ∥ 2 ≤ 2 s η k
z k + 1 ← y k + β ( y k − y k − 1 ) z_{k+1} \leftarrow y_k + \beta(y_k-y_{k-1}) z k + 1 ← y k + β ( y k − y k − 1 )
end for
w 1 = ( p 1 , y 1 ) ← ( 0 , y J ) w_1 = (p_1,y_1) \leftarrow(0,y_J) w 1 = ( p 1 , y 1 ) ← ( 0 , y J )
for t = 1 , … , T o u t = ⌈ 60 D 0 / e 0 2 ⌉ t = 1,\ldots,T_{\mathrm{out}} = \left\lceil60D_0/e_0^2\right\rceil t = 1 , … , T out = ⌈ 60 D 0 / e 0 2 ⌉ do ▷ outer steps, Appendices F.3–F.4
compute R t 2 = ∥ ∇ f ^ t ( w t ) ∥ 2 R_t^2 = \lVert\nabla\widehat{f}_t(w_t)\rVert^2 R t 2 = ∥ ∇ f t ( w t ) ∥ 2 with n n n calls; if R t = 0 R_t = 0 R t = 0 then return p t p_t p t
B t ← ν R t 2 / 72 B_t \leftarrow\nu R_t^2/72 B t ← ν R t 2 /72 , w ~ 0 = ( p ~ 0 , y ~ 0 ) ← w t \widetilde{w}_0 = (\widetilde{p}_0,\widetilde{y}_0) \leftarrow w_t w 0 = ( p 0 , y 0 ) ← w t , z 1 ← y t z_1 \leftarrow y_t z 1 ← y t
for k = 1 , … , K k = 1,\ldots,K k = 1 , … , K do
from w ~ k − 1 \widetilde{w}_{k-1} w k − 1 , run certified L-SVRE epochs on f ^ t , k \widehat{f}_{t,k} f t , k of (124) until ∥ ∇ f ^ t , k ( w ~ k ) ∥ 2 ≤ δ t , k \lVert\nabla\widehat{f}_{t,k}(\widetilde{w}_k)\rVert^2 \leq\delta_{t,k} ∥ ∇ f t , k ( w k ) ∥ 2 ≤ δ t , k
z k + 1 ← y ~ k + β ( y ~ k − y ~ k − 1 ) z_{k+1} \leftarrow\widetilde{y}_k + \beta(\widetilde{y}_k-\widetilde{y}_{k-1}) z k + 1 ← y k + β ( y k − y k − 1 )
end for
w t + 1 ← w ~ K w_{t+1} \leftarrow\widetilde{w}_K w t + 1 ← w K ▷ ∥ ∇ f ^ t ( w ~ K ) ∥ 2 ≤ α R t 2 \lVert\nabla\widehat{f}_t(\widetilde{w}_K)\rVert^2 \leq\alpha R_t^2 ∥ ∇ f t ( w K ) ∥ 2 ≤ α R t 2 by (125)
end for
return p t + 1 p_{t+1} p t + 1 for t t t drawn uniformly from { 1 , … , T o u t } \{1,\ldots,T_{\mathrm{out}}\} { 1 , … , T out }
Residual Certification with the Shared Solver
Lemma 25 (Certified L-SVRE epochs). Under Lemma 24, restart from r = w = u r = w = u r = w = u for j 0 = ⌈ θ − 1 log 8 ⌉ j_0 = \lceil\theta^{-1}\log8\rceil j 0 = ⌈ θ − 1 log 8 ⌉ updates per epoch, paying n n n calls to test ∥ A ( u + ) ∥ 2 ≤ δ \lVert A(u^+)\rVert^2 \leq\delta ∥ A ( u + ) ∥ 2 ≤ δ afterward. If the test fails, restart from u + u^+ u + . For δ > 0 \delta> 0 δ > 0 , this terminates almost surely with a certified output. Writing H = M s 2 ∥ u 0 − z ∗ ∥ 2 / δ H = M_s^2\lVert u_0-z^*\rVert^2/\delta H = M s 2 ∥ u 0 − z ∗ ∥ 2 / δ and log 2 + 0 : = 0 \log_2^+ 0 := 0 log 2 + 0 := 0 , the cost Q c e r t Q_{\mathrm{cert}} Q cert from the given start satisfies
E Q c e r t ≤ [ 2 n + 5 2 ⌈ θ − 1 log 8 ⌉ ] [ 2 + ⌈ log 2 + H ⌉ ] = O ( ( n + n M s / α ) [ 1 + log + H ] ) . (118) \mathbb{E}Q_{\mathrm{cert}} \leq\left[2n+\frac{5}{2}\left\lceil\theta^{-1}\log8\right\rceil\right]\left[2+\left\lceil\log_2^+ H\right\rceil\right]
= O\left((n+\sqrt{n}M_s/\alpha)[1+\log^+ H]\right). \tag*{(118)} E Q cert ≤ [ 2 n + 2 5 ⌈ θ − 1 log 8 ⌉ ] [ 2 + ⌈ log 2 + H ⌉ ] = O ( ( n + n M s / α ) [ 1 + log + H ] ) . (118)
Proof. Each epoch halves conditional mean-squared distance by (109). Continue virtually after acceptance, and let N e p N_{\mathrm{ep}} N ep be the number of epochs. If N e p > r N_{\mathrm{ep}}>r N ep > r , the r r r th virtual output fails its residual test; mean Lipschitzness and Markov’s inequality therefore give P ( N e p > r ) ≤ min { 1 , H 2 − r } \mathbb{P}(N_{\mathrm{ep}}>r)\leq\min\{1,H2^{-r}\} P ( N ep > r ) ≤ min { 1 , H 2 − r } for r ≥ 1 r\geq1 r ≥ 1 . Summation gives E N e p ≤ 2 + ⌈ log 2 + H ⌉ \mathbb{E}N_{\mathrm{ep}}\leq2+\lceil\log_2^+H\rceil E N ep ≤ 2 + ⌈ log 2 + H ⌉ and almost-sure termination. Each epoch costs at most C e p = 2 n + ( 5 / 2 ) j 0 C_{\mathrm{ep}}=2n+(5/2)j_0 C ep = 2 n + ( 5/2 ) j 0 in conditional expectation, including its initial snapshot, random refreshes, and final test. The epoch-start indicator is measurable before that epoch. Multiply its uniform conditional cost bound by this indicator and sum by the tower property to obtain E Q c e r t ≤ C e p E N e p \mathbb{E}Q_{\mathrm{cert}}\leq C_{\mathrm{ep}}\mathbb{E}N_{\mathrm{ep}} E Q cert ≤ C ep E N ep .
Both applications below use α = s = n − 1 / 2 \alpha=s=n^{-1/2} α = s = n − 1/2 and M s ≤ 3 M_s\leq3 M s ≤ 3 , so their uniform conditional epoch charge is at most
C e p H : = 2 n + 5 2 ⌈ 28 n log 8 ⌉ . (119) C_{\mathrm{ep}}^H := 2n+\frac{5}{2}\lceil28n\log8\rceil. \tag*{(119)} C ep H := 2 n + 2 5 ⌈ 28 n log 8 ⌉ . (119)
Initialization Cost and a Pathwise Certificate
We use the normalization and parameters of Section 4, and write f = F / L f=F/L f = F / L and V f ( p ) = max y f ( p , y ) = V ( p ) / L V_f(p)=\max_y f(p,y)=V(p)/L V f ( p ) = max y f ( p , y ) = V ( p ) / L . For g ( y ) = − f ( 0 , y ) g(y)=-f(0,y) g ( y ) = − f ( 0 , y ) let g ∗ = min y g ( y ) g^*=\min_y g(y) g ∗ = min y g ( y ) and B = D 0 / ν B=D_0/\nu B = D 0 / ν . Mean strong convexity and (5) imply
g ( 0 ) − g ∗ ≤ ∥ ∇ g ( 0 ) ∥ 2 2 ν ≤ B , ∥ y ∗ ( 0 ) ∥ 2 ≤ 2 B ν . g(0)-g^* \le\frac{\|\nabla g(0)\|^2}{2\nu} \le B,\qquad\|y^*(0)\|^2 \le\frac{2B}{\nu}. g ( 0 ) − g ∗ ≤ 2 ν ∥∇ g ( 0 ) ∥ 2 ≤ B , ∥ y ∗ ( 0 ) ∥ 2 ≤ ν 2 B .
Starting at y 0 = z 1 = 0 y_0=z_1=0 y 0 = z 1 = 0 , set
g k ( y ) = g ( y ) + τ 2 ∥ y − z k ∥ 2 , η k = 2 9 B ( 1 − ρ ) k , g_k(y)=g(y)+\frac{\tau}{2}\|y-z_k\|^2,\qquad\eta_k=\frac{2}{9}B(1-\rho)^k, g k ( y ) = g ( y ) + 2 τ ∥ y − z k ∥ 2 , η k = 9 2 B ( 1 − ρ ) k ,
{#eq:120}
z_{k+1}=y_k+\beta(y_k-y_{k-1}),\qquad\beta=\frac{1-\sqrt{q}}{1+\sqrt{q}}.
For each k k k , solve from y = 0 y=0 y = 0 using Lemma 25 until the computable condition ∥ ∇ g k ( y k ) ∥ 2 ≤ 2 s η k \|\nabla g_k(y_k)\|^2\le2s\eta_k ∥∇ g k ( y k ) ∥ 2 ≤ 2 s η k holds. Each subproblem has strong convexity s s s and RMS smoothness constant at most 1 + τ ≤ 2 1+\tau\le2 1 + τ ≤ 2 , so this certifies g k ( y k ) − min g k ≤ η k g_k(y_k)-\min g_k\le\eta_k g k ( y k ) − min g k ≤ η k .
The Catalyst estimate for strongly convex objectives (Lin et al., [9 ] ) states that the recurrence (120), with these subproblem errors and 0 < ρ < q 0<\rho<\sqrt{q} 0 < ρ < q , satisfies
g ( y k ) − g ∗ ≤ 8 B ( q − ρ ) 2 ( 1 − ρ ) k + 1 ≤ 800 B q ( 1 − ρ ) k . (120) g(y_k)-g^* \le\frac{8B}{(\sqrt{q}-\rho)^2}(1-\rho)^{k+1}\le\frac{800B}{q}(1-\rho)^k. \tag*{(120)} g ( y k ) − g ∗ ≤ ( q − ρ ) 2 8 B ( 1 − ρ ) k + 1 ≤ q 800 B ( 1 − ρ ) k . (120)
The error sum in the proof of Proposition 5 is monotone in the initial error and potential, bounded here by B B B and 2 ( 1 − q ) B 2(1-\sqrt{q})B 2 ( 1 − q ) B ; prescribing all errors with the same B B B therefore permits this supplied upper bound in place of the unknown initial gap.
Choose the deterministic number of accepted subproblems
J = ⌈ log ( 1600 / ( q ν 2 ) ) − log ( 1 − ρ ) ⌉ . (121) J=\left\lceil\frac{\log(1600/(q\nu^2))}{-\log(1-\rho)}\right\rceil. \tag*{(121)} J = ⌈ − log ( 1 − ρ ) log ( 1600/ ( q ν 2 )) ⌉ . (121)
Then (120) gives g ( y J ) − g ∗ ≤ ν D 0 / 2 g(y_J)-g^*\le\nu D_0/2 g ( y J ) − g ∗ ≤ ν D 0 /2 , so ∥ y J − y ∗ ( 0 ) ∥ 2 ≤ D 0 \|y_J-y^*(0)\|^2\le D_0 ∥ y J − y ∗ ( 0 ) ∥ 2 ≤ D 0 on every terminating run. Each certified subproblem terminates almost surely; the finite composition therefore yields the pathwise certificate (21) for the warm start y w = y J y_w=y_J y w = y J .
To charge the cold starts, (120) and strong convexity bound ∥ y k − y ∗ ( 0 ) ∥ 2 ≤ 1600 B / ( q ν ) \|y_k-y^*(0)\|^2\le1600B/(q\nu) ∥ y k − y ∗ ( 0 ) ∥ 2 ≤ 1600 B / ( q ν ) for all accepted k ≤ J k\le J k ≤ J . Since 0 ≤ β ≤ 1 0\le\beta\le1 0 ≤ β ≤ 1 , ∥ z k − y ∗ ( 0 ) ∥ 2 ≤ 14400 B / ( q ν ) \|z_k-y^*(0)\|^2\le14400B/(q\nu) ∥ z k − y ∗ ( 0 ) ∥ 2 ≤ 14400 B / ( q ν ) . If y k ∗ = a r g m i n g k y_k^*=\mathop{\rm argmin} g_k y k ∗ = argmin g k , contraction of its proximal map under strong convexity gives ∥ y k ∗ − y ∗ ( 0 ) ∥ ≤ τ 1 + τ ∥ z k − y ∗ ( 0 ) ∥ ≤ ∥ z k − y ∗ ( 0 ) ∥ \|y_k^*-y^*(0)\|\le\frac{\tau}{1+\tau}\|z_k-y^*(0)\|\le\|z_k-y^*(0)\| ∥ y k ∗ − y ∗ ( 0 ) ∥ ≤ 1 + τ τ ∥ z k − y ∗ ( 0 ) ∥ ≤ ∥ z k − y ∗ ( 0 ) ∥ . Thus ∥ 0 − y k ∗ ∥ 2 ≤ C B / ( q ν ) \|0-y_k^*\|^2\le CB/(q\nu) ∥0 − y k ∗ ∥ 2 ≤ C B / ( q ν ) . The threshold is 2 s η k = ( 4 / 9 ) s B ( 1 − ρ ) k 2s\eta_k=(4/9)sB(1-\rho)^k 2 s η k = ( 4/9 ) s B ( 1 − ρ ) k . Because q s = ν qs=\nu q s = ν , the ratio in (118) is at most H k ≤ C ν − 2 ( 1 − ρ ) − k ≤ C κ 5 H_k\le C\nu^{-2}(1-\rho)^{-k}\le C\kappa^5 H k ≤ C ν − 2 ( 1 − ρ ) − k ≤ C κ 5 . The last bound follows from (121) , q ≥ ν q\ge\nu q ≥ ν , and 1 − ρ ≥ 0.1 1-\rho\ge0.1 1 − ρ ≥ 0.1 . Each subproblem costs O ( n log ( 2 + κ ) ) O(n\log(2+\kappa)) O ( n log ( 2 + κ )) expected calls, uniformly conditional on all previous accepted outputs. Since J = O ( q − 1 / 2 log ( 2 + κ ) ) J=O(q^{-1/2}\log(2+\kappa)) J = O ( q − 1/2 log ( 2 + κ )) , startup costs
O ( n q − 1 / 2 log 2 ( 2 + κ ) ) . (122) O\bigl(nq^{-1/2}\log^2(2+\kappa)\bigr). \tag*{(122)} O ( n q − 1/2 log 2 ( 2 + κ ) ) . (122)
Auxiliary Saddle Solves with Computable Tolerances
At outer iterate w t = ( p t , y t ) w_t=(p_t,y_t) w t = ( p t , y t ) let f ^ t ( p , y ) = f ( p , y ) + ∥ p − p t ∥ 2 \widehat f_t(p,y)=f(p,y)+\|p-p_t\|^2 f t ( p , y ) = f ( p , y ) + ∥ p − p t ∥ 2 and R t 2 = ∥ ∇ f ^ t ( w t ) ∥ 2 R_t^2=\|\nabla\widehat f_t(w_t)\|^2 R t 2 = ∥∇ f t ( w t ) ∥ 2 . The mean is ( 1 , ν ) (1,\nu) ( 1 , ν ) -strongly convex–strongly concave and has joint smoothness at most three. Let w t ∗ w_t^* w t ∗ be its saddle point and G t G_t G t its saddle gap at w t w_t w t . Then
G t ≥ ν 2 ∥ w t − w t ∗ ∥ 2 ≥ ν R t 2 18 , G t ≤ R t 2 2 ν . G_t\ge\frac{\nu}{2}\|w_t-w_t^*\|^2\ge\frac{\nu R_t^2}{18},\qquad G_t\le\frac{R_t^2}{2\nu}. G t ≥ 2 ν ∥ w t − w t ∗ ∥ 2 ≥ 18 ν R t 2 , G t ≤ 2 ν R t 2 .
The first uses strong curvature and R t ≤ 3 ∥ w t − w t ∗ ∥ R_t \le3\lVert w_t-w_t^*\rVert R t ≤ 3 ∥ w t − w t ∗ ∥ ; the second applies strong convexity/concavity separately to the two gap terms. Thus the computable B t = ν R t 2 / 72 B_t=\nu R_t^2/72 B t = ν R t 2 /72 satisfies (22) . If R t = 0 R_t=0 R t = 0 , uniqueness of the dual maximizer gives y t = y ∗ ( p t ) y_t=y^*(p_t) y t = y ∗ ( p t ) and ∇ V f ( p t ) = 0 \nabla V_f(p_t)=0 ∇ V f ( p t ) = 0 .
Otherwise start the dual acceleration at w ~ 0 = ( p ~ 0 , y ~ 0 ) = w t \widetilde{w}_0=(\widetilde{p}_0,\widetilde{y}_0)=w_t w 0 = ( p 0 , y 0 ) = w t , z 1 = y t z_1=y_t z 1 = y t , write w ~ k = ( p ~ k , y ~ k ) \widetilde{w}_k=(\widetilde{p}_k,\widetilde{y}_k) w k = ( p k , y k ) for its iterates, and, with the same β \beta β as in (120), form
f ^ t , k ( p , y ) = f ^ t ( p , y ) − τ 2 ∥ y − z k ∥ 2 , δ t , k = ν 2 ( 1 − ρ ) k B t , z k + 1 = y ~ k + β ( y ~ k − y ~ k − 1 ) . (123) \begin{aligned}
\widehat{f}_{t,k}(p,y)&=\widehat{f}_t(p,y)-\frac{\tau}{2}\lVert y-z_k\rVert^2,\\
\delta_{t,k}&=\frac{\nu}{\sqrt{2}}(1-\rho)^kB_t,\\
z_{k+1}&=\widetilde{y}_k+\beta(\widetilde{y}_k-\widetilde{y}_{k-1}). \tag*{(123)}
\end{aligned} f t , k ( p , y ) δ t , k z k + 1 = f t ( p , y ) − 2 τ ∥ y − z k ∥ 2 , = 2 ν ( 1 − ρ ) k B t , = y k + β ( y k − y k − 1 ) . (123)
Solve f ^ t , k \widehat{f}_{t,k} f t , k from the preceding accepted w ~ k − 1 \widetilde{w}_{k-1} w k − 1 using Lemma 25, until ∥ ∇ f ^ t , k ( w ~ k ) ∥ 2 ≤ δ t , k \lVert\nabla\widehat{f}_{t,k}(\widetilde{w}_k)\rVert^2\le\delta_{t,k} ∥ ∇ f t , k ( w k ) ∥ 2 ≤ δ t , k . Its mean strong monotonicity is s s s and its RMS smoothness constant is at most three, because the regularizer Hessian has norm at most two and the RMS smoothness constant of f f f is one.
Since B t ≤ G t B_t\le G_t B t ≤ G t , our accepted residual thresholds are no larger than ( ν / 2 ) ( 1 − ρ ) k G t (\nu/\sqrt{2})(1-\rho)^kG_t ( ν / 2 ) ( 1 − ρ ) k G t . The residual estimate of Zhang et al. [25 ] therefore gives, pathwise for every accepted finite prefix,
∥ ∇ f ^ t ( w ~ k ) ∥ 2 ≤ C R ( 1 − ρ ) k R t 2 , C R = 5508 ν 2 ( q − ρ ) 2 + 18 2 ν . (124) \lVert\nabla\widehat{f}_t(\widetilde{w}_k)\rVert^2\le C_R(1-\rho)^kR_t^2,\qquad C_R=\frac{5508}{\nu^2(\sqrt{q}-\rho)^2}+\frac{18\sqrt{2}}{\nu}. \tag*{(124)} ∥ ∇ f t ( w k ) ∥ 2 ≤ C R ( 1 − ρ ) k R t 2 , C R = ν 2 ( q − ρ ) 2 5508 + ν 18 2 . (124)
The same residual estimate directly controls the next solver’s starting distance. Write a = 1 − ρ a=1-\rho a = 1 − ρ and A = ( ∇ p f ^ t , − ∇ y f ^ t ) A=(\nabla_p\widehat{f}_t,-\nabla_y\widehat{f}_t) A = ( ∇ p f t , − ∇ y f t ) . Its ν \nu ν -strong monotonicity gives
d j : = ∥ w ~ j − w t ∗ ∥ ≤ ν − 1 C R a j / 2 R t ( j ≥ 0 ) , d_j:=\lVert\widetilde{w}_j-w_t^*\rVert\le\nu^{-1}\sqrt{C_R}a^{j/2}R_t\qquad(j\ge0), d j := ∥ w j − w t ∗ ∥ ≤ ν − 1 C R a j /2 R t ( j ≥ 0 ) ,
where j = 0 j=0 j = 0 follows from ∥ A ( w ~ 0 ) ∥ = R t \lVert A(\widetilde{w}_0)\rVert=R_t ∥ A ( w 0 )∥ = R t and C R ≥ 1 C_R\ge1 C R ≥ 1 . Writing w t ∗ = ( p t ∗ , y t ∗ ) w_t^*=(p_t^*,y_t^*) w t ∗ = ( p t ∗ , y t ∗ ) , the auxiliary operator A k = A + ( 0 , τ ( y − z k ) ) A_k=A+(0,\tau(y-z_k)) A k = A + ( 0 , τ ( y − z k )) is s s s -strongly monotone, so
∥ w t , k ∗ − w t ∗ ∥ ≤ ( τ / s ) ∥ z k − y t ∗ ∥ ≤ ∥ z k − y t ∗ ∥ . \lVert w_{t,k}^*-w_t^*\rVert\le(\tau/s)\lVert z_k-y_t^*\rVert\le\lVert z_k-y_t^*\rVert. ∥ w t , k ∗ − w t ∗ ∥ ≤ ( τ / s ) ∥ z k − y t ∗ ∥ ≤ ∥ z k − y t ∗ ∥ .
For k ≥ 2 k\ge2 k ≥ 2 , z k = y ~ k − 1 + β ( y ~ k − 1 − y ~ k − 2 ) z_k=\widetilde{y}_{k-1}+\beta(\widetilde{y}_{k-1}-\widetilde{y}_{k-2}) z k = y k − 1 + β ( y k − 1 − y k − 2 ) and 0 ≤ β ≤ 1 0\le\beta\le1 0 ≤ β ≤ 1 consequently give ∥ w ~ k − 1 − w t , k ∗ ∥ ≤ 3 d k − 1 + d k − 2 \lVert\widetilde{w}_{k-1}-w_{t,k}^*\rVert\le3d_{k-1}+d_{k-2} ∥ w k − 1 − w t , k ∗ ∥ ≤ 3 d k − 1 + d k − 2 . For k = 1 k=1 k = 1 , z 1 = y ~ 0 z_1=\widetilde{y}_0 z 1 = y 0 gives A 1 ( w ~ 0 ) = A ( w ~ 0 ) A_1(\widetilde{w}_0)=A(\widetilde{w}_0) A 1 ( w 0 ) = A ( w 0 ) . Thus
∥ w ~ k − 1 − w t , k ∗ ∥ 2 ≤ { R t 2 / s 2 , k = 1 , 16 C R ν − 2 a k − 2 R t 2 , k ≥ 2. \lVert\widetilde{w}_{k-1}-w_{t,k}^*\rVert^2\le
\begin{cases}
R_t^2/s^2, & k=1,\\
16C_R\nu^{-2}a^{k-2}R_t^2, & k\ge2.
\end{cases} ∥ w k − 1 − w t , k ∗ ∥ 2 ≤ { R t 2 / s 2 , 16 C R ν − 2 a k − 2 R t 2 , k = 1 , k ≥ 2.
Since δ t , k = ν 2 R t 2 a k / ( 72 2 ) \delta_{t,k}=\nu^2R_t^2a^k/(72\sqrt{2}) δ t , k = ν 2 R t 2 a k / ( 72 2 ) , a ≥ 1 / 10 a\ge1/10 a ≥ 1/10 , s ≥ ν s\ge\nu s ≥ ν , and C R = 550800 / ( ν 2 q ) + 18 2 / ν ≤ C ν − 3 C_R=550800/(\nu^2q)+18\sqrt{2}/\nu\le C\nu^{-3} C R = 550800/ ( ν 2 q ) + 18 2 / ν ≤ C ν − 3 , the certification ratio obeys
9 ∥ w ~ k − 1 − w t , k ∗ ∥ 2 δ t , k ≤ C κ 7 ( k ≥ 1 ) . \frac{9\lVert\widetilde{w}_{k-1}-w_{t,k}^*\rVert^2}{\delta_{t,k}}\le C\kappa^7\qquad(k\ge1). δ t , k 9 ∥ w k − 1 − w t , k ∗ ∥ 2 ≤ C κ 7 ( k ≥ 1 ) .
Here q = ν / s ≥ ν q=\nu/s\ge\nu q = ν / s ≥ ν was used. Conditional on the complete history at each auxiliary start, all these inputs are fixed and subsequent solver indices are fresh. Lemma 25 therefore gives almost-sure termination and expected cost
O ( ( n + 3 n / s ) [ 1 + log + ( C κ 7 ) ] ) = O ( n log ( 2 + κ ) ) . O\left((n+3\sqrt{n}/s)[1+\log^+(C\kappa^7)]\right)=O(n\log(2+\kappa)). O ( ( n + 3 n / s ) [ 1 + log + ( C κ 7 )] ) = O ( n log ( 2 + κ )) .
for that call, uniformly over the preceding accepted history, by induction from k = 1 k=1 k = 1 . Take α = ν 5 / 576 \alpha=\nu^5/576 α = ν 5 /576 and K = ⌈ log ( C R / α ) − log ( 1 − ρ ) ⌉ = O ( q − 1 / 2 log ( 2 + κ ) ) K=\left\lceil\frac{\log(C_R/\alpha)}{-\log(1-\rho)}\right\rceil=O(q^{-1/2}\log(2+\kappa)) K = ⌈ − l o g ( 1 − ρ ) l o g ( C R / α ) ⌉ = O ( q − 1/2 log ( 2 + κ )) . Equation (124) guarantees ∥ ∇ f ^ t ( w ~ K ) ∥ 2 ≤ α R t 2 \lVert\nabla\hat{f}_t(\widetilde{w}_K)\rVert^2\leq\alpha R_t^2 ∥ ∇ f ^ t ( w K ) ∥ 2 ≤ α R t 2 on every run. The choice of α \alpha α is no larger than the two outer tolerances in Zhang et al. [25 ] , Theorem 4.1. Consequently one outer step costs
O ( n q − 1 / 2 log 2 ( 2 + κ ) ) (125) O(nq^{-1/2}\log^2(2+\kappa)) \tag*{(125)} O ( n q − 1/2 log 2 ( 2 + κ )) (125)
in conditional expectation.
Stationarity, Total Cost, and Fixed Confidence
Start the outer method at ( 0 , y w ) (0,y_w) ( 0 , y w ) and run T o u t = ⌈ 60 D 0 / ϵ 0 2 ⌉ T_{\mathrm{out}}=\lceil60D_0/\epsilon_0^2\rceil T out = ⌈ 60 D 0 / ϵ 0 2 ⌉ outer steps, unless an exactly zero residual permits early return. The outer theorem of Zhang et al. [25 ] , Theorem 4.1, under the relative residual conditions used above, bounds the gradients at the outputs p t + 1 p_{t+1} p t + 1 of the outer steps:
1 T o u t ∑ t = 1 T o u t ∥ ∇ V f ( p t + 1 ) ∥ 2 ≤ 268 D 0 5 T o u t + 28 ∥ y w − y ∗ ( 0 ) ∥ 2 5 T o u t ≤ 296 D 0 5 T o u t ≤ ϵ 0 2 . \frac{1}{T_{\mathrm{out}}}\sum_{t=1}^{T_{\mathrm{out}}}\lVert\nabla V_f(p_{t+1})\rVert^2\leq\frac{268D_0}{5T_{\mathrm{out}}}+\frac{28\lVert y_w-y^{\ast(0)}\rVert^2}{5T_{\mathrm{out}}}\leq\frac{296D_0}{5T_{\mathrm{out}}}\leq\epsilon_0^2. T out 1 t = 1 ∑ T out ∥ ∇ V f ( p t + 1 ) ∥ 2 ≤ 5 T out 268 D 0 + 5 T out 28 ∥ y w − y ∗ ( 0 ) ∥ 2 ≤ 5 T out 296 D 0 ≤ ϵ 0 2 .
This holds pathwise for certified accepted outputs. Returning p t + 1 p_{t+1} p t + 1 for a uniform t ∈ { 1 , … , T o u t } t\in\{1,\ldots,T_{\mathrm{out}}\} t ∈ { 1 , … , T out } and scaling by L L L give (14) for the gradient of the unnormalized value function. The conditional expected costs (123) and (126) are uniform. The tower property therefore bounds total expected queries by
C n q − 1 / 2 ( 1 + S ) log 2 ( 2 + κ ) = C n 3 / 4 κ ( 1 + S ) log 2 ( 2 + κ ) . Cnq^{-1/2}(1+S)\log^2(2+\kappa)=Cn^{3/4}\sqrt{\kappa}(1+S)\log^2(2+\kappa). C n q − 1/2 ( 1 + S ) log 2 ( 2 + κ ) = C n 3/4 κ ( 1 + S ) log 2 ( 2 + κ ) .
All auxiliary saddle gaps are finite by strong convexity–concavity.
For a deterministic cap with success probability 2 / 3 2/3 2/3 , run with target squared gradient norm ϵ 2 / 6 \epsilon^2/6 ϵ 2 /6 . Choose a universal C c e r t ≥ 1 C_{\mathrm{cert}}\geq1 C cert ≥ 1 bounding both certification ratios above by C c e r t κ 7 C_{\mathrm{cert}}\kappa^7 C cert κ 7 . With T 6 = ⌈ 360 D 0 / ϵ 0 2 ⌉ T_6=\lceil360D_0/\epsilon_0^2\rceil T 6 = ⌈ 360 D 0 / ϵ 0 2 ⌉ , a valid expected-cost bound is
B Q = C e p H ( J + T 6 K ) [ 2 + ⌈ log + 2 ( C c e r t κ 7 ) ⌉ ] + n T 6 . (126) B_Q=C_{\mathrm{ep}}^H(J+T_6K)\left[2+\left\lceil\log_+^2(C_{\mathrm{cert}}\kappa^7)\right\rceil\right]+nT_6. \tag*{(126)} B Q = C ep H ( J + T 6 K ) [ 2 + ⌈ log + 2 ( C cert κ 7 ) ⌉ ] + n T 6 . (126)
This uses the L-SVRE epoch charge (119) and includes the residual R t R_t R t computed at each outer step. Cap execution at 6 B Q 6B_Q 6 B Q , returning the origin on timeout; Markov’s inequality then bounds the failure probability by 1 / 3 1/3 1/3 as in Appendix E.8, at the same rate.
Ahmet Alacaoglu and Yura Malitsky. Stochastic variance reduction for variational inequality methods. In Proceedings of Thirty Fifth Conference on Learning Theory , volume 178 of Proceedings of Machine Learning Research , pages 778–816, 2022.
Yossi Arjevani, Yair Carmon, John C. Duchi, Dylan J. Foster, Nathan Srebro, and Blake Woodworth. Lower bounds for non-convex stochastic optimization. Mathematical Programming , 199:165–214, 2023. doi: 10.1007/s10107-022-01822-7. Revised full version: arXiv:1912.02365v2.
AI Usage Statement
We used AI assistants, including OpenAI’s GPT models and Anthropic’s Claude, to help develop and check proofs, formalize them in Lean, and edit the text. The authors take full responsibility for the content of this paper.