Introduction
Sample average approximation (SAA) replaces an expected cost by an average over observed scenarios. The accuracy of a computed solution depends on both sampling error and optimization error. We study how quickly direct subgradient methods reduce the optimization error to the scale needed for inference when the scenario costs are nonsmooth.
Averaging piecewise linear costs against a distribution with a bounded conditional density produces a smooth expected objective. We show how its local strong convexity speeds computation on the nonsmooth sample objective and determines the sampling distribution of the computed solution.
Our main probabilistic result relates subgradient increments to the distance between decisions. Consider L ( θ ) = n − 1 ∑ i f ( Y i − Z i T θ ) + b T θ \mathcal{L}(\theta)=n^{-1}\sum_i f(Y_i-Z_i^{T}\theta)+b^{T}\theta L ( θ ) = n − 1 ∑ i f ( Y i − Z i T θ ) + b T θ on Θ ⊆ R p \Theta\subseteq\mathbb{R}^{p} Θ ⊆ R p , where f f f is convex and Lipschitz. Let L ‾ \overline{\mathcal{L}} L be its expected objective, write ∂ L \partial\mathcal{L} ∂ L for a selected subgradient, and set a n = p log n / n a_n=p\log n/n a n = p log n / n . We prove
∥ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ‾ ( θ 1 ) + ∇ L ‾ ( θ 2 ) ∥ 2 ≤ C { a n ∥ θ 1 − θ 2 ∥ 2 + a n } \left\lVert\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)-\nabla\overline{\mathcal{L}}(\theta_1)+\nabla\overline{\mathcal{L}}(\theta_2)\right\rVert_2 \le C\left\{\sqrt{a_n}\lVert\theta_1-\theta_2\rVert_2+a_n\right\} ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ( θ 1 ) + ∇ L ( θ 2 ) 2 ≤ C { a n ∥ θ 1 − θ 2 ∥ 2 + a n }
with high probability, simultaneously for every pair ( θ 1 , θ 2 ) ∈ Θ × Θ (\theta_1,\theta_2)\in\Theta\times\Theta ( θ 1 , θ 2 ) ∈ Θ × Θ . A pinball example shows that the smoothness approximation scale a n a_n a n is necessary up to a logarithmic factor. The assumptions control the conditional density of Y Y Y and projections of Z Z Z , without a moment condition on Y Y Y . The same bound extends to finite maxima of random affine costs, including two-stage stochastic linear programs with fixed recourse.
The increment bound controls successive iterates, extrapolated points, and the SAA solution on a common event, although all depend on the same observations. At distances larger than a n a_n a n , transfers smoothness and strong convexity from the expected objective to approximate properties of the SAA objective. Near the population solution, the distance dependence gives a linear subgradient approximation that identifies the random term governing inference.
For Lipschitz convex objectives, subgradient descent with step sizes proportional to t − ζ t^{-\zeta} t − ζ , 1 / 2 < ζ < 1 1/2 < \zeta< 1 1/2 < ζ < 1 , attains objective error O ( T − 1 ) O(T^{-1}) O ( T − 1 ) for a weighted average. Under strong convexity, diminishing steps and averaging give O ( T − 1 ) O(T^{-1}) O ( T − 1 ) ([32 ] ). Our increment bound gives three phases of faster convergence for SAA. In Phase 1, subgradient descent and Nesterov acceleration attain objective rates T − 1 T^{-1} T − 1 and T − 2 T^{-2} T − 2 and enter a region of approximate strong convexity. In Phase 2, their errors decrease toward approximation floors at rates exp ( − c T / κ ) \exp(-cT/\kappa) exp ( − c T / κ ) and exp ( − c T / κ ) \exp(-cT/\sqrt{\kappa}) exp ( − c T / κ ) , where κ \kappa κ is the ratio of the smoothness and strong convexity parameters. A one-dimensional example shows that, from almost every initialization, subgradient descent cannot reduce its error below order ϵ 1 \epsilon_1 ϵ 1 , the smoothness approximation scale. Phase 3 separates the remaining optimization error from the leading random term. Acceleration improves the geometric rate while preserving the expansion remainder.
These convergence bounds complement analyses based on linear growth and sharpness ([44 , 12 , 17 ] ). The two-phase method of Shen et al. [53 ] uses decreasing step sizes initially and a constant step size in its second phase. Our methods use a constant step size throughout and yield a distributional approximation of the computed solution. Beyond SAA, bounded local subgradient variation and quadratic growth give geometric convergence of subgradient descent and restarted Nesterov acceleration to neighborhoods whose radii have the order of the variation scale.
The iterates need not converge to a minimizer to support inference. After O ( log n ) O(\log n) O ( log n ) updates, their optimization error is O ( a n ) O(a_n) O ( a n ) , and the computed solution has the same first-order expansion as an SAA solution, with remainder O P { ( p / n ) 3 / 4 ( log n ) 1 / 2 + a n } O_P\{(p/n)^{3/4}(\log n)^{1/2}+a_n\} O P {( p / n ) 3/4 ( log n ) 1/2 + a n } . First-order equivalence holds under weaker dimension growth conditions than those needed for a Gaussian limit. The same analysis gives bounds on the population optimality gap and the sample complexity of an ϵ \epsilon ϵ -optimal decision. Each update evaluates n n n scenario subgradients, giving a total of O ( n log n ) O(n\log n) O ( n log n ) scenario evaluations.
Moreau smoothing can change the computed solution’s limiting distribution through a drift n h v T B w \sqrt{n}h\,v^\mathsf{T}B_w n h v T B w in a unit contrast v v v . We identify the first-order population displacement coefficient B w B_w B w for losses with two affine pieces and general prox-functions. We derive a residual shift h c w h_{c_w} h c w that removes the first-order displacement. The expansion remainder then improves to O P { ( p / n ) 4 / 5 ( log n ) 2 / 5 } O_P\{(p/n)^{4/5}(\log n)^{2/5}\} O P {( p / n ) 4/5 ( log n ) 2/5 } , with the limiting distribution preserved. Sample averaging keeps the smoothness and local strong convexity constants bounded as the bandwidth decreases to C a n C a_n C a n , permitting constant step sizes and convergence to the smoothed SAA solution.
Sparse decisions lead to regularized SAA. Restricting the increments to sparse vectors gives an approximation scale s log p / n s\log p/n s log p / n when the working sparsity is proportional to the population sparsity s s s . For proximal subgradient descent and iterative hard thresholding, this yields geometric convergence, support recovery, and expansions of the computed solution. On a recovered support or a deterministic retained set containing the population support, the expansions give coordinatewise normal limits after O ( log n ) O(\log n) O ( log n ) updates.
We determine the iteration budget needed to retain the sampling distribution described by the asymptotic theory of SAA solutions ([29 , 51 , 52 ] ). When fewer updates are used, the expansion also quantifies the remaining optimization error. Retrospective approximation and sample-allocation analyses use an inner solver’s convergence rate to choose sampling and stopping schedules ([43 , 48 ] ). We derive that rate for the nonsmooth SAA solver and relate its stopping accuracy to the solution expansion. The same result gives a full-batch counterpart to asymptotic normality for stochastic approximation with iterate averaging ([50 , 45 Davis et al.,] ) 2024). Here the increment bound controls dependence on the repeatedly used sample and gives an expansion of the final iterate.
The increment bound supplies both the optimization models and the probability control needed for this expansion. The objective recursions follow the inexact-oracle framework [14 , 15 ] , whose accelerated bounds accumulate model error with iterations. Here long updates remove that error, and we prove containment of iterates and extrapolated points within the regions where the models hold. The resulting approximation scales also give high-probability values of the regularity parameters in universal-gradient and local-variation analyses [39 , 16 ] .
To pass from optimization bounds to an expansion, we use the decrease of the increment error with distance. Localization of objective deviations over near-optimal sets gives bounds on SAA solutions [41 b] . Other nonsmooth SAA analyses give concentration of subdifferentials and Moreau envelopes [11 ] . In particular, Ruan [49 ] establishes uniform Hausdorff convergence of empirical subdifferentials. Subtracting two such approximation bounds retains their full error at nearby decisions. Our direct control of increments makes the error decrease as the decisions approach each other, which gives the smaller remainder in the computed solution’s expansion.
Organization and notation
Section 2 states the regularity conditions. Section 3 gives the convergence bounds and their combination. Section 4 verifies the conditions for SAA objectives and develops applications. Sections 5 and 6 treat smoothing and regularized SAA. Proofs are collected in the Appendix.
For a vector v ∈ R p v \in\mathbb{R}^{p} v ∈ R p , we use ∣ v ∣ 2 |v|_{2} ∣ v ∣ 2 , ∣ v ∣ 1 |v|_{1} ∣ v ∣ 1 , and ∣ v ∣ ∞ |v|_{\infty} ∣ v ∣ ∞ for its Euclidean, ℓ 1 \ell_{1} ℓ 1 , and maximum norms. For an index set I I I , v I v_{I} v I is the subvector indexed by I I I , and v − l v_{-l} v − l omits coordinate l l l . For a matrix A A A , ∥ A ∥ \lVert A \rVert ∥ A ∥ and ∥ A ∥ ∞ \lVert A \rVert_{\infty} ∥ A ∥ ∞ are the induced Euclidean and maximum norms. For a symmetric matrix, Λ max ( A ) \Lambda_{\max}(A) Λ m a x ( A ) and Λ min ( A ) \Lambda_{\min}(A) Λ m i n ( A ) are its largest and smallest eigenvalues. We write A I × J A_{I \times J} A I × J for the submatrix indexed by rows I I I and columns J J J . The indicator I ( A ) \mathbb{I}(A) I ( A ) equals one when the event or condition A A A holds and zero otherwise. We write B ( θ , r ) B(\theta,r) B ( θ , r ) for the closed Euclidean ball with center θ \theta θ and radius r r r . For a random variable W W W and q > 0 q>0 q > 0 , write ∥ W ∥ ψ q = inf { c > 0 : E exp ( ∣ W / c ∣ q ) ≤ 2 } \lVert W \rVert_{\psi_{q}} = \inf\{c>0:\mathbb{E}\exp(|W/c|^{q}) \le2\} ∥ W ∥ ψ q = inf { c > 0 : E exp ( ∣ W / c ∣ q ) ≤ 2 } . For positive sequences, a n ≍ b n a_{n} \asymp b_{n} a n ≍ b n means that their ratio is bounded above and away from zero. Unless stated otherwise, constants and the step sizes used in asymptotic iteration bounds are independent of n n n and p p p .
Problem and regularity conditions
Let L \mathcal{L} L be an SAA objective on the decision region Θ ⊆ R p \Theta\subseteq\mathbb{R}^{p} Θ ⊆ R p , and let L ~ \widetilde{\mathcal{L}} L be its expected objective. When individual losses are not integrable, use the loss-difference definition in Section 4. We write
θ ∗ ∈ argmin θ ∈ Θ L ~ ( θ ) , θ ^ ∈ argmin θ ∈ Θ L ( θ ) (1) \theta^{*} \in\underset{\theta\in\Theta}{\operatorname{argmin}}\,\widetilde{\mathcal{L}}(\theta), \qquad\widehat{\theta} \in\underset{\theta\in\Theta}{\operatorname{argmin}}\,\mathcal{L}(\theta) \tag*{(1)} θ ∗ ∈ θ ∈ Θ argmin L ( θ ) , θ ∈ θ ∈ Θ argmin L ( θ ) (1)
for the population solution and an SAA solution. For a computed solution θ ~ \widetilde{\theta} θ , the differences θ ~ − θ ^ \widetilde{\theta}-\widehat{\theta} θ − θ and θ ^ − θ ∗ \widehat{\theta}-\theta^{*} θ − θ ∗ are the optimization error and sampling error, respectively. The sample remains fixed throughout the algorithm. We use ∂ L ( θ ) \partial\mathcal{L}(\theta) ∂ L ( θ ) for a measurable selection of a subgradient at θ \theta θ . At a differentiable point it equals ∇ L ( θ ) \nabla\mathcal{L}(\theta) ∇ L ( θ ) . All updates and derivatives are taken in the free decision coordinates. Affine feasible sets are parametrized by these coordinates.
We first state deterministic conditions under which subgradient descent and Nesterov acceleration inherit convergence properties of smooth optimization. Section 4 verifies these conditions with high probability for a class of SAA objectives.
Approximate smoothness and strong convexity
Condition 1. For L > 0 L > 0 L > 0 and ϵ 1 ≥ 0 \epsilon_1 \ge0 ϵ 1 ≥ 0 , the function L \mathcal{L} L is ( L , ϵ 1 ) (L,\epsilon_1) ( L , ϵ 1 ) -smooth on a set U U U if, for all θ 1 , θ 2 ∈ U \theta_1,\theta_2 \in U θ 1 , θ 2 ∈ U ,
∥ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) ∥ 2 ≤ L max { ∥ θ 1 − θ 2 ∥ 2 , ϵ 1 } . \lVert\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)\rVert_2 \le L\max\{\lVert\theta_1-\theta_2\rVert_2,\epsilon_1\}. ∥ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) ∥ 2 ≤ L max {∥ θ 1 − θ 2 ∥ 2 , ϵ 1 } .
Condition 1 bounds subgradient increments by distance when that distance is at least ϵ 1 \epsilon_1 ϵ 1 , and by L ϵ 1 L\epsilon_1 L ϵ 1 at smaller distances. This permits discontinuities in the selected subgradient while retaining the quadratic upper bound used in descent analysis. Every function with maximum local subgradient variation bounded by V ϵ 1 V_{\epsilon_1} V ϵ 1 at radius ϵ 1 > 0 \epsilon_1 > 0 ϵ 1 > 0 on a convex region satisfies Condition 1 with L = 2 V ϵ 1 / ϵ 1 L = 2V_{\epsilon_1}/\epsilon_1 L = 2 V ϵ 1 / ϵ 1 . Partitioning a segment into pieces of length at most ϵ 1 \epsilon_1 ϵ 1 gives this implication from the BVGmax _{\max} m a x class of Diakonikolas and Guzmán [16 ] .
Lemma 1. Suppose Condition 1 holds for a finite convex function L \mathcal{L} L on a convex region. For every pair θ 1 , θ 2 \theta_1,\theta_2 θ 1 , θ 2 in that region with ∥ θ 1 − θ 2 ∥ 2 ≥ ϵ 1 \lVert\theta_1-\theta_2\rVert_2 \ge\epsilon_1 ∥ θ 1 − θ 2 ∥ 2 ≥ ϵ 1 ,
L ( θ 1 ) − L ( θ 2 ) ≤ ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ + L ∥ θ 1 − θ 2 ∥ 2 2 . (2) \mathcal{L}(\theta_1)-\mathcal{L}(\theta_2) \le\langle\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle+L\lVert\theta_1-\theta_2\rVert_2^2. \tag*{(2)} L ( θ 1 ) − L ( θ 2 ) ≤ ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ + L ∥ θ 1 − θ 2 ∥ 2 2 . (2)
For 0 < η ≤ 1 / L 0 < \eta\le1/L 0 < η ≤ 1/ L and θ , θ + = θ − η ∂ L ( θ ) \theta,\theta^+ = \theta-\eta\partial\mathcal{L}(\theta) θ , θ + = θ − η ∂ L ( θ ) in the region,
L ( θ + ) ≤ L ( θ ) − η 2 ∥ ∂ L ( θ ) ∥ 2 2 + η L 2 ϵ 1 2 2 . (3) \mathcal{L}(\theta^+) \le\mathcal{L}(\theta)-\frac{\eta}{2}\lVert\partial\mathcal{L}(\theta)\rVert_2^2+\frac{\eta L^2\epsilon_1^2}{2}. \tag*{(3)} L ( θ + ) ≤ L ( θ ) − 2 η ∥ ∂ L ( θ ) ∥ 2 2 + 2 η L 2 ϵ 1 2 . (3)
The descent bound (3) applies at every update. For η ≤ 1 / ( 2 L ) \eta\le1/(2L) η ≤ 1/ ( 2 L ) , an update of length at least ϵ 1 \epsilon_1 ϵ 1 removes the additive error by (2).
Condition 2. For μ > 0 \mu> 0 μ > 0 and ϵ 2 ≥ 0 \epsilon_2 \ge0 ϵ 2 ≥ 0 , the function L \mathcal{L} L is ( μ , ϵ 2 ) (\mu,\epsilon_2) ( μ , ϵ 2 ) -strongly convex on a set U U U if
L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ μ 4 ∥ θ 1 − θ 2 ∥ 2 2 \mathcal{L}(\theta_1)-\mathcal{L}(\theta_2)-\langle\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle\ge\frac{\mu}{4}\lVert\theta_1-\theta_2\rVert_2^2 L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ 4 μ ∥ θ 1 − θ 2 ∥ 2 2
for all θ 1 , θ 2 ∈ U \theta_1,\theta_2 \in U θ 1 , θ 2 ∈ U with ∥ θ 1 − θ 2 ∥ 2 ≥ ϵ 2 \lVert\theta_1-\theta_2\rVert_2 \ge\epsilon_2 ∥ θ 1 − θ 2 ∥ 2 ≥ ϵ 2 .
Condition 2 requires a quadratic lower bound at distances of at least ϵ 2 \epsilon_2 ϵ 2 . Section 4 proves that the SAA objective satisfies this condition wherever the expected objective is strongly convex. Suppose both conditions hold on a convex region, L \mathcal{L} L is finite and convex there, and its minimizer θ ^ \widehat{\theta} θ lies in the interior. For ∥ θ − θ ^ ∥ 2 ≥ max { ϵ 1 , 2 ϵ 2 } \lVert\theta-\widehat{\theta}\rVert_2 \ge\max\{\epsilon_1,\sqrt{2\epsilon_2}\} ∥ θ − θ ∥ 2 ≥ max { ϵ 1 , 2 ϵ 2 } , Lemma 3 gives
4 L 2 ∥ θ − θ ^ ∥ 2 2 ≥ ∥ ∂ L ( θ ) ∥ 2 2 ≥ μ { L ( θ ) − L ( θ ^ ) } ≥ μ 2 8 ∥ θ − θ ^ ∥ 2 2 . (4) 4L^2\lVert\theta-\widehat{\theta}\rVert_2^2 \ge\lVert\partial\mathcal{L}(\theta)\rVert_2^2 \ge\mu\{\mathcal{L}(\theta)-\mathcal{L}(\widehat{\theta})\} \ge\frac{\mu^2}{8}\lVert\theta-\widehat{\theta}\rVert_2^2. \tag*{(4)} 4 L 2 ∥ θ − θ ∥ 2 2 ≥ ∥ ∂ L ( θ ) ∥ 2 2 ≥ μ { L ( θ ) − L ( θ )} ≥ 8 μ 2 ∥ θ − θ ∥ 2 2 . (4)
This approximate form of metric subregularity ([18 ] ) lets a subgradient norm threshold control the objective gap and distance without knowing θ ^ \widehat{\theta} θ .
Local subgradient approximation
Condition 3. The function L \mathcal{L} L satisfies the local subgradient approximation with matrix H H H , radius ϵ 3 \epsilon_3 ϵ 3 , and subgradient remainder bound r n r_n r n if H H H is symmetric positive definite and
sup ∥ θ − θ ∗ ∥ 2 ≤ ϵ 3 ∥ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∥ 2 ≤ r n . \sup_{\lVert\theta-\theta^*\rVert_2\le\epsilon_3}\lVert\partial\mathcal{L}(\theta)-\partial\mathcal{L}(\theta^*)-H(\theta-\theta^*)\rVert_2 \le r_n. ∥ θ − θ ∗ ∥ 2 ≤ ϵ 3 sup ∥ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∥ 2 ≤ r n .
Condition 3 approximates the subgradient increment by the linear map H ( θ − θ ∗ ) H(\theta-\theta^*) H ( θ − θ ∗ ) . Typically, H = ∇ 2 L ( θ ∗ ) H=\nabla^2\mathcal{L}(\theta^*) H = ∇ 2 L ( θ ∗ ) . When r n = o ( ϵ 3 ) r_n=o(\epsilon_3) r n = o ( ϵ 3 ) , this approximation gives a linear recursion for iterates near θ ∗ \theta^* θ ∗ , up to a smaller remainder. Its solution identifies the contribution of the random subgradient ∂ L ( θ ∗ ) \partial\mathcal{L}(\theta^*) ∂ L ( θ ∗ ) and the remaining optimization error.
A three-phase scheme
The three phases use progressively finer consequences of the same subgradient increment bound, verified in § 4 . Phase 1 reaches a region where approximate strong convexity holds. Phase 2 uses this curvature to reduce the optimization error geometrically. Phase 3 reduces the transient term to the order of the subgradient remainder, identifying the leading random term of the computed solution.
Algorithm 1 allows the update method and its parameters to vary between phases. Appendix § 8 .§ 3 gives complete procedures, iteration counts, and proofs of containment. The procedures use (4) to set subgradient norm thresholds that ensure the required update lengths and distances before Phase 3 and determine entry into its local region.
Algorithm 1 Three-phase subgradient method
Input: θ ( 0 ) \theta^{(0)} θ ( 0 ) , methods A j \mathcal{A}_j A j , and stopping criteria Cri j \operatorname{Cri}_j Cri j , j = 1 , 2 , 3 j=1,2,3 j = 1 , 2 , 3 .
Initialize θ ~ = θ ( 0 ) \widetilde{\theta}=\theta^{(0)} θ = θ ( 0 ) .
for j = 1 , 2 , 3 j=1,2,3 j = 1 , 2 , 3 do
while Cri j \operatorname{Cri}_j Cri j is not met do
Apply the Phase j j j update θ ~ + = A j ( θ ~ ) \widetilde{\theta}^{+}=\mathcal{A}_j(\widetilde{\theta}) θ + = A j ( θ ) .
Set θ ~ = θ ~ + \widetilde{\theta}=\widetilde{\theta}^{+} θ = θ + .
end while
end for
Output: Computed solution θ ~ \widetilde{\theta} θ .
Subgradient methods
We establish phasewise convergence bounds for subgradient descent and Nesterov’s accelerated subgradient method. § 3.3 combines the objective bounds to obtain geometric convergence with a constant step size throughout the computation. The proofs transfer the quadratic upper and lower models at the point pairs where each algorithm uses them. At distances above ϵ 1 \epsilon_1 ϵ 1 and ϵ 2 \epsilon_2 ϵ 2 , these models have no additive error. At shorter distances, the potential weights determine how the errors in Lemma 2 in Appendix § 8 enter the final rate.
Subgradient descent
Algorithm 2 Subgradient descent
Input: Initial iterate θ ( 0 ) \theta^{(0)} θ ( 0 ) , iterations T T T , and step size η \eta η .
for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 do
Subgradient update:
θ ( t + 1 ) = θ ( t ) − η ∂ L ( θ ( t ) ) . \theta^{(t+1)}=\theta^{(t)}-\eta\partial\mathcal{L}(\theta^{(t)}). θ ( t + 1 ) = θ ( t ) − η ∂ L ( θ ( t ) ) .
end for
Output: Final iterate θ ( T ) \theta^{(T)} θ ( T ) .
Approximate smoothness gives the T − 1 T^{-1} T − 1 objective rate in Phase 1, and approximate strong convexity gives geometric convergence in Phase 2. Theorem 1 also bounds the error when the updates continue below the approximation scales.
Theorem 1. Index each phase from t = 0 t=0 t = 0 . For parts (i) and (ii), suppose L \mathcal{L} L is convex and satisfies Condition 1 on a convex set Θ \Theta Θ containing θ ( 0 ) , … , θ ( T ) \theta^{(0)},\ldots,\theta^{(T)} θ ( 0 ) , … , θ ( T ) and θ ^ \hat{\theta} θ ^ .
(i) Phase 1. Let 0 < η ≤ 1 / L 0<\eta\le1/L 0 < η ≤ 1/ L . Then
1 T ∑ t = 1 T { L ( θ ( t ) ) − L ( θ ^ ) } ≤ ∣ θ ( 0 ) − θ ^ ∣ 2 2 2 T η + η L 2 ϵ 1 2 2 . \frac{1}{T}\sum_{t=1}^{T}\{\mathcal{L}(\theta^{(t)})-\mathcal{L}(\hat{\theta})\}\le\frac{|\theta^{(0)}-\hat{\theta}|_{2}^{2}}{2T\eta}+\frac{\eta L^{2}\epsilon_{1}^{2}}{2}. T 1 t = 1 ∑ T { L ( θ ( t ) ) − L ( θ ^ )} ≤ 2 T η ∣ θ ( 0 ) − θ ^ ∣ 2 2 + 2 η L 2 ϵ 1 2 . If η ≤ 1 / ( 2 L ) \eta\le1/(2L) η ≤ 1/ ( 2 L ) and ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 ≥ ϵ 1 |\theta^{(t+1)}-\theta^{(t)}|_{2}\ge\epsilon_{1} ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 ≥ ϵ 1 for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 , the objective values and distances to θ ^ \hat{\theta} θ ^ are nonincreasing, and
L ( θ ( T ) ) − L ( θ ^ ) ≤ ∣ θ ( 0 ) − θ ^ ∣ 2 2 2 T η . \mathcal{L}(\theta^{(T)})-\mathcal{L}(\hat{\theta})\le\frac{|\theta^{(0)}-\hat{\theta}|_{2}^{2}}{2T\eta}. L ( θ ( T ) ) − L ( θ ^ ) ≤ 2 T η ∣ θ ( 0 ) − θ ^ ∣ 2 2 . (ii) Phase 2. Let Θ s c ⊆ Θ \Theta_{\mathrm{sc}}\subseteq\Theta Θ sc ⊆ Θ , and impose Condition 2 on that subset. Assume θ ( t ) , θ ^ ∈ Θ s c \theta^{(t)},\hat{\theta}\in\Theta_{\mathrm{sc}} θ ( t ) , θ ^ ∈ Θ sc for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 . For 0 < η ≤ 1 / L 0<\eta\le1/L 0 < η ≤ 1/ L and 0 < μ ≤ 2 L 0<\mu\le2L 0 < μ ≤ 2 L ,
L ( θ ( T ) ) − L ( θ ^ ) ≤ { 1 − μ ( η − L η 2 / 2 ) } T { L ( θ ( 0 ) ) − L ( θ ^ ) } + μ ϵ 2 2 4 + L ϵ 1 2 2 μ ( η − L η 2 / 2 ) . \mathcal{L}(\theta^{(T)})-\mathcal{L}(\hat{\theta})\le\{1-\mu(\eta-L\eta^{2}/2)\}^{T}\{\mathcal{L}(\theta^{(0)})-\mathcal{L}(\hat{\theta})\}+\frac{\mu\epsilon_{2}^{2}}{4}+\frac{L\epsilon_{1}^{2}}{2\mu(\eta-L\eta^{2}/2)}. L ( θ ( T ) ) − L ( θ ^ ) ≤ { 1 − μ ( η − L η 2 /2 ) } T { L ( θ ( 0 ) ) − L ( θ ^ )} + 4 μ ϵ 2 2 + 2 μ ( η − L η 2 /2 ) L ϵ 1 2 . For any step size satisfying 0 < μ ( η − η 2 L ) ≤ 1 0<\mu(\eta-\eta^{2}L)\le1 0 < μ ( η − η 2 L ) ≤ 1 , if ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 ≥ ϵ 1 |\theta^{(t+1)}-\theta^{(t)}|_{2}\ge\epsilon_{1} ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 ≥ ϵ 1 and ∣ θ ( t ) − θ ^ ∣ 2 ≥ ϵ 2 |\theta^{(t)}-\hat{\theta}|_{2}\ge\epsilon_{2} ∣ θ ( t ) − θ ^ ∣ 2 ≥ ϵ 2 for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 , then
L ( θ ( T ) ) − L ( θ ^ ) ≤ { 1 − μ ( η − η 2 L ) } T { L ( θ ( 0 ) ) − L ( θ ^ ) } . \mathcal{L}(\theta^{(T)})-\mathcal{L}(\hat{\theta})\le\{1-\mu(\eta-\eta^{2}L)\}^{T}\{\mathcal{L}(\theta^{(0)})-\mathcal{L}(\hat{\theta})\}. L ( θ ( T ) ) − L ( θ ^ ) ≤ { 1 − μ ( η − η 2 L ) } T { L ( θ ( 0 ) ) − L ( θ ^ )} . (iii) Phase 3. Suppose the local subgradient approximation in Condition 3 holds with remainder r n r_n r n , and m H I ⪯ H ⪯ M H I m_H I\preceq H\preceq M_H I m H I ⪯ H ⪯ M H I for constants 0 < m H ≤ M H 0<m_H\le M_H 0 < m H ≤ M H . Assume ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ ϵ 3 / 2 |\theta^{(0)}-\theta^*|_{2}\le\epsilon_{3}/2 ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ ϵ 3 /2 , ∣ ∂ L ( θ ∗ ) ∣ 2 + r n ≤ m H ϵ 3 / 2 |\partial\mathcal{L}(\theta^*)|_{2}+r_n\le m_H\epsilon_{3}/2 ∣ ∂ L ( θ ∗ ) ∣ 2 + r n ≤ m H ϵ 3 /2 , and 0 < η ≤ M H − 1 0<\eta\le M_H^{-1} 0 < η ≤ M H − 1 . Then ∣ θ ( t ) − θ ∗ ∣ 2 ≤ ϵ 3 / 2 |\theta^{(t)}-\theta^*|_{2}\le\epsilon_{3}/2 ∣ θ ( t ) − θ ∗ ∣ 2 ≤ ϵ 3 /2 for every t ≥ 0 t\ge0 t ≥ 0 , and
θ ( T ) − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + ( I − η H ) T { θ ( 0 ) − θ ∗ + H − 1 ∂ L ( θ ∗ ) } + R T , (5) \theta^{(T)}-\theta^*=-H^{-1}\partial\mathcal{L}(\theta^*)+(I-\eta H)^{T}\{\theta^{(0)}-\theta^*+H^{-1}\partial\mathcal{L}(\theta^*)\}+R_T, \tag*{(5)} θ ( T ) − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + ( I − η H ) T { θ ( 0 ) − θ ∗ + H − 1 ∂ L ( θ ∗ )} + R T , (5) where ∣ R T ∣ 2 ≤ r n / m H |R_T|_{2}\le r_n/m_H ∣ R T ∣ 2 ≤ r n / m H .
In (5), the term − H − 1 ∂ L ( θ ∗ ) -H^{-1}\partial\mathcal{L}(\theta^*) − H − 1 ∂ L ( θ ∗ ) approximates the sampling error. The transient term depends on initialization and decreases geometrically. Condition 3 controls the expansion remainder. § 4 determines the iteration count needed to match the first-order expansion of an SAA solution. The expansion is often called a Bahadur representation in statistics [21 ] .
Projected subgradient descent and projected Nesterov acceleration attain the Phase 1 and geometric Phase 2 bounds on any closed convex decision region. Corollary 10 in Appendix § 8 proves these bounds and, for an interior population solution, the Phase 3 expansions (5) and (7).
Nesterov's accelerated subgradient method
Algorithm 3 combines a subgradient step with Nesterov’s extrapolation [37 ] .
Algorithm 3 Nesterov’s accelerated subgradient method
Input: Initial iterate θ ( 0 ) \theta^{(0)} θ ( 0 ) , iterations T T T , step size η \eta η , and momentum γ t \gamma_t γ t .
Initialize β ( 0 ) = θ ( 0 ) \beta^{(0)} = \theta^{(0)} β ( 0 ) = θ ( 0 ) .
for t = 0 , … , T − 1 t = 0,\ldots,T-1 t = 0 , … , T − 1 do
Subgradient update:
θ ( t + 1 ) = β ( t ) − η ∂ L ( β ( t ) ) . \theta^{(t+1)} = \beta^{(t)} - \eta\partial\mathcal{L}(\beta^{(t)}). θ ( t + 1 ) = β ( t ) − η ∂ L ( β ( t ) ) . Extrapolation:
β ( t + 1 ) = ( 1 + γ t ) θ ( t + 1 ) − γ t θ ( t ) . \beta^{(t+1)} = (1+\gamma_t)\theta^{(t+1)}-\gamma_t\theta^{(t)}. β ( t + 1 ) = ( 1 + γ t ) θ ( t + 1 ) − γ t θ ( t ) . end for
Output: Final iterate θ ( T ) \theta^{(T)} θ ( T ) .
Theorem 2. Index each phase from t = 0 t=0 t = 0 . For parts (i) and (ii), suppose L \mathcal{L} L is convex on a convex set Θ \Theta Θ containing θ ( 0 ) , … , θ ( T ) \theta^{(0)},\ldots,\theta^{(T)} θ ( 0 ) , … , θ ( T ) , β ( 0 ) , … , β ( T − 1 ) \beta^{(0)},\ldots,\beta^{(T-1)} β ( 0 ) , … , β ( T − 1 ) , and θ ^ \hat{\theta} θ ^ . Take η > 0 \eta>0 η > 0 .
(i) Phase 1. Let γ t = ( λ t − 1 ) / λ t + 1 \gamma_t=(\lambda_t-1)/\lambda_{t+1} γ t = ( λ t − 1 ) / λ t + 1 , where λ 0 = 1 \lambda_0=1 λ 0 = 1 and λ t + 1 = ( 1 + 1 + 4 λ t 2 ) / 2 \lambda_{t+1}=(1+\sqrt{1+4\lambda_t^2})/2 λ t + 1 = ( 1 + 1 + 4 λ t 2 ) /2 . Suppose δ t ≥ 0 \delta_t\geq0 δ t ≥ 0 satisfies
L ( θ ( t + 1 ) ) ≤ L ( β ( t ) ) − η 2 ∥ ∂ L ( β ( t ) ) ∥ 2 2 + δ t . \mathcal{L}(\theta^{(t+1)})\leq\mathcal{L}(\beta^{(t)})-\frac{\eta}{2}\lVert\partial\mathcal{L}(\beta^{(t)})\rVert_2^2+\delta_t. L ( θ ( t + 1 ) ) ≤ L ( β ( t ) ) − 2 η ∥ ∂ L ( β ( t ) ) ∥ 2 2 + δ t . Then
L ( θ ( T ) ) − L ( θ ^ ) ≤ ∥ θ ( 0 ) − θ ^ ∥ 2 2 / ( 2 η ) + ∑ t = 0 T − 1 λ t 2 δ t λ T − 1 2 . (6) \mathcal{L}(\theta^{(T)})-\mathcal{L}(\hat{\theta})\leq\frac{\lVert\theta^{(0)}-\hat{\theta}\rVert_2^2/(2\eta)+\sum_{t=0}^{T-1}\lambda_t^2\delta_t}{\lambda_{T-1}^2}. \tag*{(6)} L ( θ ( T ) ) − L ( θ ^ ) ≤ λ T − 1 2 ∥ θ ( 0 ) − θ ^ ∥ 2 2 / ( 2 η ) + ∑ t = 0 T − 1 λ t 2 δ t . (6) Under Condition 1 and η ≤ 1 / ( 2 L ) \eta\leq1/(2L) η ≤ 1/ ( 2 L ) , take δ t = η L 2 ϵ 1 2 I { ∥ θ ( t + 1 ) − β ( t ) ∥ 2 < ϵ 1 } / 2 \delta_t=\eta L^2\epsilon_1^2\mathbb{I}\{\lVert\theta^{(t+1)}-\beta^{(t)}\rVert_2<\epsilon_1\}/2 δ t = η L 2 ϵ 1 2 I {∥ θ ( t + 1 ) − β ( t ) ∥ 2 < ϵ 1 } /2 . This gives
L ( θ ( T ) ) − L ( θ ^ ) ≤ 2 ∥ θ ( 0 ) − θ ^ ∥ 2 2 η T 2 + T η L 2 ϵ 1 2 2 . \mathcal{L}(\theta^{(T)})-\mathcal{L}(\hat{\theta})\leq\frac{2\lVert\theta^{(0)}-\hat{\theta}\rVert_2^2}{\eta T^2}+\frac{T\eta L^2\epsilon_1^2}{2}. L ( θ ( T ) ) − L ( θ ^ ) ≤ η T 2 2 ∥ θ ( 0 ) − θ ^ ∥ 2 2 + 2 T η L 2 ϵ 1 2 . If ∥ θ ( t + 1 ) − β ( t ) ∥ 2 ≥ ϵ 1 \lVert\theta^{(t+1)}-\beta^{(t)}\rVert_2\geq\epsilon_1 ∥ θ ( t + 1 ) − β ( t ) ∥ 2 ≥ ϵ 1 for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 , then
L ( θ ( T ) ) − L ( θ ^ ) ≤ 2 ∥ θ ( 0 ) − θ ^ ∥ 2 2 η T 2 . \mathcal{L}(\theta^{(T)})-\mathcal{L}(\hat{\theta})\leq\frac{2\lVert\theta^{(0)}-\hat{\theta}\rVert_2^2}{\eta T^2}. L ( θ ( T ) ) − L ( θ ^ ) ≤ η T 2 2 ∥ θ ( 0 ) − θ ^ ∥ 2 2 . (ii) Phase 2. Suppose Condition 1 holds on Θ \Theta Θ and η ≤ 1 / L \eta\leq1/L η ≤ 1/ L . Let Θ s c ⊆ Θ \Theta_{\mathrm{sc}}\subseteq\Theta Θ sc ⊆ Θ , and impose Condition 2 on that subset. Assume β ( t ) , θ ^ ∈ Θ s c \beta^{(t)},\hat{\theta}\in\Theta_{\mathrm{sc}} β ( t ) , θ ^ ∈ Θ sc for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 and γ t ≡ γ ∈ [ 0 , 1 ) \gamma_t\equiv\gamma\in[0,1) γ t ≡ γ ∈ [ 0 , 1 ) . For α = min { ( 1 − γ ) / ( 1 + γ ) , ( η μ / 2 ) ( 1 + γ ) / ( 1 − γ ) } \alpha=\min\{(1-\gamma)/(1+\gamma),(\eta\mu/2)(1+\gamma)/(1-\gamma)\} α = min {( 1 − γ ) / ( 1 + γ ) , ( η μ /2 ) ( 1 + γ ) / ( 1 − γ )} ,
L ( θ ( T ) ) − L ( θ ^ ) ≤ ( 1 − α ) T { L ( θ ( 0 ) ) − L ( θ ^ ) + α 2 2 η ∥ θ ( 0 ) − θ ^ ∥ 2 2 } + η L 2 ϵ 1 2 2 α + μ ϵ 2 2 4 . \mathcal{L}(\theta^{(T)})-\mathcal{L}(\hat{\theta})\leq(1-\alpha)^T\left\{\mathcal{L}(\theta^{(0)})-\mathcal{L}(\hat{\theta})+\frac{\alpha^2}{2\eta}\lVert\theta^{(0)}-\hat{\theta}\rVert_2^2\right\}+\frac{\eta L^2\epsilon_1^2}{2\alpha}+\frac{\mu\epsilon_2^2}{4}. L ( θ ( T ) ) − L ( θ ^ ) ≤ ( 1 − α ) T { L ( θ ( 0 ) ) − L ( θ ^ ) + 2 η α 2 ∥ θ ( 0 ) − θ ^ ∥ 2 2 } + 2 α η L 2 ϵ 1 2 + 4 μ ϵ 2 2 . If η ≤ 1 / ( 2 L ) \eta\leq1/(2L) η ≤ 1/ ( 2 L ) , ∥ θ ( t + 1 ) − β ( t ) ∥ 2 ≥ ϵ 1 \lVert\theta^{(t+1)}-\beta^{(t)}\rVert_2\geq\epsilon_1 ∥ θ ( t + 1 ) − β ( t ) ∥ 2 ≥ ϵ 1 , and ∥ β ( t ) − θ ^ ∥ 2 ≥ ϵ 2 \lVert\beta^{(t)}-\hat{\theta}\rVert_2\geq\epsilon_2 ∥ β ( t ) − θ ^ ∥ 2 ≥ ϵ 2 for t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 , then
L ( θ ( T ) ) − L ( θ ^ ) ≤ ( 1 − α ) T { L ( θ ( 0 ) ) − L ( θ ^ ) + α 2 2 η ∥ θ ( 0 ) − θ ^ ∥ 2 2 } . \mathcal{L}(\theta^{(T)})-\mathcal{L}(\hat{\theta})\leq(1-\alpha)^T\left\{\mathcal{L}(\theta^{(0)})-\mathcal{L}(\hat{\theta})+\frac{\alpha^2}{2\eta}\lVert\theta^{(0)}-\hat{\theta}\rVert_2^2\right\}. L ( θ ( T ) ) − L ( θ ^ ) ≤ ( 1 − α ) T { L ( θ ( 0 ) ) − L ( θ ^ ) + 2 η α 2 ∥ θ ( 0 ) − θ ^ ∥ 2 2 } . (iii) Phase 3. Suppose Condition 3 holds with remainder r n r_n r n , and m H I ⪯ H ⪯ M H I m_H I\preceq H\preceq M_H I m H I ⪯ H ⪯ M H I for 0 < m H ≤ M H 0<m_H\leq M_H 0 < m H ≤ M H . Let 0 < η ≤ M H − 1 0<\eta\leq M_H^{-1} 0 < η ≤ M H − 1 and γ = ( 1 − η m H ) / ( 1 + η m H ) \gamma=(1-\sqrt{\eta m_H})/(1+\sqrt{\eta m_H}) γ = ( 1 − η m H ) / ( 1 + η m H ) . Restart Algorithm 3 with γ t ≡ γ \gamma_t\equiv\gamma γ t ≡ γ .
If ( 1 + 2 γ ) ( ∣ θ ( 0 ) − θ ∗ + H − 1 ∂ L ( θ ∗ ) ∣ 2 + r n / m H ) + ∣ H − 1 ∂ L ( θ ∗ ) ∣ 2 ≤ ϵ 3 (1+2\gamma)(\lvert\theta^{(0)}-\theta^{*}+\mathbf{H}^{-1}\partial\mathcal{L}(\theta^{*})\rvert_{2}+r_n/m_H)+\lvert\mathbf{H}^{-1}\partial\mathcal{L}(\theta^{*})\rvert_{2}\le\epsilon_3 ( 1 + 2 γ ) (∣ θ ( 0 ) − θ ∗ + H − 1 ∂ L ( θ ∗ ) ∣ 2 + r n / m H ) + ∣ H − 1 ∂ L ( θ ∗ ) ∣ 2 ≤ ϵ 3 , then every iterate and extrapolated point lies in B ( θ ∗ , ϵ 3 ) \mathbb{B}(\theta^{*},\epsilon_3) B ( θ ∗ , ϵ 3 ) . For every T ≥ 1 T\ge1 T ≥ 1 ,
θ ( T ) − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + P T { θ ( 0 ) − θ ∗ + H − 1 ∂ L ( θ ∗ ) } + R T . (7) \theta^{(T)}-\theta^{*}=-\mathbf{H}^{-1}\partial\mathcal{L}(\theta^{*})+P_T\{\theta^{(0)}-\theta^{*}+\mathbf{H}^{-1}\partial\mathcal{L}(\theta^{*})\}+R_T. \tag*{(7)} θ ( T ) − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + P T { θ ( 0 ) − θ ∗ + H − 1 ∂ L ( θ ∗ )} + R T . (7)
where P T P_T P T is a matrix polynomial in H \mathbf{H} H satisfying ∥ P T ∥ ≤ ( 1 + T η m H ) ( 1 − η m H ) T ≤ 2 exp ( − T η m H / 2 ) \lVert P_T\rVert\le(1+T\sqrt{\eta m_H})(1-\sqrt{\eta m_H})^T\le2\exp(-T\sqrt{\eta m_H}/2) ∥ P T ∥ ≤ ( 1 + T η m H ) ( 1 − η m H ) T ≤ 2 exp ( − T η m H /2 ) , and ∣ R T ∣ 2 ≤ r n / m H \lvert R_T\rvert_2\le r_n/m_H ∣ R T ∣ 2 ≤ r n / m H .
For the bound without additive terms in part (ii), the proof uses approximate smoothness on ( θ ( t + 1 ) , β ( t ) ) (\theta^{(t+1)},\beta^{(t)}) ( θ ( t + 1 ) , β ( t ) ) and approximate strong convexity on ( β ( t ) , θ ^ ) (\beta^{(t)},\widehat{\theta}) ( β ( t ) , θ ) , at distances of at least ϵ 1 \epsilon_1 ϵ 1 and ϵ 2 \epsilon_2 ϵ 2 , respectively. For the bounds with additive terms, take 0 < μ ≤ 2 L 0<\mu\le2L 0 < μ ≤ 2 L and η = 1 / L \eta=1/L η = 1/ L . Nesterov acceleration with γ = ( 2 L − μ ) / ( 2 L + μ ) \gamma=(\sqrt{2L}-\sqrt{\mu})/(\sqrt{2L}+\sqrt{\mu}) γ = ( 2 L − μ ) / ( 2 L + μ ) gives factor 1 − μ / ( 2 L ) 1-\sqrt{\mu/(2L)} 1 − μ / ( 2 L ) , compared with 1 − μ / ( 2 L ) 1-\mu/(2L) 1 − μ / ( 2 L ) for subgradient descent in Theorem 1(ii). The Nesterov factor is smaller when μ < 2 L \mu<2L μ < 2 L , and both factors are zero at μ = 2 L \mu=2L μ = 2 L .
With η = 1 / M H \eta=1/M_H η = 1/ M H , part (iii) gives the accelerated geometric factor 1 − m H / M H 1-\sqrt{m_H/M_H} 1 − m H / M H and preserves the sampling term and remainder bound r n / m H r_n/m_H r n / m H of subgradient descent. Consequently, acceleration reduces the condition-number dependence of the Phase 3 iteration count from M H / m H M_H/m_H M H / m H to M H / m H \sqrt{M_H/m_H} M H / m H .
Proposition 4 in Appendix § 8 treats an arbitrary prescribed momentum under a smaller step-size bound and compares its geometric factor with subgradient descent at the same step size. Corollary 11 in Appendix § 8 .§ 4 gives geometric convergence and a Phase 3 expansion for heavy-ball updates.
Convergence with a constant step size
The phase bounds combine into an optimization guarantee for every later iterate. The next corollary proves entry into the region of approximate strong convexity and containment after entry, including updates shorter than ϵ 1 \epsilon_1 ϵ 1 . § 4 uses the resulting distance bound to compare the computed and SAA solutions.
Corollary 1. Suppose L \mathcal{L} L is finite and convex on R p \mathbb{R}^{p} R p and has a minimizer θ ^ \widehat{\theta} θ . Let R 0 ≥ ∣ θ ( 0 ) − θ ^ ∣ 2 R_0\ge\lvert\theta^{(0)}-\widehat{\theta}\rvert_2 R 0 ≥ ∣ θ ( 0 ) − θ ∣ 2 , 0 < μ ≤ 2 L 0<\mu\le2L 0 < μ ≤ 2 L , κ = L / μ \kappa=L/\mu κ = L / μ , and 0 < η ≤ 1 / ( 2 L ) 0<\eta\le1/(2L) 0 < η ≤ 1/ ( 2 L ) . Suppose Condition 1 holds on B ( θ ^ , 2 max { R 0 , R } + 2 ϵ 1 ) \mathbb{B}(\widehat{\theta},2\max\{R_0,R\}+2\epsilon_1) B ( θ , 2 max { R 0 , R } + 2 ϵ 1 ) and Condition 2 holds on B ( θ ^ , R ) \mathbb{B}(\widehat{\theta},R) B ( θ , R ) , where R > 0 R>0 R > 0 and R ≥ max { 4 ϵ 2 , 16 ϵ 1 / ( η μ ) } R\ge\max\{4\epsilon_2,16\epsilon_1/(\eta\mu)\} R ≥ max { 4 ϵ 2 , 16 ϵ 1 / ( η μ )} . Let T 1 = ⌊ 8 R 0 2 / ( η μ R 2 ) ⌋ + 1 T_1=\lfloor8R_0^2/(\eta\mu R^2)\rfloor+1 T 1 = ⌊ 8 R 0 2 / ( η μ R 2 )⌋ + 1 .
(i) Subgradient descent. The iterates of Algorithm 2 satisfy ∣ θ ( t ) − θ ^ ∣ 2 ≤ max { R 0 , R } \lvert\theta^{(t)}-\widehat{\theta}\rvert_2\le\max\{R_0,R\} ∣ θ ( t ) − θ ∣ 2 ≤ max { R 0 , R } for every t ≥ 0 t\ge0 t ≥ 0 , and
∣ θ ( t ) − θ ^ ∣ 2 ≤ ( 1 − 3 η μ 4 ) ( t − T 1 ) / 2 R 2 + ϵ 2 2 + 4 η L 2 3 μ ϵ 1 2 , t ≥ T 1 . (8) \lvert\theta^{(t)}-\widehat{\theta}\rvert_2\le\left(1-\frac{3\eta\mu}{4}\right)^{(t-T_1)/2}\frac{R}{\sqrt{2}}+\sqrt{\epsilon_2^2+\frac{4\eta L^2}{3\mu}\epsilon_1^2},\qquad t\ge T_1. \tag*{(8)} ∣ θ ( t ) − θ ∣ 2 ≤ ( 1 − 4 3 η μ ) ( t − T 1 ) /2 2 R + ϵ 2 2 + 3 μ 4 η L 2 ϵ 1 2 , t ≥ T 1 . (8) (ii) Nesterov acceleration. Suppose additionally that R ≥ 64 ( ϵ 2 + κ ϵ 1 ) R\ge64(\epsilon_2+\kappa\epsilon_1) R ≥ 64 ( ϵ 2 + κ ϵ 1 ) . Let N 1 = max { 1 , ⌈ 512 / ( η μ ) R 0 / R ⌉ } N_1=\max\{1,\lceil\sqrt{512/(\eta\mu)}R_0/R\rceil\} N 1 = max { 1 , ⌈ 512/ ( η μ ) R 0 / R ⌉} . Run Algorithm 3 with the momentum sequence in Theorem 2(i) for at most N 1 N_1 N 1 updates, stopping before an update if η ∣ ∂ L ( β ( t ) ) ∣ 2 < ϵ 1 \eta\lvert\partial\mathcal{L}(\beta^{(t)})\rvert_2<\epsilon_1 η ∣ ∂ L ( β ( t ) ) ∣ 2 < ϵ 1 . Restart at this extrapolated point if the test is met, or at θ ( N 1 ) \theta^{(N_1)} θ ( N 1 ) otherwise, using step size η \eta η and constant momentum γ = ( 1 − η μ / 2 ) / ( 1 + η μ / 2 ) \gamma=(1-\sqrt{\eta\mu/2})/(1+\sqrt{\eta\mu/2}) γ = ( 1 − η μ /2 ) / ( 1 + η μ /2 ) . Indexing the restarted iterates from t = 0 t=0 t = 0 , they satisfy
∣ θ ( t ) − θ ^ ∣ 2 ≤ ( 1 − η μ 2 ) t / 2 R 3 + 2 ϵ 2 + 2 κ ( η μ 2 ) 1 / 4 ϵ 1 , t ≥ 0. (9) \lvert\theta^{(t)}-\widehat{\theta}\rvert_2\le\left(1-\sqrt{\frac{\eta\mu}{2}}\right)^{t/2}\frac{R}{3}+\sqrt{2}\epsilon_2+2\kappa\left(\frac{\eta\mu}{2}\right)^{1/4}\epsilon_1,\qquad t\ge0. \tag*{(9)} ∣ θ ( t ) − θ ∣ 2 ≤ ( 1 − 2 η μ ) t /2 3 R + 2 ϵ 2 + 2 κ ( 2 η μ ) 1/4 ϵ 1 , t ≥ 0. (9) Every Phase 1 iterate and extrapolated point lies in B ( θ ^ , R 0 ) \mathbb{B}(\widehat{\theta},R_0) B ( θ , R 0 ) . Every iterate and extrapolated point after the restart lies in B ( θ ^ , R ) \mathbb{B}(\widehat{\theta},R) B ( θ , R ) .
At η = 1 / ( 2 L ) \eta= 1/(2L) η = 1/ ( 2 L ) , parts (i) and (ii) give radii O ( ϵ 2 + κ 1 / 2 ϵ 1 ) O(\epsilon_2+\kappa^{1/2}\epsilon_1) O ( ϵ 2 + κ 1/2 ϵ 1 ) and O ( ϵ 2 + κ 3 / 4 ϵ 1 ) O(\epsilon_2+\kappa^{3/4}\epsilon_1) O ( ϵ 2 + κ 3/4 ϵ 1 ) , respectively. Acceleration improves the condition-number dependence of the iteration count from κ \kappa κ to κ \sqrt{\kappa} κ at the cost of a factor κ 1 / 4 \kappa^{1/4} κ 1/4 in the ϵ 1 \epsilon_1 ϵ 1 term. The ϵ 1 \epsilon_1 ϵ 1 term also follows from the accelerated inexact-oracle objective floor O ( κ δ ) O(\sqrt{\kappa}\delta) O ( κ δ ) in Devolder et al. [14 ] and quadratic growth when ϵ 2 = 0 \epsilon_2=0 ϵ 2 = 0 and δ = L ϵ 1 2 / 2 \delta=L\epsilon_1^2/2 δ = L ϵ 1 2 /2 .
The ϵ 1 \epsilon_1 ϵ 1 dependence in (8) is necessary even when ϵ 2 = 0 \epsilon_2=0 ϵ 2 = 0 . At η = 1 / ( 2 L ) \eta= 1/(2L) η = 1/ ( 2 L ) , Appendix § 8 gives an example with limiting distance at least ϵ 1 / 14 \epsilon_1/14 ϵ 1 /14 for every subgradient selection and Lebesgue-almost every initialization. The upper and lower radii therefore differ by at most a factor of order κ \sqrt{\kappa} κ .
Corollary 12 in Appendix § 8 extends the analysis of the BVGmax _{\max} m a x class of Diakonikolas and Guzmán [16 ] to geometric convergence under quadratic growth. Subgradient descent and restarted Nesterov acceleration attain the two rates above, to radii O ( κ r ) O(\sqrt{\kappa}r) O ( κ r ) and O ( κ 3 / 4 r ) O(\kappa^{3/4}r) O ( κ 3/4 r ) , where r r r is the variation scale. The lower bound makes the dependence on r r r sharp for subgradient descent when κ \kappa κ is bounded.
Subgradient increments and applications
The subgradient increment bound gives optimization rates and a sampling expansion for the computed solution. We first treat residual costs,
L ( θ ) = 1 n ∑ i = 1 n f ( Y i − Z i T θ ) + b T θ , θ ∈ Θ ⊆ R p , (10) \mathcal{L}(\theta)=\frac{1}{n}\sum_{i=1}^{n}f(Y_i-Z_i^{\mathsf{T}}\theta)+b^{\mathsf{T}}\theta,\qquad\theta\in\Theta\subseteq\mathbb{R}^{p}, \tag*{(10)} L ( θ ) = n 1 i = 1 ∑ n f ( Y i − Z i T θ ) + b T θ , θ ∈ Θ ⊆ R p , (10)
where ( Y i , Z i ) (Y_i,Z_i) ( Y i , Z i ) are independent copies of ( Y , Z ) (Y,Z) ( Y , Z ) and b b b is deterministic. For a reference decision θ r e f ∈ Θ \theta_{\mathrm{ref}}\in\Theta θ ref ∈ Θ , define L ‾ ( θ ) = E { f ( Y − Z T θ ) − f ( Y − Z T θ r e f ) } + b T ( θ − θ r e f ) \overline{\mathcal{L}}(\theta)=\mathbb{E}\{f(Y-Z^{\mathsf{T}}\theta)-f(Y-Z^{\mathsf{T}}\theta_{\mathrm{ref}})\}+b^{\mathsf{T}}(\theta-\theta_{\mathrm{ref}}) L ( θ ) = E { f ( Y − Z T θ ) − f ( Y − Z T θ ref )} + b T ( θ − θ ref ) . The difference inside the expectation is bounded in absolute value by C f ∣ Z T ( θ − θ r e f ) ∣ C_f|Z^{\mathsf{T}}(\theta-\theta_{\mathrm{ref}})| C f ∣ Z T ( θ − θ ref ) ∣ and is therefore integrable without a moment condition on Y Y Y . Reference subtraction preserves objective differences, subgradients, and updates.
Assumption 1. The decision region Θ \Theta Θ is convex and satisfies sup θ ∈ Θ ∣ θ ∣ 2 ≤ C Θ \sup_{\theta\in\Theta}|\theta|_2\leq C_\Theta sup θ ∈ Θ ∣ θ ∣ 2 ≤ C Θ .
Assumption 2. (i) The conditional distribution of Y Y Y given Z Z Z has a density ρ ( y ∣ Z ) \rho(y\mid Z) ρ ( y ∣ Z ) with sup y ρ ( y ∣ Z ) ≤ C ρ \sup_y\rho(y\mid Z)\leq C_\rho sup y ρ ( y ∣ Z ) ≤ C ρ almost surely.
(ii) The covariates satisfy sup ∣ v ∣ 2 = 1 ∥ v T Z ∥ ψ 2 ≤ C Z \sup_{|v|_2=1}\|v^{\mathsf{T}}Z\|_{\psi_2}\leq C_Z sup ∣ v ∣ 2 = 1 ∥ v T Z ∥ ψ 2 ≤ C Z .
Assumption 3. The function f f f is finite and convex on R \mathbb{R} R , and every subgradient of f f f has absolute value at most C f C_f C f .
We assume a n → 0 a_n\to0 a n → 0 . Assumption 3 includes piecewise linear, Huber, and smooth convex Lipschitz losses. Write g = f + ′ g=f'_+ g = f + ′ and let ν = d g \nu=dg ν = d g be its Stieltjes measure, with ν ( R ) ≤ 2 C f \nu(\mathbb{R})\leq2C_f ν ( R ) ≤ 2 C f .
Theorem 3. Suppose Assumptions 1–3 hold. For each deterministic θ 0 ∈ Θ \theta_0\in\Theta θ 0 ∈ Θ , there are C , γ > 0 C,\gamma>0 C , γ > 0 and an event of probability at least 1 − C n − γ p 1-Cn^{-\gamma p} 1 − C n − γ p on which
∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ‾ ( θ 1 ) + ∇ L ‾ ( θ 2 ) ∣ 2 ≤ C { ∣ θ 1 − θ 2 ∣ 2 a n + a n } (11) \left|\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)-\nabla\overline{\mathcal{L}}(\theta_1)+\nabla\overline{\mathcal{L}}(\theta_2)\right|_2\leq C\left\{\sqrt{|\theta_1-\theta_2|_2a_n}+a_n\right\} \tag*{(11)} ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ( θ 1 ) + ∇ L ( θ 2 ) 2 ≤ C { ∣ θ 1 − θ 2 ∣ 2 a n + a n } (11) for every θ 1 , θ 2 ∈ Θ \theta_1,\theta_2\in\Theta θ 1 , θ 2 ∈ Θ and every selection of scenario subgradients, and
∣ ∂ L ( θ 0 ) − ∇ L ‾ ( θ 0 ) ∣ 2 ≤ C a n . |\partial\mathcal{L}(\theta_0)-\nabla\overline{\mathcal{L}}(\theta_0)|_2\leq C\sqrt{a_n}. ∣ ∂ L ( θ 0 ) − ∇ L ( θ 0 ) ∣ 2 ≤ C a n .
The error decreases as the decisions approach each other. Uniformity over Θ \Theta Θ allows the same bound to follow the iterates from initialization to the region used for inference. At a common decision, (11) bounds the difference between any two subgradients by C a n Ca_n C a n , controlling differences between the scenario-subgradient choices used by an implementation. Here a n = p log n / n a_n = p\log n/n a n = p log n / n is the empirical complexity scale arising from the Bernstein and covering arguments in the proof of Theorem 3.
Remark 1. Uniform objective approximation alone does not identify the solution’s expansion. Even if a uniform error δ n ≍ p / n \delta_n \asymp\sqrt{p/n} δ n ≍ p / n is available, population strong convexity gives only an O { ( p / n ) 1 / 4 } O\{(p/n)^{1/4}\} O {( p / n ) 1/4 } minimizer bound, above the sampling scale p / n \sqrt{p/n} p / n . This square-root dependence is attained by φ ( x ) = x 2 / 2 \varphi(x)=x^2/2 φ ( x ) = x 2 /2 and φ a ( x ) = max { x 2 / 2 , a 2 / 2 − a x } \varphi_a(x)=\max\{x^2/2,a^2/2-ax\} φ a ( x ) = max { x 2 /2 , a 2 /2 − a x } , whose uniform difference is a 2 a^2 a 2 and whose minimizers differ by ( 2 − 1 ) a (\sqrt{2}-1)a ( 2 − 1 ) a for a > 0 a>0 a > 0 . For decisions θ 1 , θ 2 \boldsymbol{\theta}_1,\boldsymbol{\theta}_2 θ 1 , θ 2 at distance d = ∣ θ 1 − θ 2 ∣ 2 d=|\boldsymbol{\theta}_1-\boldsymbol{\theta}_2|_2 d = ∣ θ 1 − θ 2 ∣ 2 , the subgradient increment error is bounded by C { d a n + a n } C\{\sqrt{d a_n}+a_n\} C { d a n + a n } . This bound decreases to C a n Ca_n C a n as d d d shrinks, enabling the population Hessian approximation to identify the leading random term.
Corollary 2. Under the assumptions of Theorem 3, the following conclusions hold on its event.
The SAA objective satisfies Condition 1 on Θ \Theta Θ with L = L 0 L=L_0 L = L 0 and ϵ 1 = C 1 a n \epsilon_1=C_1a_n ϵ 1 = C 1 a n .
If L \mathcal L L is μ 0 \mu_0 μ 0 -strongly convex on a convex subset Θ s c ⊆ Θ \Theta_{\mathrm{sc}}\subseteq\Theta Θ sc ⊆ Θ , then Condition 2 holds for pairs in that subset with μ = μ 0 \mu=\mu_0 μ = μ 0 and ϵ 2 = C 2 a n \epsilon_2=C_2a_n ϵ 2 = C 2 a n .
Suppose the population Hessian is Lipschitz in operator norm on a neighborhood of θ ∗ \theta^* θ ∗ , as when ρ \rho ρ has a bounded continuous first derivative. If H = ∇ 2 L ( θ ∗ ) \mathbf H=\nabla^2\mathcal L(\theta^*) H = ∇ 2 L ( θ ∗ ) is positive definite, then, for every C 3 > 0 C_3>0 C 3 > 0 and all θ \theta θ in the ball ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 = C 3 a n |\theta-\theta^*|_2\leq\epsilon_3=C_3\sqrt{a_n} ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 = C 3 a n contained in that neighborhood and in Θ \Theta Θ ,
∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∣ 2 ≤ C 4 a n 3 / 4 . \left|\partial\mathcal L(\theta)-\partial\mathcal L(\theta^*)-\mathbf H(\theta-\theta^*)\right|_2\leq C_4a_n^{3/4}. ∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∣ 2 ≤ C 4 a n 3/4 . Thus Condition 3 holds with subgradient remainder r n = C 4 a n 3 / 4 r_n=C_4a_n^{3/4} r n = C 4 a n 3/4 .
A positive lower eigenvalue bound at θ ∗ \theta^* θ ∗ and the Hessian Lipschitz property give population strong convexity on a neighborhood of θ ∗ \theta^* θ ∗ . With ϵ 1 , ϵ 2 ≍ a n \epsilon_1,\epsilon_2\asymp a_n ϵ 1 , ϵ 2 ≍ a n , both radii in Corollary 1 are O ( a n ) O(a_n) O ( a n ) , below the sampling scale p / n \sqrt{p/n} p / n when p ( log n ) 2 = o ( n ) p(\log n)^2=o(n) p ( log n ) 2 = o ( n ) . Thus the extra κ 1 / 4 \kappa^{1/4} κ 1/4 factor from acceleration does not alter the first-order expansion in the next theorem.
Theorem 4. Suppose the assumptions of Theorem 3 hold, a n → 0 a_n\to0 a n → 0 , ρ \rho ρ has a bounded continuous first derivative, and θ ∗ \theta^* θ ∗ is an interior population minimizer with m H I ⪯ H = ∇ 2 L ( θ ∗ ) ⪯ M H I m_H\mathbf I\preceq\mathbf H=\nabla^2\mathcal L(\theta^*)\preceq M_H\mathbf I m H I ⪯ H = ∇ 2 L ( θ ∗ ) ⪯ M H I , where 0 < m H ≤ M H < ∞ 0<m_H\leq M_H<\infty 0 < m H ≤ M H < ∞ . Use L = L 0 L=L_0 L = L 0 , μ = m H / 2 \mu=m_H/2 μ = m H /2 , ϵ j = C j a n \epsilon_j=C_ja_n ϵ j = C j a n for j = 1 , 2 j=1,2 j = 1 , 2 , and 0 < η ≤ 1 / ( 2 L ) 0<\eta\leq1/(2L) 0 < η ≤ 1/ ( 2 L ) . Let Θ \Theta Θ be closed with B ( θ ∗ , 2 R ) ⊆ Θ B(\theta^*,2R)\subseteq\Theta B ( θ ∗ , 2 R ) ⊆ Θ for some R > 0 R>0 R > 0 . From any θ ( 0 ) ∈ Θ \theta^{(0)}\in\Theta θ ( 0 ) ∈ Θ , use either projected procedure in Corollary 10(iii)(b). Let θ ( t ) \theta^{(t)} θ ( t ) denote the iterates, counting all phases.
(i) There are c , C , γ > 0 c,C,\gamma>0 c , C , γ > 0 such that, for sufficiently large n n n , there is an event of probability at least 1 − C n − γ p 1-Cn^{-\gamma p} 1 − C n − γ p on which an SAA solution θ ^ \hat\theta θ ^ exists and the following bounds hold simultaneously over θ ( 0 ) ∈ Θ \theta^{(0)}\in\Theta θ ( 0 ) ∈ Θ and all selections of scenario subgradients.
sup t ≥ ⌈ c log n ⌉ ∣ θ ( t ) − θ ^ ∣ 2 ≤ C a n , max { L ‾ ( θ ^ ) − L ‾ ( θ ∗ ) , sup t ≥ ⌈ c log n ⌉ { L ‾ ( θ ( t ) ) − L ‾ ( θ ∗ ) } } ≤ C a n . \sup_{t\geq\lceil c\log n\rceil}|\theta^{(t)}-\hat\theta|_2\leq Ca_n,\qquad
\max\left\{\overline{\mathcal L}(\hat\theta)-\overline{\mathcal L}(\theta^*),\ \sup_{t\geq\lceil c\log n\rceil}\left\{\overline{\mathcal L}(\theta^{(t)})-\overline{\mathcal L}(\theta^*)\right\}\right\}\leq Ca_n. t ≥ ⌈ c l o g n ⌉ sup ∣ θ ( t ) − θ ^ ∣ 2 ≤ C a n , max { L ( θ ^ ) − L ( θ ∗ ) , t ≥ ⌈ c l o g n ⌉ sup { L ( θ ( t ) ) − L ( θ ∗ ) } } ≤ C a n . (ii) Write ψ i = b − Z i f + ′ ( Y i − Z i T θ ∗ ) \psi_i=\mathbf b-\mathbf Z_i f'_+(Y_i-\mathbf Z_i^\mathsf{T}\theta^*) ψ i = b − Z i f + ′ ( Y i − Z i T θ ∗ ) . Every output θ ~ = θ ( T ) \tilde\theta=\theta^{(T)} θ ~ = θ ( T ) with T ≥ ⌈ c log n ⌉ T\geq\lceil c\log n\rceil T ≥ ⌈ c log n ⌉ satisfies
θ ~ − θ ∗ = − H − 1 1 n ∑ i = 1 n ψ i + O P { ( p / n ) 3 / 4 log n + a n } . \tilde\theta-\theta^*=-\mathbf H^{-1}\frac{1}{n}\sum_{i=1}^{n}\psi_i+O_{\mathbb P}\left\{(p/n)^{3/4}\sqrt{\log n}+a_n\right\}. θ ~ − θ ∗ = − H − 1 n 1 i = 1 ∑ n ψ i + O P { ( p / n ) 3/4 log n + a n } .
If p log n = o ( n ) p\log n=o(\sqrt{n}) p log n = o ( n ) , then n ∣ θ ~ − θ ^ ∣ 2 = o P ( 1 ) \sqrt{n}\lvert\widetilde{\theta}-\widehat{\theta}\rvert_{2}=o_{P}(1) n ∣ θ − θ ∣ 2 = o P ( 1 ) .
(iii) For a deterministic unit vector v n \mathbf{v}_{n} v n , let σ n 2 = Var ( v n T H − 1 ψ i ) \sigma_{n}^{2}=\operatorname{Var}(\mathbf{v}_{n}^{\mathsf{T}}\mathbf{H}^{-1}\psi_{i}) σ n 2 = Var ( v n T H − 1 ψ i ) . If inf n σ n > 0 \inf_{n}\sigma_{n}>0 inf n σ n > 0 and p = o { n 1 / 3 ( log n ) − 2 / 3 } p=o\{n^{1/3}(\log n)^{-2/3}\} p = o { n 1/3 ( log n ) − 2/3 } , then
n v n T ( θ ~ − θ ∗ ) σ n → d N ( 0 , 1 ) , n { L ‾ ( θ ~ ) − L ‾ ( θ ^ ) } = O P ( p 3 / 2 log n / n ) = o P ( 1 ) . \frac{\sqrt{n}\mathbf{v}_{n}^{\mathsf{T}}(\widetilde{\theta}-\theta^{*})}{\sigma_{n}}\xrightarrow{d}\mathcal{N}(0,1),\qquad n\{\overline{L}(\widetilde{\theta})-\overline{L}(\widehat{\theta})\}=O_{P}(p^{3/2}\log n/\sqrt{n})=o_{P}(1). σ n n v n T ( θ − θ ∗ ) d N ( 0 , 1 ) , n { L ( θ ) − L ( θ )} = O P ( p 3/2 log n / n ) = o P ( 1 ) .
Appendix A.3 gives the unprojected extension of Theorem 4. Under their respective region and initialization requirements, the complete procedures there and the interior switches in Corollary 10(iii) attain the same expansion, normal limit, and population-gap conclusions, with Phase 3 counts ⌈ c ( η 3 m H ) − 1 log n ⌉ \lceil c(\eta_{3}m_{H})^{-1}\log n\rceil ⌈ c ( η 3 m H ) − 1 log n ⌉ for subgradient descent and ⌈ c ( η N m H ) − 1 / 2 log n ⌉ \lceil c(\eta_{N}m_{H})^{-1/2}\log n\rceil ⌈ c ( η N m H ) − 1/2 log n ⌉ for Nesterov acceleration, where η 3 \eta_{3} η 3 and η N \eta_{N} η N are the Phase 3 step sizes. Corollary 13 (ii) in Appendix B.6 treats directional sub-Weibull tails and preserves the dimension exponent in the normal limit, with additional logarithmic factors.
For 0 < ε ≤ 1 0<\varepsilon\leq1 0 < ε ≤ 1 and sufficiently large n n n , Theorem 4(i) gives L ‾ ( θ ( T ) ) − L ‾ ( θ ∗ ) ≤ ε \overline{L}(\theta^{(T)})-\overline{L}(\theta^{*})\leq\varepsilon L ( θ ( T ) ) − L ( θ ∗ ) ≤ ε with probability at least 1 − C n − γ p 1-Cn^{-\gamma p} 1 − C n − γ p when n ≥ C ( p / ε ) log ( e p / ε ) n\geq C(p/\varepsilon)\log(ep/\varepsilon) n ≥ C ( p / ε ) log ( e p / ε ) and T ≥ ⌈ c log n ⌉ T\geq\lceil c\log n\rceil T ≥ ⌈ c log n ⌉ .
Remark 2. When p ( log n ) 2 = o ( n ) p(\log n)^{2}=o(n) p ( log n ) 2 = o ( n ) , further iterations after ⌈ c log n ⌉ \lceil c\log n\rceil ⌈ c log n ⌉ updates do not improve the first-order statistical accuracy of the computed solution. Theorem 4 confines every later iterate to an O ( a n ) O(a_{n}) O ( a n ) neighborhood of the SAA solution, below both the sampling scale and the expansion remainder order, for every subgradient selection. Additional updates therefore retain the same leading sampling error, as well as the normal limit and population-gap equivalence under part (iii). The pinball trajectories in Proposition 5 show that fluctuations persist, but they need not vanish to attain SAA accuracy.
Proposition 1. Consider (10) with f ( x ) = c x { τ − I ( x ≤ 0 ) } f(x)=cx\{\tau-\mathbb{I}(x\leq0)\} f ( x ) = c x { τ − I ( x ≤ 0 )} , where c > 0 c>0 c > 0 and τ ∈ ( 0 , 1 ] \tau\in(0,1] τ ∈ ( 0 , 1 ] . Suppose the matrix with rows Z i T \mathbf{Z}_{i}^{\mathsf{T}} Z i T has rank p p p , there is an index l l l with Z i l = 1 Z_{il}=1 Z i l = 1 for every i i i , and L \mathcal{L} L attains its minimum on R p \mathbb{R}^{p} R p . There is a minimizer θ ^ \widehat{\theta} θ such that every pair ( L , ϵ 1 ) (L,\epsilon_{1}) ( L , ϵ 1 ) satisfying Condition 1 on a neighborhood of θ ^ \widehat{\theta} θ obeys
L ϵ 1 ≥ c p n . L_{\epsilon_{1}}\geq\frac{cp}{n}. L ϵ 1 ≥ n c p .
Thus, for L L L of constant order, the scale p log n / n p\log n/n p log n / n differs from the necessary scale in this class by at most a logarithmic factor. For pinball SAA, Proposition 5 in Appendix B gives constant-step trajectory lower bounds simultaneously for every scenario-subgradient selection.
The subgradient increment bound also applies to scenario costs whose active affine piece changes with the decision.
Corollary 3. Let Θ ⊆ R p \Theta\subseteq\mathbb{R}^{p} Θ ⊆ R p be convex and bounded, ξ \xi ξ be a random scenario whose cost has K K K affine pieces with intercepts α k ( ξ ) \alpha_{k}(\xi) α k ( ξ ) and slopes β k ( ξ ) ∈ R p \beta_{k}(\xi)\in\mathbb{R}^{p} β k ( ξ ) ∈ R p , and let ξ 1 , … , ξ n \xi_{1},\ldots,\xi_{n} ξ 1 , … , ξ n be independent copies of ξ \xi ξ with cost
Q ( θ , ξ ) = max 1 ≤ k ≤ K { α k ( ξ ) + β k T ( ξ ) θ } , L ( θ ) = 1 n ∑ i = 1 n Q ( θ , ξ i ) + b T θ . Q(\theta,\xi)=\max_{1\leq k\leq K}\{\alpha_{k}(\xi)+\beta_{k}^{\mathsf{T}}(\xi)\theta\},\qquad\mathcal{L}(\theta)=\frac{1}{n}\sum_{i=1}^{n}Q(\theta,\xi_{i})+\mathbf{b}^{\mathsf{T}}\theta. Q ( θ , ξ ) = 1 ≤ k ≤ K max { α k ( ξ ) + β k T ( ξ ) θ } , L ( θ ) = n 1 i = 1 ∑ n Q ( θ , ξ i ) + b T θ . Suppose sup ∣ v ∣ 2 = 1 ∥ v T β k ∥ ψ 2 ≤ C β \sup_{\lvert v\rvert_{2}=1}\lVert\mathbf{v}^{\mathsf{T}}\boldsymbol{\beta}_{k}\rVert_{\psi_{2}}\leq C_{\beta} sup ∣ v ∣ 2 = 1 ∥ v T β k ∥ ψ 2 ≤ C β and, for each j < k j<k j < k , the difference α j − α k \alpha_{j}-\alpha_{k} α j − α k conditional on β j − β k \beta_{j}-\beta_{k} β j − β k has a density bounded by C ρ C_{\rho} C ρ . Define L ‾ \overline{L} L through expected loss differences. Then the increment and pointwise bounds in Theorem 3 and parts 1 and 2 of Corollary 2 hold, with constants depending on K K K , for every selection from the convex hull of active slopes. If the population Hessian is Lipschitz near an interior minimizer θ ∗ \theta^{*} θ ∗ and m H I ⪯ ∇ 2 L ‾ ( θ ∗ ) ⪯ M H I m_{H}\mathbf{I}\preceq\nabla^{2}\overline{L}(\theta^{*})\preceq M_{H}\mathbf{I} m H I ⪯ ∇ 2 L ( θ ∗ ) ⪯ M H I , the procedures in Theorem 4 and Appendix A.3, with their region and initialization requirements, give all conclusions of Theorem 4 with score ψ i = b + β k i ∗ ( ξ i ) \psi_{i}=\mathbf{b}+\beta_{k_{i}^{*}}(\xi_{i}) ψ i = b + β k i ∗ ( ξ i ) , where k i ∗ k_{i}^{*} k i ∗ maximizes the affine pieces at θ ∗ \theta^{*} θ ∗ .
For a two-stage stochastic linear program with fixed recourse matrix W W W and cost q q q , the recourse value is
Q ( θ , ξ ) = min { q T y : W y = d ( ξ ) − T ( ξ ) θ , y ≥ 0 } . Q(\theta,\xi)=\min\{q^\mathsf{T}y: Wy=d(\xi)-T(\xi)\theta,\ y\geq0\}. Q ( θ , ξ ) = min { q T y : W y = d ( ξ ) − T ( ξ ) θ , y ≥ 0 } .
If this value is finite and the dual feasible polytope has vertices π 1 , … , π K \pi_1,\ldots,\pi_K π 1 , … , π K , linear programming duality gives α k = π k T d ( ξ ) \alpha_k=\pi_k^\mathsf{T}d(\xi) α k = π k T d ( ξ ) and β k = − T ( ξ ) T π k \beta_k=-T(\xi)^\mathsf{T}\pi_k β k = − T ( ξ ) T π k . The corollary then gives the increment bound and optimization rates from density conditions on the scenario differences. The constants depend on the number of dual vertices.
Applications
The examples below give ordering rules, hedging, and classification. Appendices B.5 and C.4 treat rank regression.
Example 1. A decision maker uses demand scenarios to compute a linear ordering rule. For demand D D D and features Z Z Z , consider the rule Z T β Z^\mathsf{T}\beta Z T β . For underage and overage costs c u , c o > 0 c_u,c_o>0 c u , c o > 0 , the SAA objective is
L N V ( β ) = 1 n ∑ i = 1 n [ c u ( D i − Z i T β ) + + c o ( Z i T β − D i ) + ] . \mathcal{L}_{\mathrm{NV}}(\beta)=\frac{1}{n}\sum_{i=1}^{n}\left[c_u(D_i-Z_i^\mathsf{T}\beta)_+ + c_o(Z_i^\mathsf{T}\beta-D_i)_+\right]. L NV ( β ) = n 1 i = 1 ∑ n [ c u ( D i − Z i T β ) + + c o ( Z i T β − D i ) + ] . This includes the decision rules in Ban and Rudin [2 ] and the sparse inventory policies in Chang et al. [7 ] . Its scenario cost equals ( c u + c o ) (c_u+c_o) ( c u + c o ) times the pinball loss used in quantile regression, f ( x ) = x { τ − I ( x ≤ 0 ) } f(x)=x\{\tau-\mathbb{I}(x\leq0)\} f ( x ) = x { τ − I ( x ≤ 0 )} , with τ = c u / ( c u + c o ) \tau=c_u/(c_u+c_o) τ = c u / ( c u + c o ) [30 ] . The population solution minimizes expected cost differences over linear ordering rules, including when the conditional demand quantile is nonlinear.
Suppose Z Z Z satisfies Assumption 2(ii), and the conditional demand density is bounded with a bounded first derivative. At an interior population solution β ∗ \beta^* β ∗ ,
H = ( c u + c o ) E [ ρ D ( Z T β ∗ ∣ Z ) Z Z T ] . H=(c_u+c_o)\mathbb{E}[\rho_D(Z^\mathsf{T}\beta^*\mid Z)ZZ^\mathsf{T}]. H = ( c u + c o ) E [ ρ D ( Z T β ∗ ∣ Z ) Z Z T ] . Positive lower bounds on this density and on Λ min { E [ Z Z T ] } \Lambda_{\min}\{\mathbb{E}[ZZ^\mathsf{T}]\} Λ m i n { E [ Z Z T ]} give the Hessian condition in Theorem 4. Starting from any feasible ordering rule in a closed convex decision region containing a ball about β ∗ \beta^* β ∗ , the projected methods attain the SAA expansion and normal limit after O ( log n ) O(\log n) O ( log n ) updates. Evaluating scenario subgradients costs O ( n p log n ) O(np\log n) O ( n p log n ) operations in total, nearly linear in n n n , without requiring an exact solution of the SAA linear program. Their later iterates remain within C α n C\alpha_n C α n of the SAA solution. Nonnegativity and capacity constraints are therefore inactive at those iterates.
Example 2. Let B B B be an exogenous loss with E B + < ∞ \mathbb{E}B_+<\infty E B + < ∞ and Z Z Z a vector of hedge payoffs. Positions w w w leave a loss B − Z T w B-Z^\mathsf{T}w B − Z T w . At confidence level α C V a R ∈ ( 0 , 1 ) \alpha_{\mathrm{CVaR}}\in(0,1) α CVaR ∈ ( 0 , 1 ) , the Rockafellar–Uryasev SAA objective is [47 ]
L C V a R ( a , w ) = a + 1 n ( 1 − α C V a R ) ∑ i = 1 n ( B i − Z i T w − a ) + . \mathcal{L}_{\mathrm{CVaR}}(a,w)=a+\frac{1}{n(1-\alpha_{\mathrm{CVaR}})}\sum_{i=1}^{n}(B_i-Z_i^\mathsf{T}w-a)_+. L CVaR ( a , w ) = a + n ( 1 − α CVaR ) 1 i = 1 ∑ n ( B i − Z i T w − a ) + . Minimizing this objective gives the hedge and its risk threshold. It has the form (10) with response B B B , covariate vector V = ( Z T , 1 ) T V=(Z^\mathsf{T},1)^\mathsf{T} V = ( Z T , 1 ) T , decision ( a , w ) (a,w) ( a , w ) , and affine coefficient ( 0 T , 1 ) T (0^\mathsf{T},1)^\mathsf{T} ( 0 T , 1 ) T . The affine term cancels in subgradient increments.
Suppose V V V satisfies the directional sub-Gaussian bound and the conditional density of B B B given Z Z Z is bounded with a bounded first derivative. At an interior population solution ( a ∗ , w ∗ ) (a^*,w^*) ( a ∗ , w ∗ ) ,
H C V a R = 1 1 − α C V a R E [ ρ B ( Z T w ∗ + a ∗ ∣ Z ) V V T ] . H_{\mathrm{CVaR}}=\frac{1}{1-\alpha_{\mathrm{CVaR}}}\mathbb{E}[\rho_B(Z^\mathsf{T}w^*+a^*\mid Z)VV^\mathsf{T}]. H CVaR = 1 − α CVaR 1 E [ ρ B ( Z T w ∗ + a ∗ ∣ Z ) V V T ] .
Positive lower bounds on this density and on Λ min { E [ V V T ] } \Lambda_{\min}\{\mathbb{E}[\mathbf{V}\mathbf{V}^{\mathsf{T}}]\} Λ m i n { E [ V V T ]} give the Hessian condition in Theorem 4, whose score here is ψ i = ( 0 T , 1 ) T − I ( B i > Z i T w ∗ + a ∗ ) V i / ( 1 − α C V a R ) \boldsymbol{\psi}_i=(\mathbf{0}^{\mathsf{T}},1)^{\mathsf{T}}-\mathbb{I}(B_i>\mathbf{Z}_i^{\mathsf{T}}\mathbf{w}^{*}+a^{*})\mathbf{V}_i/(1-\alpha_{\mathrm{CVaR}}) ψ i = ( 0 T , 1 ) T − I ( B i > Z i T w ∗ + a ∗ ) V i / ( 1 − α CVaR ) . After O ( log n ) O(\log n) O ( log n ) updates, the computed hedge and threshold have the same limiting distribution as the SAA decision, with covariance H C V a R − 1 Var ( ψ i ) H C V a R − 1 \mathbf{H}_{\mathrm{CVaR}}^{-1}\operatorname{Var}(\boldsymbol{\psi}_i)\mathbf{H}_{\mathrm{CVaR}}^{-1} H CVaR − 1 Var ( ψ i ) H CVaR − 1 in fixed dimension. For growing dimension, Theorem 4 gives the normal limit of decision contrasts. The conditional expected-shortfall formulation in He et al. [23 ] provides a related regression approach to tail-risk decisions.
Example 3. For a class label Y ∈ { − 1 , 1 } Y\in\{-1,1\} Y ∈ { − 1 , 1 } and covariates Z \mathbf{Z} Z , the linear support vector machine uses the hinge objective [31 ]
L S V M ( θ ) = 1 n ∑ i = 1 n ( 1 − X i T θ ) + , X i = Y i Z i . L_{\mathrm{SVM}}(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=1}^{n}(1-\mathbf{X}_i^{\mathsf{T}}\boldsymbol{\theta})_{+},\quad\mathbf{X}_i=Y_i\mathbf{Z}_i. L SVM ( θ ) = n 1 i = 1 ∑ n ( 1 − X i T θ ) + , X i = Y i Z i . This is an index objective n − 1 ∑ i f ( X i T θ ) n^{-1}\sum_i f(\mathbf{X}_i^{\mathsf{T}}\boldsymbol{\theta}) n − 1 ∑ i f ( X i T θ ) with f ( t ) = ( 1 − t ) + f(t)=(1-t)_{+} f ( t ) = ( 1 − t ) + and every coordinate of θ \boldsymbol{\theta} θ varying. On a bounded convex region with ∣ θ 1 ∣ ≥ c 0 > 0 |\theta_1|\ge c_0>0 ∣ θ 1 ∣ ≥ c 0 > 0 , Proposition 6 in Appendix B.3 gives the increment bound when X 1 \mathbf{X}_1 X 1 given X − 1 \mathbf{X}_{-1} X − 1 has a bounded density and X \mathbf{X} X satisfies the directional sub-Gaussian bound, since ν = δ 1 \boldsymbol{\nu}=\boldsymbol{\delta}_1 ν = δ 1 has compact support. Under the Hessian and initialization conditions of Theorem 4 or its unprojected extension, the expansion holds after O ( log n ) O(\log n) O ( log n ) updates with ψ i = − X i I ( X i T θ ∗ < 1 ) \boldsymbol{\psi}_i=-\mathbf{X}_i\mathbb{I}(\mathbf{X}_i^{\mathsf{T}}\boldsymbol{\theta}^{*}<1) ψ i = − X i I ( X i T θ ∗ < 1 ) , and the normal limit holds under its dimension and variance conditions.
Smoothing
Direct subgradient updates already yield the SAA expansion without smoothing. Proposition 5 gives a lower bound on persistent fluctuations of nonsmooth iterates. Smoothing sets ϵ 1 = ϵ 2 = 0 \epsilon_1=\epsilon_2=0 ϵ 1 = ϵ 2 = 0 , and Corollary 1 then gives geometric convergence to the smoothed SAA solution. Its effect on inference depends on the population displacement. Infimal convolution has displacement O ( h ) O(h) O ( h ) , with a leading coefficient determined by the prox-function. An explicit shift of the residual removes its first-order term. The expansion remainder then improves, with the limiting distribution preserved. The corresponding results for integral convolution, together with its definition and relation to existing inference methods, are given in Appendix C.
Convolution and regularity
For a strongly convex prox-function ω \omega ω , define the scalar infimal convolution
f h ( x ) = inf y ∈ R { f ( y ) + h ω ( ( x − y ) / h ) } . (12) f_h(x)=\inf_{y\in\mathbb{R}}\{f(y)+h\omega((x-y)/h)\}. \tag*{(12)} f h ( x ) = y ∈ R inf { f ( y ) + hω (( x − y ) / h )} . (12)
The choice ω ( u ) = u 2 / 2 \omega(u)=u^2/2 ω ( u ) = u 2 /2 gives the Moreau envelope Moreau, [34 ] . The Fenchel-dual representation of (12) is a scalar instance of Nesterov smoothing Nesterov, [38 ] . We analyze both the regularity and the population displacement for general prox-functions ω \omega ω , including nonquadratic choices. Write
L h ( θ ) = 1 n ∑ i = 1 n f h ( Y i − Z i T θ ) + b T θ . L_h(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=1}^{n}f_h(Y_i-\mathbf{Z}_i^{\mathsf{T}}\boldsymbol{\theta})+\mathbf{b}^{\mathsf{T}}\boldsymbol{\theta}. L h ( θ ) = n 1 i = 1 ∑ n f h ( Y i − Z i T θ ) + b T θ .
and let L ‾ h \overline{L}_h L h be its expected objective, defined through loss differences as in Section 4. Each update evaluates a scalar proximal map, which is explicit for piecewise linear losses.
Assumption 4. The prox-function ω \omega ω is twice continuously differentiable, with second derivative bounded above and bounded away from zero, and ω ′ ′ \omega'' ω ′′ is Lipschitz continuous. The function f f f is C 3 C^3 C 3 on the intervals separated by a finite number of breakpoints, with second and third derivatives bounded on those intervals. The conditional density in Assumption 2(i) is continuously differentiable with bounded first derivative.
Assumption 4 includes piecewise linear and Huber losses. The convolution acts on a scalar scenario loss and remains one-dimensional as p p p grows, whereas Diakonikolas and Guzmán [16 ] smooth the p p p -dimensional decision vector.
Theorem 5. Suppose Assumptions 1–4 hold. Let A a n ≤ h ≤ h 0 A a_n \le h \le h_0 A a n ≤ h ≤ h 0 , for sufficiently large A A A and sufficiently small h 0 h_0 h 0 . With probability at least 1 − C n − γ p 1-Cn^{-\gamma p} 1 − C n − γ p ,
∥ ∇ L h ( θ 1 ) − ∇ L h ( θ 2 ) − ∇ L ‾ h ( θ 1 ) + ∇ L ‾ h ( θ 2 ) ∥ 2 ≤ C min { ∣ θ 1 − θ 2 ∣ 2 a n , ∣ θ 1 − θ 2 ∣ 2 a n / h } , \left\|\nabla L_h(\theta_1)-\nabla L_h(\theta_2)-\nabla\overline{L}_h(\theta_1)+\nabla\overline{L}_h(\theta_2)\right\|_2
\le C\min\left\{\sqrt{\lvert\theta_1-\theta_2\rvert_2a_n},\lvert\theta_1-\theta_2\rvert_2\sqrt{a_n/h}\right\}, ∇ L h ( θ 1 ) − ∇ L h ( θ 2 ) − ∇ L h ( θ 1 ) + ∇ L h ( θ 2 ) 2 ≤ C min { ∣ θ 1 − θ 2 ∣ 2 a n , ∣ θ 1 − θ 2 ∣ 2 a n / h } , uniformly over $\theta_1,\theta_2\in\Theta.
The bound is linear in distance below the bandwidth and retains the square-root bound above it. Lemma 9 in Appendix C bounds the Hessian fluctuation by O ( a n / h + a n / h ) O(\sqrt{a_n/h}+a_n/h) O ( a n / h + a n / h ) .
Corollary 4. Under the assumptions of Theorem 5, the following hold with probability at least 1 − C n − γ p 1-Cn^{-\gamma p} 1 − C n − γ p .
Condition 1 holds for L h L_h L h with ϵ 1 = 0 \epsilon_1=0 ϵ 1 = 0 and a constant L 0 L_0 L 0 .
If L ‾ \overline{L} L is μ 0 \mu_0 μ 0 -strongly convex on a convex set Θ s c ⊆ Θ \Theta_{\mathrm{sc}}\subseteq\Theta Θ sc ⊆ Θ , then Condition 2 holds for L h L_h L h on that set with μ = μ 0 \mu=\mu_0 μ = μ 0 and ϵ 2 = 0 \epsilon_2=0 ϵ 2 = 0 .
Suppose H = ∇ 2 L ‾ ( θ ∗ ) H=\nabla^2\overline{L}(\theta^*) H = ∇ 2 L ( θ ∗ ) is positive definite and B ( θ ∗ , C 3 a n ) ⊆ Θ B(\theta^*,C_3\sqrt{a_n})\subseteq\Theta B ( θ ∗ , C 3 a n ) ⊆ Θ . Then uniformly on that ball,
∥ ∇ L h ( θ ) − ∇ L ‾ h ( θ ∗ ) − H ( θ − θ ∗ ) ∥ 2 ≤ C 4 { min ( a n / h , a n 3 / 4 ) + h a n } . (13) \left\|\nabla L_h(\theta)-\nabla\overline{L}_h(\theta^*)-H(\theta-\theta^*)\right\|_2
\le C_4\left\{\min\left(a_n/\sqrt{h},a_n^{3/4}\right)+h\sqrt{a_n}\right\}. \tag*{(13)} ∇ L h ( θ ) − ∇ L h ( θ ∗ ) − H ( θ − θ ∗ ) 2 ≤ C 4 { min ( a n / h , a n 3/4 ) + h a n } . (13)
The density derivative bound gives the population Hessian Lipschitz property used in part 3.
The O ( 1 / h ) O(1/h) O ( 1/ h ) smoothness bound for an individual smoothed loss also enters homotopy smoothing methods (Xu et al., 2016). Combined with an O ( h ) O(h) O ( h ) function approximation error, it gives the accelerated bound O ( h ) + O ( 1 / ( h T 2 ) ) O(h)+O(1/(hT^2)) O ( h ) + O ( 1/ ( h T 2 )) , which becomes O ( T − 1 ) O(T^{-1}) O ( T − 1 ) after balancing h h h with T T T (Beck and Teboulle, [4 ] ). Under sample averaging, the smoothed SAA objective instead permits the Phase 1 optimization rate O ( T − 2 ) O(T^{-2}) O ( T − 2 ) at each bandwidth h ≥ A a n h\ge Aa_n h ≥ A a n , with a step size independent of h h h .
The threshold is necessary up to a logarithmic factor. For infimal convolution of the pinball loss, the smoothed second derivative equals ω ′ ′ ( x / h ) / h \omega''(x/h)/h ω ′′ ( x / h ) / h within its transition interval. An intercept shift of size O ( h ) O(h) O ( h ) from the minimizer in Proposition 1 places at least p p p residuals inside this interval. A smoothness bound of constant order on a fixed neighborhood of that minimizer therefore requires h ≳ p / n h\gtrsim p/n h ≳ p / n , as shown in Appendix C.1.
Population displacement and inference
Smoothing can displace the population minimizer. For infimal convolution the displacement is O ( h ) O(h) O ( h ) , and Proposition 2 identifies its leading term. Uniform function approximation bounds control objective values in smoothing analyses (Beck and Teboulle, [4 ] ), whereas inference for θ ∗ \theta^* θ ∗ depends on this displacement.
For a loss with one breakpoint k k k and affine pieces of slopes a < b a < b a < b , let ω ∗ \omega^* ω ∗ be the convex conjugate of ω \omega ω and define
c ω = ω ∗ ( b ) − ω ∗ ( a ) b − a , B ω = − ( b − a ) c ω H − 1 E [ Z ρ ( k + Z T θ ∗ ∣ Z ) ] . c_\omega= \frac{\omega^*(b)-\omega^*(a)}{b-a}, \qquad B_\omega= -(b-a)c_\omega H^{-1}\mathbb{E}[Z\rho(k+Z^{\mathsf T}\theta^* \mid Z)]. c ω = b − a ω ∗ ( b ) − ω ∗ ( a ) , B ω = − ( b − a ) c ω H − 1 E [ Z ρ ( k + Z T θ ∗ ∣ Z )] .
Proposition 2. Suppose the model and population assumptions of Theorem 4 and Assumption 4 hold, and f f f has one breakpoint k k k , with affine pieces of slopes a < b a < b a < b . For sufficiently small h h h , the infimal population minimizer is unique, lies in the interior of Θ \Theta Θ , and satisfies
θ h ∗ − θ ∗ = h B ω + O ( h 2 ) . \theta_h^*-\theta^*=hB_\omega+O(h^2). θ h ∗ − θ ∗ = h B ω + O ( h 2 ) .
For the losses in Proposition 2, define the shifted objective
L h s h ( θ ) = 1 n ∑ i = 1 n f h ( Y i − Z i T θ + h c ω ) + b T θ . (14) \mathcal{L}_h^{\mathrm{sh}}(\theta)=\frac{1}{n}\sum_{i=1}^{n}f_h(Y_i-Z_i^{\mathsf T}\theta+hc_\omega)+\mathbf{b}^{\mathsf T}\theta. \tag*{(14)} L h sh ( θ ) = n 1 i = 1 ∑ n f h ( Y i − Z i T θ + h c ω ) + b T θ . (14)
Let U 0 U_0 U 0 be uniform on [ a , b ] [a,b] [ a , b ] , independently of ( Y , Z ) (Y,Z) ( Y , Z ) , and put W = ( ω ′ ) − 1 ( U 0 ) W=(\omega')^{-1}(U_0) W = ( ω ′ ) − 1 ( U 0 ) . The following identity explains how the residual shift removes the first-order displacement.
Proposition 3. Under the assumptions of Proposition 2, E W = c ω \mathbb{E}W=c_\omega E W = c ω and, for every r ∈ R r\in\mathbb{R} r ∈ R and h > 0 h>0 h > 0 ,
f h ( r + h c ω ) = E [ f { r − h ( W − c ω ) } ] + h { a c ω − ω ∗ ( a ) } . (15) f_h(r+hc_\omega)=\mathbb{E}[f\{r-h(W-c_\omega)\}]+h\{ac_\omega-\omega^*(a)\}. \tag*{(15)} f h ( r + h c ω ) = E [ f { r − h ( W − c ω )}] + h { a c ω − ω ∗ ( a )} . (15) For sufficiently small h h h , the expected objective L h s h \mathcal{L}_h^{\mathrm{sh}} L h sh has a unique minimizer θ h ∗ , s h \theta_h^{*,\mathrm{sh}} θ h ∗ , sh in the interior of Θ \Theta Θ , with
∣ θ h ∗ , s h − θ ∗ ∣ 2 ≤ C h 2 . \lvert\theta_h^{*,\mathrm{sh}}-\theta^*\rvert_2\le Ch^2. ∣ θ h ∗ , sh − θ ∗ ∣ 2 ≤ C h 2 . For ω ( u ) = u 2 / 2 \omega(u)=u^2/2 ω ( u ) = u 2 /2 , the variable W − c ω W-c_\omega W − c ω is uniform on [ − ( b − a ) / 2 , ( b − a ) / 2 ] [-(b-a)/2,(b-a)/2] [ − ( b − a ) /2 , ( b − a ) /2 ] , and the additive constant in (15) is h a b / 2 hab/2 hab /2 .
The perturbation W − c ω W-c_\omega W − c ω has mean zero, which removes the first-order term in the population gradient. For Moreau smoothing, the shifted loss therefore has the same gradient as integral convolution with the uniform kernel on [ − ( b − a ) / 2 , ( b − a ) / 2 ] [-(b-a)/2,(b-a)/2] [ − ( b − a ) /2 , ( b − a ) /2 ] . The scalar proximal formula gives this gradient without computing a kernel integral.
Remark 3. Without the residual shift, infimal smoothing can alter first-order inference for the original solution θ ∗ \theta^* θ ∗ . At the n − 1 / 2 n^{-1/2} n − 1/2 scale, its effect on a contrast is determined by the drift n h v n T B ω \sqrt{n}h\,\mathbf{v}_n^{\mathsf T}B_\omega n h v n T B ω . A nonnegligible drift alters the limiting distribution even after the optimization error becomes negligible. The residual shift in Proposition 3 removes this first-order bias. Under the normal-limit conditions of Corollary 5, the remaining O ( h 2 ) O(h^2) O ( h 2 ) displacement is negligible at this scale, giving a centered normal limit at θ ∗ \theta^* θ ∗ with the original score variance.
For Moreau smoothing, c ω = ( a + b ) / 2 c_\omega=(a+b)/2 c ω = ( a + b ) /2 . For unit pinball loss at level τ \tau τ , the intercept displacement is − ( 2 τ − 1 ) h / 2 + O ( h 2 ) -(2\tau-1)h/2+O(h^2) − ( 2 τ − 1 ) h /2 + O ( h 2 ) , including the scalar location model with Z = 1 Z=1 Z = 1 , and the residual shift is h ( 2 τ − 1 ) / 2 h(2\tau-1)/2 h ( 2 τ − 1 ) /2 .
For the CVaR objective in Example 2, the risk threshold has displacement − h / { 2 ( 1 − α C V a R ) } + O ( h 2 ) -h/\{2(1-\alpha_{\mathrm{CVaR}})\}+O(h^2) − h / { 2 ( 1 − α CVaR )} + O ( h 2 ) , while the hedge weights have displacement O ( h 2 ) O(h^2) O ( h 2 ) . The threshold displacement is downward, with a first-order coefficient increasing in magnitude as α C V a R → 1 \alpha_{\mathrm{CVaR}}\to1 α CVaR → 1 . The residual shift is h / { 2 ( 1 − α C V a R ) } h/\{2(1-\alpha_{\mathrm{CVaR}})\} h / { 2 ( 1 − α CVaR )} .
The Moreau displacement also depends on whether a linear term is placed in f f f or in b T θ \mathbf{b}^{\mathsf T}\theta b T θ . To see this for scalar VaR estimation without hedge payoffs, write α = α C V a R \alpha=\alpha_{\mathrm{CVaR}} α = α CVaR , q = ( 1 − α ) − 1 q=(1-\alpha)^{-1} q = ( 1 − α ) − 1 , and f α ( r ) = r { α − I ( r ≤ 0 ) } f_{\alpha}(r)=r\{\alpha-\mathbb{I}(r\le0)\} f α ( r ) = r { α − I ( r ≤ 0 )} . The identity q r + = q f α ( r ) + r qr_{+}=qf_{\alpha}(r)+r q r + = q f α ( r ) + r makes the CVaR objective and the scaled pinball objective differ only by the sample average of B i B_i B i . They therefore have the same minimizer. Their Moreau coefficients are q / 2 q/2 q /2 and q ( 2 α − 1 ) / 2 q(2\alpha-1)/2 q ( 2 α − 1 ) /2 , respectively, which differ by one. For unit pinball loss the coefficient is ( 2 α − 1 ) / 2 (2\alpha-1)/2 ( 2 α − 1 ) /2 , and comparison at the same h h h also changes the loss scaling. More generally, for f ~ ( r ) = f ( r ) + ℓ r \widetilde{f}(r)=f(r)+\ell r f ( r ) = f ( r ) + ℓ r , the quadratic prox-function gives
f ~ h ( r ) = f h ( r − h ℓ ) + ℓ r − h ℓ 2 / 2. \widetilde{f}_h(r)=f_h(r-h\ell)+\ell r-h\ell^2/2. f h ( r ) = f h ( r − h ℓ ) + ℓ r − h ℓ 2 /2.
Adding ℓ r \ell r ℓ r to the loss increases c ω c_{\omega} c ω by ℓ \ell ℓ . The corresponding residual shift in Proposition 3 removes the first-order displacement for either placement of the linear term.
The O ( h 2 ) O(h^2) O ( h 2 ) displacement in Proposition 3 and the localized increment term p a n / ( n h ) \sqrt{pa_n/(nh)} p a n / ( nh ) yield the bandwidth below. The unshifted expansion is centered at θ h ∗ \theta_h^* θ h ∗ , and the shifted expansion is centered at θ ∗ \theta^* θ ∗ .
Corollary 5. Impose the model and population assumptions of Theorem 4 and Assumption 4. Suppose a n → 0 a_n\to0 a n → 0 and choose h ≍ max { a n , ( p / n ) 2 / 5 ( log n ) 1 / 5 } h\asymp\max\{a_n,(p/n)^{2/5}(\log n)^{1/5}\} h ≍ max { a n , ( p / n ) 2/5 ( log n ) 1/5 } with h ≥ A a n h\ge Aa_n h ≥ A a n for sufficiently large A A A . Use a method from Theorem 4 or its unprojected extension, with its initialization and region requirements, step size 0 < η ≤ 1 / ( 2 L 0 ) 0<\eta\le1/(2L_0) 0 < η ≤ 1/ ( 2 L 0 ) , and ϵ 1 = ϵ 2 = 0 \epsilon_1=\epsilon_2=0 ϵ 1 = ϵ 2 = 0 . For either complete procedure use the local refinement in Phase 3 in Appendix A.3. After ⌈ c log n ⌉ \lceil c\log n\rceil ⌈ c log n ⌉ updates for sufficiently large c c c , the computed solution has distance O P ( n − 1 ) O_P(n^{-1}) O P ( n − 1 ) from the smoothed SAA solution and satisfies
θ ~ h − θ h ∗ = − H − 1 { ∇ L h ( θ h ∗ ) − ∇ L ‾ h ( θ h ∗ ) } + O P { ( p / n ) 4 / 5 ( log n ) 2 / 5 + a n 2 } . \widetilde{\theta}_h-\theta_h^*=-H^{-1}\{\nabla\mathcal{L}_h(\theta_h^*)-\nabla\overline{\mathcal{L}}_h(\theta_h^*)\}+O_P\left\{(p/n)^{4/5}(\log n)^{2/5}+a_n^2\right\}. θ h − θ h ∗ = − H − 1 { ∇ L h ( θ h ∗ ) − ∇ L h ( θ h ∗ )} + O P { ( p / n ) 4/5 ( log n ) 2/5 + a n 2 } . If p ( log n ) 4 / 3 = o ( n ) p(\log n)^{4/3}=o(n) p ( log n ) 4/3 = o ( n ) , then h ≍ ( p / n ) 2 / 5 ( log n ) 1 / 5 h\asymp(p/n)^{2/5}(\log n)^{1/5} h ≍ ( p / n ) 2/5 ( log n ) 1/5 and the remainder is O P { ( p / n ) 4 / 5 ( log n ) 2 / 5 } O_P\{(p/n)^{4/5}(\log n)^{2/5}\} O P {( p / n ) 4/5 ( log n ) 2/5 } . For any deterministic unit contrast v n v_n v n whose unsmoothed score variance σ n 2 = Var ( v n T H − 1 ψ i ) \sigma_n^2=\operatorname{Var}(v_n^{\mathsf T}H^{-1}\psi_i) σ n 2 = Var ( v n T H − 1 ψ i ) is bounded below, p = o { n 3 / 8 ( log n ) − 1 / 2 } p=o\{n^{3/8}(\log n)^{-1/2}\} p = o { n 3/8 ( log n ) − 1/2 } gives
n v n T ( θ ~ h − θ h ∗ ) σ n → d N ( 0 , 1 ) . \frac{\sqrt{n}v_n^{\mathsf T}(\widetilde{\theta}_h-\theta_h^*)}{\sigma_n}\xrightarrow{d}\mathcal{N}(0,1). σ n n v n T ( θ h − θ h ∗ ) d N ( 0 , 1 ) . For an output from the shifted objective in Proposition 3, the preceding conclusions hold with L h s h \mathcal{L}_h^{\mathrm{sh}} L h sh in place of L h \mathcal{L}_h L h and θ ∗ \theta^* θ ∗ in place of θ h ∗ \theta_h^* θ h ∗ . For the unshifted objective, put d n = n v n T ( θ h ∗ − θ ∗ ) / σ n d_n=\sqrt{n}v_n^{\mathsf T}(\theta_h^*-\theta^*)/\sigma_n d n = n v n T ( θ h ∗ − θ ∗ ) / σ n . The standardized contrast centered at θ ∗ \theta^* θ ∗ has limiting distribution N ( λ , 1 ) \mathcal{N}(\lambda,1) N ( λ , 1 ) if d n → λ d_n\to\lambda d n → λ , retains the displayed limit if d n → 0 d_n\to0 d n → 0 , and diverges in absolute value in probability if ∣ d n ∣ → ∞ |d_n|\to\infty ∣ d n ∣ → ∞ . For the losses in Proposition 2,
d n = n h v n T B ω / σ n + o ( 1 ) d_n=\sqrt{nh}\,v_n^{\mathsf T}B_{\omega}/\sigma_n+o(1) d n = nh v n T B ω / σ n + o ( 1 ) under the dimension condition for the normal limit.
The iteration bounds and expansions extend to the index objectives covered by Proposition 9 in Appendix C.4.
Smoothing serves two distinct purposes at two bandwidths. At h = A a n h=Aa_n h = A a n with sufficiently large A A A , it gives exact geometric convergence. At this bandwidth, the population displacement is o ( n − 1 / 2 ) o(n^{-1/2}) o ( n − 1/2 ) when p log n = o ( n ) p\log n=o(\sqrt{n}) p log n = o ( n ) . At h ≍ ( p / n ) 2 / 5 ( log n ) 1 / 5 h\asymp(p/n)^{2/5}(\log n)^{1/5} h ≍ ( p / n ) 2/5 ( log n ) 1/5 , the shifted objective improves the expansion remainder, with the expansion centered at θ ∗ \theta^* θ ∗ . Without the shift, the Moreau drift diverges in absolute value under the normal-limit conditions of Corollary 5 if ∣ v n T B ω ∣ / σ n |v_n^{\mathsf T}B_{\omega}|/\sigma_n ∣ v n T B ω ∣/ σ n stays bounded away from zero.
Regularized SAA
When p > n p > n p > n , strong convexity of the expected objective need not carry over to the SAA objective. Sparsity allows us to control objective changes along the directions used by sparse decisions. Write S = supp ( θ ∗ ) S = \operatorname{supp}(\theta^*) S = supp ( θ ∗ ) and s = ∣ S ∣ ≪ p s = |S| \ll p s = ∣ S ∣ ≪ p . In this section, θ ^ \hat{\theta} θ ^ denotes a solution of the regularized SAA problem
θ ^ ∈ arg min θ ∈ Θ F ( θ ) , F ( θ ) = L ( θ ) + λ r ( θ ) , \hat{\theta} \in\underset{\theta\in\Theta}{\arg\min}\ F(\theta), \qquad F(\theta) = \mathcal{L}(\theta) + \lambda r(\theta), θ ^ ∈ θ ∈ Θ arg min F ( θ ) , F ( θ ) = L ( θ ) + λ r ( θ ) ,
where r r r is a penalty and λ > 0 \lambda> 0 λ > 0 . Throughout this section, L \mathcal{L} L is convex. We extend the increment bound to sparse directions and use the resulting regularity to analyze proximal subgradient descent and iterative hard thresholding. Their local expansions give inference on retained sets containing S S S .
Restricted regularity conditions
Restricting the increment bound to sparse decisions makes its approximation scales depend on sparsity, with only logarithmic dependence on p p p . We use Θ r e s = { θ ∈ Θ : ∣ θ ∣ 0 ≤ s ~ } \Theta_{\mathrm{res}} = \{\theta\in\Theta: |\theta|_0 \le\tilde{s}\} Θ res = { θ ∈ Θ : ∣ θ ∣ 0 ≤ s ~ } , where s ≤ s ~ ≍ s s \le\tilde{s} \asymp s s ≤ s ~ ≍ s . For a ∈ R p a \in\mathbb{R}^p a ∈ R p , define ∣ a ∣ ( k ) = max ∣ J ∣ ≤ k ∣ a J ∣ 2 |a|_{(k)} = \max_{|J| \le k} |a_J|_2 ∣ a ∣ ( k ) = max ∣ J ∣ ≤ k ∣ a J ∣ 2 , the Euclidean norm of its k k k largest coordinates in magnitude.
Condition 1-R. The SAA objective L \mathcal{L} L is ( L , ϵ 1 ) (L,\epsilon_1) ( L , ϵ 1 ) -restricted smooth on Θ r e s \Theta_{\mathrm{res}} Θ res if, for every θ 1 , θ 2 ∈ Θ r e s \theta_1,\theta_2 \in\Theta_{\mathrm{res}} θ 1 , θ 2 ∈ Θ res ,
∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) ∣ ( 2 s ~ ) ≤ L max { ∣ θ 1 − θ 2 ∣ 2 , ϵ 1 } . |\partial\mathcal{L}(\theta_1) - \partial\mathcal{L}(\theta_2)|_{(2\tilde{s})} \le L\max\{|\theta_1-\theta_2|_2,\epsilon_1\}. ∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) ∣ ( 2 s ~ ) ≤ L max { ∣ θ 1 − θ 2 ∣ 2 , ϵ 1 } .
The norm in Condition 1-R pairs with a 2 s ~ 2\tilde{s} 2 s ~ -sparse decision difference. A maximum-norm increment bound with constant L 0 L_0 L 0 implies this condition with L = 2 s ~ L 0 L = \sqrt{2\tilde{s}}L_0 L = 2 s ~ L 0 .
Condition 2-R. The SAA objective L \mathcal{L} L is ( μ , ϵ 2 ) (\mu,\epsilon_2) ( μ , ϵ 2 ) -restricted strongly convex on Θ s c ⊆ Θ \Theta_{\mathrm{sc}} \subseteq\Theta Θ sc ⊆ Θ if, for every θ 1 , θ 2 ∈ Θ s c ∩ Θ r e s \theta_1,\theta_2 \in\Theta_{\mathrm{sc}} \cap\Theta_{\mathrm{res}} θ 1 , θ 2 ∈ Θ sc ∩ Θ res ,
L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ μ 4 ∣ θ 1 − θ 2 ∣ 2 2 , for ∣ θ 1 − θ 2 ∣ 2 ≥ ϵ 2 . \mathcal{L}(\theta_1) - \mathcal{L}(\theta_2) - \langle\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle\ge\frac{\mu}{4}|\theta_1-\theta_2|_2^2,\qquad\text{for }|\theta_1-\theta_2|_2 \ge\epsilon_2. L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ 4 μ ∣ θ 1 − θ 2 ∣ 2 2 , for ∣ θ 1 − θ 2 ∣ 2 ≥ ϵ 2 .
Condition 2-R gives quadratic growth between sparse vectors separated by at least ϵ 2 \epsilon_2 ϵ 2 . The distance cutoff makes this condition applicable to piecewise linear SAA objectives. Together with the restricted increment bounds, it extends the use of restricted strong convexity in recovery analysis ([36 , 56 , 57 ] ) and fast optimization ([1 ] ) to expansions of subgradient iterates. Phase 3 uses a subgradient approximation near θ ∗ \theta^* θ ∗ .
Condition 3-R. The SAA objective L \mathcal{L} L admits a restricted local subgradient approximation at θ ∗ \theta^* θ ∗ with matrix H H H , radius ϵ 3 \epsilon_3 ϵ 3 , and subgradient remainder bound r n r_n r n if, for every θ ∈ Θ r e s \theta\in\Theta_{\mathrm{res}} θ ∈ Θ res ,
∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∣ ∞ ≤ r n , for ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 . |\partial\mathcal{L}(\theta) - \partial\mathcal{L}(\theta^*) - H(\theta-\theta^*)|_\infty\le r_n,\qquad\text{for }|\theta-\theta^*|_2 \le\epsilon_3. ∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∣ ∞ ≤ r n , for ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 .
The matrix H H H is symmetric and satisfies m H I ⪯ H ⪯ M H I m_H I \preceq H \preceq M_H I m H I ⪯ H ⪯ M H I for constants 0 < m H ≤ M H < ∞ 0 < m_H \le M_H < \infty 0 < m H ≤ M H < ∞ .
Theorem 6. Consider the residual objective (10) with i.i.d. observations and p > n ≥ 2 p > n \ge2 p > n ≥ 2 , and write a s = s ~ log p / n a_s = \tilde{s}\log p/n a s = s ~ log p / n . Suppose Θ \Theta Θ is convex with sup θ ∈ Θ ∣ θ ∣ 2 ≤ C Θ \sup_{\theta\in\Theta}|\theta|_2 \le C_\Theta sup θ ∈ Θ ∣ θ ∣ 2 ≤ C Θ , the conditional density of Y Y Y given Z Z Z is bounded by C ρ C_\rho C ρ , and f f f is finite, convex, and Lipschitz.
If ∥ Z ∥ ∞ ≤ C Z \lVert Z\rVert_{\infty} \le C_Z ∥ Z ∥ ∞ ≤ C Z almost surely and Λ max { E ( Z Z T ) } = O ( 1 ) \Lambda_{\max}\{\mathbb{E}(ZZ^{\mathsf{T}})\}=O(1) Λ m a x { E ( Z Z T )} = O ( 1 ) , then, with probability at least 1 − C p − γ s ~ 1-Cp^{-\gamma\tilde{s}} 1 − C p − γ s ~ ,
∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ‾ ( θ 1 ) + ∇ L ‾ ( θ 2 ) ∣ ∞ ≤ C { ∣ θ 1 − θ 2 ∣ 2 2 s ~ + a s ~ } . (16) \left\lvert\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)-\nabla\overline{\mathcal{L}}(\theta_1)+\nabla\overline{\mathcal{L}}(\theta_2)\right\rvert_{\infty}\le C\left\{\sqrt{\lvert\theta_1-\theta_2\rvert_{2}^{2}\tilde{s}}+a_{\tilde{s}}\right\}. \tag*{(16)} ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ( θ 1 ) + ∇ L ( θ 2 ) ∞ ≤ C { ∣ θ 1 − θ 2 ∣ 2 2 s ~ + a s ~ } . (16)
If sup ∣ v ∣ 2 = 1 , ∣ v ∣ 0 ≤ 2 s ~ ∥ v T Z ∥ ψ 2 ≤ C Z \sup_{\lvert v\rvert_2=1,\lvert v\rvert_0\le2\tilde{s}}\lVert v^{\mathsf{T}}Z\rVert_{\psi_2}\le C_Z sup ∣ v ∣ 2 = 1 , ∣ v ∣ 0 ≤ 2 s ~ ∥ v T Z ∥ ψ 2 ≤ C Z , then (16) holds with ∣ ⋅ ∣ ∞ \lvert\cdot\rvert_{\infty} ∣ ⋅ ∣ ∞ replaced by ∣ ⋅ ∣ ( 2 s ~ ) \lvert\cdot\rvert_{(2\tilde{s})} ∣ ⋅ ∣ ( 2 s ~ ) , with the same probability bound.
Both increment bounds hold for some C , γ > 0 C,\gamma>0 C , γ > 0 , uniformly over θ 1 , θ 2 ∈ Θ \theta_1,\theta_2\in\Theta θ 1 , θ 2 ∈ Θ with ∣ supp ( θ 1 ) ∪ supp ( θ 2 ) ∣ ≤ 2 s ~ \lvert\operatorname{supp}(\theta_1)\cup\operatorname{supp}(\theta_2)\rvert\le2\tilde{s} ∣ supp ( θ 1 ) ∪ supp ( θ 2 )∣ ≤ 2 s ~ , for every selection of scenario subgradients. Under either set of assumptions, at θ ∗ ∈ Θ \theta^*\in\Theta θ ∗ ∈ Θ , the bound ∣ ∂ L ( θ ∗ ) − ∇ L ‾ ( θ ∗ ) ∣ ∞ ≤ C log p / n \lvert\partial\mathcal{L}(\theta^*)-\nabla\overline{\mathcal{L}}(\theta^*)\rvert_{\infty}\le C\sqrt{\log p/n} ∣ ∂ L ( θ ∗ ) − ∇ L ( θ ∗ ) ∣ ∞ ≤ C log p / n holds with probability at least 1 − C p − γ 1-Cp^{-\gamma} 1 − C p − γ . The increment and pointwise bounds hold jointly with probability at least 1 − C p − γ 1-Cp^{-\gamma} 1 − C p − γ .
These bounds require no response moment condition. Corollary 16 in Appendix § 11 gives these results for index objectives.
The restricted norm controls objective changes without an additional factor s ~ \sqrt{\tilde{s}} s ~ . To verify the local approximation, assume that the expected objective is differentiable near θ ∗ \theta^* θ ∗ and, for some matrix H H H ,
∣ ∇ L ‾ ( θ ) − ∇ L ‾ ( θ ∗ ) − H ( θ − θ ∗ ) ∣ ∞ ≤ C ∣ θ − θ ∗ ∣ 2 2 (17) \left\lvert\nabla\overline{\mathcal{L}}(\theta)-\nabla\overline{\mathcal{L}}(\theta^*)-H(\theta-\theta^*)\right\rvert_{\infty}\le C\lvert\theta-\theta^*\rvert_2^2 \tag*{(17)} ∇ L ( θ ) − ∇ L ( θ ∗ ) − H ( θ − θ ∗ ) ∞ ≤ C ∣ θ − θ ∗ ∣ 2 2 (17)
for sparse θ \theta θ with ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 \lvert\theta-\theta^*\rvert_2\le\epsilon_3 ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 . Under Theorem 6(2), a bounded derivative of the conditional density of Y Y Y given Z Z Z gives this Taylor bound and its version in ∣ ⋅ ∣ ( 2 s ~ ) \lvert\cdot\rvert_{(2\tilde{s})} ∣ ⋅ ∣ ( 2 s ~ ) used for inference below.
Corollary 6. Under the assumptions of Theorem 6(2), suppose a s = o ( 1 ) a_s=o(1) a s = o ( 1 ) . On its increment event, the following hold.
Condition 1-R holds with L = L 0 L=L_0 L = L 0 and ϵ 1 = C 1 a s \epsilon_1=C_1a_s ϵ 1 = C 1 a s .
If L ‾ \overline{\mathcal{L}} L is μ 0 \mu_0 μ 0 -strongly convex on a convex set Θ s c ⊆ Θ \Theta_{\mathrm{sc}}\subseteq\Theta Θ sc ⊆ Θ , then Condition 2-R holds with μ = μ 0 \mu=\mu_0 μ = μ 0 and ϵ 2 = C 2 a s \epsilon_2=C_2a_s ϵ 2 = C 2 a s .
If (17) holds and H H H satisfies the spectral bounds in Condition 3-R, then that condition holds with ϵ 3 = C 3 a s \epsilon_3=C_3\sqrt{a_s} ϵ 3 = C 3 a s and r n = C 4 a s 3 / 4 r_n=C_4a_s^{3/4} r n = C 4 a s 3/4 .
Under the bounded-coordinate assumptions of Theorem 6, these conclusions hold with L = 2 s ~ L 0 L=\sqrt{2\tilde{s}}L_0 L = 2 s ~ L 0 and ϵ 2 = C 2 s ~ a s \epsilon_2=C_2\tilde{s}a_s ϵ 2 = C 2 s ~ a s . The values of ϵ 1 , ϵ 3 , r n \epsilon_1,\epsilon_3,r_n ϵ 1 , ϵ 3 , r n are unchanged.
The bound L = L 0 L=L_0 L = L 0 permits step sizes independent of s s s in Phases 1 and 2.
Proximal subgradient descent
We use proximal subgradient descent (PGD) with the ℓ 1 \ell_1 ℓ 1 -penalty in Phases 1 and 2 and the ℓ 0 \ell_0 ℓ 0 -penalty in Phase 3. The ℓ 1 \ell_1 ℓ 1 -penalty preserves convexity of the regularized objective and gives soft thresholding. Under the sufficient condition in Appendix § 11 .§ 3 , soft thresholding preserves a working set of size at most s ~ \tilde{s} s ~ . The ℓ 0 \ell_0 ℓ 0 -penalty gives hard thresholding, which removes small coordinates without shrinking the retained coordinates. On the recovered support S S S , the updates coincide with unpenalized subgradient descent and yield the expansion in Theorem 7(iii).
Algorithm 4 Proximal subgradient descent
Input: Initial iterate θ ( 0 ) \theta^{(0)} θ ( 0 ) , iterations T T T , step size η \eta η , and penalty weight λ \lambda λ .
for t = 0 , … , T − 1 t = 0,\ldots,T-1 t = 0 , … , T − 1 do
Compute the proximal subgradient update
θ ( t + 1 ) = arg min θ { ⟨ ∂ L ( θ ( t ) ) , θ − θ ( t ) ⟩ + λ r ( θ ) + 1 2 η ∥ θ − θ ( t ) ∥ 2 2 } . (18) \theta^{(t+1)} = \underset{\theta}{\arg\min}\ \left\{\left\langle\partial\mathcal{L}(\theta^{(t)}),\theta-\theta^{(t)}\right\rangle+\lambda r(\theta)+\frac{1}{2\eta}\lVert\theta-\theta^{(t)}\rVert_2^2\right\}. \tag*{(18)} θ ( t + 1 ) = θ arg min { ⟨ ∂ L ( θ ( t ) ) , θ − θ ( t ) ⟩ + λ r ( θ ) + 2 η 1 ∥ θ − θ ( t ) ∥ 2 2 } . (18)
end for
Output: Final iterate θ ( T ) \theta^{(T)} θ ( T ) .
In Phase 3, the ℓ 0 \ell_0 ℓ 0 -penalty thresholds the coordinates of θ − η ∂ L ( θ ) \theta-\eta\partial\mathcal{L}(\theta) θ − η ∂ L ( θ ) at 2 η λ \sqrt{2\eta\lambda} 2 η λ . Let b n = ∣ ∂ L ( θ ∗ ) ∣ ∞ b_n=\lvert\partial\mathcal{L}(\theta^*)\rvert_\infty b n = ∣ ∂ L ( θ ∗ ) ∣ ∞ and use r n r_n r n from Condition 3-R. For a step size η > 0 \eta>0 η > 0 , write q = ∥ I − η H ∥ ∞ q=\lVert I-\eta H\rVert_\infty q = ∥ I − η H ∥ ∞ . For d > 0 d>0 d > 0 , define B d = { θ ∈ R p : ∣ θ ∣ 0 ≤ s , e ∥ θ − θ ∗ ∥ ∞ ≤ d } B_d=\{\theta\in\mathbb{R}^p:\lvert\theta\rvert_0\le s,\ e\lVert\theta-\theta^*\rVert_\infty\le d\} B d = { θ ∈ R p : ∣ θ ∣ 0 ≤ s , e ∥ θ − θ ∗ ∥ ∞ ≤ d } .
Theorem 7. Consider Algorithm 4, with t = 0 t=0 t = 0 denoting the first iterate of each phase.
(i) Phase 1. Under Condition 1-R, let r ( θ ) = ∣ θ ∣ 1 r(\theta)=\lvert\theta\rvert_1 r ( θ ) = ∣ θ ∣ 1 and let θ ^ ∈ Θ r e s \hat{\theta}\in\Theta_{\mathrm{res}} θ ^ ∈ Θ res minimize F F F . Suppose θ ( t ) ∈ Θ r e s \theta^{(t)}\in\Theta_{\mathrm{res}} θ ( t ) ∈ Θ res for 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T and 0 < η ≤ 1 / ( 2 L ) 0<\eta\le1/(2L) 0 < η ≤ 1/ ( 2 L ) . Then
1 T ∑ t = 1 T { F ( θ ( t ) ) − F ( θ ^ ) } ≤ ∣ θ ( 0 ) − θ ^ ∣ 2 2 2 T η + η L 2 ϵ 1 2 2 . \frac{1}{T}\sum_{t=1}^{T}\{F(\theta^{(t)})-F(\hat{\theta})\}\le\frac{\lvert\theta^{(0)}-\hat{\theta}\rvert_2^2}{2T\eta}+\frac{\eta L^2\epsilon_1^2}{2}. T 1 t = 1 ∑ T { F ( θ ( t ) ) − F ( θ ^ )} ≤ 2 T η ∣ θ ( 0 ) − θ ^ ∣ 2 2 + 2 η L 2 ϵ 1 2 . If ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 ≥ ϵ 1 \lvert\theta^{(t+1)}-\theta^{(t)}\rvert_2\ge\epsilon_1 ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 ≥ ϵ 1 for 0 ≤ t < T 0\le t<T 0 ≤ t < T , then
F ( θ ( T ) ) − F ( θ ^ ) ≤ ∣ θ ( 0 ) − θ ^ ∣ 2 2 2 T η . F(\theta^{(T)})-F(\hat{\theta})\le\frac{\lvert\theta^{(0)}-\hat{\theta}\rvert_2^2}{2T\eta}. F ( θ ( T ) ) − F ( θ ^ ) ≤ 2 T η ∣ θ ( 0 ) − θ ^ ∣ 2 2 . (ii) Phase 2. Under Conditions 1-R and 2-R, let r ( θ ) = ∣ θ ∣ 1 r(\theta)=\lvert\theta\rvert_1 r ( θ ) = ∣ θ ∣ 1 and let θ ^ ∈ Θ s c ∩ Θ r e s \hat{\theta}\in\Theta_{\mathrm{sc}}\cap\Theta_{\mathrm{res}} θ ^ ∈ Θ sc ∩ Θ res minimize F F F . Suppose θ ( t ) ∈ Θ s c ∩ Θ r e s \theta^{(t)}\in\Theta_{\mathrm{sc}}\cap\Theta_{\mathrm{res}} θ ( t ) ∈ Θ sc ∩ Θ res for 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T . For 0 < η ≤ 1 / ( 2 L ) 0<\eta\le1/(2L) 0 < η ≤ 1/ ( 2 L ) , define α = min { η μ / 2 , 1 } \alpha=\min\{\eta\mu/2,1\} α = min { η μ /2 , 1 } . Then
∣ θ ( T ) − θ ^ ∣ 2 2 ≤ ( 1 − α ) T ∣ θ ( 0 ) − θ ^ ∣ 2 2 + ϵ 2 2 + η 2 L 2 ϵ 1 2 α . \lvert\theta^{(T)}-\hat{\theta}\rvert_2^2\le(1-\alpha)^T\lvert\theta^{(0)}-\hat{\theta}\rvert_2^2+\epsilon_2^2+\frac{\eta^2L^2\epsilon_1^2}{\alpha}. ∣ θ ( T ) − θ ^ ∣ 2 2 ≤ ( 1 − α ) T ∣ θ ( 0 ) − θ ^ ∣ 2 2 + ϵ 2 2 + α η 2 L 2 ϵ 1 2 . If ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 ≥ ϵ 1 \lvert\theta^{(t+1)}-\theta^{(t)}\rvert_2\ge\epsilon_1 ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 ≥ ϵ 1 and ∣ θ ( t ) − θ ^ ∣ 2 ≥ ϵ 2 \lvert\theta^{(t)}-\hat{\theta}\rvert_2\ge\epsilon_2 ∣ θ ( t ) − θ ^ ∣ 2 ≥ ϵ 2 for 0 ≤ t < T 0\le t<T 0 ≤ t < T , both additive terms can be omitted.
(iii) Phase 3. Assume Condition 3-R holds. Suppose that B d ⊆ Θ B_d\subseteq\Theta B d ⊆ Θ , s d ≤ ϵ 3 \sqrt{s}d\le\epsilon_3 s d ≤ ϵ 3 , and q < 1 q<1 q < 1 . Assume θ ( 0 ) ∈ B d \theta^{(0)}\in B_d θ ( 0 ) ∈ B d , η ( b n + r n ) ≤ ( 1 − q ) d \eta(b_n+r_n)\le(1-q)d η ( b n + r n ) ≤ ( 1 − q ) d , and d < 2 η λ < min j ∈ S ∣ θ j ∗ ∣ − d d<\sqrt{2\eta\lambda}<\min_{j\in S}\lvert\theta_j^*\rvert-d d < 2 η λ < min j ∈ S ∣ θ j ∗ ∣ − d . Then the ℓ 0 \ell_0 ℓ 0 -penalized proximal subgradient iterates satisfy supp ( θ ( t ) ) = S \operatorname{supp}(\theta^{(t)})=S supp ( θ ( t ) ) = S for every t ≥ 1 t\ge1 t ≥ 1 . For T ≥ 1 T\ge1 T ≥ 1 ,
∣ { θ ( T ) − θ ∗ } S + H S × S − 1 { ∂ L ( θ ∗ ) } S ∣ ∞ ≤ q T − 1 ( d + η b n 1 − q ) + η r n 1 − q . (19) \left\lvert\{\theta^{(T)}-\theta^*\}_S+H_{S\times S}^{-1}\{\partial\mathcal{L}(\theta^*)\}_S\right\rvert_\infty\le q^{T-1}\left(d+\frac{\eta b_n}{1-q}\right)+\frac{\eta r_n}{1-q}. \tag*{(19)} { θ ( T ) − θ ∗ } S + H S × S − 1 { ∂ L ( θ ∗ ) } S ∞ ≤ q T − 1 ( d + 1 − q η b n ) + 1 − q η r n . (19)
The O ( T − 1 ) O(T^{-1}) O ( T − 1 ) objective bound in Theorem 7(i) matches the smooth convex rate, improving on the O ( T − 1 / 2 ) O(T^{-1/2}) O ( T − 1/2 ) guarantee for nonsmooth proximal subgradient descent [3 Theorem 9.26] . A positive step size with q < 1 q<1 q < 1 exists precisely when H H H is strictly diagonally dominant. Theorem 8(iv) gives containment and an expansion without diagonal dominance for IHT on an unchanged retained set containing S S S . The admissible step sizes and the inverse bound for q < 1 q<1 q < 1 are given in the local recursion calculation in Appendix D. After support recovery, the leading term in (19) is the same as if S S S were known. For q ≤ q 0 < 1 q \le q_0 < 1 q ≤ q 0 < 1 , a logarithmic number of updates reduces the optimization term to order r n r_n r n when d / r n d/r_n d / r n grows at most polynomially in n n n . This adds a distributional approximation of the computed solution to geometric-convergence and recovery guarantees for ℓ 1 \ell_1 ℓ 1 -penalized methods Xiao and Zhang [59 ] and folded concave penalties Wang et al. [58 ] . Appendix D.3 gives the penalty calibration for Phase 3. Under the proximal-map condition stated there, support recovery and the expansion also hold for SCAD, MCP, and capped-ℓ 1 \ell_1 ℓ 1 penalties.
Iterative hard thresholding
Iterative hard thresholding (IHT) preserves sparsity by retaining the s ~ \tilde{s} s ~ largest coordinates in magnitude after each subgradient step [5 , 26 ] . Algorithm 5 is the proximal subgradient update (18) with a penalty equal to zero on vectors with at most s ~ \tilde{s} s ~ nonzero coordinates and + ∞ +\infty + ∞ otherwise. Its Phase 2 and Phase 3 bounds use the population solution θ ∗ \theta^* θ ∗ as the reference point.
Algorithm 5 Iterative hard thresholding
Input: Initial iterate θ ( 0 ) \theta^{(0)} θ ( 0 ) , iterations T T T , step size η \eta η , and sparsity level s ~ \tilde{s} s ~ .
for t = 0 , … , T − 1 t = 0,\ldots,T-1 t = 0 , … , T − 1 do
Compute the IHT update
θ ( t + 1 ) = argmin ∣ θ ∣ 0 ≤ s ~ { ⟨ ∂ L ( θ ( t ) ) , θ − θ ( t ) ⟩ + 1 2 η ∥ θ − θ ( t ) ∥ 2 2 } \theta^{(t+1)}=\underset{\lvert\theta\rvert_0\le\tilde{s}}{\operatorname{argmin}}\left\{\left\langle\partial\mathcal{L}(\theta^{(t)}),\theta-\theta^{(t)}\right\rangle+\frac{1}{2\eta}\lVert\theta-\theta^{(t)}\rVert_2^2\right\} θ ( t + 1 ) = ∣ θ ∣ 0 ≤ s ~ argmin { ⟨ ∂ L ( θ ( t ) ) , θ − θ ( t ) ⟩ + 2 η 1 ∥ θ − θ ( t ) ∥ 2 2 }
end for
Output: Final iterate θ ( T ) \theta^{(T)} θ ( T ) .
Theorem 8. Consider Algorithm 5, with t = 0 t=0 t = 0 denoting the first iterate of each phase. For part (iii), assume Condition 3-R holds, s ~ ≥ s \tilde{s}\ge s s ~ ≥ s , B d ⊆ Θ B_d\subseteq\Theta B d ⊆ Θ , s d ≤ ϵ 3 \sqrt{s}d\le\epsilon_3 s d ≤ ϵ 3 , q < 1 q<1 q < 1 , θ ( 0 ) ∈ B d \theta^{(0)}\in B_d θ ( 0 ) ∈ B d , and η ( b n + r n ) ≤ ( 1 − q ) d \eta(b_n+r_n)\le(1-q)d η ( b n + r n ) ≤ ( 1 − q ) d .
(i) Phase 1. Under Condition 1-R, suppose θ ( 0 ) ∈ Θ r e s \theta^{(0)}\in\Theta_{\mathrm{res}} θ ( 0 ) ∈ Θ res , θ ( t ) ∈ Θ \theta^{(t)}\in\Theta θ ( t ) ∈ Θ for 1 ≤ t ≤ T 1\le t\le T 1 ≤ t ≤ T , and inf θ ∈ Θ r e s L ( θ ) > − ∞ \inf_{\theta\in\Theta_{\mathrm{res}}}\mathcal{L}(\theta)>-\infty inf θ ∈ Θ res L ( θ ) > − ∞ . If 0 < η ≤ 1 / ( 4 L ) 0<\eta\le1/(4L) 0 < η ≤ 1/ ( 4 L ) , then
min 0 ≤ t < T ∥ θ ( t + 1 ) − θ ( t ) ∥ 2 2 ≤ 4 η T { L ( θ ( 0 ) ) − inf θ ∈ Θ r e s L ( θ ) } + 4 η 2 L 2 ϵ 1 2 . \min_{0\le t<T}\lVert\theta^{(t+1)}-\theta^{(t)}\rVert_2^2\le\frac{4\eta}{T}\left\{\mathcal{L}(\theta^{(0)})-\inf_{\theta\in\Theta_{\mathrm{res}}}\mathcal{L}(\theta)\right\}+4\eta^2L^2\epsilon_1^2. 0 ≤ t < T min ∥ θ ( t + 1 ) − θ ( t ) ∥ 2 2 ≤ T 4 η { L ( θ ( 0 ) ) − θ ∈ Θ res inf L ( θ ) } + 4 η 2 L 2 ϵ 1 2 . The final term can be omitted if ∥ θ ( t + 1 ) − θ ( t ) ∥ 2 ≥ ϵ 1 \lVert\theta^{(t+1)}-\theta^{(t)}\rVert_2\ge\epsilon_1 ∥ θ ( t + 1 ) − θ ( t ) ∥ 2 ≥ ϵ 1 for 0 ≤ t < T 0\le t<T 0 ≤ t < T .
(ii) Phase 2. Under Conditions 1-R and 2-R, suppose θ ∗ ∈ Θ s c \theta^*\in\Theta_{\mathrm{sc}} θ ∗ ∈ Θ sc and θ ( t ) ∈ Θ s c ∩ Θ r e s \theta^{(t)}\in\Theta_{\mathrm{sc}}\cap\Theta_{\mathrm{res}} θ ( t ) ∈ Θ sc ∩ Θ res with S ⊆ supp ( θ ( t ) ) S\subseteq\operatorname{supp}(\theta^{(t)}) S ⊆ supp ( θ ( t ) ) for 0 ≤ t ≤ T 0\le t\le T 0 ≤ t ≤ T . For 0 < η ≤ 1 / ( 2 L ) 0<\eta\le1/(2L) 0 < η ≤ 1/ ( 2 L ) , define α = min { η μ / 2 , 1 } \alpha=\min\{\eta\mu/2,1\} α = min { η μ /2 , 1 } . Then
L ( θ ( T ) ) − L ( θ ∗ ) ≤ ( 1 − α ) T { L ( θ ( 0 ) ) − L ( θ ∗ ) } + μ ϵ 2 2 4 + η L 2 ϵ 1 2 2 α . \mathcal{L}(\theta^{(T)})-\mathcal{L}(\theta^*)\le(1-\alpha)^T\{\mathcal{L}(\theta^{(0)})-\mathcal{L}(\theta^*)\}+\frac{\mu\epsilon_2^2}{4}+\frac{\eta L^2\epsilon_1^2}{2\alpha}. L ( θ ( T ) ) − L ( θ ∗ ) ≤ ( 1 − α ) T { L ( θ ( 0 ) ) − L ( θ ∗ )} + 4 μ ϵ 2 2 + 2 α η L 2 ϵ 1 2 . If ∥ θ ( t + 1 ) − θ ( t ) ∥ 2 ≥ ϵ 1 \lVert\theta^{(t+1)}-\theta^{(t)}\rVert_2\ge\epsilon_1 ∥ θ ( t + 1 ) − θ ( t ) ∥ 2 ≥ ϵ 1 and ∥ θ ( t ) − θ ∗ ∥ 2 ≥ ϵ 2 \lVert\theta^{(t)}-\theta^*\rVert_2\ge\epsilon_2 ∥ θ ( t ) − θ ∗ ∥ 2 ≥ ϵ 2 for 0 ≤ t < T 0\le t<T 0 ≤ t < T , both additive terms can be omitted.
(iii) Phase 3 support retention. If min j ∈ S ∣ θ j ∗ ∣ > 2 d \min_{j\in S}|\theta_j^*|>2d min j ∈ S ∣ θ j ∗ ∣ > 2 d , then θ ( t ) ∈ B d \theta^{(t)}\in B_d θ ( t ) ∈ B d and S ⊆ supp ( θ ( t ) ) S\subseteq\operatorname{supp}(\theta^{(t)}) S ⊆ supp ( θ ( t ) ) for every t ≥ 0 t\ge0 t ≥ 0 . Moreover,
∥ θ ( T ) − θ ∗ ∥ ∞ ≤ q T ∥ θ ( 0 ) − θ ∗ ∥ ∞ + η ( 1 − q T ) 1 − q ( b n + r n ) . \lVert\theta^{(T)}-\theta^*\rVert_\infty\le q^T\lVert\theta^{(0)}-\theta^*\rVert_\infty+\frac{\eta(1-q^T)}{1-q}(b_n+r_n). ∥ θ ( T ) − θ ∗ ∥ ∞ ≤ q T ∥ θ ( 0 ) − θ ∗ ∥ ∞ + 1 − q η ( 1 − q T ) ( b n + r n ) .
If s ~ = s \tilde{s}=s s ~ = s , then supp ( θ ( t ) ) = S \operatorname{supp}(\theta^{(t)})=S supp ( θ ( t ) ) = S for every t ≥ 0 t\geq0 t ≥ 0 .
(iv) Phase 3 on a retained set. Under the assumptions of Theorem 6(2), assume (17) with ∣ ⋅ ∣ ∞ |\cdot|_{\infty} ∣ ⋅ ∣ ∞ replaced by ∣ ⋅ ∣ ( 2 s ~ ) |\cdot|_{(2\tilde{s})} ∣ ⋅ ∣ ( 2 s ~ ) , ∇ L ‾ ( θ ∗ ) = 0 \nabla\overline{L}(\theta^{*})=0 ∇ L ( θ ∗ ) = 0 , and H = ∇ 2 L ‾ ( θ ∗ ) \mathbf{H}=\nabla^{2}\overline{L}(\theta^{*}) H = ∇ 2 L ( θ ∗ ) with m H I ⪯ H ⪯ M H I m_{H}\mathbf{I}\preceq\mathbf{H}\preceq M_{H}\mathbf{I} m H I ⪯ H ⪯ M H I . Assume a s → 0 a_{s}\to0 a s → 0 , take 0 < η ≤ M H − 1 0<\eta\leq M_{H}^{-1} 0 < η ≤ M H − 1 , and set ϵ 3 = C 3 a s \epsilon_{3}=C_{3}\sqrt{a_{s}} ϵ 3 = C 3 a s for sufficiently large C 3 C_{3} C 3 . Let J ^ ⊇ S \widehat{J}\supseteq S J ⊇ S be a random retained set of size s ~ \tilde{s} s ~ . Suppose that, on an event E n \mathcal{E}_{n} E n of probability at least 1 − δ n 1-\delta_{n} 1 − δ n , where δ n → 0 \delta_{n}\to0 δ n → 0 , the next T T T IHT updates retain J ^ \widehat{J} J , supp ( θ ( 0 ) ) ⊆ J ^ \operatorname{supp}(\theta^{(0)})\subseteq\widehat{J} supp ( θ ( 0 ) ) ⊆ J , ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ ϵ 3 |\theta^{(0)}-\theta^{*}|_{2}\leq\epsilon_{3} ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ ϵ 3 , and
{ θ : supp ( θ ) ⊆ J ^ , ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 } ⊆ Θ . \{\theta:\operatorname{supp}(\theta)\subseteq\widehat{J},\ |\theta-\theta^{*}|_{2}\leq\epsilon_{3}\}\subseteq\Theta. { θ : supp ( θ ) ⊆ J , ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 } ⊆ Θ.
Write e t = { θ ( t ) − θ ∗ } J ^ e_{t}=\{\theta^{(t)}-\theta^{*}\}_{\widehat{J}} e t = { θ ( t ) − θ ∗ } J , g J ^ = { ∂ L ‾ ( θ ∗ ) } J ^ g_{\widehat{J}}=\{\partial\overline{L}(\theta^{*})\}_{\widehat{J}} g J = { ∂ L ( θ ∗ ) } J , and q 2 = 1 − η m H / 2 q_{2}=1-\eta m_{H}/2 q 2 = 1 − η m H /2 , which accounts for the subgradient remainder in the containment bound. For sufficiently large n n n , with probability at least 1 − δ n − C p − γ 1-\delta_{n}-Cp^{-\gamma} 1 − δ n − C p − γ ,
∣ e t ∣ 2 ≤ q 2 t ∣ e 0 ∣ 2 + 2 ( 1 − q 2 t ) m H { ∣ g J ^ ∣ 2 + C a s } ≤ ϵ 3 , 0 ≤ t ≤ T , |e_{t}|_{2}\leq q_{2}^{t}|e_{0}|_{2}+\frac{2(1-q_{2}^{t})}{m_{H}}\{|g_{\widehat{J}}|_{2}+Ca_{s}\}\leq\epsilon_{3},\qquad0\leq t\leq T, ∣ e t ∣ 2 ≤ q 2 t ∣ e 0 ∣ 2 + m H 2 ( 1 − q 2 t ) { ∣ g J ∣ 2 + C a s } ≤ ϵ 3 , 0 ≤ t ≤ T ,
and
∣ e T + H J ^ × J ^ − 1 g J ^ ∣ 2 ≤ C { ( 1 − η m H ) T a s + a s 3 / 4 } . \left|e_{T}+\mathbf{H}_{\widehat{J}\times\widehat{J}}^{-1}g_{\widehat{J}}\right|_{2}\leq C\{(1-\eta m_{H})^{T}\sqrt{a_{s}}+a_{s}^{3/4}\}. e T + H J × J − 1 g J 2 ≤ C {( 1 − η m H ) T a s + a s 3/4 } .
Theorem 8(ii) parallels the second-phase result of Shen et al. [53 ] , whose first phase uses decreasing step sizes. For Euclidean localization, Corollary 15 in Appendix § 11 uses overspecified working sparsity to give geometric convergence to an O ( a s ) O(\sqrt{a_{s}}) O ( a s ) neighborhood. On an unchanged retained set containing S S S , part (iv) also dispenses with a minimum-coefficient condition.
Sampling distribution and stopping rules
For inference on a retained set J ⊇ S J\supseteq S J ⊇ S , use ψ i = b − Z i f + ′ ( Y i − Z i T θ ∗ ) \psi_{i}=\mathbf{b}-Z_{i}f_{+}^{\prime}(Y_{i}-Z_{i}^{\mathsf{T}}\theta^{*}) ψ i = b − Z i f + ′ ( Y i − Z i T θ ∗ ) and write σ J , j 2 = Var ( e j , J T H J × J − 1 ψ i , J ) \sigma_{J,j}^{2}=\operatorname{Var}(e_{j,J}^{\mathsf{T}}\mathbf{H}_{J\times J}^{-1}\psi_{i,J}) σ J , j 2 = Var ( e j , J T H J × J − 1 ψ i , J ) , where e j , J e_{j,J} e j , J is the coordinate vector for j j j in J J J .
Corollary 7. Under the assumptions of Theorem 6(2), impose the population assumptions of Theorem 8(iv) and suppose a s = o ( 1 ) a_{s}=o(1) a s = o ( 1 ) . Use ϵ 3 = C 3 a s \epsilon_{3}=C_{3}\sqrt{a_{s}} ϵ 3 = C 3 a s and r n = C 4 a s 3 / 4 r_{n}=C_{4}a_{s}^{3/4} r n = C 4 a s 3/4 from Corollary 6. Let J = J n ⊇ S J=J_{n}\supseteq S J = J n ⊇ S be deterministic. For IHT, suppose the localization conditions of Theorem 8(iv) or Proposition 12(i) hold with probability tending to one for a retained set J ^ \widehat{J} J satisfying Pr ( J ^ = J ) → 1 \Pr(\widehat{J}=J)\to1 Pr ( J = J ) → 1 . For PGD, take J = S J=S J = S and suppose the conditions of Theorem 7(iii) hold with probability tending to one. Under the conditions of Theorem 7(iii) or Proposition 12(i), take q ≤ q 0 < 1 q\leq q_{0}<1 q ≤ q 0 < 1 and T ≥ ⌈ c log n ⌉ T\geq\lceil c\log n\rceil T ≥ ⌈ c log n ⌉ . Under those of Theorem 8(iv), take T ≥ ⌈ c ( η m H ) − 1 log n ⌉ T\geq\lceil c(\eta m_{H})^{-1}\log n\rceil T ≥ ⌈ c ( η m H ) − 1 log n ⌉ . For sufficiently large c c c , the output satisfies
{ θ ( T ) − θ ∗ } J = − H J × J − 1 1 n ∑ i = 1 n ψ i , J + O P { ( ∣ J ∣ / n ) 3 / 4 log n + ∣ J ∣ log n n } . \{\theta^{(T)}-\theta^{*}\}_{J}=-\mathbf{H}_{J\times J}^{-1}\frac{1}{n}\sum_{i=1}^{n}\psi_{i,J}+O_{P}\left\{(|J|/n)^{3/4}\sqrt{\log n}+\frac{|J|\log n}{n}\right\}. { θ ( T ) − θ ∗ } J = − H J × J − 1 n 1 i = 1 ∑ n ψ i , J + O P { ( ∣ J ∣/ n ) 3/4 log n + n ∣ J ∣ log n } .
in Euclidean norm. For a deterministic j ∈ J j\in J j ∈ J , if σ J , j ≥ c 0 > 0 \sigma_{J,j}\geq c_{0}>0 σ J , j ≥ c 0 > 0 and ∣ J ∣ = o { n 1 / 3 ( log n ) − 2 / 3 } |J|=o\{n^{1/3}(\log n)^{-2/3}\} ∣ J ∣ = o { n 1/3 ( log n ) − 2/3 } , then
n ( θ j ( T ) − θ j ∗ ) σ J , j ⟶ d N ( 0 , 1 ) . \frac{\sqrt{n}(\theta_{j}^{(T)}-\theta_{j}^{*})}{\sigma_{J,j}}\mathrel{\mathop{\longrightarrow}^{d}}\mathcal{N}(0,1). σ J , j n ( θ j ( T ) − θ j ∗ ) ⟶ d N ( 0 , 1 ) .
The expansion depends on ∣ J ∣ |J| ∣ J ∣ and n n n , while the ambient dimension p p p enters localization through a s a_{s} a s . Taking J = S J=S J = S gives inference after support recovery. Proposition 12 in Appendix § 11 gives simultaneous confidence statements when the retained set remains random. Corollary 16 extends these results to index objectives, including sparse SVM with a bounded density derivative.
Update lengths give stopping rules for reaching the local neighborhoods. For an update from θ \theta θ to θ + \theta^{+} θ + , write k = ∣ θ + − θ ∣ 2 k = \lvert\theta^{+} - \theta\rvert_{2} k = ∣ θ + − θ ∣ 2 and G = k / η + L max { k , ϵ 1 } G = k/\eta+ L\max\{k,\epsilon_{1}\} G = k / η + L max { k , ϵ 1 } . Under the conditions of Proposition 11(i)–(ii) in Appendix § 11 , the stopping tests are
k + max { 4 G / μ , ϵ 2 } ≤ ϱ for PGD , k + \max\{4G/\mu,\epsilon_{2}\} \leq\varrho\quad\text{for PGD}, k + max { 4 G / μ , ϵ 2 } ≤ ϱ for PGD ,
k + max { 2 ( G + s ~ b n ) / μ , ϵ 2 } ≤ ϱ for IHT . k + \max\{2(G + \sqrt{\widetilde{s}b_n})/\mu,\epsilon_{2}\} \leq\varrho\quad\text{for IHT}. k + max { 2 ( G + s b n ) / μ , ϵ 2 } ≤ ϱ for IHT .
These tests give ∣ θ − θ ^ ∣ 2 ≤ ϱ \lvert\theta-\widehat{\theta}\rvert_{2} \leq\varrho ∣ θ − θ ∣ 2 ≤ ϱ for PGD with any proper convex penalty and ∣ θ − θ ∗ ∣ 2 ≤ ϱ \lvert\theta-\theta^{*}\rvert_{2} \leq\varrho ∣ θ − θ ∗ ∣ 2 ≤ ϱ for IHT. For PGD, choose ϱ + ∣ θ ^ − θ ∗ ∣ 2 ≤ ϵ 3 \varrho+ \lvert\widehat{\theta}-\theta^{*}\rvert_{2} \leq\epsilon_{3} ϱ + ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 to include the sampling error. For IHT, choose ϱ ≤ ϵ 3 \varrho\leq\epsilon_{3} ϱ ≤ ϵ 3 . Proposition 11 also gives the lower bounds on ϱ \varrho ϱ and the coordinatewise tests for entering B d \mathcal{B}_{d} B d .
Conclusion
The increment bound connects population regularity to convergence and inference for a computed SAA decision. It gives a first-order expansion after O ( log n ) O(\log n) O ( log n ) subgradient or Nesterov updates even when the iterates do not converge. Projection permits any feasible initialization in a closed convex region containing a neighborhood of the population solution. Smoothing yields convergence to the smoothed minimizer, and its population displacement determines the effect on inference. The first-order Moreau bias appears as a deterministic drift, and an explicit shift of the residual removes this bias and improves the remainder without changing the limiting distribution. Restricted increments give expansions for sparse decisions, including IHT on retained sets under spectral Hessian bounds.
The accelerated procedure uses the strong convexity parameter, approximation scales, and radius ratios to choose momentum and phase transitions. Adaptive restart (O’Donoghue and Candès, [40 ] ), local curvature estimation (Jensen et al., [27 ] ), and progress-based step search (Jin et al., [28 ] ) provide possible approaches to selecting these quantities. Applying these methods under our local regularity conditions requires containment and stopping guarantees that preserve the expansion. Nonsmooth nonconvex problems, including robust phase retrieval and models with rectified linear units, call for an extension of the local analysis. Combining increment control with sharpness may give fast convergence under linear objective growth. The expansions also suggest bootstrap inference based on computed solutions, building on smoothing-based bootstrap methods for median regression (Horowitz, [24 ] ). Whether these iteration counts also preserve bootstrap validity across resampled objectives remains open.
Optimization under approximate regularity
Oracle inequalities
Lemma 2. Let L \mathcal{L} L be finite and convex on a convex region. Under Condition 1, every pair θ 1 , θ 2 \theta_1,\theta_2 θ 1 , θ 2 in the region satisfies
0 ≤ L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≤ L 2 ∣ θ 1 − θ 2 ∣ 2 2 + L 2 ϵ 1 2 . (20) 0 \le\mathcal{L}(\theta_1)-\mathcal{L}(\theta_2)-\langle\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle\le\frac{L}{2}|\theta_1-\theta_2|_2^2+\frac{L}{2}\epsilon_1^2. \tag*{(20)} 0 ≤ L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≤ 2 L ∣ θ 1 − θ 2 ∣ 2 2 + 2 L ϵ 1 2 . (20) If θ ^ \hat{\theta} θ ^ minimizes L \mathcal{L} L over the region and lies in its interior, the same condition gives ∣ ∂ L ( θ ^ ) ∣ 2 ≤ L ϵ 1 |\partial\mathcal{L}(\hat{\theta})|_2\le L\epsilon_1 ∣ ∂ L ( θ ^ ) ∣ 2 ≤ L ϵ 1 . If Condition 2 holds on a subset of the region, every pair in that subset satisfies
L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ μ 4 ∣ θ 1 − θ 2 ∣ 2 2 − μ 4 ϵ 2 2 . (21) \mathcal{L}(\theta_1)-\mathcal{L}(\theta_2)-\langle\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle\ge\frac{\mu}{4}|\theta_1-\theta_2|_2^2-\frac{\mu}{4}\epsilon_2^2. \tag*{(21)} L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ 4 μ ∣ θ 1 − θ 2 ∣ 2 2 − 4 μ ϵ 2 2 . (21)
Proof of Lemmas 1 and 2. Write d = θ 1 − θ 2 \boldsymbol{d}=\theta_1-\theta_2 d = θ 1 − θ 2 and d = ∣ d ∣ 2 d=|\boldsymbol{d}|_2 d = ∣ d ∣ 2 . The result is immediate when d = 0 d=0 d = 0 . Convexity gives the lower bound in (20) . The restriction of L \mathcal{L} L to the segment is absolutely continuous, with derivative ⟨ ∂ L ( θ 2 + t d ) , d ⟩ \langle\partial\mathcal{L}(\theta_2+t\boldsymbol{d}),\boldsymbol{d}\rangle ⟨ ∂ L ( θ 2 + t d ) , d ⟩ almost everywhere. Hence
L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , d ⟩ ≤ L d ∫ 0 1 max { t d , ϵ 1 } d t ≤ L 2 ( d 2 + ϵ 1 2 ) . \mathcal{L}(\theta_1)-\mathcal{L}(\theta_2)-\langle\partial\mathcal{L}(\theta_2),\boldsymbol{d}\rangle\le Ld\int_0^1\max\{td,\epsilon_1\}\,dt\le\frac{L}{2}(d^2+\epsilon_1^2). L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , d ⟩ ≤ L d ∫ 0 1 max { t d , ϵ 1 } d t ≤ 2 L ( d 2 + ϵ 1 2 ) .
The integral equals L ϵ 1 d L\epsilon_1d L ϵ 1 d for d ≤ ϵ 1 d\le\epsilon_1 d ≤ ϵ 1 and L ( d 2 + ϵ 1 2 ) / 2 L(d^2+\epsilon_1^2)/2 L ( d 2 + ϵ 1 2 ) /2 for d > ϵ 1 d>\epsilon_1 d > ϵ 1 . Since ϵ 1 d ≤ ( d 2 + ϵ 1 2 ) / 2 \epsilon_1d\le(d^2+\epsilon_1^2)/2 ϵ 1 d ≤ ( d 2 + ϵ 1 2 ) /2 , this proves (20). If d ≥ ϵ 1 d\ge\epsilon_1 d ≥ ϵ 1 , the same bound is at most L d 2 Ld^2 L d 2 , proving (2).
For the subgradient update in the statement, write s = ∣ θ + − θ ∣ 2 s=|\theta^+-\theta|_2 s = ∣ θ + − θ ∣ 2 and take 0 < η ≤ 1 / L 0<\eta\le1/L 0 < η ≤ 1/ L . If s ≤ ϵ 1 s\le\epsilon_1 s ≤ ϵ 1 , the exact segment bound and L ϵ 1 s ≤ s 2 / ( 2 η ) + η L 2 ϵ 1 2 / 2 L\epsilon_1s\le s^2/(2\eta)+\eta L^2\epsilon_1^2/2 L ϵ 1 s ≤ s 2 / ( 2 η ) + η L 2 ϵ 1 2 /2 give (3). If s ≥ ϵ 1 s\ge\epsilon_1 s ≥ ϵ 1 , then
L 2 ( s 2 + ϵ 1 2 ) − s 2 2 η − η L 2 ϵ 1 2 2 = − ( 1 − η L ) ( s 2 − η L ϵ 1 2 ) 2 η ≤ 0 , \frac{L}{2}(s^2+\epsilon_1^2)-\frac{s^2}{2\eta}-\frac{\eta L^2\epsilon_1^2}{2}
=-\frac{(1-\eta L)(s^2-\eta L\epsilon_1^2)}{2\eta}\le0, 2 L ( s 2 + ϵ 1 2 ) − 2 η s 2 − 2 η L 2 ϵ 1 2 = − 2 η ( 1 − η L ) ( s 2 − η L ϵ 1 2 ) ≤ 0 ,
which proves the same bound. When η ≤ 1 / ( 2 L ) \eta\le1/(2L) η ≤ 1/ ( 2 L ) and s ≥ ϵ 1 s\ge\epsilon_1 s ≥ ϵ 1 , the segment remainder is at most L s 2 ≤ s 2 / ( 2 η ) Ls^2\le s^2/(2\eta) L s 2 ≤ s 2 / ( 2 η ) , and the additive term can be omitted.
For (21), Condition 2 gives the bound when d ≥ ϵ 2 d\ge\epsilon_2 d ≥ ϵ 2 . When d < ϵ 2 d<\epsilon_2 d < ϵ 2 , its right-hand side is nonpositive, and convexity suffices.
For the bound at the minimizer, use the directional limiting argument of Lam et al. [33 ] . Let g = ∂ L ( θ ^ ) \boldsymbol{g}=\partial\mathcal{L}(\hat{\theta}) g = ∂ L ( θ ^ ) . If g ≠ 0 \boldsymbol{g}\ne0 g = 0 , take u = − g / ∣ g ∣ 2 \boldsymbol{u}=-\boldsymbol{g}/|\boldsymbol{g}|_2 u = − g /∣ g ∣ 2 . Since θ ^ \hat{\theta} θ ^ lies in the interior, θ ^ + s u \hat{\theta}+s\boldsymbol{u} θ ^ + s u belongs to the region for sufficiently small s > 0 s>0 s > 0 . Convexity and minimality give ⟨ ∂ L ( θ ^ + s u ) , u ⟩ ≥ 0 \langle\partial\mathcal{L}(\hat{\theta}+s\boldsymbol{u}),\boldsymbol{u}\rangle\ge0 ⟨ ∂ L ( θ ^ + s u ) , u ⟩ ≥ 0 , and Condition 1 implies ∣ g ∣ 2 ≤ L max { s , ϵ 1 } |\boldsymbol{g}|_2\le L\max\{s,\epsilon_1\} ∣ g ∣ 2 ≤ L max { s , ϵ 1 } . Letting s ↓ 0 s\downarrow0 s ↓ 0 proves ∣ g ∣ 2 ≤ L ϵ 1 |\boldsymbol{g}|_2\le L\epsilon_1 ∣ g ∣ 2 ≤ L ϵ 1 . The claim is immediate when g = 0 \boldsymbol{g}=0 g = 0 .
Lemma 3. Suppose L \mathcal{L} L is finite and convex on a convex set U U U and satisfies Conditions 1 and 2 there. Let θ ^ \hat{\theta} θ ^ minimize L \mathcal{L} L over U U U and lie in its interior. For every θ ∈ U \theta\in U θ ∈ U ,
∣ θ − θ ^ ∣ 2 2 ≤ 4 μ { L ( θ ) − L ( θ ^ ) } + ϵ 2 2 . (22) |\theta-\hat{\theta}|_2^2\le\frac{4}{\mu}\{\mathcal{L}(\theta)-\mathcal{L}(\hat{\theta})\}+\epsilon_2^2. \tag*{(22)} ∣ θ − θ ^ ∣ 2 2 ≤ μ 4 { L ( θ ) − L ( θ ^ )} + ϵ 2 2 . (22) If ∣ θ − θ ^ ∣ 2 ≥ max { ϵ 1 , 2 ϵ 2 } |\theta-\hat{\theta}|_2\ge\max\{\epsilon_1,\sqrt{2}\epsilon_2\} ∣ θ − θ ^ ∣ 2 ≥ max { ϵ 1 , 2 ϵ 2 } , the inequality chain (4) also holds.
Proof of Lemma 3. Write d = ∣ θ − θ ^ ∣ 2 d=|\theta-\hat{\theta}|_2 d = ∣ θ − θ ^ ∣ 2 . If d > ϵ 2 d>\epsilon_2 d > ϵ 2 , let θ s = θ ^ + s ( θ − θ ^ ) \theta_s=\hat{\theta}+s(\theta-\hat{\theta}) θ s = θ ^ + s ( θ − θ ^ ) for sufficiently small s > 0 s>0 s > 0 . Convexity and minimality give ⟨ ∂ L ( θ s ) , θ − θ s ⟩ ≥ 0 \langle\partial\mathcal{L}(\theta_s),\theta-\theta_s\rangle\ge0 ⟨ ∂ L ( θ s ) , θ − θ s ⟩ ≥ 0 . Since ( 1 − s ) d ≥ ϵ 2 (1-s)d\ge\epsilon_2 ( 1 − s ) d ≥ ϵ 2 , Condition 2 implies
L ( θ ) − L ( θ ^ ) ≥ L ( θ ) − L ( θ s ) ≥ μ 4 ( 1 − s ) 2 d 2 . \mathcal{L}(\theta)-\mathcal{L}(\hat{\theta})\ge\mathcal{L}(\theta)-\mathcal{L}(\theta_s)\ge\frac{\mu}{4}(1-s)^2d^2. L ( θ ) − L ( θ ^ ) ≥ L ( θ ) − L ( θ s ) ≥ 4 μ ( 1 − s ) 2 d 2 .
e n d s m i d = 1 {ends_mid=1} e n d s m i d = 1
Letting s ↓ 0 s \downarrow0 s ↓ 0 gives L ( θ ) − L ( θ ^ ) ≥ μ d 2 / 4 \mathcal{L}(\theta)-\mathcal{L}(\hat{\theta}) \ge\mu d^2/4 L ( θ ) − L ( θ ^ ) ≥ μ d 2 /4 . For d ≤ ϵ 2 d \le\epsilon_2 d ≤ ϵ 2 , minimality suffices for (22) .
Lemma 2 gives ∣ ∂ L ( θ ^ ) ∣ 2 ≤ L ϵ 1 \lvert\partial\mathcal{L}(\hat{\theta})\rvert_2 \le L\epsilon_1 ∣ ∂ L ( θ ^ ) ∣ 2 ≤ L ϵ 1 . For d ≥ ϵ 1 d \ge\epsilon_1 d ≥ ϵ 1 , Condition 1 therefore gives ∣ ∂ L ( θ ) ∣ 2 ≤ L d + L ϵ 1 ≤ 2 L d \lvert\partial\mathcal{L}(\theta)\rvert_2 \le Ld+L\epsilon_1 \le2Ld ∣ ∂ L ( θ ) ∣ 2 ≤ L d + L ϵ 1 ≤ 2 L d . Applying Condition 2 at θ \theta θ with comparison point θ ^ \hat{\theta} θ ^ and completing the square yields
L ( θ ^ ) ≥ L ( θ ) + ⟨ ∂ L ( θ ) , θ ^ − θ ⟩ + μ 4 d 2 ≥ L ( θ ) − 1 μ ∣ ∂ L ( θ ) ∣ 2 2 . \mathcal{L}(\hat{\theta}) \ge\mathcal{L}(\theta)+\langle\partial\mathcal{L}(\theta),\hat{\theta}-\theta\rangle+\frac{\mu}{4}d^2 \ge\mathcal{L}(\theta)-\frac{1}{\mu}\lvert\partial\mathcal{L}(\theta)\rvert_2^2. L ( θ ^ ) ≥ L ( θ ) + ⟨ ∂ L ( θ ) , θ ^ − θ ⟩ + 4 μ d 2 ≥ L ( θ ) − μ 1 ∣ ∂ L ( θ ) ∣ 2 2 .
Finally, (22) and d 2 ≥ 2 ϵ 2 2 d^2 \ge2\epsilon_2^2 d 2 ≥ 2 ϵ 2 2 give L ( θ ) − L ( θ ^ ) ≥ μ d 2 / 8 \mathcal{L}(\theta)-\mathcal{L}(\hat{\theta}) \ge\mu d^2/8 L ( θ ) − L ( θ ^ ) ≥ μ d 2 /8 , completing the chain.
Lemma 4
Lemma 4. The sequence in Theorem 2(i) satisfies λ t ≥ ( t + 1 ) / 2 \lambda_t \ge(t+1)/2 λ t ≥ ( t + 1 ) /2 .
Proof. The initial value is λ 0 = 1 \lambda_0=1 λ 0 = 1 , and the recursion gives λ t + 1 ≥ λ t + 1 / 2 \lambda_{t+1}\ge\lambda_t+1/2 λ t + 1 ≥ λ t + 1/2 . Induction proves the bound.
Let 0 < m H ≤ M H 0<m_H\le M_H 0 < m H ≤ M H , m H I ⪯ H ⪯ M H I m_H\mathbb{I}\preceq H\preceq M_H\mathbb{I} m H I ⪯ H ⪯ M H I , γ ∈ ( 0 , 1 ) \gamma\in(0,1) γ ∈ ( 0 , 1 ) , and 0 < η ≤ { ( 1 − γ ) / ( 1 + γ ) } 2 M H − 1 0<\eta\le\{(1-\gamma)/(1+\gamma)\}^2M_H^{-1} 0 < η ≤ {( 1 − γ ) / ( 1 + γ ) } 2 M H − 1 . Write M = I − η H M=\mathbb{I}-\eta H M = I − η H and z 0 = θ ( 0 ) − θ ∗ + H − 1 ∂ L ( θ ∗ ) z_0=\theta^{(0)}-\theta^*+H^{-1}\partial\mathcal{L}(\theta^*) z 0 = θ ( 0 ) − θ ∗ + H − 1 ∂ L ( θ ∗ ) . Define ρ 1 , ρ 2 = { ( 1 + γ ) M ± ( 1 + γ ) 2 M 2 − 4 γ M } / 2 \rho_1,\rho_2=\{(1+\gamma)M\pm\sqrt{(1+\gamma)^2M^2-4\gamma M}\}/2 ρ 1 , ρ 2 = {( 1 + γ ) M ± ( 1 + γ ) 2 M 2 − 4 γ M } /2 , q ρ = ∥ ρ 1 ∥ q_\rho=\|\rho_1\| q ρ = ∥ ρ 1 ∥ , d ρ = Λ min ( ρ 1 − ρ 2 ) d_\rho=\Lambda_{\min}(\rho_1-\rho_2) d ρ = Λ m i n ( ρ 1 − ρ 2 ) , and
A ρ = max { m H − 1 , η min { 1 ( 1 − q ρ ) 2 , q ρ d ρ ( 1 − q ρ ) } } . A_\rho=\max\left\{m_H^{-1},\eta\min\left\{\frac{1}{(1-q_\rho)^2},\frac{q_\rho}{d_\rho(1-q_\rho)}\right\}\right\}. A ρ = max { m H − 1 , η min { ( 1 − q ρ ) 2 1 , d ρ ( 1 − q ρ ) q ρ } } .
with the convention 1 / 0 = ∞ 1/0=\infty 1/0 = ∞ . Let P 0 = I P_0=\mathbb{I} P 0 = I , P 1 = M P_1=M P 1 = M , and P t + 1 = ( 1 + γ ) M P t − γ M P t − 1 P_{t+1}=(1+\gamma)MP_t-\gamma MP_{t-1} P t + 1 = ( 1 + γ ) M P t − γ M P t − 1 .
Proposition 4. Suppose Condition 3 holds with remainder r n r_n r n . Restart Algorithm 3 with step size η \eta η and constant momentum γ t ≡ γ \gamma_t\equiv\gamma γ t ≡ γ . Assume ∣ θ ( 0 ) − θ ∗ ∣ 2 2 ≤ ϵ 3 / [ 2 ( 1 + 2 γ ) ] \lvert\theta^{(0)}-\theta^*\rvert_2^2\le\epsilon_3/[2(1+2\gamma)] ∣ θ ( 0 ) − θ ∗ ∣ 2 2 ≤ ϵ 3 / [ 2 ( 1 + 2 γ )] and A ρ { ∣ ∂ L ( θ ∗ ) ∣ 2 2 + r n } ≤ ϵ 3 / [ 2 ( 1 + 2 γ ) ] A_\rho\{\lvert\partial\mathcal{L}(\theta^*)\rvert_2^2+r_n\}\le\epsilon_3/[2(1+2\gamma)] A ρ {∣ ∂ L ( θ ∗ ) ∣ 2 2 + r n } ≤ ϵ 3 / [ 2 ( 1 + 2 γ )] . Then ∣ θ ( t ) − θ ∗ ∣ 2 2 ≤ ϵ 3 / ( 1 + 2 γ ) \lvert\theta^{(t)}-\theta^*\rvert_2^2\le\epsilon_3/(1+2\gamma) ∣ θ ( t ) − θ ∗ ∣ 2 2 ≤ ϵ 3 / ( 1 + 2 γ ) and ∣ β ( t ) − θ ∗ ∣ 2 2 ≤ ϵ 3 \lvert\beta^{(t)}-\theta^*\rvert_2^2\le\epsilon_3 ∣ β ( t ) − θ ∗ ∣ 2 2 ≤ ϵ 3 for every t ≥ 0 t\ge0 t ≥ 0 , and
θ ( T ) − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + P T z 0 + R T , (23) \theta^{(T)}-\theta^*=-H^{-1}\partial\mathcal{L}(\theta^*)+P_Tz_0+R_T, \tag*{(23)} θ ( T ) − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + P T z 0 + R T , (23) where ∥ P T ∥ ≤ min { 1 , ( T + 1 ) q ρ T , 2 q ρ T / d ρ } \|P_T\|\le\min\{1,(T+1)q_\rho^T,2q_\rho^T/d_\rho\} ∥ P T ∥ ≤ min { 1 , ( T + 1 ) q ρ T , 2 q ρ T / d ρ } and ∣ R T ∣ 2 ≤ A ρ r n \lvert R_T\rvert_2\le A_\rho r_n ∣ R T ∣ 2 ≤ A ρ r n . The geometric factor satisfies q ρ < ∥ I − η H ∥ q_\rho<\|\mathbb{I}-\eta H\| q ρ < ∥ I − η H ∥ .
Proof of Proposition 4. Step 1. The homogeneous recursion. All matrices below are functions of H H H and commute. The step-size inequality gives 0 ≺ ρ 2 ⪯ ρ 1 ≺ M ≺ I 0\prec\rho_2\preceq\rho_1\prec M\prec\mathbb{I} 0 ≺ ρ 2 ⪯ ρ 1 ≺ M ≺ I . Define
K j = ∑ k = 0 j ρ 1 k ρ 2 j − k ( j ≥ 0 ) , K − 1 = 0. K_j=\sum_{k=0}^{j}\rho_1^k\rho_2^{j-k}\quad(j\ge0),\qquad K_{-1}=0. K j = k = 0 ∑ j ρ 1 k ρ 2 j − k ( j ≥ 0 ) , K − 1 = 0.
This formula includes coincident roots. If d ρ > 0 d_\rho>0 d ρ > 0 , it also equals ( ρ 1 j + 1 − ρ 2 j + 1 ) ( ρ 1 − ρ 2 ) − 1 (\rho_1^{j+1}-\rho_2^{j+1})(\rho_1-\rho_2)^{-1} ( ρ 1 j + 1 − ρ 2 j + 1 ) ( ρ 1 − ρ 2 ) − 1 . The matrices are positive semidefinite, and
∥ K j ∥ ≤ min { ( j + 1 ) q ρ j , q ρ j + 1 / d ρ } , η ∑ j = 0 ∞ ∥ K j ∥ ≤ A ρ , η ∑ j = 0 ∞ K j = H − 1 . \|K_j\|\le\min\{(j+1)q_\rho^j,q_\rho^{j+1}/d_\rho\},\qquad\eta\sum_{j=0}^{\infty}\|K_j\|\le A_\rho,\qquad\eta\sum_{j=0}^{\infty}K_j=H^{-1}. ∥ K j ∥ ≤ min {( j + 1 ) q ρ j , q ρ j + 1 / d ρ } , η j = 0 ∑ ∞ ∥ K j ∥ ≤ A ρ , η j = 0 ∑ ∞ K j = H − 1 .
The two norm bounds follow from the power sum and, when d ρ > 0 d_\rho>0 d ρ > 0 , the difference formula. Summing them gives the bound by A ρ A_\rho A ρ . The last identity follows from ( I − ρ 1 ) ( I − ρ 2 ) = η H (\mathbb{I}-\rho_1)(\mathbb{I}-\rho_2)=\eta H ( I − ρ 1 ) ( I − ρ 2 ) = η H . The polynomial recursion gives
P t = K t − γ M K t − 1 , P t − P t + 1 = η H K t . P_t=K_t-\gamma MK_{t-1},\qquad P_t-P_{t+1}=\eta HK_t. P t = K t − γ M K t − 1 , P t − P t + 1 = η H K t .
Since q ρ < 1 q_\rho<1 q ρ < 1 , the first identity gives P t → 0 P_t\to0 P t → 0 . The second then implies 0 ⪯ P t ⪯ I 0\preceq P_t\preceq\mathbb{I} 0 ⪯ P t ⪯ I . Also P t ⪯ K t P_t\preceq K_t P t ⪯ K t , and ∥ P t ∥ ≤ min { 1 , ( t + 1 ) q ρ t , 2 q ρ t / d ρ } \|P_t\|\le\min\{1,(t+1)q_\rho^t,2q_\rho^t/d_\rho\} ∥ P t ∥ ≤ min { 1 , ( t + 1 ) q ρ t , 2 q ρ t / d ρ } . The convention for d ρ = 0 d_\rho=0 d ρ = 0 retains the finite bounds from the power sum at coincident roots.
Step 2. Containment of the iterates and extrapolates. For this proof, define θ ( − 1 ) = θ ( 0 ) \theta^{(-1)}=\theta^{(0)} θ ( − 1 ) = θ ( 0 ) . Write g ∗ = ∂ L ( θ ∗ ) g^*=\partial\mathcal{L}(\theta^*) g ∗ = ∂ L ( θ ∗ ) and x t = θ ( t ) − θ ∗ x_t=\theta^{(t)}-\theta^* x t = θ ( t ) − θ ∗ . If β ( j ) \beta^{(j)} β ( j ) lies in the radius-ϵ 3 \epsilon_3 ϵ 3 ball, define
e j = ∂ L ( β ( j ) ) − g ∗ − H ( β ( j ) − θ ∗ ) , ∣ e j ∣ 2 ≤ r n . e_j=\partial\mathcal{L}(\beta^{(j)})-g^*-H(\beta^{(j)}-\theta^*),\qquad|e_j|_2\le r_n. e j = ∂ L ( β ( j ) ) − g ∗ − H ( β ( j ) − θ ∗ ) , ∣ e j ∣ 2 ≤ r n .
For every index up to which these bounds hold, the restarted recursion has solution
x t = P t x 0 − η ∑ j = 0 t − 1 K t − 1 − j ( g ∗ + e j ) . x_t=P_tx_0-\eta\sum_{j=0}^{t-1}K_{t-1-j}(g^*+e_j). x t = P t x 0 − η j = 0 ∑ t − 1 K t − 1 − j ( g ∗ + e j ) .
Consequently ∣ x t ∣ 2 ≤ ∣ x 0 ∣ 2 + A ρ ( ∣ g ∗ ∣ 2 + r n ) ≤ ϵ 3 / ( 1 + 2 γ ) |x_t|_2\le|x_0|_2+A_\rho(|g^*|_2+r_n)\le\epsilon_3/(1+2\gamma) ∣ x t ∣ 2 ≤ ∣ x 0 ∣ 2 + A ρ ( ∣ g ∗ ∣ 2 + r n ) ≤ ϵ 3 / ( 1 + 2 γ ) . The restart gives the same bound at t = − 1 t=-1 t = − 1 . Hence
∣ β ( t ) − θ ∗ ∣ 2 ≤ ( 1 + γ ) ∣ x t ∣ 2 + γ ∣ x t − 1 ∣ 2 ≤ ϵ 3 . |\beta^{(t)}-\theta^*|_2\le(1+\gamma)|x_t|_2+\gamma|x_{t-1}|_2\le\epsilon_3. ∣ β ( t ) − θ ∗ ∣ 2 ≤ ( 1 + γ ) ∣ x t ∣ 2 + γ ∣ x t − 1 ∣ 2 ≤ ϵ 3 .
Starting with β ( 0 ) = θ ( 0 ) \beta^{(0)}=\theta^{(0)} β ( 0 ) = θ ( 0 ) , induction validates the recursion at every update.
Step 3. The output expansion. The identities in Step 1 give η ∑ j = 0 T − 1 K j = ( I − P T ) H − 1 \eta\sum_{j=0}^{T-1}K_j=(I-P_T)H^{-1} η ∑ j = 0 T − 1 K j = ( I − P T ) H − 1 . Separating the constant input g ∗ g^* g ∗ therefore yields
x T = − H − 1 g ∗ + P T ( x 0 + H − 1 g ∗ ) − η ∑ j = 0 T − 1 K T − 1 − j e j . x_T=-H^{-1}g^*+P_T(x_0+H^{-1}g^*)-\eta\sum_{j=0}^{T-1}K_{T-1-j}e_j. x T = − H − 1 g ∗ + P T ( x 0 + H − 1 g ∗ ) − η j = 0 ∑ T − 1 K T − 1 − j e j .
The last term has norm at most A ρ r n A_\rho r_n A ρ r n , and the bound for P T P_T P T gives the stated transient bound, including coincident roots. The strict inequality ρ 1 ≺ M \rho_1\prec M ρ 1 ≺ M gives the stated rate comparison at a common step size. □ \square □
Lemma 5. Let 0 < m H ≤ M H 0<m_H\le M_H 0 < m H ≤ M H , m H I ⪯ H ⪯ M H I m_HI\preceq H\preceq M_HI m H I ⪯ H ⪯ M H I , 0 < η ≤ M H − 1 0<\eta\le M_H^{-1} 0 < η ≤ M H − 1 , and γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) . Write M = I − η H M=I-\eta H M = I − η H . For an eigenvalue m m m of M M M , let r ( γ , m ) r(\gamma,m) r ( γ , m ) denote the largest modulus of the roots of
x 2 − ( 1 + γ ) m x + γ m = 0 , x^2-(1+\gamma)mx+\gamma m=0, x 2 − ( 1 + γ ) m x + γ m = 0 , and define q ( η , γ ) = max m ∈ spec ( M ) r ( γ , m ) q(\eta,\gamma)=\max_{m\in\operatorname{spec}(M)}r(\gamma,m) q ( η , γ ) = max m ∈ spec ( M ) r ( γ , m ) .
(i) If η ≤ { ( 1 − γ ) / ( 1 + γ ) } 2 M H − 1 \eta\le\left\{(1-\gamma)/(1+\gamma)\right\}^2M_H^{-1} η ≤ { ( 1 − γ ) / ( 1 + γ ) } 2 M H − 1 , the matrices ρ 1 , ρ 2 \rho_1,\rho_2 ρ 1 , ρ 2 given by the root formula preceding Proposition 4 satisfy
γ I ⪯ ρ 2 ⪯ 2 γ 1 + γ I ⪯ 1 + γ 2 M ⪯ ρ 1 ⪯ M . \gamma I\preceq\rho_2\preceq\frac{2\gamma}{1+\gamma}I\preceq\frac{1+\gamma}{2}M\preceq\rho_1\preceq M. γ I ⪯ ρ 2 ⪯ 1 + γ 2 γ I ⪯ 2 1 + γ M ⪯ ρ 1 ⪯ M . Consequently q ( η , γ ) = ∥ ρ 1 ∥ ≤ 1 − η m H q(\eta,\gamma)=\|\rho_1\|\le1-\eta m_H q ( η , γ ) = ∥ ρ 1 ∥ ≤ 1 − η m H .
(ii) If γ = ( 1 − η m H ) / ( 1 + η m H ) \gamma=(1-\sqrt{\eta m_H})/(1+\sqrt{\eta m_H}) γ = ( 1 − η m H ) / ( 1 + η m H ) , then q ( η , γ ) ≤ 1 − η m H q(\eta,\gamma)\le1-\sqrt{\eta m_H} q ( η , γ ) ≤ 1 − η m H . If m H = Λ min ( H ) m_H=\Lambda_{\min}(H) m H = Λ m i n ( H ) , equality holds and this momentum minimizes q ( η , γ ) q(\eta,\gamma) q ( η , γ ) over γ ∈ [ 0 , 1 ) \gamma\in[0,1) γ ∈ [ 0 , 1 ) .
Proof of Lemma 5. All matrices are functions of H H H and commute. It therefore suffices to compare scalar eigenvalues. For part (i), m ≥ 4 γ / ( 1 + γ ) 2 m\ge4\gamma/(1+\gamma)^2 m ≥ 4 γ / ( 1 + γ ) 2 gives real nonnegative roots. The case γ = 0 \gamma=0 γ = 0 is immediate. If γ > 0 \gamma>0 γ > 0 , the root formula and m ≤ 1 m\le1 m ≤ 1 give
1 + γ 2 m ≤ x 1 ≤ m , x 2 = 2 γ m ( 1 + γ ) m + ( 1 + γ ) 2 m 2 − 4 γ m , γ ≤ x 2 ≤ 2 γ 1 + γ . \frac{1+\gamma}{2}m\le x_1\le m,\qquad x_2=\frac{2\gamma m}{(1+\gamma)m+\sqrt{(1+\gamma)^2m^2-4\gamma m}},\qquad\gamma\le x_2\le\frac{2\gamma}{1+\gamma}. 2 1 + γ m ≤ x 1 ≤ m , x 2 = ( 1 + γ ) m + ( 1 + γ ) 2 m 2 − 4 γ m 2 γ m , γ ≤ x 2 ≤ 1 + γ 2 γ .
The lower bound on m m m completes the displayed chain.
For part (ii), write a = η m H a=\sqrt{\eta m_H} a = η m H . If a = 1 a=1 a = 1 , then M = 0 M=0 M = 0 and the conclusion follows. Otherwise γ = ( 1 − a ) / ( 1 + a ) \gamma=(1-a)/(1+a) γ = ( 1 − a ) / ( 1 + a ) and 4 γ / ( 1 + γ ) 2 = 1 − a 2 4\gamma/(1+\gamma)^2=1-a^2 4 γ / ( 1 + γ ) 2 = 1 − a 2 . For 0 < m < 1 − a 2 0<m<1-a^2 0 < m < 1 − a 2 , the roots are complex conjugates with modulus γ m < 1 − a \sqrt{\gamma m}<1-a γ m < 1 − a . At m = 1 − a 2 m=1-a^2 m = 1 − a 2 , both roots equal 1 − a 1-a 1 − a , and at m = 0 m=0 m = 0 they are zero. This proves the spectral bound and equality when m H = Λ min ( H ) m_H=\Lambda_{\min}(H) m H = Λ m i n ( H ) . To verify optimality in the latter case, fix m ∗ = 1 − η m H m_*=1-\eta m_H m ∗ = 1 − η m H . Below the stated momentum, its larger real root decreases with γ \gamma γ , since implicit differentiation gives
∂ x 1 ∂ γ = m ∗ ( x 1 − 1 ) 2 x 1 − ( 1 + γ ) m ∗ < 0. \frac{\partial x_1}{\partial\gamma}=\frac{m_*(x_1-1)}{2x_1-(1+\gamma)m_*}<0. ∂ γ ∂ x 1 = 2 x 1 − ( 1 + γ ) m ∗ m ∗ ( x 1 − 1 ) < 0.
Above that momentum, its root modulus is γ m ∗ \sqrt{\gamma m_*} γ m ∗ and increases with γ \gamma γ . The bound already proved holds simultaneously for every eigenvalue, establishing optimality. □ \square □
Proofs of the convergence bounds
Proof of Theorem 1(i). Write Δ t = L ( θ ( t ) ) − L ( θ ^ ) \Delta_t = \mathcal{L}(\theta^{(t)}) - \mathcal{L}(\hat{\theta}) Δ t = L ( θ ( t ) ) − L ( θ ^ ) and g t = ∂ L ( θ ( t ) ) \mathbf{g}_t = \partial\mathcal{L}(\theta^{(t)}) g t = ∂ L ( θ ( t ) ) . Convexity gives
Δ t ≤ ⟨ g t , θ ( t ) − θ ^ ⟩ . (24) \Delta_t \le\langle\mathbf{g}_t,\theta^{(t)}-\hat{\theta}\rangle. \tag*{(24)} Δ t ≤ ⟨ g t , θ ( t ) − θ ^ ⟩ . (24)
Combining this inequality with (3) and expanding the squared distance along θ ( t + 1 ) = θ ( t ) − η g t \theta^{(t+1)} = \theta^{(t)} - \eta\mathbf{g}_t θ ( t + 1 ) = θ ( t ) − η g t yields
Δ t + 1 ≤ ∣ θ ( t ) − θ ^ ∣ 2 2 − ∣ θ ( t + 1 ) − θ ^ ∣ 2 2 2 η + η L 2 ϵ 1 2 2 . (25) \Delta_{t+1} \le\frac{\lvert\theta^{(t)}-\hat{\theta}\rvert_2^2-\lvert\theta^{(t+1)}-\hat{\theta}\rvert_2^2}{2\eta}+\frac{\eta L^2\epsilon_1^2}{2}. \tag*{(25)} Δ t + 1 ≤ 2 η ∣ θ ( t ) − θ ^ ∣ 2 2 − ∣ θ ( t + 1 ) − θ ^ ∣ 2 2 + 2 η L 2 ϵ 1 2 . (25)
Summing over t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 proves the averaged bound. If every update has length at least ϵ 1 \epsilon_1 ϵ 1 , (2) gives
L ( θ ( t + 1 ) ) − L ( θ ( t ) ) ≤ − ( η − η 2 L ) ∣ g t ∣ 2 2 . (26) \mathcal{L}(\theta^{(t+1)})-\mathcal{L}(\theta^{(t)})\le-(\eta-\eta^2L)\lvert\mathbf{g}_t\rvert_2^2. \tag*{(26)} L ( θ ( t + 1 ) ) − L ( θ ( t ) ) ≤ − ( η − η 2 L ) ∣ g t ∣ 2 2 . (26)
For η ≤ 1 / ( 2 L ) \eta\le1/(2L) η ≤ 1/ ( 2 L ) , this proves objective descent and removes the additive term from (25) . Nonnegativity of Δ t + 1 \Delta_{t+1} Δ t + 1 then makes the distances nonincreasing. Summation and objective descent give Δ T ≤ T − 1 ∑ t = 1 T Δ t ≤ ∣ θ ( 0 ) − θ ^ ∣ 2 2 / ( 2 η T ) \Delta_T\le T^{-1}\sum_{t=1}^{T}\Delta_t\le\lvert\theta^{(0)}-\hat{\theta}\rvert_2^2/(2\eta T) Δ T ≤ T − 1 ∑ t = 1 T Δ t ≤ ∣ θ ( 0 ) − θ ^ ∣ 2 2 / ( 2 η T ) .
Proof of Theorem 1(ii). Let Δ t = L ( θ ( t ) ) − L ( θ ^ ) \Delta_t=\mathcal{L}(\theta^{(t)})-\mathcal{L}(\hat{\theta}) Δ t = L ( θ ( t ) ) − L ( θ ^ ) and g t = ∂ L ( θ ( t ) ) \mathbf{g}_t=\partial\mathcal{L}(\theta^{(t)}) g t = ∂ L ( θ ( t ) ) . Inequality (21) and completion of the square give
Δ t ≤ ∣ g t ∣ 2 ∣ θ ( t ) − θ ^ ∣ 2 − μ 4 ∣ θ ( t ) − θ ^ ∣ 2 2 + μ 4 ϵ 2 2 ≤ ∣ g t ∣ 2 2 μ + μ 4 ϵ 2 2 . \Delta_t\le\lvert\mathbf{g}_t\rvert_2\lvert\theta^{(t)}-\hat{\theta}\rvert_2-\frac{\mu}{4}\lvert\theta^{(t)}-\hat{\theta}\rvert_2^2+\frac{\mu}{4}\epsilon_2^2\le\frac{\lvert\mathbf{g}_t\rvert_2^2}{\mu}+\frac{\mu}{4}\epsilon_2^2. Δ t ≤ ∣ g t ∣ 2 ∣ θ ( t ) − θ ^ ∣ 2 − 4 μ ∣ θ ( t ) − θ ^ ∣ 2 2 + 4 μ ϵ 2 2 ≤ μ ∣ g t ∣ 2 2 + 4 μ ϵ 2 2 .
With c η = μ ( η − L η 2 / 2 ) c_\eta=\mu(\eta-L\eta^2/2) c η = μ ( η − L η 2 /2 ) , applying (20) along the update therefore yields
Δ t + 1 ≤ ( 1 − c η ) Δ t + c η μ ϵ 2 2 4 + L 2 ϵ 1 2 . \Delta_{t+1}\le(1-c_\eta)\Delta_t+\frac{c_\eta\mu\epsilon_2^2}{4}+\frac{L}{2}\epsilon_1^2. Δ t + 1 ≤ ( 1 − c η ) Δ t + 4 c η μ ϵ 2 2 + 2 L ϵ 1 2 .
Here 0 < c η ≤ μ / ( 2 L ) ≤ 1 0<c_\eta\le\mu/(2L)\le1 0 < c η ≤ μ / ( 2 L ) ≤ 1 . Summing the geometric recursion gives the bound with additive terms. Under the two distance conditions, Condition 2 gives the first display without its ϵ 2 \epsilon_2 ϵ 2 term, hence ∣ g t ∣ 2 2 ≥ μ Δ t \lvert\mathbf{g}_t\rvert_2^2\ge\mu\Delta_t ∣ g t ∣ 2 2 ≥ μ Δ t . Equation (26) then gives Δ t + 1 ≤ { 1 − μ ( η − η 2 L ) } Δ t \Delta_{t+1}\le\{1-\mu(\eta-\eta^2L)\}\Delta_t Δ t + 1 ≤ { 1 − μ ( η − η 2 L )} Δ t . Iteration proves the remaining bound whenever 0 < μ ( η − η 2 L ) ≤ 1 0<\mu(\eta-\eta^2L)\le1 0 < μ ( η − η 2 L ) ≤ 1 .
Proof of Theorem 1(iii). Write g ∗ = ∂ L ( θ ∗ ) \mathbf{g}^*=\partial\mathcal{L}(\theta^*) g ∗ = ∂ L ( θ ∗ ) and M = I − η H M=I-\eta H M = I − η H . If ∣ θ ( t ) − θ ∗ ∣ 2 ≤ ϵ 3 / 2 \lvert\theta^{(t)}-\theta^*\rvert_2\le\epsilon_3/2 ∣ θ ( t ) − θ ∗ ∣ 2 ≤ ϵ 3 /2 , the local subgradient approximation gives
θ ( t + 1 ) − θ ∗ = M ( θ ( t ) − θ ∗ ) − η g ∗ − η e t , ∣ e t ∣ 2 ≤ r n . \theta^{(t+1)}-\theta^*=M(\theta^{(t)}-\theta^*)-\eta\mathbf{g}^*-\eta\mathbf{e}_t,\qquad\lvert\mathbf{e}_t\rvert_2\le r_n. θ ( t + 1 ) − θ ∗ = M ( θ ( t ) − θ ∗ ) − η g ∗ − η e t , ∣ e t ∣ 2 ≤ r n .
Whenever ∣ θ ( t ) − θ ∗ ∣ 2 ≤ ϵ 3 / 2 \lvert\theta^{(t)}-\theta^*\rvert_2\le\epsilon_3/2 ∣ θ ( t ) − θ ∗ ∣ 2 ≤ ϵ 3 /2 , the bound ∥ M ∥ ≤ 1 − η m H \lVert M\rVert\le1-\eta m_H ∥ M ∥ ≤ 1 − η m H implies
∣ θ ( t + 1 ) − θ ∗ ∣ 2 ≤ ( 1 − η m H ) ϵ 3 / 2 + η ( ∣ g ∗ ∣ 2 + r n ) ≤ ϵ 3 / 2. \lvert\theta^{(t+1)}-\theta^*\rvert_2\le(1-\eta m_H)\epsilon_3/2+\eta(\lvert\mathbf{g}^*\rvert_2+r_n)\le\epsilon_3/2. ∣ θ ( t + 1 ) − θ ∗ ∣ 2 ≤ ( 1 − η m H ) ϵ 3 /2 + η (∣ g ∗ ∣ 2 + r n ) ≤ ϵ 3 /2.
Induction keeps every iterate in this ball. Solving the recursion yields (5) with
R T = − η ∑ t = 0 T − 1 M T − 1 − t e t , ∣ R T ∣ 2 ≤ η r n ∑ j = 0 T − 1 ( 1 − η m H ) j ≤ r n / m H . R_T=-\eta\sum_{t=0}^{T-1}M^{T-1-t}\mathbf{e}_t,\qquad\lvert R_T\rvert_2\le\eta r_n\sum_{j=0}^{T-1}(1-\eta m_H)^j\le r_n/m_H. R T = − η t = 0 ∑ T − 1 M T − 1 − t e t , ∣ R T ∣ 2 ≤ η r n j = 0 ∑ T − 1 ( 1 − η m H ) j ≤ r n / m H .
Proof of Theorem 2(i). For this proof, define λ − 1 = 0 \lambda_{-1}=0 λ − 1 = 0 , γ − 1 = − 1 \gamma_{-1}=-1 γ − 1 = − 1 , and θ ( − 1 ) = θ ( 0 ) \boldsymbol{\theta}^{(-1)}=\boldsymbol{\theta}^{(0)} θ ( − 1 ) = θ ( 0 ) . Write Δ t = L ( θ ( t ) ) − L ( θ ^ ) \Delta_t=\mathcal{L}(\boldsymbol{\theta}^{(t)})-\mathcal{L}(\hat{\boldsymbol{\theta}}) Δ t = L ( θ ( t ) ) − L ( θ ^ ) , g t = ∂ L ( β ( t ) ) \boldsymbol{g}_t=\partial\mathcal{L}(\boldsymbol{\beta}^{(t)}) g t = ∂ L ( β ( t ) ) , and u t = λ t − 1 θ ( t ) − ( λ t − 1 − 1 ) θ ( t − 1 ) \boldsymbol{u}_t=\lambda_{t-1}\boldsymbol{\theta}^{(t)}-(\lambda_{t-1}-1)\boldsymbol{\theta}^{(t-1)} u t = λ t − 1 θ ( t ) − ( λ t − 1 − 1 ) θ ( t − 1 ) . Convexity gives
L ( β ( t ) ) − L ( θ ^ ) ≤ ⟨ g t , β ( t ) − θ ^ ⟩ , (27) \mathcal{L}(\boldsymbol{\beta}^{(t)})-\mathcal{L}(\hat{\boldsymbol{\theta}})\leq\langle\boldsymbol{g}_t,\boldsymbol{\beta}^{(t)}-\hat{\boldsymbol{\theta}}\rangle, \tag*{(27)} L ( β ( t ) ) − L ( θ ^ ) ≤ ⟨ g t , β ( t ) − θ ^ ⟩ , (27)
L ( β ( t ) ) − L ( θ ( t ) ) ≤ ⟨ g t , β ( t ) − θ ( t ) ⟩ . (28) \mathcal{L}(\boldsymbol{\beta}^{(t)})-\mathcal{L}(\boldsymbol{\theta}^{(t)})\leq\langle\boldsymbol{g}_t,\boldsymbol{\beta}^{(t)}-\boldsymbol{\theta}^{(t)}\rangle. \tag*{(28)} L ( β ( t ) ) − L ( θ ( t ) ) ≤ ⟨ g t , β ( t ) − θ ( t ) ⟩ . (28)
The momentum and update identities imply λ t ( λ t − 1 ) = λ t − 1 2 \lambda_t(\lambda_t-1)=\lambda_{t-1}^2 λ t ( λ t − 1 ) = λ t − 1 2 , λ t β ( t ) − ( λ t − 1 ) θ ( t ) = u t \lambda_t\boldsymbol{\beta}^{(t)}-(\lambda_t-1)\boldsymbol{\theta}^{(t)}=\boldsymbol{u}_t λ t β ( t ) − ( λ t − 1 ) θ ( t ) = u t , and u t + 1 = u t − η λ t g t \boldsymbol{u}_{t+1}=\boldsymbol{u}_t-\eta\lambda_t\boldsymbol{g}_t u t + 1 = u t − η λ t g t . Weight the two convexity bounds by λ t \lambda_t λ t and λ t ( λ t − 1 ) \lambda_t(\lambda_t-1) λ t ( λ t − 1 ) . Combining them with the one-step model multiplied by λ t 2 \lambda_t^2 λ t 2 and completing the square gives
λ t 2 Δ t + 1 − λ t − 1 2 Δ t ≤ ∥ u t − θ ^ ∥ 2 2 − ∥ u t + 1 − θ ^ ∥ 2 2 2 η + λ t 2 δ t . (29) \lambda_t^2\Delta_{t+1}-\lambda_{t-1}^2\Delta_t\leq\frac{\lVert\boldsymbol{u}_t-\hat{\boldsymbol{\theta}}\rVert_2^2-\lVert\boldsymbol{u}_{t+1}-\hat{\boldsymbol{\theta}}\rVert_2^2}{2\eta}+\lambda_t^2\delta_t. \tag*{(29)} λ t 2 Δ t + 1 − λ t − 1 2 Δ t ≤ 2 η ∥ u t − θ ^ ∥ 2 2 − ∥ u t + 1 − θ ^ ∥ 2 2 + λ t 2 δ t . (29)
Since u 0 = θ ( 0 ) \boldsymbol{u}_0=\boldsymbol{\theta}^{(0)} u 0 = θ ( 0 ) , summation proves (6). Under Condition 1 and η ≤ 1 / ( 2 L ) \eta\leq1/(2L) η ≤ 1/ ( 2 L ) , (3) supplies the stated δ t \delta_t δ t . Monotonicity of λ t \lambda_t λ t gives ∑ t < T λ t 2 δ t / λ T − 1 2 ≤ T η L 2 ϵ 1 2 / 2 \sum_{t<T}\lambda_t^2\delta_t/\lambda_{T-1}^2\leq T\eta L^2\epsilon_1^2/2 ∑ t < T λ t 2 δ t / λ T − 1 2 ≤ T η L 2 ϵ 1 2 /2 , and Lemma 4 gives the T − 2 T^{-2} T − 2 term. For an update of length at least ϵ 1 \epsilon_1 ϵ 1 , (2) gives
L ( θ ( t + 1 ) ) ≤ L ( β ( t ) ) − ( η − η 2 L ) ∥ g t ∥ 2 2 ≤ L ( β ( t ) ) − η 2 ∥ g t ∥ 2 2 . (30) \mathcal{L}(\boldsymbol{\theta}^{(t+1)})\leq\mathcal{L}(\boldsymbol{\beta}^{(t)})-(\eta-\eta^2L)\lVert\boldsymbol{g}_t\rVert_2^2\leq\mathcal{L}(\boldsymbol{\beta}^{(t)})-\frac{\eta}{2}\lVert\boldsymbol{g}_t\rVert_2^2. \tag*{(30)} L ( θ ( t + 1 ) ) ≤ L ( β ( t ) ) − ( η − η 2 L ) ∥ g t ∥ 2 2 ≤ L ( β ( t ) ) − 2 η ∥ g t ∥ 2 2 . (30)
Thus all δ t \delta_t δ t vanish when every update is long enough, proving the error-free objective bound. Before the first update shorter than ϵ 1 \epsilon_1 ϵ 1 , (29) and the momentum identities give, with d 0 = ∥ θ ( 0 ) − θ ^ ∥ 2 d_0=\lVert\boldsymbol{\theta}^{(0)}-\hat{\boldsymbol{\theta}}\rVert_2 d 0 = ∥ θ ( 0 ) − θ ^ ∥ 2 ,
λ t − 1 2 { L ( θ ( t ) ) − L ( θ ^ ) } + ∥ u t − θ ^ ∥ 2 2 2 η ≤ d 0 2 2 η , \lambda_{t-1}^2\{\mathcal{L}(\boldsymbol{\theta}^{(t)})-\mathcal{L}(\hat{\boldsymbol{\theta}})\}+\frac{\lVert\boldsymbol{u}_t-\hat{\boldsymbol{\theta}}\rVert_2^2}{2\eta}\leq\frac{d_0^2}{2\eta}, λ t − 1 2 { L ( θ ( t ) ) − L ( θ ^ )} + 2 η ∥ u t − θ ^ ∥ 2 2 ≤ 2 η d 0 2 ,
θ ( t ) = ( 1 − 1 λ t − 1 ) θ ( t − 1 ) + u t λ t − 1 , ( t ≥ 1 ) , β ( t ) = ( 1 − 1 λ t ) θ ( t ) + u t λ t . (31) \begin{aligned}
\boldsymbol{\theta}^{(t)}&=\left(1-\frac{1}{\lambda_{t-1}}\right)\boldsymbol{\theta}^{(t-1)}+\frac{\boldsymbol{u}_t}{\lambda_{t-1}},\qquad(t\geq1),\\
\boldsymbol{\beta}^{(t)}&=\left(1-\frac{1}{\lambda_t}\right)\boldsymbol{\theta}^{(t)}+\frac{\boldsymbol{u}_t}{\lambda_t}. \tag*{(31)}
\end{aligned} θ ( t ) β ( t ) = ( 1 − λ t − 1 1 ) θ ( t − 1 ) + λ t − 1 u t , ( t ≥ 1 ) , = ( 1 − λ t 1 ) θ ( t ) + λ t u t . (31)
Since λ t ≥ 1 \lambda_t\geq1 λ t ≥ 1 and u 0 = θ ( 0 ) \boldsymbol{u}_0=\boldsymbol{\theta}^{(0)} u 0 = θ ( 0 ) , the last two identities are convex combinations. Nonnegativity of the objective gap and induction therefore keep u t \boldsymbol{u}_t u t , θ ( t ) \boldsymbol{\theta}^{(t)} θ ( t ) , and β ( t ) \boldsymbol{\beta}^{(t)} β ( t ) within distance d 0 d_0 d 0 of θ ^ \hat{\boldsymbol{\theta}} θ ^ whenever the preceding updates satisfy the distance and domain conditions. □
Proof of Theorem 2(ii). Define θ ( − 1 ) = θ ( 0 ) \boldsymbol{\theta}^{(-1)}=\boldsymbol{\theta}^{(0)} θ ( − 1 ) = θ ( 0 ) , g t = ∂ L ( β ( t ) ) \boldsymbol{g}_t=\partial\mathcal{L}(\boldsymbol{\beta}^{(t)}) g t = ∂ L ( β ( t ) ) , and Δ t = L ( θ ( t ) ) − L ( θ ^ ) \Delta_t=\mathcal{L}(\boldsymbol{\theta}^{(t)})-\mathcal{L}(\hat{\boldsymbol{\theta}}) Δ t = L ( θ ( t ) ) − L ( θ ^ ) . Convexity and (21) give
L ( β ( t ) ) ≤ L ( θ ( t ) ) + ⟨ g t , β ( t ) − θ ( t ) ⟩ , (32) \mathcal{L}(\boldsymbol{\beta}^{(t)})\leq\mathcal{L}(\boldsymbol{\theta}^{(t)})+\langle\boldsymbol{g}_t,\boldsymbol{\beta}^{(t)}-\boldsymbol{\theta}^{(t)}\rangle, \tag*{(32)} L ( β ( t ) ) ≤ L ( θ ( t ) ) + ⟨ g t , β ( t ) − θ ( t ) ⟩ , (32)
L ( β ( t ) ) ≤ L ( θ ^ ) + ⟨ g t , β ( t ) − θ ^ ⟩ − μ 4 ∥ β ( t ) − θ ^ ∥ 2 2 + μ 4 ϵ 2 2 . (33) \mathcal{L}(\boldsymbol{\beta}^{(t)})\leq\mathcal{L}(\hat{\boldsymbol{\theta}})+\langle\boldsymbol{g}_t,\boldsymbol{\beta}^{(t)}-\hat{\boldsymbol{\theta}}\rangle-\frac{\mu}{4}\lVert\boldsymbol{\beta}^{(t)}-\hat{\boldsymbol{\theta}}\rVert_2^2+\frac{\mu}{4}\epsilon_2^2. \tag*{(33)} L ( β ( t ) ) ≤ L ( θ ^ ) + ⟨ g t , β ( t ) − θ ^ ⟩ − 4 μ ∥ β ( t ) − θ ^ ∥ 2 2 + 4 μ ϵ 2 2 . (33)
Set r ( 0 ) = α ( θ ( 0 ) − θ ^ ) \boldsymbol{r}^{(0)}=\alpha(\boldsymbol{\theta}^{(0)}-\hat{\boldsymbol{\theta}}) r ( 0 ) = α ( θ ( 0 ) − θ ^ ) , w ( t ) = β ( t ) − ( 1 − α ) θ ( t ) − α θ ^ \boldsymbol{w}^{(t)}=\boldsymbol{\beta}^{(t)}-(1-\alpha)\boldsymbol{\theta}^{(t)}-\alpha\hat{\boldsymbol{\theta}} w ( t ) = β ( t ) − ( 1 − α ) θ ( t ) − α θ ^ , and r ( t + 1 ) = w ( t ) − η g t = θ ( t + 1 ) − ( 1 − α ) θ ( t ) − α θ ^ \boldsymbol{r}^{(t+1)}=\boldsymbol{w}^{(t)}-\eta\boldsymbol{g}_t=\boldsymbol{\theta}^{(t+1)}-(1-\alpha)\boldsymbol{\theta}^{(t)}-\alpha\hat{\boldsymbol{\theta}} r ( t + 1 ) = w ( t ) − η g t = θ ( t + 1 ) − ( 1 − α ) θ ( t ) − α θ ^ . Weight the two inequalities by 1 − α 1-\alpha 1 − α and α \alpha α . Combining them with (3) and completing the square gives
Δ t + 1 + ∥ r ( t + 1 ) ∥ 2 2 2 η ≤ ( 1 − α ) Δ t + ∥ w ( t ) ∥ 2 2 2 η − α μ 4 ∥ β ( t ) − θ ^ ∥ 2 2 + η L 2 ϵ 1 2 2 + α μ ϵ 2 2 4 . (34) \Delta_{t+1}+\frac{\lVert\boldsymbol{r}^{(t+1)}\rVert_2^2}{2\eta}\leq(1-\alpha)\Delta_t+\frac{\lVert\boldsymbol{w}^{(t)}\rVert_2^2}{2\eta}-\frac{\alpha\mu}{4}\lVert\boldsymbol{\beta}^{(t)}-\hat{\boldsymbol{\theta}}\rVert_2^2+\frac{\eta L^2\epsilon_1^2}{2}+\frac{\alpha\mu\epsilon_2^2}{4}. \tag*{(34)} Δ t + 1 + 2 η ∥ r ( t + 1 ) ∥ 2 2 ≤ ( 1 − α ) Δ t + 2 η ∥ w ( t ) ∥ 2 2 − 4 α μ ∥ β ( t ) − θ ^ ∥ 2 2 + 2 η L 2 ϵ 1 2 + 4 α μ ϵ 2 2 . (34)
For c = ( 1 − γ ) / ( 1 + γ ) c=(1-\gamma)/(1+\gamma) c = ( 1 − γ ) / ( 1 + γ ) , direct expansion gives
1 2 η ∥ w ( t ) ∥ 2 2 − α μ 4 ∥ θ ^ − β ( t ) ∥ 2 2 − 1 − α 2 η ∥ r ( t ) ∥ 2 2 = − 1 2 η ∥ ( c − α ) r ( t ) + γ ( θ ( t ) − θ ( t − 1 ) ) ∥ 2 2 − α γ 2 ( 1 − α ) 2 η ∥ θ ( t ) − θ ( t − 1 ) ∥ 2 2 + α 2 ( μ η − 2 α c ) ∥ β ( t ) − θ ^ ∥ 2 2 ≤ 0. \begin{aligned}
\frac{1}{2\eta}\lVert\boldsymbol{w}^{(t)}\rVert_2^2-\frac{\alpha\mu}{4}\lVert\hat{\boldsymbol{\theta}}-\boldsymbol{\beta}^{(t)}\rVert_2^2-\frac{1-\alpha}{2\eta}\lVert\boldsymbol{r}^{(t)}\rVert_2^2
={}&-\frac{1}{2\eta}\left\lVert(c-\alpha)\boldsymbol{r}^{(t)}+\gamma(\boldsymbol{\theta}^{(t)}-\boldsymbol{\theta}^{(t-1)})\right\rVert_2^2\\
&-\frac{\alpha\gamma^2(1-\alpha)}{2\eta}\lVert\boldsymbol{\theta}^{(t)}-\boldsymbol{\theta}^{(t-1)}\rVert_2^2\\
&+\frac{\alpha}{2}\left(\mu\eta-2\alpha c\right)\lVert\boldsymbol{\beta}^{(t)}-\hat{\boldsymbol{\theta}}\rVert_2^2\leq0.
\end{aligned} 2 η 1 ∥ w ( t ) ∥ 2 2 − 4 α μ ∥ θ ^ − β ( t ) ∥ 2 2 − 2 η 1 − α ∥ r ( t ) ∥ 2 2 = − 2 η 1 ( c − α ) r ( t ) + γ ( θ ( t ) − θ ( t − 1 ) ) 2 2 − 2 η α γ 2 ( 1 − α ) ∥ θ ( t ) − θ ( t − 1 ) ∥ 2 2 + 2 α ( μ η − 2 α c ) ∥ β ( t ) − θ ^ ∥ 2 2 ≤ 0.
The last inequality uses α ≤ c \alpha\le c α ≤ c and 2 α c ≤ η μ 2\alpha c \le\eta\mu 2 α c ≤ η μ . Hence the potential
Φ t = Δ t + ∣ r ( t ) ∣ 2 2 2 η (35) \Phi_t = \Delta_t + \frac{\lvert r^{(t)}\rvert_2^2}{2\eta} \tag*{(35)} Φ t = Δ t + 2 η ∣ r ( t ) ∣ 2 2 (35)
satisfies
Φ t + 1 ≤ ( 1 − α ) Φ t + η L 2 ϵ 1 2 2 + α μ ϵ 2 2 4 . (36) \Phi_{t+1} \le(1-\alpha)\Phi_t + \frac{\eta L^2\epsilon_1^2}{2} + \frac{\alpha\mu\epsilon_2^2}{4}. \tag*{(36)} Φ t + 1 ≤ ( 1 − α ) Φ t + 2 η L 2 ϵ 1 2 + 4 α μ ϵ 2 2 . (36)
Since 0 < α ≤ 1 0 < \alpha\le1 0 < α ≤ 1 and Φ 0 = Δ 0 + α 2 ∣ θ ( 0 ) − θ ^ 2 ∣ 2 2 η \Phi_0 = \Delta_0 + \frac{\alpha^2\lvert\theta^{(0)}-\widehat{\theta}_2\rvert^2}{2\eta} Φ 0 = Δ 0 + 2 η α 2 ∣ θ ( 0 ) − θ 2 ∣ 2 , summing this recursion proves the bound with additive terms. For η ≤ 1 / ( 2 L ) \eta\le1/(2L) η ≤ 1/ ( 2 L ) and under the two distance conditions, Condition 2 removes the ϵ 2 \epsilon_2 ϵ 2 term and (30) removes the ϵ 1 \epsilon_1 ϵ 1 term. The same square completion then gives (36) without either additive term and proves the remaining bound. □
Proof of Theorem 2(iii). For this proof, define θ ( − 1 ) = θ ( 0 ) \theta^{(-1)}=\theta^{(0)} θ ( − 1 ) = θ ( 0 ) . Write g ∗ = ∂ L ( θ ∗ ) \mathbf{g}^*=\partial\mathcal{L}(\theta^*) g ∗ = ∂ L ( θ ∗ ) , M = I − η H M=\mathbb{I}-\eta\mathbf{H} M = I − η H , and z 0 = θ ( 0 ) − θ ∗ + H − 1 g ∗ \mathbf{z}_0=\theta^{(0)}-\theta^*+\mathbf{H}^{-1}\mathbf{g}^* z 0 = θ ( 0 ) − θ ∗ + H − 1 g ∗ . Define P 0 = I P_0=\mathbb{I} P 0 = I , P 1 = M P_1=M P 1 = M , and P t + 1 = ( 1 + γ ) M P t − γ M P t − 1 P_{t+1}=(1+\gamma)MP_t-\gamma MP_{t-1} P t + 1 = ( 1 + γ ) M P t − γ M P t − 1 . Write a = η m H a=\sqrt{\eta m_H} a = η m H and ρ = 1 − a \rho=1-a ρ = 1 − a . Let K − 1 = 0 K_{-1}=0 K − 1 = 0 , K 0 = I K_0=\mathbb{I} K 0 = I , and K j + 1 = ( 1 + γ ) M K j − γ M K j − 1 K_{j+1}=(1+\gamma)MK_j-\gamma MK_{j-1} K j + 1 = ( 1 + γ ) M K j − γ M K j − 1 . Then P t = K t − γ M K t − 1 P_t=K_t-\gamma MK_{t-1} P t = K t − γ M K t − 1 . Diagonalizing the symmetric matrix H \mathbf{H} H reduces both sequences to scalar polynomials. For an eigenvalue m ∈ [ 0 , 1 − a 2 ] m\in[0,1-a^2] m ∈ [ 0 , 1 − a 2 ] of M M M , the characteristic roots x 1 , x 2 x_1,x_2 x 1 , x 2 satisfy ∣ x i ∣ ≤ ρ \lvert x_i\rvert\le\rho ∣ x i ∣ ≤ ρ by Lemma 5(ii). The polynomial K j K_j K j therefore satisfies
K j ( m ) = ∑ k = 0 j x 1 k x 2 j − k , ∣ K j ( m ) ∣ ≤ ( j + 1 ) ρ j , K_j(m)=\sum_{k=0}^{j}x_1^k x_2^{j-k},\qquad\lvert K_j(m)\rvert\le(j+1)\rho^j, K j ( m ) = k = 0 ∑ j x 1 k x 2 j − k , ∣ K j ( m )∣ ≤ ( j + 1 ) ρ j ,
including complex and repeated roots. Consequently ∥ K j ∥ ≤ ( j + 1 ) ρ j \lVert K_j\rVert\le(j+1)\rho^j ∥ K j ∥ ≤ ( j + 1 ) ρ j and η ∑ j ≥ 0 ∥ K j ∥ ≤ η / ( 1 − ρ ) 2 = 1 / m H \eta\sum_{j\ge0}\lVert K_j\rVert\le\eta/(1-\rho)^2=1/m_H η ∑ j ≥ 0 ∥ K j ∥ ≤ η / ( 1 − ρ ) 2 = 1/ m H .
We next bound the homogeneous polynomial without separating the roots. If 0 < a < 1 0<a<1 0 < a < 1 and 0 < m < 1 − a 2 0<m<1-a^2 0 < m < 1 − a 2 , write the roots as r exp ( ± i ϕ ) r\exp(\pm i\phi) r exp ( ± i ϕ ) , where r = γ m ≤ ρ r=\sqrt{\gamma m}\le\rho r = γ m ≤ ρ and cos ϕ = ( 1 + γ ) m / ( 2 r ) \cos\phi=(1+\gamma)m/(2r) cos ϕ = ( 1 + γ ) m / ( 2 r ) . Since ( 1 − γ ) / ( 1 + γ ) = a (1-\gamma)/(1+\gamma)=a ( 1 − γ ) / ( 1 + γ ) = a ,
P t ( m ) = r t { cos ( t ϕ ) + a cos ϕ sin ( t ϕ ) sin ϕ } . P_t(m)=r^t\left\{\cos(t\phi)+a\cos\phi\frac{\sin(t\phi)}{\sin\phi}\right\}. P t ( m ) = r t { cos ( tϕ ) + a cos ϕ sin ϕ sin ( tϕ ) } .
The inequality ∣ sin ( t ϕ ) ∣ ≤ t sin ϕ \lvert\sin(t\phi)\rvert\le t\sin\phi ∣ sin ( tϕ )∣ ≤ t sin ϕ gives ∣ P t ( m ) ∣ ≤ ( 1 + a t ) ρ t \lvert P_t(m)\rvert\le(1+at)\rho^t ∣ P t ( m )∣ ≤ ( 1 + a t ) ρ t . At m = 1 − a 2 m=1-a^2 m = 1 − a 2 , the double root is ρ \rho ρ and P t ( m ) = ( 1 + a t ) ρ t P_t(m)=(1+at)\rho^t P t ( m ) = ( 1 + a t ) ρ t . At m = 0 m=0 m = 0 , P t ( m ) = 0 P_t(m)=0 P t ( m ) = 0 for t ≥ 1 t\ge1 t ≥ 1 . When a = 1 a=1 a = 1 , M = 0 M=0 M = 0 and the same bounds hold directly. Hence
∥ P t ∥ ≤ ( 1 + a t ) ρ t ≤ 1 , ∥ P t ∥ ≤ ( 1 + a t ) e − a t ≤ 2 e − a t / 2 . \lVert P_t\rVert\le(1+at)\rho^t\le1,\qquad\lVert P_t\rVert\le(1+at)e^{-at}\le2e^{-at/2}. ∥ P t ∥ ≤ ( 1 + a t ) ρ t ≤ 1 , ∥ P t ∥ ≤ ( 1 + a t ) e − a t ≤ 2 e − a t /2 .
The bound by 1 follows from 1 + a t ≤ ( 1 − a ) − t 1+at\le(1-a)^{-t} 1 + a t ≤ ( 1 − a ) − t for 0 < a < 1 0<a<1 0 < a < 1 .
Write w t = θ ( t ) − θ ∗ + H − 1 g ∗ \mathbf{w}_t=\theta^{(t)}-\theta^*+\mathbf{H}^{-1}\mathbf{g}^* w t = θ ( t ) − θ ∗ + H − 1 g ∗ . While the extrapolated points lie in the local ball, define e t = ∂ L ( β ( t ) ) − g ∗ − H ( β ( t ) − θ ∗ ) \mathbf{e}_t=\partial\mathcal{L}(\beta^{(t)})-\mathbf{g}^*-\mathbf{H}(\beta^{(t)}-\theta^*) e t = ∂ L ( β ( t ) ) − g ∗ − H ( β ( t ) − θ ∗ ) . Condition 3 gives
w t + 1 = ( 1 + γ ) M w t − γ M w t − 1 − η e t , ∣ e t ∣ 2 ≤ r n , \mathbf{w}_{t+1}=(1+\gamma)M\mathbf{w}_t-\gamma M\mathbf{w}_{t-1}-\eta\mathbf{e}_t,\qquad\lvert\mathbf{e}_t\rvert_2\le r_n, w t + 1 = ( 1 + γ ) M w t − γ M w t − 1 − η e t , ∣ e t ∣ 2 ≤ r n ,
with w − 1 = w 0 = z 0 \mathbf{w}_{-1}=\mathbf{w}_0=\mathbf{z}_0 w − 1 = w 0 = z 0 . Solving this recursion yields
w t = P t z 0 − η ∑ j = 0 t − 1 K t − 1 − j e j , ∣ w t ∣ 2 ≤ ∣ z 0 ∣ 2 + r n / m H . \mathbf{w}_t=P_t\mathbf{z}_0-\eta\sum_{j=0}^{t-1}K_{t-1-j}\mathbf{e}_j,\qquad\lvert\mathbf{w}_t\rvert_2\le\lvert\mathbf{z}_0\rvert_2+r_n/m_H. w t = P t z 0 − η j = 0 ∑ t − 1 K t − 1 − j e j , ∣ w t ∣ 2 ≤ ∣ z 0 ∣ 2 + r n / m H .
Let C = ∣ z 0 ∣ 2 + r n / m H C=\lvert\mathbf{z}_0\rvert_2+r_n/m_H C = ∣ z 0 ∣ 2 + r n / m H . Initially ∣ w − 1 ∣ 2 = ∣ w 0 ∣ 2 ≤ C \lvert\mathbf{w}_{-1}\rvert_2=\lvert\mathbf{w}_0\rvert_2\le C ∣ w − 1 ∣ 2 = ∣ w 0 ∣ 2 ≤ C . If the errors through index t − 1 t-1 t − 1 are bounded by r n r_n r n , the solution formula gives ∣ w t ∣ 2 ≤ C \lvert\mathbf{w}_t\rvert_2\le C ∣ w t ∣ 2 ≤ C . Since β ( t ) − θ ∗ = ( 1 + γ ) w t − γ w t − 1 − H − 1 g ∗ \beta^{(t)}-\theta^*=(1+\gamma)\mathbf{w}_t-\gamma\mathbf{w}_{t-1}-\mathbf{H}^{-1}\mathbf{g}^* β ( t ) − θ ∗ = ( 1 + γ ) w t − γ w t − 1 − H − 1 g ∗ , the entry condition in Theorem 2(iii) gives ∣ β ( t ) − θ ∗ ∣ 2 ≤ ( 1 + 2 γ ) C + ∣ H − 1 g ∗ ∣ 2 ≤ ϵ 3 |\boldsymbol{\beta}^{(t)}-\theta^*|_2 \le(1+2\gamma)C+|H^{-1}g^*|_2 \le\epsilon_3 ∣ β ( t ) − θ ∗ ∣ 2 ≤ ( 1 + 2 γ ) C + ∣ H − 1 g ∗ ∣ 2 ≤ ϵ 3 and validates the bound on e t e_t e t . Induction therefore covers every t ≥ 0 t\ge0 t ≥ 0 . Also ∣ θ ( t ) − θ ∗ ∣ 2 ≤ C + ∣ H − 1 g ∗ ∣ 2 ≤ ϵ 3 |\theta^{(t)}-\theta^*|_2\le C+|H^{-1}g^*|_2\le\epsilon_3 ∣ θ ( t ) − θ ∗ ∣ 2 ≤ C + ∣ H − 1 g ∗ ∣ 2 ≤ ϵ 3 . The solution formula and the bounds for P t P_t P t and K j K_j K j give (7).
Proof of Corollary 1. Write Δ ( θ ) = L ( θ ) − L ( θ ^ ) \Delta(\theta)=\mathcal{L}(\theta)-\mathcal{L}(\hat{\theta}) Δ ( θ ) = L ( θ ) − L ( θ ^ ) , d ( θ ) = ∣ θ − θ ^ ∣ 2 d(\theta)=|\theta-\hat{\theta}|_2 d ( θ ) = ∣ θ − θ ^ ∣ 2 , and R 1 = 2 max { R 0 , R } + 2 ϵ 1 R_1=2\max\{R_0,R\}+2\epsilon_1 R 1 = 2 max { R 0 , R } + 2 ϵ 1 . Lemma 2 gives ∣ ∂ L ( θ ^ ) ∣ 2 ≤ L ϵ 1 |\partial\mathcal{L}(\hat{\theta})|_2\le L\epsilon_1 ∣ ∂ L ( θ ^ ) ∣ 2 ≤ L ϵ 1 , hence
∣ ∂ L ( θ ) ∣ 2 ≤ L { d ( θ ) + 2 ϵ 1 } , θ ∈ B ( θ ^ , R 1 ) . |\partial\mathcal{L}(\theta)|_2\le L\{d(\theta)+2\epsilon_1\},\quad\theta\in\mathbb{B}(\hat{\theta},R_1). ∣ ∂ L ( θ ) ∣ 2 ≤ L { d ( θ ) + 2 ϵ 1 } , θ ∈ B ( θ ^ , R 1 ) .
For θ ∈ B ( θ ^ , R ) \theta\in\mathbb{B}(\hat{\theta},R) θ ∈ B ( θ ^ , R ) , (21) and (22) give
∣ ∂ L ( θ ) ∣ 2 2 ≥ μ { Δ ( θ ) − μ ϵ 2 2 / 4 } , d ( θ ) 2 ≤ 4 μ Δ ( θ ) + ϵ 2 2 . (37) |\partial\mathcal{L}(\theta)|_2^2\ge\mu\{\Delta(\theta)-\mu\epsilon_2^2/4\},\quad d(\theta)^2\le\frac{4}{\mu}\Delta(\theta)+\epsilon_2^2. \tag*{(37)} ∣ ∂ L ( θ ) ∣ 2 2 ≥ μ { Δ ( θ ) − μ ϵ 2 2 /4 } , d ( θ ) 2 ≤ μ 4 Δ ( θ ) + ϵ 2 2 . (37)
The first inequality follows by completing the square in ∣ ∂ L ( θ ) ∣ 2 d − μ d 2 / 4 |\partial\mathcal{L}(\theta)|_2d-\mu d^2/4 ∣ ∂ L ( θ ) ∣ 2 d − μ d 2 /4 .
Set b = μ R 2 / 16 b=\mu R^2/16 b = μ R 2 /16 . Since R ≥ 4 ϵ 2 R\ge4\epsilon_2 R ≥ 4 ϵ 2 , (37) gives Δ ≥ b \Delta\ge b Δ ≥ b on the sphere d = R d=R d = R . Convexity along rays from θ ^ \hat{\theta} θ ^ then gives Δ ( θ ) ≥ μ R d ( θ ) / 16 \Delta(\theta)\ge\mu Rd(\theta)/16 Δ ( θ ) ≥ μ R d ( θ ) /16 for d ( θ ) > R d(\theta)>R d ( θ ) > R . Thus { Δ < b } ⊆ B ( θ ^ , R ) \{\Delta<b\}\subseteq\mathbb{B}(\hat{\theta},R) { Δ < b } ⊆ B ( θ ^ , R ) . Every point in the smoothness region with Δ ≥ b \Delta\ge b Δ ≥ b has a subgradient step of length at least ϵ 1 \epsilon_1 ϵ 1 . Inside the radius-R R R ball, (37) gives ∣ ∂ L ∣ 2 ≥ μ R / ( 4 2 ) |\partial\mathcal{L}|_2\ge\mu R/(4\sqrt{2}) ∣ ∂ L ∣ 2 ≥ μ R / ( 4 2 ) , and outside it convexity gives ∣ ∂ L ∣ 2 ≥ Δ / d ≥ μ R / 16 |\partial\mathcal{L}|_2\ge\Delta/d\ge\mu R/16 ∣ ∂ L ∣ 2 ≥ Δ/ d ≥ μ R /16 . The condition R ≥ 16 ϵ 1 / ( η μ ) R\ge16\epsilon_1/(\eta\mu) R ≥ 16 ϵ 1 / ( η μ ) suffices in both cases.
Part (i). Let τ = inf { t : Δ ( θ ( t ) ) < b } \tau=\inf\{t:\Delta(\theta^{(t)})<b\} τ = inf { t : Δ ( θ ( t ) ) < b } and q = 1 − 3 η μ / 4 q=1-3\eta\mu/4 q = 1 − 3 η μ /4 . Before τ \tau τ , suppose d ( θ ( t ) ) ≤ R 0 d(\theta^{(t)})\le R_0 d ( θ ( t ) ) ≤ R 0 . The proposed next iterate has distance at most 3 R 0 / 2 + ϵ 1 ≤ R 1 3R_0/2+\epsilon_1\le R_1 3 R 0 /2 + ϵ 1 ≤ R 1 , and the step is long enough for the error-free bound in Theorem 1(i). Its distance to θ ^ \hat{\theta} θ ^ therefore cannot increase. Induction gives Δ ( θ ( t ) ) ≤ R 0 2 / ( 2 η t ) \Delta(\theta^{(t)})\le R_0^2/(2\eta t) Δ ( θ ( t ) ) ≤ R 0 2 / ( 2 η t ) up to τ \tau τ . Hence T 1 > 8 R 0 2 / ( η μ R 2 ) T_1>8R_0^2/(\eta\mu R^2) T 1 > 8 R 0 2 / ( η μ R 2 ) implies τ ≤ T 1 \tau\le T_1 τ ≤ T 1 .
For a current point θ \theta θ with d ( θ ) ≤ R d(\theta)\le R d ( θ ) ≤ R , write g = ∂ L ( θ ) g=\partial\mathcal{L}(\theta) g = ∂ L ( θ ) . The auxiliary point θ − g / L \theta-g/L θ − g / L lies in B ( θ ^ , R 1 ) \mathbb{B}(\hat{\theta},R_1) B ( θ ^ , R 1 ) since its distance is at most 2 R + 2 ϵ 1 2R+2\epsilon_1 2 R + 2 ϵ 1 . Minimality and (20) applied to this step yield ∣ g ∣ 2 2 ≤ 2 L Δ ( θ ) + L 2 ϵ 1 2 |g|_2^2\le2L\Delta(\theta)+L^2\epsilon_1^2 ∣ g ∣ 2 2 ≤ 2 L Δ ( θ ) + L 2 ϵ 1 2 . Applying (21) with comparison point θ ^ \hat{\theta} θ ^ and then (22) gives
d ( θ − η g ) 2 ≤ ( 1 − η μ 2 ) d ( θ ) 2 − η Δ ( θ ) + η μ 2 ϵ 2 2 + η 2 L 2 ϵ 1 2 ≤ q d ( θ ) 2 + ( 1 − q ) ( ϵ 2 2 + 4 η L 2 3 μ ϵ 1 2 ) . \begin{aligned}
d(\theta-\eta g)^2\le\left(1-\frac{\eta\mu}{2}\right)d(\theta)^2-\eta\Delta(\theta)+\frac{\eta\mu}{2}\epsilon_2^2+\eta^2L^2\epsilon_1^2 \\
&\le qd(\theta)^2+(1-q)\left(\epsilon_2^2+\frac{4\eta L^2}{3\mu}\epsilon_1^2\right).
\end{aligned} d ( θ − η g ) 2 ≤ ( 1 − 2 η μ ) d ( θ ) 2 − η Δ ( θ ) + 2 η μ ϵ 2 2 + η 2 L 2 ϵ 1 2 ≤ q d ( θ ) 2 + ( 1 − q ) ( ϵ 2 2 + 3 μ 4 η L 2 ϵ 1 2 ) .
The first inequality uses 2 η ( 1 − η L ) ≥ η 2\eta(1-\eta L)\ge\eta 2 η ( 1 − η L ) ≥ η . The radius assumptions imply R 2 ≥ ϵ 2 2 + 4 η L 2 ϵ 1 2 / ( 3 μ ) R^2\ge\epsilon_2^2+4\eta L^2\epsilon_1^2/(3\mu) R 2 ≥ ϵ 2 2 + 4 η L 2 ϵ 1 2 / ( 3 μ ) , making this ball invariant. At entry, (37) gives d ( θ ( τ ) ) 2 ≤ R 2 / 4 + ϵ 2 2 ≤ R 2 / 2 d(\theta^{(\tau)})^2\le R^2/4+\epsilon_2^2\le R^2/2 d ( θ ( τ ) ) 2 ≤ R 2 /4 + ϵ 2 2 ≤ R 2 /2 . Iterating the distance recursion therefore gives
d ( θ ( t ) ) 2 ≤ q t − τ R 2 2 + ϵ 2 2 + 4 η L 2 3 μ ϵ 1 2 , t ≥ τ . d(\theta^{(t)})^2\le q^{t-\tau}\frac{R^2}{2}+\epsilon_2^2+\frac{4\eta L^2}{3\mu}\epsilon_1^2,\quad t\ge\tau. d ( θ ( t ) ) 2 ≤ q t − τ 2 R 2 + ϵ 2 2 + 3 μ 4 η L 2 ϵ 1 2 , t ≥ τ .
Taking square roots and using τ ≤ T 1 \tau\le T_1 τ ≤ T 1 proves (8) and the distance bound.
Part (ii). During the initial Nesterov phase, every executed update has length at least ϵ 1 \epsilon_1 ϵ 1 . The potential and convex-combination identities in (31) keep the completed iterates and the next extrapolated point within distance R 0 R_0 R 0 of θ ^ \hat{\theta} θ ^ . The proposed next iterate lies within distance 3 R 0 / 2 + ϵ 1 ≤ R 1 3R_0/2+\epsilon_1\le R_1 3 R 0 /2 + ϵ 1 ≤ R 1 . Induction therefore validates each application of the error-free potential bound until the short-step test stops this phase or all N 1 N_1 N 1 updates are completed.
Let θ ˉ \bar{\theta} θ ˉ be the point at which the constant-momentum phase starts. If the short-step test is met, ∣ ∂ L ( θ ˉ ) ∣ 2 < ϵ 1 / η ≤ μ R / 16 |\partial\mathcal{L}(\bar{\theta})|_2<\epsilon_1/\eta\le\mu R/16 ∣ ∂ L ( θ ˉ ) ∣ 2 < ϵ 1 / η ≤ μ R /16 . The lower bound outside the radius-R R R ball proved above places θ ˉ \bar{\theta} θ ˉ inside that ball. Completing the square in (21) and then applying (22) give
Δ ( θ ˉ ) ≤ ∣ ∂ L ( θ ˉ ) ∣ 2 2 μ + μ 4 ϵ 2 2 ≤ 65 μ R 2 16384 , d ( θ ˉ ) 2 ≤ 264 R 2 16384 . \Delta(\bar{\theta})\le\frac{|\partial\mathcal{L}(\bar{\theta})|_2^2}{\mu}+\frac{\mu}{4}\epsilon_2^2\le\frac{65\mu R^2}{16384},\qquad d(\bar{\theta})^2\le\frac{264R^2}{16384}. Δ ( θ ˉ ) ≤ μ ∣ ∂ L ( θ ˉ ) ∣ 2 2 + 4 μ ϵ 2 2 ≤ 16384 65 μ R 2 , d ( θ ˉ ) 2 ≤ 16384 264 R 2 .
If all N 1 N_1 N 1 updates are completed, Theorem 2(i) gives instead
Δ ( θ ˉ ) ≤ 2 R 0 2 η N 1 2 ≤ μ R 2 256 , d ( θ ˉ ) 2 ≤ 65 R 2 4096 . \Delta(\bar{\theta}) \le\frac{2R_0^2}{\eta N_1^2} \le\frac{\mu R^2}{256}, \qquad d(\bar{\theta})^2 \le\frac{65R^2}{4096}. Δ ( θ ˉ ) ≤ η N 1 2 2 R 0 2 ≤ 256 μ R 2 , d ( θ ˉ ) 2 ≤ 4096 65 R 2 .
Here the objective bound first places θ ˉ \bar{\theta} θ ˉ inside the radius-R R R ball, and (22) gives the distance bound. The additional radius assumption gives ϵ 2 ≤ R / 64 \epsilon_2 \le R/64 ϵ 2 ≤ R /64 in both cases.
Reset the time index at the restart with θ ( 0 ) = θ ˉ \theta^{(0)}=\bar{\theta} θ ( 0 ) = θ ˉ , and define θ ( − 1 ) = θ ( 0 ) \theta^{(-1)}=\theta^{(0)} θ ( − 1 ) = θ ( 0 ) for this proof. Let α = η μ / 2 \alpha=\sqrt{\eta\mu/2} α = η μ /2 and use the potential from the proof of Theorem 2(ii),
Φ t = Δ ( θ ( t ) ) + 1 2 η ∣ θ ( t ) − ( 1 − α ) θ ( t − 1 ) − α θ ˉ ∣ 2 2 . \Phi_t=\Delta(\theta^{(t)})+\frac{1}{2\eta}\left\lvert\theta^{(t)}-(1-\alpha)\theta^{(t-1)}-\alpha\bar{\theta}\right\rvert_2^2. Φ t = Δ ( θ ( t ) ) + 2 η 1 θ ( t ) − ( 1 − α ) θ ( t − 1 ) − α θ ˉ 2 2 .
Since α 2 / ( 2 η ) = μ / 4 \alpha^2/(2\eta)=\mu/4 α 2 / ( 2 η ) = μ /4 , its initial value and forcing bound satisfy
Φ 0 ≤ 131 μ R 2 16384 < μ R 2 120 , E N = η L 2 ϵ 1 2 2 α + μ ϵ 2 2 4 = μ α κ 2 ϵ 1 2 + μ 4 ϵ 2 2 ≤ μ R 2 4096 . \Phi_0\le\frac{131\mu R^2}{16384}<\frac{\mu R^2}{120},\qquad E_N=\frac{\eta L^2\epsilon_1^2}{2\alpha}+\frac{\mu\epsilon_2^2}{4}=\mu\alpha\kappa^2\epsilon_1^2+\frac{\mu}{4}\epsilon_2^2\le\frac{\mu R^2}{4096}. Φ 0 ≤ 16384 131 μ R 2 < 120 μ R 2 , E N = 2 α η L 2 ϵ 1 2 + 4 μ ϵ 2 2 = μα κ 2 ϵ 1 2 + 4 μ ϵ 2 2 ≤ 4096 μ R 2 .
The last inequality uses α ≤ 1 \alpha\le1 α ≤ 1 and $\epsilon_2+\kappa\epsilon_1\le R/64.
We prove Φ t ≤ ( 1 − α ) t Φ 0 + E N < μ R 2 / 100 \Phi_t\le(1-\alpha)^t\Phi_0+E_N<\mu R^2/100 Φ t ≤ ( 1 − α ) t Φ 0 + E N < μ R 2 /100 while verifying the points used by the potential recursion. If the bound holds through time t t t , the sublevel-set inclusion places θ ( t ) \theta^{(t)} θ ( t ) in the radius-R R R ball. Inequality (22) then gives d ( θ ( t ) ) 2 < R 2 / 16 d(\theta^{(t)})^2<R^2/16 d ( θ ( t ) ) 2 < R 2 /16 . The same holds at t − 1 t-1 t − 1 , including the initial index − 1 -1 − 1 by the restart. Thus ∣ β ( t ) − θ ˉ ∣ 2 ≤ 3 R / 4 < R \lvert\beta^{(t)}-\bar{\theta}\rvert_2\le3R/4<R ∣ β ( t ) − θ ˉ ∣ 2 ≤ 3 R /4 < R , and the proposed θ ( t + 1 ) \theta^{(t+1)} θ ( t + 1 ) lies within distance 3 R / 2 + ϵ 1 ≤ R 1 3R/2+\epsilon_1\le R_1 3 R /2 + ϵ 1 ≤ R 1 . The one-step potential bound is therefore valid and gives
Φ t + 1 ≤ ( 1 − α ) Φ t + η L 2 ϵ 1 2 2 + α μ ϵ 2 2 4 = ( 1 − α ) Φ t + α E N . \Phi_{t+1}\le(1-\alpha)\Phi_t+\frac{\eta L^2\epsilon_1^2}{2}+\frac{\alpha\mu\epsilon_2^2}{4}=(1-\alpha)\Phi_t+\alpha E_N. Φ t + 1 ≤ ( 1 − α ) Φ t + 2 η L 2 ϵ 1 2 + 4 α μ ϵ 2 2 = ( 1 − α ) Φ t + α E N .
This closes the induction. Finally, (22) gives
d ( θ ( t ) ) 2 ≤ 1 30 ( 1 − α ) t R 2 + 2 ϵ 2 2 + 4 α κ 2 ϵ 1 2 . d(\theta^{(t)})^2\le\frac{1}{30}(1-\alpha)^tR^2+2\epsilon_2^2+4\alpha\kappa^2\epsilon_1^2. d ( θ ( t ) ) 2 ≤ 30 1 ( 1 − α ) t R 2 + 2 ϵ 2 2 + 4 α κ 2 ϵ 1 2 .
Taking square roots and using 1 / 30 < 1 / 3 1/\sqrt{30}<1/3 1/ 30 < 1/3 proves (9) .
Proof of the radius lower bound following Corollary 1. For 0 < μ ≤ L / 2 0<\mu\le L/2 0 < μ ≤ L /2 and ϵ 1 > 0 \epsilon_1>0 ϵ 1 > 0 , consider L ( θ ) = a ∣ θ ∣ 1 + ( μ / 2 ) ∣ θ ∣ 2 2 \mathcal{L}(\theta)=a\lvert\theta\rvert_1+(\mu/2)\lvert\theta\rvert_2^2 L ( θ ) = a ∣ θ ∣ 1 + ( μ /2 ) ∣ θ ∣ 2 2 , where a = ( L − μ ) ϵ 1 / ( 2 p ) a=(L-\mu)\epsilon_1/(2\sqrt{p}) a = ( L − μ ) ϵ 1 / ( 2 p ) . Its minimizer is θ ˉ = 0 \bar{\theta}=0 θ ˉ = 0 . Use step size η = 1 / ( 2 L ) \eta=1/(2L) η = 1/ ( 2 L ) . For every subgradient selection, ∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) ∣ 2 ≤ μ ∣ θ 1 − θ 2 ∣ 2 + 2 a p = μ ∣ θ 1 − θ 2 ∣ 2 + ( L − μ ) ϵ 1 \lvert\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)\rvert_2\le\mu\lvert\theta_1-\theta_2\rvert_2+2a\sqrt{p}=\mu\lvert\theta_1-\theta_2\rvert_2+(L-\mu)\epsilon_1 ∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) ∣ 2 ≤ μ ∣ θ 1 − θ 2 ∣ 2 + 2 a p = μ ∣ θ 1 − θ 2 ∣ 2 + ( L − μ ) ϵ 1 , which verifies Condition 1. The quadratic term verifies Condition 2 with ϵ 2 = 0 \epsilon_2=0 ϵ 2 = 0 . Away from zero, each coordinate evolves by x ↦ ( 1 − η μ ) x − η a sign ( x ) x\mapsto(1-\eta\mu)x-\eta a\operatorname{sign}(x) x ↦ ( 1 − η μ ) x − η a sign ( x ) . Both affine branches have nonzero slope. Thus at most 2 t 2^t 2 t scalar initial values first reach zero at time t t t , and their union over t ≥ 0 t\ge0 t ≥ 0 is countable. The initial vectors having a coordinate in this set form a Lebesgue-null set common to all selections. Outside it, every iterate has nonzero coordinates, hence a unique subgradient, and all selections generate the same trajectory. The coordinate magnitudes evolve by u ↦ ∣ ( 1 − η μ ) u − η a ∣ u\mapsto\lvert(1-\eta\mu)u-\eta a\rvert u ↦ ∣( 1 − η μ ) u − η a ∣ , a contraction with fixed point η a / ( 2 − η μ ) \eta a/(2-\eta\mu) η a / ( 2 − η μ ) . Each magnitude therefore converges to this value, and ∣ θ ( t ) − θ ˉ ∣ 2 \lvert\theta^{(t)}-\bar{\theta}\rvert_2 ∣ θ ( t ) − θ ˉ ∣ 2 converges to p η a / ( 2 − η μ ) = ( L − μ ) ϵ 1 / { 2 ( 4 L − μ ) } \sqrt{p}\eta a/(2-\eta\mu)=(L-\mu)\epsilon_1/\{2(4L-\mu)\} p η a / ( 2 − η μ ) = ( L − μ ) ϵ 1 / { 2 ( 4 L − μ )} . Since 0 < μ ≤ L / 2 0<\mu\le L/2 0 < μ ≤ L /2 , this limit is at least ϵ 1 / 14 \epsilon_1/14 ϵ 1 /14 .
Complete procedures
Under the model, population, and step-size assumptions of Theorem 4, suppose ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ D \lvert\theta^{(0)}-\theta^*\rvert_2\le D ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ D and B ( θ ∗ , 2 D + R ) ⊆ Θ \mathbb{B}(\theta^*,2D+R)\subseteq\Theta B ( θ ∗ , 2 D + R ) ⊆ Θ for D > 0 D>0 D > 0 and sufficiently small R > 0 R>0 R > 0 . Run either unprojected method in Corollary 1 with radius R / 4 R/4 R /4 and initialization bound R 0 = D + R / 8 R_0=D+R/8 R 0 = D + R /8 , counting all phases. All three conclusions of Theorem 4 hold for these iterates, uniformly over the stated initial decisions and all scenario-subgradient selections.
Proof of the unprojected extension. The localization argument in the proof of Theorem 4 gives δ n = ∣ θ ^ − θ ∗ ∣ 2 ≤ R / 8 \delta_n = |\hat{\theta} - \theta^*|_2 \le R/8 δ n = ∣ θ ^ − θ ∗ ∣ 2 ≤ R /8 on the common increment event for sufficiently large n n n . For sufficiently small R R R , the same argument verifies Conditions 1 and 2 on Θ \Theta Θ and B ( θ ∗ , R ) \mathbb{B}(\theta^*, R) B ( θ ∗ , R ) , respectively. With R a b s = R / 4 R_{\mathrm{abs}} = R/4 R abs = R /4 and R 0 = D + R / 8 R_0 = D + R/8 R 0 = D + R /8 , the two regions needed by Corollary 1 satisfy
B ( θ ^ , R a b s ) ⊆ B ( θ ∗ , R ) , B ( θ ^ , 2 max { R 0 , R a b s } + 2 ϵ 1 ) ⊆ B ( θ ∗ , 2 D + R ) \mathbb{B}(\hat{\theta}, R_{\mathrm{abs}}) \subseteq\mathbb{B}(\theta^*, R), \qquad\mathbb{B}(\hat{\theta}, 2\max\{R_0, R_{\mathrm{abs}}\} + 2\epsilon_1) \subseteq\mathbb{B}(\theta^*, 2D + R) B ( θ ^ , R abs ) ⊆ B ( θ ∗ , R ) , B ( θ ^ , 2 max { R 0 , R abs } + 2 ϵ 1 ) ⊆ B ( θ ∗ , 2 D + R )
for sufficiently large n n n . For the second inclusion, use δ n + 2 max { D + R / 8 , R / 4 } + 2 ϵ 1 ≤ 2 D + R \delta_n + 2\max\{D + R/8, R/4\} + 2\epsilon_1 \le2D + R δ n + 2 max { D + R /8 , R /4 } + 2 ϵ 1 ≤ 2 D + R . Also R a b s ≥ max { 64 ( ϵ 2 + κ ϵ 1 ) , 16 ϵ 1 / ( η μ ) } R_{\mathrm{abs}} \ge\max\{64(\epsilon_2 + \kappa\epsilon_1), 16\epsilon_1/(\eta\mu)\} R abs ≥ max { 64 ( ϵ 2 + κ ϵ 1 ) , 16 ϵ 1 / ( η μ )} eventually, where κ = L / μ \kappa= L/\mu κ = L / μ . The initialization bound is valid and Phase 1 takes a bounded number of updates. Since a n ≥ n − 1 a_n \ge n^{-1} a n ≥ n − 1 eventually, Corollary 1 gives ∣ θ ( t ) − θ ^ ∣ 2 ≤ C a n |\theta^{(t)} - \hat{\theta}|_2 \le Ca_n ∣ θ ( t ) − θ ^ ∣ 2 ≤ C a n for every t ≥ ⌈ c log n ⌉ t \ge\lceil c\log n\rceil t ≥ ⌈ c log n ⌉ . Combining this bound with the SAA expansion and population-gap calculation in the proof of Theorem 4 proves the extension.
The following corollaries combine the phase bounds with subgradient stopping rules. They provide explicit iteration counts and a first-order expansion of the computed solution.
For subgradient descent with D , R > 0 D, R > 0 D , R > 0 and eigenvalue bounds 0 < m H ≤ M H 0 < m_H \le M_H 0 < m H ≤ M H , define
η = 1 2 L , η 3 = M H − 1 , τ n = μ ϵ 3 8 , ϵ 0 = max { ϵ 1 , 2 ϵ 2 } , q = 1 − μ 4 L . \eta= \frac{1}{2L}, \qquad\eta_3 = M_H^{-1}, \qquad\tau_n = \frac{\mu\epsilon_3}{8}, \qquad\epsilon_0 = \max\{\epsilon_1,\sqrt{2}\epsilon_2\}, \qquad q = 1 - \frac{\mu}{4L}. η = 2 L 1 , η 3 = M H − 1 , τ n = 8 μ ϵ 3 , ϵ 0 = max { ϵ 1 , 2 ϵ 2 } , q = 1 − 4 L μ .
Write log + ( x ) = max { 0 , log x } \log_+(x) = \max\{0,\log x\} log + ( x ) = max { 0 , log x } . For r n > 0 r_n > 0 r n > 0 , define the iteration counts
T 1 = ⌈ 8 ( D + R / 4 ) 2 η μ R 2 ⌉ , T 2 = ⌈ log + ( 2 L 2 R 2 / τ n 2 ) − log q ⌉ , K = ⌈ 1 η 3 m H log ( m H ϵ 3 r n ) ⌉ . T_1 = \left\lceil\frac{8(D + R/4)^2}{\eta\mu R^2}\right\rceil,\qquad T_2 = \left\lceil\frac{\log_+(2L^2R^2/\tau_n^2)}{-\log q}\right\rceil,\qquad K = \left\lceil\frac{1}{\eta_3m_H}\log\left(\frac{m_H\epsilon_3}{r_n}\right)\right\rceil. T 1 = ⌈ η μ R 2 8 ( D + R /4 ) 2 ⌉ , T 2 = ⌈ − log q log + ( 2 L 2 R 2 / τ n 2 ) ⌉ , K = ⌈ η 3 m H 1 log ( r n m H ϵ 3 ) ⌉ .
Phase 1 uses step size η \eta η for at most T 1 T_1 T 1 updates. If ∣ ∂ L ( θ ) ∣ 2 ≤ τ n |\partial\mathcal{L}(\theta)|_2 \le\tau_n ∣ ∂ L ( θ ) ∣ 2 ≤ τ n at any check, the algorithm proceeds directly to Phase 3. Otherwise Phase 2 continues with the same step size until this inequality holds. Phase 3 uses step size η 3 \eta_3 η 3 for K K K updates.
Corollary 8. Suppose an event E n \mathcal{E}_n E n has probability at least 1 − δ n 1-\delta_n 1 − δ n . Let K ⊆ Θ \mathcal{K} \subseteq\Theta K ⊆ Θ be convex and contain the ball of radius 2 D + R 2D + R 2 D + R about θ ∗ \theta^* θ ∗ . On E n \mathcal{E}_n E n , assume that L \mathcal{L} L is finite and convex on R p \mathbb{R}^p R p , Condition 1 holds for pairs in K \mathcal{K} K , and Condition 2 holds for pairs in the radius-R R R ball about θ ∗ \theta^* θ ∗ . Suppose the local subgradient approximation in Condition 3 holds with remainder bound r n r_n r n , and
m H I ⪯ H ⪯ M H I , 0 < μ ≤ 2 L , 0 < ϵ 3 ≤ R / 2 , ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ D , m_H\mathbf{I} \preceq H \preceq M_H\mathbf{I}, \qquad0 < \mu\le2L, \qquad0 < \epsilon_3 \le R/2, \qquad|\theta^{(0)} - \theta^*|_2 \le D, m H I ⪯ H ⪯ M H I , 0 < μ ≤ 2 L , 0 < ϵ 3 ≤ R /2 , ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ D , ϵ 0 ≤ τ n 4 L , ∣ ∂ L ( θ ∗ ) ∣ 2 ≤ min { μ / 16 , m H / 4 } ϵ 3 , r n ≤ m H ϵ 3 / 4. \epsilon_0 \le\frac{\tau_n}{4L}, \qquad|\partial\mathcal{L}(\theta^*)|_2 \le\min\{\mu/16,m_H/4\}\epsilon_3, \qquad r_n \le m_H\epsilon_3/4. ϵ 0 ≤ 4 L τ n , ∣ ∂ L ( θ ∗ ) ∣ 2 ≤ min { μ /16 , m H /4 } ϵ 3 , r n ≤ m H ϵ 3 /4. On E n \mathcal{E}_n E n , every iterate remains in K \mathcal{K} K , Phase 2 stops within T 2 T_2 T 2 updates, and Phase 3 starts within distance ϵ 3 / 2 \epsilon_3/2 ϵ 3 /2 of θ ∗ \theta^* θ ∗ . After at most T 1 + T 2 + K T_1 + T_2 + K T 1 + T 2 + K updates, the computed solution and the SAA solution satisfy
θ ~ − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + R n , θ ^ − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + R ^ n , (38) \widetilde{\theta} - \theta^* = -H^{-1}\partial\mathcal{L}(\theta^*) + R_n,\qquad\widehat{\theta} - \theta^* = -H^{-1}\partial\mathcal{L}(\theta^*) + \widehat{R}_n, \tag*{(38)} θ − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + R n , θ − θ ∗ = − H − 1 ∂ L ( θ ∗ ) + R n , (38) where ∣ R n ∣ 2 ≤ 2 r n / m H |R_n|_2 \le2r_n/m_H ∣ R n ∣ 2 ≤ 2 r n / m H and ∣ R ^ n ∣ 2 ≤ r n / m H |\widehat{R}_n|_2 \le r_n/m_H ∣ R n ∣ 2 ≤ r n / m H . Consequently, ∣ θ ~ − θ ^ ∣ 2 ≤ 3 r n / m H |\widetilde{\theta} - \widehat{\theta}|_2 \le3r_n/m_H ∣ θ − θ ∣ 2 ≤ 3 r n / m H .
Proof of Corollary 8. Step 1. Localization of the SAA solution. Work on E n \mathcal{E}_n E n and write g ( θ ) = ∂ L ( θ ) g(\theta) = \partial\mathcal{L}(\theta) g ( θ ) = ∂ L ( θ ) and s n = ∣ g ( θ ∗ ) ∣ 2 s_n = |g(\theta^*)|_2 s n = ∣ g ( θ ∗ ) ∣ 2 . We first locate the SAA solution. For each unit vector u u u , Condition 2 gives
L ( θ ∗ + R u ) − L ( θ ∗ ) ≥ − s n R + μ R 2 4 ≥ μ R 2 8 . \mathcal{L}(\theta^* + Ru) - \mathcal{L}(\theta^*) \ge-s_nR + \frac{\mu R^2}{4} \ge\frac{\mu R^2}{8}. L ( θ ∗ + R u ) − L ( θ ∗ ) ≥ − s n R + 4 μ R 2 ≥ 8 μ R 2 .
@page 31/87 starts_mid=0 ends_mid=1 For t > R t > R t > R , convexity along the same direction implies
L ( θ ∗ + t u ) − L ( θ ∗ ) ≥ t R { L ( θ ∗ + R u ) − L ( θ ∗ ) } ≥ μ R t 8 . \mathcal{L}(\theta^* + tu) - \mathcal{L}(\theta^*) \ge\frac{t}{R}\{\mathcal{L}(\theta^* + Ru) - \mathcal{L}(\theta^*)\} \ge\frac{\mu Rt}{8}. L ( θ ∗ + t u ) − L ( θ ∗ ) ≥ R t { L ( θ ∗ + R u ) − L ( θ ∗ )} ≥ 8 μ R t .
The SAA objective is therefore coercive, and every SAA solution lies inside the ball of radius R R R . Since L ( θ ^ ) ≤ L ( θ ∗ ) \mathcal{L}(\hat{\theta}) \le\mathcal{L}(\theta^*) L ( θ ^ ) ≤ L ( θ ∗ ) , Condition 2 gives
∣ θ ^ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 4 s n / μ } ≤ R / 4. \lvert\hat{\theta} - \theta^* \rvert_2 \le\max\{\epsilon_2,4s_n/\mu\} \le R/4. ∣ θ ^ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 4 s n / μ } ≤ R /4.
The assumptions also bound this distance by ϵ 3 / 4 \epsilon_3/4 ϵ 3 /4 . Hence every global minimizer belongs to Θ \Theta Θ , and the global minimizers are exactly the SAA solutions in (1).
Since θ ^ \hat{\theta} θ ^ lies in the interior of B ( θ ∗ , 2 D + R ) \mathbb{B}(\theta^*,2D+R) B ( θ ∗ , 2 D + R ) , Lemma 2 gives ∣ g ( θ ^ ) ∣ 2 ≤ L ϵ 1 \lvert g(\hat{\theta})\rvert_2 \le L\epsilon_1 ∣ g ( θ ^ ) ∣ 2 ≤ L ϵ 1 .
Step 2. Feasible iterates and entry into Phase 2. Initially ∣ θ ( 0 ) − θ ^ ∣ 2 ≤ D + R / 4 \lvert\theta^{(0)}-\hat{\theta}\rvert_2 \le D+R/4 ∣ θ ( 0 ) − θ ^ ∣ 2 ≤ D + R /4 . Suppose this bound holds at a current iterate with ∣ g ( θ ( t ) ) ∣ 2 > τ n \lvert g(\theta^{(t)})\rvert_2 > \tau_n ∣ g ( θ ( t ) ) ∣ 2 > τ n . Condition 1 and ∣ g ( θ ^ ) ∣ 2 ≤ L ϵ 1 \lvert g(\hat{\theta})\rvert_2 \le L\epsilon_1 ∣ g ( θ ^ ) ∣ 2 ≤ L ϵ 1 give ∣ g ( θ ( t ) ) ∣ 2 ≤ L { ∣ θ ( t ) − θ ^ ∣ 2 + 2 ϵ 1 } \lvert g(\theta^{(t)})\rvert_2 \le L\{\lvert\theta^{(t)}-\hat{\theta}\rvert_2+2\epsilon_1\} ∣ g ( θ ( t ) ) ∣ 2 ≤ L {∣ θ ( t ) − θ ^ ∣ 2 + 2 ϵ 1 } . The proposed update therefore satisfies
∣ θ ( t + 1 ) − θ ∗ ∣ 2 ≤ 3 2 ( D + R / 4 ) + ϵ 1 + R / 4 ≤ 2 D + R , \lvert\theta^{(t+1)}-\theta^*\rvert_2 \le\frac{3}{2}(D+R/4)+\epsilon_1+R/4 \le2D+R, ∣ θ ( t + 1 ) − θ ∗ ∣ 2 ≤ 2 3 ( D + R /4 ) + ϵ 1 + R /4 ≤ 2 D + R ,
since the assumptions imply ϵ 1 ≤ R / 32 \epsilon_1 \le R/32 ϵ 1 ≤ R /32 . The update lies in K \mathcal{K} K and has length greater than η τ n ≥ ϵ 1 \eta\tau_n \ge\epsilon_1 η τ n ≥ ϵ 1 . Equations (25) and (26), applied to this update with comparator θ ^ \hat{\theta} θ ^ , give nonincreasing distance and objective values, with no additive error. Induction validates every update before Phase 3. If all T 1 T_1 T 1 updates are used, the sum of (25) gives
L ( θ ( T 1 ) ) − L ( θ ^ ) ≤ ( D + R / 4 ) 2 2 η T 1 ≤ μ R 2 16 . \mathcal{L}(\theta^{(T_1)})-\mathcal{L}(\hat{\theta}) \le\frac{(D+R/4)^2}{2\eta T_1} \le\frac{\mu R^2}{16}. L ( θ ( T 1 ) ) − L ( θ ^ ) ≤ 2 η T 1 ( D + R /4 ) 2 ≤ 16 μ R 2 .
The lower bound in Step 1 places this iterate inside the radius-R R R ball, and objective descent keeps Phase 2 there.
Step 3. The Phase 2 stopping time. At a Phase 2 iterate with subgradient norm greater than τ n \tau_n τ n , write d = ∣ θ − θ ^ ∣ 2 d=\lvert\theta-\hat{\theta}\rvert_2 d = ∣ θ − θ ^ ∣ 2 and Δ = L ( θ ) − L ( θ ^ ) \Delta=\mathcal{L}(\theta)-\mathcal{L}(\hat{\theta}) Δ = L ( θ ) − L ( θ ^ ) . If d < ϵ 0 d<\epsilon_0 d < ϵ 0 , approximate smoothness would give ∣ g ( θ ) ∣ 2 ≤ 2 L ϵ 0 ≤ τ n / 2 \lvert g(\theta)\rvert_2 \le2L\epsilon_0 \le\tau_n/2 ∣ g ( θ ) ∣ 2 ≤ 2 L ϵ 0 ≤ τ n /2 . Hence d ≥ ϵ 0 d\ge\epsilon_0 d ≥ ϵ 0 . Applying Lemma 3 on B ( θ ∗ , R ) \mathbb{B}(\theta^*,R) B ( θ ∗ , R ) gives
μ Δ ≤ ∣ g ( θ ) ∣ 2 2 ≤ 32 L 2 μ Δ . \mu\Delta\le\lvert g(\theta)\rvert_2^2 \le\frac{32L^2}{\mu}\Delta. μ Δ ≤ ∣ g ( θ ) ∣ 2 2 ≤ μ 32 L 2 Δ.
The first inequality and the descent bound contract Δ \Delta Δ by q q q at each Phase 2 update. After T 2 T_2 T 2 updates, the objective gap is at most μ τ n 2 / ( 32 L 2 ) \mu\tau_n^2/(32L^2) μ τ n 2 / ( 32 L 2 ) . If the subgradient norm still exceeded τ n \tau_n τ n , the second inequality would give ∣ g ( θ ) ∣ 2 2 ≤ τ n 2 \lvert g(\theta)\rvert_2^2 \le\tau_n^2 ∣ g ( θ ) ∣ 2 2 ≤ τ n 2 , a contradiction. This proves the bound on the Phase 2 stopping time.
Step 4. Entry into Phase 3 and the output expansion. We now locate the iterate that satisfies ∣ g ( θ ) ∣ 2 ≤ τ n \lvert g(\theta)\rvert_2 \le\tau_n ∣ g ( θ ) ∣ 2 ≤ τ n . This argument also covers a direct transition from Phase 1 to Phase 3. Applying Condition 2 in both orders between θ ∗ + R u \theta^*+Ru θ ∗ + R u and θ ∗ \theta^* θ ∗ yields
⟨ g ( θ ∗ + R u ) , u ⟩ ≥ μ R 2 − s n > τ n . \langle g(\theta^*+Ru),u\rangle\ge\frac{\mu R}{2}-s_n > \tau_n. ⟨ g ( θ ∗ + R u ) , u ⟩ ≥ 2 μ R − s n > τ n .
Monotonicity of subgradients gives the same lower bound at θ ∗ + t u \theta^*+tu θ ∗ + t u for t > R t>R t > R . An iterate meeting the stopping condition therefore lies inside the ball of radius R R R . Applying Condition 2 in both orders between this iterate and θ ∗ \theta^* θ ∗ gives
∣ θ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 2 ( τ n + s n ) / μ } ≤ ϵ 3 / 2. \lvert\theta-\theta^*\rvert_2 \le\max\{\epsilon_2,2(\tau_n+s_n)/\mu\} \le\epsilon_3/2. ∣ θ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 2 ( τ n + s n ) / μ } ≤ ϵ 3 /2.
Since s n + r n ≤ m H ϵ 3 / 2 s_n+r_n\le m_H\epsilon_3/2 s n + r n ≤ m H ϵ 3 /2 , Theorem 1(iii) keeps every Phase 3 iterate in the required neighborhood. The choice of K K K gives ( 1 − η 3 m H ) K ≤ exp ( − η 3 m H K ) ≤ r n / ( m H ϵ 3 ) (1-\eta_3m_H)^K\le\exp(-\eta_3m_HK)\le r_n/(m_H\epsilon_3) ( 1 − η 3 m H ) K ≤ exp ( − η 3 m H K ) ≤ r n / ( m H ϵ 3 ) . The transient term consequently has norm at most 3 r n / ( 4 m H ) 3r_n/(4m_H) 3 r n / ( 4 m H ) , and the expansion remainder has norm at most r n / m H r_n/m_H r n / m H . Their sum is bounded by 2 r n / m H 2r_n/m_H 2 r n / m H .
Step 5. Accuracy relative to the SAA solution. The same approximation gives the expansion of the SAA solution. Its localization bound above implies ∣ θ ^ − θ ∗ ∣ 2 ≤ ϵ 3 / 4 |\hat{\theta}-\theta^*|_2\le\epsilon_3/4 ∣ θ ^ − θ ∗ ∣ 2 ≤ ϵ 3 /4 . Define A = g ( θ ∗ ) + H ( θ ^ − θ ∗ ) A=g(\theta^*)+H(\hat{\theta}-\theta^*) A = g ( θ ∗ ) + H ( θ ^ − θ ∗ ) . If A ≠ 0 A\ne0 A = 0 , let u = A / ∣ A ∣ 2 u=A/|A|_2 u = A /∣ A ∣ 2 . For sufficiently small t > 0 t>0 t > 0 , the point θ ^ − t u \hat{\theta}-tu θ ^ − t u lies in the ball of radius ϵ 3 \epsilon_3 ϵ 3 . The subgradient inequality and minimality of θ ^ \hat{\theta} θ ^ give
0 ≥ ⟨ g ( θ ^ − t u ) , u ⟩ ≥ ∣ A ∣ 2 − t M H − r n . 0\ge\langle g(\hat{\theta}-tu),u\rangle\ge|A|_2-tM_H-r_n. 0 ≥ ⟨ g ( θ ^ − t u ) , u ⟩ ≥ ∣ A ∣ 2 − t M H − r n .
Letting t ↓ 0 t\downarrow0 t ↓ 0 yields ∣ A ∣ 2 ≤ r n |A|_2\le r_n ∣ A ∣ 2 ≤ r n , which also holds when A = 0 A=0 A = 0 . Thus R ^ n = H − 1 A \widehat{R}_n=H^{-1}A R n = H − 1 A has norm at most r n / m H r_n/m_H r n / m H . Subtracting the two first-order expansions gives ∣ θ ~ − θ ^ ∣ 2 ≤ 3 r n / m H |\tilde{\theta}-\hat{\theta}|_2\le3r_n/m_H ∣ θ ~ − θ ^ ∣ 2 ≤ 3 r n / m H . All conclusions hold on E n \mathcal{E}_n E n , and their joint probability is at least 1 − δ n 1-\delta_n 1 − δ n .
For Nesterov acceleration, fix 0 < η N ≤ M H − 1 0<\eta_N\le M_H^{-1} 0 < η N ≤ M H − 1 and let γ 3 = ( 1 − η N m H ) / ( 1 + η N m H ) \gamma_3=(1-\sqrt{\eta_Nm_H})/(1+\sqrt{\eta_Nm_H}) γ 3 = ( 1 − η N m H ) / ( 1 + η N m H ) . Define the remaining algorithm parameters by
α 2 = μ 4 L , γ 2 = 1 − α 2 1 + α 2 , c 2 = 1 + 2 γ 2 , c 3 = 1 + 2 γ 3 , τ N = μ ϵ 3 8 c 3 . \alpha_2=\sqrt{\frac{\mu}{4L}},\qquad\gamma_2=\frac{1-\alpha_2}{1+\alpha_2},\qquad c_2=1+2\gamma_2,\qquad c_3=1+2\gamma_3,\qquad\tau_N=\frac{\mu\epsilon_3}{8c_3}. α 2 = 4 L μ , γ 2 = 1 + α 2 1 − α 2 , c 2 = 1 + 2 γ 2 , c 3 = 1 + 2 γ 3 , τ N = 8 c 3 μ ϵ 3 .
The iteration counts are
N 1 = ⌈ 16 c 2 ( D + R / 4 ) R μ ⌉ , N 2 = 1 + ⌈ log + ( 4 L 2 R 2 / τ N 2 ) − log ( 1 − α 2 ) ⌉ , N 3 = ⌈ 2 η N m H log + ( 2 m H ϵ 3 r n ) ⌉ . N_1=\left\lceil\frac{16c_2(D+R/4)}{R\sqrt{\mu}}\right\rceil,\qquad N_2=1+\left\lceil\frac{\log_+(4L^2R^2/\tau_N^2)}{-\log(1-\alpha_2)}\right\rceil,\qquad N_3=\left\lceil\frac{2}{\sqrt{\eta_Nm_H}}\log_+\left(\frac{2m_H\epsilon_3}{r_n}\right)\right\rceil. N 1 = ⌈ R μ 16 c 2 ( D + R /4 ) ⌉ , N 2 = 1 + ⌈ − log ( 1 − α 2 ) log + ( 4 L 2 R 2 / τ N 2 ) ⌉ , N 3 = ⌈ η N m H 2 log + ( r n 2 m H ϵ 3 ) ⌉ .
Phase 1 runs Algorithm 3 with η = 1 / ( 2 L ) \eta=1/(2L) η = 1/ ( 2 L ) and the momentum sequence in Theorem 2(i) for at most N 1 N_1 N 1 updates. Before each update, check whether ∣ ∂ L ( β ( t ) ) ∣ 2 ≤ τ N |\partial\mathcal{L}(\beta^{(t)})|_2\le\tau_N ∣ ∂ L ( β ( t ) ) ∣ 2 ≤ τ N . If so, proceed directly to Phase 3 at this extrapolated point. Otherwise, after N 1 N_1 N 1 updates restart Nesterov’s method at θ ( N 1 ) \theta^{(N_1)} θ ( N 1 ) with γ 2 \gamma_2 γ 2 and the same η \eta η in Phase 2, using the same norm test at β ( t ) \beta^{(t)} β ( t ) . Phase 3 restarts at the extrapolated point meeting the test, uses γ 3 , η N \gamma_3,\eta_N γ 3 , η N , and runs N 3 N_3 N 3 updates.
Corollary 9. Suppose the assumptions of Corollary 8 hold. In addition, assume
ϵ 0 ≤ min { τ N 4 L c 2 , R 8 c 2 } , ∣ ∂ L ( θ ∗ ) ∣ 2 ≤ min { μ 16 c 3 , m H 4 ( c 3 + 1 ) } ϵ 3 , r n ≤ m H ϵ 3 4 c 3 . \epsilon_0\le\min\left\{\frac{\tau_N}{4Lc_2},\frac{R}{8c_2}\right\},\qquad|\partial\mathcal{L}(\theta^*)|_2\le\min\left\{\frac{\mu}{16c_3},\frac{m_H}{4(c_3+1)}\right\}\epsilon_3,\qquad r_n\le\frac{m_H\epsilon_3}{4c_3}. ϵ 0 ≤ min { 4 L c 2 τ N , 8 c 2 R } , ∣ ∂ L ( θ ∗ ) ∣ 2 ≤ min { 16 c 3 μ , 4 ( c 3 + 1 ) m H } ϵ 3 , r n ≤ 4 c 3 m H ϵ 3 . On E n \mathcal{E}_n E n , every Phase 1 iterate and extrapolated point remains in K \mathcal{K} K . Phase 2 remains in the strongly convex region and stops within N 2 N_2 N 2 updates. Phase 3 starts within distance ϵ 3 / ( 2 c 3 ) \epsilon_3/(2c_3) ϵ 3 / ( 2 c 3 ) of θ ∗ \theta^* θ ∗ , and every subsequent extrapolated iterate remains in the radius-ϵ 3 \epsilon_3 ϵ 3 ball. After at most N 1 + N 2 + N 3 N_1+N_2+N_3 N 1 + N 2 + N 3 updates, the computed solution satisfies (38) with ∣ R n ∣ 2 ≤ 2 r n / m H |R_n|_2\le2r_n/m_H ∣ R n ∣ 2 ≤ 2 r n / m H and ∣ θ ~ − θ ^ ∣ 2 ≤ 3 r n / m H |\tilde{\theta}-\hat{\theta}|_2\le3r_n/m_H ∣ θ ~ − θ ^ ∣ 2 ≤ 3 r n / m H .
Proof of Corollary 9. Work on E n \mathcal{E}_n E n and write g = ∂ L g=\partial\mathcal{L} g = ∂ L and s n = ∣ g ( θ ∗ ) ∣ 2 s_n=|g(\theta^*)|_2 s n = ∣ g ( θ ∗ ) ∣ 2 .
Step 1. Entry into Phase 2. The localization and selected-subgradient arguments in the proof of Corollary 8 give
∣ θ ^ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 4 s n / μ } ≤ ϵ 3 / 4 ≤ R / 8 , ∣ g ( θ ^ ) ∣ 2 ≤ L ϵ 1 . |\hat{\theta}-\theta^*|_2\le\max\{\epsilon_2,4s_n/\mu\}\le\epsilon_3/4\le R/8,\qquad|g(\hat{\theta})|_2\le L\epsilon_1. ∣ θ ^ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 4 s n / μ } ≤ ϵ 3 /4 ≤ R /8 , ∣ g ( θ ^ ) ∣ 2 ≤ L ϵ 1 .
Write d 0 = ∣ θ ( 0 ) − θ ^ ∣ 2 ≤ D + R / 4 d_0=|\theta^{(0)}-\hat{\theta}|_2\le D+R/4 d 0 = ∣ θ ( 0 ) − θ ^ ∣ 2 ≤ D + R /4 . The Phase 1 threshold obeys η τ N ≥ ϵ 1 \eta\tau_N\ge\epsilon_1 η τ N ≥ ϵ 1 , and every executed update has length at least ϵ 1 \epsilon_1 ϵ 1 . The potential and convex-combination identities in (31) keep every completed iterate and the next extrapolated point within distance d 0 d_0 d 0 of θ ^ \hat{\theta} θ ^ . The proposed next iterate satisfies
∣ θ ( t + 1 ) − θ ∗ ∣ 2 ≤ 3 2 d 0 + ϵ 1 + ∣ θ ^ − θ ∗ ∣ 2 ≤ 2 D + R . (39) |\theta^{(t+1)}-\theta^*|_2\le\frac{3}{2}d_0+\epsilon_1+|\hat{\theta}-\theta^*|_2\le2D+R. \tag*{(39)} ∣ θ ( t + 1 ) − θ ∗ ∣ 2 ≤ 2 3 d 0 + ϵ 1 + ∣ θ ^ − θ ∗ ∣ 2 ≤ 2 D + R . (39)
It therefore lies in K \mathcal{K} K , validating the next potential inequality. Induction proves Phase 1 containment before the norm test stops the phase or all N 1 N_1 N 1 updates are completed. If Phase 1 does not stop early, Theorem 2(i) gives
L ( θ ( N 1 ) ) − L ( θ ^ ) ≤ 2 ( D + R / 4 ) 2 η N 1 2 ≤ μ R 2 128 c 2 2 . (40) \mathcal{L}(\theta^{(N_1)})-\mathcal{L}(\widehat{\theta})\leq\frac{2(D+R/4)^2}{\eta N_1^2}\leq\frac{\mu R^2}{128c_2^2}. \tag*{(40)} L ( θ ( N 1 ) ) − L ( θ ) ≤ η N 1 2 2 ( D + R /4 ) 2 ≤ 128 c 2 2 μ R 2 . (40)
This iterate lies in the radius-R R R ball, since every point outside that ball has objective gap at least μ R 2 / 8 \mu R^2/8 μ R 2 /8 .
Step 2. Containment during accelerated Phase 2. Index Phase 2 from zero and restart with θ ( − 1 ) = θ ( 0 ) \theta^{(-1)}=\theta^{(0)} θ ( − 1 ) = θ ( 0 ) . Use the potential (35) with α = α 2 \alpha=\alpha_2 α = α 2 and η = 1 / ( 2 L ) \eta=1/(2L) η = 1/ ( 2 L ) , and denote it by E t E_t E t . Write Δ t = L ( θ ( t ) ) − L ( θ ^ ) \Delta_t=\mathcal{L}(\theta^{(t)})-\mathcal{L}(\widehat{\theta}) Δ t = L ( θ ( t ) ) − L ( θ ) . For a point in the radius-R R R ball, (22) and ϵ 0 ≥ 2 ϵ 2 \epsilon_0\geq\sqrt{2}\epsilon_2 ϵ 0 ≥ 2 ϵ 2 give
∥ θ − θ ^ ∥ 2 2 ≤ max { ϵ 0 2 , 8 [ L ( θ ) − L ( θ ^ ) ] / μ } . \lVert\theta-\widehat{\theta}\rVert_2^2\leq\max\{\epsilon_0^2,8[\mathcal{L}(\theta)-\mathcal{L}(\widehat{\theta})]/\mu\}. ∥ θ − θ ∥ 2 2 ≤ max { ϵ 0 2 , 8 [ L ( θ ) − L ( θ )] / μ } .
Since α 2 2 / ( 2 η ) = μ / 4 \alpha_2^2/(2\eta)=\mu/4 α 2 2 / ( 2 η ) = μ /4 , the initial potential satisfies
E 0 ≤ 3 Δ 0 + μ 4 ϵ 0 2 ≤ 7 μ R 2 256 c 2 2 < μ R 2 32 c 2 2 . E_0\leq3\Delta_0+\frac{\mu}{4}\epsilon_0^2\leq\frac{7\mu R^2}{256c_2^2}<\frac{\mu R^2}{32c_2^2}. E 0 ≤ 3 Δ 0 + 4 μ ϵ 0 2 ≤ 256 c 2 2 7 μ R 2 < 32 c 2 2 μ R 2 .
We prove inductively that every Phase 2 iterate lies in the radius-R R R ball and E t ≤ ( 1 − α 2 ) t E 0 E_t\leq(1-\alpha_2)^tE_0 E t ≤ ( 1 − α 2 ) t E 0 until the norm test stops the phase. The bound holds at initialization. If it holds at two consecutive iterates, the last two displays give
∥ θ ( t ) − θ ^ ∥ 2 , ∥ θ ( t − 1 ) − θ ^ ∥ 2 ≤ R 2 c 2 , ∥ β ( t ) − θ ^ ∥ 2 ≤ R 2 . \lVert\theta^{(t)}-\widehat{\theta}\rVert_2,\lVert\theta^{(t-1)}-\widehat{\theta}\rVert_2\leq\frac{R}{2c_2},\qquad\lVert\beta^{(t)}-\widehat{\theta}\rVert_2\leq\frac{R}{2}. ∥ θ ( t ) − θ ∥ 2 , ∥ θ ( t − 1 ) − θ ∥ 2 ≤ 2 c 2 R , ∥ β ( t ) − θ ∥ 2 ≤ 2 R .
The same conclusion at t = 0 t=0 t = 0 follows from the restart. Thus β ( t ) \beta^{(t)} β ( t ) lies in the radius-R R R ball. Approximate smoothness bounds the next proposed iterate by
∥ θ ( t + 1 ) − θ ∗ ∥ 2 ≤ 3 2 ∥ β ( t ) − θ ^ ∥ 2 + ϵ 1 + ∥ θ ^ − θ ∗ ∥ 2 ≤ 7 R 8 + ϵ 1 ≤ R . \lVert\theta^{(t+1)}-\theta^*\rVert_2\leq\frac{3}{2}\lVert\beta^{(t)}-\widehat{\theta}\rVert_2+\epsilon_1+\lVert\widehat{\theta}-\theta^*\rVert_2\leq\frac{7R}{8}+\epsilon_1\leq R. ∥ θ ( t + 1 ) − θ ∗ ∥ 2 ≤ 2 3 ∥ β ( t ) − θ ∥ 2 + ϵ 1 + ∥ θ − θ ∗ ∥ 2 ≤ 8 7 R + ϵ 1 ≤ R .
If the phase has not stopped, ∥ g ( β ( t ) ) ∥ 2 > τ N \lVert g(\beta^{(t)})\rVert_2>\tau_N ∥ g ( β ( t ) ) ∥ 2 > τ N implies ∥ β ( t ) − θ ^ ∥ 2 ≥ ϵ 0 \lVert\beta^{(t)}-\widehat{\theta}\rVert_2\geq\epsilon_0 ∥ β ( t ) − θ ∥ 2 ≥ ϵ 0 by the selected-subgradient bound and approximate smoothness. Its update length is at least η τ N ≥ ϵ 1 \eta\tau_N\geq\epsilon_1 η τ N ≥ ϵ 1 . All pairs used in Theorem 2(ii) therefore lie in their required regions before its potential inequality is applied. Equation (36) , with α = α 2 \alpha=\alpha_2 α = α 2 and both additive terms absent under these distance conditions, gives E t + 1 ≤ ( 1 − α 2 ) E t E_{t+1}\leq(1-\alpha_2)E_t E t + 1 ≤ ( 1 − α 2 ) E t , completing the induction.
Step 3. The norm stopping rule. After N 2 − 1 N_2-1 N 2 − 1 updates, the potential is at most μ τ N 2 / ( 128 L 2 c 2 2 ) \mu\tau_N^2/(128L^2c_2^2) μ τ N 2 / ( 128 L 2 c 2 2 ) . If another update is needed, the same bound holds for both E N 2 − 1 E_{N_2-1} E N 2 − 1 and E N 2 E_{N_2} E N 2 . The growth inequality in Step 2 and ϵ 0 ≤ τ N / ( 4 L c 2 ) \epsilon_0\leq\tau_N/(4Lc_2) ϵ 0 ≤ τ N / ( 4 L c 2 ) then give
∥ θ ( N 2 ) − θ ^ ∥ 2 , ∥ θ ( N 2 − 1 ) − θ ^ ∥ 2 ≤ τ N 4 L c 2 , ∥ β ( N 2 ) − θ ^ ∥ 2 ≤ τ N 4 L . \lVert\theta^{(N_2)}-\widehat{\theta}\rVert_2,\lVert\theta^{(N_2-1)}-\widehat{\theta}\rVert_2\leq\frac{\tau_N}{4Lc_2},\qquad\lVert\beta^{(N_2)}-\widehat{\theta}\rVert_2\leq\frac{\tau_N}{4L}. ∥ θ ( N 2 ) − θ ∥ 2 , ∥ θ ( N 2 − 1 ) − θ ∥ 2 ≤ 4 L c 2 τ N , ∥ β ( N 2 ) − θ ∥ 2 ≤ 4 L τ N .
Consequently ∥ g ( β ( N 2 ) ) ∥ 2 ≤ τ N / 4 + L ϵ 1 ≤ τ N / 2 \lVert g(\beta^{(N_2)})\rVert_2\leq\tau_N/4+L\epsilon_1\leq\tau_N/2 ∥ g ( β ( N 2 ) ) ∥ 2 ≤ τ N /4 + L ϵ 1 ≤ τ N /2 . Hence the phase stops within N 2 N_2 N 2 updates.
Step 4. Entry into Phase 3 and the final error. Step 4 of the proof of Corollary 8 applies to the stopping point in either Phase 1 or Phase 2 and yields
∥ θ ( 0 ) − θ ∗ ∥ 2 ≤ max { ϵ 2 , 2 ( τ N + s n ) / μ } ≤ ϵ 3 2 c 3 . \lVert\theta^{(0)}-\theta^*\rVert_2\leq\max\{\epsilon_2,2(\tau_N+s_n)/\mu\}\leq\frac{\epsilon_3}{2c_3}. ∥ θ ( 0 ) − θ ∗ ∥ 2 ≤ max { ϵ 2 , 2 ( τ N + s n ) / μ } ≤ 2 c 3 ϵ 3 .
Define z 0 = θ ( 0 ) − θ ∗ + H − 1 g ( θ ∗ ) z_0 = \theta^{(0)} - \theta^* + H^{-1}g(\theta^*) z 0 = θ ( 0 ) − θ ∗ + H − 1 g ( θ ∗ ) . The score and remainder bounds give
c 3 ( ∣ z 0 ∣ 2 + r n / m H ) + ∣ H − 1 g ( θ ∗ ) ∣ 2 ≤ ϵ 3 2 + ( c 3 + 1 ) s n + c 3 r n m H ≤ ϵ 3 . c_3\left(|z_0|_2 + r_n/m_H\right) + |H^{-1}g(\theta^*)|_2 \le\frac{\epsilon_3}{2} + \frac{(c_3+1)s_n+c_3r_n}{m_H} \le\epsilon_3. c 3 ( ∣ z 0 ∣ 2 + r n / m H ) + ∣ H − 1 g ( θ ∗ ) ∣ 2 ≤ 2 ϵ 3 + m H ( c 3 + 1 ) s n + c 3 r n ≤ ϵ 3 .
Theorem 2(iii) therefore proves containment. Also ∣ z 0 ∣ 2 ≤ ϵ 3 |z_0|_2 \le\epsilon_3 ∣ z 0 ∣ 2 ≤ ϵ 3 , and the choice of N 3 N_3 N 3 gives 2 exp { − η N m H N 3 / 2 } ∣ z 0 ∣ 2 ≤ r n / m H 2\exp\{-\sqrt{\eta_Nm_H}N_3/2\}|z_0|_2 \le r_n/m_H 2 exp { − η N m H N 3 /2 } ∣ z 0 ∣ 2 ≤ r n / m H . Adding the local remainder gives ∣ R n ∣ 2 ≤ 2 r n / m H |R_n|_2 \le2r_n/m_H ∣ R n ∣ 2 ≤ 2 r n / m H . The SAA expansion in Corollary 8 then gives ∣ θ ~ − θ ^ ∣ 2 ≤ 3 r n / m H |\widetilde{\theta}-\widehat{\theta}|_2 \le3r_n/m_H ∣ θ − θ ∣ 2 ≤ 3 r n / m H on E n \mathcal{E}_n E n .
Acceleration reduces the Phase 1 count from order ( L / μ ) ( 1 + D / R ) 2 (L/\mu)(1+D/R)^2 ( L / μ ) ( 1 + D / R ) 2 to L / μ ( 1 + D / R ) \sqrt{L/\mu}(1+D/R) L / μ ( 1 + D / R ) and the Phase 2 count from order ( L / μ ) log ( R / τ n ) (L/\mu)\log(R/\tau_n) ( L / μ ) log ( R / τ n ) to L / μ log ( R / τ N ) \sqrt{L/\mu}\log(R/\tau_N) L / μ log ( R / τ N ) . At η N = M H − 1 \eta_N=M_H^{-1} η N = M H − 1 , Phase 3 takes O { M H / m H log ( m H ϵ 3 / r n ) } O\{\sqrt{M_H/m_H}\log(m_H\epsilon_3/r_n)\} O { M H / m H log ( m H ϵ 3 / r n )} updates with the same expansion remainder bound as subgradient descent. Polynomially decreasing approximation scales give O ( log n ) O(\log n) O ( log n ) updates. Each update evaluates n n n scenario subgradients.
Condition 3 and the entry conditions of the complete procedures give containment in B ( θ ∗ , ϵ 3 ) \mathbb{B}(\theta^*,\epsilon_3) B ( θ ∗ , ϵ 3 ) for every Phase 3 iterate and extrapolated point by Theorems 1(iii) and 2(iii). The SAA solution also lies in this ball. Write g = ∂ L g = \partial\mathcal{L} g = ∂ L . Suppose that, on this ball, for some 0 ≤ δ ≤ m H / 64 0 \le\delta\le m_H/64 0 ≤ δ ≤ m H /64 and ζ n ≥ 0 \zeta_n \ge0 ζ n ≥ 0 ,
∣ g ( θ ) − H ( θ − θ ^ ) ∣ 2 ≤ δ ∣ θ − θ ^ ∣ 2 + ζ n . (41) |g(\theta)-H(\theta-\widehat{\theta})|_2 \le\delta|\theta-\widehat{\theta}|_2+\zeta_n. \tag*{(41)} ∣ g ( θ ) − H ( θ − θ ) ∣ 2 ≤ δ ∣ θ − θ ∣ 2 + ζ n . (41)
Index Phase 3 from zero and use the restarted momentum in the accelerated procedure. Then
∣ θ ( t ) − θ ^ ∣ 2 ≤ 4 q l o c t ∣ θ ( 0 ) − θ ^ ∣ 2 + 2 ζ n m H , q l o c = { 1 − η 3 m H / 2 , subgradient descent , 1 − η N m H / 2 , Nesterov acceleration . (42) |\theta^{(t)}-\widehat{\theta}|_2 \le4q_{\mathrm{loc}}^t|\theta^{(0)}-\widehat{\theta}|_2+\frac{2\zeta_n}{m_H}, \qquad
q_{\mathrm{loc}} =
\begin{cases}
1-\eta_3m_H/2, & \text{subgradient descent},\\
1-\sqrt{\eta_Nm_H}/2, & \text{Nesterov acceleration}.
\end{cases} \tag*{(42)} ∣ θ ( t ) − θ ∣ 2 ≤ 4 q loc t ∣ θ ( 0 ) − θ ∣ 2 + m H 2 ζ n , q loc = { 1 − η 3 m H /2 , 1 − η N m H /2 , subgradient descent , Nesterov acceleration . (42)
For subgradient descent, set d t = θ ( t ) − θ ^ d_t=\theta^{(t)}-\widehat{\theta} d t = θ ( t ) − θ . Since 0 < η 3 ≤ M H − 1 0<\eta_3\le M_H^{-1} 0 < η 3 ≤ M H − 1 , (41) gives
∣ d t + 1 ∣ 2 ≤ ( 1 − η 3 m H + η 3 δ ) ∣ d t ∣ 2 + η 3 ζ n ≤ q l o c ∣ d t ∣ 2 + η 3 ζ n . |d_{t+1}|_2 \le(1-\eta_3m_H+\eta_3\delta)|d_t|_2+\eta_3\zeta_n \le q_{\mathrm{loc}}|d_t|_2+\eta_3\zeta_n. ∣ d t + 1 ∣ 2 ≤ ( 1 − η 3 m H + η 3 δ ) ∣ d t ∣ 2 + η 3 ζ n ≤ q loc ∣ d t ∣ 2 + η 3 ζ n .
Summing this recursion proves (42) .
For Nesterov acceleration, let d − 1 = d 0 d_{-1}=d_0 d − 1 = d 0 , a = η N m H a=\sqrt{\eta_Nm_H} a = η N m H , ρ = 1 − a \rho=1-a ρ = 1 − a , and q l o c = 1 − a / 2 q_{\mathrm{loc}}=1-a/2 q loc = 1 − a /2 . Use the polynomials P t P_t P t and K j K_j K j from the proof of Theorem 2(iii), with η N \eta_N η N in place of η \eta η . Their bounds imply
∥ P t ∥ ≤ 2 q l o c t , η N ∑ j ≥ 0 ∥ K j ∥ ≤ 1 m H , η N ∑ j ≥ 0 ∥ K j ∥ q l o c − j − 1 ≤ η N q l o c ( q l o c − ρ ) 2 ≤ 4 m H . \lVert P_t\rVert\le2q_{\mathrm{loc}}^t,\qquad
\eta_N\sum_{j\ge0}\lVert K_j\rVert\le\frac{1}{m_H},\qquad
\eta_N\sum_{j\ge0}\lVert K_j\rVert q_{\mathrm{loc}}^{-j-1}
\le\frac{\eta_Nq_{\mathrm{loc}}}{(q_{\mathrm{loc}}-\rho)^2}
\le\frac{4}{m_H}. ∥ P t ∥ ≤ 2 q loc t , η N j ≥ 0 ∑ ∥ K j ∥ ≤ m H 1 , η N j ≥ 0 ∑ ∥ K j ∥ q loc − j − 1 ≤ ( q loc − ρ ) 2 η N q loc ≤ m H 4 .
Here the first bound follows from ( 1 + a t ) ( ρ / q l o c ) t ≤ ( 1 + a t ) e − a t / 2 ≤ 2 (1+at)(\rho/q_{\mathrm{loc}})^t\le(1+at)e^{-at/2}\le2 ( 1 + a t ) ( ρ / q loc ) t ≤ ( 1 + a t ) e − a t /2 ≤ 2 . Let b t = β ( t ) − θ ^ = ( 1 + γ 3 ) d t − γ 3 d t − 1 b_t=\beta^{(t)}-\widehat{\theta}=(1+\gamma_3)d_t-\gamma_3d_{t-1} b t = β ( t ) − θ = ( 1 + γ 3 ) d t − γ 3 d t − 1 and e t = g ( β ( t ) ) − H b t e_t=g(\beta^{(t)})-Hb_t e t = g ( β ( t ) ) − H b t . The local recursion has solution
d t = P t d 0 − η N ∑ j = 0 t − 1 K t − 1 − j e j , ∣ e j ∣ 2 ≤ δ ∣ b j ∣ 2 + ζ n . d_t=P_td_0-\eta_N\sum_{j=0}^{t-1}K_{t-1-j}e_j,\qquad|e_j|_2\le\delta|b_j|_2+\zeta_n. d t = P t d 0 − η N j = 0 ∑ t − 1 K t − 1 − j e j , ∣ e j ∣ 2 ≤ δ ∣ b j ∣ 2 + ζ n .
Put D 0 = ∣ d 0 ∣ 2 D_0=|d_0|_2 D 0 = ∣ d 0 ∣ 2 and B = 2 ζ n / m H B=2\zeta_n/m_H B = 2 ζ n / m H . If ∣ d j ∣ 2 ≤ 4 q l o c j D 0 + B |d_j|_2\le4q_{\mathrm{loc}}^jD_0+B ∣ d j ∣ 2 ≤ 4 q loc j D 0 + B up to index t − 1 t-1 t − 1 , then q l o c ≥ 1 / 2 q_{\mathrm{loc}}\ge1/2 q loc ≥ 1/2 and 0 ≤ γ 3 < 1 0\le\gamma_3<1 0 ≤ γ 3 < 1 give ∣ b j ∣ 2 ≤ 16 q l o c j D 0 + 3 B |b_j|_2\le16q_{\mathrm{loc}}^jD_0+3B ∣ b j ∣ 2 ≤ 16 q loc j D 0 + 3 B . The three polynomial bounds therefore yield
∣ d t ∣ 2 ≤ ( 2 + 64 δ m H ) q l o c t D 0 + 3 δ B + ζ n m H ≤ 4 q l o c t D 0 + B . |d_t|_2\le\left(2+\frac{64\delta}{m_H}\right)q_{\mathrm{loc}}^tD_0+\frac{3\delta B+\zeta_n}{m_H}\le4q_{\mathrm{loc}}^tD_0+B. ∣ d t ∣ 2 ≤ ( 2 + m H 64 δ ) q loc t D 0 + m H 3 δ B + ζ n ≤ 4 q loc t D 0 + B .
The initialization satisfies the same bound, including the convention d − 1 = d 0 d_{-1}=d_0 d − 1 = d 0 . Induction proves (42) for every t t t .
Under the model and population assumptions of Theorem 4 and the region and initialization conditions in the unprojected extension, choose ϵ 3 = C 3 a n \epsilon_3 = C_3\sqrt{a_n} ϵ 3 = C 3 a n and r n = C 4 a n 3 / 4 r_n = C_4a_n^{3/4} r n = C 4 a n 3/4 . For sufficiently large C 3 C_3 C 3 , the score bound in the proof of that theorem verifies the score inequalities in Corollaries 8 and 9. Their remaining scale inequalities follow from ϵ 0 = O ( a n ) = o ( ϵ 3 ) \epsilon_0 = O(a_n) = o(\epsilon_3) ϵ 0 = O ( a n ) = o ( ϵ 3 ) and r n / ϵ 3 → 0 r_n/\epsilon_3 \to0 r n / ϵ 3 → 0 . The ball inclusion verifies their regions and gives containment for the Phase 3 iterates and extrapolated points. The increment bound in Theorem 3, applied to θ \theta θ and θ ^ \hat{\theta} θ ^ , and the Lipschitz population Hessian give, with d = ∣ θ − θ ^ ∣ 2 d = \lvert\theta-\hat{\theta}\rvert_2 d = ∣ θ − θ ^ ∣ 2 ,
∣ g ( θ ) − H ( θ − θ ^ ) ∣ 2 ≤ ∣ g ( θ ^ ) ∣ 2 + C { d a n + a n + ϵ 3 d } . \lvert g(\theta)-H(\theta-\hat{\theta})\rvert_2 \leq\lvert g(\hat{\theta})\rvert_2+C\{\sqrt{d a_n}+a_n+\epsilon_3d\}. ∣ g ( θ ) − H ( θ − θ ^ ) ∣ 2 ≤ ∣ g ( θ ^ ) ∣ 2 + C { d a n + a n + ϵ 3 d } .
Lemma 2 bounds ∣ g ( θ ^ ) ∣ 2 \lvert g(\hat{\theta})\rvert_2 ∣ g ( θ ^ ) ∣ 2 by L ϵ 1 = O ( a n ) L\epsilon_1=O(a_n) L ϵ 1 = O ( a n ) . Young’s inequality bounds C d a n C\sqrt{d a_n} C d a n by δ d / 2 + C δ a n \delta d/2+C_\delta a_n δ d /2 + C δ a n , and C ϵ 3 ≤ δ / 2 C\epsilon_3\leq\delta/2 C ϵ 3 ≤ δ /2 for sufficiently large n n n . Taking δ = m H / 64 \delta=m_H/64 δ = m H /64 thus verifies (39) with ζ n = O ( a n ) \zeta_n=O(a_n) ζ n = O ( a n ) . Equation (42) gives the O ( a n ) O(a_n) O ( a n ) optimization error with the Phase 3 counts stated after Theorem 4. At η 3 = η N = M H − 1 \eta_3=\eta_N=M_H^{-1} η 3 = η N = M H − 1 , these counts have condition-number dependence M H / m H M_H/m_H M H / m H and M H / m H \sqrt{M_H/m_H} M H / m H , respectively.
The projected procedures admit the interior switches in Corollary 10(iii)(a), with the SAA calibration in part (iii)(b). Combining each optimization bound with the SAA expansion in the proof of Theorem 4 gives its parts (ii) and (iii) and the population-gap bound in part (i) for the terminal outputs.
Other update rules
Let C ⊆ R p C\subseteq\mathbb{R}^p C ⊆ R p be nonempty, closed, and convex, and let Π C \Pi_C Π C be its Euclidean projection. Projected subgradient descent starts at x 0 ∈ C x_0\in C x 0 ∈ C and uses x t + 1 = Π C { x t − η ∂ L ( x t ) } x_{t+1}=\Pi_C\{x_t-\eta\partial\mathcal{L}(x_t)\} x t + 1 = Π C { x t − η ∂ L ( x t )} . The accelerated method starts a block at x 0 = z 0 ∈ C x_0=z_0\in C x 0 = z 0 ∈ C , sets λ 0 = 1 \lambda_0=1 λ 0 = 1 and λ t + 1 = ( 1 + 1 + 4 λ t 2 ) / 2 \lambda_{t+1}=(1+\sqrt{1+4\lambda_t^2})/2 λ t + 1 = ( 1 + 1 + 4 λ t 2 ) /2 , and uses
y t = ( 1 − λ t − 1 ) x t + λ t − 1 z t , z t + 1 = Π C { z t − η λ t ∂ L ( y t ) } , x t + 1 = ( 1 − λ t − 1 ) x t + λ t − 1 z t + 1 . (43) \begin{aligned}
y_t&=(1-\lambda_t^{-1})x_t+\lambda_t^{-1}z_t,\\
z_{t+1}&=\Pi_C\{z_t-\eta\lambda_t\partial\mathcal{L}(y_t)\},\\
x_{t+1}&=(1-\lambda_t^{-1})x_t+\lambda_t^{-1}z_{t+1}. \tag*{(43)}
\end{aligned} y t z t + 1 x t + 1 = ( 1 − λ t − 1 ) x t + λ t − 1 z t , = Π C { z t − η λ t ∂ L ( y t )} , = ( 1 − λ t − 1 ) x t + λ t − 1 z t + 1 . (43)
All three sequences are feasible. A restart sets λ 0 = 1 \lambda_0=1 λ 0 = 1 and both initial sequences to the preceding block output.
Corollary 10. Let L \mathcal{L} L be finite and convex on C C C . For parts (i) and (ii), suppose θ ^ ∈ argmin θ ∈ C L ( θ ) \hat{\theta}\in\operatorname*{argmin}_{\theta\in C}\mathcal{L}(\theta) θ ^ ∈ argmin θ ∈ C L ( θ ) exists, impose Condition 1 on C C C , and take 0 < η ≤ 1 / ( 2 L ) 0<\eta\leq1/(2L) 0 < η ≤ 1/ ( 2 L ) . Write Δ ( x ) = L ( x ) − L ( θ ^ ) \Delta(x)=\mathcal{L}(x)-\mathcal{L}(\hat{\theta}) Δ ( x ) = L ( x ) − L ( θ ^ ) , D 0 = ∣ x 0 − θ ^ ∣ 2 D_0=\lvert x_0-\hat{\theta}\rvert_2 D 0 = ∣ x 0 − θ ^ ∣ 2 , and δ = η L 2 ϵ 1 2 / 2 \delta=\eta L^2\epsilon_1^2/2 δ = η L 2 ϵ 1 2 /2 .
(i) Let T ≥ 1 T\geq1 T ≥ 1 .
(a) Projected subgradient descent satisfies
1 T ∑ t = 1 T Δ ( x t ) ≤ D 0 2 2 η T + δ . \frac{1}{T}\sum_{t=1}^{T}\Delta(x_t)\leq\frac{D_0^2}{2\eta T}+\delta. T 1 t = 1 ∑ T Δ ( x t ) ≤ 2 η T D 0 2 + δ . If ∣ x t + 1 − x t ∣ 2 ≥ ϵ 1 \lvert x_{t+1}-x_t\rvert_2\geq\epsilon_1 ∣ x t + 1 − x t ∣ 2 ≥ ϵ 1 for 0 ≤ t < T 0\leq t<T 0 ≤ t < T , its objective values are nonincreasing and Δ ( x T ) ≤ D 0 2 / ( 2 η T ) \Delta(x_T)\leq D_0^2/(2\eta T) Δ ( x T ) ≤ D 0 2 / ( 2 η T ) .
(b) For a block of T T T accelerated updates, put δ t = δ I { ∣ x t + 1 − y t ∣ 2 < ϵ 1 } \delta_t=\delta\mathbb{I}\{\lvert x_{t+1}-y_t\rvert_2<\epsilon_1\} δ t = δ I {∣ x t + 1 − y t ∣ 2 < ϵ 1 } . Then
Δ ( x T ) ≤ D 0 2 / ( 2 η ) + ∑ t = 0 T − 1 λ t 2 δ t λ T − 1 2 ≤ 2 D 0 2 η T 2 + T δ . (44) \Delta(x_T)\leq\frac{D_0^2/(2\eta)+\sum_{t=0}^{T-1}\lambda_t^2\delta_t}{\lambda_{T-1}^2}\leq\frac{2D_0^2}{\eta T^2}+T\delta. \tag*{(44)} Δ ( x T ) ≤ λ T − 1 2 D 0 2 / ( 2 η ) + ∑ t = 0 T − 1 λ t 2 δ t ≤ η T 2 2 D 0 2 + T δ . (44)
Every x t x_t x t , z t z_t z t for 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T and y t y_t y t for 0 ≤ t < T 0 \le t < T 0 ≤ t < T has distance at most { D 0 2 + 2 η δ T 3 } 1 / 2 \{D_0^2 + 2\eta\delta T^3\}^{1/2} { D 0 2 + 2 η δ T 3 } 1/2 from θ ^ \hat{\theta} θ ^ . If every actual displacement ∣ x t + 1 − y t ∣ 2 \lvert x_{t+1}-y_t\rvert_2 ∣ x t + 1 − y t ∣ 2 is at least ϵ 1 \epsilon_1 ϵ 1 , the radius is D 0 D_0 D 0 and the additive objective error vanishes.
(ii) (a) Suppose Condition 2 holds on C C C and α = η μ / 2 ≤ 1 \alpha=\eta\mu/2 \le1 α = η μ /2 ≤ 1 . Projected subgradient descent satisfies
∣ x T − θ ^ ∣ 2 2 ≤ ( 1 − α ) T D 0 2 + F G , F G = ϵ 2 2 + 2 η δ α , \lvert x_T-\hat{\theta}\rvert_2^2 \le(1-\alpha)^T D_0^2+F_G,\qquad F_G=\epsilon_2^2+\frac{2\eta\delta}{\alpha}, ∣ x T − θ ^ ∣ 2 2 ≤ ( 1 − α ) T D 0 2 + F G , F G = ϵ 2 2 + α 2 η δ ,
Δ ( x T ) ≤ ( 1 − α ) T Δ ( x 0 ) + μ ϵ 2 2 4 + δ α . \Delta(x_T) \le(1-\alpha)^T\Delta(x_0)+\frac{\mu\epsilon_2^2}{4}+\frac{\delta}{\alpha}. Δ ( x T ) ≤ ( 1 − α ) T Δ ( x 0 ) + 4 μ ϵ 2 2 + α δ .
If ∣ x t + 1 − x t ∣ 2 ≥ ϵ 1 \lvert x_{t+1}-x_t\rvert_2 \ge\epsilon_1 ∣ x t + 1 − x t ∣ 2 ≥ ϵ 1 and ∣ x t − θ ^ ∣ 2 ≥ ϵ 2 \lvert x_t-\hat{\theta}\rvert_2 \ge\epsilon_2 ∣ x t − θ ^ ∣ 2 ≥ ϵ 2 for 0 ≤ t < T 0 \le t<T 0 ≤ t < T , both additive errors vanish.
(b) Suppose Condition 2 holds on C C C and η μ ≤ 1 \eta\mu\le1 η μ ≤ 1 . Set K = ⌈ 32 / ( η μ ) ⌉ K=\lceil\sqrt{32/(\eta\mu)}\rceil K = ⌈ 32/ ( η μ ) ⌉ and E = ϵ 2 2 + 4 K δ / μ E=\epsilon_2^2+4K\delta/\mu E = ϵ 2 2 + 4 K δ / μ . Restart the accelerated method every K K K updates, write θ ^ j \hat{\theta}_j θ ^ j for the output of j j j blocks, and put D j = ∣ θ ^ j − θ ^ ∣ 2 D_j=\lvert\hat{\theta}_j-\hat{\theta}\rvert_2 D j = ∣ θ ^ j − θ ^ ∣ 2 . Then
D j 2 ≤ 4 − j D 0 2 + 4 3 ( 1 − 4 − j ) E , j ≥ 0 , (45) D_j^2 \le4^{-j}D_0^2+\frac{4}{3}(1-4^{-j})E,\qquad j\ge0, \tag*{(45)} D j 2 ≤ 4 − j D 0 2 + 3 4 ( 1 − 4 − j ) E , j ≥ 0 , (45)
and Δ ( θ ^ j ) ≤ ( μ / 4 ) 4 − j D 0 2 + μ E / 3 \Delta(\hat{\theta}_j)\le(\mu/4)4^{-j}D_0^2+\mu E/3 Δ ( θ ^ j ) ≤ ( μ /4 ) 4 − j D 0 2 + μ E /3 for j ≥ 1 j\ge1 j ≥ 1 . Block j j j starts at θ ^ j \hat{\theta}_j θ ^ j . The points in this block satisfy
∣ x t − θ ^ ∣ 2 2 , ∣ y t − θ ^ ∣ 2 2 , ∣ z t − θ ^ ∣ 2 2 ≤ 4 − j D 0 2 + 26 E , (46) \lvert x_t-\hat{\theta}\rvert_2^2,\ \lvert y_t-\hat{\theta}\rvert_2^2,\ \lvert z_t-\hat{\theta}\rvert_2^2 \le4^{-j}D_0^2+26E, \tag*{(46)} ∣ x t − θ ^ ∣ 2 2 , ∣ y t − θ ^ ∣ 2 2 , ∣ z t − θ ^ ∣ 2 2 ≤ 4 − j D 0 2 + 26 E , (46)
for x t , z t x_t,z_t x t , z t with 0 ≤ t ≤ K 0\le t\le K 0 ≤ t ≤ K and y t y_t y t with 0 ≤ t < K 0\le t<K 0 ≤ t < K .
(c) Suppose Condition 2 holds only on C ∩ B ( θ ^ , R ) C\cap\mathbb{B}(\hat{\theta},R) C ∩ B ( θ ^ , R ) , for R > 0 R>0 R > 0 . For projected subgradient descent, if α ≤ 1 \alpha\le1 α ≤ 1 , F G ≤ R 2 / 4 F_G\le R^2/4 F G ≤ R 2 /4 , and D 0 ≤ R / 2 D_0\le R/2 D 0 ≤ R /2 , all iterates stay within distance R / 2 R/2 R /2 and the bounds in part (ii)(a) hold. For the accelerated method, retain η μ ≤ 1 \eta\mu\le1 η μ ≤ 1 and assume
ϵ 2 2 + 2 K δ μ ≤ R 2 256 . (47) \epsilon_2^2+\frac{2K\delta}{\mu}\le\frac{R^2}{256}. \tag*{(47)} ϵ 2 2 + μ 2 K δ ≤ 256 R 2 . (47)
If D 0 ≤ R / 2 D_0\le R/2 D 0 ≤ R /2 , all points in each block remain in C ∩ B ( θ ^ , R ) C\cap\mathbb{B}(\hat{\theta},R) C ∩ B ( θ ^ , R ) and the bounds in part (ii)(b) hold.
For entry from any feasible point at distance at most R 0 R_0 R 0 , assume ϵ 2 ≤ R / 16 \epsilon_2\le R/16 ϵ 2 ≤ R /16 . Projected subgradient descent first runs N G = max { 1 , ⌈ 64 R 0 2 / ( η μ R 2 ) ⌉ } N_G=\max\{1,\lceil64R_0^2/(\eta\mu R^2)\rceil\} N G = max { 1 , ⌈ 64 R 0 2 / ( η μ R 2 )⌉} updates and restarts from their average. If δ ≤ μ R 2 / 128 \delta\le\mu R^2/128 δ ≤ μ R 2 /128 , this average lies within distance R / 2 R/2 R /2 . The accelerated method first runs one block of N = max { 1 , ⌈ 16 R 0 / ( R η μ ) ⌉ } N=\max\{1,\lceil16R_0/(R\sqrt{\eta\mu})\rceil\} N = max { 1 , ⌈ 16 R 0 / ( R η μ )⌉} updates. If
N δ ≤ μ R 2 128 , (48) N\delta\le\frac{\mu R^2}{128}, \tag*{(48)} N δ ≤ 128 μ R 2 , (48)
its output lies within distance R / 2 R/2 R /2 . All three sequences in this initial block have distance at most { R 0 2 + 2 η δ N 3 } 1 / 2 \{R_0^2+2\eta\delta N^3\}^{1/2} { R 0 2 + 2 η δ N 3 } 1/2 from θ ^ \hat{\theta} θ ^ . Each method then continues under its local conditions above.
(iii) (a) Suppose Condition 3 holds on B ( θ ∗ , ϵ 3 ) ⊆ C \mathbb{B}(\theta^*,\epsilon_3)\subseteq C B ( θ ∗ , ϵ 3 ) ⊆ C and m H Π ⪯ H ⪯ M H Π m_H\boldsymbol{\Pi}\preceq H\preceq M_H\boldsymbol{\Pi} m H Π ⪯ H ⪯ M H Π . Under the initialization and score conditions of Theorem 1(iii), projected subgradient descent with 0 < η H ≤ M H − 1 0<\eta_H\le M_H^{-1} 0 < η H ≤ M H − 1 has inactive projections and satisfies (5).
For Nesterov acceleration, let θ ^ \hat{\theta} θ ^ be a constrained minimizer and put w = H − 1 ∂ L ( θ ∗ ) \mathbf{w}=H^{-1}\partial L(\theta^*) w = H − 1 ∂ L ( θ ∗ ) , b ∗ = ∣ θ ^ − θ ∗ + w ∣ 2 b_*=\lvert\hat{\theta}-\theta^*+\mathbf{w}\rvert_2 b ∗ = ∣ θ ^ − θ ∗ + w ∣ 2 , and γ H = ( 1 − η H m H ) / ( 1 + η H m H ) \gamma_H=(1-\sqrt{\eta_Hm_H})/(1+\sqrt{\eta_Hm_H}) γ H = ( 1 − η H m H ) / ( 1 + η H m H ) , where 0 < η H ≤ M H − 1 0<\eta_H\le M_H^{-1} 0 < η H ≤ M H − 1 . Choose ρ > 0 \rho>0 ρ > 0 with
( 1 + 2 γ H ) ( ρ + b ∗ + r n / m H ) + ∣ w ∣ 2 ≤ ϵ 3 . (49) (1+2\gamma_H)(\rho+b_*+r_n/m_H)+\lvert\mathbf{w}\rvert_2\le\epsilon_3. \tag*{(49)} ( 1 + 2 γ H ) ( ρ + b ∗ + r n / m H ) + ∣ w ∣ 2 ≤ ϵ 3 . (49)
From any feasible point within distance ρ \rho ρ of θ ^ \hat{\theta} θ ^ , restart Algorithm 3 with η H , γ H \eta_H,\gamma_H η H , γ H . Its iterates and extrapolated points remain in B ( θ ∗ , ϵ 3 ) B(\theta^*,\epsilon_3) B ( θ ∗ , ϵ 3 ) and satisfy (7) . Projection onto C C C leaves these updates unchanged. The block outputs in part (ii)(b) or (ii)(c) meet this distance requirement when 4 E / 3 ≤ ρ / 2 \sqrt{4E/3}\leq\rho/2 4 E /3 ≤ ρ /2 and 2 − j D 0 ≤ ρ / 2 2^{-j}D_0\leq\rho/2 2 − j D 0 ≤ ρ /2 .
(b) Suppose the assumptions of Theorem 3 or Proposition 6 hold with a n → 0 a_n\to0 a n → 0 , and take C = Θ C=\Theta C = Θ closed and bounded. Let θ ∗ \theta^* θ ∗ be an interior population minimizer with a Lipschitz population Hessian on a neighborhood and m H I ⪯ H = ∇ 2 L ( θ ∗ ) ⪯ M H I m_H I\preceq H=\nabla^2\mathcal{L}(\theta^*)\preceq M_H I m H I ⪯ H = ∇ 2 L ( θ ∗ ) ⪯ M H I . Choose R > 0 R>0 R > 0 sufficiently small that B ( θ ∗ , 2 R ) ⊆ Θ B(\theta^*,2R)\subseteq\Theta B ( θ ∗ , 2 R ) ⊆ Θ and the population Hessian is at least m H I / 2 m_H I/2 m H I /2 there. Use L = L 0 L=L_0 L = L 0 , μ = m H / 2 \mu=m_H/2 μ = m H /2 , and ϵ j = C j a n \epsilon_j=C_j a_n ϵ j = C j a n from Corollary 2, increasing L L L if needed so that μ ≤ 2 L \mu\leq2L μ ≤ 2 L . Take 0 < η ≤ 1 / ( 2 L ) 0<\eta\leq1/(2L) 0 < η ≤ 1/ ( 2 L ) and R 0 ≥ diam ( Θ ) R_0\geq\operatorname{diam}(\Theta) R 0 ≥ diam ( Θ ) . Use either entry and continuation procedure in part (ii)(c).
For sufficiently large n n n , there are c , C , γ > 0 c,C,\gamma>0 c , C , γ > 0 and one event of probability at least 1 − C n − γ p 1-Cn^{-\gamma p} 1 − C n − γ p on which an SAA solution lies in the interior of Θ \Theta Θ and the following bounds hold for every initial decision in Θ \Theta Θ and every selection of scenario subgradients. Every decision after ⌈ c log n ⌉ \lceil c\log n\rceil ⌈ c log n ⌉ updates has distance at most C a n Ca_n C a n from the SAA solution, including all three accelerated sequences. The population optimality gaps of the SAA solution and these decisions are at most C a n Ca_n C a n . The expansion and distributional conclusions of Theorem 4 consequently hold under their dimension and variance conditions. Switching to either interior Phase 3 in part (iii)(a) preserves these conclusions after another O ( log n ) O(\log n) O ( log n ) updates, with the optimization error given by (42) .
Proof. Write g = ∂ L g=\partial\mathcal{L} g = ∂ L . For z + = Π C ( z − α g ( y ) ) z^+=\Pi_C(z-\alpha g(y)) z + = Π C ( z − α g ( y )) , projection gives, for every v ∈ C v\in C v ∈ C ,
⟨ g ( y ) , z + − v ⟩ ≤ ∥ z − v ∥ 2 − ∥ z + − v ∥ 2 − ∥ z + − z ∥ 2 2 α . \langle g(y),z^+-v\rangle\leq\frac{\lVert z-v\rVert^2-\lVert z^+-v\rVert^2-\lVert z^+-z\rVert^2}{2\alpha}. ⟨ g ( y ) , z + − v ⟩ ≤ 2 α ∥ z − v ∥ 2 − ∥ z + − v ∥ 2 − ∥ z + − z ∥ 2 .
The segment bound in the proof of Lemma 1 and η L ≤ 1 / 2 \eta L\leq1/2 η L ≤ 1/2 give L max { k , ϵ 1 } k − k 2 / ( 2 η ) ≤ δ I { k < ϵ 1 } L\max\{k,\epsilon_1\}k-k^2/(2\eta)\leq\delta\mathbb{I}\{k<\epsilon_1\} L max { k , ϵ 1 } k − k 2 / ( 2 η ) ≤ δ I { k < ϵ 1 } for every feasible displacement of length k k k . For k < ϵ 1 k<\epsilon_1 k < ϵ 1 , this follows by completing the square, and for k ≥ ϵ 1 k\geq\epsilon_1 k ≥ ϵ 1 the left side is nonpositive.
Part (i). For a projected subgradient update, use y = z = x y=z=x y = z = x , α = η \alpha=\eta α = η , and v = θ ^ v=\hat{\theta} v = θ ^ in the projection inequality. Convexity and the remainder bound give
Δ ( x + ) ≤ ∥ x − θ ^ ∥ 2 − ∥ x + − θ ^ ∥ 2 2 η + δ . \Delta(x^+)\leq\frac{\lVert x-\hat{\theta}\rVert^2-\lVert x^+-\hat{\theta}\rVert^2}{2\eta}+\delta. Δ ( x + ) ≤ 2 η ∥ x − θ ^ ∥ 2 − ∥ x + − θ ^ ∥ 2 + δ .
Summation proves part (i)(a). If each update length is at least ϵ 1 \epsilon_1 ϵ 1 , the same calculation has zero error. The comparison v = x v=x v = x gives monotonicity and hence the last-iterate bound.
For acceleration, feasibility follows from (43) . Set g t = g ( y t ) g_t=g(y_t) g t = g ( y t ) , k t = ∥ x t + 1 − y t ∥ 2 k_t=\lVert x_{t+1}-y_t\rVert^2 k t = ∥ x t + 1 − y t ∥ 2 , and λ − 1 = 0 \lambda_{-1}=0 λ − 1 = 0 . Weight the convexity inequalities at x t x_t x t and θ ^ \hat{\theta} θ ^ by λ t ( λ t − 1 ) \lambda_t(\lambda_t-1) λ t ( λ t − 1 ) and λ t \lambda_t λ t , respectively. Applying the projection inequality with α = η λ t \alpha=\eta\lambda_t α = η λ t , z = z t z=z_t z = z t , and v = θ ^ v=\hat{\theta} v = θ ^ , and using ∥ z t + 1 − z t ∥ 2 = λ t k t \lVert z_{t+1}-z_t\rVert^2=\lambda_t k_t ∥ z t + 1 − z t ∥ 2 = λ t k t and λ t ( λ t − 1 ) = λ t − 1 2 \lambda_t(\lambda_t-1)=\lambda_{t-1}^2 λ t ( λ t − 1 ) = λ t − 1 2 , gives
λ t 2 Δ ( x t + 1 ) + ∥ z t + 1 − θ ^ ∥ 2 2 η ≤ λ t − 1 2 Δ ( x t ) + ∥ z t − θ ^ ∥ 2 2 η + λ t 2 δ t . (50) \lambda_t^2\Delta(x_{t+1})+\frac{\lVert z_{t+1}-\hat{\theta}\rVert^2}{2\eta}\leq\lambda_{t-1}^2\Delta(x_t)+\frac{\lVert z_t-\hat{\theta}\rVert^2}{2\eta}+\lambda_t^2\delta_t. \tag*{(50)} λ t 2 Δ ( x t + 1 ) + 2 η ∥ z t + 1 − θ ^ ∥ 2 ≤ λ t − 1 2 Δ ( x t ) + 2 η ∥ z t − θ ^ ∥ 2 + λ t 2 δ t . (50)
Summation and λ T − 1 ≥ T / 2 \lambda_{T-1}\geq T/2 λ T − 1 ≥ T /2 prove (44) . Since Δ ≥ 0 \Delta\geq0 Δ ≥ 0 , each auxiliary squared distance is at most D 0 2 + 2 η ∑ s < t λ s 2 δ s D_0^2+2\eta\sum_{s<t}\lambda_s^2\delta_s D 0 2 + 2 η ∑ s < t λ s 2 δ s . The bounds λ s ≤ s + 1 \lambda_s\leq s+1 λ s ≤ s + 1 and ∑ s < T ( s + 1 ) 2 ≤ T 3 \sum_{s<T}(s+1)^2\leq T^3 ∑ s < T ( s + 1 ) 2 ≤ T 3 prove the asserted radius for z t z_t z t . Each x t x_t x t is a convex combination of preceding x x x and z z z points, as is y t y_t y t . This proves their radii as well. If all k t ≤ ϵ 1 k_t\leq\epsilon_1 k t ≤ ϵ 1 , each δ t = 0 \delta_t=0 δ t = 0 and all distances are at most D 0 D_0 D 0 .
Part (ii). The proof of (22) applies to a constrained minimizer. Indeed, for x s = θ ^ + s ( x − θ ^ ) x_s=\hat{\theta}+s(x-\hat{\theta}) x s = θ ^ + s ( x − θ ^ ) , convexity and constrained minimality give ⟨ g ( x s ) , x − x s ⟩ ≥ 0 \langle g(x_s),x-x_s\rangle\ge0 ⟨ g ( x s ) , x − x s ⟩ ≥ 0 . If d = ∣ x − θ ^ ∣ 2 > ϵ 2 d=|x-\hat{\theta}|_2>\epsilon_2 d = ∣ x − θ ^ ∣ 2 > ϵ 2 , Condition 2 at ( x , x s ) (x,x_s) ( x , x s ) and s ↓ 0 s\downarrow0 s ↓ 0 give Δ ( x ) ≥ μ d 2 / 4 \Delta(x)\ge\mu d^2/4 Δ ( x ) ≥ μ d 2 /4 . Together with minimality for d ≤ ϵ 2 d\le\epsilon_2 d ≤ ϵ 2 , this gives
d 2 ≤ 4 Δ ( x ) μ + ϵ 2 2 . (51) d^2\le\frac{4\Delta(x)}{\mu}+\epsilon_2^2. \tag*{(51)} d 2 ≤ μ 4Δ ( x ) + ϵ 2 2 . (51)
For projected subgradient descent, retain the lower-model term (21) at ( θ ^ , x ) (\hat{\theta},x) ( θ ^ , x ) in the calculation for part (i)(a). Since Δ ( x + ) ≥ 0 \Delta(x^+)\ge0 Δ ( x + ) ≥ 0 and α = η μ / 2 \alpha=\eta\mu/2 α = η μ /2 , this gives
∣ x + − θ ^ ∣ 2 2 ≤ ( 1 − α ) ∣ x − θ ^ ∣ 2 2 + α ϵ 2 2 + 2 η δ . |x^+-\hat{\theta}|_2^2\le(1-\alpha)|x-\hat{\theta}|_2^2+\alpha\epsilon_2^2+2\eta\delta. ∣ x + − θ ^ ∣ 2 2 ≤ ( 1 − α ) ∣ x − θ ^ ∣ 2 2 + α ϵ 2 2 + 2 η δ .
For the objective bound, the projected update minimizes Q x ( v ) = ⟨ g ( x ) , v − x ⟩ + ∣ v − x ∣ 2 2 / ( 2 η ) Q_x(v)=\langle g(x),v-x\rangle+|v-x|_2^2/(2\eta) Q x ( v ) = ⟨ g ( x ) , v − x ⟩ + ∣ v − x ∣ 2 2 / ( 2 η ) over C C C . Compare its value with v = ( 1 − α ) x + α θ ^ v=(1-\alpha)x+\alpha\hat{\theta} v = ( 1 − α ) x + α θ ^ . Since α = η μ / 2 \alpha=\eta\mu/2 α = η μ /2 , the squared-distance terms cancel and
Δ ( x + ) ≤ ( 1 − α ) Δ ( x ) + α μ ϵ 2 2 4 + δ . \Delta(x^+)\le(1-\alpha)\Delta(x)+\frac{\alpha\mu\epsilon_2^2}{4}+\delta. Δ ( x + ) ≤ ( 1 − α ) Δ ( x ) + 4 α μ ϵ 2 2 + δ .
Iteration proves part (ii)(a). The stated distance cutoffs remove the two error terms. Both one-step inequalities use strong convexity only at the pair ( θ ^ , x ) (\hat{\theta},x) ( θ ^ , x ) . If ∣ x − θ ^ ∣ 2 2 ≤ R / 2 |x-\hat{\theta}|_2^2\le R/2 ∣ x − θ ^ ∣ 2 2 ≤ R /2 and F G ≤ R 2 / 4 F_G\le R^2/4 F G ≤ R 2 /4 , the distance recursion therefore proves local containment before invoking strong convexity at the next point. This proves the projected subgradient conclusion in part (ii)(c).
For an accelerated block of length K K K , part (i)(b) and (51) give
D j + 1 2 ≤ 8 η μ K 2 D j 2 + 4 K δ μ + ϵ 2 2 ≤ 1 4 D j 2 + E . D_{j+1}^2\le\frac{8}{\eta\mu K^2}D_j^2+\frac{4K\delta}{\mu}+\epsilon_2^2\le\frac{1}{4}D_j^2+E. D j + 1 2 ≤ η μ K 2 8 D j 2 + μ 4 K δ + ϵ 2 2 ≤ 4 1 D j 2 + E .
Iteration proves (45) . Since 2 / ( η K 2 ) ≤ μ / 16 2/(\eta K^2)\le\mu/16 2/ ( η K 2 ) ≤ μ /16 , for j ≥ 1 j\ge1 j ≥ 1 ,
Δ ( θ ˉ j ) ≤ μ 16 D j − 1 2 + K δ ≤ μ 4 4 − j D 0 2 + μ 12 E + K δ ≤ μ 4 4 − j D 0 2 + μ 3 E . \Delta(\bar{\theta}_j)\le\frac{\mu}{16}D_{j-1}^2+K\delta\le\frac{\mu}{4}4^{-j}D_0^2+\frac{\mu}{12}E+K\delta\le\frac{\mu}{4}4^{-j}D_0^2+\frac{\mu}{3}E. Δ ( θ ˉ j ) ≤ 16 μ D j − 1 2 + K δ ≤ 4 μ 4 − j D 0 2 + 12 μ E + K δ ≤ 4 μ 4 − j D 0 2 + 3 μ E .
Since η μ K 2 ≤ ( 32 + 1 ) 2 < 49 \eta\mu K^2\le(\sqrt{32}+1)^2<49 η μ K 2 ≤ ( 32 + 1 ) 2 < 49 , we have 2 η δ K 3 ≤ 49 E / 2 2\eta\delta K^3\le49E/2 2 η δ K 3 ≤ 49 E /2 . The bound on squared distances within each block in part (i)(b) is therefore D j 2 + 2 η δ K 3 ≤ 4 − j D 0 2 + 26 E D_j^2+2\eta\delta K^3\le4^{-j}D_0^2+26E D j 2 + 2 η δ K 3 ≤ 4 − j D 0 2 + 26 E , proving part (ii)(b).
For local containment, put H = ϵ 2 2 + 2 K δ / μ ≤ R 2 / 256 H=\epsilon_2^2+2K\delta/\mu\le R^2/256 H = ϵ 2 2 + 2 K δ / μ ≤ R 2 /256 . A block starting within distance R / 2 R/2 R /2 has every inner squared distance at most R 2 / 4 + 49 H ≤ 113 R 2 / 256 < R 2 R^2/4+49H\le113R^2/256<R^2 R 2 /4 + 49 H ≤ 113 R 2 /256 < R 2 by part (i)(b), which uses only approximate smoothness on C C C . Applying (51) to its output then gives D j + 1 2 ≤ R 2 / 16 + E ≤ R 2 / 16 + 2 H ≤ 9 R 2 / 128 < R 2 / 4 D_{j+1}^2\le R^2/16+E\le R^2/16+2H\le9R^2/128<R^2/4 D j + 1 2 ≤ R 2 /16 + E ≤ R 2 /16 + 2 H ≤ 9 R 2 /128 < R 2 /4 . Induction proves the accelerated conclusions in part (ii)(c).
For entry, (51) and ϵ 2 ≤ R / 16 \epsilon_2\le R/16 ϵ 2 ≤ R /16 give Δ ( x ) > μ R 2 / 8 \Delta(x)>\mu R^2/8 Δ ( x ) > μ R 2 /8 on the feasible radius-R R R sphere. Convexity along segments to θ ^ \hat{\theta} θ ^ gives the same lower bound at every feasible point outside that sphere. A feasible point of gap at most μ R 2 / 64 \mu R^2/64 μ R 2 /64 must therefore lie inside it, where (51) gives squared distance at most R 2 / 16 + ϵ 2 2 < R 2 / 4 R^2/16+\epsilon_2^2<R^2/4 R 2 /16 + ϵ 2 2 < R 2 /4 . For projected subgradient descent, its feasible average satisfies
Δ ( 1 N G ∑ t = 1 N G x t ) ≤ R 0 2 2 η N G + δ ≤ μ R 2 64 . \Delta\left(\frac{1}{N_G}\sum_{t=1}^{N_G}x_t\right)\le\frac{R_0^2}{2\eta N_G}+\delta\le\frac{\mu R^2}{64}. Δ ( N G 1 t = 1 ∑ N G x t ) ≤ 2 η N G R 0 2 + δ ≤ 64 μ R 2 .
For acceleration, (44) gives Δ ( x N ) ≤ 2 R 0 2 / ( η N 2 ) + N δ ≤ μ R 2 / 64 \Delta(x_N)\le2R_0^2/(\eta N^2)+N\delta\le\mu R^2/64 Δ ( x N ) ≤ 2 R 0 2 / ( η N 2 ) + N δ ≤ μ R 2 /64 . The initial accelerated radius is the bound in part (i)(b). This proves both entry procedures.
Part (iii). For projected subgradient descent in part (iii)(a), Theorem 1(iii) places each unprojected proposal in B ( θ ∗ , ϵ 3 / 2 ) ⊆ C \mathbb{B}(\theta^*,\epsilon_3/2)\subseteq C B ( θ ∗ , ϵ 3 /2 ) ⊆ C . Each projection is therefore inactive, proving (5) . For the accelerated switch from x x x , the bound ∣ x − θ ^ ∣ 2 ≤ ρ |x-\hat{\theta}|_2\le\rho ∣ x − θ ^ ∣ 2 ≤ ρ gives ∣ x − θ ∗ + w 2 ∣ ≤ ρ + b ∗ |x-\theta^*+w_2|\le\rho+b_* ∣ x − θ ∗ + w 2 ∣ ≤ ρ + b ∗ . The block bound (45) implies ∣ θ ˉ j − θ ^ ∣ 2 ≤ ρ |\bar{\theta}_j-\hat{\theta}|_2\le\rho ∣ θ ˉ j − θ ^ ∣ 2 ≤ ρ under the two stated block conditions. Condition (49) verifies the initialization in Theorem 2(iii), which proves the expansion and containment of all iterates and extrapolated points. Projection onto C C C leaves each proposed update unchanged.
For part (iii)(b), work on the joint increment and pointwise event in Theorem 3 or Proposition 6. The population first-order condition and pointwise score bound give s n = ∣ g ( θ ∗ ) ∣ 2 ≤ C a n s_n = \lvert g(\theta^*)\rvert_2 \le C\sqrt{a_n} s n = ∣ g ( θ ∗ ) ∣ 2 ≤ C a n . Corollary 2 gives approximate smoothness on Θ \Theta Θ and approximate strong convexity on B ( θ ∗ , 2 R ) \mathbb{B}(\theta^*,2R) B ( θ ∗ , 2 R ) . The model objective is continuous. Hence compactness of Θ \Theta Θ gives a minimizer. On the radius-R / 2 R/2 R /2 sphere about θ ∗ \theta^* θ ∗ ,
L ( θ ∗ + R u / 2 ) − L ( θ ∗ ) ≥ − s n R / 2 + μ R 2 / 16 ≥ μ R 2 / 32 \mathcal{L}(\theta^*+Ru/2)-\mathcal{L}(\theta^*)\ge-s_nR/2+\mu R^2/16\ge\mu R^2/32 L ( θ ∗ + R u /2 ) − L ( θ ∗ ) ≥ − s n R /2 + μ R 2 /16 ≥ μ R 2 /32
for every unit u u u and sufficiently large n n n . Convexity along feasible segments excludes every point outside this sphere from the minimizer set. Each SAA minimizer therefore lies in the interior of Θ \Theta Θ and satisfies
δ n : = ∣ θ ^ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 4 s n / μ } = O ( a n ) . \delta_n:=\lvert\widehat{\theta}-\theta^*\rvert_2\le\max\{\epsilon_2,4s_n/\mu\}=O(\sqrt{a_n}). δ n := ∣ θ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 4 s n / μ } = O ( a n ) .
Eventually δ n ≤ R / 8 \delta_n\le R/8 δ n ≤ R /8 . Hence B ( θ ^ , R ) ⊆ B ( θ ∗ , 2 R ) ⊆ Θ \mathbb{B}(\widehat{\theta},R)\subseteq\mathbb{B}(\theta^*,2R)\subseteq\Theta B ( θ , R ) ⊆ B ( θ ∗ , 2 R ) ⊆ Θ and (49) holds throughout B ( θ ^ , R ) \mathbb{B}(\widehat{\theta},R) B ( θ , R ) . Since R 0 ≥ diam ( Θ ) R_0\ge\operatorname{diam}(\Theta) R 0 ≥ diam ( Θ ) , every initialization has distance at most R 0 R_0 R 0 . The entry lengths N G N_G N G , N N N and restart length K K K have constant order, while δ = O ( a n 2 ) \delta=O(a_n^2) δ = O ( a n 2 ) and ϵ 2 = O ( a n ) \epsilon_2=O(a_n) ϵ 2 = O ( a n ) . All local and entry accuracy conditions in part (ii)(c) hold eventually. That part therefore proves entry and containment for both methods. Their distance floors are F G = O ( a n 2 ) F_G=O(a_n^2) F G = O ( a n 2 ) and E = O ( a n 2 ) E=O(a_n^2) E = O ( a n 2 ) . The geometric bounds give distance O ( a n ) O(a_n) O ( a n ) after O ( log n ) O(\log n) O ( log n ) updates for all later projected subgradient iterates and all points in the accelerated blocks. The constants are uniform over initial decisions, and the event controls all scenario-subgradient choices.
Population smoothness and stationarity bound the population gap by C ∣ x − θ ∗ ∣ 2 2 C\lvert x-\theta^*\rvert_2^2 C ∣ x − θ ∗ ∣ 2 2 . The localization and optimization bounds therefore give the asserted gaps on the same event. At θ ∗ \theta^* θ ∗ , the scenario subgradient is unique almost surely, and g ( θ ∗ ) = n − 1 ∑ i ψ i g(\theta^*)=n^{-1}\sum_i\psi_i g ( θ ∗ ) = n − 1 ∑ i ψ i has mean zero and second moment at most C p / n Cp/n C p / n . Hence δ n = O P ( p / n + a n ) \delta_n=O_P(\sqrt{p/n}+a_n) δ n = O P ( p / n + a n ) . Interiority, Lemma 2, the increment bound, and the population Taylor expansion give
∣ H ( θ ^ − θ ∗ ) + g ( θ ∗ ) ∣ 2 ≤ L ϵ 1 + C { δ n a n + a n + δ n 2 } = O P { ( p / n ) 3 / 4 log n + a n } . \left\lvert H(\widehat{\theta}-\theta^*)+g(\theta^*)\right\rvert_2\le L\epsilon_1+C\left\{\sqrt{\delta_na_n}+a_n+\delta_n^2\right\}=O_P\left\{(p/n)^{3/4}\sqrt{\log n}+a_n\right\}. H ( θ − θ ∗ ) + g ( θ ∗ ) 2 ≤ L ϵ 1 + C { δ n a n + a n + δ n 2 } = O P { ( p / n ) 3/4 log n + a n } .
Adding the O ( a n ) O(a_n) O ( a n ) optimization error proves the computed-solution expansion. The first-order equivalence conclusions follow from a n = o ( p / n ) a_n=o(\sqrt{p/n}) a n = o ( p / n ) when p = o { n ( log n ) − 2 } p=o\{n(\log n)^{-2}\} p = o { n ( log n ) − 2 } and n a n → 0 \sqrt{n}a_n\to0 n a n → 0 when p log n = o ( n ) p\log n=o(\sqrt{n}) p log n = o ( n ) . Every deterministic unit projection of H − 1 ψ i H^{-1}\psi_i H − 1 ψ i has bounded third absolute moment. Thus the variance lower bound gives Lyapunov’s condition, and p = o { n 1 / 3 ( log n ) − 2 / 3 } p=o\{n^{1/3}(\log n)^{-2/3}\} p = o { n 1/3 ( log n ) − 2/3 } makes the expansion remainder o P ( n − 1 / 2 ) o_P(n^{-1/2}) o P ( n − 1/2 ) . Slutsky’s theorem proves the normal limit. Finally, population smoothness gives
∣ L ‾ ( θ ~ ) − L ‾ ( θ ^ ) ∣ ≤ C { δ n ∣ θ ~ − θ ^ ∣ 2 + ∣ θ ~ − θ ^ ∣ 2 2 } = O P { a n p / n + a n 2 } . \left\lvert\overline{\mathcal{L}}(\widetilde{\theta})-\overline{\mathcal{L}}(\widehat{\theta})\right\rvert\le C\left\{\delta_n\lvert\widetilde{\theta}-\widehat{\theta}\rvert_2+\lvert\widetilde{\theta}-\widehat{\theta}\rvert_2^2\right\}=O_P\{a_n\sqrt{p/n}+a_n^2\}. L ( θ ) − L ( θ ) ≤ C { δ n ∣ θ − θ ∣ 2 + ∣ θ − θ ∣ 2 2 } = O P { a n p / n + a n 2 } .
Multiplication by n n n gives O P ( p 3 / 2 log n / n ) = o P ( 1 ) O_P(p^{3/2}\log n/\sqrt{n})=o_P(1) O P ( p 3/2 log n / n ) = o P ( 1 ) under the same dimension condition.
For the interior switch, choose ϵ 3 = C 3 a n \epsilon_3=C_3\sqrt{a_n} ϵ 3 = C 3 a n with sufficiently large C 3 C_3 C 3 . The local approximation has remainder r n = C 4 a n 3 / 4 r_n=C_4a_n^{3/4} r n = C 4 a n 3/4 . The score and localization bounds verify the projected subgradient initialization in part (iii)(a) after reaching optimization error O ( a n ) O(a_n) O ( a n ) . For acceleration, w = H − 1 g ( θ ∗ ) w=H^{-1}g(\theta^*) w = H − 1 g ( θ ∗ ) satisfies ∣ w ∣ 2 ≤ C a n \lvert w\rvert_2\le C\sqrt{a_n} ∣ w ∣ 2 ≤ C a n , and approximation at the interior minimizer gives b ∗ ≤ ( L ϵ 1 + r n ) / m H = o ( ϵ 3 ) b_*\le(L\epsilon_1+r_n)/m_H=o(\epsilon_3) b ∗ ≤ ( L ϵ 1 + r n ) / m H = o ( ϵ 3 ) . Choose ρ = c 3 ϵ 3 \rho=c_3\epsilon_3 ρ = c 3 ϵ 3 for small c 3 > 0 c_3>0 c 3 > 0 and increase C 3 C_3 C 3 to verify (49) . Theorems 1(iii) and 2(iii) make subsequent projections inactive. Finally, the increment bound applied to Θ \Theta Θ and Θ ^ \widehat{\Theta} Θ verifies (41) with ζ n = O ( a n ) \zeta_n=O(a_n) ζ n = O ( a n ) , as in Appendix A.3. Its local comparison (42) gives optimization error O ( a n ) O(a_n) O ( a n ) after another O ( log n ) O(\log n) O ( log n ) updates. This completes part (iii). □
Corollary 11. Let L \mathcal{L} L be finite and convex on R p \mathbb{R}^{p} R p , let θ ^ ∈ argmin θ ∈ R p L ( θ ) \hat{\theta} \in\operatorname*{argmin}_{\theta\in\mathbb{R}^{p}} \mathcal{L}(\theta) θ ^ ∈ argmin θ ∈ R p L ( θ ) , and consider the heavy-ball updates
θ ( t + 1 ) = θ ( t ) − η ∂ L ( θ ( t ) ) + β ( θ ( t ) − θ ( t − 1 ) ) , θ ( − 1 ) = θ ( 0 ) . (52) \theta^{(t+1)}=\theta^{(t)}-\eta\partial\mathcal{L}(\theta^{(t)})+\beta\left(\theta^{(t)}-\theta^{(t-1)}\right),\qquad\theta^{(-1)}=\theta^{(0)}. \tag*{(52)} θ ( t + 1 ) = θ ( t ) − η ∂ L ( θ ( t ) ) + β ( θ ( t ) − θ ( t − 1 ) ) , θ ( − 1 ) = θ ( 0 ) . (52) (i) Suppose Conditions 1 and 2 hold on R p \mathbb{R}^{p} R p . Take β = 1 / 2 \beta=1/2 β = 1/2 and 0 < η ≤ 1 / ( 4 L ) 0<\eta\leq1/(4L) 0 < η ≤ 1/ ( 4 L ) . Write
E t = L ( θ ( t ) ) − L ( θ ^ ) + ∣ θ ( t ) − θ ( t − 1 ) ∣ 2 2 4 η , q C = μ η 2 ( 1 + μ η ) , B C = μ ϵ 2 2 4 + 2 L 2 ( 1 + μ η ) μ ϵ 1 2 . E_t=\mathcal{L}(\theta^{(t)})-\mathcal{L}(\hat{\theta})+\frac{\left\lvert\theta^{(t)}-\theta^{(t-1)}\right\rvert_2^2}{4\eta},\qquad q_C=\frac{\mu\eta}{2(1+\mu\eta)},\qquad B_C=\frac{\mu\epsilon_2^2}{4}+\frac{2L^2(1+\mu\eta)}{\mu}\epsilon_1^2. E t = L ( θ ( t ) ) − L ( θ ^ ) + 4 η θ ( t ) − θ ( t − 1 ) 2 2 , q C = 2 ( 1 + μ η ) μ η , B C = 4 μ ϵ 2 2 + μ 2 L 2 ( 1 + μ η ) ϵ 1 2 . For every T ≥ 0 T\geq0 T ≥ 0 ,
E T ≤ ( 1 − q C ) T E 0 + { 1 − ( 1 − q C ) T } B C , ∣ θ ( T ) − θ ^ ∣ 2 2 ≤ 4 μ E T + ϵ 2 2 . (53) E_T\leq(1-q_C)^T E_0+\left\{1-(1-q_C)^T\right\}B_C,\qquad\left\lvert\theta^{(T)}-\hat{\theta}\right\rvert_2^2\leq\frac{4}{\mu}E_T+\epsilon_2^2. \tag*{(53)} E T ≤ ( 1 − q C ) T E 0 + { 1 − ( 1 − q C ) T } B C , θ ( T ) − θ ^ 2 2 ≤ μ 4 E T + ϵ 2 2 . (53) (ii) Suppose Conditions 1 and 2 hold on B ( θ ^ , R ) B(\hat{\theta},R) B ( θ ^ , R ) for R > 0 R>0 R > 0 , and take θ ( 0 ) ∈ B ( θ ^ , R ) \theta^{(0)}\in B(\hat{\theta},R) θ ( 0 ) ∈ B ( θ ^ , R ) . Use the parameters and quantities in part (i). Set r = { 4 max { E 0 , B C } / μ + ϵ 2 2 } 1 / 2 r=\{4\max\{E_0,B_C\}/\mu+\epsilon_2^2\}^{1/2} r = { 4 max { E 0 , B C } / μ + ϵ 2 2 } 1/2 . If
2 r + η L { max { r , ϵ 1 } + ϵ 1 } ≤ R , (54) 2r+\eta L\{\max\{r,\epsilon_1\}+\epsilon_1\}\leq R, \tag*{(54)} 2 r + η L { max { r , ϵ 1 } + ϵ 1 } ≤ R , (54) then ∣ θ ( t ) − θ ^ ∣ 2 ≤ r \left\lvert\theta^{(t)}-\hat{\theta}\right\rvert_2\leq r θ ( t ) − θ ^ 2 ≤ r for every t ≥ 0 t\geq0 t ≥ 0 , and (51) holds.
(iii) Suppose Condition 3 holds with remainder r n r_n r n , and m H I ⪯ H ⪯ M H I m_H\mathbb{I}\preceq H\preceq M_H\mathbb{I} m H I ⪯ H ⪯ M H I for 0 < m H ≤ M H 0<m_H\leq M_H 0 < m H ≤ M H . Restart (50) with
η = 4 ( M H + m H ) 2 , q H = M H − m H M H + m H , β = q H 2 , A H = 1 + q H 1 − q H . \eta=\frac{4}{(\sqrt{M_H}+\sqrt{m_H})^2},\qquad q_H=\frac{\sqrt{M_H}-\sqrt{m_H}}{\sqrt{M_H}+\sqrt{m_H}},\qquad\beta=q_H^2,\qquad A_H=\frac{1+q_H}{1-q_H}. η = ( M H + m H ) 2 4 , q H = M H + m H M H − m H , β = q H 2 , A H = 1 − q H 1 + q H . Write g ∗ = ∂ L ( θ ∗ ) g^*=\partial\mathcal{L}(\theta^*) g ∗ = ∂ L ( θ ∗ ) and z 0 = θ ( 0 ) − θ ∗ + H − 1 g ∗ z_0=\theta^{(0)}-\theta^*+H^{-1}g^* z 0 = θ ( 0 ) − θ ∗ + H − 1 g ∗ . If
A H ∣ θ ( 0 ) − θ ∗ ∣ 2 2 + ∣ g ∗ ∣ 2 2 + r n m H ≤ ϵ 3 , (55) A_H\left\lvert\theta^{(0)}-\theta^*\right\rvert_2^2+\frac{\left\lvert g^*\right\rvert_2^2+r_n}{m_H}\leq\epsilon_3, \tag*{(55)} A H θ ( 0 ) − θ ∗ 2 2 + m H ∣ g ∗ ∣ 2 2 + r n ≤ ϵ 3 , (55) then ∣ θ ( t ) − θ ∗ ∣ 2 ≤ ϵ 3 \left\lvert\theta^{(t)}-\theta^*\right\rvert_2\leq\epsilon_3 θ ( t ) − θ ∗ 2 ≤ ϵ 3 for every t ≥ 0 t\geq0 t ≥ 0 . Define P 0 = I P_0=\mathbb{I} P 0 = I , P 1 = I − η H P_1=\mathbb{I}-\eta H P 1 = I − η H , and P t + 1 = { ( 1 + β ) I − η H } P t − β P t − 1 P_{t+1}=\{(1+\beta)\mathbb{I}-\eta H\}P_t-\beta P_{t-1} P t + 1 = {( 1 + β ) I − η H } P t − β P t − 1 . For every T ≥ 1 T\geq1 T ≥ 1 ,
θ ( T ) − θ ∗ = − H − 1 g ∗ + P T z 0 + R T , ∣ R T ∣ 2 ≤ r n / m H , \theta^{(T)}-\theta^*=-H^{-1}g^*+P_Tz_0+R_T,\qquad\left\lvert R_T\right\rvert_2\leq r_n/m_H, θ ( T ) − θ ∗ = − H − 1 g ∗ + P T z 0 + R T , ∣ R T ∣ 2 ≤ r n / m H , ∥ P T ∥ ≤ min { A H , { 1 + ( 1 + q H ) T } q H T , 2 A H exp { − ( 1 − q H ) T / 2 } } . (56) \left\lVert P_T\right\rVert\leq\min\left\{A_H,\{1+(1+q_H)T\}q_H^T,2A_H\exp\{-(1-q_H)T/2\}\right\}. \tag*{(56)} ∥ P T ∥ ≤ min { A H , { 1 + ( 1 + q H ) T } q H T , 2 A H exp { − ( 1 − q H ) T /2 } } . (56)
Proof. Write v t = θ ( t ) − θ ( t − 1 ) v_t=\theta^{(t)}-\theta^{(t-1)} v t = θ ( t ) − θ ( t − 1 ) , g t = ∂ L ( θ ( t ) ) g_t=\partial\mathcal{L}(\theta^{(t)}) g t = ∂ L ( θ ( t ) ) , and Δ t = L ( θ ( t ) ) − L ( θ ^ ) \Delta_t=\mathcal{L}(\theta^{(t)})-\mathcal{L}(\hat{\theta}) Δ t = L ( θ ( t ) ) − L ( θ ^ ) . The segment bound in the proof of Lemma 1 and Young’s inequality bound the linearization remainder for an update of length k k k by L max { k , ϵ 1 } k ≤ k 2 / ( 4 η ) + η L 2 ϵ 1 2 L\max\{k,\epsilon_1\}k\leq k^2/(4\eta)+\eta L^2\epsilon_1^2 L max { k , ϵ 1 } k ≤ k 2 / ( 4 η ) + η L 2 ϵ 1 2 when η ≤ 1 / ( 4 L ) \eta\leq1/(4L) η ≤ 1/ ( 4 L ) . With β = 1 / 2 \beta=1/2 β = 1/2 , the update gives η g t = v t / 2 − v t + 1 \eta g_t=v_t/2-v_{t+1} η g t = v t /2 − v t + 1 . Adding the difference of the kinetic terms to the upper bound for Δ t + 1 − Δ t \Delta_{t+1}-\Delta_t Δ t + 1 − Δ t yields
E t + 1 ≤ E t − ∣ v t + 1 ∣ 2 2 + ∣ v t − v t + 1 ∣ 2 2 4 η + η L 2 ϵ 1 2 . (57) E_{t+1}\leq E_t-\frac{\left\lvert v_{t+1}\right\rvert_2^2+\left\lvert v_t-v_{t+1}\right\rvert_2^2}{4\eta}+\eta L^2\epsilon_1^2. \tag*{(57)} E t + 1 ≤ E t − 4 η ∣ v t + 1 ∣ 2 2 + ∣ v t − v t + 1 ∣ 2 2 + η L 2 ϵ 1 2 . (57)
The lower bound (21) with comparison point θ ^ \hat{\theta} θ ^ and a square completion give Δ t ≤ ∣ g t ∣ 2 2 / μ + μ ϵ 2 2 / 4 \Delta_t\leq\left\lvert g_t\right\rvert_2^2/\mu+\mu\epsilon_2^2/4 Δ t ≤ ∣ g t ∣ 2 2 / μ + μ ϵ 2 2 /4 . Since 2 η g t = ( v t − v t + 1 ) − v t + 1 2\eta g_t=(v_t-v_{t+1})-v_{t+1} 2 η g t = ( v t − v t + 1 ) − v t + 1 ,
E t ≤ 1 + μ η 2 μ η 2 { ∣ v t + 1 ∣ 2 2 + ∣ v t − v t + 1 ∣ 2 2 } + μ ϵ 2 2 4 . E_t\leq\frac{1+\mu\eta}{2\mu\eta^2}\{\left\lvert v_{t+1}\right\rvert_2^2+\left\lvert v_t-v_{t+1}\right\rvert_2^2\}+\frac{\mu\epsilon_2^2}{4}. E t ≤ 2 μ η 2 1 + μ η { ∣ v t + 1 ∣ 2 2 + ∣ v t − v t + 1 ∣ 2 2 } + 4 μ ϵ 2 2 .
Substitution into (55) gives E t + 1 ≤ ( 1 − q C ) E t + q C B C E_{t+1} \le(1-q_C)E_t+q_CB_C E t + 1 ≤ ( 1 − q C ) E t + q C B C . Iteration proves the first bound in (51). Applying (22) and using Δ T ≤ E T \Delta_T \le E_T Δ T ≤ E T proves its distance bound.
For part (ii), write B = max { E 0 , B C } B=\max\{E_0,B_C\} B = max { E 0 , B C } . The initial point belongs to the radius-R R R ball, and (22) gives ∣ θ ( 0 ) − θ ^ ∣ 2 ≤ r \lvert\theta^{(0)}-\hat{\theta}\rvert_2\le r ∣ θ ( 0 ) − θ ^ ∣ 2 ≤ r . The restart gives this bound at t = − 1 t=-1 t = − 1 . Suppose the bounds E t ≤ B E_t\le B E t ≤ B and ∣ θ ( j ) − θ ^ ∣ 2 ≤ r \lvert\theta^{(j)}-\hat{\theta}\rvert_2\le r ∣ θ ( j ) − θ ^ ∣ 2 ≤ r hold through index t t t . Lemma 2 bounds ∣ ∂ L ( θ ^ ) ∣ 2 \lvert\partial\mathcal{L}(\hat{\theta})\rvert_2 ∣ ∂ L ( θ ^ ) ∣ 2 by L ϵ 1 L\epsilon_1 L ϵ 1 . Hence Condition 1 gives ∣ g t ∣ 2 ≤ L { max { r , ϵ 1 } + ϵ 1 } \lvert g_t\rvert_2\le L\{\max\{r,\epsilon_1\}+\epsilon_1\} ∣ g t ∣ 2 ≤ L { max { r , ϵ 1 } + ϵ 1 } . The proposed update satisfies
∣ θ ( t + 1 ) − θ ^ ∣ 2 ≤ 3 2 ∣ θ ( t ) − θ ^ ∣ 2 + 1 2 ∣ θ ( t − 1 ) − θ ^ ∣ 2 + η ∣ g t ∣ 2 ≤ R . \lvert\theta^{(t+1)}-\hat{\theta}\rvert_2\le\frac{3}{2}\lvert\theta^{(t)}-\hat{\theta}\rvert_2+\frac{1}{2}\lvert\theta^{(t-1)}-\hat{\theta}\rvert_2+\eta\lvert g_t\rvert_2\le R. ∣ θ ( t + 1 ) − θ ^ ∣ 2 ≤ 2 3 ∣ θ ( t ) − θ ^ ∣ 2 + 2 1 ∣ θ ( t − 1 ) − θ ^ ∣ 2 + η ∣ g t ∣ 2 ≤ R .
Its segment with the current point therefore lies in the radius-R R R ball. The calculation for part (i) gives E t + 1 ≤ ( 1 − q C ) B + q C B C ≤ B E_{t+1}\le(1-q_C)B+q_CB_C\le B E t + 1 ≤ ( 1 − q C ) B + q C B C ≤ B , and (22) then gives ∣ θ ( t + 1 ) − θ ^ ∣ 2 ≤ r \lvert\theta^{(t+1)}-\hat{\theta}\rvert_2\le r ∣ θ ( t + 1 ) − θ ^ ∣ 2 ≤ r . Induction proves containment and validates (51) at every update.
For part (iii), let A = ( 1 + β ) I − η H \boldsymbol{A}=(1+\beta)\boldsymbol{I}-\eta\boldsymbol{H} A = ( 1 + β ) I − η H , define K − 1 = 0 \boldsymbol{K}_{-1}=0 K − 1 = 0 , K 0 = I \boldsymbol{K}_0=\boldsymbol{I} K 0 = I , and set K j + 1 = A K j − β K j − 1 \boldsymbol{K}_{j+1}=\boldsymbol{A}\boldsymbol{K}_j-\beta\boldsymbol{K}_{j-1} K j + 1 = A K j − β K j − 1 . For an eigenvalue λ ∈ [ m H , M H ] \lambda\in[m_H,M_H] λ ∈ [ m H , M H ] of H \boldsymbol{H} H , the scalar coefficient 1 + q H 2 − η λ 1+q_H^2-\eta\lambda 1 + q H 2 − η λ belongs to [ − 2 q H , 2 q H ] [-2q_H,2q_H] [ − 2 q H , 2 q H ] . Its characteristic roots therefore have modulus q H q_H q H . If these roots are z 1 , z 2 z_1,z_2 z 1 , z 2 , the polynomial K j ( λ ) = ∑ k = 0 j z 1 k z 2 j − k K_j(\lambda)=\sum_{k=0}^{j}z_1^kz_2^{j-k} K j ( λ ) = ∑ k = 0 j z 1 k z 2 j − k satisfies ∣ K j ( λ ) ∣ ≤ ( j + 1 ) q H j \lvert K_j(\lambda)\rvert\le(j+1)q_H^j ∣ K j ( λ )∣ ≤ ( j + 1 ) q H j . Diagonalization of H \boldsymbol{H} H and summation give
∥ K j ∥ ≤ ( j + 1 ) q H j , η ∑ j = 0 ∞ ∥ K j ∥ ≤ η ( 1 − q H ) 2 = 1 m H . (58) \lVert\boldsymbol{K}_j\rVert\le(j+1)q_H^j,\qquad\eta\sum_{j=0}^{\infty}\lVert\boldsymbol{K}_j\rVert\le\frac{\eta}{(1-q_H)^2}=\frac{1}{m_H}. \tag*{(58)} ∥ K j ∥ ≤ ( j + 1 ) q H j , η j = 0 ∑ ∞ ∥ K j ∥ ≤ ( 1 − q H ) 2 η = m H 1 . (58)
The polynomial sequence in the statement satisfies P t = K t − β K t − 1 \boldsymbol{P}_t=\boldsymbol{K}_t-\beta\boldsymbol{K}_{t-1} P t = K t − β K t − 1 . For q H > 0 q_H>0 q H > 0 and t ≥ 1 t\ge1 t ≥ 1 , this identity gives ∥ P t ∥ ≤ { 1 + ( 1 + q H ) t } q H t \lVert\boldsymbol{P}_t\rVert\le\{1+(1+q_H)t\}q_H^t ∥ P t ∥ ≤ { 1 + ( 1 + q H ) t } q H t . Since ( t + 1 ) q H t ≤ ∑ j = 0 t q H j (t+1)q_H^t\le\sum_{j=0}^{t}q_H^j ( t + 1 ) q H t ≤ ∑ j = 0 t q H j , it also gives ∥ P t ∥ ≤ A H \lVert\boldsymbol{P}_t\rVert\le A_H ∥ P t ∥ ≤ A H . With x = ( 1 − q H ) t x=(1-q_H)t x = ( 1 − q H ) t , the inequality ( 1 + x ) e − x ≤ 2 e − x / 2 (1+x)e^{-x}\le2e^{-x/2} ( 1 + x ) e − x ≤ 2 e − x /2 gives the exponential bound in (54). If q H = 0 q_H=0 q H = 0 , then H = m H I \boldsymbol{H}=m_H\boldsymbol{I} H = m H I , A = 0 \boldsymbol{A}=0 A = 0 , K j = 0 \boldsymbol{K}_j=0 K j = 0 for j ≥ 1 j\ge1 j ≥ 1 , and P t = 0 \boldsymbol{P}_t=0 P t = 0 for t ≥ 1 t\ge1 t ≥ 1 . The bounds follow directly in this case.
Write x t = θ ( t ) − θ ∗ x_t=\theta^{(t)}-\theta^* x t = θ ( t ) − θ ∗ . While ∣ x t ∣ 2 ≤ ϵ 3 \lvert x_t\rvert_2\le\epsilon_3 ∣ x t ∣ 2 ≤ ϵ 3 , Condition 3 gives u t = ∂ L ( θ ( t ) ) − g ∗ − H x t u_t=\partial\mathcal{L}(\theta^{(t)})-g^*-\boldsymbol{H}x_t u t = ∂ L ( θ ( t ) ) − g ∗ − H x t with ∣ u t ∣ 2 ≤ r n \lvert u_t\rvert_2\le r_n ∣ u t ∣ 2 ≤ r n . The restarted recursion has the solution
x t = P t x 0 − η ∑ j = 0 t − 1 K t − 1 − j ( g ∗ + u j ) . x_t=P_tx_0-\eta\sum_{j=0}^{t-1}K_{t-1-j}(g^*+u_j). x t = P t x 0 − η j = 0 ∑ t − 1 K t − 1 − j ( g ∗ + u j ) .
The entry condition implies ∣ x 0 ∣ 2 ≤ ϵ 3 \lvert x_0\rvert_2\le\epsilon_3 ∣ x 0 ∣ 2 ≤ ϵ 3 . If the error bounds hold through index t − 1 t-1 t − 1 , the solution and (56) give ∣ x t ∣ 2 ≤ A H ∣ x 0 ∣ 2 + ( ∣ g ∗ ∣ 2 + r n ) / m H ≤ ϵ 3 \lvert x_t\rvert_2\le A_H\lvert x_0\rvert_2+(\lvert g^*\rvert_2+r_n)/m_H\le\epsilon_3 ∣ x t ∣ 2 ≤ A H ∣ x 0 ∣ 2 + (∣ g ∗ ∣ 2 + r n ) / m H ≤ ϵ 3 . Induction therefore validates every update. The polynomial identity P t − P t + 1 = η H K t P_t-P_{t+1}=\eta HK_t P t − P t + 1 = η H K t gives η ∑ j = 0 T − 1 K j = ( I − P T ) H − 1 \eta\sum_{j=0}^{T-1}K_j=(I-P_T)H^{-1} η ∑ j = 0 T − 1 K j = ( I − P T ) H − 1 . Separating the constant term g ∗ g^* g ∗ proves (54) with R T = − η ∑ j = 0 T − 1 K T − 1 − j u j R_T=-\eta\sum_{j=0}^{T-1}K_{T-1-j}u_j R T = − η ∑ j = 0 T − 1 K T − 1 − j u j . Its norm is bounded by r n / m H r_n/m_H r n / m H through (56).
Functions with bounded local subgradient variation
Corollary 12. Let L \mathcal{L} L be finite and convex on R p \mathbb{R}^p R p , with a minimizer θ ^ \hat{\theta} θ ^ . Suppose that, for r , V r > 0 r,V_r>0 r , V r > 0 , its selected subgradient satisfies ∣ ∂ L ( x ) − ∂ L ( y ) ∣ 2 ≤ V r \lvert\partial\mathcal{L}(x)-\partial\mathcal{L}(y)\rvert_2\le V_r ∣ ∂ L ( x ) − ∂ L ( y ) ∣ 2 ≤ V r whenever ∣ x − y ∣ 2 ≤ r \lvert x-y\rvert_2\le r ∣ x − y ∣ 2 ≤ r , and that
L ( θ ) − L ( θ ^ ) ≥ μ 2 ∣ θ − θ ^ ∣ 2 2 , θ ∈ R p . \mathcal{L}(\theta)-\mathcal{L}(\hat{\theta})\ge\frac{\mu}{2}\lvert\theta-\hat{\theta}\rvert_2^2,\qquad\theta\in\mathbb{R}^p. L ( θ ) − L ( θ ^ ) ≥ 2 μ ∣ θ − θ ^ ∣ 2 2 , θ ∈ R p . Set L = 2 V r / r L=2V_r/r L = 2 V r / r , κ = L / μ ≥ 1 \kappa=L/\mu\ge1 κ = L / μ ≥ 1 , and η = 1 / ( 2 L ) \eta=1/(2L) η = 1/ ( 2 L ) .
Subgradient descent satisfies
∣ θ ( t ) − θ ^ ∣ 2 2 ≤ q t ∣ θ ( 0 ) − θ ^ ∣ 2 2 + ( 1 − q t ) κ r 2 , q = ( 1 + 1 / ( 2 κ ) ) − 1 . \lvert\theta^{(t)}-\hat{\theta}\rvert_2^2\le q^t\lvert\theta^{(0)}-\hat{\theta}\rvert_2^2+(1-q^t)\kappa r^2,\qquad q=(1+1/(2\kappa))^{-1}. ∣ θ ( t ) − θ ^ ∣ 2 2 ≤ q t ∣ θ ( 0 ) − θ ^ ∣ 2 2 + ( 1 − q t ) κ r 2 , q = ( 1 + 1/ ( 2 κ ) ) − 1 .
(ii) Restart the Phase 1 Nesterov method every K = ⌈ 32 κ ⌉ K=\lceil\sqrt{32\kappa}\rceil K = ⌈ 32 κ ⌉ updates, and write z ( j ) z^{(j)} z ( j ) for the output after j j j such blocks. Then
∣ z ( j ) − θ ^ ∣ 2 2 ≤ 4 − j ∣ z ( 0 ) − θ ^ ∣ 2 2 + 2 K κ 3 ( 1 − 4 − j ) r 2 . \left|z^{(j)}-\hat{\theta}\right|_{2}^{2}\leq4^{-j}\left|z^{(0)}-\hat{\theta}\right|_{2}^{2}+\frac{2K\kappa}{3}(1-4^{-j})r^{2}. z ( j ) − θ ^ 2 2 ≤ 4 − j z ( 0 ) − θ ^ 2 2 + 3 2 K κ ( 1 − 4 − j ) r 2 .
Thus the distances decrease at rates exp ( − c t / κ ) \exp(-ct/\kappa) exp ( − c t / κ ) and exp ( − c t / κ ) \exp(-ct/\sqrt{\kappa}) exp ( − c t / κ ) to radii of orders κ r \sqrt{\kappa}r κ r and κ 3 / 4 r \kappa^{3/4}r κ 3/4 r , respectively. The order r r r is necessary for constant-step subgradient descent even when κ \kappa κ is bounded.
Proof. Partitioning a segment into pieces of length at most r r r gives Condition 1 with ϵ 1 = r \epsilon_1=r ϵ 1 = r and L = 2 V r / r L=2V_r/r L = 2 V r / r . Write D t = ∣ θ ( t ) − θ ^ ∣ 2 D_t=|\theta^{(t)}-\hat{\theta}|_2 D t = ∣ θ ( t ) − θ ^ ∣ 2 . The upper oracle bound (20) , convexity, and the subgradient update give
2 η { L ( θ ( t + 1 ) ) − L ( θ ^ ) } ≤ D t 2 − D t + 1 2 − η 2 ( 1 − η L ) ∣ ∂ L ( θ ( t ) ) ∣ 2 2 + η L r 2 . 2\eta\{\mathcal{L}(\theta^{(t+1)})-\mathcal{L}(\hat{\theta})\}\leq D_t^2-D_{t+1}^2-\eta^2(1-\eta L)|\partial\mathcal{L}(\theta^{(t)})|_2^2+\eta Lr^2. 2 η { L ( θ ( t + 1 ) ) − L ( θ ^ )} ≤ D t 2 − D t + 1 2 − η 2 ( 1 − η L ) ∣ ∂ L ( θ ( t ) ) ∣ 2 2 + η L r 2 .
Quadratic growth and 1 − η L ≥ 0 1-\eta L\geq0 1 − η L ≥ 0 imply ( 1 + η μ ) D t + 1 2 ≤ D t 2 + η L r 2 (1+\eta\mu)D_{t+1}^2\leq D_t^2+\eta Lr^2 ( 1 + η μ ) D t + 1 2 ≤ D t 2 + η L r 2 . Summing this recursion proves part (i).
For one Nesterov block, Theorem 2(i) and quadratic growth yield
∣ z ( j + 1 ) − θ ^ ∣ 2 2 ≤ 4 η μ K 2 ∣ z ( j ) − θ ^ ∣ 2 2 + K η L 2 μ r 2 ≤ 1 4 ∣ z ( j ) − θ ^ ∣ 2 2 + K κ 2 r 2 . |z^{(j+1)}-\hat{\theta}|_2^2\leq\frac{4}{\eta\mu K^2}|z^{(j)}-\hat{\theta}|_2^2+\frac{K\eta L^2}{\mu}r^2\leq\frac{1}{4}|z^{(j)}-\hat{\theta}|_2^2+\frac{K\kappa}{2}r^2. ∣ z ( j + 1 ) − θ ^ ∣ 2 2 ≤ η μ K 2 4 ∣ z ( j ) − θ ^ ∣ 2 2 + μ K η L 2 r 2 ≤ 4 1 ∣ z ( j ) − θ ^ ∣ 2 2 + 2 K κ r 2 .
Iteration proves part (ii). Since K ≍ κ K\asymp\sqrt{\kappa} K ≍ κ , the block count gives the stated rate in the total number of updates.
For the lower bound take p = 1 p=1 p = 1 and L ( x ) = r ∣ x ∣ / 4 + x 2 / 4 \mathcal{L}(x)=r|x|/4+x^2/4 L ( x ) = r ∣ x ∣/4 + x 2 /4 . Its local subgradient variation at radius r r r is at most V r = r V_r=r V r = r , and its quadratic growth parameter can be chosen as μ = 1 / 2 \mu=1/2 μ = 1/2 . Hence L = 2 L=2 L = 2 , κ = 4 \kappa=4 κ = 4 , and η = 1 / 4 \eta=1/4 η = 1/4 . Except on the countable set of initial values whose trajectory reaches zero, every subgradient selection gives x t + 1 = 7 x t / 8 − r sign ( x t ) / 16 x_{t+1}=7x_t/8-r\operatorname{sign}(x_t)/16 x t + 1 = 7 x t /8 − r sign ( x t ) /16 . The magnitudes converge to r / 30 r/30 r /30 by the contraction u ↦ ∣ 7 u / 8 − r / 16 ∣ u\mapsto|7u/8-r/16| u ↦ ∣7 u /8 − r /16∣ .
Subgradient increment bounds
Auxiliary results
We use Bernstein’s inequality in its moment form Vershynin, [55 ] .
Lemma 6. Let W 1 , … , W n W_1,\ldots,W_n W 1 , … , W n be independent mean-zero random variables. Suppose that, for every integer k ≥ 2 k\geq2 k ≥ 2 ,
∑ i = 1 n E ∣ W i ∣ k ≤ k ! 2 V B k − 2 . \sum_{i=1}^{n}\mathbb{E}|W_i|^k\leq\frac{k!}{2}VB^{k-2}. i = 1 ∑ n E ∣ W i ∣ k ≤ 2 k ! V B k − 2 . Then, for every t > 0 t>0 t > 0 ,
P { ∣ ∑ i = 1 n W i ∣ > 2 V t + 2 B t } ≤ 2 e − t . \mathbb{P}\left\{\left|\sum_{i=1}^{n}W_i\right|>\sqrt{2Vt}+2Bt\right\}\leq2e^{-t}. P { i = 1 ∑ n W i > 2 V t + 2 B t } ≤ 2 e − t . Proof. For ∣ λ ∣ B < 1 |\lambda|B<1 ∣ λ ∣ B < 1 , independence, the power series for the exponential, and log ( 1 + x ) ≤ x \log(1+x)\leq x log ( 1 + x ) ≤ x give
log E exp ( λ ∑ i W i ) ≤ ∑ i ∑ k ≥ 2 ∣ λ ∣ k E ∣ W i ∣ k k ! ≤ V λ 2 2 ( 1 − B ∣ λ ∣ ) . \log\mathbb{E}\exp\left(\lambda\sum_i W_i\right)\leq\sum_i\sum_{k\geq2}\frac{|\lambda|^k\mathbb{E}|W_i|^k}{k!}\leq\frac{V\lambda^2}{2(1-B|\lambda|)}. log E exp ( λ i ∑ W i ) ≤ i ∑ k ≥ 2 ∑ k ! ∣ λ ∣ k E ∣ W i ∣ k ≤ 2 ( 1 − B ∣ λ ∣ ) V λ 2 . Chernoff’s bound with λ = 2 t / V / { 1 + B 2 t / V } \lambda=\sqrt{2t/V}/\{1+B\sqrt{2t/V}\} λ = 2 t / V / { 1 + B 2 t / V } gives the upper-tail bound at 2 V t + B t \sqrt{2Vt}+Bt 2 V t + B t , and applying the same calculation to − W i -W_i − W i gives the lower tail. The stated, slightly larger threshold follows. If V = 0 V=0 V = 0 , all summands vanish almost surely.
Residual objectives
Proof of Theorem 3. Let g = f + ′ g=f'_+ g = f + ′ and let ν = d g \nu=dg ν = d g be its Stieltjes measure. Convexity and the Lipschitz bound give ν ≥ 0 \nu\ge0 ν ≥ 0 and ν ( R ) ≤ 2 C f \nu(\mathbb{R})\le2C_f ν ( R ) ≤ 2 C f . Write r θ = Y − Z T θ r_\theta=Y-Z^{\mathsf T}\theta r θ = Y − Z T θ , P n q = n − 1 ∑ i q ( Y i , Z i ) P_nq=n^{-1}\sum_i q(Y_i,Z_i) P n q = n − 1 ∑ i q ( Y i , Z i ) , and P q = E [ q ( Y , Z ) ] Pq=\mathbb{E}[q(Y,Z)] P q = E [ q ( Y , Z )] . Conditional absolute continuity and dominated differentiation of the expected objective defined through loss differences give
∇ L ‾ ( θ ) = b − P [ Z g ( r θ ) ] . \nabla\overline{L}(\theta)=b-P[Zg(r_\theta)]. ∇ L ( θ ) = b − P [ Z g ( r θ )] .
For unit vectors u , v u,v u , v and every integer k ≥ 1 k\ge1 k ≥ 1 , the directional sub-Gaussian assumption gives
P [ ∣ v T Z ∣ k ] ≤ ( C k ) k , P [ ∣ v T Z ∣ k ∣ u T Z ∣ ] ≤ C k + 1 k k / 2 . P[|v^{\mathsf T}Z|^k]\le(C\sqrt{k})^k,\qquad P[|v^{\mathsf T}Z|^k|u^{\mathsf T}Z|]\le C^{k+1}k^{k/2}. P [ ∣ v T Z ∣ k ] ≤ ( C k ) k , P [ ∣ v T Z ∣ k ∣ u T Z ∣ ] ≤ C k + 1 k k /2 .
For d = ∣ θ 1 − θ 2 ∣ 2 > 0 d=|\theta_1-\theta_2|_2>0 d = ∣ θ 1 − θ 2 ∣ 2 > 0 , write θ 1 − θ 2 = d u \theta_1-\theta_2=du θ 1 − θ 2 = d u . For every t ∈ R t\in\mathbb{R} t ∈ R , conditioning on Z Z Z gives
P [ ∣ v T Z ∣ k 1 { t lies between r θ 1 and r θ 2 } ] ≤ C ρ d P [ ∣ v T Z ∣ k ∣ u T Z ∣ ] ≤ d k ! C k , k ≥ 2. P[|v^{\mathsf T}Z|^k\mathbb{1}\{t\text{ lies between }r_{\theta_1}\text{ and }r_{\theta_2}\}]\le C_\rho dP[|v^{\mathsf T}Z|^k|u^{\mathsf T}Z|]\le d\,k!C^k,\qquad k\ge2. P [ ∣ v T Z ∣ k 1 { t lies between r θ 1 and r θ 2 }] ≤ C ρ d P [ ∣ v T Z ∣ k ∣ u T Z ∣ ] ≤ d k ! C k , k ≥ 2.
Since ∣ g ( x ) − g ( y ) ∣ ≤ 2 C f |g(x)-g(y)|\le2C_f ∣ g ( x ) − g ( y ) ∣ ≤ 2 C f , integration with respect to ν \nu ν gives, for W v = v T Z { g ( r θ 1 ) − g ( r θ 2 ) } W_v=v^{\mathsf T}Z\{g(r_{\theta_1})-g(r_{\theta_2})\} W v = v T Z { g ( r θ 1 ) − g ( r θ 2 )} ,
P ∣ W v ∣ ≤ C d , P ∣ W v ∣ k ≤ d k ! C k ( k ≥ 2 ) . (59) P|W_v|\le Cd,\qquad P|W_v|^k\le d\,k!C^k\qquad(k\ge2). \tag*{(59)} P ∣ W v ∣ ≤ C d , P ∣ W v ∣ k ≤ d k ! C k ( k ≥ 2 ) . (59)
The same bounds, with larger constants, hold after subtracting the mean. Lemma 6 therefore yields
P { ∣ ( P n − P ) W v ∣ > C ( d t n + t n ) } ≤ 2 e − t . (60) \mathbb{P}\left\{|(P_n-P)W_v|>C\left(\sqrt{\frac{dt}{n}}+\frac{t}{n}\right)\right\}\le2e^{-t}. \tag*{(60)} P { ∣ ( P n − P ) W v ∣ > C ( n d t + n t ) } ≤ 2 e − t . (60)
We now extend the fixed-pair bound to the full decision region. Let δ = n − 1 / 2 \delta=n^{-1/2} δ = n − 1/2 , take a δ \delta δ -net N \mathcal{N} N of Θ \Theta Θ , and take a 1 / 4 1/4 1/4 -net V \mathcal{V} V of the unit sphere. Their logarithmic cardinalities are at most C p log n C_p\log n C p log n and C p C_p C p . A sphere net and the sub-Gaussian tail bound imply, for a sufficiently large C B C_B C B ,
P { max i ≤ n ∣ Z i ∣ 2 > B n } ≤ n − γ p , B n = C B p log n . \mathbb{P}\left\{\max_{i\le n}|Z_i|_2>B_n\right\}\le n^{-\gamma p},\qquad B_n=C_B\sqrt{p\log n}. P { i ≤ n max ∣ Z i ∣ 2 > B n } ≤ n − γ p , B n = C B p log n .
Increasing C B C_B C B if needed also gives sup v ∈ V P [ ∣ v T Z ∣ 1 { ∣ Z ∣ 2 > B n } ] ≤ n − 10 \sup_{v\in\mathcal{V}}P[|v^{\mathsf T}Z|\mathbb{1}\{|Z|_2>B_n\}]\le n^{-10} sup v ∈ V P [ ∣ v T Z ∣1 { ∣ Z ∣ 2 > B n }] ≤ n − 10 . Here p log n ≤ n p\log n\le n p log n ≤ n eventually, since a n → 0 a_n\to0 a n → 0 .
For each θ \theta θ , choose θ ~ ∈ N \widetilde{\theta}\in\mathcal{N} θ ∈ N within δ \delta δ . On the displayed sample event, the difference between their residuals is at most δ B n \delta B_n δ B n . With w n = 2 δ B n w_n=2\delta B_n w n = 2 δ B n , monotonicity bounds every subgradient selection by the strict bracket
B θ ~ = g ( r θ ~ + w n ) − f − ′ ( r θ ~ − w n ) . B_{\widetilde{\theta}}=g(r_{\widetilde{\theta}}+w_n)-f'_-(r_{\widetilde{\theta}}-w_n). B θ = g ( r θ + w n ) − f − ′ ( r θ − w n ) .
For each v ∈ V v\in\mathcal{V} v ∈ V , the nonnegative variable ∣ v T Z ∣ B θ ~ |v^{\mathsf T}Z|B_{\widetilde{\theta}} ∣ v T Z ∣ B θ has mean at most C w n Cw_n C w n and k k k th moment at most w n k ! C k w_n k!C^k w n k ! C k . These assertions follow by integrating P ( ∣ r θ ~ − t ∣ ≤ w n ∣ Z ) ≤ 2 C ρ w n P(|r_{\widetilde{\theta}}-t|\le w_n\mid Z)\le2C_\rho w_n P ( ∣ r θ − t ∣ ≤ w n ∣ Z ) ≤ 2 C ρ w n against ν \nu ν . Bernstein’s inequality and a union bound over N × V \mathcal{N}\times\mathcal{V} N × V therefore give
max θ ~ ∈ N , v ∈ V ( P n + P ) [ ∣ v T Z ∣ B θ ~ ] ≤ C ( w n + a n ) ≤ C a n (61) \max_{\widetilde{\theta}\in\mathcal{N},\,v\in\mathcal{V}}(P_n+P)[|v^{\mathsf T}Z|B_{\widetilde{\theta}}]\le C(w_n+a_n)\le Ca_n \tag*{(61)} θ ∈ N , v ∈ V max ( P n + P ) [ ∣ v T Z ∣ B θ ] ≤ C ( w n + a n ) ≤ C a n (61)
with failure probability at most C n − γ p Cn^{-\gamma p} C n − γ p . At a common decision θ \theta θ , the difference between any two scenario-subgradient choices is bounded by the same strict bracket. Each projection of the resulting difference between subgradients of L \mathcal{L} L is therefore bounded by P n [ ∣ v T Z ∣ B θ ~ ] ≤ C a n P_n[|v^{\mathsf T}Z|B_{\widetilde{\theta}}]\le Ca_n P n [ ∣ v T Z ∣ B θ ] ≤ C a n . The sphere-net inequality gives the Euclidean bound simultaneously for all θ ∈ Θ \theta\in\Theta θ ∈ Θ and all choices.
The population part outside ∣ Z ∣ 2 ≤ B n |Z|_2\le B_n ∣ Z ∣ 2 ≤ B n contributes at most C n − 10 Cn^{-10} C n − 10 . Consequently,
∣ ( P n − P ) v T Z { g ( r θ 1 ) − g ( r θ 2 ) } ∣ ≤ ∣ ( P n − P ) v T Z { g ( r θ ~ 1 ) − g ( r θ ~ 2 ) } ∣ + C a n (62) |(P_n-P)v^{\mathsf T}Z\{g(r_{\theta_1})-g(r_{\theta_2})\}|
\le|(P_n-P)v^{\mathsf T}Z\{g(r_{\widetilde{\theta}_1})-g(r_{\widetilde{\theta}_2})\}|+Ca_n \tag*{(62)} ∣ ( P n − P ) v T Z { g ( r θ 1 ) − g ( r θ 2 )} ∣ ≤ ∣ ( P n − P ) v T Z { g ( r θ 1 ) − g ( r θ 2 )} ∣ + C a n (62)
for all v ∈ V v \in\mathcal{V} v ∈ V . Applying (60) to every net pair with t = A p log n t = Ap\log n t = A p log n , where A A A exceeds the cardinality constants, gives the right side C { ( d + 2 δ ) a n + a n } C\{\sqrt{(d+2\delta)a_n}+a_n\} C { ( d + 2 δ ) a n + a n } . The sphere-net inequality ∣ z ∣ 2 ≤ 2 max v ∈ V ∣ v T z ∣ |z|_2 \le2\max_{v\in\mathcal{V}}|v^{\mathsf{T}}z| ∣ z ∣ 2 ≤ 2 max v ∈ V ∣ v T z ∣ proves (11) . Summing the failure probabilities for truncation, brackets, and net pairs preserves the order C n − γ p Cn^{-\gamma p} C n − γ p . The truncation was used only for interpolation.
For the pointwise bound, each projection of Z g ( r θ 0 ) Zg(r\theta_0) Z g ( r θ 0 ) is sub-Gaussian. Apply the scalar tail bound on V \mathcal{V} V with t = A p log n t = Ap\log n t = A p log n , followed by the sphere-net inequality. This gives C a n C\sqrt{a_n} C a n with the same failure order. Add that failure probability to the preceding event.
Proof of Corollary 2. The crossing bound for the first moment in the proof of Theorem 3, integrated against ν \nu ν , gives ∣ ∇ L ‾ ( θ 1 ) − ∇ L ‾ ( θ 2 ) ∣ 2 ≤ C ∣ θ 1 − θ 2 ∣ 2 |\nabla\overline{\mathcal{L}}(\theta_1)-\nabla\overline{\mathcal{L}}(\theta_2)|_2 \le C|\theta_1-\theta_2|_2 ∣∇ L ( θ 1 ) − ∇ L ( θ 2 ) ∣ 2 ≤ C ∣ θ 1 − θ 2 ∣ 2 . On the event of Theorem 3, with d = ∣ θ 1 − θ 2 ∣ 2 d=|\theta_1-\theta_2|_2 d = ∣ θ 1 − θ 2 ∣ 2 ,
∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ‾ ( θ 1 ) + ∇ L ‾ ( θ 2 ) ∣ 2 ≤ C ( d a n + a n ) . (63) |\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)-\nabla\overline{\mathcal{L}}(\theta_1)+\nabla\overline{\mathcal{L}}(\theta_2)|_2 \le C(\sqrt{d a_n}+a_n). \tag*{(63)} ∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ( θ 1 ) + ∇ L ( θ 2 ) ∣ 2 ≤ C ( d a n + a n ) . (63)
Adding the population bound and considering d ≥ a n d\ge a_n d ≥ a n and d < a n d<a_n d < a n proves part 1.
For part 2, integration along the segment in the convex region Θ s c \Theta_{\mathrm{sc}} Θ sc gives
L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ μ 0 2 d 2 − 2 C 3 a n d 3 / 2 − C a n d ≥ μ 0 4 d 2 . \mathcal{L}(\theta_1)-\mathcal{L}(\theta_2)-\langle\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle\ge\frac{\mu_0}{2}d^2-\frac{2C}{3}\sqrt{a_n}\,d^{3/2}-Ca_n d\ge\frac{\mu_0}{4}d^2. L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ 2 μ 0 d 2 − 3 2 C a n d 3/2 − C a n d ≥ 4 μ 0 d 2 .
when d ≥ C 2 a n d\ge C_2a_n d ≥ C 2 a n . This is Condition 2 with μ = μ 0 \mu=\mu_0 μ = μ 0 .
For part 3, on ∣ θ − θ ∗ ∣ 2 ≤ C 3 a n |\theta-\theta^*|_2\le C_3\sqrt{a_n} ∣ θ − θ ∗ ∣ 2 ≤ C 3 a n , (63) gives
∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − ∇ L ‾ ( θ ) + ∇ L ‾ ( θ ∗ ) ∣ 2 ≤ C a n 3 / 4 . (64) |\partial\mathcal{L}(\theta)-\partial\mathcal{L}(\theta^*)-\nabla\overline{\mathcal{L}}(\theta)+\nabla\overline{\mathcal{L}}(\theta^*)|_2\le Ca_n^{3/4}. \tag*{(64)} ∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − ∇ L ( θ ) + ∇ L ( θ ∗ ) ∣ 2 ≤ C a n 3/4 . (64)
Taylor’s formula bounds the remaining population increment by C ∣ θ − θ ∗ ∣ 2 2 ≤ C a n C|\theta-\theta^*|_2^2\le Ca_n C ∣ θ − θ ∗ ∣ 2 2 ≤ C a n . Since a n ≤ a n 3 / 4 a_n\le a_n^{3/4} a n ≤ a n 3/4 eventually, part 3 follows.
To verify the sufficient density condition in part 3, suppose ρ \rho ρ has a bounded continuous first derivative. Conditional integration gives
H ( θ ) = P [ Z Z T ∫ R ρ ( t + Z T θ ∣ Z ) ν ( d t ) ] . H(\theta)=P\left[ZZ^{\mathsf{T}}\int_{\mathbb{R}}\rho(t+Z^{\mathsf{T}}\theta\mid Z)\nu(dt)\right]. H ( θ ) = P [ Z Z T ∫ R ρ ( t + Z T θ ∣ Z ) ν ( d t ) ] .
For unit u , v u,v u , v ,
∣ v T { H ( θ + d u ) − H ( θ ) } v ∣ ≤ C d P [ ( v T Z ) 2 ∣ u T Z ∣ ] ≤ C d . |v^{\mathsf{T}}\{H(\theta+du)-H(\theta)\}v|\le Cd\,P[(v^{\mathsf{T}}Z)^2|u^{\mathsf{T}}Z|]\le Cd. ∣ v T { H ( θ + d u ) − H ( θ )} v ∣ ≤ C d P [( v T Z ) 2 ∣ u T Z ∣ ] ≤ C d .
Hence the population Hessian is Lipschitz in operator norm. The loss-difference bound justifies dominated differentiation without a moment assumption on Y Y Y .
Proof of Theorem 4. Work on the event supplied by Theorem 3 and write g ( θ ) = ∂ L ( θ ) g(\theta)=\partial\mathcal{L}(\theta) g ( θ ) = ∂ L ( θ ) . The population Hessian is Lipschitz by assumption. For sufficiently small R R R , its smallest eigenvalue is therefore at least m H / 2 m_H/2 m H /2 throughout B ( θ ∗ , R ) B(\theta^*,R) B ( θ ∗ , R ) . The population first-order condition and the pointwise bound give s n = ∣ g ( θ ∗ ) ∣ 2 ≤ C a n s_n=|g(\theta^*)|_2\le C\sqrt{a_n} s n = ∣ g ( θ ∗ ) ∣ 2 ≤ C a n . Corollary 2 gives approximate smoothness on Θ \Theta Θ and approximate strong convexity on B ( θ ∗ , R ) B(\theta^*,R) B ( θ ∗ , R ) , with L = L 0 L=L_0 L = L 0 , μ = m H / 2 \mu=m_H/2 μ = m H /2 , and ϵ j = C j a n \epsilon_j=C_ja_n ϵ j = C j a n for j = 1 , 2 j=1,2 j = 1 , 2 . We may increase L L L so that μ ≤ 2 L \mu\le2L μ ≤ 2 L .
For every unit vector u u u and sufficiently large n n n ,
L ( θ ∗ + R u ) − L ( θ ∗ ) ≥ − s n R + μ R 2 4 ≥ μ R 2 8 . \mathcal{L}(\theta^*+Ru)-\mathcal{L}(\theta^*)\ge-s_nR+\frac{\mu R^2}{4}\ge\frac{\mu R^2}{8}. L ( θ ∗ + R u ) − L ( θ ∗ ) ≥ − s n R + 4 μ R 2 ≥ 8 μ R 2 .
Convexity along rays therefore makes the finite convex objective in (10) coercive, with every minimizer inside B ( θ ∗ , R ) B(\theta^*,R) B ( θ ∗ , R ) . Approximate strong convexity and optimality then give
δ n : = ∣ θ ^ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 4 s n / μ } = O ( a n ) . \delta_n:=|\widehat{\theta}-\theta^*|_2\le\max\{\epsilon_2,4s_n/\mu\}=O(\sqrt{a_n}). δ n := ∣ θ − θ ∗ ∣ 2 ≤ max { ϵ 2 , 4 s n / μ } = O ( a n ) .
Corollary 10(iii)(b) gives geometric convergence to an O ( a n ) O(a_n) O ( a n ) neighborhood of θ ^ \hat{\theta} θ ^ from every initial decision in Θ \Theta Θ . Since a n ≥ n − 1 a_n \ge n^{-1} a n ≥ n − 1 eventually, the geometric terms are at most a n a_n a n for every t ≥ ⌈ c log n ⌉ t \ge\lceil c\log n\rceil t ≥ ⌈ c log n ⌉ when c c c is sufficiently large. The population gradient vanishes at θ ∗ \theta^* θ ∗ and is Lipschitz. Hence its objective gap is at most C ∥ θ − θ ∗ ∥ 2 2 C\lVert\theta-\theta^*\rVert_2^2 C ∥ θ − θ ∗ ∥ 2 2 . The localization bound and the uniform optimization bound prove both asserted population-gap bounds on the same event. This event controls all scenario-subgradient choices simultaneously. The deterministic arguments therefore also cover choices depending on the preceding iterates.
At the deterministic decision θ ∗ \theta^* θ ∗ , the scenario loss is differentiable almost surely. Thus g ( θ ∗ ) = n − 1 ∑ i ψ i \boldsymbol{g}(\theta^*)=n^{-1}\sum_i\psi_i g ( θ ∗ ) = n − 1 ∑ i ψ i , with P ψ i = 0 P\psi_i=0 P ψ i = 0 . The directional sub-Gaussian assumption and ∣ f ± ′ ∣ ≤ C f |f'_{\pm}|\le C_f ∣ f ± ′ ∣ ≤ C f give E ∥ ψ i ∥ 2 2 ≤ C p \mathbb{E}\lVert\psi_i\rVert_2^2\le C p E ∥ ψ i ∥ 2 2 ≤ C p , and independence yields E ∥ g ( θ ∗ ) ∥ 2 2 ≤ C p / n \mathbb{E}\lVert\boldsymbol{g}(\theta^*)\rVert_2^2\le Cp/n E ∥ g ( θ ∗ ) ∥ 2 2 ≤ C p / n . Consequently s n = O P ( p / n ) s_n=O_P(\sqrt{p/n}) s n = O P ( p / n ) , and the preceding localization inequality sharpens to δ n = O P ( p / n + a n ) \delta_n=O_P(\sqrt{p/n}+a_n) δ n = O P ( p / n + a n ) . The uniform increment bound and the Lipschitz population Hessian apply at the random minimizer. Lemma 2 gives ∥ g ( θ ^ ) ∥ 2 ≤ L ϵ 1 \lVert\boldsymbol{g}(\hat{\theta})\rVert_2\le L\epsilon_1 ∥ g ( θ ^ ) ∥ 2 ≤ L ϵ 1 . Hence
∥ H ( θ ^ − θ ∗ ) + g ( θ ∗ ) ∥ 2 ≤ L ϵ 1 + C { δ n a n + a n + δ n 2 } = O P { ( p / n ) 3 / 4 log n + a n } . \left\lVert H(\hat{\theta}-\theta^*)+\boldsymbol{g}(\theta^*)\right\rVert_2\le L\epsilon_1+C\left\{\sqrt{\delta_n a_n}+a_n+\delta_n^2\right\}=O_P\left\{(p/n)^{3/4}\sqrt{\log n}+a_n\right\}. H ( θ ^ − θ ∗ ) + g ( θ ∗ ) 2 ≤ L ϵ 1 + C { δ n a n + a n + δ n 2 } = O P { ( p / n ) 3/4 log n + a n } .
Combining this expansion with the O ( a n ) O(a_n) O ( a n ) optimization bound proves the output expansion. When p = o { n ( log n ) − 2 } p=o\{n(\log n)^{-2}\} p = o { n ( log n ) − 2 } , the a n a_n a n term is smaller than ( p / n ) 3 / 4 log n (p/n)^{3/4}\sqrt{\log n} ( p / n ) 3/4 log n .
For every deterministic unit v n v_n v n , the third absolute moment of v n T H − 1 ψ i v_n^{\mathsf T}H^{-1}\psi_i v n T H − 1 ψ i is bounded. Indeed, ∣ f ± ′ ∣ ≤ C f |f'_{\pm}|\le C_f ∣ f ± ′ ∣ ≤ C f , ∥ H − 1 ∥ ≤ m H − 1 \lVert H^{-1}\rVert\le m_H^{-1} ∥ H − 1 ∥ ≤ m H − 1 , and Assumption 2(ii) bounds every covariate projection. The population first-order condition gives E ψ i = 0 \mathbb{E}\psi_i=0 E ψ i = 0 . The lower bound on σ n \sigma_n σ n therefore gives Lyapunov’s condition. If p = o { n 1 / 3 ( log n ) − 2 / 3 } p=o\{n^{1/3}(\log n)^{-2/3}\} p = o { n 1/3 ( log n ) − 2/3 } , then both n ( p / n ) 3 / 4 log n \sqrt{n}(p/n)^{3/4}\sqrt{\log n} n ( p / n ) 3/4 log n and n a n \sqrt{n}a_n n a n vanish. The expansion and Slutsky’s theorem prove the normal limit. For fixed p p p , a score law and H H H that do not depend on n n n give the vector limit N { 0 , H − 1 E [ ψ i ψ i T ] H − 1 } \mathcal{N}\{0,H^{-1}\mathbb{E}[\psi_i\psi_i^{\mathsf T}]H^{-1}\} N { 0 , H − 1 E [ ψ i ψ i T ] H − 1 } by the multivariate central limit theorem.
Population smoothness and ∇ L ( θ ∗ ) = 0 \nabla\mathcal{L}(\theta^*)=0 ∇ L ( θ ∗ ) = 0 give
∣ L ( θ ~ ) − L ( θ ^ ) ∣ ≤ C { ∥ θ ^ − θ ∗ ∥ 2 ∥ θ ~ − θ ^ ∥ 2 + ∥ θ ~ − θ ^ ∥ 2 2 } = O P { a n p / n + a n 2 } . \left|\mathcal{L}(\tilde{\theta})-\mathcal{L}(\hat{\theta})\right|\le C\left\{\lVert\hat{\theta}-\theta^*\rVert_2\lVert\tilde{\theta}-\hat{\theta}\rVert_2+\lVert\tilde{\theta}-\hat{\theta}\rVert_2^2\right\}=O_P\left\{a_n\sqrt{p/n}+a_n^2\right\}. L ( θ ~ ) − L ( θ ^ ) ≤ C { ∥ θ ^ − θ ∗ ∥ 2 ∥ θ ~ − θ ^ ∥ 2 + ∥ θ ~ − θ ^ ∥ 2 2 } = O P { a n p / n + a n 2 } .
Under the dimension condition for the normal limit, a n = o ( p / n ) a_n=o(\sqrt{p/n}) a n = o ( p / n ) and n a n p / n = p 3 / 2 log n / n → 0 na_n\sqrt{p/n}=p^{3/2}\log n/\sqrt{n}\to0 n a n p / n = p 3/2 log n / n → 0 , proving the asserted equivalence. The sample-complexity consequence follows from part (i), since n ≥ A ( p / ϵ ) log ( e p / ϵ ) n\ge A(p/\epsilon)\log(ep/\epsilon) n ≥ A ( p / ϵ ) log ( e p / ϵ ) gives C p log n / n ≤ ϵ Cp\log n/n\le\epsilon C p log n / n ≤ ϵ for sufficiently large A A A . □
Proof of Proposition 1. Write r i ( θ ) = Y i − Z i T θ r_i(\theta)=Y_i-Z_i^{\mathsf T}\theta r i ( θ ) = Y i − Z i T θ . Choose a minimizer θ ^ \hat{\theta} θ ^ for which the rows with r i ( θ ^ ) = 0 r_i(\hat{\theta})=0 r i ( θ ^ ) = 0 span a space of largest possible dimension, and let A A A denote their index set. We first show that this dimension is p p p . Otherwise choose a nonzero v v v orthogonal to these rows. For small ∣ s ∣ |s| ∣ s ∣ , the active residuals remain zero and all other residuals keep their signs at θ ^ + s v \hat{\theta}+sv θ ^ + s v . The objective is affine on this segment, and minimality at its midpoint makes its slope zero. Full column rank ensures that some inactive residual changes along v v v . Choose the sign of v v v so that a residual first reaches zero at a finite positive s s s . The objective remains affine, hence constant, up to that point. This gives another minimizer with all previous active rows and a new row not in their span, a contradiction. Therefore ∣ A ∣ ≥ p |A|\ge p ∣ A ∣ ≥ p .
Let l l l be the coordinate with Z i l = 1 Z_{il}=1 Z i l = 1 for every i i i , and let e l e_l e l be its unit vector. For sufficiently small δ > 0 \delta>0 δ > 0 , residuals outside A A A keep their signs at θ ^ ± δ e l \hat{\theta}\pm\delta e_l θ ^ ± δ e l , while those in A A A equal ∓ δ \mp\delta ∓ δ . Both points are differentiable, and
∇ L ( θ ^ + δ e l ) − ∇ L ( θ ^ − δ e l ) = c n ∑ i ∈ A Z i . \nabla\mathcal{L}(\hat{\theta}+\delta e_l)-\nabla\mathcal{L}(\hat{\theta}-\delta e_l)=\frac{c}{n}\sum_{i\in A}Z_i. ∇ L ( θ ^ + δ e l ) − ∇ L ( θ ^ − δ e l ) = n c i ∈ A ∑ Z i .
Its l l l th coordinate is c ∣ A ∣ / n ≥ c p / n c|A|/n\ge cp/n c ∣ A ∣/ n ≥ c p / n . Condition 1 therefore gives c p / n ≤ L max { 2 δ , ϵ 1 } cp/n\le L\max\{2\delta,\epsilon_1\} c p / n ≤ L max { 2 δ , ϵ 1 } . Letting δ ↓ 0 \delta\downarrow0 δ ↓ 0 proves the claim. □
Proposition 5. Suppose the observations are independent and the conditional distribution of Y i Y_i Y i given Z i Z_i Z i has a density for every i i i . Consider (10) on R p \mathbb{R}^p R p with a finite convex piecewise linear loss f f f having finitely many pieces. Fix a deterministic initialization θ ( 0 ) \boldsymbol{\theta}^{(0)} θ ( 0 ) and a constant step size η > 0 \eta> 0 η > 0 . With probability one, every selection of scenario subgradients generates the same subgradient descent trajectory, and no iterate has a residual at a breakpoint of f f f . On this event, the following bound holds for every t ≥ 0 t \ge0 t ≥ 0 and every minimizer θ ^ \widehat{\boldsymbol{\theta}} θ . If f ( x ) = c x { τ − I ( x ≤ 0 ) } f(x) = cx\{\tau- \mathbb{I}(x \le0)\} f ( x ) = c x { τ − I ( x ≤ 0 )} , where c > 0 c > 0 c > 0 and τ ∈ ( 0 , 1 ] \tau\in(0,1] τ ∈ ( 0 , 1 ] , and some coordinate l l l satisfies Z i l = 1 Z_{il} = 1 Z i l = 1 for every i i i and n ( τ − b l / c ) ∉ Z n(\tau- b_l/c) \notin\mathbb{Z} n ( τ − b l / c ) ∈ / Z , then
∣ θ ( t ) − θ ^ ∣ 2 + ∣ θ ( t + 1 ) − θ ^ ∣ 2 ≥ η c n min k ∈ Z ∣ n ( τ − b l / c ) − k ∣ . \left|\boldsymbol{\theta}^{(t)}-\widehat{\boldsymbol{\theta}}\right|_2+\left|\boldsymbol{\theta}^{(t+1)}-\widehat{\boldsymbol{\theta}}\right|_2\geq\frac{\eta c}{n}\min_{k\in\mathbb{Z}}\left|n(\tau-b_l/c)-k\right|. θ ( t ) − θ 2 + θ ( t + 1 ) − θ 2 ≥ n η c k ∈ Z min ∣ n ( τ − b l / c ) − k ∣ . For f ( x ) = ∣ x ∣ f(x)=|x| f ( x ) = ∣ x ∣ , b = 0 \boldsymbol{b}=\boldsymbol{0} b = 0 , Z i k ∈ { − 1 , 1 } Z_{ik}\in\{-1,1\} Z ik ∈ { − 1 , 1 } for every i , k i,k i , k , and odd n n n , the left side is at least η p / n \eta\sqrt{p}/n η p / n .
Proof. Condition on Z 1 , … , Z n \boldsymbol{Z}_1,\ldots,\boldsymbol{Z}_n Z 1 , … , Z n . Independence implies that each Y i Y_i Y i still has a conditional density. Before the first time that a residual reaches a breakpoint, every scenario subgradient belongs to the finite set of slopes of f f f . At a first hitting time t t t , the iterate therefore belongs to a finite set determined by the design, θ ( 0 ) \boldsymbol{\theta}^{(0)} θ ( 0 ) , and η \eta η . A hit would require Y i Y_i Y i to equal Z i T θ \boldsymbol{Z}_i^{\mathsf T}\boldsymbol{\theta} Z i T θ plus a breakpoint for one of these finitely many points. Each equality has conditional probability zero. Taking the countable union over t t t excludes every first hit on one event, simultaneously for every selection. Hence all selections yield the same iterates and unique scenario subgradients.
For the pinball loss, let K t K_t K t be the number of negative residuals at θ ( t ) \boldsymbol{\theta}^{(t)} θ ( t ) . The l l l th coordinate of the objective gradient is ( c / n ) { K t − n ( τ − b l / c ) } (c/n)\{K_t-n(\tau-b_l/c)\} ( c / n ) { K t − n ( τ − b l / c )} , hence each step has length at least ( η c / n ) min k ∈ Z ∣ n ( τ − b l / c ) − k ∣ (\eta c/n)\min_{k\in\mathbb{Z}}|n(\tau-b_l/c)-k| ( η c / n ) min k ∈ Z ∣ n ( τ − b l / c ) − k ∣ . For the absolute loss, every gradient coordinate is a sum of n n n numbers in { − 1 , 1 } \{-1,1\} { − 1 , 1 } divided by n n n . Its absolute value is at least 1 / n 1/n 1/ n when n n n is odd, giving step length at least η p / n \eta\sqrt{p}/n η p / n . The triangle inequality gives both conclusions.
:::
Index objectives
The index formulation allows every coefficient of a random scenario vector to vary,
L ( θ ) = 1 n ∑ i = 1 n f ( X i T θ ) + b T θ . (65) \mathcal{L}(\boldsymbol{\theta})=\frac{1}{n}\sum_{i=1}^{n}f(\boldsymbol{X}_i^{\mathsf T}\boldsymbol{\theta})+\boldsymbol{b}^{\mathsf T}\boldsymbol{\theta}. \tag*{(65)} L ( θ ) = n 1 i = 1 ∑ n f ( X i T θ ) + b T θ . (65)
Let X i \boldsymbol{X}_i X i be independent copies of X \boldsymbol{X} X .
Assumption 5. The decision region Θ \Theta Θ is convex and satisfies sup θ ∈ Θ ∣ θ ∣ 2 ≤ C Θ \sup_{\boldsymbol{\theta}\in\Theta}|\boldsymbol{\theta}|_2\leq C_{\Theta} sup θ ∈ Θ ∣ θ ∣ 2 ≤ C Θ and inf θ ∈ Θ ∣ θ 1 ∣ ≥ c 0 > 0 \inf_{\boldsymbol{\theta}\in\Theta}|\theta_1|\geq c_0>0 inf θ ∈ Θ ∣ θ 1 ∣ ≥ c 0 > 0 .
Assumption 6. (i) The conditional distribution of X 1 X_1 X 1 given X − 1 \boldsymbol{X}_{-1} X − 1 has a density ρ X ( x ∣ X − 1 ) \rho_X(x\mid\boldsymbol{X}_{-1}) ρ X ( x ∣ X − 1 ) with sup x ρ X ( x ∣ X − 1 ) ≤ C ρ \sup_x\rho_X(x\mid\boldsymbol{X}_{-1})\leq C_{\rho} sup x ρ X ( x ∣ X − 1 ) ≤ C ρ almost surely.
(ii) The vector X \boldsymbol{X} X is sub-Gaussian with sup ∣ v ∣ 2 = 1 ∥ v T X ∥ ψ 2 ≤ C X \sup_{|\boldsymbol{v}|_2=1}\|\boldsymbol{v}^{\mathsf T}\boldsymbol{X}\|_{\psi_2}\leq C_X sup ∣ v ∣ 2 = 1 ∥ v T X ∥ ψ 2 ≤ C X .
Proposition 6. Suppose Assumptions 3, 5, and 6 hold and at least one of the following conditions holds.
(a) ∣ X 2 ∣ ≤ C X |X_2|\leq C_X ∣ X 2 ∣ ≤ C X almost surely.
(b) The measure ν \nu ν has compact support.
(c) The conditional density ρ X \rho_X ρ X has a bounded continuous first derivative.
For each deterministic θ 0 ∈ Θ \theta_0 \in\Theta θ 0 ∈ Θ , there are C , γ > 0 C,\gamma> 0 C , γ > 0 and an event of probability at least 1 − C n − γ p 1-Cn^{-\gamma p} 1 − C n − γ p on which the following bounds hold. For every θ 1 , θ 2 ∈ Θ \theta_1,\theta_2 \in\Theta θ 1 , θ 2 ∈ Θ and every selection of scenario subgradients,
∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ˉ ( θ 1 ) + ∇ L ˉ ( θ 2 ) ∣ 2 ≤ C { ∣ θ 1 − θ 2 ∣ 2 a n + a n } . (66) \left\lvert\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)-\nabla\bar{L}(\theta_1)+\nabla\bar{L}(\theta_2)\right\rvert_2 \le C\left\{\sqrt{\lvert\theta_1-\theta_2\rvert_2a_n}+a_n\right\}. \tag*{(66)} ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ˉ ( θ 1 ) + ∇ L ˉ ( θ 2 ) 2 ≤ C { ∣ θ 1 − θ 2 ∣ 2 a n + a n } . (66)
At θ 0 \theta_0 θ 0 , ∣ ∂ L ( θ 0 ) − ∇ L ˉ ( θ 0 ) ∣ 2 ≤ C a n \lvert\partial\mathcal{L}(\theta_0)-\nabla\bar{L}(\theta_0)\rvert_2 \le C\sqrt{a_n} ∣ ∂ L ( θ 0 ) − ∇ L ˉ ( θ 0 ) ∣ 2 ≤ C a n . Parts 1 and 2 of Corollary 2 follow. In cases (a) and (b), a bounded continuous first derivative of ρ X \rho_X ρ X also gives a Lipschitz population Hessian. If the population Hessian is Lipschitz near an interior minimizer θ ∗ \theta^* θ ∗ and m H I ⪯ ∇ 2 L ( θ ∗ ) ⪯ M H I m_H\mathbb{I} \preceq\nabla^2\mathcal{L}(\theta^*) \preceq M_H\mathbb{I} m H I ⪯ ∇ 2 L ( θ ∗ ) ⪯ M H I , part 3 of Corollary 2 holds. The procedures in Theorem 4 and Appendix A.3, with their region and initialization requirements, give all its conclusions with ψ i = b + X i f + ′ ( X i T θ ∗ ) \psi_i=b+X_i f_+'(X_i^{\mathsf T}\theta^*) ψ i = b + X i f + ′ ( X i T θ ∗ ) .
Proof of Proposition 6. Write g = f + g=f_+ g = f + , ν = d g \nu=dg ν = d g , and U = X − 1 U=X_{-1} U = X − 1 , and let C Θ = sup θ ∈ Θ ∣ θ ∣ 2 C_\Theta=\sup_{\theta\in\Theta}\lvert\theta\rvert_2 C Θ = sup θ ∈ Θ ∣ θ ∣ 2 . Case (b) includes every piecewise affine loss with finitely many breakpoints. Cases (a) and (c) allow arbitrary finite convex Lipschitz losses. The Lipschitz bound on f f f and the sub-Gaussian assumption make f ( X T θ ) f(X^{\mathsf T}\theta) f ( X T θ ) integrable.
The curvature measure satisfies ν ≥ 0 \nu\ge0 ν ≥ 0 and ν ( R ) ≤ 2 C f \nu(\mathbb{R})\le2C_f ν ( R ) ≤ 2 C f . For c ∈ R c\in\mathbb{R} c ∈ R , define
a θ ( c ) = c − U T θ − 1 θ 1 , x θ ( c ) = ( a θ ( c ) , U T ) T , ρ θ ( c ∣ U ) = ρ X ( a θ ( c ) ∣ U ) ∣ θ 1 ∣ . a_\theta(c)=\frac{c-U^{\mathsf T}\theta_{-1}}{\theta_1},\qquad x_\theta(c)=\left(a_\theta(c),U^{\mathsf T}\right)^{\mathsf T},\qquad\rho_\theta(c\mid U)=\frac{\rho_X(a_\theta(c)\mid U)}{\lvert\theta_1\rvert}. a θ ( c ) = θ 1 c − U T θ − 1 , x θ ( c ) = ( a θ ( c ) , U T ) T , ρ θ ( c ∣ U ) = ∣ θ 1 ∣ ρ X ( a θ ( c ) ∣ U ) .
Convexity of Θ \Theta Θ fixes the sign of θ 1 \theta_1 θ 1 . In case (a), only ∣ c ∣ ≤ C X C Θ \lvert c\rvert\le C_XC_\Theta ∣ c ∣ ≤ C X C Θ can contribute, and all projections of x θ ( c ) x_\theta(c) x θ ( c ) on this interval are bounded. In the sub-Gaussian cases, those projections have k k k th moments at most C k ( ∣ c ∣ + k ) k C^k(\lvert c\rvert+\sqrt{k})^k C k (∣ c ∣ + k ) k . This suffices on every compact level interval, including a neighborhood of the support of ν \nu ν in case (b).
In case (c) we need a bound at every level. The marginal density p θ ( c ) = E ρ θ ( c ∣ U ) p_\theta(c)=\mathbb{E}\rho_\theta(c\mid U) p θ ( c ) = E ρ θ ( c ∣ U ) is uniformly bounded and Lipschitz, while X T θ X^{\mathsf T}\theta X T θ is uniformly sub-Gaussian. Thus
p θ ( c ) ≤ C exp ( − c 1 c 2 ) , c ∈ R , θ ∈ Θ . (67) p_\theta(c)\le C\exp(-c_1c^2),\qquad c\in\mathbb{R},\quad\theta\in\Theta. \tag*{(67)} p θ ( c ) ≤ C exp ( − c 1 c 2 ) , c ∈ R , θ ∈ Θ. (67)
Indeed, if L 1 > 0 L_1>0 L 1 > 0 bounds the density Lipschitz constants, density at least p θ ( c ) / 2 p_\theta(c)/2 p θ ( c ) /2 persists on an interval of radius p θ ( c ) / ( 2 L 1 ) p_\theta(c)/(2L_1) p θ ( c ) / ( 2 L 1 ) about c c c . Its probability is at least p θ ( c ) 2 / ( 2 L 1 ) p_\theta(c)^2/(2L_1) p θ ( c ) 2 / ( 2 L 1 ) , and for large ∣ c ∣ \lvert c\rvert ∣ c ∣ it lies in the sub-Gaussian tail ∣ X T θ ∣ ≥ ∣ c ∣ / 2 \lvert X^{\mathsf T}\theta\rvert\ge\lvert c\rvert/2 ∣ X T θ ∣ ≥ ∣ c ∣ /2 . This proves the display. For unit u , v u,v u , v and nonnegative integers j , k j,k j , k with m = j + k ≥ 1 m=j+k\ge1 m = j + k ≥ 1 , Cauchy–Schwarz and ρ θ ≤ C \rho_\theta\le C ρ θ ≤ C now give
E [ ρ θ ( c ∣ U ) ∣ v T x θ ( c ) ∣ j ∣ u T x θ ( c ) ∣ k ] ≤ C m + 1 e − c 1 c 2 / 2 ( ∣ c ∣ + m ) m ≤ C m + 1 m m / 2 . \mathbb{E}\left[\rho_\theta(c\mid U)\lvert v^{\mathsf T}x_\theta(c)\rvert^j\lvert u^{\mathsf T}x_\theta(c)\rvert^k\right]\le C^{m+1}e^{-c_1c^2/2}(\lvert c\rvert+\sqrt{m})^m\le C^{m+1}m^{m/2}. E [ ρ θ ( c ∣ U ) ∣ v T x θ ( c ) ∣ j ∣ u T x θ ( c ) ∣ k ] ≤ C m + 1 e − c 1 c 2 /2 (∣ c ∣ + m ) m ≤ C m + 1 m m /2 .
In particular, the section moments
E [ ρ θ ( c ∣ U ) ∣ v T x θ ( c ) ∣ 2 k ] ≤ k ! C k , \mathbb{E}\left[\rho_\theta(c\mid U)\lvert v^{\mathsf T}x_\theta(c)\rvert^{2k}\right]\le k!C^k, E [ ρ θ ( c ∣ U ) ∣ v T x θ ( c ) ∣ 2 k ] ≤ k ! C k ,
E [ ρ θ ( c ∣ U ) ∣ v T x θ ( c ) ∣ k ∣ u T x θ ( c ) ∣ ] ≤ k ! C k (68) \mathbb{E}\left[\rho_\theta(c\mid U)\lvert v^{\mathsf T}x_\theta(c)\rvert^k\lvert u^{\mathsf T}x_\theta(c)\rvert\right]\le k!C^k \tag*{(68)} E [ ρ θ ( c ∣ U ) ∣ v T x θ ( c ) ∣ k ∣ u T x θ ( c )∣ ] ≤ k ! C k (68)
hold uniformly at all levels in cases (a) and (c), and on every compact level interval in case (b). Write θ 1 − θ 0 = d u \theta_1-\theta_0=du θ 1 − θ 0 = d u and θ s = θ 0 + s d u \theta_s=\theta_0+sdu θ s = θ 0 + s d u . Conditional on U U U , the threshold a θ s ( c ) a_{\theta_s}(c) a θ s ( c ) is monotone in s s s , with derivative of absolute value d ∣ u T x θ s ( c ) ∣ / ∣ θ s , 1 ∣ d\lvert u^{\mathsf T}x_{\theta_s}(c)\rvert/\lvert\theta_{s,1}\rvert d ∣ u T x θ s ( c )∣ / ∣ θ s , 1 ∣ . Integrating along this interval gives
E [ ∣ v T X ∣ k ∣ I ( X T θ 1 < c ) − I ( X T θ 0 < c ) ∣ ] = d ∫ 0 1 E [ ρ θ s ( c ∣ U ) ∣ v T x θ s ( c ) ∣ k ∣ u T x θ s ( c ) ∣ ] d s ≤ d k ! C k . (69) \mathbb{E}\left[\lvert v^{\mathsf T}X\rvert^k\left\lvert\mathbb{I}(X^{\mathsf T}\theta_1<c)-\mathbb{I}(X^{\mathsf T}\theta_0<c)\right\rvert\right]
=d\int_0^1\mathbb{E}\left[\rho_{\theta_s}(c\mid U)\lvert v^{\mathsf T}x_{\theta_s}(c)\rvert^k\lvert u^{\mathsf T}x_{\theta_s}(c)\rvert\right]\,ds\le dk!C^k. \tag*{(69)} E [ ∣ v T X ∣ k I ( X T θ 1 < c ) − I ( X T θ 0 < c ) ] = d ∫ 0 1 E [ ρ θ s ( c ∣ U ) ∣ v T x θ s ( c ) ∣ k ∣ u T x θ s ( c )∣ ] d s ≤ d k ! C k . (69)
Integration over [ c − w , c + w ] [c-w,c+w] [ c − w , c + w ] similarly gives
E [ ∣ v T X ∣ k I { ∣ X T θ − c ∣ ≤ w } ] ≤ w C k + 1 k k / 2 . (70) \mathbb{E}\left[\lvert v^{\mathsf T}X\rvert^k\mathbb{I}\{\lvert X^{\mathsf T}\theta-c\rvert\le w\}\right]\le wC^{k+1}k^{k/2}. \tag*{(70)} E [ ∣ v T X ∣ k I {∣ X T θ − c ∣ ≤ w } ] ≤ w C k + 1 k k /2 . (70)
In case (b) take c c c in the support of ν \nu ν and 0 < w ≤ 1 0 < w \le1 0 < w ≤ 1 . The other two cases allow every c c c and w > 0 w > 0 w > 0 . Integrating the crossing bound against ν \nu ν gives, for W v = v T X { g ( X T θ 1 ) − g ( X T θ 0 ) } W_v = v^{\mathsf{T}}X\{g(X^{\mathsf{T}}\theta_1) - g(X^{\mathsf{T}}\theta_0)\} W v = v T X { g ( X T θ 1 ) − g ( X T θ 0 )} ,
E ∣ W v ∣ ≤ C d , E ∣ W v ∣ k ≤ d k ! C k , k ≥ 2. \mathbb{E}|W_v| \le Cd,\qquad\mathbb{E}|W_v|^k \le d\,k!C^k,\quad k \ge2. E ∣ W v ∣ ≤ C d , E ∣ W v ∣ k ≤ d k ! C k , k ≥ 2.
The same bounds hold after subtracting the mean, with larger constants. Write P n P_n P n for the empirical average and P P P for expectation under the law of X X X . Lemma 6 yields
Pr { ∣ ( P n − P ) W v ∣ > C ( d t / n + t / n ) } ≤ 2 e − t . \Pr\left\{\left|(P_n-P)W_v\right| > C\left(\sqrt{dt/n}+t/n\right)\right\} \le2e^{-t}. Pr { ∣ ( P n − P ) W v ∣ > C ( d t / n + t / n ) } ≤ 2 e − t .
Let δ = n − 12 \delta=n^{-12} δ = n − 12 , let N \mathcal{N} N be a δ \delta δ -net of Θ \Theta Θ , and let V \mathcal{V} V be a 1 / 4 1/4 1/4 -net of the unit sphere. Their logarithmic cardinalities are at most C p log n C p\log n C p log n and C p C p C p . The sub-Gaussian bound gives
Pr { max i ≤ n ∣ X i ∣ 2 > B n } ≤ n − γ p , B n = C B p log n . \Pr\left\{\max_{i\le n}|X_i|_2>B_n\right\}\le n^{-\gamma p},\qquad B_n=C_B\sqrt{p\log n}. Pr { i ≤ n max ∣ X i ∣ 2 > B n } ≤ n − γ p , B n = C B p log n .
Choose C B C_B C B sufficiently large that also sup v ∈ V P [ ∣ v T X ∣ I { ∣ X ∣ 2 > B n } ] ≤ n − 10 \sup_{v\in\mathcal{V}}P[|v^{\mathsf{T}}X|\mathbb{I}\{|X|_2>B_n\}]\le n^{-10} sup v ∈ V P [ ∣ v T X ∣ I { ∣ X ∣ 2 > B n }] ≤ n − 10 . For each decision choose a representative θ ~ ∈ N \widetilde{\theta}\in\mathcal{N} θ ∈ N within δ \delta δ and put w n = 2 δ B n w_n=2\delta B_n w n = 2 δ B n . On the sample norm event, monotonicity bounds the difference between its scenario subgradient and the derivative at the representative by
B θ ~ = g ( X T θ ~ + w n ) − f − ′ ( X T θ ~ − w n ) . B_{\widetilde{\theta}}=g(X^{\mathsf{T}}\widetilde{\theta}+w_n)-f'_{-}(X^{\mathsf{T}}\widetilde{\theta}-w_n). B θ = g ( X T θ + w n ) − f − ′ ( X T θ − w n ) .
The slab bound integrated against ν \nu ν gives mean at most C w n Cw_n C w n and k k k th moment at most w n k ! C k w_n k!C^k w n k ! C k for ∣ v T X ∣ B θ ~ |v^{\mathsf{T}}X|B_{\widetilde{\theta}} ∣ v T X ∣ B θ . In case (b), w n ≤ 1 w_n\le1 w n ≤ 1 eventually and only the specified neighborhood of the support of ν \nu ν is used. Bernstein’s inequality and the union bound over N × V \mathcal{N}\times\mathcal{V} N × V give
max θ ~ ∈ N , v ∈ V ( P n + P ) [ ∣ v T X ∣ B θ ~ ] ≤ C ( w n + a n ) ≤ C a n \max_{\widetilde{\theta}\in\mathcal{N},\,v\in\mathcal{V}}(P_n+P)[|v^{\mathsf{T}}X|B_{\widetilde{\theta}}]\le C(w_n+a_n)\le Ca_n θ ∈ N , v ∈ V max ( P n + P ) [ ∣ v T X ∣ B θ ] ≤ C ( w n + a n ) ≤ C a n
with failure probability at most C n − γ p Cn^{-\gamma p} C n − γ p . The population tail outside the norm event contributes at most C n − 10 Cn^{-10} C n − 10 . Thus interpolation of each projected mean-zero increment from its two net representatives costs at most C a n Ca_n C a n . This also bounds the difference between any two scenario-subgradient selections at a common decision.
Apply the fixed-pair bound on N × N × V \mathcal{N}\times\mathcal{N}\times\mathcal{V} N × N × V with t = A p log n t=Ap\log n t = A p log n , taking A A A larger than the cardinality constants. The representative distance is at most d + 2 δ d+2\delta d + 2 δ . Adding the interpolation error and using the sphere-net inequality ∣ z ∣ 2 ≤ 2 max v ∈ V ∣ v T z ∣ |z|_2\le2\max_{v\in\mathcal{V}}|v^{\mathsf{T}}z| ∣ z ∣ 2 ≤ 2 max v ∈ V ∣ v T z ∣ prove (66) . Conditional absolute continuity and dominated differentiation give
∇ L ‾ ( θ ) = b + P [ X g ( X T θ ) ] . \nabla\overline{L}(\theta)=b+P[Xg(X^{\mathsf{T}}\theta)]. ∇ L ( θ ) = b + P [ X g ( X T θ )] .
At a deterministic decision every scenario index has a density, and each projection of X g ( X T θ 0 ) Xg(X^{\mathsf{T}}\theta_0) X g ( X T θ 0 ) is sub-Gaussian. A scalar tail bound on V \mathcal{V} V gives the pointwise bound C a n C\sqrt{a_n} C a n . Adding the failure probabilities for truncation, brackets, net pairs, and the pointwise bound preserves the order C n − γ p Cn^{-\gamma p} C n − γ p .
We verify the asserted Hessian Lipschitz property in cases (a) and (b) under the density derivative bound. Conditional integration with the notation above gives
H ( θ ) = E ∫ R ρ θ ( c ∣ U ) x θ ( c ) x θ ( c ) T ν ( d c ) . H(\theta)=\mathbb{E}\int_{\mathbb{R}}\rho_\theta(c\mid U)x_\theta(c)x_\theta(c)^{\mathsf{T}}\nu(dc). H ( θ ) = E ∫ R ρ θ ( c ∣ U ) x θ ( c ) x θ ( c ) T ν ( d c ) .
For bounded X X X , only the compact interval ∣ c ∣ ≤ C X C Θ |c|\le C_XC_\Theta ∣ c ∣ ≤ C X C Θ contributes. For compactly supported ν \nu ν , the integral is already over a compact interval. On either interval, projections of x θ ( c ) x_\theta(c) x θ ( c ) have bounded sub-Gaussian norms, and subtracting the defining fractions gives ∥ a θ ( c ) − a θ ′ ( c ) ∥ ψ 2 ≤ C ∣ θ − θ ′ ∣ 2 \|a_\theta(c)-a_{\theta'}(c)\|_{\psi_2}\le C|\theta-\theta'|_2 ∥ a θ ( c ) − a θ ′ ( c ) ∥ ψ 2 ≤ C ∣ θ − θ ′ ∣ 2 . The density derivative bound and Hölder’s inequality applied to differences of the integrand matrices therefore yield ∥ H ( θ ) − H ( θ ′ ) ∥ ≤ C ∣ θ − θ ′ ∣ 2 \|H(\theta)-H(\theta')\|\le C|\theta-\theta'|_2 ∥ H ( θ ) − H ( θ ′ ) ∥ ≤ C ∣ θ − θ ′ ∣ 2 .
Proof of the conclusions in Example 3. Set f ( t ) = ( 1 − t ) + f(t)=(1-t)_{+} f ( t ) = ( 1 − t ) + and b = 0 \mathbf{b}=\mathbf{0} b = 0 in the general index formulation. Then f + ′ ( t ) = − I ( t < 1 ) f_{+}'(t)=-\mathbb{I}(t<1) f + ′ ( t ) = − I ( t < 1 ) and ν = δ 1 \nu=\delta_{1} ν = δ 1 . Case (b) therefore gives the increment and pointwise score bounds under the density assumption.
Under the additional density derivative bound, the compact support of ν \nu ν gives the population Hessian Lipschitz property proved in Proposition 6. The Hessian formula becomes
∇ 2 L S V M ( θ ) = E [ ρ X ( a θ ( 1 ) ∣ U ) ∣ θ 1 ∣ x θ ( 1 ) x θ ( 1 ) T ] . \nabla^{2}\mathcal{L}_{\mathrm{SVM}}(\theta)=\mathbb{E}\left[\frac{\rho_{X}(a_{\theta}(1)\mid U)}{|\theta_{1}|}x_{\theta}(1)x_{\theta}(1)^{\mathsf{T}}\right]. ∇ 2 L SVM ( θ ) = E [ ∣ θ 1 ∣ ρ X ( a θ ( 1 ) ∣ U ) x θ ( 1 ) x θ ( 1 ) T ] .
The population score at the interior solution is ψ i = − X i I ( X i T θ ∗ < 1 ) \psi_i=-X_i\mathbb{I}(X_i^{\mathsf{T}}\theta^{*}<1) ψ i = − X i I ( X i T θ ∗ < 1 ) and has mean zero. Applying Theorem 4 and its unprojected extension with this score gives the containment, expansion, and normal limits asserted in the example. Appendix § 8 .§ 3 gives the same conclusions for the complete procedures.
Piecewise linear costs
Proof of Corollary 3. Write Δ α j k = α j − α k \Delta\alpha_{jk}=\alpha_j-\alpha_k Δ α j k = α j − α k and Δ β j k = β j − β k \Delta\beta_{jk}=\beta_j-\beta_k Δ β j k = β j − β k . Each pair of affine pieces ties with probability zero at a deterministic decision. If the maximizing pieces at θ 0 \theta_0 θ 0 and θ 1 \theta_1 θ 1 differ, their difference changes sign between these decisions. For θ 1 − θ 0 = d u \theta_1-\theta_0=d u θ 1 − θ 0 = d u and any unit v v v , the conditional density assumption in Corollary 3 gives
E [ ∣ v T Δ β j k ∣ q I { pieces j , k cross between θ 0 and θ 1 } ] ≤ C ρ d E [ ∣ v T Δ β j k ∣ q ∣ u T Δ β j k ] ≤ d q ! C q , q ≥ 1. \mathbb{E}\left[|v^{\mathsf{T}}\Delta\beta_{jk}|^{q}\mathbb{I}\{\text{pieces }j,k\text{ cross between }\theta_0\text{ and }\theta_1\}\right]
\le C_{\rho}d\mathbb{E}[|v^{\mathsf{T}}\Delta\beta_{jk}|^{q}\mid u^{\mathsf{T}}\Delta\beta_{jk}]
\le d\,q!C^{q},\qquad q\ge1. E [ ∣ v T Δ β j k ∣ q I { pieces j , k cross between θ 0 and θ 1 } ] ≤ C ρ d E [ ∣ v T Δ β j k ∣ q ∣ u T Δ β j k ] ≤ d q ! C q , q ≥ 1.
The absolute projected gradient increment is bounded by the sum of these weighted crossing indicators over j < k j<k j < k . Since there are at most K ( K − 1 ) / 2 K(K-1)/2 K ( K − 1 ) /2 pairs, its first moment is at most C d Cd C d and its q q q th moment at most d q ! C q d\,q!C^{q} d q ! C q . Thus the Bernstein bound in (58) holds for each pair of decisions and direction, with constants depending on K K K .
To cover ties and data-dependent decisions, use the decision and sphere nets from the proof of Theorem 3. On the sample event max i , k ∣ β i k ∣ 2 ≤ B n = C B p log n \max_{i,k}|\beta_{ik}|_{2}\le B_n=C_B\sqrt{p\log n} max i , k ∣ β ik ∣ 2 ≤ B n = C B p log n , every affine piece active at a decision within δ \delta δ of a net point either agrees with its maximizing piece or belongs to a pair satisfying
∣ Δ α j k + Δ β j k T θ ~ ∣ ≤ 2 δ B n . |\Delta\alpha_{jk}+\Delta\beta_{jk}^{\mathsf{T}}\widetilde{\theta}|\le2\delta B_n. ∣Δ α j k + Δ β j k T θ ∣ ≤ 2 δ B n .
For w > 0 w>0 w > 0 , the conditional density assumption in Corollary 3 gives
E [ ∣ v T Δ β j k ∣ q I { ∣ Δ α j k + Δ β j k T θ ~ ∣ ≤ w } ] ≤ 2 C ρ w E ∣ v T Δ β j k ∣ q ≤ w q ! C q . \mathbb{E}[|v^{\mathsf{T}}\Delta\beta_{jk}|^{q}\mathbb{I}\{|\Delta\alpha_{jk}+\Delta\beta_{jk}^{\mathsf{T}}\widetilde{\theta}|\le w\}]
\le2C_{\rho}w\mathbb{E}|v^{\mathsf{T}}\Delta\beta_{jk}|^{q}
\le w\,q!C^{q}. E [ ∣ v T Δ β j k ∣ q I { ∣Δ α j k + Δ β j k T θ ∣ ≤ w }] ≤ 2 C ρ w E ∣ v T Δ β j k ∣ q ≤ w q ! C q .
Summing over pairs bounds every convex combination of active pieces by the same bracket. Its Bernstein bound and the union bound give interpolation error C a n Ca_n C a n . The sample norm tail, population tail, and net-pair bounds proceed exactly as in the proof of Theorem 3, establishing the increment bound for all subgradient choices. At a deterministic decision, each score projection is bounded in absolute value by max k ∣ v T β k ∣ \max_k|v^{\mathsf{T}}\beta_k| max k ∣ v T β k ∣ and is therefore sub-Gaussian. This gives the pointwise bound.
The crossing bound for the first moment proves that ∇ ‾ L \overline{\nabla}\mathcal{L} ∇ L is Lipschitz. The segment integration and Taylor arguments in the proof of Corollary 2 then give its conclusions under the stated Hessian assumption. At an interior population solution, ψ i \psi_i ψ i has mean zero, second moment at most C p C_p C p , and bounded third moments in deterministic unit directions. The proof of Theorem 4 consequently applies. Finally,
∣ Q ( θ , ξ ) − Q ( θ r e f , ξ ) ∣ ≤ max k ∣ β k T ( θ − θ r e f ) ∣ |Q(\theta,\xi)-Q(\theta_{\mathrm{ref}},\xi)|\le\max_k|\beta_k^{\mathsf{T}}(\theta-\theta_{\mathrm{ref}})| ∣ Q ( θ , ξ ) − Q ( θ ref , ξ ) ∣ ≤ k max ∣ β k T ( θ − θ ref ) ∣
is integrable, which justifies the expected loss differences and their derivatives without offset moments. For any pair j < k j<k j < k , replacing the conditional density assumption by Assumptions 5 and 6 on W j k = ( Δ β j k T , Δ α j k ) T W_{jk}=(\Delta\beta_{jk}^{\mathsf{T}},\Delta\alpha_{jk})^{\mathsf{T}} W j k = ( Δ β j k T , Δ α j k ) T and the region { ( θ T , 1 ) T : θ ∈ Θ } \{(\theta^{\mathsf{T}},1)^{\mathsf{T}}:\theta\in\Theta\} {( θ T , 1 ) T : θ ∈ Θ } , after the same permutation of coordinates, gives the same conclusions. Apply (67) and (68) at level zero with test direction ( v T , 0 ) T (v^{\mathsf{T}},0)^{\mathsf{T}} ( v T , 0 ) T and displacement direction ( u T , 0 ) T (u^{\mathsf{T}},0)^{\mathsf{T}} ( u T , 0 ) T . The positive-part loss has curvature measure δ 0 \delta_0 δ 0 , and case (b) of Proposition 6 gives the crossing and slab bounds used above.
Rank regression
Example 4. Rank regression uses pairwise residual differences [25 ] , with objective
L r a n k ( θ ) = 1 n ( n − 1 ) ∑ i ≠ j ∣ ( Y i − Z i T θ ) − ( Y j − Z j T θ ) ∣ . \mathcal{L}_{\mathrm{rank}}(\theta)=\frac{1}{n(n-1)}\sum_{i\ne j}\left|(Y_i-Z_i^{\mathsf{T}}\theta)-(Y_j-Z_j^{\mathsf{T}}\theta)\right|. L rank ( θ ) = n ( n − 1 ) 1 i = j ∑ ( Y i − Z i T θ ) − ( Y j − Z j T θ ) . This formulation is used in tuning-free sparse regression and can improve efficiency over single-quantile regression [57 ] . We use slope coordinates, since pairwise differences eliminate an intercept.
On a convex bounded decision region, under the distributional assumptions of Theorem 3, the pairwise objective satisfies its uniform increment and pointwise score bounds. If the conditional density derivative bound holds and the expected pairwise objective satisfies the Hessian, region, and initialization conditions of Theorem 4 or its unprojected extension, the corresponding methods and complete procedures attain distance O P ( a n ) O_P(a_n) O P ( a n ) from a minimizer after O ( log n ) O(\log n) O ( log n ) updates using the pairwise objective. For independent copies X = ( Y , Z T ) T X=(Y,Z^{\mathsf{T}})^{\mathsf{T}} X = ( Y , Z T ) T and X ′ = ( Y ′ , Z ′ T ) T X'=(Y',Z'^{\mathsf{T}})^{\mathsf{T}} X ′ = ( Y ′ , Z ′ T ) T , define the score kernel and its first-order projection by
k ( X , X ′ ) = − ( Z − Z ′ ) sign { Y − Y ′ − ( Z − Z ′ ) T θ ∗ } , ψ ( X ) = 2 E [ k ( X , X ′ ) ∣ X ] . k(X,X')=-(Z-Z')\operatorname{sign}\{Y-Y'-(Z-Z')^{\mathsf{T}}\theta^*\},\qquad\psi(X)=2\mathbb{E}[k(X,X')\mid X]. k ( X , X ′ ) = − ( Z − Z ′ ) sign { Y − Y ′ − ( Z − Z ′ ) T θ ∗ } , ψ ( X ) = 2 E [ k ( X , X ′ ) ∣ X ] . Their outputs satisfy
θ ~ − θ ∗ = − H − 1 1 n ∑ i = 1 n ψ ( X i ) + O P { ( p n ) 3 / 4 log n + a n } . \widetilde{\theta}-\theta^*=-H^{-1}\frac{1}{n}\sum_{i=1}^{n}\psi(X_i)+O_P\left\{\left(\frac{p}{n}\right)^{3/4}\sqrt{\log n}+a_n\right\}. θ − θ ∗ = − H − 1 n 1 i = 1 ∑ n ψ ( X i ) + O P { ( n p ) 3/4 log n + a n } . The remainder reduces to O P { ( p / n ) 3 / 4 log n } O_P\{(p/n)^{3/4}\sqrt{\log n}\} O P {( p / n ) 3/4 log n } when p = o { n ( log n ) − 2 } p=o\{n(\log n)^{-2}\} p = o { n ( log n ) − 2 } . For any deterministic unit contrast with variance bounded below, p = o { n 1 / 3 ( log n ) − 2 / 3 } p=o\{n^{1/3}(\log n)^{-2/3}\} p = o { n 1/3 ( log n ) − 2/3 } gives the normal limit in Theorem 4, with variance Var { v n T H − 1 ψ ( X ) } \operatorname{Var}\{v_n^{\mathsf{T}}H^{-1}\psi(X)\} Var { v n T H − 1 ψ ( X )} .
Proof of the rank-regression conclusions in Example 4. Write Δ Y = Y − Y ′ \Delta Y=Y-Y' Δ Y = Y − Y ′ and Δ Z = Z − Z ′ \Delta Z=Z-Z' Δ Z = Z − Z ′ for independent observations. The conditional density
ρ Δ ( t ∣ Z , Z ′ ) = ∫ ρ ( t + y ∣ Z ) ρ ( y ∣ Z ′ ) d y \rho_{\Delta}(t\mid Z,Z')=\int\rho(t+y\mid Z)\rho(y\mid Z')\,\mathrm{d}y ρ Δ ( t ∣ Z , Z ′ ) = ∫ ρ ( t + y ∣ Z ) ρ ( y ∣ Z ′ ) d y
is bounded by C ρ C_{\rho} C ρ and, under the density derivative bound, has derivative bounded by C ρ , 1 C_{\rho,1} C ρ , 1 . Every projection of Δ Z \Delta Z Δ Z has sub-Gaussian norm at most 2 C Z 2C_Z 2 C Z . Thus an independent pair satisfies the crossing and strict-bracket moments in the proof of Theorem 3, with f ( x ) = ∣ x ∣ f(x)=|x| f ( x ) = ∣ x ∣ . Expected loss differences are defined as in Section 4.
To handle the dependent pairs, put U n Q = ( n 2 ) − 1 ∑ i < j Q ( X i , X j ) U_nQ=\binom{n}{2}^{-1}\sum_{i<j}Q(X_i,X_j) U n Q = ( 2 n ) − 1 ∑ i < j Q ( X i , X j ) , P Q = E Q ( X , X ′ ) PQ=\mathbb{E}Q(X,X') P Q = E Q ( X , X ′ ) , and m = ⌊ n / 2 ⌋ m=\lfloor n/2\rfloor m = ⌊ n /2 ⌋ for a symmetric scalar kernel Q Q Q . Counting pairs over permutations π \pi π , followed by Jensen’s inequality and independence of disjoint pairs, gives
U n Q = 1 n ! ∑ π 1 m ∑ j = 1 m Q ( X π ( 2 j − 1 ) , X π ( 2 j ) ) , U_nQ=\frac{1}{n!}\sum_{\pi}\frac{1}{m}\sum_{j=1}^{m}Q(X_{\pi(2j-1)},X_{\pi(2j)}), U n Q = n ! 1 π ∑ m 1 j = 1 ∑ m Q ( X π ( 2 j − 1 ) , X π ( 2 j ) ) ,
E exp { λ ( U n Q − P Q ) } ≤ [ E exp { λ ( Q ( X , X ′ ) − P Q ) / m } ] m . \mathbb{E}\exp\{\lambda(U_nQ-PQ)\}\leq\left[\mathbb{E}\exp\{\lambda(Q(X,X')-PQ)/m\}\right]^m. E exp { λ ( U n Q − P Q )} ≤ [ E exp { λ ( Q ( X , X ′ ) − P Q ) / m } ] m .
Hence the independent-pair Bernstein bounds hold with m m m in place of n n n . Nonsymmetric kernels are first symmetrized, preserving their expectations and moment bounds by convexity.
For projected score increments, the crossing moments d k ! C k d\,k!C^k d k ! C k give deviation C { d t / n + t / n } C\{\sqrt{dt/n}+t/n\} C { d t / n + t / n } , where d = ∣ θ 1 − θ 2 ∣ 2 d=|\theta_1-\theta_2|_2 d = ∣ θ 1 − θ 2 ∣ 2 . Use the decision and sphere nets in the proof of Theorem 3. On max i ∣ Z i ∣ 2 ≤ C p log n \max_i|Z_i|_2\le C\sqrt{p\log n} max i ∣ Z i ∣ 2 ≤ C p log n , pairwise differences have at most twice this norm and the strict brackets use twice the width. A bracket of width w w w has independent-pair moments w k ! C k w\,k!C^k w k ! C k , and the corresponding population tail is C n − 10 Cn^{-10} C n − 10 . Applying the displayed permutation bound to these brackets and the net pairs at t = A p log n t=Ap\log n t = A p log n gives (11) for every decision pair and subgradient selection. Projected pairwise scores are sub-Gaussian, giving the pointwise bound C a n C\sqrt{a_n} C a n by the permutation bound and sphere net. Summing the sample-norm, bracket, net-pair, and pointwise failure probabilities gives C n − γ p Cn^{-\gamma p} C n − γ p .
Conditional-density integration gives
H ( θ ) = 2 E [ ρ Δ ( Δ Z T θ ∣ Z , Z ′ ) Δ Z Δ Z T ] . H(\theta)=2\mathbb{E}[\rho_{\Delta}(\Delta Z^{T}\theta\mid Z,Z')\Delta Z\Delta Z^{T}]. H ( θ ) = 2 E [ ρ Δ ( Δ Z T θ ∣ Z , Z ′ ) Δ Z Δ Z T ] .
The density derivative and directional moments bound its operator norm and Lipschitz constant. At deterministic θ ∗ \theta^* θ ∗ the score is unique almost surely, has mean zero, and has kernel second moment at most C p C_p C p . The permutation representation and Jensen’s inequality give E ∣ ∂ L r a n k ( θ ∗ ) ∣ 2 2 ≤ C p / n \mathbb{E}|\partial\mathcal{L}_{\mathrm{rank}}(\theta^*)|_2^2\le C_p/n E ∣ ∂ L rank ( θ ∗ ) ∣ 2 2 ≤ C p / n . These bounds verify the localization and expansion inputs in the proof of Theorem 4. The methods in Theorem 4 and Appendix § 8 .§ 3 , with their initialization requirements and Phase 3 counts, therefore give the claimed optimization accuracy and expansion with leading term − H − 1 ∂ L r a n k ( θ ∗ ) -H^{-1}\partial\mathcal{L}_{\mathrm{rank}}(\theta^*) − H − 1 ∂ L rank ( θ ∗ ) .
To identify this term, let k 1 ( X ) = E [ k ( X , X ′ ) ∣ X ] k_1(X)=\mathbb{E}[k(X,X')\mid X] k 1 ( X ) = E [ k ( X , X ′ ) ∣ X ] and k 2 ( X , X ′ ) = k ( X , X ′ ) − k 1 ( X ) − k 1 ( X ′ ) k_2(X,X')=k(X,X')-k_1(X)-k_1(X') k 2 ( X , X ′ ) = k ( X , X ′ ) − k 1 ( X ) − k 1 ( X ′ ) . Since E [ k 2 ( X , X ′ ) ∣ X ] = 0 \mathbb{E}[k_2(X,X')\mid X]=0 E [ k 2 ( X , X ′ ) ∣ X ] = 0 , the Hoeffding decomposition and orthogonality of degenerate pairs, including pairs sharing one observation, give
∂ L r a n k ( θ ∗ ) = 1 n ∑ i = 1 n ψ ( X i ) + D n , D n = U n k 2 . \partial\mathcal{L}_{\mathrm{rank}}(\theta^*)=\frac{1}{n}\sum_{i=1}^{n}\psi(X_i)+D_n,\qquad D_n=U_nk_2. ∂ L rank ( θ ∗ ) = n 1 i = 1 ∑ n ψ ( X i ) + D n , D n = U n k 2 .
E ∣ D n ∣ 2 2 = 2 n ( n − 1 ) E ∣ k 2 ( X , X ′ ) ∣ 2 2 ≤ C p / n 2 . \mathbb{E}|D_n|_2^2=\frac{2}{n(n-1)}\mathbb{E}|k_2(X,X')|_2^2\le C_p/n^2. E ∣ D n ∣ 2 2 = n ( n − 1 ) 2 E ∣ k 2 ( X , X ′ ) ∣ 2 2 ≤ C p / n 2 .
Here ∣ k ∣ 2 ≤ ∣ Δ Z ∣ 2 |k|_2\le|\Delta Z|_2 ∣ k ∣ 2 ≤ ∣Δ Z ∣ 2 and E ∣ Δ Z ∣ 2 2 ≤ C p \mathbb{E}|\Delta Z|_2^2\le C_p E ∣Δ Z ∣ 2 2 ≤ C p . Thus ∣ D n ∣ 2 = O P ( p / n ) |D_n|_2=O_P(\sqrt{p}/n) ∣ D n ∣ 2 = O P ( p / n ) is negligible relative to the expansion remainder. Conditional Jensen’s inequality makes v n T H − 1 ψ ( X ) v_n^TH^{-1}\psi(X) v n T H − 1 ψ ( X ) sub-Gaussian. Its mean is zero, and a variance bounded below gives Lyapunov’s condition. The dimension condition in the example makes the expansion remainder o P ( n − 1 / 2 ) o_P(n^{-1/2}) o P ( n − 1/2 ) , proving the contrast limit by Slutsky’s theorem. For fixed dimension, the multivariate CLT gives covariance H − 1 Var { ψ ( X ) } H − 1 H^{-1}\operatorname{Var}\{\psi(X)\}H^{-1} H − 1 Var { ψ ( X )} H − 1 .
Heavier tails
Corollary 13. For parts (i)–(iii), consider i.i.d. observations within each n n n and either the residual objective (10) under Assumptions 1, 2(i), and 3, or the index objective (63) under Assumptions 3, 5, and 6(i). Write V = Z V=Z V = Z for the residual objective and V = X V=X V = X for the index objective. In parts (i) and (ii), the index objective also satisfies one of cases (a)–(c) in Proposition 6, with its sub-Gaussian assumption replaced by the displayed tail assumption.
(i) Suppose sup ∣ v ∣ 2 = 1 ∥ v T V ∥ ψ α ≤ C V \sup_{|v|_2=1}\|v^TV\|_{\psi_\alpha}\le C_V sup ∣ v ∣ 2 = 1 ∥ v T V ∥ ψ α ≤ C V for some α > 0 \alpha>0 α > 0 . Put
α ˉ = min { α , 1 } , u n = ( p log n ) 1 / α ˉ n , π n = C n − γ p . \bar{\alpha}=\min\{\alpha,1\},\qquad u_n=\frac{(p\log n)^{1/\bar{\alpha}}}{n},\qquad\pi_n=Cn^{-\gamma p}. α ˉ = min { α , 1 } , u n = n ( p log n ) 1/ α ˉ , π n = C n − γ p . (ii) Suppose sup ∣ v ∣ 2 = 1 ∥ v T V ∥ ψ α ≤ C V \sup_{|v|_2=1}\|v^TV\|_{\psi_\alpha}\le C_V sup ∣ v ∣ 2 = 1 ∥ v T V ∥ ψ α ≤ C V for 0 < α < 1 0<\alpha<1 0 < α < 1 . For every M > 0 M>0 M > 0 , choose C M C_M C M sufficiently large and put
B n = C M p + log n ( log n ) 1 / α − 1 / 2 , u n = B n a n , π n = C n − M + C n − γ p . B_n=C_M\sqrt{p+\log n}\,(\log n)^{1/\alpha-1/2},\qquad u_n=B_na_n,\qquad\pi_n=Cn^{-M}+Cn^{-\gamma p}. B n = C M p + log n ( log n ) 1/ α − 1/2 , u n = B n a n , π n = C n − M + C n − γ p .
(iii) Suppose ( V i , R i ) (V_i,R_i) ( V i , R i ) are i.i.d. within each n n n , R ≥ 1 R \ge1 R ≥ 1 , and, for some β > 0 \beta> 0 β > 0 ,
P ( R > t ) ≤ C exp ( − c t β ) , sup ∣ v ∣ 2 = 1 ∥ v T V ∣ R = r ∥ ψ 2 ≤ C r , r ≥ 1. \mathbb{P}(R>t) \le C\exp(-ct^\beta), \qquad\sup_{|v|_2=1}\lVert v^\mathsf{T}V\mid R=r\rVert_{\psi_2}\le Cr,\qquad r\ge1. P ( R > t ) ≤ C exp ( − c t β ) , ∣ v ∣ 2 = 1 sup ∥ v T V ∣ R = r ∥ ψ 2 ≤ C r , r ≥ 1.
For the residual objective, suppose the conditional density of Y Y Y given ( Z , R ) (Z,R) ( Z , R ) is bounded. For the index objective, suppose the conditional density of X 1 X_1 X 1 given ( X − 1 , R ) (X_{-1},R) ( X − 1 , R ) is bounded and either ν \nu ν has compact support or this conditional density has a bounded continuous first derivative. All conditional bounds are uniform in r r r . For every M > 0 M>0 M > 0 , choose C M C_M C M sufficiently large and put
τ n = ( C M log n ) 1 / β , u n = τ n a n , π n = C n − M + C n − γ p . \tau_n=(C_M\log n)^{1/\beta},\qquad u_n=\tau_n a_n,\qquad\pi_n=Cn^{-M}+Cn^{-\gamma p}. τ n = ( C M log n ) 1/ β , u n = τ n a n , π n = C n − M + C n − γ p .
Under any of parts (i)–(iii), suppose u n → 0 u_n\to0 u n → 0 . For each deterministic θ 0 ∈ Θ \theta_0\in\Theta θ 0 ∈ Θ , there is an event of probability at least 1 − π n 1-\pi_n 1 − π n on which the following bounds hold simultaneously for every endpoint pair in Θ \Theta Θ and every selection of scenario subgradients.
∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ‾ ( θ 1 ) + ∇ L ‾ ( θ 2 ) ∣ 2 ≤ C { ∣ θ 1 − θ 2 ∣ 2 a n + u n } , \left\lvert\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)-\nabla\overline{\mathcal{L}}(\theta_1)+\nabla\overline{\mathcal{L}}(\theta_2)\right\rvert_2\le C\left\{\sqrt{|\theta_1-\theta_2|_2a_n+u_n}\right\}, ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ( θ 1 ) + ∇ L ( θ 2 ) 2 ≤ C { ∣ θ 1 − θ 2 ∣ 2 a n + u n } ,
∣ ∂ L ( θ 0 ) − ∇ L ‾ ( θ 0 ) ∣ 2 ≤ C { a n + u n } . \left\lvert\partial\mathcal{L}(\theta_0)-\nabla\overline{\mathcal{L}}(\theta_0)\right\rvert_2\le C\left\{\sqrt{a_n}+u_n\right\}. ∂ L ( θ 0 ) − ∇ L ( θ 0 ) 2 ≤ C { a n + u n } .
Parts 1 and 2 of Corollary 2 hold with ϵ j = C j u n \epsilon_j=C_ju_n ϵ j = C j u n , j = 1 , 2 j=1,2 j = 1 , 2 . Under the Hessian conditions of its part 3, if B ( θ ∗ , C 3 u n ) ⊆ Θ B(\theta^*,C_3\sqrt{u_n})\subseteq\Theta B ( θ ∗ , C 3 u n ) ⊆ Θ , that conclusion holds with radius C 3 u n C_3\sqrt{u_n} C 3 u n and remainder C 4 u n 3 / 4 C_4u_n^{3/4} C 4 u n 3/4 . For the residual objective, a bounded continuous conditional density derivative gives the population Hessian Lipschitz property for every finite convex Lipschitz loss. For the index objective, this verification holds in cases (a) and (b) in parts (i) and (ii), and with compactly supported ν \nu ν in part (iii). In part (iii), these derivative bounds are conditional on R R R and uniform in r r r .
Suppose the population Hessian is Lipschitz near an interior minimizer θ ∗ \theta^* θ ∗ and m H I ⪯ H = ∇ 2 L ( θ ∗ ) ⪯ M H I m\mathbf{H}\mathbf{I}\preceq H=\nabla^2\mathcal{L}(\theta^*)\preceq M\mathbf{H}\mathbf{I} m HI ⪯ H = ∇ 2 L ( θ ∗ ) ⪯ M HI . Use a procedure from Theorem 4 or its unprojected extension, with its region and initialization requirements and ϵ j = C j u n \epsilon_j=C_ju_n ϵ j = C j u n . On the same event, an SAA solution θ ^ \widehat{\theta} θ exists, every iterate after ⌈ c log n ⌉ \lceil c\log n\rceil ⌈ c log n ⌉ updates has distance at most C u n Cu_n C u n from it, and the population objective gaps of the SAA solution and these iterates are at most C u n Cu_n C u n . Every such output satisfies
θ ~ − θ ∗ = − H − 1 1 n ∑ i = 1 n ψ i + O P { ( p / n ) 3 / 4 log n + u n } . \widetilde{\theta}-\theta^*=-H^{-1}\frac{1}{n}\sum_{i=1}^{n}\psi_i+O_P\left\{(p/n)^{3/4}\sqrt{\log n}+u_n\right\}. θ − θ ∗ = − H − 1 n 1 i = 1 ∑ n ψ i + O P { ( p / n ) 3/4 log n + u n } .
where ψ i = b − Z i f + ′ ( Y i − Z i T θ ∗ ) \psi_i=b-Z_i f'_+(Y_i-Z_i^\mathsf{T}\theta^*) ψ i = b − Z i f + ′ ( Y i − Z i T θ ∗ ) for the residual objective and ψ i = b + X i f + ′ ( X i T θ ∗ ) \psi_i=b+X_i f'_+(X_i^\mathsf{T}\theta^*) ψ i = b + X i f + ′ ( X i T θ ∗ ) for the index objective. The two complete procedures give the same bounds with their local Phase 3 refinement calibrated at u n u_n u n . If n u n → 0 \sqrt{n}u_n\to0 n u n → 0 , then n ∣ θ ~ − θ ^ ∣ 2 = o P ( 1 ) \sqrt{n}\lvert\widetilde{\theta}-\widehat{\theta}\rvert_2=o_P(1) n ∣ θ − θ ∣ 2 = o P ( 1 ) . For a deterministic unit contrast whose variance in Theorem 4 is bounded below, its normal limit holds under
p = o { n 1 / 3 ( log n ) − 2 / 3 } , n u n → 0 , in part (i) , p=o\{n^{1/3}(\log n)^{-2/3}\},\qquad\sqrt{n}u_n\to0,\qquad\text{in part (i)}, p = o { n 1/3 ( log n ) − 2/3 } , n u n → 0 , in part (i) ,
p = o { n 1 / 3 ( log n ) − ( 2 / α + 1 ) / 3 } , in part (ii) , p=o\{n^{1/3}(\log n)^{-(2/\alpha+1)/3}\},\qquad\text{in part (ii)}, p = o { n 1/3 ( log n ) − ( 2/ α + 1 ) /3 } , in part (ii) ,
p = o { n 1 / 3 ( log n ) − 2 / 3 } , in part (iii) . p=o\{n^{1/3}(\log n)^{-2/3}\},\qquad\text{in part (iii)}. p = o { n 1/3 ( log n ) − 2/3 } , in part (iii) .
In part (ii), p ≤ n 1 / 3 p\le n^{1/3} p ≤ n 1/3 also gives remainder O P { ( p / n ) 3 / 4 ( log n ) 1 / α + 1 / 2 } O_P\{(p/n)^{3/4}(\log n)^{1/\alpha+1/2}\} O P {( p / n ) 3/4 ( log n ) 1/ α + 1/2 } . The mixtures in part (iii) satisfy the directional ψ α \psi_\alpha ψ α bound with α = 2 β / ( β + 2 ) \alpha=2\beta/(\beta+2) α = 2 β / ( β + 2 ) . For the index objective they include X = R G X=RG X = R G , where G ∼ N ( 0 , Σ ) G\sim N(0,\Sigma) G ∼ N ( 0 , Σ ) is independent of R R R and c I ⪯ Σ ⪯ C I c\mathbf{I}\preceq\Sigma\preceq C\mathbf{I} c I ⪯ Σ ⪯ C I .
(iv) Consider the residual objective (10) with i.i.d. observations, p > n ≥ 2 p > n \ge2 p > n ≥ 2 , a convex bounded region Θ \Theta Θ , a bounded conditional density of Y Y Y given Z Z Z , and a finite convex Lipschitz loss. Fix a deterministic θ ∗ ∈ Θ \theta^* \in\Theta θ ∗ ∈ Θ and suppose
max j ≤ p ∥ Z j ∥ ψ α ≤ C X , Λ max { E ( Z Z T ) } ≤ C Σ , α > 0. \max_{j \le p} \|Z_j\|_{\psi_\alpha} \le C_X,\qquad\Lambda_{\max}\{\mathbb{E}(ZZ^{\mathsf T})\} \le C_\Sigma,\qquad\alpha> 0. j ≤ p max ∥ Z j ∥ ψ α ≤ C X , Λ m a x { E ( Z Z T )} ≤ C Σ , α > 0.
Write m = s ~ m = \tilde{s} m = s ~ and use a s a_s a s from Section 6. For every γ > 0 \gamma> 0 γ > 0 , choose C τ C_\tau C τ sufficiently large and put τ = C τ { log ( n p ) } 1 / α \tau= C_\tau\{\log(np)\}^{1/\alpha} τ = C τ { log ( n p ) } 1/ α . For sufficiently large n n n , there is an event of probability at least 1 − C p − γ 1-Cp^{-\gamma} 1 − C p − γ on which the following conclusions hold.
(a) Uniformly over endpoint pairs in Θ \Theta Θ with joint support of size at most 2 m 2m 2 m , and over every selection of scenario subgradients,
∥ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ‾ ( θ 1 ) + ∇ L ‾ ( θ 2 ) ∥ ∞ ≤ C { D a s + τ a s } , D = ∣ θ 1 − θ 2 ∣ 2 . \left\|\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)-\nabla\overline{\mathcal{L}}(\theta_1)+\nabla\overline{\mathcal{L}}(\theta_2)\right\|_\infty\le C\{\sqrt{Da_s}+\tau a_s\},\qquad D=|\theta_1-\theta_2|_2. ∂ L ( θ 1 ) − ∂ L ( θ 2 ) − ∇ L ( θ 1 ) + ∇ L ( θ 2 ) ∞ ≤ C { D a s + τ a s } , D = ∣ θ 1 − θ 2 ∣ 2 .
At θ ∗ \theta^* θ ∗ ,
∣ ∂ L ( θ ∗ ) − ∇ L ‾ ( θ ∗ ) ∣ ∞ ≤ C { log p / n + τ log p / n } . \left|\partial\mathcal{L}(\theta^*)-\nabla\overline{\mathcal{L}}(\theta^*)\right|_\infty\le C\{\sqrt{\log p/n}+\tau\log p/n\}. ∂ L ( θ ∗ ) − ∇ L ( θ ∗ ) ∞ ≤ C { log p / n + τ log p / n } .
(b) Condition 1-R holds with L = C m τ 2 L=C\sqrt{m}\tau^2 L = C m τ 2 and ϵ 1 = C a s \epsilon_1=Ca_s ϵ 1 = C a s . If L ‾ \overline{\mathcal{L}} L is μ 0 \mu_0 μ 0 -strongly convex on a convex set Θ s c ⊆ Θ \Theta_{\mathrm{sc}}\subseteq\Theta Θ sc ⊆ Θ , then Condition 2-R holds there with μ = μ 0 \mu=\mu_0 μ = μ 0 and ϵ 2 = C m τ 3 a s \epsilon_2=Cm\tau^3a_s ϵ 2 = C m τ 3 a s .
(c) Suppose θ ∗ ∈ Θ r e s \theta^*\in\Theta_{\mathrm{res}} θ ∗ ∈ Θ res , a s → 0 a_s\to0 a s → 0 , and, for all θ ∈ Θ r e s \theta\in\Theta_{\mathrm{res}} θ ∈ Θ res with ∣ θ − θ ∗ ∣ 2 ≤ C 3 a s |\theta-\theta^*|_2\le C_3\sqrt{a_s} ∣ θ − θ ∗ ∣ 2 ≤ C 3 a s ,
∣ ∇ L ‾ ( θ ) − ∇ L ‾ ( θ ∗ ) − H ( θ − θ ∗ ) ∣ ∞ ≤ C 0 ∣ θ − θ ∗ ∣ 2 2 , \left|\nabla\overline{\mathcal{L}}(\theta)-\nabla\overline{\mathcal{L}}(\theta^*)-H(\theta-\theta^*)\right|_\infty\le C_0|\theta-\theta^*|_2^2, ∇ L ( θ ) − ∇ L ( θ ∗ ) − H ( θ − θ ∗ ) ∞ ≤ C 0 ∣ θ − θ ∗ ∣ 2 2 ,
where m Π H ⪯ H ⪯ M Π H m_\Pi H \preceq H\preceq M_\Pi H m Π H ⪯ H ⪯ M Π H . Then Condition 3-R holds with ϵ 3 = C 3 a s \epsilon_3=C_3\sqrt{a_s} ϵ 3 = C 3 a s and r n = C τ 3 / 2 a s 3 / 4 r_n=C\tau^{3/2}a_s^{3/4} r n = C τ 3/2 a s 3/4 .
For the index objective under Assumptions 3, 5, and 6(i), impose the displayed tail and second-moment bounds on X X X . Then part (a) holds with increment bound C { τ 3 / 2 D a s + τ a s } C\{\tau^{3/2}\sqrt{Da_s}+\tau a_s\} C { τ 3/2 D a s + τ a s } , and parts (b) and (c) hold under their population assumptions. Substitution of these parameters into Theorems 7 and 8, under their remaining hypotheses, gives their optimization and expansion bounds.
Proof. Part (i). We give the index calculation first. The residual calculation uses the conditional density of Y Y Y given Z Z Z in each crossing and slab bound. Write g = f + ′ g=f'_+ g = f + ′ and ν = d g \nu=d g ν = d g . The directional condition gives ∥ v T X ∥ L q ≤ C q 1 / α ˉ \|v^{\mathsf T}X\|_{L_q}\le Cq^{1/\bar{\alpha}} ∥ v T X ∥ L q ≤ C q 1/ α ˉ for every unit v v v . Use the reconstructed sections in the proof of Proposition 6. On compact level intervals,
∥ v T x θ ( c ) ∥ L q ≤ C ( ∣ c ∣ + q 1 / α ˉ ) . \|v^{\mathsf T}x_\theta(c)\|_{L_q}\le C(|c|+q^{1/\bar{\alpha}}). ∥ v T x θ ( c ) ∥ L q ≤ C ( ∣ c ∣ + q 1/ α ˉ ) .
In case (c), the marginal density is bounded and Lipschitz. Its value at level c c c persists on an interval whose length is proportional to that value. The directional tail bound consequently gives p θ ( c ) ≤ C exp ( − c 1 ∣ c ∣ α ˉ ) p_\theta(c)\le C\exp(-c_1|c|^{\bar{\alpha}}) p θ ( c ) ≤ C exp ( − c 1 ∣ c ∣ α ˉ ) . Cauchy–Schwarz then bounds a weighted section moment of total degree q q q by
C q + 1 e − c 1 ∣ c ∣ α ˉ / 2 ( ∣ c ∣ + q 1 / α ˉ ) q ≤ C q + 1 q q / α ˉ . C^{q+1}e^{-c_1|c|^{\bar{\alpha}}/2}(|c|+q^{1/\bar{\alpha}})^q\le C^{q+1}q^{q/\bar{\alpha}}. C q + 1 e − c 1 ∣ c ∣ α ˉ /2 ( ∣ c ∣ + q 1/ α ˉ ) q ≤ C q + 1 q q / α ˉ .
These bounds hold at every level in case (c), on the required compact level interval in case (b), and with bounded sections in case (a). For W v = v T X { g ( X T θ 1 ) − g ( X T θ 0 ) } W_v=v^{\mathsf T}X\{g(X^{\mathsf T}\theta_1)-g(X^{\mathsf T}\theta_0)\} W v = v T X { g ( X T θ 1 ) − g ( X T θ 0 )} and d = ∣ θ 1 − θ 0 ∣ 2 d=|\theta_1-\theta_0|_2 d = ∣ θ 1 − θ 0 ∣ 2 , integration along the threshold crossing and against ν \nu ν gives
E ∣ W v ∣ ≤ C d , E ∣ W v ∣ k ≤ d C k k k / α ˉ , k ≥ 2. \mathbb{E}|W_v|\le Cd,\qquad\mathbb{E}|W_v|^k\le dC^k k^{k/\bar{\alpha}},\qquad k\ge2. E ∣ W v ∣ ≤ C d , E ∣ W v ∣ k ≤ d C k k k / α ˉ , k ≥ 2.
For residual objectives, these inequalities follow by conditioning on Z Z Z . A slab of width w w w has the same moment bounds with d d d replaced by w w w .
Write Y i = W i , v − E W i , v Y_i = W_{i,v} - \mathbb{E}W_{i,v} Y i = W i , v − E W i , v and let q ≥ 2 q \ge2 q ≥ 2 be even. For 2 ≤ k ≤ q 2 \le k \le q 2 ≤ k ≤ q , the preceding moments imply
E ∣ Y i ∣ k ≤ k ! 2 V B q k − 2 , V = C d , B q = C q 1 / α ˉ − 1 . \mathbb{E}|Y_i|^k \le\frac{k!}{2}VB_q^{k-2}, \qquad V = Cd,\quad B_q = Cq^{1/\bar{\alpha}-1}. E ∣ Y i ∣ k ≤ 2 k ! V B q k − 2 , V = C d , B q = C q 1/ α ˉ − 1 .
Indeed, k k / α ˉ / k ! ≤ C k k k ( 1 / α ˉ − 1 ) k^{k/\bar{\alpha}}/k! \le C^k k^{k(1/\bar{\alpha}-1)} k k / α ˉ / k ! ≤ C k k k ( 1/ α ˉ − 1 ) , and the factor k 2 ( 1 / α ˉ − 1 ) k^{2(1/\bar{\alpha}-1)} k 2 ( 1/ α ˉ − 1 ) is absorbed into C k C^k C k . Expansion of the q q q th moment and comparison of nonnegative power-series coefficients give
E ∣ ∑ i Y i ∣ q ≤ q ! [ z q ] exp { n V z 2 2 ( 1 − B q z ) } . \mathbb{E}\left|\sum_i Y_i\right|^q \le q![z^q]\exp\left\{\frac{nVz^2}{2(1-B_qz)}\right\}. E i ∑ Y i q ≤ q ! [ z q ] exp { 2 ( 1 − B q z ) nV z 2 } .
If V = 0 V=0 V = 0 , every summand vanishes almost surely. Otherwise, for 0 < z < 1 / B q 0<z<1/B_q 0 < z < 1/ B q , the coefficient is at most z − q z^{-q} z − q times the displayed exponential. Taking z = min { q / ( 2 n V ) , 1 / ( 2 B q ) } z=\min\{\sqrt{q/(2nV)},1/(2B_q)\} z = min { q / ( 2 nV ) , 1/ ( 2 B q )} gives
∥ ∑ i Y i ∥ L q ≤ C { n q d + q 1 / α ˉ } . \left\|\sum_i Y_i\right\|_{L^q} \le C\{\sqrt{nqd}+q^{1/\bar{\alpha}}\}. i ∑ Y i L q ≤ C { n q d + q 1/ α ˉ } .
Markov’s inequality with even q ∈ [ t , t + 2 ] q\in[t,t+2] q ∈ [ t , t + 2 ] gives, for t ≥ 2 t\ge2 t ≥ 2 ,
P { ∣ ( P n − P ) W v ∣ > C ( d t / n + t 1 / α ˉ / n ) } ≤ 2 e − t . \mathbb{P}\{|(P_n-P)W_v|>C(\sqrt{dt/n}+t^{1/\bar{\alpha}}/n)\}\le2e^{-t}. P { ∣ ( P n − P ) W v ∣ > C ( d t / n + t 1/ α ˉ / n )} ≤ 2 e − t .
The slab variables satisfy this bound, and the pointwise projected score satisfies it with d d d replaced by one.
Take the decision and sphere nets from the proof of Theorem 3, with decision mesh n − 12 n^{-12} n − 12 and logarithmic cardinalities O ( p log n ) O(p\log n) O ( p log n ) . A sphere net and the directional tail bound give
P { max i ≤ n ∣ X i ∣ 2 > C ( p log n ) 1 / α ˉ } ≤ n − γ p . \mathbb{P}\left\{\max_{i\le n}|X_i|_2>C(p\log n)^{1/\bar{\alpha}}\right\}\le n^{-\gamma p}. P { i ≤ n max ∣ X i ∣ 2 > C ( p log n ) 1/ α ˉ } ≤ n − γ p .
Increasing the constant makes the corresponding population tail in any unit projection negligible. Since ( p log n ) 1 / α ˉ = n u n = o ( n ) (p\log n)^{1/\bar{\alpha}}=nu_n=o(n) ( p log n ) 1/ α ˉ = n u n = o ( n ) , the strict-bracket width is o ( n − 10 ) o(n^{-10}) o ( n − 10 ) . Applying the slab and pairwise bounds with t = A p log n t=Ap\log n t = A p log n over the nets gives bracket error C u n Cu_n C u n . The representative pair has distance at most d + 2 n − 12 d+2n^{-12} d + 2 n − 12 . The sphere-net inequality proves the increment bound for all endpoint pairs and subgradient selections. Applying the score bound on the sphere net proves the pointwise claim.
Part (ii). Let G ∼ N ( 0 , I p ) G\sim\mathcal{N}(0,I_p) G ∼ N ( 0 , I p ) be independent of X X X and let G 1 ∼ N ( 0 , 1 ) G_1\sim\mathcal{N}(0,1) G 1 ∼ N ( 0 , 1 ) . Gaussian integration and the directional moment bound give, for q ≥ 2 q\ge2 q ≥ 2 ,
E ∣ G 1 ∣ q E ∣ X ∣ 2 2 = E ∣ G T X ∣ q ≤ C q q q / α E ∣ G ∣ 2 q . \mathbb{E}|G_1|^q\mathbb{E}|X|_2^2=\mathbb{E}|G^\mathsf{T}X|^q\le C^q q^{q/\alpha}\mathbb{E}|G|_2^q. E ∣ G 1 ∣ q E ∣ X ∣ 2 2 = E ∣ G T X ∣ q ≤ C q q q / α E ∣ G ∣ 2 q .
Since ∥ G 1 ∥ L q ≥ c q \|G_1\|_{L^q}\ge c\sqrt{q} ∥ G 1 ∥ L q ≥ c q and ∥ ∣ G ∣ 2 ∥ L q ≤ C p + q \||G|_2\|_{L^q}\le C\sqrt{p+q} ∥∣ G ∣ 2 ∥ L q ≤ C p + q ,
∥ ∣ X ∣ 2 ∥ L q ≤ C p + q q 1 / α − 1 / 2 . \||X|_2\|_{L^q}\le C\sqrt{p+q}\,q^{1/\alpha-1/2}. ∥∣ X ∣ 2 ∥ L q ≤ C p + q q 1/ α − 1/2 .
Markov’s inequality with q = C 1 log n q=C_1\log n q = C 1 log n and a union bound give P { max i ∣ X i ∣ 2 > B n } ≤ C n − M \mathbb{P}\{\max_i|X_i|_2>B_n\}\le Cn^{-M} P { max i ∣ X i ∣ 2 > B n } ≤ C n − M . Increasing C M C_M C M and using bounded directional second moments also give
sup ∣ v ∣ 2 = 1 E [ ∣ v T X ∣ I { ∣ X ∣ 2 > B n } ] ≤ C n − M . \sup_{|v|_2=1}\mathbb{E}[|v^\mathsf{T}X|\mathbb{I}\{|X|_2>B_n\}]\le Cn^{-M}. ∣ v ∣ 2 = 1 sup E [ ∣ v T X ∣ I { ∣ X ∣ 2 > B n }] ≤ C n − M .
Set A n = { ∣ X ∣ 2 ≤ B n } A_n=\{|X|_2\le B_n\} A n = { ∣ X ∣ 2 ≤ B n } and weight each projected increment by I A n \mathbb{I}_{A_n} I A n . Evaluate this indicator at the reconstructed vector in the crossing identity, taking expectations under the original law. The section moments from part (i) give crossing first moment and variance C d Cd C d and slab first moment and variance C w Cw C w , while the envelopes are C B n CB_n C B n . Bernstein’s inequality gives fixed-pair deviation C { d t / n + B n t / n } C\{\sqrt{dt/n}+B_nt/n\} C { d t / n + B n t / n } and the analogous slab bound.
Apply the decision and sphere nets from part (i) at t = A p log n t=Ap\log n t = A p log n . Their mesh n − 12 n^{-12} n − 12 gives bracket width 2 B n n − 12 = o ( a n ) 2B_nn^{-12}=o(a_n) 2 B n n − 12 = o ( a n ) , since B n a n → 0 B_na_n\to0 B n a n → 0 , and interpolation error C B n a n = C u n CB_na_n=Cu_n C B n a n = C u n . On the sample norm event the weighted and original empirical fields agree. Their population difference is uniformly C n − M Cn^{-M} C n − M , which is absorbed into u n u_n u n by increasing the tail exponent. The pointwise score has bounded projected variance and envelope C B n CB_n C B n , giving its bound by the same sphere-net Bernstein argument. The total failure probability is C n − M + C n − γ p Cn^{-M}+Cn^{-\gamma p} C n − M + C n − γ p . For the residual objective use Z Z Z and conditional response-density integration.
Part (iii). Let A n = { R ≤ τ n } A_n=\{R\leq\tau_n\} A n = { R ≤ τ n } and P τ = P ( ⋅ ∣ A n ) P_\tau=P(\mathord\cdot\mid A_n) P τ = P ( ⋅ ∣ A n ) . Conditional on R = r R=r R = r , the reconstructed sections satisfy
∥ v T x θ ( c ) ∥ L q ( P ∣ R = r ) ≤ C ( ∣ c ∣ + r q ) . \lVert v^{\mathsf{T}}x_\theta(c)\rVert_{L_q(P\mid R=r)}\leq C(|c|+r\sqrt{q}). ∥ v T x θ ( c ) ∥ L q ( P ∣ R = r ) ≤ C ( ∣ c ∣ + r q ) .
On compact level intervals this is at most C r q Cr\sqrt{q} C r q . Under the conditional density derivative bound, the density-tail argument in Proposition 6 gives p θ , r ( c ) ≤ C exp ( − c 1 c 2 / r 2 ) p_{\theta,r}(c)\leq C\exp(-c_1c^2/r^2) p θ , r ( c ) ≤ C exp ( − c 1 c 2 / r 2 ) . Cauchy–Schwarz then bounds a weighted section moment of total degree q q q by
C q + 1 e − c 1 c 2 / ( 2 r 2 ) ( ∣ c ∣ + r q ) q ≤ C q + 1 r q q q / 2 . C^{q+1}e^{-c_1c^2/(2r^2)}(|c|+r\sqrt{q})^q\leq C^{q+1}r^qq^{q/2}. C q + 1 e − c 1 c 2 / ( 2 r 2 ) ( ∣ c ∣ + r q ) q ≤ C q + 1 r q q q /2 .
For W v W_v W v as in part (i), integration against ν \nu ν gives
E ( ∣ W v ∣ ∣ R = r ) ≤ C d r 2 , E ( ∣ W v ∣ k ∣ R = r ) ≤ d k ! C r k + 1 , k ≥ 2. \mathbb{E}(|W_v|\mid R=r)\leq Cdr^2,\qquad\mathbb{E}(|W_v|^k\mid R=r)\leq dk!C_r^{k+1},\qquad k\geq2. E ( ∣ W v ∣ ∣ R = r ) ≤ C d r 2 , E ( ∣ W v ∣ k ∣ R = r ) ≤ d k ! C r k + 1 , k ≥ 2.
For residual objectives, conditioning on ( Z , R ) (Z,R) ( Z , R ) and integrating the bounded response density gives the same inequalities without a response moment assumption. Every fixed moment of R R R is finite, and P ( A n ) ≥ 1 / 2 P(A_n)\geq1/2 P ( A n ) ≥ 1/2 eventually. Hence
E τ R k + 1 ≤ 2 τ n k − 2 E R 3 , k ≥ 2. \mathbb{E}_\tau R^{k+1}\leq2\tau_n^{k-2}\mathbb{E}R^3,\qquad k\geq2. E τ R k + 1 ≤ 2 τ n k − 2 E R 3 , k ≥ 2.
The centered moments satisfy Lemma 6 with variance parameter C d Cd C d and scale C τ n C\tau_n C τ n . Thus
P τ { ∣ ( P n − P τ ) W v ∣ > C ( d t / n + τ n t / n ) } ≤ 2 e − t . P_\tau\{|(P_n-P_\tau)W_v|>C(\sqrt{dt/n}+\tau_nt/n)\}\leq2e^{-t}. P τ { ∣ ( P n − P τ ) W v ∣ > C ( d t / n + τ n t / n )} ≤ 2 e − t .
A level slab of width w w w has conditional k k k th moment at most w k ! C r k r k wk!C_r^kr^k w k ! C r k r k . The bound E τ R k ≤ 2 τ n k − 2 E R 2 \mathbb{E}_\tau R^k\leq2\tau_n^{k-2}\mathbb{E}R^2 E τ R k ≤ 2 τ n k − 2 E R 2 gives variance parameter C w Cw C w and scale C τ n C\tau_n C τ n , and its first moment is at most C w E R Cw\mathbb{E}R C w E R . At a deterministic decision, the projected score has bounded variance and scale C τ n C\tau_n C τ n by the same argument.
Under P τ P_\tau P τ , apply the nets from part (i) with norm cutoff C τ n p log n C\tau_n\sqrt{p\log n} C τ n p log n and Bernstein scale C τ n C\tau_n C τ n . Conditional sub-Gaussian tails give failure C n − γ p Cn^{-\gamma p} C n − γ p and a negligible population interpolation tail. The width C n − 12 τ n p log n = o ( a n ) Cn^{-12}\tau_n\sqrt{p\log n}=o(a_n) C n − 12 τ n p log n = o ( a n ) and t = A p log n t=Ap\log n t = A p log n give bracket error C τ n a n C\tau_na_n C τ n a n and increment error C { d a n + τ n a n } C\{\sqrt{d a_n}+\tau_na_n\} C { d a n + τ n a n } for all endpoint pairs and selections, with conditional failure C n − γ p Cn^{-\gamma p} C n − γ p .
Choose C M C_M C M sufficiently large that
n P ( R > τ n ) ≤ C n − M , E [ R j I { R > τ n } ] ≤ C n − M , j = 1 , 2 , 3. nP(R>\tau_n)\leq Cn^{-M},\qquad\mathbb{E}[R^j\mathbb{I}\{R>\tau_n\}]\leq Cn^{-M},\qquad j=1,2,3. n P ( R > τ n ) ≤ C n − M , E [ R j I { R > τ n }] ≤ C n − M , j = 1 , 2 , 3.
Conditional on A n A_n A n for every observation, the observations remain i.i.d. with law P τ P_\tau P τ . The conditional density and derivative bounds are preserved, since, given the remaining covariates, they are averages of the bounds conditional on R R R . Taking a supremum over unit projections gives
sup θ ∈ Θ ∣ ∇ L ~ τ ( θ ) − ∇ L ‾ ( θ ) ∣ 2 ≤ C E [ R I { R > τ n } ] + C P ( R > τ n ) ≤ C n − M . \sup_{\theta\in\Theta}\lvert\nabla\widetilde{\mathcal{L}}_\tau(\theta)-\nabla\overline{\mathcal{L}}(\theta)\rvert_2\leq C\mathbb{E}[R\mathbb{I}\{R>\tau_n\}]+CP(R>\tau_n)\leq Cn^{-M}. θ ∈ Θ sup ∣ ∇ L τ ( θ ) − ∇ L ( θ ) ∣ 2 ≤ C E [ R I { R > τ n }] + C P ( R > τ n ) ≤ C n − M .
A larger exponent in the tail estimate absorbs this difference into τ n a n \tau_n a_n τ n a n . Removing the conditioning proves the increment and pointwise claims. Finally,
∥ v T V ∥ L q ≤ C q ∥ R ∥ L q ≤ C q 1 / 2 + 1 / β , q ≥ 2 , \|v^{\mathsf T}V\|_{L^q} \le C\sqrt{q}\|R\|_{L^q} \le Cq^{1/2+1/\beta}, \qquad q\ge2, ∥ v T V ∥ L q ≤ C q ∥ R ∥ L q ≤ C q 1/2 + 1/ β , q ≥ 2 ,
which gives the asserted directional ψ 2 β / ( β + 2 ) \psi_{2\beta/(\beta+2)} ψ 2 β / ( β + 2 ) bound. For X = R G X=RG X = R G , the Gaussian conditional density of X 1 X_1 X 1 given ( X − 1 , R ) (X_{-1},R) ( X − 1 , R ) is bounded by C / R ≤ C C/R\le C C / R ≤ C and its derivative by C / R 2 ≤ C C/R^2\le C C / R 2 ≤ C . The covariance bounds also give the required conditional projection bound.
The consequences in parts (i)–(iii). The crossing first moment gives population Lipschitz continuity. The density derivative and moments of degree at most three give the Hessian Lipschitz verifications. The residual calculation uses only ν ( R ) ≤ 2 C f \nu(R)\le2C_f ν ( R ) ≤ 2 C f , while the index calculation uses bounded sections or compact support of ν \nu ν . In part (iii), integrate these bounds over R R R using its finite moments. Since a n ≤ u n a_n\le u_n a n ≤ u n , the proof of Corollary 2 gives ϵ j = C j u n \epsilon_j=C_j u_n ϵ j = C j u n . On a ball of radius C 3 u n C_3\sqrt{u_n} C 3 u n , the increment error is at most C u n 3 / 4 Cu_n^{3/4} C u n 3/4 and the population Taylor error is at most C u n Cu_n C u n , proving its part 3.
At a stationary population solution the score has mean zero, second moment at most C p Cp C p , and bounded third moments in deterministic unit directions. The localization argument gives
δ n : = ∣ θ ^ − θ ∗ ∣ 2 = O P ( p / n + u n ) . \delta_n:=|\widehat{\theta}-\theta^*|_2=O_P\left(\sqrt{p/n}+u_n\right). δ n := ∣ θ − θ ∗ ∣ 2 = O P ( p / n + u n ) .
The increment bound at an SAA solution, Taylor’s formula, and Lemma 2 give
∣ H ( θ ^ − θ ∗ ) + ∂ L ( θ ∗ ) ∣ 2 ≤ C { δ n a n + u n + δ n 2 } = O P { ( p / n ) 3 / 4 log n + u n } . |H(\widehat{\theta}-\theta^*)+\partial\mathcal{L}(\theta^*)|_2
\le C\{\sqrt{\delta_n a_n}+u_n+\delta_n^2\}
=O_P\left\{(p/n)^{3/4}\sqrt{\log n}+u_n\right\}. ∣ H ( θ − θ ∗ ) + ∂ L ( θ ∗ ) ∣ 2 ≤ C { δ n a n + u n + δ n 2 } = O P { ( p / n ) 3/4 log n + u n } .
Corollary 1 and the local recursions in Corollary 10(ii)(c), calibrated at u n u_n u n , give the optimization bound for all later iterates. The local comparison (42) gives the complete procedures’ bound at their Phase 3 iteration rates. On the displayed event, the pointwise bound gives SAA localization C u n C\sqrt{u_n} C u n . Population smoothness and stationarity then bound the population objective gaps by C u n Cu_n C u n . Adding optimization error proves the output expansion, and multiplying that error by n \sqrt{n} n proves first-order equivalence whenever n u n → 0 \sqrt{n}u_n\to0 n u n → 0 .
For the normal limit, the variance lower bound and third moments give Lyapunov’s condition. In part (i), the two stated dimension conditions make the remainder o P ( n − 1 / 2 ) o_P(n^{-1/2}) o P ( n − 1/2 ) . In part (ii), if p ≥ log n p\ge\log n p ≥ log n ,
n u n ≤ C p 3 ( log n ) 2 / α + 1 n ⟶ 0. \sqrt{n}u_n\le C\sqrt{\frac{p^3(\log n)^{2/\alpha+1}}{n}}\longrightarrow0. n u n ≤ C n p 3 ( log n ) 2/ α + 1 ⟶ 0.
For p < log n p<\log n p < log n , the same conclusion follows from a polylogarithmic bound. Its stated dimension condition also makes ( p / n ) 3 / 4 n log n → 0 (p/n)^{3/4}\sqrt{n\log n}\to0 ( p / n ) 3/4 n log n → 0 . In part (iii), since u n = τ n a n u_n=\tau_n a_n u n = τ n a n and τ n \tau_n τ n is a power of log n \log n log n , both remainder terms multiplied by n \sqrt{n} n vanish under p = o { n 1 / 3 ( log n ) − 2 / 3 } p=o\{n^{1/3}(\log n)^{-2/3}\} p = o { n 1/3 ( log n ) − 2/3 } . Slutsky’s theorem proves the limits. If p ≤ n 1 / 3 p\le n^{1/3} p ≤ n 1/3 and p ≥ log n p\ge\log n p ≥ log n in part (ii),
u n ( p / n ) 3 / 4 ≤ C p 3 / 4 n 1 / 4 ( log n ) 1 / α + 1 / 2 ≤ C ( log n ) 1 / α + 1 / 2 . \frac{u_n}{(p/n)^{3/4}}\le C\frac{p^{3/4}}{n^{1/4}}(\log n)^{1/\alpha+1/2}\le C(\log n)^{1/\alpha+1/2}. ( p / n ) 3/4 u n ≤ C n 1/4 p 3/4 ( log n ) 1/ α + 1/2 ≤ C ( log n ) 1/ α + 1/2 .
For p < log n p<\log n p < log n , direct substitution gives the same bound eventually. This proves its additional remainder assertion.
Part (iv). First consider the index objective. Let g = f + ′ g=f'_{+} g = f + ′ , ν = d g \nu=dg ν = d g , A τ = { x : ∣ x ∣ ∞ ≤ τ } \mathcal{A}_{\tau}=\{x:|x|_\infty\le\tau\} A τ = { x : ∣ x ∣ ∞ ≤ τ } , and U = X − 1 U=X_{-1} U = X − 1 . All expectations use the original law with indicator weight 1 A τ \mathbf{1}_{\mathcal{A}_{\tau}} 1 A τ . For any prescribed M > 0 M>0 M > 0 , increasing C τ C_{\tau} C τ gives
Pr { some i ≤ n : X i ∉ A τ } ≤ C p − M , max j E [ ∣ X j ∣ 1 A τ ] ≤ C p − M . \Pr\{\text{some }i\le n:X_i\notin\mathcal{A}_{\tau}\}\le Cp^{-M},
\qquad
\max_j\mathbb{E}[|X_j|\mathbf{1}_{\mathcal{A}_{\tau}}]\le Cp^{-M}. Pr { some i ≤ n : X i ∈ / A τ } ≤ C p − M , j max E [ ∣ X j ∣ 1 A τ ] ≤ C p − M .
The first inequality follows from the coordinate tails and a union bound over n p np n p . The second follows from Cauchy–Schwarz, bounded second moments, and the same tail probability. Take M > max { γ , 12 } M > \max\{\gamma,12\} M > max { γ , 12 } .
Use the sections in Proposition 6. When x θ ( c ) ∈ A τ x_\theta(c) \in A_\tau x θ ( c ) ∈ A τ , every section coordinate is bounded by τ \tau τ . For every unit u u u , the conditional density and second-moment bounds give, at every level c c c ,
E [ ρ θ ( c ∣ U ) I { x θ ( c ) ∈ A τ } ∣ u T x θ ( c ) ∣ ] ≤ C { ∣ u 1 ∣ τ + E ∣ u − 1 T U ∣ } ≤ C τ . \mathbb{E}\left[\rho_\theta(c \mid U)\mathbb{I}\{x_\theta(c) \in A_\tau\}\lvert u^{\mathsf T}x_\theta(c)\rvert\right] \le C\{|u_1|\tau+\mathbb{E}|u_{-1}^{\mathsf T}U|\} \le C\tau. E [ ρ θ ( c ∣ U ) I { x θ ( c ) ∈ A τ } ∣ u T x θ ( c )∣ ] ≤ C { ∣ u 1 ∣ τ + E ∣ u − 1 T U ∣ } ≤ C τ .
For endpoints at distance d d d , apply the crossing formula along their segment with its unit displacement direction and the displayed indicator. Integrating against ν \nu ν gives, for W j τ = X j I A τ { g ( X T θ 1 ) − g ( X T θ 2 ) } W_j^\tau=X_j\mathbb{I}_{A_\tau}\{g(X^{\mathsf T}\theta_1)-g(X^{\mathsf T}\theta_2)\} W j τ = X j I A τ { g ( X T θ 1 ) − g ( X T θ 2 )} ,
E ∣ W j τ ∣ k ≤ d C f k τ k + 1 , k ≥ 2 , ∣ W j τ ∣ ≤ 2 C f τ . \mathbb{E}|W_j^\tau|^k \le dC_f^k\tau^{k+1}, \qquad k \ge2, \qquad|W_j^\tau| \le2C_f\tau. E ∣ W j τ ∣ k ≤ d C f k τ k + 1 , k ≥ 2 , ∣ W j τ ∣ ≤ 2 C f τ .
The variance is at most C τ 3 d C\tau^3d C τ 3 d and the centered envelope at most 4 C f τ 4C_f\tau 4 C f τ . Bernstein’s inequality gives fixed-pair deviation C { τ 3 / 2 d t / n + τ t / n } C\{\tau^{3/2}\sqrt{dt/n}+\tau t/n\} C { τ 3/2 d t / n + τ t / n } with failure probability 2 e − t 2e^{-t} 2 e − t .
Use the sparse decision nets in the bounded-coordinate proof of Theorem 6(1), with mesh p − 12 p^{-12} p − 12 on subspaces of dimension at most 2 m 2m 2 m and logarithmic cardinality O ( m log p ) O(m\log p) O ( m log p ) . On A τ A_\tau A τ the bracket width is w = τ 2 m p − 12 ≤ a s w=\tau\sqrt{2m}\,p^{-12}\le a_s w = τ 2 m p − 12 ≤ a s eventually. Conditional integration gives
E [ { ∣ X j ∣ I A τ [ g ( X T θ + 2 w ) − f ′ ( X T θ − 2 w ) ] } k ] ≤ w C f k τ k . \mathbb{E}\left[\left\{|X_j|\mathbb{I}_{A_\tau}[g(X^{\mathsf T}\theta+2w)-f'(X^{\mathsf T}\theta-2w)]\right\}^k\right]\le wC_f^k\tau^k. E [ { ∣ X j ∣ I A τ [ g ( X T θ + 2 w ) − f ′ ( X T θ − 2 w )] } k ] ≤ w C f k τ k .
At t = A m log p t=Am\log p t = A m log p , the coordinate union bound and strict-bracket interpolation give error C τ a s C\tau a_s C τ a s and cover every selection. Combining the fixed-pair bound with this interpolation proves the weighted increment bound. On { X i ∈ A τ for all i } \{X_i\in A_\tau\text{ for all }i\} { X i ∈ A τ for all i } the weighted and original empirical fields agree, while their population difference is uniformly C p − M Cp^{-M} C p − M and is absorbed into τ a s \tau a_s τ a s . Combining these losses gives failure probability C p − γ Cp^{-\gamma} C p − γ .
At a deterministic decision, each weighted score coordinate has variance at most C f 2 E X j 2 ≤ C C_f^2\mathbb{E}X_j^2\le C C f 2 E X j 2 ≤ C and envelope at most C f τ C_f\tau C f τ . Coordinatewise Bernstein bounds and the same population-tail comparison prove the pointwise claim. The first-moment crossing calculation gives a truncated population increment at most C τ 2 D C\tau^2D C τ 2 D in maximum norm, while the full population increment differs by at most C p − M Cp^{-M} C p − M . Adding the empirical increment bound, separately for D ≥ a s D\ge a_s D ≥ a s and D < a s D<a_s D < a s , gives
∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) ∣ ∞ ≤ C τ 2 max { D , a s } . |\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2)|_\infty\le C\tau^2\max\{D,a_s\}. ∣ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) ∣ ∞ ≤ C τ 2 max { D , a s } .
Conversion to ∣ ⋅ ∣ ( 2 m ) |\cdot|_{(2m)} ∣ ⋅ ∣ ( 2 m ) proves (b)’s smoothness assertion. Under population strong convexity, pairing the increment error with a sparse displacement and integrating gives
μ 0 2 D 2 − C m τ 3 / 2 a s D 3 / 2 − C m τ a s D . \frac{\mu_0}{2}D^2-C\sqrt{m}\tau^{3/2}\sqrt{a_s}D^{3/2}-C\sqrt{m}\tau a_sD. 2 μ 0 D 2 − C m τ 3/2 a s D 3/2 − C m τ a s D .
For D ≥ C m τ 3 a s D\ge Cm\tau^3a_s D ≥ C m τ 3 a s , this is at least μ 0 D 2 / 4 \mu_0D^2/4 μ 0 D 2 /4 , proving restricted strong convexity. Combining (a) with the assumed Taylor bound on D ≤ C 3 a s D\le C_3\sqrt{a_s} D ≤ C 3 a s gives
C { τ 3 / 2 D a s + τ a s + D 2 } ≤ C 1 τ 3 / 2 a s 3 / 4 , C\{\tau^{3/2}\sqrt{Da_s}+\tau a_s+D^2\}\le C_1\tau^{3/2}a_s^{3/4}, C { τ 3/2 D a s + τ a s + D 2 } ≤ C 1 τ 3/2 a s 3/4 ,
which proves (c).
For residual objectives, conditional response-density integration and the tail and second-moment bounds give
E [ Z j 2 ∣ u T Z ∣ ] ≤ ( E Z j 4 ) 1 / 2 ( E ∣ u T Z ∣ 2 ) 1 / 2 ≤ C . \mathbb{E}[Z_j^2|u^{\mathsf T}Z|]\le(\mathbb{E}Z_j^4)^{1/2}(\mathbb{E}|u^{\mathsf T}Z|^2)^{1/2}\le C. E [ Z j 2 ∣ u T Z ∣ ] ≤ ( E Z j 4 ) 1/2 ( E ∣ u T Z ∣ 2 ) 1/2 ≤ C .
The truncated crossing variance is therefore C d Cd C d , while its Bernstein scale remains C τ C\tau C τ . The net and tail argument just given proves the residual increment bound. Its first-moment population bound follows from E [ ∣ Z j ∣ ∣ u T Z ∣ ] ≤ C \mathbb{E}[|Z_j||u^{\mathsf T}Z|]\le C E [ ∣ Z j ∣∣ u T Z ∣ ] ≤ C , and the segment and Taylor calculations prove (b) and (c) with the stated parameters.
Smoothing
Write g = f + ′ g=f_{+}^{\prime} g = f + ′ and ν = d g \nu=dg ν = d g , and let P n P_n P n and P P P denote the empirical average and expectation. Write ∘ ∈ { inf , i n t } \circ\in\{\inf,\mathrm{int}\} ∘ ∈ { inf , int } for infimal convolution in (12) and integral convolution in (78) . In this appendix, the subscript i n f , h \mathrm{inf},h inf , h denotes the infimal convolution written with subscript h h h in Section 5. For the residual objective, impose Assumptions 1–3, together with Assumption 4 for infimal convolution or Assumption 4-int for integral convolution. For the index objective, impose Assumptions 3, 5, and 6, with a bounded continuous first derivative of ρ X \rho_X ρ X . Its infimal-convolution calculations use the conditions on f f f and ω \omega ω in Assumption 4, and its integral-convolution calculations use the conditions on K K K in Assumption 4-int.
For the common moment and concentration arguments, put ( Q θ , V ) = ( Y − Z T θ , Z ) (Q_\theta,V)=(Y-Z^{\mathsf T}\theta,Z) ( Q θ , V ) = ( Y − Z T θ , Z ) for the residual objective and ( Q θ , V ) = ( X T θ , X ) (Q_\theta,V)=(X^{\mathsf T}\theta,X) ( Q θ , V ) = ( X T θ , X ) for the index objective. Write q h = f ∘ , h ′ ′ q_h=f_{\circ,h}^{\prime\prime} q h = f ∘ , h ′′ where this derivative exists. The scenario gradients have opposite signs, and both Hessians equal q h ( Q θ ) V V T q_h(Q_\theta)VV^{\mathsf T} q h ( Q θ ) V V T .
Write U = X − 1 U=X_{-1} U = X − 1 and define
a θ ( c ) = c − U T θ − 1 θ 1 , x θ ( c ) = ( a θ ( c ) , U T ) T , ρ θ ( c ∣ U ) = ρ X ( a θ ( c ) ∣ U ) ∣ θ 1 ∣ . a_\theta(c)=\frac{c-U^{\mathsf T}\theta_{-1}}{\theta_1},\qquad x_\theta(c)=\left(a_\theta(c),U^{\mathsf T}\right)^{\mathsf T},\qquad\rho_\theta(c\mid U)=\frac{\rho_X(a_\theta(c)\mid U)}{|\theta_1|}. a θ ( c ) = θ 1 c − U T θ − 1 , x θ ( c ) = ( a θ ( c ) , U T ) T , ρ θ ( c ∣ U ) = ∣ θ 1 ∣ ρ X ( a θ ( c ) ∣ U ) .
Also write
A θ ( c ) = E [ ρ θ ( c ∣ U ) x θ ( c ) x θ ( c ) T ] , B θ ( c ) = E [ X I ( X T θ ≥ c ) ] . A_\theta(c)=\mathbb{E}\left[\rho_\theta(c\mid U)x_\theta(c)x_\theta(c)^{\mathsf T}\right],\qquad B_\theta(c)=\mathbb{E}\left[X\mathbb{I}(X^{\mathsf T}\theta\ge c)\right]. A θ ( c ) = E [ ρ θ ( c ∣ U ) x θ ( c ) x θ ( c ) T ] , B θ ( c ) = E [ X I ( X T θ ≥ c ) ] .
Infimal convolution
Fix constants 0 < μ ω ≤ C ω < ∞ 0<\mu_\omega\le C_\omega<\infty 0 < μ ω ≤ C ω < ∞ such that μ ω ≤ ω ′ ′ ≤ C ω \mu_\omega\le\omega^{\prime\prime}\le C_\omega μ ω ≤ ω ′′ ≤ C ω . Write y h ( x ) y_h(x) y h ( x ) for the unique minimizer in (12) . Its optimality equation is
ω ′ ( ( x − y h ( x ) ) / h ) ∈ ∂ f ( y h ( x ) ) , f i n f , h ′ ( x ) = ω ′ ( ( x − y h ( x ) ) / h ) . (71) \omega^{\prime}\left((x-y_h(x))/h\right)\in\partial f(y_h(x)),\qquad f_{\mathrm{inf},h}^{\prime}(x)=\omega^{\prime}\left((x-y_h(x))/h\right). \tag*{(71)} ω ′ ( ( x − y h ( x )) / h ) ∈ ∂ f ( y h ( x )) , f inf , h ′ ( x ) = ω ′ ( ( x − y h ( x )) / h ) . (71)
Since ∣ ∂ f ∣ ≤ C f |\partial f|\le C_f ∣ ∂ f ∣ ≤ C f and ω ′ ′ ≥ μ ω \omega^{\prime\prime}\ge\mu_\omega ω ′′ ≥ μ ω , the quantity ( x − y h ( x ) ) / h (x-y_h(x))/h ( x − y h ( x )) / h is bounded uniformly. In particular,
∣ y h ( x ) − x ∣ ≤ C h , ∣ f i n f , h ′ ( x ) ∣ ≤ C f . (72) |y_h(x)-x|\le Ch,\qquad|f_{\mathrm{inf},h}^{\prime}(x)|\le C_f. \tag*{(72)} ∣ y h ( x ) − x ∣ ≤ C h , ∣ f inf , h ′ ( x ) ∣ ≤ C f . (72)
Let g h = f i n f , h ′ g_h=f_{\mathrm{inf},h}^{\prime} g h = f inf , h ′ and ν h = d g h \nu_h=dg_h ν h = d g h . The derivative is nondecreasing, and ν h ( R ) = ν ( R ) ≤ 2 C f \nu_h(\mathbb{R})=\nu(\mathbb{R})\le2C_f ν h ( R ) = ν ( R ) ≤ 2 C f . The inverse subgradient graphs satisfy
g h − 1 ( s ) = g − 1 ( s ) + h ( ω ′ ) − 1 ( s ) , g ( − ∞ ) < s < g ( + ∞ ) , (73) g_h^{-1}(s)=g^{-1}(s)+h(\omega^{\prime})^{-1}(s),\qquad g(-\infty)<s<g(+\infty), \tag*{(73)} g h − 1 ( s ) = g − 1 ( s ) + h ( ω ′ ) − 1 ( s ) , g ( − ∞ ) < s < g ( + ∞ ) , (73)
where generalized inverses can be used at flat portions. This follows from (71) , or from ( f i n f , h ) ∗ = f ∗ + h ω ∗ (f_{\mathrm{inf},h})^*=f^*+h\omega^* ( f inf , h ) ∗ = f ∗ + h ω ∗ , where ∗ * ∗ denotes convex conjugacy [46 ] . Thus the measures ν h \nu_h ν h and ν \nu ν admit a coupling that moves mass by at most C h Ch C h .
Lemma 7. The proximal minimizer satisfies ∣ y h ( x ) − y h ( z ) ∣ ≤ ∣ x − z ∣ |y_h(x)-y_h(z)|\le|x-z| ∣ y h ( x ) − y h ( z ) ∣ ≤ ∣ x − z ∣ . Consequently, g h g_h g h is nondecreasing and C / h C/h C / h -Lipschitz.
Proof. Monotonicity of ∂ f \partial f ∂ f and (71) give
{ ω ′ ( ( x − y h ( x ) ) / h ) − ω ′ ( ( z − y h ( z ) ) / h ) } { y h ( x ) − y h ( z ) } ≥ 0. \left\{\omega^{\prime}\left((x-y_h(x))/h\right)-\omega^{\prime}\left((z-y_h(z))/h\right)\right\}\{y_h(x)-y_h(z)\}\ge0. { ω ′ ( ( x − y h ( x )) / h ) − ω ′ ( ( z − y h ( z )) / h ) } { y h ( x ) − y h ( z )} ≥ 0.
Since ω ′ \omega^{\prime} ω ′ is strictly increasing, the differences y h ( x ) − y h ( z ) y_h(x)-y_h(z) y h ( x ) − y h ( z ) and x − z − y h ( x ) + y h ( z ) x-z-y_h(x)+y_h(z) x − z − y h ( x ) + y h ( z ) have the same sign. Both lie between zero and x − z x-z x − z when x ≥ z x\ge z x ≥ z . This proves nonexpansiveness and, using ω ′ ′ ≤ C ω \omega^{\prime\prime}\le C_\omega ω ′′ ≤ C ω in (71) , the Lipschitz bound for g h g_h g h .
Let q h = g h ′ q_h=g_h' q h = g h ′ wherever this derivative exists. Lemma 7 gives
0 ≤ q h ≤ C / h , ∫ R q h ( t ) d t ≤ 2 C f . (74) 0\le q_h\le C/h,\qquad\int_{\mathbb{R}}q_h(t)\mathrm{d}t\le2C_f. \tag*{(74)} 0 ≤ q h ≤ C / h , ∫ R q h ( t ) d t ≤ 2 C f . (74)
Under Assumption 4, q h q_h q h is Lipschitz with constant C / h 2 C/h^2 C / h 2 between finitely many transition endpoints. Indeed, on a smooth piece, differentiation of (69) gives
q h ( x ) = f ′ ′ ( y h ( x ) ) ω ′ ′ ( ( x − y h ( x ) ) / h ) ω ′ ′ ( ( x − y h ( x ) ) / h ) + h f ′ ′ ( y h ( x ) ) . q_h(x)=\frac{f''(y_h(x))\omega''((x-y_h(x))/h)}{\omega''((x-y_h(x))/h)+hf''(y_h(x))}. q h ( x ) = ω ′′ (( x − y h ( x )) / h ) + h f ′′ ( y h ( x )) f ′′ ( y h ( x )) ω ′′ (( x − y h ( x )) / h ) .
The assumed derivative bounds and ω ′ ′ ≥ μ ω \omega''\ge\mu_\omega ω ′′ ≥ μ ω bound its derivative on that piece. If y h ( x ) y_h(x) y h ( x ) equals a breakpoint t j t_j t j whose one-sided derivatives are m j , − , m j , + m_{j,-},m_{j,+} m j , − , m j , + , then
q h ( x ) = h − 1 ω ′ ′ ( ( x − t j ) / h ) , t j + h ( ω ′ ) − 1 ( m j , − ) < x < t j + h ( ω ′ ) − 1 ( m j , + ) . q_h(x)=h^{-1}\omega''((x-t_j)/h),\qquad t_j+h(\omega')^{-1}(m_{j,-})<x<t_j+h(\omega')^{-1}(m_{j,+}). q h ( x ) = h − 1 ω ′′ (( x − t j ) / h ) , t j + h ( ω ′ ) − 1 ( m j , − ) < x < t j + h ( ω ′ ) − 1 ( m j , + ) .
Thus there are finitely many mapped boundaries, and the claimed piecewise Lipschitz constant follows. Define q h = 0 q_h=0 q h = 0 at those boundaries. Its values there do not affect any population expectation or any integral along a segment.
For the residual objective, the curvature coupling and conditional-density integration give
sup θ ∈ Θ ∣ ∇ L ‾ i n f , h ( θ ) − ∇ L ‾ ( θ ) ∣ 2 ≤ C h , sup θ ∈ Θ ∥ ∇ 2 L ‾ i n f , h ( θ ) − ∇ 2 L ‾ ( θ ) ∥ ≤ C h . (75) \sup_{\theta\in\Theta}\left|\nabla\overline{\mathcal{L}}_{\mathrm{inf},h}(\theta)-\nabla\overline{\mathcal{L}}(\theta)\right|_2\le Ch,\qquad
\sup_{\theta\in\Theta}\left\|\nabla^2\overline{\mathcal{L}}_{\mathrm{inf},h}(\theta)-\nabla^2\overline{\mathcal{L}}(\theta)\right\|\le Ch. \tag*{(75)} θ ∈ Θ sup ∇ L inf , h ( θ ) − ∇ L ( θ ) 2 ≤ C h , θ ∈ Θ sup ∇ 2 L inf , h ( θ ) − ∇ 2 L ( θ ) ≤ C h . (75)
Each unit projection of the gradient difference is bounded by C h ν ( R ) E ∣ v T Z ∣ Ch\nu(\mathbb{R})\mathbb{E}|v^\mathsf{T}Z| C h ν ( R ) E ∣ v T Z ∣ . The Hessian difference is bounded by C h ν ( R ) E ( v T Z ) 2 Ch\nu(\mathbb{R})\mathbb{E}(v^\mathsf{T}Z)^2 C h ν ( R ) E ( v T Z ) 2 , using the bounded first derivative of the conditional response density. This derivative bound also gives the population Hessian Lipschitz property.
For the index objective, the population gradient and Hessian are b + g ( − ∞ ) E X + ∫ B θ ( c ) ν ( d c ) b+g(-\infty)\mathbb{E}X+\int B_\theta(c)\nu(\mathrm{d}c) b + g ( − ∞ ) E X + ∫ B θ ( c ) ν ( d c ) and ∫ A θ ( c ) ν ( d c ) \int A_\theta(c)\nu(\mathrm{d}c) ∫ A θ ( c ) ν ( d c ) . The all-level section moments in the proof of Proposition 6 bound A θ ( c ) A_\theta(c) A θ ( c ) , B θ ( c ) B_\theta(c) B θ ( c ) , and B θ ′ ( c ) = − E [ ρ θ ( c ∣ U ) x θ ( c ) ] B'_\theta(c)=-\mathbb{E}[\rho_\theta(c\mid U)x_\theta(c)] B θ ′ ( c ) = − E [ ρ θ ( c ∣ U ) x θ ( c )] in operator or Euclidean norm by C C C . The curvature coupling in (71) therefore gives the gradient bound in (73). For the Hessian bound, decompose ν \nu ν into the bounded density f ′ ′ f'' f ′′ on its smooth pieces and the finitely many atoms at its breakpoints. On a smooth piece, the formula for q h q_h q h above and ∣ y h ( x ) − x ∣ ≤ C h |y_h(x)-x|\le Ch ∣ y h ( x ) − x ∣ ≤ C h give ∣ q h ( x ) − f ′ ′ ( x ) ∣ ≤ C h |q_h(x)-f''(x)|\le Ch ∣ q h ( x ) − f ′′ ( x ) ∣ ≤ C h away from C h Ch C h -neighborhoods of the breakpoints. The component of ν h \nu_h ν h transported from the density f ′ ′ f'' f ′′ has density at most f ′ ′ ( y h ( x ) ) ≤ C f''(y_h(x))\le C f ′′ ( y h ( x )) ≤ C on its mapped smooth pieces and zero on the intervals receiving an atom. Its contribution, and that of f ′ ′ f'' f ′′ , in the breakpoint neighborhoods is O ( h ) O(h) O ( h ) by the weighted slab bound (68). Each atom is transported by at most C h Ch C h near a fixed level, where A θ ( c ) A_\theta(c) A θ ( c ) is Lipschitz in c c c by the density derivative bound and the section moments. Its Hessian contribution consequently changes by O ( h ) O(h) O ( h ) . This proves (73) for the index objective. The original index Hessian is also Lipschitz in θ \theta θ under Assumption 4. For its continuous-curvature part, bounded f ′ ′ ′ f''' f ′′′ gives C d E [ ( v T X ) 2 ∣ u T X ∣ ] Cd\mathbb{E}[(v^\mathsf{T}X)^2|u^\mathsf{T}X|] C d E [( v T X ) 2 ∣ u T X ∣ ] off the finitely many breakpoint crossings, while (67) bounds their contributions by C d Cd C d . The atomic parts are Lipschitz by the compact-level calculation in the proof of Proposition 6. Here u u u is the displacement direction, v v v is a unit test vector, and d d d is the displacement length.
Proposition 7. Under the infimal-convolution assumptions specified above, suppose θ ∗ \theta^* θ ∗ is an interior population solution and L ‾ \overline{\mathcal{L}} L is strongly convex on a ball about θ ∗ \theta^* θ ∗ . For sufficiently small h h h , the smoothed population solution θ i n f , h ∗ \theta^*_{\mathrm{inf},h} θ inf , h ∗ is unique, lies in the interior of Θ \Theta Θ , and satisfies
∣ θ i n f , h ∗ − θ ∗ ∣ 2 ≤ C h . \left|\theta^*_{\mathrm{inf},h}-\theta^*\right|_2\le Ch. θ inf , h ∗ − θ ∗ 2 ≤ C h .
Proof of Proposition 7. Choose a ball B ( θ ∗ , r ) ⊆ Θ \mathbb{B}(\theta^*,r) \subseteq\Theta B ( θ ∗ , r ) ⊆ Θ on which the expected objective is strongly convex. By (73), the smoothed expected Hessian is bounded below on this ball for small h h h . The original gradient points strictly outward on its boundary, and its perturbation has norm O ( h ) O(h) O ( h ) . Hence the smoothed gradient also points strictly outward there. Minimizing over the closed ball gives an interior stationary point. Convexity makes it a global minimizer, and strict convexity on the ball makes it the unique global minimizer. Its gradient equation and (73) give ∣ θ i n f , h ∗ − θ ∗ ∣ 2 ≤ C ∣ ∇ L ~ i n f , h ( θ ∗ ) ∣ 2 ≤ C h |\theta^*_{\mathrm{inf},h}-\theta^*|_2 \leq C|\nabla\widetilde{L}_{\mathrm{inf},h}(\theta^*)|_2 \leq Ch ∣ θ inf , h ∗ − θ ∗ ∣ 2 ≤ C ∣∇ L inf , h ( θ ∗ ) ∣ 2 ≤ C h .
Lemma 8. Under the infimal-convolution assumptions specified above, g h g_h g h is bounded and nondecreasing with total variation at most 2 C f 2C_f 2 C f . For d = ∣ θ 1 − θ 2 ∣ 2 d=|\theta_1-\theta_2|_2 d = ∣ θ 1 − θ 2 ∣ 2 and every unit v v v ,
P ∣ v T { g h ( Q θ 1 ) − g h ( Q θ 2 ) } ∣ k ≤ d k ! C k , k ≥ 2. P|v^\mathsf{T}\{g_h(Q\theta_1)-g_h(Q\theta_2)\}|^k \leq d\,k!C^k,\qquad k\geq2. P ∣ v T { g h ( Q θ 1 ) − g h ( Q θ 2 )} ∣ k ≤ d k ! C k , k ≥ 2. The smoothed population gradients are Lipschitz with a constant independent of h h h .
Proof. For the index objective, integrate the all-level crossing bound (67) against ν h \nu_h ν h . For the residual objective, use conditional response-density integration as in (57). Monotonicity and ν h ( R ) ≤ 2 C f \nu_h(\mathbb{R}) \leq2C_f ν h ( R ) ≤ 2 C f give the displayed moments. Their first-moment version proves the population Lipschitz bound.
Lemma 9. Under the infimal-convolution assumptions specified above,
sup θ ∈ Θ ∥ ( P n − P ) [ q h ( Q θ ) V V T ] ∥ ≤ C { a n / h + a n / h } \sup_{\theta\in\Theta}\|(P_n-P)[q_h(Q\theta)\mathbf{V}\mathbf{V}^{\mathsf{T}}]\| \leq C\{\sqrt{a_n/h}+a_n/h\} θ ∈ Θ sup ∥ ( P n − P ) [ q h ( Qθ ) V V T ] ∥ ≤ C { a n / h + a n / h } with probability at least 1 − C n − γ p 1-Cn^{-\gamma p} 1 − C n − γ p . The same assertion holds for integral convolution under Assumption 4-int, with q h = f i n t , h ′ ′ q_h=f''_{\mathrm{int},h} q h = f int , h ′′ wherever the derivative exists.
Proof. First fix θ \theta θ and a unit vector v v v . For the index objective, integrate the section moment (66) against q h ( c ) d c q_h(c)\,dc q h ( c ) d c . For the residual objective, condition on Z Z Z and use its bounded response density. Together with (72), these give, for every k ≥ 2 k\geq2 k ≥ 2 ,
P ∣ q h ( Q θ ) ( v T V ) 2 ∣ k ≤ h 1 − k k ! C k . P|q_h(Q\theta)(v^\mathsf{T}V)^2|^k \leq h^{1-k}k!C^k. P ∣ q h ( Qθ ) ( v T V ) 2 ∣ k ≤ h 1 − k k ! C k .
After subtracting the mean, Lemma 6 applies with variance constant C / h C/h C / h and scale constant C / h C/h C / h . The fixed-pair deviation is therefore C { t / ( n h ) + t / ( n h ) } C\{\sqrt{t/(nh)}+t/(nh)\} C { t / ( nh ) + t / ( nh )} with failure probability 2 e − t 2e^{-t} 2 e − t .
Take a decision net with mesh δ = n − 12 \delta=n^{-12} δ = n − 12 and a 1 / 4 1/4 1/4 -sphere net V \mathcal{V} V . Their combined logarithmic cardinality is O ( p log n ) O(p\log n) O ( p log n ) . On the sample event max i ∣ V i ∣ 2 ≤ B n = C B p log n \max_i|V_i|_2\leq B_n=C_B\sqrt{p\log n} max i ∣ V i ∣ 2 ≤ B n = C B p log n established above, let w n = 2 δ B n w_n=2\delta B_n w n = 2 δ B n . For a decision and its net representative, the piecewise Lipschitz description gives
∣ q h ( Q θ ) − q h ( Q θ ~ ) ∣ ≤ C w n / h 2 + ( C / h ) ∑ j I { ∣ Q θ ~ − e j , h ∣ ≤ w n } , |q_h(Q\theta)-q_h(Q\widetilde{\theta})|\leq Cw_n/h^2+(C/h)\sum_j\mathbb{I}\{|Q\widetilde{\theta}-e_{j,h}|\leq w_n\}, ∣ q h ( Qθ ) − q h ( Q θ ) ∣ ≤ C w n / h 2 + ( C / h ) j ∑ I { ∣ Q θ − e j , h ∣ ≤ w n } ,
where the finite set { e j , h } \{e_{j,h}\} { e j , h } consists of the mapped boundaries. The index slab bound (68), or conditional response-density integration, gives
P [ ∣ v T V ∣ 2 k I { ∣ Q θ ~ − e j , h ∣ ≤ w n } ] ≤ w n k ! C k . P[|v^\mathsf{T}V|^{2k}\mathbb{I}\{|Q\widetilde{\theta}-e_{j,h}|\leq w_n\}]\leq w_n k!C^k. P [ ∣ v T V ∣ 2 k I { ∣ Q θ − e j , h ∣ ≤ w n }] ≤ w n k ! C k .
Bernstein’s inequality and the net union bound show that every empirical weighted slab average, and its expectation, is at most C ( w n + a n ) C(w_n+a_n) C ( w n + a n ) with failure probability C n − γ p Cn^{-\gamma p} C n − γ p . A separate application to ( v T V ) 2 (v^{\mathsf{T}}V)^2 ( v T V ) 2 gives ( P n + P ) ( v T V ) 2 ≤ C (P_n+P)(v^{\mathsf{T}}V)^2 \le C ( P n + P ) ( v T V ) 2 ≤ C for all v ∈ V v \in V v ∈ V . Thus the interpolation error for quadratic forms is at most
C w n / h 2 + C ( w n + a n ) / h + C n − 10 / h . Cw_n/h^2 + C(w_n+a_n)/h + Cn^{-10}/h. C w n / h 2 + C ( w n + a n ) / h + C n − 10 / h .
The last term bounds the population contribution outside ∣ V ∣ 2 ≤ B n |V|_2 \le B_n ∣ V ∣ 2 ≤ B n , by the fourth directional moment and the exponentially small norm tail. Since h ≥ A a n ≥ A log n / n h \ge Aa_n \ge A\log n/n h ≥ A a n ≥ A log n / n and p log n ≤ n p\log n \le n p log n ≤ n , the terms involving w n w_n w n and n − 10 n^{-10} n − 10 are negligible relative to a n / h a_n/h a n / h .
For integral convolution, the finite Stieltjes measure ν = d g \nu=dg ν = d g gives
q h ( x ) = ∫ h − 1 K ( ( x − t ) / h ) ν ( d t ) , ∥ q h ∥ ∞ ≤ C / h , ∫ q h ≤ 2 C f . q_h(x)=\int h^{-1}K((x-t)/h)\nu(dt),\qquad\|q_h\|_\infty\le C/h,\qquad\int q_h \le2Cf. q h ( x ) = ∫ h − 1 K (( x − t ) / h ) ν ( d t ) , ∥ q h ∥ ∞ ≤ C / h , ∫ q h ≤ 2 C f .
The fixed-pair moment calculation is therefore identical. If K K K is globally Lipschitz, then Lip ( q h ) ≤ C / h 2 \operatorname{Lip}(q_h) \le C/h^2 Lip ( q h ) ≤ C / h 2 , and interpolation uses only the C w n / h 2 Cw_n/h^2 C w n / h 2 term, without boundary slabs.
If K K K has bounded variation, choose the right-continuous representative of q h q_h q h and let λ h = d q h \lambda_h=dq_h λ h = d q h be its signed Stieltjes measure. Convolution gives ∣ λ h ∣ ( R ) ≤ ν ( R ) TV ( K ) / h ≤ C / h |\lambda_h|(\mathbb{R}) \le\nu(\mathbb{R})\operatorname{TV}(K)/h \le C/h ∣ λ h ∣ ( R ) ≤ ν ( R ) TV ( K ) / h ≤ C / h . For scalar indices or residuals within w n w_n w n , their curvature difference is bounded by
A θ ~ = ∫ I { ∣ Q θ ~ − t ∣ ≤ w n } ∣ λ h ∣ ( d t ) . A_{\widetilde{\theta}}=\int\mathbb{I}\{|Q_{\widetilde{\theta}}-t|\le w_n\}|\lambda_h|(dt). A θ = ∫ I { ∣ Q θ − t ∣ ≤ w n } ∣ λ h ∣ ( d t ) .
The same slab bound and the directional moments give, for k ≥ 1 k \ge1 k ≥ 1 ,
P [ ( ( v T V ) 2 A θ ~ ) k ] ≤ w n h − k k ! C k . P[((v^{\mathsf{T}}V)^2A_{\widetilde{\theta}})^k]\le w_nh^{-k}k!C^k. P [(( v T V ) 2 A θ ) k ] ≤ w n h − k k ! C k .
Bernstein’s inequality and the same net union bound therefore bound the empirical bracket and its expectation by C { w n + w n a n + a n } / h ≤ C a n / h C\{w_n+\sqrt{w_na_n}+a_n\}/h \le Ca_n/h C { w n + w n a n + a n } / h ≤ C a n / h . The population tail outside the sample event is at most C n − 10 / h Cn^{-10}/h C n − 10 / h . This proves the same interpolation bound without a Lipschitz assumption on K K K . In both cases apply the fixed-pair deviation with t = A p log n t=Ap\log n t = A p log n over the nets. The symmetric-matrix sphere-net inequality converts their quadratic-form bounds to the operator norm bound. Adding the finite-net, sample-tail, covariance, and bracket failure probabilities gives C n − γ p Cn^{-\gamma p} C n − γ p .
Infimal-convolution case of Theorem 5. Write d = ∣ θ 1 − θ 2 ∣ 2 d=|\theta_1-\theta_2|_2 d = ∣ θ 1 − θ 2 ∣ 2 . Lemma 8 and the corresponding index or residual net proof give
∣ ∇ L i n f , h ( θ 1 ) − ∇ L i n f , h ( θ 2 ) − ∇ L ‾ i n f , h ( θ 1 ) + ∇ L ‾ i n f , h ( θ 2 ) ∣ 2 ≤ C ( d a n + a n ) . (76) |\nabla L_{\mathrm{inf},h}(\theta_1)-\nabla L_{\mathrm{inf},h}(\theta_2)-\nabla\overline{L}_{\mathrm{inf},h}(\theta_1)+\nabla\overline{L}_{\mathrm{inf},h}(\theta_2)|_2 \le C(\sqrt{da_n}+a_n). \tag*{(76)} ∣∇ L inf , h ( θ 1 ) − ∇ L inf , h ( θ 2 ) − ∇ L inf , h ( θ 1 ) + ∇ L inf , h ( θ 2 ) ∣ 2 ≤ C ( d a n + a n ) . (76)
Alternatively, integrate the uniform Hessian bound in Lemma 9 along the segment, which lies in Θ \Theta Θ by convexity. Absolute continuity of g h g_h g h then gives
∣ ∇ L i n f , h ( θ 1 ) − ∇ L i n f , h ( θ 2 ) − ∇ L ‾ i n f , h ( θ 1 ) + ∇ L ‾ i n f , h ( θ 2 ) ∣ 2 ≤ C d { a n / h + a n / h } . (77) |\nabla L_{\mathrm{inf},h}(\theta_1)-\nabla L_{\mathrm{inf},h}(\theta_2)-\nabla\overline{L}_{\mathrm{inf},h}(\theta_1)+\nabla\overline{L}_{\mathrm{inf},h}(\theta_2)|_2 \le Cd\{\sqrt{a_n/h}+a_n/h\}. \tag*{(77)} ∣∇ L inf , h ( θ 1 ) − ∇ L inf , h ( θ 2 ) − ∇ L inf , h ( θ 1 ) + ∇ L inf , h ( θ 2 ) ∣ 2 ≤ C d { a n / h + a n / h } . (77)
A segment lying in a mapped boundary contributes a constant scenario derivative, and the assigned value q h = 0 q_h=0 q h = 0 gives its correct contribution. At all other points the ordinary chain rule applies almost everywhere. For d ≤ h d \le h d ≤ h , (75) gives C d a n / h Cd\sqrt{a_n/h} C d a n / h , since a n / h ≤ 1 a_n/h \le1 a n / h ≤ 1 . For d ≥ h ≥ A a n d \ge h \ge Aa_n d ≥ h ≥ A a n , the additive term in (74) is absorbed by d a n \sqrt{da_n} d a n . These two ranges prove the minimum bound on the joint event.
Infimal-convolution case of Corollary 4. The population gradient is Lipschitz by Lemma 8. Adding the linear bound in Theorem 5 proves part 1. For part 2, consider any segment in Θ s c \Theta_{\mathrm{sc}} Θ sc and its unit direction u u u . Strong convexity gives a second directional derivative of the original expected objective. of at least μ 0 \mu_0 μ 0 in the interior of this segment. Equation (75) makes the smoothed expected second directional derivative at least 3 μ 0 / 4 3\mu_0/4 3 μ 0 /4 for small h 0 h_0 h 0 . Choose A A A in h ≥ A a n h \ge Aa_n h ≥ A a n large enough that Lemma 9 bounds its empirical deviation in this direction by μ 0 / 4 \mu_0/4 μ 0 /4 . The SAA second derivative along the segment is then at least μ 0 / 2 \mu_0/2 μ 0 /2 almost everywhere. Integrating along the segment proves Condition 2 with μ = μ 0 \mu=\mu_0 μ = μ 0 and ϵ 2 = 0 \epsilon_2=0 ϵ 2 = 0 .
For part 3, write d = ∥ θ − θ ∗ ∥ 2 ≤ C 3 a n d = \lVert\theta-\theta^*\rVert_2 \le C_3\sqrt{a_n} d = ∥ θ − θ ∗ ∥ 2 ≤ C 3 a n . Theorem 5 bounds the mean-zero gradient increment by C min ( a n 3 / 4 , a n / h ) C\min(a_n^{3/4},a_n/\sqrt{h}) C min ( a n 3/4 , a n / h ) . The original population Hessian is Lipschitz, and (73) bounds the smoothing perturbation by C h Ch C h . Its difference from H H H along the segment is therefore at most C ( h + d ) C(h+d) C ( h + d ) . The remaining population increment is at most C ( h d + d 2 ) C(hd+d^2) C ( h d + d 2 ) . The d 2 = O ( a n ) d^2=O(a_n) d 2 = O ( a n ) term is absorbed by the displayed minimum, since h ≤ h 0 < 1 h\le h_0<1 h ≤ h 0 < 1 . This proves part 3.
Proof of Proposition 2. Let U 0 U_0 U 0 be uniform on [ a , b ] [a,b] [ a , b ] and put W = ( ω ′ ) − 1 ( U 0 ) W=(\omega')^{-1}(U_0) W = ( ω ′ ) − 1 ( U 0 ) . The variable W W W is bounded and
E W = 1 b − a ∫ a b ( ω ′ ) − 1 ( u ) d u = ω ∗ ( b ) − ω ∗ ( a ) b − a = c ω . \mathbb{E}W=\frac{1}{b-a}\int_a^b(\omega')^{-1}(u)\,du=\frac{\omega^*(b)-\omega^*(a)}{b-a}=c_\omega. E W = b − a 1 ∫ a b ( ω ′ ) − 1 ( u ) d u = b − a ω ∗ ( b ) − ω ∗ ( a ) = c ω .
The proximal optimality equation gives f inf , h ′ ( k + h s ) = min { b , max { a , ω ′ ( s ) } } f'_{\inf,h}(k+hs)=\min\{b,\max\{a,\omega'(s)\}\} f i n f , h ′ ( k + h s ) = min { b , max { a , ω ′ ( s )}} . Hence f inf , h ′ ( x ) = E [ f + ′ ( x − h W ) ] f'_{\inf,h}(x)=\mathbb{E}[f'_+(x-hW)] f i n f , h ′ ( x ) = E [ f + ′ ( x − hW )] , and matching the left affine tails gives
f inf , h ( x ) = E [ f ( x − h W ) ] + h { a c ω − ω ∗ ( a ) } . f_{\inf,h}(x)=\mathbb{E}[f(x-hW)]+h\{ac_\omega-\omega^*(a)\}. f i n f , h ( x ) = E [ f ( x − hW )] + h { a c ω − ω ∗ ( a )} .
Conditioning on Z Z Z and expanding the conditional response distribution gives
∇ L inf , h ( θ ∗ ) = h ( b − a ) c ω E [ Z ρ ( k + Z T θ ∗ ∣ Z ) ] + O ( h 2 ) . \nabla\mathcal{L}_{\inf,h}(\theta^*)=h(b-a)c_\omega\mathbb{E}[Z\rho(k+Z^\mathsf{T}\theta^*\mid Z)]+O(h^2). ∇ L i n f , h ( θ ∗ ) = h ( b − a ) c ω E [ Z ρ ( k + Z T θ ∗ ∣ Z )] + O ( h 2 ) .
The remainder has norm at most C h 2 Ch^2 C h 2 by the bounded density derivative and the directional moment bounds. Proposition 7 gives a unique interior population minimizer with displacement O ( h ) O(h) O ( h ) . Its gradient equation, (73), and the population Hessian Lipschitz bound give the h B ω + O ( h 2 ) hB_\omega+O(h^2) h B ω + O ( h 2 ) displacement.
For the intercept calculation in Remark 3, Z l = 1 Z_l=1 Z l = 1 almost surely gives
H e l = ( b − a ) E [ Z ρ ( k + Z T θ ∗ ∣ Z ) ] , B ω = − c ω e l . H e_l=(b-a)\mathbb{E}[Z\rho(k+Z^\mathsf{T}\theta^*\mid Z)],\qquad B_\omega=-c_\omega e_l. H e l = ( b − a ) E [ Z ρ ( k + Z T θ ∗ ∣ Z )] , B ω = − c ω e l .
For Moreau smoothing, c ω = ( a + b ) / 2 c_\omega=(a+b)/2 c ω = ( a + b ) /2 . Substitution of the pinball slopes τ − 1 , τ \tau-1,\tau τ − 1 , τ and the CVaR slopes 0 , ( 1 − α C V a R ) − 1 0,(1-\alpha_{\mathrm{CVaR}})^{-1} 0 , ( 1 − α CVaR ) − 1 gives the coefficients in the remark.
Proof of Proposition 3. The proof of Proposition 2 gives E W = c ω \mathbb{E}W=c_\omega E W = c ω . Substitution of x = r + h c ω x=r+hc_\omega x = r + h c ω in its convolution identity proves (15) . Write V = W − c ω V=W-c_\omega V = W − c ω and t Z = k + Z T θ ∗ t_Z=k+Z^\mathsf{T}\theta^* t Z = k + Z T θ ∗ . The variable V V V is bounded and has mean zero. Conditioning on Z Z Z and using ∇ L ( θ ∗ ) = 0 \nabla\mathcal{L}(\theta^*)=0 ∇ L ( θ ∗ ) = 0 therefore gives
∇ L inf , h s h ( θ ∗ ) = ( b − a ) E [ Z E V ∫ 0 h V { ρ ( t Z + s ∣ Z ) − ρ ( t Z ∣ Z ) } d s ] . \nabla\mathcal{L}^{\mathrm{sh}}_{\inf,h}(\theta^*)=(b-a)\mathbb{E}\left[Z\mathbb{E}_V\int_0^{hV}\{\rho(t_Z+s\mid Z)-\rho(t_Z\mid Z)\}\,ds\right]. ∇ L i n f , h sh ( θ ∗ ) = ( b − a ) E [ Z E V ∫ 0 hV { ρ ( t Z + s ∣ Z ) − ρ ( t Z ∣ Z )} d s ] .
The bounded density derivative and sup ∥ v ∥ 2 = 1 E ∣ v T Z ∣ ≤ C \sup_{\lVert v\rVert_2=1}\mathbb{E}|v^\mathsf{T}Z|\le C sup ∥ v ∥ 2 = 1 E ∣ v T Z ∣ ≤ C bound the norm of this gradient by C h 2 Ch^2 C h 2 . The conditional-density calculation in (73), with the bounded shift V V V , gives an O ( h ) O(h) O ( h ) Hessian perturbation. The boundary-gradient argument in the proof of Proposition 7 then gives a unique population minimizer in the interior of Θ \Theta Θ , at distance O ( h 2 ) O(h^2) O ( h 2 ) from θ ∗ \theta^* θ ∗ . For ω ( u ) = u 2 / 2 \omega(u)=u^2/2 ω ( u ) = u 2 /2 , we have W = U 0 W=U_0 W = U 0 , c ω = ( a + b ) / 2 c_\omega=(a+b)/2 c ω = ( a + b ) /2 , and a c ω − ω ∗ ( a ) = a b / 2 ac_\omega-\omega^*(a)=ab/2 a c ω − ω ∗ ( a ) = ab /2 , proving the last assertion.
For the bandwidth lower bound in § 5.1 , consider the pinball objective in Proposition 1. Let θ ^ \hat{\theta} θ ^ be the minimizer supplied by that proposition, let l l l be the intercept coordinate, and put A = { i : Y i − Z i T θ ^ = 0 } A = \{i : Y_i - Z_i^{\mathsf{T}}\hat{\theta} = 0\} A = { i : Y i − Z i T θ ^ = 0 } , with ∣ A ∣ ≥ p \lvert A\rvert\ge p ∣ A ∣ ≥ p . Write a = c ( τ − 1 ) a = c(\tau- 1) a = c ( τ − 1 ) , b = c τ b = c\tau b = c τ , s 0 = ( ω ′ ) − 1 ( ( a + b ) / 2 ) s_0 = (\omega')^{-1}((a+b)/2) s 0 = ( ω ′ ) − 1 (( a + b ) /2 ) , and θ ~ h = θ ^ − h s 0 e l \tilde{\theta}_h = \hat{\theta} - hs_0e_l θ ~ h = θ ^ − h s 0 e l . At this decision, each scenario in A A A has residual h s 0 hs_0 h s 0 . Since a < ω ′ ( s 0 ) < b a < \omega'(s_0) < b a < ω ′ ( s 0 ) < b , it lies in the interior of the transition interval h [ ( ω ′ ) − 1 ( a ) , ( ω ′ ) − 1 ( b ) ] h[(\omega')^{-1}(a),(\omega')^{-1}(b)] h [( ω ′ ) − 1 ( a ) , ( ω ′ ) − 1 ( b )] , where f i n f , h ′ ( x ) = ω ′ ( x / h ) f'_{\mathrm{inf},h}(x) = \omega'(x/h) f inf , h ′ ( x ) = ω ′ ( x / h ) . Monotonicity makes every other scenario’s contribution to the following directional increment nonnegative. Consequently,
lim inf t ↓ 0 ∇ l L i n f , h ( θ ~ h + t e l ) − ∇ l L i n f , h ( θ ~ h ) t ≥ ∣ A ∣ ω ′ ′ ( s 0 ) n h ≥ p μ ω n h . \liminf_{t\downarrow0}\frac{\nabla_l\mathcal{L}_{\mathrm{inf},h}(\tilde{\theta}_h+t e_l)-\nabla_l\mathcal{L}_{\mathrm{inf},h}(\tilde{\theta}_h)}{t}\ge\frac{\lvert A\rvert\omega''(s_0)}{nh}\ge\frac{p\mu_\omega}{nh}. t ↓ 0 lim inf t ∇ l L inf , h ( θ ~ h + t e l ) − ∇ l L inf , h ( θ ~ h ) ≥ nh ∣ A ∣ ω ′′ ( s 0 ) ≥ nh p μ ω .
The distance from θ ~ h \tilde{\theta}_h θ ~ h to θ ^ \hat{\theta} θ ^ is h ∣ s 0 ∣ h\lvert s_0\rvert h ∣ s 0 ∣ . Hence any Lipschitz bound on a fixed neighborhood of θ ^ \hat{\theta} θ ^ satisfies L h ≥ p μ ω / ( n h ) L_h \ge p\mu_\omega/(nh) L h ≥ p μ ω / ( nh ) for sufficiently small h h h , proving the necessary bandwidth order for every prox-function in Assumption 4.
Integral convolution
Integral convolution has been studied for inference and regularization in quantile regression. Bahadur representations and asymptotic inference are developed by Fernandes et al. [20 ] , and nonasymptotic expansions, normal approximation, and bootstrap validity in growing dimension are established by He et al. [22 ] . Convolution smoothing is combined with concave regularization in Tan et al. [54 ] and with noisy iterative hard thresholding for sparse newsvendor policies in Chang et al. [7 ] . Related smoothing-based refinement under memory constraints is studied by Chen et al. [9 ] . These works establish inference and regularization benefits of smoothing for quantile losses. The following results give the corresponding regularity, population displacement, and computed-solution expansions for the convex Lipschitz losses considered here.
For a probability kernel K K K , define the scalar integral convolution
f i n t , h ( x ) = ∫ R h − 1 K ( ( x − y ) / h ) f ( y ) d y , (78) f_{\mathrm{int},h}(x)=\int_{\mathbb{R}} h^{-1}K((x-y)/h)f(y)\,dy, \tag*{(78)} f int , h ( x ) = ∫ R h − 1 K (( x − y ) / h ) f ( y ) d y , (78)
and the objective L i n t , h ( θ ) = n − 1 ∑ i = 1 n f i n t , h ( Y i − Z i T θ ) + b T θ \mathcal{L}_{\mathrm{int},h}(\theta)=n^{-1}\sum_{i=1}^{n}f_{\mathrm{int},h}(Y_i-Z_i^{\mathsf{T}}\theta)+b^{\mathsf{T}}\theta L int , h ( θ ) = n − 1 ∑ i = 1 n f int , h ( Y i − Z i T θ ) + b T θ with expected objective L ‾ i n t , h \overline{\mathcal{L}}_{\mathrm{int},h} L int , h defined through loss differences. Gaussian and uniform convolution give closed-form gradients for the piecewise linear losses in quantile regression and SVM. Without a closed form, the gradient E [ f ′ ( x − h V ) ] \mathbb{E}[f'(x-hV)] E [ f ′ ( x − hV )] , where V V V has density K K K , requires numerical integration or sampling, and sampling turns a full-batch update into a stochastic update (Duchi et al. [19 ] ).
Assumption 4-int. The kernel K K K is a symmetric, bounded, nonnegative probability density with finite second moment, and is either globally Lipschitz continuous or of bounded variation. The conditional density in Assumption 2(i) has a bounded continuous first derivative.
Assumption 4-int includes Gaussian and uniform kernels and permits every convex Lipschitz loss in Assumption 3, without additional piecewise smoothness.
Corollary 14. Suppose Assumptions 1–3 and 4-int hold. Theorem 5 and Corollary 4 hold for L i n t , h \mathcal{L}_{\mathrm{int},h} L int , h in place of L i n f , h \mathcal{L}_{\mathrm{inf},h} L inf , h , with Assumption 4-int in place of Assumption 4. If θ ∗ \theta^* θ ∗ is an interior population solution and L ‾ \overline{\mathcal{L}} L is strongly convex on a ball about it, then ∣ θ i n t , h ∗ − θ ∗ ∣ 2 = O ( h 2 ) \lvert\theta^*_{\mathrm{int},h}-\theta^*\rvert_2=O(h^2) ∣ θ int , h ∗ − θ ∗ ∣ 2 = O ( h 2 ) for sufficiently small h h h . Under the hypotheses of Corollary 5, with Assumption 4-int in place of Assumption 4, its optimization bound, expansion, and normal limit hold with L i n t , h \mathcal{L}_{\mathrm{int},h} L int , h in place of L h L_h L h and θ ∗ \theta^* θ ∗ in place of θ h ∗ \theta_h^* θ h ∗ , without a residual shift.
The additional condition used for the integral index perturbation bounds is
sup θ ∈ Θ , ∣ v ∣ 2 = 1 sup ∣ s ∣ ≤ 1 ∫ R E [ ∣ ρ θ ′ ( c + s ∣ U ) ∣ { v T x θ ( c + s ) } 2 ] ν ( d c ) ≤ C , (79) \sup_{\theta\in\Theta,\,|v|_2=1}\sup_{|s|\leq1}\int_{\mathbb{R}}\mathbb{E}\left[\left|\rho'_{\theta}(c+s\mid U)\right|\left\{v^{\mathsf{T}}x_{\theta}(c+s)\right\}^{2}\right]\nu(dc)\leq C, \tag*{(79)} θ ∈ Θ , ∣ v ∣ 2 = 1 sup ∣ s ∣ ≤ 1 sup ∫ R E [ ∣ ρ θ ′ ( c + s ∣ U ) ∣ { v T x θ ( c + s ) } 2 ] ν ( d c ) ≤ C , (79)
where the prime denotes differentiation in c c c .
Proposition 8. Under the integral-convolution assumptions specified above, suppose θ ∗ \theta^* θ ∗ is an interior population solution and L ‾ \overline{L} L is strongly convex on a ball about θ ∗ \theta^* θ ∗ . For an index objective impose (77). For sufficiently small h h h , the integral-convolution population minimizer is unique, lies in the interior of Θ \Theta Θ , and satisfies ∣ θ i n t , h ∗ − θ ∗ ∣ 2 ≤ C h 2 |\theta^*_{\mathrm{int},h}-\theta^*|_2\leq Ch^2 ∣ θ int , h ∗ − θ ∗ ∣ 2 ≤ C h 2 . For the residual objective, no condition on the curvature support is needed.
Proof. Let V V V have density K K K . For the index objective, differentiation of the conditional integrals gives
B θ ′ ( c ) = − E [ ρ θ ( c ∣ U ) x θ ( c ) ] , B'_{\theta}(c)=-\mathbb{E}[\rho_{\theta}(c\mid U)x_{\theta}(c)], B θ ′ ( c ) = − E [ ρ θ ( c ∣ U ) x θ ( c )] ,
B θ ′ ′ ( c ) = − E [ ρ θ ′ ( c ∣ U ) x θ ( c ) + ρ θ ( c ∣ U ) e 1 / θ 1 ] , B''_{\theta}(c)=-\mathbb{E}[\rho'_{\theta}(c\mid U)x_{\theta}(c)+\rho_{\theta}(c\mid U)e_1/\theta_1], B θ ′′ ( c ) = − E [ ρ θ ′ ( c ∣ U ) x θ ( c ) + ρ θ ( c ∣ U ) e 1 / θ 1 ] ,
A θ ′ ( c ) = E [ ρ θ ′ ( c ∣ U ) x θ ( c ) x θ ( c ) T + ρ θ ( c ∣ U ) { e 1 x θ ( c ) T + x θ ( c ) e 1 T } / θ 1 ] . A'_{\theta}(c)=\mathbb{E}[\rho'_{\theta}(c\mid U)x_{\theta}(c)x_{\theta}(c)^{\mathsf{T}}+\rho_{\theta}(c\mid U)\{e_1x_{\theta}(c)^{\mathsf{T}}+x_{\theta}(c)e_1^{\mathsf{T}}\}/\theta_1]. A θ ′ ( c ) = E [ ρ θ ′ ( c ∣ U ) x θ ( c ) x θ ( c ) T + ρ θ ( c ∣ U ) { e 1 x θ ( c ) T + x θ ( c ) e 1 T } / θ 1 ] .
Equation (79) , Cauchy–Schwarz, and the section moments bound the integrals against ν \nu ν of the local second derivative of B θ B_{\theta} B θ and the local first derivative of A θ A_{\theta} A θ by C C C . Their global norm bounds extend Taylor’s inequalities to every shift, giving
sup θ ∈ Θ ∥ ∫ [ B θ ( c + s ) − B θ ( c ) − s B θ ′ ( c ) ] ν ( d c ) ∥ 2 ≤ C s 2 , \sup_{\theta\in\Theta}\left\|\int[B_{\theta}(c+s)-B_{\theta}(c)-sB'_{\theta}(c)]\nu(dc)\right\|_2\leq Cs^2, θ ∈ Θ sup ∫ [ B θ ( c + s ) − B θ ( c ) − s B θ ′ ( c )] ν ( d c ) 2 ≤ C s 2 ,
sup θ ∈ Θ ∥ ∫ [ A θ ( c + s ) − A θ ( c ) ] ν ( d c ) ∥ ≤ C ∣ s ∣ . \sup_{\theta\in\Theta}\left\|\int[A_{\theta}(c+s)-A_{\theta}(c)]\nu(dc)\right\|\leq C|s|. θ ∈ Θ sup ∫ [ A θ ( c + s ) − A θ ( c )] ν ( d c ) ≤ C ∣ s ∣.
For ∣ s ∣ ≤ 1 |s|\leq1 ∣ s ∣ ≤ 1 these follow by integrating the displayed derivatives. For ∣ s ∣ > 1 |s|>1 ∣ s ∣ > 1 , use the bounded norms of A A A , B B B , and B ′ B' B ′ , and ν ( R ) ≤ 2 C f \nu(\mathbb{R})\leq2C_f ν ( R ) ≤ 2 C f . The smoothed population gradient replaces B θ ( c ) B_{\theta}(c) B θ ( c ) by E V B θ ( c + h V ) \mathbb{E}_V B_{\theta}(c+hV) E V B θ ( c + hV ) , and its Hessian makes the same replacement for A θ A_{\theta} A θ . Thus E V = 0 \mathbb{E}V=0 E V = 0 and E V 2 < ∞ \mathbb{E}V^2<\infty E V 2 < ∞ give
sup θ ∈ Θ ∥ ∇ L ‾ i n t , h ( θ ) − ∇ L ‾ ( θ ) ∥ 2 ≤ C h 2 , (80) \sup_{\theta\in\Theta}\left\|\nabla\overline{L}_{\mathrm{int},h}(\theta)-\nabla\overline{L}(\theta)\right\|_2\leq Ch^2, \tag*{(80)} θ ∈ Θ sup ∇ L int , h ( θ ) − ∇ L ( θ ) 2 ≤ C h 2 , (80)
sup θ ∈ Θ ∥ ∇ 2 L ‾ i n t , h ( θ ) − ∇ 2 L ‾ ( θ ) ∥ ≤ C h . (81) \sup_{\theta\in\Theta}\left\|\nabla^2\overline{L}_{\mathrm{int},h}(\theta)-\nabla^2\overline{L}(\theta)\right\|\leq Ch. \tag*{(81)} θ ∈ Θ sup ∇ 2 L int , h ( θ ) − ∇ 2 L ( θ ) ≤ C h . (81)
Here are the sufficient conditions for (77). If ν \nu ν has compact support, the reconstructed projections have bounded second moments on its unit neighborhood and ρ θ ′ \rho'_{\theta} ρ θ ′ is bounded. If X X X is bounded, the conditional density and its derivative vanish outside the conditional support of X 1 X_1 X 1 , including its boundary. Wherever the derivative contributes, x θ ( c ) x_{\theta}(c) x θ ( c ) is therefore bounded. If ρ X ′ ′ \rho''_X ρ X ′′ is bounded, nonnegativity and Taylor’s inequality give ∣ ρ X ′ ∣ 2 ≤ C ρ X |\rho'_X|^2\leq C\rho_X ∣ ρ X ′ ∣ 2 ≤ C ρ X , and hence ∣ ρ θ ′ ∣ 2 ≤ C ρ θ |\rho'_{\theta}|^2\leq C\rho_{\theta} ∣ ρ θ ′ ∣ 2 ≤ C ρ θ . Cauchy–Schwarz and (65) imply
E [ ∣ ρ θ ′ ( c ∣ U ) ∣ ( v T x θ ( c ) ) 2 ] ≤ C { E ρ θ ( c ∣ U ) } 1 / 2 { E ( v T x θ ( c ) ) 4 } 1 / 2 ≤ C ( 1 + c 2 ) e − c 1 c 2 ≤ C . \mathbb{E}[|\rho'_{\theta}(c\mid U)|(v^{\mathsf{T}}x_{\theta}(c))^2]\leq C\{\mathbb{E}\rho_{\theta}(c\mid U)\}^{1/2}\{\mathbb{E}(v^{\mathsf{T}}x_{\theta}(c))^4\}^{1/2}\leq C(1+c^2)e^{-c_1c^2}\leq C. E [ ∣ ρ θ ′ ( c ∣ U ) ∣ ( v T x θ ( c ) ) 2 ] ≤ C { E ρ θ ( c ∣ U ) } 1/2 { E ( v T x θ ( c ) ) 4 } 1/2 ≤ C ( 1 + c 2 ) e − c 1 c 2 ≤ C .
This also bounds the supremum over a unit level neighborhood, and integration against the finite measure ν \nu ν verifies the condition.
For a residual objective, conditional integration gives the expected gradient through g ( − ∞ ) + ∫ { 1 − F ( c + Z T θ ∣ Z ) } ν ( d c ) g(-\infty)+\int\{1-F(c+Z^{\mathsf{T}}\theta\mid Z)\}\nu(dc) g ( − ∞ ) + ∫ { 1 − F ( c + Z T θ ∣ Z )} ν ( d c ) . Taylor’s inequality for F F F , the bounded response density derivative, and E V = 0 \mathbb{E}V=0 E V = 0 give (78). The density Lipschitz bound gives (79). Each unit projection uses only E ∣ v T Z ∣ \mathbb{E}|v^\mathsf{T}Z| E ∣ v T Z ∣ or E ( v T Z ) 2 \mathbb{E}(v^\mathsf{T}Z)^2 E ( v T Z ) 2 , respectively, and no response moment is needed. Finally, the outward-gradient and strict-convexity argument in Proposition 7 gives the unique interior minimizer and its O ( h 2 ) O(h^2) O ( h 2 ) displacement.
Without (77), the index Hessian perturbation still tends uniformly to zero. Indeed, the section-density tail and the reconstructed fourth moments give ∥ A θ ( c ) ∥ ≤ C e − c 1 c 2 \|A_\theta(c)\| \leq Ce^{-c_1c^2} ∥ A θ ( c ) ∥ ≤ C e − c 1 c 2 , while the bounded density derivative gives ∥ A θ ′ ( c ) ∥ ≤ C ( 1 + c 2 ) \|A'_\theta(c)\| \leq C(1+c^2) ∥ A θ ′ ( c ) ∥ ≤ C ( 1 + c 2 ) . Splitting levels at ∣ c ∣ = 2 ∣ s ∣ − 1 / 4 |c| = 2|s|^{-1/4} ∣ c ∣ = 2∣ s ∣ − 1/4 yields sup θ , c ∥ A θ ( c + s ) − A θ ( c ) ∥ ≤ C ∣ s ∣ \sup_{\theta,c}\|A_\theta(c+s)-A_\theta(c)\| \leq C\sqrt{|s|} sup θ , c ∥ A θ ( c + s ) − A θ ( c ) ∥ ≤ C ∣ s ∣ for ∣ s ∣ ≤ 1 |s| \leq1 ∣ s ∣ ≤ 1 , with a bounded extension for larger shifts. Consequently the Hessian perturbation is at most C h E ∣ V ∣ = o ( 1 ) C\sqrt{h}\mathbb{E}\sqrt{|V|} = o(1) C h E ∣ V ∣ = o ( 1 ) . This suffices for the ordinary strong convexity conclusion in Corollary 4, which does not impose (77).
Lemma 10. Under the integral-convolution assumptions specified above, its smoothed derivative satisfies the moments in Lemma 8, and its population gradient is Lipschitz with a constant independent of h h h .
Proof. Convolution preserves monotonicity and the total variation bound of g g g . Integrate the index crossing bound or the residual conditional-density bound against the smoothed curvature measure. The first-moment calculation proves the population Lipschitz bound.
Integral-convolution case of Theorem 5 and Corollary 4. Write d = ∣ θ 1 − θ 2 ∣ 2 d = |\theta_1-\theta_2|_2 d = ∣ θ 1 − θ 2 ∣ 2 . Lemma 10 and the corresponding index or residual net argument give
∥ ∇ L i n t , h ( θ 1 ) − ∇ L i n t , h ( θ 2 ) − ∇ L ‾ i n t , h ( θ 1 ) + ∇ L ‾ i n t , h ( θ 2 ) ∥ 2 ≤ C ( d a n + a n ) . (82) \left\|\nabla L_{\mathrm{int},h}(\theta_1)-\nabla L_{\mathrm{int},h}(\theta_2)-\nabla\overline{L}_{\mathrm{int},h}(\theta_1)+\nabla\overline{L}_{\mathrm{int},h}(\theta_2)\right\|_2 \leq C\left(\sqrt{d a_n}+a_n\right). \tag*{(82)} ∇ L int , h ( θ 1 ) − ∇ L int , h ( θ 2 ) − ∇ L int , h ( θ 1 ) + ∇ L int , h ( θ 2 ) 2 ≤ C ( d a n + a n ) . (82)
For integral convolution, integrating the bound in Lemma 9 along the segment between the arguments gives
∥ ∇ L i n t , h ( θ 1 ) − ∇ L i n t , h ( θ 2 ) − ∇ L ‾ i n t , h ( θ 1 ) + ∇ L ‾ i n t , h ( θ 2 ) ∥ 2 ≤ C d { a n h + a n h } . (83) \left\|\nabla L_{\mathrm{int},h}(\theta_1)-\nabla L_{\mathrm{int},h}(\theta_2)-\nabla\overline{L}_{\mathrm{int},h}(\theta_1)+\nabla\overline{L}_{\mathrm{int},h}(\theta_2)\right\|_2 \leq Cd\left\{\sqrt{\frac{a_n}{h}}+\frac{a_n}{h}\right\}. \tag*{(83)} ∇ L int , h ( θ 1 ) − ∇ L int , h ( θ 2 ) − ∇ L int , h ( θ 1 ) + ∇ L int , h ( θ 2 ) 2 ≤ C d { h a n + h a n } . (83)
For d ≤ h d \leq h d ≤ h , use (81). For d ≥ h ≥ A a n d \geq h \geq Aa_n d ≥ h ≥ A a n , absorb a n a_n a n into the square-root term of (80). The sum of the two failure probabilities is C n − γ p Cn^{-\gamma p} C n − γ p , proving the theorem. Combining the population gradient Lipschitz bound in Lemma 10 with (81) proves part 1. For part 2, use the uniform o ( 1 ) o(1) o ( 1 ) population Hessian perturbation proved above and the same Hessian concentration event along each segment in Θ s c \Theta_{\mathrm{sc}} Θ sc , as in the infimal-convolution case of Corollary 4. The second directional derivative along that segment is at least μ 0 / 2 \mu_0/2 μ 0 /2 almost everywhere, and integration gives Condition 2 with μ = μ 0 \mu=\mu_0 μ = μ 0 and ϵ 2 = 0 \epsilon_2=0 ϵ 2 = 0 . For part 3, the mean-zero increment on the radius C 3 a n C_3\sqrt{a_n} C 3 a n is at most C min ( a n 3 / 4 , a n / h ) C\min(a_n^{3/4},a_n/\sqrt{h}) C min ( a n 3/4 , a n / h ) . Under the additional section-derivative condition for an index objective, or the response density derivative condition for a residual objective, equation (79) and the unsmoothed Hessian Lipschitz bound contribute at most C ( a n + a n ) C(\sqrt{a_n}+a_n) C ( a n + a n ) . Absorb a n a_n a n as in the infimal-convolution case of Corollary 4 to obtain (13) for integral convolution.
For the bandwidth lower bound, consider the pinball objective in Proposition 1 and a kernel satisfying Assumption 4-int, continuous at zero with K ( 0 ) > 0 K(0)>0 K ( 0 ) > 0 . Let θ ^ \widehat{\theta} θ be the minimizer supplied by that proposition, write r i = Y i − Z i T θ ^ r_i=Y_i-Z_i^\mathsf{T}\widehat{\theta} r i = Y i − Z i T θ , and put A = { i : r i ≥ 0 } A=\{i:r_i\geq0\} A = { i : r i ≥ 0 } , with ∣ A ∣ ≥ p |A|\geq p ∣ A ∣ ≥ p . Let l l l be the intercept coordinate, with Z i l = 1 Z_{il}=1 Z i l = 1 for every i i i , and let F K ( x ) = ∫ − ∞ x K ( u ) d u F_K(x)=\int_{-\infty}^x K(u)\,du F K ( x ) = ∫ − ∞ x K ( u ) d u . For f ( x ) = c x { τ − I ( x ≤ 0 ) } f(x)=cx\{\tau-\mathbb{I}(x\leq0)\} f ( x ) = c x { τ − I ( x ≤ 0 )} , differentiation under the convolution gives f i n t , h ′ ( x ) = c { τ − 1 + F K ( x / h ) } f'_{\mathrm{int},h}(x)=c\{\tau-1+F_K(x/h)\} f int , h ′ ( x ) = c { τ − 1 + F K ( x / h )} . Hence, for t > 0 t>0 t > 0 ,
∇ l L i n t , h ( θ ^ + t e l ) − ∇ l L i n t , h ( θ ^ ) t = c n t ∑ i = 1 n ∫ ( r i − t ) / h r i / h K ( u ) d u ≥ c ∣ A ∣ n t ∫ − t / h 0 K ( u ) d u . \frac{\nabla_l L_{\mathrm{int},h}(\widehat{\theta}+t e_l)-\nabla_l L_{\mathrm{int},h}(\widehat{\theta})}{t}
=
\frac{c}{nt}\sum_{i=1}^{n}\int_{(r_i-t)/h}^{r_i/h}K(u)\,du
\geq\frac{c|A|}{nt}\int_{-t/h}^{0}K(u)\,du. t ∇ l L int , h ( θ + t e l ) − ∇ l L int , h ( θ ) = n t c i = 1 ∑ n ∫ ( r i − t ) / h r i / h K ( u ) d u ≥ n t c ∣ A ∣ ∫ − t / h 0 K ( u ) d u .
Every summand is nonnegative. Continuity of K K K at zero therefore gives
lim inf t ↓ 0 ∇ L i n t , h ( θ ^ + t e l ) − ∇ L i n t , h ( θ ^ ) t ≥ c ∣ A ∣ K ( 0 ) n h ≥ c p K ( 0 ) n h . \liminf_{t\downarrow0}\frac{\nabla\mathcal{L}_{\mathrm{int},h}(\hat{\theta}+t e_l)-\nabla\mathcal{L}_{\mathrm{int},h}(\hat{\theta})}{t}\geq\frac{c|A|K(0)}{nh}\geq\frac{cpK(0)}{nh}. t ↓ 0 lim inf t ∇ L int , h ( θ ^ + t e l ) − ∇ L int , h ( θ ^ ) ≥ nh c ∣ A ∣ K ( 0 ) ≥ nh c p K ( 0 ) .
If the smoothed gradient is L h L_h L h -Lipschitz on a neighborhood of θ ^ \hat{\theta} θ ^ , then L h ≥ c p K ( 0 ) / ( n h ) L_h\geq cpK(0)/(nh) L h ≥ c p K ( 0 ) / ( nh ) . Thus a constant-order bound on L h L_h L h requires h ≳ p / n h\gtrsim p/n h ≳ p / n , compared with the sufficient scale A a n = A p log n / n A a_n=Ap\log n/n A a n = A p log n / n .
Proof of Corollary 5
Proof. Write L h L_h L h for the objective in question and θ h \theta_h θ h for its population minimizer. The population bounds give ∣ θ h − θ ∗ ∣ 2 ≤ C h |\theta_h-\theta^*|_2\leq Ch ∣ θ h − θ ∗ ∣ 2 ≤ C h for the unshifted infimal objective and C h 2 Ch^2 C h 2 for integral convolution. Proposition 3 gives ∣ θ h − θ ∗ ∣ 2 ≤ C h 2 |\theta_h-\theta^*|_2\leq Ch^2 ∣ θ h − θ ∗ ∣ 2 ≤ C h 2 for the shifted objective. This objective uses the translated loss f ( ⋅ + h c ω ) f(\cdot+hc\omega) f ( ⋅ + h c ω ) , whose breakpoint is k − h c ω k-hc\omega k − h c ω , and whose derivative bounds in Assumption 4 are unchanged.
On a fixed ball about θ ∗ \theta^* θ ∗ , the smoothed population and SAA objectives have strong convexity of constant order. At the deterministic minimizer θ h \theta_h θ h , the expected score is zero. Boundedness of the smoothed scalar derivative and the directional moment bounds give its high-probability norm bound C a n C\sqrt{a_n} C a n and E ∣ ∇ L h ( θ h ) ∣ 2 2 ≤ C p / n \mathbb{E}|\nabla L_h(\theta_h)|_2^2\leq Cp/n E ∣∇ L h ( θ h ) ∣ 2 2 ≤ C p / n . The boundary-gradient argument therefore gives a unique interior SAA minimizer θ ^ h \hat{\theta}_h θ ^ h with ∣ θ ^ h − θ h ∣ 2 = O P ( p / n ) |\hat{\theta}_h-\theta_h|_2=O_P(\sqrt{p/n}) ∣ θ ^ h − θ h ∣ 2 = O P ( p / n ) .
Since the population displacement tends to zero, the local strong-convexity ball about θ ^ h \hat{\theta}_h θ ^ h lies in the population curvature region for sufficiently large n n n . For the projected methods, the initialization distance is bounded by diam ( Θ ) \operatorname{diam}(\Theta) diam ( Θ ) . Corollary 10(ii)(c), with ϵ 1 = ϵ 2 = 0 \epsilon_1=\epsilon_2=0 ϵ 1 = ϵ 2 = 0 , gives both methods distance recursions with zero additive terms. Their initial phases take a bounded number of updates, and their geometric bounds give distance O P ( n − 1 ) O_P(n^{-1}) O P ( n − 1 ) after ⌈ c log n ⌉ \lceil c\log n\rceil ⌈ c log n ⌉ updates from any permitted initialization. For an unprojected method, impose the ball-inclusion requirement in the unprojected extension. The vanishing population displacement then permits the region verification in Corollary 1, whose bound with zero additive terms gives the same optimization accuracy.
For the complete procedures, center their Phase 3 neighborhood at θ h \theta_h θ h , with radius ϵ 3 = C 3 a n \epsilon_3=C_3\sqrt{a_n} ϵ 3 = C 3 a n and sufficiently large C 3 C_3 C 3 . The score at θ h \theta_h θ h has mean zero and satisfies the preceding high-probability bound. The gradient increment relative to H H H has remainder O ( a n 3 / 4 + h a n ) = o ( ϵ 3 ) O(a_n^{3/4}+h\sqrt{a_n})=o(\epsilon_3) O ( a n 3/4 + h a n ) = o ( ϵ 3 ) on this ball. This uses the population Hessian perturbation O ( h ) O(h) O ( h ) and the unsmoothed Hessian Lipschitz property, since ∣ θ h − θ ∗ ∣ 2 = O ( h ) |\theta_h-\theta^*|_2=O(h) ∣ θ h − θ ∗ ∣ 2 = O ( h ) . The pointwise, region, and scale conditions in the complete procedures follow as in the proof of Theorem 4. On this ball, Lemma 9 and the population bounds give
∣ ∇ L h ( θ ) − H ( θ − θ ^ h ) ∣ 2 ≤ C { a n / h + a n / h + h + ϵ 3 } ∣ θ − θ ^ h ∣ 2 . \left|\nabla L_h(\theta)-H(\theta-\hat{\theta}_h)\right|_2\leq C\left\{\sqrt{a_n/h}+a_n/h+h+\epsilon_3\right\}|\theta-\hat{\theta}_h|_2. ∇ L h ( θ ) − H ( θ − θ ^ h ) 2 ≤ C { a n / h + a n / h + h + ϵ 3 } ∣ θ − θ ^ h ∣ 2 .
Choose A A A sufficiently large, then n n n sufficiently large, so that this coefficient is at most m H / 64 m_H/64 m H /64 . The local comparison (40), with c n = 0 c_n=0 c n = 0 , reduces the optimization error to O ( n − 1 ) O(n^{-1}) O ( n − 1 ) in O ( log n ) O(\log n) O ( log n ) updates for both procedures. For the unshifted infimal objective, the neighborhood is centered at θ h \theta_h θ h , since its displacement can exceed a n \sqrt{a_n} a n .
Put ϑ h = θ h \vartheta_h=\theta_h ϑ h = θ h for the unshifted infimal objective and ϑ h = θ ∗ \vartheta_h=\theta^* ϑ h = θ ∗ for the shifted objective and for integral convolution. Independence gives a second moment at most C p / n Cp/n C p / n for the score deviation from its expectation at this deterministic point. The expected score is zero for the unshifted infimal objective and has norm O ( h 2 ) O(h^2) O ( h 2 ) for the shifted objective and for integral convolution. Strong convexity consequently gives
δ n , h : = ∣ θ ^ h − ϑ h ∣ 2 = O P ( p / n + h 2 ) . \delta_{n,h}:=|\hat{\theta}_h-\vartheta_h|_2=O_P\left(\sqrt{p/n}+h^2\right). δ n , h := ∣ θ ^ h − ϑ h ∣ 2 = O P ( p / n + h 2 ) .
The uniform linear increment bound and the population Hessian bounds imply
∣ H ( θ ^ h − ϑ h ) + ∇ L h ( ϑ h ) ∣ 2 ≤ C { δ n , h a n / h + h δ n , h + δ n , h 2 } = O P { p a n n h + h 2 } . \left|H(\hat{\theta}_h-\vartheta_h)+\nabla L_h(\vartheta_h)\right|_2\leq C\left\{\delta_{n,h}\sqrt{a_n/h}+h\delta_{n,h}+\delta_{n,h}^2\right\}=O_P\left\{\sqrt{\frac{pa_n}{nh}}+h^2\right\}. H ( θ ^ h − ϑ h ) + ∇ L h ( ϑ h ) 2 ≤ C { δ n , h a n / h + h δ n , h + δ n , h 2 } = O P { nh p a n + h 2 } .
Subtracting the expected score preserves this remainder. For the chosen bandwidth, p a n / ( n h ) ≤ C ( p / n ) 4 / 5 ( log n ) 2 / 5 \sqrt{pa_n/(nh)} \le C(p/n)^{4/5}(\log n)^{2/5} p a n / ( nh ) ≤ C ( p / n ) 4/5 ( log n ) 2/5 and h 2 ≤ C { ( p / n ) 4 / 5 ( log n ) 2 / 5 + a n 2 } h^2 \le C\{(p/n)^{4/5}(\log n)^{2/5}+a_n^2\} h 2 ≤ C {( p / n ) 4/5 ( log n ) 2/5 + a n 2 } . The optimization error O P ( n − 1 ) O_P(n^{-1}) O P ( n − 1 ) is absorbed. If p ( log n ) 4 / 3 = o ( n ) p(\log n)^{4/3}=o(n) p ( log n ) 4/3 = o ( n ) , then a n / h o p t → 0 a_n/h_{\mathrm{opt}}\to0 a n / h opt → 0 for h o p t = ( p / n ) 2 / 5 ( log n ) 1 / 5 h_{\mathrm{opt}}=(p/n)^{2/5}(\log n)^{1/5} h opt = ( p / n ) 2/5 ( log n ) 1/5 and a n 2 = o ( h o p t ) a_n^2=o(h_{\mathrm{opt}}) a n 2 = o ( h opt ) , proving the sharper remainder.
To compare variances, write u n = H − 1 v n \boldsymbol{u}_n=\boldsymbol{H}^{-1}\boldsymbol{v}_n u n = H − 1 v n , which has bounded norm. The curvature coupling gives
∣ f i n f , h ′ ( x ) − g ( x ) ∣ ≤ ∫ I { ∣ x − c ∣ ≤ C h } ν ( d c ) . |f'_{\mathrm{inf},h}(x)-g(x)|\le\int\mathbb{I}\{|x-c|\le Ch\}\nu(dc). ∣ f inf , h ′ ( x ) − g ( x ) ∣ ≤ ∫ I { ∣ x − c ∣ ≤ C h } ν ( d c ) .
For the index objective, the weighted slab bound makes its squared score difference in the direction u n \boldsymbol{u}_n u n at most C h Ch C h . Moving from θ h \theta_h θ h to θ ∗ \theta^* θ ∗ adds at most C ∣ θ h − θ ∗ ∣ 2 ≤ C h C|\theta_h-\theta^*|^2\le Ch C ∣ θ h − θ ∗ ∣ 2 ≤ C h by the weighted crossing bound. For the shifted residual objective, (15) gives the same bound with the bounded perturbation W − c ω W-c_\omega W − c ω . For integral convolution, Jensen’s inequality and the level shift h V hV hV give C h E ∣ V ∣ Ch\mathbb{E}|V| C h E ∣ V ∣ . Conditional response-density integration gives the same bounds for the residual objective. Subtracting means cannot increase these variances. The smoothed projected score variance therefore differs from the original σ n 2 \sigma_n^2 σ n 2 by O ( h ) O(\sqrt{h}) O ( h ) , and third absolute moments remain bounded. Lyapunov’s theorem gives the normal limit for the score after subtracting its expectation. The condition p = o { n 3 / 8 ( log n ) − 1 / 2 } p=o\{n^{3/8}(\log n)^{-1/2}\} p = o { n 3/8 ( log n ) − 1/2 } makes n \sqrt{n} n times the expansion remainder vanish. Slutsky’s theorem proves the limiting distribution. Centering the unshifted infimal output at θ ∗ \theta^* θ ∗ adds the exact deterministic drift d n d_n d n . If d n → λ d_n\to\lambda d n → λ , Slutsky’s theorem gives N ( λ , 1 ) \mathcal{N}(\lambda,1) N ( λ , 1 ) , while ∣ d n ∣ → ∞ |d_n|\to\infty ∣ d n ∣ → ∞ gives divergence in absolute value in probability. For the losses in Proposition 2, its displacement expansion gives d n = n h v n T B ω / σ n + O ( n h 2 ) d_n=\sqrt{nh}\,\boldsymbol{v}_n^{\mathsf T}\boldsymbol{B}_\omega/\sigma_n+O(\sqrt{n}h^2) d n = nh v n T B ω / σ n + O ( n h 2 ) . The dimension condition for the normal limit makes n h 2 = o ( 1 ) \sqrt{n}h^2=o(1) n h 2 = o ( 1 ) , proving the drift expansion in Corollary 5.
Other objectives
Proposition 9. Consider the index objective (63) under Assumptions 3, 5, and 6, and suppose ρ X \rho_X ρ X has a bounded continuous first derivative. For infimal convolution impose the conditions on f f f and ω \omega ω in Assumption 4, and for integral convolution impose the conditions on K K K in Assumption 4-int. Then Theorem 5 and parts 1 and 2 of Corollary 4 hold for the index objective.
For integral convolution additionally impose (77). Suppose the population Hessian is Lipschitz near θ ∗ \theta^* θ ∗ and m H I ⪯ ∇ 2 L ( θ ∗ ) ⪯ M H I m\boldsymbol{H}_{I}\preceq\nabla^2\mathcal{L}(\theta^*)\preceq M\boldsymbol{H}_{I} m H I ⪯ ∇ 2 L ( θ ∗ ) ⪯ M H I . Then part 3 of Corollary 4 applies. The unshifted infimal conclusion of Corollary 5 and the integral conclusion of Corollary 14 apply with their step sizes, bandwidths, and initialization requirements, using score ψ i = b + X i f + ′ ( X i T θ ∗ ) \boldsymbol{\psi}_i=\boldsymbol{b}+\boldsymbol{X}_if'_+(\boldsymbol{X}_i^{\mathsf T}\theta^*) ψ i = b + X i f + ′ ( X i T θ ∗ ) . For infimal convolution these conclusions do not require (77). The integral condition holds when X \boldsymbol{X} X is bounded or ν \nu ν has compact support, under the first density derivative bound, and for arbitrary ν \nu ν when ρ X \rho_X ρ X has a bounded second derivative.
If f f f has one breakpoint k k k and affine pieces of slopes a < b a<b a < b , the expected index objective has an interior minimizer θ ∗ \theta^* θ ∗ , and its Hessian H \boldsymbol{H} H is positive definite and Lipschitz near that point, then
θ i n f , h ∗ − θ ∗ = h B ω i d x + O ( h 2 ) , B ω i d x = ( b − a ) c ω H − 1 E [ ρ θ ∗ ( k ∣ U ) X θ ∗ ( k ) ] . \theta^*_{\mathrm{inf},h}-\theta^*=h\boldsymbol{B}^{\mathrm{idx}}_\omega+O(h^2),\qquad\boldsymbol{B}^{\mathrm{idx}}_\omega=(b-a)c_\omega\boldsymbol{H}^{-1}\mathbb{E}[\rho_{\theta^*}(k\mid U)\boldsymbol{X}_{\theta^*}(k)]. θ inf , h ∗ − θ ∗ = h B ω idx + O ( h 2 ) , B ω idx = ( b − a ) c ω H − 1 E [ ρ θ ∗ ( k ∣ U ) X θ ∗ ( k )] . with c ω c_\omega c ω as in Proposition 2. For k ≠ 0 k\ne0 k = 0 , B ω i d x = ( c ω / k ) θ ∗ \boldsymbol{B}^{\mathrm{idx}}_\omega=(c_\omega/k)\theta^* B ω idx = ( c ω / k ) θ ∗ . In particular, the Moreau envelope of the hinge loss has displacement − h θ ∗ / 2 + O ( h 2 ) -h\theta^*/2+O(h^2) − h θ ∗ /2 + O ( h 2 ) .
Proof of Proposition 9. The index section moments and all-level crossing bounds in Proposition 6 verify Lemmas 8, 9, and 10. Their proofs above therefore give the increment and Hessian concentration bounds for the index objective. Equations (73), (78), and (79), with the section-derivative condition in the integral case, give the population perturbation bounds. The sufficient conditions for (77) were verified in the proof of Proposition 8. These estimates prove the regularity conclusions and supply every input to the localization, optimization, expansion, and variance calculation in the proof of Corollary 5. For infimal convolution, the expansion is centered at θ i n f , h ∗ \theta^*_{\mathrm{inf},h} θ inf , h ∗ , and the unsmoothed index score gives the limiting variance.
For the displacement coefficient, use the bounded-shift convolution identity in the proof of Proposition 2. It gives
∇ L ‾ i n f , h ( θ ) = b + a E X + ( b − a ) E W B θ ( k + h W ) . \nabla\overline{L}_{\mathrm{inf},h}(\theta)=b+a\mathbb{E}X+(b-a)\mathbb{E}_{W}B_{\theta}(k+hW). ∇ L inf , h ( θ ) = b + a E X + ( b − a ) E W B θ ( k + hW ) .
The first two level derivatives of B θ B_{\theta} B θ are bounded near the fixed level k k k by the conditional density derivative and the section moments. Since E W = c ω \mathbb{E}W=c_{\omega} E W = c ω , Taylor expansion yields
∇ L ‾ i n f , h ( θ ∗ ) = − h ( b − a ) c ω E [ ρ θ ∗ ( k ∣ U ) x θ ∗ ( k ) ] + O ( h 2 ) . \nabla\overline{L}_{\mathrm{inf},h}(\theta^*)=-h(b-a)c_{\omega}\mathbb{E}[\rho_{\theta^*}(k\mid U)x_{\theta^*}(k)]+O(h^2). ∇ L inf , h ( θ ∗ ) = − h ( b − a ) c ω E [ ρ θ ∗ ( k ∣ U ) x θ ∗ ( k )] + O ( h 2 ) .
Proposition 7, the Hessian perturbation (73), and the gradient equation give the coefficient B ω i d x B_{\omega}^{\mathrm{idx}} B ω idx . Since x θ ∗ ( k ) T θ ∗ = k x_{\theta^*}(k)^{\mathsf{T}}\theta^*=k x θ ∗ ( k ) T θ ∗ = k , we have H θ ∗ = ( b − a ) k E [ ρ θ ∗ ( k ∣ U ) x θ ∗ ( k ) ] H_{\theta^*}=(b-a)k\mathbb{E}[\rho_{\theta^*}(k\mid U)x_{\theta^*}(k)] H θ ∗ = ( b − a ) k E [ ρ θ ∗ ( k ∣ U ) x θ ∗ ( k )] . For k ≠ 0 k\ne0 k = 0 this proves B ω i d x = ( c ω / k ) θ ∗ B_{\omega}^{\mathrm{idx}}=(c_{\omega}/k)\theta^* B ω idx = ( c ω / k ) θ ∗ . The hinge loss has a = − 1 a=-1 a = − 1 , b = 0 b=0 b = 0 , and k = 1 k=1 k = 1 . The quadratic prox-function gives c ω = − 1 / 2 c_{\omega}=-1/2 c ω = − 1/2 .
For SVM, f ( t ) = ( 1 − t ) + f(t)=(1-t)_+ f ( t ) = ( 1 − t ) + has curvature measure ν = δ 1 \nu=\delta_1 ν = δ 1 . The density derivative bound in Proposition 9 therefore verifies (77). For integral convolution, smoothing this hinge is equivalent to smoothing x + x_+ x + at 1 − X T θ 1-X^{\mathsf{T}}\theta 1 − X T θ , since K K K is symmetric. For infimal convolution, the latter convention is the former with the reflected prox-function u ↦ ω ( − u ) u\mapsto\omega(-u) u ↦ ω ( − u ) , which satisfies the same assumptions. Both conventions therefore have the regularity and inference conclusions of that proposition, centered at the population minimizer of the chosen smoothed objective in the infimal case.
For rank regression, set f ( x ) = ∣ x ∣ f(x)=|x| f ( x ) = ∣ x ∣ , write r i j ( θ ) = Y i − Y j − ( Z i − Z j ) T θ r_{ij}(\theta)=Y_i-Y_j-(Z_i-Z_j)^{\mathsf{T}}\theta r ij ( θ ) = Y i − Y j − ( Z i − Z j ) T θ , and define
L r a n k , ∘ , h ( θ ) = 1 n ( n − 1 ) ∑ i ≠ j f ∘ , h ( r i j ( θ ) ) , ∘ ∈ { i n f , i n t } . L_{\mathrm{rank},\circ,h}(\theta)=\frac{1}{n(n-1)}\sum_{i\ne j}f_{\circ,h}(r_{ij}(\theta)),\qquad\circ\in\{\mathrm{inf},\mathrm{int}\}. L rank , ∘ , h ( θ ) = n ( n − 1 ) 1 i = j ∑ f ∘ , h ( r ij ( θ )) , ∘ ∈ { inf , int } .
Proposition 10. Impose the distributional assumptions in Example 4, including its density derivative bound, on a convex bounded decision region Θ \Theta Θ . Impose the conditions on the population minimizer and Hessian in Theorem 4 on the pairwise expected objective, and assume a n → 0 a_n\to0 a n → 0 . For infimal convolution impose the conditions on ω \omega ω in Assumption 4, and for integral convolution impose the conditions on K K K in Assumption 4-int. Let A a n ≤ h ≤ h 0 Aa_n\le h\le h_0 A a n ≤ h ≤ h 0 , with A A A sufficiently large and h 0 h_0 h 0 sufficiently small. The conclusions of Theorem 5 and Corollary 4 hold for the pairwise smoothed objective. In particular, its mean-zero gradient increments satisfy the same minimum bound uniformly over decision pairs, with probability at least 1 − C n − γ p 1-Cn^{-\gamma p} 1 − C n − γ p , and its smoothness and strong convexity bounds have ϵ 1 = ϵ 2 = 0 \epsilon_1=\epsilon_2=0 ϵ 1 = ϵ 2 = 0 . Its population minimizers are unique and lie in the interior of Θ \Theta Θ for sufficiently small h h h , with displacements O ( h ) O(h) O ( h ) and O ( h 2 ) O(h^2) O ( h 2 ) for infimal and integral convolution, respectively.
With h ≍ max { a n , ( p / n ) 2 / 5 ( log n ) 1 / 5 } h\asymp\max\{a_n,(p/n)^{2/5}(\log n)^{1/5}\} h ≍ max { a n , ( p / n ) 2/5 ( log n ) 1/5 } and h ≥ A a n h\ge Aa_n h ≥ A a n , the procedures in Corollary 5, with their region and initialization requirements, produce an output after O ( log n ) O(\log n) O ( log n ) updates with distance O P ( n − 1 ) O_P(n^{-1}) O P ( n − 1 ) from the smoothed SAA solution and its expansion with remainder O P { ( p / n ) 4 / 5 ( log n ) 2 / 5 + a n 2 } O_P\{(p/n)^{4/5}(\log n)^{2/5}+a_n^2\} O P {( p / n ) 4/5 ( log n ) 2/5 + a n 2 } . The expansion is centered at θ i n f , h ∗ \theta^*_{\mathrm{inf},h} θ inf , h ∗ for infimal convolution and at θ ∗ \theta^* θ ∗ for integral convolution. For any deterministic unit contrast with unsmoothed rank-score variance bounded below, p = o { n 3 / 8 ( log n ) − 1 / 2 } p=o\{n^{3/8}(\log n)^{-1/2}\} p = o { n 3/8 ( log n ) − 1/2 } gives the limiting distribution in that corollary, with the unsmoothed rank-regression variance in Example 4.
Proof. Let W = ( Y , Z T ) T \mathbf{W}=(Y,\mathbf{Z}^{\mathsf{T}})^{\mathsf{T}} W = ( Y , Z T ) T and let W ′ \mathbf{W}' W ′ be an independent copy. The proof of Example 4 shows that Δ Y = Y − Y ′ \Delta Y=Y-Y' Δ Y = Y − Y ′ conditional on ( Z , Z ′ ) (\mathbf{Z},\mathbf{Z}') ( Z , Z ′ ) has bounded density and bounded first density derivative, and every projection of Δ Z = Z − Z ′ \Delta\mathbf{Z}=\mathbf{Z}-\mathbf{Z}' Δ Z = Z − Z ′ is sub-Gaussian. Since the ordered-pair objective includes both orientations, it equals the U-statistic with scalar loss ℓ ∘ , h ( x ) = { f ∘ , h ( x ) + f ∘ , h ( − x ) } / 2 \ell_{\circ,h}(x)=\{f_{\circ,h}(x)+f_{\circ,h}(-x)\}/2 ℓ ∘ , h ( x ) = { f ∘ , h ( x ) + f ∘ , h ( − x )} /2 . For integral convolution, symmetry of K K K gives ℓ i n t , h = f i n t , h \ell_{\mathrm{int},h}=f_{\mathrm{int},h} ℓ int , h = f int , h . For infimal convolution, its derivative is bounded and nondecreasing, and its curvature is the average of f i n f , h ′ ′ ( x ) f_{\mathrm{inf},h}''(x) f inf , h ′′ ( x ) and f i n f , h ′ ′ ( − x ) f_{\mathrm{inf},h}''(-x) f inf , h ′′ ( − x ) . This curvature has bounded total mass, support in [ − C h , C h ] [-C h,C h] [ − C h , C h ] , and the piecewise Lipschitz property used in Lemma 9. The absolute loss satisfies the piecewise derivative assumptions.
An independent pair therefore satisfies all fixed-decision, crossing, and slab moments in the residual proofs in Appendix § 10 . The permutation moment-generating-function inequality in the proof of Example 4, applied to each projected gradient increment, quadratic form, and nonnegative interpolation bracket, gives the same Bernstein bounds with ⌊ n / 2 ⌋ \lfloor n/2\rfloor ⌊ n /2 ⌋ in place of n n n . On max i ∣ Z i ∣ 2 ≤ C p log n \max_i\lvert Z_i\rvert_2\le C\sqrt{p\log n} max i ∣ Z i ∣ 2 ≤ C p log n , pairwise design differences have at most twice this norm, and twice the bracket width suffices for interpolation. The same nets and union bounds give the increment bound and Hessian fluctuation C { a n / h + a n / h } C\{\sqrt{a_n/h}+a_n/h\} C { a n / h + a n / h } . The mean-zero pointwise gradient deviation has second moment at most C p / n Cp/n C p / n by the permutation representation and Jensen’s inequality. Conditional-density integration for the independent pair gives the population gradient perturbations O ( h ) O(h) O ( h ) and O ( h 2 ) O(h^2) O ( h 2 ) and Hessian perturbation O ( h ) O(h) O ( h ) . Symmetrization preserves these bounds. Thus the regularity and population-minimizer arguments in Appendix § 10 apply.
Put ϑ h = θ i n f , h ∗ \vartheta_h=\theta^*_{\mathrm{inf},h} ϑ h = θ inf , h ∗ for infimal convolution and ϑ h = θ ∗ \vartheta_h=\theta^* ϑ h = θ ∗ for integral convolution. Define
k h ( W , W ′ ) = − Δ Z ℓ ∘ , h ′ ( Δ Y − Δ Z T ϑ h ) , k_h(\mathbf{W},\mathbf{W}')=-\Delta\mathbf{Z}\,\ell'_{\circ,h}(\Delta Y-\Delta\mathbf{Z}^{\mathsf{T}}\vartheta_h), k h ( W , W ′ ) = − Δ Z ℓ ∘ , h ′ ( Δ Y − Δ Z T ϑ h ) ,
ψ h ( W ) = 2 { E [ k h ( W , W ′ ) ∣ W ] − E k h ( W , W ′ ) } . \psi_h(\mathbf{W})=2\{\mathbb{E}[k_h(\mathbf{W},\mathbf{W}')\mid\mathbf{W}]-\mathbb{E}k_h(\mathbf{W},\mathbf{W}')\}. ψ h ( W ) = 2 { E [ k h ( W , W ′ ) ∣ W ] − E k h ( W , W ′ )} .
This kernel is symmetric, since ℓ ∘ , h ′ \ell'_{\circ,h} ℓ ∘ , h ′ is odd. The Hoeffding decomposition gives
∇ L r a n k , ∘ , h ( ϑ h ) − ∇ L ‾ r a n k , ∘ , h ( ϑ h ) = 1 n ∑ i = 1 n ψ h ( W i ) + D n , h , E ∣ D n , h ∣ 2 2 ≤ C p n 2 . \nabla\mathcal{L}_{\mathrm{rank},\circ,h}(\vartheta_h)-\nabla\overline{\mathcal{L}}_{\mathrm{rank},\circ,h}(\vartheta_h)=\frac{1}{n}\sum_{i=1}^{n}\psi_h(\mathbf{W}_i)+D_{n,h},\qquad\mathbb{E}\lvert D_{n,h}\rvert_2^2\le\frac{Cp}{n^2}. ∇ L rank , ∘ , h ( ϑ h ) − ∇ L rank , ∘ , h ( ϑ h ) = n 1 i = 1 ∑ n ψ h ( W i ) + D n , h , E ∣ D n , h ∣ 2 2 ≤ n 2 C p .
Orthogonality of distinct degenerate pairs gives the factor 2 / { n ( n − 1 ) } 2/\{n(n-1)\} 2/ { n ( n − 1 )} multiplying their kernel second moment, which is at most C p Cp C p by ∣ ℓ ∘ , h ′ ∣ ≤ 1 \lvert\ell'_{\circ,h}\rvert\le1 ∣ ℓ ∘ , h ′ ∣ ≤ 1 . Thus ∣ D n , h ∣ 2 = O P ( p / n ) \lvert D_{n,h}\rvert_2=O_{\mathbb{P}}(\sqrt{p}/n) ∣ D n , h ∣ 2 = O P ( p / n ) is absorbed by the remainder.
The population and score bounds permit the localization argument in the proof of Corollary 5. For the complete procedures, the Phase 3 neighborhood is centered at the corresponding smoothed population minimizer. The local comparison (42) gives the stated optimization error. The same increment expansion then gives the remainder for both convolutions. Conditional-density integration for an independent pair bounds the projected second moment of the smoothed score difference by C h C h C h , including the O ( h ) O(h) O ( h ) movement of the infimal population minimizer. Conditional Jensen’s inequality gives the same bound for the difference between ψ h \psi_h ψ h and the original influence vector in Example 4. Subtracting means preserves this bound, and third absolute moments remain bounded. Lyapunov’s theorem and the remainder calculation prove the stated limiting distributions. □ \square □
Regularized SAA
Auxiliary results
Lemma 11. Suppose L \mathcal{L} L is convex and satisfies Condition 1-R. Then, for every θ 1 , θ 2 ∈ Θ r e s \theta_1,\theta_2\in\Theta_{\mathrm{res}} θ 1 , θ 2 ∈ Θ res ,
0 ≤ L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≤ L max { ∣ θ 1 − θ 2 ∣ 2 , ϵ 1 , ∣ θ 1 − θ 2 ∣ 2 } ∣ θ 1 − θ 2 ∣ 2 . (84) 0\le\mathcal{L}(\theta_1)-\mathcal{L}(\theta_2)-\langle\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle\le L\max\{\lvert\theta_1-\theta_2\rvert_{2,\epsilon_1},\lvert\theta_1-\theta_2\rvert_2\}\lvert\theta_1-\theta_2\rvert_2. \tag*{(84)} 0 ≤ L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≤ L max {∣ θ 1 − θ 2 ∣ 2 , ϵ 1 , ∣ θ 1 − θ 2 ∣ 2 } ∣ θ 1 − θ 2 ∣ 2 . (84)
Proof of Lemma 11. Convexity, ∣ θ 1 − θ 2 ∣ 0 ≤ 2 s ~ |\theta_1-\theta_2|_0 \le2\widetilde{s} ∣ θ 1 − θ 2 ∣ 0 ≤ 2 s , and Condition 1-R give
0 ≤ L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≤ ⟨ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≤ L max { ∣ θ 1 − θ 2 ∣ 2 , ϵ 1 } ∣ θ 1 − θ 2 ∣ 2 . 0 \le\mathcal{L}(\theta_1)-\mathcal{L}(\theta_2)-\langle\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle
\le\langle\partial\mathcal{L}(\theta_1)-\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle
\le L\max\{|\theta_1-\theta_2|_2,\epsilon_1\}|\theta_1-\theta_2|_2. 0 ≤ L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≤ ⟨ ∂ L ( θ 1 ) − ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≤ L max { ∣ θ 1 − θ 2 ∣ 2 , ϵ 1 } ∣ θ 1 − θ 2 ∣ 2 .
Lemma 12. The update (18) with the ℓ 0 \ell_0 ℓ 0 -penalty r ( θ ) = ∣ θ ∣ 0 r(\theta)=|\theta|_0 r ( θ ) = ∣ θ ∣ 0 has the closed-form solution
θ l + = θ T , l I { ∣ θ T , l ∣ ≥ 2 η λ } , \theta_l^+ = \theta_{T,l}\mathbb{I}\{|\theta_{T,l}|\ge\sqrt{2\eta\lambda}\}, θ l + = θ T , l I { ∣ θ T , l ∣ ≥ 2 η λ } ,
where θ T = θ − η ∂ L ( θ ) \theta_T=\theta-\eta\partial\mathcal{L}(\theta) θ T = θ − η ∂ L ( θ ) .
Proof of Lemma 12. Multiplying the proximal objective by η \eta η and removing terms independent of β \beta β gives − ⟨ θ T , β ⟩ + η λ ∣ β ∣ 0 + ∣ β ∣ 2 2 / 2 -\langle\theta_T,\beta\rangle+\eta\lambda|\beta|_0+|\beta|_2^2/2 − ⟨ θ T , β ⟩ + η λ ∣ β ∣ 0 + ∣ β ∣ 2 2 /2 , which separates across coordinates. For each l l l , choosing β l = 0 \beta_l=0 β l = 0 gives value zero in its coordinate contribution. Among nonzero choices, the minimum is attained at β l = θ T , l \beta_l=\theta_{T,l} β l = θ T , l when θ T , l ≠ 0 \theta_{T,l}\ne0 θ T , l = 0 , with value η λ − θ T , l 2 / 2 \eta\lambda-\theta_{T,l}^2/2 η λ − θ T , l 2 /2 . Therefore, one minimizer is
θ l + = θ T , l I { ∣ θ T , l ∣ ≥ 2 η λ } . \theta_l^+ = \theta_{T,l}\mathbb{I}\{|\theta_{T,l}|\ge\sqrt{2\eta\lambda}\}. θ l + = θ T , l I { ∣ θ T , l ∣ ≥ 2 η λ } .
At equality either choice minimizes the coordinate contribution. This proves the lemma.
Lemma 13. Under the assumptions of Theorem 7(iii), supp ( θ ( t ) ) = S \operatorname{supp}(\theta^{(t)})=S supp ( θ ( t ) ) = S for every t ≥ 1 t\ge1 t ≥ 1 and ∣ θ ( t ) − θ ∗ ∣ ∞ ≤ d |\theta^{(t)}-\theta^*|_\infty\le d ∣ θ ( t ) − θ ∗ ∣ ∞ ≤ d for every t ≥ 0 t\ge0 t ≥ 0 .
Proof. The bound on the minimum nonzero coefficient gives S ⊆ supp ( θ ( 0 ) ) S\subseteq\operatorname{supp}(\theta^{(0)}) S ⊆ supp ( θ ( 0 ) ) . Suppose that θ ( t ) ∈ B d \theta^{(t)}\in B_d θ ( t ) ∈ B d and S ⊆ supp ( θ ( t ) ) S\subseteq\operatorname{supp}(\theta^{(t)}) S ⊆ supp ( θ ( t ) ) . Its error has at most s ~ \widetilde{s} s nonzero coordinates, hence ∣ θ ( t ) − θ ∗ ∣ 2 ≤ s ~ d ≤ ϵ 3 |\theta^{(t)}-\theta^*|_2\le\sqrt{\widetilde{s}}d\le\epsilon_3 ∣ θ ( t ) − θ ∗ ∣ 2 ≤ s d ≤ ϵ 3 . Define the unthresholded update and the subgradient remainder by
z ( t ) = θ ( t ) − η ∂ L ( θ ( t ) ) , u ( t ) = ∂ L ( θ ( t ) ) − ∂ L ( θ ∗ ) − H ( θ ( t ) − θ ∗ ) . z^{(t)}=\theta^{(t)}-\eta\partial\mathcal{L}(\theta^{(t)}),\qquad u^{(t)}=\partial\mathcal{L}(\theta^{(t)})-\partial\mathcal{L}(\theta^*)-H(\theta^{(t)}-\theta^*). z ( t ) = θ ( t ) − η ∂ L ( θ ( t ) ) , u ( t ) = ∂ L ( θ ( t ) ) − ∂ L ( θ ∗ ) − H ( θ ( t ) − θ ∗ ) .
The definitions of q q q , b n b_n b n , and r n r_n r n give
∣ z ( t ) − θ ∗ ∣ ∞ ≤ q ∣ θ ( t ) − θ ∗ ∣ ∞ + η ( b n + r n ) ≤ q d + η ( b n + r n ) ≤ d . |z^{(t)}-\theta^*|_\infty\le q|\theta^{(t)}-\theta^*|_\infty+\eta(b_n+r_n)\le qd+\eta(b_n+r_n)\le d. ∣ z ( t ) − θ ∗ ∣ ∞ ≤ q ∣ θ ( t ) − θ ∗ ∣ ∞ + η ( b n + r n ) ≤ q d + η ( b n + r n ) ≤ d .
For j ∉ S j\notin S j ∈ / S , this gives ∣ z j ( t ) ∣ ≤ d < 2 η λ |z_j^{(t)}|\le d<\sqrt{2\eta\lambda} ∣ z j ( t ) ∣ ≤ d < 2 η λ . For j ∈ S j\in S j ∈ S , it gives ∣ z j ( t ) ∣ ≥ ∣ θ j ∗ ∣ − d > 2 η λ |z_j^{(t)}|\ge|\theta_j^*|-d>\sqrt{2\eta\lambda} ∣ z j ( t ) ∣ ≥ ∣ θ j ∗ ∣ − d > 2 η λ . Lemma 12 therefore gives supp ( θ ( t + 1 ) ) = S \operatorname{supp}(\theta^{(t+1)})=S supp ( θ ( t + 1 ) ) = S and ∣ θ ( t + 1 ) − θ ∗ ∣ ∞ ≤ d |\theta^{(t+1)}-\theta^*|_\infty\le d ∣ θ ( t + 1 ) − θ ∗ ∣ ∞ ≤ d . Since B d ⊆ Θ B_d\subseteq\Theta B d ⊆ Θ , this proves the induction.
Restricted subgradient increments
Proof of Theorem 6. Let g = f + ′ g=f'_+ g = f + ′ , let ν = d g \nu=dg ν = d g , and write r θ = Y − Z T θ r_\theta=Y-Z^{\mathsf T}\theta r θ = Y − Z T θ . Assumption 3 gives ν ≥ 0 \nu\ge0 ν ≥ 0 and ν ( R ) ≤ 2 C f \nu(\mathbb{R})\le2C_f ν ( R ) ≤ 2 C f . Write P n P_n P n and P P P for the empirical average and expectation, respectively, and let m = s ~ m=\widetilde{s} m = s and a s = m log p / n a_s=m\log p/n a s = m log p / n . The affine term cancels from every subgradient increment.
First consider the bounded-coordinate assumptions. For d = ∣ θ 1 − θ 2 ∣ 2 > 0 d=|\theta_1-\theta_2|_2>0 d = ∣ θ 1 − θ 2 ∣ 2 > 0 , write θ 1 − θ 2 = d u \theta_1-\theta_2=du θ 1 − θ 2 = d u . At every level t t t , conditional-density integration gives
P { t lies between r θ 1 and r θ 2 } ≤ C ρ d P ∣ u T Z ∣ ≤ C d . P\{t\text{ lies between }r_{\theta_1}\text{ and }r_{\theta_2}\}\le C_{\rho}dP|u^{\mathsf T}Z|\le Cd. P { t lies between r θ 1 and r θ 2 } ≤ C ρ d P ∣ u T Z ∣ ≤ C d .
Integrating against ν \nu ν , and using ∣ g ∣ ≤ C f |g| \le C_f ∣ g ∣ ≤ C f and ∣ Z j ∣ ≤ C Z |Z_j| \le C_Z ∣ Z j ∣ ≤ C Z , shows that
W j = Z j { g ( r θ 1 ) − g ( r θ 2 ) } satisfies P ∣ W j ∣ k ≤ d k ! C k , k ≥ 2. W_j = Z_j\{g(r_{\theta_1}) - g(r_{\theta_2})\}\quad\text{satisfies}\quad P|W_j|^k \le d\,k!C^k,\quad k \ge2. W j = Z j { g ( r θ 1 ) − g ( r θ 2 )} satisfies P ∣ W j ∣ k ≤ d k ! C k , k ≥ 2.
Bernstein’s inequality gives ∣ ( P n − P ) W j ∣ ≤ C ( d t / n + t / n ) |(P_n-P)W_j| \le C(\sqrt{dt/n}+t/n) ∣ ( P n − P ) W j ∣ ≤ C ( d t / n + t / n ) with failure probability 2 e − t 2e^{-t} 2 e − t .
Take a p − 12 p^{-12} p − 12 -net of the bounded decision region on each coordinate subspace of dimension at most 2 m 2m 2 m . The union has logarithmic cardinality at most C m log p Cm\log p C m log p . Every pair of m m m -sparse decisions belongs to one of these subspaces. The same is true of all pairs on the segment joining them. A decision and a net representative on the same subspace have residuals within w = C Z 2 m p − 12 w=C_Z\sqrt{2m}p^{-12} w = C Z 2 m p − 12 . Monotonicity bounds their selected-subgradient difference by
B θ ~ = g ( r θ ~ + 2 w ) − f θ ~ ′ ( r θ ~ − 2 w ) . B_{\widetilde{\theta}}=g(r_{\widetilde{\theta}}+2w)-f'_{\widetilde{\theta}}(r_{\widetilde{\theta}}-2w). B θ = g ( r θ + 2 w ) − f θ ′ ( r θ − 2 w ) .
Conditional integration against ν \nu ν gives P B θ ~ k ≤ w k ! C k PB_{\widetilde{\theta}}^k\le wk!C^k P B θ k ≤ w k ! C k . Applying Bernstein’s inequality over the decision nets and coordinates yields
max θ ~ ( P n + P ) B θ ~ ≤ C ( w + a s ) \max_{\widetilde{\theta}}(P_n+P)B_{\widetilde{\theta}}\le C(w+a_s) θ max ( P n + P ) B θ ≤ C ( w + a s )
with failure probability C p − γ m Cp^{-\gamma m} C p − γ m . Since w ≤ C a s w\le Ca_s w ≤ C a s , this bounds the interpolation error in maximum norm by C a s Ca_s C a s . Apply the fixed-pair bound with t = A m log p t=Am\log p t = A m log p over all net pairs and coordinates. Approximating both endpoints proves the asserted maximum-norm increment bound. The strict brackets also cover arbitrary selections at nondifferentiable residuals.
Under the projection bound in Theorem 6(2), test the increment against a unit vector v v v with at most 2 m 2m 2 m nonzero coordinates. If the two decisions have joint support of size at most 2 m 2m 2 m , their displacement direction u u u has the same sparsity bound. The directional moment assumption and conditional-density integration imply, uniformly in the crossing level,
P [ ∣ v T Z ∣ k I { t lies between r θ 1 and r θ 2 } ] ≤ C ρ d P [ ∣ v T Z ∣ k ∣ u T Z ∣ ] ≤ d k ! C k . P\bigl[|v^{\mathsf T}Z|^k\mathbb{I}\{t\text{ lies between }r_{\theta_1}\text{ and }r_{\theta_2}\}\bigr]\le C_{\rho}d\,P[|v^{\mathsf T}Z|^k|u^{\mathsf T}Z|]\le d\,k!C^k. P [ ∣ v T Z ∣ k I { t lies between r θ 1 and r θ 2 } ] ≤ C ρ d P [ ∣ v T Z ∣ k ∣ u T Z ∣ ] ≤ d k ! C k .
Thus the same fixed-pair Bernstein bound holds for W v = v T Z { g ( r θ 1 ) − g ( r θ 2 ) } W_v=v^{\mathsf T}Z\{g(r_{\theta_1})-g(r_{\theta_2})\} W v = v T Z { g ( r θ 1 ) − g ( r θ 2 )} . Use the preceding decision nets and a 1 / 4 1/4 1/4 -net on the unit sphere of each coordinate subspace of dimension at most 2 m 2m 2 m . The total logarithmic cardinality remains C m log p Cm\log p C m log p . A union bound over these sphere nets and the n ≤ p n\le p n ≤ p observations gives
max i ≤ n ∣ Z i ∣ ( 2 m ) ≤ B p = C B m log p \max_{i\le n}|Z_i|_{(2m)}\le B_p=C_B\sqrt{m\log p} i ≤ n max ∣ Z i ∣ ( 2 m ) ≤ B p = C B m log p
with failure probability C p − γ m Cp^{-\gamma m} C p − γ m . Increasing C B C_B C B also makes P [ ∣ v T Z ∣ I { ∣ Z ∣ ( 2 m ) > B p } ] ≤ p − 10 P[|v^{\mathsf T}Z|\mathbb{I}\{|Z|_{(2m)}>B_p\}]\le p^{-10} P [ ∣ v T Z ∣ I { ∣ Z ∣ ( 2 m ) > B p }] ≤ p − 10 uniformly over the test vectors.
On this event the residual interpolation width is w = p − 12 B p w=p^{-12}B_p w = p − 12 B p . The Stieltjes bracket has the moment bound
P [ { ∣ v T Z ∣ B θ ~ } k ] ≤ w k ! C k , k ≥ 1. P\bigl[\{|v^{\mathsf T}Z|B_{\widetilde{\theta}}\}^k\bigr]\le wk!C^k,\quad k\ge1. P [ { ∣ v T Z ∣ B θ } k ] ≤ w k ! C k , k ≥ 1.
Bernstein’s inequality over the decision and sphere nets bounds its empirical average and expectation by C ( w + a s ) C(w+a_s) C ( w + a s ) . The population contribution outside the displayed sample event is at most C p − 10 Cp^{-10} C p − 10 . Hence interpolation again costs at most C a s Ca_s C a s . Applying the fixed-pair bound over the nets, followed by the sphere-net inequality on each test support, gives the stated bound on ∣ ⋅ ∣ ( 2 m ) |\cdot|_{(2m)} ∣ ⋅ ∣ ( 2 m ) . The proof applies to every pair whose joint support has size at most 2 m 2m 2 m , including every pair along the segments used below.
Finally, at θ ∗ \theta^* θ ∗ , each coordinate of Z g ( r θ ∗ ) Zg(r_{\theta^*}) Z g ( r θ ∗ ) is bounded under the first assumptions and sub-Gaussian under the second. A coordinate union bound gives ∣ ∂ L ( θ ∗ ) − ∇ L ( θ ∗ ) ∣ ∞ ≤ C log p / n |\partial\mathcal{L}(\theta^*)-\nabla\mathcal{L}(\theta^*)|_\infty\le C\sqrt{\log p/n} ∣ ∂ L ( θ ∗ ) − ∇ L ( θ ∗ ) ∣ ∞ ≤ C log p / n with failure probability C p − γ Cp^{-\gamma} C p − γ . Intersecting this event with the increment event gives failure probability C p − γ Cp^{-\gamma} C p − γ , while the increment bound alone retains the sharper probability C p − γ m Cp^{-\gamma m} C p − γ m .
Proof of Corollary 6. Write m = s ~ m=\tilde{s} m = s ~ and a s = m log p / n a_s=m\log p/n a s = m log p / n . For unit 2 m 2m 2 m -sparse vectors u , v u,v u , v , conditional-density integration and the sparse projection moments give
P [ ∣ v T Z ∣ ∣ f + ′ ( Y − Z T ( θ + d u ) ) − f + ′ ( Y − Z T θ ) ∣ ] ≤ C d . P\left[\left|v^{\mathsf T}Z\right|\left|f'_+\left(Y-Z^{\mathsf T}(\theta+du)\right)-f'_+\left(Y-Z^{\mathsf T}\theta\right)\right|\right]\le Cd. P [ v T Z f + ′ ( Y − Z T ( θ + d u ) ) − f + ′ ( Y − Z T θ ) ] ≤ C d .
Thus the population gradient increment has ∣ ⋅ ∣ ( 2 m ) \lvert\cdot\rvert_{(2m)} ∣ ⋅ ∣ ( 2 m ) -norm at most C d Cd C d . Combining this bound with Theorem 6, separately for d ≥ a s d\ge a_s d ≥ a s and d < a s d<a_s d < a s , proves part 1.
For part 2, let D = ∣ θ 1 − θ 2 ∣ 2 D=\lvert\theta_1-\theta_2\rvert_2 D = ∣ θ 1 − θ 2 ∣ 2 and θ t = θ 2 + t ( θ 1 − θ 2 ) \theta_t=\theta_2+t(\theta_1-\theta_2) θ t = θ 2 + t ( θ 1 − θ 2 ) . This segment stays in Θ s c \Theta_{\mathrm{sc}} Θ sc , and every pair on it has joint support of size at most 2 m 2m 2 m . Pairing the restricted empirical increment error with its sparse displacement and integrating along the segment gives
L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ μ 0 2 D 2 − C a s D 3 / 2 − C a s D ≥ μ 0 4 D 2 \mathcal{L}(\theta_1)-\mathcal{L}(\theta_2)-\langle\partial\mathcal{L}(\theta_2),\theta_1-\theta_2\rangle\ge\frac{\mu_0}{2}D^2-C\sqrt{a_s}D^{3/2}-Ca_sD\ge\frac{\mu_0}{4}D^2 L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ 2 μ 0 D 2 − C a s D 3/2 − C a s D ≥ 4 μ 0 D 2
when D ≥ C 2 a s D\ge C_2a_s D ≥ C 2 a s , proving part 2.
For part 3, combine the empirical increment bound with the assumed Taylor bound. If D = ∣ θ − θ ∗ ∣ 2 ≤ C 3 a s D=\lvert\theta-\theta^*\rvert_2\le C_3\sqrt{a_s} D = ∣ θ − θ ∗ ∣ 2 ≤ C 3 a s , then
∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∣ ∞ ≤ C { D a s + a s + D 2 } ≤ C 4 a s 3 / 4 . \left|\partial\mathcal{L}(\theta)-\partial\mathcal{L}(\theta^*)-H(\theta-\theta^*)\right|_\infty\le C\left\{\sqrt{Da_s}+a_s+D^2\right\}\le C_4a_s^{3/4}. ∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∣ ∞ ≤ C { D a s + a s + D 2 } ≤ C 4 a s 3/4 .
To verify the Taylor bound under a bounded conditional density derivative, let ν = d f + ′ \nu=df'_+ ν = d f + ′ . Conditional integration gives
H ( θ ) = P [ Z Z T ∫ ρ ( t + Z T θ ∣ Z ) ν ( d t ) ] . H(\theta)=P\left[ZZ^{\mathsf T}\int\rho(t+Z^{\mathsf T}\theta\mid Z)\nu(dt)\right]. H ( θ ) = P [ Z Z T ∫ ρ ( t + Z T θ ∣ Z ) ν ( d t ) ] .
The sparse projection moments imply P [ ∣ v T Z ∣ ∣ Z T ( θ − θ ∗ ) ∣ 2 ] ≤ C ∣ θ − θ ∗ ∣ 2 2 P[\lvert v^{\mathsf T}Z\rvert\lvert Z^{\mathsf T}(\theta-\theta^*)\rvert^2]\le C\lvert\theta-\theta^*\rvert_2^2 P [∣ v T Z ∣ ∣ Z T ( θ − θ ∗ ) ∣ 2 ] ≤ C ∣ θ − θ ∗ ∣ 2 2 for each unit 2 m 2m 2 m -sparse v v v . Taylor’s inequality for the conditional distribution function therefore gives the population remainder in ∣ ⋅ ∣ ( 2 m ) \lvert\cdot\rvert_{(2m)} ∣ ⋅ ∣ ( 2 m ) .
Under the bounded-coordinate assumptions, the first-moment crossing bound gives a population maximum-norm increment at most C d Cd C d . Conversion to ∣ ⋅ ∣ ( 2 m ) \lvert\cdot\rvert_{(2m)} ∣ ⋅ ∣ ( 2 m ) multiplies the smoothness constant by 2 m \sqrt{2m} 2 m . In the strong convexity calculation the scalar empirical error acquires the same factor, and D ≥ C 2 m a s D\ge C_2ma_s D ≥ C 2 m a s suffices. The maximum-norm local approximation has the same remainder C 4 a s 3 / 4 C_4a_s^{3/4} C 4 a s 3/4 . □
Proximal subgradient descent
For Phases 1 and 2, suppose there is a set J 0 J_0 J 0 with ∣ J 0 ∣ ≤ s ~ \lvert J_0\rvert\le\tilde{s} ∣ J 0 ∣ ≤ s ~ , supp ( θ ( 0 ) ) ⊆ J 0 \operatorname{supp}(\theta^{(0)})\subseteq J_0 supp ( θ ( 0 ) ) ⊆ J 0 , and ∣ ∂ j L ( θ ) ∣ ≤ λ \lvert\partial_j\mathcal{L}(\theta)\rvert\le\lambda ∣ ∂ j L ( θ )∣ ≤ λ for every j ∉ J 0 j\notin J_0 j ∈ / J 0 and every θ ∈ Θ \theta\in\Theta θ ∈ Θ supported on J 0 J_0 J 0 . At each such iterate, soft thresholding keeps the next iterate supported on J 0 J_0 J 0 .
Proof of Theorem 7(i). Write x = θ ( t ) x=\theta^{(t)} x = θ ( t ) , x + = θ ( t + 1 ) x^+=\theta^{(t+1)} x + = θ ( t + 1 ) , and g = ∂ L ( x ) g=\partial\mathcal{L}(x) g = ∂ L ( x ) . The proximal objective is 1 / η 1/\eta 1/ η -strongly convex, and its optimality condition gives, for every u ∈ Θ r e s u\in\Theta_{\mathrm{res}} u ∈ Θ res ,
⟨ g , x + − u ⟩ + λ ( ∣ x + ∣ 1 − ∣ u ∣ 1 ) ≤ ∣ x − u ∣ 2 2 − ∣ x + − u ∣ 2 2 − ∣ x + − x ∣ 2 2 2 η . \langle g,x^+-u\rangle+\lambda(\lvert x^+\rvert_1-\lvert u\rvert_1)\le\frac{\lvert x-u\rvert_2^2-\lvert x^+-u\rvert_2^2-\lvert x^+-x\rvert_2^2}{2\eta}. ⟨ g , x + − u ⟩ + λ (∣ x + ∣ 1 − ∣ u ∣ 1 ) ≤ 2 η ∣ x − u ∣ 2 2 − ∣ x + − u ∣ 2 2 − ∣ x + − x ∣ 2 2 .
Inequality (82) gives an upper remainder L max { ∣ x + − x ∣ 2 , ϵ 1 } ∣ x + − x ∣ 2 L\max\{\lvert x^+-x\rvert_2,\epsilon_1\}\lvert x^+-x\rvert_2 L max {∣ x + − x ∣ 2 , ϵ 1 } ∣ x + − x ∣ 2 . After subtracting ∣ x + − x ∣ 2 2 / ( 2 η ) \lvert x^+-x\rvert_2^2/(2\eta) ∣ x + − x ∣ 2 2 / ( 2 η ) , this remainder is nonpositive if ∣ x + − x ∣ 2 ≥ ϵ 1 \lvert x^+-x\rvert_2\ge\epsilon_1 ∣ x + − x ∣ 2 ≥ ϵ 1 , since η ≤ 1 / ( 2 L ) \eta\le1/(2L) η ≤ 1/ ( 2 L ) . Below ϵ 1 \epsilon_1 ϵ 1 , completing the square bounds it by η L 2 ϵ 1 2 / 2 \eta L^2\epsilon_1^2/2 η L 2 ϵ 1 2 /2 . Thus
F ( x + ) − F ( u ) ≤ ∣ x − u ∣ 2 2 − ∣ x + − u ∣ 2 2 2 η − D ( u , x ) + η L 2 ϵ 1 2 2 . (85) F(x^+)-F(u)\le\frac{\lvert x-u\rvert_2^2-\lvert x^+-u\rvert_2^2}{2\eta}-D(u,x)+\frac{\eta L^2\epsilon_1^2}{2}. \tag*{(85)} F ( x + ) − F ( u ) ≤ 2 η ∣ x − u ∣ 2 2 − ∣ x + − u ∣ 2 2 − D ( u , x ) + 2 η L 2 ϵ 1 2 . (85)
where D ( u , x ^ ) = L ( u ) − L ( x ) − ⟨ g , u − x ⟩ ≥ 0 D(u,\hat{x})=\mathcal{L}(u)-\mathcal{L}(x)-\langle g,u-x\rangle\ge0 D ( u , x ^ ) = L ( u ) − L ( x ) − ⟨ g , u − x ⟩ ≥ 0 . Take u = θ ^ u=\hat{\theta} u = θ ^ and sum over t t t to obtain the average-objective bound. If all update lengths are at least ϵ 1 \epsilon_1 ϵ 1 , the last term is absent. Taking u = x u=x u = x then gives F ( x + ) ≤ F ( x ) F(x^+)\le F(x) F ( x + ) ≤ F ( x ) . Hence the final objective gap is bounded by the average gap.
Proof of Theorem 7(ii). Let α = min { η μ / 2 , 1 } \alpha=\min\{\eta\mu/2,1\} α = min { η μ /2 , 1 } . Convexity and Condition 2-R give
D ( θ ^ , θ ( t ) ) ≥ α 2 η { ∣ θ ( t ) − θ ^ ∣ 2 2 − ϵ 2 2 } . D(\hat{\theta},\theta^{(t)})\ge\frac{\alpha}{2\eta}\left\{\left\lvert\theta^{(t)}-\hat{\theta}\right\rvert_2^2-\epsilon_2^2\right\}. D ( θ ^ , θ ( t ) ) ≥ 2 η α { θ ( t ) − θ ^ 2 2 − ϵ 2 2 } .
Indeed, below ϵ 2 \epsilon_2 ϵ 2 the right side is nonpositive. Above it, use α / ( 2 η ) ≤ μ / 4 \alpha/(2\eta)\le\mu/4 α / ( 2 η ) ≤ μ /4 . Since F ( θ ( t + 1 ) ) ≥ F ( θ ^ ) F(\theta^{(t+1)})\ge F(\hat{\theta}) F ( θ ( t + 1 ) ) ≥ F ( θ ^ ) , (83) gives
∣ θ ( t + 1 ) − θ ^ ∣ 2 2 ≤ ( 1 − α ) ∣ θ ( t ) − θ ^ ∣ 2 2 + α ϵ 2 2 + η 2 L 2 ϵ 1 2 . \left\lvert\theta^{(t+1)}-\hat{\theta}\right\rvert_2^2\le(1-\alpha)\left\lvert\theta^{(t)}-\hat{\theta}\right\rvert_2^2+\alpha\epsilon_2^2+\eta^2L^2\epsilon_1^2. θ ( t + 1 ) − θ ^ 2 2 ≤ ( 1 − α ) θ ( t ) − θ ^ 2 2 + α ϵ 2 2 + η 2 L 2 ϵ 1 2 .
Iteration and summation of the geometric series prove the stated bound. When both distance conditions hold, the two additive terms are absent at every step, which proves the sharper conclusion.
Proof of Theorem 7(iii). By Lemma 13, the iterates have support S S S for t ≥ 1 t\ge1 t ≥ 1 . Using u ( t ) u^{(t)} u ( t ) from its proof, the update on S S S is
{ θ ( t + 1 ) − θ ∗ } S = ( I − η H S × S ) { θ ( t ) − θ ∗ } S − η { ∂ L ( θ ∗ ) } S − η u S ( t ) . \{\theta^{(t+1)}-\theta^*\}_S=(I-\eta H_{S\times S})\{\theta^{(t)}-\theta^*\}_S-\eta\{\partial\mathcal{L}(\theta^*)\}_S-\eta u_S^{(t)}. { θ ( t + 1 ) − θ ∗ } S = ( I − η H S × S ) { θ ( t ) − θ ∗ } S − η { ∂ L ( θ ∗ ) } S − η u S ( t ) .
where ∣ u ( t ) ∣ ∞ ≤ r n \lvert u^{(t)}\rvert_\infty\le r_n ∣ u ( t ) ∣ ∞ ≤ r n . The principal submatrix satisfies ∥ I − η H S × S ∥ ∞ ≤ q < 1 \lVert I-\eta H_{S\times S}\rVert_\infty\le q<1 ∥ I − η H S × S ∥ ∞ ≤ q < 1 , and the Neumann series gives
∥ H S × S − 1 ∥ ∞ ≤ η ∑ k = 0 ∞ q k = η 1 − q . \lVert H_{S\times S}^{-1}\rVert_\infty\le\eta\sum_{k=0}^{\infty}q^k=\frac{\eta}{1-q}. ∥ H S × S − 1 ∥ ∞ ≤ η k = 0 ∑ ∞ q k = 1 − q η .
Adding H S × S − 1 { ∂ L ( θ ∗ ) } S H_{S\times S}^{-1}\{\partial\mathcal{L}(\theta^*)\}_S H S × S − 1 { ∂ L ( θ ∗ ) } S to the recursion and iterating from t = 1 t=1 t = 1 gives
∣ { θ ( T ) − θ ∗ } S + H S × S − 1 { ∂ L ( θ ∗ ) } S ∣ ∞ ≤ q T − 1 ( d + η b n 1 − q ) + η r n ∑ k = 0 T − 2 q k . \left\lvert\{\theta^{(T)}-\theta^*\}_S+H_{S\times S}^{-1}\{\partial\mathcal{L}(\theta^*)\}_S\right\rvert_\infty\le q^{T-1}\left(d+\frac{\eta b_n}{1-q}\right)+\eta r_n\sum_{k=0}^{T-2}q^k. { θ ( T ) − θ ∗ } S + H S × S − 1 { ∂ L ( θ ∗ ) } S ∞ ≤ q T − 1 ( d + 1 − q η b n ) + η r n k = 0 ∑ T − 2 q k .
with the sum understood as zero when T = 1 T=1 T = 1 . This proves (19).
Choosing d ≳ b n + r n d\gtrsim b_n+r_n d ≳ b n + r n and λ ≳ d 2 \lambda\gtrsim d^2 λ ≳ d 2 gives the order λ ≳ b n 2 \lambda\gtrsim b_n^2 λ ≳ b n 2 when r n = O ( b n ) r_n=O(b_n) r n = O ( b n ) ([6 , 8 ] ).
The support recovery and expansion also extend to SCAD, MCP, and capped-ℓ 1 \ell_1 ℓ 1 penalties. For a separable penalty λ r ( θ ) = ∑ j = 1 p p λ ( θ j ) \lambda r(\theta)=\sum_{j=1}^{p}p_\lambda(\theta_j) λ r ( θ ) = ∑ j = 1 p p λ ( θ j ) , write P η ( u ) = argmin v ∈ R { p λ ( v ) + ( v − u ) 2 / ( 2 η ) } P_\eta(u)=\operatorname*{argmin}_{v\in\mathbb{R}}\{p_\lambda(v)+(v-u)^2/(2\eta)\} P η ( u ) = argmin v ∈ R { p λ ( v ) + ( v − u ) 2 / ( 2 η )} . Suppose P η ( u ) = { 0 } P_\eta(u)=\{0\} P η ( u ) = { 0 } for ∣ u ∣ < τ 0 \lvert u\rvert<\tau_0 ∣ u ∣ < τ 0 and P η ( u ) = { u } P_\eta(u)=\{u\} P η ( u ) = { u } for ∣ u ∣ > τ 1 \lvert u\rvert>\tau_1 ∣ u ∣ > τ 1 , where d < τ 0 ≤ τ 1 < min j ∈ S ∣ θ j ∗ ∣ − d d<\tau_0\le\tau_1<\min_{j\in S}\lvert\theta_j^*\rvert-d d < τ 0 ≤ τ 1 < min j ∈ S ∣ θ j ∗ ∣ − d . Replacing the ℓ 0 \ell_0 ℓ 0 -threshold condition in Theorem 7(iii) by this separation gives the same support recovery after one update and expansion (19).
Proof of the penalty extension in Section 6.2. The threshold separation gives min j ∈ S ∣ θ j ∗ ∣ > 2 d \min_{j\in S}\lvert\theta_j^*\rvert>2d min j ∈ S ∣ θ j ∗ ∣ > 2 d , hence S ⊆ supp ( θ ( 0 ) ) S\subseteq\operatorname{supp}(\theta^{(0)}) S ⊆ supp ( θ ( 0 ) ) . For θ ∈ B d \theta\in B_d θ ∈ B d containing S S S in its support, write z = θ − η ∂ L ( θ ) z=\theta-\eta\partial\mathcal{L}(\theta) z = θ − η ∂ L ( θ ) . The neighborhood, contraction, and initialization conditions of Theorem 7(iii) give ∣ z − θ ∗ ∣ ∞ ≤ q d + η ( b n + r n ) ≤ d \lvert z-\theta^*\rvert_\infty\le qd+\eta(b_n+r_n)\le d ∣ z − θ ∗ ∣ ∞ ≤ q d + η ( b n + r n ) ≤ d . Thus ∣ z j ∣ ≤ d < τ 0 \lvert z_j\rvert\le d<\tau_0 ∣ z j ∣ ≤ d < τ 0 for j ∉ S j\notin S j ∈ / S , whereas ∣ z j ∣ ≥ ∣ θ j ∗ ∣ − d > τ 1 \lvert z_j\rvert\ge\lvert\theta_j^*\rvert-d>\tau_1 ∣ z j ∣ ≥ ∣ θ j ∗ ∣ − d > τ 1 for j ∈ S j\in S j ∈ S . Every coordinate proximal minimizer therefore equals zero outside S S S and equals z j z_j z j on S S S . The updated vector has support S S S and remains in B d B_d B d . The induction in Lemma 13 and the unpenalized recursion in the proof of Theorem 7(iii) then give (19).
Iterative hard thresholding
Proof of Theorem 8(i). Define g t = ∂ L ( θ ( t ) ) g_t = \partial\mathcal{L}(\theta^{(t)}) g t = ∂ L ( θ ( t ) ) and Q t ( u ) = ⟨ g t , u − θ ( t ) ⟩ + ( 2 η ) − 1 ∣ u − θ ( t ) ∣ 2 2 Q_t(u) = \langle g_t, u-\theta^{(t)}\rangle+ (2\eta)^{-1}\lvert u-\theta^{(t)}\rvert_2^2 Q t ( u ) = ⟨ g t , u − θ ( t ) ⟩ + ( 2 η ) − 1 ∣ u − θ ( t ) ∣ 2 2 . The IHT update minimizes Q t Q_t Q t over all s ~ \tilde{s} s ~ -sparse vectors. Inequality (84) yields
L ( θ ( t + 1 ) ) ≤ L ( θ ( t ) ) + Q t ( θ ( t + 1 ) ) − ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 2 2 η + L max { ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 , ϵ 1 } ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 . (86) \mathcal{L}(\theta^{(t+1)}) \leq\mathcal{L}(\theta^{(t)}) + Q_t(\theta^{(t+1)}) - \frac{\lvert\theta^{(t+1)}-\theta^{(t)}\rvert_2^2}{2\eta} + L\max\{\lvert\theta^{(t+1)}-\theta^{(t)}\rvert_2,\epsilon_1\}\lvert\theta^{(t+1)}-\theta^{(t)}\rvert_2. \tag*{(86)} L ( θ ( t + 1 ) ) ≤ L ( θ ( t ) ) + Q t ( θ ( t + 1 ) ) − 2 η ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 2 + L max {∣ θ ( t + 1 ) − θ ( t ) ∣ 2 , ϵ 1 } ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 . (86)
Since η ≤ 1 / ( 4 L ) \eta\leq1/(4L) η ≤ 1/ ( 4 L ) , the last two terms are at most − ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 2 / ( 4 η ) -\lvert\theta^{(t+1)}-\theta^{(t)}\rvert_2^2/(4\eta) − ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 2 / ( 4 η ) when the update length is at least ϵ 1 \epsilon_1 ϵ 1 . Below ϵ 1 \epsilon_1 ϵ 1 , completing the square gives the same bound with additive error η L 2 ϵ 1 2 \eta L^2\epsilon_1^2 η L 2 ϵ 1 2 . Since Q t ( θ ( t + 1 ) ) ≤ Q t ( θ ( t ) ) = 0 Q_t(\theta^{(t+1)}) \leq Q_t(\theta^{(t)}) = 0 Q t ( θ ( t + 1 ) ) ≤ Q t ( θ ( t ) ) = 0 ,
∣ θ ( t + 1 ) − θ ( t ) ∣ 2 2 4 η ≤ L ( θ ( t ) ) − L ( θ ( t + 1 ) ) + η L 2 ϵ 1 2 . \frac{\lvert\theta^{(t+1)}-\theta^{(t)}\rvert_2^2}{4\eta} \leq\mathcal{L}(\theta^{(t)})-\mathcal{L}(\theta^{(t+1)})+\eta L^2\epsilon_1^2. 4 η ∣ θ ( t + 1 ) − θ ( t ) ∣ 2 2 ≤ L ( θ ( t ) ) − L ( θ ( t + 1 ) ) + η L 2 ϵ 1 2 .
Summing proves the claim. If every update length is at least ϵ 1 \epsilon_1 ϵ 1 , the final term is absent.
Proof of Theorem 8(ii). Let α = min { η μ / 2 , 1 } \alpha= \min\{\eta\mu/2,1\} α = min { η μ /2 , 1 } , u t = ( 1 − α ) θ ( t ) + α θ ∗ u_t = (1-\alpha)\theta^{(t)}+\alpha\theta^* u t = ( 1 − α ) θ ( t ) + α θ ∗ , and Q t ( u ) = ⟨ ∂ L ( θ ( t ) ) , u − θ ( t ) ⟩ + ( 2 η ) − 1 ∣ u − θ ( t ) ∣ 2 2 Q_t(u) = \langle\partial\mathcal{L}(\theta^{(t)}),u-\theta^{(t)}\rangle+(2\eta)^{-1}\lvert u-\theta^{(t)}\rvert_2^2 Q t ( u ) = ⟨ ∂ L ( θ ( t ) ) , u − θ ( t ) ⟩ + ( 2 η ) − 1 ∣ u − θ ( t ) ∣ 2 2 . Since S ⊆ supp ( θ ( t ) ) S\subseteq\operatorname{supp}(\theta^{(t)}) S ⊆ supp ( θ ( t ) ) , the vector u t u_t u t is s ~ \tilde{s} s ~ -sparse. Comparison with u t u_t u t in the IHT update gives
Q t ( θ ( t + 1 ) ) ≤ Q t ( u t ) = α ⟨ ∂ L ( θ ( t ) ) , θ ∗ − θ ( t ) ⟩ + α 2 2 η ∣ θ ( t ) − θ ∗ ∣ 2 2 . Q_t(\theta^{(t+1)}) \leq Q_t(u_t) = \alpha\langle\partial\mathcal{L}(\theta^{(t)}),\theta^*-\theta^{(t)}\rangle+ \frac{\alpha^2}{2\eta}\lvert\theta^{(t)}-\theta^*\rvert_2^2. Q t ( θ ( t + 1 ) ) ≤ Q t ( u t ) = α ⟨ ∂ L ( θ ( t ) ) , θ ∗ − θ ( t ) ⟩ + 2 η α 2 ∣ θ ( t ) − θ ∗ ∣ 2 2 .
By convexity and Condition 2-R,
⟨ ∂ L ( θ ( t ) ) , θ ∗ − θ ( t ) ⟩ ≤ L ( θ ∗ ) − L ( θ ( t ) ) − μ 4 ∣ θ ( t ) − θ ∗ ∣ 2 2 + μ 4 ϵ 2 2 . \langle\partial\mathcal{L}(\theta^{(t)}),\theta^*-\theta^{(t)}\rangle\leq\mathcal{L}(\theta^*)-\mathcal{L}(\theta^{(t)})-\frac{\mu}{4}\lvert\theta^{(t)}-\theta^*\rvert_2^2+\frac{\mu}{4}\epsilon_2^2. ⟨ ∂ L ( θ ( t ) ) , θ ∗ − θ ( t ) ⟩ ≤ L ( θ ∗ ) − L ( θ ( t ) ) − 4 μ ∣ θ ( t ) − θ ∗ ∣ 2 2 + 4 μ ϵ 2 2 .
Write Δ t = L ( θ ( t ) ) − L ( θ ∗ ) \Delta_t=\mathcal{L}(\theta^{(t)})-\mathcal{L}(\theta^*) Δ t = L ( θ ( t ) ) − L ( θ ∗ ) . For η ≤ 1 / ( 2 L ) \eta\leq1/(2L) η ≤ 1/ ( 2 L ) , the last two terms in (86) are bounded by η L 2 ϵ 1 2 / 2 \eta L^2\epsilon_1^2/2 η L 2 ϵ 1 2 /2 , as shown in the proof of Theorem 7(i). Substituting the bound for Q t ( u t ) Q_t(u_t) Q t ( u t ) gives
Δ t + 1 ≤ ( 1 − α ) Δ t − ( α μ 4 − α 2 2 η ) ∣ θ ( t ) − θ ∗ ∣ 2 2 + α μ ϵ 2 2 4 + η L 2 ϵ 1 2 2 . \Delta_{t+1}\leq(1-\alpha)\Delta_t-\left(\frac{\alpha\mu}{4}-\frac{\alpha^2}{2\eta}\right)\lvert\theta^{(t)}-\theta^*\rvert_2^2+\frac{\alpha\mu\epsilon_2^2}{4}+\frac{\eta L^2\epsilon_1^2}{2}. Δ t + 1 ≤ ( 1 − α ) Δ t − ( 4 α μ − 2 η α 2 ) ∣ θ ( t ) − θ ∗ ∣ 2 2 + 4 α μ ϵ 2 2 + 2 η L 2 ϵ 1 2 .
The squared-distance coefficient is nonnegative since α ≤ η μ / 2 \alpha\leq\eta\mu/2 α ≤ η μ /2 . Drop that term and iterate the resulting scalar inequality. The sum of the geometric weights is at most 1 / α 1/\alpha 1/ α , which gives the remainder. Under the two distance conditions, both additive terms vanish before iteration.
Proof of Theorem 8(iii). The bound on the minimum nonzero coefficient and θ ( 0 ) ∈ B d \theta^{(0)}\in B_d θ ( 0 ) ∈ B d give S ⊆ supp ( θ ( 0 ) ) S\subseteq\operatorname{supp}(\theta^{(0)}) S ⊆ supp ( θ ( 0 ) ) . Suppose θ ( t ) ∈ B d \theta^{(t)}\in B_d θ ( t ) ∈ B d and S ⊆ supp ( θ ( t ) ) S\subseteq\operatorname{supp}(\theta^{(t)}) S ⊆ supp ( θ ( t ) ) . The error has at most s ~ \tilde{s} s ~ nonzero coordinates, hence ∣ θ ( t ) − θ ∗ ∣ 2 ≤ s ~ d ≤ ϵ 3 \lvert\theta^{(t)}-\theta^*\rvert_2\leq\sqrt{\tilde{s}}d\leq\epsilon_3 ∣ θ ( t ) − θ ∗ ∣ 2 ≤ s ~ d ≤ ϵ 3 . Define z ( t ) z^{(t)} z ( t ) and u ( t ) u^{(t)} u ( t ) as in the proof of Lemma 13. Then
∣ z ( t ) − θ ∗ ∣ ∞ ≤ q ∣ θ ( t ) − θ ∗ ∣ ∞ + η ( b n + r n ) ≤ d . \lvert z^{(t)}-\theta^*\rvert_\infty\leq q\lvert\theta^{(t)}-\theta^*\rvert_\infty+\eta(b_n+r_n)\leq d. ∣ z ( t ) − θ ∗ ∣ ∞ ≤ q ∣ θ ( t ) − θ ∗ ∣ ∞ + η ( b n + r n ) ≤ d .
For j ∈ S j\in S j ∈ S , the inequality ∣ z j ( t ) ∣ ≥ ∣ θ j ∗ ∣ − d > d \lvert z_j^{(t)}\rvert\geq\lvert\theta_j^*\rvert-d>d ∣ z j ( t ) ∣ ≥ ∣ θ j ∗ ∣ − d > d holds. For j ∉ S j\notin S j ∈ / S , ∣ z j ( t ) ∣ ≤ d \lvert z_j^{(t)}\rvert\leq d ∣ z j ( t ) ∣ ≤ d . Since s ~ ≥ s \tilde{s}\geq s s ~ ≥ s , hard thresholding retains every coordinate in S S S . It follows that
∣ θ ( t + 1 ) − θ ∗ ∣ ∞ ≤ q ∣ θ ( t ) − θ ∗ ∣ ∞ + η ( b n + r n ) . \lvert\theta^{(t+1)}-\theta^*\rvert_\infty\leq q\lvert\theta^{(t)}-\theta^*\rvert_\infty+\eta(b_n+r_n). ∣ θ ( t + 1 ) − θ ∗ ∣ ∞ ≤ q ∣ θ ( t ) − θ ∗ ∣ ∞ + η ( b n + r n ) .
The iterate θ ( t + 1 ) \theta^{(t+1)} θ ( t + 1 ) has at most s ~ \tilde{s} s ~ nonzero coordinates and belongs to Θ \Theta Θ by B d ⊆ Θ \mathcal{B}_d \subseteq\Theta B d ⊆ Θ . Induction gives the support inclusion, invariance of B d \mathcal{B}_d B d , and geometric bound. When s ~ = s \tilde{s}=s s ~ = s , every iterate has support S S S , including t = 0 t=0 t = 0 .
To recover S S S when s s s is unknown, IHT can instead be followed by ℓ 0 \ell_0 ℓ 0 -penalized PGD. Under the conditions of Theorem 8(iii), the next unthresholded iterate has maximum-norm error at most d d d . Every threshold satisfying d < 2 η λ < min j ∈ S ∣ θ j ∗ ∣ − d d < \sqrt{2\eta\lambda} < \min_{j\in S}|\theta_j^*|-d d < 2 η λ < min j ∈ S ∣ θ j ∗ ∣ − d removes all coordinates outside S S S and retains all coordinates in S S S . The theorem’s minimum-coefficient condition makes this interval nonempty.
Proof of Theorem 8(iv). The joint increment and score event in Theorem 6 has probability at least 1 − C p − γ 1-Cp^{-\gamma} 1 − C p − γ . On this event, every retained set J ⊇ S J \supseteq S J ⊇ S of size s ~ \tilde{s} s ~ satisfies ∣ g J ∣ 2 ≤ s ~ ∣ ∂ L ( θ ∗ ) ∣ ∞ ≤ C g s ~ a s \lvert g_J\rvert_2 \le\sqrt{\tilde{s}}\lvert\partial\mathcal{L}(\theta^*)\rvert_\infty\le C_g\sqrt{\tilde{s}a_s} ∣ g J ∣ 2 ≤ s ~ ∣ ∂ L ( θ ∗ ) ∣ ∞ ≤ C g s ~ a s . The assumed population Taylor bound gives its Euclidean remainder on each retained set. Consequently, for every θ ∈ Θ \theta\in\Theta θ ∈ Θ supported on J J J with D = ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 D=\lvert\theta-\theta^*\rvert_2\le\epsilon_3 D = ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 ,
∣ { ∂ L ( θ ) − ∂ L ( θ ∗ ) } J − H J × J ( θ − θ ∗ ) J ∣ 2 ≤ C { D a s + a s + D 2 } . \left\lvert\{\partial\mathcal{L}(\theta)-\partial\mathcal{L}(\theta^*)\}_J-H_{J\times J}(\theta-\theta^*)_J\right\rvert_2\le C\{\sqrt{Da_s}+a_s+D^2\}. ∣ { ∂ L ( θ ) − ∂ L ( θ ∗ ) } J − H J × J ( θ − θ ∗ ) J ∣ 2 ≤ C { D a s + a s + D 2 } .
Intersect this event with E n \mathcal{E}_n E n and write J = J ^ J=\widehat{J} J = J . Define u t = { ∂ L ( θ ( t ) ) − ∂ L ( θ ∗ ) } J − H J × J e t u_t=\{\partial\mathcal{L}(\theta^{(t)})-\partial\mathcal{L}(\theta^*)\}_J-H_{J\times J}e_t u t = { ∂ L ( θ ( t ) ) − ∂ L ( θ ∗ ) } J − H J × J e t . Since η ≤ M H − 1 \eta\le M_H^{-1} η ≤ M H − 1 , the matrix A J = I − η H J × J A_J=I-\eta H_{J\times J} A J = I − η H J × J satisfies ∥ A J ∥ ≤ 1 − η m H \lVert A_J\rVert\le1-\eta m_H ∥ A J ∥ ≤ 1 − η m H . When an update retains J J J , its error follows e t + 1 = A J e t − η g J − η u t e_{t+1}=A_Je_t-\eta g_J-\eta u_t e t + 1 = A J e t − η g J − η u t . Young’s inequality gives C D a s ≤ m H D / 4 + C 2 a s / m H C\sqrt{Da_s}\le m_HD/4+C^2a_s/m_H C D a s ≤ m H D /4 + C 2 a s / m H . For sufficiently large n n n , D ≤ ϵ 3 D\le\epsilon_3 D ≤ ϵ 3 also gives C D 2 ≤ m H D / 4 CD^2\le m_HD/4 C D 2 ≤ m H D /4 . Hence, for a constant C u C_u C u ,
∣ e t + 1 ∣ 2 ≤ ( 1 − η m H / 2 ) ∣ e t ∣ 2 + η { ∣ g J ∣ 2 + C u a s } . \lvert e_{t+1}\rvert_2\le(1-\eta m_H/2)\lvert e_t\rvert_2+\eta\{\lvert g_J\rvert_2+C_ua_s\}. ∣ e t + 1 ∣ 2 ≤ ( 1 − η m H /2 ) ∣ e t ∣ 2 + η {∣ g J ∣ 2 + C u a s } .
Choose C 3 ≥ 4 ( C g + C u ) / m H C_3\ge4(C_g+C_u)/m_H C 3 ≥ 4 ( C g + C u ) / m H . Since a s ≤ 1 a_s\le1 a s ≤ 1 eventually, the forcing term is at most η m H ϵ 3 / 4 \eta m_H\epsilon_3/4 η m H ϵ 3 /4 . Every current error of norm at most ϵ 3 \epsilon_3 ϵ 3 therefore produces an error of norm at most ( 1 − η m H / 4 ) ϵ 3 (1-\eta m_H/4)\epsilon_3 ( 1 − η m H /4 ) ϵ 3 . The region assumption places this next iterate in Θ \Theta Θ , completing the containment induction. Iterating the displayed recursion proves the first bound.
Containment also gives ∣ u t ∣ 2 ≤ C a s 3 / 4 \lvert u_t\rvert_2\le Ca_s^{3/4} ∣ u t ∣ 2 ≤ C a s 3/4 . Apply the recursion to e t + H J × J − 1 g J e_t+H_{J\times J}^{-1}g_J e t + H J × J − 1 g J . The geometric kernel has total mass at most η / { 1 − ( 1 − η m H ) } = m H − 1 \eta/\{1-(1-\eta m_H)\}=m_H^{-1} η / { 1 − ( 1 − η m H )} = m H − 1 , proving the expansion bound.
A positive step size with q = ∥ I − η H ∥ ∞ < 1 q=\lVert I-\eta H\rVert_\infty<1 q = ∥ I − η H ∥ ∞ < 1 exists if and only if H i i > ∑ j ≠ i ∣ H i j ∣ H_{ii}>\sum_{j\ne i}\lvert H_{ij}\rvert H ii > ∑ j = i ∣ H ij ∣ for every i i i . The admissible step sizes are exactly 0 < η < min i 2 / ( H i i + ∑ j ≠ i ∣ H i j ∣ ) 0<\eta<\min_i 2/(H_{ii}+\sum_{j\ne i}\lvert H_{ij}\rvert) 0 < η < min i 2/ ( H ii + ∑ j = i ∣ H ij ∣) . This follows from
q = max i { ∣ 1 − η H i i ∣ + η ∑ j ≠ i ∣ H i j ∣ } . q=\max_i\left\{\lvert1-\eta H_{ii}\rvert+\eta\sum_{j\ne i}\lvert H_{ij}\rvert\right\}. q = i max ⎩ ⎨ ⎧ ∣ 1 − η H ii ∣ + η j = i ∑ ∣ H ij ∣ ⎭ ⎬ ⎫ .
For 0 < η ≤ 1 / max i H i i 0<\eta\le1/\max_i H_{ii} 0 < η ≤ 1/ max i H ii , we have q = 1 − η min i ( H i i − ∑ j ≠ i ∣ H i j ∣ ) q=1-\eta\min_i(H_{ii}-\sum_{j\ne i}\lvert H_{ij}\rvert) q = 1 − η min i ( H ii − ∑ j = i ∣ H ij ∣) and ∥ H − 1 ∥ ∞ ≤ { min i ( H i i − ∑ j ≠ i ∣ H i j ∣ ) } − 1 \lVert H^{-1}\rVert_\infty\le\{\min_i(H_{ii}-\sum_{j\ne i}\lvert H_{ij}\rvert)\}^{-1} ∥ H − 1 ∥ ∞ ≤ { min i ( H ii − ∑ j = i ∣ H ij ∣) } − 1 . A lower bound on the row margins then gives q ≤ q 0 < 1 q\le q_0<1 q ≤ q 0 < 1 . For the inverse bound, choose a coordinate with ∣ v i ∣ = ∣ v ∣ ∞ \lvert v_i\rvert=\lvert v\rvert_\infty ∣ v i ∣ = ∣ v ∣ ∞ and use
∣ H v ∣ ∞ ≥ { H i i − ∑ j ≠ i ∣ H i j ∣ } ∣ v ∣ ∞ . \lvert Hv\rvert_\infty\ge\{H_{ii}-\sum_{j\ne i}\lvert H_{ij}\rvert\}\lvert v\rvert_\infty. ∣ H v ∣ ∞ ≥ { H ii − j = i ∑ ∣ H ij ∣} ∣ v ∣ ∞ .
Corollary 15. Let m = s ~ m=\tilde{s} m = s ~ , 1 ≤ s < m ≤ p 1\le s<m\le p 1 ≤ s < m ≤ p , p > n ≥ 2 p>n\ge2 p > n ≥ 2 , and a s = m log p / n → 0 a_s=m\log p/n\to0 a s = m log p / n → 0 . Consider the residual objective under the assumptions of Theorem 6(2). Let θ ∗ \theta^* θ ∗ be an interior stationary point of the expected objective, let S = supp ( θ ∗ ) S=\operatorname{supp}(\theta^*) S = supp ( θ ∗ ) have size s s s , and suppose B ( θ ∗ , R ) ⊆ Θ \mathbb{B}(\theta^*,R)\subseteq\Theta B ( θ ∗ , R ) ⊆ Θ for R > 0 R>0 R > 0 . Assume that 0 < μ 0 ≤ L 0 0<\mu_0\le L_0 0 < μ 0 ≤ L 0 and, for every index set I ⊇ S I\supseteq S I ⊇ S with ∣ I ∣ ≤ 2 m + s \lvert I\rvert\le2m+s ∣ I ∣ ≤ 2 m + s ,
μ 0 I I ⪯ ∇ 2 L ( θ ) I × I ⪯ L 0 I I \mu_0 I_I\preceq\nabla^2\mathcal{L}(\theta)_{I\times I}\preceq L_0I_I μ 0 I I ⪯ ∇ 2 L ( θ ) I × I ⪯ L 0 I I whenever supp ( θ ) ⊆ I \operatorname{supp}(\theta)\subseteq I supp ( θ ) ⊆ I and ∣ θ − θ ∗ ∣ 2 ≤ R \lvert\theta-\theta^*\rvert_2\le R ∣ θ − θ ∗ ∣ 2 ≤ R . Put κ 0 = L 0 / μ 0 \kappa_0=L_0/\mu_0 κ 0 = L 0 / μ 0 , suppose m − s ≥ 64 κ 0 2 s m-s\ge64\kappa_0^2s m − s ≥ 64 κ 0 2 s , and run IHT with working sparsity m m m and step size η = L 0 − 1 \eta=L_0^{-1} η = L 0 − 1 . There exist C , γ , K , c > 0 C,\gamma,K,c>0 C , γ , K , c > 0 such that, for sufficiently large n n n , on one event of probability at least 1 − C p − γ 1-Cp^{-\gamma} 1 − C p − γ , every trajectory initialized at an m m m -sparse θ ( 0 ) \theta^{(0)} θ ( 0 ) with ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ R / 2 \lvert\theta^{(0)}-\theta^*\rvert_2\le R/2 ∣ θ ( 0 ) − θ ∗ ∣ 2 ≤ R /2 remains in B ( θ ∗ , R ) \mathbb{B}(\theta^*,R) B ( θ ∗ , R ) and satisfies
∣ θ ( t ) − θ ∗ ∣ 2 ≤ ( 1 − 1 2 κ 0 ) t ∣ θ ( 0 ) − θ ∗ ∣ 2 + K a s , t ≥ 0. \lvert\theta^{(t)}-\theta^*\rvert_2\le\left(1-\frac{1}{2\kappa_0}\right)^t\lvert\theta^{(0)}-\theta^*\rvert_2+K\sqrt{a_s},\qquad t\ge0. ∣ θ ( t ) − θ ∗ ∣ 2 ≤ ( 1 − 2 κ 0 1 ) t ∣ θ ( 0 ) − θ ∗ ∣ 2 + K a s , t ≥ 0.
In particular, ∥ θ ( t ) − θ ∗ ∥ 2 ≤ 2 K a s \lVert\theta^{(t)}-\theta^*\rVert_2 \le2K\sqrt{a_s} ∥ θ ( t ) − θ ∗ ∥ 2 ≤ 2 K a s for every t ≥ ⌈ c κ 0 log n ⌉ t \ge\lceil c\kappa_0\log n\rceil t ≥ ⌈ c κ 0 log n ⌉ .
Proof. Work on the increment and pointwise score event in Theorem 6. Its probability is at least 1 − C p − γ 1-Cp^{-\gamma} 1 − C p − γ . Write g = ∂ L g=\partial\mathcal{L} g = ∂ L , G = ∇ L G=\nabla\mathcal{L} G = ∇ L , and Γ = ∥ g ( θ ∗ ) ∥ ( 2 m + s ) \Gamma=\lVert g(\theta^*)\rVert_{(2m+s)} Γ = ∥ g ( θ ∗ ) ∥ ( 2 m + s ) . Stationarity gives G ( θ ∗ ) = 0 G(\theta^*)=0 G ( θ ∗ ) = 0 . The pointwise maximum-norm score bound therefore yields Γ ≤ C g a s \Gamma\le C_g\sqrt{a_s} Γ ≤ C g a s . If an index set has more than 2 m 2m 2 m coordinates but at most 2 m + s ≤ 3 m 2m+s\le3m 2 m + s ≤ 3 m , split it into two sets of size at most 2 m 2m 2 m . The restricted increment bound then gives, for every m m m -sparse θ ∈ Θ \theta\in\Theta θ ∈ Θ and every such index set I I I ,
∥ { g ( θ ) − g ( θ ∗ ) − G ( θ ) + G ( θ ∗ ) } I ∥ 2 ≤ C { D a s + a s } , D = ∥ θ − θ ∗ ∥ 2 . \left\lVert\{g(\theta)-g(\theta^*)-G(\theta)+G(\theta^*)\}_I\right\rVert_2 \le C\{\sqrt{D a_s}+a_s\},\qquad D=\lVert\theta-\theta^*\rVert_2. ∥ { g ( θ ) − g ( θ ∗ ) − G ( θ ) + G ( θ ∗ ) } I ∥ 2 ≤ C { D a s + a s } , D = ∥ θ − θ ∗ ∥ 2 .
The endpoint pair has joint support of size at most m + s ≤ 2 m m+s\le2m m + s ≤ 2 m , as required by that theorem. Both bounds hold simultaneously for the decisions and index sets selected by the algorithm.
We use the projection mechanism of Jain et al. [26 ] and give the needed distance estimate. For an arbitrary vector z z z , let J J J be a retained set of its m m m largest coordinates in magnitude, let y = z J y=z_J y = z J extended by zero, and put A = S ∖ J A=S\setminus J A = S ∖ J , B = J ∖ S B=J\setminus S B = J ∖ S , and r = s / ( m − s ) r=\sqrt{s/(m-s)} r = s / ( m − s ) . The ordering of the retained coordinates gives ∥ z A ∥ 2 ≤ r ∥ z B ∥ 2 \lVert z_A\rVert_2\le r\lVert z_B\rVert_2 ∥ z A ∥ 2 ≤ r ∥ z B ∥ 2 . With e = z − θ ∗ e=z-\theta^* e = z − θ ∗ on J ∪ S J\cup S J ∪ S and E = ∥ e J ∪ S ∥ 2 E=\lVert e_{J\cup S}\rVert_2 E = ∥ e J ∪ S ∥ 2 , we have e B = z B e_B=z_B e B = z B and
∥ y − θ ∗ ∥ 2 2 = ∥ e J ∥ 2 2 + ∥ e A − z A ∥ 2 2 ≤ E 2 + r 2 ∥ e B ∥ 2 2 + 2 r ∥ e A ∥ 2 ∥ e B ∥ 2 ≤ ( 1 + r + r 2 ) E 2 ≤ ( 1 + r ) 2 E 2 . \begin{aligned}
\lVert y-\theta^*\rVert_2^2=\lVert e_J\rVert_2^2+\lVert e_A-z_A\rVert_2^2 \\
&\le E^2+r^2\lVert e_B\rVert_2^2+2r\lVert e_A\rVert_2\lVert e_B\rVert_2\le(1+r+r^2)E^2\le(1+r)^2E^2.
\end{aligned} ∥ y − θ ∗ ∥ 2 2 = ∥ e J ∥ 2 2 + ∥ e A − z A ∥ 2 2 ≤ E 2 + r 2 ∥ e B ∥ 2 2 + 2 r ∥ e A ∥ 2 ∥ e B ∥ 2 ≤ ( 1 + r + r 2 ) E 2 ≤ ( 1 + r ) 2 E 2 .
Suppose that the current iterate θ \theta θ has distance D ≤ R D\le R D ≤ R , set z = θ − η g ( θ ) z=\theta-\eta g(\theta) z = θ − η g ( θ ) , and let J J J be its next retained set. Set I = supp ( θ ) ∪ J ∪ S I=\operatorname{supp}(\theta)\cup J\cup S I = supp ( θ ) ∪ J ∪ S . Then ∣ I ∣ ≤ 2 m + s \lvert I\rvert\le2m+s ∣ I ∣ ≤ 2 m + s . The segment from θ ∗ \theta^* θ ∗ to θ \theta θ stays in the assumed coordinate ball. Integration of the population Hessian along this segment gives
∥ { θ − θ ∗ − η [ G ( θ ) − G ( θ ∗ ) ] } I ∥ 2 ≤ ( 1 − 1 κ 0 ) D . \left\lVert\{\theta-\theta^*-\eta[G(\theta)-G(\theta^*)]\}_I\right\rVert_2\le\left(1-\frac{1}{\kappa_0}\right)D. ∥ { θ − θ ∗ − η [ G ( θ ) − G ( θ ∗ )] } I ∥ 2 ≤ ( 1 − κ 0 1 ) D .
Combining this inequality with the increment bound and the projection estimate yields
D + ≤ ( 1 + r ) [ ( 1 − 1 κ 0 ) D + η { Γ + C D a s + C a s } ] . D^+\le(1+r)\left[\left(1-\frac{1}{\kappa_0}\right)D+\eta\{\Gamma+C\sqrt{D a_s}+C a_s\}\right]. D + ≤ ( 1 + r ) [ ( 1 − κ 0 1 ) D + η { Γ + C D a s + C a s } ] .
Young’s inequality gives C D a s ≤ μ 0 D / 8 + 2 C 2 a s / μ 0 C\sqrt{D a_s}\le\mu_0D/8+2C^2a_s/\mu_0 C D a s ≤ μ 0 D /8 + 2 C 2 a s / μ 0 . Put C 1 = C + 2 C 2 / μ 0 C_1=C+2C^2/\mu_0 C 1 = C + 2 C 2 / μ 0 . Since r ≤ 1 / ( 8 κ 0 ) r\le1/(8\kappa_0) r ≤ 1/ ( 8 κ 0 ) ,
( 1 + r ) ( 1 − 1 κ 0 ) ≤ 1 − 7 4 κ 0 ≤ 3 4 κ 0 ≤ Q : = 1 − 1 2 κ 0 . (1+r)\left(1-\frac{1}{\kappa_0}\right)\le1-\frac{7}{4\kappa_0}\le\frac{3}{4\kappa_0}\le Q:=1-\frac{1}{2\kappa_0}. ( 1 + r ) ( 1 − κ 0 1 ) ≤ 1 − 4 κ 0 7 ≤ 4 κ 0 3 ≤ Q := 1 − 2 κ 0 1 .
Consequently
D + ≤ Q D + η ( 1 + r ) ( Γ + C 1 a s ) , η ( 1 + r ) ( Γ + C 1 a s ) 1 − Q ≤ 4 μ 0 ( Γ + C 1 a s ) = : R n . D^+\le QD+\eta(1+r)(\Gamma+C_1a_s),\qquad\frac{\eta(1+r)(\Gamma+C_1a_s)}{1-Q}\le\frac{4}{\mu_0}(\Gamma+C_1a_s)=:R_n. D + ≤ Q D + η ( 1 + r ) ( Γ + C 1 a s ) , 1 − Q η ( 1 + r ) ( Γ + C 1 a s ) ≤ μ 0 4 ( Γ + C 1 a s ) =: R n .
Choose K = 4 ( C g + C 1 ) / μ 0 K=4(C_g+C_1)/\mu_0 K = 4 ( C g + C 1 ) / μ 0 . For a s ≤ 1 a_s\le1 a s ≤ 1 , R n ≤ K a s R_n\le K\sqrt{a_s} R n ≤ K a s , which is at most R / 2 R/2 R /2 for sufficiently large n n n . For D ≤ R D\le R D ≤ R , the one-step bound is at most Q R + ( 1 − Q ) R / 2 < R QR+(1-Q)R/2<R QR + ( 1 − Q ) R /2 < R . Starting with D 0 ≤ R / 2 D_0\le R/2 D 0 ≤ R /2 , induction proves containment and
D t ≤ Q t D 0 + ( 1 − Q t ) R n . D_t\le Q^tD_0+(1-Q^t)R_n. D t ≤ Q t D 0 + ( 1 − Q t ) R n .
Since Q t ≤ exp { − t / ( 2 κ 0 ) } Q^t\le\exp\{-t/(2\kappa_0)\} Q t ≤ exp { − t / ( 2 κ 0 )} and a s ≥ n − 1 a_s\ge n^{-1} a s ≥ n − 1 eventually, increasing c c c gives the stated entry count. A bounded conditional density derivative gives Lipschitz bounds on the population Hessian on these coordinate subspaces by the proof of Corollary 6. Projections with up to 2 m + s 2m+s 2 m + s coordinates satisfy the required moment bounds by splitting them into two projections with at most 2 m 2m 2 m coordinates and applying the triangle inequality for the sub-Gaussian norm. Under this density derivative condition, the spectral bounds m H I ⪯ ∇ 2 L ( θ ∗ ) ⪯ M H I m_H I\preceq\nabla^2\mathcal{L}(\theta^*)\preceq M_H I m H I ⪯ ∇ 2 L ( θ ∗ ) ⪯ M H I imply the displayed local Hessian bounds for sufficiently small R R R , with μ 0 = m H / 2 \mu_0=m_H/2 μ 0 = m H /2 and L 0 = M H + m H / 2 L_0=M_H+m_H/2 L 0 = M H + m H /2 .
Stopping rules and sampling distribution
For a proximal or IHT update from θ \theta θ to θ + \theta^{+} θ + with step size η > 0 \eta> 0 η > 0 , write k = ∣ θ − θ + ∣ 2 k = \lvert\theta- \theta^{+} \rvert_{2} k = ∣ θ − θ + ∣ 2 and G = k / η + L max { k , ϵ 1 } G = k/\eta+ L\max\{k,\epsilon_{1}\} G = k / η + L max { k , ϵ 1 } .
Proposition 11. For parts (i) and (ii), suppose L \mathcal{L} L is convex and satisfies Conditions 1-R and 2-R.
(i) Let θ + \theta^{+} θ + solve (18) with a proper convex penalty r r r , and let θ ^ ∈ arg min θ ∈ Θ F ( θ ) \hat{\theta} \in\operatorname*{arg\,min}_{\theta\in\Theta} F(\theta) θ ^ ∈ arg min θ ∈ Θ F ( θ ) . Suppose θ ∈ Θ r e s \theta\in\Theta_{\mathrm{res}} θ ∈ Θ res and θ + , θ ^ ∈ Θ s c ∩ Θ r e s \theta^{+}, \hat{\theta} \in\Theta_{\mathrm{sc}} \cap\Theta_{\mathrm{res}} θ + , θ ^ ∈ Θ sc ∩ Θ res . Then
∣ θ + − θ ^ ∣ 2 ≤ max { 4 G / μ , ϵ 2 } , \lvert\theta^{+} - \hat{\theta} \rvert_{2} \le\max\{4G/\mu,\epsilon_{2}\}, ∣ θ + − θ ^ ∣ 2 ≤ max { 4 G / μ , ϵ 2 } , 0 ≤ F ( θ + ) − F ( θ ^ ) ≤ max { G 2 / μ , G ϵ 2 } . 0 \le F(\theta^{+}) - F(\hat{\theta}) \le\max\{G^{2}/\mu,G\epsilon_{2}\}. 0 ≤ F ( θ + ) − F ( θ ^ ) ≤ max { G 2 / μ , G ϵ 2 } . (ii) Let θ + \theta^{+} θ + be an IHT update from θ \theta θ . Suppose θ ∈ Θ r e s \theta\in\Theta_{\mathrm{res}} θ ∈ Θ res , θ + , θ ∗ ∈ Θ s c ∩ Θ r e s \theta^{+},\theta^{*} \in\Theta_{\mathrm{sc}} \cap\Theta_{\mathrm{res}} θ + , θ ∗ ∈ Θ sc ∩ Θ res , and S ⊆ supp ( θ + ) S \subseteq\operatorname{supp}(\theta^{+}) S ⊆ supp ( θ + ) . Then
∣ θ + − θ ∗ ∣ 2 ≤ max { 2 ( G + s ~ b n ) / μ , ϵ 2 } , \lvert\theta^{+} - \theta^{*} \rvert_{2} \le\max\{2(G+\sqrt{\tilde{s}}b_{n})/\mu,\epsilon_{2}\}, ∣ θ + − θ ∗ ∣ 2 ≤ max { 2 ( G + s ~ b n ) / μ , ϵ 2 } , L ( θ + ) − L ( θ ∗ ) ≤ max { G 2 / μ , G ϵ 2 } . \mathcal{L}(\theta^{+}) - \mathcal{L}(\theta^{*}) \le\max\{G^{2}/\mu,G\epsilon_{2}\}. L ( θ + ) − L ( θ ∗ ) ≤ max { G 2 / μ , G ϵ 2 } . (iii) Suppose Condition 3-R holds and θ ∈ Θ r e s \theta\in\Theta_{\mathrm{res}} θ ∈ Θ res satisfies ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 \lvert\theta-\theta^{*}\rvert_{2} \le\epsilon_{3} ∣ θ − θ ∗ ∣ 2 ≤ ϵ 3 . For d > 0 d > 0 d > 0 , either of the following conditions gives ∣ θ − θ ∗ ∣ ∞ ≤ d \lvert\theta-\theta^{*}\rvert_{\infty} \le d ∣ θ − θ ∗ ∣ ∞ ≤ d ,
∣ θ + − θ ∣ ∞ η + b n + r n + λ ≤ d ∥ H − 1 ∥ ∞ for ℓ 1 -penalized PGD , \frac{\lvert\theta^{+}-\theta\rvert_{\infty}}{\eta}+b_{n}+r_{n}+\lambda\le\frac{d}{\lVert H^{-1}\rVert_{\infty}}
\qquad\text{for $\ell_{1}$-penalized PGD}, η ∣ θ + − θ ∣ ∞ + b n + r n + λ ≤ ∥ H − 1 ∥ ∞ d for ℓ 1 -penalized PGD , ∣ { ∂ L ( θ ) } J ∣ ∞ + b n + r n ≤ d ∥ H J × J − 1 ∥ ∞ if supp ( θ ) ⊆ J ⊇ S . \lvert\{\partial\mathcal{L}(\theta)\}_{J}\rvert_{\infty}+b_{n}+r_{n} \le\frac{d}{\lVert H^{-1}_{J\times J}\rVert_{\infty}}
\qquad\text{if $\operatorname{supp}(\theta) \subseteq J \supseteq S$.} ∣{ ∂ L ( θ ) } J ∣ ∞ + b n + r n ≤ ∥ H J × J − 1 ∥ ∞ d if supp ( θ ) ⊆ J ⊇ S . In the second condition, assume ∣ J ∣ ≤ s ~ \lvert J\rvert\le\tilde{s} ∣ J ∣ ≤ s ~ .
Proof. For part (i), proximal optimality gives ξ + ∈ ∂ r ( θ + ) \xi^{+} \in\partial r(\theta^{+}) ξ + ∈ ∂ r ( θ + ) with ∂ L ( θ ) + λ ξ + = ( θ − θ + ) / η \partial\mathcal{L}(\theta)+\lambda\xi^{+}=(\theta-\theta^{+})/\eta ∂ L ( θ ) + λ ξ + = ( θ − θ + ) / η . Thus v = ∂ L ( θ + ) + λ ξ + ∈ ∂ F ( θ + ) v=\partial\mathcal{L}(\theta^{+})+\lambda\xi^{+}\in\partial F(\theta^{+}) v = ∂ L ( θ + ) + λ ξ + ∈ ∂ F ( θ + ) , and Condition 1-R gives ∣ v ∣ ( 2 s ~ ) ≤ k / η + L max { k , ϵ 1 } = G \lvert v\rvert_{(2\tilde{s})}\le k/\eta+L\max\{k,\epsilon_{1}\}=G ∣ v ∣ ( 2 s ~ ) ≤ k / η + L max { k , ϵ 1 } = G . Write δ = θ + − θ ^ \delta=\theta^{+}-\hat{\theta} δ = θ + − θ ^ and D = ∣ δ ∣ 2 D=\lvert\delta\rvert_{2} D = ∣ δ ∣ 2 . Since ∣ δ ∣ 0 ≤ 2 s ~ \lvert\delta\rvert_{0}\le2\tilde{s} ∣ δ ∣ 0 ≤ 2 s ~ , convexity and optimality of θ ^ \hat{\theta} θ ^ give 0 ≤ F ( θ + ) − F ( θ ^ ) ≤ ⟨ v , δ ⟩ ≤ G D 0\le F(\theta^{+})-F(\hat{\theta})\le\langle v,\delta\rangle\le GD 0 ≤ F ( θ + ) − F ( θ ^ ) ≤ ⟨ v , δ ⟩ ≤ G D . If D ≥ ϵ 2 D\ge\epsilon_{2} D ≥ ϵ 2 , Condition 2-R for ( θ ^ , θ + ) (\hat{\theta},\theta^{+}) ( θ ^ , θ + ) and convexity of r r r give
0 ≤ F ( θ + ) − F ( θ ^ ) ≤ G D − μ 4 D 2 ≤ G 2 μ , 0 \le F(\theta^{+})-F(\hat{\theta}) \le GD-\frac{\mu}{4}D^{2}\le\frac{G^{2}}{\mu}, 0 ≤ F ( θ + ) − F ( θ ^ ) ≤ G D − 4 μ D 2 ≤ μ G 2 ,
which also implies D ≤ 4 G / μ D\le4G/\mu D ≤ 4 G / μ . If D < ϵ 2 D<\epsilon_{2} D < ϵ 2 , the convexity bound gives F ( θ + ) − F ( θ ^ ) ≤ G ϵ 2 F(\theta^{+})-F(\hat{\theta})\le G\epsilon_{2} F ( θ + ) − F ( θ ^ ) ≤ G ϵ 2 , proving part (i).
For part (ii), put J + = supp ( θ + ) J^{+}=\operatorname{supp}(\theta^{+}) J + = supp ( θ + ) , δ = θ + − θ ∗ \delta=\theta^{+}-\theta^{*} δ = θ + − θ ∗ , and D = ∣ δ ∣ 2 D=\lvert\delta\rvert_{2} D = ∣ δ ∣ 2 . The inclusion S ⊆ J + S\subseteq J^{+} S ⊆ J + gives supp ( δ ) ⊆ J + \operatorname{supp}(\delta)\subseteq J^{+} supp ( δ ) ⊆ J + and ∣ J + ∣ ≤ s ~ \lvert J^{+}\rvert\le\tilde{s} ∣ J + ∣ ≤ s ~ . Coordinatewise optimality gives { ∂ L ( θ ) } J + = { θ − θ + } J + / η \{\partial\mathcal{L}(\theta)\}_{J^{+}}=\{\theta-\theta^{+}\}_{J^{+}}/\eta { ∂ L ( θ ) } J + = { θ − θ + } J + / η . Condition 1-R therefore yields ∣ { ∂ L ( θ + ) } J + ∣ 2 ≤ G \lvert\{\partial\mathcal{L}(\theta^{+})\}_{J^{+}}\rvert_{2}\le G ∣{ ∂ L ( θ + ) } J + ∣ 2 ≤ G , while ∣ { ∂ L ( θ ∗ ) } J + ∣ 2 ≤ s ~ b n \lvert\{\partial\mathcal{L}(\theta^{*})\}_{J^{+}}\rvert_{2}\le\sqrt{\tilde{s}}b_{n} ∣{ ∂ L ( θ ∗ ) } J + ∣ 2 ≤ s ~ b n . Convexity gives L ( θ + ) − L ( θ ∗ ) ≤ G D \mathcal{L}(\theta^{+})-\mathcal{L}(\theta^{*})\le GD L ( θ + ) − L ( θ ∗ ) ≤ G D . If D ≥ ϵ 2 D\ge\epsilon_{2} D ≥ ϵ 2 , adding Condition 2-R for both orientations gives
μ 2 D 2 ≤ ⟨ ∂ L ( θ + ) − ∂ L ( θ ∗ ) , δ ⟩ ≤ ( G + s ~ b n ) D . \frac{\mu}{2}D^{2}\le\langle\partial\mathcal{L}(\theta^{+})-\partial\mathcal{L}(\theta^{*}),\delta\rangle\le(G+\sqrt{\tilde{s}}b_{n})D. 2 μ D 2 ≤ ⟨ ∂ L ( θ + ) − ∂ L ( θ ∗ ) , δ ⟩ ≤ ( G + s ~ b n ) D .
Hence D ≤ 2 ( G + s ~ b n ) / μ D\le2(G+\sqrt{\tilde{s}}b_{n})/\mu D ≤ 2 ( G + s ~ b n ) / μ . The same condition for ( θ ∗ , θ + ) (\theta^{*},\theta^{+}) ( θ ∗ , θ + ) gives L ( θ + ) − L ( θ ∗ ) ≤ G D − μ D 2 / 4 ≤ G 2 / μ \mathcal{L}(\theta^{+})-\mathcal{L}(\theta^{*})\le GD-\mu D^{2}/4\le G^{2}/\mu L ( θ + ) − L ( θ ∗ ) ≤ G D − μ D 2 /4 ≤ G 2 / μ . If D < ϵ 2 D<\epsilon_{2} D < ϵ 2 , convexity gives L ( θ + ) − L ( θ ∗ ) ≤ G ϵ 2 \mathcal{L}(\theta^{+})-\mathcal{L}(\theta^{*})\le G\epsilon_{2} L ( θ + ) − L ( θ ∗ ) ≤ G ϵ 2 , proving part (ii).
Adding k k k to the distance bounds in parts (i) and (ii) gives the current-iterate stopping tests in Section 6.4. Choose ϱ > ϵ 2 \varrho>\epsilon_{2} ϱ > ϵ 2 , with ϱ > 4 L ϵ 1 / μ \varrho>4L\epsilon_{1}/\mu ϱ > 4 L ϵ 1 / μ for PGD or ϱ > 2 ( L ϵ 1 + s ~ b n ) / μ \varrho>2(L\epsilon_{1}+\sqrt{\tilde{s}}b_{n})/\mu ϱ > 2 ( L ϵ 1 + s ~ b n ) / μ for IHT, to place the stopping radius above the approximation and score terms at k = 0 k=0 k = 0 .
For part (iii), Condition 3-R gives ∣ H ( θ − θ ∗ ) ∣ ∞ ≤ ∣ ∂ L ( θ ) ∣ ∞ + b n + r n \lvert H(\theta-\theta^*)\rvert_\infty\leq\lvert\partial\mathcal{L}(\theta)\rvert_\infty+b_n+r_n ∣ H ( θ − θ ∗ ) ∣ ∞ ≤ ∣ ∂ L ( θ ) ∣ ∞ + b n + r n . For ℓ 1 \ell_1 ℓ 1 -penalized PGD, proximal optimality gives ( θ − θ + ) / η = ∂ L ( θ ) + λ ξ (\theta-\theta^+)/\eta=\partial\mathcal{L}(\theta)+\lambda\xi ( θ − θ + ) / η = ∂ L ( θ ) + λ ξ , with ξ ∈ ∂ ∣ θ + ∣ 1 \xi\in\partial\lvert\theta^+\rvert_1 ξ ∈ ∂ ∣ θ + ∣ 1 and ∣ ξ ∣ ∞ ≤ 1 \lvert\xi\rvert_\infty\leq1 ∣ ξ ∣ ∞ ≤ 1 . Thus ∣ ∂ L ( θ ) ∣ ∞ ≤ ∣ θ + − θ ∣ ∞ / η + λ \lvert\partial\mathcal{L}(\theta)\rvert_\infty\leq\lvert\theta^+-\theta\rvert_\infty/\eta+\lambda ∣ ∂ L ( θ ) ∣ ∞ ≤ ∣ θ + − θ ∣ ∞ / η + λ , and multiplication by H − 1 H^{-1} H − 1 proves the first criterion. For the second, supp ( θ − θ ∗ ) ⊆ J \operatorname{supp}(\theta-\theta^*)\subseteq J supp ( θ − θ ∗ ) ⊆ J gives ∣ H J × J ( θ − θ ∗ ) J ∣ ∞ ≤ ∣ { ∂ L ( θ ) } J ∣ ∞ + b n + r n \lvert H_{J\times J}(\theta-\theta^*)_J\rvert_\infty\leq\lvert\{\partial\mathcal{L}(\theta)\}_J\rvert_\infty+b_n+r_n ∣ H J × J ( θ − θ ∗ ) J ∣ ∞ ≤ ∣{ ∂ L ( θ ) } J ∣ ∞ + b n + r n . Multiplication by H J × J − 1 H_{J\times J}^{-1} H J × J − 1 proves the second criterion.
Proof of Corollary 7. Restrict the residual objective to the deterministic coordinate set J J J . The conditional density of Y Y Y given Z J Z_J Z J is ρ J ( y ∣ Z J ) = E { ρ ( y ∣ Z ) ∣ Z J } \rho_J(y\mid Z_J)=\mathbb{E}\{\rho(y\mid Z)\mid Z_J\} ρ J ( y ∣ Z J ) = E { ρ ( y ∣ Z ) ∣ Z J } and has the same bound. Every unit projection of Z J Z_J Z J has bounded sub-Gaussian norm, and the restricted region is convex and bounded. Theorem 3 therefore applies with dimension ∣ J ∣ \lvert J\rvert ∣ J ∣ and scale a J = ∣ J ∣ log n / n ≤ a s a_J=\lvert J\rvert\log n/n\leq a_s a J = ∣ J ∣ log n / n ≤ a s . Its uniform increment event has probability at least 1 − C n − γ ∣ J ∣ 1-Cn^{-\gamma\lvert J\rvert} 1 − C n − γ ∣ J ∣ . Intersect it with the localization event and, for IHT, with { J ^ = J } \{\widehat{J}=J\} { J = J } . Their intersection has probability tending to one. Put e t = { θ ( t ) − θ ∗ } J e_t=\{\theta^{(t)}-\theta^*\}_J e t = { θ ( t ) − θ ∗ } J and g J = { ∂ L ( θ ∗ ) } J g_J=\{\partial\mathcal{L}(\theta^*)\}_J g J = { ∂ L ( θ ∗ ) } J . Independence and stationarity give
E ∣ g J ∣ 2 2 = 1 n E ∣ ψ i , J ∣ 2 2 ≤ C ∣ J ∣ / n . \mathbb{E}\lvert g_J\rvert_2^2=\frac{1}{n}\mathbb{E}\lvert\psi_{i,J}\rvert_2^2\leq C\lvert J\rvert/n. E ∣ g J ∣ 2 2 = n 1 E ∣ ψ i , J ∣ 2 2 ≤ C ∣ J ∣ / n .
The PGD iterates have support S S S from time 1. Each IHT update retaining J J J copies the unthresholded update on J J J . Thus, for 1 ≤ t < T 1\leq t<T 1 ≤ t < T ,
e t + 1 = ( I − η H J × J ) e t − η g J − η u t , ∣ u t ∣ 2 ≤ C { ∣ e t ∣ 2 a J + a J + ∣ e t ∣ 2 2 } . e_{t+1}=(I-\eta H_{J\times J})e_t-\eta g_J-\eta u_t,\qquad\lvert u_t\rvert_2\leq C\{\sqrt{\lvert e_t\rvert_2a_J}+a_J+\lvert e_t\rvert_2^2\}. e t + 1 = ( I − η H J × J ) e t − η g J − η u t , ∣ u t ∣ 2 ≤ C { ∣ e t ∣ 2 a J + a J + ∣ e t ∣ 2 2 } .
Under Theorem 7(iii) or Proposition 12(i), the same recursion coordinatewise gives ∣ e t + 1 ∣ ∞ ≤ q ∣ e t ∣ ∞ + η ( b n + r n ) ≤ d \lvert e_{t+1}\rvert_\infty\leq q\lvert e_t\rvert_\infty+\eta(b_n+r_n)\leq d ∣ e t + 1 ∣ ∞ ≤ q ∣ e t ∣ ∞ + η ( b n + r n ) ≤ d , proving containment by induction. Theorem 8(iv) gives Euclidean containment for the other IHT case. Hence ∣ e t ∣ 2 ≤ C 3 a s = o ( 1 ) \lvert e_t\rvert_2\leq C_3\sqrt{a_s}=o(1) ∣ e t ∣ 2 ≤ C 3 a s = o ( 1 ) throughout the window. Let q J = ∥ I − η H J × J ∥ q_J=\lVert I-\eta H_{J\times J}\rVert q J = ∥ I − η H J × J ∥ and α J = 1 − q J \alpha_J=1-q_J α J = 1 − q J . The bound q ≤ q 0 < 1 q\leq q_0<1 q ≤ q 0 < 1 gives q J ≤ q 0 q_J\leq q_0 q J ≤ q 0 and η m H ≤ 1 + q 0 \eta m_H\leq1+q_0 η m H ≤ 1 + q 0 , while η ≤ M H − 1 \eta\leq M_H^{-1} η ≤ M H − 1 gives α J ≥ η m H \alpha_J\geq\eta m_H α J ≥ η m H . In both cases η / α J ≤ C \eta/\alpha_J\leq C η / α J ≤ C . Young’s inequality and the vanishing containment radius yield
∣ e t + 1 ∣ 2 ≤ ( 1 − α J / 2 ) ∣ e t ∣ 2 + η { ∣ g J ∣ 2 + C a J } . \lvert e_{t+1}\rvert_2\leq(1-\alpha_J/2)\lvert e_t\rvert_2+\eta\{\lvert g_J\rvert_2+Ca_J\}. ∣ e t + 1 ∣ 2 ≤ ( 1 − α J /2 ) ∣ e t ∣ 2 + η {∣ g J ∣ 2 + C a J } .
The geometric kernel has mass at most 2 η / α J ≤ C 2\eta/\alpha_J\leq C 2 η / α J ≤ C . For t 0 = ⌊ T / 2 ⌋ t_0=\lfloor T/2\rfloor t 0 = ⌊ T /2 ⌋ , the stated iteration counts therefore give
sup t 0 ≤ t ≤ T ∣ e t ∣ 2 = O P ( ∣ J ∣ / n + a J ) , sup t 0 ≤ t < T ∣ u t ∣ 2 = O P { ( ∣ J ∣ / n ) 3 / 4 log n + a J } . \sup_{t_0\leq t\leq T}\lvert e_t\rvert_2=O_P\left(\sqrt{\lvert J\rvert/n}+a_J\right),\qquad
\sup_{t_0\leq t<T}\lvert u_t\rvert_2=O_P\left\{(\lvert J\rvert/n)^{3/4}\sqrt{\log n}+a_J\right\}. t 0 ≤ t ≤ T sup ∣ e t ∣ 2 = O P ( ∣ J ∣ / n + a J ) , t 0 ≤ t < T sup ∣ u t ∣ 2 = O P { (∣ J ∣ / n ) 3/4 log n + a J } .
Apply the linear recursion from t 0 t_0 t 0 to T T T to e t + H J × J − 1 g J e_t+H_{J\times J}^{-1}g_J e t + H J × J − 1 g J . Its kernel has mass at most η / α J ≤ C \eta/\alpha_J\leq C η / α J ≤ C , and its transient is O P ( n − 1 ) O_P(n^{-1}) O P ( n − 1 ) after increasing c c c . This proves the expansion. Each deterministic coordinate of H J × J − 1 ψ i , J H_{J\times J}^{-1}\psi_{i,J} H J × J − 1 ψ i , J has bounded third absolute moment. The variance lower bound gives Lyapunov’s condition, and the dimension condition makes both remainder terms o ( n − 1 / 2 ) o(n^{-1/2}) o ( n − 1/2 ) . Slutsky’s theorem proves the normal limit. The complement of the joint event has vanishing probability and does not affect either conclusion.
Proposition 12. (i) Suppose Condition 3-R holds, s ~ ≥ s \widetilde{s}\geq s s ≥ s , B d ⊆ Θ \mathcal{B}_d\subseteq\Theta B d ⊆ Θ , s ~ d ≤ ϵ 3 \sqrt{\widetilde{s}}d\leq\epsilon_3 s d ≤ ϵ 3 , q < 1 q<1 q < 1 , θ ( 0 ) ∈ B d \theta^{(0)}\in\mathcal{B}_d θ ( 0 ) ∈ B d , and η ( b n + r n ) ≤ ( 1 − q ) d \eta(b_n+r_n)\leq(1-q)d η ( b n + r n ) ≤ ( 1 − q ) d . For every J ⊇ S J\supseteq S J ⊇ S with ∣ J ∣ = s ~ \lvert J\rvert=\widetilde{s} ∣ J ∣ = s , if supp ( θ ( 0 ) ) ⊆ J \operatorname{supp}(\theta^{(0)})\subseteq J supp ( θ ( 0 ) ) ⊆ J and the next T T T IHT updates retain J J J , then every iterate belongs to B d \mathcal{B}_d B d and
{ θ ( T ) − θ ∗ } J = − H J × J − 1 { ∂ L ( θ ∗ ) } J \{\theta^{(T)}-\theta^*\}_J=-H_{J\times J}^{-1}\{\partial\mathcal{L}(\theta^*)\}_J { θ ( T ) − θ ∗ } J = − H J × J − 1 { ∂ L ( θ ∗ ) } J + ( I − η H J × J ) T [ { θ ( 0 ) − θ ∗ } J + H J × J − 1 { ∂ L ( θ ∗ ) } J ] + R J , T , \qquad+ (I-\eta H_{J\times J})^T\left[\{\theta^{(0)}-\theta^*\}_J+H_{J\times J}^{-1}\{\partial\mathcal{L}(\theta^*)\}_J\right]+R_{J,T}, + ( I − η H J × J ) T [ { θ ( 0 ) − θ ∗ } J + H J × J − 1 { ∂ L ( θ ∗ ) } J ] + R J , T , where ∣ R J , T ∣ ∞ ≤ η ( 1 − q T ) r n / ( 1 − q ) \lvert R_{J,T}\rvert_\infty\leq\eta(1-q^T)r_n/(1-q) ∣ R J , T ∣ ∞ ≤ η ( 1 − q T ) r n / ( 1 − q ) .
(ii) Consider the residual objective under either set of assumptions in Theorem 6, with ∇ L ( θ ∗ ) = 0 \nabla\mathcal{L}(\theta^*) = \mathbf{0} ∇ L ( θ ∗ ) = 0 . Let J ^ ⊇ S \widehat{J} \supseteq S J ⊇ S be a random retained set of size s ~ \widetilde{s} s , and suppose the conditions of part (i) hold on an event E n \mathcal{E}_n E n of probability at least 1 − δ n 1-\delta_n 1 − δ n , where δ n → 0 \delta_n \to0 δ n → 0 . For every 0 < α < 1 0 < \alpha< 1 0 < α < 1 , let B n , α = C log ( 2 p / α ) / n B_{n,\alpha}=C\sqrt{\log(2p/\alpha)/n} B n , α = C log ( 2 p / α ) / n . Then
Pr { ∣ θ j ( T ) − θ j ∗ ∣ ≤ q T d + η ( 1 − q T ) 1 − q ( B n , α + r n ) for every j ∈ J ^ } ≥ 1 − α − δ n . \Pr\left\{\left|\theta_j^{(T)}-\theta_j^*\right|\leq q^T d+\frac{\eta(1-q^T)}{1-q}(B_{n,\alpha}+r_n)\ \text{for every }j\in\widehat{J}\right\}\geq1-\alpha-\delta_n. Pr { θ j ( T ) − θ j ∗ ≤ q T d + 1 − q η ( 1 − q T ) ( B n , α + r n ) for every j ∈ J } ≥ 1 − α − δ n .
If q ≤ q 0 < 1 q\leq q_0<1 q ≤ q 0 < 1 , r n = C 4 a s 3 / 4 r_n=C_4a_s^{3/4} r n = C 4 a s 3/4 , and T ≥ ⌈ c log n ⌉ T\geq\lceil c\log n\rceil T ≥ ⌈ c log n ⌉ for sufficiently large c c c , the radius is O { log p / n + a s 3 / 4 } O\{\sqrt{\log p/n}+a_s^{3/4}\} O { log p / n + a s 3/4 } and tends to zero throughout as a s → 0 a_s\to0 a s → 0 .
(iii) Let J = J n ⊇ S J=J_n\supseteq S J = J n ⊇ S be deterministic. For IHT, suppose the conditions of part (i) hold with probability tending to one for a random retained set J ^ \widehat{J} J , and Pr ( J ^ = J ) → 1 \Pr(\widehat{J}=J)\to1 Pr ( J = J ) → 1 . For PGD, take J = S J=S J = S and suppose the conditions of Theorem 7(iii) hold with probability tending to one. Assume H H H is deterministic, q ≤ q 0 < 1 q\leq q_0<1 q ≤ q 0 < 1 , and ∂ L ( θ ∗ ) = n − 1 ∑ i ψ i \partial\mathcal{L}(\theta^*)=n^{-1}\sum_i\psi_i ∂ L ( θ ∗ ) = n − 1 ∑ i ψ i with i.i.d. mean-zero scores. For a deterministic j ∈ J j\in J j ∈ J , write
ξ i , J , j = e j , J T H J × J − 1 ψ i , J , σ J , j 2 = Var ( ξ i , J , j ) . \xi_{i,J,j}=e_{j,J}^{\mathsf{T}}H_{J\times J}^{-1}\psi_{i,J},\qquad\sigma_{J,j}^{2}=\operatorname{Var}(\xi_{i,J,j}). ξ i , J , j = e j , J T H J × J − 1 ψ i , J , σ J , j 2 = Var ( ξ i , J , j ) .
Suppose σ J , j ≥ c > 0 \sigma_{J,j}\geq c>0 σ J , j ≥ c > 0 , E ∣ ξ i , J , j ∣ 2 + ν ≤ C σ J , j 2 + ν \mathbb{E}|\xi_{i,J,j}|^{2+\nu}\leq C\sigma_{J,j}^{2+\nu} E ∣ ξ i , J , j ∣ 2 + ν ≤ C σ J , j 2 + ν for some ν > 0 \nu>0 ν > 0 , and n r n = o P ( 1 ) \sqrt{n}r_n=o_P(1) n r n = o P ( 1 ) . If
n q T ( d + η b n 1 − q ) = o P ( 1 ) for IHT , n q T − 1 ( d + η b n 1 − q ) = o P ( 1 ) for PGD , \sqrt{n}q^T\left(d+\frac{\eta b_n}{1-q}\right)=o_P(1)\quad\text{for IHT},\qquad
\sqrt{n}q^{T-1}\left(d+\frac{\eta b_n}{1-q}\right)=o_P(1)\quad\text{for PGD}, n q T ( d + 1 − q η b n ) = o P ( 1 ) for IHT , n q T − 1 ( d + 1 − q η b n ) = o P ( 1 ) for PGD ,
then
n ( θ j ( T ) − θ j ∗ ) σ J , j → d N ( 0 , 1 ) . \frac{\sqrt{n}(\theta_j^{(T)}-\theta_j^*)}{\sigma_{J,j}}\xrightarrow{d}\mathcal{N}(0,1). σ J , j n ( θ j ( T ) − θ j ∗ ) d N ( 0 , 1 ) .
For the residual objective, Corollary 7 gives the sharp expansion and normal limit.
(iv) Consider the residual objective under the assumptions of Theorem 6(2), with ∇ L ( θ ∗ ) = 0 \nabla\mathcal{L}(\theta^*)=0 ∇ L ( θ ∗ ) = 0 and ψ i = b − Z i f + ′ ( Y i − Z i T θ ∗ ) \psi_i=\mathbf{b}-Z_i f_+'(Y_i-Z_i^{\mathsf{T}}\theta^*) ψ i = b − Z i f + ′ ( Y i − Z i T θ ∗ ) . Let J ^ ⊇ S \widehat{J}\supseteq S J ⊇ S be a random retained set of size s ~ \widetilde{s} s . Suppose the conditions of part (i) hold on an event E n \mathcal{E}_n E n of probability at least 1 − δ n 1-\delta_n 1 − δ n , where δ n → 0 \delta_n\to0 δ n → 0 , and assume H H H is deterministic and q ≤ q 0 < 1 q\leq q_0<1 q ≤ q 0 < 1 . Use ξ i , J , j , σ J , j \xi_{i,J,j},\sigma_{J,j} ξ i , J , j , σ J , j from part (iii). Take a nonempty deterministic family J n \mathcal{J}_n J n of nonempty retained sets of size s ~ \widetilde{s} s , with Pr ( J ^ ∈ J n ) → 1 \Pr(\widehat{J}\in\mathcal{J}_n)\to1 Pr ( J ∈ J n ) → 1 . Let I n = { ( J , j ) : J ∈ J n , j ∈ J } \mathcal{I}_n=\{(J,j):J\in\mathcal{J}_n,\ j\in J\} I n = {( J , j ) : J ∈ J n , j ∈ J } , D n = ∣ I n ∣ D_n=|\mathcal{I}_n| D n = ∣ I n ∣ , and ℓ n = log ( 2 n D n ) \ell_n=\log(2nD_n) ℓ n = log ( 2 n D n ) . Assume σ J , j ≥ c > 0 \sigma_{J,j}\geq c>0 σ J , j ≥ c > 0 uniformly over I n \mathcal{I}_n I n and
ℓ n 5 = o ( n ) , n ℓ n [ q T ( d + η b n 1 − q ) + η r n 1 − q ] = o P ( 1 ) . \ell_n^5=o(n),\qquad\sqrt{n\ell_n}\left[q^T\left(d+\frac{\eta b_n}{1-q}\right)+\frac{\eta r_n}{1-q}\right]=o_P(1). ℓ n 5 = o ( n ) , n ℓ n [ q T ( d + 1 − q η b n ) + 1 − q η r n ] = o P ( 1 ) .
Let c n , 1 − α c_{n,1-\alpha} c n , 1 − α be the ( 1 − α ) (1-\alpha) ( 1 − α ) -quantile of max ( J , j ) ∈ I n ∣ G J , j ∣ \max_{(J,j)\in\mathcal{I}_n}|G_{J,j}| max ( J , j ) ∈ I n ∣ G J , j ∣ , where G G G is mean-zero Gaussian with the covariance of ( ξ i , J , j / σ J , j ) ( J , j ) ∈ I n (\xi_{i,J,j}/\sigma_{J,j})_{(J,j)\in\mathcal{I}_n} ( ξ i , J , j / σ J , j ) ( J , j ) ∈ I n . For every 0 < α < 1 0<\alpha<1 0 < α < 1 ,
lim inf n → ∞ Pr { θ j ∗ ∈ [ θ j ( T ) ± σ J ^ , j c n , 1 − α n ] for every j ∈ J ^ } ≥ 1 − α . \liminf_{n\to\infty}\Pr\left\{\theta_j^*\in\left[\theta_j^{(T)}\pm\frac{\sigma_{\widehat{J},j}c_{n,1-\alpha}}{\sqrt{n}}\right]\text{ for every }j\in\widehat{J}\right\}\geq1-\alpha. n → ∞ lim inf Pr { θ j ∗ ∈ [ θ j ( T ) ± n σ J , j c n , 1 − α ] for every j ∈ J } ≥ 1 − α .
The critical value can also be Φ − 1 { 1 − α / ( 2 D n ) } \Phi^{-1}\{1-\alpha/(2D_n)\} Φ − 1 { 1 − α / ( 2 D n )} . For the family of all retained sets of size s ~ \widetilde{s} s , with r n = C 4 a s 3 / 4 r_n=C_4a_s^{3/4} r n = C 4 a s 3/4 and T ≥ ⌈ c log n ⌉ T\geq\lceil c\log n\rceil T ≥ ⌈ c log n ⌉ for sufficiently large c c c , the regime { s ~ log p + log n } 5 = o ( n ) \{\widetilde{s}\log p+\log n\}^5=o(n) { s log p + log n } 5 = o ( n ) suffices for the two displayed rate conditions.
Proof. For part (i), write e t = { θ ( t ) − θ ∗ } J e_t=\{\theta^{(t)}-\theta^*\}_J e t = { θ ( t ) − θ ∗ } J , g J = { ∂ L ( θ ∗ ) } J g_J=\{\partial L(\theta^*)\}_J g J = { ∂ L ( θ ∗ ) } J , and A J = I − η H J × J A_J=I-\eta H_{J\times J} A J = I − η H J × J . The initial error is supported on J J J , and every retained coordinate copies the unthresholded update. Therefore
e t + 1 = A J e t − η g J − η u t , u t = { ∂ L ( θ ( t ) ) − ∂ L ( θ ∗ ) } J − H J × J e t . e_{t+1}=A_Je_t-\eta g_J-\eta u_t,\qquad u_t=\{\partial L(\theta^{(t)})-\partial L(\theta^*)\}_J-H_{J\times J}e_t. e t + 1 = A J e t − η g J − η u t , u t = { ∂ L ( θ ( t ) ) − ∂ L ( θ ∗ ) } J − H J × J e t .
If θ ( t ) ∈ B d \theta^{(t)}\in B_d θ ( t ) ∈ B d , its error has Euclidean norm at most s ~ d ≤ ϵ 3 \sqrt{\tilde{s}}d\le\epsilon_3 s ~ d ≤ ϵ 3 , and Condition 3-R gives ∣ u t ∣ ∞ ≤ r n \lvert u_t\rvert_\infty\le r_n ∣ u t ∣ ∞ ≤ r n . Since ∥ A J ∥ ∞ ≤ q \lVert A_J\rVert_\infty\le q ∥ A J ∥ ∞ ≤ q , the next error has maximum norm at most q d + η ( b n + r n ) ≤ d qd+\eta(b_n+r_n)\le d q d + η ( b n + r n ) ≤ d . Its support is in J J J , and B d ⊆ Θ B_d\subseteq\Theta B d ⊆ Θ gives the induction. Iterating the recursion proves the expansion and the displayed remainder bound. The same Neumann series gives
∥ H J × J − 1 ∥ ∞ ≤ η 1 − q , ∣ e T + H J × J − 1 g J ∣ ∞ ≤ q T ( d + η b n 1 − q ) + η r n 1 − q . \lVert H_{J\times J}^{-1}\rVert_\infty\le\frac{\eta}{1-q},\qquad\lvert e_T+H_{J\times J}^{-1}g_J\rvert_\infty\le q^T\left(d+\frac{\eta b_n}{1-q}\right)+\frac{\eta r_n}{1-q}. ∥ H J × J − 1 ∥ ∞ ≤ 1 − q η , ∣ e T + H J × J − 1 g J ∣ ∞ ≤ q T ( d + 1 − q η b n ) + 1 − q η r n .
For part (ii), each score coordinate has a uniformly bounded sub-Gaussian norm, since its random part is bounded by C f ∣ Z j ∣ C_f\lvert Z_j\rvert C f ∣ Z j ∣ and its affine term subtracts its expectation. A coordinatewise tail bound and a union bound give Pr { b n > B n , α } ≤ α \Pr\{b_n>B_{n,\alpha}\}\le\alpha Pr { b n > B n , α } ≤ α . On E n E_n E n , the recursion in part (i) gives ∣ θ ( T ) − θ ∗ ∣ ∞ ≤ q T d + η ( 1 − q T ) ( b n + r n ) / ( 1 − q ) \lvert\theta^{(T)}-\theta^*\rvert_\infty\le q^Td+\eta(1-q^T)(b_n+r_n)/(1-q) ∣ θ ( T ) − θ ∗ ∣ ∞ ≤ q T d + η ( 1 − q T ) ( b n + r n ) / ( 1 − q ) . Intersecting these events proves the confidence bound. Boundedness of Θ \Theta Θ and B d ⊆ Θ B_d\subseteq\Theta B d ⊆ Θ bound d d d , and a sufficiently large c c c makes the transient negligible, proving the stated radius.
For part (iii), the localization event, intersected with { J ^ = J } \{\widehat{J}=J\} { J = J } for IHT, has probability tending to one. Part (i) gives the IHT expansion, and Theorem 7(iii) gives the PGD expansion. The respective transient conditions and n r n = o P ( 1 ) \sqrt{n}r_n=o_P(1) n r n = o P ( 1 ) imply n ( θ j ( T ) − θ j ∗ ) / σ J , j = − ( n σ J , j ) − 1 ∑ i = 1 n ξ i , J , j + o P ( 1 ) \sqrt{n}(\theta_j^{(T)}-\theta_j^*)/\sigma_{J,j}=-(\sqrt{n}\,\sigma_{J,j})^{-1}\sum_{i=1}^n\xi_{i,J,j}+o_P(1) n ( θ j ( T ) − θ j ∗ ) / σ J , j = − ( n σ J , j ) − 1 ∑ i = 1 n ξ i , J , j + o P ( 1 ) . The moment assumption bounds the Lyapunov ratio by C n − ν / 2 Cn^{-\nu/2} C n − ν /2 , and the Lyapunov central limit theorem and Slutsky’s theorem prove the normal limit.
For part (iv), extend H J × J − 1 e j , J H_{J\times J}^{-1}e_{j,J} H J × J − 1 e j , J by zeros outside J J J . This vector is s ~ \tilde{s} s ~ -sparse with Euclidean norm at most m H − 1 m_H^{-1} m H − 1 . Hence ξ i , J , j / σ J , j \xi_{i,J,j}/\sigma_{J,j} ξ i , J , j / σ J , j has a uniformly bounded sub-Gaussian norm. Its exponential and fourth-moment bounds and its unit variance verify the assumptions of Chernozhukov et al. [10 ] , which gives rectangle Gaussian approximation error C ( ℓ n 5 / n ) 1 / 4 C(\ell_n^5/n)^{1/4} C ( ℓ n 5 / n ) 1/4 for the standardized score array. Nazarov [35 ] ’s inequality, applied to ( G , − G ) (G,-G) ( G , − G ) , bounds the probability that max ( J , j ) ∈ I n ∣ G J , j ∣ \max_{(J,j)\in\mathcal{I}_n}\lvert G_{J,j}\rvert max ( J , j ) ∈ I n ∣ G J , j ∣ lies in any interval of width δ > 0 \delta>0 δ > 0 by C δ log ( 2 D n ) C\delta\sqrt{\log(2D_n)} C δ log ( 2 D n ) . The second rate condition gives a deterministic ϵ n = o ( ℓ n − 1 / 2 ) \epsilon_n=o(\ell_n^{-1/2}) ϵ n = o ( ℓ n − 1/2 ) for which the selected standardized expansion remainder is at most ϵ n \epsilon_n ϵ n with probability tending to one. On this event, bounding every standardized influence coordinate in absolute value by c n , 1 − α − ϵ n c_{n,1-\alpha}-\epsilon_n c n , 1 − α − ϵ n implies simultaneous coverage, and anti-concentration proves the bound. The Gaussian union bound permits the normal critical value. For the full retained-set family, ℓ n ≤ C { s ~ log p + log n } \ell_n\le C\{\tilde{s}\log p+\log n\} ℓ n ≤ C { s ~ log p + log n } , and
n ℓ n a s 3 / 4 ≤ C [ { s ~ log p + log n } 5 n ] 1 / 4 . \sqrt{n}\ell_n a_s^{3/4}\le C\left[\frac{\{\tilde{s}\log p+\log n\}^5}{n}\right]^{1/4}. n ℓ n a s 3/4 ≤ C [ n { s ~ log p + log n } 5 ] 1/4 .
A sufficiently large c c c also makes the transient negligible, proving the sufficient regime.
Index objectives
Corollary 16. *Consider the index objective (63) with i.i.d. observations and p ≥ n ≥ 2 p\ge n\ge2 p ≥ n ≥ 2 . Impose Assumptions 3, 5, and 6(i), and write ν = d f + ′ \nu=df'_+ ν = d f + ′ . *
If ∣ X ∣ ∞ ≤ C X \lvert X\rvert_\infty\le C_X ∣ X ∣ ∞ ≤ C X almost surely and Λ max { E ( X X T ) } = O ( 1 ) \Lambda_{\max}\{\mathbb{E}(XX^\mathsf{T})\}=O(1) Λ m a x { E ( X X T )} = O ( 1 ) , then (16) holds with probability at least 1 − C P n − γ 1-C_Pn^{-\gamma} 1 − C P n − γ .
If sup ∣ v ∣ 2 = 1 , ∣ v ∣ 0 ≤ 2 s ~ ∥ v T X ∥ ψ 2 ≤ C X \sup_{|v|_2=1,\,|v|_0\le2\tilde{s}} \|v^{\mathsf{T}}X\|_{\psi_2}\le C_X sup ∣ v ∣ 2 = 1 , ∣ v ∣ 0 ≤ 2 s ~ ∥ v T X ∥ ψ 2 ≤ C X and either ν \nu ν has compact support or ρ X \rho_X ρ X has a bounded continuous first derivative, then (16) holds with ∣ ⋅ ∣ ∞ |\cdot|_\infty ∣ ⋅ ∣ ∞ replaced by ∣ ⋅ ∣ ( 2 s ~ ) |\cdot|_{(2\tilde{s})} ∣ ⋅ ∣ ( 2 s ~ ) , with the same probability bound.
Both increment bounds hold for some C , γ > 0 C,\gamma>0 C , γ > 0 , uniformly over θ 1 , θ 2 ∈ Θ \theta_1,\theta_2\in\Theta θ 1 , θ 2 ∈ Θ with ∣ supp ( θ 1 ) ∪ supp ( θ 2 ) ∣ ≤ 2 s ~ |\operatorname{supp}(\theta_1)\cup\operatorname{supp}(\theta_2)|\le2\tilde{s} ∣ supp ( θ 1 ) ∪ supp ( θ 2 ) ∣ ≤ 2 s ~ , for every selection of scenario subgradients. Under either set of assumptions, at θ ∗ ∈ Θ \theta^*\in\Theta θ ∗ ∈ Θ , the bound ∣ ∂ L ( θ ∗ ) − ∇ L ( θ ∗ ) ∣ ∞ ≤ C log p / n |\partial L(\theta^*)-\nabla L(\theta^*)|_\infty\le C\sqrt{\log p/n} ∣ ∂ L ( θ ∗ ) − ∇ L ( θ ∗ ) ∣ ∞ ≤ C log p / n holds with probability at least 1 − C p − γ 1-Cp^{-\gamma} 1 − C p − γ . The increment and pointwise bounds therefore hold jointly with probability at least 1 − C p − γ 1-Cp^{-\gamma} 1 − C p − γ . Under the population strong convexity and Taylor assumptions of Corollary 6, its conclusions hold for this objective. Compact support of ν \nu ν and a bounded continuous density derivative give the Taylor bound in ∣ ⋅ ∣ ( 2 s ~ ) |\cdot|_{(2\tilde{s})} ∣ ⋅ ∣ ( 2 s ~ ) . At a stationary population solution, use the score ψ i = b + X i f + ′ ( X i T θ ∗ ) \psi_i=b+X_i f'_+(X_i^{\mathsf{T}}\theta^*) ψ i = b + X i f + ′ ( X i T θ ∗ ) . Under their respective hypotheses, Corollary 7, Proposition 12, Theorem 8(iv), and Corollary 15 hold with this objective and score.
Proof. Write m = s ~ m=\tilde{s} m = s ~ , g = f + ′ g=f'_+ g = f + ′ , ν = d g \nu=dg ν = d g , and a s = m log p / n a_s=m\log p/n a s = m log p / n . Use a θ ( c ) a_\theta(c) a θ ( c ) , x θ ( c ) x_\theta(c) x θ ( c ) , and ρ θ ( c ∣ U ) \rho_\theta(c\mid U) ρ θ ( c ∣ U ) from the proof of Proposition 6, with U = X − 1 U=X_{-1} U = X − 1 . Convexity of Θ \Theta Θ and ∣ θ 1 ∣ ≥ c 0 |\theta_1|\ge c_0 ∣ θ 1 ∣ ≥ c 0 fix the sign of its first coefficient. Every decision support contains coordinate 1 1 1 , and a segment between decisions with joint support of size at most 2 m 2m 2 m stays in that subspace and in Θ \Theta Θ .
For part 1, the conditional distribution of X 1 X_1 X 1 is supported on [ − C X , C X ] [-C_X,C_X] [ − C X , C X ] . Thus ρ θ ( c ∣ U ) > 0 \rho_\theta(c\mid U)>0 ρ θ ( c ∣ U ) > 0 implies ∣ a θ ( c ) ∣ ≤ C X |a_\theta(c)|\le C_X ∣ a θ ( c ) ∣ ≤ C X , and every coordinate of x θ ( c ) x_\theta(c) x θ ( c ) is bounded by C X C_X C X on this event. For every unit vector u u u , the second-moment bound gives, uniformly in c c c ,
E [ ρ θ ( c ∣ U ) ∣ u T x θ ( c ) ∣ ] ≤ C { ∣ u 1 ∣ C X + E ∣ u − 1 T U ∣ } ≤ C . \mathbb{E}[\rho_\theta(c\mid U)|u^{\mathsf{T}}x_\theta(c)|]\le C\{|u_1|C_X+\mathbb{E}|u_{-1}^{\mathsf{T}}U|\}\le C. E [ ρ θ ( c ∣ U ) ∣ u T x θ ( c ) ∣ ] ≤ C { ∣ u 1 ∣ C X + E ∣ u − 1 T U ∣ } ≤ C .
Apply the crossing formula (69) along the segment between θ 0 \theta_0 θ 0 and θ 1 \theta_1 θ 1 . Since ν ( R ) ≤ 2 C f \nu(\mathbb{R})\le2C_f ν ( R ) ≤ 2 C f , the coordinate bound on each section gives, for d = ∣ θ 1 − θ 0 ∣ 2 d=|\theta_1-\theta_0|_2 d = ∣ θ 1 − θ 0 ∣ 2 ,
E ∣ X j { g ( X T θ 1 ) − g ( X T θ 0 ) } ∣ k ≤ d k ! C k , k ≥ 1. \mathbb{E}\left|X_j\{g(X^{\mathsf{T}}\theta_1)-g(X^{\mathsf{T}}\theta_0)\}\right|^k\le d k! C^k,\qquad k\ge1. E X j { g ( X T θ 1 ) − g ( X T θ 0 )} k ≤ d k ! C k , k ≥ 1.
The conditional density bound also gives the weighted slab moments at every level, and integration against ν \nu ν gives the strict-bracket moments. Bernstein’s inequality therefore gives deviation C { d t / n + t / n } C\{\sqrt{dt/n}+t/n\} C { d t / n + t / n } for a fixed decision pair and coordinate. Take p − 12 p^{-12} p − 12 -nets of Θ \Theta Θ on coordinate subspaces of dimension at most 2 m 2m 2 m . Their combined logarithmic cardinality is O ( m log p ) O(m\log p) O ( m log p ) . A decision and its representative have index difference at most w = C X 2 m p − 12 w=C_X\sqrt{2m}p^{-12} w = C X 2 m p − 12 . The strict bracket g ( X T θ ~ + 2 w ) − f − ′ ( X T θ ~ − 2 w ) g(X^{\mathsf{T}}\tilde{\theta}+2w)-f'_-(X^{\mathsf{T}}\tilde{\theta}-2w) g ( X T θ ~ + 2 w ) − f − ′ ( X T θ ~ − 2 w ) has weighted moments bounded by w k ! C k wk!C^k w k ! C k . Applying Bernstein’s inequality over the nets and coordinates bounds interpolation by C ( w + a s ) ≤ C a s C(w+a_s)\le Ca_s C ( w + a s ) ≤ C a s . Applying the pairwise bound with t = A m log p t=Am\log p t = A m log p gives part 1, with failure probability C p − γ m Cp^{-\gamma m} C p − γ m , for every subgradient selection.
For part 2, the sparse projection assumption supplies the section moments used in the proof of Proposition 6.
For every unit 2 m 2m 2 m -sparse vector v v v and every 2 m 2m 2 m -sparse θ ∈ Θ \theta\in\Theta θ ∈ Θ ,
v T x θ ( c ) = v T X − v 1 θ 1 θ T X + v 1 θ 1 c . v^{\mathsf{T}}x_\theta(c)=v^{\mathsf{T}}X-\frac{v_1}{\theta_1}\theta^{\mathsf{T}}X+\frac{v_1}{\theta_1}c. v T x θ ( c ) = v T X − θ 1 v 1 θ T X + θ 1 v 1 c .
The triangle inequality for ∥ ⋅ ∥ ψ 2 \|\cdot\|_{\psi_2} ∥ ⋅ ∥ ψ 2 , applied separately to the projections along v v v and θ \theta θ , bounds the sub-Gaussian norm of the random part by C C C . Hence all section moments used in (68) hold for sparse test vectors on compact level intervals. When the conditional density derivative is bounded, the proof of (67) also applies, since X T θ X^{\mathsf{T}}\theta X T θ is sub-Gaussian for these sparse decisions. Cauchy–Schwarz then gives the same section moments at every level. In particular, for a displacement direction $u and a test vector v \boldsymbol{v} v , each of sparsity at most 2 m 2m 2 m , the mixed section moments are bounded by k ! C k k!C^k k ! C k without a factor depending on m m m .
Apply (67) along the preceding sparse segment. For d = ∣ θ 1 − θ 0 ∣ 2 d = \lvert\boldsymbol{\theta}_1-\boldsymbol{\theta}_0\rvert_2 d = ∣ θ 1 − θ 0 ∣ 2 , integration against the finite measure ν \nu ν gives
E ∣ v T X { g ( X T θ 1 ) − g ( X T θ 0 ) } ∣ k ≤ d k ! C k , k ≥ 2. \mathbb{E}\left|\boldsymbol{v}^{\mathsf{T}}\boldsymbol{X}\{g(\boldsymbol{X}^{\mathsf{T}}\boldsymbol{\theta}_1)-g(\boldsymbol{X}^{\mathsf{T}}\boldsymbol{\theta}_0)\}\right|^k \le d\,k!C^k,\qquad k\ge2. E v T X { g ( X T θ 1 ) − g ( X T θ 0 )} k ≤ d k ! C k , k ≥ 2.
The first moment is at most C d Cd C d . The slab bound (68), integrated against ν \nu ν , gives the strict-bracket moments. When ν \nu ν has compact support, only a fixed neighborhood of the support of ν \nu ν is used, and its slab widths are eventually smaller than one. When the density derivative is bounded, the bounds hold at all levels.
Take p − 12 p^{-12} p − 12 -nets of Θ \Theta Θ on each coordinate subspace of dimension at most 2 m 2m 2 m , and 1 / 4 1/4 1/4 -nets of the unit spheres on all test supports of that size. The logarithm of their combined cardinality is O ( m log p ) O(m\log p) O ( m log p ) . All decision net points are chosen in Θ \Theta Θ , and hence retain the coefficient bound on coordinate 1. The sub-Gaussian bound on sparse projections gives
max i ≤ n ∣ X i ∣ ( 2 m ) ≤ C m log p . \max_{i\le n}\lvert\boldsymbol{X}_i\rvert_{(2m)}\le C\sqrt{m\log p}. i ≤ n max ∣ X i ∣ ( 2 m ) ≤ C m log p .
with failure probability C p − γ m Cp^{-\gamma m} C p − γ m . For a decision and a representative on the same subspace, their index values differ by at most w = C p − 12 m log p w=Cp^{-12}\sqrt{m\log p} w = C p − 12 m log p on this event. Monotonicity brackets every subgradient selection by
g ( X T θ ~ + 2 w ) − f − ′ ( X T θ ~ − 2 w ) . g(\boldsymbol{X}^{\mathsf{T}}\widetilde{\boldsymbol{\theta}}+2w)-f'_{-}(\boldsymbol{X}^{\mathsf{T}}\widetilde{\boldsymbol{\theta}}-2w). g ( X T θ + 2 w ) − f − ′ ( X T θ − 2 w ) .
The slab moments bound each weighted bracket moment by w k ! C k wk!C^k w k ! C k . The population contribution outside the displayed sample event is at most C p − 10 Cp^{-10} C p − 10 by the sparse projection moments. Bernstein’s inequality over the decision and sphere nets therefore bounds interpolation by C a s Ca_s C a s . Apply the pairwise bound with t = A m log p t=Am\log p t = A m log p over the net pairs and test vectors. The sphere-net inequality gives the restricted increment bound with failure probability C p − γ m Cp^{-\gamma m} C p − γ m . The strict brackets cover every subgradient selection.
Under either set of assumptions, ∣ g ∣ ≤ C f \lvert g\rvert\le C_f ∣ g ∣ ≤ C f makes each pointwise score coordinate bounded or sub-Gaussian. A coordinate union bound gives the pointwise deviation with failure probability C p − γ Cp^{-\gamma} C p − γ . Intersecting it with the increment event proves the joint probability bound.
Under part 2 of this corollary, conditional-density integration gives, for unit 2 m 2m 2 m -sparse u , v \boldsymbol{u},\boldsymbol{v} u , v ,
P ∣ v T X { f + ′ ( X T ( θ + d u ) ) − f + ′ ( X T θ ) } ∣ ≤ C d . P\left|\boldsymbol{v}^{\mathsf{T}}\boldsymbol{X}\{f'_{+}(\boldsymbol{X}^{\mathsf{T}}(\boldsymbol{\theta}+d\boldsymbol{u}))-f'_{+}(\boldsymbol{X}^{\mathsf{T}}\boldsymbol{\theta})\}\right|\le Cd. P v T X { f + ′ ( X T ( θ + d u )) − f + ′ ( X T θ )} ≤ C d .
Consequently, the population gradient increment has ∣ ⋅ ∣ ( 2 m ) \lvert\cdot\rvert_{(2m)} ∣ ⋅ ∣ ( 2 m ) -norm at most C d Cd C d . Combining this bound with the increment bound above, separately for d ≥ a s d\ge a_s d ≥ a s and d < a s d<a_s d < a s , proves part 1 of Corollary 6 for the index objective.
For restricted strong convexity, let D = ∣ θ 1 − θ 2 ∣ 2 D=\lvert\boldsymbol{\theta}_1-\boldsymbol{\theta}_2\rvert_2 D = ∣ θ 1 − θ 2 ∣ 2 and θ t = θ 2 + t ( θ 1 − θ 2 ) \boldsymbol{\theta}_t=\boldsymbol{\theta}_2+t(\boldsymbol{\theta}_1-\boldsymbol{\theta}_2) θ t = θ 2 + t ( θ 1 − θ 2 ) . This segment stays in Θ s c \Theta_{\mathrm{sc}} Θ sc , and every pair on it has joint support of size at most 2 m 2m 2 m . The increment bound proved above therefore controls the empirical subgradient increment minus its expectation at θ t \boldsymbol{\theta}_t θ t and θ 2 \boldsymbol{\theta}_2 θ 2 on the same event. Pairing it with the 2 m 2m 2 m -sparse displacement and integrating gives
L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ μ 0 2 D 2 − C a s D 3 / 2 − C a s D ≥ μ 0 4 D 2 \mathcal{L}(\boldsymbol{\theta}_1)-\mathcal{L}(\boldsymbol{\theta}_2)-\langle\partial\mathcal{L}(\boldsymbol{\theta}_2),\boldsymbol{\theta}_1-\boldsymbol{\theta}_2\rangle\ge\frac{\mu_0}{2}D^2-C\sqrt{a_s}D^{3/2}-Ca_sD\ge\frac{\mu_0}{4}D^2 L ( θ 1 ) − L ( θ 2 ) − ⟨ ∂ L ( θ 2 ) , θ 1 − θ 2 ⟩ ≥ 2 μ 0 D 2 − C a s D 3/2 − C a s D ≥ 4 μ 0 D 2
when D ≥ C 2 a s D\ge C_2a_s D ≥ C 2 a s . This proves the stated restricted strong convexity with μ = μ 0 \mu=\mu_0 μ = μ 0 .
For the local approximation, ∣ ⋅ ∣ ∞ ≤ ∣ ⋅ ∣ ( 2 m ) \lvert\cdot\rvert_\infty\le\lvert\cdot\rvert_{(2m)} ∣ ⋅ ∣ ∞ ≤ ∣ ⋅ ∣ ( 2 m ) and the assumed population gradient expansion give, for D = ∣ θ − θ ∗ ∣ 2 ≤ C 3 a s D=\lvert\boldsymbol{\theta}-\boldsymbol{\theta}^*\rvert_2\le C_3\sqrt{a_s} D = ∣ θ − θ ∗ ∣ 2 ≤ C 3 a s ,
∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∣ ∞ ≤ C ( D a s + a s + D 2 ) ≤ C 4 a s 3 / 4 . \lvert\partial\mathcal{L}(\boldsymbol{\theta})-\partial\mathcal{L}(\boldsymbol{\theta}^*)-\boldsymbol{H}(\boldsymbol{\theta}-\boldsymbol{\theta}^*)\rvert_\infty\le C(\sqrt{Da_s}+a_s+D^2)\le C_4a_s^{3/4}. ∣ ∂ L ( θ ) − ∂ L ( θ ∗ ) − H ( θ − θ ∗ ) ∣ ∞ ≤ C ( D a s + a s + D 2 ) ≤ C 4 a s 3/4 .
This proves the local condition with the radius and remainder.
Suppose now that ν \nu ν has compact support and the density derivative is bounded. On its support, for two decisions with joint support of size at most 2 m 2m 2 m ,
∥ a θ ( c ) − a θ ′ ( c ) ∥ ψ 2 ≤ C ∣ θ − θ ′ ∣ 2 . \lVert a_{\theta}(c)-a_{\theta'}(c)\rVert_{\psi_2}\le C\lvert\theta-\theta'\rvert_2. ∥ a θ ( c ) − a θ ′ ( c ) ∥ ψ 2 ≤ C ∣ θ − θ ′ ∣ 2 .
Subtracting their defining fractions and applying the sparse projection bound separately to θ \theta θ , θ ′ \theta' θ ′ , and their displacement gives this inequality. Conditional integration gives H ( θ ) = E ∫ x θ ( c ) x θ ( c ) T ρ θ ( c ∣ U ) ν ( d c ) \mathbf{H}(\theta)=\mathbb{E}\int x_{\theta}(c)x_{\theta}(c)^{\mathsf{T}}\rho_{\theta}(c\mid U)\nu(\mathrm{d}c) H ( θ ) = E ∫ x θ ( c ) x θ ( c ) T ρ θ ( c ∣ U ) ν ( d c ) . The density derivative bound and Hölder’s inequality therefore give
∣ v T { H ( θ ) − H ( θ ′ ) } u ∣ ≤ C ∣ θ − θ ′ ∣ 2 \lvert v^{\mathsf{T}}\{\mathbf{H}(\theta)-\mathbf{H}(\theta')\}u\rvert\le C\lvert\theta-\theta'\rvert_2 ∣ v T { H ( θ ) − H ( θ ′ )} u ∣ ≤ C ∣ θ − θ ′ ∣ 2
for unit 2 m 2m 2 m -sparse u , v u,v u , v . Integrating this bound along the segment to θ ∗ \theta^* θ ∗ proves (17), and also its version in ∣ ⋅ ∣ ( 2 m ) \lvert\cdot\rvert_{(2m)} ∣ ⋅ ∣ ( 2 m ) . Combining that Taylor bound with the increment event proves the local subgradient approximation in Corollary 6. When ν \nu ν is not compactly supported, the stated Taylor assumption supplies this step.
Under the bounded-coordinate assumptions, the first-moment crossing calculation gives a population maximum-norm increment at most C d Cd C d . Conversion to ∣ ⋅ ∣ ( 2 m ) \lvert\cdot\rvert_{(2m)} ∣ ⋅ ∣ ( 2 m ) multiplies the smoothness constant by 2 m \sqrt{2m} 2 m . In the strong convexity calculation the scalar empirical remainder acquires the same factor, and D ≥ C 2 m a s D\ge C_{2m}a_s D ≥ C 2 m a s suffices. The maximum-norm local approximation still has remainder C 4 a s 3 / 4 C_4a_s^{3/4} C 4 a s 3/4 .
For a deterministic J ⊇ S J\supseteq S J ⊇ S , coordinate 1 belongs to J J J . The conditional density of X 1 X_1 X 1 given X J ∖ { 1 } X_{J\setminus\{1\}} X J ∖ { 1 } is the conditional average of ρ X ( x 1 ∣ X − 1 ) \rho_X(x_1\mid X_{-1}) ρ X ( x 1 ∣ X − 1 ) over the omitted coordinates. Its density and derivative bounds are preserved by conditional averaging and dominated difference quotients. Every unit projection of X J X_J X J has bounded sub-Gaussian norm. The restricted region remains convex and bounded, with ∣ θ 1 ∣ ≥ c 0 \lvert\theta_1\rvert\ge c_0 ∣ θ 1 ∣ ≥ c 0 . Applying Proposition 6 to the restriction to J J J gives increment scale ∣ J ∣ log n / n \lvert J\rvert\log n/n ∣ J ∣ log n / n . Independence gives ∣ { ∂ L ( θ ∗ ) } J ∣ 2 = O P ( ∣ J ∣ / n ) \lvert\{\partial\mathcal{L}(\theta^*)\}_J\rvert_2=O_{\mathsf{P}}(\sqrt{\lvert J\rvert/n}) ∣{ ∂ L ( θ ∗ ) } J ∣ 2 = O P ( ∣ J ∣ / n ) . Together with the Taylor bound above, these estimates give the inputs used in the recursion and normal limit for a deterministic retained set in Corollary 7. The uniform increment and pointwise score bounds proved above supply the events used for data-dependent retained sets and for the warm-start recursion.