Introduction
Modern optimizers are non-Euclidean in practice. Modern deep-learning training increasingly uses geometry-aware primitives, including matrix preconditioning and orthogonalized updates. Matrix-aware optimizers include Shampoo [32 ] , which uses matrix preconditioning, and Muon [38 ] and Dion [1 ] , which use orthogonalized matrix updates. Recent studies report improved training efficiency over AdamW for Muon [54 ] and Dion [1 ] in Transformer language-model experiments. Their empirical advantages are often interpreted through the connection between these matrix updates and steepest descent with respect to non-Euclidean norms, such as the spectral norm and related Schatten norms [9 , 52 ] . These norms respect the matrix structure of neural-network parameters, including the weight matrices in Transformer attention and feed-forward layers. This geometric perspective motivates a stochastic theory developed directly in general normed spaces.
Theory gap: stochastic non-Euclidean SGD under the “right” noise model. A central modeling question in stochastic non-Euclidean optimization is how to quantify oracle noise. Most existing analyses [50 , 30 , 53 , 71 , 91 , 80 ] assume Euclidean (or Frobenius) moment bounds or bounded variance and then transfer results to a target non-Euclidean norm via norm equivalence. Such conversions can introduce dimension-dependent inflation and may obscure the regimes in which a geometry-aware method should provably help. We therefore seek guarantees under a noise model stated in the native norm .
Our lens: dual-norm light tails and κ \kappa κ -regularity. We measure oracle noise directly in the dual norm ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ induced by the chosen update norm ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ (Assumption 3). Using the κ \kappa κ -regularity framework [40 ] , we obtain high-probability guarantees with the additional geometric dependence captured by the dual-space regularity constant κ \kappa κ . Every nonzero finite-dimensional normed space admits such a constant, so this approach does not restrict the choice of update norm (Section 3.3). For suitable geometries and noise models, it yields sharper bounds than Euclidean/Frobenius conversions (§ 3.6.1 ).
Problem setup. Throughout the paper, E E E denotes a finite-dimensional space with dim ( E ) ≥ 1 \dim(E) \ge1 dim ( E ) ≥ 1 . We consider the distributed, non-convex stochastic optimization problem expressed as a finite sum over N N N workers:
min x ∈ E { f ( x ) : = 1 N ∑ i = 1 N f i ( x ) } (1) \min_{x\in E}\left\{f(x) := \frac{1}{N}\sum_{i=1}^{N} f_i(x)\right\} \tag*{(1)} x ∈ E min { f ( x ) := N 1 i = 1 ∑ N f i ( x ) } (1)
where each f i f_i f i is an expectation over a worker-specific data distribution D i \mathcal{D}_i D i :
f i ( x ) : = E ξ i ∼ D i [ f i ( x ; ξ i ) ] . (2) f_i(x) := \mathbb{E}_{\xi_i\sim\mathcal{D}_i}[f_i(x;\xi_i)] . \tag*{(2)} f i ( x ) := E ξ i ∼ D i [ f i ( x ; ξ i )] . (2)
The space E E E may represent, for instance, vectors in R d \mathbb{R}^d R d or matrices in R m × n \mathbb{R}^{m\times n} R m × n . For each worker i i i , the function f i : E × S i → R f_i:E\times S_i\to\mathbb{R} f i : E × S i → R is defined over a sample space S i S_i S i , and ξ i \xi_i ξ i is a random variable drawn from D i \mathcal{D}_i D i . The problem is unconstrained : the methods we analyze contain no projection, so nothing confines the iterates to a proper subset of E E E , and every assumption below is stated on all of E E E .
This formulation is particularly relevant to Federated Learning (FL) [48 , 56 , 42 ] . In this context, two primary forms of heterogeneity arise: device heterogeneity , which concerns variations in system capabilities, and statistical heterogeneity , where data distributions D i \mathcal{D}_i D i differ across workers. To isolate the effects of device heterogeneity, we assume that all workers share the same sample loss f ( ⋅ ; ⋅ ) f(\cdot;\cdot) f ( ⋅ ; ⋅ ) and sampling distribution D \mathcal{D} D . Thus, f i ( x ; ξ ) = f ( x ; ξ ) f_i(x;\xi)=f(x;\xi) f i ( x ; ξ ) = f ( x ; ξ ) and D i = D \mathcal{D}_i=\mathcal{D} D i = D for every worker i i i , so all local objectives coincide with f ( x ) = E ξ ∼ D [ f ( x ; ξ ) ] f(x)=\mathbb{E}_{\xi\sim\mathcal{D}}[f(x;\xi)] f ( x ) = E ξ ∼ D [ f ( x ; ξ )] . In this case, (1) reduces to
min x ∈ E { f ( x ) : = E ξ ∼ D [ f ( x ; ξ ) ] } . (3) \min_{x\in E}\{f(x) := \mathbb{E}_{\xi\sim\mathcal{D}}[f(x;\xi)]\}. \tag*{(3)} x ∈ E min { f ( x ) := E ξ ∼ D [ f ( x ; ξ )]} . (3)
Our algorithms access f f f through a stochastic first-order oracle: given a query point x x x , it returns a random vector ∇ f ( x ; ξ ) ∈ E ∗ \nabla f(x;\xi)\in E^* ∇ f ( x ; ξ ) ∈ E ∗ that estimates ∇ f ( x ) \nabla f(x) ∇ f ( x ) . In the sample-loss setting above, this response is the gradient of f ( x ; ξ ) f(x;\xi) f ( x ; ξ ) with respect to x x x . More generally, we allow stochastic estimates that need not be gradients of sample losses; the oracle assumptions are stated formally in § 3.2 . In the parallel setting, all workers use the same oracle map and sampling distribution D \mathcal{D} D .
Throughout the paper, we assume that f f f is L L L -smooth (Assumption 1) and bounded from below (Assumption 2), and that the oracle responses are unbiased and satisfy a light-tail condition in the induced dual norm ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ . (Assumption 3).
We work in the following parallel computing setup:
A fixed number of N N N workers are available to perform computations in parallel.
Worker i i i requires a fixed time τ i \tau_i τ i seconds to compute one stochastic gradient after receiving a query point. These times remain fixed throughout the run, and we index the workers so that 0 < τ 1 ≤ τ 2 ≤ ⋯ ≤ τ N 0 < \tau_1 \le\tau_2 \le\cdots\le\tau_N 0 < τ 1 ≤ τ 2 ≤ ⋯ ≤ τ N .
Communication in both directions between the server and the workers, and server-side processing, including evaluation of the sharp update, are treated as instantaneous, as in the computation model of § 4 .
From oracle complexity to wall-clock time: heterogeneous parallelism. On the systems side, asynchronous SGD has been shown to improve wall-clock convergence guarantees over synchronous minibatch [57 ] that waits for all workers, in settings with heterogeneous worker speeds. This line of work emphasizes time complexity —the wall-clock time needed to reach a prescribed accuracy—as a natural measure of parallel performance. In the Euclidean setting, Tyurin and Richtárik [ [84 ] establish a lower bound for their zero-respecting algorithm class and a matching upper bound, up to constants, for Rennala SGD under their specified unbiased bounded-variance oracle model with fixed worker computation times. On the optimization side, non-Euclidean methods are motivated by modern matrix-aware updates, yet their interaction with computational heterogeneity remains poorly understood. Bridging these two directions requires (i) tight control of stochastic gradient noise in the induced dual norm ∥ ⋅ ∥ ∗ \|\cdot\|_{*} ∥ ⋅ ∥ ∗ associated with the chosen update norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ and (ii) a wall-clock analysis that accounts for heterogeneous worker speeds under non-Euclidean updates. Together, these two directions motivate the following question:
Should stochastic gradient noise be controlled in a Euclidean or Frobenius norm and then converted, via norm equivalence, to the dual norm ∥ ⋅ ∥ ∗ \|\cdot\|_{*} ∥ ⋅ ∥ ∗ induced by the update norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ ? Or can a noise condition stated directly in ∥ ⋅ ∥ ∗ \|\cdot\|_{*} ∥ ⋅ ∥ ∗ yield sharper oracle and wall-clock guarantees?
Notation. The notation used throughout the paper is collected in Section A, which gives the main definitions followed by a consolidated reference table.
Our contributions. We study non-convex stochastic optimization in a normed space ( E , ∥ ⋅ ∥ ) (E,\|\cdot\|) ( E , ∥ ⋅ ∥ ) and focus on the statistically homogeneous but computationally heterogeneous parallel regime. Our goal is to characterize both (i) the number of stochastic gradients needed to reach an ε \varepsilon ε -stationary point and (ii) the wall-clock time needed when workers have different speeds.
Our contributions are threefold:
(i) Minibatch non-Euclidean SGD under a dual-norm light-tail oracle. We establish high-probability oracle-complexity bounds for minibatch SGD under the stated smoothness, lower-boundedness, and unbiased dual-norm light-tail assumptions. The key ingredient is a minibatch concentration inequality for fresh i.i.d. samples in regular dual spaces (Lemma 1). Combining it with non-Euclidean descent, we show that an ε \varepsilon ε -stationary point is reached with probability at least 1 − δ 1-\delta 1 − δ using
O ~ ( L Δ ε + κ L Δ σ 2 ε 2 ) (4) \widetilde{O}\left(\frac{L\Delta}{\varepsilon}+\frac{\kappa L\Delta\sigma^{2}}{\varepsilon^{2}}\right) \tag*{(4)} O ( ε L Δ + ε 2 κ L Δ σ 2 ) (4)
stochastic gradient evaluations (Theorem 1 and Corollary 1), where Δ = f ( x 0 ) − f inf \Delta=f(x^{0})-f^{\inf} Δ = f ( x 0 ) − f i n f is the initial functional gap and κ \kappa κ is a regularity constant of the dual space ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^{*},\|\cdot\|_{*}) ( E ∗ , ∥ ⋅ ∥ ∗ ) . In the Euclidean case, where κ = 1 \kappa=1 κ = 1 , this matches the optimal oracle complexity up to logarithmic factors for general unbiased light-tail oracles, in the regime described in § 3.6 .
(ii) Sharp regularity bounds and improvements over norm-equivalence conversions. We establish matching upper and lower bounds on the optimal κ \kappa κ -regularity parameter for ℓ p d \ell_{p}^{d} ℓ p d and Schatten-p p p spaces, exact for 1 ≤ p < 2 1\leq p<2 1 ≤ p < 2 and up to absolute constants for 2 ≤ p ≤ ∞ 2\leq p\leq\infty 2 ≤ p ≤ ∞ (Table 3 and Section L). For some non-Euclidean geometries and noise models, the native-norm analysis yields significantly smaller oracle-complexity bounds than those obtained by converting a Euclidean/Frobenius noise assumption. In particular, for the ℓ 1 \ell_{1} ℓ 1 update norm under dense Rademacher perturbations, the statistical coefficient grows logarithmically rather than linearly with the dimension. We quantify these comparisons for ℓ p d \ell_{p}^{d} ℓ p d and Schatten-p p p dual geometries in § 3.6.1 and Section J.
Table 3. Examples of κ \kappa κ -regular spaces for V = E ∗ V=E^* V = E ∗ with corresponding regularity parameters κ \kappa κ and auxiliary norm ∥ ⋅ ∥ ∗ , + \|\cdot\|_{*,+} ∥ ⋅ ∥ ∗ , + .
(iii) Wall-clock guarantees for semi-asynchronous non-Euclidean SGD: Rennala NSGD. Building on the minibatch theory, we introduce Rennala NSGD, a semi-asynchronous parallel method for spaces with non-Euclidean geometry, which collects fresh minibatches at the current iterate and discards stale gradients. In a system with worker computation times τ i \tau_i τ i , we prove (Theorem 2) the wall-clock time bound
O ~ ( ( min S ∈ [ N ] 1 S ∑ i = 1 S 1 τ i ) − 1 ( L Δ ε + κ L Δ σ 2 S ε 2 ) ) . (5) \widetilde{O}\left(\left(\min_{S\in[N]}\frac{1}{S}\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1}\left(\frac{L\Delta}{\varepsilon}+\frac{\kappa L\Delta\sigma^{2}}{S\varepsilon^{2}}\right)\right). \tag*{(5)} O ( S ∈ [ N ] min S 1 i = 1 ∑ S τ i 1 ) − 1 ( ε L Δ + S ε 2 κ L Δ σ 2 ) . (5)
In the Euclidean case, where κ = 1 \kappa=1 κ = 1 , (5) matches the parameter dependence of the classical Rennala SGD time-complexity bound up to logarithmic factors [84 ] ({xref}Section 4.1`).
Related Work
Our work connects two research threads: (i) stochastic optimization in a general normed space ( E , ∥ ⋅ ∥ ) (E,\|\cdot\|) ( E , ∥ ⋅ ∥ ) with non-Euclidean norm , and (ii) time-aware parallel optimization under computational heterogeneity.
Non-Euclidean Optimization. Recent work on large-scale training has explored matrix-aware optimizers, including the Muon and Shampoo families, that approximate steepest-descent directions in non-Euclidean norms [38 , 70 , 6 ] . Several existing theoretical analyses of these methods in the non-distributed setting [53 , 71 , 45 , 58 , 50 ] control stochastic gradient noise in Euclidean or Frobenius norms. Some of these analyses use norm equivalence to obtain guarantees in a target non-Euclidean norm . Such conversions can introduce dimension- or rank-dependent factors. Our analysis instead imposes a light-tail condition directly in the dual norm induced by the update geometry.
In distributed settings, several works have explored Muon-style methods. Zhang and Gao [ [91 ] and Takezawa et al. [ [80 ] analyze synchronous federated training and address the statistical bias of local updates, while Gruntkowska et al. [ [30 ] focuses on communication efficiency via error feedback. These studies primarily target communication overhead or local-update bias within synchronized rounds (typically under Frobenius/Euclidean noise models), and do not address computational heterogeneity or wall-clock bounds.
Stochastic Optimization in Regular Spaces. The κ \kappa κ -regularity framework of Juditsky and Nemirovski [ [40 ] provides large-deviation bounds for martingale sums in finite-dimensional normed spaces, with geometric dependence captured by the regularity constant κ \kappa κ . Under conditional exponential-moment bounds on the squared norms of the increments, they obtain sub-Gaussian deviation bounds for the norm of the martingale sum. These tools were subsequently incorporated into the analysis of stochastic first-order methods [see, e.g., Juditsky et al., 2013]. Related martingale large-deviation bounds have also been used to derive non-asymptotic confidence bounds for assessing the quality of solutions to stochastic optimization problems [31 ] .
Time Complexity in Heterogeneous Parallel SGD. In the distributed regime, parallelizing SGD is a natural strategy for accelerating training. The simplest approach, minibatch SGD , synchronizes N N N workers at each step to compute gradients in parallel, after which a central server performs the update [26 , 14 , 27 ] : x k + 1 = x k − γ N ∑ i = 1 N ∇ f ( x k ; ξ i k ) x^{k+1}=x^k-\frac{\gamma}{N}\sum_{i=1}^{N}\nabla f(x^k;\xi_i^k) x k + 1 = x k − N γ ∑ i = 1 N ∇ f ( x k ; ξ i k ) . For a general minibatch size B B B and a suitable stepsize, the standard Euclidean/Frobenius analysis gives the iteration-complexity bound T S G D F ( B ) = O ( L F Δ ϵ F + σ F 2 L F Δ B ϵ F 2 ) T_{\mathrm{SGD}}^{\mathrm{F}}(B)=O\left(\frac{L_{\mathrm{F}}\Delta}{\epsilon_{\mathrm{F}}}+\frac{\sigma_{\mathrm{F}}^2L_{\mathrm{F}}\Delta}{B\epsilon_{\mathrm{F}}^2}\right) T SGD F ( B ) = O ( ϵ F L F Δ + B ϵ F 2 σ F 2 L F Δ ) [24 ] .4 with B = N B=N B = N for the synchronized method above. Since each iteration waits for all workers, heterogeneous worker speeds make the slowest worker the bottleneck [20 , 12 ] : if worker i i i takes τ i \tau_i τ i seconds per gradient, the runtime scales as τ max T S G D F ( N ) \tau_{\max}T_{\mathrm{SGD}}^{\mathrm{F}}(N) τ m a x T SGD F ( N ) , where τ max = max i τ i \tau_{\max}=\max_i\tau_i τ m a x = max i τ i .
To overcome this synchronization barrier, Asynchronous SGD (ASGD) was introduced [61 , 22 , 59 , 7 ] . In its standard form, the server applies each arriving stochastic gradient immediately: x k + 1 = x k − γ k ∇ f ( x k − δ k ; ξ k − δ k ) x^{k+1}=x^k-\gamma_k\nabla f(x^{k-\delta_k};\xi_{k-\delta_k}) x k + 1 = x k − γ k ∇ f ( x k − δ k ; ξ k − δ k ) , where δ k \delta_k δ k is the gradient delay and γ k \gamma_k γ k may depend on it. Recent theoretical advances [57 , 47 , 13 ] establish wall-clock bounds for suitable asynchronous methods with fixed worker computation times τ i \tau_i τ i of order ( 1 N ∑ i = 1 N 1 τ i ) − 1 T S G D F ( N ) \left(\frac{1}{N}\sum_{i=1}^{N}\frac{1}{\tau_i}\right)^{-1}T_{\mathrm{SGD}}^{\mathrm{F}}(N) ( N 1 ∑ i = 1 N τ i 1 ) − 1 T SGD F ( N ) , governed by the harmonic-mean worker time rather than the slowest worker (Section C.2).
However, to the best of our knowledge, the available wall-clock bounds for classical ASGD remain suboptimal in general. Tyurin and Richtárik [ [84 ] close this gap with Rennala SGD , a semi-asynchronous method that uses only gradients computed at the current iterate. Its wall-clock bound is min S ∈ [ N ] { ( 1 S ∑ i = 1 S 1 τ i ) − 1 T S G D F G D ( S ) } \min_{S \in[N]} \left\{\left(\frac{1}{S}\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1} T_{\mathrm{SGD}}^{\mathrm{FGD}}(S)\right\} min S ∈ [ N ] { ( S 1 ∑ i = 1 S τ i 1 ) − 1 T SGD FGD ( S ) } , where 0 < τ 1 ≤ ⋯ ≤ τ N 0 < \tau_1 \le\cdots\le\tau_N 0 < τ 1 ≤ ⋯ ≤ τ N and S S S counts the fastest workers considered in the bound. This bound is minimax-optimal up to constants in their model.5
Asynchrony with non-Euclidean updates has been explored mostly in convex/online regimes [18 , 22 , 39 ] . These methods use mirror descent or dual averaging, with non-Euclidean geometry specified through a prox function and its associated Bregman divergence. Our approach instead uses steepest-descent steps defined by an update norm and a fixed sharp selection. These works focus on delay-robust iteration/regret guarantees, and do not provide time-optimal wall-clock bounds under computational heterogeneity, nor nonconvex stationarity guarantees in general normed spaces.
We combine our stochastic analysis of SGD for arbitrary non-Euclidean norms with a Rennala-style semi-asynchronous architecture to obtain oracle and wall-clock guarantees beyond the Euclidean/Frobenius regime. Additional literature review references are deferred to Appendix § 9 .
SGD Under Non-Euclidean Norms
We derive high-probability convergence and oracle-complexity guarantees for minibatch SGD in nonconvex optimization, using steepest-descent updates in an arbitrary norm ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ and measuring oracle noise in its induced dual norm ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ . Notation is collected in Section A.
Non-Euclidean Update Rule and the Sharp Operator
Minibatch non-Euclidean SGD. At each iteration k k k , draw a fresh minibatch of i.i.d. samples { ξ j k } j = 1 B \{\xi_j^k\}_{j=1}^{B} { ξ j k } j = 1 B from D \mathcal{D} D , independent of the past. Denote the minibatch stochastic gradient as
g k : = 1 B ∑ j = 1 B ∇ f ( x k ; ξ j k ) ∈ E ∗ . (6) g^k := \frac{1}{B}\sum_{j=1}^{B}\nabla f(x^k;\xi_j^k) \in E^*. \tag*{(6)} g k := B 1 j = 1 ∑ B ∇ f ( x k ; ξ j k ) ∈ E ∗ . (6)
Fix a Borel-measurable selection s ↦ s ♯ s \mapsto s^\sharp s ↦ s ♯ with s ♯ ∈ ∂ ϕ ∗ ( s ) s^\sharp\in\partial\phi^*(s) s ♯ ∈ ∂ ϕ ∗ ( s ) for every s ∈ E ∗ s \in E^* s ∈ E ∗ . Given a prescribed sequence of positive stepsizes { γ k } \{\gamma_k\} { γ k } , the non-Euclidean minibatch SGD update is
x k + 1 = x k − γ k ( g k ) ♯ , where ( g k ) ♯ ∈ ∂ ϕ ∗ ( g k ) . (7) x^{k+1}=x^k-\gamma_k(g^k)^\sharp,\qquad\text{where }(g^k)^\sharp\in\partial\phi^*(g^k). \tag*{(7)} x k + 1 = x k − γ k ( g k ) ♯ , where ( g k ) ♯ ∈ ∂ ϕ ∗ ( g k ) . (7)
The convergence and complexity bounds below use the constant choice γ k ≡ 1 2 L \gamma_k \equiv\frac{1}{2L} γ k ≡ 2 L 1 .
Steepest descent / LMO form. The update in (7) admits equivalent steepest-descent and Linear Minimization Oracle (LMO) characterizations in ( E , ∥ ⋅ ∥ ) (E,\lVert\cdot\rVert) ( E , ∥ ⋅ ∥) (Section G).6 The LMO form, with the minimizer determined by the fixed sharp selection, is
x k + 1 ∈ lmo B ( x k , t k ) ( g k ) , t k : = γ k ∥ g k ∥ ∗ , (8) x^{k+1}\in\operatorname{lmo}_{\mathcal{B}(x^k,t_k)}(g^k),\qquad t_k:=\gamma_k\lVert g^k\rVert_*, \tag*{(8)} x k + 1 ∈ lmo B ( x k , t k ) ( g k ) , t k := γ k ∥ g k ∥ ∗ , (8)
where lmo B ( x k , t k ) ( g k ) : = arg min x ∈ B ( x k , t k ) ⟨ g k , x ⟩ \operatorname{lmo}_{\mathcal{B}(x^k,t_k)}(g^k):=\arg\min_{x\in\mathcal{B}(x^k,t_k)}\langle g^k,x\rangle lmo B ( x k , t k ) ( g k ) := arg min x ∈ B ( x k , t k ) ⟨ g k , x ⟩ and B ( x k , t k ) : = { x ∈ E : ∥ x − x k ∥ ≤ t k } \mathcal{B}(x^k,t_k):=\{x\in E:\lVert x-x^k\rVert\le t_k\} B ( x k , t k ) := { x ∈ E : ∥ x − x k ∥ ≤ t k } .
Special cases of the update (7). Different choices of the update norm ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ recover familiar update rules (up to stepsize reparameterizations).
ℓ 2 \ell_2 ℓ 2 norm (Euclidean case). If ∥ ⋅ ∥ = ∥ ⋅ ∥ 2 \lVert\cdot\rVert=\lVert\cdot\rVert_2 ∥ ⋅ ∥ = ∥ ⋅ ∥ 2 , then ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 \lVert\cdot\rVert_*=\lVert\cdot\rVert_2 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 and ( g k ) ♯ = g k (g^k)^\sharp=g^k ( g k ) ♯ = g k , hence (7) reduces to classical SGD [26 , 25 , 15 ] .
ℓ ∞ \ell_\infty ℓ ∞ norm. If ∥ ⋅ ∥ = ∥ ⋅ ∥ ∞ \lVert\cdot\rVert=\lVert\cdot\rVert_\infty ∥ ⋅ ∥ = ∥ ⋅ ∥ ∞ (so ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 1 \lVert\cdot\rVert_*=\lVert\cdot\rVert_1 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 1 ), then ( g k ) ♯ ∈ ∥ g k ∥ 1 ∂ ∥ g k ∥ 1 (g^k)^\sharp\in\lVert g^k\rVert_1\partial\lVert g^k\rVert_1 ( g k ) ♯ ∈ ∥ g k ∥ 1 ∂ ∥ g k ∥ 1 and a canonical selection is ( g k ) ♯ = ∥ g k ∥ 1 sign ( g k ) (g^k)^\sharp=\lVert g^k\rVert_1\operatorname{sign}(g^k) ( g k ) ♯ = ∥ g k ∥ 1 sign ( g k ) , where sign ( g k ) i = 0 \operatorname{sign}(g^k)_i=0 sign ( g k ) i = 0 . This yields a sign-type update closely related to signSGD [10 , 79 ] .
ℓ 1 \ell_1 ℓ 1 norm. If ∥ ⋅ ∥ = ∥ ⋅ ∥ 1 \lVert\cdot\rVert=\lVert\cdot\rVert_1 ∥ ⋅ ∥ = ∥ ⋅ ∥ 1 (so ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_*=\lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ ∞ ), then ( g k ) ♯ ∈ ∥ g k ∥ ∞ ∂ ∥ g k ∥ ∞ (g^k)^\sharp\in\lVert g^k\rVert_\infty\partial\lVert g^k\rVert_\infty ( g k ) ♯ ∈ ∥ g k ∥ ∞ ∂ ∥ g k ∥ ∞ , i.e., ( g k ) ♯ (g^k)^\sharp ( g k ) ♯ is supported on (a subset of) the coordinates attaining ∥ g k ∥ ∞ \lVert g^k\rVert_\infty ∥ g k ∥ ∞ . For g k ≠ 0 g^k\ne0 g k = 0 , an extreme-point selection yields a
greedy single-coordinate step in (7) [81 , 82 , 60 ] . A general sharp selection may update several coordinates tied for the largest ∣ g i k ∣ |g_i^k| ∣ g i k ∣ .
A detailed derivation of the update (7) for the ℓ 1 \ell_1 ℓ 1 and ℓ ∞ \ell_\infty ℓ ∞ update norms is deferred to Section F.
Spectral norm and orthogonalized matrix updates. When E = R m × n E = \mathbb{R}^{m \times n} E = R m × n and ∥ ⋅ ∥ = ∥ ⋅ ∥ o p \lVert\cdot\rVert= \lVert\cdot\rVert_{\mathrm{op}} ∥ ⋅ ∥ = ∥ ⋅ ∥ op is the spectral norm, the dual norm is the nuclear norm ∥ ⋅ ∥ n u c \lVert\cdot\rVert_{\mathrm{nuc}} ∥ ⋅ ∥ nuc . With the canonical sharp selection ( G k ) ♯ = ∥ G k ∥ n u c orth ( G k ) (G^k)^\sharp= \lVert G^k \rVert_{\mathrm{nuc}} \operatorname{orth}(G^k) ( G k ) ♯ = ∥ G k ∥ nuc orth ( G k ) , the update takes the following orthogonalized form (derivation in Appendix § 14 ):
X k + 1 = X k − γ k ∥ G k ∥ n u c ⋅ O k , O k = orth ( G k ) , (9) X^{k+1} = X^k - \gamma_k \lVert G^k \rVert_{\mathrm{nuc}} \cdot O^k,\qquad O^k = \operatorname{orth}(G^k), \tag*{(9)} X k + 1 = X k − γ k ∥ G k ∥ nuc ⋅ O k , O k = orth ( G k ) , (9)
where G k G^k G k is the minibatch gradient in E ∗ E^* E ∗ and
orth ( G ) : = ( G G ⊤ ) † / 2 G . (10) \operatorname{orth}(G) := (GG^\top)^{\dagger/2}G. \tag*{(10)} orth ( G ) := ( G G ⊤ ) † /2 G . (10)
For a compact rank-r r r SVD G = U r Σ r V r ⊤ G = U_r \Sigma_r V_r^\top G = U r Σ r V r ⊤ , where r = rank ( G ) r = \operatorname{rank}(G) r = rank ( G ) and only positive singular values are retained, one has orth ( G ) = U r V r ⊤ \operatorname{orth}(G) = U_r V_r^\top orth ( G ) = U r V r ⊤ (the polar factor / partial isometry).7
Oracle Model
Following Tyurin and Richtárik [ [84 ] , we consider a jointly measurable stochastic gradient oracle ( x , ξ ) ↦ ∇ f ( x ; ξ ) ∈ E ∗ (x,\xi) \mapsto\nabla f(x;\xi) \in E^* ( x , ξ ) ↦ ∇ f ( x ; ξ ) ∈ E ∗ , with a fresh independent sample ξ ∼ D \xi\sim\mathcal{D} ξ ∼ D at each call.8 We analyze (7) under the following assumptions.
Assumption 1 (Non-Euclidean L L L -smoothness). The function f f f is L L L -smooth with respect to ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ if
∥ ∇ f ( x ) − ∇ f ( y ) ∥ ∗ ≤ L ∥ x − y ∥ for all x , y ∈ E . \lVert\nabla f(x) - \nabla f(y) \rVert_* \le L \lVert x-y \rVert\quad\text{for all } x,y \in E. ∥ ∇ f ( x ) − ∇ f ( y ) ∥ ∗ ≤ L ∥ x − y ∥ for all x , y ∈ E .
Assumption 2 (Lower bound). There exists f i n f ∈ R f^{\mathrm{inf}} \in\mathbb{R} f inf ∈ R such that f ( x ) ≥ f i n f f(x) \ge f^{\mathrm{inf}} f ( x ) ≥ f inf for all x ∈ E x \in E x ∈ E .
Assumption 3 (Dual-norm light-tail oracle induced by the update norm). For all x ∈ E x \in E x ∈ E , the stochastic gradient ∇ f ( x ; ξ ) \nabla f(x;\xi) ∇ f ( x ; ξ ) is unbiased, E ξ [ ∇ f ( x ; ξ ) ] = ∇ f ( x ) \mathbb{E}_{\xi}[\nabla f(x;\xi)] = \nabla f(x) E ξ [ ∇ f ( x ; ξ )] = ∇ f ( x ) , and satisfies the light-tail condition in the induced dual norm
E ξ [ exp ( ∥ ∇ f ( x ; ξ ) − ∇ f ( x ) ∥ ∗ 2 σ 2 ) ] ≤ exp ( 1 ) (11) \mathbb{E}_{\xi}\left[\exp\left(\frac{\lVert\nabla f(x;\xi)-\nabla f(x)\rVert_*^2}{\sigma^2}\right)\right] \le\exp(1) \tag*{(11)} E ξ [ exp ( σ 2 ∥ ∇ f ( x ; ξ ) − ∇ f ( x ) ∥ ∗ 2 ) ] ≤ exp ( 1 ) (11)
for some σ > 0 \sigma> 0 σ > 0 .
Assumptions 1 and 2 are standard in the analysis of stochastic first-order methods [24 , 26 ] . Assumption 3 represents a non-Euclidean generalization of the standard light-tail condition—typically formulated in the Euclidean or Frobenius norm—widely used in optimization theory [83 ] (Lan 2020).
Our bounds use the dual-norm noise scale σ \sigma σ and regularity constant κ \kappa κ directly, without Euclidean conversion. Whether this improves the bound depends on both the geometry and the noise model (§ 3.6.1 ). Further discussion of Assumption 3 appears in Section D.
κ \kappa κ -Regularity and Norm-Dependent Constants
To obtain stochastic guarantees under an arbitrary update norm, we rely on the framework of κ \kappa κ -smooth and κ \kappa κ -regular normed spaces introduced by Juditsky and Nemirovski [ [40 ] . We briefly recall the key definitions.
Definition 1 (κ V \kappa_V κ V -Smooth and κ V \kappa_V κ V -Regular Spaces). Let ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) be a finite-dimensional normed space and define ϕ ( u ) : = 1 2 ∥ u ∥ V 2 \phi(u) := \frac{1}{2}\lVert u\rVert_V^2 ϕ ( u ) := 2 1 ∥ u ∥ V 2 . (i) We say that ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) is κ V \kappa_V κ V -smooth for some κ V ≥ 1 \kappa_V \ge1 κ V ≥ 1 if ϕ \phi ϕ is continuously differentiable and satisfies ϕ ( u + v ) ≤ ϕ ( u ) + ⟨ ∇ ϕ ( u ) , v ⟩ + κ V 2 ∥ v ∥ V 2 \phi(u+v) \le\phi(u) + \langle\nabla\phi(u),v\rangle+ \frac{\kappa_V}{2}\lVert v\rVert_V^2 ϕ ( u + v ) ≤ ϕ ( u ) + ⟨ ∇ ϕ ( u ) , v ⟩ + 2 κ V ∥ v ∥ V 2 for all $u,v \in V.
(ii) We say that ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) is κ V \kappa_V κ V -regular if there exists an auxiliary norm ∥ ⋅ ∥ V , + \|\cdot\|_{V,+} ∥ ⋅ ∥ V , + on V V V that is κ V , + \kappa_{V,+} κ V , + -smooth for some κ V , + ∈ [ 1 , κ V ] \kappa_{V,+}\in[1,\kappa_V] κ V , + ∈ [ 1 , κ V ] and satisfies the compatibility chain
∥ u ∥ V 2 ≤ ∥ u ∥ V , + 2 ≤ κ V κ V , + ∥ u ∥ V 2 , ∀ u ∈ V . (12) \|u\|_V^2 \le\|u\|_{V,+}^2 \le\frac{\kappa_V}{\kappa_{V,+}}\|u\|_V^2,\qquad\forall u\in V. \tag*{(12)} ∥ u ∥ V 2 ≤ ∥ u ∥ V , + 2 ≤ κ V , + κ V ∥ u ∥ V 2 , ∀ u ∈ V . (12)
The constant κ V ≥ 1 \kappa_V\ge1 κ V ≥ 1 for which ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) is κ V \kappa_V κ V -regular is called the constant of regularity of ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) .
Remark 1 (How we use Definition 1 in this paper). Definition 1 is stated for an arbitrary finite-dimensional normed space ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) . In the remainder of the paper, we will apply it primarily to the dual space V = E ∗ V=E^* V = E ∗ equipped with the induced dual norm ∥ ⋅ ∥ V = ∥ ⋅ ∥ ∗ \|\cdot\|_V=\|\cdot\|_* ∥ ⋅ ∥ V = ∥ ⋅ ∥ ∗ , since stochastic gradients and noise live in E ∗ E^* E ∗ and are measured in ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ . To simplify notation, we write κ : = κ E ∗ \kappa:=\kappa_{E^*} κ := κ E ∗ , κ + : = κ E ∗ , + \kappa_+:=\kappa_{E^*,+} κ + := κ E ∗ , + , and denote the auxiliary norm on E ∗ E^* E ∗ by ∥ ⋅ ∥ ∗ , + : = ∥ ⋅ ∥ E ∗ , + \|\cdot\|_{*,+}:=\|\cdot\|_{E^*,+} ∥ ⋅ ∥ ∗ , + := ∥ ⋅ ∥ E ∗ , + . With this convention, the compatibility chain (12) specializes to
∥ s ∥ ∗ 2 ≤ ∥ s ∥ ∗ , + 2 ≤ κ κ + ∥ s ∥ ∗ 2 , ∀ s ∈ E ∗ . (13) \|s\|_*^2 \le\|s\|_{*,+}^2 \le\frac{\kappa}{\kappa_+}\|s\|_*^2,\qquad\forall s\in E^*. \tag*{(13)} ∥ s ∥ ∗ 2 ≤ ∥ s ∥ ∗ , + 2 ≤ κ + κ ∥ s ∥ ∗ 2 , ∀ s ∈ E ∗ . (13)
The condition (13) does not restrict the choice of the update norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ (and hence of the dual norm ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ ). Indeed, for any nonzero finite-dimensional space E E E , its dual ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*,\|\cdot\|_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) is automatically κ \kappa κ -regular for some κ ≤ dim ( E ) \kappa\le\dim(E) κ ≤ dim ( E ) by Proposition 10, and this worst-case linear dependence on the dimension cannot be improved in general (Proposition 11). For ℓ p d \ell_p^d ℓ p d and Schatten-p p p norms, however, the regularity parameter can often be much smaller.9 Section L collects these bounds and their proofs, with a summary of the auxiliary norms in Table 3 .
Regularity vs. Euclidean Norm Equivalence
A common alternative is to compare the induced dual norm ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ with a Euclidean reference norm and then apply Euclidean concentration. Regularity offers a more flexible version of this approach: it allows a smooth auxiliary norm that is comparable to ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ and tailored to the target geometry. This freedom can substantially sharpen the geometric bounds. For dual ℓ p d \ell_p^d ℓ p d spaces with 2 ≤ p ≤ ∞ 2\le p\le\infty 2 ≤ p ≤ ∞ , tuning the auxiliary norm yields the optimal order κ = Θ ( min { p , ln ( e d ) } ) \kappa=\Theta(\min\{p,\ln(ed)\}) κ = Θ ( min { p , ln ( e d )}) , compared with κ = d 1 − 2 / p \kappa=d^{1-2/p} κ = d 1 − 2/ p for the Euclidean choice. At p = ∞ p=\infty p = ∞ , the gain is striking: linear dimension dependence becomes logarithmic. At p = 2 p=2 p = 2 , both choices give κ = 1 \kappa=1 κ = 1 ; for 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 , a scaled Euclidean proxy is already optimal.
These gains require no change to the algorithm: the auxiliary norm is used only in the analysis. For suitable noise models, they also translate into sharper oracle-complexity bounds (§ 3.6.1 ). Details and the Schatten-p p p counterpart appear in Section I.
Key Lemma: Minibatch Concentration in Regular Dual Spaces
The following concentration inequality is the technical core of the analysis: it controls the minibatch error directly in the induced dual norm ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ , with the geometry entering only through κ \kappa κ .
Lemma 1 (Concentration in Regular Normed Spaces). Let κ \kappa κ be a regularity constant of the dual space ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*,\|\cdot\|_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) . Assume that { η j } j = 1 B \{\eta^j\}_{j=1}^B { η j } j = 1 B is a sequence of i.i.d. random vectors in E ∗ E^* E ∗ such that E [ η j ] = 0 \mathbb{E}[\eta^j]=0 E [ η j ] = 0 and they satisfy the light-tail condition: E [ exp ( ∥ η j ∥ ∗ 2 σ 2 ) ] ≤ exp ( 1 ) \mathbb{E}\left[\exp\left(\frac{\|\eta^j\|_*^2}{\sigma^2}\right)\right]\le\exp(1) E [ exp ( σ 2 ∥ η j ∥ ∗ 2 ) ] ≤ exp ( 1 ) . Then for any λ ≥ 0 \lambda\ge0 λ ≥ 0 , the average η ˉ = 1 B ∑ j = 1 B η j \bar{\eta}=\frac{1}{B}\sum_{j=1}^B\eta^j η ˉ = B 1 ∑ j = 1 B η j satisfies the concentration inequality: Pr { ∥ η ˉ ∥ ∗ ≥ ( 2 κ + 2 λ ) σ B } ≤ exp ( − λ 2 / 3 ) \Pr\left\{\|\bar{\eta}\|_*\ge(\sqrt{2\kappa}+\sqrt{2\lambda})\frac{\sigma}{\sqrt{B}}\right\}\le\exp(-\lambda^2/3) Pr { ∥ η ˉ ∥ ∗ ≥ ( 2 κ + 2 λ ) B σ } ≤ exp ( − λ 2 /3 ) .
Proof idea. κ \kappa κ -regularity provides a smooth auxiliary norm on E ∗ E^* E ∗ , enabling a martingale concentration argument. The comparison (13) then yields concentration in ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ without a detour through Euclidean/Frobenius conversions, with geometric dependence captured by κ \kappa κ . Further explanation and the proof are given in Section M.1.
Main Theorem: High-Probability Oracle Complexity
We now combine the descent analysis under non-Euclidean smoothness with the minibatch concentration lemma.
Theorem 1 (Minibatch non-Euclidean SGD under light tails). Let κ \kappa κ be a regularity constant of the dual space ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*,\|\cdot\|_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) . Suppose Assumptions 1, 2, and 3 hold. Run the update (7) with the constant stepsize γ k ≡ 1 2 L \gamma_k\equiv\frac{1}{2L} γ k ≡ 2 L 1 for T = ⌈ 128 L Δ 15 ε ⌉ T=\left\lceil\frac{128L\Delta}{15\varepsilon}\right\rceil T = ⌈ 15 ε 128 L Δ ⌉ iterations, with a minibatch size chosen as
B = max { ⌈ 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 ε ⌉ , 1 } . B=\max\left\{\left\lceil\frac{16\sigma^2\left(\sqrt{2\kappa}+\sqrt{6\log(T/\delta)}\right)^2}{\varepsilon}\right\rceil,1\right\}. B = max ⎩ ⎨ ⎧ ε 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 , 1 ⎭ ⎬ ⎫ . Then, with probability at least 1 − δ 1-\delta 1 − δ , the iterates satisfy min 0 ≤ k ≤ T − 1 ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε \min_{0\le k\le T-1}\|\nabla f(x^k)\|_*^2\le\varepsilon min 0 ≤ k ≤ T − 1 ∥∇ f ( x k ) ∥ ∗ 2 ≤ ε .
Theorem 1 sets T T T using the descent argument and chooses B B B to control the gradient-estimation error simultaneously over all T T T iterations. Since each iteration uses B B B stochastic gradients, the following corollary combines these choices to bound the total oracle cost B T BT B T .
Corollary 1 (Norm-native oracle complexity under dual-norm light tails). Suppose the assumptions of Theorem 1 hold. Run the update (7) with the constant stepsize γ k ≡ 1 2 L \gamma_k\equiv\frac{1}{2L} γ k ≡ 2 L 1 , and choose T T T and B B B as in that theorem. Let N ∗ : = B T N_*:=BT N ∗ := B T denote the total number of stochastic-gradient evaluations. Then with probability at least 1 − δ 1-\delta 1 − δ , the method produces an iterate satisfying min 0 ≤ k ≤ T − 1 ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε \min_{0\le k\le T-1}\|\nabla f(x^k)\|_*^2\le\varepsilon min 0 ≤ k ≤ T − 1 ∥∇ f ( x k ) ∥ ∗ 2 ≤ ε after at most
N ∗ = O ~ ( L Δ ε + κ L Δ σ 2 ε 2 ) (14) N_*=\widetilde{\mathcal O}\left(\frac{L\Delta}{\varepsilon}+\frac{\kappa L\Delta\sigma^2}{\varepsilon^2}\right) \tag*{(14)} N ∗ = O ( ε L Δ + ε 2 κ L Δ σ 2 ) (14) stochastic-gradient evaluations, where O ~ ( ⋅ ) \widetilde{\mathcal O}(\cdot) O ( ⋅ ) hides logarithmic factors in T T T and 1 / δ 1/\delta 1/ δ .
The rate is optimal in ε \varepsilon ε and σ \sigma σ . Jin [ [36 ] establishes a lower bound of Ω ( L Δ / ε + L Δ σ 2 / ε 2 ) \Omega(L\Delta/\varepsilon+L\Delta\sigma^2/\varepsilon^2) Ω ( L Δ/ ε + L Δ σ 2 / ε 2 ) oracle calls for general unbiased first-order oracles with uniformly bounded noise, which satisfy our oracle model and light-tail condition (§ 3.2 ).12 In the Euclidean case, where ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 \|\cdot\|_*=\|\cdot\|_2 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 and κ = 1 \kappa=1 κ = 1 , this matches (14), so our rate is optimal up to logarithmic factors over the general unbiased light-tail oracle class. For a general update norm the two are not literally comparable, since σ \sigma σ is then measured in ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ rather than in ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 ; what (14) adds in that case is the single geometry factor κ \kappa κ .
On the size of the minibatch. Theorem 1 uses B = O ~ ( 1 + κ σ 2 / ε ) B=\widetilde{\mathcal O}(1+\kappa\sigma^2/\varepsilon) B = O ( 1 + κ σ 2 / ε ) , with B ≥ 1 B\ge1 B ≥ 1 . The Euclidean analyses of Tyurin and Richtárik [ [84 ] and Tyurin [ [83 ] have the same dependence on σ 2 / ε \sigma^2/\varepsilon σ 2 / ε , up to logarithmic factors.13 For exact gradients, we set B = 1 B=1 B = 1 and retain γ k = 1 2 L \gamma_k=\frac{1}{2L} γ k = 2 L 1 .14 Deterministic descent then gives N ∗ = O ( L Δ / ε ) N_*=\mathcal O(L\Delta/\varepsilon) N ∗ = O ( L Δ/ ε ) , explaining the noise-independent leading term in (14).
Algorithm 1 Rennala NSGD (server)
Input: initial point x 0 ∈ E x^0 \in E x 0 ∈ E , stepsizes { γ k > 0 } \{\gamma_k > 0\} { γ k > 0 } , minibatch size B B B , number of iterations T T T .
Start Algorithm 2 on all N N N workers.
for k = 0 , 1 , … , T − 1 k = 0, 1, \ldots, T - 1 k = 0 , 1 , … , T − 1 do
Set g k = 0 g^k = 0 g k = 0 and b = 0 b = 0 b = 0 .
while b < B b < B b < B do
Receive the next gradient message ( g , k ′ ) (g, k') ( g , k ′ ) from any worker.
if k ′ = k k' = k k ′ = k then
g k ← g k + 1 B g g^k \leftarrow g^k + \frac{1}{B}g g k ← g k + B 1 g and b ← b + 1 b \leftarrow b + 1 b ← b + 1 .
end if
if b < B b < B b < B then
Send ( x k , k ) (x^k, k) ( x k , k ) to that worker.
end if
end while
x k + 1 = x k − γ k ( g k ) # x^{k+1} = x^k - \gamma_k (g^k)^\# x k + 1 = x k − γ k ( g k ) # .
Send ( x k + 1 , k + 1 ) (x^{k+1}, k + 1) ( x k + 1 , k + 1 ) to the worker whose gradient completed the minibatch.
end for
Algorithm 2 Rennala NSGD (worker loop)
Set g = 0 g = 0 g = 0 and k ′ = − 1 k' = -1 k ′ = − 1 .
loop
Send ( g , k ′ ) (g, k') ( g , k ′ ) to the server.
Receive ( x k , k ) (x^k, k) ( x k , k ) from the server.
Set k ′ = k k' = k k ′ = k .
Sample ξ ∼ D \xi\sim\mathcal{D} ξ ∼ D .
Compute g = ∇ f ( x k ; ξ ) g = \nabla f(x^k; \xi) g = ∇ f ( x k ; ξ ) .
end loop
Light-Tail Assumption in the Native Norm Can Lead to Substantial Improvement
Measuring noise directly in the induced dual norm can substantially sharpen oracle bounds. Our native guarantee (14) uses the statistical coefficient C ∗ = κ σ 2 C_* = \kappa\sigma^2 C ∗ = κ σ 2 . For comparison, consider a Euclidean/Frobenius light-tail oracle with scale σ F \sigma_F σ F and α > 0 \alpha> 0 α > 0 satisfying α ∥ z ∥ ∗ ≤ ∥ z ∥ F \alpha\lVert z \rVert_* \le\lVert z \rVert_F α ∥ z ∥ ∗ ≤ ∥ z ∥ F . Concentrating the minibatch error in that norm and then converting it yields N F = O ~ ( L Δ ϵ + L Δ ϵ 2 α − 2 σ 2 ) N_F = \widetilde{\mathcal{O}}\left(\frac{L\Delta}{\epsilon} + \frac{L\Delta}{\epsilon^2}\alpha^{-2}\sigma^2\right) N F = O ( ϵ L Δ + ϵ 2 L Δ α − 2 σ 2 ) .
The two bounds have the same leading term, so we compare their statistical coefficients for two dense-noise models in dual ℓ p d \ell_p^d ℓ p d geometries. Both use ∇ f ( x ; ξ ) = ∇ f ( x ) + ν u \nabla f(x; \xi) = \nabla f(x) + \nu u ∇ f ( x ; ξ ) = ∇ f ( x ) + ν u , where ν > 0 \nu> 0 ν > 0 and each call draws a fresh independent u u u with independent Rademacher or standard Gaussian entries.15 Table 1 reports the resulting statistical terms.
Table 1. Statistical oracle-cost terms, omitting the factor ν 2 \nu^2 ν 2 and the common leading term O ~ ( L Δ / ε ) \widetilde{\mathcal O}(L\Delta/\varepsilon) O ( L Δ/ ε ) . Formulas hold for d ≥ 1 d\ge1 d ≥ 1 . Within each row, green and red mark smaller and larger asymptotic orders; blue marks agreement. Colours do not imply strict improvement in every finite dimension.
For these models at dual p = ∞ p = \infty p = ∞ , corresponding to the ℓ 1 \ell_1 ℓ 1 update norm, the native coefficient grows as ln ( e d ) \ln(ed) ln ( e d ) for Rademacher noise and ( ln ( e d ) ) 2 (\ln(ed))^2 ( ln ( e d ) ) 2 for Gaussian noise, while the converted coefficient grows as d d d . At p = 2 p = 2 p = 2 the bounds agree; at p = 1 p = 1 p = 1 , the native coefficient has the larger asymptotic order, d 3 d^3 d 3 instead of d 2 d^2 d 2 . These conclusions depend on the noise model: p p p alone does not determine which analysis gives a smaller bound. Full derivations, intermediate p p p regimes, and the diagonal Schatten counterparts appear in Section J.
Rennala NSGD: Semi-Asynchronous Parallelism Beyond Euclidean Norms
We present Rennala NSGD, a semi-asynchronous method combining the minibatch non-Euclidean update (7) with a modified collection protocol of Rennala SGD [84 ] .16 A central server coordinates N N N workers with heterogeneous computation speeds, using only gradients computed at the current iterate and discarding stale ones (Algorithm 1 and Algorithm 2).
Computation model. We use the fixed computation model of Tyurin and Richtárik [ [84 ] . All N N N workers draw fresh independent samples from the same distribution D \mathcal{D} D , and worker i i i takes a fixed time τ i \tau_i τ i for stochastic gradient, with 0 < τ 1 ≤ ⋯ ≤ τ N 0 < \tau_1 \le\cdots\le\tau_N 0 < τ 1 ≤ ⋯ ≤ τ N . Communication and server processing are instantaneous. The server processes messages one at a time, including any update and reply, and handles simultaneous returns in a fixed worker order.
Fresh gradients. Each gradient carries its query index k ′ k' k ′ . At iteration k k k , the server accepts only fresh gradients, with k ′ = k k'=k k ′ = k , and discards stale ones, with k ′ < k k'<k k ′ < k . Conditional on the history, accepted gradients are i.i.d. oracle responses at x k x^k x k , so the oracle model of § 3.2 and the serial convergence analysis apply.
Algorithm: Rennala NSGD. Algorithm 1 and Algorithm 2 give the pseudocode. At iteration k k k , the server collects the first B B B fresh gradients, possibly several from the same worker, averages them as in (6) , and applies (7) . After each message, the server replies with the current iterate and its index. If the message completes the minibatch, it first updates and then sends ( x k + 1 , k + 1 ) (x^{k+1}, k+1) ( x k + 1 , k + 1 ) to that worker; otherwise, it sends ( x k , k ) (x^k,k) ( x k , k ) . 17 Other workers finish their pending computations and receive the latest iterate when their results are processed.
Main Theorem: Wall-Clock Time Complexity
The following theorem converts the high-probability oracle guarantee of Theorem 1 into a wall-clock guarantee under heterogeneous worker speeds.
Theorem 2 (Wall-clock time complexity of Rennala NSGD ). Suppose the assumptions of Theorem 1 hold, and use the fixed computation model of § 4 . Run Rennala NSGD with the stepsize, iteration count T T T , and minibatch size B B B prescribed in that theorem. For 0 < ε ≤ L Δ 0<\varepsilon\le L\Delta 0 < ε ≤ L Δ and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , the iterates satisfy min 0 ≤ k < T ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε \min_{0\le k<T}\|\nabla f(x^k)\|_*^2\le\varepsilon min 0 ≤ k < T ∥∇ f ( x k ) ∥ ∗ 2 ≤ ε with probability at least 1 − δ 1-\delta 1 − δ within total wall-clock time
T t o t a l = O ~ ( min S ∈ [ N ] [ ( 1 S ∑ i = 1 S 1 τ i ) − 1 ( L Δ ε + κ L Δ σ 2 S ε 2 ) ] ) . (15) T_{\mathrm{total}}=\widetilde{\mathcal{O}}\left(\min_{S\in[N]}\left[\left(\frac{1}{S}\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1}\left(\frac{L\Delta}{\varepsilon}+\frac{\kappa L\Delta\sigma^2}{S\varepsilon^2}\right)\right]\right). \tag*{(15)} T total = O S ∈ [ N ] min ( S 1 i = 1 ∑ S τ i 1 ) − 1 ( ε L Δ + S ε 2 κ L Δ σ 2 ) . (15)
The minimum in (15) balances the harmonic-mean time of the S S S fastest workers against a statistical term reduced by S S S . Neither the worker times nor the minimizing S S S are needed to run the algorithm. The update norm affects L L L , σ \sigma σ , κ \kappa κ , and the stationarity criterion, so bounds for different update norms certify different notions of accuracy.
For a fixed update norm, accuracy, and worker times, the native-versus-converted comparison depends on κ σ 2 \kappa\sigma^2 κ σ 2 and α − 2 σ τ 2 \alpha^{-2}\sigma_\tau^2 α − 2 σ τ 2 . The smaller coefficient gives a wall-clock bound that is no larger and can be significantly smaller.
In the Euclidean case, κ = 1 \kappa=1 κ = 1 recovers the parameter dependence of classical Rennala SGD up to logarithmic factors [84 ] . Our guarantee holds with high probability under light tails rather than in expectation under bounded variance; this agreement does not establish optimality in our oracle model or for general update norms. See Section M.3 for the proof and Section M.4 for the full discussion.
Conclusion
We analyzed minibatch SGD in finite-dimensional normed spaces under dual-norm light-tailed noise. Concentration in regular dual spaces yields high-probability oracle-complexity bounds with geometric dependence captured by κ \kappa κ (Lemma 1 and Corollary 1). We established matching upper and lower regularity bounds for ℓ p \ell_p ℓ p and Schatten-p p p spaces (Table 3 ) and compared native and Euclidean/Frobenius noise analyses. For example, dense Rademacher noise with the ℓ 1 \ell_1 ℓ 1 update norm gives logarithmic rather than linear dimension dependence in the statistical coefficient (Proposition 5).
Building on these results, Rennala NSGD combines non-Euclidean updates with fresh-minibatch collection, yielding a high-probability wall-clock bound under fixed, heterogeneous worker computation times (Theorem 2). In the Euclidean case, our oracle and time bounds match the parameter dependence of standard minibatch SGD and classical Rennala SGD, respectively, up to logarithmic factors. Together, these results connect update geometry and stochastic noise to parallel optimization time. Section B discusses the limitations of our assumptions and computation model.
Acknowledgments and Disclosure of Funding
The research reported in this publication was supported by funding from King Abdullah University of Science and Technology (KAUST): i) KAUST Baseline Research Scheme, ii) CRG Grant ORFS-CRG12-2024-6460, and iii) Center of Excellence for Generative AI, under award number 5940.
The authors declare no competing interests.
Notation
We collect here the notation used throughout the paper; the objects introduced are developed further in § 3 .
Spaces and norms. Our analysis takes place in a finite-dimensional normed space ( E , ∥ ⋅ ∥ ) (E,\|\cdot\|) ( E , ∥ ⋅ ∥ ) with dual space ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*,\|\cdot\|_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) . We use the canonical dual pairing ⟨ y , x ⟩ : = y ( x ) \langle y,x\rangle:= y(x) ⟨ y , x ⟩ := y ( x ) for y ∈ E ∗ y \in E^* y ∈ E ∗ and x ∈ E x \in E x ∈ E , and define the dual norm by ∥ y ∥ ∗ : = sup ∥ x ∥ ≤ 1 ⟨ y , x ⟩ \|y\|_* := \sup_{\|x\|\le1}\langle y,x\rangle ∥ y ∥ ∗ := sup ∥ x ∥ ≤ 1 ⟨ y , x ⟩ , y ∈ E ∗ y \in E^* y ∈ E ∗ . We refer to ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ as the update norm : it determines ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ and, for a given gradient and positive stepsize, the set of admissible steepest-descent steps. The fixed sharp selection described below specifies a concrete step from this set. Gradients and stochastic gradients are elements of E ∗ E^* E ∗ . Our analysis uses the primal norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ for parameter displacements and its induced dual norm ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ for gradient smoothness, oracle noise, and stationarity. We also use auxiliary norms in the proofs and Euclidean or Frobenius norms when comparing oracle models. We write κ \kappa κ for a regularity constant of ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*,\|\cdot\|_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) , and ∥ ⋅ ∥ ∗ , + \|\cdot\|_{*,+} ∥ ⋅ ∥ ∗ , + and κ + \kappa_+ κ + for a corresponding auxiliary norm and its smoothness constant (Definition 1).
The sharp operator. Let ϕ ( x ) : = 1 2 ∥ x ∥ 2 \phi(x) := \frac{1}{2}\|x\|^2 ϕ ( x ) := 2 1 ∥ x ∥ 2 for x ∈ E x \in E x ∈ E , with Fenchel conjugate ϕ ∗ ( s ) : = 1 2 ∥ s ∥ ∗ 2 \phi^*(s) := \frac{1}{2}\|s\|_*^2 ϕ ∗ ( s ) := 2 1 ∥ s ∥ ∗ 2 for s ∈ E ∗ s \in E^* s ∈ E ∗ . The subdifferential ∂ ϕ ∗ ( s ) ⊆ E \partial\phi^*(s) \subseteq E ∂ ϕ ∗ ( s ) ⊆ E may contain more than one element. A sharp selection assigns to each s ∈ E ∗ s \in E^* s ∈ E ∗ an element s ♯ ∈ ∂ ϕ ∗ ( s ) s^\sharp\in\partial\phi^*(s) s ♯ ∈ ∂ ϕ ∗ ( s ) . For the stochastic algorithm, we fix a Borel-measurable selection s ↦ s ♯ s \mapsto s^\sharp s ↦ s ♯ as in § 3.1 and call this map the sharp operator . Every such selected vector satisfies ∥ s ♯ ∥ 2 = ∥ s ∥ ∗ 2 \|s^\sharp\|^2 = \|s\|_*^2 ∥ s ♯ ∥ 2 = ∥ s ∥ ∗ 2 and ⟨ s , s ♯ ⟩ = ∥ s ∥ ∗ 2 \langle s,s^\sharp\rangle= \|s\|_*^2 ⟨ s , s ♯ ⟩ = ∥ s ∥ ∗ 2 . In the Euclidean case s ♯ = s s^\sharp= s s ♯ = s . In the quadratic case, ∥ x ∥ 2 = ⟨ A x , x ⟩ \|x\|^2 = \langle Ax,x\rangle ∥ x ∥ 2 = ⟨ A x , x ⟩ for a symmetric positive-definite linear map A : E → E ∗ A:E\to E^* A : E → E ∗ , so ∇ ϕ ( x ) = A x \nabla\phi(x)=Ax ∇ ϕ ( x ) = A x , ∇ 2 ϕ = A \nabla^2\phi=A ∇ 2 ϕ = A , and s ♯ = ∇ ϕ ∗ ( s ) = A − 1 s s^\sharp=\nabla\phi^*(s)=A^{-1}s s ♯ = ∇ ϕ ∗ ( s ) = A − 1 s . Here the sharp operator is the fixed linear map A − 1 A^{-1} A − 1 . For a general norm, the squared-norm functions ϕ \phi ϕ and ϕ ∗ \phi^* ϕ ∗ need not be differentiable everywhere, and the sharp operator need not admit such a linear representation. The subdifferential formulation remains valid without differentiability assumptions: ∂ ϕ ∗ \partial\phi^* ∂ ϕ ∗ is positively homogeneous and may contain multiple vectors at a given point. Basic properties of the sharp operator are collected in Lemma 12 , and § 3.1 derives the resulting update together with its steepest-descent and LMO forms.
Vector geometries. For vector-valued parameters we write ℓ p d \ell_p^d ℓ p d for R d \mathbb{R}^d R d equipped with the ℓ p \ell_p ℓ p norm, ∥ x ∥ p : = ( ∑ i = 1 d ∣ x i ∣ p ) 1 / p \|x\|_p := \left(\sum_{i=1}^d |x_i|^p\right)^{1/p} ∥ x ∥ p := ( ∑ i = 1 d ∣ x i ∣ p ) 1/ p for 1 ≤ p < ∞ 1 \le p < \infty 1 ≤ p < ∞ and ∥ x ∥ ∞ : = max 1 ≤ i ≤ d ∣ x i ∣ \|x\|_\infty:= \max_{1\le i\le d}|x_i| ∥ x ∥ ∞ := max 1 ≤ i ≤ d ∣ x i ∣ , with ℓ 2 \ell_2 ℓ 2 the Euclidean norm. The ℓ p \ell_p ℓ p and ℓ q \ell_q ℓ q norms are dual to one another whenever 1 / p + 1 / q = 1 1/p+1/q=1 1/ p + 1/ q = 1 , so an ℓ p \ell_p ℓ p update norm induces an ℓ q \ell_q ℓ q dual norm; in particular ℓ 1 \ell_1 ℓ 1 and ℓ ∞ \ell_\infty ℓ ∞ form a dual pair, which is the source of the coordinate-type and sign-type updates in § 3.1 .
Matrix geometries. When the parameter space contains matrix-valued blocks, as in modern neural networks, non-Euclidean update norms are naturally expressed through Schatten norms. For X ∈ R m × n X\in\mathbb{R}^{m\times n} X ∈ R m × n with m , n ≥ 1 m,n\ge1 m , n ≥ 1 , set r : = min { m , n } r:=\min\{m,n\} r := min { m , n } and write σ ( X ) : = ( σ 1 ( X ) , … , σ r ( X ) ) ∈ R r \sigma(X):=(\sigma_1(X),\ldots,\sigma_r(X))\in\mathbb{R}^r σ ( X ) := ( σ 1 ( X ) , … , σ r ( X )) ∈ R r for the full vector of singular values, ordered as σ 1 ( X ) ≥ ⋯ ≥ σ r ( X ) ≥ 0 \sigma_1(X)\ge\cdots\ge\sigma_r(X)\ge0 σ 1 ( X ) ≥ ⋯ ≥ σ r ( X ) ≥ 0 . Zero singular values are included, so σ ( 0 ) = 0 ∈ R r \sigma(0)=0\in\mathbb{R}^r σ ( 0 ) = 0 ∈ R r . For 1 ≤ p < ∞ 1\le p<\infty 1 ≤ p < ∞ , the Schatten-p p p norm is ∥ X ∥ S p : = ∥ σ ( X ) ∥ p = ( ∑ i = 1 r σ i ( X ) p ) 1 / p \|X\|_{S_p}:=\|\sigma(X)\|_p=\left(\sum_{i=1}^r\sigma_i(X)^p\right)^{1/p} ∥ X ∥ S p := ∥ σ ( X ) ∥ p = ( ∑ i = 1 r σ i ( X ) p ) 1/ p ; for p = ∞ p=\infty p = ∞ , we define ∥ X ∥ S ∞ : = ∥ σ ( X ) ∥ ∞ = σ 1 ( X ) \|X\|_{S_\infty}:=\|\sigma(X)\|_\infty=\sigma_1(X) ∥ X ∥ S ∞ := ∥ σ ( X ) ∥ ∞ = σ 1 ( X ) . Its special cases are the nuclear norm ∥ X ∥ n u c = ∥ X ∥ S 1 \|X\|_{\mathrm{nuc}}=\|X\|_{S_1} ∥ X ∥ nuc = ∥ X ∥ S 1 , the Frobenius norm ∥ X ∥ F = ∥ X ∥ S 2 \|X\|_{\mathrm{F}}=\|X\|_{S_2} ∥ X ∥ F = ∥ X ∥ S 2 , and the spectral norm ∥ X ∥ o p = ∥ X ∥ S ∞ \|X\|_{\mathrm{op}}=\|X\|_{S_\infty} ∥ X ∥ op = ∥ X ∥ S ∞ . The spectral and nuclear norms are dual to one another. The dimension-dependent bounds use r = min { m , n } r=\min\{m,n\} r = min { m , n } , whereas the actual rank rank ( X ) \operatorname{rank}(X) rank ( X ) is the number of positive singular values. In compact-SVD arguments, r r r is explicitly redefined locally as the actual matrix rank.
Accuracy and complexity. We write Δ : = f ( x 0 ) − f inf \Delta:= f(x^0)-f^{\inf} Δ := f ( x 0 ) − f i n f for the initial functional gap. A point x ∈ E x\in E x ∈ E is ε \varepsilon ε -stationary if ∥ ∇ f ( x ) ∥ ∗ 2 ≤ ε \|\nabla f(x)\|_*^2\le\varepsilon ∥∇ f ( x ) ∥ ∗ 2 ≤ ε ; stationarity is thus measured in the dual norm induced by the chosen update norm. Our light-tail guarantees are of high-probability type: for a failure probability δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) they assert that, with probability at least 1 − δ 1-\delta 1 − δ , at least one iterate among x 0 , … , x T − 1 x^0,\ldots,x^{T-1} x 0 , … , x T − 1 is ε \varepsilon ε -stationary. By iteration complexity we mean the number T T T of algorithmic iterations required, by oracle complexity the number of stochastic gradient evaluations, and by time complexity the wall-clock time. For serial minibatch SGD, the oracle count is N ∗ = B T N_* = BT N ∗ = B T , with B B B evaluations per iteration; N N N denotes the number of workers in the parallel setting. In wall-clock bounds, S ∈ [ N ] S\in[N] S ∈ [ N ] denotes the number of fastest workers considered in the minimization. We write [ N ] : = { 1 , … , N } [N]:=\{1,\ldots,N\} [ N ] := { 1 , … , N } , and O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) for O ( ⋅ ) O(\cdot) O ( ⋅ ) up to factors logarithmic in T T T and 1 / δ 1/\delta 1/ δ .
Throughout the convergence and complexity analysis, we take a positive smoothness bound L > 0 L>0 L > 0 , an accuracy ε > 0 \varepsilon>0 ε > 0 , and consider the nontrivial case Δ > 0 \Delta>0 Δ > 0 . If Δ = 0 \Delta=0 Δ = 0 , then x 0 x^0 x 0 is a global minimizer and hence ∇ f ( x o ) = 0 \nabla f(x^o) = 0 ∇ f ( x o ) = 0 , so it can be returned without any iterations. The simplified oracle and wall-clock complexity bounds are stated in the accuracy regime 0 < ε ≤ L Δ 0 < \varepsilon\le L\Delta 0 < ε ≤ L Δ .
Notation reference. The following table collects the notation used in the main text and the principal constructions in the appendices. The last column points to the definition or the relevant discussion. Symbols reused locally are listed with their scope; temporary variables in individual calculations retain their local definitions.
Table 2. Notation reference. Definitions with a stated local scope do not replace the conventions used elsewhere.
Notation reference (continued)
Table 4.
Table 5.
Limitations
Noise assumptions. Our high-probability guarantees rely on a dual-norm light-tail condition, which is stronger than bounded variance. Under bounded variance, the counterpart discussed in Section D gives an expectation guarantee, while the high-probability analysis requires stronger control of the noise.
Dependence on geometry and noise. The benefits of the native-norm formulation also depend on the problem: the regularity parameter κ \kappa κ can be as large as dim ( E ) \dim(E) dim ( E ) , and whether our bounds improve on Euclidean/Frobenius conversions depends jointly on the geometry and the noise model.
Statistical homogeneity. In the parallel setting, statistical homogeneity allows gradients from different workers to form an unbiased minibatch at the current iterate. When local distributions differ, accepting the first available fresh gradients can instead favor faster workers and bias the estimate of the global gradient. Extending our guarantees to this setting therefore requires revisiting minibatch collection and its time analysis.
Computation and communication model. The current analysis also assumes fixed worker computation times and instantaneous communication and server processing, leaving arbitrary delays and communication costs outside its scope.
Cost of the sharp operator. This paper does not account for the cost of computing the update direction. The computation model charges the workers only for stochastic gradients and treats the work of the server, including the evaluation of the sharp operator, as free. For a non-Euclidean update norm this cost need not be negligible: for the spectral norm, the sharp operator requires an orthogonalization of G k G^k G k (§ 3.1 ). Accordingly, (15) should be read as the time spent on computing stochastic gradients. Our guarantees assume an exact sharp selection, so numerical approximations require additional error control. Reducing the cost of the sharp operator is an important problem in its own right, which lies outside the scope of this work; existing approaches are reviewed in Section C.4.
Experimental validation. The present contribution is theoretical; assessing its practical benefits and computational tradeoffs through experiments remains future work.
Extended Related Work
Non-Euclidean Optimization
Kovalev and Borodich [ [50 ] develop a unified analysis of non-Euclidean SGD with momentum and weight decay under structured smoothness and gradient-noise assumptions, including variants with extrapolation and momentum variance reduction. Their Assumption 2 bounds the noise second moment in a weighted Euclidean norm, whereas our light-tail assumption is formulated directly in the dual norm induced by the update geometry. In the convex setting, they compare the leading complexity terms for signSGD and Muon with those of normalized SGD. Their comparisons identify favorable regimes involving dense solutions together with strongly nonuniform coordinatewise smoothness and noise scales, or solutions of high effective rank together with approximately low-rank matrices controlling curvature and gradient noise. Although they provide explicit comparisons in terms of structural parameters, they do not construct specific objective–oracle pairs realizing the favorable regimes. Moreover, the conditions yielding an improvement may be restrictive when the smoothness and noise parameters have many components or singular values of comparable magnitude.
Concurrent work by Xie et al. [ [88 ] studies non-Euclidean noise through an adaptive variance model, whereas we use a light-tail condition in the dual norm induced by the update geometry. They prove that Normalized Steepest Descent (NSD) with momentum admits a dimension-free convergence rate for nonconvex optimization, in the sense that it avoids the norm-distortion factor that arises when controlling noise in a Euclidean reference norm and then converting to the target dual norm .
Time Complexity in Heterogeneous Parallel SGD
In the asynchronous update described in § 2 , the gradient applied at iteration k k k is evaluated at the earlier iterate x k − δ k x^{k-\delta_k} x k − δ k . It therefore need not be an unbiased estimator of ∇ f ( x k ) \nabla f(x^k) ∇ f ( x k ) ; the analysis must account for this bias, as in frameworks for biased gradient estimators [15 ] .
The harmonic-mean wall-clock bounds cited in § 2 rely on suitable stepsize or filtering rules. The results cover ASGD with delay-adaptive stepsizes and filtering variants such as Picky SGD, which skips updates when the query point is too far from the current iterate [57 , 47 , 13 ] .
Sadiev et al. [ [68 ] propose Ringmaster LMO, an asynchronous LMO-based momentum method for unconstrained nonconvex stochastic optimization, and translate its iteration complexity into wall-clock time under heterogeneous worker speeds, recovering the optimal time-complexity scaling of Tyurin and Richtárik [ [84 ] in the standard smooth Euclidean case. This is the method closest to ours, and the two are best read as addressing different questions. The key difference lies in the noise model , which is the subject of the present paper: they assume bounded variance in the Euclidean norm and transfer it to the update geometry through the norm-equivalence constant ρ : = sup z ≠ 0 ∥ z ∥ ∗ / ∥ z ∥ 2 \rho:= \sup_{z \ne0} \lVert z\rVert_* / \lVert z\rVert_2 ρ := sup z = 0 ∥ z ∥ ∗ / ∥ z ∥ 2 , so that the product ρ σ \rho\sigma ρ σ enters their complexity bounds—precisely the conversion that Assumption 3 is designed to avoid by measuring noise directly in the induced dual norm ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ .
Federated Learning: Challenges and Algorithmic Remedies
Problem (1) is the standard formulation of federated learning (FL), in which a server coordinates a large number of clients that keep their data local [48 , 56 , 42 ] .
This setting presents several distinct challenges. Statistical heterogeneity means that the local distributions D i \mathcal{D}_i D i differ across clients, while device or system heterogeneity refers to differences in compute speed, availability, and reliability. With partial participation , only a sampled cohort is active in each round. Privacy motivates keeping data local, while the communication bottleneck arises because transmitting model updates is often much more expensive than computing them [72 ] .
The same nested averaging structure also appears in centralized settings: the objective averages over groups, and each group averages over its own samples. Examples include pooled data silos, out-of-core learning, and deliberate stratification. There, the aim is to exploit this structure to reduce memory use and oracle calls, with complexity depending on similarity both across groups and within each group [75 ] .
The challenge closest to the agenda of the present paper is how computation is organized in a server-based data-parallel setting, where workers repeatedly exchange gradients or model updates with a coordinating server. Two widely studied algorithmic levers aim to reduce the communication cost of these exchanges.
Communication compression. The first lever is to reduce the size of each transmitted message by quantizing, sparsifying, or otherwise compressing the update while controlling the resulting error. Unbiased compressors [3 , 33 ] (Wangni et al. 2018) simplify the analysis: conditional on the vectors being compressed, independent compression randomness across workers ensures that the Euclidean second moment of the average compression error equals 1 / N 1/N 1/ N times the average of the individual error second moments.
Biased compression requires additional care. Naive compression of local gradients can lead to exponential divergence in heterogeneous distributed settings, as shown by the Top-1 counterexample of Beznosikov et al. [ [11 ] . Error feedback is a standard way to obtain convergence guarantees with biased compressors [69 , 77 , 4 , 43 , 76 ] . For smooth nonconvex objectives, EF21 matches the O ( 1 / T ) \mathcal{O}(1/T) O ( 1/ T ) iteration dependence of gradient descent, with compression-dependent constants and without bounded-gradient assumptions [63 Theorem 2] . Subsequent developments include practical variants [21 ] and normalized methods that combine momentum with time-varying stepsizes requiring no tuning from problem constants [67 ] . The framework of three-point compressors connects error feedback with lazy aggregation [64 ] .
Other work studies which messages to compress and how to exploit their structure. This includes server-to-worker compression [29 , 73 ] and low-rank or structured compression of transmitted gradients [85 ] (Wang et al. 2018). Compression can also be combined with parameter-efficient fine-tuning, where the update itself is restricted to low rank [34 ] , with quantization or error feedback applied on top [74 ] .
Compression also interacts with without-replacement sampling. Its variance can mask the benefits of random reshuffling in Q-RR, whereas DIANA-RR improves the accuracy dependence in the strongly convex setting by compressing differences between gradients and learned shifts [65 , 66 ] . Lower bounds characterize the limits of communication compression [35 ] .
Local steps. The second lever is to communicate less often, letting each client take several local update steps between synchronizations, as in FedAvg [56 ] . The obstacle is client drift: local progress is made towards local optima, and the effects of the resulting bias must be controlled [44 , 90 , 46 ] . Subsequent work combines local steps with probabilistic synchronization [55 ] , with random reshuffling [16 ] , and with compression itself [65 ] . A complementary question is how much can be extracted from a single cohort: instead of the customary single communication round per sampled cohort, one may run several proximal-point rounds inside a cohort and thereby cut the total communication cost substantially [89 ] .
Our work addresses a complementary question: how computational heterogeneity interacts with the geometry of the update. We study the wall-clock time of a fresh-minibatch protocol when workers differ in speed and updates follow steepest-descent directions in a non-Euclidean norm. Our setting is statistically homogeneous (§ 1 ), which removes heterogeneity-induced client drift, although local stochastic trajectories can still differ.
Our time model counts stochastic-gradient computation and treats communication in both directions and server-side processing, including evaluation of the sharp update, as instantaneous. We therefore do not analyze communication compression or local-step methods here. Combining these approaches with our treatment of worker speeds and update geometry is a natural direction for future work.
Computational Cost of the Sharp Operator
The two approaches discussed above reduce communication costs by changing the size or frequency of messages exchanged between workers and the server. A separate computational cost arises from evaluating the non-Euclidean update direction itself. As shown in § 3.1 , for the spectral update norm the canonical sharp selection is G ♯ = ∥ G ∥ n u c orth ( G ) G^{\sharp} = \lVert G\rVert_{\mathrm{nuc}} \operatorname{orth}(G) G ♯ = ∥ G ∥ nuc orth ( G ) , where orth ( G ) \operatorname{orth}(G) orth ( G ) is the polar factor. A direct implementation uses a compact rank-r r r singular value decomposition (SVD), where r = rank ( G ) r = \operatorname{rank}(G) r = rank ( G ) , retaining only the nonzero singular values and their corresponding singular vectors. The complete orthogonal factors of a full SVD are unnecessary. This SVD-based implementation can be computationally expensive at the scale of modern models, motivating alternative algorithms for computing the polar factor. The analysis in this paper assumes exact access to the chosen sharp selection; extending its guarantees to numerical approximations requires separate control of the approximation error.
Several lines of work aim to reduce this cost. The original Muon algorithm replaces the SVD by the computationally cheaper Newton–Schulz iteration. Subsequent work has proposed efficient polar-decomposition subroutines [52 , 5 ] , acceleration of the Newton–Schulz iteration with Chebyshev-type polynomials [28 ] , and SVD-based orthogonalization of momentum within an adaptively updated low-dimensional subspace [62 ] . A different approach, Dion, replaces the full-rank step by a rank-r r r distributed orthonormalization built on power iteration and randomized Cholesky QR [1 ] .
Additional Discussion of Assumption 3
This section gives examples satisfying Assumption 3, explains the choice of norm and tail condition, and compares the resulting guarantees with those under bounded variance.
Examples Satisfying the Assumption
Additive perturbations. Assumption 3 is satisfied, for instance, by the classical perturbed oracle ∇ f ( x ; ξ ) = ∇ f ( x ) + Z \nabla f(x;\xi)=\nabla f(x)+Z ∇ f ( x ; ξ ) = ∇ f ( x ) + Z with Z Z Z zero-mean and sub-Gaussian in the induced dual norm, that is, E [ exp ( ∥ Z ∥ ∗ 2 σ 2 ) ] ≤ exp ( 1 ) \mathbb{E}\left[\exp\left(\frac{\|Z\|_{*}^{2}}{\sigma^{2}}\right)\right]\leq\exp(1) E [ exp ( σ 2 ∥ Z ∥ ∗ 2 ) ] ≤ exp ( 1 ) . More generally, our framework applies to the additively perturbed and inexact-gradient models studied in the literature [2 , 23 , 78 , 17 ] only when their errors have zero mean and satisfy the dual-norm light-tail condition (11) with a common σ > 0 \sigma>0 σ > 0 for all x ∈ E x\in E x ∈ E .
Bounded errors. An unbiased oracle with almost surely bounded error, ∥ ∇ f ( x ; ξ ) − ∇ f ( x ) ∥ ∗ ≤ G \|\nabla f(x;\xi)-\nabla f(x)\|_{*}\leq G ∥∇ f ( x ; ξ ) − ∇ f ( x ) ∥ ∗ ≤ G for every x ∈ E x\in E x ∈ E with a common G ≥ 0 G\geq0 G ≥ 0 , also satisfies Assumption 3. If G > 0 G>0 G > 0 , this pointwise bound is sufficient for the light-tail condition with σ = G \sigma=G σ = G , since E ξ [ exp ( ∥ ∇ f ( x ; ξ ) − ∇ f ( x ) ∥ ∗ 2 G 2 ) ] ≤ exp ( 1 ) \mathbb{E}_{\xi}\left[\exp\left(\frac{\|\nabla f(x;\xi)-\nabla f(x)\|_{*}^{2}}{G^{2}}\right)\right]\leq\exp(1) E ξ [ exp ( G 2 ∥∇ f ( x ; ξ ) − ∇ f ( x ) ∥ ∗ 2 ) ] ≤ exp ( 1 ) . If G = 0 G=0 G = 0 , then for each fixed x ∈ E x\in E x ∈ E the oracle error vanishes almost surely. Hence the expectation in (11) equals 1 for any σ > 0 \sigma>0 σ > 0 .
Noise in the Induced Dual Norm
Stationarity is measured in ∥ ⋅ ∥ ∗ \|\cdot\|_{*} ∥ ⋅ ∥ ∗ , and the update direction ( g k ) ♯ (g^{k})^{\sharp} ( g k ) ♯ is produced from g k g^{k} g k through the dual norm, so ∥ ⋅ ∥ ∗ \|\cdot\|_{*} ∥ ⋅ ∥ ∗ is the norm in which the estimation error actually enters the descent inequality. A reference Euclidean/Frobenius light-tail assumption also suffices: first concentrate the minibatch error in that norm, then use ∥ z ∥ ∗ ≤ α − 1 ∥ z ∥ F \|z\|_{*}\leq\alpha^{-1}\|z\|_{F} ∥ z ∥ ∗ ≤ α − 1 ∥ z ∥ F to transfer the error bound. This yields the statistical coefficient C F = α − 2 σ F 2 C_{F}=\alpha^{-2}\sigma_{F}^{2} C F = α − 2 σ F 2 in (63). The native analysis instead uses C ∗ = κ σ 2 C_{*}=\kappa\sigma^{2} C ∗ = κ σ 2 ; which coefficient is smaller depends on the geometry and the noise model (Section J).
Exponential Moments and High-Probability Guarantees
For a prescribed finite horizon T ≥ 1 T\geq1 T ≥ 1 , our analysis requires ∥ g k − ∇ f ( x k ) ∥ ∗ ≤ ε / 4 \|g^{k}-\nabla f(x^{k})\|_{*}\leq\sqrt{\varepsilon}/4 ∥ g k − ∇ f ( x k ) ∥ ∗ ≤ ε /4 to hold simultaneously for 0 ≤ k < T 0\leq k<T 0 ≤ k < T with probability at least 1 − δ 1-\delta 1 − δ . We apply the concentration bound conditionally at each iterate with failure probability at most δ / T \delta/T δ / T , then take a union bound over the T T T iterations. The resulting minibatch size contains log ( T / δ ) = log T + log ( 1 / δ ) \log(T/\delta)=\log T+\log(1/\delta) log ( T / δ ) = log T + log ( 1/ δ ) : the union bound contributes the horizon dependence log T \log T log T , while the concentration tail supplies the failure-probability dependence log ( 1 / δ ) \log(1/\delta) log ( 1/ δ ) .
Comparison with Bounded-Variance Oracles
For comparison, replace the exponential-moment condition by the dual-norm variance bound E ξ [ ∥ ∇ f ( x ; ξ ) − ∇ f ( x ) ∥ ∗ 2 ] ≤ σ 2 \mathbb{E}_{\xi}\left[\|\nabla f(x;\xi)-\nabla f(x)\|_{*}^{2}\right]\leq\sigma^{2} E ξ [ ∥∇ f ( x ; ξ ) − ∇ f ( x ) ∥ ∗ 2 ] ≤ σ 2 for all x ∈ E x\in E x ∈ E . Retain unbiasedness, fresh i.i.d. samples independent of the past at each iteration, and the smoothness and lower-bound assumptions Assumption 1 and Assumption 2. As above, κ \kappa κ denotes a regularity constant of the dual space ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^{*},\|\cdot\|_{*}) ( E ∗ , ∥ ⋅ ∥ ∗ ) . At any fixed query point, Lemma 15 gives E [ ∥ 1 B ∑ j = 1 B η j ∥ ∗ 2 ] ≤ κ σ 2 / B \mathbb{E}\left[\left\|\frac{1}{B}\sum_{j=1}^{B}\eta_{j}\right\|_{*}^{2}\right]\leq\kappa\sigma^{2}/B E [ B 1 ∑ j = 1 B η j ∗ 2 ] ≤ κ σ 2 / B for the centered minibatch noise vectors η j \eta_{j} η j ; fresh sampling gives the same bound conditionally on the history at each iterate.
With γ k ≡ 1 2 L \gamma_{k}\equiv\frac{1}{2L} γ k ≡ 2 L 1 and suitable choices of T T T and B B B , the corresponding analysis guarantees E [ ∥ ∇ f ( x ^ ) ∥ ∗ 2 ] ≤ ε \mathbb{E}\left[\|\nabla f(\hat{x})\|_{*}^{2}\right]\leq\varepsilon E [ ∥∇ f ( x ^ ) ∥ ∗ 2 ] ≤ ε , where x ^ = x R \hat{x}=x^{R} x ^ = x R and R R R is uniform on { 0 , … , T − 1 } \{0,\ldots,T-1\} { 0 , … , T − 1 } independently of the run. For L > 0 L>0 L > 0 , Δ > 0 \Delta>0 Δ > 0 , and 0 < ε ≤ L Δ 0<\varepsilon\leq L\Delta 0 < ε ≤ L Δ , the resulting oracle complexity is N ∗ = B T = O ( L Δ / ε + κ L Δ σ 2 / ε 2 ) N_{*}=BT=\mathcal{O}(L\Delta/\varepsilon+\kappa L\Delta\sigma^{2}/\varepsilon^{2}) N ∗ = B T = O ( L Δ/ ε + κ L Δ σ 2 / ε 2 ) . This gives an expectation counterpart to our light-tail guarantees; the present paper focuses on high-probability bounds.
Useful Auxiliary Results
This section collects definitions and auxiliary results used throughout the proofs. We begin with convex analysis and properties of primal and dual norms, then describe squared norms and their conjugates. We next give a generalized Young inequality and compare Schatten norms. Finally, we introduce the probabilistic definitions needed for vector-valued martingales and prove a second-moment bound for martingale sums in κ \kappa κ -regular spaces.
Let E E E be a finite-dimensional vector space, and let E ∗ E^{*} E ∗ be its dual. For s ∈ E ∗ s \in E^{*} s ∈ E ∗ , we write ⟨ s , x ⟩ \langle s,x\rangle ⟨ s , x ⟩ to denote the action of s s s on x x x .
Foundations of Convex Analysis
To analyze non-smooth optimization objectives and generalized updates, we require the standard definition of the subgradient.
Definition 2 (Subgradient). Let f : E → R ∪ { + ∞ } f:E \to\mathbb{R} \cup\{+\infty\} f : E → R ∪ { + ∞ } be proper. A vector s ∈ E ∗ s \in E^{*} s ∈ E ∗ is a subgradient of f f f at a point x ∈ E x \in E x ∈ E if
f ( y ) ≥ f ( x ) + ⟨ s , y − x ⟩ ∀ y ∈ E . (16) f(y) \ge f(x) + \langle s,y-x\rangle\quad\forall y \in E. \tag*{(16)} f ( y ) ≥ f ( x ) + ⟨ s , y − x ⟩ ∀ y ∈ E . (16)
The collection of all subgradients forms the subdifferential.
Definition 3 (Subdifferential). Let f : E → R ∪ { + ∞ } f:E \to\mathbb{R} \cup\{+\infty\} f : E → R ∪ { + ∞ } be proper. The set of all subgradients of f f f at x x x is denoted ∂ f ( x ) \partial f(x) ∂ f ( x ) , and called the subdifferential of f f f at x x x . We can think of the subdifferential as a set-valued operator x ↦ ∂ f ( x ) x \mapsto\partial f(x) x ↦ ∂ f ( x ) . If ∂ f ( x ) ≠ ∅ \partial f(x) \ne\varnothing ∂ f ( x ) = ∅ , we say that f f f is subdifferentiable at $x . The set of all such points x x x is denoted dom ∂ f \operatorname{dom}\partial f dom ∂ f .
To ensure our algorithm is well-defined, we rely on the fact that the subdifferential is non-empty in the relative interior of the domain.
Theorem 3 (Existence of subgradients). Let E E E be a finite-dimensional real vector space and let f : E → R ∪ { + ∞ } f:E \to\mathbb{R} \cup\{+\infty\} f : E → R ∪ { + ∞ } be a proper convex function.
(i) For every x ∈ rint ( dom f ) x \in\operatorname{rint}(\operatorname{dom} f) x ∈ rint ( dom f ) , the subdifferential ∂ f ( x ) \partial f(x) ∂ f ( x ) is non-empty, closed, and convex.
(ii) If x ∈ int ( dom f ) x \in\operatorname{int}(\operatorname{dom} f) x ∈ int ( dom f ) , where the interior is taken in E E E , then ∂ f ( x ) \partial f(x) ∂ f ( x ) is also compact.
Note that inequality (16) can be rearranged into
f ( x ) − ⟨ s , x ⟩ ≤ f ( y ) − ⟨ s , y ⟩ ∀ y ∈ E , f(x) - \langle s,x\rangle\le f(y) - \langle s,y\rangle\quad\forall y \in E, f ( x ) − ⟨ s , x ⟩ ≤ f ( y ) − ⟨ s , y ⟩ ∀ y ∈ E ,
which can be interpreted as follows: a vector s s s is a subgradient of f f f at x x x if and only if x x x minimizes the function y ↦ f ( y ) − ⟨ s , y ⟩ y \mapsto f(y) - \langle s,y\rangle y ↦ f ( y ) − ⟨ s , y ⟩ . We have arrived at the following result:
Lemma 2. Let f : E → R ∪ { + ∞ } f:E \to\mathbb{R} \cup\{+\infty\} f : E → R ∪ { + ∞ } be proper and consider a point x ∈ dom f x \in\operatorname{dom} f x ∈ dom f . Then
s ∈ ∂ f ( x ) ⟺ x ∈ arg min y ∈ E { f ( y ) − ⟨ s , y ⟩ } . (17) s \in\partial f(x) \quad\Longleftrightarrow\quad x \in\underset{y \in E}{\arg\min}\{f(y) - \langle s,y\rangle\}. \tag*{(17)} s ∈ ∂ f ( x ) ⟺ x ∈ y ∈ E arg min { f ( y ) − ⟨ s , y ⟩} . (17)
We note that the set of minimizers of a function is identical to the set of maximizers of the negative of the function. Applying this simple principle, we observe that
arg min y ∈ E { f ( y ) − ⟨ s , y ⟩ } = arg max y ∈ E { ⟨ s , y ⟩ − f ( y ) } . (18) \underset{y \in E}{\arg\min}\{f(y) - \langle s,y\rangle\} = \underset{y \in E}{\arg\max}\{\langle s,y\rangle- f(y)\}. \tag*{(18)} y ∈ E arg min { f ( y ) − ⟨ s , y ⟩} = y ∈ E arg max {⟨ s , y ⟩ − f ( y )} . (18)
We also recall the fundamental relationship between a function and its convex conjugate, which allows us to invert the subdifferential mapping.
Theorem 4 (Fenchel-Moreau-Rockafellar). Let f : E → R ∪ { + ∞ } f:E \to\mathbb{R} \cup\{+\infty\} f : E → R ∪ { + ∞ } be proper, closed and convex. Then
s ∈ ∂ f ( x ) ⟺ x ∈ ∂ f ∗ ( s ) . s \in\partial f(x) \quad\Longleftrightarrow\quad x \in\partial f^{*}(s). s ∈ ∂ f ( x ) ⟺ x ∈ ∂ f ∗ ( s ) .
Properties of the Primal and Dual Norms
Our analysis depends heavily on the interplay between the primal norm ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ and the dual norm ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ . We first verify that the dual norm satisfies the standard axioms.
Lemma 3 (The Dual Norm is a Norm). Let ( E , ∥ ⋅ ∥ ) (E,\lVert\cdot\rVert) ( E , ∥ ⋅ ∥) be a finite-dimensional normed space. The dual norm ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ , defined for any s ∈ E ∗ s\in E^* s ∈ E ∗ as
∥ s ∥ ∗ ≔ sup ∥ x ∥ ≤ 1 ⟨ s , x ⟩ , \lVert s\rVert_* \coloneqq\sup_{\lVert x\rVert\le1}\langle s,x\rangle, ∥ s ∥ ∗ : = ∥ x ∥ ≤ 1 sup ⟨ s , x ⟩ , satisfies the three axioms of a norm: definiteness, absolute homogeneity, and the triangle inequality.
Proof. We first show that ∥ s ∥ ∗ \lVert s\rVert_* ∥ s ∥ ∗ is finite for every s ∈ E ∗ s\in E^* s ∈ E ∗ . Since E E E is finite-dimensional, its closed unit ball is compact, and every linear functional on E E E is continuous. Therefore, the supremum defining the dual norm is attained and finite:
∥ s ∥ ∗ = sup ∥ x ∥ ≤ 1 ⟨ s , x ⟩ = max ∥ x ∥ ≤ 1 ⟨ s , x ⟩ < + ∞ . \lVert s\rVert_*=\sup_{\lVert x\rVert\le1}\langle s,x\rangle=\max_{\lVert x\rVert\le1}\langle s,x\rangle<+\infty. ∥ s ∥ ∗ = ∥ x ∥ ≤ 1 sup ⟨ s , x ⟩ = ∥ x ∥ ≤ 1 max ⟨ s , x ⟩ < + ∞.
We now verify the three norm axioms.
Proof of Property (i): Definiteness. We must show that ∥ s ∥ ∗ ≥ 0 \lVert s\rVert_*\ge0 ∥ s ∥ ∗ ≥ 0 , with equality if and only if s = 0 s=0 s = 0 .
Non-negativity: The supremum is taken over the unit ball { x ∈ E ∣ ∥ x ∥ ≤ 1 } \{x\in E\mid\lVert x\rVert\le1\} { x ∈ E ∣ ∥ x ∥ ≤ 1 } , which contains the origin 0 E 0_E 0 E . At this point, the dual pairing evaluates to ⟨ s , 0 E ⟩ = 0 \langle s,0_E\rangle=0 ⟨ s , 0 E ⟩ = 0 . Since the supremum over a set must be greater than or equal to the value of the objective at any point in that set, we conclude that ∥ s ∥ ∗ ≥ 0 \lVert s\rVert_*\ge0 ∥ s ∥ ∗ ≥ 0 .
∥ 0 ∥ ∗ = sup ∥ x ∥ ≤ 1 ⟨ 0 , x ⟩ = 0. \lVert0\rVert_*=\sup_{\lVert x\rVert\le1}\langle0,x\rangle=0. ∥ 0 ∥ ∗ = ∥ x ∥ ≤ 1 sup ⟨ 0 , x ⟩ = 0.
For the converse, assume that ∥ s ∥ ∗ = 0 \lVert s\rVert_*=0 ∥ s ∥ ∗ = 0 . By definition of the supremum, this implies that ⟨ s , x ⟩ ≤ 0 \langle s,x\rangle\le0 ⟨ s , x ⟩ ≤ 0 for all x x x in the unit ball { x ∈ E ∣ ∥ x ∥ ≤ 1 } \{x\in E\mid\lVert x\rVert\le1\} { x ∈ E ∣ ∥ x ∥ ≤ 1 } . We will now show that this can only be true if s = 0 s=0 s = 0 .
Assume for contradiction that s ≠ 0 s\ne0 s = 0 . As a non-zero linear functional, s s s cannot be zero for all inputs, so there must exist some vector y ∈ E y\in E y ∈ E such that ⟨ s , y ⟩ ≠ 0 \langle s,y\rangle\ne0 ⟨ s , y ⟩ = 0 . Let’s construct a specific vector x 0 x_0 x 0 in the unit ball. If ⟨ s , y ⟩ > 0 \langle s,y\rangle>0 ⟨ s , y ⟩ > 0 , we can simply set x 0 = y / ∥ y ∥ x_0=y/\lVert y\rVert x 0 = y / ∥ y ∥ . If ⟨ s , y ⟩ < 0 \langle s,y\rangle<0 ⟨ s , y ⟩ < 0 , we can set x 0 = − y / ∥ y ∥ x_0=-y/\lVert y\rVert x 0 = − y / ∥ y ∥ . In either case, we have a vector x 0 x_0 x 0 such that ∥ x 0 ∥ = 1 \lVert x_0\rVert=1 ∥ x 0 ∥ = 1 and, crucially,
⟨ s , x 0 ⟩ = ∣ ⟨ s , y ⟩ ∣ ∥ y ∥ > 0. \langle s,x_0\rangle=\frac{\lvert\langle s,y\rangle\rvert}{\lVert y\rVert}>0. ⟨ s , x 0 ⟩ = ∥ y ∥ ∣⟨ s , y ⟩∣ > 0.
The existence of such an x 0 x_0 x 0 contradicts our initial deduction that ⟨ s , x ⟩ ≤ 0 \langle s,x\rangle\le0 ⟨ s , x ⟩ ≤ 0 for all vectors in the unit ball. Therefore, the initial assumption that s ≠ 0 s\ne0 s = 0 must be false. We conclude that if ∥ s ∥ ∗ = 0 \lVert s\rVert_*=0 ∥ s ∥ ∗ = 0 , then s = 0 s=0 s = 0 .
Proof of Property (ii): Absolute Homogeneity. We need to show that ∥ t s ∥ ∗ = ∣ t ∣ ∥ s ∥ ∗ \lVert ts\rVert_*=\lvert t\rvert\lVert s\rVert_* ∥ t s ∥ ∗ = ∣ t ∣ ∥ s ∥ ∗ for any scalar t ∈ R t\in\mathbb{R} t ∈ R .
∥ t s ∥ ∗ = sup ∥ x ∥ ≤ 1 ⟨ t s , x ⟩ = sup ∥ x ∥ ≤ 1 t ⟨ s , x ⟩ . \lVert ts\rVert_*=\sup_{\lVert x\rVert\le1}\langle ts,x\rangle=\sup_{\lVert x\rVert\le1}t\langle s,x\rangle. ∥ t s ∥ ∗ = ∥ x ∥ ≤ 1 sup ⟨ t s , x ⟩ = ∥ x ∥ ≤ 1 sup t ⟨ s , x ⟩ .
We consider three cases for the scalar t t t :
∥ t s ∥ ∗ = t sup ∥ x ∥ ≤ 1 ⟨ s , x ⟩ = t ∥ s ∥ ∗ = ∣ t ∣ ∥ s ∥ ∗ . \lVert ts\rVert_*=t\sup_{\lVert x\rVert\le1}\langle s,x\rangle=t\lVert s\rVert_*=\lvert t\rvert\lVert s\rVert_*. ∥ t s ∥ ∗ = t ∥ x ∥ ≤ 1 sup ⟨ s , x ⟩ = t ∥ s ∥ ∗ = ∣ t ∣ ∥ s ∥ ∗ .
∥ t s ∥ ∗ = sup ∥ x ∥ ≤ 1 ( − ∣ t ∣ ⟨ s , x ⟩ ) = ∣ t ∣ sup ∥ x ∥ ≤ 1 ( − ⟨ s , x ⟩ ) . \lVert ts\rVert_*=\sup_{\lVert x\rVert\le1}\left(-\lvert t\rvert\langle s,x\rangle\right)=\lvert t\rvert\sup_{\lVert x\rVert\le1}\left(-\langle s,x\rangle\right). ∥ t s ∥ ∗ = ∥ x ∥ ≤ 1 sup ( − ∣ t ∣ ⟨ s , x ⟩ ) = ∣ t ∣ ∥ x ∥ ≤ 1 sup ( − ⟨ s , x ⟩ ) .
Let y = − x y=-x y = − x . The set { y ∈ E ∣ ∥ y ∥ ≤ 1 } \{y\in E\mid\lVert y\rVert\le1\} { y ∈ E ∣ ∥ y ∥ ≤ 1 } is the same as { x ∈ E ∣ ∥ x ∥ ≤ 1 } \{x\in E\mid\lVert x\rVert\le1\} { x ∈ E ∣ ∥ x ∥ ≤ 1 } because ∥ − x ∥ = ∥ x ∥ \lVert-x\rVert=\lVert x\rVert ∥ − x ∥ = ∥ x ∥ . Therefore, we can change the variable in the supremum:
sup ∥ x ∥ ≤ 1 ( − ⟨ s , x ⟩ ) = sup ∥ y ∥ ≤ 1 ⟨ s , y ⟩ = ∥ s ∥ ∗ . \sup_{\lVert x\rVert\le1}\left(-\langle s,x\rangle\right)=\sup_{\lVert y\rVert\le1}\langle s,y\rangle=\lVert s\rVert_*. ∥ x ∥ ≤ 1 sup ( − ⟨ s , x ⟩ ) = ∥ y ∥ ≤ 1 sup ⟨ s , y ⟩ = ∥ s ∥ ∗ .
Combining these gives ∥ t s ∥ ∗ = ∣ t ∣ ∥ s ∥ ∗ \lVert ts\rVert_*=\lvert t\rvert\lVert s\rVert_* ∥ t s ∥ ∗ = ∣ t ∣ ∥ s ∥ ∗ .
The identity holds for all t ∈ R t \in\mathbb{R} t ∈ R .
Proof of Property (iii): Triangle Inequality. We need to show that ∥ s 1 + s 2 ∥ ∗ ≤ ∥ s 1 ∥ ∗ + ∥ s 2 ∥ ∗ \lVert s_1 + s_2\rVert_* \le\lVert s_1\rVert_* + \lVert s_2\rVert_* ∥ s 1 + s 2 ∥ ∗ ≤ ∥ s 1 ∥ ∗ + ∥ s 2 ∥ ∗ . The proof follows directly from the definition of the dual norm and the properties of the supremum.
∥ s 1 + s 2 ∥ ∗ = sup ∥ x ∥ ≤ 1 ⟨ s 1 + s 2 , x ⟩ = sup ∥ x ∥ ≤ 1 ( ⟨ s 1 , x ⟩ + ⟨ s 2 , x ⟩ ) (by linearity of the dual pairing) ≤ sup ∥ x ∥ ≤ 1 ⟨ s 1 , x ⟩ + sup ∥ x ∥ ≤ 1 ⟨ s 2 , x ⟩ (by subadditivity of supremum) = ∥ s 1 ∥ ∗ + ∥ s 2 ∥ ∗ . \begin{aligned}
\lVert s_1+s_2\rVert_* &= \sup_{\lVert x\rVert\le1} \langle s_1+s_2,x\rangle\\
&= \sup_{\lVert x\rVert\le1} \left(\langle s_1,x\rangle+\langle s_2,x\rangle\right) && \text{(by linearity of the dual pairing)}\\
&\le\sup_{\lVert x\rVert\le1}\langle s_1,x\rangle+\sup_{\lVert x\rVert\le1}\langle s_2,x\rangle&& \text{(by subadditivity of supremum)}\\
&= \lVert s_1\rVert_*+\lVert s_2\rVert_*.
\end{aligned} ∥ s 1 + s 2 ∥ ∗ = ∥ x ∥ ≤ 1 sup ⟨ s 1 + s 2 , x ⟩ = ∥ x ∥ ≤ 1 sup ( ⟨ s 1 , x ⟩ + ⟨ s 2 , x ⟩ ) ≤ ∥ x ∥ ≤ 1 sup ⟨ s 1 , x ⟩ + ∥ x ∥ ≤ 1 sup ⟨ s 2 , x ⟩ = ∥ s 1 ∥ ∗ + ∥ s 2 ∥ ∗ . (by linearity of the dual pairing) (by subadditivity of supremum)
This completes the proof.
Lemma 4 (Duality of Norm Equivalence). Let ∥ ⋅ ∥ a \lVert\cdot\rVert_a ∥ ⋅ ∥ a and ∥ ⋅ ∥ b \lVert\cdot\rVert_b ∥ ⋅ ∥ b be two norms on a finite-dimensional real vector space E E E . The subscripts a a a and b b b are neutral labels for the two auxiliary norms considered in this lemma. Let ∥ ⋅ ∥ a , ∗ \lVert\cdot\rVert_{a,*} ∥ ⋅ ∥ a , ∗ and ∥ ⋅ ∥ b , ∗ \lVert\cdot\rVert_{b,*} ∥ ⋅ ∥ b , ∗ denote their respective dual norms on E ∗ E^* E ∗ . If
∥ x ∥ a ≤ ∥ x ∥ b ≤ C ∥ x ∥ a ∀ x ∈ E , \lVert x\rVert_a \le\lVert x\rVert_b \le C\lVert x\rVert_a \qquad\forall x\in E, ∥ x ∥ a ≤ ∥ x ∥ b ≤ C ∥ x ∥ a ∀ x ∈ E , for some constant C ≥ 1 C\ge1 C ≥ 1 , then their dual norms satisfy the reverse equivalence inequality:
1 C ∥ s ∥ a , ∗ ≤ ∥ s ∥ b , ∗ ≤ ∥ s ∥ a , ∗ ∀ s ∈ E ∗ . \frac{1}{C}\lVert s\rVert_{a,*}\le\lVert s\rVert_{b,*}\le\lVert s\rVert_{a,*}\qquad\forall s\in E^*. C 1 ∥ s ∥ a , ∗ ≤ ∥ s ∥ b , ∗ ≤ ∥ s ∥ a , ∗ ∀ s ∈ E ∗ . Proof. The proof relies on the definition of the dual norm as the supremum of the dual pairing over the unit ball of the primal norm. Let B a = { x ∈ E ∣ ∥ x ∥ a ≤ 1 } B_a=\{x\in E\mid\lVert x\rVert_a\le1\} B a = { x ∈ E ∣ ∥ x ∥ a ≤ 1 } and B b = { x ∈ E ∣ ∥ x ∥ b ≤ 1 } B_b=\{x\in E\mid\lVert x\rVert_b\le1\} B b = { x ∈ E ∣ ∥ x ∥ b ≤ 1 } be the unit balls of the respective primal norms.
Step 1: Proving ∥ s ∥ b , ∗ ≤ ∥ s ∥ a , ∗ \lVert s\rVert_{b,*}\le\lVert s\rVert_{a,*} ∥ s ∥ b , ∗ ≤ ∥ s ∥ a , ∗ . The first part of the primal norm inequality is ∥ x ∥ a ≤ ∥ x ∥ b \lVert x\rVert_a\le\lVert x\rVert_b ∥ x ∥ a ≤ ∥ x ∥ b . This inequality has a direct geometric interpretation for the unit balls. If a point y y y is in B b B_b B b , then ∥ y ∥ b ≤ 1 \lVert y\rVert_b\le1 ∥ y ∥ b ≤ 1 . From the inequality, this implies that ∥ y ∥ a ≤ 1 \lVert y\rVert_a\le1 ∥ y ∥ a ≤ 1 , which means y y y must also be in B a B_a B a . Therefore, we have the set inclusion:
B b ⊆ B a . B_b\subseteq B_a. B b ⊆ B a . Now we use the definition of the dual norms. For any s ∈ E ∗ s\in E^* s ∈ E ∗ :
∥ s ∥ b , ∗ = sup x ∈ B b ⟨ s , x ⟩ . \lVert s\rVert_{b,*}=\sup_{x\in B_b}\langle s,x\rangle. ∥ s ∥ b , ∗ = x ∈ B b sup ⟨ s , x ⟩ . Since we are taking the supremum of the same linear function ⟨ s , x ⟩ \langle s,x\rangle ⟨ s , x ⟩ over a smaller set (B b B_b B b ) compared to the set B a B_a B a , its value cannot be larger:
sup x ∈ B b ⟨ s , x ⟩ ≤ sup x ∈ B a ⟨ s , x ⟩ . \sup_{x\in B_b}\langle s,x\rangle\le\sup_{x\in B_a}\langle s,x\rangle. x ∈ B b sup ⟨ s , x ⟩ ≤ x ∈ B a sup ⟨ s , x ⟩ . By definition, the right-hand side is ∥ s ∥ a , ∗ \lVert s\rVert_{a,*} ∥ s ∥ a , ∗ . Thus, we have shown:
∥ s ∥ b , ∗ ≤ ∥ s ∥ a , ∗ . \lVert s\rVert_{b,*}\le\lVert s\rVert_{a,*}. ∥ s ∥ b , ∗ ≤ ∥ s ∥ a , ∗ . Step 2: Proving 1 C ∥ s ∥ a , ∗ ≤ ∥ s ∥ b , ∗ \frac{1}{C}\lVert s\rVert_{a,*}\le\lVert s\rVert_{b,*} C 1 ∥ s ∥ a , ∗ ≤ ∥ s ∥ b , ∗ . The second part of the primal norm inequality is ∥ x ∥ b ≤ C ∥ x ∥ a \lVert x\rVert_b\le C\lVert x\rVert_a ∥ x ∥ b ≤ C ∥ x ∥ a . This can be rewritten as 1 C ∥ x ∥ b ≤ ∥ x ∥ a \frac{1}{C}\lVert x\rVert_b\le\lVert x\rVert_a C 1 ∥ x ∥ b ≤ ∥ x ∥ a . This implies a relationship between the unit balls in the opposite direction. If a point y y y is in B a B_a B a , then ∥ y ∥ a ≤ 1 \lVert y\rVert_a\le1 ∥ y ∥ a ≤ 1 . From the inequality, we have ∥ y ∥ b ≤ C ⋅ ∥ y ∥ a ≤ C ⋅ 1 = C \lVert y\rVert_b\le C\cdot\lVert y\rVert_a\le C\cdot1=C ∥ y ∥ b ≤ C ⋅ ∥ y ∥ a ≤ C ⋅ 1 = C . This means that any point in B a B_a B a is also in the ball of radius C C C for the ∥ ⋅ ∥ b \lVert\cdot\rVert_b ∥ ⋅ ∥ b norm. This gives the set inclusion:
B a ⊆ C ⋅ B b = { x ∈ E ∣ ∥ x ∥ b ≤ C } . B_a\subseteq C\cdot B_b=\{x\in E\mid\lVert x\rVert_b\le C\}. B a ⊆ C ⋅ B b = { x ∈ E ∣ ∥ x ∥ b ≤ C } .
Now we consider the definition of the dual norm ∥ s ∥ a , ∗ \lVert s\rVert_{a,*} ∥ s ∥ a , ∗ :
∥ s ∥ a , ∗ = sup x ∈ B a ⟨ s , x ⟩ . \lVert s\rVert_{a,*}=\sup_{x\in B_a}\langle s,x\rangle. ∥ s ∥ a , ∗ = x ∈ B a sup ⟨ s , x ⟩ .
Since B a B_a B a is a subset of C ⋅ B b C\cdot B_b C ⋅ B b , the supremum over B a B_a B a cannot be larger than the supremum over C ⋅ B b C\cdot B_b C ⋅ B b :
sup x ∈ B a ⟨ s , x ⟩ ≤ sup x ∈ C ⋅ B b ⟨ s , x ⟩ . \sup_{x\in B_a}\langle s,x\rangle\leq\sup_{x\in C\cdot B_b}\langle s,x\rangle. x ∈ B a sup ⟨ s , x ⟩ ≤ x ∈ C ⋅ B b sup ⟨ s , x ⟩ .
The supremum of a linear functional over a scaled ball is the scaling factor times the supremum over the unscaled ball:
sup x ∈ C ⋅ B b ⟨ s , x ⟩ = sup y ∈ B b ⟨ s , C y ⟩ = C sup y ∈ B b ⟨ s , y ⟩ = C ∥ s ∥ b , ∗ . \sup_{x\in C\cdot B_b}\langle s,x\rangle=\sup_{y\in B_b}\langle s,Cy\rangle=C\sup_{y\in B_b}\langle s,y\rangle=C\lVert s\rVert_{b,*}. x ∈ C ⋅ B b sup ⟨ s , x ⟩ = y ∈ B b sup ⟨ s , C y ⟩ = C y ∈ B b sup ⟨ s , y ⟩ = C ∥ s ∥ b , ∗ .
Combining these steps, we have shown:
∥ s ∥ a , ∗ ≤ C ∥ s ∥ b , ∗ . \lVert s\rVert_{a,*}\leq C\lVert s\rVert_{b,*}. ∥ s ∥ a , ∗ ≤ C ∥ s ∥ b , ∗ .
Since C ≥ 1 C\geq1 C ≥ 1 , we can divide by C C C to get the final inequality:
1 C ∥ s ∥ a , ∗ ≤ ∥ s ∥ b , ∗ . \frac{1}{C}\lVert s\rVert_{a,*}\leq\lVert s\rVert_{b,*}. C 1 ∥ s ∥ a , ∗ ≤ ∥ s ∥ b , ∗ .
Putting the two parts together completes the proof of the dual norm equivalence.
We often require the existence of a dual vector that "certifies" the norm of a primal vector. This is guaranteed by the Hahn-Banach theorem.
Lemma 5 (Consequence of Hahn-Banach theorem). Let ( E , ∥ ⋅ ∥ ) (E,\lVert\cdot\rVert) ( E , ∥ ⋅ ∥) be a nonzero finite-dimensional real normed space. For every x ∈ E x\in E x ∈ E , there exists s ∈ E ∗ s\in E^* s ∈ E ∗ such that
⟨ s , x ⟩ = ∥ x ∥ and ∥ s ∥ ∗ = 1. \langle s,x\rangle=\lVert x\rVert\quad\text{and}\quad\lVert s\rVert_*=1. ⟨ s , x ⟩ = ∥ x ∥ and ∥ s ∥ ∗ = 1.
For completeness, we also state the reflexivity property for finite-dimensional normed spaces.
Theorem 5 (Bidual norm equals primal norm). Let ∥ ⋅ ∥ ∗ ∗ \lVert\cdot\rVert_{**} ∥ ⋅ ∥ ∗∗ be the dual of the dual norm ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_* ∥ ⋅ ∥ ∗ , called the bidual norm. Then
∥ ⋅ ∥ ∗ ∗ = ∥ ⋅ ∥ . \lVert\cdot\rVert_{**}=\lVert\cdot\rVert. ∥ ⋅ ∥ ∗∗ = ∥ ⋅ ∥ .
Proof. The proof consists of two parts, showing the inequality in both directions.
We first show that ∥ x ∥ ∗ ∗ ≤ ∥ x ∥ \lVert x\rVert_{**}\leq\lVert x\rVert ∥ x ∥ ∗∗ ≤ ∥ x ∥ . Using the definition of the bidual norm and applying the duality inequality, we get
∥ x ∥ ∗ ∗ = sup ∥ s ∥ ∗ ≤ 1 ⟨ s , x ⟩ ≤ sup ∥ s ∥ ∗ ≤ 1 ∥ s ∥ ∗ ∥ x ∥ = ∥ x ∥ . \lVert x\rVert_{**}=\sup_{\lVert s\rVert_*\leq1}\langle s,x\rangle\leq\sup_{\lVert s\rVert_*\leq1}\lVert s\rVert_*\lVert x\rVert=\lVert x\rVert. ∥ x ∥ ∗∗ = ∥ s ∥ ∗ ≤ 1 sup ⟨ s , x ⟩ ≤ ∥ s ∥ ∗ ≤ 1 sup ∥ s ∥ ∗ ∥ x ∥ = ∥ x ∥ .
We will now prove that ∥ x ∥ ∗ ∗ ≥ ∥ x ∥ \lVert x\rVert_{**}\geq\lVert x\rVert ∥ x ∥ ∗∗ ≥ ∥ x ∥ . This holds trivially for x = 0 x=0 x = 0 . So, let us consider x ≠ 0 x\neq0 x = 0 . The definition of the bidual norm is
∥ x ∥ ∗ ∗ = sup ∥ s ∥ ∗ ≤ 1 ⟨ s , x ⟩ . \lVert x\rVert_{**}=\sup_{\lVert s\rVert_*\leq1}\langle s,x\rangle. ∥ x ∥ ∗∗ = ∥ s ∥ ∗ ≤ 1 sup ⟨ s , x ⟩ .
Using Lemma 5, we know that there exists s ′ ∈ E ∗ s'\in E^* s ′ ∈ E ∗ such that ∥ s ′ ∥ ∗ = 1 \lVert s'\rVert_*=1 ∥ s ′ ∥ ∗ = 1 and ⟨ s ′ , x ⟩ = ∥ x ∥ \langle s',x\rangle=\lVert x\rVert ⟨ s ′ , x ⟩ = ∥ x ∥ . Since this specific s ′ s' s ′ is in the set over which the supremum is taken, the supremum must be at least the value obtained at s ′ s' s ′ . Therefore,
∥ x ∥ ∗ ∗ = sup ∥ s ∥ ∗ ≤ 1 ⟨ s , x ⟩ ≥ ⟨ s ′ , x ⟩ = ∥ x ∥ . \lVert x\rVert_{**}=\sup_{\lVert s\rVert_*\leq1}\langle s,x\rangle\geq\langle s',x\rangle=\lVert x\rVert. ∥ x ∥ ∗∗ = ∥ s ∥ ∗ ≤ 1 sup ⟨ s , x ⟩ ≥ ⟨ s ′ , x ⟩ = ∥ x ∥ .
Combining both inequalities proves the theorem.
The following two lemmas characterize the subdifferentials of the norm function itself. These are building blocks for deriving the subdifferentials of squared norms.
Lemma 6 (Norm of the Subgradient of a Norm). Let ( E , ∥ ⋅ ∥ ) (E,\lVert\cdot\rVert) ( E , ∥ ⋅ ∥) be a finite-dimensional real normed space, and let ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_{*} ∥ ⋅ ∥ ∗ be its dual norm. For every nonzero s ∈ E ∗ s \in E^{*} s ∈ E ∗ and every x ∈ ∂ ∥ s ∥ ∗ x \in\partial\lVert s\rVert_{*} x ∈ ∂ ∥ s ∥ ∗ ,19 we have ∥ x ∥ = 1 \lVert x\rVert= 1 ∥ x ∥ = 1 .
Proof. The proof relies on two fundamental inequalities that characterize the subdifferential of a convex function.
Let s ∈ E ∗ s \in E^{*} s ∈ E ∗ be a non-zero vector. By definition, a vector x ∈ E x \in E x ∈ E is a subgradient of the dual norm ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_{*} ∥ ⋅ ∥ ∗ at s s s if the following inequality holds for all z ∈ E ∗ z \in E^{*} z ∈ E ∗ :
∥ z ∥ ∗ ≥ ∥ s ∥ ∗ + ⟨ x , z − s ⟩ . (19) \lVert z\rVert_{*} \ge\lVert s\rVert_{*} + \langle x,z-s\rangle. \tag*{(19)} ∥ z ∥ ∗ ≥ ∥ s ∥ ∗ + ⟨ x , z − s ⟩ . (19)
Step 1: Proving ∥ x ∥ ≤ 1 \lVert x\rVert\le1 ∥ x ∥ ≤ 1 . We can choose a specific value for z z z in the subgradient inequality. Let us choose z = s + h z=s+h z = s + h for any h ∈ E ∗ h \in E^{*} h ∈ E ∗ . The inequality becomes ∥ s + h ∥ ∗ ≥ ∥ s ∥ ∗ + ⟨ x , h ⟩ \lVert s+h\rVert_{*} \ge\lVert s\rVert_{*}+\langle x,h\rangle ∥ s + h ∥ ∗ ≥ ∥ s ∥ ∗ + ⟨ x , h ⟩ . By the triangle inequality, ∥ s + h ∥ ∗ ≤ ∥ s ∥ ∗ + ∥ h ∥ ∗ \lVert s+h\rVert_{*}\le\lVert s\rVert_{*}+\lVert h\rVert_{*} ∥ s + h ∥ ∗ ≤ ∥ s ∥ ∗ + ∥ h ∥ ∗ . This gives:
∥ s ∥ ∗ + ∥ h ∥ ∗ ≥ ∥ s ∥ ∗ + ⟨ x , h ⟩ ⟹ ⟨ x , h ⟩ ≤ ∥ h ∥ ∗ . \lVert s\rVert_{*}+\lVert h\rVert_{*}\ge\lVert s\rVert_{*}+\langle x,h\rangle\Longrightarrow\langle x,h\rangle\le\lVert h\rVert_{*}. ∥ s ∥ ∗ + ∥ h ∥ ∗ ≥ ∥ s ∥ ∗ + ⟨ x , h ⟩ ⟹ ⟨ x , h ⟩ ≤ ∥ h ∥ ∗ .
This inequality must hold for all h ∈ E ∗ h \in E^{*} h ∈ E ∗ . By the definition of the primal norm as the dual of the dual norm, we have:
∥ x ∥ = sup h ∈ E ∗ , ∥ h ∥ ∗ ≤ 1 ⟨ x , h ⟩ . \lVert x\rVert=\sup_{\substack{h\in E^{*},\ \lVert h\rVert_{*}\le1}}\langle x,h\rangle. ∥ x ∥ = h ∈ E ∗ , ∥ h ∥ ∗ ≤ 1 sup ⟨ x , h ⟩ .
Since ⟨ x , h ⟩ ≤ ∥ h ∥ ∗ \langle x,h\rangle\le\lVert h\rVert_{*} ⟨ x , h ⟩ ≤ ∥ h ∥ ∗ , this implies that for any h h h with ∥ h ∥ ∗ ≤ 1 \lVert h\rVert_{*}\le1 ∥ h ∥ ∗ ≤ 1 , we have ⟨ x , h ⟩ ≤ 1 \langle x,h\rangle\le1 ⟨ x , h ⟩ ≤ 1 . Therefore, the supremum must be less than or equal to 1, which proves ∥ x ∥ ≤ 1 \lVert x\rVert\le1 ∥ x ∥ ≤ 1 .
Step 2: Proving ∥ x ∥ ≥ 1 \lVert x\rVert\ge1 ∥ x ∥ ≥ 1 . Now we choose a different value for z z z in the subgradient inequality. Let us choose z = 0 z=0 z = 0 . Since ∥ 0 ∥ ∗ = 0 \lVert0\rVert_{*}=0 ∥ 0 ∥ ∗ = 0 , substituting into (19) gives:
0 ≥ ∥ s ∥ ∗ + ⟨ x , 0 − s ⟩ 0\ge\lVert s\rVert_{*}+\langle x,0-s\rangle 0 ≥ ∥ s ∥ ∗ + ⟨ x , 0 − s ⟩
0 ≥ ∥ s ∥ ∗ − ⟨ x , s ⟩ . 0\ge\lVert s\rVert_{*}-\langle x,s\rangle. 0 ≥ ∥ s ∥ ∗ − ⟨ x , s ⟩ .
Rearranging this gives the inequality:
⟨ s , x ⟩ ≥ ∥ s ∥ ∗ . \langle s,x\rangle\ge\lVert s\rVert_{*}. ⟨ s , x ⟩ ≥ ∥ s ∥ ∗ .
Now, by the duality inequality, we know that ⟨ s , x ⟩ ≤ ∥ s ∥ ∗ ∥ x ∥ \langle s,x\rangle\le\lVert s\rVert_{*}\lVert x\rVert ⟨ s , x ⟩ ≤ ∥ s ∥ ∗ ∥ x ∥ . Combining these two results:
∥ s ∥ ∗ ≤ ⟨ s , x ⟩ ≤ ∥ s ∥ ∗ ∥ x ∥ . \lVert s\rVert_{*}\le\langle s,x\rangle\le\lVert s\rVert_{*}\lVert x\rVert. ∥ s ∥ ∗ ≤ ⟨ s , x ⟩ ≤ ∥ s ∥ ∗ ∥ x ∥ .
This chain of inequalities implies ∥ s ∥ ∗ ≤ ∥ s ∥ ∗ ∥ x ∥ \lVert s\rVert_{*}\le\lVert s\rVert_{*}\lVert x\rVert ∥ s ∥ ∗ ≤ ∥ s ∥ ∗ ∥ x ∥ . Since we assumed s s s is a non-zero vector, its norm ∥ s ∥ ∗ \lVert s\rVert_{*} ∥ s ∥ ∗ is strictly positive. We can therefore divide both sides by ∥ s ∥ ∗ \lVert s\rVert_{*} ∥ s ∥ ∗ without changing the direction of the inequality:
1 ≤ ∥ x ∥ . 1\le\lVert x\rVert. 1 ≤ ∥ x ∥ .
Combining the results from Step 1 (∥ x ∥ ≤ 1 \lVert x\rVert\le1 ∥ x ∥ ≤ 1 ) and Step 2 (∥ x ∥ ≥ 1 \lVert x\rVert\ge1 ∥ x ∥ ≥ 1 ), we conclude that we must have exactly ∥ x ∥ = 1 \lVert x\rVert=1 ∥ x ∥ = 1 .
Lemma 7 (Subdifferential of the norm). The subdifferentials of ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ and ∥ ⋅ ∥ ∗ \lVert\cdot\rVert_{*} ∥ ⋅ ∥ ∗ are given by
∂ ∥ x ∥ = { { s ∈ E ∗ ∣ ∥ s ∥ ∗ ≤ 1 } if x = 0 , { s ∈ E ∗ ∣ ⟨ s , x ⟩ = ∥ x ∥ and ∥ s ∥ ∗ = 1 } if x ≠ 0 , (20) \partial\lVert x\rVert=
\begin{cases}
\{s\in E^{*}\mid\lVert s\rVert_{*}\le1\} & \text{if }x=0,\\
\{s\in E^{*}\mid\langle s,x\rangle=\lVert x\rVert\ \text{and}\ \lVert s\rVert_{*}=1\} & \text{if }x\ne0,
\end{cases} \tag*{(20)} ∂ ∥ x ∥ = { { s ∈ E ∗ ∣ ∥ s ∥ ∗ ≤ 1 } { s ∈ E ∗ ∣ ⟨ s , x ⟩ = ∥ x ∥ and ∥ s ∥ ∗ = 1 } if x = 0 , if x = 0 , (20) ∂ ∥ s ∥ ∗ = { { x ∈ E ∣ ∥ x ∥ ≤ 1 } if s = 0 , { x ∈ E ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ and ∥ x ∥ = 1 } if s ≠ 0. (21) \partial\lVert s\rVert_{*}=
\begin{cases}
\{x\in E\mid\lVert x\rVert\le1\} & \text{if }s=0,\\
\{x\in E\mid\langle s,x\rangle=\lVert s\rVert_{*}\ \text{and}\ \lVert x\rVert=1\} & \text{if }s\ne0.
\end{cases} \tag*{(21)} ∂ ∥ s ∥ ∗ = { { x ∈ E ∣ ∥ x ∥ ≤ 1 } { x ∈ E ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ and ∥ x ∥ = 1 } if s = 0 , if s = 0. (21)
Proof. We prove (20). The dual formula follows by applying the primal formula to E ∗ E^{*} E ∗ and identifying the bidual norm with the primal norm using Theorem 5. By definition,
s ∈ ∂ ∥ x ∥ ⟺ ∥ y ∥ ≥ ∥ x ∥ + ⟨ s , y − x ⟩ ∀ y ∈ E s\in\partial\lVert x\rVert\Longleftrightarrow\lVert y\rVert\ge\lVert x\rVert+\langle s,y-x\rangle\quad\forall y\in E s ∈ ∂ ∥ x ∥ ⟺ ∥ y ∥ ≥ ∥ x ∥ + ⟨ s , y − x ⟩ ∀ y ∈ E
⟺ ⟨ s , x ⟩ − ∥ x ∥ ≥ ⟨ s , y ⟩ − ∥ y ∥ ∀ y ∈ E \Longleftrightarrow\langle s,x\rangle-\lVert x\rVert\ge\langle s,y\rangle-\lVert y\rVert\quad\forall y\in E ⟺ ⟨ s , x ⟩ − ∥ x ∥ ≥ ⟨ s , y ⟩ − ∥ y ∥ ∀ y ∈ E
⟺ x ∈ arg max y ∈ E { ⟨ s , y ⟩ − ∥ y ∥ } . (22) \Longleftrightarrow x\in\underset{y\in E}{\arg\max}\{\langle s,y\rangle-\lVert y\rVert\}. \tag*{(22)} ⟺ x ∈ y ∈ E arg max {⟨ s , y ⟩ − ∥ y ∥} . (22)
We know from the proof of Lemma 8 that
sup y ∈ E { ⟨ s , y ⟩ − ∥ y ∥ } = { + ∞ when ∥ s ∥ ∗ > 1 , 0 when ∥ s ∥ ∗ ≤ 1. \sup_{y\in E}\{\langle s,y\rangle-\|y\|\} =
\begin{cases}
+\infty& \text{when }\|s\|_* > 1,\\
0 & \text{when }\|s\|_* \le1.
\end{cases} y ∈ E sup {⟨ s , y ⟩ − ∥ y ∥ } = { + ∞ 0 when ∥ s ∥ ∗ > 1 , when ∥ s ∥ ∗ ≤ 1.
Hence, for x x x to be a maximizer, we must have that the maximum is finite (so ∥ s ∥ ∗ ≤ 1 \|s\|_* \le1 ∥ s ∥ ∗ ≤ 1 ) and that the value ⟨ s , x ⟩ − ∥ x ∥ \langle s,x\rangle-\|x\| ⟨ s , x ⟩ − ∥ x ∥ is equal to this maximum value of 0. Therefore,
x ∈ arg max y ∈ E { ⟨ s , y ⟩ − ∥ y ∥ } ⟺ ⟨ s , x ⟩ = ∥ x ∥ and ∥ s ∥ ∗ ≤ 1. x\in\mathop{\operatorname{arg\,max}}_{y\in E}\{\langle s,y\rangle-\|y\|\}
\quad\Longleftrightarrow\quad
\langle s,x\rangle=\|x\|\quad\text{and}\quad\|s\|_*\le1. x ∈ arg max y ∈ E {⟨ s , y ⟩ − ∥ y ∥ } ⟺ ⟨ s , x ⟩ = ∥ x ∥ and ∥ s ∥ ∗ ≤ 1.
We now consider two cases for x x x .
If x = 0 x=0 x = 0 , the condition ⟨ s , x ⟩ = ∥ x ∥ \langle s,x\rangle=\|x\| ⟨ s , x ⟩ = ∥ x ∥ is automatically satisfied. Thus, the characterization becomes:
0 ∈ arg max y ∈ E { ⟨ s , y ⟩ − ∥ y ∥ } ⟺ ∥ s ∥ ∗ ≤ 1. (23) 0\in\mathop{\operatorname{arg\,max}}_{y\in E}\{\langle s,y\rangle-\|y\|\}
\quad\Longleftrightarrow\quad
\|s\|_*\le1. \tag*{(23)} 0 ∈ arg max y ∈ E {⟨ s , y ⟩ − ∥ y ∥ } ⟺ ∥ s ∥ ∗ ≤ 1. (23)
If x ≠ 0 x\ne0 x = 0 , by the duality inequality, it is impossible for ⟨ s , x ⟩ = ∥ x ∥ \langle s,x\rangle=\|x\| ⟨ s , x ⟩ = ∥ x ∥ to hold unless ∥ s ∥ ∗ ≥ 1 \|s\|_*\ge1 ∥ s ∥ ∗ ≥ 1 . Combining this with the condition ∥ s ∥ ∗ ≤ 1 \|s\|_*\le1 ∥ s ∥ ∗ ≤ 1 , we must have ∥ s ∥ ∗ = 1 \|s\|_*=1 ∥ s ∥ ∗ = 1 . Hence,
x ∈ arg max y ∈ E { ⟨ s , y ⟩ − ∥ y ∥ } ⟺ ⟨ s , x ⟩ = ∥ x ∥ and ∥ s ∥ ∗ = 1. (24) x\in\mathop{\operatorname{arg\,max}}_{y\in E}\{\langle s,y\rangle-\|y\|\}
\quad\Longleftrightarrow\quad
\langle s,x\rangle=\|x\|\quad\text{and}\quad\|s\|_*=1. \tag*{(24)} x ∈ arg max y ∈ E {⟨ s , y ⟩ − ∥ y ∥ } ⟺ ⟨ s , x ⟩ = ∥ x ∥ and ∥ s ∥ ∗ = 1. (24)
It only remains to combine (22) with (23) and (24) to establish the result.
It is also useful to identify the convex conjugate of a norm, which corresponds to the indicator function of the dual unit ball.
Lemma 8 (Fenchel conjugate of the norm). The Fenchel conjugate of the norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ is the indicator function of the unit ball with respect to the dual norm. That is,
( ∥ ⋅ ∥ ) ∗ ( s ) = ι B ∗ ( s ) : = { 0 if ∥ s ∥ ∗ ≤ 1 , + ∞ if ∥ s ∥ ∗ > 1 , (\|\cdot\|)^*(s)=\iota_{B_*}(s):=
\begin{cases}
0 & \text{if }\|s\|_*\le1,\\
+\infty& \text{if }\|s\|_*>1,
\end{cases} ( ∥ ⋅ ∥ ) ∗ ( s ) = ι B ∗ ( s ) := { 0 + ∞ if ∥ s ∥ ∗ ≤ 1 , if ∥ s ∥ ∗ > 1 , where B ∗ : = { s ∈ E ∗ ∣ ∥ s ∥ ∗ ≤ 1 } B_*:=\{s\in E^*\mid\|s\|_*\le1\} B ∗ := { s ∈ E ∗ ∣ ∥ s ∥ ∗ ≤ 1 } .
Proof. We consider two cases based on the value of ∥ s ∥ ∗ \|s\|_* ∥ s ∥ ∗ .
Case 1: ∥ s ∥ ∗ ≤ 1 \|s\|_*\le1 ∥ s ∥ ∗ ≤ 1 . By the definition of the Fenchel conjugate and the duality inequality, we have:
( ∥ ⋅ ∥ ) ∗ ( s ) = sup y ∈ E { ⟨ s , y ⟩ − ∥ y ∥ } ≤ sup y ∈ E { ∥ s ∥ ∗ ∥ y ∥ − ∥ y ∥ } ≤ sup y ∈ E { ∥ y ∥ − ∥ y ∥ } = 0. \begin{aligned}
(\|\cdot\|)^*(s) &= \sup_{y\in E}\{\langle s,y\rangle-\|y\|\}\\
&\le\sup_{y\in E}\{\|s\|_*\|y\|-\|y\|\}\\
&\le\sup_{y\in E}\{\|y\|-\|y\|\}=0.
\end{aligned} ( ∥ ⋅ ∥ ) ∗ ( s ) = y ∈ E sup {⟨ s , y ⟩ − ∥ y ∥ } ≤ y ∈ E sup { ∥ s ∥ ∗ ∥ y ∥ − ∥ y ∥ } ≤ y ∈ E sup { ∥ y ∥ − ∥ y ∥ } = 0.
On the other hand, by choosing y = 0 y=0 y = 0 , we see that the expression ⟨ s , y ⟩ − ∥ y ∥ \langle s,y\rangle-\|y\| ⟨ s , y ⟩ − ∥ y ∥ evaluates to 0. Since the supremum must be at least this value, we have ( ∥ ⋅ ∥ ) ∗ ( s ) ≥ 0 (\|\cdot\|)^*(s)\ge0 ( ∥ ⋅ ∥ ) ∗ ( s ) ≥ 0 . Combining these, we conclude that ( ∥ ⋅ ∥ ) ∗ ( s ) = 0 (\|\cdot\|)^*(s)=0 ( ∥ ⋅ ∥ ) ∗ ( s ) = 0 .
Case 2: ∥ s ∥ ∗ > 1 \|s\|_*>1 ∥ s ∥ ∗ > 1 . Since ∥ s ∥ ∗ = sup ∥ x ∥ ≤ 1 ⟨ s , x ⟩ \|s\|_*=\sup_{\|x\|\le1}\langle s,x\rangle ∥ s ∥ ∗ = sup ∥ x ∥ ≤ 1 ⟨ s , x ⟩ , there must exist some x ∈ E x\in E x ∈ E such that ∥ x ∥ ≤ 1 \|x\|\le1 ∥ x ∥ ≤ 1 and ⟨ s , x ⟩ > 1 \langle s,x\rangle>1 ⟨ s , x ⟩ > 1 . For such an x x x , it follows that ⟨ s , x ⟩ − ∥ x ∥ > 0 \langle s,x\rangle-\|x\|>0 ⟨ s , x ⟩ − ∥ x ∥ > 0 . Therefore, we can lower-bound the supremum:
( ∥ ⋅ ∥ ) ∗ ( s ) = sup y ∈ E { ⟨ s , y ⟩ − ∥ y ∥ } ≥ sup t > 0 { ⟨ s , t x ⟩ − ∥ t x ∥ } = sup t > 0 { t ( ⟨ s , x ⟩ − ∥ x ∥ ) } . \begin{aligned}
(\|\cdot\|)^*(s) &= \sup_{y\in E}\{\langle s,y\rangle-\|y\|\}\\
&\ge\sup_{t>0}\{\langle s,tx\rangle-\|tx\|\}\\
&= \sup_{t>0}\{t(\langle s,x\rangle-\|x\|)\}.
\end{aligned} ( ∥ ⋅ ∥ ) ∗ ( s ) = y ∈ E sup {⟨ s , y ⟩ − ∥ y ∥ } ≥ t > 0 sup {⟨ s , t x ⟩ − ∥ t x ∥ } = t > 0 sup { t (⟨ s , x ⟩ − ∥ x ∥ )} .
Since the term ( ⟨ s , x ⟩ − ∥ x ∥ ) (\langle s,x\rangle-\|x\|) (⟨ s , x ⟩ − ∥ x ∥ ) is a positive constant, the supremum over t > 0 t>0 t > 0 is + ∞ +\infty + ∞ .
Properties of Squared Norms and their Conjugates
In this section, we establish fundamental properties relating the squared norm function ϕ ( x ) : = 1 2 ∥ x ∥ 2 \phi(x) := \frac{1}{2}\lVert x\rVert^2 ϕ ( x ) := 2 1 ∥ x ∥ 2 and its Fenchel conjugate ϕ ∗ ( s ) : = 1 2 ∥ s ∥ ∗ 2 \phi^*(s) := \frac{1}{2}\lVert s\rVert_*^2 ϕ ∗ ( s ) := 2 1 ∥ s ∥ ∗ 2 to their respective subgradients. These identities are central to the analysis of non-Euclidean methods and will be used extensively in our proofs.
First, we state the classic inequality bounding the dual pairing by the sum of a function and its conjugate.
Theorem 6 (Fenchel-Young inequality). Let f : E → R ∪ { + ∞ } f : E \to\mathbb{R} \cup\{+\infty\} f : E → R ∪ { + ∞ } be proper, and define its Fenchel conjugate by
f ∗ ( s ) : = sup y ∈ E { ⟨ s , y ⟩ − f ( y ) } , s ∈ E ∗ . f^*(s) := \sup_{y \in E} \{\langle s,y\rangle- f(y)\}, \qquad s \in E^*. f ∗ ( s ) := y ∈ E sup {⟨ s , y ⟩ − f ( y )} , s ∈ E ∗ .
Then, for any x ∈ dom f x \in\operatorname{dom} f x ∈ dom f and s ∈ E ∗ s \in E^* s ∈ E ∗ ,
⟨ s , x ⟩ ≤ f ∗ ( s ) + f ( x ) . (25) \langle s,x\rangle\le f^*(s) + f(x). \tag*{(25)} ⟨ s , x ⟩ ≤ f ∗ ( s ) + f ( x ) . (25)
Moreover,
s ∈ ∂ f ( x ) ⟺ ⟨ s , x ⟩ = f ∗ ( s ) + f ( x ) . (26) s \in\partial f(x) \quad\Longleftrightarrow\quad\langle s,x\rangle= f^*(s) + f(x). \tag*{(26)} s ∈ ∂ f ( x ) ⟺ ⟨ s , x ⟩ = f ∗ ( s ) + f ( x ) . (26)
We next characterize the subdifferentials of the squared primal and dual norms. These are generally set-valued; the sharp operator selects an element of ∂ ϕ ∗ ( s ) \partial\phi^*(s) ∂ ϕ ∗ ( s ) for each s ∈ E ∗ s \in E^* s ∈ E ∗ , where ϕ ∗ ( s ) : = 1 2 ∥ s ∥ ∗ 2 \phi^*(s) := \frac{1}{2}\lVert s\rVert_*^2 ϕ ∗ ( s ) := 2 1 ∥ s ∥ ∗ 2 .
Lemma 9 (Subdifferential of ϕ \phi ϕ and ϕ ∗ \phi^* ϕ ∗ ). The subdifferentials of ϕ ( x ) : = 1 2 ∥ x ∥ 2 \phi(x) := \frac{1}{2}\lVert x\rVert^2 ϕ ( x ) := 2 1 ∥ x ∥ 2 and ϕ ∗ ( s ) : = 1 2 ∥ s ∥ ∗ 2 \phi^*(s) := \frac{1}{2}\lVert s\rVert_*^2 ϕ ∗ ( s ) := 2 1 ∥ s ∥ ∗ 2 are given by:
(i) ∂ ϕ ( x ) = { s ∈ E ∗ ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ ∥ x ∥ and ∥ s ∥ ∗ = ∥ x ∥ } \partial\phi(x) = \{s \in E^* \mid\langle s,x\rangle= \lVert s\rVert_*\lVert x\rVert\text{ and } \lVert s\rVert_* = \lVert x\rVert\} ∂ ϕ ( x ) = { s ∈ E ∗ ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ ∥ x ∥ and ∥ s ∥ ∗ = ∥ x ∥} .
(ii) ∂ ϕ ∗ ( s ) = { x ∈ E ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ ∥ x ∥ and ∥ x ∥ = ∥ s ∥ ∗ } \partial\phi^*(s) = \{x \in E \mid\langle s,x\rangle= \lVert s\rVert_*\lVert x\rVert\text{ and } \lVert x\rVert= \lVert s\rVert_*\} ∂ ϕ ∗ ( s ) = { x ∈ E ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ ∥ x ∥ and ∥ x ∥ = ∥ s ∥ ∗ } .
Proof. We first verify that ϕ ∗ \phi^* ϕ ∗ is the Fenchel conjugate of ϕ \phi ϕ . By the duality inequality, for every s ∈ E ∗ s \in E^* s ∈ E ∗ ,
sup x ∈ E { ⟨ s , x ⟩ − 1 2 ∥ x ∥ 2 } ≤ sup r ≥ 0 { r ∥ s ∥ ∗ − 1 2 r 2 } = 1 2 ∥ s ∥ ∗ 2 . \sup_{x \in E} \left\{\langle s,x\rangle- \frac{1}{2}\lVert x\rVert^2\right\}
\le\sup_{r \ge0} \left\{r\lVert s\rVert_* - \frac{1}{2}r^2\right\}
= \frac{1}{2}\lVert s\rVert_*^2. x ∈ E sup { ⟨ s , x ⟩ − 2 1 ∥ x ∥ 2 } ≤ r ≥ 0 sup { r ∥ s ∥ ∗ − 2 1 r 2 } = 2 1 ∥ s ∥ ∗ 2 .
If s = 0 s = 0 s = 0 , this upper bound is attained at x = 0 x = 0 x = 0 . If s ≠ 0 s \ne0 s = 0 , compactness of the unit ball and continuity of the linear functional give a maximizer u u u with ⟨ s , u ⟩ = ∥ s ∥ ∗ > 0 \langle s,u\rangle= \lVert s\rVert_* > 0 ⟨ s , u ⟩ = ∥ s ∥ ∗ > 0 . This maximizer satisfies ∥ u ∥ = 1 \lVert u\rVert= 1 ∥ u ∥ = 1 , since otherwise scaling it to have norm one would increase ⟨ s , u ⟩ \langle s,u\rangle ⟨ s , u ⟩ . Taking x = ∥ s ∥ ∗ u x = \lVert s\rVert_*u x = ∥ s ∥ ∗ u , we obtain
⟨ s , x ⟩ − 1 2 ∥ x ∥ 2 = ∥ s ∥ ∗ 2 − 1 2 ∥ s ∥ ∗ 2 = 1 2 ∥ s ∥ ∗ 2 . \langle s,x\rangle- \frac{1}{2}\lVert x\rVert^2
= \lVert s\rVert_*^2 - \frac{1}{2}\lVert s\rVert_*^2
= \frac{1}{2}\lVert s\rVert_*^2. ⟨ s , x ⟩ − 2 1 ∥ x ∥ 2 = ∥ s ∥ ∗ 2 − 2 1 ∥ s ∥ ∗ 2 = 2 1 ∥ s ∥ ∗ 2 .
Thus the supremum equals ϕ ∗ ( s ) = 1 2 ∥ s ∥ ∗ 2 \phi^*(s) = \frac{1}{2}\lVert s\rVert_*^2 ϕ ∗ ( s ) = 2 1 ∥ s ∥ ∗ 2 for every s ∈ E ∗ s \in E^* s ∈ E ∗ . We now prove the two subdifferential characterizations.
(i) We first characterize the subdifferential ∂ ϕ ( x ) \partial\phi(x) ∂ ϕ ( x ) . By the Fenchel-Young equality (Theorem 6), an element s ∈ E ∗ s \in E^* s ∈ E ∗ is in ∂ ϕ ( x ) \partial\phi(x) ∂ ϕ ( x ) if and only if the equality holds in the Fenchel-Young inequality:
⟨ s , x ⟩ = ϕ ( x ) + ϕ ∗ ( s ) . \langle s,x\rangle= \phi(x) + \phi^*(s). ⟨ s , x ⟩ = ϕ ( x ) + ϕ ∗ ( s ) .
Substituting the definitions of ϕ \phi ϕ and ϕ ∗ \phi^* ϕ ∗ gives the equivalent condition:
⟨ s , x ⟩ = 1 2 ∥ x ∥ 2 + 1 2 ∥ s ∥ ∗ 2 . (27) \langle s,x\rangle= \frac{1}{2}\lVert x\rVert^2 + \frac{1}{2}\lVert s\rVert_*^2. \tag*{(27)} ⟨ s , x ⟩ = 2 1 ∥ x ∥ 2 + 2 1 ∥ s ∥ ∗ 2 . (27)
We now establish a general chain of inequalities. By the duality inequality, we have:
⟨ s , x ⟩ ≤ ∥ s ∥ ∗ ∥ x ∥ . \langle s,x\rangle\le\lVert s\rVert_*\lVert x\rVert. ⟨ s , x ⟩ ≤ ∥ s ∥ ∗ ∥ x ∥ .
Furthermore, the standard Young’s inequality for scalars, a b ≤ 1 2 a 2 + 1 2 b 2 ab \le\frac{1}{2}a^2 + \frac{1}{2}b^2 ab ≤ 2 1 a 2 + 2 1 b 2 , applied with a = ∥ s ∥ ∗ a = \lVert s\rVert_* a = ∥ s ∥ ∗ and b = ∥ x ∥ b = \lVert x\rVert b = ∥ x ∥ , yields:
∥ s ∥ ∗ ∥ x ∥ ≤ 1 2 ∥ s ∥ ∗ 2 + 1 2 ∥ x ∥ 2 . \lVert s\rVert_*\lVert x\rVert\le\frac{1}{2}\lVert s\rVert_*^2 + \frac{1}{2}\lVert x\rVert^2. ∥ s ∥ ∗ ∥ x ∥ ≤ 2 1 ∥ s ∥ ∗ 2 + 2 1 ∥ x ∥ 2 .
Combining these two results gives the chain:
⟨ s , x ⟩ ≤ ∥ s ∥ ∗ ∥ x ∥ ≤ 1 2 ∥ s ∥ ∗ 2 + 1 2 ∥ x ∥ 2 . (28) \langle s,x\rangle\le\lVert s\rVert_* \lVert x\rVert\le\frac{1}{2}\lVert s\rVert_*^2+\frac{1}{2}\lVert x\rVert^2. \tag*{(28)} ⟨ s , x ⟩ ≤ ∥ s ∥ ∗ ∥ x ∥ ≤ 2 1 ∥ s ∥ ∗ 2 + 2 1 ∥ x ∥ 2 . (28)
For s s s to be an element of ∂ φ ( x ) \partial\varphi(x) ∂ φ ( x ) , the condition in (27) must hold, which means the first and last terms of the chain in (28) must be equal. This can only occur if both inequalities in the chain hold with equality.
The first inequality becomes an equality, ⟨ s , x ⟩ = ∥ s ∥ ∗ ∥ x ∥ \langle s,x\rangle=\lVert s\rVert_*\lVert x\rVert ⟨ s , x ⟩ = ∥ s ∥ ∗ ∥ x ∥ , which is the equality condition in the duality inequality.
The second inequality becomes an equality, ∥ s ∥ ∗ ∥ x ∥ = 1 2 ∥ s ∥ ∗ 2 + 1 2 ∥ x ∥ 2 \lVert s\rVert_*\lVert x\rVert=\frac{1}{2}\lVert s\rVert_*^2+\frac{1}{2}\lVert x\rVert^2 ∥ s ∥ ∗ ∥ x ∥ = 2 1 ∥ s ∥ ∗ 2 + 2 1 ∥ x ∥ 2 . Rearranging gives ∥ s ∥ ∗ 2 − 2 ∥ s ∥ ∗ ∥ x ∥ + ∥ x ∥ 2 = 0 \lVert s\rVert_*^2-2\lVert s\rVert_*\lVert x\rVert+\lVert x\rVert^2=0 ∥ s ∥ ∗ 2 − 2 ∥ s ∥ ∗ ∥ x ∥ + ∥ x ∥ 2 = 0 , which is equivalent to ( ∥ s ∥ ∗ − ∥ x ∥ ) 2 = 0 (\lVert s\rVert_*-\lVert x\rVert)^2=0 (∥ s ∥ ∗ − ∥ x ∥ ) 2 = 0 . This holds if and only if ∥ s ∥ ∗ = ∥ x ∥ \lVert s\rVert_*=\lVert x\rVert ∥ s ∥ ∗ = ∥ x ∥ .
Therefore, s ∈ ∂ φ ( x ) s\in\partial\varphi(x) s ∈ ∂ φ ( x ) if and only if both conditions are satisfied, which proves the first statement.
(ii) The proof for ∂ φ ∗ ( s ) \partial\varphi^*(s) ∂ φ ∗ ( s ) follows from a symmetric argument. We apply the result from part (i) to the function φ ∗ ( s ) \varphi^*(s) φ ∗ ( s ) defined on the space E ∗ E^* E ∗ . The dual of E ∗ E^* E ∗ is the bidual space E ∗ ∗ E^{**} E ∗∗ , and the corresponding dual norm is the bidual norm, ∥ ⋅ ∥ ∗ ∗ \lVert\cdot\rVert_{**} ∥ ⋅ ∥ ∗∗ .
Applying the characterization from (i) to φ ∗ \varphi^* φ ∗ (with the roles of the spaces and norms interchanged) gives that x ∈ ∂ φ ∗ ( s ) x\in\partial\varphi^*(s) x ∈ ∂ φ ∗ ( s ) if and only if:
x ∈ E ∗ ∗ , ⟨ x , s ⟩ = ∥ x ∥ ∗ ∗ ∥ s ∥ ∗ , and ∥ x ∥ ∗ ∗ = ∥ s ∥ ∗ . x\in E^{**},\qquad\langle x,s\rangle=\lVert x\rVert_{**}\lVert s\rVert_*,\qquad\text{and}\qquad\lVert x\rVert_{**}=\lVert s\rVert_*. x ∈ E ∗∗ , ⟨ x , s ⟩ = ∥ x ∥ ∗∗ ∥ s ∥ ∗ , and ∥ x ∥ ∗∗ = ∥ s ∥ ∗ .
As stated in Theorem 5, for a finite-dimensional space, there is a canonical isomorphism between E E E and E ∗ ∗ E^{**} E ∗∗ , and the bidual norm is equal to the primal norm, i.e., ∥ ⋅ ∥ ∗ ∗ = ∥ ⋅ ∥ \lVert\cdot\rVert_{**}=\lVert\cdot\rVert ∥ ⋅ ∥ ∗∗ = ∥ ⋅ ∥ . We can thus identify x ∈ E ∗ ∗ x\in E^{**} x ∈ E ∗∗ with x ∈ E x\in E x ∈ E and replace the bidual norm with the primal norm. This substitution yields the desired characterization for ∂ φ ∗ ( s ) \partial\varphi^*(s) ∂ φ ∗ ( s ) :
∂ φ ∗ ( s ) = { x ∈ E ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ ∥ x ∥ and ∥ x ∥ = ∥ s ∥ ∗ } . \partial\varphi^*(s)=\{x\in E\mid\langle s,x\rangle=\lVert s\rVert_*\lVert x\rVert\ \text{and}\ \lVert x\rVert=\lVert s\rVert_*\}. ∂ φ ∗ ( s ) = { x ∈ E ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ ∥ x ∥ and ∥ x ∥ = ∥ s ∥ ∗ } .
This completes the proof. □
Nonemptiness follows directly from Lemma 5. For x ≠ 0 x\ne0 x = 0 , choose s 0 ∈ E ∗ s_0\in E^* s 0 ∈ E ∗ with ∥ s 0 ∥ ∗ = 1 \lVert s_0\rVert_*=1 ∥ s 0 ∥ ∗ = 1 and ⟨ s 0 , x ⟩ = ∥ x ∥ \langle s_0,x\rangle=\lVert x\rVert ⟨ s 0 , x ⟩ = ∥ x ∥ ; then s = ∥ x ∥ s 0 s=\lVert x\rVert s_0 s = ∥ x ∥ s 0 satisfies the conditions in part (i). For x = 0 x=0 x = 0 , take s = 0 s=0 s = 0 . The same argument on E ∗ E^* E ∗ establishes nonemptiness of ∂ φ ∗ ( s ) \partial\varphi^*(s) ∂ φ ∗ ( s ) .
For s ≠ 0 s\ne0 s = 0 , every x ∈ ∂ φ ∗ ( s ) x\in\partial\varphi^*(s) x ∈ ∂ φ ∗ ( s ) has ∥ x ∥ = ∥ s ∥ ∗ > 0 \lVert x\rVert=\lVert s\rVert_*>0 ∥ x ∥ = ∥ s ∥ ∗ > 0 . Writing x = ∥ s ∥ ∗ u x=\lVert s\rVert_*u x = ∥ s ∥ ∗ u gives
∂ φ ∗ ( s ) = { x ∈ E ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ 2 , ∥ x ∥ = ∥ s ∥ ∗ } = ∥ s ∥ ∗ ⋅ { u ∈ E ∣ ∥ u ∥ = 1 , ⟨ s , u ⟩ = ∥ s ∥ ∗ } = ∥ s ∥ ∗ ⋅ ∂ ∥ s ∥ ∗ , (29) \begin{aligned}
\partial\varphi^*(s)&=\{x\in E\mid\langle s,x\rangle=\lVert s\rVert_*^2,\ \lVert x\rVert=\lVert s\rVert_*\}\\
&=\lVert s\rVert_*\cdot\{u\in E\mid\lVert u\rVert=1,\ \langle s,u\rangle=\lVert s\rVert_*\}\\
&=\lVert s\rVert_*\cdot\partial\lVert s\rVert_*, \tag*{(29)}
\end{aligned} ∂ φ ∗ ( s ) = { x ∈ E ∣ ⟨ s , x ⟩ = ∥ s ∥ ∗ 2 , ∥ x ∥ = ∥ s ∥ ∗ } = ∥ s ∥ ∗ ⋅ { u ∈ E ∣ ∥ u ∥ = 1 , ⟨ s , u ⟩ = ∥ s ∥ ∗ } = ∥ s ∥ ∗ ⋅ ∂ ∥ s ∥ ∗ , (29)
where the last equality follows from Lemma 7. At s = 0 s=0 s = 0 , the characterization above gives ∂ φ ∗ ( 0 ) = { 0 } \partial\varphi^*(0)=\{0\} ∂ φ ∗ ( 0 ) = { 0 } , and ∂ ∥ 0 ∥ ∗ \partial\lVert0\rVert_* ∂ ∥ 0 ∥ ∗ is the primal unit ball, so (29) also holds at zero.
Lemma 10 (Subdifferential of the Squared Dual Norm). Let φ ∗ ( s ) : = 1 2 ∥ s ∥ ∗ 2 \varphi^*(s):=\frac{1}{2}\lVert s\rVert_*^2 φ ∗ ( s ) := 2 1 ∥ s ∥ ∗ 2 . For any s ∈ E ∗ s\in E^* s ∈ E ∗ , its subdifferential is given by
∂ φ ∗ ( s ) = ∥ s ∥ ∗ ⋅ ∂ ∥ s ∥ ∗ , \partial\varphi^*(s)=\lVert s\rVert_*\cdot\partial\lVert s\rVert_*, ∂ φ ∗ ( s ) = ∥ s ∥ ∗ ⋅ ∂ ∥ s ∥ ∗ , where ∂ ∥ s ∥ ∗ \partial\lVert s\rVert_* ∂ ∥ s ∥ ∗ is the subdifferential of the dual norm at s s s . If the dual norm is differentiable at s s s , then this relationship simplifies to ∇ φ ∗ ( s ) = ∥ s ∥ ∗ ⋅ ∇ ∥ s ∥ ∗ \nabla\varphi^*(s)=\lVert s\rVert_*\cdot\nabla\lVert s\rVert_* ∇ φ ∗ ( s ) = ∥ s ∥ ∗ ⋅ ∇ ∥ s ∥ ∗ .
Proof. For s ≠ 0 s\ne0 s = 0 , the identity follows from (29) , obtained by normalizing the subgradients characterized in Lemma 9.
At s = 0 s=0 s = 0 , Lemma 9 gives ∂ φ ∗ ( 0 ) = { 0 } \partial\varphi^*(0)=\{0\} ∂ φ ∗ ( 0 ) = { 0 } , while Lemma 7 gives
∂ ∥ 0 ∥ ∗ = { x ∈ E : ∥ x ∥ ≤ 1 } . \partial\lVert0\rVert_*=\{x\in E:\lVert x\rVert\le1\}. ∂ ∥ 0 ∥ ∗ = { x ∈ E : ∥ x ∥ ≤ 1 } .
Consequently,
∂ ϕ ∗ ( 0 ) = { 0 } = ∥ 0 ∥ ∗ ⋅ ∂ ∥ 0 ∥ ∗ . \partial\phi^*(0)=\{0\}=\lVert0\rVert_* \cdot\partial\lVert0\rVert_*. ∂ ϕ ∗ ( 0 ) = { 0 } = ∥ 0 ∥ ∗ ⋅ ∂ ∥ 0 ∥ ∗ .
Thus the subdifferential identity holds for every s ∈ E ∗ s \in E^* s ∈ E ∗ .
If the dual norm is differentiable at s s s , the ordinary chain rule applied to ϕ ∗ ( s ) = 1 2 ∥ s ∥ ∗ 2 \phi^*(s)=\frac{1}{2}\lVert s\rVert_*^2 ϕ ∗ ( s ) = 2 1 ∥ s ∥ ∗ 2 yields
∇ ϕ ∗ ( s ) = ∥ s ∥ ∗ ⋅ ∇ ∥ s ∥ ∗ . \nabla\phi^*(s)=\lVert s\rVert_* \cdot\nabla\lVert s\rVert_*. ∇ ϕ ∗ ( s ) = ∥ s ∥ ∗ ⋅ ∇ ∥ s ∥ ∗ .
These subdifferential mappings satisfy useful scaling properties.
Lemma 11 (Homogeneity of ∂ ϕ \partial\phi ∂ ϕ and ∂ ϕ ∗ \partial\phi^* ∂ ϕ ∗ ). The subdifferentials of ϕ \phi ϕ and ϕ ∗ \phi^* ϕ ∗ satisfy the relations
∂ ϕ ( t x ) = t ∂ ϕ ( x ) , ∀ x ∈ E , t ∈ R , \partial\phi(tx)=t\partial\phi(x), \qquad\forall x\in E,t\in\mathbb{R}, ∂ ϕ ( t x ) = t ∂ ϕ ( x ) , ∀ x ∈ E , t ∈ R , ∂ ϕ ∗ ( t s ) = t ∂ ϕ ∗ ( s ) , ∀ s ∈ E ∗ , t ∈ R . \partial\phi^*(ts)=t\partial\phi^*(s), \qquad\forall s\in E^*,t\in\mathbb{R}. ∂ ϕ ∗ ( t s ) = t ∂ ϕ ∗ ( s ) , ∀ s ∈ E ∗ , t ∈ R . In particular, ∂ ϕ ( 0 ) = { 0 } \partial\phi(0)=\{0\} ∂ ϕ ( 0 ) = { 0 } and ∂ ϕ ∗ ( 0 ) = { 0 } \partial\phi^*(0)=\{0\} ∂ ϕ ∗ ( 0 ) = { 0 } .
Finally, we derive key identities involving the sharp operator ( ⋅ ) # (\cdot)^\# ( ⋅ ) # , which are used directly in the derivation of the descent lemma.
Lemma 12 (Properties of the Sharp Operators). Let x ∈ E x\in E x ∈ E and s ∈ E ∗ s\in E^* s ∈ E ∗ . The following four identities hold for any element selected from the respective subdifferentials:
(i) For any u ∈ ∂ ϕ ∗ ( s ) u\in\partial\phi^*(s) u ∈ ∂ ϕ ∗ ( s ) , we have ⟨ s , u ⟩ = ∥ s ∥ ∗ 2 \langle s,u\rangle=\lVert s\rVert_*^2 ⟨ s , u ⟩ = ∥ s ∥ ∗ 2 .
(ii) For any u ∈ ∂ ϕ ∗ ( s ) u\in\partial\phi^*(s) u ∈ ∂ ϕ ∗ ( s ) , we have ∥ u ∥ = ∥ s ∥ ∗ \lVert u\rVert=\lVert s\rVert_* ∥ u ∥ = ∥ s ∥ ∗ .
(iii) For any v ∈ ∂ ϕ ( x ) v\in\partial\phi(x) v ∈ ∂ ϕ ( x ) , we have ⟨ v , x ⟩ = ∥ x ∥ 2 \langle v,x\rangle=\lVert x\rVert^2 ⟨ v , x ⟩ = ∥ x ∥ 2 .
(iv) For any v ∈ ∂ ϕ ( x ) v\in\partial\phi(x) v ∈ ∂ ϕ ( x ) , we have ∥ v ∥ ∗ = ∥ x ∥ \lVert v\rVert_*=\lVert x\rVert ∥ v ∥ ∗ = ∥ x ∥ .
Proof. Let u ∈ ∂ ϕ ∗ ( s ) u\in\partial\phi^*(s) u ∈ ∂ ϕ ∗ ( s ) and v ∈ ∂ ϕ ( x ) v\in\partial\phi(x) v ∈ ∂ ϕ ( x ) be arbitrary. By Lemma 9(ii),
∥ u ∥ = ∥ s ∥ ∗ , ⟨ s , u ⟩ = ∥ s ∥ ∗ ∥ u ∥ = ∥ s ∥ ∗ 2 , \lVert u\rVert=\lVert s\rVert_*, \qquad\langle s,u\rangle=\lVert s\rVert_*\lVert u\rVert=\lVert s\rVert_*^2, ∥ u ∥ = ∥ s ∥ ∗ , ⟨ s , u ⟩ = ∥ s ∥ ∗ ∥ u ∥ = ∥ s ∥ ∗ 2 ,
which proves (i) and (ii). Similarly, Lemma 9(i) gives
∥ v ∥ ∗ = ∥ x ∥ , ⟨ v , x ⟩ = ∥ v ∥ ∗ ∥ x ∥ = ∥ x ∥ 2 , \lVert v\rVert_*=\lVert x\rVert, \qquad\langle v,x\rangle=\lVert v\rVert_*\lVert x\rVert=\lVert x\rVert^2, ∥ v ∥ ∗ = ∥ x ∥ , ⟨ v , x ⟩ = ∥ v ∥ ∗ ∥ x ∥ = ∥ x ∥ 2 ,
which proves (iii) and (iv).
Inequalities and Specific Geometries
To bound dual pairings in our convergence analysis, we require the following generalized version of Young’s inequality.
Lemma 13 (Generalized Young inequality). Let ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ be any norm on R d \mathbb{R}^d R d with dual norm ∥ u ∥ ∗ = sup ∥ x ∥ ≤ 1 ⟨ u , x ⟩ \lVert u\rVert_*=\sup_{\lVert x\rVert\leq1}\langle u,x\rangle ∥ u ∥ ∗ = sup ∥ x ∥ ≤ 1 ⟨ u , x ⟩ . Then for all u , v ∈ R d u,v\in\mathbb{R}^d u , v ∈ R d and every s > 0 s>0 s > 0 ,
⟨ u , v ⟩ ≤ ∥ u ∥ ∗ ∥ v ∥ ≤ 1 2 s ∥ u ∥ ∗ 2 + s 2 ∥ v ∥ 2 . \langle u,v\rangle\leq\lVert u\rVert_*\lVert v\rVert\leq\frac{1}{2s}\lVert u\rVert_*^2+\frac{s}{2}\lVert v\rVert^2. ⟨ u , v ⟩ ≤ ∥ u ∥ ∗ ∥ v ∥ ≤ 2 s 1 ∥ u ∥ ∗ 2 + 2 s ∥ v ∥ 2 . Proof. (i) Duality (Hölder) inequality. If v = 0 v=0 v = 0 the claim is trivial. Otherwise, set w : = v / ∥ v ∥ w:=v/\lVert v\rVert w := v / ∥ v ∥ , so ∥ w ∥ = 1 \lVert w\rVert=1 ∥ w ∥ = 1 . By the definition of the dual norm,
⟨ u , w ⟩ ≤ sup ∥ x ∥ ≤ 1 ⟨ u , x ⟩ = ∥ u ∥ ∗ . \langle u,w\rangle\leq\sup_{\lVert x\rVert\leq1}\langle u,x\rangle=\lVert u\rVert_*. ⟨ u , w ⟩ ≤ ∥ x ∥ ≤ 1 sup ⟨ u , x ⟩ = ∥ u ∥ ∗ . Multiplying by ∥ v ∥ \lVert v\rVert ∥ v ∥ gives ⟨ u , v ⟩ = ∥ v ∥ ⟨ u , w ⟩ ≤ ∥ u ∥ ∗ ∥ v ∥ \langle u,v\rangle=\lVert v\rVert\langle u,w\rangle\leq\lVert u\rVert_*\lVert v\rVert ⟨ u , v ⟩ = ∥ v ∥ ⟨ u , w ⟩ ≤ ∥ u ∥ ∗ ∥ v ∥ .
(ii) Young’s inequality with a parameter. For any a , b ∈ R a,b\in\mathbb{R} a , b ∈ R and any s > 0 s>0 s > 0 ,
0 ≤ ( a 2 s − s 2 b ) 2 = a 2 2 s + s 2 b 2 − a b , 0\leq\left(\frac{a}{\sqrt{2s}}-\sqrt{\frac{s}{2}}b\right)^2=\frac{a^2}{2s}+\frac{s}{2}b^2-ab, 0 ≤ ( 2 s a − 2 s b ) 2 = 2 s a 2 + 2 s b 2 − ab ,
hence a b ≤ a 2 2 s + s 2 b 2 ab \le\frac{a^2}{2s}+\frac{s}{2}b^2 ab ≤ 2 s a 2 + 2 s b 2 . Applying this with a = ∥ u ∥ ∗ a=\lVert u\rVert_* a = ∥ u ∥ ∗ and b = ∥ v ∥ b=\lVert v\rVert b = ∥ v ∥ yields
∥ u ∥ ∗ ∥ v ∥ ≤ 1 2 s ∥ u ∥ ∗ 2 + s 2 ∥ v ∥ 2 . \lVert u\rVert_*\lVert v\rVert\le\frac{1}{2s}\lVert u\rVert_*^2+\frac{s}{2}\lVert v\rVert^2. ∥ u ∥ ∗ ∥ v ∥ ≤ 2 s 1 ∥ u ∥ ∗ 2 + 2 s ∥ v ∥ 2 .
Combining (i) and (ii) proves the statement.
For our examples involving matrix optimization, we utilize the comparison inequalities between different Schatten norms.
Lemma 14 (Schatten norm comparison). Let X ∈ R m × n X \in\mathbb{R}^{m\times n} X ∈ R m × n and r : = min { m , n } r := \min\{m,n\} r := min { m , n } . For any 1 ≤ ρ ≤ p ≤ ∞ 1 \le\rho\le p \le\infty 1 ≤ ρ ≤ p ≤ ∞ ,
∥ X ∥ S p ≤ ∥ X ∥ S ρ ≤ r 1 ρ − 1 p ∥ X ∥ S p . \lVert X\rVert_{S_p} \le\lVert X\rVert_{S_\rho} \le r^{\frac{1}{\rho}-\frac{1}{p}}\lVert X\rVert_{S_p}. ∥ X ∥ S p ≤ ∥ X ∥ S ρ ≤ r ρ 1 − p 1 ∥ X ∥ S p . Proof. Let σ ( X ) ∈ R + r \sigma(X) \in\mathbb{R}_+^r σ ( X ) ∈ R + r be the vector of singular values of X X X . By definition, ∥ X ∥ S q = ∥ σ ( X ) ∥ q \lVert X\rVert_{S_q}=\lVert\sigma(X)\rVert_q ∥ X ∥ S q = ∥ σ ( X ) ∥ q for every q ∈ [ 1 , ∞ ] q \in[1,\infty] q ∈ [ 1 , ∞ ] . Applying the standard ℓ p \ell_p ℓ p comparison inequality in R r \mathbb{R}^r R r to σ ( X ) \sigma(X) σ ( X ) yields the claim.
Probabilistic Preliminaries for Martingale Bounds
To rigorously analyze the behavior of stochastic gradients in non-Euclidean norms, we need tools from the theory of vector-valued martingales. We begin by recalling the necessary probabilistic definitions.
Definition 4 (Measurable space). A measurable space is a pair ( S , S ) (S,\mathcal{S}) ( S , S ) , where S S S is a set and S \mathcal{S} S is a σ \sigma σ -algebra of subsets of S S S , i.e., a collection S ⊆ 2 S \mathcal{S} \subseteq2^S S ⊆ 2 S such that:
(i) S ∈ S S \in\mathcal{S} S ∈ S ;
(ii) if A ∈ S A \in\mathcal{S} A ∈ S , then its complement S ∖ A ∈ S S \setminus A \in\mathcal{S} S ∖ A ∈ S ;
(iii) if A 1 , A 2 , ⋯ ∈ S A_1,A_2,\cdots\in\mathcal{S} A 1 , A 2 , ⋯ ∈ S , then the countable union ⋃ n = 1 ∞ A n ∈ S \bigcup_{n=1}^{\infty} A_n \in\mathcal{S} ⋃ n = 1 ∞ A n ∈ S .
Elements of S \mathcal{S} S are called measurable sets.
Definition 5 (F \mathcal{F} F -measurable random variable). Let ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) be a probability space and let ( S , S ) (S,\mathcal{S}) ( S , S ) be a measurable space. A map ξ : Ω → S \xi:\Omega\to S ξ : Ω → S is called F \mathcal{F} F -measurable (or simply measurable) if for every measurable set A ∈ S A \in\mathcal{S} A ∈ S , the preimage
ξ − 1 ( A ) : = { ω ∈ Ω : ξ ( ω ) ∈ A } \xi^{-1}(A):=\{\omega\in\Omega:\xi(\omega)\in A\} ξ − 1 ( A ) := { ω ∈ Ω : ξ ( ω ) ∈ A } belongs to F \mathcal{F} F , i.e.,
∀ A ∈ S : ξ − 1 ( A ) ∈ F . \forall A \in\mathcal{S}:\quad\xi^{-1}(A)\in\mathcal{F}. ∀ A ∈ S : ξ − 1 ( A ) ∈ F . In particular, when S = R S=\mathbb{R} S = R and S = B ( R ) \mathcal{S}=\mathcal{B}(\mathbb{R}) S = B ( R ) is the Borel σ \sigma σ -algebra, this becomes:
∀ B ∈ B ( R ) : { ω ∈ Ω : ξ ( ω ) ∈ B } ∈ F . \forall B \in\mathcal{B}(\mathbb{R}):\quad\{\omega\in\Omega:\xi(\omega)\in B\}\in\mathcal{F}. ∀ B ∈ B ( R ) : { ω ∈ Ω : ξ ( ω ) ∈ B } ∈ F .
Definition 6 (Sub-σ \sigma σ -algebra). Let ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) be a probability space. A collection of sets G \mathcal{G} G is called a sub-σ \sigma σ -algebra of F \mathcal{F} F if:
(i) G ⊆ F \mathcal{G}\subseteq\mathcal{F} G ⊆ F (i.e., ∀ A ⊆ Ω : A ∈ G ⇒ A ∈ F \forall A\subseteq\Omega:A\in\mathcal{G}\Rightarrow A\in\mathcal{F} ∀ A ⊆ Ω : A ∈ G ⇒ A ∈ F ),
(ii) G \mathcal{G} G is a σ \sigma σ -algebra on Ω \Omega Ω , i.e., Ω ∈ G \Omega\in\mathcal{G} Ω ∈ G , A ∈ G ⇒ A c ∈ G A\in\mathcal{G}\Rightarrow A^c\in\mathcal{G} A ∈ G ⇒ A c ∈ G , and A 1 , A 2 , ⋯ ∈ G ⇒ ⋃ n = 1 ∞ A n ∈ G A_1,A_2,\cdots\in\mathcal{G}\Rightarrow\bigcup_{n=1}^{\infty}A_n\in\mathcal{G} A 1 , A 2 , ⋯ ∈ G ⇒ ⋃ n = 1 ∞ A n ∈ G .
Definition 7 (Filtration). Let ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) be a probability space and let T T T be a totally ordered index set (e.g., T = N 0 T=\mathbb{N}_0 T = N 0 or T = [ 0 , ∞ ) T=[0,\infty) T = [ 0 , ∞ ) ). A family of σ \sigma σ -algebras { F t } t ∈ T \{\mathcal{F}_t\}_{t\in T} { F t } t ∈ T is called a filtration if:
(i) F t ⊆ F \mathcal{F}_t\subseteq\mathcal{F} F t ⊆ F for every t ∈ T t\in T t ∈ T (each F t \mathcal{F}_t F t is a sub-σ \sigma σ -algebra of F \mathcal{F} F );
(ii) for all s , t ∈ T s,t\in T s , t ∈ T with s ≤ t s\le t s ≤ t , one has F s ⊆ F t \mathcal{F}_s\subseteq\mathcal{F}_t F s ⊆ F t .
The inclusion F s ⊆ F t \mathcal{F}_s\subseteq\mathcal{F}_t F s ⊆ F t is an inclusion of collections of sets: it means ∀ A ⊆ Ω \forall A\subseteq\Omega ∀ A ⊆ Ω , A ∈ F s ⇒ A ∈ F t A\in\mathcal{F}_s\Rightarrow A\in\mathcal{F}_t A ∈ F s ⇒ A ∈ F t .
Generated filtrations (common construction). Given random variables { X u } u ∈ T \{X_u\}_{u\in T} { X u } u ∈ T on ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) , one can define the filtration generated by ( X u ) (X_u) ( X u ) by
F t : = σ ( { X u : u ≤ t } ) , \mathcal{F}_t:=\sigma(\{X_u:u\le t\}), F t := σ ({ X u : u ≤ t }) ,
i.e., F t \mathcal{F}_t F t is the smallest σ \sigma σ -algebra with respect to which all X u X_u X u for u ≤ t u\le t u ≤ t are measurable. In particular, if s ≤ t s\le t s ≤ t then { X u : u ≤ s } ⊆ { X u : u ≤ t } \{X_u:u\le s\}\subseteq\{X_u:u\le t\} { X u : u ≤ s } ⊆ { X u : u ≤ t } , and therefore F s = σ ( { X u : u ≤ s } ) ⊆ σ ( { X u : u ≤ t } ) = F t \mathcal{F}_s=\sigma(\{X_u:u\le s\})\subseteq\sigma(\{X_u:u\le t\})=\mathcal{F}_t F s = σ ({ X u : u ≤ s }) ⊆ σ ({ X u : u ≤ t }) = F t .
We interpret F t \mathcal{F}_t F t as the information available up to time t t t .
Definition 8 (Martingale-difference sequence). Let ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) be a probability space, let { F t } t ≥ 0 \{\mathcal{F}_t\}_{t\ge0} { F t } t ≥ 0 be a filtration ( F t − 1 ⊆ F t ⊆ F ) (\mathcal{F}_{t-1}\subseteq\mathcal{F}_t\subseteq\mathcal{F}) ( F t − 1 ⊆ F t ⊆ F ) , and let ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) be a finite-dimensional normed space. A sequence of V V V -valued random variables { ξ t } t ≥ 1 \{\xi_t\}_{t\ge1} { ξ t } t ≥ 1 is called an { F t } \{\mathcal{F}_t\} { F t } -martingale-difference sequence if for every t ≥ 1 t\ge1 t ≥ 1 :
(i) ξ t \xi_t ξ t is F t \mathcal{F}_t F t -measurable,
(ii) E [ ∥ ξ t ∥ V ] < ∞ \mathbb{E}[\|\xi_t\|_V]<\infty E [ ∥ ξ t ∥ V ] < ∞ (integrability),
(iii) E [ ξ t ∣ F t − 1 ] = 0 \mathbb{E}[\xi_t\mid\mathcal{F}_{t-1}]=0 E [ ξ t ∣ F t − 1 ] = 0 almost surely.
In this case, the partial sums S t : = ∑ i = 1 t ξ i S_t:=\sum_{i=1}^t\xi_i S t := ∑ i = 1 t ξ i form a V V V -valued martingale with respect to { F t } \{\mathcal{F}_t\} { F t } .
The next lemma converts κ \kappa κ -regularity into a second-moment bound for vector-valued martingale sums. Intuitively, κ \kappa κ controls how large E ∥ S T ∥ 2 \mathbb{E}\|S_T\|^2 E ∥ S T ∥ 2 can be relative to the accumulated ∑ t = 1 T E ∥ ξ t ∥ 2 \sum_{t=1}^T\mathbb{E}\|\xi_t\|^2 ∑ t = 1 T E ∥ ξ t ∥ 2 .
Lemma 15 (Second-moment bound from κ V \kappa_V κ V -regularity). Assume that the normed space ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) is κ V \kappa_V κ V -regular in the sense of Definition 1. Let { ξ t } t = 1 T \{\xi_t\}_{t=1}^T { ξ t } t = 1 T be a V V V -valued { F t } \{\mathcal{F}_t\} { F t } -martingale-difference sequence such that
E [ ∥ ξ t ∥ V 2 ] < ∞ , t = 1 , … , T . \mathbb{E}[\|\xi_t\|_V^2]<\infty,\qquad t=1,\ldots,T. E [ ∥ ξ t ∥ V 2 ] < ∞ , t = 1 , … , T . Then, for
S T : = ∑ t = 1 T ξ t , S_T:=\sum_{t=1}^T\xi_t, S T := t = 1 ∑ T ξ t , one has
E [ ∥ S T ∥ V 2 ] ≤ κ V ∑ t = 1 T E [ ∥ ξ t ∥ V 2 ] . (30) \mathbb{E}[\|S_T\|_V^2]\le\kappa_V\sum_{t=1}^T\mathbb{E}[\|\xi_t\|_V^2]. \tag*{(30)} E [ ∥ S T ∥ V 2 ] ≤ κ V t = 1 ∑ T E [ ∥ ξ t ∥ V 2 ] . (30)
Proof. To simplify notation throughout this proof, we write
∥ x ∥ : = ∥ x ∥ V , ∥ x ∥ + : = ∥ x ∥ V , + , κ : = κ V , κ + : = κ V , + . \|x\|:=\|x\|_V,\qquad\|x\|_+:=\|x\|_{V,+},\qquad\kappa:=\kappa_V,\qquad\kappa_+:=\kappa_{V,+}. ∥ x ∥ := ∥ x ∥ V , ∥ x ∥ + := ∥ x ∥ V , + , κ := κ V , κ + := κ V , + .
By κ \kappa κ -regularity, there exist κ + ∈ [ 1 , κ ] \kappa_+\in[1,\kappa] κ + ∈ [ 1 , κ ] and a κ + \kappa_+ κ + -smooth auxiliary norm ∥ ⋅ ∥ + \|\cdot\|_+ ∥ ⋅ ∥ + on V V V such that
∥ x ∥ 2 ≤ ∥ x ∥ + 2 ≤ κ κ + ∥ x ∥ 2 , ∀ x ∈ V . (31) \|x\|^2\le\|x\|_+^2\le\frac{\kappa}{\kappa_+}\|x\|^2,\qquad\forall x\in V. \tag*{(31)} ∥ x ∥ 2 ≤ ∥ x ∥ + 2 ≤ κ + κ ∥ x ∥ 2 , ∀ x ∈ V . (31)
Define the auxiliary function
φ + ( x ) : = 1 2 ∥ x ∥ + 2 . \varphi_+(x):=\frac{1}{2}\|x\|_+^2. φ + ( x ) := 2 1 ∥ x ∥ + 2 .
Since ( V , ∥ ⋅ ∥ + ) (V,\|\cdot\|_+) ( V , ∥ ⋅ ∥ + ) is κ + \kappa_+ κ + -smooth, for all x , h ∈ V x,h\in V x , h ∈ V we have
φ + ( x + h ) ≤ φ + ( x ) + ⟨ ∇ φ + ( x ) , h ⟩ + κ + 2 ∥ h ∥ + 2 . (32) \varphi_+(x+h)\le\varphi_+(x)+\langle\nabla\varphi_+(x),h\rangle+\frac{\kappa_+}{2}\|h\|_+^2. \tag*{(32)} φ + ( x + h ) ≤ φ + ( x ) + ⟨ ∇ φ + ( x ) , h ⟩ + 2 κ + ∥ h ∥ + 2 . (32)
For each t = 0 , 1 , … , T t=0,1,\ldots,T t = 0 , 1 , … , T , let
S t : = ∑ i = 1 t ξ i , S 0 : = 0. S_t:=\sum_{i=1}^{t}\xi_i,\qquad S_0:=0. S t := i = 1 ∑ t ξ i , S 0 := 0.
Applying (32) with x = S t − 1 x=S_{t-1} x = S t − 1 and h = ξ t h=\xi_t h = ξ t , we get
ϕ + ( S t ) ≤ ϕ + ( S t − 1 ) + ⟨ ∇ ϕ + ( S t − 1 ) , ξ t ⟩ + κ + 2 ∥ ξ t ∥ + 2 . \phi_+(S_t)\leq\phi_+(S_{t-1})+\langle\nabla\phi_+(S_{t-1}),\xi_t\rangle+\frac{\kappa_+}{2}\lVert\xi_t\rVert_+^2. ϕ + ( S t ) ≤ ϕ + ( S t − 1 ) + ⟨ ∇ ϕ + ( S t − 1 ) , ξ t ⟩ + 2 κ + ∥ ξ t ∥ + 2 .
Before taking conditional expectations, we verify integrability. By (31) , E [ ∥ ξ i ∥ + 2 ] < ∞ \mathbb{E}\left[\lVert\xi_i\rVert_+^2\right]<\infty E [ ∥ ξ i ∥ + 2 ] < ∞ for every i i i . The triangle inequality and the scalar Cauchy-Schwarz inequality give
E [ ∥ S t ∥ + 2 ] ≤ t ∑ i = 1 t E [ ∥ ξ i ∥ + 2 ] < ∞ , t = 1 , … , T , \mathbb{E}\left[\lVert S_t\rVert_+^2\right]\leq t\sum_{i=1}^{t}\mathbb{E}\left[\lVert\xi_i\rVert_+^2\right]<\infty,\qquad t=1,\ldots,T, E [ ∥ S t ∥ + 2 ] ≤ t i = 1 ∑ t E [ ∥ ξ i ∥ + 2 ] < ∞ , t = 1 , … , T ,
and S 0 = 0 S_0=0 S 0 = 0 . Let ∥ ⋅ ∥ + , ∗ \lVert\cdot\rVert_{+,*} ∥ ⋅ ∥ + , ∗ denote the dual norm of ∥ ⋅ ∥ + \lVert\cdot\rVert_+ ∥ ⋅ ∥ + . By Lemma 12 (iv), applied to ϕ + \phi_+ ϕ + ,
∥ ∇ ϕ + ( z ) ∥ + , ∗ = ∥ z ∥ + , z ∈ V . \lVert\nabla\phi_+(z)\rVert_{+,*}=\lVert z\rVert_+,\qquad z\in V. ∥ ∇ ϕ + ( z ) ∥ + , ∗ = ∥ z ∥ + , z ∈ V .
Consequently, the duality inequality and Cauchy-Schwarz yield
E [ ∣ ⟨ ∇ ϕ + ( S t − 1 ) , ξ t ⟩ ∣ ] ≤ E [ ∥ S t − 1 ∥ + ∥ ξ t ∥ + ] ≤ ( E [ ∥ S t − 1 ∥ + 2 ] ) 1 / 2 ( E [ ∥ ξ t ∥ + 2 ] ) 1 / 2 < ∞ . \mathbb{E}\left[\left|\langle\nabla\phi_+(S_{t-1}),\xi_t\rangle\right|\right]\leq\mathbb{E}\left[\lVert S_{t-1}\rVert_+\lVert\xi_t\rVert_+\right]
\leq\left(\mathbb{E}\left[\lVert S_{t-1}\rVert_+^2\right]\right)^{1/2}\left(\mathbb{E}\left[\lVert\xi_t\rVert_+^2\right]\right)^{1/2}<\infty. E [ ∣ ⟨ ∇ ϕ + ( S t − 1 ) , ξ t ⟩ ∣ ] ≤ E [ ∥ S t − 1 ∥ + ∥ ξ t ∥ + ] ≤ ( E [ ∥ S t − 1 ∥ + 2 ] ) 1/2 ( E [ ∥ ξ t ∥ + 2 ] ) 1/2 < ∞.
Thus all terms in the preceding smoothness inequality are integrable. Since ∇ ϕ + \nabla\phi_+ ∇ ϕ + is continuous and S t − 1 S_{t-1} S t − 1 is F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable, ∇ ϕ + ( S t − 1 ) \nabla\phi_+(S_{t-1}) ∇ ϕ + ( S t − 1 ) is also F t − 1 \mathcal{F}_{t-1} F t − 1 -measurable. We may therefore take conditional expectations and use E [ ξ t ∣ F t − 1 ] = 0 \mathbb{E}[\xi_t\mid\mathcal{F}_{t-1}]=0 E [ ξ t ∣ F t − 1 ] = 0 a.s. to obtain
E [ ⟨ ∇ ϕ + ( S t − 1 ) , ξ t ⟩ ∣ F t − 1 ] = ⟨ ∇ ϕ + ( S t − 1 ) , E [ ξ t ∣ F t − 1 ] ⟩ = 0 a.s. \mathbb{E}\left[\langle\nabla\phi_+(S_{t-1}),\xi_t\rangle\mid\mathcal{F}_{t-1}\right]
=\left\langle\nabla\phi_+(S_{t-1}),\mathbb{E}[\xi_t\mid\mathcal{F}_{t-1}]\right\rangle
=0\quad\text{a.s.} E [ ⟨ ∇ ϕ + ( S t − 1 ) , ξ t ⟩ ∣ F t − 1 ] = ⟨ ∇ ϕ + ( S t − 1 ) , E [ ξ t ∣ F t − 1 ] ⟩ = 0 a.s.
Therefore,
E [ ϕ + ( S t ) ∣ F t − 1 ] ≤ ϕ + ( S t − 1 ) + κ + 2 E [ ∥ ξ t ∥ + 2 ∣ F t − 1 ] . \mathbb{E}\left[\phi_+(S_t)\mid\mathcal{F}_{t-1}\right]\leq\phi_+(S_{t-1})+\frac{\kappa_+}{2}\mathbb{E}\left[\lVert\xi_t\rVert_+^2\mid\mathcal{F}_{t-1}\right]. E [ ϕ + ( S t ) ∣ F t − 1 ] ≤ ϕ + ( S t − 1 ) + 2 κ + E [ ∥ ξ t ∥ + 2 ∣ F t − 1 ] .
Taking full expectation and summing over t = 1 , … , T t=1,\ldots,T t = 1 , … , T yields
E [ ϕ + ( S T ) ] ≤ κ + 2 ∑ t = 1 T E [ ∥ ξ t ∥ + 2 ] . (33) \mathbb{E}\left[\phi_+(S_T)\right]\leq\frac{\kappa_+}{2}\sum_{t=1}^{T}\mathbb{E}\left[\lVert\xi_t\rVert_+^2\right]. \tag*{(33)} E [ ϕ + ( S T ) ] ≤ 2 κ + t = 1 ∑ T E [ ∥ ξ t ∥ + 2 ] . (33)
Since ϕ + ( x ) = 1 2 ∥ x ∥ + 2 \phi_+(x)=\frac{1}{2}\lVert x\rVert_+^2 ϕ + ( x ) = 2 1 ∥ x ∥ + 2 , (33) is equivalent to
E [ ∥ S T ∥ + 2 ] ≤ κ + ∑ t = 1 T E [ ∥ ξ t ∥ + 2 ] . (34) \mathbb{E}\left[\lVert S_T\rVert_+^2\right]\leq\kappa_+\sum_{t=1}^{T}\mathbb{E}\left[\lVert\xi_t\rVert_+^2\right]. \tag*{(34)} E [ ∥ S T ∥ + 2 ] ≤ κ + t = 1 ∑ T E [ ∥ ξ t ∥ + 2 ] . (34)
Finally, using (31) and then (34) , we obtain
E [ ∥ S T ∥ 2 ] ≤ E [ ∥ S T ∥ + 2 ] ≤ κ + ∑ t = 1 T E [ ∥ ξ t ∥ + 2 ] ≤ κ + ∑ t = 1 T E [ κ κ + ∥ ξ t ∥ 2 ] = κ ∑ t = 1 T E [ ∥ ξ t ∥ 2 ] , \mathbb{E}\left[\lVert S_T\rVert^2\right]\leq\mathbb{E}\left[\lVert S_T\rVert_+^2\right]
\leq\kappa_+\sum_{t=1}^{T}\mathbb{E}\left[\lVert\xi_t\rVert_+^2\right]
\leq\kappa_+\sum_{t=1}^{T}\mathbb{E}\left[\frac{\kappa}{\kappa_+}\lVert\xi_t\rVert^2\right]
=\kappa\sum_{t=1}^{T}\mathbb{E}\left[\lVert\xi_t\rVert^2\right], E [ ∥ S T ∥ 2 ] ≤ E [ ∥ S T ∥ + 2 ] ≤ κ + t = 1 ∑ T E [ ∥ ξ t ∥ + 2 ] ≤ κ + t = 1 ∑ T E [ κ + κ ∥ ξ t ∥ 2 ] = κ t = 1 ∑ T E [ ∥ ξ t ∥ 2 ] ,
which is exactly (30) .
Sharp Operator for ℓ 1 \ell_1 ℓ 1 and ℓ ∞ \ell_\infty ℓ ∞ Update Norms
In this section, we specialize the abstract definition of the sharp operator s ♯ ∈ ∂ ϕ ∗ ( s ) s^\sharp\in\partial\phi^*(s) s ♯ ∈ ∂ ϕ ∗ ( s ) , where ϕ ∗ ( s ) : = 1 2 ∥ s ∥ ∗ 2 \phi^*(s) := \frac{1}{2}\lVert s\rVert_*^2 ϕ ∗ ( s ) := 2 1 ∥ s ∥ ∗ 2 , to the nonsmooth update norms ∥ ⋅ ∥ 1 \lVert\cdot\rVert_1 ∥ ⋅ ∥ 1 and ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∞ on R d \mathbb{R}^d R d . Throughout, we identify E = R d E = \mathbb{R}^d E = R d and E ∗ = R d E^* = \mathbb{R}^d E ∗ = R d via the standard pairing ⟨ s , x ⟩ = ∑ i = 1 d s i x i \langle s,x\rangle= \sum_{i=1}^{d}s_i x_i ⟨ s , x ⟩ = ∑ i = 1 d s i x i , and we write { e i } i = 1 d \{e_i\}_{i=1}^{d} { e i } i = 1 d for the canonical basis of R d \mathbb{R}^d R d .
ℓ 1 \ell_1 ℓ 1 Update Norm: Coordinate-Type Steps
Assume the update norm is ∥ ⋅ ∥ = ∥ ⋅ ∥ 1 \lVert\cdot\rVert= \lVert\cdot\rVert_1 ∥ ⋅ ∥ = ∥ ⋅ ∥ 1 , so the induced dual norm is ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_* = \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ ∞ . Then the squared dual-norm potential is ϕ ∗ ( s ) = 1 2 ∥ s ∥ ∞ 2 \phi^*(s) = \frac{1}{2}\lVert s\rVert_\infty^2 ϕ ∗ ( s ) = 2 1 ∥ s ∥ ∞ 2 . By Lemma 10, for every s ∈ R d s \in\mathbb{R}^d s ∈ R d ,
∂ ϕ ∗ ( s ) = ∥ s ∥ ∞ ⋅ ∂ ∥ s ∥ ∞ . (35) \partial\phi^*(s) = \lVert s\rVert_\infty\cdot\partial\lVert s\rVert_\infty. \tag*{(35)} ∂ ϕ ∗ ( s ) = ∥ s ∥ ∞ ⋅ ∂ ∥ s ∥ ∞ . (35)
An explicit description of ∂ ∥ s ∥ ∞ \partial\lVert s\rVert_\infty ∂ ∥ s ∥ ∞ (and hence of ∂ ϕ ∗ ( s ) \partial\phi^*(s) ∂ ϕ ∗ ( s ) ) follows from a standard result in convex analysis; for completeness, we state it below.
Proposition 1 (Subdifferential of ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∞ ). For s ∈ R d s \in\mathbb{R}^d s ∈ R d , define the active set
I ( s ) : = { i ∈ [ d ] : ∣ s i ∣ = ∥ s ∥ ∞ } . I(s) := \{i \in[d] : |s_i| = \lVert s\rVert_\infty\}. I ( s ) := { i ∈ [ d ] : ∣ s i ∣ = ∥ s ∥ ∞ } . If s = 0 s = 0 s = 0 , then ∂ ∥ s ∥ ∞ = { x ∈ R d : ∥ x ∥ 1 ≤ 1 } \partial\lVert s\rVert_\infty= \{x \in\mathbb{R}^d : \lVert x\rVert_1 \le1\} ∂ ∥ s ∥ ∞ = { x ∈ R d : ∥ x ∥ 1 ≤ 1 } . If s ≠ 0 s \ne0 s = 0 , then
∂ ∥ s ∥ ∞ = conv { sign ( s i ) e i : i ∈ I ( s ) } \partial\lVert s\rVert_\infty= \operatorname{conv}\{\operatorname{sign}(s_i)e_i : i \in I(s)\} ∂ ∥ s ∥ ∞ = conv { sign ( s i ) e i : i ∈ I ( s )} = { ∑ i ∈ I ( s ) α i sign ( s i ) e i : α i ≥ 0 , ∑ i ∈ I ( s ) α i = 1 } . (36) = \left\{\sum_{i\in I(s)}\alpha_i\operatorname{sign}(s_i)e_i : \alpha_i \ge0,\ \sum_{i\in I(s)}\alpha_i = 1\right\}. \tag*{(36)} = ⎩ ⎨ ⎧ i ∈ I ( s ) ∑ α i sign ( s i ) e i : α i ≥ 0 , i ∈ I ( s ) ∑ α i = 1 ⎭ ⎬ ⎫ . (36) Proof. See Beck [ [8 ] .
Corollary 2 (Sharp operator under ∥ ⋅ ∥ 1 \lVert\cdot\rVert_1 ∥ ⋅ ∥ 1 ). Assume ∥ ⋅ ∥ = ∥ ⋅ ∥ 1 \lVert\cdot\rVert= \lVert\cdot\rVert_1 ∥ ⋅ ∥ = ∥ ⋅ ∥ 1 and ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_* = \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ ∞ . For s ∈ R d s \in\mathbb{R}^d s ∈ R d with active set I ( s ) I(s) I ( s ) as above,
∂ ϕ ∗ ( s ) = ∥ s ∥ ∞ ∂ ∥ s ∥ ∞ = ∥ s ∥ ∞ ⋅ conv { sign ( s i ) e i : i ∈ I ( s ) } . (37) \partial\phi^*(s) = \lVert s\rVert_\infty\partial\lVert s\rVert_\infty= \lVert s\rVert_\infty\cdot\operatorname{conv}\{\operatorname{sign}(s_i)e_i : i \in I(s)\}. \tag*{(37)} ∂ ϕ ∗ ( s ) = ∥ s ∥ ∞ ∂ ∥ s ∥ ∞ = ∥ s ∥ ∞ ⋅ conv { sign ( s i ) e i : i ∈ I ( s )} . (37) In particular, if s ≠ 0 s \ne0 s = 0 , then every selection s ♯ ∈ ∂ ϕ ∗ ( s ) s^\sharp\in\partial\phi^*(s) s ♯ ∈ ∂ ϕ ∗ ( s ) satisfies:
(i) s i ♯ = 0 s_i^\sharp= 0 s i ♯ = 0 for all i ∉ I ( s ) i \notin I(s) i ∈ / I ( s ) (support contained in the maximizers of ∣ s i ∣ |s_i| ∣ s i ∣ ),
(ii) ∥ s ♯ ∥ 1 = ∥ s ∥ ∞ \lVert s^\sharp\rVert_1 = \lVert s\rVert_\infty ∥ s ♯ ∥ 1 = ∥ s ∥ ∞ ,
(iii) s i ♯ s i ≥ 0 s_i^\sharp s_i \ge0 s i ♯ s i ≥ 0 for all i i i (sign alignment on the active set).
Moreover, any extreme-point selection has the coordinate form
s ♯ = ∥ s ∥ ∞ sign ( s i ⋆ ) e i ⋆ , i ⋆ ∈ I ( s ) , (38) s^\sharp= \lVert s\rVert_\infty\operatorname{sign}(s_{i^\star})e_{i^\star},\qquad i^\star\in I(s), \tag*{(38)} s ♯ = ∥ s ∥ ∞ sign ( s i ⋆ ) e i ⋆ , i ⋆ ∈ I ( s ) , (38) and yields a single-coordinate update when substituted into (7) (with s = g k s = g^k s = g k ).
Proof. The identity (37) follows from (35) and Proposition 1. Properties (i)–(iii) follow immediately from the convex-hull representation in (37) . Finally, the extreme points of the simplex { α ≥ 0 : ∑ i ∈ I ( s ) α i = 1 } \{\alpha\ge0 : \sum_{i\in I(s)}\alpha_i = 1\} { α ≥ 0 : ∑ i ∈ I ( s ) α i = 1 } correspond to placing all mass on one index i ⋆ ∈ I ( s ) i^\star\in I(s) i ⋆ ∈ I ( s ) , which gives (38) .
ℓ ∞ \ell_\infty ℓ ∞ Update Norm: Sign-Type Steps
Assume the update norm is ∥ ⋅ ∥ = ∥ ⋅ ∥ ∞ \lVert\cdot\rVert= \lVert\cdot\rVert_\infty ∥ ⋅ ∥ = ∥ ⋅ ∥ ∞ , so the induced dual norm is ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 1 \lVert\cdot\rVert_* = \lVert\cdot\rVert_1 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 1 . Then ϕ ∗ ( s ) = 1 2 ∥ s ∥ 1 2 \phi^*(s) = \frac{1}{2}\lVert s\rVert_1^2 ϕ ∗ ( s ) = 2 1 ∥ s ∥ 1 2 and Lemma 10 yields
∂ ϕ ∗ ( s ) = ∥ s ∥ 1 ⋅ ∂ ∥ s ∥ 1 . (39) \partial\phi^*(s) = \lVert s\rVert_1 \cdot\partial\lVert s\rVert_1 . \tag*{(39)} ∂ ϕ ∗ ( s ) = ∥ s ∥ 1 ⋅ ∂ ∥ s ∥ 1 . (39)
Similarly, one can characterize ∂ ∥ s ∥ 1 \partial\lVert s\rVert_1 ∂ ∥ s ∥ 1 explicitly.
Proposition 2 (Subdifferential of ∥ ⋅ ∥ 1 \lVert\cdot\rVert_1 ∥ ⋅ ∥ 1 ). Let s ∈ R d s \in\mathbb{R}^d s ∈ R d . If s = 0 s=0 s = 0 , then ∂ ∥ s ∥ 1 = { x ∈ R d : ∥ x ∥ ∞ ≤ 1 } \partial\lVert s\rVert_1=\{x\in\mathbb{R}^d:\lVert x\rVert_\infty\le1\} ∂ ∥ s ∥ 1 = { x ∈ R d : ∥ x ∥ ∞ ≤ 1 } . If s ≠ 0 s\ne0 s = 0 , then
∂ ∥ s ∥ 1 = { x ∈ R d : x i = sign ( s i ) for all i with s i ≠ 0 , ∣ x i ∣ ≤ 1 for all i with s i = 0 } . (40) \partial\lVert s\rVert_1=\{x\in\mathbb{R}^d:x_i=\operatorname{sign}(s_i)\text{ for all }i\text{ with }s_i\ne0,\ |x_i|\le1\text{ for all }i\text{ with }s_i=0\}. \tag*{(40)} ∂ ∥ s ∥ 1 = { x ∈ R d : x i = sign ( s i ) for all i with s i = 0 , ∣ x i ∣ ≤ 1 for all i with s i = 0 } . (40) Equivalently, ∂ ∥ s ∥ 1 \partial\lVert s\rVert_1 ∂ ∥ s ∥ 1 consists of the componentwise sign vectors of s s s , with an arbitrary choice in [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] on coordinates where s i = 0 s_i=0 s i = 0 .
Proof. See Beck [ [8 ] .
Corollary 3 (Sharp operator under ∥ ⋅ ∥ ∞ \lVert\cdot\rVert_\infty ∥ ⋅ ∥ ∞ ). Assume ∥ ⋅ ∥ = ∥ ⋅ ∥ ∞ \lVert\cdot\rVert=\lVert\cdot\rVert_\infty ∥ ⋅ ∥ = ∥ ⋅ ∥ ∞ and ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 1 \lVert\cdot\rVert_*=\lVert\cdot\rVert_1 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 1 . Then for any s ∈ R d s\in\mathbb{R}^d s ∈ R d ,
∂ ϕ ∗ ( s ) = ∥ s ∥ 1 ⋅ ∂ ∥ s ∥ 1 . (41) \partial\phi^*(s)=\lVert s\rVert_1\cdot\partial\lVert s\rVert_1. \tag*{(41)} ∂ ϕ ∗ ( s ) = ∥ s ∥ 1 ⋅ ∂ ∥ s ∥ 1 . (41) In particular, if s ≠ 0 s\ne0 s = 0 then any selection s ♯ ∈ ∂ ϕ ∗ ( s ) s^\sharp\in\partial\phi^*(s) s ♯ ∈ ∂ ϕ ∗ ( s ) satisfies
s ♯ = ∥ s ∥ 1 ⋅ v , v i = sign ( s i ) for s i ≠ 0 , ∣ v i ∣ ≤ 1 for s i = 0. (42) s^\sharp=\lVert s\rVert_1\cdot v,\qquad v_i=\operatorname{sign}(s_i)\text{ for }s_i\ne0,\quad|v_i|\le1\text{ for }s_i=0. \tag*{(42)} s ♯ = ∥ s ∥ 1 ⋅ v , v i = sign ( s i ) for s i = 0 , ∣ v i ∣ ≤ 1 for s i = 0. (42) A canonical selection is v = sign ( s ) v=\operatorname{sign}(s) v = sign ( s ) , where sign \operatorname{sign} sign is applied componentwise with sign ( 0 ) = 0 \operatorname{sign}(0)=0 sign ( 0 ) = 0 . This yields the sign-type update
x k + 1 = x k − γ k ∥ g k ∥ 1 sign ( g k ) , x^{k+1}=x^k-\gamma_k\lVert g^k\rVert_1\operatorname{sign}(g^k), x k + 1 = x k − γ k ∥ g k ∥ 1 sign ( g k ) , closely related to signSGD-style directions [10 , 79 ] .
Proof. This follows immediately from (39) and Proposition 2.
Derivation of the Non-Euclidean SGD Update
In this section, we provide a mathematical justification for the non-Euclidean SGD update rule used in the main text:
x k + 1 = x k − γ k ( g k ) ♯ . (43) x^{k+1}=x^k-\gamma_k(g^k)^\sharp. \tag*{(43)} x k + 1 = x k − γ k ( g k ) ♯ . (43)
We show that the steepest-descent and Linear Minimization Oracle (LMO) formulations characterize the same set of admissible steps, from which the fixed sharp selection determines the update.
Derivation from Steepest Descent
The steepest descent method generates the next iterate x k + 1 = x k + h k x^{k+1}=x^k+h_k x k + 1 = x k + h k by minimizing a linearized model regularized by the squared norm. In minibatch SGD, the linear term uses the stochastic gradient g k g^k g k , and the stepsize γ k \gamma_k γ k sets the regularization weight 1 / ( 2 γ k ) 1/(2\gamma_k) 1/ ( 2 γ k ) . The step h k h_k h k is a solution to the following optimization problem:
h k ∈ arg min h ∈ E { ⟨ g k , h ⟩ + 1 2 γ k ∥ h ∥ 2 } . (44) h_k\in\mathop{\arg\min}_{h\in E}\left\{\langle g^k,h\rangle+\frac{1}{2\gamma_k}\lVert h\rVert^2\right\}. \tag*{(44)} h k ∈ arg min h ∈ E { ⟨ g k , h ⟩ + 2 γ k 1 ∥ h ∥ 2 } . (44)
where γ k > 0 \gamma_k>0 γ k > 0 and ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ is an arbitrary norm on the space E E E . The objective function in (44) is convex, and its first-order optimality condition states that the zero vector must be in the subdifferential of the objective at h k h_k h k . Letting ϕ ( h ) : = 1 2 ∥ h ∥ 2 \phi(h):=\frac{1}{2}\lVert h\rVert^2 ϕ ( h ) := 2 1 ∥ h ∥ 2 , this condition is:
0 ∈ g k + 1 γ k ∂ ϕ ( h k ) . 0\in g^k+\frac{1}{\gamma_k}\partial\phi(h_k). 0 ∈ g k + γ k 1 ∂ ϕ ( h k ) .
Rearranging this expression, we get:
− γ k g k ∈ ∂ ϕ ( h k ) . -\gamma_k g^k\in\partial\phi(h_k). − γ k g k ∈ ∂ ϕ ( h k ) .
This is a condition on the subdifferential of the primal squared norm ϕ \phi ϕ . We can invert this relationship using the Fenchel-Moreau-Rockafellar theorem (Theorem 4), which states that s ∈ ∂ ϕ ( x ) ⇔ x ∈ ∂ ϕ ∗ ( s ) s\in\partial\phi(x)\Leftrightarrow x\in\partial\phi^*(s) s ∈ ∂ ϕ ( x ) ⇔ x ∈ ∂ ϕ ∗ ( s ) . Applying this gives:
h k ∈ ∂ ϕ ∗ ( − γ k g k ) . h_k\in\partial\phi^*(-\gamma_k g^k). h k ∈ ∂ ϕ ∗ ( − γ k g k ) .
Using the homogeneity of the subdifferential of the squared norm (Lemma 11), which states that ∂ ϕ ∗ ( t s ) = t ∂ ϕ ∗ ( s ) \partial\phi^*(ts)=t\partial\phi^*(s) ∂ ϕ ∗ ( t s ) = t ∂ ϕ ∗ ( s ) for all t ∈ R t\in\mathbb{R} t ∈ R , we can factor out the stepsize γ k \gamma_k γ k :
h k ∈ − γ k ∂ ϕ ∗ ( g k ) . h_k\in-\gamma_k\partial\phi^*(g^k). h k ∈ − γ k ∂ ϕ ∗ ( g k ) .
All preceding conditions are equivalent, so the full set of minimizers is − γ k ∂ ϕ ∗ ( g k ) -\gamma_k\partial\phi^*(g^k) − γ k ∂ ϕ ∗ ( g k ) . For the fixed sharp selection from the main text, choose h k = − γ k ( g k ) ♯ h_k=-\gamma_k(g^k)^\sharp h k = − γ k ( g k ) ♯ ; this is a minimizer because ( g k ) ♯ ∈ ∂ ϕ ∗ ( g k ) (g^k)^\sharp\in\partial\phi^*(g^k) ( g k ) ♯ ∈ ∂ ϕ ∗ ( g k ) . Substituting this choice into x k + 1 = x k + h k x^{k+1}=x^k+h_k x k + 1 = x k + h k gives the specified non-Euclidean SGD update:
x k + 1 = x k − γ k ( g k ) ♯ . x^{k+1}=x^k-\gamma_k(g^k)^\sharp. x k + 1 = x k − γ k ( g k ) ♯ .
We now show that a Linear Minimization Oracle (LMO) gives the same set of admissible next iterates as the steepest-descent formulation. An LMO is an operator which, for a given set C ⊂ E \mathcal{C}\subset E C ⊂ E and a vector s ∈ E ∗ s\in E^* s ∈ E ∗ , solves the linear problem:
lmo C ( s ) : = arg min x ∈ C ⟨ s , x ⟩ . \operatorname{lmo}_{\mathcal{C}}(s):=\mathop{\arg\min}_{x\in\mathcal{C}}\langle s,x\rangle. lmo C ( s ) := arg min x ∈ C ⟨ s , x ⟩ .
Our derivation will rely on two key properties, which we first establish as lemmas.
Lemma 16 (Characterization of the Sharp Operator via LMO). For any non-zero s ∈ E ∗ s\in E^* s ∈ E ∗ , the subdifferential ∂ ϕ ∗ ( s ) \partial\phi^*(s) ∂ ϕ ∗ ( s ) can be expressed in terms of an LMO over the unit ball B ( 0 , 1 ) = { x ∈ E ∣ ∥ x ∥ ≤ 1 } \mathcal{B}(0,1)=\{x\in E\mid\lVert x\rVert\le1\} B ( 0 , 1 ) = { x ∈ E ∣ ∥ x ∥ ≤ 1 } as follows:
∂ ϕ ∗ ( s ) = − ∥ s ∥ ∗ ⋅ lmo B ( 0 , 1 ) ( s ) . \partial\phi^*(s)=-\lVert s\rVert_*\cdot\operatorname{lmo}_{\mathcal{B}(0,1)}(s). ∂ ϕ ∗ ( s ) = − ∥ s ∥ ∗ ⋅ lmo B ( 0 , 1 ) ( s ) .
Proof. From Equation (29) and the variational characterization of the subdifferential of the norm from Lemma 7, we have for any s ≠ 0 s \ne0 s = 0 :
∂ φ ∗ ( s ) = ∥ s ∥ ∗ ⋅ ∂ ∥ s ∥ ∗ = ∥ s ∥ ∗ ⋅ arg max x ∈ E { ⟨ s , x ⟩ ∣ ∥ x ∥ ≤ 1 } = − ∥ s ∥ ∗ ⋅ arg min x ∈ E { ⟨ s , x ⟩ ∣ ∥ x ∥ ≤ 1 } = − ∥ s ∥ ∗ ⋅ lmo B ( 0 , 1 ) ( s ) . \partial\varphi^{*}(s) = \lVert s\rVert_{*} \cdot\partial\lVert s\rVert_{*}
= \lVert s\rVert_{*} \cdot\underset{x \in E}{\operatorname{arg\,max}}\{\langle s,x\rangle\mid\lVert x\rVert\le1\}
= -\lVert s\rVert_{*} \cdot\underset{x \in E}{\operatorname{arg\,min}}\{\langle s,x\rangle\mid\lVert x\rVert\le1\}
= -\lVert s\rVert_{*} \cdot\operatorname{lmo}_{\mathcal{B}(0,1)}(s). ∂ φ ∗ ( s ) = ∥ s ∥ ∗ ⋅ ∂ ∥ s ∥ ∗ = ∥ s ∥ ∗ ⋅ x ∈ E arg max {⟨ s , x ⟩ ∣ ∥ x ∥ ≤ 1 } = − ∥ s ∥ ∗ ⋅ x ∈ E arg min {⟨ s , x ⟩ ∣ ∥ x ∥ ≤ 1 } = − ∥ s ∥ ∗ ⋅ lmo B ( 0 , 1 ) ( s ) .
This completes the proof.
Lemma 17 (Translation Property of the LMO). Let B ( y , t ) = { x ∈ E ∣ ∥ x − y ∥ ≤ t } \mathcal{B}(y,t)=\{x\in E\mid\lVert x-y\rVert\le t\} B ( y , t ) = { x ∈ E ∣ ∥ x − y ∥ ≤ t } be a ball of radius t > 0 t>0 t > 0 centered at y ∈ E y\in E y ∈ E . The LMO over this ball can be expressed in terms of the LMO over the unit ball B ( 0 , 1 ) \mathcal{B}(0,1) B ( 0 , 1 ) as:
lmo B ( y , t ) ( s ) = y + t ⋅ lmo B ( 0 , 1 ) ( s ) . \operatorname{lmo}_{\mathcal{B}(y,t)}(s)=y+t\cdot\operatorname{lmo}_{\mathcal{B}(0,1)}(s). lmo B ( y , t ) ( s ) = y + t ⋅ lmo B ( 0 , 1 ) ( s ) . Proof. The proof follows from a change of variables. The LMO problem is to find a minimizer x ∗ x^{*} x ∗ for ⟨ s , x ⟩ \langle s,x\rangle ⟨ s , x ⟩ over the set B ( y , t ) \mathcal{B}(y,t) B ( y , t ) .
lmo B ( y , t ) ( s ) = arg min x ∈ E : ∥ x − y ∥ ≤ t ⟨ s , x ⟩ = y + arg min x ′ ∈ E : ∥ x ′ ∥ ≤ t ⟨ s , y + x ′ ⟩ (letting x ′ = x − y ) = y + arg min x ′ ∈ E : ∥ x ′ ∥ ≤ t ⟨ s , x ′ ⟩ = y + t ⋅ arg min x ′ ′ ∈ E : ∥ x ′ ′ ∥ ≤ 1 ⟨ s , t x ′ ′ ⟩ (letting x ′ = t x ′ ′ ) = y + t ⋅ arg min x ′ ′ ∈ E : ∥ x ′ ′ ∥ ≤ 1 ⟨ s , x ′ ′ ⟩ = y + t ⋅ lmo B ( 0 , 1 ) ( s ) . \begin{aligned}
\operatorname{lmo}_{\mathcal{B}(y,t)}(s)
&=\underset{x\in E:\lVert x-y\rVert\le t}{\operatorname{arg\,min}}\ \langle s,x\rangle\\
&=y+\underset{x'\in E:\lVert x'\rVert\le t}{\operatorname{arg\,min}}\ \langle s,y+x'\rangle\qquad\text{(letting }x'=x-y\text{)}\\
&=y+\underset{x'\in E:\lVert x'\rVert\le t}{\operatorname{arg\,min}}\ \langle s,x'\rangle\\
&=y+t\cdot\underset{x''\in E:\lVert x''\rVert\le1}{\operatorname{arg\,min}}\ \langle s,tx''\rangle\qquad\text{(letting }x'=tx''\text{)}\\
&=y+t\cdot\underset{x''\in E:\lVert x''\rVert\le1}{\operatorname{arg\,min}}\ \langle s,x''\rangle\\
&=y+t\cdot\operatorname{lmo}_{\mathcal{B}(0,1)}(s).
\end{aligned} lmo B ( y , t ) ( s ) = x ∈ E : ∥ x − y ∥ ≤ t arg min ⟨ s , x ⟩ = y + x ′ ∈ E : ∥ x ′ ∥ ≤ t arg min ⟨ s , y + x ′ ⟩ (letting x ′ = x − y ) = y + x ′ ∈ E : ∥ x ′ ∥ ≤ t arg min ⟨ s , x ′ ⟩ = y + t ⋅ x ′′ ∈ E : ∥ x ′′ ∥ ≤ 1 arg min ⟨ s , t x ′′ ⟩ (letting x ′ = t x ′′ ) = y + t ⋅ x ′′ ∈ E : ∥ x ′′ ∥ ≤ 1 arg min ⟨ s , x ′′ ⟩ = y + t ⋅ lmo B ( 0 , 1 ) ( s ) . This completes the derivation.
For g k ≠ 0 g^{k}\ne0 g k = 0 , define t k : = γ k ∥ g k ∥ ∗ > 0 t_{k}:=\gamma_{k}\lVert g^{k}\rVert_{*}>0 t k := γ k ∥ g k ∥ ∗ > 0 . Applying Lemma 17 and then Lemma 16 gives the equality of solution sets:
lmo B ( x k , t k ) ( g k ) = x k + t k ⋅ lmo B ( 0 , 1 ) ( g k ) = x k − γ k ∂ φ ∗ ( g k ) . \operatorname{lmo}_{\mathcal{B}(x^{k},t_{k})}(g^{k})
=x^{k}+t_{k}\cdot\operatorname{lmo}_{\mathcal{B}(0,1)}(g^{k})
=x^{k}-\gamma_{k}\partial\varphi^{*}(g^{k}). lmo B ( x k , t k ) ( g k ) = x k + t k ⋅ lmo B ( 0 , 1 ) ( g k ) = x k − γ k ∂ φ ∗ ( g k ) .
The right-hand side is exactly the set of next iterates obtained from all steepest-descent minimizers. Thus the LMO formulation is
x k + 1 ∈ lmo B ( x k , t k ) ( g k ) , where t k = γ k ∥ g k ∥ ∗ . (45) x^{k+1}\in\operatorname{lmo}_{\mathcal{B}(x^{k},t_{k})}(g^{k}),\qquad\text{where }t_{k}=\gamma_{k}\lVert g^{k}\rVert_{*}. \tag*{(45)} x k + 1 ∈ lmo B ( x k , t k ) ( g k ) , where t k = γ k ∥ g k ∥ ∗ . (45)
To reproduce the algorithm with the fixed sharp selection, choose the LMO minimizer x k + 1 = x k − γ k ( g k ) ♯ x^{k+1}=x^{k}-\gamma_{k}(g^{k})^{\sharp} x k + 1 = x k − γ k ( g k ) ♯ . The equality of solution sets ensures that this choice is valid; an arbitrary different minimizer may produce a different admissible step. The radius of the ball, t k t_{k} t k , depends on the stepsize and the dual norm of the stochastic gradient.
We note that if g k = 0 g^{k}=0 g k = 0 , then t k = 0 t_{k}=0 t k = 0 and both formulations reduce to the identity update x k + 1 = x k x^{k+1}=x^{k} x k + 1 = x k : in this case the ball B ( x k , 0 ) \mathcal{B}(x^{k},0) B ( x k , 0 ) is the singleton { x k } \{x^{k}\} { x k } , and the LMO formulation still coincides with the steepest-descent solution.
Derivation of the Orthogonalized Gradient Update
In this section, we derive the spectral-norm update (7) for the canonical sharp selection and characterize the full family of updates allowed by the LMO formulation.
The Non-Euclidean Update with the Spectral Norm
The non-Euclidean SGD update is given by
X k + 1 = X k − γ k ( G k ) ♯ , X^{k+1}=X^k-\gamma_k(G^k)^{\sharp}, X k + 1 = X k − γ k ( G k ) ♯ ,
where ( G k ) ♯ ∈ ∂ ϕ ∗ ( G k ) (G^k)^{\sharp}\in\partial\phi^*(G^k) ( G k ) ♯ ∈ ∂ ϕ ∗ ( G k ) . In our setting, the primal space is the space of matrices equipped with the spectral norm, ∥ ⋅ ∥ = ∥ ⋅ ∥ o p \|\cdot\|=\|\cdot\|_{\mathrm{op}} ∥ ⋅ ∥ = ∥ ⋅ ∥ op . The corresponding dual norm is the nuclear norm, ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ n u c \|\cdot\|_*=\|\cdot\|_{\mathrm{nuc}} ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ nuc . The function ϕ ∗ \phi^* ϕ ∗ is therefore the squared nuclear norm:
ϕ ∗ ( G ) = 1 2 ∥ G ∥ n u c 2 . \phi^*(G)=\frac{1}{2}\|G\|_{\mathrm{nuc}}^2. ϕ ∗ ( G ) = 2 1 ∥ G ∥ nuc 2 .
To characterize the sharp operator ( G k ) ♯ (G^k)^{\sharp} ( G k ) ♯ , we must find the subdifferential of ϕ ∗ \phi^* ϕ ∗ . By Lemma 10, we have ∂ ϕ ∗ ( G ) = ∥ G ∥ n u c ⋅ ∂ ∥ G ∥ n u c \partial\phi^*(G)=\|G\|_{\mathrm{nuc}}\cdot\partial\|G\|_{\mathrm{nuc}} ∂ ϕ ∗ ( G ) = ∥ G ∥ nuc ⋅ ∂ ∥ G ∥ nuc . We therefore first characterize the subdifferential of the nuclear norm.
Lemma 18 (Subdifferential of the Nuclear Norm). Let G ∈ R m × n G\in\mathbb{R}^{m\times n} G ∈ R m × n have thin singular value decomposition G = U r Σ r V r ⊤ G=U_r\Sigma_rV_r^\top G = U r Σ r V r ⊤ , where r = rank ( G ) r=\operatorname{rank}(G) r = rank ( G ) , U r ∈ R m × r U_r\in\mathbb{R}^{m\times r} U r ∈ R m × r , V r ∈ R n × r V_r\in\mathbb{R}^{n\times r} V r ∈ R n × r , and Σ r = diag ( σ 1 , … , σ r ) ∈ R r × r \Sigma_r=\operatorname{diag}(\sigma_1,\ldots,\sigma_r)\in\mathbb{R}^{r\times r} Σ r = diag ( σ 1 , … , σ r ) ∈ R r × r with σ i > 0 \sigma_i>0 σ i > 0 . The subdifferential of the nuclear norm ∥ ⋅ ∥ n u c \|\cdot\|_{\mathrm{nuc}} ∥ ⋅ ∥ nuc at G G G is given by
∂ ∥ G ∥ n u c = { U r V r ⊤ + W ∣ U r ⊤ W = 0 , W V r = 0 , and ∥ W ∥ o p ≤ 1 } . \partial\|G\|_{\mathrm{nuc}}=\left\{U_rV_r^\top+W\mid U_r^\top W=0,\ WV_r=0,\ \text{and }\|W\|_{\mathrm{op}}\leq1\right\}. ∂ ∥ G ∥ nuc = { U r V r ⊤ + W ∣ U r ⊤ W = 0 , W V r = 0 , and ∥ W ∥ op ≤ 1 } . A canonical and valid selection for an element of the subdifferential is the partial isometry U r V r ⊤ U_rV_r^\top U r V r ⊤ .
Proof. We will derive this result by applying the characterization of the subdifferential of the dual norm from Lemma 7.
We apply the general norm characterization with the following identification:
The primal norm is the spectral norm: ∥ ⋅ ∥ ≡ ∥ ⋅ ∥ o p \|\cdot\|\equiv\|\cdot\|_{\mathrm{op}} ∥ ⋅ ∥ ≡ ∥ ⋅ ∥ op .
The corresponding dual norm is the nuclear norm: ∥ ⋅ ∥ ∗ ≡ ∥ ⋅ ∥ n u c \|\cdot\|_*\equiv\|\cdot\|_{\mathrm{nuc}} ∥ ⋅ ∥ ∗ ≡ ∥ ⋅ ∥ nuc .
We consider the two cases from (21) .
Case 1: G = 0 G=0 G = 0 . According to (21) , the subdifferential of the dual norm at the origin is the unit ball of the primal norm. In our case, this is the unit ball of the spectral norm:
∂ ∥ 0 ∥ n u c = { S ∈ R m × n ∣ ∥ S ∥ o p ≤ 1 } . \partial\|0\|_{\mathrm{nuc}}=\left\{S\in\mathbb{R}^{m\times n}\mid\|S\|_{\mathrm{op}}\leq1\right\}. ∂ ∥0 ∥ nuc = { S ∈ R m × n ∣ ∥ S ∥ op ≤ 1 } .
This matches the general formula: when G = 0 G=0 G = 0 , we have r = 0 r=0 r = 0 , and the empty factors U 0 ∈ R m × 0 U_0\in\mathbb{R}^{m\times0} U 0 ∈ R m × 0 and V 0 ∈ R n × 0 V_0\in\mathbb{R}^{n\times0} V 0 ∈ R n × 0 satisfy U 0 V 0 ⊤ = 0 U_0V_0^\top=0 U 0 V 0 ⊤ = 0 . The constraints U 0 ⊤ W = 0 U_0^\top W=0 U 0 ⊤ W = 0 and W V 0 = 0 WV_0=0 W V 0 = 0 are then vacuous, so the formula reduces to { W ∈ R m × n ∣ ∥ W ∥ o p ≤ 1 } \left\{W\in\mathbb{R}^{m\times n}\mid\|W\|_{\mathrm{op}}\leq1\right\} { W ∈ R m × n ∣ ∥ W ∥ op ≤ 1 } .
Case 2: G ≠ 0 G\neq0 G = 0 . According to (21) , an element S ∈ R m × n S\in\mathbb{R}^{m\times n} S ∈ R m × n belongs to the subdifferential ∂ ∥ G ∥ n u c \partial\|G\|_{\mathrm{nuc}} ∂ ∥ G ∥ nuc if and only if it satisfies two conditions:
⟨ S , G ⟩ = ∥ G ∥ n u c \langle S,G\rangle=\|G\|_{\mathrm{nuc}} ⟨ S , G ⟩ = ∥ G ∥ nuc
∥ S ∥ o p = 1 \|S\|_{\mathrm{op}}=1 ∥ S ∥ op = 1
Let u i u_i u i and v i v_i v i be the columns of the fixed thin SVD factors U r U_r U r and V r V_r V r . Suppose first that S ∈ ∂ ∥ G ∥ n u c S\in\partial\|G\|_{\mathrm{nuc}} S ∈ ∂ ∥ G ∥ nuc . The two conditions above give
∑ i = 1 r σ i u i ⊤ S v i = ⟨ S , G ⟩ F = ∥ G ∥ n u c = ∑ i = 1 r σ i . \sum_{i=1}^{r}\sigma_i u_i^\top S v_i=\langle S,G\rangle_{\mathrm{F}}=\|G\|_{\mathrm{nuc}}=\sum_{i=1}^{r}\sigma_i. i = 1 ∑ r σ i u i ⊤ S v i = ⟨ S , G ⟩ F = ∥ G ∥ nuc = i = 1 ∑ r σ i .
Each u i T S v i ≤ ∥ S v i ∥ 2 ≤ 1 u_i^{\mathsf{T}}Sv_i \le\lVert Sv_i\rVert_2 \le1 u i T S v i ≤ ∥ S v i ∥ 2 ≤ 1 , and all σ i \sigma_i σ i are positive, so u i T S v i = 1 u_i^{\mathsf{T}}Sv_i=1 u i T S v i = 1 for every i ≤ r i\le r i ≤ r . Since ∥ S T ∥ o p = ∥ S ∥ o p = 1 \lVert S^{\mathsf{T}}\rVert_{\mathrm{op}}=\lVert S\rVert_{\mathrm{op}}=1 ∥ S T ∥ op = ∥ S ∥ op = 1 , equality in the Euclidean Cauchy–Schwarz inequality, applied to S v i Sv_i S v i and to S T u i S^{\mathsf{T}}u_i S T u i , yields
S v i = u i , S T u i = v i . Sv_i=u_i,\qquad S^{\mathsf{T}}u_i=v_i. S v i = u i , S T u i = v i .
Thus W : = S − U r V r T W:=S-U_rV_r^{\mathsf{T}} W := S − U r V r T satisfies U r T W = 0 U_r^{\mathsf{T}}W=0 U r T W = 0 and W V r = 0 WV_r=0 W V r = 0 . Moreover,
W = ( I m − U r U r T ) S ( I n − V r V r T ) , W=(I_m-U_rU_r^{\mathsf{T}})S(I_n-V_rV_r^{\mathsf{T}}), W = ( I m − U r U r T ) S ( I n − V r V r T ) ,
so ∥ W ∥ o p ≤ ∥ S ∥ o p = 1 \lVert W\rVert_{\mathrm{op}}\le\lVert S\rVert_{\mathrm{op}}=1 ∥ W ∥ op ≤ ∥ S ∥ op = 1 , because both orthogonal projections have spectral norm at most one. This proves that every subgradient has the stated form.
Conversely, let W W W satisfy the three conditions in the lemma and set S = U r V r T + W S=U_rV_r^{\mathsf{T}}+W S = U r V r T + W . For any z ∈ R n z\in\mathbb{R}^n z ∈ R n , write z = V r a + b z=V_ra+b z = V r a + b with V r T b = 0 V_r^{\mathsf{T}}b=0 V r T b = 0 . Then S z = U r a + W b Sz=U_ra+Wb S z = U r a + W b , and the two terms are orthogonal because U r T W = 0 U_r^{\mathsf{T}}W=0 U r T W = 0 . Hence
∥ S z ∥ 2 2 = ∥ a ∥ 2 2 + ∥ W b ∥ 2 2 ≤ ∥ a ∥ 2 2 + ∥ b ∥ 2 2 = ∥ z ∥ 2 2 . \lVert Sz\rVert_2^2=\lVert a\rVert_2^2+\lVert Wb\rVert_2^2\le\lVert a\rVert_2^2+\lVert b\rVert_2^2=\lVert z\rVert_2^2. ∥ S z ∥ 2 2 = ∥ a ∥ 2 2 + ∥ W b ∥ 2 2 ≤ ∥ a ∥ 2 2 + ∥ b ∥ 2 2 = ∥ z ∥ 2 2 .
Therefore ∥ S ∥ o p ≤ 1 \lVert S\rVert_{\mathrm{op}}\le1 ∥ S ∥ op ≤ 1 . Since r ≥ 1 r\ge1 r ≥ 1 and S v i = u i Sv_i=u_i S v i = u i for every i ≤ r i\le r i ≤ r , we also have ∥ S ∥ o p ≥ 1 \lVert S\rVert_{\mathrm{op}}\ge1 ∥ S ∥ op ≥ 1 and
⟨ S , G ⟩ F = ∑ i = 1 r σ i u i T S v i = ∑ i = 1 r σ i = ∥ G ∥ n u c . \langle S,G\rangle_{\mathrm{F}}=\sum_{i=1}^{r}\sigma_i u_i^{\mathsf{T}}Sv_i=\sum_{i=1}^{r}\sigma_i=\lVert G\rVert_{\mathrm{nuc}}. ⟨ S , G ⟩ F = i = 1 ∑ r σ i u i T S v i = i = 1 ∑ r σ i = ∥ G ∥ nuc .
Both conditions from (21) hold, so S ∈ ∂ ∥ G ∥ n u c S\in\partial\lVert G\rVert_{\mathrm{nuc}} S ∈ ∂ ∥ G ∥ nuc . This proves the reverse inclusion.
Choosing W = 0 W=0 W = 0 yields the canonical element S = U r V r T S=U_rV_r^{\mathsf{T}} S = U r V r T .
With this result, we can characterize the sharp operator at iteration k k k . Let G k G^k G k have thin singular value decomposition
G k = U r k Σ r k ( V r k ) T . G^k=U_r^k\Sigma_r^k(V_r^k)^{\mathsf{T}}. G k = U r k Σ r k ( V r k ) T .
where r = rank ( G k ) r=\operatorname{rank}(G^k) r = rank ( G k ) . By Lemma 18, a canonical choice of subgradient of the nuclear norm is the partial isometry
O k : = U r k ( V r k ) T ∈ ∂ ∥ G k ∥ n u c . O^k:=U_r^k(V_r^k)^{\mathsf{T}}\in\partial\lVert G^k\rVert_{\mathrm{nuc}}. O k := U r k ( V r k ) T ∈ ∂ ∥ G k ∥ nuc .
Using the chain rule for subdifferentials, we have
( G k ) ♯ ∈ ∂ φ ∗ ( G k ) = ∥ G k ∥ n u c ⋅ ∂ ∥ G k ∥ n u c , (G^k)^\sharp\in\partial\varphi^*(G^k)=\lVert G^k\rVert_{\mathrm{nuc}}\cdot\partial\lVert G^k\rVert_{\mathrm{nuc}}, ( G k ) ♯ ∈ ∂ φ ∗ ( G k ) = ∥ G k ∥ nuc ⋅ ∂ ∥ G k ∥ nuc ,
and we select
( G k ) ♯ : = ∥ G k ∥ n u c ⋅ O k . (G^k)^\sharp:=\lVert G^k\rVert_{\mathrm{nuc}}\cdot O^k. ( G k ) ♯ := ∥ G k ∥ nuc ⋅ O k .
Substituting this back into the SGD update rule gives
X k + 1 = X k − γ k ∥ G k ∥ n u c O k . X^{k+1}=X^k-\gamma_k\lVert G^k\rVert_{\mathrm{nuc}}O^k. X k + 1 = X k − γ k ∥ G k ∥ nuc O k .
For G k ≠ 0 G^k\ne0 G k = 0 , the direction O k O^k O k has spectral norm one. If we define a state-dependent stepsize η k : = γ k ∥ G k ∥ n u c \eta_k:=\gamma_k\lVert G^k\rVert_{\mathrm{nuc}} η k := γ k ∥ G k ∥ nuc , the update becomes
X k + 1 = X k − η k O k . X^{k+1}=X^k-\eta_kO^k. X k + 1 = X k − η k O k .
This is the canonical orthogonalized update used in § 3.1 .
Equivalence of Orthogonalization Definitions
We now prove that the partial isometry U r V r T U_rV_r^{\mathsf{T}} U r V r T obtained from the thin SVD agrees with the explicit formula given in (10) .
Lemma 19. Let G ∈ R m × n G\in\mathbb{R}^{m\times n} G ∈ R m × n have thin singular value decomposition G = U r Σ r V r T G=U_r\Sigma_rV_r^{\mathsf{T}} G = U r Σ r V r T , where r = rank ( G ) r=\operatorname{rank}(G) r = rank ( G ) , U r ∈ R m × r U_r\in\mathbb{R}^{m\times r} U r ∈ R m × r , V r ∈ R n × r V_r\in\mathbb{R}^{n\times r} V r ∈ R n × r , and Σ r = diag ( σ 1 , … , σ r ) ∈ R r × r \Sigma_r=\operatorname{diag}(\sigma_1,\ldots,\sigma_r)\in\mathbb{R}^{r\times r} Σ r = diag ( σ 1 , … , σ r ) ∈ R r × r with σ i > 0 \sigma_i>0 σ i > 0 . Then the orthogonalization function yields the partial isometry:
orth ( G ) = ( G G T ) † / 2 G = U r V r T . \operatorname{orth}(G)=(GG^{\mathsf{T}})^{\dagger/2}G=U_rV_r^{\mathsf{T}}. orth ( G ) = ( G G T ) † /2 G = U r V r T .
Proof. If G = 0 \mathbf{G}=0 G = 0 , both sides are zero, with the empty product U 0 V 0 T \mathbf{U}_0\mathbf{V}_0^{\mathsf{T}} U 0 V 0 T interpreted as the zero matrix. Assume henceforth that G ≠ 0 \mathbf{G}\ne0 G = 0 . Using the thin SVD, we have
G G T = ( U r Σ r V r T ) ( U r Σ r V r T ) T = ( U r Σ r V r T ) ( V r Σ r T U r T ) = U r Σ r 2 U r T , \mathbf{G}\mathbf{G}^{\mathsf{T}}=(\mathbf{U}_r\mathbf{\Sigma}_r\mathbf{V}_r^{\mathsf{T}})(\mathbf{U}_r\mathbf{\Sigma}_r\mathbf{V}_r^{\mathsf{T}})^{\mathsf{T}}
=(\mathbf{U}_r\mathbf{\Sigma}_r\mathbf{V}_r^{\mathsf{T}})(\mathbf{V}_r\mathbf{\Sigma}_r^{\mathsf{T}}\mathbf{U}_r^{\mathsf{T}})=\mathbf{U}_r\mathbf{\Sigma}_r^2\mathbf{U}_r^{\mathsf{T}}, G G T = ( U r Σ r V r T ) ( U r Σ r V r T ) T = ( U r Σ r V r T ) ( V r Σ r T U r T ) = U r Σ r 2 U r T ,
where Σ r 2 = diag ( σ 1 2 , … , σ r 2 ) \mathbf{\Sigma}_r^2=\operatorname{diag}(\sigma_1^2,\ldots,\sigma_r^2) Σ r 2 = diag ( σ 1 2 , … , σ r 2 ) . The Moore–Penrose pseudoinverse is then
( G G T ) † = U r Σ r − 2 U r T , (\mathbf{G}\mathbf{G}^{\mathsf{T}})^\dagger=\mathbf{U}_r\mathbf{\Sigma}_r^{-2}\mathbf{U}_r^{\mathsf{T}}, ( G G T ) † = U r Σ r − 2 U r T ,
where Σ r − 2 = diag ( σ 1 − 2 , … , σ r − 2 ) \mathbf{\Sigma}_r^{-2}=\operatorname{diag}(\sigma_1^{-2},\ldots,\sigma_r^{-2}) Σ r − 2 = diag ( σ 1 − 2 , … , σ r − 2 ) . Since ( G G T ) † (\mathbf{G}\mathbf{G}^{\mathsf{T}})^\dagger ( G G T ) † is positive semidefinite, its unique positive semidefinite square root is
( G G T ) † / 2 = U r Σ r − 1 U r T , (\mathbf{G}\mathbf{G}^{\mathsf{T}})^{\dagger/2}=\mathbf{U}_r\mathbf{\Sigma}_r^{-1}\mathbf{U}_r^{\mathsf{T}}, ( G G T ) † /2 = U r Σ r − 1 U r T ,
with Σ r − 1 = diag ( σ 1 − 1 , … , σ r − 1 ) \mathbf{\Sigma}_r^{-1}=\operatorname{diag}(\sigma_1^{-1},\ldots,\sigma_r^{-1}) Σ r − 1 = diag ( σ 1 − 1 , … , σ r − 1 ) . Multiplying by G \mathbf{G} G gives
orth ( G ) = ( G G T ) † / 2 G = U r Σ r − 1 U r T ( U r Σ r V r T ) = U r Σ r − 1 ( U r T U r ) Σ r V r T = U r ( Σ r − 1 Σ r ) V r T = U r V r T , \begin{aligned}
\operatorname{orth}(\mathbf{G})&=(\mathbf{G}\mathbf{G}^{\mathsf{T}})^{\dagger/2}\mathbf{G}\\
&=\mathbf{U}_r\mathbf{\Sigma}_r^{-1}\mathbf{U}_r^{\mathsf{T}}(\mathbf{U}_r\mathbf{\Sigma}_r\mathbf{V}_r^{\mathsf{T}})\\
&=\mathbf{U}_r\mathbf{\Sigma}_r^{-1}(\mathbf{U}_r^{\mathsf{T}}\mathbf{U}_r)\mathbf{\Sigma}_r\mathbf{V}_r^{\mathsf{T}}\\
&=\mathbf{U}_r(\mathbf{\Sigma}_r^{-1}\mathbf{\Sigma}_r)\mathbf{V}_r^{\mathsf{T}}=\mathbf{U}_r\mathbf{V}_r^{\mathsf{T}},
\end{aligned} orth ( G ) = ( G G T ) † /2 G = U r Σ r − 1 U r T ( U r Σ r V r T ) = U r Σ r − 1 ( U r T U r ) Σ r V r T = U r ( Σ r − 1 Σ r ) V r T = U r V r T ,
since Σ r − 1 Σ r = I r \mathbf{\Sigma}_r^{-1}\mathbf{\Sigma}_r=\mathbf{I}_r Σ r − 1 Σ r = I r . This proves the claim.
LMO and the Canonical Orthogonalized Update for the Spectral Norm
We characterize all solutions of the spectral-norm LMO and recover the canonical orthogonalized update by selecting the negative partial isometry.
The LMO formulation of the update is:
X k + 1 ∈ lmo B ( X k , t k ) ( G k ) . \mathbf{X}^{k+1}\in\operatorname{lmo}_{\mathcal{B}(\mathbf{X}^k,t_k)}(\mathbf{G}^k). X k + 1 ∈ lmo B ( X k , t k ) ( G k ) .
where the geometry is defined by the spectral norm, ∥ ⋅ ∥ = ∥ ⋅ ∥ o p \lVert\cdot\rVert=\lVert\cdot\rVert_{\mathrm{op}} ∥ ⋅ ∥ = ∥ ⋅ ∥ op . Consequently, the dual norm is the nuclear norm, ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ n u c \lVert\cdot\rVert_*=\lVert\cdot\rVert_{\mathrm{nuc}} ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ nuc , and the radius of the ball is t k = γ k ∥ G k ∥ n u c t_k=\gamma_k\lVert\mathbf{G}^k\rVert_{\mathrm{nuc}} t k = γ k ∥ G k ∥ nuc with γ k > 0 \gamma_k>0 γ k > 0 . We first consider G k ≠ 0 \mathbf{G}^k\ne0 G k = 0 , so that t k > 0 t_k>0 t k > 0 . The LMO problem is to solve:
X k + 1 ∈ arg min X ∈ R m × n { ⟨ G k , X ⟩ F | ∥ X − X k ∥ o p ≤ t k } . \mathbf{X}^{k+1}\in\underset{\mathbf{X}\in\mathbb{R}^{m\times n}}{\arg\min}\left\{\langle\mathbf{G}^k,\mathbf{X}\rangle_{\mathrm{F}}\ \middle|\ \lVert\mathbf{X}-\mathbf{X}^k\rVert_{\mathrm{op}}\le t_k\right\}. X k + 1 ∈ X ∈ R m × n arg min { ⟨ G k , X ⟩ F ∥ X − X k ∥ op ≤ t k } .
Using the translation property of the LMO (Lemma 17 ), we can simplify this problem:
X k + 1 ∈ X k + t k ⋅ lmo B ( 0 , 1 ) ( G k ) , \mathbf{X}^{k+1}\in\mathbf{X}^k+t_k\cdot\operatorname{lmo}_{B(0,1)}(\mathbf{G}^k), X k + 1 ∈ X k + t k ⋅ lmo B ( 0 , 1 ) ( G k ) ,
where B ( 0 , 1 ) B(0,1) B ( 0 , 1 ) is the unit ball of the spectral norm, i.e., B ( 0 , 1 ) = { X ∈ R m × n ∣ ∥ X ∥ o p ≤ 1 } B(0,1)=\{\mathbf{X}\in\mathbb{R}^{m\times n}\mid\lVert\mathbf{X}\rVert_{\mathrm{op}}\le1\} B ( 0 , 1 ) = { X ∈ R m × n ∣ ∥ X ∥ op ≤ 1 } . The following lemma describes the full set of minimizers over this ball.
Lemma 20 (LMO over the Spectral Norm Ball). Let G ∈ R m × n \mathbf{G}\in\mathbb{R}^{m\times n} G ∈ R m × n be non-zero with thin singular value decomposition G = U r Σ r V r T \mathbf{G}=\mathbf{U}_r\mathbf{\Sigma}_r\mathbf{V}_r^{\mathsf{T}} G = U r Σ r V r T , where r = rank ( G ) r=\operatorname{rank}(\mathbf{G}) r = rank ( G ) and Σ r = diag ( σ 1 , … , σ r ) \mathbf{\Sigma}_r=\operatorname{diag}(\sigma_1,\ldots,\sigma_r) Σ r = diag ( σ 1 , … , σ r ) has positive diagonal entries. Then
lmo B ( 0 , 1 ) ( G ) = arg min ∥ X ∥ o p ≤ 1 ⟨ G , X ⟩ F = { − U r V r T + W : U r T W = 0 , W V r = 0 , ∥ W ∥ o p ≤ 1 } , \operatorname{lmo}_{B(0,1)}(\mathbf{G})=\underset{\lVert\mathbf{X}\rVert_{\mathrm{op}}\le1}{\arg\min}\langle\mathbf{G},\mathbf{X}\rangle_{\mathrm{F}}
=\left\{-\mathbf{U}_r\mathbf{V}_r^{\mathsf{T}}+\mathbf{W}:\mathbf{U}_r^{\mathsf{T}}\mathbf{W}=0,\ \mathbf{W}\mathbf{V}_r=0,\ \lVert\mathbf{W}\rVert_{\mathrm{op}}\le1\right\}, lmo B ( 0 , 1 ) ( G ) = ∥ X ∥ op ≤ 1 arg min ⟨ G , X ⟩ F = { − U r V r T + W : U r T W = 0 , W V r = 0 , ∥ W ∥ op ≤ 1 } , where W ∈ R m × n \mathbf{W}\in\mathbb{R}^{m\times n} W ∈ R m × n . In particular, − U r V r T = − orth ( G ) -\mathbf{U}_r\mathbf{V}_r^{\mathsf{T}}=-\operatorname{orth}(\mathbf{G}) − U r V r T = − orth ( G ) is a canonical minimizer. The minimizer is unique if and only if r = min { m , n } r=\min\{m,n\} r = min { m , n } . For G = 0 \mathbf{G}=0 G = 0 , the LMO solution set is the entire ball B ( 0 , 1 ) B(0,1) B ( 0 , 1 ) .
Proof. Let u i \mathbf{u}_i u i and v i \mathbf{v}_i v i be the columns of U r \mathbf{U}_r U r and V r \mathbf{V}_r V r . For every feasible X \mathbf{X} X , the bound ∥ X ∥ o p ≤ 1 \|\mathbf{X}\|_{\mathrm{op}} \le1 ∥ X ∥ op ≤ 1 gives u i T X v i ≥ − 1 \mathbf{u}_i^{\mathsf{T}}\mathbf{X}\mathbf{v}_i \ge-1 u i T X v i ≥ − 1 . Hence
⟨ G , X ⟩ F = ∑ i = 1 r σ i u i T X v i ≥ − ∑ i = 1 r σ i = − ∥ G ∥ n u c . \langle\mathbf{G},\mathbf{X}\rangle_{\mathrm{F}} = \sum_{i=1}^{r} \sigma_i\mathbf{u}_i^{\mathsf{T}}\mathbf{X}\mathbf{v}_i \ge-\sum_{i=1}^{r}\sigma_i = -\|\mathbf{G}\|_{\mathrm{nuc}}. ⟨ G , X ⟩ F = i = 1 ∑ r σ i u i T X v i ≥ − i = 1 ∑ r σ i = − ∥ G ∥ nuc .
The matrix X 0 = − U r V r T \mathbf{X}_0=-\mathbf{U}_r\mathbf{V}_r^{\mathsf{T}} X 0 = − U r V r T is feasible and attains this bound.
If X \mathbf{X} X is any minimizer, positivity of all σ i \sigma_i σ i implies u i T X v i = − 1 \mathbf{u}_i^{\mathsf{T}}\mathbf{X}\mathbf{v}_i=-1 u i T X v i = − 1 for every i ≤ r i\le r i ≤ r . Since u i \mathbf{u}_i u i and v i \mathbf{v}_i v i are unit vectors and ∥ X ∥ o p = ∥ X T ∥ o p ≤ 1 \|\mathbf{X}\|_{\mathrm{op}}=\|\mathbf{X}^{\mathsf{T}}\|_{\mathrm{op}}\le1 ∥ X ∥ op = ∥ X T ∥ op ≤ 1 , equality in the Euclidean Cauchy–Schwarz inequality gives
X V r = − U r , X T U r = − V r . \mathbf{X}\mathbf{V}_r=-\mathbf{U}_r,\qquad\mathbf{X}^{\mathsf{T}}\mathbf{U}_r=-\mathbf{V}_r. X V r = − U r , X T U r = − V r .
Therefore W : = X + U r V r T \mathbf{W}:=\mathbf{X}+\mathbf{U}_r\mathbf{V}_r^{\mathsf{T}} W := X + U r V r T satisfies U r T W = 0 \mathbf{U}_r^{\mathsf{T}}\mathbf{W}=0 U r T W = 0 and W V r = 0 \mathbf{W}\mathbf{V}_r=0 W V r = 0 . Moreover,
W = ( I m − U r U r T ) X ( I n − V r V r T ) , \mathbf{W}=(\mathbf{I}_m-\mathbf{U}_r\mathbf{U}_r^{\mathsf{T}})\mathbf{X}(\mathbf{I}_n-\mathbf{V}_r\mathbf{V}_r^{\mathsf{T}}), W = ( I m − U r U r T ) X ( I n − V r V r T ) ,
so ∥ W ∥ o p ≤ ∥ X ∥ o p ≤ 1 \|\mathbf{W}\|_{\mathrm{op}}\le\|\mathbf{X}\|_{\mathrm{op}}\le1 ∥ W ∥ op ≤ ∥ X ∥ op ≤ 1 because both orthogonal projections have spectral norm at most one.
Conversely, let W \mathbf{W} W satisfy the three stated conditions and set X = − U r V r T + W \mathbf{X}=-\mathbf{U}_r\mathbf{V}_r^{\mathsf{T}}+\mathbf{W} X = − U r V r T + W . For any z = V r a + b \mathbf{z}=\mathbf{V}_r\mathbf{a}+\mathbf{b} z = V r a + b with V r T b = 0 \mathbf{V}_r^{\mathsf{T}}\mathbf{b}=0 V r T b = 0 , the vectors U r a \mathbf{U}_r\mathbf{a} U r a and W b \mathbf{W}\mathbf{b} Wb are orthogonal, so
∥ X z ∥ 2 2 = ∥ − U r a + W b ∥ 2 2 = ∥ a ∥ 2 2 + ∥ W b ∥ 2 2 ≤ ∥ a ∥ 2 2 + ∥ b ∥ 2 2 = ∥ z ∥ 2 2 . \|\mathbf{X}\mathbf{z}\|_2^2=\|-\mathbf{U}_r\mathbf{a}+\mathbf{W}\mathbf{b}\|_2^2=\|\mathbf{a}\|_2^2+\|\mathbf{W}\mathbf{b}\|_2^2\le\|\mathbf{a}\|_2^2+\|\mathbf{b}\|_2^2=\|\mathbf{z}\|_2^2. ∥ Xz ∥ 2 2 = ∥ − U r a + Wb ∥ 2 2 = ∥ a ∥ 2 2 + ∥ Wb ∥ 2 2 ≤ ∥ a ∥ 2 2 + ∥ b ∥ 2 2 = ∥ z ∥ 2 2 .
Thus ∥ X ∥ o p ≤ 1 \|\mathbf{X}\|_{\mathrm{op}}\le1 ∥ X ∥ op ≤ 1 , and X v i = − u i \mathbf{X}\mathbf{v}_i=-\mathbf{u}_i X v i = − u i implies ⟨ G , X ⟩ F = − ∥ G ∥ n u c \langle\mathbf{G},\mathbf{X}\rangle_{\mathrm{F}}=-\|\mathbf{G}\|_{\mathrm{nuc}} ⟨ G , X ⟩ F = − ∥ G ∥ nuc . This proves the full characterization. The identity U r V r T = orth ( G ) \mathbf{U}_r\mathbf{V}_r^{\mathsf{T}}=\operatorname{orth}(\mathbf{G}) U r V r T = orth ( G ) follows from Lemma 19.
If r = min { m , n } r=\min\{m,n\} r = min { m , n } , one of the two orthogonal complements is trivial, forcing W = 0 \mathbf{W}=0 W = 0 . If r < min { m , n } r<\min\{m,n\} r < min { m , n } , choose unit vectors u \mathbf{u} u and v \mathbf{v} v with U r T u = 0 \mathbf{U}_r^{\mathsf{T}}\mathbf{u}=0 U r T u = 0 and V r T v = 0 \mathbf{V}_r^{\mathsf{T}}\mathbf{v}=0 V r T v = 0 . Then W = u v T ≠ 0 \mathbf{W}=\mathbf{u}\mathbf{v}^{\mathsf{T}}\ne0 W = u v T = 0 is admissible, so the minimizer is not unique. Finally, if G = 0 \mathbf{G}=0 G = 0 , the objective is identically zero on B ( 0 , 1 ) \mathbb{B}(0,1) B ( 0 , 1 ) .
For G k ≠ 0 \mathbf{G}^k\ne0 G k = 0 , Lemma 20 shows that the general LMO update has the form
X k + 1 = X k − t k orth ( G k ) + t k W k . \mathbf{X}^{k+1}=\mathbf{X}^k-t_k\operatorname{orth}(\mathbf{G}^k)+t_k\mathbf{W}^k. X k + 1 = X k − t k orth ( G k ) + t k W k .
where r = rank ( G k ) r=\operatorname{rank}(\mathbf{G}^k) r = rank ( G k ) and W k \mathbf{W}^k W k satisfies
( U r k ) T W k = 0 , W k V r k = 0 , ∥ W k ∥ o p ≤ 1. (\mathbf{U}_r^k)^{\mathsf{T}}\mathbf{W}^k=0,\qquad\mathbf{W}^k\mathbf{V}_r^k=0,\qquad\|\mathbf{W}^k\|_{\mathrm{op}}\le1. ( U r k ) T W k = 0 , W k V r k = 0 , ∥ W k ∥ op ≤ 1.
By Lemma 16, this family is exactly the family of sharp updates, with corresponding selections
( G k ) ♯ = ∥ G k ∥ n u c ( orth ( G k ) − W k ) . (\mathbf{G}^k)^\sharp=\|\mathbf{G}^k\|_{\mathrm{nuc}}\left(\operatorname{orth}(\mathbf{G}^k)-\mathbf{W}^k\right). ( G k ) ♯ = ∥ G k ∥ nuc ( orth ( G k ) − W k ) .
Selecting W k = 0 \mathbf{W}^k=0 W k = 0 gives the canonical update
X k + 1 = X k − t k orth ( G k ) = X k − γ k ∥ G k ∥ n u c orth ( G k ) , \begin{aligned}
\mathbf{X}^{k+1}=\mathbf{X}^k-t_k\operatorname{orth}(\mathbf{G}^k) \\
&=\mathbf{X}^k-\gamma_k\|\mathbf{G}^k\|_{\mathrm{nuc}}\operatorname{orth}(\mathbf{G}^k),
\end{aligned} X k + 1 = X k − t k orth ( G k ) = X k − γ k ∥ G k ∥ nuc orth ( G k ) ,
which agrees with the canonical sharp selection used in the steepest-descent derivation and in (9) . With the state-dependent stepsize η k : = γ k ∥ G k ∥ n u c \eta_k:=\gamma_k\|\mathbf{G}^k\|_{\mathrm{nuc}} η k := γ k ∥ G k ∥ nuc and direction O k : = orth ( G k ) \mathbf{O}^k:=\operatorname{orth}(\mathbf{G}^k) O k := orth ( G k ) , it takes the form X k + 1 = X k − η k O k \mathbf{X}^{k+1}=\mathbf{X}^k-\eta_k\mathbf{O}^k X k + 1 = X k − η k O k . Other admissible LMO selections can produce different updates when rank ( G k ) < min { m , n } \operatorname{rank}(\mathbf{G}^k)<\min\{m,n\} rank ( G k ) < min { m , n } .
When G k = 0 \mathbf{G}^k=0 G k = 0 , we have t k = 0 t_k=0 t k = 0 , ∂ φ ∗ ( 0 ) = { 0 } \partial\varphi^*(0)=\{0\} ∂ φ ∗ ( 0 ) = { 0 } , and orth ( 0 ) = 0 \operatorname{orth}(0)=0 orth ( 0 ) = 0 by (10) . The translated ball is then B ( X k , 0 ) = { X k } \mathbb{B}(\mathbf{X}^k,0)=\{\mathbf{X}^k\} B ( X k , 0 ) = { X k } , so both the sharp and LMO formulations reduce to the identity update X k + 1 = X k \mathbf{X}^{k+1}=\mathbf{X}^k X k + 1 = X k .
Regularity vs. Norm Equivalence: Why Not Just Use Euclidean Smoothness?
To connect non-Euclidean geometries with the standard Euclidean setting, existing analyses (e.g., [50 , 30 ] ) often proceed via norm equivalence . Concretely, when the geometry is expressed in terms of a dual norm ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ , one compares it to a Euclidean reference norm (Frobenius for matrices, ℓ 2 \ell_2 ℓ 2 for vectors). In the matrix case, there exist constants α , β > 0 \alpha, \beta> 0 α , β > 0 (typically dimension- and/or rank-dependent) such that for all X ∈ R m × n X \in\mathbb{R}^{m \times n} X ∈ R m × n ,
α ∥ X ∥ ∗ ≤ ∥ X ∥ F ≤ β ∥ X ∥ ∗ . (46) \alpha\|X\|_* \le\|X\|_F \le\beta\|X\|_* . \tag*{(46)} α ∥ X ∥ ∗ ≤ ∥ X ∥ F ≤ β ∥ X ∥ ∗ . (46)
Analogously, in the vector case there exist α , β > 0 \alpha, \beta> 0 α , β > 0 such that for all x ∈ R d x \in\mathbb{R}^d x ∈ R d ,
α ∥ x ∥ ∗ ≤ ∥ x ∥ 2 ≤ β ∥ x ∥ ∗ . (47) \alpha\|x\|_* \le\|x\|_2 \le\beta\|x\|_* . \tag*{(47)} α ∥ x ∥ ∗ ≤ ∥ x ∥ 2 ≤ β ∥ x ∥ ∗ . (47)
These inequalities allow one to translate smoothness and noise assumptions stated in Euclidean/Frobenius norms into the target geometry. However, since α \alpha α and β \beta β typically scale with the ambient dimension (or matrix dimension/rank), such conversions can introduce large (often polynomial) inflation factors and thus obscure the true benefits of the chosen geometry.
For an E ∗ E^* E ∗ -valued noise vector ζ \zeta ζ with finite Euclidean second moment, only the left inequality in (47) is needed to obtain E [ ∥ ζ ∥ ∗ 2 ] ≤ α − 2 E [ ∥ ζ ∥ 2 2 ] \mathbb{E}\left[\|\zeta\|_*^2\right] \le\alpha^{-2}\mathbb{E}\left[\|\zeta\|_2^2\right] E [ ∥ ζ ∥ ∗ 2 ] ≤ α − 2 E [ ∥ ζ ∥ 2 2 ] . Thus the noise second-moment conversion uses α − 2 \alpha^{-2} α − 2 ; the Frobenius comparison in (46) gives the same factor for matrices.
The κ \kappa κ -regularity framework allows an auxiliary norm that need not be Euclidean. As in Definition 1, this norm is κ V , + \kappa_{V,+} κ V , + -smooth, meaning that its squared-norm function 1 2 ∥ ⋅ ∥ V , + 2 \frac{1}{2}\|\cdot\|_{V,+}^2 2 1 ∥ ⋅ ∥ V , + 2 is continuously differentiable and satisfies the stated smoothness inequality. It also satisfies the comparison (12) :
∥ u ∥ V 2 ≤ ∥ u ∥ V , + 2 ≤ C V ∥ u ∥ V 2 , C V = κ V κ V , + , u ∈ V , \|u\|_V^2 \le\|u\|_{V,+}^2 \le C_V\|u\|_V^2,\qquad C_V = \frac{\kappa_V}{\kappa_{V,+}},\qquad u \in V, ∥ u ∥ V 2 ≤ ∥ u ∥ V , + 2 ≤ C V ∥ u ∥ V 2 , C V = κ V , + κ V , u ∈ V ,
where 1 ≤ κ V , + ≤ κ V 1 \le\kappa_{V,+} \le\kappa_V 1 ≤ κ V , + ≤ κ V . Smoothness supports the analysis in the auxiliary norm, while the comparison transfers the resulting bounds to the original norm. Choosing the auxiliary norm therefore requires balancing its smoothness parameter κ V , + \kappa_{V,+} κ V , + against the factor C V C_V C V comparing the squared norms. To obtain a small regularity parameter, we minimize their product κ V = κ V , + C V \kappa_V = \kappa_{V,+}C_V κ V = κ V , + C V , rather than either factor alone. The two propositions below illustrate this choice for ℓ p \ell_p ℓ p and Schatten-p p p spaces.
Which space carries the norm. As in Definition 1, Proposition 10, Proposition 12 and Proposition 14, the results of this section are stated for a generic finite-dimensional normed space ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) , with constant of regularity κ V \kappa_V κ V , auxiliary norm ∥ ⋅ ∥ V , + \|\cdot\|_{V,+} ∥ ⋅ ∥ V , + and smoothness parameter κ V , + \kappa_{V,+} κ V , + . Following Remark 1, we apply them with V = E ∗ V = E^* V = E ∗ and ∥ ⋅ ∥ V = ∥ ⋅ ∥ ∗ \|\cdot\|_V = \|\cdot\|_* ∥ ⋅ ∥ V = ∥ ⋅ ∥ ∗ , in which case κ V \kappa_V κ V , ∥ ⋅ ∥ V , + \|\cdot\|_{V,+} ∥ ⋅ ∥ V , + and κ V , + \kappa_{V,+} κ V , + become κ \kappa κ , ∥ ⋅ ∥ ∗ , + \|\cdot\|_{*,+} ∥ ⋅ ∥ ∗ , + and κ + \kappa_+ κ + , and the generic compatibility chain (12) specializes to (13) . It is therefore the regularity constant of the dual space that enters our bounds, since that is where stochastic gradients live and where the noise is measured; an ℓ p \ell_p ℓ p or Schatten-p p p norm appearing below is a norm on V = E ∗ V = E^* V = E ∗ , that is, the norm induced on E ∗ E^* E ∗ by the update norm on E E E .
The Euclidean proxy and the role of the auxiliary norm. Constructing a Euclidean proxy uses both sides of (47) : the scaled norm ∥ ⋅ ∥ ∗ , + : = α − 1 ∥ ⋅ ∥ 2 \|\cdot\|_{*,+} := \alpha^{-1}\|\cdot\|_2 ∥ ⋅ ∥ ∗ , + := α − 1 ∥ ⋅ ∥ 2 is 1 1 1 -smooth and satisfies ∥ s ∥ ∗ 2 ≤ ∥ s ∥ ∗ , + 2 ≤ ( β / α ) 2 ∥ s ∥ ∗ 2 \|s\|_*^2 \le\|s\|_{*,+}^2 \le(\beta/\alpha)^2\|s\|_*^2 ∥ s ∥ ∗ 2 ≤ ∥ s ∥ ∗ , + 2 ≤ ( β / α ) 2 ∥ s ∥ ∗ 2 . It therefore gives the admissible regularity parameter κ = ( β / α ) 2 \kappa= (\beta/\alpha)^2 κ = ( β / α ) 2 , distinct from the noise-conversion factor α − 2 \alpha^{-2} α − 2 . The same construction applies to the Frobenius norm. Allowing other auxiliary norms can reduce the product κ = κ + C \kappa= \kappa_+C κ = κ + C , where C = κ / κ + C = \kappa/\kappa_+ C = κ / κ + is the squared-norm comparison factor in (13) . The auxiliary norm is used only in the analysis and enters the guarantee through κ \kappa κ ; the update (7) is determined by the update norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ and the fixed sharp selection, and never evaluates ∥ ⋅ ∥ ∗ , + \|\cdot\|_{*,+} ∥ ⋅ ∥ ∗ , + .
What this means for oracle complexity. When these constructions are applied to V = E ∗ V = E^* V = E ∗ , their effect on oracle complexity depends on the full statistical coefficients for the chosen noise model. Suppressing logarithmic factors, the relevant comparison is between C ∗ = κ σ 2 C_* = \kappa\sigma^2 C ∗ = κ σ 2 for the native analysis and C F = α − 2 σ F 2 C_F = \alpha^{-2}\sigma_F^2 C F = α − 2 σ F 2 for the Euclidean/Frobenius conversion, as in (61) . Here σ \sigma σ and σ F \sigma_F σ F measure the noise of the same oracle in the induced dual norm and the Euclidean/Frobenius norm, respectively. A smaller κ \kappa κ alone does not guarantee a sharper oracle-complexity bound; any improvement depends on these full coefficients, as illustrated by the noise models in Section J. With this distinction in place, we compare the auxiliary norms for vectors and matrices below, before stating and proving the corresponding propositions.
ell-p Spaces in Dimension d
We first specialize the preceding discussion to ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R d , ∥ ⋅ ∥ p ) (E^*,\|\cdot\|_*)=(\mathbb{R}^d,\|\cdot\|_p) ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R d , ∥ ⋅ ∥ p ) . Whether a non-Euclidean auxiliary norm improves the regularity bound depends on p p p .
The regime 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 . A scaled Euclidean auxiliary norm is already optimal: taking ∥ ⋅ ∥ ∗ , + = d 1 p − 1 2 ∥ ⋅ ∥ 2 \|\cdot\|_{*,+}=d^{\frac{1}{p}-\frac{1}{2}}\|\cdot\|_2 ∥ ⋅ ∥ ∗ , + = d p 1 − 2 1 ∥ ⋅ ∥ 2 reduces (13) to the classical ℓ p \ell_p ℓ p –ℓ 2 \ell_2 ℓ 2 equivalence, ∥ s ∥ 2 ≤ ∥ s ∥ p ≤ d 1 p − 1 2 ∥ s ∥ 2 \|s\|_2 \le\|s\|_p \le d^{\frac{1}{p}-\frac{1}{2}}\|s\|_2 ∥ s ∥ 2 ≤ ∥ s ∥ p ≤ d p 1 − 2 1 ∥ s ∥ 2 , and gives κ = d 2 p − 1 \kappa=d^{\frac{2}{p}-1} κ = d p 2 − 1 for every d ≥ 1 d \ge1 d ≥ 1 . This parameter cannot be improved by choosing another auxiliary norm; see Remark 5 and case 1 of Proposition 3 below.
The regime 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ . Here we may use ∥ ⋅ ∥ ∗ , + = ∥ ⋅ ∥ ρ \|\cdot\|_{*,+}=\|\cdot\|_\rho ∥ ⋅ ∥ ∗ , + = ∥ ⋅ ∥ ρ with 2 ≤ ρ ≤ p 2 \le\rho\le p 2 ≤ ρ ≤ p and ρ < ∞ \rho< \infty ρ < ∞ . Its smoothness constant κ + = ρ − 1 \kappa_+=\rho-1 κ + = ρ − 1 and squared-norm comparison factor d 2 / ρ − 2 / p d^{2/\rho-2/p} d 2/ ρ − 2/ p give the regularity bound κ ( ρ ) = ( ρ − 1 ) d 2 / ρ − 2 / p \kappa(\rho)=(\rho-1)d^{2/\rho-2/p} κ ( ρ ) = ( ρ − 1 ) d 2/ ρ − 2/ p . The minimizing choice is ρ ∗ : = 2 \rho_*:=2 ρ ∗ := 2 for 1 ≤ d < 8 1 \le d < 8 1 ≤ d < 8 and ρ ∗ : = min { p , ln d + ( ln d ) 2 − 2 ln d } \rho_*:=\min\{p,\ln d+\sqrt{(\ln d)^2-2\ln d}\} ρ ∗ := min { p , ln d + ( ln d ) 2 − 2 ln d } for d ≥ 8 d \ge8 d ≥ 8 ; in particular, ρ ∗ \rho_* ρ ∗ remains finite when p = ∞ p=\infty p = ∞ . For 1 ≤ d < 8 1 \le d < 8 1 ≤ d < 8 , it gives κ = d 1 − 2 / p ≤ 7.18 \kappa=d^{1-2/p}\le7.18 κ = d 1 − 2/ p ≤ 7.18 .20 For d ≥ 8 d \ge8 d ≥ 8 , the comparison takes the following form:
Euclidean proxy: ∥ ⋅ ∥ ∗ , + = ∥ ⋅ ∥ 2 \|\cdot\|_{*,+}=\|\cdot\|_2 ∥ ⋅ ∥ ∗ , + = ∥ ⋅ ∥ 2
Tuned proxy: ∥ ⋅ ∥ ∗ , + = ∥ ⋅ ∥ ρ ∗ \|\cdot\|_{*,+}=\|\cdot\|_{\rho_*} ∥ ⋅ ∥ ∗ , + = ∥ ⋅ ∥ ρ ∗
∥ s ∥ ∗ 2 ≤ ∥ s ∥ 2 2 ≤ d 1 − 2 / p ∥ s ∥ ∗ 2 \|s\|_*^2\le\|s\|_2^2\le d^{1-2/p}\|s\|_*^2 ∥ s ∥ ∗ 2 ≤ ∥ s ∥ 2 2 ≤ d 1 − 2/ p ∥ s ∥ ∗ 2
∥ s ∥ ∗ 2 ≤ ∥ s ∥ ρ ∗ 2 ≤ O ( 1 ) ∥ s ∥ ∗ 2 \|s\|_*^2\le\|s\|_{\rho_*}^2\le\mathcal{O}(1)\|s\|_*^2 ∥ s ∥ ∗ 2 ≤ ∥ s ∥ ρ ∗ 2 ≤ O ( 1 ) ∥ s ∥ ∗ 2
κ = d 1 − 2 p \kappa=d^{1-\frac{2}{p}} κ = d 1 − p 2
κ = Θ ( min { p , ln ( e d ) } ) \kappa=\Theta(\min\{p,\ln(ed)\}) κ = Θ ( min { p , ln ( e d )})
Both choices of ∥ ⋅ ∥ ∗ , + \|\cdot\|_{*,+} ∥ ⋅ ∥ ∗ , + are κ + \kappa_+ κ + -smooth and satisfy the required norm-comparison inequalities, so each establishes κ \kappa κ -regularity according to Definition 1, with its corresponding values of κ + \kappa_+ κ + and κ \kappa κ . Since ρ = 2 \rho=2 ρ = 2 is allowed in the minimization of κ ( ρ ) \kappa(\rho) κ ( ρ ) , the tuned regularity bound is no larger than the Euclidean bound d 1 − 2 / p d^{1-2/p} d 1 − 2/ p and can be significantly smaller. The constant comparison factor has a simple explanation: when p ≤ ρ ^ ( d ) p\le\hat{\rho}(d) p ≤ ρ ^ ( d ) , where ρ ^ ( d ) = ln d + ( ln d ) 2 − 2 ln d \hat{\rho}(d)=\ln d+\sqrt{(\ln d)^2-2\ln d} ρ ^ ( d ) = ln d + ( ln d ) 2 − 2 ln d , we have ρ ∗ = p \rho_*=p ρ ∗ = p and the factor is 1 1 1 ; when p > ρ ^ ( d ) p>\hat{\rho}(d) p > ρ ^ ( d ) , it is at most e 2 e^2 e 2 .
Interpretation. At p = 2 p=2 p = 2 , both proxies coincide with ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 and give κ = 1 \kappa=1 κ = 1 . For 2 ≤ p ≤ ∞ 2\le p\le\infty 2 ≤ p ≤ ∞ , the tuned bound has the optimal order Θ ( min { p , ln ( e d ) } ) \Theta(\min\{p,\ln(ed)\}) Θ ( min { p , ln ( e d )}) in all positive dimensions: case 2 of Proposition 3 establishes the comparison for d ≥ 8 d\ge8 d ≥ 8 , and Remark 7 covers 1 ≤ d < 8 1\le d<8 1 ≤ d < 8 . Thus fixed finite p ≥ 2 p\ge2 p ≥ 2 admits a bound independent of d d d , while at p = ∞ p=\infty p = ∞ , the dual of the ℓ 1 \ell_1 ℓ 1 update norm, tuning reduces the regularity parameter from d d d to Θ ( ln ( e d ) ) \Theta(\ln(ed)) Θ ( ln ( e d )) . These are comparisons of regularity parameters; the oracle-complexity comparison also depends on the noise scales, as explained above. The following proposition gives the formal statements and their proof in the generic notation ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) .
Proposition 3 (Regularity vs. Euclidean equivalence in ℓ p d \ell_p^d ℓ p d spaces). Fix d ∈ N d\in\mathbb{N} d ∈ N and p ∈ [ 1 , ∞ ] p\in[1,\infty] p ∈ [ 1 , ∞ ] . Let V : = R d V:=\mathbb{R}^d V := R d and equip it with the norm ∥ ⋅ ∥ V : = ∥ ⋅ ∥ p \|\cdot\|_V:=\|\cdot\|_p ∥ ⋅ ∥ V := ∥ ⋅ ∥ p . Then the following hold.
Case 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 : regularity recovers the classical ℓ p \ell_p ℓ p –ℓ 2 \ell_2 ℓ 2 equivalence. Take the auxiliary norm
∥ x ∥ V , + : = d 1 p − 1 2 ∥ x ∥ 2 . \|x\|_{V,+}:=d^{\frac{1}{p}-\frac{1}{2}}\|x\|_2. ∥ x ∥ V , + := d p 1 − 2 1 ∥ x ∥ 2 . Then ( V , ∥ ⋅ ∥ V , + ) (V,\|\cdot\|_{V,+}) ( V , ∥ ⋅ ∥ V , + ) is 1 1 1 -smooth, so κ V , + = 1 \kappa_{V,+}=1 κ V , + = 1 , and the compatibility chain (12) holds with
κ V = κ p ( d ) = d 2 p − 1 . \kappa_V=\kappa_p(d)=d^{\frac{2}{p}-1}. κ V = κ p ( d ) = d p 2 − 1 . For this choice, (12) is nothing but the standard ℓ p \ell_p ℓ p –ℓ 2 \ell_2 ℓ 2 norm equivalence:
∥ x ∥ 2 ≤ ∥ x ∥ p ≤ d 1 p − 1 2 ∥ x ∥ 2 ⟺ ∥ x ∥ 2 2 ≤ ∥ x ∥ p 2 ≤ d 2 p − 1 ∥ x ∥ 2 2 . (48) \|x\|_2\le\|x\|_p\le d^{\frac{1}{p}-\frac{1}{2}}\|x\|_2
\quad\Longleftrightarrow\quad
\|x\|_2^2\le\|x\|_p^2\le d^{\frac{2}{p}-1}\|x\|_2^2. \tag*{(48)} ∥ x ∥ 2 ≤ ∥ x ∥ p ≤ d p 1 − 2 1 ∥ x ∥ 2 ⟺ ∥ x ∥ 2 2 ≤ ∥ x ∥ p 2 ≤ d p 2 − 1 ∥ x ∥ 2 2 . (48) Moreover, this value of κ V \kappa_V κ V cannot be improved: the matching lower bound is (106) in Proposition 13.
Case 2 ≤ p ≤ ∞ \le p \le\infty ≤ p ≤ ∞ : comparison with the Euclidean proxy. For any ρ ∈ [ 2 , p ] \rho\in[2,p] ρ ∈ [ 2 , p ] with ρ < ∞ \rho< \infty ρ < ∞ , take the auxiliary norm ∥ ⋅ ∥ V , + : = ∥ ⋅ ∥ ρ \|\cdot\|_{V,+}:=\|\cdot\|_{\rho} ∥ ⋅ ∥ V , + := ∥ ⋅ ∥ ρ . Then ( V , ∥ ⋅ ∥ ρ ) (V,\|\cdot\|_{\rho}) ( V , ∥ ⋅ ∥ ρ ) is ( ρ − 1 ) (\rho-1) ( ρ − 1 ) -smooth, so κ V , + = ρ − 1 \kappa_{V,+}=\rho-1 κ V , + = ρ − 1 , and (12) holds with compatibility factor and admissible regularity parameter
κ V κ V , + = d 2 ρ − 2 p and κ V ( ρ ) = ( ρ − 1 ) d 2 ρ − 2 p , \frac{\kappa_V}{\kappa_{V,+}}=d^{\frac{2}{\rho}-\frac{2}{p}}
\qquad\text{and}\qquad
\kappa_V(\rho)=(\rho-1)d^{\frac{2}{\rho}-\frac{2}{p}}, κ V , + κ V = d ρ 2 − p 2 and κ V ( ρ ) = ( ρ − 1 ) d ρ 2 − p 2 ,
respectively. Optimizing over ρ ∈ [ 2 , p ] \rho\in[2,p] ρ ∈ [ 2 , p ] with ρ < ∞ \rho<\infty ρ < ∞ yields the best regularity parameter obtainable this way,
κ V : = κ p ( d ) = min 2 ≤ ρ ≤ p ρ < ∞ ( ρ − 1 ) d 2 ρ − 2 p . (49) \kappa_V:=\kappa_p(d)=\min_{\substack{2\le\rho\le p\\ \rho<\infty}}(\rho-1)d^{\frac{2}{\rho}-\frac{2}{p}}. \tag*{(49)} κ V := κ p ( d ) = 2 ≤ ρ ≤ p ρ < ∞ min ( ρ − 1 ) d ρ 2 − p 2 . (49)
When p = ∞ p=\infty p = ∞ , we use 1 / p = 0 1/p=0 1/ p = 0 and optimize over ρ ∈ [ 2 , ∞ ) \rho\in[2,\infty) ρ ∈ [ 2 , ∞ ) . Since ρ = 2 \rho=2 ρ = 2 is admissible, κ p ( d ) ≤ d 1 − 2 / p \kappa_p(d)\le d^{1-2/p} κ p ( d ) ≤ d 1 − 2/ p , the parameter supplied by the Euclidean proxy. At p = 2 p=2 p = 2 , both choices coincide with ∥ ⋅ ∥ 2 \|\cdot\|_2 ∥ ⋅ ∥ 2 and give κ p ( d ) = 1 \kappa_p(d)=1 κ p ( d ) = 1 . Assume in addition that ln d ≥ 2 \ln d\ge2 ln d ≥ 2 (e.g., d ≥ 8 d\ge8 d ≥ 8 ) and set
ρ ^ ( d ) : = ln d + ( ln d ) 2 − 2 ln d , ρ ∗ : = min { p , ρ ^ ( d ) } . \widehat{\rho}(d):=\ln d+\sqrt{(\ln d)^2-2\ln d},
\qquad
\rho_*:=\min\{p,\widehat{\rho}(d)\}. ρ ( d ) := ln d + ( ln d ) 2 − 2 ln d , ρ ∗ := min { p , ρ ( d )} .
The Euclidean choice ρ = 2 \rho=2 ρ = 2 produces the classical compatibility factor d 1 − 2 p d^{1-\frac{2}{p}} d 1 − p 2 , whereas the tuned choice ρ = ρ ∗ \rho=\rho_* ρ = ρ ∗ produces a factor d 2 ρ ∗ − 2 p d^{\frac{2}{\rho_*}-\frac{2}{p}} d ρ ∗ 2 − p 2 that is never larger and is in fact dimension-free in both regimes p ≤ ρ ^ ( d ) p\le\widehat{\rho}(d) p ≤ ρ ( d ) and p > ρ ^ ( d ) p>\widehat{\rho}(d) p > ρ ( d ) :
∥ x ∥ p 2 ≤ ∥ x ∥ ρ ∗ 2 ≤ O ( 1 ) ∥ x ∥ p 2 ∀ x ∈ R d . (50) \|x\|_{p}^{2}\le\|x\|_{\rho_*}^{2}\le\mathcal{O}(1)\|x\|_{p}^{2}
\qquad\forall x\in\mathbb{R}^{d}. \tag*{(50)} ∥ x ∥ p 2 ≤ ∥ x ∥ ρ ∗ 2 ≤ O ( 1 ) ∥ x ∥ p 2 ∀ x ∈ R d . (50)
Finally, the scaling κ p ( d ) = Θ ( min { p , ln ( e d ) } ) \kappa_p(d)=\Theta(\min\{p,\ln(ed)\}) κ p ( d ) = Θ ( min { p , ln ( e d )}) is unavoidable for d ≥ 8 d\ge8 d ≥ 8 : the matching lower bound is (107) in Proposition 13.
Proof. Both cases instantiate Definition 1 on V = R d V=\mathbb{R}^{d} V = R d ; the constructions and the underlying smoothness estimates are those of Proposition 12, and the only new content is the explicit comparison with the Euclidean conversion.
Case 1 ≤ p < 2 \le p<2 ≤ p < 2 . The norm ∥ ⋅ ∥ V , + \|\cdot\|_{V,+} ∥ ⋅ ∥ V , + is a scaled Euclidean norm and is therefore 1-smooth, so κ V , + = 1 \kappa_{V,+}=1 κ V , + = 1 . The standard inequalities ∥ x ∥ 2 ≤ ∥ x ∥ p ≤ d 1 p − 1 2 ∥ x ∥ 2 \|x\|_2\le\|x\|_p\le d^{\frac{1}{p}-\frac{1}{2}}\|x\|_2 ∥ x ∥ 2 ≤ ∥ x ∥ p ≤ d p 1 − 2 1 ∥ x ∥ 2 give ∥ x ∥ p ≤ ∥ x ∥ V , + \|x\|_p\le\|x\|_{V,+} ∥ x ∥ p ≤ ∥ x ∥ V , + and ∥ x ∥ V , + 2 = d 2 p − 1 ∥ x ∥ 2 2 ≤ d 2 p − 1 ∥ x ∥ p 2 \|x\|_{V,+}^{2}=d^{\frac{2}{p}-1}\|x\|_2^{2}\le d^{\frac{2}{p}-1}\|x\|_p^{2} ∥ x ∥ V , + 2 = d p 2 − 1 ∥ x ∥ 2 2 ≤ d p 2 − 1 ∥ x ∥ p 2 , which is (12) with κ V = d 2 p − 1 \kappa_V=d^{\frac{2}{p}-1} κ V = d p 2 − 1 and κ V , + = 1 \kappa_{V,+}=1 κ V , + = 1 ; squaring the same two inequalities gives (48) . The lower bound is (106) .
Case 2 ≤ p ≤ ∞ \le p\le\infty ≤ p ≤ ∞ . Fix 2 ≤ ρ ≤ p 2\le\rho\le p 2 ≤ ρ ≤ p with ρ < ∞ \rho<\infty ρ < ∞ . That ( R d , ∥ ⋅ ∥ ρ ) (\mathbb{R}^{d},\|\cdot\|_{\rho}) ( R d , ∥ ⋅ ∥ ρ ) is ( ρ − 1 ) (\rho-1) ( ρ − 1 ) -smooth for 2 ≤ ρ < ∞ 2\le\rho<\infty 2 ≤ ρ < ∞ is shown in Step 1 of the proof of Proposition 12. The standard ℓ p \ell_p ℓ p –ℓ ρ \ell_\rho ℓ ρ inequalities give ∥ x ∥ p ≤ ∥ x ∥ ρ ≤ d 1 ρ − 1 p ∥ x ∥ p \|x\|_p\le\|x\|_\rho\le d^{\frac{1}{\rho}-\frac{1}{p}}\|x\|_p ∥ x ∥ p ≤ ∥ x ∥ ρ ≤ d ρ 1 − p 1 ∥ x ∥ p , and squaring yields (12) with κ V / κ V , + = d 2 ρ − 2 p \kappa_V/\kappa_{V,+}=d^{\frac{2}{\rho}-\frac{2}{p}} κ V / κ V , + = d ρ 2 − p 2 , hence κ V ( ρ ) = ( ρ − 1 ) d 2 ρ − 2 p \kappa_V(\rho)=(\rho-1)d^{\frac{2}{\rho}-\frac{2}{p}} κ V ( ρ ) = ( ρ − 1 ) d ρ 2 − p 2 . Minimizing κ V ( ρ ) \kappa_V(\rho) κ V ( ρ ) over ρ ∈ [ 2 , p ] \rho\in[2,p] ρ ∈ [ 2 , p ] with ρ < ∞ \rho<\infty ρ < ∞ gives (49) . When p = ∞ p=\infty p = ∞ , this continuous function satisfies κ V ( ρ ) = ( ρ − 1 ) d 2 / ρ → ∞ \kappa_V(\rho)=(\rho-1)d^{2/\rho}\to\infty κ V ( ρ ) = ( ρ − 1 ) d 2/ ρ → ∞ as ρ → ∞ \rho\to\infty ρ → ∞ , so the minimum on [ 2 , ∞ ) [2,\infty) [ 2 , ∞ ) is attained at a finite ρ \rho ρ . The expression for ρ ^ ( d ) \widehat{\rho}(d) ρ ( d ) and the bound d 2 / ρ ^ ( d ) ≤ e 2 d^{2/\widehat{\rho}(d)}\le e^2 d 2/ ρ ( d ) ≤ e 2 (valid whenever ln d ≥ 2 \ln d\ge2 ln d ≥ 2 ) are derived in Step 3 of the proof of Proposition 12; in particular ρ ^ ( d ) ≥ ln d ≥ 2 \widehat{\rho}(d)\ge\ln d\ge2 ρ ( d ) ≥ ln d ≥ 2 , so 2 ≤ ρ ∗ < ∞ 2\le\rho_*<\infty 2 ≤ ρ ∗ < ∞ even when p = ∞ p=\infty p = ∞ , and the tuned compatibility factor d 2 ρ ∗ − 2 p d^{\frac{2}{\rho_*}-\frac{2}{p}} d ρ ∗ 2 − p 2 is never larger than the Euclidean one, d 1 − 2 p d^{1-\frac{2}{p}} d 1 − p 2 . It remains to prove (50) , and we distinguish two subcases.
Subcase 1: p ≤ ρ ^ ( d ) p\le\widehat{\rho}(d) p ≤ ρ ( d ) . Then ρ ∗ = p \rho_*=p ρ ∗ = p , so ∥ x ∥ ρ ∗ = ∥ x ∥ p \|x\|_{\rho_*}=\|x\|_p ∥ x ∥ ρ ∗ = ∥ x ∥ p and (50) holds with constant 1.
Subcase 2: p > ρ ^ ( d ) p>\widehat{\rho}(d) p > ρ ( d ) . Then ρ ∗ = ρ ^ ( d ) \rho_*=\widehat{\rho}(d) ρ ∗ = ρ ( d ) and
∥ x ∥ ρ ∗ 2 ≤ d 2 ρ ∗ − 2 p ∥ x ∥ p 2 = d 2 / ρ ^ ( d ) d − 2 / p ∥ x ∥ p 2 ≤ d 2 / ρ ^ ( d ) ∥ x ∥ p 2 ≤ e 2 ∥ x ∥ p 2 , \|x\|_{\rho_*}^{2}\le d^{\frac{2}{\rho_*}-\frac{2}{p}}\|x\|_p^{2}
=d^{2/\widehat{\rho}(d)}d^{-2/p}\|x\|_p^{2}
\le d^{2/\widehat{\rho}(d)}\|x\|_p^{2}
\le e^{2}\|x\|_p^{2}, ∥ x ∥ ρ ∗ 2 ≤ d ρ ∗ 2 − p 2 ∥ x ∥ p 2 = d 2/ ρ ( d ) d − 2/ p ∥ x ∥ p 2 ≤ d 2/ ρ ( d ) ∥ x ∥ p 2 ≤ e 2 ∥ x ∥ p 2 ,
which yields (50) with O ( 1 ) = e 2 \mathcal{O}(1)=e^2 O ( 1 ) = e 2 . The lower bound is (107) .
Spaces with Schatten-p Norms on real m-by-n matrices
For matrices, we compare the fixed Frobenius proxy with auxiliary Schatten norms. The two p p p regimes follow the vector case, although the smoothness bounds for the auxiliary norms differ. Let V : = R m × n V := \mathbb{R}^{m \times n} V := R m × n and denote r : = min { m , n } r := \min\{m,n\} r := min { m , n } . For p ∈ [ 1 , ∞ ] p \in[1,\infty] p ∈ [ 1 , ∞ ] , the Schatten-p p p norm is ∥ X ∥ S p : = ∥ σ ( X ) ∥ p \|X\|_{S_p} := \|\sigma(X)\|_p ∥ X ∥ S p := ∥ σ ( X ) ∥ p , where σ ( X ) ∈ R r \sigma(X) \in\mathbb{R}^r σ ( X ) ∈ R r is the vector of singular values of X X X (padded with zeros when rank ( X ) < r \operatorname{rank}(X) < r rank ( X ) < r ). Since Schatten norms are ℓ p \ell_p ℓ p norms on σ ( X ) \sigma(X) σ ( X ) , the standard comparison inequalities (see Lemma 14) give, for 1 ≤ ρ ≤ p ≤ ∞ 1 \le\rho\le p \le\infty 1 ≤ ρ ≤ p ≤ ∞ ,
∥ X ∥ S p ≤ ∥ X ∥ S ρ ≤ r 1 ρ − 1 p ∥ X ∥ S p . (51) \|X\|_{S_p} \le\|X\|_{S_\rho} \le r^{\frac{1}{\rho}-\frac{1}{p}}\|X\|_{S_p}. \tag*{(51)} ∥ X ∥ S p ≤ ∥ X ∥ S ρ ≤ r ρ 1 − p 1 ∥ X ∥ S p . (51)
(For X ≠ 0 X \ne0 X = 0 , one may replace r r r by rank ( X ) \operatorname{rank}(X) rank ( X ) in (51) for an instance-dependent bound, which can be tighter when rank ( X ) ≪ r \operatorname{rank}(X) \ll r rank ( X ) ≪ r ; the zero case is immediate.)
The regime 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 . The scaled Frobenius auxiliary norm ∥ ⋅ ∥ V , + : = r 1 p − 1 2 ∥ ⋅ ∥ F \|\cdot\|_{V,+} := r^{\frac{1}{p}-\frac{1}{2}}\|\cdot\|_F ∥ ⋅ ∥ V , + := r p 1 − 2 1 ∥ ⋅ ∥ F is 1-smooth and gives κ V = r 2 / p − 1 \kappa_V = r^{2/p-1} κ V = r 2/ p − 1 for every r ≥ 1 r \ge1 r ≥ 1 . The matching lower bound shows that this parameter is optimal, so allowing non-Frobenius auxiliary norms cannot improve it.
The regime 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ . The Frobenius proxy is 1-smooth and gives the squared-norm comparison factor and regularity parameter r 1 − 2 / p r^{1-2/p} r 1 − 2/ p . Alternatively, for 2 ≤ ρ ≤ p 2 \le\rho\le p 2 ≤ ρ ≤ p with ρ < ∞ \rho< \infty ρ < ∞ , the auxiliary norm ∥ ⋅ ∥ S ρ \|\cdot\|_{S_\rho} ∥ ⋅ ∥ S ρ has comparison factor r 2 / ρ − 2 / p r^{2/\rho-2/p} r 2/ ρ − 2/ p and admits the smoothness bound max { 2 , ρ − 1 } \max\{2,\rho-1\} max { 2 , ρ − 1 } . Their product is the regularity bound optimized in the proposition below. For r ≥ 8 r \ge8 r ≥ 8 , the minimizing exponent for this bound is ρ ∗ = min { p , max { 3 , ρ ^ ( r ) } } \rho_* = \min\{p,\max\{3,\hat{\rho}(r)\}\} ρ ∗ = min { p , max { 3 , ρ ^ ( r )}} , where ρ ^ ( r ) = ln r + ( ln r ) 2 − 2 ln r \hat{\rho}(r) = \ln r + \sqrt{(\ln r)^2 - 2\ln r} ρ ^ ( r ) = ln r + ( ln r ) 2 − 2 ln r . If ρ ∗ = p \rho_* = p ρ ∗ = p , the auxiliary and target norms coincide and the comparison factor is 1; otherwise the factor is at most e 2 e^2 e 2 . Thus the tuned squared-norm comparison factor is no larger than the Frobenius factor and is bounded independently of r r r .
Interpretation. The resulting regularity bound has the optimal order Θ ( min { p , ln ( e r ) } ) \Theta(\min\{p,\ln(er)\}) Θ ( min { p , ln ( er )}) for r ≥ 8 r \ge8 r ≥ 8 ; Remark 7 extends this order to 1 ≤ r < 8 1 \le r < 8 1 ≤ r < 8 . For fixed finite p ≥ 2 p \ge2 p ≥ 2 this order is independent of r r r , while at p = ∞ p = \infty p = ∞ it is logarithmic, compared with the Frobenius proxy’s linear parameter r r r . At p = 2 p = 2 p = 2 , both auxiliary norms are Frobenius and one can take κ V = 1 \kappa_V = 1 κ V = 1 : the uniform smoothness estimate max { 2 , ρ − 1 } \max\{2,\rho-1\} max { 2 , ρ − 1 } is not sharp at ρ = 2 \rho= 2 ρ = 2 . As in the vector case, these geometric improvements give a smaller oracle-complexity bound only when the full statistical coefficients favor the native analysis. We now state the precise comparison and prove it.
Proposition 4 (Regularity vs. Frobenius equivalence in Schatten classes). Fix m , n ∈ N m,n \in\mathbb{N} m , n ∈ N , let r : = min { m , n } r := \min\{m,n\} r := min { m , n } and let p ∈ [ 1 , ∞ ] p \in[1,\infty] p ∈ [ 1 , ∞ ] . Let V : = R m × n V := \mathbb{R}^{m \times n} V := R m × n and equip it with the norm ∥ ⋅ ∥ V : = ∥ ⋅ ∥ S p \|\cdot\|_V := \|\cdot\|_{S_p} ∥ ⋅ ∥ V := ∥ ⋅ ∥ S p . Then the following hold.
Case 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 : regularity recovers the Schatten-p p p –Frobenius equivalence. Take the scaled Frobenius auxiliary norm
∥ X ∥ V , + : = r 1 p − 1 2 ∥ X ∥ F , \|X\|_{V,+} := r^{\frac{1}{p}-\frac{1}{2}}\|X\|_F, ∥ X ∥ V , + := r p 1 − 2 1 ∥ X ∥ F , which is 1-smooth, so κ V , + = 1 \kappa_{V,+}=1 κ V , + = 1 . Then (12) holds with
κ V : = κ p ( m , n ) = r 2 p − 1 , \kappa_V := \kappa_p(m,n) = r^{\frac{2}{p}-1}, κ V := κ p ( m , n ) = r p 2 − 1 , and it is equivalent to the standard Schatten-p p p –Frobenius norm equivalence:
∥ X ∥ F ≤ ∥ X ∥ S p ≤ r 1 p − 1 2 ∥ X ∥ F ⟺ ∥ X ∥ F 2 ≤ ∥ X ∥ S p 2 ≤ r 2 p − 1 ∥ X ∥ F 2 . (52) \|X\|_F \le\|X\|_{S_p} \le r^{\frac{1}{p}-\frac{1}{2}}\|X\|_F
\qquad\Longleftrightarrow\qquad
\|X\|_F^2 \le\|X\|_{S_p}^2 \le r^{\frac{2}{p}-1}\|X\|_F^2. \tag*{(52)} ∥ X ∥ F ≤ ∥ X ∥ S p ≤ r p 1 − 2 1 ∥ X ∥ F ⟺ ∥ X ∥ F 2 ≤ ∥ X ∥ S p 2 ≤ r p 2 − 1 ∥ X ∥ F 2 . (52) Moreover, this value of κ V \kappa_V κ V cannot be improved: the matching lower bound is (108) in Proposition 15.
Case 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ : comparison with the Frobenius proxy. For any ρ ∈ [ 2 , p ] \rho\in[2,p] ρ ∈ [ 2 , p ] with ρ < ∞ \rho< \infty ρ < ∞ , take the auxiliary norm ∥ ⋅ ∥ V , + : = ∥ ⋅ ∥ S ρ \|\cdot\|_{V,+} := \|\cdot\|_{S_\rho} ∥ ⋅ ∥ V , + := ∥ ⋅ ∥ S ρ . Then (51) yields the compatibility chain
∥ X ∥ S p 2 ≤ ∥ X ∥ S ρ 2 ≤ r 2 ρ − 2 p ∥ X ∥ S p 2 , (53) \|X\|_{S_p}^2 \le\|X\|_{S_\rho}^2 \le r^{\frac{2}{\rho}-\frac{2}{p}}\|X\|_{S_p}^2, \tag*{(53)} ∥ X ∥ S p 2 ≤ ∥ X ∥ S ρ 2 ≤ r ρ 2 − p 2 ∥ X ∥ S p 2 , (53) and ( V , ∥ ⋅ ∥ S ρ ) (V,\|\cdot\|_{S_\rho}) ( V , ∥ ⋅ ∥ S ρ ) is κ V , + \kappa_{V,+} κ V , + -smooth with κ V , + = max { 2 , ρ − 1 } \kappa_{V,+}=\max\{2,\rho-1\} κ V , + = max { 2 , ρ − 1 } for ρ ≥ 2 \rho\ge2 ρ ≥ 2 (see Proposition 14 and the references therein), so that (12) holds with
κ V ( ρ ) = max { 2 , ρ − 1 } r 2 ρ − 2 p . \kappa_V(\rho) = \max\{2,\rho-1\}r^{\frac{2}{\rho}-\frac{2}{p}}. κ V ( ρ ) = max { 2 , ρ − 1 } r ρ 2 − p 2 . Assume now that r ≥ 8 r \ge8 r ≥ 8 , so ln r ≥ 2 \ln r \ge2 ln r ≥ 2 . Under this condition, optimizing over ρ \rho ρ yields a regularity parameter of order κ V = O ( min { p , ln ( e r ) } ) \kappa_V = O(\min\{p,\ln(er)\}) κ V = O ( min { p , ln ( er )}) . More explicitly, set
ρ ^ ( r ) : = ln r + ( ln r ) 2 − 2 ln r , ρ ∗ : = min { p , max { 3 , ρ ^ ( r ) } } . \hat{\rho}(r) := \ln r + \sqrt{(\ln r)^2 - 2\ln r}, \qquad\rho_* := \min\{p,\max\{3,\hat{\rho}(r)\}\}. ρ ^ ( r ) := ln r + ( ln r ) 2 − 2 ln r , ρ ∗ := min { p , max { 3 , ρ ^ ( r )}} .
The Frobenius choice ρ = 2 \rho= 2 ρ = 2 produces the compatibility factor r 1 − 2 p r^{1-\frac{2}{p}} r 1 − p 2 , whereas the tuned choice ρ = ρ ∗ \rho= \rho_* ρ = ρ ∗ produces a factor that is no larger and is bounded by an absolute constant independent of the dimension parameter r = min { m , n } r = \min\{m,n\} r = min { m , n } in both regimes:
∥ X ∥ S p 2 ≤ ∥ X ∥ S ρ ∗ 2 ≤ O ( 1 ) ∥ X ∥ S p 2 ∀ X ∈ R m × n . (54) \lVert X\rVert_{S_p}^2 \le\lVert X\rVert_{S_{\rho_*}}^2 \le\mathcal{O}(1)\,\lVert X\rVert_{S_p}^2 \qquad\forall X \in\mathbb{R}^{m\times n}. \tag*{(54)} ∥ X ∥ S p 2 ≤ ∥ X ∥ S ρ ∗ 2 ≤ O ( 1 ) ∥ X ∥ S p 2 ∀ X ∈ R m × n . (54)
At p = 2 p = 2 p = 2 , both proxies coincide with the Frobenius norm and the compatibility factor is 1. Its exact smoothness constant is also 1, so κ V = 1 \kappa_V = 1 κ V = 1 is admissible in this case; the uniform bound max { 2 , ρ − 1 } \max\{2,\rho-1\} max { 2 , ρ − 1 } used above is not sharp at ρ = 2 \rho= 2 ρ = 2 . Finally, the scaling κ p ( m , n ) = Θ ( min { p , ln ( e r ) } ) \kappa_p(m,n) = \Theta\left(\min\{p,\ln(er)\}\right) κ p ( m , n ) = Θ ( min { p , ln ( er )} ) is unavoidable for r ≥ 8 r \ge8 r ≥ 8 : the matching lower bound is (109) in Proposition 15.
Proof. As in the vector case, both cases instantiate Definition 1 on V = R m × n V = \mathbb{R}^{m\times n} V = R m × n , and the constructions and smoothness estimates are those of Proposition 14.
Case 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 . This is the Schatten analogue of case 1 of Proposition 3. Since ∥ ⋅ ∥ V , + \lVert\cdot\rVert_{V,+} ∥ ⋅ ∥ V , + is a scaled Hilbert norm, it is 1-smooth. Applying (51) with smaller exponent p p p and larger exponent 2 yields ∥ X ∥ S 2 = ∥ X ∥ F ≤ ∥ X ∥ S p ≤ r 1 p − 1 2 ∥ X ∥ F \lVert X\rVert_{S_2} = \lVert X\rVert_F \le\lVert X\rVert_{S_p} \le r^{\frac{1}{p}-\frac{1}{2}}\lVert X\rVert_F ∥ X ∥ S 2 = ∥ X ∥ F ≤ ∥ X ∥ S p ≤ r p 1 − 2 1 ∥ X ∥ F , i.e., (52) . The compatibility chain (12) then follows by construction with κ V = r 2 p − 1 \kappa_V = r^{\frac{2}{p}-1} κ V = r p 2 − 1 and κ V , + = 1 \kappa_{V,+} = 1 κ V , + = 1 . The lower bound is (108) .
Case 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ . The compatibility chain (53) is exactly (51) squared. The smoothness bound for Schatten-ρ \rho ρ spaces with ρ ≥ 2 \rho\ge2 ρ ≥ 2 , and the resulting regularity parameter, are stated and proved in Proposition 14 (see also [40 ] ). For the remaining claims, assume r ≥ 8 r \ge8 r ≥ 8 . Under this condition, the optimization over ρ \rho ρ is the same calculus as in the ℓ p \ell_p ℓ p case, with d d d replaced by r r r , and leads to κ V = O ( min { p , ln ( e r ) } ) \kappa_V = \mathcal{O}(\min\{p,\ln(er)\}) κ V = O ( min { p , ln ( er )}) . It remains to justify (54) , and we distinguish two subcases.
Subcase 1: ρ ∗ = p \rho_* = p ρ ∗ = p . Then ∥ X ∥ S ρ ∗ = ∥ X ∥ S p \lVert X\rVert_{S_{\rho_*}} = \lVert X\rVert_{S_p} ∥ X ∥ S ρ ∗ = ∥ X ∥ S p and (54) holds with constant 1.
Subcase 2: ρ ∗ = max { 3 , ρ ^ ( r ) } \rho_* = \max\{3,\widehat{\rho}(r)\} ρ ∗ = max { 3 , ρ ( r )} . Since ln r ≥ 2 \ln r \ge2 ln r ≥ 2 , we have ρ ^ ( r ) ≥ ln r \widehat{\rho}(r) \ge\ln r ρ ( r ) ≥ ln r and therefore ρ ∗ ≥ ln r \rho_* \ge\ln r ρ ∗ ≥ ln r , whence r 2 / ρ ∗ ≤ r 2 / ln r = e 2 r^{2/\rho_*} \le r^{2/\ln r} = e^2 r 2/ ρ ∗ ≤ r 2/ l n r = e 2 . Combining this with (53) gives
∥ X ∥ S ρ ∗ 2 ≤ r 2 ρ ∗ − 2 p ∥ X ∥ S p 2 ≤ r 2 / ρ ∗ ∥ X ∥ S p 2 ≤ e 2 ∥ X ∥ S p 2 , \lVert X\rVert_{S_{\rho_*}}^2 \le r^{\frac{2}{\rho_*}-\frac{2}{p}}\lVert X\rVert_{S_p}^2 \le r^{2/\rho_*}\lVert X\rVert_{S_p}^2 \le e^2\lVert X\rVert_{S_p}^2, ∥ X ∥ S ρ ∗ 2 ≤ r ρ ∗ 2 − p 2 ∥ X ∥ S p 2 ≤ r 2/ ρ ∗ ∥ X ∥ S p 2 ≤ e 2 ∥ X ∥ S p 2 ,
which yields (54) with O ( 1 ) = e 2 \mathcal{O}(1) = e^2 O ( 1 ) = e 2 . The lower bound is (109) .
A Direct Comparison of Euclidean/Frobenius and Geometry-Native Oracles
There are two ways to model stochastic-gradient noise in a non-Euclidean geometry. One assumes light tails in a fixed Euclidean or Frobenius norm and converts the resulting bound into the dual norm induced by the update geometry; the other states the light-tail condition directly in that dual norm. This section compares them. We keep the conversion constant α − 1 \alpha^{-1} α − 1 , the regularity constant κ \kappa κ and the two noise scales separate throughout, so that one can see which part of each bound comes from the geometry and which part comes from the perturbation model.
Notation. The index p p p always refers to the norm on the dual space E ∗ E^* E ∗ , exactly as in § 3.4 and Section I: we write ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R d , ∥ ⋅ ∥ p ) (E^*, \|\cdot\|_*) = (\mathbb{R}^d, \|\cdot\|_p) ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R d , ∥ ⋅ ∥ p ) in the vector case and ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R m × n , ∥ ⋅ ∥ S p ) (E^*, \|\cdot\|_*) = (\mathbb{R}^{m \times n}, \|\cdot\|_{S_p}) ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R m × n , ∥ ⋅ ∥ S p ) with r : = min { m , n } r := \min\{m,n\} r := min { m , n } in the matrix case, and we adopt the convention d 2 / ∞ = r 2 / ∞ = 1 d^{2/\infty} = r^{2/\infty} = 1 d 2/∞ = r 2/∞ = 1 .
Overview of the comparison. We compare the same oracle under the two norm-based analyses. The examples use fresh independent additive perturbations: independent Rademacher signs or standard Gaussian entries in the vector case, and their diagonal matrix counterparts (Definition 9 and Definition 10). Write D = d D = d D = d for vectors and D = r D = r D = r for these diagonal matrix models. Under the common assumptions stated below, the two oracle bounds have the same leading term; their statistical terms are governed by C ∗ = κ σ 2 C_* = \kappa\sigma^2 C ∗ = κ σ 2 and C F = α − 2 σ F 2 C_F = \alpha^{-2}\sigma_F^2 C F = α − 2 σ F 2 .
For 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 , the squared conversion factor α − 2 \alpha^{-2} α − 2 and the regularity constant are both D 2 / p − 1 D^{2/p-1} D 2/ p − 1 . Their ratio therefore cancels, leaving the comparison between the noise scales. For these dense perturbations, the ratio of squared noise scales is D 2 / p − 1 D^{2/p-1} D 2/ p − 1 , which is also the ratio of the native and converted coefficients. At p = 1 p = 1 p = 1 , the converted and native coefficients consequently have orders D 2 D^2 D 2 and D 3 D^3 D 3 , respectively. At p = 2 p = 2 p = 2 , the norms coincide: choosing σ F = σ \sigma_F = \sigma σ F = σ , α = 1 \alpha= 1 α = 1 and κ = 1 \kappa= 1 κ = 1 makes the bounds agree.
For each fixed 2 < p < ∞ 2 < p < \infty 2 < p < ∞ , the displayed native coefficient is bounded by a constant depending on p p p times D 2 / p ν 2 D^{2/p}\nu^2 D 2/ p ν 2 , while the converted coefficient grows as D ν 2 D\nu^2 D ν 2 . Since D 2 / p = o ( D ) D^{2/p} = o(D) D 2/ p = o ( D ) as D → ∞ D \to\infty D → ∞ , the native bound has smaller asymptotic dimension dependence. At p = ∞ p = \infty p = ∞ , its coefficients have orders ln ( e D ) ν 2 \ln(eD)\nu^2 ln ( eD ) ν 2 for Rademacher perturbations and ( ln ( e D ) ) 2 ν 2 (\ln(eD))^2\nu^2 ( ln ( eD ) ) 2 ν 2 for the chosen Gaussian scales, compared with D ν 2 D\nu^2 D ν 2 for conversion. Thus the ratios C F / C ∗ C_F/C_* C F / C ∗ have orders D / ln ( e D ) D/\ln(eD) D / ln ( eD ) and D / ( ln ( e D ) ) 2 D/(\ln(eD))^2 D / ( ln ( eD ) ) 2 , respectively. For finite dimensions, the native coefficient can still coincide with or exceed the converted one.
These outcomes depend on the noise model. For example, the one-sparse perturbations in Remark 3 give equal coefficients throughout 1 ≤ p ≤ 2 1 \le p \le2 1 ≤ p ≤ 2 , so the larger native coefficient for dense noise below p = 2 p = 2 p = 2 is not universal. We therefore compare the full coefficients rather than infer an advantage from p p p alone, and claim no uniform improvement over Euclidean/Frobenius conversion. In the vector case, dual p = ∞ p = \infty p = ∞ corresponds to an ℓ 1 \ell_1 ℓ 1 update norm, while dual p = 1 p = 1 p = 1 corresponds to an ℓ ∞ \ell_\infty ℓ ∞ update norm. For matrices, dual p = ∞ p = \infty p = ∞ corresponds to the nuclear update norm, whose dual is the spectral norm. The analogous improvement here concerns diagonal perturbations, not independent noise in all matrix entries (Remark 4).
We now specify the oracle models and common assumptions, derive the complexity comparison, and give the detailed vector and matrix results.
Assumption 4 (Frobenius light-tail oracle). For every x ∈ E x \in E x ∈ E , the stochastic gradient is unbiased, E ξ [ ∇ f ( x ; ξ ) ] = ∇ f ( x ) \mathbb{E}_{\xi}[\nabla f(x;\xi)] = \nabla f(x) E ξ [ ∇ f ( x ; ξ )] = ∇ f ( x ) , and its error satisfies
E ξ [ exp ( ∥ ∇ f ( x ; ξ ) − ∇ f ( x ) ∥ F 2 σ F 2 ) ] ≤ exp ( 1 ) (55) \mathbb{E}_{\xi}\left[\exp\left(\frac{\|\nabla f(x;\xi)-\nabla f(x)\|_F^2}{\sigma_F^2}\right)\right] \le\exp(1) \tag*{(55)} E ξ [ exp ( σ F 2 ∥∇ f ( x ; ξ ) − ∇ f ( x ) ∥ F 2 ) ] ≤ exp ( 1 ) (55)
for some σ F > 0 \sigma_F > 0 σ F > 0 .
Since E ∗ E^* E ∗ is finite-dimensional, there are constants α , β > 0 \alpha,\beta> 0 α , β > 0 with
α ∥ s ∥ ∗ ≤ ∥ s ∥ F ≤ β ∥ s ∥ ∗ , ∀ s ∈ E ∗ . (56) \alpha\|s\|_* \le\|s\|_F \le\beta\|s\|_*, \qquad\forall s \in E^*. \tag*{(56)} α ∥ s ∥ ∗ ≤ ∥ s ∥ F ≤ β ∥ s ∥ ∗ , ∀ s ∈ E ∗ . (56)
Only the left inequality is used below, in the form ∥ s ∥ ∗ ≤ α − 1 ∥ s ∥ F \|s\|_* \le\alpha^{-1}\|s\|_F ∥ s ∥ ∗ ≤ α − 1 ∥ s ∥ F : it is what turns a Frobenius bound into a bound in ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ , and α − 1 \alpha^{-1} α − 1 is the entire price of doing so.
The converted analysis first concentrates the minibatch error in the Euclidean/Frobenius norm, whose regularity constant is 1, and then uses ∥ g k − ∇ f ( x k ) ∥ ∗ ≤ α − 1 ∥ g k − ∇ f ( x k ) ∥ F \lVert g^k-\nabla f(x^k)\rVert_* \le\alpha^{-1}\lVert g^k-\nabla f(x^k)\rVert_F ∥ g k − ∇ f ( x k ) ∥ ∗ ≤ α − 1 ∥ g k − ∇ f ( x k ) ∥ F . Consequently, its statistical coefficient is α − 2 σ F 2 \alpha^{-2}\sigma_F^2 α − 2 σ F 2 , with no additional factor of κ \kappa κ ; Lemma 21 gives the concentration bound used below.
Two perturbation models. For u = ( u 1 , … , u r ) ∈ R r u=(u_1,\ldots,u_r)\in\mathbb{R}^r u = ( u 1 , … , u r ) ∈ R r we write Diag m , n ( u ) \operatorname{Diag}_{m,n}(u) Diag m , n ( u ) for the rectangular diagonal embedding, defined entrywise for 1 ≤ i ≤ m 1\le i\le m 1 ≤ i ≤ m and 1 ≤ j ≤ n 1\le j\le n 1 ≤ j ≤ n by
[ Diag m , n ( u ) ] i j : = { u i , i = j ≤ r , 0 , otherwise . (57) [\operatorname{Diag}_{m,n}(u)]_{ij}:=
\begin{cases}
u_i, & i=j\le r,\\
0, & \text{otherwise}.
\end{cases} \tag*{(57)} [ Diag m , n ( u ) ] ij := { u i , 0 , i = j ≤ r , otherwise . (57)
The comparison is carried out on two canonical oracles.
Definition 9 (Rademacher perturbed-gradient model). The oracle is ∇ f ( x ; ξ ) = ∇ f ( x ) + ζ ( ξ ) \nabla f(x;\xi)=\nabla f(x)+\zeta(\xi) ∇ f ( x ; ξ ) = ∇ f ( x ) + ζ ( ξ ) with E [ ζ ( ξ ) ] = 0 \mathbb{E}[\zeta(\xi)]=0 E [ ζ ( ξ )] = 0 . In the vector case ζ : = ν χ \zeta:=\nu\chi ζ := ν χ with χ : = ( χ 1 , … , χ d ) \chi:=(\chi_1,\ldots,\chi_d) χ := ( χ 1 , … , χ d ) , and in the matrix case ζ : = ν Diag m , n ( ( χ i ) i = 1 r ) \zeta:=\nu\operatorname{Diag}_{m,n}((\chi_i)_{i=1}^r) ζ := ν Diag m , n (( χ i ) i = 1 r ) , where ν > 0 \nu>0 ν > 0 and { χ i } i ≥ 1 \{\chi_i\}_{i\ge1} { χ i } i ≥ 1 are independent Rademacher variables, Prob { χ i = 1 } = Prob { χ i = − 1 } = 1 2 \operatorname{Prob}\{\chi_i=1\}=\operatorname{Prob}\{\chi_i=-1\}=\frac{1}{2} Prob { χ i = 1 } = Prob { χ i = − 1 } = 2 1 . A fresh perturbation is drawn at every oracle call.
Definition 10 (Gaussian perturbed-gradient model). The oracle is ∇ f ( x ; ξ ) = ∇ f ( x ) + ζ ( ξ ) \nabla f(x;\xi)=\nabla f(x)+\zeta(\xi) ∇ f ( x ; ξ ) = ∇ f ( x ) + ζ ( ξ ) with E [ ζ ( ξ ) ] = 0 \mathbb{E}[\zeta(\xi)]=0 E [ ζ ( ξ )] = 0 . In the vector case ζ : = ν ω \zeta:=\nu\omega ζ := ν ω with ω : = ( ω 1 , … , ω d ) \omega:=(\omega_1,\ldots,\omega_d) ω := ( ω 1 , … , ω d ) , and in the matrix case ζ : = ν Diag m , n ( ( ω i ) i = 1 r ) \zeta:=\nu\operatorname{Diag}_{m,n}((\omega_i)_{i=1}^r) ζ := ν Diag m , n (( ω i ) i = 1 r ) , where ν > 0 \nu>0 ν > 0 and { ω i } i ≥ 1 \{\omega_i\}_{i\ge1} { ω i } i ≥ 1 are i.i.d. standard Gaussian variables, ω i ∼ N ( 0 , 1 ) \omega_i\sim\mathcal{N}(0,1) ω i ∼ N ( 0 , 1 ) . A fresh perturbation is drawn at every oracle call.
Both matrix models are diagonal, so the perturbation carries exactly r r r independent coordinates rather than m n mn mn . Write D D D for that number of independent coordinates: D = d D=d D = d in the vector model and D = r D=r D = r in the matrix model. The Gaussian model additionally needs the shape factor
ϑ p ( D ) : = { 1 , 1 ≤ p ≤ 2 , min { p , ln ( e D ) } , 2 < p ≤ ∞ , (58) \vartheta_p(D):=
\begin{cases}
1, & 1\le p\le2,\\
\min\{p,\ln(eD)\}, & 2<p\le\infty,
\end{cases} \tag*{(58)} ϑ p ( D ) := { 1 , min { p , ln ( eD )} , 1 ≤ p ≤ 2 , 2 < p ≤ ∞ , (58)
again with D 2 / ∞ : = 1 D^{2/\infty}:=1 D 2/∞ := 1 . With this notation the two analyses can be placed side by side before we turn to the proofs.
Common Assumptions and Complexity Bounds
Throughout this comparison, suppose that Assumption 1 and Assumption 2 hold with L > 0 L>0 L > 0 . Let κ \kappa κ denote a regularity constant of the dual space ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*,\lVert\cdot\rVert_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) . Fix an initial point x 0 x^0 x 0 with Δ : = f ( x 0 ) − f inf > 0 \Delta:=f(x^0)-f^{\inf}>0 Δ := f ( x 0 ) − f i n f > 0 , an accuracy 0 < ε ≤ L Δ 0<\varepsilon\le L\Delta 0 < ε ≤ L Δ , and a failure probability δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Both routes use the update (7) with the same update norm and fixed Borel-measurable sharp selection. Each iteration draws fresh i.i.d. oracle samples independently of the preceding history. The Frobenius route uses Assumption 4, whereas the geometry-native route uses Assumption 3. In both cases, choose
γ k ≡ 1 2 L , T : = ⌈ 128 L Δ 15 ε ⌉ . \gamma_k\equiv\frac{1}{2L},\qquad T:=\left\lceil\frac{128L\Delta}{15\varepsilon}\right\rceil. γ k ≡ 2 L 1 , T := ⌈ 15 ε 128 L Δ ⌉ .
The two analyses differ only in the minibatch size they prescribe. The Frobenius route pays the conversion constant,
B F : = max { ⌈ 16 α − 2 σ F 2 ε ( 2 + 6 log ( T / δ ) ) 2 ⌉ , 1 } . (59) B_{\mathrm F}:=\max\left\{\left\lceil\frac{16\alpha^{-2}\sigma_{\mathrm F}^2}{\varepsilon}\left(\sqrt{2}+\sqrt{6\log(T/\delta)}\right)^2\right\rceil,1\right\}. \tag*{(59)} B F := max { ⌈ ε 16 α − 2 σ F 2 ( 2 + 6 log ( T / δ ) ) 2 ⌉ , 1 } . (59)
whereas the geometry-native route pays the regularity constant instead,
B ∗ : = max { ⌈ 16 σ 2 ε ( 2 κ + 6 log ( T / δ ) ) 2 ⌉ , 1 } . (60) B_*:=\max\left\{\left\lceil\frac{16\sigma^2}{\varepsilon}\left(\sqrt{2\kappa}+\sqrt{6\log(T/\delta)}\right)^2\right\rceil,1\right\}. \tag*{(60)} B ∗ := max { ⌈ ε 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 ⌉ , 1 } . (60)
To justify these choices, condition on the history F k \mathcal{F}_k F k before iteration k k k , which fixes x k x^k x k . Apply Lemma 21 for the Frobenius route and Lemma 1 for the geometry-native route, with λ = 3 log ( T / δ ) \lambda=\sqrt{3\log(T/\delta)} λ = 3 log ( T / δ ) . The corresponding batch size makes the concentration threshold at most ε / 4 \sqrt{\varepsilon}/4 ε /4 , so the conditional probability of exceeding this error is at most exp ( − λ 2 / 3 ) = δ / T \exp(-\lambda^2/3)=\delta/T exp ( − λ 2 /3 ) = δ / T . Taking expectations of these conditional probabilities and applying a union bound gives, for either run,
Prob { ∃ k ∈ { 0 , … , T − 1 } : ∥ g k − ∇ f ( x k ) ∥ ∗ > ε 4 } ≤ T exp ( − λ 2 / 3 ) = δ . \operatorname{Prob}\left\{\exists k\in\{0,\ldots,T-1\}:\lVert g^k-\nabla f(x^k)\rVert_*>\frac{\sqrt{\varepsilon}}{4}\right\}\le T\exp(-\lambda^2/3)=\delta. Prob { ∃ k ∈ { 0 , … , T − 1 } : ∥ g k − ∇ f ( x k ) ∥ ∗ > 4 ε } ≤ T exp ( − λ 2 /3 ) = δ .
On the complementary event, the deterministic descent estimate in the proof of Theorem 1 gives
f ( x k + 1 ) ≤ f ( x k ) − 15 128 L ∥ ∇ f ( x k ) ∥ ∗ 2 f(x^{k+1}) \le f(x^k) - \frac{15}{128L}\lVert\nabla f(x^k) \rVert_*^2 f ( x k + 1 ) ≤ f ( x k ) − 128 L 15 ∥ ∇ f ( x k ) ∥ ∗ 2
whenever ∥ ∇ f ( x k ) ∥ ∗ 2 > ε \lVert\nabla f(x^k) \rVert_*^2 > \varepsilon ∥ ∇ f ( x k ) ∥ ∗ 2 > ε .
If this strict gradient inequality held at every iteration, summing the descent inequalities along that trajectory would yield
f ( x T ) < f ( x 0 ) − 15 T ε 128 L ≤ f inf , f(x^T) < f(x^0) - \frac{15T\varepsilon}{128L} \le f^{\inf}, f ( x T ) < f ( x 0 ) − 128 L 15 T ε ≤ f i n f ,
contradicting Assumption 2. Thus each run satisfies min 0 ≤ k < T ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε \min_{0\le k<T}\lVert\nabla f(x^k) \rVert_*^2 \le\varepsilon min 0 ≤ k < T ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε with probability at least 1 − δ 1-\delta 1 − δ .
Suppressing logarithmic factors, each minibatch size contributes a single coefficient to the statistical term,
C F : = α − 2 σ F 2 , C ∗ : = κ σ 2 . (61) C_{\mathrm{F}} := \alpha^{-2}\sigma_{\mathrm{F}}^2,\qquad C_* := \kappa\sigma^2. \tag*{(61)} C F := α − 2 σ F 2 , C ∗ := κ σ 2 . (61)
Since ε ≤ L Δ \varepsilon\le L\Delta ε ≤ L Δ , the ceiling in T T T gives T = O ( L Δ / ε ) T = \mathcal{O}(L\Delta/\varepsilon) T = O ( L Δ/ ε ) . The batch sizes satisfy B F = O ~ ( 1 + C F / ε ) B_{\mathrm{F}} = \widetilde{\mathcal{O}}(1 + C_{\mathrm{F}}/\varepsilon) B F = O ( 1 + C F / ε ) and B ∗ = O ~ ( 1 + C ∗ / ε ) B_* = \widetilde{\mathcal{O}}(1 + C_*/\varepsilon) B ∗ = O ( 1 + C ∗ / ε ) , where the suppressed factors are logarithmic in T T T and 1 / δ 1/\delta 1/ δ . Multiplying by T T T , the total numbers of oracle calls N F : = T B F N_{\mathrm{F}} := TB_{\mathrm{F}} N F := T B F and N ∗ : = T B ∗ N_* := TB_* N ∗ := T B ∗ therefore satisfy
N F = O ~ ( L Δ ε + L Δ ε 2 C F ) (62) N_{\mathrm{F}} = \widetilde{\mathcal{O}}\left(\frac{L\Delta}{\varepsilon} + \frac{L\Delta}{\varepsilon^2}C_{\mathrm{F}}\right) \tag*{(62)} N F = O ( ε L Δ + ε 2 L Δ C F ) (62)
= O ~ ( L Δ ε + L Δ σ F 2 ε 2 α 2 ) (63) = \widetilde{\mathcal{O}}\left(\frac{L\Delta}{\varepsilon} + \frac{L\Delta\sigma_{\mathrm{F}}^2}{\varepsilon^2\alpha^2}\right) \tag*{(63)} = O ( ε L Δ + ε 2 α 2 L Δ σ F 2 ) (63)
and
N ∗ = O ~ ( L Δ ε + L Δ ε 2 C ∗ ) . (64) N_* = \widetilde{\mathcal{O}}\left(\frac{L\Delta}{\varepsilon} + \frac{L\Delta}{\varepsilon^2}C_*\right). \tag*{(64)} N ∗ = O ( ε L Δ + ε 2 L Δ C ∗ ) . (64)
The first term is identical in both, so the comparison reduces, up to the suppressed logarithmic factors, to C F C_{\mathrm{F}} C F against C ∗ C_* C ∗ . These coefficients refer to the same oracle and update geometry, but use noise scales measured in different norms. We compare the coefficients rather than interpret their ratio as an exact ratio of oracle complexities, since both analyses provide upper bounds.
Side-by-side comparison of the noise terms. The tables below place the two statistical terms next to each other, separating the regimes 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 and 2 < p < ∞ 2 < p < \infty 2 < p < ∞ and displaying the boundary cases p = 1 p = 1 p = 1 , p = 2 p = 2 p = 2 and p = ∞ p = \infty p = ∞ in full, so that every dimension exponent can be read off directly rather than inferred. Four conventions apply throughout. We take ν 2 = O ( 1 ) \nu^2 = \mathcal{O}(1) ν 2 = O ( 1 ) to expose the dependence on dimension; a general ν \nu ν is restored by multiplying every displayed term by ν 2 \nu^2 ν 2 . A formula indexed by M ∈ { R a d , G a u } M \in\{\mathrm{Rad},\mathrm{Gau}\} M ∈ { Rad , Gau } holds for both perturbation models, the numerical Gaussian factor 4 4 4 being absorbed into O ~ \widetilde{\mathcal{O}} O ; the two models are written separately only where they genuinely differ. The symbols κ ℓ p ( d ) \kappa_{\ell_p}(d) κ ℓ p ( d ) and κ S p ( r ) \kappa_{S_p}(r) κ S p ( r ) are the admissible regularity parameters established in Section J.2. Finally, for every fixed finite p p p the two perturbation models carry the same dependence on dimension up to constants, and they part company only at p = ∞ p = \infty p = ∞ , where the admissible Gaussian scale costs one additional logarithmic factor.
One exponent is worth unpacking before the tables, since it is the only one that is not read off a single quantity. For 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 the regularity factor is κ = D 2 / p − 1 \kappa= D^{2/p-1} κ = D 2/ p − 1 . In the Rademacher model, the native noise scale is σ 2 = D 2 / p ν 2 \sigma^2 = D^{2/p}\nu^2 σ 2 = D 2/ p ν 2 , so the exact coefficient is
C ∗ R a d = κ σ 2 = D 2 / p − 1 ⋅ D 2 / p ν 2 = D 4 / p − 1 ν 2 . \begin{aligned}
C_*^{\mathrm{Rad}} = \kappa\sigma^2 \\
&= D^{2/p-1}\cdot D^{2/p}\nu^2 \\
&= D^{4/p-1}\nu^2.
\end{aligned} C ∗ Rad = κ σ 2 = D 2/ p − 1 ⋅ D 2/ p ν 2 = D 4/ p − 1 ν 2 .
In the Gaussian model, the chosen admissible scale σ 2 = 4 D 2 / p ν 2 \sigma^2 = 4D^{2/p}\nu^2 σ 2 = 4 D 2/ p ν 2 instead gives
C ∗ G a u = κ σ 2 = D 2 / p − 1 ⋅ 4 D 2 / p ν 2 = 4 D 4 / p − 1 ν 2 . C_*^{\mathrm{Gau}} = \kappa\sigma^2 = D^{2/p-1}\cdot4D^{2/p}\nu^2 = 4D^{4/p-1}\nu^2. C ∗ Gau = κ σ 2 = D 2/ p − 1 ⋅ 4 D 2/ p ν 2 = 4 D 4/ p − 1 ν 2 .
Thus both models have the dimension exponent 4 / p − 1 4/p - 1 4/ p − 1 ; only the absolute factor differs.
ℓ p \ell_p ℓ p spaces. Read the two columns against each other row by row. The left column never improves on d d d and degrades to d 2 / p d^{2/p} d 2/ p below p = 2 p=2 p = 2 . The native coefficient has a larger dimension exponent for p < 2 p<2 p < 2 , and the two coefficients coincide at p = 2 p=2 p = 2 . For each fixed 2 < p < ∞ 2<p<\infty 2 < p < ∞ , the native coefficient has a smaller asymptotic order as d → ∞ d\to\infty d → ∞ . At p = ∞ p=\infty p = ∞ , its dependence is logarithmic for Rademacher noise and squared-logarithmic for Gaussian noise, again asymptotically smaller than linear. These comparisons do not guarantee strict improvement in every finite dimension.
Euclidean/Frobenius approach
For 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 ,
N F , n o i s e M = O ~ ( d 2 / p L Δ ε 2 ) . N_{\mathrm{F,noise}}^{\mathrm{M}}=\widetilde{\mathcal{O}}\left(\frac{d^{2/p}L\Delta}{\varepsilon^2}\right). N F , noise M = O ( ε 2 d 2/ p L Δ ) .
In particular, at p = 1 p=1 p = 1 ,
N F , n o i s e M ∣ p = 1 = O ~ ( d 2 L Δ ε 2 ) . \left.N_{\mathrm{F,noise}}^{\mathrm{M}}\right|_{p=1}=\widetilde{\mathcal{O}}\left(\frac{d^2L\Delta}{\varepsilon^2}\right). N F , noise M p = 1 = O ( ε 2 d 2 L Δ ) .
For every 2 ≤ p ≤ ∞ 2\le p\le\infty 2 ≤ p ≤ ∞ ,
N F , n o i s e M = O ~ ( d L Δ ε 2 ) . N_{\mathrm{F,noise}}^{\mathrm{M}}=\widetilde{\mathcal{O}}\left(\frac{dL\Delta}{\varepsilon^2}\right). N F , noise M = O ( ε 2 d L Δ ) .
Geometry-native approach
For 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 ,
N ∗ , n o i s e M = O ~ ( d 4 / p − 1 L Δ ε 2 ) . N_{\ast,\mathrm{noise}}^{\mathrm{M}}=\widetilde{\mathcal{O}}\left(\frac{d^{4/p-1}L\Delta}{\varepsilon^2}\right). N ∗ , noise M = O ( ε 2 d 4/ p − 1 L Δ ) .
In particular, at p = 1 p=1 p = 1 ,
N ∗ , n o i s e M ∣ p = 1 = O ~ ( d 3 L Δ ε 2 ) . \left.N_{\ast,\mathrm{noise}}^{\mathrm{M}}\right|_{p=1}=\widetilde{\mathcal{O}}\left(\frac{d^3L\Delta}{\varepsilon^2}\right). N ∗ , noise M p = 1 = O ( ε 2 d 3 L Δ ) .
At p = 2 p=2 p = 2 ,
N ∗ , n o i s e M ∣ p = 2 = O ~ ( d L Δ ε 2 ) . \left.N_{\ast,\mathrm{noise}}^{\mathrm{M}}\right|_{p=2}=\widetilde{\mathcal{O}}\left(\frac{dL\Delta}{\varepsilon^2}\right). N ∗ , noise M p = 2 = O ( ε 2 d L Δ ) .
For 2 < p < ∞ 2<p<\infty 2 < p < ∞ ,
N ∗ , n o i s e R a d = O ~ ( κ ℓ p ( d ) d 2 / p L Δ ε 2 ) , N_{\ast,\mathrm{noise}}^{\mathrm{Rad}}=\widetilde{\mathcal{O}}\left(\frac{\kappa_{\ell_p}(d)d^{2/p}L\Delta}{\varepsilon^2}\right), N ∗ , noise Rad = O ( ε 2 κ ℓ p ( d ) d 2/ p L Δ ) ,
N ∗ , n o i s e G a u = O ~ ( κ ℓ p ( d ) ϑ p ( d ) d 2 / p L Δ ε 2 ) . N_{\ast,\mathrm{noise}}^{\mathrm{Gau}}=\widetilde{\mathcal{O}}\left(\frac{\kappa_{\ell_p}(d)\vartheta_p(d)d^{2/p}L\Delta}{\varepsilon^2}\right). N ∗ , noise Gau = O ( ε 2 κ ℓ p ( d ) ϑ p ( d ) d 2/ p L Δ ) .
At p = ∞ p=\infty p = ∞ , for every d ≥ 1 d\ge1 d ≥ 1 ,
N ∗ , n o i s e R a d ∣ p = ∞ = O ~ ( ln ( e d ) L Δ ε 2 ) , \left.N_{\ast,\mathrm{noise}}^{\mathrm{Rad}}\right|_{p=\infty}=\widetilde{\mathcal{O}}\left(\frac{\ln(ed)L\Delta}{\varepsilon^2}\right), N ∗ , noise Rad p = ∞ = O ( ε 2 ln ( e d ) L Δ ) ,
N ∗ , n o i s e G a u ∣ p = ∞ = O ~ ( ( ln ( e d ) ) 2 L Δ ε 2 ) . \left.N_{\ast,\mathrm{noise}}^{\mathrm{Gau}}\right|_{p=\infty}=\widetilde{\mathcal{O}}\left(\frac{(\ln(ed))^2L\Delta}{\varepsilon^2}\right). N ∗ , noise Gau p = ∞ = O ( ε 2 ( ln ( e d ) ) 2 L Δ ) .
Spaces with Schatten-p p p norms. The diagonal perturbations in Definition 9 and Definition 10 allow the vector noise-scale calculations to carry over with d d d replaced by r = min { m , n } r=\min\{m,n\} r = min { m , n } . For u ∈ R ν > 0 u\in\mathbb{R}^{\nu}>0 u ∈ R ν > 0 , the singular values of ν Diag m , n ( u ) \nu\operatorname{Diag}_{m,n}(u) ν Diag m , n ( u ) are the numbers ν ∣ u i ∣ \nu|u_i| ν ∣ u i ∣ arranged in nonincreasing order. Consequently,
∥ ν Diag m , n ( u ) ∥ S p = ν ∥ u ∥ p , 1 ≤ p ≤ ∞ . \left\|\nu\operatorname{Diag}_{m,n}(u)\right\|_{S_p}=\nu\|u\|_p,\qquad1\leq p\leq\infty. ν Diag m , n ( u ) S p = ν ∥ u ∥ p , 1 ≤ p ≤ ∞.
The resulting statistical coefficients have analogous asymptotic dimension dependence, but use the separately defined Schatten regularity parameter κ S p ( r ) \kappa_{S_p}(r) κ S p ( r ) . It need not equal κ ℓ p ( d ) \kappa_{\ell_p}(d) κ ℓ p ( d ) even when d = r d=r d = r .
Euclidean/Frobenius approach
For 1 ≤ p < 2 1\leq p<2 1 ≤ p < 2 ,
N F , n o i s e M = O ~ ( r 2 / p L Δ ε 2 ) . N_{\mathrm{F},\mathrm{noise}}^{\mathrm{M}}=\widetilde{\mathcal{O}}\left(\frac{r^{2/p}L\Delta}{\varepsilon^2}\right). N F , noise M = O ( ε 2 r 2/ p L Δ ) .
In particular, at p = 1 p=1 p = 1 ,
N F , n o i s e M ∣ p = 1 = O ~ ( r 2 L Δ ε 2 ) . \left.N_{\mathrm{F},\mathrm{noise}}^{\mathrm{M}}\right|_{p=1}=\widetilde{\mathcal{O}}\left(\frac{r^2L\Delta}{\varepsilon^2}\right). N F , noise M p = 1 = O ( ε 2 r 2 L Δ ) .
For every 2 ≤ p ≤ ∞ 2\leq p\leq\infty 2 ≤ p ≤ ∞ ,
N F , n o i s e M = O ~ ( r L Δ ε 2 ) . N_{\mathrm{F},\mathrm{noise}}^{\mathrm{M}}=\widetilde{\mathcal{O}}\left(\frac{rL\Delta}{\varepsilon^2}\right). N F , noise M = O ( ε 2 r L Δ ) .
Geometry-native approach
For 1 ≤ p < 2 1\leq p<2 1 ≤ p < 2 ,
N ∗ , n o i s e M = O ~ ( r 4 / p − 1 L Δ ε 2 ) . N_{\ast,\mathrm{noise}}^{\mathrm{M}}=\widetilde{\mathcal{O}}\left(\frac{r^{4/p-1}L\Delta}{\varepsilon^2}\right). N ∗ , noise M = O ( ε 2 r 4/ p − 1 L Δ ) .
In particular, at p = 1 p=1 p = 1 ,
N ∗ , n o i s e M ∣ p = 1 = O ~ ( r 3 L Δ ε 2 ) . \left.N_{\ast,\mathrm{noise}}^{\mathrm{M}}\right|_{p=1}=\widetilde{\mathcal{O}}\left(\frac{r^3L\Delta}{\varepsilon^2}\right). N ∗ , noise M p = 1 = O ( ε 2 r 3 L Δ ) .
At p = 2 p=2 p = 2 ,
N ∗ , n o i s e M ∣ p = 2 = O ~ ( r L Δ ε 2 ) . \left.N_{\ast,\mathrm{noise}}^{\mathrm{M}}\right|_{p=2}=\widetilde{\mathcal{O}}\left(\frac{rL\Delta}{\varepsilon^2}\right). N ∗ , noise M p = 2 = O ( ε 2 r L Δ ) .
For 2 < p < ∞ 2<p<\infty 2 < p < ∞ ,
N ∗ , n o i s e R a d = O ~ ( κ S p ( r ) r 2 / p L Δ ε 2 ) , N_{\ast,\mathrm{noise}}^{\mathrm{Rad}}=\widetilde{\mathcal{O}}\left(\frac{\kappa_{S_p}(r)r^{2/p}L\Delta}{\varepsilon^2}\right), N ∗ , noise Rad = O ( ε 2 κ S p ( r ) r 2/ p L Δ ) ,
N ∗ , n o i s e G a u = O ~ ( κ S p ( r ) ϑ p ( r ) r 2 / p L Δ ε 2 ) . N_{\ast,\mathrm{noise}}^{\mathrm{Gau}}=\widetilde{\mathcal{O}}\left(\frac{\kappa_{S_p}(r)\vartheta_p(r)r^{2/p}L\Delta}{\varepsilon^2}\right). N ∗ , noise Gau = O ( ε 2 κ S p ( r ) ϑ p ( r ) r 2/ p L Δ ) .
At p = ∞ p=\infty p = ∞ , for every r ≥ 1 r\geq1 r ≥ 1 ,
N ∗ , n o i s e R a d ∣ p = ∞ = O ~ ( ln ( e r ) L Δ ε 2 ) , \left.N_{\ast,\mathrm{noise}}^{\mathrm{Rad}}\right|_{p=\infty}=\widetilde{\mathcal{O}}\left(\frac{\ln(er)L\Delta}{\varepsilon^2}\right), N ∗ , noise Rad p = ∞ = O ( ε 2 ln ( er ) L Δ ) ,
N ∗ , n o i s e G a u ∣ p = ∞ = O ~ ( ( ln ( e r ) ) 2 L Δ ε 2 ) . \left.N_{\ast,\mathrm{noise}}^{\mathrm{Gau}}\right|_{p=\infty}=\widetilde{\mathcal{O}}\left(\frac{(\ln(er))^2L\Delta}{\varepsilon^2}\right). N ∗ , noise Gau p = ∞ = O ( ε 2 ( ln ( er ) ) 2 L Δ ) .
Detailed Results and Proofs
We first establish the Euclidean/Frobenius conversion bound used in the summary.
Lemma 21 (Frobenius-to-dual concentration). Let { η j } j = 1 B \{\eta^j\}_{j=1}^{B} { η j } j = 1 B be i.i.d. random elements of E ∗ E^* E ∗ such that E [ η j ] = 0 \mathbb{E}[\eta^j]=0 E [ η j ] = 0 and E [ exp ( ∥ η j ∥ F 2 σ F 2 ) ] ≤ exp ( 1 ) \mathbb{E}\left[\exp\left(\frac{\|\eta^j\|_F^2}{\sigma_F^2}\right)\right]\leq\exp(1) E [ exp ( σ F 2 ∥ η j ∥ F 2 ) ] ≤ exp ( 1 ) . Define η ˉ : = B − 1 ∑ j = 1 B η j \bar{\eta}:=B^{-1}\sum_{j=1}^{B}\eta^j η ˉ := B − 1 ∑ j = 1 B η j . Then, for every λ ≥ 0 \lambda\geq0 λ ≥ 0 ,
Prob { ∥ η ˉ ∥ ∗ ≥ 1 α ( 2 + 2 λ ) σ F B } ≤ exp ( − λ 2 / 3 ) . (65) \operatorname{Prob}\left\{\|\bar{\eta}\|_*\geq\frac{1}{\alpha}\left(\sqrt{2}+\sqrt{2\lambda}\right)\frac{\sigma_F}{\sqrt{B}}\right\}\leq\exp\left(-\lambda^2/3\right). \tag*{(65)} Prob { ∥ η ˉ ∥ ∗ ≥ α 1 ( 2 + 2 λ ) B σ F } ≤ exp ( − λ 2 /3 ) . (65)
Proof. The Euclidean space ( E ∗ , ∥ ⋅ ∥ F ) (E^*,\|\cdot\|_F) ( E ∗ , ∥ ⋅ ∥ F ) is 1-regular. Applying Lemma 1 in this space gives
Prob { ∥ η ˉ ∥ F ≥ ( 2 + 2 λ ) σ F B } ≤ exp ( − λ 2 / 3 ) . \operatorname{Prob}\left\{\|\bar{\eta}\|_F\geq\left(\sqrt{2}+\sqrt{2\lambda}\right)\frac{\sigma_F}{\sqrt{B}}\right\}\leq\exp\left(-\lambda^2/3\right). Prob { ∥ η ˉ ∥ F ≥ ( 2 + 2 λ ) B σ F } ≤ exp ( − λ 2 /3 ) .
For any t ≥ 0 t\geq0 t ≥ 0 , the left inequality in (56) implies the event inclusion
{ ∥ η ˉ ∥ ∗ ≥ t } ⊆ { ∥ η ˉ ∥ F ≥ α t } . \{\|\bar{\eta}\|_*\geq t\}\subseteq\{\|\bar{\eta}\|_F\geq\alpha t\}. { ∥ η ˉ ∥ ∗ ≥ t } ⊆ { ∥ η ˉ ∥ F ≥ α t } .
Taking
t : = 1 α ( 2 + 2 λ ) σ F B t:=\frac{1}{\alpha}\left(\sqrt{2}+\sqrt{2\lambda}\right)\frac{\sigma_F}{\sqrt{B}} t := α 1 ( 2 + 2 λ ) B σ F
proves (65) . ∎
We next establish the Gaussian light-tail scale used for both norm families.
Lemma 22 (Gaussian light tails in ℓ p \ell_p ℓ p ). Let ω ∼ N ( 0 , I D ) \omega\sim\mathcal{N}(0,I_D) ω ∼ N ( 0 , I D ) and p ∈ [ 1 , ∞ ] p \in[1,\infty] p ∈ [ 1 , ∞ ] . For every ν > 0 \nu> 0 ν > 0 ,
E [ exp ( ∥ ν ω ∥ p 2 4 ϑ p ( D ) D 2 / p ν 2 ) ] ≤ exp ( 1 ) . (66) \mathbb{E}\left[\exp\left(\frac{\|\nu\omega\|_p^2}{4\vartheta_p(D)D^{2/p}\nu^2}\right)\right]\leq\exp(1). \tag*{(66)} E [ exp ( 4 ϑ p ( D ) D 2/ p ν 2 ∥ ν ω ∥ p 2 ) ] ≤ exp ( 1 ) . (66)
Proof. We first consider 1 ≤ p ≤ 2 1 \leq p \leq2 1 ≤ p ≤ 2 . The norm comparison
∥ ω ∥ p 2 ≤ D 2 p − 1 ∥ ω ∥ 2 2 \|\omega\|_p^2 \leq D^{\frac{2}{p}-1}\|\omega\|_2^2 ∥ ω ∥ p 2 ≤ D p 2 − 1 ∥ ω ∥ 2 2
gives
E [ exp ( ∥ ν ω ∥ p 2 4 D 2 / p ν 2 ) ] ≤ E [ exp ( ∥ ω ∥ 2 2 4 D ) ] . \mathbb{E}\left[\exp\left(\frac{\|\nu\omega\|_p^2}{4D^{2/p}\nu^2}\right)\right]\leq\mathbb{E}\left[\exp\left(\frac{\|\omega\|_2^2}{4D}\right)\right]. E [ exp ( 4 D 2/ p ν 2 ∥ ν ω ∥ p 2 ) ] ≤ E [ exp ( 4 D ∥ ω ∥ 2 2 ) ] .
Since ∥ ω ∥ 2 2 \|\omega\|_2^2 ∥ ω ∥ 2 2 has a chi-squared distribution with D D D degrees of freedom,
E [ exp ( ∥ ω ∥ 2 2 4 D ) ] = ( 1 − 1 2 D ) − D / 2 . \mathbb{E}\left[\exp\left(\frac{\|\omega\|_2^2}{4D}\right)\right]=\left(1-\frac{1}{2D}\right)^{-D/2}. E [ exp ( 4 D ∥ ω ∥ 2 2 ) ] = ( 1 − 2 D 1 ) − D /2 .
For x ∈ [ 0 , 1 ) x \in[0,1) x ∈ [ 0 , 1 ) , the inequality − ln ( 1 − x ) ≤ x / ( 1 − x ) -\ln(1-x)\leq x/(1-x) − ln ( 1 − x ) ≤ x / ( 1 − x ) yields
− D 2 ln ( 1 − 1 2 D ) ≤ 1 4 1 1 − 1 2 D ≤ 1 2 < 1. -\frac{D}{2}\ln\left(1-\frac{1}{2D}\right)\leq\frac{1}{4}\frac{1}{1-\frac{1}{2D}}\leq\frac{1}{2}<1. − 2 D ln ( 1 − 2 D 1 ) ≤ 4 1 1 − 2 D 1 1 ≤ 2 1 < 1.
Thus (66) holds for 1 ≤ p ≤ 2 1 \leq p \leq2 1 ≤ p ≤ 2 .
Now let 2 < p ≤ ∞ 2 < p \leq\infty 2 < p ≤ ∞ and set
m p : = E [ ∥ ω ∥ p ] . m_p:=\mathbb{E}\left[\|\omega\|_p\right]. m p := E [ ∥ ω ∥ p ] .
For finite p p p , Jensen’s inequality and the Gaussian moment formula give
m p ≤ ( E [ ∥ ω ∥ p p ] ) 1 / p = D 1 / p ( E [ ∣ ω 1 ∣ p ] ) 1 / p = D 1 / p ( 2 p / 2 Γ ( ( p + 1 ) / 2 ) π ) 1 / p ≤ D 1 / p p . \begin{aligned}
m_p\leq\left(\mathbb{E}\left[\|\omega\|_p^p\right]\right)^{1/p} \\
&=D^{1/p}\left(\mathbb{E}\left[|\omega_1|^p\right]\right)^{1/p} \\
&=D^{1/p}\left(\frac{2^{p/2}\Gamma((p+1)/2)}{\sqrt{\pi}}\right)^{1/p} \\
&\leq D^{1/p}\sqrt{p}.
\end{aligned} m p ≤ ( E [ ∥ ω ∥ p p ] ) 1/ p = D 1/ p ( E [ ∣ ω 1 ∣ p ] ) 1/ p = D 1/ p ( π 2 p /2 Γ (( p + 1 ) /2 ) ) 1/ p ≤ D 1/ p p .
The last inequality is the standard Gaussian moment bound ( E [ ∣ ω 1 ∣ p ] ) 1 / p ≤ p (\mathbb{E}[|\omega_1|^p])^{1/p}\leq\sqrt{p} ( E [ ∣ ω 1 ∣ p ] ) 1/ p ≤ p . A second bound follows from
∥ ω ∥ p ≤ D 1 / p ∥ ω ∥ ∞ . \|\omega\|_p\leq D^{1/p}\|\omega\|_\infty. ∥ ω ∥ p ≤ D 1/ p ∥ ω ∥ ∞ .
For any t > 0 t>0 t > 0 , we first have
E [ exp ( t ∥ ω ∥ ∞ ) ] ≤ ∑ i = 1 D E [ exp ( t ∣ ω i ∣ ) ] \mathbb{E}[\exp(t\|\omega\|_\infty)]\leq\sum_{i=1}^{D}\mathbb{E}[\exp(t|\omega_i|)] E [ exp ( t ∥ ω ∥ ∞ )] ≤ i = 1 ∑ D E [ exp ( t ∣ ω i ∣ )]
≤ 2 D exp ( t 2 / 2 ) . \leq2D\exp(t^2/2). ≤ 2 D exp ( t 2 /2 ) .
The first inequality uses
exp ( t max 1 ≤ i ≤ D ∣ ω i ∣ ) ≤ ∑ i = 1 D exp ( t ∣ ω i ∣ ) , \exp\left(t\max_{1\leq i\leq D}|\omega_i|\right)\leq\sum_{i=1}^{D}\exp(t|\omega_i|), exp ( t 1 ≤ i ≤ D max ∣ ω i ∣ ) ≤ i = 1 ∑ D exp ( t ∣ ω i ∣ ) ,
and the second uses
E [ exp ( t ∣ ω i ∣ ) ] ≤ E [ exp ( t ω i ) ] + E [ exp ( − t ω i ) ] = 2 exp ( t 2 / 2 ) . \begin{aligned}
\mathbb{E}[\exp(t|\omega_i|)]\leq\mathbb{E}[\exp(t\omega_i)]+\mathbb{E}[\exp(-t\omega_i)] \\
&=2\exp(t^2/2).
\end{aligned} E [ exp ( t ∣ ω i ∣ )] ≤ E [ exp ( t ω i )] + E [ exp ( − t ω i )] = 2 exp ( t 2 /2 ) .
Consequently,
E [ ∥ ω ∥ ∞ ] ≤ 1 t ln E [ exp ( t ∥ ω ∥ ∞ ) ] ≤ ln ( 2 D ) t + t 2 . \mathbb{E}\left[\lVert\omega\rVert_{\infty}\right] \le\frac{1}{t}\ln\mathbb{E}\left[\exp\left(t\lVert\omega\rVert_{\infty}\right)\right] \le\frac{\ln(2D)}{t}+\frac{t}{2}. E [ ∥ ω ∥ ∞ ] ≤ t 1 ln E [ exp ( t ∥ ω ∥ ∞ ) ] ≤ t ln ( 2 D ) + 2 t .
Taking t = 2 ln ( 2 D ) t=\sqrt{2\ln(2D)} t = 2 ln ( 2 D ) gives
E [ ∥ ω ∥ ∞ ] ≤ 2 ln ( 2 D ) . \mathbb{E}\left[\lVert\omega\rVert_{\infty}\right] \le\sqrt{2\ln(2D)}. E [ ∥ ω ∥ ∞ ] ≤ 2 ln ( 2 D ) .
Combining the two estimates, including the second one at p = ∞ p=\infty p = ∞ , gives
m p 2 ≤ min { p , 2 ln ( 2 D ) } D 2 / p . (67) m_p^2 \le\min\{p,2\ln(2D)\}D^{2/p}. \tag*{(67)} m p 2 ≤ min { p , 2 ln ( 2 D )} D 2/ p . (67)
Set
H p ( D ) : = min { p , ln ( e D ) } D 2 / p . H_p(D):=\min\{p,\ln(eD)\}D^{2/p}. H p ( D ) := min { p , ln ( eD )} D 2/ p .
If D = 1 D=1 D = 1 , then H p ( 1 ) = 1 H_p(1)=1 H p ( 1 ) = 1 and ∥ ω ∥ p = ∣ ω 1 ∣ \lVert\omega\rVert_p=|\omega_1| ∥ ω ∥ p = ∣ ω 1 ∣ for every p > 2 p>2 p > 2 . Therefore,
E [ exp ( ∥ ω ∥ p 2 4 H p ( 1 ) ) ] = E [ exp ( ω 1 2 4 ) ] = 2 < exp ( 1 ) . \mathbb{E}\left[\exp\left(\frac{\lVert\omega\rVert_p^2}{4H_p(1)}\right)\right]
=\mathbb{E}\left[\exp\left(\frac{\omega_1^2}{4}\right)\right]
=\sqrt{2}
<\exp(1). E [ exp ( 4 H p ( 1 ) ∥ ω ∥ p 2 ) ] = E [ exp ( 4 ω 1 2 ) ] = 2 < exp ( 1 ) .
It remains to consider D ≥ 2 D\ge2 D ≥ 2 .
For p > 2 p>2 p > 2 , the map z ↦ ∥ z ∥ p z\mapsto\lVert z\rVert_p z ↦ ∥ z ∥ p is 1-Lipschitz with respect to ∥ ⋅ ∥ 2 \lVert\cdot\rVert_2 ∥ ⋅ ∥ 2 , because
∣ ∥ u ∥ p − ∥ v ∥ p ∣ ≤ ∥ u − v ∥ p ≤ ∥ u − v ∥ 2 . \left|\lVert u\rVert_p-\lVert v\rVert_p\right|\le\lVert u-v\rVert_p\le\lVert u-v\rVert_2. ∣ ∥ u ∥ p − ∥ v ∥ p ∣ ≤ ∥ u − v ∥ p ≤ ∥ u − v ∥ 2 .
The moment-generating-function form of Gaussian concentration, obtained from the Gaussian log-Sobolev inequality by the Herbst argument, therefore gives, for
X p : = ∥ ω ∥ p − m p , X_p:=\lVert\omega\rVert_p-m_p, X p := ∥ ω ∥ p − m p ,
the bound
E [ exp ( s X p ) ] ≤ exp ( s 2 / 2 ) , s ∈ R . (68) \mathbb{E}\left[\exp(sX_p)\right]\le\exp(s^2/2),\qquad s\in\mathbb{R}. \tag*{(68)} E [ exp ( s X p ) ] ≤ exp ( s 2 /2 ) , s ∈ R . (68)
Let γ ∼ N ( 0 , 1 ) \gamma\sim\mathcal{N}(0,1) γ ∼ N ( 0 , 1 ) be independent of ω \omega ω . Gaussian linearization and (68) give
E [ exp ( ∥ ω ∥ p 2 4 H p ( D ) ) ] = E ω , γ [ exp ( γ ∥ ω ∥ p 2 H p ( D ) ) ] = E γ [ exp ( γ m p 2 H p ( D ) ) E ω [ exp ( γ X p 2 H p ( D ) ) ] ] ≤ E γ [ exp ( γ m p 2 H p ( D ) + γ 2 4 H p ( D ) ) ] = ( 1 − 1 2 H p ( D ) ) − 1 / 2 exp ( m p 2 4 H p ( D ) − 2 ) . \begin{aligned}
\mathbb{E}\left[\exp\left(\frac{\lVert\omega\rVert_p^2}{4H_p(D)}\right)\right]
&=\mathbb{E}_{\omega,\gamma}\left[\exp\left(\frac{\gamma\lVert\omega\rVert_p}{\sqrt{2H_p(D)}}\right)\right]\\
&=\mathbb{E}_{\gamma}\left[\exp\left(\frac{\gamma m_p}{\sqrt{2H_p(D)}}\right)\mathbb{E}_{\omega}\left[\exp\left(\frac{\gamma X_p}{\sqrt{2H_p(D)}}\right)\right]\right]\\
&\le\mathbb{E}_{\gamma}\left[\exp\left(\frac{\gamma m_p}{\sqrt{2H_p(D)}}+\frac{\gamma^2}{4H_p(D)}\right)\right]\\
&=\left(1-\frac{1}{2H_p(D)}\right)^{-1/2}\exp\left(\frac{m_p^2}{4H_p(D)-2}\right).
\end{aligned} E [ exp ( 4 H p ( D ) ∥ ω ∥ p 2 ) ] = E ω , γ [ exp ( 2 H p ( D ) γ ∥ ω ∥ p ) ] = E γ [ exp ( 2 H p ( D ) γ m p ) E ω [ exp ( 2 H p ( D ) γ X p ) ] ] ≤ E γ [ exp ( 2 H p ( D ) γ m p + 4 H p ( D ) γ 2 ) ] = ( 1 − 2 H p ( D ) 1 ) − 1/2 exp ( 4 H p ( D ) − 2 m p 2 ) .
Suppose first that 2 < p ≤ ln ( e D ) 2<p\le\ln(eD) 2 < p ≤ ln ( eD ) . Then H p ( D ) = p D 2 / p > 2 H_p(D)=pD^{2/p}>2 H p ( D ) = p D 2/ p > 2 , and (67) gives m p 2 ≤ H p ( D ) m_p^2\le H_p(D) m p 2 ≤ H p ( D ) . Hence,
ln E [ exp ( ∥ ω ∥ p 2 4 H p ( D ) ) ] ≤ − 1 2 ln ( 1 − 1 2 H p ( D ) ) + m p 2 4 H p ( D ) − 2 < 1 2 ln ( 4 3 ) + 1 3 < 1. \begin{aligned}
\ln\mathbb{E}\left[\exp\left(\frac{\lVert\omega\rVert_p^2}{4H_p(D)}\right)\right]
&\le-\frac{1}{2}\ln\left(1-\frac{1}{2H_p(D)}\right)+\frac{m_p^2}{4H_p(D)-2}\\
&<\frac{1}{2}\ln\left(\frac{4}{3}\right)+\frac{1}{3}\\
&<1.
\end{aligned} ln E [ exp ( 4 H p ( D ) ∥ ω ∥ p 2 ) ] ≤ − 2 1 ln ( 1 − 2 H p ( D ) 1 ) + 4 H p ( D ) − 2 m p 2 < 2 1 ln ( 3 4 ) + 3 1 < 1.
Suppose next that p > ln ( e D ) p>\ln(eD) p > ln ( eD ) . Since D ≥ 2 D\ge2 D ≥ 2 ,
ln ( 2 D ) ≤ ln ( e D ) . \ln(2D)\le\ln(eD). ln ( 2 D ) ≤ ln ( eD ) .
Therefore, (67) gives m p 2 ≤ 2 H p ( D ) m_p^2\le2H_p(D) m p 2 ≤ 2 H p ( D ) . Moreover,
H p ( D ) = ln ( e D ) D 2 / p ≥ ln ( 2 e ) > 3 2 . \begin{aligned}
H_p(D)&=\ln(eD)D^{2/p}\\
&\ge\ln(2e)\\
&>\frac{3}{2}.
\end{aligned} H p ( D ) = ln ( eD ) D 2/ p ≥ ln ( 2 e ) > 2 3 .
It follows that
ln E [ exp ( ∥ ω ∥ 2 4 H p ( D ) ) ] ≤ − 1 2 ln ( 1 − 1 2 H p ( D ) ) + m p 2 4 H p ( D ) − 2 < 1 2 ln ( 3 2 ) + 3 4 < 1. \begin{aligned}
\ln\mathbb{E}\left[\exp\left(\frac{\lVert\omega\rVert^2}{4H_p(D)}\right)\right]\le-\frac{1}{2}\ln\left(1-\frac{1}{2H_p(D)}\right)+\frac{m_p^2}{4H_p(D)-2} \\
&<\frac{1}{2}\ln\left(\frac{3}{2}\right)+\frac{3}{4} \\
&<1.
\end{aligned} ln E [ exp ( 4 H p ( D ) ∥ ω ∥ 2 ) ] ≤ − 2 1 ln ( 1 − 2 H p ( D ) 1 ) + 4 H p ( D ) − 2 m p 2 < 2 1 ln ( 2 3 ) + 4 3 < 1.
Rescaling ω \omega ω by ν \nu ν proves (66) for p > 2 p>2 p > 2 .
ell-p Spaces in Dimension d
Let ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R d , ∥ ⋅ ∥ p ) (E^*,\lVert\cdot\rVert_*)=(\mathbb{R}^d,\lVert\cdot\rVert_p) ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R d , ∥ ⋅ ∥ p ) with p ∈ [ 1 , ∞ ] p\in[1,\infty] p ∈ [ 1 , ∞ ] . The sharp one-sided Euclidean-to-ℓ p \ell_p ℓ p constant is
α p ( d ) − 2 : = { d 2 p − 1 , 1 ≤ p < 2 , 1 , 2 ≤ p ≤ ∞ . (69) \alpha_p(d)^{-2}:=
\begin{cases}
d^{\frac{2}{p}-1}, & 1\le p<2,\\
1, & 2\le p\le\infty.
\end{cases} \tag*{(69)} α p ( d ) − 2 := { d p 2 − 1 , 1 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞. (69)
Indeed,
∥ z ∥ p ≤ d 1 p − 1 2 ∥ z ∥ 2 , 1 ≤ p < 2 , \lVert z\rVert_p\le d^{\frac{1}{p}-\frac{1}{2}}\lVert z\rVert_2,\qquad1\le p<2, ∥ z ∥ p ≤ d p 1 − 2 1 ∥ z ∥ 2 , 1 ≤ p < 2 ,
whereas
∥ z ∥ p ≤ ∥ z ∥ 2 , 2 ≤ p ≤ ∞ . \lVert z\rVert_p\le\lVert z\rVert_2,\qquad2\le p\le\infty. ∥ z ∥ p ≤ ∥ z ∥ 2 , 2 ≤ p ≤ ∞.
Write κ ℓ p ( d ) \kappa_{\ell_p}(d) κ ℓ p ( d ) for the following admissible regularity parameter of ( R d , ∥ ⋅ ∥ p ) (\mathbb{R}^d,\lVert\cdot\rVert_p) ( R d , ∥ ⋅ ∥ p ) :
κ ℓ p ( d ) : = { d 2 p − 1 , 1 ≤ p < 2 , min 2 ≤ ρ ≤ p , ρ < ∞ ( ρ − 1 ) d 2 ρ − 2 p , 2 ≤ p ≤ ∞ . (70) \kappa_{\ell_p}(d):=
\begin{cases}
d^{\frac{2}{p}-1}, & 1\le p<2,\\
\min_{2\le\rho\le p,\ \rho<\infty}(\rho-1)d^{\frac{2}{\rho}-\frac{2}{p}}, & 2\le p\le\infty.
\end{cases} \tag*{(70)} κ ℓ p ( d ) := { d p 2 − 1 , min 2 ≤ ρ ≤ p , ρ < ∞ ( ρ − 1 ) d ρ 2 − p 2 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞. (70)
This is the parameter constructed in Proposition 12. It satisfies
κ ℓ p ( d ) ≤ 2 e min { p , ln ( e d ) } , 2 ≤ p ≤ ∞ . (71) \kappa_{\ell_p}(d)\le2e\min\{p,\ln(ed)\},\qquad2\le p\le\infty. \tag*{(71)} κ ℓ p ( d ) ≤ 2 e min { p , ln ( e d )} , 2 ≤ p ≤ ∞. (71)
The explicit factor 2 e 2e 2 e follows by taking ρ = p \rho=p ρ = p when p ≤ 2 ln ( e d ) p\le2\ln(ed) p ≤ 2 ln ( e d ) and ρ = 2 ln ( e d ) \rho=2\ln(ed) ρ = 2 ln ( e d ) when p > 2 ln ( e d ) p>2\ln(ed) p > 2 ln ( e d ) . The use of ln ( e d ) \ln(ed) ln ( e d ) keeps this exact bound valid also at d = 1 d=1 d = 1 .
Proposition 5 (ℓ p d \ell_p^d ℓ p d geometry under Rademacher perturbations). Let the oracle be as in Definition 9. Then the smallest admissible light-tail scales are
σ F 2 = d ν 2 , σ 2 = d 2 / p ν 2 . (72) \sigma_{\mathrm{F}}^2=d\nu^2,\qquad\sigma^2=d^{2/p}\nu^2. \tag*{(72)} σ F 2 = d ν 2 , σ 2 = d 2/ p ν 2 . (72) With α − 2 = α p ( d ) − 2 \alpha^{-2}=\alpha_p(d)^{-2} α − 2 = α p ( d ) − 2 and κ = κ ℓ p ( d ) \kappa=\kappa_{\ell_p}(d) κ = κ ℓ p ( d ) , the coefficients in (61) are
C F R a d : = { d 2 / p ν 2 , 1 ≤ p < 2 , d ν 2 , 2 ≤ p ≤ ∞ , (73) C_{\mathrm{F}}^{\mathrm{Rad}}:=
\begin{cases}
d^{2/p}\nu^2, & 1\le p<2,\\
d\nu^2, & 2\le p\le\infty,
\end{cases} \tag*{(73)} C F Rad := { d 2/ p ν 2 , d ν 2 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞ , (73) C ∗ R a d : = { d 4 p − 1 ν 2 , 1 ≤ p < 2 , κ ℓ p ( d ) d 2 / p ν 2 , 2 ≤ p ≤ ∞ . (74) C_*^{\mathrm{Rad}}:=
\begin{cases}
d^{\frac{4}{p}-1}\nu^2, & 1\le p<2,\\
\kappa_{\ell_p}(d)d^{2/p}\nu^2, & 2\le p\le\infty.
\end{cases} \tag*{(74)} C ∗ Rad := { d p 4 − 1 ν 2 , κ ℓ p ( d ) d 2/ p ν 2 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞. (74)
The coefficients agree at p = 2 p = 2 p = 2 . For p < 2 p < 2 p < 2 , the native coefficient contains one additional factor d 2 / p − 1 d^{2/p-1} d 2/ p − 1 . For every fixed p > 2 p > 2 p > 2 ,
C ∗ R a d ≤ 2 e min { p , ln ( e d ) } d 2 / p ν 2 , C^{\mathrm{Rad}}_* \le2e\min\{p,\ln(ed)\}d^{2/p}\nu^2, C ∗ Rad ≤ 2 e min { p , ln ( e d )} d 2/ p ν 2 ,
while C F R a d = d ν 2 C^{\mathrm{Rad}}_F = d\nu^2 C F Rad = d ν 2 . At p = ∞ p = \infty p = ∞ and d ≥ 1 d \ge1 d ≥ 1 , the displayed dependences are O ( ln ( e d ) ν 2 ) \mathcal{O}(\ln(ed)\nu^2) O ( ln ( e d ) ν 2 ) and d ν 2 d\nu^2 d ν 2 , respectively.
Proof. Every coordinate of ζ = ν χ \zeta= \nu\chi ζ = ν χ has absolute value ν \nu ν . Therefore,
∥ ζ ∥ 2 2 = ν 2 ∑ i = 1 d χ i 2 = d ν 2 . \lVert\zeta\rVert_2^2 = \nu^2\sum_{i=1}^{d}\chi_i^2
= d\nu^2. ∥ ζ ∥ 2 2 = ν 2 i = 1 ∑ d χ i 2 = d ν 2 .
For finite p p p ,
∥ ζ ∥ p 2 = ( ∑ i = 1 d ∣ ν χ i ∣ p ) 2 / p = ( d ν p ) 2 / p = d 2 / p ν 2 . \lVert\zeta\rVert_p^2 = \left(\sum_{i=1}^{d}|\nu\chi_i|^p\right)^{2/p}
= (d\nu^p)^{2/p}
= d^{2/p}\nu^2. ∥ ζ ∥ p 2 = ( i = 1 ∑ d ∣ ν χ i ∣ p ) 2/ p = ( d ν p ) 2/ p = d 2/ p ν 2 .
At p = ∞ p = \infty p = ∞ ,
∥ ζ ∥ ∞ 2 = max 1 ≤ i ≤ d ∣ ν χ i ∣ 2 = ν 2 , \lVert\zeta\rVert_\infty^2 = \max_{1\le i\le d}|\nu\chi_i|^2
= \nu^2, ∥ ζ ∥ ∞ 2 = 1 ≤ i ≤ d max ∣ ν χ i ∣ 2 = ν 2 ,
which agrees with d 2 / ∞ = 1 d^{2/\infty} = 1 d 2/∞ = 1 .
Let σ ~ F > 0 \widetilde{\sigma}_F > 0 σ F > 0 and σ ~ > 0 \widetilde{\sigma} > 0 σ > 0 be arbitrary candidate light-tail scales for the Euclidean and ℓ p \ell_p ℓ p norms, respectively. Since both norms of ζ \zeta ζ are deterministic,
E [ exp ( ∥ ζ ∥ 2 2 σ ~ F 2 ) ] ≤ exp ( 1 ) ⟺ σ ~ F 2 ≥ d ν 2 , \mathbb{E}\left[\exp\left(\frac{\lVert\zeta\rVert_2^2}{\widetilde{\sigma}_F^2}\right)\right]\le\exp(1)
\Longleftrightarrow\widetilde{\sigma}_F^2\ge d\nu^2, E [ exp ( σ F 2 ∥ ζ ∥ 2 2 ) ] ≤ exp ( 1 ) ⟺ σ F 2 ≥ d ν 2 ,
E [ exp ( ∥ ζ ∥ p 2 σ ~ 2 ) ] ≤ exp ( 1 ) ⟺ σ ~ 2 ≥ d 2 / p ν 2 . \mathbb{E}\left[\exp\left(\frac{\lVert\zeta\rVert_p^2}{\widetilde{\sigma}^2}\right)\right]\le\exp(1)
\Longleftrightarrow\widetilde{\sigma}^2\ge d^{2/p}\nu^2. E [ exp ( σ 2 ∥ ζ ∥ p 2 ) ] ≤ exp ( 1 ) ⟺ σ 2 ≥ d 2/ p ν 2 .
This proves (72) and shows that both scales are minimal.
For 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 , substituting (69) , (70) , and (72) gives
C F R a d = d 2 / p − 1 ⋅ d ν 2 = d 2 / p ν 2 , C_F^{\mathrm{Rad}}=d^{2/p-1}\cdot d\nu^2=d^{2/p}\nu^2, C F Rad = d 2/ p − 1 ⋅ d ν 2 = d 2/ p ν 2 ,
C ∗ R a d = d 2 / p − 1 ⋅ d 2 / p ν 2 = d 4 / p − 1 ν 2 . C_*^{\mathrm{Rad}}=d^{2/p-1}\cdot d^{2/p}\nu^2=d^{4/p-1}\nu^2. C ∗ Rad = d 2/ p − 1 ⋅ d 2/ p ν 2 = d 4/ p − 1 ν 2 .
For 2 ≤ p ≤ ∞ 2\le p\le\infty 2 ≤ p ≤ ∞ , α p ( d ) − 2 = 1 \alpha_p(d)^{-2}=1 α p ( d ) − 2 = 1 , so
C F R a d = d ν 2 , C_F^{\mathrm{Rad}}=d\nu^2, C F Rad = d ν 2 ,
C ∗ R a d = κ ℓ p ( d ) d 2 / p ν 2 . C_*^{\mathrm{Rad}}=\kappa_{\ell_p}(d)d^{2/p}\nu^2. C ∗ Rad = κ ℓ p ( d ) d 2/ p ν 2 .
The bound (71) gives the stated consequences.
Proposition 6 (ℓ p \ell_p ℓ p geometry under Gaussian perturbations). Let the oracle be as in Definition 10. Using the explicit universal constant from Lemma 22, the scales
σ F 2 : = 4 d ν 2 , σ 2 : = 4 ϑ p ( d ) d 2 / p ν 2 (75) \sigma_F^2:=4d\nu^2,\qquad\sigma^2:=4\vartheta_p(d)d^{2/p}\nu^2 \tag*{(75)} σ F 2 := 4 d ν 2 , σ 2 := 4 ϑ p ( d ) d 2/ p ν 2 (75) are admissible. The Frobenius scale is also of the smallest possible order: every admissible scale σ ~ F \widetilde{\sigma}_F σ F satisfies σ ~ F 2 ≥ d ν 2 \widetilde{\sigma}_F^2\ge d\nu^2 σ F 2 ≥ d ν 2 .
For the choices in (75), the coefficients are
C F G a u : = 4 { d 2 / p ν 2 , 1 ≤ p < 2 , d ν 2 , 2 ≤ p ≤ ∞ , (76) C_{\mathrm{F}}^{\mathrm{Gau}} := 4\begin{cases} d^{2/p}\nu^2, & 1 \le p < 2, \\ d\nu^2, & 2 \le p \le\infty, \end{cases} \tag*{(76)} C F Gau := 4 { d 2/ p ν 2 , d ν 2 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞ , (76)
C ∗ G a u = 4 { d 4 p − 1 ν 2 , 1 ≤ p < 2 , κ ℓ p ( d ) ϑ p ( d ) d 2 / p ν 2 , 2 ≤ p ≤ ∞ . (77) C_*^{\mathrm{Gau}} = 4\begin{cases} d^{\frac{4}{p}-1}\nu^2, & 1 \le p < 2, \\ \kappa_{\ell_p}(d)\vartheta_p(d)d^{2/p}\nu^2, & 2 \le p \le\infty. \end{cases} \tag*{(77)} C ∗ Gau = 4 { d p 4 − 1 ν 2 , κ ℓ p ( d ) ϑ p ( d ) d 2/ p ν 2 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞. (77)
At p = 2 p=2 p = 2 , both coefficients are 4 d ν 2 4d\nu^2 4 d ν 2 . For every fixed p > 2 p>2 p > 2 ,
C ∗ G a u ≤ 8 e min { p , ln ( e d ) } 2 d 2 / p ν 2 , C_*^{\mathrm{Gau}} \le8e\min\{p,\ln(ed)\}^2d^{2/p}\nu^2, C ∗ Gau ≤ 8 e min { p , ln ( e d ) } 2 d 2/ p ν 2 ,
whereas C F G a u = 4 d ν 2 C_{\mathrm{F}}^{\mathrm{Gau}}=4d\nu^2 C F Gau = 4 d ν 2 . At p = ∞ p=\infty p = ∞ and d ≥ 1 d\ge1 d ≥ 1 , the displayed dependences are O ( ( ln ( e d ) ) 2 ν 2 ) \mathcal{O}((\ln(ed))^2\nu^2) O (( ln ( e d ) ) 2 ν 2 ) and d ν 2 d\nu^2 d ν 2 , respectively.
Proof. The Frobenius claim follows from Lemma 22 with D = d D=d D = d and p = 2 p=2 p = 2 . To see that its order cannot be smaller, let σ ~ F \widetilde{\sigma}_{\mathrm{F}} σ F be any admissible Frobenius scale. Jensen’s inequality gives
exp ( 1 ) ≥ E [ exp ( ∥ ν ω ∥ 2 2 σ ~ F 2 ) ] ≥ exp ( E [ ∥ ν ω ∥ 2 2 ] σ ~ F 2 ) . \exp(1) \ge\mathbb{E}\left[\exp\left(\frac{\lVert\nu\omega\rVert_2^2}{\widetilde{\sigma}_{\mathrm{F}}^2}\right)\right]
\ge\exp\left(\frac{\mathbb{E}\left[\lVert\nu\omega\rVert_2^2\right]}{\widetilde{\sigma}_{\mathrm{F}}^2}\right). exp ( 1 ) ≥ E [ exp ( σ F 2 ∥ ν ω ∥ 2 2 ) ] ≥ exp ( σ F 2 E [ ∥ ν ω ∥ 2 2 ] ) .
Since
E [ ∥ ν ω ∥ 2 2 ] = d ν 2 , \mathbb{E}\left[\lVert\nu\omega\rVert_2^2\right]=d\nu^2, E [ ∥ ν ω ∥ 2 2 ] = d ν 2 ,
we obtain
σ ~ F 2 ≥ d ν 2 . \widetilde{\sigma}_{\mathrm{F}}^2\ge d\nu^2. σ F 2 ≥ d ν 2 .
The native scale follows from the same lemma with D = d D=d D = d and the given value of p p p . It remains to substitute the two scales into (61). If 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 , then
C F G a u = d 2 p − 1 ⋅ 4 d ν 2 = 4 d 2 / p ν 2 , C_{\mathrm{F}}^{\mathrm{Gau}}=d^{\frac{2}{p}-1}\cdot4d\nu^2=4d^{2/p}\nu^2, C F Gau = d p 2 − 1 ⋅ 4 d ν 2 = 4 d 2/ p ν 2 ,
C ∗ G a u = d 2 p − 1 ⋅ 4 d 2 / p ν 2 = 4 d 4 p − 1 ν 2 . C_*^{\mathrm{Gau}}=d^{\frac{2}{p}-1}\cdot4d^{2/p}\nu^2=4d^{\frac{4}{p}-1}\nu^2. C ∗ Gau = d p 2 − 1 ⋅ 4 d 2/ p ν 2 = 4 d p 4 − 1 ν 2 .
If 2 ≤ p ≤ ∞ 2\le p\le\infty 2 ≤ p ≤ ∞ , then
C F G a u = 4 d ν 2 , C_{\mathrm{F}}^{\mathrm{Gau}}=4d\nu^2, C F Gau = 4 d ν 2 ,
C ∗ G a u = 4 κ ℓ p ( d ) ϑ p ( d ) d 2 / p ν 2 . C_*^{\mathrm{Gau}}=4\kappa_{\ell_p}(d)\vartheta_p(d)d^{2/p}\nu^2. C ∗ Gau = 4 κ ℓ p ( d ) ϑ p ( d ) d 2/ p ν 2 .
Using (71) proves the final two statements.
Remark 3 (One-sparse perturbations). Fix ν > 0 \nu>0 ν > 0 . Let χ \chi χ be Rademacher and I I I be uniform on [ D ] [D] [ D ] , independently of χ \chi χ , and let e 1 , … , e D e_1,\ldots,e_D e 1 , … , e D be the standard basis of R D \mathbb{R}^D R D . Define
ζ : = ν χ e I in the vector case, D = d , \zeta:=\nu\chi e_I \qquad\text{in the vector case, }D=d, ζ := ν χ e I in the vector case, D = d ,
ζ : = ν Diag m , n ( χ e I ) in the matrix case, D = r = min { m , n } . \zeta:=\nu\operatorname{Diag}_{m,n}(\chi e_I) \qquad\text{in the matrix case, }D=r=\min\{m,n\}. ζ := ν Diag m , n ( χ e I ) in the matrix case, D = r = min { m , n } .
The oracle returns ∇ f ( x ) + ζ \nabla f(x)+\zeta ∇ f ( x ) + ζ with a fresh independent pair ( χ , I ) (\chi,I) ( χ , I ) at every call. In both cases, E [ ζ ] = 0 \mathbb{E}[\zeta]=0 E [ ζ ] = 0 , and the Euclidean/Frobenius norm and the native ℓ p \ell_p ℓ p /Schatten-p p p norm of ζ \zeta ζ equal ν \nu ν almost surely for every 1 ≤ p ≤ ∞ 1\le p\le\infty 1 ≤ p ≤ ∞ . Thus the light-tail condition reduces to exp ( ν 2 σ 2 ) ≤ e \exp\left(\frac{\nu^2}{\sigma^2}\right)\le e exp ( σ 2 ν 2 ) ≤ e , and the minimal squared light-tail scales are σ F 2 = σ ∗ 2 = ν 2 \sigma_{\mathrm{F}}^2=\sigma_*^2=\nu^2 σ F 2 = σ ∗ 2 = ν 2 . For 1 ≤ p ≤ 2 1\le p\le2 1 ≤ p ≤ 2 , the chosen conversion and regularity factors both equal D 2 / p − 1 D^{2/p-1} D 2/ p − 1 , so
C F = C ∗ = D 2 / p − 1 ν 2 . C_{\mathrm{F}}=C_*=D^{2/p-1}\nu^2. C F = C ∗ = D 2/ p − 1 ν 2 .
Sparsity alone does not ensure that the coefficients coincide for other perturbation models.
Spaces with Schatten-p p p Norms on real m m m -by-n n n matrices
Let ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R m × n , ∥ ⋅ ∥ S p ) (E^*, \|\cdot\|_*) = (\mathbb{R}^{m \times n}, \|\cdot\|_{S_p}) ( E ∗ , ∥ ⋅ ∥ ∗ ) = ( R m × n , ∥ ⋅ ∥ S p ) , where p ∈ [ 1 , ∞ ] p \in[1,\infty] p ∈ [ 1 , ∞ ] and r : = min { m , n } r := \min\{m,n\} r := min { m , n } . The Frobenius norm is ∥ ⋅ ∥ F = ∥ ⋅ ∥ S 2 \|\cdot\|_F = \|\cdot\|_{S_2} ∥ ⋅ ∥ F = ∥ ⋅ ∥ S 2 . Applying the vector norm comparison to the singular-value vector gives
α S p ( r ) − 2 : = { r 2 p − 1 , 1 ≤ p < 2 , 1 , 2 ≤ p ≤ ∞ . (78) \alpha_{S_p}(r)^{-2} := \begin{cases}
r^{\frac{2}{p}-1}, & 1 \le p < 2,\\
1, & 2 \le p \le\infty.
\end{cases} \tag*{(78)} α S p ( r ) − 2 := { r p 2 − 1 , 1 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞. (78)
Indeed,
∥ Z ∥ S p ≤ r 1 p − 1 2 ∥ Z ∥ F , 1 ≤ p < 2 , \|Z\|_{S_p} \le r^{\frac{1}{p}-\frac{1}{2}}\|Z\|_F,\qquad1 \le p < 2, ∥ Z ∥ S p ≤ r p 1 − 2 1 ∥ Z ∥ F , 1 ≤ p < 2 ,
whereas
∥ Z ∥ S p ≤ ∥ Z ∥ F , 2 ≤ p ≤ ∞ . \|Z\|_{S_p} \le\|Z\|_F,\qquad2 \le p \le\infty. ∥ Z ∥ S p ≤ ∥ Z ∥ F , 2 ≤ p ≤ ∞.
Choose an admissible regularity parameter κ S p ( r ) \kappa_{S_p}(r) κ S p ( r ) as follows:
κ S p ( r ) : = { r 2 p − 1 , 1 ≤ p < 2 , 1 , p = 2 , min 2 ≤ ρ ≤ p ρ < ∞ max { 2 , ρ − 1 } r 2 ρ − 2 p , 2 < p ≤ ∞ . (79) \kappa_{S_p}(r) := \begin{cases}
r^{\frac{2}{p}-1}, & 1 \le p < 2,\\
1, & p = 2,\\
\min_{\substack{2 \le\rho\le p\\ \rho< \infty}}\max\{2,\rho-1\}r^{\frac{2}{\rho}-\frac{2}{p}}, & 2 < p \le\infty.
\end{cases} \tag*{(79)} κ S p ( r ) := ⎩ ⎨ ⎧ r p 2 − 1 , 1 , min 2 ≤ ρ ≤ p ρ < ∞ max { 2 , ρ − 1 } r ρ 2 − p 2 , 1 ≤ p < 2 , p = 2 , 2 < p ≤ ∞. (79)
The p = 2 p=2 p = 2 line uses the exact Hilbert-space constant. The remaining lines follow from Proposition 14. In particular,
κ S p ( r ) ≤ 2 e min { p , ln ( e r ) } , 2 ≤ p ≤ ∞ . (80) \kappa_{S_p}(r) \le2e\min\{p,\ln(er)\},\qquad2 \le p \le\infty. \tag*{(80)} κ S p ( r ) ≤ 2 e min { p , ln ( er )} , 2 ≤ p ≤ ∞. (80)
As in the vector case, for p > 2 p>2 p > 2 take ρ = p \rho=p ρ = p when p ≤ 2 ln ( e r ) p\le2\ln(er) p ≤ 2 ln ( er ) and ρ = 2 ln ( e r ) \rho=2\ln(er) ρ = 2 ln ( er ) otherwise. The factor 2 e 2e 2 e is valid also at r = 1 r=1 r = 1 .
Proposition 7 (Schatten-p p p geometry under diagonal Rademacher perturbations). Let the oracle be as in Definition 9. Then the smallest admissible light-tail scales are
σ F 2 = r ν 2 , σ 2 = r 2 / p ν 2 . (81) \sigma_F^2 = r\nu^2,\qquad\sigma^2 = r^{2/p}\nu^2. \tag*{(81)} σ F 2 = r ν 2 , σ 2 = r 2/ p ν 2 . (81) With α − 2 = α S p ( r ) − 2 \alpha^{-2}=\alpha_{S_p}(r)^{-2} α − 2 = α S p ( r ) − 2 and κ = κ S p ( r ) \kappa=\kappa_{S_p}(r) κ = κ S p ( r ) , the statistical coefficients are
C F R a d : = { r 2 / p ν 2 , 1 ≤ p < 2 , r ν 2 , 2 ≤ p ≤ ∞ , (82) C_F^{\mathrm{Rad}} := \begin{cases}
r^{2/p}\nu^2, & 1 \le p < 2,\\
r\nu^2, & 2 \le p \le\infty,
\end{cases} \tag*{(82)} C F Rad := { r 2/ p ν 2 , r ν 2 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞ , (82) C ∗ R a d : = { r 4 p − 1 ν 2 , 1 ≤ p < 2 , κ S p ( r ) r 2 / p ν 2 , 2 ≤ p ≤ ∞ . (83) C_*^{\mathrm{Rad}} := \begin{cases}
r^{\frac{4}{p}-1}\nu^2, & 1 \le p < 2,\\
\kappa_{S_p}(r)r^{2/p}\nu^2, & 2 \le p \le\infty.
\end{cases} \tag*{(83)} C ∗ Rad := { r p 4 − 1 ν 2 , κ S p ( r ) r 2/ p ν 2 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞. (83) The coefficients agree at p = 2 p=2 p = 2 . For p < 2 p<2 p < 2 , the native coefficient contains one additional factor r 2 / p − 1 r^{2/p-1} r 2/ p − 1 . For every fixed p > 2 p>2 p > 2 ,
C ∗ R a d ≤ 2 e min { p , ln ( e r ) } r 2 / p ν 2 , C_*^{\mathrm{Rad}} \le2e\min\{p,\ln(er)\}r^{2/p}\nu^2, C ∗ Rad ≤ 2 e min { p , ln ( er )} r 2/ p ν 2 , whereas C F R a d = r ν 2 C_F^{\mathrm{Rad}}=r\nu^2 C F Rad = r ν 2 . At p = ∞ p=\infty p = ∞ and r ≥ 1 r\ge1 r ≥ 1 , the displayed dependences are O ( ln ( e r ) ν 2 ) O(\ln(er)\nu^2) O ( ln ( er ) ν 2 ) and r ν 2 r\nu^2 r ν 2 , respectively.
Proof. The matrix
ζ = ν Diag m , n ( ( χ i ) i = 1 r ) \zeta=\nu\operatorname{Diag}_{m,n}\left((\chi_i)_{i=1}^{r}\right) ζ = ν Diag m , n ( ( χ i ) i = 1 r )
has singular values
σ i ( ζ ) = ν ∣ χ i ∣ = ν , i = 1 , … , r . \sigma_i(\zeta)=\nu|\chi_i|=\nu,\qquad i=1,\ldots,r. σ i ( ζ ) = ν ∣ χ i ∣ = ν , i = 1 , … , r .
Therefore,
∥ ζ ∥ F 2 = ∑ i = 1 r σ i ( ζ ) 2 = r ν 2 . \|\zeta\|_F^2=\sum_{i=1}^{r}\sigma_i(\zeta)^2
=r\nu^2. ∥ ζ ∥ F 2 = i = 1 ∑ r σ i ( ζ ) 2 = r ν 2 .
For finite p p p ,
∥ ζ ∥ S p 2 = ( ∑ i = 1 r σ i ( ζ ) p ) 2 / p = ( r ν p ) 2 / p = r 2 / p ν 2 . \lVert\zeta\rVert_{S_p}^{2}=\left(\sum_{i=1}^{r}\sigma_i(\zeta)^p\right)^{2/p}
=(r\nu^p)^{2/p}
=r^{2/p}\nu^2. ∥ ζ ∥ S p 2 = ( i = 1 ∑ r σ i ( ζ ) p ) 2/ p = ( r ν p ) 2/ p = r 2/ p ν 2 .
At p = ∞ p=\infty p = ∞ ,
∥ ζ ∥ S ∞ 2 = max 1 ≤ i ≤ r σ i ( ζ ) 2 = ν 2 . \lVert\zeta\rVert_{S_\infty}^{2}=\max_{1\leq i\leq r}\sigma_i(\zeta)^2
=\nu^2. ∥ ζ ∥ S ∞ 2 = 1 ≤ i ≤ r max σ i ( ζ ) 2 = ν 2 .
Let σ ~ F > 0 \widetilde{\sigma}_F>0 σ F > 0 and σ ~ > 0 \widetilde{\sigma}>0 σ > 0 be arbitrary candidate light-tail scales for the Frobenius and Schatten-p p p norms, respectively. Since both norms of ζ \zeta ζ are deterministic,
E [ exp ( ∥ ζ ∥ F 2 σ ~ F 2 ) ] ≤ exp ( 1 ) \mathbb{E}\left[\exp\left(\frac{\lVert\zeta\rVert_F^2}{\widetilde{\sigma}_F^2}\right)\right]\leq\exp(1) E [ exp ( σ F 2 ∥ ζ ∥ F 2 ) ] ≤ exp ( 1 )
⟺ exp ( r ν 2 σ ~ F 2 ) ≤ exp ( 1 ) \Longleftrightarrow\exp\left(\frac{r\nu^2}{\widetilde{\sigma}_F^2}\right)\leq\exp(1) ⟺ exp ( σ F 2 r ν 2 ) ≤ exp ( 1 )
⟺ σ ~ F 2 ≥ r ν 2 , \Longleftrightarrow\widetilde{\sigma}_F^2\geq r\nu^2, ⟺ σ F 2 ≥ r ν 2 ,
and, with the convention r 2 / ∞ = 1 r^{2/\infty}=1 r 2/∞ = 1 ,
E [ exp ( ∥ ζ ∥ S p 2 σ ~ 2 ) ] ≤ exp ( 1 ) \mathbb{E}\left[\exp\left(\frac{\lVert\zeta\rVert_{S_p}^2}{\widetilde{\sigma}^2}\right)\right]\leq\exp(1) E [ exp ( σ 2 ∥ ζ ∥ S p 2 ) ] ≤ exp ( 1 )
⟺ exp ( r 2 / p ν 2 σ ~ 2 ) ≤ exp ( 1 ) \Longleftrightarrow\exp\left(\frac{r^{2/p}\nu^2}{\widetilde{\sigma}^2}\right)\leq\exp(1) ⟺ exp ( σ 2 r 2/ p ν 2 ) ≤ exp ( 1 )
⟺ σ ~ 2 ≥ r 2 / p ν 2 . \Longleftrightarrow\widetilde{\sigma}^2\geq r^{2/p}\nu^2. ⟺ σ 2 ≥ r 2/ p ν 2 .
Thus the choices σ F 2 = r ν 2 \sigma_F^2=r\nu^2 σ F 2 = r ν 2 and σ 2 = r 2 / p ν 2 \sigma^2=r^{2/p}\nu^2 σ 2 = r 2/ p ν 2 in (81) are admissible and are the smallest possible scales.
For 1 ≤ p < 2 1\leq p<2 1 ≤ p < 2 , substitution gives
C F R a d = r 2 p − 1 ⋅ r ν 2 = r 2 / p ν 2 , C_F^{\mathrm{Rad}}=r^{\frac{2}{p}-1}\cdot r\nu^2=r^{2/p}\nu^2, C F Rad = r p 2 − 1 ⋅ r ν 2 = r 2/ p ν 2 ,
C ∗ R a d = r 2 p − 1 ⋅ r 2 / p ν 2 = r 4 p − 1 ν 2 . C_*^{\mathrm{Rad}}=r^{\frac{2}{p}-1}\cdot r^{2/p}\nu^2=r^{\frac{4}{p}-1}\nu^2. C ∗ Rad = r p 2 − 1 ⋅ r 2/ p ν 2 = r p 4 − 1 ν 2 .
For 2 ≤ p ≤ ∞ 2\leq p\leq\infty 2 ≤ p ≤ ∞ , α S p ( r ) − 2 = 1 \alpha_{S_p}(r)^{-2}=1 α S p ( r ) − 2 = 1 , and hence
C F R a d = r ν 2 , C_F^{\mathrm{Rad}}=r\nu^2, C F Rad = r ν 2 ,
C ∗ R a d = κ S p ( r ) r 2 / p ν 2 . C_*^{\mathrm{Rad}}=\kappa_{S_p}(r)r^{2/p}\nu^2. C ∗ Rad = κ S p ( r ) r 2/ p ν 2 .
The bound (80) gives the stated consequences.
Proposition 8 (Schatten-p p p geometry under diagonal Gaussian perturbations). Let the oracle be as in Definition 10. Using the explicit universal constant from Lemma 22 , the scales
σ F 2 : = 4 r ν 2 , σ 2 : = 4 ϑ p ( r ) r 2 / p ν 2 (84) \sigma_F^2:=4r\nu^2,\qquad\sigma^2:=4\vartheta_p(r)r^{2/p}\nu^2 \tag*{(84)} σ F 2 := 4 r ν 2 , σ 2 := 4 ϑ p ( r ) r 2/ p ν 2 (84) are admissible. Every admissible Frobenius scale σ ~ F \widetilde{\sigma}_F σ F satisfies σ ~ F 2 ≥ r ν 2 \widetilde{\sigma}_F^2\geq r\nu^2 σ F 2 ≥ r ν 2 , so the displayed Frobenius scale has the smallest possible order.
For the choices in (84) , the coefficients are
C F G a u : = 4 { r 2 / p ν 2 , 1 ≤ p < 2 , r ν 2 , 2 ≤ p ≤ ∞ , (85) C_F^{\mathrm{Gau}}:=4
\begin{cases}
r^{2/p}\nu^2, & 1\leq p<2,\\
r\nu^2, & 2\leq p\leq\infty,
\end{cases} \tag*{(85)} C F Gau := 4 { r 2/ p ν 2 , r ν 2 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞ , (85) C ∗ G a u : = 4 { r 4 p − 1 ν 2 , 1 ≤ p < 2 , κ S p ( r ) ϑ p ( r ) r 2 / p ν 2 , 2 ≤ p ≤ ∞ . (86) C_*^{\mathrm{Gau}}:=4
\begin{cases}
r^{\frac{4}{p}-1}\nu^2, & 1\leq p<2,\\
\kappa_{S_p}(r)\vartheta_p(r)r^{2/p}\nu^2, & 2\leq p\leq\infty.
\end{cases} \tag*{(86)} C ∗ Gau := 4 { r p 4 − 1 ν 2 , κ S p ( r ) ϑ p ( r ) r 2/ p ν 2 , 1 ≤ p < 2 , 2 ≤ p ≤ ∞. (86)
At p = 2 p = 2 p = 2 , both coefficients are 4 r ν 2 4r\nu^2 4 r ν 2 . For every fixed p > 2 p > 2 p > 2 ,
C ∗ G a u ≤ 8 e min { p , ln ( e r ) } 2 r 2 / p ν 2 , C_*^{\mathrm{Gau}} \le8e\min\{p,\ln(er)\}^2r^{2/p}\nu^2, C ∗ Gau ≤ 8 e min { p , ln ( er ) } 2 r 2/ p ν 2 ,
whereas C F G a u = 4 r ν 2 C_F^{\mathrm{Gau}} = 4r\nu^2 C F Gau = 4 r ν 2 . At p = ∞ p = \infty p = ∞ and r ≥ 1 r \ge1 r ≥ 1 , the displayed dependences are O ( ( ln ( e r ) ) 2 ν 2 ) \mathcal{O}((\ln(er))^2\nu^2) O (( ln ( er ) ) 2 ν 2 ) and r ν 2 r\nu^2 r ν 2 , respectively.
Proof. For
ζ = ν Diag m , n ( ( ω i ) i = 1 r ) , \zeta= \nu\operatorname{Diag}_{m,n}((\omega_i)_{i=1}^r), ζ = ν Diag m , n (( ω i ) i = 1 r ) ,
the singular values are the numbers ν ∣ ω i ∣ \nu|\omega_i| ν ∣ ω i ∣ , i = 1 , … , r i = 1,\ldots,r i = 1 , … , r , arranged in nonincreasing order. Since the corresponding vector norms are invariant under permutations, we obtain
∥ ζ ∥ F = ν ∥ ω ∥ 2 , \|\zeta\|_F = \nu\|\omega\|_2, ∥ ζ ∥ F = ν ∥ ω ∥ 2 ,
∥ ζ ∥ S p = ν ∥ ω ∥ p , \|\zeta\|_{S_p} = \nu\|\omega\|_p, ∥ ζ ∥ S p = ν ∥ ω ∥ p ,
where ω = ( ω 1 , … , ω r ) ∼ N ( 0 , I r ) \omega= (\omega_1,\ldots,\omega_r) \sim\mathcal{N}(0,I_r) ω = ( ω 1 , … , ω r ) ∼ N ( 0 , I r ) . Applying Lemma 22 with D = r D = r D = r and p = 2 p = 2 p = 2 proves that 4 r ν 2 4r\nu^2 4 r ν 2 is an admissible Frobenius scale. If σ ~ F \widetilde{\sigma}_F σ F is any admissible Frobenius scale, then Jensen’s inequality gives
exp ( 1 ) ≥ E [ exp ( ∥ ζ ∥ F 2 σ ~ F 2 ) ] ≥ exp ( E [ ∥ ζ ∥ F 2 ] σ ~ F 2 ) . \begin{aligned}
\exp(1) \ge\mathbb{E}\left[\exp\left(\frac{\|\zeta\|_F^2}{\widetilde{\sigma}_F^2}\right)\right] \\
&\ge\exp\left(\frac{\mathbb{E}\left[\|\zeta\|_F^2\right]}{\widetilde{\sigma}_F^2}\right).
\end{aligned} exp ( 1 ) ≥ E [ exp ( σ F 2 ∥ ζ ∥ F 2 ) ] ≥ exp ( σ F 2 E [ ∥ ζ ∥ F 2 ] ) .
Since
E [ ∥ ζ ∥ F 2 ] = r ν 2 , \mathbb{E}\left[\|\zeta\|_F^2\right] = r\nu^2, E [ ∥ ζ ∥ F 2 ] = r ν 2 ,
we obtain
σ ~ F 2 ≥ r ν 2 . \widetilde{\sigma}_F^2 \ge r\nu^2. σ F 2 ≥ r ν 2 .
Applying the same lemma with D = r D = r D = r and the stated value of p p p proves that the native scale in (84) is admissible. For 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 , substitution yields
C F G a u = r 2 p − 1 ⋅ 4 r ν 2 = 4 r 2 / p ν 2 , C_F^{\mathrm{Gau}} = r^{\frac{2}{p}-1}\cdot4r\nu^2 = 4r^{2/p}\nu^2, C F Gau = r p 2 − 1 ⋅ 4 r ν 2 = 4 r 2/ p ν 2 ,
C ∗ G a u = r 2 p − 1 ⋅ 4 r 2 / p ν 2 = 4 r 4 p − 1 ν 2 . C_*^{\mathrm{Gau}} = r^{\frac{2}{p}-1}\cdot4r^{2/p}\nu^2 = 4r^{\frac{4}{p}-1}\nu^2. C ∗ Gau = r p 2 − 1 ⋅ 4 r 2/ p ν 2 = 4 r p 4 − 1 ν 2 .
For 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ , α S p ( r ) − 2 = 1 \alpha_{S_p}(r)^{-2} = 1 α S p ( r ) − 2 = 1 , and hence
C F G a u = 4 r ν 2 C_F^{\mathrm{Gau}} = 4r\nu^2 C F Gau = 4 r ν 2
C ∗ G a u = 4 κ S p ( r ) ϑ p ( r ) r 2 / p ν 2 . C_*^{\mathrm{Gau}} = 4\kappa_{S_p}(r)\vartheta_p(r)r^{2/p}\nu^2. C ∗ Gau = 4 κ S p ( r ) ϑ p ( r ) r 2/ p ν 2 .
Using (80) proves the final statements.
Fundamental Properties of Smooth and Regular Spaces
This section collects general results for finite-dimensional smooth and regular normed spaces. We first state a lemma that provides a smooth auxiliary norm uniformly equivalent to a regular norm, and then recall equivalent characterizations of smoothness in terms of the squared norm and its conjugate. The ℓ p d \ell_p^d ℓ p d and Schatten-p p p examples, with matching upper and lower regularity bounds, are developed in the next section, Section L.
The following lemma supplies the smooth auxiliary norm used in our proof of the concentration inequality (Section M.1).
Lemma 23 (Juditsky and Nemirovski [ [40 ] ). Let ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) be a finite-dimensional κ V \kappa_V κ V -regular space in the sense of Definition 1. Then there exists a κ V \kappa_V κ V -smooth norm ∥ ⋅ ∥ V , + \lVert\cdot\rVert_{V,+} ∥ ⋅ ∥ V , + on V V V such that
∥ u ∥ V 2 ≤ ∥ u ∥ V , + 2 ≤ 2 ∥ u ∥ V 2 , ∀ u ∈ V . (87) \lVert u\rVert_V^2 \le\lVert u\rVert_{V,+}^2 \le2\lVert u\rVert_V^2,\qquad\forall u\in V. \tag*{(87)} ∥ u ∥ V 2 ≤ ∥ u ∥ V , + 2 ≤ 2 ∥ u ∥ V 2 , ∀ u ∈ V . (87)
Next, we recall standard equivalent characterizations of smoothness in the primal space. For φ ( u ) : = 1 2 ∥ u ∥ V , + 2 \varphi(u):=\frac{1}{2}\lVert u\rVert_{V,+}^2 φ ( u ) := 2 1 ∥ u ∥ V , + 2 , these properties relate the smoothness of the norm ∥ ⋅ ∥ V , + \lVert\cdot\rVert_{V,+} ∥ ⋅ ∥ V , + to the strong convexity of the Fenchel conjugate φ ∗ \varphi^* φ ∗ and the Lipschitz continuity of the gradient ∇ φ \nabla\varphi ∇ φ .
Proposition 9 (Juditsky and Nemirovski [ [40 ] ). Let ( V , ∥ ⋅ ∥ V , + ) (V,\lVert\cdot\rVert_{V,+}) ( V , ∥ ⋅ ∥ V , + ) be a finite-dimensional normed space with dual space V ∗ V^* V ∗ and induced dual norm ∥ ⋅ ∥ V , + , ∗ \lVert\cdot\rVert_{V,+,*} ∥ ⋅ ∥ V , + , ∗ . Define φ ( u ) : = 1 2 ∥ u ∥ V , + 2 \varphi(u):=\frac{1}{2}\lVert u\rVert_{V,+}^2 φ ( u ) := 2 1 ∥ u ∥ V , + 2 and φ ∗ ( s ) : = 1 2 ∥ s ∥ V , + , ∗ 2 \varphi^*(s):=\frac{1}{2}\lVert s\rVert_{V,+,*}^2 φ ∗ ( s ) := 2 1 ∥ s ∥ V , + , ∗ 2 . Then the following six properties are equivalent:
(i) The space ( V , ∥ ⋅ ∥ V , + ) (V,\lVert\cdot\rVert_{V,+}) ( V , ∥ ⋅ ∥ V , + ) is κ V , + \kappa_{V,+} κ V , + -smooth.
(ii) The function φ \varphi φ is everywhere differentiable, and its gradient satisfies
⟨ ∇ φ ( u ) − ∇ φ ( v ) , u − v ⟩ ≤ κ V , + ∥ u − v ∥ V , + 2 , ∀ u , v ∈ V . \langle\nabla\varphi(u)-\nabla\varphi(v),u-v\rangle\le\kappa_{V,+}\lVert u-v\rVert_{V,+}^2,\qquad\forall u,v\in V. ⟨ ∇ φ ( u ) − ∇ φ ( v ) , u − v ⟩ ≤ κ V , + ∥ u − v ∥ V , + 2 , ∀ u , v ∈ V . (iii) The function φ \varphi φ is continuously differentiable, and the mapping ∇ φ : V → V ∗ \nabla\varphi:V\to V^* ∇ φ : V → V ∗ is κ V , + \kappa_{V,+} κ V , + -Lipschitz continuous:
∥ ∇ φ ( u ) − ∇ φ ( v ) ∥ V , + , ∗ ≤ κ V , + ∥ u − v ∥ V , + , ∀ u , v ∈ V . \lVert\nabla\varphi(u)-\nabla\varphi(v)\rVert_{V,+,*}\le\kappa_{V,+}\lVert u-v\rVert_{V,+},\qquad\forall u,v\in V. ∥ ∇ φ ( u ) − ∇ φ ( v ) ∥ V , + , ∗ ≤ κ V , + ∥ u − v ∥ V , + , ∀ u , v ∈ V . (iv) The set-valued subdifferential operator ∂ φ ∗ : V ∗ ⇉ V \partial\varphi^*:V^*\rightrightarrows V ∂ φ ∗ : V ∗ ⇉ V is strongly monotone with constant 1 / κ V , + 1/\kappa_{V,+} 1/ κ V , + :
⟨ s 1 − s 2 , u 1 − u 2 ⟩ ≥ 1 κ V , + ∥ s 1 − s 2 ∥ V , + , ∗ 2 \langle s_1-s_2,u_1-u_2\rangle\ge\frac{1}{\kappa_{V,+}}\lVert s_1-s_2\rVert_{V,+,*}^2 ⟨ s 1 − s 2 , u 1 − u 2 ⟩ ≥ κ V , + 1 ∥ s 1 − s 2 ∥ V , + , ∗ 2 for all s 1 , s 2 ∈ V ∗ s_1,s_2\in V^* s 1 , s 2 ∈ V ∗ and all u 1 ∈ ∂ φ ∗ ( s 1 ) u_1\in\partial\varphi^*(s_1) u 1 ∈ ∂ φ ∗ ( s 1 ) , u 2 ∈ ∂ φ ∗ ( s 2 ) u_2\in\partial\varphi^*(s_2) u 2 ∈ ∂ φ ∗ ( s 2 ) .
(v) The inverse mapping ( ∂ φ ∗ ) − 1 = ∂ φ (\partial\varphi^*)^{-1}=\partial\varphi ( ∂ φ ∗ ) − 1 = ∂ φ is single-valued and κ V , + \kappa_{V,+} κ V , + -Lipschitz continuous, which can be written as
∥ u 1 − u 2 ∥ V , + ≥ 1 κ V , + ∥ s 1 − s 2 ∥ V , + , ∗ , \lVert u_1-u_2\rVert_{V,+}\ge\frac{1}{\kappa_{V,+}}\lVert s_1-s_2\rVert_{V,+,*}, ∥ u 1 − u 2 ∥ V , + ≥ κ V , + 1 ∥ s 1 − s 2 ∥ V , + , ∗ , for all s 1 , s 2 ∈ V ∗ s_1,s_2\in V^* s 1 , s 2 ∈ V ∗ and all u 1 ∈ ∂ φ ∗ ( s 1 ) u_1\in\partial\varphi^*(s_1) u 1 ∈ ∂ φ ∗ ( s 1 ) , u 2 ∈ ∂ φ ∗ ( s 2 ) u_2\in\partial\varphi^*(s_2) u 2 ∈ ∂ φ ∗ ( s 2 ) .
(vi) The function φ ∗ \varphi^* φ ∗ is strongly convex with constant 1 / κ V , + 1/\kappa_{V,+} 1/ κ V , + with respect to ∥ ⋅ ∥ V , + , ∗ \lVert\cdot\rVert_{V,+,*} ∥ ⋅ ∥ V , + , ∗ :
φ ∗ ( s 2 ) ≥ φ ∗ ( s 1 ) + ⟨ u 1 , s 2 − s 1 ⟩ + 1 2 κ V , + ∥ s 2 − s 1 ∥ V , + , ∗ 2 \varphi^*(s_2)\ge\varphi^*(s_1)+\langle u_1,s_2-s_1\rangle+\frac{1}{2\kappa_{V,+}}\lVert s_2-s_1\rVert_{V,+,*}^2 φ ∗ ( s 2 ) ≥ φ ∗ ( s 1 ) + ⟨ u 1 , s 2 − s 1 ⟩ + 2 κ V , + 1 ∥ s 2 − s 1 ∥ V , + , ∗ 2 for all s 1 , s 2 ∈ V ∗ s_1,s_2\in V^* s 1 , s 2 ∈ V ∗ and all $u_1\in\partial\varphi^*(s_1).
Examples of κ \kappa κ -Regular Spaces and Sharp Bounds on κ \kappa κ
This section establishes the universal regularity bound and gives explicit upper and matching lower bounds for the norm families used in the main text. For ℓ p d \ell_p^d ℓ p d and Schatten-p p p spaces, write D = d D=d D = d and D = r = min { m , n } D=r=\min\{m,n\} D = r = min { m , n } , respectively. The optimal parameter equals D 2 / p − 1 D^{2/p-1} D 2/ p − 1 for 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 and has order min { p , ln ( e D ) } \min\{p,\ln(eD)\} min { p , ln ( eD )} for 2 ≤ p ≤ ∞ 2\le p\le\infty 2 ≤ p ≤ ∞ . Thus the latter regime admits a bound independent of D D D for each fixed finite p p p , with logarithmic growth in the branch p ≥ ln ( e D ) p\ge\ln(eD) p ≥ ln ( eD ) , including p = ∞ p=\infty p = ∞ .
Table 3 summarizes the bounds and auxiliary norms using the main-text convention V = E ∗ V=E^* V = E ∗ . We begin with the universal bound via John’s ellipsoid, followed by the ℓ p d \ell_p^d ℓ p d and Schatten-p p p constructions and their matching lower bounds.
Universal Bound via John’s Ellipsoid
The universal regularity bound follows from the symmetric form of John’s ellipsoid theorem [37 ] . We give a proof based on a convex optimization problem for the ellipsoid’s shape matrix.
Theorem 7 (John’s Ellipsoid Theorem for Symmetric Bodies). Let V V V be a d d d -dimensional real vector space and let B ⊂ V B\subset V B ⊂ V be a compact, convex body with non-empty interior containing the origin. Assume B B B is centrally symmetric, i.e., if x ∈ B x\in B x ∈ B then − x ∈ B -x\in B − x ∈ B . Then:
(i) There exists a unique ellipsoid E i n \mathcal{E}_{\mathrm{in}} E in of maximal volume contained in B B B (the John ellipsoid).
(ii) The body B B B is “sandwiched” by this ellipsoid and its scaled-up version:
E i n ⊆ B ⊆ d E i n . \mathcal{E}_{\mathrm{in}}\subseteq B\subseteq\sqrt{d}\mathcal{E}_{\mathrm{in}}. E in ⊆ B ⊆ d E in .
Proof. The case d = 0 d=0 d = 0 is immediate, so assume d ≥ 1 d\ge1 d ≥ 1 . Fix an identification V ≃ R d V\simeq\mathbb{R}^d V ≃ R d , and let B 2 \mathcal{B}_2 B 2 denote the Euclidean unit ball. Volume ratios and the claimed inclusions are invariant under invertible linear changes of coordinates. Symmetry and convexity imply 0 ∈ int B 0\in\operatorname{int}B 0 ∈ int B , so there exist ε , R > 0 \varepsilon,R>0 ε , R > 0 such that
ε B 2 ⊆ B ⊆ R B 2 . \varepsilon\mathcal{B}_2\subseteq B\subseteq R\mathcal{B}_2. ε B 2 ⊆ B ⊆ R B 2 .
Centering and the optimal shape matrix. Every ellipsoid of positive volume can be written uniquely as c + Q B 2 c+Q\mathcal{B}_2 c + Q B 2 , where c ∈ R d c\in\mathbb{R}^d c ∈ R d and Q ∈ S + + d Q\in S_{++}^d Q ∈ S ++ d is symmetric positive definite. If c + Q B 2 ⊆ B c+Q\mathcal{B}_2\subseteq B c + Q B 2 ⊆ B , symmetry gives − c + Q B 2 ⊆ B -c+Q\mathcal{B}_2\subseteq B − c + Q B 2 ⊆ B , and convexity then yields
Q B 2 = 1 2 [ ( c + Q B 2 ) + ( − c + Q B 2 ) ] ⊆ B . Q\mathcal{B}_2=\tfrac{1}{2}\bigl[(c+Q\mathcal{B}_2)+(-c+Q\mathcal{B}_2)\bigr]\subseteq B. Q B 2 = 2 1 [ ( c + Q B 2 ) + ( − c + Q B 2 ) ] ⊆ B .
Centering therefore preserves feasibility and volume, without any appeal to uniqueness.
Consider the set
Q : = { Q ∈ S + d : Q B 2 ⊆ B } , \mathcal{Q}:=\{Q\in S_+^d:Q\mathcal{B}_2\subseteq B\}, Q := { Q ∈ S + d : Q B 2 ⊆ B } ,
where S + d S_+^d S + d is the cone of symmetric positive semidefinite matrices. This set is closed and convex. Indeed, for Q 1 , Q 2 ∈ Q Q_1,Q_2\in\mathcal{Q} Q 1 , Q 2 ∈ Q , t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] , and z ∈ B 2 z\in\mathcal{B}_2 z ∈ B 2 , convexity of B B B gives ( ( 1 − t ) Q 1 + t Q 2 ) z ∈ B ((1-t)Q_1+tQ_2)z\in B (( 1 − t ) Q 1 + t Q 2 ) z ∈ B . Moreover, Q \mathcal{Q} Q is bounded because every Q ∈ Q Q\in\mathcal{Q} Q ∈ Q satisfies ∥ Q ∥ o p ≤ R \lVert Q\rVert_{\mathrm{op}}\le R ∥ Q ∥ op ≤ R . Thus Q \mathcal{Q} Q is compact. Since ε I d ∈ Q \varepsilon I_d\in\mathcal{Q} ε I d ∈ Q , the maximum of det Q \det Q det Q over Q \mathcal{Q} Q is positive and is attained at a positive definite matrix Q ∗ Q^* Q ∗ . The volume of Q B 2 Q\mathcal{B}_2 Q B 2 is det Q \det Q det Q times the volume of B 2 \mathcal{B}_2 B 2 , so Q ∗ B 2 Q^*\mathcal{B}_2 Q ∗ B 2 is a maximal-volume ellipsoid among all feasible ellipsoids by the centering argument.
The centered maximizer is unique: maximizing det Q \det Q det Q on positive definite matrices is equivalent to maximizing the strictly concave function log det Q \log\det Q log det Q over the convex set Q ∩ S + + d \mathcal{Q}\cap S_{++}^d Q ∩ S ++ d . Strict concavity follows from
D 2 ( log det Q ) [ H , H ] = − Tr ( Q − 1 H Q − 1 H ) < 0 D^2(\log\det Q)[H,H]=-\operatorname{Tr}(Q^{-1}HQ^{-1}H)<0 D 2 ( log det Q ) [ H , H ] = − Tr ( Q − 1 H Q − 1 H ) < 0
for every nonzero symmetric matrix H H H . We prove below that an optimal ellipsoid cannot have a nonzero center.
A finite contact decomposition. Normalize by the linear map T : = ( Q ∗ ) − 1 T:=(Q^*)^{-1} T := ( Q ∗ ) − 1 and write B ~ : = T ( B ) \widetilde{B}:=T(B) B := T ( B ) . The body B ~ \widetilde{B} B is centrally symmetric, contains B 2 \mathcal{B}_2 B 2 , and has B 2 \mathcal{B}_2 B 2 as its unique centered maximal-volume ellipsoid. Define its polar body by
B ~ ∘ : = { y ∈ R d : ⟨ y , x ⟩ ≤ 1 for all x ∈ B ~ } . \widetilde{B}^{\circ}:=\{y\in\mathbb{R}^d:\langle y,x\rangle\le1\text{ for all }x\in\widetilde{B}\}. B ∘ := { y ∈ R d : ⟨ y , x ⟩ ≤ 1 for all x ∈ B } .
It is compact and contained in B 2 \mathcal{B}_2 B 2 . By the separating-hyperplane characterization of a closed convex body, for a symmetric positive definite matrix Q Q Q ,
Q B 2 ⊆ B ~ ⟺ ∥ Q y ∥ 2 ≤ 1 for all y ∈ B ~ ∘ . Q\mathcal{B}_2\subseteq\widetilde{B}\quad\Longleftrightarrow\quad\lVert Qy\rVert_2\le1\quad\text{for all }y\in\widetilde{B}^{\circ}. Q B 2 ⊆ B ⟺ ∥ Q y ∥ 2 ≤ 1 for all y ∈ B ∘ .
Here we used sup z ∈ B 2 ⟨ y , Q z ⟩ = ∥ Q y ∥ 2 \sup_{z\in\mathcal{B}_2}\langle y,Qz\rangle=\lVert Qy\rVert_2 sup z ∈ B 2 ⟨ y , Q z ⟩ = ∥ Q y ∥ 2 . Consequently, I d I_d I d solves the finite-dimensional convex problem
min Q ∈ S + + d − log det Q subject to g ( Q ) : = max y ∈ B ~ ∘ ∥ Q y ∥ 2 2 − 1 ≤ 0. \min_{Q\in S_{++}^d}-\log\det Q\quad\text{subject to}\quad g(Q):=\max_{y\in\widetilde{B}^{\circ}}\lVert Qy\rVert_2^2-1\le0. Q ∈ S ++ d min − log det Q subject to g ( Q ) := y ∈ B ∘ max ∥ Q y ∥ 2 2 − 1 ≤ 0.
The function g g g is finite, continuous, and convex, since it is the maximum of continuous convex quadratic functions indexed by a compact set. Slater’s condition holds: Q = 1 2 I d Q=\tfrac{1}{2}I_d Q = 2 1 I d is positive definite and satisfies g ( Q ) ≤ − 3 4 g(Q)\le-\tfrac{3}{4} g ( Q ) ≤ − 4 3 . The convex KKT theorem therefore supplies a multiplier μ ≥ 0 \mu\ge0 μ ≥ 0 and a subgradient G ∈ ∂ g ( I d ) G\in\partial g(I_d) G ∈ ∂ g ( I d ) such that the Lagrangian
L ( Q , μ ) = − log det Q + μ g ( Q ) \mathcal{L}(Q,\mu)=-\log\det Q+\mu g(Q) L ( Q , μ ) = − log det Q + μg ( Q )
satisfies stationarity and complementary slackness:
− I d + μ G = 0 , μ g ( I d ) = 0. -I_d+\mu G=0,\qquad\mu g(I_d)=0. − I d + μ G = 0 , μg ( I d ) = 0.
In particular, μ > 0 \mu>0 μ > 0 and g ( I d ) = 0 g(I_d)=0 g ( I d ) = 0 .
Equip the space of symmetric matrices with the pairing ⟨ H , K ⟩ = Tr ( H K ) \langle H,K\rangle=\operatorname{Tr}(HK) ⟨ H , K ⟩ = Tr ( H K ) . The subdifferential rule for a maximum of differentiable convex functions over a compact index set gives
∂ g ( I d ) = conv { 2 y y T : y ∈ B ~ ∘ , ∥ y ∥ 2 = 1 } , \partial g(I_d)=\operatorname{conv}\{2yy^{\mathsf T}:y\in\widetilde{B}^{\circ},\ \lVert y\rVert_2=1\}, ∂ g ( I d ) = conv { 2 y y T : y ∈ B ∘ , ∥ y ∥ 2 = 1 } ,
because the gradient of Q ↦ ∥ Q y ∥ 2 2 Q \mapsto\lVert Qy\rVert_{2}^{2} Q ↦ ∥ Q y ∥ 2 2 at I d I_d I d is 2 y y T 2yy^{\mathsf{T}} 2 y y T . The maximum is attained, and these gradients depend continuously on y y y , so the compact-index maximum rule applies. By Carathéodory’s theorem in the finite-dimensional space of symmetric matrices, G G G is a finite convex combination of these active gradients. Absorbing the factor 2 μ 2\mu 2 μ into the coefficients and discarding zero coefficients gives contact points y 1 , … , y k y_1,\ldots,y_k y 1 , … , y k and weights λ 1 , … , λ k > 0 \lambda_1,\ldots,\lambda_k>0 λ 1 , … , λ k > 0 satisfying
I d = ∑ i = 1 k λ i y i y i T . (88) I_d=\sum_{i=1}^{k}\lambda_i y_i y_i^{\mathsf{T}}. \tag*{(88)} I d = i = 1 ∑ k λ i y i y i T . (88)
Each y i y_i y i belongs to B 2 ⊆ B ~ \mathcal{B}_2\subseteq\widetilde{B} B 2 ⊆ B and to B ~ ∘ \widetilde{B}^{\circ} B ∘ , and ⟨ y i , y i ⟩ = 1 \langle y_i,y_i\rangle=1 ⟨ y i , y i ⟩ = 1 . Thus
y i ∈ ∂ B ~ ∩ ∂ B 2 , ∥ y i ∥ 2 = 1. (89) y_i\in\partial\widetilde{B}\cap\partial\mathcal{B}_2,\qquad\lVert y_i\rVert_2=1. \tag*{(89)} y i ∈ ∂ B ∩ ∂ B 2 , ∥ y i ∥ 2 = 1. (89)
Taking traces in (88) yields
∑ i = 1 k λ i = d . (90) \sum_{i=1}^{k}\lambda_i=d. \tag*{(90)} i = 1 ∑ k λ i = d . (90)
At each contact point y i y_i y i , a supporting hyperplane of B ~ \widetilde{B} B exists and also supports B 2 \mathcal{B}_2 B 2 , because B 2 ⊆ B ~ \mathcal{B}_2\subseteq\widetilde{B} B 2 ⊆ B . The Euclidean ball has the unique supporting hyperplane { x : ⟨ x , y i ⟩ = 1 } \{x:\langle x,y_i\rangle=1\} { x : ⟨ x , y i ⟩ = 1 } at y i y_i y i . Hence
⟨ x , y i ⟩ ≤ 1 , x ∈ B ~ , i = 1 , … , k . (91) \langle x,y_i\rangle\leq1,\qquad x\in\widetilde{B},\quad i=1,\ldots,k. \tag*{(91)} ⟨ x , y i ⟩ ≤ 1 , x ∈ B , i = 1 , … , k . (91)
By central symmetry, the same inequality applied to − x -x − x gives ∣ ⟨ x , y i ⟩ ∣ ≤ 1 \lvert\langle x,y_i\rangle\rvert\leq1 ∣⟨ x , y i ⟩∣ ≤ 1 .
Uniqueness among all ellipsoids. Let c + Q B 2 ⊆ B c+Q\mathcal{B}_2\subseteq B c + Q B 2 ⊆ B be any maximal-volume ellipsoid. Centering it preserves its volume, so uniqueness of the centered maximizer gives Q = Q ∗ Q=Q_* Q = Q ∗ . After applying T T T , this ellipsoid is a + B 2 ⊆ B ~ a+\mathcal{B}_2\subseteq\widetilde{B} a + B 2 ⊆ B , where a = T c a=Tc a = T c . For each i i i , the two points a + y i a+y_i a + y i and a − y i a-y_i a − y i belong to B ~ \widetilde{B} B . Applying the two-sided supporting inequality to them gives ⟨ a , y i ⟩ ≤ 0 \langle a,y_i\rangle\leq0 ⟨ a , y i ⟩ ≤ 0 and ⟨ a , y i ⟩ ≥ 0 \langle a,y_i\rangle\geq0 ⟨ a , y i ⟩ ≥ 0 , respectively. Thus ⟨ a , y i ⟩ = 0 \langle a,y_i\rangle=0 ⟨ a , y i ⟩ = 0 for every i i i , and (88) implies
∥ a ∥ 2 2 = ∑ i = 1 k λ i ⟨ a , y i ⟩ 2 = 0. \lVert a\rVert_2^2=\sum_{i=1}^{k}\lambda_i\langle a,y_i\rangle^2=0. ∥ a ∥ 2 2 = i = 1 ∑ k λ i ⟨ a , y i ⟩ 2 = 0.
Therefore c = 0 c=0 c = 0 , proving uniqueness among all ellipsoids and identifying E i n = Q ∗ B 2 \mathcal{E}_{\mathrm{in}}=Q_*\mathcal{B}_2 E in = Q ∗ B 2 .
The sandwich inequality. For every x ∈ B ~ x\in\widetilde{B} x ∈ B , the contact decomposition and the supporting inequalities give
∥ x ∥ 2 2 = ∑ i = 1 k λ i ⟨ x , y i ⟩ 2 ≤ ∑ i = 1 k λ i = d . \lVert x\rVert_2^2=\sum_{i=1}^{k}\lambda_i\langle x,y_i\rangle^2\leq\sum_{i=1}^{k}\lambda_i=d. ∥ x ∥ 2 2 = i = 1 ∑ k λ i ⟨ x , y i ⟩ 2 ≤ i = 1 ∑ k λ i = d .
Hence B 2 ⊆ B ~ ⊆ d B 2 \mathcal{B}_2\subseteq\widetilde{B}\subseteq\sqrt{d}\mathcal{B}_2 B 2 ⊆ B ⊆ d B 2 . Applying T − 1 T^{-1} T − 1 yields E i n ⊆ B ⊆ d E i n \mathcal{E}_{\mathrm{in}}\subseteq B\subseteq\sqrt{d}\mathcal{E}_{\mathrm{in}} E in ⊆ B ⊆ d E in .
We now proceed with a dimension-dependent regularity guarantee that holds for every d d d -dimensional normed space. It will serve as a baseline for comparison with the sharper, structure-dependent bounds stated next.
Proposition 10 (Universal Regularity Bound). Let d ≥ 1 d\geq1 d ≥ 1 . Any d d d -dimensional normed space ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) is d d d -regular in the sense of Definition 1, i.e., it is κ V \kappa_V κ V -regular with the choice κ V = d \kappa_V=d κ V = d .
Proof. To simplify notation throughout this proof, we omit the subscript V V V and write ∥ ⋅ ∥ : = ∥ ⋅ ∥ V \lVert\cdot\rVert:=\lVert\cdot\rVert_V ∥ ⋅ ∥ := ∥ ⋅ ∥ V , ∥ ⋅ ∥ + : = ∥ ⋅ ∥ V , + \lVert\cdot\rVert_+:=\lVert\cdot\rVert_{V,+} ∥ ⋅ ∥ + := ∥ ⋅ ∥ V , + , κ + : = κ V , + \kappa_+:=\kappa_{V,+} κ + := κ V , + , and κ : = κ V \kappa:=\kappa_V κ := κ V . The proof proceeds by explicitly constructing the required smooth norm and verifying that it satisfies the conditions of Definition 1 with κ = d \kappa=d κ = d . The construction relies on the geometric sandwich inequality provided by John’s Ellipsoid Theorem.
Step 1: Applying John’s Theorem to the Unit Ball. Let ( V , ∥ ⋅ ∥ ) (V,\lVert\cdot\rVert) ( V , ∥ ⋅ ∥) be a d d d -dimensional normed space. Its unit ball, B = { x ∈ V : ∥ x ∥ ≤ 1 } B=\{x\in V:\lVert x\rVert\leq1\} B = { x ∈ V : ∥ x ∥ ≤ 1 } , is a centrally symmetric, compact, convex body containing the origin. By John’s Ellipsoid Theorem (Theorem 7), there exists a unique ellipsoid of maximal volume E i n \mathcal{E}_{\mathrm{in}} E in contained in B B B , such that
E i n ⊆ B ⊆ d E i n . \mathcal{E}_{\mathrm{in}} \subseteq B \subseteq\sqrt{d}\mathcal{E}_{\mathrm{in}}. E in ⊆ B ⊆ d E in .
Step 2: From Geometric Inclusion to an Algebraic Norm Inequality. Any ellipsoid centered at the origin can be represented as the unit ball of a Euclidean norm. Let us define the Euclidean norm ∥ ⋅ ∥ + \lVert\cdot\rVert_{+} ∥ ⋅ ∥ + on V V V as the one whose unit ball is precisely the John ellipsoid E i n \mathcal{E}_{\mathrm{in}} E in , i.e., E i n = { x ∈ V : ∥ x ∥ + ≤ 1 } \mathcal{E}_{\mathrm{in}} = \{x \in V : \lVert x \rVert_{+} \le1\} E in = { x ∈ V : ∥ x ∥ + ≤ 1 } .
We can now translate the geometric inclusions into algebraic inequalities between the norms ∥ ⋅ ∥ \lVert\cdot\rVert ∥ ⋅ ∥ and ∥ ⋅ ∥ + \lVert\cdot\rVert_{+} ∥ ⋅ ∥ + :
The inclusion E i n ⊆ B \mathcal{E}_{\mathrm{in}} \subseteq B E in ⊆ B means that if ∥ x ∥ + ≤ 1 \lVert x \rVert_{+} \le1 ∥ x ∥ + ≤ 1 , then ∥ x ∥ ≤ 1 \lVert x \rVert\le1 ∥ x ∥ ≤ 1 . By homogeneity of norms, this implies ∥ x ∥ ≤ ∥ x ∥ + \lVert x \rVert\le\lVert x \rVert_{+} ∥ x ∥ ≤ ∥ x ∥ + for all x ∈ V x \in V x ∈ V .
The inclusion B ⊆ d E i n B \subseteq\sqrt{d}\mathcal{E}_{\mathrm{in}} B ⊆ d E in means that if ∥ x ∥ ≤ 1 \lVert x \rVert\le1 ∥ x ∥ ≤ 1 , then ∥ x ∥ + ≤ d \lVert x \rVert_{+} \le\sqrt{d} ∥ x ∥ + ≤ d . By homogeneity, this implies ∥ x ∥ + ≤ d ∥ x ∥ \lVert x \rVert_{+} \le\sqrt{d}\lVert x \rVert ∥ x ∥ + ≤ d ∥ x ∥ for all x ∈ V x \in V x ∈ V .
Combining these two results, we have the norm equivalence inequality ∥ x ∥ ≤ ∥ x ∥ + ≤ d ∥ x ∥ \lVert x \rVert\le\lVert x \rVert_{+} \le\sqrt{d}\lVert x \rVert ∥ x ∥ ≤ ∥ x ∥ + ≤ d ∥ x ∥ for all x ∈ V x \in V x ∈ V . Squaring this expression gives the form required by the definition of regularity:
∥ x ∥ 2 ≤ ∥ x ∥ + 2 ≤ d ∥ x ∥ 2 , ∀ x ∈ V . (92) \lVert x \rVert^{2} \le\lVert x \rVert_{+}^{2} \le d\lVert x \rVert^{2}, \qquad\forall x \in V. \tag*{(92)} ∥ x ∥ 2 ≤ ∥ x ∥ + 2 ≤ d ∥ x ∥ 2 , ∀ x ∈ V . (92)
Step 3: Verifying the Conditions for d d d -Regularity. To show that ( V , ∥ ⋅ ∥ ) (V, \lVert\cdot\rVert) ( V , ∥ ⋅ ∥) is d d d -regular, we must find a κ + \kappa_{+} κ + -smooth auxiliary norm ∥ ⋅ ∥ + \lVert\cdot\rVert_{+} ∥ ⋅ ∥ + and a constant κ ≥ κ + \kappa\ge\kappa_{+} κ ≥ κ + such that
∥ x ∥ 2 ≤ ∥ x ∥ + 2 ≤ κ κ + ∥ x ∥ 2 . \lVert x \rVert^{2} \le\lVert x \rVert_{+}^{2} \le\frac{\kappa}{\kappa_{+}}\lVert x \rVert^{2}. ∥ x ∥ 2 ≤ ∥ x ∥ + 2 ≤ κ + κ ∥ x ∥ 2 .
Let us choose the Euclidean norm ∥ ⋅ ∥ + \lVert\cdot\rVert_{+} ∥ ⋅ ∥ + from Step 2. As a Euclidean norm, the space ( V , ∥ ⋅ ∥ + ) (V, \lVert\cdot\rVert_{+}) ( V , ∥ ⋅ ∥ + ) is 1-smooth. To see this, note that for φ + ( x ) : = 1 2 ∥ x ∥ + 2 \varphi_{+}(x) := \frac{1}{2}\lVert x \rVert_{+}^{2} φ + ( x ) := 2 1 ∥ x ∥ + 2 , there exists an inner product ⟨ ⋅ , ⋅ ⟩ + \langle\cdot,\cdot\rangle_{+} ⟨ ⋅ , ⋅ ⟩ + with ∥ x ∥ + 2 = ⟨ x , x ⟩ + \lVert x \rVert_{+}^{2} = \langle x,x\rangle_{+} ∥ x ∥ + 2 = ⟨ x , x ⟩ + , and the descent inequality holds with equality for κ + = 1 \kappa_{+} = 1 κ + = 1 :
φ + ( x + y ) = 1 2 ∥ x + y ∥ + 2 = 1 2 ∥ x ∥ + 2 + ⟨ x , y ⟩ + + 1 2 ∥ y ∥ + 2 = φ + ( x ) + ⟨ ∇ φ + ( x ) , y ⟩ + 1 2 ∥ y ∥ + 2 . \begin{aligned}
\varphi_{+}(x+y) &= \frac{1}{2}\lVert x+y\rVert_{+}^{2}
= \frac{1}{2}\lVert x\rVert_{+}^{2}+\langle x,y\rangle_{+}+\frac{1}{2}\lVert y\rVert_{+}^{2} \\
&= \varphi_{+}(x)+\langle\nabla\varphi_{+}(x),y\rangle+\frac{1}{2}\lVert y\rVert_{+}^{2}.
\end{aligned} φ + ( x + y ) = 2 1 ∥ x + y ∥ + 2 = 2 1 ∥ x ∥ + 2 + ⟨ x , y ⟩ + + 2 1 ∥ y ∥ + 2 = φ + ( x ) + ⟨ ∇ φ + ( x ) , y ⟩ + 2 1 ∥ y ∥ + 2 .
where ∇ φ + ( x ) ∈ V ∗ \nabla\varphi_{+}(x) \in V^{*} ∇ φ + ( x ) ∈ V ∗ is the linear functional y ↦ ⟨ x , y ⟩ + y \mapsto\langle x,y\rangle_{+} y ↦ ⟨ x , y ⟩ + . Therefore, we can set κ + = 1 \kappa_{+} = 1 κ + = 1 .
Now we check the compatibility inequality from Definition 1:
∥ x ∥ + 2 ≤ κ κ + ∥ x ∥ 2 . \lVert x \rVert_{+}^{2} \le\frac{\kappa}{\kappa_{+}}\lVert x \rVert^{2}. ∥ x ∥ + 2 ≤ κ + κ ∥ x ∥ 2 .
Substituting κ + = 1 \kappa_{+} = 1 κ + = 1 and comparing with (92) , we see a direct match if we set κ = d \kappa= d κ = d :
∥ x ∥ + 2 ≤ d ∥ x ∥ 2 . \lVert x \rVert_{+}^{2} \le d\lVert x \rVert^{2}. ∥ x ∥ + 2 ≤ d ∥ x ∥ 2 .
The constructed norm ∥ ⋅ ∥ + \lVert\cdot\rVert_{+} ∥ ⋅ ∥ + is 1-smooth (so κ + = 1 \kappa_{+} = 1 κ + = 1 ) and satisfies the required inequality for κ = d \kappa= d κ = d . Therefore, by Definition 1, the space ( V , ∥ ⋅ ∥ V ) (V, \lVert\cdot\rVert_{V}) ( V , ∥ ⋅ ∥ V ) is d d d -regular.
While Proposition 10 establishes that κ V = d \kappa_{V} = d κ V = d is sufficient for any finite-dimensional space, one might ask whether a sub-linear dependence is achievable universally. The following proposition answers this in the negative by demonstrating that for the ℓ 1 \ell_{1} ℓ 1 norm, the constant of regularity must scale linearly with the dimension. Thus, the universal bound is tight up to absolute constants.
Proposition 11 (Tightness of the universal d d d -regularity bound). Fix d ∈ N d \in\mathbb{N} d ∈ N and consider the normed space ( V , ∥ ⋅ ∥ V ) (V, \lVert\cdot\rVert_{V}) ( V , ∥ ⋅ ∥ V ) with V : = R d V := \mathbb{R}^{d} V := R d and ∥ ⋅ ∥ V : = ∥ ⋅ ∥ 1 \lVert\cdot\rVert_{V} := \lVert\cdot\rVert_{1} ∥ ⋅ ∥ V := ∥ ⋅ ∥ 1 . If ( V , ∥ ⋅ ∥ V ) (V, \lVert\cdot\rVert_{V}) ( V , ∥ ⋅ ∥ V ) is κ V \kappa_{V} κ V -regular in the sense of Definition 1, then necessarily
κ V ≥ d . (93) \kappa_{V} \ge d. \tag*{(93)} κ V ≥ d . (93) In particular, the universal upper bound of Proposition 10 is tight (up to absolute constants): there exist d d d -dimensional normed spaces whose constant of regularity is of order d d d .
Proof. Let { ε i } i = 1 d \{\varepsilon_i\}_{i=1}^d { ε i } i = 1 d be i.i.d. Rademacher random variables, i.e. Prob { ε i = 1 } = Prob { ε i = − 1 } = 1 2 \operatorname{Prob}\{\varepsilon_i=1\}=\operatorname{Prob}\{\varepsilon_i=-1\}=\frac{1}{2} Prob { ε i = 1 } = Prob { ε i = − 1 } = 2 1 , and let { e i } i = 1 d \{e_i\}_{i=1}^d { e i } i = 1 d be the standard basis of R d \mathbb{R}^d R d . Define the filtration F i : = σ ( ε 1 , … , ε i ) \mathcal{F}_i:=\sigma(\varepsilon_1,\ldots,\varepsilon_i) F i := σ ( ε 1 , … , ε i ) and the sequence
ξ i : = ε i e i , i = 1 , … , d . \xi_i:=\varepsilon_i e_i,\qquad i=1,\ldots,d. ξ i := ε i e i , i = 1 , … , d .
Then { ξ i } i = 1 d \{\xi_i\}_{i=1}^d { ξ i } i = 1 d is an { F i } \{\mathcal{F}_i\} { F i } -martingale-difference sequence: it is F i \mathcal{F}_i F i -measurable and
E [ ξ i ∣ F i − 1 ] = e i E [ ε i ∣ F i − 1 ] = e i E [ ε i ] = 0 a.s. \mathbb{E}\left[\xi_i\mid\mathcal{F}_{i-1}\right]=e_i\mathbb{E}\left[\varepsilon_i\mid\mathcal{F}_{i-1}\right]=e_i\mathbb{E}[\varepsilon_i]=0\quad\text{a.s.} E [ ξ i ∣ F i − 1 ] = e i E [ ε i ∣ F i − 1 ] = e i E [ ε i ] = 0 a.s.
Moreover, ∥ ξ i ∥ 1 = ∥ e i ∥ 1 = 1 \|\xi_i\|_1=\|e_i\|_1=1 ∥ ξ i ∥ 1 = ∥ e i ∥ 1 = 1 , hence E [ ∥ ξ i ∥ 1 2 ] = 1 \mathbb{E}\left[\|\xi_i\|_1^2\right]=1 E [ ∥ ξ i ∥ 1 2 ] = 1 and therefore
∑ i = 1 d E [ ∥ ξ i ∥ 1 2 ] = d . (94) \sum_{i=1}^d\mathbb{E}\left[\|\xi_i\|_1^2\right]=d. \tag*{(94)} i = 1 ∑ d E [ ∥ ξ i ∥ 1 2 ] = d . (94)
Let S d : = ∑ i = 1 d ξ i = ( ε 1 , … , ε d ) S_d:=\sum_{i=1}^d\xi_i=(\varepsilon_1,\ldots,\varepsilon_d) S d := ∑ i = 1 d ξ i = ( ε 1 , … , ε d ) . Since ∣ ε i ∣ = 1 |\varepsilon_i|=1 ∣ ε i ∣ = 1 a.s., we have deterministically
∥ S d ∥ 1 = ∑ i = 1 d ∣ ε i ∣ = d , (95) \|S_d\|_1=\sum_{i=1}^d|\varepsilon_i|=d, \tag*{(95)} ∥ S d ∥ 1 = i = 1 ∑ d ∣ ε i ∣ = d , (95)
and consequently
E [ ∥ S d ∥ 1 2 ] = d 2 . (96) \mathbb{E}\left[\|S_d\|_1^2\right]=d^2. \tag*{(96)} E [ ∥ S d ∥ 1 2 ] = d 2 . (96)
On the other hand, since ( R d , ∥ ⋅ ∥ 1 ) (\mathbb{R}^d,\|\cdot\|_1) ( R d , ∥ ⋅ ∥ 1 ) is assumed κ V \kappa_V κ V -regular and { ξ i } \{\xi_i\} { ξ i } is a martingale-difference sequence, the second-moment inequality of Lemma 15 yields
E [ ∥ S d ∥ 1 2 ] ≤ κ V ∑ i = 1 d E [ ∥ ξ i ∥ 1 2 ] . (97) \mathbb{E}\left[\|S_d\|_1^2\right]\leq\kappa_V\sum_{i=1}^d\mathbb{E}\left[\|\xi_i\|_1^2\right]. \tag*{(97)} E [ ∥ S d ∥ 1 2 ] ≤ κ V i = 1 ∑ d E [ ∥ ξ i ∥ 1 2 ] . (97)
Combining (97) with (94) and (96) gives
d 2 ≤ κ V d , d^2\leq\kappa_Vd, d 2 ≤ κ V d ,
which implies κ V ≥ d \kappa_V\geq d κ V ≥ d . This is exactly (93) . □
ℓ p \ell_p ℓ p Spaces
We now determine essentially tight regularity constants for ℓ p d \ell_p^d ℓ p d normed spaces ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) with V = R d V=\mathbb{R}^d V = R d and ∥ ⋅ ∥ V = ∥ ⋅ ∥ p \|\cdot\|_V=\|\cdot\|_p ∥ ⋅ ∥ V = ∥ ⋅ ∥ p , distinguishing between the regime p ≥ 2 p\geq2 p ≥ 2 (where κ V \kappa_V κ V can be as small as logarithmic in d d d ) and the regime p < 2 p<2 p < 2 (where κ V \kappa_V κ V necessarily grows polynomially in d d d ).
Proposition 12 (ℓ p d \ell_p^d ℓ p d spaces). Fix d ∈ N d\in\mathbb{N} d ∈ N and p ∈ [ 1 , ∞ ] p\in[1,\infty] p ∈ [ 1 , ∞ ] . Let V : = R d V:=\mathbb{R}^d V := R d and equip it with the norm ∥ ⋅ ∥ V : = ∥ ⋅ ∥ p \|\cdot\|_V:=\|\cdot\|_p ∥ ⋅ ∥ V := ∥ ⋅ ∥ p , where
∥ x ∥ p : = ( ∑ i = 1 d ∣ x i ∣ p ) 1 / p ( 1 ≤ p < ∞ ) , ∥ x ∥ ∞ : = max 1 ≤ i ≤ d ∣ x i ∣ . \|x\|_p:=\left(\sum_{i=1}^d|x_i|^p\right)^{1/p}\quad(1\leq p<\infty),\qquad\|x\|_\infty:=\max_{1\leq i\leq d}|x_i|. ∥ x ∥ p := ( i = 1 ∑ d ∣ x i ∣ p ) 1/ p ( 1 ≤ p < ∞ ) , ∥ x ∥ ∞ := 1 ≤ i ≤ d max ∣ x i ∣.
Then the space ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) is κ V \kappa_V κ V -regular in the sense of Definition 1, with admissible regularity parameter κ V : = κ p ( d ) \kappa_V:=\kappa_p(d) κ V := κ p ( d ) given as follows:
Let 2 ≤ p ≤ ∞ 2\leq p\leq\infty 2 ≤ p ≤ ∞ . Then
κ V : = κ p ( d ) = min 2 ≤ ρ ≤ p ( ρ − 1 ) d 2 ρ − 2 p . \kappa_V:=\kappa_p(d)=\min_{2\leq\rho\leq p}(\rho-1)d^{\frac{2}{\rho}-\frac{2}{p}}. κ V := κ p ( d ) = 2 ≤ ρ ≤ p min ( ρ − 1 ) d ρ 2 − p 2 .
When p = ∞ p=\infty p = ∞ , we use 1 / p = 0 1/p=0 1/ p = 0 and the auxiliary exponent ranges over [ 2 , ∞ ) [2,\infty) [ 2 , ∞ ) . Furthermore, if d ≥ 8 d\geq8 d ≥ 8 , then
κ V : = κ p ( d ) ≤ min { p − 1 , 2 e 2 ln ( d ) } , \kappa_V:=\kappa_p(d)\leq\min\{p-1,2e^2\ln(d)\}, κ V := κ p ( d ) ≤ min { p − 1 , 2 e 2 ln ( d )} ,
with the corresponding auxiliary norm
∥ ⋅ ∥ V , + : = ∥ ⋅ ∥ ρ ∗ , ρ ∗ : = min { p , ln d + ( ln d ) 2 − 2 ln d } . \|\cdot\|_{V,+}:=\|\cdot\|_{\rho_*},\qquad\rho_*:=\min\left\{p,\ln d+\sqrt{(\ln d)^2-2\ln d}\right\}. ∥ ⋅ ∥ V , + := ∥ ⋅ ∥ ρ ∗ , ρ ∗ := min { p , ln d + ( ln d ) 2 − 2 ln d } .
Let 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 . Then
κ V : = κ p ( d ) = d 2 p − 1 , \kappa_V := \kappa_p(d) = d^{\frac{2}{p}-1}, κ V := κ p ( d ) = d p 2 − 1 ,
with the corresponding auxiliary norm
∥ x ∥ V , + : = d 1 p − 1 2 ∥ x ∥ 2 , x ∈ R d . \|x\|_{V,+} := d^{\frac{1}{p}-\frac{1}{2}}\|x\|_2,\qquad x \in\mathbb{R}^d. ∥ x ∥ V , + := d p 1 − 2 1 ∥ x ∥ 2 , x ∈ R d .
Proof. To simplify notation throughout this proof, we write
∥ x ∥ : = ∥ x ∥ V = ∥ x ∥ p , κ : = κ V . \|x\| := \|x\|_V = \|x\|_p,\qquad\kappa:= \kappa_V. ∥ x ∥ := ∥ x ∥ V = ∥ x ∥ p , κ := κ V .
Whenever an auxiliary norm is introduced, we denote it by
∥ x ∥ + : = ∥ x ∥ V , + , κ + : = κ V , + . \|x\|_+ := \|x\|_{V,+},\qquad\kappa_+ := \kappa_{V,+}. ∥ x ∥ + := ∥ x ∥ V , + , κ + := κ V , + .
The letters x , h ∈ V = R d x,h \in V = \mathbb{R}^d x , h ∈ V = R d denote generic vectors; this is only a local renaming of the dummy variables in Definition 1.
We start with Case 1: 2 ≤ p ≤ + ∞ 2 \le p \le+\infty 2 ≤ p ≤ + ∞ .
The proof proceeds in three steps. First, we establish that for any 2 ≤ ρ < ∞ 2 \le\rho< \infty 2 ≤ ρ < ∞ , the space ( R d , ∥ ⋅ ∥ ρ ) (\mathbb{R}^d,\|\cdot\|_\rho) ( R d , ∥ ⋅ ∥ ρ ) is ( ρ − 1 ) (\rho-1) ( ρ − 1 ) -smooth. Second, we use this result with a standard norm equivalence inequality to derive the regularity constant for ( R d , ∥ ⋅ ∥ p ) (\mathbb{R}^d,\|\cdot\|_p) ( R d , ∥ ⋅ ∥ p ) . Lastly, we provide a bound depending only on p p p and d d d .
Step 1: Smoothness of ( R d , ∥ ⋅ ∥ ρ ) (\mathbb{R}^d,\|\cdot\|_\rho) ( R d , ∥ ⋅ ∥ ρ ) for $2 \le \rho < \infty. Fix ρ ∈ [ 2 , p ] \rho\in[2,p] ρ ∈ [ 2 , p ] with ρ < ∞ \rho< \infty ρ < ∞ and use ∥ x ∥ + : = ∥ x ∥ ρ \|x\|_+ := \|x\|_\rho ∥ x ∥ + := ∥ x ∥ ρ as the candidate auxiliary norm on V V V . Define
ϕ + ( x ) : = 1 2 ∥ x ∥ + 2 = 1 2 ∥ x ∥ ρ 2 , ψ ( x ) : = 2 ϕ + ( x ) = ∥ x ∥ ρ 2 . \phi_+(x) := \frac{1}{2}\|x\|^2_+ = \frac{1}{2}\|x\|_\rho^2,\qquad\psi(x) := 2\phi_+(x) = \|x\|_\rho^2. ϕ + ( x ) := 2 1 ∥ x ∥ + 2 = 2 1 ∥ x ∥ ρ 2 , ψ ( x ) := 2 ϕ + ( x ) = ∥ x ∥ ρ 2 .
To prove that ( V , ∥ ⋅ ∥ + ) (V,\|\cdot\|_+) ( V , ∥ ⋅ ∥ + ) is κ + \kappa_+ κ + -smooth with κ + = ρ − 1 \kappa_+ = \rho-1 κ + = ρ − 1 , it suffices to show that
⟨ ∇ 2 ψ ( x ) h , h ⟩ ≤ 2 ( ρ − 1 ) ∥ h ∥ + 2 = 2 ( ρ − 1 ) ∥ h ∥ ρ 2 \langle\nabla^2\psi(x)h,h\rangle\le2(\rho-1)\|h\|_+^2 = 2(\rho-1)\|h\|_\rho^2 ⟨ ∇ 2 ψ ( x ) h , h ⟩ ≤ 2 ( ρ − 1 ) ∥ h ∥ + 2 = 2 ( ρ − 1 ) ∥ h ∥ ρ 2
for all x ≠ 0 x \ne0 x = 0 and all h ∈ V h \in V h ∈ V . The case ρ = 2 \rho=2 ρ = 2 is Euclidean, hence 1-smooth, so we assume ρ > 2 \rho>2 ρ > 2 .
Since ρ > 2 \rho>2 ρ > 2 , the function ψ \psi ψ is twice continuously differentiable on R d ∖ { 0 } \mathbb{R}^d \setminus\{0\} R d ∖ { 0 } . For x ≠ 0 x \ne0 x = 0 , its gradient is
∇ ψ ( x ) = 2 ( ∑ i ∣ x i ∣ ρ ) 2 ρ − 1 ⋅ ( ∣ x 1 ∣ ρ − 1 sign ( x 1 ) , … , ∣ x d ∣ ρ − 1 sign ( x d ) ) T . \nabla\psi(x) = 2\left(\sum_i |x_i|^\rho\right)^{\frac{2}{\rho}-1}\cdot\left(|x_1|^{\rho-1}\operatorname{sign}(x_1),\ldots,|x_d|^{\rho-1}\operatorname{sign}(x_d)\right)^{\mathsf{T}}. ∇ ψ ( x ) = 2 ( i ∑ ∣ x i ∣ ρ ) ρ 2 − 1 ⋅ ( ∣ x 1 ∣ ρ − 1 sign ( x 1 ) , … , ∣ x d ∣ ρ − 1 sign ( x d ) ) T .
At the origin,
∣ ψ ( h ) − ψ ( 0 ) ∣ ∥ h ∥ ρ = ∥ h ∥ ρ ⟶ 0 as h → 0 , h ≠ 0 , \frac{|\psi(h)-\psi(0)|}{\|h\|_\rho} = \|h\|_\rho\longrightarrow0\qquad\text{as }h\to0,\ h\ne0, ∥ h ∥ ρ ∣ ψ ( h ) − ψ ( 0 ) ∣ = ∥ h ∥ ρ ⟶ 0 as h → 0 , h = 0 ,
so ψ \psi ψ is differentiable at 0 0 0 with ∇ ψ ( 0 ) = 0 \nabla\psi(0)=0 ∇ ψ ( 0 ) = 0 . Moreover, the gradient formula gives
∥ ∇ ψ ( x ) ∥ ρ / ( ρ − 1 ) = 2 ( ∑ i ∣ x i ∣ ρ ) 2 ρ − 1 ( ∑ i ∣ x i ∣ ρ ) ρ − 1 ρ = 2 ∥ x ∥ ρ ⟶ 0 as x → 0 , x ≠ 0. \|\nabla\psi(x)\|_{\rho/(\rho-1)}
=2\left(\sum_i |x_i|^\rho\right)^{\frac{2}{\rho}-1}
\left(\sum_i |x_i|^\rho\right)^{\frac{\rho-1}{\rho}}
=2\|x\|_\rho\longrightarrow0
\qquad\text{as }x\to0,\ x\ne0. ∥∇ ψ ( x ) ∥ ρ / ( ρ − 1 ) = 2 ( i ∑ ∣ x i ∣ ρ ) ρ 2 − 1 ( i ∑ ∣ x i ∣ ρ ) ρ ρ − 1 = 2∥ x ∥ ρ ⟶ 0 as x → 0 , x = 0.
Thus the gradient is also continuous at the origin, and ψ ∈ C 1 ( R d ) \psi\in C^1(\mathbb{R}^d) ψ ∈ C 1 ( R d ) .
Since ψ ( x ) \psi(x) ψ ( x ) is homogeneous of degree 2, it suffices to verify the bound for vectors on the unit sphere, i.e., where ∥ x ∥ ρ = 1 \|x\|_\rho=1 ∥ x ∥ ρ = 1 . For such an x x x , the Hessian action ⟨ ∇ 2 ψ ( x ) h , h ⟩ \langle\nabla^2\psi(x)h,h\rangle ⟨ ∇ 2 ψ ( x ) h , h ⟩ is:
⟨ ∇ 2 ψ ( x ) h , h ⟩ = 2 ( 2 − ρ ) ( ∑ i ∣ x i ∣ ρ ) 2 ρ − 2 ( ∑ i ∣ x i ∣ ρ − 1 sign ( x i ) h i ) 2 + 2 ( ∑ i ∣ x i ∣ ρ ) 2 ρ − 1 ∑ i ( ρ − 1 ) ∣ x i ∣ ρ − 2 h i 2 . \begin{aligned}
\langle\nabla^2\psi(x)h,h\rangle
&=2(2-\rho)\left(\sum_i |x_i|^\rho\right)^{\frac{2}{\rho}-2}
\left(\sum_i |x_i|^{\rho-1}\operatorname{sign}(x_i)h_i\right)^2 \\
&\quad+2\left(\sum_i |x_i|^\rho\right)^{\frac{2}{\rho}-1}
\sum_i(\rho-1)|x_i|^{\rho-2}h_i^2.
\end{aligned} ⟨ ∇ 2 ψ ( x ) h , h ⟩ = 2 ( 2 − ρ ) ( i ∑ ∣ x i ∣ ρ ) ρ 2 − 2 ( i ∑ ∣ x i ∣ ρ − 1 sign ( x i ) h i ) 2 + 2 ( i ∑ ∣ x i ∣ ρ ) ρ 2 − 1 i ∑ ( ρ − 1 ) ∣ x i ∣ ρ − 2 h i 2 .
Since ρ > 2 \rho> 2 ρ > 2 , the first term is non-positive. Dropping it and substituting ∑ i ∣ x i ∣ ρ = 1 \sum_i |x_i|^\rho= 1 ∑ i ∣ x i ∣ ρ = 1 gives the upper bound:
⟨ ∇ 2 ψ ( x ) h , h ⟩ ≤ 2 ( ρ − 1 ) ∑ i = 1 d ∣ x i ∣ ρ − 2 h i 2 . \langle\nabla^2\psi(x)h,h\rangle\le2(\rho-1)\sum_{i=1}^{d}|x_i|^{\rho-2}h_i^2. ⟨ ∇ 2 ψ ( x ) h , h ⟩ ≤ 2 ( ρ − 1 ) i = 1 ∑ d ∣ x i ∣ ρ − 2 h i 2 .
We now apply Hölder’s inequality to the sum on the right-hand side. Recall that for any two vectors a , b ∈ R d a,b \in\mathbb{R}^d a , b ∈ R d and conjugate exponents q , r > 1 q,r>1 q , r > 1 (i.e., 1 q + 1 r = 1 \frac{1}{q}+\frac{1}{r}=1 q 1 + r 1 = 1 ), the inequality states: ∑ i = 1 d ∣ a i b i ∣ ≤ ( ∑ i = 1 d ∣ a i ∣ q ) 1 / q ( ∑ i = 1 d ∣ b i ∣ r ) 1 / r \sum_{i=1}^{d}|a_i b_i| \le\left(\sum_{i=1}^{d}|a_i|^q\right)^{1/q}\left(\sum_{i=1}^{d}|b_i|^r\right)^{1/r} ∑ i = 1 d ∣ a i b i ∣ ≤ ( ∑ i = 1 d ∣ a i ∣ q ) 1/ q ( ∑ i = 1 d ∣ b i ∣ r ) 1/ r . In our case, we set a i = ∣ x i ∣ ρ − 2 a_i=|x_i|^{\rho-2} a i = ∣ x i ∣ ρ − 2 and b i = h i 2 b_i=h_i^2 b i = h i 2 . Since both terms are non-negative, the absolute value on the left-hand side is not needed. The conjugate exponents are q = ρ ρ − 2 q=\frac{\rho}{\rho-2} q = ρ − 2 ρ and r = ρ 2 r=\frac{\rho}{2} r = 2 ρ . Applying the inequality yields:
∑ i = 1 d ∣ x i ∣ ρ − 2 h i 2 ≤ ( ∑ i = 1 d ( ∣ x i ∣ ρ − 2 ) q ) 1 / q ( ∑ i = 1 d ( h i 2 ) r ) 1 / r = ( ∑ i = 1 d ∣ x i ∣ ρ ) ( ρ − 2 ) / ρ ( ∑ i = 1 d ∣ h i ∣ ρ ) 2 / ρ = ( ∥ x ∥ ρ ρ ) ( ρ − 2 ) / ρ ( ∥ h ∥ ρ ρ ) 2 / ρ = ∥ x ∥ ρ ρ − 2 ∥ h ∥ ρ 2 . \begin{aligned}
\sum_{i=1}^{d}|x_i|^{\rho-2}h_i^2
\le\left(\sum_{i=1}^{d}\left(|x_i|^{\rho-2}\right)^q\right)^{1/q}
\left(\sum_{i=1}^{d}\left(h_i^2\right)^r\right)^{1/r} \\
&= \left(\sum_{i=1}^{d}|x_i|^\rho\right)^{(\rho-2)/\rho}
\left(\sum_{i=1}^{d}|h_i|^\rho\right)^{2/\rho} \\
&= \left(\|x\|_\rho^\rho\right)^{(\rho-2)/\rho}
\left(\|h\|_\rho^\rho\right)^{2/\rho}
= \|x\|_\rho^{\rho-2}\|h\|_\rho^2.
\end{aligned} i = 1 ∑ d ∣ x i ∣ ρ − 2 h i 2 ≤ ( i = 1 ∑ d ( ∣ x i ∣ ρ − 2 ) q ) 1/ q ( i = 1 ∑ d ( h i 2 ) r ) 1/ r = ( i = 1 ∑ d ∣ x i ∣ ρ ) ( ρ − 2 ) / ρ ( i = 1 ∑ d ∣ h i ∣ ρ ) 2/ ρ = ( ∥ x ∥ ρ ρ ) ( ρ − 2 ) / ρ ( ∥ h ∥ ρ ρ ) 2/ ρ = ∥ x ∥ ρ ρ − 2 ∥ h ∥ ρ 2 .
Since we assumed ∥ x ∥ ρ = 1 \|x\|_\rho=1 ∥ x ∥ ρ = 1 , this gives ∑ i = 1 d ∣ x i ∣ ρ − 2 h i 2 ≤ ∥ h ∥ ρ 2 \sum_{i=1}^{d}|x_i|^{\rho-2}h_i^2\le\|h\|_\rho^2 ∑ i = 1 d ∣ x i ∣ ρ − 2 h i 2 ≤ ∥ h ∥ ρ 2 . Substituting back yields:
⟨ ∇ 2 ψ ( x ) h , h ⟩ ≤ 2 ( ρ − 1 ) ∥ h ∥ ρ 2 = 2 ( ρ − 1 ) ∥ h ∥ + 2 . \langle\nabla^2\psi(x)h,h\rangle\le2(\rho-1)\|h\|_\rho^2=2(\rho-1)\|h\|_+^2. ⟨ ∇ 2 ψ ( x ) h , h ⟩ ≤ 2 ( ρ − 1 ) ∥ h ∥ ρ 2 = 2 ( ρ − 1 ) ∥ h ∥ + 2 .
This confirms that the Hessian bound holds for every x ≠ 0 x\ne0 x = 0 . To obtain the global smoothness inequality, fix x , h ∈ R d x,h\in\mathbb{R}^d x , h ∈ R d . The case h = 0 h=0 h = 0 is immediate, so assume h ≠ 0 h\ne0 h = 0 and define F ( t ) : = ψ ( x + t h ) F(t):=\psi(x+th) F ( t ) := ψ ( x + t h ) for t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] . Then F ∈ C 1 ( [ 0 , 1 ] ) F\in C^1([0,1]) F ∈ C 1 ([ 0 , 1 ]) , and whenever x + t h ≠ 0 x+th\ne0 x + t h = 0 ,
F ′ ′ ( t ) = ⟨ ∇ 2 ψ ( x + t h ) h , h ⟩ ≤ 2 ( ρ − 1 ) ∥ h ∥ ρ 2 . F''(t)=\langle\nabla^2\psi(x+th)h,h\rangle\le2(\rho-1)\|h\|_\rho^2. F ′′ ( t ) = ⟨ ∇ 2 ψ ( x + t h ) h , h ⟩ ≤ 2 ( ρ − 1 ) ∥ h ∥ ρ 2 .
The segment can pass through the origin at at most one parameter value t 0 t_0 t 0 . Integrating this bound on intervals avoiding t 0 t_0 t 0 , taking limits using continuity of F ′ F' F ′ , and adding the two inequalities when the interval crosses t 0 t_0 t 0 gives
F ′ ( t ) − F ′ ( 0 ) ≤ 2 ( ρ − 1 ) ∥ h ∥ ρ 2 t , t ∈ [ 0 , 1 ] . F'(t)-F'(0)\le2(\rho-1)\|h\|_\rho^2t,\qquad t\in[0,1]. F ′ ( t ) − F ′ ( 0 ) ≤ 2 ( ρ − 1 ) ∥ h ∥ ρ 2 t , t ∈ [ 0 , 1 ] .
Integrating once more yields
ψ ( x + h ) − ψ ( x ) = ∫ 0 1 F ′ ( t ) d t ≤ ⟨ ∇ ψ ( x ) , h ⟩ + ( ρ − 1 ) ∥ h ∥ ρ 2 . \psi(x+h)-\psi(x)=\int_0^1F'(t)\,dt
\le\langle\nabla\psi(x),h\rangle+(\rho-1)\|h\|_\rho^2. ψ ( x + h ) − ψ ( x ) = ∫ 0 1 F ′ ( t ) d t ≤ ⟨ ∇ ψ ( x ) , h ⟩ + ( ρ − 1 ) ∥ h ∥ ρ 2 .
Since ϕ + = ψ / 2 \phi_+=\psi/2 ϕ + = ψ /2 , this is equivalent to
ϕ + ( x + h ) ≤ ϕ + ( x ) + ⟨ ∇ ϕ + ( x ) , h ⟩ + ρ − 1 2 ∥ h ∥ + 2 , x , h ∈ V . \phi_+(x+h)\le\phi_+(x)+\langle\nabla\phi_+(x),h\rangle+\frac{\rho-1}{2}\|h\|_+^2,\qquad x,h\in V. ϕ + ( x + h ) ≤ ϕ + ( x ) + ⟨ ∇ ϕ + ( x ) , h ⟩ + 2 ρ − 1 ∥ h ∥ + 2 , x , h ∈ V .
Therefore, the auxiliary norm ∥ ⋅ ∥ + = ∥ ⋅ ∥ ρ \|\cdot\|_+=\|\cdot\|_\rho ∥ ⋅ ∥ + = ∥ ⋅ ∥ ρ is κ + \kappa_+ κ + -smooth with
κ + = ρ − 1. \kappa_+=\rho-1. κ + = ρ − 1.
Step 2: From Smoothness to Regularity. To establish the regularity of ( V , ∥ ⋅ ∥ ) (V,\|\cdot\|) ( V , ∥ ⋅ ∥ ) for p ∈ [ 2 , ∞ ] p\in[2,\infty] p ∈ [ 2 , ∞ ] , we use ∥ ⋅ ∥ + = ∥ ⋅ ∥ ρ \|\cdot\|_+=\|\cdot\|_\rho ∥ ⋅ ∥ + = ∥ ⋅ ∥ ρ with ρ ∈ [ 2 , p ] \rho\in[2,p] ρ ∈ [ 2 , p ] and ρ < ∞ \rho<\infty ρ < ∞ as the auxiliary smooth norm. The standard equivalence inequality for ℓ p \ell_p ℓ p -norms states that for any x ∈ R d x\in\mathbb{R}^d x ∈ R d and ρ ≤ p \rho\le p ρ ≤ p ,
∥ x ∥ = ∥ x ∥ p ≤ ∥ x ∥ ρ = ∥ x ∥ + ≤ d 1 ρ − 1 p ∥ x ∥ p = d 1 ρ − 1 p ∥ x ∥ . \|x\|=\|x\|_p\le\|x\|_\rho=\|x\|_+\le d^{\frac{1}{\rho}-\frac{1}{p}}\|x\|_p=d^{\frac{1}{\rho}-\frac{1}{p}}\|x\|. ∥ x ∥ = ∥ x ∥ p ≤ ∥ x ∥ ρ = ∥ x ∥ + ≤ d ρ 1 − p 1 ∥ x ∥ p = d ρ 1 − p 1 ∥ x ∥.
Squaring yields
∥ x ∥ 2 ≤ ∥ x ∥ + 2 ≤ d 2 ρ − 2 p ∥ x ∥ 2 , \|x\|^2\le\|x\|_+^2\le d^{\frac{2}{\rho}-\frac{2}{p}}\|x\|^2, ∥ x ∥ 2 ≤ ∥ x ∥ + 2 ≤ d ρ 2 − p 2 ∥ x ∥ 2 ,
which is exactly the compatibility chain required by Definition 1.
From Step 1, the auxiliary norm ∥ ⋅ ∥ + \|\cdot\|_{+} ∥ ⋅ ∥ + is κ + \kappa_{+} κ + -smooth with
κ + = ρ − 1. \kappa_{+}=\rho-1. κ + = ρ − 1.
Therefore, Definition 1 gives
κ = κ + d 2 ρ − 2 p = ( ρ − 1 ) d 2 ρ − 2 p . \kappa=\kappa_{+}d^{\frac{2}{\rho}-\frac{2}{p}}=(\rho-1)d^{\frac{2}{\rho}-\frac{2}{p}}. κ = κ + d ρ 2 − p 2 = ( ρ − 1 ) d ρ 2 − p 2 .
The tightest regularity constant supplied by this construction is found by minimizing this expression over all ρ ∈ [ 2 , p ] \rho\in[2,p] ρ ∈ [ 2 , p ] with ρ < ∞ \rho<\infty ρ < ∞ . This leads to the formula in the example statement.
To find the tightest bound on the regularity constant, we must solve
κ V : = κ p ( d ) = min 2 ≤ ρ ≤ p ρ < ∞ κ ( ρ ) , κ ( ρ ) : = ( ρ − 1 ) d 2 ρ − 2 p . (98) \kappa_{V}:=\kappa_{p}(d)=\min_{\substack{2\leq\rho\leq p\\ \rho<\infty}}\kappa(\rho),\qquad\kappa(\rho):=(\rho-1)d^{\frac{2}{\rho}-\frac{2}{p}}. \tag*{(98)} κ V := κ p ( d ) = 2 ≤ ρ ≤ p ρ < ∞ min κ ( ρ ) , κ ( ρ ) := ( ρ − 1 ) d ρ 2 − p 2 . (98)
The minimum is attained: for p < ∞ p<\infty p < ∞ , the objective is continuous on [ 2 , p ] [2,p] [ 2 , p ] ; for p = ∞ p=\infty p = ∞ , it is continuous on [ 2 , ∞ ) [2,\infty) [ 2 , ∞ ) and tends to ∞ \infty ∞ as ρ → ∞ \rho\to\infty ρ → ∞ . The term d − 2 / p d^{-2/p} d − 2/ p is a constant with respect to ρ \rho ρ , so we can focus on minimizing the function g ( ρ ) : = ( ρ − 1 ) d 2 / ρ g(\rho):=(\rho-1)d^{2/\rho} g ( ρ ) := ( ρ − 1 ) d 2/ ρ .
Step 3: A ρ \rho ρ -Independent Bound on the Regularity Constant. We compute the first and second derivatives of g ( ρ ) g(\rho) g ( ρ ) with respect to ρ \rho ρ .
g ′ ( ρ ) = d 2 / ρ ( 1 − 2 ( ρ − 1 ) ln d ρ 2 ) . g'(\rho)=d^{2/\rho}\left(1-\frac{2(\rho-1)\ln d}{\rho^{2}}\right). g ′ ( ρ ) = d 2/ ρ ( 1 − ρ 2 2 ( ρ − 1 ) ln d ) .
g ′ ′ ( ρ ) = d 2 / ρ 4 ln d ρ 4 ( ( ρ − 1 ) ln d − ρ ) . g''(\rho)=d^{2/\rho}\frac{4\ln d}{\rho^{4}}\left((\rho-1)\ln d-\rho\right). g ′′ ( ρ ) = d 2/ ρ ρ 4 4 ln d ( ( ρ − 1 ) ln d − ρ ) .
g ′ ′ ( ρ ) g''(\rho) g ′′ ( ρ ) is non-negative if and only if ( ln d ) ( ρ − 1 ) − ρ ≥ 0 (\ln d)(\rho-1)-\rho\geq0 ( ln d ) ( ρ − 1 ) − ρ ≥ 0 , which is equivalent to the condition ρ ≥ ln d ln d − 1 \rho\geq\frac{\ln d}{\ln d-1} ρ ≥ l n d − 1 l n d . We show this holds for any ρ ≥ 2 \rho\geq2 ρ ≥ 2 when d ≥ 8 d\geq8 d ≥ 8 .
Since we assume d ≥ 8 d\geq8 d ≥ 8 , it follows that d > e 2 ≈ 7.39 d>e^{2}\approx7.39 d > e 2 ≈ 7.39 . This implies ln d > 2 \ln d>2 ln d > 2 . From ln d > 2 \ln d>2 ln d > 2 , we have 2 ( ln d − 1 ) = 2 ln d − 2 > ln d 2(\ln d-1)=2\ln d-2>\ln d 2 ( ln d − 1 ) = 2 ln d − 2 > ln d , which rearranges to:
2 > ln d ln d − 1 . 2>\frac{\ln d}{\ln d-1}. 2 > ln d − 1 ln d .
At the same time, d ≥ 8 > e d\geq8>e d ≥ 8 > e implies ln d > 1 \ln d>1 ln d > 1 , so ln d − 1 > 0 \ln d-1>0 ln d − 1 > 0 , which ensures that ln d ln d − 1 > 1 \frac{\ln d}{\ln d-1}>1 l n d − 1 l n d > 1 . Combining these gives the sandwich inequality:
2 > ln d ln d − 1 > 1. 2>\frac{\ln d}{\ln d-1}>1. 2 > ln d − 1 ln d > 1.
Therefore, for any ρ ≥ 2 \rho\geq2 ρ ≥ 2 , we have:
ρ ≥ 2 > ln d ln d − 1 . \rho\geq2>\frac{\ln d}{\ln d-1}. ρ ≥ 2 > ln d − 1 ln d .
This confirms that g ′ ′ ( ρ ) ≥ 0 g''(\rho)\geq0 g ′′ ( ρ ) ≥ 0 , and thus g ( ρ ) g(\rho) g ( ρ ) is convex for all ρ ≥ 2 \rho\geq2 ρ ≥ 2 under the mild condition d ≥ 8 d\geq8 d ≥ 8 .
Finding the Optimal ρ ∗ \rho^{*} ρ ∗ . Since we have established that the function g ( ρ ) g(\rho) g ( ρ ) is convex on our interval of interest for d ≥ 8 d\geq8 d ≥ 8 , we can find its global minimizer by setting the first derivative to zero:
g ′ ( ρ ) = d 2 / ρ ( 1 − 2 ( ln d ) ( ρ − 1 ) ρ 2 ) = 0. g'(\rho)=d^{2/\rho}\left(1-\frac{2(\ln d)(\rho-1)}{\rho^{2}}\right)=0. g ′ ( ρ ) = d 2/ ρ ( 1 − ρ 2 2 ( ln d ) ( ρ − 1 ) ) = 0.
This requires the term in the parenthesis to be zero, which leads to a quadratic equation for ρ \rho ρ :
ρ 2 = 2 ( ln d ) ( ρ − 1 ) ⟹ ρ 2 − ( 2 ln d ) ρ + 2 ln d = 0. \rho^{2}=2(\ln d)(\rho-1)\Longrightarrow\rho^{2}-(2\ln d)\rho+2\ln d=0. ρ 2 = 2 ( ln d ) ( ρ − 1 ) ⟹ ρ 2 − ( 2 ln d ) ρ + 2 ln d = 0.
The solutions to this quadratic equation are:
ρ ∗ = ln d ± ( ln d ) 2 − 2 ln d . \rho^* = \ln d \pm\sqrt{(\ln d)^2 - 2\ln d}. ρ ∗ = ln d ± ( ln d ) 2 − 2 ln d .
For the solutions to be real, the discriminant must be non-negative, which requires ( ln d ) 2 − 2 ln d ≥ 0 (\ln d)^2 - 2\ln d \ge0 ( ln d ) 2 − 2 ln d ≥ 0 , or ln d ≥ 2 \ln d \ge2 ln d ≥ 2 . This condition, d ≥ e 2 ≈ 7.4 d \ge e^2 \approx7.4 d ≥ e 2 ≈ 7.4 , is consistent with our assumption that d ≥ 8 d \ge8 d ≥ 8 .
The quadratic equation yields two roots.
To determine which of these corresponds to a local minimum, we apply the second derivative test. Note that sign ( g ′ ′ ( ρ ) ) = sign ( ( ln d ) ( ρ − 1 ) − ρ ) \operatorname{sign}(g''(\rho)) = \operatorname{sign}((\ln d)(\rho- 1) - \rho) sign ( g ′′ ( ρ )) = sign (( ln d ) ( ρ − 1 ) − ρ ) .
At either critical point ρ ∗ ∈ { ρ − , ρ + } \rho^* \in\{\rho_-, \rho_+\} ρ ∗ ∈ { ρ − , ρ + } , the first-order condition gives ρ ∗ 2 = 2 ( ln d ) ( ρ ∗ − 1 ) \rho^{*2} = 2(\ln d)(\rho^* - 1) ρ ∗ 2 = 2 ( ln d ) ( ρ ∗ − 1 ) . We can therefore simplify the expression by substituting ( ln d ) ( ρ ∗ − 1 ) = ρ ∗ 2 2 (\ln d)(\rho^* - 1) = \frac{\rho^{*2}}{2} ( ln d ) ( ρ ∗ − 1 ) = 2 ρ ∗ 2 . Plugging this into our sign-determining term:
( ln d ) ( ρ ∗ − 1 ) − ρ ∗ = ρ ∗ 2 2 − ρ ∗ = ρ ∗ ( ρ ∗ − 2 ) 2 . (\ln d)(\rho^* - 1) - \rho^* = \frac{\rho^{*2}}{2} - \rho^* = \frac{\rho^*(\rho^* - 2)}{2}. ( ln d ) ( ρ ∗ − 1 ) − ρ ∗ = 2 ρ ∗ 2 − ρ ∗ = 2 ρ ∗ ( ρ ∗ − 2 ) .
Therefore, the sign of the second derivative at a critical point ρ ∗ \rho^* ρ ∗ is the same as the sign of ρ ∗ − 2 \rho^* - 2 ρ ∗ − 2 .
Let us now analyze the location of the two roots relative to the value 2. We are working under the condition ln d ≥ 2 \ln d \ge2 ln d ≥ 2 (since d ≥ 8 > e 2 d \ge8 > e^2 d ≥ 8 > e 2 ).
ln d − 2 > ( ln d ) 2 − 2 ln d \ln d - 2 > \sqrt{(\ln d)^2 - 2\ln d} ln d − 2 > ( ln d ) 2 − 2 ln d
( ln d − 2 ) 2 > ( ln d ) 2 − 2 ln d (squaring is valid as ln d > 2 ) (\ln d - 2)^2 > (\ln d)^2 - 2\ln d \qquad\text{(squaring is valid as }\ln d > 2\text{)} ( ln d − 2 ) 2 > ( ln d ) 2 − 2 ln d (squaring is valid as ln d > 2 )
( ln d ) 2 − 4 ln d + 4 > ( ln d ) 2 − 2 ln d (\ln d)^2 - 4\ln d + 4 > (\ln d)^2 - 2\ln d ( ln d ) 2 − 4 ln d + 4 > ( ln d ) 2 − 2 ln d
This is a contradiction to our premise that ln d ≥ 2 \ln d \ge2 ln d ≥ 2 . Therefore, the inequality must be reversed, which means ρ − ≤ 2 \rho_- \le2 ρ − ≤ 2 .
ρ + = ln d + ( ln d ) 2 − 2 ln d ≥ ln d ≥ 2. \rho_+ = \ln d + \sqrt{(\ln d)^2 - 2\ln d} \ge\ln d \ge2. ρ + = ln d + ( ln d ) 2 − 2 ln d ≥ ln d ≥ 2.
Now we apply our simplified second derivative test:
At ρ − \rho_- ρ − , since ρ − ≤ 2 \rho_- \le2 ρ − ≤ 2 , we have ρ − − 2 ≤ 0 \rho_- - 2 \le0 ρ − − 2 ≤ 0 . This implies g ′ ′ ( ρ − ) ≤ 0 g''(\rho_-) \le0 g ′′ ( ρ − ) ≤ 0 , so ρ − \rho_- ρ − corresponds to a local maximum .
At ρ + \rho_+ ρ + , since ρ + ≥ 2 \rho_+ \ge2 ρ + ≥ 2 , we have ρ + − 2 ≥ 0 \rho_+ - 2 \ge0 ρ + − 2 ≥ 0 . This implies g ′ ′ ( ρ + ) ≥ 0 g''(\rho_+) \ge0 g ′′ ( ρ + ) ≥ 0 , so ρ + \rho_+ ρ + corresponds to a local minimum .
Since g g g is convex on [ 2 , ∞ ) [2,\infty) [ 2 , ∞ ) , its minimizer on this interval is the larger root ρ + \rho_+ ρ + .
Analyzing the Location of the Minimum. The optimal regularity bound within this construction depends on where this unconstrained minimizer ρ ∗ \rho^* ρ ∗ falls relative to the domain { ρ : 2 ≤ ρ ≤ p , ρ < ∞ } \{\rho: 2 \le\rho\le p,\ \rho< \infty\} { ρ : 2 ≤ ρ ≤ p , ρ < ∞ } .
**If ρ ∗ > p : ∗ ∗ T h i s o c c u r s w h e n \rho^* > p:** This occurs when ρ ∗ > p : ∗ ∗ T hi socc u r s w h e n p$ is small compared to ln d \ln d ln d . Since g ( ρ ) g(\rho) g ( ρ ) is convex, its minimum over [ 2 , p ] [2,p] [ 2 , p ] must occur at the boundary. As g ′ ( ρ ) < 0 g'(\rho) < 0 g ′ ( ρ ) < 0 for ρ < ρ ∗ \rho< \rho^* ρ < ρ ∗ , the function is decreasing on [ 2 , p ] [2,p] [ 2 , p ] , so the minimum is achieved at the right endpoint, ρ = p \rho= p ρ = p . In this case, the regularity constant is κ ( p ) = ( p − 1 ) d 2 / p − 2 / p = p − 1 \kappa(p) = (p - 1)d^{2/p - 2/p} = p - 1 κ ( p ) = ( p − 1 ) d 2/ p − 2/ p = p − 1 .
If 2 ≤ ρ ∗ ≤ p 2 \le\rho^* \le p 2 ≤ ρ ∗ ≤ p : The minimum is achieved at the unconstrained minimizer ρ ∗ \rho^* ρ ∗ . We now find an upper bound for the value of κ ( ρ ∗ ) \kappa(\rho^*) κ ( ρ ∗ ) .
κ ( ρ ∗ ) = ( ρ ∗ − 1 ) d 2 ρ ∗ − 2 p . \kappa(\rho^*) = (\rho^* - 1)d^{\frac{2}{\rho^*} - \frac{2}{p}}. κ ( ρ ∗ ) = ( ρ ∗ − 1 ) d ρ ∗ 2 − p 2 .
Since ρ ∗ ≤ p \rho^* \le p ρ ∗ ≤ p and d ≥ 1 d \ge1 d ≥ 1 , we have d − 2 / ρ ∗ ≤ d − 2 / p ≤ 1 d^{-2/\rho^*} \le d^{-2/p} \le1 d − 2/ ρ ∗ ≤ d − 2/ p ≤ 1 . Using the upper bound d − 2 / p ≤ 1 d^{-2/p} \le1 d − 2/ p ≤ 1 gives
κ ( ρ ∗ ) ≤ ( ρ ∗ − 1 ) d 2 ρ ∗ . \kappa(\rho^*) \le(\rho^*-1)d^{\frac{2}{\rho^*}}. κ ( ρ ∗ ) ≤ ( ρ ∗ − 1 ) d ρ ∗ 2 .
From the first-order optimality condition, we have the identity 2 ( ln d ) ( ρ ∗ − 1 ) = ( ρ ∗ ) 2 2(\ln d)(\rho^*-1)=(\rho^*)^2 2 ( ln d ) ( ρ ∗ − 1 ) = ( ρ ∗ ) 2 . This allows us to substitute ( ρ ∗ − 1 ) = ( ρ ∗ ) 2 2 ln d (\rho^*-1)=\frac{(\rho^*)^2}{2\ln d} ( ρ ∗ − 1 ) = 2 l n d ( ρ ∗ ) 2 .
κ ( ρ ∗ ) ≤ ( ρ ∗ ) 2 2 ln d ⋅ d 2 ρ ∗ = ( ρ ∗ ) 2 2 ln d exp ( 2 ln d ρ ∗ ) . \kappa(\rho^*) \le\frac{(\rho^*)^2}{2\ln d}\cdot d^{\frac{2}{\rho^*}}=\frac{(\rho^*)^2}{2\ln d}\exp\left(\frac{2\ln d}{\rho^*}\right). κ ( ρ ∗ ) ≤ 2 ln d ( ρ ∗ ) 2 ⋅ d ρ ∗ 2 = 2 ln d ( ρ ∗ ) 2 exp ( ρ ∗ 2 ln d ) .
We know that ln d ≤ ρ ∗ = ln d + ( ln d ) 2 − 2 ln d ≤ 2 ln d \ln d \le\rho^*=\ln d+\sqrt{(\ln d)^2-2\ln d}\le2\ln d ln d ≤ ρ ∗ = ln d + ( ln d ) 2 − 2 ln d ≤ 2 ln d . Substituting these bounds:
κ ( ρ ∗ ) ≤ ( 2 ln d ) 2 2 ln d exp ( 2 ln d ln d ) = ( 2 ln d ) ⋅ e 2 . \kappa(\rho^*) \le\frac{(2\ln d)^2}{2\ln d}\exp\left(\frac{2\ln d}{\ln d}\right)=(2\ln d)\cdot e^2. κ ( ρ ∗ ) ≤ 2 ln d ( 2 ln d ) 2 exp ( ln d 2 ln d ) = ( 2 ln d ) ⋅ e 2 .
For p < ∞ p<\infty p < ∞ , combining the endpoint bound p − 1 p-1 p − 1 with the bound 2 e 2 ln d 2e^2\ln d 2 e 2 ln d gives the estimate below. For p = ∞ p=\infty p = ∞ , the finite minimizer ρ ∗ \rho^* ρ ∗ gives the bound 2 e 2 ln d 2e^2\ln d 2 e 2 ln d directly:
κ p ( d ) = min 2 ≤ ρ ≤ p κ ( ρ ) ≤ min { p − 1 , 2 e 2 ( ln d ) } . \kappa_p(d)=\min_{2\le\rho\le p}\kappa(\rho)\le\min\{p-1,2e^2(\ln d)\}. κ p ( d ) = 2 ≤ ρ ≤ p min κ ( ρ ) ≤ min { p − 1 , 2 e 2 ( ln d )} .
We now proceed with Case 2: 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 .
We prove that ( V , ∥ ⋅ ∥ ) (V,\|\cdot\|) ( V , ∥ ⋅ ∥ ) is κ \kappa κ -regular with
κ : = κ V = κ p ( d ) = d 2 p − 1 . (99) \kappa:=\kappa_V=\kappa_p(d)=d^{\frac{2}{p}-1}. \tag*{(99)} κ := κ V = κ p ( d ) = d p 2 − 1 . (99)
In particular, the regularity constant is polynomial in d d d for p < 2 p<2 p < 2 .
Fix p ∈ [ 1 , 2 ) p\in[1,2) p ∈ [ 1 , 2 ) and let d ≥ 1 d\ge1 d ≥ 1 .
Step 1: Construct a smooth auxiliary norm. Define the auxiliary norm on V V V by
∥ x ∥ + : = ∥ x ∥ V , + : = d 1 p − 1 2 ∥ x ∥ 2 , x ∈ R d . (100) \|x\|_+:=\|x\|_{V,+}:=d^{\frac{1}{p}-\frac{1}{2}}\|x\|_2,\qquad x\in\mathbb{R}^d. \tag*{(100)} ∥ x ∥ + := ∥ x ∥ V , + := d p 1 − 2 1 ∥ x ∥ 2 , x ∈ R d . (100)
Let a : = d 1 p − 1 2 > 0 a:=d^{\frac{1}{p}-\frac{1}{2}}>0 a := d p 1 − 2 1 > 0 and define
φ + ( x ) : = 1 2 ∥ x ∥ + 2 = a 2 2 ∥ x ∥ 2 2 . \varphi_+(x):=\frac{1}{2}\|x\|_+^2=\frac{a^2}{2}\|x\|_2^2. φ + ( x ) := 2 1 ∥ x ∥ + 2 = 2 a 2 ∥ x ∥ 2 2 .
This is a quadratic function and is continuously differentiable on R d \mathbb{R}^d R d . Using the standard Euclidean inner product to identify V V V with V ∗ V^* V ∗ , we obtain
∇ φ + ( x ) = a 2 x . \nabla\varphi_+(x)=a^2x. ∇ φ + ( x ) = a 2 x .
For any x , y ∈ R d x,y\in\mathbb{R}^d x , y ∈ R d , expanding the square gives
φ + ( x + y ) = a 2 2 ∥ x + y ∥ 2 2 = a 2 2 ( ∥ x ∥ 2 2 + 2 ⟨ x , y ⟩ + ∥ y ∥ 2 2 ) = φ + ( x ) + ⟨ a 2 x , y ⟩ + a 2 2 ∥ y ∥ 2 2 = φ + ( x ) + ⟨ ∇ φ + ( x ) , y ⟩ + 1 2 ∥ y ∥ + 2 . \begin{aligned}
\varphi_+(x+y)&=\frac{a^2}{2}\|x+y\|_2^2\\
&=\frac{a^2}{2}\left(\|x\|_2^2+2\langle x,y\rangle+\|y\|_2^2\right)\\
&=\varphi_+(x)+\langle a^2x,y\rangle+\frac{a^2}{2}\|y\|_2^2\\
&=\varphi_+(x)+\langle\nabla\varphi_+(x),y\rangle+\frac{1}{2}\|y\|_+^2.
\end{aligned} φ + ( x + y ) = 2 a 2 ∥ x + y ∥ 2 2 = 2 a 2 ( ∥ x ∥ 2 2 + 2 ⟨ x , y ⟩ + ∥ y ∥ 2 2 ) = φ + ( x ) + ⟨ a 2 x , y ⟩ + 2 a 2 ∥ y ∥ 2 2 = φ + ( x ) + ⟨ ∇ φ + ( x ) , y ⟩ + 2 1 ∥ y ∥ + 2 .
Thus the inequality in Definition 1(i) holds with equality for κ + = 1 \kappa_+=1 κ + = 1 , so ( V , ∥ ⋅ ∥ + ) (V,\|\cdot\|_+) ( V , ∥ ⋅ ∥ + ) is 1-smooth.
Step 2: Verify the compatibility inequalities. We claim that for every x ∈ R d x \in\mathbb{R}^{d} x ∈ R d ,
∥ x ∥ 2 ≤ ∥ x ∥ p ≤ d 1 p − 1 2 ∥ x ∥ 2 . (101) \lVert x\rVert_{2} \le\lVert x\rVert_{p} \le d^{\frac{1}{p}-\frac{1}{2}}\lVert x\rVert_{2}. \tag*{(101)} ∥ x ∥ 2 ≤ ∥ x ∥ p ≤ d p 1 − 2 1 ∥ x ∥ 2 . (101)
Proof of the left inequality in (101) . If x = 0 x=0 x = 0 it is trivial. Otherwise, let a i : = ∣ x i ∣ / ∥ x ∥ p a_i := |x_i|/\lVert x\rVert_p a i := ∣ x i ∣/ ∥ x ∥ p . Then ∑ i = 1 d a i p = 1 \sum_{i=1}^{d}a_i^p=1 ∑ i = 1 d a i p = 1 and each a i ∈ [ 0 , 1 ] a_i \in[0,1] a i ∈ [ 0 , 1 ] . Since p < 2 p<2 p < 2 and a i ∈ [ 0 , 1 ] a_i \in[0,1] a i ∈ [ 0 , 1 ] , we have a i 2 ≤ a i p a_i^2 \le a_i^p a i 2 ≤ a i p . Hence
∥ x ∥ 2 2 = ∑ i = 1 d ∣ x i ∣ 2 = ∥ x ∥ p 2 ∑ i = 1 d a i 2 ≤ ∥ x ∥ p 2 ∑ i = 1 d a i p = ∥ x ∥ p 2 , \begin{aligned}
\lVert x\rVert_2^2 = \sum_{i=1}^{d}|x_i|^2 \\
&= \lVert x\rVert_p^2 \sum_{i=1}^{d}a_i^2 \\
&\le\lVert x\rVert_p^2 \sum_{i=1}^{d}a_i^p \\
&= \lVert x\rVert_p^2,
\end{aligned} ∥ x ∥ 2 2 = i = 1 ∑ d ∣ x i ∣ 2 = ∥ x ∥ p 2 i = 1 ∑ d a i 2 ≤ ∥ x ∥ p 2 i = 1 ∑ d a i p = ∥ x ∥ p 2 ,
which yields ∥ x ∥ 2 ≤ ∥ x ∥ p \lVert x\rVert_2 \le\lVert x\rVert_p ∥ x ∥ 2 ≤ ∥ x ∥ p .
Proof of the right inequality in (101) . Let r : = 2 / p > 1 r := 2/p > 1 r := 2/ p > 1 and r ′ : = 2 / ( 2 − p ) r' := 2/(2-p) r ′ := 2/ ( 2 − p ) be its conjugate exponent. Using Hölder’s inequality,
∑ i = 1 d ∣ x i ∣ p = ∑ i = 1 d ( ∣ x i ∣ 2 ) p / 2 ⋅ 1 ≤ ( ∑ i = 1 d ∣ x i ∣ 2 ) p / 2 ( ∑ i = 1 d 1 r ′ ) 1 / r ′ = ∥ x ∥ 2 p d 1 − p 2 . \begin{aligned}
\sum_{i=1}^{d}|x_i|^p = \sum_{i=1}^{d}(|x_i|^2)^{p/2}\cdot1 \\
&\le\left(\sum_{i=1}^{d}|x_i|^2\right)^{p/2}\left(\sum_{i=1}^{d}1^{r'}\right)^{1/r'} \\
&= \lVert x\rVert_2^p d^{1-\frac{p}{2}}.
\end{aligned} i = 1 ∑ d ∣ x i ∣ p = i = 1 ∑ d ( ∣ x i ∣ 2 ) p /2 ⋅ 1 ≤ ( i = 1 ∑ d ∣ x i ∣ 2 ) p /2 ( i = 1 ∑ d 1 r ′ ) 1/ r ′ = ∥ x ∥ 2 p d 1 − 2 p .
Taking p p p -th roots gives ∥ x ∥ p ≤ d 1 p − 1 2 ∥ x ∥ 2 \lVert x\rVert_p \le d^{\frac{1}{p}-\frac{1}{2}}\lVert x\rVert_2 ∥ x ∥ p ≤ d p 1 − 2 1 ∥ x ∥ 2 .
Combining the two inequalities proves (101) . Now, by the definition (100) and (101) ,
∥ x ∥ p ≤ d 1 p − 1 2 ∥ x ∥ 2 = ∥ x ∥ + , \lVert x\rVert_p \le d^{\frac{1}{p}-\frac{1}{2}}\lVert x\rVert_2 = \lVert x\rVert_+, ∥ x ∥ p ≤ d p 1 − 2 1 ∥ x ∥ 2 = ∥ x ∥ + ,
so ∥ x ∥ p 2 ≤ ∥ x ∥ + 2 \lVert x\rVert_p^2 \le\lVert x\rVert_+^2 ∥ x ∥ p 2 ≤ ∥ x ∥ + 2 . Also, using ∥ x ∥ 2 ≤ ∥ x ∥ p \lVert x\rVert_2 \le\lVert x\rVert_p ∥ x ∥ 2 ≤ ∥ x ∥ p (the left inequality),
∥ x ∥ + 2 = d 2 p − 1 ∥ x ∥ 2 2 ≤ d 2 p − 1 ∥ x ∥ p 2 . \lVert x\rVert_+^2 = d^{\frac{2}{p}-1}\lVert x\rVert_2^2 \le d^{\frac{2}{p}-1}\lVert x\rVert_p^2. ∥ x ∥ + 2 = d p 2 − 1 ∥ x ∥ 2 2 ≤ d p 2 − 1 ∥ x ∥ p 2 .
Therefore, for all x ∈ R d x \in\mathbb{R}^{d} x ∈ R d ,
∥ x ∥ p 2 ≤ ∥ x ∥ + 2 ≤ d 2 p − 1 ∥ x ∥ p 2 . (102) \lVert x\rVert_p^2 \le\lVert x\rVert_+^2 \le d^{\frac{2}{p}-1}\lVert x\rVert_p^2. \tag*{(102)} ∥ x ∥ p 2 ≤ ∥ x ∥ + 2 ≤ d p 2 − 1 ∥ x ∥ p 2 . (102)
Step 3: Conclude κ \kappa κ -regularity. We have shown that the auxiliary norm ( V , ∥ ⋅ ∥ + ) (V,\lVert\cdot\rVert_+) ( V , ∥ ⋅ ∥ + ) is 1 1 1 -smooth, i.e., κ + = 1 \kappa_+ = 1 κ + = 1 , and that (102) gives the compatibility chain
∥ x ∥ p 2 ≤ ∥ x ∥ + 2 ≤ d 2 p − 1 ∥ x ∥ p 2 . \lVert x\rVert_p^2 \le\lVert x\rVert_+^2 \le d^{\frac{2}{p}-1}\lVert x\rVert_p^2. ∥ x ∥ p 2 ≤ ∥ x ∥ + 2 ≤ d p 2 − 1 ∥ x ∥ p 2 .
Therefore, by Definition 1, the space ( V , ∥ ⋅ ∥ ) (V,\lVert\cdot\rVert) ( V , ∥ ⋅ ∥) is κ \kappa κ -regular with
κ = d 2 p − 1 . \kappa= d^{\frac{2}{p}-1}. κ = d p 2 − 1 .
Equivalently,
κ V = κ p ( d ) = d 2 p − 1 . \kappa_V = \kappa_p(d) = d^{\frac{2}{p}-1}. κ V = κ p ( d ) = d p 2 − 1 .
This establishes the upper bound in (99) .
Remark 5. (Optimality of the Euclidean auxiliary norm for p < 2 p < 2 p < 2 within the scaled ℓ p \ell_p ℓ p family.) Fix d ∈ N d \in\mathbb{N} d ∈ N and p ∈ [ 1 , 2 ) p \in[1,2) p ∈ [ 1 , 2 ) . Let V : = R d V := \mathbb{R}^d V := R d and equip it with the norm ∥ ⋅ ∥ V : = ∥ ⋅ ∥ p \lVert\cdot\rVert_V := \lVert\cdot\rVert_p ∥ ⋅ ∥ V := ∥ ⋅ ∥ p . For every ρ ∈ [ 2 , ∞ ) \rho\in[2,\infty) ρ ∈ [ 2 , ∞ ) , consider the scaled auxiliary norm
∥ x ∥ V , + , ρ : = d 1 p − 1 ρ ∥ x ∥ ρ , x ∈ R d . (103) \lVert x\rVert_{V,+,\rho} := d^{\frac{1}{p}-\frac{1}{\rho}}\lVert x\rVert_\rho,\qquad x\in\mathbb{R}^d. \tag*{(103)} ∥ x ∥ V , + , ρ := d p 1 − ρ 1 ∥ x ∥ ρ , x ∈ R d . (103) Then ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) is κ V ( ρ ) \kappa_V(\rho) κ V ( ρ ) -regular with respect to this auxiliary norm, where
κ V ( ρ ) : = ( ρ − 1 ) d 2 p − 2 ρ . (104) \kappa_V(\rho):=(\rho-1)d^{\frac{2}{p}-\frac{2}{\rho}}. \tag*{(104)} κ V ( ρ ) := ( ρ − 1 ) d p 2 − ρ 2 . (104) Moreover, the function ρ ↦ κ V ( ρ ) \rho\mapsto\kappa_V(\rho) ρ ↦ κ V ( ρ ) is minimized over ρ ∈ [ 2 , ∞ ) \rho\in[2,\infty) ρ ∈ [ 2 , ∞ ) at ρ = 2 \rho=2 ρ = 2 . Hence, within this family of scaled ℓ ρ \ell_\rho ℓ ρ auxiliary norms, the optimal choice is the Euclidean norm
∥ x ∥ V , + : = d 1 p − 1 2 ∥ x ∥ 2 . \lVert x\rVert_{V,+}:=d^{\frac{1}{p}-\frac{1}{2}}\lVert x\rVert_2. ∥ x ∥ V , + := d p 1 − 2 1 ∥ x ∥ 2 .
Proof. To simplify notation throughout this proof, we write
∥ x ∥ : = ∥ x ∥ V = ∥ x ∥ p , ∥ x ∥ + , ρ : = ∥ x ∥ V , + , ρ . \lVert x\rVert:=\lVert x\rVert_V=\lVert x\rVert_p,\qquad\lVert x\rVert_{+,\rho}:=\lVert x\rVert_{V,+,\rho}. ∥ x ∥ := ∥ x ∥ V = ∥ x ∥ p , ∥ x ∥ + , ρ := ∥ x ∥ V , + , ρ .
Fix ρ ∈ [ 2 , ∞ ) \rho\in[2,\infty) ρ ∈ [ 2 , ∞ ) . Since p < ρ p<\rho p < ρ , the standard norm equivalence inequalities for ℓ p \ell_p ℓ p spaces give
∥ x ∥ ρ ≤ ∥ x ∥ ≤ d 1 p − 1 ρ ∥ x ∥ ρ , ∀ x ∈ R d . \lVert x\rVert_\rho\leq\lVert x\rVert\leq d^{\frac{1}{p}-\frac{1}{\rho}}\lVert x\rVert_\rho,\qquad\forall x\in\mathbb{R}^d. ∥ x ∥ ρ ≤ ∥ x ∥ ≤ d p 1 − ρ 1 ∥ x ∥ ρ , ∀ x ∈ R d .
Substituting the definition (103) , we obtain
∥ x ∥ ≤ ∥ x ∥ + , ρ ≤ d 1 p − 1 ρ ∥ x ∥ , ∀ x ∈ R d . (105) \lVert x\rVert\leq\lVert x\rVert_{+,\rho}\leq d^{\frac{1}{p}-\frac{1}{\rho}}\lVert x\rVert,\qquad\forall x\in\mathbb{R}^d. \tag*{(105)} ∥ x ∥ ≤ ∥ x ∥ + , ρ ≤ d p 1 − ρ 1 ∥ x ∥ , ∀ x ∈ R d . (105)
Squaring gives the compatibility chain
∥ x ∥ 2 ≤ ∥ x ∥ + , ρ 2 ≤ d 2 p − 2 ρ ∥ x ∥ 2 , ∀ x ∈ R d . \lVert x\rVert^2\leq\lVert x\rVert_{+,\rho}^2\leq d^{\frac{2}{p}-\frac{2}{\rho}}\lVert x\rVert^2,\qquad\forall x\in\mathbb{R}^d. ∥ x ∥ 2 ≤ ∥ x ∥ + , ρ 2 ≤ d p 2 − ρ 2 ∥ x ∥ 2 , ∀ x ∈ R d .
By Step 1 in the proof of Proposition 12, the norm ∥ ⋅ ∥ ρ \lVert\cdot\rVert_\rho ∥ ⋅ ∥ ρ is ( ρ − 1 ) (\rho-1) ( ρ − 1 ) -smooth for every ρ ∈ [ 2 , ∞ ) \rho\in[2,\infty) ρ ∈ [ 2 , ∞ ) . Since scaling a norm by a positive constant does not change its smoothness parameter, the scaled norm ∥ ⋅ ∥ + , ρ \lVert\cdot\rVert_{+,\rho} ∥ ⋅ ∥ + , ρ is also ( ρ − 1 ) (\rho-1) ( ρ − 1 ) -smooth. Hence ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) is κ V ( ρ ) \kappa_V(\rho) κ V ( ρ ) -regular with respect to the auxiliary norm ∥ ⋅ ∥ V , + , ρ \lVert\cdot\rVert_{V,+,\rho} ∥ ⋅ ∥ V , + , ρ , where
κ V ( ρ ) = ( ρ − 1 ) d 2 p − 2 ρ . \kappa_V(\rho)=(\rho-1)d^{\frac{2}{p}-\frac{2}{\rho}}. κ V ( ρ ) = ( ρ − 1 ) d p 2 − ρ 2 .
To identify the best choice within this family, compare an arbitrary ρ > 2 \rho>2 ρ > 2 against ρ = 2 \rho=2 ρ = 2 :
κ V ( ρ ) κ V ( 2 ) = ( ρ − 1 ) d 2 p − 2 ρ 1 ⋅ d 2 p − 1 = ( ρ − 1 ) d 1 − 2 ρ . \frac{\kappa_V(\rho)}{\kappa_V(2)}
=\frac{(\rho-1)d^{\frac{2}{p}-\frac{2}{\rho}}}{1\cdot d^{\frac{2}{p}-1}}
=(\rho-1)d^{1-\frac{2}{\rho}}. κ V ( 2 ) κ V ( ρ ) = 1 ⋅ d p 2 − 1 ( ρ − 1 ) d p 2 − ρ 2 = ( ρ − 1 ) d 1 − ρ 2 .
Since ρ > 2 \rho>2 ρ > 2 , we have ρ − 1 > 1 \rho-1>1 ρ − 1 > 1 and 1 − 2 ρ > 0 1-\frac{2}{\rho}>0 1 − ρ 2 > 0 . Also, for every d ≥ 1 d\geq1 d ≥ 1 , we have d 1 − 2 ρ ≥ 1 d^{1-\frac{2}{\rho}}\geq1 d 1 − ρ 2 ≥ 1 . Therefore,
κ V ( ρ ) κ V ( 2 ) > 1 , ∀ ρ > 2. \frac{\kappa_V(\rho)}{\kappa_V(2)}>1,\qquad\forall\rho>2. κ V ( 2 ) κ V ( ρ ) > 1 , ∀ ρ > 2.
Thus κ V ( ρ ) \kappa_V(\rho) κ V ( ρ ) is minimized at ρ = 2 \rho=2 ρ = 2 , which yields the Euclidean auxiliary norm
∥ x ∥ V , + : = d 1 p − 1 2 ∥ x ∥ 2 . \lVert x\rVert_{V,+}:=d^{\frac{1}{p}-\frac{1}{2}}\lVert x\rVert_2. ∥ x ∥ V , + := d p 1 − 2 1 ∥ x ∥ 2 .
We now complement the upper bounds from Proposition 12 with matching lower bounds on the admissible regularity parameter κ V \kappa_V κ V for ℓ p d \ell_p^d ℓ p d spaces. For 1 ≤ p < 2 1\leq p<2 1 ≤ p < 2 , the optimal parameter is d 2 / p − 1 d^{2/p-1} d 2/ p − 1 . For 2 ≤ p ≤ ∞ 2\leq p\leq\infty 2 ≤ p ≤ ∞ , its optimal order is min { p , ln ( e d ) } \min\{p,\ln(ed)\} min { p , ln ( e d )} , uniformly in d ≥ 1 d\geq1 d ≥ 1 ; see Remark 7. Consequently, the optimal parameter remains bounded as d d d varies with a fixed finite p ≥ 2 p\geq2 p ≥ 2 , and equals 1 1 1 at p = 2 p=2 p = 2 . The logarithmic branch of the minimum corresponds to p ≥ ln ( e d ) p\geq\ln(ed) p ≥ ln ( e d ) , including p = ∞ p=\infty p = ∞ , where logarithmic growth is unavoidable up to absolute constants.
Proposition 13. (Lower bounds for ℓ p d \ell_p^d ℓ p d spaces). Fix d ∈ N d\in\mathbb{N} d ∈ N and p ∈ [ 1 , ∞ ] p\in[1,\infty] p ∈ [ 1 , ∞ ] . Let V : = R d V:=\mathbb{R}^d V := R d and equip it with the norm ∥ ⋅ ∥ V : = ∥ ⋅ ∥ p \lVert\cdot\rVert_V:=\lVert\cdot\rVert_p ∥ ⋅ ∥ V := ∥ ⋅ ∥ p . If the space ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) is κ V \kappa_V κ V -regular, then the following hold:
If 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 , then
κ V ≥ d 2 p − 1 . (106) \kappa_V \ge d^{\frac{2}{p}-1}. \tag*{(106)} κ V ≥ d p 2 − 1 . (106)
If 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ and d ≥ 8 d \ge8 d ≥ 8 , then
κ V ≥ 1 8 min { p − 1 , ln d } . (107) \kappa_V \ge\frac{1}{8}\min\{p-1,\ln d\}. \tag*{(107)} κ V ≥ 8 1 min { p − 1 , ln d } . (107)
Proof. To simplify notation throughout this proof, we write
∥ x ∥ : = ∥ x ∥ V = ∥ x ∥ p , κ : = κ V . \lVert x\rVert:= \lVert x\rVert_V = \lVert x\rVert_p,\qquad\kappa:= \kappa_V. ∥ x ∥ := ∥ x ∥ V = ∥ x ∥ p , κ := κ V .
Case 1: 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 .
Let { ε i } i = 1 d \{\varepsilon_i\}_{i=1}^{d} { ε i } i = 1 d be i.i.d. Rademacher random variables, and let { e i } i = 1 d \{e_i\}_{i=1}^{d} { e i } i = 1 d be the standard basis of R d \mathbb{R}^d R d . Define
F i : = σ ( ε 1 , … , ε i ) , ξ i : = ε i e i , i = 1 , … , d . \mathcal{F}_i := \sigma(\varepsilon_1,\ldots,\varepsilon_i),\qquad\xi_i := \varepsilon_i e_i,\qquad i=1,\ldots,d. F i := σ ( ε 1 , … , ε i ) , ξ i := ε i e i , i = 1 , … , d .
Then { ξ i } i = 1 d \{\xi_i\}_{i=1}^{d} { ξ i } i = 1 d is an { F i } \{\mathcal{F}_i\} { F i } -martingale-difference sequence: it is F i \mathcal{F}_i F i -measurable and
E [ ξ i ∣ F i − 1 ] = e i E [ ε i ∣ F i − 1 ] = 0 a.s. \mathbb{E}[\xi_i\mid\mathcal{F}_{i-1}] = e_i\mathbb{E}[\varepsilon_i\mid\mathcal{F}_{i-1}] = 0\qquad\text{a.s.} E [ ξ i ∣ F i − 1 ] = e i E [ ε i ∣ F i − 1 ] = 0 a.s.
Moreover,
∥ ξ i ∥ = ∥ e i ∥ p = 1 , i = 1 , … , d . \lVert\xi_i\rVert= \lVert e_i\rVert_p = 1,\qquad i=1,\ldots,d. ∥ ξ i ∥ = ∥ e i ∥ p = 1 , i = 1 , … , d .
Let
S d : = ∑ i = 1 d ξ i = ( ε 1 , … , ε d ) . S_d := \sum_{i=1}^{d}\xi_i = (\varepsilon_1,\ldots,\varepsilon_d). S d := i = 1 ∑ d ξ i = ( ε 1 , … , ε d ) .
Since ∣ ε i ∣ = 1 |\varepsilon_i|=1 ∣ ε i ∣ = 1 almost surely, we have deterministically
∥ S d ∥ = ∥ S d ∥ p = ( ∑ i = 1 d ∣ ε i ∣ p ) 1 / p = d 1 / p . \lVert S_d\rVert= \lVert S_d\rVert_p = \left(\sum_{i=1}^{d}|\varepsilon_i|^p\right)^{1/p}=d^{1/p}. ∥ S d ∥ = ∥ S d ∥ p = ( i = 1 ∑ d ∣ ε i ∣ p ) 1/ p = d 1/ p .
Hence
E [ ∥ S d ∥ 2 ] = d 2 / p , ∑ i = 1 d E [ ∥ ξ i ∥ 2 ] = ∑ i = 1 d 1 = d . \mathbb{E}\left[\lVert S_d\rVert^2\right]=d^{2/p},\qquad\sum_{i=1}^{d}\mathbb{E}\left[\lVert\xi_i\rVert^2\right]=\sum_{i=1}^{d}1=d. E [ ∥ S d ∥ 2 ] = d 2/ p , i = 1 ∑ d E [ ∥ ξ i ∥ 2 ] = i = 1 ∑ d 1 = d .
Since ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) is κ \kappa κ -regular and { ξ i } \{\xi_i\} { ξ i } is a martingale-difference sequence, Lemma 15 yields
E [ ∥ S d ∥ 2 ] ≤ κ ∑ i = 1 d E [ ∥ ξ i ∥ 2 ] = κ d . \mathbb{E}\left[\lVert S_d\rVert^2\right]\le\kappa\sum_{i=1}^{d}\mathbb{E}\left[\lVert\xi_i\rVert^2\right]=\kappa d. E [ ∥ S d ∥ 2 ] ≤ κ i = 1 ∑ d E [ ∥ ξ i ∥ 2 ] = κ d .
Combining with E [ ∥ S d ∥ 2 ] = d 2 / p \mathbb{E}\left[\lVert S_d\rVert^2\right]=d^{2/p} E [ ∥ S d ∥ 2 ] = d 2/ p gives
d 2 / p ≤ κ d , d^{2/p}\le\kappa d, d 2/ p ≤ κ d ,
and therefore
κ ≥ d 2 p − 1 . \kappa\ge d^{\frac{2}{p}-1}. κ ≥ d p 2 − 1 .
This proves (106) .
Case 2: 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ and d ≥ 8 d \ge8 d ≥ 8 .
We first treat the subcase 2 ≤ p < ∞ 2 \le p < \infty 2 ≤ p < ∞ .
Step 1: Parameter choice and an explicit family of vectors. Let
n : = ⌊ min { p , log 2 d } ⌋ . n := \left\lfloor\min\{p,\log_{2} d\}\right\rfloor. n := ⌊ min { p , log 2 d } ⌋ .
Since d ≥ 8 d \ge8 d ≥ 8 , we have log 2 d ≥ 3 \log_{2} d \ge3 log 2 d ≥ 3 and hence n ≥ 2 n \ge2 n ≥ 2 . Define
Ω : = { ± 1 } n , ∣ Ω ∣ = 2 n ≤ d . \Omega:= \{\pm1\}^{n}, \qquad|\Omega| = 2^{n} \le d. Ω := { ± 1 } n , ∣Ω∣ = 2 n ≤ d .
Fix an arbitrary enumeration
Ω = { ω ( 1 ) , ω ( 2 ) , … , ω ( 2 n ) } , ω ( j ) = ( ω 1 ( j ) , … , ω n ( j ) ) ∈ { ± 1 } n . \Omega= \{\omega^{(1)},\omega^{(2)},\ldots,\omega^{(2^{n})}\}, \qquad\omega^{(j)} = (\omega_{1}^{(j)},\ldots,\omega_{n}^{(j)}) \in\{\pm1\}^{n}. Ω = { ω ( 1 ) , ω ( 2 ) , … , ω ( 2 n ) } , ω ( j ) = ( ω 1 ( j ) , … , ω n ( j ) ) ∈ { ± 1 } n .
We identify the first 2 n 2^{n} 2 n coordinates of R d \mathbb{R}^{d} R d with the index set { 1 , … , 2 n } \{1,\ldots,2^{n}\} { 1 , … , 2 n } and set all remaining coordinates equal to 0 0 0 .
For each i ∈ { 1 , … , n } i \in\{1,\ldots,n\} i ∈ { 1 , … , n } , define a vector u i ∈ R d u_i \in\mathbb{R}^{d} u i ∈ R d by
( u i ) j : = 2 − n / p ω i ( j ) for j = 1 , … , 2 n , ( u i ) j : = 0 for j = 2 n + 1 , … , d . (u_i)_j := 2^{-n/p}\omega_i^{(j)} \quad\text{for } j=1,\ldots,2^{n}, \qquad(u_i)_j := 0 \quad\text{for } j=2^{n}+1,\ldots,d. ( u i ) j := 2 − n / p ω i ( j ) for j = 1 , … , 2 n , ( u i ) j := 0 for j = 2 n + 1 , … , d .
Then each u i u_i u i has exactly 2 n 2^{n} 2 n nonzero entries, each equal to ± 2 − n / p \pm2^{-n/p} ± 2 − n / p , and therefore
∥ u i ∥ p = ∥ u i ∥ p p = ∑ j = 1 2 n ∣ 2 − n / p ω i ( j ) ∣ p = 2 n ⋅ 2 − n = 1. \|u_i\|^{p} = \|u_i\|_{p}^{p} = \sum_{j=1}^{2^{n}}\left|2^{-n/p}\omega_i^{(j)}\right|^{p} = 2^{n}\cdot2^{-n} = 1. ∥ u i ∥ p = ∥ u i ∥ p p = j = 1 ∑ 2 n 2 − n / p ω i ( j ) p = 2 n ⋅ 2 − n = 1.
Hence
∥ u i ∥ = 1 , i = 1 , … , n . \|u_i\| = 1, \qquad i=1,\ldots,n. ∥ u i ∥ = 1 , i = 1 , … , n .
Step 2: A martingale-difference sequence. Let { ε i } i = 1 n \{\varepsilon_i\}_{i=1}^{n} { ε i } i = 1 n be i.i.d. Rademacher random variables, define
F i : = σ ( ε 1 , … , ε i ) , ξ i : = ε i u i , i = 1 , … , n . \mathcal{F}_i := \sigma(\varepsilon_1,\ldots,\varepsilon_i), \qquad\xi_i := \varepsilon_i u_i, \qquad i=1,\ldots,n. F i := σ ( ε 1 , … , ε i ) , ξ i := ε i u i , i = 1 , … , n .
Then { ξ i } i = 1 n \{\xi_i\}_{i=1}^{n} { ξ i } i = 1 n is an { F i } \{\mathcal{F}_i\} { F i } -martingale-difference sequence and
∥ ξ i ∥ = ∥ u i ∥ = 1 , i = 1 , … , n , \|\xi_i\| = \|u_i\| = 1, \qquad i=1,\ldots,n, ∥ ξ i ∥ = ∥ u i ∥ = 1 , i = 1 , … , n ,
so
∑ i = 1 n E [ ∥ ξ i ∥ 2 ] = n . \sum_{i=1}^{n}\mathbb{E}\left[\|\xi_i\|^{2}\right] = n. i = 1 ∑ n E [ ∥ ξ i ∥ 2 ] = n .
Step 3: Lower bound on E [ ∥ S n ∥ p ] \mathbb{E}\left[\|S_n\|^{p}\right] E [ ∥ S n ∥ p ] . Let
S n : = ∑ i = 1 n ξ i . S_n := \sum_{i=1}^{n}\xi_i. S n := i = 1 ∑ n ξ i .
For each coordinate j ∈ { 1 , … , 2 n } j \in\{1,\ldots,2^{n}\} j ∈ { 1 , … , 2 n } , we have
( S n ) j = ∑ i = 1 n ε i ( u i ) j = 2 − n / p ∑ i = 1 n ε i ω i ( j ) . (S_n)_j = \sum_{i=1}^{n}\varepsilon_i(u_i)_j = 2^{-n/p}\sum_{i=1}^{n}\varepsilon_i\omega_i^{(j)}. ( S n ) j = i = 1 ∑ n ε i ( u i ) j = 2 − n / p i = 1 ∑ n ε i ω i ( j ) .
Therefore,
∥ S n ∥ p = ∥ S n ∥ p p = ∑ j = 1 2 n ∣ 2 − n / p ∑ i = 1 n ε i ω i ( j ) ∣ p = 2 − n ∑ j = 1 2 n ∣ ∑ i = 1 n ε i ω i ( j ) ∣ p = E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 n ε i ω i ∣ p ] , \begin{aligned}
\|S_n\|^{p} = \|S_n\|_{p}^{p}
&= \sum_{j=1}^{2^{n}}\left|2^{-n/p}\sum_{i=1}^{n}\varepsilon_i\omega_i^{(j)}\right|^{p} \\
&= 2^{-n}\sum_{j=1}^{2^{n}}\left|\sum_{i=1}^{n}\varepsilon_i\omega_i^{(j)}\right|^{p} \\
&= \mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^{n}\varepsilon_i\omega_i\right|^{p}\right],
\end{aligned} ∥ S n ∥ p = ∥ S n ∥ p p = j = 1 ∑ 2 n 2 − n / p i = 1 ∑ n ε i ω i ( j ) p = 2 − n j = 1 ∑ 2 n i = 1 ∑ n ε i ω i ( j ) p = E ω ∼ Unif ( Ω ) [ i = 1 ∑ n ε i ω i p ] ,
where the last equality holds because { ω ( j ) } j = 1 2 n \{\omega^{(j)}\}_{j=1}^{2^n} { ω ( j ) } j = 1 2 n is merely an enumeration of Ω \Omega Ω .
Since ω ∼ Unif ( Ω ) \omega\sim\operatorname{Unif}(\Omega) ω ∼ Unif ( Ω ) is uniform on { ± 1 } n \{\pm1\}^n { ± 1 } n , for every fixed realization of ε = ( ε 1 , … , ε n ) ∈ { ± 1 } n \varepsilon=(\varepsilon_1,\ldots,\varepsilon_n)\in\{\pm1\}^n ε = ( ε 1 , … , ε n ) ∈ { ± 1 } n we have
E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 n ε i ω i ∣ p ] = E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 n ω i ∣ p ] . \mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^n\varepsilon_i\omega_i\right|^p\right]
=
\mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^n\omega_i\right|^p\right]. E ω ∼ Unif ( Ω ) [ i = 1 ∑ n ε i ω i p ] = E ω ∼ Unif ( Ω ) [ i = 1 ∑ n ω i p ] .
because the coordinatewise sign-flip map
ω ↦ ( ε 1 ω 1 , … , ε n ω n ) \omega\mapsto(\varepsilon_1\omega_1,\ldots,\varepsilon_n\omega_n) ω ↦ ( ε 1 ω 1 , … , ε n ω n )
is a bijection of Ω \Omega Ω and hence preserves the uniform law. Therefore,
∥ S n ∥ = ( E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 n ω i ∣ p ] ) 1 / p . \lVert S_n\rVert
=
\left(\mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^n\omega_i\right|^p\right]\right)^{1/p}. ∥ S n ∥ = ( E ω ∼ Unif ( Ω ) [ i = 1 ∑ n ω i p ] ) 1/ p .
In particular, ∥ S n ∥ \lVert S_n\rVert ∥ S n ∥ is deterministic, and thus
E [ ∥ S n ∥ 2 ] = ∥ S n ∥ 2 = ( E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 n ω i ∣ p ] ) 2 / p . \mathbb{E}\left[\lVert S_n\rVert^2\right]
=
\lVert S_n\rVert^2
=
\left(\mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^n\omega_i\right|^p\right]\right)^{2/p}. E [ ∥ S n ∥ 2 ] = ∥ S n ∥ 2 = ( E ω ∼ Unif ( Ω ) [ i = 1 ∑ n ω i p ] ) 2/ p .
We now lower bound this quantity using a single atom of the distribution. Let
Z n : = ∑ i = 1 n ω i , ω ∼ Unif ( Ω ) . Z_n:=\sum_{i=1}^n\omega_i,\qquad\omega\sim\operatorname{Unif}(\Omega). Z n := i = 1 ∑ n ω i , ω ∼ Unif ( Ω ) .
Since Prob { Z n = n } = 2 − n \operatorname{Prob}\{Z_n=n\}=2^{-n} Prob { Z n = n } = 2 − n , we obtain
E ω ∼ Unif ( Ω ) [ ∣ Z n ∣ p ] ≥ n p Prob { Z n = n } = n p 2 − n . \mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}[|Z_n|^p]
\ge n^p\operatorname{Prob}\{Z_n=n\}
=n^p2^{-n}. E ω ∼ Unif ( Ω ) [ ∣ Z n ∣ p ] ≥ n p Prob { Z n = n } = n p 2 − n .
Therefore,
∥ S n ∥ = ( E ω ∼ Unif ( Ω ) [ ∣ Z n ∣ p ] ) 1 / p ≥ n ⋅ 2 − n / p . \lVert S_n\rVert
=
\left(\mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}[|Z_n|^p]\right)^{1/p}
\ge n\cdot2^{-n/p}. ∥ S n ∥ = ( E ω ∼ Unif ( Ω ) [ ∣ Z n ∣ p ] ) 1/ p ≥ n ⋅ 2 − n / p .
By construction n ≤ p n\le p n ≤ p , hence 2 − n / p ≥ 1 / 2 2^{-n/p}\ge1/2 2 − n / p ≥ 1/2 , and consequently
E [ ∥ S n ∥ 2 ] = ∥ S n ∥ 2 ≥ n 2 4 . \mathbb{E}\left[\lVert S_n\rVert^2\right]
=
\lVert S_n\rVert^2
\ge\frac{n^2}{4}. E [ ∥ S n ∥ 2 ] = ∥ S n ∥ 2 ≥ 4 n 2 .
Step 4: Conclude via the second-moment bound. Up to this point, all expectations involving ω \omega ω were taken with respect to the auxiliary randomness ω ∼ Unif ( Ω ) \omega\sim\operatorname{Unif}(\Omega) ω ∼ Unif ( Ω ) . We now switch back to the original probability space on which the martingale-difference sequence { ξ i } i = 1 n \{\xi_i\}_{i=1}^n { ξ i } i = 1 n (equivalently, the Rademacher signs { ε i } i = 1 n \{\varepsilon_i\}_{i=1}^n { ε i } i = 1 n ) is defined. In particular, in the remainder of the proof, E [ ⋅ ] \mathbb{E}[\cdot] E [ ⋅ ] denotes expectation with respect to the randomness of ( ε 1 , … , ε n ) (\varepsilon_1,\ldots,\varepsilon_n) ( ε 1 , … , ε n ) .
Since ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) is κ \kappa κ -regular and { ξ i } \{\xi_i\} { ξ i } is a martingale-difference sequence, Lemma 15 yields
E [ ∥ S n ∥ 2 ] ≤ κ ∑ i = 1 n E [ ∥ ξ i ∥ 2 ] = κ n . \mathbb{E}\left[\lVert S_n\rVert^2\right]
\le\kappa\sum_{i=1}^n\mathbb{E}\left[\lVert\xi_i\rVert^2\right]
=\kappa n. E [ ∥ S n ∥ 2 ] ≤ κ i = 1 ∑ n E [ ∥ ξ i ∥ 2 ] = κn .
Combining this with the lower bound E [ ∥ S n ∥ 2 ] ≥ n 2 / 4 \mathbb{E}\left[\lVert S_n\rVert^2\right]\ge n^2/4 E [ ∥ S n ∥ 2 ] ≥ n 2 /4 gives
κ ≥ n 4 . \kappa\ge\frac{n}{4}. κ ≥ 4 n .
Finally, since n = ⌊ min { p , log 2 d } ⌋ n=\left\lfloor\min\{p,\log_2 d\}\right\rfloor n = ⌊ min { p , log 2 d } ⌋ and d ≥ 8 d\ge8 d ≥ 8 , we have
n ≥ min { p − 1 , log 2 d − 1 } ≥ 1 2 min { p − 1 , log 2 d } ≥ 1 2 min { p − 1 , ln d } . \begin{aligned}
n\ge\min\{p-1,\log_2 d-1\} \\
&\ge\frac{1}{2}\min\{p-1,\log_2 d\} \\
&\ge\frac{1}{2}\min\{p-1,\ln d\}.
\end{aligned} n ≥ min { p − 1 , log 2 d − 1 } ≥ 2 1 min { p − 1 , log 2 d } ≥ 2 1 min { p − 1 , ln d } .
Therefore,
κ ≥ n 4 ≥ 1 8 min { p − 1 , ln d } , \kappa\ge\frac{n}{4}\ge\frac{1}{8}\min\{p-1,\ln d\}, κ ≥ 4 n ≥ 8 1 min { p − 1 , ln d } ,
which proves (107) for 2 ≤ p < ∞ 2\le p<\infty 2 ≤ p < ∞ .
Subcase p = ∞ p=\infty p = ∞ . Let
n : = ⌊ log 2 d ⌋ , Ω : = { ± 1 } n , ∣ Ω ∣ = 2 n ≤ d . n := \lfloor\log_{2} d \rfloor,\qquad\Omega:= \{\pm1\}^{n},\qquad|\Omega|=2^{n}\le d. n := ⌊ log 2 d ⌋ , Ω := { ± 1 } n , ∣Ω∣ = 2 n ≤ d .
Fix an enumeration
Ω = { ω ( 1 ) , … , ω ( 2 n ) } , ω ( j ) = ( ω 1 ( j ) , … , ω n ( j ) ) ∈ { ± 1 } n . \Omega=\{\omega^{(1)},\ldots,\omega^{(2^{n})}\},\qquad\omega^{(j)}=(\omega_{1}^{(j)},\ldots,\omega_{n}^{(j)})\in\{\pm1\}^{n}. Ω = { ω ( 1 ) , … , ω ( 2 n ) } , ω ( j ) = ( ω 1 ( j ) , … , ω n ( j ) ) ∈ { ± 1 } n .
As before, we identify the first 2 n 2^{n} 2 n coordinates of R d \mathbb{R}^{d} R d with the index set { 1 , … , 2 n } \{1,\ldots,2^{n}\} { 1 , … , 2 n } and set all remaining coordinates equal to 0 0 0 .
For each i ∈ { 1 , … , n } i\in\{1,\ldots,n\} i ∈ { 1 , … , n } , define u i ∈ R d u_i\in\mathbb{R}^{d} u i ∈ R d by
( u i ) j : = ω i ( j ) for j = 1 , … , 2 n , ( u i ) j : = 0 for j = 2 n + 1 , … , d . (u_i)_j:=\omega_i^{(j)}\quad\text{for }j=1,\ldots,2^{n},\qquad(u_i)_j:=0\quad\text{for }j=2^{n}+1,\ldots,d. ( u i ) j := ω i ( j ) for j = 1 , … , 2 n , ( u i ) j := 0 for j = 2 n + 1 , … , d .
Then every nonzero entry of u i u_i u i equals ± 1 \pm1 ± 1 , and hence
∥ u i ∥ = ∥ u i ∥ ∞ = 1 , i = 1 , … , n . \|u_i\|=\|u_i\|_{\infty}=1,\qquad i=1,\ldots,n. ∥ u i ∥ = ∥ u i ∥ ∞ = 1 , i = 1 , … , n .
Let { ε i } i = 1 n \{\varepsilon_i\}_{i=1}^{n} { ε i } i = 1 n be i.i.d. Rademacher random variables and define
F i : = σ ( ε 1 , … , ε i ) , ξ i : = ε i u i , i = 1 , … , n . \mathcal{F}_i:=\sigma(\varepsilon_1,\ldots,\varepsilon_i),\qquad\xi_i:=\varepsilon_i u_i,\qquad i=1,\ldots,n. F i := σ ( ε 1 , … , ε i ) , ξ i := ε i u i , i = 1 , … , n .
Then { ξ i } i = 1 n \{\xi_i\}_{i=1}^{n} { ξ i } i = 1 n is an { F i } \{\mathcal{F}_i\} { F i } -martingale-difference sequence and
∑ i = 1 n E [ ∥ ξ i ∥ 2 ] = ∑ i = 1 n E [ ∥ ξ i ∥ ∞ 2 ] = n . \sum_{i=1}^{n}\mathbb{E}\left[\|\xi_i\|^2\right]=\sum_{i=1}^{n}\mathbb{E}\left[\|\xi_i\|_{\infty}^2\right]=n. i = 1 ∑ n E [ ∥ ξ i ∥ 2 ] = i = 1 ∑ n E [ ∥ ξ i ∥ ∞ 2 ] = n .
Now consider
S n : = ∑ i = 1 n ξ i . S_n:=\sum_{i=1}^{n}\xi_i. S n := i = 1 ∑ n ξ i .
For each j ∈ { 1 , … , 2 n } j\in\{1,\ldots,2^{n}\} j ∈ { 1 , … , 2 n } , we have
( S n ) j = ∑ i = 1 n ε i ( u i ) j = ∑ i = 1 n ε i ω i ( j ) . (S_n)_j=\sum_{i=1}^{n}\varepsilon_i(u_i)_j=\sum_{i=1}^{n}\varepsilon_i\omega_i^{(j)}. ( S n ) j = i = 1 ∑ n ε i ( u i ) j = i = 1 ∑ n ε i ω i ( j ) .
Hence
∥ S n ∥ = ∥ S n ∥ ∞ = max 1 ≤ j ≤ 2 n ∣ ( S n ) j ∣ = max ω ∈ Ω ∣ ∑ i = 1 n ε i ω i ∣ . \|S_n\|=\|S_n\|_{\infty}=\max_{1\le j\le2^{n}}|(S_n)_j|=\max_{\omega\in\Omega}\left|\sum_{i=1}^{n}\varepsilon_i\omega_i\right|. ∥ S n ∥ = ∥ S n ∥ ∞ = 1 ≤ j ≤ 2 n max ∣ ( S n ) j ∣ = ω ∈ Ω max i = 1 ∑ n ε i ω i .
For any fixed realization of ε = ( ε 1 , … , ε n ) ∈ { ± 1 } n \varepsilon=(\varepsilon_1,\ldots,\varepsilon_n)\in\{\pm1\}^{n} ε = ( ε 1 , … , ε n ) ∈ { ± 1 } n , we may choose ω = ε \omega=\varepsilon ω = ε . Then
∑ i = 1 n ε i ω i = ∑ i = 1 n ε i 2 = n , \sum_{i=1}^{n}\varepsilon_i\omega_i=\sum_{i=1}^{n}\varepsilon_i^2=n, i = 1 ∑ n ε i ω i = i = 1 ∑ n ε i 2 = n ,
so the maximum is at least n n n . On the other hand, for every ω ∈ { ± 1 } n \omega\in\{\pm1\}^{n} ω ∈ { ± 1 } n ,
∣ ∑ i = 1 n ε i ω i ∣ ≤ ∑ i = 1 n ∣ ε i ω i ∣ = n , \left|\sum_{i=1}^{n}\varepsilon_i\omega_i\right|\le\sum_{i=1}^{n}|\varepsilon_i\omega_i|=n, i = 1 ∑ n ε i ω i ≤ i = 1 ∑ n ∣ ε i ω i ∣ = n ,
so the maximum is also at most n n n . Therefore,
∥ S n ∥ = n deterministically, and hence E [ ∥ S n ∥ 2 ] = n 2 . \|S_n\|=n\qquad\text{deterministically, and hence}\qquad\mathbb{E}\left[\|S_n\|^2\right]=n^2. ∥ S n ∥ = n deterministically, and hence E [ ∥ S n ∥ 2 ] = n 2 .
Finally, since ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) is κ \kappa κ -regular and { ξ i } \{\xi_i\} { ξ i } is a martingale-difference sequence, Lemma 15 yields
E [ ∥ S n ∥ 2 ] ≤ κ ∑ i = 1 n E [ ∥ ξ i ∥ 2 ] = κ n . \mathbb{E}\left[\|S_n\|^2\right]\le\kappa\sum_{i=1}^{n}\mathbb{E}\left[\|\xi_i\|^2\right]=\kappa n. E [ ∥ S n ∥ 2 ] ≤ κ i = 1 ∑ n E [ ∥ ξ i ∥ 2 ] = κn .
Combining with E [ ∥ S n ∥ 2 ] = n 2 \mathbb{E}\left[\|S_n\|^2\right]=n^2 E [ ∥ S n ∥ 2 ] = n 2 gives
Since n = ⌊ log 2 d ⌋ ≥ 1 2 ln d n=\lfloor\log_2 d\rfloor\ge\frac{1}{2}\ln d n = ⌊ log 2 d ⌋ ≥ 2 1 ln d for d ≥ 8 d\ge8 d ≥ 8 , we obtain
κ ≥ 1 2 ln d , \kappa\ge\frac{1}{2}\ln d, κ ≥ 2 1 ln d ,
which is stronger than (107) in the case p = ∞ p=\infty p = ∞ , because
min { p − 1 , ln d } = ln d when p = ∞ . \min\{p-1,\ln d\}=\ln d\qquad\text{when }p=\infty. min { p − 1 , ln d } = ln d when p = ∞.
Remark 6 (Prior work). Dümbgen et al. [ [19 ] give a logarithmic lower bound for the second-moment constant in ℓ ∞ \ell^\infty ℓ ∞ ; their Lemma 3.6 (p. 146) gives corresponding bounds for the squared type-2 constant. These results concern moment inequalities, and imply lower bounds on our regularity parameter through Lemma 15.
For 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ , the general ℓ p d \ell_p^d ℓ p d order follows with an additional argument. Let K p ( d ) K_p(d) K p ( d ) be the smallest constant such that
E [ ∥ ∑ i X i ∥ p 2 ] ≤ K p ( d ) ∑ i E [ ∥ X i ∥ p 2 ] . \mathbb{E}\left[\left\|\sum_i X_i\right\|_p^2\right] \le K_p(d)\sum_i \mathbb{E}\left[\|X_i\|_p^2\right]. E i ∑ X i p 2 ≤ K p ( d ) i ∑ E [ ∥ X i ∥ p 2 ] .
for every finite sequence of independent, mean-zero random vectors in R d \mathbb{R}^d R d with finite second moments. The cited results, together with K ∞ ( m ) ≥ 1 K_\infty(m) \ge1 K ∞ ( m ) ≥ 1 , give K ∞ ( m ) ≥ c ln ( e m ) K_\infty(m) \ge c\ln(em) K ∞ ( m ) ≥ c ln ( e m ) for all m ≥ 1 m \ge1 m ≥ 1 and an absolute constant c > 0 c > 0 c > 0 . Embedding R m \mathbb{R}^m R m as a coordinate subspace of R d \mathbb{R}^d R d and using ∥ z ∥ ∞ ≤ ∥ z ∥ p ≤ m 1 / p ∥ z ∥ ∞ \|z\|_\infty\le\|z\|_p \le m^{1/p}\|z\|_\infty ∥ z ∥ ∞ ≤ ∥ z ∥ p ≤ m 1/ p ∥ z ∥ ∞ gives, for every 1 ≤ m ≤ d 1 \le m \le d 1 ≤ m ≤ d ,
κ V ≥ K p ( d ) ≥ K p ( m ) ≥ m − 2 / p K ∞ ( m ) ≥ c m − 2 / p ln ( e m ) . \kappa_V \ge K_p(d) \ge K_p(m) \ge m^{-2/p}K_\infty(m) \ge c m^{-2/p}\ln(em). κ V ≥ K p ( d ) ≥ K p ( m ) ≥ m − 2/ p K ∞ ( m ) ≥ c m − 2/ p ln ( e m ) .
For finite p ≥ 2 p \ge2 p ≥ 2 , choose m = min { d , ⌊ e p ⌋ } m=\min\{d,\lfloor e^p\rfloor\} m = min { d , ⌊ e p ⌋} ; then m 2 / p ≤ e 2 m^{2/p} \le e^2 m 2/ p ≤ e 2 and ln ( e m ) ≥ min { p , ln ( e d ) } \ln(em) \ge\min\{p,\ln(ed)\} ln ( e m ) ≥ min { p , ln ( e d )} . For p = ∞ p=\infty p = ∞ , take m = d m=d m = d and use m − 2 / p = 1 m^{-2/p}=1 m − 2/ p = 1 . Thus the lower-bound order min { p , ln ( e d ) } \min\{p,\ln(ed)\} min { p , ln ( e d )} follows, with an unspecified absolute constant.
The explicit factor 1 / 8 1/8 1/8 in (107) is supplied by our direct proof above. That proof was obtained independently, before consulting this reference; we include it for completeness and to keep the exposition self-contained within our framework.
Spaces with Schatten-p p p Norms on R m × n \mathbb{R}^{m\times n} R m × n
We next record the analogous regularity bounds for Schatten-p p p norms on matrix spaces. Since the Schatten-p p p norm is simply the ℓ p \ell_p ℓ p norm of the singular-value vector, the behavior closely parallels the vector case: for p ≥ 2 p \ge2 p ≥ 2 , one may take a regularity parameter that grows at most logarithmically in the effective dimension r : = min { m , n } r:=\min\{m,n\} r := min { m , n } , whereas for p < 2 p<2 p < 2 , the natural admissible parameter grows polynomially in r r r .
Proposition 14 (Schatten-p p p norms for matrices). Fix m , n ∈ N m,n\in\mathbb{N} m , n ∈ N and let r : = min { m , n } r:=\min\{m,n\} r := min { m , n } . Let V : = R m × n V:=\mathbb{R}^{m\times n} V := R m × n and equip it with the norm ∥ ⋅ ∥ V : = ∥ ⋅ ∥ S p \|\cdot\|_V:=\|\cdot\|_{S_p} ∥ ⋅ ∥ V := ∥ ⋅ ∥ S p , where
∥ X ∥ S p : = ∥ σ ( X ) ∥ p = ( ∑ i = 1 r σ i ( X ) p ) 1 / p , p ∈ [ 1 , ∞ ) , \|X\|_{S_p}:=\|\sigma(X)\|_p=\left(\sum_{i=1}^{r}\sigma_i(X)^p\right)^{1/p},\qquad p\in[1,\infty), ∥ X ∥ S p := ∥ σ ( X ) ∥ p = ( i = 1 ∑ r σ i ( X ) p ) 1/ p , p ∈ [ 1 , ∞ ) , and
∥ X ∥ S ∞ : = max 1 ≤ i ≤ r σ i ( X ) . \|X\|_{S_\infty}:=\max_{1\le i\le r}\sigma_i(X). ∥ X ∥ S ∞ := 1 ≤ i ≤ r max σ i ( X ) . Then the space ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) is κ V \kappa_V κ V -regular in the sense of Definition 1, with admissible regularity parameter κ V : = κ p ( m , n ) \kappa_V:=\kappa_p(m,n) κ V := κ p ( m , n ) given as follows:
Let 2 ≤ p ≤ ∞ 2\le p\le\infty 2 ≤ p ≤ ∞ . Then
κ V : = κ p ( m , n ) = min 2 ≤ ρ ≤ p max { 2 , ρ − 1 } r 2 ρ − 2 p . \kappa_V:=\kappa_p(m,n)=\min_{2\le\rho\le p}\max\{2,\rho-1\}r^{\frac{2}{\rho}-\frac{2}{p}}. κ V := κ p ( m , n ) = 2 ≤ ρ ≤ p min max { 2 , ρ − 1 } r ρ 2 − p 2 . For r ≥ 8 r\ge8 r ≥ 8 , this parameter is realized by the auxiliary norm
∥ X ∥ V , + : = ∥ X ∥ S ρ ∗ , ρ ∗ : = min { p , max { 3 , ln r + ( ln r ) 2 − 2 ln r } } . \|X\|_{V,+}:=\|X\|_{S_{\rho_*}},\qquad\rho_*:=\min\left\{p,\max\left\{3,\ln r+\sqrt{(\ln r)^2-2\ln r}\right\}\right\}. ∥ X ∥ V , + := ∥ X ∥ S ρ ∗ , ρ ∗ := min { p , max { 3 , ln r + ( ln r ) 2 − 2 ln r } } . Under the same condition r ≥ 8 r\ge8 r ≥ 8 ,
κ V : = κ p ( m , n ) = O ( min { p , ln ( e r ) } ) . \kappa_V:=\kappa_p(m,n)=\mathcal{O}\left(\min\{p,\ln(er)\}\right). κ V := κ p ( m , n ) = O ( min { p , ln ( er )} ) . Let 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 . Then
κ V : = κ p ( m , n ) = r 2 p − 1 . \kappa_V:=\kappa_p(m,n)=r^{\frac{2}{p}-1}. κ V := κ p ( m , n ) = r p 2 − 1 .
Proof. To simplify notation throughout this proof, we write
∥ X ∥ : = ∥ X ∥ V = ∥ X ∥ S p , κ : = κ V . \lVert X\rVert:= \lVert X\rVert_V = \lVert X\rVert_{S_p}, \qquad\kappa:= \kappa_V. ∥ X ∥ := ∥ X ∥ V = ∥ X ∥ S p , κ := κ V .
Whenever an auxiliary norm is introduced, we denote it by
∥ X ∥ + : = ∥ X ∥ V , + , κ + : = κ V , + . \lVert X\rVert_+ := \lVert X\rVert_{V,+}, \qquad\kappa_+ := \kappa_{V,+}. ∥ X ∥ + := ∥ X ∥ V , + , κ + := κ V , + .
We consider separately the cases 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ and 1 ≤ p < 2 1 \le p < 2 1 ≤ p < 2 .
(i) Case 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ . This case is established in [40 ] ; see the proof of Example 3.3 therein. In our notation, this yields that ( V , ∥ ⋅ ∥ ) (V,\lVert\cdot\rVert) ( V , ∥ ⋅ ∥) is κ \kappa κ -regular with
κ : = κ p ( m , n ) = min 2 ≤ ρ ≤ p ( max { 2 , ρ − 1 } r 2 ρ − 2 p ) . \kappa:= \kappa_p(m,n) = \min_{2 \le\rho\le p} \left(\max\{2,\rho-1\}r^{\frac{2}{\rho}-\frac{2}{p}}\right). κ := κ p ( m , n ) = 2 ≤ ρ ≤ p min ( max { 2 , ρ − 1 } r ρ 2 − p 2 ) .
We now derive the minimizing exponent for r ≥ 8 r \ge8 r ≥ 8 . Set a : = ln r > 2 a := \ln r > 2 a := ln r > 2 and define, for finite ρ ≥ 2 \rho\ge2 ρ ≥ 2 ,
F ( ρ ) : = max { 2 , ρ − 1 } r 2 ρ − 2 p , F(\rho) := \max\{2,\rho-1\}r^{\frac{2}{\rho}-\frac{2}{p}}, F ( ρ ) := max { 2 , ρ − 1 } r ρ 2 − p 2 ,
with the convention 1 / p = 0 1/p=0 1/ p = 0 when p = ∞ p=\infty p = ∞ . On 2 ≤ ρ ≤ 3 2 \le\rho\le3 2 ≤ ρ ≤ 3 , we have F ( ρ ) = 2 r 2 / ρ − 2 / p F(\rho)=2r^{2/\rho-2/p} F ( ρ ) = 2 r 2/ ρ − 2/ p and
d d ρ ln F ( ρ ) = − 2 a ρ 2 < 0 , 2 < ρ < 3. \frac{d}{d\rho}\ln F(\rho)=-\frac{2a}{\rho^2}<0,\qquad2<\rho<3. d ρ d ln F ( ρ ) = − ρ 2 2 a < 0 , 2 < ρ < 3.
Thus F F F is strictly decreasing on [ 2 , 3 ] [2,3] [ 2 , 3 ] ; in particular, if 2 ≤ p ≤ 3 2\le p\le3 2 ≤ p ≤ 3 , its minimum over [ 2 , p ] [2,p] [ 2 , p ] is attained at ρ = p \rho=p ρ = p .
On ρ ≥ 3 \rho\ge3 ρ ≥ 3 , we have F ( ρ ) = ( ρ − 1 ) r 2 / ρ − 2 / p F(\rho)=(\rho-1)r^{2/\rho-2/p} F ( ρ ) = ( ρ − 1 ) r 2/ ρ − 2/ p , so
d d ρ ln F ( ρ ) = 1 ρ − 1 − 2 a ρ 2 = ρ 2 − 2 a ρ + 2 a ρ 2 ( ρ − 1 ) , ρ > 3. \frac{d}{d\rho}\ln F(\rho)=\frac{1}{\rho-1}-\frac{2a}{\rho^2}=\frac{\rho^2-2a\rho+2a}{\rho^2(\rho-1)},\qquad\rho>3. d ρ d ln F ( ρ ) = ρ − 1 1 − ρ 2 2 a = ρ 2 ( ρ − 1 ) ρ 2 − 2 a ρ + 2 a , ρ > 3.
The numerator has roots
ρ − = a − a 2 − 2 a , ρ + = a + a 2 − 2 a . \rho_- = a-\sqrt{a^2-2a},\qquad\rho_+ = a+\sqrt{a^2-2a}. ρ − = a − a 2 − 2 a , ρ + = a + a 2 − 2 a .
Since a > 2 a>2 a > 2 and the numerator at ρ = 2 \rho=2 ρ = 2 equals 4 − 2 a < 0 4-2a<0 4 − 2 a < 0 , these roots satisfy ρ − < 2 < ρ + \rho_-<2<\rho_+ ρ − < 2 < ρ + . Consequently, if ρ + > 3 \rho_+>3 ρ + > 3 , then F F F decreases on [ 3 , ρ + ] [3,\rho_+] [ 3 , ρ + ] and increases on [ ρ + , ∞ ) [\rho_+,\infty) [ ρ + , ∞ ) ; if ρ + ≤ 3 \rho_+\le3 ρ + ≤ 3 , it increases on [ 3 , ∞ ) [3,\infty) [ 3 , ∞ ) . Together with continuity at ρ = 3 \rho=3 ρ = 3 and the decrease on [ 2 , 3 ] [2,3] [ 2 , 3 ] , this shows that the minimum over all finite ρ ≥ 2 \rho\ge2 ρ ≥ 2 is attained at max { 3 , ρ + } \max\{3,\rho_+\} max { 3 , ρ + } . Imposing ρ ≤ p \rho\le p ρ ≤ p therefore gives
ρ ∗ = min { p , max { 3 , ln r + ( ln r ) 2 − 2 ln r } } . \rho_*=\min\left\{p,\max\left\{3,\ln r+\sqrt{(\ln r)^2-2\ln r}\right\}\right\}. ρ ∗ = min { p , max { 3 , ln r + ( ln r ) 2 − 2 ln r } } .
For p = ∞ p=\infty p = ∞ , this yields the finite exponent ρ ∗ = max { 3 , ρ + } \rho_*=\max\{3,\rho_+\} ρ ∗ = max { 3 , ρ + } , so the auxiliary norm ∥ ⋅ ∥ S ρ ∗ \lVert\cdot\rVert_{S_{\rho_*}} ∥ ⋅ ∥ S ρ ∗ remains within the construction above.
For completeness, the order bound follows by choosing ρ = p \rho=p ρ = p when 2 ≤ p < 2 a 2\le p<2a 2 ≤ p < 2 a , which gives κ ≤ max { 2 , p − 1 } ≤ p < 2 a \kappa\le\max\{2,p-1\}\le p<2a κ ≤ max { 2 , p − 1 } ≤ p < 2 a . When p ≥ 2 a p\ge2a p ≥ 2 a , including p = ∞ p=\infty p = ∞ , the admissible choice ρ = 2 a \rho=2a ρ = 2 a gives
κ ≤ ( 2 a − 1 ) exp ( 1 − 2 a p ) ≤ 2 e a . \kappa\le(2a-1)\exp\left(1-\frac{2a}{p}\right)\le2ea. κ ≤ ( 2 a − 1 ) exp ( 1 − p 2 a ) ≤ 2 e a .
For every finite p ≥ 2 p\ge2 p ≥ 2 , the choice ρ = p \rho=p ρ = p also gives κ ≤ p \kappa\le p κ ≤ p . Hence, for r ≥ 8 r\ge8 r ≥ 8 ,
κ : = κ p ( m , n ) = O ( min { p , ln ( e r ) } ) . \kappa:=\kappa_p(m,n)=\mathcal{O}\left(\min\{p,\ln(er)\}\right). κ := κ p ( m , n ) = O ( min { p , ln ( er )} ) .
(ii) Case 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 .
Upper bound. Let ∥ ⋅ ∥ S 2 \lVert\cdot\rVert_{S_2} ∥ ⋅ ∥ S 2 be the Frobenius (Schatten-2) norm and define the auxiliary norm
∥ X ∥ + : = r 1 p − 1 2 ∥ X ∥ S 2 . \lVert X\rVert_+ := r^{\frac{1}{p}-\frac{1}{2}}\lVert X\rVert_{S_2}. ∥ X ∥ + := r p 1 − 2 1 ∥ X ∥ S 2 .
This is a scaled Hilbert norm, hence 1-smooth, so we may take
Moreover, for every X ∈ R m × n X \in\mathbb{R}^{m\times n} X ∈ R m × n ,
∥ X ∥ = ∥ X ∥ S p = ∥ σ ( X ) ∥ p ≤ r 1 p − 1 2 ∥ σ ( X ) ∥ 2 = r 1 p − 1 2 ∥ X ∥ S 2 = ∥ X ∥ + , \lVert X\rVert=\lVert X\rVert_{S_p}=\lVert\sigma(X)\rVert_p\le r^{\frac{1}{p}-\frac{1}{2}}\lVert\sigma(X)\rVert_2=r^{\frac{1}{p}-\frac{1}{2}}\lVert X\rVert_{S_2}=\lVert X\rVert_{+}, ∥ X ∥ = ∥ X ∥ S p = ∥ σ ( X ) ∥ p ≤ r p 1 − 2 1 ∥ σ ( X ) ∥ 2 = r p 1 − 2 1 ∥ X ∥ S 2 = ∥ X ∥ + ,
and since p < 2 p<2 p < 2 implies ∥ X ∥ S 2 ≤ ∥ X ∥ S p \lVert X\rVert_{S_2}\le\lVert X\rVert_{S_p} ∥ X ∥ S 2 ≤ ∥ X ∥ S p , we also have
∥ X ∥ + 2 = r 2 p − 1 ∥ X ∥ S 2 2 ≤ r 2 p − 1 ∥ X ∥ S p 2 = r 2 p − 1 ∥ X ∥ 2 . \lVert X\rVert_{+}^{2}=r^{\frac{2}{p}-1}\lVert X\rVert_{S_2}^{2}\le r^{\frac{2}{p}-1}\lVert X\rVert_{S_p}^{2}=r^{\frac{2}{p}-1}\lVert X\rVert^{2}. ∥ X ∥ + 2 = r p 2 − 1 ∥ X ∥ S 2 2 ≤ r p 2 − 1 ∥ X ∥ S p 2 = r p 2 − 1 ∥ X ∥ 2 .
Thus
∥ X ∥ 2 ≤ ∥ X ∥ + 2 ≤ r 2 p − 1 ∥ X ∥ 2 , ∀ X ∈ V . \lVert X\rVert^{2}\le\lVert X\rVert_{+}^{2}\le r^{\frac{2}{p}-1}\lVert X\rVert^{2},\qquad\forall X\in V. ∥ X ∥ 2 ≤ ∥ X ∥ + 2 ≤ r p 2 − 1 ∥ X ∥ 2 , ∀ X ∈ V .
Therefore, by Definition 1, the space ( V , ∥ ⋅ ∥ ) (V,\lVert\cdot\rVert) ( V , ∥ ⋅ ∥) is κ \kappa κ -regular with
κ : = κ p ( m , n ) = r 2 p − 1 . \kappa:=\kappa_p(m,n)=r^{\frac{2}{p}-1}. κ := κ p ( m , n ) = r p 2 − 1 .
We now complement Proposition 14 with corresponding lower bounds for the matrix setting. With r : = min { m , n } r:=\min\{m,n\} r := min { m , n } , the optimal parameter is r 2 / p − 1 r^{2/p-1} r 2/ p − 1 for 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 , and its optimal order is min { p , ln ( e r ) } \min\{p,\ln(er)\} min { p , ln ( er )} for 2 ≤ p ≤ ∞ 2\le p\le\infty 2 ≤ p ≤ ∞ , uniformly in r ≥ 1 r\ge1 r ≥ 1 ; see Remark 7. Consequently, the optimal parameter remains bounded as r r r varies with a fixed finite p ≥ 2 p\ge2 p ≥ 2 , and equals 1 1 1 at p = 2 p=2 p = 2 . The logarithmic branch of the minimum corresponds to p ≥ ln ( e r ) p\ge\ln(er) p ≥ ln ( er ) , including p = ∞ p=\infty p = ∞ , where logarithmic growth is unavoidable up to absolute constants.
Proposition 15 (Lower bounds for Schatten classes). Fix m , n ∈ N m,n\in\mathbb{N} m , n ∈ N , let r : = min { m , n } r:=\min\{m,n\} r := min { m , n } , let V : = R m × n V:=\mathbb{R}^{m\times n} V := R m × n , and equip it with the norm ∥ ⋅ ∥ V : = ∥ ⋅ ∥ S p \lVert\cdot\rVert_V:=\lVert\cdot\rVert_{S_p} ∥ ⋅ ∥ V := ∥ ⋅ ∥ S p . If the space ( V , ∥ ⋅ ∥ V ) (V,\lVert\cdot\rVert_V) ( V , ∥ ⋅ ∥ V ) is κ V \kappa_V κ V -regular, then the following hold:
If 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 , then
κ V ≥ r 2 p − 1 . (108) \kappa_V\ge r^{\frac{2}{p}-1}. \tag*{(108)} κ V ≥ r p 2 − 1 . (108) If 2 ≤ p ≤ ∞ 2\le p\le\infty 2 ≤ p ≤ ∞ and r ≥ 8 r\ge8 r ≥ 8 , then
κ V ≥ 1 8 min { p − 1 , ln r } . (109) \kappa_V\ge\frac{1}{8}\min\{p-1,\ln r\}. \tag*{(109)} κ V ≥ 8 1 min { p − 1 , ln r } . (109)
Proof. To simplify notation throughout this proof, we write
∥ X ∥ : = ∥ X ∥ V = ∥ X ∥ S p , κ : = κ V . \lVert X\rVert:=\lVert X\rVert_V=\lVert X\rVert_{S_p},\qquad\kappa:=\kappa_V. ∥ X ∥ := ∥ X ∥ V = ∥ X ∥ S p , κ := κ V .
Case 1: 1 ≤ p < 2 1\le p<2 1 ≤ p < 2 .
Let { ε i } i = 1 r \{\varepsilon_i\}_{i=1}^{r} { ε i } i = 1 r be independent Rademacher random variables and define the filtration
F i : = σ ( ε 1 , … , ε i ) , i = 1 , … , r . \mathcal{F}_i:=\sigma(\varepsilon_1,\ldots,\varepsilon_i),\qquad i=1,\ldots,r. F i := σ ( ε 1 , … , ε i ) , i = 1 , … , r .
For each i = 1 , … , r i=1,\ldots,r i = 1 , … , r , let E i i ∈ R m × n E_{ii}\in\mathbb{R}^{m\times n} E ii ∈ R m × n denote the matrix with a 1 in position ( i , i ) (i,i) ( i , i ) and zeros elsewhere, and set
ξ i : = ε i E i i , i = 1 , … , r . \xi_i:=\varepsilon_i E_{ii},\qquad i=1,\ldots,r. ξ i := ε i E ii , i = 1 , … , r .
Then { ξ i } i = 1 r \{\xi_i\}_{i=1}^{r} { ξ i } i = 1 r is an { F i } \{\mathcal{F}_i\} { F i } -martingale-difference sequence, since each ξ i \xi_i ξ i is F i \mathcal{F}_i F i -measurable and
E [ ξ i ∣ F i − 1 ] = E i i E [ ε i ∣ F i − 1 ] = 0 a.s. \mathbb{E}[\xi_i\mid\mathcal{F}_{i-1}]=E_{ii}\mathbb{E}[\varepsilon_i\mid\mathcal{F}_{i-1}]=0\qquad\text{a.s.} E [ ξ i ∣ F i − 1 ] = E ii E [ ε i ∣ F i − 1 ] = 0 a.s.
Next, note that E i i E_{ii} E ii has exactly one singular value equal to 1 and all remaining singular values equal to 0, hence
∥ ξ i ∥ = ∣ ε i ∣ ∥ E i i ∥ S p = 1 , E [ ∥ ξ i ∥ 2 ] = 1. \lVert\xi_i\rVert=|\varepsilon_i|\lVert E_{ii}\rVert_{S_p}=1,\qquad\mathbb{E}\bigl[\lVert\xi_i\rVert^2\bigr]=1. ∥ ξ i ∥ = ∣ ε i ∣ ∥ E ii ∥ S p = 1 , E [ ∥ ξ i ∥ 2 ] = 1.
Moreover, for
S r : = ∑ i = 1 r ξ i S_r := \sum_{i=1}^{r} \xi_i S r := i = 1 ∑ r ξ i
we have
S r = ∑ i = 1 r ε i E i i . S_r = \sum_{i=1}^{r} \varepsilon_i E_{ii}. S r = i = 1 ∑ r ε i E ii .
Recall that E i i ∈ R m × n E_{ii} \in\mathbb{R}^{m \times n} E ii ∈ R m × n is the matrix with a single 1 1 1 in entry ( i , i ) (i,i) ( i , i ) and zeros elsewhere. Therefore, S r S_r S r is the rectangular diagonal matrix whose first r r r diagonal entries are ( ε 1 , … , ε r ) (\varepsilon_1,\ldots,\varepsilon_r) ( ε 1 , … , ε r ) and whose remaining entries are zero:
( S r ) a b = { ε a , if a = b ≤ r , 0 , otherwise. (S_r)_{ab} =
\begin{cases}
\varepsilon_a, & \text{if } a=b\le r,\\
0, & \text{otherwise.}
\end{cases} ( S r ) ab = { ε a , 0 , if a = b ≤ r , otherwise.
For brevity, we denote this by
S r = Diag m × n ( ε 1 , … , ε r ) . S_r = \operatorname{Diag}_{m\times n}(\varepsilon_1,\ldots,\varepsilon_r). S r = Diag m × n ( ε 1 , … , ε r ) .
The singular values of S r S_r S r are ∣ ε 1 ∣ , … , ∣ ε r ∣ |\varepsilon_1|,\ldots,|\varepsilon_r| ∣ ε 1 ∣ , … , ∣ ε r ∣ (and zeros thereafter), so deterministically
∥ S r ∥ = ∥ S r ∥ S p = ( ∑ i = 1 r ∣ ε i ∣ p ) 1 / p = r 1 / p . \|S_r\| = \|S_r\|_{S_p} = \left(\sum_{i=1}^{r}|\varepsilon_i|^p\right)^{1/p} = r^{1/p}. ∥ S r ∥ = ∥ S r ∥ S p = ( i = 1 ∑ r ∣ ε i ∣ p ) 1/ p = r 1/ p .
Hence
E [ ∥ S r ∥ 2 ] = r 2 / p , ∑ i = 1 r E [ ∥ ξ i ∥ 2 ] = r . \mathbb{E}\left[\|S_r\|^2\right] = r^{2/p}, \qquad\sum_{i=1}^{r}\mathbb{E}\left[\|\xi_i\|^2\right] = r. E [ ∥ S r ∥ 2 ] = r 2/ p , i = 1 ∑ r E [ ∥ ξ i ∥ 2 ] = r .
Finally, since ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) is κ \kappa κ -regular and { ξ i } \{\xi_i\} { ξ i } is a martingale-difference sequence, Lemma 15 gives
E [ ∥ S r ∥ 2 ] ≤ κ ∑ i = 1 r E [ ∥ ξ i ∥ 2 ] = κ r . \mathbb{E}\left[\|S_r\|^2\right] \le\kappa\sum_{i=1}^{r}\mathbb{E}\left[\|\xi_i\|^2\right] = \kappa r. E [ ∥ S r ∥ 2 ] ≤ κ i = 1 ∑ r E [ ∥ ξ i ∥ 2 ] = κ r .
Combining this with E [ ∥ S r ∥ 2 ] = r 2 / p \mathbb{E}\left[\|S_r\|^2\right] = r^{2/p} E [ ∥ S r ∥ 2 ] = r 2/ p yields
r 2 / p ≤ κ r , r^{2/p} \le\kappa r, r 2/ p ≤ κ r ,
that is,
κ ≥ r 2 p − 1 . \kappa\ge r^{\frac{2}{p}-1}. κ ≥ r p 2 − 1 .
This proves (108).
Case 2: 2 ≤ p ≤ ∞ 2 \le p \le\infty 2 ≤ p ≤ ∞ and r ≥ 8 r \ge8 r ≥ 8 .
We first treat the subcase 2 ≤ p < ∞ 2 \le p < \infty 2 ≤ p < ∞ .
Step 1: Parameter choice and an explicit family of diagonal matrices. Let
N : = ⌊ min { p , log 2 r } ⌋ . N := \left\lfloor\min\{p,\log_2 r\}\right\rfloor. N := ⌊ min { p , log 2 r } ⌋ .
Since r ≥ 8 r \ge8 r ≥ 8 , we have log 2 r ≥ 3 \log_2 r \ge3 log 2 r ≥ 3 , so in particular N ≥ 2 N \ge2 N ≥ 2 and
2 N ≤ r ≤ min { m , n } . 2^N \le r \le\min\{m,n\}. 2 N ≤ r ≤ min { m , n } .
Define the discrete cube
Ω : = { ± 1 } N , ∣ Ω ∣ = 2 N . \Omega:= \{\pm1\}^N, \qquad|\Omega| = 2^N. Ω := { ± 1 } N , ∣Ω∣ = 2 N .
Fix an arbitrary enumeration
Ω = { ω ( 1 ) , … , ω ( 2 N ) } , ω ( j ) = ( ω 1 ( j ) , … , ω N ( j ) ) ∈ { ± 1 } N . \Omega= \{\omega^{(1)},\ldots,\omega^{(2^N)}\}, \qquad\omega^{(j)} = \left(\omega_1^{(j)},\ldots,\omega_N^{(j)}\right) \in\{\pm1\}^N. Ω = { ω ( 1 ) , … , ω ( 2 N ) } , ω ( j ) = ( ω 1 ( j ) , … , ω N ( j ) ) ∈ { ± 1 } N .
For each i ∈ { 1 , … , N } i \in\{1,\ldots,N\} i ∈ { 1 , … , N } , define a diagonal matrix U i ∈ R m × n U_i \in\mathbb{R}^{m\times n} U i ∈ R m × n by setting
( U i ) a b : = 0 , for a ≠ b , (U_i)_{ab}:=0,\qquad\text{for }a\ne b, ( U i ) ab := 0 , for a = b ,
and
( U i ) j j : = 2 − N / p ω i ( j ) , j = 1 , … , 2 N , (U_i)_{jj}:=2^{-N/p}\omega_i^{(j)},\qquad j=1,\ldots,2^N, ( U i ) j j := 2 − N / p ω i ( j ) , j = 1 , … , 2 N ,
while
( U i ) j j : = 0 , j = 2 N + 1 , … , r . (U_i)_{jj}:=0,\qquad j=2^N+1,\ldots,r. ( U i ) j j := 0 , j = 2 N + 1 , … , r .
Thus U i U_i U i has exactly 2 N 2^N 2 N nonzero diagonal entries, each equal to ± 2 − N / p \pm2^{-N/p} ± 2 − N / p . Since U i U_i U i is diagonal, its singular values are the absolute values of its diagonal entries (and zeros thereafter). Hence
∥ U i ∥ p = ∥ U i ∥ S p p = ∑ j = 1 2 N ∣ 2 − N / p ω i ( j ) ∣ p = 2 N ⋅ 2 − N = 1. \|U_i\|^p=\|U_i\|_{S_p}^p=\sum_{j=1}^{2^N}\left|2^{-N/p}\omega_i^{(j)}\right|^p=2^N\cdot2^{-N}=1. ∥ U i ∥ p = ∥ U i ∥ S p p = j = 1 ∑ 2 N 2 − N / p ω i ( j ) p = 2 N ⋅ 2 − N = 1.
Therefore,
∥ U i ∥ = 1 , i = 1 , … , N . \|U_i\|=1,\qquad i=1,\ldots,N. ∥ U i ∥ = 1 , i = 1 , … , N .
Step 2: A martingale-difference sequence. Let { ε i } i = 1 N \{\varepsilon_i\}_{i=1}^N { ε i } i = 1 N be i.i.d. Rademacher random variables, let
F i : = σ ( ε 1 , … , ε i ) , ξ i : = ε i U i , i = 1 , … , N . \mathcal{F}_i:=\sigma(\varepsilon_1,\ldots,\varepsilon_i),\qquad\xi_i:=\varepsilon_i U_i,\qquad i=1,\ldots,N. F i := σ ( ε 1 , … , ε i ) , ξ i := ε i U i , i = 1 , … , N .
Then { ξ i } i = 1 N \{\xi_i\}_{i=1}^N { ξ i } i = 1 N is an { F i } \{\mathcal{F}_i\} { F i } -martingale-difference sequence and
∥ ξ i ∥ = ∥ U i ∥ = 1 , i = 1 , … , N , \|\xi_i\|=\|U_i\|=1,\qquad i=1,\ldots,N, ∥ ξ i ∥ = ∥ U i ∥ = 1 , i = 1 , … , N ,
so
∑ i = 1 N E [ ∥ ξ i ∥ 2 ] = N . \sum_{i=1}^N\mathbb{E}\left[\|\xi_i\|^2\right]=N. i = 1 ∑ N E [ ∥ ξ i ∥ 2 ] = N .
Step 3: Lower bound on E [ ∥ S N ∥ 2 ] \mathbb{E}\left[\|S_N\|^2\right] E [ ∥ S N ∥ 2 ] . Let
S N : = ∑ i = 1 N ξ i . S_N:=\sum_{i=1}^N\xi_i. S N := i = 1 ∑ N ξ i .
Then S N S_N S N is diagonal, and for each j ∈ { 1 , … , 2 N } j\in\{1,\ldots,2^N\} j ∈ { 1 , … , 2 N } ,
( S N ) j j = ∑ i = 1 N ε i ( U i ) j j = 2 − N / p ∑ i = 1 N ε i ω i ( j ) . (S_N)_{jj}=\sum_{i=1}^N\varepsilon_i(U_i)_{jj}=2^{-N/p}\sum_{i=1}^N\varepsilon_i\omega_i^{(j)}. ( S N ) j j = i = 1 ∑ N ε i ( U i ) j j = 2 − N / p i = 1 ∑ N ε i ω i ( j ) .
Therefore,
∥ S N ∥ p = ∥ S N ∥ S p p = ∑ j = 1 2 N ∣ ( S N ) j j ∣ p = ∑ j = 1 2 N ∣ 2 − N / p ∑ i = 1 N ε i ω i ( j ) ∣ p = 2 − N ∑ j = 1 2 N ∣ ∑ i = 1 N ε i ω i ( j ) ∣ p = E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 N ε i ω i ∣ p ] , \begin{aligned}
\|S_N\|^p
&=\|S_N\|_{S_p}^p\\
&=\sum_{j=1}^{2^N}|(S_N)_{jj}|^p\\
&=\sum_{j=1}^{2^N}\left|2^{-N/p}\sum_{i=1}^N\varepsilon_i\omega_i^{(j)}\right|^p\\
&=2^{-N}\sum_{j=1}^{2^N}\left|\sum_{i=1}^N\varepsilon_i\omega_i^{(j)}\right|^p\\
&=\mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^N\varepsilon_i\omega_i\right|^p\right],
\end{aligned} ∥ S N ∥ p = ∥ S N ∥ S p p = j = 1 ∑ 2 N ∣ ( S N ) j j ∣ p = j = 1 ∑ 2 N 2 − N / p i = 1 ∑ N ε i ω i ( j ) p = 2 − N j = 1 ∑ 2 N i = 1 ∑ N ε i ω i ( j ) p = E ω ∼ Unif ( Ω ) [ i = 1 ∑ N ε i ω i p ] ,
where the last equality holds because { ω ( j ) } j = 1 N \{\omega^{(j)}\}_{j=1}^{N} { ω ( j ) } j = 1 N is an enumeration of Ω \Omega Ω .
Since ω ∼ Unif ( Ω ) \omega\sim\operatorname{Unif}(\Omega) ω ∼ Unif ( Ω ) is uniform on { ± 1 } N \{\pm1\}^{N} { ± 1 } N , for every fixed realization of ε = ( ε 1 , … , ε N ) ∈ { ± 1 } N \varepsilon=(\varepsilon_1,\ldots,\varepsilon_N)\in\{\pm1\}^{N} ε = ( ε 1 , … , ε N ) ∈ { ± 1 } N we have
E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 N ε i ω i ∣ p ] = E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 N ω i ∣ p ] . \mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^{N}\varepsilon_i\omega_i\right|^p\right]
=
\mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^{N}\omega_i\right|^p\right]. E ω ∼ Unif ( Ω ) [ i = 1 ∑ N ε i ω i p ] = E ω ∼ Unif ( Ω ) [ i = 1 ∑ N ω i p ] .
because the coordinatewise sign-flip map
ω ↦ ( ε 1 ω 1 , … , ε N ω N ) \omega\mapsto(\varepsilon_1\omega_1,\ldots,\varepsilon_N\omega_N) ω ↦ ( ε 1 ω 1 , … , ε N ω N )
is a bijection of Ω \Omega Ω and therefore preserves the uniform law. Consequently,
∥ S N ∥ = ( E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 N ω i ∣ p ] ) 1 / p , \|S_N\|=\left(\mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^{N}\omega_i\right|^p\right]\right)^{1/p}, ∥ S N ∥ = ( E ω ∼ Unif ( Ω ) [ i = 1 ∑ N ω i p ] ) 1/ p ,
so ∥ S N ∥ \|S_N\| ∥ S N ∥ is deterministic. In particular,
E [ ∥ S N ∥ 2 ] = ∥ S N ∥ 2 = ( E ω ∼ Unif ( Ω ) [ ∣ ∑ i = 1 N ω i ∣ p ] ) 2 / p . \mathbb{E}\left[\|S_N\|^2\right]=\|S_N\|^2=\left(\mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}\left[\left|\sum_{i=1}^{N}\omega_i\right|^p\right]\right)^{2/p}. E [ ∥ S N ∥ 2 ] = ∥ S N ∥ 2 = ( E ω ∼ Unif ( Ω ) [ i = 1 ∑ N ω i p ] ) 2/ p .
We now lower bound this quantity using a single atom. Let
Z N : = ∑ i = 1 N ω i , ω ∼ Unif ( Ω ) . Z_N:=\sum_{i=1}^{N}\omega_i,\qquad\omega\sim\operatorname{Unif}(\Omega). Z N := i = 1 ∑ N ω i , ω ∼ Unif ( Ω ) .
Since Prob { Z N = N } = 2 − N \operatorname{Prob}\{Z_N=N\}=2^{-N} Prob { Z N = N } = 2 − N , we obtain
E ω ∼ Unif ( Ω ) [ ∣ Z N ∣ p ] ≥ N p Prob { Z N = N } = N p 2 − N . \mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}[|Z_N|^p]\geq N^p\operatorname{Prob}\{Z_N=N\}=N^p2^{-N}. E ω ∼ Unif ( Ω ) [ ∣ Z N ∣ p ] ≥ N p Prob { Z N = N } = N p 2 − N .
Hence
∥ S N ∥ = ( E ω ∼ Unif ( Ω ) [ ∣ Z N ∣ p ] ) 1 / p ≥ N ⋅ 2 − N / p . \|S_N\|=\left(\mathbb{E}_{\omega\sim\operatorname{Unif}(\Omega)}[|Z_N|^p]\right)^{1/p}\geq N\cdot2^{-N/p}. ∥ S N ∥ = ( E ω ∼ Unif ( Ω ) [ ∣ Z N ∣ p ] ) 1/ p ≥ N ⋅ 2 − N / p .
By construction, N ≤ p N\leq p N ≤ p , so 2 − N / p ≥ 1 / 2 2^{-N/p}\geq1/2 2 − N / p ≥ 1/2 , and thus
E [ ∥ S N ∥ 2 ] = ∥ S N ∥ 2 ≥ N 2 4 . \mathbb{E}[\|S_N\|^2]=\|S_N\|^2\geq\frac{N^2}{4}. E [ ∥ S N ∥ 2 ] = ∥ S N ∥ 2 ≥ 4 N 2 .
Step 4: Conclude via the second-moment bound. Up to this point, all expectations involving ω \omega ω were taken with respect to the auxiliary randomness ω ∼ Unif ( Ω ) \omega\sim\operatorname{Unif}(\Omega) ω ∼ Unif ( Ω ) . We now switch back to the original probability space on which the martingale-difference sequence { ξ i } i = 1 N \{\xi_i\}_{i=1}^{N} { ξ i } i = 1 N is defined. In particular, in the remainder of the proof, E [ ⋅ ] \mathbb{E}[\cdot] E [ ⋅ ] denotes expectation with respect to the randomness of ε \varepsilon ε .
Since ( V , ∥ ⋅ ∥ V ) (V,\|\cdot\|_V) ( V , ∥ ⋅ ∥ V ) is κ \kappa κ -regular and { ξ i } \{\xi_i\} { ξ i } is a martingale-difference sequence, Lemma 15 gives
E [ ∥ S N ∥ 2 ] ≤ κ ∑ i = 1 N E [ ∥ ξ i ∥ 2 ] = κ N . \mathbb{E}[\|S_N\|^2]\leq\kappa\sum_{i=1}^{N}\mathbb{E}[\|\xi_i\|^2]=\kappa N. E [ ∥ S N ∥ 2 ] ≤ κ i = 1 ∑ N E [ ∥ ξ i ∥ 2 ] = κ N .
Combining this with E [ ∥ S N ∥ 2 ] ≥ N 2 / 4 \mathbb{E}[\|S_N\|^2]\geq N^2/4 E [ ∥ S N ∥ 2 ] ≥ N 2 /4 yields
κ ≥ N 4 . \kappa\geq\frac{N}{4}. κ ≥ 4 N .
Finally, since
N = ⌊ min { p , log 2 r } ⌋ , N=\left\lfloor\min\{p,\log_2 r\}\right\rfloor, N = ⌊ min { p , log 2 r } ⌋ ,
we have
N ≥ min { p − 1 , log 2 r − 1 } . N\geq\min\{p-1,\log_2 r-1\}. N ≥ min { p − 1 , log 2 r − 1 } .
Since r ≥ 8 r\geq8 r ≥ 8 , we have log 2 r ≥ 3 \log_2 r\geq3 log 2 r ≥ 3 , and therefore
log 2 r − 1 ≥ 1 2 log 2 r ≥ 1 2 ln r . \log_2 r-1\geq\frac{1}{2}\log_2 r\geq\frac{1}{2}\ln r. log 2 r − 1 ≥ 2 1 log 2 r ≥ 2 1 ln r .
Hence
κ ≥ N 4 ≥ 1 8 min { p − 1 , ln r } , \kappa\geq\frac{N}{4}\geq\frac{1}{8}\min\{p-1,\ln r\}, κ ≥ 4 N ≥ 8 1 min { p − 1 , ln r } ,
which proves (109) for 2 ≤ p < ∞ 2\leq p<\infty 2 ≤ p < ∞ .
Subcase p = ∞ p=\infty p = ∞ . Let
N : = ⌊ log 2 r ⌋ , Ω : = { ± 1 } N , ∣ Ω ∣ = 2 N . N:=\left\lfloor\log_2 r\right\rfloor,\qquad\Omega:=\{\pm1\}^N,\qquad|\Omega|=2^N. N := ⌊ log 2 r ⌋ , Ω := { ± 1 } N , ∣Ω∣ = 2 N .
Fix the same enumeration
Ω = { ω ( 1 ) , … , ω ( 2 N ) } , ω ( j ) = ( ω 1 ( j ) , … , ω N ( j ) ) ∈ { ± 1 } N . \Omega=\{\omega^{(1)},\ldots,\omega^{(2^N)}\},\qquad\omega^{(j)}=(\omega_1^{(j)},\ldots,\omega_N^{(j)})\in\{\pm1\}^N. Ω = { ω ( 1 ) , … , ω ( 2 N ) } , ω ( j ) = ( ω 1 ( j ) , … , ω N ( j ) ) ∈ { ± 1 } N .
Define diagonal matrices U i ∈ R m × n U_i\in\mathbb{R}^{m\times n} U i ∈ R m × n by
( U i ) a b : = 0 , for a ≠ b , (U_i)_{ab}:=0,\qquad\text{for }a\ne b, ( U i ) ab := 0 , for a = b ,
and
( U i ) j j : = ω i ( j ) , j = 1 , … , 2 N , (U_i)_{jj}:=\omega_i^{(j)},\qquad j=1,\ldots,2^N, ( U i ) j j := ω i ( j ) , j = 1 , … , 2 N ,
while
( U i ) j j : = 0 , j = 2 N + 1 , … , r . (U_i)_{jj}:=0,\qquad j=2^N+1,\ldots,r. ( U i ) j j := 0 , j = 2 N + 1 , … , r .
Then every nonzero diagonal entry of U i U_i U i equals ± 1 \pm1 ± 1 , hence
∥ U i ∥ = ∥ U i ∥ S ∞ = 1 , i = 1 , … , N . \lVert U_i\rVert=\lVert U_i\rVert_{S_\infty}=1,\qquad i=1,\ldots,N. ∥ U i ∥ = ∥ U i ∥ S ∞ = 1 , i = 1 , … , N .
With
F i : = σ ( ε 1 , … , ε i ) , ξ i : = ε i U i , i = 1 , … , N , \mathcal{F}_i:=\sigma(\varepsilon_1,\ldots,\varepsilon_i),\qquad\xi_i:=\varepsilon_iU_i,\qquad i=1,\ldots,N, F i := σ ( ε 1 , … , ε i ) , ξ i := ε i U i , i = 1 , … , N ,
we again obtain an { F i } \{\mathcal{F}_i\} { F i } -martingale-difference sequence satisfying
∑ i = 1 N E [ ∥ ξ i ∥ 2 ] = ∑ i = 1 N E [ ∥ ξ i ∥ S ∞ 2 ] = N . \sum_{i=1}^{N}\mathbb{E}\left[\lVert\xi_i\rVert^2\right]=\sum_{i=1}^{N}\mathbb{E}\left[\lVert\xi_i\rVert_{S_\infty}^2\right]=N. i = 1 ∑ N E [ ∥ ξ i ∥ 2 ] = i = 1 ∑ N E [ ∥ ξ i ∥ S ∞ 2 ] = N .
For
S N : = ∑ i = 1 N ξ i , S_N:=\sum_{i=1}^{N}\xi_i, S N := i = 1 ∑ N ξ i ,
we have, for each j ∈ { 1 , … , 2 N } j\in\{1,\ldots,2^N\} j ∈ { 1 , … , 2 N } ,
( S N ) j j = ∑ i = 1 N ε i ( U i ) j j = ∑ i = 1 N ε i ω i ( j ) . (S_N)_{jj}=\sum_{i=1}^{N}\varepsilon_i(U_i)_{jj}=\sum_{i=1}^{N}\varepsilon_i\omega_i^{(j)}. ( S N ) j j = i = 1 ∑ N ε i ( U i ) j j = i = 1 ∑ N ε i ω i ( j ) .
Since S N S_N S N is diagonal,
∥ S N ∥ = ∥ S N ∥ S ∞ = max 1 ≤ j ≤ 2 N ∣ ( S N ) j j ∣ = max ω ∈ Ω ∣ ∑ i = 1 N ε i ω i ∣ . \lVert S_N\rVert=\lVert S_N\rVert_{S_\infty}=\max_{1\le j\le2^N}|(S_N)_{jj}|=\max_{\omega\in\Omega}\left|\sum_{i=1}^{N}\varepsilon_i\omega_i\right|. ∥ S N ∥ = ∥ S N ∥ S ∞ = 1 ≤ j ≤ 2 N max ∣ ( S N ) j j ∣ = ω ∈ Ω max i = 1 ∑ N ε i ω i .
For any fixed realization of ε ∈ { ± 1 } N \varepsilon\in\{\pm1\}^N ε ∈ { ± 1 } N , choosing ω = ε \omega=\varepsilon ω = ε gives
∑ i = 1 N ε i ω i = ∑ i = 1 N ε i 2 = N , \sum_{i=1}^{N}\varepsilon_i\omega_i=\sum_{i=1}^{N}\varepsilon_i^2=N, i = 1 ∑ N ε i ω i = i = 1 ∑ N ε i 2 = N ,
so ∥ S N ∥ ≥ N \lVert S_N\rVert\ge N ∥ S N ∥ ≥ N . On the other hand, for every ω ∈ Ω \omega\in\Omega ω ∈ Ω ,
∣ ∑ i = 1 N ε i ω i ∣ ≤ ∑ i = 1 N ∣ ε i ω i ∣ = N , \left|\sum_{i=1}^{N}\varepsilon_i\omega_i\right|\le\sum_{i=1}^{N}|\varepsilon_i\omega_i|=N, i = 1 ∑ N ε i ω i ≤ i = 1 ∑ N ∣ ε i ω i ∣ = N ,
so ∥ S N ∥ ≤ N \lVert S_N\rVert\le N ∥ S N ∥ ≤ N . Therefore,
∥ S N ∥ = N deterministically, and hence E [ ∥ S N ∥ 2 ] = N 2 . \lVert S_N\rVert=N\qquad\text{deterministically, and hence}\qquad\mathbb{E}\left[\lVert S_N\rVert^2\right]=N^2. ∥ S N ∥ = N deterministically, and hence E [ ∥ S N ∥ 2 ] = N 2 .
Applying Lemma 15 gives
N 2 = E [ ∥ S N ∥ 2 ] ≤ κ ∑ i = 1 N E [ ∥ ξ i ∥ 2 ] = κ N . N^2 = \mathbb{E}\left[\lVert S_N\rVert^2\right] \le\kappa\sum_{i=1}^{N} \mathbb{E}\left[\lVert\xi_i\rVert^2\right] = \kappa N. N 2 = E [ ∥ S N ∥ 2 ] ≤ κ i = 1 ∑ N E [ ∥ ξ i ∥ 2 ] = κ N .
so
Since
N = ⌊ log 2 r ⌋ ≥ 1 2 ln r N = \left\lfloor\log_2 r \right\rfloor\ge\frac{1}{2}\ln r N = ⌊ log 2 r ⌋ ≥ 2 1 ln r
for r ≥ 8 r \ge8 r ≥ 8 , this yields
κ ≥ 1 2 ln r , \kappa\ge\frac{1}{2}\ln r, κ ≥ 2 1 ln r ,
which is stronger than (109) in the case p = ∞ p = \infty p = ∞ , since
min { p − 1 , ln r } = ln r when p = ∞ . \min\{p-1,\ln r\} = \ln r \qquad\text{when } p = \infty. min { p − 1 , ln r } = ln r when p = ∞.
Analysis of SGD Under Dual-Norm Light-Tail Assumption
This section develops the analysis under Assumption 3, from minibatch concentration to oracle and wall-clock complexity. In Section M.1, we prove Lemma 1 following the martingale large-deviation approach of Juditsky and Nemirovski [ [40 ] . Regularity provides a smooth auxiliary norm on E ∗ E^* E ∗ in which we control the minibatch noise; comparison with ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ then yields concentration in the induced dual norm.
In Section M.2, we apply this concentration bound conditionally at each iterate and use a union bound to control the minibatch error simultaneously over the T T T iterations. Combining this error bound with descent under non-Euclidean smoothness gives the high-probability convergence guarantee and the oracle complexity. As in Section A and Corollary 1, N ∗ = B T N_* = BT N ∗ = B T counts the stochastic-gradient oracle calls of serial minibatch SGD: B B B calls at each of the T T T iterations. We use O ~ ( ⋅ ) \widetilde{\mathcal{O}}(\cdot) O ( ⋅ ) with the convention of Section A, suppressing logarithmic factors in both T T T and 1 / δ 1/\delta 1/ δ , where δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) is the failure probability.
Finally, Section M.3 combines the same iteration guarantee with a bound on minibatch collection time to establish the wall-clock complexity of Rennala NSGD under fixed worker computation times.
Proof of the Key Concentration Inequality
How κ \kappa κ -regularity enables minibatch concentration. For independent centered noise with finite second moments, the Euclidean second-moment identity
E [ ∥ η ˉ ∥ ∗ 2 ] = 1 B 2 ∑ j = 1 B E [ ∥ η j ∥ ∗ 2 ] \mathbb{E}\left[\left\|\bar{\eta}\right\|_*^2\right] = \frac{1}{B^2}\sum_{j=1}^{B}\mathbb{E}\left[\left\|\eta^j\right\|_*^2\right] E [ ∥ η ˉ ∥ ∗ 2 ] = B 2 1 j = 1 ∑ B E [ η j ∗ 2 ]
follows by expanding the squared norm and cancelling the cross terms. This identity need not hold for a norm not induced by an inner product. Even then, norm equivalence gives O ( 1 / B ) \mathcal{O}(1/B) O ( 1/ B ) second-moment decay in any fixed finite-dimensional space, with a geometry-dependent constant; κ \kappa κ -regularity bounds this constant by κ \kappa κ (Lemma 15).
The high-probability argument additionally controls exponential moments of a potential built from the squared norm. Smoothness gives a quadratic upper bound on this potential (Lemma 25), and conditional centering removes the linear term associated with each martingale increment. However, the ℓ 1 \ell_1 ℓ 1 , ℓ ∞ \ell_\infty ℓ ∞ , nuclear, and spectral norms are generally nonsmooth away from the origin, apart from low-dimensional cases in which they are Euclidean. Their squared norms therefore need not provide the required bound.
κ \kappa κ -regularity resolves this difficulty by supplying a κ + \kappa_+ κ + -smooth auxiliary norm and the comparison (13) . Together, these allow the martingale argument to yield concentration in the original dual norm, with the geometric dependence captured by κ \kappa κ . A suitably scaled Euclidean norm is always admissible as the 1 1 1 -smooth choice, but other auxiliary norms can give significantly smaller regularity parameters. The regimes for dual ℓ p \ell_p ℓ p and Schatten-p p p norms are described in Section 3.4, with orders valid in all positive dimensions summarized in Remark 7. As discussed there, improvement is not uniform across p p p , and a smaller regularity parameter alone does not guarantee a smaller oracle-complexity bound.
We now collect the auxiliary lemmas needed to prove Lemma 1.
Lemma 24 (Fenchel Conjugate of the Restricted Squared Norm). Let ϕ ^ : E → R ∪ { + ∞ } \hat{\phi}: E \to\mathbb{R} \cup\{+\infty\} ϕ ^ : E → R ∪ { + ∞ } be the squared norm function restricted to the primal unit ball B = { x ∈ E ∣ ∥ x ∥ ≤ 1 } \mathbb{B} = \{x \in E \mid\|x\| \le1\} B = { x ∈ E ∣ ∥ x ∥ ≤ 1 } :
ϕ ^ ( x ) = { 1 2 ∥ x ∥ 2 , if ∥ x ∥ ≤ 1 , + ∞ , otherwise . \hat{\phi}(x) =
\begin{cases}
\frac{1}{2}\|x\|^2, & \text{if }\|x\| \le1,\\
+\infty, & \text{otherwise}.
\end{cases} ϕ ^ ( x ) = { 2 1 ∥ x ∥ 2 , + ∞ , if ∥ x ∥ ≤ 1 , otherwise . Its Fenchel conjugate, ( ϕ ^ ) ∗ ( s ) = sup x ∈ E { ⟨ s , x ⟩ − ϕ ^ ( x ) } (\hat{\phi})^*(s) = \sup_{x\in E}\{\langle s,x\rangle- \hat{\phi}(x)\} ( ϕ ^ ) ∗ ( s ) = sup x ∈ E {⟨ s , x ⟩ − ϕ ^ ( x )} , is precisely the potential function V ( s ) V(s) V ( s ) :
( ϕ ^ ) ∗ ( s ) = V ( s ) : = { 1 2 ∥ s ∥ ∗ 2 , if ∥ s ∥ ∗ ≤ 1 , ∥ s ∥ ∗ − 1 2 , if ∥ s ∥ ∗ > 1. (\hat{\phi})^*(s) = V(s) :=
\begin{cases}
\frac{1}{2}\|s\|_*^2, & \text{if }\|s\|_* \le1,\\
\|s\|_* - \frac{1}{2}, & \text{if }\|s\|_* > 1.
\end{cases} ( ϕ ^ ) ∗ ( s ) = V ( s ) := { 2 1 ∥ s ∥ ∗ 2 , ∥ s ∥ ∗ − 2 1 , if ∥ s ∥ ∗ ≤ 1 , if ∥ s ∥ ∗ > 1.
Proof. Since E E E is finite-dimensional, the primal unit ball B \mathbb{B} B is compact. The function x ↦ ⟨ s , x ⟩ − 1 2 ∥ x ∥ 2 x \mapsto\langle s,x\rangle- \frac{1}{2}\|x\|^2 x ↦ ⟨ s , x ⟩ − 2 1 ∥ x ∥ 2 is continuous, so
( ϕ ^ ) ∗ ( s ) = max x ∈ B { ⟨ s , x ⟩ − 1 2 ∥ x ∥ 2 } . (\hat{\phi})^*(s) = \max_{x\in\mathbb{B}}\left\{\langle s,x\rangle- \frac{1}{2}\|x\|^2\right\}. ( ϕ ^ ) ∗ ( s ) = x ∈ B max { ⟨ s , x ⟩ − 2 1 ∥ x ∥ 2 } .
If s = 0 s = 0 s = 0 , this maximum is zero, attained at x = 0 x = 0 x = 0 , and it equals max 0 ≤ r ≤ 1 { − r 2 / 2 } = 0 \max_{0 \le r \le1}\{-r^2/2\} = 0 max 0 ≤ r ≤ 1 { − r 2 /2 } = 0 .
Now suppose s ≠ 0 s \ne0 s = 0 . For every x ∈ B x \in\mathbb{B} x ∈ B , the duality inequality gives
⟨ s , x ⟩ − 1 2 ∥ x ∥ 2 ≤ ∥ s ∥ ∗ ∥ x ∥ − 1 2 ∥ x ∥ 2 ≤ max 0 ≤ r ≤ 1 { r ∥ s ∥ ∗ − r 2 2 } . \langle s,x\rangle-\frac{1}{2}\|x\|^2 \le\|s\|_*\|x\|-\frac{1}{2}\|x\|^2 \le\max_{0\le r\le1}\left\{r\|s\|_*-\frac{r^2}{2}\right\}. ⟨ s , x ⟩ − 2 1 ∥ x ∥ 2 ≤ ∥ s ∥ ∗ ∥ x ∥ − 2 1 ∥ x ∥ 2 ≤ 0 ≤ r ≤ 1 max { r ∥ s ∥ ∗ − 2 r 2 } .
The continuous linear functional u ↦ ⟨ s , u ⟩ u \mapsto\langle s,u\rangle u ↦ ⟨ s , u ⟩ attains its maximum ∥ s ∥ ∗ > 0 \|s\|_* > 0 ∥ s ∥ ∗ > 0 on B \mathbb{B} B . Any maximizer u u u has ∥ u ∥ = 1 \|u\| = 1 ∥ u ∥ = 1 : otherwise, rescaling u u u to the unit sphere would increase the pairing and contradict maximality. For every r ∈ [ 0 , 1 ] r \in[0,1] r ∈ [ 0 , 1 ] , the point x = r u x = ru x = r u therefore belongs to B \mathbb{B} B and satisfies
⟨ s , r u ⟩ − 1 2 ∥ r u ∥ 2 = r ∥ s ∥ ∗ − r 2 2 . \langle s,ru\rangle-\frac{1}{2}\|ru\|^2=r\|s\|_*-\frac{r^2}{2}. ⟨ s , r u ⟩ − 2 1 ∥ r u ∥ 2 = r ∥ s ∥ ∗ − 2 r 2 .
This proves the reverse bound. Together with the case s = 0 s = 0 s = 0 , we obtain
( φ ^ ) ∗ ( s ) = max 0 ≤ r ≤ 1 { r ∥ s ∥ ∗ − r 2 2 } . (\widehat{\varphi})^*(s)=\max_{0\le r\le1}\left\{r\|s\|_*-\frac{r^2}{2}\right\}. ( φ ) ∗ ( s ) = 0 ≤ r ≤ 1 max { r ∥ s ∥ ∗ − 2 r 2 } .
Completing the square shows that the scalar maximum is attained at r = min { ∥ s ∥ ∗ , 1 } r = \min\{\|s\|_*,1\} r = min { ∥ s ∥ ∗ , 1 } . Hence
( φ ^ ) ∗ ( s ) = { 1 2 ∥ s ∥ ∗ 2 , if ∥ s ∥ ∗ ≤ 1 , ∥ s ∥ ∗ − 1 2 , if ∥ s ∥ ∗ > 1 , = V ( s ) . (\widehat{\varphi})^*(s)=
\begin{cases}
\frac{1}{2}\|s\|_*^2, & \text{if }\|s\|_* \le1,\\
\|s\|_*-\frac{1}{2}, & \text{if }\|s\|_* > 1,
\end{cases}
=V(s). ( φ ) ∗ ( s ) = { 2 1 ∥ s ∥ ∗ 2 , ∥ s ∥ ∗ − 2 1 , if ∥ s ∥ ∗ ≤ 1 , if ∥ s ∥ ∗ > 1 , = V ( s ) .
At ∥ s ∥ ∗ = 1 \|s\|_* = 1 ∥ s ∥ ∗ = 1 , both expressions equal 1 2 \frac{1}{2} 2 1 , attained at r = 1 r = 1 r = 1 .
Lemma 25 (Descent Property of the Potential Function V β V_\beta V β ). Let the space ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*,\|\cdot\|_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) be κ \kappa κ -smooth. Define the function V β : E ∗ → R V_\beta:E^* \to\mathbb{R} V β : E ∗ → R as:
V β ( s ) : = β V ( s / β ) , where V ( s ) = { 1 2 ∥ s ∥ ∗ 2 , if ∥ s ∥ ∗ ≤ 1 , ∥ s ∥ ∗ − 1 2 , if ∥ s ∥ ∗ > 1. V_\beta(s):=\beta V(s/\beta), \qquad\text{where } V(s)=
\begin{cases}
\frac{1}{2}\|s\|_*^2, & \text{if }\|s\|_* \le1,\\
\|s\|_*-\frac{1}{2}, & \text{if }\|s\|_* > 1.
\end{cases} V β ( s ) := β V ( s / β ) , where V ( s ) = { 2 1 ∥ s ∥ ∗ 2 , ∥ s ∥ ∗ − 2 1 , if ∥ s ∥ ∗ ≤ 1 , if ∥ s ∥ ∗ > 1. Then for any β > 0 \beta> 0 β > 0 , the potential function V β ( s ) V_\beta(s) V β ( s ) is continuously differentiable and satisfies the inequality:
V β ( s + η ) ≤ V β ( s ) + ⟨ ∇ V β ( s ) , η ⟩ + κ 2 β ∥ η ∥ ∗ 2 , ∀ s , η ∈ E ∗ . V_\beta(s+\eta)\le V_\beta(s)+\langle\nabla V_\beta(s),\eta\rangle+\frac{\kappa}{2\beta}\|\eta\|_*^2,\qquad\forall s,\eta\in E^*. V β ( s + η ) ≤ V β ( s ) + ⟨ ∇ V β ( s ) , η ⟩ + 2 β κ ∥ η ∥ ∗ 2 , ∀ s , η ∈ E ∗ . Proof. The proof strategy is to first establish that the gradient of the base potential function, ∇ V ( s ) \nabla V(s) ∇ V ( s ) , is κ \kappa κ -Lipschitz continuous. The desired descent inequality is then a direct consequence of this property. We adapt the proof technique from the appendix of Juditsky and Nemirovski [2008].
Step 1: Establishing the κ \kappa κ -Lipschitz Continuity of ∇ V ( s ) \nabla V(s) ∇ V ( s ) . Let ϕ ( x ) : = 1 2 ∥ x ∥ 2 \phi(x):=\frac{1}{2}\|x\|^2 ϕ ( x ) := 2 1 ∥ x ∥ 2 , whose Fenchel conjugate is ϕ ∗ ( s ) = 1 2 ∥ s ∥ ∗ 2 \phi^*(s)=\frac{1}{2}\|s\|_*^2 ϕ ∗ ( s ) = 2 1 ∥ s ∥ ∗ 2 . Apply Proposition 9 with V = E ∗ V=E^* V = E ∗ , ∥ ⋅ ∥ V , + = ∥ ⋅ ∥ ∗ \|\cdot\|_{V,+}=\|\cdot\|_* ∥ ⋅ ∥ V , + = ∥ ⋅ ∥ ∗ , and κ V , + = κ \kappa_{V,+}=\kappa κ V , + = κ . Under the canonical identification V ∗ = E ∗ ∗ ≃ E V^*=E^{**}\simeq E V ∗ = E ∗∗ ≃ E , the induced dual norm is ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ . The proposition’s squared-norm function is therefore ϕ ∗ \phi^* ϕ ∗ , and its Fenchel conjugate is ϕ \phi ϕ . By implication (i) ⇒ \Rightarrow ⇒ (vi), the assumed κ \kappa κ -smoothness of ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*,\|\cdot\|_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) makes ϕ \phi ϕ 1 κ \frac{1}{\kappa} κ 1 -strongly convex with respect to ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ .
Define
ϕ ^ ( x ) = { 1 2 ∥ x ∥ 2 , if ∥ x ∥ ≤ 1 , + ∞ , otherwise . \widehat{\phi}(x)=
\begin{cases}
\frac{1}{2}\|x\|^2, & \text{if }\|x\|\le1,\\
+\infty, & \text{otherwise}.
\end{cases} ϕ ( x ) = { 2 1 ∥ x ∥ 2 , + ∞ , if ∥ x ∥ ≤ 1 , otherwise . Since the unit ball B = { x ∈ E : ∥ x ∥ ≤ 1 } \mathbb{B}=\{x\in E:\|x\|\le1\} B = { x ∈ E : ∥ x ∥ ≤ 1 } is closed and convex, ϕ ^ \widehat{\phi} ϕ is proper and closed, and restricting ϕ \phi ϕ to B \mathbb{B} B preserves its 1 κ \frac{1}{\kappa} κ 1 -strong convexity. By Lemma 24 , its Fenchel conjugate is V = ( ϕ ^ ) ∗ V=(\widehat{\phi})^* V = ( ϕ ) ∗ .
The conjugate of a proper closed μ \mu μ -strongly convex function is differentiable, with a 1 μ \frac{1}{\mu} μ 1 -Lipschitz gradient in the corresponding dual norms. Applying this result with μ = 1 κ \mu=\frac{1}{\kappa} μ = κ 1 shows that V V V is continuously differentiable and
∥ ∇ V ( s 1 ) − ∇ V ( s 2 ) ∥ ≤ κ ∥ s 1 − s 2 ∥ ∗ , s 1 , s 2 ∈ E ∗ . \|\nabla V(s_1)-\nabla V(s_2)\|\le\kappa\|s_1-s_2\|_*,\qquad s_1,s_2\in E^*. ∥∇ V ( s 1 ) − ∇ V ( s 2 ) ∥ ≤ κ ∥ s 1 − s 2 ∥ ∗ , s 1 , s 2 ∈ E ∗ .
Step 2: From Lipschitz Gradient to the Descent Inequality. The desired inequality is a standard consequence of a function having a Lipschitz continuous gradient. For any function F F F with a κ \kappa κ -Lipschitz gradient, the descent lemma states:
F ( s + η ) ≤ F ( s ) + ⟨ ∇ F ( s ) , η ⟩ + κ 2 ∥ η ∥ ∗ 2 . F(s+\eta) \le F(s) + \langle\nabla F(s),\eta\rangle+ \frac{\kappa}{2}\lVert\eta\rVert_*^2. F ( s + η ) ≤ F ( s ) + ⟨ ∇ F ( s ) , η ⟩ + 2 κ ∥ η ∥ ∗ 2 .
Applying this directly to our function V ( s ) V(s) V ( s ) , which we have just shown to have a κ \kappa κ -Lipschitz gradient, gives:
V ( s + η ) ≤ V ( s ) + ⟨ ∇ V ( s ) , η ⟩ + κ 2 ∥ η ∥ ∗ 2 . V(s+\eta) \le V(s) + \langle\nabla V(s),\eta\rangle+ \frac{\kappa}{2}\lVert\eta\rVert_*^2. V ( s + η ) ≤ V ( s ) + ⟨ ∇ V ( s ) , η ⟩ + 2 κ ∥ η ∥ ∗ 2 .
This proves the lemma for the base case where β = 1 \beta=1 β = 1 .
Step 3: Scaling the Result for V β V_\beta V β . The final step is to extend the result to the scaled function V β ( s ) = β V ( s / β ) V_\beta(s)=\beta V(s/\beta) V β ( s ) = β V ( s / β ) . Using the chain rule, the gradient is ∇ V β ( s ) = ∇ V ( s / β ) \nabla V_\beta(s)=\nabla V(s/\beta) ∇ V β ( s ) = ∇ V ( s / β ) .
We start with the inequality proven in Step 2, but apply it to the points s / β s/\beta s / β and η / β \eta/\beta η / β :
V ( s / β + η / β ) ≤ V ( s / β ) + ⟨ ∇ V ( s / β ) , η / β ⟩ + κ 2 ∥ η / β ∥ ∗ 2 . V(s/\beta+\eta/\beta) \le V(s/\beta) + \langle\nabla V(s/\beta),\eta/\beta\rangle+ \frac{\kappa}{2}\lVert\eta/\beta\rVert_*^2. V ( s / β + η / β ) ≤ V ( s / β ) + ⟨ ∇ V ( s / β ) , η / β ⟩ + 2 κ ∥ η / β ∥ ∗ 2 .
Multiplying the entire inequality by the positive scalar β \beta β :
β V ( s + η β ) ≤ β V ( s / β ) + β ⟨ ∇ V ( s / β ) , η / β ⟩ + β κ 2 ∥ η / β ∥ ∗ 2 \beta V\left(\frac{s+\eta}{\beta}\right) \le\beta V(s/\beta) + \beta\langle\nabla V(s/\beta),\eta/\beta\rangle+ \beta\frac{\kappa}{2}\lVert\eta/\beta\rVert_*^2 β V ( β s + η ) ≤ β V ( s / β ) + β ⟨ ∇ V ( s / β ) , η / β ⟩ + β 2 κ ∥ η / β ∥ ∗ 2
V β ( s + η ) ≤ V β ( s ) + ⟨ ∇ V ( s / β ) , η ⟩ + κ 2 β ∥ η ∥ ∗ 2 . V_\beta(s+\eta) \le V_\beta(s) + \langle\nabla V(s/\beta),\eta\rangle+ \frac{\kappa}{2\beta}\lVert\eta\rVert_*^2. V β ( s + η ) ≤ V β ( s ) + ⟨ ∇ V ( s / β ) , η ⟩ + 2 β κ ∥ η ∥ ∗ 2 .
Finally, substituting the identity for the gradient, ∇ V ( s / β ) = ∇ V β ( s ) \nabla V(s/\beta)=\nabla V_\beta(s) ∇ V ( s / β ) = ∇ V β ( s ) , we arrive at the desired conclusion:
V β ( s + η ) ≤ V β ( s ) + ⟨ ∇ V β ( s ) , η ⟩ + κ 2 β ∥ η ∥ ∗ 2 . V_\beta(s+\eta) \le V_\beta(s) + \langle\nabla V_\beta(s),\eta\rangle+ \frac{\kappa}{2\beta}\lVert\eta\rVert_*^2. V β ( s + η ) ≤ V β ( s ) + ⟨ ∇ V β ( s ) , η ⟩ + 2 β κ ∥ η ∥ ∗ 2 .
This completes the proof.
Lemma 26 (Scalar Concentration for Sub-Gaussian Martingales) [40 ] . Let { ψ j } j = 1 B \{\psi_j\}_{j=1}^{B} { ψ j } j = 1 B be a sequence of real-valued random variables adapted to a filtration { F j } j = 0 B \{\mathcal{F}_j\}_{j=0}^{B} { F j } j = 0 B . Assume that for constants μ j ≥ 0 \mu_j\ge0 μ j ≥ 0 and ν j > 0 \nu_j>0 ν j > 0 , the following conditions hold almost surely for all j = 1 , … , B j=1,\ldots,B j = 1 , … , B :
(i) E [ ψ j ∣ F j − 1 ] ≤ μ j \mathbb{E}[\psi_j\mid\mathcal{F}_{j-1}]\le\mu_j E [ ψ j ∣ F j − 1 ] ≤ μ j .
(ii) E [ exp ( ψ j 2 ν j 2 ) | F j − 1 ] ≤ exp ( 1 ) \mathbb{E}\left[\exp\left(\frac{\psi_j^2}{\nu_j^2}\right)\middle|\mathcal{F}_{j-1}\right]\le\exp(1) E [ exp ( ν j 2 ψ j 2 ) F j − 1 ] ≤ exp ( 1 ) .
Then for any λ ≥ 0 \lambda\ge0 λ ≥ 0 , the sum Ψ B = ∑ j = 1 B ψ j \Psi_B=\sum_{j=1}^{B}\psi_j Ψ B = ∑ j = 1 B ψ j satisfies:
Prob { Ψ B > ∑ j = 1 B μ j + λ ∑ j = 1 B ν j 2 } ≤ exp ( − λ 2 / 3 ) . \operatorname{Prob}\left\{\Psi_B>\sum_{j=1}^{B}\mu_j+\lambda\sqrt{\sum_{j=1}^{B}\nu_j^2}\right\}\le\exp(-\lambda^2/3). Prob ⎩ ⎨ ⎧ Ψ B > j = 1 ∑ B μ j + λ j = 1 ∑ B ν j 2 ⎭ ⎬ ⎫ ≤ exp ( − λ 2 /3 ) .
Lemma 27 (Lower Bound on the Potential Function). For any β > 0 \beta>0 β > 0 , the following inequality gives a lower bound on the potential V β ( s ) V_\beta(s) V β ( s ) :
∥ s ∥ ∗ ≤ β 2 + V β ( s ) , ∀ s ∈ E ∗ . \lVert s\rVert_* \le\frac{\beta}{2}+V_\beta(s),\qquad\forall s\in E^*. ∥ s ∥ ∗ ≤ 2 β + V β ( s ) , ∀ s ∈ E ∗ . Proof. The proof is most easily done in two steps. First, we prove the inequality for the base function V ( s ) V(s) V ( s ) (i.e., for β = 1 \beta=1 β = 1 ). Second, we use the scaling property of V β ( s ) V_\beta(s) V β ( s ) to generalize the result.
Step 1: Proof for the Base Case (β = 1 \beta= 1 β = 1 ). Our goal is to prove that for any s ∈ E ∗ s \in E^* s ∈ E ∗ , the following holds:
∥ s ∥ ∗ ≤ 1 2 + V ( s ) . \lVert s\rVert_* \le\frac{1}{2} + V(s). ∥ s ∥ ∗ ≤ 2 1 + V ( s ) .
We analyze this by considering the two cases from the piecewise definition of V ( s ) V(s) V ( s ) .
∥ s ∥ ∗ ≤ 1 2 + 1 2 ∥ s ∥ ∗ 2 . \lVert s\rVert_* \le\frac{1}{2} + \frac{1}{2}\lVert s\rVert_*^2. ∥ s ∥ ∗ ≤ 2 1 + 2 1 ∥ s ∥ ∗ 2 .
Let t = ∥ s ∥ ∗ t = \lVert s\rVert_* t = ∥ s ∥ ∗ . The inequality becomes t ≤ 1 2 + 1 2 t 2 t \le\frac{1}{2} + \frac{1}{2}t^2 t ≤ 2 1 + 2 1 t 2 . We can rearrange this into a simple quadratic inequality:
1 2 t 2 − t + 1 2 ≥ 0 \frac{1}{2}t^2 - t + \frac{1}{2} \ge0 2 1 t 2 − t + 2 1 ≥ 0
t 2 − 2 t + 1 ≥ 0 t^2 - 2t + 1 \ge0 t 2 − 2 t + 1 ≥ 0
( t − 1 ) 2 ≥ 0. (t - 1)^2 \ge0. ( t − 1 ) 2 ≥ 0.
A squared real number is always non-negative, so this inequality is always true. Thus, the property holds for this case.
∥ s ∥ ∗ ≤ 1 2 + ( ∥ s ∥ ∗ − 1 2 ) . \lVert s\rVert_* \le\frac{1}{2} + \left(\lVert s\rVert_* - \frac{1}{2}\right). ∥ s ∥ ∗ ≤ 2 1 + ( ∥ s ∥ ∗ − 2 1 ) .
The right-hand side simplifies to ∥ s ∥ ∗ \lVert s\rVert_* ∥ s ∥ ∗ . The inequality becomes:
∥ s ∥ ∗ ≤ ∥ s ∥ ∗ . \lVert s\rVert_* \le\lVert s\rVert_*. ∥ s ∥ ∗ ≤ ∥ s ∥ ∗ .
This holds with equality. Thus, the property also holds for this case.
Since the inequality holds in both cases, we have proven that ∥ s ∥ ∗ ≤ 1 2 + V ( s ) \lVert s\rVert_* \le\frac{1}{2} + V(s) ∥ s ∥ ∗ ≤ 2 1 + V ( s ) for all s ∈ E ∗ s \in E^* s ∈ E ∗ .
Step 2: Scaling the Result to V β ( s ) V_\beta(s) V β ( s ) . We now use the result from Step 1, which is valid for any vector in E ∗ E^* E ∗ . Let us apply it to the specific vector s / β s/\beta s / β :
∥ s / β ∥ ∗ ≤ 1 2 + V ( s / β ) . \lVert s/\beta\rVert_* \le\frac{1}{2} + V(s/\beta). ∥ s / β ∥ ∗ ≤ 2 1 + V ( s / β ) .
By the absolute homogeneity property of norms, ∥ s / β ∥ ∗ = 1 β ∥ s ∥ ∗ \lVert s/\beta\rVert_* = \frac{1}{\beta}\lVert s\rVert_* ∥ s / β ∥ ∗ = β 1 ∥ s ∥ ∗ . Substituting this in gives:
1 β ∥ s ∥ ∗ ≤ 1 2 + V ( s / β ) . \frac{1}{\beta}\lVert s\rVert_* \le\frac{1}{2} + V(s/\beta). β 1 ∥ s ∥ ∗ ≤ 2 1 + V ( s / β ) .
Since β > 0 \beta> 0 β > 0 , we can multiply the entire inequality by β \beta β without changing its direction:
∥ s ∥ ∗ ≤ β 2 + β V ( s / β ) . \lVert s\rVert_* \le\frac{\beta}{2} + \beta V(s/\beta). ∥ s ∥ ∗ ≤ 2 β + β V ( s / β ) .
By definition, V β ( s ) = β V ( s / β ) V_\beta(s) = \beta V(s/\beta) V β ( s ) = β V ( s / β ) . Substituting this into the right-hand side gives the final result:
∥ s ∥ ∗ ≤ β 2 + V β ( s ) . \lVert s\rVert_* \le\frac{\beta}{2} + V_\beta(s). ∥ s ∥ ∗ ≤ 2 β + V β ( s ) .
This completes the proof.
Lemma 1 (Concentration in Regular Normed Spaces). Let κ \kappa κ be a regularity constant of the dual space ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*, \lVert\cdot\rVert_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) . Assume that { η j } j = 1 B \{\eta^j\}_{j=1}^B { η j } j = 1 B is a sequence of i.i.d. random vectors in E ∗ E^* E ∗ such that E [ η j ] = 0 \mathbb{E}[\eta^j] = 0 E [ η j ] = 0 and they satisfy the light-tail condition: E [ exp ( ∥ η j ∥ ∗ 2 σ 2 ) ] ≤ exp ( 1 ) \mathbb{E}\left[\exp\left(\frac{\lVert\eta^j\rVert_*^2}{\sigma^2}\right)\right] \le\exp(1) E [ exp ( σ 2 ∥ η j ∥ ∗ 2 ) ] ≤ exp ( 1 ) . Then for any λ ≥ 0 \lambda\ge0 λ ≥ 0 , the average η ˉ = 1 B ∑ j = 1 B η j \bar{\eta} = \frac{1}{B}\sum_{j=1}^B \eta^j η ˉ = B 1 ∑ j = 1 B η j satisfies the concentration inequality: Prob { ∥ η ˉ ∥ ∗ ≥ ( 2 κ + 2 λ ) σ B } ≤ exp ( − λ 2 / 3 ) \operatorname{Prob}\left\{\lVert\bar{\eta}\rVert_* \ge\left(\sqrt{2\kappa}+\sqrt{2\lambda}\right)\frac{\sigma}{\sqrt{B}}\right\} \le\exp(-\lambda^2/3) Prob { ∥ η ˉ ∥ ∗ ≥ ( 2 κ + 2 λ ) B σ } ≤ exp ( − λ 2 /3 ) .
Proof of Lemma 1. Let κ \kappa κ be the regularity constant specified in the lemma. By Lemma 23, there exists a κ \kappa κ -smooth norm ∥ ⋅ ∥ ∗ , + \lVert\cdot\rVert_{*,+} ∥ ⋅ ∥ ∗ , + on E ∗ E^* E ∗ such that
∥ s ∥ ∗ 2 ≤ ∥ s ∥ ∗ , + 2 ≤ 2 ∥ s ∥ ∗ 2 ∀ s ∈ E ∗ . (110) \lVert s\rVert_*^2 \le\lVert s\rVert_{*,+}^2 \le2\lVert s\rVert_*^2 \qquad\forall s\in E^*. \tag*{(110)} ∥ s ∥ ∗ 2 ≤ ∥ s ∥ ∗ , + 2 ≤ 2 ∥ s ∥ ∗ 2 ∀ s ∈ E ∗ . (110)
Define σ + 2 : = 2 σ 2 \sigma_+^2:=2\sigma^2 σ + 2 := 2 σ 2 . Then for every j j j ,
∥ η j ∥ ∗ , + 2 σ + 2 ≤ 2 ∥ η j ∥ ∗ 2 2 σ 2 = ∥ η j ∥ ∗ 2 σ 2 , \frac{\lVert\eta^j\rVert_{*,+}^2}{\sigma_+^2}\le\frac{2\lVert\eta^j\rVert_*^2}{2\sigma^2}=\frac{\lVert\eta^j\rVert_*^2}{\sigma^2}, σ + 2 ∥ η j ∥ ∗ , + 2 ≤ 2 σ 2 2 ∥ η j ∥ ∗ 2 = σ 2 ∥ η j ∥ ∗ 2 ,
and hence by the assumed light-tail condition,
E [ exp ( ∥ η j ∥ ∗ , + 2 σ + 2 ) ] ≤ E [ exp ( ∥ η j ∥ ∗ 2 σ 2 ) ] ≤ e . (111) \mathbb{E}\left[\exp\left(\frac{\lVert\eta^j\rVert_{*,+}^2}{\sigma_+^2}\right)\right]\le\mathbb{E}\left[\exp\left(\frac{\lVert\eta^j\rVert_*^2}{\sigma^2}\right)\right]\le e. \tag*{(111)} E [ exp ( σ + 2 ∥ η j ∥ ∗ , + 2 ) ] ≤ E [ exp ( σ 2 ∥ η j ∥ ∗ 2 ) ] ≤ e . (111)
Step 1: set up a martingale difference sequence. Let F j : = σ ( η 1 , … , η j ) \mathcal{F}_j:=\sigma(\eta^1,\ldots,\eta^j) F j := σ ( η 1 , … , η j ) and define the partial sums S 0 ≡ 0 S_0\equiv0 S 0 ≡ 0 , S j : = ∑ t = 1 j η t S_j:=\sum_{t=1}^j\eta^t S j := ∑ t = 1 j η t . Then ( S j ) j = 0 B (S_j)_{j=0}^B ( S j ) j = 0 B is a martingale and ( η j ) j = 1 B (\eta^j)_{j=1}^B ( η j ) j = 1 B is a martingale-difference sequence w.r.t. ( F j ) (\mathcal{F}_j) ( F j ) .
Step 2: the potential function. For β > 0 \beta>0 β > 0 , define V β : E ∗ → R V_\beta:E^*\to\mathbb{R} V β : E ∗ → R by
V β ( s ) : = β V ( s / β ) , V ( u ) : = { 1 2 ∥ u ∥ ∗ , + 2 , ∥ u ∥ ∗ , + ≤ 1 , ∥ u ∥ ∗ , + − 1 2 , ∥ u ∥ ∗ , + > 1. V_\beta(s):=\beta V(s/\beta),\qquad V(u):=
\begin{cases}
\frac{1}{2}\lVert u\rVert_{*,+}^2, & \lVert u\rVert_{*,+}\le1,\\
\lVert u\rVert_{*,+}-\frac{1}{2}, & \lVert u\rVert_{*,+}>1.
\end{cases} V β ( s ) := β V ( s / β ) , V ( u ) := { 2 1 ∥ u ∥ ∗ , + 2 , ∥ u ∥ ∗ , + − 2 1 , ∥ u ∥ ∗ , + ≤ 1 , ∥ u ∥ ∗ , + > 1.
We use the following three properties:
(a) V β V_\beta V β is 1-Lipschitz w.r.t. ∥ ⋅ ∥ ∗ , + \lVert\cdot\rVert_{*,+} ∥ ⋅ ∥ ∗ , + . Indeed, write
V β ( s ) = h β ( ∥ s ∥ ∗ , + ) , h β ( r ) = { r 2 2 β , 0 ≤ r ≤ β , r − β 2 , r > β . V_\beta(s)=h_\beta(\lVert s\rVert_{*,+}),\qquad h_\beta(r)=
\begin{cases}
\frac{r^2}{2\beta}, & 0\le r\le\beta,\\
r-\frac{\beta}{2}, & r>\beta.
\end{cases} V β ( s ) = h β (∥ s ∥ ∗ , + ) , h β ( r ) = { 2 β r 2 , r − 2 β , 0 ≤ r ≤ β , r > β .
The two branches have matching derivatives at r = β r=\beta r = β , and h β ′ ( r ) = min { r / β , 1 } ∈ [ 0 , 1 ] h_\beta'(r)=\min\{r/\beta,1\}\in[0,1] h β ′ ( r ) = min { r / β , 1 } ∈ [ 0 , 1 ] for r > 0 r>0 r > 0 . Thus h β h_\beta h β is 1-Lipschitz on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) . Combining this with the reverse triangle inequality gives
∣ V β ( s + η ) − V β ( s ) ∣ ≤ ∣ ∥ s + η ∥ ∗ , + − ∥ s ∥ ∗ , + ∣ ≤ ∥ η ∥ ∗ , + ∀ s , η ∈ E ∗ . (112) \left|V_\beta(s+\eta)-V_\beta(s)\right|\le\left|\lVert s+\eta\rVert_{*,+}-\lVert s\rVert_{*,+}\right|\le\lVert\eta\rVert_{*,+}\qquad\forall s,\eta\in E^*. \tag*{(112)} ∣ V β ( s + η ) − V β ( s ) ∣ ≤ ∣ ∥ s + η ∥ ∗ , + − ∥ s ∥ ∗ , + ∣ ≤ ∥ η ∥ ∗ , + ∀ s , η ∈ E ∗ . (112)
(b) By Lemma 25, applied to the κ \kappa κ -smooth space ( E ∗ , ∥ ⋅ ∥ ∗ , + ) (E^*,\lVert\cdot\rVert_{*,+}) ( E ∗ , ∥ ⋅ ∥ ∗ , + ) , V β V_\beta V β is continuously differentiable and satisfies the descent inequality
V β ( s + η ) ≤ V β ( s ) + ⟨ ∇ V β ( s ) , η ⟩ + κ 2 β ∥ η ∥ ∗ , + 2 ∀ s , η ∈ E ∗ . (113) V_\beta(s+\eta)\le V_\beta(s)+\langle\nabla V_\beta(s),\eta\rangle+\frac{\kappa}{2\beta}\lVert\eta\rVert_{*,+}^2\qquad\forall s,\eta\in E^*. \tag*{(113)} V β ( s + η ) ≤ V β ( s ) + ⟨ ∇ V β ( s ) , η ⟩ + 2 β κ ∥ η ∥ ∗ , + 2 ∀ s , η ∈ E ∗ . (113)
(c) By Lemma 27, applied to the same auxiliary norm, for all s ∈ E ∗ s\in E^* s ∈ E ∗ ,
V β ( s ) ≥ ∥ s ∥ ∗ , + − β 2 V_\beta(s)\ge\lVert s\rVert_{*,+}-\frac{\beta}{2} V β ( s ) ≥ ∥ s ∥ ∗ , + − 2 β
equivalently
∥ s ∥ ∗ , + ≤ V β ( s ) + β 2 . (114) \lVert s\rVert_{*,+}\le V_\beta(s)+\frac{\beta}{2}. \tag*{(114)} ∥ s ∥ ∗ , + ≤ V β ( s ) + 2 β . (114)
Step 3: define increments and verify scalar conditions. Define
ψ j : = V β ( S j ) − V β ( S j − 1 ) , j = 1 , … , B . \psi_j:=V_\beta(S_j)-V_\beta(S_{j-1}),\qquad j=1,\ldots,B. ψ j := V β ( S j ) − V β ( S j − 1 ) , j = 1 , … , B .
First, (111) and Jensen’s inequality imply the second moment bound:
E [ ∥ η j ∥ ∗ , + 2 σ + 2 ] ≤ log E [ exp ( ∥ η j ∥ ∗ , + 2 σ + 2 ) ] ≤ 1 ⟹ E [ ∥ η j ∥ ∗ , + 2 ] ≤ σ + 2 . (115) \mathbb{E}\left[\frac{\lVert\eta^j\rVert_{*,+}^2}{\sigma_+^2}\right]\le\log\mathbb{E}\left[\exp\left(\frac{\lVert\eta^j\rVert_{*,+}^2}{\sigma_+^2}\right)\right]\le1\quad\Longrightarrow\quad\mathbb{E}\left[\lVert\eta^j\rVert_{*,+}^2\right]\le\sigma_+^2. \tag*{(115)} E [ σ + 2 ∥ η j ∥ ∗ , + 2 ] ≤ log E [ exp ( σ + 2 ∥ η j ∥ ∗ , + 2 ) ] ≤ 1 ⟹ E [ ∥ η j ∥ ∗ , + 2 ] ≤ σ + 2 . (115)
Since S j S_j S j is F j \mathcal{F}_j F j -measurable and V β V_\beta V β is continuous, ψ j \psi_j ψ j is F j \mathcal{F}_j F j -measurable. The continuity of ∇ V β \nabla V_\beta ∇ V β also makes ∇ V β ( S j − 1 ) \nabla V_\beta(S_{j-1}) ∇ V β ( S j − 1 ) F j − 1 \mathcal{F}_{j-1} F j − 1 -measurable. Differentiability and (112) give, for every s , η ∈ E ∗ s,\eta\in E^* s , η ∈ E ∗ ,
∣ ⟨ ∇ V β ( s ) , η ⟩ ∣ = lim t → 0 ∣ V β ( s + t η ) − V β ( s ) ∣ ∣ t ∣ ≤ ∥ η ∥ ∗ , + . \left|\langle\nabla V_\beta(s),\eta\rangle\right|=\lim_{t\to0}\frac{\left|V_\beta(s+t\eta)-V_\beta(s)\right|}{|t|}\le\lVert\eta\rVert_{*,+}. ∣ ⟨ ∇ V β ( s ) , η ⟩ ∣ = t → 0 lim ∣ t ∣ ∣ V β ( s + t η ) − V β ( s ) ∣ ≤ ∥ η ∥ ∗ , + .
Equivalently, ∇ V β ( s ) \nabla V_{\beta}(s) ∇ V β ( s ) has norm at most 1 in the norm dual to ∥ ⋅ ∥ ∗ , + \|\cdot\|_{*,+} ∥ ⋅ ∥ ∗ , + . Consequently, by Cauchy–Schwarz and (115),
E [ ∣ ⟨ ∇ V β ( S j − 1 ) , η j ⟩ ∣ ] ≤ E [ ∥ η j ∥ ∗ , + ] ≤ E [ ∥ η j ∥ ∗ , + 2 ] ≤ σ + < ∞ . \mathbb{E}\left[\left|\left\langle\nabla V_{\beta}(S_{j-1}),\eta^j\right\rangle\right|\right]\leq\mathbb{E}\left[\|\eta^j\|_{*,+}\right]\leq\sqrt{\mathbb{E}\left[\|\eta^j\|_{*,+}^2\right]}\leq\sigma_+<\infty. E [ ⟨ ∇ V β ( S j − 1 ) , η j ⟩ ] ≤ E [ ∥ η j ∥ ∗ , + ] ≤ E [ ∥ η j ∥ ∗ , + 2 ] ≤ σ + < ∞.
Also, (112) implies E [ ∣ ψ j ∣ ] ≤ E [ ∥ η j ∥ ∗ , + ] ≤ σ + \mathbb{E}[|\psi_j|]\leq\mathbb{E}\left[\|\eta^j\|_{*,+}\right]\leq\sigma_+ E [ ∣ ψ j ∣ ] ≤ E [ ∥ η j ∥ ∗ , + ] ≤ σ + . Thus the increment, the mixed term, and the squared-noise term are all integrable. We may therefore take conditional expectations in (113) and pull out the F j − 1 \mathcal{F}_{j-1} F j − 1 -measurable gradient. Since η j \eta^j η j is independent of F j − 1 \mathcal{F}_{j-1} F j − 1 and has mean zero, the mixed term has conditional mean zero, and we obtain
E [ ψ j ∣ F j − 1 ] = E [ V β ( S j − 1 + η j ) − V β ( S j − 1 ) ∣ F j − 1 ] ≤ E [ ⟨ ∇ V β ( S j − 1 ) , η j ⟩ + κ 2 β ∥ η j ∥ ∗ , + 2 ∣ F j − 1 ] = κ 2 β E [ ∥ η j ∥ ∗ , + 2 ] . \begin{aligned}
\mathbb{E}[\psi_j\mid\mathcal{F}_{j-1}]=\mathbb{E}\left[V_{\beta}(S_{j-1}+\eta^j)-V_{\beta}(S_{j-1})\mid\mathcal{F}_{j-1}\right] \\
&\leq\mathbb{E}\left[\left\langle\nabla V_{\beta}(S_{j-1}),\eta^j\right\rangle+\frac{\kappa}{2\beta}\|\eta^j\|_{*,+}^2\mid\mathcal{F}_{j-1}\right] \\
&=\frac{\kappa}{2\beta}\mathbb{E}\left[\|\eta^j\|_{*,+}^2\right].
\end{aligned} E [ ψ j ∣ F j − 1 ] = E [ V β ( S j − 1 + η j ) − V β ( S j − 1 ) ∣ F j − 1 ] ≤ E [ ⟨ ∇ V β ( S j − 1 ) , η j ⟩ + 2 β κ ∥ η j ∥ ∗ , + 2 ∣ F j − 1 ] = 2 β κ E [ ∥ η j ∥ ∗ , + 2 ] .
Therefore,
E [ ψ j ∣ F j − 1 ] ≤ κ σ + 2 2 β for all j . (116) \mathbb{E}[\psi_j\mid\mathcal{F}_{j-1}]\leq\frac{\kappa\sigma_+^2}{2\beta}\qquad\text{for all }j. \tag*{(116)} E [ ψ j ∣ F j − 1 ] ≤ 2 β κ σ + 2 for all j . (116)
Also, by (112) and (111),
E [ exp ( ψ j 2 σ + 2 ) ∣ F j − 1 ] ≤ E [ exp ( ∥ η j ∥ ∗ , + 2 σ + 2 ) ] ≤ e . (117) \mathbb{E}\left[\exp\left(\frac{\psi_j^2}{\sigma_+^2}\right)\mid\mathcal{F}_{j-1}\right]\leq\mathbb{E}\left[\exp\left(\frac{\|\eta^j\|_{*,+}^2}{\sigma_+^2}\right)\right]\leq e. \tag*{(117)} E [ exp ( σ + 2 ψ j 2 ) ∣ F j − 1 ] ≤ E [ exp ( σ + 2 ∥ η j ∥ ∗ , + 2 ) ] ≤ e . (117)
Step 4: apply scalar concentration to ∑ j = 1 B ψ j = V β ( S B ) \sum_{j=1}^{B}\psi_j=V_{\beta}(S_B) ∑ j = 1 B ψ j = V β ( S B ) . Although Lemma 26 is stated with a strict tail event, the same bound holds with ≥ \geq ≥ . For λ > 0 \lambda>0 λ > 0 , apply the lemma with 0 ≤ λ n < λ 0\leq\lambda_n<\lambda 0 ≤ λ n < λ and λ n ↑ λ \lambda_n\uparrow\lambda λ n ↑ λ . The strict tail events decrease to the non-strict event at λ \lambda λ , so continuity of probability from above and exp ( − λ n 2 / 3 ) → exp ( − λ 2 / 3 ) \exp(-\lambda_n^2/3)\to\exp(-\lambda^2/3) exp ( − λ n 2 /3 ) → exp ( − λ 2 /3 ) yield the claim. For λ = 0 \lambda=0 λ = 0 , the bound is trivial because its right-hand side is 1. No continuity assumption on the distribution is needed.
Applying Lemma 26 (the sub-Gaussian case) with μ j = κ σ + 2 / 2 β \mu_j=\kappa\sigma_+^2/2\beta μ j = κ σ + 2 /2 β and ν j = σ + \nu_j=\sigma_+ ν j = σ + yields that for all λ ≥ 0 \lambda\geq0 λ ≥ 0 ,
Prob { V β ( S B ) ≥ B κ σ + 2 2 β + λ B σ + } ≤ exp ( − λ 2 / 3 ) . \operatorname{Prob}\left\{V_{\beta}(S_B)\geq\frac{B\kappa\sigma_+^2}{2\beta}+\lambda\sqrt{B}\sigma_+\right\}\leq\exp(-\lambda^2/3). Prob { V β ( S B ) ≥ 2 β B κ σ + 2 + λ B σ + } ≤ exp ( − λ 2 /3 ) .
Using (114), we obtain
Prob { ∥ S B ∥ ∗ , + ≥ β 2 + B κ σ + 2 2 β + λ B σ + } ≤ exp ( − λ 2 / 3 ) . \operatorname{Prob}\left\{\|S_B\|_{*,+}\geq\frac{\beta}{2}+\frac{B\kappa\sigma_+^2}{2\beta}+\lambda\sqrt{B}\sigma_+\right\}\leq\exp(-\lambda^2/3). Prob { ∥ S B ∥ ∗ , + ≥ 2 β + 2 β B κ σ + 2 + λ B σ + } ≤ exp ( − λ 2 /3 ) .
Optimizing the deterministic term in β \beta β gives β ∗ = B κ σ + \beta^*=\sqrt{B\kappa}\,\sigma_+ β ∗ = B κ σ + and thus
Prob { ∥ S B ∥ ∗ , + ≥ ( κ + λ ) B σ + } ≤ exp ( − λ 2 / 3 ) . \operatorname{Prob}\left\{\|S_B\|_{*,+}\geq(\sqrt{\kappa}+\lambda)\sqrt{B}\sigma_+\right\}\leq\exp(-\lambda^2/3). Prob { ∥ S B ∥ ∗ , + ≥ ( κ + λ ) B σ + } ≤ exp ( − λ 2 /3 ) .
Step 5: return to ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ and to the average. Since ∥ S B ∥ ∗ ≤ ∥ S B ∥ ∗ , + \|S_B\|_*\leq\|S_B\|_{*,+} ∥ S B ∥ ∗ ≤ ∥ S B ∥ ∗ , + and σ + = 2 σ \sigma_+=\sqrt{2}\sigma σ + = 2 σ ,
Prob { ∥ S B ∥ ∗ ≥ ( 2 κ + 2 λ ) B σ } ≤ exp ( − λ 2 / 3 ) . \operatorname{Prob}\left\{\|S_B\|_*\geq(\sqrt{2\kappa}+\sqrt{2}\lambda)\sqrt{B}\sigma\right\}\leq\exp(-\lambda^2/3). Prob { ∥ S B ∥ ∗ ≥ ( 2 κ + 2 λ ) B σ } ≤ exp ( − λ 2 /3 ) .
Finally, η ˉ = S B / B \bar{\eta}=S_B/B η ˉ = S B / B and norms are homogeneous, so ∥ η ˉ ∥ ∗ = ∥ S B ∥ ∗ / B \|\bar{\eta}\|_*=\|S_B\|_*/B ∥ η ˉ ∥ ∗ = ∥ S B ∥ ∗ / B . Dividing the threshold by B B B yields
Prob { ∥ η ˉ ∥ ∗ ≥ ( 2 κ + 2 λ ) σ B } ≤ exp ( − λ 2 / 3 ) . \operatorname{Prob}\left\{\|\bar{\eta}\|_*\geq(\sqrt{2\kappa}+\sqrt{2}\lambda)\frac{\sigma}{\sqrt{B}}\right\}\leq\exp(-\lambda^2/3). Prob { ∥ η ˉ ∥ ∗ ≥ ( 2 κ + 2 λ ) B σ } ≤ exp ( − λ 2 /3 ) .
This completes the proof.
Proof of the Convergence Rate Theorem
Theorem 1 (Minibatch non-Euclidean SGD under light tails). Let κ \kappa κ be a regularity constant of the dual space ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*, \lVert\cdot\rVert_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) . Suppose Assumptions 1, 2, and 3 hold. Run the update (7) with the constant stepsize γ k ≡ 1 2 L \gamma_k \equiv\frac{1}{2L} γ k ≡ 2 L 1 for T = ⌈ 128 L Δ 15 ε ⌉ T = \left\lceil\frac{128L\Delta}{15\varepsilon}\right\rceil T = ⌈ 15 ε 128 L Δ ⌉ iterations, with a minibatch size chosen as
B = max { ⌈ 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 ε ⌉ , 1 } . B = \max\left\{\left\lceil\frac{16\sigma^2\left(\sqrt{2\kappa}+\sqrt{6\log(T/\delta)}\right)^2}{\varepsilon}\right\rceil,1\right\}. B = max ⎩ ⎨ ⎧ ε 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 , 1 ⎭ ⎬ ⎫ . Then, with probability at least 1 − δ 1-\delta 1 − δ , the iterates satisfy min 0 ≤ k ≤ T − 1 ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε \min_{0\leq k\leq T-1}\lVert\nabla f(x^k)\rVert_*^2\leq\varepsilon min 0 ≤ k ≤ T − 1 ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε .
Proof of Theorem 1. Fix the initial point x 0 x^0 x 0 and define the filtration
F k : = σ ( x 0 , ξ t j : 0 ≤ t < k , 1 ≤ j ≤ B ) , k ≥ 0. \mathcal{F}_k := \sigma\left(x^0,\xi_t^j:0\leq t<k,\ 1\leq j\leq B\right),\qquad k\geq0. F k := σ ( x 0 , ξ t j : 0 ≤ t < k , 1 ≤ j ≤ B ) , k ≥ 0.
By induction from the update (7) and the fixed Borel-measurable sharp selection in § 3.1 , x k x^k x k is F k \mathcal{F}_k F k -measurable. As specified in the serial algorithm, the new batch { ξ j k } j = 1 B \{\xi_j^k\}_{j=1}^B { ξ j k } j = 1 B consists of i.i.d. samples from D \mathcal{D} D and is independent of F k \mathcal{F}_k F k . Let η j k : = ∇ f ( x k ; ξ j k ) − ∇ f ( x k ) \eta_j^k:=\nabla f(x^k;\xi_j^k)-\nabla f(x^k) η j k := ∇ f ( x k ; ξ j k ) − ∇ f ( x k ) be the noise of the j j j -th sample at iteration k k k . Conditionally on F k \mathcal{F}_k F k , the query x k x^k x k is fixed, so these noise vectors are independent and identically distributed. By Assumption 3, they satisfy, almost surely,
E [ η j k ∣ F k ] = 0 , E [ exp ( ∥ η j k ∥ ∗ 2 σ 2 ) | F k ] ≤ e . \mathbb{E}\left[\eta_j^k\mid\mathcal{F}_k\right]=0,\qquad
\mathbb{E}\left[\exp\left(\frac{\lVert\eta_j^k\rVert_*^2}{\sigma^2}\right)\middle|\mathcal{F}_k\right]\leq e. E [ η j k ∣ F k ] = 0 , E [ exp ( σ 2 ∥ η j k ∥ ∗ 2 ) F k ] ≤ e .
The minibatch error is 1 B ∑ j = 1 B η j k = g k − ∇ f ( x k ) \frac{1}{B}\sum_{j=1}^B\eta_j^k=g^k-\nabla f(x^k) B 1 ∑ j = 1 B η j k = g k − ∇ f ( x k ) . Thus Lemma 1 applies conditionally on F k \mathcal{F}_k F k and gives, for each k k k and λ ≥ 0 \lambda\geq0 λ ≥ 0 , almost surely,
P r o b { ∥ g k − ∇ f ( x k ) ∥ ∗ ≥ ( 2 κ + 2 λ ) σ B ∣ F k } ≤ exp ( − λ 2 / 3 ) . \mathop{\mathrm{Prob}}\left\{\left.\lVert g^k-\nabla f(x^k)\rVert_*\geq(\sqrt{2\kappa}+\sqrt{2}\lambda)\frac{\sigma}{\sqrt{B}}\right|\mathcal{F}_k\right\}\leq\exp(-\lambda^2/3). Prob { ∥ g k − ∇ f ( x k ) ∥ ∗ ≥ ( 2 κ + 2 λ ) B σ F k } ≤ exp ( − λ 2 /3 ) .
Let A k ( λ ) A_k(\lambda) A k ( λ ) denote the event in this conditional probability. Taking expectations and using the tower property yields
P r o b { A k ( λ ) } = E [ P r o b { A k ( λ ) ∣ F k } ] ≤ exp ( − λ 2 / 3 ) . \mathop{\mathrm{Prob}}\{A_k(\lambda)\}=\mathbb{E}\left[\mathop{\mathrm{Prob}}\{A_k(\lambda)\mid\mathcal{F}_k\}\right]\leq\exp(-\lambda^2/3). Prob { A k ( λ )} = E [ Prob { A k ( λ ) ∣ F k } ] ≤ exp ( − λ 2 /3 ) .
We may now apply the unconditional union bound over the prescribed iterations k ∈ { 0 , … , T − 1 } k\in\{0,\ldots,T-1\} k ∈ { 0 , … , T − 1 } ; independence of the events A k ( λ ) A_k(\lambda) A k ( λ ) is not needed:
P r o b { ⋃ k = 0 T − 1 A k ( λ ) } ≤ ∑ k = 0 T − 1 P r o b { A k ( λ ) } ≤ ∑ k = 0 T − 1 exp ( − λ 2 / 3 ) = T exp ( − λ 2 / 3 ) . \mathop{\mathrm{Prob}}\left\{\bigcup_{k=0}^{T-1}A_k(\lambda)\right\}\leq\sum_{k=0}^{T-1}\mathop{\mathrm{Prob}}\{A_k(\lambda)\}\leq\sum_{k=0}^{T-1}\exp(-\lambda^2/3)=T\exp(-\lambda^2/3). Prob { k = 0 ⋃ T − 1 A k ( λ ) } ≤ k = 0 ∑ T − 1 Prob { A k ( λ )} ≤ k = 0 ∑ T − 1 exp ( − λ 2 /3 ) = T exp ( − λ 2 /3 ) .
We want this failure probability to be at most δ \delta δ . By setting T exp ( − λ 2 / 3 ) ≤ δ T\exp(-\lambda^2/3)\leq\delta T exp ( − λ 2 /3 ) ≤ δ , we find that choosing λ = 3 log ( T / δ ) \lambda=\sqrt{3\log(T/\delta)} λ = 3 log ( T / δ ) suffices. With this choice, we can conclude that with probability at least 1 − δ 1-\delta 1 − δ , the following holds for all k ∈ { 0 , … , T − 1 } k\in\{0,\ldots,T-1\} k ∈ { 0 , … , T − 1 } :
∥ g k − ∇ f ( x k ) ∥ ∗ ≤ ( 2 κ + 2 λ ) σ B . (118) \lVert g^k-\nabla f(x^k)\rVert_*\leq\left(\sqrt{2\kappa}+\sqrt{2\lambda}\right)\frac{\sigma}{\sqrt{B}}. \tag*{(118)} ∥ g k − ∇ f ( x k ) ∥ ∗ ≤ ( 2 κ + 2 λ ) B σ . (118)
We choose the error threshold ε 4 \frac{\sqrt{\varepsilon}}{4} 4 ε and select the batch size B B B so that the right-hand side of (118) is at most this threshold. Thus, we require:
( 2 κ + 2 3 log ( T / δ ) ) σ B ≤ ε 4 . \left(\sqrt{2\kappa}+\sqrt{2}\sqrt{3\log(T/\delta)}\right)\frac{\sigma}{\sqrt{B}}\leq\frac{\sqrt{\varepsilon}}{4}. ( 2 κ + 2 3 log ( T / δ ) ) B σ ≤ 4 ε .
Solving for B B B yields the choice:
B = max { ⌈ 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 ε ⌉ , 1 } . B=\max\left\{\left\lceil\frac{16\sigma^2\left(\sqrt{2\kappa}+\sqrt{6\log(T/\delta)}\right)^2}{\varepsilon}\right\rceil,1\right\}. B = max ⎩ ⎨ ⎧ ε 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 , 1 ⎭ ⎬ ⎫ .
With these choices of λ \lambda λ and B B B , we conclude that with probability at least 1 − δ 1-\delta 1 − δ , we have for all k ∈ { 0 , … , T − 1 } k \in\{0,\ldots,T-1\} k ∈ { 0 , … , T − 1 } :
∥ g k − ∇ f ( x k ) ∥ ∗ ≤ ε 4 . (119) \left\lVert g^k-\nabla f(x^k)\right\rVert_* \le\frac{\sqrt{\varepsilon}}{4}. \tag*{(119)} g k − ∇ f ( x k ) ∗ ≤ 4 ε . (119)
Fix any realization on which (119) holds for every 0 ≤ k < T 0 \le k < T 0 ≤ k < T ; these realizations form an event of probability at least 1 − δ 1-\delta 1 − δ . All subsequent descent inequalities are evaluated along this fixed realization. We proceed by contradiction to show that there must exist some k ∈ { 0 , … , T − 1 } k \in\{0,\ldots,T-1\} k ∈ { 0 , … , T − 1 } such that ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε \left\lVert\nabla f(x^k)\right\rVert_*^2 \le\varepsilon ∇ f ( x k ) ∗ 2 ≤ ε .
Assume the contrary, i.e., that for all k ∈ { 0 , … , T − 1 } k \in\{0,\ldots,T-1\} k ∈ { 0 , … , T − 1 } :
∥ ∇ f ( x k ) ∥ ∗ 2 > ε . (120) \left\lVert\nabla f(x^k)\right\rVert_*^2 > \varepsilon. \tag*{(120)} ∇ f ( x k ) ∗ 2 > ε . (120)
This implies ∥ ∇ f ( x k ) ∥ ∗ > ε \left\lVert\nabla f(x^k)\right\rVert_* > \sqrt{\varepsilon} ∇ f ( x k ) ∗ > ε . Combining this with the error bound from (119) , we get:
∥ g k − ∇ f ( x k ) ∥ ∗ ≤ ε 4 < ∥ ∇ f ( x k ) ∥ ∗ 4 . (121) \left\lVert g^k-\nabla f(x^k)\right\rVert_* \le\frac{\sqrt{\varepsilon}}{4} < \frac{\left\lVert\nabla f(x^k)\right\rVert_*}{4}. \tag*{(121)} g k − ∇ f ( x k ) ∗ ≤ 4 ε < 4 ∇ f ( x k ) ∗ . (121)
By the triangle inequality for the dual norm, we can bound the norm of the stochastic gradient g k g^k g k :
3 4 ∥ ∇ f ( x k ) ∥ ∗ ≤ ∥ g k ∥ ∗ ≤ 5 4 ∥ ∇ f ( x k ) ∥ ∗ . (122) \frac{3}{4}\left\lVert\nabla f(x^k)\right\rVert_* \le\left\lVert g^k\right\rVert_* \le\frac{5}{4}\left\lVert\nabla f(x^k)\right\rVert_*. \tag*{(122)} 4 3 ∇ f ( x k ) ∗ ≤ g k ∗ ≤ 4 5 ∇ f ( x k ) ∗ . (122)
We now have all the necessary components to analyze the one-step progress of the algorithm. The proof proceeds by combining the standard descent lemma for L L L -smooth functions with the high-probability bounds on the stochastic gradient error.
Let us analyze the descent from iterate x k x^k x k to x k + 1 x^{k+1} x k + 1 . Since the function f f f is L L L -smooth, we have:
f ( x k + 1 ) ≤ f ( x k ) + ⟨ ∇ f ( x k ) , x k + 1 − x k ⟩ + L 2 ∥ x k + 1 − x k ∥ 2 = f ( x k ) − γ k ⟨ ∇ f ( x k ) , ( g k ) ♯ ⟩ + L γ k 2 2 ∥ ( g k ) ♯ ∥ 2 = f ( x k ) − γ k ⟨ ∇ f ( x k ) , ( g k ) ♯ ⟩ + L γ k 2 2 ∥ g k ∥ ∗ 2 . \begin{aligned}
f(x^{k+1}) &\le f(x^k)+\langle\nabla f(x^k),x^{k+1}-x^k\rangle+\frac{L}{2}\left\lVert x^{k+1}-x^k\right\rVert^2 \\
&= f(x^k)-\gamma_k\langle\nabla f(x^k),(g^k)^\sharp\rangle+\frac{L\gamma_k^2}{2}\left\lVert(g^k)^\sharp\right\rVert^2 \\
&= f(x^k)-\gamma_k\langle\nabla f(x^k),(g^k)^\sharp\rangle+\frac{L\gamma_k^2}{2}\left\lVert g^k\right\rVert_*^2.
\end{aligned} f ( x k + 1 ) ≤ f ( x k ) + ⟨ ∇ f ( x k ) , x k + 1 − x k ⟩ + 2 L x k + 1 − x k 2 = f ( x k ) − γ k ⟨ ∇ f ( x k ) , ( g k ) ♯ ⟩ + 2 L γ k 2 ( g k ) ♯ 2 = f ( x k ) − γ k ⟨ ∇ f ( x k ) , ( g k ) ♯ ⟩ + 2 L γ k 2 g k ∗ 2 .
In the last step, we used the fundamental property that ∥ s ♯ ∥ 2 = ∥ s ∥ ∗ 2 \left\lVert s^\sharp\right\rVert^2=\left\lVert s\right\rVert_*^2 s ♯ 2 = ∥ s ∥ ∗ 2 . We now rewrite the dual pairing term by adding and subtracting g k g^k g k :
− ⟨ ∇ f ( x k ) , ( g k ) ♯ ⟩ = ⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ − ⟨ g k , ( g k ) ♯ ⟩ . -\langle\nabla f(x^k),(g^k)^\sharp\rangle=\langle g^k-\nabla f(x^k),(g^k)^\sharp\rangle-\langle g^k,(g^k)^\sharp\rangle. − ⟨ ∇ f ( x k ) , ( g k ) ♯ ⟩ = ⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ − ⟨ g k , ( g k ) ♯ ⟩ .
Using the identity ⟨ s , s ♯ ⟩ = ∥ s ∥ ∗ 2 \langle s,s^\sharp\rangle=\left\lVert s\right\rVert_*^2 ⟨ s , s ♯ ⟩ = ∥ s ∥ ∗ 2 , this becomes:
− ⟨ ∇ f ( x k ) , ( g k ) ♯ ⟩ = ⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ − ∥ g k ∥ ∗ 2 . -\langle\nabla f(x^k),(g^k)^\sharp\rangle=\langle g^k-\nabla f(x^k),(g^k)^\sharp\rangle-\left\lVert g^k\right\rVert_*^2. − ⟨ ∇ f ( x k ) , ( g k ) ♯ ⟩ = ⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ − g k ∗ 2 .
Substituting this back into the descent inequality, we get:
f ( x k + 1 ) ≤ f ( x k ) + γ k ⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ − γ k ∥ g k ∥ ∗ 2 + L γ k 2 2 ∥ g k ∥ ∗ 2 = f ( x k ) + γ k ⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ − γ k ( 1 − L γ k / 2 ) ∥ g k ∥ ∗ 2 . \begin{aligned}
f(x^{k+1}) &\le f(x^k)+\gamma_k\langle g^k-\nabla f(x^k),(g^k)^\sharp\rangle-\gamma_k\left\lVert g^k\right\rVert_*^2+\frac{L\gamma_k^2}{2}\left\lVert g^k\right\rVert_*^2 \\
&= f(x^k)+\gamma_k\langle g^k-\nabla f(x^k),(g^k)^\sharp\rangle-\gamma_k(1-L\gamma_k/2)\left\lVert g^k\right\rVert_*^2.
\end{aligned} f ( x k + 1 ) ≤ f ( x k ) + γ k ⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ − γ k g k ∗ 2 + 2 L γ k 2 g k ∗ 2 = f ( x k ) + γ k ⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ − γ k ( 1 − L γ k /2 ) g k ∗ 2 .
We now apply the generalized Young’s inequality, ⟨ u , v ⟩ ≤ 1 2 s ∥ u ∥ ∗ 2 + s 2 ∥ v ∥ 2 \langle u,v\rangle\le\frac{1}{2s}\left\lVert u\right\rVert_*^2+\frac{s}{2}\left\lVert v\right\rVert^2 ⟨ u , v ⟩ ≤ 2 s 1 ∥ u ∥ ∗ 2 + 2 s ∥ v ∥ 2 , to the dual pairing term with u = g k − ∇ f ( x k ) u=g^k-\nabla f(x^k) u = g k − ∇ f ( x k ) , v = ( g k ) ♯ v=(g^k)^\sharp v = ( g k ) ♯ , and a parameter s > 0 s>0 s > 0 :
⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ ≤ 1 2 s ∥ g k − ∇ f ( x k ) ∥ ∗ 2 + s 2 ∥ ( g k ) ♯ ∥ 2 = 1 2 s ∥ g k − ∇ f ( x k ) ∥ ∗ 2 + s 2 ∥ g k ∥ ∗ 2 . \begin{aligned}
\langle g^k-\nabla f(x^k),(g^k)^\sharp\rangle
&\le\frac{1}{2s}\left\lVert g^k-\nabla f(x^k)\right\rVert_*^2+\frac{s}{2}\left\lVert(g^k)^\sharp\right\rVert^2 \\
&= \frac{1}{2s}\left\lVert g^k-\nabla f(x^k)\right\rVert_*^2+\frac{s}{2}\left\lVert g^k\right\rVert_*^2.
\end{aligned} ⟨ g k − ∇ f ( x k ) , ( g k ) ♯ ⟩ ≤ 2 s 1 g k − ∇ f ( x k ) ∗ 2 + 2 s ( g k ) ♯ 2 = 2 s 1 g k − ∇ f ( x k ) ∗ 2 + 2 s g k ∗ 2 .
Plugging this bound into our main inequality gives:
f ( x k + 1 ) ≤ f ( x k ) + γ k ( 1 2 s ∥ g k − ∇ f ( x k ) ∥ ∗ 2 + s 2 ∥ g k ∥ ∗ 2 ) − γ k ( 1 − L γ k 2 ) ∥ g k ∥ ∗ 2 = f ( x k ) + γ k 2 s ∥ g k − ∇ f ( x k ) ∥ ∗ 2 − γ k ( 1 − L γ k 2 − s 2 ) ∥ g k ∥ ∗ 2 . \begin{aligned}
f(x^{k+1}) \leq f(x^k) + \gamma_k \left( \frac{1}{2s} \lVert g^k - \nabla f(x^k) \rVert_*^2 + \frac{s}{2} \lVert g^k \rVert_*^2 \right) - \gamma_k \left(1 - \frac{L\gamma_k}{2}\right) \lVert g^k \rVert_*^2 \\
&= f(x^k) + \frac{\gamma_k}{2s} \lVert g^k - \nabla f(x^k) \rVert_*^2 - \gamma_k \left(1 - \frac{L\gamma_k}{2} - \frac{s}{2}\right) \lVert g^k \rVert_*^2.
\end{aligned} f ( x k + 1 ) ≤ f ( x k ) + γ k ( 2 s 1 ∥ g k − ∇ f ( x k ) ∥ ∗ 2 + 2 s ∥ g k ∥ ∗ 2 ) − γ k ( 1 − 2 L γ k ) ∥ g k ∥ ∗ 2 = f ( x k ) + 2 s γ k ∥ g k − ∇ f ( x k ) ∥ ∗ 2 − γ k ( 1 − 2 L γ k − 2 s ) ∥ g k ∥ ∗ 2 .
This inequality holds for any s > 0 s > 0 s > 0 . To substitute the lower bound ∥ g k ∥ ∗ 2 ≥ 9 16 ∥ ∇ f ( x k ) ∥ ∗ 2 \lVert g^k \rVert_*^2 \geq\frac{9}{16} \lVert\nabla f(x^k) \rVert_*^2 ∥ g k ∥ ∗ 2 ≥ 16 9 ∥ ∇ f ( x k ) ∥ ∗ 2 into its negative term, we require
1 − L γ k 2 − s 2 ≥ 0. 1 - \frac{L\gamma_k}{2} - \frac{s}{2} \geq0. 1 − 2 L γ k − 2 s ≥ 0.
With the prescribed stepsize γ k = 1 2 L \gamma_k = \frac{1}{2L} γ k = 2 L 1 , this condition is 3 4 − s 2 ≥ 0 \frac{3}{4} - \frac{s}{2} \geq0 4 3 − 2 s ≥ 0 . Hence we restrict the Young parameter to 0 < s ≤ 3 2 0 < s \leq\frac{3}{2} 0 < s ≤ 2 3 from now on. On this interval, the coefficient multiplying ∥ g k ∥ ∗ 2 \lVert g^k \rVert_*^2 ∥ g k ∥ ∗ 2 is nonpositive, so we may use the high-probability bounds from (121) and (122) to obtain
f ( x k + 1 ) ≤ f ( x k ) + γ k 2 s ( 1 4 ∥ ∇ f ( x k ) ∥ ∗ ) 2 − γ k ( 1 − L γ k 2 − s 2 ) ( 3 4 ∥ ∇ f ( x k ) ∥ ∗ ) 2 = f ( x k ) + γ k 32 s ∥ ∇ f ( x k ) ∥ ∗ 2 − 9 γ k 16 ( 1 − L γ k 2 − s 2 ) ∥ ∇ f ( x k ) ∥ ∗ 2 = f ( x k ) − γ k [ 9 16 ( 1 − L γ k 2 − s 2 ) − 1 32 s ] ∥ ∇ f ( x k ) ∥ ∗ 2 . \begin{aligned}
f(x^{k+1}) \leq f(x^k) + \frac{\gamma_k}{2s} \left(\frac{1}{4} \lVert\nabla f(x^k) \rVert_* \right)^2 - \gamma_k \left(1 - \frac{L\gamma_k}{2} - \frac{s}{2}\right) \left(\frac{3}{4} \lVert\nabla f(x^k) \rVert_* \right)^2 \\
&= f(x^k) + \frac{\gamma_k}{32s} \lVert\nabla f(x^k) \rVert_*^2 - \frac{9\gamma_k}{16} \left(1 - \frac{L\gamma_k}{2} - \frac{s}{2}\right) \lVert\nabla f(x^k) \rVert_*^2 \\
&= f(x^k) - \gamma_k \left[\frac{9}{16}\left(1 - \frac{L\gamma_k}{2} - \frac{s}{2}\right) - \frac{1}{32s}\right] \lVert\nabla f(x^k) \rVert_*^2.
\end{aligned} f ( x k + 1 ) ≤ f ( x k ) + 2 s γ k ( 4 1 ∥ ∇ f ( x k ) ∥ ∗ ) 2 − γ k ( 1 − 2 L γ k − 2 s ) ( 4 3 ∥ ∇ f ( x k ) ∥ ∗ ) 2 = f ( x k ) + 32 s γ k ∥ ∇ f ( x k ) ∥ ∗ 2 − 16 9 γ k ( 1 − 2 L γ k − 2 s ) ∥ ∇ f ( x k ) ∥ ∗ 2 = f ( x k ) − γ k [ 16 9 ( 1 − 2 L γ k − 2 s ) − 32 s 1 ] ∥ ∇ f ( x k ) ∥ ∗ 2 .
To ensure descent, the step size γ k \gamma_k γ k and the parameter s s s must make the term in the square brackets positive. Using the constant step size γ k ≡ γ = 1 2 L \gamma_k \equiv\gamma= \frac{1}{2L} γ k ≡ γ = 2 L 1 prescribed in the theorem, the expression becomes:
f ( x k + 1 ) ≤ f ( x k ) − 1 2 L [ 9 16 ( 1 − 1 4 − s 2 ) − 1 32 s ] ∥ ∇ f ( x k ) ∥ ∗ 2 . f(x^{k+1}) \leq f(x^k) - \frac{1}{2L} \left[\frac{9}{16}\left(1 - \frac{1}{4} - \frac{s}{2}\right) - \frac{1}{32s}\right] \lVert\nabla f(x^k) \rVert_*^2. f ( x k + 1 ) ≤ f ( x k ) − 2 L 1 [ 16 9 ( 1 − 4 1 − 2 s ) − 32 s 1 ] ∥ ∇ f ( x k ) ∥ ∗ 2 .
To obtain the strongest possible descent guarantee from this inequality, we maximize the term in the square brackets over the admissible interval 0 < s ≤ 3 2 0 < s \leq\frac{3}{2} 0 < s ≤ 2 3 . Define
h ( s ) : = 9 16 ( 1 − 1 4 − s 2 ) − 1 32 s . h(s) := \frac{9}{16}\left(1 - \frac{1}{4} - \frac{s}{2}\right) - \frac{1}{32s}. h ( s ) := 16 9 ( 1 − 4 1 − 2 s ) − 32 s 1 .
Its derivative is
h ′ ( s ) = − 9 32 + 1 32 s 2 . h'(s) = -\frac{9}{32} + \frac{1}{32s^2}. h ′ ( s ) = − 32 9 + 32 s 2 1 .
Setting h ′ ( s ) = 0 h'(s) = 0 h ′ ( s ) = 0 gives 9 s 2 = 1 9s^2 = 1 9 s 2 = 1 , whose positive solution is s ∗ = 1 3 s^* = \frac{1}{3} s ∗ = 3 1 . Since h ′ ′ ( s ) = − 1 16 s 3 < 0 h''(s) = -\frac{1}{16s^3} < 0 h ′′ ( s ) = − 16 s 3 1 < 0 for s > 0 s > 0 s > 0 , this is the unique maximizer on the admissible interval 0 < s ≤ 3 2 0 < s \leq\frac{3}{2} 0 < s ≤ 2 3 . In particular, s ∗ = 1 3 s^* = \frac{1}{3} s ∗ = 3 1 is admissible and gives 1 − L γ k 2 − s ∗ 2 = 7 12 > 0 1 - \frac{L\gamma_k}{2} - \frac{s^*}{2} = \frac{7}{12} > 0 1 − 2 L γ k − 2 s ∗ = 12 7 > 0 . We now substitute this optimal value s ∗ = 1 / 3 s^* = 1/3 s ∗ = 1/3 back into our one-step descent inequality:
h ( 1 / 3 ) = 9 16 ( 1 − 1 4 − 1 6 ) − 3 32 = 15 64 . h(1/3) = \frac{9}{16}\left(1 - \frac{1}{4} - \frac{1}{6}\right) - \frac{3}{32} = \frac{15}{64}. h ( 1/3 ) = 16 9 ( 1 − 4 1 − 6 1 ) − 32 3 = 64 15 .
Substituting this optimized constant back into the full inequality gives our final one-step progress guarantee:
f ( x k + 1 ) ≤ f ( x k ) − 1 2 L 15 64 ∥ ∇ f ( x k ) ∥ ∗ 2 . f(x^{k+1}) \leq f(x^k) - \frac{1}{2L}\frac{15}{64} \lVert\nabla f(x^k) \rVert_*^2. f ( x k + 1 ) ≤ f ( x k ) − 2 L 1 64 15 ∥ ∇ f ( x k ) ∥ ∗ 2 .
Summing this inequality over k = 0 , … , T − 1 k = 0,\ldots,T-1 k = 0 , … , T − 1 along the fixed realization and using (120) , we obtain
f ( x T ) ≤ f ( x 0 ) − 15 128 L ∑ k = 0 T − 1 ∥ ∇ f ( x k ) ∥ ∗ 2 < f ( x 0 ) − 15 T ε 128 L ≤ f ( x 0 ) − Δ = f inf . \begin{aligned}
f(x^T) \leq f(x^0) - \frac{15}{128L} \sum_{k=0}^{T-1} \lVert\nabla f(x^k) \rVert_*^2 \\
&< f(x^0) - \frac{15T\varepsilon}{128L} \\
&\leq f(x^0) - \Delta= f^{\inf}.
\end{aligned} f ( x T ) ≤ f ( x 0 ) − 128 L 15 k = 0 ∑ T − 1 ∥ ∇ f ( x k ) ∥ ∗ 2 < f ( x 0 ) − 128 L 15 T ε ≤ f ( x 0 ) − Δ = f i n f .
The last inequality uses the prescribed iteration count T = ⌈ 128 L Δ 15 ε ⌉ T = \left\lceil\frac{128L\Delta}{15\varepsilon} \right\rceil T = ⌈ 15 ε 128 L Δ ⌉ and Δ = f ( x 0 ) − f inf \Delta= f(x^{0}) - f^{\inf} Δ = f ( x 0 ) − f i n f . This contradicts the lower bound f ( x T ) ≥ f inf f(x^{T}) \ge f^{\inf} f ( x T ) ≥ f i n f from Assumption 2. Thus every realization in the good event contains an iterate with ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε \lVert\nabla f(x^{k}) \rVert_{*}^{2} \le\varepsilon ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε for some 0 ≤ k < T 0 \le k < T 0 ≤ k < T . Since this event has probability at least 1 − δ 1-\delta 1 − δ , we conclude that
Prob { min 0 ≤ k < T ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε } ≥ 1 − δ . \operatorname{Prob}\left\{\min_{0\leq k<T}\lVert\nabla f(x^{k})\rVert_{*}^{2}\leq\varepsilon\right\}\geq1-\delta. Prob { 0 ≤ k < T min ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε } ≥ 1 − δ .
This completes the proof.
Corollary 1 (Norm-native oracle complexity under dual-norm light tails). Suppose the assumptions of Theorem 1 hold. Run the update (7) with the constant stepsize γ k ≡ 1 2 L \gamma_k \equiv\frac{1}{2L} γ k ≡ 2 L 1 , and choose T T T and B B B as in that theorem. Let N ∗ : = B T N_* := BT N ∗ := B T denote the total number of stochastic-gradient evaluations. Then with probability at least 1 − δ 1-\delta 1 − δ , the method produces an iterate satisfying min 0 ≤ k ≤ T − 1 ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε \min_{0\leq k\leq T-1}\lVert\nabla f(x^{k})\rVert_{*}^{2}\leq\varepsilon min 0 ≤ k ≤ T − 1 ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε after at most
N ∗ = O ~ ( L Δ ε + κ L Δ σ 2 ε 2 ) . (123) N_*=\widetilde{\mathcal{O}}\left(\frac{L\Delta}{\varepsilon}+\frac{\kappa L\Delta\sigma^{2}}{\varepsilon^{2}}\right). \tag*{(123)} N ∗ = O ( ε L Δ + ε 2 κ L Δ σ 2 ) . (123) stochastic-gradient evaluations, where O ~ ( ⋅ ) \widetilde{\mathcal{O}}(\cdot) O ( ⋅ ) hides logarithmic factors in T T T and 1 / δ 1/\delta 1/ δ .
Proof of Corollary 1. By Theorem 1, the required number of iterations is
T = ⌈ 128 L Δ 15 ε ⌉ . T=\left\lceil\frac{128L\Delta}{15\varepsilon}\right\rceil. T = ⌈ 15 ε 128 L Δ ⌉ .
Since ε ≤ L Δ \varepsilon\leq L\Delta ε ≤ L Δ , this implies
T = O ( L Δ ε ) . T=\mathcal{O}\left(\frac{L\Delta}{\varepsilon}\right). T = O ( ε L Δ ) .
The same theorem prescribes the minibatch size
B = max { ⌈ 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 ε ⌉ , 1 } . B=\max\left\{\left\lceil\frac{16\sigma^{2}\left(\sqrt{2\kappa}+\sqrt{6\log(T/\delta)}\right)^{2}}{\varepsilon}\right\rceil,1\right\}. B = max ⎩ ⎨ ⎧ ε 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 , 1 ⎭ ⎬ ⎫ .
Using ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^{2}\leq2a^{2}+2b^{2} ( a + b ) 2 ≤ 2 a 2 + 2 b 2 , we have
( 2 κ + 6 log ( T / δ ) ) 2 ≤ 4 κ + 12 log ( T / δ ) . \left(\sqrt{2\kappa}+\sqrt{6\log(T/\delta)}\right)^{2}\leq4\kappa+12\log(T/\delta). ( 2 κ + 6 log ( T / δ ) ) 2 ≤ 4 κ + 12 log ( T / δ ) .
Consequently,
B = O ~ ( 1 + κ σ 2 ε ) , B=\widetilde{\mathcal{O}}\left(1+\frac{\kappa\sigma^{2}}{\varepsilon}\right), B = O ( 1 + ε κ σ 2 ) ,
where the suppressed factors are logarithmic in T T T and 1 / δ 1/\delta 1/ δ . Therefore,
N ∗ = B T = O ~ ( L Δ ε ( 1 + κ σ 2 ε ) ) = O ~ ( L Δ ε + κ L Δ σ 2 ε 2 ) , \begin{aligned}
N_*=BT=\widetilde{\mathcal{O}}\left(\frac{L\Delta}{\varepsilon}\left(1+\frac{\kappa\sigma^{2}}{\varepsilon}\right)\right) \\
&=\widetilde{\mathcal{O}}\left(\frac{L\Delta}{\varepsilon}+\frac{\kappa L\Delta\sigma^{2}}{\varepsilon^{2}}\right),
\end{aligned} N ∗ = B T = O ( ε L Δ ( 1 + ε κ σ 2 ) ) = O ( ε L Δ + ε 2 κ L Δ σ 2 ) ,
which proves (14). The stated success probability and stationarity guarantee follow directly from Theorem 1.
Time-Complexity Analysis of Rennala NSGD Under the Light-Tailed Assumption
We explain the role of fresh gradients and the modified reply order in § 4 , then prove the wall-clock guarantee under the fixed computation model.
Why discard stale gradients? The server may update the iterate while a worker is still computing, so a gradient can be outdated when it arrives. As discussed in Section C.2, a gradient computed at x k − δ k x^{k-\delta_k} x k − δ k need not be an unbiased estimator of ∇ f ( x k ) \nabla f(x^k) ∇ f ( x k ) . Moreover, to the best of our knowledge, the available wall-clock bounds for classical ASGD, which applies every arriving gradient, are suboptimal in general relative to the lower bound of Tyurin and Richtárik [ [84 ] for their zero-respecting algorithm class and specified unbiased bounded-variance oracle model with fixed worker computation times. Rennala SGD avoids staleness by using only gradients computed at the current iterate; in that Euclidean oracle model, its time complexity matches the lower bound up to a constant factor [84 ] . For our analysis, this restriction also connects the parallel method to the serial one: conditional on the history, the accepted gradients are i.i.d. oracle responses at x k x^k x k . The first part of the proof below establishes this property and applies the serial convergence guarantee.
Updating before replying. We slightly modify Method 4 of Tyurin and Richtárik [ [84 ] , whose server replies to the worker that completes the minibatch before updating the iterate. Our server updates first and then sends the new iterate and its index to that worker, allowing it to start a fresh computation immediately. Other workers finish their pending computations and receive the latest iterate when their messages are processed. Since each message is fully processed before the next is handled, including any update and reply, a simultaneous return processed after an update is checked against the new index. This change preserves the fresh-minibatch property and the wall-clock upper bound: a worker still computing at an earlier iterate needs at most τ i \tau_i τ i additional time before receiving the current one, while the worker that completed the minibatch starts immediately. The timing argument below accounts for both cases.
With these protocol details in place, we restate and prove the wall-clock guarantee.
Theorem 2 (Wall-clock time complexity of Rennala NSGD). Suppose the assumptions of Theorem 1 hold, and use the fixed computation model of § 4 . Run Rennala NSGD with the stepsize, iteration count T T T , and minibatch size B B B prescribed in that theorem. For 0 < ε ≤ L Δ 0 < \varepsilon\le L\Delta 0 < ε ≤ L Δ and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , the iterates satisfy min 0 ≤ k < T ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε \min_{0 \le k < T}\lVert\nabla f(x^k) \rVert_*^2 \le\varepsilon min 0 ≤ k < T ∥ ∇ f ( x k ) ∥ ∗ 2 ≤ ε with probability at least 1 − δ 1-\delta 1 − δ within total wall-clock time
T t o t a l = O ~ ( min S ∈ [ N ] [ ( 1 S ∑ i = 1 S 1 τ i ) − 1 ( L Δ ε + κ L Δ σ 2 S ε 2 ) ] ) . (124) T_{\mathrm{total}}=\widetilde{O}\left(\min_{S\in[N]}\left[\left(\frac{1}{S}\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1}\left(\frac{L\Delta}{\varepsilon}+\frac{\kappa L\Delta\sigma^2}{S\varepsilon^2}\right)\right]\right). \tag*{(124)} T total = O S ∈ [ N ] min ( S 1 i = 1 ∑ S τ i 1 ) − 1 ( ε L Δ + S ε 2 κ L Δ σ 2 ) . (124) We will also prove that, for any positive integers T T T and B B B , the time needed to complete T T T iterations satisfies
T t o t a l ≤ 2 T min S ∈ [ N ] [ ( ∑ i = 1 S 1 τ i ) − 1 ( B + S ) ] . (125) T_{\mathrm{total}}\le2T\min_{S\in[N]}\left[\left(\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1}(B+S)\right]. \tag*{(125)} T total ≤ 2 T S ∈ [ N ] min ( i = 1 ∑ S τ i 1 ) − 1 ( B + S ) . (125) This deterministic estimate will also be used in the noise-model comparison in Section M.4.
Proof of Theorem 2. The argument has two parts. We first verify that the stochastic gradients accepted during an iteration form a fresh minibatch at the current iterate, so that the serial guarantee of Theorem 1 applies verbatim; we then bound the wall-clock time required to complete T T T such iterations under the fixed computation model.
Step 1: the accepted gradients form a fresh minibatch. Fix an iteration k k k , and let F k \mathcal{F}_k F k denote the complete algorithmic history immediately after x k x^k x k is formed and before it is sent to any worker, including the states of workers with computations or messages pending. For k > 0 k>0 k > 0 , this is just before the reply to the worker that completed the preceding minibatch; for k = 0 k=0 k = 0 , it is before the initial replies. Thus x k x^k x k is F k \mathcal{F}_k F k -measurable, and every sample used for a gradient tagged with k k k is drawn after this history is fixed.
The server accepts a stochastic gradient during iteration k k k only if it is tagged with the current iteration index k k k . Hence, every accepted gradient is computed at x k x^k x k . Whenever a worker receives ( x k , k ) (x^k,k) ( x k , k ) , it draws a fresh sample from the common data distribution, independently of the past and of the samples used by other oracle calls, and computes a stochastic gradient at x k x^k x k .
Under the fixed computation model, the order of worker returns is determined by the constants { τ i } i = 1 N \{\tau_i\}_{i=1}^N { τ i } i = 1 N , the worker states recorded in F k \mathcal{F}_k F k , and any independent tie-breaking randomness. By the tie-breaking requirement in § 4 , this order does not depend on the fresh oracle samples or their gradient values. Thus selecting the first B B B returns tagged with k k k preserves the sampling law: conditional on F k \mathcal{F}_k F k , the accepted samples { ξ j k } j = 1 B \{\xi_j^k\}_{j=1}^B { ξ j k } j = 1 B are i.i.d. with distribution D \mathcal{D} D . Since x k x^k x k is F k \mathcal{F}_k F k -measurable, the accepted gradients are conditionally i.i.d. given F k \mathcal{F}_k F k , with the conditional distribution of ∇ f ( x k ; ξ ) \nabla f(x^k;\xi) ∇ f ( x k ; ξ ) for a fresh ξ ∼ D \xi\sim\mathcal{D} ξ ∼ D . This conditional distribution can depend on the history through x k x^k x k .
Writing the accepted minibatch as g k : = 1 B ∑ j = 1 B ∇ f ( x k ; ξ j k ) g^k := \frac{1}{B}\sum_{j=1}^B \nabla f(x^k;\xi_j^k) g k := B 1 ∑ j = 1 B ∇ f ( x k ; ξ j k ) , we consequently have E [ g k ∣ F k ] = ∇ f ( x k ) \mathbb{E}[g^k \mid\mathcal{F}_k] = \nabla f(x^k) E [ g k ∣ F k ] = ∇ f ( x k ) , and the zero-mean noise vectors η j k : = ∇ f ( x k ; ξ j k ) − ∇ f ( x k ) \eta_j^k := \nabla f(x^k;\xi_j^k) - \nabla f(x^k) η j k := ∇ f ( x k ; ξ j k ) − ∇ f ( x k ) satisfy Assumption 3 conditionally on F k \mathcal{F}_k F k . Applying Lemma 1 conditionally with x = x k x=x^k x = x k therefore gives, for every λ > 0 \lambda>0 λ > 0 ,
Prob { ∥ g k − ∇ f ( x k ) ∥ ∗ ≥ ( 2 κ + 2 λ ) σ B | F k } ≤ exp ( − λ 2 / 3 ) . (126) \operatorname{Prob}\left\{\left\|g^k-\nabla f(x^k)\right\|_* \geq\left(\sqrt{2\kappa}+\sqrt{2}\lambda\right)\frac{\sigma}{\sqrt{B}} \,\middle|\, \mathcal{F}_k\right\}\leq\exp(-\lambda^2/3). \tag*{(126)} Prob { g k − ∇ f ( x k ) ∗ ≥ ( 2 κ + 2 λ ) B σ F k } ≤ exp ( − λ 2 /3 ) . (126)
Thus the iterates produced by Rennala NSGD satisfy the same conditional minibatch oracle properties as the serial minibatch method analyzed in Theorem 1, so that theorem applies.
Step 2: iteration complexity. By Theorem 1, with the choice of T T T and B B B in the statement, Rennala NSGD finds an ε \varepsilon ε -stationary point after at most T T T iterations with probability at least 1 − δ 1-\delta 1 − δ . Our subsequent wall-clock time analysis is adapted from the proof of Theorem 7.5 in [84 ] to the non-Euclidean setting. Thus, it remains to upper bound the wall-clock time required to complete T T T iterations under the fixed computation model where worker i i i returns one stochastic gradient every τ i \tau_i τ i seconds. Without loss of generality, assume 0 < τ 1 ≤ ⋯ ≤ τ N 0<\tau_1\leq\cdots\leq\tau_N 0 < τ 1 ≤ ⋯ ≤ τ N .
Time to collect one minibatch. Fix an iteration index k k k . In iteration k k k , the server forms the minibatch estimator g k g^k g k by collecting B B B stochastic gradients computed at the current iterate x k x^k x k (i.e., gradients whose iteration index equals k k k ), ignoring all stale gradients.
Choose nonnegative integer target counts B i B_i B i for the workers. These are sufficient quotas for bounding the collection time, not necessarily the numbers of gradients actually accepted from each worker before the minibatch is completed.
At the instant x k x^k x k is formed, each worker has at most one computation or message pending from an earlier iteration. Completing and processing it takes at most τ i \tau_i τ i additional time, after which the worker receives the current iterate. The worker that completed the preceding minibatch instead receives x k x^k x k immediately and starts a fresh computation; at k = 0 k=0 k = 0 , all workers receive the initial iterate without first completing a stale computation. As long as iteration k k k has not ended, every subsequent reply sends x k x^k x k again. Consequently, unless the minibatch has already been completed, worker i i i can deliver B i > 0 B_i>0 B i > 0 fresh gradients within τ i ( 1 + B i ) \tau_i(1+B_i) τ i ( 1 + B i ) time after x k x^k x k is formed. The extra τ i \tau_i τ i is unnecessary for a worker that starts immediately, but remains a valid upper bound. Define
t i : = { τ i ( 1 + B i ) , B i > 0 , 0 , B i = 0. t_i := \begin{cases}
\tau_i(1+B_i), & B_i>0,\\
0, & B_i=0.
\end{cases} t i := { τ i ( 1 + B i ) , 0 , B i > 0 , B i = 0.
Since workers run in parallel, a sufficient condition to collect a batch of size B B B is the existence of t ′ ≥ 0 t'\geq0 t ′ ≥ 0 and integers B 1 , … , B N ∈ N 0 B_1,\ldots,B_N\in\mathbb{N}_0 B 1 , … , B N ∈ N 0 such that
t ′ ≥ max i ∈ [ N ] t i , ∑ i = 1 N B i ≥ B . (127) t'\geq\max_{i\in[N]}t_i,\qquad\sum_{i=1}^{N}B_i\geq B. \tag*{(127)} t ′ ≥ i ∈ [ N ] max t i , i = 1 ∑ N B i ≥ B . (127)
Indeed, if the iteration had not ended by time t ′ t' t ′ , each worker would have delivered its quota, giving at least B B B accepted gradients, a contradiction. Thus the modified reply order admits the same collection-time bound as the original protocol.
Relaxation and rounding. We relax (127) by allowing B i ∈ R + B_i\in\mathbb{R}_+ B i ∈ R + . If ( t ′ , B 1 , … , B N ) (t',B_1,\ldots,B_N) ( t ′ , B 1 , … , B N ) is feasible for the relaxed problem, then ( 2 t ′ , ⌈ B 1 ⌉ , … , ⌈ B N ⌉ ) (2t',\lceil B_1\rceil,\ldots,\lceil B_N\rceil) ( 2 t ′ , ⌈ B 1 ⌉ , … , ⌈ B N ⌉) is feasible for the integer problem. Indeed, ∑ i ⌈ B i ⌉ ≥ ∑ i B i ≥ B \sum_i\lceil B_i\rceil\geq\sum_i B_i\geq B ∑ i ⌈ B i ⌉ ≥ ∑ i B i ≥ B , and for any i i i with B i > 0 B_i>0 B i > 0 ,
τ i ( 1 + ⌈ B i ⌉ ) ≤ τ i ( 2 + B i ) ≤ 2 τ i ( 1 + B i ) ≤ 2 t ′ . \tau_i(1+\lceil B_i\rceil)\leq\tau_i(2+B_i)\leq2\tau_i(1+B_i)\leq2t'. τ i ( 1 + ⌈ B i ⌉) ≤ τ i ( 2 + B i ) ≤ 2 τ i ( 1 + B i ) ≤ 2 t ′ .
(For B i = 0 B_i = 0 B i = 0 , rounding leaves B i B_i B i and t i t_i t i equal to zero.) Hence it suffices to construct a feasible solution to the relaxed problem and then pay a factor 2.
Constructing a feasible solution. For any S ∈ [ N ] S \in[N] S ∈ [ N ] , define
t ′ ( S ) : = ( ∑ i = 1 S 1 τ i ) − 1 ( B + S ) , t'(S) := \left(\sum_{i=1}^{S} \frac{1}{\tau_i}\right)^{-1}(B+S), t ′ ( S ) := ( i = 1 ∑ S τ i 1 ) − 1 ( B + S ) ,
and let S ⋆ ∈ arg min S ∈ [ N ] t ′ ( S ) S^\star\in\arg\min_{S\in[N]} t'(S) S ⋆ ∈ arg min S ∈ [ N ] t ′ ( S ) be the smallest minimizer. Consider the choice
B i : = { t ′ ( S ⋆ ) τ i − 1 , i ≤ S ⋆ , 0 , i > S ⋆ . B_i := \begin{cases}
\frac{t'(S^\star)}{\tau_i}-1, & i \leq S^\star,\\
0, & i > S^\star.
\end{cases} B i := { τ i t ′ ( S ⋆ ) − 1 , 0 , i ≤ S ⋆ , i > S ⋆ .
Then
∑ i = 1 N B i = ∑ i = 1 S ⋆ ( t ′ ( S ⋆ ) τ i − 1 ) = t ′ ( S ⋆ ) ∑ i = 1 S ⋆ 1 τ i − S ⋆ = ( B + S ⋆ ) − S ⋆ = B , \begin{aligned}
\sum_{i=1}^{N} B_i
&= \sum_{i=1}^{S^\star}\left(\frac{t'(S^\star)}{\tau_i}-1\right)\\
&= t'(S^\star)\sum_{i=1}^{S^\star}\frac{1}{\tau_i}-S^\star\\
&= (B+S^\star)-S^\star=B,
\end{aligned} i = 1 ∑ N B i = i = 1 ∑ S ⋆ ( τ i t ′ ( S ⋆ ) − 1 ) = t ′ ( S ⋆ ) i = 1 ∑ S ⋆ τ i 1 − S ⋆ = ( B + S ⋆ ) − S ⋆ = B ,
so the batch-size constraint holds with equality. Moreover, one can verify that t ′ ( S ⋆ ) > τ S ⋆ t'(S^\star)>\tau_{S^\star} t ′ ( S ⋆ ) > τ S ⋆ , hence B i > 0 B_i>0 B i > 0 for all i ≤ S ⋆ i\leq S^\star i ≤ S ⋆ .
Showing that B i > 0 B_i > 0 B i > 0 for all i ≤ S ⋆ i \leq S^\star i ≤ S ⋆ . Recall that
t ′ ( S ) : = ( ∑ i = 1 S 1 τ i ) − 1 ( B + S ) , S ⋆ ∈ arg min S ∈ [ N ] t ′ ( S ) is the smallest minimizer , t'(S) := \left(\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1}(B+S), \qquad S^\star\in\arg\min_{S\in[N]} t'(S) \text{ is the \emph{smallest minimizer}}, t ′ ( S ) := ( i = 1 ∑ S τ i 1 ) − 1 ( B + S ) , S ⋆ ∈ arg S ∈ [ N ] min t ′ ( S ) is the smallest minimizer ,
and we set
B i : = { t ′ ( S ⋆ ) τ i − 1 , i ≤ S ⋆ , 0 , i > S ⋆ . B_i := \begin{cases}
\frac{t'(S^\star)}{\tau_i}-1, & i \leq S^\star,\\
0, & i > S^\star.
\end{cases} B i := { τ i t ′ ( S ⋆ ) − 1 , 0 , i ≤ S ⋆ , i > S ⋆ .
We prove that t ′ ( S ⋆ ) > τ S ⋆ t'(S^\star)>\tau_{S^\star} t ′ ( S ⋆ ) > τ S ⋆ , which will imply B i > 0 B_i>0 B i > 0 for all i ≤ S ⋆ i\leq S^\star i ≤ S ⋆ .
Case 1: S ⋆ = 1 S^\star=1 S ⋆ = 1 . Then
t ′ ( 1 ) = τ 1 ( B + 1 ) > τ 1 = τ S ⋆ . t'(1)=\tau_1(B+1)>\tau_1=\tau_{S^\star}. t ′ ( 1 ) = τ 1 ( B + 1 ) > τ 1 = τ S ⋆ .
Case 2: S ⋆ > 1 S^\star>1 S ⋆ > 1 . Since S ⋆ S^\star S ⋆ is the \emph{smallest} minimizer of t ′ ( ⋅ ) t'(\cdot) t ′ ( ⋅ ) , we must have
t ′ ( S ⋆ ) < t ′ ( S ⋆ − 1 ) . t'(S^\star)<t'(S^\star-1). t ′ ( S ⋆ ) < t ′ ( S ⋆ − 1 ) .
Writing A S : = ∑ i = 1 S 1 τ i A_S:=\sum_{i=1}^{S}\frac{1}{\tau_i} A S := ∑ i = 1 S τ i 1 , this inequality is equivalent to
A S ⋆ − 1 ( B + S ⋆ ) < A S ⋆ − 1 − 1 ( B + S ⋆ − 1 ) , A_{S^\star}^{-1}(B+S^\star)<A_{S^\star-1}^{-1}(B+S^\star-1), A S ⋆ − 1 ( B + S ⋆ ) < A S ⋆ − 1 − 1 ( B + S ⋆ − 1 ) ,
and hence (multiplying by A S ⋆ A S ⋆ − 1 > 0 A_{S^\star}A_{S^\star-1}>0 A S ⋆ A S ⋆ − 1 > 0 )
A S ⋆ − 1 ( B + S ⋆ ) < A S ⋆ ( B + S ⋆ − 1 ) . A_{S^\star-1}(B+S^\star)<A_{S^\star}(B+S^\star-1). A S ⋆ − 1 ( B + S ⋆ ) < A S ⋆ ( B + S ⋆ − 1 ) .
Using A S ⋆ = A S ⋆ − 1 + 1 τ S ⋆ A_{S^\star}=A_{S^\star-1}+\frac{1}{\tau_{S^\star}} A S ⋆ = A S ⋆ − 1 + τ S ⋆ 1 , we obtain
A S ⋆ − 1 ( B + S ⋆ ) < ( A S ⋆ − 1 + 1 τ S ⋆ ) ( B + S ⋆ − 1 ) = A S ⋆ − 1 ( B + S ⋆ − 1 ) + B + S ⋆ − 1 τ S ⋆ . \begin{aligned}
A_{S^\star-1}(B+S^\star)<\left(A_{S^\star-1}+\frac{1}{\tau_{S^\star}}\right)(B+S^\star-1) \\
&= A_{S^\star-1}(B+S^\star-1)+\frac{B+S^\star-1}{\tau_{S^\star}}.
\end{aligned} A S ⋆ − 1 ( B + S ⋆ ) < ( A S ⋆ − 1 + τ S ⋆ 1 ) ( B + S ⋆ − 1 ) = A S ⋆ − 1 ( B + S ⋆ − 1 ) + τ S ⋆ B + S ⋆ − 1 .
Subtracting A S ⋆ − 1 ( B + S ⋆ − 1 ) A_{S^\star-1}(B+S^\star-1) A S ⋆ − 1 ( B + S ⋆ − 1 ) from both sides yields
A S ⋆ − 1 < B + S ⋆ − 1 τ S ⋆ , A_{S^\star-1}<\frac{B+S^\star-1}{\tau_{S^\star}}, A S ⋆ − 1 < τ S ⋆ B + S ⋆ − 1 ,
and therefore
A S ⋆ = A S ⋆ − 1 + 1 τ S ⋆ < B + S ⋆ − 1 τ S ⋆ + 1 τ S ⋆ = B + S ⋆ τ S ⋆ . A_{S^\star}=A_{S^\star-1}+\frac{1}{\tau_{S^\star}}<\frac{B+S^\star-1}{\tau_{S^\star}}+\frac{1}{\tau_{S^\star}}=\frac{B+S^\star}{\tau_{S^\star}}. A S ⋆ = A S ⋆ − 1 + τ S ⋆ 1 < τ S ⋆ B + S ⋆ − 1 + τ S ⋆ 1 = τ S ⋆ B + S ⋆ .
Finally,
t ′ ( S ⋆ ) = B + S ⋆ A S ⋆ > τ S ⋆ . t'(S^\star)=\frac{B+S^\star}{A_{S^\star}}>\tau_{S^\star}. t ′ ( S ⋆ ) = A S ⋆ B + S ⋆ > τ S ⋆ .
Since τ 1 ≤ ⋯ ≤ τ S ⋆ \tau_1\leq\cdots\leq\tau_{S^\star} τ 1 ≤ ⋯ ≤ τ S ⋆ , for every i ≤ S ⋆ i\leq S^\star i ≤ S ⋆ we have
B i = t ′ ( S ⋆ ) τ i − 1 ≥ t ′ ( S ⋆ ) τ S ⋆ − 1 > 0. B_i=\frac{t'(S^\star)}{\tau_i}-1\geq\frac{t'(S^\star)}{\tau_{S^\star}}-1>0. B i = τ i t ′ ( S ⋆ ) − 1 ≥ τ S ⋆ t ′ ( S ⋆ ) − 1 > 0.
Thus indeed B i > 0 B_i>0 B i > 0 for all i ≤ S ⋆ i\leq S^\star i ≤ S ⋆ .
Finally,
max i ∈ [ N ] t i = max i ≤ S ⋆ τ i ( 1 + B i ) = max i ≤ S ⋆ τ i ⋅ t ′ ( S ⋆ ) τ i = t ′ ( S ⋆ ) , \begin{aligned}
\max_{i\in[N]}t_i=\max_{i\leq S^\star}\tau_i(1+B_i) \\
&=\max_{i\leq S^\star}\tau_i\cdot\frac{t'(S^\star)}{\tau_i} \\
&=t'(S^\star),
\end{aligned} i ∈ [ N ] max t i = i ≤ S ⋆ max τ i ( 1 + B i ) = i ≤ S ⋆ max τ i ⋅ τ i t ′ ( S ⋆ ) = t ′ ( S ⋆ ) ,
so ( t ′ ( S ⋆ ) , B 1 , … , B N ) (t'(S^\star),B_1,\ldots,B_N) ( t ′ ( S ⋆ ) , B 1 , … , B N ) is feasible for the relaxed problem. By the rounding argument, the server can therefore collect a batch of size B B B within at most 2 t ′ ( S ⋆ ) 2t'(S^\star) 2 t ′ ( S ⋆ ) seconds in every iteration.
Total time. Thus, the wall-clock time to execute T T T iterations is at most
T t o t a l ≤ 2 T ⋅ min S ∈ [ N ] [ ( ∑ i = 1 S 1 τ i ) − 1 ( B + S ) ] . (128) T_{\mathrm{total}}\leq2T\cdot\min_{S\in[N]}\left[\left(\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1}(B+S)\right]. \tag*{(128)} T total ≤ 2 T ⋅ S ∈ [ N ] min ( i = 1 ∑ S τ i 1 ) − 1 ( B + S ) . (128)
Combining this deterministic runtime bound with Theorem 1 (which guarantees success after T T T iterations with probability at least 1 − δ 1-\delta 1 − δ ) yields the high-probability time guarantee (125) .
From the exact bound to a O ~ \widetilde{\mathcal{O}} O complexity bound. Fix S ∈ [ N ] S\in[N] S ∈ [ N ] and define the average inverse time (average rate) of the fastest S S S workers by
r ˉ S : = 1 S ∑ i = 1 S 1 τ i . \bar{r}_S:=\frac{1}{S}\sum_{i=1}^{S}\frac{1}{\tau_i}. r ˉ S := S 1 i = 1 ∑ S τ i 1 .
Then we can rewrite
( ∑ i = 1 S 1 τ i ) − 1 ( B + S ) = B + S ∑ i = 1 S 1 τ i = S ∑ i = 1 S 1 τ i ( 1 + B S ) = r ˉ S − 1 ( 1 + B S ) . \begin{aligned}
\left(\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1}(B+S)=\frac{B+S}{\sum_{i=1}^{S}\frac{1}{\tau_i}}=\frac{S}{\sum_{i=1}^{S}\frac{1}{\tau_i}}\left(1+\frac{B}{S}\right) \\
&=\bar{r}_S^{-1}\left(1+\frac{B}{S}\right).
\end{aligned} ( i = 1 ∑ S τ i 1 ) − 1 ( B + S ) = ∑ i = 1 S τ i 1 B + S = ∑ i = 1 S τ i 1 S ( 1 + S B ) = r ˉ S − 1 ( 1 + S B ) .
Plugging this into (128) gives
T t o t a l ≤ 2 T ⋅ min S ∈ [ N ] r ˉ S − 1 ( 1 + B S ) = 2 min S ∈ [ N ] r ˉ S − 1 ( T + T B S ) . (129) T_{\mathrm{total}}\leq2T\cdot\min_{S\in[N]}\bar{r}_S^{-1}\left(1+\frac{B}{S}\right)=2\min_{S\in[N]}\bar{r}_S^{-1}\left(T+\frac{TB}{S}\right). \tag*{(129)} T total ≤ 2 T ⋅ S ∈ [ N ] min r ˉ S − 1 ( 1 + S B ) = 2 S ∈ [ N ] min r ˉ S − 1 ( T + S T B ) . (129)
Next, we substitute the iteration and batch-size choices from Theorem 1:
T = ⌈ 128 L Δ 15 ε ⌉ ⟹ T = O ( L Δ ε ) , T=\left\lceil\frac{128L\Delta}{15\varepsilon}\right\rceil\Longrightarrow T=\mathcal{O}\left(\frac{L\Delta}{\varepsilon}\right), T = ⌈ 15 ε 128 L Δ ⌉ ⟹ T = O ( ε L Δ ) ,
and
B = max { ⌈ 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 ε ⌉ , 1 } . B=\max\left\{\left\lceil\frac{16\sigma^2\left(\sqrt{2\kappa}+\sqrt{6\log(T/\delta)}\right)^2}{\varepsilon}\right\rceil,1\right\}. B = max ⎩ ⎨ ⎧ ε 16 σ 2 ( 2 κ + 6 log ( T / δ ) ) 2 , 1 ⎭ ⎬ ⎫ .
Using ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^2\le2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2 with a = 2 κ a=\sqrt{2\kappa} a = 2 κ and b = 6 log ( T / δ ) b=\sqrt{6\log(T/\delta)} b = 6 log ( T / δ ) , we obtain
( 2 κ + 6 log ( T / δ ) ) 2 ≤ 2 ( 2 κ ) + 2 ( 6 log ( T / δ ) ) = 4 κ + 12 log ( T / δ ) . \left(\sqrt{2\kappa}+\sqrt{6\log(T/\delta)}\right)^2\le2(2\kappa)+2(6\log(T/\delta))=4\kappa+12\log(T/\delta). ( 2 κ + 6 log ( T / δ ) ) 2 ≤ 2 ( 2 κ ) + 2 ( 6 log ( T / δ )) = 4 κ + 12 log ( T / δ ) .
Therefore,
B ≤ 1 + 16 σ 2 ( 4 κ + 12 log ( T / δ ) ) ε = 1 + 64 κ σ 2 ε + 192 σ 2 log ( T / δ ) ε . (130) B\le1+\frac{16\sigma^2(4\kappa+12\log(T/\delta))}{\varepsilon}=1+\frac{64\kappa\sigma^2}{\varepsilon}+\frac{192\sigma^2\log(T/\delta)}{\varepsilon}. \tag*{(130)} B ≤ 1 + ε 16 σ 2 ( 4 κ + 12 log ( T / δ )) = 1 + ε 64 κ σ 2 + ε 192 σ 2 log ( T / δ ) . (130)
In particular, using κ ≥ 1 \kappa\ge1 κ ≥ 1 and hiding the logarithmic factor log ( T / δ ) \log(T/\delta) log ( T / δ ) inside O ~ ( ⋅ ) \widetilde{\mathcal{O}}(\cdot) O ( ⋅ ) , we have
B = O ~ ( 1 + κ σ 2 ε ) . B=\widetilde{\mathcal{O}}\left(1+\frac{\kappa\sigma^2}{\varepsilon}\right). B = O ( 1 + ε κ σ 2 ) .
Consequently,
T B S = O ~ ( T S + κ T σ 2 S ε ) = O ~ ( L Δ S ε + κ L Δ σ 2 S ε 2 ) . \begin{aligned}
\frac{TB}{S}=\widetilde{\mathcal{O}}\left(\frac{T}{S}+\frac{\kappa T\sigma^2}{S\varepsilon}\right) \\
&=\widetilde{\mathcal{O}}\left(\frac{L\Delta}{S\varepsilon}+\frac{\kappa L\Delta\sigma^2}{S\varepsilon^2}\right).
\end{aligned} S T B = O ( S T + S ε κ T σ 2 ) = O ( S ε L Δ + S ε 2 κ L Δ σ 2 ) .
Since S ≥ 1 S\ge1 S ≥ 1 , the deterministic contribution T / S T/S T / S is at most T T T and can be absorbed into the leading T T T term in (129) . Substituting these bounds into (129) gives
T t o t a l = O ~ ( min S ∈ [ N ] [ ( 1 S ∑ i = 1 S 1 τ i ) − 1 ( L Δ ε + κ L Δ σ 2 S ε 2 ) ] ) . T_{\mathrm{total}}=\widetilde{\mathcal{O}}\left(\min_{S\in[N]}\left[\left(\frac{1}{S}\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1}\left(\frac{L\Delta}{\varepsilon}+\frac{\kappa L\Delta\sigma^2}{S\varepsilon^2}\right)\right]\right). T total = O S ∈ [ N ] min ( S 1 i = 1 ∑ S τ i 1 ) − 1 ( ε L Δ + S ε 2 κ L Δ σ 2 ) .
which is the claimed O ~ \widetilde{\mathcal{O}} O time complexity bound.
Discussion
We now interpret (15), explain how the update norm and noise model affect it, and compare its Euclidean specialization with existing guarantees.
A system factor and a problem factor. For each S ∈ [ N ] S\in[N] S ∈ [ N ] , the bound (15) is the product of two factors. The first, ( 1 S ∑ i = 1 S 1 τ i ) − 1 \left(\frac{1}{S}\sum_{i=1}^{S}\frac{1}{\tau_i}\right)^{-1} ( S 1 ∑ i = 1 S τ i 1 ) − 1 , is the harmonic mean of the computation times of the S S S fastest workers, and it depends only on the system. The second, L Δ ε + κ L Δ σ 2 S ε 2 \frac{L\Delta}{\varepsilon}+\frac{\kappa L\Delta\sigma^2}{S\varepsilon^2} ε L Δ + S ε 2 κ L Δ σ 2 , is the oracle complexity (14) with its statistical term divided by S S S , and apart from S S S it depends only on the problem and the update norm. Enlarging S S S shrinks the statistical term but admits slower workers into the harmonic mean, and the minimum over S S S selects the best balance. Running the algorithm does not require knowing either the minimizing number of workers or their computation times τ i \tau_i τ i . The prescribed T T T and B B B do not depend on these quantities: the server simply accepts the first B B B fresh gradients to arrive.
Where the update norm enters. The update norm ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ leaves the system factor untouched and acts only through the second factor, but there it affects every constant except Δ \Delta Δ . The smoothness constant L L L is defined through the pair ( ∥ ⋅ ∥ , ∥ ⋅ ∥ ∗ ) (\|\cdot\|,\|\cdot\|_*) ( ∥ ⋅ ∥ , ∥ ⋅ ∥ ∗ ) in Assumption 1, the noise level σ \sigma σ is measured in ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ in Assumption 3, the accuracy ε \varepsilon ε refers to stationarity measured in ∥ ⋅ ∥ ∗ \|\cdot\|_* ∥ ⋅ ∥ ∗ , and κ \kappa κ is the regularity constant of ( E ∗ , ∥ ⋅ ∥ ∗ ) (E^*,\|\cdot\|_*) ( E ∗ , ∥ ⋅ ∥ ∗ ) . Bounds obtained for different update norms therefore certify stationarity in different dual norms, and they should not be compared as if they measured the same quantity.
Native versus converted noise in wall-clock time. The comparison of § 3.6.1 carries over from oracle complexity to wall-clock time. Under the comparison assumptions of Section J.1 and the fixed computation model of Theorem 2, apply Lemma 21 conditionally to each accepted minibatch and use B F B_F B F from (59) . The same union bound and descent argument give the required stationarity guarantee. The deterministic minibatch-collection bound (125) applies with B = B F B = B_F B = B F . Since B F = O ~ ( 1 + C F / ε ) B_F = \widetilde{O}(1 + C_F/\varepsilon) B F = O ( 1 + C F / ε ) , substituting into that bound gives the form (15) with the full native coefficient κ σ 2 \kappa\sigma^2 κ σ 2 replaced by C F = α − 2 σ F 2 C_F = \alpha^{-2}\sigma_F^2 C F = α − 2 σ F 2 ; the system factor is unchanged. With the update norm, target accuracy, and worker times fixed, a smaller statistical coefficient reduces the noise-dependent term for every choice of S S S . Taking the minimum over S S S therefore cannot increase the resulting expression. Thus, a smaller statistical coefficient also gives a wall-clock upper bound that is no larger and can be significantly smaller, up to the suppressed logarithmic factors.
The Euclidean case. When the update norm is Euclidean, ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 \|\cdot\|_* = \|\cdot\|_2 ∥ ⋅ ∥ ∗ = ∥ ⋅ ∥ 2 and κ = 1 \kappa= 1 κ = 1 , and (15) coincides, up to logarithmic factors, with the time complexity of Rennala SGD [84 ] .23 The assumptions differ, however: our light-tail condition is stronger than bounded variance, and our guarantee holds with high probability rather than in expectation. We therefore read this agreement as a consistency check with the Euclidean theory rather than as an optimality result, and we make no optimality claim for a general update norm.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.