Introduction
Feature learning, informally defined as the ability to extract informative low-dimensional representations out of high-dimensional data, is the defining characteristic of Neural Networks trained with gradient descent —and the basis for the modern AI paradigm of pretraining then fine-tuning. Over the recent years, a feature learning theory has emerged, focusing on a stylized model for high-dimensional learning based on multi-index models [2 , 16 , 12 , 13 , 14 , 3 , 11 ] ; see [6 ] for a recent survey, demonstrating the ability of shallow neural networks to perform efficient feature learning, and even identifying sharp phase transitions [25 ] .
A natural shortcoming for such theory is that shallow NNs are limited idealizations of modern NNs, where depth is a critical component. Towards addressing this gap, several works [1 , 26 , 28 , 15 , 31 ] have studied feature learning in deeper architectures, raising novel questions such as (i) elucidating the hierarchy between features at different layers, or (ii) how to adapt gradient methods to facilitate learning across all layers.
In this work, motivated by applications in data domains such as images or text, we consider a stylized model for deep feature learning based on the principle of scale separation , where each layer of the network can be associated with a different physical scale. In the context of computer vision, the model becomes a Convolutional Neural Network (CNN) with disjoint receptive fields, as originally introduced in [27 ] ; the first hidden-layer extracts local features at each patch, and subsequent layers capture local information at the next scale, progressively coarse-graining the domain [7 , 22 ] .
We instantiate each feature learning stage using a single-index model. Specifically, we consider a setting with Gaussian inputs X ∈ ( R d ) ⊗ K X \in(\mathbb{R}^{d})^{\otimes K} X ∈ ( R d ) ⊗ K , capturing a multiscale structure with K K K scales and ‘patches’ of size d d d , and a target obtained by a K K K -layer network, whereby each layer k ∈ [ K ] k \in[K] k ∈ [ K ] coarse-grains the feature map Z ( k − 1 ) ∈ ( R d ) ⊗ K − k + 1 Z^{(k-1)} \in(\mathbb{R}^{d})^{\otimes K-k+1} Z ( k − 1 ) ∈ ( R d ) ⊗ K − k + 1 as Z ( k ) = ϕ ( Z ( k − 1 ) θ k ∗ ) ∈ ( R d ) ⊗ K − k Z^{(k)} = \phi(Z^{(k-1)}\theta_k^*) \in(\mathbb{R}^{d})^{\otimes K-k} Z ( k ) = ϕ ( Z ( k − 1 ) θ k ∗ ) ∈ ( R d ) ⊗ K − k ; see Figure 1 . The model X ↦ f θ ∗ ( X ) : = Z ( K ) ∈ R X \mapsto f_{\theta^*}(X) := Z^{(K)} \in\mathbb{R} X ↦ f θ ∗ ( X ) := Z ( K ) ∈ R is thus parametrized by hidden directions or features θ 1 ∗ , … , θ K ∗ ∈ S d − 1 \theta_1^*,\ldots,\theta_K^* \in\mathbb{S}^{d-1} θ 1 ∗ , … , θ K ∗ ∈ S d − 1 , and a non-linear activation function ϕ : R → R \phi:\mathbb{R}\to\mathbb{R} ϕ : R → R .
Figure 1. Schema of the Multiscale SIM model. Each patch within the same layer k k k shares the same local feature, parametrized by a SIM of direction θ k \theta_k θ k .
By considering the Wiener chaos expansion of the target f ( X ) f(X) f ( X ) , this model can be viewed as a non-linear analogue of Tensor PCA [24 ] , where the planted directions θ 1 ∗ , … , θ K ∗ \theta_1^*,\ldots,\theta_K^* θ 1 ∗ , … , θ K ∗ may be estimated from a spike in the associated chaos decomposition. This naturally leads to the study of two different algorithmic frameworks: (i) a spectral method, motivated by its efficiency in related problems, eg single- and multi-index models [23 , 13 , 20 ] , spiked tensor PCA, and tensor-decomposition methods for non-overlapping CNNs [27 ] ; and (ii) stochastic gradient-descent, inspired by the practical connection with deep learning. The latter setting is not addressed by these tensor-decomposition analyses and therefore is the main focus of our work.
Our analysis makes two key structural assumptions: we consider the setting where ϕ \phi ϕ is non-degenerate , in the sense that E G ∼ N ( 0 , 1 ) [ ϕ ′ ( G ) ] : = κ ≠ 0 \mathbb{E}_{G\sim\mathcal{N}(0,1)}[\phi'(G)] := \kappa\ne0 E G ∼ N ( 0 , 1 ) [ ϕ ′ ( G )] := κ = 0 1 , and further assume that the planted directions are delocalized , unaligned with the basis on which the non-linear activations act. Exploiting these two assumptions, we analyze online SGD in the case ( K = 2 ) (K=2) ( K = 2 ) , the immediate multiscale extension of the single-index setting, we establish a sample complexity of n = O ( d ( log ( d ) 2 + log ϵ − 1 ) ) n = O(d(\log(d)^2 + \log\epsilon^{-1})) n = O ( d ( log ( d ) 2 + log ϵ − 1 )) to achieve error ϵ \epsilon ϵ , nearly matching the spectral estimator up to logarithmic factor. It is important to emphasize that the machinery of Ben Arous et al. [2 ] does not directly apply in this setting, due to the non-Gaussian features seen by the second hidden layer activations.
Related Works. Understanding the benefits of depth in Neural Networks is a longstanding question in the field, with initial progress centered on approximation questions [30 , 17 ] . The efforts then moved towards incorporating optimization aspects, with early notable works such as [29 , 21 ] , and later developed in [32 , 26 , 1 ] . One aspect shared by all these works is the fact that the input high-dimensional distribution is rotation-invariant, ie with no particular physical structure. Early works that also exploited tensor representations of data to capture scale separation are [10 ] , which focused on expressivity.
The closest works to ours are [15 ] and [28 ] , which we discuss briefly. Dandi et al. [15 ] study Single/Multi-Index Gaussian Hierarchical Targets f ∗ ( x ) = g ∗ ( a ∗ ⊤ P k ( W ∗ x ) / d e 1 ) f^*(x) = g^*(a^{*\top}P_k(W^*x)/\sqrt{d^{e_1}}) f ∗ ( x ) = g ∗ ( a ∗ ⊤ P k ( W ∗ x ) / d e 1 ) with W ∗ ∈ R d e 1 × d W^* \in\mathbb{R}^{d^{e_1}\times d} W ∗ ∈ R d e 1 × d , where the hierarchy is one of dimensions d → d e 1 → 1 d \to d^{e_1} \to1 d → d e 1 → 1 . They train a three-layer MLP layer-wise and propagate Hermite decompositions across layers via the asymptotic Gaussianity of the inner statistic h ∗ h^* h ∗ , which holds because E ( P k ) = 2 \mathbb{E}(P_k)=2 E ( P k ) = 2 . Our setup differs in three essentials: (1) the hierarchy is one of physical scales (within- versus cross-patch), with both layers acting in the same dimension d d d on single-index teachers β ∗ , θ ∗ ∈ S d − 1 \beta^*,\theta^* \in\mathbb{S}^{d-1} β ∗ , θ ∗ ∈ S d − 1 ; (2) both layers are trained jointly by online SGD on the same timescale, so the second-layer student sees a moving, non-Gaussian feature map throughout; and (3) at E ( ϕ ) = 1 \mathbb{E}(\phi)=1 E ( ϕ ) = 1 the inner statistic is not asymptotically Gaussian, so we replace the CLT by a quantitative Stein/Lindeberg comparison driven by the delocalization. Ren et al. [28 ] prove that L L L -layer convolutional networks trained layer-wise learn L L L -level Random Hierarchy Models with O ( m 1 + o ( 1 ) L ) O(m^{1+o(1)}L) O ( m 1 + o ( 1 ) L ) samples, via a problem-independent “shallow-to-deep chaining” reduction: layer-wise training succeeds whenever the target correlates with lower-level latents, the signal to lower layers is clean, and the lower-level features are weakly identifiable. Their setting is complementary to ours: The data are discrete tokens drawn from a probabilistic context-free grammar rather than continuous Gaussian patches, and the targets are categorical class labels rather than scalar regressions, so the relevant geometry is a tree of synonym classes rather than the product sphere. Our companion work [5 ] shows that under correlation loss between teacher model and student model E [ f θ ( X ) f θ ∗ ( X ) ] \mathbb{E}[f_{\theta}(X)f_{\theta^*}(X)] E [ f θ ( X ) f θ ∗ ( X )] when initialized in a favorable basin of constant-in-d d d probability, online SGD achieves achieves weak recovery in O ~ ( d K − 1 ) \widetilde{O}(d^{K-1}) O ( d K − 1 ) samples, and continues to 1 − α d ( 1 ) 1-\alpha_d(1) 1 − α d ( 1 ) overlap after a lower-order continuation phase, for any fixed depth K K K . Yet it remains unclear how the online SGD behaves under standard loss such as Mean Square Error (MSE) loss, which is the main focus of this work.
Notation: All expectations E X \mathbb{E}_X E X are with respect to X X X , with the directions held fixed, unless explicitly stated otherwise. For unit vectors u , v u,v u , v , define the sign-invariant distance dist ± ( u , v ) : = min { ∥ u − v ∥ 2 , ∥ u + v ∥ 2 } \operatorname{dist}_{\pm}(u,v) := \min\{\lVert u-v\rVert_2,\lVert u+v\rVert_2\} dist ± ( u , v ) := min {∥ u − v ∥ 2 , ∥ u + v ∥ 2 } .
The Multiscale Single-Index Model
Let K ≥ 1 K \ge1 K ≥ 1 be fixed. We model the input as a K K K -th order Gaussian tensor of dimension d d d , ie X ∈ ( R d ) ⊗ K X \in(\mathbb{R}^d)^{\otimes K} X ∈ ( R d ) ⊗ K have independent N ( 0 , 1 ) N(0,1) N ( 0 , 1 ) entries. Given unit vectors θ 1 , … , θ K ∈ S d − 1 \theta_1,\ldots,\theta_K \in S^{d-1} θ 1 , … , θ K ∈ S d − 1 , define Z ( 0 ) = X Z^{(0)} = X Z ( 0 ) = X , and recursively
Y i r + 1 , … , i K ( r ) = ∑ j = 1 d θ r ( j ) Z j , i r + 1 , … , i K ( r − 1 ) , 1 ≤ r ≤ K , (1) Y^{(r)}_{i_{r+1},\ldots,i_K} = \sum_{j=1}^{d} \theta_r(j) Z^{(r-1)}_{j,i_{r+1},\ldots,i_K}, \qquad1 \le r \le K, \tag*{(1)} Y i r + 1 , … , i K ( r ) = j = 1 ∑ d θ r ( j ) Z j , i r + 1 , … , i K ( r − 1 ) , 1 ≤ r ≤ K , (1)
Z ( r ) = ϕ ( Y ( r ) ) entrywise . Z^{(r)} = \phi(Y^{(r)}) \quad\text{entrywise}. Z ( r ) = ϕ ( Y ( r ) ) entrywise .
At the final layer Z ( K ) ∈ R Z^{(K)} \in\mathbb{R} Z ( K ) ∈ R , and we write f θ ( X ) : = Z ( K ) f_\theta(X) := Z^{(K)} f θ ( X ) := Z ( K ) . As illustrated in Figure 1 , this is a stylized model of a NN with physical structure: each layer computes local and translation-invariant features at a fixed spatial scale. In the case where K = 2 K = 2 K = 2 , for instance, the first layer extracts a single shared ‘local’ feature θ 1 ∗ \theta_1^* θ 1 ∗ capturing internal correlations within each patch, while the second layer extracts a coarse feature θ 2 ∗ \theta_2^* θ 2 ∗ capturing correlations across patches.
Our algorithmic approaches will be based on two main assumptions:
Assumption 2.1 (Non-degeneracy and regularity, informal). Let G ∼ N ( 0 , 1 ) G \sim N(0,1) G ∼ N ( 0 , 1 ) . We assume
E ϕ ( G ) = 0 , E ϕ ( G ) 2 = 1 , κ : = E ϕ ′ ( G ) ≠ 0. (2) \mathbb{E}\phi(G) = 0,\qquad\mathbb{E}\phi(G)^2 = 1,\qquad\kappa:= \mathbb{E}\phi'(G) \ne0. \tag*{(2)} E ϕ ( G ) = 0 , E ϕ ( G ) 2 = 1 , κ := E ϕ ′ ( G ) = 0. (2) Further, ϕ \phi ϕ satisfies certain standard regularity assumptions, which differ slightly on each setting (discussed in each section).
In the language of Ben Arous et al. [2 ] , non-degeneracy corresponds to the case where the information exponent of ϕ \phi ϕ is 1. In the shallow setting K = 1 K = 1 K = 1 , this assumption greatly simplifies the geometry of the model, ensuring a descent direction from random initializations. In our setting K > 1 K > 1 K > 1 , the non-degeneracy is instrumental for gradient methods to escape the mediocrity, as will be shown in later sections.
Assumption 2.2 (Incoherence, informal). The planted directions are delocalized, in the sense that ∥ θ k ∥ ∞ ≪ 1 \lVert\theta_k\rVert_\infty\ll1 ∥ θ k ∥ ∞ ≪ 1 .
Incoherence corresponds to the notion that no patch exerts an dominant influence on the output, and enables a mean-field approximation, or equivalently a Gaussian universality in our SGD analysis, similarly as in [18 ] .
Main Result
We now turn to verify that, indeed, K = 2 K = 2 K = 2 is a tractable setting for SGD dynamics to achieve strong recovery under the natural MSE loss rather than linear correlation in our companion paper [5 ] . For ease of presentation, we name θ 1 ∗ = θ ∗ \theta_1^* = \theta^* θ 1 ∗ = θ ∗ and θ 2 ∗ = β ∗ \theta_2^* = \beta^* θ 2 ∗ = β ∗ in the remainder of this section.
Setup
Given the teacher direction ( θ ∗ , β ∗ ) ∈ S d − 1 × S d − 1 (\theta^*, \beta^*) \in\mathbb{S}^{d-1} \times\mathbb{S}^{d-1} ( θ ∗ , β ∗ ) ∈ S d − 1 × S d − 1 we want to study the ability of online stochastic gradient descent (online SGD) to minimize the population loss
L ( β t , θ t ) : = ∥ f β t , θ t − f β ∗ , θ ∗ ∥ 2 = E x ∼ γ D [ f β t , θ t ( x ) − f β ∗ , θ ∗ ( x ) ] 2 , (3) \mathcal{L}(\beta_t, \theta_t) := \lVert f_{\beta_t,\theta_t} - f_{\beta^*,\theta^*} \rVert^2 = \mathbb{E}_{x \sim\gamma_D}[f_{\beta_t,\theta_t}(x) - f_{\beta^*,\theta^*}(x)]^2, \tag*{(3)} L ( β t , θ t ) := ∥ f β t , θ t − f β ∗ , θ ∗ ∥ 2 = E x ∼ γ D [ f β t , θ t ( x ) − f β ∗ , θ ∗ ( x ) ] 2 , (3)
where the label is generated by z i : = ϕ ( X i ⋅ β ∗ ) z_i := \phi(X_i \cdot\beta^*) z i := ϕ ( X i ⋅ β ∗ ) , f β ∗ , θ ∗ ( X ) : = ψ ( z ⋅ θ ∗ ) f_{\beta^*,\theta^*}(X) := \psi(z \cdot\theta^*) f β ∗ , θ ∗ ( X ) := ψ ( z ⋅ θ ∗ ) , i ∈ [ d ] i \in[d] i ∈ [ d ] . To recover the teacher, we run online SGD on S d − 1 × S d − 1 \mathbb{S}^{d-1} \times\mathbb{S}^{d-1} S d − 1 × S d − 1 :
β t + 1 = β t − δ ∇ β S ℓ ( β t , θ t ; x t ) ∥ β t − δ ∇ β S ℓ ( β t , θ t ; x t ) ∥ , θ t + 1 = θ t − δ ∇ θ S ℓ ( β t , θ t ; x t ) ∥ θ t − δ ∇ θ S ℓ ( β t , θ t ; x t ) ∥ , (4) \beta_{t+1} = \frac{\beta_t - \delta\nabla^S_\beta\ell(\beta_t,\theta_t;x_t)}{\lVert\beta_t - \delta\nabla^S_\beta\ell(\beta_t,\theta_t;x_t)\rVert}, \qquad\theta_{t+1} = \frac{\theta_t - \delta\nabla^S_\theta\ell(\beta_t,\theta_t;x_t)}{\lVert\theta_t - \delta\nabla^S_\theta\ell(\beta_t,\theta_t;x_t)\rVert}, \tag*{(4)} β t + 1 = ∥ β t − δ ∇ β S ℓ ( β t , θ t ; x t )∥ β t − δ ∇ β S ℓ ( β t , θ t ; x t ) , θ t + 1 = ∥ θ t − δ ∇ θ S ℓ ( β t , θ t ; x t )∥ θ t − δ ∇ θ S ℓ ( β t , θ t ; x t ) , (4)
for t = 0 , 1 , 2 , … t = 0,1,2,\ldots t = 0 , 1 , 2 , … , where ( x t ∈ R d × d ) (x_t \in\mathbb{R}^{d \times d}) ( x t ∈ R d × d ) are fresh samples independent of the past, δ \delta δ is the step-size, and the spherical gradient is defined as:
∇ β S ℓ ( β , θ ; x ) : = ( I − β β T ) ∇ β ℓ ( β , θ ; x ) , ∇ θ S ℓ ( β , θ ; x ) : = ( I − θ θ T ) ∇ θ ℓ ( β , θ ; x ) . \nabla^S_\beta\ell(\beta,\theta;x) := (I - \beta\beta^\mathsf{T})\nabla_\beta\ell(\beta,\theta;x), \qquad\nabla^S_\theta\ell(\beta,\theta;x) := (I - \theta\theta^\mathsf{T})\nabla_\theta\ell(\beta,\theta;x). ∇ β S ℓ ( β , θ ; x ) := ( I − β β T ) ∇ β ℓ ( β , θ ; x ) , ∇ θ S ℓ ( β , θ ; x ) := ( I − θ θ T ) ∇ θ ℓ ( β , θ ; x ) .
Following [2 , 8 ] , throughout the learning process, we track the evolution of summary statistics, defined as overlaps ρ t : = β t ⋅ β ∗ \rho_t := \beta_t \cdot\beta^* ρ t := β t ⋅ β ∗ and χ t : = θ t ⋅ θ ∗ \chi_t := \theta_t \cdot\theta^* χ t := θ t ⋅ θ ∗ for the first and second layer’s parameter respectively.
Theorem 3.1 (Informal). With our assumptions (A1)-(A4) (see Section B), for any ϵ > 0 \epsilon> 0 ϵ > 0 , under online SGD training (3), we have:
Weak Recovery: starting from initialization, with probability at least 1 − K ϵ 1 - K\epsilon 1 − K ϵ , choosing step size δ ≍ 1 / ( d log d ) \delta\asymp1/(d\log d) δ ≍ 1/ ( d log d ) , SGD achieves weak recovery: min { ρ t , χ t } = Θ ( 1 ) \min\{\rho_t,\chi_t\} = \Theta(1) min { ρ t , χ t } = Θ ( 1 ) , using O ( d log 2 d ) O(d\log^2 d) O ( d log 2 d ) samples.
Strong Recovery: conditioned on the weak recovery event, with probability at least 1 − O ( log ( 1 / ϵ ) / d ) 1 - O(\log(1/\epsilon)/d) 1 − O ( log ( 1/ ϵ ) / d ) , choosing step size δ ≍ 1 / d \delta\asymp1/d δ ≍ 1/ d , SGD achieves strong recovery: min { ρ t , χ t } ≥ 1 − ϵ \min\{\rho_t,\chi_t\} \ge1 - \epsilon min { ρ t , χ t } ≥ 1 − ϵ , using O ( d log ( 1 / ϵ ) ) O(d\log(1/\epsilon)) O ( d log ( 1/ ϵ )) additional samples.
We build our analysis on top of the analytical framework introduced in Ben Arous et al. [2 ] by modeling the learning process as drifts with fluctuations. The main technical point is to leverage delocalization (which, as remarked in B.2, will be true w.h.p. under (A4)) to control the gaussian approximation L G \mathcal{L}_G L G of the true loss L \mathcal{L} L to derive the continuous-population level on the gradient flow dynamics: Based on Stein’s Coupling (see Theorem E.1 ), for each layer, in Lemma C.2, we bound the distance between the spherical gradient of the true loss ∇ β S L \nabla^S_\beta\mathcal{L} ∇ β S L and its Gaussian proxy ∇ β S L G \nabla^S_\beta\mathcal{L}_G ∇ β S L G by some constant factor of the incoherence η t : = ∥ θ t ∥ ∞ + ∥ θ ∗ ∥ ∞ \eta_t := \lVert\theta_t\rVert_\infty+ \lVert\theta^*\rVert_\infty η t := ∥ θ t ∥ ∞ + ∥ θ ∗ ∥ ∞ that holds true uniformly in time:
∣ ⟨ ∇ β S ( L − L G ) ( θ t , β t ) , β ∗ ⟩ ∣ ≤ C β ( 1 ) η t , ∣ ⟨ ∇ θ S ( L − L G ) ( θ t , β t ) , θ ∗ ⟩ ∣ ≤ C θ ( 1 ) η t . (5) \left|\left\langle\nabla^S_\beta(\mathcal{L} - \mathcal{L}_G)(\theta_t,\beta_t),\beta^*\right\rangle\right| \le C_\beta^{(1)}\eta_t,\qquad\left|\left\langle\nabla^S_\theta(\mathcal{L} - \mathcal{L}_G)(\theta_t,\beta_t),\theta^*\right\rangle\right| \le C_\theta^{(1)}\eta_t. \tag*{(5)} ⟨ ∇ β S ( L − L G ) ( θ t , β t ) , β ∗ ⟩ ≤ C β ( 1 ) η t , ⟨ ∇ θ S ( L − L G ) ( θ t , β t ) , θ ∗ ⟩ ≤ C θ ( 1 ) η t . (5)
Continuous-time Analysis In continuous time, we consider the population-level spherical gradient flow defined as
β ˙ t = − ∇ β S L ( θ t , β t ) , θ ˙ t = − ∇ θ S L ( θ t , β t ) . (6) \dot{\beta}_t = -\nabla^S_\beta\mathcal{L}(\theta_t,\beta_t),\qquad\dot{\theta}_t = -\nabla^S_\theta\mathcal{L}(\theta_t,\beta_t). \tag*{(6)} β ˙ t = − ∇ β S L ( θ t , β t ) , θ ˙ t = − ∇ θ S L ( θ t , β t ) . (6)
Based on (4), in Lemma C.3 we derive a system of coupled ODE inequalities that captures the evolution of overlaps ρ t \rho_t ρ t and χ t \chi_t χ t :
ρ ˙ t ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t , χ ˙ t ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t , (7) \dot{\rho}_t \ge a_\rho\chi_t(1-\rho_t^2) - b_\rho\eta_t,\qquad\dot{\chi}_t \ge a_\chi\rho_t(1-\chi_t^2) - b_\chi\eta_t, \tag*{(7)} ρ ˙ t ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t , χ ˙ t ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t , (7)
where a ρ , a χ , b ρ , b χ > 0 a_\rho,a_\chi,b_\rho,b_\chi> 0 a ρ , a χ , b ρ , b χ > 0 are independent of d d d and ( θ t , β t ) (\theta_t,\beta_t) ( θ t , β t ) . From here, we can see that for the RHS to be non-vacuous and provide enough signal for the growth of ( ρ t , θ t ) (\rho_t,\theta_t) ( ρ t , θ t ) , terms b ρ η t b_\rho\eta_t b ρ η t and b χ η t b_\chi\eta_t b χ η t must not exceed a ρ χ t ( 1 − ρ t 2 ) a_\rho\chi_t(1-\rho_t^2) a ρ χ t ( 1 − ρ t 2 ) and a χ ρ t ( 1 − χ t 2 ) a_\chi\rho_t(1-\chi_t^2) a χ ρ t ( 1 − χ t 2 ) , respectively. This requires us to provide a quantitative control of η t \eta_t η t , derived in Lemma C.4 :
η ˙ t ≤ C 1 ρ t ∥ θ ∗ ∥ ∞ + C 2 χ t ρ t η t + O ( η t 2 ) . (8) \dot{\eta}_t \le C_1\rho_t\lVert\theta^*\rVert_\infty+ C_2\chi_t\rho_t\eta_t + O(\eta_t^2). \tag*{(8)} η ˙ t ≤ C 1 ρ t ∥ θ ∗ ∥ ∞ + C 2 χ t ρ t η t + O ( η t 2 ) . (8)
We make the following remarks on our setting:
On the Delocalization hypothesis: our proof crucially relies on the Gaussian comparison in order to obtain an overlap growth differential inequality. Delocalization is not necessary for efficient recovery, e.g. if ∥ θ ∗ ∥ 0 = O d ( 1 ) \|\theta^*\|_0 = O_d(1) ∥ θ ∗ ∥ 0 = O d ( 1 ) the target is a multi-index model, for which gradient-based guarantees exist [12 , 4 ] . A natural follow-up question is to understand whether our SGD can obtain recovery guarantees uniform on θ ∗ \theta^* θ ∗ .
ρ ˙ t ≥ a ρ χ t ρ t s − 1 ( 1 − ρ t 2 ) − b ρ η t , χ ˙ t ≥ a χ ρ t χ t s − 1 ( 1 − χ t 2 ) − b χ η t , (9) \dot{\rho}_t \ge a_\rho\chi_t \rho_t^{s-1}(1-\rho_t^2)-b_\rho\eta_t,\qquad\dot{\chi}_t \ge a_\chi\rho_t \chi_t^{s-1}(1-\chi_t^2)-b_\chi\eta_t, \tag*{(9)} ρ ˙ t ≥ a ρ χ t ρ t s − 1 ( 1 − ρ t 2 ) − b ρ η t , χ ˙ t ≥ a χ ρ t χ t s − 1 ( 1 − χ t 2 ) − b χ η t , (9)
in which the perturbative η t \eta_t η t term dominates the positive drift at initialization. This difficulty is not merely technical; our numerical simulations in Section A show that online SGD does not manage to learn in this setting.
On Gaussian inputs For simplicity we state our result in the Gaussian setting, but using delocalization in the first layer feature, our arguments naturally extend to the class of well-behaved product measures for which Stein’s CLT coupling applies.
Main Elements of the Proof
Below, we present the high-level roadmap of proofs in getting Theorem 3.1.
Weak Recovery
The proof has two dynamical regimes, Phase I and Phase II, separated by
τ A : = inf { t : Z t 2 ≥ A η t } , Z t : = ρ t + χ t . \tau_A := \inf\{t: Z_t^2 \ge A\eta_t\},\qquad Z_t := \rho_t+\chi_t. τ A := inf { t : Z t 2 ≥ A η t } , Z t := ρ t + χ t .
In Phase I, Z t Z_t Z t is still below the incoherence scale. In this phase, the crude O ( η t ) O(\eta_t) O ( η t ) comparison error is too large under spherical initialization; in Lemma C.5 we bound the deviation of true population drifts from its Gaussian proxy by a refined third-moment quantity
ω t : = ∑ i = 1 d ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 , (10) \omega_t := \sum_{i=1}^{d} \left(|\theta_{t,i}|+|\theta_i^*|\right)^3, \tag*{(10)} ω t := i = 1 ∑ d ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 , (10)
which itself is controlled in Lemma C.6 and Lemma C.7. Under the small-overlap regime of weak recovery, the dynamics is well approximated by the coupled drift inequalities
ρ ˙ t ≥ a χ t − b ω t , χ ˙ t ≥ a ρ t − b ω t . (11) \dot{\rho}_t \ge a\chi_t-b\omega_t,\qquad\dot{\chi}_t \ge a\rho_t-b\omega_t. \tag*{(11)} ρ ˙ t ≥ a χ t − b ω t , χ ˙ t ≥ a ρ t − b ω t . (11)
In Lemma C.8, we showed that (i) starting from initialization, (10) pulls Z t Z_t Z t from d − 1 / 2 d^{-1/2} d − 1/2 to the signal-dominant scale Z t 2 ∼ η t Z_t^2\sim\eta_t Z t 2 ∼ η t in O ( log d ) O(\log d) O ( log d ) time; (ii) after τ A \tau_A τ A , the incoherence error is of a lower order compared with the positive Z Z Z -drift. The sum Z t Z_t Z t then grows from order d − 1 / 4 d^{-1/4} d − 1/4 to a fixed small constant, while the imbalance D t = ρ t − χ t D_t=\rho_t-\chi_t D t = ρ t − χ t contracts. The entry estimate D τ A = o ( Z τ A ) D_{\tau_A}=o(Z_{\tau_A}) D τ A = o ( Z τ A ) ensures that both overlaps become positive constants simultaneously under another O ( log d ) O(\log d) O ( log d ) time, obtaining weak recovery.
Extension to Discrete-Time SGD The discrete-time weak-recovery proof mirrors the same two-regime mechanism, with additional martingale and discretization controls: We start from the one-step expansion in Lemma D.2,
ρ t + 1 = ρ t + δ F t ( ρ ) + δ M t ( ρ ) + R t ( ρ ) χ t + 1 = χ t + δ F t ( χ ) + δ M t ( χ ) + R t ( χ ) , (12) \rho_{t+1}=\rho_t+\delta F_t^{(\rho)}+\delta M_t^{(\rho)}+R_t^{(\rho)}
\qquad
\chi_{t+1}=\chi_t+\delta F_t^{(\chi)}+\delta M_t^{(\chi)}+R_t^{(\chi)}, \tag*{(12)} ρ t + 1 = ρ t + δ F t ( ρ ) + δ M t ( ρ ) + R t ( ρ ) χ t + 1 = χ t + δ F t ( χ ) + δ M t ( χ ) + R t ( χ ) , (12)
where we take the step size δ ≲ 1 / d log d \delta\lesssim1/d\log d δ ≲ 1/ d log d . During Phase I, plugging the ω \omega ω -controlled population drift in Lemma C.6 into (11), we obtained the aggregate one-step update expression
Z t + 1 ≥ ( 1 + β δ ) Z t − C δ ω t + ϵ t ( Z ) , (13) Z_{t+1}\ge(1+\beta\delta)Z_t-C\delta\omega_t+\epsilon_t^{(Z)}, \tag*{(13)} Z t + 1 ≥ ( 1 + β δ ) Z t − C δ ω t + ϵ t ( Z ) , (13)
where ϵ t ( Z ) \epsilon_t^{(Z)} ϵ t ( Z ) as the one-step increment of martingale and discretization errors, with their time-accumulation is controlled by Abel summation from the event E T ( Z ) E_T^{(Z)} E T ( Z ) . Besides, for the third-moment term ω t \omega_t ω t in (12), under Lemma D.12 as the discretized version of third-moment control made in Lemma C.7 we showed that for Phase I, ω t ≲ ( 1 + t δ ) / d \omega_t\lesssim(1+t\delta)/\sqrt{d} ω t ≲ ( 1 + t δ ) / d , making the weighted forcing remains of order d − 1 / 2 d^{-1/2} d − 1/2 . Therefore Z t Z_t Z t grows from its initialization scale d − 1 / 2 d^{-1/2} d − 1/2 to the signal-dominant scale Z τ A ≳ d − 1 / 4 Z_{\tau_A} \gtrsim d^{-1/4} Z τ A ≳ d − 1/4 in O ( δ − 1 log d ) O(\delta^{-1}\log d) O ( δ − 1 log d ) iterations, while η t ≲ ( log d ) / d \eta_t \lesssim(\log d)/\sqrt{d} η t ≲ ( log d ) / d . Same as continuous case, we arrived at an entry-balance estimate D τ A = o ( Z τ A ) D_{\tau_A}=o(Z_{\tau_A}) D τ A = o ( Z τ A ) using the small-overlap D D D -drift derived in D.15.
Entering Phase II, as shown in Lemma D.16, once Z t 2 ≳ η t Z_t^2 \gtrsim\eta_t Z t 2 ≳ η t , the incoherence error is negligible compared with the positive Z Z Z -drift, so Z t Z_t Z t grows from order d − 1 / 4 d^{-1/4} d − 1/4 to a fixed constant c Z c_Z c Z within another O ( δ − 1 log d ) O(\delta^{-1}\log d) O ( δ − 1 log d ) iterations. At the same time, the D D D -recursion is contractive up to lower-order forcing, which keeps ∣ D t ∣ ≤ Z t / 2 |D_t| \le Z_t/2 ∣ D t ∣ ≤ Z t /2 . Hence, as shown in Theorem D.17 , at the hitting time t e s c : = inf { t : Z t ≥ c Z } t_{\mathrm{esc}}:=\inf\{t: Z_t\ge c_Z\} t esc := inf { t : Z t ≥ c Z } , both overlaps are bounded below with high probability:
min { ρ t e s c , χ t e s c } = Z t e s c − ∣ D t e s c ∣ 2 ≥ c Z 4 . \min\{\rho_{t_{\mathrm{esc}}},\chi_{t_{\mathrm{esc}}}\}=\frac{Z_{t_{\mathrm{esc}}}-|D_{t_{\mathrm{esc}}}|}{2}\ge\frac{c_Z}{4}. min { ρ t esc , χ t esc } = 2 Z t esc − ∣ D t esc ∣ ≥ 4 c Z .
Strong Recovery
Continuous-time Strong Recovery Results After weak recovery, we show in Lemma C.9 that under additional O ( 1 ) O(1) O ( 1 ) time, the population dynamics drives ( θ t , β t ) (\theta_t,\beta_t) ( θ t , β t ) into a fixed local cap of arbitrary level around y ∗ : = ( θ ∗ , β ∗ ) y^*:=(\theta^*,\beta^*) y ∗ := ( θ ∗ , β ∗ ) , the global minimizer of the true population loss. While the small-overlap estimate from weak recovery no longer holds, we employ the uniform bounds in Lemma C.3 and Lemma C.4 to show η t = o d ( 1 ) \eta_t=o_d(1) η t = o d ( 1 ) , which guarantees both overlaps enter the claimed local cap.
Once inside this local cap, the proof switches from overlap drift to local geometry. We analyze the Hessian of the loss function by decomposing it as:
H L ( y ) = ( H L ( y ) − H L ( y ∗ ) ) ⏟ ( A ) + ( H L ( y ∗ ) − H L G ( y ∗ ) ) ⏟ ( B ) + H L G ( y ∗ ) ⏟ ( C ) . (14) H_L(y)=\underbrace{\left(H_L(y)-H_L(y^*)\right)}_{(\mathrm{A})}+\underbrace{\left(H_L(y^*)-H_{L_G}(y^*)\right)}_{(\mathrm{B})}+\underbrace{H_{L_G}(y^*)}_{(\mathrm{C})}. \tag*{(14)} H L ( y ) = ( A ) ( H L ( y ) − H L ( y ∗ ) ) + ( B ) ( H L ( y ∗ ) − H L G ( y ∗ ) ) + ( C ) H L G ( y ∗ ) . (14)
From (14) , we derive Lemma C.12 to lower bound term (B) by some constant factor of − ∥ θ ∗ ∥ ∞ -\|\theta^*\|_\infty − ∥ θ ∗ ∥ ∞ , and Lemma C.13 to lower bound term (A) by some constant factor of − dist M ( y , y ∗ ) : = − arccos ( ρ ) 2 + arccos ( χ ) 2 -\operatorname{dist}_{\mathcal{M}}(y,y^*):=-\sqrt{\arccos(\rho)^2+\arccos(\chi)^2} − dist M ( y , y ∗ ) := − arccos ( ρ ) 2 + arccos ( χ ) 2 , taken arbitrarily small. Finally, as (C) is lower bounded by a Hermite-coefficient quantity that dominates the other constants, in Lemma C.14, we obtain the positivity of the Hessian for the true loss H L ( y ) H_L(y) H L ( y ) throughout the local cap.
Hence, under this implied local strong convexity, in Lemma C.15 , by exploiting the the log-drift inequality
⟨ ∇ M L ( y t ) , Log y t ( y ∗ ) ⟩ ≤ − 2 μ l o c Q t , Q t : = 1 2 arccos ( χ t ) 2 + 1 2 arccos ( ρ t ) 2 , (15) \langle\nabla_{\mathcal{M}}L(y_t),\operatorname{Log}_{y_t}(y^*)\rangle\le-2\mu_{\mathrm{loc}}Q_t,\qquad Q_t:=\frac{1}{2}\arccos(\chi_t)^2+\frac{1}{2}\arccos(\rho_t)^2, \tag*{(15)} ⟨ ∇ M L ( y t ) , Log y t ( y ∗ )⟩ ≤ − 2 μ loc Q t , Q t := 2 1 arccos ( χ t ) 2 + 2 1 arccos ( ρ t ) 2 , (15)
we make a standard Riemannian gradient-flow argument to obtain the exponential contraction of dist M ( y , y ∗ ) \operatorname{dist}_{\mathcal{M}}(y,y^*) dist M ( y , y ∗ ) , in the sense that
dist M ( y t , y ∗ ) 2 ≤ e − 2 μ l o c ( t − t e s c ) dist M ( y t e s c , y ∗ ) 2 . (16) \operatorname{dist}_{\mathcal{M}}(y_t,y^*)^2\le e^{-2\mu_{\mathrm{loc}}(t-t_{\mathrm{esc}})}\operatorname{dist}_{\mathcal{M}}(y_{t_{\mathrm{esc}}},y^*)^2. \tag*{(16)} dist M ( y t , y ∗ ) 2 ≤ e − 2 μ loc ( t − t esc ) dist M ( y t esc , y ∗ ) 2 . (16)
Converting the product distance back to overlap gives the strong recovery of
min { ρ t , χ t } ≥ 1 − ε (17) \min\{\rho_t,\chi_t\}\ge1-\varepsilon \tag*{(17)} min { ρ t , χ t } ≥ 1 − ε (17)
after an additional O ( log ( 1 / ε ) ) O(\log(1/\varepsilon)) O ( log ( 1/ ε )) time from weak recovery. We formalize this result in Theorem C.16 .
Extension to Discrete-Time SGD Same as the extension to discrete-time SGD in § 0.3.2.1 , here we must control stochastic fluctuations of SGD inside the strongly-convex basin to ensure the local contraction. Finishing the discrete-time weak recovery gave by Theorem D.17 , we switch the SGD step size to δ l o c = c l o c d \delta_{\mathrm{loc}}=\frac{c_{\mathrm{loc}}}{d} δ loc = d c loc , that is optimal to strong recovery as remarked in Remark D.18.
Mirroring Lemma C.9 , Proposition D.21 shows that after an additional O ( d ) O(d) O ( d ) iterations beyond weak recovery, the trajectory enters the local strongly-convex cap in discrete-time. Inside this cap, we track the local distance
Q t : = 1 2 arccos ( χ t ) 2 + 1 2 arccos ( ρ t ) 2 . (18) Q_t:=\frac{1}{2}\arccos(\chi_t)^2+\frac{1}{2}\arccos(\rho_t)^2. \tag*{(18)} Q t := 2 1 arccos ( χ t ) 2 + 2 1 arccos ( ρ t ) 2 . (18)
From its one-step recursion in Lemma D.22 :
Q t + 1 ≤ Q t + δ l o c ⟨ g ˉ t , ξ t ⟩ + δ l o c Ξ t + C r e t δ l o c 2 G t , (19) Q_{t+1}\le Q_t+\delta_{\mathrm{loc}}\langle\bar{g}_t,\xi_t\rangle+\delta_{\mathrm{loc}}\Xi_t+C_{\mathrm{ret}}\delta_{\mathrm{loc}}^2G_t, \tag*{(19)} Q t + 1 ≤ Q t + δ loc ⟨ g ˉ t , ξ t ⟩ + δ loc Ξ t + C ret δ loc 2 G t , (19)
we bound the population drift ⟨ g ˉ t , L t ⟩ \langle\bar{g}_t,L_t\rangle ⟨ g ˉ t , L t ⟩ by the log-drift inequality (15) in continuous time, and we bound both the martingale fluctuation term Ξ t : = ⟨ g t − g ˉ t , L t ⟩ \Xi_t := \langle g_t-\bar{g}_t,L_t\rangle Ξ t := ⟨ g t − g ˉ t , L t ⟩ and the quadratic gradient term G t : = ∥ g t ∥ 2 G_t := \lVert g_t\rVert^2 G t := ∥ g t ∥ 2 in Lemma D.23 by the sharp local self-quenching estimate:
E [ Ξ t 2 ∣ F t ] ≲ Q t 2 , and E [ G t ∣ F t ] ≲ d Q t , (20) \mathbb{E}[\Xi_t^2 \mid\mathcal{F}_t] \lesssim Q_t^2,\quad\text{and}\quad\mathbb{E}[G_t \mid\mathcal{F}_t] \lesssim dQ_t, \tag*{(20)} E [ Ξ t 2 ∣ F t ] ≲ Q t 2 , and E [ G t ∣ F t ] ≲ d Q t , (20)
which show that both the gradient energy and the martingale variance vanish proportionally to the current distance to the teacher. Thus the SGD noise becomes weaker as the iterates approach the teacher: Proposition D.24 proves that one local SGD epoch of length O ( δ l o c − 1 ) O(\delta_{\mathrm{loc}}^{-1}) O ( δ loc − 1 ) contracts the distance scale by a factor of 1 / 2 1/2 1/2 with probability 1 − O ( δ l o c ) 1-O(\delta_{\mathrm{loc}}) 1 − O ( δ loc ) , while the trajectory remains inside the local cap. Iterating dyadic epochs in Proposition D.25 yields Q t ≤ ϵ Q_t \le\epsilon Q t ≤ ϵ after O ( d log 1 ϵ ) O(d\log\frac{1}{\epsilon}) O ( d log ϵ 1 ) additional samples. We conclude our discrete-time result in Theorem D.26, where we obtained (17) , namely the strong recovery under discrete-time SGD, with a total sample complexity of O ( d log 2 d ) + O ( d log 1 ϵ ) O(d\log^2 d)+O(d\log\frac{1}{\epsilon}) O ( d log 2 d ) + O ( d log ϵ 1 ) .
Conclusions and Future Directions
In this work, we studied the Multiscale Single-Index Model, in which a d K d^K d K -dimensional input with multiscale structure is iteratively coarse-grained via local, translation-invariant single-index models, and where the number of scales K K K corresponds directly to the depth of the associated neural network. Provided the non-linearities are non-degenerate and the planted directions are delocalized, we proved weak and strong recovery of online SGD in the three-layer setting K = 2 K=2 K = 2 using O ( d ( log ( d ) ) 2 ) O(d(\log(d))^2) O ( d ( log ( d ) ) 2 ) samples.
Our work leaves several open questions ahead of us: the first, most pressing one is to extend the SGD recoverability to general depth K > 2 K>2 K > 2 . While [5 ] provides d d d -asymptotic strong recovery for general K K K under the correlation loss, it remains unclear how a general K K K model behaves under canonical regression objectives such as MSE loss. In our current SGD analysis, the restriction to K = 2 K=2 K = 2 is tied to the weak-recovery phase near random initialization. Indeed, the MSE objective decomposes as
L M S E ( Θ ) = E [ f Θ ( X ) 2 ] − 2 E [ f Θ ( X ) f Θ ⋆ ( X ) ] + E [ f Θ ⋆ ( X ) 2 ] . (21) \mathcal{L}_{\mathrm{MSE}}(\Theta)=\mathbb{E}\left[f_{\Theta}(X)^2\right]-2\mathbb{E}\left[f_{\Theta}(X)f_{\Theta^\star}(X)\right]+\mathbb{E}\left[f_{\Theta^\star}(X)^2\right]. \tag*{(21)} L MSE ( Θ ) = E [ f Θ ( X ) 2 ] − 2 E [ f Θ ( X ) f Θ ⋆ ( X ) ] + E [ f Θ ⋆ ( X ) 2 ] . (21)
When m j = ⟨ θ j , θ j ⋆ ⟩ ≲ d − 1 / 2 m_j=\langle\theta_j,\theta_j^\star\rangle\lesssim d^{-1/2} m j = ⟨ θ j , θ j ⋆ ⟩ ≲ d − 1/2 , the teacher-dependent correlation drift in mode i i i has size ∏ ℓ ≠ i m ℓ ≲ d − ( K − 1 ) / 2 \prod_{\ell\ne i}m_\ell\lesssim d^{-(K-1)/2} ∏ ℓ = i m ℓ ≲ d − ( K − 1 ) /2 . In contrast, our current first-order bounds allow teacher-independent finite-d d d self-energy corrections at scale d − 1 / 2 d^{-1/2} d − 1/2 . For K = 2 K=2 K = 2 , these terms occur at the same scale, and our refined third-moment and incoherence estimates allow us to prove that the resulting two-overlap dynamics still has a positive cooperative drift to escape mediocrity. For K > 2 K>2 K > 2 , however, the planted drift is of strictly smaller order, and a first-order Gaussian or norm comparison no longer determines the sign of the overlap evolution. Moreover, unlike the perturbations in the correlation objective, the self-energy corrections are not naturally accompanied by planted-overlap factors. Figure 3 , Section A shows the empirical evidence of hardness of MSE loss for K > 2 K>2 K > 2 .
Figure 3. Comparison between MSE and Correlation Loss, for various pairs of K , d K, d K , d with d K = 16777216 d^K = 16777216 d K = 16777216 , using ϕ = G e L U \phi= \mathrm{GeLU} ϕ = GeLU .
The next natural question is to consider the regime of fixed d d d (the so-called ‘receptive field’) and diverging depth K K K ; this will also require substantial technical innovation, since our CLT/universality arguments do not directly apply.
Finally, an important limitation of our work is that it imposes a form of scale separation: each layer lives on its dedicated scale. While this is a valid assumption for certain physical data (e.g images), most popular deep architectures do not have scale separation; for instance in ResNets [19 ] /NeuralODEs [9 ] the features remain at the same scale throughout several layers. Building toy models that explain feature learning in these settings remains an outstanding challenge.
Acknowledgments
GD was partially supported by the Dean’s Undergraduate Research Fund from NYU’s College of Arts and Science. This work was completed during the JB sabbatical visit at the Flatiron Institute, which also provided the computing resources used in the project. The authors would also like to thank Hengzhi Li for helping them running experiments, and Dr. Denny Wu, Yunwei Ren, Hanzhe Wu for reviewing our draft and providing helpful feedback.
Appendix
Numerical Experiments
We illustrate the performance of online SGD on several representative experiments. We use minibatch online SGD. The figure captions and experiment descriptions specify the batch sizes and learning rates. We initialize the parameters ( θ ∗ , β ∗ ) (\theta^{*}, \beta^{*}) ( θ ∗ , β ∗ ) drawn i.i.d. from the uniform measure, and we initialize ( θ 0 , β 0 ) (\theta_{0}, \beta_{0}) ( θ 0 , β 0 ) uniformly at random from the non-negative-overlap hemispheres { θ ; θ ⋅ θ ∗ ≥ 0 } \{\theta; \theta\cdot\theta^{*} \ge0\} { θ ; θ ⋅ θ ∗ ≥ 0 } , { β ; β ⋅ β ∗ ≥ 0 } \{\beta; \beta\cdot\beta^{*} \ge0\} { β ; β ⋅ β ∗ ≥ 0 } .
Sample complexity In Figure 2 we plot the recovery performance of online SGD for d ∈ { 20 , 50 , 100 , 200 } d \in\{20, 50, 100, 200\} d ∈ { 20 , 50 , 100 , 200 } , using normalized GELU. We track min ( ∣ χ t ∣ , ∣ ρ t ∣ ) \min(\lvert\chi_t\rvert, \lvert\rho_t\rvert) min (∣ χ t ∣ , ∣ ρ t ∣) over 10 independent seeds and show variability across seeds. As predicted by our theory, the algorithm successfully recovers the planted parameters, at a rate that is roughly n ≃ d log d n \simeq d \log d n ≃ d log d .
Figure 2. Minimum absolute overlap for K = 2 K = 2 K = 2 with normalized GELU at d ∈ { 20 , 50 , 100 , 200 } d \in\{20, 50, 100, 200\} d ∈ { 20 , 50 , 100 , 200 } . We use η = 18.4207 / ( d log d ) \eta= 18.4207/(d \log d) η = 18.4207/ ( d log d ) , batch size 256, and 10 seeds.
Comparing different K K K , d d d pair choices To empirically see the potential of online SGD under MSE loss extends beyond K > 2 K > 2 K > 2 and compare its performance with correlation loss as studied in our companion work [5 ] , we ran a set of experiments with varying K K K . Specifically, based on our model set-up, we consider a fixed dimension of model input, d K = 16777216 d^{K} = 16777216 d K = 16777216 , and tested pair choices ( K , d ) ∈ { ( 2 , 4096 ) , ( 3 , 256 ) , ( 4 , 64 ) , ( 6 , 16 ) } (K, d) \in\{(2, 4096), (3, 256), (4, 64), (6, 16)\} ( K , d ) ∈ {( 2 , 4096 ) , ( 3 , 256 ) , ( 4 , 64 ) , ( 6 , 16 )} for both correlation and MSE loss. All experiments use normalized ϕ = GeLU \phi= \operatorname{GeLU} ϕ = GeLU activations, a learning rate of η = 0.000391 \eta= 0.000391 η = 0.000391 , minibatches of size 16, and 2 27 2^{27} 2 27 training examples across 10 random seeds. As Figure 4 shows, only for K = 2 K = 2 K = 2 do we see reliable recovery for both losses. Their performance diverges substantially as depth increases, with correlation loss exhibits meaningful recovery among all remaining trials.
Figure 4. Minimum absolute overlap for K = 2 K = 2 K = 2 with normalized GELU and with He2 _2 2 . We use d = 64 d = 64 d = 64 , η = 0.02 \eta= 0.02 η = 0.02 , batch size 64, and 10 seeds.
Testing non-degeneracy assumption Finally, we investigate the effect of removing the non-degeneracy assumption, by using instead ϕ = He 2 \phi= \operatorname{He}_2 ϕ = He 2 , which has information exponent 2. Figure 4 shows that, in this setting, online SGD does not make significant progress, even at timescales n ≫ d 2 n \gg d^2 n ≫ d 2 .
Formalized Assumptions
Let G ∼ N ( 0 , 1 ) G \sim N(0,1) G ∼ N ( 0 , 1 ) , and write the Hermite expansions ϕ = ∑ k ≥ 0 a k h k \phi= \sum_{k\geq0}a_kh_k ϕ = ∑ k ≥ 0 a k h k and ψ = ∑ k ≥ 0 α k h k \psi= \sum_{k\geq0}\alpha_kh_k ψ = ∑ k ≥ 0 α k h k in the normalized Hermite basis. We assume:
(A1) Normalization. E ϕ ( G ) = 0 \mathbb{E}\phi(G)=0 E ϕ ( G ) = 0 and E ϕ ( G ) 2 = 1 \mathbb{E}\phi(G)^2=1 E ϕ ( G ) 2 = 1 .
(A2) Non-degeneracy. The first Hermite coefficients are nonzero:
a 1 = E [ G ϕ ( G ) ] = E [ ϕ ′ ( G ) ] ≠ 0 , α 1 = E [ G ψ ( G ) ] = E [ ψ ′ ( G ) ] ≠ 0. a_1 = \mathbb{E}[G\phi(G)] = \mathbb{E}[\phi'(G)] \ne0,\qquad\alpha_1 = \mathbb{E}[G\psi(G)] = \mathbb{E}[\psi'(G)] \ne0. a 1 = E [ Gϕ ( G )] = E [ ϕ ′ ( G )] = 0 , α 1 = E [ G ψ ( G )] = E [ ψ ′ ( G )] = 0.
(A3) Regularity of activation function. The inner activation ϕ ∈ C 3 ( R ) \phi\in C^3(\mathbb{R}) ϕ ∈ C 3 ( R ) and outer activation ψ ∈ C b 4 ( R ) \psi\in C_b^4(\mathbb{R}) ψ ∈ C b 4 ( R ) have finite regularity constants
L ϕ : = ∥ ϕ ′ ∥ ∞ + ∥ ϕ ′ ′ ∥ ∞ + ∥ ϕ ′ ′ ′ ∥ ∞ , L ψ : = max 0 ≤ j ≤ 4 ∥ ψ ( j ) ∥ ∞ . L_\phi:= \|\phi'\|_\infty+ \|\phi''\|_\infty+ \|\phi'''\|_\infty,\qquad L_\psi:= \max_{0\leq j\leq4}\|\psi^{(j)}\|_\infty. L ϕ := ∥ ϕ ′ ∥ ∞ + ∥ ϕ ′′ ∥ ∞ + ∥ ϕ ′′′ ∥ ∞ , L ψ := 0 ≤ j ≤ 4 max ∥ ψ ( j ) ∥ ∞ .
Here C b 4 ( R ) C_b^4(\mathbb{R}) C b 4 ( R ) requires ψ \psi ψ and its derivatives through order four to be bounded; ϕ \phi ϕ itself need not be bounded.
(A4) Initialization. The initial second layer student parameter and the teacher parameter are uniformly sampled from the unit sphere: θ 0 , θ ∗ ∼ Unif ( S d − 1 ) \theta_0,\theta^\ast\sim\operatorname{Unif}(\mathbb{S}^{d-1}) θ 0 , θ ∗ ∼ Unif ( S d − 1 ) and set the initial overlap to be of the order d − 1 / 2 d^{-1/2} d − 1/2 . The initialization distribution of the overlap satisfies
E o v : = { ρ 0 , χ 0 ∈ [ c i n i t d − 1 / 2 , C i n i t d − 1 / 2 ] } . \mathcal{E}_{\mathrm{ov}} := \left\{\rho_0,\chi_0 \in[c_{\mathrm{init}}d^{-1/2},C_{\mathrm{init}}d^{-1/2}]\right\}. E ov := { ρ 0 , χ 0 ∈ [ c init d − 1/2 , C init d − 1/2 ] } .
Below, we made two remarks, for (A3) and for (A4) respectively:
Remark B.1. Assumption (A3) implies all moment conditions used below. Indeed, since ∥ ϕ ′ ∥ ∞ < ∞ \|\phi'\|_\infty< \infty ∥ ϕ ′ ∥ ∞ < ∞ ,
∣ ϕ ( x ) ∣ ≤ ∣ ϕ ( 0 ) ∣ + ∥ ϕ ′ ∥ ∞ ∣ x ∣ . |\phi(x)| \leq|\phi(0)|+\|\phi'\|_\infty|x|. ∣ ϕ ( x ) ∣ ≤ ∣ ϕ ( 0 ) ∣ + ∥ ϕ ′ ∥ ∞ ∣ x ∣.
Hence ϕ ( G ) \phi(G) ϕ ( G ) has finite Gaussian moments of every order. Moreover, for 1 ≤ j ≤ 3 1 \leq j \leq3 1 ≤ j ≤ 3 , the derivatives ϕ ( j ) \phi^{(j)} ϕ ( j ) are bounded. Therefore, for every finite p ≥ 1 p \geq1 p ≥ 1 ,
max 0 ≤ j ≤ 3 E [ ( 1 + ∣ G ∣ ) p ∣ ϕ ( j ) ( G ) ∣ p ] < ∞ , G ∼ N ( 0 , 1 ) , \max_{0\leq j\leq3}\mathbb{E}\left[(1+|G|)^p|\phi^{(j)}(G)|^p\right] < \infty,\qquad G\sim N(0,1), 0 ≤ j ≤ 3 max E [ ( 1 + ∣ G ∣ ) p ∣ ϕ ( j ) ( G ) ∣ p ] < ∞ , G ∼ N ( 0 , 1 ) ,
where ϕ ( 0 ) = ϕ \phi^{(0)}=\phi ϕ ( 0 ) = ϕ . In particular,
E ∣ ϕ ( G ) ∣ 8 < ∞ , E ∣ ϕ ′ ( G ) ∣ 8 < ∞ , \mathbb{E}|\phi(G)|^8 < \infty,\qquad\mathbb{E}|\phi'(G)|^8 < \infty, E ∣ ϕ ( G ) ∣ 8 < ∞ , E ∣ ϕ ′ ( G ) ∣ 8 < ∞ ,
and
E [ ( 1 + ∣ ϕ ( G ) ∣ 2 ) ∣ ϕ ′ ( G ) ∣ 2 ] < ∞ , E [ ∣ ϕ ( G ) ∣ ∣ ϕ ′ ( G ) ∣ 2 ] < ∞ . \mathbb{E}\left[(1+|\phi(G)|^2)|\phi'(G)|^2\right] < \infty,\qquad\mathbb{E}\left[|\phi(G)|\,|\phi'(G)|^2\right] < \infty. E [ ( 1 + ∣ ϕ ( G ) ∣ 2 ) ∣ ϕ ′ ( G ) ∣ 2 ] < ∞ , E [ ∣ ϕ ( G ) ∣ ∣ ϕ ′ ( G ) ∣ 2 ] < ∞.
Finally, if Y Y Y is sub-Gaussian, then ϕ ( Y ) \phi(Y) ϕ ( Y ) is sub-Gaussian, with
∥ ϕ ( Y ) ∥ ψ 2 ≤ C ( ∣ ϕ ( 0 ) ∣ + L ϕ ∥ Y ∥ ψ 2 ) . \lVert\phi(Y)\rVert_{\psi_2} \le C\bigl(\lvert\phi(0)\rvert+ L_\phi\lVert Y\rVert_{\psi_2}\bigr). ∥ ϕ ( Y ) ∥ ψ 2 ≤ C ( ∣ ϕ ( 0 )∣ + L ϕ ∥ Y ∥ ψ 2 ) .
for a universal constant C C C . This follows from the same Lipschitz bound.
Remark B.2. By standard concentration of the uniform measure (e.g. [4 ] ), the event E o v \mathcal{E}_{\mathrm{ov}} E ov in (A4) will happen with probability at least p 0 > 0 p_0 > 0 p 0 > 0 independent of d d d for the uniform initialization.
Fix K > 0 K > 0 K > 0 . We work on the event
E 0 ( K ) : = E o v ∩ E d e l o c ( K ) , \mathcal{E}_0(K) := \mathcal{E}_{\mathrm{ov}} \cap\mathcal{E}_{\mathrm{deloc}}(K), E 0 ( K ) := E ov ∩ E deloc ( K ) ,
where
E d e l o c ( K ) : = { ∥ θ 0 ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ C K log d d , ∥ θ 0 ∥ 3 3 + ∥ θ ∗ ∥ 3 3 ≤ C K d − 1 / 2 } . \mathcal{E}_{\mathrm{deloc}}(K) := \left\{\lVert\theta_0\rVert_\infty+ \lVert\theta^*\rVert_\infty\le C_K\sqrt{\frac{\log d}{d}},\quad\lVert\theta_0\rVert_3^3 + \lVert\theta^*\rVert_3^3 \le C_Kd^{-1/2}\right\}. E deloc ( K ) := { ∥ θ 0 ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ C K d log d , ∥ θ 0 ∥ 3 3 + ∥ θ ∗ ∥ 3 3 ≤ C K d − 1/2 } .
By Lemma C.1, given θ 0 , θ ∗ ∼ Unif ( S d − 1 ) \theta_0,\theta^* \sim\operatorname{Unif}(\mathbb{S}^{d-1}) θ 0 , θ ∗ ∼ Unif ( S d − 1 ) , E d e l o c ( K ) \mathcal{E}_{\mathrm{deloc}}(K) E deloc ( K ) holds with probability at least 1 − d − K 1-d^{-K} 1 − d − K .
Proof of Continuous-Time Population Dynamics
Initialization Event
Lemma C.1 (Delocalization of random spherical vectors). Let u ∼ Unif ( S d − 1 ) u \sim\operatorname{Unif}(\mathbb{S}^{d-1}) u ∼ Unif ( S d − 1 ) . Then, for every fixed K > 0 K > 0 K > 0 , there exists a constant C K < ∞ C_K < \infty C K < ∞ , independent of d d d , such that
P ( ∥ u ∥ ∞ ≤ C K log d d and ∥ u ∥ 3 3 ≤ C K d − 1 / 2 ) ≥ 1 − d − K . (22) \mathbb{P}\left(\lVert u\rVert_\infty\le C_K\sqrt{\frac{\log d}{d}}\ \text{and}\ \lVert u\rVert_3^3 \le C_Kd^{-1/2}\right) \ge1-d^{-K}. \tag*{(22)} P ( ∥ u ∥ ∞ ≤ C K d log d and ∥ u ∥ 3 3 ≤ C K d − 1/2 ) ≥ 1 − d − K . (22)
Consequently, if θ 0 , θ ∗ \theta_0,\theta^* θ 0 , θ ∗ are independent uniform random vectors on S d − 1 \mathbb{S}^{d-1} S d − 1 , then, after increasing C K C_K C K , with probability at least 1 − d − K 1-d^{-K} 1 − d − K ,
∥ θ 0 ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ C K log d d , (23) \lVert\theta_0\rVert_\infty+ \lVert\theta^*\rVert_\infty\le C_K\sqrt{\frac{\log d}{d}}, \tag*{(23)} ∥ θ 0 ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ C K d log d , (23)
and
∥ θ 0 ∥ 3 3 + ∥ θ ∗ ∥ 3 3 ≤ C K d − 1 / 2 . (24) \lVert\theta_0\rVert_3^3 + \lVert\theta^*\rVert_3^3 \le C_Kd^{-1/2}. \tag*{(24)} ∥ θ 0 ∥ 3 3 + ∥ θ ∗ ∥ 3 3 ≤ C K d − 1/2 . (24)
Proof. Write
u = g ∥ g ∥ 2 , g = ( g 1 , … , g d ) , g i ∼ i i d N ( 0 , 1 ) . u=\frac{g}{\lVert g\rVert_2}, \qquad g=(g_1,\ldots,g_d), \qquad g_i \mathrel{\overset{\mathrm{iid}}{\sim}} N(0,1). u = ∥ g ∥ 2 g , g = ( g 1 , … , g d ) , g i ∼ iid N ( 0 , 1 ) .
Standard Gaussian concentration gives, for every fixed K > 0 K>0 K > 0 , after choosing C K C_K C K large enough,
max 1 ≤ i ≤ d ∣ g i ∣ ≤ C K log d , ∥ g ∥ 2 2 ≥ d 2 \max_{1\le i\le d}|g_i|\le C_K\sqrt{\log d}, \qquad\lVert g\rVert_2^2\ge\frac{d}{2} 1 ≤ i ≤ d max ∣ g i ∣ ≤ C K log d , ∥ g ∥ 2 2 ≥ 2 d
with probability at least 1 − d − K 1-d^{-K} 1 − d − K . Hence
∥ u ∥ ∞ = max i ∣ g i ∣ ∥ g ∥ 2 ≤ C K log d d . \lVert u\rVert_\infty=\frac{\max_i|g_i|}{\lVert g\rVert_2}\le C_K\sqrt{\frac{\log d}{d}}. ∥ u ∥ ∞ = ∥ g ∥ 2 max i ∣ g i ∣ ≤ C K d log d .
For the third moment,
∥ u ∥ 3 3 = ∑ i = 1 d ∣ u i ∣ 3 = ∑ i = 1 d ∣ g i ∣ 3 ∥ g ∥ 2 3 . \lVert u\rVert_3^3=\sum_{i=1}^d|u_i|^3=\frac{\sum_{i=1}^d|g_i|^3}{\lVert g\rVert_2^3}. ∥ u ∥ 3 3 = i = 1 ∑ d ∣ u i ∣ 3 = ∥ g ∥ 2 3 ∑ i = 1 d ∣ g i ∣ 3 .
By a standard concentration bound for sums of iid variables with finite moments, for every fixed K > 0 K>0 K > 0 there exists C K < ∞ C_K<\infty C K < ∞ such that
∑ i = 1 d ∣ g i ∣ 3 ≤ C K d \sum_{i=1}^d|g_i|^3\le C_Kd i = 1 ∑ d ∣ g i ∣ 3 ≤ C K d
with probability at least 1 − d − K 1-d^{-K} 1 − d − K . Combining this with ∥ g ∥ 2 2 ≥ d / 2 \lVert g\rVert_2^2\ge d/2 ∥ g ∥ 2 2 ≥ d /2 , we get
∥ u ∥ 3 3 ≤ C K d ( d / 2 ) 3 / 2 ≤ C K d − 1 / 2 . \lVert u\rVert_3^3\le\frac{C_Kd}{(d/2)^{3/2}}\le C_Kd^{-1/2}. ∥ u ∥ 3 3 ≤ ( d /2 ) 3/2 C K d ≤ C K d − 1/2 .
This proves (21). By a union bound applied to θ 0 \theta_0 θ 0 and θ ∗ \theta^* θ ∗ , we get (22) and (23), respectively.
General Results on Dynamics
Lemma C.2 (Uniform first-order gradient comparison). *Assume (A1) and (A3), then there exist constants C β ( 1 ) , C θ ( 1 ) > 0 C_\beta^{(1)},C_\theta^{(1)}>0 C β ( 1 ) , C θ ( 1 ) > 0 , independent of d d d , such that uniformly for all ( θ , β ) (\theta,\beta) ( θ , β ) with ρ , χ ≥ 0 \rho,\chi\ge0 ρ , χ ≥ 0 ,
∣ ⟨ ∇ S , β ( L − L G ) ( θ , β ) , β ∗ ⟩ ∣ ≤ C β ( 1 ) η . (25) \left|\left\langle\nabla_{S,\beta}(L-L_G)(\theta,\beta),\beta^*\right\rangle\right|\le C_\beta^{(1)}\eta. \tag*{(25)} ∣ ⟨ ∇ S , β ( L − L G ) ( θ , β ) , β ∗ ⟩ ∣ ≤ C β ( 1 ) η . (25)
∣ ⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ ∣ ≤ C θ ( 1 ) η . (26) \left|\left\langle\nabla_{S,\theta}(L-L_G)(\theta,\beta),\theta^*\right\rangle\right|\le C_\theta^{(1)}\eta. \tag*{(26)} ∣ ⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ ∣ ≤ C θ ( 1 ) η . (26)
Proof. Throughout the proof, constants may change from line to line but depend only on the model and moment constants, not on d , θ , θ ∗ , β , β ∗ d,\theta,\theta^*,\beta,\beta^* d , θ , θ ∗ , β , β ∗ .
Set
U i : = x ( i ) ⋅ β , U i ∗ : = x ( i ) ⋅ β ∗ , U_i:=x^{(i)}\cdot\beta,\qquad U_i^*:=x^{(i)}\cdot\beta^*, U i := x ( i ) ⋅ β , U i ∗ := x ( i ) ⋅ β ∗ ,
so that ( U i , U i ∗ ) i = 1 d (U_i,U_i^*)_{i=1}^d ( U i , U i ∗ ) i = 1 d are iid centered Gaussian pairs with correlation ρ \rho ρ . Define
X i : = ϕ ( U i ) , X i ∗ : = ϕ ( U i ∗ ) , X_i:=\phi(U_i),\qquad X_i^*:=\phi(U_i^*), X i := ϕ ( U i ) , X i ∗ := ϕ ( U i ∗ ) ,
and
S : = ∑ i = 1 d θ i X i , S ∗ : = ∑ i = 1 d θ i ∗ X i ∗ . S:=\sum_{i=1}^d\theta_iX_i,\qquad S^*:=\sum_{i=1}^d\theta_i^*X_i^*. S := i = 1 ∑ d θ i X i , S ∗ := i = 1 ∑ d θ i ∗ X i ∗ .
Then
E [ X i ] = E [ X i ∗ ] = 0 , E [ X i 2 ] = E [ ( X i ∗ ) 2 ] = 1 , E [ X i X i ∗ ] = κ ( ρ ) . \mathbb{E}[X_i]=\mathbb{E}[X_i^*]=0,\qquad\mathbb{E}[X_i^2]=\mathbb{E}[(X_i^*)^2]=1,\qquad\mathbb{E}[X_iX_i^*]=\kappa(\rho). E [ X i ] = E [ X i ∗ ] = 0 , E [ X i 2 ] = E [( X i ∗ ) 2 ] = 1 , E [ X i X i ∗ ] = κ ( ρ ) .
Let ( Γ i , Γ i ∗ ) i = 1 d (\Gamma_i,\Gamma_i^*)_{i=1}^d ( Γ i , Γ i ∗ ) i = 1 d be iid Gaussian pairs with the same covariance matrix
E [ Γ i 2 ] = E [ ( Γ i ∗ ) 2 ] = 1 , E [ Γ i Γ i ∗ ] = κ ( ρ ) , \mathbb{E}[\Gamma_i^2]=\mathbb{E}[(\Gamma_i^*)^2]=1,\qquad\mathbb{E}[\Gamma_i\Gamma_i^*]=\kappa(\rho), E [ Γ i 2 ] = E [( Γ i ∗ ) 2 ] = 1 , E [ Γ i Γ i ∗ ] = κ ( ρ ) ,
and write
S G : = ∑ i = 1 d θ i Γ i , S G ∗ : = ∑ i = 1 d θ i ∗ Γ i ∗ . S_G:=\sum_{i=1}^d\theta_i\Gamma_i,\qquad S_G^*:=\sum_{i=1}^d\theta_i^*\Gamma_i^*. S G := i = 1 ∑ d θ i Γ i , S G ∗ := i = 1 ∑ d θ i ∗ Γ i ∗ .
Step 1: the θ \theta θ -gradient comparison. Let
w : = P θ ⊥ θ ∗ = θ ∗ − χ θ . w := P_{\theta^\perp}\theta^* = \theta^*-\chi\theta. w := P θ ⊥ θ ∗ = θ ∗ − χ θ .
Then
⟨ ∇ S , θ L ( θ , β ) , θ ∗ ⟩ = 2 E [ ( ψ ( S ) − ψ ( S ∗ ) ) ψ ′ ( S ) ∑ i = 1 d w i X i ] . \left\langle\nabla_{S,\theta}L(\theta,\beta),\theta^*\right\rangle
=2\mathbb{E}\left[(\psi(S)-\psi(S^*))\psi'(S)\sum_{i=1}^{d}w_iX_i\right]. ⟨ ∇ S , θ L ( θ , β ) , θ ∗ ⟩ = 2 E [ ( ψ ( S ) − ψ ( S ∗ )) ψ ′ ( S ) i = 1 ∑ d w i X i ] .
The same formula holds for L G L_G L G , with ( X , X ∗ ) (X,X^*) ( X , X ∗ ) replaced by ( Γ , Γ ∗ ) (\Gamma,\Gamma^*) ( Γ , Γ ∗ ) . Hence it is enough to compare the expectations of
h ( x , x ∗ ) : = 2 ( ψ ( x ⋅ θ ) − ψ ( x ∗ ⋅ θ ∗ ) ) ψ ′ ( x ⋅ θ ) ( x ⋅ w ) . h(x,x^*) := 2(\psi(x\cdot\theta)-\psi(x^*\cdot\theta^*))\psi'(x\cdot\theta)(x\cdot w). h ( x , x ∗ ) := 2 ( ψ ( x ⋅ θ ) − ψ ( x ∗ ⋅ θ ∗ )) ψ ′ ( x ⋅ θ ) ( x ⋅ w ) .
We apply Corollary E.2. Fix a block j j j , condition on all other blocks, and write
s : = x j , t : = x j ∗ , s:=x_j,\qquad t:=x_j^*, s := x j , t := x j ∗ ,
ℓ 1 = θ j s + S 1 , ℓ 2 = w j s + S 2 , ℓ 3 = θ j ∗ t + S 3 , \ell_1=\theta_js+S_1,\qquad\ell_2=w_js+S_2,\qquad\ell_3=\theta_j^*t+S_3, ℓ 1 = θ j s + S 1 , ℓ 2 = w j s + S 2 , ℓ 3 = θ j ∗ t + S 3 ,
where S 1 , S 2 , S 3 S_1,S_2,S_3 S 1 , S 2 , S 3 are the leave-out sums. Then
h ( s , t ) = 2 ( ψ ( ℓ 1 ) − ψ ( ℓ 3 ) ) ψ ′ ( ℓ 1 ) ℓ 2 . h(s,t)=2(\psi(\ell_1)-\psi(\ell_3))\psi'(\ell_1)\ell_2. h ( s , t ) = 2 ( ψ ( ℓ 1 ) − ψ ( ℓ 3 )) ψ ′ ( ℓ 1 ) ℓ 2 .
Since ψ ∈ C b 4 \psi\in C_b^4 ψ ∈ C b 4 , every third partial derivative ∂ s a ∂ t b h \partial_s^a\partial_t^b h ∂ s a ∂ t b h , a + b = 3 a+b=3 a + b = 3 , is a finite sum of terms of the two following types:
C ∣ θ j ∣ a ∣ θ j ∗ ∣ b ∣ ℓ 2 ∣ , a + b = 3 , C|\theta_j|^a|\theta_j^*|^b|\ell_2|,\qquad a+b=3, C ∣ θ j ∣ a ∣ θ j ∗ ∣ b ∣ ℓ 2 ∣ , a + b = 3 ,
and
C ∣ w j ∣ ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ , a ′ + b ′ = 2. C|w_j||\theta_j|^{a'}|\theta_j^*|^{b'},\qquad a'+b'=2. C ∣ w j ∣∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ , a ′ + b ′ = 2.
At every hybrid vector in the Lindeberg interpolation,
E ∣ ℓ 2 ∣ ≤ C ∥ w ∥ 2 ≤ C . \mathbb{E}|\ell_2|\le C\|w\|_2\le C. E ∣ ℓ 2 ∣ ≤ C ∥ w ∥ 2 ≤ C .
Therefore
E [ ∣ ∂ s a ∂ t b h ( W ) ∣ ] ≤ C ( ∣ θ j ∣ a ∣ θ j ∗ ∣ b + ∣ w j ∣ ∑ a ′ + b ′ = 2 ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ ) . \mathbb{E}\left[|\partial_s^a\partial_t^b h(W)|\right]
\le C\left(|\theta_j|^a|\theta_j^*|^b+|w_j|\sum_{a'+b'=2}|\theta_j|^{a'}|\theta_j^*|^{b'}\right). E [ ∣ ∂ s a ∂ t b h ( W ) ∣ ] ≤ C ( ∣ θ j ∣ a ∣ θ j ∗ ∣ b + ∣ w j ∣ a ′ + b ′ = 2 ∑ ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ ) .
Summing over j j j and over a + b = 3 a+b=3 a + b = 3 , we get
∣ E h ( X , X ∗ ) − E h ( Γ , Γ ∗ ) ∣ ≤ C [ ∑ j ∑ a + b = 3 ∣ θ j ∣ a ∣ θ j ∗ ∣ b + ∑ j ∣ w j ∣ ∑ a ′ + b ′ = 2 ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ ] . (27) |\mathbb{E}h(X,X^*)-\mathbb{E}h(\Gamma,\Gamma^*)|
\le C\left[\sum_j\sum_{a+b=3}|\theta_j|^a|\theta_j^*|^b
+\sum_j|w_j|\sum_{a'+b'=2}|\theta_j|^{a'}|\theta_j^*|^{b'}\right]. \tag*{(27)} ∣ E h ( X , X ∗ ) − E h ( Γ , Γ ∗ ) ∣ ≤ C [ j ∑ a + b = 3 ∑ ∣ θ j ∣ a ∣ θ j ∗ ∣ b + j ∑ ∣ w j ∣ a ′ + b ′ = 2 ∑ ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ ] . (27)
The first sum is bounded by
C ( ∥ θ ∥ ∞ + ∥ θ ∗ ∥ ∞ ) . C(\|\theta\|_\infty+\|\theta^*\|_\infty). C ( ∥ θ ∥ ∞ + ∥ θ ∗ ∥ ∞ ) .
Indeed,
∑ j ∣ θ j ∣ 3 ≤ ∥ θ ∥ ∞ , ∑ j ∣ θ j ∣ 2 ∣ θ j ∗ ∣ ≤ ∥ θ ∗ ∥ ∞ , \sum_j|\theta_j|^3\le\|\theta\|_\infty,\qquad
\sum_j|\theta_j|^2|\theta_j^*|\le\|\theta^*\|_\infty, j ∑ ∣ θ j ∣ 3 ≤ ∥ θ ∥ ∞ , j ∑ ∣ θ j ∣ 2 ∣ θ j ∗ ∣ ≤ ∥ θ ∗ ∥ ∞ ,
and similarly for the remaining degree-three monomials. For the second sum, use
∥ w ∥ ∞ = ∥ θ ∗ − χ θ ∥ ∞ ≤ ∥ θ ∗ ∥ ∞ + ∥ θ ∥ ∞ = η , \|w\|_\infty=\|\theta^*-\chi\theta\|_\infty\le\|\theta^*\|_\infty+\|\theta\|_\infty=\eta, ∥ w ∥ ∞ = ∥ θ ∗ − χ θ ∥ ∞ ≤ ∥ θ ∗ ∥ ∞ + ∥ θ ∥ ∞ = η ,
and ∥ w ∥ 2 , ∥ θ ∥ 2 , ∥ θ ∗ ∥ 2 ≤ C \|w\|_2,\|\theta\|_2,\|\theta^*\|_2\le C ∥ w ∥ 2 , ∥ θ ∥ 2 , ∥ θ ∗ ∥ 2 ≤ C , to obtain
∑ j ∣ w j ∣ ∑ a ′ + b ′ = 2 ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ ≤ C η . \sum_j|w_j|\sum_{a'+b'=2}|\theta_j|^{a'}|\theta_j^*|^{b'}\le C\eta. j ∑ ∣ w j ∣ a ′ + b ′ = 2 ∑ ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ ≤ C η .
Thus
∣ ⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ ∣ ≤ C η . \left|\left\langle\nabla_{S,\theta}(L-L_G)(\theta,\beta),\theta^*\right\rangle\right|\le C\eta. ∣ ⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ ∣ ≤ C η .
This proves the θ \theta θ -gradient comparison.
Step 2: exact formula for the β \beta β -gradient. Let
b : = P β ⊥ β ∗ = β ∗ − ρ β . b:=P_{\beta^\perp}\beta^*=\beta^*-\rho\beta. b := P β ⊥ β ∗ = β ∗ − ρβ .
Then
⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = ⟨ ∇ β L ( θ , β ) , b ⟩ . \langle\nabla_{S,\beta}L(\theta,\beta),\beta^* \rangle= \langle\nabla_\beta L(\theta,\beta),b\rangle. ⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = ⟨ ∇ β L ( θ , β ) , b ⟩ .
Differentiating S S S in the direction b b b gives
D b S = ∑ i = 1 d θ i ϕ ′ ( U i ) H i , H i : = x ( i ) ⋅ b . D_bS=\sum_{i=1}^{d}\theta_i\phi'(U_i)H_i,\qquad H_i:=x^{(i)}\cdot b. D b S = i = 1 ∑ d θ i ϕ ′ ( U i ) H i , H i := x ( i ) ⋅ b .
Hence
⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = 2 E [ ( ψ ( S ) − ψ ( S ∗ ) ) ψ ′ ( S ) ∑ i = 1 d θ i ϕ ′ ( U i ) H i ] . \langle\nabla_{S,\beta}L(\theta,\beta),\beta^* \rangle= 2\mathbb{E}\left[\left(\psi(S)-\psi(S^*)\right)\psi'(S)\sum_{i=1}^{d}\theta_i\phi'(U_i)H_i\right]. ⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = 2 E [ ( ψ ( S ) − ψ ( S ∗ ) ) ψ ′ ( S ) i = 1 ∑ d θ i ϕ ′ ( U i ) H i ] .
For each i i i , H i = U i ∗ − ρ U i H_i=U_i^*-\rho U_i H i = U i ∗ − ρ U i , so H i H_i H i is centered Gaussian with variance 1 − ρ 2 1-\rho^2 1 − ρ 2 , independent of U i U_i U i . Moreover U i ∗ = ρ U i + H i U_i^*=\rho U_i+H_i U i ∗ = ρ U i + H i . Applying Gaussian integration by parts in the H i H_i H i -variable gives
E [ ( ψ ( S ) − ψ ( S ∗ ) ) ψ ′ ( S ) θ i ϕ ′ ( U i ) H i ] = − ( 1 − ρ 2 ) θ i θ i ∗ E [ ψ ′ ( S ) ψ ′ ( S ∗ ) ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) ] . \mathbb{E}\left[\left(\psi(S)-\psi(S^*)\right)\psi'(S)\theta_i\phi'(U_i)H_i\right]=-(1-\rho^2)\theta_i\theta_i^*\mathbb{E}\left[\psi'(S)\psi'(S^*)\phi'(U_i)\phi'(U_i^*)\right]. E [ ( ψ ( S ) − ψ ( S ∗ ) ) ψ ′ ( S ) θ i ϕ ′ ( U i ) H i ] = − ( 1 − ρ 2 ) θ i θ i ∗ E [ ψ ′ ( S ) ψ ′ ( S ∗ ) ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) ] .
Therefore
− ⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = 2 ( 1 − ρ 2 ) ∑ i = 1 d θ i θ i ∗ E [ ψ ′ ( S ) ψ ′ ( S ∗ ) ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) ] . (28) -\langle\nabla_{S,\beta}L(\theta,\beta),\beta^* \rangle=2(1-\rho^2)\sum_{i=1}^{d}\theta_i\theta_i^*\mathbb{E}\left[\psi'(S)\psi'(S^*)\phi'(U_i)\phi'(U_i^*)\right]. \tag*{(28)} − ⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = 2 ( 1 − ρ 2 ) i = 1 ∑ d θ i θ i ∗ E [ ψ ′ ( S ) ψ ′ ( S ∗ ) ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) ] . (28)
Step 3: leave-one-out reduction for the true β \beta β -gradient. Set
F ( r , s ) : = ψ ′ ( r ) ψ ′ ( s ) , q i : = ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) , c i : = θ i θ i ∗ . F(r,s):=\psi'(r)\psi'(s),\qquad q_i:=\phi'(U_i)\phi'(U_i^*),\qquad c_i:=\theta_i\theta_i^*. F ( r , s ) := ψ ′ ( r ) ψ ′ ( s ) , q i := ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) , c i := θ i θ i ∗ .
Also define
S − i : = S − θ i X i , S − i ∗ : = S ∗ − θ i ∗ X i ∗ . S_{-i}:=S-\theta_iX_i,\qquad S_{-i}^*:=S^*-\theta_i^*X_i^*. S − i := S − θ i X i , S − i ∗ := S ∗ − θ i ∗ X i ∗ .
Since S − i , S − i ∗ S_{-i},S_{-i}^* S − i , S − i ∗ are independent of ( U i , U i ∗ ) (U_i,U_i^*) ( U i , U i ∗ ) ,
E [ F ( S − i , S − i ∗ ) q i ] = E [ F ( S − i , S − i ∗ ) ] E [ q i ] . \mathbb{E}[F(S_{-i},S_{-i}^*)q_i]=\mathbb{E}[F(S_{-i},S_{-i}^*)]\mathbb{E}[q_i]. E [ F ( S − i , S − i ∗ ) q i ] = E [ F ( S − i , S − i ∗ )] E [ q i ] .
By the Hermite identity,
E [ q i ] = E [ ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) ] = κ ′ ( ρ ) . \mathbb{E}[q_i]=\mathbb{E}[\phi'(U_i)\phi'(U_i^*)]=\kappa'(\rho). E [ q i ] = E [ ϕ ′ ( U i ) ϕ ′ ( U i ∗ )] = κ ′ ( ρ ) .
Moreover, since F F F is Lipschitz and the stated moment assumptions imply
sup ρ ∈ [ 0 , 1 ] E [ ( ∣ X i ∣ + ∣ X i ∗ ∣ ) ∣ q i ∣ ] < ∞ , \sup_{\rho\in[0,1]}\mathbb{E}\left[(|X_i|+|X_i^*|)|q_i|\right]<\infty, ρ ∈ [ 0 , 1 ] sup E [ ( ∣ X i ∣ + ∣ X i ∗ ∣ ) ∣ q i ∣ ] < ∞ ,
we have
∣ E [ F ( S , S ∗ ) q i ] − E [ F ( S − i , S − i ∗ ) q i ] ∣ ≤ C ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) . \left|\mathbb{E}[F(S,S^*)q_i]-\mathbb{E}[F(S_{-i},S_{-i}^*)q_i]\right|\le C(|\theta_i|+|\theta_i^*|). E [ F ( S , S ∗ ) q i ] − E [ F ( S − i , S − i ∗ ) q i ] ≤ C ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) .
Multiplying by ∣ c i ∣ |c_i| ∣ c i ∣ and summing,
∑ i = 1 d ∣ c i ∣ ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) ≤ ∑ i ∣ θ i ∣ 2 ∣ θ i ∗ ∣ + ∑ i ∣ θ i ∣ ∣ θ i ∗ ∣ 2 ≤ ∥ θ ∗ ∥ ∞ + ∥ θ ∥ ∞ = η . \sum_{i=1}^{d}|c_i|(|\theta_i|+|\theta_i^*|)\le\sum_i|\theta_i|^2|\theta_i^*|+\sum_i|\theta_i||\theta_i^*|^2\le\|\theta^*\|_\infty+\|\theta\|_\infty=\eta. i = 1 ∑ d ∣ c i ∣ ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) ≤ i ∑ ∣ θ i ∣ 2 ∣ θ i ∗ ∣ + i ∑ ∣ θ i ∣∣ θ i ∗ ∣ 2 ≤ ∥ θ ∗ ∥ ∞ + ∥ θ ∥ ∞ = η .
Thus
∑ i = 1 d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) ∑ i = 1 d c i E [ F ( S − i , S − i ∗ ) ] + O ( η ) . \sum_{i=1}^{d}c_i\mathbb{E}[F(S,S^*)q_i]=\kappa'(\rho)\sum_{i=1}^{d}c_i\mathbb{E}[F(S_{-i},S_{-i}^*)]+O(\eta). i = 1 ∑ d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) i = 1 ∑ d c i E [ F ( S − i , S − i ∗ )] + O ( η ) .
Similarly,
∣ E [ F ( S , S ∗ ) ] − E [ F ( S − i , S − i ∗ ) ] ∣ ≤ C ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) , \left|\mathbb{E}[F(S,S^*)]-\mathbb{E}[F(S_{-i},S_{-i}^*)]\right|\le C(|\theta_i|+|\theta_i^*|), E [ F ( S , S ∗ )] − E [ F ( S − i , S − i ∗ )] ≤ C ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) ,
and the same summation estimate gives
∑ i = 1 d c i E [ F ( S − i , S − i ∗ ) ] = ( ∑ i = 1 d c i ) E [ F ( S , S ∗ ) ] + O ( η ) . \sum_{i=1}^{d}c_i\mathbb{E}[F(S_{-i},S_{-i}^*)]=\left(\sum_{i=1}^{d}c_i\right)\mathbb{E}[F(S,S^*)]+O(\eta). i = 1 ∑ d c i E [ F ( S − i , S − i ∗ )] = ( i = 1 ∑ d c i ) E [ F ( S , S ∗ )] + O ( η ) .
Since ∑ i c i = θ ⋅ θ ∗ = χ \sum_i c_i=\theta\cdot\theta^*=\chi ∑ i c i = θ ⋅ θ ∗ = χ , we conclude
∑ i = 1 d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) χ E [ F ( S , S ∗ ) ] + O ( η ) . (29) \sum_{i=1}^{d}c_i\mathbb{E}[F(S,S^*)q_i]=\kappa'(\rho)\chi\mathbb{E}[F(S,S^*)]+O(\eta). \tag*{(29)} i = 1 ∑ d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) χ E [ F ( S , S ∗ )] + O ( η ) . (29)
Step 4: Stein comparison for the two-dimensional statistic. Apply Corollary E.2 to
h ( x , x ∗ ) : = F ( x ⋅ θ , x ∗ ⋅ θ ∗ ) . h(x,x^*) := F(x\cdot\theta,x^*\cdot\theta^*). h ( x , x ∗ ) := F ( x ⋅ θ , x ∗ ⋅ θ ∗ ) .
For a fixed block j j j , the third partials have the form
∂ s a ∂ t b h = θ j a ( θ j ∗ ) b ∂ 1 a ∂ 2 b F ( ℓ 1 , ℓ 2 ) , a + b = 3. \partial_s^a\partial_t^b h=\theta_j^a(\theta_j^*)^b\partial_1^a\partial_2^b F(\ell_1,\ell_2),\qquad a+b=3. ∂ s a ∂ t b h = θ j a ( θ j ∗ ) b ∂ 1 a ∂ 2 b F ( ℓ 1 , ℓ 2 ) , a + b = 3.
Since F ∈ C b 3 ( R 2 ) F\in C_b^3(\mathbb{R}^2) F ∈ C b 3 ( R 2 ) ,
E [ ∣ ∂ s a ∂ t b h ( W ) ∣ ] ≤ C ∣ θ j ∣ a ∣ θ j ∗ ∣ b . \mathbb{E}\left[|\partial_s^a\partial_t^b h(W)|\right]\le C|\theta_j|^a|\theta_j^*|^b. E [ ∣ ∂ s a ∂ t b h ( W ) ∣ ] ≤ C ∣ θ j ∣ a ∣ θ j ∗ ∣ b .
Therefore
∣ E [ F ( S , S ∗ ) ] − E [ F ( S G , S G ∗ ) ] ∣ ≤ C ∑ j ∑ a + b = 3 ∣ θ j ∣ a ∣ θ j ∗ ∣ b ≤ C η . \left|\mathbb{E}[F(S,S^*)]-\mathbb{E}[F(S_G,S_G^*)]\right|
\le C\sum_j\sum_{a+b=3}|\theta_j|^a|\theta_j^*|^b\le C\eta. ∣ E [ F ( S , S ∗ )] − E [ F ( S G , S G ∗ )] ∣ ≤ C j ∑ a + b = 3 ∑ ∣ θ j ∣ a ∣ θ j ∗ ∣ b ≤ C η .
Combining this with (29) , and using that κ ′ ( ρ ) \kappa'(\rho) κ ′ ( ρ ) and χ \chi χ are bounded on the entry corridor, gives
∑ i = 1 d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) χ E [ F ( S G , S G ∗ ) ] + O ( η ) . (30) \sum_{i=1}^{d}c_i\mathbb{E}[F(S,S^*)q_i]
=\kappa'(\rho)\chi\mathbb{E}[F(S_G,S_G^*)]+O(\eta). \tag*{(30)} i = 1 ∑ d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) χ E [ F ( S G , S G ∗ )] + O ( η ) . (30)
Step 5: identify the Gaussian β \beta β -gradient. In the Gaussian equivalent model,
L G ( θ , β ) = Φ ( χ , ρ ) , Φ ( χ , ρ ) = 2 ∑ k ≥ 1 α k 2 [ 1 − ( κ ( ρ ) χ ) k ] . L_G(\theta,\beta)=\Phi(\chi,\rho),\qquad
\Phi(\chi,\rho)=2\sum_{k\ge1}\alpha_k^2\left[1-(\kappa(\rho)\chi)^k\right]. L G ( θ , β ) = Φ ( χ , ρ ) , Φ ( χ , ρ ) = 2 k ≥ 1 ∑ α k 2 [ 1 − ( κ ( ρ ) χ ) k ] .
Equivalently, by Price’s identity applied to the Gaussian pair ( S G , S G ∗ ) (S_G,S_G^*) ( S G , S G ∗ ) ,
− Φ ρ ( χ , ρ ) = 2 κ ′ ( ρ ) χ E [ ψ ′ ( S G ) ψ ′ ( S G ∗ ) ] . -\Phi_\rho(\chi,\rho)=2\kappa'(\rho)\chi\mathbb{E}[\psi'(S_G)\psi'(S_G^*)]. − Φ ρ ( χ , ρ ) = 2 κ ′ ( ρ ) χ E [ ψ ′ ( S G ) ψ ′ ( S G ∗ )] .
Since
∇ S , β L G = Φ ρ ( χ , ρ ) P β ⊥ β ∗ , \nabla_{S,\beta}L_G=\Phi_\rho(\chi,\rho)P_{\beta^\perp}\beta^*, ∇ S , β L G = Φ ρ ( χ , ρ ) P β ⊥ β ∗ ,
we have
− ⟨ ∇ S , β L G ( θ , β ) , β ∗ ⟩ = 2 ( 1 − ρ 2 ) κ ′ ( ρ ) χ E [ ψ ′ ( S G ) ψ ′ ( S G ∗ ) ] . (31) -\langle\nabla_{S,\beta}L_G(\theta,\beta),\beta^*\rangle
=2(1-\rho^2)\kappa'(\rho)\chi\mathbb{E}[\psi'(S_G)\psi'(S_G^*)]. \tag*{(31)} − ⟨ ∇ S , β L G ( θ , β ) , β ∗ ⟩ = 2 ( 1 − ρ 2 ) κ ′ ( ρ ) χ E [ ψ ′ ( S G ) ψ ′ ( S G ∗ )] . (31)
Comparing (28) , (30) , and (31) , we obtain
∣ ⟨ ∇ S , β ( L − L G ) ( θ , β ) , β ∗ ⟩ ∣ ≤ C ( 1 − ρ 2 ) η ≤ C η . \left|\langle\nabla_{S,\beta}(L-L_G)(\theta,\beta),\beta^*\rangle\right|
\le C(1-\rho^2)\eta\le C\eta. ∣ ⟨ ∇ S , β ( L − L G ) ( θ , β ) , β ∗ ⟩ ∣ ≤ C ( 1 − ρ 2 ) η ≤ C η .
This proves the β \beta β -gradient comparison.
Combining the two block estimates gives the lemma.
Lemma C.3 (Uniform positive-overlap drift). Assume Lemma C.2. On the positive overlap region ρ , χ ≥ 0 \rho,\chi\ge0 ρ , χ ≥ 0 , the population gradient flow satisfies
ρ ˙ t ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t , χ ˙ t ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t , (32) \dot{\rho}_t\ge a_\rho\chi_t(1-\rho_t^2)-b_\rho\eta_t,\qquad
\dot{\chi}_t\ge a_\chi\rho_t(1-\chi_t^2)-b_\chi\eta_t, \tag*{(32)} ρ ˙ t ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t , χ ˙ t ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t , (32) where a ρ , a χ , b ρ , b χ > 0 a_\rho,a_\chi,b_\rho,b_\chi>0 a ρ , a χ , b ρ , b χ > 0 are independent of d d d .
Proof. With the population flow defined in (5), we have
ρ ˙ t = − ⟨ ∇ S , β L ( θ t , β t ) , β ∗ ⟩ , χ ˙ t = − ⟨ ∇ S , θ L ( θ t , β t ) , θ ∗ ⟩ . \dot{\rho}_t=-\langle\nabla_{S,\beta}L(\theta_t,\beta_t),\beta^*\rangle,\qquad
\dot{\chi}_t=-\langle\nabla_{S,\theta}L(\theta_t,\beta_t),\theta^*\rangle. ρ ˙ t = − ⟨ ∇ S , β L ( θ t , β t ) , β ∗ ⟩ , χ ˙ t = − ⟨ ∇ S , θ L ( θ t , β t ) , θ ∗ ⟩ .
We first compute the Gaussian drifts exactly. Recall that
L G ( θ , β ) = Φ ( χ , ρ ) , Φ ( χ , ρ ) = 2 ∑ k ≥ 1 α k 2 [ 1 − ( κ ( ρ ) χ ) k ] , L_G(\theta,\beta)=\Phi(\chi,\rho),\qquad
\Phi(\chi,\rho)=2\sum_{k\ge1}\alpha_k^2\left[1-(\kappa(\rho)\chi)^k\right], L G ( θ , β ) = Φ ( χ , ρ ) , Φ ( χ , ρ ) = 2 k ≥ 1 ∑ α k 2 [ 1 − ( κ ( ρ ) χ ) k ] ,
where
κ ( ρ ) = ∑ ℓ ≥ 1 α ℓ 2 ρ ℓ . \kappa(\rho)=\sum_{\ell\ge1}\alpha_\ell^2\rho^\ell. κ ( ρ ) = ℓ ≥ 1 ∑ α ℓ 2 ρ ℓ .
For the first layer,
∇ S , β L G = Φ ρ ( χ , ρ ) P β ⊥ β ∗ = Φ ρ ( χ , ρ ) ( β ∗ − ρ β ) . \nabla_{S,\beta}L_G=\Phi_\rho(\chi,\rho)P_{\beta^\perp}\beta^*
=\Phi_\rho(\chi,\rho)(\beta^*-\rho\beta). ∇ S , β L G = Φ ρ ( χ , ρ ) P β ⊥ β ∗ = Φ ρ ( χ , ρ ) ( β ∗ − ρβ ) .
Hence
− ⟨ ∇ S , β L G , β ∗ ⟩ = − Φ ρ ( χ , ρ ) ( 1 − ρ 2 ) . -\langle\nabla_{S,\beta}L_G,\beta^*\rangle
=-\Phi_\rho(\chi,\rho)(1-\rho^2). − ⟨ ∇ S , β L G , β ∗ ⟩ = − Φ ρ ( χ , ρ ) ( 1 − ρ 2 ) .
Moreover,
− Φ ρ ( χ , ρ ) = 2 χ κ ′ ( ρ ) ∑ k ≥ 1 k a k 2 ( κ ( ρ ) χ ) k − 1 . -\Phi_{\rho}(\chi,\rho)=2\chi\kappa'(\rho)\sum_{k\ge1}ka_k^2\bigl(\kappa(\rho)\chi\bigr)^{k-1}. − Φ ρ ( χ , ρ ) = 2 χ κ ′ ( ρ ) k ≥ 1 ∑ k a k 2 ( κ ( ρ ) χ ) k − 1 .
On the positive-overlap region ρ , χ ≥ 0 \rho,\chi\ge0 ρ , χ ≥ 0 , all terms are nonnegative. Keeping only the k = 1 k=1 k = 1 term gives
− Φ ρ ( χ , ρ ) ≥ 2 a 1 2 χ κ ′ ( ρ ) . -\Phi_{\rho}(\chi,\rho)\ge2a_1^2\chi\kappa'(\rho). − Φ ρ ( χ , ρ ) ≥ 2 a 1 2 χ κ ′ ( ρ ) .
Since
κ ′ ( ρ ) = ∑ ℓ ≥ 1 ℓ a ℓ 2 ρ ℓ − 1 ≥ a 1 2 ( ρ ≥ 0 ) , \kappa'(\rho)=\sum_{\ell\ge1}\ell a_\ell^2\rho^{\ell-1}\ge a_1^2\qquad(\rho\ge0), κ ′ ( ρ ) = ℓ ≥ 1 ∑ ℓ a ℓ 2 ρ ℓ − 1 ≥ a 1 2 ( ρ ≥ 0 ) ,
we obtain
− ⟨ ∇ S , β L G , β ∗ ⟩ ≥ 2 a 1 2 a 1 2 χ ( 1 − ρ 2 ) . -\langle\nabla_{S,\beta}L_G,\beta^*\rangle\ge2a_1^2a_1^2\chi(1-\rho^2). − ⟨ ∇ S , β L G , β ∗ ⟩ ≥ 2 a 1 2 a 1 2 χ ( 1 − ρ 2 ) .
Similarly,
∇ S , θ L G = Φ χ ( χ , ρ ) P θ ⊥ θ ∗ = Φ χ ( χ , ρ ) ( θ ∗ − χ θ ) , \nabla_{S,\theta}L_G=\Phi_{\chi}(\chi,\rho)P_{\theta^\perp}\theta^*=\Phi_{\chi}(\chi,\rho)(\theta^*-\chi\theta), ∇ S , θ L G = Φ χ ( χ , ρ ) P θ ⊥ θ ∗ = Φ χ ( χ , ρ ) ( θ ∗ − χ θ ) ,
so
− ⟨ ∇ S , θ L G , θ ∗ ⟩ = − Φ χ ( χ , ρ ) ( 1 − χ 2 ) . -\langle\nabla_{S,\theta}L_G,\theta^*\rangle=-\Phi_{\chi}(\chi,\rho)(1-\chi^2). − ⟨ ∇ S , θ L G , θ ∗ ⟩ = − Φ χ ( χ , ρ ) ( 1 − χ 2 ) .
Now
− Φ χ ( χ , ρ ) = 2 κ ( ρ ) ∑ k ≥ 1 k a k 2 ( κ ( ρ ) χ ) k − 1 . -\Phi_{\chi}(\chi,\rho)=2\kappa(\rho)\sum_{k\ge1}ka_k^2\bigl(\kappa(\rho)\chi\bigr)^{k-1}. − Φ χ ( χ , ρ ) = 2 κ ( ρ ) k ≥ 1 ∑ k a k 2 ( κ ( ρ ) χ ) k − 1 .
Again all terms are nonnegative on ρ , χ ≥ 0 \rho,\chi\ge0 ρ , χ ≥ 0 , so
− Φ χ ( χ , ρ ) ≥ 2 a 1 2 κ ( ρ ) . -\Phi_{\chi}(\chi,\rho)\ge2a_1^2\kappa(\rho). − Φ χ ( χ , ρ ) ≥ 2 a 1 2 κ ( ρ ) .
Since
κ ( ρ ) = ∑ ℓ ≥ 1 a ℓ 2 ρ ℓ ≥ a 1 2 ρ ( ρ ≥ 0 ) , \kappa(\rho)=\sum_{\ell\ge1}a_\ell^2\rho^\ell\ge a_1^2\rho\qquad(\rho\ge0), κ ( ρ ) = ℓ ≥ 1 ∑ a ℓ 2 ρ ℓ ≥ a 1 2 ρ ( ρ ≥ 0 ) ,
we get
− ⟨ ∇ S , θ L G , θ ∗ ⟩ ≥ 2 a 1 2 a 1 2 ρ ( 1 − χ 2 ) . -\langle\nabla_{S,\theta}L_G,\theta^*\rangle\ge2a_1^2a_1^2\rho(1-\chi^2). − ⟨ ∇ S , θ L G , θ ∗ ⟩ ≥ 2 a 1 2 a 1 2 ρ ( 1 − χ 2 ) .
Now transfer the Gaussian drift to the true population loss. By (24) from Lemma C.2,
ρ ˙ t = − ⟨ ∇ S , β L G ( θ t , β t ) , β ∗ ⟩ − ⟨ ∇ S , β ( L − L G ) ( θ t , β t ) , β ∗ ⟩ \dot{\rho}_t=-\langle\nabla_{S,\beta}L_G(\theta_t,\beta_t),\beta^*\rangle-\langle\nabla_{S,\beta}(L-L_G)(\theta_t,\beta_t),\beta^*\rangle ρ ˙ t = − ⟨ ∇ S , β L G ( θ t , β t ) , β ∗ ⟩ − ⟨ ∇ S , β ( L − L G ) ( θ t , β t ) , β ∗ ⟩
≥ 2 a 1 2 a 1 2 χ t ( 1 − ρ t 2 ) − C β ( 1 ) η t . \ge2a_1^2a_1^2\chi_t(1-\rho_t^2)-C_\beta^{(1)}\eta_t. ≥ 2 a 1 2 a 1 2 χ t ( 1 − ρ t 2 ) − C β ( 1 ) η t .
Likewise, by (25) from Lemma C.2,
χ ˙ t = − ⟨ ∇ S , θ L G ( θ t , β t ) , θ ∗ ⟩ − ⟨ ∇ S , θ ( L − L G ) ( θ t , β t ) , θ ∗ ⟩ \dot{\chi}_t=-\langle\nabla_{S,\theta}L_G(\theta_t,\beta_t),\theta^*\rangle-\langle\nabla_{S,\theta}(L-L_G)(\theta_t,\beta_t),\theta^*\rangle χ ˙ t = − ⟨ ∇ S , θ L G ( θ t , β t ) , θ ∗ ⟩ − ⟨ ∇ S , θ ( L − L G ) ( θ t , β t ) , θ ∗ ⟩
≥ 2 a 1 2 a 1 2 ρ t ( 1 − χ t 2 ) − C θ ( 1 ) η t . \ge2a_1^2a_1^2\rho_t(1-\chi_t^2)-C_\theta^{(1)}\eta_t. ≥ 2 a 1 2 a 1 2 ρ t ( 1 − χ t 2 ) − C θ ( 1 ) η t .
Thus the claimed inequalities hold with
a ρ = a χ : = 2 a 1 2 a 1 2 , b ρ : = C β ( 1 ) , b χ : = C θ ( 1 ) . a_\rho=a_\chi:=2a_1^2a_1^2,\qquad b_\rho:=C_\beta^{(1)},\qquad b_\chi:=C_\theta^{(1)}. a ρ = a χ := 2 a 1 2 a 1 2 , b ρ := C β ( 1 ) , b χ := C θ ( 1 ) .
Lemma C.4 (ℓ ∞ \ell_\infty ℓ ∞ control). Under our assumptions,
η ˙ t ≤ C 1 ρ t ∥ θ ∗ ∥ ∞ + C 2 χ t ρ t η t + O ( η t 2 ) . (33) \dot{\eta}_t\le C_1\rho_t\lVert\theta^*\rVert_\infty+C_2\chi_t\rho_t\eta_t+O(\eta_t^2). \tag*{(33)} η ˙ t ≤ C 1 ρ t ∥ θ ∗ ∥ ∞ + C 2 χ t ρ t η t + O ( η t 2 ) . (33)
Proof. By definition, we have
L = E ( z , z ∗ ) ∼ φ # ⊗ d γ β ⋅ β ∗ [ ∣ ψ ( z ⋅ θ ) − ψ ( z ∗ ⋅ θ ∗ ) ∣ 2 ] \mathcal{L}=\mathbb{E}_{(z,z^*)\sim\varphi^{\otimes d}_{\#}\gamma_{\beta\cdot\beta^*}}\left[\left|\psi(z\cdot\theta)-\psi(z^*\cdot\theta^*)\right|^2\right] L = E ( z , z ∗ ) ∼ φ # ⊗ d γ β ⋅ β ∗ [ ∣ ψ ( z ⋅ θ ) − ψ ( z ∗ ⋅ θ ∗ ) ∣ 2 ]
so
∇ θ L = E ( z , z ∗ ) ∼ φ # ⊗ d γ β ⋅ β ∗ [ ( ψ ( z ⋅ θ ) − ψ ( z ∗ ⋅ θ ∗ ) ) ψ ′ ( z ⋅ θ ) z ] . \nabla_\theta\mathcal{L}=\mathbb{E}_{(z,z^*)\sim\varphi^{\otimes d}_{\#}\gamma_{\beta\cdot\beta^*}}\left[\left(\psi(z\cdot\theta)-\psi(z^*\cdot\theta^*)\right)\psi'(z\cdot\theta)z\right]. ∇ θ L = E ( z , z ∗ ) ∼ φ # ⊗ d γ β ⋅ β ∗ [ ( ψ ( z ⋅ θ ) − ψ ( z ∗ ⋅ θ ∗ ) ) ψ ′ ( z ⋅ θ ) z ] .
Call ν \nu ν the law of φ ( w ) \varphi(w) φ ( w ) , where w ∼ N ( 0 , 1 ) w\sim N(0,1) w ∼ N ( 0 , 1 ) . The random vector z z z is then drawn from ν ⊗ d \nu^{\otimes d} ν ⊗ d . Let us now analyze the two terms in the RHS. The first term reads E [ F ( z ⋅ θ ) z ] \mathbb{E}[F(z\cdot\theta)z] E [ F ( z ⋅ θ ) z ] for F F F of the form F = ψ ψ ′ F=\psi\psi' F = ψ ψ ′ . Assume that F ′ F' F ′ is Lipschitz. A second-order Taylor expansion of F F F yields
F ( z ⋅ θ ) = F ( z 1 θ 1 + z − 1 ⋅ θ − 1 ) (34) F(z\cdot\theta)=F(z_1\theta_1+z_{-1}\cdot\theta_{-1}) \tag*{(34)} F ( z ⋅ θ ) = F ( z 1 θ 1 + z − 1 ⋅ θ − 1 ) (34)
= F ( z − 1 ⋅ θ − 1 ) + F ′ ( z − 1 ⋅ θ − 1 ) θ 1 z 1 + 1 2 θ 1 2 z 1 2 F ′ ′ ( M ) , (35) =F(z_{-1}\cdot\theta_{-1})+F'(z_{-1}\cdot\theta_{-1})\theta_1z_1+\frac{1}{2}\theta_1^2z_1^2F''(M), \tag*{(35)} = F ( z − 1 ⋅ θ − 1 ) + F ′ ( z − 1 ⋅ θ − 1 ) θ 1 z 1 + 2 1 θ 1 2 z 1 2 F ′′ ( M ) , (35)
and therefore, for all i ∈ [ d ] i\in[d] i ∈ [ d ] ,
E [ F ( z ⋅ θ ) z i ] = θ i E [ F ′ ( z − i ⋅ θ − i ) ] + O ( θ i 2 ) . (36) \mathbb{E}[F(z\cdot\theta)z_i]=\theta_i\mathbb{E}[F'(z_{-i}\cdot\theta_{-i})]+O(\theta_i^2). \tag*{(36)} E [ F ( z ⋅ θ ) z i ] = θ i E [ F ′ ( z − i ⋅ θ − i )] + O ( θ i 2 ) . (36)
Now, for each θ \theta θ and i ∈ [ d ] i\in[d] i ∈ [ d ] , consider μ i = Law ( ∑ j ≠ i θ j z j ) ∈ P ( R ) \mu_i=\operatorname{Law}(\sum_{j\ne i}\theta_jz_j)\in\mathcal{P}(\mathbb{R}) μ i = Law ( ∑ j = i θ j z j ) ∈ P ( R ) , where z j ∼ ν z_j\sim\nu z j ∼ ν iid. We claim that
sup i , i ′ W 1 ( μ i , μ i ′ ) = O ( ∥ θ ∥ ∞ ) . (37) \sup_{i,i'}W_1(\mu_i,\mu_{i'})=O(\lVert\theta\rVert_\infty). \tag*{(37)} i , i ′ sup W 1 ( μ i , μ i ′ ) = O (∥ θ ∥ ∞ ) . (37)
Indeed, fix i , i ′ i,i' i , i ′ and consider the coupling ( Y i , Y i ′ ) ∼ Γ μ i , μ i ′ (Y_i,Y_{i'})\sim\Gamma_{\mu_i,\mu_{i'}} ( Y i , Y i ′ ) ∼ Γ μ i , μ i ′ given by Y i = W + θ i Z Y_i=W+\theta_iZ Y i = W + θ i Z , Y i ′ = W + θ i ′ Z Y_{i'}=W+\theta_{i'}Z Y i ′ = W + θ i ′ Z , where Z ∼ ν Z\sim\nu Z ∼ ν and W = ∑ j ≠ i , i ′ θ j z j W=\sum_{j\ne i,i'}\theta_jz_j W = ∑ j = i , i ′ θ j z j , with z j ∼ ν z_j\sim\nu z j ∼ ν independently of Z Z Z . We then have
W 1 ( μ i , μ i ′ ) ≤ E ∣ Y i − Y i ′ ∣ = ∣ θ i − θ i ′ ∣ E ∣ Z ∣ ≤ 2 E [ ∣ φ ( G ) ∣ 2 ] ∥ θ ∥ ∞ ≤ 2 ∥ θ ∥ ∞ . W_1(\mu_i,\mu_{i'})\le\mathbb{E}|Y_i-Y_{i'}|=|\theta_i-\theta_{i'}|\mathbb{E}|Z|\le2\sqrt{\mathbb{E}[|\varphi(G)|^2]}\lVert\theta\rVert_\infty\le2\lVert\theta\rVert_\infty. W 1 ( μ i , μ i ′ ) ≤ E ∣ Y i − Y i ′ ∣ = ∣ θ i − θ i ′ ∣ E ∣ Z ∣ ≤ 2 E [ ∣ φ ( G ) ∣ 2 ] ∥ θ ∥ ∞ ≤ 2 ∥ θ ∥ ∞ .
Now, since F ′ F' F ′ is Lipschitz, we have
sup i , j ∣ E [ F ′ ( z − i ⋅ θ − i ) ] − E [ F ′ ( z − j ⋅ θ − j ) ] ∣ ≤ Lip ( F ′ ) sup i , i ′ W 1 ( μ i , μ i ′ ) = O ( ∥ θ ∥ ∞ ) , (38) \sup_{i,j}\left|\mathbb{E}[F'(z_{-i}\cdot\theta_{-i})]-\mathbb{E}[F'(z_{-j}\cdot\theta_{-j})]\right|\le\operatorname{Lip}(F')\sup_{i,i'}W_1(\mu_i,\mu_{i'})=O(\lVert\theta\rVert_\infty), \tag*{(38)} i , j sup ∣ E [ F ′ ( z − i ⋅ θ − i )] − E [ F ′ ( z − j ⋅ θ − j )] ∣ ≤ Lip ( F ′ ) i , i ′ sup W 1 ( μ i , μ i ′ ) = O (∥ θ ∥ ∞ ) , (38)
and therefore, denoting by v ( θ ) = 1 d ∑ i E [ F ′ ( z − i ⋅ θ − i ) ] v(\theta)=\frac{1}{d}\sum_i\mathbb{E}[F'(z_{-i}\cdot\theta_{-i})] v ( θ ) = d 1 ∑ i E [ F ′ ( z − i ⋅ θ − i )] , we have
E [ F ( z ⋅ θ ) z ] = θ v ( θ ) + ε , with ∣ ε i ∣ ≤ C ∣ θ i ∣ ∥ θ ∥ ∞ . (39) \mathbb{E}[F(z\cdot\theta)z]=\theta v(\theta)+\varepsilon,\quad\text{with }|\varepsilon_i|\le C|\theta_i|\lVert\theta\rVert_\infty. \tag*{(39)} E [ F ( z ⋅ θ ) z ] = θ v ( θ ) + ε , with ∣ ε i ∣ ≤ C ∣ θ i ∣ ∥ θ ∥ ∞ . (39)
It follows that
P θ ⊥ E [ F ( z ⋅ θ ) z ] = P θ ⊥ ε = ε − ( ε ⋅ θ ) θ (40) P_{\theta^\perp}\mathbb{E}[F(z\cdot\theta)z]=P_{\theta^\perp}\varepsilon=\varepsilon-(\varepsilon\cdot\theta)\theta \tag*{(40)} P θ ⊥ E [ F ( z ⋅ θ ) z ] = P θ ⊥ ε = ε − ( ε ⋅ θ ) θ (40)
satisfies
∥ P θ ⊥ E [ F ( z ⋅ θ ) z ] ∥ ∞ ≤ C ~ ∥ θ ∥ ∞ 2 . (41) \lVert P_{\theta^\perp}\mathbb{E}[F(z\cdot\theta)z]\rVert_\infty\le\widetilde{C}\lVert\theta\rVert_\infty^2. \tag*{(41)} ∥ P θ ⊥ E [ F ( z ⋅ θ ) z ] ∥ ∞ ≤ C ∥ θ ∥ ∞ 2 . (41)
Now we repeat the argument for a term of the form E [ G ( z ⋅ θ , z ~ ⋅ θ ∗ ) z ] \mathbb{E}[G(z\cdot\theta,\widetilde{z}\cdot\theta^*)z] E [ G ( z ⋅ θ , z ⋅ θ ∗ ) z ] for G G G of the form G ( t 1 , t 2 ) = ψ ′ ( t 1 ) ψ ( t 2 ) G(t_1,t_2)=\psi'(t_1)\psi(t_2) G ( t 1 , t 2 ) = ψ ′ ( t 1 ) ψ ( t 2 ) . Doing a second-order Taylor expansion leads to
G ( z ⋅ θ , z ~ ⋅ θ ∗ ) = G ( z 1 θ 1 + z − 1 ⋅ θ − 1 , z ~ 1 θ 1 ∗ + z ~ − 1 ⋅ θ − 1 ∗ ) (42) G(z\cdot\theta,\widetilde{z}\cdot\theta^*)=G(z_1\theta_1+z_{-1}\cdot\theta_{-1},\widetilde{z}_1\theta_1^*+\widetilde{z}_{-1}\cdot\theta_{-1}^*) \tag*{(42)} G ( z ⋅ θ , z ⋅ θ ∗ ) = G ( z 1 θ 1 + z − 1 ⋅ θ − 1 , z 1 θ 1 ∗ + z − 1 ⋅ θ − 1 ∗ ) (42)
= ψ ′ ( z − 1 ⋅ θ − 1 ) ψ ( z ~ − 1 ⋅ θ − 1 ∗ ) + ψ ′ ′ ( z − 1 ⋅ θ − 1 ) ψ ( z ~ − 1 ⋅ θ − 1 ∗ ) θ 1 z 1 + ψ ′ ( z − 1 ⋅ θ − 1 ) ψ ′ ( z ~ − 1 ⋅ θ − 1 ∗ ) z ~ 1 θ 1 ∗ (43) \begin{aligned}
=\psi'(z_{-1}\cdot\theta_{-1})\psi(\widetilde{z}_{-1}\cdot\theta_{-1}^*)+\psi''(z_{-1}\cdot\theta_{-1})\psi(\widetilde{z}_{-1}\cdot\theta_{-1}^*)\theta_1z_1 \\
&+\psi'(z_{-1}\cdot\theta_{-1})\psi'(\widetilde{z}_{-1}\cdot\theta_{-1}^*)\widetilde{z}_1\theta_1^* \tag*{(43)}
\end{aligned} = ψ ′ ( z − 1 ⋅ θ − 1 ) ψ ( z − 1 ⋅ θ − 1 ∗ ) + ψ ′′ ( z − 1 ⋅ θ − 1 ) ψ ( z − 1 ⋅ θ − 1 ∗ ) θ 1 z 1 + ψ ′ ( z − 1 ⋅ θ − 1 ) ψ ′ ( z − 1 ⋅ θ − 1 ∗ ) z 1 θ 1 ∗ (43)
+ 1 2 [ z 1 θ 1 z ~ 1 θ 1 ∗ ] ⊤ ∇ 2 G ( M ) [ z 1 θ 1 z ~ 1 θ 1 ∗ ] , (44) +\frac{1}{2}[z_1\theta_1\ \widetilde{z}_1\theta_1^*]^\top\nabla^2G(M)[z_1\theta_1\ \widetilde{z}_1\theta_1^*], \tag*{(44)} + 2 1 [ z 1 θ 1 z 1 θ 1 ∗ ] ⊤ ∇ 2 G ( M ) [ z 1 θ 1 z 1 θ 1 ∗ ] , (44)
and thus, for all i ∈ [ d ] i\in[d] i ∈ [ d ] , denoting α = E [ z i z ~ i ] \alpha=\mathbb{E}[z_i\widetilde{z}_i] α = E [ z i z i ] , and assuming that ∇ G \nabla G ∇ G is Lipschitz, we have
E [ G ( z ⋅ θ , z ~ ⋅ θ ∗ ) z i ] = θ i E [ ψ ′ ′ ( z − i ⋅ θ − i ) ψ ( z ~ − i ⋅ θ − i ∗ ) ] + α θ i ∗ E [ ψ ′ ( z − i ⋅ θ − i ) ψ ′ ( z ~ − i ⋅ θ − i ∗ ) ] + O ( θ i 2 + ( θ i ∗ ) 2 ) . (45) \begin{aligned}
\mathbb{E}[G(z\cdot\theta,\widetilde{z}\cdot\theta^*)z_i]=\theta_i\mathbb{E}[\psi''(z_{-i}\cdot\theta_{-i})\psi(\widetilde{z}_{-i}\cdot\theta_{-i}^*)] \\
&+\alpha\theta_i^*\mathbb{E}[\psi'(z_{-i}\cdot\theta_{-i})\psi'(\widetilde{z}_{-i}\cdot\theta_{-i}^*)]+O(\theta_i^2+(\theta_i^*)^2). \tag*{(45)}
\end{aligned} E [ G ( z ⋅ θ , z ⋅ θ ∗ ) z i ] = θ i E [ ψ ′′ ( z − i ⋅ θ − i ) ψ ( z − i ⋅ θ − i ∗ )] + α θ i ∗ E [ ψ ′ ( z − i ⋅ θ − i ) ψ ′ ( z − i ⋅ θ − i ∗ )] + O ( θ i 2 + ( θ i ∗ ) 2 ) . (45)
Following our previous argument, we now consider for each θ , θ ∗ \theta,\theta^* θ , θ ∗ and each i ∈ [ d ] i\in[d] i ∈ [ d ] the bivariate distributions μ ˉ i = Law ( ∑ j ≠ i θ j z j , ∑ j ≠ i θ j ∗ z ~ j ) \bar{\mu}_i=\operatorname{Law}(\sum_{j\ne i}\theta_jz_j,\sum_{j\ne i}\theta_j^*\widetilde{z}_j) μ ˉ i = Law ( ∑ j = i θ j z j , ∑ j = i θ j ∗ z j ) , where ( z i , z ~ i ) (z_i,\widetilde{z}_i) ( z i , z i ) are drawn i.i.d. from ν ˉ \bar{\nu} ν ˉ . We now have
sup i , i ′ W 1 ( μ ˉ i , μ ˉ i ′ ) = O ( ∥ θ ∥ ∞ + ∥ θ ∗ ∥ ∞ ) , (46) \sup_{i,i'}W_1(\bar{\mu}_i,\bar{\mu}_{i'})=O(\lVert\theta\rVert_\infty+\lVert\theta^*\rVert_\infty), \tag*{(46)} i , i ′ sup W 1 ( μ ˉ i , μ ˉ i ′ ) = O (∥ θ ∥ ∞ + ∥ θ ∗ ∥ ∞ ) , (46)
leading to
E [ G ( z ⋅ θ , z ~ ⋅ θ ∗ ) z ] = θ v 1 ( θ , θ ∗ ) + α θ ∗ v 2 ( θ , θ ∗ ) + ε , (47) \mathbb{E}[G(z\cdot\theta,\widetilde{z}\cdot\theta^*)z]=\theta v_1(\theta,\theta^*)+\alpha\theta^*v_2(\theta,\theta^*)+\varepsilon, \tag*{(47)} E [ G ( z ⋅ θ , z ⋅ θ ∗ ) z ] = θ v 1 ( θ , θ ∗ ) + α θ ∗ v 2 ( θ , θ ∗ ) + ε , (47)
with ∣ ε i ∣ ≤ ( C 1 ∣ θ i ∣ + C 2 ∣ α ∣ ∣ θ i ∗ ∣ ) ( ∥ θ ∥ ∞ + ∥ θ ∗ ∥ ∞ ) |\varepsilon_i|\le(C_1|\theta_i|+C_2|\alpha||\theta_i^*|)(\lVert\theta\rVert_\infty+\lVert\theta^*\rVert_\infty) ∣ ε i ∣ ≤ ( C 1 ∣ θ i ∣ + C 2 ∣ α ∣∣ θ i ∗ ∣ ) (∥ θ ∥ ∞ + ∥ θ ∗ ∥ ∞ ) and v 2 = O ( 1 ) v_2=O(1) v 2 = O ( 1 ) . We then arrive at
P θ ⊥ E [ G ( z ⋅ θ , z ~ ⋅ θ ∗ ) z ] = α θ ∗ v 2 ( θ , θ ∗ ) − χ α v 2 θ + ε − ( ε ⋅ θ ) θ . (48) P_{\theta^\perp}\mathbb{E}[G(z\cdot\theta,\widetilde{z}\cdot\theta^*)z]=\alpha\theta^*v_2(\theta,\theta^*)-\chi\alpha v_2\theta+\varepsilon-(\varepsilon\cdot\theta)\theta. \tag*{(48)} P θ ⊥ E [ G ( z ⋅ θ , z ⋅ θ ∗ ) z ] = α θ ∗ v 2 ( θ , θ ∗ ) − χ α v 2 θ + ε − ( ε ⋅ θ ) θ . (48)
Again, since ∣ ε ⋅ θ ∣ = O ( ∥ θ ∥ ∞ + ∥ θ ∗ ∥ ∞ ) |\varepsilon\cdot\theta|=O(\lVert\theta\rVert_\infty+\lVert\theta^*\rVert_\infty) ∣ ε ⋅ θ ∣ = O (∥ θ ∥ ∞ + ∥ θ ∗ ∥ ∞ ) , we have
∥ ∇ θ L ∥ ∞ ≤ C 1 ρ ∥ θ ∗ ∥ ∞ + C 2 ρ χ ∥ θ ∥ ∞ + O ( ∥ θ ∥ ∞ 2 + ∥ θ ∗ ∥ ∞ 2 ) , (49) \lVert\nabla_\theta\mathcal{L}\rVert_\infty\le C_1\rho\lVert\theta^*\rVert_\infty+C_2\rho\chi\lVert\theta\rVert_\infty+O(\lVert\theta\rVert_\infty^2+\lVert\theta^*\rVert_\infty^2), \tag*{(49)} ∥ ∇ θ L ∥ ∞ ≤ C 1 ρ ∥ θ ∗ ∥ ∞ + C 2 ρ χ ∥ θ ∥ ∞ + O (∥ θ ∥ ∞ 2 + ∥ θ ∗ ∥ ∞ 2 ) , (49)
proving the lemma.
Weak Recovery
At the weak recovery, the total overlap Z t Z_t Z t is still below the incoherence scale. In this regime the crude O ( η t ) O(\eta_t) O ( η t ) comparison error from Lemma C.3 is too large under spherical initialization. We instead track the third moment quantity ω t : = ∑ i = 1 d ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 \omega_t := \sum_{i=1}^{d} (|\theta_{t,i}| + |\theta^*_{i}|)^3 ω t := ∑ i = 1 d ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 and show it is controlled, at population level, under the order 1 + t d \frac{1+t}{\sqrt{d}} d 1 + t .
Lemma C.5 (First-order comparison in the small-overlap regime). Assume the hypotheses of Lemma C.2 and the regularity needed for Corollary E.2. Define
ω ( θ , θ ∗ ) : = ∑ i = 1 d ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) 3 . \omega(\theta,\theta^*) := \sum_{i=1}^{d} (|\theta_i| + |\theta_i^*|)^3. ω ( θ , θ ∗ ) := i = 1 ∑ d ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) 3 . Let
ρ ˙ : = − ⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ , χ ˙ : = − ⟨ ∇ S , θ L ( θ , β ) , θ ∗ ⟩ , \dot{\rho} := -\langle\nabla_{S,\beta}L(\theta,\beta),\beta^*\rangle,\qquad
\dot{\chi} := -\langle\nabla_{S,\theta}L(\theta,\beta),\theta^*\rangle, ρ ˙ := − ⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ , χ ˙ := − ⟨ ∇ S , θ L ( θ , β ) , θ ∗ ⟩ , and define ρ ˙ G , χ ˙ G \dot{\rho}_G,\dot{\chi}_G ρ ˙ G , χ ˙ G analogously with L G L_G L G . Then, uniformly on the small-overlap region 0 ≤ ρ , χ 0 \le\rho,\chi 0 ≤ ρ , χ and Z : = ρ + χ ≤ c s m Z := \rho+\chi\le c_{\mathrm{sm}} Z := ρ + χ ≤ c sm ,
∣ ρ ˙ − ρ ˙ G ∣ + ∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) , (50) |\dot{\rho}-\dot{\rho}_G|+|\dot{\chi}-\dot{\chi}_G| \le C\omega(\theta,\theta^*), \tag*{(50)} ∣ ρ ˙ − ρ ˙ G ∣ + ∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) , (50) where C C C is independent of d d d .
Proof. Throughout, constants may change from line to line and depend only on the model and moment constants. Set
U i : = x ( i ) ⋅ β , U i ∗ : = x ( i ) ⋅ β ∗ , U_i := x^{(i)}\cdot\beta,\qquad U_i^* := x^{(i)}\cdot\beta^*, U i := x ( i ) ⋅ β , U i ∗ := x ( i ) ⋅ β ∗ ,
so that ( U i , U i ∗ ) i = 1 d (U_i,U_i^*)_{i=1}^{d} ( U i , U i ∗ ) i = 1 d are iid centered Gaussian pairs with correlation ρ \rho ρ . Let
X i : = ϕ ( U i ) , X i ∗ : = ϕ ( U i ∗ ) , X_i := \phi(U_i),\qquad X_i^* := \phi(U_i^*), X i := ϕ ( U i ) , X i ∗ := ϕ ( U i ∗ ) ,
and
S : = ∑ i = 1 d θ i X i , S ∗ : = ∑ i = 1 d θ i ∗ X i ∗ . S := \sum_{i=1}^{d}\theta_iX_i,\qquad S^* := \sum_{i=1}^{d}\theta_i^*X_i^*. S := i = 1 ∑ d θ i X i , S ∗ := i = 1 ∑ d θ i ∗ X i ∗ .
Let ( Γ i , Γ i ∗ ) i = 1 d (\Gamma_i,\Gamma_i^*)_{i=1}^{d} ( Γ i , Γ i ∗ ) i = 1 d be iid centered Gaussian pairs with the same block covariance as ( X i , X i ∗ ) (X_i,X_i^*) ( X i , X i ∗ ) :
E [ Γ i 2 ] = E [ ( Γ i ∗ ) 2 ] = 1 , E [ Γ i Γ i ∗ ] = κ ( ρ ) , \mathbb{E}[\Gamma_i^2]=\mathbb{E}[(\Gamma_i^*)^2]=1,\qquad
\mathbb{E}[\Gamma_i\Gamma_i^*]=\kappa(\rho), E [ Γ i 2 ] = E [( Γ i ∗ ) 2 ] = 1 , E [ Γ i Γ i ∗ ] = κ ( ρ ) ,
and write
S G : = ∑ i = 1 d θ i Γ i , S G ∗ : = ∑ i = 1 d θ i ∗ Γ i ∗ . S_G := \sum_{i=1}^{d}\theta_i\Gamma_i,\qquad S_G^* := \sum_{i=1}^{d}\theta_i^*\Gamma_i^*. S G := i = 1 ∑ d θ i Γ i , S G ∗ := i = 1 ∑ d θ i ∗ Γ i ∗ .
We prove the two block comparisons separately.
The θ \theta θ -block. Let
w : = P θ ⊥ θ ∗ = θ ∗ − χ θ . w := P_{\theta^\perp}\theta^*=\theta^*-\chi\theta. w := P θ ⊥ θ ∗ = θ ∗ − χ θ .
Since w ∈ T θ S d − 1 w\in T_\theta S^{d-1} w ∈ T θ S d − 1 ,
⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ = ⟨ ∇ θ ( L − L G ) ( θ , β ) , w ⟩ . \langle\nabla_{S,\theta}(L-L_G)(\theta,\beta),\theta^*\rangle
=
\langle\nabla_\theta(L-L_G)(\theta,\beta),w\rangle. ⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ = ⟨ ∇ θ ( L − L G ) ( θ , β ) , w ⟩ .
Thus the θ \theta θ -comparison is the difference between the true and Gaussian expectations of a function of the form
h ( x , x ∗ ) = 2 ( ψ ( x ⋅ θ ) − ψ ( x ∗ ⋅ θ ∗ ) ) ψ ′ ( x ⋅ θ ) ( x ⋅ w ) . h(x,x^*)=2(\psi(x\cdot\theta)-\psi(x^*\cdot\theta^*))\psi'(x\cdot\theta)(x\cdot w). h ( x , x ∗ ) = 2 ( ψ ( x ⋅ θ ) − ψ ( x ∗ ⋅ θ ∗ )) ψ ′ ( x ⋅ θ ) ( x ⋅ w ) .
Apply Corollary E.2 to h h h . For a fixed block j j j , every third partial derivative ∂ s a ∂ t b h \partial_s^a\partial_t^b h ∂ s a ∂ t b h , a + b = 3 a+b=3 a + b = 3 , is a finite sum of terms of the following two types:
C ∣ θ j ∣ a ∣ θ j ∗ ∣ b ∣ ℓ 2 ∣ , a + b = 3 , C|\theta_j|^a|\theta_j^*|^b|\ell_2|,\qquad a+b=3, C ∣ θ j ∣ a ∣ θ j ∗ ∣ b ∣ ℓ 2 ∣ , a + b = 3 ,
and
C ∣ w j ∣ ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ , a ′ + b ′ = 2 , C|w_j||\theta_j|^{a'}|\theta_j^*|^{b'},\qquad a'+b'=2, C ∣ w j ∣∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ , a ′ + b ′ = 2 ,
where ℓ 2 \ell_2 ℓ 2 denotes the linear form w ⋅ W w\cdot W w ⋅ W evaluated at the corresponding Lindeberg hybrid vector. Since ∥ w ∥ 2 ≤ 1 \|w\|_2\le1 ∥ w ∥ 2 ≤ 1 , the moment assumptions imply
E ∣ ℓ 2 ∣ ≤ C . \mathbb{E}|\ell_2|\le C. E ∣ ℓ 2 ∣ ≤ C .
Therefore
∣ ⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ ∣ ≤ C ∑ j = 1 d ∑ a + b = 3 ∣ θ j ∣ a ∣ θ j ∗ ∣ b + C ∑ j = 1 d ∣ w j ∣ ∑ a ′ + b ′ = 2 ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ . \left|\left\langle\nabla_{S,\theta}(L-L_G)(\theta,\beta),\theta^*\right\rangle\right| \le C\sum_{j=1}^{d}\sum_{a+b=3}|\theta_j|^a|\theta_j^*|^b+C\sum_{j=1}^{d}|w_j|\sum_{a'+b'=2}|\theta_j|^{a'}|\theta_j^*|^{b'}. ∣ ⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ ∣ ≤ C j = 1 ∑ d a + b = 3 ∑ ∣ θ j ∣ a ∣ θ j ∗ ∣ b + C j = 1 ∑ d ∣ w j ∣ a ′ + b ′ = 2 ∑ ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ .
Now
∣ w j ∣ = ∣ θ j ∗ − χ θ j ∣ ≤ ∣ θ j ∗ ∣ + ∣ θ j ∣ , |w_j|=|\theta_j^*-\chi\theta_j|\leq|\theta_j^*|+|\theta_j|, ∣ w j ∣ = ∣ θ j ∗ − χ θ j ∣ ≤ ∣ θ j ∗ ∣ + ∣ θ j ∣ ,
because ∣ χ ∣ ≤ 1 |\chi|\leq1 ∣ χ ∣ ≤ 1 . Hence, for every j j j ,
∑ a + b = 3 ∣ θ j ∣ a ∣ θ j ∗ ∣ b ≤ C ( ∣ θ j ∣ + ∣ θ j ∗ ∣ ) 3 , \sum_{a+b=3}|\theta_j|^a|\theta_j^*|^b\leq C(|\theta_j|+|\theta_j^*|)^3, a + b = 3 ∑ ∣ θ j ∣ a ∣ θ j ∗ ∣ b ≤ C ( ∣ θ j ∣ + ∣ θ j ∗ ∣ ) 3 ,
and
∣ w j ∣ ∑ a ′ + b ′ = 2 ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ ≤ C ( ∣ θ j ∣ + ∣ θ j ∗ ∣ ) 3 . |w_j|\sum_{a'+b'=2}|\theta_j|^{a'}|\theta_j^*|^{b'}\leq C(|\theta_j|+|\theta_j^*|)^3. ∣ w j ∣ a ′ + b ′ = 2 ∑ ∣ θ j ∣ a ′ ∣ θ j ∗ ∣ b ′ ≤ C ( ∣ θ j ∣ + ∣ θ j ∗ ∣ ) 3 .
Summing over j j j , we obtain
∣ ⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ ∣ ≤ C ω ( θ , θ ∗ ) . \left|\left\langle\nabla_{S,\theta}(L-L_G)(\theta,\beta),\theta^*\right\rangle\right|\leq C\omega(\theta,\theta^*). ∣ ⟨ ∇ S , θ ( L − L G ) ( θ , β ) , θ ∗ ⟩ ∣ ≤ C ω ( θ , θ ∗ ) .
Equivalently,
∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) . |\dot{\chi}-\dot{\chi}_G|\leq C\omega(\theta,\theta^*). ∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) .
The β \beta β -block. Let
b : = P β ⊥ β ∗ = β ∗ − ρ β . b:=P_{\beta^\perp}\beta^*=\beta^*-\rho\beta. b := P β ⊥ β ∗ = β ∗ − ρβ .
Then
⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = ⟨ ∇ β L ( θ , β ) , b ⟩ . \left\langle\nabla_{S,\beta}L(\theta,\beta),\beta^*\right\rangle=\left\langle\nabla_\beta L(\theta,\beta),b\right\rangle. ⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = ⟨ ∇ β L ( θ , β ) , b ⟩ .
As in the proof of Lemma C.2, Gaussian integration by parts in the b b b -direction gives the exact formula
− ⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = 2 ( 1 − ρ 2 ) ∑ i = 1 d θ i θ i ∗ E [ ψ ′ ( S ) ψ ′ ( S ∗ ) ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) ] . -\left\langle\nabla_{S,\beta}L(\theta,\beta),\beta^*\right\rangle=2(1-\rho^2)\sum_{i=1}^{d}\theta_i\theta_i^*\mathbb{E}\left[\psi'(S)\psi'(S^*)\phi'(U_i)\phi'(U_i^*)\right]. − ⟨ ∇ S , β L ( θ , β ) , β ∗ ⟩ = 2 ( 1 − ρ 2 ) i = 1 ∑ d θ i θ i ∗ E [ ψ ′ ( S ) ψ ′ ( S ∗ ) ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) ] .
Set
F ( s , t ) : = ψ ′ ( s ) ψ ′ ( t ) , q i : = ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) , c i : = θ i θ i ∗ . F(s,t):=\psi'(s)\psi'(t),\qquad q_i:=\phi'(U_i)\phi'(U_i^*),\qquad c_i:=\theta_i\theta_i^*. F ( s , t ) := ψ ′ ( s ) ψ ′ ( t ) , q i := ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) , c i := θ i θ i ∗ .
We compare
∑ i = 1 d c i E [ F ( S , S ∗ ) q i ] \sum_{i=1}^{d}c_i\mathbb{E}[F(S,S^*)q_i] i = 1 ∑ d c i E [ F ( S , S ∗ ) q i ]
with its Gaussian analogue.
Let
S − i : = S − θ i X i , S − i ∗ : = S ∗ − θ i ∗ X i ∗ . S_{-i}:=S-\theta_iX_i,\qquad S_{-i}^*:=S^*-\theta_i^*X_i^*. S − i := S − θ i X i , S − i ∗ := S ∗ − θ i ∗ X i ∗ .
Since ( S − i , S − i ∗ ) (S_{-i},S_{-i}^*) ( S − i , S − i ∗ ) is independent of ( U i , U i ∗ ) (U_i,U_i^*) ( U i , U i ∗ ) ,
E [ F ( S − i , S − i ∗ ) q i ] = E [ F ( S − i , S − i ∗ ) ] E [ q i ] . \mathbb{E}[F(S_{-i},S_{-i}^*)q_i]=\mathbb{E}[F(S_{-i},S_{-i}^*)]\mathbb{E}[q_i]. E [ F ( S − i , S − i ∗ ) q i ] = E [ F ( S − i , S − i ∗ )] E [ q i ] .
By the Hermite identity,
E [ q i ] = E [ ϕ ′ ( U i ) ϕ ′ ( U i ∗ ) ] = κ ′ ( ρ ) . \mathbb{E}[q_i]=\mathbb{E}[\phi'(U_i)\phi'(U_i^*)]=\kappa'(\rho). E [ q i ] = E [ ϕ ′ ( U i ) ϕ ′ ( U i ∗ )] = κ ′ ( ρ ) .
Moreover, by the Lipschitz bound on F F F and the moment assumptions,
∣ E [ F ( S , S ∗ ) q i ] − E [ F ( S − i , S − i ∗ ) q i ] ∣ ≤ C ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) . \left|\mathbb{E}[F(S,S^*)q_i]-\mathbb{E}[F(S_{-i},S_{-i}^*)q_i]\right|\leq C(|\theta_i|+|\theta_i^*|). E [ F ( S , S ∗ ) q i ] − E [ F ( S − i , S − i ∗ ) q i ] ≤ C ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) .
Multiplying by ∣ c i ∣ = ∣ θ i θ i ∗ ∣ |c_i|=|\theta_i\theta_i^*| ∣ c i ∣ = ∣ θ i θ i ∗ ∣ and summing yields
∑ i ∣ c i ∣ ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) = ∑ i ∣ θ i θ i ∗ ∣ ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) ≤ C ω ( θ , θ ∗ ) . \sum_i|c_i|(|\theta_i|+|\theta_i^*|)=\sum_i|\theta_i\theta_i^*|(|\theta_i|+|\theta_i^*|)\leq C\omega(\theta,\theta^*). i ∑ ∣ c i ∣ ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) = i ∑ ∣ θ i θ i ∗ ∣ ( ∣ θ i ∣ + ∣ θ i ∗ ∣ ) ≤ C ω ( θ , θ ∗ ) .
Hence
∑ i = 1 d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) ∑ i = 1 d c i E [ F ( S − i , S − i ∗ ) ] + O ( ω ) . \sum_{i=1}^{d}c_i\mathbb{E}[F(S,S^*)q_i]=\kappa'(\rho)\sum_{i=1}^{d}c_i\mathbb{E}[F(S_{-i},S_{-i}^*)]+O(\omega). i = 1 ∑ d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) i = 1 ∑ d c i E [ F ( S − i , S − i ∗ )] + O ( ω ) .
The same leave-one-out argument gives
∑ i = 1 d c i E [ F ( S − i , S − i ∗ ) ] = ( ∑ i = 1 d c i ) E [ F ( S , S ∗ ) ] + O ( ω ) . \sum_{i=1}^{d} c_i\mathbb{E}[F(S_{-i},S^*_{-i})] = \left(\sum_{i=1}^{d}c_i\right)\mathbb{E}[F(S,S^*)] + O(\omega). i = 1 ∑ d c i E [ F ( S − i , S − i ∗ )] = ( i = 1 ∑ d c i ) E [ F ( S , S ∗ )] + O ( ω ) .
Since
∑ i = 1 d c i = θ ⋅ θ ∗ = χ , \sum_{i=1}^{d}c_i=\theta\cdot\theta^*=\chi, i = 1 ∑ d c i = θ ⋅ θ ∗ = χ ,
we obtain
∑ i = 1 d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) χ E [ F ( S , S ∗ ) ] + O ( ω ) . \sum_{i=1}^{d}c_i\mathbb{E}[F(S,S^*)q_i]=\kappa'(\rho)\chi\mathbb{E}[F(S,S^*)]+O(\omega). i = 1 ∑ d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) χ E [ F ( S , S ∗ )] + O ( ω ) .
It remains to replace ( S , S ∗ ) (S,S^*) ( S , S ∗ ) by ( S G , S G ∗ ) (S_G,S_G^*) ( S G , S G ∗ ) . Applying Corollary E.2 to
h ( x , x ∗ ) = F ( x ⋅ θ , x ∗ ⋅ θ ∗ ) h(x,x^*)=F(x\cdot\theta,x^*\cdot\theta^*) h ( x , x ∗ ) = F ( x ⋅ θ , x ∗ ⋅ θ ∗ )
gives
∣ E [ F ( S , S ∗ ) ] − E [ F ( S G , S G ∗ ) ] ∣ ≤ C ∑ j = 1 d ∑ a + b = 3 ∣ θ j ∣ a ∣ θ j ∗ ∣ b ≤ C ω ( θ , θ ∗ ) . \left|\mathbb{E}[F(S,S^*)]-\mathbb{E}[F(S_G,S_G^*)]\right|
\le C\sum_{j=1}^{d}\sum_{a+b=3}|\theta_j|^a|\theta_j^*|^b
\le C\omega(\theta,\theta^*). ∣ E [ F ( S , S ∗ )] − E [ F ( S G , S G ∗ )] ∣ ≤ C j = 1 ∑ d a + b = 3 ∑ ∣ θ j ∣ a ∣ θ j ∗ ∣ b ≤ C ω ( θ , θ ∗ ) .
Since κ ′ ( ρ ) \kappa'(\rho) κ ′ ( ρ ) and χ \chi χ are bounded, we conclude
∑ i = 1 d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) χ E [ F ( S G , S G ∗ ) ] + O ( ω ) . \sum_{i=1}^{d}c_i\mathbb{E}[F(S,S^*)q_i]=\kappa'(\rho)\chi\mathbb{E}[F(S_G,S_G^*)]+O(\omega). i = 1 ∑ d c i E [ F ( S , S ∗ ) q i ] = κ ′ ( ρ ) χ E [ F ( S G , S G ∗ )] + O ( ω ) .
In the Gaussian equivalent model, Price’s identity gives
− ⟨ ∇ S , β L G ( θ , β ) , β ∗ ⟩ = 2 ( 1 − ρ 2 ) κ ′ ( ρ ) χ E [ F ( S G , S G ∗ ) ] . -\langle\nabla_{S,\beta}L_G(\theta,\beta),\beta^*\rangle
=2(1-\rho^2)\kappa'(\rho)\chi\mathbb{E}[F(S_G,S_G^*)]. − ⟨ ∇ S , β L G ( θ , β ) , β ∗ ⟩ = 2 ( 1 − ρ 2 ) κ ′ ( ρ ) χ E [ F ( S G , S G ∗ )] .
Comparing the last two displays and using 0 ≤ 1 − ρ 2 ≤ 1 0\le1-\rho^2\le1 0 ≤ 1 − ρ 2 ≤ 1 , we get
∣ ⟨ ∇ S , β ( L − L G ) ( θ , β ) , β ∗ ⟩ ∣ ≤ C ω ( θ , θ ∗ ) . \left|\langle\nabla_{S,\beta}(L-L_G)(\theta,\beta),\beta^*\rangle\right|
\le C\omega(\theta,\theta^*). ∣ ⟨ ∇ S , β ( L − L G ) ( θ , β ) , β ∗ ⟩ ∣ ≤ C ω ( θ , θ ∗ ) .
Equivalently,
∣ ρ ˙ − ρ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) . |\dot{\rho}-\dot{\rho}_G|\le C\omega(\theta,\theta^*). ∣ ρ ˙ − ρ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) .
Combining the θ \theta θ - and β \beta β -block estimates proves
∣ ρ ˙ − ρ ˙ G ∣ + ∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) . |\dot{\rho}-\dot{\rho}_G|+|\dot{\chi}-\dot{\chi}_G|
\le C\omega(\theta,\theta^*). ∣ ρ ˙ − ρ ˙ G ∣ + ∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) .
Lemma C.6 (Small-overlap drift estimates). Under the assumptions of Lemma C.2, with the same definition of ω ( θ , θ ∗ ) \omega(\theta,\theta^*) ω ( θ , θ ∗ ) in C.5. There exist constants c s m , a , b , C > 0 c_{\rm sm},a,b,C>0 c sm , a , b , C > 0 , independent of d d d , such that whenever ρ , χ ≥ 0 \rho,\chi\ge0 ρ , χ ≥ 0 , Z : = ρ + χ ≤ c s m Z:=\rho+\chi\le c_{\rm sm} Z := ρ + χ ≤ c sm , the population overlap drifts satisfy
ρ ˙ ≥ a χ − b ω ( θ , θ ∗ ) , χ ˙ ≥ a ρ − b ω ( θ , θ ∗ ) . (51) \dot{\rho}\ge a\chi-b\omega(\theta,\theta^*),\qquad
\dot{\chi}\ge a\rho-b\omega(\theta,\theta^*). \tag*{(51)} ρ ˙ ≥ a χ − bω ( θ , θ ∗ ) , χ ˙ ≥ a ρ − bω ( θ , θ ∗ ) . (51) Moreover, with C ~ : = 2 α 1 2 α 1 2 > 0 \widetilde{C}:=2\alpha_1^2\alpha_1^2>0 C := 2 α 1 2 α 1 2 > 0 , one has the refined two-sided expansion
ρ ˙ = C ~ χ ( 1 − ρ 2 ) + R ρ , χ ˙ = C ~ ρ ( 1 − χ 2 ) + R χ , (52) \dot{\rho}=\widetilde{C}\chi(1-\rho^2)+R_\rho,\qquad
\dot{\chi}=\widetilde{C}\rho(1-\chi^2)+R_\chi, \tag*{(52)} ρ ˙ = C χ ( 1 − ρ 2 ) + R ρ , χ ˙ = C ρ ( 1 − χ 2 ) + R χ , (52) where ∣ R ρ ∣ + ∣ R χ ∣ ≤ C ( Z 2 + ω ( θ , θ ∗ ) ) |R_\rho|+|R_\chi|\le C(Z^2+\omega(\theta,\theta^*)) ∣ R ρ ∣ + ∣ R χ ∣ ≤ C ( Z 2 + ω ( θ , θ ∗ )) . Consequently, for D : = ρ − χ D:=\rho-\chi D := ρ − χ ,
D ˙ = − C ~ ( 1 + ρ χ ) D + R D , ∣ R D ∣ ≤ C ( Z 2 + ω ( θ , θ ∗ ) ) . (53) \dot{D}=-\widetilde{C}(1+\rho\chi)D+R_D,\qquad
|R_D|\le C(Z^2+\omega(\theta,\theta^*)). \tag*{(53)} D ˙ = − C ( 1 + ρ χ ) D + R D , ∣ R D ∣ ≤ C ( Z 2 + ω ( θ , θ ∗ )) . (53)
Proof. Let L G L_G L G denote the Gaussian-equivalent population loss. As in the proof of Lemma C.3, the Gaussian overlap drifts are
ρ ˙ G = 2 χ κ ′ ( ρ ) A ( q ) ( 1 − ρ 2 ) , χ ˙ G = 2 κ ( ρ ) A ( q ) ( 1 − χ 2 ) , \dot{\rho}_G=2\chi\kappa'(\rho)A(q)(1-\rho^2),\qquad
\dot{\chi}_G=2\kappa(\rho)A(q)(1-\chi^2), ρ ˙ G = 2 χ κ ′ ( ρ ) A ( q ) ( 1 − ρ 2 ) , χ ˙ G = 2 κ ( ρ ) A ( q ) ( 1 − χ 2 ) ,
where
q : = κ ( ρ ) χ , A ( q ) : = ∑ k ≥ 1 k α k 2 q k − 1 , κ ( ρ ) : = ∑ ℓ ≥ 1 α ℓ 2 ρ ℓ . (54) q:=\kappa(\rho)\chi,\qquad
A(q):=\sum_{k\ge1}k\alpha_k^2q^{k-1},\qquad
\kappa(\rho):=\sum_{\ell\ge1}\alpha_\ell^2\rho^\ell. \tag*{(54)} q := κ ( ρ ) χ , A ( q ) := k ≥ 1 ∑ k α k 2 q k − 1 , κ ( ρ ) := ℓ ≥ 1 ∑ α ℓ 2 ρ ℓ . (54)
Since a 1 ≠ 0 a_1 \ne0 a 1 = 0 and α 1 ≠ 0 \alpha_1 \ne0 α 1 = 0 ,
κ ′ ( 0 ) = a 1 2 > 0 , A ( 0 ) = α 1 2 > 0. \kappa'(0) = a_1^2 > 0,\qquad A(0) = \alpha_1^2 > 0. κ ′ ( 0 ) = a 1 2 > 0 , A ( 0 ) = α 1 2 > 0.
First we prove the cooperative lower bounds. On the positive-overlap region ρ , χ ≥ 0 \rho, \chi\ge0 ρ , χ ≥ 0 , all terms in the Gaussian drift formulas are nonnegative. Keeping only the leading terms gives
ρ ˙ G ≥ 2 α 1 2 a 1 2 χ ( 1 − ρ 2 ) , χ ˙ G ≥ 2 α 1 2 a 1 2 ρ ( 1 − χ 2 ) . \dot{\rho}_G \ge2\alpha_1^2a_1^2\chi(1-\rho^2),\qquad\dot{\chi}_G \ge2\alpha_1^2a_1^2\rho(1-\chi^2). ρ ˙ G ≥ 2 α 1 2 a 1 2 χ ( 1 − ρ 2 ) , χ ˙ G ≥ 2 α 1 2 a 1 2 ρ ( 1 − χ 2 ) .
Choose c s m > 0 c_{\mathrm{sm}}>0 c sm > 0 sufficiently small so that Z ≤ c s m Z\le c_{\mathrm{sm}} Z ≤ c sm implies ρ , χ ≤ c s m \rho,\chi\le c_{\mathrm{sm}} ρ , χ ≤ c sm and hence
1 − ρ 2 ≥ 1 2 , 1 − χ 2 ≥ 1 2 . 1-\rho^2 \ge\frac{1}{2},\qquad1-\chi^2 \ge\frac{1}{2}. 1 − ρ 2 ≥ 2 1 , 1 − χ 2 ≥ 2 1 .
Therefore
ρ ˙ G ≥ c 0 χ , χ ˙ G ≥ c 0 ρ \dot{\rho}_G \ge c_0\chi,\qquad\dot{\chi}_G \ge c_0\rho ρ ˙ G ≥ c 0 χ , χ ˙ G ≥ c 0 ρ
for c 0 : = α 1 2 a 1 2 > 0 c_0 := \alpha_1^2a_1^2 > 0 c 0 := α 1 2 a 1 2 > 0 .
It remains to transfer these estimates from L G L_G L G to the true population loss. By Lemma C.5,
∣ ρ ˙ − ρ ˙ G ∣ + ∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) . |\dot{\rho}-\dot{\rho}_G|+|\dot{\chi}-\dot{\chi}_G|\le C\omega(\theta,\theta^*). ∣ ρ ˙ − ρ ˙ G ∣ + ∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) .
Hence, after setting a : = c 0 / 2 a:=c_0/2 a := c 0 /2 and increasing b b b ,
ρ ˙ ≥ a χ − b ω ( θ , θ ∗ ) , χ ˙ ≥ a ρ − b ω ( θ , θ ∗ ) , \dot{\rho}\ge a\chi-b\omega(\theta,\theta^*),\qquad\dot{\chi}\ge a\rho-b\omega(\theta,\theta^*), ρ ˙ ≥ a χ − bω ( θ , θ ∗ ) , χ ˙ ≥ a ρ − bω ( θ , θ ∗ ) ,
which proves (50).
We now prove the two-sided expansion. Since Z ≤ c s m Z\le c_{\mathrm{sm}} Z ≤ c sm , we have
∣ ρ ∣ + ∣ χ ∣ ≤ c s m . |\rho|+|\chi|\le c_{\mathrm{sm}}. ∣ ρ ∣ + ∣ χ ∣ ≤ c sm .
Taylor expanding κ \kappa κ , κ ′ \kappa' κ ′ , and A A A near the origin gives
κ ( ρ ) = a 1 2 ρ + O ( ρ 2 ) , κ ′ ( ρ ) = a 1 2 + O ( ∣ ρ ∣ ) , \kappa(\rho)=a_1^2\rho+O(\rho^2),\qquad\kappa'(\rho)=a_1^2+O(|\rho|), κ ( ρ ) = a 1 2 ρ + O ( ρ 2 ) , κ ′ ( ρ ) = a 1 2 + O ( ∣ ρ ∣ ) ,
and since
q = κ ( ρ ) χ = O ( Z 2 ) , q=\kappa(\rho)\chi=O(Z^2), q = κ ( ρ ) χ = O ( Z 2 ) ,
also
A ( q ) = α 1 2 + O ( Z 2 ) . A(q)=\alpha_1^2+O(Z^2). A ( q ) = α 1 2 + O ( Z 2 ) .
Substituting these expansions into the exact Gaussian drift formulas yields
ρ ˙ G = 2 α 1 2 a 1 2 χ ( 1 − ρ 2 ) + O ( Z 2 ) , \dot{\rho}_G=2\alpha_1^2a_1^2\chi(1-\rho^2)+O(Z^2), ρ ˙ G = 2 α 1 2 a 1 2 χ ( 1 − ρ 2 ) + O ( Z 2 ) ,
and
χ ˙ G = 2 α 1 2 a 1 2 ρ ( 1 − χ 2 ) + O ( Z 2 ) . \dot{\chi}_G=2\alpha_1^2a_1^2\rho(1-\chi^2)+O(Z^2). χ ˙ G = 2 α 1 2 a 1 2 ρ ( 1 − χ 2 ) + O ( Z 2 ) .
Indeed, in the first formula the error from κ ′ ( ρ ) − a 1 2 \kappa'(\rho)-a_1^2 κ ′ ( ρ ) − a 1 2 is O ( ∣ ρ ∣ ∣ χ ∣ ) = O ( Z 2 ) O(|\rho||\chi|)=O(Z^2) O ( ∣ ρ ∣∣ χ ∣ ) = O ( Z 2 ) , while the error from A ( q ) − α 1 2 A(q)-\alpha_1^2 A ( q ) − α 1 2 is O ( ∣ χ ∣ Z 2 ) = O ( Z 3 ) O(|\chi|Z^2)=O(Z^3) O ( ∣ χ ∣ Z 2 ) = O ( Z 3 ) . In the second formula, the error from κ ( ρ ) − a 1 2 ρ \kappa(\rho)-a_1^2\rho κ ( ρ ) − a 1 2 ρ is O ( ρ 2 ) = O ( Z 2 ) O(\rho^2)=O(Z^2) O ( ρ 2 ) = O ( Z 2 ) , and the error from A ( q ) − α 1 2 A(q)-\alpha_1^2 A ( q ) − α 1 2 is O ( ∣ ρ ∣ Z 2 ) = O ( Z 3 ) O(|\rho|Z^2)=O(Z^3) O ( ∣ ρ ∣ Z 2 ) = O ( Z 3 ) .
Thus, with C ~ : = 2 α 1 2 a 1 2 \widetilde{C}:=2\alpha_1^2a_1^2 C := 2 α 1 2 a 1 2 , we have
ρ ˙ G = C ~ χ ( 1 − ρ 2 ) + O ( Z 2 ) , χ ˙ G = C ~ ρ ( 1 − χ 2 ) + O ( Z 2 ) . \dot{\rho}_G=\widetilde{C}\chi(1-\rho^2)+O(Z^2),\qquad\dot{\chi}_G=\widetilde{C}\rho(1-\chi^2)+O(Z^2). ρ ˙ G = C χ ( 1 − ρ 2 ) + O ( Z 2 ) , χ ˙ G = C ρ ( 1 − χ 2 ) + O ( Z 2 ) .
Using again the refined comparison estimate
∣ ρ ˙ − ρ ˙ G ∣ + ∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) , |\dot{\rho}-\dot{\rho}_G|+|\dot{\chi}-\dot{\chi}_G|\le C\omega(\theta,\theta^*), ∣ ρ ˙ − ρ ˙ G ∣ + ∣ χ ˙ − χ ˙ G ∣ ≤ C ω ( θ , θ ∗ ) ,
we obtain
ρ ˙ = C ~ χ ( 1 − ρ 2 ) + R ρ , χ ˙ = C ~ ρ ( 1 − χ 2 ) + R χ , \dot{\rho}=\widetilde{C}\chi(1-\rho^2)+R_\rho,\qquad\dot{\chi}=\widetilde{C}\rho(1-\chi^2)+R_\chi, ρ ˙ = C χ ( 1 − ρ 2 ) + R ρ , χ ˙ = C ρ ( 1 − χ 2 ) + R χ ,
with ∣ R ρ ∣ + ∣ R χ ∣ ≤ C ( Z 2 + ω ( θ , θ ∗ ) ) |R_\rho|+|R_\chi|\le C(Z^2+\omega(\theta,\theta^*)) ∣ R ρ ∣ + ∣ R χ ∣ ≤ C ( Z 2 + ω ( θ , θ ∗ )) . This proves (51).
Finally, subtract the two expansions. Since
χ ( 1 − ρ 2 ) − ρ ( 1 − χ 2 ) = χ − ρ − ρ 2 χ + ρ χ 2 = − ( ρ − χ ) − ρ χ ( ρ − χ ) = − ( 1 + ρ χ ) D , \chi(1-\rho^2)-\rho(1-\chi^2)=\chi-\rho-\rho^2\chi+\rho\chi^2=-(\rho-\chi)-\rho\chi(\rho-\chi)=-(1+\rho\chi)D, χ ( 1 − ρ 2 ) − ρ ( 1 − χ 2 ) = χ − ρ − ρ 2 χ + ρ χ 2 = − ( ρ − χ ) − ρ χ ( ρ − χ ) = − ( 1 + ρ χ ) D ,
we get
D ˙ = − C ~ ( 1 + ρ χ ) D + R D , \dot{D}=-\widetilde{C}(1+\rho\chi)D+R_D, D ˙ = − C ( 1 + ρ χ ) D + R D ,
where
∣ R D ∣ ≤ ∣ R ρ ∣ + ∣ R χ ∣ ≤ C ( Z 2 + ω ( θ , θ ∗ ) ) . |R_D|\le|R_\rho|+|R_\chi|\le C(Z^2+\omega(\theta,\theta^*)). ∣ R D ∣ ≤ ∣ R ρ ∣ + ∣ R χ ∣ ≤ C ( Z 2 + ω ( θ , θ ∗ )) .
This proves (52).
Lemma C.7 (Third-moment control at population level). Let Ω t : = ∥ θ t ∥ 3 3 = ∑ i = 1 d ∣ θ t , i ∣ 3 \Omega_t := \|\theta_t\|_3^3 = \sum_{i=1}^d |\theta_{t,i}|^3 Ω t := ∥ θ t ∥ 3 3 = ∑ i = 1 d ∣ θ t , i ∣ 3 , ω t : = ∑ i = 1 d ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 \omega_t := \sum_{i=1}^d (|\theta_{t,i}| + |\theta_i^*|)^3 ω t := ∑ i = 1 d ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 . Work on the initialization event of Lemma C.1. Then, on the positive-overlap region ρ t , χ t ≥ 0 \rho_t, \chi_t \ge0 ρ t , χ t ≥ 0 , with Z t : = ρ t + χ t Z_t := \rho_t + \chi_t Z t := ρ t + χ t , the population flow satisfies
Ω ˙ t ≤ C ( Z t ∥ θ ∗ ∥ ∞ + Z t 2 η t + η t 2 ) . (55) \dot{\Omega}_t \le C\left(Z_t\|\theta^*\|_\infty+ Z_t^2\eta_t + \eta_t^2\right). \tag*{(55)} Ω ˙ t ≤ C ( Z t ∥ θ ∗ ∥ ∞ + Z t 2 η t + η t 2 ) . (55) Consequently, suppose that on an interval 0 ≤ t ≤ T 0 \le t \le T 0 ≤ t ≤ T , with T ≤ C T log d T \le C_T \log d T ≤ C T log d , one has
Z t 2 ≤ A η t , η t ≤ C η log d + 1 + t d . Z_t^2 \le A\eta_t,\qquad\eta_t \le C_\eta\frac{\sqrt{\log d}+1+t}{\sqrt{d}}. Z t 2 ≤ A η t , η t ≤ C η d log d + 1 + t . Then, for all sufficiently large d d d ,
Ω t ≤ C Ω 1 + t d , ω t ≤ C ω 1 + t d , 0 ≤ t ≤ T . (56) \Omega_t \le C_\Omega\frac{1+t}{\sqrt{d}},\qquad\omega_t \le C_\omega\frac{1+t}{\sqrt{d}},\qquad0 \le t \le T. \tag*{(56)} Ω t ≤ C Ω d 1 + t , ω t ≤ C ω d 1 + t , 0 ≤ t ≤ T . (56) The constants C Ω , C ω C_\Omega, C_\omega C Ω , C ω may depend on K K K , A A A , C η C_\eta C η , C T C_T C T , but not on d d d .
Proof. The population θ \theta θ -flow is
θ ˙ t = − ∇ S , θ L ( θ t , β t ) . \dot{\theta}_t = -\nabla_{S,\theta}L(\theta_t,\beta_t). θ ˙ t = − ∇ S , θ L ( θ t , β t ) .
Let
v t = ( ∣ θ t , i ∣ θ t , i ) i = 1 d . v_t = (|\theta_{t,i}|\theta_{t,i})_{i=1}^d. v t = ( ∣ θ t , i ∣ θ t , i ) i = 1 d .
Since x ↦ ∣ x ∣ 3 x \mapsto|x|^3 x ↦ ∣ x ∣ 3 is C 1 C^1 C 1 , we have
d d t Ω t = 3 ∑ i ∣ θ t , i ∣ θ t , i θ ˙ t , i = − 3 ⟨ v t , ∇ S , θ L ( θ t , β t ) ⟩ . \frac{d}{dt}\Omega_t = 3\sum_i |\theta_{t,i}|\theta_{t,i}\dot{\theta}_{t,i} = -3\langle v_t,\nabla_{S,\theta}L(\theta_t,\beta_t)\rangle. d t d Ω t = 3 i ∑ ∣ θ t , i ∣ θ t , i θ ˙ t , i = − 3 ⟨ v t , ∇ S , θ L ( θ t , β t )⟩ .
Moreover,
∥ v t ∥ 1 = ∑ i ∣ θ t , i ∣ 2 = 1. \|v_t\|_1 = \sum_i |\theta_{t,i}|^2 = 1. ∥ v t ∥ 1 = i ∑ ∣ θ t , i ∣ 2 = 1.
Hence
Ω ˙ t ≤ 3 ∥ ∇ S , θ L ( θ t , β t ) ∥ ∞ . \dot{\Omega}_t \le3\|\nabla_{S,\theta}L(\theta_t,\beta_t)\|_\infty. Ω ˙ t ≤ 3∥ ∇ S , θ L ( θ t , β t ) ∥ ∞ .
By Lemma C.4 , more precisely by (48), we have
∥ ∇ S , θ L ( θ t , β t ) ∥ ∞ ≤ C 1 ∣ ρ t ∣ ∥ θ ∗ ∥ ∞ + C 2 ∣ ρ t χ t ∣ ∥ θ t ∥ ∞ + C 3 η t 2 . \|\nabla_{S,\theta}L(\theta_t,\beta_t)\|_\infty\le C_1|\rho_t|\|\theta^*\|_\infty+ C_2|\rho_t\chi_t|\|\theta_t\|_\infty+ C_3\eta_t^2. ∥ ∇ S , θ L ( θ t , β t ) ∥ ∞ ≤ C 1 ∣ ρ t ∣∥ θ ∗ ∥ ∞ + C 2 ∣ ρ t χ t ∣∥ θ t ∥ ∞ + C 3 η t 2 .
On the positive-overlap region, ρ t , χ t ≥ 0 \rho_t, \chi_t \ge0 ρ t , χ t ≥ 0 , and hence
∣ ρ t ∣ ≤ Z t , ∣ ρ t χ t ∣ ≤ Z t 2 , ∥ θ t ∥ ∞ ≤ η t . |\rho_t| \le Z_t,\qquad|\rho_t\chi_t| \le Z_t^2,\qquad\|\theta_t\|_\infty\le\eta_t. ∣ ρ t ∣ ≤ Z t , ∣ ρ t χ t ∣ ≤ Z t 2 , ∥ θ t ∥ ∞ ≤ η t .
Therefore
Ω ˙ t ≤ C ( Z t ∥ θ ∗ ∥ ∞ + Z t 2 η t + η t 2 ) , \dot{\Omega}_t \le C\left(Z_t\|\theta^*\|_\infty+ Z_t^2\eta_t + \eta_t^2\right), Ω ˙ t ≤ C ( Z t ∥ θ ∗ ∥ ∞ + Z t 2 η t + η t 2 ) ,
which proves (53).
Now with the conditions
Z t 2 ≤ A η t , η t ≤ C η log d + 1 + t d , 0 ≤ t ≤ T ≤ C T log d . Z_t^2 \le A\eta_t,\qquad\eta_t \le C_\eta\frac{\sqrt{\log d}+1+t}{\sqrt{d}},\qquad0 \le t \le T \le C_T\log d. Z t 2 ≤ A η t , η t ≤ C η d log d + 1 + t , 0 ≤ t ≤ T ≤ C T log d .
Then
Z t ∥ θ ∗ ∥ ∞ ≤ A η t C K log d d . Z_t\|\theta^*\|_\infty\le\sqrt{A\eta_t}\,C_K\sqrt{\frac{\log d}{d}}. Z t ∥ θ ∗ ∥ ∞ ≤ A η t C K d log d .
Using the envelope for η t \eta_t η t ,
Z t ∥ θ ∗ ∥ ∞ ≤ C 1 log d + 1 + t d log d d . Z_t\|\theta^*\|_\infty\le C_1\sqrt{\frac{\sqrt{\log d}+1+t}{\sqrt{d}}}\sqrt{\frac{\log d}{d}}. Z t ∥ θ ∗ ∥ ∞ ≤ C 1 d log d + 1 + t d log d .
Since t ≤ C T log d t \le C_T\log d t ≤ C T log d , the right-hand side is bounded by
C log d d 3 / 4 ≤ C d . C\frac{\log d}{d^{3/4}} \le\frac{C}{\sqrt{d}}. C d 3/4 log d ≤ d C .
for all sufficiently large d d d . Also,
Z t 2 η t ≤ A η t 2 . Z_t^{2}\eta_t \le A\eta_t^{2}. Z t 2 η t ≤ A η t 2 .
Therefore, throughout the Phase-I interval,
Ω ˙ t ≤ C d + C η t 2 . \dot{\Omega}_t \le\frac{C}{\sqrt{d}}+C\eta_t^{2}. Ω ˙ t ≤ d C + C η t 2 .
Integrating from 0 0 0 to t ≤ T t\le T t ≤ T , we get
Ω t ≤ Ω 0 + C t d + C ∫ 0 t η s 2 d s . \Omega_t\le\Omega_0+\frac{Ct}{\sqrt{d}}+C\int_0^t\eta_s^2\,\mathrm{d}s. Ω t ≤ Ω 0 + d C t + C ∫ 0 t η s 2 d s .
By Lemma C.1,
Ω 0 = ∥ θ 0 ∥ 3 3 ≤ C K d − 1 / 2 . \Omega_0=\|\theta_0\|_3^3\le C_Kd^{-1/2}. Ω 0 = ∥ θ 0 ∥ 3 3 ≤ C K d − 1/2 .
Moreover,
∫ 0 t η s 2 d s ≤ C d ∫ 0 t ( log d + 1 + s ) 2 d s . \int_0^t\eta_s^2\,\mathrm{d}s\le\frac{C}{d}\int_0^t(\sqrt{\log d}+1+s)^2\,\mathrm{d}s. ∫ 0 t η s 2 d s ≤ d C ∫ 0 t ( log d + 1 + s ) 2 d s .
For t ≤ C T log d t\le C_T\log d t ≤ C T log d ,
∫ 0 t ( log d + 1 + s ) 2 d s ≤ C ( log d ) 3 . \int_0^t(\sqrt{\log d}+1+s)^2\,\mathrm{d}s\le C(\log d)^3. ∫ 0 t ( log d + 1 + s ) 2 d s ≤ C ( log d ) 3 .
Thus
∫ 0 t η s 2 d s ≤ C ( log d ) 3 d = o ( d − 1 / 2 ) . \int_0^t\eta_s^2\,\mathrm{d}s\le\frac{C(\log d)^3}{d}=o(d^{-1/2}). ∫ 0 t η s 2 d s ≤ d C ( log d ) 3 = o ( d − 1/2 ) .
Hence
Ω t ≤ C Ω 0 1 + t d , 0 ≤ t ≤ T . \Omega_t\le C\Omega_0\frac{1+t}{\sqrt{d}},\qquad0\le t\le T. Ω t ≤ C Ω 0 d 1 + t , 0 ≤ t ≤ T .
Finally, by the elementary inequality
( a + b ) 3 ≤ 4 a 3 + 4 b 3 ( a , b ≥ 0 ) , (a+b)^3\le4a^3+4b^3\qquad(a,b\ge0), ( a + b ) 3 ≤ 4 a 3 + 4 b 3 ( a , b ≥ 0 ) ,
we have
ω t = ∑ i ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 ≤ 4 ∥ θ t ∥ 3 3 + 4 ∥ θ ∗ ∥ 3 3 . \omega_t=\sum_i\left(|\theta_{t,i}|+|\theta_i^*|\right)^3\le4\|\theta_t\|_3^3+4\|\theta^*\|_3^3. ω t = i ∑ ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 ≤ 4∥ θ t ∥ 3 3 + 4∥ θ ∗ ∥ 3 3 .
Using the bound on Ω t \Omega_t Ω t and the initialization event
∥ θ ∗ ∥ 3 3 ≤ C K d − 1 / 2 , \|\theta^*\|_3^3\le C_Kd^{-1/2}, ∥ θ ∗ ∥ 3 3 ≤ C K d − 1/2 ,
we obtain
ω t ≤ C ω 1 + t d . \omega_t\le C_\omega\frac{1+t}{\sqrt{d}}. ω t ≤ C ω d 1 + t .
This proves (54).
Lemma C.8 (Escape from mediocrity at population level). Assume Lemmas C.4, C.6, and C.7. Work on the random-spherical delocalization event of Lemma C.1, namely
∥ θ 0 ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ C K log d d , ∥ θ 0 ∥ 3 3 + ∥ θ ∗ ∥ 3 3 ≤ C K d − 1 / 2 , \|\theta_0\|_\infty+\|\theta^*\|_\infty\le C_K\sqrt{\frac{\log d}{d}},\qquad
\|\theta_0\|_3^3+\|\theta^*\|_3^3\le C_Kd^{-1/2}, ∥ θ 0 ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ C K d log d , ∥ θ 0 ∥ 3 3 + ∥ θ ∗ ∥ 3 3 ≤ C K d − 1/2 , and the favorable initialization event defined in (A4). Then there exist constants c e s c , C e s c > 0 c_{\mathrm{esc}},C_{\mathrm{esc}}>0 c esc , C esc > 0 , independent of d d d , such that for all sufficiently large d d d , there is a time
t e s c ≤ C e s c log d (57) t_{\mathrm{esc}}\le C_{\mathrm{esc}}\log d \tag*{(57)} t esc ≤ C esc log d (57) with
min { ρ t e s c , χ t e s c } ≥ c e s c . (58) \min\{\rho_{t_{\mathrm{esc}}},\chi_{t_{\mathrm{esc}}}\}\ge c_{\mathrm{esc}}. \tag*{(58)} min { ρ t esc , χ t esc } ≥ c esc . (58) Moreover, throughout the escape window,
η t ≤ C log d d = o d ( 1 ) . (59) \eta_t\le C\frac{\log d}{\sqrt{d}}=o_d(1). \tag*{(59)} η t ≤ C d log d = o d ( 1 ) . (59)
Proof. Fix a large constant C T > 0 C_T > 0 C T > 0 , to be chosen below, and set
T : = C T log d . T := C_T \log d. T := C T log d .
Let A > 0 A > 0 A > 0 be a fixed large constant and define the stopping times
τ A : = inf { t : Z t 2 ≥ A η t } , \tau_A := \inf\{t : Z_t^2 \ge A\eta_t\}, τ A := inf { t : Z t 2 ≥ A η t } ,
τ B : = inf { t : ρ t ≥ 1 / 2 or χ t ≥ 1 / 2 } , \tau_B := \inf\{t : \rho_t \ge1/2 \text{ or } \chi_t \ge1/2\}, τ B := inf { t : ρ t ≥ 1/2 or χ t ≥ 1/2 } ,
τ + : = inf { t : ρ t < 0 or χ t < 0 } . \tau_+ := \inf\{t : \rho_t < 0 \text{ or } \chi_t < 0\}. τ + := inf { t : ρ t < 0 or χ t < 0 } .
We first work on Phase I, defined as when t ∈ [ 0 , σ ] t \in[0,\sigma] t ∈ [ 0 , σ ] where σ : = τ A ∧ τ B ∧ τ + ∧ T \sigma:= \tau_A \wedge\tau_B \wedge\tau_+ \wedge T σ := τ A ∧ τ B ∧ τ + ∧ T .
Step 1: Incoherence Control in Phase I We begin with the Phase-I control of η t \eta_t η t . By Lemma C.4, on the positive-overlap region,
η ˙ t ≤ C Z t ∥ θ ∗ ∥ ∞ + C Z t 2 η t + C η t 2 . \dot{\eta}_t \le CZ_t\|\theta^*\|_\infty+ CZ_t^2\eta_t + C\eta_t^2. η ˙ t ≤ C Z t ∥ θ ∗ ∥ ∞ + C Z t 2 η t + C η t 2 .
For t < τ A t < \tau_A t < τ A , we have Z t 2 < A η t Z_t^2 < A\eta_t Z t 2 < A η t . Therefore
Z t ∥ θ ∗ ∥ ∞ ≤ A η t ∥ θ ∗ ∥ ∞ . Z_t\|\theta^*\|_\infty\le\sqrt{A\eta_t}\|\theta^*\|_\infty. Z t ∥ θ ∗ ∥ ∞ ≤ A η t ∥ θ ∗ ∥ ∞ .
Define
H I ( t ) : = K I log d + 1 + t d , H_I(t) := K_I\frac{\sqrt{\log d}+1+t}{\sqrt{d}}, H I ( t ) := K I d log d + 1 + t ,
where K I K_I K I is a large constant. We claim that
η t ≤ H I ( t ) ( 0 ≤ t ≤ σ ) . \eta_t \le H_I(t) \qquad(0 \le t \le\sigma). η t ≤ H I ( t ) ( 0 ≤ t ≤ σ ) .
Indeed, at initialization,
η 0 = ∥ θ 0 ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ C K log d d ≤ 1 2 H I ( 0 ) \eta_0 = \|\theta_0\|_\infty+ \|\theta^*\|_\infty\le C_K\sqrt{\frac{\log d}{d}} \le\frac{1}{2}H_I(0) η 0 = ∥ θ 0 ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ C K d log d ≤ 2 1 H I ( 0 )
after choosing K I K_I K I large. Suppose the estimate holds up to a first violation time. Since t ≤ T = C T log d t \le T = C_T\log d t ≤ T = C T log d , the bootstrap bound gives
η t ≤ H I ( t ) ≤ C log d d . \eta_t \le H_I(t) \le C\frac{\log d}{\sqrt{d}}. η t ≤ H I ( t ) ≤ C d log d .
Thus
Z t ∥ θ ∗ ∥ ∞ ≤ A H I ( t ) C K log d d ≤ C log d d 3 / 4 ≤ C d Z_t\|\theta^*\|_\infty\le\sqrt{AH_I(t)}C_K\sqrt{\frac{\log d}{d}} \le\frac{C\log d}{d^{3/4}} \le\frac{C}{\sqrt{d}} Z t ∥ θ ∗ ∥ ∞ ≤ A H I ( t ) C K d log d ≤ d 3/4 C log d ≤ d C
for all sufficiently large d d d . Also,
Z t 2 η t ≤ A η t 2 . Z_t^2\eta_t \le A\eta_t^2. Z t 2 η t ≤ A η t 2 .
Consequently, up to the first violation time,
η ˙ t ≤ C d + C η t 2 . \dot{\eta}_t \le\frac{C}{\sqrt{d}} + C\eta_t^2. η ˙ t ≤ d C + C η t 2 .
Integrating,
η t ≤ η 0 + C t d + C ∫ 0 t η s 2 d s . \eta_t \le\eta_0 + \frac{Ct}{\sqrt{d}} + C\int_0^t\eta_s^2\,ds. η t ≤ η 0 + d C t + C ∫ 0 t η s 2 d s .
Under the bootstrap η s ≤ H I ( s ) \eta_s \le H_I(s) η s ≤ H I ( s ) ,
∫ 0 t η s 2 d s ≤ C d ∫ 0 C T log d ( log d + 1 + s ) 2 d s ≤ C ( log d ) 3 d = o ( log d + 1 + t d ) . \int_0^t\eta_s^2\,ds \le\frac{C}{d}\int_0^{C_T\log d}(\sqrt{\log d}+1+s)^2\,ds \le\frac{C(\log d)^3}{d} = o\left(\frac{\sqrt{\log d}+1+t}{\sqrt{d}}\right). ∫ 0 t η s 2 d s ≤ d C ∫ 0 C T l o g d ( log d + 1 + s ) 2 d s ≤ d C ( log d ) 3 = o ( d log d + 1 + t ) .
Choosing K I K_I K I large gives
η t ≤ 1 2 H I ( t ) \eta_t \le\frac{1}{2}H_I(t) η t ≤ 2 1 H I ( t )
at the putative first violation, a contradiction. Hence for 0 ≤ t ≤ σ 0 \le t \le\sigma 0 ≤ t ≤ σ ,
η t ≤ H I ( t ) . \eta_t \le H_I(t). η t ≤ H I ( t ) .
So by Lemma C.7, we have for 0 ≤ t ≤ σ 0 \le t \le\sigma 0 ≤ t ≤ σ ,
Ω t : = ∥ θ t ∥ 3 3 ≤ C Ω 1 + t d , \Omega_t := \|\theta_t\|_3^3 \le C_\Omega\frac{1+t}{\sqrt{d}}, Ω t := ∥ θ t ∥ 3 3 ≤ C Ω d 1 + t ,
and
ω t ≤ C ω 1 + t d . \omega_t \le C_\omega\frac{1+t}{\sqrt{d}}. ω t ≤ C ω d 1 + t .
Step 2: Phase I Positivity, Exponential Pull-Up, and Balancing. We next prove positivity and exponential pull-up in Phase I. On t < σ t < \sigma t < σ , we have
0 ≤ ρ t , χ t ≤ 1 / 2. 0 \le\rho_t,\chi_t \le1/2. 0 ≤ ρ t , χ t ≤ 1/2.
By Lemma C.6, since 0 ≤ ρ t , χ t ≤ 1 / 2 0 \le\rho_t,\chi_t \le1/2 0 ≤ ρ t , χ t ≤ 1/2 on the stopped Phase-I interval and d d d is large, after decreasing a > 0 a > 0 a > 0 and increasing b > 0 b > 0 b > 0 ,
ρ ˙ t ≥ a χ t − b ω t , χ ˙ t ≥ a ρ t − b ω t . \dot{\rho}_t \ge a\chi_t-b\omega_t,\qquad\dot{\chi}_t \ge a\rho_t-b\omega_t. ρ ˙ t ≥ a χ t − b ω t , χ ˙ t ≥ a ρ t − b ω t .
Let
u t : = ( ρ t , χ t ) ⊤ , J : = ( 0 1 1 0 ) , 1 : = ( 1 , 1 ) ⊤ . u_t := (\rho_t,\chi_t)^\top,\qquad J := \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix},\qquad\mathbf{1} := (1,1)^\top. u t := ( ρ t , χ t ) ⊤ , J := ( 0 1 1 0 ) , 1 := ( 1 , 1 ) ⊤ .
Then, componentwise,
u ˙ t ≥ a J u t − b ω t 1. \dot{u}_t \ge aJu_t-b\omega_t\mathbf{1}. u ˙ t ≥ a J u t − b ω t 1 .
Since e a J r e^{aJr} e a J r has nonnegative entries and
e a J r 1 = e a r 1 , e^{aJr}\mathbf{1}=e^{ar}\mathbf{1}, e a J r 1 = e a r 1 ,
variation of constants gives, for 0 ≤ t ≤ σ 0 \le t \le\sigma 0 ≤ t ≤ σ ,
u t ≥ e a J t u 0 − b ∫ 0 t e a J ( t − s ) ω s 1 d s . u_t \ge e^{aJt}u_0-b\int_0^t e^{aJ(t-s)}\omega_s\mathbf{1}\,ds. u t ≥ e a J t u 0 − b ∫ 0 t e a J ( t − s ) ω s 1 d s .
Using ρ 0 , χ 0 ≥ z i n i t d − 1 / 2 \rho_0,\chi_0 \ge z_{\mathrm{init}}d^{-1/2} ρ 0 , χ 0 ≥ z init d − 1/2 , from (A4), we get componentwise
e a J t u 0 ≥ e a t z i n i t d 1. e^{aJt}u_0 \ge e^{at}\frac{z_{\mathrm{init}}}{\sqrt{d}}\mathbf{1}. e a J t u 0 ≥ e a t d z init 1 .
Also,
b ∫ 0 t e a J ( t − s ) ω s 1 d s = b e a t ( ∫ 0 t e − a s ω s d s ) 1. b\int_0^t e^{aJ(t-s)}\omega_s\mathbf{1}\,ds=be^{at}\left(\int_0^t e^{-as}\omega_s\,ds\right)\mathbf{1}. b ∫ 0 t e a J ( t − s ) ω s 1 d s = b e a t ( ∫ 0 t e − a s ω s d s ) 1 .
Since
ω s ≤ C ω 1 + s d , \omega_s \le C_\omega\frac{1+s}{\sqrt{d}}, ω s ≤ C ω d 1 + s ,
we have
∫ 0 ∞ e − a s ω s d s ≤ C d . \int_0^\infty e^{-as}\omega_s\,ds \le\frac{C}{\sqrt{d}}. ∫ 0 ∞ e − a s ω s d s ≤ d C .
Let
M ω : = b C ω ∫ 0 ∞ e − a s ( 1 + s ) d s . M_\omega:= bC_\omega\int_0^\infty e^{-as}(1+s)\,ds. M ω := b C ω ∫ 0 ∞ e − a s ( 1 + s ) d s .
By choosing z i n i t > 2 M ω z_{\mathrm{init}}>2M_\omega z init > 2 M ω , we obtain
ρ t , χ t ≥ e a t z i n i t − M ω d ≥ c − d e a t ( 0 ≤ t ≤ σ ) , \rho_t,\chi_t\ge e^{at}\frac{z_{\mathrm{init}}-M_\omega}{\sqrt{d}}\ge\frac{c_-}{\sqrt{d}}e^{at}\qquad(0\le t\le\sigma), ρ t , χ t ≥ e a t d z init − M ω ≥ d c − e a t ( 0 ≤ t ≤ σ ) ,
for some c − > 0 c_->0 c − > 0 . In particular,
Z t ≥ 2 c − d e a t ( 0 ≤ t ≤ σ ) , Z_t\ge\frac{2c_-}{\sqrt{d}}e^{at}\qquad(0\le t\le\sigma), Z t ≥ d 2 c − e a t ( 0 ≤ t ≤ σ ) ,
and τ + \tau_+ τ + cannot be the first stopping time.
Choose
t 0 : = C 0 log d t_0:=C_0\log d t 0 := C 0 log d
with C 0 > 0 C_0>0 C 0 > 0 sufficiently large, and then choose C T > C 0 C_T>C_0 C T > C 0 . Here we show that
σ ≤ t 0 . (60) \sigma\le t_0. \tag*{(60)} σ ≤ t 0 . (60)
Suppose instead that σ > t 0 \sigma>t_0 σ > t 0 . Then t 0 < τ A t_0<\tau_A t 0 < τ A , so
Z t 0 2 < A η t 0 . Z_{t_0}^2<A\eta_{t_0}. Z t 0 2 < A η t 0 .
But the lower bound above gives
Z t 0 2 ≥ c d e 2 a t 0 = c d 2 a C 0 − 1 , Z_{t_0}^2 \ge\frac{c}{d} e^{2at_0} = c d^{2aC_0-1}, Z t 0 2 ≥ d c e 2 a t 0 = c d 2 a C 0 − 1 ,
whereas the Phase-I η \eta η -bound gives
A η t 0 ≤ C log d + 1 + t 0 d ≤ C log d d . A\eta_{t_0} \le C\frac{\sqrt{\log d}+1+t_0}{\sqrt{d}} \le C\frac{\log d}{\sqrt{d}}. A η t 0 ≤ C d log d + 1 + t 0 ≤ C d log d .
For C 0 C_0 C 0 sufficiently large, this is impossible for large d d d . Hence (58) holds.
Since t 0 < T t_0<T t 0 < T and τ + \tau_+ τ + cannot occur first, the first stopping time is either τ A \tau_A τ A or τ B \tau_B τ B . We claim it must be τ A \tau_A τ A . If σ = τ B ≤ τ A \sigma=\tau_B\le\tau_A σ = τ B ≤ τ A , then by continuity and the definition of τ B \tau_B τ B ,
Z τ B ≥ 1 2 . Z_{\tau_B} \ge\frac{1}{2}. Z τ B ≥ 2 1 .
On the other hand, since τ B ≤ τ A \tau_B\le\tau_A τ B ≤ τ A ,
Z τ B 2 ≤ A η τ B ≤ C log d d , Z_{\tau_B}^2 \le A\eta_{\tau_B} \le C\frac{\log d}{\sqrt{d}}, Z τ B 2 ≤ A η τ B ≤ C d log d ,
and hence
Z τ B ≤ C log d d − 1 / 4 = o ( 1 ) , Z_{\tau_B} \le C\sqrt{\log d}\,d^{-1/4}=o(1), Z τ B ≤ C log d d − 1/4 = o ( 1 ) ,
a contradiction. Therefore
σ = τ A , τ A ≤ C 0 log d , τ A < τ B . \sigma=\tau_A,\qquad\tau_A\le C_0\log d,\qquad\tau_A<\tau_B. σ = τ A , τ A ≤ C 0 log d , τ A < τ B .
Since Z 0 ≤ C i n i t d − 1 / 2 Z_0\le C_{\mathrm{init}}d^{-1/2} Z 0 ≤ C init d − 1/2 by assumption and given the lower bound η 0 ≥ ∥ θ ∗ ∥ ∞ ≥ d − 1 / 2 \eta_0\ge\lVert\theta^*\rVert_\infty\ge d^{-1/2} η 0 ≥ ∥ θ ∗ ∥ ∞ ≥ d − 1/2 , we have Z 0 2 < A η 0 Z_0^2<A\eta_0 Z 0 2 < A η 0 for all large d d d . Thus, by continuity,
Z τ A 2 = A η τ A . Z_{\tau_A}^2=A\eta_{\tau_A}. Z τ A 2 = A η τ A .
Moreover,
η τ A ≥ ∥ θ ∗ ∥ ∞ ≥ d − 1 / 2 , \eta_{\tau_A}\ge\lVert\theta^*\rVert_\infty\ge d^{-1/2}, η τ A ≥ ∥ θ ∗ ∥ ∞ ≥ d − 1/2 ,
and from the Phase-I bound,
η τ A ≤ C log d d . \eta_{\tau_A}\le C\frac{\log d}{\sqrt{d}}. η τ A ≤ C d log d .
Therefore
Z τ A ≥ c A d − 1 / 4 , Z τ A ≤ C A log d d − 1 / 4 = o ( 1 ) , Z_{\tau_A}\ge c_A d^{-1/4},\qquad Z_{\tau_A}\le C_A\sqrt{\log d}\,d^{-1/4}=o(1), Z τ A ≥ c A d − 1/4 , Z τ A ≤ C A log d d − 1/4 = o ( 1 ) ,
and
η τ A ≤ C log d d . \eta_{\tau_A}\le C\frac{\log d}{\sqrt{d}}. η τ A ≤ C d log d .
We also need the two overlaps to be balanced at τ A \tau_A τ A . On [ 0 , τ A ] [0,\tau_A] [ 0 , τ A ] , by the D D D -drift consequence (52) of Lemma C.6, we have
D ˙ t = − C ~ ( 1 + ρ t χ t ) D t + O ( Z t 2 + ω t ) . (61) \dot{D}_t=-\widetilde{C}(1+\rho_t\chi_t)D_t+O(Z_t^2+\omega_t). \tag*{(61)} D ˙ t = − C ( 1 + ρ t χ t ) D t + O ( Z t 2 + ω t ) . (61)
Integrating (59) on [ 0 , τ A ] [0,\tau_A] [ 0 , τ A ] , and using Z t 2 ≤ A η t Z_t^2\le A\eta_t Z t 2 ≤ A η t , gives
∣ D τ A ∣ ≤ ∣ D 0 ∣ + C ∫ 0 τ A ( Z s 2 + ω s ) d s . \lvert D_{\tau_A}\rvert\le\lvert D_0\rvert+C\int_0^{\tau_A}(Z_s^2+\omega_s)\,ds. ∣ D τ A ∣ ≤ ∣ D 0 ∣ + C ∫ 0 τ A ( Z s 2 + ω s ) d s .
Using
∣ D 0 ∣ ≤ C i n i t d − 1 / 2 , η s ≤ C log d + 1 + s d , ω s ≤ C 1 + s d , \lvert D_0\rvert\le C_{\mathrm{init}}d^{-1/2},\qquad
\eta_s\le C\frac{\sqrt{\log d}+1+s}{\sqrt{d}},\qquad
\omega_s\le C\frac{1+s}{\sqrt{d}}, ∣ D 0 ∣ ≤ C init d − 1/2 , η s ≤ C d log d + 1 + s , ω s ≤ C d 1 + s ,
and τ A ≤ C 0 log d \tau_A\le C_0\log d τ A ≤ C 0 log d , we obtain
∣ D τ A ∣ ≤ C d + C ∫ 0 C 0 log d ( log d + 1 + s d + 1 + s d ) d s ≤ C ( log d ) 2 d . \lvert D_{\tau_A}\rvert\le\frac{C}{\sqrt{d}}+
C\int_0^{C_0\log d}\left(\frac{\sqrt{\log d}+1+s}{\sqrt{d}}+\frac{1+s}{\sqrt{d}}\right)\,ds
\le C\frac{(\log d)^2}{\sqrt{d}}. ∣ D τ A ∣ ≤ d C + C ∫ 0 C 0 l o g d ( d log d + 1 + s + d 1 + s ) d s ≤ C d ( log d ) 2 .
Since Z τ A ≥ c A d − 1 / 4 Z_{\tau_A}\ge c_A d^{-1/4} Z τ A ≥ c A d − 1/4 , we have
∣ D τ A ∣ = o ( Z τ A ) . \lvert D_{\tau_A}\rvert=o(Z_{\tau_A}). ∣ D τ A ∣ = o ( Z τ A ) .
In particular, for all sufficiently large d d d ,
∣ D τ A ∣ ≤ 1 8 Z τ A . \lvert D_{\tau_A}\rvert\le\frac{1}{8}Z_{\tau_A}. ∣ D τ A ∣ ≤ 8 1 Z τ A .
Step 3: Phase III Incoherence Control and Drift Pull-Up Exiting Phase I, we next show that Z t Z_t Z t will be pulled from the d − 1 / 4 d^{-1/4} d − 1/4 -scale to a fixed constant. We define Phase II as when t ∈ [ τ A , t ∗ ] t \in[\tau_A,t^*] t ∈ [ τ A , t ∗ ] , where t ∗ : = inf { t ≥ τ A : Z t ≥ c Z } t^*:=\inf\{t\ge\tau_A: Z_t\ge c_Z\} t ∗ := inf { t ≥ τ A : Z t ≥ c Z } . Choose a small constant c Z > 0 c_Z>0 c Z > 0 , to be fixed below, and define
H : = K 2 log d d , T 2 : = τ A + C 2 log d , H:=K_2\frac{\log d}{\sqrt{d}},\qquad T_2:=\tau_A+C_2\log d, H := K 2 d log d , T 2 := τ A + C 2 log d ,
where K 2 , C 2 K_2,C_2 K 2 , C 2 are large constants. Define the bootstrap stopping time
σ 2 : = inf { t ≥ τ A : η t > H or Z t > 2 c Z or ∣ D t ∣ > 1 2 Z t or t = T 2 } . \sigma_2:=\inf\left\{t\ge\tau_A:\eta_t>H\text{ or }Z_t>2c_Z\text{ or }|D_t|>\frac{1}{2}Z_t\text{ or }t=T_2\right\}. σ 2 := inf { t ≥ τ A : η t > H or Z t > 2 c Z or ∣ D t ∣ > 2 1 Z t or t = T 2 } .
At t = τ A t=\tau_A t = τ A , none of the first three clauses holds for large d d d . Hence σ 2 > τ A \sigma_2>\tau_A σ 2 > τ A . For t < σ 2 t<\sigma_2 t < σ 2 , we have
∣ D t ∣ ≤ 1 2 Z t , |D_t|\le\frac{1}{2}Z_t, ∣ D t ∣ ≤ 2 1 Z t ,
and therefore
1 4 Z t ≤ ρ t , χ t ≤ 3 4 Z t . \frac{1}{4}Z_t\le\rho_t,\chi_t\le\frac{3}{4}Z_t. 4 1 Z t ≤ ρ t , χ t ≤ 4 3 Z t .
Also Z t ≤ 2 c Z Z_t\le2c_Z Z t ≤ 2 c Z .
We first obtain a crude bound on ω t \omega_t ω t on the Phase-II bootstrap interval. From the third-moment differential inequality,
Ω ˙ t ≤ C ( Z t ∥ θ ∗ ∥ ∞ + Z t 2 η t + η t 2 ) . \dot{\Omega}_t\le C\left(Z_t\|\theta^*\|_\infty+Z_t^2\eta_t+\eta_t^2\right). Ω ˙ t ≤ C ( Z t ∥ θ ∗ ∥ ∞ + Z t 2 η t + η t 2 ) .
For t < σ 2 t<\sigma_2 t < σ 2 , we have Z t ≤ 2 c Z Z_t\le2c_Z Z t ≤ 2 c Z , η t ≤ H \eta_t\le H η t ≤ H , and t − τ A ≤ C 2 log d t-\tau_A\le C_2\log d t − τ A ≤ C 2 log d . Thus
Ω t ≤ Ω τ A + C ∥ θ ∗ ∥ ∞ ∫ τ A t Z s d s + C ∫ τ A t Z s 2 η s d s + C ∫ τ A t η s 2 d s . \Omega_t\le\Omega_{\tau_A}+C\|\theta^*\|_\infty\int_{\tau_A}^{t}Z_s\,\mathrm{d}s+C\int_{\tau_A}^{t}Z_s^2\eta_s\,\mathrm{d}s+C\int_{\tau_A}^{t}\eta_s^2\,\mathrm{d}s. Ω t ≤ Ω τ A + C ∥ θ ∗ ∥ ∞ ∫ τ A t Z s d s + C ∫ τ A t Z s 2 η s d s + C ∫ τ A t η s 2 d s .
Using
Ω τ A ≤ C 1 + τ A d ≤ C log d d , \Omega_{\tau_A}\le C\frac{1+\tau_A}{\sqrt{d}}\le C\frac{\log d}{\sqrt{d}}, Ω τ A ≤ C d 1 + τ A ≤ C d log d ,
∥ θ ∗ ∥ ∞ ≤ C K log d d , \|\theta^*\|_\infty\le CK\sqrt{\frac{\log d}{d}}, ∥ θ ∗ ∥ ∞ ≤ C K d log d ,
and the crude bounds Z s ≤ 2 c Z Z_s\le2c_Z Z s ≤ 2 c Z , η s ≤ H \eta_s\le H η s ≤ H , we get
Ω t ≤ C log d d + C c Z ( log d ) 3 / 2 d + C c Z 2 ( log d ) 2 d + C ( log d ) 3 d . \Omega_t\le C\frac{\log d}{\sqrt{d}}+Cc_Z\frac{(\log d)^{3/2}}{\sqrt{d}}+Cc_Z^2\frac{(\log d)^2}{\sqrt{d}}+C\frac{(\log d)^3}{d}. Ω t ≤ C d log d + C c Z d ( log d ) 3/2 + C c Z 2 d ( log d ) 2 + C d ( log d ) 3 .
After increasing the constant, this yields
Ω t ≤ C ( log d ) 2 d ( t < σ 2 ) . \Omega_t\le C\frac{(\log d)^2}{\sqrt{d}}\qquad(t<\sigma_2). Ω t ≤ C d ( log d ) 2 ( t < σ 2 ) .
Since ω t ≤ 4 Ω t + 4 ∥ θ ∗ ∥ 3 3 \omega_t\le4\Omega_t+4\|\theta^*\|_3^3 ω t ≤ 4 Ω t + 4∥ θ ∗ ∥ 3 3 and ∥ θ ∗ ∥ 3 3 ≤ C K d − 1 / 2 \|\theta^*\|_3^3\le CKd^{-1/2} ∥ θ ∗ ∥ 3 3 ≤ C K d − 1/2 by Lemma C.1, we obtain
ω t ≤ W d : = C ( log d ) 2 d ( t < σ 2 ) . \omega_t\le W_d:=C\frac{(\log d)^2}{\sqrt{d}}\qquad(t<\sigma_2). ω t ≤ W d := C d ( log d ) 2 ( t < σ 2 ) .
We next prove that Z t Z_t Z t grows exponentially on the Phase-II bootstrap interval. Let
τ − : = inf { t ≥ τ A : Z t < Z τ A / 2 } . \tau_-:=\inf\{t\ge\tau_A: Z_t<Z_{\tau_A}/2\}. τ − := inf { t ≥ τ A : Z t < Z τ A /2 } .
On [ τ A , σ 2 ∧ τ − ) [\tau_A,\sigma_2\wedge\tau_-) [ τ A , σ 2 ∧ τ − ) , we have Z t ≥ Z τ A / 2 Z_t\ge Z_{\tau_A}/2 Z t ≥ Z τ A /2 , the balance bounds imply ρ t , χ t ≥ Z t / 4 \rho_t,\chi_t\ge Z_t/4 ρ t , χ t ≥ Z t /4 , and, by choosing c Z c_Z c Z small, ρ t , χ t ≤ 1 / 2 \rho_t,\chi_t\le1/2 ρ t , χ t ≤ 1/2 . Hence the refined drift gives
Z ˙ t = ρ ˙ t + χ ˙ t ≥ a Z Z t − b Z ω t ≥ a Z Z t − b Z W d . \dot{Z}_t=\dot{\rho}_t+\dot{\chi}_t\ge a_ZZ_t-b_Z\omega_t\ge a_ZZ_t-b_ZW_d. Z ˙ t = ρ ˙ t + χ ˙ t ≥ a Z Z t − b Z ω t ≥ a Z Z t − b Z W d .
Since
W d Z τ A ≤ C ( log d ) 2 / d d − 1 / 4 = C ( log d ) 2 d 1 / 4 = o ( 1 ) , \frac{W_d}{Z_{\tau_A}}\le C\frac{(\log d)^2/\sqrt{d}}{d^{-1/4}}=C\frac{(\log d)^2}{d^{1/4}}=o(1), Z τ A W d ≤ C d − 1/4 ( log d ) 2 / d = C d 1/4 ( log d ) 2 = o ( 1 ) ,
we have, for all sufficiently large d d d ,
b Z W d ≤ 1 2 a Z Z t on [ τ A , σ 2 ∧ τ − ) . b_ZW_d \le\frac{1}{2}a_ZZ_t \qquad\text{on }[\tau_A,\sigma_2 \wedge\tau_-). b Z W d ≤ 2 1 a Z Z t on [ τ A , σ 2 ∧ τ − ) .
Thus
Z ˙ t ≥ μ Z t ( τ A ≤ t < σ 2 ∧ τ − ) \dot{Z}_t \ge\mu Z_t \qquad(\tau_A \le t < \sigma_2 \wedge\tau_-) Z ˙ t ≥ μ Z t ( τ A ≤ t < σ 2 ∧ τ − )
for some μ > 0 \mu> 0 μ > 0 . This implies τ − > σ 2 \tau_- > \sigma_2 τ − > σ 2 , and hence
Z t ≥ Z τ A e μ ( t − τ A ) ( τ A ≤ t < σ 2 ) . Z_t \ge Z_{\tau_A}e^{\mu(t-\tau_A)} \qquad(\tau_A \le t < \sigma_2). Z t ≥ Z τ A e μ ( t − τ A ) ( τ A ≤ t < σ 2 ) .
In particular, for s ≤ t < σ 2 s \le t < \sigma_2 s ≤ t < σ 2 ,
Z s ≤ e − μ ( t − s ) Z t . Z_s \le e^{-\mu(t-s)}Z_t. Z s ≤ e − μ ( t − s ) Z t .
We now close the η \eta η -bootstrap before t ∗ t^* t ∗ . For τ A ≤ t < σ 2 ∧ t ∗ \tau_A \le t < \sigma_2 \wedge t^* τ A ≤ t < σ 2 ∧ t ∗ , Lemma C.4 gives
η t ≤ η τ A + C ∥ θ ∗ ∥ ∞ ∫ τ A t Z s d s + C ∫ τ A t Z s 2 η s d s + C ∫ τ A t η s 2 d s . \eta_t \le\eta_{\tau_A} + C\lVert\theta^*\rVert_\infty\int_{\tau_A}^{t} Z_s\,ds + C\int_{\tau_A}^{t} Z_s^2\eta_s\,ds + C\int_{\tau_A}^{t}\eta_s^2\,ds. η t ≤ η τ A + C ∥ θ ∗ ∥ ∞ ∫ τ A t Z s d s + C ∫ τ A t Z s 2 η s d s + C ∫ τ A t η s 2 d s .
Since t < t ∗ t < t^* t < t ∗ , we have Z t < c Z Z_t < c_Z Z t < c Z . Using the exponential comparison Z s ≤ e − μ ( t − s ) Z t Z_s \le e^{-\mu(t-s)}Z_t Z s ≤ e − μ ( t − s ) Z t , we obtain
∫ τ A t Z s d s ≤ C c Z , ∫ τ A t Z s 2 d s ≤ C c Z 2 . \int_{\tau_A}^{t} Z_s\,ds \le Cc_Z,\qquad\int_{\tau_A}^{t} Z_s^2\,ds \le Cc_Z^2. ∫ τ A t Z s d s ≤ C c Z , ∫ τ A t Z s 2 d s ≤ C c Z 2 .
Therefore
η t ≤ C log d d + C c Z log d d + C c Z 2 H + C H 2 log d . \eta_t \le C\frac{\log d}{\sqrt d}+Cc_Z\sqrt{\frac{\log d}{d}}+Cc_Z^2H+CH^2\log d. η t ≤ C d log d + C c Z d log d + C c Z 2 H + C H 2 log d .
Choose c Z > 0 c_Z > 0 c Z > 0 small enough that
C c Z 2 H ≤ 1 8 H . Cc_Z^2H \le\frac{1}{8}H. C c Z 2 H ≤ 8 1 H .
Then choose K 2 K_2 K 2 large enough so that
C log d d ≤ 1 4 H . C\frac{\log d}{\sqrt d} \le\frac{1}{4}H. C d log d ≤ 4 1 H .
Moreover,
C c Z log d d = o ( H ) , H 2 log d = K 2 2 ( log d ) 3 d = o ( H ) . Cc_Z\sqrt{\frac{\log d}{d}}=o(H),\qquad H^2\log d=K_2^2\frac{(\log d)^3}{d}=o(H). C c Z d log d = o ( H ) , H 2 log d = K 2 2 d ( log d ) 3 = o ( H ) .
Thus, for all sufficiently large d d d ,
η t ≤ 1 2 H ( τ A ≤ t < σ 2 ∧ t ∗ ) . \eta_t \le\frac{1}{2}H \qquad(\tau_A \le t < \sigma_2 \wedge t^*). η t ≤ 2 1 H ( τ A ≤ t < σ 2 ∧ t ∗ ) .
By a first-violation argument, the clause η t > H \eta_t > H η t > H cannot trigger before t ∗ t^* t ∗ .
Next we close the D / Z D/Z D / Z -balance bootstrap. On the balanced small-overlap band, the same subtraction estimate as above gives
D ˙ t = − C ~ ( 1 + ρ t χ t ) D t + O ( Z t 2 + ω t ) . \dot{D}_t=-\widetilde{C}(1+\rho_t\chi_t)D_t+O(Z_t^2+\omega_t). D ˙ t = − C ( 1 + ρ t χ t ) D t + O ( Z t 2 + ω t ) .
Since c Z c_Z c Z is small, 1 + ρ t χ t ≥ 1 / 2 1+\rho_t\chi_t \ge1/2 1 + ρ t χ t ≥ 1/2 . Hence, for τ A ≤ t < σ 2 ∧ t ∗ \tau_A \le t < \sigma_2 \wedge t^* τ A ≤ t < σ 2 ∧ t ∗ ,
∣ D t ∣ ≤ e − c ( t − τ A ) ∣ D τ A ∣ + C ∫ τ A t e − c ( t − s ) ( Z s 2 + ω s ) d s . |D_t| \le e^{-c(t-\tau_A)}|D_{\tau_A}|+C\int_{\tau_A}^{t}e^{-c(t-s)}(Z_s^2+\omega_s)\,ds. ∣ D t ∣ ≤ e − c ( t − τ A ) ∣ D τ A ∣ + C ∫ τ A t e − c ( t − s ) ( Z s 2 + ω s ) d s .
The entry term is bounded by
e − c ( t − τ A ) ∣ D τ A ∣ ≤ ∣ D τ A ∣ ≤ 1 8 Z τ A ≤ 1 8 Z t . e^{-c(t-\tau_A)}|D_{\tau_A}| \le|D_{\tau_A}| \le\frac{1}{8}Z_{\tau_A} \le\frac{1}{8}Z_t. e − c ( t − τ A ) ∣ D τ A ∣ ≤ ∣ D τ A ∣ ≤ 8 1 Z τ A ≤ 8 1 Z t .
The ω \omega ω -term is bounded by
C W d = o ( Z τ A ) ≤ o ( Z t ) . CW_d=o(Z_{\tau_A})\le o(Z_t). C W d = o ( Z τ A ) ≤ o ( Z t ) .
For the Z s 2 Z_s^2 Z s 2 -term, using
Z s ≤ e − μ ( t − s ) Z t , Z_s \le e^{-\mu(t-s)}Z_t, Z s ≤ e − μ ( t − s ) Z t ,
we get
∫ τ A t e − c ( t − s ) Z s 2 d s ≤ C Z t 2 ≤ c Z Z t . \int_{\tau_A}^{t} e^{-c(t-s)}Z_s^2\,\mathrm{d}s \le CZ_t^2 \le c_ZZ_t. ∫ τ A t e − c ( t − s ) Z s 2 d s ≤ C Z t 2 ≤ c Z Z t .
Choosing c Z c_Z c Z smaller if necessary, and then d d d large, gives
∣ D t ∣ ≤ 3 8 Z t < 1 2 Z t ( τ A ≤ t < σ 2 ∧ t ∗ ) . |D_t| \le\frac{3}{8}Z_t < \frac{1}{2}Z_t \qquad(\tau_A \le t < \sigma_2 \wedge t^*). ∣ D t ∣ ≤ 8 3 Z t < 2 1 Z t ( τ A ≤ t < σ 2 ∧ t ∗ ) .
Thus the clause ∣ D t ∣ > Z t / 2 |D_t| > Z_t/2 ∣ D t ∣ > Z t /2 cannot trigger before t ∗ t^* t ∗ .
We now show that t ∗ ≤ T 2 t^* \le T_2 t ∗ ≤ T 2 . Suppose, for contradiction, that t ∗ > T 2 t^* > T_2 t ∗ > T 2 . Then Z t < c Z Z_t < c_Z Z t < c Z for all t ≤ T 2 t \le T_2 t ≤ T 2 . The clauses η t > H \eta_t > H η t > H and ∣ D t ∣ > Z t / 2 |D_t| > Z_t/2 ∣ D t ∣ > Z t /2 cannot trigger before t ∗ t^* t ∗ , as just shown, and the clause Z t > 2 c Z Z_t > 2c_Z Z t > 2 c Z cannot trigger while Z t < c Z Z_t < c_Z Z t < c Z . Hence
T 2 < σ 2 . T_2 < \sigma_2. T 2 < σ 2 .
The exponential lower bound then gives
Z T 2 ≥ Z τ A e μ C 2 log d ≥ c A d − 1 / 4 d μ C 2 . Z_{T_2} \ge Z_{\tau_A}e^{\mu C_2\log d} \ge c_A d^{-1/4}d^{\mu C_2}. Z T 2 ≥ Z τ A e μ C 2 l o g d ≥ c A d − 1/4 d μ C 2 .
Choosing C 2 > 1 / ( 4 μ ) C_2 > 1/(4\mu) C 2 > 1/ ( 4 μ ) large enough gives
Z T 2 ≥ c Z Z_{T_2} \ge c_Z Z T 2 ≥ c Z
for all sufficiently large d d d , contradicting t ∗ > T 2 t^* > T_2 t ∗ > T 2 . Therefore
t ∗ ≤ T 2 . t^* \le T_2. t ∗ ≤ T 2 .
By continuity,
Z t ∗ = c Z . Z_{t^*} = c_Z. Z t ∗ = c Z .
The bootstrap balance gives
∣ D t ∗ ∣ ≤ 1 2 Z t ∗ . |D_{t^*}| \le\frac{1}{2}Z_{t^*}. ∣ D t ∗ ∣ ≤ 2 1 Z t ∗ .
Therefore
min { ρ t ∗ , χ t ∗ } = Z t ∗ − ∣ D t ∗ ∣ 2 ≥ c Z 4 . \min\{\rho_{t^*},\chi_{t^*}\} = \frac{Z_{t^*}-|D_{t^*}|}{2} \ge\frac{c_Z}{4}. min { ρ t ∗ , χ t ∗ } = 2 Z t ∗ − ∣ D t ∗ ∣ ≥ 4 c Z .
Set
t e s c : = t ∗ , c e s c : = c Z 4 . t_{\mathrm{esc}} := t^*, \qquad c_{\mathrm{esc}} := \frac{c_Z}{4}. t esc := t ∗ , c esc := 4 c Z .
Since
τ A ≤ C 0 log d \tau_A \le C_0\log d τ A ≤ C 0 log d
and
t ∗ ≤ τ A + C 2 log d , t^* \le\tau_A+C_2\log d, t ∗ ≤ τ A + C 2 log d ,
we have
t e s c ≤ ( C 0 + C 2 ) log d . t_{\mathrm{esc}} \le(C_0+C_2)\log d. t esc ≤ ( C 0 + C 2 ) log d .
Thus the theorem holds with
C e s c : = C 0 + C 2 . C_{\mathrm{esc}} := C_0+C_2. C esc := C 0 + C 2 .
Finally, the escape-window η t \eta_t η t -bound follows by combining the Phase-I estimate
η t ≤ K I log d + 1 + t d ≤ C log d d ( 0 ≤ t ≤ τ A ) \eta_t \le K_I\frac{\sqrt{\log d}+1+t}{\sqrt d}
\le C\frac{\log d}{\sqrt d}
\qquad(0 \le t \le\tau_A) η t ≤ K I d log d + 1 + t ≤ C d log d ( 0 ≤ t ≤ τ A )
with the Phase-II bootstrap bound
η t ≤ H = K 2 log d d ( τ A ≤ t ≤ t ∗ ) . \eta_t \le H = K_2\frac{\log d}{\sqrt d}
\qquad(\tau_A \le t \le t^*). η t ≤ H = K 2 d log d ( τ A ≤ t ≤ t ∗ ) .
Hence, throughout weak recovery window 0 ≤ t ≤ t e s c 0 \le t \le t_{\mathrm{esc}} 0 ≤ t ≤ t esc ,
η t ≤ C log d d = o d ( 1 ) . \eta_t \le C\frac{\log d}{\sqrt d}=o_d(1). η t ≤ C d log d = o d ( 1 ) .
This completes the proof.
Strong Recovery
Building on Lemma C.8, we show that after weak recovery the population dynamics enters the fixed local strong-convexity cap around the teacher in an additional O ( 1 ) O(1) O ( 1 ) time. Once inside this cap, local strong convexity gives exponential contraction to the teacher, achieving strong recovery.
Lemma C.9 (Entry into the local cap after weak recovery). Let r l o c > 0 r_{\mathrm{loc}}>0 r loc > 0 be the radius from Lemma C.14. After possibly shrinking r l o c r_{\mathrm{loc}} r loc , assume
r l o c < π 2 . r_{\mathrm{loc}} < \frac{\pi}{2}. r loc < 2 π . Set
s l o c : = cos ( r l o c / 4 ) . s_{\mathrm{loc}} := \cos(r_{\mathrm{loc}}/4). s loc := cos ( r loc /4 ) . Assume C.8 with t e s c t_{\mathrm{esc}} t esc , c e s c c_{\mathrm{esc}} c esc , and C e s c C_{\mathrm{esc}} C esc . Under the same assumptions as Lemmas C.3 and C.4, for all sufficiently large d d d , there exists a time
t l o c ≤ t e s c + C i n t_{\mathrm{loc}} \le t_{\mathrm{esc}}+C_{\mathrm{in}} t loc ≤ t esc + C in where C i n < ∞ C_{\mathrm{in}}<\infty C in < ∞ is independent of d d d , such that
ρ t l o c ≥ s l o c , χ t l o c ≥ s l o c . \rho_{t_{\mathrm{loc}}}\ge s_{\mathrm{loc}}, \qquad\chi_{t_{\mathrm{loc}}}\ge s_{\mathrm{loc}}. ρ t loc ≥ s loc , χ t loc ≥ s loc . Consequently,
dist M ( y t l o c , y ∗ ) 2 ≤ r l o c 2 8 . \operatorname{dist}_{\mathcal{M}}(y_{t_{\mathrm{loc}}},y^*)^2\le\frac{r_{\mathrm{loc}}^2}{8}. dist M ( y t loc , y ∗ ) 2 ≤ 8 r loc 2 . In particular, since t e s c ≤ C e s c log d t_{\mathrm{esc}}\le C_{\mathrm{esc}}\log d t esc ≤ C esc log d ,
t l o c ≤ C l o c log d t_{\mathrm{loc}}\le C_{\mathrm{loc}}\log d t loc ≤ C loc log d for some constant C l o c < ∞ C_{\mathrm{loc}}<\infty C loc < ∞ independent of d d d .
Proof. The proof is deterministic on the favorable event supplied by Lemma C.8. By that lemma, there are constants c e s c ∈ ( 0 , 1 ) c_{\mathrm{esc}}\in(0,1) c esc ∈ ( 0 , 1 ) and C e s c < ∞ C_{\mathrm{esc}}<\infty C esc < ∞ , independent of d d d , and a time t e s c t_{\mathrm{esc}} t esc such that
t e s c ≤ C e s c log d , ρ t e s c ≥ c e s c , χ t e s c ≥ c e s c , η t e s c = o d ( 1 ) . t_{\mathrm{esc}}\le C_{\mathrm{esc}}\log d,\qquad\rho_{t_{\mathrm{esc}}}\ge c_{\mathrm{esc}},\qquad\chi_{t_{\mathrm{esc}}}\ge c_{\mathrm{esc}},\qquad\eta_{t_{\mathrm{esc}}}=o_d(1). t esc ≤ C esc log d , ρ t esc ≥ c esc , χ t esc ≥ c esc , η t esc = o d ( 1 ) .
Set
c − : = c e s c 2 . c_-:=\frac{c_{\mathrm{esc}}}{2}. c − := 2 c esc .
Since r l o c > 0 r_{\mathrm{loc}}>0 r loc > 0 , we have
s l o c = cos ( r l o c / 4 ) < 1. s_{\mathrm{loc}}=\cos(r_{\mathrm{loc}}/4)<1. s loc = cos ( r loc /4 ) < 1.
Hence
m l o c : = 1 − s l o c 2 > 0. m_{\mathrm{loc}}:=1-s_{\mathrm{loc}}^2>0. m loc := 1 − s loc 2 > 0.
By Lemma C.3, on the positive-overlap region ρ t , χ t ≥ 0 \rho_t,\chi_t\ge0 ρ t , χ t ≥ 0 , the flow satisfies
ρ ˙ t ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t , \dot{\rho}_t\ge a_\rho\chi_t(1-\rho_t^2)-b_\rho\eta_t, ρ ˙ t ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t ,
χ ˙ t ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t , \dot{\chi}_t\ge a_\chi\rho_t(1-\chi_t^2)-b_\chi\eta_t, χ ˙ t ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t ,
where a ρ , a χ , b ρ , b χ > 0 a_\rho,a_\chi,b_\rho,b_\chi>0 a ρ , a χ , b ρ , b χ > 0 are independent of d d d .
Choose ε ∗ > 0 \varepsilon_*>0 ε ∗ > 0 , independent of d d d , so small that
b ρ ε ∗ ≤ 1 2 a ρ c − m l o c , b χ ε ∗ ≤ 1 2 a χ c − m l o c , b_\rho\varepsilon_*\le\frac{1}{2}a_\rho c_-m_{\mathrm{loc}},\qquad b_\chi\varepsilon_*\le\frac{1}{2}a_\chi c_-m_{\mathrm{loc}}, b ρ ε ∗ ≤ 2 1 a ρ c − m loc , b χ ε ∗ ≤ 2 1 a χ c − m loc ,
and also
b ρ ε ∗ ≤ 1 2 a ρ c − ( 1 − c − 2 ) , b χ ε ∗ ≤ 1 2 a χ c − ( 1 − c − 2 ) . b_\rho\varepsilon_*\le\frac{1}{2}a_\rho c_-(1-c_-^2),\qquad b_\chi\varepsilon_*\le\frac{1}{2}a_\chi c_-(1-c_-^2). b ρ ε ∗ ≤ 2 1 a ρ c − ( 1 − c − 2 ) , b χ ε ∗ ≤ 2 1 a χ c − ( 1 − c − 2 ) .
Define
v ρ : = 1 2 a ρ c − m l o c , v χ : = 1 2 a χ c − m l o c . v_\rho:=\frac{1}{2}a_\rho c_-m_{\mathrm{loc}},\qquad v_\chi:=\frac{1}{2}a_\chi c_-m_{\mathrm{loc}}. v ρ := 2 1 a ρ c − m loc , v χ := 2 1 a χ c − m loc .
Let
T ρ : = ( s l o c − c e s c ) + v ρ , T χ : = ( s l o c − c e s c ) + v χ , T_\rho:=\frac{(s_{\mathrm{loc}}-c_{\mathrm{esc}})_+}{v_\rho},\qquad T_\chi:=\frac{(s_{\mathrm{loc}}-c_{\mathrm{esc}})_+}{v_\chi}, T ρ := v ρ ( s loc − c esc ) + , T χ := v χ ( s loc − c esc ) + ,
and set
C i n : = 1 + max { T ρ , T χ } . C_{\mathrm{in}} := 1 + \max\{T_{\rho}, T_{\chi}\}. C in := 1 + max { T ρ , T χ } .
This constant depends only on the fixed population constants and on r l o c r_{\mathrm{loc}} r loc , and therefore is independent of d d d .
We first show that η t \eta_t η t remains smaller than ε ∗ \varepsilon_* ε ∗ on the fixed window
[ t e s c , t e s c + C i n ] . [t_{\mathrm{esc}}, t_{\mathrm{esc}} + C_{\mathrm{in}}]. [ t esc , t esc + C in ] .
By Lemma C.4,
η ˙ t ≤ C 1 ∣ ρ t ∣ ∥ θ ∗ ∥ ∞ + C 2 ∣ χ t ρ t ∣ η t + C 3 η t 2 . \dot{\eta}_t \le C_1|\rho_t|\|\theta^*\|_\infty+ C_2|\chi_t\rho_t|\eta_t + C_3\eta_t^2. η ˙ t ≤ C 1 ∣ ρ t ∣∥ θ ∗ ∥ ∞ + C 2 ∣ χ t ρ t ∣ η t + C 3 η t 2 .
Since ∣ ρ t ∣ , ∣ χ t ∣ ≤ 1 |\rho_t|, |\chi_t| \le1 ∣ ρ t ∣ , ∣ χ t ∣ ≤ 1 , on the present fixed window this gives
η ˙ t ≤ A d + B η t + C η t 2 , \dot{\eta}_t \le A_d + B\eta_t + C\eta_t^2, η ˙ t ≤ A d + B η t + C η t 2 ,
where
A d : = C 1 ∥ θ ∗ ∥ ∞ ≤ C log d d = o d ( 1 ) , A_d := C_1\|\theta^*\|_\infty\le C\sqrt{\frac{\log d}{d}} = o_d(1), A d := C 1 ∥ θ ∗ ∥ ∞ ≤ C d log d = o d ( 1 ) ,
and B , C < ∞ B,C < \infty B , C < ∞ are independent of d d d .
Define
τ η : = inf { t ≥ t e s c : η t ≥ ε ∗ } ∧ ( t e s c + C i n ) . \tau_\eta:= \inf\{t \ge t_{\mathrm{esc}} : \eta_t \ge\varepsilon_*\} \wedge(t_{\mathrm{esc}} + C_{\mathrm{in}}). τ η := inf { t ≥ t esc : η t ≥ ε ∗ } ∧ ( t esc + C in ) .
Since η t e s c = o d ( 1 ) \eta_{t_{\mathrm{esc}}} = o_d(1) η t esc = o d ( 1 ) , for all sufficiently large d d d ,
η t e s c < ε ∗ . \eta_{t_{\mathrm{esc}}} < \varepsilon_*. η t esc < ε ∗ .
Thus τ η > t e s c \tau_\eta> t_{\mathrm{esc}} τ η > t esc . On [ t e s c , τ η ] [t_{\mathrm{esc}}, \tau_\eta] [ t esc , τ η ] , we have η t ≤ ε ∗ \eta_t \le\varepsilon_* η t ≤ ε ∗ , and therefore for almost every t ∈ [ t e s c , τ η ] t \in[t_{\mathrm{esc}}, \tau_\eta] t ∈ [ t esc , τ η ] ,
η ˙ t ≤ A d + ( B + C ε ∗ ) η t . \dot{\eta}_t \le A_d + (B + C\varepsilon_*)\eta_t. η ˙ t ≤ A d + ( B + C ε ∗ ) η t .
Set
B ∗ : = B + C ε ∗ . B_* := B + C\varepsilon_*. B ∗ := B + C ε ∗ .
By Gronwall’s inequality, for every t ∈ [ t e s c , τ η ] t \in[t_{\mathrm{esc}}, \tau_\eta] t ∈ [ t esc , τ η ] ,
η t ≤ e B ∗ ( t − t e s c ) η t e s c + A d ∫ t e s c t e B ∗ ( t − u ) d u . \eta_t \le e^{B_*(t-t_{\mathrm{esc}})}\eta_{t_{\mathrm{esc}}} + A_d\int_{t_{\mathrm{esc}}}^{t}e^{B_*(t-u)}\,du. η t ≤ e B ∗ ( t − t esc ) η t esc + A d ∫ t esc t e B ∗ ( t − u ) d u .
Hence
η t ≤ e B ∗ C i n η t e s c + A d C i n e B ∗ C i n . \eta_t \le e^{B_*C_{\mathrm{in}}}\eta_{t_{\mathrm{esc}}} + A_d C_{\mathrm{in}}e^{B_*C_{\mathrm{in}}}. η t ≤ e B ∗ C in η t esc + A d C in e B ∗ C in .
The right-hand side is o d ( 1 ) o_d(1) o d ( 1 ) . Therefore, for all sufficiently large d d d ,
η t < ε ∗ for all t ∈ [ t e s c , τ η ] . \eta_t < \varepsilon_* \qquad\text{for all }t \in[t_{\mathrm{esc}}, \tau_\eta]. η t < ε ∗ for all t ∈ [ t esc , τ η ] .
By the definition of τ η \tau_\eta τ η , this forces
τ η = t e s c + C i n . \tau_\eta= t_{\mathrm{esc}} + C_{\mathrm{in}}. τ η = t esc + C in .
Consequently,
η t ≤ ε ∗ for all t ∈ [ t e s c , t e s c + C i n ] . \eta_t \le\varepsilon_* \qquad\text{for all }t \in[t_{\mathrm{esc}}, t_{\mathrm{esc}} + C_{\mathrm{in}}]. η t ≤ ε ∗ for all t ∈ [ t esc , t esc + C in ] .
Next we prove that the rectangle
R − : = { ( ρ , χ ) : ρ ≥ c − , χ ≥ c − } \mathcal{R}_{-} := \{(\rho,\chi) : \rho\ge c_{-},\ \chi\ge c_{-}\} R − := {( ρ , χ ) : ρ ≥ c − , χ ≥ c − }
is forward invariant on the same window. At time t e s c t_{\mathrm{esc}} t esc ,
ρ t e s c ≥ c e s c > c − , χ t e s c ≥ c e s c > c − . \rho_{t_{\mathrm{esc}}} \ge c_{\mathrm{esc}} > c_{-}, \qquad\chi_{t_{\mathrm{esc}}} \ge c_{\mathrm{esc}} > c_{-}. ρ t esc ≥ c esc > c − , χ t esc ≥ c esc > c − .
Suppose, for contradiction, that the trajectory exits R − \mathcal{R}_{-} R − before t e s c + C i n t_{\mathrm{esc}} + C_{\mathrm{in}} t esc + C in . Let
τ − : = inf { t ∈ [ t e s c , t e s c + C i n ] : min { ρ t , χ t } ≤ c − } . \tau_{-} := \inf\{t \in[t_{\mathrm{esc}}, t_{\mathrm{esc}} + C_{\mathrm{in}}] : \min\{\rho_t,\chi_t\} \le c_{-}\}. τ − := inf { t ∈ [ t esc , t esc + C in ] : min { ρ t , χ t } ≤ c − } .
Then τ − < ∞ \tau_{-} < \infty τ − < ∞ , and by continuity,
ρ τ − ≥ c − , χ τ − ≥ c − , \rho_{\tau_{-}} \ge c_{-}, \qquad\chi_{\tau_{-}} \ge c_{-}, ρ τ − ≥ c − , χ τ − ≥ c − ,
with at least one equality. If ρ τ − = c − \rho_{\tau_-}=c_- ρ τ − = c − , then ρ τ − , χ τ − ≥ 0 \rho_{\tau_-},\chi_{\tau_-}\ge0 ρ τ − , χ τ − ≥ 0 , so Lemma C.3 applies. Using also η τ − ≤ ε ∗ \eta_{\tau_-}\le\varepsilon_* η τ − ≤ ε ∗ , we get
ρ ˙ τ − ≥ a ρ c − ( 1 − c − 2 ) − b ρ ε ∗ ≥ 1 2 a ρ c − ( 1 − c − 2 ) > 0. \dot{\rho}_{\tau_-}\ge a_\rho c_-(1-c_-^2)-b_\rho\varepsilon_*\ge\frac{1}{2}a_\rho c_-(1-c_-^2)>0. ρ ˙ τ − ≥ a ρ c − ( 1 − c − 2 ) − b ρ ε ∗ ≥ 2 1 a ρ c − ( 1 − c − 2 ) > 0.
Thus ρ t \rho_t ρ t cannot cross downward through c − c_- c − at τ − \tau_- τ − . Similarly, if χ τ − = c − \chi_{\tau_-}=c_- χ τ − = c − , then
χ ˙ τ − ≥ a χ c − ( 1 − c − 2 ) − b χ ε ∗ ≥ 1 2 a χ c − ( 1 − c − 2 ) > 0 , \dot{\chi}_{\tau_-}\ge a_\chi c_-(1-c_-^2)-b_\chi\varepsilon_*\ge\frac{1}{2}a_\chi c_-(1-c_-^2)>0, χ ˙ τ − ≥ a χ c − ( 1 − c − 2 ) − b χ ε ∗ ≥ 2 1 a χ c − ( 1 − c − 2 ) > 0 ,
so χ t \chi_t χ t cannot cross downward through c − c_- c − . This contradicts the definition of τ − \tau_- τ − . Therefore,
ρ t , χ t ≥ c − for all t ∈ [ t e s c , t e s c + C i n ] . \rho_t,\chi_t\ge c_-\qquad\text{for all }t\in[t_{\mathrm{esc}},t_{\mathrm{esc}}+C_{\mathrm{in}}]. ρ t , χ t ≥ c − for all t ∈ [ t esc , t esc + C in ] .
We now show that ρ t \rho_t ρ t reaches s l o c s_{\mathrm{loc}} s loc within time T ρ T_\rho T ρ and then cannot fall below s l o c s_{\mathrm{loc}} s loc during the window. Define
τ ρ : = inf { t ≥ t e s c : ρ t ≥ s l o c } . \tau_\rho:=\inf\{t\ge t_{\mathrm{esc}}:\rho_t\ge s_{\mathrm{loc}}\}. τ ρ := inf { t ≥ t esc : ρ t ≥ s loc } .
If ρ t e s c ≥ s l o c \rho_{t_{\mathrm{esc}}}\ge s_{\mathrm{loc}} ρ t esc ≥ s loc , then τ ρ = t e s c \tau_\rho=t_{\mathrm{esc}} τ ρ = t esc . Otherwise, before ρ t \rho_t ρ t reaches s l o c s_{\mathrm{loc}} s loc , we have ρ t < s l o c \rho_t<s_{\mathrm{loc}} ρ t < s loc . On this interval, the rectangle invariance gives χ t ≥ c − \chi_t\ge c_- χ t ≥ c − , and the previous step gives η t ≤ ε ∗ \eta_t\le\varepsilon_* η t ≤ ε ∗ . Hence Lemma C.3 implies
ρ ˙ t ≥ a ρ c − ( 1 − ρ t 2 ) − b ρ ε ∗ . \dot{\rho}_t\ge a_\rho c_-(1-\rho_t^2)-b_\rho\varepsilon_*. ρ ˙ t ≥ a ρ c − ( 1 − ρ t 2 ) − b ρ ε ∗ .
Since ρ t < s l o c \rho_t<s_{\mathrm{loc}} ρ t < s loc ,
1 − ρ t 2 ≥ 1 − s l o c 2 = m l o c . 1-\rho_t^2\ge1-s_{\mathrm{loc}}^2=m_{\mathrm{loc}}. 1 − ρ t 2 ≥ 1 − s loc 2 = m loc .
Therefore
ρ ˙ t ≥ a ρ c − m l o c − b ρ ε ∗ ≥ 1 2 a ρ c − m l o c = v ρ . \dot{\rho}_t\ge a_\rho c_-m_{\mathrm{loc}}-b_\rho\varepsilon_*\ge\frac{1}{2}a_\rho c_-m_{\mathrm{loc}}=v_\rho. ρ ˙ t ≥ a ρ c − m loc − b ρ ε ∗ ≥ 2 1 a ρ c − m loc = v ρ .
Thus ρ t \rho_t ρ t reaches s l o c s_{\mathrm{loc}} s loc no later than
t e s c + T ρ . t_{\mathrm{esc}}+T_\rho. t esc + T ρ .
Moreover, after ρ t \rho_t ρ t reaches s l o c s_{\mathrm{loc}} s loc , it cannot cross downward through s l o c s_{\mathrm{loc}} s loc inside the same window. Indeed, at a hypothetical first downward crossing through s l o c s_{\mathrm{loc}} s loc , we would have
ρ t = s l o c , χ t ≥ c − , η t ≤ ε ∗ . \rho_t=s_{\mathrm{loc}},\qquad\chi_t\ge c_-,\qquad\eta_t\le\varepsilon_*. ρ t = s loc , χ t ≥ c − , η t ≤ ε ∗ .
Therefore
ρ ˙ t ≥ a ρ c − ( 1 − s l o c 2 ) − b ρ ε ∗ ≥ v ρ > 0 , \dot{\rho}_t\ge a_\rho c_-(1-s_{\mathrm{loc}}^2)-b_\rho\varepsilon_*\ge v_\rho>0, ρ ˙ t ≥ a ρ c − ( 1 − s loc 2 ) − b ρ ε ∗ ≥ v ρ > 0 ,
which is incompatible with a first downward crossing. The same argument applies to χ t \chi_t χ t . Define
τ χ : = inf { t ≥ t e s c : χ t ≥ s l o c } . \tau_\chi:=\inf\{t\ge t_{\mathrm{esc}}:\chi_t\ge s_{\mathrm{loc}}\}. τ χ := inf { t ≥ t esc : χ t ≥ s loc } .
If χ t e s c ≥ s l o c \chi_{t_{\mathrm{esc}}}\ge s_{\mathrm{loc}} χ t esc ≥ s loc , then τ χ = t e s c \tau_\chi=t_{\mathrm{esc}} τ χ = t esc . Otherwise, before χ t \chi_t χ t reaches s l o c s_{\mathrm{loc}} s loc , we have χ t < s l o c \chi_t<s_{\mathrm{loc}} χ t < s loc , while ρ t ≥ c − \rho_t\ge c_- ρ t ≥ c − and η t ≤ ε ∗ \eta_t\le\varepsilon_* η t ≤ ε ∗ . Hence
χ ˙ t ≥ a χ c − ( 1 − χ t 2 ) − b χ ε ∗ ≥ a χ c − m l o c − b χ ε ∗ ≥ v χ . \dot{\chi}_t\ge a_\chi c_-(1-\chi_t^2)-b_\chi\varepsilon_*\ge a_\chi c_-m_{\mathrm{loc}}-b_\chi\varepsilon_*\ge v_\chi. χ ˙ t ≥ a χ c − ( 1 − χ t 2 ) − b χ ε ∗ ≥ a χ c − m loc − b χ ε ∗ ≥ v χ .
Thus χ t \chi_t χ t reaches s l o c s_{\mathrm{loc}} s loc no later than
t e s c + T χ . t_{\mathrm{esc}}+T_\chi. t esc + T χ .
Once it reaches s l o c s_{\mathrm{loc}} s loc , it cannot cross downward through s l o c s_{\mathrm{loc}} s loc , because at such a crossing the same estimate gives
χ ˙ t ≥ v χ > 0. \dot{\chi}_t\ge v_\chi>0. χ ˙ t ≥ v χ > 0.
Now set
t l o c : = max { τ ρ , τ χ } . t_{\mathrm{loc}}:=\max\{\tau_\rho,\tau_\chi\}. t loc := max { τ ρ , τ χ } .
By the previous estimates,
t l o c ≤ t e s c + max { T ρ , T χ } ≤ t e s c + C i n , t_{\mathrm{loc}}\le t_{\mathrm{esc}}+\max\{T_\rho,T_\chi\}\le t_{\mathrm{esc}}+C_{\mathrm{in}}, t loc ≤ t esc + max { T ρ , T χ } ≤ t esc + C in ,
and by the no-downcrossing arguments,
ρ t l o c ≥ s l o c , χ t l o c ≥ s l o c . \rho_{t_{\mathrm{loc}}}\ge s_{\mathrm{loc}},\qquad\chi_{t_{\mathrm{loc}}}\ge s_{\mathrm{loc}}. ρ t loc ≥ s loc , χ t loc ≥ s loc .
It remains to convert the overlap bounds into a product-manifold distance bound. Since
ρ t l o c = ⟨ β t l o c , β ∗ ⟩ ≥ s l o c = cos ( r l o c / 4 ) , \rho_{t_{\mathrm{loc}}}=\langle\beta_{t_{\mathrm{loc}}},\beta^*\rangle\ge s_{\mathrm{loc}}=\cos(r_{\mathrm{loc}}/4), ρ t loc = ⟨ β t loc , β ∗ ⟩ ≥ s loc = cos ( r loc /4 ) ,
and since arccos \arccos arccos is decreasing on [ − 1 , 1 ] [-1,1] [ − 1 , 1 ] ,
dist S d − 1 ( β t l o c , β ∗ ) = arccos ( ρ t l o c ) ≤ r l o c 4 . \operatorname{dist}_{\mathbb{S}^{d-1}}(\beta_{t_{\mathrm{loc}}},\beta^*)=\arccos(\rho_{t_{\mathrm{loc}}})\le\frac{r_{\mathrm{loc}}}{4}. dist S d − 1 ( β t loc , β ∗ ) = arccos ( ρ t loc ) ≤ 4 r loc .
Similarly,
dist S d − 1 ( θ t l o c , θ ∗ ) = arccos ( χ t l o c ) ≤ r l o c 4 . \operatorname{dist}_{\mathbb{S}^{d-1}}(\theta_{t_{\mathrm{loc}}},\theta^*)=\arccos(\chi_{t_{\mathrm{loc}}})\le\frac{r_{\mathrm{loc}}}{4}. dist S d − 1 ( θ t loc , θ ∗ ) = arccos ( χ t loc ) ≤ 4 r loc .
Using the product metric on M = S d − 1 × S d − 1 \mathcal{M}=\mathbb{S}^{d-1}\times\mathbb{S}^{d-1} M = S d − 1 × S d − 1 , we get
dist M ( y t l o c , y ∗ ) 2 = dist S d − 1 ( θ t l o c , θ ∗ ) 2 + dist S d − 1 ( β t l o c , β ∗ ) 2 . \operatorname{dist}_{\mathcal{M}}(y_{t_{\mathrm{loc}}},y^*)^2=\operatorname{dist}_{\mathbb{S}^{d-1}}(\theta_{t_{\mathrm{loc}}},\theta^*)^2+\operatorname{dist}_{\mathbb{S}^{d-1}}(\beta_{t_{\mathrm{loc}}},\beta^*)^2. dist M ( y t loc , y ∗ ) 2 = dist S d − 1 ( θ t loc , θ ∗ ) 2 + dist S d − 1 ( β t loc , β ∗ ) 2 .
Therefore,
dist M ( y t l o c , y ∗ ) 2 ≤ ( r l o c 4 ) 2 + ( r l o c 4 ) 2 = r l o c 2 8 . \operatorname{dist}_{\mathcal{M}}(y_{t_{\mathrm{loc}}},y^*)^2\le\left(\frac{r_{\mathrm{loc}}}{4}\right)^2+\left(\frac{r_{\mathrm{loc}}}{4}\right)^2=\frac{r_{\mathrm{loc}}^2}{8}. dist M ( y t loc , y ∗ ) 2 ≤ ( 4 r loc ) 2 + ( 4 r loc ) 2 = 8 r loc 2 .
Finally, since Lemma C.8 gives t e s c ≤ C e s c log d t_{\mathrm{esc}}\le C_{\mathrm{esc}}\log d t esc ≤ C esc log d , we have
t l o c ≤ C e s c log d + C i n . t_{\mathrm{loc}}\le C_{\mathrm{esc}}\log d+C_{\mathrm{in}}. t loc ≤ C esc log d + C in .
For d ≥ e d\ge e d ≥ e ,
C i n ≤ C i n log d , C_{\mathrm{in}}\le C_{\mathrm{in}}\log d, C in ≤ C in log d ,
and hence
t l o c ≤ ( C e s c + C i n ) log d . t_{\mathrm{loc}}\le(C_{\mathrm{esc}}+C_{\mathrm{in}})\log d. t loc ≤ ( C esc + C in ) log d .
Thus the last claim holds with
C l o c : = C e s c + C i n . C_{\mathrm{loc}}:=C_{\mathrm{esc}}+C_{\mathrm{in}}. C loc := C esc + C in .
The proof is complete. □
Curvature near the teacher. Here, we prove the local curvature estimate used for the contraction phase. The comparison with the Gaussian model is needed only at the teacher; the extension to a small cap uses Hessian continuity of the true loss.
Lemma C.10 (Gaussian local strong convexity). Let M : = S d − 1 × S d − 1 \mathcal{M}:=\mathbb{S}^{d-1}\times\mathbb{S}^{d-1} M := S d − 1 × S d − 1 , y = ( θ , β ) y=(\theta,\beta) y = ( θ , β ) , y ∗ = ( θ ∗ , β ∗ ) y^*=(\theta^*,\beta^*) y ∗ = ( θ ∗ , β ∗ ) . Let L G L_G L G be the Gaussian population loss
L G ( θ , β ) = E [ ( ψ ( g ⋅ θ ) − ψ ( g ∗ ⋅ θ ∗ ) ) 2 ] , L_G(\theta,\beta)=\mathbb{E}\left[\left(\psi(g\cdot\theta)-\psi(g^*\cdot\theta^*)\right)^2\right], L G ( θ , β ) = E [ ( ψ ( g ⋅ θ ) − ψ ( g ∗ ⋅ θ ∗ ) ) 2 ] , where ( g , g ∗ ) ∈ R d × R d (g,g^*)\in\mathbb{R}^d\times\mathbb{R}^d ( g , g ∗ ) ∈ R d × R d is centered Gaussian with E [ g i g j ] = δ i j \mathbb{E}[g_i g_j]=\delta_{ij} E [ g i g j ] = δ ij , E [ g i ∗ g j ∗ ] = δ i j \mathbb{E}[g_i^*g_j^*]=\delta_{ij} E [ g i ∗ g j ∗ ] = δ ij , E [ g i g j ∗ ] = κ ( ρ ) δ i j \mathbb{E}[g_i g_j^*]=\kappa(\rho)\delta_{ij} E [ g i g j ∗ ] = κ ( ρ ) δ ij , with
ρ : = β ⋅ β ∗ , κ ( ρ ) : = ∑ ℓ ≥ 1 a ℓ 2 ρ ℓ . \rho:=\beta\cdot\beta^*,\qquad\kappa(\rho):=\sum_{\ell\ge1}a_\ell^2\rho^\ell. ρ := β ⋅ β ∗ , κ ( ρ ) := ℓ ≥ 1 ∑ a ℓ 2 ρ ℓ . Assume ∥ ϕ ∥ L 2 ( γ ) = 1 \|\phi\|_{L^2(\gamma)}=1 ∥ ϕ ∥ L 2 ( γ ) = 1 , so κ ( 1 ) = 1 \kappa(1)=1 κ ( 1 ) = 1 , and assume the information-exponent-one conditions α 1 ≠ 0 \alpha_1\ne0 α 1 = 0 , a 1 ≠ 0 a_1\ne0 a 1 = 0 , where
ψ = ∑ k ≥ 0 α k h k , ϕ = ∑ ℓ ≥ 1 a ℓ h ℓ . \psi=\sum_{k\ge0}\alpha_k h_k,\qquad\phi=\sum_{\ell\ge1}a_\ell h_\ell. ψ = k ≥ 0 ∑ α k h k , ϕ = ℓ ≥ 1 ∑ a ℓ h ℓ . Assume also enough regularity so that the function below is C 2 C^2 C 2 in a neighborhood of ( 1 , 1 ) (1,1) ( 1 , 1 ) . Then there exist constants r G > 0 r_G>0 r G > 0 and μ G > 0 \mu_G>0 μ G > 0 , independent of d d d , such that for every
( θ , β ) ∈ U r G : = { ( θ , β ) : dist S ( θ , θ ∗ ) 2 + dist S ( β , β ∗ ) 2 ≤ r G 2 } , (\theta,\beta)\in U_{r_G}:=\left\{(\theta,\beta):\operatorname{dist}_{\mathbb{S}}(\theta,\theta^*)^2+\operatorname{dist}_{\mathbb{S}}(\beta,\beta^*)^2\le r_G^2\right\}, ( θ , β ) ∈ U r G := { ( θ , β ) : dist S ( θ , θ ∗ ) 2 + dist S ( β , β ∗ ) 2 ≤ r G 2 } , and every product tangent vector ξ = ( u , v ) ∈ T θ S d − 1 × T β S d − 1 \xi=(u,v)\in T_\theta\mathbb{S}^{d-1}\times T_\beta\mathbb{S}^{d-1} ξ = ( u , v ) ∈ T θ S d − 1 × T β S d − 1 , one has
Hess M L G ( θ , β ) [ ξ , ξ ] ≥ μ G ( ∥ u ∥ 2 + ∥ v ∥ 2 ) . (62) \operatorname{Hess}_{\mathcal{M}}L_G(\theta,\beta)[\xi,\xi]\ge\mu_G\left(\|u\|^2+\|v\|^2\right). \tag*{(62)} Hess M L G ( θ , β ) [ ξ , ξ ] ≥ μ G ( ∥ u ∥ 2 + ∥ v ∥ 2 ) . (62)
Proof. In the Gaussian model, the scalar pair ( g ⋅ θ , g ∗ ⋅ θ ∗ ) (g \cdot\theta, g^* \cdot\theta^*) ( g ⋅ θ , g ∗ ⋅ θ ∗ ) is centered Gaussian with unit variances and correlation m = κ ( ρ ) χ m = \kappa(\rho)\chi m = κ ( ρ ) χ . Therefore, using the Hermite expansion of ψ \psi ψ ,
E [ ψ ( g ⋅ θ ) ψ ( g ∗ ⋅ θ ∗ ) ] = ∑ k ≥ 0 α k 2 ( κ ( ρ ) χ ) k . \mathbb{E}[\psi(g \cdot\theta)\psi(g^* \cdot\theta^*)] = \sum_{k \ge0} \alpha_k^2(\kappa(\rho)\chi)^k. E [ ψ ( g ⋅ θ ) ψ ( g ∗ ⋅ θ ∗ )] = k ≥ 0 ∑ α k 2 ( κ ( ρ ) χ ) k .
Since both marginals have variance one,
E [ ψ ( g ⋅ θ ) 2 ] = E [ ψ ( g ∗ ⋅ θ ∗ ) 2 ] = ∑ k ≥ 0 α k 2 . \mathbb{E}[\psi(g \cdot\theta)^2] = \mathbb{E}[\psi(g^* \cdot\theta^*)^2] = \sum_{k \ge0} \alpha_k^2. E [ ψ ( g ⋅ θ ) 2 ] = E [ ψ ( g ∗ ⋅ θ ∗ ) 2 ] = k ≥ 0 ∑ α k 2 .
Hence
L G ( θ , β ) = 2 ∑ k ≥ 0 α k 2 [ 1 − ( κ ( ρ ) χ ) k ] . (63) L_G(\theta,\beta) = 2\sum_{k \ge0}\alpha_k^2\left[1-(\kappa(\rho)\chi)^k\right]. \tag*{(63)} L G ( θ , β ) = 2 k ≥ 0 ∑ α k 2 [ 1 − ( κ ( ρ ) χ ) k ] . (63)
with k = 0 k=0 k = 0 term is zero. We define
Φ ( χ , ρ ) : = 2 ∑ k ≥ 1 α k 2 [ 1 − ( κ ( ρ ) χ ) k ] = L G ( θ , β ) . \Phi(\chi,\rho) := 2\sum_{k \ge1}\alpha_k^2\left[1-(\kappa(\rho)\chi)^k\right] = L_G(\theta,\beta). Φ ( χ , ρ ) := 2 k ≥ 1 ∑ α k 2 [ 1 − ( κ ( ρ ) χ ) k ] = L G ( θ , β ) .
Now we first compute the Hessian at the planted point ( θ , β ) = ( θ ∗ , β ∗ ) (\theta,\beta)=(\theta^*,\beta^*) ( θ , β ) = ( θ ∗ , β ∗ ) : we have χ = 1 \chi=1 χ = 1 , ρ = 1 \rho=1 ρ = 1 , κ ( 1 ) = 1 \kappa(1)=1 κ ( 1 ) = 1 . Also Φ χ ( 1 , 1 ) = − 2 ∑ k ≥ 1 k α k 2 \Phi_\chi(1,1)=-2\sum_{k \ge1}k\alpha_k^2 Φ χ ( 1 , 1 ) = − 2 ∑ k ≥ 1 k α k 2 , and Φ ρ ( 1 , 1 ) = − 2 κ ′ ( 1 ) ∑ k ≥ 1 k α k 2 \Phi_\rho(1,1)=-2\kappa'(1)\sum_{k \ge1}k\alpha_k^2 Φ ρ ( 1 , 1 ) = − 2 κ ′ ( 1 ) ∑ k ≥ 1 k α k 2 . Define
λ θ : = − Φ χ ( 1 , 1 ) = 2 ∑ k ≥ 1 k α k 2 , \lambda_\theta:= -\Phi_\chi(1,1)=2\sum_{k \ge1}k\alpha_k^2, λ θ := − Φ χ ( 1 , 1 ) = 2 k ≥ 1 ∑ k α k 2 ,
and
λ β : = − Φ ρ ( 1 , 1 ) = 2 κ ′ ( 1 ) ∑ k ≥ 1 k α k 2 . \lambda_\beta:= -\Phi_\rho(1,1)=2\kappa'(1)\sum_{k \ge1}k\alpha_k^2. λ β := − Φ ρ ( 1 , 1 ) = 2 κ ′ ( 1 ) k ≥ 1 ∑ k α k 2 .
The information-exponent-one condition α 1 ≠ 0 \alpha_1 \ne0 α 1 = 0 implies λ θ > 0 \lambda_\theta> 0 λ θ > 0 . Moreover,
κ ′ ( 1 ) = ∑ ℓ ≥ 1 ℓ a ℓ 2 > 0 , \kappa'(1)=\sum_{\ell\ge1}\ell a_\ell^2>0, κ ′ ( 1 ) = ℓ ≥ 1 ∑ ℓ a ℓ 2 > 0 ,
because a 1 ≠ 0 a_1 \ne0 a 1 = 0 . Hence λ β > 0 \lambda_\beta> 0 λ β > 0 .
Now let u ∈ T θ ∗ S d − 1 u \in T_{\theta^*}S^{d-1} u ∈ T θ ∗ S d − 1 , v ∈ T β ∗ S d − 1 v \in T_{\beta^*}S^{d-1} v ∈ T β ∗ S d − 1 and let ( θ t , β t ) (\theta_t,\beta_t) ( θ t , β t ) be the product geodesic with
( θ 0 , β 0 ) = ( θ ∗ , β ∗ ) , ( θ ˙ 0 , β ˙ 0 ) = ( u , v ) . (\theta_0,\beta_0)=(\theta^*,\beta^*), \qquad(\dot{\theta}_0,\dot{\beta}_0)=(u,v). ( θ 0 , β 0 ) = ( θ ∗ , β ∗ ) , ( θ ˙ 0 , β ˙ 0 ) = ( u , v ) .
Then
χ ( t ) : = θ t ⋅ θ ∗ = cos ( t ∥ u ∥ ) , \chi(t):=\theta_t\cdot\theta^*=\cos(t\lVert u\rVert), χ ( t ) := θ t ⋅ θ ∗ = cos ( t ∥ u ∥) ,
and
ρ ( t ) : = β t ⋅ β ∗ = cos ( t ∥ v ∥ ) . \rho(t):=\beta_t\cdot\beta^*=\cos(t\lVert v\rVert). ρ ( t ) := β t ⋅ β ∗ = cos ( t ∥ v ∥) .
Thus
χ ′ ( 0 ) = 0 , χ ′ ′ ( 0 ) = − ∥ u ∥ 2 , \chi'(0)=0,\qquad\chi''(0)=-\lVert u\rVert^2, χ ′ ( 0 ) = 0 , χ ′′ ( 0 ) = − ∥ u ∥ 2 ,
and
ρ ′ ( 0 ) = 0 , ρ ′ ′ ( 0 ) = − ∥ v ∥ 2 . \rho'(0)=0,\qquad\rho''(0)=-\lVert v\rVert^2. ρ ′ ( 0 ) = 0 , ρ ′′ ( 0 ) = − ∥ v ∥ 2 .
Since the curve is a product geodesic, the Riemannian Hessian is computed by the second derivative along this curve:
Hess M L G ( y ∗ ) [ ( u , v ) , ( u , v ) ] = d 2 d t 2 ∣ t = 0 Φ ( χ ( t ) , ρ ( t ) ) . \operatorname{Hess}_{\mathcal{M}}L_G(y^*)[(u,v),(u,v)] =
\left.\frac{\mathrm{d}^2}{\mathrm{d}t^2}\right|_{t=0}\Phi(\chi(t),\rho(t)). Hess M L G ( y ∗ ) [( u , v ) , ( u , v )] = d t 2 d 2 t = 0 Φ ( χ ( t ) , ρ ( t )) .
By the chain rule and the fact that χ ′ ( 0 ) = ρ ′ ( 0 ) = 0 \chi'(0)=\rho'(0)=0 χ ′ ( 0 ) = ρ ′ ( 0 ) = 0 ,
d 2 d t 2 ∣ t = 0 Φ ( χ ( t ) , ρ ( t ) ) = Φ χ ( 1 , 1 ) χ ′ ′ ( 0 ) + Φ ρ ( 1 , 1 ) ρ ′ ′ ( 0 ) . \left.\frac{\mathrm{d}^2}{\mathrm{d}t^2}\right|_{t=0}\Phi(\chi(t),\rho(t)) =
\Phi_\chi(1,1)\chi''(0)+\Phi_\rho(1,1)\rho''(0). d t 2 d 2 t = 0 Φ ( χ ( t ) , ρ ( t )) = Φ χ ( 1 , 1 ) χ ′′ ( 0 ) + Φ ρ ( 1 , 1 ) ρ ′′ ( 0 ) .
Therefore
Hess M L G ( y ∗ ) [ ( u , v ) , ( u , v ) ] = λ θ ∥ u ∥ 2 + λ β ∥ v ∥ 2 . \operatorname{Hess}_{\mathcal{M}}L_G(y^*)[(u,v),(u,v)] =
\lambda_\theta\lVert u\rVert^2+\lambda_\beta\lVert v\rVert^2. Hess M L G ( y ∗ ) [( u , v ) , ( u , v )] = λ θ ∥ u ∥ 2 + λ β ∥ v ∥ 2 .
Consequently, with
λ 0 : = min { λ θ , λ β } > 0 , \lambda_0:=\min\{\lambda_\theta,\lambda_\beta\}>0, λ 0 := min { λ θ , λ β } > 0 ,
we have
Hess M L G ( y ∗ ) [ ( u , v ) , ( u , v ) ] ≥ λ 0 ( ∥ u ∥ 2 + ∥ v ∥ 2 ) . \operatorname{Hess}_{\mathcal M} L_G(y^*)[(u,v),(u,v)] \ge\lambda_0(\lVert u\rVert^2+\lVert v\rVert^2). Hess M L G ( y ∗ ) [( u , v ) , ( u , v )] ≥ λ 0 (∥ u ∥ 2 + ∥ v ∥ 2 ) .
It remains to extend this lower bound to a small product cap around y ∗ y^* y ∗ . We do this explicitly.
Let
( θ , β ) ∈ M , u ∈ T θ S d − 1 , v ∈ T β S d − 1 . (\theta,\beta)\in\mathcal M,\qquad u\in T_\theta S^{d-1},\qquad v\in T_\beta S^{d-1}. ( θ , β ) ∈ M , u ∈ T θ S d − 1 , v ∈ T β S d − 1 .
Let ( θ t , β t ) (\theta_t,\beta_t) ( θ t , β t ) be the product geodesic starting from ( θ , β ) (\theta,\beta) ( θ , β ) with initial velocity ( u , v ) (u,v) ( u , v ) . Then
χ ′ ( 0 ) = u ⋅ θ ∗ , χ ′ ′ ( 0 ) = − χ ∥ u ∥ 2 , \chi'(0)=u\cdot\theta^*,\qquad\chi''(0)=-\chi\lVert u\rVert^2, χ ′ ( 0 ) = u ⋅ θ ∗ , χ ′′ ( 0 ) = − χ ∥ u ∥ 2 ,
and
ρ ′ ( 0 ) = v ⋅ β ∗ , ρ ′ ′ ( 0 ) = − ρ ∥ v ∥ 2 . \rho'(0)=v\cdot\beta^*,\qquad\rho''(0)=-\rho\lVert v\rVert^2. ρ ′ ( 0 ) = v ⋅ β ∗ , ρ ′′ ( 0 ) = − ρ ∥ v ∥ 2 .
Therefore
Hess M L G ( θ , β ) [ ( u , v ) , ( u , v ) ] (64) \operatorname{Hess}_{\mathcal M} L_G(\theta,\beta)[(u,v),(u,v)] \tag*{(64)} Hess M L G ( θ , β ) [( u , v ) , ( u , v )] (64)
= Φ χ χ ( χ , ρ ) ( u ⋅ θ ∗ ) 2 + 2 Φ χ ρ ( χ , ρ ) ( u ⋅ θ ∗ ) ( v ⋅ β ∗ ) + Φ ρ ρ ( χ , ρ ) ( v ⋅ β ∗ ) 2 − χ Φ χ ( χ , ρ ) ∥ u ∥ 2 − ρ Φ ρ ( χ , ρ ) ∥ v ∥ 2 . (65) =\Phi_{\chi\chi}(\chi,\rho)(u\cdot\theta^*)^2+2\Phi_{\chi\rho}(\chi,\rho)(u\cdot\theta^*)(v\cdot\beta^*)+\Phi_{\rho\rho}(\chi,\rho)(v\cdot\beta^*)^2
-\chi\Phi_\chi(\chi,\rho)\lVert u\rVert^2-\rho\Phi_\rho(\chi,\rho)\lVert v\rVert^2. \tag*{(65)} = Φ χχ ( χ , ρ ) ( u ⋅ θ ∗ ) 2 + 2 Φ χ ρ ( χ , ρ ) ( u ⋅ θ ∗ ) ( v ⋅ β ∗ ) + Φ ρρ ( χ , ρ ) ( v ⋅ β ∗ ) 2 − χ Φ χ ( χ , ρ ) ∥ u ∥ 2 − ρ Φ ρ ( χ , ρ ) ∥ v ∥ 2 . (65)
We now estimate the terms in this expression. Choose a small neighborhood
N : = { ( χ , ρ ) : ∣ χ − 1 ∣ ≤ r 0 , ∣ ρ − 1 ∣ ≤ r 0 } \mathcal N:=\{(\chi,\rho):|\chi-1|\le r_0,\ |\rho-1|\le r_0\} N := {( χ , ρ ) : ∣ χ − 1∣ ≤ r 0 , ∣ ρ − 1∣ ≤ r 0 }
on which all second derivatives of Φ \Phi Φ are bounded:
∣ Φ χ χ ∣ + ∣ Φ χ ρ ∣ + ∣ Φ ρ ρ ∣ ≤ M . |\Phi_{\chi\chi}|+|\Phi_{\chi\rho}|+|\Phi_{\rho\rho}|\le M. ∣ Φ χχ ∣ + ∣ Φ χ ρ ∣ + ∣ Φ ρρ ∣ ≤ M .
By continuity of Φ χ , Φ ρ \Phi_\chi,\Phi_\rho Φ χ , Φ ρ , after shrinking r 0 r_0 r 0 if necessary, we also have
− χ Φ χ ( χ , ρ ) ≥ λ θ 2 , − ρ Φ ρ ( χ , ρ ) ≥ λ β 2 -\chi\Phi_\chi(\chi,\rho)\ge\frac{\lambda_\theta}{2},\qquad
-\rho\Phi_\rho(\chi,\rho)\ge\frac{\lambda_\beta}{2} − χ Φ χ ( χ , ρ ) ≥ 2 λ θ , − ρ Φ ρ ( χ , ρ ) ≥ 2 λ β
throughout N \mathcal N N .
Now suppose
dist S ( θ , θ ∗ ) ≤ r , dist S ( β , β ∗ ) ≤ r , \operatorname{dist}_{S}(\theta,\theta^*)\le r,\qquad
\operatorname{dist}_{S}(\beta,\beta^*)\le r, dist S ( θ , θ ∗ ) ≤ r , dist S ( β , β ∗ ) ≤ r ,
with r > 0 r>0 r > 0 small enough so that ( χ , ρ ) ∈ N (\chi,\rho)\in\mathcal N ( χ , ρ ) ∈ N . Since u ⊥ θ u\perp\theta u ⊥ θ ,
∣ u ⋅ θ ∗ ∣ = ∣ u ⋅ P θ ⊥ θ ∗ ∣ ≤ ∥ u ∥ ∥ P θ ⊥ θ ∗ ∥ = ∥ u ∥ 1 − χ 2 ≤ r ∥ u ∥ . |u\cdot\theta^*|=|u\cdot P_{\theta^\perp}\theta^*|
\le\lVert u\rVert\lVert P_{\theta^\perp}\theta^*\rVert
=\lVert u\rVert\sqrt{1-\chi^2}\le r\lVert u\rVert. ∣ u ⋅ θ ∗ ∣ = ∣ u ⋅ P θ ⊥ θ ∗ ∣ ≤ ∥ u ∥ ∥ P θ ⊥ θ ∗ ∥ = ∥ u ∥ 1 − χ 2 ≤ r ∥ u ∥ .
Similarly,
∣ v ⋅ β ∗ ∣ ≤ r ∥ v ∥ . |v\cdot\beta^*|\le r\lVert v\rVert. ∣ v ⋅ β ∗ ∣ ≤ r ∥ v ∥ .
Hence the second-derivative terms are bounded in absolute value by
M r 2 ∥ u ∥ 2 + 2 M r 2 ∥ u ∥ ∥ v ∥ + M r 2 ∥ v ∥ 2 ≤ 2 M r 2 ( ∥ u ∥ 2 + ∥ v ∥ 2 ) . Mr^2\lVert u\rVert^2+2Mr^2\lVert u\rVert\lVert v\rVert+Mr^2\lVert v\rVert^2
\le2Mr^2(\lVert u\rVert^2+\lVert v\rVert^2). M r 2 ∥ u ∥ 2 + 2 M r 2 ∥ u ∥ ∥ v ∥ + M r 2 ∥ v ∥ 2 ≤ 2 M r 2 (∥ u ∥ 2 + ∥ v ∥ 2 ) .
Therefore,
Hess M L G ( θ , β ) [ ( u , v ) , ( u , v ) ] ≥ λ θ 2 ∥ u ∥ 2 + λ β 2 ∥ v ∥ 2 − 2 M r 2 ( ∥ u ∥ 2 + ∥ v ∥ 2 ) . \operatorname{Hess}_{\mathcal M} L_G(\theta,\beta)[(u,v),(u,v)]
\ge\frac{\lambda_\theta}{2}\lVert u\rVert^2+\frac{\lambda_\beta}{2}\lVert v\rVert^2
-2Mr^2(\lVert u\rVert^2+\lVert v\rVert^2). Hess M L G ( θ , β ) [( u , v ) , ( u , v )] ≥ 2 λ θ ∥ u ∥ 2 + 2 λ β ∥ v ∥ 2 − 2 M r 2 (∥ u ∥ 2 + ∥ v ∥ 2 ) .
Choose r G > 0 r_G>0 r G > 0 small enough that 2 M r G 2 ≤ 1 4 min { λ θ , λ β } 2Mr_G^2\le\frac14\min\{\lambda_\theta,\lambda_\beta\} 2 M r G 2 ≤ 4 1 min { λ θ , λ β } . Then, for every r ≤ r G r\le r_G r ≤ r G ,
Hess M L G ( θ , β ) [ ( u , v ) , ( u , v ) ] ≥ 1 4 min { λ θ , λ β } ( ∥ u ∥ 2 + ∥ v ∥ 2 ) . \operatorname{Hess}_{\mathcal M} L_G(\theta,\beta)[(u,v),(u,v)]
\ge\frac14\min\{\lambda_\theta,\lambda_\beta\}(\lVert u\rVert^2+\lVert v\rVert^2). Hess M L G ( θ , β ) [( u , v ) , ( u , v )] ≥ 4 1 min { λ θ , λ β } (∥ u ∥ 2 + ∥ v ∥ 2 ) .
Thus the lemma holds with
μ G : = 1 4 min { λ θ , λ β } > 0. \mu_G:=\frac14\min\{\lambda_\theta,\lambda_\beta\}>0. μ G := 4 1 min { λ θ , λ β } > 0.
Lemma C.11 (Teacher Hessian block formula). Let y ∗ : = ( θ ∗ , β ∗ ) y^* := (\theta^*, \beta^*) y ∗ := ( θ ∗ , β ∗ ) . Define
G i : = x ( i ) ⋅ β ∗ , S : = ∑ i = 1 d θ i ∗ ϕ ( G i ) . G_i := x^{(i)} \cdot\beta^*, \qquad S := \sum_{i=1}^{d} \theta_i^* \phi(G_i). G i := x ( i ) ⋅ β ∗ , S := i = 1 ∑ d θ i ∗ ϕ ( G i ) . For u ∈ T θ ∗ S d − 1 u \in T_{\theta^*}S^{d-1} u ∈ T θ ∗ S d − 1 , set
A u : = ∑ i = 1 d u i ϕ ( G i ) . A_u := \sum_{i=1}^{d} u_i \phi(G_i). A u := i = 1 ∑ d u i ϕ ( G i ) . Then, for u 1 , u 2 ∈ T θ ∗ S d − 1 u_1,u_2 \in T_{\theta^*}S^{d-1} u 1 , u 2 ∈ T θ ∗ S d − 1 and v 1 , v 2 ∈ T β ∗ S d − 1 v_1,v_2 \in T_{\beta^*}S^{d-1} v 1 , v 2 ∈ T β ∗ S d − 1 , the Hessian blocks of L L L at the teacher satisfy
H L , θ θ ( y ∗ ) [ u 1 , u 2 ] = 2 E [ ψ ′ ( S ) 2 A u 1 A u 2 ] , (66) H_{L,\theta\theta}(y^*)[u_1,u_2] = 2\mathbb{E}\left[\psi'(S)^2 A_{u_1} A_{u_2}\right], \tag*{(66)} H L , θ θ ( y ∗ ) [ u 1 , u 2 ] = 2 E [ ψ ′ ( S ) 2 A u 1 A u 2 ] , (66) H L , θ β ( y ∗ ) [ u , v ] = 0 , (67) H_{L,\theta\beta}(y^*)[u,v] = 0, \tag*{(67)} H L , θ β ( y ∗ ) [ u , v ] = 0 , (67) H L , β β ( y ∗ ) [ v 1 , v 2 ] = 2 ⟨ v 1 , v 2 ⟩ ∑ i = 1 d ( θ i ∗ ) 2 E [ ψ ′ ( S ) 2 ϕ ′ ( G i ) 2 ] . (68) H_{L,\beta\beta}(y^*)[v_1,v_2] = 2\langle v_1,v_2\rangle\sum_{i=1}^{d}(\theta_i^*)^2\mathbb{E}\left[\psi'(S)^2\phi'(G_i)^2\right]. \tag*{(68)} H L , β β ( y ∗ ) [ v 1 , v 2 ] = 2 ⟨ v 1 , v 2 ⟩ i = 1 ∑ d ( θ i ∗ ) 2 E [ ψ ′ ( S ) 2 ϕ ′ ( G i ) 2 ] . (68) Consequently, for every ( u , v ) ∈ T θ ∗ S d − 1 × T β ∗ S d − 1 (u,v) \in T_{\theta^*}S^{d-1} \times T_{\beta^*}S^{d-1} ( u , v ) ∈ T θ ∗ S d − 1 × T β ∗ S d − 1 ,
Hess M L ( y ∗ ) [ ( u , v ) , ( u , v ) ] = 2 E [ ψ ′ ( S ) 2 A u 2 ] + 2 ∥ v ∥ 2 ∑ i = 1 d ( θ i ∗ ) 2 E [ ψ ′ ( S ) 2 ϕ ′ ( G i ) 2 ] . (69) \operatorname{Hess}_{\mathcal{M}}L(y^*)[(u,v),(u,v)] = 2\mathbb{E}\left[\psi'(S)^2 A_u^2\right] + 2\lVert v\rVert^2\sum_{i=1}^{d}(\theta_i^*)^2\mathbb{E}\left[\psi'(S)^2\phi'(G_i)^2\right]. \tag*{(69)} Hess M L ( y ∗ ) [( u , v ) , ( u , v )] = 2 E [ ψ ′ ( S ) 2 A u 2 ] + 2 ∥ v ∥ 2 i = 1 ∑ d ( θ i ∗ ) 2 E [ ψ ′ ( S ) 2 ϕ ′ ( G i ) 2 ] . (69) Proof. We first compute the full quadratic form at the teacher and then polarize.
Let y t = ( θ t , β t ) y_t=(\theta_t,\beta_t) y t = ( θ t , β t ) be any product geodesic on M = S d − 1 × S d − 1 \mathcal{M}=S^{d-1}\times S^{d-1} M = S d − 1 × S d − 1 with
y 0 = y ∗ , y ˙ 0 = ( u , v ) . y_0=y^*, \qquad\dot{y}_0=(u,v). y 0 = y ∗ , y ˙ 0 = ( u , v ) . Write
f t ( x ) : = f β t , θ t ( x ) , f ∗ ( x ) : = f β ∗ , θ ∗ ( x ) . f_t(x):=f_{\beta_t,\theta_t}(x), \qquad f_*(x):=f_{\beta^*,\theta^*}(x). f t ( x ) := f β t , θ t ( x ) , f ∗ ( x ) := f β ∗ , θ ∗ ( x ) . Since f 0 = f ∗ f_0=f_* f 0 = f ∗ pointwise,
L ( y t ) = E [ ( f t − f ∗ ) 2 ] , L(y_t)=\mathbb{E}[(f_t-f_*)^2], L ( y t ) = E [( f t − f ∗ ) 2 ] , and therefore
d 2 d t 2 ∣ t = 0 L ( y t ) = 2 E [ f ˙ 0 2 ] + 2 E [ ( f 0 − f ∗ ) f ¨ 0 ] = 2 E [ f ˙ 0 2 ] . \left.\frac{d^2}{dt^2}\right|_{t=0}L(y_t)=2\mathbb{E}[\dot{f}_0^2]+2\mathbb{E}[(f_0-f_*)\ddot{f}_0]=2\mathbb{E}[\dot{f}_0^2]. d t 2 d 2 t = 0 L ( y t ) = 2 E [ f ˙ 0 2 ] + 2 E [( f 0 − f ∗ ) f ¨ 0 ] = 2 E [ f ˙ 0 2 ] . Because y t y_t y t is a geodesic, this second derivative is exactly the Riemannian Hessian:
Hess M L ( y ∗ ) [ ( u , v ) , ( u , v ) ] = 2 E [ f ˙ 0 2 ] . \operatorname{Hess}_{\mathcal{M}}L(y^*)[(u,v),(u,v)]=2\mathbb{E}[\dot{f}_0^2]. Hess M L ( y ∗ ) [( u , v ) , ( u , v )] = 2 E [ f ˙ 0 2 ] . Now compute f ˙ 0 \dot{f}_0 f ˙ 0 . At y ∗ y^* y ∗ ,
f t ( x ) = ψ ( ∑ i = 1 d θ t , i ϕ ( x ( i ) ⋅ β t ) ) . f_t(x)=\psi\left(\sum_{i=1}^{d}\theta_{t,i}\phi(x^{(i)}\cdot\beta_t)\right). f t ( x ) = ψ ( i = 1 ∑ d θ t , i ϕ ( x ( i ) ⋅ β t ) ) . Thus
f ˙ 0 = ψ ′ ( S ) [ ∑ i = 1 d u i ϕ ( G i ) + ∑ i = 1 d θ i ∗ ϕ ′ ( G i ) ( x ( i ) ⋅ v ) ] . \dot{f}_0=\psi'(S)\left[\sum_{i=1}^{d}u_i\phi(G_i)+\sum_{i=1}^{d}\theta_i^*\phi'(G_i)(x^{(i)}\cdot v)\right]. f ˙ 0 = ψ ′ ( S ) [ i = 1 ∑ d u i ϕ ( G i ) + i = 1 ∑ d θ i ∗ ϕ ′ ( G i ) ( x ( i ) ⋅ v ) ] . Define, for the proof,
B v : = ∑ i = 1 d θ i ∗ ϕ ′ ( G i ) H i ( v ) , H i ( v ) : = x ( i ) ⋅ v . B_v:=\sum_{i=1}^{d}\theta_i^*\phi'(G_i)H_i(v), \qquad H_i(v):=x^{(i)}\cdot v. B v := i = 1 ∑ d θ i ∗ ϕ ′ ( G i ) H i ( v ) , H i ( v ) := x ( i ) ⋅ v . Then
f ˙ 0 = ψ ′ ( S ) ( A u + B v ) , \dot{f}_0=\psi'(S)(A_u+B_v), f ˙ 0 = ψ ′ ( S ) ( A u + B v ) , and hence
Hess M L ( y ∗ ) [ ( u , v ) , ( u , v ) ] = 2 E [ ψ ′ ( S ) 2 ( A u + B v ) 2 ] . (70) \operatorname{Hess}_{\mathcal{M}}L(y^*)[(u,v),(u,v)]=2\mathbb{E}\left[\psi'(S)^2(A_u+B_v)^2\right]. \tag*{(70)} Hess M L ( y ∗ ) [( u , v ) , ( u , v )] = 2 E [ ψ ′ ( S ) 2 ( A u + B v ) 2 ] . (70)
Since the right-hand side of (68) is a quadratic form in ( u , v ) (u,v) ( u , v ) , its associated symmetric bilinear form is obtained by polarization:
Hess M L ( y ∗ ) [ ( u 1 , v 1 ) , ( u 2 , v 2 ) ] = 2 E [ ψ ′ ( S ) 2 ( A u 1 + B v 1 ) ( A u 2 + B v 2 ) ] . (71) \operatorname{Hess}_{M}L(y^*)[(u_1,v_1),(u_2,v_2)] = 2\mathbb{E}\left[\psi'(S)^2(Au_1+Bv_1)(Au_2+Bv_2)\right]. \tag*{(71)} Hess M L ( y ∗ ) [( u 1 , v 1 ) , ( u 2 , v 2 )] = 2 E [ ψ ′ ( S ) 2 ( A u 1 + B v 1 ) ( A u 2 + B v 2 ) ] . (71)
Now, we derive the three Hessian blocks: First, take v 1 = v 2 = 0 v_1=v_2=0 v 1 = v 2 = 0 in (69). This gives
H L , θ θ ( y ∗ ) [ u 1 , u 2 ] = 2 E [ ψ ′ ( S ) 2 A u 1 A u 2 ] , H_{L,\theta\theta}(y^*)[u_1,u_2] = 2\mathbb{E}\left[\psi'(S)^2Au_1Au_2\right], H L , θ θ ( y ∗ ) [ u 1 , u 2 ] = 2 E [ ψ ′ ( S ) 2 A u 1 A u 2 ] ,
which proves (64), the θ θ \theta\theta θ θ -block.
Second, take ( u 1 , v 1 ) = ( u , 0 ) (u_1,v_1)=(u,0) ( u 1 , v 1 ) = ( u , 0 ) and ( u 2 , v 2 ) = ( 0 , v ) (u_2,v_2)=(0,v) ( u 2 , v 2 ) = ( 0 , v ) . Then
H L , θ β ( y ∗ ) [ u , v ] = 2 E [ ψ ′ ( S ) 2 A u B v ] . H_{L,\theta\beta}(y^*)[u,v] = 2\mathbb{E}\left[\psi'(S)^2AuBv\right]. H L , θ β ( y ∗ ) [ u , v ] = 2 E [ ψ ′ ( S ) 2 A u B v ] .
We show this expectation is zero. Since v ∈ T β ∗ S d − 1 v\in T_{\beta^*}\mathbb{S}^{d-1} v ∈ T β ∗ S d − 1 , we have v ⊥ β ∗ v\perp\beta^* v ⊥ β ∗ . For each i i i , the pair
G i = x ( i ) ⋅ β ∗ , H i ( v ) = x ( i ) ⋅ v G_i = x^{(i)}\cdot\beta^*,\qquad H_i(v)=x^{(i)}\cdot v G i = x ( i ) ⋅ β ∗ , H i ( v ) = x ( i ) ⋅ v
is jointly Gaussian with covariance
E [ G i H i ( v ) ] = ⟨ β ∗ , v ⟩ = 0. \mathbb{E}[G_iH_i(v)] = \langle\beta^*,v\rangle=0. E [ G i H i ( v )] = ⟨ β ∗ , v ⟩ = 0.
Hence G i G_i G i and H i ( v ) H_i(v) H i ( v ) are independent. Moreover, conditional on ( G 1 , … , G d ) (G_1,\ldots,G_d) ( G 1 , … , G d ) , the variables H i ( v ) H_i(v) H i ( v ) are centered Gaussian and independent across i i i . Therefore
E [ B v ∣ G 1 , … , G d ] = ∑ i = 1 d θ i ∗ ϕ ′ ( G i ) E [ H i ( v ) ∣ G 1 , … , G d ] = 0. \mathbb{E}[B_v\mid G_1,\ldots,G_d] = \sum_{i=1}^{d}\theta_i^*\phi'(G_i)\mathbb{E}[H_i(v)\mid G_1,\ldots,G_d]=0. E [ B v ∣ G 1 , … , G d ] = i = 1 ∑ d θ i ∗ ϕ ′ ( G i ) E [ H i ( v ) ∣ G 1 , … , G d ] = 0.
Since ψ ′ ( S ) 2 A u \psi'(S)^2A_u ψ ′ ( S ) 2 A u is measurable with respect to σ ( G 1 , … , G d ) \sigma(G_1,\ldots,G_d) σ ( G 1 , … , G d ) , we obtain
E [ ψ ′ ( S ) 2 A u B v ] = E [ ψ ′ ( S ) 2 A u E [ B v ∣ G 1 , … , G d ] ] = 0. \mathbb{E}[\psi'(S)^2A_uB_v] = \mathbb{E}\left[\psi'(S)^2A_u\mathbb{E}[B_v\mid G_1,\ldots,G_d]\right]=0. E [ ψ ′ ( S ) 2 A u B v ] = E [ ψ ′ ( S ) 2 A u E [ B v ∣ G 1 , … , G d ] ] = 0.
Thus we have (65), the θ β \theta\beta θ β -block:
H L , θ β ( y ∗ ) [ u , v ] = 0. H_{L,\theta\beta}(y^*)[u,v]=0. H L , θ β ( y ∗ ) [ u , v ] = 0.
Finally, take u 1 = u 2 = 0 u_1=u_2=0 u 1 = u 2 = 0 in (69). Then
H L , β β ( y ∗ ) [ v 1 , v 2 ] = 2 E [ ψ ′ ( S ) 2 B v 1 B v 2 ] . H_{L,\beta\beta}(y^*)[v_1,v_2] = 2\mathbb{E}\left[\psi'(S)^2B_{v_1}B_{v_2}\right]. H L , β β ( y ∗ ) [ v 1 , v 2 ] = 2 E [ ψ ′ ( S ) 2 B v 1 B v 2 ] .
Conditioning again on G : = ( G 1 , … , G d ) G:=(G_1,\ldots,G_d) G := ( G 1 , … , G d ) ,
B v 1 B v 2 = ∑ i , j = 1 d θ i ∗ θ j ∗ ϕ ′ ( G i ) ϕ ′ ( G j ) H i ( v 1 ) H j ( v 2 ) . B_{v_1}B_{v_2}=\sum_{i,j=1}^{d}\theta_i^*\theta_j^*\phi'(G_i)\phi'(G_j)H_i(v_1)H_j(v_2). B v 1 B v 2 = i , j = 1 ∑ d θ i ∗ θ j ∗ ϕ ′ ( G i ) ϕ ′ ( G j ) H i ( v 1 ) H j ( v 2 ) .
For i ≠ j i\ne j i = j , conditional independence and centering give
E [ H i ( v 1 ) H j ( v 2 ) ∣ G ] = 0. \mathbb{E}[H_i(v_1)H_j(v_2)\mid G]=0. E [ H i ( v 1 ) H j ( v 2 ) ∣ G ] = 0.
For i = j i=j i = j , since v 1 , v 2 ⊥ β ∗ v_1,v_2\perp\beta^* v 1 , v 2 ⊥ β ∗ ,
E [ H i ( v 1 ) H i ( v 2 ) ∣ G ] = ⟨ v 1 , v 2 ⟩ . \mathbb{E}[H_i(v_1)H_i(v_2)\mid G]=\langle v_1,v_2\rangle. E [ H i ( v 1 ) H i ( v 2 ) ∣ G ] = ⟨ v 1 , v 2 ⟩ .
Hence
E [ B v 1 B v 2 ∣ G ] = ⟨ v 1 , v 2 ⟩ ∑ i = 1 d ( θ i ∗ ) 2 ϕ ′ ( G i ) 2 . \mathbb{E}[B_{v_1}B_{v_2}\mid G]=\langle v_1,v_2\rangle\sum_{i=1}^{d}(\theta_i^*)^2\phi'(G_i)^2. E [ B v 1 B v 2 ∣ G ] = ⟨ v 1 , v 2 ⟩ i = 1 ∑ d ( θ i ∗ ) 2 ϕ ′ ( G i ) 2 .
Because ψ ′ ( S ) 2 \psi'(S)^2 ψ ′ ( S ) 2 is G G G -measurable,
E [ ψ ′ ( S ) 2 B v 1 B v 2 ] = ⟨ v 1 , v 2 ⟩ ∑ i = 1 d ( θ i ∗ ) 2 E [ ψ ′ ( S ) 2 ϕ ′ ( G i ) 2 ] . \mathbb{E}[\psi'(S)^2B_{v_1}B_{v_2}]=\langle v_1,v_2\rangle\sum_{i=1}^{d}(\theta_i^*)^2\mathbb{E}[\psi'(S)^2\phi'(G_i)^2]. E [ ψ ′ ( S ) 2 B v 1 B v 2 ] = ⟨ v 1 , v 2 ⟩ i = 1 ∑ d ( θ i ∗ ) 2 E [ ψ ′ ( S ) 2 ϕ ′ ( G i ) 2 ] .
Therefore we have the final β β \beta\beta β β -block in (66):
H L , β β ( y ∗ ) [ v 1 , v 2 ] = 2 ⟨ v 1 , v 2 ⟩ ∑ i = 1 d ( θ i ∗ ) 2 E [ ψ ′ ( S ) 2 ϕ ′ ( G i ) 2 ] . H_{L,\beta\beta}(y^*)[v_1,v_2]=2\langle v_1,v_2\rangle\sum_{i=1}^{d}(\theta_i^*)^2\mathbb{E}[\psi'(S)^2\phi'(G_i)^2]. H L , β β ( y ∗ ) [ v 1 , v 2 ] = 2 ⟨ v 1 , v 2 ⟩ i = 1 ∑ d ( θ i ∗ ) 2 E [ ψ ′ ( S ) 2 ϕ ′ ( G i ) 2 ] .
Combining the θ θ \theta\theta θ θ -block, the vanishing mixed block, and the β β \beta\beta β β -block with u 1 = u 2 = u u_1=u_2=u u 1 = u 2 = u , v 1 = v 2 = v v_1=v_2=v v 1 = v 2 = v , gives the final quadratic formula. This completes the proof.
Stein inputs for the teacher Hessian. We next state the two blockwise teacher-level comparisons used below. Their proofs are deferred to Subsection E.1.
Lemma C.12 (Teacher Hessian comparison). Let
A ψ : = E [ ψ ′ ( G ) 2 ] , B ϕ : = E [ ϕ ′ ( G ) 2 ] = κ ′ ( 1 ) , λ θ : = 2 A ψ , λ β : = 2 A ψ B ϕ . A_{\psi}:=\mathbb{E}[\psi'(G)^2],\qquad B_{\phi}:=\mathbb{E}[\phi'(G)^2]=\kappa'(1),\qquad\lambda_{\theta}:=2A_{\psi},\qquad\lambda_{\beta}:=2A_{\psi}B_{\phi}. A ψ := E [ ψ ′ ( G ) 2 ] , B ϕ := E [ ϕ ′ ( G ) 2 ] = κ ′ ( 1 ) , λ θ := 2 A ψ , λ β := 2 A ψ B ϕ . Under the regularity assumptions of Lemma E.4 and Lemma E.6, there exists C ∗ > 0 C_{*}>0 C ∗ > 0 , independent of d d d , such that
sup ∥ ξ ∥ = 1 ∣ Hess M L ( y ∗ ) [ ξ , ξ ] − Hess M L G ( y ∗ ) [ ξ , ξ ] ∣ ≤ C ∗ ∥ θ ∗ ∥ ∞ . (72) \sup_{\lVert\xi\rVert=1}\left|\operatorname{Hess}_{\mathcal{M}}L(y^{*})[\xi,\xi]-\operatorname{Hess}_{\mathcal{M}}L_{G}(y^{*})[\xi,\xi]\right|\leq C_{*}\lVert\theta^{*}\rVert_{\infty}. \tag*{(72)} ∥ ξ ∥ = 1 sup ∣ Hess M L ( y ∗ ) [ ξ , ξ ] − Hess M L G ( y ∗ ) [ ξ , ξ ] ∣ ≤ C ∗ ∥ θ ∗ ∥ ∞ . (72) Moreover,
Hess M L G ( y ∗ ) [ ( u , v ) , ( u , v ) ] = λ θ ∥ u ∥ 2 + λ β ∥ v ∥ 2 . (73) \operatorname{Hess}_{\mathcal{M}}L_{G}(y^{*})[(u,v),(u,v)]=\lambda_{\theta}\lVert u\rVert^{2}+\lambda_{\beta}\lVert v\rVert^{2}. \tag*{(73)} Hess M L G ( y ∗ ) [( u , v ) , ( u , v )] = λ θ ∥ u ∥ 2 + λ β ∥ v ∥ 2 . (73) Proof. Let
ξ = ( u , v ) ∈ T y ∗ M , ∥ ξ ∥ 2 = ∥ u ∥ 2 + ∥ v ∥ 2 . \xi=(u,v)\in T_{y^{*}}\mathcal{M},\qquad\lVert\xi\rVert^{2}=\lVert u\rVert^{2}+\lVert v\rVert^{2}. ξ = ( u , v ) ∈ T y ∗ M , ∥ ξ ∥ 2 = ∥ u ∥ 2 + ∥ v ∥ 2 . First, the Gaussian teacher Hessian formula follows from the explicit Gaussian loss:
L G ( θ , β ) = 2 ∑ k ≥ 1 α k 2 ( 1 − ( κ ( ρ ) χ ) k ) . L_{G}(\theta,\beta)=2\sum_{k\geq1}\alpha_{k}^{2}\left(1-\bigl(\kappa(\rho)\chi\bigr)^{k}\right). L G ( θ , β ) = 2 k ≥ 1 ∑ α k 2 ( 1 − ( κ ( ρ ) χ ) k ) . At y ∗ y^{*} y ∗ , tangent geodesics satisfy
χ ′ ′ ( 0 ) = − ∥ u ∥ 2 , ρ ′ ′ ( 0 ) = − ∥ v ∥ 2 , χ ′ ( 0 ) = ρ ′ ( 0 ) = 0. \chi''(0)=-\lVert u\rVert^{2},\qquad\rho''(0)=-\lVert v\rVert^{2},\qquad\chi'(0)=\rho'(0)=0. χ ′′ ( 0 ) = − ∥ u ∥ 2 , ρ ′′ ( 0 ) = − ∥ v ∥ 2 , χ ′ ( 0 ) = ρ ′ ( 0 ) = 0. Therefore
Hess M L G ( y ∗ ) [ ( u , v ) , ( u , v ) ] = λ θ ∥ u ∥ 2 + λ β ∥ v ∥ 2 , \operatorname{Hess}_{\mathcal{M}}L_{G}(y^{*})[(u,v),(u,v)]=\lambda_{\theta}\lVert u\rVert^{2}+\lambda_{\beta}\lVert v\rVert^{2}, Hess M L G ( y ∗ ) [( u , v ) , ( u , v )] = λ θ ∥ u ∥ 2 + λ β ∥ v ∥ 2 , with
λ θ = 2 ∑ k ≥ 1 k α k 2 = 2 A ψ , λ β = 2 κ ′ ( 1 ) ∑ k ≥ 1 k α k 2 = 2 A ψ B ϕ . \lambda_{\theta}=2\sum_{k\geq1}k\alpha_{k}^{2}=2A_{\psi},\qquad\lambda_{\beta}=2\kappa'(1)\sum_{k\geq1}k\alpha_{k}^{2}=2A_{\psi}B_{\phi}. λ θ = 2 k ≥ 1 ∑ k α k 2 = 2 A ψ , λ β = 2 κ ′ ( 1 ) k ≥ 1 ∑ k α k 2 = 2 A ψ B ϕ . For the true loss, decompose the Hessian into blocks. By Lemma C.11 ,
H L , θ β ( y ∗ ) = 0. H_{L,\theta\beta}(y^{*})=0. H L , θ β ( y ∗ ) = 0. The same geodesic computation above gives
H L G , θ β ( y ∗ ) = 0. H_{L_{G},\theta\beta}(y^{*})=0. H L G , θ β ( y ∗ ) = 0. For the θ θ \theta\theta θ θ -block, Lemma E.4 at ( θ , β ) = ( θ ∗ , β ∗ ) (\theta,\beta)=(\theta^{*},\beta^{*}) ( θ , β ) = ( θ ∗ , β ∗ ) gives
∣ H L , θ θ ( y ∗ ) [ u , u ] − H L G , θ θ ( y ∗ ) [ u , u ] ∣ ≤ C θ ∥ θ ∗ ∥ ∞ ∥ u ∥ 2 . \left|H_{L,\theta\theta}(y^{*})[u,u]-H_{L_{G},\theta\theta}(y^{*})[u,u]\right|\leq C_{\theta}\lVert\theta^{*}\rVert_{\infty}\lVert u\rVert^{2}. ∣ H L , θ θ ( y ∗ ) [ u , u ] − H L G , θ θ ( y ∗ ) [ u , u ] ∣ ≤ C θ ∥ θ ∗ ∥ ∞ ∥ u ∥ 2 . For the β β \beta\beta β β -block, Lemma E.6 gives
∣ H L , β β ( y ∗ ) [ v , v ] − H L G , β β ( y ∗ ) [ v , v ] ∣ ≤ C β ∥ θ ∗ ∥ ∞ ∥ v ∥ 2 . \left|H_{L,\beta\beta}(y^{*})[v,v]-H_{L_{G},\beta\beta}(y^{*})[v,v]\right|\leq C_{\beta}\lVert\theta^{*}\rVert_{\infty}\lVert v\rVert^{2}. ∣ H L , β β ( y ∗ ) [ v , v ] − H L G , β β ( y ∗ ) [ v , v ] ∣ ≤ C β ∥ θ ∗ ∥ ∞ ∥ v ∥ 2 . Combining the three block estimates,
∣ H L ( y ∗ ) [ ξ , ξ ] − H L G ( y ∗ ) [ ξ , ξ ] ∣ ≤ ( C θ + C β ) ∥ θ ∗ ∥ ∞ ( ∥ u ∥ 2 + ∥ v ∥ 2 ) . \left|H_{L}(y^{*})[\xi,\xi]-H_{L_{G}}(y^{*})[\xi,\xi]\right|\leq(C_{\theta}+C_{\beta})\lVert\theta^{*}\rVert_{\infty}\left(\lVert u\rVert^{2}+\lVert v\rVert^{2}\right). ∣ H L ( y ∗ ) [ ξ , ξ ] − H L G ( y ∗ ) [ ξ , ξ ] ∣ ≤ ( C θ + C β ) ∥ θ ∗ ∥ ∞ ( ∥ u ∥ 2 + ∥ v ∥ 2 ) . Taking ∥ ξ ∥ = 1 \lVert\xi\rVert=1 ∥ ξ ∥ = 1 and setting C ∗ : = C θ + C β C_{*}:=C_{\theta}+C_{\beta} C ∗ := C θ + C β proves the claim.
Lemma C.13 (Uniform Hessian continuity near the teacher). Assume ψ ∈ C b 3 ( R ) \psi\in C_{b}^{3}(\mathbb{R}) ψ ∈ C b 3 ( R ) , and assume that ϕ ∈ C 3 ( R ) \phi\in C^{3}(\mathbb{R}) ϕ ∈ C 3 ( R ) satisfies
max 0 ≤ j ≤ 3 E [ ( 1 + ∣ G ∣ ) 12 ∣ ϕ ( j ) ( G ) ∣ 12 ] < ∞ , G ∼ N ( 0 , 1 ) . \max_{0\leq j\leq3}\mathbb{E}\left[(1+|G|)^{12}|\phi^{(j)}(G)|^{12}\right]<\infty,\qquad G\sim N(0,1). 0 ≤ j ≤ 3 max E [ ( 1 + ∣ G ∣ ) 12 ∣ ϕ ( j ) ( G ) ∣ 12 ] < ∞ , G ∼ N ( 0 , 1 ) . Assume also
E [ ϕ ( G ) ] = 0 , E [ ϕ ( G ) 2 ] = 1. \mathbb{E}[\phi(G)]=0,\qquad\mathbb{E}[\phi(G)^{2}]=1. E [ ϕ ( G )] = 0 , E [ ϕ ( G ) 2 ] = 1. Then there exist constants r 0 > 0 r_{0}>0 r 0 > 0 and C 3 > 0 C_{3}>0 C 3 > 0 , independent of d d d , such that for every y ∈ U r 0 y\in U_{r_{0}} y ∈ U r 0 ,
∥ P y → y ∗ Hess M L ( y ) P y ∗ → y − Hess M L ( y ∗ ) ∥ o p ≤ C 3 dist M ( y , y ∗ ) . (74) \left\|P_{y\to y^{*}}\operatorname{Hess}_{\mathcal{M}}L(y)P_{y^{*}\to y}-\operatorname{Hess}_{\mathcal{M}}L(y^{*})\right\|_{\mathrm{op}}\leq C_{3}\operatorname{dist}_{\mathcal{M}}(y,y^{*}). \tag*{(74)} ∥ P y → y ∗ Hess M L ( y ) P y ∗ → y − Hess M L ( y ∗ ) ∥ op ≤ C 3 dist M ( y , y ∗ ) . (74) Here P y → y ∗ P_{y\to y^{*}} P y → y ∗ denotes parallel transport along the minimizing product geodesic from y y y to y ∗ y^{*} y ∗ .
Proof. It is enough to prove that the covariant derivative of the Hessian is uniformly bounded on a fixed product cap. Indeed, let γ : [ 0 , ℓ ] → M \gamma: [0,\ell] \to\mathcal{M} γ : [ 0 , ℓ ] → M be the minimizing product geodesic from y ∗ y^* y ∗ to y y y , parametrized by arclength, so that
ℓ = dist M ( y , y ∗ ) . \ell= \operatorname{dist}_{\mathcal{M}}(y,y^*). ℓ = dist M ( y , y ∗ ) .
Let ξ 0 ∈ T y ∗ M \xi_0 \in T_{y^*}\mathcal{M} ξ 0 ∈ T y ∗ M be a unit vector, and let ξ t \xi_t ξ t be its parallel transport along γ \gamma γ . Then
d d t Hess M L ( γ ( t ) ) [ ξ t , ξ t ] = ( ∇ Hess M L ) ( γ ( t ) ) [ γ ˙ ( t ) , ξ t , ξ t ] . \frac{d}{dt}\operatorname{Hess}_{\mathcal{M}}L(\gamma(t))[\xi_t,\xi_t]
=
(\nabla\operatorname{Hess}_{\mathcal{M}}L)(\gamma(t))[\dot{\gamma}(t),\xi_t,\xi_t]. d t d Hess M L ( γ ( t )) [ ξ t , ξ t ] = ( ∇ Hess M L ) ( γ ( t )) [ γ ˙ ( t ) , ξ t , ξ t ] .
Thus, if
sup γ ( t ) ∈ U r 0 sup ∥ ζ ∥ = ∥ ξ ∥ = 1 ∣ ( ∇ Hess M L ) ( γ ( t ) ) [ ζ , ξ , ξ ] ∣ ≤ C 3 , \sup_{\gamma(t)\in U_{r_0}}\ \sup_{\|\zeta\|=\|\xi\|=1}
\left|(\nabla\operatorname{Hess}_{\mathcal{M}}L)(\gamma(t))[\zeta,\xi,\xi]\right|
\le C_3, γ ( t ) ∈ U r 0 sup ∥ ζ ∥ = ∥ ξ ∥ = 1 sup ∣ ( ∇ Hess M L ) ( γ ( t )) [ ζ , ξ , ξ ] ∣ ≤ C 3 ,
then integration along γ \gamma γ gives the desired transported-Hessian Lipschitz bound. We prove this uniform bound. Write
s y ( x ) : = ∑ i = 1 d θ i ϕ ( x ( i ) ⋅ β ) , s ∗ ( x ) : = ∑ i = 1 d θ i ∗ ϕ ( x ( i ) ⋅ β ∗ ) , s_y(x):=\sum_{i=1}^{d}\theta_i\phi(x^{(i)}\cdot\beta),
\qquad
s_*(x):=\sum_{i=1}^{d}\theta_i^*\phi(x^{(i)}\cdot\beta^*), s y ( x ) := i = 1 ∑ d θ i ϕ ( x ( i ) ⋅ β ) , s ∗ ( x ) := i = 1 ∑ d θ i ∗ ϕ ( x ( i ) ⋅ β ∗ ) ,
so that
L ( y ) = E [ ( ψ ( s y ( x ) ) − ψ ( s ∗ ( x ) ) ) 2 ] . L(y)=\mathbb{E}\left[\left(\psi(s_y(x))-\psi(s_*(x))\right)^2\right]. L ( y ) = E [ ( ψ ( s y ( x )) − ψ ( s ∗ ( x )) ) 2 ] .
Let y t = ( θ t , β t ) y_t=(\theta_t,\beta_t) y t = ( θ t , β t ) be any unit-speed product geodesic contained in U r 0 U_{r_0} U r 0 , and set
s t : = s y t ( x ) . s_t:=s_{y_t}(x). s t := s y t ( x ) .
We first record the basic moment estimate
sup t ( ∥ s ˙ t ∥ L 6 + ∥ s ¨ t ∥ L 3 + ∥ s t ( 3 ) ∥ L 1 ) ≤ C , (75) \sup_t\left(\|\dot{s}_t\|_{L^6}+\|\ddot{s}_t\|_{L^3}+\|s_t^{(3)}\|_{L^1}\right)\le C, \tag*{(75)} t sup ( ∥ s ˙ t ∥ L 6 + ∥ s ¨ t ∥ L 3 + ∥ s t ( 3 ) ∥ L 1 ) ≤ C , (75)
where C C C is independent of d d d and of the chosen unit-speed product geodesic. To prove (73), observe that
s t = ∑ i = 1 d θ t , i ϕ ( U t , i ) , U t , i : = x ( i ) ⋅ β t . s_t=\sum_{i=1}^{d}\theta_{t,i}\phi(U_{t,i}),
\qquad
U_{t,i}:=x^{(i)}\cdot\beta_t. s t = i = 1 ∑ d θ t , i ϕ ( U t , i ) , U t , i := x ( i ) ⋅ β t .
Along a unit-speed spherical geodesic,
∥ θ ˙ t ∥ + ∥ θ ¨ t ∥ + ∥ θ t ( 3 ) ∥ ≤ C , ∥ β ˙ t ∥ + ∥ β ¨ t ∥ + ∥ β t ( 3 ) ∥ ≤ C . \|\dot{\theta}_t\|+\|\ddot{\theta}_t\|+\|\theta_t^{(3)}\|\le C,
\qquad
\|\dot{\beta}_t\|+\|\ddot{\beta}_t\|+\|\beta_t^{(3)}\|\le C. ∥ θ ˙ t ∥ + ∥ θ ¨ t ∥ + ∥ θ t ( 3 ) ∥ ≤ C , ∥ β ˙ t ∥ + ∥ β ¨ t ∥ + ∥ β t ( 3 ) ∥ ≤ C .
Moreover,
U ˙ t , i = x ( i ) ⋅ β ˙ t , U ¨ t , i = x ( i ) ⋅ β ¨ t , U t , i ( 3 ) = x ( i ) ⋅ β t ( 3 ) . \dot{U}_{t,i}=x^{(i)}\cdot\dot{\beta}_t,
\qquad
\ddot{U}_{t,i}=x^{(i)}\cdot\ddot{\beta}_t,
\qquad
U_{t,i}^{(3)}=x^{(i)}\cdot\beta_t^{(3)}. U ˙ t , i = x ( i ) ⋅ β ˙ t , U ¨ t , i = x ( i ) ⋅ β ¨ t , U t , i ( 3 ) = x ( i ) ⋅ β t ( 3 ) .
The variables coming from distinct blocks i i i are independent, and each block has moments bounded uniformly in t t t . Differentiating θ t , i ϕ ( U t , i ) \theta_{t,i}\phi(U_{t,i}) θ t , i ϕ ( U t , i ) up to order three gives a finite sum of terms whose coefficient vectors have uniformly bounded ℓ 2 \ell_2 ℓ 2 -norm and whose block variables have uniformly bounded moments by the assumptions on ϕ \phi ϕ . The only point that needs care is centering. For each i i i ,
E [ ϕ ( U t , i ) ] = E [ ϕ ( G ) ] = 0 \mathbb{E}[\phi(U_{t,i})]=\mathbb{E}[\phi(G)]=0 E [ ϕ ( U t , i )] = E [ ϕ ( G )] = 0
for every t t t , because U t , i ∼ N ( 0 , 1 ) U_{t,i}\sim N(0,1) U t , i ∼ N ( 0 , 1 ) . Differentiating this identity in t t t shows that the differentiated block variables appearing in s ˙ t \dot{s}_t s ˙ t , s ¨ t \ddot{s}_t s ¨ t , s t ( 3 ) s_t^{(3)} s t ( 3 ) are centered. Therefore Rosenthal’s inequality applies to each differentiated sum and gives the bounds in (73), uniformly in d d d .
Now define
R t : = ψ ( s t ) − ψ ( s ∗ ) . R_t:=\psi(s_t)-\psi(s_*). R t := ψ ( s t ) − ψ ( s ∗ ) .
Since ψ \psi ψ and its derivatives up to order three are bounded,
R ˙ t = ψ ′ ( s t ) s ˙ t . \dot{R}_t=\psi'(s_t)\dot{s}_t. R ˙ t = ψ ′ ( s t ) s ˙ t .
R ¨ t = ψ ′ ′ ( s t ) s ˙ t 2 + ψ ′ ( s t ) s ¨ t , \ddot{R}_t=\psi''(s_t)\dot{s}_t^2+\psi'(s_t)\ddot{s}_t, R ¨ t = ψ ′′ ( s t ) s ˙ t 2 + ψ ′ ( s t ) s ¨ t ,
and
R t ( 3 ) = ψ ( 3 ) ( s t ) s ˙ t 3 + 3 ψ ′ ′ ( s t ) s ˙ t s ¨ t + ψ ′ ( s t ) s t ( 3 ) . R_t^{(3)}=\psi^{(3)}(s_t)\dot{s}_t^3+3\psi''(s_t)\dot{s}_t\ddot{s}_t+\psi'(s_t)s_t^{(3)}. R t ( 3 ) = ψ ( 3 ) ( s t ) s ˙ t 3 + 3 ψ ′′ ( s t ) s ˙ t s ¨ t + ψ ′ ( s t ) s t ( 3 ) .
Since L ( y t ) = E [ R t 2 ] L(y_t)=\mathbb{E}[R_t^2] L ( y t ) = E [ R t 2 ] ,
d 3 d t 3 L ( y t ) = 2 E [ 3 R ˙ t R ¨ t + R t R t ( 3 ) ] . \frac{d^3}{dt^3}L(y_t)=2\mathbb{E}\left[3\dot{R}_t\ddot{R}_t+R_tR_t^{(3)}\right]. d t 3 d 3 L ( y t ) = 2 E [ 3 R ˙ t R ¨ t + R t R t ( 3 ) ] .
Using the boundedness of ψ , ψ ′ , ψ ′ ′ , ψ ( 3 ) \psi,\psi',\psi'',\psi^{(3)} ψ , ψ ′ , ψ ′′ , ψ ( 3 ) , together with (73) and Hölder’s inequality, we obtain
sup t ∣ d 3 d t 3 L ( y t ) ∣ ≤ C . \sup_t\left|\frac{d^3}{dt^3}L(y_t)\right|\le C. t sup d t 3 d 3 L ( y t ) ≤ C .
The same argument, applied to geodesic variations and then polarized, gives
sup y ∈ U r 0 sup ∥ ζ ∥ = ∥ ξ ∥ = 1 ∣ ( ∇ Hess M L ) ( y ) [ ζ , ξ , ξ ] ∣ ≤ C . \sup_{y\in U_{r_0}}\sup_{\|\zeta\|=\|\xi\|=1}\left|(\nabla\operatorname{Hess}_{\mathcal M}L)(y)[\zeta,\xi,\xi]\right|\le C. y ∈ U r 0 sup ∥ ζ ∥ = ∥ ξ ∥ = 1 sup ∣ ( ∇ Hess M L ) ( y ) [ ζ , ξ , ξ ] ∣ ≤ C .
Equivalently, the transported Hessian is C C C -Lipschitz along every minimizing product geodesic in U r 0 U_{r_0} U r 0 . Integrating this bound along the geodesic from y ∗ y^* y ∗ to y y y gives
∥ P y → y ∗ Hess M L ( y ) P y ∗ → y − Hess M L ( y ∗ ) ∥ o p ≤ C dist M ( y , y ∗ ) . \left\|P_{y\to y^*}\operatorname{Hess}_{\mathcal M}L(y)P_{y^*\to y}-\operatorname{Hess}_{\mathcal M}L(y^*)\right\|_{\mathrm{op}}\le C\operatorname{dist}_{\mathcal M}(y,y^*). ∥ P y → y ∗ Hess M L ( y ) P y ∗ → y − Hess M L ( y ∗ ) ∥ op ≤ C dist M ( y , y ∗ ) .
This proves the lemma, with C 3 = C C_3=C C 3 = C .
Lemma C.14 (True local product strong convexity). Assume the teacher Hessian comparison of Lemma C.12 and the uniform Hessian continuity of Lemma C.13. On the random-spherical delocalization event of Lemma C.1, so that
∥ θ ∗ ∥ ∞ ≤ C K log d d , \|\theta^*\|_\infty\le C_K\sqrt{\frac{\log d}{d}}, ∥ θ ∗ ∥ ∞ ≤ C K d log d , there exist constants r l o c > 0 r_{\mathrm{loc}}>0 r loc > 0 and μ l o c > 0 \mu_{\mathrm{loc}}>0 μ loc > 0 , independent of d d d , such that for all sufficiently large d d d and every y ∈ U r l o c y\in U_{r_{\mathrm{loc}}} y ∈ U r loc ,
Hess M L ( y ) ⪰ μ l o c I . (76) \operatorname{Hess}_{\mathcal M}L(y)\succeq\mu_{\mathrm{loc}}I. \tag*{(76)} Hess M L ( y ) ⪰ μ loc I . (76) Proof. Let
λ 0 : = min { λ θ , λ β } = 2 A ψ min { 1 , B ϕ } . \lambda_0:=\min\{\lambda_\theta,\lambda_\beta\}=2A_\psi\min\{1,B_\phi\}. λ 0 := min { λ θ , λ β } = 2 A ψ min { 1 , B ϕ } . By the information-exponent-one assumptions, A ψ > 0 A_\psi>0 A ψ > 0 and B ϕ > 0 B_\phi>0 B ϕ > 0 , so λ 0 > 0 \lambda_0>0 λ 0 > 0 .
By Lemma C.12, for every ξ ∈ T y ∗ M \xi\in T_{y^*}\mathcal M ξ ∈ T y ∗ M ,
Hess M L ( y ∗ ) [ ξ , ξ ] ≥ Hess M L G ( y ∗ ) [ ξ , ξ ] − C ∗ ∥ θ ∗ ∥ ∞ ∥ ξ ∥ 2 . \operatorname{Hess}_{\mathcal M}L(y^*)[\xi,\xi]\ge\operatorname{Hess}_{\mathcal M}L_G(y^*)[\xi,\xi]-C_*\|\theta^*\|_\infty\|\xi\|^2. Hess M L ( y ∗ ) [ ξ , ξ ] ≥ Hess M L G ( y ∗ ) [ ξ , ξ ] − C ∗ ∥ θ ∗ ∥ ∞ ∥ ξ ∥ 2 . Since by Lemma C.10
Hess M L G ( y ∗ ) [ ξ , ξ ] ≥ λ 0 ∥ ξ ∥ 2 , \operatorname{Hess}_{\mathcal M}L_G(y^*)[\xi,\xi]\ge\lambda_0\|\xi\|^2, Hess M L G ( y ∗ ) [ ξ , ξ ] ≥ λ 0 ∥ ξ ∥ 2 , we obtain
Hess M L ( y ∗ ) [ ξ , ξ ] ≥ ( λ 0 − C ∗ ∥ θ ∗ ∥ ∞ ) ∥ ξ ∥ 2 . \operatorname{Hess}_{\mathcal M}L(y^*)[\xi,\xi]\ge(\lambda_0-C_*\|\theta^*\|_\infty)\|\xi\|^2. Hess M L ( y ∗ ) [ ξ , ξ ] ≥ ( λ 0 − C ∗ ∥ θ ∗ ∥ ∞ ) ∥ ξ ∥ 2 . By Lemma C.1,
∥ θ ∗ ∥ ∞ ≤ C K log d d . \|\theta^*\|_\infty\le C_K\sqrt{\frac{\log d}{d}}. ∥ θ ∗ ∥ ∞ ≤ C K d log d . Hence, for all sufficiently large d d d ,
C ∗ ∥ θ ∗ ∥ ∞ ≤ C ∗ C K log d d ≤ λ 0 4 . C_*\|\theta^*\|_\infty\le C_*C_K\sqrt{\frac{\log d}{d}}\le\frac{\lambda_0}{4}. C ∗ ∥ θ ∗ ∥ ∞ ≤ C ∗ C K d log d ≤ 4 λ 0 . Hence
Hess M L ( y ∗ ) ⪰ 3 λ 0 4 I . \operatorname{Hess}_{\mathcal M}L(y^*)\succeq\frac{3\lambda_0}{4}I. Hess M L ( y ∗ ) ⪰ 4 3 λ 0 I . Now choose
r l o c ≤ r 0 and C 3 r l o c ≤ λ 0 4 . r_{\mathrm{loc}}\le r_0\qquad\text{and}\qquad C_3r_{\mathrm{loc}}\le\frac{\lambda_0}{4}. r loc ≤ r 0 and C 3 r loc ≤ 4 λ 0 .
Let y ∈ U r l o c y \in\mathcal{U}_{r_{\mathrm{loc}}} y ∈ U r loc , and let ξ ∈ T y M \xi\in T_y\mathcal{M} ξ ∈ T y M be a unit tangent vector. Parallel transport ξ \xi ξ to T y ∗ M T_{y^*}\mathcal{M} T y ∗ M , writing
ξ ~ : = P y → y ∗ ξ . \widetilde{\xi} := P_{y \to y^*}\xi. ξ := P y → y ∗ ξ .
Then ∥ ξ ~ ∥ = 1 \|\widetilde{\xi}\| = 1 ∥ ξ ∥ = 1 . By Lemma C.13 ,
Hess M L ( y ) [ ξ , ξ ] ≥ Hess M L ( y ∗ ) [ ξ ~ , ξ ~ ] − C 3 dist M ( y , y ∗ ) . \operatorname{Hess}_{\mathcal{M}} L(y)[\xi,\xi] \geq\operatorname{Hess}_{\mathcal{M}} L(y^*)[\widetilde{\xi},\widetilde{\xi}] - C_3\operatorname{dist}_{\mathcal{M}}(y,y^*). Hess M L ( y ) [ ξ , ξ ] ≥ Hess M L ( y ∗ ) [ ξ , ξ ] − C 3 dist M ( y , y ∗ ) .
Since y ∈ U r l o c y \in\mathcal{U}_{r_{\mathrm{loc}}} y ∈ U r loc ,
dist M ( y , y ∗ ) ≤ r l o c . \operatorname{dist}_{\mathcal{M}}(y,y^*) \leq r_{\mathrm{loc}}. dist M ( y , y ∗ ) ≤ r loc .
Therefore
Hess M L ( y ) [ ξ , ξ ] ≥ 3 λ 0 4 − C 3 r l o c ≥ λ 0 2 . \operatorname{Hess}_{\mathcal{M}} L(y)[\xi,\xi] \geq\frac{3\lambda_0}{4} - C_3r_{\mathrm{loc}} \geq\frac{\lambda_0}{2}. Hess M L ( y ) [ ξ , ξ ] ≥ 4 3 λ 0 − C 3 r loc ≥ 2 λ 0 .
Thus the lemma holds with
μ l o c : = λ 0 2 . \mu_{\mathrm{loc}} := \frac{\lambda_0}{2}. μ loc := 2 λ 0 .
Strong Recovery Dynamics
Lemma C.15 (Local contraction under product strong convexity). Let M = S d − 1 × S d − 1 \mathcal{M} = \mathbb{S}^{d-1} \times\mathbb{S}^{d-1} M = S d − 1 × S d − 1 equipped with the product Riemannian metric, and let y ∗ = ( θ ∗ , β ∗ ) ∈ M y^* = (\theta^*,\beta^*) \in\mathcal{M} y ∗ = ( θ ∗ , β ∗ ) ∈ M . Fix r ∈ ( 0 , π / 2 ) r \in(0,\pi/2) r ∈ ( 0 , π /2 ) and suppose L : M → R L : \mathcal{M} \to\mathbb{R} L : M → R is C 2 C^2 C 2 and satisfies
Hess M L ( y ) ⪰ μ I for all y ∈ U r : = { y ∈ M : dist M ( y , y ∗ ) ≤ r } , \operatorname{Hess}_{\mathcal{M}} L(y) \succeq\mu I \qquad\text{for all } y \in\mathcal{U}_r := \{y \in\mathcal{M} : \operatorname{dist}_{\mathcal{M}}(y,y^*) \leq r\}, Hess M L ( y ) ⪰ μ I for all y ∈ U r := { y ∈ M : dist M ( y , y ∗ ) ≤ r } , for some μ > 0 \mu> 0 μ > 0 (Lemma C.14, with r = r l o c r = r_{\mathrm{loc}} r = r loc and μ = μ l o c \mu= \mu_{\mathrm{loc}} μ = μ loc ), and that y ∗ y^* y ∗ is a critical point of L L L , i.e., ∇ M L ( y ∗ ) = 0 \nabla_{\mathcal{M}}L(y^*) = 0 ∇ M L ( y ∗ ) = 0 .
Let y t y_t y t be the population gradient flow
y ˙ t = − ∇ M L ( y t ) , y t 0 ∈ M , dist M ( y t 0 , y ∗ ) ≤ r / 2. \dot{y}_t = -\nabla_{\mathcal{M}}L(y_t), \qquad y_{t_0} \in\mathcal{M}, \qquad\operatorname{dist}_{\mathcal{M}}(y_{t_0},y^*) \leq r/2. y ˙ t = − ∇ M L ( y t ) , y t 0 ∈ M , dist M ( y t 0 , y ∗ ) ≤ r /2. Then y t ∈ U r y_t \in\mathcal{U}_r y t ∈ U r for all t ≥ t 0 t \geq t_0 t ≥ t 0 , and
dist M ( y t , y ∗ ) 2 ≤ e − 2 μ ( t − t 0 ) dist M ( y t 0 , y ∗ ) 2 . (77) \operatorname{dist}_{\mathcal{M}}(y_t,y^*)^2 \leq e^{-2\mu(t-t_0)}\operatorname{dist}_{\mathcal{M}}(y_{t_0},y^*)^2. \tag*{(77)} dist M ( y t , y ∗ ) 2 ≤ e − 2 μ ( t − t 0 ) dist M ( y t 0 , y ∗ ) 2 . (77)
Proof. Write
D t : = dist M ( y t , y ∗ ) , E t : = 1 2 D t 2 . D_t := \operatorname{dist}_{\mathcal{M}}(y_t,y^*), \qquad E_t := \frac{1}{2}D_t^2. D t := dist M ( y t , y ∗ ) , E t := 2 1 D t 2 .
Since r < π / 2 r < \pi/2 r < π /2 , every point y = ( θ , β ) ∈ U r y = (\theta,\beta) \in\mathcal{U}_r y = ( θ , β ) ∈ U r satisfies
dist S ( θ , θ ∗ ) < π / 2 , dist S ( β , β ∗ ) < π / 2. \operatorname{dist}_{\mathbb{S}}(\theta,\theta^*) < \pi/2, \qquad\operatorname{dist}_{\mathbb{S}}(\beta,\beta^*) < \pi/2. dist S ( θ , θ ∗ ) < π /2 , dist S ( β , β ∗ ) < π /2.
Thus neither component is antipodal to its target. Hence the minimizing geodesic from y y y to y ∗ y^* y ∗ is unique and is obtained by taking the componentwise minimizing geodesics on the two spheres. In particular, Log y ( y ∗ ) \operatorname{Log}_y(y^*) Log y ( y ∗ ) is well-defined, and
∥ Log y ( y ∗ ) ∥ = dist M ( y , y ∗ ) . \|\operatorname{Log}_y(y^*)\| = \operatorname{dist}_{\mathcal{M}}(y,y^*). ∥ Log y ( y ∗ ) ∥ = dist M ( y , y ∗ ) .
Moreover, if γ : [ 0 , 1 ] → M \gamma: [0,1] \to\mathcal{M} γ : [ 0 , 1 ] → M is this minimizing product geodesic from y y y to y ∗ y^* y ∗ , then
dist M ( γ ( s ) , y ∗ ) = ( 1 − s ) dist M ( y , y ∗ ) , \operatorname{dist}_{\mathcal{M}}(\gamma(s),y^*) = (1-s)\operatorname{dist}_{\mathcal{M}}(y,y^*), dist M ( γ ( s ) , y ∗ ) = ( 1 − s ) dist M ( y , y ∗ ) ,
so γ ( s ) ∈ U r \gamma(s) \in\mathcal{U}_r γ ( s ) ∈ U r whenever y ∈ U r y \in\mathcal{U}_r y ∈ U r .
Define the exit time
τ : = inf { t ≥ t 0 : D t > r } . \tau:= \inf\{t \geq t_0 : D_t > r\}. τ := inf { t ≥ t 0 : D t > r } .
For t ∈ [ t 0 , τ ) t \in[t_0,\tau) t ∈ [ t 0 , τ ) , we have y t ∈ U r y_t \in\mathcal{U}_r y t ∈ U r . We will prove
d d t E t ≤ − 2 μ E t for all t ∈ [ t 0 , τ ) . \frac{d}{dt}E_t \leq-2\mu E_t \qquad\text{for all } t \in[t_0,\tau). d t d E t ≤ − 2 μ E t for all t ∈ [ t 0 , τ ) .
Once this is shown, Gronwall gives
E t ≤ e − 2 μ ( t − t 0 ) E t 0 for all t ∈ [ t 0 , τ ) . E_t \leq e^{-2\mu(t-t_0)}E_{t_0} \qquad\text{for all } t \in[t_0,\tau). E t ≤ e − 2 μ ( t − t 0 ) E t 0 for all t ∈ [ t 0 , τ ) .
Since D t 0 ≤ r / 2 D_{t_0} \le r/2 D t 0 ≤ r /2 , this implies D t ≤ r / 2 D_t \le r/2 D t ≤ r /2 on [ t 0 , τ ) [t_0,\tau) [ t 0 , τ ) . If τ < ∞ \tau< \infty τ < ∞ , then by continuity D τ ≤ r / 2 < r D_\tau\le r/2 < r D τ ≤ r /2 < r , contradicting the definition of τ \tau τ . Hence τ = ∞ \tau= \infty τ = ∞ , and the trajectory stays in U r U_r U r for all t ≥ t 0 t \ge t_0 t ≥ t 0 .
It remains to prove the differential inequality. Fix t < τ t < \tau t < τ , and let γ : [ 0 , 1 ] → M \gamma: [0,1] \to\mathcal{M} γ : [ 0 , 1 ] → M be the minimizing geodesic from y t y_t y t to y ∗ y^* y ∗ , parametrized with constant speed:
γ ( 0 ) = y t , γ ( 1 ) = y ∗ , ∥ γ ′ ( s ) ∥ ≡ D t . \gamma(0) = y_t,\qquad\gamma(1) = y^*,\qquad\lVert\gamma'(s)\rVert\equiv D_t. γ ( 0 ) = y t , γ ( 1 ) = y ∗ , ∥ γ ′ ( s )∥ ≡ D t .
Then
γ ′ ( 0 ) = Log y t ( y ∗ ) . \gamma'(0) = \operatorname{Log}_{y_t}(y^*). γ ′ ( 0 ) = Log y t ( y ∗ ) .
Set
φ ( s ) : = L ( γ ( s ) ) . \varphi(s) := L(\gamma(s)). φ ( s ) := L ( γ ( s )) .
Since γ \gamma γ is a geodesic,
φ ′ ′ ( s ) = Hess M L ( γ ( s ) ) [ γ ′ ( s ) , γ ′ ( s ) ] . \varphi''(s) = \operatorname{Hess}_{\mathcal{M}}L(\gamma(s))[\gamma'(s),\gamma'(s)]. φ ′′ ( s ) = Hess M L ( γ ( s )) [ γ ′ ( s ) , γ ′ ( s )] .
The geodesic remains in U r U_r U r , so the Hessian lower bound gives
φ ′ ′ ( s ) ≥ μ ∥ γ ′ ( s ) ∥ 2 = μ D t 2 . \varphi''(s) \ge\mu\lVert\gamma'(s)\rVert^2 = \mu D_t^2. φ ′′ ( s ) ≥ μ ∥ γ ′ ( s ) ∥ 2 = μ D t 2 .
Integrating from 0 0 0 to 1 1 1 ,
φ ′ ( 1 ) − φ ′ ( 0 ) ≥ μ D t 2 . \varphi'(1) - \varphi'(0) \ge\mu D_t^2. φ ′ ( 1 ) − φ ′ ( 0 ) ≥ μ D t 2 .
Because y ∗ y^* y ∗ is critical,
φ ′ ( 1 ) = ⟨ ∇ M L ( y ∗ ) , γ ′ ( 1 ) ⟩ = 0. \varphi'(1) = \langle\nabla_{\mathcal{M}}L(y^*),\gamma'(1)\rangle= 0. φ ′ ( 1 ) = ⟨ ∇ M L ( y ∗ ) , γ ′ ( 1 )⟩ = 0.
Therefore
φ ′ ( 0 ) ≤ − μ D t 2 . \varphi'(0) \le-\mu D_t^2. φ ′ ( 0 ) ≤ − μ D t 2 .
But
φ ′ ( 0 ) = ⟨ ∇ M L ( y t ) , Log y t ( y ∗ ) ⟩ . \varphi'(0) = \langle\nabla_{\mathcal{M}}L(y_t),\operatorname{Log}_{y_t}(y^*)\rangle. φ ′ ( 0 ) = ⟨ ∇ M L ( y t ) , Log y t ( y ∗ )⟩ .
Thus
⟨ ∇ M L ( y t ) , Log y t ( y ∗ ) ⟩ ≤ − μ D t 2 . \langle\nabla_{\mathcal{M}}L(y_t),\operatorname{Log}_{y_t}(y^*)\rangle\le-\mu D_t^2. ⟨ ∇ M L ( y t ) , Log y t ( y ∗ )⟩ ≤ − μ D t 2 .
Finally, the first-variation formula for the squared distance gives
d d t E t = − ⟨ y ˙ t , Log y t ( y ∗ ) ⟩ . \frac{d}{dt}E_t = -\langle\dot{y}_t,\operatorname{Log}_{y_t}(y^*)\rangle. d t d E t = − ⟨ y ˙ t , Log y t ( y ∗ )⟩ .
Since y ˙ t = − ∇ M L ( y t ) \dot{y}_t = -\nabla_{\mathcal{M}}L(y_t) y ˙ t = − ∇ M L ( y t ) ,
d d t E t = ⟨ ∇ M L ( y t ) , Log y t ( y ∗ ) ⟩ ≤ − μ D t 2 = − 2 μ E t . \frac{d}{dt}E_t = \langle\nabla_{\mathcal{M}}L(y_t),\operatorname{Log}_{y_t}(y^*)\rangle\le-\mu D_t^2 = -2\mu E_t. d t d E t = ⟨ ∇ M L ( y t ) , Log y t ( y ∗ )⟩ ≤ − μ D t 2 = − 2 μ E t .
This proves the desired differential inequality, and hence the contraction estimate.
Theorem C.16 (Continuous-time strong recovery). Assume the weak-recovery conclusion of Lemma C.8, together with the assumptions of Lemmas C.3 and C.4. Assume also the regularity and incoherence assumptions under which Lemmas C.12, C.13, C.14, and C.15 hold.
Let y t : = ( θ t , β t ) ∈ M : = S d − 1 × S d − 1 y_t := (\theta_t,\beta_t) \in\mathcal{M} := \mathbb{S}^{d-1} \times\mathbb{S}^{d-1} y t := ( θ t , β t ) ∈ M := S d − 1 × S d − 1 , y ∗ : = ( θ ∗ , β ∗ ) y^* := (\theta^*,\beta^*) y ∗ := ( θ ∗ , β ∗ ) , and write R t : = dist M ( y t , y ∗ ) R_t := \operatorname{dist}_{\mathcal{M}}(y_t,y^*) R t := dist M ( y t , y ∗ ) . Then there exist constants r l o c > 0 r_{\mathrm{loc}} > 0 r loc > 0 , μ l o c > 0 \mu_{\mathrm{loc}} > 0 μ loc > 0 , C l o c < ∞ C_{\mathrm{loc}} < \infty C loc < ∞ , independent of d d d , such that for all sufficiently large d d d , on the favorable initialization event of (A4), there exists a time t l o c ≤ C l o c log d t_{\mathrm{loc}} \le C_{\mathrm{loc}}\log d t loc ≤ C loc log d for which
R t l o c 2 ≤ r l o c 2 8 . (78) R_{t_{\mathrm{loc}}}^2 \le\frac{r_{\mathrm{loc}}^2}{8}. \tag*{(78)} R t loc 2 ≤ 8 r loc 2 . (78) Moreover, for every t ≥ t l o c t \ge t_{\mathrm{loc}} t ≥ t loc ,
R t 2 ≤ e − 2 μ l o c ( t − t l o c ) R t l o c 2 . R_t^2 \le e^{-2\mu_{\mathrm{loc}}(t-t_{\mathrm{loc}})}R_{t_{\mathrm{loc}}}^2. R t 2 ≤ e − 2 μ loc ( t − t loc ) R t loc 2 . Consequently,
( 1 − χ t ) + ( 1 − ρ t ) ≤ r l o c 2 16 e − 2 μ l o c ( t − t l o c ) . (1-\chi_t) + (1-\rho_t) \le\frac{r_{\mathrm{loc}}^2}{16}e^{-2\mu_{\mathrm{loc}}(t-t_{\mathrm{loc}})}. ( 1 − χ t ) + ( 1 − ρ t ) ≤ 16 r loc 2 e − 2 μ loc ( t − t loc ) . In particular, there exists ϵ 0 > 0 \epsilon_0 > 0 ϵ 0 > 0 , independent of d d d , such that for every ϵ ∈ ( 0 , ϵ 0 ) \epsilon\in(0,\epsilon_0) ϵ ∈ ( 0 , ϵ 0 ) ,
min { χ t , ρ t } ≥ 1 − ϵ (79) \min\{\chi_t,\rho_t\} \ge1-\epsilon \tag*{(79)} min { χ t , ρ t } ≥ 1 − ϵ (79) whenever
t ≥ C ′ log d + C log ( 1 / ϵ ) , t \ge C'\log d + C\log(1/\epsilon), t ≥ C ′ log d + C log ( 1/ ϵ ) , *where C < ∞ C < \infty C < ∞ is independent of d d d and ϵ \epsilon ϵ .
Proof. By Lemma C.14, there exist constants r l o c > 0 r_{\mathrm{loc}}>0 r loc > 0 and μ l o c > 0 \mu_{\mathrm{loc}}>0 μ loc > 0 , independent of d d d , such that for all sufficiently large d d d ,
Hess M L ( y ) ⪰ μ l o c I for every y ∈ U r l o c : = { y ∈ M : dist M ( y , y ∗ ) ≤ r l o c } . \operatorname{Hess}_{\mathcal{M}} L(y) \succeq\mu_{\mathrm{loc}}I \qquad\text{for every } y\in U_{r_{\mathrm{loc}}}:=\{y\in\mathcal{M}:\operatorname{dist}_{\mathcal{M}}(y,y^*)\le r_{\mathrm{loc}}\}. Hess M L ( y ) ⪰ μ loc I for every y ∈ U r loc := { y ∈ M : dist M ( y , y ∗ ) ≤ r loc } .
After shrinking r l o c r_{\mathrm{loc}} r loc , if necessary, we may assume
r l o c ≤ π 2 . r_{\mathrm{loc}}\le\frac{\pi}{2}. r loc ≤ 2 π .
The same Hessian lower bound remains valid on the smaller neighborhood.
Apply Lemma C.9 with this value of r l o c r_{\mathrm{loc}} r loc . On the favorable event of Lemma C.8, it gives a time t l o c t_{\mathrm{loc}} t loc satisfying
t l o c ≤ t e s c + C i n , t_{\mathrm{loc}}\le t_{\mathrm{esc}}+C_{\mathrm{in}}, t loc ≤ t esc + C in ,
where C i n < ∞ C_{\mathrm{in}}<\infty C in < ∞ is independent of d d d , and
ρ t l o c ≥ cos ( r l o c / 4 ) , χ t l o c ≥ cos ( r l o c / 4 ) . \rho_{t_{\mathrm{loc}}}\ge\cos(r_{\mathrm{loc}}/4),\qquad\chi_{t_{\mathrm{loc}}}\ge\cos(r_{\mathrm{loc}}/4). ρ t loc ≥ cos ( r loc /4 ) , χ t loc ≥ cos ( r loc /4 ) .
Equivalently
R t l o c 2 = dist M ( y t l o c , y ∗ ) 2 ≤ r l o c 2 8 , R_{t_{\mathrm{loc}}}^{2}=\operatorname{dist}_{\mathcal{M}}(y_{t_{\mathrm{loc}}},y^*)^{2}\le\frac{r_{\mathrm{loc}}^{2}}{8}, R t loc 2 = dist M ( y t loc , y ∗ ) 2 ≤ 8 r loc 2 ,
which proves (76). Since Lemma C.8 gives
t e s c ≤ C e s c log d , t_{\mathrm{esc}}\le C_{\mathrm{esc}}\log d, t esc ≤ C esc log d ,
we have, for d ≥ e d\ge e d ≥ e ,
t l o c ≤ C e s c log d + C i n ≤ ( C e s c + C i n ) log d . t_{\mathrm{loc}}\le C_{\mathrm{esc}}\log d+C_{\mathrm{in}}\le(C_{\mathrm{esc}}+C_{\mathrm{in}})\log d. t loc ≤ C esc log d + C in ≤ ( C esc + C in ) log d .
Thus the claimed bound holds with
C l o c : = C e s c + C i n . C_{\mathrm{loc}}:=C_{\mathrm{esc}}+C_{\mathrm{in}}. C loc := C esc + C in .
Next, observe that y ∗ y^* y ∗ is a critical point of L L L . Indeed, at the planted pair the residual vanishes pointwise, so
L ( y ∗ ) = 0. L(y^*)=0. L ( y ∗ ) = 0.
Since L ≥ 0 L\ge0 L ≥ 0 , y ∗ y^* y ∗ is a global minimizer of L L L , and therefore
∇ M L ( y ∗ ) = 0. \nabla_{\mathcal{M}}L(y^*)=0. ∇ M L ( y ∗ ) = 0.
The entry estimate gives
R t l o c ≤ r l o c 2 2 < r l o c 2 . R_{t_{\mathrm{loc}}}\le\frac{r_{\mathrm{loc}}}{2\sqrt{2}}<\frac{r_{\mathrm{loc}}}{2}. R t loc ≤ 2 2 r loc < 2 r loc .
Hence we may apply Lemma C.15 with
r = r l o c , μ = μ l o c , t 0 = t l o c . r=r_{\mathrm{loc}},\qquad\mu=\mu_{\mathrm{loc}},\qquad t_0=t_{\mathrm{loc}}. r = r loc , μ = μ loc , t 0 = t loc .
It follows that the trajectory remains in U r l o c U_{r_{\mathrm{loc}}} U r loc for all t ≥ t l o c t\ge t_{\mathrm{loc}} t ≥ t loc , and that
R t 2 ≤ e − 2 μ l o c ( t − t l o c ) R t l o c 2 for every t ≥ t l o c . R_t^2\le e^{-2\mu_{\mathrm{loc}}(t-t_{\mathrm{loc}})}R_{t_{\mathrm{loc}}}^2\qquad\text{for every }t\ge t_{\mathrm{loc}}. R t 2 ≤ e − 2 μ loc ( t − t loc ) R t loc 2 for every t ≥ t loc .
We now convert the product-manifold distance decay into overlap decay. Since χ t = ⟨ θ t , θ ∗ ⟩ = cos ( dist S d − 1 ( θ t , θ ∗ ) ) \chi_t=\langle\theta_t,\theta^*\rangle=\cos\left(\operatorname{dist}_{\mathbb{S}^{d-1}}(\theta_t,\theta^*)\right) χ t = ⟨ θ t , θ ∗ ⟩ = cos ( dist S d − 1 ( θ t , θ ∗ ) ) , and ρ t = ⟨ β t , β ∗ ⟩ = cos ( dist S d − 1 ( β t , β ∗ ) ) \rho_t=\langle\beta_t,\beta^*\rangle=\cos\left(\operatorname{dist}_{\mathbb{S}^{d-1}}(\beta_t,\beta^*)\right) ρ t = ⟨ β t , β ∗ ⟩ = cos ( dist S d − 1 ( β t , β ∗ ) ) , the elementary bound 1 − cos x ≤ x 2 2 1-\cos x\le\frac{x^2}{2} 1 − cos x ≤ 2 x 2 (x ≥ 0 x\ge0 x ≥ 0 ) gives
1 − χ t ≤ 1 2 dist S d − 1 ( θ t , θ ∗ ) 2 , 1-\chi_t\le\frac{1}{2}\operatorname{dist}_{\mathbb{S}^{d-1}}(\theta_t,\theta^*)^2, 1 − χ t ≤ 2 1 dist S d − 1 ( θ t , θ ∗ ) 2 ,
and
1 − ρ t ≤ 1 2 dist S d − 1 ( β t , β ∗ ) 2 . 1-\rho_t\le\frac{1}{2}\operatorname{dist}_{\mathbb{S}^{d-1}}(\beta_t,\beta^*)^2. 1 − ρ t ≤ 2 1 dist S d − 1 ( β t , β ∗ ) 2 .
Summing these two estimates and using the product metric, we have
( 1 − χ t ) + ( 1 − ρ t ) ≤ 1 2 dist M ( y t , y ∗ ) 2 = 1 2 R t 2 . (1-\chi_t)+(1-\rho_t)\le\frac{1}{2}\operatorname{dist}_{\mathcal{M}}(y_t,y^*)^2=\frac{1}{2}R_t^2. ( 1 − χ t ) + ( 1 − ρ t ) ≤ 2 1 dist M ( y t , y ∗ ) 2 = 2 1 R t 2 .
Therefore,
( 1 − χ t ) + ( 1 − ρ t ) ≤ 1 2 e − 2 μ l o c ( t − t l o c ) R t l o c 2 . (1-\chi_t)+(1-\rho_t)\le\frac{1}{2}e^{-2\mu_{\mathrm{loc}}(t-t_{\mathrm{loc}})}R_{t_{\mathrm{loc}}}^{2}. ( 1 − χ t ) + ( 1 − ρ t ) ≤ 2 1 e − 2 μ loc ( t − t loc ) R t loc 2 .
Using (78) that we proved, we obtain
( 1 − χ t ) + ( 1 − ρ t ) ≤ r l o c 2 16 e − 2 μ l o c ( t − t l o c ) . (1-\chi_t)+(1-\rho_t)\le\frac{r_{\mathrm{loc}}^{2}}{16}e^{-2\mu_{\mathrm{loc}}(t-t_{\mathrm{loc}})}. ( 1 − χ t ) + ( 1 − ρ t ) ≤ 16 r loc 2 e − 2 μ loc ( t − t loc ) .
Finally, set
ε 0 : = r l o c 2 16 . \varepsilon_0:=\frac{r_{\mathrm{loc}}^{2}}{16}. ε 0 := 16 r loc 2 .
Let ε ∈ ( 0 , ε 0 ) \varepsilon\in(0,\varepsilon_0) ε ∈ ( 0 , ε 0 ) . If
r l o c 2 16 e − 2 μ l o c ( t − t l o c ) ≤ ε , \frac{r_{\mathrm{loc}}^{2}}{16}e^{-2\mu_{\mathrm{loc}}(t-t_{\mathrm{loc}})}\le\varepsilon, 16 r loc 2 e − 2 μ loc ( t − t loc ) ≤ ε ,
then
( 1 − χ t ) + ( 1 − ρ t ) ≤ ε . (1-\chi_t)+(1-\rho_t)\le\varepsilon. ( 1 − χ t ) + ( 1 − ρ t ) ≤ ε .
Since both summands are nonnegative, this implies
1 − χ t ≤ ε , 1 − ρ t ≤ ε , 1-\chi_t\le\varepsilon,\qquad1-\rho_t\le\varepsilon, 1 − χ t ≤ ε , 1 − ρ t ≤ ε ,
and hence we arrive at (77):
min { χ t , ρ t } ≥ 1 − ε . \min\{\chi_t,\rho_t\}\ge1-\varepsilon. min { χ t , ρ t } ≥ 1 − ε .
The displayed exponential condition is equivalent to
t − t l o c ≥ 1 2 μ l o c log r l o c 2 16 ε . t-t_{\mathrm{loc}}\ge\frac{1}{2\mu_{\mathrm{loc}}}\log\frac{r_{\mathrm{loc}}^{2}}{16\varepsilon}. t − t loc ≥ 2 μ loc 1 log 16 ε r loc 2 .
Using t l o c ≤ C l o c log d t_{\mathrm{loc}}\le C_{\mathrm{loc}}\log d t loc ≤ C loc log d , we conclude that
min { χ t , ρ t } ≥ 1 − ε \min\{\chi_t,\rho_t\}\ge1-\varepsilon min { χ t , ρ t } ≥ 1 − ε
whenever
t ≥ C l o c log d + 1 2 μ l o c log r l o c 2 16 ε . t\ge C_{\mathrm{loc}}\log d+\frac{1}{2\mu_{\mathrm{loc}}}\log\frac{r_{\mathrm{loc}}^{2}}{16\varepsilon}. t ≥ C loc log d + 2 μ loc 1 log 16 ε r loc 2 .
Since r l o c r_{\mathrm{loc}} r loc and μ l o c \mu_{\mathrm{loc}} μ loc are fixed constants independent of d d d and ε \varepsilon ε , this is of the form
t ≥ C log d + C log ( 1 / ε ) , t\ge C\log d+C\log(1/\varepsilon), t ≥ C log d + C log ( 1/ ε ) ,
for a constant C < ∞ C<\infty C < ∞ independent of d d d and ε \varepsilon ε . This proves the theorem.
Proof of Discrete-Time Online SGD Dynamics
We now move to analyze the learning dynamics under SGD discretization.
Weak Recovery Setups
Assumption D.1 (SGD regime and initialization for weak recovery). Fix 0 < ε ≤ ε ∗ 0<\varepsilon\le\varepsilon_* 0 < ε ≤ ε ∗ . Consider online blockwise spherical SGD on S d − 1 × S d − 1 \mathbb{S}^{d-1}\times\mathbb{S}^{d-1} S d − 1 × S d − 1 follows (3). The horizon and step-size follows
T δ ≤ C T log d , δ ≤ c 0 ε d log d , 0 < ε ≤ ε ∗ . T\delta\le C_T\log d,\qquad\delta\le\frac{c_0\varepsilon}{d\log d},\qquad0<\varepsilon\le\varepsilon_*. T δ ≤ C T log d , δ ≤ d log d c 0 ε , 0 < ε ≤ ε ∗ . for an absolute constant c 0 > 0 c_0>0 c 0 > 0 , so we have the weak recovery horizon T : = ⌈ C T δ − 1 log d ⌉ ∈ N T:=\lceil C_T\delta^{-1}\log d\rceil\in\mathbb{N} T := ⌈ C T δ − 1 log d ⌉ ∈ N . Initialization condition (A4) also follows.
We start the discretizations analysis by deriving the one-step evolution of the overlaps under SGD:
Lemma D.2 (One-step evolution of the overlaps under SGD). Under Assumption D.1 , on the event
E t ( T ) : = { δ ∥ g t ( β ) ∥ ≤ 1 / 2 , δ ∥ g t ( θ ) ∥ ≤ 1 / 2 } , (80) E_t^{(\mathrm{T})} := \left\{\delta\lVert g_t^{(\beta)}\rVert\le1/2,\ \delta\lVert g_t^{(\theta)}\rVert\le1/2\right\}, \tag*{(80)} E t ( T ) := { δ ∥ g t ( β ) ∥ ≤ 1/2 , δ ∥ g t ( θ ) ∥ ≤ 1/2 } , (80) there exist F t \mathcal{F}_t F t -measurable predictable drifts F t ( ρ ) , F t ( χ ) F_t^{(\rho)}, F_t^{(\chi)} F t ( ρ ) , F t ( χ ) , martingale increments M t ( ρ ) , M t ( χ ) M_t^{(\rho)}, M_t^{(\chi)} M t ( ρ ) , M t ( χ ) , and remainders R t ( ρ ) , R t ( χ ) R_t^{(\rho)}, R_t^{(\chi)} R t ( ρ ) , R t ( χ ) such that for all t t t ,
ρ t + 1 = ρ t + δ F t ( ρ ) + δ M t ( ρ ) + R t ( ρ ) (81) \rho_{t+1} = \rho_t + \delta F_t^{(\rho)} + \delta M_t^{(\rho)} + R_t^{(\rho)} \tag*{(81)} ρ t + 1 = ρ t + δ F t ( ρ ) + δ M t ( ρ ) + R t ( ρ ) (81) χ t + 1 = χ t + δ F t ( χ ) + δ M t ( χ ) + R t ( χ ) (82) \chi_{t+1} = \chi_t + \delta F_t^{(\chi)} + \delta M_t^{(\chi)} + R_t^{(\chi)} \tag*{(82)} χ t + 1 = χ t + δ F t ( χ ) + δ M t ( χ ) + R t ( χ ) (82) where E [ M t ( ⋅ ) ∣ F t ] = 0 \mathbb{E}[M_t^{(\cdot)} \mid\mathcal{F}_t] = 0 E [ M t ( ⋅ ) ∣ F t ] = 0 for each component, R t ( ρ ) = O ( δ 3 ∥ g t ( β ) ∥ 3 ) R_t^{(\rho)} = O(\delta^3\lVert g_t^{(\beta)}\rVert^3) R t ( ρ ) = O ( δ 3 ∥ g t ( β ) ∥ 3 ) , and R t ( χ ) = O ( δ 3 ∥ g t ( θ ) ∥ 3 ) R_t^{(\chi)} = O(\delta^3\lVert g_t^{(\theta)}\rVert^3) R t ( χ ) = O ( δ 3 ∥ g t ( θ ) ∥ 3 ) .
Proof. Let g t ( β ) : = ∇ β S l ( β t , θ t ; x t ) g_t^{(\beta)} := \nabla_{\beta}^{S} l(\beta_t,\theta_t;x_t) g t ( β ) := ∇ β S l ( β t , θ t ; x t ) and g t ( θ ) : = ∇ θ S l ( β t , θ t ; x t ) g_t^{(\theta)} := \nabla_{\theta}^{S} l(\beta_t,\theta_t;x_t) g t ( θ ) := ∇ θ S l ( β t , θ t ; x t ) be the spherical gradients taken with respect β \beta β and θ \theta θ respectively, which implies g t ( β ) ⊥ β t g_t^{(\beta)} \perp\beta_t g t ( β ) ⊥ β t and g t ( θ ) ⊥ θ t g_t^{(\theta)} \perp\theta_t g t ( θ ) ⊥ θ t . From (3) and under E t E_t E t , we have the normalized update as
β t + 1 = β t − δ g t ( β ) ∥ β t − δ g t ( β ) ∥ = β t − δ g t ( β ) − δ 2 2 ∥ g t ( β ) ∥ 2 β t + O ( δ 3 ∥ g t ( β ) ∥ 3 ) (83) \beta_{t+1} = \frac{\beta_t-\delta g_t^{(\beta)}}{\lVert\beta_t-\delta g_t^{(\beta)}\rVert}
= \beta_t-\delta g_t^{(\beta)}-\frac{\delta^2}{2}\lVert g_t^{(\beta)}\rVert^2\beta_t+O(\delta^3\lVert g_t^{(\beta)}\rVert^3) \tag*{(83)} β t + 1 = ∥ β t − δ g t ( β ) ∥ β t − δ g t ( β ) = β t − δ g t ( β ) − 2 δ 2 ∥ g t ( β ) ∥ 2 β t + O ( δ 3 ∥ g t ( β ) ∥ 3 ) (83)
θ t + 1 = θ t − δ g t ( θ ) ∥ θ t − δ g t ( θ ) ∥ = θ t − δ g t ( θ ) − δ 2 2 ∥ g t ( θ ) ∥ 2 θ t + O ( δ 3 ∥ g t ( θ ) ∥ 3 ) (84) \theta_{t+1} = \frac{\theta_t-\delta g_t^{(\theta)}}{\lVert\theta_t-\delta g_t^{(\theta)}\rVert}
= \theta_t-\delta g_t^{(\theta)}-\frac{\delta^2}{2}\lVert g_t^{(\theta)}\rVert^2\theta_t+O(\delta^3\lVert g_t^{(\theta)}\rVert^3) \tag*{(84)} θ t + 1 = ∥ θ t − δ g t ( θ ) ∥ θ t − δ g t ( θ ) = θ t − δ g t ( θ ) − 2 δ 2 ∥ g t ( θ ) ∥ 2 θ t + O ( δ 3 ∥ g t ( θ ) ∥ 3 ) (84)
Hence taking overlap with the teacher direction β ∗ \beta^* β ∗ and θ ∗ \theta^* θ ∗ , respectively, we have
⟨ β t + 1 , β ∗ ⟩ = ⟨ β t , β ∗ ⟩ − δ ⟨ g t ( β ) , β ∗ ⟩ − δ 2 2 ∥ g t ( β ) ∥ 2 ⟨ β t , β ∗ ⟩ + O ( δ 3 ∥ g t ( β ) ∥ 3 ) (85) \langle\beta_{t+1},\beta^*\rangle= \langle\beta_t,\beta^*\rangle-\delta\langle g_t^{(\beta)},\beta^*\rangle-\frac{\delta^2}{2}\lVert g_t^{(\beta)}\rVert^2\langle\beta_t,\beta^*\rangle+O(\delta^3\lVert g_t^{(\beta)}\rVert^3) \tag*{(85)} ⟨ β t + 1 , β ∗ ⟩ = ⟨ β t , β ∗ ⟩ − δ ⟨ g t ( β ) , β ∗ ⟩ − 2 δ 2 ∥ g t ( β ) ∥ 2 ⟨ β t , β ∗ ⟩ + O ( δ 3 ∥ g t ( β ) ∥ 3 ) (85)
⟨ θ t + 1 , θ ∗ ⟩ = ⟨ θ t , θ ∗ ⟩ − δ ⟨ g t ( θ ) , θ ∗ ⟩ − δ 2 2 ∥ g t ( θ ) ∥ 2 ⟨ θ t , θ ∗ ⟩ + O ( δ 3 ∥ g t ( θ ) ∥ 3 ) , (86) \langle\theta_{t+1},\theta^*\rangle= \langle\theta_t,\theta^*\rangle-\delta\langle g_t^{(\theta)},\theta^*\rangle-\frac{\delta^2}{2}\lVert g_t^{(\theta)}\rVert^2\langle\theta_t,\theta^*\rangle+O(\delta^3\lVert g_t^{(\theta)}\rVert^3), \tag*{(86)} ⟨ θ t + 1 , θ ∗ ⟩ = ⟨ θ t , θ ∗ ⟩ − δ ⟨ g t ( θ ) , θ ∗ ⟩ − 2 δ 2 ∥ g t ( θ ) ∥ 2 ⟨ θ t , θ ∗ ⟩ + O ( δ 3 ∥ g t ( θ ) ∥ 3 ) , (86)
where ρ t : = ⟨ β t , β ∗ ⟩ \rho_t := \langle\beta_t,\beta^*\rangle ρ t := ⟨ β t , β ∗ ⟩ and χ t : = ⟨ θ t , θ ∗ ⟩ \chi_t := \langle\theta_t,\theta^*\rangle χ t := ⟨ θ t , θ ∗ ⟩ . Now, we decompose the middle two terms involving spherical gradient as the population drift plus the martingale increment, namely,
⟨ g t ( β ) , β ∗ ⟩ + δ 2 ∥ g t ( β ) ∥ 2 ρ t = E [ ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ∥ g t ( β ) ∥ 2 ρ t ∣ F t ] + ( ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ∥ g t ( β ) ∥ 2 ρ t − E [ ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ∥ g t ( β ) ∥ 2 ρ t ∣ F t ] ) , (87) \begin{aligned}
\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\beta)}\rVert^2\rho_t
&= \mathbb{E}\left[\left.\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\beta)}\rVert^2\rho_t\right|\mathcal{F}_t\right] \\
&\quad+\left(\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\beta)}\rVert^2\rho_t
-\mathbb{E}\left[\left.\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\beta)}\rVert^2\rho_t\right|\mathcal{F}_t\right]\right), \tag*{(87)}
\end{aligned} ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ∥ g t ( β ) ∥ 2 ρ t = E [ ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ∥ g t ( β ) ∥ 2 ρ t F t ] + ( ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ∥ g t ( β ) ∥ 2 ρ t − E [ ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ∥ g t ( β ) ∥ 2 ρ t F t ] ) , (87)
⟨ g t ( θ ) , θ ∗ ⟩ + δ 2 ∥ g t ( θ ) ∥ 2 χ t = E [ ⟨ g t ( θ ) , θ ∗ ⟩ + δ 2 ∥ g t ( θ ) ∥ 2 χ t ∣ F t ] + ( ⟨ g t ( θ ) , θ ∗ ⟩ + δ 2 ∥ g t ( θ ) ∥ 2 χ t − E [ ⟨ g t ( θ ) , θ ∗ ⟩ + δ 2 ∥ g t ( θ ) ∥ 2 χ t ∣ F t ] ) . (88) \begin{aligned}
\langle g_t^{(\theta)},\theta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\theta)}\rVert^2\chi_t
&= \mathbb{E}\left[\left.\langle g_t^{(\theta)},\theta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\theta)}\rVert^2\chi_t\right|\mathcal{F}_t\right] \\
&\quad+\left(\langle g_t^{(\theta)},\theta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\theta)}\rVert^2\chi_t
-\mathbb{E}\left[\left.\langle g_t^{(\theta)},\theta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\theta)}\rVert^2\chi_t\right|\mathcal{F}_t\right]\right). \tag*{(88)}
\end{aligned} ⟨ g t ( θ ) , θ ∗ ⟩ + 2 δ ∥ g t ( θ ) ∥ 2 χ t = E [ ⟨ g t ( θ ) , θ ∗ ⟩ + 2 δ ∥ g t ( θ ) ∥ 2 χ t F t ] + ( ⟨ g t ( θ ) , θ ∗ ⟩ + 2 δ ∥ g t ( θ ) ∥ 2 χ t − E [ ⟨ g t ( θ ) , θ ∗ ⟩ + 2 δ ∥ g t ( θ ) ∥ 2 χ t F t ] ) . (88)
Therefore, we can write (85) and (86) respectively as
ρ t + 1 = ρ t + δ F t ( ρ ) + δ M t ( ρ ) + O ( δ 3 ∥ g t ( β ) ∥ 3 ) (89) \rho_{t+1} = \rho_t+\delta F_t^{(\rho)}+\delta M_t^{(\rho)}+O(\delta^3\lVert g_t^{(\beta)}\rVert^3) \tag*{(89)} ρ t + 1 = ρ t + δ F t ( ρ ) + δ M t ( ρ ) + O ( δ 3 ∥ g t ( β ) ∥ 3 ) (89)
χ t + 1 = χ t + δ F t ( χ ) + δ M t ( χ ) + O ( δ 3 ∥ g t ( θ ) ∥ 3 ) (90) \chi_{t+1} = \chi_t+\delta F_t^{(\chi)}+\delta M_t^{(\chi)}+O(\delta^3\lVert g_t^{(\theta)}\rVert^3) \tag*{(90)} χ t + 1 = χ t + δ F t ( χ ) + δ M t ( χ ) + O ( δ 3 ∥ g t ( θ ) ∥ 3 ) (90)
Defining R t ( ρ ) : = O ( δ 3 ∥ g t ( β ) ∥ 3 ) R_t^{(\rho)} := O(\delta^3\lVert g_t^{(\beta)}\rVert^3) R t ( ρ ) := O ( δ 3 ∥ g t ( β ) ∥ 3 ) and R t ( χ ) : = O ( δ 3 ∥ g t ( θ ) ∥ 3 ) R_t^{(\chi)} := O(\delta^3\lVert g_t^{(\theta)}\rVert^3) R t ( χ ) := O ( δ 3 ∥ g t ( θ ) ∥ 3 ) gives the claimed expression.
Next we provide a class of concentration inequalities to better control the iteration of ( ρ t , χ t ) (\rho_t, \chi_t) ( ρ t , χ t ) that has been used in studying the aggregate SGD dynamics, on top of (81) and (82) . In particular, Lemma D.5 controls the aggregate martingale term, and Lemma D.7 controls the aggregate discretization remainder term. Besides, As the one-step expansion in Lemma D.2 relies on the event E t \mathcal{E}_t E t , we provide Lemma D.3 to show that E t \mathcal{E}_t E t is true with high probability. All of these inequalities are obtained via using the technical bounds specified in Lemma E.7 in E.2.
Taylor Expansion Error Control
Lemma D.3 (Uniform Taylor event on the Weak-Recovery Horizon). Assume the moment bounds of Lemma E.7 and following Assumption D.1 suppose
T δ ≤ C T log d , δ ≤ c 0 ε d log d , 0 < ε ≤ ε ∗ . T\delta\le C_T \log d,\qquad\delta\le\frac{c_0\varepsilon}{d\log d},\qquad0 < \varepsilon\le\varepsilon_*. T δ ≤ C T log d , δ ≤ d log d c 0 ε , 0 < ε ≤ ε ∗ .
Then there is a constant K T a y K_{\mathrm{Tay}} K Tay , depending only on model constants, C T C_T C T , c 0 c_0 c 0 , ε ∗ \varepsilon_* ε ∗ , such that for all sufficiently large d d d , the Taylor Event E T ( T ) E_T^{(T)} E T ( T ) satisfies:
P ( ( E T ( T ) ) c ) ≤ 32 K T δ 4 d 2 ≤ K T a y ε 3 d log 2 d ≤ K T a y ε . (91) \mathbb{P}\left((E_T^{(T)})^c\right) \le32KT\delta^4d^2 \le K_{\mathrm{Tay}}\frac{\varepsilon^3}{d\log^2 d} \le K_{\mathrm{Tay}}\varepsilon. \tag*{(91)} P ( ( E T ( T ) ) c ) ≤ 32 K T δ 4 d 2 ≤ K Tay d log 2 d ε 3 ≤ K Tay ε . (91)
Moreover, on E T ( T ) E_T^{(T)} E T ( T ) , the one-step expansions (81) and (82) in Lemma D.2 are valid simultaneously for every t < T t<T t < T , with remainders
∣ R t ( ρ ) ∣ ≤ C δ 3 ∥ g t ( β ) ∥ 3 , ∣ R t ( χ ) ∣ ≤ C δ 3 ∥ g t ( θ ) ∥ 3 . \left|R_t^{(\rho)}\right| \le C\delta^3\left\|g_t^{(\beta)}\right\|^3,\qquad\left|R_t^{(\chi)}\right| \le C\delta^3\left\|g_t^{(\theta)}\right\|^3. R t ( ρ ) ≤ C δ 3 g t ( β ) 3 , R t ( χ ) ≤ C δ 3 g t ( θ ) 3 .
Proof. By Lemma E.7, conditionally on F t \mathcal{F}_t F t , we have
E [ ∥ g t ( β ) ∥ 4 ∣ F t ] ≤ K d 2 , E [ ∥ g t ( θ ) ∥ 4 ∣ F t ] ≤ K d 2 . \mathbb{E}\left[\left\|g_t^{(\beta)}\right\|^4\mid\mathcal{F}_t\right]\le Kd^2,\qquad\mathbb{E}\left[\left\|g_t^{(\theta)}\right\|^4\mid\mathcal{F}_t\right]\le Kd^2. E [ g t ( β ) 4 ∣ F t ] ≤ K d 2 , E [ g t ( θ ) 4 ∣ F t ] ≤ K d 2 .
Hence, by Markov’s inequality,
P ( δ ∥ g t ( β ) ∥ > 1 2 ∣ F t ) ≤ 16 δ 4 E [ ∥ g t ( β ) ∥ 4 ∣ F t ] ≤ 16 K δ 4 d 2 . \mathbb{P}\left(\delta\left\|g_t^{(\beta)}\right\|>\frac{1}{2}\mid\mathcal{F}_t\right)\le16\delta^4\mathbb{E}\left[\left\|g_t^{(\beta)}\right\|^4\mid\mathcal{F}_t\right]\le16K\delta^4d^2. P ( δ g t ( β ) > 2 1 ∣ F t ) ≤ 16 δ 4 E [ g t ( β ) 4 ∣ F t ] ≤ 16 K δ 4 d 2 .
The same bound holds for the θ \theta θ -block. Therefore
P ( ( E t ( T ) ) c ) ≤ 32 K δ 4 d 2 . \mathbb{P}\left((E_t^{(T)})^c\right)\le32K\delta^4d^2. P ( ( E t ( T ) ) c ) ≤ 32 K δ 4 d 2 .
Union bounding over t = 0 , … , T − 1 t=0,\ldots,T-1 t = 0 , … , T − 1 ,
P ( ( E T ( T ) ) c ) ≤ 32 K T δ 4 d 2 . \mathbb{P}\left((E_T^{(T)})^c\right)\le32KT\delta^4d^2. P ( ( E T ( T ) ) c ) ≤ 32 K T δ 4 d 2 .
Since T δ ≤ C T log d T\delta\le C_T\log d T δ ≤ C T log d ,
T δ 4 d 2 = ( T δ ) δ 3 d 2 ≤ C T δ 3 d 2 log d . T\delta^4d^2=(T\delta)\delta^3d^2\le C_T\delta^3d^2\log d. T δ 4 d 2 = ( T δ ) δ 3 d 2 ≤ C T δ 3 d 2 log d .
Using δ ≤ c 0 ε d log d \delta\le\frac{c_0\varepsilon}{d\log d} δ ≤ d l o g d c 0 ε , we get
δ 3 d 2 log d ≤ c 0 3 ε 3 d log 2 d . \delta^3d^2\log d\le\frac{c_0^3\varepsilon^3}{d\log^2d}. δ 3 d 2 log d ≤ d log 2 d c 0 3 ε 3 .
Thus
P ( ( E T ( T ) ) c ) ≤ K T a y ε 3 d log 2 d ≤ K T a y ε \mathbb{P}\left((E_T^{(T)})^c\right)\le K_{\mathrm{Tay}}\frac{\varepsilon^3}{d\log^2d}\le K_{\mathrm{Tay}}\varepsilon P ( ( E T ( T ) ) c ) ≤ K Tay d log 2 d ε 3 ≤ K Tay ε
for all sufficiently large d d d . This proves the lemma.
Martingale and Discretization Error Control for Z t Z_t Z t
The Martingale Error for Z t Z_t Z t depends on this intermediate Corollary, derived from Lemma E.7.
Corollary D.4 (Negligible remainder for drift). With the martingale increments M t ( ρ ) M_t^{(\rho)} M t ( ρ ) and M t ( χ ) M_t^{(\chi)} M t ( χ ) we defined in Lemma D.2 and adopting the technical gradient bounds in Lemma E.7, we have
E [ ( M t ( ρ ) ) 2 ∣ F t ] ≤ K M ρ , E [ ( M t ( χ ) ) 2 ∣ F t ] ≤ K M χ , (92) \mathbb{E}\left[(M_t^{(\rho)})^2\mid\mathcal{F}_t\right]\le K_M^\rho,\qquad\mathbb{E}\left[(M_t^{(\chi)})^2\mid\mathcal{F}_t\right]\le K_M^\chi, \tag*{(92)} E [ ( M t ( ρ ) ) 2 ∣ F t ] ≤ K M ρ , E [ ( M t ( χ ) ) 2 ∣ F t ] ≤ K M χ , (92)
where K M ρ , K M χ K_M^\rho,K_M^\chi K M ρ , K M χ absolute for large d d d under step size δ ≤ c 0 ε / d log d \delta\le c_0\varepsilon/d\log d δ ≤ c 0 ε / d log d .
Proof. Let A t ( ρ ) = ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ∥ g t ( β ) ∥ 2 ρ t A_t^{(\rho)}=\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\beta)}\rVert^2\rho_t A t ( ρ ) = ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ∥ g t ( β ) ∥ 2 ρ t , then the martingale term writes as M t ( ρ ) = − ( A t ( ρ ) − E [ A t ( ρ ) ∣ F t ] ) M_t^{(\rho)}=-(A_t^{(\rho)}-\mathbb{E}[A_t^{(\rho)}\mid\mathcal{F}_t]) M t ( ρ ) = − ( A t ( ρ ) − E [ A t ( ρ ) ∣ F t ]) , so the conditional second moment of M t ( ρ ) M_t^{(\rho)} M t ( ρ ) can be written as
E [ ( M t ( ρ ) ) 2 ∣ F t ] = E [ ( A t ( ρ ) − E [ A t ( ρ ) ∣ F t ] ) 2 ∣ F t ] (93) \mathbb{E}[(M_t^{(\rho)})^2\mid\mathcal{F}_t]=\mathbb{E}\left[\left(A_t^{(\rho)}-\mathbb{E}[A_t^{(\rho)}\mid\mathcal{F}_t]\right)^2\mid\mathcal{F}_t\right] \tag*{(93)} E [( M t ( ρ ) ) 2 ∣ F t ] = E [ ( A t ( ρ ) − E [ A t ( ρ ) ∣ F t ] ) 2 ∣ F t ] (93)
Since we have:
A t ( ρ ) − E [ A t ( ρ ) ∣ F t ] = ( ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ∥ g t ( β ) ∥ 2 ρ t ) − E [ ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ∥ g t ( β ) ∥ 2 ρ t ∣ F t ] = ( ⟨ g t ( β ) , β ∗ ⟩ − E [ ⟨ g t ( β ) , β ∗ ⟩ ∣ F t ] ) + δ 2 ρ t ( ∥ g t ( β ) ∥ 2 − E [ ∥ g t ( β ) ∥ 2 ∣ F t ] ) by regrouping = ⟨ Δ g t ( β ) , β ∗ ⟩ + δ 2 ρ t Δ q t ( β ) , \begin{aligned}
A_t^{(\rho)}-\mathbb{E}[A_t^{(\rho)}\mid\mathcal{F}_t]
&=\left(\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\lVert g_t^{(\beta)}\rVert^2\rho_t\right)\\
&\quad-\mathbb{E}\left[\left\langle g_t^{(\beta)},\beta^*\right\rangle+\frac{\delta}{2}\lVert g_t^{(\beta)}\rVert^2\rho_t\mid\mathcal{F}_t\right]\\
&=\left(\langle g_t^{(\beta)},\beta^*\rangle-\mathbb{E}[\langle g_t^{(\beta)},\beta^*\rangle\mid\mathcal{F}_t]\right)\\
&\quad+\frac{\delta}{2}\rho_t\left(\lVert g_t^{(\beta)}\rVert^2-\mathbb{E}[\lVert g_t^{(\beta)}\rVert^2\mid\mathcal{F}_t]\right)\text{ by regrouping}\\
&=\langle\Delta g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\rho_t\Delta q_t^{(\beta)},
\end{aligned} A t ( ρ ) − E [ A t ( ρ ) ∣ F t ] = ( ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ∥ g t ( β ) ∥ 2 ρ t ) − E [ ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ∥ g t ( β ) ∥ 2 ρ t ∣ F t ] = ( ⟨ g t ( β ) , β ∗ ⟩ − E [⟨ g t ( β ) , β ∗ ⟩ ∣ F t ] ) + 2 δ ρ t ( ∥ g t ( β ) ∥ 2 − E [∥ g t ( β ) ∥ 2 ∣ F t ] ) by regrouping = ⟨ Δ g t ( β ) , β ∗ ⟩ + 2 δ ρ t Δ q t ( β ) ,
where we defined Δ g t ( β ) : = g t ( β ) − E [ g t ( β ) ∣ F t ] \Delta g_t^{(\beta)}:=g_t^{(\beta)}-\mathbb{E}[g_t^{(\beta)}\mid\mathcal{F}_t] Δ g t ( β ) := g t ( β ) − E [ g t ( β ) ∣ F t ] , and Δ q t ( β ) : = ∥ g t ( β ) ∥ 2 − E [ ∥ g t ( β ) ∥ 2 ∣ F t ] \Delta q_t^{(\beta)}:=\lVert g_t^{(\beta)}\rVert^2-\mathbb{E}[\lVert g_t^{(\beta)}\rVert^2\mid\mathcal{F}_t] Δ q t ( β ) := ∥ g t ( β ) ∥ 2 − E [∥ g t ( β ) ∥ 2 ∣ F t ] , (91) can be written as:
E [ ( M t ( ρ ) ) 2 ∣ F t ] = E [ ( ⟨ Δ g t ( β ) , β ∗ ⟩ + δ 2 ρ t Δ q t ( β ) ) 2 ∣ F t ] (94) \mathbb{E}[(M_t^{(\rho)})^2\mid\mathcal{F}_t]=\mathbb{E}\left[\left(\langle\Delta g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\rho_t\Delta q_t^{(\beta)}\right)^2\mid\mathcal{F}_t\right] \tag*{(94)} E [( M t ( ρ ) ) 2 ∣ F t ] = E [ ( ⟨ Δ g t ( β ) , β ∗ ⟩ + 2 δ ρ t Δ q t ( β ) ) 2 ∣ F t ] (94)
= 2 E [ ⟨ Δ g t ( β ) , β ∗ ⟩ 2 ∣ F t ] + δ 2 2 ρ t 2 E [ ( Δ q t ( β ) ) 2 ] , given ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (95) =2\mathbb{E}[\langle\Delta g_t^{(\beta)},\beta^*\rangle^2\mid\mathcal{F}_t]+\frac{\delta^2}{2}\rho_t^2\mathbb{E}[(\Delta q_t^{(\beta)})^2],\text{ given }(a+b)^2\leq2a^2+2b^2 \tag*{(95)} = 2 E [⟨ Δ g t ( β ) , β ∗ ⟩ 2 ∣ F t ] + 2 δ 2 ρ t 2 E [( Δ q t ( β ) ) 2 ] , given ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (95)
By Lemma E.7, we can bound the first term of (93) by K K K . For the second term of (93), note that ( Δ q ) 2 = ( ∥ g t ( β ) ∥ 2 − E [ ∥ g t ( β ) ∥ 2 ∣ F t ] ) 2 ≤ ∥ g t ( β ) ∥ 4 (\Delta q)^2=(\lVert g_t^{(\beta)}\rVert^2-\mathbb{E}[\lVert g_t^{(\beta)}\rVert^2\mid\mathcal{F}_t])^2\leq\lVert g_t^{(\beta)}\rVert^4 ( Δ q ) 2 = (∥ g t ( β ) ∥ 2 − E [∥ g t ( β ) ∥ 2 ∣ F t ] ) 2 ≤ ∥ g t ( β ) ∥ 4 , we can use the forth moment bound in Lemma E.7 to obtain the K d 2 Kd^2 K d 2 bound, so
E [ ( M t ( ρ ) ) 2 ∣ F t ] ≤ 2 K + ( δ ρ t ) 2 2 K d 2 (96) \mathbb{E}[(M_t^{(\rho)})^2\mid\mathcal{F}_t]\leq2K+\frac{(\delta\rho_t)^2}{2}Kd^2 \tag*{(96)} E [( M t ( ρ ) ) 2 ∣ F t ] ≤ 2 K + 2 ( δ ρ t ) 2 K d 2 (96)
≤ 2 K + δ 2 2 K d 2 . (97) \leq2K+\frac{\delta^2}{2}Kd^2. \tag*{(97)} ≤ 2 K + 2 δ 2 K d 2 . (97)
Given our step size δ ≤ c 0 ε ∗ d log d \delta\leq\frac{c_0\varepsilon_*}{d\log d} δ ≤ d l o g d c 0 ε ∗ by Assumption D.1, δ 2 2 K d 2 ≤ c 0 2 ε ∗ 2 2 log d K \frac{\delta^2}{2}Kd^2\leq\frac{c_0^2\varepsilon_*^2}{2\log d}K 2 δ 2 K d 2 ≤ 2 l o g d c 0 2 ε ∗ 2 K , enabling us to write the bound
E [ ( M t ( ρ ) ) 2 ∣ F t ] ≤ K M ρ , (98) \mathbb{E}[(M_t^{(\rho)})^2\mid\mathcal{F}_t]\leq K_M^\rho, \tag*{(98)} E [( M t ( ρ ) ) 2 ∣ F t ] ≤ K M ρ , (98)
for an absolute K M ρ K_M^\rho K M ρ that depends only on model constants and c 0 , ε ∗ c_0,\varepsilon_* c 0 , ε ∗ . The exact argument for χ \chi χ gives
E [ ( M t ( χ ) ) 2 ∣ F t ] ≤ K M χ , (99) \mathbb{E}[(M_t^{(\chi)})^2\mid\mathcal{F}_t]\leq K_M^\chi, \tag*{(99)} E [( M t ( χ ) ) 2 ∣ F t ] ≤ K M χ , (99)
for an absolute K M χ K_M^\chi K M χ .
Lemma D.5 (Martingale inequality for Z t Z_t Z t ). Let
M t ( Z ) : = ∑ k = 0 t − 1 ( M k ( ρ ) + M k ( χ ) ) , t ≥ 0. \mathcal{M}_t^{(Z)}:=\sum_{k=0}^{t-1}(M_k^{(\rho)}+M_k^{(\chi)}),\qquad t\geq0. M t ( Z ) := k = 0 ∑ t − 1 ( M k ( ρ ) + M k ( χ ) ) , t ≥ 0. Under Assumption D.1 and the bounds of Corollary D.4, for all T ∈ N T\in\mathbb{N} T ∈ N and all λ > 0 \lambda>0 λ > 0 ,
P ( sup 0 ≤ t ≤ T δ ∣ M t ( Z ) ∣ ≥ λ ) ≤ K T δ 2 λ 2 , (100) \mathbb{P}\left(\sup_{0\leq t\leq T}\delta\lvert\mathcal{M}_t^{(Z)}\rvert\geq\lambda\right)\leq\frac{KT\delta^2}{\lambda^2}, \tag*{(100)} P ( 0 ≤ t ≤ T sup δ ∣ M t ( Z ) ∣ ≥ λ ) ≤ λ 2 K T δ 2 , (100) for some constant K > 0 K>0 K > 0 depending only on the model.
Proof. Let D k : = M k ( ρ ) + M k ( χ ) D_k:=M_k^{(\rho)}+M_k^{(\chi)} D k := M k ( ρ ) + M k ( χ ) , so that M t ( Z ) = ∑ k = 0 t − 1 D k \mathcal{M}_t^{(Z)}=\sum_{k=0}^{t-1}D_k M t ( Z ) = ∑ k = 0 t − 1 D k is a martingale with respect to ( F t ) (\mathcal{F}_t) ( F t ) . By Doob’s L 2 L^2 L 2 maximal inequality,
E [ sup 0 ≤ t ≤ T ∣ M t ( Z ) ∣ 2 ] ≤ 4 E [ ∣ M T ( Z ) ∣ 2 ] . (101) \mathbb{E}\left[\sup_{0\leq t\leq T}\lvert\mathcal{M}_t^{(Z)}\rvert^2\right]\leq4\mathbb{E}[\lvert\mathcal{M}_T^{(Z)}\rvert^2]. \tag*{(101)} E [ 0 ≤ t ≤ T sup ∣ M t ( Z ) ∣ 2 ] ≤ 4 E [∣ M T ( Z ) ∣ 2 ] . (101)
Moreover, using ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^2 \le2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2 and Corollary D.4,
E [ D k 2 ∣ F k ] ≤ 2 E [ ( M k ( ρ ) ) 2 ∣ F k ] + 2 E [ ( M k ( χ ) ) 2 ∣ F k ] ≤ 4 K . \mathbb{E}[D_k^2 \mid\mathcal{F}_k] \le2\mathbb{E}[(M_k^{(\rho)})^2 \mid\mathcal{F}_k] + 2\mathbb{E}[(M_k^{(\chi)})^2 \mid\mathcal{F}_k] \le4K. E [ D k 2 ∣ F k ] ≤ 2 E [( M k ( ρ ) ) 2 ∣ F k ] + 2 E [( M k ( χ ) ) 2 ∣ F k ] ≤ 4 K .
Since martingale increments are orthogonal in L 2 L^2 L 2 ,
E [ ∣ M T ( Z ) ∣ 2 ] = ∑ k = 0 T − 1 E [ D k 2 ] ≤ ∑ k = 0 T − 1 4 K = 4 K T . \mathbb{E}[|\mathcal{M}_T^{(Z)}|^2] = \sum_{k=0}^{T-1}\mathbb{E}[D_k^2] \le\sum_{k=0}^{T-1}4K = 4KT. E [ ∣ M T ( Z ) ∣ 2 ] = k = 0 ∑ T − 1 E [ D k 2 ] ≤ k = 0 ∑ T − 1 4 K = 4 K T .
Combining with (101) and applying Markov’s inequality yields
P ( sup 0 ≤ t ≤ T ∣ δ M t ( Z ) ∣ ≥ λ ) ≤ δ 2 λ 2 E [ sup 0 ≤ t ≤ T ∣ M t ( Z ) ∣ 2 ] ≤ δ 2 λ 2 ⋅ 4 E [ ∣ M T ( Z ) ∣ 2 ] ≤ 16 K T δ 2 λ 2 . \begin{aligned}
\mathbb{P}\left(\sup_{0\le t\le T}|\delta\mathcal{M}_t^{(Z)}|\ge\lambda\right) \le\frac{\delta^2}{\lambda^2}\mathbb{E}\left[\sup_{0\le t\le T}|\mathcal{M}_t^{(Z)}|^2\right] \\
&\le\frac{\delta^2}{\lambda^2}\cdot4\mathbb{E}[|\mathcal{M}_T^{(Z)}|^2] \le\frac{16KT\delta^2}{\lambda^2}.
\end{aligned} P ( 0 ≤ t ≤ T sup ∣ δ M t ( Z ) ∣ ≥ λ ) ≤ λ 2 δ 2 E [ 0 ≤ t ≤ T sup ∣ M t ( Z ) ∣ 2 ] ≤ λ 2 δ 2 ⋅ 4 E [ ∣ M T ( Z ) ∣ 2 ] ≤ λ 2 16 K T δ 2 .
Absorbing the numerical factor into the constant completes the proof.
Lemma D.7 (Discretization terms for Z t Z_t Z t ). Under Assumption D.1 and Lemma E.7, there exists K > 0 K>0 K > 0 such that for all T ∈ N T\in\mathbb{N} T ∈ N and all λ > 0 \lambda>0 λ > 0 ,
P ( E T ( T ) ∩ { sup t ≤ T ∣ R t ( Z ) ∣ ≥ λ } ) ≤ K T δ 3 d 3 / 2 λ . \mathbb{P}\left(E_T^{(T)}\cap\left\{\sup_{t\le T}|\mathcal{R}_t^{(Z)}|\ge\lambda\right\}\right)\le\frac{KT\delta^3d^{3/2}}{\lambda}. P ( E T ( T ) ∩ { t ≤ T sup ∣ R t ( Z ) ∣ ≥ λ } ) ≤ λ K T δ 3 d 3/2 . Consequently, under the assumptions of Lemma D.3,
P ( sup t ≤ T ∣ R t ( Z ) ∣ ≥ λ ) ≤ P ( ( E T ( T ) ) c ) + K T δ 3 d 3 / 2 λ . (102) \mathbb{P}\left(\sup_{t\le T}|\mathcal{R}_t^{(Z)}|\ge\lambda\right)\le\mathbb{P}\left((E_T^{(T)})^c\right)+\frac{KT\delta^3d^{3/2}}{\lambda}. \tag*{(102)} P ( t ≤ T sup ∣ R t ( Z ) ∣ ≥ λ ) ≤ P ( ( E T ( T ) ) c ) + λ K T δ 3 d 3/2 . (102) Proof. Let
r k ( Z ) : = R k ( ρ ) + R k ( χ ) , R t ( Z ) : = ∑ k = 0 t − 1 r k ( Z ) . r_k^{(Z)} := R_k^{(\rho)}+R_k^{(\chi)},\qquad\mathcal{R}_t^{(Z)} := \sum_{k=0}^{t-1}r_k^{(Z)}. r k ( Z ) := R k ( ρ ) + R k ( χ ) , R t ( Z ) := k = 0 ∑ t − 1 r k ( Z ) . On the Taylor event E T ( T ) E_T^{(T)} E T ( T ) , Lemma D.3 gives, simultaneously for all k < T k<T k < T ,
∣ R k ( ρ ) ∣ ≤ C δ 3 ∥ g k ( β ) ∥ 3 , ∣ R k ( χ ) ∣ ≤ C δ 3 ∥ g k ( θ ) ∥ 3 . |R_k^{(\rho)}|\le C\delta^3\|g_k^{(\beta)}\|^3,\qquad|R_k^{(\chi)}|\le C\delta^3\|g_k^{(\theta)}\|^3. ∣ R k ( ρ ) ∣ ≤ C δ 3 ∥ g k ( β ) ∥ 3 , ∣ R k ( χ ) ∣ ≤ C δ 3 ∥ g k ( θ ) ∥ 3 . Therefore, on E T ( T ) E_T^{(T)} E T ( T ) ,
∣ r k ( Z ) ∣ ≤ C δ 3 ( ∥ g k ( β ) ∥ 3 + ∥ g k ( θ ) ∥ 3 ) . |r_k^{(Z)}|\le C\delta^3\left(\|g_k^{(\beta)}\|^3+\|g_k^{(\theta)}\|^3\right). ∣ r k ( Z ) ∣ ≤ C δ 3 ( ∥ g k ( β ) ∥ 3 + ∥ g k ( θ ) ∥ 3 ) . Hence
E T ( T ) ∩ { sup t ≤ T ∣ R t ( Z ) ∣ ≥ λ } ⊆ { C δ 3 ∑ k < T ( ∥ g k ( β ) ∥ 3 + ∥ g k ( θ ) ∥ 3 ) ≥ λ } . E_T^{(T)}\cap\left\{\sup_{t\le T}|\mathcal{R}_t^{(Z)}|\ge\lambda\right\}\subseteq\left\{C\delta^3\sum_{k<T}\left(\|g_k^{(\beta)}\|^3+\|g_k^{(\theta)}\|^3\right)\ge\lambda\right\}. E T ( T ) ∩ { t ≤ T sup ∣ R t ( Z ) ∣ ≥ λ } ⊆ { C δ 3 k < T ∑ ( ∥ g k ( β ) ∥ 3 + ∥ g k ( θ ) ∥ 3 ) ≥ λ } . By Markov’s inequality,
P ( E T ( T ) ∩ { sup t ≤ T ∣ R t ( Z ) ∣ ≥ λ } ) ≤ C δ 3 ∑ k < T E [ ∥ g k ( β ) ∥ 3 + ∥ g k ( θ ) ∥ 3 ] λ . \mathbb{P}\left(E_T^{(T)}\cap\left\{\sup_{t\le T}|\mathcal{R}_t^{(Z)}|\ge\lambda\right\}\right)\le\frac{C\delta^3\sum_{k<T}\mathbb{E}\left[\|g_k^{(\beta)}\|^3+\|g_k^{(\theta)}\|^3\right]}{\lambda}. P ( E T ( T ) ∩ { t ≤ T sup ∣ R t ( Z ) ∣ ≥ λ } ) ≤ λ C δ 3 ∑ k < T E [ ∥ g k ( β ) ∥ 3 + ∥ g k ( θ ) ∥ 3 ] . By Hölder’s inequality and Lemma E.7,
E [ ∥ g k ( β ) ∥ 3 ∣ F k ] ≤ ( E [ ∥ g k ( β ) ∥ 4 ∣ F k ] ) 3 / 4 ≤ ( K d 2 ) 3 / 4 ≤ K d 3 / 2 . \mathbb{E}\left[\|g_k^{(\beta)}\|^3\mid\mathcal{F}_k\right]\le\left(\mathbb{E}\left[\|g_k^{(\beta)}\|^4\mid\mathcal{F}_k\right]\right)^{3/4}\le(Kd^2)^{3/4}\le Kd^{3/2}. E [ ∥ g k ( β ) ∥ 3 ∣ F k ] ≤ ( E [ ∥ g k ( β ) ∥ 4 ∣ F k ] ) 3/4 ≤ ( K d 2 ) 3/4 ≤ K d 3/2 . The same bound holds for g k ( θ ) g_k^{(\theta)} g k ( θ ) . Therefore
∑ k < T E [ ∥ g k ( β ) ∥ 3 + ∥ g k ( θ ) ∥ 3 ] ≤ K T d 3 / 2 . \sum_{k<T}\mathbb{E}\left[\|g_k^{(\beta)}\|^3+\|g_k^{(\theta)}\|^3\right]\le KTd^{3/2}. k < T ∑ E [ ∥ g k ( β ) ∥ 3 + ∥ g k ( θ ) ∥ 3 ] ≤ K T d 3/2 .
Thus
P ( E T ( T ) ∩ { sup t ≤ T ∣ R t ( Z ) ∣ ≥ λ } ) ≤ K T δ 3 d 3 / 2 λ . \mathbb{P}\left(E_T^{(T)} \cap\left\{\sup_{t\le T} |\mathcal{R}_t^{(Z)}| \ge\lambda\right\}\right) \le\frac{KT\delta^3d^{3/2}}{\lambda}. P ( E T ( T ) ∩ { t ≤ T sup ∣ R t ( Z ) ∣ ≥ λ } ) ≤ λ K T δ 3 d 3/2 .
Finally,
{ sup t ≤ T ∣ R t ( Z ) ∣ ≥ λ } ⊆ ( E T ( T ) ) c ∪ ( E T ( T ) ∩ { sup t ≤ T ∣ R t ( T ) ∣ ≥ λ } ) , \left\{\sup_{t\le T} |\mathcal{R}_t^{(Z)}| \ge\lambda\right\} \subseteq(E_T^{(T)})^c \cup\left(E_T^{(T)} \cap\left\{\sup_{t\le T} |\mathcal{R}_t^{(T)}| \ge\lambda\right\}\right), { t ≤ T sup ∣ R t ( Z ) ∣ ≥ λ } ⊆ ( E T ( T ) ) c ∪ ( E T ( T ) ∩ { t ≤ T sup ∣ R t ( T ) ∣ ≥ λ } ) ,
so
P ( sup t ≤ T ∣ R t ( Z ) ∣ ≥ λ ) ≤ P ( ( E T ( T ) ) c ) + K T δ 3 d 3 / 2 λ . \mathbb{P}\left(\sup_{t\le T} |\mathcal{R}_t^{(Z)}| \ge\lambda\right) \le\mathbb{P}\left((E_T^{(T)})^c\right) + \frac{KT\delta^3d^{3/2}}{\lambda}. P ( t ≤ T sup ∣ R t ( Z ) ∣ ≥ λ ) ≤ P ( ( E T ( T ) ) c ) + λ K T δ 3 d 3/2 .
The second displayed bound follows from Lemma D.3.
Remark D.8. Taking T δ ≤ c 2 log d T\delta\le c_2\log d T δ ≤ c 2 log d , δ ≤ c 0 ϵ / ( d log d ) \delta\le c_0\epsilon/(d\log d) δ ≤ c 0 ϵ / ( d log d ) , and λ = C 0 / d \lambda= C_0/\sqrt{d} λ = C 0 / d , Lemma D.7 gives
K T δ 3 d 3 / 2 λ = K T δ 3 d 2 C 0 ≤ K c 2 C 0 δ 2 d 2 log d ≤ K c 2 c 0 2 C 0 ϵ 2 log d . \frac{KT\delta^3d^{3/2}}{\lambda} = \frac{KT\delta^3d^2}{C_0} \le\frac{Kc_2}{C_0}\delta^2d^2\log d \le\frac{Kc_2c_0^2}{C_0}\frac{\epsilon^2}{\log d}. λ K T δ 3 d 3/2 = C 0 K T δ 3 d 2 ≤ C 0 K c 2 δ 2 d 2 log d ≤ C 0 K c 2 c 0 2 log d ϵ 2 .
Together with Lemma D.3,
P ( ( E T ( T ) ) c ) ≤ K T a y ϵ 3 d log 2 d . \mathbb{P}\left((E_T^{(T)})^c\right) \le K_{\mathrm{Tay}}\frac{\epsilon^3}{d\log^2 d}. P ( ( E T ( T ) ) c ) ≤ K Tay d log 2 d ϵ 3 .
Hence the discretization contribution is absorbed into the existing O ( ϵ ) O(\epsilon) O ( ϵ ) good-event failure probability.
Good Events for $Z_t Now, we formalize the definition and the high-likelihood of good events for Z t Z_t Z t iterate: From the two-layer aggregate martingale and discretization remainder for each time step defined as M t ( Z ) : = M t ( ρ ) + M t ( χ ) M_t^{(Z)} := M_t^{(\rho)} + M_t^{(\chi)} M t ( Z ) := M t ( ρ ) + M t ( χ ) , r t ( Z ) : = R t ( ρ ) + R t ( χ ) r_t^{(Z)} := R_t^{(\rho)} + R_t^{(\chi)} r t ( Z ) := R t ( ρ ) + R t ( χ ) , we write their time accumulation as
M t ( Z ) : = ∑ k < t M k ( Z ) , R t ( Z ) : = ∑ k < t r k ( Z ) , N t ( Z ) : = δ M t ( Z ) + R t ( Z ) . \mathcal{M}_t^{(Z)} := \sum_{k<t} M_k^{(Z)}, \qquad\mathcal{R}_t^{(Z)} := \sum_{k<t} r_k^{(Z)}, \qquad N_t^{(Z)} := \delta\mathcal{M}_t^{(Z)} + \mathcal{R}_t^{(Z)}. M t ( Z ) := k < t ∑ M k ( Z ) , R t ( Z ) := k < t ∑ r k ( Z ) , N t ( Z ) := δ M t ( Z ) + R t ( Z ) .
For a threshold λ Z = C 0 / d \lambda_Z = C_0/\sqrt{d} λ Z = C 0 / d , define
E T ( Z , M ) : = { sup t ≤ T ∣ δ M t ( Z ) ∣ ≤ λ Z } , E T ( Z , R ) : = { sup t ≤ T ∣ R t ( Z ) ∣ ≤ λ Z } , E_T^{(Z,M)} := \left\{\sup_{t\le T} |\delta\mathcal{M}_t^{(Z)}| \le\lambda_Z\right\}, \qquad E_T^{(Z,R)} := \left\{\sup_{t\le T} |\mathcal{R}_t^{(Z)}| \le\lambda_Z\right\}, E T ( Z , M ) := { t ≤ T sup ∣ δ M t ( Z ) ∣ ≤ λ Z } , E T ( Z , R ) := { t ≤ T sup ∣ R t ( Z ) ∣ ≤ λ Z } ,
and
E T ( Z ) : = E T ( Z , M ) ∩ E T ( Z , R ) . (103) E_T^{(Z)} := E_T^{(Z,M)} \cap E_T^{(Z,R)}. \tag*{(103)} E T ( Z ) := E T ( Z , M ) ∩ E T ( Z , R ) . (103)
Then on E T ( Z ) E_T^{(Z)} E T ( Z ) ,
sup t ≤ T ∣ N t ( Z ) ∣ ≤ 2 λ Z . (104) \sup_{t\le T} |N_t^{(Z)}| \le2\lambda_Z. \tag*{(104)} t ≤ T sup ∣ N t ( Z ) ∣ ≤ 2 λ Z . (104)
By Lemma D.5 and Lemma D.7, together with Lemma D.3, we have
P ( ( E T ( Z ) ) c ) ≤ P ( ( E T ( Z , M ) ) c ) + P ( ( E T ( Z , R ) ) c ) ≤ K ϵ , (105) \mathbb{P}\left((E_T^{(Z)})^c\right) \le\mathbb{P}\left((E_T^{(Z,M)})^c\right) + \mathbb{P}\left((E_T^{(Z,R)})^c\right) \le K\epsilon, \tag*{(105)} P ( ( E T ( Z ) ) c ) ≤ P ( ( E T ( Z , M ) ) c ) + P ( ( E T ( Z , R ) ) c ) ≤ K ϵ , (105)
for some constant K K K .
Good Events for D t D_t D t Besides, we keep track of the good events for D t : = ρ t − χ t D_t := \rho_t - \chi_t D t := ρ t − χ t : for a ∈ { ρ , χ } a \in\{\rho,\chi\} a ∈ { ρ , χ } , we have the individual-layer time accumulation as:
M t ( a ) : = ∑ k < t M k ( a ) , R t ( a ) : = ∑ k < t R k ( a ) , N t ( a ) : = δ M t ( a ) + R t ( a ) . \mathcal{M}_t^{(a)} := \sum_{k<t} M_k^{(a)}, \qquad\mathcal{R}_t^{(a)} := \sum_{k<t} R_k^{(a)}, \qquad N_t^{(a)} := \delta\mathcal{M}_t^{(a)} + \mathcal{R}_t^{(a)}. M t ( a ) := k < t ∑ M k ( a ) , R t ( a ) := k < t ∑ R k ( a ) , N t ( a ) := δ M t ( a ) + R t ( a ) .
We then track their difference N t ( D ) : = N t ( ρ ) − N t ( χ ) N_t^{(D)} := N_t^{(\rho)} - N_t^{(\chi)} N t ( D ) := N t ( ρ ) − N t ( χ ) . With parameter λ ρ = λ χ : = C i n d / d \lambda_\rho= \lambda_\chi:= C_{\mathrm{ind}}/\sqrt{d} λ ρ = λ χ := C ind / d , λ D : = λ ρ + λ χ \lambda_D := \lambda_\rho+ \lambda_\chi λ D := λ ρ + λ χ , we define the good events here as
E T ( ρ , χ ) : = { sup t ≤ T ∣ N t ( ρ ) ∣ ≤ 2 λ ρ } ∩ { sup t ≤ T ∣ N t ( χ ) ∣ ≤ 2 λ χ } , (106) E_T^{(\rho,\chi)} := \left\{\sup_{t\le T}|N_t^{(\rho)}| \le2\lambda_\rho\right\} \cap\left\{\sup_{t\le T}|N_t^{(\chi)}| \le2\lambda_\chi\right\}, \tag*{(106)} E T ( ρ , χ ) := { t ≤ T sup ∣ N t ( ρ ) ∣ ≤ 2 λ ρ } ∩ { t ≤ T sup ∣ N t ( χ ) ∣ ≤ 2 λ χ } , (106)
So on E T ( ρ , χ ) E_T^{(\rho,\chi)} E T ( ρ , χ ) ,
sup t ≤ T ∣ N t ( D ) ∣ ≤ sup t ≤ T ∣ N t ( ρ ) ∣ + sup t ≤ T ∣ N t ( χ ) ∣ ≤ 2 ( λ ρ + λ χ ) = 2 λ D . \sup_{t\le T}|N_t^{(D)}| \le\sup_{t\le T}|N_t^{(\rho)}| + \sup_{t\le T}|N_t^{(\chi)}| \le2(\lambda_\rho+\lambda_\chi) = 2\lambda_D. t ≤ T sup ∣ N t ( D ) ∣ ≤ t ≤ T sup ∣ N t ( ρ ) ∣ + t ≤ T sup ∣ N t ( χ ) ∣ ≤ 2 ( λ ρ + λ χ ) = 2 λ D .
By the individual versions of Lemmas D.5 and D.7, choosing C i n d C_{\mathrm{ind}} C ind large enough gives
P ( ( E T ( ρ , χ ) ) c ) ≤ K ϵ . (107) \mathbb{P}\left((E_T^{(\rho,\chi)})^c\right) \le K\epsilon. \tag*{(107)} P ( ( E T ( ρ , χ ) ) c ) ≤ K ϵ . (107)
Martingale and Discretization Error Control for η t \eta_t η t
Deriving the discrete-time evolution η t \eta_t η t requires a different strategy compares to that of Z t Z_t Z t . To start with, we derive the one-step evolution η t \eta_t η t :
Lemma D.9 (Evolution of η t \eta_t η t ). Work under Assumption D.1. Let g t ( θ ) : = ∇ S θ ℓ ( β t , θ t ; x t ) g_t^{(\theta)} := \nabla_{S}^{\theta}\ell(\beta_t,\theta_t;x_t) g t ( θ ) := ∇ S θ ℓ ( β t , θ t ; x t ) so that g t ( θ ) ⊥ θ t g_t^{(\theta)} \perp\theta_t g t ( θ ) ⊥ θ t , and define the mean-field spherical gradient g ˉ t ( θ ) : = E [ g t ( θ ) ∣ F t ] \bar{g}_t^{(\theta)} := \mathbb{E}[g_t^{(\theta)} \mid\mathcal{F}_t] g ˉ t ( θ ) := E [ g t ( θ ) ∣ F t ] and Δ g t ( θ ) : = g t ( θ ) − g ˉ t ( θ ) \Delta g_t^{(\theta)} := g_t^{(\theta)}-\bar{g}_t^{(\theta)} Δ g t ( θ ) := g t ( θ ) − g ˉ t ( θ ) . Then for all t t t , we have
∥ θ t ∥ ∞ ≤ ∥ θ 0 ∥ ∞ + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ + max i ∣ δ ∑ k < t ξ k , i ∣ + δ 2 2 ∑ k < t ∥ g k ( θ ) ∥ 2 2 ∥ θ k ∥ ∞ + δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 . (108) \|\theta_t\|_\infty\le\|\theta_0\|_\infty+\delta\sum_{k<t}\|\bar{g}_k^{(\theta)}\|_\infty+\max_i\left|\delta\sum_{k<t}\xi_{k,i}\right|+\frac{\delta^2}{2}\sum_{k<t}\|g_k^{(\theta)}\|_2^2\|\theta_k\|_\infty+\frac{\delta^3}{2}\sum_{k<t}\|g_k^{(\theta)}\|_2^3. \tag*{(108)} ∥ θ t ∥ ∞ ≤ ∥ θ 0 ∥ ∞ + δ k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + i max δ k < t ∑ ξ k , i + 2 δ 2 k < t ∑ ∥ g k ( θ ) ∥ 2 2 ∥ θ k ∥ ∞ + 2 δ 3 k < t ∑ ∥ g k ( θ ) ∥ 2 3 . (108)
Proof. Since g t ( θ ) ⊥ θ t g_t^{(\theta)}\perp\theta_t g t ( θ ) ⊥ θ t and ∥ θ t ∥ 2 = 1 \|\theta_t\|_2=1 ∥ θ t ∥ 2 = 1 , we have a t : = 1 / ∥ θ t − δ g t ( θ ) ∥ 2 = 1 / 1 + δ 2 ∥ g t ( θ ) ∥ 2 2 a_t:=1/\|\theta_t-\delta g_t^{(\theta)}\|_2=1/\sqrt{1+\delta^2\|g_t^{(\theta)}\|_2^2} a t := 1/∥ θ t − δ g t ( θ ) ∥ 2 = 1/ 1 + δ 2 ∥ g t ( θ ) ∥ 2 2 and
θ t + 1 = a t ( θ t − δ g t ( θ ) ) . \theta_{t+1}=a_t(\theta_t-\delta g_t^{(\theta)}). θ t + 1 = a t ( θ t − δ g t ( θ ) ) .
Define the "telescoping correlation"
r t = θ t + 1 − ( θ t − δ g t ( θ ) ) = ( a t − 1 ) ( θ t − δ g t ( θ ) ) , (109) r_t=\theta_{t+1}-(\theta_t-\delta g_t^{(\theta)})=(a_t-1)(\theta_t-\delta g_t^{(\theta)}), \tag*{(109)} r t = θ t + 1 − ( θ t − δ g t ( θ ) ) = ( a t − 1 ) ( θ t − δ g t ( θ ) ) , (109)
so as θ t + 1 = θ t − δ g t ( θ ) + r t \theta_{t+1}=\theta_t-\delta g_t^{(\theta)}+r_t θ t + 1 = θ t − δ g t ( θ ) + r t , under time-cumulation we have
θ t = θ 0 − δ ∑ k < t g k ( θ ) + ∑ k < t r k . \theta_t=\theta_0-\delta\sum_{k<t}g_k^{(\theta)}+\sum_{k<t}r_k. θ t = θ 0 − δ k < t ∑ g k ( θ ) + k < t ∑ r k .
Now we decompose the gradient g k ( θ ) g_k^{(\theta)} g k ( θ ) as the mean-field and martingale noise g k ( θ ) = g ˉ k ( θ ) + ξ k ( θ ) g_k^{(\theta)}=\bar{g}_k^{(\theta)}+\xi_k^{(\theta)} g k ( θ ) = g ˉ k ( θ ) + ξ k ( θ ) , where g ˉ k : = E [ g k ( θ ) ∣ F k ] \bar{g}_k:=\mathbb{E}[g_k^{(\theta)}\mid\mathcal{F}_k] g ˉ k := E [ g k ( θ ) ∣ F k ] and ξ k : = g k − g ˉ k \xi_k:=g_k-\bar{g}_k ξ k := g k − g ˉ k . Thus, taking the l ∞ l_\infty l ∞ -norm
∥ θ t ∥ ∞ ≤ ∥ θ 0 ∥ ∞ + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ + max i ∣ δ ∑ k < t ξ k , i ( θ ) ∣ + ∑ k < t ∥ r k ( θ ) ∥ ∞ (110) \|\theta_t\|_\infty\le\|\theta_0\|_\infty+\delta\sum_{k<t}\|\bar{g}_k^{(\theta)}\|_\infty+\max_i\left|\delta\sum_{k<t}\xi_{k,i}^{(\theta)}\right|+\sum_{k<t}\|r_k^{(\theta)}\|_\infty \tag*{(110)} ∥ θ t ∥ ∞ ≤ ∥ θ 0 ∥ ∞ + δ k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + i max δ k < t ∑ ξ k , i ( θ ) + k < t ∑ ∥ r k ( θ ) ∥ ∞ (110)
To bound ∥ r t ( θ ) ∥ ∞ \|r_t^{(\theta)}\|_\infty ∥ r t ( θ ) ∥ ∞ , taking l ∞ l_\infty l ∞ -norm of (107):
∥ r t ( θ ) ∥ ∞ ≤ ∣ a t − 1 ∣ ⋅ ∥ θ t − δ g t ( θ ) ∥ ∞ (111) \|r_t^{(\theta)}\|_\infty\le|a_t-1|\cdot\|\theta_t-\delta g_t^{(\theta)}\|_\infty \tag*{(111)} ∥ r t ( θ ) ∥ ∞ ≤ ∣ a t − 1∣ ⋅ ∥ θ t − δ g t ( θ ) ∥ ∞ (111)
≤ ∣ 1 − a t ∣ ⋅ ( ∥ θ t ∥ ∞ + δ ∥ g t ( θ ) ∥ ∞ ) (112) \le|1-a_t|\cdot(\|\theta_t\|_\infty+\delta\|g_t^{(\theta)}\|_\infty) \tag*{(112)} ≤ ∣1 − a t ∣ ⋅ ( ∥ θ t ∥ ∞ + δ ∥ g t ( θ ) ∥ ∞ ) (112)
≤ δ 2 2 ∥ g t ( θ ) ∥ 2 2 ⋅ ( ∥ θ t ∥ ∞ + δ ∥ g t ( θ ) ∥ ∞ ) , since for x ≥ 0 , 1 + x ≥ 1 + x / 2. (113) \le\frac{\delta^2}{2}\|g_t^{(\theta)}\|_2^2\cdot(\|\theta_t\|_\infty+\delta\|g_t^{(\theta)}\|_\infty),\text{ since for }x\ge0,\sqrt{1+x}\ge1+x/2. \tag*{(113)} ≤ 2 δ 2 ∥ g t ( θ ) ∥ 2 2 ⋅ ( ∥ θ t ∥ ∞ + δ ∥ g t ( θ ) ∥ ∞ ) , since for x ≥ 0 , 1 + x ≥ 1 + x /2. (113)
= δ 2 2 ∥ g t ( θ ) ∥ 2 2 ∥ θ t ∥ ∞ + δ 3 2 ∥ g t ( θ ) ∥ 2 3 , given ∥ g t ( θ ) ∥ ∞ ≤ ∥ g t ( θ ) ∥ . (114) =\frac{\delta^2}{2}\|g_t^{(\theta)}\|_2^2\|\theta_t\|_\infty+\frac{\delta^3}{2}\|g_t^{(\theta)}\|_2^3,\text{ given }\|g_t^{(\theta)}\|_\infty\le\|g_t^{(\theta)}\|. \tag*{(114)} = 2 δ 2 ∥ g t ( θ ) ∥ 2 2 ∥ θ t ∥ ∞ + 2 δ 3 ∥ g t ( θ ) ∥ 2 3 , given ∥ g t ( θ ) ∥ ∞ ≤ ∥ g t ( θ ) ∥. (114)
So we have the bound
∥ θ t ∥ ∞ ≤ ∥ θ 0 ∥ ∞ + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ + max i ∣ δ ∑ k < t ξ k , i ( θ ) ∣ + δ 2 2 ∑ k < t ∥ g k ( θ ) ∥ 2 2 ∥ θ k ∥ ∞ + δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 . (115) \|\theta_t\|_\infty\le\|\theta_0\|_\infty+\delta\sum_{k<t}\|\bar{g}_k^{(\theta)}\|_\infty+\max_i\left|\delta\sum_{k<t}\xi_{k,i}^{(\theta)}\right|+\frac{\delta^2}{2}\sum_{k<t}\|g_k^{(\theta)}\|_2^2\|\theta_k\|_\infty+\frac{\delta^3}{2}\sum_{k<t}\|g_k^{(\theta)}\|_2^3. \tag*{(115)} ∥ θ t ∥ ∞ ≤ ∥ θ 0 ∥ ∞ + δ k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + i max δ k < t ∑ ξ k , i ( θ ) + 2 δ 2 k < t ∑ ∥ g k ( θ ) ∥ 2 2 ∥ θ k ∥ ∞ + 2 δ 3 k < t ∑ ∥ g k ( θ ) ∥ 2 3 . (115)
Now, we analyze the aggregation through time: let Θ t : = max s ≤ t ∥ θ s ∥ ∞ \Theta_t:=\max_{s\le t}\|\theta_s\|_\infty Θ t := max s ≤ t ∥ θ s ∥ ∞ to be the maximum over the trajectory, we then have for any t ≤ T t\le T t ≤ T ,
Θ t ≤ ∥ θ 0 ∥ ∞ + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ + max i sup s ≤ t ∣ δ ∑ k < s ξ k , i ( θ ) ∣ + ( δ 2 2 ∑ k < t ∥ g k ( θ ) ∥ 2 2 ) Θ t + δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 . (116) \Theta_t\le\|\theta_0\|_\infty+\delta\sum_{k<t}\|\bar{g}_k^{(\theta)}\|_\infty+\max_i\sup_{s\le t}\left|\delta\sum_{k<s}\xi_{k,i}^{(\theta)}\right|+\left(\frac{\delta^2}{2}\sum_{k<t}\|g_k^{(\theta)}\|_2^2\right)\Theta_t+\frac{\delta^3}{2}\sum_{k<t}\|g_k^{(\theta)}\|_2^3. \tag*{(116)} Θ t ≤ ∥ θ 0 ∥ ∞ + δ k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + i max s ≤ t sup δ k < s ∑ ξ k , i ( θ ) + ( 2 δ 2 k < t ∑ ∥ g k ( θ ) ∥ 2 2 ) Θ t + 2 δ 3 k < t ∑ ∥ g k ( θ ) ∥ 2 3 . (116)
To bound term 3 in (114), we have by Markov Inequality,
P ( δ 2 2 ∑ k < T ∥ g k ( θ ) ∥ 2 2 ≥ λ ) ≤ E [ δ 2 2 ∑ k < T ∥ g k ( θ ) ∥ 2 2 ] λ ≤ δ 2 T K d λ . \mathbb{P}\left(\frac{\delta^2}{2}\sum_{k<T}\lVert g_k^{(\theta)}\rVert_2^2\ge\lambda\right)\le\frac{\mathbb{E}\left[\frac{\delta^2}{2}\sum_{k<T}\lVert g_k^{(\theta)}\rVert_2^2\right]}{\lambda}\le\frac{\delta^2TKd}{\lambda}. P ( 2 δ 2 k < T ∑ ∥ g k ( θ ) ∥ 2 2 ≥ λ ) ≤ λ E [ 2 δ 2 ∑ k < T ∥ g k ( θ ) ∥ 2 2 ] ≤ λ δ 2 T K d .
Taking λ = 1 / 4 \lambda=1/4 λ = 1/4 , we have
P ( δ 2 2 ∑ k < T ∥ g k ( θ ) ∥ 2 2 ≥ 1 4 ) = O ( ε ) , (117) \mathbb{P}\left(\frac{\delta^2}{2}\sum_{k<T}\lVert g_k^{(\theta)}\rVert_2^2\ge\frac14\right)=O(\varepsilon), \tag*{(117)} P ( 2 δ 2 k < T ∑ ∥ g k ( θ ) ∥ 2 2 ≥ 4 1 ) = O ( ε ) , (117)
giving that w.p. 1 − O ( ε ) 1-O(\varepsilon) 1 − O ( ε ) ,
Θ t ≤ ∥ θ 0 ∥ ∞ + δ ∑ k < t ∥ g k ( θ ) ∥ ∞ ⏟ 1 + max i sup s ≤ t ∣ δ ∑ k < s ξ k , i ( θ ) ∣ ⏟ 2 + 1 4 Θ t + δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 ⏟ 4 . \Theta_t\le\lVert\theta_0\rVert_\infty+\delta\underbrace{\sum_{k<t}\lVert g_k^{(\theta)}\rVert_\infty}_{1}+\underbrace{\max_i\sup_{s\le t}\left|\delta\sum_{k<s}\xi_{k,i}^{(\theta)}\right|}_{2}+\frac14\Theta_t+\frac{\delta^3}{2}\underbrace{\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3}_{4}. Θ t ≤ ∥ θ 0 ∥ ∞ + δ 1 k < t ∑ ∥ g k ( θ ) ∥ ∞ + 2 i max s ≤ t sup δ k < s ∑ ξ k , i ( θ ) + 4 1 Θ t + 2 δ 3 4 k < t ∑ ∥ g k ( θ ) ∥ 2 3 .
( 1 − 1 4 ) Θ t ≤ ∥ θ 0 ∥ ∞ + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ ⏟ 1 + max i sup s ≤ t ∣ δ ∑ k < s ξ k , i ( θ ) ∣ ⏟ 2 + δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 ⏟ 4 . \left(1-\frac14\right)\Theta_t\le\lVert\theta_0\rVert_\infty+\delta\underbrace{\sum_{k<t}\lVert\bar g_k^{(\theta)}\rVert_\infty}_{1}+\underbrace{\max_i\sup_{s\le t}\left|\delta\sum_{k<s}\xi_{k,i}^{(\theta)}\right|}_{2}+\frac{\delta^3}{2}\underbrace{\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3}_{4}. ( 1 − 4 1 ) Θ t ≤ ∥ θ 0 ∥ ∞ + δ 1 k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + 2 i max s ≤ t sup δ k < s ∑ ξ k , i ( θ ) + 2 δ 3 4 k < t ∑ ∥ g k ( θ ) ∥ 2 3 .
Θ t ≤ 4 3 ( ∥ θ 0 ∥ ∞ + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ ⏟ 1 + max i sup s ≤ t ∣ δ ∑ k < s ξ k , i ( θ ) ∣ ⏟ 2 + δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 ⏟ 4 ) . (118) \Theta_t\le\frac43\left(\lVert\theta_0\rVert_\infty+\underbrace{\delta\sum_{k<t}\lVert\bar g_k^{(\theta)}\rVert_\infty}_{1}+\underbrace{\max_i\sup_{s\le t}\left|\delta\sum_{k<s}\xi_{k,i}^{(\theta)}\right|}_{2}+\frac{\delta^3}{2}\underbrace{\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3}_{4}\right). \tag*{(118)} Θ t ≤ 3 4 ∥ θ 0 ∥ ∞ + 1 δ k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + 2 i max s ≤ t sup δ k < s ∑ ξ k , i ( θ ) + 2 δ 3 4 k < t ∑ ∥ g k ( θ ) ∥ 2 3 . (118)
We start by bounding term 2 using a Freedman Inequality. Yet to show it, we need this corollary below:
Corollary D.10 (Coordinate variance and 8th-moment bounds for ξ ( θ ) \xi^{(\theta)} ξ ( θ ) ). Work under Assumption D.1 and Lemma E.7. Let ξ t ( θ ) : = g t ( θ ) − E [ g t ( θ ) ∣ F t ] \xi_t^{(\theta)}:=g_t^{(\theta)}-\mathbb{E}[g_t^{(\theta)}\mid\mathcal{F}_t] ξ t ( θ ) := g t ( θ ) − E [ g t ( θ ) ∣ F t ] and write ξ t , i ( θ ) \xi_{t,i}^{(\theta)} ξ t , i ( θ ) for its i i i -th coordinate. Then there exist constants σ 2 > 0 \sigma^2>0 σ 2 > 0 and K 8 > 0 K_8>0 K 8 > 0 (depending only on model constants) such that for all t t t and all i ≤ d i\le d i ≤ d .
E [ ( ξ t , i ( θ ) ) 2 ∣ F t ] ≤ σ 2 , E [ ∣ ξ t , i ( θ ) ∣ 8 ∣ F t ] ≤ K 8 . (119) \mathbb{E}\left[(\xi_{t,i}^{(\theta)})^2\mid\mathcal{F}_t\right]\le\sigma^2,\qquad\mathbb{E}\left[|\xi_{t,i}^{(\theta)}|^8\mid\mathcal{F}_t\right]\le K_8. \tag*{(119)} E [ ( ξ t , i ( θ ) ) 2 ∣ F t ] ≤ σ 2 , E [ ∣ ξ t , i ( θ ) ∣ 8 ∣ F t ] ≤ K 8 . (119)
Proof. Conditional variance. Lemma E.7 gives the directional variance bound
sup ∥ u ∥ = 1 E [ ( ( Δ g t ( θ ) ) ⋅ u ) 2 ∣ F t ] ≤ K . \sup_{\lVert u\rVert=1}\mathbb{E}\left[\left((\Delta g_t^{(\theta)})\cdot u\right)^2\mid\mathcal{F}_t\right]\le K. ∥ u ∥ = 1 sup E [ ( ( Δ g t ( θ ) ) ⋅ u ) 2 ∣ F t ] ≤ K .
Taking u = e i u=e_i u = e i yields E [ ( ξ t , i ( θ ) ) 2 ∣ F t ] ≤ K \mathbb{E}[(\xi_{t,i}^{(\theta)})^2\mid\mathcal{F}_t]\le K E [( ξ t , i ( θ ) ) 2 ∣ F t ] ≤ K . Set σ 2 : = K \sigma^2:=K σ 2 := K .
8-th moment. Recall that in Lemma E.7, from the per-sample gradient ∇ θ ℓ ( β t , θ t ; x t ) = A t z t \nabla_\theta\ell(\beta_t,\theta_t;x_t)=A_tz_t ∇ θ ℓ ( β t , θ t ; x t ) = A t z t with A t : = 2 ( f t − f t ∗ ) ψ ′ ( z t ⋅ θ t ) A_t:=2(f_t-f_t^*)\psi'(z_t\cdot\theta_t) A t := 2 ( f t − f t ∗ ) ψ ′ ( z t ⋅ θ t ) defined in (215), using the boundedness of ψ , ψ ′ \psi,\psi' ψ , ψ ′ we showed ∣ A t ∣ ≤ C A |A_t|\le C_A ∣ A t ∣ ≤ C A in (219). From there, the spherical gradient is
g t ( θ ) = ( I − θ t θ t T ) ∇ θ ℓ = A t ( z t − ( z t ⋅ θ t ) θ t ) , g_t^{(\theta)}=(I-\theta_t\theta_t^\mathsf{T})\nabla_\theta\ell=A_t\left(z_t-(z_t\cdot\theta_t)\theta_t\right), g t ( θ ) = ( I − θ t θ t T ) ∇ θ ℓ = A t ( z t − ( z t ⋅ θ t ) θ t ) ,
so for each coordinate
∣ g t , i ( θ ) ∣ ≤ ∣ A t ∣ ( ∣ z t , i ∣ + ∣ z t ⋅ θ t ∣ ) ≤ C A ( ∣ z t , i ∣ + ∣ z t ⋅ θ t ∣ ) . (120) |g_{t,i}^{(\theta)}|\le|A_t|\left(|z_{t,i}|+|z_t\cdot\theta_t|\right)\le C_A\left(|z_{t,i}|+|z_t\cdot\theta_t|\right). \tag*{(120)} ∣ g t , i ( θ ) ∣ ≤ ∣ A t ∣ ( ∣ z t , i ∣ + ∣ z t ⋅ θ t ∣ ) ≤ C A ( ∣ z t , i ∣ + ∣ z t ⋅ θ t ∣ ) . (120)
Condition on F t \mathcal{F}_t F t . Then θ t \theta_t θ t is deterministic with ∥ θ t ∥ 2 = 1 \lVert\theta_t\rVert_2=1 ∥ θ t ∥ 2 = 1 and z t = ( z t , 1 , … , z t , d ) z_t=(z_{t,1},\ldots,z_{t,d}) z t = ( z t , 1 , … , z t , d ) has i.i.d. coordinates z t , j = ϕ ( γ t , j ) z_{t,j}=\phi(\gamma_{t,j}) z t , j = ϕ ( γ t , j ) with γ t , j ∼ N ( 0 , 1 ) \gamma_{t,j}\sim\mathcal{N}(0,1) γ t , j ∼ N ( 0 , 1 ) . In particular,
μ 2 : = E [ z t , 1 2 ] = E [ ϕ ( γ ) 2 ] < ∞ , μ 8 : = E [ ∣ z t , 1 ∣ 8 ] = E [ ∣ ϕ ( γ ) ∣ 8 ] < ∞ . \mu_2:=\mathbb{E}[z_{t,1}^2]=\mathbb{E}[\phi(\gamma)^2]<\infty,\qquad\mu_8:=\mathbb{E}[|z_{t,1}|^8]=\mathbb{E}[|\phi(\gamma)|^8]<\infty. μ 2 := E [ z t , 1 2 ] = E [ ϕ ( γ ) 2 ] < ∞ , μ 8 := E [ ∣ z t , 1 ∣ 8 ] = E [ ∣ ϕ ( γ ) ∣ 8 ] < ∞.
Let X j : = θ t , j z t , j X_j:=\theta_{t,j}z_{t,j} X j := θ t , j z t , j . Then ( X j ) j ≤ d (X_j)_{j\le d} ( X j ) j ≤ d are independent and mean-zero, and
∑ j = 1 d E [ X j 2 ∣ F t ] = ∑ j = 1 d θ t , j 2 E [ z t , 1 2 ] = μ 2 ∑ j = 1 d θ t , j 2 = μ 2 . \sum_{j=1}^{d}\mathbb{E}[X_j^2\mid\mathcal{F}_t]=\sum_{j=1}^{d}\theta_{t,j}^2\mathbb{E}[z_{t,1}^2]=\mu_2\sum_{j=1}^{d}\theta_{t,j}^2=\mu_2. j = 1 ∑ d E [ X j 2 ∣ F t ] = j = 1 ∑ d θ t , j 2 E [ z t , 1 2 ] = μ 2 j = 1 ∑ d θ t , j 2 = μ 2 .
By Rosenthal’s inequality for p = 8 p = 8 p = 8 , there exists a constant C 8 > 0 C_8 > 0 C 8 > 0 such that
E [ ∣ ∑ j = 1 d X j ∣ 8 | F t ] ≤ C 8 ( ( ∑ j = 1 d E [ ∣ X j ∣ 2 ∣ F t ] ) 4 + ∑ j = 1 d E [ ∣ X j ∣ 8 ∣ F t ] ) . \mathbb{E}\left[\left|\sum_{j=1}^{d} X_j\right|^8 \middle| \mathcal{F}_t\right] \le C_8\left(\left(\sum_{j=1}^{d} \mathbb{E}[|X_j|^2 \mid\mathcal{F}_t]\right)^4 + \sum_{j=1}^{d} \mathbb{E}[|X_j|^8 \mid\mathcal{F}_t]\right). E j = 1 ∑ d X j 8 F t ≤ C 8 ( j = 1 ∑ d E [ ∣ X j ∣ 2 ∣ F t ] ) 4 + j = 1 ∑ d E [ ∣ X j ∣ 8 ∣ F t ] .
Moreover,
∑ j = 1 d E [ ∣ X j ∣ 8 ∣ F t ] = ∑ j = 1 d ∣ θ t , j ∣ 8 E [ ∣ z t , 1 ∣ 8 ] = μ 8 ∑ j = 1 d ∣ θ t , j ∣ 8 ≤ μ 8 ∑ j = 1 d θ t , j 2 = μ 8 , \sum_{j=1}^{d} \mathbb{E}[|X_j|^8 \mid\mathcal{F}_t] = \sum_{j=1}^{d} |\theta_{t,j}|^8 \mathbb{E}[|z_{t,1}|^8] = \mu_8 \sum_{j=1}^{d} |\theta_{t,j}|^8 \le\mu_8 \sum_{j=1}^{d} \theta_{t,j}^2 = \mu_8, j = 1 ∑ d E [ ∣ X j ∣ 8 ∣ F t ] = j = 1 ∑ d ∣ θ t , j ∣ 8 E [ ∣ z t , 1 ∣ 8 ] = μ 8 j = 1 ∑ d ∣ θ t , j ∣ 8 ≤ μ 8 j = 1 ∑ d θ t , j 2 = μ 8 ,
since ∣ θ t , j ∣ ≤ 1 |\theta_{t,j}| \le1 ∣ θ t , j ∣ ≤ 1 implies ∣ θ t , j ∣ 8 ≤ ∣ θ t , j ∣ 2 |\theta_{t,j}|^8 \le|\theta_{t,j}|^2 ∣ θ t , j ∣ 8 ≤ ∣ θ t , j ∣ 2 . Therefore,
E [ ∣ z t ⋅ θ t ∣ 8 ∣ F t ] = E [ ∣ ∑ j = 1 d X j ∣ 8 | F t ] ≤ C 8 ( μ 2 4 + μ 8 ) = : C , \mathbb{E}[|z_t \cdot\theta_t|^8 \mid\mathcal{F}_t] = \mathbb{E}\left[\left|\sum_{j=1}^{d} X_j\right|^8 \middle| \mathcal{F}_t\right] \le C_8(\mu_2^4+\mu_8) =: C, E [ ∣ z t ⋅ θ t ∣ 8 ∣ F t ] = E j = 1 ∑ d X j 8 F t ≤ C 8 ( μ 2 4 + μ 8 ) =: C ,
which is a constant independent of t t t and d d d . Hence, plugging into (120) , we have E [ ∣ g t , i ( θ ) ∣ 8 ∣ F t ] ≤ K 8 ′ \mathbb{E}[|g_{t,i}^{(\theta)}|^8 \mid\mathcal{F}_t] \le K_8' E [ ∣ g t , i ( θ ) ∣ 8 ∣ F t ] ≤ K 8 ′ uniformly.
Finally, by Jensen and ( a − b ) 8 ≤ 2 7 ( a 8 + b 8 ) (a-b)^8 \le2^7(a^8+b^8) ( a − b ) 8 ≤ 2 7 ( a 8 + b 8 ) ,
E [ ∣ ξ t , i ( θ ) ∣ 8 ∣ F t ] ≤ 2 8 E [ ∣ g t , i ( θ ) ∣ 8 ∣ F t ] ≤ 2 8 K 8 ′ . \mathbb{E}[|\xi_{t,i}^{(\theta)}|^8 \mid\mathcal{F}_t] \le2^8\mathbb{E}[|g_{t,i}^{(\theta)}|^8 \mid\mathcal{F}_t] \le2^8K_8'. E [ ∣ ξ t , i ( θ ) ∣ 8 ∣ F t ] ≤ 2 8 E [ ∣ g t , i ( θ ) ∣ 8 ∣ F t ] ≤ 2 8 K 8 ′ .
Set K 8 : = 2 8 K 8 ′ K_8 := 2^8K_8' K 8 := 2 8 K 8 ′ gives the claimed result. □
Now we move to prove the Freedman bound:
Lemma D.11 (Coordinate martingale control for the η \eta η -bootstrap). Assume Corollary D.10. Let T δ ≤ C T log d T\delta\le C_T \log d T δ ≤ C T log d and δ ≤ c 0 ε / ( d log d ) \delta\le c_0\varepsilon/(d\log d) δ ≤ c 0 ε / ( d log d ) . Then there is a constant C M C_M C M such that
P ( max i ≤ d sup t ≤ T ∣ δ ∑ k < t ξ k , i ( θ ) ∣ > C M log ( 40 d / ε ) d ) ≤ ε / 5. (121) \mathbb{P}\left(\max_{i\le d}\sup_{t\le T}\left|\delta\sum_{k<t}\xi_{k,i}^{(\theta)}\right| > C_M\sqrt{\frac{\log(40d/\varepsilon)}{d}}\right) \le\varepsilon/5. \tag*{(121)} P ( i ≤ d max t ≤ T sup δ k < t ∑ ξ k , i ( θ ) > C M d log ( 40 d / ε ) ) ≤ ε /5. (121)
Proof. Let
L : = log 40 d ε . L := \log\frac{40d}{\varepsilon}. L := log ε 40 d .
The case T = 0 T = 0 T = 0 is trivial, so assume T ≥ 1 T \ge1 T ≥ 1 . Let σ 2 \sigma^2 σ 2 , K 8 K_8 K 8 be the constants from Corollary D.10. Define the truncation level
B : = ( 40 d T K 8 ε ) 1 / 8 . B := \left(\frac{40dTK_8}{\varepsilon}\right)^{1/8}. B := ( ε 40 d T K 8 ) 1/8 .
For k < T k < T k < T and i ≤ d i \le d i ≤ d , write
ξ k , i : = ξ k , i ( θ ) . \xi_{k,i} := \xi_{k,i}^{(\theta)}. ξ k , i := ξ k , i ( θ ) .
Define the centered truncated martingale difference
Y k , i : = ξ k , i 1 { ∣ ξ k , i ∣ ≤ B } − E [ ξ k , i 1 { ∣ ξ k , i ∣ ≤ B } ∣ F k ] . Y_{k,i} := \xi_{k,i}\mathbf{1}_{\{|\xi_{k,i}|\le B\}}-\mathbb{E}\left[\xi_{k,i}\mathbf{1}_{\{|\xi_{k,i}|\le B\}}\mid\mathcal{F}_k\right]. Y k , i := ξ k , i 1 { ∣ ξ k , i ∣ ≤ B } − E [ ξ k , i 1 { ∣ ξ k , i ∣ ≤ B } ∣ F k ] .
Then
E [ Y k , i ∣ F k ] = 0 , ∣ Y k , i ∣ ≤ 2 B , \mathbb{E}[Y_{k,i}\mid\mathcal{F}_k]=0,\qquad|Y_{k,i}|\le2B, E [ Y k , i ∣ F k ] = 0 , ∣ Y k , i ∣ ≤ 2 B ,
and
E [ Y k , i 2 ∣ F k ] ≤ E [ ξ k , i 2 ∣ F k ] ≤ σ 2 . \mathbb{E}[Y_{k,i}^2\mid\mathcal{F}_k]\le\mathbb{E}[\xi_{k,i}^2\mid\mathcal{F}_k]\le\sigma^2. E [ Y k , i 2 ∣ F k ] ≤ E [ ξ k , i 2 ∣ F k ] ≤ σ 2 .
Hence, for each fixed i i i , the process
M t ( i ) : = ∑ k < t Y k , i M_t^{(i)} := \sum_{k<t}Y_{k,i} M t ( i ) := k < t ∑ Y k , i
is a martingale with increments bounded by 2 B 2B 2 B and predictable quadratic variation at most σ 2 t ≤ σ 2 T \sigma^2t\le\sigma^2T σ 2 t ≤ σ 2 T .
By the maximal form of Freedman’s inequality, for every x > 0 x>0 x > 0 ,
P ( sup t ≤ T M t ( i ) ≥ x ) ≤ exp ( − x 2 2 ( σ 2 T + 2 B x / 3 ) ) . (122) \mathbb{P}\left(\sup_{t\le T}M_t^{(i)}\ge x\right)\le\exp\left(-\frac{x^2}{2(\sigma^2T+2Bx/3)}\right). \tag*{(122)} P ( t ≤ T sup M t ( i ) ≥ x ) ≤ exp ( − 2 ( σ 2 T + 2 B x /3 ) x 2 ) . (122)
and the same estimate holds for − M t ( i ) -M_t^{(i)} − M t ( i ) . Set
x T : = 2 σ 2 T L + 4 3 B L . x_T := \sqrt{2\sigma^2TL}+\frac{4}{3}BL. x T := 2 σ 2 T L + 3 4 B L .
Then
x T 2 2 ( σ 2 T + 2 B x T / 3 ) ≥ L . \frac{x_T^2}{2(\sigma^2T+2Bx_T/3)}\ge L. 2 ( σ 2 T + 2 B x T /3 ) x T 2 ≥ L .
Indeed, writing a = σ 2 T a=\sigma^2T a = σ 2 T , u = 2 a L u=\sqrt{2aL} u = 2 a L , and v = 4 3 B L v=\frac{4}{3}BL v = 3 4 B L , we have x T = u + v x_T=u+v x T = u + v , while
2 L ( a + 2 B x T 3 ) = 2 a L + v ( u + v ) . 2L\left(a+\frac{2Bx_T}{3}\right)=2aL+v(u+v). 2 L ( a + 3 2 B x T ) = 2 a L + v ( u + v ) .
Since
x T 2 = ( u + v ) 2 = 2 a L + 2 u v + v 2 , x_T^2=(u+v)^2=2aL+2uv+v^2, x T 2 = ( u + v ) 2 = 2 a L + 2 uv + v 2 ,
we get
x T 2 − 2 L ( a + 2 B x T 3 ) = u v ≥ 0. x_T^2-2L\left(a+\frac{2Bx_T}{3}\right)=uv\ge0. x T 2 − 2 L ( a + 3 2 B x T ) = uv ≥ 0.
Therefore
P ( sup t ≤ T ∣ M t ( i ) ∣ ≥ x T ) ≤ 2 e − L . \mathbb{P}\left(\sup_{t\le T}|M_t^{(i)}|\ge x_T\right)\le2e^{-L}. P ( t ≤ T sup ∣ M t ( i ) ∣ ≥ x T ) ≤ 2 e − L .
Taking a union bound over i ≤ d i\le d i ≤ d , and using L = log ( 40 d / ε ) L=\log(40d/\varepsilon) L = log ( 40 d / ε ) , gives
P ( max i ≤ d sup t ≤ T ∣ M t ( i ) ∣ ≥ x T ) ≤ 2 d e − L = ε 20 . \mathbb{P}\left(\max_{i\le d}\sup_{t\le T}|M_t^{(i)}|\ge x_T\right)\le2de^{-L}=\frac{\varepsilon}{20}. P ( i ≤ d max t ≤ T sup ∣ M t ( i ) ∣ ≥ x T ) ≤ 2 d e − L = 20 ε .
It remains to compare the original martingale to the truncated one. Let
J : = { max k < T max i ≤ d ∣ ξ k , i ∣ ≤ B } . \mathcal{J}:=\left\{\max_{k<T}\max_{i\le d}|\xi_{k,i}|\le B\right\}. J := { k < T max i ≤ d max ∣ ξ k , i ∣ ≤ B } .
By Corollary D.10 and Markov’s inequality,
P ( J c ) ≤ ∑ k < T ∑ i ≤ d P ( ∣ ξ k , i ∣ > B ) ≤ d T K 8 B 8 = ε 40 . \mathbb{P}(\mathcal{J}^{c})\le\sum_{k<T}\sum_{i\le d}\mathbb{P}(|\xi_{k,i}|>B)\le dT\frac{K_8}{B^8}=\frac{\varepsilon}{40}. P ( J c ) ≤ k < T ∑ i ≤ d ∑ P ( ∣ ξ k , i ∣ > B ) ≤ d T B 8 K 8 = 40 ε .
Now define
b k , i : = E [ ξ k , i 1 { ∣ ξ k , i ∣ > B } ∣ F k ] . b_{k,i}:=\mathbb{E}\left[\xi_{k,i}1_{\{|\xi_{k,i}|>B\}}\mid\mathcal{F}_k\right]. b k , i := E [ ξ k , i 1 { ∣ ξ k , i ∣ > B } ∣ F k ] .
Since E [ ξ k , i ∣ F k ] = 0 \mathbb{E}[\xi_{k,i}\mid\mathcal{F}_k]=0 E [ ξ k , i ∣ F k ] = 0 ,
E [ ξ k , i 1 { ∣ ξ k , i ∣ ≤ B } ∣ F k ] = − b k , i . \mathbb{E}\left[\xi_{k,i}1_{\{|\xi_{k,i}|\le B\}}\mid\mathcal{F}_k\right]=-b_{k,i}. E [ ξ k , i 1 { ∣ ξ k , i ∣ ≤ B } ∣ F k ] = − b k , i .
Hence
Y k , i = ξ k , i 1 { ∣ ξ k , i ∣ ≤ B } + b k , i . Y_{k,i}=\xi_{k,i}1_{\{|\xi_{k,i}|\le B\}}+b_{k,i}. Y k , i = ξ k , i 1 { ∣ ξ k , i ∣ ≤ B } + b k , i .
On J \mathcal{J} J , the indicator 1 { ∣ ξ k , i ∣ ≤ B } 1_{\{|\xi_{k,i}|\le B\}} 1 { ∣ ξ k , i ∣ ≤ B } is identically one for all k < T k<T k < T and i ≤ d i\le d i ≤ d . Therefore, on J \mathcal{J} J ,
ξ k , i = Y k , i − b k , i . \xi_{k,i}=Y_{k,i}-b_{k,i}. ξ k , i = Y k , i − b k , i .
Moreover, by the conditional eighth-moment bound,
∣ b k , i ∣ ≤ E [ ∣ ξ k , i ∣ 1 { ∣ ξ k , i ∣ > B } ∣ F k ] ≤ E [ ∣ ξ k , i ∣ 8 ∣ F k ] B 7 ≤ K 8 B 7 . |b_{k,i}|\le\mathbb{E}\left[|\xi_{k,i}|1_{\{|\xi_{k,i}|>B\}}\mid\mathcal{F}_k\right]\le\frac{\mathbb{E}[|\xi_{k,i}|^8\mid\mathcal{F}_k]}{B^7}\le\frac{K_8}{B^7}. ∣ b k , i ∣ ≤ E [ ∣ ξ k , i ∣ 1 { ∣ ξ k , i ∣ > B } ∣ F k ] ≤ B 7 E [ ∣ ξ k , i ∣ 8 ∣ F k ] ≤ B 7 K 8 .
Thus, on J \mathcal{J} J ,
sup t ≤ T ∣ ∑ k < t ξ k , i ∣ ≤ sup t ≤ T ∣ M t ( i ) ∣ + T K 8 B 7 . \sup_{t\le T}\left|\sum_{k<t}\xi_{k,i}\right|\le\sup_{t\le T}|M_t^{(i)}|+\frac{TK_8}{B^7}. t ≤ T sup k < t ∑ ξ k , i ≤ t ≤ T sup ∣ M t ( i ) ∣ + B 7 T K 8 .
Combining the previous estimates, with probability at least 1 − ε / 20 − ε / 40 1-\varepsilon/20-\varepsilon/40 1 − ε /20 − ε /40 , we have
max i ≤ d sup t ≤ T ∣ ∑ k < t ξ k , i ∣ ≤ x T + T K 8 B 7 . \max_{i\le d}\sup_{t\le T}\left|\sum_{k<t}\xi_{k,i}\right|\le x_T+\frac{TK_8}{B^7}. i ≤ d max t ≤ T sup k < t ∑ ξ k , i ≤ x T + B 7 T K 8 .
Multiplying by δ \delta δ , it remains to show that
δ x T + δ T K 8 B 7 ≤ C M L d . \delta x_T+\delta\frac{TK_8}{B^7}\le C_M\sqrt{\frac{L}{d}}. δ x T + δ B 7 T K 8 ≤ C M d L .
First,
δ 2 σ 2 T L = 2 σ 2 L δ ( δ T ) ≤ 2 σ 2 C T c 0 ε L d ≤ C L d , \delta\sqrt{2\sigma^2TL}=\sqrt{2\sigma^2L\delta(\delta T)}\le\sqrt{2\sigma^2C_Tc_0\varepsilon}\sqrt{\frac{L}{d}}\le C\sqrt{\frac{L}{d}}, δ 2 σ 2 T L = 2 σ 2 L δ ( δ T ) ≤ 2 σ 2 C T c 0 ε d L ≤ C d L ,
because T δ ≤ C T log d T\delta\le C_T\log d T δ ≤ C T log d , δ ≤ c 0 ε / ( d log d ) \delta\le c_0\varepsilon/(d\log d) δ ≤ c 0 ε / ( d log d ) , and ε ≤ 1 \varepsilon\le1 ε ≤ 1 .
Next, using T ≤ C T ( log d ) / δ T\le C_T(\log d)/\delta T ≤ C T ( log d ) / δ ,
δ B L = δ L ( 40 d T K 8 ε ) 1 / 8 ≤ C L δ 7 / 8 ( d log d ε ) 1 / 8 . \delta BL=\delta L\left(\frac{40dTK_8}{\varepsilon}\right)^{1/8}\le CL\delta^{7/8}\left(\frac{d\log d}{\varepsilon}\right)^{1/8}. δ B L = δ L ( ε 40 d T K 8 ) 1/8 ≤ C L δ 7/8 ( ε d log d ) 1/8 .
Using again δ ≤ c 0 ε / ( d log d ) \delta\le c_0\varepsilon/(d\log d) δ ≤ c 0 ε / ( d log d ) , this gives
δ B L ≤ C c 0 7 / 8 ε 3 / 4 L d − 3 / 4 ( log d ) − 3 / 4 . \delta BL\le Cc_0^{7/8}\varepsilon^{3/4}Ld^{-3/4}(\log d)^{-3/4}. δ B L ≤ C c 0 7/8 ε 3/4 L d − 3/4 ( log d ) − 3/4 .
Since L = log ( 40 d / ε ) L=\log(40d/\varepsilon) L = log ( 40 d / ε ) and 0 < ε ≤ 1 0<\varepsilon\le1 0 < ε ≤ 1 ,
ε 3 / 4 L 1 / 2 ≤ C ( log d ) 1 / 2 \varepsilon^{3/4}L^{1/2}\le C(\log d)^{1/2} ε 3/4 L 1/2 ≤ C ( log d ) 1/2
for d ≥ 3 d\ge3 d ≥ 3 . Hence
δ B L ≤ C c 0 7 / 8 L 1 / 2 d − 3 / 4 ( log d ) − 1 / 4 ≤ C L d . \delta BL\le Cc_0^{7/8}L^{1/2}d^{-3/4}(\log d)^{-1/4}\le C\sqrt{\frac{L}{d}}. δ B L ≤ C c 0 7/8 L 1/2 d − 3/4 ( log d ) − 1/4 ≤ C d L .
Finally,
δ T K 8 B 7 = δ T K 8 ( ε 40 d T K 8 ) 7 / 8 ≤ C δ T 1 / 8 ε 7 / 8 d − 7 / 8 . \delta\frac{TK_8}{B^7}=\delta TK_8\left(\frac{\varepsilon}{40dTK_8}\right)^{7/8}\le C\delta T^{1/8}\varepsilon^{7/8}d^{-7/8}. δ B 7 T K 8 = δ T K 8 ( 40 d T K 8 ε ) 7/8 ≤ C δ T 1/8 ε 7/8 d − 7/8 .
Using T ≤ C T ( log d ) / δ T\le C_T(\log d)/\delta T ≤ C T ( log d ) / δ and δ ≤ c 0 ε / ( d log d ) \delta\le c_0\varepsilon/(d\log d) δ ≤ c 0 ε / ( d log d ) , we get
δ T K 8 B 7 ≤ C c 0 7 / 8 ε 7 / 4 d − 7 / 4 ( log d ) − 3 / 4 ≤ C L d . \delta\frac{TK_8}{B^7}\le Cc_0^{7/8}\varepsilon^{7/4}d^{-7/4}(\log d)^{-3/4}\le C\sqrt{\frac{L}{d}}. δ B 7 T K 8 ≤ C c 0 7/8 ε 7/4 d − 7/4 ( log d ) − 3/4 ≤ C d L .
Therefore, after increasing C M C_M C M to absorb constants depending only on σ 2 \sigma^2 σ 2 , K 8 K_8 K 8 , C T C_T C T , c 0 c_0 c 0 , we have
δ x T + δ T K 8 B 7 ≤ C M L d = C M log ( 40 d / ε ) d . \delta x_T+\delta\frac{TK_8}{B^7}\le C_M\sqrt{\frac{L}{d}}=C_M\sqrt{\frac{\log(40d/\varepsilon)}{d}}. δ x T + δ B 7 T K 8 ≤ C M d L = C M d log ( 40 d / ε ) .
Since
ε 20 + ε 40 < ε 5 , \frac{\varepsilon}{20}+\frac{\varepsilon}{40}<\frac{\varepsilon}{5}, 20 ε + 40 ε < 5 ε ,
the desired bound follows.
We plug this proved Freedman bound into (116) to obtain
Θ t ≤ 4 3 ( ∥ θ 0 ∥ ∞ + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ ⏟ 1 + Λ η + δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 ⏟ 4 ) . (123) \Theta_t\le\frac{4}{3}\left(\lVert\theta_0\rVert_\infty+\delta\underbrace{\sum_{k<t}\lVert\bar g_k^{(\theta)}\rVert_\infty}_{1}+\Lambda\eta+\frac{\delta^3}{2}\underbrace{\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3}_{4}\right). \tag*{(123)} Θ t ≤ 3 4 ∥ θ 0 ∥ ∞ + δ 1 k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + Λ η + 2 δ 3 4 k < t ∑ ∥ g k ( θ ) ∥ 2 3 . (123)
Since η t = ∥ θ t ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ Θ t + ∥ θ ∗ ∥ ∞ \eta_t=\lVert\theta_t\rVert_\infty+\lVert\theta^*\rVert_\infty\le\Theta_t+\lVert\theta^*\rVert_\infty η t = ∥ θ t ∥ ∞ + ∥ θ ∗ ∥ ∞ ≤ Θ t + ∥ θ ∗ ∥ ∞ , we re-write it as:
η t ≤ 4 3 ( η 0 + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ ⏟ 1 + Λ η + δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 ⏟ 4 ) . (124) \eta_t\le\frac{4}{3}\left(\eta_0+\delta\underbrace{\sum_{k<t}\lVert\bar g_k^{(\theta)}\rVert_\infty}_{1}+\Lambda\eta+\frac{\delta^3}{2}\underbrace{\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3}_{4}\right). \tag*{(124)} η t ≤ 3 4 η 0 + δ 1 k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + Λ η + 2 δ 3 4 k < t ∑ ∥ g k ( θ ) ∥ 2 3 . (124)
Next we bound 4: notice that since δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 \frac{\delta^3}{2}\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3 2 δ 3 ∑ k < t ∥ g k ( θ ) ∥ 2 3 is non decreasing in t t t , we have
sup t ≤ T δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 = δ 3 2 ∑ k < T ∥ g k ( θ ) ∥ 2 3 . \sup_{t\le T}\frac{\delta^3}{2}\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3=\frac{\delta^3}{2}\sum_{k<T}\lVert g_k^{(\theta)}\rVert_2^3. t ≤ T sup 2 δ 3 k < t ∑ ∥ g k ( θ ) ∥ 2 3 = 2 δ 3 k < T ∑ ∥ g k ( θ ) ∥ 2 3 .
On the other hand,
E [ ∥ g k ( θ ) ∥ 2 3 ] ≤ E [ ∥ g k ( θ ) ∥ 2 4 ] 3 / 4 by Holder inequality (125) \mathbb{E}\left[\lVert g_k^{(\theta)}\rVert_2^3\right] \le\mathbb{E}\left[\lVert g_k^{(\theta)}\rVert_2^4\right]^{3/4} \text{ by Holder inequality} \tag*{(125)} E [ ∥ g k ( θ ) ∥ 2 3 ] ≤ E [ ∥ g k ( θ ) ∥ 2 4 ] 3/4 by Holder inequality (125)
≤ ( K d 2 ) 3 / 4 by the technical bounds in x r e f ‘ L e m m a E .7 ‘ (126) \le(K d^2)^{3/4} \text{ by the technical bounds in } {xref}`Lemma E.7` \tag*{(126)} ≤ ( K d 2 ) 3/4 by the technical bounds in x r e f ‘ L e mma E .7‘ (126)
= K 3 / 4 d 3 / 2 . (127) =K^{3/4}d^{3/2}. \tag*{(127)} = K 3/4 d 3/2 . (127)
Therefore, for a given λ c u b \lambda_{\mathrm{cub}} λ cub ,
P ( sup t ≤ T δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 ≥ λ c u b ) = P ( δ 3 2 ∑ k < T ∥ g k ( θ ) ∥ 2 3 ≥ λ c u b ) by monotonicity (128) \mathbb{P}\left(\sup_{t\le T}\frac{\delta^3}{2}\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3\ge\lambda_{\mathrm{cub}}\right)=\mathbb{P}\left(\frac{\delta^3}{2}\sum_{k<T}\lVert g_k^{(\theta)}\rVert_2^3\ge\lambda_{\mathrm{cub}}\right)\text{ by monotonicity} \tag*{(128)} P ( t ≤ T sup 2 δ 3 k < t ∑ ∥ g k ( θ ) ∥ 2 3 ≥ λ cub ) = P ( 2 δ 3 k < T ∑ ∥ g k ( θ ) ∥ 2 3 ≥ λ cub ) by monotonicity (128)
≤ E [ δ 3 2 ∑ k < T ∥ g k ( θ ) ∥ 2 3 ] λ c u b , by Markov Inequality (129) \le\frac{\mathbb{E}\left[\frac{\delta^3}{2}\sum_{k<T}\lVert g_k^{(\theta)}\rVert_2^3\right]}{\lambda_{\mathrm{cub}}},\text{ by Markov Inequality} \tag*{(129)} ≤ λ cub E [ 2 δ 3 ∑ k < T ∥ g k ( θ ) ∥ 2 3 ] , by Markov Inequality (129)
≤ K 3 / 4 d 3 / 2 T δ 3 2 λ c u b , following x r e f ‘ ( 124 ) ‘ (130) \le\frac{K^{3/4}d^{3/2}T\frac{\delta^3}{2}}{\lambda_{\mathrm{cub}}},\text{ following } {xref}`(124)` \tag*{(130)} ≤ λ cub K 3/4 d 3/2 T 2 δ 3 , following x r e f ‘ ( 124 ) ‘ (130)
Taking λ c u b = C c u b / d \lambda_{\mathrm{cub}}=C_{\mathrm{cub}}/\sqrt{d} λ cub = C cub / d , and plugging in T ≤ c 2 ( log d ) / δ T\le c_2(\log d)/\delta T ≤ c 2 ( log d ) / δ and δ ≤ c 0 ε / ( d log d ) \delta\le c_0\varepsilon/(d\log d) δ ≤ c 0 ε / ( d log d ) , we have
P ( sup t ≤ T δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 ≥ C c u b d ) ≤ c 2 , c 0 2 , K 3 / 4 2 C c u b ⋅ ε 2 log d . (131) \mathbb{P}\left(\sup_{t\le T}\frac{\delta^3}{2}\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3\ge\frac{C_{\mathrm{cub}}}{\sqrt{d}}\right)\le\frac{c_2,c_0^2,K^{3/4}}{2C_{\mathrm{cub}}}\cdot\frac{\varepsilon^2}{\log d}. \tag*{(131)} P ( t ≤ T sup 2 δ 3 k < t ∑ ∥ g k ( θ ) ∥ 2 3 ≥ d C cub ) ≤ 2 C cub c 2 , c 0 2 , K 3/4 ⋅ log d ε 2 . (131)
Choose C c u b C_{\mathrm{cub}} C cub large enough (absorbing constants) and take d d d sufficiently large so that ε 2 log d ≤ ε / 10 \frac{\varepsilon^2}{\log d}\le\varepsilon/10 l o g d ε 2 ≤ ε /10 , we obtain
P ( sup t ≤ T δ 3 2 ∑ k < t ∥ g k ( θ ) ∥ 2 3 ≥ C c u b d ) ≤ ε 10 (132) \mathbb{P}\left(\sup_{t\le T}\frac{\delta^3}{2}\sum_{k<t}\lVert g_k^{(\theta)}\rVert_2^3\ge\frac{C_{\mathrm{cub}}}{\sqrt{d}}\right)\le\frac{\varepsilon}{10} \tag*{(132)} P ( t ≤ T sup 2 δ 3 k < t ∑ ∥ g k ( θ ) ∥ 2 3 ≥ d C cub ) ≤ 10 ε (132)
Plugging it in the η \eta η bound in (118) it becomes
η t ≤ C η ( η 0 + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ + Λ η + C c u b d ) , (133) \eta_t\le C_\eta\left(\eta_0+\delta\sum_{k<t}\lVert\bar g_k^{(\theta)}\rVert_\infty+\Lambda_\eta+\frac{C_{\mathrm{cub}}}{\sqrt{d}}\right), \tag*{(133)} η t ≤ C η ( η 0 + δ k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + Λ η + d C cub ) , (133)
with high probability. The bound of 1 is contingent to dynamics’s phase that we will derive it later. We later bound them in Step 1 of the Phase I proof (Lemma D.13) and Step 1(a) of the Phase II proof (Lemma D.16).
Good event for η \eta η iteration Define
E T ( η ) : = E T ( η , M ) ∩ E T ( η , 2 ) ∩ E T ( η , 3 ) , (134) E_T^{(\eta)}:=E_T^{(\eta,M)}\cap E_T^{(\eta,2)}\cap E_T^{(\eta,3)}, \tag*{(134)} E T ( η ) := E T ( η , M ) ∩ E T ( η , 2 ) ∩ E T ( η , 3 ) , (134)
where
E T ( η , M ) : = { max i ≤ d sup t ≤ T ∣ δ ∑ k < t ξ k , i ( θ ) ∣ ≤ Λ η } , (135) E_T^{(\eta,M)}:=\left\{\max_{i\le d}\sup_{t\le T}\left|\delta\sum_{k<t}\xi_{k,i}^{(\theta)}\right|\le\Lambda_\eta\right\}, \tag*{(135)} E T ( η , M ) := { i ≤ d max t ≤ T sup δ k < t ∑ ξ k , i ( θ ) ≤ Λ η } , (135)
E T ( η , 2 ) : = { δ 2 2 ∑ k < T ∥ g k ( θ ) ∥ 2 2 ≤ 1 4 } , (136) E_T^{(\eta,2)}:=\left\{\frac{\delta^2}{2}\sum_{k<T}\lVert g_k^{(\theta)}\rVert_2^2\le\frac{1}{4}\right\}, \tag*{(136)} E T ( η , 2 ) := { 2 δ 2 k < T ∑ ∥ g k ( θ ) ∥ 2 2 ≤ 4 1 } , (136)
E T ( η , 3 ) : = { sup t ≤ T δ 3 2 ∑ k < T ∥ g k ( θ ) ∥ 2 3 ≤ C c u b d } . (137) E_T^{(\eta,3)}:=\left\{\sup_{t\le T}\frac{\delta^3}{2}\sum_{k<T}\lVert g_k^{(\theta)}\rVert_2^3\le\frac{C_{\mathrm{cub}}}{\sqrt{d}}\right\}. \tag*{(137)} E T ( η , 3 ) := { t ≤ T sup 2 δ 3 k < T ∑ ∥ g k ( θ ) ∥ 2 3 ≤ d C cub } . (137)
Therefore, with Bound (119) from Lemma E.7 to ensure (132), Bound (115) to ensure (133), and Bound (129) to ensure (134), we obtain a union bound,
P ( ( E T ( η ) ) c ) ≤ P ( ( E T ( η , M ) ) c ) + P ( ( E T ( η , 2 ) ) c ) + P ( ( E T ( η , 3 ) ) c ) ≤ K ε , (138) \mathbb{P}\left((E_T^{(\eta)})^c\right)\le\mathbb{P}\left((E_T^{(\eta,M)})^c\right)+\mathbb{P}\left((E_T^{(\eta,2)})^c\right)+\mathbb{P}\left((E_T^{(\eta,3)})^c\right)\le K\varepsilon, \tag*{(138)} P ( ( E T ( η ) ) c ) ≤ P ( ( E T ( η , M ) ) c ) + P ( ( E T ( η , 2 ) ) c ) + P ( ( E T ( η , 3 ) ) c ) ≤ K ε , (138)
after increasing K K K .
Martingale and Discretization Error Control for Ω t \Omega_t Ω t and ω t \omega_t ω t
As a discrete analogy of Lemma C.7, we next provide the control of third moment quantities Ω t : = ∥ θ t ∥ 3 3 = ∑ i = 1 d ∣ θ t , i ∣ 3 \Omega_t := \|\theta_t\|_3^3 = \sum_{i=1}^d |\theta_{t,i}|^3 Ω t := ∥ θ t ∥ 3 3 = ∑ i = 1 d ∣ θ t , i ∣ 3 , and ω t : = ∑ i = 1 d ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 \omega_t := \sum_{i=1}^d (|\theta_{t,i}|+|\theta_i^*|)^3 ω t := ∑ i = 1 d ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 , upon their martingale and discretization noises.
Lemma D.12 (Third-moment control for SGD through Phase I). Let Ω t \Omega_t Ω t and ω t \omega_t ω t follow the definition made in Lemma C.7. Fix T ∈ N T \in\mathbb{N} T ∈ N with T δ ≤ C T log d T\delta\le C_T \log d T δ ≤ C T log d . Under Assumption D.1 , Lemma E.7 , and Corollary D.10, there is an event E T ( Ω ) E_T^{(\Omega)} E T ( Ω ) satisfying
P ( ( E T ( Ω ) ) c ) ≤ K Ω ε \mathbb{P}\left((E_T^{(\Omega)})^c\right) \le K_\Omega\varepsilon P ( ( E T ( Ω ) ) c ) ≤ K Ω ε such that the following holds. Work on E 0 ∩ E T ( Ω ) E_0 \cap E_T^{(\Omega)} E 0 ∩ E T ( Ω ) . Let
τ A : = inf { t : Z t 2 ≥ A η t } , \tau_A := \inf\{t: Z_t^2 \ge A\eta_t\}, τ A := inf { t : Z t 2 ≥ A η t } , τ B : = inf { t : ρ t ≥ 1 / 2 or χ t ≥ 1 / 2 } , \tau_B := \inf\{t: \rho_t \ge1/2\text{ or }\chi_t \ge1/2\}, τ B := inf { t : ρ t ≥ 1/2 or χ t ≥ 1/2 } , τ + : = inf { t : ρ t < 0 or χ t < 0 } , \tau_+ := \inf\{t: \rho_t < 0\text{ or }\chi_t < 0\}, τ + := inf { t : ρ t < 0 or χ t < 0 } , and set
σ : = τ A ∧ τ B ∧ τ + ∧ T . \sigma:= \tau_A \wedge\tau_B \wedge\tau_+ \wedge T. σ := τ A ∧ τ B ∧ τ + ∧ T . Suppose that, for some constant C η > 0 C_\eta> 0 C η > 0 ,
η t ≤ C η log d + 1 + t δ d ( 0 ≤ t ≤ σ ) . (139) \eta_t \le C_\eta\frac{\sqrt{\log d}+1+t\delta}{\sqrt{d}} \qquad(0\le t\le\sigma). \tag*{(139)} η t ≤ C η d log d + 1 + t δ ( 0 ≤ t ≤ σ ) . (139) Then, for all sufficiently large d d d ,
Ω t ≤ C Ω 1 + t δ d , ω t ≤ C ω 1 + t δ d , 0 ≤ t ≤ σ . (140) \Omega_t \le C_\Omega\frac{1+t\delta}{\sqrt{d}}, \qquad\omega_t \le C_\omega\frac{1+t\delta}{\sqrt{d}}, \qquad0\le t\le\sigma. \tag*{(140)} Ω t ≤ C Ω d 1 + t δ , ω t ≤ C ω d 1 + t δ , 0 ≤ t ≤ σ . (140) The constants K Ω K_\Omega K Ω , C Ω C_\Omega C Ω , C ω C_\omega C ω depend only on model constants, A A A , C η C_\eta C η , C T C_T C T , and the step-size regime.
Proof. Write g t : = g t ( θ ) g_t := g_t^{(\theta)} g t := g t ( θ ) , g ˉ t : = E [ g t ∣ F t ] \bar g_t := \mathbb{E}[g_t\mid\mathcal{F}_t] g ˉ t := E [ g t ∣ F t ] , and ξ t : = g t − g ˉ t \xi_t := g_t-\bar g_t ξ t := g t − g ˉ t . Since g t ⊥ θ t g_t \perp\theta_t g t ⊥ θ t , the normalized spherical update is
θ t + 1 = a t ( θ t − δ g t ) , a t = ( 1 + δ 2 ∥ g t ∥ 2 2 ) − 1 / 2 ≤ 1. \theta_{t+1}=a_t(\theta_t-\delta g_t),\qquad a_t=(1+\delta^2\|g_t\|_2^2)^{-1/2}\le1. θ t + 1 = a t ( θ t − δ g t ) , a t = ( 1 + δ 2 ∥ g t ∥ 2 2 ) − 1/2 ≤ 1.
Hence
Ω t + 1 = ∥ θ t + 1 ∥ 3 3 ≤ ∥ θ t − δ g t ∥ 3 3 . \Omega_{t+1}=\|\theta_{t+1}\|_3^3\le\|\theta_t-\delta g_t\|_3^3. Ω t + 1 = ∥ θ t + 1 ∥ 3 3 ≤ ∥ θ t − δ g t ∥ 3 3 .
We use the elementary inequality
∣ x − y ∣ 3 ≤ ∣ x ∣ 3 − 3 ∣ x ∣ x y + C ( ∣ x ∣ y 2 + ∣ y ∣ 3 ) , x , y ∈ R . |x-y|^3\le|x|^3-3|x|xy+C\left(|x|y^2+|y|^3\right),\qquad x,y\in\mathbb{R}. ∣ x − y ∣ 3 ≤ ∣ x ∣ 3 − 3∣ x ∣ x y + C ( ∣ x ∣ y 2 + ∣ y ∣ 3 ) , x , y ∈ R .
Applying it coordinatewise with x = θ t , i x=\theta_{t,i} x = θ t , i and y = δ g t , i y=\delta g_{t,i} y = δ g t , i , and letting v t : = ( ∣ θ t , i ∣ θ t , i ) i = 1 d v_t:=\left(|\theta_{t,i}|\theta_{t,i}\right)_{i=1}^d v t := ( ∣ θ t , i ∣ θ t , i ) i = 1 d , gives
Ω t + 1 ≤ Ω t − 3 δ ⟨ v t , g ˉ t ⟩ − 3 δ ⟨ v t , ξ t ⟩ + C δ 2 ∑ i = 1 d ∣ θ t , i ∣ g t , i 2 + C δ 3 ∥ g t ∥ 3 3 . (141) \Omega_{t+1}\le\Omega_t-3\delta\langle v_t,\bar g_t\rangle-3\delta\langle v_t,\xi_t\rangle+C\delta^2\sum_{i=1}^d|\theta_{t,i}|g_{t,i}^2+C\delta^3\|g_t\|_3^3. \tag*{(141)} Ω t + 1 ≤ Ω t − 3 δ ⟨ v t , g ˉ t ⟩ − 3 δ ⟨ v t , ξ t ⟩ + C δ 2 i = 1 ∑ d ∣ θ t , i ∣ g t , i 2 + C δ 3 ∥ g t ∥ 3 3 . (141)
Since ∥ v t ∥ 1 = ∑ i ∣ θ t , i ∣ 2 = 1 \|v_t\|_1=\sum_i|\theta_{t,i}|^2=1 ∥ v t ∥ 1 = ∑ i ∣ θ t , i ∣ 2 = 1 ,
− 3 δ ⟨ v t , g ˉ t ⟩ ≤ 3 δ ∥ g ˉ t ∥ ∞ . -3\delta\langle v_t,\bar g_t\rangle\le3\delta\|\bar g_t\|_\infty. − 3 δ ⟨ v t , g ˉ t ⟩ ≤ 3 δ ∥ g ˉ t ∥ ∞ .
Summing (141) gives
Ω t ≤ Ω 0 + 3 δ ∑ k < t ∥ g ˉ k ∥ ∞ + M t ( Ω ) + C Q t ( Ω ) + C C t ( Ω ) , (142) \Omega_t\le\Omega_0+3\delta\sum_{k<t}\|\bar g_k\|_\infty+\mathcal{M}_t^{(\Omega)}+C Q_t^{(\Omega)}+C C_t^{(\Omega)}, \tag*{(142)} Ω t ≤ Ω 0 + 3 δ k < t ∑ ∥ g ˉ k ∥ ∞ + M t ( Ω ) + C Q t ( Ω ) + C C t ( Ω ) , (142)
where
M t ( Ω ) : = − 3 δ ∑ k < t ⟨ v k , ξ k ⟩ , \mathcal{M}_t^{(\Omega)}:=-3\delta\sum_{k<t}\langle v_k,\xi_k\rangle, M t ( Ω ) := − 3 δ k < t ∑ ⟨ v k , ξ k ⟩ ,
Q t ( Ω ) : = δ 2 ∑ k < t ∑ i = 1 d ∣ θ k , i ∣ g k , i 2 , C t ( Ω ) : = δ 3 ∑ k < t ∥ g k ∥ 3 3 . Q_t^{(\Omega)}:=\delta^2\sum_{k<t}\sum_{i=1}^d|\theta_{k,i}|g_{k,i}^2,\qquad C_t^{(\Omega)}:=\delta^3\sum_{k<t}\|g_k\|_3^3. Q t ( Ω ) := δ 2 k < t ∑ i = 1 ∑ d ∣ θ k , i ∣ g k , i 2 , C t ( Ω ) := δ 3 k < t ∑ ∥ g k ∥ 3 3 .
We first define the good event E T ( Ω ) E_T^{(\Omega)} E T ( Ω ) . Let
λ Ω , M : = C Ω , M d , λ Ω , Q : = C Ω , Q d , λ Ω , C : = C Ω , C d , \lambda_{\Omega,M}:=\frac{C_{\Omega,M}}{\sqrt{d}},\qquad\lambda_{\Omega,Q}:=\frac{C_{\Omega,Q}}{\sqrt{d}},\qquad\lambda_{\Omega,C}:=\frac{C_{\Omega,C}}{\sqrt{d}}, λ Ω , M := d C Ω , M , λ Ω , Q := d C Ω , Q , λ Ω , C := d C Ω , C ,
where the constants will be chosen large. Set
E T ( Ω ) : = { sup t ≤ T ∣ M t ( Ω ) ∣ ≤ λ Ω , M } ∩ { sup t ≤ T Q t ( Ω ) ≤ λ Ω , Q } ∩ { sup t ≤ T C t ( Ω ) ≤ λ Ω , C } . (143) E_T^{(\Omega)}:=\left\{\sup_{t\le T}|\mathcal{M}_t^{(\Omega)}|\le\lambda_{\Omega,M}\right\}\cap\left\{\sup_{t\le T}\mathcal{Q}_t^{(\Omega)}\le\lambda_{\Omega,Q}\right\}\cap\left\{\sup_{t\le T}\mathcal{C}_t^{(\Omega)}\le\lambda_{\Omega,C}\right\}. \tag*{(143)} E T ( Ω ) := { t ≤ T sup ∣ M t ( Ω ) ∣ ≤ λ Ω , M } ∩ { t ≤ T sup Q t ( Ω ) ≤ λ Ω , Q } ∩ { t ≤ T sup C t ( Ω ) ≤ λ Ω , C } . (143)
We verify that this event has high probability. The process M t ( Ω ) \mathcal{M}_t^{(\Omega)} M t ( Ω ) is a martingale. Moreover,
∥ v k ∥ 2 2 = ∑ i ∣ θ k , i ∣ 4 ≤ ∑ i ∣ θ k , i ∣ 2 = 1. \|v_k\|_2^2=\sum_i|\theta_{k,i}|^4\le\sum_i|\theta_{k,i}|^2=1. ∥ v k ∥ 2 2 = i ∑ ∣ θ k , i ∣ 4 ≤ i ∑ ∣ θ k , i ∣ 2 = 1.
By Lemma E.7,
E [ ⟨ v k , ξ k ⟩ 2 ∣ F k ] ≤ K . \mathbb{E}\left[\langle v_k,\xi_k\rangle^2\mid\mathcal{F}_k\right]\le K. E [ ⟨ v k , ξ k ⟩ 2 ∣ F k ] ≤ K .
Thus Doob’s L 2 L^2 L 2 inequality gives
P ( sup t ≤ T ∣ M t ( Ω ) ∣ ≥ λ Ω , M ) ≤ K T δ 2 λ Ω , M 2 = K T δ 2 d C Ω , M 2 ≤ K C T c 0 C Ω , M 2 ε . \mathbb{P}\left(\sup_{t\le T}|\mathcal{M}_t^{(\Omega)}|\ge\lambda_{\Omega,M}\right)\le\frac{KT\delta^2}{\lambda_{\Omega,M}^2}=\frac{KT\delta^2d}{C_{\Omega,M}^2}\le\frac{KC_Tc_0}{C_{\Omega,M}^2}\varepsilon. P ( t ≤ T sup ∣ M t ( Ω ) ∣ ≥ λ Ω , M ) ≤ λ Ω , M 2 K T δ 2 = C Ω , M 2 K T δ 2 d ≤ C Ω , M 2 K C T c 0 ε .
Choosing C Ω , M C_{\Omega,M} C Ω , M large makes this contribution O ( ε ) O(\varepsilon) O ( ε ) .
Next, by the coordinate moment bound proved in Corollary D.10 and (120) , we have
E [ g k , i 2 ∣ F k ] ≤ K \mathbb{E}[g_{k,i}^2\mid\mathcal{F}_k]\le K E [ g k , i 2 ∣ F k ] ≤ K
uniformly in i , k i,k i , k . Therefore
E [ ∑ i ∣ θ k , i ∣ ∣ g k , i ∣ ∣ F k ] ≤ K ∑ i ∣ θ k , i ∣ ≤ K d . \mathbb{E}\left[\left.\sum_i|\theta_{k,i}|\,|g_{k,i}|\right|\mathcal{F}_k\right]\le K\sum_i|\theta_{k,i}|\le K\sqrt{d}. E [ i ∑ ∣ θ k , i ∣ ∣ g k , i ∣ F k ] ≤ K i ∑ ∣ θ k , i ∣ ≤ K d .
Hence
E [ Q T ( Ω ) ] ≤ K T δ 2 d . \mathbb{E}[\mathcal{Q}_T^{(\Omega)}]\le KT\delta^2\sqrt{d}. E [ Q T ( Ω ) ] ≤ K T δ 2 d .
By Markov’s inequality,
P ( sup t ≤ T Q t ( Ω ) ≥ λ Ω , Q ) ≤ K T δ 2 d C Ω , Q / d = K T δ 2 d C Ω , Q ≤ K C T c 0 C Ω , Q ε . \mathbb{P}\left(\sup_{t\le T}\mathcal{Q}_t^{(\Omega)}\ge\lambda_{\Omega,Q}\right)\le\frac{KT\delta^2\sqrt{d}}{C_{\Omega,Q}/\sqrt{d}}=\frac{KT\delta^2d}{C_{\Omega,Q}}\le\frac{KC_Tc_0}{C_{\Omega,Q}}\varepsilon. P ( t ≤ T sup Q t ( Ω ) ≥ λ Ω , Q ) ≤ C Ω , Q / d K T δ 2 d = C Ω , Q K T δ 2 d ≤ C Ω , Q K C T c 0 ε .
Choosing C Ω , Q C_{\Omega,Q} C Ω , Q large makes this contribution O ( ε ) O(\varepsilon) O ( ε ) .
Similarly, the same coordinate moment bounds imply
E [ ∣ g k , i ∣ 3 ∣ F k ] ≤ K , E [ ∥ g k ∥ 3 3 ∣ F k ] ≤ K d . \mathbb{E}[|g_{k,i}|^3\mid\mathcal{F}_k]\le K,\qquad\mathbb{E}[\|g_k\|_3^3\mid\mathcal{F}_k]\le Kd. E [ ∣ g k , i ∣ 3 ∣ F k ] ≤ K , E [ ∥ g k ∥ 3 3 ∣ F k ] ≤ K d .
Thus
E [ C T ( Ω ) ] ≤ K T δ 3 d . \mathbb{E}[\mathcal{C}_T^{(\Omega)}]\le KT\delta^3d. E [ C T ( Ω ) ] ≤ K T δ 3 d .
Again by Markov’s inequality,
P ( sup t ≤ T C t ( Ω ) ≥ λ Ω , C ) ≤ K T δ 3 d C Ω , C / d = K T δ 3 d 3 / 2 C Ω , C . \mathbb{P}\left(\sup_{t\le T}\mathcal{C}_t^{(\Omega)}\ge\lambda_{\Omega,C}\right)\le\frac{KT\delta^3d}{C_{\Omega,C}/\sqrt{d}}=\frac{KT\delta^3d^{3/2}}{C_{\Omega,C}}. P ( t ≤ T sup C t ( Ω ) ≥ λ Ω , C ) ≤ C Ω , C / d K T δ 3 d = C Ω , C K T δ 3 d 3/2 .
Since T δ ≤ C T log d T\delta\le C_T\log d T δ ≤ C T log d and δ ≤ c 0 ε / ( d log d ) \delta\le c_0\varepsilon/(d\log d) δ ≤ c 0 ε / ( d log d ) , the last display is o ( ε ) o(\varepsilon) o ( ε ) for d d d large. Therefore, after increasing the constants if necessary,
P ( ( E T ( Ω ) ) c ) ≤ K Ω ε . (144) \mathbb{P}\left((E_T^{(\Omega)})^c\right)\le K_{\Omega}\varepsilon. \tag*{(144)} P ( ( E T ( Ω ) ) c ) ≤ K Ω ε . (144)
It remains to prove the pathwise bound on Ω t \Omega_t Ω t . Work on E 0 ∩ E T ( Ω ) E_0\cap E_T^{(\Omega)} E 0 ∩ E T ( Ω ) . From (142) and (143) ,
Ω t ≤ Ω 0 + 3 δ ∑ k < t ∥ g ˉ k ∥ ∞ + C d , t ≤ T . (145) \Omega_t\le\Omega_0+3\delta\sum_{k<t}\|\bar{g}_k\|_\infty+\frac{C}{\sqrt{d}},\qquad t\le T. \tag*{(145)} Ω t ≤ Ω 0 + 3 δ k < t ∑ ∥ g ˉ k ∥ ∞ + d C , t ≤ T . (145)
We now bound the mean-field term for t ≤ σ t \le\sigma t ≤ σ . Since k < σ k < \sigma k < σ implies k < τ + k < \tau_{+} k < τ + , we have ρ k , χ k ≥ 0 \rho_k, \chi_k \ge0 ρ k , χ k ≥ 0 . Since k < σ k < \sigma k < σ also implies k < τ A k < \tau_A k < τ A , we have Z k 2 < A η k Z_k^2 < A\eta_k Z k 2 < A η k . By (48) in Lemma C.4,
∥ g ˉ k ∥ ∞ ≤ C Z k ∥ θ ∗ ∥ ∞ + C Z k 2 η k + C η k 2 . \|\bar{g}_k\|_{\infty} \le C Z_k\|\theta^*\|_{\infty} + C Z_k^2\eta_k + C\eta_k^2. ∥ g ˉ k ∥ ∞ ≤ C Z k ∥ θ ∗ ∥ ∞ + C Z k 2 η k + C η k 2 .
On E 0 \mathcal{E}_0 E 0 , ∥ θ ∗ ∥ ∞ ≤ C log d / d \|\theta^*\|_{\infty} \le C\sqrt{\log d/d} ∥ θ ∗ ∥ ∞ ≤ C log d / d . Using Z k 2 < A η k Z_k^2 < A\eta_k Z k 2 < A η k and the assumed envelope for η k \eta_k η k , we obtain
Z k ∥ θ ∗ ∥ ∞ ≤ A η k ∥ θ ∗ ∥ ∞ ≤ C log d + 1 + k δ d log d d . Z_k\|\theta^*\|_{\infty} \le\sqrt{A\eta_k}\|\theta^*\|_{\infty} \le C\sqrt{\frac{\sqrt{\log d}+1+k\delta}{\sqrt{d}}}\sqrt{\frac{\log d}{d}}. Z k ∥ θ ∗ ∥ ∞ ≤ A η k ∥ θ ∗ ∥ ∞ ≤ C d log d + 1 + k δ d log d .
Since k δ ≤ T δ ≤ C T log d k\delta\le T\delta\le C_T\log d k δ ≤ T δ ≤ C T log d , the right-hand side is bounded by C ( log d ) d − 3 / 4 ≤ C d − 1 / 2 C(\log d)d^{-3/4} \le Cd^{-1/2} C ( log d ) d − 3/4 ≤ C d − 1/2 for all sufficiently large d d d . Also, Z k 2 η k ≤ A η k 2 Z_k^2\eta_k \le A\eta_k^2 Z k 2 η k ≤ A η k 2 . Hence, for k < σ k < \sigma k < σ ,
∥ g ˉ k ∥ ∞ ≤ C d + C η k 2 . (146) \|\bar{g}_k\|_{\infty} \le\frac{C}{\sqrt{d}}+C\eta_k^2. \tag*{(146)} ∥ g ˉ k ∥ ∞ ≤ d C + C η k 2 . (146)
Therefore, for t ≤ σ t \le\sigma t ≤ σ ,
δ ∑ k < t ∥ g ˉ k ∥ ∞ ≤ C t δ d + C δ ∑ k < t η k 2 . \delta\sum_{k<t}\|\bar{g}_k\|_{\infty} \le C\frac{t\delta}{\sqrt{d}}+C\delta\sum_{k<t}\eta_k^2. δ k < t ∑ ∥ g ˉ k ∥ ∞ ≤ C d t δ + C δ k < t ∑ η k 2 .
Using the assumed η \eta η -envelope,
δ ∑ k < t η k 2 ≤ C d δ ∑ k < t ( log d + 1 + k δ ) 2 ≤ C ( log d ) 3 d = o ( d − 1 / 2 ) . \delta\sum_{k<t}\eta_k^2 \le\frac{C}{d}\delta\sum_{k<t}\left(\sqrt{\log d}+1+k\delta\right)^2 \le\frac{C(\log d)^3}{d}=o(d^{-1/2}). δ k < t ∑ η k 2 ≤ d C δ k < t ∑ ( log d + 1 + k δ ) 2 ≤ d C ( log d ) 3 = o ( d − 1/2 ) .
Plugging this into (142), and using Ω 0 ≤ C / d \Omega_0 \le C/\sqrt{d} Ω 0 ≤ C / d on E 0 \mathcal{E}_0 E 0 , gives
Ω t ≤ C 1 + t δ d , 0 ≤ t ≤ σ . \Omega_t \le C\frac{1+t\delta}{\sqrt{d}},\qquad0\le t\le\sigma. Ω t ≤ C d 1 + t δ , 0 ≤ t ≤ σ .
This proves the desired Ω t \Omega_t Ω t -bound.
Finally,
ω t = ∑ i ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 ≤ 4 ∑ i ∣ θ t , i ∣ 3 + 4 ∑ i ∣ θ i ∗ ∣ 3 = 4 Ω t + 4 ∥ θ ∗ ∥ 3 3 . \omega_t=\sum_i\left(|\theta_{t,i}|+|\theta_i^*|\right)^3\le4\sum_i|\theta_{t,i}|^3+4\sum_i|\theta_i^*|^3=4\Omega_t+4\|\theta^*\|_3^3. ω t = i ∑ ( ∣ θ t , i ∣ + ∣ θ i ∗ ∣ ) 3 ≤ 4 i ∑ ∣ θ t , i ∣ 3 + 4 i ∑ ∣ θ i ∗ ∣ 3 = 4 Ω t + 4∥ θ ∗ ∥ 3 3 .
On E 0 \mathcal{E}_0 E 0 , ∥ θ ∗ ∥ 3 3 ≤ C / d \|\theta^*\|_3^3\le C/\sqrt{d} ∥ θ ∗ ∥ 3 3 ≤ C / d . Therefore
ω t ≤ C ω 1 + t δ d , 0 ≤ t ≤ σ . \omega_t\le C_\omega\frac{1+t\delta}{\sqrt{d}},\qquad0\le t\le\sigma. ω t ≤ C ω d 1 + t δ , 0 ≤ t ≤ σ .
This completes the proof.
Weak Recovery Dynamics
Good Events for Weak Recovery. Now, we define the good event for weak recovery by taking the intersection of E d e l o c \mathcal{E}_{\mathrm{deloc}} E deloc from B.2, E T ( T ) \mathcal{E}^{(T)}_T E T ( T ) from (78), E T ( Z ) \mathcal{E}^{(Z)}_T E T ( Z ) from (101), E T ( ρ , χ ) \mathcal{E}^{(\rho,\chi)}_T E T ( ρ , χ ) from (104), E T ( η ) \mathcal{E}^{(\eta)}_T E T ( η ) from (131), and E T ( Ω ) \mathcal{E}^{(\Omega)}_T E T ( Ω ) from (140):
G T w e a k : = E 0 ∩ E T ( T ) ∩ E T ( Z ) ∩ E T ( ρ , χ ) ∩ E T ( η ) ∩ E T ( Ω ) . (147) \mathcal{G}^{\mathrm{weak}}_T:=\mathcal{E}_0\cap\mathcal{E}^{(T)}_T\cap\mathcal{E}^{(Z)}_T\cap\mathcal{E}^{(\rho,\chi)}_T\cap\mathcal{E}^{(\eta)}_T\cap\mathcal{E}^{(\Omega)}_T. \tag*{(147)} G T weak := E 0 ∩ E T ( T ) ∩ E T ( Z ) ∩ E T ( ρ , χ ) ∩ E T ( η ) ∩ E T ( Ω ) . (147)
With their respective probabilistic controls made in (21), (89), (103), (105), (135), (141), they are true with probability ( 1 − K a l g ϵ ) ( 1 − d − K i n i ) (1-K_{\mathrm{alg}}\epsilon)(1-d^{-K_{\mathrm{ini}}}) ( 1 − K alg ϵ ) ( 1 − d − K ini ) .
Phase I Dynamics, Weak Recovery
Lemma D.13 (Phase I: pull-up to the signal-dominant scale). Work on G T w e a k \mathcal{G}^{\mathrm{weak}}_T G T weak . Then, up to τ A ∧ τ B \tau_A\wedge\tau_B τ A ∧ τ B ,
η t ≤ C η log d + 1 + t δ d , ω t ≤ C ω 1 + t δ d . (148) \eta_t\le C_\eta\frac{\sqrt{\log d}+1+t\delta}{\sqrt{d}},\qquad\omega_t\le C_\omega\frac{1+t\delta}{\sqrt{d}}. \tag*{(148)} η t ≤ C η d log d + 1 + t δ , ω t ≤ C ω d 1 + t δ . (148) Moreover,
τ A < τ B , τ + > τ A , τ A δ ≤ C log d , Z τ A ≥ c A d − 1 / 4 . (149) \tau_A<\tau_B,\qquad\tau_+>\tau_A,\qquad\tau_A\delta\le C\log d,\qquad Z_{\tau_A}\ge c_A d^{-1/4}. \tag*{(149)} τ A < τ B , τ + > τ A , τ A δ ≤ C log d , Z τ A ≥ c A d − 1/4 . (149) Equivalently,
ρ t , χ t > 0 for all 0 ≤ t ≤ τ A . (150) \rho_t,\chi_t>0\qquad\text{for all }0\le t\le\tau_A. \tag*{(150)} ρ t , χ t > 0 for all 0 ≤ t ≤ τ A . (150) Finally,
Z τ A ≤ C A log d d − 1 / 4 + o ( 1 ) . (151) Z_{\tau_A}\le C_A\sqrt{\log d}\,d^{-1/4}+o(1). \tag*{(151)} Z τ A ≤ C A log d d − 1/4 + o ( 1 ) . (151)
Proof. Work throughout on G T w e a k G_T^{\mathrm{weak}} G T weak . We use the favorable initialization event in the form
ρ 0 , χ 0 ≥ z − d , η 0 ≤ C log d d . \rho_0,\chi_0 \ge\frac{z_-}{\sqrt{d}}, \qquad\eta_0 \le C\sqrt{\frac{\log d}{d}}. ρ 0 , χ 0 ≥ d z − , η 0 ≤ C d log d .
where z − > 0 z_- > 0 z − > 0 is fixed large enough below. Recall that
τ + : = inf { t : ρ t < 0 or χ t < 0 } . \tau_+ := \inf\{t:\rho_t<0\text{ or }\chi_t<0\}. τ + := inf { t : ρ t < 0 or χ t < 0 } .
Set
σ : = τ A ∧ τ B ∧ τ + ∧ T . \sigma:= \tau_A\wedge\tau_B\wedge\tau_+\wedge T. σ := τ A ∧ τ B ∧ τ + ∧ T .
It is enough to prove the desired estimates up to σ \sigma σ , and then show that σ = τ A \sigma=\tau_A σ = τ A .
Step 1: Phase-I η \eta η -envelope. Let
Θ t : = max 0 ≤ s ≤ t η s , H I ( t ) : = K I log d + 1 + t δ d , \Theta_t := \max_{0\le s\le t}\eta_s,\qquad H_I(t):=K_I\frac{\sqrt{\log d}+1+t\delta}{\sqrt d}, Θ t := 0 ≤ s ≤ t max η s , H I ( t ) := K I d log d + 1 + t δ ,
where K I K_I K I will be chosen large. By the discrete η \eta η -estimate (133) , on E T ( η ) E_T^{(\eta)} E T ( η ) ,
η t ≤ C ( η 0 + δ ∑ k < t ∥ g ˉ k ( θ ) ∥ ∞ + Λ η + C c u b d ) , t ≤ T . \eta_t\le C\left(\eta_0+\delta\sum_{k<t}\lVert\bar g_k^{(\theta)}\rVert_\infty+\Lambda_\eta+\frac{C_{\mathrm{cub}}}{\sqrt d}\right),\qquad t\le T. η t ≤ C ( η 0 + δ k < t ∑ ∥ g ˉ k ( θ ) ∥ ∞ + Λ η + d C cub ) , t ≤ T .
The definition of E T ( η ) E_T^{(\eta)} E T ( η ) gives
Λ η ≤ C log ( 40 d / ε ) d ≤ C log d + 1 + t δ d , t ≤ T . \Lambda_\eta\le C\sqrt{\frac{\log(40d/\varepsilon)}{d}}\le C\frac{\sqrt{\log d}+1+t\delta}{\sqrt d},\qquad t\le T. Λ η ≤ C d log ( 40 d / ε ) ≤ C d log d + 1 + t δ , t ≤ T .
For k < σ k<\sigma k < σ , we have k < τ + k<\tau_+ k < τ + , hence ρ k , χ k ≥ 0 \rho_k,\chi_k\ge0 ρ k , χ k ≥ 0 , and also k < τ A k<\tau_A k < τ A , hence Z k 2 < A η k Z_k^2<A\eta_k Z k 2 < A η k . The population ℓ ∞ \ell_\infty ℓ ∞ -gradient bound (49) gives
∥ g ˉ k ( θ ) ∥ ∞ ≤ C Z k ∥ θ ∗ ∥ ∞ + C Z k 2 η k + C η k 2 . \lVert\bar g_k^{(\theta)}\rVert_\infty\le CZ_k\lVert\theta^*\rVert_\infty+CZ_k^2\eta_k+C\eta_k^2. ∥ g ˉ k ( θ ) ∥ ∞ ≤ C Z k ∥ θ ∗ ∥ ∞ + C Z k 2 η k + C η k 2 .
Using Z k 2 < A η k Z_k^2<A\eta_k Z k 2 < A η k , this becomes
∥ g ˉ k ( θ ) ∥ ∞ ≤ C η k ∥ θ ∗ ∥ ∞ + C η k 2 . \lVert\bar g_k^{(\theta)}\rVert_\infty\le C\sqrt{\eta_k}\lVert\theta^*\rVert_\infty+C\eta_k^2. ∥ g ˉ k ( θ ) ∥ ∞ ≤ C η k ∥ θ ∗ ∥ ∞ + C η k 2 .
Since ∥ θ ∗ ∥ ∞ ≤ C log d / d \lVert\theta^*\rVert_\infty\le C\sqrt{\log d/d} ∥ θ ∗ ∥ ∞ ≤ C log d / d , and since throughout the bootstrap k δ ≤ T δ ≤ C T log d k\delta\le T\delta\le C_T\log d k δ ≤ T δ ≤ C T log d , the implication η k ≤ H I ( k ) \eta_k\le H_I(k) η k ≤ H I ( k ) gives
η k ∥ θ ∗ ∥ ∞ ≤ C log d d 3 / 4 ≤ C d \sqrt{\eta_k}\lVert\theta^*\rVert_\infty\le C\frac{\log d}{d^{3/4}}\le\frac{C}{\sqrt d} η k ∥ θ ∗ ∥ ∞ ≤ C d 3/4 log d ≤ d C
for all sufficiently large d d d . Thus, under the bootstrap,
∥ g ˉ k ( θ ) ∥ ∞ ≤ C d + C η k 2 . \lVert\bar g_k^{(\theta)}\rVert_\infty\le\frac{C}{\sqrt d}+C\eta_k^2. ∥ g ˉ k ( θ ) ∥ ∞ ≤ d C + C η k 2 .
Let
τ H : = inf { t ≤ σ : Θ t > H I ( t ) } . \tau_H:=\inf\{t\le\sigma:\Theta_t>H_I(t)\}. τ H := inf { t ≤ σ : Θ t > H I ( t )} .
Suppose for contradiction that τ H ≤ σ \tau_H\le\sigma τ H ≤ σ . Then for every k < τ H k<\tau_H k < τ H , η k ≤ Θ k ≤ H I ( k ) \eta_k\le\Theta_k\le H_I(k) η k ≤ Θ k ≤ H I ( k ) , and therefore
δ ∑ k < τ H ∥ g ˉ k ( θ ) ∥ ∞ ≤ C τ H δ d + C δ ∑ k < τ H H I ( k ) 2 . \delta\sum_{k<\tau_H}\lVert\bar g_k^{(\theta)}\rVert_\infty\le C\frac{\tau_H\delta}{\sqrt d}+C\delta\sum_{k<\tau_H}H_I(k)^2. δ k < τ H ∑ ∥ g ˉ k ( θ ) ∥ ∞ ≤ C d τ H δ + C δ k < τ H ∑ H I ( k ) 2 .
Moreover,
δ ∑ k < τ H H I ( k ) 2 ≤ C K I 2 d δ ∑ k < τ H ( log d + 1 + k δ ) 2 ≤ C ( log d ) 3 d = o ( log d + 1 + τ H δ d ) . \delta\sum_{k<\tau_H}H_I(k)^2\le\frac{CK_I^2}{d}\delta\sum_{k<\tau_H}(\sqrt{\log d}+1+k\delta)^2\le\frac{C(\log d)^3}{d}=o\left(\frac{\sqrt{\log d}+1+\tau_H\delta}{\sqrt d}\right). δ k < τ H ∑ H I ( k ) 2 ≤ d C K I 2 δ k < τ H ∑ ( log d + 1 + k δ ) 2 ≤ d C ( log d ) 3 = o ( d log d + 1 + τ H δ ) .
Combining the previous estimates with the bound on η t \eta_t η t , we get
Θ τ H ≤ C log d + 1 + τ H δ d + o ( log d + 1 + τ H δ d ) . \Theta_{\tau_H}\le C\frac{\sqrt{\log d}+1+\tau_H\delta}{\sqrt d}+o\left(\frac{\sqrt{\log d}+1+\tau_H\delta}{\sqrt d}\right). Θ τ H ≤ C d log d + 1 + τ H δ + o ( d log d + 1 + τ H δ ) .
Choosing K I K_I K I large enough, and then d d d large enough, gives
Θ τ H ≤ 1 2 H I ( τ H ) , \Theta_{\tau_H} \le\frac{1}{2}H_I(\tau_H), Θ τ H ≤ 2 1 H I ( τ H ) ,
contradicting the definition of τ H \tau_H τ H . Hence τ H > σ \tau_H > \sigma τ H > σ , and therefore
η t ≤ C η log d + 1 + t δ d , 0 ≤ t ≤ σ . (152) \eta_t \le C_\eta\frac{\sqrt{\log d}+1+t\delta}{\sqrt{d}}, \qquad0 \le t \le\sigma. \tag*{(152)} η t ≤ C η d log d + 1 + t δ , 0 ≤ t ≤ σ . (152)
which is exactly the assumption (136) made in Lemma D.12. Alongside the condition Z t 2 < A η t Z_t^2 < A\eta_t Z t 2 < A η t for t < τ A t < \tau_A t < τ A , we can apply Lemma D.12 to obtain
ω t ≤ C ω 1 + t δ d , 0 ≤ t ≤ σ . (153) \omega_t \le C_\omega\frac{1+t\delta}{\sqrt{d}}, \qquad0 \le t \le\sigma. \tag*{(153)} ω t ≤ C ω d 1 + t δ , 0 ≤ t ≤ σ . (153)
Step 2: Positivity and scalar Z Z Z -pull-up. For t < σ t < \sigma t < σ , we have t < τ A t < \tau_A t < τ A , hence Z t 2 < A η t Z_t^2 < A\eta_t Z t 2 < A η t . By Step 1,
η t ≤ C η log d + 1 + t δ d ≤ C log d d . \eta_t \le C_\eta\frac{\sqrt{\log d}+1+t\delta}{\sqrt{d}} \le C\frac{\log d}{\sqrt{d}}. η t ≤ C η d log d + 1 + t δ ≤ C d log d .
Thus
Z t ≤ C log d d − 1 / 4 = o ( 1 ) . Z_t \le C\sqrt{\log d}\,d^{-1/4}=o(1). Z t ≤ C log d d − 1/4 = o ( 1 ) .
For d d d large, Z t ≤ c s m Z_t \le c_{\mathrm{sm}} Z t ≤ c sm , so we can apply Lemma C.6
ρ ˙ t ≥ a χ t − b ω t , χ ˙ t ≥ a ρ t − b ω t . \dot{\rho}_t \ge a\chi_t-b\omega_t,\qquad\dot{\chi}_t \ge a\rho_t-b\omega_t. ρ ˙ t ≥ a χ t − b ω t , χ ˙ t ≥ a ρ t − b ω t .
Combining this with Lemma D.2, the Taylor event E T ( T ) E_T^{(T)} E T ( T ) , and Lemma E.7, we obtain
u t + 1 ≥ [ ( 1 − C δ 2 d ) I + a δ J ] u t − C δ ω t 1 + ε t + 1 ( u ) , u t : = ( ρ t , χ t ) ⊤ , u_{t+1} \ge\left[(1-C\delta^2d)I+a\delta J\right]u_t-C\delta\omega_t\mathbf{1}+\varepsilon_{t+1}^{(u)},\qquad u_t:=(\rho_t,\chi_t)^\top, u t + 1 ≥ [ ( 1 − C δ 2 d ) I + a δ J ] u t − C δ ω t 1 + ε t + 1 ( u ) , u t := ( ρ t , χ t ) ⊤ ,
where
J = ( 0 1 1 0 ) , 1 = ( 1 , 1 ) ⊤ , ε t + 1 ( u ) = ( N t + 1 ( ρ ) − N t ( ρ ) , N t + 1 ( χ ) − N t ( χ ) ) ⊤ . J=\begin{pmatrix}0&1\\1&0\end{pmatrix},\qquad\mathbf{1}=(1,1)^\top,\qquad\varepsilon_{t+1}^{(u)}=\left(N_{t+1}^{(\rho)}-N_t^{(\rho)},N_{t+1}^{(\chi)}-N_t^{(\chi)}\right)^\top. J = ( 0 1 1 0 ) , 1 = ( 1 , 1 ) ⊤ , ε t + 1 ( u ) = ( N t + 1 ( ρ ) − N t ( ρ ) , N t + 1 ( χ ) − N t ( χ ) ) ⊤ .
Let
B : = ( 1 − C δ 2 d ) I + a δ J . B:=(1-C\delta^2d)I+a\delta J. B := ( 1 − C δ 2 d ) I + a δ J .
For d d d large, B B B has nonnegative entries. Its eigenvalues in the 1 \mathbf{1} 1 - and ( 1 , − 1 ) ⊤ (1,-1)^\top ( 1 , − 1 ) ⊤ -directions are
γ + : = 1 + a δ − C δ 2 d , γ − : = 1 − a δ − C δ 2 d . \gamma_+:=1+a\delta-C\delta^2d,\qquad\gamma_-:=1-a\delta-C\delta^2d. γ + := 1 + a δ − C δ 2 d , γ − := 1 − a δ − C δ 2 d .
After decreasing a a a , we may assume
γ + ≥ 1 + β δ , 0 < γ − < 1 , β > 0. \gamma_+\ge1+\beta\delta,\qquad0<\gamma_-<1,\qquad\beta>0. γ + ≥ 1 + β δ , 0 < γ − < 1 , β > 0.
Unrolling the previous cooperative recursion and applying the summation-by-parts bounds to the Z Z Z -noise and D D D -noise gives, componentwise,
ρ t , χ t ≥ γ + t [ z − d − C ∑ k < t γ + − ( k + 1 ) δ ω k − C ( λ Z + λ ρ + λ χ ) ] , 0 ≤ t ≤ σ . (154) \rho_t,\chi_t\ge\gamma_+^t\left[\frac{z_-}{\sqrt{d}}-C\sum_{k<t}\gamma_+^{-(k+1)}\delta\omega_k-C(\lambda_Z+\lambda_\rho+\lambda_\chi)\right],\qquad0\le t\le\sigma. \tag*{(154)} ρ t , χ t ≥ γ + t [ d z − − C k < t ∑ γ + − ( k + 1 ) δ ω k − C ( λ Z + λ ρ + λ χ ) ] , 0 ≤ t ≤ σ . (154)
Indeed, the Z Z Z -noise is controlled by Corollary E.9, while the contracting D D D -noise is controlled by Corollary E.10.
By (150),
∑ k < t γ + − ( k + 1 ) δ ω k ≤ C d ( δ ∑ k ≥ 0 γ + − ( k + 1 ) + δ 2 ∑ k ≥ 0 k γ + − ( k + 1 ) ) ≤ C d . \sum_{k<t}\gamma_+^{-(k+1)}\delta\omega_k\le\frac{C}{\sqrt{d}}\left(\delta\sum_{k\ge0}\gamma_+^{-(k+1)}+\delta^2\sum_{k\ge0}k\gamma_+^{-(k+1)}\right)\le\frac{C}{\sqrt{d}}. k < t ∑ γ + − ( k + 1 ) δ ω k ≤ d C ( δ k ≥ 0 ∑ γ + − ( k + 1 ) + δ 2 k ≥ 0 ∑ k γ + − ( k + 1 ) ) ≤ d C .
Since
λ Z + λ ρ + λ χ = O ( d − 1 / 2 ) , \lambda_Z+\lambda_\rho+\lambda_\chi=O(d^{-1/2}), λ Z + λ ρ + λ χ = O ( d − 1/2 ) ,
choosing z − z_- z − larger than the constants in the bracket yields
ρ t , χ t > 0 , 0 ≤ t ≤ σ . (155) \rho_t,\chi_t>0,\qquad0\le t\le\sigma. \tag*{(155)} ρ t , χ t > 0 , 0 ≤ t ≤ σ . (155)
In particular, σ ≠ τ + \sigma\ne\tau_{+} σ = τ + .
Now sum the two one-step inequalities. For
ε t ( Z ) : = N t + 1 ( Z ) − N t ( Z ) , \varepsilon_t^{(Z)} := N_{t+1}^{(Z)} - N_t^{(Z)}, ε t ( Z ) := N t + 1 ( Z ) − N t ( Z ) ,
we get
Z t + 1 ≥ ( 1 + β δ ) Z t − C δ ω t + ε t ( Z ) , t < σ . Z_{t+1} \ge(1+\beta\delta)Z_t - C\delta\omega_t + \varepsilon_t^{(Z)}, \qquad t<\sigma. Z t + 1 ≥ ( 1 + β δ ) Z t − C δ ω t + ε t ( Z ) , t < σ .
Set γ : = 1 + β δ \gamma:= 1+\beta\delta γ := 1 + β δ . Multiplying by γ − ( t + 1 ) \gamma^{-(t+1)} γ − ( t + 1 ) and summing,
γ − t Z t ≥ Z 0 − C ∑ k < t γ − ( k + 1 ) δ ω k + ∑ k < t γ − ( k + 1 ) ε k ( Z ) . \gamma^{-t}Z_t \ge Z_0 - C\sum_{k<t}\gamma^{-(k+1)}\delta\omega_k + \sum_{k<t}\gamma^{-(k+1)}\varepsilon_k^{(Z)}. γ − t Z t ≥ Z 0 − C k < t ∑ γ − ( k + 1 ) δ ω k + k < t ∑ γ − ( k + 1 ) ε k ( Z ) .
The weighted ω \omega ω -sum is O ( d − 1 / 2 ) O(d^{-1/2}) O ( d − 1/2 ) by (150). Also, on E T ( Z ) E_T^{(Z)} E T ( Z ) ,
sup u ≤ T ∣ N u ( Z ) ∣ ≤ 2 λ Z , \sup_{u\le T}|N_u^{(Z)}|\le2\lambda Z, u ≤ T sup ∣ N u ( Z ) ∣ ≤ 2 λ Z ,
so Corollary E.9 gives
∣ ∑ k < t γ − ( k + 1 ) ε k ( Z ) ∣ ≤ 4 λ Z . \left|\sum_{k<t}\gamma^{-(k+1)}\varepsilon_k^{(Z)}\right|\le4\lambda Z. k < t ∑ γ − ( k + 1 ) ε k ( Z ) ≤ 4 λ Z .
Since Z 0 ≥ 2 z − d − 1 / 2 Z_0\ge2z_-d^{-1/2} Z 0 ≥ 2 z − d − 1/2 , increasing z − z_- z − if necessary gives a constant c − > 0 c_->0 c − > 0 such that
Z t ≥ c − d γ t , 0 ≤ t ≤ σ . (156) Z_t\ge\frac{c_-}{\sqrt d}\gamma^t,\qquad0\le t\le\sigma. \tag*{(156)} Z t ≥ d c − γ t , 0 ≤ t ≤ σ . (156)
Step 3: Some stopping event occurs by O ( δ − 1 log d ) O(\delta^{-1}\log d) O ( δ − 1 log d ) . Let
t 0 : = ⌈ c 6 δ − 1 log d ⌉ , t_0:=\left\lceil c_6\delta^{-1}\log d\right\rceil, t 0 := ⌈ c 6 δ − 1 log d ⌉ ,
where c 6 c_6 c 6 will be chosen large. Choose the weak-recovery horizon constant C T C_T C T so that t 0 < T t_0<T t 0 < T . Suppose, for contradiction, that σ > t 0 \sigma>t_0 σ > t 0 . Then t 0 < τ A t_0<\tau_A t 0 < τ A , hence
Z t 0 2 < A η t 0 . Z_{t_0}^2<A\eta_{t_0}. Z t 0 2 < A η t 0 .
Using (153) and (149),
c − 2 d γ 2 t 0 ≤ Z t 0 2 < A η t 0 ≤ C log d + 1 + t 0 δ d ≤ C log d d . \frac{c_-^2}{d}\gamma^{2t_0}\le Z_{t_0}^2<A\eta_{t_0}\le C\frac{\sqrt{\log d}+1+t_0\delta}{\sqrt d}\le C\frac{\log d}{\sqrt d}. d c − 2 γ 2 t 0 ≤ Z t 0 2 < A η t 0 ≤ C d log d + 1 + t 0 δ ≤ C d log d .
Since log γ ≥ c δ \log\gamma\ge c\delta log γ ≥ cδ for some c > 0 c>0 c > 0 , we have
γ 2 t 0 ≥ d 2 c c 6 + o ( 1 ) . \gamma^{2t_0}\ge d^{2cc_6+o(1)}. γ 2 t 0 ≥ d 2 c c 6 + o ( 1 ) .
Choosing c 6 c_6 c 6 so large that 2 c c 6 > 1 / 2 2cc_6>1/2 2 c c 6 > 1/2 gives a contradiction for all large d d d . Therefore
σ ≤ t 0 , σ δ ≤ c 6 log d + O ( δ ) . (157) \sigma\le t_0,\qquad\sigma\delta\le c_6\log d+O(\delta). \tag*{(157)} σ ≤ t 0 , σ δ ≤ c 6 log d + O ( δ ) . (157)
Step 4: The first stopping event is τ A \tau_A τ A . Since t 0 < T t_0<T t 0 < T , (154) implies σ ≠ T \sigma\ne T σ = T . By (152), σ ≠ τ + \sigma\ne\tau_+ σ = τ + . Thus σ \sigma σ is either τ A \tau_A τ A or τ B \tau_B τ B .
Suppose for contradiction that σ = τ B ≤ τ A \sigma=\tau_B\le\tau_A σ = τ B ≤ τ A . Since τ B ≤ t 0 \tau_B\le t_0 τ B ≤ t 0 , at time τ B − 1 \tau_B-1 τ B − 1 we have
Z τ B − 1 2 < A η τ B − 1 . Z_{\tau_B-1}^2<A\eta_{\tau_B-1}. Z τ B − 1 2 < A η τ B − 1 .
Thus, by (149),
Z τ B − 1 ≤ C log d d 1 / 4 = o ( 1 ) . Z_{\tau_B-1}\le C\frac{\sqrt{\log d}}{d^{1/4}}=o(1). Z τ B − 1 ≤ C d 1/4 log d = o ( 1 ) .
The one-step expansion gives
Z τ B = Z τ B − 1 + δ F τ B − 1 ( Z ) + ε τ B − 1 ( Z ) . Z_{\tau_B}=Z_{\tau_B-1}+\delta F_{\tau_B-1}^{(Z)}+\varepsilon_{\tau_B-1}^{(Z)}. Z τ B = Z τ B − 1 + δ F τ B − 1 ( Z ) + ε τ B − 1 ( Z ) .
By Lemma E.7,
∣ F t ( Z ) ∣ ≤ C d + C δ d , |F_t^{(Z)}|\le C\sqrt d+C\delta d, ∣ F t ( Z ) ∣ ≤ C d + C δ d ,
and hence
δ ∣ F t ( Z ) ∣ ≤ C δ d + C δ 2 d = o ( 1 ) . \delta|F_t^{(Z)}| \le C\delta\sqrt{d}+C\delta^2d=o(1). δ ∣ F t ( Z ) ∣ ≤ C δ d + C δ 2 d = o ( 1 ) .
Also,
∣ ε τ B − 1 ( Z ) ∣ ≤ ∣ N τ B ( Z ) ∣ + ∣ N τ B − 1 ( Z ) ∣ ≤ 4 λ Z = o ( 1 ) . |\varepsilon_{\tau_B-1}^{(Z)}|\le|N_{\tau_B}^{(Z)}|+|N_{\tau_B-1}^{(Z)}|\le4\lambda Z=o(1). ∣ ε τ B − 1 ( Z ) ∣ ≤ ∣ N τ B ( Z ) ∣ + ∣ N τ B − 1 ( Z ) ∣ ≤ 4 λ Z = o ( 1 ) .
Therefore Z τ B = o ( 1 ) Z_{\tau_B}=o(1) Z τ B = o ( 1 ) . But (152) gives ρ τ B , χ τ B > 0 \rho_{\tau_B},\chi_{\tau_B}>0 ρ τ B , χ τ B > 0 , and by definition of τ B \tau_B τ B , one of ρ τ B , χ τ B \rho_{\tau_B},\chi_{\tau_B} ρ τ B , χ τ B is at least 1 / 2 1/2 1/2 . Hence Z τ B ≥ 1 / 2 Z_{\tau_B}\ge1/2 Z τ B ≥ 1/2 , a contradiction. Therefore
σ = τ A , τ A < τ B , τ A δ ≤ C log d . \sigma=\tau_A,\qquad\tau_A<\tau_B,\qquad\tau_A\delta\le C\log d. σ = τ A , τ A < τ B , τ A δ ≤ C log d .
The bounds
η t ≤ C η log d + 1 + t δ d , ω t ≤ C ω 1 + t δ d \eta_t\le C_\eta\frac{\sqrt{\log d}+1+t\delta}{\sqrt d},\qquad\omega_t\le C_\omega\frac{1+t\delta}{\sqrt d} η t ≤ C η d log d + 1 + t δ , ω t ≤ C ω d 1 + t δ
now hold for all t ≤ τ A t\le\tau_A t ≤ τ A , and since τ A < τ B \tau_A<\tau_B τ A < τ B , also up to τ A ∧ τ B \tau_A\wedge\tau_B τ A ∧ τ B .
Step 5: Size of Z τ A Z_{\tau_A} Z τ A . By definition of τ A \tau_A τ A ,
Z τ A 2 ≥ A η τ A . Z_{\tau_A}^2\ge A\eta_{\tau_A}. Z τ A 2 ≥ A η τ A .
Since θ τ A ∈ S d − 1 \theta_{\tau_A}\in S^{d-1} θ τ A ∈ S d − 1 ,
∥ θ τ A ∥ ∞ ≥ d − 1 / 2 , \|\theta_{\tau_A}\|_\infty\ge d^{-1/2}, ∥ θ τ A ∥ ∞ ≥ d − 1/2 ,
and therefore η τ A ≥ d − 1 / 2 \eta_{\tau_A}\ge d^{-1/2} η τ A ≥ d − 1/2 . Hence
Z τ A ≥ A d − 1 / 4 . Z_{\tau_A}\ge\sqrt{A}d^{-1/4}. Z τ A ≥ A d − 1/4 .
This proves the lower bound with c A : = A c_A:=\sqrt{A} c A := A .
For the upper bound, first note that τ A ≥ 1 \tau_A\ge1 τ A ≥ 1 for large d d d , since Z 0 2 = O ( d − 1 ) Z_0^2=O(d^{-1}) Z 0 2 = O ( d − 1 ) while η 0 ≥ d − 1 / 2 \eta_0\ge d^{-1/2} η 0 ≥ d − 1/2 . Hence τ A − 1 < τ A \tau_A-1<\tau_A τ A − 1 < τ A , and so
Z τ A − 1 2 < A η τ A − 1 . Z_{\tau_A-1}^2<A\eta_{\tau_A-1}. Z τ A − 1 2 < A η τ A − 1 .
Using the η \eta η -bound and τ A δ ≤ C log d \tau_A\delta\le C\log d τ A δ ≤ C log d ,
Z τ A − 1 ≤ C log d d 1 / 4 . Z_{\tau_A-1}\le C\frac{\sqrt{\log d}}{d^{1/4}}. Z τ A − 1 ≤ C d 1/4 log d .
The one-step expansion gives
Z τ A ≤ Z τ A − 1 + δ ∣ F τ A − 1 ( Z ) ∣ + ∣ ε τ A − 1 ( Z ) ∣ . Z_{\tau_A}\le Z_{\tau_A-1}+\delta|F_{\tau_A-1}^{(Z)}|+|\varepsilon_{\tau_A-1}^{(Z)}|. Z τ A ≤ Z τ A − 1 + δ ∣ F τ A − 1 ( Z ) ∣ + ∣ ε τ A − 1 ( Z ) ∣.
As above,
δ ∣ F τ A − 1 ( Z ) ∣ ≤ C δ d + C δ 2 d = o ( 1 ) , ∣ ε τ A − 1 ( Z ) ∣ ≤ 4 λ Z = o ( 1 ) . \delta|F_{\tau_A-1}^{(Z)}|\le C\delta\sqrt{d}+C\delta^2d=o(1),\qquad|\varepsilon_{\tau_A-1}^{(Z)}|\le4\lambda Z=o(1). δ ∣ F τ A − 1 ( Z ) ∣ ≤ C δ d + C δ 2 d = o ( 1 ) , ∣ ε τ A − 1 ( Z ) ∣ ≤ 4 λ Z = o ( 1 ) .
Therefore
Z τ A ≤ C A log d d − 1 / 4 + o ( 1 ) . Z_{\tau_A}\le C_A\sqrt{\log d}\,d^{-1/4}+o(1). Z τ A ≤ C A log d d − 1/4 + o ( 1 ) .
This completes the proof.
Lemma D.14 (Entry balance at τ A \tau_A τ A ). Work on G T w e a k G_T^{\mathrm{weak}} G T weak , and assume the conclusions of Lemma D.13.
Let
D t : = ρ t − χ t , λ D : = λ ρ + λ χ . D_t:=\rho_t-\chi_t,\qquad\lambda_D:=\lambda_\rho+\lambda_\chi. D t := ρ t − χ t , λ D := λ ρ + λ χ . Then
∣ D τ A ∣ ≤ C ( log d ) 2 d + C λ D + C δ d log d d 1 / 4 . |D_{\tau_A}|\le C\frac{(\log d)^2}{\sqrt d}+C\lambda_D+C\delta d\frac{\sqrt{\log d}}{d^{1/4}}. ∣ D τ A ∣ ≤ C d ( log d ) 2 + C λ D + C δ d d 1/4 log d . In particular,
∣ D τ A ∣ = o ( Z τ A ) . |D_{\tau_A}|=o(Z_{\tau_A}). ∣ D τ A ∣ = o ( Z τ A ) . Hence, for all sufficiently large d d d ,
∣ D τ A ∣ ≤ 1 8 Z τ A . (158) |D_{\tau_A}|\le\frac{1}{8}Z_{\tau_A}. \tag*{(158)} ∣ D τ A ∣ ≤ 8 1 Z τ A . (158)
Proof. By Lemma D.13, on 0 ≤ t ≤ τ A 0 \le t \le\tau_A 0 ≤ t ≤ τ A we have
τ A < τ B , τ + > τ A , τ A δ ≤ C log d , \tau_A < \tau_B,\qquad\tau_+ > \tau_A,\qquad\tau_A\delta\le C\log d, τ A < τ B , τ + > τ A , τ A δ ≤ C log d ,
and for t ≤ τ A t \le\tau_A t ≤ τ A ,
η t ≤ C η log d + 1 + t δ d . \eta_t \le C_\eta\frac{\sqrt{\log d}+1+t\delta}{\sqrt{d}}. η t ≤ C η d log d + 1 + t δ .
Moreover, for t < τ A t < \tau_A t < τ A ,
Z t 2 < A η t . Z_t^2 < A\eta_t. Z t 2 < A η t .
Therefore, for t < τ A t < \tau_A t < τ A ,
Z t ≤ C log d d 1 / 4 = o ( 1 ) . Z_t \le C\frac{\sqrt{\log d}}{d^{1/4}}=o(1). Z t ≤ C d 1/4 log d = o ( 1 ) .
Thus, throughout the interval 0 ≤ t < τ A 0 \le t < \tau_A 0 ≤ t < τ A , the overlaps are nonnegative and lie in the small-overlap regime. Hence, by (52) from Lemma C.6,
D ˙ t = − C ~ ( 1 + ρ t χ t ) D t + E t , ∣ E t ∣ ≤ C ( Z t 2 + ω t ) . \dot{D}_t=-\widetilde{C}(1+\rho_t\chi_t)D_t+E_t,\qquad|E_t|\le C(Z_t^2+\omega_t). D ˙ t = − C ( 1 + ρ t χ t ) D t + E t , ∣ E t ∣ ≤ C ( Z t 2 + ω t ) .
Now use the one-step SGD expansion. Let
a t : = E [ ∥ g t ( β ) ∥ 2 ∣ F t ] , b t : = E [ ∥ g t ( θ ) ∥ 2 ∣ F t ] . a_t:=\mathbb{E}[\lVert g_t^{(\beta)}\rVert^2\mid\mathcal{F}_t],\qquad b_t:=\mathbb{E}[\lVert g_t^{(\theta)}\rVert^2\mid\mathcal{F}_t]. a t := E [∥ g t ( β ) ∥ 2 ∣ F t ] , b t := E [∥ g t ( θ ) ∥ 2 ∣ F t ] .
Subtracting the one-step overlap expansions gives
D t + 1 = D t + δ ( ρ ˙ t − χ ˙ t ) − δ 2 2 ( a t ρ t − b t χ t ) + ε t ( D ) , D_{t+1}=D_t+\delta(\dot{\rho}_t-\dot{\chi}_t)-\frac{\delta^2}{2}(a_t\rho_t-b_t\chi_t)+\varepsilon_t^{(D)}, D t + 1 = D t + δ ( ρ ˙ t − χ ˙ t ) − 2 δ 2 ( a t ρ t − b t χ t ) + ε t ( D ) ,
where
ε t ( D ) : = N t + 1 ( D ) − N t ( D ) , N t ( D ) : = N t ( ρ ) − N t ( χ ) . \varepsilon_t^{(D)}:=N_{t+1}^{(D)}-N_t^{(D)},\qquad N_t^{(D)}:=N_t^{(\rho)}-N_t^{(\chi)}. ε t ( D ) := N t + 1 ( D ) − N t ( D ) , N t ( D ) := N t ( ρ ) − N t ( χ ) .
Decompose
a t ρ t − b t χ t = a t + b t 2 D t + a t − b t 2 Z t . a_t\rho_t-b_t\chi_t=\frac{a_t+b_t}{2}D_t+\frac{a_t-b_t}{2}Z_t. a t ρ t − b t χ t = 2 a t + b t D t + 2 a t − b t Z t .
Therefore
D t + 1 = ( 1 − α t δ ) D t + δ e t + ε t ( D ) (159) D_{t+1}=(1-\alpha_t\delta)D_t+\delta e_t+\varepsilon_t^{(D)} \tag*{(159)} D t + 1 = ( 1 − α t δ ) D t + δ e t + ε t ( D ) (159)
where
α t : = C ~ ( 1 + ρ t χ t ) + δ 4 ( a t + b t ) , \alpha_t:=\widetilde{C}(1+\rho_t\chi_t)+\frac{\delta}{4}(a_t+b_t), α t := C ( 1 + ρ t χ t ) + 4 δ ( a t + b t ) ,
and
e t : = E t − δ 4 ( a t − b t ) Z t . e_t:=E_t-\frac{\delta}{4}(a_t-b_t)Z_t. e t := E t − 4 δ ( a t − b t ) Z t .
By Lemma E.7, a t , b t ≤ K d a_t,b_t\le Kd a t , b t ≤ K d . Hence
∣ e t ∣ ≤ C ( Z t 2 + η t ) + C δ d Z t . (160) |e_t|\le C(Z_t^2+\eta_t)+C\delta d Z_t. \tag*{(160)} ∣ e t ∣ ≤ C ( Z t 2 + η t ) + C δ d Z t . (160)
Also, since ρ t , χ t ≥ 0 \rho_t,\chi_t\ge0 ρ t , χ t ≥ 0 ,
α t ≥ C ~ . \alpha_t\ge\widetilde{C}. α t ≥ C .
After reducing the step-size constant, we may assume
0 ≤ 1 − α t δ ≤ 1 − α 0 δ < 1 0\le1-\alpha_t\delta\le1-\alpha_0\delta<1 0 ≤ 1 − α t δ ≤ 1 − α 0 δ < 1
for a fixed α 0 > 0 \alpha_0>0 α 0 > 0 .
Unroll (1) up to t = τ A t=\tau_A t = τ A . Define
P m , t : = ∏ ℓ = m t − 1 ( 1 − α ℓ δ ) , P t , t : = 1. P_{m,t}:=\prod_{\ell=m}^{t-1}(1-\alpha_\ell\delta),\qquad P_{t,t}:=1. P m , t := ℓ = m ∏ t − 1 ( 1 − α ℓ δ ) , P t , t := 1.
Then
D τ A = P 0 , τ A D 0 + ∑ k = 0 τ A − 1 P k + 1 , τ A δ e k + ∑ k = 0 τ A − 1 P k + 1 , τ A ε k ( D ) . D_{\tau_A}=P_{0,\tau_A}D_0+\sum_{k=0}^{\tau_A-1}P_{k+1,\tau_A}\delta e_k+\sum_{k=0}^{\tau_A-1}P_{k+1,\tau_A}\varepsilon_k^{(D)}. D τ A = P 0 , τ A D 0 + k = 0 ∑ τ A − 1 P k + 1 , τ A δ e k + k = 0 ∑ τ A − 1 P k + 1 , τ A ε k ( D ) .
Since 0 ≤ P k + 1 , τ A ≤ 1 0\le P_{k+1,\tau_A}\le1 0 ≤ P k + 1 , τ A ≤ 1 ,
∣ P 0 , τ A D 0 ∣ ≤ ∣ D 0 ∣ ≤ C d − 1 / 2 . |P_{0,\tau_A}D_0|\le|D_0|\le Cd^{-1/2}. ∣ P 0 , τ A D 0 ∣ ≤ ∣ D 0 ∣ ≤ C d − 1/2 .
by the favorable initialization event.
For the first forcing term, using Z k 2 ≤ A η k Z_k^2 \le A\eta_k Z k 2 ≤ A η k for k < τ A k < \tau_A k < τ A ,
δ ∑ k < τ A ( Z k 2 + η k ) ≤ C δ ∑ k < τ A η k . \delta\sum_{k<\tau_A} (Z_k^2+\eta_k) \le C\delta\sum_{k<\tau_A}\eta_k. δ k < τ A ∑ ( Z k 2 + η k ) ≤ C δ k < τ A ∑ η k .
By the Phase-I η \eta η -bound and τ A δ ≤ C log d \tau_A\delta\le C\log d τ A δ ≤ C log d ,
δ ∑ k < τ A η k ≤ C d δ ∑ k < τ A ( log d + 1 + k δ ) ≤ C ( log d ) 2 d . (161) \delta\sum_{k<\tau_A}\eta_k \le\frac{C}{\sqrt{d}}\delta\sum_{k<\tau_A}\left(\sqrt{\log d}+1+k\delta\right) \le C\frac{(\log d)^2}{\sqrt{d}}. \tag*{(161)} δ k < τ A ∑ η k ≤ d C δ k < τ A ∑ ( log d + 1 + k δ ) ≤ C d ( log d ) 2 . (161)
For the discretization forcing term, since
P k + 1 , τ A ≤ ( 1 − α 0 δ ) τ A − 1 − k , P_{k+1,\tau_A} \le(1-\alpha_0\delta)^{\tau_A-1-k}, P k + 1 , τ A ≤ ( 1 − α 0 δ ) τ A − 1 − k ,
we have
δ ∑ k < τ A P k + 1 , τ A Z k ≤ ( sup k < τ A Z k ) δ ∑ j ≥ 0 ( 1 − α 0 δ ) j ≤ C sup k < τ A Z k . \delta\sum_{k<\tau_A}P_{k+1,\tau_A}Z_k \le\left(\sup_{k<\tau_A}Z_k\right)\delta\sum_{j\ge0}(1-\alpha_0\delta)^j \le C\sup_{k<\tau_A}Z_k. δ k < τ A ∑ P k + 1 , τ A Z k ≤ ( k < τ A sup Z k ) δ j ≥ 0 ∑ ( 1 − α 0 δ ) j ≤ C k < τ A sup Z k .
As observed above,
sup k < τ A Z k ≤ C log d d 1 / 4 . \sup_{k<\tau_A} Z_k \le C\frac{\sqrt{\log d}}{d^{1/4}}. k < τ A sup Z k ≤ C d 1/4 log d .
Therefore
δ ∑ k < τ A P k + 1 , τ A δ d Z k ≤ C δ d log d d 1 / 4 . (162) \delta\sum_{k<\tau_A}P_{k+1,\tau_A}\delta d Z_k \le C\delta d\frac{\sqrt{\log d}}{d^{1/4}}. \tag*{(162)} δ k < τ A ∑ P k + 1 , τ A δ d Z k ≤ C δ d d 1/4 log d . (162)
It remains to control the noise term. For fixed τ A \tau_A τ A , the weights P k + 1 , τ A P_{k+1,\tau_A} P k + 1 , τ A , 0 ≤ k ≤ τ A − 1 0\le k\le\tau_A-1 0 ≤ k ≤ τ A − 1 , lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] and are nondecreasing in k k k . On E T ( ρ , χ ) E_T^{(\rho,\chi)} E T ( ρ , χ ) ,
sup u ≤ T ∣ N u ( D ) ∣ ≤ 2 ( λ ρ + λ χ ) = 2 λ D . \sup_{u\le T}|N_u^{(D)}|\le2(\lambda_\rho+\lambda_\chi)=2\lambda_D. u ≤ T sup ∣ N u ( D ) ∣ ≤ 2 ( λ ρ + λ χ ) = 2 λ D .
Thus Lemma E.8 gives
∣ ∑ k = 0 τ A − 1 P k + 1 , τ A ε k ( D ) ∣ ≤ 4 λ D . (163) \left|\sum_{k=0}^{\tau_A-1}P_{k+1,\tau_A}\varepsilon_k^{(D)}\right|\le4\lambda_D. \tag*{(163)} k = 0 ∑ τ A − 1 P k + 1 , τ A ε k ( D ) ≤ 4 λ D . (163)
Combining (2)–(5), and absorbing ∣ D 0 ∣ ≤ C d − 1 / 2 |D_0|\le Cd^{-1/2} ∣ D 0 ∣ ≤ C d − 1/2 into the first term, gives
∣ D τ A ∣ ≤ C ( log d ) 2 d + C λ D + C δ d log d d 1 / 4 . |D_{\tau_A}|\le C\frac{(\log d)^2}{\sqrt{d}}+C\lambda_D+C\delta d\frac{\sqrt{\log d}}{d^{1/4}}. ∣ D τ A ∣ ≤ C d ( log d ) 2 + C λ D + C δ d d 1/4 log d .
Finally, Lemma D.13 gives
Z τ A ≥ c A d − 1 / 4 . Z_{\tau_A}\ge c_A d^{-1/4}. Z τ A ≥ c A d − 1/4 .
Hence
∣ D τ A ∣ Z τ A ≤ C ( log d ) 2 d 1 / 4 + C λ D d − 1 / 4 + C δ d log d . \frac{|D_{\tau_A}|}{Z_{\tau_A}}\le C\frac{(\log d)^2}{d^{1/4}}+C\frac{\lambda_D}{d^{-1/4}}+C\delta d\sqrt{\log d}. Z τ A ∣ D τ A ∣ ≤ C d 1/4 ( log d ) 2 + C d − 1/4 λ D + C δ d log d .
Since λ D = O ( d − 1 / 2 ) \lambda_D=O(d^{-1/2}) λ D = O ( d − 1/2 ) and
δ d ≤ c 0 ε log d , \delta d\le\frac{c_0\varepsilon}{\log d}, δ d ≤ log d c 0 ε ,
the right-hand side tends to zero. Therefore
∣ D τ A ∣ = o ( Z τ A ) . |D_{\tau_A}|=o(Z_{\tau_A}). ∣ D τ A ∣ = o ( Z τ A ) .
In particular, for all sufficiently large d d d ,
∣ D τ A ∣ ≤ 1 8 Z τ A . |D_{\tau_A}|\le\frac{1}{8}Z_{\tau_A}. ∣ D τ A ∣ ≤ 8 1 Z τ A .
Before Phase II, we provide another lemma below that controls the difference of drift
Lemma D.15 (Balanced D D D -drift on the Phase-II band). Assume Lemma C.2. Let D : = ρ − χ D := \rho- \chi D := ρ − χ , Z : = ρ + χ Z := \rho+ \chi Z := ρ + χ , and define
C ~ : = 2 α 1 2 a 1 2 > 0. \widetilde{C} := 2\alpha_1^2a_1^2 > 0. C := 2 α 1 2 a 1 2 > 0. There exist constants c b d ∈ ( 0 , 1 / 8 ) c_{\mathrm{bd}} \in(0,1/8) c bd ∈ ( 0 , 1/8 ) and C E > 0 C_E > 0 C E > 0 , depending only on the model, such that the following holds. Suppose
0 < Z ≤ 2 c b d , ∣ D ∣ ≤ 1 2 Z . (164) 0 < Z \le2c_{\mathrm{bd}}, \qquad|D| \le\frac{1}{2}Z. \tag*{(164)} 0 < Z ≤ 2 c bd , ∣ D ∣ ≤ 2 1 Z . (164) Then ρ , χ ≥ 0 \rho,\chi\ge0 ρ , χ ≥ 0 , ∣ ρ ∣ , ∣ χ ∣ ≤ 3 c b d / 2 |\rho|,|\chi| \le3c_{\mathrm{bd}}/2 ∣ ρ ∣ , ∣ χ ∣ ≤ 3 c bd /2 , and the population drifts satisfy
ρ ˙ − χ ˙ = − C ~ ( 1 + ρ χ ) D + E , (165) \dot{\rho}-\dot{\chi}=-\widetilde{C}(1+\rho\chi)D+E, \tag*{(165)} ρ ˙ − χ ˙ = − C ( 1 + ρ χ ) D + E , (165) where the error obeys the sharper bound
∣ E ∣ ≤ C E ( Z 2 + η ) . (166) |E| \le C_E(Z^2+\eta). \tag*{(166)} ∣ E ∣ ≤ C E ( Z 2 + η ) . (166) In particular, since Z ≤ 2 c b d Z \le2c_{\mathrm{bd}} Z ≤ 2 c bd , one also has the weaker bound
∣ E ∣ ≤ C E ( c b d 2 + η ) . (167) |E| \le C_E(c_{\mathrm{bd}}^2+\eta). \tag*{(167)} ∣ E ∣ ≤ C E ( c bd 2 + η ) . (167)
Proof. From Z = ρ + χ Z=\rho+\chi Z = ρ + χ and D = ρ − χ D=\rho-\chi D = ρ − χ ,
ρ = Z + D 2 , χ = Z − D 2 . \rho=\frac{Z+D}{2}, \qquad\chi=\frac{Z-D}{2}. ρ = 2 Z + D , χ = 2 Z − D .
The balance condition ∣ D ∣ ≤ Z / 2 |D| \le Z/2 ∣ D ∣ ≤ Z /2 gives
1 4 Z ≤ ρ , χ ≤ 3 4 Z . \frac{1}{4}Z \le\rho,\chi\le\frac{3}{4}Z. 4 1 Z ≤ ρ , χ ≤ 4 3 Z .
Thus ρ , χ ≥ 0 \rho,\chi\ge0 ρ , χ ≥ 0 , and since Z ≤ 2 c b d Z \le2c_{\mathrm{bd}} Z ≤ 2 c bd , we also have
∣ ρ ∣ , ∣ χ ∣ ≤ 3 4 Z ≤ 3 2 c b d < 1 2 . (168) |\rho|,|\chi| \le\frac{3}{4}Z \le\frac{3}{2}c_{\mathrm{bd}} < \frac{1}{2}. \tag*{(168)} ∣ ρ ∣ , ∣ χ ∣ ≤ 4 3 Z ≤ 2 3 c bd < 2 1 . (168)
Let L G L_G L G denote the Gaussian-equivalent population loss. As in the proof of Lemma C.3,
L G ( θ , β ) = Φ ( χ , ρ ) , Φ ( χ , ρ ) = 2 ∑ k ≥ 1 α k 2 [ 1 − ( κ ( ρ ) χ ) k ] . L_G(\theta,\beta)=\Phi(\chi,\rho), \qquad\Phi(\chi,\rho)=2\sum_{k\ge1}\alpha_k^2\left[1-(\kappa(\rho)\chi)^k\right]. L G ( θ , β ) = Φ ( χ , ρ ) , Φ ( χ , ρ ) = 2 k ≥ 1 ∑ α k 2 [ 1 − ( κ ( ρ ) χ ) k ] .
where κ ( ρ ) = ∑ ℓ ≥ 1 a ℓ 2 ρ ℓ \kappa(\rho)=\sum_{\ell\ge1}a_\ell^2\rho^\ell κ ( ρ ) = ∑ ℓ ≥ 1 a ℓ 2 ρ ℓ . Set
q : = κ ( ρ ) χ , A ( q ) : = ∑ k ≥ 1 k α k 2 q k − 1 . q:=\kappa(\rho)\chi, \qquad A(q):=\sum_{k\ge1}k\alpha_k^2q^{k-1}. q := κ ( ρ ) χ , A ( q ) := k ≥ 1 ∑ k α k 2 q k − 1 .
Then A ( 0 ) = α 1 2 A(0)=\alpha_1^2 A ( 0 ) = α 1 2 . By the Gaussian β \beta β -gradient identity (30), equivalently by differentiating Φ \Phi Φ in ρ \rho ρ , and by the corresponding χ \chi χ -derivative displayed in the proof of Lemma C.3, the Gaussian-model overlap drifts are
ρ ˙ G = 2 χ κ ′ ( ρ ) A ( q ) ( 1 − ρ 2 ) , χ ˙ G = 2 κ ( ρ ) A ( q ) ( 1 − χ 2 ) . (169) \dot{\rho}_G=2\chi\kappa'(\rho)A(q)(1-\rho^2), \qquad\dot{\chi}_G=2\kappa(\rho)A(q)(1-\chi^2). \tag*{(169)} ρ ˙ G = 2 χ κ ′ ( ρ ) A ( q ) ( 1 − ρ 2 ) , χ ˙ G = 2 κ ( ρ ) A ( q ) ( 1 − χ 2 ) . (169)
We now expand these identities on the small band (156). Since ∣ ρ ∣ ≤ 3 c b d / 2 < 1 / 2 |\rho| \le3c_{\mathrm{bd}}/2 < 1/2 ∣ ρ ∣ ≤ 3 c bd /2 < 1/2 , the Hermite covariance function satisfies
κ ( ρ ) = a 1 2 ρ + O ( ρ 2 ) , κ ′ ( ρ ) = a 1 2 + O ( ∣ ρ ∣ ) . (170) \kappa(\rho)=a_1^2\rho+O(\rho^2), \qquad\kappa'(\rho)=a_1^2+O(|\rho|). \tag*{(170)} κ ( ρ ) = a 1 2 ρ + O ( ρ 2 ) , κ ′ ( ρ ) = a 1 2 + O ( ∣ ρ ∣ ) . (170)
Moreover, by (160), q = κ ( ρ ) χ = O ( Z 2 ) q=\kappa(\rho)\chi=O(Z^2) q = κ ( ρ ) χ = O ( Z 2 ) , so
A ( q ) = α 1 2 + O ( Z 2 ) . (171) A(q)=\alpha_1^2+O(Z^2). \tag*{(171)} A ( q ) = α 1 2 + O ( Z 2 ) . (171)
Using (161)–(163),
ρ ˙ G = 2 α 1 2 a 1 2 χ ( 1 − ρ 2 ) + O ( Z 2 ) , \dot{\rho}_G=2\alpha_1^2a_1^2\chi(1-\rho^2)+O(Z^2), ρ ˙ G = 2 α 1 2 a 1 2 χ ( 1 − ρ 2 ) + O ( Z 2 ) ,
because the error from κ ′ ( ρ ) − a 1 2 \kappa'(\rho)-a_1^2 κ ′ ( ρ ) − a 1 2 is O ( ∣ χ ∣ ∣ ρ ∣ ) = O ( Z 2 ) O(|\chi||\rho|)=O(Z^2) O ( ∣ χ ∣∣ ρ ∣ ) = O ( Z 2 ) , and the error from A ( q ) − α 1 2 A(q)-\alpha_1^2 A ( q ) − α 1 2 is O ( ∣ χ ∣ Z 2 ) = O ( Z 3 ) O(|\chi|Z^2)=O(Z^3) O ( ∣ χ ∣ Z 2 ) = O ( Z 3 ) . Similarly,
χ ˙ G = 2 α 1 2 a 1 2 ρ ( 1 − χ 2 ) + O ( Z 2 ) . \dot{\chi}_G=2\alpha_1^2a_1^2\rho(1-\chi^2)+O(Z^2). χ ˙ G = 2 α 1 2 a 1 2 ρ ( 1 − χ 2 ) + O ( Z 2 ) .
because κ ( ρ ) − a 1 2 ρ = O ( ρ 2 ) = O ( Z 2 ) \kappa(\rho)-a_1^2\rho=O(\rho^2)=O(Z^2) κ ( ρ ) − a 1 2 ρ = O ( ρ 2 ) = O ( Z 2 ) .
Thus, with C ~ : = 2 α 1 2 a 1 2 \widetilde{C}:=2\alpha_1^2a_1^2 C := 2 α 1 2 a 1 2 ,
ρ ˙ G = C ~ χ ( 1 − ρ 2 ) + O ( Z 2 ) , χ ˙ G = C ~ ρ ( 1 − χ 2 ) + O ( Z 2 ) . (172) \dot{\rho}_G=\widetilde{C}\chi(1-\rho^2)+O(Z^2),\qquad\dot{\chi}_G=\widetilde{C}\rho(1-\chi^2)+O(Z^2). \tag*{(172)} ρ ˙ G = C χ ( 1 − ρ 2 ) + O ( Z 2 ) , χ ˙ G = C ρ ( 1 − χ 2 ) + O ( Z 2 ) . (172)
It remains to transfer the Gaussian drift estimates to the true population loss. By (25) and (26) from Lemma C.2,
∣ ρ ˙ − ρ ˙ G ∣ ≤ C η , ∣ χ ˙ − χ ˙ G ∣ ≤ C η . \lvert\dot{\rho}-\dot{\rho}_G\rvert\le C\eta,\qquad\lvert\dot{\chi}-\dot{\chi}_G\rvert\le C\eta. ∣ ρ ˙ − ρ ˙ G ∣ ≤ C η , ∣ χ ˙ − χ ˙ G ∣ ≤ C η .
Combining this with (172) , we get
ρ ˙ − χ ˙ = C ~ [ χ ( 1 − ρ 2 ) − ρ ( 1 − χ 2 ) ] + O ( Z 2 + η ) . \dot{\rho}-\dot{\chi}=\widetilde{C}\left[\chi(1-\rho^2)-\rho(1-\chi^2)\right]+O(Z^2+\eta). ρ ˙ − χ ˙ = C [ χ ( 1 − ρ 2 ) − ρ ( 1 − χ 2 ) ] + O ( Z 2 + η ) .
Finally,
χ ( 1 − ρ 2 ) − ρ ( 1 − χ 2 ) = ( χ − ρ ) ( 1 + ρ χ ) = − D ( 1 + ρ χ ) . (173) \chi(1-\rho^2)-\rho(1-\chi^2)=(\chi-\rho)(1+\rho\chi)=-D(1+\rho\chi). \tag*{(173)} χ ( 1 − ρ 2 ) − ρ ( 1 − χ 2 ) = ( χ − ρ ) ( 1 + ρ χ ) = − D ( 1 + ρ χ ) . (173)
Therefore
ρ ˙ − χ ˙ = − C ~ ( 1 + ρ χ ) D + E , ∣ E ∣ ≤ C E ( Z 2 + η ) , \dot{\rho}-\dot{\chi}=-\widetilde{C}(1+\rho\chi)D+E,\qquad\lvert E\rvert\le C_E(Z^2+\eta), ρ ˙ − χ ˙ = − C ( 1 + ρ χ ) D + E , ∣ E ∣ ≤ C E ( Z 2 + η ) ,
which proves (165) and (166) . Since Z ≤ 2 c b d Z\le2c_{\mathrm{bd}} Z ≤ 2 c bd , (167) follows immediately.
Phase II Dynamics, Weak Recovery
Lemma D.16 (Phase II: signal-dominated amplification and balance). Assume Lemmas D.13, D.14, and D.15. Work on G T w e a k G_T^{\mathrm{weak}} G T weak . There exist constants
c Z ∈ ( 0 , c b d ∧ 1 / 8 ) , C 2 , C η , I I , C I I > 0 , c_Z\in(0,c_{\mathrm{bd}}\wedge1/8),\qquad C_2,C_{\eta,\mathrm{II}},C_{\mathrm{II}}>0, c Z ∈ ( 0 , c bd ∧ 1/8 ) , C 2 , C η , II , C II > 0 , depending only on the model constants, such that the following holds for all sufficiently large d d d . If
C I I log d ≤ T δ ≤ C T log d , C_{\mathrm{II}}\log d\le T\delta\le C_T\log d, C II log d ≤ T δ ≤ C T log d , and
t ∗ : = inf { t ≥ τ A : Z t ≥ c Z } , t^*:=\inf\{t\ge\tau_A:Z_t\ge c_Z\}, t ∗ := inf { t ≥ τ A : Z t ≥ c Z } , then
t ∗ ≤ τ A + ⌈ C 2 δ − 1 log d ⌉ , Z t ∗ ∈ [ c Z , 2 c Z ] . t^*\le\tau_A+\lceil C_2\delta^{-1}\log d\rceil,\qquad Z_{t^*}\in[c_Z,2c_Z]. t ∗ ≤ τ A + ⌈ C 2 δ − 1 log d ⌉ , Z t ∗ ∈ [ c Z , 2 c Z ] . Moreover, for every τ A ≤ t ≤ t ∗ \tau_A\le t\le t^* τ A ≤ t ≤ t ∗ ,
η t ≤ C η , I I log d d , ∣ D t ∣ ≤ 1 2 Z t . \eta_t\le C_{\eta,\mathrm{II}}\frac{\log d}{\sqrt d},\qquad\lvert D_t\rvert\le\frac{1}{2}Z_t. η t ≤ C η , II d log d , ∣ D t ∣ ≤ 2 1 Z t . Consequently,
ρ t , χ t ≥ 1 4 Z t ( τ A ≤ t ≤ t ∗ ) , (174) \rho_t,\chi_t\ge\frac{1}{4}Z_t\qquad(\tau_A\le t\le t^*), \tag*{(174)} ρ t , χ t ≥ 4 1 Z t ( τ A ≤ t ≤ t ∗ ) , (174) with
t ∗ < τ B , min { ρ t ∗ , χ t ∗ } ≥ c Z 4 . (175) t^*<\tau_B,\qquad\min\{\rho_{t^*},\chi_{t^*}\}\ge\frac{c_Z}{4}. \tag*{(175)} t ∗ < τ B , min { ρ t ∗ , χ t ∗ } ≥ 4 c Z . (175)
Proof. Work on G T w e a k G_T^{\mathrm{weak}} G T weak . By Lemma D.13 and Lemma D.14,
τ A < τ B , τ A δ ≤ C log d , Z τ A ≥ c A d − 1 / 4 , (176) \tau_A<\tau_B,\qquad\tau_A\delta\le C\log d,\qquad Z_{\tau_A}\ge c_A d^{-1/4}, \tag*{(176)} τ A < τ B , τ A δ ≤ C log d , Z τ A ≥ c A d − 1/4 , (176)
and
η τ A ≤ C log d d , Z τ A = o ( 1 ) , ∣ D τ A ∣ ≤ 1 8 Z τ A . (177) \eta_{\tau_A}\le C\frac{\log d}{\sqrt d},\qquad Z_{\tau_A}=o(1),\qquad\lvert D_{\tau_A}\rvert\le\frac{1}{8}Z_{\tau_A}. \tag*{(177)} η τ A ≤ C d log d , Z τ A = o ( 1 ) , ∣ D τ A ∣ ≤ 8 1 Z τ A . (177)
Also, on G T w e a k G_T^{\mathrm{weak}} G T weak ,
λ Z , λ D = O ( d − 1 / 2 ) = o ( Z τ A ) . \lambda_Z,\lambda_D=O(d^{-1/2})=o(Z_{\tau_A}). λ Z , λ D = O ( d − 1/2 ) = o ( Z τ A ) .
Choose
H : = C η , I I log d d , T 2 : = τ A + ⌈ C 2 δ − 1 log d ⌉ . H:=C_{\eta,\mathrm{II}}\frac{\log d}{\sqrt d},\qquad T_2:=\tau_A+\lceil C_2\delta^{-1}\log d\rceil. H := C η , II d log d , T 2 := τ A + ⌈ C 2 δ − 1 log d ⌉ .
The constants C η , I I C_{\eta,\mathrm{II}} C η , II , c Z c_Z c Z , and C 2 C_2 C 2 will be fixed below, and C I I C_{\mathrm{II}} C II is chosen large enough so that T 2 ≤ T T_2 \le T T 2 ≤ T , using τ A δ ≤ C log d \tau_A\delta\le C\log d τ A δ ≤ C log d .
Define the bootstrap stopping time
σ : = inf { t ≥ τ A : η t > H or Z t > 2 c Z or ∣ D t ∣ > 1 2 Z t or t = T 2 } . \sigma:= \inf\left\{t \ge\tau_A : \eta_t > H \text{ or } Z_t > 2c_Z \text{ or } \lvert D_t\rvert> \frac{1}{2}Z_t \text{ or } t = T_2\right\}. σ := inf { t ≥ τ A : η t > H or Z t > 2 c Z or ∣ D t ∣ > 2 1 Z t or t = T 2 } .
At t = τ A t=\tau_A t = τ A , none of the first three clauses holds for large d d d . For t < σ t<\sigma t < σ ,
0 < Z t ≤ 2 c Z , ∣ D t ∣ ≤ 1 2 Z t . 0 < Z_t \le2c_Z,\qquad\lvert D_t\rvert\le\frac{1}{2}Z_t. 0 < Z t ≤ 2 c Z , ∣ D t ∣ ≤ 2 1 Z t .
Hence
ρ t = Z t + D t 2 , χ t = Z t − D t 2 , 1 4 Z t ≤ ρ t , χ t ≤ 3 4 Z t . (178) \rho_t = \frac{Z_t+D_t}{2},\qquad\chi_t = \frac{Z_t-D_t}{2},\qquad\frac{1}{4}Z_t \le\rho_t,\chi_t \le\frac{3}{4}Z_t. \tag*{(178)} ρ t = 2 Z t + D t , χ t = 2 Z t − D t , 4 1 Z t ≤ ρ t , χ t ≤ 4 3 Z t . (178)
Taking c Z < 1 / 3 c_Z<1/3 c Z < 1/3 , this implies ρ t , χ t < 1 / 2 \rho_t,\chi_t<1/2 ρ t , χ t < 1/2 , so all one-step expansions below are valid before σ \sigma σ .
Step 1: Z Z Z -growth and no-drop. Let
ε t ( Z ) : = N t + 1 ( Z ) − N t ( Z ) . \varepsilon_t^{(Z)} := N_{t+1}^{(Z)}-N_t^{(Z)}. ε t ( Z ) := N t + 1 ( Z ) − N t ( Z ) .
For t < σ t<\sigma t < σ , the one-step overlap expansion and the aggregate drift bound give
Z t + 1 ≥ Z t + δ ( C 9 Z t − C 11 η t ) − C δ 2 d Z t + ε t ( Z ) . Z_{t+1} \ge Z_t+\delta(C_9Z_t-C_{11}\eta_t)-C\delta^2d\,Z_t+\varepsilon_t^{(Z)}. Z t + 1 ≥ Z t + δ ( C 9 Z t − C 11 η t ) − C δ 2 d Z t + ε t ( Z ) .
Since δ d = o ( 1 ) \delta d=o(1) δ d = o ( 1 ) , for large d d d ,
C 9 − C δ d ≥ 3 4 C 9 . C_9-C\delta d\ge\frac{3}{4}C_9. C 9 − C δ d ≥ 4 3 C 9 .
Set
τ − : = inf { t ≥ τ A : Z t < 1 2 Z τ A } . \tau_- := \inf\left\{t\ge\tau_A: Z_t<\frac{1}{2}Z_{\tau_A}\right\}. τ − := inf { t ≥ τ A : Z t < 2 1 Z τ A } .
For t < σ ∧ τ − t<\sigma\wedge\tau_- t < σ ∧ τ − , we have Z t ≥ Z τ A / 2 Z_t\ge Z_{\tau_A}/2 Z t ≥ Z τ A /2 and η t ≤ H \eta_t\le H η t ≤ H . Therefore
η t Z t ≤ 2 H Z τ A ≤ C log d d 1 / 4 = o ( 1 ) . \frac{\eta_t}{Z_t}\le\frac{2H}{Z_{\tau_A}}\le C\frac{\log d}{d^{1/4}}=o(1). Z t η t ≤ Z τ A 2 H ≤ C d 1/4 log d = o ( 1 ) .
Thus, for large d d d ,
C 11 η t ≤ 1 4 C 9 Z t , C_{11}\eta_t\le\frac{1}{4}C_9Z_t, C 11 η t ≤ 4 1 C 9 Z t ,
and hence
Z t + 1 ≥ ( 1 + β δ ) Z t + ε t ( Z ) , t < σ ∧ τ − . (179) Z_{t+1}\ge(1+\beta\delta)Z_t+\varepsilon_t^{(Z)},\qquad t<\sigma\wedge\tau_-. \tag*{(179)} Z t + 1 ≥ ( 1 + β δ ) Z t + ε t ( Z ) , t < σ ∧ τ − . (179)
where β : = C 9 / 2 \beta:=C_9/2 β := C 9 /2 . Let γ : = 1 + β δ \gamma:=1+\beta\delta γ := 1 + β δ .
Unrolling (171), for every s ≥ 0 s\ge0 s ≥ 0 with τ A + s < σ ∧ τ − \tau_A+s<\sigma\wedge\tau_- τ A + s < σ ∧ τ − ,
Z τ A + s ≥ γ s ( Z τ A + ∑ k = 0 s − 1 γ − ( k + 1 ) ε τ A + k ( Z ) ) . Z_{\tau_A+s}\ge\gamma^s\left(Z_{\tau_A}+\sum_{k=0}^{s-1}\gamma^{-(k+1)}\varepsilon_{\tau_A+k}^{(Z)}\right). Z τ A + s ≥ γ s ( Z τ A + k = 0 ∑ s − 1 γ − ( k + 1 ) ε τ A + k ( Z ) ) .
On E T ( Z ) E_T^{(Z)} E T ( Z ) , (102) gives
sup u ≤ T ∣ N u ( Z ) ∣ ≤ 2 λ Z . \sup_{u\le T}\lvert N_u^{(Z)}\rvert\le2\lambda_Z. u ≤ T sup ∣ N u ( Z ) ∣ ≤ 2 λ Z .
By Corollary E.9 applied on the shifted interval starting at τ A \tau_A τ A ,
∣ ∑ k = 0 s − 1 γ − ( k + 1 ) ε τ A + k ( Z ) ∣ ≤ 4 λ Z . \left\lvert\sum_{k=0}^{s-1}\gamma^{-(k+1)}\varepsilon_{\tau_A+k}^{(Z)}\right\rvert\le4\lambda_Z. k = 0 ∑ s − 1 γ − ( k + 1 ) ε τ A + k ( Z ) ≤ 4 λ Z .
Therefore
Z τ A + s ≥ ( Z τ A − 4 λ Z ) γ s . (180) Z_{\tau_A+s}\ge\left(Z_{\tau_A}-4\lambda_Z\right)\gamma^s. \tag*{(180)} Z τ A + s ≥ ( Z τ A − 4 λ Z ) γ s . (180)
Since λ Z = o ( Z τ A ) \lambda_Z=o(Z_{\tau_A}) λ Z = o ( Z τ A ) , the right side is at least ( 3 / 4 ) Z τ A γ s (3/4)Z_{\tau_A}\gamma^s ( 3/4 ) Z τ A γ s . This rules out τ − ≤ σ \tau_-\le\sigma τ − ≤ σ . Hence
Z t ≥ 1 2 Z τ A ( τ A ≤ t < σ ) . Z_t\ge\frac{1}{2}Z_{\tau_A}\qquad(\tau_A\le t<\sigma). Z t ≥ 2 1 Z τ A ( τ A ≤ t < σ ) .
and (180) holds for every τ A + s < σ \tau_A+s<\sigma τ A + s < σ .
We shall use the following summability estimates. Summing (179) from τ A \tau_A τ A to t − 1 t-1 t − 1 , for t < σ t<\sigma t < σ , gives
β δ ∑ k = τ A t − 1 Z k ≤ Z t − Z τ A − ( N t ( Z ) − N τ A ( Z ) ) . \beta\delta\sum_{k=\tau_A}^{t-1} Z_k \le Z_t-Z_{\tau_A}-\left(N_t^{(Z)}-N_{\tau_A}^{(Z)}\right). β δ k = τ A ∑ t − 1 Z k ≤ Z t − Z τ A − ( N t ( Z ) − N τ A ( Z ) ) .
Since Z t ≤ 2 c Z Z_t\le2c_Z Z t ≤ 2 c Z and ∣ N t ( Z ) − N τ A ( Z ) ∣ ≤ 4 λ Z \left|N_t^{(Z)}-N_{\tau_A}^{(Z)}\right|\le4\lambda Z N t ( Z ) − N τ A ( Z ) ≤ 4 λ Z ,
δ ∑ k = τ A t − 1 Z k ≤ C c Z . (181) \delta\sum_{k=\tau_A}^{t-1} Z_k\le Cc_Z. \tag*{(181)} δ k = τ A ∑ t − 1 Z k ≤ C c Z . (181)
Also, since Z k ≤ 2 c Z Z_k\le2c_Z Z k ≤ 2 c Z ,
δ ∑ k = τ A t − 1 Z k 2 ≤ C c Z 2 . (182) \delta\sum_{k=\tau_A}^{t-1} Z_k^2\le Cc_Z^2. \tag*{(182)} δ k = τ A ∑ t − 1 Z k 2 ≤ C c Z 2 . (182)
Finally, the backward form of (179) gives, for τ A ≤ k < t < σ \tau_A\le k<t<\sigma τ A ≤ k < t < σ ,
Z k ≤ γ − ( t − k ) Z t + C λ Z . (183) Z_k\le\gamma^{-(t-k)}Z_t+C\lambda Z. \tag*{(183)} Z k ≤ γ − ( t − k ) Z t + C λ Z . (183)
Indeed, unroll (179) from k k k to t t t , divide by γ t − k \gamma^{t-k} γ t − k , and apply Corollary E.9 to the shifted weighted noise sum.
Step 2: Closing the η \eta η -bootstrap. Since E T ( η ) E_T^{(\eta)} E T ( η ) controls the cumulative coordinate martingales from time 0, for every t ≥ τ A t\ge\tau_A t ≥ τ A and every i ≤ d i\le d i ≤ d ,
∣ δ ∑ k = τ A t − 1 ξ k , i ( θ ) ∣ ≤ ∣ δ ∑ k = τ A t ξ k , i ( θ ) ∣ + ∣ δ ∑ k < τ A ξ k , i ( θ ) ∣ ≤ 2 Λ η . \left|\delta\sum_{k=\tau_A}^{t-1}\xi_{k,i}^{(\theta)}\right|
\le\left|\delta\sum_{k=\tau_A}^{t}\xi_{k,i}^{(\theta)}\right|
+\left|\delta\sum_{k<\tau_A}\xi_{k,i}^{(\theta)}\right|
\le2\Lambda\eta. δ k = τ A ∑ t − 1 ξ k , i ( θ ) ≤ δ k = τ A ∑ t ξ k , i ( θ ) + δ k < τ A ∑ ξ k , i ( θ ) ≤ 2Λ η .
Given that the quadratic and cubic remainder sums over [ τ A , t ] [\tau_A,t] [ τ A , t ] are bounded by their global good-event bounds, therefore, applying Lemma D.9 on the interval [ τ A , t ] [\tau_A,t] [ τ A , t ] , for every τ A ≤ t ≤ σ \tau_A\le t\le\sigma τ A ≤ t ≤ σ ,
η t ≤ C ( η τ A + δ ∑ k = τ A t − 1 ∥ g ˉ k ( θ ) ∥ ∞ + 2 Λ η + λ c u b ) . \eta_t\le C\left(\eta_{\tau_A}+\delta\sum_{k=\tau_A}^{t-1}\|\bar g_k^{(\theta)}\|_\infty+2\Lambda\eta+\lambda_{\mathrm{cub}}\right). η t ≤ C ( η τ A + δ k = τ A ∑ t − 1 ∥ g ˉ k ( θ ) ∥ ∞ + 2Λ η + λ cub ) .
After increasing C C C , we write this as
η t ≤ C ( η τ A + δ ∑ k = τ A t − 1 ∥ g ˉ k ( θ ) ∥ ∞ + Λ η + λ c u b ) . \eta_t\le C\left(\eta_{\tau_A}+\delta\sum_{k=\tau_A}^{t-1}\|\bar g_k^{(\theta)}\|_\infty+\Lambda\eta+\lambda_{\mathrm{cub}}\right). η t ≤ C ( η τ A + δ k = τ A ∑ t − 1 ∥ g ˉ k ( θ ) ∥ ∞ + Λ η + λ cub ) .
On the Phase-II band, the mean-field ℓ ∞ \ell_\infty ℓ ∞ -gradient bound gives
∥ g ˉ k ( θ ) ∥ ∞ ≤ C Z k ∥ θ ∗ ∥ ∞ + C Z k 2 η k + C η k 2 . \|\bar g_k^{(\theta)}\|_\infty\le CZ_k\|\theta^*\|_\infty+CZ_k^2\eta_k+C\eta_k^2. ∥ g ˉ k ( θ ) ∥ ∞ ≤ C Z k ∥ θ ∗ ∥ ∞ + C Z k 2 η k + C η k 2 .
Using ∥ θ ∗ ∥ ∞ ≤ C log d / d \|\theta^*\|_\infty\le C\sqrt{\log d/d} ∥ θ ∗ ∥ ∞ ≤ C log d / d and (181) ,
δ ∑ k = τ A t − 1 Z k ∥ θ ∗ ∥ ∞ ≤ C c Z log d d = o ( H ) . \delta\sum_{k=\tau_A}^{t-1}Z_k\|\theta^*\|_\infty
\le Cc_Z\sqrt{\frac{\log d}{d}}=o(H). δ k = τ A ∑ t − 1 Z k ∥ θ ∗ ∥ ∞ ≤ C c Z d log d = o ( H ) .
By (182) and the bootstrap bound η k ≤ H \eta_k\le H η k ≤ H ,
δ ∑ k = τ A t − 1 Z k 2 η k ≤ H δ ∑ k = τ A t − 1 Z k 2 ≤ C c Z 2 H . \delta\sum_{k=\tau_A}^{t-1}Z_k^2\eta_k
\le H\delta\sum_{k=\tau_A}^{t-1}Z_k^2
\le Cc_Z^2H. δ k = τ A ∑ t − 1 Z k 2 η k ≤ H δ k = τ A ∑ t − 1 Z k 2 ≤ C c Z 2 H .
Also,
δ ∑ k = τ A t − 1 η k 2 ≤ H 2 ( t − τ A ) δ ≤ C ( log d ) 3 d = o ( H ) . \delta\sum_{k=\tau_A}^{t-1}\eta_k^2
\le H^2(t-\tau_A)\delta
\le C\frac{(\log d)^3}{d}=o(H). δ k = τ A ∑ t − 1 η k 2 ≤ H 2 ( t − τ A ) δ ≤ C d ( log d ) 3 = o ( H ) .
Together with
η τ A + λ η ( t ) + λ c u b ≤ C log d d , \eta_{\tau_A}+\lambda_\eta(t)+\lambda_{\mathrm{cub}}\le C\frac{\log d}{\sqrt{d}}, η τ A + λ η ( t ) + λ cub ≤ C d log d ,
we obtain
η t ≤ C log d d + C c Z 2 H + o ( H ) . \eta_t\le C\frac{\log d}{\sqrt{d}}+Cc_Z^2H+o(H). η t ≤ C d log d + C c Z 2 H + o ( H ) .
Choose c Z c_Z c Z small enough that C c Z 2 ≤ 1 / 8 Cc_Z^2\le1/8 C c Z 2 ≤ 1/8 , then choose C η , I I C_{\eta,\mathrm{II}} C η , II large enough that C ( log d ) / d ≤ H / 4 C(\log d)/\sqrt{d}\le H/4 C ( log d ) / d ≤ H /4 . For large d d d ,
η t ≤ 1 2 H ( τ A ≤ t ≤ σ ) . \eta_t\le\frac{1}{2}H\qquad(\tau_A\le t\le\sigma). η t ≤ 2 1 H ( τ A ≤ t ≤ σ ) .
Thus the stopping clause η t > H \eta_t>H η t > H cannot be the first active clause before t ∗ t^* t ∗ .
Step 3: Closing the D / Z D/Z D / Z -bootstrap. For t < σ t<\sigma t < σ , choose c Z ≤ c b d c_Z\le c_{\mathrm{bd}} c Z ≤ c bd . Then (170) allows us to apply Lemma D.15:
D ˙ t = − C ~ ( 1 + ρ t χ t ) D t + E t , ∣ E t ∣ ≤ C ( Z t 2 + η t ) . \dot{D}_t=-\widetilde{C}(1+\rho_t\chi_t)D_t+E_t,\qquad|E_t|\le C(Z_t^2+\eta_t). D ˙ t = − C ( 1 + ρ t χ t ) D t + E t , ∣ E t ∣ ≤ C ( Z t 2 + η t ) .
The discrete one-step expansion for D t D_t D t has the form
D t + 1 = ( 1 − α t δ ) D t + δ e t + ε t ( D ) . (184) D_{t+1}=(1-\alpha_t\delta)D_t+\delta e_t+\varepsilon_t^{(D)}. \tag*{(184)} D t + 1 = ( 1 − α t δ ) D t + δ e t + ε t ( D ) . (184)
where
ε t ( D ) : = N t + 1 ( D ) − N t ( D ) , \varepsilon_t^{(D)}:=N_{t+1}^{(D)}-N_t^{(D)}, ε t ( D ) := N t + 1 ( D ) − N t ( D ) ,
α t = C ~ ( 1 + ρ t χ t ) + δ 4 ( E [ ∥ g t ( β ) ∥ 2 ∣ F t ] + E [ ∥ g t ( θ ) ∥ 2 ∣ F t ] ) , \alpha_t=\widetilde{C}(1+\rho_t\chi_t)+\frac{\delta}{4}\left(\mathbb{E}[\lVert g_t^{(\beta)}\rVert^2\mid\mathcal{F}_t]+\mathbb{E}[\lVert g_t^{(\theta)}\rVert^2\mid\mathcal{F}_t]\right), α t = C ( 1 + ρ t χ t ) + 4 δ ( E [∥ g t ( β ) ∥ 2 ∣ F t ] + E [∥ g t ( θ ) ∥ 2 ∣ F t ] ) ,
and
∣ e t ∣ ≤ C ( Z t 2 + η t ) + C δ d Z t . (185) |e_t|\le C(Z_t^2+\eta_t)+C\delta d Z_t. \tag*{(185)} ∣ e t ∣ ≤ C ( Z t 2 + η t ) + C δ d Z t . (185)
Here the last term comes from the difference of the two normalization corrections. Since c Z c_Z c Z is small, α t ≥ α 0 > 0 \alpha_t\ge\alpha_0>0 α t ≥ α 0 > 0 , and after decreasing the step-size constant,
0 ≤ 1 − α t δ ≤ 1 − α 0 δ = : γ D < 1. 0\le1-\alpha_t\delta\le1-\alpha_0\delta=:\gamma_D<1. 0 ≤ 1 − α t δ ≤ 1 − α 0 δ =: γ D < 1.
Unroll (176). For τ A ≤ t ≤ σ \tau_A\le t\le\sigma τ A ≤ t ≤ σ , define
P k + 1 , t : = ∏ ℓ = k + 1 t − 1 ( 1 − α ℓ δ ) , τ A ≤ k ≤ t − 1 , P_{k+1,t}:=\prod_{\ell=k+1}^{t-1}(1-\alpha_\ell\delta),\qquad\tau_A\le k\le t-1, P k + 1 , t := ℓ = k + 1 ∏ t − 1 ( 1 − α ℓ δ ) , τ A ≤ k ≤ t − 1 ,
with P t , t = 1 P_{t,t}=1 P t , t = 1 . Then
D t = P τ A , t D τ A + ∑ k = τ A t − 1 P k + 1 , t δ e k + ∑ k = τ A t − 1 P k + 1 , t ε k ( D ) . D_t=P_{\tau_A,t}D_{\tau_A}+\sum_{k=\tau_A}^{t-1}P_{k+1,t}\delta e_k+\sum_{k=\tau_A}^{t-1}P_{k+1,t}\varepsilon_k^{(D)}. D t = P τ A , t D τ A + k = τ A ∑ t − 1 P k + 1 , t δ e k + k = τ A ∑ t − 1 P k + 1 , t ε k ( D ) .
The entry term is controlled by Lemma D.14 and the no-drop estimate:
∣ P τ A , t D τ A ∣ ≤ ∣ D τ A ∣ ≤ 1 8 Z τ A ≤ 1 4 Z t . |P_{\tau_A,t}D_{\tau_A}|\le|D_{\tau_A}|\le\frac{1}{8}Z_{\tau_A}\le\frac{1}{4}Z_t. ∣ P τ A , t D τ A ∣ ≤ ∣ D τ A ∣ ≤ 8 1 Z τ A ≤ 4 1 Z t .
For the noise term, the weights P k + 1 , t P_{k+1,t} P k + 1 , t lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] and are nondecreasing in k k k . On E T ( ρ , χ ) E_T^{(\rho,\chi)} E T ( ρ , χ ) ,
sup u ≤ T ∣ N u ( D ) ∣ ≤ 2 λ D . \sup_{u\le T}|N_u^{(D)}|\le2\lambda_D. u ≤ T sup ∣ N u ( D ) ∣ ≤ 2 λ D .
Therefore Lemma E.8 gives
∣ ∑ k = τ A t − 1 P k + 1 , t ε k ( D ) ∣ ≤ 4 λ D = o ( Z τ A ) = o ( Z t ) . \left|\sum_{k=\tau_A}^{t-1}P_{k+1,t}\varepsilon_k^{(D)}\right|\le4\lambda_D=o(Z_{\tau_A})=o(Z_t). k = τ A ∑ t − 1 P k + 1 , t ε k ( D ) ≤ 4 λ D = o ( Z τ A ) = o ( Z t ) .
It remains to bound the forcing. Using (175),
Z k ≤ γ − ( t − k ) Z t + C λ Z . Z_k\le\gamma^{-(t-k)}Z_t+C\lambda_Z. Z k ≤ γ − ( t − k ) Z t + C λ Z .
Since P k + 1 , t ≤ γ D t − 1 − k P_{k+1,t} \leq\gamma_D^{t-1-k} P k + 1 , t ≤ γ D t − 1 − k , we have
δ ∑ k = τ A t − 1 P k + 1 , t Z k ≤ C δ ∑ j ≥ 0 γ D j γ − j Z t + C λ Z δ ∑ j ≥ 0 γ D j ≤ C Z t + C λ Z . \delta\sum_{k=\tau_A}^{t-1} P_{k+1,t}Z_k \leq C\delta\sum_{j\geq0}\gamma_D^j\gamma^{-j}Z_t + C\lambda_Z\delta\sum_{j\geq0}\gamma_D^j \leq CZ_t+C\lambda_Z. δ k = τ A ∑ t − 1 P k + 1 , t Z k ≤ C δ j ≥ 0 ∑ γ D j γ − j Z t + C λ Z δ j ≥ 0 ∑ γ D j ≤ C Z t + C λ Z .
Since λ Z = o ( Z t ) \lambda_Z=o(Z_t) λ Z = o ( Z t ) , this is C Z t + o ( Z t ) CZ_t+o(Z_t) C Z t + o ( Z t ) . Similarly,
δ ∑ k = τ A t − 1 P k + 1 , t Z k 2 ≤ C Z t 2 + C λ Z 2 = C Z t 2 + o ( Z t ) . \delta\sum_{k=\tau_A}^{t-1}P_{k+1,t}Z_k^2\leq CZ_t^2+C\lambda_Z^2=CZ_t^2+o(Z_t). δ k = τ A ∑ t − 1 P k + 1 , t Z k 2 ≤ C Z t 2 + C λ Z 2 = C Z t 2 + o ( Z t ) .
Also, by the η \eta η -bootstrap,
δ ∑ k = τ A t − 1 P k + 1 , t η k ≤ C H = o ( Z τ A ) = o ( Z t ) . \delta\sum_{k=\tau_A}^{t-1}P_{k+1,t}\eta_k\leq CH=o(Z_{\tau_A})=o(Z_t). δ k = τ A ∑ t − 1 P k + 1 , t η k ≤ C H = o ( Z τ A ) = o ( Z t ) .
Using (177), we conclude
∑ k = τ A t − 1 P k + 1 , t δ ∣ e k ∣ ≤ C c Z Z t + o ( Z t ) + C δ d Z t . \sum_{k=\tau_A}^{t-1}P_{k+1,t}\delta|e_k|\leq Cc_ZZ_t+o(Z_t)+C\delta d Z_t. k = τ A ∑ t − 1 P k + 1 , t δ ∣ e k ∣ ≤ C c Z Z t + o ( Z t ) + C δ d Z t .
Since δ d = o ( 1 ) \delta d=o(1) δ d = o ( 1 ) ,
∑ k = τ A t − 1 P k + 1 , t δ ∣ e k ∣ ≤ C c Z Z t + o ( Z t ) . \sum_{k=\tau_A}^{t-1}P_{k+1,t}\delta|e_k|\leq Cc_ZZ_t+o(Z_t). k = τ A ∑ t − 1 P k + 1 , t δ ∣ e k ∣ ≤ C c Z Z t + o ( Z t ) .
Thus
∣ D t ∣ ≤ 1 4 Z t + C c Z Z t + o ( Z t ) . |D_t|\leq\frac{1}{4}Z_t+Cc_ZZ_t+o(Z_t). ∣ D t ∣ ≤ 4 1 Z t + C c Z Z t + o ( Z t ) .
Choose c Z c_Z c Z smaller, if necessary, so that C c Z ≤ 1 / 16 Cc_Z\leq1/16 C c Z ≤ 1/16 , and then take d d d large enough so that the o ( Z t ) o(Z_t) o ( Z t ) -term is at most Z t / 16 Z_t/16 Z t /16 . Hence
∣ D t ∣ ≤ 3 8 Z t < 1 2 Z t ( τ A ≤ t ≤ σ ) . |D_t|\leq\frac{3}{8}Z_t<\frac{1}{2}Z_t\qquad(\tau_A\leq t\leq\sigma). ∣ D t ∣ ≤ 8 3 Z t < 2 1 Z t ( τ A ≤ t ≤ σ ) .
Therefore the stopping clause ∣ D t ∣ > Z t / 2 |D_t|>Z_t/2 ∣ D t ∣ > Z t /2 cannot be the first active clause before t ∗ t^* t ∗ .
Step 4: Hitting c Z c_Z c Z before T 2 T_2 T 2 . From (172),
Z τ A + s ≥ ( Z τ A − 4 λ Z ) γ s Z_{\tau_A+s}\geq(Z_{\tau_A}-4\lambda_Z)\gamma^s Z τ A + s ≥ ( Z τ A − 4 λ Z ) γ s
as long as τ A + s < σ \tau_A+s<\sigma τ A + s < σ . Define
s ∗ : = ⌈ log ( c Z / ( Z τ A − 4 λ Z ) ) log γ ⌉ . s_*:=\left\lceil\frac{\log(c_Z/(Z_{\tau_A}-4\lambda_Z))}{\log\gamma}\right\rceil. s ∗ := ⌈ log γ log ( c Z / ( Z τ A − 4 λ Z )) ⌉ .
Since Z τ A − 4 λ Z ≥ c d − 1 / 4 Z_{\tau_A}-4\lambda_Z\geq cd^{-1/4} Z τ A − 4 λ Z ≥ c d − 1/4 and log γ ≥ c δ \log\gamma\geq c\delta log γ ≥ cδ , we have
s ∗ ≤ C δ − 1 log d . s_*\leq C\delta^{-1}\log d. s ∗ ≤ C δ − 1 log d .
Choose C 2 C_2 C 2 larger than this constant.
Suppose, for contradiction, that t ∗ > τ A + s ∗ t^*>\tau_A+s_* t ∗ > τ A + s ∗ . Then Z t < c Z Z_t<c_Z Z t < c Z up to τ A + s ∗ \tau_A+s_* τ A + s ∗ , so the Z t > 2 c Z Z_t>2c_Z Z t > 2 c Z clause cannot trigger before that time. By Steps 2 and 3, the η \eta η - and D / Z D/Z D / Z -clauses cannot trigger before t ∗ t^* t ∗ . By the choice of C 2 C_2 C 2 , the time clause t = T 2 t=T_2 t = T 2 also cannot trigger before τ A + s ∗ \tau_A+s_* τ A + s ∗ . Hence τ A + s ∗ < σ \tau_A+s_*<\sigma τ A + s ∗ < σ , and
Z τ A + s ∗ ≥ ( Z τ A − 4 λ Z ) γ s ∗ ≥ c Z , Z_{\tau_A+s_*}\geq(Z_{\tau_A}-4\lambda_Z)\gamma^{s_*}\geq c_Z, Z τ A + s ∗ ≥ ( Z τ A − 4 λ Z ) γ s ∗ ≥ c Z ,
contradicting the definition of t ∗ t^* t ∗ . Therefore
t ∗ ≤ τ A + s ∗ ≤ τ A + ⌈ C 2 δ − 1 log d ⌉ . t^*\leq\tau_A+s_*\leq\tau_A+\left\lceil C_2\delta^{-1}\log d\right\rceil. t ∗ ≤ τ A + s ∗ ≤ τ A + ⌈ C 2 δ − 1 log d ⌉ .
Step 5: Overshoot and conclusion. Since Z τ A = o ( 1 ) Z_{\tau_A}=o(1) Z τ A = o ( 1 ) , for large d d d , Z τ A < c Z Z_{\tau_A}<c_Z Z τ A < c Z , so t ∗ > τ A t^*>\tau_A t ∗ > τ A and Z t ∗ − 1 < c Z Z_{t^*-1}<c_Z Z t ∗ − 1 < c Z . The one-step expansion gives
Z t ∗ = Z t ∗ − 1 + δ F t ∗ − 1 ( Z ) + ε t ∗ − 1 ( Z ) . Z_{t^*}=Z_{t^*-1}+\delta F_{t^*-1}^{(Z)}+\varepsilon_{t^*-1}^{(Z)}. Z t ∗ = Z t ∗ − 1 + δ F t ∗ − 1 ( Z ) + ε t ∗ − 1 ( Z ) .
By Lemma E.7 ,
∣ F t ( Z ) ∣ ≤ C d + C δ d . \left|F_t^{(Z)}\right|\le C\sqrt{d}+C\delta d. F t ( Z ) ≤ C d + C δ d .
Hence
δ ∣ F t ( Z ) ∣ ≤ C δ d + C δ 2 d = o ( 1 ) . \delta\left|F_t^{(Z)}\right|\le C\delta\sqrt{d}+C\delta^2d=o(1). δ F t ( Z ) ≤ C δ d + C δ 2 d = o ( 1 ) .
Also, on E T ( Z ) E_T^{(Z)} E T ( Z ) ,
∣ ε t ( Z ) ∣ ≤ ∣ N t + 1 ( Z ) ∣ + ∣ N t ( Z ) ∣ ≤ 4 λ Z = o ( 1 ) . \left|\varepsilon_t^{(Z)}\right|\le\left|N_{t+1}^{(Z)}\right|+\left|N_t^{(Z)}\right|\le4\lambda_Z=o(1). ε t ( Z ) ≤ N t + 1 ( Z ) + N t ( Z ) ≤ 4 λ Z = o ( 1 ) .
Therefore
Z t ∗ ≤ c Z + o ( 1 ) ≤ 2 c Z Z_{t^*}\le c_Z+o(1)\le2c_Z Z t ∗ ≤ c Z + o ( 1 ) ≤ 2 c Z
for all large d d d . By definition, Z t ∗ ≥ c Z Z_{t^*}\ge c_Z Z t ∗ ≥ c Z , so
Z t ∗ ∈ [ c Z , 2 c Z ] . Z_{t^*}\in[c_Z,2c_Z]. Z t ∗ ∈ [ c Z , 2 c Z ] .
The bootstrap conclusions give, for τ A ≤ t ≤ t ∗ \tau_A\le t\le t^* τ A ≤ t ≤ t ∗ ,
η t ≤ H = C η , I I log d d , ∣ D t ∣ ≤ 1 2 Z t . \eta_t\le H=C_{\eta,\mathrm{II}}\frac{\log d}{\sqrt{d}},\qquad|D_t|\le\frac{1}{2}Z_t. η t ≤ H = C η , II d log d , ∣ D t ∣ ≤ 2 1 Z t .
Thus
ρ t = Z t + D t 2 ≥ 1 4 Z t , χ t = Z t − D t 2 ≥ 1 4 Z t . \rho_t=\frac{Z_t+D_t}{2}\ge\frac{1}{4}Z_t,\qquad\chi_t=\frac{Z_t-D_t}{2}\ge\frac{1}{4}Z_t. ρ t = 2 Z t + D t ≥ 4 1 Z t , χ t = 2 Z t − D t ≥ 4 1 Z t .
At t = t ∗ t=t^* t = t ∗ ,
min { ρ t ∗ , χ t ∗ } ≥ 1 4 Z t ∗ ≥ c Z 4 . \min\{\rho_{t^*},\chi_{t^*}\}\ge\frac{1}{4}Z_{t^*}\ge\frac{c_Z}{4}. min { ρ t ∗ , χ t ∗ } ≥ 4 1 Z t ∗ ≥ 4 c Z .
Moreover,
max { ρ t ∗ , χ t ∗ } ≤ Z t ∗ + ∣ D t ∗ ∣ 2 ≤ 3 4 Z t ∗ ≤ 3 2 c Z < 1 2 , \max\{\rho_{t^*},\chi_{t^*}\}\le\frac{Z_{t^*}+|D_{t^*}|}{2}\le\frac{3}{4}Z_{t^*}\le\frac{3}{2}c_Z<\frac{1}{2}, max { ρ t ∗ , χ t ∗ } ≤ 2 Z t ∗ + ∣ D t ∗ ∣ ≤ 4 3 Z t ∗ ≤ 2 3 c Z < 2 1 ,
after choosing c Z < 1 / 3 c_Z<1/3 c Z < 1/3 . Hence t ∗ < τ B t^*<\tau_B t ∗ < τ B . This completes the proof.
Weak Recovery Theorem
Now, we’re prepared to derive the final theorem for weak recovery, which states:
Theorem D.17 (Escape from mediocrity under discrete-time SGD). Assume Assumption D.1 and the auxiliary estimates used in Lemmas D.13, D.14, and D.16. Let E 0 \mathcal{E}_0 E 0 be the favorable initialization event. Then there exist constants
c e s c , C t i m e , K > 0 c_{\mathrm{esc}},C_{\mathrm{time}},K>0 c esc , C time , K > 0 depending only on the model constants such that, for all sufficiently large d d d , the following holds. Let
T : = ⌈ C t i m e δ − 1 log d ⌉ . T:=\left\lceil C_{\mathrm{time}}\delta^{-1}\log d\right\rceil. T := ⌈ C time δ − 1 log d ⌉ . Conditional on E 0 \mathcal{E}_0 E 0 , with probability at least 1 − K ε 1-K\varepsilon 1 − K ε , there exists a time
t e s c ≤ C t i m e δ − 1 log d (186) t_{\mathrm{esc}}\le C_{\mathrm{time}}\delta^{-1}\log d \tag*{(186)} t esc ≤ C time δ − 1 log d (186) such that
min { ρ t e s c , χ t e s c } ≥ c e s c . (187) \min\{\rho_{t_{\mathrm{esc}}},\chi_{t_{\mathrm{esc}}}\}\ge c_{\mathrm{esc}}. \tag*{(187)} min { ρ t esc , χ t esc } ≥ c esc . (187) Moreover, throughout the escape window,
η t ≤ C log d d , 0 ≤ t ≤ t e s c . (188) \eta_t\le C\frac{\log d}{\sqrt{d}},\qquad0\le t\le t_{\mathrm{esc}}. \tag*{(188)} η t ≤ C d log d , 0 ≤ t ≤ t esc . (188) Equivalently, if P ( E 0 ) ≥ p 0 > 0 \mathbb{P}(\mathcal{E}_0)\ge p_0>0 P ( E 0 ) ≥ p 0 > 0 , then
P ( ∃ t e s c ≤ C t i m e δ − 1 log d : min { ρ t e s c , χ t e s c } ≥ c e s c ) ≥ p 0 ( 1 − K ε ) . (189) \mathbb{P}\left(\exists\,t_{\mathrm{esc}}\le C_{\mathrm{time}}\delta^{-1}\log d:\min\{\rho_{t_{\mathrm{esc}}},\chi_{t_{\mathrm{esc}}}\}\ge c_{\mathrm{esc}}\right)\ge p_0(1-K\varepsilon). \tag*{(189)} P ( ∃ t esc ≤ C time δ − 1 log d : min { ρ t esc , χ t esc } ≥ c esc ) ≥ p 0 ( 1 − K ε ) . (189)
Proof. Choose C t i m e C_{\mathrm{time}} C time large enough so that
C t i m e > C I + C I I + 2 , C_{\mathrm{time}} > C_{\mathrm{I}} + C_{\mathrm{II}} + 2, C time > C I + C II + 2 ,
where C I C_{\mathrm{I}} C I is the Phase-I time constant from Lemma D.13, and C I I C_{\mathrm{II}} C II is the Phase-II time constant from Lemma D.16. Increasing C t i m e C_{\mathrm{time}} C time if necessary, the horizon T T T satisfies all horizon assumptions required in the Phase-I, entry-balance, and Phase-II lemmas.
Define the weak-recovery good event, excluding initialization, by
H T w e a k : = E T ( T ) ∩ E T ( Z ) ∩ E T ( η ) ∩ E T ( ρ , χ ) ∩ E T ( Ω ) . \mathcal{H}^{\mathrm{weak}}_{T} := E^{(T)}_{T} \cap E^{(Z)}_{T} \cap E^{(\eta)}_{T} \cap E^{(\rho,\chi)}_{T} \cap E^{(\Omega)}_{T}. H T weak := E T ( T ) ∩ E T ( Z ) ∩ E T ( η ) ∩ E T ( ρ , χ ) ∩ E T ( Ω ) .
By bounds of the Taylor (89), accumulative (103) and difference (105) martingale and discretization control, η \eta η -control (135), (141) and third-moment estimates,
P ( ( H T w e a k ) c ∣ E 0 ) ≤ K ε . (190) \mathbb{P}\left((\mathcal{H}^{\mathrm{weak}}_{T})^{c} \mid\mathcal{E}_{0}\right) \le K\varepsilon. \tag*{(190)} P ( ( H T weak ) c ∣ E 0 ) ≤ K ε . (190)
Set
G T w e a k : = E 0 ∩ H T w e a k . \mathcal{G}^{\mathrm{weak}}_{T} := \mathcal{E}_{0} \cap\mathcal{H}^{\mathrm{weak}}_{T}. G T weak := E 0 ∩ H T weak .
We now work on G T w e a k \mathcal{G}^{\mathrm{weak}}_{T} G T weak .
By Lemma D.13,
τ A < τ B , τ A δ ≤ C I log d , Z τ A ≥ c A d − 1 / 4 . (191) \tau_{A} < \tau_{B}, \qquad\tau_{A}\delta\le C_{\mathrm{I}}\log d, \qquad Z_{\tau_{A}} \ge c_{A}d^{-1/4}. \tag*{(191)} τ A < τ B , τ A δ ≤ C I log d , Z τ A ≥ c A d − 1/4 . (191)
The same lemma also gives, for 0 ≤ t ≤ τ A 0 \le t \le\tau_{A} 0 ≤ t ≤ τ A ,
η t ≤ C η log d + 1 + t δ d . (192) \eta_{t} \le C_{\eta}\frac{\sqrt{\log d}+1+t\delta}{\sqrt{d}}. \tag*{(192)} η t ≤ C η d log d + 1 + t δ . (192)
Next, Lemma D.14 gives
∣ D τ A ∣ = o ( Z τ A ) . |D_{\tau_{A}}| = o(Z_{\tau_{A}}). ∣ D τ A ∣ = o ( Z τ A ) .
Hence, for all sufficiently large d d d ,
∣ D τ A ∣ ≤ 1 8 Z τ A . (193) |D_{\tau_{A}}| \le\frac{1}{8}Z_{\tau_{A}}. \tag*{(193)} ∣ D τ A ∣ ≤ 8 1 Z τ A . (193)
Now apply Lemma D.16. Define
t ∗ : = inf { t ≥ τ A : Z t ≥ c Z } , t^{*} := \inf\{t \ge\tau_{A}: Z_{t} \ge c_{Z}\}, t ∗ := inf { t ≥ τ A : Z t ≥ c Z } ,
where c Z > 0 c_{Z}>0 c Z > 0 is the constant from Lemma D.16. The Phase-II lemma gives
t ∗ ≤ τ A + ⌈ C I I δ − 1 log d ⌉ , t^{*} \le\tau_{A} + \left\lceil C_{\mathrm{II}}\delta^{-1}\log d \right\rceil, t ∗ ≤ τ A + ⌈ C II δ − 1 log d ⌉ ,
Z t ∗ ∈ [ c Z , 2 c Z ] , Z_{t^{*}} \in[c_{Z},2c_{Z}], Z t ∗ ∈ [ c Z , 2 c Z ] ,
and, for every τ A ≤ t ≤ t ∗ \tau_{A} \le t \le t^{*} τ A ≤ t ≤ t ∗ ,
η t ≤ C η , I I log d d , ∣ D t ∣ ≤ 1 2 Z t . (194) \eta_{t} \le C_{\eta,\mathrm{II}}\frac{\log d}{\sqrt{d}}, \qquad|D_{t}| \le\frac{1}{2}Z_{t}. \tag*{(194)} η t ≤ C η , II d log d , ∣ D t ∣ ≤ 2 1 Z t . (194)
Therefore,
t ∗ ≤ C t i m e δ − 1 log d t^{*} \le C_{\mathrm{time}}\delta^{-1}\log d t ∗ ≤ C time δ − 1 log d
for all sufficiently large d d d .
Set
t e s c : = t ∗ , c e s c : = c Z 4 . t_{\mathrm{esc}} := t^{*}, \qquad c_{\mathrm{esc}} := \frac{c_{Z}}{4}. t esc := t ∗ , c esc := 4 c Z .
Since
ρ t = Z t + D t 2 , χ t = Z t − D t 2 , \rho_{t} = \frac{Z_{t}+D_{t}}{2}, \qquad\chi_{t} = \frac{Z_{t}-D_{t}}{2}, ρ t = 2 Z t + D t , χ t = 2 Z t − D t ,
the balance estimate at t = t ∗ t=t^{*} t = t ∗ gives
min { ρ t e s c , χ t e s c } = Z t e s c − ∣ D t e s c ∣ 2 ≥ 1 4 Z t e s c ≥ c Z 4 = c e s c . \min\{\rho_{t_{\mathrm{esc}}},\chi_{t_{\mathrm{esc}}}\}
= \frac{Z_{t_{\mathrm{esc}}}-|D_{t_{\mathrm{esc}}}|}{2}
\ge\frac{1}{4}Z_{t_{\mathrm{esc}}}
\ge\frac{c_{Z}}{4}
= c_{\mathrm{esc}}. min { ρ t esc , χ t esc } = 2 Z t esc − ∣ D t esc ∣ ≥ 4 1 Z t esc ≥ 4 c Z = c esc .
This proves weak recovery on G T w e a k \mathcal{G}^{\mathrm{weak}}_{T} G T weak .
It remains to verify the claimed η t \eta_t η t -bound. For 0 ≤ t ≤ τ A 0 \le t \le\tau_A 0 ≤ t ≤ τ A , (192) and τ A δ ≤ C I log d \tau_A\delta\le C_I\log d τ A δ ≤ C I log d imply
η t ≤ C log d d . \eta_t \le C\frac{\log d}{\sqrt{d}}. η t ≤ C d log d .
For τ A ≤ t ≤ t e s c \tau_A \le t \le t_{\mathrm{esc}} τ A ≤ t ≤ t esc , the same bound follows from (194) . Hence
η t ≤ C log d d , 0 ≤ t ≤ t e s c . \eta_t \le C\frac{\log d}{\sqrt{d}}, \qquad0 \le t \le t_{\mathrm{esc}}. η t ≤ C d log d , 0 ≤ t ≤ t esc .
Finally, the conditional probability statement follows from (190) . If P ( E 0 ) ≥ p 0 \mathbb{P}(\mathcal{E}_0) \ge p_0 P ( E 0 ) ≥ p 0 , then
P ( E 0 ∩ H T w e a k ) = P ( H T w e a k ∣ E 0 ) P ( E 0 ) ≥ p 0 ( 1 − K ε ) , \mathbb{P}(\mathcal{E}_0 \cap\mathcal{H}^{\mathrm{weak}}_T) = \mathbb{P}(\mathcal{H}^{\mathrm{weak}}_T \mid\mathcal{E}_0)\mathbb{P}(\mathcal{E}_0) \ge p_0(1-K\varepsilon), P ( E 0 ∩ H T weak ) = P ( H T weak ∣ E 0 ) P ( E 0 ) ≥ p 0 ( 1 − K ε ) ,
which gives the unconditional statement. □ \square □
Strong Recovery Setups
Population inputs reused from the continuous-time analysis. Throughout this subsection, let r l o c > 0 r_{\mathrm{loc}} > 0 r loc > 0 and μ l o c > 0 \mu_{\mathrm{loc}} > 0 μ loc > 0 be the constants from Lemma C.14. In particular, for every
y ∈ U r l o c , U r l o c : = { y ∈ M : dist M ( y , y ∗ ) ≤ r l o c } , y \in\mathcal{U}_{r_{\mathrm{loc}}}, \qquad\mathcal{U}_{r_{\mathrm{loc}}} := \{y \in\mathcal{M} : \operatorname{dist}_{\mathcal{M}}(y,y^*) \le r_{\mathrm{loc}}\}, y ∈ U r loc , U r loc := { y ∈ M : dist M ( y , y ∗ ) ≤ r loc } ,
we have
Hess M L ( y ) ⪰ μ l o c I . \operatorname{Hess}_{\mathcal{M}} L(y) \succeq\mu_{\mathrm{loc}}I. Hess M L ( y ) ⪰ μ loc I .
Discrete-time basin-entry. Mirroring Lemma C.9, the following lemmas establish that after an additional O ( d ) O(d) O ( d ) iterations beyond weak recovery, the trajectory enters the local strongly-convex cap in discrete-time:
Lemma D.19 (Entry-window delocalization and overlap-noise control). Work on the weak-recovery event at t e s c t_{\mathrm{esc}} t esc . Assume the incoherence bound (A4). After t e s c t_{\mathrm{esc}} t esc , run SGD with the local step size δ l o c = c l o c d \delta_{\mathrm{loc}} = \frac{c_{\mathrm{loc}}}{d} δ loc = d c loc , where c l o c > 0 c_{\mathrm{loc}} > 0 c loc > 0 is chosen sufficiently small depending only on the model constants, C i n C_{\mathrm{in}} C in , and the desired entry-window failure level. Fix C i n < ∞ C_{\mathrm{in}} < \infty C in < ∞ , and set N i n : = ⌈ C i n δ l o c − 1 ⌉ N_{\mathrm{in}} := \lceil C_{\mathrm{in}}\delta_{\mathrm{loc}}^{-1}\rceil N in := ⌈ C in δ loc − 1 ⌉ . Then, conditionally on F t e s c \mathcal{F}_{t_{\mathrm{esc}}} F t esc , there exists an event E i n \mathcal{E}_{\mathrm{in}} E in , measurable with respect to the samples in [ t e s c , t e s c + N i n ] [t_{\mathrm{esc}},t_{\mathrm{esc}}+N_{\mathrm{in}}] [ t esc , t esc + N in ] , such that
P ( E i n ∣ F t e s c ) ≥ 1 − ν i n . (195) \mathbb{P}(\mathcal{E}_{\mathrm{in}} \mid\mathcal{F}_{t_{\mathrm{esc}}}) \ge1-\nu_{\mathrm{in}}. \tag*{(195)} P ( E in ∣ F t esc ) ≥ 1 − ν in . (195) On E i n \mathcal{E}_{\mathrm{in}} E in ,
sup t e s c ≤ t ≤ t e s c + N i n η t ≤ η ˉ d , (196) \sup_{t_{\mathrm{esc}}\le t\le t_{\mathrm{esc}}+N_{\mathrm{in}}}\eta_t \le\bar{\eta}_d, \tag*{(196)} t esc ≤ t ≤ t esc + N in sup η t ≤ η ˉ d , (196) where, with L ν , d : = 1 + log ( d / ν i n ) L_{\nu,d} := 1+\log(d/\nu_{\mathrm{in}}) L ν , d := 1 + log ( d / ν in ) ,
η ˉ d ≤ C ( log d d + log d d + δ l o c L ν , d + L ν , d δ l o c 2 d 3 / 2 ) . (197) \bar{\eta}_d \le C\left(\frac{\log d}{\sqrt{d}}+\sqrt{\frac{\log d}{d}}+\sqrt{\delta_{\mathrm{loc}}L_{\nu,d}}+L_{\nu,d}\delta_{\mathrm{loc}}^2d^{3/2}\right). \tag*{(197)} η ˉ d ≤ C ( d log d + d log d + δ loc L ν , d + L ν , d δ loc 2 d 3/2 ) . (197) In particular, for fixed or polynomially small ν i n \nu_{\mathrm{in}} ν in ,$
η ˉ d = O ( log d d ) = o d ( 1 ) . (198) \bar{\eta}_d = O\left(\frac{\log d}{\sqrt{d}}\right) = o_d(1). \tag*{(198)} η ˉ d = O ( d log d ) = o d ( 1 ) . (198)
Moreover, for j ∈ { ρ , χ } j \in\{\rho,\chi\} j ∈ { ρ , χ } ,
sup t e s c ≤ s ≤ u ≤ t e s c + N i n ∣ ∑ k = s u − 1 Δ N k + 1 ( j ) ∣ ≤ Λ d , (199) \sup_{t_{\mathrm{esc}}\leq s\leq u\leq t_{\mathrm{esc}}+N_{\mathrm{in}}}\left|\sum_{k=s}^{u-1}\Delta N_{k+1}^{(j)}\right|\leq\Lambda_d, \tag*{(199)} t esc ≤ s ≤ u ≤ t esc + N in sup k = s ∑ u − 1 Δ N k + 1 ( j ) ≤ Λ d , (199)
where
Λ d ≤ C ( δ l o c log ( 1 / ν i n ) + L ν , d δ l o c d 3 / 2 ) = o d ( 1 ) (200) \Lambda_d\leq C\left(\sqrt{\delta_{\mathrm{loc}}\log(1/\nu_{\mathrm{in}})}+L_{\nu,d}\delta_{\mathrm{loc}}d^{3/2}\right)=o_d(1) \tag*{(200)} Λ d ≤ C ( δ loc log ( 1/ ν in ) + L ν , d δ loc d 3/2 ) = o d ( 1 ) (200)
for fixed or polynomially small ν i n \nu_{\mathrm{in}} ν in .
Proof. Write
t 0 : = t e s c , δ : = δ l o c , N : = N i n = ⌈ C i n δ − 1 ⌉ . t_0:=t_{\mathrm{esc}},\qquad\delta:=\delta_{\mathrm{loc}},\qquad N:=N_{\mathrm{in}}=\left\lceil C_{\mathrm{in}}\delta^{-1}\right\rceil. t 0 := t esc , δ := δ loc , N := N in = ⌈ C in δ − 1 ⌉ .
Throughout the proof we condition on F t 0 \mathcal{F}_{t_0} F t 0 and on the weak-recovery event stated in the lemma. Since the samples after t 0 t_0 t 0 are fresh and independent of F t 0 \mathcal{F}_{t_0} F t 0 , the martingale and moment estimates from the weak-recovery section apply conditionally on F t 0 \mathcal{F}_{t_0} F t 0 , with the same constants.
We shall use the normalized spherical one-step expansion
ρ t + 1 = ρ t + δ F t ( ρ ) + Δ N t + 1 ( ρ ) , χ t + 1 = χ t + δ F t ( χ ) + Δ N t + 1 ( χ ) . \rho_{t+1}=\rho_t+\delta F_t^{(\rho)}+\Delta N_{t+1}^{(\rho)},\qquad\chi_{t+1}=\chi_t+\delta F_t^{(\chi)}+\Delta N_{t+1}^{(\chi)}. ρ t + 1 = ρ t + δ F t ( ρ ) + Δ N t + 1 ( ρ ) , χ t + 1 = χ t + δ F t ( χ ) + Δ N t + 1 ( χ ) .
where
Δ N t + 1 ( j ) = δ M t ( j ) + R t ( j ) , j ∈ { ρ , χ } . \Delta N_{t+1}^{(j)}=\delta M_t^{(j)}+R_t^{(j)},\qquad j\in\{\rho,\chi\}. Δ N t + 1 ( j ) = δ M t ( j ) + R t ( j ) , j ∈ { ρ , χ } .
Here M t ( j ) M_t^{(j)} M t ( j ) is a scalar martingale increment and R t ( j ) R_t^{(j)} R t ( j ) is the normalization/retraction remainder.
Step 1: overlap-noise control.
By the scalar martingale moment bounds used in the weak-recovery proof, there is K M < ∞ K_M<\infty K M < ∞ , independent of d d d , such that
E [ ( M t ( j ) ) 2 ∣ F t ] ≤ K M ( 1 + δ 2 d 2 ) , j ∈ { ρ , χ } . \mathbb{E}\left[(M_t^{(j)})^2\mid\mathcal{F}_t\right]\leq K_M(1+\delta^2d^2),\qquad j\in\{\rho,\chi\}. E [ ( M t ( j ) ) 2 ∣ F t ] ≤ K M ( 1 + δ 2 d 2 ) , j ∈ { ρ , χ } .
Since δ d = c l o c \delta d=c_{\mathrm{loc}} δ d = c loc , this is bounded by another constant K M ′ K'_M K M ′ .
For u ∈ [ t 0 , t 0 + N ] u\in[t_0,t_0+N] u ∈ [ t 0 , t 0 + N ] , define
S u ( j ) : = δ ∑ k = t 0 u − 1 M k ( j ) . S_u^{(j)}:=\delta\sum_{k=t_0}^{u-1}M_k^{(j)}. S u ( j ) := δ k = t 0 ∑ u − 1 M k ( j ) .
Then
⟨ S ( j ) ⟩ t 0 + N ≤ K M ′ δ 2 N ≤ C δ , \langle S^{(j)}\rangle_{t_0+N}\leq K'_M\delta^2N\leq C\delta, ⟨ S ( j ) ⟩ t 0 + N ≤ K M ′ δ 2 N ≤ C δ ,
because N ≤ C δ − 1 N\leq C\delta^{-1} N ≤ C δ − 1 . By the same conditional Freedman/truncation argument used for the weak-recovery overlap martingales, with conditional probability at least 1 − ν i n / 8 1-\nu_{\mathrm{in}}/8 1 − ν in /8 ,
sup t 0 ≤ u ≤ t 0 + N ∣ δ ∑ k = t 0 u − 1 M k ( j ) ∣ ≤ C δ log ( 1 / ν i n ) . \sup_{t_0\leq u\leq t_0+N}\left|\delta\sum_{k=t_0}^{u-1}M_k^{(j)}\right|\leq C\sqrt{\delta\log(1/\nu_{\mathrm{in}})}. t 0 ≤ u ≤ t 0 + N sup δ k = t 0 ∑ u − 1 M k ( j ) ≤ C δ log ( 1/ ν in ) .
Consequently,
sup t 0 ≤ s ≤ u ≤ t 0 + N ∣ δ ∑ k = s u − 1 M k ( j ) ∣ ≤ C δ log ( 1 / ν i n ) , \sup_{t_0\leq s\leq u\leq t_0+N}\left|\delta\sum_{k=s}^{u-1}M_k^{(j)}\right|\leq C\sqrt{\delta\log(1/\nu_{\mathrm{in}})}, t 0 ≤ s ≤ u ≤ t 0 + N sup δ k = s ∑ u − 1 M k ( j ) ≤ C δ log ( 1/ ν in ) ,
after increasing C C C .
Next consider the remainders. On the local Taylor event for the normalized spherical step,
∣ R t ( ρ ) ∣ ≤ C δ 3 ∥ g t ( β ) ∥ 3 , ∣ R t ( χ ) ∣ ≤ C δ 3 ∥ g t ( θ ) ∥ 3 . \left|R_t^{(\rho)}\right|\leq C\delta^3\lVert g_t^{(\beta)}\rVert^3,\qquad\left|R_t^{(\chi)}\right|\leq C\delta^3\lVert g_t^{(\theta)}\rVert^3. R t ( ρ ) ≤ C δ 3 ∥ g t ( β ) ∥ 3 , R t ( χ ) ≤ C δ 3 ∥ g t ( θ ) ∥ 3 .
Moreover, the gradient moment bounds give
E [ ∥ g t ( β ) ∥ 3 ∣ F t ] + E [ ∥ g t ( θ ) ∥ 3 ∣ F t ] ≤ C d 3 / 2 . \mathbb{E}\left[\lVert g_t^{(\beta)}\rVert^3\mid\mathcal{F}_t\right]+\mathbb{E}\left[\lVert g_t^{(\theta)}\rVert^3\mid\mathcal{F}_t\right]\leq Cd^{3/2}. E [ ∥ g t ( β ) ∥ 3 ∣ F t ] + E [ ∥ g t ( θ ) ∥ 3 ∣ F t ] ≤ C d 3/2 .
Hence, for j ∈ { ρ , χ } j \in\{\rho,\chi\} j ∈ { ρ , χ } ,
E [ ∑ k = t 0 t 0 + N − 1 ∣ R k ( j ) ∣ ∣ F t 0 ] ≤ C N δ 3 d 3 / 2 ≤ C δ 2 d 3 / 2 . \mathbb{E}\left[\left.\sum_{k=t_0}^{t_0+N-1} \left|R_k^{(j)}\right| \right| \mathcal{F}_{t_0}\right] \le CN\delta^3d^{3/2} \le C\delta^2d^{3/2}. E [ k = t 0 ∑ t 0 + N − 1 R k ( j ) F t 0 ] ≤ C N δ 3 d 3/2 ≤ C δ 2 d 3/2 .
Using the same high-moment/truncation estimate as in the weak-recovery discretization control, there is an event of conditional probability at least 1 − ν i n / 8 1-\nu_{\mathrm{in}}/8 1 − ν in /8 on which
sup t 0 ≤ s ≤ u ≤ t 0 + N ∣ ∑ k = s u − 1 R k ( j ) ∣ ≤ C L ν , d δ 2 d 3 / 2 , L ν , d : = 1 + log ( d / ν i n ) . \sup_{t_0\le s\le u\le t_0+N}\left|\sum_{k=s}^{u-1}R_k^{(j)}\right| \le CL_{\nu,d}\delta^2d^{3/2}, \qquad L_{\nu,d}:=1+\log(d/\nu_{\mathrm{in}}). t 0 ≤ s ≤ u ≤ t 0 + N sup k = s ∑ u − 1 R k ( j ) ≤ C L ν , d δ 2 d 3/2 , L ν , d := 1 + log ( d / ν in ) .
Therefore, on an event E o v E_{\mathrm{ov}} E ov of conditional probability at least 1 − ν i n / 2 1-\nu_{\mathrm{in}}/2 1 − ν in /2 , for both j ∈ { ρ , χ } j\in\{\rho,\chi\} j ∈ { ρ , χ } ,
sup t 0 ≤ s ≤ u ≤ t 0 + N ∣ ∑ k = s u − 1 Δ N k + 1 ( j ) ∣ ≤ Λ d , \sup_{t_0\le s\le u\le t_0+N}\left|\sum_{k=s}^{u-1}\Delta N_{k+1}^{(j)}\right| \le\Lambda_d, t 0 ≤ s ≤ u ≤ t 0 + N sup k = s ∑ u − 1 Δ N k + 1 ( j ) ≤ Λ d ,
where
Λ d : = C ( δ log ( 1 / ν i n ) + L ν , d δ 2 d 3 / 2 ) . \Lambda_d:=C\left(\sqrt{\delta\log(1/\nu_{\mathrm{in}})}+L_{\nu,d}\delta^2d^{3/2}\right). Λ d := C ( δ log ( 1/ ν in ) + L ν , d δ 2 d 3/2 ) .
Since δ = c l o c / d \delta=c_{\mathrm{loc}}/d δ = c loc / d ,
Λ d ≤ C log ( 1 / ν i n ) d + C L ν , d 1 d = o d ( 1 ) \Lambda_d\le C\sqrt{\frac{\log(1/\nu_{\mathrm{in}})}{d}}+CL_{\nu,d}\frac{1}{\sqrt{d}}=o_d(1) Λ d ≤ C d log ( 1/ ν in ) + C L ν , d d 1 = o d ( 1 )
for fixed or polynomially small ν i n \nu_{\mathrm{in}} ν in .
Step 2: η t \eta_t η t -control on the entry window.
To control η t = ∥ θ t ∥ ∞ + ∥ θ ∗ ∥ ∞ \eta_t=\|\theta_t\|_\infty+\|\theta^*\|_\infty η t = ∥ θ t ∥ ∞ + ∥ θ ∗ ∥ ∞ , we use the same ℓ ∞ \ell_\infty ℓ ∞ inequality as in the weak-recovery analysis:
∥ θ t ∥ ∞ ≤ ∥ θ t 0 ∥ ∞ + δ ∑ k = t 0 t − 1 ∥ g ˉ k ( θ ) ∥ ∞ + max i ≤ d ∣ δ ∑ k = t 0 t − 1 ξ k , i ( θ ) ∣ + δ 2 2 ∑ k = t 0 t − 1 ∥ g k ( θ ) ∥ 2 ∥ θ k ∥ ∞ + δ 3 2 ∑ k = t 0 t − 1 ∥ g k ( θ ) ∥ 3 , \|\theta_t\|_\infty\le\|\theta_{t_0}\|_\infty+\delta\sum_{k=t_0}^{t-1}\|\bar g_k^{(\theta)}\|_\infty+\max_{i\le d}\left|\delta\sum_{k=t_0}^{t-1}\xi_{k,i}^{(\theta)}\right|+\frac{\delta^2}{2}\sum_{k=t_0}^{t-1}\|g_k^{(\theta)}\|^2\|\theta_k\|_\infty+\frac{\delta^3}{2}\sum_{k=t_0}^{t-1}\|g_k^{(\theta)}\|^3, ∥ θ t ∥ ∞ ≤ ∥ θ t 0 ∥ ∞ + δ k = t 0 ∑ t − 1 ∥ g ˉ k ( θ ) ∥ ∞ + i ≤ d max δ k = t 0 ∑ t − 1 ξ k , i ( θ ) + 2 δ 2 k = t 0 ∑ t − 1 ∥ g k ( θ ) ∥ 2 ∥ θ k ∥ ∞ + 2 δ 3 k = t 0 ∑ t − 1 ∥ g k ( θ ) ∥ 3 ,
where
g ˉ k ( θ ) : = E [ g k ( θ ) ∣ F k ] , ξ k ( θ ) : = g k ( θ ) − g ˉ k ( θ ) . \bar g_k^{(\theta)}:=\mathbb{E}[g_k^{(\theta)}\mid\mathcal{F}_k],\qquad\xi_k^{(\theta)}:=g_k^{(\theta)}-\bar g_k^{(\theta)}. g ˉ k ( θ ) := E [ g k ( θ ) ∣ F k ] , ξ k ( θ ) := g k ( θ ) − g ˉ k ( θ ) .
Set
Θ u : = sup t 0 ≤ t ≤ u η t , t 0 ≤ u ≤ t 0 + N . \Theta_u:=\sup_{t_0\le t\le u}\eta_t,\qquad t_0\le u\le t_0+N. Θ u := t 0 ≤ t ≤ u sup η t , t 0 ≤ u ≤ t 0 + N .
By the conditional coordinate Freedman bound, with conditional probability at least 1 − ν i n / 8 1-\nu_{\mathrm{in}}/8 1 − ν in /8 ,
max i ≤ d sup t 0 ≤ u ≤ t 0 + N ∣ δ ∑ k = t 0 u − 1 ξ k , i ( θ ) ∣ ≤ C δ L ν , d . \max_{i\le d}\sup_{t_0\le u\le t_0+N}\left|\delta\sum_{k=t_0}^{u-1}\xi_{k,i}^{(\theta)}\right|\le C\sqrt{\delta L_{\nu,d}}. i ≤ d max t 0 ≤ u ≤ t 0 + N sup δ k = t 0 ∑ u − 1 ξ k , i ( θ ) ≤ C δ L ν , d .
The gradient moment estimates imply
E [ δ 2 ∑ k = t 0 t 0 + N − 1 ∥ g k ( θ ) ∥ 2 ∣ F t 0 ] ≤ C δ 2 N d ≤ C δ d = C c l o c . \mathbb{E}\left[\left.\delta^2\sum_{k=t_0}^{t_0+N-1}\|g_k^{(\theta)}\|^2\right|\mathcal{F}_{t_0}\right]\le C\delta^2Nd\le C\delta d=Cc_{\mathrm{loc}}. E [ δ 2 k = t 0 ∑ t 0 + N − 1 ∥ g k ( θ ) ∥ 2 F t 0 ] ≤ C δ 2 N d ≤ C δ d = C c loc .
Choosing c l o c > 0 c_{\mathrm{loc}}>0 c loc > 0 sufficiently small, the same concentration/Markov argument used in the weak phase gives, with conditional probability at least 1 − ν i n / 8 1-\nu_{\mathrm{in}}/8 1 − ν in /8 ,
δ 2 2 ∑ k = t 0 t 0 + N − 1 ∥ g k ( θ ) ∥ 2 ≤ 1 4 . \frac{\delta^2}{2}\sum_{k=t_0}^{t_0+N-1}\|g_k^{(\theta)}\|^2\le\frac14. 2 δ 2 k = t 0 ∑ t 0 + N − 1 ∥ g k ( θ ) ∥ 2 ≤ 4 1 .
Similarly,
E [ δ 3 ∑ k = t 0 t 0 + N − 1 ∥ g k ( θ ) ∥ 3 ∣ F t 0 ] ≤ C δ 3 N d 3 / 2 ≤ C δ 2 d 3 / 2 . \mathbb{E}\left[\left.\delta^3\sum_{k=t_0}^{t_0+N-1}\|g_k^{(\theta)}\|^3\right|\mathcal{F}_{t_0}\right]\le C\delta^3Nd^{3/2}\le C\delta^2d^{3/2}. E [ δ 3 k = t 0 ∑ t 0 + N − 1 ∥ g k ( θ ) ∥ 3 F t 0 ] ≤ C δ 3 N d 3/2 ≤ C δ 2 d 3/2 .
and hence, on an event of conditional probability at least 1 − ν i n / 8 1-\nu_{\mathrm{in}}/8 1 − ν in /8 ,
δ 3 ∑ k = t 0 t 0 + N − 1 ∥ g k ( θ ) ∥ 3 ≤ C L ν , d δ 2 d 3 / 2 . \delta^3\sum_{k=t_0}^{t_0+N-1}\left\|g_k^{(\theta)}\right\|^3 \le CL_{\nu,d}\delta^2d^{3/2}. δ 3 k = t 0 ∑ t 0 + N − 1 g k ( θ ) 3 ≤ C L ν , d δ 2 d 3/2 .
Let E η , n o i s e E_{\eta,\mathrm{noise}} E η , noise be the intersection of these three η \eta η -control events. On E η , n o i s e E_{\eta,\mathrm{noise}} E η , noise , for every u ∈ [ t 0 , t 0 + N ] u\in[t_0,t_0+N] u ∈ [ t 0 , t 0 + N ] ,
Θ u ≤ C η t 0 + C δ ∑ k = t 0 u − 1 ∥ g ˉ k ( θ ) ∥ ∞ + C δ L ν , d + C L ν , d δ 2 d 3 / 2 + 1 4 Θ u . \Theta_u\le C\eta_{t_0}+C\delta\sum_{k=t_0}^{u-1}\left\|\bar g_k^{(\theta)}\right\|_\infty+C\sqrt{\delta}L_{\nu,d}+CL_{\nu,d}\delta^2d^{3/2}+\frac{1}{4}\Theta_u. Θ u ≤ C η t 0 + C δ k = t 0 ∑ u − 1 g ˉ k ( θ ) ∞ + C δ L ν , d + C L ν , d δ 2 d 3/2 + 4 1 Θ u .
Absorbing the last term,
Θ u ≤ C η t 0 + C δ ∑ k = t 0 u − 1 ∥ g ˉ k ( θ ) ∥ ∞ + C δ L ν , d + C L ν , d δ 2 d 3 / 2 . (201) \Theta_u\le C\eta_{t_0}+C\delta\sum_{k=t_0}^{u-1}\left\|\bar g_k^{(\theta)}\right\|_\infty+C\sqrt{\delta}L_{\nu,d}+CL_{\nu,d}\delta^2d^{3/2}. \tag*{(201)} Θ u ≤ C η t 0 + C δ k = t 0 ∑ u − 1 g ˉ k ( θ ) ∞ + C δ L ν , d + C L ν , d δ 2 d 3/2 . (201)
It remains to bound the mean-field term. The population ℓ ∞ \ell_\infty ℓ ∞ gradient estimate gives, uniformly along the entry window,
∥ g ˉ k ( θ ) ∥ ∞ ≤ C ∥ θ ∗ ∥ ∞ + C η k + C η k 2 . \left\|\bar g_k^{(\theta)}\right\|_\infty\le C\|\theta^*\|_\infty+C\eta_k+C\eta_k^2. g ˉ k ( θ ) ∞ ≤ C ∥ θ ∗ ∥ ∞ + C η k + C η k 2 .
Therefore, since N δ ≤ C i n + 1 N\delta\le C_{\mathrm{in}}+1 N δ ≤ C in + 1 , (193) implies
Θ u ≤ A d + C δ ∑ k = t 0 u − 1 Θ k + C δ ∑ k = t 0 u − 1 Θ k 2 , \Theta_u\le A_d+C\delta\sum_{k=t_0}^{u-1}\Theta_k+C\delta\sum_{k=t_0}^{u-1}\Theta_k^2, Θ u ≤ A d + C δ k = t 0 ∑ u − 1 Θ k + C δ k = t 0 ∑ u − 1 Θ k 2 ,
where
A d : = C ( η t 0 + ∥ θ ∗ ∥ ∞ + δ L ν , d + L ν , d δ 2 d 3 / 2 ) . A_d:=C\left(\eta_{t_0}+\|\theta^*\|_\infty+\sqrt{\delta}L_{\nu,d}+L_{\nu,d}\delta^2d^{3/2}\right). A d := C ( η t 0 + ∥ θ ∗ ∥ ∞ + δ L ν , d + L ν , d δ 2 d 3/2 ) .
Now insert the scales. On the weak-recovery event,
η t 0 ≤ C η , e s c log d d , \eta_{t_0}\le C_{\eta,\mathrm{esc}}\frac{\log d}{\sqrt d}, η t 0 ≤ C η , esc d log d ,
and by the teacher incoherence assumption,
∥ θ ∗ ∥ ∞ ≤ C ∗ log d d . \|\theta^*\|_\infty\le C_*\sqrt{\frac{\log d}{d}}. ∥ θ ∗ ∥ ∞ ≤ C ∗ d log d .
Also δ = c l o c / d \delta=c_{\mathrm{loc}}/d δ = c loc / d . Hence, for fixed or polynomially small ν i n \nu_{\mathrm{in}} ν in ,
A d ≤ C ( log d d + log d d + L ν , d d + L ν , d d ) = O ( log d d ) = o d ( 1 ) . A_d\le C\left(\frac{\log d}{\sqrt d}+\sqrt{\frac{\log d}{d}}+\sqrt{\frac{L_{\nu,d}}{d}}+\frac{L_{\nu,d}}{\sqrt d}\right)=O\left(\frac{\log d}{\sqrt d}\right)=o_d(1). A d ≤ C ( d log d + d log d + d L ν , d + d L ν , d ) = O ( d log d ) = o d ( 1 ) .
We close the bootstrap. Define
τ η : = inf { u ∈ [ t 0 , t 0 + N ] : Θ u ≥ 1 } ∧ ( t 0 + N ) . \tau_\eta:=\inf\{u\in[t_0,t_0+N]:\Theta_u\ge1\}\wedge(t_0+N). τ η := inf { u ∈ [ t 0 , t 0 + N ] : Θ u ≥ 1 } ∧ ( t 0 + N ) .
For u ≤ τ η u\le\tau_\eta u ≤ τ η , one has Θ k 2 ≤ Θ k \Theta_k^2\le\Theta_k Θ k 2 ≤ Θ k , and therefore
Θ u ≤ A d + C δ ∑ k = t 0 u − 1 Θ k . \Theta_u\le A_d+C\delta\sum_{k=t_0}^{u-1}\Theta_k. Θ u ≤ A d + C δ k = t 0 ∑ u − 1 Θ k .
The discrete Gronwall inequality gives
Θ u ≤ A d exp ( C ( u − t 0 ) δ ) ≤ A d exp ( C ( C i n + 1 ) ) ≤ C i n ′ A d , u ≤ τ η . \Theta_u\le A_d\exp(C(u-t_0)\delta)\le A_d\exp(C(C_{\mathrm{in}}+1))\le C'_{\mathrm{in}}A_d,\qquad u\le\tau_\eta. Θ u ≤ A d exp ( C ( u − t 0 ) δ ) ≤ A d exp ( C ( C in + 1 )) ≤ C in ′ A d , u ≤ τ η .
Since A d = o d ( 1 ) A_d=o_d(1) A d = o d ( 1 ) , for all sufficiently large d d d the right-hand side is strictly smaller than 1 1 1 . Hence τ η = t 0 + N \tau_\eta=t_0+N τ η = t 0 + N , and
sup t 0 ≤ t ≤ t 0 + N η t = Θ t 0 + N ≤ η ˉ d . \sup_{t_0\le t\le t_0+N}\eta_t=\Theta_{t_0+N}\le\bar\eta_d. t 0 ≤ t ≤ t 0 + N sup η t = Θ t 0 + N ≤ η ˉ d .
where
η ˉ d : = C l ′ A d ≤ C ( log d d + log d d + δ L ν , d + L ν , d δ d 3 / 2 ) . \bar{\eta}_d := C'_l A_d \le C\left(\frac{\log d}{\sqrt{d}}+\sqrt{\frac{\log d}{d}}+\sqrt{\delta L_{\nu,d}}+L_{\nu,d}\delta d^{3/2}\right). η ˉ d := C l ′ A d ≤ C ( d log d + d log d + δ L ν , d + L ν , d δ d 3/2 ) .
This proves the claimed delocalization bound.
conclusion.
Define
E i n : = E o v ∩ E η , n o i s e . E_{\mathrm{in}} := E_{\mathrm{ov}} \cap E_{\eta,\mathrm{noise}}. E in := E ov ∩ E η , noise .
By the conditional probability bounds above and a union bound, after adjusting constants,
P ( E i n ∣ F t 0 ) ≥ 1 − ν i n . \mathbb{P}(E_{\mathrm{in}}\mid\mathcal{F}_{t_0})\ge1-\nu_{\mathrm{in}}. P ( E in ∣ F t 0 ) ≥ 1 − ν in .
On E i n E_{\mathrm{in}} E in , both conclusions hold:
sup t 0 ≤ t ≤ t 0 + N η t ≤ η ˉ d \sup_{t_0\le t\le t_0+N}\eta_t\le\bar{\eta}_d t 0 ≤ t ≤ t 0 + N sup η t ≤ η ˉ d
and, for j ∈ { ρ , χ } j\in\{\rho,\chi\} j ∈ { ρ , χ } ,
sup t 0 ≤ s ≤ u ≤ t 0 + N ∣ ∑ k = s u − 1 Δ N k + 1 ( j ) ∣ ≤ Λ d . \sup_{t_0\le s\le u\le t_0+N}\left|\sum_{k=s}^{u-1}\Delta N_{k+1}^{(j)}\right|\le\Lambda_d. t 0 ≤ s ≤ u ≤ t 0 + N sup k = s ∑ u − 1 Δ N k + 1 ( j ) ≤ Λ d .
Substituting back t 0 = t e s c t_0=t_{\mathrm{esc}} t 0 = t esc , N = N i n N=N_{\mathrm{in}} N = N in , and δ = δ l o c \delta=\delta_{\mathrm{loc}} δ = δ loc completes the proof. □
Lemma D.20 (One-step overlap recursion on the entry corridor). Assume Lemma C.3. After t e s c t_{\mathrm{esc}} t esc , run SGD with step size δ = δ l o c \delta=\delta_{\mathrm{loc}} δ = δ loc . On any time t t t in the entry window for which the one-step Taylor expansion is valid and
ρ t ≥ 0 , χ t ≥ 0 , \rho_t\ge0,\qquad\chi_t\ge0, ρ t ≥ 0 , χ t ≥ 0 , the overlap recursions can be written as
ρ t + 1 = ρ t + δ F t ( ρ ) + Δ N t + 1 ( ρ ) , χ t + 1 = χ t + δ F t ( χ ) + Δ N t + 1 ( χ ) , \rho_{t+1}=\rho_t+\delta F_t^{(\rho)}+\Delta N_{t+1}^{(\rho)},\qquad\chi_{t+1}=\chi_t+\delta F_t^{(\chi)}+\Delta N_{t+1}^{(\chi)}, ρ t + 1 = ρ t + δ F t ( ρ ) + Δ N t + 1 ( ρ ) , χ t + 1 = χ t + δ F t ( χ ) + Δ N t + 1 ( χ ) , where
Δ N t + 1 ( ρ ) = δ M t ( ρ ) + R t ( ρ ) , Δ N t + 1 ( χ ) = δ M t ( χ ) + R t ( χ ) . \Delta N_{t+1}^{(\rho)}=\delta M_t^{(\rho)}+R_t^{(\rho)},\qquad\Delta N_{t+1}^{(\chi)}=\delta M_t^{(\chi)}+R_t^{(\chi)}. Δ N t + 1 ( ρ ) = δ M t ( ρ ) + R t ( ρ ) , Δ N t + 1 ( χ ) = δ M t ( χ ) + R t ( χ ) . Here M t ( ρ ) , M t ( χ ) M_t^{(\rho)},M_t^{(\chi)} M t ( ρ ) , M t ( χ ) are martingale increments and the remainders satisfy
∣ R t ( ρ ) ∣ ≤ C δ 3 ∥ g t ( β ) ∥ 3 , ∣ R t ( χ ) ∣ ≤ C δ 3 ∥ g t ( θ ) ∥ 3 . |R_t^{(\rho)}|\le C\delta^3\lVert g_t^{(\beta)}\rVert^3,\qquad|R_t^{(\chi)}|\le C\delta^3\lVert g_t^{(\theta)}\rVert^3. ∣ R t ( ρ ) ∣ ≤ C δ 3 ∥ g t ( β ) ∥ 3 , ∣ R t ( χ ) ∣ ≤ C δ 3 ∥ g t ( θ ) ∥ 3 . Moreover, the predictable drifts satisfy
F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t − C δ d , F_t^{(\rho)}\ge a_\rho\chi_t(1-\rho_t^2)-b_\rho\eta_t-C\delta d, F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t − C δ d , and
F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t − C δ d . F_t^{(\chi)}\ge a_\chi\rho_t(1-\chi_t^2)-b_\chi\eta_t-C\delta d. F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t − C δ d . Consequently, on the event E i n E_{\mathrm{in}} E in from Lemma D.19,
F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − B d , F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − B d , F_t^{(\rho)}\ge a_\rho\chi_t(1-\rho_t^2)-B_d,\qquad F_t^{(\chi)}\ge a_\chi\rho_t(1-\chi_t^2)-B_d, F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − B d , F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − B d , where
B d : = C ( η ˉ d + δ d ) . B_d:=C(\bar{\eta}_d+\delta d). B d := C ( η ˉ d + δ d ) . In particular, for δ = c l o c / d \delta=c_{\mathrm{loc}}/d δ = c loc / d ,
B d ≤ C ( log d d + c l o c ) B_d\le C\left(\frac{\log d}{\sqrt{d}}+c_{\mathrm{loc}}\right) B d ≤ C ( d log d + c loc ) for fixed or polynomially small ν i n \nu_{\mathrm{in}} ν in .
Proof. We prove the ρ \rho ρ -recursion; the proof for χ \chi χ is identical. By the normalized spherical update,
β t + 1 = β t − δ g t ( β ) ∥ β t − δ g t ( β ) ∥ \beta_{t+1}=\frac{\beta_t-\delta g_t^{(\beta)}}{\lVert\beta_t-\delta g_t^{(\beta)}\rVert} β t + 1 = ∥ β t − δ g t ( β ) ∥ β t − δ g t ( β )
Since g t ( β ) ⊥ β t g_t^{(\beta)} \perp\beta_t g t ( β ) ⊥ β t , on the one-step Taylor event δ ∥ g t ( β ) ∥ ≤ 1 / 2 \delta\lVert g_t^{(\beta)}\rVert\le1/2 δ ∥ g t ( β ) ∥ ≤ 1/2 , we have
β t + 1 = β t − δ g t ( β ) − δ 2 2 ∥ g t ( β ) ∥ 2 β t + O ( δ 3 ∥ g t ( β ) ∥ 3 ) . \beta_{t+1}=\beta_t-\delta g_t^{(\beta)}-\frac{\delta^2}{2}\lVert g_t^{(\beta)}\rVert^2\beta_t+O(\delta^3\lVert g_t^{(\beta)}\rVert^3). β t + 1 = β t − δ g t ( β ) − 2 δ 2 ∥ g t ( β ) ∥ 2 β t + O ( δ 3 ∥ g t ( β ) ∥ 3 ) .
Taking inner product with β ∗ \beta^* β ∗ , and using ρ t = ⟨ β t , β ∗ ⟩ \rho_t=\langle\beta_t,\beta^*\rangle ρ t = ⟨ β t , β ∗ ⟩ , gives
ρ t + 1 = ρ t − δ ⟨ g t ( β ) , β ∗ ⟩ − δ 2 2 ρ t ∥ g t ( β ) ∥ 2 + R t ( ρ ) , \rho_{t+1}=\rho_t-\delta\langle g_t^{(\beta)},\beta^*\rangle-\frac{\delta^2}{2}\rho_t\lVert g_t^{(\beta)}\rVert^2+R_t^{(\rho)}, ρ t + 1 = ρ t − δ ⟨ g t ( β ) , β ∗ ⟩ − 2 δ 2 ρ t ∥ g t ( β ) ∥ 2 + R t ( ρ ) ,
where
∣ R t ( ρ ) ∣ ≤ C δ 3 ∥ g t ( β ) ∥ 3 . \lvert R_t^{(\rho)}\rvert\le C\delta^3\lVert g_t^{(\beta)}\rVert^3. ∣ R t ( ρ ) ∣ ≤ C δ 3 ∥ g t ( β ) ∥ 3 .
Equivalently,
ρ t + 1 = ρ t − δ ( ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ρ t ∥ g t ( β ) ∥ 2 ) + R t ( ρ ) . \rho_{t+1}=\rho_t-\delta\left(\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\rho_t\lVert g_t^{(\beta)}\rVert^2\right)+R_t^{(\rho)}. ρ t + 1 = ρ t − δ ( ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ρ t ∥ g t ( β ) ∥ 2 ) + R t ( ρ ) .
Define
F t ( ρ ) : = − E [ ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ρ t ∥ g t ( β ) ∥ 2 ∣ F t ] , F_t^{(\rho)}:=-\mathbb{E}\left[\left.\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\rho_t\lVert g_t^{(\beta)}\rVert^2\right|\mathcal{F}_t\right], F t ( ρ ) := − E [ ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ρ t ∥ g t ( β ) ∥ 2 F t ] ,
and
M t ( ρ ) : = − [ ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ρ t ∥ g t ( β ) ∥ 2 − E [ ⟨ g t ( β ) , β ∗ ⟩ + δ 2 ρ t ∥ g t ( β ) ∥ 2 ∣ F t ] ] . M_t^{(\rho)}:=-\left[\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\rho_t\lVert g_t^{(\beta)}\rVert^2-\mathbb{E}\left[\left.\langle g_t^{(\beta)},\beta^*\rangle+\frac{\delta}{2}\rho_t\lVert g_t^{(\beta)}\rVert^2\right|\mathcal{F}_t\right]\right]. M t ( ρ ) := − [ ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ρ t ∥ g t ( β ) ∥ 2 − E [ ⟨ g t ( β ) , β ∗ ⟩ + 2 δ ρ t ∥ g t ( β ) ∥ 2 F t ] ] .
Then
E [ M t ( ρ ) ∣ F t ] = 0 , \mathbb{E}[M_t^{(\rho)}\mid\mathcal{F}_t]=0, E [ M t ( ρ ) ∣ F t ] = 0 ,
and
ρ t + 1 = ρ t + δ F t ( ρ ) + δ M t ( ρ ) + R t ( ρ ) . \rho_{t+1}=\rho_t+\delta F_t^{(\rho)}+\delta M_t^{(\rho)}+R_t^{(\rho)}. ρ t + 1 = ρ t + δ F t ( ρ ) + δ M t ( ρ ) + R t ( ρ ) .
Thus
Δ N t + 1 ( ρ ) : = δ M t ( ρ ) + R t ( ρ ) . \Delta N_{t+1}^{(\rho)}:=\delta M_t^{(\rho)}+R_t^{(\rho)}. Δ N t + 1 ( ρ ) := δ M t ( ρ ) + R t ( ρ ) .
It remains to lower-bound F t ( ρ ) F_t^{(\rho)} F t ( ρ ) . By definition,
F t ( ρ ) = − E [ ⟨ g t ( β ) , β ∗ ⟩ ∣ F t ] − δ 2 ρ t E [ ∥ g t ( β ) ∥ 2 ∣ F t ] . F_t^{(\rho)}=-\mathbb{E}[\langle g_t^{(\beta)},\beta^*\rangle\mid\mathcal{F}_t]-\frac{\delta}{2}\rho_t\mathbb{E}[\lVert g_t^{(\beta)}\rVert^2\mid\mathcal{F}_t]. F t ( ρ ) = − E [⟨ g t ( β ) , β ∗ ⟩ ∣ F t ] − 2 δ ρ t E [∥ g t ( β ) ∥ 2 ∣ F t ] .
The first term is the population spherical-gradient drift:
− E [ ⟨ g t ( β ) , β ∗ ⟩ ∣ F t ] = − ⟨ ∇ S , β L ( θ t , β t ) , β ∗ ⟩ . -\mathbb{E}[\langle g_t^{(\beta)},\beta^*\rangle\mid\mathcal{F}_t]=-\langle\nabla_{S,\beta}L(\theta_t,\beta_t),\beta^*\rangle. − E [⟨ g t ( β ) , β ∗ ⟩ ∣ F t ] = − ⟨ ∇ S , β L ( θ t , β t ) , β ∗ ⟩ .
Since ρ t , χ t ≥ 0 \rho_t,\chi_t\ge0 ρ t , χ t ≥ 0 , Lemma C.3 implies
− ⟨ ∇ S , β L ( θ t , β t ) , β ∗ ⟩ ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t . -\langle\nabla_{S,\beta}L(\theta_t,\beta_t),\beta^*\rangle\ge a_\rho\chi_t(1-\rho_t^2)-b_\rho\eta_t. − ⟨ ∇ S , β L ( θ t , β t ) , β ∗ ⟩ ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t .
For the normalization correction, use ∣ ρ t ∣ ≤ 1 |\rho_t|\le1 ∣ ρ t ∣ ≤ 1 and the conditional second-moment gradient bound:
− δ 2 ρ t E [ ∥ g t ( β ) ∥ 2 ∣ F t ] ≥ − δ 2 E [ ∥ g t ( β ) ∥ 2 ∣ F t ] ≥ − C δ d . -\frac{\delta}{2}\rho_t\mathbb{E}[\lVert g_t^{(\beta)}\rVert^2\mid\mathcal{F}_t]\ge-\frac{\delta}{2}\mathbb{E}[\lVert g_t^{(\beta)}\rVert^2\mid\mathcal{F}_t]\ge-C\delta d. − 2 δ ρ t E [∥ g t ( β ) ∥ 2 ∣ F t ] ≥ − 2 δ E [∥ g t ( β ) ∥ 2 ∣ F t ] ≥ − C δ d .
Combining the two estimates gives
F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t − C δ d . F_t^{(\rho)}\ge a_\rho\chi_t(1-\rho_t^2)-b_\rho\eta_t-C\delta d. F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − b ρ η t − C δ d .
The χ \chi χ -recursion follows from the same argument applied to
θ t + 1 = θ t − δ g t ( θ ) ∥ θ t − δ g t ( θ ) ∥ . \theta_{t+1}=\frac{\theta_t-\delta g_t^{(\theta)}}{\lVert\theta_t-\delta g_t^{(\theta)}\rVert}. θ t + 1 = ∥ θ t − δ g t ( θ ) ∥ θ t − δ g t ( θ ) .
Thus
χ t + 1 = χ t + δ F t ( χ ) + Δ N t + 1 ( χ ) , \chi_{t+1}=\chi_t+\delta F_t^{(\chi)}+\Delta N_{t+1}^{(\chi)}, χ t + 1 = χ t + δ F t ( χ ) + Δ N t + 1 ( χ ) ,
with
F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t − C δ d . F_t^{(\chi)}\ge a_\chi\rho_t(1-\chi_t^2)-b_\chi\eta_t-C\delta d. F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − b χ η t − C δ d .
Finally, on E i n E_{\mathrm{in}} E in ,
throughout the entry window. Increasing C C C to absorb b ρ , b χ b_{\rho}, b_{\chi} b ρ , b χ , the two drift bounds become
F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − C ( η ˉ d + δ d ) , F_t^{(\rho)} \ge a_{\rho}\chi_t(1-\rho_t^2)-C(\bar{\eta}_d+\delta d), F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − C ( η ˉ d + δ d ) ,
and
F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − C ( η ˉ d + δ d ) . F_t^{(\chi)} \ge a_{\chi}\rho_t(1-\chi_t^2)-C(\bar{\eta}_d+\delta d). F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − C ( η ˉ d + δ d ) .
This proves the stated bounds with
B d : = C ( η ˉ d + δ d ) . B_d := C(\bar{\eta}_d+\delta d). B d := C ( η ˉ d + δ d ) .
Since Lemma D.19 gives η ˉ d = O ( log d / d ) \bar{\eta}_d=O(\log d/\sqrt{d}) η ˉ d = O ( log d / d ) for fixed or polynomially small ν i n \nu_{\mathrm{in}} ν in , and since δ = c l o c / d \delta=c_{\mathrm{loc}}/d δ = c loc / d , we obtain
B d ≤ C ( log d d + c l o c ) . B_d \le C\left(\frac{\log d}{\sqrt{d}}+c_{\mathrm{loc}}\right). B d ≤ C ( d log d + c loc ) .
The proof is complete.
Proposition D.21 (Discrete entry into the local product basin). Let r l o c r_{\mathrm{loc}} r loc be the radius from Lemma C.14. After possibly shrinking r l o c r_{\mathrm{loc}} r loc , assume 0 < r l o c < π 0<r_{\mathrm{loc}}<\pi 0 < r loc < π . Define
s b : = cos ( r l o c / 8 ) , s + : = 1 + s b 2 . s_b:=\cos(r_{\mathrm{loc}}/8),\qquad s_+:=\frac{1+s_b}{2}. s b := cos ( r loc /8 ) , s + := 2 1 + s b . Apply Lemma D.19 with a constant C i n C_{\mathrm{in}} C in at least as large as the fixed constant C e n t C_{\mathrm{ent}} C ent constructed in the proof below.
Then there exist constants c l o c , 0 > 0 c_{\mathrm{loc},0}>0 c loc , 0 > 0 , d 0 < ∞ d_0<\infty d 0 < ∞ , and C e n t < ∞ C_{\mathrm{ent}}<\infty C ent < ∞ , depending only on r l o c r_{\mathrm{loc}} r loc , c e s c c_{\mathrm{esc}} c esc , and the model constants, such that the following holds. If
δ l o c = c l o c d , 0 < c l o c ≤ c l o c , 0 , d ≥ d 0 , \delta_{\mathrm{loc}}=\frac{c_{\mathrm{loc}}}{d},\qquad0<c_{\mathrm{loc}}\le c_{\mathrm{loc},0},\qquad d\ge d_0, δ loc = d c loc , 0 < c loc ≤ c loc , 0 , d ≥ d 0 , then on the event E i n E_{\mathrm{in}} E in from Lemma D.19, there exists a time t 1 ≤ t e s c + C e n t δ l o c − 1 t_1\le t_{\mathrm{esc}}+C_{\mathrm{ent}}\delta_{\mathrm{loc}}^{-1} t 1 ≤ t esc + C ent δ loc − 1 such that
ρ t 1 ≥ s b , χ t 1 ≥ s b . \rho_{t_1}\ge s_b,\qquad\chi_{t_1}\ge s_b. ρ t 1 ≥ s b , χ t 1 ≥ s b . Consequently,
Q t 1 = 1 2 dist S ( θ t 1 , θ ∗ ) 2 + 1 2 dist S ( β t 1 , β ∗ ) 2 ≤ r l o c 2 64 ≤ r l o c 2 32 . Q_{t_1}=\frac{1}{2}\operatorname{dist}_{S}(\theta_{t_1},\theta^*)^2+\frac{1}{2}\operatorname{dist}_{S}(\beta_{t_1},\beta^*)^2\le\frac{r_{\mathrm{loc}}^2}{64}\le\frac{r_{\mathrm{loc}}^2}{32}. Q t 1 = 2 1 dist S ( θ t 1 , θ ∗ ) 2 + 2 1 dist S ( β t 1 , β ∗ ) 2 ≤ 64 r loc 2 ≤ 32 r loc 2 .
Proof. Write
t 0 : = t e s c , δ : = δ l o c , c − : = c e s c 2 . t_0:=t_{\mathrm{esc}},\qquad\delta:=\delta_{\mathrm{loc}},\qquad c_-:=\frac{c_{\mathrm{esc}}}{2}. t 0 := t esc , δ := δ loc , c − := 2 c esc .
We work deterministically on the event E i n E_{\mathrm{in}} E in , together with the weak recovery event at t e s c t_{\mathrm{esc}} t esc . Thus
ρ t 0 ≥ c e s c , χ t 0 ≥ c e s c , \rho_{t_0}\ge c_{\mathrm{esc}},\qquad\chi_{t_0}\ge c_{\mathrm{esc}}, ρ t 0 ≥ c esc , χ t 0 ≥ c esc ,
and, throughout the entry window,
η t ≤ η ˉ d , sup t 0 ≤ s ≤ u ≤ t 0 + C i n δ − 1 ∣ ∑ k = s u − 1 Δ N k + 1 ( j ) ∣ ≤ Λ d , j ∈ { ρ , χ } . \eta_t\le\bar{\eta}_d,\qquad
\sup_{t_0\le s\le u\le t_0+C_{\mathrm{in}}\delta^{-1}}
\left|\sum_{k=s}^{u-1}\Delta N_{k+1}^{(j)}\right|
\le\Lambda_d,\qquad j\in\{\rho,\chi\}. η t ≤ η ˉ d , t 0 ≤ s ≤ u ≤ t 0 + C in δ − 1 sup k = s ∑ u − 1 Δ N k + 1 ( j ) ≤ Λ d , j ∈ { ρ , χ } .
By Lemma D.20, on E i n E_{\mathrm{in}} E in ,
F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − B d , F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − B d , F_t^{(\rho)}\ge a_{\rho}\chi_t(1-\rho_t^2)-B_d,\qquad
F_t^{(\chi)}\ge a_{\chi}\rho_t(1-\chi_t^2)-B_d, F t ( ρ ) ≥ a ρ χ t ( 1 − ρ t 2 ) − B d , F t ( χ ) ≥ a χ ρ t ( 1 − χ t 2 ) − B d ,
where
B d : = C ( η ˉ d + δ d ) . B_d:=C(\bar{\eta}_d+\delta d). B d := C ( η ˉ d + δ d ) .
Since
η ˉ d = O ( log d d ) , δ d = c l o c , \bar{\eta}_d=O\left(\frac{\log d}{\sqrt{d}}\right),\qquad\delta d=c_{\mathrm{loc}}, η ˉ d = O ( d log d ) , δ d = c loc ,
we may choose c l o c > 0 c_{\mathrm{loc}}>0 c loc > 0 sufficiently small and then d d d sufficiently large so that all smallness conditions below hold. Define
m + : = 1 − s + 2 > 0 , m_+ := 1-s_+^2>0, m + := 1 − s + 2 > 0 ,
and set
γ ρ : = 1 2 a ρ c − m + , γ χ : = 1 2 a χ c − m + . \gamma_\rho:= \frac{1}{2}a_\rho c_-m_+,\qquad\gamma_\chi:= \frac{1}{2}a_\chi c_-m_+. γ ρ := 2 1 a ρ c − m + , γ χ := 2 1 a χ c − m + .
These are fixed positive constants independent of d d d . Also define
α ∗ : = 1 4 min { c e s c , s + − s b , γ ρ , γ χ } > 0. \alpha_* := \frac{1}{4}\min\{c_{\mathrm{esc}},s_+-s_b,\gamma_\rho,\gamma_\chi\}>0. α ∗ := 4 1 min { c esc , s + − s b , γ ρ , γ χ } > 0.
Let
T ρ : = ( s + − c e s c ) + + α ∗ γ ρ , T χ : = ( s + − c e s c ) + + α ∗ γ χ , T_\rho:= \frac{(s_+-c_{\mathrm{esc}})_++\alpha_*}{\gamma_\rho},\qquad T_\chi:= \frac{(s_+-c_{\mathrm{esc}})_++\alpha_*}{\gamma_\chi}, T ρ := γ ρ ( s + − c esc ) + + α ∗ , T χ := γ χ ( s + − c esc ) + + α ∗ ,
and choose
C e n t : = 2 + max { T ρ , T χ } . C_{\mathrm{ent}}:=2+\max\{T_\rho,T_\chi\}. C ent := 2 + max { T ρ , T χ } .
We choose c l o c c_{\mathrm{loc}} c loc small enough and then d d d large enough so that
B d ≤ min { γ ρ , γ χ } , (202) B_d\leq\min\{\gamma_\rho,\gamma_\chi\}, \tag*{(202)} B d ≤ min { γ ρ , γ χ } , (202)
Λ d ≤ α ∗ , (203) \Lambda_d\leq\alpha_*, \tag*{(203)} Λ d ≤ α ∗ , (203)
C e n t B d + Λ d ≤ α ∗ . (204) C_{\mathrm{ent}}B_d+\Lambda_d\leq\alpha_*. \tag*{(204)} C ent B d + Λ d ≤ α ∗ . (204)
These conditions are consistent because B d ≤ C ( log d / d + c l o c ) B_d\leq C(\log d/\sqrt{d}+c_{\mathrm{loc}}) B d ≤ C ( log d / d + c loc ) and Λ d = o d ( 1 ) \Lambda_d=o_d(1) Λ d = o d ( 1 ) .
Let
N e n t : = ⌈ max { T ρ , T χ } δ − 1 ⌉ . N_{\mathrm{ent}}:=\left\lceil\max\{T_\rho,T_\chi\}\delta^{-1}\right\rceil. N ent := ⌈ max { T ρ , T χ } δ − 1 ⌉ .
Since δ ≤ 1 \delta\leq1 δ ≤ 1 , we have
N e n t δ ≤ C e n t . N_{\mathrm{ent}}\delta\leq C_{\mathrm{ent}}. N ent δ ≤ C ent .
By the assumption C i n ≥ C e n t C_{\mathrm{in}}\geq C_{\mathrm{ent}} C in ≥ C ent , all estimates from E i n E_{\mathrm{in}} E in hold on [ t 0 , t 0 + N e n t ] [t_0,t_0+N_{\mathrm{ent}}] [ t 0 , t 0 + N ent ] .
Step 1: the overlaps stay above the lower barrier.
We claim that
ρ t ≥ c − , χ t ≥ c − , t 0 ≤ t ≤ t 0 + N e n t . \rho_t\geq c_-,\qquad\chi_t\geq c_-,\qquad t_0\leq t\leq t_0+N_{\mathrm{ent}}. ρ t ≥ c − , χ t ≥ c − , t 0 ≤ t ≤ t 0 + N ent .
Suppose not, and define the first lower-barrier exit time
τ − : = inf { t ∈ [ t 0 , t 0 + N e n t ] : min { ρ t , χ t } < c − } . \tau_-:=\inf\{t\in[t_0,t_0+N_{\mathrm{ent}}]:\min\{\rho_t,\chi_t\}<c_-\}. τ − := inf { t ∈ [ t 0 , t 0 + N ent ] : min { ρ t , χ t } < c − } .
Since ρ t 0 , χ t 0 ≥ c e s c > c − \rho_{t_0},\chi_{t_0}\geq c_{\mathrm{esc}}>c_- ρ t 0 , χ t 0 ≥ c esc > c − , one has τ − > t 0 \tau_->t_0 τ − > t 0 if τ − < ∞ \tau_-<\infty τ − < ∞ .
For every k < τ − k<\tau_- k < τ − , both overlaps are at least c − c_- c − , and hence nonnegative. Therefore the crude drift bounds
F k ( ρ ) ≥ − B d , F k ( χ ) ≥ − B d F_k^{(\rho)}\geq-B_d,\qquad F_k^{(\chi)}\geq-B_d F k ( ρ ) ≥ − B d , F k ( χ ) ≥ − B d
hold. Summing the ρ \rho ρ -recursion from t 0 t_0 t 0 to τ − \tau_- τ − , and using the cumulative noise bound from E i n E_{\mathrm{in}} E in , gives
ρ τ − ≥ ρ t 0 − ( τ − − t 0 ) δ B d − Λ d . \rho_{\tau_-}\geq\rho_{t_0}-(\tau_--t_0)\delta B_d-\Lambda_d. ρ τ − ≥ ρ t 0 − ( τ − − t 0 ) δ B d − Λ d .
Since ( τ − − t 0 ) δ ≤ C e n t (\tau_--t_0)\delta\leq C_{\mathrm{ent}} ( τ − − t 0 ) δ ≤ C ent , (196) implies
ρ τ − ≥ c e s c − C e n t B d − Λ d ≥ c e s c − α ∗ ≥ 3 4 c e s c > c − . \rho_{\tau_-}\geq c_{\mathrm{esc}}-C_{\mathrm{ent}}B_d-\Lambda_d\geq c_{\mathrm{esc}}-\alpha_*\geq\frac{3}{4}c_{\mathrm{esc}}>c_-. ρ τ − ≥ c esc − C ent B d − Λ d ≥ c esc − α ∗ ≥ 4 3 c esc > c − .
The same argument gives χ τ − > c − \chi_{\tau_-}>c_- χ τ − > c − . This contradicts the definition of τ − \tau_- τ − . Hence no such exit occurs.
Step 2: each overlap reaches the buffer level s + s_+ s + .
First consider ρ t \rho_t ρ t . If ρ t 0 ≥ s + \rho_{t_0}\geq s_+ ρ t 0 ≥ s + , set τ ρ : = t 0 \tau_\rho:=t_0 τ ρ := t 0 . Otherwise, suppose for contradiction that
ρ t < s + for all t 0 ≤ t ≤ t 0 + ⌈ T ρ δ − 1 ⌉ . \rho_t<s_+\qquad\text{for all }t_0\leq t\leq t_0+\left\lceil T_\rho\delta^{-1}\right\rceil. ρ t < s + for all t 0 ≤ t ≤ t 0 + ⌈ T ρ δ − 1 ⌉ .
By Step 1, χ t ≥ c − \chi_t\geq c_- χ t ≥ c − on this interval. Hence, for every such k k k ,
F k ( ρ ) ≥ a ρ c − ( 1 − ρ k 2 ) − B d . F_k^{(\rho)}\geq a_\rho c_-(1-\rho_k^2)-B_d. F k ( ρ ) ≥ a ρ c − ( 1 − ρ k 2 ) − B d .
Since ρ k < s + \rho_k < s_+ ρ k < s + , we have 1 − ρ k 2 ≥ 1 − s + 2 = m + 1-\rho_k^2 \ge1-s_+^2 = m_+ 1 − ρ k 2 ≥ 1 − s + 2 = m + , and therefore
F k ( ρ ) ≥ a ρ c − m + − B d = 2 γ ρ − B d ≥ γ ρ F_k^{(\rho)} \ge a_\rho c_{-}m_+ - B_d = 2\gamma_\rho- B_d \ge\gamma_\rho F k ( ρ ) ≥ a ρ c − m + − B d = 2 γ ρ − B d ≥ γ ρ
by (202) . Summing the recursion gives
ρ t 0 + ⌈ T ρ δ − 1 ⌉ ≥ ρ t 0 + γ ρ T ρ − Λ d . \rho_{t_0+\lceil T_\rho\delta^{-1}\rceil} \ge\rho_{t_0}+\gamma_\rho T_\rho-\Lambda_d. ρ t 0 + ⌈ T ρ δ − 1 ⌉ ≥ ρ t 0 + γ ρ T ρ − Λ d .
Using ρ t 0 ≥ c e s c \rho_{t_0} \ge c_{\mathrm{esc}} ρ t 0 ≥ c esc , the definition of T ρ T_\rho T ρ , and (203) ,
ρ t 0 + ⌈ T ρ δ − 1 ⌉ ≥ c e s c + ( s + − c e s c ) + + α ∗ − Λ d ≥ s + . \rho_{t_0+\lceil T_\rho\delta^{-1}\rceil} \ge c_{\mathrm{esc}}+(s_+-c_{\mathrm{esc}})_+ + \alpha_*-\Lambda_d \ge s_+. ρ t 0 + ⌈ T ρ δ − 1 ⌉ ≥ c esc + ( s + − c esc ) + + α ∗ − Λ d ≥ s + .
This contradicts the assumption that ρ t < s + \rho_t < s_+ ρ t < s + throughout the interval. Thus there exists
τ ρ ≤ t 0 + ⌈ T ρ δ − 1 ⌉ \tau_\rho\le t_0+\lceil T_\rho\delta^{-1}\rceil τ ρ ≤ t 0 + ⌈ T ρ δ − 1 ⌉
such that
ρ τ ρ ≥ s + . \rho_{\tau_\rho} \ge s_+. ρ τ ρ ≥ s + .
The proof for χ t \chi_t χ t is identical. If χ t 0 ≥ s + \chi_{t_0} \ge s_+ χ t 0 ≥ s + , set τ χ : = t 0 \tau_\chi:= t_0 τ χ := t 0 . Otherwise, using Step 1 and the lower bound for F k ( χ ) F_k^{(\chi)} F k ( χ ) , one obtains a time
τ χ ≤ t 0 + ⌈ T χ δ − 1 ⌉ \tau_\chi\le t_0+\lceil T_\chi\delta^{-1}\rceil τ χ ≤ t 0 + ⌈ T χ δ − 1 ⌉
such that
χ τ χ ≥ s + . \chi_{\tau_\chi} \ge s_+. χ τ χ ≥ s + .
Step 3: retain both overlaps above s b s_b s b .
Define
t 1 : = max { τ ρ , τ χ } . t_1 := \max\{\tau_\rho,\tau_\chi\}. t 1 := max { τ ρ , τ χ } .
Then
t 1 ≤ t 0 + N e n t ≤ t e s c + C e n t δ − 1 . t_1 \le t_0+N_{\mathrm{ent}} \le t_{\mathrm{esc}}+C_{\mathrm{ent}}\delta^{-1}. t 1 ≤ t 0 + N ent ≤ t esc + C ent δ − 1 .
We show that both overlaps are at least s b s_b s b at time t 1 t_1 t 1 . Since ρ τ ρ ≥ s + \rho_{\tau_\rho} \ge s_+ ρ τ ρ ≥ s + , and since Step 1 gives the lower barrier on the whole window, the crude drift bound F k ( ρ ) ≥ − B d F_k^{(\rho)} \ge-B_d F k ( ρ ) ≥ − B d holds on [ τ ρ , t 1 ] [\tau_\rho,t_1] [ τ ρ , t 1 ] . Therefore
ρ t 1 ≥ ρ τ ρ − ( t 1 − τ ρ ) δ B d − Λ d . \rho_{t_1} \ge\rho_{\tau_\rho}-(t_1-\tau_\rho)\delta B_d-\Lambda_d. ρ t 1 ≥ ρ τ ρ − ( t 1 − τ ρ ) δ B d − Λ d .
As ( t 1 − τ ρ ) δ ≤ C e n t (t_1-\tau_\rho)\delta\le C_{\mathrm{ent}} ( t 1 − τ ρ ) δ ≤ C ent , (204) gives
ρ t 1 ≥ s + − C e n t B d − Λ d ≥ s + − α ∗ . \rho_{t_1} \ge s_+-C_{\mathrm{ent}}B_d-\Lambda_d \ge s_+-\alpha_*. ρ t 1 ≥ s + − C ent B d − Λ d ≥ s + − α ∗ .
By the definition of α ∗ \alpha_* α ∗ ,
α ∗ ≤ 1 4 ( s + − s b ) , \alpha_* \le\frac{1}{4}(s_+-s_b), α ∗ ≤ 4 1 ( s + − s b ) ,
so
ρ t 1 ≥ s b . \rho_{t_1} \ge s_b. ρ t 1 ≥ s b .
The same argument gives
χ t 1 ≥ s b . \chi_{t_1} \ge s_b. χ t 1 ≥ s b .
Step 4: convert overlap entry into product-distance entry.
Since
ρ t 1 = ⟨ β t 1 , β ∗ ⟩ ≥ s b = cos ( r l o c / 8 ) , \rho_{t_1}=\langle\beta_{t_1},\beta^*\rangle\ge s_b=\cos(r_{\mathrm{loc}}/8), ρ t 1 = ⟨ β t 1 , β ∗ ⟩ ≥ s b = cos ( r loc /8 ) ,
and 0 < r l o c < π 0<r_{\mathrm{loc}}<\pi 0 < r loc < π ,
dist S ( β t 1 , β ∗ ) = arccos ( ρ t 1 ) ≤ r l o c 8 . \operatorname{dist}_{\mathbb{S}}(\beta_{t_1},\beta^*)=\arccos(\rho_{t_1})\le\frac{r_{\mathrm{loc}}}{8}. dist S ( β t 1 , β ∗ ) = arccos ( ρ t 1 ) ≤ 8 r loc .
Similarly,
dist S ( θ t 1 , θ ∗ ) ≤ r l o c 8 . \operatorname{dist}_{\mathbb{S}}(\theta_{t_1},\theta^*)\le\frac{r_{\mathrm{loc}}}{8}. dist S ( θ t 1 , θ ∗ ) ≤ 8 r loc .
Therefore
Q t 1 = 1 2 dist S ( θ t 1 , θ ∗ ) 2 + 1 2 dist S ( β t 1 , β ∗ ) 2 ≤ 1 2 ( r l o c 8 ) 2 + 1 2 ( r l o c 8 ) 2 = r l o c 2 64 . Q_{t_1}=\frac{1}{2}\operatorname{dist}_{\mathbb{S}}(\theta_{t_1},\theta^*)^2+\frac{1}{2}\operatorname{dist}_{\mathbb{S}}(\beta_{t_1},\beta^*)^2\le\frac{1}{2}\left(\frac{r_{\mathrm{loc}}}{8}\right)^2+\frac{1}{2}\left(\frac{r_{\mathrm{loc}}}{8}\right)^2=\frac{r_{\mathrm{loc}}^2}{64}. Q t 1 = 2 1 dist S ( θ t 1 , θ ∗ ) 2 + 2 1 dist S ( β t 1 , β ∗ ) 2 ≤ 2 1 ( 8 r loc ) 2 + 2 1 ( 8 r loc ) 2 = 64 r loc 2 .
This proves the proposition.
Deriving Q t Q_t Q t -iteration and its controlling bounds.
Lemma D.22 (One-step local distance recursion). Let
y t = ( θ t , β t ) , y ∗ = ( θ ∗ , β ∗ ) , g t = ( g t ( θ ) , g t ( β ) ) , y_t=(\theta_t,\beta_t),\qquad y^*=(\theta^*,\beta^*),\qquad g_t=(g_t^{(\theta)},g_t^{(\beta)}), y t = ( θ t , β t ) , y ∗ = ( θ ∗ , β ∗ ) , g t = ( g t ( θ ) , g t ( β ) ) , and define
Q t : = 1 2 dist M ( y t , y ∗ ) 2 = 1 2 dist S ( θ t , θ ∗ ) 2 + 1 2 dist S ( β t , β ∗ ) 2 . Q_t:=\frac{1}{2}\operatorname{dist}_{\mathcal{M}}(y_t,y^*)^2
=\frac{1}{2}\operatorname{dist}_{\mathbb{S}}(\theta_t,\theta^*)^2
+\frac{1}{2}\operatorname{dist}_{\mathbb{S}}(\beta_t,\beta^*)^2. Q t := 2 1 dist M ( y t , y ∗ ) 2 = 2 1 dist S ( θ t , θ ∗ ) 2 + 2 1 dist S ( β t , β ∗ ) 2 . Assume r l o c < π / 4 r_{\mathrm{loc}}<\pi/4 r loc < π /4 . There exists a constant C r e t < ∞ C_{\mathrm{ret}}<\infty C ret < ∞ , depending only on r l o c r_{\mathrm{loc}} r loc , such that whenever y t ∈ U r l o c y_t\in U_{r_{\mathrm{loc}}} y t ∈ U r loc , the normalized SGD step satisfies
Q t + 1 ≤ Q t + δ ⟨ g t , L t ⟩ + C r e t δ 2 ∥ g t ∥ 2 , Q_{t+1}\leq Q_t+\delta\langle g_t,\mathcal{L}_t\rangle+C_{\mathrm{ret}}\delta^2\lVert g_t\rVert^2, Q t + 1 ≤ Q t + δ ⟨ g t , L t ⟩ + C ret δ 2 ∥ g t ∥ 2 , where
L t : = ( Log θ t ( θ ∗ ) , Log β t ( β ∗ ) ) . \mathcal{L}_t:=\left(\operatorname{Log}_{\theta_t}(\theta^*),\operatorname{Log}_{\beta_t}(\beta^*)\right). L t := ( Log θ t ( θ ∗ ) , Log β t ( β ∗ ) ) . Consequently, if
g ˉ t : = E [ g t ∣ F t ] , Ξ t : = ⟨ g t − g ˉ t , L t ⟩ , G t : = ∥ g t ∥ 2 , \bar{g}_t:=\mathbb{E}[g_t\mid\mathcal{F}_t],\qquad\Xi_t:=\langle g_t-\bar{g}_t,\mathcal{L}_t\rangle,\qquad G_t:=\lVert g_t\rVert^2, g ˉ t := E [ g t ∣ F t ] , Ξ t := ⟨ g t − g ˉ t , L t ⟩ , G t := ∥ g t ∥ 2 , then
Q t + 1 ≤ Q t + δ ⟨ g ˉ t , L t ⟩ + δ Ξ t + C r e t δ 2 G t . (205) Q_{t+1}\leq Q_t+\delta\langle\bar{g}_t,\mathcal{L}_t\rangle+\delta\Xi_t+C_{\mathrm{ret}}\delta^2G_t. \tag*{(205)} Q t + 1 ≤ Q t + δ ⟨ g ˉ t , L t ⟩ + δ Ξ t + C ret δ 2 G t . (205)
Proof. We use the product metric on
M = S d − 1 × S d − 1 , \mathcal{M}=\mathbb{S}^{d-1}\times\mathbb{S}^{d-1}, M = S d − 1 × S d − 1 ,
namely
⟨ ( a θ , a β ) , ( b θ , b β ) ⟩ : = ⟨ a θ , b θ ⟩ + ⟨ a β , b β ⟩ , \langle(a_\theta,a_\beta),(b_\theta,b_\beta)\rangle
:=\langle a_\theta,b_\theta\rangle+\langle a_\beta,b_\beta\rangle, ⟨( a θ , a β ) , ( b θ , b β )⟩ := ⟨ a θ , b θ ⟩ + ⟨ a β , b β ⟩ ,
and
∥ ( a θ , a β ) ∥ 2 : = ∥ a θ ∥ 2 + ∥ a β ∥ 2 . \lVert(a_\theta,a_\beta)\rVert^2:=\lVert a_\theta\rVert^2+\lVert a_\beta\rVert^2. ∥( a θ , a β ) ∥ 2 := ∥ a θ ∥ 2 + ∥ a β ∥ 2 .
Fix t t t such that y t ∈ U r l o c y_t\in U_{r_{\mathrm{loc}}} y t ∈ U r loc . We prove the one-step estimate on one sphere and then add the two block estimates.
Let u ∈ S d − 1 u\in\mathbb{S}^{d-1} u ∈ S d − 1 , let v ∈ S d − 1 v\in\mathbb{S}^{d-1} v ∈ S d − 1 , and let h ∈ T u S d − 1 h\in T_u\mathbb{S}^{d-1} h ∈ T u S d − 1 . Consider the normalized spherical step
u ^ : = u − δ h ∥ u − δ h ∥ . \hat{u}:=\frac{u-\delta h}{\lVert u-\delta h\rVert}. u ^ := ∥ u − δ h ∥ u − δ h .
Since h ⊥ u h\perp u h ⊥ u ,
∥ u − δ h ∥ 2 = 1 + δ 2 ∥ h ∥ 2 . \lVert u-\delta h\rVert^2=1+\delta^2\lVert h\rVert^2. ∥ u − δ h ∥ 2 = 1 + δ 2 ∥ h ∥ 2 .
Define
τ ( h ) : = { arctan ( δ ∥ h ∥ ) ∥ h ∥ , h ≠ 0 , 0 , h = 0. \tau(h):=
\begin{cases}
\dfrac{\arctan(\delta\lVert h\rVert)}{\lVert h\rVert}, & h\ne0,\\
0, & h=0.
\end{cases} τ ( h ) := ⎩ ⎨ ⎧ ∥ h ∥ arctan ( δ ∥ h ∥) , 0 , h = 0 , h = 0.
Then
u ^ = Exp u ( − τ ( h ) h ) . \hat{u}=\operatorname{Exp}_u(-\tau(h)h). u ^ = Exp u ( − τ ( h ) h ) .
Indeed, when h ≠ 0 h\ne0 h = 0 , the normalized point lies on the great circle through u u u in direction − h / ∥ h ∥ -h/\lVert h\rVert − h / ∥ h ∥ , with angular displacement arctan ( δ ∥ h ∥ ) \arctan(\delta\lVert h\rVert) arctan ( δ ∥ h ∥) . The case h = 0 h=0 h = 0 is immediate.
Now set
q v ( u ) : = 1 2 dist S ( u , v ) 2 . q_v(u):=\frac{1}{2}\operatorname{dist}_{\mathbb{S}}(u,v)^2. q v ( u ) := 2 1 dist S ( u , v ) 2 .
Since u u u lies in the local cap around v v v , and since the normalized step has spherical length at most π / 2 \pi/2 π /2 , after shrinking r l o c r_{\mathrm{loc}} r loc if necessary the geodesic segment used above remains in a compact subset away from the cut locus of v v v . Therefore q v q_v q v has uniformly bounded Hessian on this region:
∥ Hess S q v ∥ ≤ C s p h . \lVert\operatorname{Hess}_{\mathbb{S}}q_v\rVert\leq C_{\mathrm{sph}}. ∥ Hess S q v ∥ ≤ C sph .
Taylor expansion along
s ↦ Exp u ( − s τ ( h ) h ) s\mapsto\operatorname{Exp}_u(-s\tau(h)h) s ↦ Exp u ( − s τ ( h ) h )
gives
q v ( u ^ ) ≤ q v ( u ) + ⟨ ∇ S q v ( u ) , − τ ( h ) h ⟩ + C s p h 2 τ ( h ) 2 ∥ h ∥ 2 . q_v(\widehat{u}) \le q_v(u) + \langle\nabla_S q_v(u),-\tau(h)h\rangle+ \frac{C_{\mathrm{sph}}}{2}\tau(h)^2\lVert h\rVert^2. q v ( u ) ≤ q v ( u ) + ⟨ ∇ S q v ( u ) , − τ ( h ) h ⟩ + 2 C sph τ ( h ) 2 ∥ h ∥ 2 .
Since
∇ S q v ( u ) = − Log u ( v ) , \nabla_S q_v(u)=-\operatorname{Log}_u(v), ∇ S q v ( u ) = − Log u ( v ) ,
we get
q v ( u ^ ) ≤ q v ( u ) + τ ( h ) ⟨ h , Log u ( v ) ⟩ + C s p h 2 τ ( h ) 2 ∥ h ∥ 2 . q_v(\widehat{u}) \le q_v(u) + \tau(h)\langle h,\operatorname{Log}_u(v)\rangle+ \frac{C_{\mathrm{sph}}}{2}\tau(h)^2\lVert h\rVert^2. q v ( u ) ≤ q v ( u ) + τ ( h ) ⟨ h , Log u ( v )⟩ + 2 C sph τ ( h ) 2 ∥ h ∥ 2 .
We compare τ ( h ) \tau(h) τ ( h ) with δ \delta δ . Since
0 ≤ τ ( h ) ≤ δ , 0\le\tau(h)\le\delta, 0 ≤ τ ( h ) ≤ δ ,
and, writing x = δ ∥ h ∥ x=\delta\lVert h\rVert x = δ ∥ h ∥ ,
0 ≤ δ ∥ h ∥ − arctan ( δ ∥ h ∥ ) = x − arctan x ≤ C x 2 , 0\le\delta\lVert h\rVert-\arctan(\delta\lVert h\rVert)=x-\arctan x\le Cx^2, 0 ≤ δ ∥ h ∥ − arctan ( δ ∥ h ∥) = x − arctan x ≤ C x 2 ,
we have
∣ δ − τ ( h ) ∣ ∥ h ∥ ≤ C δ 2 ∥ h ∥ 2 . \lvert\delta-\tau(h)\rvert\lVert h\rVert\le C\delta^2\lVert h\rVert^2. ∣ δ − τ ( h )∣ ∥ h ∥ ≤ C δ 2 ∥ h ∥ 2 .
Also u u u is within distance r l o c r_{\mathrm{loc}} r loc of v v v , so
∥ Log u ( v ) ∥ = dist S ( u , v ) ≤ r l o c . \lVert\operatorname{Log}_u(v)\rVert=\operatorname{dist}_S(u,v)\le r_{\mathrm{loc}}. ∥ Log u ( v )∥ = dist S ( u , v ) ≤ r loc .
Therefore
τ ( h ) ⟨ h , Log u ( v ) ⟩ ≤ δ ⟨ h , Log u ( v ) ⟩ + C r l o c δ 2 ∥ h ∥ 2 . \tau(h)\langle h,\operatorname{Log}_u(v)\rangle\le\delta\langle h,\operatorname{Log}_u(v)\rangle+Cr_{\mathrm{loc}}\delta^2\lVert h\rVert^2. τ ( h ) ⟨ h , Log u ( v )⟩ ≤ δ ⟨ h , Log u ( v )⟩ + C r loc δ 2 ∥ h ∥ 2 .
Moreover,
τ ( h ) 2 ∥ h ∥ 2 ≤ δ 2 ∥ h ∥ 2 . \tau(h)^2\lVert h\rVert^2\le\delta^2\lVert h\rVert^2. τ ( h ) 2 ∥ h ∥ 2 ≤ δ 2 ∥ h ∥ 2 .
Thus
q v ( u ^ ) ≤ q v ( u ) + δ ⟨ h , Log u ( v ) ⟩ + C r e t ( 1 ) δ 2 ∥ h ∥ 2 , q_v(\widehat{u})\le q_v(u)+\delta\langle h,\operatorname{Log}_u(v)\rangle+C_{\mathrm{ret}}^{(1)}\delta^2\lVert h\rVert^2, q v ( u ) ≤ q v ( u ) + δ ⟨ h , Log u ( v )⟩ + C ret ( 1 ) δ 2 ∥ h ∥ 2 ,
where C r e t ( 1 ) C_{\mathrm{ret}}^{(1)} C ret ( 1 ) depends only on r l o c r_{\mathrm{loc}} r loc and C s p h C_{\mathrm{sph}} C sph .
Apply this estimate first with
u = θ t , v = θ ∗ , h = g t ( θ ) , u=\theta_t,\qquad v=\theta^*,\qquad h=g_t^{(\theta)}, u = θ t , v = θ ∗ , h = g t ( θ ) ,
and then with
u = β t , v = β ∗ , h = g t ( β ) . u=\beta_t,\qquad v=\beta^*,\qquad h=g_t^{(\beta)}. u = β t , v = β ∗ , h = g t ( β ) .
The normalized SGD updates are
θ t + 1 = θ t − δ g t ( θ ) ∥ θ t − δ g t ( θ ) ∥ , β t + 1 = β t − δ g t ( β ) ∥ β t − δ g t ( β ) ∥ . \theta_{t+1}=\frac{\theta_t-\delta g_t^{(\theta)}}{\lVert\theta_t-\delta g_t^{(\theta)}\rVert},\qquad
\beta_{t+1}=\frac{\beta_t-\delta g_t^{(\beta)}}{\lVert\beta_t-\delta g_t^{(\beta)}\rVert}. θ t + 1 = ∥ θ t − δ g t ( θ ) ∥ θ t − δ g t ( θ ) , β t + 1 = ∥ β t − δ g t ( β ) ∥ β t − δ g t ( β ) .
Hence
1 2 dist S ( θ t + 1 , θ ∗ ) 2 ≤ 1 2 dist S ( θ t , θ ∗ ) 2 + δ ⟨ g t ( θ ) , Log θ t ( θ ∗ ) ⟩ + C r e t ( 1 ) δ 2 ∥ g t ( θ ) ∥ 2 , \frac{1}{2}\operatorname{dist}_S(\theta_{t+1},\theta^*)^2\le\frac{1}{2}\operatorname{dist}_S(\theta_t,\theta^*)^2+\delta\langle g_t^{(\theta)},\operatorname{Log}_{\theta_t}(\theta^*)\rangle+C_{\mathrm{ret}}^{(1)}\delta^2\lVert g_t^{(\theta)}\rVert^2, 2 1 dist S ( θ t + 1 , θ ∗ ) 2 ≤ 2 1 dist S ( θ t , θ ∗ ) 2 + δ ⟨ g t ( θ ) , Log θ t ( θ ∗ )⟩ + C ret ( 1 ) δ 2 ∥ g t ( θ ) ∥ 2 ,
and
1 2 dist S ( β t + 1 , β ∗ ) 2 ≤ 1 2 dist S ( β t , β ∗ ) 2 + δ ⟨ g t ( β ) , Log β t ( β ∗ ) ⟩ + C r e t ( 1 ) δ 2 ∥ g t ( β ) ∥ 2 . \frac{1}{2}\operatorname{dist}_S(\beta_{t+1},\beta^*)^2\le\frac{1}{2}\operatorname{dist}_S(\beta_t,\beta^*)^2+\delta\langle g_t^{(\beta)},\operatorname{Log}_{\beta_t}(\beta^*)\rangle+C_{\mathrm{ret}}^{(1)}\delta^2\lVert g_t^{(\beta)}\rVert^2. 2 1 dist S ( β t + 1 , β ∗ ) 2 ≤ 2 1 dist S ( β t , β ∗ ) 2 + δ ⟨ g t ( β ) , Log β t ( β ∗ )⟩ + C ret ( 1 ) δ 2 ∥ g t ( β ) ∥ 2 .
Adding the two estimates gives
Q t + 1 ≤ Q t + δ ⟨ g t , L t ⟩ + C r e t δ 2 ∥ g t ∥ 2 , Q_{t+1}\le Q_t+\delta\langle g_t,L_t\rangle+C_{\mathrm{ret}}\delta^2\lVert g_t\rVert^2, Q t + 1 ≤ Q t + δ ⟨ g t , L t ⟩ + C ret δ 2 ∥ g t ∥ 2 ,
after increasing the constant C r e t C_{\mathrm{ret}} C ret .
Finally,
g t = g ˉ t + ( g t − g ˉ t ) , g_t=\bar{g}_t+(g_t-\bar{g}_t), g t = g ˉ t + ( g t − g ˉ t ) ,
so
⟨ g t , L t ⟩ = ⟨ g ˉ t , L t ⟩ + ⟨ g t − g ˉ t , L t ⟩ = ⟨ g ˉ t , L t ⟩ + Ξ t . \langle g_t,L_t\rangle=\langle\bar{g}_t,L_t\rangle+\langle g_t-\bar{g}_t,L_t\rangle=\langle\bar{g}_t,L_t\rangle+\Xi_t. ⟨ g t , L t ⟩ = ⟨ g ˉ t , L t ⟩ + ⟨ g t − g ˉ t , L t ⟩ = ⟨ g ˉ t , L t ⟩ + Ξ t .
With G t = ∥ g t ∥ 2 G_t=\lVert g_t\rVert^2 G t = ∥ g t ∥ 2 , this yields
Q t + 1 ≤ Q t + δ ⟨ g ˉ t , L t ⟩ + δ Ξ t + C r e t δ 2 G t , Q_{t+1}\le Q_t+\delta\langle\bar{g}_t,L_t\rangle+\delta\Xi_t+C_{\mathrm{ret}}\delta^2G_t, Q t + 1 ≤ Q t + δ ⟨ g ˉ t , L t ⟩ + δ Ξ t + C ret δ 2 G t ,
which proves (197).
Lemma D.23 (Sharp local self-quenching). Inside U r l o c U_{r_{\mathrm{loc}}} U r loc , there exists C s q > 0 C_{\mathrm{sq}}>0 C sq > 0 , independent of d d d , such that
E [ G t ∣ F t ] ≤ C s q d Q t , \mathbb{E}[G_t\mid\mathcal{F}_t]\le C_{\mathrm{sq}}dQ_t, E [ G t ∣ F t ] ≤ C sq d Q t , E [ Ξ t 2 ∣ F t ] ≤ C s q Q t 2 , \mathbb{E}[\Xi_t^2\mid\mathcal{F}_t]\le C_{\mathrm{sq}}Q_t^2, E [ Ξ t 2 ∣ F t ] ≤ C sq Q t 2 , and
E [ ( G t − E [ G t ∣ F t ] ) 2 ∣ F t ] ≤ C s q d 2 Q t 2 . \mathbb{E}\left[(G_t-\mathbb{E}[G_t\mid\mathcal{F}_t])^2\mid\mathcal{F}_t\right]\le C_{\mathrm{sq}}d^2Q_t^2. E [ ( G t − E [ G t ∣ F t ] ) 2 ∣ F t ] ≤ C sq d 2 Q t 2 . Proof. Condition on F t \mathcal{F}_t F t , so that ( θ t , β t ) (\theta_t,\beta_t) ( θ t , β t ) are deterministic. To lighten notation, write
θ : = θ t , β : = β t , \theta:=\theta_t,\qquad\beta:=\beta_t, θ := θ t , β := β t , and set
d θ : = dist S ( θ , θ ∗ ) , d β : = dist S ( β , β ∗ ) . d_\theta:=\operatorname{dist}_{\mathbb{S}}(\theta,\theta^*),\qquad d_\beta:=\operatorname{dist}_{\mathbb{S}}(\beta,\beta^*). d θ := dist S ( θ , θ ∗ ) , d β := dist S ( β , β ∗ ) . Since
Q t = 1 2 ( d θ 2 + d β 2 ) , Q_t=\frac{1}{2}\left(d_\theta^2+d_\beta^2\right), Q t = 2 1 ( d θ 2 + d β 2 ) , we have
d θ 2 + d β 2 = 2 Q t . d_\theta^2+d_\beta^2=2Q_t. d θ 2 + d β 2 = 2 Q t . Let
a : = Log θ ( θ ∗ ) ∈ T θ S d − 1 , b : = Log β ( β ∗ ) ∈ T β S d − 1 , a:=\operatorname{Log}_{\theta}(\theta^*)\in T_\theta\mathbb{S}^{d-1},\qquad b:=\operatorname{Log}_{\beta}(\beta^*)\in T_\beta\mathbb{S}^{d-1}, a := Log θ ( θ ∗ ) ∈ T θ S d − 1 , b := Log β ( β ∗ ) ∈ T β S d − 1 , so that
∥ a ∥ = d θ , ∥ b ∥ = d β . \lVert a\rVert=d_\theta,\qquad\lVert b\rVert=d_\beta. ∥ a ∥ = d θ , ∥ b ∥ = d β . Write the one-sample prediction as
f β , θ ( x ) = ψ ( s ) , s : = z β ( x ) ⋅ θ = ∑ i = 1 d θ i ϕ ( u i ) , u i : = x ( i ) ⋅ β , f_{\beta,\theta}(x)=\psi(s),\qquad s:=z_\beta(x)\cdot\theta=\sum_{i=1}^{d}\theta_i\phi(u_i),\qquad u_i:=x^{(i)}\cdot\beta, f β , θ ( x ) = ψ ( s ) , s := z β ( x ) ⋅ θ = i = 1 ∑ d θ i ϕ ( u i ) , u i := x ( i ) ⋅ β , and the teacher prediction as
f β ∗ , θ ∗ ( x ) = ψ ( s ∗ ) , s ∗ : = z β ∗ ( x ) ⋅ θ ∗ = ∑ i = 1 d θ i ∗ ϕ ( u i ∗ ) , u i ∗ : = x ( i ) ⋅ β ∗ . f_{\beta^*,\theta^*}(x)=\psi(s^*),\qquad s^*:=z_{\beta^*}(x)\cdot\theta^*=\sum_{i=1}^{d}\theta_i^*\phi(u_i^*),\qquad u_i^*:=x^{(i)}\cdot\beta^*. f β ∗ , θ ∗ ( x ) = ψ ( s ∗ ) , s ∗ := z β ∗ ( x ) ⋅ θ ∗ = i = 1 ∑ d θ i ∗ ϕ ( u i ∗ ) , u i ∗ := x ( i ) ⋅ β ∗ . Define the residual
R : = f β , θ ( x ) − f β ∗ , θ ∗ ( x ) = ψ ( s ) − ψ ( s ∗ ) . R:=f_{\beta,\theta}(x)-f_{\beta^*,\theta^*}(x)=\psi(s)-\psi(s^*). R := f β , θ ( x ) − f β ∗ , θ ∗ ( x ) = ψ ( s ) − ψ ( s ∗ ) . The spherical one-sample gradients are
g t ( θ ) = 2 R ψ ′ ( s ) P θ z β ( x ) , g_t^{(\theta)}=2R\psi'(s)P_\theta z_\beta(x), g t ( θ ) = 2 R ψ ′ ( s ) P θ z β ( x ) , and
g t ( β ) = 2 R ψ ′ ( s ) P β ( ∑ i = 1 d θ i ϕ ′ ( u i ) x ( i ) ) . g_t^{(\beta)}=2R\psi'(s)P_\beta\left(\sum_{i=1}^{d}\theta_i\phi'(u_i)x^{(i)}\right). g t ( β ) = 2 R ψ ′ ( s ) P β ( i = 1 ∑ d θ i ϕ ′ ( u i ) x ( i ) ) . Set
g ( θ ) : = g t ( θ ) , g ( β ) : = g t ( β ) , g t = ( g ( θ ) , g ( β ) ) , g^{(\theta)}:=g_t^{(\theta)},\qquad g^{(\beta)}:=g_t^{(\beta)},\qquad g_t=(g^{(\theta)},g^{(\beta)}), g ( θ ) := g t ( θ ) , g ( β ) := g t ( β ) , g t = ( g ( θ ) , g ( β ) ) , and
G t : = ∥ g t ∥ 2 = ∥ g ( θ ) ∥ 2 + ∥ g ( β ) ∥ 2 . G_t:=\lVert g_t\rVert^2=\lVert g^{(\theta)}\rVert^2+\lVert g^{(\beta)}\rVert^2. G t := ∥ g t ∥ 2 = ∥ g ( θ ) ∥ 2 + ∥ g ( β ) ∥ 2 . Also recall
Ξ t = ⟨ g t − g ˉ t , ( a , b ) ⟩ , g ˉ t : = E [ g t ∣ F t ] . \Xi_t=\langle g_t-\bar g_t,(a,b)\rangle,\qquad\bar g_t:=\mathbb{E}[g_t\mid\mathcal{F}_t]. Ξ t = ⟨ g t − g ˉ t , ( a , b )⟩ , g ˉ t := E [ g t ∣ F t ] . All moment estimates below follow from Assumption (A3) and Remark B.1.
residual self-quenching.
We claim that for m = 2 , 4 , 8 m=2,4,8 m = 2 , 4 , 8 ,
E [ ∣ R ∣ m ∣ F t ] ≤ C m Q t m / 2 . \mathbb{E}[|R|^m\mid\mathcal{F}_t]\le C_mQ_t^{m/2}. E [ ∣ R ∣ m ∣ F t ] ≤ C m Q t m /2 .
Since ψ ′ \psi' ψ ′ is bounded,
∣ R ∣ ≤ ∥ ψ ′ ∥ ∞ ∣ s − s ∗ ∣ . |R| \le\|\psi'\|_{\infty}|s-s^*|. ∣ R ∣ ≤ ∥ ψ ′ ∥ ∞ ∣ s − s ∗ ∣.
Decompose
s − s ∗ = A + B , s-s^* = A+B, s − s ∗ = A + B ,
where
A : = ∑ i = 1 d ( θ i − θ i ∗ ) ϕ ( u i ) , A := \sum_{i=1}^{d}(\theta_i-\theta_i^*)\phi(u_i), A := i = 1 ∑ d ( θ i − θ i ∗ ) ϕ ( u i ) ,
and
B : = ∑ i = 1 d θ i ∗ ( ϕ ( u i ) − ϕ ( u i ∗ ) ) . B := \sum_{i=1}^{d}\theta_i^*(\phi(u_i)-\phi(u_i^*)). B := i = 1 ∑ d θ i ∗ ( ϕ ( u i ) − ϕ ( u i ∗ )) .
For A A A : the variables ϕ ( u i ) \phi(u_i) ϕ ( u i ) are iid, centered, and have finite moments up to order 8. Hence by Rosenthal’s inequality, for m ∈ { 2 , 4 , 8 } m \in\{2,4,8\} m ∈ { 2 , 4 , 8 } ,
∥ A ∥ L m ( P ( ⋅ ∣ F t ) ) ≤ C m ∥ θ − θ ∗ ∥ 2 . \|A\|_{L^m(P(\cdot|\mathcal{F}_t))} \le C_m\|\theta-\theta^*\|_2. ∥ A ∥ L m ( P ( ⋅ ∣ F t )) ≤ C m ∥ θ − θ ∗ ∥ 2 .
Since Euclidean distance is bounded by geodesic distance on the sphere,
∥ θ − θ ∗ ∥ 2 ≤ d θ , \|\theta-\theta^*\|_2 \le d_\theta, ∥ θ − θ ∗ ∥ 2 ≤ d θ ,
we get
∥ A ∥ L m ( P ( ⋅ ∣ F t ) ) ≤ C m d θ . \|A\|_{L^m(P(\cdot|\mathcal{F}_t))} \le C_m d_\theta. ∥ A ∥ L m ( P ( ⋅ ∣ F t )) ≤ C m d θ .
For B B B : the blocks x ( i ) x^{(i)} x ( i ) are independent, so the summands
Y i : = θ i ∗ ( ϕ ( u i ) − ϕ ( u i ∗ ) ) Y_i:=\theta_i^*(\phi(u_i)-\phi(u_i^*)) Y i := θ i ∗ ( ϕ ( u i ) − ϕ ( u i ∗ ))
are independent across i i i . They are centered because u i ∼ N ( 0 , 1 ) u_i\sim N(0,1) u i ∼ N ( 0 , 1 ) and u i ∗ ∼ N ( 0 , 1 ) u_i^*\sim N(0,1) u i ∗ ∼ N ( 0 , 1 ) , hence
E [ ϕ ( u i ) − ϕ ( u i ∗ ) ] = 0. \mathbb{E}[\phi(u_i)-\phi(u_i^*)]=0. E [ ϕ ( u i ) − ϕ ( u i ∗ )] = 0.
Using the Lipschitzness of ϕ \phi ϕ ,
∣ Y i ∣ ≤ ∣ θ i ∗ ∣ Lip ( ϕ ) ∣ u i − u i ∗ ∣ = ∣ θ i ∗ ∣ Lip ( ϕ ) ∣ x ( i ) ⋅ ( β − β ∗ ) ∣ . |Y_i| \le|\theta_i^*|\operatorname{Lip}(\phi)|u_i-u_i^*|=|\theta_i^*|\operatorname{Lip}(\phi)|x^{(i)}\cdot(\beta-\beta^*)|. ∣ Y i ∣ ≤ ∣ θ i ∗ ∣ Lip ( ϕ ) ∣ u i − u i ∗ ∣ = ∣ θ i ∗ ∣ Lip ( ϕ ) ∣ x ( i ) ⋅ ( β − β ∗ ) ∣.
Thus for m ∈ { 2 , 4 , 8 } m\in\{2,4,8\} m ∈ { 2 , 4 , 8 } ,
∑ i = 1 d E [ ∣ Y i ∣ m ∣ F t ] ≤ C m ∥ β − β ∗ ∥ 2 m ∑ i = 1 d ∣ θ i ∗ ∣ m ≤ C m d β m , \sum_{i=1}^{d}\mathbb{E}[|Y_i|^m\mid\mathcal{F}_t]\le C_m\|\beta-\beta^*\|_2^m\sum_{i=1}^{d}|\theta_i^*|^m\le C_m d_\beta^m, i = 1 ∑ d E [ ∣ Y i ∣ m ∣ F t ] ≤ C m ∥ β − β ∗ ∥ 2 m i = 1 ∑ d ∣ θ i ∗ ∣ m ≤ C m d β m ,
since ∑ i ∣ θ i ∗ ∣ m ≤ ∥ θ ∗ ∥ 2 m = 1 \sum_i|\theta_i^*|^m\le\|\theta^*\|_2^m=1 ∑ i ∣ θ i ∗ ∣ m ≤ ∥ θ ∗ ∥ 2 m = 1 for m ≥ 2 m\ge2 m ≥ 2 . Applying Rosenthal again,
∥ B ∥ L m ( P ( ⋅ ∣ F t ) ) ≤ C m d β , m = 2 , 4 , 8. \|B\|_{L^m(P(\cdot|\mathcal{F}_t))}\le C_m d_\beta,\qquad m=2,4,8. ∥ B ∥ L m ( P ( ⋅ ∣ F t )) ≤ C m d β , m = 2 , 4 , 8.
Combining the bounds for A A A and B B B ,
∥ s − s ∗ ∥ L m ( P ( ⋅ ∣ F t ) ) ≤ C m ( d θ + d β ) ≤ C m d θ 2 + d β 2 = C m 2 Q t . \|s-s^*\|_{L^m(P(\cdot|\mathcal{F}_t))}\le C_m(d_\theta+d_\beta)\le C_m\sqrt{d_\theta^2+d_\beta^2}=C_m\sqrt{2Q_t}. ∥ s − s ∗ ∥ L m ( P ( ⋅ ∣ F t )) ≤ C m ( d θ + d β ) ≤ C m d θ 2 + d β 2 = C m 2 Q t .
Hence
∥ R ∥ L m ( P ( ⋅ ∣ F t ) ) ≤ C m Q t , m = 2 , 4 , 8 , \|R\|_{L^m(P(\cdot|\mathcal{F}_t))}\le C_m\sqrt{Q_t},\qquad m=2,4,8, ∥ R ∥ L m ( P ( ⋅ ∣ F t )) ≤ C m Q t , m = 2 , 4 , 8 ,
which is equivalent to
E [ ∣ R ∣ m ∣ F t ] ≤ C m Q t m / 2 , m = 2 , 4 , 8. \mathbb{E}[|R|^m\mid\mathcal{F}_t]\le C_m Q_t^{m/2},\qquad m=2,4,8. E [ ∣ R ∣ m ∣ F t ] ≤ C m Q t m /2 , m = 2 , 4 , 8.
feature-side moment bounds.
Define
z : = z β ( x ) = ( ϕ ( u 1 ) , … , ϕ ( u d ) ) , h : = P β ∑ i = 1 d θ i ϕ ′ ( u i ) x ( i ) , z:=z_\beta(x)=(\phi(u_1),\ldots,\phi(u_d)),\qquad h:=P_\beta\sum_{i=1}^{d}\theta_i\phi'(u_i)x^{(i)}, z := z β ( x ) = ( ϕ ( u 1 ) , … , ϕ ( u d )) , h := P β i = 1 ∑ d θ i ϕ ′ ( u i ) x ( i ) ,
so that
g ( θ ) = A ⋅ P θ z , g ( β ) = A ⋅ h , A : = 2 ψ ′ ( s ) R , g^{(\theta)}=A\cdot P_\theta z,\qquad g^{(\beta)}=A\cdot h,\qquad A:=2\psi'(s)R, g ( θ ) = A ⋅ P θ z , g ( β ) = A ⋅ h , A := 2 ψ ′ ( s ) R ,
and G t ≤ C R 2 W G_t \le CR^2W G t ≤ C R 2 W where W : = ∥ z ∥ 2 + ∥ h ∥ 2 W := \|z\|^2 + \|h\|^2 W := ∥ z ∥ 2 + ∥ h ∥ 2 . The same Gaussian moment computations used in the proof of Lemma E.7 give, for k = 1 , 2 , 4 k = 1, 2, 4 k = 1 , 2 , 4 ,
E [ ∥ z ∥ 2 k ∣ F t ] ≤ C d k , E [ ∥ h ∥ 2 k ∣ F t ] ≤ C d k . \mathbb{E}[\|z\|^{2k} \mid\mathcal{F}_t] \le Cd^k,\qquad\mathbb{E}[\|h\|^{2k} \mid\mathcal{F}_t] \le Cd^k. E [ ∥ z ∥ 2 k ∣ F t ] ≤ C d k , E [ ∥ h ∥ 2 k ∣ F t ] ≤ C d k .
Hence E [ W k ∣ F t ] ≤ C d k \mathbb{E}[W^k \mid\mathcal{F}_t] \le Cd^k E [ W k ∣ F t ] ≤ C d k for k = 1 , 2 , 4 k = 1, 2, 4 k = 1 , 2 , 4 .
bound on E [ G t ∣ F t ] \mathbb{E}[G_t \mid\mathcal{F}_t] E [ G t ∣ F t ] . Since G t ≤ C R 2 W G_t \le CR^2W G t ≤ C R 2 W , Cauchy–Schwarz gives
E [ G t ∣ F t ] ≤ C ( E [ R 4 ∣ F t ] ) 1 / 2 ( E [ W 2 ∣ F t ] ) 1 / 2 . \mathbb{E}[G_t \mid\mathcal{F}_t] \le C\left(\mathbb{E}[R^4 \mid\mathcal{F}_t]\right)^{1/2}\left(\mathbb{E}[W^2 \mid\mathcal{F}_t]\right)^{1/2}. E [ G t ∣ F t ] ≤ C ( E [ R 4 ∣ F t ] ) 1/2 ( E [ W 2 ∣ F t ] ) 1/2 .
Using Step 1 with m = 4 m = 4 m = 4 and Step 2,
E [ G t ∣ F t ] ≤ C ( Q t 2 ) 1 / 2 ( d 2 ) 1 / 2 = C d Q t . \mathbb{E}[G_t \mid\mathcal{F}_t] \le C\left(Q_t^2\right)^{1/2}\left(d^2\right)^{1/2} = CdQ_t. E [ G t ∣ F t ] ≤ C ( Q t 2 ) 1/2 ( d 2 ) 1/2 = C d Q t .
bound on E [ Ξ t 2 ∣ F t ] \mathbb{E}[\Xi_t^2 \mid\mathcal{F}_t] E [ Ξ t 2 ∣ F t ] . Since conditional variance is bounded by the second moment,
E [ Ξ t 2 ∣ F t ] ≤ E [ ( ⟨ g ( θ ) , a ⟩ + ⟨ g ( β ) , b ⟩ ) 2 | F t ] . \mathbb{E}[\Xi_t^2 \mid\mathcal{F}_t] \le\mathbb{E}\left[\left(\langle g^{(\theta)},a\rangle+ \langle g^{(\beta)},b\rangle\right)^2 \,\middle|\, \mathcal{F}_t\right]. E [ Ξ t 2 ∣ F t ] ≤ E [ ( ⟨ g ( θ ) , a ⟩ + ⟨ g ( β ) , b ⟩ ) 2 F t ] .
Hence
E [ Ξ t 2 ∣ F t ] ≤ 2 E [ ⟨ g ( θ ) , a ⟩ 2 ∣ F t ] + 2 E [ ⟨ g ( β ) , b ⟩ 2 ∣ F t ] . \mathbb{E}[\Xi_t^2 \mid\mathcal{F}_t] \le2\mathbb{E}[\langle g^{(\theta)},a\rangle^2 \mid\mathcal{F}_t] + 2\mathbb{E}[\langle g^{(\beta)},b\rangle^2 \mid\mathcal{F}_t]. E [ Ξ t 2 ∣ F t ] ≤ 2 E [⟨ g ( θ ) , a ⟩ 2 ∣ F t ] + 2 E [⟨ g ( β ) , b ⟩ 2 ∣ F t ] .
For the θ \theta θ -block, a ∈ T θ S d − 1 a \in T_\theta S^{d-1} a ∈ T θ S d − 1 , so
⟨ P θ z , a ⟩ = ⟨ z , a ⟩ . \langle P_\theta z,a\rangle= \langle z,a\rangle. ⟨ P θ z , a ⟩ = ⟨ z , a ⟩ .
Thus
⟨ g ( θ ) , a ⟩ = 2 R ψ ′ ( s ) ⟨ z , a ⟩ . \langle g^{(\theta)},a\rangle= 2R\psi'(s)\langle z,a\rangle. ⟨ g ( θ ) , a ⟩ = 2 R ψ ′ ( s ) ⟨ z , a ⟩ .
Therefore
E [ ⟨ g ( θ ) , a ⟩ 2 ∣ F t ] ≤ C ( E [ R 4 ∣ F t ] ) 1 / 2 ( E [ ⟨ z , a ⟩ 4 ∣ F t ] ) 1 / 2 . \mathbb{E}[\langle g^{(\theta)},a\rangle^2 \mid\mathcal{F}_t] \le C\left(\mathbb{E}[R^4 \mid\mathcal{F}_t]\right)^{1/2}\left(\mathbb{E}[\langle z,a\rangle^4 \mid\mathcal{F}_t]\right)^{1/2}. E [⟨ g ( θ ) , a ⟩ 2 ∣ F t ] ≤ C ( E [ R 4 ∣ F t ] ) 1/2 ( E [⟨ z , a ⟩ 4 ∣ F t ] ) 1/2 .
Now ⟨ z , a ⟩ = ∑ i a i ϕ ( u i ) \langle z,a\rangle= \sum_i a_i\phi(u_i) ⟨ z , a ⟩ = ∑ i a i ϕ ( u i ) is a linear statistic of iid centered variables with finite fourth moment, so Rosenthal yields
E [ ⟨ z , a ⟩ 4 ∣ F t ] ≤ C ∥ a ∥ 4 . \mathbb{E}[\langle z,a\rangle^4 \mid\mathcal{F}_t] \le C\|a\|^4. E [⟨ z , a ⟩ 4 ∣ F t ] ≤ C ∥ a ∥ 4 .
Hence
E [ ⟨ g ( θ ) , a ⟩ 2 ∣ F t ] ≤ C Q t ∥ a ∥ 2 ≤ C Q t 2 . \mathbb{E}[\langle g^{(\theta)},a\rangle^2 \mid\mathcal{F}_t] \le CQ_t\|a\|^2 \le CQ_t^2. E [⟨ g ( θ ) , a ⟩ 2 ∣ F t ] ≤ C Q t ∥ a ∥ 2 ≤ C Q t 2 .
For the β \beta β -block,
⟨ g ( β ) , b ⟩ = 2 R ψ ′ ( s ) ⟨ h , b ⟩ . \langle g^{(\beta)},b\rangle= 2R\psi'(s)\langle h,b\rangle. ⟨ g ( β ) , b ⟩ = 2 R ψ ′ ( s ) ⟨ h , b ⟩ .
Conditionally on ( u 1 , … , u d ) (u_1,\ldots,u_d) ( u 1 , … , u d ) , the scalar ⟨ h , b ⟩ \langle h,b\rangle ⟨ h , b ⟩ is Gaussian with variance
∥ b ∥ 2 σ 2 , σ 2 = ∑ i = 1 d θ i 2 ϕ ′ ( u i ) 2 . \|b\|^2\sigma^2,\qquad\sigma^2 = \sum_{i=1}^d \theta_i^2\phi'(u_i)^2. ∥ b ∥ 2 σ 2 , σ 2 = i = 1 ∑ d θ i 2 ϕ ′ ( u i ) 2 .
Therefore
E [ ⟨ h , b ⟩ 4 ∣ u , F t ] ≤ C ∥ b ∥ 4 σ 4 . \mathbb{E}[\langle h,b\rangle^4 \mid u,\mathcal{F}_t] \le C\|b\|^4\sigma^4. E [⟨ h , b ⟩ 4 ∣ u , F t ] ≤ C ∥ b ∥ 4 σ 4 .
Averaging and using E [ σ 4 ∣ F t ] ≤ C \mathbb{E}[\sigma^4 \mid\mathcal{F}_t] \le C E [ σ 4 ∣ F t ] ≤ C ,
E [ ⟨ h , b ⟩ 4 ∣ F t ] ≤ C ∥ b ∥ 4 . \mathbb{E}[\langle h,b\rangle^4 \mid\mathcal{F}_t] \le C\|b\|^4. E [⟨ h , b ⟩ 4 ∣ F t ] ≤ C ∥ b ∥ 4 .
Thus
E [ ⟨ g ( β ) , b ⟩ 2 ∣ F t ] ≤ C ( E [ R 4 ∣ F t ] ) 1 / 2 ( E [ ⟨ h , b ⟩ 4 ∣ F t ] ) 1 / 2 ≤ C Q t ∥ b ∥ 2 ≤ C Q t 2 . \mathbb{E}[\langle g^{(\beta)},b\rangle^2 \mid\mathcal{F}_t] \le C\left(\mathbb{E}[R^4 \mid\mathcal{F}_t]\right)^{1/2}\left(\mathbb{E}[\langle h,b\rangle^4 \mid\mathcal{F}_t]\right)^{1/2} \le CQ_t\|b\|^2 \le CQ_t^2. E [⟨ g ( β ) , b ⟩ 2 ∣ F t ] ≤ C ( E [ R 4 ∣ F t ] ) 1/2 ( E [⟨ h , b ⟩ 4 ∣ F t ] ) 1/2 ≤ C Q t ∥ b ∥ 2 ≤ C Q t 2 .
Combining the two block estimates,
E [ Ξ t 2 ∣ F t ] ≤ C Q t 2 . \mathbb{E}[\Xi_t^2 \mid\mathcal{F}_t] \le CQ_t^2. E [ Ξ t 2 ∣ F t ] ≤ C Q t 2 .
centered normalization fluctuation.
Since
E [ ( G t − E [ G t ∣ F t ] ) 2 ∣ F t ] ≤ E [ G t 2 ∣ F t ] , \mathbb{E}\left[\left(G_t-\mathbb{E}[G_t\mid\mathcal{F}_t]\right)^2\mid\mathcal{F}_t\right]\leq\mathbb{E}[G_t^2\mid\mathcal{F}_t], E [ ( G t − E [ G t ∣ F t ] ) 2 ∣ F t ] ≤ E [ G t 2 ∣ F t ] ,
it suffices to bound E [ G t 2 ∣ F t ] \mathbb{E}[G_t^2\mid\mathcal{F}_t] E [ G t 2 ∣ F t ] . Using G t ≤ C R 2 W G_t\leq CR^2W G t ≤ C R 2 W ,
G t 2 ≤ C R 4 W 2 . G_t^2\leq CR^4W^2. G t 2 ≤ C R 4 W 2 .
Hence, by Cauchy–Schwarz,
E [ G t 2 ∣ F t ] ≤ C ( E [ R 8 ∣ F t ] ) 1 / 2 ( E [ W 4 ∣ F t ] ) 1 / 2 . \mathbb{E}[G_t^2\mid\mathcal{F}_t]\leq C\left(\mathbb{E}[R^8\mid\mathcal{F}_t]\right)^{1/2}\left(\mathbb{E}[W^4\mid\mathcal{F}_t]\right)^{1/2}. E [ G t 2 ∣ F t ] ≤ C ( E [ R 8 ∣ F t ] ) 1/2 ( E [ W 4 ∣ F t ] ) 1/2 .
Applying Step 1 with m = 8 m=8 m = 8 and Step 2,
E [ G t 2 ∣ F t ] ≤ C ( Q t 4 ) 1 / 2 ( d 4 ) 1 / 2 = C d 2 Q t 2 . \mathbb{E}[G_t^2\mid\mathcal{F}_t]\leq C\left(Q_t^4\right)^{1/2}\left(d^4\right)^{1/2}=Cd^2Q_t^2. E [ G t 2 ∣ F t ] ≤ C ( Q t 4 ) 1/2 ( d 4 ) 1/2 = C d 2 Q t 2 .
Therefore
E [ ( G t − E [ G t ∣ F t ] ) 2 ∣ F t ] ≤ C d 2 Q t 2 . \mathbb{E}\left[\left(G_t-\mathbb{E}[G_t\mid\mathcal{F}_t]\right)^2\mid\mathcal{F}_t\right]\leq Cd^2Q_t^2. E [ ( G t − E [ G t ∣ F t ] ) 2 ∣ F t ] ≤ C d 2 Q t 2 .
This proves the lemma. □
Strong Recovery Dynamics
With all the controlling lemmas needed, we now move to prove the strong recovery dynamics.
Proposition D.24 (One local SGD epoch contracts). Assume Lemma C.15, Lemma D.22, and Lemma D.23. Let the local step size satisfy
δ = δ l o c = c l o c d , C r e t C s q δ d ≤ μ l o c . \delta=\delta_{\mathrm{loc}}=\frac{c_{\mathrm{loc}}}{d},\qquad C_{\mathrm{ret}}C_{\mathrm{sq}}\delta d\leq\mu_{\mathrm{loc}}. δ = δ loc = d c loc , C ret C sq δ d ≤ μ loc . Let s s s be a stopping time, and suppose that, conditionally on F s \mathcal{F}_s F s ,
Q s ≤ B , 0 ≤ B ≤ r l o c 2 32 . Q_s\leq B,\qquad0\leq B\leq\frac{r_{\mathrm{loc}}^2}{32}. Q s ≤ B , 0 ≤ B ≤ 32 r loc 2 . Define τ B : = inf { t ≥ s : Q t > 2 B } \tau_B:=\inf\{t\geq s:Q_t>2B\} τ B := inf { t ≥ s : Q t > 2 B } . Then there exist constants C e p < ∞ C_{\mathrm{ep}}<\infty C ep < ∞ , C < ∞ C<\infty C < ∞ , and d 0 < ∞ d_0<\infty d 0 < ∞ , depending only on the model constants, such that for all d ≥ d 0 d\geq d_0 d ≥ d 0 , with
L e p : = ⌈ C e p δ ⌉ , L_{\mathrm{ep}}:=\left\lceil\frac{C_{\mathrm{ep}}}{\delta}\right\rceil, L ep := ⌈ δ C ep ⌉ , one has
P ( Q s + L e p ≤ B 2 and τ B > s + L e p ∣ F s ) ≥ 1 − C δ . \mathbb{P}\left(Q_{s+L_{\mathrm{ep}}}\leq\frac{B}{2}\text{ and }\tau_B>s+L_{\mathrm{ep}}\mid\mathcal{F}_s\right)\geq1-C\delta. P ( Q s + L ep ≤ 2 B and τ B > s + L ep ∣ F s ) ≥ 1 − C δ . Equivalently, with probability at least 1 − C δ 1-C\delta 1 − C δ ,
Q s + L e p ≤ B 2 and Q t ≤ 2 B for every s ≤ t ≤ s + L e p . Q_{s+L_{\mathrm{ep}}}\leq\frac{B}{2}\qquad\text{and}\qquad Q_t\leq2B\quad\text{for every }s\leq t\leq s+L_{\mathrm{ep}}. Q s + L ep ≤ 2 B and Q t ≤ 2 B for every s ≤ t ≤ s + L ep .
Proof. If B = 0 B=0 B = 0 , then Q s = 0 Q_s=0 Q s = 0 , so y s = y ∗ y_s=y^* y s = y ∗ . By Lemma D.23,
E [ G s ∣ F s ] ≤ C s q d Q s = 0 , \mathbb{E}[G_s\mid\mathcal{F}_s]\leq C_{\mathrm{sq}}dQ_s=0, E [ G s ∣ F s ] ≤ C sq d Q s = 0 ,
hence G s = 0 G_s=0 G s = 0 almost surely conditionally on F s \mathcal{F}_s F s . Thus the SGD step is trivial at y ∗ y^* y ∗ , and the same argument iterates. The conclusion is therefore immediate. Hence assume B > 0 B>0 B > 0 .
Throughout the proof we condition on F s \mathcal{F}_s F s . Since the samples after time s s s are fresh, all martingale and conditional moment estimates hold with the same constants.
Step 1: stopped multiplicative recursion.
For every t < τ B t<\tau_B t < τ B , one has Q t ≤ 2 B Q_t\leq2B Q t ≤ 2 B . Hence
dist M ( y t , y ∗ ) 2 = 2 Q t ≤ 4 B ≤ r l o c 2 8 < r l o c 2 . \operatorname{dist}_{\mathcal{M}}(y_t,y^*)^2=2Q_t\leq4B\leq\frac{r_{\mathrm{loc}}^2}{8}<r_{\mathrm{loc}}^2. dist M ( y t , y ∗ ) 2 = 2 Q t ≤ 4 B ≤ 8 r loc 2 < r loc 2 .
Thus y t ∈ U r l o c y_t \in\mathcal{U}_{r_{\mathrm{loc}}} y t ∈ U r loc , and the local one-step recursion is valid. By Lemma D.22,
Q t + 1 ≤ Q t + δ ⟨ g ˉ t , L t ⟩ + δ Ξ t + C r e t δ 2 G t . Q_{t+1} \le Q_t + \delta\langle\bar{g}_t,\mathcal{L}_t\rangle+ \delta\Xi_t + C_{\mathrm{ret}}\delta^2G_t. Q t + 1 ≤ Q t + δ ⟨ g ˉ t , L t ⟩ + δ Ξ t + C ret δ 2 G t .
Since g ˉ t = ∇ M L ( y t ) \bar{g}_t = \nabla_M L(y_t) g ˉ t = ∇ M L ( y t ) , Lemma C.15 gives
⟨ g ˉ t , L t ⟩ = ⟨ ∇ M L ( y t ) , L t ⟩ ≤ − 2 μ l o c Q t . \langle\bar{g}_t,\mathcal{L}_t\rangle= \langle\nabla_M L(y_t),\mathcal{L}_t\rangle\le-2\mu_{\mathrm{loc}}Q_t. ⟨ g ˉ t , L t ⟩ = ⟨ ∇ M L ( y t ) , L t ⟩ ≤ − 2 μ loc Q t .
Define
ζ t : = G t − E [ G t ∣ F t ] . \zeta_t := G_t-\mathbb{E}[G_t\mid\mathcal{F}_t]. ζ t := G t − E [ G t ∣ F t ] .
Then, for t < τ B t<\tau_B t < τ B ,
Q t + 1 ≤ Q t − 2 μ l o c δ Q t + δ Ξ t + C r e t δ 2 E [ G t ∣ F t ] + C r e t δ 2 ζ t . Q_{t+1}\le Q_t-2\mu_{\mathrm{loc}}\delta Q_t+\delta\Xi_t+C_{\mathrm{ret}}\delta^2\mathbb{E}[G_t\mid\mathcal{F}_t]+C_{\mathrm{ret}}\delta^2\zeta_t. Q t + 1 ≤ Q t − 2 μ loc δ Q t + δ Ξ t + C ret δ 2 E [ G t ∣ F t ] + C ret δ 2 ζ t .
By Lemma D.23,
E [ G t ∣ F t ] ≤ C s q d Q t . \mathbb{E}[G_t\mid\mathcal{F}_t]\le C_{\mathrm{sq}}dQ_t. E [ G t ∣ F t ] ≤ C sq d Q t .
Therefore
Q t + 1 ≤ ( 1 − ( 2 μ l o c − C r e t C s q δ d ) δ ) Q t + δ Ξ t + C r e t δ 2 ζ t . Q_{t+1}\le\left(1-(2\mu_{\mathrm{loc}}-C_{\mathrm{ret}}C_{\mathrm{sq}}\delta d)\delta\right)Q_t+\delta\Xi_t+C_{\mathrm{ret}}\delta^2\zeta_t. Q t + 1 ≤ ( 1 − ( 2 μ loc − C ret C sq δ d ) δ ) Q t + δ Ξ t + C ret δ 2 ζ t .
The local step-size condition implies
2 μ l o c − C r e t C s q δ d ≥ μ l o c . 2\mu_{\mathrm{loc}}-C_{\mathrm{ret}}C_{\mathrm{sq}}\delta d\ge\mu_{\mathrm{loc}}. 2 μ loc − C ret C sq δ d ≥ μ loc .
Set
λ : = μ l o c , α : = 1 − λ δ . \lambda:=\mu_{\mathrm{loc}},\qquad\alpha:=1-\lambda\delta. λ := μ loc , α := 1 − λ δ .
For d d d large enough, α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) . Thus, for every t < τ B t<\tau_B t < τ B ,
Q t + 1 ≤ α Q t + Δ N t + 1 , (206) Q_{t+1}\le\alpha Q_t+\Delta N_{t+1}, \tag*{(206)} Q t + 1 ≤ α Q t + Δ N t + 1 , (206)
where
Δ N t + 1 : = δ Ξ t + C r e t δ 2 ζ t . \Delta N_{t+1}:=\delta\Xi_t+C_{\mathrm{ret}}\delta^2\zeta_t. Δ N t + 1 := δ Ξ t + C ret δ 2 ζ t .
Moreover,
E [ Δ N t + 1 ∣ F t ] = 0. \mathbb{E}[\Delta N_{t+1}\mid\mathcal{F}_t]=0. E [ Δ N t + 1 ∣ F t ] = 0.
Step 2: martingale variance on the stopped epoch.
By Lemma D.23,
E [ Ξ t 2 ∣ F t ] ≤ C s q Q t 2 , \mathbb{E}[\Xi_t^2\mid\mathcal{F}_t]\le C_{\mathrm{sq}}Q_t^2, E [ Ξ t 2 ∣ F t ] ≤ C sq Q t 2 ,
and
E [ ζ t 2 ∣ F t ] = E [ ( G t − E [ G t ∣ F t ] ) 2 ∣ F t ] ≤ C s q d 2 Q t 2 . \mathbb{E}[\zeta_t^2\mid\mathcal{F}_t]=\mathbb{E}\left[(G_t-\mathbb{E}[G_t\mid\mathcal{F}_t])^2\mid\mathcal{F}_t\right]\le C_{\mathrm{sq}}d^2Q_t^2. E [ ζ t 2 ∣ F t ] = E [ ( G t − E [ G t ∣ F t ] ) 2 ∣ F t ] ≤ C sq d 2 Q t 2 .
Hence, using ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^2\le2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2 ,
E [ ( Δ N t + 1 ) 2 ∣ F t ] ≤ C δ 2 Q t 2 + C δ 4 d 2 Q t 2 . \mathbb{E}[(\Delta N_{t+1})^2\mid\mathcal{F}_t]\le C\delta^2Q_t^2+C\delta^4d^2Q_t^2. E [( Δ N t + 1 ) 2 ∣ F t ] ≤ C δ 2 Q t 2 + C δ 4 d 2 Q t 2 .
Since δ d \delta d δ d is bounded under the local step-size regime,
E [ ( Δ N t + 1 ) 2 ∣ F t ] ≤ C δ 2 Q t 2 . \mathbb{E}[(\Delta N_{t+1})^2\mid\mathcal{F}_t]\le C\delta^2Q_t^2. E [( Δ N t + 1 ) 2 ∣ F t ] ≤ C δ 2 Q t 2 .
On { t < τ B } \{t<\tau_B\} { t < τ B } , Q t ≤ 2 B Q_t\le2B Q t ≤ 2 B , and therefore
E [ ( Δ N t + 1 ) 2 ∣ F t ] ≤ C δ 2 B 2 . \mathbb{E}[(\Delta N_{t+1})^2\mid\mathcal{F}_t]\le C\delta^2B^2. E [( Δ N t + 1 ) 2 ∣ F t ] ≤ C δ 2 B 2 .
Define the stopped martingale increments
Δ N ~ t + 1 : = 1 { t < τ B } Δ N t + 1 , \Delta\widetilde{N}_{t+1}:=\mathbf{1}_{\{t<\tau_B\}}\Delta N_{t+1}, Δ N t + 1 := 1 { t < τ B } Δ N t + 1 ,
and for u = s , … , s + L e p u=s,\ldots,s+L_{\mathrm{ep}} u = s , … , s + L ep ,
N ~ u : = ∑ k = s u − 1 Δ N ~ k + 1 , N ~ s : = 0. \widetilde{N}_u:=\sum_{k=s}^{u-1}\Delta\widetilde{N}_{k+1},\qquad\widetilde{N}_s:=0. N u := k = s ∑ u − 1 Δ N k + 1 , N s := 0.
Then ( N ~ u ) u = s s + L e p (\widetilde{N}_u)_{u=s}^{s+L_{\mathrm{ep}}} ( N u ) u = s s + L ep is a martingale, and
E [ ∣ N ~ s + L e p ∣ 2 ∣ F s ] = E [ ∑ k = s s + L e p − 1 E [ ( Δ N ~ k + 1 ) 2 ∣ F k ] ∣ F s ] ≤ C L e p δ 2 B 2 . \mathbb{E}\left[|\widetilde{N}_{s+L_{\mathrm{ep}}}|^2\mid\mathcal{F}_s\right]
=\mathbb{E}\left[\left.\sum_{k=s}^{s+L_{\mathrm{ep}}-1}\mathbb{E}[(\Delta\widetilde{N}_{k+1})^2\mid\mathcal{F}_k]\right|\mathcal{F}_s\right]
\le CL_{\mathrm{ep}}\delta^2B^2. E [ ∣ N s + L ep ∣ 2 ∣ F s ] = E k = s ∑ s + L ep − 1 E [( Δ N k + 1 ) 2 ∣ F k ] F s ≤ C L ep δ 2 B 2 .
Since
L e p = ⌊ C e p δ ⌋ ≤ C δ . L_{\mathrm{ep}}=\left\lfloor\frac{C_{\mathrm{ep}}}{\delta}\right\rfloor\le\frac{C}{\delta}. L ep = ⌊ δ C ep ⌋ ≤ δ C .
for d d d large enough,
E [ ∣ N ~ s + L e p ∣ 2 ∣ F s ] ≤ C δ B 2 . \mathbb{E}\left[\left|\widetilde{N}_{s+L_{\mathrm{ep}}}\right|^2\mid\mathcal{F}_s\right]\le C\delta B^2. E [ N s + L ep 2 ∣ F s ] ≤ C δ B 2 .
By Doob’s L 2 L^2 L 2 maximal inequality,
E [ sup s ≤ u ≤ s + L e p ∣ N ~ u ∣ 2 ∣ F s ] ≤ C δ B 2 . \mathbb{E}\left[\sup_{s\le u\le s+L_{\mathrm{ep}}}\left|\widetilde{N}_u\right|^2\mid\mathcal{F}_s\right]\le C\delta B^2. E [ s ≤ u ≤ s + L ep sup N u 2 ∣ F s ] ≤ C δ B 2 .
Therefore, by Markov’s inequality,
P ( sup s ≤ u ≤ s + L e p ∣ N ~ u ∣ > B 8 | F s ) ≤ C δ . \mathbb{P}\left(\sup_{s\le u\le s+L_{\mathrm{ep}}}\left|\widetilde{N}_u\right|>\frac{B}{8}\,\middle|\,\mathcal{F}_s\right)\le C\delta. P ( s ≤ u ≤ s + L ep sup N u > 8 B F s ) ≤ C δ .
Step 3: weighted martingale control.
For 1 ≤ ℓ ≤ L e p 1\le\ell\le L_{\mathrm{ep}} 1 ≤ ℓ ≤ L ep , define the stopped geometric convolution
S ~ ℓ : = ∑ j = 0 ℓ − 1 α ℓ − 1 − j Δ N ~ s + j + 1 . \widetilde{S}_{\ell}:=\sum_{j=0}^{\ell-1}\alpha^{\ell-1-j}\Delta\widetilde{N}_{s+j+1}. S ℓ := j = 0 ∑ ℓ − 1 α ℓ − 1 − j Δ N s + j + 1 .
We claim that
sup 1 ≤ ℓ ≤ L e p ∣ S ~ ℓ ∣ ≤ 2 sup s ≤ u ≤ s + L e p ∣ N ~ u ∣ . \sup_{1\le\ell\le L_{\mathrm{ep}}}\left|\widetilde{S}_{\ell}\right|\le2\sup_{s\le u\le s+L_{\mathrm{ep}}}\left|\widetilde{N}_u\right|. 1 ≤ ℓ ≤ L ep sup S ℓ ≤ 2 s ≤ u ≤ s + L ep sup N u .
Indeed, for fixed ℓ \ell ℓ , write
A j : = N ~ s + j , j = 0 , … , ℓ . A_j:=\widetilde{N}_{s+j},\qquad j=0,\ldots,\ell. A j := N s + j , j = 0 , … , ℓ .
Then A 0 = 0 A_0=0 A 0 = 0 and
Δ N ~ s + j + 1 = A j + 1 − A j . \Delta\widetilde{N}_{s+j+1}=A_{j+1}-A_j. Δ N s + j + 1 = A j + 1 − A j .
By summation by parts,
S ~ ℓ = A ℓ − ( 1 − α ) ∑ j = 1 ℓ − 1 α ℓ − 1 − j A j . \widetilde{S}_{\ell}=A_{\ell}-(1-\alpha)\sum_{j=1}^{\ell-1}\alpha^{\ell-1-j}A_j. S ℓ = A ℓ − ( 1 − α ) j = 1 ∑ ℓ − 1 α ℓ − 1 − j A j .
Thus
∣ S ~ ℓ ∣ ≤ sup 0 ≤ j ≤ ℓ ∣ A j ∣ + ( 1 − α ) ∑ j = 1 ℓ − 1 α ℓ − 1 − j sup 0 ≤ j ≤ ℓ ∣ A j ∣ ≤ 2 sup 0 ≤ j ≤ ℓ ∣ A j ∣ . \left|\widetilde{S}_{\ell}\right|\le\sup_{0\le j\le\ell}|A_j|+(1-\alpha)\sum_{j=1}^{\ell-1}\alpha^{\ell-1-j}\sup_{0\le j\le\ell}|A_j|\le2\sup_{0\le j\le\ell}|A_j|. S ℓ ≤ 0 ≤ j ≤ ℓ sup ∣ A j ∣ + ( 1 − α ) j = 1 ∑ ℓ − 1 α ℓ − 1 − j 0 ≤ j ≤ ℓ sup ∣ A j ∣ ≤ 2 0 ≤ j ≤ ℓ sup ∣ A j ∣.
Taking the supremum over ℓ \ell ℓ proves the claim.
Define the epoch good event
E e p : = { sup 1 ≤ ℓ ≤ L e p ∣ S ~ ℓ ∣ ≤ B 4 } . \mathcal{E}_{\mathrm{ep}}:=\left\{\sup_{1\le\ell\le L_{\mathrm{ep}}}\left|\widetilde{S}_{\ell}\right|\le\frac{B}{4}\right\}. E ep := { 1 ≤ ℓ ≤ L ep sup S ℓ ≤ 4 B } .
By the preceding maximal bound,
P ( E e p ∣ F s ) ≥ 1 − C δ . \mathbb{P}(\mathcal{E}_{\mathrm{ep}}\mid\mathcal{F}_s)\ge1-C\delta. P ( E ep ∣ F s ) ≥ 1 − C δ .
Step 4: contraction on E e p \mathcal{E}_{\mathrm{ep}} E ep .
We prove by induction that, on E e p \mathcal{E}_{\mathrm{ep}} E ep ,
Q s + ℓ ≤ α ℓ B + B 4 for every 0 ≤ ℓ ≤ L e p . Q_{s+\ell}\le\alpha^{\ell}B+\frac{B}{4}\qquad\text{for every }0\le\ell\le L_{\mathrm{ep}}. Q s + ℓ ≤ α ℓ B + 4 B for every 0 ≤ ℓ ≤ L ep .
For ℓ = 0 \ell=0 ℓ = 0 , this follows from Q s ≤ B Q_s\le B Q s ≤ B . Assume the claim has been proved up to time s + ℓ − 1 s+\ell-1 s + ℓ − 1 , where 1 ≤ ℓ ≤ L e p 1\le\ell\le L_{\mathrm{ep}} 1 ≤ ℓ ≤ L ep . Then, for every 0 ≤ j ≤ ℓ − 1 0\le j\le\ell-1 0 ≤ j ≤ ℓ − 1 ,
Q s + j ≤ α j B + B 4 ≤ B + B 4 < 2 B . Q_{s+j}\le\alpha^j B+\frac{B}{4}\le B+\frac{B}{4}<2B. Q s + j ≤ α j B + 4 B ≤ B + 4 B < 2 B .
Hence τ B > s + ℓ − 1 \tau_B>s+\ell-1 τ B > s + ℓ − 1 , so the original increments and the stopped increments agree on the interval s , … , s + ℓ − 1 s,\ldots,s+\ell-1 s , … , s + ℓ − 1 . Iterating (206) gives
Q s + ℓ ≤ α ℓ Q s + ∑ j = 0 ℓ − 1 α ℓ − 1 − j Δ N s + j + 1 = α ℓ Q s + S ~ ℓ . Q_{s+\ell}\le\alpha^\ell Q_s+\sum_{j=0}^{\ell-1}\alpha^{\ell-1-j}\Delta N_{s+j+1}=\alpha^\ell Q_s+\widetilde S_\ell. Q s + ℓ ≤ α ℓ Q s + j = 0 ∑ ℓ − 1 α ℓ − 1 − j Δ N s + j + 1 = α ℓ Q s + S ℓ .
On E e p \mathcal{E}_{\mathrm{ep}} E ep , this yields
Q s + ℓ ≤ α ℓ B + B 4 . Q_{s+\ell}\le\alpha^\ell B+\frac{B}{4}. Q s + ℓ ≤ α ℓ B + 4 B .
This closes the induction. In particular,
Q s + ℓ < 2 B for every 0 ≤ ℓ ≤ L e p , Q_{s+\ell}<2B\qquad\text{for every }0\le\ell\le L_{\mathrm{ep}}, Q s + ℓ < 2 B for every 0 ≤ ℓ ≤ L ep ,
and therefore
τ B > s + L e p . \tau_B>s+L_{\mathrm{ep}}. τ B > s + L ep .
It remains to choose C e p C_{\mathrm{ep}} C ep so that the endpoint is at most B / 2 B/2 B /2 . Choose C e p C_{\mathrm{ep}} C ep large enough that
e − λ C e p ≤ 1 4 . e^{-\lambda C_{\mathrm{ep}}}\le\frac{1}{4}. e − λ C ep ≤ 4 1 .
Since
L e p = ⌈ C e p δ ⌉ , α = 1 − λ δ , L_{\mathrm{ep}}=\left\lceil\frac{C_{\mathrm{ep}}}{\delta}\right\rceil,\qquad\alpha=1-\lambda\delta, L ep = ⌈ δ C ep ⌉ , α = 1 − λ δ ,
we have, for d d d large enough,
α L e p ≤ e − λ δ L e p ≤ e − λ C e p ≤ 1 4 . \alpha^{L_{\mathrm{ep}}}\le e^{-\lambda\delta L_{\mathrm{ep}}}\le e^{-\lambda C_{\mathrm{ep}}}\le\frac{1}{4}. α L ep ≤ e − λ δ L ep ≤ e − λ C ep ≤ 4 1 .
Thus, on E e p \mathcal{E}_{\mathrm{ep}} E ep ,
Q s + L e p ≤ α L e p B + B 4 ≤ B 4 + B 4 = B 2 . Q_{s+L_{\mathrm{ep}}}\le\alpha^{L_{\mathrm{ep}}}B+\frac{B}{4}\le\frac{B}{4}+\frac{B}{4}=\frac{B}{2}. Q s + L ep ≤ α L ep B + 4 B ≤ 4 B + 4 B = 2 B .
Together with τ B > s + L e p \tau_B>s+L_{\mathrm{ep}} τ B > s + L ep , this gives
E e p ⊆ { Q s + L e p ≤ B 2 and τ B > s + L e p } . \mathcal{E}_{\mathrm{ep}}\subseteq\left\{Q_{s+L_{\mathrm{ep}}}\le\frac{B}{2}\ \text{and}\ \tau_B>s+L_{\mathrm{ep}}\right\}. E ep ⊆ { Q s + L ep ≤ 2 B and τ B > s + L ep } .
Since
P ( E e p ∣ F s ) ≥ 1 − C δ , \mathbb{P}(\mathcal{E}_{\mathrm{ep}}\mid\mathcal{F}_s)\ge1-C\delta, P ( E ep ∣ F s ) ≥ 1 − C δ ,
the proposition follows. □
Proposition D.25 (Iterated local contraction). Assume Proposition D.24. Let
δ = δ l o c = c l o c d \delta=\delta_{\mathrm{loc}}=\frac{c_{\mathrm{loc}}}{d} δ = δ loc = d c loc satisfy the local step-size condition, and let
L e p : = ⌈ C e p δ ⌉ L_{\mathrm{ep}}:=\left\lceil\frac{C_{\mathrm{ep}}}{\delta}\right\rceil L ep := ⌈ δ C ep ⌉ be the epoch length from Proposition D.24. Let s s s be a stopping time and suppose that, conditionally on F s \mathcal{F}_s F s ,
Q s ≤ B 0 , 0 < B 0 ≤ r l o c 2 32 . Q_s\le B_0,\qquad0<B_0\le\frac{r_{\mathrm{loc}}^2}{32}. Q s ≤ B 0 , 0 < B 0 ≤ 32 r loc 2 . For ε ∈ ( 0 , B 0 ] \varepsilon\in(0,B_0] ε ∈ ( 0 , B 0 ] , define
J ε : = ⌈ log 2 B 0 ε ⌉ , N ε : = J ε L e p . J_\varepsilon:=\left\lceil\log_2\frac{B_0}{\varepsilon}\right\rceil,\qquad N_\varepsilon:=J_\varepsilon L_{\mathrm{ep}}. J ε := ⌈ log 2 ε B 0 ⌉ , N ε := J ε L ep .
Then
P ( Q s + N ε ≤ ε and Q t ≤ 2 B 0 for every s ≤ t ≤ s + N ε ∣ F s ) ≥ 1 − C δ J ε . (207) \mathbb{P}\left(Q_{s+N_{\varepsilon}} \le\varepsilon\mathbin{\text{ and }} Q_t \le2B_0 \text{ for every } s \le t \le s+N_{\varepsilon} \mid\mathcal{F}_s\right) \ge1-C\delta J_{\varepsilon}. \tag*{(207)} P ( Q s + N ε ≤ ε and Q t ≤ 2 B 0 for every s ≤ t ≤ s + N ε ∣ F s ) ≥ 1 − C δ J ε . (207)
In particular, since δ = c l o c / d \delta=c_{\mathrm{loc}}/d δ = c loc / d ,
N ε ≤ C d ( 1 + log B 0 ε ) , (208) N_{\varepsilon} \le Cd\left(1+\log\frac{B_0}{\varepsilon}\right), \tag*{(208)} N ε ≤ C d ( 1 + log ε B 0 ) , (208)
and
P ( Q s + N ε ≤ ε ∣ F s ) ≥ 1 − C log ( 2 B 0 / ε ) d . (209) \mathbb{P}\left(Q_{s+N_{\varepsilon}}\le\varepsilon\mid\mathcal{F}_s\right)\ge1-C\frac{\log(2B_0/\varepsilon)}{d}. \tag*{(209)} P ( Q s + N ε ≤ ε ∣ F s ) ≥ 1 − C d log ( 2 B 0 / ε ) . (209)
Proof. If ε = B 0 \varepsilon=B_0 ε = B 0 , then J ε = 0 J_{\varepsilon}=0 J ε = 0 , N ε = 0 N_{\varepsilon}=0 N ε = 0 , and the claim follows from Q s ≤ B 0 = ε Q_s\le B_0=\varepsilon Q s ≤ B 0 = ε . Hence assume ε < B 0 \varepsilon<B_0 ε < B 0 , so J ε ≥ 1 J_{\varepsilon}\ge1 J ε ≥ 1 .
Set
s j : = s + j L e p , B j : = 2 − j B 0 , j = 0 , 1 , … , J ε . s_j:=s+jL_{\mathrm{ep}},\qquad B_j:=2^{-j}B_0,\qquad j=0,1,\ldots,J_{\varepsilon}. s j := s + j L ep , B j := 2 − j B 0 , j = 0 , 1 , … , J ε .
Since s s s is a stopping time, each s j s_j s j is also a stopping time. Moreover,
B j ≤ B 0 ≤ r l o c 2 32 , B_j\le B_0\le\frac{r_{\mathrm{loc}}^2}{32}, B j ≤ B 0 ≤ 32 r loc 2 ,
so Proposition D.24 is applicable at every epoch as soon as Q s j ≤ B j Q_{s_j}\le B_j Q s j ≤ B j .
For j = 0 , … , J ε − 1 j=0,\ldots,J_{\varepsilon}-1 j = 0 , … , J ε − 1 , define the epoch-success event
A j : = { Q s j + 1 ≤ B j + 1 and Q t ≤ 2 B j for every s j ≤ t ≤ s j + 1 } . A_j:=\left\{Q_{s_{j+1}}\le B_{j+1}\text{ and }Q_t\le2B_j\text{ for every }s_j\le t\le s_{j+1}\right\}. A j := { Q s j + 1 ≤ B j + 1 and Q t ≤ 2 B j for every s j ≤ t ≤ s j + 1 } .
Also define
G m : = ⋂ j = 0 m − 1 A j , m = 1 , … , J ε , G_m:=\bigcap_{j=0}^{m-1}A_j,\qquad m=1,\ldots,J_{\varepsilon}, G m := j = 0 ⋂ m − 1 A j , m = 1 , … , J ε ,
with G 0 G_0 G 0 equal to the whole probability space.
We claim that, for every m ≤ J ε m\le J_{\varepsilon} m ≤ J ε , on G m G_m G m ,
Q s m ≤ B m Q_{s_m}\le B_m Q s m ≤ B m
and
Q t ≤ 2 B 0 for every s ≤ t ≤ s m . Q_t\le2B_0\qquad\text{for every }s\le t\le s_m. Q t ≤ 2 B 0 for every s ≤ t ≤ s m .
Indeed, the first statement follows by induction from the endpoint conclusion in each A j A_j A j . The second follows because, on the j j j -th epoch,
Q t ≤ 2 B j ≤ 2 B 0 . Q_t\le2B_j\le2B_0. Q t ≤ 2 B j ≤ 2 B 0 .
Now fix j ∈ { 0 , … , J ε − 1 } j\in\{0,\ldots,J_{\varepsilon}-1\} j ∈ { 0 , … , J ε − 1 } . On G j G_j G j , we have Q s j ≤ B j Q_{s_j}\le B_j Q s j ≤ B j . Applying Proposition D.24 conditionally on F s j \mathcal{F}_{s_j} F s j gives
P ( A j c ∣ F s j ) ≤ C δ on G j . \mathbb{P}\left(A_j^{\mathrm{c}}\mid\mathcal{F}_{s_j}\right)\le C\delta\qquad\text{on }G_j. P ( A j c ∣ F s j ) ≤ C δ on G j .
Therefore,
P ( G j ∩ A j c ∣ F s ) = E [ 1 G j P ( A j c ∣ F s j ) ∣ F s ] ≤ C δ . \mathbb{P}\left(G_j\cap A_j^{\mathrm{c}}\mid\mathcal{F}_s\right)
=\mathbb{E}\left[\mathbf{1}_{G_j}\mathbb{P}\left(A_j^{\mathrm{c}}\mid\mathcal{F}_{s_j}\right)\mid\mathcal{F}_s\right]\le C\delta. P ( G j ∩ A j c ∣ F s ) = E [ 1 G j P ( A j c ∣ F s j ) ∣ F s ] ≤ C δ .
Taking a union bound over the J ε J_{\varepsilon} J ε epochs yields
P ( G J ε c ∣ F s ) ≤ C δ J ε . \mathbb{P}\left(G_{J_{\varepsilon}}^{\mathrm{c}}\mid\mathcal{F}_s\right)\le C\delta J_{\varepsilon}. P ( G J ε c ∣ F s ) ≤ C δ J ε .
Hence
P ( G J ε ∣ F s ) ≥ 1 − C δ J ε . \mathbb{P}\left(G_{J_{\varepsilon}}\mid\mathcal{F}_s\right)\ge1-C\delta J_{\varepsilon}. P ( G J ε ∣ F s ) ≥ 1 − C δ J ε .
On G J ε G_{J_{\varepsilon}} G J ε , we have
Q s + N ε = Q s J ε ≤ B J ε = 2 − J ε B 0 ≤ ε , Q_{s+N_{\varepsilon}}=Q_{s_{J_{\varepsilon}}}\le B_{J_{\varepsilon}}=2^{-J_{\varepsilon}}B_0\le\varepsilon, Q s + N ε = Q s J ε ≤ B J ε = 2 − J ε B 0 ≤ ε ,
by the definition of J ε J_{\varepsilon} J ε . Also, as shown above,
Q t ≤ 2 B 0 for every s ≤ t ≤ s + N ε . Q_t\le2B_0\qquad\text{for every }s\le t\le s+N_{\varepsilon}. Q t ≤ 2 B 0 for every s ≤ t ≤ s + N ε .
This proves the probability estimate.
It remains only to record the sample complexity. Since
L e p = ⌈ C e p δ ⌉ ≤ C δ L_{\mathrm{ep}}=\left\lceil\frac{C_{\mathrm{ep}}}{\delta}\right\rceil\le\frac{C}{\delta} L ep = ⌈ δ C ep ⌉ ≤ δ C
for d d d sufficiently large, and
J ε = ⌈ log 2 B 0 ε ⌉ ≤ C ( 1 + log B 0 ε ) , J_{\varepsilon}=\left\lceil\log_{2}\frac{B_{0}}{\varepsilon}\right\rceil\le C\left(1+\log\frac{B_{0}}{\varepsilon}\right), J ε = ⌈ log 2 ε B 0 ⌉ ≤ C ( 1 + log ε B 0 ) ,
we get
N ε = J ε L e p ≤ C δ − 1 ( 1 + log B 0 ε ) . N_{\varepsilon}=J_{\varepsilon}L_{\mathrm{ep}}\le C\delta^{-1}\left(1+\log\frac{B_{0}}{\varepsilon}\right). N ε = J ε L ep ≤ C δ − 1 ( 1 + log ε B 0 ) .
With δ = c l o c / d \delta=c_{\mathrm{loc}}/d δ = c loc / d , this becomes
N ε ≤ C d ( 1 + log B 0 ε ) . N_{\varepsilon}\le Cd\left(1+\log\frac{B_{0}}{\varepsilon}\right). N ε ≤ C d ( 1 + log ε B 0 ) .
Finally,
δ J ε ≤ C d log 2 B 0 ε , \delta J_{\varepsilon}\le\frac{C}{d}\log\frac{2B_{0}}{\varepsilon}, δ J ε ≤ d C log ε 2 B 0 ,
after increasing C C C . This gives
P ( Q s + N ε ≤ ε ∣ F s ) ≥ 1 − C log ( 2 B 0 / ε ) d . \mathbb{P}\left(Q_{s+N_{\varepsilon}}\le\varepsilon\mid\mathcal{F}_{s}\right)\ge1-C\frac{\log(2B_{0}/\varepsilon)}{d}. P ( Q s + N ε ≤ ε ∣ F s ) ≥ 1 − C d log ( 2 B 0 / ε ) .
The proof is complete.
Theorem D.26 (Discrete-time weak-to-strong recovery). *Assume Theorem D.17 holds with event G T w e a k G_{T}^{\mathrm{weak}} G T weak , stopping time t e s c t_{\mathrm{esc}} t esc , and constants T e s c T_{\mathrm{esc}} T esc , c e s c c_{\mathrm{esc}} c esc , C η , e s c C_{\eta,\mathrm{esc}} C η , esc , such that on G T w e a k G_{T}^{\mathrm{weak}} G T weak ,
t e s c ≤ T e s c , min { ρ t e s c , χ t e s c } ≥ c e s c , η t e s c ≤ C η , e s c log d d . t_{\mathrm{esc}}\le T_{\mathrm{esc}},\qquad\min\{\rho_{t_{\mathrm{esc}}},\chi_{t_{\mathrm{esc}}}\}\ge c_{\mathrm{esc}},\qquad\eta_{t_{\mathrm{esc}}}\le C_{\eta,\mathrm{esc}}\frac{\log d}{\sqrt{d}}. t esc ≤ T esc , min { ρ t esc , χ t esc } ≥ c esc , η t esc ≤ C η , esc d log d . *Assume (A4). After t e s c t_{\mathrm{esc}} t esc , switch to the local step size δ l o c = c l o c d \delta_{\mathrm{loc}}=\frac{c_{\mathrm{loc}}}{d} δ loc = d c loc , where c l o c > 0 c_{\mathrm{loc}}>0 c loc > 0 is chosen sufficiently small so that the assumptions of Lemma D.19, Proposition D.21, and Proposition D.24 hold. Let
Q i n : = r l o c 2 64 . Q_{\mathrm{in}}:=\frac{r_{\mathrm{loc}}^{2}}{64}. Q in := 64 r loc 2 . *Then, for every fixed or polynomially small ν i n \nu_{\mathrm{in}} ν in , there exists an event E s t r E_{\mathrm{str}} E str satisfying
P ( E s t r ) ≥ 1 − P ( G T w e a k c ) − ν i n , (210) \mathbb{P}(E_{\mathrm{str}})\ge1-\mathbb{P}(G_{T}^{\mathrm{weak}\,c})-\nu_{\mathrm{in}}, \tag*{(210)} P ( E str ) ≥ 1 − P ( G T weak c ) − ν in , (210) on which there is a stopping time t 1 t_{1} t 1 such that
t 1 ≤ t e s c + C e n t δ l o c − 1 ≤ T e s c + C e n t δ l o c − 1 , (211) t_{1}\le t_{\mathrm{esc}}+C_{\mathrm{ent}}\delta_{\mathrm{loc}}^{-1}\le T_{\mathrm{esc}}+C_{\mathrm{ent}}\delta_{\mathrm{loc}}^{-1}, \tag*{(211)} t 1 ≤ t esc + C ent δ loc − 1 ≤ T esc + C ent δ loc − 1 , (211) and
Q t 1 ≤ Q i n ≤ r l o c 2 32 . (212) Q_{t_{1}}\le Q_{\mathrm{in}}\le\frac{r_{\mathrm{loc}}^{2}}{32}. \tag*{(212)} Q t 1 ≤ Q in ≤ 32 r loc 2 . (212) *Moreover, for every ε ∈ ( 0 , Q i n ] \varepsilon\in(0,Q_{\mathrm{in}}] ε ∈ ( 0 , Q in ] , define
J ε : = ⌈ log 2 Q i n ε ⌉ , L e p : = ⌈ C e p δ l o c ⌉ , N ε : = J ε L e p . J_{\varepsilon}:=\left\lceil\log_{2}\frac{Q_{\mathrm{in}}}{\varepsilon}\right\rceil,\qquad L_{\mathrm{ep}}:=\left\lceil\frac{C_{\mathrm{ep}}}{\delta_{\mathrm{loc}}}\right\rceil,\qquad N_{\varepsilon}:=J_{\varepsilon}L_{\mathrm{ep}}. J ε := ⌈ log 2 ε Q in ⌉ , L ep := ⌈ δ loc C ep ⌉ , N ε := J ε L ep . *Then
P ( Q t 1 + N ε ≤ ε ) ≥ 1 − P ( G T w e a k c ) − ν i n − C δ l o c J ε . (213) \mathbb{P}(Q_{t_{1}+N_{\varepsilon}}\le\varepsilon)\ge1-\mathbb{P}(G_{T}^{\mathrm{weak}\,c})-\nu_{\mathrm{in}}-C\delta_{\mathrm{loc}}J_{\varepsilon}. \tag*{(213)} P ( Q t 1 + N ε ≤ ε ) ≥ 1 − P ( G T weak c ) − ν in − C δ loc J ε . (213) *Equivalently, since δ l o c = c l o c / d \delta_{\mathrm{loc}}=c_{\mathrm{loc}}/d δ loc = c loc / d ,
P ( Q t 1 + N ε ≤ ε ) ≥ 1 − P ( G T w e a k c ) − ν i n − C log ( 2 Q i n / ε ) d . \mathbb{P}(Q_{t_{1}+N_{\varepsilon}}\le\varepsilon)\ge1-\mathbb{P}(G_{T}^{\mathrm{weak}\,c})-\nu_{\mathrm{in}}-C\frac{\log(2Q_{\mathrm{in}}/\varepsilon)}{d}. P ( Q t 1 + N ε ≤ ε ) ≥ 1 − P ( G T weak c ) − ν in − C d log ( 2 Q in / ε ) . *On the same event,
1 − ρ t 1 + N ε ≤ ε , 1 − χ t 1 + N ε ≤ ε , 1-\rho_{t_{1}+N_{\varepsilon}}\le\varepsilon,\qquad1-\chi_{t_{1}+N_{\varepsilon}}\le\varepsilon, 1 − ρ t 1 + N ε ≤ ε , 1 − χ t 1 + N ε ≤ ε ,
and hence
min { ρ t 1 + N ε , χ t 1 + N ε } ≥ 1 − ε . (214) \min\{\rho_{t_1+N_\varepsilon},\chi_{t_1+N_\varepsilon}\}\ge1-\varepsilon. \tag*{(214)} min { ρ t 1 + N ε , χ t 1 + N ε } ≥ 1 − ε . (214)
In particular,
N ε ≤ C d ( 1 + log Q i n ε ) . N_\varepsilon\le Cd\left(1+\log\frac{Q_{\mathrm{in}}}{\varepsilon}\right). N ε ≤ C d ( 1 + log ε Q in ) .
With the weak phase satisfies T e s c = O ( d log 2 d ) T_{\mathrm{esc}}=O(d\log^2 d) T esc = O ( d log 2 d ) , the total number of SGD samples needed to reach Q ≤ ε Q\le\varepsilon Q ≤ ε is
O ( d log 2 d ) + O ( d log 1 ε ) . (215) O(d\log^2 d)+O\left(d\log\frac{1}{\varepsilon}\right). \tag*{(215)} O ( d log 2 d ) + O ( d log ε 1 ) . (215)
Proof. Let
t 0 : = t e s c , δ : = δ l o c . t_0:=t_{\mathrm{esc}},\qquad\delta:=\delta_{\mathrm{loc}}. t 0 := t esc , δ := δ loc .
Work first on the weak-recovery event G T w e a k G_T^{\mathrm{weak}} G T weak . On this event,
min { ρ t 0 , χ t 0 } ≥ c e s c , η t 0 ≤ C η , e s c log d d . \min\{\rho_{t_0},\chi_{t_0}\}\ge c_{\mathrm{esc}},\qquad
\eta_{t_0}\le C_{\eta,\mathrm{esc}}\frac{\log d}{\sqrt d}. min { ρ t 0 , χ t 0 } ≥ c esc , η t 0 ≤ C η , esc d log d .
Together with the incoherence bound
∥ θ ∗ ∥ ∞ ≤ C ∗ log d d , \|\theta^*\|_\infty\le C_*\sqrt{\frac{\log d}{d}}, ∥ θ ∗ ∥ ∞ ≤ C ∗ d log d ,
these are exactly the assumptions of Lemma D.19 .
Apply Lemma D.19 with C i n ≥ C e n t C_{\mathrm{in}}\ge C_{\mathrm{ent}} C in ≥ C ent . Conditionally on F t 0 \mathcal{F}_{t_0} F t 0 , there is an event E i n E_{\mathrm{in}} E in such that
P ( E i n ∣ F t 0 ) ≥ 1 − ν i n . \mathbb{P}(E_{\mathrm{in}}\mid\mathcal{F}_{t_0})\ge1-\nu_{\mathrm{in}}. P ( E in ∣ F t 0 ) ≥ 1 − ν in .
Hence, by the tower property,
P ( G T w e a k ∩ E i n ) = E [ 1 G T w e a k P ( E i n ∣ F t 0 ) ] ≥ ( 1 − ν i n ) P ( G T w e a k ) . \mathbb{P}(G_T^{\mathrm{weak}}\cap E_{\mathrm{in}})
=\mathbb{E}\left[\mathbf{1}_{G_T^{\mathrm{weak}}}\mathbb{P}(E_{\mathrm{in}}\mid\mathcal{F}_{t_0})\right]
\ge(1-\nu_{\mathrm{in}})\mathbb{P}(G_T^{\mathrm{weak}}). P ( G T weak ∩ E in ) = E [ 1 G T weak P ( E in ∣ F t 0 ) ] ≥ ( 1 − ν in ) P ( G T weak ) .
Therefore
P ( G T w e a k ∩ E i n ) ≥ 1 − P ( ( G T w e a k ) c ) − ν i n . \mathbb{P}(G_T^{\mathrm{weak}}\cap E_{\mathrm{in}})
\ge1-\mathbb{P}\left((G_T^{\mathrm{weak}})^c\right)-\nu_{\mathrm{in}}. P ( G T weak ∩ E in ) ≥ 1 − P ( ( G T weak ) c ) − ν in .
On G T w e a k ∩ E i n G_T^{\mathrm{weak}}\cap E_{\mathrm{in}} G T weak ∩ E in , Proposition D.21 gives a time
t 1 ≤ t e s c + C e n t δ − 1 t_1\le t_{\mathrm{esc}}+C_{\mathrm{ent}}\delta^{-1} t 1 ≤ t esc + C ent δ − 1
such that
ρ t 1 ≥ cos ( r l o c / 8 ) , χ t 1 ≥ cos ( r l o c / 8 ) , \rho_{t_1}\ge\cos(r_{\mathrm{loc}}/8),\qquad
\chi_{t_1}\ge\cos(r_{\mathrm{loc}}/8), ρ t 1 ≥ cos ( r loc /8 ) , χ t 1 ≥ cos ( r loc /8 ) ,
and consequently
Q t 1 ≤ r l o c 2 64 = Q i n . Q_{t_1}\le\frac{r_{\mathrm{loc}}^2}{64}=Q_{\mathrm{in}}. Q t 1 ≤ 64 r loc 2 = Q in .
Choosing t 1 t_1 t 1 as the stopping time constructed in Proposition D.21 , this proves the first part. Since t e s c ≤ T e s c t_{\mathrm{esc}}\le T_{\mathrm{esc}} t esc ≤ T esc on G T w e a k G_T^{\mathrm{weak}} G T weak , we also have
t 1 ≤ T e s c + C e n t δ − 1 . t_1\le T_{\mathrm{esc}}+C_{\mathrm{ent}}\delta^{-1}. t 1 ≤ T esc + C ent δ − 1 .
Set
E s t r ( 0 ) : = G T w e a k ∩ E i n . E_{\mathrm{str}}^{(0)}:=G_T^{\mathrm{weak}}\cap E_{\mathrm{in}}. E str ( 0 ) := G T weak ∩ E in .
Then
P ( E s t r ( 0 ) ) ≥ 1 − P ( ( G T w e a k ) c ) − ν i n . \mathbb{P}(E_{\mathrm{str}}^{(0)})
\ge1-\mathbb{P}\left((G_T^{\mathrm{weak}})^c\right)-\nu_{\mathrm{in}}. P ( E str ( 0 ) ) ≥ 1 − P ( ( G T weak ) c ) − ν in .
Now fix ε ∈ ( 0 , Q i n ] \varepsilon\in(0,Q_{\mathrm{in}}] ε ∈ ( 0 , Q in ] . On E s t r ( 0 ) E_{\mathrm{str}}^{(0)} E str ( 0 ) , we may apply Proposition D.25 conditionally on F t 1 \mathcal{F}_{t_1} F t 1 , with
s = t 1 , B 0 = Q i n . s=t_1,\qquad B_0=Q_{\mathrm{in}}. s = t 1 , B 0 = Q in .
Indeed,
Q t 1 ≤ Q i n ≤ r l o c 2 32 . Q_{t_1}\le Q_{\mathrm{in}}\le\frac{r_{\mathrm{loc}}^2}{32}. Q t 1 ≤ Q in ≤ 32 r loc 2 .
Thus, conditionally on F t 1 \mathcal{F}_{t_1} F t 1 ,
P ( Q t 1 + N ε ≤ ε ∣ F t 1 ) ≥ 1 − C δ J ε . \mathbb{P}\left(Q_{t_1+N_\varepsilon}\le\varepsilon\mid\mathcal{F}_{t_1}\right)\ge1-C\delta J_\varepsilon. P ( Q t 1 + N ε ≤ ε ∣ F t 1 ) ≥ 1 − C δ J ε .
Using the tower property again,
P ( E s t r ( 0 ) ∩ { Q t 1 + N ε > ε } ) ≤ C δ J ε . \mathbb{P}\left(E_{\mathrm{str}}^{(0)}\cap\{Q_{t_1+N_\varepsilon}>\varepsilon\}\right)\le C\delta J_\varepsilon. P ( E str ( 0 ) ∩ { Q t 1 + N ε > ε } ) ≤ C δ J ε .
Therefore
P ( Q t 1 + N ε ≤ ε ) ≥ P ( E s t r ( 0 ) ) − C δ J ε . \mathbb{P}\left(Q_{t_1+N_\varepsilon}\le\varepsilon\right)\ge\mathbb{P}\left(E_{\mathrm{str}}^{(0)}\right)-C\delta J_\varepsilon. P ( Q t 1 + N ε ≤ ε ) ≥ P ( E str ( 0 ) ) − C δ J ε .
Combining this with the lower bound on P ( E s t r ( 0 ) ) \mathbb{P}\left(E_{\mathrm{str}}^{(0)}\right) P ( E str ( 0 ) ) gives
P ( Q t 1 + N ε ≤ ε ) ≥ 1 − P ( G T w e a k , c ) − ν i n − C δ J ε . \mathbb{P}\left(Q_{t_1+N_\varepsilon}\le\varepsilon\right)\ge1-\mathbb{P}\left(G_T^{\mathrm{weak},c}\right)-\nu_{\mathrm{in}}-C\delta J_\varepsilon. P ( Q t 1 + N ε ≤ ε ) ≥ 1 − P ( G T weak , c ) − ν in − C δ J ε .
Since
δ = c l o c d \delta=\frac{c_{\mathrm{loc}}}{d} δ = d c loc
and
J ε = ⌈ log 2 Q i n ε ⌉ ≤ C log 2 Q i n ε , J_\varepsilon=\left\lceil\log_2\frac{Q_{\mathrm{in}}}{\varepsilon}\right\rceil\le C\log\frac{2Q_{\mathrm{in}}}{\varepsilon}, J ε = ⌈ log 2 ε Q in ⌉ ≤ C log ε 2 Q in ,
we obtain
C δ J ε ≤ C log ( 2 Q i n / ε ) d . C\delta J_\varepsilon\le C\frac{\log(2Q_{\mathrm{in}}/\varepsilon)}{d}. C δ J ε ≤ C d log ( 2 Q in / ε ) .
It remains to convert the Q Q Q -bound into the overlap bound. If
Q t 1 + N ε ≤ ε , Q_{t_1+N_\varepsilon}\le\varepsilon, Q t 1 + N ε ≤ ε ,
then
dist S ( θ t 1 + N ε , θ ∗ ) 2 + dist S ( β t 1 + N ε , β ∗ ) 2 = 2 Q t 1 + N ε ≤ 2 ε . \operatorname{dist}_{S}(\theta_{t_1+N_\varepsilon},\theta^*)^2+\operatorname{dist}_{S}(\beta_{t_1+N_\varepsilon},\beta^*)^2=2Q_{t_1+N_\varepsilon}\le2\varepsilon. dist S ( θ t 1 + N ε , θ ∗ ) 2 + dist S ( β t 1 + N ε , β ∗ ) 2 = 2 Q t 1 + N ε ≤ 2 ε .
In particular,
dist S ( θ t 1 + N ε , θ ∗ ) 2 ≤ 2 ε , dist S ( β t 1 + N ε , β ∗ ) 2 ≤ 2 ε . \operatorname{dist}_{S}(\theta_{t_1+N_\varepsilon},\theta^*)^2\le2\varepsilon,\qquad\operatorname{dist}_{S}(\beta_{t_1+N_\varepsilon},\beta^*)^2\le2\varepsilon. dist S ( θ t 1 + N ε , θ ∗ ) 2 ≤ 2 ε , dist S ( β t 1 + N ε , β ∗ ) 2 ≤ 2 ε .
Using 1 − cos x ≤ x 2 / 2 1-\cos x\le x^2/2 1 − cos x ≤ x 2 /2 , we get
1 − χ t 1 + N ε = 1 − cos ( dist S ( θ t 1 + N ε , θ ∗ ) ) ≤ ε , 1-\chi_{t_1+N_\varepsilon}=1-\cos\left(\operatorname{dist}_{S}(\theta_{t_1+N_\varepsilon},\theta^*)\right)\le\varepsilon, 1 − χ t 1 + N ε = 1 − cos ( dist S ( θ t 1 + N ε , θ ∗ ) ) ≤ ε ,
and similarly
1 − ρ t 1 + N ε ≤ ε . 1-\rho_{t_1+N_\varepsilon}\le\varepsilon. 1 − ρ t 1 + N ε ≤ ε .
Thus
min { ρ t 1 + N ε , χ t 1 + N ε } ≥ 1 − ε . \min\{\rho_{t_1+N_\varepsilon},\chi_{t_1+N_\varepsilon}\}\ge1-\varepsilon. min { ρ t 1 + N ε , χ t 1 + N ε } ≥ 1 − ε .
Finally, Proposition D.25 gives
N ε = J ε L e p ≤ C d ( 1 + log Q i n ε ) . N_\varepsilon=J_\varepsilon L_{\mathrm{ep}}\le Cd\left(1+\log\frac{Q_{\mathrm{in}}}{\varepsilon}\right). N ε = J ε L ep ≤ C d ( 1 + log ε Q in ) .
Since
t 1 ≤ t e s c + C e n t δ − 1 = t e s c + O ( d ) , t_1\le t_{\mathrm{esc}}+C_{\mathrm{ent}}\delta^{-1}=t_{\mathrm{esc}}+O(d), t 1 ≤ t esc + C ent δ − 1 = t esc + O ( d ) ,
the total number of samples after initialization is bounded by
t e s c + O ( d ) + O ( d log Q i n ε ) . t_{\mathrm{esc}}+O(d)+O\left(d\log\frac{Q_{\mathrm{in}}}{\varepsilon}\right). t esc + O ( d ) + O ( d log ε Q in ) .
Because Q i n = r l o c 2 / 64 Q_{\mathrm{in}}=r_{\mathrm{loc}}^2/64 Q in = r loc 2 /64 is a fixed constant independent of d d d , this is
t e s c + O ( d log 1 ε ) . t_{\mathrm{esc}}+O\left(d\log\frac{1}{\varepsilon}\right). t esc + O ( d log ε 1 ) .
If T e s c = O ( d log 2 d ) T_{\mathrm{esc}}=O(d\log^2d) T esc = O ( d log 2 d ) , then on G T w e a k G_T^{\mathrm{weak}} G T weak ,
t 1 + N ε ≤ O ( d log 2 d ) + O ( d log 1 ε ) . t_1+N_\varepsilon\le O(d\log^2d)+O\left(d\log\frac{1}{\varepsilon}\right). t 1 + N ε ≤ O ( d log 2 d ) + O ( d log ε 1 ) .
The proof is complete.
Auxiliary Lemmas
Stein Bounds
Here, we present the comparison bounds between the Riemannian Hessian of the true loss and that of the Gaussian model via a Stein-type approximation, for θ \theta θ -Hessian and β \beta β -Hessian respectively. They are used in the Lemma C.12. We prove our results based on this theorem from [8 ] :
Theorem E.1 (Stein coupling for product measures [8 Thm. D.1] ). Let X = ( X 1 , … , X d ) X=(X_1,\ldots,X_d) X = ( X 1 , … , X d ) be a random vector with independent coordinates such that E [ X ] = 0 \mathbb{E}[X]=0 E [ X ] = 0 , E [ X X T ] = I d \mathbb{E}[XX^\mathsf{T}]=I_d E [ X X T ] = I d , and E ∣ X i ∣ 3 = τ i 3 < ∞ \mathbb{E}|X_i|^3=\tau_i^3<\infty E ∣ X i ∣ 3 = τ i 3 < ∞ . Let G ∼ N ( 0 , I d ) G\sim\mathcal{N}(0,I_d) G ∼ N ( 0 , I d ) be standard Gaussian. If h : R d → R h:\mathbb{R}^d\to\mathbb{R} h : R d → R is three-times differentiable, then
∣ E [ h ( X ) ] − E [ h ( G ) ] ∣ ≤ 5 6 ∑ i = 1 d τ i 3 ∥ ∂ x i 3 h ∥ ∞ . (216) \left|\mathbb{E}[h(X)]-\mathbb{E}[h(G)]\right|\leq\frac{5}{6}\sum_{i=1}^{d}\tau_i^3\left\|\partial_{x_i}^3h\right\|_\infty. \tag*{(216)} ∣ E [ h ( X )] − E [ h ( G )] ∣ ≤ 6 5 i = 1 ∑ d τ i 3 ∂ x i 3 h ∞ . (216)
Stein Bound for θ \theta θ -Hessian
Corollary E.2 (Block Stein coupling with moment remainder). Let ( X i , X i ∗ ) i = 1 d (X_i,X_i^*)_{i=1}^d ( X i , X i ∗ ) i = 1 d be independent R 2 \mathbb{R}^2 R 2 -valued random vectors with
E [ X i ] = E [ X i ∗ ] = 0 , E [ X i 2 ] = E [ ( X i ∗ ) 2 ] = 1 , E [ X i X i ∗ ] = m , \mathbb{E}[X_i]=\mathbb{E}[X_i^*]=0,\qquad\mathbb{E}[X_i^2]=\mathbb{E}[(X_i^*)^2]=1,\qquad\mathbb{E}[X_iX_i^*]=m, E [ X i ] = E [ X i ∗ ] = 0 , E [ X i 2 ] = E [( X i ∗ ) 2 ] = 1 , E [ X i X i ∗ ] = m , and τ : = max i ( E ∣ X i ∣ 5 + E ∣ X i ∗ ∣ 5 ) < ∞ \tau:=\max_i(\mathbb{E}|X_i|^5+\mathbb{E}|X_i^*|^5)<\infty τ := max i ( E ∣ X i ∣ 5 + E ∣ X i ∗ ∣ 5 ) < ∞ . Let ( G i , G i ∗ ) i = 1 d (G_i,G_i^*)_{i=1}^d ( G i , G i ∗ ) i = 1 d be iid N ( 0 , Σ m ) \mathcal{N}(0,\Sigma_m) N ( 0 , Σ m ) with Σ m = ( 1 m m 1 ) \Sigma_m=\begin{pmatrix}1&m\\m&1\end{pmatrix} Σ m = ( 1 m m 1 ) . If h : R d × R d → R h:\mathbb{R}^d\times\mathbb{R}^d\to\mathbb{R} h : R d × R d → R is three-times differentiable in each block ( x i , x i ∗ ) (x_i,x_i^*) ( x i , x i ∗ ) , then
∣ E [ h ( X , X ∗ ) ] − E [ h ( G , G ∗ ) ] ∣ ≤ C ( τ ) ∑ i = 1 d ∑ a , b ≥ 0 a + b = 3 sup j ≤ d E [ ∣ ∂ a + b h ∂ x i a ∂ ( x i ∗ ) b ( W ( j ) , W ∗ ( j ) ) ∣ ] . (217) \left|\mathbb{E}[h(X,X^*)]-\mathbb{E}[h(G,G^*)]\right|
\leq C(\tau)\sum_{i=1}^{d}\sum_{\substack{a,b\geq0\\a+b=3}}\sup_{j\leq d}\mathbb{E}\left[\left|\frac{\partial^{a+b}h}{\partial x_i^a\partial(x_i^*)^b}(W^{(j)},W^{*(j)})\right|\right]. \tag*{(217)} ∣ E [ h ( X , X ∗ )] − E [ h ( G , G ∗ )] ∣ ≤ C ( τ ) i = 1 ∑ d a , b ≥ 0 a + b = 3 ∑ j ≤ d sup E [ ∂ x i a ∂ ( x i ∗ ) b ∂ a + b h ( W ( j ) , W ∗ ( j ) ) ] . (217) where ( W ( j ) , W ∗ ( j ) ) (W^{(j)},W^{*(j)}) ( W ( j ) , W ∗ ( j ) ) is the hybrid that agrees with ( G , G ∗ ) (G,G^*) ( G , G ∗ ) on blocks 1 , … , j 1,\ldots,j 1 , … , j and with ( X , X ∗ ) (X,X^*) ( X , X ∗ ) on blocks j + 1 , … , d j+1,\ldots,d j + 1 , … , d .
Proof. We have ( W ( 0 ) , W ∗ ( 0 ) ) = ( X , X ∗ ) (W^{(0)},W^{*(0)})=(X,X^*) ( W ( 0 ) , W ∗ ( 0 ) ) = ( X , X ∗ ) and ( W ( d ) , W ∗ ( d ) ) = ( G , G ∗ ) (W^{(d)},W^{*(d)})=(G,G^*) ( W ( d ) , W ∗ ( d ) ) = ( G , G ∗ ) , so
E [ h ( X , X ∗ ) ] − E [ h ( G , G ∗ ) ] = ∑ j = 1 d ( E [ h ( W ( j − 1 ) , W ∗ ( j − 1 ) ) ] − E [ h ( W ( j ) , W ∗ ( j ) ) ] ) . (218) \mathbb{E}[h(X,X^*)]-\mathbb{E}[h(G,G^*)]
=\sum_{j=1}^{d}\left(\mathbb{E}[h(W^{(j-1)},W^{*(j-1)})]-\mathbb{E}[h(W^{(j)},W^{*(j)})]\right). \tag*{(218)} E [ h ( X , X ∗ )] − E [ h ( G , G ∗ )] = j = 1 ∑ d ( E [ h ( W ( j − 1 ) , W ∗ ( j − 1 ) )] − E [ h ( W ( j ) , W ∗ ( j ) )] ) . (218)
Each summand differs only in block j j j . With all other blocks frozen, define φ j ( s , t ) : = h ( W ( j ) ) ( s , t ) in block j \varphi_j(s,t):=h(W^{(j)})_{(s,t)\text{ in block }j} φ j ( s , t ) := h ( W ( j ) ) ( s , t ) in block j , so the j j j -th summand equals E [ φ j ( X j , X j ∗ ) − φ j ( G j , G j ∗ ) ] \mathbb{E}[\varphi_j(X_j,X_j^*)-\varphi_j(G_j,G_j^*)] E [ φ j ( X j , X j ∗ ) − φ j ( G j , G j ∗ )] . Taylor-expand φ j \varphi_j φ j around ( 0 , 0 ) (0,0) ( 0 , 0 ) to second order with integral remainder:
φ j ( s , t ) = ∑ a , b ≥ 0 a + b ≤ 2 1 a ! b ! ∂ a + b φ j ∂ s a ∂ t b ∣ ( 0 , 0 ) s a t b + R j ( s , t ) , \varphi_j(s,t)=\sum_{\substack{a,b\geq0\\a+b\leq2}}\frac{1}{a!b!}\left.\frac{\partial^{a+b}\varphi_j}{\partial s^a\partial t^b}\right|_{(0,0)}s^at^b+R_j(s,t), φ j ( s , t ) = a , b ≥ 0 a + b ≤ 2 ∑ a ! b ! 1 ∂ s a ∂ t b ∂ a + b φ j ( 0 , 0 ) s a t b + R j ( s , t ) ,
where the integral form of the remainder gives
∣ R j ( s , t ) ∣ ≤ C 0 ( ∣ s ∣ + ∣ t ∣ ) 3 ∑ a , b ≥ 0 a + b = 3 sup ∣ ξ ∣ ≤ ∣ s ∣ , ∣ ζ ∣ ≤ ∣ t ∣ ∣ ∂ a + b φ j ∂ s a ∂ t b ( ξ , ζ ) ∣ . |R_j(s,t)|\leq C_0(|s|+|t|)^3\sum_{\substack{a,b\geq0\\a+b=3}}\sup_{\substack{|\xi|\leq|s|,\,|\zeta|\leq|t|}}\left|\frac{\partial^{a+b}\varphi_j}{\partial s^a\partial t^b}(\xi,\zeta)\right|. ∣ R j ( s , t ) ∣ ≤ C 0 ( ∣ s ∣ + ∣ t ∣ ) 3 a , b ≥ 0 a + b = 3 ∑ ∣ ξ ∣ ≤ ∣ s ∣ , ∣ ζ ∣ ≤ ∣ t ∣ sup ∂ s a ∂ t b ∂ a + b φ j ( ξ , ζ ) .
Taking E [ φ j ( X j , X j ∗ ) ] − E [ φ j ( G j , G j ∗ ) ] \mathbb{E}[\varphi_j(X_j,X_j^*)]-\mathbb{E}[\varphi_j(G_j,G_j^*)] E [ φ j ( X j , X j ∗ )] − E [ φ j ( G j , G j ∗ )] : the zeroth-order terms cancel trivially; the first-order terms cancel since E [ X j ] = E [ G j ] = 0 \mathbb{E}[X_j]=\mathbb{E}[G_j]=0 E [ X j ] = E [ G j ] = 0 and E [ X j ∗ ] = E [ G j ∗ ] = 0 \mathbb{E}[X_j^*]=\mathbb{E}[G_j^*]=0 E [ X j ∗ ] = E [ G j ∗ ] = 0 ; the second-order terms cancel since
E [ X j 2 ] = E [ G j 2 ] = 1 , E [ ( X j ∗ ) 2 ] = E [ ( G j ∗ ) 2 ] = 1 , E [ X j X j ∗ ] = E [ G j G j ∗ ] = m . \mathbb{E}[X_j^2]=\mathbb{E}[G_j^2]=1,\qquad\mathbb{E}[(X_j^*)^2]=\mathbb{E}[(G_j^*)^2]=1,\qquad\mathbb{E}[X_jX_j^*]=\mathbb{E}[G_jG_j^*]=m. E [ X j 2 ] = E [ G j 2 ] = 1 , E [( X j ∗ ) 2 ] = E [( G j ∗ ) 2 ] = 1 , E [ X j X j ∗ ] = E [ G j G j ∗ ] = m .
Hence only the remainder contributes:
∣ E [ φ j ( X j , X j ∗ ) − φ j ( G j , G j ∗ ) ] ∣ ≤ C 0 ( E ( ∣ X j ∣ + ∣ X j ∗ ∣ ) 3 + E ( ∣ G j ∣ + ∣ G j ∗ ∣ ) 3 ) ∑ a , b ≥ 0 a + b = 3 E [ ∣ ∂ a + b φ j ∂ s a ∂ t b ∣ ] . \left|\mathbb{E}[\varphi_j(X_j,X_j^*)-\varphi_j(G_j,G_j^*)]\right|
\leq C_0\left(\mathbb{E}(|X_j|+|X_j^*|)^3+\mathbb{E}(|G_j|+|G_j^*|)^3\right)
\sum_{\substack{a,b\geq0\\a+b=3}}\mathbb{E}\left[\left|\frac{\partial^{a+b}\varphi_j}{\partial s^a\partial t^b}\right|\right]. E [ φ j ( X j , X j ∗ ) − φ j ( G j , G j ∗ )] ≤ C 0 ( E ( ∣ X j ∣ + ∣ X j ∗ ∣ ) 3 + E ( ∣ G j ∣ + ∣ G j ∗ ∣ ) 3 ) a , b ≥ 0 a + b = 3 ∑ E [ ∂ s a ∂ t b ∂ a + b φ j ] .
where the moment prefactors are bounded by C ′ ( τ ) C'(\tau) C ′ ( τ ) , and the expectations of the third-order partials use Hölder to separate the ( s , t ) (s,t) ( s , t ) -dependence from the frozen blocks. Summing over j j j and noting ∂ a + b φ j / ∂ s a ∂ t b = ∂ a + b h / ∂ x j a ∂ ( x j ∗ ) b \partial^{a+b}\varphi_j/\partial s^a\partial t^b=\partial^{a+b}h/\partial x_j^a\partial(x_j^*)^b ∂ a + b φ j / ∂ s a ∂ t b = ∂ a + b h / ∂ x j a ∂ ( x j ∗ ) b evaluated at the hybrid yields (209). ∎
Corollary E.3 (Stein estimate for a weighted quadratic statistic). Let X 1 , … , X d X_1,\ldots,X_d X 1 , … , X d be independent, centered, unit-variance random variables with sup i E ∣ X i ∣ 8 < ∞ \sup_i \mathbb{E}|X_i|^8<\infty sup i E ∣ X i ∣ 8 < ∞ , and let Γ ∼ N ( 0 , I d ) \Gamma\sim N(0,I_d) Γ ∼ N ( 0 , I d ) . Let a , u ∈ R d a,u\in\mathbb{R}^d a , u ∈ R d satisfy ∥ a ∥ 2 = ∥ u ∥ 2 = 1 \|a\|_2=\|u\|_2=1 ∥ a ∥ 2 = ∥ u ∥ 2 = 1 . If F ∈ C 3 ( R ) F\in C^3(\mathbb{R}) F ∈ C 3 ( R ) and ∥ F ( k ) ∥ ∞ < ∞ \|F^{(k)}\|_\infty<\infty ∥ F ( k ) ∥ ∞ < ∞ for 1 ≤ k ≤ 3 1\le k\le3 1 ≤ k ≤ 3 , then
∣ E [ F ( a ⋅ X ) ( u ⋅ X ) 2 ] − E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] ∣ ≤ C F ( ∑ i = 1 d ∣ a i ∣ u i 2 + ∑ i = 1 d a i 2 ∣ u i ∣ + ∑ i = 1 d ∣ a i ∣ 3 ) . \left|\mathbb{E}\left[F(a\cdot X)(u\cdot X)^2\right]-\mathbb{E}\left[F(a\cdot\Gamma)(u\cdot\Gamma)^2\right]\right|\le C_F\left(\sum_{i=1}^{d}|a_i|u_i^2+\sum_{i=1}^{d}a_i^2|u_i|+\sum_{i=1}^{d}|a_i|^3\right). E [ F ( a ⋅ X ) ( u ⋅ X ) 2 ] − E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] ≤ C F ( i = 1 ∑ d ∣ a i ∣ u i 2 + i = 1 ∑ d a i 2 ∣ u i ∣ + i = 1 ∑ d ∣ a i ∣ 3 ) . where C F C_F C F depends only on F F F and the law of X 1 X_1 X 1 , but not on d , a , u d,a,u d , a , u . In particular,
∣ E [ F ( a ⋅ X ) ( u ⋅ X ) 2 ] − E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] ∣ ≤ C F ∥ a ∥ ∞ . \left|\mathbb{E}\left[F(a\cdot X)(u\cdot X)^2\right]-\mathbb{E}\left[F(a\cdot\Gamma)(u\cdot\Gamma)^2\right]\right|\le C_F\|a\|_\infty. E [ F ( a ⋅ X ) ( u ⋅ X ) 2 ] − E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] ≤ C F ∥ a ∥ ∞ . Proof. Apply the scalar-block moment version of the Lindeberg–Stein comparison, as in Corollary E.2, to
h ( x ) : = F ( a ⋅ x ) ( u ⋅ x ) 2 . h(x):=F(a\cdot x)(u\cdot x)^2. h ( x ) := F ( a ⋅ x ) ( u ⋅ x ) 2 . For each coordinate i i i ,
∂ i 3 h ( x ) = a i 3 F ( 3 ) ( a ⋅ x ) ( u ⋅ x ) 2 + 6 a i 2 u i F ′ ′ ( a ⋅ x ) ( u ⋅ x ) + 6 a i u i 2 F ′ ( a ⋅ x ) . \partial_i^3h(x)=a_i^3F^{(3)}(a\cdot x)(u\cdot x)^2+6a_i^2u_iF''(a\cdot x)(u\cdot x)+6a_iu_i^2F'(a\cdot x). ∂ i 3 h ( x ) = a i 3 F ( 3 ) ( a ⋅ x ) ( u ⋅ x ) 2 + 6 a i 2 u i F ′′ ( a ⋅ x ) ( u ⋅ x ) + 6 a i u i 2 F ′ ( a ⋅ x ) . At every hybrid vector appearing in the Lindeberg interpolation,
E ∣ u ⋅ W ∣ 2 ≤ C ∥ u ∥ 2 2 = C , E ∣ u ⋅ W ∣ ≤ C , \mathbb{E}|u\cdot W|^2\le C\|u\|_2^2=C,\qquad\mathbb{E}|u\cdot W|\le C, E ∣ u ⋅ W ∣ 2 ≤ C ∥ u ∥ 2 2 = C , E ∣ u ⋅ W ∣ ≤ C , because the coordinates have uniformly bounded second moments. Hence
E ∣ ∂ i 3 h ( W ) ∣ ≤ C F ( ∣ a i ∣ 3 + a i 2 ∣ u i ∣ + ∣ a i ∣ u i 2 ) . \mathbb{E}|\partial_i^3h(W)|\le C_F\left(|a_i|^3+a_i^2|u_i|+|a_i|u_i^2\right). E ∣ ∂ i 3 h ( W ) ∣ ≤ C F ( ∣ a i ∣ 3 + a i 2 ∣ u i ∣ + ∣ a i ∣ u i 2 ) . Summing over i i i gives the first estimate.
For the final bound, use ∥ a ∥ 2 = ∥ u ∥ 2 = 1 \|a\|_2=\|u\|_2=1 ∥ a ∥ 2 = ∥ u ∥ 2 = 1 :
∑ i ∣ a i ∣ u i 2 ≤ ∥ a ∥ ∞ , ∑ i a i 2 ∣ u i ∣ ≤ ( ∑ i a i 4 ) 1 / 2 ∥ u ∥ 2 ≤ ∥ a ∥ ∞ , \sum_i|a_i|u_i^2\le\|a\|_\infty,\qquad\sum_i a_i^2|u_i|\le\left(\sum_i a_i^4\right)^{1/2}\|u\|_2\le\|a\|_\infty, i ∑ ∣ a i ∣ u i 2 ≤ ∥ a ∥ ∞ , i ∑ a i 2 ∣ u i ∣ ≤ ( i ∑ a i 4 ) 1/2 ∥ u ∥ 2 ≤ ∥ a ∥ ∞ , and
∑ i ∣ a i ∣ 3 ≤ ∥ a ∥ ∞ ∑ i a i 2 = ∥ a ∥ ∞ . \sum_i|a_i|^3\le\|a\|_\infty\sum_i a_i^2=\|a\|_\infty. i ∑ ∣ a i ∣ 3 ≤ ∥ a ∥ ∞ i ∑ a i 2 = ∥ a ∥ ∞ . This proves the claim. □
Lemma E.4 (Teacher-level θ \theta θ -Hessian comparison). Let
A ψ : = E [ ψ ′ ( G ) 2 ] , G ∼ N ( 0 , 1 ) . A_\psi:=\mathbb{E}[\psi'(G)^2],\qquad G\sim N(0,1). A ψ := E [ ψ ′ ( G ) 2 ] , G ∼ N ( 0 , 1 ) . Assume E [ ϕ ( G ) ] = 0 \mathbb{E}[\phi(G)]=0 E [ ϕ ( G )] = 0 , E [ ϕ ( G ) 2 ] = 1 \mathbb{E}[\phi(G)^2]=1 E [ ϕ ( G ) 2 ] = 1 , E ∣ ϕ ( G ) ∣ 8 < ∞ \mathbb{E}|\phi(G)|^8<\infty E ∣ ϕ ( G ) ∣ 8 < ∞ , and ψ ∈ C 4 ( R ) \psi\in C^4(\mathbb{R}) ψ ∈ C 4 ( R ) with bounded derivatives up to order 4. Then there exists C θ > 0 C_\theta>0 C θ > 0 , independent of d d d , such that for every u ∈ T θ ∗ S d − 1 u\in T_{\theta^*}S^{d-1} u ∈ T θ ∗ S d − 1 ,
∣ Hess S , θ L ( y ∗ ) [ u , u ] − 2 A ψ ∥ u ∥ 2 ∣ ≤ C θ ∥ θ ∗ ∥ ∞ ∥ u ∥ 2 . \left|\operatorname{Hess}_{S,\theta}L(y^*)[u,u]-2A_\psi\|u\|^2\right|\le C_\theta\|\theta^*\|_\infty\|u\|^2. Hess S , θ L ( y ∗ ) [ u , u ] − 2 A ψ ∥ u ∥ 2 ≤ C θ ∥ θ ∗ ∥ ∞ ∥ u ∥ 2 . Equivalently,
∣ Hess S , θ L ( y ∗ ) [ u , u ] − Hess S , θ L G ( y ∗ ) [ u , u ] ∣ ≤ C θ ∥ θ ∗ ∥ ∞ ∥ u ∥ 2 . \left|\operatorname{Hess}_{S,\theta}L(y^*)[u,u]-\operatorname{Hess}_{S,\theta}L_G(y^*)[u,u]\right|\le C_\theta\|\theta^*\|_\infty\|u\|^2. ∣ Hess S , θ L ( y ∗ ) [ u , u ] − Hess S , θ L G ( y ∗ ) [ u , u ] ∣ ≤ C θ ∥ θ ∗ ∥ ∞ ∥ u ∥ 2 . Proof. By quadratic scaling, it is enough to prove the result for ∥ u ∥ = 1 \|u\|=1 ∥ u ∥ = 1 . Set
a : = θ ∗ , X i : = ϕ ( G i ) , S : = a ⋅ X = ∑ i = 1 d a i X i , A u : = u ⋅ X = ∑ i = 1 d u i X i , a:=\theta^*,\qquad X_i:=\phi(G_i),\qquad S:=a\cdot X=\sum_{i=1}^{d}a_iX_i,\qquad A_u:=u\cdot X=\sum_{i=1}^{d}u_iX_i, a := θ ∗ , X i := ϕ ( G i ) , S := a ⋅ X = i = 1 ∑ d a i X i , A u := u ⋅ X = i = 1 ∑ d u i X i , where G i ∼ i i d N ( 0 , 1 ) G_i\mathrel{\overset{\mathrm{iid}}{\sim}}N(0,1) G i ∼ iid N ( 0 , 1 ) . Also define
F : = ψ ′ 2 . F:=\psi'^2. F := ψ ′2 . Since u ∈ T θ ∗ S d − 1 u\in T_{\theta^*}S^{d-1} u ∈ T θ ∗ S d − 1 , we have a ⋅ u = 0 a\cdot u=0 a ⋅ u = 0 .
At the teacher, the residual vanishes pointwise. Therefore, along a spherical geodesic in the θ \theta θ -variable with initial velocity u u u ,
Hess S , θ L ( y ∗ ) [ u , u ] = 2 E [ F ( S ) A u 2 ] . \operatorname{Hess}_{S,\theta}L(y^*)[u,u]=2\mathbb{E}\left[F(S)A_u^2\right]. Hess S , θ L ( y ∗ ) [ u , u ] = 2 E [ F ( S ) A u 2 ] .
Let Γ ∼ N ( 0 , I d ) \Gamma\sim N(0,I_d) Γ ∼ N ( 0 , I d ) . By Corollary E.3 ,
∣ E [ F ( a ⋅ X ) ( u ⋅ X ) 2 ] − E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] ∣ ≤ C F ∥ a ∥ ∞ = C F ∥ θ ∗ ∥ ∞ . \left|\mathbb{E}\left[F(a \cdot X)(u \cdot X)^2\right]-\mathbb{E}\left[F(a \cdot\Gamma)(u \cdot\Gamma)^2\right]\right| \le C_F\|a\|_\infty= C_F\|\theta^*\|_\infty. E [ F ( a ⋅ X ) ( u ⋅ X ) 2 ] − E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] ≤ C F ∥ a ∥ ∞ = C F ∥ θ ∗ ∥ ∞ .
Since a ⋅ u = 0 a \cdot u=0 a ⋅ u = 0 and ∥ a ∥ = ∥ u ∥ = 1 \|a\|=\|u\|=1 ∥ a ∥ = ∥ u ∥ = 1 , the Gaussian random variables a ⋅ Γ a \cdot\Gamma a ⋅ Γ and u ⋅ Γ u \cdot\Gamma u ⋅ Γ are independent standard normals. Hence
E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] = E [ F ( G ) ] E [ G 2 ] = A ψ . \mathbb{E}\left[F(a \cdot\Gamma)(u \cdot\Gamma)^2\right]=\mathbb{E}[F(G)]\mathbb{E}[G^2]=A_\psi. E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] = E [ F ( G )] E [ G 2 ] = A ψ .
Thus
∣ Hess S , θ L ( y ∗ ) [ u , u ] − 2 A ψ ∣ ≤ C θ ∥ θ ∗ ∥ ∞ \left|\operatorname{Hess}_{S,\theta}L(y^*)[u,u]-2A_\psi\right|\le C_\theta\|\theta^*\|_\infty ∣ Hess S , θ L ( y ∗ ) [ u , u ] − 2 A ψ ∣ ≤ C θ ∥ θ ∗ ∥ ∞
for every unit u ∈ T θ ∗ S d − 1 u\in T_{\theta^*}\mathbb{S}^{d-1} u ∈ T θ ∗ S d − 1 . Homogeneity gives the claim for arbitrary u u u .
Finally, the Gaussian teacher Hessian satisfies
Hess S , θ L G ( y ∗ ) [ u , u ] = 2 A ψ ∥ u ∥ 2 , \operatorname{Hess}_{S,\theta}L_G(y^*)[u,u]=2A_\psi\|u\|^2, Hess S , θ L G ( y ∗ ) [ u , u ] = 2 A ψ ∥ u ∥ 2 ,
because in the Gaussian model the same computation gives 2 E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] 2\mathbb{E}[F(a\cdot\Gamma)(u\cdot\Gamma)^2] 2 E [ F ( a ⋅ Γ ) ( u ⋅ Γ ) 2 ] , and a ⋅ Γ a\cdot\Gamma a ⋅ Γ is independent of u ⋅ Γ u\cdot\Gamma u ⋅ Γ . This proves the comparison with L G L_G L G .
Stein Bound for β \beta β -Hessian. To give a Stein bound for β \beta β -Hessian, we start by deriving this corollary from Theorem E.1 .
Corollary E.5 (One-dimensional linear statistic). Assume the setting of Theorem E.1. Let a ∈ R d a\in\mathbb{R}^d a ∈ R d and f ∈ C 3 ( R ) f\in C^3(\mathbb{R}) f ∈ C 3 ( R ) with ∥ f ( 3 ) ∥ ∞ < ∞ \|f^{(3)}\|_\infty<\infty ∥ f ( 3 ) ∥ ∞ < ∞ . Define h ( x ) : = f ( ⟨ a , x ⟩ ) h(x):=f(\langle a,x\rangle) h ( x ) := f (⟨ a , x ⟩) . Then
∣ E [ f ( ⟨ a , X ⟩ ) ] − E [ f ( ⟨ a , G ⟩ ) ] ∣ ≤ 5 6 ∥ f ( 3 ) ∥ ∞ ∑ i = 1 d τ i 3 ∣ a i ∣ 3 . \left|\mathbb{E}[f(\langle a,X\rangle)]-\mathbb{E}[f(\langle a,G\rangle)]\right|\le\frac{5}{6}\|f^{(3)}\|_\infty\sum_{i=1}^{d}\tau_i^3|a_i|^3. ∣ E [ f (⟨ a , X ⟩)] − E [ f (⟨ a , G ⟩)] ∣ ≤ 6 5 ∥ f ( 3 ) ∥ ∞ i = 1 ∑ d τ i 3 ∣ a i ∣ 3 . In particular, if sup i τ i ≤ τ \sup_i\tau_i\le\tau sup i τ i ≤ τ and ∥ a ∥ 2 = 1 \|a\|_2=1 ∥ a ∥ 2 = 1 , then
∣ E [ f ( ⟨ a , X ⟩ ) ] − E [ f ( ⟨ a , G ⟩ ) ] ∣ ≤ C ( τ , f ) ∥ a ∥ ∞ , (219) \left|\mathbb{E}[f(\langle a,X\rangle)]-\mathbb{E}[f(\langle a,G\rangle)]\right|\le C(\tau,f)\|a\|_\infty, \tag*{(219)} ∣ E [ f (⟨ a , X ⟩)] − E [ f (⟨ a , G ⟩)] ∣ ≤ C ( τ , f ) ∥ a ∥ ∞ , (219) since ∑ i ∣ a i ∣ 3 ≤ ∥ a ∥ ∞ ∑ i a i 2 = ∥ a ∥ ∞ \sum_i|a_i|^3\le\|a\|_\infty\sum_i a_i^2=\|a\|_\infty ∑ i ∣ a i ∣ 3 ≤ ∥ a ∥ ∞ ∑ i a i 2 = ∥ a ∥ ∞ .
Proof. Define h ( x ) : = f ( ⟨ a , x ⟩ ) h(x):=f(\langle a,x\rangle) h ( x ) := f (⟨ a , x ⟩) . By the chain rule, all partial derivatives act only through ⟨ a , x ⟩ \langle a,x\rangle ⟨ a , x ⟩ :
∂ 3 h ∂ x i 3 ( x ) = a i 3 f ′ ′ ′ ( ⟨ a , x ⟩ ) . \frac{\partial^3 h}{\partial x_i^3}(x)=a_i^3f^{\prime\prime\prime}(\langle a,x\rangle). ∂ x i 3 ∂ 3 h ( x ) = a i 3 f ′′′ (⟨ a , x ⟩) . Therefore
∥ ∂ 3 h ∂ x i 3 ∥ ∞ = ∣ a i ∣ 3 ∥ f ( 3 ) ∥ ∞ . \left\|\frac{\partial^3h}{\partial x_i^3}\right\|_\infty=|a_i|^3\|f^{(3)}\|_\infty. ∂ x i 3 ∂ 3 h ∞ = ∣ a i ∣ 3 ∥ f ( 3 ) ∥ ∞ . Applying Theorem E.1 directly gives
∣ E [ f ( ⟨ a , X ⟩ ) ] − E [ f ( ⟨ a , G ⟩ ) ] ∣ ≤ 5 6 ∑ i = 1 d τ i 3 ∥ ∂ x i 3 h ∥ ∞ = 5 6 ∥ f ( 3 ) ∥ ∞ ∑ i = 1 d τ i 3 ∣ a i ∣ 3 . \left|\mathbb{E}[f(\langle a,X\rangle)]-\mathbb{E}[f(\langle a,G\rangle)]\right|
\le\frac{5}{6}\sum_{i=1}^{d}\tau_i^3\left\|\partial_{x_i}^3h\right\|_\infty
=\frac{5}{6}\|f^{(3)}\|_\infty\sum_{i=1}^{d}\tau_i^3|a_i|^3. ∣ E [ f (⟨ a , X ⟩)] − E [ f (⟨ a , G ⟩)] ∣ ≤ 6 5 i = 1 ∑ d τ i 3 ∂ x i 3 h ∞ = 6 5 ∥ f ( 3 ) ∥ ∞ i = 1 ∑ d τ i 3 ∣ a i ∣ 3 . If sup i τ i ≤ τ \sup_i\tau_i\le\tau sup i τ i ≤ τ and ∥ a ∥ 2 = 1 \|a\|_2=1 ∥ a ∥ 2 = 1 , then
∑ i = 1 d τ i 3 ∣ a i ∣ 3 ≤ τ 3 ∑ i = 1 d ∣ a i ∣ 3 ≤ τ 3 ∥ a ∥ ∞ ∑ i = 1 d a i 2 = τ 3 ∥ a ∥ ∞ , \sum_{i=1}^{d}\tau_i^3|a_i|^3\le\tau^3\sum_{i=1}^{d}|a_i|^3\le\tau^3\|a\|_\infty\sum_{i=1}^{d}a_i^2=\tau^3\|a\|_\infty, i = 1 ∑ d τ i 3 ∣ a i ∣ 3 ≤ τ 3 i = 1 ∑ d ∣ a i ∣ 3 ≤ τ 3 ∥ a ∥ ∞ i = 1 ∑ d a i 2 = τ 3 ∥ a ∥ ∞ , giving C ( τ , f ) = 5 6 τ 3 ∥ f ( 3 ) ∥ ∞ C(\tau,f)=\frac{5}{6}\tau^3\|f^{(3)}\|_\infty C ( τ , f ) = 6 5 τ 3 ∥ f ( 3 ) ∥ ∞ , so we obtained (211).
Lemma E.6 (Teacher-level β \beta β -Hessian comparison). Let
A ψ : = E [ ψ ′ ( G ) 2 ] , B ϕ : = E [ ϕ ′ ( G ) 2 ] = κ ′ ( 1 ) , G ∼ N ( 0 , 1 ) . A_\psi:=\mathbb{E}[\psi'(G)^2],\qquad B_\phi:=\mathbb{E}[\phi'(G)^2]=\kappa'(1),\qquad G\sim N(0,1). A ψ := E [ ψ ′ ( G ) 2 ] , B ϕ := E [ ϕ ′ ( G ) 2 ] = κ ′ ( 1 ) , G ∼ N ( 0 , 1 ) . Assume (A3). Then there exists a constant C β > 0 C_\beta>0 C β > 0 , independent of d d d , such that for every v ∈ T β ∗ S d − 1 v\in T_{\beta^*}\mathbb{S}^{d-1} v ∈ T β ∗ S d − 1 ,
∣ H L , β β ( y ∗ ) [ v , v ] − 2 A ψ B ϕ ∥ v ∥ 2 ∣ ≤ C β ∥ θ ∗ ∥ ∞ ∥ v ∥ 2 . \left|H_{L,\beta\beta}(y^*)[v,v]-2A_\psi B_\phi\|v\|^2\right|\le C_\beta\|\theta^*\|_\infty\|v\|^2. H L , β β ( y ∗ ) [ v , v ] − 2 A ψ B ϕ ∥ v ∥ 2 ≤ C β ∥ θ ∗ ∥ ∞ ∥ v ∥ 2 . Equivalently,
∣ H L , β β ( y ∗ ) [ v , v ] − H L G , β β ( y ∗ ) [ v , v ] ∣ ≤ C β ∥ θ ∗ ∥ ∞ ∥ v ∥ 2 . \left|H_{L,\beta\beta}(y^*)[v,v]-H_{L_G,\beta\beta}(y^*)[v,v]\right|\le C_\beta\|\theta^*\|_\infty\|v\|^2. ∣ H L , β β ( y ∗ ) [ v , v ] − H L G , β β ( y ∗ ) [ v , v ] ∣ ≤ C β ∥ θ ∗ ∥ ∞ ∥ v ∥ 2 .
Proof. By quadratic scaling, it is enough to consider ∥ v ∥ = 1 \lVert v\rVert= 1 ∥ v ∥ = 1 . Set
a i : = θ i ∗ , X i : = ϕ ( G i ) , q i : = ϕ ′ ( G i ) 2 , S : = ∑ i = 1 d a i X i , S − i : = S − a i X i . a_i := \theta_i^*, \qquad X_i := \phi(G_i), \qquad q_i := \phi'(G_i)^2, \qquad S := \sum_{i=1}^{d} a_i X_i, \qquad S_{-i} := S - a_i X_i. a i := θ i ∗ , X i := ϕ ( G i ) , q i := ϕ ′ ( G i ) 2 , S := i = 1 ∑ d a i X i , S − i := S − a i X i .
and define
F ( s ) : = ψ ′ ( s ) 2 . F(s) := \psi'(s)^2. F ( s ) := ψ ′ ( s ) 2 .
By Lemma C.11 ,
1 2 H L , β β ( v ∗ ) [ v , v ] = ∑ i = 1 d a i 2 E [ F ( S ) q i ] . \frac{1}{2} H_{L,\beta\beta}(v^*)[v,v] = \sum_{i=1}^{d} a_i^2 \mathbb{E}[F(S)q_i]. 2 1 H L , β β ( v ∗ ) [ v , v ] = i = 1 ∑ d a i 2 E [ F ( S ) q i ] .
We first decouple the i i i -th coordinate. Since S − i S_{-i} S − i is independent of ( X i , q i ) (X_i,q_i) ( X i , q i ) ,
E [ F ( S − i ) q i ] = E [ F ( S − i ) ] E [ q i ] = B ϕ E [ F ( S − i ) ] . \mathbb{E}[F(S_{-i})q_i] = \mathbb{E}[F(S_{-i})]\mathbb{E}[q_i] = B_\phi\mathbb{E}[F(S_{-i})]. E [ F ( S − i ) q i ] = E [ F ( S − i )] E [ q i ] = B ϕ E [ F ( S − i )] .
Moreover, by the boundedness of F ′ F' F ′ and the moment assumption,
∣ E [ F ( S ) q i ] − E [ F ( S − i ) q i ] ∣ ≤ ∥ F ′ ∥ ∞ ∣ a i ∣ E [ ∣ X i ∣ q i ] ≤ C ∣ a i ∣ . \left|\mathbb{E}[F(S)q_i] - \mathbb{E}[F(S_{-i})q_i]\right| \le\lVert F'\rVert_\infty|a_i|\mathbb{E}[|X_i|q_i] \le C|a_i|. ∣ E [ F ( S ) q i ] − E [ F ( S − i ) q i ] ∣ ≤ ∥ F ′ ∥ ∞ ∣ a i ∣ E [ ∣ X i ∣ q i ] ≤ C ∣ a i ∣.
Multiplying by a i 2 a_i^2 a i 2 and summing gives
∑ i = 1 d a i 2 E [ F ( S ) q i ] = B ϕ ∑ i = 1 d a i 2 E [ F ( S − i ) ] + O ( ∑ i = 1 d ∣ a i ∣ 3 ) . \sum_{i=1}^{d} a_i^2\mathbb{E}[F(S)q_i] = B_\phi\sum_{i=1}^{d} a_i^2\mathbb{E}[F(S_{-i})] + O\left(\sum_{i=1}^{d}|a_i|^3\right). i = 1 ∑ d a i 2 E [ F ( S ) q i ] = B ϕ i = 1 ∑ d a i 2 E [ F ( S − i )] + O ( i = 1 ∑ d ∣ a i ∣ 3 ) .
Since ∑ i a i 2 = 1 \sum_i a_i^2 = 1 ∑ i a i 2 = 1 and
∣ E [ F ( S ) ] − E [ F ( S − i ) ] ∣ ≤ ∥ F ′ ∥ ∞ ∣ a i ∣ E ∣ X i ∣ , |\mathbb{E}[F(S)] - \mathbb{E}[F(S_{-i})]| \le\lVert F'\rVert_\infty|a_i|\mathbb{E}|X_i|, ∣ E [ F ( S )] − E [ F ( S − i )] ∣ ≤ ∥ F ′ ∥ ∞ ∣ a i ∣ E ∣ X i ∣ ,
we also have
∑ i = 1 d a i 2 E [ F ( S − i ) ] = E [ F ( S ) ] + O ( ∑ i = 1 d ∣ a i ∣ 3 ) . \sum_{i=1}^{d} a_i^2\mathbb{E}[F(S_{-i})] = \mathbb{E}[F(S)] + O\left(\sum_{i=1}^{d}|a_i|^3\right). i = 1 ∑ d a i 2 E [ F ( S − i )] = E [ F ( S )] + O ( i = 1 ∑ d ∣ a i ∣ 3 ) .
Therefore,
∑ i = 1 d a i 2 E [ F ( S ) q i ] = B ϕ E [ F ( S ) ] + O ( ∑ i = 1 d ∣ a i ∣ 3 ) . \sum_{i=1}^{d} a_i^2\mathbb{E}[F(S)q_i] = B_\phi\mathbb{E}[F(S)] + O\left(\sum_{i=1}^{d}|a_i|^3\right). i = 1 ∑ d a i 2 E [ F ( S ) q i ] = B ϕ E [ F ( S )] + O ( i = 1 ∑ d ∣ a i ∣ 3 ) .
Now apply the one-dimensional Stein comparison in Corollary E.5 with
X i : = ϕ ( G i ) , a i : = θ i ∗ , f : = F = ψ ′ 2 . X_i := \phi(G_i), \qquad a_i := \theta_i^*, \qquad f := F = \psi'^2. X i := ϕ ( G i ) , a i := θ i ∗ , f := F = ψ ′2 .
By the assumptions on ϕ \phi ϕ ,
E [ X i ] = 0 , E [ X i 2 ] = 1 , τ ϕ 3 : = E ∣ X i ∣ 3 < ∞ , \mathbb{E}[X_i] = 0, \qquad\mathbb{E}[X_i^2] = 1, \qquad\tau_\phi^3 := \mathbb{E}|X_i|^3 < \infty, E [ X i ] = 0 , E [ X i 2 ] = 1 , τ ϕ 3 := E ∣ X i ∣ 3 < ∞ ,
and by the regularity assumptions on ψ \psi ψ , F ∈ C 3 ( R ) F \in C^3(\mathbb{R}) F ∈ C 3 ( R ) with ∥ F ( 3 ) ∥ ∞ < ∞ \lVert F^{(3)}\rVert_\infty< \infty ∥ F ( 3 ) ∥ ∞ < ∞ . Moreover ∥ a ∥ 2 = ∥ θ ∗ ∥ 2 = 1 \lVert a\rVert_2 = \lVert\theta^*\rVert_2 = 1 ∥ a ∥ 2 = ∥ θ ∗ ∥ 2 = 1 . Let Γ = ( Γ 1 , … , Γ d ) ∼ N ( 0 , I d ) \Gamma= (\Gamma_1,\ldots,\Gamma_d) \sim N(0,I_d) Γ = ( Γ 1 , … , Γ d ) ∼ N ( 0 , I d ) . Then ⟨ a , Γ ⟩ ∼ N ( 0 , 1 ) \langle a,\Gamma\rangle\sim N(0,1) ⟨ a , Γ ⟩ ∼ N ( 0 , 1 ) . Hence Corollary E.5 gives
∣ E F ( ∑ i = 1 d a i X i ) − E F ( ⟨ a , Γ ⟩ ) ∣ ≤ 5 6 ∥ F ( 3 ) ∥ ∞ ∑ i = 1 d τ ϕ 3 ∣ a i ∣ 3 . \left|\mathbb{E}F\left(\sum_{i=1}^{d}a_iX_i\right) - \mathbb{E}F(\langle a,\Gamma\rangle)\right| \le\frac{5}{6}\lVert F^{(3)}\rVert_\infty\sum_{i=1}^{d}\tau_\phi^3|a_i|^3. E F ( i = 1 ∑ d a i X i ) − E F (⟨ a , Γ ⟩) ≤ 6 5 ∥ F ( 3 ) ∥ ∞ i = 1 ∑ d τ ϕ 3 ∣ a i ∣ 3 .
Since ⟨ a , Γ ⟩ ∼ G ∼ N ( 0 , 1 ) \langle a,\Gamma\rangle\sim G \sim N(0,1) ⟨ a , Γ ⟩ ∼ G ∼ N ( 0 , 1 ) , this becomes
∣ E [ F ( S ) ] − E [ F ( G ) ] ∣ ≤ C F ∑ i = 1 d ∣ a i ∣ 3 , |\mathbb{E}[F(S)] - \mathbb{E}[F(G)]| \le C_F\sum_{i=1}^{d}|a_i|^3, ∣ E [ F ( S )] − E [ F ( G )] ∣ ≤ C F i = 1 ∑ d ∣ a i ∣ 3 ,
where C F : = 5 6 τ ϕ 3 ∥ F ( 3 ) ∥ ∞ C_F := \frac{5}{6}\tau_\phi^3\lVert F^{(3)}\rVert_\infty C F := 6 5 τ ϕ 3 ∥ F ( 3 ) ∥ ∞ . Finally,
∑ i = 1 d ∣ a i ∣ 3 ≤ ∥ a ∥ ∞ ∑ i = 1 d a i 2 = ∥ θ ∗ ∥ ∞ . \sum_{i=1}^{d}|a_i|^3 \le\lVert a\rVert_\infty\sum_{i=1}^{d}a_i^2 = \lVert\theta^*\rVert_\infty. i = 1 ∑ d ∣ a i ∣ 3 ≤ ∥ a ∥ ∞ i = 1 ∑ d a i 2 = ∥ θ ∗ ∥ ∞ .
Therefore
∣ E [ F ( S ) ] − E [ F ( G ) ] ∣ ≤ C F ∥ θ ∗ ∥ ∞ . |\mathbb{E}[F(S)] - \mathbb{E}[F(G)]| \le C_F\lVert\theta^*\rVert_\infty. ∣ E [ F ( S )] − E [ F ( G )] ∣ ≤ C F ∥ θ ∗ ∥ ∞ .
Combining this with the preceding leave-one-out estimate yields
∑ i = 1 d a i 2 E [ F ( S ) q i ] = B ϕ E [ F ( G ) ] + O ( ∥ θ ∗ ∥ ∞ ) = B ϕ A ψ + O ( ∥ θ ∗ ∥ ∞ ) . \sum_{i=1}^{d} a_i^2 \mathbb{E}[F(S)q_i] = B_\phi\mathbb{E}[F(G)] + O(\|\theta^*\|_\infty) = B_\phi A_\psi+ O(\|\theta^*\|_\infty). i = 1 ∑ d a i 2 E [ F ( S ) q i ] = B ϕ E [ F ( G )] + O ( ∥ θ ∗ ∥ ∞ ) = B ϕ A ψ + O ( ∥ θ ∗ ∥ ∞ ) .
Multiplying by 2 ∥ v ∥ 2 2\|v\|^2 2∥ v ∥ 2 proves the true β β \beta\beta β β -block estimate.
For the Gaussian loss, since we have (61) in Lemma C.10,
L G ( θ , β ) = 2 ∑ k ≥ 1 α k 2 ( 1 − ( κ ( ρ ) χ ) k ) L_G(\theta,\beta) = 2\sum_{k\geq1}\alpha_k^2\left(1-(\kappa(\rho)\chi)^k\right) L G ( θ , β ) = 2 k ≥ 1 ∑ α k 2 ( 1 − ( κ ( ρ ) χ ) k )
, gives, at y ∗ y^* y ∗ ,
H L G , β β ( y ∗ ) [ v , v ] = 2 κ ′ ( 1 ) ( ∑ k ≥ 1 k α k 2 ) ∥ v ∥ 2 . H_{L_G,\beta\beta}(y^*)[v,v] = 2\kappa'(1)\left(\sum_{k\geq1} k\alpha_k^2\right)\|v\|^2. H L G , β β ( y ∗ ) [ v , v ] = 2 κ ′ ( 1 ) ( k ≥ 1 ∑ k α k 2 ) ∥ v ∥ 2 .
But
∑ k ≥ 1 k α k 2 = E [ ψ ′ ( G ) 2 ] = A ψ , κ ′ ( 1 ) = B ϕ . \sum_{k\geq1} k\alpha_k^2 = \mathbb{E}[\psi'(G)^2] = A_\psi,\qquad\kappa'(1)=B_\phi. k ≥ 1 ∑ k α k 2 = E [ ψ ′ ( G ) 2 ] = A ψ , κ ′ ( 1 ) = B ϕ .
Thus
H L G , β β ( y ∗ ) [ v , v ] = 2 A ψ B ϕ ∥ v ∥ 2 , H_{L_G,\beta\beta}(y^*)[v,v] = 2A_\psi B_\phi\|v\|^2, H L G , β β ( y ∗ ) [ v , v ] = 2 A ψ B ϕ ∥ v ∥ 2 ,
and the comparison follows.
Gradient Bounds
In the lemma below, we provide bounds for gradients g ( θ ) g^{(\theta)} g ( θ ) , g ( β ) g^{(\beta)} g ( β ) to help us to control the evolution of Z t Z_t Z t and η t \eta_t η t in weak recovery, and that of Q t Q_t Q t in strong recovery.
Lemma E.7 (Technical gradient bounds for blockwise spherical SGD). Assume (A3). Then there exists a constant K < ∞ K < \infty K < ∞ , depending only on L ϕ L_\phi L ϕ , L ψ L_\psi L ψ , ϕ ( 0 ) \phi(0) ϕ ( 0 ) , such that if we draw a fresh sample x t x_t x t then with
g ( β ) ( β , θ ; x t ) : = ∇ β S ℓ ( β , θ ; x t ) , g ( θ ) ( β , θ ; x t ) : = ∇ θ S ℓ ( β , θ ; x t ) , g^{(\beta)}(\beta,\theta;x_t):=\nabla_\beta^S\ell(\beta,\theta;x_t),\qquad g^{(\theta)}(\beta,\theta;x_t):=\nabla_\theta^S\ell(\beta,\theta;x_t), g ( β ) ( β , θ ; x t ) := ∇ β S ℓ ( β , θ ; x t ) , g ( θ ) ( β , θ ; x t ) := ∇ θ S ℓ ( β , θ ; x t ) , we have the moment bounds
sup β , θ E [ ∥ g ( β ) ( β , θ ; x t ) ∥ 2 ] ≤ K d , \sup_{\beta,\theta}\mathbb{E}\left[\|g^{(\beta)}(\beta,\theta;x_t)\|^2\right]\leq Kd, β , θ sup E [ ∥ g ( β ) ( β , θ ; x t ) ∥ 2 ] ≤ K d , sup β , θ E [ ∥ g ( θ ) ( β , θ ; x t ) ∥ 2 ] ≤ K d , \sup_{\beta,\theta}\mathbb{E}\left[\|g^{(\theta)}(\beta,\theta;x_t)\|^2\right]\leq Kd, β , θ sup E [ ∥ g ( θ ) ( β , θ ; x t ) ∥ 2 ] ≤ K d , sup β , θ E [ ∥ g ( β ) ( β , θ ; x t ) ∥ 4 ] ≤ K d 2 , \sup_{\beta,\theta}\mathbb{E}\left[\|g^{(\beta)}(\beta,\theta;x_t)\|^4\right]\leq Kd^2, β , θ sup E [ ∥ g ( β ) ( β , θ ; x t ) ∥ 4 ] ≤ K d 2 , sup β , θ E [ ∥ g ( θ ) ( β , θ ; x t ) ∥ 4 ] ≤ K d 2 . \sup_{\beta,\theta}\mathbb{E}\left[\|g^{(\theta)}(\beta,\theta;x_t)\|^4\right]\leq Kd^2. β , θ sup E [ ∥ g ( θ ) ( β , θ ; x t ) ∥ 4 ] ≤ K d 2 . Moreover, for the centered gradient-noise fields defined as
Δ g ( β ) ( β , θ ; x t ) : = g ( β ) ( β , θ ; x t ) − E [ g ( β ) ( β , θ ; x t ) ] , \Delta g^{(\beta)}(\beta,\theta;x_t):=g^{(\beta)}(\beta,\theta;x_t)-\mathbb{E}[g^{(\beta)}(\beta,\theta;x_t)], Δ g ( β ) ( β , θ ; x t ) := g ( β ) ( β , θ ; x t ) − E [ g ( β ) ( β , θ ; x t )] , Δ g ( θ ) ( β , θ ; x t ) : = g ( θ ) ( β , θ ; x t ) − E [ g ( θ ) ( β , θ ; x t ) ] , \Delta g^{(\theta)}(\beta,\theta;x_t):=g^{(\theta)}(\beta,\theta;x_t)-\mathbb{E}[g^{(\theta)}(\beta,\theta;x_t)], Δ g ( θ ) ( β , θ ; x t ) := g ( θ ) ( β , θ ; x t ) − E [ g ( θ ) ( β , θ ; x t )] , we have the directional variance bounds
sup β , θ sup ∥ u ∥ = 1 E [ ( Δ g ( β ) ( β , θ ; x t ) ⋅ u ) 2 ] ≤ K , \sup_{\beta,\theta}\sup_{\|u\|=1}\mathbb{E}\left[(\Delta g^{(\beta)}(\beta,\theta;x_t)\cdot u)^2\right]\leq K, β , θ sup ∥ u ∥ = 1 sup E [ ( Δ g ( β ) ( β , θ ; x t ) ⋅ u ) 2 ] ≤ K , sup β , θ sup ∥ u ∥ = 1 E [ ( Δ g ( θ ) ( β , θ ; x t ) ⋅ u ) 2 ] ≤ K . \sup_{\beta,\theta}\sup_{\|u\|=1}\mathbb{E}\left[(\Delta g^{(\theta)}(\beta,\theta;x_t)\cdot u)^2\right]\leq K. β , θ sup ∥ u ∥ = 1 sup E [ ( Δ g ( θ ) ( β , θ ; x t ) ⋅ u ) 2 ] ≤ K . Consequently, along the SGD trajectory ( β t , θ t ) (\beta_t,\theta_t) ( β t , θ t ) , since x t x_t x t is independent of F t \mathcal{F}_t F t , the same bounds hold conditionally on F t \mathcal{F}_t F t :
E [ ∥ g t ( β ) ∥ 2 ∣ F t ] ≤ K d , E [ ∥ g t ( β ) ∥ 4 ∣ F t ] ≤ K d 2 , sup ∥ u ∥ = 1 E [ ( ( Δ g t ( β ) ) ⋅ u ) 2 ∣ F t ] ≤ K , \mathbb{E}\left[\|g_t^{(\beta)}\|^2\mid\mathcal{F}_t\right]\leq Kd,\qquad
\mathbb{E}\left[\|g_t^{(\beta)}\|^4\mid\mathcal{F}_t\right]\leq Kd^2,\qquad
\sup_{\|u\|=1}\mathbb{E}\left[((\Delta g_t^{(\beta)})\cdot u)^2\mid\mathcal{F}_t\right]\leq K, E [ ∥ g t ( β ) ∥ 2 ∣ F t ] ≤ K d , E [ ∥ g t ( β ) ∥ 4 ∣ F t ] ≤ K d 2 , ∥ u ∥ = 1 sup E [ (( Δ g t ( β ) ) ⋅ u ) 2 ∣ F t ] ≤ K , and similarly for the θ \theta θ -block.
Proof. Since spherical gradient is the orthogonal projection of the Euclidean gradient onto the tangent space via ∇ S F ( x ) = ∇ F ( x ) − ( ∇ F ( x ) ⋅ x ) x \nabla^{S}F(x)=\nabla F(x)-(\nabla F(x)\cdot x)x ∇ S F ( x ) = ∇ F ( x ) − ( ∇ F ( x ) ⋅ x ) x , we have ∥ ∇ S F ( x ) ∥ ≤ ∥ ∇ F ( x ) ∥ \|\nabla^{S}F(x)\|\le\|\nabla F(x)\| ∥ ∇ S F ( x ) ∥ ≤ ∥∇ F ( x ) ∥ and ∣ u ⋅ ∇ S F ( x ) ∣ ≤ ∣ u ⋅ ∇ F ( x ) ∣ ≤ ∥ ∇ F ( x ) ∥ |u\cdot\nabla^{S}F(x)|\le|u\cdot\nabla F(x)|\le\|\nabla F(x)\| ∣ u ⋅ ∇ S F ( x ) ∣ ≤ ∣ u ⋅ ∇ F ( x ) ∣ ≤ ∥∇ F ( x ) ∥ for ∣ u ∣ = 1 |u|=1 ∣ u ∣ = 1 , it suffices to bound the Euclidean gradients.
Based on the model definition in § 0.3.1 and define the per-sample loss as
l = ( f − f ∗ ) 2 (220) l=(f-f^{*})^{2} \tag*{(220)} l = ( f − f ∗ ) 2 (220)
we have the per-sample Euclidean gradients w.r.t. each layer as:
∇ θ l = 2 ( f − f ∗ ) ⋅ ψ ′ ( s ) ⋅ z (221) \nabla_{\theta}l=2(f-f^{*})\cdot\psi'(s)\cdot z \tag*{(221)} ∇ θ l = 2 ( f − f ∗ ) ⋅ ψ ′ ( s ) ⋅ z (221)
∇ β l = 2 ( f − f ∗ ) ⋅ ψ ′ ( s ) ⋅ ∑ i = 1 d θ i ϕ i ′ ( u i ) x ( i ) . (222) \nabla_{\beta}l=2(f-f^{*})\cdot\psi'(s)\cdot\sum_{i=1}^{d}\theta_i\phi_i'(u_i)x^{(i)}. \tag*{(222)} ∇ β l = 2 ( f − f ∗ ) ⋅ ψ ′ ( s ) ⋅ i = 1 ∑ d θ i ϕ i ′ ( u i ) x ( i ) . (222)
We simplify the expression by defining
A ( x ; β , θ ) : = 2 ( f − f ∗ ) ⋅ ψ ′ ( s ) ∈ R . (223) A(x;\beta,\theta):=2(f-f^{*})\cdot\psi'(s)\in\mathbb{R}. \tag*{(223)} A ( x ; β , θ ) := 2 ( f − f ∗ ) ⋅ ψ ′ ( s ) ∈ R . (223)
θ \theta θ bounds
E [ ∥ g ( θ ) ∥ 2 ] ≤ E [ ∥ ∇ θ l ∥ 2 ] (224) \mathbb{E}[\|g^{(\theta)}\|^{2}]\le\mathbb{E}[\|\nabla_{\theta}l\|^{2}] \tag*{(224)} E [ ∥ g ( θ ) ∥ 2 ] ≤ E [ ∥ ∇ θ l ∥ 2 ] (224)
= E [ ∣ A ∣ 2 ∥ z ∥ 2 ] (225) =\mathbb{E}[|A|^{2}\|z\|^{2}] \tag*{(225)} = E [ ∣ A ∣ 2 ∥ z ∥ 2 ] (225)
≤ ( E ∣ A ∣ 4 ) 1 / 2 ⋅ ( E ∥ z ∥ 4 ) 1 / 2 by Cauchy-Schwarz (226) \le(\mathbb{E}|A|^{4})^{1/2}\cdot(\mathbb{E}\|z\|^{4})^{1/2}\text{ by Cauchy-Schwarz} \tag*{(226)} ≤ ( E ∣ A ∣ 4 ) 1/2 ⋅ ( E ∥ z ∥ 4 ) 1/2 by Cauchy-Schwarz (226)
For the first term of (218), with the bounded ψ , ψ ′ \psi,\psi' ψ , ψ ′ assumption, we have
∣ A ∣ = 2 ∣ ψ ( s ) − ψ ( s ∗ ) ∣ ∣ ψ ′ ( s ) ∣ ≤ 2 ( ∣ ψ ( s ) ∣ + ∣ ψ ( s ∗ ) ∣ ) ∣ ψ ′ ( s ) ∣ ≤ 4 C ψ C ψ ′ . (227) |A|=2|\psi(s)-\psi(s^{*})||\psi'(s)|\le2(|\psi(s)|+|\psi(s^{*})|)|\psi'(s)|\le4C_{\psi}C_{\psi'}. \tag*{(227)} ∣ A ∣ = 2∣ ψ ( s ) − ψ ( s ∗ ) ∣∣ ψ ′ ( s ) ∣ ≤ 2 ( ∣ ψ ( s ) ∣ + ∣ ψ ( s ∗ ) ∣ ) ∣ ψ ′ ( s ) ∣ ≤ 4 C ψ C ψ ′ . (227)
For the second term, given z z z has i.i.d. entries, we have
E [ ∥ z ∥ 4 ] = E [ ( ∑ i = 1 d z i 2 ) 2 ] (228) \mathbb{E}[\|z\|^{4}]=\mathbb{E}\left[\left(\sum_{i=1}^{d}z_i^{2}\right)^{2}\right] \tag*{(228)} E [ ∥ z ∥ 4 ] = E ( i = 1 ∑ d z i 2 ) 2 (228)
= E [ ( ∑ i = 1 d z i 2 ) ( ∑ i = 1 d z i 2 ) ] (229) =\mathbb{E}\left[\left(\sum_{i=1}^{d}z_i^{2}\right)\left(\sum_{i=1}^{d}z_i^{2}\right)\right] \tag*{(229)} = E [ ( i = 1 ∑ d z i 2 ) ( i = 1 ∑ d z i 2 ) ] (229)
= E [ ∑ i = 1 d z i 4 + ∑ i ≠ j z i 2 z j 2 ] (230) =\mathbb{E}\left[\sum_{i=1}^{d}z_i^{4}+\sum_{i\ne j}z_i^{2}z_j^{2}\right] \tag*{(230)} = E i = 1 ∑ d z i 4 + i = j ∑ z i 2 z j 2 (230)
= ∑ i = 1 d E [ z i 4 ] + ∑ i ≠ j E [ z i 2 ] E [ z j 2 ] (231) =\sum_{i=1}^{d}\mathbb{E}[z_i^{4}]+\sum_{i\ne j}\mathbb{E}[z_i^{2}]\mathbb{E}[z_j^{2}] \tag*{(231)} = i = 1 ∑ d E [ z i 4 ] + i = j ∑ E [ z i 2 ] E [ z j 2 ] (231)
= d μ 4 + d ( d − 1 ) μ 2 2 (232) =d\mu_4+d(d-1)\mu_2^{2} \tag*{(232)} = d μ 4 + d ( d − 1 ) μ 2 2 (232)
= Θ ( d 2 ) . (233) =\Theta(d^{2}). \tag*{(233)} = Θ ( d 2 ) . (233)
Therefore, there exists a unifying constant K K K such that
E [ ∥ g ( θ ) ∥ 2 ] ≤ [ 4 C ψ C ψ ′ ] 2 [ Θ ( d 2 ) ] 1 / 2 = K d . (234) \mathbb{E}[\|g^{(\theta)}\|^{2}]\le[4C_{\psi}C_{\psi'}]^{2}[\Theta(d^{2})]^{1/2}=Kd. \tag*{(234)} E [ ∥ g ( θ ) ∥ 2 ] ≤ [ 4 C ψ C ψ ′ ] 2 [ Θ ( d 2 ) ] 1/2 = K d . (234)
Similarly, to bound the fourth moment, using the Minkowski inequality in L 4 L^{4} L 4 and the i.i.d. of { z i } \{z_i\} { z i } , we have
∥ ∑ i = 1 d z i 2 ∥ L 4 = ( E [ ∑ i = 1 d z i 2 ] 4 ) 1 / 4 ≤ ∑ i = 1 d ( E [ z i 8 ] ) 1 / 4 = d ( E [ z i 8 ] ) 1 / 4 \left\|\sum_{i=1}^{d}z_i^{2}\right\|_{L^{4}}=(\mathbb{E}[\sum_{i=1}^{d}z_i^{2}]^{4})^{1/4}\le\sum_{i=1}^{d}(\mathbb{E}[z_i^{8}])^{1/4}=d(\mathbb{E}[z_i^{8}])^{1/4} i = 1 ∑ d z i 2 L 4 = ( E [ i = 1 ∑ d z i 2 ] 4 ) 1/4 ≤ i = 1 ∑ d ( E [ z i 8 ] ) 1/4 = d ( E [ z i 8 ] ) 1/4
Raise both sides to the 4th power, we have
E [ ∥ z ∥ 8 ] = E [ ( ∑ i = 1 d z i 2 ) 4 ] ≤ d 4 E [ z 1 8 ] = O ( d 4 ) . \mathbb{E}[\|z\|^{8}]=\mathbb{E}\left[\left(\sum_{i=1}^{d}z_i^{2}\right)^{4}\right]\le d^{4}\mathbb{E}[z_1^{8}]=O(d^{4}). E [ ∥ z ∥ 8 ] = E ( i = 1 ∑ d z i 2 ) 4 ≤ d 4 E [ z 1 8 ] = O ( d 4 ) .
Thus, with the same, finite A A A , there exists a unifying constant K K K such that
E [ ∥ g ( θ ) ∥ 4 ] ≤ E [ ∣ A ∣ 4 ∥ z ∥ 4 ] ≤ E [ ∣ A ∣ 8 ] 1 / 2 E [ ∥ z ∥ 8 ] 1 / 2 ≤ K d 2 (235) \mathbb{E}[\|g^{(\theta)}\|^{4}]\le\mathbb{E}[|A|^{4}\|z\|^{4}]\le\mathbb{E}[|A|^{8}]^{1/2}\mathbb{E}[\|z\|^{8}]^{1/2}\le Kd^{2} \tag*{(235)} E [ ∥ g ( θ ) ∥ 4 ] ≤ E [ ∣ A ∣ 4 ∥ z ∥ 4 ] ≤ E [ ∣ A ∣ 8 ] 1/2 E [ ∥ z ∥ 8 ] 1/2 ≤ K d 2 (235)
θ \theta θ -block directional variance. Fix ( β , θ ) ∈ S d − 1 × S d − 1 (\beta,\theta) \in\mathbb{S}^{d-1} \times\mathbb{S}^{d-1} ( β , θ ) ∈ S d − 1 × S d − 1 , and condition on F t \mathcal{F}_t F t , so that ( β , θ ) (\beta,\theta) ( β , θ ) are deterministic. For a fresh sample x t x_t x t , write
u i : = x t ( i ) ⋅ β , z i : = ϕ ( u i ) , z : = ( z 1 , … , z d ) , s : = z ⋅ θ . u_i := x_t^{(i)} \cdot\beta,\qquad z_i := \phi(u_i),\qquad z := (z_1,\ldots,z_d),\qquad s := z \cdot\theta. u i := x t ( i ) ⋅ β , z i := ϕ ( u i ) , z := ( z 1 , … , z d ) , s := z ⋅ θ .
Also write
A t : = 2 ( f t − f t ∗ ) ψ ′ ( s ) . A_t := 2(f_t-f_t^*)\psi'(s). A t := 2 ( f t − f t ∗ ) ψ ′ ( s ) .
By the boundedness of ψ \psi ψ and ψ ′ \psi' ψ ′ ,
∣ A t ∣ ≤ C A . |A_t| \le C_A. ∣ A t ∣ ≤ C A .
The Euclidean θ \theta θ -gradient is A t z A_tz A t z , hence the spherical θ \theta θ -gradient is
g t ( θ ) = P θ ⊥ ( A t z ) = A t P θ ⊥ z , P θ ⊥ : = I − θ θ T . g_t^{(\theta)} = P_{\theta^\perp}(A_tz) = A_tP_{\theta^\perp}z,\qquad P_{\theta^\perp} := I-\theta\theta^{\mathsf{T}}. g t ( θ ) = P θ ⊥ ( A t z ) = A t P θ ⊥ z , P θ ⊥ := I − θ θ T .
Let v ∈ R d v \in\mathbb{R}^d v ∈ R d be any unit vector. Since P θ ⊥ P_{\theta^\perp} P θ ⊥ is self-adjoint,
⟨ g t ( θ ) , v ⟩ = A t ⟨ z , P θ ⊥ v ⟩ . \langle g_t^{(\theta)},v\rangle= A_t\langle z,P_{\theta^\perp}v\rangle. ⟨ g t ( θ ) , v ⟩ = A t ⟨ z , P θ ⊥ v ⟩ .
Set
v ~ : = P θ ⊥ v . \widetilde{v} := P_{\theta^\perp}v. v := P θ ⊥ v .
Then ∥ v ~ ∥ ≤ 1 \|\widetilde{v}\| \le1 ∥ v ∥ ≤ 1 . Since E [ ϕ ( G ) ] = 0 \mathbb{E}[\phi(G)] = 0 E [ ϕ ( G )] = 0 and E [ ϕ ( G ) 2 ] < ∞ \mathbb{E}[\phi(G)^2] < \infty E [ ϕ ( G ) 2 ] < ∞ , the coordinates z i = ϕ ( u i ) z_i=\phi(u_i) z i = ϕ ( u i ) are iid centered variables with bounded second moment. Therefore
E [ ⟨ z , v ~ ⟩ 2 ∣ F t ] = E [ ϕ ( G ) 2 ] ∥ v ~ ∥ 2 ≤ C . \mathbb{E}\left[\langle z,\widetilde{v}\rangle^2 \mid\mathcal{F}_t\right] = \mathbb{E}[\phi(G)^2]\|\widetilde{v}\|^2 \le C. E [ ⟨ z , v ⟩ 2 ∣ F t ] = E [ ϕ ( G ) 2 ] ∥ v ∥ 2 ≤ C .
Using ∣ A t ∣ ≤ C A |A_t| \le C_A ∣ A t ∣ ≤ C A ,
E [ ⟨ g t ( θ ) , v ⟩ 2 ∣ F t ] ≤ C . \mathbb{E}\left[\langle g_t^{(\theta)},v\rangle^2 \mid\mathcal{F}_t\right] \le C. E [ ⟨ g t ( θ ) , v ⟩ 2 ∣ F t ] ≤ C .
Hence
E [ ( ⟨ g t ( θ ) , v ⟩ − E [ ⟨ g t ( θ ) , v ⟩ ∣ F t ] ) 2 ∣ F t ] ≤ E [ ⟨ g t ( θ ) , v ⟩ 2 ∣ F t ] ≤ C . \mathbb{E}\left[\left(\langle g_t^{(\theta)},v\rangle-\mathbb{E}[\langle g_t^{(\theta)},v\rangle\mid\mathcal{F}_t]\right)^2\mid\mathcal{F}_t\right]
\le\mathbb{E}\left[\langle g_t^{(\theta)},v\rangle^2\mid\mathcal{F}_t\right]\le C. E [ ( ⟨ g t ( θ ) , v ⟩ − E [⟨ g t ( θ ) , v ⟩ ∣ F t ] ) 2 ∣ F t ] ≤ E [ ⟨ g t ( θ ) , v ⟩ 2 ∣ F t ] ≤ C .
Equivalently,
sup ∥ v ∥ 2 = 1 E [ ( ( Δ g t ( θ ) ) ⋅ v ) 2 ∣ F t ] ≤ C . \sup_{\|v\|_2=1}\mathbb{E}\left[\left((\Delta g_t^{(\theta)})\cdot v\right)^2\mid\mathcal{F}_t\right]\le C. ∥ v ∥ 2 = 1 sup E [ ( ( Δ g t ( θ ) ) ⋅ v ) 2 ∣ F t ] ≤ C .
β \beta β bounds Recall
∇ β l = 2 ( f − f ∗ ) ψ ′ ( s ) ∑ i = 1 d θ i ϕ ′ ( u i ) x ( i ) , u i : = x ( i ) ⋅ β , s : = z ⋅ θ , z i = ϕ ( u i ) , \nabla_\beta l = 2(f-f^*)\psi'(s)\sum_{i=1}^{d}\theta_i\phi'(u_i)x^{(i)},\qquad u_i:=x^{(i)}\cdot\beta,\qquad s:=z\cdot\theta,\ z_i=\phi(u_i), ∇ β l = 2 ( f − f ∗ ) ψ ′ ( s ) i = 1 ∑ d θ i ϕ ′ ( u i ) x ( i ) , u i := x ( i ) ⋅ β , s := z ⋅ θ , z i = ϕ ( u i ) ,
and define the scalar
A ( x ; β , θ ) : = 2 ( f − f ∗ ) ψ ′ ( s ) . A(x;\beta,\theta):=2(f-f^*)\psi'(s). A ( x ; β , θ ) := 2 ( f − f ∗ ) ψ ′ ( s ) .
By boundedness of ψ , ψ ′ \psi,\psi' ψ , ψ ′ , we have the uniform bound
∣ A ∣ ≤ 2 ( ∣ ψ ( s ) ∣ + ∣ ψ ( s ∗ ) ∣ ) ∣ ψ ′ ( s ) ∣ ≤ 4 C ψ C ψ ′ = : C A . |A|\le2(|\psi(s)|+|\psi(s^*)|)|\psi'(s)|\le4C_\psi C_{\psi'}=:C_A. ∣ A ∣ ≤ 2 ( ∣ ψ ( s ) ∣ + ∣ ψ ( s ∗ ) ∣ ) ∣ ψ ′ ( s ) ∣ ≤ 4 C ψ C ψ ′ =: C A .
Let P β : = I − β β T P_\beta:=I-\beta\beta^{\mathsf{T}} P β := I − β β T be the orthogonal projector onto β ⊥ \beta^\perp β ⊥ . The β \beta β -spherical gradient is
g ( β ) ( β , θ ; x ) = ∇ β S l ( β , θ ; x ) = P β ∇ β l = A ∑ i = 1 d θ i ϕ ′ ( u i ) P β x ( i ) . g^{(\beta)}(\beta,\theta;x)=\nabla_\beta^{S}l(\beta,\theta;x)=P_\beta\nabla_\beta l
=A\sum_{i=1}^{d}\theta_i\phi'(u_i)P_\beta x^{(i)}. g ( β ) ( β , θ ; x ) = ∇ β S l ( β , θ ; x ) = P β ∇ β l = A i = 1 ∑ d θ i ϕ ′ ( u i ) P β x ( i ) .
Now exploit the Gaussian block structure: each block x ( i ) ∼ N ( 0 , I d ) x^{(i)}\sim\mathcal{N}(0,I_d) x ( i ) ∼ N ( 0 , I d ) independently. Decompose
x ( i ) = u i β + v i , u i : = x ( i ) ⋅ β ∼ N ( 0 , 1 ) , v i : = P β x ( i ) ∼ N ( 0 , P β ) , x^{(i)}=u_i\beta+v_i,\qquad u_i:=x^{(i)}\cdot\beta\sim\mathcal{N}(0,1),\qquad v_i:=P_\beta x^{(i)}\sim\mathcal{N}(0,P_\beta), x ( i ) = u i β + v i , u i := x ( i ) ⋅ β ∼ N ( 0 , 1 ) , v i := P β x ( i ) ∼ N ( 0 , P β ) ,
and for each i i i , u i u_i u i is independent of v i v_i v i (and the pairs ( u i , v i ) (u_i,v_i) ( u i , v i ) are i.i.d. over i i i ). Hence
g ( β ) = A ∑ i = 1 d θ i ϕ ′ ( u i ) v i . g^{(\beta)}=A\sum_{i=1}^{d}\theta_i\phi'(u_i)v_i. g ( β ) = A i = 1 ∑ d θ i ϕ ′ ( u i ) v i .
Conditioned on ( u 1 , … , u d ) (u_1,\ldots,u_d) ( u 1 , … , u d ) , the vector
S : = ∑ i = 1 d θ i ϕ ′ ( u i ) v i S := \sum_{i=1}^{d}\theta_i\phi'(u_i)v_i S := i = 1 ∑ d θ i ϕ ′ ( u i ) v i
is Gaussian with mean 0 0 0 and covariance
Cov ( S ∣ u ) = ( ∑ i = 1 d θ i 2 ϕ ′ ( u i ) 2 ) P β . \operatorname{Cov}(S \mid u)=\left(\sum_{i=1}^{d}\theta_i^2\phi'(u_i)^2\right)P_\beta. Cov ( S ∣ u ) = ( i = 1 ∑ d θ i 2 ϕ ′ ( u i ) 2 ) P β .
Write
σ 2 ( u ) : = ∑ i = 1 d θ i 2 ϕ ′ ( u i ) 2 . \sigma^2(u):=\sum_{i=1}^{d}\theta_i^2\phi'(u_i)^2. σ 2 ( u ) := i = 1 ∑ d θ i 2 ϕ ′ ( u i ) 2 .
Since rank ( P β ) = d − 1 \operatorname{rank}(P_\beta)=d-1 rank ( P β ) = d − 1 , standard Gaussian moment formulas give
E [ ∥ S ∥ 2 ∣ u ] = ( d − 1 ) σ 2 ( u ) , E [ ∥ S ∥ 4 ∣ u ] = ( d − 1 ) ( d + 1 ) σ 4 ( u ) . \mathbb{E}[\lVert S\rVert^2\mid u]=(d-1)\sigma^2(u),\qquad\mathbb{E}[\lVert S\rVert^4\mid u]=(d-1)(d+1)\sigma^4(u). E [∥ S ∥ 2 ∣ u ] = ( d − 1 ) σ 2 ( u ) , E [∥ S ∥ 4 ∣ u ] = ( d − 1 ) ( d + 1 ) σ 4 ( u ) .
At this point we use the regularity of ϕ \phi ϕ . For instance, if ϕ \phi ϕ is L ϕ L_\phi L ϕ -Lipschitz (as in our regularity assumptions), then ∣ ϕ ′ ∣ ≤ L ϕ |\phi'|\leq L_\phi ∣ ϕ ′ ∣ ≤ L ϕ a.e., so
σ 2 ( u ) ≤ L ϕ 2 ∑ i = 1 d θ i 2 = L ϕ 2 , σ 4 ( u ) ≤ L ϕ 4 . \sigma^2(u)\leq L_\phi^2\sum_{i=1}^{d}\theta_i^2=L_\phi^2,\qquad\sigma^4(u)\leq L_\phi^4. σ 2 ( u ) ≤ L ϕ 2 i = 1 ∑ d θ i 2 = L ϕ 2 , σ 4 ( u ) ≤ L ϕ 4 .
Therefore,
E ∥ g ( β ) ∥ 2 = E [ ∣ A ∣ 2 ∥ S ∥ 2 ] ≤ C A 2 E ∥ S ∥ 2 ≤ C A 2 ( d − 1 ) L ϕ 2 ≤ K d , \mathbb{E}\lVert g^{(\beta)}\rVert^2=\mathbb{E}[|A|^2\lVert S\rVert^2]\leq C_A^2\mathbb{E}\lVert S\rVert^2\leq C_A^2(d-1)L_\phi^2\leq Kd, E ∥ g ( β ) ∥ 2 = E [ ∣ A ∣ 2 ∥ S ∥ 2 ] ≤ C A 2 E ∥ S ∥ 2 ≤ C A 2 ( d − 1 ) L ϕ 2 ≤ K d ,
and similarly
E ∥ g ( β ) ∥ 4 = E [ ∣ A ∣ 4 ∥ S ∥ 4 ] ≤ C A 4 E ∥ S ∥ 4 ≤ C A 4 ( d − 1 ) ( d + 1 ) L ϕ 4 ≤ K d 2 . \mathbb{E}\lVert g^{(\beta)}\rVert^4=\mathbb{E}[|A|^4\lVert S\rVert^4]\leq C_A^4\mathbb{E}\lVert S\rVert^4\leq C_A^4(d-1)(d+1)L_\phi^4\leq Kd^2. E ∥ g ( β ) ∥ 4 = E [ ∣ A ∣ 4 ∥ S ∥ 4 ] ≤ C A 4 E ∥ S ∥ 4 ≤ C A 4 ( d − 1 ) ( d + 1 ) L ϕ 4 ≤ K d 2 .
Finally, we bound the directional variance of the centered noise field. Fix any unit vector q ∈ R d q\in\mathbb{R}^d q ∈ R d . Conditioned on u u u , ( q ⋅ S ) (q\cdot S) ( q ⋅ S ) is Gaussian with variance
Var ( q ⋅ S ∣ u ) = q ⊤ Cov ( S ∣ u ) q = σ 2 ( u ) q ⊤ P β q ≤ σ 2 ( u ) . \operatorname{Var}(q\cdot S\mid u)=q^\top\operatorname{Cov}(S\mid u)q=\sigma^2(u)q^\top P_\beta q\leq\sigma^2(u). Var ( q ⋅ S ∣ u ) = q ⊤ Cov ( S ∣ u ) q = σ 2 ( u ) q ⊤ P β q ≤ σ 2 ( u ) .
Thus,
E [ ( q ⋅ g ( β ) ) 2 ] = E [ ∣ A ∣ 2 ( q ⋅ S ) 2 ] ≤ C A 2 E [ σ 2 ( u ) ] ≤ C A 2 L ϕ 2 . \mathbb{E}[(q\cdot g^{(\beta)})^2]=\mathbb{E}[|A|^2(q\cdot S)^2]\leq C_A^2\mathbb{E}[\sigma^2(u)]\leq C_A^2L_\phi^2. E [( q ⋅ g ( β ) ) 2 ] = E [ ∣ A ∣ 2 ( q ⋅ S ) 2 ] ≤ C A 2 E [ σ 2 ( u )] ≤ C A 2 L ϕ 2 .
Since Var ( X ) ≤ E [ X 2 ] \operatorname{Var}(X)\leq\mathbb{E}[X^2] Var ( X ) ≤ E [ X 2 ] , we get
sup ∥ q ∥ = 1 E [ ( ( Δ g ( β ) ) ⋅ q ) 2 ] ≤ sup ∥ q ∥ = 1 E [ ( g ( β ) ⋅ q ) 2 ] ≤ C A 2 L ϕ 2 ≤ K , \sup_{\lVert q\rVert=1}\mathbb{E}[((\Delta g^{(\beta)})\cdot q)^2]\leq\sup_{\lVert q\rVert=1}\mathbb{E}[(g^{(\beta)}\cdot q)^2]\leq C_A^2L_\phi^2\leq K, ∥ q ∥ = 1 sup E [(( Δ g ( β ) ) ⋅ q ) 2 ] ≤ ∥ q ∥ = 1 sup E [( g ( β ) ⋅ q ) 2 ] ≤ C A 2 L ϕ 2 ≤ K ,
which proves the claimed directional variance bound (absorbing constants into K K K ).
Along the SGD trajectory ( β t , θ t ) (\beta_t,\theta_t) ( β t , θ t ) is F t \mathcal{F}_t F t -measurable and x t x_t x t is independent of F t \mathcal{F}_t F t , hence the above bounds apply conditionally on F t \mathcal{F}_t F t by evaluation at ( β , θ ) = ( β t , θ t ) (\beta,\theta)=(\beta_t,\theta_t) ( β , θ ) = ( β t , θ t ) .
Summation by Parts
The following estimates are deterministic and pathwise; no martingale property is used. They are stated on subintervals because the SGD proof applies them after stopping times.
Lemma E.8 (Summation by parts for monotone weights). Let ( N t ) 0 ≤ t ≤ T (N_t)_{0\leq t\leq T} ( N t ) 0 ≤ t ≤ T be a real-valued sequence and set
ε t : = N t + 1 − N t . \varepsilon_t:=N_{t+1}-N_t. ε t := N t + 1 − N t . Fix integers 0 ≤ r < t ≤ T 0\leq r<t\leq T 0 ≤ r < t ≤ T , and let ( b k ) k = r t − 1 (b_k)_{k=r}^{t-1} ( b k ) k = r t − 1 be monotone with 0 ≤ b k ≤ 1 0\leq b_k\leq1 0 ≤ b k ≤ 1 . If
sup 0 ≤ u ≤ T ∣ N u ∣ ≤ M , \sup_{0\leq u\leq T}|N_u|\leq M, 0 ≤ u ≤ T sup ∣ N u ∣ ≤ M , then
∣ ∑ k = r t − 1 b k ε k ∣ ≤ 2 M . \left|\sum_{k=r}^{t-1}b_k\varepsilon_k\right|\leq2M. k = r ∑ t − 1 b k ε k ≤ 2 M .
Proof. Abel summation gives
∑ k = r t − 1 b k ( N k + 1 − N k ) = b t − 1 N t − b r N r − ∑ k = r t − 2 N k + 1 ( b k + 1 − b k ) . \sum_{k=r}^{t-1} b_k(N_{k+1}-N_k)=b_{t-1}N_t-b_rN_r-\sum_{k=r}^{t-2}N_{k+1}(b_{k+1}-b_k). k = r ∑ t − 1 b k ( N k + 1 − N k ) = b t − 1 N t − b r N r − k = r ∑ t − 2 N k + 1 ( b k + 1 − b k ) .
Hence
∣ ∑ k = r t − 1 b k ε k ∣ ≤ M ( b t − 1 + b r + ∑ k = r t − 2 ∣ b k + 1 − b k ∣ ) . \left|\sum_{k=r}^{t-1}b_k\varepsilon_k\right|\le M\left(b_{t-1}+b_r+\sum_{k=r}^{t-2}|b_{k+1}-b_k|\right). k = r ∑ t − 1 b k ε k ≤ M ( b t − 1 + b r + k = r ∑ t − 2 ∣ b k + 1 − b k ∣ ) .
If ( b k ) (b_k) ( b k ) is increasing, then
∑ k = r t − 2 ∣ b k + 1 − b k ∣ = b t − 1 − b r , \sum_{k=r}^{t-2}|b_{k+1}-b_k|=b_{t-1}-b_r, k = r ∑ t − 2 ∣ b k + 1 − b k ∣ = b t − 1 − b r ,
so the coefficient is 2 b t − 1 ≤ 2 2b_{t-1}\le2 2 b t − 1 ≤ 2 . If ( b k ) (b_k) ( b k ) is decreasing, then
∑ k = r t − 2 ∣ b k + 1 − b k ∣ = b r − b t − 1 , \sum_{k=r}^{t-2}|b_{k+1}-b_k|=b_r-b_{t-1}, k = r ∑ t − 2 ∣ b k + 1 − b k ∣ = b r − b t − 1 ,
so the coefficient is 2 b r ≤ 2 2b_r\le2 2 b r ≤ 2 . This proves the claim.
Corollary E.9 (Summation by parts for decreasing geometric weights). Let ( N t ) 0 ≤ t ≤ T (N_t)_{0\le t\le T} ( N t ) 0 ≤ t ≤ T be a real-valued sequence and set
ε t : = N t + 1 − N t . \varepsilon_t:=N_{t+1}-N_t. ε t := N t + 1 − N t . Let γ > 1 \gamma>1 γ > 1 . If
sup 0 ≤ u ≤ T ∣ N u ∣ ≤ M , \sup_{0\le u\le T}|N_u|\le M, 0 ≤ u ≤ T sup ∣ N u ∣ ≤ M , then for every 0 ≤ r < T 0\le r<T 0 ≤ r < T and every s ∈ { 1 , … , T − r } s\in\{1,\ldots,T-r\} s ∈ { 1 , … , T − r } ,
∣ ∑ k = 0 s − 1 γ − ( k + 1 ) ε r + k ∣ ≤ 2 M . \left|\sum_{k=0}^{s-1}\gamma^{-(k+1)}\varepsilon_{r+k}\right|\le2M. k = 0 ∑ s − 1 γ − ( k + 1 ) ε r + k ≤ 2 M . In particular, if M = 2 λ M=2\lambda M = 2 λ , the bound is 4 λ 4\lambda 4 λ . If moreover r = 0 r=0 r = 0 and N 0 = 0 N_0=0 N 0 = 0 , then the bound improves to M M M .
Proof. Apply Lemma E.8 with
b r + k : = γ − ( k + 1 ) , 0 ≤ k ≤ s − 1. b_{r+k}:=\gamma^{-(k+1)},\qquad0\le k\le s-1. b r + k := γ − ( k + 1 ) , 0 ≤ k ≤ s − 1. These weights are decreasing and lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] , so the first bound follows.
If r = 0 r=0 r = 0 and N 0 = 0 N_0=0 N 0 = 0 , then the initial boundary term in Abel summation vanishes:
∑ k = 0 s − 1 γ − ( k + 1 ) ( N k + 1 − N k ) = γ − s N s − ∑ k = 0 s − 2 N k + 1 ( γ − ( k + 2 ) − γ − ( k + 1 ) ) . \sum_{k=0}^{s-1}\gamma^{-(k+1)}(N_{k+1}-N_k)=\gamma^{-s}N_s-\sum_{k=0}^{s-2}N_{k+1}\left(\gamma^{-(k+2)}-\gamma^{-(k+1)}\right). k = 0 ∑ s − 1 γ − ( k + 1 ) ( N k + 1 − N k ) = γ − s N s − k = 0 ∑ s − 2 N k + 1 ( γ − ( k + 2 ) − γ − ( k + 1 ) ) . Therefore
∣ ∑ k = 0 s − 1 γ − ( k + 1 ) ε k ∣ ≤ M ( γ − s + ∑ k = 0 s − 2 ( γ − ( k + 1 ) − γ − ( k + 2 ) ) ) = M γ − 1 ≤ M . \left|\sum_{k=0}^{s-1}\gamma^{-(k+1)}\varepsilon_k\right|\le M\left(\gamma^{-s}+\sum_{k=0}^{s-2}\left(\gamma^{-(k+1)}-\gamma^{-(k+2)}\right)\right)=M\gamma^{-1}\le M. k = 0 ∑ s − 1 γ − ( k + 1 ) ε k ≤ M ( γ − s + k = 0 ∑ s − 2 ( γ − ( k + 1 ) − γ − ( k + 2 ) ) ) = M γ − 1 ≤ M .
Corollary E.10 (Summation by parts for increasing geometric weights). Let ( N t ) 0 ≤ t ≤ T (N_t)_{0\le t\le T} ( N t ) 0 ≤ t ≤ T be a real-valued sequence and set
ε t : = N t + 1 − N t . \varepsilon_t:=N_{t+1}-N_t. ε t := N t + 1 − N t . Let α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) . If
sup 0 ≤ u ≤ T ∣ N u ∣ ≤ M , \sup_{0\le u\le T}|N_u|\le M, 0 ≤ u ≤ T sup ∣ N u ∣ ≤ M , then for every 0 ≤ r < t ≤ T 0\le r<t\le T 0 ≤ r < t ≤ T ,
∣ ∑ k = r t − 1 α t − 1 − k ε k ∣ ≤ 2 M . \left|\sum_{k=r}^{t-1}\alpha^{t-1-k}\varepsilon_k\right|\le2M. k = r ∑ t − 1 α t − 1 − k ε k ≤ 2 M . In particular, if M = λ M=\lambda M = λ , the bound is 2 λ 2\lambda 2 λ , and if M = 2 λ M=2\lambda M = 2 λ , the bound is 4 λ 4\lambda 4 λ .
Proof. Apply Lemma E.8 with
b k ≔ α t − 1 − k , r ≤ k ≤ t − 1. b_k \coloneqq\alpha^{t-1-k}, \qquad r \le k \le t-1. b k : = α t − 1 − k , r ≤ k ≤ t − 1.
Since α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) , these weights are increasing in k k k and lie in [ 0 , 1 ] [0,1] [ 0 , 1 ] . The claim follows. □ \square □
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
Answer: [Yes]
Justification: We provide proofs of the main claims and clearly distinguish between claims and open questions.
Guidelines:
The answer [N/A] means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the final conclusion section
Guidelines:
The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate “Limitations” section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes] Justification: N/A
Guidelines:
The answer [N/A] means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include an appendix with experimental details
Guidelines:
The answer [N/A] means that the paper does not include experiments.
If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general, releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: We provide the code used to run the experiments.
Guidelines:
The answer [N/A] means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://neurips.cc/public/guides/CodeSubmissionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results?
Answer: [Yes]
Justification:
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates).
If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification:
Guidelines:
The answer [N/A] means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper).
Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?
Answer: [Yes]
Justification:
Guidelines:
The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that there is no societal impact of the work performed.
If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required.
Answer: [N/A]
Justification:
Guidelines:
The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.