Introduction
In continual learning (CL), a learner sequentially learns from a stream of tasks. The key challenge is to accumulate expertise by learning new tasks without forgetting previously seen ones [12 ] (McCloskey & Cohen 1989; Lopez-Paz & Ranzato 2017; Parisi et al. 2019). In modern large models, continual adaptation is especially important, as environments evolve over time (e.g., in user-facing agents, e-commerce, and social networks), requiring models to incorporate new knowledge while retaining previously acquired capabilities, whereas retraining from scratch requires enormous computational resources.
We theoretically study continual classification in a statistical setting. Specifically, our work provides a precise high-dimensional characterization of continual logistic regression with linear predictors under Gaussian features. Such statistical settings have recently facilitated theoretical advances in continual linear regression, shedding light on factors such as task similarity, model complexity, and regularization strength (e.g., (Lin et al., 2023; Zhao et al., 2025; Karpel et al., 2026). The framework we develop allows us, for the first time, to theoretically study both L2 and other regularizers in a statistical setting for continual classification, focusing on the effects of the number of learned tasks and the amount of sparsity in the system.
Regularization methods have been a central focus of the continual learning community (e.g., (Kirkpatrick et al. 2017; see survey in Shi et al. 2025). Typically, these works incorporate an additive L2 regularizer in the objective of task t t t , i.e., minimize L t ( w ) + λ ∥ w − w t − 1 ∥ M 2 \mathcal{L}_t(\mathbf{w}) + \lambda\|\mathbf{w} - \mathbf{w}_{t-1}\|_{\mathbf{M}}^2 L t ( w ) + λ ∥ w − w t − 1 ∥ M 2 over w \mathbf{w} w , with L t \mathcal{L}_t L t being the per-task loss, w t − 1 \mathbf{w}_{t-1} w t − 1 the solution to the previous task, and M \mathbf{M} M typically being diagonal (e.g., [6 ] ) or even simply isotropic, M = I \mathbf{M} = \mathbf{I} M = I (e.g., Lubana et al., 2022).
While empirical works have focused on regularizing changes in the weights of neural networks (NNs), the theoretical literature on regularization has mostly analyzed shallow, linear models (e.g., Li et al., 2023, 2026; Cai & Diakonikolas, 2025; Levinstein et al., 2025; Karpel et al., 2026). Thus, it remains unclear whether even isotropic L2 regularization induces a similar effect or algorithmic bias in linear models as it does in NNs. For example, for a single task, minimizing L2 regularization over the weights of a two-layer ReLU network is equivalent to minimizing L1 regularization on the second layer (Neyshabur et al., 2014, 2015; Savarese et al., 2019). In the infinite-width limit, this further corresponds to sparse penalties on the predictor in function space (Savarese et al., 2019; Ongie et al., 2020). Similar effects arise in other homogeneous models, where L2 regularization on the weights of depth-h h h models typically translates into “sparser” penalties, such as the Lp quasi-norm with p = 2 / h p = 2/h p = 2/ h for linear networks [9 ] and certain deep ReLU networks (Savarese et al., 2019; Zhang & Wang, 2023).
Such explicit or implicit sparse penalties are often viewed as driving the rich feature-learning behavior of NNs, enabling them to outperform standard kernel methods (Woodworth et al., 2020; Parhi & Nowak, 2022). Even in simple linear settings, when a sparse solution exists, an L1 penalty yields sample complexity with only logarithmic dependence on the dimension d d d , compared with the linear dependence under L2 regularization (Ng, 2004). In other words, sparsity bias may be a key mechanism enabling overparameterized models (e.g., NNs) to generalize.
So far, the effect of sparsity bias in continual learning has not been explored theoretically. Intuitively, sparse solutions may interfere less across tasks and thereby reduce forgetting. As our paper shows, this intuition can be effective, but only when the sparsity-inducing penalty is designed appropriately. A natural candidate is the L1 regularizer ∥ w − w t − 1 ∥ 1 \lVert\mathbf{w} - \mathbf{w}_{t-1}\rVert_1 ∥ w − w t − 1 ∥ 1 , the direct analogue of the standard L2 penalty ∥ w − w t − 1 ∥ 2 2 \lVert\mathbf{w} - \mathbf{w}_{t-1}\rVert_2^2 ∥ w − w t − 1 ∥ 2 2 . We prove that this L1 regularizer can exploit a sparse ground truth, with excess risk decaying nearly linearly in the sparsity factor s s s . However, both of these continual regularizers are nonhomogeneous in w \mathbf{w} w , making it hard to align them with the natural sparsity bias we observe in a single task of deeper models.
We design a continual regularizer that avoids this non-homogeneity pitfall, preserving a direct connection to the literature on sparsity bias. Specifically, we introduce a homogeneous and analyzable “sign weighted L1” regularizer in predictor space, defined as
f ( w , w t − 1 ) = ∑ i = 1 d ∣ w i ∣ [ sign ( w i ) w t − 1 , i ] + + ϵ . f(\mathbf{w},\mathbf{w}_{t-1}) = \sum_{i=1}^{d} \frac{|w_i|}{[\operatorname{sign}(w_i)w_{t-1,i}]_{+}+\epsilon}. f ( w , w t − 1 ) = i = 1 ∑ d [ sign ( w i ) w t − 1 , i ] + + ϵ ∣ w i ∣ .
Intuitively, this regularizer promotes sparsity in the predictor while encouraging preservation of the coordinate-wise signs of the previous iterate (see Figure 1 ).
Figure 1. Coordinate-wise comparison of L1 and our sign-weighted L1 regularization. (a) Previously inactive coordinate (w t − 1 , i = 0 w_{t-1,i}=0 w t − 1 , i = 0 ). SW-L1 and L1 coincide, so activation is equally penalized in either direction. (b) Previously positive coordinate (w t − 1 , i > 0 w_{t-1,i}>0 w t − 1 , i > 0 ). SW-L1 penalizes same-sign updates less than sign flips. Here w t − 1 , i = ϵ = 1 w_{t-1,i}=\epsilon=1 w t − 1 , i = ϵ = 1 .
Importantly, our proposed regularizer arises naturally at the predictor level when weighted L2 regularization is applied to the parameter space of a two-layer diagonal linear network (DLN; Woodworth et al. 2020). Its benefit goes beyond this connection to the sparsity-bias literature: we show theoretically and empirically that this regularizer achieves a substantially better convergence rate and exhibits less forgetting than the direct L1 analogue.
Specifically, our contributions are:
We develop the first high-dimensional continual classification , deriving an analytic expression for the asymptotic performance of general regularized continual logistic regression.
In a 1-layer setting (where simply y ^ = w T x \hat{y} = \mathbf{w}^{\mathsf{T}}\mathbf{x} y ^ = w T x ):
We derive the optimal L2 regularization strength and excess risk in continual linear classification in high dimensions. We show that the optimal strength scales with the task horizon T T T as T / log T T/\log T T / log T and the excess risk scales as log T / T \log T/T log T / T , mirroring the linear regression case (Karpel et al., 2026).
For the direct L1 regularizer ∥ w − w t − 1 ∥ 1 \lVert\mathbf{w} - \mathbf{w}_{t-1}\rVert_1 ∥ w − w t − 1 ∥ 1 , we establish global convergence guarantees with near-linear dependence on the sparsity factor s s s , namely s log ( 1 s ) s\log\left(\frac{1}{s}\right) s log ( s 1 ) , but only polylogarithmic dependence on the task horizon T T T .
In a two-layer DLN (where y ^ = ∑ j θ ( 1 ) , j θ ( 2 ) , j x j \hat{y} = \sum_j \theta_{(1),j}\theta_{(2),j}x_j y ^ = ∑ j θ ( 1 ) , j θ ( 2 ) , j x j ):
We design a weighting scheme for L2 regularization on the weight space that induces a sign weighted L1 regularizer the predictor space .
We prove that the sign weighted regularizer attains a tight linear dependence on s s s .
We empirically validate, using linear models and neural networks, that sign weighted L1 outperforms competing regularizers in the presence of sparsity.
Setting: Regularized Continual Logistic Regression
Continual classification tasks. In our setting, the learner receives a sequence of T T T tasks { ( X t , y t ) } t = 1 T \{(X_t,y_t)\}_{t=1}^{T} {( X t , y t ) } t = 1 T , each having n n n samples. In each iteration t t t a learner is exposed to the samples from the current task ( X t , y t ) (X_t,y_t) ( X t , y t ) , with no access to data from previous tasks .
Statistical data model: Gaussian features and a noisy linear teacher. For each task t t t , datapoints are sampled i.i.d. from the Gaussian distribution, i.e. , { x t , i } i = 1 n ∼ i i d N ( 0 d , d − 1 I ) \{x_{t,i}\}_{i=1}^{n} \mathrel{\overset{\mathrm{iid}}{\sim}} \mathcal{N}(0_d,d^{-1}I) { x t , i } i = 1 n ∼ iid N ( 0 d , d − 1 I ) independently of all other tasks. Labels are drawn from a logistic model using a teacher vector w ⋆ ∈ R d \mathbf{w}_{\star}\in\mathbb{R}^{d} w ⋆ ∈ R d whose entries are sampled independently from a distribution Π \Pi Π , and a label-flip noise level of ρ ∈ [ 0 , 1 2 ) \rho\in[0,\frac{1}{2}) ρ ∈ [ 0 , 2 1 ) . That is, for any task t = 1 , … , T t=1,\ldots,T t = 1 , … , T and sample i = 1 , … , n i=1,\ldots,n i = 1 , … , n , the label is drawn as
P ( y t , i = 1 ∣ x t , i ) = 1 − P ( y t , i = 0 ∣ x t , i ) = ρ + ( 1 − 2 ρ ) e x t , i T w ⋆ 1 + e x t , i T w ⋆ . (1) \mathbb{P}(y_{t,i}=1\mid x_{t,i})=1-\mathbb{P}(y_{t,i}=0\mid x_{t,i})
=\rho+(1-2\rho)\frac{e^{x_{t,i}^{\mathsf{T}}\mathbf{w}_{\star}}}{1+e^{x_{t,i}^{\mathsf{T}}\mathbf{w}_{\star}}}. \tag*{(1)} P ( y t , i = 1 ∣ x t , i ) = 1 − P ( y t , i = 0 ∣ x t , i ) = ρ + ( 1 − 2 ρ ) 1 + e x t , i T w ⋆ e x t , i T w ⋆ . (1)
This setting has been extensively studied in the deep learning theory literature [17 ] (Salehi et al. 2019; Thrampoulidis et al. 2020).
Studied algorithm: Regularized continual logistic regression. To learn sequentially without forgetting, we follow a large body of continual learning works that penalize deviations from the previous predictor (Kirkpatrick et al., 2017; Schwarz et al., 2018; Zhao et al., 2024) using weighted L2 regularization. Extending the theoretical analysis of continual learning with weighted L2 regularization on linear predictors (i.e. , depth-one neural networks) to deep neural networks is nontrivial. We therefore consider a depth-h h h diagonal linear network (DLN), which is a popular [2 ] (Woodworth et al. 2020; Nacson et al. 2022) minimal model for the effect of depth, of the form
w ( θ ) = θ ( 1 ) ⊙ ⋯ ⊙ θ ( h ) , \mathbf{w}(\boldsymbol{\theta})=\boldsymbol{\theta}_{(1)}\odot\cdots\odot\boldsymbol{\theta}_{(h)}, w ( θ ) = θ ( 1 ) ⊙ ⋯ ⊙ θ ( h ) ,
where θ = [ θ ( 1 ) , … , θ ( h ) ] ∈ R d × h \boldsymbol{\theta}=[\boldsymbol{\theta}_{(1)},\ldots,\boldsymbol{\theta}_{(h)}]\in\mathbb{R}^{d\times h} θ = [ θ ( 1 ) , … , θ ( h ) ] ∈ R d × h are the parameters of all the layers, ⊙ \odot ⊙ denotes component-wise product, and for an input x \mathbf{x} x the predictor is y ^ = w T ( θ ) x \hat{y}=\mathbf{w}^{\mathsf{T}}(\boldsymbol{\theta})\mathbf{x} y ^ = w T ( θ ) x . In this paper, we focus on two cases: h = 1 h=1 h = 1 and h = 2 h=2 h = 2 .
Specifically, the objective function incorporates an additive regularization term f ( θ , θ t − 1 ) f(\boldsymbol{\theta},\boldsymbol{\theta}_{t-1}) f ( θ , θ t − 1 ) to discourage deviations from the previously learned predictor.
Table 1.
We consider regularizers that are convex in their first argument and decompose additively across coordinates. That is, there exists f ~ : R × R → R ≥ 0 \tilde{f}:\mathbb{R}\times\mathbb{R}\to\mathbb{R}_{\geq0} f ~ : R × R → R ≥ 0 such that,
f ( θ , ν ) = ∑ j = 1 d f ~ ( θ j , ν j ) for all θ , ν ∈ R d . f(\boldsymbol{\theta},\boldsymbol{\nu})=\sum_{j=1}^{d}\tilde{f}(\theta_j,\nu_j)\quad\text{for all }\boldsymbol{\theta},\boldsymbol{\nu}\in\mathbb{R}^{d}. f ( θ , ν ) = j = 1 ∑ d f ~ ( θ j , ν j ) for all θ , ν ∈ R d .
Notation. Throughout the paper, we write → d \xrightarrow{d} d for convergence in distribution, and define the softplus function ϕ ( z ) = log ( 1 + e z ) \phi(z)=\log(1+e^z) ϕ ( z ) = log ( 1 + e z ) . We write a n = Θ x ( b n ) a_n=\Theta_x(b_n) a n = Θ x ( b n ) if there exist constants 0 < c x < C x < ∞ 0<c_x<C_x<\infty 0 < c x < C x < ∞ , depending only on x x x , such that c x b n ≤ a n ≤ C x b n c_xb_n\leq a_n\leq C_xb_n c x b n ≤ a n ≤ C x b n for all sufficiently large n n n . Similarly, a n = O x ( b n ) a_n=O_x(b_n) a n = O x ( b n ) and a n = Ω x ( b n ) a_n=\Omega_x(b_n) a n = Ω x ( b n ) denote the corresponding one-sided bounds. A function f : R d → R f:\mathbb{R}^{d}\to\mathbb{R} f : R d → R is called α \alpha α homogeneous if f ( α w ) = α ξ f ( w ) f(\alpha\mathbf{w})=\alpha^\xi f(\mathbf{w}) f ( α w ) = α ξ f ( w ) for every α ≥ 0 \alpha\geq0 α ≥ 0 , and w ∈ R d \mathbf{w}\in\mathbb{R}^{d} w ∈ R d . Its proximal operator is defined by
Prox λ f ( y ) = arg min x ∈ R d 1 2 λ ∥ y − x ∥ 2 2 + f ( x ) . \operatorname{Prox}_{\lambda f}(\mathbf{y})=\underset{\mathbf{x}\in\mathbb{R}^{d}}{\arg\min}\ \frac{1}{2\lambda}\|\mathbf{y}-\mathbf{x}\|_2^2+f(\mathbf{x}). Prox λ f ( y ) = x ∈ R d arg min 2 λ 1 ∥ y − x ∥ 2 2 + f ( x ) .
Loss metric. Following earlier theoretical work (Guo et al., 2017; Ovadia et al., 2019), we analyze the negative log-likelihood on an independent test sample, i.e. ,
L ( w ) ≜ E x ∼ N ( 0 d , d − 1 I ) [ log ( 1 + e x T w ) − π ρ ( x T w ⋆ ) x T w ] . \mathcal{L}(\mathbf{w})\triangleq\mathbb{E}_{\mathbf{x}\sim\mathcal{N}(0_d,d^{-1}I)}\left[\log(1+e^{\mathbf{x}^{\mathsf{T}}\mathbf{w}})-\pi_{\rho}(\mathbf{x}^{\mathsf{T}}\mathbf{w}_{\star})\mathbf{x}^{\mathsf{T}}\mathbf{w}\right]. L ( w ) ≜ E x ∼ N ( 0 d , d − 1 I ) [ log ( 1 + e x T w ) − π ρ ( x T w ⋆ ) x T w ] .
where π ρ ( u ) = ρ + ( 1 − 2 ρ ) e u 1 + e u \pi_{\rho}(u)=\rho+(1-2\rho)\frac{e^u}{1+e^u} π ρ ( u ) = ρ + ( 1 − 2 ρ ) 1 + e u e u is the link function (1) .
The nature of our analysis. In this paper, we study regularized continual learning in the high-dimensional regime, where n , d → ∞ n,d\to\infty n , d → ∞ with n d → δ > 0 \frac{n}{d}\to\delta>0 d n → δ > 0 . Our main results characterize the performance of the predictors produced by Algorithm 1 under different choices of regularization, as a function of the task horizon T T T , noise level ρ \rho ρ , aspect ratio δ \delta δ , and signal strength ν ≜ ∥ w ⋆ ∥ d \nu\triangleq\frac{\|\mathbf{w}_{\star}\|}{\sqrt{d}} ν ≜ d ∥ w ⋆ ∥ .
For ease of presentation, throughout the paper we state simplified versions of our lemmas and theorems that capture the essence of the results, deferring the precise statements to the appendices.
Single Layer Analysis
In this section we have a single layer (h = 1 h = 1 h = 1 ), i.e., a simple linear model where w ( θ ) = θ \mathbf{w}(\boldsymbol{\theta}) = \boldsymbol{\theta} w ( θ ) = θ . Thus, we simply denote w = w ( θ ) \mathbf{w} = \mathbf{w}(\boldsymbol{\theta}) w = w ( θ ) for brevity.
High-Dimensional Characterization
In this section, we characterize the iterates of Algorithm 1 asymptotically as n , d → ∞ n, d \to\infty n , d → ∞ with n / d → δ n/d \to\delta n / d → δ . Our result shows that each coordinate of the learned predictor behaves as a scalar proximal estimator applied to an effective signal corrupted by Gaussian noise. This general expression provides a foundation for studying specific regularization functions.
For each task t t t , let η t \eta_t η t , a t a_t a t , and b t b_t b t denote the effective regularization, signal, and noise parameters, respectively, given by the state evolution (Eq. (9) ) in App. C.1.
Simplified Theorem 1 (Asymptotic Performance). Under the statistical model (§ 2 ), suppose that the state-evolution system (9) in App. C.1 admits a unique solution ( η t , a t , b t ) (\eta_t, a_t, b_t) ( η t , a t , b t ) at each task t ≤ T t \le T t ≤ T . Then, in the proportional high-dimensional limit n , d → ∞ n, d \to\infty n , d → ∞ , after sequentially learning T T T tasks using Algorithm 1, the entries of w T \mathbf{w}_T w T converge as follows,
w T , j → d Prox η T f ~ ( ⋅ , F T − 1 ) ( a T w ⋆ + b T Z ) , w_{T,j} \xrightarrow{d} \operatorname{Prox}_{\eta_T \widetilde{f}(\cdot,F_{T-1})}(a_T w^\star+ b_T Z), w T , j d Prox η T f ( ⋅ , F T − 1 ) ( a T w ⋆ + b T Z ) , where w ⋆ ∼ Π w^\star\sim\Pi w ⋆ ∼ Π , F T − 1 F_{T-1} F T − 1 are independent of Z ∼ N ( 0 , 1 ) Z \sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) , F T − 1 F_{T-1} F T − 1 is history statistic (that may depend on previous tasks), and F T − 1 F_{T-1} F T − 1 is the limiting law of a uniformly sampled coordinate from F T − 1 F_{T-1} F T − 1 .
The precise version of Theorem 1 in App. C.1. Throughout the remainder of the paper, unless stated otherwise, we assume that the system in Eq. (9) admits a unique saddle point solution for every t ≤ T t \le T t ≤ T .
Next, we use our derived characterization to analyze common regularizers and their pitfalls, and to design a new regularizer that outperforms them under sparsity.
Isotropic L2 Regularization:Optimal Regularization Scaling Law
Following recent theoretical work (e.g., (Levinstein et al., 2025; Karpel et al., 2026)), we consider the L2 (isotropic) regularization, where
f ( w , w t − 1 ) = ∥ w − w t − 1 ∥ 2 2 = ∑ j = 1 d ( w j − w t − 1 , j ) 2 . (2) f(\mathbf{w},\mathbf{w}_{t-1}) = \lVert\mathbf{w} - \mathbf{w}_{t-1} \rVert_2^2 = \sum_{j=1}^{d}(w_j-w_{t-1,j})^2. \tag*{(2)} f ( w , w t − 1 ) = ∥ w − w t − 1 ∥ 2 2 = j = 1 ∑ d ( w j − w t − 1 , j ) 2 . (2)
In practice, Hsu et al. (2018); Lubana et al. (2022) find that this simple regularizer often performs similarly to more sophisticated L2 regularizers that, e.g., use Fisher information from previous tasks to weight deviations across coordinates ([1 , 6 ] (Kirkpatrick et al. 2017; Zenke et al. 2017)).
First, we derive an explicit asymptotic characterization of the optimal regularization strength, clarifying how it should scale with the number of learned tasks.
Simplified Theorem 2 (L2 regularization decays near-linearly in the task horizon T T T ). In the proportional high-dimensional limit n , d → ∞ n,d \to\infty n , d → ∞ , let λ ⋆ \lambda^\star λ ⋆ denote the fixed regularization strength that minimizes the generalization loss after T T T tasks. Then, there exist constants C = C ( ν , ρ , δ ) > 0 C = C(\nu,\rho,\delta) > 0 C = C ( ν , ρ , δ ) > 0 and κ = κ ( ν , ρ ) > 0 \kappa= \kappa(\nu,\rho) > 0 κ = κ ( ν , ρ ) > 0 such that, for all ϵ > 0 \epsilon> 0 ϵ > 0 and T > T 0 = O ( 1 / ϵ 2 ) T > T_0 = O(1/\epsilon^2) T > T 0 = O ( 1/ ϵ 2 ) , the optimal regularization strength is bounded as,
( 1 − ϵ ) κ T log ( C κ T ) < λ ⋆ < ( 1 + ϵ ) κ T log ( C κ T ) . (1-\epsilon)\frac{\kappa T}{\log(C\kappa T)} < \lambda^\star< (1+\epsilon)\frac{\kappa T}{\log(C\kappa T)}. ( 1 − ϵ ) log ( C κ T ) κ T < λ ⋆ < ( 1 + ϵ ) log ( C κ T ) κ T . Moreover, under this strength scaling the excess risk decreases in T T T as,
L ( w T ) − inf w L ( w ) = O ( log T T ) . \mathcal{L}(\mathbf{w}_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w}) = O\left(\frac{\log T}{T}\right). L ( w T ) − w inf L ( w ) = O ( T log T ) . In App. D.3, we prove Theorem 2 and give explicit expressions for κ \kappa κ , C C C , and the excess risk, making their dependence on ν , δ , ρ \nu,\delta,\rho ν , δ , ρ explicit.
Empirical validation on synthetic data. Figure 2 validates the predictions of Theorems 1 and 2 using finite-dimensional data. We set dimension d = 50 d = 50 d = 50 and noise level ρ = 0.1 \rho= 0.1 ρ = 0.1 , draw the teacher as w ⋆ ∼ N ( 1 d , I ) \mathbf{w}^{\star} \sim\mathcal{N}(\mathbf{1}_d, \mathbf{I}) w ⋆ ∼ N ( 1 d , I ) , and sample data points as { x t , i } i = 1 n ∼ i i d N ( 0 , d − 1 I ) \{\mathbf{x}_{t,i}\}_{i=1}^{n} \stackrel{\mathrm{iid}}{\sim} \mathcal{N}(0,d^{-1}\mathbf{I}) { x t , i } i = 1 n ∼ iid N ( 0 , d − 1 I ) .
(a) Optimal regularization strength grows as predicted by the full Theorem 2 (App. D.3). Theoretical λ ⋆ \lambda^\star λ ⋆ is computed from Theorem 2. Empirical λ ^ ⋆ \hat{\lambda}^\star λ ^ ⋆ is obtained by an exhaustive search (choosing δ ≜ n d = 2 \delta\triangleq\frac{n}{d}=2 δ ≜ d n = 2 ).
(b) L2-regularized continual logistic regression over 10 tasks. Dashed lines delineate the theoretical risk from the full Theorem 1 (App. C.1). Circles show empirical results under different aspect ratios δ ≜ n d \delta\triangleq\frac{n}{d} δ ≜ d n .
Figure 2. Qualitative validation of Theorems 1 and 2. Empirical values are averaged over 100 runs.
Near-optimal convergence under L2. Interestingly, L2 is already near-optimal: under the optimal scaling of λ \lambda λ given in Theorem 2, the generalization error decays as log ( T ) / T \log(T)/T log ( T ) / T . This is within a logarithmic factor of the lower bound Ω ν , δ , ρ ( 1 / T ) 1 \Omega_{\nu,\delta,\rho}(1/T)^1 Ω ν , δ , ρ ( 1/ T ) 1 applicable to any learning algorithm as we establish in App. § 12 .8.
Is L2 always the optimal choice? Across multiple tasks, L2 regularization does not take advantage of sparsity because it penalizes changes from the previous solution quadratically. It therefore spreads each update across many coordinates rather than exploit a sparse underlying predictor. In App. B, we empirically validate with synthetic data that across various sparsity levels the excess risk attained by regularizing with L2 remains approximately fixed (see Figure 4 ).
Figure 4. Temporal L2 does not exploit sparsity of the teacher. Excess risk L ( w T ) − inf w L ( w ) \mathcal{L}(\mathbf{w}_T) - \inf_{\mathbf{w}} \mathcal{L}(\mathbf{w}) L ( w T ) − inf w L ( w ) after T = 8 T = 8 T = 8 sequential tasks. The dimension d = 256 d = 256 d = 256 , the per-task sample size n = 128 n = 128 n = 128 (hence δ = n / d = 0.5 \delta= n/d = 0.5 δ = n / d = 0.5 ), the horizon T T T , the noise level ρ = 0 \rho= 0 ρ = 0 and the signal strength ν = ∥ w ⋆ ∥ / d = 0.25 \nu= \lVert\mathbf{w}_{\star} \rVert/\sqrt{d} = 0.25 ν = ∥ w ⋆ ∥ / d = 0.25 are all held fixed. Error bars are one standard error over 3 seeds. The dashed line is the least-squares fit on the log-log scale, with slope +0.005 (R 2 = 0.11 R^2 = 0.11 R 2 = 0.11 ).
L1 Regularization Can Exploit Sparsity
This lack of sparsity adaptation motivates us to study sparsity-inducing regularizers that can better exploit a sparse teacher and thereby improve generalization.
In single-task learning, L1 regularization is one of the most common approaches to recover sparse solutions (Vaddireddy et al., 2020; Islam et al., 2024). Such regularization has played a significant rule in compressed sensing (Yin et al., 2008), MRI (Lustig et al., 2007), medical image reconstruction (Zhu et al., 2013), and biological image deconvolution (Ikoma et al., 2018).
A natural, yet underexplored, continual-learning analogue is to set, at iteration t t t ,
f ( w , w t − 1 ) = ∥ w − w t − 1 ∥ 1 = ∑ j = 1 d ∣ w j − w t − 1 , j ∣ . (3) f(\mathbf{w},\mathbf{w}_{t-1}) = \lVert\mathbf{w} - \mathbf{w}_{t-1}\rVert_1 = \sum_{j=1}^{d} \lvert w_j - w_{t-1,j}\rvert. \tag*{(3)} f ( w , w t − 1 ) = ∥ w − w t − 1 ∥ 1 = j = 1 ∑ d ∣ w j − w t − 1 , j ∣ . (3)
To examine the potential sparsity advantage of this regularizer, we specialize the teacher distribution to a standard teacher prior. This lets us to explicitly track how the excess risk depends on the sparsity level.
Sparsity modeling. Following earlier works (Sun et al., 2022; Jantre et al., 2024; Kumar et al., 2025), we consider a sparse teacher prior. Specifically, we assume the entries of w ⋆ \mathbf{w}_{\star} w ⋆ are drawn i.i.d. from a distribution,
Π ( w ) = ( 1 − s ) δ 0 ( w ) + s N ( w ∣ 0 , ν 2 s ) , (4) \Pi(w) = (1-s)\delta_0(w) + s\mathcal{N}\left(w \mid0,\frac{\nu^2}{s}\right), \tag*{(4)} Π ( w ) = ( 1 − s ) δ 0 ( w ) + s N ( w ∣ 0 , s ν 2 ) , (4)
where s ∈ ( 0 , 1 ) s \in(0,1) s ∈ ( 0 , 1 ) is the sparsity factor, N ( w ∣ 0 , σ 2 ) \mathcal{N}(w \mid0,\sigma^2) N ( w ∣ 0 , σ 2 ) is the density of the Gaussian distribution with zero mean and standard deviation σ \sigma σ , and δ 0 ( w ) \delta_0(w) δ 0 ( w ) represents the Dirac delta at zero. Namely, each entry of w ⋆ \mathbf{w}_{\star} w ⋆ equals 0 0 0 with probability 1 − s 1-s 1 − s , while with probability s s s it is nonzero and follows a Gaussian distribution.
Finite-horizon sparsity gains under L1. We first consider a fixed, finite task horizon and show that L1 regularization exploits the sparsity of the solution. In this regime, the excess risk is governed by the number of active coordinates and vanishes at a rate of s log ( 1 s ) s\log\left(\frac{1}{s}\right) s log ( s 1 ) as s → 0 s \to0 s → 0 . Thus, even after finitely many tasks, L1 regularization can achieve small error by exploiting the underlying sparse structure.
Simplified Lemma 3 (L1 regularization achieves near-linear decay in the sparsity factor s s s ). For every fixed task horizon T ≥ 1 T \ge1 T ≥ 1 and every s ≤ s 0 = Θ ν , δ ( 1 ) s \le s_0 = \Theta_{\nu,\delta}(1) s ≤ s 0 = Θ ν , δ ( 1 ) , the excess risk satisfies, for all λ > 0 , \lambda> 0, λ > 0 ,
L ( w T ) − inf w L ( w ) = Ω ν , δ , ρ ( s log ( 1 s ) ) . \mathcal{L}(\mathbf{w}_T) - \inf_{\mathbf{w}} \mathcal{L}(\mathbf{w}) = \Omega_{\nu,\delta,\rho}\left(s\log\left(\frac{1}{s}\right)\right). L ( w T ) − w inf L ( w ) = Ω ν , δ , ρ ( s log ( s 1 ) ) . Moreover, there exists λ = Θ δ ( log ( 1 / s ) + log T ) \lambda= \Theta_{\delta}\left(\sqrt{\log(1/s)+\log T}\right) λ = Θ δ ( log ( 1/ s ) + log T ) such that
L ( w T ) − inf w L ( w ) = O ν , δ , ρ ( s log ( 1 s ) ) . \mathcal{L}(\mathbf{w}_T) - \inf_{\mathbf{w}} \mathcal{L}(\mathbf{w}) = \mathcal{O}_{\nu,\delta,\rho}\left(s\log\left(\frac{1}{s}\right)\right). L ( w T ) − w inf L ( w ) = O ν , δ , ρ ( s log ( s 1 ) ) .
Limited long-horizon gains under L1. Lemma 3 clarifies that the excess risk vanishes as the teacher becomes sparse. However, this guarantee alone does not characterize the long-term behavior. Next, we show the (slow) convergence rate of the L1 regularizer, being almost logarithmic as the task horizon increases.
Simplified Lemma 4 (L1 regularization decays only poly-logarithmically in the task horizon T T T ). For any fixed s ≤ s 0 = Θ ν , δ ( 1 ) s \le s_0 = \Theta_{\nu,\delta}(1) s ≤ s 0 = Θ ν , δ ( 1 ) and every λ > 0 , \lambda> 0, λ > 0 ,
L ( w T ) − inf w L ( w ) = Ω ν , δ , ρ ( 1 log 2 T ) . \mathcal{L}(\mathbf{w}_T) - \inf_{\mathbf{w}} \mathcal{L}(\mathbf{w}) = \Omega_{\nu,\delta,\rho}\left(\frac{1}{\log^2 T}\right). L ( w T ) − w inf L ( w ) = Ω ν , δ , ρ ( log 2 T 1 ) . Moreover, there exists λ = Θ ν , δ ( log T ) \lambda= \Theta_{\nu,\delta}(\sqrt{\log T}) λ = Θ ν , δ ( log T ) such that
L ( w T ) − inf w L ( w ) = O ν , δ ( 1 log T ) . \mathcal{L}(\mathbf{w}_T) - \inf_{\mathbf{w}} \mathcal{L}(\mathbf{w}) = \mathcal{O}_{\nu,\delta}\left(\frac{1}{\log T}\right). L ( w T ) − w inf L ( w ) = O ν , δ ( log T 1 ) .
The upper bounds in Lemmas 3 and 4 are proved in Appendices F.3 and F.4, respectively. The corresponding lower bound is proved in Appendix F.5. These appendices provide explicit choices of λ \lambda λ and retain the full dependence of the bounds on ν , δ \nu,\delta ν , δ . In particular, we prove slightly stronger results, allowing the task horizon T T T to grow while the sparsity factor decreases at the same time, unlike the results presented here, which assume that one of these parameters is fixed.
Two-Layer Analysis
In this section we use two diagonal layers h = 2 h=2 h = 2 , inducing a linear model parameterized as
y ^ = w T ( θ ) x = ( θ ( 1 ) ⊙ θ ( 2 ) ) T x . \hat{y}=\mathbf{w}^{\mathsf{T}}(\theta) \mathbf{x}=(\theta_{(1)}\odot\theta_{(2)})^{\mathsf{T}}\mathbf{x}. y ^ = w T ( θ ) x = ( θ ( 1 ) ⊙ θ ( 2 ) ) T x .
From Parameter-Space L2 Regularization to Predictor-Space Sparsity
We aim to analyze the behavior of this model with L2 regularization on the parameters θ \theta θ . To do so, we will first show that the dynamics of Algorithm 1 can be reduced to the dynamics of the same algorithm in a single layer, but with a different penalty f f f .
Our main tool for this would be the following proposition (which is a slight generalization of the result in Dai et al. [9 ] , proved in App. F.7):
Proposition 5 (Weighted L2 equivalence for DLN). Denote by w ( θ ) \mathbf{w}(\theta) w ( θ ) the linear predictor induced by a depth-h h h DLN parameterized by θ \theta θ . Then, for any predictor w ∈ R d \mathbf{w}\in\mathbb{R}^{d} w ∈ R d and weighting function c ( ⋅ , ⋅ ) > 0 c(\cdot,\cdot)>0 c ( ⋅ , ⋅ ) > 0 ,
min θ ∈ R d × h : w ( θ ) = w 1 h ∑ j = 1 d c ( w j , ⋅ ) ∑ ℓ = 1 h ∣ θ ( ℓ ) , j ∣ 2 = ∑ j = 1 d c ( w j , ⋅ ) ∣ w j ∣ 2 / h . \min_{\theta\in\mathbb{R}^{d\times h}:\mathbf{w}(\theta)=\mathbf{w}}\frac{1}{h}\sum_{j=1}^{d}c(w_j,\cdot)\sum_{\ell=1}^{h}\left|\theta_{(\ell),j}\right|^2=\sum_{j=1}^{d}c(w_j,\cdot)|w_j|^{2/h}. θ ∈ R d × h : w ( θ ) = w min h 1 j = 1 ∑ d c ( w j , ⋅ ) ℓ = 1 ∑ h θ ( ℓ ) , j 2 = j = 1 ∑ d c ( w j , ⋅ ) ∣ w j ∣ 2/ h .
Example 6 (Sparsity bias in L2-regularized DLNs). In a single-task setup, learning a two-layer DLN under a standard L2 penalty on the weights is equivalent to an L1 penalty on the predictor. That is, under common loss functions L ( w ) \mathcal{L}(\mathbf{w}) L ( w ) (e.g., the logistic loss), the following equivalence hold:
min θ ∈ R d × h L ( w ( θ ) ) + λ 2 d ∥ θ ∥ F 2 = min w ∈ R d L ( w ) + λ d ∥ w ∥ 1 . \min_{\theta\in\mathbb{R}^{d\times h}}\mathcal{L}(\mathbf{w}(\theta))+\frac{\lambda}{2d}\|\theta\|_{F}^{2}=\min_{\mathbf{w}\in\mathbb{R}^{d}}\mathcal{L}(\mathbf{w})+\frac{\lambda}{d}\|\mathbf{w}\|_{1}. θ ∈ R d × h min L ( w ( θ )) + 2 d λ ∥ θ ∥ F 2 = w ∈ R d min L ( w ) + d λ ∥ w ∥ 1 .
Such a sparsity bias can substantially improve generalization when the ground-truth predictor is sparse. For example, L1-regularized learning requires sample complexity only logarithmic in the dimension d d d , whereas L2 regularization generally retains linear dependence on d d d (Ng, 2004). Related sparsity-promoting biases have also been associated (Woodworth et al., 2020) with the “rich” (a.k.a. active) feature-learning regime of NNs, which can generalize beyond standard kernel methods ([9 ] (Parhi & Nowak 2022)). Sparsity bias may therefore be an important mechanism through which overparameterized models generalize well.
Can regularized CL also induce a sparsity bias? The analytical mechanism above does not extend to CL by simply centering the regularizer at previous-task parameters. Specifically, the proof of Proposition 5 relies on the homogeneity of the weighted L2 regularizer while ∥ w − w t − 1 ∥ 2 2 \|\mathbf{w}-\mathbf{w}_{t-1}\|_{2}^{2} ∥ w − w t − 1 ∥ 2 2 (Eq. (2) ) is nonhomogeneous in w \mathbf{w} w . Next, we overcome this by shifting the preservation of past expertise to a weighting function c ( ⋅ , ⋅ ) c(\cdot,\cdot) c ( ⋅ , ⋅ ) .
Sign Weighted L1 Regularization: From Rich-Regime Sparsity to Stable CL
We aim to induce a sparsity bias that is both practical for continual learning and readily analyzable using linear dynamics (e.g., with the high-dimensional characterization in Section 3.1). To this end, we employ the following weighted L2 regularizer:
f ( θ , θ t − 1 ) = 1 2 ∑ j = 1 d c ( w j ( θ ) , w j ( θ t − 1 ) ) ∑ ℓ = 1 h θ ( ℓ ) , j 2 . f(\theta,\theta_{t-1})=\frac{1}{2}\sum_{j=1}^{d}c(w_j(\theta),w_j(\theta_{t-1}))\sum_{\ell=1}^{h}\theta_{(\ell),j}^{2}. f ( θ , θ t − 1 ) = 2 1 j = 1 ∑ d c ( w j ( θ ) , w j ( θ t − 1 )) ℓ = 1 ∑ h θ ( ℓ ) , j 2 .
choosing c ( w j , w t − 1 , j ) = 1 [ sign ( w j ) w t − 1 , j ] + ϵ c(w_j,w_{t-1,j})=\frac{1}{[\operatorname{sign}(w_j)w_{t-1,j}]+\epsilon} c ( w j , w t − 1 , j ) = [ sign ( w j ) w t − 1 , j ] + ϵ 1 where ϵ > 0 \epsilon>0 ϵ > 0 is a fixed constant.
Importantly, by Proposition 5, this regularizer, which acts in the DLN parameter space, induces a predictor-space sign weighted L1 regularization (SW-L1):
f ( w , w t − 1 ) = ∑ i = 1 d ∣ w i ∣ [ sign ( w i ) w t − 1 , i ] + ϵ . (5) f(\mathbf{w},\mathbf{w}_{t-1})=\sum_{i=1}^{d}\frac{|w_i|}{[\operatorname{sign}(w_i)w_{t-1,i}]+\epsilon}. \tag*{(5)} f ( w , w t − 1 ) = i = 1 ∑ d [ sign ( w i ) w t − 1 , i ] + ϵ ∣ w i ∣ . (5)
Our sign weighted L1 regularizer f f f is designed to satisfy two key properties.
A) Convexity and homogeneity in the first argument. We required convexity in Algorithm 1 to facilitate the high-dimensional characterization of Section 3.1. Homogeneity implies a lower bound on the excess risk, as we will show in Lemma 8.
B) Promotes sparse but stable continual learning. Common nonhomogeneous regularizers in CL (e.g., ∥ w − w t − 1 ∥ 2 2 \|\mathbf{w}-\mathbf{w}_{t-1}\|_{2}^{2} ∥ w − w t − 1 ∥ 2 2 ) aim to preserve past expertise by centering norms at previous iterates. In contrast, we encode this preservation through the weighting function c ( w i , w t − 1 , i ) c(w_i,w_{t-1,i}) c ( w i , w t − 1 , i ) while maintaining homogeneity. As illustrated in fig:1a , this weighting promotes not only sparsity but also preservation of the coordinate-wise signs of the previous iterate.
Our results in this section show that sign weighted L1 regularization achieves the optimal convergence rate in sparsity factor s s s within the broader family of positively homogeneous regularizers. Proofs are in App. F.
Theorem 7 (SW-L1 decays linearly in the sparsity factor s s s ). For any s ≤ s 0 = Θ ν , δ ( 1 ) s\le s_0=\Theta_{\nu,\delta}(1) s ≤ s 0 = Θ ν , δ ( 1 ) and any λ ≥ λ 0 > 0 \lambda\ge\lambda_0>0 λ ≥ λ 0 > 0 , set ϵ = 2 s log ( 64 / s ) \epsilon=\sqrt{\frac{2}{s}\log(64/s)} ϵ = s 2 log ( 64/ s ) . Then, for every task T ≥ 2 T\ge2 T ≥ 2 ,
L ( w T ) − inf w L ( w ) = O ν , δ ( s ) . \mathcal{L}(\mathbf{w}_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})=O_{\nu,\delta}(s). L ( w T ) − w inf L ( w ) = O ν , δ ( s ) .
The rate above contrasts with L1 regularization which converges only as s log ( 1 s ) s\log\left(\frac{1}{s}\right) s log ( s 1 ) , as shown in Lemma 3. This also shows that the advantage of SW-L1 emerges only from the second task: at the first task, where w 0 = 0 \mathbf{w}_0 = 0 w 0 = 0 , the sign-weighted penalty reduces to L1.
Under homogeneous weighting functions c ( ⋅ , ⋅ ) c(\cdot,\cdot) c ( ⋅ , ⋅ ) (like ours), the penalties induced by Proposition 5 are overall positively homogeneous as well. This class has an intrinsic limitation: regardless of how the weights are chosen, its excess risk does not vanish with the task horizon for a fixed sparsity level. This shows that the upper bound of Theorem 7 is tight, and that sign weighted L1 is an optimal positively homogeneous regularizer.
Lemma 8 (Non-vanishing error for positively homogeneous regularizers). Under the statistical setting (Section 2). Assume that the regularizer is order ξ > 0 \xi> 0 ξ > 0 homogeneous. Then, for every T ≥ 1 T \ge1 T ≥ 1 and s ∈ ( 0 , 1 ) s \in(0,1) s ∈ ( 0 , 1 ) ,
L ( w T ) − inf w L ( w ) = Ω ξ , ν , ρ , δ ( s ) . L(\mathbf{w}_T) - \inf_{\mathbf{w}} L(\mathbf{w}) = \Omega_{\xi,\nu,\rho,\delta}(s). L ( w T ) − w inf L ( w ) = Ω ξ , ν , ρ , δ ( s ) .
Experiments: SW-L1 Best Exploits Sparsity
Setup
In this section, we empirically evaluate the performance of the three regularizers considered in our paper: L2, L1, and sign weighted L1 (Eqs. (2) , (3) and (5) , respectively). We consider three increasingly complex settings: logistic regression on synthetic Gaussian data, logistic regression on MNIST, and a two-layer ReLU network on MNIST. In each setting, a fixed sparse teacher generates binary labels with label-flip noise of ρ = 0.1 \rho= 0.1 ρ = 0.1 , and the student is updated sequentially. Performance is measured by the normalized loss ( L ( w t ) − L B a y e s ) / L B a y e s (L(\mathbf{w}_t)-L_{\mathrm{Bayes}})/L_{\mathrm{Bayes}} ( L ( w t ) − L Bayes ) / L Bayes , where L B a y e s L_{\mathrm{Bayes}} L Bayes denotes the entropy of the teacher-induced label distribution representing the irreducible minimum loss.
At each task, the student observes n = 1,000 n = 1{,}000 n = 1 , 000 randomly selected samples and minimizes the regularized objective: using proximal FISTA for linear classification (Beck & Teboulle [3 ] ) and Adam for NNs (Kingma & Ba (2015)). Full implementation details are in App. A.
Linear classifier for Gaussian synthetic tasks. We consider data points sampled according to x i ∼ N ( 0 , d − 1 I ) \mathbf{x}_i \sim\mathcal{N}(0,d^{-1}\mathbf{I}) x i ∼ N ( 0 , d − 1 I ) with d = 500 d = 500 d = 500 . Also, the teacher’s components are sampled from the distribution, as specified in Eq. (4) with ν = 5 \nu= 5 ν = 5 . This experiment most closely matches our theoretical setting and provides direct empirical validation of our theoretical predictions.
Linear classifier for MNIST. We convert MNIST (LeCun et al. (1998)) into a binary classification problem by partitioning digits into 0–4 versus 5–9. Each pixel is standardized using the training set statistics. The teacher is a sparse logistic model whose support follows a Fisher-based pixel ranking (Duda et al. [13 ] ).
Two-layer ReLU for MNIST. We use the previously considered MNIST setting, with the student as a two-layer ReLU network with m = 2,000 m = 2{,}000 m = 2 , 000 neurons trained with Adam. The teacher is a sparse subset of a pretrained dense network, where the active neurons are selected by neuron contribution score. The sign weighted L1 regularizer is regularized by the equivalence established in Proposition 5.
Results
Figure 3 reports the held-out test loss after learning a sequence of T T T tasks across three sparsity levels. Curves are averaged over 5 seeds, with shaded regions indicating one standard deviation.
Figure 3. Effect of teacher sparsity across models and datasets. sign weighted L1 performs best at high teacher sparsity and remains competitive at moderate sparsity, while L2 becomes preferable as sparsity decreases.
Across all settings, regularizer performance depends on teacher sparsity. With highly sparse teachers, L1 leverages sparsity and often beats L2, while SW-L1 provides even larger gain by reducing the loss rapidly and attaining the best performance. As sparsity decreases, this advantage becomes less pronounced, the L1 plateaus early, whereas L2 keeps improving and eventually reaches lower loss. Still, in the two linear model setups, SW-L1 converges quickly and remains better than L2 over the first tasks even at low sparsity.
Related Work & Discussion
We propose a sign weighted L1 regularizer that exploits sparsity, remains stable over long task horizons, and preserves sparsity bias in continual learning with two-layer diagonal linear networks. It converges substantially faster than existing regularizers, both theoretically and empirically. Below, we relate our results to prior work and discuss future directions.
CL theory beyond linear models. While a few theoretical works extend beyond linear models, their guarantees are limited to local convergence (Han et al. (2023), Schrliesman et al. (2026)) or neural networks in the lazy regime, where the model stays near its initialization (Bennani et al. [5 ] , Doan et al. [11 ] , Lee et al. (2021), Taheri et al. (2026)). We contribute a complementary direction by exploiting implicit sparsity biases of deeper models through their induced regularization in predictor space. This allows us to derive global convergence guarantees in a statistical setting while capturing rich-regime feature learning and sparsity bias.
Graldi et al. (2025) study the impact of model scale and feature learning (lazy and rich regimes), for a model trained sequentially via gradient descent. However, their analysis is specific to these dynamics, whereas ours applies to general decomposable convex regularizers used in practice (e.g., Kirkpatrick et al. (2017)).
L1 variants. Improving the performance of L1 regularization has been the focus of several works in the single-task setting . Examples include reweighted L1 regularization (Candes et al., 2008; Foucart & Lai, 2009; Daubechies et al., 2010; Zhao & Li, 2012), nonconcave penalties for variable selection [15 ] , the minimax concave penalty (Zhang, 2010), and, more recently, L1 − 2 _{1-2} 1 − 2 regularization (Hu et al., 2026).
Continual L2 regularization in linear settings. Several theoretical works have analyzed isotropic L2 regularization in continual learning (e.g., Li et al., 2023; Zhao et al., 2024; Levinstein et al., 2025; Karpel et al., 2026). However, these works consider the more analytically tractable continual linear regression setting, whereas ours is the first to rigorously characterize continual classification under different regularization types and strengths. Interestingly, our findings in § 3.2 qualitatively mirror several results from the regression setting. For example, Karpel et al. (2026), whose statistical setup is closest to ours, also find that the optimal L2 regularization strength scales nearly linearly with the number of tasks, consistent with Theorem 2. In a related setting, Karpel & Soudry (2026) find a similar inverse scaling law for the step size in a model trained continually using N N N gradient steps and step size γ \gamma γ . Levinstein et al. (2025) derive a near-optimal regularization strength scaling as ln ( T ) \ln(T) ln ( T ) ; however, other choices may perform better, since their result optimizes a worst-case upper bound . Cai & Diakonikolas [7 ] analyze a broader proximal framework, but require a cyclic ordering of tasks. We also note Evron et al. [14 ] , who analyze continual linear classification but require a regularization strength λ → 0 \lambda\to0 λ → 0 , primarily yielding a theoretical proxy to unregularized settings.
Finally, concurrent work (Guo et al., 2026) studies sparse continual linear regression with an additional regularization matrix constructed from the empirical covariance matrix. Their (global) iterates are essentially equivalent to
w T = argmin w ( 1 2 n ∑ m = 1 T ∥ X m w − y m ∥ 2 2 + λ ∥ w ∥ 1 ) \mathbf{w}_{T}=\underset{\mathbf{w}}{\operatorname{argmin}}\left(\frac{1}{2n}\sum_{m=1}^{T}\lVert\mathbf{X}_{m}\mathbf{w}-\mathbf{y}_{m}\rVert_{2}^{2}+\lambda\lVert\mathbf{w}\rVert_{1}\right) w T = w argmin ( 2 n 1 m = 1 ∑ T ∥ X m w − y m ∥ 2 2 + λ ∥ w ∥ 1 )
i.e. , their method balances the cumulative loss over all tasks seen so far against the L1 norm of the shared parameter vector. However, even for their linear setup, this method requires the storage and continual updating of d 2 d^{2} d 2 parameters, which can limit practical scalability (Kirkpatrick et al., 2017; Schwarz et al., 2018; Karpel & Soudry, 2026). In contrast, we use a practical alternative that requires maintaining only two parameters and induces analyzable implicit biases beyond shallow linear models, specifically in diagonal linear networks.
Open directions. Several assumptions could be relaxed, e.g. , by allowing multiple teachers, broader statistical feature models, or multiclass settings. A particularly interesting direction is to identify other implicit biases, potentially in deeper or more practical models, and use them to design improved weighting schemes. Such theory-guided designs could lead to better empirical continual regularization methods.
Supplementary Materials: Experiments and Proofs
Additional implementation details for the Experiments
Setup
All three experiments share the following design. A fixed sparse teacher generates binary labels at each task via a bit-flip model with noise parameter ρ = 0.1 \rho= 0.1 ρ = 0.1 . In the Gaussian and linear MNIST settings the teacher is a sparse linear model w ⋆ \mathbf{w}_{\star} w ⋆ and labels satisfy
P ( y = 1 ∣ x ) = ρ + ( 1 − 2 ρ ) σ ( x T w ⋆ ) . (6) \mathbb{P}(y = 1 \mid\mathbf{x}) = \rho+ (1 - 2\rho)\sigma(\mathbf{x}^{\mathsf{T}}\mathbf{w}_{\star}). \tag*{(6)} P ( y = 1 ∣ x ) = ρ + ( 1 − 2 ρ ) σ ( x T w ⋆ ) . (6)
In the ReLU MNIST setting the teacher is a sparse two-layer ReLU network f θ ⋆ f_{\theta}^{\star} f θ ⋆ with k k k active neurons, and labels satisfy
P ( y = 1 ∣ x ) = ρ + ( 1 − 2 ρ ) σ ( ∑ i = 1 k w 2 , i ⋆ ReLU ( w 1 , i ⋆ T x ) ) . (7) \mathbb{P}(y = 1 \mid\mathbf{x}) = \rho+ (1 - 2\rho)\sigma\left(\sum_{i=1}^{k} w_{2,i}^{\star}\operatorname{ReLU}(\mathbf{w}_{1,i}^{\star\mathsf{T}}\mathbf{x})\right). \tag*{(7)} P ( y = 1 ∣ x ) = ρ + ( 1 − 2 ρ ) σ ( i = 1 ∑ k w 2 , i ⋆ ReLU ( w 1 , i ⋆ T x ) ) . (7)
The student is initialized at zero (or from a random Gaussian for the ReLU model) and never revisits past data. Hyperparameters ( λ , ϵ ) (\lambda,\epsilon) ( λ , ϵ ) are selected by pooled validation NLL: the configuration minimizing the mean validation loss across all seeds at the final task is applied to every seed. For all experiments we find the optimal hyperparameter λ \lambda λ using an exhaustive search. The test set is withheld until final reporting.
The three regularizers are:
L 2 : λ ∥ w − w t − 1 ∥ 2 2 , L 1 : λ ∥ w − w t − 1 ∥ 1 , sign weighted L1 : λ ∑ i ∣ w i ∣ [ sign ( w i ) w t − 1 , i ] + + ϵ . \mathrm{L2}: \quad\lambda\lVert\mathbf{w}-\mathbf{w}_{t-1}\rVert_{2}^{2}, \qquad\mathrm{L1}: \quad\lambda\lVert\mathbf{w}-\mathbf{w}_{t-1}\rVert_{1}, \qquad\text{sign weighted L1}: \quad\lambda\sum_i\frac{\lvert w_i\rvert}{[\operatorname{sign}(w_i)w_{t-1,i}]_{+}+\epsilon}. L2 : λ ∥ w − w t − 1 ∥ 2 2 , L1 : λ ∥ w − w t − 1 ∥ 1 , sign weighted L1 : λ i ∑ [ sign ( w i ) w t − 1 , i ] + + ϵ ∣ w i ∣ .
Gaussian Synthetic Experiment
Data. Data samples are drawn i.i.d. as x ∼ N ( 0 , d − 1 I ) \mathbf{x}\sim\mathcal{N}(\mathbf{0},d^{-1}\mathbf{I}) x ∼ N ( 0 , d − 1 I ) . A fixed training pool of 50,000 points is drawn once; validation and test sets each contain 5,000 independent points.
Teacher. w ⋆ ∈ R d \mathbf{w}_{\star}\in\mathbb{R}^{d} w ⋆ ∈ R d has exactly k k k nonzero entries drawn using the spike-and-slab model as in Eq. (4) . One teacher is used per sparsity level, shared across seeds.
Protocol. The dimension is d = 500 d=500 d = 500 , T = 30 T=30 T = 30 tasks, n = 1 , 000 n=1,000 n = 1 , 000 samples per task, three seeds. Sparsity levels: s ∈ { 0.001 , 0.01 , 0.1 } s\in\{0.001,0.01,0.1\} s ∈ { 0.001 , 0.01 , 0.1 } .
Student optimizer. Proximal FISTA ([3 ] ) with adaptive restart and backtracking line search. Step size initialized from a power-iteration estimate of the Lipschitz constant (40 iterations, 1.05 × 1.05\times 1.05 × inflation). Up to 1,000 iterations; early stopping when the normalized proximal-gradient residual falls below 10 − 6 10^{-6} 1 0 − 6 .
Linear MNIST Experiment
Data. MNIST is binarized as { 0 , … , 4 } \{0,\ldots,4\} { 0 , … , 4 } vs. { 5 , … , 9 } \{5,\ldots,9\} { 5 , … , 9 } , d = 784 d=784 d = 784 . Pixels are standardized per-coordinate on the training split using the empirical mean and std (and divided by d \sqrt{d} d ). Training pool consisting of 50,000 images. Validation of 5,000 images and Test with 10,000 images.
Teacher. Pixels are ranked by a supervised Fisher score on all 60,000 training images. For each sparsity level, the top-k k k pixels are extracted and a logistic teacher is refitted on the full training set.
More specifically, for each pixel coordinate j ∈ { 1 , … , 784 } j \in\{1,\ldots,784\} j ∈ { 1 , … , 784 } , let μ j , c = 1 n c ∑ i : y i = c x i j \mu_{j,c} = \frac{1}{n_c}\sum_{i:y_i=c}x_{ij} μ j , c = n c 1 ∑ i : y i = c x ij and σ j , c 2 = 1 n c ∑ i : y i = c ( x i j − μ j , c ) 2 \sigma_{j,c}^{2} = \frac{1}{n_c}\sum_{i:y_i=c}(x_{ij}-\mu_{j,c})^2 σ j , c 2 = n c 1 ∑ i : y i = c ( x ij − μ j , c ) 2 denote the class-conditional mean and variance of coordinate j j j for class c ∈ { 0 , 1 } c \in\{0,1\} c ∈ { 0 , 1 } , computed once on the full training set. The Fisher score of coordinate j j j is then
F j = ( μ j , 1 − μ j , 0 ) 2 σ j , 1 2 + σ j , 0 2 . (8) F_j = \frac{(\mu_{j,1}-\mu_{j,0})^2}{\sigma_{j,1}^{2}+\sigma_{j,0}^{2}}. \tag*{(8)} F j = σ j , 1 2 + σ j , 0 2 ( μ j , 1 − μ j , 0 ) 2 . (8)
We rank all 784 coordinates in descending order of F j F_j F j and retain the top k = ⌊ s ⋅ 784 ⌋ k = \lfloor s \cdot784\rfloor k = ⌊ s ⋅ 784 ⌋ coordinates from the teacher support.
Protocol. The dimension is d = 28 × 28 = 784 d = 28 \times28 = 784 d = 28 × 28 = 784 with T = 30 T = 30 T = 30 tasks, n = 1 , 000 n = 1,000 n = 1 , 000 samples per task, and three seeds. Sparsity levels: s ∈ { 0.001 , 0.01 , 0.1 } s \in\{0.001, 0.01, 0.1\} s ∈ { 0.001 , 0.01 , 0.1 }
Student optimizer. Identical to the Gaussian experiment.
Two-Layer ReLU MNIST Experiment
Data. Same binarized MNIST as the linear experiment. Pixels are standardized per-coordinate using the empirical mean and std.
Protocol. The dimension is d = 28 × 28 = 784 d = 28 \times28 = 784 d = 28 × 28 = 784 with T = 30 T = 30 T = 30 tasks, n = 1 , 000 n = 1,000 n = 1 , 000 samples per task, three seeds. Validation set: 2,000 images (held out from the 10,000 MNIST test images); test set: the remaining 8,000.
Teacher. A two-layer ReLU network with m = 2 , 000 m = 2,000 m = 2 , 000 neurons, f ( x ) = ∑ i w 2 , i ReLU ( w 1 , i T x ) f(x) = \sum_i w_{2,i}\operatorname{ReLU}(w_{1,i}^{\mathsf{T}}x) f ( x ) = ∑ i w 2 , i ReLU ( w 1 , i T x ) , is trained by Adam for 25 epochs (lr 10 − 3 10^{-3} 1 0 − 3 , weight decay 10 − 4 10^{-4} 1 0 − 4 , batch size 512) on the 50,000 training images. Neurons are ranked by contribution score ∣ w 2 , i ∣ ⋅ std ( ReLU ( w 1 , i T X ) ) \lvert w_{2,i}\rvert\cdot\operatorname{std}(\operatorname{ReLU}(w_{1,i}^{\mathsf{T}}X)) ∣ w 2 , i ∣ ⋅ std ( ReLU ( w 1 , i T X )) . For each sparsity level, the top-k k k neurons are retained and fine-tuned for 10 additional epochs; the remaining neurons are zeroed.
Sign weighted regularizer for ReLU. The sign weighted L1 penalty is applied in its weighted-L2 on the weights form. For neuron i i i the coefficient c i = 1 [ sign ( w t − 1 , i ( 2 ) ) w t − 1 , i ( 2 ) ] + + ϵ c_i = \frac{1}{[\operatorname{sign}(w_{t-1,i}^{(2)})w_{t-1,i}^{(2)}]_{+}+\epsilon} c i = [ sign ( w t − 1 , i ( 2 ) ) w t − 1 , i ( 2 ) ] + + ϵ 1 is computed with a stop-gradient on sign ( w i ( 2 ) ) \operatorname{sign}(w_i^{(2)}) sign ( w i ( 2 ) ) , consistent with treating c i c_i c i as fixed at the previous iterate, the penalty 1 2 c i ( ∥ w 1 , i ∥ 2 + w 2 , i 2 ) \frac{1}{2}c_i(\lVert w_{1,i}\rVert^2+w_{2,i}^2) 2 1 c i (∥ w 1 , i ∥ 2 + w 2 , i 2 ) is added to the loss.
Student optimizer. Both layers trained jointly by Adam (lr 3 × 10 − 3 3 \times10^{-3} 3 × 1 0 − 3 , batch size 256, 300 epochs per task). Student initialized with w 1 , i ∼ N ( 0 , d − 1 ) w_{1,i} \sim\mathcal{N}(0,d^{-1}) w 1 , i ∼ N ( 0 , d − 1 ) and w 2 , i ∼ N ( 0 , m − 1 ) w_{2,i} \sim\mathcal{N}(0,m^{-1}) w 2 , i ∼ N ( 0 , m − 1 ) . Proximal step for L1/L2 applied once per epoch (after all mini-batches).
Summary Table
Table 1 (PDF p. 14). Key parameters across the three experimental settings.
L2 regularization does not exploit sparsity
Experimental setup. We work in the teacher–student setting of Algorithm 1. Features are drawn i.i.d. as x ∼ N ( 0 , I ) \mathbf{x} \sim\mathcal{N}(0, I) x ∼ N ( 0 , I ) with d = 256 d = 256 d = 256 . The teacher w ⋆ \mathbf{w}_{\star} w ⋆ is k k k -sparse: a support of size k = s d k = sd k = s d is drawn uniformly at random, the entries on it are drawn from Π = N ( 0 , 1 ) \Pi= \mathcal{N}(0, 1) Π = N ( 0 , 1 ) , and w ⋆ \mathbf{w}_{\star} w ⋆ is then rescaled so that ν = ∥ w ⋆ ∥ / d \nu= \lVert\mathbf{w}_{\star} \rVert/\sqrt{d} ν = ∥ w ⋆ ∥ / d equals a constant. Labels are generated by the teacher through the noisy logistic model described in § 2 with label noise ρ = 0 \rho= 0 ρ = 0 .
The predictor is obtained from Algorithm 1. Each task is solved to convergence by full-batch Adam. We report the excess risk of the last predictor estimated on an independent test set of 8,000 samples labeled by the same teacher, and inf w L ( w ) \inf_{\mathbf{w}} \mathcal{L}(\mathbf{w}) inf w L ( w ) is the corresponding Bayes risk, computed in closed form from the teacher logits. The regularization strength λ \lambda λ is selected for each method and each configuration separately, by minimizing the loss at the final task on a held-out validation set of 4,000 samples labeled by the same teacher.
To isolate the effect of sparsity we sweep the teacher sparsity s s s while holding every other quantity of the problem fixed, and fit log L ( w T ) − inf w L ( w ) ≈ β log s + c o n s t \log\mathcal{L}(\mathbf{w}_T) - \inf_{\mathbf{w}} \mathcal{L}(\mathbf{w}) \approx\beta\log s + \mathrm{const} log L ( w T ) − inf w L ( w ) ≈ β log s + const . Figure 4 reports the outcome for temporal L2: the fitted slope is β = + 0.005 \beta= +0.005 β = + 0.005 , and the excess risk stays within 0.155-0.163 across the entire range. This indicates that the L2 penalty does not exploit sparsity of the teacher.
Proofs for Section 3.1
CGMT proof for the bit-flip logistic teacher (general decomposable regularizer)
Full Theorem 1. Under the statistical model (§ 2 ), assume that the system (9) admits a unique solution ( α ˉ t , σ ˉ t , r ˉ t , τ ˉ t , γ ˉ t , θ ˉ t ) (\bar{\alpha}_t,\bar{\sigma}_t,\bar{r}_t,\bar{\tau}_t,\bar{\gamma}_t,\bar{\theta}_t) ( α ˉ t , σ ˉ t , r ˉ t , τ ˉ t , γ ˉ t , θ ˉ t ) for every t ≤ T t \le T t ≤ T . Then as n , d → ∞ n,d \to\infty n , d → ∞ , after learning T T T tasks with Algorithm 1, for every locally Lipschitz function with at most quadratic growth Ψ : R 2 → R 2 \Psi:\mathbb{R}^2 \to\mathbb{R}^2 Ψ : R 2 → R 2 ,
1 d ∑ j = 1 d Ψ ( w T , j , w ⋆ , j ) → P E [ Ψ ( Prox η T f ~ ( ⋅ , F T − 1 ) ( a T w ⋆ + b T Z ) , w ⋆ ) ] . \frac{1}{d}\sum_{j=1}^{d}\Psi(w_{T,j},w_{\star,j}) \xrightarrow{\mathbb{P}} \mathbb{E}\left[\Psi\left(\operatorname{Prox}_{\eta_T\tilde{f}(\cdot,F_{T-1})}(a_Tw^\star+b_TZ),w^\star\right)\right]. d 1 j = 1 ∑ d Ψ ( w T , j , w ⋆ , j ) P E [ Ψ ( Prox η T f ~ ( ⋅ , F T − 1 ) ( a T w ⋆ + b T Z ) , w ⋆ ) ] . where F T − 1 F_{T-1} F T − 1 and w ⋆ ∼ Π w^\star\sim\Pi w ⋆ ∼ Π are independent of Z ∼ N ( 0 , 1 ) Z \sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) , F t − 1 F_{t-1} F t − 1 is any coordinatewise state so that for c > 0 c>0 c > 0 bounded away from 0 0 0 , M λ f ~ ( ⋅ , F ) ( x , c ) 3 M_{\lambda\tilde{f}(\cdot,F)}(x,c)^{3} M λ f ~ ( ⋅ , F ) ( x , c ) 3 is continuous and satisfies M λ f ~ ( ⋅ , F ) ( x , c ) ≤ C ( 1 + x 2 + F 2 ) M_{\lambda\tilde{f}(\cdot,F)}(x,c)\le C(1+x^2+F^2) M λ f ~ ( ⋅ , F ) ( x , c ) ≤ C ( 1 + x 2 + F 2 ) for C > 0 C>0 C > 0 . Also, the past state has a deterministic empirical limit i.e., 1 d ∑ j = 1 d δ ( w ⋆ , j , F t − 1 , j ) → P μ t − 1 \frac{1}{d}\sum_{j=1}^{d}\delta_{(w_{\star,j},F_{t-1,j})}\xrightarrow{\mathbb{P}}\mu_{t-1} d 1 ∑ j = 1 d δ ( w ⋆ , j , F t − 1 , j ) P μ t − 1 in the 2-Wasserstein distance. Moreover, the saddle-point parameters satisfy for all t ≤ T t\le T t ≤ T the nonlinear system
ν 2 α t = E [ w ⋆ Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + r t δ Z ) ) ] , γ t = 1 r t δ E [ Z Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + r t δ Z ) ) ] , ν 2 α t 2 + σ t 2 = E [ Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + r t δ Z ) ) 2 ] , γ t 2 = 2 r t 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ( ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] , γ t θ t = − 2 E [ π ρ ( ν Z 1 ) Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ] , 1 − γ t σ t τ t = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . (9) \begin{aligned}
\nu^2\alpha_t&=\mathbb{E}\left[w^\star\operatorname{Prox}_{\lambda\sigma_t\tau_t\tilde{f}(\cdot,F_{t-1})}\left(\sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z\right)\right)\right],\\
\gamma_t&=\frac{1}{r_t\sqrt{\delta}}\mathbb{E}\left[Z\operatorname{Prox}_{\lambda\sigma_t\tau_t\tilde{f}(\cdot,F_{t-1})}\left(\sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z\right)\right)\right],\\
\nu^2\alpha_t^2+\sigma_t^2&=\mathbb{E}\left[\operatorname{Prox}_{\lambda\sigma_t\tau_t\tilde{f}(\cdot,F_{t-1})}\left(\sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z\right)\right)^2\right],\\
\gamma_t^2&=\frac{2}{r_t^2}\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\left(\nu\alpha_tZ_1+\sigma_tZ_2-\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)^2\right],\\
\gamma_t\theta_t&=-2\mathbb{E}\left[\pi_\rho(\nu Z_1)\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right],\\
1-\frac{\gamma_t}{\sigma_t\tau_t}&=\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))}{1+\gamma_t\phi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}\right]. \tag*{(9)}
\end{aligned} ν 2 α t γ t ν 2 α t 2 + σ t 2 γ t 2 γ t θ t 1 − σ t τ t γ t = E [ w ⋆ Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + δ r t Z ) ) ] , = r t δ 1 E [ Z Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + δ r t Z ) ) ] , = E [ Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + δ r t Z ) ) 2 ] , = r t 2 2 E [ ( 1 − π ρ ( ν Z 1 )) ( ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] , = − 2 E [ π ρ ( ν Z 1 ) Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ] , = E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ( 1 − π ρ ( ν Z 1 )) ] . (9) Here Z , Z 1 , Z 2 ∼ i i d N ( 0 , 1 ) Z,Z_1,Z_2 \overset{\mathrm{iid}}{\sim}\mathcal{N}(0,1) Z , Z 1 , Z 2 ∼ iid N ( 0 , 1 ) , and η t = λ σ ˉ t τ ˉ t \eta_t=\lambda\bar{\sigma}_t\bar{\tau}_t η t = λ σ ˉ t τ ˉ t , a t = σ ˉ t τ ˉ t θ ˉ t a_t=\bar{\sigma}_t\bar{\tau}_t\bar{\theta}_t a t = σ ˉ t τ ˉ t θ ˉ t , b t = σ ˉ t τ ˉ t r ˉ t δ b_t=\frac{\bar{\sigma}_t\bar{\tau}_t\bar{r}_t}{\sqrt{\delta}} b t = δ σ ˉ t τ ˉ t r ˉ t .
Remark 9 (About the uniqueness of the non-linear system (9)). Earlier CGMT (Thrampoulidis et al. (2015)) analyses of regularized logistic regression, such as Salehi et al. (2019), assume uniqueness of the associated nonlinear system rather than prove it. However, through numerical evaluations they observed that a fixed-point iterative method converges to a stable solution (see their Remark 2). More recent work has established existence and uniqueness of analogous state-equation systems for broad classes of convex regularized M M M -estimators in linear models (Bellec & Koriyama [4 ] ), and for unregularized single-index models, including logistic regression, above some critical threshold δ ∞ \delta_\infty δ ∞ (Koriyama & Bellec (2025)). Although their results do not directly cover our regularized single-index setting, they provide further support for the plausibility of the uniqueness assumption.
Proof. The proof will consist of three main steps, as in Salehi et al. (2019) (see Appendix § 9 )
We begin by expressing the optimization problem in bilinear form in order to apply CGMT and derive its corresponding auxiliary optimization problem.
We then simplify the auxiliary optimization problem. Characterizing the problem via a scalar optimization.
Finally, we solve the auxiliary optimization problem by deriving and solving the first-order optimality conditions.
We begin with characterizing the limiting performance of a specific iteration t t t , for ease of presentation we will omit the dependence on t t t and write X t = X X_t = X X t = X , y t = y y_t = y y t = y . The optimization problem is
w ^ ∈ argmin w ∈ R d 1 n ∑ i = 1 n [ log ( 1 + e x i T w ) − y i x i T w ] + λ d f ( w ) (10) \widehat{\mathbf{w}} \in\operatorname*{argmin}_{\mathbf{w}\in\mathbb{R}^d} \frac{1}{n}\sum_{i=1}^{n}\left[\log\left(1+e^{\mathbf{x}_i^{\mathsf{T}}\mathbf{w}}\right)-y_i\mathbf{x}_i^{\mathsf{T}}\mathbf{w}\right]+\frac{\lambda}{d}f(\mathbf{w}) \tag*{(10)} w ∈ w ∈ R d argmin n 1 i = 1 ∑ n [ log ( 1 + e x i T w ) − y i x i T w ] + d λ f ( w ) (10)
where f f f is convex, and decomposable.
The optimization problem in Eq. (10) can be re-written as
w ^ ∈ argmin w ∈ R d { 1 n 1 T ϕ ( X w ) − 1 n y T X w + λ d f ( w ) } . (11) \widehat{\mathbf{w}} \in\operatorname*{argmin}_{\mathbf{w}\in\mathbb{R}^d}\left\{\frac{1}{n}\mathbf{1}^{\mathsf{T}}\phi(\mathbf{X}\mathbf{w})-\frac{1}{n}\mathbf{y}^{\mathsf{T}}\mathbf{X}\mathbf{w}+\frac{\lambda}{d}f(\mathbf{w})\right\}. \tag*{(11)} w ∈ w ∈ R d argmin { n 1 1 T ϕ ( Xw ) − n 1 y T Xw + d λ f ( w ) } . (11)
Under the statistical model, P ( y i = 1 ∣ x i ) = ρ + ( 1 − 2 ρ ) σ ( x i T w ⋆ ) \mathbb{P}(y_i=1\mid x_i)=\rho+(1-2\rho)\sigma(x_i^{\mathsf{T}}w_\star) P ( y i = 1 ∣ x i ) = ρ + ( 1 − 2 ρ ) σ ( x i T w ⋆ ) , with σ ( x ) = 1 1 + e − x \sigma(x)=\frac{1}{1+e^{-x}} σ ( x ) = 1 + e − x 1 hence P ( y i = 1 ∣ x i ) = π ρ ( x i T w ⋆ ) \mathbb{P}(y_i=1\mid x_i)=\pi_\rho(x_i^{\mathsf{T}}w_\star) P ( y i = 1 ∣ x i ) = π ρ ( x i T w ⋆ ) , where π ρ ( g ) = ρ + ( 1 − 2 ρ ) σ ( g ) \pi_\rho(g)=\rho+(1-2\rho)\sigma(g) π ρ ( g ) = ρ + ( 1 − 2 ρ ) σ ( g ) . We therefore obtain the following identities,
π ρ ( − g ) = 1 − π ρ ( g ) , π ρ ′ ( g ) = ( 1 − 2 ρ ) σ ( g ) σ ( − g ) = ( 1 − 2 ρ ) ϕ ′ ′ ( g ) . (12) \pi_\rho(-g)=1-\pi_\rho(g),\qquad\pi_\rho'(g)=(1-2\rho)\sigma(g)\sigma(-g)=(1-2\rho)\phi''(g). \tag*{(12)} π ρ ( − g ) = 1 − π ρ ( g ) , π ρ ′ ( g ) = ( 1 − 2 ρ ) σ ( g ) σ ( − g ) = ( 1 − 2 ρ ) ϕ ′′ ( g ) . (12)
Let
H = d [ − x 1 T ⋮ − x n T ] ∈ R n × d , δ = n d . \mathbf{H}=\sqrt{d}\begin{bmatrix}-\mathbf{x}_1^{\mathsf{T}}\\ \vdots\\ -\mathbf{x}_n^{\mathsf{T}}\end{bmatrix}\in\mathbb{R}^{n\times d},\qquad\delta=\frac{n}{d}. H = d − x 1 T ⋮ − x n T ∈ R n × d , δ = d n .
Then the entries of H \mathbf{H} H are i.i.d. standard Gaussian and Eq. (11) is equivalent to
w ^ ∈ argmin w ∈ R d { 1 n 1 T ϕ ( 1 d H w ) − 1 n y T 1 d H w + λ d f ( w ) } . (13) \widehat{\mathbf{w}} \in\operatorname*{argmin}_{\mathbf{w}\in\mathbb{R}^d}\left\{\frac{1}{n}\mathbf{1}^{\mathsf{T}}\phi\left(\frac{1}{\sqrt{d}}\mathbf{H}\mathbf{w}\right)-\frac{1}{n}\mathbf{y}^{\mathsf{T}}\frac{1}{\sqrt{d}}\mathbf{H}\mathbf{w}+\frac{\lambda}{d}f(\mathbf{w})\right\}. \tag*{(13)} w ∈ w ∈ R d argmin { n 1 1 T ϕ ( d 1 Hw ) − n 1 y T d 1 Hw + d λ f ( w ) } . (13)
In order to apply the CGMT, we need a min-max optimization, so we introduce u ∈ R n \mathbf{u}\in\mathbb{R}^n u ∈ R n and rewrite Eq. (13) as
min w ∈ R d , u ∈ R n 1 n 1 T ϕ ( u ) − 1 n y T u + λ d f ( w ) s.t. u = 1 d H w . (14) \min_{\mathbf{w}\in\mathbb{R}^d,\mathbf{u}\in\mathbb{R}^n}\frac{1}{n}\mathbf{1}^{\mathsf{T}}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\mathsf{T}}\mathbf{u}+\frac{\lambda}{d}f(\mathbf{w})\quad\text{s.t.}\quad\mathbf{u}=\frac{1}{\sqrt{d}}\mathbf{H}\mathbf{w}. \tag*{(14)} w ∈ R d , u ∈ R n min n 1 1 T ϕ ( u ) − n 1 y T u + d λ f ( w ) s.t. u = d 1 Hw . (14)
Introducing a Lagrange multiplier v ∈ R n \mathbf{v}\in\mathbb{R}^n v ∈ R n gives us the desired min-max optimization,
min w , u v max 1 n 1 T ϕ ( u ) − 1 n y T u + λ d f ( w ) + 1 n v T ( u − 1 d H w ) . (15) \min_{\mathbf{w},\mathbf{u}\atop\mathbf{v}}\max\frac{1}{n}\mathbf{1}^{\mathsf{T}}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\mathsf{T}}\mathbf{u}+\frac{\lambda}{d}f(\mathbf{w})+\frac{1}{n}\mathbf{v}^{\mathsf{T}}\left(\mathbf{u}-\frac{1}{\sqrt{d}}\mathbf{H}\mathbf{w}\right). \tag*{(15)} v w , u min max n 1 1 T ϕ ( u ) − n 1 y T u + d λ f ( w ) + n 1 v T ( u − d 1 Hw ) . (15)
Remark 10. To apply the CGMT, we would also need to restrict the optimization variables w , u , v \mathbf{w},\mathbf{u},\mathbf{v} w , u , v to some compact and convex sets S w ⊂ R d S_{\mathbf{w}}\subset\mathbb{R}^d S w ⊂ R d , S u ⊂ R n S_{\mathbf{u}}\subset\mathbb{R}^n S u ⊂ R n , S v ⊂ R n S_{\mathbf{v}}\subset\mathbb{R}^n S v ⊂ R n , and perform the min–max optimization over these sets. These sets can be chosen to be arbitrary large and do not affect the optimization phase. For simplicity, we suppress the dependence on ( S w , S u , S v ) (S_{\mathbf{w}},S_{\mathbf{u}},S_{\mathbf{v}}) ( S w , S u , S v ) from the notation.
Since y ~ \widetilde{\mathbf{y}} y depends on H \mathbf{H} H we can not directly apply CGMT. To deal with this problem we introduce P = w ⋆ w ⋆ T ∥ w ⋆ ∥ 2 \mathbf{P}=\frac{\mathbf{w}_\star\mathbf{w}_\star^{\mathsf{T}}}{\lVert\mathbf{w}_\star\rVert^2} P = ∥ w ⋆ ∥ 2 w ⋆ w ⋆ T , and P ⊥ = I d − P \mathbf{P}^{\perp}=\mathbf{I}_d-\mathbf{P} P ⊥ = I d − P . We can now decompose the matrix H \mathbf{H} H to H = H 1 + H 2 \mathbf{H}=\mathbf{H}_1+\mathbf{H}_2 H = H 1 + H 2 where H 1 = H P \mathbf{H}_1=\mathbf{H}\mathbf{P} H 1 = HP , and H 2 = H P ⊥ \mathbf{H}_2=\mathbf{H}\mathbf{P}^{\perp} H 2 = H P ⊥ . Then the optimization becomes
min w , u v max 1 n 1 T ϕ ( u ) − 1 n y T u + λ d f ( w ) + 1 n v T ( u − 1 d H 1 w ) − 1 n d v T H 2 w . (16) \min_{\mathbf{w},\mathbf{u}\atop\mathbf{v}}\max\frac{1}{n}\mathbf{1}^{\mathsf{T}}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\mathsf{T}}\mathbf{u}+\frac{\lambda}{d}f(\mathbf{w})+\frac{1}{n}\mathbf{v}^{\mathsf{T}}\left(\mathbf{u}-\frac{1}{\sqrt{d}}\mathbf{H}_1\mathbf{w}\right)-\frac{1}{n\sqrt{d}}\mathbf{v}^{\mathsf{T}}\mathbf{H}_2\mathbf{w}. \tag*{(16)} v w , u min max n 1 1 T ϕ ( u ) − n 1 y T u + d λ f ( w ) + n 1 v T ( u − d 1 H 1 w ) − n d 1 v T H 2 w . (16)
Since P w ⋆ = w ⋆ \mathbf{P}\mathbf{w}_\star=\mathbf{w}_\star P w ⋆ = w ⋆ , we have
1 d H w ⋆ = 1 d H 1 w ⋆ = ∥ w ⋆ ∥ d q = ν q , q ∼ N ( 0 , I n ) . \frac{1}{\sqrt{d}}\mathbf{H}\mathbf{w}_\star=\frac{1}{\sqrt{d}}\mathbf{H}_1\mathbf{w}_\star=\frac{\lVert\mathbf{w}_\star\rVert}{\sqrt{d}}\mathbf{q}=\nu\mathbf{q},\qquad\mathbf{q}\sim\mathcal{N}(\mathbf{0},\mathbf{I}_n). d 1 H w ⋆ = d 1 H 1 w ⋆ = d ∥ w ⋆ ∥ q = ν q , q ∼ N ( 0 , I n ) .
Under our statistical model Section 2, y i ∣ q i ∼ Ber ( π ρ ( ν q i ) ) y_i\mid q_i\sim\operatorname{Ber}(\pi_\rho(\nu q_i)) y i ∣ q i ∼ Ber ( π ρ ( ν q i )) , where π ρ ( g ) = ρ + ( 1 − 2 ρ ) σ ( g ) \pi_\rho(g)=\rho+(1-2\rho)\sigma(g) π ρ ( g ) = ρ + ( 1 − 2 ρ ) σ ( g ) . Hence ( y , H 1 ) (\mathbf{y},\mathbf{H}_1) ( y , H 1 ) is independent of H 2 \mathbf{H}_2 H 2 , so the CGMT applies. Since the optimization problem as well as the conditions of CGMT apply we write its corresponding auxiliary optimization problem
min w , u v max 1 n 1 T ϕ ( u ) − 1 n y T u + λ d f ( w ) + 1 n v T ( u − 1 d H 1 w ) − 1 n d ( v T h ∥ P ⊥ w ∥ + ∥ v ∥ g T P ⊥ w ) . \min_{\mathbf{w},\mathbf{u}\atop\mathbf{v}}\max\frac{1}{n}\mathbf{1}^{\mathsf{T}}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\mathsf{T}}\mathbf{u}+\frac{\lambda}{d}f(\mathbf{w})+\frac{1}{n}\mathbf{v}^{\mathsf{T}}\left(\mathbf{u}-\frac{1}{\sqrt{d}}\mathbf{H}_1\mathbf{w}\right)-\frac{1}{n\sqrt{d}}\left(\mathbf{v}^{\mathsf{T}}\mathbf{h}\lVert\mathbf{P}^{\perp}\mathbf{w}\rVert+\lVert\mathbf{v}\rVert\mathbf{g}^{\mathsf{T}}\mathbf{P}^{\perp}\mathbf{w}\right). v w , u min max n 1 1 T ϕ ( u ) − n 1 y T u + d λ f ( w ) + n 1 v T ( u − d 1 H 1 w ) − n d 1 ( v T h ∥ P ⊥ w ∥ + ∥ v ∥ g T P ⊥ w ) .
where g ∼ N ( 0 , I d ) \mathbf{g} \sim\mathcal{N}(0,\mathbf{I}_d) g ∼ N ( 0 , I d ) and h ∼ N ( 0 , I n ) \mathbf{h} \sim\mathcal{N}(0,\mathbf{I}_n) h ∼ N ( 0 , I n ) .
We first maximize over v \mathbf{v} v . For fixed ( w , u ) (\mathbf{w},\mathbf{u}) ( w , u ) we have,
max v 1 n d ∥ v ∥ g ⊤ P ⊥ w + 1 n v ⊤ ( u − 1 d H 1 w − ∥ P ⊥ w ∥ d h ) . (17) \max_{\mathbf{v}} \frac{1}{n\sqrt{d}}\|\mathbf{v}\|\mathbf{g}^{\top}\mathbf{P}^{\perp}\mathbf{w}+\frac{1}{n}\mathbf{v}^{\top}\left(\mathbf{u}-\frac{1}{\sqrt{d}}\mathbf{H}_1\mathbf{w}-\frac{\|\mathbf{P}^{\perp}\mathbf{w}\|}{\sqrt{d}}\mathbf{h}\right). \tag*{(17)} v max n d 1 ∥ v ∥ g ⊤ P ⊥ w + n 1 v ⊤ ( u − d 1 H 1 w − d ∥ P ⊥ w ∥ h ) . (17)
Let r = ∥ v ∥ d r=\frac{\|\mathbf{v}\|}{\sqrt{d}} r = d ∥ v ∥ , maximizing the objective would yield a vector v \mathbf{v} v with the same direction of the vector it multiplies, therefore (17) is equivalent to
max r ≥ 0 1 n 1 ⊤ ϕ ( u ) − 1 n y ⊤ u + λ d f ( w ) + r n d g ⊤ P ⊥ w + r ∥ u n − 1 n d H 1 w − ∥ P ⊥ w ∥ n d h ∥ . \max_{r\ge0}\frac{1}{n}\mathbf{1}^{\top}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\top}\mathbf{u}+\frac{\lambda}{d}f(\mathbf{w})+\frac{r}{\sqrt{n d}}\mathbf{g}^{\top}\mathbf{P}^{\perp}\mathbf{w}+r\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{1}{\sqrt{n d}}\mathbf{H}_1\mathbf{w}-\frac{\|\mathbf{P}^{\perp}\mathbf{w}\|}{\sqrt{n d}}\mathbf{h}\right\|. r ≥ 0 max n 1 1 ⊤ ϕ ( u ) − n 1 y ⊤ u + d λ f ( w ) + n d r g ⊤ P ⊥ w + r n u − n d 1 H 1 w − n d ∥ P ⊥ w ∥ h .
Since H 1 = H P \mathbf{H}_1=\mathbf{H}\mathbf{P} H 1 = HP , and writing α = w ⊤ w ⋆ ∥ w ⋆ ∥ 2 \alpha=\frac{\mathbf{w}^{\top}\mathbf{w}_{\star}}{\|\mathbf{w}_{\star}\|^2} α = ∥ w ⋆ ∥ 2 w ⊤ w ⋆ , σ = ∥ P ⊥ w ∥ d \sigma=\frac{\|\mathbf{P}^{\perp}\mathbf{w}\|}{\sqrt{d}} σ = d ∥ P ⊥ w ∥ , q = 1 ∥ w ⋆ ∥ H w ⋆ \mathbf{q}=\frac{1}{\|\mathbf{w}_{\star}\|}\mathbf{H}\mathbf{w}_{\star} q = ∥ w ⋆ ∥ 1 H w ⋆ , we have q ∼ N ( 0 , I n ) \mathbf{q}\sim\mathcal{N}(0,\mathbf{I}_n) q ∼ N ( 0 , I n ) and
1 d H ( P w ) = 1 d H ( α w ⋆ ) = α ∥ w ⋆ ∥ d q = ν α q , \frac{1}{\sqrt{d}}\mathbf{H}(\mathbf{P}\mathbf{w})=\frac{1}{\sqrt{d}}\mathbf{H}(\alpha\mathbf{w}_{\star})=\alpha\frac{\|\mathbf{w}_{\star}\|}{\sqrt{d}}\mathbf{q}=\nu\alpha\mathbf{q}, d 1 H ( Pw ) = d 1 H ( α w ⋆ ) = α d ∥ w ⋆ ∥ q = ν α q ,
with ν = ∥ w ⋆ ∥ d \nu=\frac{\|\mathbf{w}_{\star}\|}{\sqrt{d}} ν = d ∥ w ⋆ ∥ . Therefore the AO becomes
min w , u , α , σ ≥ 0 max r ≥ 0 1 n 1 ⊤ ϕ ( u ) − 1 n y ⊤ u + λ d f ( w ) + r n d g ⊤ P ⊥ w + r ∥ u n − ν α n q − σ n h ∥ . (18) \min_{\mathbf{w},\mathbf{u},\alpha,\sigma\ge0}\max_{r\ge0}\frac{1}{n}\mathbf{1}^{\top}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\top}\mathbf{u}+\frac{\lambda}{d}f(\mathbf{w})+\frac{r}{\sqrt{n d}}\mathbf{g}^{\top}\mathbf{P}^{\perp}\mathbf{w}+r\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{\nu\alpha}{\sqrt{n}}\mathbf{q}-\frac{\sigma}{\sqrt{n}}\mathbf{h}\right\|. \tag*{(18)} w , u , α , σ ≥ 0 min r ≥ 0 max n 1 1 ⊤ ϕ ( u ) − n 1 y ⊤ u + d λ f ( w ) + n d r g ⊤ P ⊥ w + r n u − n ν α q − n σ h . (18)
Finding the optimal direction of w \mathbf{w} w is cumbersome due to the existence of f ( w ) f(\mathbf{w}) f ( w ) in the objective, therefore we now introduce μ , s ∈ R d \boldsymbol{\mu},\mathbf{s}\in\mathbb{R}^d μ , s ∈ R d and rewrite the problem as
min w , u , μ , s , r ≥ 0 max 1 n 1 ⊤ ϕ ( u ) − 1 n y ⊤ u + λ d f ( μ ) + r n d g ⊤ P ⊥ w + r ∥ u n − ν α n q − σ n h ∥ + 1 d s ⊤ ( μ − w ) . (19) \min_{\mathbf{w},\mathbf{u},\boldsymbol{\mu},s,r\ge0}\max\frac{1}{n}\mathbf{1}^{\top}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\top}\mathbf{u}+\frac{\lambda}{d}f(\boldsymbol{\mu})+\frac{r}{\sqrt{n d}}\mathbf{g}^{\top}\mathbf{P}^{\perp}\mathbf{w}+r\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{\nu\alpha}{\sqrt{n}}\mathbf{q}-\frac{\sigma}{\sqrt{n}}\mathbf{h}\right\|+\frac{1}{d}\mathbf{s}^{\top}(\boldsymbol{\mu}-\mathbf{w}). \tag*{(19)} w , u , μ , s , r ≥ 0 min max n 1 1 ⊤ ϕ ( u ) − n 1 y ⊤ u + d λ f ( μ ) + n d r g ⊤ P ⊥ w + r n u − n ν α q − n σ h + d 1 s ⊤ ( μ − w ) . (19)
We then use the same asymptotically-valid min–max flip where flipping the order of optimizations in (AO) is justified whenever such a flipping is allowed in the (PO) (see Appendix C.2 of Salehi et al. (2019)), and optimize first over the direction of P ⊥ w \mathbf{P}^{\perp}\mathbf{w} P ⊥ w . This yields
min μ , u , α , σ ≥ 0 max s , r ≥ 0 1 n 1 ⊤ ϕ ( u ) − 1 n y ⊤ u + λ d f ( μ ) − σ ∥ 1 d P ⊥ ( r δ g − s ) ∥ + r ∥ u n − ν α n q − σ n h ∥ + 1 d ( P s ) ⊤ μ + 1 d ( P ⊥ s ) ⊤ μ − 1 d ( P s ) ⊤ w . \min_{\boldsymbol{\mu},\mathbf{u},\alpha,\sigma\ge0}\max_{s,r\ge0}\frac{1}{n}\mathbf{1}^{\top}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\top}\mathbf{u}+\frac{\lambda}{d}f(\boldsymbol{\mu})-\sigma\left\|\frac{1}{\sqrt{d}}\mathbf{P}^{\perp}\left(\frac{r}{\sqrt{\delta}}\mathbf{g}-\mathbf{s}\right)\right\|+r\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{\nu\alpha}{\sqrt{n}}\mathbf{q}-\frac{\sigma}{\sqrt{n}}\mathbf{h}\right\|+\frac{1}{d}(\mathbf{P}\mathbf{s})^{\top}\boldsymbol{\mu}+\frac{1}{d}(\mathbf{P}^{\perp}\mathbf{s})^{\top}\boldsymbol{\mu}-\frac{1}{d}(\mathbf{P}\mathbf{s})^{\top}\mathbf{w}. μ , u , α , σ ≥ 0 min s , r ≥ 0 max n 1 1 ⊤ ϕ ( u ) − n 1 y ⊤ u + d λ f ( μ ) − σ d 1 P ⊥ ( δ r g − s ) + r n u − n ν α q − n σ h + d 1 ( Ps ) ⊤ μ + d 1 ( P ⊥ s ) ⊤ μ − d 1 ( Ps ) ⊤ w .
Next, we introduce τ , υ ≥ 0 \tau,\upsilon\ge0 τ , υ ≥ 0 to replace the norms by squared norms which simplifies the preceding analysis
min μ , u , α , σ , υ ≥ 0 max s , r , τ ≥ 0 1 n 1 ⊤ ϕ ( u ) − 1 n y ⊤ u + λ d f ( μ ) − σ 2 τ − σ τ 2 ∥ 1 d P ⊥ ( r δ g − s ) ∥ 2 + r 2 υ + r υ 2 ∥ u n − ν α n q − σ n h ∥ 2 + 1 d ( P s ) ⊤ μ + 1 d ( P ⊥ s ) ⊤ μ − 1 d ( P s ) ⊤ w . (20) \begin{aligned}
\min_{\boldsymbol{\mu},\mathbf{u},\alpha,\sigma,\upsilon\ge0}\max_{s,r,\tau\ge0}\frac{1}{n}\mathbf{1}^{\top}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\top}\mathbf{u}+\frac{\lambda}{d}f(\boldsymbol{\mu})-\frac{\sigma}{2\tau}-\frac{\sigma\tau}{2}\left\|\frac{1}{\sqrt{d}}\mathbf{P}^{\perp}\left(\frac{r}{\sqrt{\delta}}\mathbf{g}-\mathbf{s}\right)\right\|^2+\frac{r}{2\upsilon} \\
&+\frac{r\upsilon}{2}\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{\nu\alpha}{\sqrt{n}}\mathbf{q}-\frac{\sigma}{\sqrt{n}}\mathbf{h}\right\|^2+\frac{1}{d}(\mathbf{P}\mathbf{s})^{\top}\boldsymbol{\mu}+\frac{1}{d}(\mathbf{P}^{\perp}\mathbf{s})^{\top}\boldsymbol{\mu}-\frac{1}{d}(\mathbf{P}\mathbf{s})^{\top}\mathbf{w}. \tag*{(20)}
\end{aligned} μ , u , α , σ , υ ≥ 0 min s , r , τ ≥ 0 max n 1 1 ⊤ ϕ ( u ) − n 1 y ⊤ u + d λ f ( μ ) − 2 τ σ − 2 σ τ d 1 P ⊥ ( δ r g − s ) 2 + 2 υ r + 2 r υ n u − n ν α q − n σ h 2 + d 1 ( Ps ) ⊤ μ + d 1 ( P ⊥ s ) ⊤ μ − d 1 ( Ps ) ⊤ w . (20)
We now optimize over s \mathbf{s} s . Using a completion of squares,
− σ τ 2 ∥ 1 d P ⊥ ( r δ g − s ) ∥ 2 + 1 d ( P ⊥ s ) ⊤ μ = − σ τ 2 ∥ 1 d P ⊥ ( r δ g − s + 1 σ τ μ ) ∥ 2 + 1 2 d σ τ ∥ P ⊥ μ + σ τ r δ P ⊥ g ∥ 2 − σ τ r 2 2 n ∥ P ⊥ g ∥ 2 . \begin{aligned}
-\frac{\sigma\tau}{2}\left\|\frac{1}{\sqrt{d}}\mathbf{P}^{\perp}\left(\frac{r}{\sqrt{\delta}}\mathbf{g}-\mathbf{s}\right)\right\|^2+\frac{1}{d}(\mathbf{P}^{\perp}\mathbf{s})^{\top}\boldsymbol{\mu} \\
&=-\frac{\sigma\tau}{2}\left\|\frac{1}{\sqrt{d}}\mathbf{P}^{\perp}\left(\frac{r}{\sqrt{\delta}}\mathbf{g}-\mathbf{s}+\frac{1}{\sigma\tau}\boldsymbol{\mu}\right)\right\|^2+\frac{1}{2d\sigma\tau}\left\|\mathbf{P}^{\perp}\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{P}^{\perp}\mathbf{g}\right\|^2-\frac{\sigma\tau r^2}{2n}\|\mathbf{P}^{\perp}\mathbf{g}\|^2.
\end{aligned} − 2 σ τ d 1 P ⊥ ( δ r g − s ) 2 + d 1 ( P ⊥ s ) ⊤ μ = − 2 σ τ d 1 P ⊥ ( δ r g − s + σ τ 1 μ ) 2 + 2 d σ τ 1 P ⊥ μ + σ τ δ r P ⊥ g 2 − 2 n σ τ r 2 ∥ P ⊥ g ∥ 2 .
Since g ∼ N ( 0 , I d ) \mathbf{g}\sim\mathcal{N}(0,\mathbf{I}_d) g ∼ N ( 0 , I d ) , we have σ τ r 2 2 n ∥ P ⊥ g ∥ 2 ≲ σ τ r 2 2 δ \frac{\sigma\tau r^2}{2n}\|\mathbf{P}^{\perp}\mathbf{g}\|^2\lesssim\frac{\sigma\tau r^2}{2\delta} 2 n σ τ r 2 ∥ P ⊥ g ∥ 2 ≲ 2 δ σ τ r 2 , and,
1 d ∥ P ⊥ μ + σ τ r δ P ⊥ g ∥ 2 = 1 d ∥ P ⊥ ( μ + σ τ r δ g ) ∥ 2 = 1 d ∥ μ + σ τ r δ g ∥ 2 − 1 d ∥ P ( μ + σ τ r δ g ) ∥ 2 = 1 d ∥ μ + σ τ r δ g ∥ 2 − 1 d ∥ P μ ∥ 2 − σ 2 τ 2 r 2 δ d ∥ P g ∥ 2 − 2 σ τ r δ d ( P g ) ⊤ μ . \begin{aligned}
\frac{1}{d}\left\|\mathbf{P}^{\perp}\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{P}^{\perp}\mathbf{g}\right\|^2=\frac{1}{d}\left\|\mathbf{P}^{\perp}\left(\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{g}\right)\right\|^2=\frac{1}{d}\left\|\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{g}\right\|^2-\frac{1}{d}\left\|\mathbf{P}\left(\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{g}\right)\right\|^2 \\
&=\frac{1}{d}\left\|\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{g}\right\|^2-\frac{1}{d}\|\mathbf{P}\boldsymbol{\mu}\|^2-\frac{\sigma^2\tau^2r^2}{\delta d}\|\mathbf{P}\mathbf{g}\|^2-\frac{2\sigma\tau r}{\sqrt{\delta}d}(\mathbf{P}\mathbf{g})^{\top}\boldsymbol{\mu}.
\end{aligned} d 1 P ⊥ μ + σ τ δ r P ⊥ g 2 = d 1 P ⊥ ( μ + σ τ δ r g ) 2 = d 1 μ + σ τ δ r g 2 − d 1 P ( μ + σ τ δ r g ) 2 = d 1 μ + σ τ δ r g 2 − d 1 ∥ P μ ∥ 2 − δ d σ 2 τ 2 r 2 ∥ Pg ∥ 2 − δ d 2 σ τ r ( Pg ) ⊤ μ .
And as d → ∞ d \to\infty d → ∞ , 1 d ∥ P g ∥ 2 → 0 \frac{1}{d}\|\mathbf{P}\mathbf{g}\|^2 \to0 d 1 ∥ Pg ∥ 2 → 0 and 1 d ( P g ) ⊤ μ → 0 \frac{1}{d}(\mathbf{P}\mathbf{g})^{\top}\boldsymbol{\mu} \to0 d 1 ( Pg ) ⊤ μ → 0 almost surely. Since P μ \mathbf{P}\boldsymbol{\mu} P μ is constrained by 1 d w ⋆ ⊤ μ = α ν 2 \frac{1}{d}\mathbf{w}_{\star}^{\top}\boldsymbol{\mu}=\alpha\nu^2 d 1 w ⋆ ⊤ μ = α ν 2 , it follows that 1 d ∥ P μ ∥ 2 = ν 2 α 2 \frac{1}{d}\|\mathbf{P}\boldsymbol{\mu}\|^2=\nu^2\alpha^2 d 1 ∥ P μ ∥ 2 = ν 2 α 2 . Therefore (20) reduces asymptotically to
min μ , u , α , σ , ν ≥ 0 max r , τ ≥ 0 1 n 1 ⊤ ϕ ( u ) − 1 n y ⊤ u − σ 2 τ − σ τ r 2 2 δ + r 2 ν − ν 2 α 2 2 σ τ + r ν 2 ∥ u n − ν α n q − σ n h ∥ 2 + 1 2 d σ τ ∥ μ + σ τ r δ g ∥ 2 + λ d f ( μ ) s.t. 1 d w ⋆ ⊤ μ = α ν 2 . (21) \begin{aligned}
\min_{\boldsymbol{\mu},\mathbf{u},\alpha,\sigma,\nu\geq0}\max_{r,\tau\geq0}\;&\frac{1}{n}\mathbf{1}^{\top}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\top}\mathbf{u}-\frac{\sigma}{2\tau}-\frac{\sigma\tau r^2}{2\delta}+\frac{r}{2\nu}-\frac{\nu^2\alpha^2}{2\sigma\tau}+\frac{r\nu}{2}\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{\nu\alpha}{\sqrt{n}}\mathbf{q}-\frac{\sigma}{\sqrt{n}}\mathbf{h}\right\|^2\\
&+\frac{1}{2d\sigma\tau}\left\|\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{g}\right\|^2+\frac{\lambda}{d}f(\boldsymbol{\mu})\quad\text{s.t.}\quad\frac{1}{d}\mathbf{w}_{\star}^{\top}\boldsymbol{\mu}=\alpha\nu^2. \tag*{(21)}
\end{aligned} μ , u , α , σ , ν ≥ 0 min r , τ ≥ 0 max n 1 1 ⊤ ϕ ( u ) − n 1 y ⊤ u − 2 τ σ − 2 δ σ τ r 2 + 2 ν r − 2 σ τ ν 2 α 2 + 2 r ν n u − n ν α q − n σ h 2 + 2 d σ τ 1 μ + σ τ δ r g 2 + d λ f ( μ ) s.t. d 1 w ⋆ ⊤ μ = α ν 2 . (21)
We now optimize over μ \boldsymbol{\mu} μ and attain the following optimization problem,
min μ ∈ R d max θ ∈ R 1 2 d σ τ ∥ μ + σ τ r δ g ∥ 2 + λ d f ( μ ) s.t. 1 d w ⋆ ⊤ μ = α ν 2 . (22) \min_{\boldsymbol{\mu}\in\mathbb{R}^d}\max_{\theta\in\mathbb{R}}\frac{1}{2d\sigma\tau}\left\|\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{g}\right\|^2+\frac{\lambda}{d}f(\boldsymbol{\mu})\quad\text{s.t.}\quad\frac{1}{d}\mathbf{w}_{\star}^{\top}\boldsymbol{\mu}=\alpha\nu^2. \tag*{(22)} μ ∈ R d min θ ∈ R max 2 d σ τ 1 μ + σ τ δ r g 2 + d λ f ( μ ) s.t. d 1 w ⋆ ⊤ μ = α ν 2 . (22)
Introducing a Lagrange multiplier θ ∈ R \theta\in\mathbb{R} θ ∈ R for the linear constraint, we obtain
min μ ∈ R d max θ ∈ R 1 2 d σ τ ∥ μ + σ τ r δ g ∥ 2 + λ d f ( μ ) − θ d w ⋆ ⊤ μ + α θ ν 2 . (23) \min_{\boldsymbol{\mu}\in\mathbb{R}^d}\max_{\theta\in\mathbb{R}}\frac{1}{2d\sigma\tau}\left\|\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{g}\right\|^2+\frac{\lambda}{d}f(\boldsymbol{\mu})-\frac{\theta}{d}\mathbf{w}_{\star}^{\top}\boldsymbol{\mu}+\alpha\theta\nu^2. \tag*{(23)} μ ∈ R d min θ ∈ R max 2 d σ τ 1 μ + σ τ δ r g 2 + d λ f ( μ ) − d θ w ⋆ ⊤ μ + α θ ν 2 . (23)
Completing the square,
1 2 d σ τ ∥ μ + σ τ r δ g ∥ 2 − θ d w ⋆ ⊤ μ = 1 2 d σ τ ∥ μ + σ τ r δ g − σ τ θ w ⋆ ∥ 2 − σ τ θ 2 ν d 2 2 . \frac{1}{2d\sigma\tau}\left\|\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{g}\right\|^2-\frac{\theta}{d}\mathbf{w}_{\star}^{\top}\boldsymbol{\mu}=\frac{1}{2d\sigma\tau}\left\|\boldsymbol{\mu}+\sigma\tau\frac{r}{\sqrt{\delta}}\mathbf{g}-\sigma\tau\theta\mathbf{w}_{\star}\right\|^2-\frac{\sigma\tau\theta^2\nu_d^2}{2}. 2 d σ τ 1 μ + σ τ δ r g 2 − d θ w ⋆ ⊤ μ = 2 d σ τ 1 μ + σ τ δ r g − σ τ θ w ⋆ 2 − 2 σ τ θ 2 ν d 2 .
where we omitted the term 1 d g ⊤ w ⋆ \frac{1}{d}\mathbf{g}^{\top}\mathbf{w}_{\star} d 1 g ⊤ w ⋆ since 1 d g ⊤ w ⋆ → 0 \frac{1}{d}\mathbf{g}^{\top}\mathbf{w}_{\star}\to0 d 1 g ⊤ w ⋆ → 0 almost surely. Hence the μ \boldsymbol{\mu} μ -minimization is
max θ ∈ R { 1 d M λ f ( ⋅ , F t − 1 ) ( σ τ ( r δ g + θ w ⋆ ) , σ τ ) + α θ ν 2 − σ τ θ 2 ν d 2 2 } . \max_{\theta\in\mathbb{R}}\left\{\frac{1}{d}M_{\lambda f(\cdot,\mathcal{F}_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}\mathbf{g}+\theta\mathbf{w}_{\star}\right),\sigma\tau\right)+\alpha\theta\nu^2-\frac{\sigma\tau\theta^2\nu_d^2}{2}\right\}. θ ∈ R max { d 1 M λ f ( ⋅ , F t − 1 ) ( σ τ ( δ r g + θ w ⋆ ) , σ τ ) + α θ ν 2 − 2 σ τ θ 2 ν d 2 } .
Substituting this into (22) , we arrive at
min u , α , σ , ν ≥ 0 max r , τ ≥ 0 , θ ∈ R 1 n 1 ⊤ ϕ ( u ) − 1 n y ⊤ u + r ν 2 ∥ u n − ν α n q − σ n h ∥ 2 − σ 2 τ − σ τ r 2 2 δ + r 2 ν − ν 2 α 2 2 σ τ + ν 2 θ − ν 2 σ τ θ 2 2 + 1 d M λ f ( ⋅ , F t − 1 ) ( σ τ ( r δ g + θ w ⋆ ) , σ τ ) . (24) \begin{aligned}
\min_{\mathbf{u},\alpha,\sigma,\nu\geq0}\max_{r,\tau\geq0,\theta\in\mathbb{R}}\;&\frac{1}{n}\mathbf{1}^{\top}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\top}\mathbf{u}+\frac{r\nu}{2}\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{\nu\alpha}{\sqrt{n}}\mathbf{q}-\frac{\sigma}{\sqrt{n}}\mathbf{h}\right\|^2-\frac{\sigma}{2\tau}-\frac{\sigma\tau r^2}{2\delta}+\frac{r}{2\nu}\\
&-\frac{\nu^2\alpha^2}{2\sigma\tau}+\nu^2\theta-\frac{\nu^2\sigma\tau\theta^2}{2}+\frac{1}{d}M_{\lambda f(\cdot,\mathcal{F}_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}\mathbf{g}+\theta\mathbf{w}_{\star}\right),\sigma\tau\right). \tag*{(24)}
\end{aligned} u , α , σ , ν ≥ 0 min r , τ ≥ 0 , θ ∈ R max n 1 1 ⊤ ϕ ( u ) − n 1 y ⊤ u + 2 r ν n u − n ν α q − n σ h 2 − 2 τ σ − 2 δ σ τ r 2 + 2 ν r − 2 σ τ ν 2 α 2 + ν 2 θ − 2 ν 2 σ τ θ 2 + d 1 M λ f ( ⋅ , F t − 1 ) ( σ τ ( δ r g + θ w ⋆ ) , σ τ ) . (24)
We next optimize over u \mathbf{u} u . Consider the u \mathbf{u} u -dependent term
1 n 1 ⊤ ϕ ( u ) − 1 n y ⊤ u + r ν 2 ∥ u n − ν α n q − σ n h ∥ 2 . \frac{1}{n}\mathbf{1}^{\top}\phi(\mathbf{u})-\frac{1}{n}\mathbf{y}^{\top}\mathbf{u}+\frac{r\nu}{2}\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{\nu\alpha}{\sqrt{n}}\mathbf{q}-\frac{\sigma}{\sqrt{n}}\mathbf{h}\right\|^2. n 1 1 ⊤ ϕ ( u ) − n 1 y ⊤ u + 2 r ν n u − n ν α q − n σ h 2 .
Completing the square gives
− 1 n y ⊤ u + r ν 2 ∥ u n − ν α n q − σ n h ∥ 2 = r ν 2 ∥ u n − ν α n q − σ n h − 1 r ν n y ∥ 2 − 1 2 r ν ∥ y ∥ 2 n − ν α y ⊤ q n − σ y ⊤ h n . -\frac{1}{n}\mathbf{y}^{\top}\mathbf{u}+\frac{r\nu}{2}\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{\nu\alpha}{\sqrt{n}}\mathbf{q}-\frac{\sigma}{\sqrt{n}}\mathbf{h}\right\|^2=\frac{r\nu}{2}\left\|\frac{\mathbf{u}}{\sqrt{n}}-\frac{\nu\alpha}{\sqrt{n}}\mathbf{q}-\frac{\sigma}{\sqrt{n}}\mathbf{h}-\frac{1}{r\nu\sqrt{n}}\mathbf{y}\right\|^2-\frac{1}{2r\nu}\frac{\|\mathbf{y}\|^2}{n}-\nu\alpha\frac{\mathbf{y}^{\top}\mathbf{q}}{n}-\sigma\frac{\mathbf{y}^{\top}\mathbf{h}}{n}. − n 1 y ⊤ u + 2 r ν n u − n ν α q − n σ h 2 = 2 r ν n u − n ν α q − n σ h − r ν n 1 y 2 − 2 r ν 1 n ∥ y ∥ 2 − ν α n y ⊤ q − σ n y ⊤ h .
Using, y i ∣ q i ∼ Ber ( π ρ ( ν q i ) ) y_i\mid q_i\sim\operatorname{Ber}(\pi_{\rho}(\nu q_i)) y i ∣ q i ∼ Ber ( π ρ ( ν q i )) . Therefore
1 n ∥ y ∥ 2 = 1 n ∑ i = 1 n y i → a . s . E [ π ρ ( ν Z ) ] = 1 2 , \frac{1}{n}\|\mathbf{y}\|^2=\frac{1}{n}\sum_{i=1}^{n}y_i\xrightarrow{\mathrm{a.s.}}\mathbb{E}[\pi_{\rho}(\nu Z)]=\frac{1}{2}, n 1 ∥ y ∥ 2 = n 1 i = 1 ∑ n y i a.s. E [ π ρ ( ν Z )] = 2 1 ,
using π ρ ( − t ) = 1 − π ρ ( t ) \pi_{\rho}(-t)=1-\pi_{\rho}(t) π ρ ( − t ) = 1 − π ρ ( t ) , and
1 n y ⊤ q = 1 n ∑ i = 1 n y i q i → a . s . E [ Z π ρ ( ν Z ) ] = ν E [ π ρ ′ ( ν Z ) ] , \frac{1}{n}\mathbf{y}^{\top}\mathbf{q}=\frac{1}{n}\sum_{i=1}^{n}y_iq_i\xrightarrow{\mathrm{a.s.}}\mathbb{E}[Z\pi_{\rho}(\nu Z)]=\nu\mathbb{E}[\pi_{\rho}'(\nu Z)], n 1 y ⊤ q = n 1 i = 1 ∑ n y i q i a.s. E [ Z π ρ ( ν Z )] = ν E [ π ρ ′ ( ν Z )] ,
by Stein’s lemma. Also, 1 n y T h → P 0 \frac{1}{n}\mathbf{y}^{\mathsf{T}}\mathbf{h}\xrightarrow{\mathbb{P}}0 n 1 y T h P 0 . Thus the u \mathbf{u} u -optimization is asymptotically equivalent to
1 n M ϕ ( v α q + θ h + 1 r v y , 1 r v ) − 1 4 r v − v 2 E [ π ′ ( ρ ( v Z ) ) ] . \frac{1}{n}M_{\phi}\left(v\alpha\mathbf{q}+\theta\mathbf{h}+\frac{1}{rv}\mathbf{y},\frac{1}{rv}\right)-\frac{1}{4rv}-v^{2}\mathbb{E}\left[\pi^{\prime}(\rho(vZ))\right]. n 1 M ϕ ( v α q + θ h + r v 1 y , r v 1 ) − 4 r v 1 − v 2 E [ π ′ ( ρ ( v Z )) ] .
Hence (24) becomes
min α , σ , v ≥ 0 , r , τ ≥ 0 , θ ∈ R max − σ 2 τ − σ τ r 2 2 δ + r 2 v − v 2 α 2 2 σ τ + v 2 α θ − v 2 σ τ θ 2 2 − 1 4 r v − v 2 E [ π ′ ( ρ ( v Z ) ) ] + 1 d M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ g + θ w ⋆ ) , σ τ ) + 1 n M ϕ ( v α q + σ h + 1 r v y , 1 r v ) . (25) \begin{aligned}
\min_{\alpha,\sigma,v\geq0,\ r,\tau\geq0,\ \theta\in\mathbb{R}}\max\quad
&-\frac{\sigma}{2\tau}-\frac{\sigma\tau r^{2}}{2\delta}+\frac{r}{2v}-\frac{v^{2}\alpha^{2}}{2\sigma\tau}+v^{2}\alpha\theta-\frac{v^{2}\sigma\tau\theta^{2}}{2}-\frac{1}{4rv}-v^{2}\mathbb{E}\left[\pi^{\prime}(\rho(vZ))\right] \\
&+\frac{1}{d}M_{\lambda\tilde f_{(\cdot,\mathbf{F}_{t-1})}}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}\mathbf{g}+\theta\mathbf{w}_{\star}\right),\sigma\tau\right)
+\frac{1}{n}M_{\phi}\left(v\alpha\mathbf{q}+\sigma\mathbf{h}+\frac{1}{rv}\mathbf{y},\frac{1}{rv}\right). \tag*{(25)}
\end{aligned} α , σ , v ≥ 0 , r , τ ≥ 0 , θ ∈ R min max − 2 τ σ − 2 δ σ τ r 2 + 2 v r − 2 σ τ v 2 α 2 + v 2 α θ − 2 v 2 σ τ θ 2 − 4 r v 1 − v 2 E [ π ′ ( ρ ( v Z )) ] + d 1 M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r g + θ w ⋆ ) , σ τ ) + n 1 M ϕ ( v α q + σ h + r v 1 y , r v 1 ) . (25)
Since the regularizer is coordinatewise decomposable,
1 d M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ g + θ w ⋆ ) , σ τ ) = 1 d ∑ j = 1 d M λ f ~ ( ⋅ , F t − 1 , j ) ( σ τ ( r δ g j + θ w ⋆ , j ) , σ τ ) . \frac{1}{d}M_{\lambda\tilde f_{(\cdot,\mathbf{F}_{t-1})}}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}\mathbf{g}+\theta\mathbf{w}_{\star}\right),\sigma\tau\right)
=\frac{1}{d}\sum_{j=1}^{d}M_{\lambda\tilde f_{(\cdot,\mathbf{F}_{t-1,j})}}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}g_j+\theta w_{\star,j}\right),\sigma\tau\right). d 1 M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r g + θ w ⋆ ) , σ τ ) = d 1 j = 1 ∑ d M λ f ~ ( ⋅ , F t − 1 , j ) ( σ τ ( δ r g j + θ w ⋆ , j ) , σ τ ) .
Let Ξ : R 3 → R \Xi:\mathbb{R}^{3}\to\mathbb{R} Ξ : R 3 → R be locally Lipschitz and have at most quadratic growth, so that
∣ Ξ ( u , w , F ) ∣ ≤ C Ξ ( 1 + u 2 + w 2 + F 2 ) . \left|\Xi(u,w,F)\right|\leq C_{\Xi}(1+u^{2}+w^{2}+F^{2}). ∣ Ξ ( u , w , F ) ∣ ≤ C Ξ ( 1 + u 2 + w 2 + F 2 ) .
We first show the continuity and growth properties of the proximal operator that will be used later. Fix a compact parameter set ζ = ( σ , τ , θ , r ) ∈ K \zeta=(\sigma,\tau,\theta,r)\in K ζ = ( σ , τ , θ , r ) ∈ K on which σ τ \sigma\tau σ τ is bounded away from zero. By the Moreau-envelope identity,
∇ x M λ f ~ ( ⋅ , F ) ( x , c ) = 1 c ( x − Prox λ c f ~ ( ⋅ , F ) ( x ) ) . \nabla_x M_{\lambda\tilde f_{(\cdot,F)}}(x,c)=\frac{1}{c}\left(x-\operatorname{Prox}_{\lambda c\tilde f_{(\cdot,F)}}(x)\right). ∇ x M λ f ~ ( ⋅ , F ) ( x , c ) = c 1 ( x − Prox λ c f ~ ( ⋅ , F ) ( x ) ) .
For c c c bounded away from zero, these gradients are uniformly Lipschitz in x x x . Because,
∣ ( x − Prox λ c f ~ ( ⋅ , F ) ( x ) ) − ( y − Prox λ c f ~ ( ⋅ , F ) ( y ) ) ∣ 2 ≤ ∣ x − y ∣ 2 + ∣ Prox λ c f ~ ( ⋅ , F ) ( x ) − Prox λ c f ~ ( ⋅ , F ) ( y ) ∣ 2 − 2 ( Prox λ c f ~ ( ⋅ , F ) ( x ) − Prox λ c f ~ ( ⋅ , F ) ( y ) ) ( x − y ) ≤ ∣ x − y ∣ 2 . \begin{aligned}
\left|\left(x-\operatorname{Prox}_{\lambda c\tilde f_{(\cdot,F)}}(x)\right)-\left(y-\operatorname{Prox}_{\lambda c\tilde f_{(\cdot,F)}}(y)\right)\right|^{2} \\
&\leq|x-y|^{2}+\left|\operatorname{Prox}_{\lambda c\tilde f_{(\cdot,F)}}(x)-\operatorname{Prox}_{\lambda c\tilde f_{(\cdot,F)}}(y)\right|^{2}
-2\left(\operatorname{Prox}_{\lambda c\tilde f_{(\cdot,F)}}(x)-\operatorname{Prox}_{\lambda c\tilde f_{(\cdot,F)}}(y)\right)(x-y)
\leq|x-y|^{2}.
\end{aligned} ( x − Prox λ c f ~ ( ⋅ , F ) ( x ) ) − ( y − Prox λ c f ~ ( ⋅ , F ) ( y ) ) 2 ≤ ∣ x − y ∣ 2 + Prox λ c f ~ ( ⋅ , F ) ( x ) − Prox λ c f ~ ( ⋅ , F ) ( y ) 2 − 2 ( Prox λ c f ~ ( ⋅ , F ) ( x ) − Prox λ c f ~ ( ⋅ , F ) ( y ) ) ( x − y ) ≤ ∣ x − y ∣ 2 .
from the firm non-expansiveness of proximal operator. Hence
∣ ( x − Prox λ c f ~ ( ⋅ , F ) ( x ) ) − ( y − Prox λ c f ~ ( ⋅ , F ) ( y ) ) ∣ ≤ ∣ x − y ∣ , \left|\left(x-\operatorname{Prox}_{\lambda c\tilde f_{(\cdot,F)}}(x)\right)-\left(y-\operatorname{Prox}_{\lambda c\tilde f_{(\cdot,F)}}(y)\right)\right|\leq|x-y|, ( x − Prox λ c f ~ ( ⋅ , F ) ( x ) ) − ( y − Prox λ c f ~ ( ⋅ , F ) ( y ) ) ≤ ∣ x − y ∣ ,
∣ ∇ x M λ f ~ ( ⋅ , F ) ( x , c ) − ∇ x M λ f ~ ( ⋅ , F ) ( y , c ) ∣ ≤ 1 c ∣ x − y ∣ . \left|\nabla_xM_{\lambda\tilde f_{(\cdot,F)}}(x,c)-\nabla_xM_{\lambda\tilde f_{(\cdot,F)}}(y,c)\right|\leq\frac{1}{c}|x-y|. ∇ x M λ f ~ ( ⋅ , F ) ( x , c ) − ∇ x M λ f ~ ( ⋅ , F ) ( y , c ) ≤ c 1 ∣ x − y ∣.
On compact sets, using
∣ ∇ x M ( x , z ) − M ( x + h , z ) − M ( x , z ) h ∣ ≤ 1 h ∫ 0 h ∣ ∇ x M ( x , z ) − ∇ x M ( x + t , z ) ∣ d t ≤ h 2 c min . \left|\nabla_xM(x,z)-\frac{M(x+h,z)-M(x,z)}{h}\right|\leq\frac{1}{h}\int_{0}^{h}\left|\nabla_xM(x,z)-\nabla_xM(x+t,z)\right|\,\mathrm{d}t\leq\frac{h}{2c_{\min}}. ∇ x M ( x , z ) − h M ( x + h , z ) − M ( x , z ) ≤ h 1 ∫ 0 h ∣ ∇ x M ( x , z ) − ∇ x M ( x + t , z ) ∣ d t ≤ 2 c m i n h .
Let ( x n , z n ) → ( x , z ) (x_n,z_n)\to(x,z) ( x n , z n ) → ( x , z ) . For fixed h h h , joint continuity of M M M gives M ( x n + h , z n ) − M ( x n , z n ) h → M ( x + h , z ) − M ( x , z ) h \frac{M(x_n+h,z_n)-M(x_n,z_n)}{h}\to\frac{M(x+h,z)-M(x,z)}{h} h M ( x n + h , z n ) − M ( x n , z n ) → h M ( x + h , z ) − M ( x , z ) , together with the uniform Lipschitz of the Moreau gradient which handles x n → x x_n\to x x n → x imply joint continuity of its gradient. Also the proximal map is jointly continuity when restricted to compact set K K K by the Moreau envelope identity.
By the direct definition of the Moreau envelope M λ f ~ ( ⋅ , v ) ( x , τ ) = min y { f ~ ( y , v ) + 1 2 τ ( y − x ) 2 } M_{\lambda\tilde f_{(\cdot,v)} }(x,\tau)=\min_y\left\{\tilde f(y,v)+\frac{1}{2\tau}(y-x)^2\right\} M λ f ~ ( ⋅ , v ) ( x , τ ) = min y { f ~ ( y , v ) + 2 τ 1 ( y − x ) 2 } , non negativity of f ~ \tilde f f ~ also gives,
1 2 σ τ ∣ Prox λ σ τ f ~ ( ⋅ , F ) ( σ τ ( θ w + r δ z ) ) − σ τ ( θ w + r δ z ) ∣ 2 ≤ M λ f ~ ( ⋅ , F ) ( σ τ ( θ w + r δ z ) , σ τ ) . \frac{1}{2\sigma\tau}\left|\operatorname{Prox}_{\lambda\sigma\tau\tilde f_{(\cdot,F)}}\left(\sigma\tau\left(\theta w+\frac{r}{\sqrt{\delta}}z\right)\right)-\sigma\tau\left(\theta w+\frac{r}{\sqrt{\delta}}z\right)\right|^{2}
\leq M_{\lambda\tilde f_{(\cdot,F)}}\left(\sigma\tau\left(\theta w+\frac{r}{\sqrt{\delta}}z\right),\sigma\tau\right). 2 σ τ 1 Prox λσ τ f ~ ( ⋅ , F ) ( σ τ ( θ w + δ r z ) ) − σ τ ( θ w + δ r z ) 2 ≤ M λ f ~ ( ⋅ , F ) ( σ τ ( θ w + δ r z ) , σ τ ) .
The quadratic-growth bound on the Moreau envelope and compactness of K K K therefore imply,
sup ζ ∈ K ∣ Prox λ σ τ f ~ ( ⋅ , F ) ( σ τ ( θ w + r δ z ) ) ∣ ≤ C K ( 1 + ∣ w ∣ + ∣ F ∣ + ∣ z ∣ ) . \sup_{\zeta\in K}\left|\operatorname{Prox}_{\lambda\sigma\tau\tilde f_{(\cdot,F)}}\left(\sigma\tau\left(\theta w+\frac{r}{\sqrt{\delta}}z\right)\right)\right|\leq C_K(1+|w|+|F|+|z|). ζ ∈ K sup Prox λσ τ f ~ ( ⋅ , F ) ( σ τ ( θ w + δ r z ) ) ≤ C K ( 1 + ∣ w ∣ + ∣ F ∣ + ∣ z ∣ ) .
Consequently,
sup ζ ∈ K ∣ Ξ ( Prox λ σ τ f ~ ( ⋅ , F ) ( σ τ ( θ w + r δ z ) ) , w , F ) ∣ ≤ C K , Ξ ( 1 + w 2 + F 2 + z 2 ) . \sup_{\zeta\in K}\left|\Xi\left(\operatorname{Prox}_{\lambda\sigma\tau\tilde f_{(\cdot,F)}}\left(\sigma\tau\left(\theta w+\frac{r}{\sqrt{\delta}}z\right)\right),w,F\right)\right|\leq C_{K,\Xi}(1+w^{2}+F^{2}+z^{2}). ζ ∈ K sup Ξ ( Prox λσ τ f ~ ( ⋅ , F ) ( σ τ ( θ w + δ r z ) ) , w , F ) ≤ C K , Ξ ( 1 + w 2 + F 2 + z 2 ) .
We now freeze the scalar parameters at ζ ˉ t = ( σ ˉ t , τ ˉ t , r ˉ t , θ ˉ t ) \bar{\zeta}_t=(\bar{\sigma}_t,\bar{\tau}_t,\bar{r}_t,\bar{\theta}_t) ζ ˉ t = ( σ ˉ t , τ ˉ t , r ˉ t , θ ˉ t ) . Conditionally on the teacher and the first t − 1 t-1 t − 1 tasks, the pairs ( w ⋆ , j , F t − 1 , j ) (w_{\star,j},F_{t-1,j}) ( w ⋆ , j , F t − 1 , j ) are fixed, while Z 1 , … , Z d Z_1,\ldots,Z_d Z 1 , … , Z d are independent standard Gaussians. Therefore, the random variables Ξ ( y , w ⋆ , j , F t − 1 , j ) − E Z [ Ξ ( y , w ⋆ , j , F t − 1 , j ) ] \Xi(y,w_{\star,j},F_{t-1,j})-\mathbb{E}_Z[\Xi(y,w_{\star,j},F_{t-1,j})] Ξ ( y , w ⋆ , j , F t − 1 , j ) − E Z [ Ξ ( y , w ⋆ , j , F t − 1 , j )] with y = Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + r ˉ t δ Z j ) ) y=\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde{f}(\cdot,F_{t-1,j})}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_tw_{\star,j}+\frac{\bar{r}_t}{\sqrt{\delta}}Z_j\right)\right) y = Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + δ r ˉ t Z j ) ) are conditionally independent and centered.
We split the domain to two regimes, when ( w , F , Z ) (w,F,Z) ( w , F , Z ) norm is bounded by some R R R and when its norm is larger than R R R . Let χ R : R 3 → [ 0 , 1 ] \chi_R:\mathbb{R}^3\to[0,1] χ R : R 3 → [ 0 , 1 ] be the indicator function that equals 1 when w 2 + F 2 + z 2 ≤ R 2 w^2+F^2+z^2\le R^2 w 2 + F 2 + z 2 ≤ R 2 and 0 when w 2 + F 2 + z 2 > R 2 w^2+F^2+z^2>R^2 w 2 + F 2 + z 2 > R 2 . Conditionally on the teacher and the first t − 1 t-1 t − 1 tasks, the centered summands
χ R ( w ⋆ , j , F t − 1 , j , Z j ) Ξ ( y , w ⋆ , j , F t − 1 , j ) − E Z [ χ R ( w ⋆ , j , F t − 1 , j , Z ) Ξ ( y , w ⋆ , j , F t − 1 , j ) ] \chi_R(w_{\star,j},F_{t-1,j},Z_j)\Xi(y,w_{\star,j},F_{t-1,j})-\mathbb{E}_Z\left[\chi_R(w_{\star,j},F_{t-1,j},Z)\Xi(y,w_{\star,j},F_{t-1,j})\right] χ R ( w ⋆ , j , F t − 1 , j , Z j ) Ξ ( y , w ⋆ , j , F t − 1 , j ) − E Z [ χ R ( w ⋆ , j , F t − 1 , j , Z ) Ξ ( y , w ⋆ , j , F t − 1 , j ) ]
are independent and have conditional mean zero. The map
( w , F , z ) ⟼ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + r ˉ t δ z ) ) , w , F ) (w,F,z)\longmapsto\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde{f}(\cdot,F)}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_tw+\frac{\bar{r}_t}{\sqrt{\delta}}z\right)\right),w,F\right) ( w , F , z ) ⟼ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + δ r ˉ t z ) ) , w , F )
is continuous, as it is implied by the local lipschitzness. It is therefore bounded on the compact set { ( w , F , z ) : w 2 + F 2 + z 2 ≤ R 2 } \{(w,F,z):w^2+F^2+z^2\le R^2\} {( w , F , z ) : w 2 + F 2 + z 2 ≤ R 2 } . In particular, there is a constant B R < ∞ B_R<\infty B R < ∞ such that
∣ χ R ( w , F , z ) Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + r ˉ t δ z ) ) , w , F ) ∣ ≤ B R . \left|\chi_R(w,F,z)\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde{f}(\cdot,F)}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_tw+\frac{\bar{r}_t}{\sqrt{\delta}}z\right)\right),w,F\right)\right|\le B_R. χ R ( w , F , z ) Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + δ r ˉ t z ) ) , w , F ) ≤ B R .
Consequently, conditionally on ( w ⋆ , F t − 1 ) (w_{\star},\mathbf{F}_{t-1}) ( w ⋆ , F t − 1 ) , the variance of each term is at most B R 2 B_R^2 B R 2 , since Var ( X ) ≤ E [ X 2 ] \operatorname{Var}(X)\le\mathbb{E}[X^2] Var ( X ) ≤ E [ X 2 ] and its absolute value is at most B R B_R B R . Subtracting its conditional expectation does not change its variance.
The centered terms are conditionally independent because each depends only on the corresponding independent Gaussian variable Z j Z_j Z j . Therefore, the variance of their average is the sum of their variances divided by d 2 d^2 d 2 , and hence is bounded by 1 d 2 ∑ j = 1 d B R 2 = B R 2 d ⟶ 0 \frac{1}{d^2}\sum_{j=1}^d B_R^2=\frac{B_R^2}{d}\longrightarrow0 d 2 1 ∑ j = 1 d B R 2 = d B R 2 ⟶ 0 . Conditional Chebyshev’s inequality then implies that the empirical average of the centered terms converges to zero in probability.
Outside this region, when w 2 + F 2 + z 2 > R 2 w^2+F^2+z^2>R^2 w 2 + F 2 + z 2 > R 2 we use the bound
∣ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + r ˉ t δ z ) ) , w , F ) ∣ ≤ C ( 1 + w 2 + F 2 + z 2 ) . \left|\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde{f}(\cdot,F)}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_tw+\frac{\bar{r}_t}{\sqrt{\delta}}z\right)\right),w,F\right)\right|\le C(1+w^2+F^2+z^2). Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + δ r ˉ t z ) ) , w , F ) ≤ C ( 1 + w 2 + F 2 + z 2 ) .
Indeed, E [ ( 1 + Z 2 ) 1 { ∣ Z ∣ > R } ] = 2 R φ ( R ) + 4 Φ ( − R ) ⟶ 0 \mathbb{E}[(1+Z^2)\mathbf{1}_{\{|Z|>R\}}]=2R\varphi(R)+4\Phi(-R)\longrightarrow0 E [( 1 + Z 2 ) 1 { ∣ Z ∣ > R } ] = 2 R φ ( R ) + 4Φ ( − R ) ⟶ 0 , and the assumed 2-Wasserstein distance convergence of the empirical state distribution implies, for every ϵ > 0 \epsilon>0 ϵ > 0 ,
lim R → ∞ lim sup d → ∞ P ( 1 d ∑ j = 1 d ( 1 + w ⋆ , j 2 + F t − 1 , j 2 ) 1 { w ⋆ , j 2 + F t − 1 , j 2 > R 2 } > ϵ ) = 0. (26) \lim_{R\to\infty}\limsup_{d\to\infty}\mathbb{P}\left(\frac{1}{d}\sum_{j=1}^d(1+w_{\star,j}^2+F_{t-1,j}^2)\mathbf{1}_{\{w_{\star,j}^2+F_{t-1,j}^2>R^2\}}>\epsilon\right)=0. \tag*{(26)} R → ∞ lim d → ∞ lim sup P ( d 1 j = 1 ∑ d ( 1 + w ⋆ , j 2 + F t − 1 , j 2 ) 1 { w ⋆ , j 2 + F t − 1 , j 2 > R 2 } > ϵ ) = 0. (26)
Since it follows from w 2 + F 2 + Z 2 > R 2 w^2+F^2+Z^2>R^2 w 2 + F 2 + Z 2 > R 2 that w 2 + F 2 > R 2 2 w^2+F^2>\frac{R^2}{2} w 2 + F 2 > 2 R 2 or Z 2 > R 2 2 Z^2>\frac{R^2}{2} Z 2 > 2 R 2 ,
( 1 + w 2 + F 2 + Z 2 ) 1 { w 2 + F 2 + Z 2 > R 2 } ≤ ( 1 + w 2 + F 2 + Z 2 ) 1 { w 2 + F 2 > R 2 2 } + ( 1 + w 2 + F 2 + Z 2 ) 1 { Z 2 > R 2 2 } , (1+w^2+F^2+Z^2)\mathbf{1}_{\{w^2+F^2+Z^2>R^2\}}\le(1+w^2+F^2+Z^2)\mathbf{1}_{\{w^2+F^2>\frac{R^2}{2}\}}+(1+w^2+F^2+Z^2)\mathbf{1}_{\{Z^2>\frac{R^2}{2}\}}, ( 1 + w 2 + F 2 + Z 2 ) 1 { w 2 + F 2 + Z 2 > R 2 } ≤ ( 1 + w 2 + F 2 + Z 2 ) 1 { w 2 + F 2 > 2 R 2 } + ( 1 + w 2 + F 2 + Z 2 ) 1 { Z 2 > 2 R 2 } ,
The first term is controlled by the 2-Wasserstein tail bound, and the second by the Gaussian tail together with uniform boundedness of the empirical second moments and P ( Z 2 > R 2 2 ) → 0 \mathbb{P}(Z^2>\frac{R^2}{2})\to0 P ( Z 2 > 2 R 2 ) → 0 . It follows that,
1 d ∑ j = 1 d { Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + r ˉ t δ Z j ) ) , w ⋆ , j , F t − 1 , j ) − \frac{1}{d}\sum_{j=1}^d\left\{\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde{f}(\cdot,F_{t-1,j})}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_tw_{\star,j}+\frac{\bar{r}_t}{\sqrt{\delta}}Z_j\right)\right),w_{\star,j},F_{t-1,j}\right)-\right. d 1 j = 1 ∑ d { Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + δ r ˉ t Z j ) ) , w ⋆ , j , F t − 1 , j ) −
E Z [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + r ˉ t δ Z ) ) , w ⋆ , j , F t − 1 , j ) ] } → P 0. (27) \left.\mathbb{E}_Z\left[\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde{f}(\cdot,F_{t-1,j})}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_tw_{\star,j}+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),w_{\star,j},F_{t-1,j}\right)\right]\right\}\xrightarrow{\mathbb{P}}0. \tag*{(27)} E Z [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + δ r ˉ t Z ) ) , w ⋆ , j , F t − 1 , j ) ] } P 0. (27)
Next, for each fixed z z z the map ( w , F ) ⟼ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + r ˉ t δ z ) ) , w , F ) (w,F)\longmapsto\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde{f}(\cdot,F)}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_tw+\frac{\bar{r}_t}{\sqrt{\delta}}z\right)\right),w,F\right) ( w , F ) ⟼ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + δ r ˉ t z ) ) , w , F ) is jointly continuous by the joint continuity of the proximal map and continuity of Ξ \Xi Ξ , this also shows that,
( w , F ) ⟼ E Z [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + r ˉ t δ Z ) ) , w , F ) ] (w,F)\longmapsto\mathbb{E}_Z\left[\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde{f}(\cdot,F)}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_tw+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),w,F\right)\right] ( w , F ) ⟼ E Z [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + δ r ˉ t Z ) ) , w , F ) ]
is continuous. Indeed, for every series ( w n , F n ) → ( w , F ) (w_n,F_n) \to(w,F) ( w n , F n ) → ( w , F ) we have (as ∀ n : ∥ ( w n , F n ) ∥ < R \forall n : \lVert(w_n,F_n)\rVert< R ∀ n : ∥( w n , F n )∥ < R because it converges),
∣ Ξ ( Prox λ σ t τ ˉ t f ~ ( ⋅ , F n ) ( σ ˉ t τ ˉ t ( θ ˉ t w n + r ˉ t δ z ) ) , w n , F n ) ∣ ≤ C R ( 1 + z 2 ) . \left|\Xi\left(\operatorname{Prox}_{\lambda\sigma_t\bar{\tau}_t\tilde f(\cdot,F_n)}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t w_n+\frac{\bar{r}_t}{\sqrt{\delta}}z\right)\right),w_n,F_n\right)\right| \le C_R(1+z^2). Ξ ( Prox λ σ t τ ˉ t f ~ ( ⋅ , F n ) ( σ ˉ t τ ˉ t ( θ ˉ t w n + δ r ˉ t z ) ) , w n , F n ) ≤ C R ( 1 + z 2 ) .
Thus C R ( 1 + z 2 ) C_R(1+z^2) C R ( 1 + z 2 ) is integrable with z ∼ N ( 0 , 1 ) z\sim\mathcal{N}(0,1) z ∼ N ( 0 , 1 ) , and dominated convergence yields convergence of the expectations. The same bound gives
∣ E Z [ Ξ ( Prox λ σ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + r ˉ t δ Z ) ) , w , F ) ] ∣ ≤ E Z [ ∣ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + r ˉ t δ Z ) ) , w , F ) ∣ ] ≤ C E Z [ 1 + w 2 + F 2 + Z 2 ] = C ( 2 + w 2 + F 2 ) ≤ C ′ ( 1 + w 2 + F 2 ) . (28) \begin{aligned}
\left|\mathbb{E}_Z\left[\Xi\left(\operatorname{Prox}_{\lambda\sigma_t\bar{\tau}_t\tilde f(\cdot,F)}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t w+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),w,F\right)\right]\right| \\
&\le\mathbb{E}_Z\left[\left|\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde f(\cdot,F)}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t w+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),w,F\right)\right|\right] \\
&\le C\mathbb{E}_Z\left[1+w^2+F^2+Z^2\right]=C\left(2+w^2+F^2\right)\le C'(1+w^2+F^2). \tag*{(28)}
\end{aligned} E Z [ Ξ ( Prox λ σ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + δ r ˉ t Z ) ) , w , F ) ] ≤ E Z [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F ) ( σ ˉ t τ ˉ t ( θ ˉ t w + δ r ˉ t Z ) ) , w , F ) ] ≤ C E Z [ 1 + w 2 + F 2 + Z 2 ] = C ( 2 + w 2 + F 2 ) ≤ C ′ ( 1 + w 2 + F 2 ) . (28)
The weak convergence on a bounded set together with the second moment tail bound (by the assumed convergence in 2-Wasserstein distance) on the complement set, and Eqs. (26) and (28) , therefore give,
1 d ∑ j = 1 d E Z [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + r ˉ t δ Z ) ) , w ⋆ , j , F t − 1 , j ) ] \frac{1}{d}\sum_{j=1}^{d}\mathbb{E}_Z\left[\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde f(\cdot,F_{t-1,j})}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t w_{\star,j}+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),w_{\star,j},F_{t-1,j}\right)\right] d 1 j = 1 ∑ d E Z [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + δ r ˉ t Z ) ) , w ⋆ , j , F t − 1 , j ) ]
→ P E [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + r ˉ t δ Z ) ) , w ⋆ , F t − 1 ) ] , (29) \xrightarrow{\mathbb{P}}\mathbb{E}\left[\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde f(\cdot,F_{t-1})}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t w^\star+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),w^\star,F_{t-1}\right)\right], \tag*{(29)} P E [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + δ r ˉ t Z ) ) , w ⋆ , F t − 1 ) ] , (29)
where Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) is independent of ( w ⋆ , F t − 1 ) (w^\star,F_{t-1}) ( w ⋆ , F t − 1 ) .
It remains to replace the limiting scalar parameters by the random AO scalar parameters. By the AO proximal representation,
w ^ j A O = Prox λ σ t , d τ t , d f ~ ( ⋅ , F t − 1 , j ) ( σ t , d τ t , d ( θ t , d w ⋆ , j + r t , d δ Z j ) ) , \widehat{w}^{\mathrm{AO}}_j=\operatorname{Prox}_{\lambda\sigma_{t,d}\tau_{t,d}\tilde f(\cdot,F_{t-1,j})}\left(\sigma_{t,d}\tau_{t,d}\left(\theta_{t,d}w_{\star,j}+\frac{r_{t,d}}{\sqrt{\delta}}Z_j\right)\right), w j AO = Prox λ σ t , d τ t , d f ~ ( ⋅ , F t − 1 , j ) ( σ t , d τ t , d ( θ t , d w ⋆ , j + δ r t , d Z j ) ) ,
and, by scalar-optimizer convergence, ζ t , d : = ( σ t , d , τ t , d , r t , d , θ t , d ) → P ζ ˉ t \zeta_{t,d}:=(\sigma_{t,d},\tau_{t,d},r_{t,d},\theta_{t,d})\xrightarrow{\mathbb{P}}\bar{\zeta}_t ζ t , d := ( σ t , d , τ t , d , r t , d , θ t , d ) P ζ ˉ t . Since σ ˉ t τ ˉ t > 0 \bar{\sigma}_t\bar{\tau}_t>0 σ ˉ t τ ˉ t > 0 , choose a compact neighborhood K K K of ζ ˉ t \bar{\zeta}_t ζ ˉ t on which σ t \sigma_t σ t is bounded away from zero. Then P ( ζ t , d ∉ K ) → 0 \mathbb{P}(\zeta_{t,d}\notin K)\to0 P ( ζ t , d ∈ / K ) → 0 , for all large d d d . On the event { ζ t , d ∈ K } \{\zeta_{t,d}\in K\} { ζ t , d ∈ K } , truncate ( w ⋆ , j , F t − 1 , j , Z j ) (w_{\star,j},F_{t-1,j},Z_j) ( w ⋆ , j , F t − 1 , j , Z j ) to a fixed compact set. Uniform continuity of the composed proximal map on this compact set and ζ t , d → P ζ ˉ t \zeta_{t,d}\xrightarrow{\mathbb{P}}\bar{\zeta}_t ζ t , d P ζ ˉ t imply that the empirical average of the differences converges to zero. The contribution outside the compact set is uniformly controlled by C K ( 1 + w 2 + F 2 + z 2 ) C_K(1+w^2+F^2+z^2) C K ( 1 + w 2 + F 2 + z 2 ) and is therefore negligible by the same 2-Wasserstein distance and Gaussian tail estimates. Hence,
1 d ∑ j = 1 d [ Ξ ( w ^ j A O , w ⋆ , j , F t − 1 , j ) − Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + r ˉ t δ Z j ) ) , w ⋆ , j , F t − 1 , j ) ] → P 0. \frac{1}{d}\sum_{j=1}^{d}\left[\Xi\left(\widehat{w}^{\mathrm{AO}}_j,w_{\star,j},F_{t-1,j}\right)-\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde f(\cdot,F_{t-1,j})}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t w_{\star,j}+\frac{\bar{r}_t}{\sqrt{\delta}}Z_j\right)\right),w_{\star,j},F_{t-1,j}\right)\right]\xrightarrow{\mathbb{P}}0. d 1 j = 1 ∑ d [ Ξ ( w j AO , w ⋆ , j , F t − 1 , j ) − Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 , j ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ , j + δ r ˉ t Z j ) ) , w ⋆ , j , F t − 1 , j ) ] P 0.
Combining the parameter-replacement step, the conditional law of large numbers, and the empirical 2-Wasserstein distance convergence gives
1 d ∑ j = 1 d Ξ ( w ^ j A O , w ⋆ , j , F t − 1 , j ) → P E [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + r ˉ t δ Z ) ) , w ⋆ , F t − 1 ) ] , (30) \frac{1}{d}\sum_{j=1}^{d}\Xi\left(\widehat{w}^{\mathrm{AO}}_j,w_{\star,j},F_{t-1,j}\right)\xrightarrow{\mathbb{P}}\mathbb{E}\left[\Xi\left(\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde f(\cdot,F_{t-1})}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t w^\star+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),w^\star,F_{t-1}\right)\right], \tag*{(30)} d 1 j = 1 ∑ d Ξ ( w j AO , w ⋆ , j , F t − 1 , j ) P E [ Ξ ( Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + δ r ˉ t Z ) ) , w ⋆ , F t − 1 ) ] , (30)
where Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) is independent of ( w ⋆ , F t − 1 ) (w^\star,F_{t-1}) ( w ⋆ , F t − 1 ) . Likewise, setting
γ = 1 r v , B ∣ Z 1 ∼ Ber ( ρ p ( ν Z 1 ) ) , \gamma=\frac{1}{r v},\qquad B\mid Z_1\sim\operatorname{Ber}(\rho_p(\nu Z_1)), γ = r v 1 , B ∣ Z 1 ∼ Ber ( ρ p ( ν Z 1 )) ,
we have
1 n M ϕ ( ν α + σ h + γ y , γ ) → a . s . E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] . \frac{1}{n}M_{\phi}(\nu\alpha+\sigma h+\gamma y,\gamma)\xrightarrow{\mathrm{a.s.}}\mathbb{E}\left[M_{\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma B,\gamma)\right]. n 1 M ϕ ( ν α + σ h + γ y , γ ) a.s. E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] .
Therefore, by the preceding convergence of the two empirical Moreau envelopes and (Thrampoulidis et al. 2018, Appendix A, Lemma 9), the finite-dimensional scalar optimization converges to its deterministic limiting counterpart, which leaves us with,
min α ∈ R , σ , ν ≥ 0 max r , τ ≥ 0 , θ ∈ R C ( α , σ , r , τ , ν , θ ) (31) \min_{\alpha\in\mathbb{R},\sigma,\nu\ge0}\max_{r,\tau\ge0,\theta\in\mathbb{R}}\mathcal{C}(\alpha,\sigma,r,\tau,\nu,\theta) \tag*{(31)} α ∈ R , σ , ν ≥ 0 min r , τ ≥ 0 , θ ∈ R max C ( α , σ , r , τ , ν , θ ) (31)
where,
C ( α , σ , r , τ , ν , θ ) = − σ 2 τ − σ τ r 2 2 δ + r 2 ν − ν 2 α 2 2 σ τ + ν 2 α θ − ν 2 σ τ θ 2 2 − 1 4 r ν − ν 2 α E [ π ρ ′ ( ν Z ) ] + E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ ω ∗ ) , σ τ ) ] + E [ M ϕ ( ν α Z 1 + σ Z 2 + 1 r ν B , 1 r ν ) ] (32) C(\alpha,\sigma,r,\tau,\nu,\theta)=-\frac{\sigma}{2\tau}-\frac{\sigma\tau r^2}{2\delta}+\frac{r}{2\nu}-\frac{\nu^2\alpha^2}{2\sigma\tau}+\nu^2\alpha\theta-\frac{\nu^2\sigma\tau\theta^2}{2}-\frac{1}{4r\nu}-\nu^2\alpha\mathbb{E}[\pi'_{\rho}(\nu Z)]
+\mathbb{E}\left[M_{\lambda\tilde{f}(\cdot,\mathbf{F}_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta\omega^*\right),\sigma\tau\right)\right]
+\mathbb{E}\left[M_{\phi}\left(\nu\alpha Z_1+\sigma Z_2+\frac{1}{r\nu}B,\frac{1}{r\nu}\right)\right] \tag*{(32)} C ( α , σ , r , τ , ν , θ ) = − 2 τ σ − 2 δ σ τ r 2 + 2 ν r − 2 σ τ ν 2 α 2 + ν 2 α θ − 2 ν 2 σ τ θ 2 − 4 r ν 1 − ν 2 α E [ π ρ ′ ( ν Z )] + E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ ω ∗ ) , σ τ ) ] + E [ M ϕ ( ν α Z 1 + σ Z 2 + r ν 1 B , r ν 1 ) ] (32)
and Z , Z 1 , Z 2 ∼ i i d N ( 0 , 1 ) Z,Z_1,Z_2\mathrel{\mathop{\sim}^{\mathrm{iid}}}N(0,1) Z , Z 1 , Z 2 ∼ iid N ( 0 , 1 ) , and B ∣ Z 1 ∼ Ber ( π ρ ( ν Z 1 ) ) B\mid Z_1\sim\operatorname{Ber}(\pi_{\rho}(\nu Z_1)) B ∣ Z 1 ∼ Ber ( π ρ ( ν Z 1 )) .
We now turn to the first-order optimality conditions of Eq. (31) . Let γ = 1 r ν \gamma=\frac{1}{r\nu} γ = r ν 1 , and write the scalar objective as
C ( α , σ , r , τ , ν , θ ) = − σ 2 τ − σ τ r 2 2 δ + r 2 ν − ν 2 α 2 2 σ τ + ν 2 α θ − ν 2 σ τ θ 2 2 − 1 4 r ν − ν 2 α E [ π ρ ′ ( ν Z ) ] + E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ ω ∗ ) , σ τ ) ] + E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] . C(\alpha,\sigma,r,\tau,\nu,\theta)=-\frac{\sigma}{2\tau}-\frac{\sigma\tau r^2}{2\delta}+\frac{r}{2\nu}-\frac{\nu^2\alpha^2}{2\sigma\tau}+\nu^2\alpha\theta-\frac{\nu^2\sigma\tau\theta^2}{2}-\frac{1}{4r\nu}-\nu^2\alpha\mathbb{E}[\pi'_{\rho}(\nu Z)]
+\mathbb{E}\left[M_{\lambda\tilde{f}(\cdot,\mathbf{F}_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta\omega^*\right),\sigma\tau\right)\right]
+\mathbb{E}\left[M_{\phi}\left(\nu\alpha Z_1+\sigma Z_2+\gamma B,\gamma\right)\right]. C ( α , σ , r , τ , ν , θ ) = − 2 τ σ − 2 δ σ τ r 2 + 2 ν r − 2 σ τ ν 2 α 2 + ν 2 α θ − 2 ν 2 σ τ θ 2 − 4 r ν 1 − ν 2 α E [ π ρ ′ ( ν Z )] + E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ ω ∗ ) , σ τ ) ] + E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] .
Since the objective is smooth on { σ > 0 , r > 0 , τ > 0 , ν > 0 } \{\sigma>0,r>0,\tau>0,\nu>0\} { σ > 0 , r > 0 , τ > 0 , ν > 0 } , the unique saddle point ( α ˉ , σ ˉ , r ˉ , τ ˉ , ν ˉ , θ ˉ ) (\bar{\alpha},\bar{\sigma},\bar{r},\bar{\tau},\bar{\nu},\bar{\theta}) ( α ˉ , σ ˉ , r ˉ , τ ˉ , ν ˉ , θ ˉ ) satisfies the first-order optimality conditions. We derive them one by one. First we differentiate the objective with respect to θ \theta θ , only the terms
ν 2 α θ − ν 2 σ τ θ 2 2 + E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ ω ∗ ) , σ τ ) ] \nu^2\alpha\theta-\frac{\nu^2\sigma\tau\theta^2}{2}+\mathbb{E}\left[M_{\lambda\tilde{f}(\cdot,\mathbf{F}_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta\omega^*\right),\sigma\tau\right)\right] ν 2 α θ − 2 ν 2 σ τ θ 2 + E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ ω ∗ ) , σ τ ) ]
depend on θ \theta θ . Using the derivative formula for the Moreau envelope Salehi et al. (2019, Lemma 2),
∂ ∂ x M λ f ( ⋅ , F t − 1 ) ( x , t ) = 1 T ( x − Prox t λ f ( ⋅ , F t − 1 ) ( x ) ) , \frac{\partial}{\partial x}M_{\lambda f(\cdot,\mathbf{F}_{t-1})}(x,t)=\frac{1}{T}\left(x-\operatorname{Prox}_{t\lambda f(\cdot,\mathbf{F}_{t-1})}(x)\right), ∂ x ∂ M λ f ( ⋅ , F t − 1 ) ( x , t ) = T 1 ( x − Prox t λ f ( ⋅ , F t − 1 ) ( x ) ) ,
we obtain
∂ C ∂ θ = ν 2 α − ν 2 σ τ θ + E [ 1 σ τ ( σ τ ( r δ Z + θ ω ∗ ) − Prox λ σ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ ω ∗ ) ) ) σ τ ω ∗ ] = ν 2 α − ν 2 σ τ θ + σ τ E [ ( ω ∗ ) 2 ] + r σ τ δ E [ Z ω ∗ ] − E [ ω ∗ Prox λ σ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ ω ∗ + r δ Z ) ) ] . \begin{aligned}
\frac{\partial C}{\partial\theta}=\nu^2\alpha-\nu^2\sigma\tau\theta+\mathbb{E}\left[\frac{1}{\sigma\tau}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta\omega^*\right)-\operatorname{Prox}_{\lambda\sigma\tau\tilde{f}(\cdot,\mathbf{F}_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta\omega^*\right)\right)\right)\sigma\tau\omega^*\right] \\
&=\nu^2\alpha-\nu^2\sigma\tau\theta+\sigma\tau\mathbb{E}[(\omega^*)^2]+\frac{r\sigma\tau}{\sqrt{\delta}}\mathbb{E}[Z\omega^*]-\mathbb{E}\left[\omega^*\operatorname{Prox}_{\lambda\sigma\tau\tilde{f}(\cdot,\mathbf{F}_{t-1})}\left(\sigma\tau\left(\theta\omega^*+\frac{r}{\sqrt{\delta}}Z\right)\right)\right].
\end{aligned} ∂ θ ∂ C = ν 2 α − ν 2 σ τ θ + E [ σ τ 1 ( σ τ ( δ r Z + θ ω ∗ ) − Prox λσ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ ω ∗ ) ) ) σ τ ω ∗ ] = ν 2 α − ν 2 σ τ θ + σ τ E [( ω ∗ ) 2 ] + δ r σ τ E [ Z ω ∗ ] − E [ ω ∗ Prox λσ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ ω ∗ + δ r Z ) ) ] .
Since E [ ( ω ∗ ) 2 ] = ν 2 \mathbb{E}[(\omega^*)^2]=\nu^2 E [( ω ∗ ) 2 ] = ν 2 and E [ Z ω ∗ ] = 0 \mathbb{E}[Z\omega^*]=0 E [ Z ω ∗ ] = 0 , the θ \theta θ -stationarity condition ∂ θ C = 0 \partial_\theta C=0 ∂ θ C = 0 becomes
ν 2 α = E [ ω ∗ Prox λ σ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ ω ∗ + r δ Z ) ) ] . \nu^2\alpha=\mathbb{E}\left[\omega^*\operatorname{Prox}_{\lambda\sigma\tau\tilde{f}(\cdot,\mathbf{F}_{t-1})}\left(\sigma\tau\left(\theta\omega^*+\frac{r}{\sqrt{\delta}}Z\right)\right)\right]. ν 2 α = E [ ω ∗ Prox λσ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ ω ∗ + δ r Z ) ) ] .
For fixed ( α , σ ) (\alpha,\sigma) ( α , σ ) we define F ρ ( γ ) = − γ 4 + E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] F_{\rho}(\gamma)=-\frac{\gamma}{4}+\mathbb{E}\left[M_{\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma B,\gamma)\right] F ρ ( γ ) = − 4 γ + E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] with B ∣ Z 1 ∼ Ber ( π ρ ( ν Z 1 ) ) B\mid Z_1\sim\operatorname{Ber}(\pi_{\rho}(\nu Z_1)) B ∣ Z 1 ∼ Ber ( π ρ ( ν Z 1 )) . Then the ( r , ν ) (r,\nu) ( r , ν ) -dependence of C C C is
− σ τ r 2 2 δ + r 2 ν + E [ M ϕ ( ν α Z 1 + σ Z 2 + 1 r ν B , 1 r ν ) ] − 1 4 r ν = − σ τ r 2 2 δ + r 2 ν + F ρ ( 1 r ν ) . -\frac{\sigma\tau r^2}{2\delta}+\frac{r}{2\nu}+\mathbb{E}\left[M_{\phi}\left(\nu\alpha Z_1+\sigma Z_2+\frac{1}{r\nu}B,\frac{1}{r\nu}\right)\right]-\frac{1}{4r\nu}
=-\frac{\sigma\tau r^2}{2\delta}+\frac{r}{2\nu}+F_{\rho}\left(\frac{1}{r\nu}\right). − 2 δ σ τ r 2 + 2 ν r + E [ M ϕ ( ν α Z 1 + σ Z 2 + r ν 1 B , r ν 1 ) ] − 4 r ν 1 = − 2 δ σ τ r 2 + 2 ν r + F ρ ( r ν 1 ) .
We now compute F ρ ′ ( γ ) F'_{\rho}(\gamma) F ρ ′ ( γ ) . By differentiation under the expectation and the Moreau-envelope derivative formula,
F ρ ′ ( γ ) = − 1 4 + d d γ E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] = − 1 4 − E [ 1 − B 2 γ 2 ( ν α Z 1 + σ Z 2 − Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ) 2 ] \begin{aligned}
F'_{\rho}(\gamma)=-\frac{1}{4}+\frac{\mathrm{d}}{\mathrm{d}\gamma}\mathbb{E}\left[M_{\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma B,\gamma)\right] \\
&=-\frac{1}{4}-\mathbb{E}\left[\frac{1-B}{2\gamma^2}\left(\nu\alpha Z_1+\sigma Z_2-\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2)\right)^2\right]
\end{aligned} F ρ ′ ( γ ) = − 4 1 + d γ d E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] = − 4 1 − E [ 2 γ 2 1 − B ( ν α Z 1 + σ Z 2 − Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ) 2 ]
− E [ B 2 γ 2 ( ν α Z 1 + σ Z 2 + γ − Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ ) ) 2 ] + E [ B γ ( ν α Z 1 + σ Z 2 + γ − Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ ) ) ] . -\mathbb{E}\left[\frac{B}{2\gamma^2}\left(\nu\alpha Z_1+\sigma Z_2+\gamma-\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma)\right)^2\right]
+\mathbb{E}\left[\frac{B}{\gamma}\left(\nu\alpha Z_1+\sigma Z_2+\gamma-\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma)\right)\right]. − E [ 2 γ 2 B ( ν α Z 1 + σ Z 2 + γ − Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ ) ) 2 ] + E [ γ B ( ν α Z 1 + σ Z 2 + γ − Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ ) ) ] .
Using
P ( B = 1 ∣ Z 1 ) = π ρ ( ν Z 1 ) , P ( B = 0 ∣ Z 1 ) = 1 − π ρ ( ν Z 1 ) , \mathbb{P}(B=1\mid Z_1)=\pi_{\rho}(\nu Z_1),\qquad\mathbb{P}(B=0\mid Z_1)=1-\pi_{\rho}(\nu Z_1), P ( B = 1 ∣ Z 1 ) = π ρ ( ν Z 1 ) , P ( B = 0 ∣ Z 1 ) = 1 − π ρ ( ν Z 1 ) ,
the symmetry identity Prox γ ϕ ( x + γ ) = − Prox γ ϕ ( − x ) \operatorname{Prox}_{\gamma\phi}(x+\gamma)=-\operatorname{Prox}_{\gamma\phi}(-x) Prox γ ϕ ( x + γ ) = − Prox γ ϕ ( − x ) , and π ρ ( − t ) = 1 − π ρ ( t ) \pi_\rho(-t)=1-\pi_\rho(t) π ρ ( − t ) = 1 − π ρ ( t ) , this simplifies to
F ρ ′ ( γ ) = − 1 γ 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ( ν α Z 1 + σ Z 2 − Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ) 2 ] . (33) F'_\rho(\gamma)=-\frac{1}{\gamma^2}\mathbb{E}\left[\left(1-\pi_\rho(\nu Z_1)\right)\left(\nu\alpha Z_1+\sigma Z_2-\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2)\right)^2\right]. \tag*{(33)} F ρ ′ ( γ ) = − γ 2 1 E [ ( 1 − π ρ ( ν Z 1 ) ) ( ν α Z 1 + σ Z 2 − Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ) 2 ] . (33)
Differentiating C \mathcal{C} C with respect to r r r and ν \nu ν , gives
∂ C ∂ r = − σ τ r δ + 1 2 ν − 1 ν r 2 F ρ ′ ( 1 r ν ) + ∂ ∂ r E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ w ⋆ ) , σ τ ) ] , \frac{\partial\mathcal{C}}{\partial r}=-\frac{\sigma\tau r}{\delta}+\frac{1}{2\nu}-\frac{1}{\nu r^2}F'_\rho\left(\frac{1}{r\nu}\right)+\frac{\partial}{\partial r}\mathbb{E}\left[M_{\lambda\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta w^\star\right),\sigma\tau\right)\right], ∂ r ∂ C = − δ σ τ r + 2 ν 1 − ν r 2 1 F ρ ′ ( r ν 1 ) + ∂ r ∂ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ w ⋆ ) , σ τ ) ] ,
∂ C ∂ ν = − r 2 ν 2 − 1 r ν 2 F ρ ′ ( 1 r ν ) . \frac{\partial\mathcal{C}}{\partial\nu}=-\frac{r}{2\nu^2}-\frac{1}{r\nu^2}F'_\rho\left(\frac{1}{r\nu}\right). ∂ ν ∂ C = − 2 ν 2 r − r ν 2 1 F ρ ′ ( r ν 1 ) .
For the regularizer-envelope term,
∂ ∂ r E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ w ⋆ ) , σ τ ) ] = E [ 1 σ τ ( σ τ ( r δ Z + θ w ⋆ ) − Prox λ σ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ w ⋆ ) ) ) σ τ δ Z ] = σ τ r δ − 1 δ E [ Z Prox λ σ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + r δ Z ) ) ] . \begin{aligned}
\frac{\partial}{\partial r}\mathbb{E}\left[M_{\lambda\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta w^\star\right),\sigma\tau\right)\right]
&=\mathbb{E}\left[\frac{1}{\sigma\tau}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta w^\star\right)-\operatorname{Prox}_{\lambda\sigma\tau\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta w^\star\right)\right)\right)\frac{\sigma\tau}{\sqrt{\delta}}Z\right]\\
&=\frac{\sigma\tau r}{\delta}-\frac{1}{\sqrt{\delta}}\mathbb{E}\left[Z\operatorname{Prox}_{\lambda\sigma\tau\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\theta w^\star+\frac{r}{\sqrt{\delta}}Z\right)\right)\right].
\end{aligned} ∂ r ∂ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ w ⋆ ) , σ τ ) ] = E [ σ τ 1 ( σ τ ( δ r Z + θ w ⋆ ) − Prox λσ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ w ⋆ ) ) ) δ σ τ Z ] = δ σ τ r − δ 1 E [ Z Prox λσ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + δ r Z ) ) ] .
Substituting this into ∂ r C = 0 \partial_r\mathcal{C}=0 ∂ r C = 0 , and combining with ∂ ν C = 0 \partial_\nu\mathcal{C}=0 ∂ ν C = 0 , we obtain (after setting γ = 1 / ( r ν ) \gamma=1/(r\nu) γ = 1/ ( r ν ) ),
γ = 1 r δ E [ Z Prox λ σ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + r δ Z ) ) ] \gamma=\frac{1}{r\sqrt{\delta}}\mathbb{E}\left[Z\operatorname{Prox}_{\lambda\sigma\tau\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\theta w^\star+\frac{r}{\sqrt{\delta}}Z\right)\right)\right] γ = r δ 1 E [ Z Prox λσ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + δ r Z ) ) ]
and
γ 2 = 2 r 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ( ν α Z 1 + σ Z 2 − Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ) 2 ] . \gamma^2=\frac{2}{r^2}\mathbb{E}\left[\left(1-\pi_\rho(\nu Z_1)\right)\left(\nu\alpha Z_1+\sigma Z_2-\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2)\right)^2\right]. γ 2 = r 2 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ( ν α Z 1 + σ Z 2 − Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ) 2 ] .
Differentiating C \mathcal{C} C with respect to τ \tau τ yields
∂ C ∂ τ = σ 2 τ 2 − σ τ r 2 2 δ + ν 2 α 2 2 σ τ 2 − ν 2 θ 2 2 + ∂ ∂ τ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ w ⋆ ) , σ τ ) ] . \frac{\partial\mathcal{C}}{\partial\tau}=\frac{\sigma}{2\tau^2}-\frac{\sigma\tau r^2}{2\delta}+\frac{\nu^2\alpha^2}{2\sigma\tau^2}-\frac{\nu^2\theta^2}{2}+\frac{\partial}{\partial\tau}\mathbb{E}\left[M_{\lambda\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta w^\star\right),\sigma\tau\right)\right]. ∂ τ ∂ C = 2 τ 2 σ − 2 δ σ τ r 2 + 2 σ τ 2 ν 2 α 2 − 2 ν 2 θ 2 + ∂ τ ∂ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ w ⋆ ) , σ τ ) ] .
Again by the Moreau-envelope derivative formula,
∂ ∂ τ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ w ⋆ ) , σ τ ) ] = σ 2 ( r 2 δ + ν 2 θ 2 ) − 1 2 σ τ 2 E [ Prox λ σ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + r δ Z ) ) 2 ] . \frac{\partial}{\partial\tau}\mathbb{E}\left[M_{\lambda\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta w^\star\right),\sigma\tau\right)\right]=\frac{\sigma}{2}\left(\frac{r^2}{\delta}+\nu^2\theta^2\right)-\frac{1}{2\sigma\tau^2}\mathbb{E}\left[\operatorname{Prox}_{\lambda\sigma\tau\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\theta w^\star+\frac{r}{\sqrt{\delta}}Z\right)\right)^2\right]. ∂ τ ∂ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ w ⋆ ) , σ τ ) ] = 2 σ ( δ r 2 + ν 2 θ 2 ) − 2 σ τ 2 1 E [ Prox λσ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + δ r Z ) ) 2 ] .
Substituting into ∂ τ C = 0 \partial_\tau\mathcal{C}=0 ∂ τ C = 0 gives
ν 2 α 2 + σ 2 = E [ Prox λ σ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + r δ Z ) ) 2 ] . \nu^2\alpha^2+\sigma^2=\mathbb{E}\left[\operatorname{Prox}_{\lambda\sigma\tau\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\theta w^\star+\frac{r}{\sqrt{\delta}}Z\right)\right)^2\right]. ν 2 α 2 + σ 2 = E [ Prox λσ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + δ r Z ) ) 2 ] .
Differentiating C \mathcal{C} C with respect to σ \sigma σ , we obtain
∂ C ∂ σ = − 1 2 τ − τ r 2 2 δ + ν 2 α 2 2 σ 2 τ − ν 2 θ 2 2 + ∂ ∂ σ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ w ⋆ ) , σ τ ) ] + ∂ ∂ σ E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] . \begin{aligned}
\frac{\partial\mathcal{C}}{\partial\sigma}
&=-\frac{1}{2\tau}-\frac{\tau r^2}{2\delta}+\frac{\nu^2\alpha^2}{2\sigma^2\tau}-\frac{\nu^2\theta^2}{2}+\frac{\partial}{\partial\sigma}\mathbb{E}\left[M_{\lambda\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta w^\star\right),\sigma\tau\right)\right]\\
&\quad+\frac{\partial}{\partial\sigma}\mathbb{E}\left[M_{\phi}\left(\nu\alpha Z_1+\sigma Z_2+\gamma\mathcal{B},\gamma\right)\right].
\end{aligned} ∂ σ ∂ C = − 2 τ 1 − 2 δ τ r 2 + 2 σ 2 τ ν 2 α 2 − 2 ν 2 θ 2 + ∂ σ ∂ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ w ⋆ ) , σ τ ) ] + ∂ σ ∂ E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] .
For the regularizer part,
∂ ∂ σ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( r δ Z + θ w ⋆ ) , σ τ ) ] = τ 2 ( r 2 δ + ν 2 θ 2 ) − 1 2 σ 2 τ E [ Prox λ σ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + r δ Z ) ) 2 ] . \frac{\partial}{\partial\sigma}\mathbb{E}\left[M_{\lambda\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta w^\star\right),\sigma\tau\right)\right]=\frac{\tau}{2}\left(\frac{r^2}{\delta}+\nu^2\theta^2\right)-\frac{1}{2\sigma^2\tau}\mathbb{E}\left[\operatorname{Prox}_{\lambda\sigma\tau\tilde{f}(\cdot,F_{t-1})}\left(\sigma\tau\left(\theta w^\star+\frac{r}{\sqrt{\delta}}Z\right)\right)^2\right]. ∂ σ ∂ E [ M λ f ~ ( ⋅ , F t − 1 ) ( σ τ ( δ r Z + θ w ⋆ ) , σ τ ) ] = 2 τ ( δ r 2 + ν 2 θ 2 ) − 2 σ 2 τ 1 E [ Prox λσ τ f ~ ( ⋅ , F t − 1 ) ( σ τ ( θ w ⋆ + δ r Z ) ) 2 ] .
For the data-fit part,
∂ ∂ σ E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] = σ γ − 2 γ E [ Z 2 ( 1 − π ρ ( ν Z 1 ) ) Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ] = σ γ ( 1 − E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ ϕ ′ ′ ( Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ) ] ) , \begin{aligned}
\frac{\partial}{\partial\sigma}\mathbb{E}\left[M_{\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma B,\gamma)\right]=\frac{\sigma}{\gamma}-\frac{2}{\gamma}\mathbb{E}\left[Z_2(1-\pi_{\rho}(\nu Z_1))\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2)\right] \\
&=\frac{\sigma}{\gamma}\left(1-\mathbb{E}\left[\frac{2(1-\pi_{\rho}(\nu Z_1))}{1+\gamma\phi^{\prime\prime}(\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2))}\right]\right),
\end{aligned} ∂ σ ∂ E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] = γ σ − γ 2 E [ Z 2 ( 1 − π ρ ( ν Z 1 )) Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ] = γ σ ( 1 − E [ 1 + γ ϕ ′′ ( Prox γ ϕ ( ν α Z 1 + σ Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ] ) ,
where in the second equality we used Stein’s lemma together with
d d x Prox γ ϕ ( x ) = 1 1 + γ ϕ ′ ′ ( Prox γ ϕ ( x ) ) . \frac{\mathrm{d}}{\mathrm{d}x}\operatorname{Prox}_{\gamma\phi}(x)=\frac{1}{1+\gamma\phi^{\prime\prime}(\operatorname{Prox}_{\gamma\phi}(x))}. d x d Prox γ ϕ ( x ) = 1 + γ ϕ ′′ ( Prox γ ϕ ( x )) 1 .
Substituting these formulas into ∂ σ C = 0 \partial_{\sigma}\mathcal{C}=0 ∂ σ C = 0 , and using the τ \tau τ -equation already derived, yields
1 − γ σ τ = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ ϕ ′ ′ ( Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ) ] . 1-\frac{\gamma}{\sigma\tau}=\mathbb{E}\left[\frac{2(1-\pi_{\rho}(\nu Z_1))}{1+\gamma\phi^{\prime\prime}(\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2))}\right]. 1 − σ τ γ = E [ 1 + γ ϕ ′′ ( Prox γ ϕ ( ν α Z 1 + σ Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ] .
Differentiating C \mathcal{C} C with respect to α \alpha α , we get
∂ C ∂ α = − ν 2 α σ τ + ν 2 θ − ν 2 E [ π ρ ′ ( ν Z ) ] + ∂ ∂ α E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] . \frac{\partial\mathcal{C}}{\partial\alpha}=-\frac{\nu^{2}\alpha}{\sigma\tau}+\nu^{2}\theta-\nu^{2}\mathbb{E}[\pi_{\rho}^{\prime}(\nu Z)]+\frac{\partial}{\partial\alpha}\mathbb{E}\left[M_{\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma B,\gamma)\right]. ∂ α ∂ C = − σ τ ν 2 α + ν 2 θ − ν 2 E [ π ρ ′ ( ν Z )] + ∂ α ∂ E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] .
Now
∂ ∂ α E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] = ν γ E [ Z 1 ( ν α Z 1 + σ Z 2 + γ B − Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ B ) ) ] = ν 2 α γ + ν 2 E [ π ρ ′ ( ν Z 1 ) ] − ν γ E [ Z 1 Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ B ) ] . \begin{aligned}
\frac{\partial}{\partial\alpha}\mathbb{E}\left[M_{\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma B,\gamma)\right]=\frac{\nu}{\gamma}\mathbb{E}\left[Z_1\left(\nu\alpha Z_1+\sigma Z_2+\gamma B-\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma B)\right)\right] \\
&=\frac{\nu^{2}\alpha}{\gamma}+\nu^{2}\mathbb{E}[\pi_{\rho}^{\prime}(\nu Z_1)]-\frac{\nu}{\gamma}\mathbb{E}\left[Z_1\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma B)\right].
\end{aligned} ∂ α ∂ E [ M ϕ ( ν α Z 1 + σ Z 2 + γ B , γ ) ] = γ ν E [ Z 1 ( ν α Z 1 + σ Z 2 + γ B − Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ B ) ) ] = γ ν 2 α + ν 2 E [ π ρ ′ ( ν Z 1 )] − γ ν E [ Z 1 Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ B ) ] .
Hence ∂ α C = 0 \partial_{\alpha}\mathcal{C}=0 ∂ α C = 0 becomes
γ ν ( θ − α σ τ ) + ν α = E [ Z 1 Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ B ) ] . (34) \gamma\nu\left(\theta-\frac{\alpha}{\sigma\tau}\right)+\nu\alpha=\mathbb{E}\left[Z_1\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2+\gamma B)\right]. \tag*{(34)} γ ν ( θ − σ τ α ) + ν α = E [ Z 1 Prox γ ϕ ( ν α Z 1 + σ Z 2 + γ B ) ] . (34)
Conditioning on B B B , using P ( B = 1 ∣ Z 1 ) = π ρ ( ν Z 1 ) \mathbb{P}(B=1\mid Z_1)=\pi_{\rho}(\nu Z_1) P ( B = 1 ∣ Z 1 ) = π ρ ( ν Z 1 ) , the symmetry identity Prox γ ϕ ( x + γ ) = − Prox γ ϕ ( − x ) \operatorname{Prox}_{\gamma\phi}(x+\gamma)=-\operatorname{Prox}_{\gamma\phi}(-x) Prox γ ϕ ( x + γ ) = − Prox γ ϕ ( − x ) , and then Stein’s lemma, the right-hand side of Eq. (34) simplifies to
− 2 ν E [ π ρ ′ ( ν Z 1 ) Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ] + ν α E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ ϕ ′ ′ ( Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ) ] . -2\nu\mathbb{E}\left[\pi_{\rho}^{\prime}(\nu Z_1)\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2)\right]+\nu\alpha\mathbb{E}\left[\frac{2(1-\pi_{\rho}(\nu Z_1))}{1+\gamma\phi^{\prime\prime}(\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2))}\right]. − 2 ν E [ π ρ ′ ( ν Z 1 ) Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ] + ν α E [ 1 + γ ϕ ′′ ( Prox γ ϕ ( ν α Z 1 + σ Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ] .
Using the σ \sigma σ -equation derived above, the last term equals ν α ( 1 − γ σ τ ) \nu\alpha\left(1-\frac{\gamma}{\sigma\tau}\right) ν α ( 1 − σ τ γ ) . Substituting this back into Eq. (34) yields
γ θ = − 2 E [ π ρ ′ ( ν Z 1 ) Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ] . \gamma\theta=-2\mathbb{E}\left[\pi_{\rho}^{\prime}(\nu Z_1)\operatorname{Prox}_{\gamma\phi}(\nu\alpha Z_1+\sigma Z_2)\right]. γ θ = − 2 E [ π ρ ′ ( ν Z 1 ) Prox γ ϕ ( ν α Z 1 + σ Z 2 ) ] .
Collecting the six stationarity equations, we recover exactly the nonlinear system stated in the theorem.
Finally, by the Moreau-envelope representation used in the scalarization step, the coordinatewise AO optimizer is
w ^ j A O = Prox λ σ ‾ τ ‾ f ~ ( ⋅ , F t − 1 , j ) ( σ ‾ τ ‾ ( θ ‾ w ⋆ , j + r ‾ δ Z j ) ) , \widehat{w}_{j}^{AO}=\operatorname{Prox}_{\lambda\overline{\sigma}\overline{\tau}\widetilde{f}(\cdot,F_{t-1,j})}\left(\overline{\sigma}\overline{\tau}\left(\overline{\theta}w_{\star,j}+\frac{\overline{r}}{\sqrt{\delta}}Z_j\right)\right), w j A O = Prox λ σ τ f ( ⋅ , F t − 1 , j ) ( σ τ ( θ w ⋆ , j + δ r Z j ) ) ,
with Z j ∼ i i d N ( 0 , 1 ) Z_j\mathrel{\overset{\mathrm{iid}}{\sim}}\mathcal{N}(0,1) Z j ∼ iid N ( 0 , 1 ) . Let Ξ : R 3 → R \Xi:\mathbb{R}^{3}\to\mathbb{R} Ξ : R 3 → R be locally Lipschitz with at most quadratic growth. For ζ = ( σ , τ , r , θ ) \zeta=(\sigma,\tau,r,\theta) ζ = ( σ , τ , r , θ ) , define
m ζ ( w , F ) = E Z [ M λ f ~ ( ⋅ , F ) ( σ τ ( r δ Z + θ w ) , σ τ ) ] . m_{\zeta}(w,F)=\mathbb{E}_{Z}\left[M_{\lambda\widetilde{f}(\cdot,F)}\left(\sigma\tau\left(\frac{r}{\sqrt{\delta}}Z+\theta w\right),\sigma\tau\right)\right]. m ζ ( w , F ) = E Z [ M λ f ( ⋅ , F ) ( σ τ ( δ r Z + θ w ) , σ τ ) ] .
Let K K K be a compact scalar-parameter set on which σ τ \sigma\tau σ τ is bounded away from zero. The assumed continuity and uniform quadratic-growth bound of the Moreau envelope, together with dominated convergence in Z Z Z , imply that ( ζ , w , F ) ↦ m ζ ( w , F ) (\zeta,w,F)\mapsto m_{\zeta}(w,F) ( ζ , w , F ) ↦ m ζ ( w , F ) is jointly continuous and that
sup ζ ∈ K ∣ m ζ ( w , F ) ∣ ≤ C K ( 1 + w 2 + F 2 ) . \sup_{\zeta\in K}|m_{\zeta}(w,F)|\le C_K(1+w^2+F^2). ζ ∈ K sup ∣ m ζ ( w , F ) ∣ ≤ C K ( 1 + w 2 + F 2 ) .
Conditionally on the teacher and the first t − 1 t-1 t − 1 tasks, the pairs ( w ∗ , j , F t − 1 , j ) (w_{*,j},F_{t-1,j}) ( w ∗ , j , F t − 1 , j ) are fixed and the g j g_j g j ’s are independent standard Gaussians. Fix R > 0 R>0 R > 0 and ϵ > 0 \epsilon>0 ϵ > 0 . By continuity of the Moreau envelope and of m ζ m_\zeta m ζ , and compactness of K K K , there exists a finite net { ζ 1 , … , ζ N } \{\zeta_1,\ldots,\zeta_N\} { ζ 1 , … , ζ N } such that, for every ζ ∈ K \zeta\in K ζ ∈ K , some ζ k \zeta_k ζ k approximates both M λ f ( ⋅ , F ) ( σ τ ( r d ζ + θ w ∗ , j ) , σ τ ) M_{\lambda f_{(\cdot,F)}}\left(\sigma\tau\left(\frac{r}{\sqrt{d}}\zeta+\theta w_{*,j}\right),\sigma\tau\right) M λ f ( ⋅ , F ) ( σ τ ( d r ζ + θ w ∗ , j ) , σ τ ) uniformly on { ( w , F ) : w 2 + F 2 + g 2 ≤ R 2 } \{(w,F):w^2+F^2+g^2\le R^2\} {( w , F ) : w 2 + F 2 + g 2 ≤ R 2 } and m ζ ( w , F ) m_\zeta(w,F) m ζ ( w , F ) uniformly on { ( w , F ) : w 2 + F 2 ≤ R 2 } \{(w,F):w^2+F^2\le R^2\} {( w , F ) : w 2 + F 2 ≤ R 2 } to within ϵ \epsilon ϵ . Since the net is finite, the pointwise conditional law of large numbers gives convergence simultaneously at all its points. Hence, on the bounded region, the triangle inequality bounds the supremum over ζ ∈ K \zeta\in K ζ ∈ K by the maximum over the finite net plus 2 ϵ 2\epsilon 2 ϵ . On the complementary region, the uniform quadratic-growth bound gives,
sup ζ ∈ K 1 d ∑ j = 1 d ∣ M λ f ( ⋅ , F t − 1 , j ) ( σ τ ( r d g j + θ w ∗ , j ) , σ τ ) ∣ 1 { w ∗ , j 2 + F t − 1 , j 2 + g j 2 > R 2 } ≤ sup ζ ∈ K 1 d ∑ j = 1 d C ( 1 + σ τ ( r d g j + θ w ∗ , j ) 2 + F t − 1 , j 2 ) ≤ C K d ∑ j = 1 d ( 1 + w ∗ , j 2 + F t − 1 , j 2 + g j 2 ) 1 { w ∗ , j 2 + F t − 1 , j 2 + g j 2 > R 2 } , \begin{aligned}
\sup_{\zeta\in K}\frac{1}{d}\sum_{j=1}^{d}\left|M_{\lambda f_{(\cdot,F_{t-1,j})}}\left(\sigma\tau\left(\frac{r}{\sqrt{d}}g_j+\theta w_{*,j}\right),\sigma\tau\right)\right|\mathbf{1}_{\{w_{*,j}^{2}+F_{t-1,j}^{2}+g_j^{2}>R^{2}\}} \\
&\le\sup_{\zeta\in K}\frac{1}{d}\sum_{j=1}^{d}C\left(1+\sigma\tau\left(\frac{r}{\sqrt{d}}g_j+\theta w_{*,j}\right)^{2}+F_{t-1,j}^{2}\right)\le\frac{C_K}{d}\sum_{j=1}^{d}\left(1+w_{*,j}^{2}+F_{t-1,j}^{2}+g_j^{2}\right)\mathbf{1}_{\{w_{*,j}^{2}+F_{t-1,j}^{2}+g_j^{2}>R^{2}\}},
\end{aligned} ζ ∈ K sup d 1 j = 1 ∑ d M λ f ( ⋅ , F t − 1 , j ) ( σ τ ( d r g j + θ w ∗ , j ) , σ τ ) 1 { w ∗ , j 2 + F t − 1 , j 2 + g j 2 > R 2 } ≤ ζ ∈ K sup d 1 j = 1 ∑ d C ( 1 + σ τ ( d r g j + θ w ∗ , j ) 2 + F t − 1 , j 2 ) ≤ d C K j = 1 ∑ d ( 1 + w ∗ , j 2 + F t − 1 , j 2 + g j 2 ) 1 { w ∗ , j 2 + F t − 1 , j 2 + g j 2 > R 2 } ,
and the analogous bound holds for the conditional expectations defining m ζ m_\zeta m ζ . The assumed convergence in 2-Wasserstein distance and Gaussian tail integrability imply that these two bounds vanish in probability after first taking d → ∞ d\to\infty d → ∞ and then R → ∞ R\to\infty R → ∞ . Therefore, first letting d → ∞ d\to\infty d → ∞ , then ϵ → 0 \epsilon\to0 ϵ → 0 , and finally R → ∞ R\to\infty R → ∞ , we obtain,
sup ζ ∈ K ∣ 1 d ∑ j = 1 d [ M λ f ( ⋅ , F t − 1 , j ) ( σ τ ( r d g j + θ w ∗ , j ) , σ τ ) − m ζ ( w ∗ , j , F t − 1 , j ) ] ∣ → P 0. \sup_{\zeta\in K}\left|\frac{1}{d}\sum_{j=1}^{d}\left[M_{\lambda f_{(\cdot,F_{t-1,j})}}\left(\sigma\tau\left(\frac{r}{\sqrt{d}}g_j+\theta w_{*,j}\right),\sigma\tau\right)-m_\zeta(w_{*,j},F_{t-1,j})\right]\right|\xrightarrow{\mathbb P}0. ζ ∈ K sup d 1 j = 1 ∑ d [ M λ f ( ⋅ , F t − 1 , j ) ( σ τ ( d r g j + θ w ∗ , j ) , σ τ ) − m ζ ( w ∗ , j , F t − 1 , j ) ] P 0.
We now use the same arguments to prove a similar result. Fix R > 0 R>0 R > 0 and B R = { ( w , F ) : w 2 + F 2 ≤ R 2 } B_R=\{(w,F):w^2+F^2\le R^2\} B R = {( w , F ) : w 2 + F 2 ≤ R 2 } , define m ζ , R ( w , F ) = m ζ ( w , F ) χ R m_{\zeta,R}(w,F)=m_\zeta(w,F)\chi_R m ζ , R ( w , F ) = m ζ ( w , F ) χ R where χ R = 1 \chi_R=1 χ R = 1 on B R B_R B R , χ R = 0 \chi_R=0 χ R = 0 outside B 2 R B_{2R} B 2 R , 0 ≤ χ R ≤ 1 0\le\chi_R\le1 0 ≤ χ R ≤ 1 and χ R \chi_R χ R is continuous. Due to joint continuity of m ζ m_\zeta m ζ we may argue (as before) for finite net in K K K : { ζ 1 , … , ζ N } \{\zeta_1,\ldots,\zeta_N\} { ζ 1 , … , ζ N } . Thus, for every ζ ∈ K \zeta\in K ζ ∈ K , ϵ > 0 \epsilon>0 ϵ > 0 there exists some ζ k \zeta_k ζ k satisfying sup ( w , F ) ∈ B 2 R ∣ m ζ , R ( w , F ) − m ζ k , R ( w , F ) ∣ ≤ ϵ \sup_{(w,F)\in B_{2R}}|m_{\zeta,R}(w,F)-m_{\zeta_k,R}(w,F)|\le\epsilon sup ( w , F ) ∈ B 2 R ∣ m ζ , R ( w , F ) − m ζ k , R ( w , F ) ∣ ≤ ϵ . Thus,
∣ ∫ m ζ , R d μ d − ∫ m ζ , R d μ t − 1 ∣ ≤ ∣ ∫ m ζ k , R d μ d − ∫ m ζ k , R d μ t − 1 ∣ + 2 ϵ , \left|\int m_{\zeta,R}\,d\mu_d-\int m_{\zeta,R}\,d\mu_{t-1}\right|\le\left|\int m_{\zeta_k,R}\,d\mu_d-\int m_{\zeta_k,R}\,d\mu_{t-1}\right|+2\epsilon, ∫ m ζ , R d μ d − ∫ m ζ , R d μ t − 1 ≤ ∫ m ζ k , R d μ d − ∫ m ζ k , R d μ t − 1 + 2 ϵ ,
where μ d = 1 d ∑ j = 1 d δ ( w ∗ , j , F t − 1 , j ) \mu_d=\frac{1}{d}\sum_{j=1}^{d}\delta_{(w_{*,j},F_{t-1,j})} μ d = d 1 ∑ j = 1 d δ ( w ∗ , j , F t − 1 , j ) and μ d → P μ t − 1 \mu_d\xrightarrow{\mathbb P}\mu_{t-1} μ d P μ t − 1 by the assumption of convergence in 2-Wasserstein distance. Taking the supremum,
sup ζ ∈ K ∣ ∫ m ζ , R d μ d − ∫ m ζ , R d μ t − 1 ∣ ≤ max 1 ≤ k ≤ N ∣ ∫ m ζ k , R d μ d − ∫ m ζ k , R d μ t − 1 ∣ + 2 ϵ (35) \sup_{\zeta\in K}\left|\int m_{\zeta,R}\,d\mu_d-\int m_{\zeta,R}\,d\mu_{t-1}\right|\le\max_{1\le k\le N}\left|\int m_{\zeta_k,R}\,d\mu_d-\int m_{\zeta_k,R}\,d\mu_{t-1}\right|+2\epsilon \tag*{(35)} ζ ∈ K sup ∫ m ζ , R d μ d − ∫ m ζ , R d μ t − 1 ≤ 1 ≤ k ≤ N max ∫ m ζ k , R d μ d − ∫ m ζ k , R d μ t − 1 + 2 ϵ (35)
There are now finitely many function m ζ k , R m_{\zeta_k,R} m ζ k , R that are bounded and continuous (if w 2 + F 2 ≤ 4 R 2 w^2+F^2\le4R^2 w 2 + F 2 ≤ 4 R 2 ) since μ d → P μ t − 1 \mu_d\xrightarrow{\mathbb P}\mu_{t-1} μ d P μ t − 1 . For all fixed k k k ,
∫ m ζ k , R d μ d → P ∫ m ζ k , R d μ t − 1 ⟹ max 1 ≤ k ≤ N ∣ ∫ m ζ k , R d μ d − ∫ m ζ k , R d μ t − 1 ∣ → ϵ → 0 in Eq. (35) P 0 \int m_{\zeta_k,R}\,d\mu_d\xrightarrow{\mathbb P}\int m_{\zeta_k,R}\,d\mu_{t-1}\quad\Longrightarrow\quad\max_{1\le k\le N}\left|\int m_{\zeta_k,R}\,d\mu_d-\int m_{\zeta_k,R}\,d\mu_{t-1}\right|\xrightarrow[\epsilon\to0\text{ in Eq. (35)}]{\mathbb P}0 ∫ m ζ k , R d μ d P ∫ m ζ k , R d μ t − 1 ⟹ 1 ≤ k ≤ N max ∫ m ζ k , R d μ d − ∫ m ζ k , R d μ t − 1 P ϵ → 0 in Eq. (35) 0
⟹ sup ζ ∈ K ∣ ∫ m ζ , R d μ d − ∫ m ζ , R d μ t − 1 ∣ → P 0. \Longrightarrow\sup_{\zeta\in K}\left|\int m_{\zeta,R}\,d\mu_d-\int m_{\zeta,R}\,d\mu_{t-1}\right|\xrightarrow{\mathbb P}0. ⟹ ζ ∈ K sup ∫ m ζ , R d μ d − ∫ m ζ , R d μ t − 1 P 0.
Now we focus on the case where w 2 + F 2 > R 2 w^2+F^2>R^2 w 2 + F 2 > R 2 , in this case the convergence is immediate as we show here. Due to the quadratic growth of the Moreau envelope,
sup ζ ∈ K ∣ ∫ ( m ζ − m ζ , R ) d μ d ∣ ≤ sup ζ ∈ K ∫ ∣ m ζ ∣ 1 { ∣ ∣ ( w , F ) ∣ ∣ > R } d μ d ≤ C K ′ ∫ ( 1 + w 2 + F 2 ) 1 { ∣ ∣ ( w , F ) ∣ ∣ > R } d μ d , \begin{aligned}
\sup_{\zeta\in K}\left|\int(m_\zeta-m_{\zeta,R})\,d\mu_d\right|\le\sup_{\zeta\in K}\int|m_\zeta|\mathbf{1}_{\{||(w,F)||>R\}}\,d\mu_d \\
&\le C'_K\int(1+w^2+F^2)\mathbf{1}_{\{||(w,F)||>R\}}\,d\mu_d,
\end{aligned} ζ ∈ K sup ∫ ( m ζ − m ζ , R ) d μ d ≤ ζ ∈ K sup ∫ ∣ m ζ ∣ 1 { ∣∣ ( w , F ) ∣∣ > R } d μ d ≤ C K ′ ∫ ( 1 + w 2 + F 2 ) 1 { ∣∣ ( w , F ) ∣∣ > R } d μ d ,
the same bound holds for μ t − 1 \mu_{t-1} μ t − 1 . The assumed convergence in 2-Wasserstein distance implies that for every ϵ > 0 \epsilon>0 ϵ > 0 ,
lim R → ∞ lim sup d → ∞ P ( ∫ ( 1 + w 2 + F 2 ) 1 { ∣ ∣ ( w , F ) ∣ ∣ > R } d μ d > ϵ ) = 0. \lim_{R\to\infty}\limsup_{d\to\infty}\mathbb{P}\left(\int(1+w^2+F^2)\mathbf{1}_{\{||(w,F)||>R\}}\,d\mu_d>\epsilon\right)=0. R → ∞ lim d → ∞ lim sup P ( ∫ ( 1 + w 2 + F 2 ) 1 { ∣∣ ( w , F ) ∣∣ > R } d μ d > ϵ ) = 0.
Consequently,
lim R → ∞ lim d → ∞ sup ζ ∈ K P ( ∣ ∫ ( m ζ − m ζ , R ) d μ d ∣ > ϵ ) = 0. \lim_{R \to\infty}\lim_{d \to\infty}\sup_{\zeta\in K}\mathbb{P}\left(\left|\int(m_{\zeta}-m_{\zeta,R})\,\mathrm{d}\mu_d\right|>\epsilon\right)=0. R → ∞ lim d → ∞ lim ζ ∈ K sup P ( ∫ ( m ζ − m ζ , R ) d μ d > ϵ ) = 0.
Moreover, μ t − 1 \mu_{t-1} μ t − 1 has finite second moment, so by dominated convergence,
∫ ( 1 + w 2 + F 2 ) 1 { ∥ ( w , F ) ∥ > R } d μ t − 1 → R → ∞ P 0 , \int\left(1+w^2+F^2\right)\mathbf{1}_{\{\lVert(w,F)\rVert>R\}}\,\mathrm{d}\mu_{t-1}\xrightarrow[R\to\infty]{\mathbb{P}}0, ∫ ( 1 + w 2 + F 2 ) 1 {∥( w , F )∥ > R } d μ t − 1 P R → ∞ 0 ,
The same quadratic-growth bound then gives sup ζ ∈ K ∣ ∫ ( m ζ − m ζ , R ) d μ t − 1 ∣ ⟶ 0 \sup_{\zeta\in K}\left|\int(m_{\zeta}-m_{\zeta,R})\,\mathrm{d}\mu_{t-1}\right|\longrightarrow0 sup ζ ∈ K ∫ ( m ζ − m ζ , R ) d μ t − 1 ⟶ 0 . Combining these bounds with the convergence of the integrals, first letting d → ∞ d\to\infty d → ∞ for fixed R R R , and then letting R → ∞ R\to\infty R → ∞ , yields sup ζ ∈ K ∣ ∫ m ζ d μ d − ∫ m ζ d μ t − 1 ∣ → P 0 \sup_{\zeta\in K}\left|\int m_{\zeta}\,\mathrm{d}\mu_d-\int m_{\zeta}\,\mathrm{d}\mu_{t-1}\right|\xrightarrow{\mathbb{P}}0 sup ζ ∈ K ∫ m ζ d μ d − ∫ m ζ d μ t − 1 P 0 . By separability of f f f and its Moreau envelope, we conclude that, uniformly in ζ \zeta ζ ,
1 d M λ f ( ⋅ , F t − 1 ) ( σ T ( r δ g + θ w ⋆ , σ T ) ) → P E [ M λ f ( ⋅ , F t − 1 ) ( σ T ( r δ Z + θ w ⋆ , σ T ) ) ] , \frac{1}{d}M_{\lambda f_{(\cdot,F_{t-1})}}\left(\sigma T\left(\frac{r}{\sqrt{\delta}}\mathbf{g}+\theta\mathbf{w}_{\star},\sigma T\right)\right)\xrightarrow{\mathbb{P}}\mathbb{E}\left[M_{\lambda f_{(\cdot,F_{t-1})}}\left(\sigma T\left(\frac{r}{\sqrt{\delta}}Z+\theta\mathbf{w}^{\star},\sigma T\right)\right)\right], d 1 M λ f ( ⋅ , F t − 1 ) ( σ T ( δ r g + θ w ⋆ , σ T ) ) P E [ M λ f ( ⋅ , F t − 1 ) ( σ T ( δ r Z + θ w ⋆ , σ T ) ) ] ,
where Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) is independent of ( w ⋆ , F t − 1 ) (\mathbf{w}^{\star},F_{t-1}) ( w ⋆ , F t − 1 ) . For ϵ > 0 \epsilon>0 ϵ > 0 , let,
S ϵ : = { u ∈ R d : ∣ 1 d ∑ j = 1 d E ( u j , w ⋆ , j , F t − 1 , j ) − E [ Ξ ( Prox λ τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + r ˉ t δ Z ) ) , w ⋆ , F t − 1 ) ] ∣ < ϵ } . S_{\epsilon}:=\left\{\mathbf{u}\in\mathbb{R}^{d}:\left|\frac{1}{d}\sum_{j=1}^{d}\mathbb{E}(u_j,w_{\star,j},F_{t-1,j})-\mathbb{E}\left[\Xi\left(\operatorname{Prox}_{\lambda\bar{\tau}_t\tilde{f}_{(\cdot,F_{t-1})}}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t\mathbf{w}^{\star}+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),\mathbf{w}^{\star},F_{t-1}\right)\right]\right|<\epsilon\right\}. S ϵ := { u ∈ R d : d 1 j = 1 ∑ d E ( u j , w ⋆ , j , F t − 1 , j ) − E [ Ξ ( Prox λ τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + δ r ˉ t Z ) ) , w ⋆ , F t − 1 ) ] < ϵ } .
By Eqs. (27) and (29), P ( w ^ A O ∈ S ϵ ) ⟶ 1 \mathbb{P}(\widehat{\mathbf{w}}^{AO}\in S_{\epsilon})\longrightarrow1 P ( w A O ∈ S ϵ ) ⟶ 1 . Conditionally on the teacher and the previous tasks, S ϵ S_{\epsilon} S ϵ is fixed. Recall the artificial bounded sets introduced at the proof and consider the variables belong to these bounded sets, the limiting AO objective is continuous and has a unique saddle point whose coordinatewise optimizer belongs to S ϵ S_{\epsilon} S ϵ with probability tending to one. The CGMT theorem (see Salehi et al. (2019, Corollary 3)) therefore yields P ( w ^ t ∈ S ϵ ) ⟶ 1 \mathbb{P}(\widehat{\mathbf{w}}_t\in S_{\epsilon})\longrightarrow1 P ( w t ∈ S ϵ ) ⟶ 1 . It follows that,
1 d ∑ j = 1 d E ( w ^ t , j , w ⋆ , j , F t − 1 , j ) → P E [ Ξ ( Prox λ τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + r ˉ t δ Z ) ) , w ⋆ , F t − 1 ) ] . \frac{1}{d}\sum_{j=1}^{d}\mathbb{E}(\widehat{w}_{t,j},w_{\star,j},F_{t-1,j})\xrightarrow{\mathbb{P}}\mathbb{E}\left[\Xi\left(\operatorname{Prox}_{\lambda\bar{\tau}_t\tilde{f}_{(\cdot,F_{t-1})}}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t\mathbf{w}^{\star}+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),\mathbf{w}^{\star},F_{t-1}\right)\right]. d 1 j = 1 ∑ d E ( w t , j , w ⋆ , j , F t − 1 , j ) P E [ Ξ ( Prox λ τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + δ r ˉ t Z ) ) , w ⋆ , F t − 1 ) ] .
In particular, taking Ξ ( x , w , F ) = Ψ ( x , w ) \Xi(x,w,F)=\Psi(x,w) Ξ ( x , w , F ) = Ψ ( x , w ) , we obtain, for every locally Lipschitz Ψ \Psi Ψ with at most quadratic growth,
1 d ∑ j = 1 d Ψ ( w ^ t , j , w ⋆ , j ) → P E [ Ψ ( Prox λ τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + r ˉ t δ Z ) ) , w ⋆ ) ] . \frac{1}{d}\sum_{j=1}^{d}\Psi(\widehat{w}_{t,j},w_{\star,j})\xrightarrow{\mathbb{P}}\mathbb{E}\left[\Psi\left(\operatorname{Prox}_{\lambda\bar{\tau}_t\tilde{f}_{(\cdot,F_{t-1})}}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t\mathbf{w}^{\star}+\frac{\bar{r}_t}{\sqrt{\delta}}Z\right)\right),\mathbf{w}^{\star}\right)\right]. d 1 j = 1 ∑ d Ψ ( w t , j , w ⋆ , j ) P E [ Ψ ( Prox λ τ ˉ t f ~ ( ⋅ , F t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + δ r ˉ t Z ) ) , w ⋆ ) ] .
By induction and the uniqueness of the system solution, we know that w t \mathbf{w}_t w t is well-defined for all t ≤ T t\leq T t ≤ T , define for some iteration t t t the following derived system of equations,
{ ν 2 α t = E [ w ⋆ Prox λ σ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + r t δ Z ) ) ] , γ t = 1 r t δ E [ Z Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + r t δ Z ) ) ] , ν 2 α t 2 + σ t 2 = E [ Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + r t δ Z ) ) 2 ] , γ t 2 = 2 r t 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ( ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] , γ t θ t = − 2 E [ π ˉ ρ ( ν Z 1 ) Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ] , 1 − γ t σ t τ t = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . \left\{
\begin{aligned}
\nu^2\alpha_t&=\mathbb{E}\left[\mathbf{w}^{\star}\operatorname{Prox}_{\lambda\sigma_t\tilde{f}_{(\cdot,F_{t-1})}}\left(\sigma_t\tau_t\left(\theta_t\mathbf{w}^{\star}+\frac{r_t}{\sqrt{\delta}}Z\right)\right)\right],\\
\gamma_t&=\frac{1}{r_t\sqrt{\delta}}\mathbb{E}\left[Z\operatorname{Prox}_{\lambda\sigma_t\tau_t\tilde{f}_{(\cdot,F_{t-1})}}\left(\sigma_t\tau_t\left(\theta_t\mathbf{w}^{\star}+\frac{r_t}{\sqrt{\delta}}Z\right)\right)\right],\\
\nu^2\alpha_t^2+\sigma_t^2&=\mathbb{E}\left[\operatorname{Prox}_{\lambda\sigma_t\tau_t\tilde{f}_{(\cdot,F_{t-1})}}\left(\sigma_t\tau_t\left(\theta_t\mathbf{w}^{\star}+\frac{r_t}{\sqrt{\delta}}Z\right)\right)^2\right],\\
\gamma_t^2&=\frac{2}{r_t^2}\mathbb{E}\left[\left(1-\pi_{\rho}(\nu Z_1)\right)\left(\nu\alpha_t Z_1+\sigma_t Z_2-\operatorname{Prox}_{\gamma_t\phi}\left(\nu\alpha_t Z_1+\sigma_t Z_2\right)\right)^2\right],\\
\gamma_t\theta_t&=-2\mathbb{E}\left[\bar{\pi}_{\rho}(\nu Z_1)\operatorname{Prox}_{\gamma_t\phi}\left(\nu\alpha_t Z_1+\sigma_t Z_2\right)\right],\\
1-\frac{\gamma_t}{\sigma_t\tau_t}&=\mathbb{E}\left[\frac{2\left(1-\pi_{\rho}(\nu Z_1)\right)}{1+\gamma_t\phi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\phi}\left(\nu\alpha_t Z_1+\sigma_t Z_2\right)\right)}\right].
\end{aligned}
\right. ⎩ ⎨ ⎧ ν 2 α t γ t ν 2 α t 2 + σ t 2 γ t 2 γ t θ t 1 − σ t τ t γ t = E [ w ⋆ Prox λ σ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + δ r t Z ) ) ] , = r t δ 1 E [ Z Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + δ r t Z ) ) ] , = E [ Prox λ σ t τ t f ~ ( ⋅ , F t − 1 ) ( σ t τ t ( θ t w ⋆ + δ r t Z ) ) 2 ] , = r t 2 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ( ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] , = − 2 E [ π ˉ ρ ( ν Z 1 ) Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ] , = E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ( 1 − π ρ ( ν Z 1 ) ) ] .
where Z , Z 1 , Z 2 ∼ i i d N ( 0 , 1 ) Z,Z_1,Z_2\overset{\mathrm{iid}}{\sim}\mathcal{N}(0,1) Z , Z 1 , Z 2 ∼ iid N ( 0 , 1 ) .
Therefore, by iterating the equations up to iteration T T T , for every locally Lipschitz function Ψ : R 2 → R \Psi:\mathbb{R}^2\to\mathbb{R} Ψ : R 2 → R , we get that,
1 d ∑ j = 1 d Ψ ( w T , j , w ⋆ , j ) → P E [ Ψ ( Prox λ τ ˉ T f ~ ( ⋅ , F T − 1 ) ( σ ˉ T τ ˉ T ( θ ˉ T w ⋆ + r ˉ T δ Z ) ) , w ⋆ ) ] . \frac{1}{d}\sum_{j=1}^{d}\Psi(w_{T,j},w_{\star,j})\xrightarrow{\mathbb{P}}\mathbb{E}\left[\Psi\left(\operatorname{Prox}_{\lambda\bar{\tau}_T\tilde{f}_{(\cdot,F_{T-1})}}\left(\bar{\sigma}_T\bar{\tau}_T\left(\bar{\theta}_T\mathbf{w}^{\star}+\frac{\bar{r}_T}{\sqrt{\delta}}Z\right)\right),\mathbf{w}^{\star}\right)\right]. d 1 j = 1 ∑ d Ψ ( w T , j , w ⋆ , j ) P E [ Ψ ( Prox λ τ ˉ T f ~ ( ⋅ , F T − 1 ) ( σ ˉ T τ ˉ T ( θ ˉ T w ⋆ + δ r ˉ T Z ) ) , w ⋆ ) ] .
Remark 11 (Convergence of 1 d ∑ j = 1 d δ ( w ⋆ , j , F t − 1 , j ) → ( μ t − 1 \frac{1}{d}\sum_{j=1}^{d}\delta(w_{\star,j},F_{t-1,j}) \xrightarrow{\mathstrut} \mu_{t-1} d 1 ∑ j = 1 d δ ( w ⋆ , j , F t − 1 , j ) ( μ t − 1 ). Throughout the paper, we will mostly analyze the case in which F t − 1 = w t − 1 \mathbf{F}_{t-1}=\mathbf{w}_{t-1} F t − 1 = w t − 1 . We prove by induction that, for every fixed t t t , 1 d ∑ j = 1 d δ ( w ⋆ , j , w t , j ) → P W 2 μ t \frac{1}{d}\sum_{j=1}^{d}\delta(w_{\star,j},w_{t,j}) \xrightarrow{\mathbb{P}}_{W_2} \mu_t d 1 ∑ j = 1 d δ ( w ⋆ , j , w t , j ) P W 2 μ t .4 At t = 0 t=0 t = 0 , w 0 = 0 \mathbf{w}_0=\mathbf{0} w 0 = 0 , so 1 d ∑ j = 1 d δ ( w ⋆ , j , 0 ) → P W 2 μ 0 \frac{1}{d}\sum_{j=1}^{d}\delta(w_{\star,j},0) \xrightarrow{\mathbb{P}}_{W_2} \mu_0 d 1 ∑ j = 1 d δ ( w ⋆ , j , 0 ) P W 2 μ 0 by the law of large numbers and the finite second moment of w ⋆ w^\star w ⋆ . Suppose now that 1 d ∑ j = 1 d δ ( w ⋆ , j , w t − 1 , j ) → P W 2 μ t − 1 \frac{1}{d}\sum_{j=1}^{d}\delta(w_{\star,j},w_{t-1,j}) \xrightarrow{\mathbb{P}}_{W_2} \mu_{t-1} d 1 ∑ j = 1 d δ ( w ⋆ , j , w t − 1 , j ) P W 2 μ t − 1 . Let ( w ⋆ , w t − 1 ) ∼ μ t − 1 (w^\star,w_{t-1})\sim\mu_{t-1} ( w ⋆ , w t − 1 ) ∼ μ t − 1 , and let Z ∼ N ( 0 , 1 ) Z\sim N(0,1) Z ∼ N ( 0 , 1 ) be independent of this pair, and define w t = Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , w t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + r ˉ t δ ˉ Z ) ) w_t=\operatorname{Prox}_{\lambda\bar{\sigma}_t\bar{\tau}_t\tilde f(\cdot,w_{t-1})}\left(\bar{\sigma}_t\bar{\tau}_t\left(\bar{\theta}_t w^\star+\frac{\bar r_t}{\sqrt{\bar\delta}}Z\right)\right) w t = Prox λ σ ˉ t τ ˉ t f ~ ( ⋅ , w t − 1 ) ( σ ˉ t τ ˉ t ( θ ˉ t w ⋆ + δ ˉ r ˉ t Z ) ) , where the scalar parameters are the unique limiting task-t t t saddle-point parameters determined by μ t − 1 \mu_{t-1} μ t − 1 .
Conditionally on the teacher and the first t − 1 t-1 t − 1 tasks, the pairs ( w j ⋆ , w t − 1 , j ) (w^\star_j,w_{t-1,j}) ( w j ⋆ , w t − 1 , j ) are fixed, while the Gaussian variables in the task-t t t AO representation are fresh and independent. Hence, for every bounded Lipschitz function ψ \psi ψ , the conditional law of large numbers, the convergence of the empirical distribution, and the convergence of the scalar AO parameters give 1 d ∑ j = 1 d ψ ( w ⋆ , j , w ^ t , j A O ) → P E [ ψ ( w ⋆ , w t ) ] \frac{1}{d}\sum_{j=1}^{d}\psi(w_{\star,j},\widehat{w}^{\mathrm{AO}}_{t,j}) \xrightarrow{\mathbb{P}} \mathbb{E}[\psi(w^\star,w_t)] d 1 ∑ j = 1 d ψ ( w ⋆ , j , w t , j AO ) P E [ ψ ( w ⋆ , w t )] (the same arguments that led to Eq. (30) ). Thus the AO empirical distribution converges weakly to μ t \mu_t μ t . Applying the same conditional truncation and tail argument as in the proof (see Eq. (26) ) to ψ ( w , u ) = w 2 + u 2 \psi(w,u)=w^2+u^2 ψ ( w , u ) = w 2 + u 2 gives convergence of the second moment. Therefore, 1 d ∑ j = 1 d δ ( w ⋆ , j , w ^ t , j A O ) → P W 2 μ t \frac{1}{d}\sum_{j=1}^{d}\delta(w_{\star,j},\widehat{w}^{\mathrm{AO}}_{t,j}) \xrightarrow{\mathbb{P}}_{W_2} \mu_t d 1 ∑ j = 1 d δ ( w ⋆ , j , w t , j AO ) P W 2 μ t . The CGMT optimizer-transfer argument transfers both the bounded-Lipschitz convergence and the quadratic-moment convergence from the AO optimizer to the actual estimator, yielding 1 d ∑ j = 1 d δ ( w ⋆ , j , w t , j ) → P W 2 μ t \frac{1}{d}\sum_{j=1}^{d}\delta(w_{\star,j},w_{t,j}) \xrightarrow{\mathbb{P}}_{W_2} \mu_t d 1 ∑ j = 1 d δ ( w ⋆ , j , w t , j ) P W 2 μ t . Since F t = w t \mathbf{F}_t=\mathbf{w}_t F t = w t , this is exactly the empirical-state condition required to analyze task t + 1 t+1 t + 1 , completing the induction.
Optimal L2 Regularization: Proofs for Section 3.2
A closed form expression for the limiting performance of the final iterate under L2 regularization
We are specifically interested in L2 regularization, where f ~ ( w , w t − 1 ) = ( w − w t − 1 ) 2 \tilde{f}(w,w_{t-1})=(w-w_{t-1})^2 f ~ ( w , w t − 1 ) = ( w − w t − 1 ) 2 . We verify the conditions of the full Theorem 1. Its Moreau envelope is given by M λ f ~ ( ⋅ , v ) ( x , c ) = λ ( x − v ) 2 1 + 2 λ c M_{\lambda\tilde{f}(\cdot,v)}(x,c)=\frac{\lambda(x-v)^2}{1+2\lambda c} M λ f ~ ( ⋅ , v ) ( x , c ) = 1 + 2 λ c λ ( x − v ) 2 it is continuous and admits quadratic bound hold, furthermore, the joint convergence of the empirical distribution follows by Remark 11.
For this specific choice, we show that the the first three equations of the six-equation saddle-point system can be solved explicitly. For every η > 0 \eta>0 η > 0 ,
Prox η f ~ ( x ) = argmin u ∈ R { 1 2 ( u − x ) 2 + η ( u − w t − 1 ) 2 } . \operatorname{Prox}_{\eta\tilde{f}}(x)=\underset{u\in\mathbb{R}}{\operatorname{argmin}}\left\{\frac{1}{2}(u-x)^2+\eta(u-w_{t-1})^2\right\}. Prox η f ~ ( x ) = u ∈ R argmin { 2 1 ( u − x ) 2 + η ( u − w t − 1 ) 2 } .
Solving the optimization problem, stationary condition gives
u − x + 2 η ( u − w t − 1 ) = 0 , u-x+2\eta(u-w_{t-1})=0, u − x + 2 η ( u − w t − 1 ) = 0 ,
hence ( 1 + 2 η ) u = x + 2 η w t − 1 (1+2\eta)u=x+2\eta w_{t-1} ( 1 + 2 η ) u = x + 2 η w t − 1 , and therefore
Prox η f ~ ( x ) = x + 2 η w t − 1 1 + 2 η . \operatorname{Prox}_{\eta\tilde{f}}(x)=\frac{x+2\eta w_{t-1}}{1+2\eta}. Prox η f ~ ( x ) = 1 + 2 η x + 2 η w t − 1 .
Let ( α t , σ t , r t , τ t , γ t , θ t ) (\alpha_t,\sigma_t,r_t,\tau_t,\gamma_t,\theta_t) ( α t , σ t , r t , τ t , γ t , θ t ) be the unique solution for every t ≤ T t\leq T t ≤ T . In the scalar saddle-point system,
Prox λ σ t τ t f ~ ( σ t τ t ( θ t w ⋆ + r t δ Z ) ) = σ t τ t ( θ t w ⋆ + r t δ Z ) + 2 λ σ t τ t w t − 1 1 + 2 λ σ t τ t = σ t τ t 1 + 2 λ σ t τ t ( θ t w ⋆ + r t δ Z + 2 λ w t − 1 ) . \operatorname{Prox}_{\lambda\sigma_t\tau_t\tilde{f}}\left(\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z\right)\right)=\frac{\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z\right)+2\lambda\sigma_t\tau_t w_{t-1}}{1+2\lambda\sigma_t\tau_t}=\frac{\sigma_t\tau_t}{1+2\lambda\sigma_t\tau_t}\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z+2\lambda w_{t-1}\right). Prox λ σ t τ t f ~ ( σ t τ t ( θ t w ⋆ + δ r t Z ) ) = 1 + 2 λ σ t τ t σ t τ t ( θ t w ⋆ + δ r t Z ) + 2 λ σ t τ t w t − 1 = 1 + 2 λ σ t τ t σ t τ t ( θ t w ⋆ + δ r t Z + 2 λ w t − 1 ) .
Then
ν 2 α t = E [ w ⋆ σ t τ t ( θ t w ⋆ + r t δ Z ) + 2 λ σ t τ t w t − 1 1 + 2 λ σ t τ t ] , γ t = 1 r t δ E [ Z σ t τ t ( θ t w ⋆ + r t δ Z ) + 2 λ σ t τ t w t − 1 1 + 2 λ σ t τ t ] , ν 2 α t 2 + σ t 2 = E [ ( σ t τ t ( θ t w ⋆ + r t δ Z ) + 2 λ σ t τ t w t − 1 1 + 2 λ σ t τ t ) 2 ] . \begin{aligned}
\nu^2\alpha_t&=\mathbb{E}\left[w^\star\frac{\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z\right)+2\lambda\sigma_t\tau_t w_{t-1}}{1+2\lambda\sigma_t\tau_t}\right],\\
\gamma_t&=\frac{1}{r_t\sqrt{\delta}}\mathbb{E}\left[Z\frac{\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z\right)+2\lambda\sigma_t\tau_t w_{t-1}}{1+2\lambda\sigma_t\tau_t}\right],\\
\nu^2\alpha_t^2+\sigma_t^2&=\mathbb{E}\left[\left(\frac{\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z\right)+2\lambda\sigma_t\tau_t w_{t-1}}{1+2\lambda\sigma_t\tau_t}\right)^2\right].
\end{aligned} ν 2 α t γ t ν 2 α t 2 + σ t 2 = E w ⋆ 1 + 2 λ σ t τ t σ t τ t ( θ t w ⋆ + δ r t Z ) + 2 λ σ t τ t w t − 1 , = r t δ 1 E Z 1 + 2 λ σ t τ t σ t τ t ( θ t w ⋆ + δ r t Z ) + 2 λ σ t τ t w t − 1 , = E 1 + 2 λ σ t τ t σ t τ t ( θ t w ⋆ + δ r t Z ) + 2 λ σ t τ t w t − 1 2 .
Using E [ Z ] = 0 \mathbb{E}[Z]=0 E [ Z ] = 0 , E [ Z 2 ] = 1 \mathbb{E}[Z^2]=1 E [ Z 2 ] = 1 , and the independence of Z Z Z from w ⋆ , w t − 1 w^\star,w_{t-1} w ⋆ , w t − 1 , this simplifies to
ν 2 α t = σ t τ t 1 + 2 λ σ t τ t ( ν 2 θ t + 2 λ E [ w ⋆ w t − 1 ] ) , γ t = σ t τ t δ ( 1 + 2 λ σ t τ t ) , ν 2 α t 2 + σ t 2 = ( σ t τ t 1 + 2 λ σ t τ t ) 2 ( ν 2 θ t 2 + 4 λ θ t E [ w ⋆ w t − 1 ] + 4 λ 2 E [ w t − 1 2 ] + r t 2 δ ) . \begin{aligned}
\nu^2\alpha_t&=\frac{\sigma_t\tau_t}{1+2\lambda\sigma_t\tau_t}\left(\nu^2\theta_t+2\lambda\mathbb{E}[w^\star w_{t-1}]\right),\\
\gamma_t&=\frac{\sigma_t\tau_t}{\delta(1+2\lambda\sigma_t\tau_t)},\\
\nu^2\alpha_t^2+\sigma_t^2&=\left(\frac{\sigma_t\tau_t}{1+2\lambda\sigma_t\tau_t}\right)^2\left(\nu^2\theta_t^2+4\lambda\theta_t\mathbb{E}[w^\star w_{t-1}]+4\lambda^2\mathbb{E}[w_{t-1}^2]+\frac{r_t^2}{\delta}\right).
\end{aligned} ν 2 α t γ t ν 2 α t 2 + σ t 2 = 1 + 2 λ σ t τ t σ t τ t ( ν 2 θ t + 2 λ E [ w ⋆ w t − 1 ] ) , = δ ( 1 + 2 λ σ t τ t ) σ t τ t , = ( 1 + 2 λ σ t τ t σ t τ t ) 2 ( ν 2 θ t 2 + 4 λ θ t E [ w ⋆ w t − 1 ] + 4 λ 2 E [ w t − 1 2 ] + δ r t 2 ) .
Since σ t τ t 1 + 2 λ σ t τ t = δ γ t \frac{\sigma_t\tau_t}{1+2\lambda\sigma_t\tau_t}=\delta\gamma_t 1 + 2 λ σ t τ t σ t τ t = δ γ t , the first identity becomes
α t = δ γ t θ t + 2 λ δ γ t E [ w ⋆ w t − 1 ] ν 2 . \alpha_t=\delta\gamma_t\theta_t+2\lambda\delta\gamma_t\frac{\mathbb{E}[w^\star w_{t-1}]}{\nu^2}. α t = δ γ t θ t + 2 λ δ γ t ν 2 E [ w ⋆ w t − 1 ] .
Using the proximal identity and the claim in Theorem 1,
w t → d Prox λ σ t τ t f ~ ( σ t τ t ( θ t w ⋆ + r t δ Z t ) ) = σ t τ t 1 + 2 λ σ t τ t ( θ t w ⋆ + r t δ Z t + 2 λ w t − 1 ) . w_t\xrightarrow{d}\operatorname{Prox}_{\lambda\sigma_t\tau_t\tilde{f}}\left(\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z_t\right)\right)=\frac{\sigma_t\tau_t}{1+2\lambda\sigma_t\tau_t}\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z_t+2\lambda w_{t-1}\right). w t d Prox λ σ t τ t f ~ ( σ t τ t ( θ t w ⋆ + δ r t Z t ) ) = 1 + 2 λ σ t τ t σ t τ t ( θ t w ⋆ + δ r t Z t + 2 λ w t − 1 ) .
For notational convenience, we use = = = to denote the deterministic values to which the random quantities converge as n , d → ∞ n,d \to\infty n , d → ∞ and n / d → δ n/d \to\delta n / d → δ . Multiplying with w ⋆ w^\star w ⋆ of previous equation and taking expectation,
E [ w t w ⋆ ] = δ γ t ( θ t E [ w ⋆ 2 ] + 2 λ E [ w t − 1 w ⋆ ] ) = δ γ t ( θ t ν 2 + 2 λ E [ w t − 1 w ⋆ ] ) = ν 2 α t . \mathbb{E}[w_t w^\star] = \delta\gamma_t\left(\theta_t\mathbb{E}[w^{\star2}] + 2\lambda\mathbb{E}[w_{t-1}w^\star]\right) = \delta\gamma_t\left(\theta_t\nu^2 + 2\lambda\mathbb{E}[w_{t-1}w^\star]\right) = \nu^2\alpha_t. E [ w t w ⋆ ] = δ γ t ( θ t E [ w ⋆ 2 ] + 2 λ E [ w t − 1 w ⋆ ] ) = δ γ t ( θ t ν 2 + 2 λ E [ w t − 1 w ⋆ ] ) = ν 2 α t .
hence α t = 2 λ δ γ t α t − 1 + δ γ t θ t \alpha_t = 2\lambda\delta\gamma_t\alpha_{t-1} + \delta\gamma_t\theta_t α t = 2 λ δ γ t α t − 1 + δ γ t θ t . Likewise, subtracting ν 2 α t 2 = ν 2 ( 2 λ δ γ t α t − 1 + δ γ t θ t ) 2 \nu^2\alpha_t^2 = \nu^2(2\lambda\delta\gamma_t\alpha_{t-1} + \delta\gamma_t\theta_t)^2 ν 2 α t 2 = ν 2 ( 2 λ δ γ t α t − 1 + δ γ t θ t ) 2 , the third equation yields
σ t 2 = ( 2 λ δ γ t ) 2 σ t − 1 2 + δ γ t 2 r t 2 . \sigma_t^2 = (2\lambda\delta\gamma_t)^2\sigma_{t-1}^2 + \delta\gamma_t^2r_t^2. σ t 2 = ( 2 λ δ γ t ) 2 σ t − 1 2 + δ γ t 2 r t 2 .
Then
w t → d 2 λ δ γ t w t − 1 + δ γ t θ t w ⋆ + δ γ t r t Z t , w_t \xrightarrow{d} 2\lambda\delta\gamma_t w_{t-1} + \delta\gamma_t\theta_t w^\star+ \sqrt{\delta}\gamma_t r_t Z_t, w t d 2 λ δ γ t w t − 1 + δ γ t θ t w ⋆ + δ γ t r t Z t ,
with Z t ∼ i i d N ( 0 , 1 ) Z_t \overset{\mathrm{iid}}{\sim} N(0,1) Z t ∼ iid N ( 0 , 1 ) , and therefore
α t = 2 λ δ γ t α t − 1 + δ γ t θ t , σ t 2 = ( 2 λ δ γ t ) 2 σ t − 1 2 + δ γ t 2 r t 2 . \alpha_t = 2\lambda\delta\gamma_t\alpha_{t-1} + \delta\gamma_t\theta_t,\qquad\sigma_t^2 = (2\lambda\delta\gamma_t)^2\sigma_{t-1}^2 + \delta\gamma_t^2r_t^2. α t = 2 λ δ γ t α t − 1 + δ γ t θ t , σ t 2 = ( 2 λ δ γ t ) 2 σ t − 1 2 + δ γ t 2 r t 2 .
Iterating back to w 0 = 0 w_0=0 w 0 = 0 ,
w T → d ∑ s = 1 T ( ∏ j = s + 1 T 2 λ δ γ j ) ( δ γ s θ s w ⋆ + δ γ s r s Z s ) . w_T \xrightarrow{d} \sum_{s=1}^{T}\left(\prod_{j=s+1}^{T}2\lambda\delta\gamma_j\right)\left(\delta\gamma_s\theta_s w^\star+\sqrt{\delta}\gamma_s r_s Z_s\right). w T d s = 1 ∑ T ( j = s + 1 ∏ T 2 λ δ γ j ) ( δ γ s θ s w ⋆ + δ γ s r s Z s ) .
Hence
α T = ∑ s = 1 T ( ∏ j = s + 1 T 2 λ δ γ j ) δ γ s θ s , σ T 2 = ∑ s = 1 T ( ∏ j = s + 1 T 2 λ δ γ j ) 2 δ γ s 2 r s 2 . (36) \alpha_T = \sum_{s=1}^{T}\left(\prod_{j=s+1}^{T}2\lambda\delta\gamma_j\right)\delta\gamma_s\theta_s,\qquad\sigma_T^2 = \sum_{s=1}^{T}\left(\prod_{j=s+1}^{T}2\lambda\delta\gamma_j\right)^2\delta\gamma_s^2r_s^2. \tag*{(36)} α T = s = 1 ∑ T ( j = s + 1 ∏ T 2 λ δ γ j ) δ γ s θ s , σ T 2 = s = 1 ∑ T ( j = s + 1 ∏ T 2 λ δ γ j ) 2 δ γ s 2 r s 2 . (36)
From the sixth saddle equation,
1 − γ t σ t τ t = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . 1-\frac{\gamma_t}{\sigma_t\tau_t}=\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))}{1+\gamma_t\phi^{\prime\prime}(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}\right]. 1 − σ t τ t γ t = E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ] .
Since γ t = σ t τ t δ ( 1 + 2 λ σ t τ t ) \gamma_t=\frac{\sigma_t\tau_t}{\delta(1+2\lambda\sigma_t\tau_t)} γ t = δ ( 1 + 2 λ σ t τ t ) σ t τ t , we have 2 λ δ γ t = 2 λ σ t τ t 1 + 2 λ σ t τ t 2\lambda\delta\gamma_t=\frac{2\lambda\sigma_t\tau_t}{1+2\lambda\sigma_t\tau_t} 2 λ δ γ t = 1 + 2 λ σ t τ t 2 λ σ t τ t , and hence
γ t σ t τ t = 1 δ ( 1 + 2 λ σ t τ t ) = 1 − 2 λ δ γ t δ . \frac{\gamma_t}{\sigma_t\tau_t}=\frac{1}{\delta(1+2\lambda\sigma_t\tau_t)}=\frac{1-2\lambda\delta\gamma_t}{\delta}. σ t τ t γ t = δ ( 1 + 2 λ σ t τ t ) 1 = δ 1 − 2 λ δ γ t .
Thus the sixth saddle equation gives the exact identity
2 λ δ γ t = 1 − δ + δ E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . 2\lambda\delta\gamma_t=1-\delta+\delta\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))}{1+\gamma_t\phi^{\prime\prime}(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}\right]. 2 λ δ γ t = 1 − δ + δ E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ] .
For the bit-flip logistic model, π ρ ( u ) = ρ + ( 1 − 2 ρ ) σ ( u ) \pi_\rho(u)=\rho+(1-2\rho)\sigma(u) π ρ ( u ) = ρ + ( 1 − 2 ρ ) σ ( u ) . By Gaussian symmetry and σ ( − x ) = 1 − σ ( x ) \sigma(-x)=1-\sigma(x) σ ( − x ) = 1 − σ ( x ) , E [ σ ( ν Z 1 ) ] = 1 2 \mathbb{E}[\sigma(\nu Z_1)]=\frac{1}{2} E [ σ ( ν Z 1 )] = 2 1 , hence E [ 1 − π ρ ( ν Z 1 ) ] = 1 2 \mathbb{E}[1-\pi_\rho(\nu Z_1)]=\frac{1}{2} E [ 1 − π ρ ( ν Z 1 )] = 2 1 . Since ϕ ′ ′ ≤ 1 4 \phi^{\prime\prime}\leq\frac{1}{4} ϕ ′′ ≤ 4 1 , γ t > 0 \gamma_t>0 γ t > 0 and 2 E [ 1 − π ρ ( ν Z 1 ) ] = 1 2\mathbb{E}[1-\pi_\rho(\nu Z_1)]=1 2 E [ 1 − π ρ ( ν Z 1 )] = 1 ,
∣ 1 − E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ∣ = ∣ γ t E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ∣ ≤ γ t 4 . \left|1-\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))}{1+\gamma_t\phi^{\prime\prime}(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}\right]\right|
=\left|\gamma_t\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))\phi^{\prime\prime}(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}{1+\gamma_t\phi^{\prime\prime}(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}\right]\right|
\leq\frac{\gamma_t}{4}. 1 − E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ] = γ t E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] ≤ 4 γ t .
Therefore, 2 λ δ γ t = 1 + O ( γ t ) 2\lambda\delta\gamma_t=1+O(\gamma_t) 2 λ δ γ t = 1 + O ( γ t ) . Using Prox γ t ϕ ( x ) = x − γ t ϕ ′ ( x ) + O ( γ t 2 ) \operatorname{Prox}_{\gamma_t\phi}(x)=x-\gamma_t\phi^\prime(x)+O(\gamma_t^2) Prox γ t ϕ ( x ) = x − γ t ϕ ′ ( x ) + O ( γ t 2 ) , and ϕ ′ ′ ( Prox γ t ϕ ( x ) ) = ϕ ′ ′ ( x ) + O ( γ t ) \phi^{\prime\prime}(\operatorname{Prox}_{\gamma_t\phi}(x))=\phi^{\prime\prime}(x)+O(\gamma_t) ϕ ′′ ( Prox γ t ϕ ( x )) = ϕ ′′ ( x ) + O ( γ t ) , we get
1 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) = 1 − γ t ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) + O ( γ t 2 ) . \frac{1}{1+\gamma_t\phi^{\prime\prime}(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}
=1-\gamma_t\phi^{\prime\prime}(\nu\alpha_tZ_1+\sigma_tZ_2)+O(\gamma_t^2). 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 1 = 1 − γ t ϕ ′′ ( ν α t Z 1 + σ t Z 2 ) + O ( γ t 2 ) .
Hence
2 λ δ γ t = 1 − δ + δ [ 2 E [ 1 − π ρ ( ν Z 1 ) ] − 2 γ t E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] + O ( γ t 2 ) ] = 1 − 2 δ γ t E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] + O ( γ t 2 ) . \begin{aligned}
2\lambda\delta\gamma_t=1-\delta+\delta\left[2\mathbb{E}[1-\pi_\rho(\nu Z_1)]-2\gamma_t\mathbb{E}[(1-\pi_\rho(\nu Z_1))\phi^{\prime\prime}(\nu\alpha_tZ_1+\sigma_tZ_2)]+O(\gamma_t^2)\right] \\
&=1-2\delta\gamma_t\mathbb{E}[(1-\pi_\rho(\nu Z_1))\phi^{\prime\prime}(\nu\alpha_tZ_1+\sigma_tZ_2)]+O(\gamma_t^2).
\end{aligned} 2 λ δ γ t = 1 − δ + δ [ 2 E [ 1 − π ρ ( ν Z 1 )] − 2 γ t E [( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( ν α t Z 1 + σ t Z 2 )] + O ( γ t 2 ) ] = 1 − 2 δ γ t E [( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( ν α t Z 1 + σ t Z 2 )] + O ( γ t 2 ) .
Since γ t = 1 2 δ λ + O ( λ − 2 ) \gamma_t=\frac{1}{2\delta\lambda}+O(\lambda^{-2}) γ t = 2 δ λ 1 + O ( λ − 2 ) , then 2 λ δ γ t = 1 − A t λ + O ( λ − 2 ) 2\lambda\delta\gamma_t=1-\frac{A_t}{\lambda}+O(\lambda^{-2}) 2 λ δ γ t = 1 − λ A t + O ( λ − 2 ) , where A t = E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] A_t=\mathbb{E}[(1-\pi_\rho(\nu Z_1))\phi^{\prime\prime}(\nu\alpha_tZ_1+\sigma_tZ_2)] A t = E [( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( ν α t Z 1 + σ t Z 2 )] . For the softplus function, ϕ ′ ′ ( x ) = e x ( 1 + e x ) 2 \phi^{\prime\prime}(x)=\frac{e^x}{(1+e^x)^2} ϕ ′′ ( x ) = ( 1 + e x ) 2 e x . Writing s ( x ) = σ ( x ) ( 1 − σ ( x ) ) s(x)=\sigma(x)(1-\sigma(x)) s ( x ) = σ ( x ) ( 1 − σ ( x )) , we have ϕ ′ ′ ( x ) = s ( x ) \phi^{\prime\prime}(x)=s(x) ϕ ′′ ( x ) = s ( x ) , so A t = A_t= A t = E [ ( 1 − π ρ ( ν Z 1 ) ) s ( ν α t Z 1 + σ t Z 2 ) ] \mathbb{E}\left[(1-\pi_\rho(\nu Z_1))s(\nu\alpha_t Z_1+\sigma_t Z_2)\right] E [ ( 1 − π ρ ( ν Z 1 )) s ( ν α t Z 1 + σ t Z 2 ) ] . By Lemma 12, we obtain that A t = 1 2 h ( ν 2 α t 2 + σ t 2 ) A_t=\frac{1}{2}h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right) A t = 2 1 h ( ν 2 α t 2 + σ t 2 ) , with h ( q ) = E [ s ( q Z ) ] h(q)=\mathbb{E}[s(qZ)] h ( q ) = E [ s ( q Z )] .
Next, using the fifth saddle equation, we have
δ γ t θ t = − 2 δ E [ π ρ ′ ( ν Z 1 ) Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ] . \delta\gamma_t\theta_t=-2\delta\mathbb{E}\left[\pi_\rho'(\nu Z_1)\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right]. δ γ t θ t = − 2 δ E [ π ρ ′ ( ν Z 1 ) Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ] .
Expanding the proximal map,
δ γ t θ t = − 2 δ E [ π ρ ′ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 ) ] + 2 δ γ t E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] + O ( γ t 2 ) . \delta\gamma_t\theta_t=-2\delta\mathbb{E}\left[\pi_\rho'(\nu Z_1)(\nu\alpha_t Z_1+\sigma_t Z_2)\right]+2\delta\gamma_t\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)\right]+O(\gamma_t^2). δ γ t θ t = − 2 δ E [ π ρ ′ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 ) ] + 2 δ γ t E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] + O ( γ t 2 ) .
Now π ρ ′ ( ν Z 1 ) \pi_\rho'(\nu Z_1) π ρ ′ ( ν Z 1 ) is even in Z 1 Z_1 Z 1 , while ν α t Z 1 + σ t Z 2 \nu\alpha_t Z_1+\sigma_t Z_2 ν α t Z 1 + σ t Z 2 is odd in Z 1 Z_1 Z 1 . Therefore E [ π ρ ′ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 ) ] = 0 \mathbb{E}[\pi_\rho'(\nu Z_1)(\nu\alpha_t Z_1+\sigma_t Z_2)]=0 E [ π ρ ′ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 )] = 0 . Consequently,
δ γ t θ t = 2 δ γ t E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] + O ( γ t 2 ) . \delta\gamma_t\theta_t=2\delta\gamma_t\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)\right]+O(\gamma_t^2). δ γ t θ t = 2 δ γ t E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] + O ( γ t 2 ) .
Using again γ t = 1 2 λ δ + O ( λ − 2 ) \gamma_t=\frac{1}{2\lambda\delta}+O(\lambda^{-2}) γ t = 2 λ δ 1 + O ( λ − 2 ) , we get δ γ t θ t = B t λ + O ( λ − 2 ) \delta\gamma_t\theta_t=\frac{B_t}{\lambda}+O(\lambda^{-2}) δ γ t θ t = λ B t + O ( λ − 2 ) , where B t = E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] B_t=\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)\right] B t = E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] . By Lemma 12 we have that, B t ≡ B B_t\equiv B B t ≡ B is constant, with B = 1 − 2 ρ 2 h ( ν ) > 0 B=\frac{1-2\rho}{2}h(\nu)>0 B = 2 1 − 2 ρ h ( ν ) > 0 . From the fourth saddle equation,
γ t 2 = 2 r t 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ( ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] . \gamma_t^2=\frac{2}{r_t^2}\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))(\nu\alpha_t Z_1+\sigma_t Z_2-\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))^2\right]. γ t 2 = r t 2 2 E [ ( 1 − π ρ ( ν Z 1 )) ( ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] .
Using
ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) = γ t ϕ ′ ( ν α t Z 1 + σ t Z 2 ) + O ( γ t 2 ) , \nu\alpha_t Z_1+\sigma_t Z_2-\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)=\gamma_t\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)+O(\gamma_t^2), ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) = γ t ϕ ′ ( ν α t Z 1 + σ t Z 2 ) + O ( γ t 2 ) ,
we obtain
γ t 2 = 2 γ t 2 r t 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] + O ( γ t 3 ) . \gamma_t^2=\frac{2\gamma_t^2}{r_t^2}\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)^2\right]+O(\gamma_t^3). γ t 2 = r t 2 2 γ t 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] + O ( γ t 3 ) .
Thus
r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] + O ( λ − 1 ) . r_t^2=2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)^2\right]+O(\lambda^{-1}). r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] + O ( λ − 1 ) .
Therefore δ γ t 2 r t 2 = V t λ 2 + O ( λ − 3 ) \delta\gamma_t^2r_t^2=\frac{V_t}{\lambda^2}+O(\lambda^{-3}) δ γ t 2 r t 2 = λ 2 V t + O ( λ − 3 ) , where
V t = 1 2 δ E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] . V_t=\frac{1}{2\delta}\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)^2\right]. V t = 2 δ 1 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] .
Overall, 2 λ δ γ t = 1 − A t λ + O ( λ − 2 ) 2\lambda\delta\gamma_t=1-\frac{A_t}{\lambda}+O(\lambda^{-2}) 2 λ δ γ t = 1 − λ A t + O ( λ − 2 ) , δ γ t θ t = B t λ + O ( λ − 2 ) \delta\gamma_t\theta_t=\frac{B_t}{\lambda}+O(\lambda^{-2}) δ γ t θ t = λ B t + O ( λ − 2 ) , and δ γ t 2 r t 2 = V t λ 2 + O ( λ − 3 ) \delta\gamma_t^2r_t^2=\frac{V_t}{\lambda^2}+O(\lambda^{-3}) δ γ t 2 r t 2 = λ 2 V t + O ( λ − 3 ) , with A t = 1 2 h ( ν 2 α t 2 + σ t 2 ) A_t=\frac{1}{2}h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right) A t = 2 1 h ( ν 2 α t 2 + σ t 2 ) , B t = E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] B_t=\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)\right] B t = E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] , and V t = 1 2 δ E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] V_t=\frac{1}{2\delta}\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)^2\right] V t = 2 δ 1 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] . Hence,
α t = ( 1 − A t λ ) α t − 1 + B t λ + O ( λ − 2 ) , \alpha_t=\left(1-\frac{A_t}{\lambda}\right)\alpha_{t-1}+\frac{B_t}{\lambda}+O(\lambda^{-2}), α t = ( 1 − λ A t ) α t − 1 + λ B t + O ( λ − 2 ) ,
σ t 2 = ( 1 − 2 A t λ ) σ t − 1 2 + V t λ 2 + O ( λ − 3 ) , (37) \sigma_t^2=\left(1-\frac{2A_t}{\lambda}\right)\sigma_{t-1}^2+\frac{V_t}{\lambda^2}+O(\lambda^{-3}), \tag*{(37)} σ t 2 = ( 1 − λ 2 A t ) σ t − 1 2 + λ 2 V t + O ( λ − 3 ) , (37)
the last equation follows by Eq. (38) showing that σ t 2 = O ( λ − 1 ) \sigma_t^2=O(\lambda^{-1}) σ t 2 = O ( λ − 1 ) . Since ϕ ′ ( x ) = e x 1 + e x \phi'(x)=\frac{e^x}{1+e^x} ϕ ′ ( x ) = 1 + e x e x , ϕ ′ ′ ( x ) = e x ( 1 + e x ) 2 \phi''(x)=\frac{e^x}{(1+e^x)^2} ϕ ′′ ( x ) = ( 1 + e x ) 2 e x . Therefore ∣ ϕ ′ ( x ) ∣ < 1 |\phi'(x)|<1 ∣ ϕ ′ ( x ) ∣ < 1 , and ∣ ϕ ′ ′ ( x ) ∣ ≤ 1 4 |\phi''(x)|\leq\frac{1}{4} ∣ ϕ ′′ ( x ) ∣ ≤ 4 1 for all x ∈ R x\in\mathbb{R} x ∈ R , so A t , B t , V t A_t,B_t,V_t A t , B t , V t are uniformly bounded by 1 4 \frac{1}{4} 4 1 , 1 − 2 ρ 4 \frac{1-2\rho}{4} 4 1 − 2 ρ , and 1 2 δ \frac{1}{2\delta} 2 δ 1 , respectively.
Auxiliary Lemmas
For this section we introduce the following notations,
σ ( x ) = 1 1 + e − x , s ( x ) = σ ( x ) ( 1 − σ ( x ) ) = ϕ ′ ′ ( x ) , π ρ ( x ) = ρ + ( 1 − 2 ρ ) σ ( x ) , \sigma(x)=\frac{1}{1+e^{-x}},\qquad s(x)=\sigma(x)(1-\sigma(x))=\phi''(x),\qquad\pi_\rho(x)=\rho+(1-2\rho)\sigma(x), σ ( x ) = 1 + e − x 1 , s ( x ) = σ ( x ) ( 1 − σ ( x )) = ϕ ′′ ( x ) , π ρ ( x ) = ρ + ( 1 − 2 ρ ) σ ( x ) ,
where 0 ≤ ρ < 1 2 0\leq\rho<\frac{1}{2} 0 ≤ ρ < 2 1 , and σ \sigma σ denotes the sigmoid function. Let the population logistic risk be
L T ( λ ) = E [ log ( 1 + e ν α t Z 1 + σ t Z 2 ) − π ρ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 ) ] . L_T(\lambda)=\mathbb{E}\left[\log(1+e^{\nu\alpha_t Z_1+\sigma_t Z_2})-\pi_\rho(\nu Z_1)(\nu\alpha_t Z_1+\sigma_t Z_2)\right]. L T ( λ ) = E [ log ( 1 + e ν α t Z 1 + σ t Z 2 ) − π ρ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 ) ] .
This definition follows because x T w ⋆ → d ν Z 1 \mathbf{x}^{\mathsf T}\mathbf{w}_\star\xrightarrow{d}\nu Z_1 x T w ⋆ d ν Z 1 , if we write w t = α t w ⋆ + ( w t − α t w ⋆ ) \mathbf{w}_t=\alpha_t\mathbf{w}_\star+(\mathbf{w}_t-\alpha_t\mathbf{w}_\star) w t = α t w ⋆ + ( w t − α t w ⋆ ) with α t = w t T w ⋆ ∥ w ⋆ ∥ 2 \alpha_t=\frac{\mathbf{w}_t^{\mathsf T}\mathbf{w}_\star}{\lVert\mathbf{w}_\star\rVert^2} α t = ∥ w ⋆ ∥ 2 w t T w ⋆ then x T w t → d ν α t Z 1 + σ t Z 2 \mathbf{x}^{\mathsf T}\mathbf{w}_t\xrightarrow{d}\nu\alpha_t Z_1+\sigma_t Z_2 x T w t d ν α t Z 1 + σ t Z 2 . Define the noiseless risk (where σ t = 0 \sigma_t=0 σ t = 0 ) as,
L 0 ( a ) = E Z [ log ( 1 + e ν a Z ) − π ρ ( ν Z ) ν a Z ] , L_0(a)=\mathbb{E}_Z\left[\log(1+e^{\nu aZ})-\pi_\rho(\nu Z)\nu aZ\right], L 0 ( a ) = E Z [ log ( 1 + e ν a Z ) − π ρ ( ν Z ) ν a Z ] ,
and let α ⋆ = argmin a ≥ 0 L 0 ( a ) \alpha^\star= \operatorname*{argmin}_{a\geq0}\mathcal{L}_0(a) α ⋆ = argmin a ≥ 0 L 0 ( a ) . We introduce few other notations, gathered from the introduction,
h ( q ) = ∫ − ∞ ∞ s ( q z ) φ ( z ) d z , φ ( z ) = 1 2 π e − z 2 2 , A t = 1 2 h ( ν 2 α t 2 + σ t 2 ) , B t = B = 1 − 2 ρ 2 h ( ν ) , r = B A , h(q)=\int_{-\infty}^{\infty}s(qz)\varphi(z)\,\mathrm{d}z,\qquad\varphi(z)=\frac{1}{\sqrt{2\pi}}e^{-\frac{z^2}{2}},\qquad A_t=\frac{1}{2}h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right),\qquad B_t=B=\frac{1-2\rho}{2}h(\nu),\qquad r=\frac{B}{A}, h ( q ) = ∫ − ∞ ∞ s ( q z ) φ ( z ) d z , φ ( z ) = 2 π 1 e − 2 z 2 , A t = 2 1 h ( ν 2 α t 2 + σ t 2 ) , B t = B = 2 1 − 2 ρ h ( ν ) , r = A B ,
A ( a ) = 1 2 h ( ν a ) , A t → t → ∞ A = 1 2 h ( ν r ) , V t = 1 2 δ E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] . A(a)=\frac{1}{2}h(\nu a),\qquad A_t\xrightarrow{t\to\infty}A=\frac{1}{2}h(\nu r),\qquad V_t=\frac{1}{2\delta}\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)^2\right]. A ( a ) = 2 1 h ( ν a ) , A t t → ∞ A = 2 1 h ( ν r ) , V t = 2 δ 1 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] .
Few of the identities provided here are proved in the following Lemmas.
Lemma 12 (Symmetry identities). We have A t = 1 2 h ( ν 2 α t 2 + σ t 2 ) A_t=\frac{1}{2}h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right) A t = 2 1 h ( ν 2 α t 2 + σ t 2 ) , and B t = 1 − 2 ρ 2 h ( ν ) B_t=\frac{1-2\rho}{2}h(\nu) B t = 2 1 − 2 ρ h ( ν ) . In particular B t ≡ B B_t\equiv B B t ≡ B is constant.
Proof. By definition, A t = E [ ( 1 − π ρ ( ν Z 1 ) ) s ( ν α t Z 1 + σ t Z 2 ) ] A_t=\mathbb{E}[(1-\pi_\rho(\nu Z_1))s(\nu\alpha_tZ_1+\sigma_tZ_2)] A t = E [( 1 − π ρ ( ν Z 1 )) s ( ν α t Z 1 + σ t Z 2 )] . Using the symmetry ( Z 1 , Z 2 ) = d ( − Z 1 , − Z 2 ) (Z_1,Z_2)\mathrel{\mathop{\stackrel{d}{=}}}(-Z_1,-Z_2) ( Z 1 , Z 2 ) = d ( − Z 1 , − Z 2 ) , together with s ( − u ) = s ( u ) s(-u)=s(u) s ( − u ) = s ( u ) , and π ρ ( u ) + π ρ ( − u ) = 1 \pi_\rho(u)+\pi_\rho(-u)=1 π ρ ( u ) + π ρ ( − u ) = 1 , we get
E [ ( 1 − π ρ ( ν Z 1 ) ) s ( ν α t Z 1 + σ t Z 2 ) ] = E [ π ρ ( ν Z 1 ) s ( ν α t Z 1 + σ t Z 2 ) ] . \mathbb{E}[(1-\pi_\rho(\nu Z_1))s(\nu\alpha_tZ_1+\sigma_tZ_2)]=\mathbb{E}[\pi_\rho(\nu Z_1)s(\nu\alpha_tZ_1+\sigma_tZ_2)]. E [( 1 − π ρ ( ν Z 1 )) s ( ν α t Z 1 + σ t Z 2 )] = E [ π ρ ( ν Z 1 ) s ( ν α t Z 1 + σ t Z 2 )] .
Adding both identities gives
2 E [ ( 1 − π ρ ( ν Z 1 ) ) s ( ν α t Z 1 + σ t Z 2 ) ] = E [ s ( ν α t Z 1 + σ t Z 2 ) ] . 2\mathbb{E}[(1-\pi_\rho(\nu Z_1))s(\nu\alpha_tZ_1+\sigma_tZ_2)]=\mathbb{E}[s(\nu\alpha_tZ_1+\sigma_tZ_2)]. 2 E [( 1 − π ρ ( ν Z 1 )) s ( ν α t Z 1 + σ t Z 2 )] = E [ s ( ν α t Z 1 + σ t Z 2 )] .
Therefore A t = 1 2 E [ s ( ν α t Z 1 + σ t Z 2 ) ] A_t=\frac{1}{2}\mathbb{E}[s(\nu\alpha_tZ_1+\sigma_tZ_2)] A t = 2 1 E [ s ( ν α t Z 1 + σ t Z 2 )] . Since ν α t Z 1 + σ t Z 2 ∼ N ( 0 , ν 2 α t 2 + σ t 2 ) \nu\alpha_tZ_1+\sigma_tZ_2\sim N(0,\nu^2\alpha_t^2+\sigma_t^2) ν α t Z 1 + σ t Z 2 ∼ N ( 0 , ν 2 α t 2 + σ t 2 ) , we get A t = 1 2 h ( ν 2 α t 2 + σ t 2 ) A_t=\frac{1}{2}h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right) A t = 2 1 h ( ν 2 α t 2 + σ t 2 ) . For B t B_t B t , using π ρ ′ ( u ) = ( 1 − 2 ρ ) s ( u ) \pi_\rho'(u)=(1-2\rho)s(u) π ρ ′ ( u ) = ( 1 − 2 ρ ) s ( u ) , we have B t = ( 1 − 2 ρ ) E [ s ( ν Z 1 ) σ ( ν α t Z 1 + σ t Z 2 ) ] B_t=(1-2\rho)\mathbb{E}[s(\nu Z_1)\sigma(\nu\alpha_tZ_1+\sigma_tZ_2)] B t = ( 1 − 2 ρ ) E [ s ( ν Z 1 ) σ ( ν α t Z 1 + σ t Z 2 )] from symmetry, let I t = E [ s ( ν Z 1 ) σ ( ν α t Z 1 + σ t Z 2 ) ] = E [ s ( − ν Z 1 ) σ ( − ( ν α t Z 1 + σ t Z 2 ) ) ] I_t=\mathbb{E}[s(\nu Z_1)\sigma(\nu\alpha_tZ_1+\sigma_tZ_2)]=\mathbb{E}[s(-\nu Z_1)\sigma(-(\nu\alpha_tZ_1+\sigma_tZ_2))] I t = E [ s ( ν Z 1 ) σ ( ν α t Z 1 + σ t Z 2 )] = E [ s ( − ν Z 1 ) σ ( − ( ν α t Z 1 + σ t Z 2 ))] . Since σ ( − ( ν α t Z 1 + σ t Z 2 ) ) = 1 − σ ( ν α t Z 1 + σ t Z 2 ) \sigma(-(\nu\alpha_tZ_1+\sigma_tZ_2))=1-\sigma(\nu\alpha_tZ_1+\sigma_tZ_2) σ ( − ( ν α t Z 1 + σ t Z 2 )) = 1 − σ ( ν α t Z 1 + σ t Z 2 ) , s ( ν Z 1 ) = s ( − ν Z 1 ) s(\nu Z_1)=s(-\nu Z_1) s ( ν Z 1 ) = s ( − ν Z 1 ) we have I t = E [ s ( ν Z 1 ) ( 1 − σ ( ν α t Z 1 + σ t Z 2 ) ) ] I_t=\mathbb{E}[s(\nu Z_1)(1-\sigma(\nu\alpha_tZ_1+\sigma_tZ_2))] I t = E [ s ( ν Z 1 ) ( 1 − σ ( ν α t Z 1 + σ t Z 2 ))] . Adding this to the definition of I t I_t I t , we have 2 I t = E [ s ( ν Z 1 ) ] = h ( ν ) 2I_t=\mathbb{E}[s(\nu Z_1)]=h(\nu) 2 I t = E [ s ( ν Z 1 )] = h ( ν ) . Therefore B t = 1 − 2 ρ 2 h ( ν ) B_t=\frac{1-2\rho}{2}h(\nu) B t = 2 1 − 2 ρ h ( ν ) .
Lemma 13 (Variance bounds)
For 1 ≲ A T λ = o ( T ) 1\lesssim\frac{AT}{\lambda}=o(T) 1 ≲ λ A T = o ( T ) , we have σ T 2 ≲ 1 λ \sigma_T^2\lesssim\frac{1}{\lambda} σ T 2 ≲ λ 1 .
Proof. Since A t → A > 0 A_t\to A>0 A t → A > 0 and V t → V > 0 V_t\to V>0 V t → V > 0 by Lemma 14, for all sufficiently large t t t ,5
0 < a − ≤ A t ≤ a + , 0 < v − ≤ V t ≤ v + . 0<a_-\leq A_t\leq a_+,\qquad0<v_-\leq V_t\leq v_+. 0 < a − ≤ A t ≤ a + , 0 < v − ≤ V t ≤ v + .
Using the developed recursion σ T \sigma_T σ T in Eq. (36) ,
σ T 2 = ∑ s = 1 T ( V s λ 2 + O ( λ − 3 ) ) ∏ j = s + 1 T ( 1 − 2 A j λ + O ( λ − 2 ) ) ≤ v − ≤ V t ≤ v + , a − ≤ A t ≤ a + C λ 2 ∑ s = 1 T exp ( − c T − s λ ) ≤ C λ , (38) \sigma_T^2=\sum_{s=1}^{T}\left(\frac{V_s}{\lambda^2}+O(\lambda^{-3})\right)\prod_{j=s+1}^{T}\left(1-\frac{2A_j}{\lambda}+O(\lambda^{-2})\right)\underset{v_-\leq V_t\leq v_+,\,a_-\leq A_t\leq a_+}{\leq}\frac{C}{\lambda^2}\sum_{s=1}^{T}\exp\left(-c\frac{T-s}{\lambda}\right)\leq\frac{C}{\lambda}, \tag*{(38)} σ T 2 = s = 1 ∑ T ( λ 2 V s + O ( λ − 3 ) ) j = s + 1 ∏ T ( 1 − λ 2 A j + O ( λ − 2 ) ) v − ≤ V t ≤ v + , a − ≤ A t ≤ a + ≤ λ 2 C s = 1 ∑ T exp ( − c λ T − s ) ≤ λ C , (38)
for some C , c > 0 C,c>0 C , c > 0 . To establish the lower bound, take only the last ⌊ η λ ⌋ \lfloor\eta\lambda\rfloor ⌊ η λ ⌋ terms. For these terms we have
∏ j = s + 1 T ( 1 − 2 A j λ ) ≥ ( 1 − 2 a + λ ) η λ ≥ c > 0. \prod_{j=s+1}^{T}\left(1-\frac{2A_j}{\lambda}\right)\geq\left(1-\frac{2a_+}{\lambda}\right)^{\eta\lambda}\geq c>0. j = s + 1 ∏ T ( 1 − λ 2 A j ) ≥ ( 1 − λ 2 a + ) η λ ≥ c > 0.
Hence
σ T 2 ≥ V t ≥ v − c ∑ s = T − ⌊ η λ ⌋ T 1 λ 2 ≥ c λ , \sigma_T^2\underset{V_t\geq v_-}{\geq}c\sum_{s=T-\lfloor\eta\lambda\rfloor}^{T}\frac{1}{\lambda^2}\geq\frac{c}{\lambda}, σ T 2 V t ≥ v − ≥ c s = T − ⌊ η λ ⌋ ∑ T λ 2 1 ≥ λ c ,
for some c > 0 c>0 c > 0 , the existence of uniform lower and upper bounds follows from compactness of the trajectory ( α t , σ t ) (\alpha_t,\sigma_t) ( α t , σ t ) , derived in Eq. (38) and Lemma 14.
Lemma 14 (Positivity of the limiting coefficients)
Assume λ → ∞ \lambda\to\infty λ → ∞ , T λ → τ ∈ [ 0 , ∞ ] \frac{T}{\lambda}\to\tau\in[0,\infty] λ T → τ ∈ [ 0 , ∞ ] . Let α ( τ ) \alpha(\tau) α ( τ ) be the solution of α ˙ = B − α A ( α ) \dot{\alpha}=B-\alpha A(\alpha) α ˙ = B − α A ( α ) , with α ( 0 ) = 0 \alpha(0)=0 α ( 0 ) = 0 , where A ( a ) = 1 2 h ( ν a ) A(a)=\frac{1}{2}h(\nu a) A ( a ) = 2 1 h ( ν a ) , B = 1 − 2 ρ 2 h ( ν ) B=\frac{1-2\rho}{2}h(\nu) B = 2 1 − 2 ρ h ( ν ) . If τ = ∞ \tau=\infty τ = ∞ , define α ( ∞ ) = r \alpha(\infty)=r α ( ∞ ) = r , where r r r is the unique fixed point B = r A ( r ) B=rA(r) B = r A ( r ) . Then A T → A τ = 1 2 h ( ν α ( τ ) ) > 0 A_T\to A_\tau=\frac{1}{2}h(\nu\alpha(\tau))>0 A T → A τ = 2 1 h ( ν α ( τ )) > 0 , and V T → V τ = 1 2 δ E [ ( 1 − π ρ ( ν Z ) ) ϕ ′ ( ν α ( τ ) Z ) 2 ] > 0 V_T\to V_\tau=\frac{1}{2\delta}\mathbb{E}[(1-\pi_\rho(\nu Z))\phi'(\nu\alpha(\tau)Z)^2]>0 V T → V τ = 2 δ 1 E [( 1 − π ρ ( ν Z )) ϕ ′ ( ν α ( τ ) Z ) 2 ] > 0 .
Proof. Recall that A t = 1 2 h ( ν 2 α t 2 + σ t 2 ) A_t = \frac{1}{2}h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right) A t = 2 1 h ( ν 2 α t 2 + σ t 2 ) , and the signal recursion (Eq. (37) ),
α t = ( 1 − A t λ ) α t − 1 + B λ + O ( λ − 2 ) , \alpha_t = \left(1-\frac{A_t}{\lambda}\right)\alpha_{t-1}+\frac{B}{\lambda}+O(\lambda^{-2}), α t = ( 1 − λ A t ) α t − 1 + λ B + O ( λ − 2 ) ,
with A t = 1 2 h ( ν 2 α t 2 + σ t 2 ) A_t = \frac{1}{2}h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right) A t = 2 1 h ( ν 2 α t 2 + σ t 2 ) , B = 1 − 2 ρ 2 h ( ν ) B=\frac{1-2\rho}{2}h(\nu) B = 2 1 − 2 ρ h ( ν ) . Hence
α t − α t − 1 = 1 λ ( B − A t α t − 1 ) + O ( λ − 2 ) . \alpha_t-\alpha_{t-1}=\frac{1}{\lambda}\left(B-A_t\alpha_{t-1}\right)+O(\lambda^{-2}). α t − α t − 1 = λ 1 ( B − A t α t − 1 ) + O ( λ − 2 ) .
Since σ t 2 = O ( λ − 1 ) \sigma_t^2=O(\lambda^{-1}) σ t 2 = O ( λ − 1 ) , we have A t = A ( α t − 1 ) + o ( 1 ) A_t=A(\alpha_{t-1})+o(1) A t = A ( α t − 1 ) + o ( 1 ) , with A ( a ) = 1 2 h ( ν a ) A(a)=\frac{1}{2}h(\nu a) A ( a ) = 2 1 h ( ν a ) , and therefore
α t − α t − 1 = 1 λ ( B − α t − 1 A ( α t − 1 ) ) + o ( λ − 1 ) . \alpha_t-\alpha_{t-1}=\frac{1}{\lambda}\left(B-\alpha_{t-1}A(\alpha_{t-1})\right)+o(\lambda^{-1}). α t − α t − 1 = λ 1 ( B − α t − 1 A ( α t − 1 ) ) + o ( λ − 1 ) .
Thus α t \alpha_t α t is the explicit Euler discretization, with step size h = 1 λ h=\frac{1}{\lambda} h = λ 1 of the ODE α ˙ = B − α A ( α ) \dot{\alpha}=B-\alpha A(\alpha) α ˙ = B − α A ( α ) , with α ( 0 ) = 0 \alpha(0)=0 α ( 0 ) = 0 . The corresponding ODE time is T h = T λ Th=\frac{T}{\lambda} T h = λ T . Therefore, if T λ → τ ∈ [ 0 , ∞ ] \frac{T}{\lambda}\to\tau\in[0,\infty] λ T → τ ∈ [ 0 , ∞ ] . Using Lemma 13 we have σ t 2 → 0 \sigma_t^2\to0 σ t 2 → 0 . Therefore ν 2 α t 2 + σ t 2 → ν 2 α ( τ ) 2 \nu^2\alpha_t^2+\sigma_t^2\to\nu^2\alpha(\tau)^2 ν 2 α t 2 + σ t 2 → ν 2 α ( τ ) 2 . By the continuity of h h h ,
A t = 1 2 h ( ν 2 α t 2 + σ t 2 ) → 1 2 h ( ν α ( τ ) ) = A τ . A_t=\frac{1}{2}h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)\to\frac{1}{2}h(\nu\alpha(\tau))=A_\tau. A t = 2 1 h ( ν 2 α t 2 + σ t 2 ) → 2 1 h ( ν α ( τ )) = A τ .
Since h ( q ) = E [ σ ( q Z ) ( 1 − σ ( q Z ) ) ] h(q)=\mathbb{E}[\sigma(qZ)(1-\sigma(qZ))] h ( q ) = E [ σ ( q Z ) ( 1 − σ ( q Z ))] and 0 < σ ( q Z ) ( 1 − σ ( q Z ) ) 0<\sigma(qZ)(1-\sigma(qZ)) 0 < σ ( q Z ) ( 1 − σ ( q Z )) almost surely for every q q q , we have A τ > 0 A_\tau>0 A τ > 0 . Next,
V t = 1 2 δ E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] . V_t=\frac{1}{2\delta}\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\nu\alpha_tZ_1+\sigma_tZ_2)^2\right]. V t = 2 δ 1 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) 2 ] .
Since α t → α ( τ ) \alpha_t\to\alpha(\tau) α t → α ( τ ) , with σ t → 0 \sigma_t\to0 σ t → 0 , and ϕ ′ \phi' ϕ ′ is bounded and continuous, dominated convergence gives
V t → 1 2 δ E [ ( 1 − π ρ ( ν Z ) ) ϕ ′ ( ν α ( τ ) Z ) 2 ] = V τ . V_t\to\frac{1}{2\delta}\mathbb{E}\left[(1-\pi_\rho(\nu Z))\phi'(\nu\alpha(\tau)Z)^2\right]=V_\tau. V t → 2 δ 1 E [ ( 1 − π ρ ( ν Z )) ϕ ′ ( ν α ( τ ) Z ) 2 ] = V τ .
Finally, 1 − π ρ ( ν Z ) > 0 1-\pi_\rho(\nu Z)>0 1 − π ρ ( ν Z ) > 0 almost surely, and ϕ ′ ( ν α ( τ ) Z ) 2 > 0 \phi'(\nu\alpha(\tau)Z)^2>0 ϕ ′ ( ν α ( τ ) Z ) 2 > 0 almost surely. Hence V τ > 0 V_\tau>0 V τ > 0 .
Lemma 15 (Monotonicity). The map q ↦ q h ( q ) q\mapsto qh(q) q ↦ q h ( q ) is strictly increasing on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) . Also, h ′ ( q ) < 0 h'(q)<0 h ′ ( q ) < 0 .
Proof. Changing variables u = q z u=qz u = q z ,
q h ( q ) = ∫ − ∞ ∞ s ( u ) φ ( u q ) d u . qh(q)=\int_{-\infty}^{\infty}s(u)\varphi\left(\frac{u}{q}\right)\,du. q h ( q ) = ∫ − ∞ ∞ s ( u ) φ ( q u ) d u . For fixed u ≠ 0 u\ne0 u = 0 , the map q ↦ φ ( u q ) q\mapsto\varphi\left(\frac{u}{q}\right) q ↦ φ ( q u ) is strictly increasing. Since s ( u ) > 0 s(u)>0 s ( u ) > 0 , q h ( q ) qh(q) q h ( q ) is strictly increasing. For the derivative, h ′ ( q ) = E [ Z s ′ ( q Z ) ] h'(q)=\mathbb{E}[Zs'(qZ)] h ′ ( q ) = E [ Z s ′ ( q Z )] , because s ′ ( x ) = − tanh ( x 2 ) s ( x ) s'(x)=-\tanh\left(\frac{x}{2}\right)s(x) s ′ ( x ) = − tanh ( 2 x ) s ( x ) , we get
Z s ′ ( q Z ) = − Z tanh ( q Z 2 ) s ( q Z ) ≤ 0 , Zs'(qZ)=-Z\tanh\left(\frac{qZ}{2}\right)s(qZ)\le0, Z s ′ ( q Z ) = − Z tanh ( 2 q Z ) s ( q Z ) ≤ 0 , strictly on a set of positive probability. Hence h ′ ( q ) < 0 h'(q)<0 h ′ ( q ) < 0 .
Lemma 16 (Relation between r r r and α ⋆ \alpha^\star α ⋆ ). The limiting fixed point r = B A ( r ) r=\frac{B}{A(r)} r = A ( r ) B satisfies r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) rh(\nu r)=(1-2\rho)h(\nu) r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) . The noiseless logistic optimum satisfies α ⋆ h ( ν α ⋆ ) = ( 1 − 2 ρ ) h ( ν ) \alpha^\star h(\nu\alpha^\star)=(1-2\rho)h(\nu) α ⋆ h ( ν α ⋆ ) = ( 1 − 2 ρ ) h ( ν ) . Consequently, r = α ⋆ r=\alpha^\star r = α ⋆ in this regime.
Proof. By Lemma 14 we let r r r be the the solution of r = B A ( r ) r=\frac{B}{A(r)} r = A ( r ) B , or equivalently, r r r is the solution of r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) rh(\nu r)=(1-2\rho)h(\nu) r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) , recall that
L 0 ′ ( a ) = ν E [ Z ( σ ( ν a Z ) − π ρ ( ν Z ) ) ] . \mathcal{L}_0'(a)=\nu\mathbb{E}\left[Z(\sigma(\nu aZ)-\pi_\rho(\nu Z))\right]. L 0 ′ ( a ) = ν E [ Z ( σ ( ν a Z ) − π ρ ( ν Z )) ] . Using Stein’s identity,
E [ Z σ ( q Z ) ] = q E [ σ ( q Z ) ( 1 − σ ( q Z ) ) ] = q h ( q ) . \mathbb{E}[Z\sigma(qZ)]=q\mathbb{E}[\sigma(qZ)(1-\sigma(qZ))]=qh(q). E [ Z σ ( q Z )] = q E [ σ ( q Z ) ( 1 − σ ( q Z ))] = q h ( q ) . Thus
L 0 ′ ( a ) = ν 2 [ a h ( ν a ) − ( 1 − 2 ρ ) h ( ν ) ] . \mathcal{L}_0'(a)=\nu^2\left[ah(\nu a)-(1-2\rho)h(\nu)\right]. L 0 ′ ( a ) = ν 2 [ ah ( ν a ) − ( 1 − 2 ρ ) h ( ν ) ] . Therefore L 0 ′ ( α ⋆ ) = 0 \mathcal{L}_0'(\alpha^\star)=0 L 0 ′ ( α ⋆ ) = 0 gives
α ⋆ h ( ν α ⋆ ) = ( 1 − 2 ρ ) h ( ν ) . \alpha^\star h(\nu\alpha^\star)=(1-2\rho)h(\nu). α ⋆ h ( ν α ⋆ ) = ( 1 − 2 ρ ) h ( ν ) . Since a ↦ a h ( ν a ) a\mapsto ah(\nu a) a ↦ ah ( ν a ) is strictly increasing (see Lemma 15), comparing the equations for r r r and α ⋆ \alpha^\star α ⋆ proves the claim.
Lemma 17 (Small-noise expansion of the logistic risk). If α t \alpha_t α t remains bounded and σ t → 0 \sigma_t \to0 σ t → 0 , then
L t ( λ ) = L 0 ( α t ) + σ t 2 2 h ( ν α t ) + O ( σ t 4 ) . \mathcal{L}_t(\lambda) = \mathcal{L}_0(\alpha_t) + \frac{\sigma_t^2}{2}h(\nu\alpha_t) + O(\sigma_t^4). L t ( λ ) = L 0 ( α t ) + 2 σ t 2 h ( ν α t ) + O ( σ t 4 ) .
Proof. For fixed Z 1 Z_1 Z 1 , define
f Z 1 ( m ) = log ( 1 + e m ) − π ρ ( ν Z 1 ) m . f_{Z_1}(m) = \log(1 + e^m) - \pi_\rho(\nu Z_1)m. f Z 1 ( m ) = log ( 1 + e m ) − π ρ ( ν Z 1 ) m .
Then
f Z 1 ′ ′ ( m ) = σ ( m ) ( 1 − σ ( m ) ) = s ( m ) . f_{Z_1}''(m) = \sigma(m)(1-\sigma(m)) = s(m). f Z 1 ′′ ( m ) = σ ( m ) ( 1 − σ ( m )) = s ( m ) .
Expanding in the Gaussian perturbation σ t Z 2 \sigma_t Z_2 σ t Z 2 ,
E Z 2 f Z 1 ( ν α t Z 1 + σ t Z 2 ) = f Z 1 ( ν α t Z 1 ) + σ t 2 2 f Z 1 ′ ′ ( ν α t Z 1 ) + O ( σ t 4 ) . \mathbb{E}_{Z_2}f_{Z_1}(\nu\alpha_t Z_1+\sigma_t Z_2) = f_{Z_1}(\nu\alpha_t Z_1) + \frac{\sigma_t^2}{2}f_{Z_1}''(\nu\alpha_t Z_1) + O(\sigma_t^4). E Z 2 f Z 1 ( ν α t Z 1 + σ t Z 2 ) = f Z 1 ( ν α t Z 1 ) + 2 σ t 2 f Z 1 ′′ ( ν α t Z 1 ) + O ( σ t 4 ) .
The odd powers vanish from symmetry of Z 2 Z_2 Z 2 , and the higher derivatives are bounded. Taking expectation over Z 1 Z_1 Z 1 yields the claim.
Lemma 18 (Bootstrap lower bound). There exists c 0 > 0 c_0 > 0 c 0 > 0 such that, if A T λ ≤ c 0 log ( T ) \frac{AT}{\lambda} \le c_0 \log(T) λ A T ≤ c 0 log ( T ) , then r − α t ≥ c e − C A T λ r-\alpha_t \ge c e^{-C\frac{AT}{\lambda}} r − α t ≥ c e − C λ A T .
Proof. Let e t = r − α t e_t=r-\alpha_t e t = r − α t . Using B = r A ( r ) B=rA(r) B = r A ( r ) in Lemma 14,
e t = ( 1 − A t λ ) e t − 1 + r ( A t − A ) λ + O ( λ − 2 ) . e_t = \left(1-\frac{A_t}{\lambda}\right)e_{t-1}+\frac{r(A_t-A)}{\lambda}+O(\lambda^{-2}). e t = ( 1 − λ A t ) e t − 1 + λ r ( A t − A ) + O ( λ − 2 ) . Since A t = 1 2 h ( ν 2 α t 2 + σ t 2 ) A_t=\frac{1}{2}h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right) A t = 2 1 h ( ν 2 α t 2 + σ t 2 ) , A = A ( r ) = 1 2 h ( ν r ) A=A(r)=\frac{1}{2}h(\nu r) A = A ( r ) = 2 1 h ( ν r ) , and h h h is smooth on bounded sets. Hence, on the bounded trajectory, ν 2 α t 2 + σ t 2 − ν r ≤ − ν e t + C σ t 2 \sqrt{\nu^2\alpha_t^2+\sigma_t^2}-\nu r\le-\nu e_t+C\sigma_t^2 ν 2 α t 2 + σ t 2 − ν r ≤ − ν e t + C σ t 2 . Since h h h is decreasing and Lipschitz on bounded sets, A t − A = 1 2 { h ( ν 2 α t 2 + σ t 2 ) − h ( ν r ) } ≥ − C σ t 2 A_t-A=\frac{1}{2}\{h(\sqrt{\nu^2\alpha_t^2+\sigma_t^2})-h(\nu r)\}\ge-C\sigma_t^2 A t − A = 2 1 { h ( ν 2 α t 2 + σ t 2 ) − h ( ν r )} ≥ − C σ t 2 . Therefore A t ≥ A − C σ t 2 A_t\ge A-C\sigma_t^2 A t ≥ A − C σ t 2 . Hence r ( A t − A ) λ ≳ − λ − 2 \frac{r(A_t-A)}{\lambda}\gtrsim-\lambda^{-2} λ r ( A t − A ) ≳ − λ − 2 , so
e T ≥ ( 1 − A T λ ) e T − 1 − C λ − 2 . e_T\ge\left(1-\frac{A_T}{\lambda}\right)e_{T-1}-C\lambda^{-2}. e T ≥ ( 1 − λ A T ) e T − 1 − C λ − 2 . for some C > 0 C>0 C > 0 . Iterating,
e T ≥ e 0 ∏ j = 1 T ( 1 − A j λ ) − C ∑ s = 1 T λ − 2 ∏ j = s + 1 T ( 1 − A j λ ) . e_T\ge e_0\prod_{j=1}^{T}\left(1-\frac{A_j}{\lambda}\right)-C\sum_{s=1}^{T}\lambda^{-2}\prod_{j=s+1}^{T}\left(1-\frac{A_j}{\lambda}\right). e T ≥ e 0 j = 1 ∏ T ( 1 − λ A j ) − C s = 1 ∑ T λ − 2 j = s + 1 ∏ T ( 1 − λ A j ) . Since A j ≤ A + A_j\le A_+ A j ≤ A + , we have ∏ j = 1 T ( 1 − A j λ ) ≥ exp ( − C T λ ) \prod_{j=1}^{T}\left(1-\frac{A_j}{\lambda}\right)\ge\exp\left(-\frac{CT}{\lambda}\right) ∏ j = 1 T ( 1 − λ A j ) ≥ exp ( − λ C T ) , and since A j ≥ A − > 0 A_j\ge A_->0 A j ≥ A − > 0 the accumulated error is
∑ s = 1 T λ − 2 ∏ j = s + 1 T ( 1 − A j λ ) ≤ 1 λ 2 ∑ k = 0 T e − c k ≲ 1 λ . \sum_{s=1}^{T}\lambda^{-2}\prod_{j=s+1}^{T}\left(1-\frac{A_j}{\lambda}\right)\le\frac{1}{\lambda^2}\sum_{k=0}^{T}e^{-ck}\lesssim\frac{1}{\lambda}. s = 1 ∑ T λ − 2 j = s + 1 ∏ T ( 1 − λ A j ) ≤ λ 2 1 k = 0 ∑ T e − c k ≲ λ 1 . for some c > 0 c>0 c > 0 . Thus
e T ≥ e 0 e − C A T λ − C 1 λ . e_T\ge e_0e^{-C\frac{AT}{\lambda}}-C\frac{1}{\lambda}. e T ≥ e 0 e − C λ A T − C λ 1 . If A T λ ≤ c 0 log ( T ) \frac{AT}{\lambda}\le c_0\log(T) λ A T ≤ c 0 log ( T ) and c 0 > 0 c_0>0 c 0 > 0 is sufficiently small, then e − C A T λ > 2 C 1 λ e^{-C\frac{AT}{\lambda}}>2C\frac{1}{\lambda} e − C λ A T > 2 C λ 1 . Hence
r − α T = e T ≥ c e − C A T λ . r-\alpha_T=e_T\ge ce^{-C\frac{AT}{\lambda}}. r − α T = e T ≥ c e − C λ A T . Moreover, the variance bound in Eq. (38) gives
σ T 2 ≤ C λ ≲ e − C A T λ ≲ e T . \sigma_T^2\le\frac{C}{\lambda}\lesssim e^{-C\frac{AT}{\lambda}}\lesssim e_T. σ T 2 ≤ λ C ≲ e − C λ A T ≲ e T . Local strong convexity of L 0 \mathcal{L}_0 L 0 gives
L 0 ( α T ) − L 0 ( α ⋆ ) ≥ c ( r − α T ) 2 ≥ c e − 2 C A T λ . \mathcal{L}_0(\alpha_T)-\mathcal{L}_0(\alpha^\star)\ge c(r-\alpha_T)^2\ge ce^{-2C\frac{AT}{\lambda}}. L 0 ( α T ) − L 0 ( α ⋆ ) ≥ c ( r − α T ) 2 ≥ c e − 2 C λ A T . For c 0 c_0 c 0 sufficiently small, this is much larger than log ( T ) T \frac{\log(T)}{T} T l o g ( T ) .
Lemma 19 (Optimal Scaling). There exist constants T 0 , C , c > 0 T_0,C,c>0 T 0 , C , c > 0 such that for all T ≥ T 0 T\ge T_0 T ≥ T 0 the optimal regularizer satisfies T c log ( T ) ≤ λ ⋆ ≤ T C log ( T ) \frac{T}{c\log(T)}\le\lambda^\star\le\frac{T}{C\log(T)} c l o g ( T ) T ≤ λ ⋆ ≤ C l o g ( T ) T .
Proof. We prove the equivalent statement x ⋆ = Θ ( log ( T ) ) x^\star=\Theta(\log(T)) x ⋆ = Θ ( log ( T )) , with x = A T λ x=\frac{AT}{\lambda} x = λ A T . First choose a comparison point x 0 = C 0 log ( T ) x_0=C_0\log(T) x 0 = C 0 log ( T ) with C 0 > 0 C_0>0 C 0 > 0 sufficiently large. By linearizing the ode around r r r and the recursion in Eq. (37) gives
d d x ( α − r ) = − β ( α − r ) + O ( ( α − r ) 2 ) + O ( 1 T ) , β = 1 + ν r h ′ ( ν r ) h ( ν r ) . \frac{\mathrm{d}}{\mathrm{d}x}(\alpha-r)=-\beta(\alpha-r)+O((\alpha-r)^2)+O\left(\frac{1}{T}\right),\qquad\beta=1+\frac{\nu r h'(\nu r)}{h(\nu r)}. d x d ( α − r ) = − β ( α − r ) + O (( α − r ) 2 ) + O ( T 1 ) , β = 1 + h ( ν r ) ν r h ′ ( ν r ) .
Solving this yields α ( x ) − r = C α e − β x + O ( e − 2 β x ) + O ( x T ) \alpha(x)-r=C_\alpha e^{-\beta x}+O(e^{-2\beta x})+O\left(\frac{x}{T}\right) α ( x ) − r = C α e − β x + O ( e − 2 β x ) + O ( T x ) , where C α C_\alpha C α is the solution to C α = lim τ → ∞ e κ τ ( r − α ( τ ) ) C_\alpha=\lim_{\tau\to\infty}e^{\kappa\tau}(r-\alpha(\tau)) C α = lim τ → ∞ e κ τ ( r − α ( τ )) and κ = 1 2 ( h ( ν r ) + ν r h ′ ( ν r ) ) \kappa=\frac{1}{2}\left(h(\nu r)+\nu r h'(\nu r)\right) κ = 2 1 ( h ( ν r ) + ν r h ′ ( ν r ) ) , therefore for a sufficiently large C C C and c c c ,
∣ α t ( x 0 ) − r ∣ ≤ C e − c x 0 + C x 0 T . |\alpha_t(x_0)-r|\le Ce^{-cx_0}+C\frac{x_0}{T}. ∣ α t ( x 0 ) − r ∣ ≤ C e − c x 0 + C T x 0 .
Choosing C 0 C_0 C 0 large enough,
∣ α t ( x 0 ) − r ∣ ≤ C log ( T ) T . |\alpha_t(x_0)-r|\le C\frac{\log(T)}{T}. ∣ α t ( x 0 ) − r ∣ ≤ C T log ( T ) .
Also, σ t 2 ( x 0 ) = O ( log ( T ) T ) \sigma_t^2(x_0)=O\left(\frac{\log(T)}{T}\right) σ t 2 ( x 0 ) = O ( T l o g ( T ) ) . Therefore in Lemma 17,
L T ( x 0 ) − L 0 ( r ) = O ( log ( T ) T ) . \mathcal{L}_T(x_0)-\mathcal{L}_0(r)=O\left(\frac{\log(T)}{T}\right). L T ( x 0 ) − L 0 ( r ) = O ( T log ( T ) ) .
Now we rule out x ≫ log ( T ) x\gg\log(T) x ≫ log ( T ) , since L 0 ( α t ) ≥ L 0 ( α ⋆ ) \mathcal{L}_0(\alpha_t)\ge\mathcal{L}_0(\alpha^\star) L 0 ( α t ) ≥ L 0 ( α ⋆ ) , so the variance lower bound in Lemma 13 gives
L T ( x ) − L 0 ( α ⋆ ) ≥ c x T . \mathcal{L}_T(x)-\mathcal{L}_0(\alpha^\star)\ge c\frac{x}{T}. L T ( x ) − L 0 ( α ⋆ ) ≥ c T x .
Thus x ≫ log ( T ) x\gg\log(T) x ≫ log ( T ) is worse than the comparison point, for x ≫ log ( T ) x\gg\log(T) x ≫ log ( T ) , x = o ( T ) x=o(T) x = o ( T ) . If x x x is not o ( T ) o(T) o ( T ) , then λ \lambda λ does not diverge, so the variance does not vanish and the risk stays a fixed positive distance above the noiseless floor L 0 ( r ) \mathcal{L}_0(r) L 0 ( r ) . This is again worse than the comparison point. Hence x ⋆ ≲ log ( T ) x^\star\lesssim\log(T) x ⋆ ≲ log ( T ) . Finally, Lemma 18 rules out x ≤ c 0 log ( T ) x\le c_0\log(T) x ≤ c 0 log ( T ) for a sufficiently small c 0 > 0 c_0>0 c 0 > 0 . Hence x ⋆ ≳ log ( T ) x^\star\gtrsim\log(T) x ⋆ ≳ log ( T ) . Therefore x ⋆ = Θ ( log ( T ) ) x^\star=\Theta(\log(T)) x ⋆ = Θ ( log ( T )) . Since λ ⋆ = A T x ⋆ \lambda^\star=\frac{AT}{x^\star} λ ⋆ = x ⋆ A T , we get λ ⋆ = Θ ( T log ( T ) ) \lambda^\star=\Theta\left(\frac{T}{\log(T)}\right) λ ⋆ = Θ ( l o g ( T ) T ) . ∎
Proof of thm:2
After the simplifications and auxiliary Lemmas established above we are ready to prove Theorem 2.
Full Theorem 2. As n → ∞ n \to\infty n → ∞ the optimal fixed step size λ ⋆ \lambda^\star λ ⋆ that minimizes the resulting generalization loss of the regularized logistic regression after T T T tasks with non-zero teacher6 satisfies λ ⋆ ∼ T log T \lambda^\star\sim\frac{T}{\log T} λ ⋆ ∼ l o g T T . Specifically, for all sufficiently large T > T 0 = O ( 1 ϵ 2 ) T>T_0=O\left(\frac{1}{\epsilon^2}\right) T > T 0 = O ( ϵ 2 1 ) , the optimal parameters satisfy
( 1 − ϵ ) κ T log ( κ ν 2 E [ Z 2 ϕ ′ ′ ( ν r Z ) ] C α 2 T 1 2 δ E [ ( 1 − π ρ ( ν Z ) ) 1 ( 1 + e − ν r Z ) 2 ] ) < λ ⋆ < ( 1 + ϵ ) κ T log ( κ ν 2 E [ Z 2 ϕ ′ ′ ( ν r Z ) ] C α 2 T 1 2 δ E [ ( 1 − π ρ ( ν Z ) ) 1 ( 1 + e − ν r Z ) 2 ] ) . (1-\epsilon)\frac{\kappa T}{\log\left(\frac{\kappa\nu^2\mathbb{E}[Z^2\phi^{\prime\prime}(\nu rZ)]C_\alpha^2T}{\frac{1}{2\delta}\mathbb{E}\left[(1-\pi_\rho(\nu Z))\frac{1}{(1+e^{-\nu rZ})^2}\right]}\right)}<\lambda^\star<(1+\epsilon)\frac{\kappa T}{\log\left(\frac{\kappa\nu^2\mathbb{E}[Z^2\phi^{\prime\prime}(\nu rZ)]C_\alpha^2T}{\frac{1}{2\delta}\mathbb{E}\left[(1-\pi_\rho(\nu Z))\frac{1}{(1+e^{-\nu rZ})^2}\right]}\right)}. ( 1 − ϵ ) log ( 2 δ 1 E [ ( 1 − π ρ ( ν Z )) ( 1 + e − ν r Z ) 2 1 ] κ ν 2 E [ Z 2 ϕ ′′ ( ν r Z )] C α 2 T ) κ T < λ ⋆ < ( 1 + ϵ ) log ( 2 δ 1 E [ ( 1 − π ρ ( ν Z )) ( 1 + e − ν r Z ) 2 1 ] κ ν 2 E [ Z 2 ϕ ′′ ( ν r Z )] C α 2 T ) κ T .
where π ρ ( u ) = ρ + ( 1 − 2 ρ ) e u 1 + e u \pi_\rho(u)=\rho+(1-2\rho)\frac{e^u}{1+e^u} π ρ ( u ) = ρ + ( 1 − 2 ρ ) 1 + e u e u , κ = E [ ϕ ′ ′ ( ν r Z ) ] + ν r E [ Z ϕ ′ ′ ′ ( ν r Z ) ] \kappa=\mathbb{E}[\phi^{\prime\prime}(\nu rZ)]+\nu r\mathbb{E}[Z\phi^{\prime\prime\prime}(\nu rZ)] κ = E [ ϕ ′′ ( ν r Z )] + ν r E [ Z ϕ ′′′ ( ν r Z )] , r r r solves the equation r = ( 1 − 2 ρ ) E [ ϕ ′ ′ ( ν Z ) ] E [ ϕ ′ ′ ( ν r Z ) ] r=\frac{(1-2\rho)\mathbb{E}[\phi^{\prime\prime}(\nu Z)]}{\mathbb{E}[\phi^{\prime\prime}(\nu rZ)]} r = E [ ϕ ′′ ( ν r Z )] ( 1 − 2 ρ ) E [ ϕ ′′ ( ν Z )] , and C α C_\alpha C α is determined by the limiting ODE
2 α ˙ = ( 1 − 2 ρ ) E [ ϕ ′ ′ ( ν Z ) ] − α E [ ϕ ′ ′ ( ν α Z ) ] , α ( 0 ) = 0 , 2\dot{\alpha}=(1-2\rho)\mathbb{E}[\phi^{\prime\prime}(\nu Z)]-\alpha\mathbb{E}[\phi^{\prime\prime}(\nu\alpha Z)],\qquad\alpha(0)=0, 2 α ˙ = ( 1 − 2 ρ ) E [ ϕ ′′ ( ν Z )] − α E [ ϕ ′′ ( ν α Z )] , α ( 0 ) = 0 ,
through
C α = lim τ → ∞ e 1 2 κ τ ( r − α ( τ ) ) ∈ ( 0 , ∞ ) , C_\alpha=\lim_{\tau\to\infty}e^{\frac{1}{2}\kappa\tau}(r-\alpha(\tau))\in(0,\infty), C α = τ → ∞ lim e 2 1 κ τ ( r − α ( τ )) ∈ ( 0 , ∞ ) ,
and Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) . Moreover, under this scaling the excess risk decreases in T T T as,
L ( w T ) − inf w L ( w ) = Θ ν , δ , ρ ( log ( T ) T ) . \mathcal{L}(w_T)-\inf_w\mathcal{L}(w)=\Theta_{\nu,\delta,\rho}\left(\frac{\log(T)}{T}\right). L ( w T ) − w inf L ( w ) = Θ ν , δ , ρ ( T log ( T ) ) .
Before presenting the proof, we direct the reader to the notations introduced in App. § 10 .§ 2 .
The positivity of κ = h ( ν r ) + ν r h ′ ( ν r ) > 0 \kappa=h(\nu r)+\nu r h'(\nu r)>0 κ = h ( ν r ) + ν r h ′ ( ν r ) > 0 follows from the strict monotonicity of q ↦ q h ( q ) q\mapsto qh(q) q ↦ q h ( q ) by Lemma 15 . Throughout the proof we set β = A A ~ = 1 + ν r h ′ ( ν r ) h ( ν r ) \beta=\frac{A}{\tilde{A}}=1+\frac{\nu r h'(\nu r)}{h(\nu r)} β = A ~ A = 1 + h ( ν r ) ν r h ′ ( ν r ) and define x = A T λ x=\frac{AT}{\lambda} x = λ A T .
Note that, by linearizing the ode around r r r gives d d x ( α − r ) = − β ( α − r ) + O ( ( α − r ) 2 ) + O ( 1 T ) \frac{d}{dx}(\alpha-r)=-\beta(\alpha-r)+O((\alpha-r)^2)+O\left(\frac{1}{T}\right) d x d ( α − r ) = − β ( α − r ) + O (( α − r ) 2 ) + O ( T 1 ) solving this yields α ( x ) − r = C α e − β x + O ( e − 2 β x ) + O ( x T ) \alpha(x)-r=C_\alpha e^{-\beta x}+O(e^{-2\beta x})+O\left(\frac{x}{T}\right) α ( x ) − r = C α e − β x + O ( e − 2 β x ) + O ( T x ) . Its stable fixed point r r r satisfies r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) rh(\nu r)=(1-2\rho)h(\nu) r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) by Lemma 16 , and taking Taylor expansion in Eq. (37) leads,
σ T 2 = V 2 A λ + O ( x T e − β x + x 2 T 2 ) = V 2 A 2 x T + O ( x T e − β x + x 2 T 2 ) , \sigma_T^2=\frac{V}{2A\lambda}+O\left(\frac{x}{T}e^{-\beta x}+\frac{x^2}{T^2}\right)=\frac{V}{2A^2}\frac{x}{T}+O\left(\frac{x}{T}e^{-\beta x}+\frac{x^2}{T^2}\right), σ T 2 = 2 A λ V + O ( T x e − β x + T 2 x 2 ) = 2 A 2 V T x + O ( T x e − β x + T 2 x 2 ) ,
where V = 1 2 δ E [ ( 1 − π ρ ( ν Z ) ) ϕ ′ ( ν r Z ) 2 ] V=\frac{1}{2\delta}\mathbb{E}\left[(1-\pi_\rho(\nu Z))\phi'(\nu rZ)^2\right] V = 2 δ 1 E [ ( 1 − π ρ ( ν Z )) ϕ ′ ( ν r Z ) 2 ] . Since r = α ⋆ r=\alpha^\star r = α ⋆ by Lemma 16 , which implies L 0 ′ ( r ) = 0 \mathcal{L}_0'(r)=0 L 0 ′ ( r ) = 0 , by performing a small-noise expansions about r = α ⋆ r=\alpha^\star r = α ⋆ , since L 0 ′ ( r ) = 0 \mathcal{L}_0'(r)=0 L 0 ′ ( r ) = 0 we obtain
L T ( α T ) − L 0 ( r ) = 1 2 L 0 ′ ′ ( r ) ( α T − r ) 2 + O ( ∣ α T − r ∣ 3 ) \mathcal{L}_T(\alpha_T)-\mathcal{L}_0(r)=\frac{1}{2}\mathcal{L}_0^{\prime\prime}(r)(\alpha_T-r)^2+O(|\alpha_T-r|^3) L T ( α T ) − L 0 ( r ) = 2 1 L 0 ′′ ( r ) ( α T − r ) 2 + O ( ∣ α T − r ∣ 3 )
then using the ode linearization,
L T ( α T ) − L 0 ( r ) = 1 2 L 0 ′ ′ ( r ) C α 2 e − 2 β x + O ( e − 3 β x + x T e − β x + x 2 T 2 ) , \mathcal{L}_T(\alpha_T)-\mathcal{L}_0(r)=\frac{1}{2}\mathcal{L}_0^{\prime\prime}(r)C_\alpha^2e^{-2\beta x}+O\left(e^{-3\beta x}+\frac{x}{T}e^{-\beta x}+\frac{x^2}{T^2}\right), L T ( α T ) − L 0 ( r ) = 2 1 L 0 ′′ ( r ) C α 2 e − 2 β x + O ( e − 3 β x + T x e − β x + T 2 x 2 ) ,
and because by Lemma 17 ,
L T ( λ ) − L 0 ( α T ) = σ T 2 2 h ( ν α T ) + O ( σ T 4 ) , \mathcal{L}_T(\lambda)-\mathcal{L}_0(\alpha_T)=\frac{\sigma_T^2}{2}h(\nu\alpha_T)+O(\sigma_T^4), L T ( λ ) − L 0 ( α T ) = 2 σ T 2 h ( ν α T ) + O ( σ T 4 ) ,
we finally get, since r = α ⋆ r=\alpha^\star r = α ⋆ and h ( ν α T ) = h ( ν r ) + ν h ′ ( ν r ) ( α T − r ) + O ( ( α T − r ) 2 ) = 2 A + O ( e − β x + x T ) h(\nu\alpha_T)=h(\nu r)+\nu h'(\nu r)(\alpha_T-r)+O((\alpha_T-r)^2)=2A+O\left(e^{-\beta x}+\frac{x}{T}\right) h ( ν α T ) = h ( ν r ) + ν h ′ ( ν r ) ( α T − r ) + O (( α T − r ) 2 ) = 2 A + O ( e − β x + T x ) ,
L T ( λ ) − L 0 ( α ⋆ ) = 1 2 L 0 ′ ′ ( r ) C α 2 e − 2 β x + V 2 A x T + O ( e − 3 β x + x T e − β x + x 2 T 2 ) , \mathcal{L}_T(\lambda)-\mathcal{L}_0(\alpha^\star)=\frac{1}{2}\mathcal{L}_0^{\prime\prime}(r)C_\alpha^2e^{-2\beta x}+\frac{V}{2A}\frac{x}{T}+O\left(e^{-3\beta x}+\frac{x}{T}e^{-\beta x}+\frac{x^2}{T^2}\right), L T ( λ ) − L 0 ( α ⋆ ) = 2 1 L 0 ′′ ( r ) C α 2 e − 2 β x + 2 A V T x + O ( e − 3 β x + T x e − β x + T 2 x 2 ) ,
The leading two-term approximation is therefore
R l e a d ( x ) = 1 2 L 0 ′ ′ ( r ) C α 2 e − 2 β x + V 2 A T x . R_{\mathrm{lead}}(x)=\frac{1}{2}L_0^{\prime\prime}(r)C_{\alpha}^{2}e^{-2\beta x}+\frac{V}{2AT}x. R lead ( x ) = 2 1 L 0 ′′ ( r ) C α 2 e − 2 β x + 2 A T V x .
Its minimizer x 0 x_0 x 0 satisfies − 2 β 1 2 L 0 ′ ′ ( r ) C α 2 e − 2 β x 0 + V 2 A T = 0 -2\beta\frac{1}{2}L_0^{\prime\prime}(r)C_{\alpha}^{2}e^{-2\beta x_0}+\frac{V}{2AT}=0 − 2 β 2 1 L 0 ′′ ( r ) C α 2 e − 2 β x 0 + 2 A T V = 0 , hence e − 2 β x 0 = V 2 A B β C 0 ′ ′ ( r ) C α 2 1 T e^{-2\beta x_0}=\frac{V}{2AB\beta C_0^{\prime\prime}(r)C_{\alpha}^{2}}\frac{1}{T} e − 2 β x 0 = 2 A B β C 0 ′′ ( r ) C α 2 V T 1 . Then
x 0 = 1 2 β log ( 2 T A B β L 0 ′ ′ ( r ) C α 2 V ) , x_0=\frac{1}{2\beta}\log\left(\frac{2TAB\beta L_0^{\prime\prime}(r)C_{\alpha}^{2}}{V}\right), x 0 = 2 β 1 log ( V 2 T A B β L 0 ′′ ( r ) C α 2 ) ,
and the centered approximation for the optimal regularization is
λ 0 = A T x 0 = 2 β A T log ( 2 T A B β L 0 ′ ′ ( r ) C α 2 V ) = ( h ( ν r ) + ν r h ′ ( ν r ) ) T log ( T ( h ( ν r ) + ν r h ′ ( ν r ) ) L 0 ′ ′ ( r ) C α 2 V ) . \lambda_0=\frac{AT}{x_0}=\frac{2\beta AT}{\log\left(\frac{2TAB\beta L_0^{\prime\prime}(r)C_{\alpha}^{2}}{V}\right)}=\frac{(h(\nu r)+\nu r h'(\nu r))T}{\log\left(\frac{T(h(\nu r)+\nu r h'(\nu r))L_0^{\prime\prime}(r)C_{\alpha}^{2}}{V}\right)}. λ 0 = x 0 A T = log ( V 2 T A B β L 0 ′′ ( r ) C α 2 ) 2 β A T = log ( V T ( h ( ν r ) + ν r h ′ ( ν r )) L 0 ′′ ( r ) C α 2 ) ( h ( ν r ) + ν r h ′ ( ν r )) T .
This is indeed the optimal regularization strength even in a regime where the expansion does not hold (i.e. remainder term explodes), this is established in Lemma 19. We now verify the accuracy threshold of the centered approximation. Let x T ∗ x_T^* x T ∗ be the minimizer. At x = x 0 x=x_0 x = x 0 , e − β x 0 = O ( 1 T ) e^{-\beta x_0}=O\left(\sqrt{\frac{1}{T}}\right) e − β x 0 = O ( T 1 ) . The second derivative of the leading risk satisfies R l e a d ′ ′ ( x 0 ) = 2 β 2 L 0 ′ ′ ( r ) C α 2 e − 2 β x 0 R_{\mathrm{lead}}^{\prime\prime}(x_0)=2\beta^2L_0^{\prime\prime}(r)C_{\alpha}^{2}e^{-2\beta x_0} R lead ′′ ( x 0 ) = 2 β 2 L 0 ′′ ( r ) C α 2 e − 2 β x 0 . The derivative of the remainder at x 0 x_0 x 0 is bounded by O ( e − 3 β x 0 + x 0 T e − β x 0 + x 0 T 2 ) O\left(e^{-3\beta x_0}+\frac{x_0}{T}e^{-\beta x_0}+\frac{x_0}{T^2}\right) O ( e − 3 β x 0 + T x 0 e − β x 0 + T 2 x 0 ) . Since R l e a d ′ ( x 0 ) = 0 R_{\mathrm{lead}}^{\prime}(x_0)=0 R lead ′ ( x 0 ) = 0 . Taylor expansion around x 0 x_0 x 0 gives
0 = R l e a d ′ ′ ( x 0 ) ( x T ∗ − x 0 ) + R m e m ′ ( x 0 ) + o ( x T ∗ − x 0 ) + o ( R m e m ′ ( x 0 ) ) 0=R_{\mathrm{lead}}^{\prime\prime}(x_0)(x_T^*-x_0)+R_{\mathrm{mem}}^{\prime}(x_0)+o(x_T^*-x_0)+o(R_{\mathrm{mem}}^{\prime}(x_0)) 0 = R lead ′′ ( x 0 ) ( x T ∗ − x 0 ) + R mem ′ ( x 0 ) + o ( x T ∗ − x 0 ) + o ( R mem ′ ( x 0 ))
Dividing by R l e a d ′ ′ ( x 0 ) R_{\mathrm{lead}}^{\prime\prime}(x_0) R lead ′′ ( x 0 ) , we obtain x T ∗ − x 0 = O ( x 0 1 T ) x_T^*-x_0=O\left(x_0\sqrt{\frac{1}{T}}\right) x T ∗ − x 0 = O ( x 0 T 1 ) . Consequently, x T ∗ x 0 = 1 + O ( 1 T ) \frac{x_T^*}{x_0}=1+O\left(\sqrt{\frac{1}{T}}\right) x 0 x T ∗ = 1 + O ( T 1 ) , and since λ = A T x \lambda=\frac{AT}{x} λ = x A T ,
λ ∗ λ 0 = 1 + O ( 1 T ) . \frac{\lambda^*}{\lambda_0}=1+O\left(\sqrt{\frac{1}{T}}\right). λ 0 λ ∗ = 1 + O ( T 1 ) .
Therefore, for any fixed 0 < ϵ < 1 0<\epsilon<1 0 < ϵ < 1 ,
( 1 − ϵ ) λ 0 ≤ λ ∗ ≤ ( 1 + ϵ ) λ 0 (1-\epsilon)\lambda_0\leq\lambda^*\leq(1+\epsilon)\lambda_0 ( 1 − ϵ ) λ 0 ≤ λ ∗ ≤ ( 1 + ϵ ) λ 0
whenever 1 T ≲ ϵ \sqrt{\frac{1}{T}}\lesssim\epsilon T 1 ≲ ϵ . Equivalently, T ≳ 1 ϵ 2 T\gtrsim\frac{1}{\epsilon^2} T ≳ ϵ 2 1 , the optimal regularizer as appears in the Theorem follow by noting that by definition V = 1 2 δ E [ ( 1 − π ρ ( ν Z ) ) ϕ ′ ( ν r Z ) 2 ] V=\frac{1}{2\delta}\mathbb{E}\left[(1-\pi_\rho(\nu Z))\phi'(\nu r Z)^2\right] V = 2 δ 1 E [ ( 1 − π ρ ( ν Z )) ϕ ′ ( ν r Z ) 2 ] , κ = 1 2 ( h ( ν r ) + ν r h ′ ( ν r ) ) \kappa=\frac{1}{2}\left(h(\nu r)+\nu r h'(\nu r)\right) κ = 2 1 ( h ( ν r ) + ν r h ′ ( ν r ) ) , and L 0 ′ ′ ( r ) = ν 2 E [ Z 2 ϕ ′ ′ ( ν r Z ) ] L_0^{\prime\prime}(r)=\nu^2\mathbb{E}[Z^2\phi^{\prime\prime}(\nu r Z)] L 0 ′′ ( r ) = ν 2 E [ Z 2 ϕ ′′ ( ν r Z )] . Now, we develop the optimal rate achieved, recall that the leading excess risk is
R l e a d ( x ) = 1 2 L 0 ′ ′ ( r ) C α 2 e − 2 β x + V 2 A T x . R_{\mathrm{lead}}(x)=\frac{1}{2}L_0^{\prime\prime}(r)C_{\alpha}^{2}e^{-2\beta x}+\frac{V}{2AT}x. R lead ( x ) = 2 1 L 0 ′′ ( r ) C α 2 e − 2 β x + 2 A T V x .
From this expression it follows immediately that for all λ = Θ ν , δ , ρ ( T log ( T ) ) \lambda=\Theta_{\nu,\delta,\rho}\left(\frac{T}{\log(T)}\right) λ = Θ ν , δ , ρ ( l o g ( T ) T ) the optimal excess risk satisfies
L ( w T ) − inf w L ( w ) = Θ ν , δ , ρ ( log ( T ) T ) . \mathcal{L}(\mathbf{w}_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})=\Theta_{\nu,\delta,\rho}\left(\frac{\log(T)}{T}\right). L ( w T ) − w inf L ( w ) = Θ ν , δ , ρ ( T log ( T ) ) .
Continual sparse classification
We verify the condition in full Theorem 1 for the regularizers considered in this section: L1 and sign weighted L1. For L1 the Moreau envelope is given by (see Lemma 20)
M λ f ~ ( ⋅ , v ) ( x , c ) = { ( x − v ) 2 2 c , ∣ x − v ∣ ≤ λ c , λ ∣ x − v ∣ − λ 2 c 2 , ∣ x − v ∣ > λ c . M_{\lambda\tilde f(\cdot,v)}(x,c)=
\begin{cases}
\dfrac{(x-v)^2}{2c}, & |x-v|\leq\lambda c,\\
\lambda|x-v|-\dfrac{\lambda^2c}{2}, & |x-v|>\lambda c.
\end{cases} M λ f ~ ( ⋅ , v ) ( x , c ) = ⎩ ⎨ ⎧ 2 c ( x − v ) 2 , λ ∣ x − v ∣ − 2 λ 2 c , ∣ x − v ∣ ≤ λ c , ∣ x − v ∣ > λ c .
and for sign weighted L1 (see Lemma 21),
M λ f ~ ( ⋅ , v ) ( x , c ) = { − λ ϵ + ( − v ) + x − c 2 ( λ ϵ + ( − v ) + ) 2 , x < − c λ ϵ + ( − v ) + , x 2 2 c , − c λ ϵ + ( − v ) + ≤ x ≤ c λ ϵ + v + , λ ϵ + v + x − c 2 ( λ ϵ + v + ) 2 , x > c λ ϵ + v + . M_{\lambda\tilde f(\cdot,v)}(x,c)=
\begin{cases}
-\dfrac{\lambda}{\epsilon+(-v)_{+}}x-\dfrac{c}{2}\left(\dfrac{\lambda}{\epsilon+(-v)_{+}}\right)^2, & x<-c\dfrac{\lambda}{\epsilon+(-v)_{+}},\\
\dfrac{x^2}{2c}, & -c\dfrac{\lambda}{\epsilon+(-v)_{+}}\leq x\leq c\dfrac{\lambda}{\epsilon+v_{+}},\\
\dfrac{\lambda}{\epsilon+v_{+}}x-\dfrac{c}{2}\left(\dfrac{\lambda}{\epsilon+v_{+}}\right)^2, & x>c\dfrac{\lambda}{\epsilon+v_{+}}.
\end{cases} M λ f ~ ( ⋅ , v ) ( x , c ) = ⎩ ⎨ ⎧ − ϵ + ( − v ) + λ x − 2 c ( ϵ + ( − v ) + λ ) 2 , 2 c x 2 , ϵ + v + λ x − 2 c ( ϵ + v + λ ) 2 , x < − c ϵ + ( − v ) + λ , − c ϵ + ( − v ) + λ ≤ x ≤ c ϵ + v + λ , x > c ϵ + v + λ .
these are both continuous and admit quadratic growth, furthermore, the joint convergence of the empirical distribution follows by Remark 11.
Auxiliary Lemmas
Lemma 20 (Proximal map of the scalar L1 penalty). Let λ > 0 \lambda>0 λ > 0 and define f ( w , z ) = ∣ w − z ∣ f(w,z)=|w-z| f ( w , z ) = ∣ w − z ∣ . Then
Prox λ f ( ⋅ , z ) ( v ) = argmin w ∈ R { ∣ w − z ∣ + 1 2 λ ( w − v ) 2 } = z + S λ ( v − z ) , \operatorname{Prox}_{\lambda f(\cdot,z)}(v)=\underset{w\in\mathbb{R}}{\operatorname{argmin}}\left\{|w-z|+\frac{1}{2\lambda}(w-v)^2\right\}=z+S_{\lambda}(v-z), Prox λ f ( ⋅ , z ) ( v ) = w ∈ R argmin { ∣ w − z ∣ + 2 λ 1 ( w − v ) 2 } = z + S λ ( v − z ) ,
where
S λ ( u ) = sign ( u ) ( ∣ u ∣ − λ ) + = { u − λ , u > λ , 0 , ∣ u ∣ ≤ λ , u + λ , u < − λ . S_{\lambda}(u)=\operatorname{sign}(u)(|u|-\lambda)_{+}=
\begin{cases}
u-\lambda, & u>\lambda,\\
0, & |u|\leq\lambda,\\
u+\lambda, & u<-\lambda.
\end{cases} S λ ( u ) = sign ( u ) ( ∣ u ∣ − λ ) + = ⎩ ⎨ ⎧ u − λ , 0 , u + λ , u > λ , ∣ u ∣ ≤ λ , u < − λ .
Proof. Consider g ( w ) = ∣ w − z ∣ + 1 2 λ ( w − v ) 2 g(w)=|w-z|+\frac{1}{2\lambda}(w-v)^2 g ( w ) = ∣ w − z ∣ + 2 λ 1 ( w − v ) 2 . If w − z > 0 w-z>0 w − z > 0 , then ∣ w − z ∣ = w − z |w-z|=w-z ∣ w − z ∣ = w − z , and g ′ ( w ) = 1 + w − v λ g'(w)=1+\frac{w-v}{\lambda} g ′ ( w ) = 1 + λ w − v . Setting g ′ ( w ) = 0 g'(w)=0 g ′ ( w ) = 0 gives w = v − λ w=v-\lambda w = v − λ , which is valid only if v > z + λ v>z+\lambda v > z + λ . If w − z < 0 w-z<0 w − z < 0 , then ∣ w − z ∣ = z − w |w-z|=z-w ∣ w − z ∣ = z − w , and g ′ ( w ) = − 1 + w − v λ g'(w)=-1+\frac{w-v}{\lambda} g ′ ( w ) = − 1 + λ w − v . Setting g ′ ( w ) = 0 g'(w)=0 g ′ ( w ) = 0 gives w = v + λ w=v+\lambda w = v + λ , which is valid only if v < z − λ v<z-\lambda v < z − λ .
Finally, at w = z w=z w = z we use the subgradient condition 0 ∈ ∂ ∣ w − z ∣ + 1 λ ( w − v ) 0\in\partial|w-z|+\frac{1}{\lambda}(w-v) 0 ∈ ∂ ∣ w − z ∣ + λ 1 ( w − v ) . Since ∂ ∣ w − z ∣ ∣ w = z = [ − 1 , 1 ] \left.\partial|w-z|\right|_{w=z}=[-1,1] ∂ ∣ w − z ∣ ∣ w = z = [ − 1 , 1 ] , this becomes 0 ∈ [ − 1 , 1 ] + z − v λ 0\in[-1,1]+\frac{z-v}{\lambda} 0 ∈ [ − 1 , 1 ] + λ z − v , which is equivalent to ∣ v − z ∣ ≤ λ |v-z|\leq\lambda ∣ v − z ∣ ≤ λ . Therefore w = z w=z w = z whenever ∣ v − z ∣ ≤ λ |v-z|\leq\lambda ∣ v − z ∣ ≤ λ . Combining the three cases gives Prox λ f ( ⋅ , z ) ( v ) = z + S λ ( v − z ) \operatorname{Prox}_{\lambda f(\cdot,z)}(v)=z+S_{\lambda}(v-z) Prox λ f ( ⋅ , z ) ( v ) = z + S λ ( v − z ) .
Lemma 21 (Proximal map of the sign weighted L1 regularizer). Fix u ∈ R u\in\mathbb{R} u ∈ R and ϵ > 0 \epsilon>0 ϵ > 0 , and define
f ( w ; u ) = { ∣ w ∣ 2 [ w u ] + + ϵ ∣ w ∣ , w ≠ 0 , 0 , w = 0. f(w;u)=
\begin{cases}
\dfrac{|w|^2}{[wu]_{+}+\epsilon|w|}, & w\neq0,\\
0, & w=0.
\end{cases} f ( w ; u ) = ⎩ ⎨ ⎧ [ w u ] + + ϵ ∣ w ∣ ∣ w ∣ 2 , 0 , w = 0 , w = 0.
Then f ( ⋅ ; u ) f(\cdot;u) f ( ⋅ ; u ) is convex and positively one-homogeneous. Moreover, for every q > 0 q>0 q > 0 , λ 0 > 0 \lambda_0>0 λ 0 > 0 , and x ∈ R x\in\mathbb{R} x ∈ R ,
Prox λ 0 q f ( ⋅ , u ) ( q x ) = q T λ 0 [ u ] + + ϵ , λ 0 [ − u ] + + ϵ ( x ) , \operatorname{Prox}_{\lambda_0qf(\cdot,u)}(qx)
=q\mathcal{T}_{\frac{\lambda_0}{[u]_{+}+\epsilon},\frac{\lambda_0}{[-u]_{+}+\epsilon}}(x), Prox λ 0 q f ( ⋅ , u ) ( q x ) = q T [ u ] + + ϵ λ 0 , [ − u ] + + ϵ λ 0 ( x ) ,
where
T a , b ( x ) = { x − a , x > a , 0 , − b ≤ x ≤ a , x + b , x < − b . \mathcal{T}_{a,b}(x)=
\begin{cases}
x-a, & x>a,\\
0, & -b\leq x\leq a,\\
x+b, & x<-b.
\end{cases} T a , b ( x ) = ⎩ ⎨ ⎧ x − a , 0 , x + b , x > a , − b ≤ x ≤ a , x < − b .
Proof. For w > 0 w>0 w > 0 , one has [ w u ] + = w [ u ] + [wu]_+=w[u]_+ [ w u ] + = w [ u ] + , while for w < 0 w<0 w < 0 , [ w u ] + = ( − w ) [ − u ] + [wu]_+=(-w)[-u]_+ [ w u ] + = ( − w ) [ − u ] + . Hence
f ( w , u ) = { w [ u ] + + ϵ , w > 0 , − w [ − u ] + + ϵ , w < 0 , 0 , w = 0. f(w,u)=
\begin{cases}
\dfrac{w}{[u]_+ + \epsilon}, & w>0,\\
\dfrac{-w}{[-u]_+ + \epsilon}, & w<0,\\
0, & w=0.
\end{cases} f ( w , u ) = ⎩ ⎨ ⎧ [ u ] + + ϵ w , [ − u ] + + ϵ − w , 0 , w > 0 , w < 0 , w = 0.
Thus f ( ⋅ , u ) f(\cdot,u) f ( ⋅ , u ) is piecewise linear, with left derivative − 1 [ − u ] + + ϵ -\dfrac{1}{[-u]_+ + \epsilon} − [ − u ] + + ϵ 1 and right derivative 1 [ u ] + + ϵ \dfrac{1}{[u]_+ + \epsilon} [ u ] + + ϵ 1 . Since the left derivative is no larger than the right derivative, f ( ⋅ , u ) f(\cdot,u) f ( ⋅ , u ) is convex. Also, f ( c w , u ) = c f ( w , u ) f(cw,u)=cf(w,u) f ( c w , u ) = c f ( w , u ) for every c ≥ 0 c\geq0 c ≥ 0 , so f f f is positively one-homogeneous. Let y = Prox λ 0 q f ( ⋅ , u ) ( q x ) y=\operatorname{Prox}_{\lambda_0qf(\cdot,u)}(qx) y = Prox λ 0 q f ( ⋅ , u ) ( q x ) . Writing y = q v y=qv y = q v and using f ( q v , u ) = q f ( v , u ) f(qv,u)=qf(v,u) f ( q v , u ) = q f ( v , u ) , the proximal objective becomes
1 2 ( q v − q x ) 2 + λ 0 q f ( q v , u ) = q 2 ( 1 2 ( v − x ) 2 + λ 0 f ( v , u ) ) . \frac{1}{2}(qv-qx)^2+\lambda_0qf(qv,u)
=q^2\left(\frac{1}{2}(v-x)^2+\lambda_0f(v,u)\right). 2 1 ( q v − q x ) 2 + λ 0 q f ( q v , u ) = q 2 ( 2 1 ( v − x ) 2 + λ 0 f ( v , u ) ) .
Therefore y = q Prox λ 0 f ( ⋅ ; u ) ( x ) y=q\operatorname{Prox}_{\lambda_0f(\cdot;u)}(x) y = q Prox λ 0 f ( ⋅ ; u ) ( x ) . On v > 0 v>0 v > 0 , the minimization problem is
min v > 0 { 1 2 ( v − x ) 2 + λ 0 [ u ] + + ϵ v } , \min_{v>0}\left\{\frac{1}{2}(v-x)^2+\frac{\lambda_0}{[u]_+ + \epsilon}v\right\}, v > 0 min { 2 1 ( v − x ) 2 + [ u ] + + ϵ λ 0 v } ,
whose minimizer is v = x − λ 0 [ u ] + + ϵ v=x-\dfrac{\lambda_0}{[u]_+ + \epsilon} v = x − [ u ] + + ϵ λ 0 when x > λ 0 [ u ] + + ϵ x>\dfrac{\lambda_0}{[u]_+ + \epsilon} x > [ u ] + + ϵ λ 0 . Similarly, on v < 0 v<0 v < 0 , the minimizer is v = x + λ 0 [ − u ] + + ϵ v=x+\dfrac{\lambda_0}{[-u]_+ + \epsilon} v = x + [ − u ] + + ϵ λ 0 when x < − λ 0 [ − u ] + + ϵ x<-\dfrac{\lambda_0}{[-u]_+ + \epsilon} x < − [ − u ] + + ϵ λ 0 . In the remaining interval the minimizer is v = 0 v=0 v = 0 . □
Proposition 22 (Connection between coefficient error and excess risk). Suppose that for some universal constant c > 0 c>0 c > 0 , E [ ( w t − r w ⋆ ) 2 ] ≥ c \mathbb{E}\left[(w_t-rw^\star)^2\right]\geq c E [ ( w t − r w ⋆ ) 2 ] ≥ c then there is a universal constant c ′ > 0 c'>0 c ′ > 0 , such that, L ( w t ) − inf w L ( w ) ≥ c ′ 1 ( 1 + ν ) 6 \mathcal{L}(w_t)-\inf_w\mathcal{L}(w)\geq c'\dfrac{1}{(1+\nu)^6} L ( w t ) − inf w L ( w ) ≥ c ′ ( 1 + ν ) 6 1 for every t t t ,
L ( w t ) − inf w L ( w ) ≤ 1 8 E [ ( w t − r w ⋆ ) 2 ] . \mathcal{L}(w_t)-\inf_w\mathcal{L}(w)\leq\frac{1}{8}\mathbb{E}\left[(w_t-rw^\star)^2\right]. L ( w t ) − w inf L ( w ) ≤ 8 1 E [ ( w t − r w ⋆ ) 2 ] . If we further assume that E = E [ ( w t − r w ⋆ ) 2 ] < ∞ \mathbb{E}=\mathbb{E}[(w_t-rw^\star)^2]<\infty E = E [( w t − r w ⋆ ) 2 ] < ∞ . Then,
1 8 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } E [ ( w t − r w ⋆ ) 2 ] ≤ L ( w t ) − inf w L ( w ) . (39) \frac{1}{8}\min\left\{\mathbb{E}\left[e^{-(r\nu+\sqrt{\mathbb{E}})|Z|}\right],\mathbb{E}\left[Z^2e^{-(r\nu+\sqrt{\mathbb{E}})|Z|}\right]\right\}\mathbb{E}\left[(w_t-rw^\star)^2\right]\leq\mathcal{L}(w_t)-\inf_w\mathcal{L}(w). \tag*{(39)} 8 1 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } E [ ( w t − r w ⋆ ) 2 ] ≤ L ( w t ) − w inf L ( w ) . (39)
Proof. For the scalar parameters ( α t , σ t ) (\alpha_t,\sigma_t) ( α t , σ t ) , define the population logistic risk
L ( w t ) = E [ ϕ ( ν α t Z 1 + σ t Z 2 ) − π ρ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 ) ] . \mathcal{L}(w_t)=\mathbb{E}\left[\phi(\nu\alpha_t Z_1+\sigma_t Z_2)-\pi_\rho(\nu Z_1)(\nu\alpha_t Z_1+\sigma_t Z_2)\right]. L ( w t ) = E [ ϕ ( ν α t Z 1 + σ t Z 2 ) − π ρ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 ) ] .
and define the optimal population risk
L ⋆ = E [ ϕ ( r ν Z 1 ) − π ρ ( ν Z 1 ) r ν Z 1 ] . \mathcal{L}_\star=\mathbb{E}\left[\phi(r\nu Z_1)-\pi_\rho(\nu Z_1)r\nu Z_1\right]. L ⋆ = E [ ϕ ( r ν Z 1 ) − π ρ ( ν Z 1 ) r ν Z 1 ] .
where Z 1 , Z 2 ∼ i i d N ( 0 , 1 ) Z_1,Z_2\mathrel{\overset{\mathrm{iid}}{\sim}}\mathcal{N}(0,1) Z 1 , Z 2 ∼ iid N ( 0 , 1 ) . Indeed, L ⋆ \mathcal{L}_\star L ⋆ is the optimal population risk. By conditional Jensen’s inequality (on Z 2 Z_2 Z 2 ) and independence, L ( α , σ ) ≥ L ( α , 0 ) \mathcal{L}(\alpha,\sigma)\geq\mathcal{L}(\alpha,0) L ( α , σ ) ≥ L ( α , 0 ) , so the optimum satisfies σ ⋆ = 0 \sigma_\star=0 σ ⋆ = 0 . Moreover,
∂ ∂ α L ( α , 0 ) = ν 2 [ α h ( ν α ) − ( 1 − 2 ρ ) h ( ν ) ] , \frac{\partial}{\partial\alpha}\mathcal{L}(\alpha,0)=\nu^2\left[\alpha h(\nu\alpha)-(1-2\rho)h(\nu)\right], ∂ α ∂ L ( α , 0 ) = ν 2 [ α h ( ν α ) − ( 1 − 2 ρ ) h ( ν ) ] ,
and hence its unique zero is α = r \alpha=r α = r , since r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) rh(\nu r)=(1-2\rho)h(\nu) r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) by Lemma 16 . Finally, strict convexity of the logistic loss implies that this stationary point is the unique global minimizer, and therefore inf w L ( w ) = L ( r , 0 ) \inf_w\mathcal{L}(w)=\mathcal{L}(r,0) inf w L ( w ) = L ( r , 0 ) .
By the definitions of α t \alpha_t α t and σ t \sigma_t σ t , E [ w ⋆ w t ] = ν 2 α t \mathbb{E}[w^\star w_t]=\nu^2\alpha_t E [ w ⋆ w t ] = ν 2 α t , E [ w t 2 ] = ν 2 α t 2 + σ t 2 \mathbb{E}[w_t^2]=\nu^2\alpha_t^2+\sigma_t^2 E [ w t 2 ] = ν 2 α t 2 + σ t 2 , E [ ( w ⋆ ) 2 ] = ν 2 \mathbb{E}[(w^\star)^2]=\nu^2 E [( w ⋆ ) 2 ] = ν 2 . Therefore,
E [ ( w t − r w ⋆ ) 2 ] = E [ w t 2 ] − 2 r E [ w ⋆ w t ] + r 2 E [ ( w ⋆ ) 2 ] = ν 2 α t 2 + σ t 2 − 2 r ν 2 α t + r 2 ν 2 = ν 2 ( α t − r ) 2 + σ t 2 . (40) \begin{aligned}
\mathbb{E}\left[(w_t-rw^\star)^2\right]
&=\mathbb{E}[w_t^2]-2r\mathbb{E}[w^\star w_t]+r^2\mathbb{E}[(w^\star)^2]\\
&=\nu^2\alpha_t^2+\sigma_t^2-2r\nu^2\alpha_t+r^2\nu^2\\
&=\nu^2(\alpha_t-r)^2+\sigma_t^2. \tag*{(40)}
\end{aligned} E [ ( w t − r w ⋆ ) 2 ] = E [ w t 2 ] − 2 r E [ w ⋆ w t ] + r 2 E [( w ⋆ ) 2 ] = ν 2 α t 2 + σ t 2 − 2 r ν 2 α t + r 2 ν 2 = ν 2 ( α t − r ) 2 + σ t 2 . (40)
The defining equation for r r r (Lemma 16 ) implies E [ Z 1 ( ϕ ′ ( r ν Z 1 ) − π ρ ( ν Z 1 ) ) ] = 0 \mathbb{E}[Z_1(\phi'(r\nu Z_1)-\pi_\rho(\nu Z_1))]=0 E [ Z 1 ( ϕ ′ ( r ν Z 1 ) − π ρ ( ν Z 1 ))] = 0 . Since Z 2 Z_2 Z 2 is independent of Z 1 Z_1 Z 1 and has zero mean,
E [ ( ϕ ′ ( r ν Z 1 ) − π ρ ( ν Z 1 ) ) ( ν ( α t − r ) Z 1 + σ t Z 2 ) ] = ν ( α t − r ) E [ Z 1 ( ϕ ′ ( r ν Z 1 ) − π ρ ( ν Z 1 ) ) ] + σ t E [ ϕ ′ ( r ν Z 1 ) − π ρ ( ν Z 1 ) ] E [ Z 2 ] = 0. \begin{aligned}
\mathbb{E}\left[(\phi'(r\nu Z_1)-\pi_\rho(\nu Z_1))(\nu(\alpha_t-r)Z_1+\sigma_t Z_2)\right]
&=\nu(\alpha_t-r)\mathbb{E}\left[Z_1(\phi'(r\nu Z_1)-\pi_\rho(\nu Z_1))\right]\\
&\quad+\sigma_t\mathbb{E}\left[\phi'(r\nu Z_1)-\pi_\rho(\nu Z_1)\right]\mathbb{E}[Z_2]=0.
\end{aligned} E [ ( ϕ ′ ( r ν Z 1 ) − π ρ ( ν Z 1 )) ( ν ( α t − r ) Z 1 + σ t Z 2 ) ] = ν ( α t − r ) E [ Z 1 ( ϕ ′ ( r ν Z 1 ) − π ρ ( ν Z 1 )) ] + σ t E [ ϕ ′ ( r ν Z 1 ) − π ρ ( ν Z 1 ) ] E [ Z 2 ] = 0.
Consequently,
L ( w t ) − inf w L ( w ) = E [ ϕ ( ν α t Z 1 + σ t Z 2 ) − ϕ ( r ν Z 1 ) − ϕ ′ ( r ν Z 1 ) ( ν ( α t − r ) Z 1 + σ t Z 2 ) ] . (41) \mathcal{L}(w_t)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})=\mathbb{E}\left[\phi(\nu\alpha_t Z_1+\sigma_t Z_2)-\phi(r\nu Z_1)-\phi'(r\nu Z_1)(\nu(\alpha_t-r)Z_1+\sigma_t Z_2)\right]. \tag*{(41)} L ( w t ) − w inf L ( w ) = E [ ϕ ( ν α t Z 1 + σ t Z 2 ) − ϕ ( r ν Z 1 ) − ϕ ′ ( r ν Z 1 ) ( ν ( α t − r ) Z 1 + σ t Z 2 ) ] . (41)
Taylor’s theorem gives
ϕ ( ν α t Z 1 + σ t Z 2 ) − ϕ ( r ν Z 1 ) − ϕ ′ ( r ν Z 1 ) ( ν ( α t − r ) Z 1 + σ t Z 2 ) = ( ν ( α t − r ) Z 1 + σ t Z 2 ) 2 ∫ 0 1 ( 1 − u ) ϕ ′ ′ ( r ν Z 1 + u ( ν ( α t − r ) Z 1 + σ t Z 2 ) ) d u . (42) \begin{aligned}
\phi(\nu\alpha_t Z_1+\sigma_t Z_2)-\phi(r\nu Z_1)-\phi'(r\nu Z_1)(\nu(\alpha_t-r)Z_1+\sigma_t Z_2)\\
={}&(\nu(\alpha_t-r)Z_1+\sigma_t Z_2)^2\int_0^1(1-u)\phi''(r\nu Z_1+u(\nu(\alpha_t-r)Z_1+\sigma_t Z_2))\,\mathrm{d}u. \tag*{(42)}
\end{aligned} ϕ ( ν α t Z 1 + σ t Z 2 ) − ϕ ( r ν Z 1 ) − ϕ ′ ( r ν Z 1 ) ( ν ( α t − r ) Z 1 + σ t Z 2 ) = ( ν ( α t − r ) Z 1 + σ t Z 2 ) 2 ∫ 0 1 ( 1 − u ) ϕ ′′ ( r ν Z 1 + u ( ν ( α t − r ) Z 1 + σ t Z 2 )) d u . (42)
Since 0 < ϕ ′ ′ ( x ) ≤ 1 4 0<\phi''(x)\le\frac{1}{4} 0 < ϕ ′′ ( x ) ≤ 4 1 , equations Eqs. (41) and (42) imply
L ( w T ) − inf w L ( w ) ≤ 1 4 ∫ 0 1 ( 1 − u ) d u E [ ( ν ( α t − r ) Z 1 + σ t Z 2 ) 2 ] = 1 8 ( ν 2 ( α t − r ) 2 + σ t 2 ) . \mathcal{L}(w_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\le\frac{1}{4}\int_0^1(1-u)\,\mathrm{d}u\,\mathbb{E}\left[(\nu(\alpha_t-r)Z_1+\sigma_t Z_2)^2\right]=\frac{1}{8}\left(\nu^2(\alpha_t-r)^2+\sigma_t^2\right). L ( w T ) − w inf L ( w ) ≤ 4 1 ∫ 0 1 ( 1 − u ) d u E [ ( ν ( α t − r ) Z 1 + σ t Z 2 ) 2 ] = 8 1 ( ν 2 ( α t − r ) 2 + σ t 2 ) .
We now prove the lower bound on bounded-error regions. For every x x x ,
ϕ ′ ′ ( x ) = e x ( 1 + e x ) 2 ≥ 1 4 e − ∣ x ∣ . \phi''(x)=\frac{e^x}{(1+e^x)^2}\ge\frac{1}{4}e^{-|x|}. ϕ ′′ ( x ) = ( 1 + e x ) 2 e x ≥ 4 1 e − ∣ x ∣ .
Suppose that ν 2 ( α t − r ) 2 + σ t 2 ≤ E \nu^2(\alpha_t-r)^2+\sigma_t^2\le E ν 2 ( α t − r ) 2 + σ t 2 ≤ E . Then
ν 2 α t 2 + σ t 2 ≤ r ν + ν 2 ( α t − r ) 2 + σ t 2 ≤ r ν + E . \sqrt{\nu^2\alpha_t^2+\sigma_t^2}\le r\nu+\sqrt{\nu^2(\alpha_t-r)^2+\sigma_t^2}\le r\nu+\sqrt{E}. ν 2 α t 2 + σ t 2 ≤ r ν + ν 2 ( α t − r ) 2 + σ t 2 ≤ r ν + E .
Therefore, for every u ∈ [ 0 , 1 ] u\in[0,1] u ∈ [ 0 , 1 ] , r ν Z 1 + u ( ν ( α t − r ) Z 1 + σ t Z 2 ) r\nu Z_1+u(\nu(\alpha_t-r)Z_1+\sigma_t Z_2) r ν Z 1 + u ( ν ( α t − r ) Z 1 + σ t Z 2 ) is a centered Gaussian random variable whose standard deviation is at most r ν + E r\nu+\sqrt{E} r ν + E . By rotational invariance of ( Z 1 , Z 2 ) (Z_1,Z_2) ( Z 1 , Z 2 ) , for every centered Gaussian linear form with standard deviation at most r ν + E r\nu+\sqrt{E} r ν + E , and every other Gaussian linear form a Z 1 + b Z 2 aZ_1+bZ_2 a Z 1 + b Z 2 ,
E [ ϕ ′ ′ ( c Z 1 + d Z 2 ) ( a Z 1 + b Z 2 ) 2 ] ≥ 1 4 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } ( a 2 + b 2 ) . \mathbb{E}\left[\phi''(cZ_1+dZ_2)(aZ_1+bZ_2)^2\right]\ge\frac{1}{4}\min\left\{\mathbb{E}\left[e^{-(r\nu+\sqrt{E})|Z|}\right],\mathbb{E}\left[Z^2e^{-(r\nu+\sqrt{E})|Z|}\right]\right\}(a^2+b^2). E [ ϕ ′′ ( c Z 1 + d Z 2 ) ( a Z 1 + b Z 2 ) 2 ] ≥ 4 1 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } ( a 2 + b 2 ) .
To see this, rotate the Gaussian coordinates so that c Z 1 + d Z 2 = q Z ~ 1 cZ_1+dZ_2=q\widetilde{Z}_1 c Z 1 + d Z 2 = q Z 1 , with q 2 = c 2 + d 2 q^2=c^2+d^2 q 2 = c 2 + d 2 . Writing a Z 1 + b Z 2 = a ∥ Z ~ 1 + a ⊥ Z ~ 2 aZ_1+bZ_2=a_{\parallel}\widetilde{Z}_1+a_{\perp}\widetilde{Z}_2 a Z 1 + b Z 2 = a ∥ Z 1 + a ⊥ Z 2 , with some a ∥ , a ⊥ a_{\parallel},a_{\perp} a ∥ , a ⊥ (that make an orthogonal rotation) the cross term vanishes, and
E [ ϕ ′ ′ ( q Z ~ 1 ) ( a ∥ Z ~ 1 + a ⊥ Z ~ 2 ) 2 ] = a ∥ 2 E [ Z ~ 1 2 ϕ ′ ′ ( q Z ~ 1 ) ] + a ⊥ 2 E [ ϕ ′ ′ ( q Z ~ 1 ) ] ≥ 1 4 a ∥ 2 E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] + 1 4 a ⊥ 2 E [ e − ( r ν + E ) ∣ Z ∣ ] ≥ 1 4 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } ( a ∥ 2 + a ⊥ 2 ) . \begin{aligned}
\mathbb{E}\left[\phi''(q\widetilde{Z}_1)(a_{\parallel}\widetilde{Z}_1+a_{\perp}\widetilde{Z}_2)^2\right]
&=a_{\parallel}^2\mathbb{E}\left[\widetilde{Z}_1^2\phi''(q\widetilde{Z}_1)\right]+a_{\perp}^2\mathbb{E}\left[\phi''(q\widetilde{Z}_1)\right]\\
&\ge\frac{1}{4}a_{\parallel}^2\mathbb{E}\left[Z^2e^{-(r\nu+\sqrt{E})|Z|}\right]+\frac{1}{4}a_{\perp}^2\mathbb{E}\left[e^{-(r\nu+\sqrt{E})|Z|}\right]\\
&\ge\frac{1}{4}\min\left\{\mathbb{E}\left[e^{-(r\nu+\sqrt{E})|Z|}\right],\mathbb{E}\left[Z^2e^{-(r\nu+\sqrt{E})|Z|}\right]\right\}(a_{\parallel}^2+a_{\perp}^2).
\end{aligned} E [ ϕ ′′ ( q Z 1 ) ( a ∥ Z 1 + a ⊥ Z 2 ) 2 ] = a ∥ 2 E [ Z 1 2 ϕ ′′ ( q Z 1 ) ] + a ⊥ 2 E [ ϕ ′′ ( q Z 1 ) ] ≥ 4 1 a ∥ 2 E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] + 4 1 a ⊥ 2 E [ e − ( r ν + E ) ∣ Z ∣ ] ≥ 4 1 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } ( a ∥ 2 + a ⊥ 2 ) .
Since,
E [ ( ν ( α t − r ) Z 1 + σ t Z 2 ) 2 ] = ν 2 ( α t − r ) 2 + σ t 2 , \mathbb{E}\left[(\nu(\alpha_t-r)Z_1+\sigma_t Z_2)^2\right]=\nu^2(\alpha_t-r)^2+\sigma_t^2, E [ ( ν ( α t − r ) Z 1 + σ t Z 2 ) 2 ] = ν 2 ( α t − r ) 2 + σ t 2 ,
we obtain
L ( w T ) − inf w L ( w ) ≥ 1 4 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } ∫ 0 1 ( 1 − u ) d u ( ν 2 ( α t − r ) 2 + σ t 2 ) = 1 8 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } ( ν 2 ( α t − r ) 2 + σ t 2 ) . \begin{aligned}
\mathcal{L}(w_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\ge\frac{1}{4}\min\left\{\mathbb{E}\left[e^{-(r\nu+\sqrt{E})|Z|}\right],\mathbb{E}\left[Z^2e^{-(r\nu+\sqrt{E})|Z|}\right]\right\}\int_0^1(1-u)\,\mathrm{d}u\left(\nu^2(\alpha_t-r)^2+\sigma_t^2\right) \\
&=\frac{1}{8}\min\left\{\mathbb{E}\left[e^{-(r\nu+\sqrt{E})|Z|}\right],\mathbb{E}\left[Z^2e^{-(r\nu+\sqrt{E})|Z|}\right]\right\}\left(\nu^2(\alpha_t-r)^2+\sigma_t^2\right).
\end{aligned} L ( w T ) − w inf L ( w ) ≥ 4 1 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } ∫ 0 1 ( 1 − u ) d u ( ν 2 ( α t − r ) 2 + σ t 2 ) = 8 1 min { E [ e − ( r ν + E ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E ) ∣ Z ∣ ] } ( ν 2 ( α t − r ) 2 + σ t 2 ) .
This proves Eq. (39) . It remains to show that a uniform positive coefficient-error floor implies a positive risk floor even when the iterates are not known to remain bounded,
min { π ρ ( u ) , 1 − π ρ ( u ) } = ρ + 1 − 2 ρ 1 + e ∣ u ∣ . \min\{\pi_\rho(u),1-\pi_\rho(u)\}=\rho+\frac{1-2\rho}{1+e^{|u|}}. min { π ρ ( u ) , 1 − π ρ ( u )} = ρ + 1 + e ∣ u ∣ 1 − 2 ρ .
For fixed u u u the right-hand side is increasing in ρ \rho ρ , therefore its minimum over 0 ≤ ρ < 1 2 0\le\rho<\frac{1}{2} 0 ≤ ρ < 2 1 is attained at ρ = 0 \rho=0 ρ = 0 , hence
min { π ρ ( u ) , 1 − π ρ ( u ) } ≥ 1 1 + e ∣ u ∣ ≥ 1 2 e − ∣ u ∣ . \min\{\pi_\rho(u),1-\pi_\rho(u)\}\ge\frac{1}{1+e^{|u|}}\ge\frac{1}{2}e^{-|u|}. min { π ρ ( u ) , 1 − π ρ ( u )} ≥ 1 + e ∣ u ∣ 1 ≥ 2 1 e − ∣ u ∣ .
Thus for a 2 + b 2 = 1 a^2+b^2=1 a 2 + b 2 = 1 ,
E [ min { π ρ ( ν Z 1 ) , 1 − π ρ ( ν Z 1 ) } ∣ a Z 1 + b Z 2 ∣ ] ≥ 1 2 E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ] . \mathbb{E}\left[\min\{\pi_\rho(\nu Z_1),1-\pi_\rho(\nu Z_1)\}|aZ_1+bZ_2|\right]\ge\frac{1}{2}\mathbb{E}\left[e^{-\nu|Z_1|}|aZ_1+bZ_2|\right]. E [ min { π ρ ( ν Z 1 ) , 1 − π ρ ( ν Z 1 )} ∣ a Z 1 + b Z 2 ∣ ] ≥ 2 1 E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ] .
First, if a ≥ 1 2 a\ge\frac{1}{\sqrt{2}} a ≥ 2 1 then,
E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ] = E Z 1 [ e − ν ∣ Z 1 ∣ E [ ∣ a Z 1 + b Z 2 ∣ ∣ Z 1 ] ] ≥ E Z 1 [ e − ν ∣ Z 1 ∣ ∣ E [ a Z 1 + b Z 2 ∣ Z 1 ] ∣ ] ≥ 1 2 E [ e − ν ∣ Z 1 ∣ ∣ Z 1 ∣ ] . \begin{aligned}
\mathbb{E}\left[e^{-\nu|Z_1|}|aZ_1+bZ_2|\right]=\mathbb{E}_{Z_1}\left[e^{-\nu|Z_1|}\mathbb{E}\left[|aZ_1+bZ_2|\mid Z_1\right]\right] \\
&\ge\mathbb{E}_{Z_1}\left[e^{-\nu|Z_1|}\left|\mathbb{E}[aZ_1+bZ_2\mid Z_1]\right|\right]\ge\frac{1}{\sqrt{2}}\mathbb{E}\left[e^{-\nu|Z_1|}|Z_1|\right].
\end{aligned} E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ] = E Z 1 [ e − ν ∣ Z 1 ∣ E [ ∣ a Z 1 + b Z 2 ∣ ∣ Z 1 ] ] ≥ E Z 1 [ e − ν ∣ Z 1 ∣ ∣ E [ a Z 1 + b Z 2 ∣ Z 1 ] ∣ ] ≥ 2 1 E [ e − ν ∣ Z 1 ∣ ∣ Z 1 ∣ ] .
which evaluates to,
E [ a ∣ Z ∣ e − ν ∣ Z ∣ ] ≥ 2 a φ ( 1 ) e − 1 ∫ 0 1 1 + ν ∣ Z ∣ d Z = φ ( 1 ) e − 1 2 ( 1 + ν ) 2 . \mathbb{E}\left[a|Z|e^{-\nu|Z|}\right]\ge2a\varphi(1)e^{-1}\int_0^{\frac{1}{1+\nu}}|Z|\,\mathrm{d}Z=\frac{\varphi(1)e^{-1}}{2(1+\nu)^2}. E [ a ∣ Z ∣ e − ν ∣ Z ∣ ] ≥ 2 a φ ( 1 ) e − 1 ∫ 0 1 + ν 1 ∣ Z ∣ d Z = 2 ( 1 + ν ) 2 φ ( 1 ) e − 1 .
Otherwise b ≥ 1 2 b\ge\frac{1}{\sqrt{2}} b ≥ 2 1 and then since E ∣ x + b Z 2 ∣ \mathbb{E}|x+bZ_2| E ∣ x + b Z 2 ∣ is minimized at x = 0 x=0 x = 0 .
E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ] = E Z 1 E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ∣ Z 1 ] ≥ ∣ b ∣ E ∣ e − ν ∣ Z 1 ∣ ∣ ∣ Z 2 ∣ ≥ 2 φ ( 1 ) e − 1 π ( 1 + ν ) . \mathbb{E}\left[e^{-\nu|Z_1|}|aZ_1+bZ_2|\right]=\mathbb{E}_{Z_1}\mathbb{E}\left[e^{-\nu|Z_1|}|aZ_1+bZ_2|\mid Z_1\right]\ge|b|\mathbb{E}|e^{-\nu|Z_1|}||Z_2|\ge\frac{2\varphi(1)e^{-1}}{\sqrt{\pi}(1+\nu)}. E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ] = E Z 1 E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ∣ Z 1 ] ≥ ∣ b ∣ E ∣ e − ν ∣ Z 1 ∣ ∣∣ Z 2 ∣ ≥ π ( 1 + ν ) 2 φ ( 1 ) e − 1 .
Then,
L t = E [ ϕ ( ν α t Z 1 + σ t Z 2 ) − π ρ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 ) ] ≥ E [ min { π ρ ( ν Z 1 ) , 1 − π ρ ( ν Z 1 ) } ∣ ν α t Z 1 + σ t Z 2 ∣ ] ≥ 1 2 E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ] ≥ φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ν 2 α t 2 + σ t 2 . \begin{aligned}
\mathcal{L}_t=\mathbb{E}\left[\phi(\nu\alpha_tZ_1+\sigma_tZ_2)-\pi_\rho(\nu Z_1)(\nu\alpha_tZ_1+\sigma_tZ_2)\right] \\
&\ge\mathbb{E}\left[\min\{\pi_\rho(\nu Z_1),1-\pi_\rho(\nu Z_1)\}|\nu\alpha_tZ_1+\sigma_tZ_2|\right]\ge\frac{1}{2}\mathbb{E}\left[e^{-\nu|Z_1|}|aZ_1+bZ_2|\right] \\
&\ge\frac{\varphi(1)e^{-1}}{2(1+\nu)^2}\sqrt{\nu^2\alpha_t^2+\sigma_t^2}.
\end{aligned} L t = E [ ϕ ( ν α t Z 1 + σ t Z 2 ) − π ρ ( ν Z 1 ) ( ν α t Z 1 + σ t Z 2 ) ] ≥ E [ min { π ρ ( ν Z 1 ) , 1 − π ρ ( ν Z 1 )} ∣ ν α t Z 1 + σ t Z 2 ∣ ] ≥ 2 1 E [ e − ν ∣ Z 1 ∣ ∣ a Z 1 + b Z 2 ∣ ] ≥ 2 ( 1 + ν ) 2 φ ( 1 ) e − 1 ν 2 α t 2 + σ t 2 .
Moreover,
ν 2 α t 2 + σ t 2 ≥ ν 2 ( α t − r ) 2 + σ t 2 − r ν . \sqrt{\nu^2\alpha_t^2+\sigma_t^2}\ge\sqrt{\nu^2(\alpha_t-r)^2+\sigma_t^2}-r\nu. ν 2 α t 2 + σ t 2 ≥ ν 2 ( α t − r ) 2 + σ t 2 − r ν .
Since the zero predictor has population risk log 2 \log2 log 2 , L ⋆ ≤ log 2 \mathcal{L}_\star\le\log2 L ⋆ ≤ log 2 . Set E 0 = ( r ν + 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) φ ( 1 ) e − 1 ) 2 E_0=\left(r\nu+\frac{2(1+\nu)^2(\log(2)+1)}{\varphi(1)e^{-1}}\right)^2 E 0 = ( r ν + φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( l o g ( 2 ) + 1 ) ) 2 . If ν 2 ( α t − r ) 2 + σ t 2 ≥ E 0 \nu^2(\alpha_t-r)^2+\sigma_t^2\ge E_0 ν 2 ( α t − r ) 2 + σ t 2 ≥ E 0 , then
L ( w t ) ≥ φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( r ν + 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) φ ( 1 ) e − 1 − r ν ) = log 2 + 1 ≥ L ⋆ + 1. \mathcal{L}(\mathbf{w}_t)\ge\frac{\varphi(1)e^{-1}}{2(1+\nu)^2}\left(r\nu+\frac{2(1+\nu)^2(\log(2)+1)}{\varphi(1)e^{-1}}-r\nu\right)=\log2+1\ge\mathcal{L}_\star+1. L ( w t ) ≥ 2 ( 1 + ν ) 2 φ ( 1 ) e − 1 ( r ν + φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) − r ν ) = log 2 + 1 ≥ L ⋆ + 1.
Hence L ( w T ) − inf w L ( w ) ≥ 1 \mathcal{L}(\mathbf{w}_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\ge1 L ( w T ) − inf w L ( w ) ≥ 1 . Combining both cases,
L ( w T ) − inf w L ( w ) ≥ min { 1 , c 8 min { E [ e − ( r ν + E 0 ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E 0 ) ∣ Z ∣ ] } } . (43) \mathcal{L}(\mathbf{w}_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\ge\min\left\{1,\frac{c}{8}\min\left\{\mathbb{E}\left[e^{-(r\nu+\sqrt{E_0})|Z|}\right],\mathbb{E}\left[Z^2e^{-(r\nu+\sqrt{E_0})|Z|}\right]\right\}\right\}. \tag*{(43)} L ( w T ) − w inf L ( w ) ≥ min { 1 , 8 c min { E [ e − ( r ν + E 0 ) ∣ Z ∣ ] , E [ Z 2 e − ( r ν + E 0 ) ∣ Z ∣ ] } } . (43)
Restricting 0 ≤ Z ≤ ( 1 + r ν + E 0 ) − 1 0\le Z\le(1+r\nu+\sqrt{E_0})^{-1} 0 ≤ Z ≤ ( 1 + r ν + E 0 ) − 1 , we have φ ( Z ) ≥ φ ( 1 ) \varphi(Z)\ge\varphi(1) φ ( Z ) ≥ φ ( 1 ) and e − ( r ν + E 0 ) Z ≥ e − 1 e^{-(r\nu+\sqrt{E_0})Z}\ge e^{-1} e − ( r ν + E 0 ) Z ≥ e − 1 , so,
E [ e − ( r ν + E 0 ) ∣ Z ∣ ] = E [ e − ( 2 r ν + 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) φ ( 1 ) e − 1 ) ∣ Z ∣ ] \mathbb{E}\left[e^{-(r\nu+\sqrt{E_0})|Z|}\right]=\mathbb{E}\left[e^{-\left(2r\nu+\frac{2(1+\nu)^2(\log(2)+1)}{\varphi(1)e^{-1}}\right)|Z|}\right] E [ e − ( r ν + E 0 ) ∣ Z ∣ ] = E [ e − ( 2 r ν + φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( l o g ( 2 ) + 1 ) ) ∣ Z ∣ ]
≥ 2 φ ( 1 ) e − 1 ∫ 0 1 1 + 2 r ν + 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) φ ( 1 ) e − 1 d Z = 2 φ ( 1 ) e − 1 1 + 2 r ν + 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) φ ( 1 ) e − 1 , \ge2\varphi(1)e^{-1}\int_0^{\frac{1}{1+2r\nu+\frac{2(1+\nu)^2(\log(2)+1)}{\varphi(1)e^{-1}}}}\mathrm{d}Z
=\frac{2\varphi(1)e^{-1}}{1+2r\nu+\frac{2(1+\nu)^2(\log(2)+1)}{\varphi(1)e^{-1}}}, ≥ 2 φ ( 1 ) e − 1 ∫ 0 1 + 2 r ν + φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( l o g ( 2 ) + 1 ) 1 d Z = 1 + 2 r ν + φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( l o g ( 2 ) + 1 ) 2 φ ( 1 ) e − 1 ,
E [ Z 2 e − ( r ν + E 0 ) ∣ Z ∣ ] = E [ Z 2 e − ( 2 r ν + 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) 4 φ ( 1 ) e − 1 ) ∣ Z ∣ ] \mathbb{E}\left[Z^2e^{-(r\nu+\sqrt{E_0})|Z|}\right]=\mathbb{E}\left[Z^2e^{-\left(2r\nu+\frac{2(1+\nu)^2(\log(2)+1)}{4\varphi(1)e^{-1}}\right)|Z|}\right] E [ Z 2 e − ( r ν + E 0 ) ∣ Z ∣ ] = E [ Z 2 e − ( 2 r ν + 4 φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( l o g ( 2 ) + 1 ) ) ∣ Z ∣ ]
≥ 2 φ ( 1 ) e − 1 ∫ 0 1 1 + 2 r ν + 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) φ ( 1 ) e − 1 Z 2 d Z = 2 φ ( 1 ) e − 1 3 ( 1 + 2 r ν + 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) φ ( 1 ) e − 1 ) 3 . \ge2\varphi(1)e^{-1}\int_0^{\frac{1}{1+2r\nu+\frac{2(1+\nu)^2(\log(2)+1)}{\varphi(1)e^{-1}}}}Z^2\,\mathrm{d}Z
=\frac{2\varphi(1)e^{-1}}{3\left(1+2r\nu+\frac{2(1+\nu)^2(\log(2)+1)}{\varphi(1)e^{-1}}\right)^3}. ≥ 2 φ ( 1 ) e − 1 ∫ 0 1 + 2 r ν + φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( l o g ( 2 ) + 1 ) 1 Z 2 d Z = 3 ( 1 + 2 r ν + φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( l o g ( 2 ) + 1 ) ) 3 2 φ ( 1 ) e − 1 .
Since r ≤ 1 r\le1 r ≤ 1 , we have
2 φ ( 1 ) e − 1 3 ( 1 + 2 r ν + 2 ( 1 + ν ) 2 ( log ( 2 ) + 1 ) φ ( 1 ) e − 1 ) 3 ≤ C 1 1 ( 1 + ν ) 6 \frac{2\varphi(1)e^{-1}}{3\left(1+2r\nu+\frac{2(1+\nu)^2(\log(2)+1)}{\varphi(1)e^{-1}}\right)^3}\le C_1\frac{1}{(1+\nu)^6} 3 ( 1 + 2 r ν + φ ( 1 ) e − 1 2 ( 1 + ν ) 2 ( l o g ( 2 ) + 1 ) ) 3 2 φ ( 1 ) e − 1 ≤ C 1 ( 1 + ν ) 6 1
for some universal constant $C_1>0.
Proofs for Section 4.2
Proof of thm:onehomogeneouserrorfloor
Recall Lemma 8. Assume that for every t t t , the regularizer is an order p > 0 p>0 p > 0 homogeneous, i.e., f ( c w , v ) = c p f ( w , v ) f(c\mathbf{w},\mathbf{v})=c^{p}f(\mathbf{w},\mathbf{v}) f ( c w , v ) = c p f ( w , v ) for every c ≥ 0 c\geq0 c ≥ 0 , w ∈ R n \mathbf{w}\in\mathbb{R}^{n} w ∈ R n and a fixed v ∈ R n \mathbf{v}\in\mathbb{R}^{n} v ∈ R n . Then there exists a universal constant C > 0 C>0 C > 0 such that for every T ≥ 1 T\geq1 T ≥ 1 and s ∈ ( 0 , 1 ) s\in(0,1) s ∈ ( 0 , 1 ) ,
L ( w T ) − inf w L ( w ) ≥ C min { 1 , ( 1 − 2 ρ ) 2 ( 1 + ν ) 2 max { 1 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } } 2 min { ν 2 , s δ } } . \mathcal{L}(\mathbf{w}_{T})-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\geq C\min\left\{1,\frac{(1-2\rho)^{2}}{(1+\nu)^{2}\max\left\{1,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}\right\}^{2}}\min\left\{\nu^{2},\frac{s}{\delta}\right\}\right\}. L ( w T ) − w inf L ( w ) ≥ C min ⎩ ⎨ ⎧ 1 , ( 1 + ν ) 2 max { 1 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } } 2 ( 1 − 2 ρ ) 2 min { ν 2 , δ s } ⎭ ⎬ ⎫ .
Proof. The first scalar equation gives
w t → d Prox λ σ t τ t f ( σ t τ t ( θ t w ⋆ + r t δ Z ) ) . \mathbf{w}_{t}\xrightarrow{d}\operatorname{Prox}_{\lambda\sigma_{t}\tau_{t}f}\left(\sigma_{t}\tau_{t}\left(\theta_{t}\mathbf{w}^{\star}+\frac{r_{t}}{\sqrt{\delta}}\mathbf{Z}\right)\right). w t d Prox λ σ t τ t f ( σ t τ t ( θ t w ⋆ + δ r t Z ) ) .
We first show directly from the fourth and sixth scalar equations that the fresh Gaussian noise inside this proximal map has non-vanishing variance. The optimality condition for the proximal map appearing in the fourth, fifth, and sixth equations is
ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) . \nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}-\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)=\gamma_{t}\phi'\left(\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)\right). ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) .
where the last transition comes from,
Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) = argmin u ∈ R { γ t ϕ ( u ) + 1 2 ( u − ν α t Z 1 − σ t Z 2 ) 2 } \operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)=\underset{u\in\mathbb{R}}{\operatorname{argmin}}\left\{\gamma_{t}\phi(u)+\frac{1}{2}\left(u-\nu\alpha_{t}Z_{1}-\sigma_{t}Z_{2}\right)^{2}\right\} Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) = u ∈ R argmin { γ t ϕ ( u ) + 2 1 ( u − ν α t Z 1 − σ t Z 2 ) 2 }
whose first-order stationary implies
0 = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) + Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) − ( ν α t Z 1 + σ t Z 2 ) 0=\gamma_{t}\phi'\left(\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)\right)+\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)-\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right) 0 = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) + Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) − ( ν α t Z 1 + σ t Z 2 )
Substituting this identity into the fourth scalar equation yields
r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] . (44) r_{t}^{2}=2\mathbb{E}\left[\left(1-\pi_{\rho}(\nu Z_{1})\right)\phi'\left(\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)\right)^{2}\right]. \tag*{(44)} r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] . (44)
The sixth scalar equation is
1 − γ t σ t τ t = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . 1-\frac{\gamma_{t}}{\sigma_{t}\tau_{t}}=\mathbb{E}\left[\frac{2\left(1-\pi_{\rho}(\nu Z_{1})\right)}{1+\gamma_{t}\phi''\left(\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)\right)}\right]. 1 − σ t τ t γ t = E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ( 1 − π ρ ( ν Z 1 ) ) ] .
Since E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ] = 1 \mathbb{E}\left[2\left(1-\pi_{\rho}(\nu Z_{1})\right)\right]=1 E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ] = 1 , subtract it from both sides, we obtain
1 σ t τ t = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] (45) \frac{1}{\sigma_{t}\tau_{t}}=2\mathbb{E}\left[\left(1-\pi_{\rho}(\nu Z_{1})\right)\frac{\phi''\left(\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)\right)}{1+\gamma_{t}\phi''\left(\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)\right)}\right] \tag*{(45)} σ t τ t 1 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] (45)
≤ 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . \leq2\mathbb{E}\left[\left(1-\pi_{\rho}(\nu Z_{1})\right)\phi''\left(\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)\right)\right]. ≤ 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] .
For ϕ ( u ) = log ( 1 + e u ) \phi(u)=\log(1+e^{u}) ϕ ( u ) = log ( 1 + e u ) , we have 0 ≤ ϕ ′ ′ ( u ) = ϕ ′ ( u ) ( 1 − ϕ ′ ( u ) ) ≤ ϕ ′ ( u ) 0\leq\phi''(u)=\phi'(u)(1-\phi'(u))\leq\phi'(u) 0 ≤ ϕ ′′ ( u ) = ϕ ′ ( u ) ( 1 − ϕ ′ ( u )) ≤ ϕ ′ ( u ) . Therefore,
1 σ t τ t ≤ 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] \frac{1}{\sigma_{t}\tau_{t}}\leq2\mathbb{E}\left[\left(1-\pi_{\rho}(\nu Z_{1})\right)\phi'\left(\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)\right)\right] σ t τ t 1 ≤ 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ]
≤ 2 E [ 1 − π ρ ( ν Z 1 ) ] E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] = 2 1 2 ⋅ r t 2 2 = r t , (46) \begin{aligned}
\leq2\sqrt{\mathbb{E}\left[1-\pi_{\rho}(\nu Z_{1})\right]\mathbb{E}\left[\left(1-\pi_{\rho}(\nu Z_{1})\right)\phi'\left(\operatorname{Prox}_{\gamma_{t}\phi}\left(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}\right)\right)^{2}\right]} \\
&=2\sqrt{\frac{1}{2}\cdot\frac{r_{t}^{2}}{2}}=r_{t}, \tag*{(46)}
\end{aligned} ≤ 2 E [ 1 − π ρ ( ν Z 1 ) ] E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] = 2 2 1 ⋅ 2 r t 2 = r t , (46)
where the second inequality is Cauchy–Schwarz and the penultimate equality uses (44) . Consequently, σ t τ t r t ≥ 1 \sigma_{t}\tau_{t}r_{t}\geq1 σ t τ t r t ≥ 1 . It follows that σ t τ t r t δ ≥ 1 δ \frac{\sigma_{t}\tau_{t}r_{t}}{\sqrt{\delta}}\geq\frac{1}{\sqrt{\delta}} δ σ t τ t r t ≥ δ 1 . We next use the assumption on f f f . By p p p -homogeneity,
f ( w , w t − 1 ) = { w p f ( 1 , w t − 1 ) , w ≥ 0 , ( − w ) p f ( − 1 , w t − 1 ) , w < 0. f(w,w_{t-1})=
\begin{cases}
w^{p}f(1,w_{t-1}), & w\geq0,\\
(-w)^{p}f(-1,w_{t-1}), & w<0.
\end{cases} f ( w , w t − 1 ) = { w p f ( 1 , w t − 1 ) , ( − w ) p f ( − 1 , w t − 1 ) , w ≥ 0 , w < 0.
Condition on w ⋆ w^{\star} w ⋆ and on all previous tasks. First, we assume that w ⋆ > 0 w^{\star}>0 w ⋆ > 0 . Since r > 0 r>0 r > 0 (see Lemma 16 ), the target r w ⋆ rw^{\star} r w ⋆ is positive. By the optimality condition for the proximal map, whenever w t > 0 w_t>0 w t > 0 ,
σ t τ t ( θ t w ⋆ + r t δ Z ) = w t + λ σ t τ t p f ( 1 , w t − 1 ) w t p − 1 . \sigma_t\tau_t\left(\theta_t w^{\star}+\frac{r_t}{\sqrt{\delta}}Z\right)=w_t+\lambda\sigma_t\tau_tpf(1,w_{t-1})w_t^{p-1}. σ t τ t ( θ t w ⋆ + δ r t Z ) = w t + λ σ t τ t p f ( 1 , w t − 1 ) w t p − 1 .
In particular, the input corresponding to the output r w ⋆ rw^{\star} r w ⋆ is r w ⋆ + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 rw^{\star}+\lambda\sigma_t\tau_tpf(1,w_{t-1})(rw^{\star})^{p-1} r w ⋆ + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 . Then, for every u ∈ [ r w ⋆ 2 , 3 r w ⋆ 2 ] u\in\left[\frac{rw^{\star}}{2},\frac{3rw^{\star}}{2}\right] u ∈ [ 2 r w ⋆ , 2 3 r w ⋆ ] , we have
1 + λ σ t τ t p ( p − 1 ) f ( 1 , w t − 1 ) u p − 2 ≤ 1 + λ σ t τ t p ( p − 1 ) f ( 1 , w t − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } ( r w ⋆ ) p − 2 ≤ max { 1 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } } ( 1 + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 2 ) = max { 1 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } } r w ⋆ + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 r w ⋆ . \begin{aligned}
1+\lambda\sigma_t\tau_tp(p-1)f(1,w_{t-1})u^{p-2}\leq1+\lambda\sigma_t\tau_tp(p-1)f(1,w_{t-1})\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}(rw^{\star})^{p-2} \\
&\leq\max\left\{1,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}\right\}\left(1+\lambda\sigma_t\tau_tpf(1,w_{t-1})(rw^{\star})^{p-2}\right) \\
&=\max\left\{1,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}\right\}\frac{rw^{\star}+\lambda\sigma_t\tau_tpf(1,w_{t-1})(rw^{\star})^{p-1}}{rw^{\star}}.
\end{aligned} 1 + λ σ t τ t p ( p − 1 ) f ( 1 , w t − 1 ) u p − 2 ≤ 1 + λ σ t τ t p ( p − 1 ) f ( 1 , w t − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } ( r w ⋆ ) p − 2 ≤ max { 1 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } } ( 1 + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 2 ) = max { 1 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } } r w ⋆ r w ⋆ + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 .
Hence, if ∣ w t − r w ⋆ ∣ < r w ⋆ 2 |w_t-rw^{\star}|<\frac{rw^{\star}}{2} ∣ w t − r w ⋆ ∣ < 2 r w ⋆ , then w t > 0 w_t>0 w t > 0 , and the mean-value theorem gives
∣ w t − r w ⋆ ∣ ≥ r w ⋆ max { 1 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } } ∣ σ t τ t ( θ t w ⋆ + r t δ Z ) − r w ⋆ − λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 ∣ r w ⋆ + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 . |w_t-rw^{\star}|\geq\frac{rw^{\star}}{\max\left\{1,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}\right\}}\frac{\left|\sigma_t\tau_t\left(\theta_t w^{\star}+\frac{r_t}{\sqrt{\delta}}Z\right)-rw^{\star}-\lambda\sigma_t\tau_tpf(1,w_{t-1})(rw^{\star})^{p-1}\right|}{rw^{\star}+\lambda\sigma_t\tau_tpf(1,w_{t-1})(rw^{\star})^{p-1}}. ∣ w t − r w ⋆ ∣ ≥ max { 1 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } } r w ⋆ r w ⋆ + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 σ t τ t ( θ t w ⋆ + δ r t Z ) − r w ⋆ − λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 .
If instead ∣ w t − r w ⋆ ∣ ≥ r w ⋆ 2 |w_t-rw^{\star}|\geq\frac{rw^{\star}}{2} ∣ w t − r w ⋆ ∣ ≥ 2 r w ⋆ , then, since the minimum below is at least r w ⋆ 2 \frac{rw^{\star}}{2} 2 r w ⋆ , combining the two cases yields
∣ w t − r w ⋆ ∣ ≥ r w ⋆ max { 2 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } } × min { 1 , ∣ σ t τ t ( θ t w ⋆ + r t δ Z ) − r w ⋆ − λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 ∣ r w ⋆ + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 } . (47) |w_t-rw^{\star}|\geq\frac{rw^{\star}}{\max\left\{2,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}\right\}}\times\min\left\{1,\frac{\left|\sigma_t\tau_t\left(\theta_t w^{\star}+\frac{r_t}{\sqrt{\delta}}Z\right)-rw^{\star}-\lambda\sigma_t\tau_tpf(1,w_{t-1})(rw^{\star})^{p-1}\right|}{rw^{\star}+\lambda\sigma_t\tau_tpf(1,w_{t-1})(rw^{\star})^{p-1}}\right\}. \tag*{(47)} ∣ w t − r w ⋆ ∣ ≥ max { 2 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } } r w ⋆ × min ⎩ ⎨ ⎧ 1 , r w ⋆ + λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 σ t τ t ( θ t w ⋆ + δ r t Z ) − r w ⋆ − λ σ t τ t p f ( 1 , w t − 1 ) ( r w ⋆ ) p − 1 ⎭ ⎬ ⎫ . (47)
The case w ⋆ < 0 w^{\star}<0 w ⋆ < 0 is identical, using f ( − 1 , w t − 1 ) f(-1,w_{t-1}) f ( − 1 , w t − 1 ) .
We next use the following bound. For every a ∈ R a\in\mathbb{R} a ∈ R , b > 0 b>0 b > 0 , and q > 0 q>0 q > 0 ,
E [ min { 1 , ( a + b Z − q ) 2 q 2 } ] ≥ 1 24 b 2 a 2 + b 2 . \mathbb{E}\left[\min\left\{1,\frac{(a+bZ-q)^2}{q^2}\right\}\right]\geq\frac{1}{24}\frac{b^2}{a^2+b^2}. E [ min { 1 , q 2 ( a + b Z − q ) 2 } ] ≥ 24 1 a 2 + b 2 b 2 .
It follows by writing Y = a + b Z − q Y=a+bZ-q Y = a + b Z − q , so we have E [ Y 2 ] = b 2 + ( a − q ) 2 \mathbb{E}[Y^2]=b^2+(a-q)^2 E [ Y 2 ] = b 2 + ( a − q ) 2 and
E [ Y 4 ] = 3 b 4 + 6 b 2 ( a − q ) 2 + ( a − q ) 4 ≤ 3 ( b 2 + ( a − q ) 2 ) 2 . \mathbb{E}[Y^4]=3b^4+6b^2(a-q)^2+(a-q)^4\leq3\left(b^2+(a-q)^2\right)^2. E [ Y 4 ] = 3 b 4 + 6 b 2 ( a − q ) 2 + ( a − q ) 4 ≤ 3 ( b 2 + ( a − q ) 2 ) 2 .
Therefore, by the Paley–Zygmund inequality, P ( Y 2 ≥ 1 2 E [ Y 2 ] ) ≥ 1 12 \mathbb{P}\left(Y^2\geq\frac{1}{2}\mathbb{E}[Y^2]\right)\geq\frac{1}{12} P ( Y 2 ≥ 2 1 E [ Y 2 ] ) ≥ 12 1 , and hence
E [ min { 1 , Y 2 q 2 } ] ≥ 1 12 min { 1 , b 2 + ( a − q ) 2 2 q 2 } ≥ 1 24 min { 1 , b 2 + ( a − q ) 2 q 2 } . \mathbb{E}\left[\min\left\{1,\frac{Y^2}{q^2}\right\}\right]\geq\frac{1}{12}\min\left\{1,\frac{b^2+(a-q)^2}{2q^2}\right\}\geq\frac{1}{24}\min\left\{1,\frac{b^2+(a-q)^2}{q^2}\right\}. E [ min { 1 , q 2 Y 2 } ] ≥ 12 1 min { 1 , 2 q 2 b 2 + ( a − q ) 2 } ≥ 24 1 min { 1 , q 2 b 2 + ( a − q ) 2 } .
Moreover,
b 2 + ( a − q ) 2 q 2 ≥ inf x > 0 b 2 + ( a − x ) 2 x 2 = b 2 a 2 + b 2 , \frac{b^2+(a-q)^2}{q^2}\geq\inf_{x>0}\frac{b^2+(a-x)^2}{x^2}=\frac{b^2}{a^2+b^2}, q 2 b 2 + ( a − q ) 2 ≥ x > 0 inf x 2 b 2 + ( a − x ) 2 = a 2 + b 2 b 2 ,
and the latter quantity is at most one, proving the claim. Squaring Eq. (47) and applying this bound conditionally on w ⋆ w^{\star} w ⋆ and all previous tasks gives
E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ r 2 w ⋆ 2 24 max { 2 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } } 2 ( σ t τ t r t δ ) 2 ( σ t τ t θ t w ⋆ ) 2 + ( σ t τ t r t δ ) 2 . \mathbb{E}\left[(w_t-rw^{\star})^2\mid w^{\star},\text{ previous tasks}\right]\geq\frac{r^2w^{\star2}}{24\max\left\{2,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}\right\}^2}\frac{\left(\frac{\sigma_t\tau_t r_t}{\sqrt{\delta}}\right)^2}{(\sigma_t\tau_t\theta_tw^{\star})^2+\left(\frac{\sigma_t\tau_t r_t}{\sqrt{\delta}}\right)^2}. E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ 24 max { 2 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } } 2 r 2 w ⋆ 2 ( σ t τ t θ t w ⋆ ) 2 + ( δ σ t τ t r t ) 2 ( δ σ t τ t r t ) 2 .
Furthermore,
θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] \theta_t = 2\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)\right] θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ]
≤ 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] \le2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)\right] ≤ 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ]
≤ 2 E [ 1 − π ρ ( ν Z 1 ) ] E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] = r t , \le2\sqrt{\mathbb{E}\left[1-\pi_\rho(\nu Z_1)\right]\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)^2\right]}=r_t, ≤ 2 E [ 1 − π ρ ( ν Z 1 ) ] E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] = r t ,
where we used π ρ ′ ( u ) = ( 1 − 2 ρ ) σ ( u ) ( 1 − σ ( u ) ) ≤ ( 1 − 2 ρ ) σ ( u ) = 1 − π ρ ( u ) \pi_\rho'(u)=(1-2\rho)\sigma(u)(1-\sigma(u))\le(1-2\rho)\sigma(u)=1-\pi_\rho(u) π ρ ′ ( u ) = ( 1 − 2 ρ ) σ ( u ) ( 1 − σ ( u )) ≤ ( 1 − 2 ρ ) σ ( u ) = 1 − π ρ ( u ) , E [ 1 − π ρ ( ν Z 1 ) ] = 1 2 \mathbb{E}[1-\pi_\rho(\nu Z_1)]=\frac{1}{2} E [ 1 − π ρ ( ν Z 1 )] = 2 1 , and (44) . Consequently, because θ t ≤ r t \theta_t\le r_t θ t ≤ r t ,
( σ t τ t r t δ ) 2 ( σ t τ t θ t w ⋆ ) 2 + ( σ t τ t r t δ ) 2 = r t 2 δ θ t 2 w ⋆ 2 + r t 2 ≥ 1 1 + δ w ⋆ 2 . \frac{\left(\frac{\sigma_t\tau_t r_t}{\sqrt{\delta}}\right)^2}{(\sigma_t\tau_t\theta_t w^\star)^2+\left(\frac{\sigma_t\tau_t r_t}{\sqrt{\delta}}\right)^2}
=\frac{r_t^2}{\delta\theta_t^2w^{\star2}+r_t^2}
\ge\frac{1}{1+\delta w^{\star2}}. ( σ t τ t θ t w ⋆ ) 2 + ( δ σ t τ t r t ) 2 ( δ σ t τ t r t ) 2 = δ θ t 2 w ⋆ 2 + r t 2 r t 2 ≥ 1 + δ w ⋆ 2 1 .
Thus
E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ r 2 w ⋆ 2 24 max { 2 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } 2 } ( 1 + δ w ⋆ 2 ) ≥ r 2 48 max { 2 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } 2 } min { w ⋆ 2 , 1 δ } . \begin{aligned}
\mathbb{E}\left[(w_t-rw^\star)^2\mid w^\star,\ \text{previous tasks}\right]\ge
\frac{r^2w^{\star2}}{24\max\left\{2,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}^2\right\}(1+\delta w^{\star2})} \\
&\ge\frac{r^2}{48\max\left\{2,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}^2\right\}}\min\left\{w^{\star2},\frac{1}{\delta}\right\}.
\end{aligned} E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ 24 max { 2 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } 2 } ( 1 + δ w ⋆ 2 ) r 2 w ⋆ 2 ≥ 48 max { 2 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } 2 } r 2 min { w ⋆ 2 , δ 1 } .
Averaging over the previous tasks and over w ⋆ w^\star w ⋆ yields
E [ ( w t − r w ⋆ ) 2 ] ≥ r 2 48 max { 2 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } 2 } E [ min { w ⋆ 2 , 1 δ } ] . \mathbb{E}\left[(w_t-rw^\star)^2\right]\ge
\frac{r^2}{48\max\left\{2,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}^2\right\}}
\mathbb{E}\left[\min\left\{w^{\star2},\frac{1}{\delta}\right\}\right]. E [ ( w t − r w ⋆ ) 2 ] ≥ 48 max { 2 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } 2 } r 2 E [ min { w ⋆ 2 , δ 1 } ] .
Finally, since w ⋆ = S ν s G w^\star=S\frac{\nu}{\sqrt{s}}G w ⋆ = S s ν G , where S ∼ Ber ( s ) S\sim\operatorname{Ber}(s) S ∼ Ber ( s ) , G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) we have E [ min { w ⋆ 2 , 1 δ } ] = s E G [ min { ν 2 s G 2 , 1 δ } ] \mathbb{E}\left[\min\left\{w^{\star2},\frac{1}{\delta}\right\}\right] =s\mathbb{E}_G\left[\min\left\{\frac{\nu^2}{s}G^2,\frac{1}{\delta}\right\}\right] E [ min { w ⋆ 2 , δ 1 } ] = s E G [ min { s ν 2 G 2 , δ 1 } ] .
On the event { ∣ G ∣ ≥ 1 } \{|G|\ge1\} { ∣ G ∣ ≥ 1 } , we have min { ν 2 s G 2 , 1 δ } ≥ min { ν 2 s , 1 δ } \min\left\{\frac{\nu^2}{s}G^2,\frac{1}{\delta}\right\}\ge\min\left\{\frac{\nu^2}{s},\frac{1}{\delta}\right\} min { s ν 2 G 2 , δ 1 } ≥ min { s ν 2 , δ 1 } , and therefore
s E G [ min { ν 2 s G 2 , 1 δ } ] ≥ s P ( ∣ G ∣ ≥ 1 ) min { ν 2 s , 1 δ } ≥ 3 10 min { ν 2 , s δ } . s\mathbb{E}_G\left[\min\left\{\frac{\nu^2}{s}G^2,\frac{1}{\delta}\right\}\right]\ge s\mathbb{P}(|G|\ge1)\min\left\{\frac{\nu^2}{s},\frac{1}{\delta}\right\}\ge\frac{3}{10}\min\left\{\nu^2,\frac{s}{\delta}\right\}. s E G [ min { s ν 2 G 2 , δ 1 } ] ≥ s P ( ∣ G ∣ ≥ 1 ) min { s ν 2 , δ 1 } ≥ 10 3 min { ν 2 , δ s } .
Hence,
E [ ( w t − r w ⋆ ) 2 ] ≥ r 2 160 max { 2 , ( p − 1 ) max { 2 2 − p , ( 3 2 ) p − 2 } 2 } min { ν 2 , s δ } . \mathbb{E}\left[(w_t-rw^\star)^2\right]\ge
\frac{r^2}{160\max\left\{2,(p-1)\max\left\{2^{2-p},\left(\frac{3}{2}\right)^{p-2}\right\}^2\right\}}
\min\left\{\nu^2,\frac{s}{\delta}\right\}. E [ ( w t − r w ⋆ ) 2 ] ≥ 160 max { 2 , ( p − 1 ) max { 2 2 − p , ( 2 3 ) p − 2 } 2 } r 2 min { ν 2 , δ s } .
It remains to consider 0 < p < 1 0<p<1 0 < p < 1 . Condition on w ⋆ w^\star w ⋆ and on all previous tasks. Suppose first that w t > 0 w_t>0 w t > 0 . Since w t w_t w t is a global minimizer of the proximal objective,
1 2 ( w t − σ t τ t ( θ t w ⋆ + r t δ Z ) ) 2 + λ σ t τ t f ( 1 , w t − 1 ) w t p ≤ 1 2 ( σ t τ t ( θ t w ⋆ + r t δ Z ) ) 2 . \frac{1}{2}\left(w_t-\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z\right)\right)^2+\lambda\sigma_t\tau_t f(1,w_{t-1})w_t^p
\le\frac{1}{2}\left(\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z\right)\right)^2. 2 1 ( w t − σ t τ t ( θ t w ⋆ + δ r t Z ) ) 2 + λ σ t τ t f ( 1 , w t − 1 ) w t p ≤ 2 1 ( σ t τ t ( θ t w ⋆ + δ r t Z ) ) 2 .
Moreover, since w t > 0 w_t>0 w t > 0 , the first-order condition gives
σ t τ t ( θ t w ⋆ + r t δ Z ) = w t + λ σ t τ t p f ( 1 , w t − 1 ) w t p − 1 . \sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z\right)
=w_t+\lambda\sigma_t\tau_tpf(1,w_{t-1})w_t^{p-1}. σ t τ t ( θ t w ⋆ + δ r t Z ) = w t + λ σ t τ t p f ( 1 , w t − 1 ) w t p − 1 .
Substituting this identity into the preceding inequality yields w t 2 − p ≥ 2 λ σ t τ t ( 1 − p ) f ( 1 , w t − 1 ) w_t^{2-p}\ge2\lambda\sigma_t\tau_t(1-p)f(1,w_{t-1}) w t 2 − p ≥ 2 λ σ t τ t ( 1 − p ) f ( 1 , w t − 1 ) . Consequently,
d d w t [ w t + λ σ t τ t p f ( 1 , w t − 1 ) w t p − 1 ] = 1 − λ σ t τ t p ( 1 − p ) f ( 1 , w t − 1 ) w t p − 2 ≥ 1 − p 2 > 0. \frac{\mathrm{d}}{\mathrm{d}w_t}\left[w_t+\lambda\sigma_t\tau_tpf(1,w_{t-1})w_t^{p-1}\right]
=1-\lambda\sigma_t\tau_tp(1-p)f(1,w_{t-1})w_t^{p-2}
\ge1-\frac{p}{2}>0. d w t d [ w t + λ σ t τ t p f ( 1 , w t − 1 ) w t p − 1 ] = 1 − λ σ t τ t p ( 1 − p ) f ( 1 , w t − 1 ) w t p − 2 ≥ 1 − 2 p > 0.
Hence the proximal map is increasing,
d d x Prox λ σ t τ t f ( x ) = 1 1 − λ σ t τ t p ( 1 − p ) f ( 1 , w t − 1 ) w t p − 2 ≥ 1. \frac{\mathrm{d}}{\mathrm{d}x}\operatorname{Prox}_{\lambda\sigma_t\tau_t f}(x)
=\frac{1}{1-\lambda\sigma_t\tau_tp(1-p)f(1,w_{t-1})w_t^{p-2}}\ge1. d x d Prox λ σ t τ t f ( x ) = 1 − λ σ t τ t p ( 1 − p ) f ( 1 , w t − 1 ) w t p − 2 1 ≥ 1.
Proof of thm:signed-weighted-l1-upper
The same conclusion holds on the negative nonzero branch, with f ( − 1 , w t − 1 ) f(-1,w_{t-1}) f ( − 1 , w t − 1 ) in place of f ( 1 , w t − 1 ) f(1,w_{t-1}) f ( 1 , w t − 1 ) . On the other hand if P ( w t = 0 ∣ w ⋆ , previous tasks ) ≥ 1 2 \mathbb{P}(w_t=0\mid w^\star,\text{ previous tasks})\ge\frac{1}{2} P ( w t = 0 ∣ w ⋆ , previous tasks ) ≥ 2 1 , then
E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ 1 2 r 2 w ⋆ 2 . \mathbb{E}\left[(w_t-rw^\star)^2\mid w^\star,\text{ previous tasks}\right]\ge\frac{1}{2}r^2w^{\star2}. E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ 2 1 r 2 w ⋆ 2 .
Otherwise, P ( w t ≠ 0 ∣ w ⋆ , previous tasks ) > 1 2 \mathbb{P}(w_t\ne0\mid w^\star,\text{ previous tasks})>\frac{1}{2} P ( w t = 0 ∣ w ⋆ , previous tasks ) > 2 1 , so at least one of the events { w t > 0 } \{w_t>0\} { w t > 0 } or { w t < 0 } \{w_t<0\} { w t < 0 } has conditional probability at least 1 4 \frac{1}{4} 4 1 . Let H H H denote such an event. Since on each nonzero branch the proximal map is increasing with derivative at least one,
Var ( w t ∣ H ) ≥ Var ( σ t τ t ( θ t w ⋆ + r t δ Z ) ∣ H ) . \operatorname{Var}(w_t\mid H)\ge\operatorname{Var}\left(\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z\right)\mid H\right). Var ( w t ∣ H ) ≥ Var ( σ t τ t ( θ t w ⋆ + δ r t Z ) ∣ H ) .
Since H H H corresponds to a one-sided Gaussian event of probability at least 1 4 \frac{1}{4} 4 1 ,
P ( H ∣ w ⋆ , previous tasks ) Var ( Z ∣ H ) ≥ 1 20 . \mathbb{P}(H\mid w^\star,\text{ previous tasks})\operatorname{Var}(Z\mid H)\ge\frac{1}{20}. P ( H ∣ w ⋆ , previous tasks ) Var ( Z ∣ H ) ≥ 20 1 .
Therefore,
E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ P ( H ∣ w ⋆ , previous tasks ) Var ( w t ∣ H ) ≥ 1 20 ( σ t τ t r t δ ) 2 ≥ 1 20 δ . \mathbb{E}\left[(w_t-rw^\star)^2\mid w^\star,\text{ previous tasks}\right]\ge\mathbb{P}(H\mid w^\star,\text{ previous tasks})\operatorname{Var}(w_t\mid H)\ge\frac{1}{20}\left(\frac{\sigma_t\tau_t r_t}{\sqrt{\delta}}\right)^2\ge\frac{1}{20\delta}. E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ P ( H ∣ w ⋆ , previous tasks ) Var ( w t ∣ H ) ≥ 20 1 ( δ σ t τ t r t ) 2 ≥ 20 δ 1 .
Thus, for every 0 < p ≤ 1 0<p\le1 0 < p ≤ 1 ,
E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ 1 20 min { r 2 w ⋆ 2 , 1 δ } . \mathbb{E}\left[(w_t-rw^\star)^2\mid w^\star,\text{ previous tasks}\right]\ge\frac{1}{20}\min\left\{r^2w^{\star2},\frac{1}{\delta}\right\}. E [ ( w t − r w ⋆ ) 2 ∣ w ⋆ , previous tasks ] ≥ 20 1 min { r 2 w ⋆ 2 , δ 1 } .
Since h ( ν r ) ≤ h ( 0 ) = 1 4 h(\nu r)\le h(0)=\frac{1}{4} h ( ν r ) ≤ h ( 0 ) = 4 1 , and r = ( 1 − 2 ρ ) h ( ν ) h ( ν r ) r=\frac{(1-2\rho)h(\nu)}{h(\nu r)} r = h ( ν r ) ( 1 − 2 ρ ) h ( ν ) implying r ≥ 4 h ( ν ) ( 1 − 2 ρ ) r\ge4h(\nu)(1-2\rho) r ≥ 4 h ( ν ) ( 1 − 2 ρ ) . Also from Eq. (61) h ( ν ) ≳ 1 1 + ν h(\nu)\gtrsim\frac{1}{1+\nu} h ( ν ) ≳ 1 + ν 1 , and therefore r ≳ 1 − 2 ρ 1 + ν r\gtrsim\frac{1-2\rho}{1+\nu} r ≳ 1 + ν 1 − 2 ρ . Then Proposition 22 proves the claim.
Proof of Theorem 7
Recall Theorem 7. Let λ > 0 \lambda>0 λ > 0 , ϵ = λ 2 δ log ( 64 δ ) \epsilon=\frac{\lambda}{\sqrt{\frac{2}{\delta}\log\left(\frac{64}{\delta}\right)}} ϵ = δ 2 l o g ( δ 64 ) λ . Then there exist universal constants C , c > 0 C,c>0 C , c > 0 such that, if
s ≤ c min { δ ( 1 + ν ) 4 log + ( 1 + ν δ ) , δ ( 1 − 2 ρ ) 2 ν 2 ( 1 + ν ) 2 log + 3 ( ( 1 + ν ) 2 δ ( 1 − 2 ρ ) 2 ν 2 ) , δ ( 1 + ν ) 2 } , s\le c\min\left\{\frac{\delta}{(1+\nu)^4\log_+\left(\frac{1+\nu}{\delta}\right)},\frac{\delta(1-2\rho)^2\nu^2}{(1+\nu)^2\log_+^3\left(\frac{(1+\nu)^2}{\delta(1-2\rho)^2\nu^2}\right)},\frac{\delta}{(1+\nu)^2}\right\}, s ≤ c min ⎩ ⎨ ⎧ ( 1 + ν ) 4 log + ( δ 1 + ν ) δ , ( 1 + ν ) 2 log + 3 ( δ ( 1 − 2 ρ ) 2 ν 2 ( 1 + ν ) 2 ) δ ( 1 − 2 ρ ) 2 ν 2 , ( 1 + ν ) 2 δ ⎭ ⎬ ⎫ ,
then for every task t ≥ 2 t\ge2 t ≥ 2 the following holds.
L ( w T ) − inf w L ( w ) ≤ C s ( 1 + ν ) 8 δ . \mathcal{L}(w_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\le C\frac{s(1+\nu)^8}{\delta}. L ( w T ) − w inf L ( w ) ≤ C δ s ( 1 + ν ) 8 .
Proof. Recall that we aim to bound L ( w t ) − inf w L ( w ) \mathcal{L}(w_t)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w}) L ( w t ) − inf w L ( w ) . By Proposition 22 we know that
L ( w t ) − inf w L ( w ) ≤ 1 2 E [ ( w t − r w ⋆ ) 2 ] , \mathcal{L}(w_t)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\le\frac{1}{2}\mathbb{E}[(w_t-rw^\star)^2], L ( w t ) − w inf L ( w ) ≤ 2 1 E [( w t − r w ⋆ ) 2 ] ,
therefore it is sufficient to minimize E [ ( w t − r w ⋆ ) 2 ] \mathbb{E}[(w_t-rw^\star)^2] E [( w t − r w ⋆ ) 2 ] . Since by Eq. (40) ,
E [ ( w t − r w ⋆ ) 2 ] = ν 2 ( α t − r ) 2 + σ t 2 . \mathbb{E}[(w_t-rw^\star)^2]=\nu^2(\alpha_t-r)^2+\sigma_t^2. E [( w t − r w ⋆ ) 2 ] = ν 2 ( α t − r ) 2 + σ t 2 .
It is sufficient to bound each one of the terms α t − r \alpha_t-r α t − r and σ t 2 \sigma_t^2 σ t 2 . The regularizer is
f ( w ) = { w [ w t − 1 ] + + ϵ s , w > 0 , − w [ − w t − 1 ] + + ϵ s , w < 0. f(w)=
\begin{cases}
\frac{w}{[w_{t-1}]_++\epsilon_s}, & w>0,\\
\frac{-w}{[-w_{t-1}]_++\epsilon_s}, & w<0.
\end{cases} f ( w ) = { [ w t − 1 ] + + ϵ s w , [ − w t − 1 ] + + ϵ s − w , w > 0 , w < 0.
By Lemma 21 it is convex and positively one-homogeneous. In particular, the first scalar equation can be written as w t = σ t τ t V t w_t=\sigma_t\tau_tV_t w t = σ t τ t V t , where
V t = T λ 0 [ w t − 1 ] + + ϵ s , λ 0 [ − w t − 1 ] + + ϵ s ( θ t w ⋆ + r t δ Z t ) V_t=\mathcal{T}_{\frac{\lambda_0}{[w_{t-1}]_++\epsilon_s},\frac{\lambda_0}{[-w_{t-1}]_++\epsilon_s}}\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z_t\right) V t = T [ w t − 1 ] + + ϵ s λ 0 , [ − w t − 1 ] + + ϵ s λ 0 ( θ t w ⋆ + δ r t Z t )
and
T a , b ( x ) = { x − a , x > a , 0 , − b ≤ x ≤ a , x + b , x < − b . \mathcal{T}_{a,b}(x)=
\begin{cases}
x-a, & x>a,\\
0, & -b\le x\le a,\\
x+b, & x<-b.
\end{cases} T a , b ( x ) = ⎩ ⎨ ⎧ x − a , 0 , x + b , x > a , − b ≤ x ≤ a , x < − b .
Both thresholds are at most λ 0 ϵ s = 2 δ log ( 64 s ) \frac{\lambda_0}{\epsilon_s}=\sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)} ϵ s λ 0 = δ 2 log ( s 64 ) . Define m t = E [ w ⋆ V t ] ν 2 m_t=\frac{\mathbb{E}[w^\star V_t]}{\nu^2} m t = ν 2 E [ w ⋆ V t ] . The first and third equations give α t = σ t τ t m t \alpha_t=\sigma_t\tau_t m_t α t = σ t τ t m t , and σ t 2 = ( σ t τ t ) 2 ( E [ V t 2 ] − ν 2 m t 2 ) \sigma_t^2=(\sigma_t\tau_t)^2\left(\mathbb{E}[V_t^2]-\nu^2m_t^2\right) σ t 2 = ( σ t τ t ) 2 ( E [ V t 2 ] − ν 2 m t 2 ) . Moreover, conditionally on ( w ⋆ , w t − 1 ) (w^\star,w_{t-1}) ( w ⋆ , w t − 1 ) ,
∂ V t ∂ Z t = r t δ 1 { V t ≠ 0 } \frac{\partial V_t}{\partial Z_t}=\frac{r_t}{\sqrt{\delta}}\mathbf{1}_{\{V_t\ne0\}} ∂ Z t ∂ V t = δ r t 1 { V t = 0 }
almost surely. Stein’s Lemma therefore gives γ t = σ t τ t δ P ( V t ≠ 0 ) \gamma_t=\frac{\sigma_t\tau_t}{\delta}\mathbb{P}(V_t\ne0) γ t = δ σ t τ t P ( V t = 0 ) . The equation for r t r_t r t is (see Eq. (44) and derivations above it),
r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] . (48) r_t^2=2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))^2\right]. \tag*{(48)} r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] . (48)
Since 0 < ϕ ′ ( u ) < 1 0<\phi'(u)<1 0 < ϕ ′ ( u ) < 1 and 2 E [ 1 − π ρ ( ν Z 1 ) ] = 1 2\mathbb{E}[1-\pi_\rho(\nu Z_1)]=1 2 E [ 1 − π ρ ( ν Z 1 )] = 1 , we have r t ≤ 1 r_t\le1 r t ≤ 1 . Using
Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) + γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) = ν α t Z 1 + σ t Z 2 , (49) \operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)+\gamma_t\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))=\nu\alpha_tZ_1+\sigma_tZ_2, \tag*{(49)} Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) + γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) = ν α t Z 1 + σ t Z 2 , (49)
in the equation for θ t \theta_t θ t , and using the fact that π ρ ′ ( ν Z 1 ) \pi_\rho'(\nu Z_1) π ρ ′ ( ν Z 1 ) is even, gives
θ t = 2 E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) − ν α t Z 1 + σ t Z 2 γ t ) ] = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] (50) \theta_t=2\mathbb{E}\left[\pi_\rho'(\nu Z_1)\left(\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))-\frac{\nu\alpha_tZ_1+\sigma_tZ_2}{\gamma_t}\right)\right]=2\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))\right] \tag*{(50)} θ t = 2 E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) − γ t ν α t Z 1 + σ t Z 2 ) ] = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] (50)
Consequently, 0 < θ t ≤ 2 ( 1 − 2 ρ ) h ( ν ) 0<\theta_t\le2(1-2\rho)h(\nu) 0 < θ t ≤ 2 ( 1 − 2 ρ ) h ( ν ) . The sixth scalar equation can be rewritten as
1 σ t τ t = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . \frac{1}{\sigma_t\tau_t}=\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}\right]. σ t τ t 1 = E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] .
Since 0 ≤ ϕ ′ ′ ( u ) = ϕ ′ ( u ) ( 1 − ϕ ′ ( u ) ) ≤ ϕ ′ ( u ) 0\le\phi''(u)=\phi'(u)(1-\phi'(u))\le\phi'(u) 0 ≤ ϕ ′′ ( u ) = ϕ ′ ( u ) ( 1 − ϕ ′ ( u )) ≤ ϕ ′ ( u ) and
γ t = σ t τ t δ P ( V t ≠ 0 ) ≥ σ t τ t δ P ( V t ≠ 0 , w ⋆ = 0 ) ≥ ( 1 − s ) σ t τ t δ P ( ∣ r t δ Z t ∣ > λ 0 ϵ s | w ⋆ = 0 ) > 0. \gamma_t=\frac{\sigma_t\tau_t}{\delta}\mathbb{P}(V_t\ne0)\ge\frac{\sigma_t\tau_t}{\delta}\mathbb{P}(V_t\ne0,w^\star=0)\ge(1-s)\frac{\sigma_t\tau_t}{\delta}\mathbb{P}\left(\left|\frac{r_t}{\sqrt{\delta}}Z_t\right|>\frac{\lambda_0}{\epsilon_s}\,\middle|\,w^\star=0\right)>0. γ t = δ σ t τ t P ( V t = 0 ) ≥ δ σ t τ t P ( V t = 0 , w ⋆ = 0 ) ≥ ( 1 − s ) δ σ t τ t P ( δ r t Z t > ϵ s λ 0 w ⋆ = 0 ) > 0.
Cauchy–Schwarz yields
1 σ t τ t ≤ 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ≤ r t . \frac{1}{\sigma_t\tau_t}\le2\mathbb{E}[(1-\pi_\rho(\nu Z_1))\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))]\le r_t. σ t τ t 1 ≤ 2 E [( 1 − π ρ ( ν Z 1 )) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ))] ≤ r t .
Thus σ t τ t r t ≥ 1 \sigma_t\tau_tr_t\ge1 σ t τ t r t ≥ 1 and σ t τ t ≥ 1 \sigma_t\tau_t\ge1 σ t τ t ≥ 1 . We next control false coordinates. Put ζ t = P ( w t ≠ 0 ∣ w ⋆ = 0 ) \zeta_t=\mathbb{P}(w_t\ne0\mid w^\star=0) ζ t = P ( w t = 0 ∣ w ⋆ = 0 ) . Suppose w t − 1 > 0 w_{t-1}>0 w t − 1 > 0 . The probability that w t w_t w t has the same sign is Φ ( λ 0 r t / δ δ w t − 1 + ϵ s ) \Phi\left(\frac{\lambda_0}{r_t/\sqrt{\delta}}\frac{\sqrt{\delta}}{w_{t-1}+\epsilon_s}\right) Φ ( r t / δ λ 0 w t − 1 + ϵ s δ ) since the expression inside Φ ( ⋅ ) \Phi(\cdot) Φ ( ⋅ ) is negative the probability is at most 1 2 \frac{1}{2} 2 1 . An output having the opposite sign requires
r t δ Z t < − 2 δ log ( 64 s ) , \frac{r_t}{\sqrt{\delta}}Z_t<-\sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)}, δ r t Z t < − δ 2 log ( s 64 ) ,
whose probability is at most Φ ( − 2 log ( 64 s ) ) \Phi\left(-\sqrt{2\log\left(\frac{64}{s}\right)}\right) Φ ( − 2 log ( s 64 ) ) . The case w t − 1 < 0 w_{t-1}<0 w t − 1 < 0 is identical. If w t − 1 = 0 w_{t-1}=0 w t − 1 = 0 , both directions have threshold 2 δ log ( 64 s ) \sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)} δ 2 log ( s 64 ) , and therefore
P ( w t ≠ 0 ∣ w ⋆ = 0 , w t − 1 = 0 ) ≤ 2 Φ ( − 2 log ( 64 s ) ) . \mathbb{P}(w_t\ne0\mid w^\star=0,w_{t-1}=0)\le2\Phi\left(-\sqrt{2\log\left(\frac{64}{s}\right)}\right). P ( w t = 0 ∣ w ⋆ = 0 , w t − 1 = 0 ) ≤ 2Φ ( − 2 log ( s 64 ) ) .
It follows that
ζ t ≤ 1 2 ζ t − 1 + 2 Φ ( − 2 log ( 64 s ) ) . \zeta_t\le\frac{1}{2}\zeta_{t-1}+2\Phi\left(-\sqrt{2\log\left(\frac{64}{s}\right)}\right). ζ t ≤ 2 1 ζ t − 1 + 2Φ ( − 2 log ( s 64 ) ) .
Since ζ 0 = 0 \zeta_0=0 ζ 0 = 0 , and 2 Φ ( − u ) ≤ e − u 2 / 2 2\Phi(-u)\le e^{-u^2/2} 2Φ ( − u ) ≤ e − u 2 /2 , we obtain, uniformly in t t t , ζ t ≤ s 32 \zeta_t\le\frac{s}{32} ζ t ≤ 32 s . Therefore P ( V t ≠ 0 ) ≤ s + ( 1 − s ) ζ t ≤ 33 s 32 \mathbb{P}(V_t\ne0)\le s+(1-s)\zeta_t\le\frac{33s}{32} P ( V t = 0 ) ≤ s + ( 1 − s ) ζ t ≤ 32 33 s , and hence γ t σ t τ t = P ( V t ≠ 0 ) δ ≤ 33 s 32 δ \frac{\gamma_t}{\sigma_t\tau_t}=\frac{\mathbb{P}(V_t\ne0)}{\delta}\le\frac{33s}{32\delta} σ t τ t γ t = δ P ( V t = 0 ) ≤ 32 δ 33 s . Next, we prove that σ t τ t \sigma_t\tau_t σ t τ t is uniformly bounded. For q > 0 q>0 q > 0 and η ≥ 0 \eta\ge0 η ≥ 0 , define g ( x ) = ϕ ′ ( Prox q η ϕ ( q x ) ) g(x)=\phi'(\operatorname{Prox}_{q\eta\phi}(qx)) g ( x ) = ϕ ′ ( Prox q η ϕ ( q x )) . Then g g g increases from zero to one and
g ′ ( x ) = q ϕ ′ ′ ( Prox q η ϕ ( q x ) ) 1 + q η ϕ ′ ′ ( Prox q η ϕ ( q x ) ) . g'(x)=\frac{q\phi''(\operatorname{Prox}_{q\eta\phi}(qx))}{1+q\eta\phi''(\operatorname{Prox}_{q\eta\phi}(qx))}. g ′ ( x ) = 1 + q η ϕ ′′ ( Prox q η ϕ ( q x )) q ϕ ′′ ( Prox q η ϕ ( q x )) .
To prove that σ t τ t \sigma_t\tau_t σ t τ t is bounded we first prove a lower estimate on E [ 2 ( 1 − π ρ ( ν Z 1 ) ) g ′ ( E [ V t 2 ] Z ) ] \mathbb{E}[2(1-\pi_\rho(\nu Z_1))g'(\sqrt{\mathbb{E}[V_t^2]Z})] E [ 2 ( 1 − π ρ ( ν Z 1 )) g ′ ( E [ V t 2 ] Z )] which appears in equation six this allows us to reach a contradiction. This form follows by orthogonal transformation, we introduce an independent G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) , and
( Z G ) = ( ν m t E [ V t 2 ] E [ V t 2 ] − ν 2 m t 2 E [ V t 2 ] E [ V t 2 ] − ν 2 m t 2 E [ V t 2 ] − ν m t E [ V t 2 ] ) ( Z 1 Z 2 ) . \begin{pmatrix} Z \\ G \end{pmatrix}
=
\begin{pmatrix}
\frac{\nu m_t}{\sqrt{\mathbb{E}[V_t^2]}} &
\frac{\sqrt{\mathbb{E}[V_t^2]-\nu^2m_t^2}}{\sqrt{\mathbb{E}[V_t^2]}} \\
\frac{\sqrt{\mathbb{E}[V_t^2]-\nu^2m_t^2}}{\sqrt{\mathbb{E}[V_t^2]}} &
-\frac{\nu m_t}{\sqrt{\mathbb{E}[V_t^2]}}
\end{pmatrix}
\begin{pmatrix} Z_1 \\ Z_2 \end{pmatrix}. ( Z G ) = E [ V t 2 ] ν m t E [ V t 2 ] E [ V t 2 ] − ν 2 m t 2 E [ V t 2 ] E [ V t 2 ] − ν 2 m t 2 − E [ V t 2 ] ν m t ( Z 1 Z 2 ) .
so Z 1 = ν m t E [ V t 2 ] h + 1 − ν 2 m t 2 E [ V t 2 ] G Z_1=\frac{\nu m_t}{\sqrt{\mathbb{E}[V_t^2]}}h+\sqrt{1-\frac{\nu^2m_t^2}{\mathbb{E}[V_t^2]}}G Z 1 = E [ V t 2 ] ν m t h + 1 − E [ V t 2 ] ν 2 m t 2 G , with G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) , conditionally on E [ V t 2 ] Z = h \sqrt{\mathbb{E}[V_t^2]}Z=h E [ V t 2 ] Z = h , and from α t = σ t τ t m t \alpha_t=\sigma_t\tau_tm_t α t = σ t τ t m t and σ t 2 = ( σ t τ t ) 2 ( E [ V t 2 ] − ν 2 m t 2 ) \sigma_t^2=(\sigma_t\tau_t)^2(\mathbb{E}[V_t^2]-\nu^2m_t^2) σ t 2 = ( σ t τ t ) 2 ( E [ V t 2 ] − ν 2 m t 2 ) which follows by the first and third equations, respectively, so
ν α t Z 1 + σ t Z 2 = ν α t ( ν m t E [ V t 2 ] Z + E [ V t 2 ] − ν 2 m t 2 E [ V t 2 ] G ) + σ t ( E [ V t 2 ] − ν 2 m t 2 E [ V t 2 ] Z − ν m t E [ V t 2 ] G ) = E [ V t 2 ] Z . \nu\alpha_t Z_1+\sigma_t Z_2
=
\nu\alpha_t\left(\frac{\nu m_t}{\sqrt{\mathbb{E}[V_t^2]}}Z+\frac{\sqrt{\mathbb{E}[V_t^2]-\nu^2m_t^2}}{\sqrt{\mathbb{E}[V_t^2]}}G\right)
+\sigma_t\left(\frac{\sqrt{\mathbb{E}[V_t^2]-\nu^2m_t^2}}{\sqrt{\mathbb{E}[V_t^2]}}Z-\frac{\nu m_t}{\sqrt{\mathbb{E}[V_t^2]}}G\right)
=\sqrt{\mathbb{E}[V_t^2]}Z. ν α t Z 1 + σ t Z 2 = ν α t ( E [ V t 2 ] ν m t Z + E [ V t 2 ] E [ V t 2 ] − ν 2 m t 2 G ) + σ t ( E [ V t 2 ] E [ V t 2 ] − ν 2 m t 2 Z − E [ V t 2 ] ν m t G ) = E [ V t 2 ] Z .
Define
ω t ( x ) = E G [ 2 ( 1 − π ρ ( ν ( ν m t E [ V t 2 ] x + 1 − ν 2 m t 2 E [ V t 2 ] G ) ) ) ] . \omega_t(x)=\mathbb{E}_G\left[2\left(1-\pi_\rho\left(\nu\left(\frac{\nu m_t}{\mathbb{E}[V_t^2]}x+\sqrt{1-\frac{\nu^2m_t^2}{\mathbb{E}[V_t^2]}}G\right)\right)\right)\right]. ω t ( x ) = E G [ 2 ( 1 − π ρ ( ν ( E [ V t 2 ] ν m t x + 1 − E [ V t 2 ] ν 2 m t 2 G ) ) ) ] .
from symmetry, ω t ( 0 ) = 1 \omega_t(0)=1 ω t ( 0 ) = 1 , and, uniformly in t t t , ∣ ω t ( x ) − 1 ∣ ≤ C ∣ x ∣ |\omega_t(x)-1|\le C|x| ∣ ω t ( x ) − 1∣ ≤ C ∣ x ∣ , because
ω t ′ ( x ) = − 2 ν ( ν m t ) E [ V t 2 ] E G [ π ρ ′ ( ν ( ν m t E [ V t 2 ] x + 1 − ν 2 m t 2 E [ V t 2 ] G ) ) ] \omega_t'(x)=-\frac{2\nu(\nu m_t)}{\mathbb{E}[V_t^2]}\mathbb{E}_G\left[\pi_\rho'\left(\nu\left(\frac{\nu m_t}{\mathbb{E}[V_t^2]}x+\sqrt{1-\frac{\nu^2m_t^2}{\mathbb{E}[V_t^2]}}G\right)\right)\right] ω t ′ ( x ) = − E [ V t 2 ] 2 ν ( ν m t ) E G [ π ρ ′ ( ν ( E [ V t 2 ] ν m t x + 1 − E [ V t 2 ] ν 2 m t 2 G ) ) ]
and because sup u π ρ ′ ( u ) = ( 1 − 2 ρ ) sup u ϕ ′ ′ ( u ) ≤ 1 − 2 ρ 4 ≤ 1 4 \sup_u\pi_\rho'(u)=(1-2\rho)\sup_u\phi''(u)\le\frac{1-2\rho}{4}\le\frac14 sup u π ρ ′ ( u ) = ( 1 − 2 ρ ) sup u ϕ ′′ ( u ) ≤ 4 1 − 2 ρ ≤ 4 1 and ω t ( 0 ) = 1 \omega_t(0)=1 ω t ( 0 ) = 1 with ν m t < E [ V t 2 ] \nu m_t<\sqrt{\mathbb{E}[V_t^2]} ν m t < E [ V t 2 ] the statement holds with C = ν 2 E [ V t 2 ] C=\frac{\nu}{2\sqrt{\mathbb{E}[V_t^2]}} C = 2 E [ V t 2 ] ν . It will be convenient to use the inverse relation a = g ( x ) = ϕ ′ ( u ) = 1 1 + e − u a=g(x)=\phi'(u)=\frac{1}{1+e^{-u}} a = g ( x ) = ϕ ′ ( u ) = 1 + e − u 1 as the variable. The first order optimality enforces u + q η ϕ ′ ( u ) = q x u+q\eta\phi'(u)=qx u + q η ϕ ′ ( u ) = q x (where q = σ t τ t q=\sigma_t\tau_t q = σ t τ t ). The inverse relation implies u = log ( a 1 − a ) u=\log\left(\frac{a}{1-a}\right) u = log ( 1 − a a ) , and
x ( a ) = 1 q log ( a 1 − a ) + η a . x(a)=\frac{1}{q}\log\left(\frac{a}{1-a}\right)+\eta a. x ( a ) = q 1 log ( 1 − a a ) + η a .
Let a 0 = g ( 0 ) a_0=g(0) a 0 = g ( 0 ) . Then
log ( a 0 1 − a 0 ) + q η a 0 = 0. \log\left(\frac{a_0}{1-a_0}\right)+q\eta a_0=0. log ( 1 − a 0 a 0 ) + q η a 0 = 0.
At a = 1 2 a=\frac12 a = 2 1 we have log ( 1 ) + 1 2 q η ≥ 0 \log(1)+\frac12q\eta\ge0 log ( 1 ) + 2 1 q η ≥ 0 and at a = 1 q η + 2 a=\frac{1}{q\eta+2} a = q η + 2 1 we have log ( 1 q η + 2 1 − 1 q η + 2 ) + q η q η + 2 ≤ − q η q η + 1 + q η q η + 2 < 0 \log\left(\frac{\frac{1}{q\eta+2}}{1-\frac{1}{q\eta+2}}\right)+\frac{q\eta}{q\eta+2}\le-\frac{q\eta}{q\eta+1}+\frac{q\eta}{q\eta+2}<0 log ( 1 − q η + 2 1 q η + 2 1 ) + q η + 2 q η ≤ − q η + 1 q η + q η + 2 q η < 0 the monotonicity of the function therefore gives 1 q η + 2 ≤ a 0 ≤ 1 2 \frac{1}{q\eta+2}\le a_0\le\frac12 q η + 2 1 ≤ a 0 ≤ 2 1 . For a ∈ [ a 0 , a 0 + 1 4 ] a\in[a_0,a_0+\frac14] a ∈ [ a 0 , a 0 + 4 1 ] ,
d x d a = 1 q a ( 1 − a ) + η ≤ 4 q ( q η + 2 ) + η ≤ 8 ( η + q − 1 ) . \frac{dx}{da}=\frac{1}{qa(1-a)}+\eta\le\frac{4}{q}(q\eta+2)+\eta\le8(\eta+q^{-1}). d a d x = q a ( 1 − a ) 1 + η ≤ q 4 ( q η + 2 ) + η ≤ 8 ( η + q − 1 ) .
If R < 2 ( 1 + ν ) ( η + q − 1 ) R<2(1+\nu)(\eta+q^{-1}) R < 2 ( 1 + ν ) ( η + q − 1 ) , the interval a 0 ≤ a ≤ a 0 + R 8 ( 1 + ν ) ( η + q − 1 ) a_0\le a\le a_0+\frac{R}{8(1+\nu)(\eta+q^{-1})} a 0 ≤ a ≤ a 0 + 8 ( 1 + ν ) ( η + q − 1 ) R is mapped into 0 ≤ x ≤ R 1 + ν 0\le x\le\frac{R}{1+\nu} 0 ≤ x ≤ 1 + ν R as x ( a ) − x ( a 0 ) = ∫ a 0 a d x d b d b x(a)-x(a_0)=\int_{a_0}^a\frac{dx}{db}\,db x ( a ) − x ( a 0 ) = ∫ a 0 a d b d x d b , so
x ( a ) ≤ 8 ( η + q − 1 ) ( a − a 0 ) ≤ 8 ( η + q − 1 ) R 8 ( 1 + ν ) ( η + q − 1 ) = R 1 + ν . x(a)\le8(\eta+q^{-1})(a-a_0)\le8(\eta+q^{-1})\frac{R}{8(1+\nu)(\eta+q^{-1})}=\frac{R}{1+\nu}. x ( a ) ≤ 8 ( η + q − 1 ) ( a − a 0 ) ≤ 8 ( η + q − 1 ) 8 ( 1 + ν ) ( η + q − 1 ) R = 1 + ν R .
Then, if R < 2 ( 1 + ν ) ( η + q − 1 ) R<2(1+\nu)(\eta+q^{-1}) R < 2 ( 1 + ν ) ( η + q − 1 ) , since d a = g ′ ( x ) d x da=g'(x)\,dx d a = g ′ ( x ) d x we have
E [ 2 ( 1 − π ρ ( ν Z 1 ) ) g ′ ( H t ) ] ≥ ∫ a 0 a 0 + R 8 ( 1 + ν ) ( η + q − 1 ) 1 2 2 π R e − x ( a ) 2 2 R 2 d a ≥ e − 1 2 2 π R R 16 ( 1 + ν ) ( η + q − 1 ) = e − 1 2 16 2 π ( 1 + ν ) ( η + q − 1 ) . \begin{aligned}
\mathbb{E}[2(1-\pi_\rho(\nu Z_1))g'(H_t)]
\ge\int_{a_0}^{a_0+\frac{R}{8(1+\nu)(\eta+q^{-1})}}\frac{1}{2\sqrt{2\pi}R}e^{-\frac{x(a)^2}{2R^2}}\,da \\
&\ge\frac{e^{-\frac12}}{\sqrt{2\pi}R}\frac{R}{16(1+\nu)(\eta+q^{-1})}
=\frac{e^{-\frac12}}{16\sqrt{2\pi}(1+\nu)(\eta+q^{-1})}.
\end{aligned} E [ 2 ( 1 − π ρ ( ν Z 1 )) g ′ ( H t )] ≥ ∫ a 0 a 0 + 8 ( 1 + ν ) ( η + q − 1 ) R 2 2 π R 1 e − 2 R 2 x ( a ) 2 d a ≥ 2 π R e − 2 1 16 ( 1 + ν ) ( η + q − 1 ) R = 16 2 π ( 1 + ν ) ( η + q − 1 ) e − 2 1 .
Similarly, if R ≥ 2 ( 1 + ν ) ( η + q − 1 ) R \ge2(1+\nu)(\eta+q^{-1}) R ≥ 2 ( 1 + ν ) ( η + q − 1 ) , the values a ∈ [ a 0 , a 0 + 1 4 ] a \in[a_0,a_0+\frac{1}{4}] a ∈ [ a 0 , a 0 + 4 1 ] are mapped into an interval 0 ≤ x ( a ) ≤ 8 ( η + q − 1 ) 1 4 = 2 ( η + q − 1 ) ≤ R 1 + ν 0 \le x(a) \le8(\eta+q^{-1})\frac{1}{4}=2(\eta+q^{-1}) \le\frac{R}{1+\nu} 0 ≤ x ( a ) ≤ 8 ( η + q − 1 ) 4 1 = 2 ( η + q − 1 ) ≤ 1 + ν R . Hence E [ 2 ( 1 − π ρ ( ν Z 1 ) ) g ′ ( H t ) ] ≥ ∫ a 0 a 0 + 1 4 1 2 2 π R e − x ( a ) 2 2 R 2 d a ≥ e − 1 2 8 2 π R \mathbb{E}[2(1-\pi_\rho(\nu Z_1))g'(H_t)] \ge\int_{a_0}^{a_0+\frac{1}{4}} \frac{1}{2\sqrt{2\pi}R}e^{-\frac{x(a)^2}{2R^2}}\,da \ge\frac{e^{-\frac{1}{2}}}{8\sqrt{2\pi}R} E [ 2 ( 1 − π ρ ( ν Z 1 )) g ′ ( H t )] ≥ ∫ a 0 a 0 + 4 1 2 2 π R 1 e − 2 R 2 x ( a ) 2 d a ≥ 8 2 π R e − 2 1 . The two cases prove E [ 2 ( 1 − π ρ ( ν Z 1 ) ) g ′ ( H t ) ] ≥ e − 1 2 16 2 π ( 1 + ν ) ( R + η + q − 1 ) \mathbb{E}[2(1-\pi_\rho(\nu Z_1))g'(H_t)] \ge\frac{e^{-\frac{1}{2}}}{16\sqrt{2\pi}(1+\nu)(R+\eta+q^{-1})} E [ 2 ( 1 − π ρ ( ν Z 1 )) g ′ ( H t )] ≥ 16 2 π ( 1 + ν ) ( R + η + q − 1 ) e − 2 1 .
Suppose now, for contradiction, that no uniform upper bound on σ t τ t \sigma_t\tau_t σ t τ t exists for a sufficiently small s s s . Then there are s n ↓ 0 s_n\downarrow0 s n ↓ 0 , tasks t n t_n t n , and scalar solutions such that σ t n τ t n → ∞ \sigma_{t_n}\tau_{t_n}\to\infty σ t n τ t n → ∞ . Let H n = ν m n Z 1 + E [ V n 2 ] − ν 2 m n 2 Z 2 H_n=\nu m_nZ_1+\sqrt{\mathbb{E}[V_n^2]-\nu^2m_n^2}Z_2 H n = ν m n Z 1 + E [ V n 2 ] − ν 2 m n 2 Z 2 , this definition follows from α t = σ t τ t m t \alpha_t=\sigma_t\tau_tm_t α t = σ t τ t m t and σ t 2 = ( σ t τ t ) 2 ( E [ V t 2 ] − ν 2 m t 2 ) \sigma_t^2=(\sigma_t\tau_t)^2(\mathbb{E}[V_t^2]-\nu^2m_t^2) σ t 2 = ( σ t τ t ) 2 ( E [ V t 2 ] − ν 2 m t 2 ) by the first and third equations, respectively. Therefore,
ν α n Z 1 + σ n Z 2 = σ n τ n ( ν m n Z 1 + E [ V n 2 ] − ν 2 m n 2 Z 2 ) = σ n τ n H n \nu\alpha_nZ_1+\sigma_nZ_2=\sigma_n\tau_n\left(\nu m_nZ_1+\sqrt{\mathbb{E}[V_n^2]-\nu^2m_n^2}Z_2\right)=\sigma_n\tau_nH_n ν α n Z 1 + σ n Z 2 = σ n τ n ( ν m n Z 1 + E [ V n 2 ] − ν 2 m n 2 Z 2 ) = σ n τ n H n
The previous bound give P ( V n ≠ 0 ) ≤ 33 s n 32 ⟶ n → ∞ 0 \mathbb{P}(V_n\ne0)\le\frac{33s_n}{32}\longrightarrow_{n\to\infty}0 P ( V n = 0 ) ≤ 32 33 s n ⟶ n → ∞ 0 . Moreover, E [ H n 2 ] = ν 2 m n 2 + E [ V n 2 ] − ν 2 m n 2 = E [ V n 2 ] \mathbb{E}[H_n^2]=\nu^2m_n^2+\mathbb{E}[V_n^2]-\nu^2m_n^2=\mathbb{E}[V_n^2] E [ H n 2 ] = ν 2 m n 2 + E [ V n 2 ] − ν 2 m n 2 = E [ V n 2 ] , which is uniformly bounded because
E [ V n 2 ] ≤ E [ ( θ n w ⋆ + r n δ Z ) 2 ] r n ≤ 1 , θ n ≤ 2 ( 1 − 2 ρ ) h ( ν ) , w ⋆ ⊥ Z ≤ 2 ν 2 ( 1 − 2 ρ ) h ( ν ) + 1 δ . \mathbb{E}[V_n^2]\le\mathbb{E}\left[\left(\theta_nw^\star+\frac{r_n}{\sqrt{\delta}}Z\right)^2\right]_{r_n\le1,\theta_n\le2(1-2\rho)h(\nu),w^\star\perp Z}\le2\nu^2(1-2\rho)h(\nu)+\frac{1}{\delta}. E [ V n 2 ] ≤ E [ ( θ n w ⋆ + δ r n Z ) 2 ] r n ≤ 1 , θ n ≤ 2 ( 1 − 2 ρ ) h ( ν ) , w ⋆ ⊥ Z ≤ 2 ν 2 ( 1 − 2 ρ ) h ( ν ) + δ 1 .
Define g n ( h ) = ϕ ( Prox σ n τ n P ( V n ≠ 0 ) δ ϕ ( σ n τ n h ) ) g_n(h)=\phi\left(\operatorname{Prox}_{\sigma_n\tau_n\frac{\mathbb{P}(V_n\ne0)}{\delta}\phi}(\sigma_n\tau_nh)\right) g n ( h ) = ϕ ( Prox σ n τ n δ P ( V n = 0 ) ϕ ( σ n τ n h ) ) . Multiplying the sixth scalar equation by σ n τ n \sigma_n\tau_n σ n τ n gives
1 = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) g n ′ ( H n ) ] . 1=\mathbb{E}[2(1-\pi_\rho(\nu Z_1))g_n'(H_n)]. 1 = E [ 2 ( 1 − π ρ ( ν Z 1 )) g n ′ ( H n )] .
From the lower bound proved for E [ 2 ( 1 − π ρ ( ν Z 1 ) ) g ′ ( E [ V n 2 ] Z ) ] \mathbb{E}[2(1-\pi_\rho(\nu Z_1))g'(\sqrt{\mathbb{E}[V_n^2]}Z)] E [ 2 ( 1 − π ρ ( ν Z 1 )) g ′ ( E [ V n 2 ] Z )] we get,
1 ≥ e − 1 2 16 2 π ( 1 + ν ) E [ H n 2 ] + P ( V n ≠ 0 ) δ + ( σ n τ n ) − 1 (51) 1\ge\frac{\dfrac{e^{-\frac{1}{2}}}{16\sqrt{2\pi}(1+\nu)}}{\sqrt{\mathbb{E}[H_n^2]}+\dfrac{\mathbb{P}(V_n\ne0)}{\delta}+(\sigma_n\tau_n)^{-1}} \tag*{(51)} 1 ≥ E [ H n 2 ] + δ P ( V n = 0 ) + ( σ n τ n ) − 1 16 2 π ( 1 + ν ) e − 2 1 (51)
⟹ E [ H n 2 ] + P ( V n ≠ 0 ) δ + ( σ n τ n ) − 1 ≥ e − 1 2 16 2 π ( 1 + ν ) > 0. \Longrightarrow\quad\sqrt{\mathbb{E}[H_n^2]}+\frac{\mathbb{P}(V_n\ne0)}{\delta}+(\sigma_n\tau_n)^{-1}\ge\frac{e^{-\frac{1}{2}}}{16\sqrt{2\pi}(1+\nu)}>0. ⟹ E [ H n 2 ] + δ P ( V n = 0 ) + ( σ n τ n ) − 1 ≥ 16 2 π ( 1 + ν ) e − 2 1 > 0.
Since E [ H n 2 ] \mathbb{E}[H_n^2] E [ H n 2 ] is uniformly bounded (from above and below) and ∣ m n ∣ ≤ E [ H n 2 ] ν |m_n|\le\frac{\sqrt{\mathbb{E}[H_n^2]}}{\nu} ∣ m n ∣ ≤ ν E [ H n 2 ] by Cauchy-Schwartz then by Bolzano–Weierstrass, there exists a subsequence, still denoted by n n n ,
m n → m ∞ , E [ V n 2 ] − ν 2 m n 2 → ξ ∞ , ν 2 m ∞ 2 + ξ ∞ 2 > 0. m_n\to m_\infty,\qquad\sqrt{\mathbb{E}[V_n^2]-\nu^2m_n^2}\to\xi_\infty,\qquad\nu^2m_\infty^2+\xi_\infty^2>0. m n → m ∞ , E [ V n 2 ] − ν 2 m n 2 → ξ ∞ , ν 2 m ∞ 2 + ξ ∞ 2 > 0.
For every bounded continuous ψ \psi ψ ,
∫ R ψ ( h ) g n ′ ( h ) d h = ∫ 0 1 ψ ( ( σ n τ n ) − 1 log ( a 1 − a ) + P ( V n ≠ 0 ) δ a ) d a ⟶ ψ ( 0 ) . \int_{\mathbb{R}}\psi(h)g_n'(h)\,dh=\int_0^1\psi\left((\sigma_n\tau_n)^{-1}\log\left(\frac{a}{1-a}\right)+\frac{\mathbb{P}(V_n\ne0)}{\delta}a\right)\,da\longrightarrow\psi(0). ∫ R ψ ( h ) g n ′ ( h ) d h = ∫ 0 1 ψ ( ( σ n τ n ) − 1 log ( 1 − a a ) + δ P ( V n = 0 ) a ) d a ⟶ ψ ( 0 ) .
Conditionally on H n = h H_n=h H n = h , Z 1 = d ν m n E [ V n 2 ] h + 1 − ν 2 m n 2 E [ V n 2 ] G Z_1\mathrel{\mathop=^{d}}\frac{\nu m_n}{\mathbb{E}[V_n^2]}h+\sqrt{1-\frac{\nu^2m_n^2}{\mathbb{E}[V_n^2]}}G Z 1 = d E [ V n 2 ] ν m n h + 1 − E [ V n 2 ] ν 2 m n 2 G , where G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) . In particular, conditional on H n = 0 H_n=0 H n = 0 , Z 1 Z_1 Z 1 is symmetric, and therefore
E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ∣ H n = 0 ] = 1. \mathbb{E}[2(1-\pi_\rho(\nu Z_1))\mid H_n=0]=1. E [ 2 ( 1 − π ρ ( ν Z 1 )) ∣ H n = 0 ] = 1.
The conditional Gaussian formula and the lower bound on E [ V n 2 ] \sqrt{\mathbb{E}[V_n^2]} E [ V n 2 ] imply local uniform convergence of the conditional expectations. Hence the weak convergence of g n ′ ( h ) d h g_n'(h)\,dh g n ′ ( h ) d h gives
1 = lim n → ∞ E [ 2 ( 1 − π ρ ( ν Z 1 ) ) g n ′ ( H n ) ] = lim n → ∞ ∫ E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ∣ H n = h ] 1 2 π E [ V n 2 ] e − h 2 2 E [ V n 2 ] g n ′ ( h ) d h = 1 2 π E [ V ∞ 2 ] . \begin{aligned}
1=\lim_{n\to\infty}\mathbb{E}[2(1-\pi_\rho(\nu Z_1))g_n'(H_n)] \\
&=\lim_{n\to\infty}\int\mathbb{E}[2(1-\pi_\rho(\nu Z_1))\mid H_n=h]\frac{1}{\sqrt{2\pi\mathbb{E}[V_n^2]}}e^{-\frac{h^2}{2\mathbb{E}[V_n^2]}}g_n'(h)\,dh=\frac{1}{\sqrt{2\pi\mathbb{E}[V_\infty^2]}}.
\end{aligned} 1 = n → ∞ lim E [ 2 ( 1 − π ρ ( ν Z 1 )) g n ′ ( H n )] = n → ∞ lim ∫ E [ 2 ( 1 − π ρ ( ν Z 1 )) ∣ H n = h ] 2 π E [ V n 2 ] 1 e − 2 E [ V n 2 ] h 2 g n ′ ( h ) d h = 2 π E [ V ∞ 2 ] 1 .
Thus E [ V ∞ 2 ] = 1 2 π \mathbb{E}[V_\infty^2]=\frac{1}{\sqrt{2\pi}} E [ V ∞ 2 ] = 2 π 1 . For every h ≠ 0 h\ne0 h = 0 , g n ( h ) ⟶ 1 { h > 0 } g_n(h)\longrightarrow1_{\{h>0\}} g n ( h ) ⟶ 1 { h > 0 } . Dominated convergence in the equation for θ n \theta_n θ n gives
θ n ⟶ 2 E [ π ρ ′ ( ν Z 1 ) 1 { ν m ∞ Z 1 + ξ ∞ Z 2 > 0 } ] . \theta_n\longrightarrow2\mathbb{E}\left[\pi'_\rho(\nu Z_1)1_{\{\nu m_\infty Z_1+\xi_\infty Z_2>0\}}\right]. θ n ⟶ 2 E [ π ρ ′ ( ν Z 1 ) 1 { ν m ∞ Z 1 + ξ ∞ Z 2 > 0 } ] .
The joint law is invariant under ( Z 1 , Z 2 ) ↦ ( − Z 1 , − Z 2 ) (Z_1,Z_2) \mapsto(-Z_1,-Z_2) ( Z 1 , Z 2 ) ↦ ( − Z 1 , − Z 2 ) , while π ρ ′ ( ν Z 1 ) \pi'_{\rho}(\nu Z_1) π ρ ′ ( ν Z 1 ) is even. Therefore θ n → ( 1 − 2 ρ ) h ( ν ) \theta_n \to(1-2\rho)h(\nu) θ n → ( 1 − 2 ρ ) h ( ν ) . Since s n ↓ 0 s_n \downarrow0 s n ↓ 0 , the bounds give as ∣ ν m n − θ n ∣ ≤ E [ ( V n − θ n w ⋆ ) 2 ] → 0 \lvert\nu m_n-\theta_n\rvert\le\sqrt{\mathbb{E}[(V_n-\theta_n w^\star)^2]} \to0 ∣ ν m n − θ n ∣ ≤ E [( V n − θ n w ⋆ ) 2 ] → 0 the inequality follows because m n w ⋆ m_n w^\star m n w ⋆ is the L 2 L^2 L 2 -projection of V t V_t V t onto w ⋆ w^\star w ⋆ and the convergence is proven later independently, see conclusion in Eq. (57) , similarly ξ n → 0 \xi_n \to0 ξ n → 0 . It follows that E [ V ∞ 2 ] = ν ( 1 − 2 ρ ) h ( ν ) \sqrt{\mathbb{E}[V_\infty^2]}=\nu(1-2\rho)h(\nu) E [ V ∞ 2 ] = ν ( 1 − 2 ρ ) h ( ν ) . This contradicts ν ( 1 − 2 ρ ) h ( ν ) < 1 2 π \nu(1-2\rho)h(\nu)<\frac{1}{\sqrt{2\pi}} ν ( 1 − 2 ρ ) h ( ν ) < 2 π 1 . Indeed, d d a ( a h ( a ) ) = E [ Z 2 ϕ ′ ( a Z ) ] > 0 \frac{d}{d a}(ah(a))=\mathbb{E}[Z^2\phi'(aZ)]>0 d a d ( ah ( a )) = E [ Z 2 ϕ ′ ( a Z )] > 0 and lim a → ∞ a h ( a ) = 1 2 π \lim_{a\to\infty}ah(a)=\frac{1}{\sqrt{2\pi}} lim a → ∞ ah ( a ) = 2 π 1 . Therefore there exist s 0 > 0 s_0>0 s 0 > 0 and Q ν , ρ , δ < ∞ Q_{\nu,\rho,\delta}<\infty Q ν , ρ , δ < ∞ such that σ t τ t ≤ Q ν , ρ , δ \sigma_t\tau_t\le Q_{\nu,\rho,\delta} σ t τ t ≤ Q ν , ρ , δ for every s ≤ s 0 s\le s_0 s ≤ s 0 and every task t t t . It is enough to consider tasks for which σ t τ t > 64 2 π e 1 2 ( 1 + ν ) \sigma_t\tau_t>64\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu) σ t τ t > 64 2 π e 2 1 ( 1 + ν ) , otherwise it immediately follows that Q ν , ρ , δ = 64 2 π e 1 2 ( 1 + ν ) Q_{\nu,\rho,\delta}=64\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu) Q ν , ρ , δ = 64 2 π e 2 1 ( 1 + ν ) . The main contradiction of the previous argument lies in the fact that ν ( 1 − 2 ρ ) h ( ν ) < 1 2 π \nu(1-2\rho)h(\nu)<\frac{1}{\sqrt{2\pi}} ν ( 1 − 2 ρ ) h ( ν ) < 2 π 1 , we investigate when it holds
1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ≤ ∣ 1 2 π − E [ V t 2 ] ∣ + ∣ E [ V t 2 ] − ν θ t ∣ + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = ∣ ∥ ( ν m t , E [ V t 2 ] − ν 2 m t 2 ) ∥ 2 − ∥ ( ν θ t , 0 ) ∥ 2 ∣ ≤ 2 2 π ∣ ∫ R ( e − h 2 2 E [ V t 2 ] E [ 1 − π ρ ( ν Z 1 ) ∣ H t = h ] − E [ 1 − π ρ ( ν Z 1 ) ∣ H t = 0 ] ) g t ′ ( h ) d h ∣ \begin{aligned}
\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu) \\
&\le\left|\frac{1}{\sqrt{2\pi}}-\sqrt{\mathbb{E}[V_t^2]}\right|+\left|\sqrt{\mathbb{E}[V_t^2]}-\nu\theta_t\right|+\nu|\theta_t-(1-2\rho)h(\nu)| \\
&=\left|\left\|(\nu m_t,\sqrt{\mathbb{E}[V_t^2]-\nu^2m_t^2})\right\|_2-\|(\nu\theta_t,0)\|_2\right| \\
&\le\frac{2}{\sqrt{2\pi}}\left|\int_{\mathbb{R}}\left(e^{-\frac{h^2}{2\mathbb{E}[V_t^2]}}\mathbb{E}\left[1-\pi_\rho(\nu Z_1)\mid H_t=h\right]-\mathbb{E}\left[1-\pi_\rho(\nu Z_1)\mid H_t=0\right]\right)g'_t(h)\,dh\right|
\end{aligned} 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ≤ 2 π 1 − E [ V t 2 ] + E [ V t 2 ] − ν θ t + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = ( ν m t , E [ V t 2 ] − ν 2 m t 2 ) 2 − ∥ ( ν θ t , 0 ) ∥ 2 ≤ 2 π 2 ∫ R ( e − 2 E [ V t 2 ] h 2 E [ 1 − π ρ ( ν Z 1 ) ∣ H t = h ] − E [ 1 − π ρ ( ν Z 1 ) ∣ H t = 0 ] ) g t ′ ( h ) d h
≤ + E [ ( V t − m t ν ⋆ ) 2 ] + ν ∣ m t − θ t ∣ + 2 ν sup u π ρ ′ ( u ) E [ ∣ g t ( H t ) − 1 { H t > 0 } ∣ ] ≤ [ 2 2 π e − 1 2 16 2 π ( 1 + ν ) ( e − 1 2 + ν ( 1 − 2 ρ ) 4 ) + ν ( 1 − 2 ρ ) 2 2 π e − 1 2 16 2 π ( 1 + ν ) ] ( 2 log 2 σ t τ t + P ( V t ≠ 0 ) 2 δ ) (52) \begin{aligned}
\phantom{\le{}}+\sqrt{\mathbb{E}[(V_t-m_t\nu^\star)^2]}+\nu|m_t-\theta_t|+2\nu\sup_u\pi'_\rho(u)\mathbb{E}\left[|g_t(H_t)-1_{\{H_t>0\}}|\right] \\
&\le\left[\frac{2}{\sqrt{2\pi}\frac{e^{-\frac{1}{2}}}{16\sqrt{2\pi}(1+\nu)}}\left(e^{-\frac{1}{2}}+\frac{\nu(1-2\rho)}{4}\right)+\frac{\nu(1-2\rho)}{2\sqrt{2\pi}\frac{e^{-\frac{1}{2}}}{16\sqrt{2\pi}(1+\nu)}}\right]\left(\frac{2\log2}{\sigma_t\tau_t}+\frac{\mathbb{P}(V_t\ne0)}{2\delta}\right) \tag*{(52)}
\end{aligned} ≤ + E [( V t − m t ν ⋆ ) 2 ] + ν ∣ m t − θ t ∣ + 2 ν u sup π ρ ′ ( u ) E [ ∣ g t ( H t ) − 1 { H t > 0 } ∣ ] ≤ 2 π 16 2 π ( 1 + ν ) e − 2 1 2 ( e − 2 1 + 4 ν ( 1 − 2 ρ ) ) + 2 2 π 16 2 π ( 1 + ν ) e − 2 1 ν ( 1 − 2 ρ ) ( σ t τ t 2 log 2 + 2 δ P ( V t = 0 ) ) (52)
≤ + 2 2 s δ ( 1 + 2 log 64 s ) . \phantom{\le{}}+2\sqrt{\frac{2s}{\delta}}\left(1+2\log\frac{64}{s}\right). ≤ + 2 δ 2 s ( 1 + 2 log s 64 ) .
Here [1] follows from the sixth equation. Indeed,
1 = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) g t ′ ( H t ) ] = 2 2 π E [ V t 2 ] ∫ R e − h 2 2 E [ V t 2 ] E [ 1 − π ρ ( ν Z 1 ) ∣ H t = h ] g t ′ ( h ) d h . 1=\mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))g'_t(H_t)\right]=\frac{2}{\sqrt{2\pi\mathbb{E}[V_t^2]}}\int_{\mathbb{R}}e^{-\frac{h^2}{2\mathbb{E}[V_t^2]}}\mathbb{E}\left[1-\pi_\rho(\nu Z_1)\mid H_t=h\right]g'_t(h)\,dh. 1 = E [ 2 ( 1 − π ρ ( ν Z 1 )) g t ′ ( H t ) ] = 2 π E [ V t 2 ] 2 ∫ R e − 2 E [ V t 2 ] h 2 E [ 1 − π ρ ( ν Z 1 ) ∣ H t = h ] g t ′ ( h ) d h .
Since E [ 1 − π ρ ( ν Z 1 ) ∣ H t = 0 ] = 1 2 \mathbb{E}[1-\pi_\rho(\nu Z_1)\mid H_t=0]=\frac{1}{2} E [ 1 − π ρ ( ν Z 1 ) ∣ H t = 0 ] = 2 1 from symmetry, and ∫ R g t ′ ( h ) d h = 1 \int_{\mathbb{R}}g'_t(h)\,dh=1 ∫ R g t ′ ( h ) d h = 1 , multiplying the sixth equation by E [ V t 2 ] \sqrt{\mathbb{E}[V_t^2]} E [ V t 2 ] gives
E [ V t 2 ] − 1 2 π = 2 2 π ∫ R ( e − h 2 2 E [ V t 2 ] E [ 1 − π ρ ( ν Z 1 ) ∣ H t = h ] − 1 2 ) g t ′ ( h ) d h . \sqrt{\mathbb{E}[V_t^2]}-\frac{1}{\sqrt{2\pi}}=\frac{2}{\sqrt{2\pi}}\int_{\mathbb{R}}\left(e^{-\frac{h^2}{2\mathbb{E}[V_t^2]}}\mathbb{E}\left[1-\pi_\rho(\nu Z_1)\mid H_t=h\right]-\frac{1}{2}\right)g'_t(h)\,dh. E [ V t 2 ] − 2 π 1 = 2 π 2 ∫ R ( e − 2 E [ V t 2 ] h 2 E [ 1 − π ρ ( ν Z 1 ) ∣ H t = h ] − 2 1 ) g t ′ ( h ) d h .
Moreover, by the opposite triangle inequality,
∣ E [ V t 2 ] − ν θ t ∣ ≤ E [ ( V t − m t ν ⋆ ) 2 ] + ν ∣ m t − θ t ∣ . \left|\sqrt{\mathbb{E}[V_t^2]}-\nu\theta_t\right|\le\sqrt{\mathbb{E}[(V_t-m_t\nu^\star)^2]}+\nu|m_t-\theta_t|. E [ V t 2 ] − ν θ t ≤ E [( V t − m t ν ⋆ ) 2 ] + ν ∣ m t − θ t ∣.
Also, E [ π ρ ′ ( ν Z 1 ) 1 { H t > 0 } ] = E [ π ρ ′ ( ν Z 1 ) 1 { H t < 0 } ] \mathbb{E}[\pi'_\rho(\nu Z_1)1_{\{H_t>0\}}]=\mathbb{E}[\pi'_\rho(\nu Z_1)1_{\{H_t<0\}}] E [ π ρ ′ ( ν Z 1 ) 1 { H t > 0 } ] = E [ π ρ ′ ( ν Z 1 ) 1 { H t < 0 } ] , because π ρ ′ ( ν Z 1 ) \pi'_\rho(\nu Z_1) π ρ ′ ( ν Z 1 ) is even and ( Z 1 , Z 2 ) ∼ d ( − Z 1 , − Z 2 ) (Z_1,Z_2)\mathrel{\mathop{\sim}^{d}}(-Z_1,-Z_2) ( Z 1 , Z 2 ) ∼ d ( − Z 1 , − Z 2 ) . Since E [ V t 2 ] > 0 \sqrt{\mathbb{E}[V_t^2]}>0 E [ V t 2 ] > 0 , P ( H t = 0 ) = 0 \mathbb{P}(H_t=0)=0 P ( H t = 0 ) = 0 , and therefore
2 E [ π ρ ′ ( ν Z 1 ) 1 { H t > 0 } ] = E [ π ρ ′ ( ν Z 1 ) ] = ( 1 − 2 ρ ) h ( ν ) . 2\mathbb{E}[\pi'_\rho(\nu Z_1)1_{\{H_t>0\}}]=\mathbb{E}[\pi'_\rho(\nu Z_1)]=(1-2\rho)h(\nu). 2 E [ π ρ ′ ( ν Z 1 ) 1 { H t > 0 } ] = E [ π ρ ′ ( ν Z 1 )] = ( 1 − 2 ρ ) h ( ν ) .
For [2], define F t ( h ) = e − h 2 2 E [ V t 2 ] E [ 1 − π ρ ( ν Z 1 ) ∣ H t = h ] F_t(h)=e^{-\frac{h^2}{2\mathbb{E}[V_t^2]}}\mathbb{E}[1-\pi_\rho(\nu Z_1)\mid H_t=h] F t ( h ) = e − 2 E [ V t 2 ] h 2 E [ 1 − π ρ ( ν Z 1 ) ∣ H t = h ] . Therefore for a sufficiently small s ≤ δ e − 1 2 66 2 π ( 1 + ν ) s\le\frac{\delta e^{-\frac{1}{2}}}{66\sqrt{2\pi}(1+\nu)} s ≤ 66 2 π ( 1 + ν ) δ e − 2 1 ,
∣ F t ′ ( h ) ∣ ≤ ∣ h ∣ E [ V t 2 ] e − h 2 2 E [ V t 2 ] + ν ∣ ν m t ∣ E [ V t 2 ] sup u π ρ ′ ( u ) ≤ e − 1 2 E [ V t 2 ] + ν ( 1 − 2 ρ ) 4 E [ V t 2 ] ≤ 1 e − 1 2 32 2 π ( 1 + ν ) ( e − 1 2 + ν ( 1 − 2 ρ ) 4 ) , |F'_t(h)|\le\frac{|h|}{\mathbb{E}[V_t^2]}e^{-\frac{h^2}{2\mathbb{E}[V_t^2]}}+\nu\frac{|\nu m_t|}{\mathbb{E}[V_t^2]}\sup_u\pi'_\rho(u)\le\frac{e^{-\frac{1}{2}}}{\sqrt{\mathbb{E}[V_t^2]}}+\frac{\nu(1-2\rho)}{4\sqrt{\mathbb{E}[V_t^2]}}\le\frac{1}{\frac{e^{-\frac{1}{2}}}{32\sqrt{2\pi}(1+\nu)}}\left(e^{-\frac{1}{2}}+\frac{\nu(1-2\rho)}{4}\right), ∣ F t ′ ( h ) ∣ ≤ E [ V t 2 ] ∣ h ∣ e − 2 E [ V t 2 ] h 2 + ν E [ V t 2 ] ∣ ν m t ∣ u sup π ρ ′ ( u ) ≤ E [ V t 2 ] e − 2 1 + 4 E [ V t 2 ] ν ( 1 − 2 ρ ) ≤ 32 2 π ( 1 + ν ) e − 2 1 1 ( e − 2 1 + 4 ν ( 1 − 2 ρ ) ) ,
where we used ∣ ν m t ∣ ≤ E [ V t 2 ] |\nu m_t| \le\sqrt{\mathbb{E}[V_t^2]} ∣ ν m t ∣ ≤ E [ V t 2 ] and sup u π ρ ′ ( u ) = 1 − 2 ρ 4 \sup_u \pi'_\rho(u) = \frac{1-2\rho}{4} sup u π ρ ′ ( u ) = 4 1 − 2 ρ . Consequently,
∣ E [ V t 2 ] − 1 2 π ∣ ≤ 64 ( 1 + ν ) e − 1 2 ( e − 1 2 + ν ( 1 − 2 ρ ) 4 ) ∫ R ∣ h g t ′ ( h ) ∣ d h . \left|\sqrt{\mathbb{E}[V_t^2]}-\frac{1}{\sqrt{2\pi}}\right| \le\frac{64(1+\nu)}{e^{-\frac{1}{2}}}\left(e^{-\frac{1}{2}}+\frac{\nu(1-2\rho)}{4}\right)\int_{\mathbb{R}} |h g_t'(h)|\,\mathrm{d}h. E [ V t 2 ] − 2 π 1 ≤ e − 2 1 64 ( 1 + ν ) ( e − 2 1 + 4 ν ( 1 − 2 ρ ) ) ∫ R ∣ h g t ′ ( h ) ∣ d h .
For [3], using the inverse relation
h ( a ) = 1 σ t τ t log ( a 1 − a ) + P ( V t ≠ 0 ) δ a , h(a)=\frac{1}{\sigma_t\tau_t}\log\left(\frac{a}{1-a}\right)+\frac{\mathbb{P}(V_t\ne0)}{\delta}a, h ( a ) = σ t τ t 1 log ( 1 − a a ) + δ P ( V t = 0 ) a ,
we obtain
∫ R ∣ h ∣ g t ′ ( h ) d h = ∫ 0 1 ∣ 1 σ t τ t log ( a 1 − a ) + P ( V t ≠ 0 ) δ a ∣ d a ≤ 1 σ t τ t ∫ 0 1 ∣ log ( a 1 − a ) ∣ d a + P ( V t ≠ 0 ) δ ∫ 0 1 a d a = 2 log 2 σ t τ t + P ( V t ≠ 0 ) 2 δ . \begin{aligned}
\int_{\mathbb{R}} |h|g_t'(h)\,\mathrm{d}h
=\int_0^1\left|\frac{1}{\sigma_t\tau_t}\log\left(\frac{a}{1-a}\right)+\frac{\mathbb{P}(V_t\ne0)}{\delta}a\right|\,\mathrm{d}a \\
&\le\frac{1}{\sigma_t\tau_t}\int_0^1\left|\log\left(\frac{a}{1-a}\right)\right|\,\mathrm{d}a+\frac{\mathbb{P}(V_t\ne0)}{\delta}\int_0^1 a\,\mathrm{d}a
=\frac{2\log2}{\sigma_t\tau_t}+\frac{\mathbb{P}(V_t\ne0)}{2\delta}.
\end{aligned} ∫ R ∣ h ∣ g t ′ ( h ) d h = ∫ 0 1 σ t τ t 1 log ( 1 − a a ) + δ P ( V t = 0 ) a d a ≤ σ t τ t 1 ∫ 0 1 log ( 1 − a a ) d a + δ P ( V t = 0 ) ∫ 0 1 a d a = σ t τ t 2 log 2 + 2 δ P ( V t = 0 ) .
Furthermore,
E [ ∣ g t ( H t ) − 1 { H t > 0 } ∣ ] ≤ 1 2 π E [ V t 2 ] ∫ R ∣ g t ( h ) − 1 { h > 0 } ∣ d h ≤ 64 ( 1 + ν ) e − 1 2 ( 2 log 2 σ t τ t + P ( V t ≠ 0 ) 2 δ ) , \mathbb{E}\left[|g_t(H_t)-\mathbf{1}_{\{H_t>0\}}|\right]
\le\frac{1}{\sqrt{2\pi\mathbb{E}[V_t^2]}}\int_{\mathbb{R}}|g_t(h)-\mathbf{1}_{\{h>0\}}|\,\mathrm{d}h
\le\frac{64(1+\nu)}{e^{-\frac{1}{2}}}\left(\frac{2\log2}{\sigma_t\tau_t}+\frac{\mathbb{P}(V_t\ne0)}{2\delta}\right), E [ ∣ g t ( H t ) − 1 { H t > 0 } ∣ ] ≤ 2 π E [ V t 2 ] 1 ∫ R ∣ g t ( h ) − 1 { h > 0 } ∣ d h ≤ e − 2 1 64 ( 1 + ν ) ( σ t τ t 2 log 2 + 2 δ P ( V t = 0 ) ) ,
where the underbrace term equals ∫ R ∣ h ∣ g t ′ ( h ) d h \int_{\mathbb{R}} |h|g_t'(h)\,\mathrm{d}h ∫ R ∣ h ∣ g t ′ ( h ) d h .
Note that,
2 2 π e − 1 2 64 2 π ( 1 + ν ) ( e − 1 2 + ν ( 1 − 2 ρ ) 4 ) + ν ( 1 − 2 ρ ) 2 2 π e − 1 2 64 2 π ( 1 + ν ) = 64 e 1 2 ( 1 + ν ) ( 2 e − 1 2 + ν ( 1 − 2 ρ ) ) . \frac{2}{\sqrt{2\pi}\frac{e^{-\frac{1}{2}}}{64\sqrt{2\pi}(1+\nu)}}\left(e^{-\frac{1}{2}}+\frac{\nu(1-2\rho)}{4}\right)
+\frac{\nu(1-2\rho)}{2\sqrt{2\pi}\frac{e^{-\frac{1}{2}}}{64\sqrt{2\pi}(1+\nu)}}
=64e^{\frac{1}{2}}(1+\nu)\left(2e^{-\frac{1}{2}}+\nu(1-2\rho)\right). 2 π 64 2 π ( 1 + ν ) e − 2 1 2 ( e − 2 1 + 4 ν ( 1 − 2 ρ ) ) + 2 2 π 64 2 π ( 1 + ν ) e − 2 1 ν ( 1 − 2 ρ ) = 64 e 2 1 ( 1 + ν ) ( 2 e − 2 1 + ν ( 1 − 2 ρ ) ) .
Thus Eq. (52) becomes
1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ≤ 64 e 1 2 ( 1 + ν ) ( 2 e − 1 2 + ν ( 1 − 2 ρ ) ) ( 2 log 2 σ t τ t + 33 s 64 δ ) + 2 2 s δ ( 1 + 2 log 64 s ) . \frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)
\le64e^{\frac{1}{2}}(1+\nu)\left(2e^{-\frac{1}{2}}+\nu(1-2\rho)\right)\left(\frac{2\log2}{\sigma_t\tau_t}+\frac{33s}{64\delta}\right)+2\sqrt{\frac{2s}{\delta}\left(1+2\log\frac{64}{s}\right)}. 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ≤ 64 e 2 1 ( 1 + ν ) ( 2 e − 2 1 + ν ( 1 − 2 ρ ) ) ( σ t τ t 2 log 2 + 64 δ 33 s ) + 2 δ 2 s ( 1 + 2 log s 64 ) .
It is sufficient to decrease s 0 s_0 s 0 so that, for every s ≤ s 0 s\le s_0 s ≤ s 0 ,
64 e 1 2 ( 1 + ν ) ( 2 e − 1 2 + ν ( 1 − 2 ρ ) ) 33 s 64 δ + 2 2 s δ ( 1 + 2 log 64 s ) ≤ 1 2 ( 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ) 64e^{\frac{1}{2}}(1+\nu)\left(2e^{-\frac{1}{2}}+\nu(1-2\rho)\right)\frac{33s}{64\delta}+2\sqrt{\frac{2s}{\delta}\left(1+2\log\frac{64}{s}\right)}
\le\frac{1}{2}\left(\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)\right) 64 e 2 1 ( 1 + ν ) ( 2 e − 2 1 + ν ( 1 − 2 ρ ) ) 64 δ 33 s + 2 δ 2 s ( 1 + 2 log s 64 ) ≤ 2 1 ( 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) )
and 33 s 32 δ ≤ e − 1 2 64 2 π ( 1 + ν ) \frac{33s}{32\delta}\le\frac{e^{-\frac{1}{2}}}{64\sqrt{2\pi}(1+\nu)} 32 δ 33 s ≤ 64 2 π ( 1 + ν ) e − 2 1 . Then,
1 2 ( 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ) ≤ 64 e 1 2 ( 1 + ν ) ( 2 e − 1 2 + ν ( 1 − 2 ρ ) ) 2 log 2 σ t τ t , \frac{1}{2}\left(\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)\right)
\le64e^{\frac{1}{2}}(1+\nu)\left(2e^{-\frac{1}{2}}+\nu(1-2\rho)\right)\frac{2\log2}{\sigma_t\tau_t}, 2 1 ( 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ) ≤ 64 e 2 1 ( 1 + ν ) ( 2 e − 2 1 + ν ( 1 − 2 ρ ) ) σ t τ t 2 log 2 ,
which implies σ t τ t ≤ 128 e 1 2 ( 1 + ν ) ( 2 e − 1 2 + ν ( 1 − 2 ρ ) ) log 2 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) \sigma_t\tau_t\le\frac{128e^{\frac{1}{2}}(1+\nu)\left(2e^{-\frac{1}{2}}+\nu(1-2\rho)\right)\log2}{\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)} σ t τ t ≤ 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) 128 e 2 1 ( 1 + ν ) ( 2 e − 2 1 + ν ( 1 − 2 ρ ) ) l o g 2 . Recall h ( ν ) = 1 2 π ∫ R ϕ ′ ′ ( ν z ) e − z 2 2 d z h(\nu)=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}\phi''(\nu z)e^{-\frac{z^2}{2}}\,\mathrm{d}z h ( ν ) = 2 π 1 ∫ R ϕ ′′ ( ν z ) e − 2 z 2 d z . We aim to upper bound 1 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) \frac{1}{\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)} 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) 1 , note that
2 π ( 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ) = 2 ρ + ν ( 1 − 2 ρ ) ∫ R ϕ ′ ′ ( ν z ) ( 1 − e − z 2 2 ) d z \sqrt{2\pi}\left(\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)\right)
=2\rho+\nu(1-2\rho)\int_{\mathbb{R}}\phi''(\nu z)\left(1-e^{-\frac{z^2}{2}}\right)\,\mathrm{d}z 2 π ( 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ) = 2 ρ + ν ( 1 − 2 ρ ) ∫ R ϕ ′′ ( ν z ) ( 1 − e − 2 z 2 ) d z
because ∫ R ϕ ′ ′ ( u ) d u = 1 \int_{\mathbb{R}}\phi''(u)\,\mathrm{d}u=1 ∫ R ϕ ′′ ( u ) d u = 1 , and
∫ R ϕ ′ ′ ( u ) ( 1 − e − u 2 2 ν 2 ) d u ≥ ( 1 − e − 1 2 ν 2 ) ∫ ∣ u ∣ ≥ 1 ϕ ′ ′ ( u ) d u ≥ 2 ( 1 − e − 1 2 ν 2 ) ( 1 − ϕ ′ ( 1 ) ) , \int_{\mathbb{R}}\phi''(u)\left(1-e^{-\frac{u^2}{2\nu^2}}\right)\,\mathrm{d}u
\ge\left(1-e^{-\frac{1}{2\nu^2}}\right)\int_{|u|\ge1}\phi''(u)\,\mathrm{d}u
\ge2\left(1-e^{-\frac{1}{2\nu^2}}\right)(1-\phi'(1)), ∫ R ϕ ′′ ( u ) ( 1 − e − 2 ν 2 u 2 ) d u ≥ ( 1 − e − 2 ν 2 1 ) ∫ ∣ u ∣ ≥ 1 ϕ ′′ ( u ) d u ≥ 2 ( 1 − e − 2 ν 2 1 ) ( 1 − ϕ ′ ( 1 )) ,
where 1 − e − 1 2 ν 2 ⏟ ≥ 1 1 + 2 ν 2 \underbrace{1-e^{-\frac{1}{2\nu^2}}}_{\ge\frac{1}{1+2\nu^2}} ≥ 1 + 2 ν 2 1 1 − e − 2 ν 2 1 and 1 − ϕ ′ ( 1 ) ⏟ = 1 1 + e \underbrace{1-\phi'(1)}_{=\frac{1}{1+e}} = 1 + e 1 1 − ϕ ′ ( 1 ) .
we can therefore take Q = 128 2 π e 1 2 ( 1 + ν ) ( 2 e − 1 2 + ν ( 1 − 2 ρ ) ) log 2 2 ρ + ( 1 + e ) ( 1 + 2 ν 2 ) 2 ( 1 − 2 ρ ) Q = \frac{128\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu)\left(2e^{-\frac{1}{2}}+\nu(1-2\rho)\right)\log2}{2\rho+\frac{(1+e)(1+2\nu^2)}{2(1-2\rho)}} Q = 2 ρ + 2 ( 1 − 2 ρ ) ( 1 + e ) ( 1 + 2 ν 2 ) 128 2 π e 2 1 ( 1 + ν ) ( 2 e − 2 1 + ν ( 1 − 2 ρ ) ) l o g 2 .
We now bound the term E [ ( V t − θ t w ⋆ ) 2 ] \mathbb{E}[(V_t-\theta_t w^\star)^2] E [( V t − θ t w ⋆ ) 2 ] for every t ≥ 2 t\ge2 t ≥ 2 . Since γ t = σ t τ t P ( V t ≠ 0 ) δ ≤ Q 3 s 3 / 2 32 δ \gamma_t=\frac{\sigma_t\tau_t\mathbb{P}(V_t\ne0)}{\delta}\le\frac{Q3s^{3/2}}{32\delta} γ t = δ σ t τ t P ( V t = 0 ) ≤ 32 δ Q 3 s 3/2 , we get,
0 < ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = 2 ∣ E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ) ] ∣ 0<\left|\theta_t-(1-2\rho)h(\nu)\right|=2\left|\mathbb{E}\left[\pi_\rho'(\nu Z_1)\left(\phi'(\nu\alpha_tZ_1+\sigma_tZ_2)-\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))\right)\right]\right| 0 < ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = 2 E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ) ]
≤ ϕ ′ ′ ( u ) ≤ 1 4 1 2 E [ ∣ ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ∣ ] = Eq. (49) γ t 2 E [ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ≤ γ t 2 ( 1 − 2 ρ ) h ( ν ) (53) \begin{aligned}
\le_{\phi''(u)\le\frac{1}{4}}\frac{1}{2}\mathbb{E}\left[\left|\nu\alpha_tZ_1+\sigma_tZ_2-\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right|\right] \\
&=_{\text{Eq. (49)}}\frac{\gamma_t}{2}\mathbb{E}\left[\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))\right]\le\frac{\gamma_t}{2}(1-2\rho)h(\nu) \tag*{(53)}
\end{aligned} ≤ ϕ ′′ ( u ) ≤ 4 1 2 1 E [ ∣ ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ∣ ] = Eq. (49) 2 γ t E [ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] ≤ 2 γ t ( 1 − 2 ρ ) h ( ν ) (53)
where 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = ( 1 − 2 ρ ) h ( ν ) 2\mathbb{E}[\pi_\rho'(\nu Z_1)\phi'(\nu\alpha_tZ_1+\sigma_tZ_2)]=(1-2\rho)h(\nu) 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 )] = ( 1 − 2 ρ ) h ( ν ) follows because ( Z 1 , Z 2 ) = d ( − Z 1 , − Z 2 ) (Z_1,Z_2)\mathrel{\mathop=^{d}}(-Z_1,-Z_2) ( Z 1 , Z 2 ) = d ( − Z 1 , − Z 2 ) and π ρ ′ ( ν Z 1 ) \pi_\rho'(\nu Z_1) π ρ ′ ( ν Z 1 ) is even. Explicitly,
2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) + ϕ ′ ( − ν α t Z 1 − σ t Z 2 ) ) ] = E [ π ρ ′ ( ν Z 1 ) ] = ( 1 − 2 ρ ) E [ ϕ ′ ′ ( ν Z 1 ) ] = ( 1 − 2 ρ ) h ( ν ) , (54) \begin{aligned}
2\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'(\nu\alpha_tZ_1+\sigma_tZ_2)\right]=\mathbb{E}\left[\pi_\rho'(\nu Z_1)\left(\phi'(\nu\alpha_tZ_1+\sigma_tZ_2)+\phi'(-\nu\alpha_tZ_1-\sigma_tZ_2)\right)\right] \\
&=\mathbb{E}[\pi_\rho'(\nu Z_1)]=(1-2\rho)\mathbb{E}[\phi''(\nu Z_1)]=(1-2\rho)h(\nu), \tag*{(54)}
\end{aligned} 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) + ϕ ′ ( − ν α t Z 1 − σ t Z 2 ) ) ] = E [ π ρ ′ ( ν Z 1 )] = ( 1 − 2 ρ ) E [ ϕ ′′ ( ν Z 1 )] = ( 1 − 2 ρ ) h ( ν ) , (54)
and by Eq. (50) θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] \theta_t=2\mathbb{E}[\pi_\rho'(\nu Z_1)\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))] θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ))] . Therefore, we have θ t = ( 1 − 2 ρ ) h ( ν ) + O ( s ) \theta_t=(1-2\rho)h(\nu)+O(s) θ t = ( 1 − 2 ρ ) h ( ν ) + O ( s ) . So, there exists s 0 s_0 s 0 , such that for every s ≤ s 0 s\le s_0 s ≤ s 0 and every t t t we have θ t ≥ 1 2 ( 1 − 2 ρ ) h ( ν ) = θ > 0 \theta_t\ge\frac{1}{2}(1-2\rho)h(\nu)=\theta>0 θ t ≥ 2 1 ( 1 − 2 ρ ) h ( ν ) = θ > 0 . We first refine the contribution of the active coordinates. From symmetry it is enough to consider w ⋆ > 0 w^\star>0 w ⋆ > 0 . Recall that, on w ⋆ ≠ 0 w^\star\ne0 w ⋆ = 0 , w ⋆ = ν s G w^\star=\frac{\nu}{\sqrt{s}}G w ⋆ = s ν G , with G ∼ N ( 0 , 1 ) G\sim N(0,1) G ∼ N ( 0 , 1 ) . Put a = 8 2 δ log ( 64 s ) s ( 1 − 2 ρ ) h ( ν ) ν a=\frac{8\sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)}\sqrt{s}}{(1-2\rho)h(\nu)\nu} a = ( 1 − 2 ρ ) h ( ν ) ν 8 δ 2 l o g ( s 64 ) s . The coordinates with ∣ G ∣ < a |G|<a ∣ G ∣ < a have probability
P ( ∣ G ∣ < a ) = 2 ∫ 0 a 1 2 π e − t 2 2 d t ≤ 2 π a . \mathbb{P}(|G|<a)=2\int_0^a\sqrt{\frac{1}{2\pi}}e^{-\frac{t^2}{2}}\,dt\le\sqrt{\frac{2}{\pi}}a. P ( ∣ G ∣ < a ) = 2 ∫ 0 a 2 π 1 e − 2 t 2 d t ≤ π 2 a .
Using the crude bound as both thresholds are at most λ 0 ϵ s = 2 δ log ( 64 s ) \frac{\lambda_0}{\epsilon_s}=\sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)} ϵ s λ 0 = δ 2 log ( s 64 ) ,
∣ V t − θ t w ⋆ ∣ 2 ≤ 2 r t 2 δ Z t 2 + ( 2 2 δ log ( 64 s ) ) 2 ≤ 2 δ Z t 2 + 8 δ log ( 64 s ) , |V_t-\theta_t w^\star|^2\le2\frac{r_t^2}{\delta}Z_t^2+\left(2\sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)}\right)^2\le\frac{2}{\delta}Z_t^2+\frac{8}{\delta}\log\left(\frac{64}{s}\right), ∣ V t − θ t w ⋆ ∣ 2 ≤ 2 δ r t 2 Z t 2 + ( 2 δ 2 log ( s 64 ) ) 2 ≤ δ 2 Z t 2 + δ 8 log ( s 64 ) ,
therefore
s E [ ( V t − θ t w ⋆ ) 2 1 { w ⋆ ≠ 0 , ∣ G ∣ < a } ] ≤ 2 s 2 π a ( 1 δ + 2 δ log ( 64 s ) ) = 16 s 4 s δ log ( 64 s ) ( 1 − 2 ρ ) h ( ν ) ν ( 1 δ + 2 δ log ( 64 s ) ) = o ( s ) . s\mathbb{E}\left[(V_t-\theta_t w^\star)^2\mathbf{1}_{\{w^\star\ne0,|G|<a\}}\right]\le2s\sqrt{\frac{2}{\pi}}a\left(\frac{1}{\delta}+\frac{2}{\delta}\log\left(\frac{64}{s}\right)\right)=\frac{16s\sqrt{\frac{4s}{\delta}\log\left(\frac{64}{s}\right)}}{(1-2\rho)h(\nu)\nu}\left(\frac{1}{\delta}+\frac{2}{\delta}\log\left(\frac{64}{s}\right)\right)=o(s). s E [ ( V t − θ t w ⋆ ) 2 1 { w ⋆ = 0 , ∣ G ∣ < a } ] ≤ 2 s π 2 a ( δ 1 + δ 2 log ( s 64 ) ) = ( 1 − 2 ρ ) h ( ν ) ν 16 s δ 4 s log ( s 64 ) ( δ 1 + δ 2 log ( s 64 ) ) = o ( s ) .
For ∣ G ∣ ≥ a |G|\ge a ∣ G ∣ ≥ a , in a ’good’ event (large enough to not destroy the positive teacher signal) Z t − 1 ≥ − θ w ⋆ δ 4 Z_{t-1}\ge-\frac{\theta w^\star\sqrt{\delta}}{4} Z t − 1 ≥ − 4 θ w ⋆ δ , we want to find the threshold in the correct direction, in that case we analyze the case w ⋆ = ν s G ≥ 8 2 δ log ( 64 s ) ( 1 − 2 ρ ) h ( ν ) w^\star=\frac{\nu}{\sqrt{s}}G\ge\frac{8\sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)}}{(1-2\rho)h(\nu)} w ⋆ = s ν G ≥ ( 1 − 2 ρ ) h ( ν ) 8 δ 2 l o g ( s 64 ) and Z t − 1 ≥ − 1 2 ( 1 − 2 ρ ) h ( ν ) w ⋆ δ 4 Z_{t-1}\ge-\frac{\frac{1}{2}(1-2\rho)h(\nu)w^\star\sqrt{\delta}}{4} Z t − 1 ≥ − 4 2 1 ( 1 − 2 ρ ) h ( ν ) w ⋆ δ . Since r t − 1 ≤ 1 r_{t-1}\le1 r t − 1 ≤ 1 and θ t − 1 ≥ θ \theta_{t-1}\ge\theta θ t − 1 ≥ θ ,
θ t − 1 w ⋆ + r t − 1 δ Z t − 1 ≥ θ w ⋆ − θ 4 w ⋆ = 3 4 ( 1 − 2 ρ ) h ( ν ) w ⋆ . \theta_{t-1}w^\star+\frac{r_{t-1}}{\sqrt{\delta}}Z_{t-1}\ge\theta w^\star-\frac{\theta}{4}w^\star=\frac{3}{4}(1-2\rho)h(\nu)w^\star. θ t − 1 w ⋆ + δ r t − 1 Z t − 1 ≥ θ w ⋆ − 4 θ w ⋆ = 4 3 ( 1 − 2 ρ ) h ( ν ) w ⋆ .
The positive threshold at task t − 1 t-1 t − 1 is at most 2 δ log ( 64 s ) \sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)} δ 2 log ( s 64 ) , and hence
V t − 1 ≥ 3 8 ( 1 − 2 ρ ) h ( ν ) w ⋆ − 2 δ log ( 64 s ) ≥ 3 8 ( 1 − 2 ρ ) h ( ν ) w ⋆ − 1 8 ( 1 − 2 ρ ) h ( ν ) w ⋆ = 1 4 ( 1 − 2 ρ ) h ( ν ) w ⋆ . V_{t-1}\ge\frac{3}{8}(1-2\rho)h(\nu)w^\star-\sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)}\ge\frac{3}{8}(1-2\rho)h(\nu)w^\star-\frac{1}{8}(1-2\rho)h(\nu)w^\star=\frac{1}{4}(1-2\rho)h(\nu)w^\star. V t − 1 ≥ 8 3 ( 1 − 2 ρ ) h ( ν ) w ⋆ − δ 2 log ( s 64 ) ≥ 8 3 ( 1 − 2 ρ ) h ( ν ) w ⋆ − 8 1 ( 1 − 2 ρ ) h ( ν ) w ⋆ = 4 1 ( 1 − 2 ρ ) h ( ν ) w ⋆ .
Since σ t − 1 τ t − 1 ≥ 1 \sigma_{t-1}\tau_{t-1}\ge1 σ t − 1 τ t − 1 ≥ 1 ,
w t − 1 = σ t − 1 τ t − 1 V t − 1 ≥ 1 4 ( 1 − 2 ρ ) h ( ν ) w ⋆ . w_{t-1}=\sigma_{t-1}\tau_{t-1}V_{t-1}\ge\frac{1}{4}(1-2\rho)h(\nu)w^\star. w t − 1 = σ t − 1 τ t − 1 V t − 1 ≥ 4 1 ( 1 − 2 ρ ) h ( ν ) w ⋆ .
Consequently, the positive threshold at task t t t satisfies
λ 0 [ w t − 1 ] + + ϵ s ≤ 2 λ 0 1 2 ( 1 − 2 ρ ) h ( ν ) w ⋆ . (55) \frac{\lambda_0}{[w_{t-1}]_++\epsilon_s}\le\frac{2\lambda_0}{\frac{1}{2}(1-2\rho)h(\nu)w^\star}. \tag*{(55)} [ w t − 1 ] + + ϵ s λ 0 ≤ 2 1 ( 1 − 2 ρ ) h ( ν ) w ⋆ 2 λ 0 . (55)
Since E [ ( V t − θ t w ⋆ ) 2 ] ≤ r t 2 Z t 2 δ + 2 ( λ 0 [ w t − 1 ] + + ϵ s ) 2 \mathbb{E}[(V_t-\theta_t w^\star)^2]\leq\frac{r_t^2 Z_t^2}{\delta}+2\left(\frac{\lambda_0}{[w_{t-1}]_++\epsilon_s}\right)^2 E [( V t − θ t w ⋆ ) 2 ] ≤ δ r t 2 Z t 2 + 2 ( [ w t − 1 ] + + ϵ s λ 0 ) 2 . The second term is bounded by 32 λ 0 2 s 2 ( 1 − 2 ρ ) 2 h ( ν ) 2 ν 2 G 2 \frac{32\lambda_0^2s^2}{(1-2\rho)^2h(\nu)^2\nu^2G^2} ( 1 − 2 ρ ) 2 h ( ν ) 2 ν 2 G 2 32 λ 0 2 s 2 because an active coordinate occurs with probability s s s , its contribution is 32 λ 0 2 s 2 ( 1 − 2 ρ ) 2 h ( ν ) 2 ν 2 E [ 1 { ∣ G ∣ ≥ a } G 2 ] \frac{32\lambda_0^2s^2}{(1-2\rho)^2h(\nu)^2\nu^2}\mathbb{E}\left[\frac{\mathbf{1}_{\{|G|\geq a\}}}{G^2}\right] ( 1 − 2 ρ ) 2 h ( ν ) 2 ν 2 32 λ 0 2 s 2 E [ G 2 1 { ∣ G ∣ ≥ a } ] at most, and
E [ 1 { ∣ G ∣ ≥ a } G 2 ] = 2 ∫ a ∞ φ ( g ) g 2 d g ≤ 2 φ ( 0 ) a ≤ 2 π 1 a . \mathbb{E}\left[\frac{\mathbf{1}_{\{|G|\geq a\}}}{G^2}\right]=2\int_a^\infty\frac{\varphi(g)}{g^2}\,dg\leq2\frac{\varphi(0)}{a}\leq\sqrt{\frac{2}{\pi}}\frac{1}{a}. E [ G 2 1 { ∣ G ∣ ≥ a } ] = 2 ∫ a ∞ g 2 φ ( g ) d g ≤ 2 a φ ( 0 ) ≤ π 2 a 1 .
this is O ( s 3 / 2 2 δ log ( 64 s ) ) = o ( s ) O\left(\frac{s^{3/2}}{\sqrt{\frac{2}{\delta}\log\left(\frac{64}{s}\right)}}\right)=o(s) O ( δ 2 l o g ( s 64 ) s 3/2 ) = o ( s ) . It remains to control the other events. We are left with the event in which ∣ G ∣ ≥ a |G|\geq a ∣ G ∣ ≥ a and Z t − 1 < − θ t w ⋆ δ 4 Z_{t-1}<-\frac{\theta_t w^\star\sqrt{\delta}}{4} Z t − 1 < − 4 θ t w ⋆ δ , because w ⋆ > 0 w^\star>0 w ⋆ > 0 , and w ⋆ = ν s G w^\star=\frac{\sqrt{\nu}}{\sqrt{s}}G w ⋆ = s ν G this event simply reduces to G ≥ a G\geq a G ≥ a and Z t − 1 < − ( 1 − 2 ρ ) h ( ν ) ν δ G 8 s Z_{t-1}<-\frac{(1-2\rho)h(\nu)\nu\sqrt{\delta}G}{8\sqrt{s}} Z t − 1 < − 8 s ( 1 − 2 ρ ) h ( ν ) ν δ G , which we denote by B t − 1 B_{t-1} B t − 1 . For every fixed b > 0 b>0 b > 0 ,
E [ Φ ( − b ∣ G ∣ s ) ] = 1 π arctan ( s b ) ≤ s π b . \mathbb{E}\left[\Phi\left(-\frac{b|G|}{\sqrt{s}}\right)\right]=\frac{1}{\pi}\arctan\left(\frac{\sqrt{s}}{b}\right)\leq\frac{\sqrt{s}}{\pi b}. E [ Φ ( − s b ∣ G ∣ ) ] = π 1 arctan ( b s ) ≤ π b s .
So,
E [ ( V t − θ t w ⋆ ) 2 1 { w ⋆ ≠ 0 } 1 { B t − 1 } ] ≤ s ( 2 δ + 4 δ log ( 64 s ) ) P ( G ≥ a , Z t − 1 < − b G s ∣ w ⋆ ≠ 0 ) ≤ s ( 2 δ + 4 δ log ( 64 s ) ) E [ Φ ( − b ∣ G ∣ s ) ] . \begin{aligned}
\mathbb{E}\left[(V_t-\theta_t w^\star)^2\mathbf{1}_{\{w^\star\neq0\}}\mathbf{1}_{\{B_{t-1}\}}\right]\leq s\left(\frac{2}{\delta}+\frac{4}{\delta}\log\left(\frac{64}{s}\right)\right)\mathbb{P}\left(G\geq a,Z_{t-1}<-\frac{bG}{\sqrt{s}}\mid w^\star\neq0\right) \\
&\leq s\left(\frac{2}{\delta}+\frac{4}{\delta}\log\left(\frac{64}{s}\right)\right)\mathbb{E}\left[\Phi\left(-\frac{b|G|}{\sqrt{s}}\right)\right].
\end{aligned} E [ ( V t − θ t w ⋆ ) 2 1 { w ⋆ = 0 } 1 { B t − 1 } ] ≤ s ( δ 2 + δ 4 log ( s 64 ) ) P ( G ≥ a , Z t − 1 < − s b G ∣ w ⋆ = 0 ) ≤ s ( δ 2 + δ 4 log ( s 64 ) ) E [ Φ ( − s b ∣ G ∣ ) ] .
We can drop G ≥ a G\geq a G ≥ a because doing so only increases the probability. Thus both the event contributes at most O ( s 3 / 2 log ( 64 s ) ) = o ( s ) O\left(s^{3/2}\log\left(\frac{64}{s}\right)\right)=o(s) O ( s 3/2 log ( s 64 ) ) = o ( s ) . The first term of the Gaussian noise on active coordinates contributes at most E [ 2 r t 2 δ Z t 2 1 { w ⋆ ≠ 0 } ] ≤ 2 s δ \mathbb{E}\left[\frac{2r_t^2}{\delta}Z_t^2\mathbf{1}_{\{w^\star\neq0\}}\right]\leq\frac{2s}{\delta} E [ δ 2 r t 2 Z t 2 1 { w ⋆ = 0 } ] ≤ δ 2 s . Therefore, after decreasing s 0 > 0 s_0>0 s 0 > 0 we can require that for all s ≤ s 0 s\leq s_0 s ≤ s 0 , o ( s ) ≤ s 2 δ o(s)\leq\frac{s}{2\delta} o ( s ) ≤ 2 δ s , then,
E [ ( V t − θ t w ⋆ ) 2 1 { w ⋆ ≠ 0 } ] ≤ 5 s 2 δ , (56) \mathbb{E}\left[(V_t-\theta_t w^\star)^2\mathbf{1}_{\{w^\star\neq0\}}\right]\leq\frac{5s}{2\delta}, \tag*{(56)} E [ ( V t − θ t w ⋆ ) 2 1 { w ⋆ = 0 } ] ≤ 2 δ 5 s , (56)
for all t ≥ 2 t\geq2 t ≥ 2 . We now look at the null-coordinate contribution. On { w ⋆ = 0 , w t − 1 ≠ 0 } \{w^\star=0,w_{t-1}\neq0\} { w ⋆ = 0 , w t − 1 = 0 } , the previous argument already gives
E [ ( V t − θ t w ⋆ ) 2 1 { w ⋆ = 0 , w t − 1 ≠ 0 } ] = E [ V t 2 1 { w ⋆ = 0 , w t − 1 ≠ 0 } ] ≤ 1 δ E [ Z t 2 ] P ( w ⋆ = 0 , w t − 1 ≠ 0 ) ≤ s 32 δ . \mathbb{E}\left[(V_t-\theta_t w^\star)^2\mathbf{1}_{\{w^\star=0,w_{t-1}\neq0\}}\right]=\mathbb{E}\left[V_t^2\mathbf{1}_{\{w^\star=0,w_{t-1}\neq0\}}\right]\leq\frac{1}{\delta}\mathbb{E}[Z_t^2]\mathbb{P}(w^\star=0,w_{t-1}\neq0)\leq\frac{s}{32\delta}. E [ ( V t − θ t w ⋆ ) 2 1 { w ⋆ = 0 , w t − 1 = 0 } ] = E [ V t 2 1 { w ⋆ = 0 , w t − 1 = 0 } ] ≤ δ 1 E [ Z t 2 ] P ( w ⋆ = 0 , w t − 1 = 0 ) ≤ 32 δ s .
On { w ⋆ = 0 , w t − 1 = 0 } \{w^\star=0,w_{t-1}=0\} { w ⋆ = 0 , w t − 1 = 0 } , V t = r t δ S 2 log ( 64 / s ) r t 2 ( Z t ) V_t=\frac{r_t}{\sqrt{\delta}}S\sqrt{\frac{2\log(64/s)}{r_t^2}}(Z_t) V t = δ r t S r t 2 2 l o g ( 64/ s ) ( Z t ) . For u > 0 u>0 u > 0 ,
E [ S u ( Z ) 2 ] = E [ ( ∣ Z ∣ − u ) + 2 ] = 2 ∫ u ∞ ( z − u ) 2 φ ( z ) d z = 2 φ ( u ) ∫ 0 ∞ y 2 e − u y − y 2 2 d y \mathbb{E}[S_u(Z)^2]=\mathbb{E}[(|Z|-u)_+^2]=2\int_u^\infty(z-u)^2\varphi(z)\,dz=2\varphi(u)\int_0^\infty y^2e^{-uy-\frac{y^2}{2}}\,dy E [ S u ( Z ) 2 ] = E [( ∣ Z ∣ − u ) + 2 ] = 2 ∫ u ∞ ( z − u ) 2 φ ( z ) d z = 2 φ ( u ) ∫ 0 ∞ y 2 e − u y − 2 y 2 d y
≤ 2 φ ( u ) ∫ 0 ∞ y 2 e − u y d y = 4 φ ( u ) u 3 . \leq2\varphi(u)\int_0^\infty y^2e^{-uy}\,dy=\frac{4\varphi(u)}{u^3}. ≤ 2 φ ( u ) ∫ 0 ∞ y 2 e − u y d y = u 3 4 φ ( u ) .
Since r t ≤ 1 r_t\leq1 r t ≤ 1 (see conclusion beneath Eq. (48) ), then 2 log ( 64 / s ) r t ≥ 2 log ( 64 / s ) \frac{\sqrt{2\log(64/s)}}{r_t}\geq\sqrt{2\log(64/s)} r t 2 l o g ( 64/ s ) ≥ 2 log ( 64/ s ) . Consequently,
E [ V t 2 1 { w ⋆ = 0 , w t − 1 = 0 } ] ≤ 4 δ φ ( 2 log ( 64 / s ) ) ( 2 log ( 64 / s ) ) 3 / 2 = 4 δ 1 2 π e − log ( 64 / s ) 1 ( 2 log ( 64 / s ) ) 3 / 2 = s 16 2 π δ ( 2 log ( 64 / s ) ) 3 / 2 = o ( s ) . \mathbb{E}\left[V_t^2\mathbf{1}_{\{w^\star=0,w_{t-1}=0\}}\right]\leq\frac{4}{\delta}\frac{\varphi\left(\sqrt{2\log(64/s)}\right)}{\left(2\log(64/s)\right)^{3/2}}=\frac{4}{\delta}\frac{1}{\sqrt{2\pi}}e^{-\log(64/s)}\frac{1}{\left(2\log(64/s)\right)^{3/2}}=\frac{s}{16\sqrt{2\pi}\delta\left(2\log(64/s)\right)^{3/2}}=o(s). E [ V t 2 1 { w ⋆ = 0 , w t − 1 = 0 } ] ≤ δ 4 ( 2 log ( 64/ s ) ) 3/2 φ ( 2 log ( 64/ s ) ) = δ 4 2 π 1 e − l o g ( 64/ s ) ( 2 log ( 64/ s ) ) 3/2 1 = 16 2 π δ ( 2 log ( 64/ s ) ) 3/2 s = o ( s ) .
Combining the active and null contributions gives, after decreasing s 0 > 0 s_0>0 s 0 > 0 we can require that for all s ≤ s 0 s\leq s_0 s ≤ s 0 , o ( s ) ≤ s 32 δ o(s)\leq\frac{s}{32\delta} o ( s ) ≤ 32 δ s , and then, uniformly in t ≥ 2 t\geq2 t ≥ 2 ,
E [ ( V t − θ t w ⋆ ) 2 ] ≤ ( 5 2 + 1 32 ) s δ = 81 s 32 δ < 3 s δ . (57) \mathbb{E}[(V_t-\theta_t w^\star)^2]\leq\left(\frac{5}{2}+\frac{1}{32}\right)\frac{s}{\delta}=\frac{81s}{32\delta}<\frac{3s}{\delta}. \tag*{(57)} E [( V t − θ t w ⋆ ) 2 ] ≤ ( 2 5 + 32 1 ) δ s = 32 δ 81 s < δ 3 s . (57)
Consequently, since m t − θ t = E [ w ∗ V t ] ν 2 − θ t = 1 ν 2 E [ w ∗ ( V t − θ t w ∗ ) ] m_t-\theta_t=\frac{\mathbb{E}[w^*V_t]}{\nu^2}-\theta_t=\frac{1}{\nu^2}\mathbb{E}[w^*(V_t-\theta_tw^*)] m t − θ t = ν 2 E [ w ∗ V t ] − θ t = ν 2 1 E [ w ∗ ( V t − θ t w ∗ )] , we have by Cauchy-Schwartz,
∣ m t − θ t ∣ ≤ 1 ν 2 E [ w ∗ 2 ] E [ ( V t − θ t w ∗ ) 2 ] = 1 ν E [ ( V t − θ t w ∗ ) 2 ] ≤ 3 s ν 2 δ |m_t-\theta_t|\leq\frac{1}{\nu^2}\sqrt{\mathbb{E}[w^{*2}]\mathbb{E}[(V_t-\theta_tw^*)^2]}=\frac{1}{\nu}\sqrt{\mathbb{E}[(V_t-\theta_tw^*)^2]}\leq\sqrt{\frac{3s}{\nu^2\delta}} ∣ m t − θ t ∣ ≤ ν 2 1 E [ w ∗ 2 ] E [( V t − θ t w ∗ ) 2 ] = ν 1 E [( V t − θ t w ∗ ) 2 ] ≤ ν 2 δ 3 s
and, combining both cases and because m t w ∗ m_tw^* m t w ∗ is the L 2 L^2 L 2 -projection of V t V_t V t onto w ∗ w^* w ∗ ,9
E [ V t 2 ] − ν 2 m t 2 ≤ 3 s δ . \mathbb{E}[V_t^2]-\nu^2m_t^2\leq\frac{3s}{\delta}. E [ V t 2 ] − ν 2 m t 2 ≤ δ 3 s .
The estimates now imply from the first equation,
∣ α t − σ t τ t θ t ∣ = σ t τ t ∣ m t − θ t ∣ ≤ Q 3 s ν 2 δ |\alpha_t-\sigma_t\tau_t\theta_t|=\sigma_t\tau_t|m_t-\theta_t|\leq Q\sqrt{\frac{3s}{\nu^2\delta}} ∣ α t − σ t τ t θ t ∣ = σ t τ t ∣ m t − θ t ∣ ≤ Q ν 2 δ 3 s
and from the third equation,
σ t 2 = ( σ t τ t ) 2 ( E [ V t 2 ] − ν 2 m t 2 ) ≤ 3 s δ Q 2 . \sigma_t^2=(\sigma_t\tau_t)^2\left(\mathbb{E}[V_t^2]-\nu^2m_t^2\right)\leq\frac{3s}{\delta}Q^2. σ t 2 = ( σ t τ t ) 2 ( E [ V t 2 ] − ν 2 m t 2 ) ≤ δ 3 s Q 2 .
Also, γ t = σ t + π ( ν V t ≠ 0 ) δ ≤ Q 33 s 32 δ \gamma_t=\frac{\sigma_t+\pi(\nu V_t\ne0)}{\delta}\leq Q\frac{33s}{32\delta} γ t = δ σ t + π ( ν V t = 0 ) ≤ Q 32 δ 33 s . Put r t = ν 2 α t 2 + σ t 2 r_t=\sqrt{\nu^2\alpha_t^2+\sigma_t^2} r t = ν 2 α t 2 + σ t 2 . The proximal identity and the bounded derivatives of ϕ ′ \phi' ϕ ′ and ϕ ′ ′ \phi'' ϕ ′′ with (54) give θ t = ( 1 − 2 ρ ) h ( ν ) + O ( γ t ) \theta_t=(1-2\rho)h(\nu)+O(\gamma_t) θ t = ( 1 − 2 ρ ) h ( ν ) + O ( γ t ) and from symmetry
2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] , 2\mathbb{E}[(1-\pi_\rho(\nu Z_1))\phi''(\nu\alpha_tZ_1+\sigma_tZ_2)]=\mathbb{E}[\phi''(\nu\alpha_tZ_1+\sigma_tZ_2)], 2 E [( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( ν α t Z 1 + σ t Z 2 )] = E [ ϕ ′′ ( ν α t Z 1 + σ t Z 2 )] ,
and 1 σ t τ t = h ( E [ V t 2 ] ) + O ( γ t ) \frac{1}{\sigma_t\tau_t}=h\left(\sqrt{\mathbb{E}[V_t^2]}\right)+O(\gamma_t) σ t τ t 1 = h ( E [ V t 2 ] ) + O ( γ t ) . Since α t σ t τ t = m t \frac{\alpha_t}{\sigma_t\tau_t}=m_t σ t τ t α t = m t , it follows that
∣ α t h ( ν 2 α t 2 + σ t 2 ) − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ α t ∣ h ( ν 2 α t 2 + σ t 2 ) − 1 σ t τ t ∣ + ∣ α t σ t τ t − θ t ∣ + ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = α t ∣ h ( ν 2 α t 2 + σ t 2 ) − 1 σ t τ t ∣ + ∣ m t − θ t ∣ + ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = ≤ α t ∣ E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] − E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ∣ = + ∣ m t − θ t ∣ + ∣ 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ ≤ α t E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ∣ ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ ] ≤ + α t E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ 1 − 1 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ ] ≤ + ∣ m t − θ t ∣ + 2 E [ π ρ ′ ( ν Z 1 ) ∣ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) − ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ∣ ] ≤ ≤ α t γ t 6 3 + α t γ t 16 + ∣ m t − θ t ∣ + γ t 2 E [ π ρ ′ ( ν Z 1 ) ] = α t γ t ( 1 6 3 + 1 16 ) + ∣ m t − θ t ∣ + γ t 2 ( 1 − 2 ρ ) h ( ν ) . \begin{aligned}
\left|\alpha_th\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)-(1-2\rho)h(\nu)\right|
&\leq\alpha_t\left|h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)-\frac{1}{\sigma_t\tau_t}\right|+\left|\frac{\alpha_t}{\sigma_t\tau_t}-\theta_t\right|+|\theta_t-(1-2\rho)h(\nu)|\\
&=\alpha_t\left|h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)-\frac{1}{\sigma_t\tau_t}\right|+|m_t-\theta_t|+|\theta_t-(1-2\rho)h(\nu)|\\
&\mathrel{\phantom{=}} \leq\alpha_t\left|\mathbb{E}[2(1-\pi_\rho(\nu Z_1))\phi''(\nu\alpha_tZ_1+\sigma_tZ_2)]-\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}\right]\right|\\
&\mathrel{\phantom{=}}+|m_t-\theta_t|+\left|2\mathbb{E}[\pi_\rho'(\nu Z_1)\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))]-(1-2\rho)h(\nu)\right|\\
&\mathrel{\phantom{\leq}} \leq\alpha_t\mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))\left|\phi''(\nu\alpha_tZ_1+\sigma_tZ_2)-\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))\right|\right]\\
&\mathrel{\phantom{\leq}}+\alpha_t\mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))\left|1-\frac{1}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}\right|\right]\\
&\mathrel{\phantom{\leq}}+|m_t-\theta_t|+2\mathbb{E}\left[\pi_\rho'(\nu Z_1)\left|\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))-\phi'(\nu\alpha_tZ_1+\sigma_tZ_2)\right|\right]\\
&\mathrel{\phantom{\leq}} \leq\frac{\alpha_t\gamma_t}{6\sqrt{3}}+\frac{\alpha_t\gamma_t}{16}+|m_t-\theta_t|+\frac{\gamma_t}{2}\mathbb{E}[\pi_\rho'(\nu Z_1)]\\
&=\alpha_t\gamma_t\left(\frac{1}{6\sqrt{3}}+\frac{1}{16}\right)+|m_t-\theta_t|+\frac{\gamma_t}{2}(1-2\rho)h(\nu).
\end{aligned} α t h ( ν 2 α t 2 + σ t 2 ) − ( 1 − 2 ρ ) h ( ν ) ≤ α t h ( ν 2 α t 2 + σ t 2 ) − σ t τ t 1 + σ t τ t α t − θ t + ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = α t h ( ν 2 α t 2 + σ t 2 ) − σ t τ t 1 + ∣ m t − θ t ∣ + ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = ≤ α t E [ 2 ( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( ν α t Z 1 + σ t Z 2 )] − E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] = + ∣ m t − θ t ∣ + 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ))] − ( 1 − 2 ρ ) h ( ν ) ≤ ≤ α t E [ 2 ( 1 − π ρ ( ν Z 1 )) ∣ ϕ ′′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ∣ ] ≤ + α t E [ 2 ( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 1 − 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 1 ] ≤ + ∣ m t − θ t ∣ + 2 E [ π ρ ′ ( ν Z 1 ) ∣ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) − ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ∣ ] ≤ ≤ 6 3 α t γ t + 16 α t γ t + ∣ m t − θ t ∣ + 2 γ t E [ π ρ ′ ( ν Z 1 )] = α t γ t ( 6 3 1 + 16 1 ) + ∣ m t − θ t ∣ + 2 γ t ( 1 − 2 ρ ) h ( ν ) .
Here, (1) follows from the sixth scalar equation
1 σ t τ t = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] , \frac{1}{\sigma_t\tau_t}=\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))}\right], σ t τ t 1 = E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 2 ( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] ,
together with
2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] = h ( ν 2 α t 2 + σ t 2 ) . 2\mathbb{E}[(1-\pi_\rho(\nu Z_1))\phi''(\nu\alpha_tZ_1+\sigma_tZ_2)]=\mathbb{E}[\phi''(\nu\alpha_tZ_1+\sigma_tZ_2)]=h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right). 2 E [( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( ν α t Z 1 + σ t Z 2 )] = E [ ϕ ′′ ( ν α t Z 1 + σ t Z 2 )] = h ( ν 2 α t 2 + σ t 2 ) .
and by Eq. (50) θ t = 2 E [ π p ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] \theta_t = 2\mathbb{E}[\pi'_p(\nu Z_1)\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))] θ t = 2 E [ π p ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ))] . Furthermore, since π p ′ ( u ) \pi'_p(u) π p ′ ( u ) is even, ϕ ′ ( u ) + ϕ ′ ( − u ) = 1 \phi'(u)+\phi'(-u)=1 ϕ ′ ( u ) + ϕ ′ ( − u ) = 1 ,
2 E [ π p ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ π p ′ ( ν Z 1 ) ] = ( 1 − 2 ρ ) h ( ν ) . 2\mathbb{E}\left[\pi'_p(\nu Z_1)\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)\right] = \mathbb{E}[\pi'_p(\nu Z_1)] = (1-2\rho)h(\nu). 2 E [ π p ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ π p ′ ( ν Z 1 )] = ( 1 − 2 ρ ) h ( ν ) .
This gives [2]. For [3], since 0 < ϕ ′ ( u ) < 1 0<\phi'(u)<1 0 < ϕ ′ ( u ) < 1 , and the proximal optimality give,
∣ ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ∣ = ∣ γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ ≤ γ t . \left|\nu\alpha_t Z_1+\sigma_t Z_2-\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right| = \left|\gamma_t\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\right| \leq\gamma_t. ∣ ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ∣ = ∣ γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ∣ ≤ γ t .
Since sup u ∈ R ∣ ϕ ′ ′ ′ ( u ) ∣ = 1 6 3 \sup_{u\in\mathbb{R}}|\phi'''(u)|=\frac{1}{6\sqrt{3}} sup u ∈ R ∣ ϕ ′′′ ( u ) ∣ = 6 3 1 , so the function ϕ ′ ′ ( u ) \phi''(u) ϕ ′′ ( u ) is 1 6 3 \frac{1}{6\sqrt{3}} 6 3 1 -Lipschitz and,
∣ ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ ≤ γ t 6 3 . \left|\phi''(\nu\alpha_t Z_1+\sigma_t Z_2)-\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\right| \leq\frac{\gamma_t}{6\sqrt{3}}. ∣ ϕ ′′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ∣ ≤ 6 3 γ t .
Moreover, 2 E [ 1 − π p ( ν Z 1 ) ] = 1 2\mathbb{E}[1-\pi_p(\nu Z_1)]=1 2 E [ 1 − π p ( ν Z 1 )] = 1 , and hence
α t E [ 2 ( 1 − π p ( ν Z 1 ) ) ∣ ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ ] ≤ α t γ t 6 3 . \alpha_t\mathbb{E}\left[2(1-\pi_p(\nu Z_1))\left|\phi''(\nu\alpha_t Z_1+\sigma_t Z_2)-\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\right|\right] \leq\frac{\alpha_t\gamma_t}{6\sqrt{3}}. α t E [ 2 ( 1 − π p ( ν Z 1 )) ∣ ϕ ′′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ∣ ] ≤ 6 3 α t γ t .
For the second term, since 0 < ϕ ′ ′ ( u ) ≤ 1 4 0<\phi''(u)\leq\frac{1}{4} 0 < ϕ ′′ ( u ) ≤ 4 1 , we have
ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ 1 − 1 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ = γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≤ γ t 16 . \phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\left|1-\frac{1}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}\right| = \frac{\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))^2}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}\leq\frac{\gamma_t}{16}. ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 1 − 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 1 = 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ≤ 16 γ t .
Therefore
α t E [ 2 ( 1 − π p ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ 1 − 1 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ ] ≤ α t γ t 16 . \alpha_t\mathbb{E}\left[2(1-\pi_p(\nu Z_1))\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\left|1-\frac{1}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}\right|\right]\leq\frac{\alpha_t\gamma_t}{16}. α t E [ 2 ( 1 − π p ( ν Z 1 )) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 1 − 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 1 ] ≤ 16 α t γ t .
Finally, since sup u ∈ R ϕ ′ ′ ( u ) = 1 4 \sup_{u\in\mathbb{R}}\phi''(u)=\frac{1}{4} sup u ∈ R ϕ ′′ ( u ) = 4 1 , which implies that ϕ ′ ( u ) \phi'(u) ϕ ′ ( u ) is 1 4 \frac{1}{4} 4 1 -Lipschitz, we have,
2 E [ π p ′ ( ν Z 1 ) ∣ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) − ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ∣ ] ≤ 1 2 E [ π p ′ ( ν Z 1 ) ∣ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) − ( ν α t Z 1 + σ t Z 2 ) ∣ ] ≤ γ t 2 E [ π p ′ ( ν Z 1 ) ] = γ t 2 ( 1 − 2 ρ ) h ( ν ) . 2\mathbb{E}\left[\pi'_p(\nu Z_1)\left|\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))-\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)\right|\right]
\leq\frac{1}{2}\mathbb{E}\left[\pi'_p(\nu Z_1)\left|\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)-(\nu\alpha_t Z_1+\sigma_t Z_2)\right|\right]\leq\frac{\gamma_t}{2}\mathbb{E}[\pi'_p(\nu Z_1)]=\frac{\gamma_t}{2}(1-2\rho)h(\nu). 2 E [ π p ′ ( ν Z 1 ) ∣ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) − ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ∣ ] ≤ 2 1 E [ π p ′ ( ν Z 1 ) ∣ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) − ( ν α t Z 1 + σ t Z 2 ) ∣ ] ≤ 2 γ t E [ π p ′ ( ν Z 1 )] = 2 γ t ( 1 − 2 ρ ) h ( ν ) .
Consequently,
∣ α t h ( ν ) ( ν 2 α t 2 + σ t 2 ) − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ α t γ t ( 1 6 3 + 1 16 ) + ∣ m t − θ t ∣ + γ t 2 ( 1 − 2 ρ ) h ( ν ) . \left|\alpha_t h(\nu)\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)-(1-2\rho)h(\nu)\right|\leq\alpha_t\gamma_t\left(\frac{1}{6\sqrt{3}}+\frac{1}{16}\right)+|m_t-\theta_t|+\frac{\gamma_t}{2}(1-2\rho)h(\nu). α t h ( ν ) ( ν 2 α t 2 + σ t 2 ) − ( 1 − 2 ρ ) h ( ν ) ≤ α t γ t ( 6 3 1 + 16 1 ) + ∣ m t − θ t ∣ + 2 γ t ( 1 − 2 ρ ) h ( ν ) .
Assume s s s is sufficiently small so that
3 s δ ≤ min { 1 , 16 33 Q , ( 1 − 2 ρ ) 2 ν 2 h ( ν ) 2 16 Q 2 } . \frac{3s}{\delta}\leq\min\left\{1,\frac{16}{33Q},\frac{(1-2\rho)^2\nu^2h(\nu)^2}{16Q^2}\right\}. δ 3 s ≤ min { 1 , 33 Q 16 , 16 Q 2 ( 1 − 2 ρ ) 2 ν 2 h ( ν ) 2 } .
Since γ t ≤ 33 Q 32 s δ \gamma_t\leq\frac{33Q}{32}\frac{s}{\delta} γ t ≤ 32 33 Q δ s , we have γ t ≤ 1 2 \gamma_t\leq\frac{1}{2} γ t ≤ 2 1 . Therefore
∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ 2 E [ π p ′ ( ν Z 1 ) ∣ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) − ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ∣ ] ≤ γ t 2 ( 1 − 2 ρ ) h ( ν ) ≤ 1 4 ( 1 − 2 ρ ) h ( ν ) . (58) |\theta_t-(1-2\rho)h(\nu)|\leq2\mathbb{E}\left[\pi'_p(\nu Z_1)\left|\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))-\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)\right|\right]
\leq\frac{\gamma_t}{2}(1-2\rho)h(\nu)\leq\frac{1}{4}(1-2\rho)h(\nu). \tag*{(58)} ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ 2 E [ π p ′ ( ν Z 1 ) ∣ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) − ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ∣ ] ≤ 2 γ t ( 1 − 2 ρ ) h ( ν ) ≤ 4 1 ( 1 − 2 ρ ) h ( ν ) . (58)
Moreover, since α t − σ t τ t θ t = σ t τ t ( m t − θ t ) \alpha_t-\sigma_t\tau_t\theta_t=\sigma_t\tau_t(m_t-\theta_t) α t − σ t τ t θ t = σ t τ t ( m t − θ t ) , and σ t τ t ≥ 1 \sigma_t\tau_t\geq1 σ t τ t ≥ 1 , the previously established bound ∣ α t − σ t τ t θ t ∣ ≤ Q 3 s ν 2 δ |\alpha_t-\sigma_t\tau_t\theta_t|\leq Q\sqrt{\frac{3s}{\nu^2\delta}} ∣ α t − σ t τ t θ t ∣ ≤ Q ν 2 δ 3 s implies ∣ m t − θ t ∣ ≤ Q 3 s ν 2 δ ≤ 1 4 ( 1 − 2 ρ ) h ( ν ) |m_t-\theta_t|\leq Q\sqrt{\frac{3s}{\nu^2\delta}}\leq\frac{1}{4}(1-2\rho)h(\nu) ∣ m t − θ t ∣ ≤ Q ν 2 δ 3 s ≤ 4 1 ( 1 − 2 ρ ) h ( ν ) . Consequently, 1 2 ( 1 − 2 ρ ) h ( ν ) ≤ m t ≤ 3 2 ( 1 − 2 ρ ) h ( ν ) \frac{1}{2}(1-2\rho)h(\nu)\leq m_t\leq\frac{3}{2}(1-2\rho)h(\nu) 2 1 ( 1 − 2 ρ ) h ( ν ) ≤ m t ≤ 2 3 ( 1 − 2 ρ ) h ( ν ) . Since 1 ≤ σ t τ t ≤ Q 1\leq\sigma_t\tau_t\leq Q 1 ≤ σ t τ t ≤ Q and α t = σ t τ t m t \alpha_t=\sigma_t\tau_t m_t α t = σ t τ t m t , we obtain 1 2 ( 1 − 2 ρ ) h ( ν ) ≤ α t ≤ 3 2 Q ( 1 − 2 ρ ) h ( ν ) \frac{1}{2}(1-2\rho)h(\nu)\leq\alpha_t\leq\frac{3}{2}Q(1-2\rho)h(\nu) 2 1 ( 1 − 2 ρ ) h ( ν ) ≤ α t ≤ 2 3 Q ( 1 − 2 ρ ) h ( ν ) . Using the bound obtained above,
∣ α t h ( ν ) ( ν 2 α t 2 + σ t 2 ) − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ α t γ t ( 1 6 3 + 1 16 ) + ∣ m t − θ t ∣ + γ t 2 ( 1 − 2 ρ ) h ( ν ) ≤ 3 Q 2 s ν 2 δ + [ 99 Q 2 ( 1 − 2 ρ ) h ( ν ) 64 ( 1 6 3 + 1 16 ) + 33 Q ( 1 − 2 ρ ) h ( ν ) 64 ] 3 s δ . \begin{aligned}
\left|\alpha_t h(\nu)\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)-(1-2\rho)h(\nu)\right| \\
&\leq\alpha_t\gamma_t\left(\frac{1}{6\sqrt{3}}+\frac{1}{16}\right)+|m_t-\theta_t|+\frac{\gamma_t}{2}(1-2\rho)h(\nu) \\
&\leq\sqrt{\frac{3Q^2s}{\nu^2\delta}}+\left[\frac{99Q^2(1-2\rho)h(\nu)}{64}\left(\frac{1}{6\sqrt{3}}+\frac{1}{16}\right)+\frac{33Q(1-2\rho)h(\nu)}{64}\right]\frac{3s}{\delta}.
\end{aligned} α t h ( ν ) ( ν 2 α t 2 + σ t 2 ) − ( 1 − 2 ρ ) h ( ν ) ≤ α t γ t ( 6 3 1 + 16 1 ) + ∣ m t − θ t ∣ + 2 γ t ( 1 − 2 ρ ) h ( ν ) ≤ ν 2 δ 3 Q 2 s + [ 64 99 Q 2 ( 1 − 2 ρ ) h ( ν ) ( 6 3 1 + 16 1 ) + 64 33 Q ( 1 − 2 ρ ) h ( ν ) ] δ 3 s .
Furthermore,
ν 2 α t 2 + σ t 2 − ν α t = σ t 2 ν 2 α t 2 + σ t 2 + ν α t ≤ σ t 2 ν ( 1 − 2 ρ ) h ( ν ) ≤ Q 2 ν ( 1 − 2 ρ ) h ( ν ) 3 s δ . \sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}-\nu\alpha_{t}
=\frac{\sigma_{t}^{2}}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}+\nu\alpha_{t}}
\leq\frac{\sigma_{t}^{2}}{\nu(1-2\rho)h(\nu)}
\leq\frac{Q^{2}}{\nu(1-2\rho)h(\nu)}\frac{3s}{\delta}. ν 2 α t 2 + σ t 2 − ν α t = ν 2 α t 2 + σ t 2 + ν α t σ t 2 ≤ ν ( 1 − 2 ρ ) h ( ν ) σ t 2 ≤ ν ( 1 − 2 ρ ) h ( ν ) Q 2 δ 3 s .
Here, [4] follows from
ν 2 α t 2 + σ t 2 + ν α t ≥ 2 ν α t ≥ ν ( 1 − 2 ρ ) h ( ν ) , \sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}+\nu\alpha_{t}
\geq2\nu\alpha_{t}
\geq\nu(1-2\rho)h(\nu), ν 2 α t 2 + σ t 2 + ν α t ≥ 2 ν α t ≥ ν ( 1 − 2 ρ ) h ( ν ) ,
and [5] follows from σ t 2 ≤ Q 2 3 s δ \sigma_{t}^{2}\leq Q^{2}\frac{3s}{\delta} σ t 2 ≤ Q 2 δ 3 s . Since h ′ ( a ) = E [ Z ϕ ′ ′ ′ ( a Z ) ] h'(a)=\mathbb{E}[Z\phi'''(aZ)] h ′ ( a ) = E [ Z ϕ ′′′ ( a Z )] and sup u ∈ R ∣ ϕ ′ ′ ′ ( u ) ∣ = 1 6 3 \sup_{u\in\mathbb{R}}|\phi'''(u)|=\frac{1}{6\sqrt{3}} sup u ∈ R ∣ ϕ ′′′ ( u ) ∣ = 6 3 1 , we have ∣ h ′ ( a ) ∣ ≤ 2 π 1 6 3 |h'(a)|\leq\sqrt{\frac{2}{\pi}}\frac{1}{6\sqrt{3}} ∣ h ′ ( a ) ∣ ≤ π 2 6 3 1 .
Hence,
α t ∣ h ( ν α t ) − h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ α t 2 π 1 6 3 ( ν 2 α t 2 + σ t 2 − ν α t ) ≤ 3 2 Q ( 1 − 2 ρ ) h ( ν ) 2 π 1 6 3 Q 2 ν ( 1 − 2 ρ ) h ( ν ) 3 s δ = Q 3 4 ν 2 3 π 3 s δ . \begin{aligned}
\alpha_{t}\left|h(\nu\alpha_{t})-h\left(\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}\right)\right|
\leq\alpha_{t}\sqrt{\frac{2}{\pi}}\frac{1}{6\sqrt{3}}\left(\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}-\nu\alpha_{t}\right) \\
&\leq\frac{3}{2}Q(1-2\rho)h(\nu)\sqrt{\frac{2}{\pi}}\frac{1}{6\sqrt{3}}\frac{Q^{2}}{\nu(1-2\rho)h(\nu)}\frac{3s}{\delta}
=\frac{Q^{3}}{4\nu}\sqrt{\frac{2}{3\pi}}\frac{3s}{\delta}.
\end{aligned} α t h ( ν α t ) − h ( ν 2 α t 2 + σ t 2 ) ≤ α t π 2 6 3 1 ( ν 2 α t 2 + σ t 2 − ν α t ) ≤ 2 3 Q ( 1 − 2 ρ ) h ( ν ) π 2 6 3 1 ν ( 1 − 2 ρ ) h ( ν ) Q 2 δ 3 s = 4 ν Q 3 3 π 2 δ 3 s .
Combining the last two estimates,
∣ α t h ( ν α t ) − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ 3 s Q 2 δ ν 2 + [ 99 Q 2 ( 1 − 2 ρ ) h ( ν ) 64 ( 1 6 3 + 1 16 ) + 33 Q ( 1 − 2 ρ ) h ( ν ) 64 + Q 3 4 ν 2 3 π ] 3 s δ . \left|\alpha_{t}h(\nu\alpha_{t})-(1-2\rho)h(\nu)\right|
\leq\sqrt{\frac{3sQ^{2}}{\delta\nu^{2}}}
+\left[\frac{99Q^{2}(1-2\rho)h(\nu)}{64}\left(\frac{1}{6\sqrt{3}}+\frac{1}{16}\right)+\frac{33Q(1-2\rho)h(\nu)}{64}+\frac{Q^{3}}{4\nu}\sqrt{\frac{2}{3\pi}}\right]\frac{3s}{\delta}. ∣ α t h ( ν α t ) − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ δ ν 2 3 s Q 2 + [ 64 99 Q 2 ( 1 − 2 ρ ) h ( ν ) ( 6 3 1 + 16 1 ) + 64 33 Q ( 1 − 2 ρ ) h ( ν ) + 4 ν Q 3 3 π 2 ] δ 3 s .
We aim to use the mean-value theorem to bound the quantity ∣ α t − r ∣ |\alpha_{t}-r| ∣ α t − r ∣ , we need to find in which region α t \alpha_{t} α t is located. If we decrease s 0 s_{0} s 0 so that Q 3 s δ ≤ C 1 s δ Q^{3}\frac{s}{\delta}\leq C_{1}\sqrt{\frac{s}{\delta}} Q 3 δ s ≤ C 1 δ s for all s ≤ s 0 s\leq s_{0} s ≤ s 0 , then, for some C 2 > 0 C_{2}>0 C 2 > 0 , α t h ( ν α t ) ≤ ( 1 − 2 ρ ) h ( ν ) + C 2 ν s δ \alpha_{t}h(\nu\alpha_{t})\leq(1-2\rho)h(\nu)+\frac{C_{2}}{\nu}\sqrt{\frac{s}{\delta}} α t h ( ν α t ) ≤ ( 1 − 2 ρ ) h ( ν ) + ν C 2 δ s , if also s 0 s_{0} s 0 is small so that C 2 ν s δ ≤ 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) 2 \frac{C_{2}}{\nu}\sqrt{\frac{s}{\delta}}\leq\frac{\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)}{2} ν C 2 δ s ≤ 2 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) then ∣ α t h ( ν α t ) − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) 2 ν |\alpha_{t}h(\nu\alpha_{t})-(1-2\rho)h(\nu)|\leq\frac{\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)}{2\nu} ∣ α t h ( ν α t ) − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ 2 ν 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) , also,
1 ν 2 π − α t h ( ν α t ) = 1 ν 2 π ∫ R ϕ ′ ′ ( u ) ( 1 − e − u 2 2 ν 2 α t 2 ) d u ≤ 1 ν 2 π 1 2 ν 2 α t 2 ∫ R u 2 ϕ ′ ′ ( u ) d u = π 2 6 2 π ν 3 α t 2 , \begin{aligned}
\frac{1}{\nu\sqrt{2\pi}}-\alpha_{t}h(\nu\alpha_{t})
=\frac{1}{\nu\sqrt{2\pi}}\int_{\mathbb{R}}\phi''(u)\left(1-e^{-\frac{u^{2}}{2\nu^{2}\alpha_{t}^{2}}}\right)\,du \\
&\leq\frac{1}{\nu\sqrt{2\pi}}\frac{1}{2\nu^{2}\alpha_{t}^{2}}\int_{\mathbb{R}}u^{2}\phi''(u)\,du
=\frac{\pi^{2}}{6\sqrt{2\pi}\nu^{3}\alpha_{t}^{2}},
\end{aligned} ν 2 π 1 − α t h ( ν α t ) = ν 2 π 1 ∫ R ϕ ′′ ( u ) ( 1 − e − 2 ν 2 α t 2 u 2 ) d u ≤ ν 2 π 1 2 ν 2 α t 2 1 ∫ R u 2 ϕ ′′ ( u ) d u = 6 2 π ν 3 α t 2 π 2 ,
1 2 π − ν ( 1 − 2 ρ ) h ( ν ) 2 ν ≤ π 2 6 2 π ν 3 α t 2 \frac{\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)}{2\nu}
\leq\frac{\pi^{2}}{6\sqrt{2\pi}\nu^{3}\alpha_{t}^{2}} 2 ν 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ≤ 6 2 π ν 3 α t 2 π 2
implies
α t ≤ 2 ν π 2 6 2 π ν 3 ( 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ) \alpha_{t}\leq\sqrt{\frac{2\nu\pi^{2}}{6\sqrt{2\pi}\nu^{3}\left(\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)\right)}} α t ≤ 6 2 π ν 3 ( 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ) 2 ν π 2
as
1 2 π ( 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ) ≤ 2 2 ρ + 2 ( 1 − 2 ρ ) ( 1 + e ) ( 1 + 2 ν 2 ) . \frac{1}{\sqrt{2\pi}\left(\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)\right)}
\leq\frac{2}{2\rho+\frac{2(1-2\rho)}{(1+e)(1+2\nu^{2})}}. 2 π ( 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ) 1 ≤ 2 ρ + ( 1 + e ) ( 1 + 2 ν 2 ) 2 ( 1 − 2 ρ ) 2 .
As function of ρ \rho ρ , 2 ρ + 2 ( 1 − 2 ρ ) ( 1 + e ) ( 1 + 2 ν 2 ) 2\rho+\frac{2(1-2\rho)}{(1+e)(1+2\nu^{2})} 2 ρ + ( 1 + e ) ( 1 + 2 ν 2 ) 2 ( 1 − 2 ρ ) is increasing indeed taking the derivative with respect to ρ \rho ρ yields that,
2 − 4 ( 1 + e ) ( 1 + 2 ν 2 ) > 0 , 2-\frac{4}{(1+e)(1+2\nu^{2})}>0, 2 − ( 1 + e ) ( 1 + 2 ν 2 ) 4 > 0 ,
then
1 2 π ( 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ) ≤ ( 1 + e ) ( 1 + 2 ν 2 ) 2 \frac{1}{\sqrt{2\pi}\left(\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)\right)}
\leq\frac{(1+e)(1+2\nu^{2})}{2} 2 π ( 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ) 1 ≤ 2 ( 1 + e ) ( 1 + 2 ν 2 )
so
α t ≤ ν π 2 6 ν 3 ( 1 + e ) ( 1 + 2 ν 2 ) , \alpha_{t}\leq\sqrt{\frac{\nu\pi^{2}}{6\nu^{3}}(1+e)(1+2\nu^{2})}, α t ≤ 6 ν 3 ν π 2 ( 1 + e ) ( 1 + 2 ν 2 ) ,
and since ν r ≤ ν \nu r\leq\nu ν r ≤ ν , thus every a a a between α t \alpha_{t} α t and r r r satisfies
ν a < π 2 6 ( 1 + e ) ( 1 + 2 ν 2 ) . \nu a<\sqrt{\frac{\pi^{2}}{6}(1+e)(1+2\nu^{2})}. ν a < 6 π 2 ( 1 + e ) ( 1 + 2 ν 2 ) .
Now d d a ( a h ( ν a ) ) = E [ Z 2 ϕ ′ ′ ( ν a Z ) ] \frac{d}{da}(ah(\nu a))=\mathbb{E}[Z^{2}\phi''(\nu aZ)] d a d ( ah ( ν a )) = E [ Z 2 ϕ ′′ ( ν a Z )] , therefore we have for all 0 < b ≤ min { 1 , 1 ν a } 0<b\leq\min\left\{1,\frac{1}{\nu a}\right\} 0 < b ≤ min { 1 , ν a 1 } ,
E [ Z 2 ϕ ′ ′ ( ν a Z ) ] ≥ [ 6 ] E [ Z 2 1 { ∣ Z ∣ ≤ b } ] ϕ ′ ′ ( 1 ) ≥ [ 7 ] 2 φ ( 1 ) 3 b 3 ϕ ′ ′ ( 1 ) . (59) \mathbb{E}[Z^{2}\phi''(\nu aZ)]
\mathrel{\geq}^{[6]}\mathbb{E}\left[Z^{2}\mathbf{1}_{\{|Z|\leq b\}}\right]\phi''(1)
\mathrel{\geq}^{[7]}\frac{2\varphi(1)}{3}b^{3}\phi''(1). \tag*{(59)} E [ Z 2 ϕ ′′ ( ν a Z )] ≥ [ 6 ] E [ Z 2 1 { ∣ Z ∣ ≤ b } ] ϕ ′′ ( 1 ) ≥ [ 7 ] 3 2 φ ( 1 ) b 3 ϕ ′′ ( 1 ) . (59)
Here, [6] follows from ϕ ′ ′ ( u ) = 1 4 cosh 2 ( u 2 ) \phi''(u)=\frac{1}{4\cosh^{2}\left(\frac{u}{2}\right)} ϕ ′′ ( u ) = 4 c o s h 2 ( 2 u ) 1 (which is ascending in R − \mathbb{R}^{-} R − and descending in R + \mathbb{R}^{+} R + ) therefore on { ∣ Z ∣ ≤ b } \{|Z|\leq b\} { ∣ Z ∣ ≤ b } we have ϕ ′ ′ ( ν a Z ) > ϕ ′ ′ ( 1 ) \phi''(\nu aZ)>\phi''(1) ϕ ′′ ( ν a Z ) > ϕ ′′ ( 1 ) , since ∣ ν a Z ∣ ≤ 1 |\nu aZ|\leq1 ∣ ν a Z ∣ ≤ 1 . Also [7] follows since for all a ≤ 1 a\leq1 a ≤ 1 ,
E [ Z 2 1 { ∣ Z ∣ ≤ a } ] = 2 ∫ 0 a z 2 φ ( z ) d z ≥ 2 φ ( 1 ) 3 a 3 . \mathbb{E}\left[Z^{2}\mathbf{1}_{\{|Z|\leq a\}}\right]
=2\int_{0}^{a}z^{2}\varphi(z)\,dz
\geq\frac{2\varphi(1)}{3}a^{3}. E [ Z 2 1 { ∣ Z ∣ ≤ a } ] = 2 ∫ 0 a z 2 φ ( z ) d z ≥ 3 2 φ ( 1 ) a 3 .
Since r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) rh(\nu r)=(1-2\rho)h(\nu) r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) , with b = min { 1 , 1 max { π 2 6 ( 1 + e ) ( 1 + 2 ν 2 ) , ν r } } b=\min\left\{1,\frac{1}{\max\left\{\sqrt{\frac{\pi^{2}}{6}(1+e)(1+2\nu^{2})},\nu r\right\}}\right\} b = min ⎩ ⎨ ⎧ 1 , m a x { 6 π 2 ( 1 + e ) ( 1 + 2 ν 2 ) , ν r } 1 ⎭ ⎬ ⎫ , the mean-value theorem gives
∣ α t − r ∣ ≤ 3 2 φ ( 1 ) ϕ ′ ′ ( 1 ) max { 1 , ( π 2 6 ( 1 + e ) ( 1 + 2 ν 2 ) ) 3 } [ 3 s Q 2 δ ν 2 + [ 99 Q 2 ( 1 − 2 ρ ) h ( ν ) 64 ( 1 6 3 + 1 16 ) + 33 Q ( 1 − 2 ρ ) h ( ν ) 64 + Q 3 4 ν 2 3 π ] 3 s δ |\alpha_{t}-r|
\leq\frac{3}{2\varphi(1)\phi''(1)}
\max\{1,(\sqrt{\frac{\pi^{2}}{6}(1+e)(1+2\nu^{2})})^{3}\}
[
\sqrt{\frac{3sQ^{2}}{\delta\nu^{2}}}
+[\frac{99Q^{2}(1-2\rho)h(\nu)}{64}(\frac{1}{6\sqrt{3}}+\frac{1}{16})+\frac{33Q(1-2\rho)h(\nu)}{64}+\frac{Q^{3}}{4\nu}\sqrt{\frac{2}{3\pi}}]\frac{3s}{\delta} ∣ α t − r ∣ ≤ 2 φ ( 1 ) ϕ ′′ ( 1 ) 3 max { 1 , ( 6 π 2 ( 1 + e ) ( 1 + 2 ν 2 ) ) 3 } [ δ ν 2 3 s Q 2 + [ 64 99 Q 2 ( 1 − 2 ρ ) h ( ν ) ( 6 3 1 + 16 1 ) + 64 33 Q ( 1 − 2 ρ ) h ( ν ) + 4 ν Q 3 3 π 2 ] δ 3 s
Finally since by Eq. (40) ,
E [ ( w t − r w ⋆ ) 2 ] = ν 2 ( α t − r ) 2 + σ t 2 , \mathbb{E}[(w_t-rw^\star)^2]=\nu^2(\alpha_t-r)^2+\sigma_t^2, E [( w t − r w ⋆ ) 2 ] = ν 2 ( α t − r ) 2 + σ t 2 ,
and therefore using Proposition 22 ,
L ( w T ) − inf w L ( w ) ≤ 3 Q 2 8 s δ + ν 2 8 [ 3 2 φ ( 1 ) ϕ ′ ′ ( 1 ) ] 2 max { 1 , ( π 2 6 ( 1 + e ) ( 1 + 2 ν 2 ) ) 6 } \mathcal{L}(\mathbf{w}_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\leq\frac{3Q^2}{8}\frac{s}{\delta}+\frac{\nu^2}{8}\left[\frac{3}{2\varphi(1)\phi''(1)}\right]^2\max\left\{1,\left(\sqrt{\frac{\pi^2}{6}(1+e)(1+2\nu^2)}\right)^6\right\} L ( w T ) − w inf L ( w ) ≤ 8 3 Q 2 δ s + 8 ν 2 [ 2 φ ( 1 ) ϕ ′′ ( 1 ) 3 ] 2 max ⎩ ⎨ ⎧ 1 , ( 6 π 2 ( 1 + e ) ( 1 + 2 ν 2 ) ) 6 ⎭ ⎬ ⎫
[ 3 s Q 2 δ ν 2 + [ 99 Q 2 ( 1 − 2 ρ ) h ( ν ) 64 ( 1 6 3 + 1 16 ) + 33 Q ( 1 − 2 ρ ) h ( ν ) 64 + Q 3 4 ν 2 3 π 3 s δ ] 2 ] . \left[\sqrt{\frac{3sQ^2}{\delta\nu^2}}+\left[\frac{99Q^2(1-2\rho)h(\nu)}{64}\left(\frac{1}{6\sqrt{3}}+\frac{1}{16}\right)+\frac{33Q(1-2\rho)h(\nu)}{64}+\frac{Q^3}{4\nu}\sqrt{\frac{2}{3\pi}}\frac{3s}{\delta}\right]^2\right]. δ ν 2 3 s Q 2 + [ 64 99 Q 2 ( 1 − 2 ρ ) h ( ν ) ( 6 3 1 + 16 1 ) + 64 33 Q ( 1 − 2 ρ ) h ( ν ) + 4 ν Q 3 3 π 2 δ 3 s ] 2 .
with Q = 128 2 π e 1 2 ( 1 + ν ) ( 2 e − 1 2 + ν ( 1 − 2 ρ ) ) log 2 2 ρ + ( 1 + ν ) ( 1 + 2 ν 2 ) ( 1 + e ) ( 1 + 2 ν 2 ) Q=\frac{128\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu)(2e^{-\frac{1}{2}}+\nu(1-2\rho))\log2}{2\rho+\frac{(1+\nu)(1+2\nu^2)}{(1+e)(1+2\nu^2)}} Q = 2 ρ + ( 1 + e ) ( 1 + 2 ν 2 ) ( 1 + ν ) ( 1 + 2 ν 2 ) 128 2 π e 2 1 ( 1 + ν ) ( 2 e − 2 1 + ν ( 1 − 2 ρ )) l o g 2 . Note that, ( 1 − 2 ρ ) h ( ν ) ≤ 1 4 (1-2\rho)h(\nu)\leq\frac{1}{4} ( 1 − 2 ρ ) h ( ν ) ≤ 4 1 and ν ( 1 − 2 ρ ) h ( ν ) ≤ 1 2 π \nu(1-2\rho)h(\nu)\leq\frac{1}{\sqrt{2\pi}} ν ( 1 − 2 ρ ) h ( ν ) ≤ 2 π 1 since r ≤ 1 r\leq1 r ≤ 1 , overall there exists a universal constant C > 0 C>0 C > 0 such that
L ( w T ) − inf w L ( w ) ≤ C ( 1 + ν ) 8 s δ \mathcal{L}(\mathbf{w}_T)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\leq C(1+\nu)^8\frac{s}{\delta} L ( w T ) − w inf L ( w ) ≤ C ( 1 + ν ) 8 δ s
Below, we summarize all conditions used to make s s s sufficiently small.
For some C 1 , C 2 , C 3 , C 4 , C 5 , C 6 > 0 C_1,C_2,C_3,C_4,C_5,C_6>0 C 1 , C 2 , C 3 , C 4 , C 5 , C 6 > 0 ,
33 s 32 δ ≤ C 1 1 + ν , \frac{33s}{32\delta}\leq\frac{C_1}{1+\nu}, 32 δ 33 s ≤ 1 + ν C 1 ,
s δ ≤ C 2 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ( 1 + ν ) 2 , \frac{s}{\delta}\leq C_2\frac{\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)}{(1+\nu)^2}, δ s ≤ C 2 ( 1 + ν ) 2 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ,
s δ ( 1 + 2 log 64 s ) ≤ C 3 ( 1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ) 2 , \frac{s}{\delta}\left(1+2\log\frac{64}{s}\right)\leq C_3\left(\frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)\right)^2, δ s ( 1 + 2 log s 64 ) ≤ C 3 ( 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ) 2 ,
s δ log 64 s ( 1 + 2 log 64 s ) 2 ≤ C 4 ( 1 − 2 ρ ) 2 h ( ν ) 2 ν 2 , \frac{s}{\delta}\log\frac{64}{s}\left(1+2\log\frac{64}{s}\right)^2\leq C_4(1-2\rho)^2h(\nu)^2\nu^2, δ s log s 64 ( 1 + 2 log s 64 ) 2 ≤ C 4 ( 1 − 2 ρ ) 2 h ( ν ) 2 ν 2 ,
s δ ≤ C 5 Q 6 . (60) \frac{s}{\delta}\leq\frac{C_5}{Q^6}. \tag*{(60)} δ s ≤ Q 6 C 5 . (60)
Using
1 2 π − ν ( 1 − 2 ρ ) h ( ν ) ≥ 1 ( 1 + ν ) 2 , \frac{1}{\sqrt{2\pi}}-\nu(1-2\rho)h(\nu)\geq\frac{1}{(1+\nu)^2}, 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) ≥ ( 1 + ν ) 2 1 ,
h ( ν ) = E [ ϕ ′ ′ ( ν Z ) ] ≥ ϕ ′ ′ ( 1 ) P ( ∣ Z ∣ ≤ min { 1 , 1 ν } ) ≥ 2 ϕ ′ ′ ( 1 ) φ ( 1 ) min { 1 , 1 ν } ≥ 2 ϕ ′ ′ ( 1 ) φ ( 1 ) 1 + ν , (61) h(\nu)=\mathbb{E}[\phi''(\nu Z)]\geq\phi''(1)\mathbb{P}\left(|Z|\leq\min\left\{1,\frac{1}{\nu}\right\}\right)\geq2\phi''(1)\varphi(1)\min\left\{1,\frac{1}{\nu}\right\}\geq\frac{2\phi''(1)\varphi(1)}{1+\nu}, \tag*{(61)} h ( ν ) = E [ ϕ ′′ ( ν Z )] ≥ ϕ ′′ ( 1 ) P ( ∣ Z ∣ ≤ min { 1 , ν 1 } ) ≥ 2 ϕ ′′ ( 1 ) φ ( 1 ) min { 1 , ν 1 } ≥ 1 + ν 2 ϕ ′′ ( 1 ) φ ( 1 ) , (61)
and, since ρ ≤ 1 2 \rho\leq\frac{1}{2} ρ ≤ 2 1 , Q ≲ ( 1 + ν ) 4 Q\lesssim(1+\nu)^4 Q ≲ ( 1 + ν ) 4 , the conditions in Eq. (60) are satisfied provided
s 0 ≤ c min { δ ( 1 + ν ) 4 log + ( 1 + ν δ ) , δ ( 1 − 2 ρ ) 2 ν 2 ( 1 + ν ) 2 log + 3 ( ( 1 + ν ) 2 δ ( 1 − 2 ρ ) 2 ν 2 ) , δ ( 1 + ν ) 24 } , s_0\leq c\min\left\{\frac{\delta}{(1+\nu)^4\log_+\left(\frac{1+\nu}{\delta}\right)},\frac{\delta(1-2\rho)^2\nu^2}{(1+\nu)^2\log_+^3\left(\frac{(1+\nu)^2}{\delta(1-2\rho)^2\nu^2}\right)},\frac{\delta}{(1+\nu)^{24}}\right\}, s 0 ≤ c min ⎩ ⎨ ⎧ ( 1 + ν ) 4 log + ( δ 1 + ν ) δ , ( 1 + ν ) 2 log + 3 ( δ ( 1 − 2 ρ ) 2 ν 2 ( 1 + ν ) 2 ) δ ( 1 − 2 ρ ) 2 ν 2 , ( 1 + ν ) 24 δ ⎭ ⎬ ⎫ ,
for some universal constant $c>0.
Lemma 23. For all x ≥ 0 x \ge0 x ≥ 0 , we have
Φ ( − x ) φ ( x ) ≥ 1 x + 1 \frac{\Phi(-x)}{\varphi(x)} \ge\frac{1}{x+1} φ ( x ) Φ ( − x ) ≥ x + 1 1 Proof. Define F ( x ) = Φ ( − x ) − φ ( x ) x + 1 F(x)=\Phi(-x)-\frac{\varphi(x)}{x+1} F ( x ) = Φ ( − x ) − x + 1 φ ( x ) . First, we evaluate the function at the boundary x = 0 x=0 x = 0 :
F ( 0 ) = 1 2 − 1 2 π > 0 F(0)=\frac{1}{2}-\frac{1}{\sqrt{2\pi}}>0 F ( 0 ) = 2 1 − 2 π 1 > 0 Next, we compute the derivative F ′ ( x ) F'(x) F ′ ( x ) using the quotient rule. Recall that Φ ′ ( − x ) = − φ ( x ) \Phi'(-x)=-\varphi(x) Φ ′ ( − x ) = − φ ( x ) and φ ′ ( x ) = − x φ ( x ) \varphi'(x)=-x\varphi(x) φ ′ ( x ) = − x φ ( x ) :
F ′ ( x ) = − φ ( x ) − φ ′ ( x ) ( x + 1 ) − φ ( x ) ( x + 1 ) 2 F'(x)=-\varphi(x)-\frac{\varphi'(x)(x+1)-\varphi(x)}{(x+1)^2} F ′ ( x ) = − φ ( x ) − ( x + 1 ) 2 φ ′ ( x ) ( x + 1 ) − φ ( x ) Substituting φ ′ ( x ) = − x φ ( x ) \varphi'(x)=-x\varphi(x) φ ′ ( x ) = − x φ ( x ) into the numerator:
F ′ ( x ) = − φ ( x ) − − x φ ( x ) ( x + 1 ) − φ ( x ) ( x + 1 ) 2 = − φ ( x ) + φ ( x ) [ x ( x + 1 ) + 1 ] ( x + 1 ) 2 = φ ( x ) [ − 1 + x 2 + x + 1 ( x + 1 ) 2 ] = φ ( x ) [ − ( x 2 + 2 x + 1 ) + ( x 2 + x + 1 ) ( x + 1 ) 2 ] = − x ( x + 1 ) 2 φ ( x ) . \begin{aligned}
F'(x)&=-\varphi(x)-\frac{-x\varphi(x)(x+1)-\varphi(x)}{(x+1)^2}
=-\varphi(x)+\frac{\varphi(x)[x(x+1)+1]}{(x+1)^2}\\
&=\varphi(x)\left[-1+\frac{x^2+x+1}{(x+1)^2}\right]
=\varphi(x)\left[\frac{-(x^2+2x+1)+(x^2+x+1)}{(x+1)^2}\right]\\
&=-\frac{x}{(x+1)^2}\varphi(x).
\end{aligned} F ′ ( x ) = − φ ( x ) − ( x + 1 ) 2 − x φ ( x ) ( x + 1 ) − φ ( x ) = − φ ( x ) + ( x + 1 ) 2 φ ( x ) [ x ( x + 1 ) + 1 ] = φ ( x ) [ − 1 + ( x + 1 ) 2 x 2 + x + 1 ] = φ ( x ) [ ( x + 1 ) 2 − ( x 2 + 2 x + 1 ) + ( x 2 + x + 1 ) ] = − ( x + 1 ) 2 x φ ( x ) . Therefore, F ′ ( x ) ≤ 0 F'(x)\le0 F ′ ( x ) ≤ 0 for all x > 0 x>0 x > 0 , meaning F ( x ) F(x) F ( x ) is monotonically decreasing on this interval. Finally, we examine the asymptotic behavior as x → ∞ x\to\infty x → ∞ :
lim x → ∞ F ( x ) = lim x → ∞ Φ ( − x ) − lim x → ∞ φ ( x ) x + 1 = 0 − 0 = 0 \lim_{x\to\infty}F(x)=\lim_{x\to\infty}\Phi(-x)-\lim_{x\to\infty}\frac{\varphi(x)}{x+1}=0-0=0 x → ∞ lim F ( x ) = x → ∞ lim Φ ( − x ) − x → ∞ lim x + 1 φ ( x ) = 0 − 0 = 0 Since F ( 0 ) > 0 F(0)>0 F ( 0 ) > 0 , and F ( x ) F(x) F ( x ) decreases monotonically toward 0 0 0 as x → ∞ x\to\infty x → ∞ , it must remain non-negative over the entire positive domain.
Lemma 24 (Shifted Gaussian tail bound). For every x , q ∈ R x,q\in\mathbb{R} x , q ∈ R and every a ∈ ( 0 , 1 ) a\in(0,1) a ∈ ( 0 , 1 ) ,
Φ ( − ( x − q ) ) ≤ exp ( a q 2 2 ( 1 − a ) ) Φ ( − x ) a \Phi(-(x-q))\le\exp\left(\frac{aq^2}{2(1-a)}\right)\Phi(-x)^a Φ ( − ( x − q )) ≤ exp ( 2 ( 1 − a ) a q 2 ) Φ ( − x ) a Proof. Let Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) . By a change of variables,
Φ ( − ( x − q ) ) = ∫ x − q ∞ φ ( z ) d z = ∫ x ∞ φ ( u − q ) d u = ∫ x ∞ exp ( q u − q 2 2 ) φ ( u ) d u = E [ exp ( q Z − q 2 2 ) 1 { Z ≥ x } ] . \Phi(-(x-q))=\int_{x-q}^{\infty}\varphi(z)\,\mathrm{d}z=\int_x^\infty\varphi(u-q)\,\mathrm{d}u=\int_x^\infty\exp\left(qu-\frac{q^2}{2}\right)\varphi(u)\,\mathrm{d}u=\mathbb{E}\left[\exp\left(qZ-\frac{q^2}{2}\right)\mathbf{1}_{\{Z\ge x\}}\right]. Φ ( − ( x − q )) = ∫ x − q ∞ φ ( z ) d z = ∫ x ∞ φ ( u − q ) d u = ∫ x ∞ exp ( q u − 2 q 2 ) φ ( u ) d u = E [ exp ( q Z − 2 q 2 ) 1 { Z ≥ x } ] . Applying Hölder’s inequality with exponents 1 1 − a \frac{1}{1-a} 1 − a 1 and 1 a \frac{1}{a} a 1 gives
Φ ( − ( x − q ) ) ≤ ( E exp ( q Z − q 2 2 1 − a ) ) 1 − a P ( Z ≥ x ) a = exp ( a q 2 2 ( 1 − a ) ) Φ ( − x ) a , \Phi(-(x-q))\le\left(\mathbb{E}\exp\left(\frac{qZ-\frac{q^2}{2}}{1-a}\right)\right)^{1-a}\mathbb{P}(Z\ge x)^a=\exp\left(\frac{aq^2}{2(1-a)}\right)\Phi(-x)^a, Φ ( − ( x − q )) ≤ ( E exp ( 1 − a q Z − 2 q 2 ) ) 1 − a P ( Z ≥ x ) a = exp ( 2 ( 1 − a ) a q 2 ) Φ ( − x ) a , where we used E [ exp ( t Z ) ] = exp ( t 2 2 ) \mathbb{E}[\exp(tZ)]=\exp\left(\frac{t^2}{2}\right) E [ exp ( tZ )] = exp ( 2 t 2 ) .
Lemma 25 (Null-coordinate excursion bound). Let T ≥ 2 T\ge2 T ≥ 2 . Suppose that
∏ j = 1 T [ 1 − 2 Φ ( − ( 1 − 1 8 log ( 2 T ) ) λ δ r j ) ] ≤ 1 2 . \prod_{j=1}^{T}\left[1-2\Phi\left(-\left(1-\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_j}\right)\right]\le\frac{1}{2}. j = 1 ∏ T [ 1 − 2Φ ( − ( 1 − 8 log ( 2 T ) 1 ) r j λ δ ) ] ≤ 2 1 . Then
ν 2 ( α T − r ) 2 + σ T 2 ≥ ( 1 − s ) λ 2 4096 log 2 ( 2 T ) . \nu^2(\alpha_T-r)^2+\sigma_T^2\ge\frac{(1-s)\lambda^2}{4096\log^2(2T)}. ν 2 ( α T − r ) 2 + σ T 2 ≥ 4096 log 2 ( 2 T ) ( 1 − s ) λ 2 .
Proof. Condition on w ⋆ = 0 w^\star= 0 w ⋆ = 0 . We write for convenience,
a j = Φ ( − ( 1 − 1 8 log ( 2 T ) ) λ δ r j ) , a_j = \Phi\left(-\left(1-\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_j}\right), a j = Φ ( − ( 1 − 8 log ( 2 T ) 1 ) r j λ δ ) ,
b j = Φ ( − ( 1 + 1 8 log ( 2 T ) ) λ δ r j ) . b_j = \Phi\left(-\left(1+\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_j}\right). b j = Φ ( − ( 1 + 8 log ( 2 T ) 1 ) r j λ δ ) .
For j = 1 , … , T j=1,\ldots,T j = 1 , … , T , define
A j + = { Z j ≥ ( 1 + 1 8 log ( 2 T ) ) λ δ r j , ∣ Z k ∣ ≤ ( 1 − 1 8 log ( 2 T ) ) λ δ r k for every k > j } , A_j^+ = \left\{Z_j \geq\left(1+\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_j},\ |Z_k| \leq\left(1-\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_k}\text{ for every } k>j\right\}, A j + = { Z j ≥ ( 1 + 8 log ( 2 T ) 1 ) r j λ δ , ∣ Z k ∣ ≤ ( 1 − 8 log ( 2 T ) 1 ) r k λ δ for every k > j } ,
and similarly
A j − = { Z j ≤ − ( 1 + 1 8 log ( 2 T ) ) λ δ r j , ∣ Z k ∣ ≤ ( 1 − 1 8 log ( 2 T ) ) λ δ r k for every k > j } . A_j^- = \left\{Z_j \leq-\left(1+\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_j},\ |Z_k| \leq\left(1-\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_k}\text{ for every } k>j\right\}. A j − = { Z j ≤ − ( 1 + 8 log ( 2 T ) 1 ) r j λ δ , ∣ Z k ∣ ≤ ( 1 − 8 log ( 2 T ) 1 ) r k λ δ for every k > j } .
The events { A j + , A j − : 1 ≤ j ≤ T } \{A_j^+, A_j^-: 1\leq j\leq T\} { A j + , A j − : 1 ≤ j ≤ T } are pairwise disjoint. We first show A j + ⊂ { w t ≥ λ 8 log ( 2 T ) } A_j^+ \subset\{w_t \geq\frac{\lambda}{8\log(2T)}\} A j + ⊂ { w t ≥ 8 l o g ( 2 T ) λ } . Indeed, at task j j j the temporal L1 projection interval (see Lemma 20) is [ σ j τ j ( r j δ Z j − λ ) , σ j τ j ( r j δ Z j + λ ) ] \left[\sigma_j\tau_j\left(\frac{r_j}{\sqrt{\delta}}Z_j-\lambda\right),\sigma_j\tau_j\left(\frac{r_j}{\sqrt{\delta}}Z_j+\lambda\right)\right] [ σ j τ j ( δ r j Z j − λ ) , σ j τ j ( δ r j Z j + λ ) ] . On A j + A_j^+ A j + , r j δ Z j ≥ ( 1 + 1 8 log ( 2 T ) ) λ \frac{r_j}{\sqrt{\delta}}Z_j\geq\left(1+\frac{1}{8\log(2T)}\right)\lambda δ r j Z j ≥ ( 1 + 8 l o g ( 2 T ) 1 ) λ , and hence the lower endpoint is at least ( σ j τ j ) λ 8 log ( 2 T ) ≥ λ 8 log ( 2 T ) (\sigma_j\tau_j)\frac{\lambda}{8\log(2T)}\geq\frac{\lambda}{8\log(2T)} ( σ j τ j ) 8 l o g ( 2 T ) λ ≥ 8 l o g ( 2 T ) λ , where we used σ j τ j ≥ 1 \sigma_j\tau_j\geq1 σ j τ j ≥ 1 . Therefore w j ≥ λ 8 log ( 2 T ) w_j\geq\frac{\lambda}{8\log(2T)} w j ≥ 8 l o g ( 2 T ) λ . Now let k > j k>j k > j . On A j + A_j^+ A j + , Z k ≥ − ( 1 − 1 8 log ( 2 T ) ) λ δ r k Z_k\geq-\left(1-\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_k} Z k ≥ − ( 1 − 8 l o g ( 2 T ) 1 ) r k λ δ , so the upper endpoint at task k k k satisfies
σ k τ k ( r k δ Z k + λ ) ≥ ( σ k τ k ) λ 8 log ( 2 T ) ≥ λ 8 log ( 2 T ) . \sigma_k\tau_k\left(\frac{r_k}{\sqrt{\delta}}Z_k+\lambda\right)\geq(\sigma_k\tau_k)\frac{\lambda}{8\log(2T)}\geq\frac{\lambda}{8\log(2T)}. σ k τ k ( δ r k Z k + λ ) ≥ ( σ k τ k ) 8 log ( 2 T ) λ ≥ 8 log ( 2 T ) λ .
Thus, if w k − 1 ≥ λ 8 log ( 2 T ) w_{k-1}\geq\frac{\lambda}{8\log(2T)} w k − 1 ≥ 8 l o g ( 2 T ) λ , projecting w k − 1 w_{k-1} w k − 1 onto the task-k k k interval cannot produce a value below λ 8 log ( 2 T ) \frac{\lambda}{8\log(2T)} 8 l o g ( 2 T ) λ . By induction, w t ≥ λ 8 log ( 2 T ) w_t\geq\frac{\lambda}{8\log(2T)} w t ≥ 8 l o g ( 2 T ) λ . Likewise, A j − ⊂ { w t ≤ − λ 8 log ( 2 T ) } A_j^-\subset\{w_t\leq-\frac{\lambda}{8\log(2T)}\} A j − ⊂ { w t ≤ − 8 l o g ( 2 T ) λ } . Consequently,
P ( ∣ w t ∣ ≥ λ 8 log ( 2 T ) ∣ w ⋆ = 0 ) ≥ 2 ∑ j = 1 T b j ∏ k = j + 1 T ( 1 − 2 a k ) . (62) \mathbb{P}\left(|w_t|\geq\frac{\lambda}{8\log(2T)}\mid w^\star=0\right)\geq2\sum_{j=1}^{T}b_j\prod_{k=j+1}^{T}(1-2a_k). \tag*{(62)} P ( ∣ w t ∣ ≥ 8 log ( 2 T ) λ ∣ w ⋆ = 0 ) ≥ 2 j = 1 ∑ T b j k = j + 1 ∏ T ( 1 − 2 a k ) . (62)
We next lower-bound the right-hand side. Set P j = ∏ k = j T ( 1 − 2 a k ) P_j=\prod_{k=j}^{T}(1-2a_k) P j = ∏ k = j T ( 1 − 2 a k ) , and P T + 1 = 1 P_{T+1}=1 P T + 1 = 1 . Then P j + 1 − P j = 2 a j P j + 1 P_{j+1}-P_j=2a_jP_{j+1} P j + 1 − P j = 2 a j P j + 1 . Hence, by telescoping and the assumption of the lemma,
2 ∑ j = 1 T a j P j + 1 = 1 − P 1 ≥ 1 2 . (63) 2\sum_{j=1}^{T}a_jP_{j+1}=1-P_1\geq\frac{1}{2}. \tag*{(63)} 2 j = 1 ∑ T a j P j + 1 = 1 − P 1 ≥ 2 1 . (63)
Call an index j j j good if a j ≥ 1 16 T a_j\geq\frac{1}{16T} a j ≥ 16 T 1 . The contribution of the remaining indices satisfies
2 ∑ { j : a j < 1 16 T } a j P j + 1 ≤ 2 ∑ { j : a j < 1 16 T } a j ≤ 1 8 . 2\sum_{\{j:a_j<\frac{1}{16T}\}}a_jP_{j+1}\leq2\sum_{\{j:a_j<\frac{1}{16T}\}}a_j\leq\frac{1}{8}. 2 { j : a j < 16 T 1 } ∑ a j P j + 1 ≤ 2 { j : a j < 16 T 1 } ∑ a j ≤ 8 1 .
Therefore Eq. (63) gives 2 ∑ { j : a j ≥ 1 16 T } a j P j + 1 ≥ 3 8 2\sum_{\{j:a_j\geq\frac{1}{16T}\}}a_jP_{j+1}\geq\frac{3}{8} 2 ∑ { j : a j ≥ 16 T 1 } a j P j + 1 ≥ 8 3 . We claim that on every good index, b j ≥ 1 16 a j b_j\geq\frac{1}{16}a_j b j ≥ 16 1 a j . To see this, put y = ( 1 − 1 8 log ( 2 T ) ) λ δ r j y=\left(1-\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_j} y = ( 1 − 8 l o g ( 2 T ) 1 ) r j λ δ , and z = ( 1 + 1 8 log ( 2 T ) ) λ δ r j z=\left(1+\frac{1}{8\log(2T)}\right)\frac{\lambda\sqrt{\delta}}{r_j} z = ( 1 + 8 l o g ( 2 T ) 1 ) r j λ δ . Since a j = Φ ( − y ) ≥ 1 16 T a_j=\Phi(-y)\geq\frac{1}{16T} a j = Φ ( − y ) ≥ 16 T 1 and Φ ( − y ) ≤ 1 2 e − y 2 / 2 \Phi(-y)\leq\frac{1}{2}e^{-y^2/2} Φ ( − y ) ≤ 2 1 e − y 2 /2 , we have y 2 ≤ 2 log ( 8 T ) y^2\leq2\log(8T) y 2 ≤ 2 log ( 8 T ) . If y ≤ 1 y\leq1 y ≤ 1 , then, since for T ≥ 2 T\geq2 T ≥ 2 , 1 + 1 8 log ( 2 T ) 1 − 1 8 log ( 2 T ) ≤ 6 5 \frac{1+\frac{1}{8\log(2T)}}{1-\frac{1}{8\log(2T)}}\leq\frac{6}{5} 1 − 8 l o g ( 2 T ) 1 1 + 8 l o g ( 2 T ) 1 ≤ 5 6 , we have
Φ ( − z ) Φ ( − y ) ≥ Φ ( − 6 5 ) 1 2 ≥ 2 Φ ( − 6 5 ) > 1 16 . \frac{\Phi(-z)}{\Phi(-y)}\geq\frac{\Phi\left(-\frac{6}{5}\right)}{\frac{1}{2}}\geq2\Phi\left(-\frac{6}{5}\right)>\frac{1}{16}. Φ ( − y ) Φ ( − z ) ≥ 2 1 Φ ( − 5 6 ) ≥ 2Φ ( − 5 6 ) > 16 1 .
If y > 1 y>1 y > 1 , the Gaussian Mills bounds Φ ( − z ) ≥ z φ ( z ) 1 + z 2 \Phi(-z)\geq\frac{z\varphi(z)}{1+z^2} Φ ( − z ) ≥ 1 + z 2 z φ ( z ) , and Φ ( − y ) ≤ φ ( y ) y \Phi(-y)\leq\frac{\varphi(y)}{y} Φ ( − y ) ≤ y φ ( y ) give
Φ ( − z ) Φ ( − y ) ≥ z y 1 + z 2 exp ( − z 2 − y 2 2 ) . \frac{\Phi(-z)}{\Phi(-y)}\geq\frac{zy}{1+z^2}\exp\left(-\frac{z^2-y^2}{2}\right). Φ ( − y ) Φ ( − z ) ≥ 1 + z 2 z y exp ( − 2 z 2 − y 2 ) .
Here z y 1 + z 2 ≥ z 2 6 ( 1 + z 2 ) ≥ 5 12 ≥ 1 3 \frac{zy}{1+z^2} \ge\frac{z^2}{6(1+z^2)} \ge\frac{5}{12} \ge\frac{1}{3} 1 + z 2 z y ≥ 6 ( 1 + z 2 ) z 2 ≥ 12 5 ≥ 3 1 , while
z 2 − y 2 2 = 2 8 log ( 2 T ) ( λ d r j ) 2 = 2 8 log ( 2 T ) ( 1 − 1 8 log ( 2 T ) ) 2 y 2 ≤ 3 2 . \frac{z^2-y^2}{2}=\frac{2}{8\log(2T)}\left(\frac{\lambda\sqrt{d}}{r_j}\right)^2=\frac{\frac{2}{8\log(2T)}}{\left(1-\frac{1}{8\log(2T)}\right)^2}y^2\le\frac{3}{2}. 2 z 2 − y 2 = 8 log ( 2 T ) 2 ( r j λ d ) 2 = ( 1 − 8 l o g ( 2 T ) 1 ) 2 8 l o g ( 2 T ) 2 y 2 ≤ 2 3 .
Thus
Φ ( − z ) Φ ( − y ) ≥ 1 3 e − 3 2 > 1 16 . \frac{\Phi(-z)}{\Phi(-y)}\ge\frac{1}{3}e^{-\frac{3}{2}}>\frac{1}{16}. Φ ( − y ) Φ ( − z ) ≥ 3 1 e − 2 3 > 16 1 .
This proves b j ≥ 1 16 a j b_j \ge\frac{1}{16}a_j b j ≥ 16 1 a j . Combining Eq. (62) , with b j ≥ 1 16 a j b_j \ge\frac{1}{16}a_j b j ≥ 16 1 a j and 2 ∑ { j : a j ≥ 1 16 T } a j P j + 1 ≥ 3 8 2\sum_{\{j:a_j\ge\frac{1}{16T}\}}a_jP_{j+1}\ge\frac{3}{8} 2 ∑ { j : a j ≥ 16 T 1 } a j P j + 1 ≥ 8 3 .
P ( ∣ w t ∣ ≥ λ 8 log ( 2 T ) ∣ w ⋆ = 0 ) ≥ 2 ∑ { j : a j ≥ 1 16 T } b j P j + 1 ≥ 2 16 ∑ { j : a j ≥ 1 16 T } a j P j + 1 ≥ 3 128 ≥ 1 64 . \mathbb{P}\left(\lvert w_t\rvert\ge\frac{\lambda}{8\log(2T)}\mathrel{\big|}w^\star=0\right)\ge2\sum_{\{j:a_j\ge\frac{1}{16T}\}}b_jP_{j+1}\ge\frac{2}{16}\sum_{\{j:a_j\ge\frac{1}{16T}\}}a_jP_{j+1}\ge\frac{3}{128}\ge\frac{1}{64}. P ( ∣ w t ∣ ≥ 8 log ( 2 T ) λ w ⋆ = 0 ) ≥ 2 { j : a j ≥ 16 T 1 } ∑ b j P j + 1 ≥ 16 2 { j : a j ≥ 16 T 1 } ∑ a j P j + 1 ≥ 128 3 ≥ 64 1 .
Finally, on { w ⋆ = 0 } \{w^\star=0\} { w ⋆ = 0 } the target r w ⋆ rw^\star r w ⋆ is zero, so
ν 2 ( α t − r ) 2 + σ t 2 = E [ ( w t − r w ⋆ ) 2 ] ≥ ( 1 − s ) E [ w t 2 ∣ w ⋆ = 0 ] ≥ ( 1 − s ) ( λ 8 log ( 2 T ) ) 2 P ( ∣ w t ∣ ≥ λ 8 log ( 2 T ) ∣ w ⋆ = 0 ] ≥ ( 1 − s ) ( 1 8 log ( 2 T ) ) 2 λ 2 64 . \begin{aligned}
\nu^2(\alpha_t-r)^2+\sigma_t^2=\mathbb{E}\left[(w_t-rw^\star)^2\right]\ge(1-s)\mathbb{E}\left[w_t^2\mid w^\star=0\right] \\
&\ge(1-s)\left(\frac{\lambda}{8\log(2T)}\right)^2\mathbb{P}\left(\lvert w_t\rvert\ge\frac{\lambda}{8\log(2T)}\mathrel{\big|}w^\star=0\right]\ge(1-s)\frac{\left(\frac{1}{8\log(2T)}\right)^2\lambda^2}{64}.
\end{aligned} ν 2 ( α t − r ) 2 + σ t 2 = E [ ( w t − r w ⋆ ) 2 ] ≥ ( 1 − s ) E [ w t 2 ∣ w ⋆ = 0 ] ≥ ( 1 − s ) ( 8 log ( 2 T ) λ ) 2 P ( ∣ w t ∣ ≥ 8 log ( 2 T ) λ w ⋆ = 0 ] ≥ ( 1 − s ) 64 ( 8 l o g ( 2 T ) 1 ) 2 λ 2 .
We conclude that
ν 2 ( α T − r ) 2 + σ T 2 ≥ ( 1 − s ) λ 2 4096 log 2 ( 2 T ) . \nu^2(\alpha_T-r)^2+\sigma_T^2\ge\frac{(1-s)\lambda^2}{4096\log^2(2T)}. ν 2 ( α T − r ) 2 + σ T 2 ≥ 4096 log 2 ( 2 T ) ( 1 − s ) λ 2 .
Lemma 26 (Uniform lower bound for L1 regularization). *There exist universal constants C , c > 0 C,c>0 C , c > 0 , such that for every s ≤ s 0 = c min { δ 1 + ν 1 log + 2 ( e ( 1 + ν ) δ ) , 1 δ ( 1 + ν ) 1 log + 4 ( e δ ( 1 + ν ) ) , δ ν 2 ( 1 − 2 ρ ) 2 ( 1 + ν ) 4 } s\le s_0=c\min\left\{\frac{\delta}{1+\nu}\frac{1}{\log_+^2\left(\frac{e(1+\nu)}{\delta}\right)},\frac{1}{\delta(1+\nu)}\frac{1}{\log_+^4\left(e\delta(1+\nu)\right)},\frac{\delta\nu^2(1-2\rho)^2}{(1+\nu)^4}\right\} s ≤ s 0 = c min { 1 + ν δ l o g + 2 ( δ e ( 1 + ν ) ) 1 , δ ( 1 + ν ) 1 l o g + 4 ( eδ ( 1 + ν ) ) 1 , ( 1 + ν ) 4 δ ν 2 ( 1 − 2 ρ ) 2 } , every T ≥ 2 T\ge2 T ≥ 2 , and every λ > 0 \lambda>0 λ > 0 ,
L ( w T ) − inf w L ( w ) ≥ C ν 7 ( 1 − 2 ρ ) 7 ( 1 + ν ) 32 ( 1 + δ ) 3 2 s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 2 . \mathcal{L}(w_T)-\inf_w\mathcal{L}(w)\ge C\frac{\nu^7(1-2\rho)^7}{(1+\nu)^{32}(1+\delta)^{\frac{3}{2}}}\frac{s\log^3\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^2}. L ( w T ) − w inf L ( w ) ≥ C ( 1 + ν ) 32 ( 1 + δ ) 2 3 ν 7 ( 1 − 2 ρ ) 7 ( log ( s 1 ) + log ( 2 T ) ) 2 s log 3 ( s 1 ) . Moreover, if T = 1 T=1 T = 1 then there exist universal constants c , C > 0 c,C>0 c , C > 0 such that, for all s ≤ c δ ν 2 ( 1 − 2 ρ ) 2 ( 1 + ν ) 2 log ( 1 + ν δ ν 2 ( 1 − 2 ρ ) 2 ) s\le c\frac{\delta\nu^2(1-2\rho)^2}{(1+\nu)^2\log\left(\frac{1+\nu}{\delta\nu^2(1-2\rho)^2}\right)} s ≤ c ( 1 + ν ) 2 l o g ( δ ν 2 ( 1 − 2 ρ ) 2 1 + ν ) δ ν 2 ( 1 − 2 ρ ) 2 and for every λ > 0 \lambda>0 λ > 0 ,
L ( w 1 ) − inf w L ( w ) ≥ C ( 1 + ν ) 6 s δ log ( 1 s ) . \mathcal{L}(w_1)-\inf_w\mathcal{L}(w)\ge\frac{C}{(1+\nu)^6}\frac{s}{\delta}\log\left(\frac{1}{s}\right). L ( w 1 ) − w inf L ( w ) ≥ ( 1 + ν ) 6 C δ s log ( s 1 ) .
Proof. Note that for the special case of T = 1 T=1 T = 1 uses the lower bound developed in Theorem 27.
For temporal L1 by Lemma 20,
w t = w t − 1 + S λ σ t τ t ( σ t τ t ( θ t w ⋆ + r t δ Z t ) − w t − 1 ) . w_t=w_{t-1}+S_{\lambda\sigma_t\tau_t}\left(\sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t\right)-w_{t-1}\right). w t = w t − 1 + S λ σ t τ t ( σ t τ t ( θ t w ⋆ + δ r t Z t ) − w t − 1 ) .
Equivalently we may write w t w_t w t as the projection of w t − 1 w_{t-1} w t − 1 onto the interval σ t τ t ( θ t w ⋆ + r t δ Z t − λ , θ t w ⋆ + r t δ Z t + λ ) \sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t-\lambda,\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t+\lambda\right) σ t τ t ( θ t w ⋆ + δ r t Z t − λ , θ t w ⋆ + δ r t Z t + λ ) ,
w t = Π [ σ t τ t ( θ t w ⋆ + r t δ Z t − λ ) , σ t τ t ( θ t w ⋆ + r t δ Z t + λ ) ] ( w t − 1 ) . w_t=\Pi_{\left[\sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t-\lambda\right),\sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t+\lambda\right)\right]}(w_{t-1}). w t = Π [ σ t τ t ( θ t w ⋆ + δ r t Z t − λ ) , σ t τ t ( θ t w ⋆ + δ r t Z t + λ ) ] ( w t − 1 ) .
Hence ∣ w t σ t τ t − r t δ Z t − θ t w ⋆ ∣ ≤ λ \left|\frac{w_t}{\sigma_t\tau_t}-\frac{r_t}{\sqrt{\delta}}Z_t-\theta_tw^\star\right|\le\lambda σ t τ t w t − δ r t Z t − θ t w ⋆ ≤ λ . We assume by contradiction that for arbitrarily small s s s there exist T ≥ 2 T\ge2 T ≥ 2 and λ > 0 \lambda>0 λ > 0 such that,
ν 2 ( α t − r ) 2 + σ t 2 < c ⋆ s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 2 , (64) \nu^2(\alpha_t-r)^2+\sigma_t^2<c_\star s\frac{\log^3\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^2}, \tag*{(64)} ν 2 ( α t − r ) 2 + σ t 2 < c ⋆ s ( log ( s 1 ) + log ( 2 T ) ) 2 log 3 ( s 1 ) , (64)
here c ⋆ > 0 c_\star>0 c ⋆ > 0 is a sufficiently small constant whose value be established at the end of the proof. Since
s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 2 ≤ s log ( 1 s ) ⟶ 0 , \frac{s\log^3\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^2}\le s\log\left(\frac{1}{s}\right)\longrightarrow0, ( log ( s 1 ) + log ( 2 T ) ) 2 s log 3 ( s 1 ) ≤ s log ( s 1 ) ⟶ 0 ,
we must have α t ⟶ r \alpha_t\longrightarrow r α t ⟶ r , and σ t ⟶ 0 \sigma_t\longrightarrow0 σ t ⟶ 0 . Then ν α t Z 1 + σ t Z 2 ⟶ ν r Z 1 \nu\alpha_tZ_1+\sigma_tZ_2\longrightarrow\nu rZ_1 ν α t Z 1 + σ t Z 2 ⟶ ν r Z 1 . Furthermore, the second equation gives γ t = 1 r t δ E [ Z t w t ] \gamma_t=\frac{1}{r_t\sqrt{\delta}}\mathbb{E}[Z_tw_t] γ t = r t δ 1 E [ Z t w t ] . Since Z t Z_t Z t is independent of w ⋆ w^\star w ⋆ , by Cauchy-Schwartz,
∣ E [ Z t w t ] ∣ = ∣ E [ Z t ( w t − r w ⋆ ) ] ∣ ≤ ν 2 ( α t − r ) 2 + σ t 2 . \left|\mathbb{E}[Z_tw_t]\right|=\left|\mathbb{E}[Z_t(w_t-rw^\star)]\right|\le\sqrt{\nu^2(\alpha_t-r)^2+\sigma_t^2}. ∣ E [ Z t w t ] ∣ = ∣ E [ Z t ( w t − r w ⋆ )] ∣ ≤ ν 2 ( α t − r ) 2 + σ t 2 .
Using σ t τ t r t ≥ 1 \sigma_t\tau_tr_t\ge1 σ t τ t r t ≥ 1 (see below Eq. (50) ), we obtain
0 ≤ γ t σ t τ t ≤ ν 2 ( α t − r ) 2 + σ t 2 δ ⟶ 0. 0\le\frac{\gamma_t}{\sigma_t\tau_t}\le\sqrt{\frac{\nu^2(\alpha_t-r)^2+\sigma_t^2}{\delta}}\longrightarrow0. 0 ≤ σ t τ t γ t ≤ δ ν 2 ( α t − r ) 2 + σ t 2 ⟶ 0.
The sixth scalar equation is
1 − γ t σ t τ t = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . 1-\frac{\gamma_t}{\sigma_t\tau_t}=\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))}{1+\gamma_t\phi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}\right]. 1 − σ t τ t γ t = E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ( 1 − π ρ ( ν Z 1 )) ] .
The left-hand side tends to 1. This forces γ t ⟶ 0 \gamma_t\longrightarrow0 γ t ⟶ 0 . Indeed, if γ t \gamma_t γ t converged to a positive finite number, then the denominator on the right would be strictly larger than one with positive probability, since in that case by the proximal relation (Eq. (65) ) ν r Z 1 − γ ∞ ≤ Prox γ t ϕ ( ν r Z 1 ) ≤ ν r Z 1 \nu rZ_1-\gamma_\infty\le\operatorname{Prox}_{\gamma_t\phi}(\nu rZ_1)\le\nu rZ_1 ν r Z 1 − γ ∞ ≤ Prox γ t ϕ ( ν r Z 1 ) ≤ ν r Z 1 , then since Z 1 Z_1 Z 1 is finite almost surely and
ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) = e Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ( 1 + e Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 > 0 \phi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)=\frac{e^{\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)}}{\left(1+e^{\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)}\right)^2}>0 ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) = ( 1 + e Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 e Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) > 0
so 1 + γ ∞ ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) > 1 1+\gamma_\infty\phi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)>1 1 + γ ∞ ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) > 1 almost surely, and from symmetry E [ π ρ ( ν Z 1 ) ] = 1 2 \mathbb{E}[\pi_\rho(\nu Z_1)] = \frac{1}{2} E [ π ρ ( ν Z 1 )] = 2 1 implying E [ 2 ( 1 − π ρ ( − ν Z 1 ) ) ] = 1 \mathbb{E}[2(1-\pi_\rho(-\nu Z_1))] = 1 E [ 2 ( 1 − π ρ ( − ν Z 1 ))] = 1 so its expectation would have limit strictly smaller than one. If γ t → ∞ \gamma_t \to\infty γ t → ∞ , then from the proximal relation
ν α t Z 1 + σ t Z 2 = Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) + γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) , (65) \nu\alpha_t Z_1+\sigma_t Z_2=\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)+\gamma_t\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)), \tag*{(65)} ν α t Z 1 + σ t Z 2 = Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) + γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) , (65)
and α t → r \alpha_t \to r α t → r , σ t → 0 \sigma_t \to0 σ t → 0 , one gets Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) → − ∞ \operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\to-\infty Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) → − ∞ on every event { ∣ ν α t Z 1 + σ t Z 2 ∣ ≤ M } \{|\nu\alpha_t Z_1+\sigma_t Z_2|\le M\} { ∣ ν α t Z 1 + σ t Z 2 ∣ ≤ M } . Indeed, if you suppose there exists B > 0 B>0 B > 0 such that Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) > − B \operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)>-B Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) > − B then by
ν α t Z 1 + σ t Z 2 = Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) + γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) > − B + γ t ϕ ′ ( − B ) → γ t → ∞ ∞ \nu\alpha_t Z_1+\sigma_t Z_2=\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)+\gamma_t\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))>-B+\gamma_t\phi'(-B)\xrightarrow{\gamma_t\to\infty}\infty ν α t Z 1 + σ t Z 2 = Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) + γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) > − B + γ t ϕ ′ ( − B ) γ t → ∞ ∞
which contradicts ∣ ν α t Z 1 + σ t Z 2 ∣ ≤ M |\nu\alpha_t Z_1+\sigma_t Z_2|\le M ∣ ν α t Z 1 + σ t Z 2 ∣ ≤ M , also we have
γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ( 1 − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ) \gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))=\gamma_t\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))(1-\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))) γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ( 1 − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )))
= 64 ( ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ( 1 − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ) → γ t → ∞ ∞ , \overset{64}{=}(\nu\alpha_t Z_1+\sigma_t Z_2-\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))(1-\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)))\xrightarrow{\gamma_t\to\infty}\infty, = 64 ( ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ( 1 − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ))) γ t → ∞ ∞ ,
the expectation in the sixth equation would then tend to zero on events whose probability tends to one, again a contradiction. Thus
∣ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) − ν α t Z 1 + σ t Z 2 ∣ = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≤ γ t ⟶ 0 , |\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)-\nu\alpha_t Z_1+\sigma_t Z_2|=\gamma_t\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\le\gamma_t\longrightarrow0, ∣ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) − ν α t Z 1 + σ t Z 2 ∣ = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≤ γ t ⟶ 0 ,
and consequently Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ⟶ ν r Z 1 \operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\longrightarrow\nu rZ_1 Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ⟶ ν r Z 1 in probability. From
r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] , r_t^2=2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))^2\right], r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] ,
dominated convergence gives
r t 2 ⟶ 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( ν r Z 1 ) 2 ] > 0. r_t^2\longrightarrow2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\nu rZ_1)^2\right]>0. r t 2 ⟶ 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( ν r Z 1 ) 2 ] > 0.
Hence r t ≥ c > 0 r_t\ge c>0 r t ≥ c > 0 for a sufficiently small s s s . Now we aim to lower bound E [ ( w t − r w ⋆ ) 2 ] = ν 2 ( α t − r ) 2 + σ t 2 \mathbb{E}[(w_t-rw^\star)^2]=\nu^2(\alpha_t-r)^2+\sigma_t^2 E [( w t − r w ⋆ ) 2 ] = ν 2 ( α t − r ) 2 + σ t 2 , note that
E [ ( w t − r w ⋆ ) 2 ] ≥ E [ E [ ( w t − r w ⋆ ) 2 ] ∣ w ⋆ , w t − 1 ] = Var ( w t ∣ w ⋆ , w t − 1 ) + ( E [ w t ∣ w ⋆ , w t − 1 ] − r w ⋆ ) 2 ≥ Var ( w t ∣ w ⋆ , w t − 1 ) \begin{aligned}
\mathbb{E}[(w_t-rw^\star)^2]\ge\mathbb{E}[\mathbb{E}[(w_t-rw^\star)^2]\mid w^\star,w_{t-1}] \\
&=\operatorname{Var}(w_t\mid w^\star,w_{t-1})+(\mathbb{E}[w_t\mid w^\star,w_{t-1}]-rw^\star)^2\ge\operatorname{Var}(w_t\mid w^\star,w_{t-1})
\end{aligned} E [( w t − r w ⋆ ) 2 ] ≥ E [ E [( w t − r w ⋆ ) 2 ] ∣ w ⋆ , w t − 1 ] = Var ( w t ∣ w ⋆ , w t − 1 ) + ( E [ w t ∣ w ⋆ , w t − 1 ] − r w ⋆ ) 2 ≥ Var ( w t ∣ w ⋆ , w t − 1 )
So it is sufficient to lower bound Var ( w t ∣ w ⋆ , w t − 1 ) \operatorname{Var}(w_t\mid w^\star,w_{t-1}) Var ( w t ∣ w ⋆ , w t − 1 ) . By Cauchy-Schwartz,
Cov ( w t , Z t ∣ w ⋆ , w t − 1 ) 2 ≤ Var ( w t ∣ w ⋆ , w t − 1 ) Var ( Z t ∣ w ⋆ , w t − 1 ) . \operatorname{Cov}(w_t,Z_t\mid w^\star,w_{t-1})^2\le\operatorname{Var}(w_t\mid w^\star,w_{t-1})\operatorname{Var}(Z_t\mid w^\star,w_{t-1}). Cov ( w t , Z t ∣ w ⋆ , w t − 1 ) 2 ≤ Var ( w t ∣ w ⋆ , w t − 1 ) Var ( Z t ∣ w ⋆ , w t − 1 ) .
Since Var ( Z t ∣ w ⋆ , w t − 1 ) = 1 \operatorname{Var}(Z_t\mid w^\star,w_{t-1})=1 Var ( Z t ∣ w ⋆ , w t − 1 ) = 1 then Var ( w t ∣ w ⋆ , w t − 1 ) ≥ Cov ( w t , Z t ∣ w ⋆ , w t − 1 ) 2 \operatorname{Var}(w_t\mid w^\star,w_{t-1})\ge\operatorname{Cov}(w_t,Z_t\mid w^\star,w_{t-1})^2 Var ( w t ∣ w ⋆ , w t − 1 ) ≥ Cov ( w t , Z t ∣ w ⋆ , w t − 1 ) 2 (due to the independence of Z t Z_t Z t in w ⋆ w^\star w ⋆ and w t − 1 w_{t-1} w t − 1 ). If w t ≠ w t − 1 w_t\ne w_{t-1} w t = w t − 1 that means that soft threshold has magnitude larger than the threshold i.e.
∣ Z t + δ r t ( θ t w ⋆ − w t − 1 σ t τ t ) ⏟ = c ∣ > δ r t λ \left|Z_t+\underbrace{\frac{\sqrt{\delta}}{r_t}\left(\theta_t w^\star-\frac{w_{t-1}}{\sigma_t\tau_t}\right)}_{=c}\right|>\frac{\sqrt{\delta}}{r_t}\lambda Z t + = c r t δ ( θ t w ⋆ − σ t τ t w t − 1 ) > r t δ λ
the probability of this event is at least as large when c = 0 c=0 c = 0 . Explicitly,
P ( ∣ Z t + c ∣ > δ r t λ ) ≥ P ( ∣ Z t ∣ > δ r t λ ) = 2 Φ ( − λ δ r t ) \mathbb{P}\left(|Z_t+c|>\frac{\sqrt{\delta}}{r_t}\lambda\right)\ge\mathbb{P}\left(|Z_t|>\frac{\sqrt{\delta}}{r_t}\lambda\right)=2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_t}\right) P ( ∣ Z t + c ∣ > r t δ λ ) ≥ P ( ∣ Z t ∣ > r t δ λ ) = 2Φ ( − r t λ δ )
By Gaussian integration by parts and since ∂ w t ∂ Z t = σ t τ t r t δ 1 { ∣ Z t + c ∣ > δ r t λ } \frac{\partial w_t}{\partial Z_t}=\frac{\sigma_t\tau_t r_t}{\sqrt{\delta}}\mathbf{1}_{\{|Z_t+c|>\frac{\sqrt{\delta}}{r_t}\lambda\}} ∂ Z t ∂ w t = δ σ t τ t r t 1 { ∣ Z t + c ∣ > r t δ λ } , this gives
ν 2 ( α t − r ) 2 + σ t 2 ≥ Cov ( w t , Z t ∣ w ⋆ , w t − 1 ) 2 = E [ w t Z t ∣ w ⋆ , w t − 1 ] 2 = E [ σ t τ t r t δ 1 { ∣ Z t + c ∣ > δ r t λ } | w ⋆ , w t − 1 ] 2 ≥ 4 ( σ t τ t ) 2 r t 2 δ Φ ( − λ δ r t ) 2 ≥ 4 δ Φ ( − λ δ r t ) 2 . \begin{aligned}
\nu^2(\alpha_t-r)^2+\sigma_t^2\ge\operatorname{Cov}(w_t,Z_t\mid w^\star,w_{t-1})^2=\mathbb{E}[w_tZ_t\mid w^\star,w_{t-1}]^2 \\
&=\mathbb{E}\left[\frac{\sigma_t\tau_t r_t}{\sqrt{\delta}}\mathbf{1}_{\{|Z_t+c|>\frac{\sqrt{\delta}}{r_t}\lambda\}}\middle|w^\star,w_{t-1}\right]^2\ge\frac{4(\sigma_t\tau_t)^2r_t^2}{\delta}\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_t}\right)^2\ge\frac{4}{\delta}\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_t}\right)^2.
\end{aligned} ν 2 ( α t − r ) 2 + σ t 2 ≥ Cov ( w t , Z t ∣ w ⋆ , w t − 1 ) 2 = E [ w t Z t ∣ w ⋆ , w t − 1 ] 2 = E [ δ σ t τ t r t 1 { ∣ Z t + c ∣ > r t δ λ } w ⋆ , w t − 1 ] 2 ≥ δ 4 ( σ t τ t ) 2 r t 2 Φ ( − r t λ δ ) 2 ≥ δ 4 Φ ( − r t λ δ ) 2 .
Together with the fact that r t > 0 r_t>0 r t > 0 and by the assumption, this implies λ → ∞ \lambda\to\infty λ → ∞ . Condition on w ⋆ = 0 w^\star=0 w ⋆ = 0 . If w t − 1 = 0 w_{t-1}=0 w t − 1 = 0 , then w t = σ t τ t S λ ( r t δ Z t ) w_t=\sigma_t\tau_t S_\lambda\left(\frac{r_t}{\sqrt{\delta}}Z_t\right) w t = σ t τ t S λ ( δ r t Z t ) , and therefore w t ≠ 0 w_t\ne0 w t = 0 if and only if ∣ Z t ∣ > λ δ r t |Z_t|>\frac{\lambda\sqrt{\delta}}{r_t} ∣ Z t ∣ > r t λ δ . Once a null coordinate becomes nonzero it remains nonzero almost surely: if w t − 1 ≠ 0 w_{t-1} \ne0 w t − 1 = 0 , a later projection can equal exactly zero only if one of the continuously distributed interval endpoints is exactly zero. Hence
P ( w t ≠ 0 ∣ w ⋆ = 0 ) = 1 − ∏ j = 1 t [ 1 − 2 Φ ( − λ δ r j ) ] (66) \mathbb{P}(w_t \ne0 \mid w^\star= 0) = 1 - \prod_{j=1}^{t}\left[1 - 2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right] \tag*{(66)} P ( w t = 0 ∣ w ⋆ = 0 ) = 1 − j = 1 ∏ t [ 1 − 2Φ ( − r j λ δ ) ] (66)
We first claim that P ( w t ≠ 0 ∣ w ⋆ = 0 ) < 1 2 \mathbb{P}(w_t \ne0 \mid w^\star= 0) < \frac{1}{2} P ( w t = 0 ∣ w ⋆ = 0 ) < 2 1 . Otherwise, ∏ j = 1 t [ 1 − 2 Φ ( − λ δ r j ) ] ≤ 1 2 \prod_{j=1}^{t}\left[1 - 2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right] \le\frac{1}{2} ∏ j = 1 t [ 1 − 2Φ ( − r j λ δ ) ] ≤ 2 1 . Since 1 8 log ( 2 T ) > 0 \frac{1}{8\log(2T)} > 0 8 l o g ( 2 T ) 1 > 0 , this implies
∏ j = 1 t [ 1 − 2 Φ ( − ( 1 − 1 8 log ( 2 t ) ) λ δ r j ) ] ≤ ∏ j = 1 t [ 1 − 2 Φ ( − λ δ r j ) ] ≤ 1 2 \prod_{j=1}^{t}\left[1 - 2\Phi\left(-\left(1-\frac{1}{8\log(2t)}\right)\frac{\lambda\sqrt{\delta}}{r_j}\right)\right]
\le\prod_{j=1}^{t}\left[1 - 2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right]\le\frac{1}{2} j = 1 ∏ t [ 1 − 2Φ ( − ( 1 − 8 log ( 2 t ) 1 ) r j λ δ ) ] ≤ j = 1 ∏ t [ 1 − 2Φ ( − r j λ δ ) ] ≤ 2 1
Since it must hold λ → ∞ \lambda\to\infty λ → ∞ it means also that λ ≥ 1 \lambda\ge1 λ ≥ 1 then by Lemma 25 ,
ν 2 ( α t − r ) 2 + σ t 2 ≥ 1 − s 4096 log 2 ( 2 T ) \nu^2(\alpha_t-r)^2+\sigma_t^2\ge\frac{1-s}{4096\log^2(2T)} ν 2 ( α t − r ) 2 + σ t 2 ≥ 4096 log 2 ( 2 T ) 1 − s
But
s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 2 ≤ s log 3 ( 1 s ) log 2 ( 2 T ) \frac{s\log^3\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^2}\le\frac{s\log^3\left(\frac{1}{s}\right)}{\log^2(2T)} ( log ( s 1 ) + log ( 2 T ) ) 2 s log 3 ( s 1 ) ≤ log 2 ( 2 T ) s log 3 ( s 1 )
so it is enough to take s s s sufficiently small that c ⋆ s log 3 ( 1 s ) < 1 8192 c_\star s\log^3\left(\frac{1}{s}\right)<\frac{1}{8192} c ⋆ s log 3 ( s 1 ) < 8192 1 . This contradicts Eq. (64) . Thus P ( w t ≠ 0 ∣ w ⋆ = 0 ) < 1 2 \mathbb{P}(w_t \ne0 \mid w^\star= 0) < \frac{1}{2} P ( w t = 0 ∣ w ⋆ = 0 ) < 2 1 . Moreover, Eq. (66) implies 2 Φ ( − λ δ r j ) < 1 2 2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)<\frac{1}{2} 2Φ ( − r j λ δ ) < 2 1 for every j ≤ T j\le T j ≤ T . Hence, we have λ δ r j ≥ Φ − 1 ( 3 4 ) > 0 \frac{\lambda\sqrt{\delta}}{r_j}\ge\Phi^{-1}\left(\frac{3}{4}\right)>0 r j λ δ ≥ Φ − 1 ( 4 3 ) > 0 , and therefore, using r j ≥ h ( ν ) 8 r_j\ge\sqrt{\frac{h(\nu)}{8}} r j ≥ 8 h ( ν ) (see Eq. (74) ), then
λ ≥ Φ − 1 ( 3 4 ) h ( ν ) 8 δ . \lambda\ge\Phi^{-1}\left(\frac{3}{4}\right)\sqrt{\frac{h(\nu)}{8\delta}}. λ ≥ Φ − 1 ( 4 3 ) 8 δ h ( ν ) .
From Eq. (66) , and since 1 − ∏ j = 1 t z j ≤ ∑ j = 1 t ( 1 − z j ) 1-\prod_{j=1}^{t}z_j\le\sum_{j=1}^{t}(1-z_j) 1 − ∏ j = 1 t z j ≤ ∑ j = 1 t ( 1 − z j ) for all z j ∈ [ 0 , 1 ] z_j\in[0,1] z j ∈ [ 0 , 1 ] where j ∈ [ 1 , t ] j\in[1,t] j ∈ [ 1 , t ] , therefore, P ( w t ≠ 0 ∣ w ⋆ = 0 ) ≤ ∑ j = 1 t 2 Φ ( − λ δ r j ) \mathbb{P}(w_t\ne0\mid w^\star=0)\le\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right) P ( w t = 0 ∣ w ⋆ = 0 ) ≤ ∑ j = 1 t 2Φ ( − r j λ δ ) . Also by Eq. (66) ,
− log ( 1 − P ( w t ≠ 0 ∣ w ⋆ = 0 ) ) = − ∑ j = 1 t log ( 1 − 2 Φ ( − λ δ r j ) ) ≥ ∑ j = 1 T 2 Φ ( − λ δ r j ) . -\log\left(1-\mathbb{P}(w_t\ne0\mid w^\star=0)\right)
=-\sum_{j=1}^{t}\log\left(1-2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)
\ge\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right). − log ( 1 − P ( w t = 0 ∣ w ⋆ = 0 ) ) = − j = 1 ∑ t log ( 1 − 2Φ ( − r j λ δ ) ) ≥ j = 1 ∑ T 2Φ ( − r j λ δ ) .
Since the probability on the left is smaller than 1 2 \frac{1}{2} 2 1 , and − log ( 1 − x ) ≤ 2 x -\log(1-x)\le2x − log ( 1 − x ) ≤ 2 x for all 0 ≤ x ≤ 1 2 0\le x\le\frac{1}{2} 0 ≤ x ≤ 2 1 so
P ( w t ≠ 0 ∣ w ⋆ = 0 ) ≤ ∑ j = 1 T 2 Φ ( − λ δ r j ) ≤ 2 P ( w t ≠ 0 ∣ w ⋆ = 0 ) . (67) \mathbb{P}(w_t\ne0\mid w^\star=0)\le\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\le2\mathbb{P}(w_t\ne0\mid w^\star=0). \tag*{(67)} P ( w t = 0 ∣ w ⋆ = 0 ) ≤ j = 1 ∑ T 2Φ ( − r j λ δ ) ≤ 2 P ( w t = 0 ∣ w ⋆ = 0 ) . (67)
We also need a quantitative relation between this probability and the null-coordinate squared error. On { w ⋆ = 0 , w t ≠ 0 } \{w^\star=0,w_t\ne0\} { w ⋆ = 0 , w t = 0 } let J J J be the last update time. Then after projecting the update to one of the endpoints,
w t = σ J τ J ( r J δ Z J − ϵ J λ ) . w_t=\sigma_J\tau_J\left(\frac{r_J}{\sqrt{\delta}}Z_J-\epsilon_J\lambda\right). w t = σ J τ J ( δ r J Z J − ϵ J λ ) .
with ϵ J ∈ { − 1 , + 1 } \epsilon_J\in\{-1,+1\} ϵ J ∈ { − 1 , + 1 } . Thus, if 0 < ∣ w t ∣ ≤ a 0<|w_t|\le a 0 < ∣ w t ∣ ≤ a , then, ∣ σ J τ J ( r J δ Z J − ϵ J λ ) ∣ ≤ a \left|\sigma_J\tau_J\left(\frac{r_J}{\sqrt{\delta}}Z_J-\epsilon_J\lambda\right)\right|\le a σ J τ J ( δ r J Z J − ϵ J λ ) ≤ a implies ∣ Z J − ϵ J λ δ r J ∣ ≤ a δ \left|Z_J-\epsilon_J\frac{\lambda\sqrt{\delta}}{r_J}\right|\le a\sqrt{\delta} Z J − ϵ J r J λ δ ≤ a δ , because σ J τ J r J ≥ 1 \sigma_J\tau_Jr_J\ge1 σ J τ J r J ≥ 1 (see Eq. (46) ). Denote for simplicity a = 1 64 δ ( 1 + L ) a=\frac{1}{64\sqrt{\delta}(1+L)} a = 64 δ ( 1 + L ) 1 where
L = 2 log ( 4 T ∑ j = 1 T 2 Φ ( − λ δ r j ) ) . L=\sqrt{2\log\left(\frac{4T}{\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)}\right)}. L = 2 log ∑ j = 1 T 2Φ ( − r j λ δ ) 4 T .
Since a δ = 1 64 ( 1 + L ) ≤ 1 64 < Φ − 1 ( 3 4 ) a\sqrt{\delta}=\frac{1}{64(1+L)}\le\frac{1}{64}<\Phi^{-1}\left(\frac{3}{4}\right) a δ = 64 ( 1 + L ) 1 ≤ 64 1 < Φ − 1 ( 4 3 ) , we have a ≤ λ r j a\le\frac{\lambda}{r_j} a ≤ r j λ . Thus, using the Gaussian density bound P ( ∣ Z − x ∣ ≤ q ) ≤ 2 q ϕ ( x − q ) \mathbb{P}(|Z-x|\le q)\le2q\phi(x-q) P ( ∣ Z − x ∣ ≤ q ) ≤ 2 q ϕ ( x − q ) , whenever x ≥ q > 0 x\ge q>0 x ≥ q > 0 together with the identity ϕ ( x ) ≤ ( x + 1 ) Φ ( − x ) \phi(x)\le(x+1)\Phi(-x) ϕ ( x ) ≤ ( x + 1 ) Φ ( − x ) , for all x ≥ 0 x\ge0 x ≥ 0 which follows from Lemma 23 , we have, if λ r j ≤ L \frac{\lambda}{r_j}\le L r j λ ≤ L ,
P ( ∣ Z − λ δ r j ∣ ≤ a δ ) ≤ 2 a δ ϕ ( λ δ r j − a δ ) = 2 a δ e a δ r j λ − ( a δ ) 2 2 ϕ ( λ δ r j ) \mathbb{P}\left(\left|Z-\frac{\lambda\sqrt{\delta}}{r_j}\right|\le a\sqrt{\delta}\right)
\le2a\sqrt{\delta}\phi\left(\frac{\lambda\sqrt{\delta}}{r_j}-a\sqrt{\delta}\right)
=2a\sqrt{\delta}e^{\frac{a\delta}{r_j}\lambda-\frac{(a\sqrt{\delta})^2}{2}}\phi\left(\frac{\lambda\sqrt{\delta}}{r_j}\right) P ( Z − r j λ δ ≤ a δ ) ≤ 2 a δ ϕ ( r j λ δ − a δ ) = 2 a δ e r j a δ λ − 2 ( a δ ) 2 ϕ ( r j λ δ )
≤ 2 a δ e L a δ ( L + 1 ) Φ ( − λ δ r j ) \le2a\sqrt{\delta}e^{L a\sqrt{\delta}}(L+1)\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right) ≤ 2 a δ e L a δ ( L + 1 ) Φ ( − r j λ δ )
≤ a δ = 1 64 ( 1 + L ) 1 16 Φ ( − λ δ r j ) . \underset{a\sqrt{\delta}=\frac{1}{64(1+L)}}{\le}\frac{1}{16}\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right). a δ = 64 ( 1 + L ) 1 ≤ 16 1 Φ ( − r j λ δ ) .
On the other hand, if λ δ r j > L \frac{\lambda\sqrt{\delta}}{r_j} > L r j λ δ > L since a δ ≤ λ δ r j a\sqrt{\delta} \le\frac{\lambda\sqrt{\delta}}{r_j} a δ ≤ r j λ δ , then
P ( ∣ Z − λ δ r j ∣ ≤ a δ ) ≤ 2 a δ ϕ ( L − a δ ) . \mathbb{P}\left(\left|Z-\frac{\lambda\sqrt{\delta}}{r_j}\right|\le a\sqrt{\delta}\right)\le2a\sqrt{\delta}\phi(L-a\sqrt{\delta}). P ( Z − r j λ δ ≤ a δ ) ≤ 2 a δ ϕ ( L − a δ ) .
Consequently, using the union bound over the last update time and the two possible signs,
P ( 0 < ∣ w t ∣ ≤ a ∣ w ∗ = 0 ) ≤ 2 ∑ j = 1 t P ( ∣ Z − λ δ r j ∣ ≤ a δ ) ≤ 1 16 ∑ { j : λ δ r j ≤ L } 2 Φ ( − λ δ r j ) + 4 t a δ ϕ ( L − a δ ) ≤ 1 16 ∑ j = 1 t 2 Φ ( − λ δ r j ) + 4 t a δ 2 π e L a δ − ( a δ ) 2 2 e − L 2 2 ≤ c i t e p ‘ [ 1 ] ‘ 1 16 ∑ j = 1 t 2 Φ ( − λ δ r j ) + a δ 2 π e L a δ − ( a δ ) 2 2 ∑ j = 1 t 2 Φ ( − λ δ r j ) ≤ 1 16 ∑ j = 1 t 2 Φ ( − λ δ r j ) + 1 16 ∑ j = 1 t 2 Φ ( − λ δ r j ) = 1 8 ∑ j = 1 t 2 Φ ( − λ δ r j ) , \begin{aligned}
\mathbb{P}(0<|w_t|\le a\mid w^*=0)
&\le2\sum_{j=1}^{t}\mathbb{P}\left(\left|Z-\frac{\lambda\sqrt{\delta}}{r_j}\right|\le a\sqrt{\delta}\right)\\
&\le\frac{1}{16}\sum_{\left\{j:\frac{\lambda\sqrt{\delta}}{r_j}\le L\right\}}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)+4ta\sqrt{\delta}\phi(L-a\sqrt{\delta})\\
&\le\frac{1}{16}\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)+\frac{4ta\sqrt{\delta}}{\sqrt{2\pi}}e^{La\sqrt{\delta}-\frac{(a\sqrt{\delta})^2}{2}}e^{-\frac{L^2}{2}}\\
&\stackrel{{citep}`[1]`}{\le}\frac{1}{16}\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)+\frac{a\sqrt{\delta}}{\sqrt{2\pi}}e^{La\sqrt{\delta}-\frac{(a\sqrt{\delta})^2}{2}}\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\\
&\le\frac{1}{16}\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)+\frac{1}{16}\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)
=\frac{1}{8}\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right),
\end{aligned} P ( 0 < ∣ w t ∣ ≤ a ∣ w ∗ = 0 ) ≤ 2 j = 1 ∑ t P ( Z − r j λ δ ≤ a δ ) ≤ 16 1 { j : r j λ δ ≤ L } ∑ 2Φ ( − r j λ δ ) + 4 t a δ ϕ ( L − a δ ) ≤ 16 1 j = 1 ∑ t 2Φ ( − r j λ δ ) + 2 π 4 t a δ e L a δ − 2 ( a δ ) 2 e − 2 L 2 ≤ c i t e p ‘ [ 1 ] ‘ 16 1 j = 1 ∑ t 2Φ ( − r j λ δ ) + 2 π a δ e L a δ − 2 ( a δ ) 2 j = 1 ∑ t 2Φ ( − r j λ δ ) ≤ 16 1 j = 1 ∑ t 2Φ ( − r j λ δ ) + 16 1 j = 1 ∑ t 2Φ ( − r j λ δ ) = 8 1 j = 1 ∑ t 2Φ ( − r j λ δ ) ,
where [1] follows from e − L 2 2 = ∑ j = 1 t 2 Φ ( − λ δ r j ) 4 t e^{-\frac{L^2}{2}}=\frac{\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)}{4t} e − 2 L 2 = 4 t ∑ j = 1 t 2Φ ( − r j λ δ ) . Therefore P ( 0 < ∣ w t ∣ ≤ a ∣ w ∗ = 0 ) ≤ 1 8 ∑ j = 1 t 2 Φ ( − λ δ r j ) \mathbb{P}(0<|w_t|\le a\mid w^*=0)\le\frac{1}{8}\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right) P ( 0 < ∣ w t ∣ ≤ a ∣ w ∗ = 0 ) ≤ 8 1 ∑ j = 1 t 2Φ ( − r j λ δ ) . By Eq. (67) it follows that P ( 0 < ∣ w t ∣ ≤ a ∣ w ∗ = 0 ) ≤ 1 4 P ( w t ≠ 0 ∣ w ∗ = 0 ) \mathbb{P}(0<|w_t|\le a\mid w^*=0)\le\frac{1}{4}\mathbb{P}(w_t\ne0\mid w^*=0) P ( 0 < ∣ w t ∣ ≤ a ∣ w ∗ = 0 ) ≤ 4 1 P ( w t = 0 ∣ w ∗ = 0 ) , so
P ( ∣ w t ∣ > a ∣ w ∗ = 0 ) = P ( w t ≠ 0 ∣ w ∗ = 0 ) − P ( 0 < ∣ w t ∣ ≤ a ∣ w ∗ = 0 ) ≥ P ( w t ≠ 0 ∣ w ∗ = 0 ) − 1 4 P ( w t ≠ 0 ∣ w ∗ = 0 ) ≥ 1 2 P ( w t ≠ 0 ∣ w ∗ = 0 ) \begin{aligned}
\mathbb{P}(|w_t|>a\mid w^*=0)=\mathbb{P}(w_t\ne0\mid w^*=0)-\mathbb{P}(0<|w_t|\le a\mid w^*=0) \\
&\ge\mathbb{P}(w_t\ne0\mid w^*=0)-\frac{1}{4}\mathbb{P}(w_t\ne0\mid w^*=0)\ge\frac{1}{2}\mathbb{P}(w_t\ne0\mid w^*=0)
\end{aligned} P ( ∣ w t ∣ > a ∣ w ∗ = 0 ) = P ( w t = 0 ∣ w ∗ = 0 ) − P ( 0 < ∣ w t ∣ ≤ a ∣ w ∗ = 0 ) ≥ P ( w t = 0 ∣ w ∗ = 0 ) − 4 1 P ( w t = 0 ∣ w ∗ = 0 ) ≥ 2 1 P ( w t = 0 ∣ w ∗ = 0 )
Therefore by Markov’s inequality,
ν 2 ( α t − r ) 2 + σ t 2 = E [ ( w t − r w ∗ ) 2 ] ≥ ( 1 − s ) E [ w t 2 ∣ w ∗ = 0 ] ≥ ( 1 − s ) a 2 P ( ∣ w t ∣ > a ∣ w ∗ = 0 ) . \nu^2(\alpha_t-r)^2+\sigma_t^2=\mathbb{E}\left[(w_t-rw^*)^2\right]\ge(1-s)\mathbb{E}\left[w_t^2\mid w^*=0\right]\ge(1-s)a^2\mathbb{P}(|w_t|>a\mid w^*=0). ν 2 ( α t − r ) 2 + σ t 2 = E [ ( w t − r w ∗ ) 2 ] ≥ ( 1 − s ) E [ w t 2 ∣ w ∗ = 0 ] ≥ ( 1 − s ) a 2 P ( ∣ w t ∣ > a ∣ w ∗ = 0 ) .
Using ( 1 + 2 x ) 2 ≤ 10 x (1+\sqrt{2x})^2\le10x ( 1 + 2 x ) 2 ≤ 10 x for all x ≥ 1 x\ge1 x ≥ 1 , then since log ( 4 T P ( w t ≠ 0 ∣ w ∗ = 0 ) ) ≥ 1 \log\left(\frac{4T}{\mathbb{P}(w_t\ne0\mid w^*=0)}\right)\ge1 log ( P ( w t = 0 ∣ w ∗ = 0 ) 4 T ) ≥ 1 ,
ν 2 ( α t − r ) 2 + σ t 2 ≥ 3 ( 1 − s ) P ( w t ≠ 0 ∣ w ∗ = 0 ) 163840 δ log ( 4 T P ( w t ≠ 0 ∣ w ∗ = 0 ) ) . (68) \nu^2(\alpha_t-r)^2+\sigma_t^2\ge\frac{3(1-s)\mathbb{P}(w_t\ne0\mid w^*=0)}{163840\delta\log\left(\frac{4T}{\mathbb{P}(w_t\ne0\mid w^*=0)}\right)}. \tag*{(68)} ν 2 ( α t − r ) 2 + σ t 2 ≥ 163840 δ log ( P ( w t = 0 ∣ w ∗ = 0 ) 4 T ) 3 ( 1 − s ) P ( w t = 0 ∣ w ∗ = 0 ) . (68)
If P ( w t ≠ 0 ∣ w ∗ = 0 ) ≤ s \mathbb{P}(w_t\ne0\mid w^*=0)\le s P ( w t = 0 ∣ w ∗ = 0 ) ≤ s , there is nothing to prove. Otherwise,
log ( 4 T P ( w t ≠ 0 ∣ w ∗ = 0 ) ) ≤ C ( log ( 1 s ) + log ( 2 T ) ) . \log\left(\frac{4T}{\mathbb{P}(w_t\ne0\mid w^*=0)}\right)\le C\left(\log\left(\frac{1}{s}\right)+\log(2T)\right). log ( P ( w t = 0 ∣ w ∗ = 0 ) 4 T ) ≤ C ( log ( s 1 ) + log ( 2 T ) ) .
Thus Eqs. (64) and (68) imply
P ( w t ≠ 0 ∣ w ∗ = 0 ) ≤ s + C C κ , s log 3 ( 1 s ) log ( 1 s ) + log ( 2 T ) ≤ C s log 2 ( 1 s ) . (69) \mathbb{P}(w_t\ne0\mid w^*=0)\le s+C C_{\kappa,s}\frac{\log^3\left(\frac{1}{s}\right)}{\log\left(\frac{1}{s}\right)+\log(2T)}\le Cs\log^2\left(\frac{1}{s}\right). \tag*{(69)} P ( w t = 0 ∣ w ∗ = 0 ) ≤ s + C C κ , s log ( s 1 ) + log ( 2 T ) log 3 ( s 1 ) ≤ C s log 2 ( s 1 ) . (69)
Combining this with Eq. (67) , we get log [ ( ∑ j = 1 t 2 Φ ( − λ δ r j ) ) − 1 ] ≥ log ( 1 s ) − 2 log log ( 1 s ) − log ( C ′ ) \log\left[\left(\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^{-1}\right]\ge\log\left(\frac{1}{s}\right)-2\log\log\left(\frac{1}{s}\right)-\log(C') log [ ( ∑ j = 1 t 2Φ ( − r j λ δ ) ) − 1 ] ≥ log ( s 1 ) − 2 log log ( s 1 ) − log ( C ′ ) , therefore,
log [ ( ∑ j = 1 t 2 Φ ( − λ δ r j ) ) − 1 ] ≥ 1 2 log ( 1 s ) (70) \log\left[\left(\sum_{j=1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^{-1}\right]\ge\frac{1}{2}\log\left(\frac{1}{s}\right) \tag*{(70)} log ( j = 1 ∑ t 2Φ ( − r j λ δ ) ) − 1 ≥ 2 1 log ( s 1 ) (70)
for a sufficiently small s s s . For a null coordinate, any update implies w t ≠ 0 w_t \ne0 w t = 0 almost surely. Hence
P ( w t ≠ w t − 1 ) ≤ s + P ( w t ≠ 0 ∣ w ⋆ = 0 ) ≤ C s log 2 ( 1 s ) . \mathbb{P}(w_t \ne w_{t-1}) \le s + \mathbb{P}(w_t \ne0 \mid w^\star= 0) \le Cs\log^2\left(\frac{1}{s}\right). P ( w t = w t − 1 ) ≤ s + P ( w t = 0 ∣ w ⋆ = 0 ) ≤ C s log 2 ( s 1 ) .
Stein’s Lemma gives γ t σ t τ t = 1 δ P ( w t ≠ w t − 1 ) \frac{\gamma_t}{\sigma_t\tau_t} = \frac{1}{\delta}\mathbb{P}(w_t \ne w_{t-1}) σ t τ t γ t = δ 1 P ( w t = w t − 1 ) . By the proximal relation (Eq. (65) ) we have ν α t Z 1 + σ t Z 2 ≥ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) \nu\alpha_t Z_1 + \sigma_t Z_2 \ge\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1 + \sigma_t Z_2) ν α t Z 1 + σ t Z 2 ≥ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) . For the logistic sigmoid,
0 ≤ ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≤ 2 γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) . (71) 0 \le\phi'(\nu\alpha_t Z_1 + \sigma_t Z_2) - \phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1 + \sigma_t Z_2)) \le\frac{2\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1 + \sigma_t Z_2))}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1 + \sigma_t Z_2))}. \tag*{(71)} 0 ≤ ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≤ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 2 γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) . (71)
Indeed, if γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≥ 1 \gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)) \ge1 γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≥ 1 this is trivial, because 0 < ϕ ′ < 1 0 < \phi' < 1 0 < ϕ ′ < 1 . If γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) < 1 \gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)) < 1 γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) < 1 and ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≥ 1 2 \phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)) \ge\frac{1}{2} ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≥ 2 1 , so ν α t Z 1 + σ t Z 2 ≥ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ≥ 0 \nu\alpha_t Z_1+\sigma_t Z_2 \ge\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2) \ge0 ν α t Z 1 + σ t Z 2 ≥ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ≥ 0 by concavity of the sigmoid (ϕ ′ ′ ′ ( v ) = ϕ ′ ′ ( v ) ( 1 − 2 ϕ ′ ( v ) ) ≤ 0 \phi'''(v)=\phi''(v)(1-2\phi'(v))\le0 ϕ ′′′ ( v ) = ϕ ′′ ( v ) ( 1 − 2 ϕ ′ ( v )) ≤ 0 for v ≥ 0 v\ge0 v ≥ 0 ), with the proximal equation, gives
ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≤ γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ≤ γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) . \begin{aligned}
\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)-\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))
&\le\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\frac{\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}{\nu\alpha_t Z_1+\sigma_t Z_2-\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)}\\
&\le\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)).
\end{aligned} ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≤ γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≤ γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) .
If ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) < 1 2 \phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)) < \frac{1}{2} ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) < 2 1 , then
ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) = γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≤ 2 γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) , \begin{aligned}
\nu\alpha_t Z_1+\sigma_t Z_2-\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)
&=\gamma_t\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\\
&=\frac{\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}{1-\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}\\
&\le2\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)),
\end{aligned} ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) = γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) = 1 − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≤ 2 γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ,
and the 1 4 \frac{1}{4} 4 1 -Lipschitz property of ϕ ′ \phi' ϕ ′ gives the same conclusion. From symmetry,
2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) + ϕ ′ ( − ν α t Z 1 − σ t Z 2 ) ) ] = E [ π ρ ′ ( ν Z 1 ) ] = ( 1 − 2 ρ ) h ( ν ) . 2\mathbb{E}[\pi'_\rho(\nu Z_1)\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)]
=\mathbb{E}[\pi'_\rho(\nu Z_1)(\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)+\phi'(-\nu\alpha_t Z_1-\sigma_t Z_2))]
=\mathbb{E}[\pi'_\rho(\nu Z_1)]
=(1-2\rho)h(\nu). 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 )] = E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) + ϕ ′ ( − ν α t Z 1 − σ t Z 2 ))] = E [ π ρ ′ ( ν Z 1 )] = ( 1 − 2 ρ ) h ( ν ) .
Therefore, since π ρ ′ ( u ) ≤ 1 − 2 ρ 4 \pi'_\rho(u)\le\frac{1-2\rho}{4} π ρ ′ ( u ) ≤ 4 1 − 2 ρ , and by Eq. (50) θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] \theta_t=2\mathbb{E}[\pi'_\rho(\nu Z_1)\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))] θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ))] with the previous inequality,
0 ≤ ( 1 − 2 ρ ) h ( ν ) − θ t = 2 E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ) ] ≤ ( 1 − 2 ρ ) E [ γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . (72) 0\le(1-2\rho)h(\nu)-\theta_t
=2\mathbb{E}[\pi'_\rho(\nu Z_1)(\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)-\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)))]
\le(1-2\rho)\mathbb{E}\left[\frac{\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}\right]. \tag*{(72)} 0 ≤ ( 1 − 2 ρ ) h ( ν ) − θ t = 2 E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )))] ≤ ( 1 − 2 ρ ) E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] . (72)
The sixth scalar equation, along with the equation for γ t \gamma_t γ t and Stein’s Lemma (note that ∂ w t ∂ Z t = σ t τ t r t δ 1 { w t ≠ w t − 1 } \frac{\partial w_t}{\partial Z_t}=\frac{\sigma_t\tau_t r_t}{\sqrt{\delta}}\mathbf{1}\{w_t\ne w_{t-1}\} ∂ Z t ∂ w t = δ σ t τ t r t 1 { w t = w t − 1 } ) then γ t = 1 r t δ E [ Z t w t ] = 1 r t δ σ t τ t r t δ P ( w t ≠ w t − 1 ) = σ t τ t δ P ( w t ≠ w t − 1 ) \gamma_t=\frac{1}{r_t\sqrt{\delta}}\mathbb{E}[Z_t w_t]=\frac{1}{r_t\sqrt{\delta}}\frac{\sigma_t\tau_t r_t}{\sqrt{\delta}}\mathbb{P}(w_t\ne w_{t-1})=\frac{\sigma_t\tau_t}{\delta}\mathbb{P}(w_t\ne w_{t-1}) γ t = r t δ 1 E [ Z t w t ] = r t δ 1 δ σ t τ t r t P ( w t = w t − 1 ) = δ σ t τ t P ( w t = w t − 1 ) , this gives,
1 δ P ( w t ≠ w t − 1 ) = γ t σ t τ t = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . \frac{1}{\delta}\mathbb{P}(w_t\ne w_{t-1})
=\frac{\gamma_t}{\sigma_t\tau_t}
=\mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))\frac{\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}\right]. δ 1 P ( w t = w t − 1 ) = σ t τ t γ t = E [ 2 ( 1 − π ρ ( ν Z 1 )) 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] .
From Eq. (71) ,
0 ≤ ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≤ 2 γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) . 0\le\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)-\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\le\frac{2\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}. 0 ≤ ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≤ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 2 γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) .
By plugging this upper bound we get ( 1 − 2 ρ ) h ( ν ) − θ t ≤ E [ 4 π ρ ′ ( ν Z 1 ) γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] (1-2\rho)h(\nu)-\theta_t\le\mathbb{E}\left[\frac{4\pi'_\rho(\nu Z_1)\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}\right] ( 1 − 2 ρ ) h ( ν ) − θ t ≤ E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) 4 π ρ ′ ( ν Z 1 ) γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] . Now we bound π ρ ′ ( ν Z 1 ) \pi'_\rho(\nu Z_1) π ρ ′ ( ν Z 1 ) from above, by definition π ρ ′ ( ν Z 1 ) = ( 1 − 2 ρ ) ϕ ′ ( ν Z 1 ) ( 1 − ϕ ′ ( ν Z 1 ) ) \pi'_\rho(\nu Z_1)=(1-2\rho)\phi'(\nu Z_1)(1-\phi'(\nu Z_1)) π ρ ′ ( ν Z 1 ) = ( 1 − 2 ρ ) ϕ ′ ( ν Z 1 ) ( 1 − ϕ ′ ( ν Z 1 )) , 1 − π ρ ( ν Z 1 ) = ρ + ( 1 − 2 ρ ) ( 1 − ϕ ′ ( ν Z 1 ) ) 1-\pi_\rho(\nu Z_1)=\rho+(1-2\rho)(1-\phi'(\nu Z_1)) 1 − π ρ ( ν Z 1 ) = ρ + ( 1 − 2 ρ ) ( 1 − ϕ ′ ( ν Z 1 )) . By these, it follows that for every 0 ≤ ρ ≤ 1 2 0\le\rho\le\frac{1}{2} 0 ≤ ρ ≤ 2 1 ,
ρ + ( 1 − 2 ρ ) ( 1 − ϕ ′ ( ν Z 1 ) ) − ϕ ′ ( ν Z 1 ) ( 1 − ϕ ′ ( ν Z 1 ) ) = ( 1 − ϕ ′ ( ν Z 1 ) ) ( 1 − ϕ ′ ( ν Z 1 ) − 2 ρ ) + ρ = ( 1 − ϕ ′ ( ν Z 1 ) − ρ ) 2 + ρ ( 1 − ρ ) ≥ 0. \begin{aligned}
\rho+(1-2\rho)(1-\phi'(\nu Z_1))-\phi'(\nu Z_1)(1-\phi'(\nu Z_1))
&=(1-\phi'(\nu Z_1))(1-\phi'(\nu Z_1)-2\rho)+\rho\\
&=(1-\phi'(\nu Z_1)-\rho)^2+\rho(1-\rho)\ge0.
\end{aligned} ρ + ( 1 − 2 ρ ) ( 1 − ϕ ′ ( ν Z 1 )) − ϕ ′ ( ν Z 1 ) ( 1 − ϕ ′ ( ν Z 1 )) = ( 1 − ϕ ′ ( ν Z 1 )) ( 1 − ϕ ′ ( ν Z 1 ) − 2 ρ ) + ρ = ( 1 − ϕ ′ ( ν Z 1 ) − ρ ) 2 + ρ ( 1 − ρ ) ≥ 0.
and therefore,
( 1 − 2 ρ ) ϕ ′ ( ν Z 1 ) ( 1 − ϕ ′ ( ν Z 1 ) ) ≤ ( 1 − 2 ρ ) ( ρ + ( 1 − 2 ρ ) ( 1 − ϕ ′ ( ν Z 1 ) ) ) . (1-2\rho)\phi'(\nu Z_1)(1-\phi'(\nu Z_1)) \le(1-2\rho)\left(\rho+(1-2\rho)(1-\phi'(\nu Z_1))\right). ( 1 − 2 ρ ) ϕ ′ ( ν Z 1 ) ( 1 − ϕ ′ ( ν Z 1 )) ≤ ( 1 − 2 ρ ) ( ρ + ( 1 − 2 ρ ) ( 1 − ϕ ′ ( ν Z 1 )) ) .
So,
( 1 − 2 ρ ) h ( ν ) − θ t ≤ 4 ( 1 − 2 ρ ) E [ ( 1 − π ρ ( ν Z 1 ) ) γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] = 2 ( 1 − 2 ρ ) γ t σ t τ t . (73) (1-2\rho)h(\nu)-\theta_t \le4(1-2\rho)\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\frac{\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}\right]=2(1-2\rho)\frac{\gamma_t}{\sigma_t\tau_t}. \tag*{(73)} ( 1 − 2 ρ ) h ( ν ) − θ t ≤ 4 ( 1 − 2 ρ ) E [ ( 1 − π ρ ( ν Z 1 )) 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] = 2 ( 1 − 2 ρ ) σ t τ t γ t . (73)
Combining the null-coordinate support bound with { w t ≠ w t − 1 , w ⋆ = 0 } ⊆ { w t ≠ 0 , w ⋆ = 0 } \{w_t\ne w_{t-1},w^\star=0\}\subseteq\{w_t\ne0,w^\star=0\} { w t = w t − 1 , w ⋆ = 0 } ⊆ { w t = 0 , w ⋆ = 0 } , almost surely, we obtain
P ( w t ≠ w t − 1 ) ≤ s + P ( w t ≠ 0 ∣ w ⋆ = 0 ) ≤ C s log 2 ( 1 s ) . \mathbb{P}(w_t\ne w_{t-1})\le s+\mathbb{P}(w_t\ne0\mid w^\star=0)\le Cs\log^2\left(\frac{1}{s}\right). P ( w t = w t − 1 ) ≤ s + P ( w t = 0 ∣ w ⋆ = 0 ) ≤ C s log 2 ( s 1 ) .
Since the contradiction argument above implies λ → ∞ \lambda\to\infty λ → ∞ , for all sufficiently small s s s we have C s log 2 ( 1 s ) ≤ λ s Cs\log^2\left(\frac{1}{s}\right)\le\lambda\sqrt{s} C s log 2 ( s 1 ) ≤ λ s . Therefore P ( w t ≠ w t − 1 ) ≤ λ s \mathbb{P}(w_t\ne w_{t-1})\le\lambda\sqrt{s} P ( w t = w t − 1 ) ≤ λ s . By Eq. (73) and 1 2 P ( w t ≠ w t − 1 ) = γ t σ t τ t \frac{1}{2}\mathbb{P}(w_t\ne w_{t-1})=\frac{\gamma_t}{\sigma_t\tau_t} 2 1 P ( w t = w t − 1 ) = σ t τ t γ t , we have 1 2 ( 1 − 2 ρ ) h ( ν ) ≤ θ t \frac{1}{2}(1-2\rho)h(\nu)\le\theta_t 2 1 ( 1 − 2 ρ ) h ( ν ) ≤ θ t uniformly in t t t for a sufficiently small s s s . Also using Eq. (50)$,
θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ≤ 1 − 2 ρ 2 E [ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . \theta_t=2\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)\right]\le\frac{1-2\rho}{2}\mathbb{E}\left[\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)\right]. θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ≤ 2 1 − 2 ρ E [ ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] .
Thus, since 1 2 ( 1 − 2 ρ ) h ( ν ) ≤ θ t \frac{1}{2}(1-2\rho)h(\nu)\le\theta_t 2 1 ( 1 − 2 ρ ) h ( ν ) ≤ θ t , and θ t = 2 ( 1 − 2 ρ ) E [ ϕ ′ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] \theta_t=2(1-2\rho)\mathbb{E}[\phi''(\nu Z_1)\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2))] θ t = 2 ( 1 − 2 ρ ) E [ ϕ ′′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ))] , hence for 0 < ρ ≤ 1 2 0<\rho\le\frac{1}{2} 0 < ρ ≤ 2 1 ,
E [ ϕ ′ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ≥ 1 4 h ( ν ) . \mathbb{E}\left[\phi''(\nu Z_1)\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)\right]\ge\frac{1}{4}h(\nu). E [ ϕ ′′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ≥ 4 1 h ( ν ) .
Now, we have uniformly for all 0 ≤ ρ ≤ 1 2 0\le\rho\le\frac{1}{2} 0 ≤ ρ ≤ 2 1 , 1 − π ρ ( u ) ≥ ϕ ′ ′ ( u ) 1-\pi_\rho(u)\ge\phi''(u) 1 − π ρ ( u ) ≥ ϕ ′′ ( u ) since 1 − π ρ ( u ) − ϕ ′ ′ ( u ) = ρ + ( 1 − 2 ρ ) ( 1 − ϕ ′ ( u ) ) − ϕ ′ ( u ) ( 1 − ϕ ′ ( u ) ) = ( 1 − ϕ ′ ( u ) − ρ ) 2 + ρ ( 1 − 2 ρ ) ≥ 0 1-\pi_\rho(u)-\phi''(u)=\rho+(1-2\rho)(1-\phi'(u))-\phi'(u)(1-\phi'(u))=(1-\phi'(u)-\rho)^2+\rho(1-2\rho)\ge0 1 − π ρ ( u ) − ϕ ′′ ( u ) = ρ + ( 1 − 2 ρ ) ( 1 − ϕ ′ ( u )) − ϕ ′ ( u ) ( 1 − ϕ ′ ( u )) = ( 1 − ϕ ′ ( u ) − ρ ) 2 + ρ ( 1 − 2 ρ ) ≥ 0 as we have already established. Hence
r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] ≥ 2 E [ ϕ ′ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] ≥ 2 ( E [ ϕ ′ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ) 2 E [ ϕ ′ ′ ( ν Z 1 ) ] = 2 ( 1 4 h ( ν ) ) 2 h ( ν ) = h ( ν ) 8 . (74) \begin{aligned}
r_t^2
&=2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)^2\right]\\
&\ge2\mathbb{E}\left[\phi''(\nu Z_1)\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)^2\right]\\
&\ge\frac{2\left(\mathbb{E}\left[\phi''(\nu Z_1)\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)\right]\right)^2}{\mathbb{E}[\phi''(\nu Z_1)]}\\
&=\frac{2\left(\frac{1}{4}h(\nu)\right)^2}{h(\nu)}=\frac{h(\nu)}{8}. \tag*{(74)}
\end{aligned} r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] ≥ 2 E [ ϕ ′′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] ≥ E [ ϕ ′′ ( ν Z 1 )] 2 ( E [ ϕ ′′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ) 2 = h ( ν ) 2 ( 4 1 h ( ν ) ) 2 = 8 h ( ν ) . (74)
for every t ≤ T t\le T t ≤ T . We also have the universal upper bound 0 ≤ θ t ≤ ( 1 − 2 ρ ) h ( ν ) 0\le\theta_t\le(1-2\rho)h(\nu) 0 ≤ θ t ≤ ( 1 − 2 ρ ) h ( ν ) . Indeed Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ≤ ν α t Z 1 + σ t Z 2 \operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\le\nu\alpha_tZ_1+\sigma_tZ_2 Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ≤ ν α t Z 1 + σ t Z 2 by the prox relation Eq. (65) , and from symmetry
2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = ( 1 − 2 ρ ) h ( ν ) . 2\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'(\nu\alpha_tZ_1+\sigma_tZ_2)\right]=(1-2\rho)h(\nu). 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = ( 1 − 2 ρ ) h ( ν ) .
The bound in Eq. (70) implies two useful events. First,
P ( ∀ j ≤ T : ∣ Z j ∣ ≤ λ δ r j − log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] 4 log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ) ≥ 1 2 . (75) \mathbb{P}\left(\forall j\le T:|Z_j|\le\frac{\lambda\sqrt{\delta}}{r_j}-\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{4\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}\right)\ge\frac{1}{2}. \tag*{(75)} P ∀ j ≤ T : ∣ Z j ∣ ≤ r j λ δ − 4 log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ≥ 2 1 . (75)
To verify this, for q > 0 q>0 q > 0 and a ∈ ( 0 , 1 ) a\in(0,1) a ∈ ( 0 , 1 ) , Lemma 24 gives Φ ( − ( x − q ) ) ≤ exp ( a q 2 2 ( 1 − a ) ) Φ ( − x ) a \Phi(-(x-q))\le\exp\left(\frac{aq^2}{2(1-a)}\right)\Phi(-x)^a Φ ( − ( x − q )) ≤ exp ( 2 ( 1 − a ) a q 2 ) Φ ( − x ) a . Note that,
P ( ∃ j ≤ T : ∣ Z j ∣ > λ δ r j − q ) ≤ ∑ j = 1 T 2 Φ ( − ( λ δ r j − q ) ) . (76) \mathbb{P}\left(\exists j\le T:|Z_j|>\frac{\lambda\sqrt{\delta}}{r_j}-q\right)\le\sum_{j=1}^{T}2\Phi\left(-\left(\frac{\lambda\sqrt{\delta}}{r_j}-q\right)\right). \tag*{(76)} P ( ∃ j ≤ T : ∣ Z j ∣ > r j λ δ − q ) ≤ j = 1 ∑ T 2Φ ( − ( r j λ δ − q ) ) . (76)
Summing over j j j and using Jensen’s inequality,
1 T ∑ j = 1 T 2 Φ ( − λ δ r j ) a ≤ ( 1 T ∑ j = 1 T 2 Φ ( − λ δ r j ) ) a = T 1 − a ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) a , \frac{1}{T}\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)^a\le\left(\frac{1}{T}\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^a=T^{1-a}\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^a, T 1 j = 1 ∑ T 2Φ ( − r j λ δ ) a ≤ ( T 1 j = 1 ∑ T 2Φ ( − r j λ δ ) ) a = T 1 − a ( j = 1 ∑ T 2Φ ( − r j λ δ ) ) a ,
From that inequality, starting from the quantity in Eq. (76) ,
∑ j = 1 T 2 Φ ( − ( λ δ r j − q ) ) ≤ 2 exp ( a q 2 2 ( 1 − a ) ) ∑ j = 1 T Φ ( − λ δ r j ) a = 2 1 − a exp ( a q 2 2 ( 1 − a ) ) ∑ j = 1 T 2 Φ ( − λ δ r j ) a ≤ 2 1 − a exp ( a q 2 2 ( 1 − a ) ) T 1 − a ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) a \begin{aligned}
\sum_{j=1}^{T}2\Phi\left(-\left(\frac{\lambda\sqrt{\delta}}{r_j}-q\right)\right)\le2\exp\left(\frac{aq^2}{2(1-a)}\right)\sum_{j=1}^{T}\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)^a \\
&=2^{1-a}\exp\left(\frac{aq^2}{2(1-a)}\right)\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)^a\le2^{1-a}\exp\left(\frac{aq^2}{2(1-a)}\right)T^{1-a}\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^a
\end{aligned} j = 1 ∑ T 2Φ ( − ( r j λ δ − q ) ) ≤ 2 exp ( 2 ( 1 − a ) a q 2 ) j = 1 ∑ T Φ ( − r j λ δ ) a = 2 1 − a exp ( 2 ( 1 − a ) a q 2 ) j = 1 ∑ T 2Φ ( − r j λ δ ) a ≤ 2 1 − a exp ( 2 ( 1 − a ) a q 2 ) T 1 − a ( j = 1 ∑ T 2Φ ( − r j λ δ ) ) a
Taking logarithm,
log ( 2 1 − a exp ( a q 2 2 ( 1 − a ) ) T 1 − a ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) a ) = ( 1 − a ) log ( 2 T ) + a q 2 2 ( 1 − a ) + a log ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) = log ( ∑ j = 1 T 2 Φ ( λ δ r j ) ) + ( 1 − a ) ( log ( 2 T ) − log ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) ) + a q 2 2 ( 1 − a ) . \begin{aligned}
\log\left(2^{1-a}\exp\left(\frac{aq^2}{2(1-a)}\right)T^{1-a}\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^a\right) \\
&= (1-a)\log(2T)+\frac{aq^2}{2(1-a)}+a\log\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right) \\
&= \log\left(\sum_{j=1}^{T}2\Phi\left(\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)+(1-a)\left(\log(2T)-\log\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)\right)+\frac{aq^2}{2(1-a)}.
\end{aligned} log ( 2 1 − a exp ( 2 ( 1 − a ) a q 2 ) T 1 − a ( j = 1 ∑ T 2Φ ( − r j λ δ ) ) a ) = ( 1 − a ) log ( 2 T ) + 2 ( 1 − a ) a q 2 + a log ( j = 1 ∑ T 2Φ ( − r j λ δ ) ) = log ( j = 1 ∑ T 2Φ ( r j λ δ ) ) + ( 1 − a ) ( log ( 2 T ) − log ( j = 1 ∑ T 2Φ ( − r j λ δ ) ) ) + 2 ( 1 − a ) a q 2 .
this is minimized with respect to a a a , if 1 − a = q 2 ( log ( 2 T ) + log ( ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) − 1 ) ) 1-a=\frac{q}{\sqrt{2\left(\log(2T)+\log\left(\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^{-1}\right)\right)}} 1 − a = 2 ( l o g ( 2 T ) + l o g ( ( ∑ j = 1 T 2Φ ( − r j λ δ ) ) − 1 ) ) q therefore the logarithm of
the resulting upper bound is at most
log ( P ( ∃ j ≤ T : ∣ Z j ∣ > λ δ r j − q ) ) ≤ − log [ ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) − 1 ] − q 2 2 + q 2 ( log ( 2 T ) + log [ ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) − 1 ] ) . \begin{aligned}
\log\left(\mathbb{P}\left(\exists j\leq T: |Z_j|>\frac{\lambda\sqrt{\delta}}{r_j}-q\right)\right) \\
&\leq-\log\left[\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^{-1}\right]-\frac{q^2}{2}+q\sqrt{2\left(\log(2T)+\log\left[\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^{-1}\right]\right)}.
\end{aligned} log ( P ( ∃ j ≤ T : ∣ Z j ∣ > r j λ δ − q ) ) ≤ − log ( j = 1 ∑ T 2Φ ( − r j λ δ ) ) − 1 − 2 q 2 + q 2 log ( 2 T ) + log ( j = 1 ∑ T 2Φ ( − r j λ δ ) ) − 1 .
Choosing q q q equal to the quantity in Eq. (75) makes this at most − ( 1 − 2 4 ) log [ ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) − 1 ] -\left(1-\frac{\sqrt{2}}{4}\right)\log\left[\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^{-1}\right] − ( 1 − 4 2 ) log [ ( ∑ j = 1 T 2Φ ( − r j λ δ ) ) − 1 ] , which tends to − ∞ -\infty − ∞ by Eq. (70) as s → 0 s\to0 s → 0 . Therefore, for a sufficiently small s s s , P ( ∃ j ≤ T : ∣ Z j ∣ > λ δ r j − q ) ≤ 1 2 \mathbb{P}\left(\exists j\leq T: |Z_j|>\frac{\lambda\sqrt{\delta}}{r_j}-q\right)\leq\frac{1}{2} P ( ∃ j ≤ T : ∣ Z j ∣ > r j λ δ − q ) ≤ 2 1 , taking the complement of the event proves the claim. Second,
P ( ∀ j ≤ T : ∣ Z j ∣ ≤ [ 1 − log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] 8 ( log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ) ] λ δ r j ) ≥ 1 2 . (77) \mathbb{P}\left(\forall j\leq T: |Z_j|\leq\left[1-\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{8\left(\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]\right)}\right]\frac{\lambda\sqrt{\delta}}{r_j}\right)\geq\frac{1}{2}. \tag*{(77)} P ∀ j ≤ T : ∣ Z j ∣ ≤ 1 − 8 ( log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ) log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] r j λ δ ≥ 2 1 . (77)
To show this we prove the following identity, for every x ≥ 0 x\geq0 x ≥ 0 and 7 8 ≤ a < 1 \frac{7}{8}\leq a<1 8 7 ≤ a < 1 , 2 Φ ( − a x ) ≤ C ( 2 Φ ( − x ) ) a 2 2\Phi(-ax)\leq C(2\Phi(-x))^{a^2} 2Φ ( − a x ) ≤ C ( 2Φ ( − x ) ) a 2 for a universal constant C > 0 C>0 C > 0 . Indeed, by Mill’s inequality φ ( y ) 1 + y ≤ Φ ( − y ) ≤ φ ( y ) y \frac{\varphi(y)}{1+y}\leq\Phi(-y)\leq\frac{\varphi(y)}{y} 1 + y φ ( y ) ≤ Φ ( − y ) ≤ y φ ( y ) for y ≥ 1 y\geq1 y ≥ 1 , φ ( y ) 1 + y 2 ≤ Φ ( − y ) ≤ φ ( y ) y \frac{\varphi(y)}{1+y^2}\leq\Phi(-y)\leq\frac{\varphi(y)}{y} 1 + y 2 φ ( y ) ≤ Φ ( − y ) ≤ y φ ( y ) for y ≥ 1 y\geq1 y ≥ 1 . For 0 ≤ y ≤ 1 0\leq y\leq1 0 ≤ y ≤ 1 , 2 Φ ( − y ) ≥ 2 Φ ( − 1 ) > 0 2\Phi(-y)\geq2\Phi(-1)>0 2Φ ( − y ) ≥ 2Φ ( − 1 ) > 0 , so the same inequality follows after increasing the universal constant C C C . Apply the upper bound with y = a x y=ax y = a x , 2 Φ ( − a x ) ≤ C e − a 2 x 2 / 2 1 + a x 2\Phi(-ax)\leq C\frac{e^{-a^2x^2/2}}{1+ax} 2Φ ( − a x ) ≤ C 1 + a x e − a 2 x 2 /2 , and apply the lower bound at y = x y=x y = x and raise it to the power of a 2 a^2 a 2 , ( 2 Φ ( − x ) ) a 2 ≥ c a 2 e − a 2 x 2 / 2 ( 1 + x ) a 2 (2\Phi(-x))^{a^2}\geq c^{a^2}\frac{e^{-a^2x^2/2}}{(1+x)^{a^2}} ( 2Φ ( − x ) ) a 2 ≥ c a 2 ( 1 + x ) a 2 e − a 2 x 2 /2 . Consequently, 2 Φ ( − a x ) ( 2 Φ ( − x ) ) a 2 ≤ C c − a 2 ( 1 + x ) a 2 1 + a x \frac{2\Phi(-ax)}{(2\Phi(-x))^{a^2}}\leq Cc^{-a^2}\frac{(1+x)^{a^2}}{1+ax} ( 2Φ ( − x ) ) a 2 2Φ ( − a x ) ≤ C c − a 2 1 + a x ( 1 + x ) a 2 . Since a ≤ 1 a\leq1 a ≤ 1 , ( 1 + x ) a 2 ≤ 1 + x (1+x)^{a^2}\leq1+x ( 1 + x ) a 2 ≤ 1 + x , and since a ≥ 7 8 a\geq\frac{7}{8} a ≥ 8 7 , 1 + a x ≥ a ( 1 + x ) ≥ 7 8 ( 1 + x ) 1+ax\geq a(1+x)\geq\frac{7}{8}(1+x) 1 + a x ≥ a ( 1 + x ) ≥ 8 7 ( 1 + x ) . Therefore, ( 1 + x ) a 2 1 + a x ≤ 1 a ≤ 8 7 \frac{(1+x)^{a^2}}{1+ax}\leq\frac{1}{a}\leq\frac{8}{7} 1 + a x ( 1 + x ) a 2 ≤ a 1 ≤ 7 8 . Also c − a 2 c^{-a^2} c − a 2 is uniformly bounded for a ∈ [ 7 8 , 1 ] a\in[\frac{7}{8},1] a ∈ [ 8 7 , 1 ] . Hence 2 Φ ( − a x ) ≤ C ( 2 Φ ( − x ) ) a 2 2\Phi(-ax)\leq C(2\Phi(-x))^{a^2} 2Φ ( − a x ) ≤ C ( 2Φ ( − x ) ) a 2 . Summing and using Jensen’s inequality,
∑ j = 1 T 2 Φ ( − a λ δ r j ) ≤ C T ( 1 T ∑ j = 1 T 2 Φ ( − λ δ r j ) ) a 2 ≤ C T 1 − a 2 ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) a 2 . \sum_{j=1}^{T}2\Phi\left(-a\frac{\lambda\sqrt{\delta}}{r_j}\right)\leq CT\left(\frac{1}{T}\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^{a^2}\leq CT^{1-a^2}\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^{a^2}. j = 1 ∑ T 2Φ ( − a r j λ δ ) ≤ C T ( T 1 j = 1 ∑ T 2Φ ( − r j λ δ ) ) a 2 ≤ C T 1 − a 2 ( j = 1 ∑ T 2Φ ( − r j λ δ ) ) a 2 .
With a a a equal to the multiplicative factor in Eq. (77) , the logarithm is at most − 3 4 log [ ( ∑ j = 1 T 2 Φ ( − λ δ r j ) ) − 1 ] -\frac{3}{4}\log\left[\left(\sum_{j=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)^{-1}\right] − 4 3 log [ ( ∑ j = 1 T 2Φ ( − r j λ δ ) ) − 1 ] , which tends to − ∞ -\infty − ∞ by Eq. (70) . Once again as before, a union bound proves Eq. (77) (note that 7 8 ≤ a ≤ 1 \frac{7}{8}\leq a\leq1 8 7 ≤ a ≤ 1 ). We now split into two cases.
Case 1: s ( log ( 1 s ) + log ( 2 T ) ) ≥ 1 s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)\ge1 s ( log ( s 1 ) + log ( 2 T ) ) ≥ 1 . On the event (74), using r t ≥ h ( ν ) 8 r_t\ge\sqrt{\frac{h(\nu)}{8}} r t ≥ 8 h ( ν ) (see Eq. (74) ),
r j δ ∣ Z j ∣ ≤ λ − h ( ν ) log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] 4 8 δ log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] . \frac{r_j}{\sqrt{\delta}}|Z_j|\le\lambda-\frac{\sqrt{h(\nu)}\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{4\sqrt{8\delta}\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}. δ r j ∣ Z j ∣ ≤ λ − 4 8 δ log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] h ( ν ) log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] .
Using 0 ≤ θ t ≤ ( 1 − 2 ρ ) h ( ν ) 0\le\theta_t\le(1-2\rho)h(\nu) 0 ≤ θ t ≤ ( 1 − 2 ρ ) h ( ν ) , every positive active coordinate satisfying
0 < w ⋆ ≤ 1 4 8 δ h ( ν ) ( 1 − 2 ρ ) log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] 0<w^\star\le\frac{1}{4\sqrt{8\delta h(\nu)}(1-2\rho)}
\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}} 0 < w ⋆ ≤ 4 8 δ h ( ν ) ( 1 − 2 ρ ) 1 log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ]
satisfies ∣ θ j w ⋆ + r j δ Z j ∣ < λ \left|\theta_jw^\star+\frac{r_j}{\sqrt{\delta}}Z_j\right|<\lambda θ j w ⋆ + δ r j Z j < λ for every j j j . Thus every projection interval contains zero. Since w 0 = 0 w_0=0 w 0 = 0 , then w j = 0 w_j=0 w j = 0 for all j ≤ T j\le T j ≤ T . Conditionally on coordinate being active, w ⋆ = ν s G w^\star=\frac{\nu}{\sqrt{s}}G w ⋆ = s ν G . Therefore, since we know this event happens with probability of at least 1 2 \frac{1}{2} 2 1 by Eq. (75) ,
ν 2 ( α t − r ) 2 + σ t 2 ≥ E [ ( w t − r w ⋆ ) 2 ] ≥ s 2 E [ r 2 ( ν s G ) 2 1 { 0 < G ≤ s 4 ν 8 δ h ( ν ) ( 1 − 2 ρ ) log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] } ] \begin{aligned}
\nu^2(\alpha_t-r)^2+\sigma_t^2\ge\mathbb{E}\left[(w_t-rw^\star)^2\right] \\
&\ge\frac{s}{2}\mathbb{E}\left[r^2\left(\frac{\nu}{\sqrt{s}}G\right)^2\mathbf{1}\left\{0<G\le\frac{\sqrt{s}}{4\nu\sqrt{8\delta h(\nu)}(1-2\rho)}
\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}\right\}\right]
\end{aligned} ν 2 ( α t − r ) 2 + σ t 2 ≥ E [ ( w t − r w ⋆ ) 2 ] ≥ 2 s E r 2 ( s ν G ) 2 1 ⎩ ⎨ ⎧ 0 < G ≤ 4 ν 8 δ h ( ν ) ( 1 − 2 ρ ) s log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ⎭ ⎬ ⎫
[!1]
≥ r 2 ν 2 2 min { − φ ( 1 ) + Φ ( 1 ) − Φ ( 0 ) , φ ( 1 ) 3 ( s 4 ν 8 δ h ( ν ) ( 1 − 2 ρ ) log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ) 3 } . \ge\frac{r^2\nu^2}{2}\min\left\{-\varphi(1)+\Phi(1)-\Phi(0),\frac{\varphi(1)}{3}\left(\frac{\sqrt{s}}{4\nu\sqrt{8\delta h(\nu)}(1-2\rho)}
\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}\right)^3\right\}. ≥ 2 r 2 ν 2 min ⎩ ⎨ ⎧ − φ ( 1 ) + Φ ( 1 ) − Φ ( 0 ) , 3 φ ( 1 ) 4 ν 8 δ h ( ν ) ( 1 − 2 ρ ) s log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] 3 ⎭ ⎬ ⎫ .
where (1) follows by,
E [ G 2 1 { 0 ≤ G ≤ u } ] = ∫ 0 u z 2 φ ( z ) d z ≥ { φ ( 1 ) ∫ 0 u z 2 d z = φ ( 1 ) 3 u 3 0 < u ≤ 1 ∫ 0 1 z 2 φ ( z ) d z = [ − z φ ( z ) ] 0 1 + ∫ 0 1 φ ( z ) d z = − φ ( 1 ) + Φ ( 1 ) − Φ ( 0 ) u ≥ 1 \begin{aligned}
\mathbb{E}\left[G^2\mathbf{1}_{\{0\le G\le u\}}\right]=\int_0^u z^2\varphi(z)\,dz \\
&\ge\begin{cases}
\varphi(1)\int_0^u z^2\,dz=\frac{\varphi(1)}{3}u^3 & 0<u\le1\\
\int_0^1z^2\varphi(z)\,dz=[-z\varphi(z)]_0^1+\int_0^1\varphi(z)\,dz=-\varphi(1)+\Phi(1)-\Phi(0) & u\ge1
\end{cases}
\end{aligned} E [ G 2 1 { 0 ≤ G ≤ u } ] = ∫ 0 u z 2 φ ( z ) d z ≥ { φ ( 1 ) ∫ 0 u z 2 d z = 3 φ ( 1 ) u 3 ∫ 0 1 z 2 φ ( z ) d z = [ − z φ ( z ) ] 0 1 + ∫ 0 1 φ ( z ) d z = − φ ( 1 ) + Φ ( 1 ) − Φ ( 0 ) 0 < u ≤ 1 u ≥ 1
Recall that we attained a lower bound on log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] \log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] in Eq. (70) . To replace this quantity with the lower bound, we define f ( x ) = x 3 ( log ( 2 T ) + x ) 3 2 f(x)=\frac{x^3}{(\log(2T)+x)^{\frac{3}{2}}} f ( x ) = ( l o g ( 2 T ) + x ) 2 3 x 3 , then
f ′ ( x ) = 3 x 2 ( log ( 2 T ) + x ) 3 2 − 3 2 x 3 ( log ( 2 T ) + x ) 1 2 ( log ( 2 T ) + x ) 3 = 3 x 2 log ( 2 T ) + 3 2 x 3 ( log ( 2 T ) + x ) 5 2 > 0 x > 0 f'(x)=\frac{3x^2(\log(2T)+x)^{\frac{3}{2}}-\frac{3}{2}x^3(\log(2T)+x)^{\frac{1}{2}}}{(\log(2T)+x)^3}
=\frac{3x^2\log(2T)+\frac{3}{2}x^3}{(\log(2T)+x)^{\frac{5}{2}}}>0\qquad x>0 f ′ ( x ) = ( log ( 2 T ) + x ) 3 3 x 2 ( log ( 2 T ) + x ) 2 3 − 2 3 x 3 ( log ( 2 T ) + x ) 2 1 = ( log ( 2 T ) + x ) 2 5 3 x 2 log ( 2 T ) + 2 3 x 3 > 0 x > 0
so the function is increasing for all x > 0 x>0 x > 0 , this justifies the replacement with the lower bound. For a sufficiently small s s s , the constant in the lower bound is harmless. Hence
ν 2 ( α t − r ) 2 + σ t 2 ≥ r 2 ν 2 φ ( 1 ) 6 ( s ν 1 4 8 δ h ( ν ) ( 1 − 2 ρ ) 1 2 log ( 1 s ) log ( 2 T ) + 1 2 log ( 1 s ) ) 3 \nu^2(\alpha_t-r)^2+\sigma_t^2\ge\frac{r^2\nu^2\varphi(1)}{6}\left(\frac{\sqrt{s}}{\nu}\frac{1}{4\sqrt{8\delta h(\nu)}(1-2\rho)}
\frac{\frac{1}{2}\log\left(\frac{1}{s}\right)}{\sqrt{\log(2T)+\frac{1}{2}\log\left(\frac{1}{s}\right)}}\right)^3 ν 2 ( α t − r ) 2 + σ t 2 ≥ 6 r 2 ν 2 φ ( 1 ) ν s 4 8 δ h ( ν ) ( 1 − 2 ρ ) 1 log ( 2 T ) + 2 1 log ( s 1 ) 2 1 log ( s 1 ) 3
But in the present case, s ( log ( 1 s ) + log ( 2 T ) ) ≥ 1 s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)\ge1 s ( log ( s 1 ) + log ( 2 T ) ) ≥ 1 , so
s 3 2 log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 3 2 = s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 2 s ( log ( 1 s ) + log ( 2 T ) ) ≥ s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 2 . s^{\frac{3}{2}}\frac{\log^{3}\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^{\frac{3}{2}}}=s\frac{\log^{3}\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^{2}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}\ge s\frac{\log^{3}\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^{2}}. s 2 3 ( log ( s 1 ) + log ( 2 T ) ) 2 3 log 3 ( s 1 ) = s ( log ( s 1 ) + log ( 2 T ) ) 2 log 3 ( s 1 ) s ( log ( s 1 ) + log ( 2 T ) ) ≥ s ( log ( s 1 ) + log ( 2 T ) ) 2 log 3 ( s 1 ) .
Therefore,
ν 2 ( α t − r ) 2 + σ t 2 ≥ r 2 φ ( 1 ) 384 8 ν δ 3 2 ( 1 − 2 ρ ) 3 1 ( log ( 1 s ) + log ( 2 T ) ) 3 s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 2 , \nu^{2}(\alpha_t-r)^{2}+\sigma_t^{2}\ge\frac{r^{2}\varphi(1)}{384\sqrt{8}\nu\,\delta^{\frac{3}{2}}(1-2\rho)^{3}}\frac{1}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^{3}}\frac{s\log^{3}\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^{2}}, ν 2 ( α t − r ) 2 + σ t 2 ≥ 384 8 ν δ 2 3 ( 1 − 2 ρ ) 3 r 2 φ ( 1 ) ( log ( s 1 ) + log ( 2 T ) ) 3 1 ( log ( s 1 ) + log ( 2 T ) ) 2 s log 3 ( s 1 ) ,
contradicting Eq. (64) if c ⋆ c_{\star} c ⋆ is small enough.
Case 2 s ( log ( 1 s ) + log ( 2 T ) ) < 1 s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)<1 s ( log ( s 1 ) + log ( 2 T ) ) < 1 .
We first show that λ s \lambda\sqrt{s} λ s must be small. On the event (76), every positive active coordinate satisfying
0 < w ⋆ ≤ λ ( 1 − 2 ρ ) h ( ν ) log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] 8 ( log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ) 0<w^{\star}\le\frac{\lambda}{(1-2\rho)h(\nu)}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{8\left(\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]\right)} 0 < w ⋆ ≤ ( 1 − 2 ρ ) h ( ν ) λ 8 ( log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ) log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ]
remains identically zero, because 0 ≤ θ t ≤ ( 1 − 2 ρ ) h ( ν ) 0\le\theta_t\le(1-2\rho)h(\nu) 0 ≤ θ t ≤ ( 1 − 2 ρ ) h ( ν ) and (77) imply ∣ θ j w ⋆ + r j δ Z j ∣ ≤ λ \left|\theta_jw^{\star}+\frac{r_j}{\sqrt{\delta}}Z_j\right|\le\lambda θ j w ⋆ + δ r j Z j ≤ λ for every j j j .
Integrating G 2 G^2 G 2 and repeating the same calculations from above
ν 2 ( α t − r ) 2 + σ t 2 = E [ ( w t − r w ⋆ ) 2 ] ≥ s 2 E [ r 2 ( ν s G ) 2 1 { 0 < G < s ν λ ( 1 − 2 ρ ) h ( ν ) log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] 8 ( log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ) } ] ≥ r 2 ν 2 2 min { − φ ( 1 ) + Φ ( 1 ) − Φ ( 0 ) , φ ( 1 ) 3 ( λ s ν ( 1 − 2 ρ ) h ( ν ) log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] 8 ( log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ) ) 3 } . (78) \begin{aligned}
\nu^{2}(\alpha_t-r)^{2}+\sigma_t^{2}&=\mathbb{E}\left[(w_t-rw^{\star})^{2}\right]\\
&\ge\frac{s}{2}\mathbb{E}\left[r^{2}\left(\frac{\nu}{\sqrt{s}}G\right)^{2}\mathbf{1}\left\{0<G<\frac{\sqrt{s}}{\nu}\frac{\lambda}{(1-2\rho)h(\nu)}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{8\left(\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]\right)}\right\}\right]\\
&\ge\frac{r^{2}\nu^{2}}{2}\min\left\{-\varphi(1)+\Phi(1)-\Phi(0),\right.\\
&\qquad\left.\frac{\varphi(1)}{3}\left(\frac{\lambda\sqrt{s}}{\nu(1-2\rho)h(\nu)}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{8\left(\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]\right)}\right)^{3}\right\}. \tag*{(78)}
\end{aligned} ν 2 ( α t − r ) 2 + σ t 2 = E [ ( w t − r w ⋆ ) 2 ] ≥ 2 s E r 2 ( s ν G ) 2 1 ⎩ ⎨ ⎧ 0 < G < ν s ( 1 − 2 ρ ) h ( ν ) λ 8 ( log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ) log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ⎭ ⎬ ⎫ ≥ 2 r 2 ν 2 min { − φ ( 1 ) + Φ ( 1 ) − Φ ( 0 ) , 3 φ ( 1 ) ν ( 1 − 2 ρ ) h ( ν ) λ s 8 ( log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ) log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] 3 ⎭ ⎬ ⎫ . (78)
Since the left side tends to zero, the constant term is harmless. Using Eqs. (63) and (69) and the fact that x ↦ x log ( 2 T ) + x x\mapsto\frac{x}{\log(2T)+x} x ↦ l o g ( 2 T ) + x x is increasing, we obtain
( λ s ) 3 ≤ 49152 ν ( 1 − 2 ρ ) 3 h ( ν ) 3 r 2 φ ( 1 ) c ⋆ s ( log ( 1 s ) + log ( 2 T ) ) . (\lambda\sqrt{s})^{3}\le\frac{49152\nu(1-2\rho)^{3}h(\nu)^{3}}{r^{2}\varphi(1)}c_{\star}s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right). ( λ s ) 3 ≤ r 2 φ ( 1 ) 49152 ν ( 1 − 2 ρ ) 3 h ( ν ) 3 c ⋆ s ( log ( s 1 ) + log ( 2 T ) ) .
Therefore,
λ s ≤ ( 1 − 2 ρ ) h ( ν ) ( 49152 ν r 2 φ ( 1 ) ) 1 3 1 c ⋆ 1 3 [ s ( log ( 1 s ) + log ( 2 T ) ) ] 1 3 . (79) \lambda\sqrt{s}\le(1-2\rho)h(\nu)\left(\frac{49152\nu}{r^{2}\varphi(1)}\right)^{\frac{1}{3}}\frac{1}{c_{\star}^{\frac{1}{3}}}\left[s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)\right]^{\frac{1}{3}}. \tag*{(79)} λ s ≤ ( 1 − 2 ρ ) h ( ν ) ( r 2 φ ( 1 ) 49152 ν ) 3 1 c ⋆ 3 1 1 [ s ( log ( s 1 ) + log ( 2 T ) ) ] 3 1 . (79)
We next control the normalized feature channel directly, that is, ∣ w t σ t τ t − θ t w ⋆ − r t δ Z t ∣ ≤ λ \left|\frac{w_t}{\sigma_t\tau_t}-\theta_tw^{\star}-\frac{r_t}{\sqrt{\delta}}Z_t\right|\le\lambda σ t τ t w t − θ t w ⋆ − δ r t Z t ≤ λ . Since w ⋆ w^{\star} w ⋆ and Z t Z_t Z t are independent and Z t Z_t Z t has mean 0, multiplying by ∣ w ⋆ ∣ |w^{\star}| ∣ w ⋆ ∣ and taking expected value with ∣ E [ X ] ∣ ≤ E [ ∣ X ∣ ] |\mathbb{E}[X]|\le\mathbb{E}[|X|] ∣ E [ X ] ∣ ≤ E [ ∣ X ∣ ] we get,
∣ E [ w ⋆ w t ] ν 2 σ t τ t − θ t ∣ ≤ λ ν 2 E ∣ w ⋆ ∣ = 2 ν π λ s . (80) \left|\frac{\mathbb{E}[w^{\star}w_t]}{\nu^{2}\sigma_t\tau_t}-\theta_t\right|\le\frac{\lambda}{\nu^{2}}\mathbb{E}|w^{\star}|=\frac{\sqrt{2}}{\nu\sqrt{\pi}}\lambda\sqrt{s}. \tag*{(80)} ν 2 σ t τ t E [ w ⋆ w t ] − θ t ≤ ν 2 λ E ∣ w ⋆ ∣ = ν π 2 λ s . (80)
Moreover,
E [ w t 2 ] ( σ t τ t ) 2 − ν 2 ( E [ w ⋆ w t ] ν 2 σ t τ t ) 2 = E [ ( w t σ t τ t − E [ w ⋆ w t ] ν 2 σ t τ t w ⋆ ) 2 ] . \frac{\mathbb{E}[w_t^2]}{(\sigma_t\tau_t)^2}-\nu^2\left(\frac{\mathbb{E}[w^\star w_t]}{\nu^2\sigma_t\tau_t}\right)^2=\mathbb{E}\left[\left(\frac{w_t}{\sigma_t\tau_t}-\frac{\mathbb{E}[w^\star w_t]}{\nu^2\sigma_t\tau_t}w^\star\right)^2\right]. ( σ t τ t ) 2 E [ w t 2 ] − ν 2 ( ν 2 σ t τ t E [ w ⋆ w t ] ) 2 = E [ ( σ t τ t w t − ν 2 σ t τ t E [ w ⋆ w t ] w ⋆ ) 2 ] .
The coefficient E [ w ⋆ w t ] ν 2 σ t τ t \frac{\mathbb{E}[w^\star w_t]}{\nu^2\sigma_t\tau_t} ν 2 σ t τ t E [ w ⋆ w t ] is the L 2 L^2 L 2 -projection coefficient of w t σ t τ t \frac{w_t}{\sigma_t\tau_t} σ t τ t w t onto the span of w ⋆ w^\star w ⋆ .10 Consequently,
E [ w t 2 ] ( σ t τ t ) 2 − ν 2 ( E [ w ⋆ w t ] ν 2 σ t τ t ) 2 ≤ E [ ( w t σ t τ t − θ t w ⋆ ) 2 ] . (81) \frac{\mathbb{E}[w_t^2]}{(\sigma_t\tau_t)^2}-\nu^2\left(\frac{\mathbb{E}[w^\star w_t]}{\nu^2\sigma_t\tau_t}\right)^2\leq\mathbb{E}\left[\left(\frac{w_t}{\sigma_t\tau_t}-\theta_t w^\star\right)^2\right]. \tag*{(81)} ( σ t τ t ) 2 E [ w t 2 ] − ν 2 ( ν 2 σ t τ t E [ w ⋆ w t ] ) 2 ≤ E [ ( σ t τ t w t − θ t w ⋆ ) 2 ] . (81)
We now bound the right-hand side. On active coordinates,
∣ w t σ t τ t − θ t w ⋆ ∣ ≤ λ + r t δ ∣ Z t ∣ ≤ λ + ∣ Z t ∣ δ , \left|\frac{w_t}{\sigma_t\tau_t}-\theta_t w^\star\right|\leq\lambda+\frac{r_t}{\sqrt{\delta}}|Z_t|\leq\lambda+\frac{|Z_t|}{\sqrt{\delta}}, σ t τ t w t − θ t w ⋆ ≤ λ + δ r t ∣ Z t ∣ ≤ λ + δ ∣ Z t ∣ ,
where we used r t ≤ 1 r_t\leq1 r t ≤ 1 (see conclusion beneath Eq. (48) ). Since P ( w ⋆ ≠ 0 ) = s \mathbb{P}(w^\star\neq0)=s P ( w ⋆ = 0 ) = s and the activity indicator is independent of Z t Z_t Z t ,
E [ ( w t σ t τ t − θ t w ⋆ ) 2 1 { w ⋆ ≠ 0 } ] ≤ s E [ ( λ + ∣ Z t ∣ δ ) 2 ] = s ( λ 2 + 2 λ 2 π δ + 1 δ ) ≤ 2 s ( λ 2 + 1 δ ) . (82) \mathbb{E}\left[\left(\frac{w_t}{\sigma_t\tau_t}-\theta_t w^\star\right)^2\mathbf{1}_{\{w^\star\neq0\}}\right]\leq s\mathbb{E}\left[\left(\lambda+\frac{|Z_t|}{\sqrt{\delta}}\right)^2\right]=s\left(\lambda^2+2\lambda\sqrt{\frac{2}{\pi\delta}}+\frac{1}{\delta}\right)\leq2s\left(\lambda^2+\frac{1}{\delta}\right). \tag*{(82)} E [ ( σ t τ t w t − θ t w ⋆ ) 2 1 { w ⋆ = 0 } ] ≤ s E [ ( λ + δ ∣ Z t ∣ ) 2 ] = s ( λ 2 + 2 λ π δ 2 + δ 1 ) ≤ 2 s ( λ 2 + δ 1 ) . (82)
On null coordinates that were already nonzero at time t − 1 t-1 t − 1 , the event { w ⋆ = 0 , w t − 1 ≠ 0 } \{w^\star=0,w_{t-1}\neq0\} { w ⋆ = 0 , w t − 1 = 0 } depends only on the previous Gaussian variables and is therefore independent of the fresh Z t Z_t Z t . Hence
E [ ( w t σ t τ t ) 2 1 { w ⋆ = 0 , w t − 1 ≠ 0 } ] ≤ 2 ( 1 − s ) P ( w t − 1 ≠ 0 ∣ w ⋆ = 0 ) ( λ 2 + 1 δ ) . (83) \mathbb{E}\left[\left(\frac{w_t}{\sigma_t\tau_t}\right)^2\mathbf{1}_{\{w^\star=0,w_{t-1}\neq0\}}\right]\leq2(1-s)\mathbb{P}(w_{t-1}\neq0\mid w^\star=0)\left(\lambda^2+\frac{1}{\delta}\right). \tag*{(83)} E [ ( σ t τ t w t ) 2 1 { w ⋆ = 0 , w t − 1 = 0 } ] ≤ 2 ( 1 − s ) P ( w t − 1 = 0 ∣ w ⋆ = 0 ) ( λ 2 + δ 1 ) . (83)
For a newly activated null coordinate, w ⋆ = 0 w^\star=0 w ⋆ = 0 and w t − 1 = 0 w_{t-1}=0 w t − 1 = 0 , we get w t σ t τ t = sign ( Z t ) ( r t δ ∣ Z t ∣ − λ ) + \frac{w_t}{\sigma_t\tau_t}=\operatorname{sign}(Z_t)\left(\frac{r_t}{\sqrt{\delta}}|Z_t|-\lambda\right)_+ σ t τ t w t = sign ( Z t ) ( δ r t ∣ Z t ∣ − λ ) + by the soft-thresholding formula. Therefore,
E [ ( w t σ t τ t ) 2 1 { w ⋆ = 0 , w t − 1 = 0 , w t ≠ 0 } ] ≤ 2 ( 1 − s ) r t 2 δ P ( w t − 1 = 0 ∣ w ⋆ = 0 ) ∫ λ δ / r t ∞ ( z − λ δ r t ) 2 φ ( z ) d z . \mathbb{E}\left[\left(\frac{w_t}{\sigma_t\tau_t}\right)^2\mathbf{1}_{\{w^\star=0,w_{t-1}=0,w_t\neq0\}}\right]\leq2(1-s)\frac{r_t^2}{\delta}\mathbb{P}(w_{t-1}=0\mid w^\star=0)\int_{\lambda\sqrt{\delta}/r_t}^{\infty}\left(z-\frac{\lambda\sqrt{\delta}}{r_t}\right)^2\varphi(z)\,dz. E [ ( σ t τ t w t ) 2 1 { w ⋆ = 0 , w t − 1 = 0 , w t = 0 } ] ≤ 2 ( 1 − s ) δ r t 2 P ( w t − 1 = 0 ∣ w ⋆ = 0 ) ∫ λ δ / r t ∞ ( z − r t λ δ ) 2 φ ( z ) d z .
For every x ≥ 0 x\geq0 x ≥ 0 using integration by parts,
∫ x ∞ ( z − x ) 2 φ ( z ) d z = ( 1 + x 2 ) Φ ( − x ) − x φ ( x ) = Φ ( − x ) + x ( x Φ ( − x ) − φ ( x ) ) ≤ Φ ( − x ) , \int_x^\infty(z-x)^2\varphi(z)\,dz=(1+x^2)\Phi(-x)-x\varphi(x)=\Phi(-x)+x\left(x\Phi(-x)-\varphi(x)\right)\leq\Phi(-x), ∫ x ∞ ( z − x ) 2 φ ( z ) d z = ( 1 + x 2 ) Φ ( − x ) − x φ ( x ) = Φ ( − x ) + x ( x Φ ( − x ) − φ ( x ) ) ≤ Φ ( − x ) ,
where we used x Φ ( − x ) ≤ φ ( x ) x\Phi(-x)\leq\varphi(x) x Φ ( − x ) ≤ φ ( x ) because f ′ ( x ) = − Φ ( − x ) < 0 f'(x)=-\Phi(-x)<0 f ′ ( x ) = − Φ ( − x ) < 0 where f ( x ) = φ ( x ) − x Φ ( − x ) f(x)=\varphi(x)-x\Phi(-x) f ( x ) = φ ( x ) − x Φ ( − x ) with lim x → ∞ f ( x ) = 0 \lim_{x\to\infty}f(x)=0 lim x → ∞ f ( x ) = 0 . Since r t ≤ 1 r_t\leq1 r t ≤ 1 (see conclusion beneath Eq. (48) ),
E [ ( w t σ t τ t ) 2 1 { w ⋆ = 0 , w t − 1 = 0 , w t ≠ 0 } ] ≤ 2 ( 1 − s ) δ P ( w t − 1 = 0 , w t ≠ 0 ∣ w ⋆ = 0 ) . (84) \mathbb{E}\left[\left(\frac{w_t}{\sigma_t\tau_t}\right)^2\mathbf{1}_{\{w^\star=0,w_{t-1}=0,w_t\neq0\}}\right]\leq\frac{2(1-s)}{\delta}\mathbb{P}(w_{t-1}=0,w_t\neq0\mid w^\star=0). \tag*{(84)} E [ ( σ t τ t w t ) 2 1 { w ⋆ = 0 , w t − 1 = 0 , w t = 0 } ] ≤ δ 2 ( 1 − s ) P ( w t − 1 = 0 , w t = 0 ∣ w ⋆ = 0 ) . (84)
Since null support is persistent, the events { w t − 1 ≠ 0 } \{w_{t-1}\neq0\} { w t − 1 = 0 } and { w t − 1 = 0 , w t ≠ 0 } \{w_{t-1}=0,w_t\neq0\} { w t − 1 = 0 , w t = 0 } partition { w t ≠ 0 } \{w_t\neq0\} { w t = 0 } almost surely. Combining Eqs. (81) to (84) , we obtain
E [ w t 2 ] ( σ t τ t ) 2 − ν 2 ( E [ w ⋆ w t ] ν 2 σ t τ t ) 2 ≤ 2 [ s + P ( w t ≠ 0 ∣ w ⋆ = 0 ) ] ( λ 2 + 1 δ ) . (85) \frac{\mathbb{E}[w_t^2]}{(\sigma_t\tau_t)^2}-\nu^2\left(\frac{\mathbb{E}[w^\star w_t]}{\nu^2\sigma_t\tau_t}\right)^2\leq2\left[s+\mathbb{P}(w_t\neq0\mid w^\star=0)\right]\left(\lambda^2+\frac{1}{\delta}\right). \tag*{(85)} ( σ t τ t ) 2 E [ w t 2 ] − ν 2 ( ν 2 σ t τ t E [ w ⋆ w t ] ) 2 ≤ 2 [ s + P ( w t = 0 ∣ w ⋆ = 0 ) ] ( λ 2 + δ 1 ) . (85)
By the scalar identities ν 2 α t = E [ w ⋆ w t ] \nu^2\alpha_t=\mathbb{E}[w^\star w_t] ν 2 α t = E [ w ⋆ w t ] and ν 2 α t 2 + σ t 2 = E [ w t 2 ] \nu^2\alpha_t^2+\sigma_t^2=\mathbb{E}[w_t^2] ν 2 α t 2 + σ t 2 = E [ w t 2 ] , the left-hand side is exactly 1 τ t 2 \frac{1}{\tau_t^2} τ t 2 1 . From Eq. (68) , for s ≤ 1 2 s\leq\frac{1}{2} s ≤ 2 1 ,
ν 2 ( α t − r ) 2 + σ t 2 ≥ 3 327680 δ P ( w t ≠ 0 ∣ w ⋆ = 0 ) log ( 4 t P ( w t ≠ 0 ∣ w ⋆ = 0 ) ) . \nu^2(\alpha_t-r)^2+\sigma_t^2\geq\frac{3}{327680\delta}\frac{\mathbb{P}(w_t\neq0\mid w^\star=0)}{\log\left(\frac{4t}{\mathbb{P}(w_t\neq0\mid w^\star=0)}\right)}. ν 2 ( α t − r ) 2 + σ t 2 ≥ 327680 δ 3 log ( P ( w t = 0 ∣ w ⋆ = 0 ) 4 t ) P ( w t = 0 ∣ w ⋆ = 0 ) .
Under the contradiction assumption,
ν 2 ( α t − r ) 2 + σ t 2 < c ⋆ s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 t ) ) 2 . \nu^{2}(\alpha_t-r)^{2}+\sigma_t^{2}<c_{\star}s\frac{\log^{3}\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2t)\right)^{2}}. ν 2 ( α t − r ) 2 + σ t 2 < c ⋆ s ( log ( s 1 ) + log ( 2 t ) ) 2 log 3 ( s 1 ) .
If P ( w t ≠ 0 ∣ w ⋆ = 0 ) ≥ s \mathbb{P}(w_t\ne0\mid w^{\star}=0)\ge s P ( w t = 0 ∣ w ⋆ = 0 ) ≥ s , then
log ( 4 t P ( w t ≠ 0 ∣ w ⋆ = 0 ) ) ≤ log ( 4 t s ) ≤ 2 ( log ( 1 s ) + log ( 2 t ) ) , \log\left(\frac{4t}{\mathbb{P}(w_t\ne0\mid w^{\star}=0)}\right)\le\log\left(\frac{4t}{s}\right)\le2\left(\log\left(\frac{1}{s}\right)+\log(2t)\right), log ( P ( w t = 0 ∣ w ⋆ = 0 ) 4 t ) ≤ log ( s 4 t ) ≤ 2 ( log ( s 1 ) + log ( 2 t ) ) ,
and hence
P ( w t ≠ 0 ∣ w ⋆ = 0 ) ≤ c ⋆ s 655360 δ log 3 ( 1 s ) 3 ( log ( 1 s ) + log ( 2 t ) ) . \mathbb{P}(w_t\ne0\mid w^{\star}=0)\le c_{\star}s\frac{655360\delta\log^{3}\left(\frac{1}{s}\right)}{3\left(\log\left(\frac{1}{s}\right)+\log(2t)\right)}. P ( w t = 0 ∣ w ⋆ = 0 ) ≤ c ⋆ s 3 ( log ( s 1 ) + log ( 2 t ) ) 655360 δ log 3 ( s 1 ) .
If instead P ( w t ≠ 0 ∣ w ⋆ = 0 ) < s \mathbb{P}(w_t\ne0\mid w^{\star}=0)<s P ( w t = 0 ∣ w ⋆ = 0 ) < s , this probability is already bounded by s s s . Since null support is persistent (almost surely, unless the endpoints are exactly 0), for every t ≤ T t\le T t ≤ T ,
P ( w t ≠ 0 ∣ w ⋆ = 0 ) ≤ s + 655360 3 δ c ⋆ s log 3 ( 1 s ) log ( 1 s ) + log ( 2 T ) ≤ log ( 1 s ) + log ( 2 T ) ≥ log ( 1 s ) s + 655360 3 δ c ⋆ s log 2 ( 1 s ) . (86) \mathbb{P}(w_t\ne0\mid w^{\star}=0)\le s+\frac{655360}{3}\delta c_{\star}s\frac{\log^{3}\left(\frac{1}{s}\right)}{\log\left(\frac{1}{s}\right)+\log(2T)}\underset{\log\left(\frac{1}{s}\right)+\log(2T)\ge\log\left(\frac{1}{s}\right)}{\le}s+\frac{655360}{3}\delta c_{\star}s\log^{2}\left(\frac{1}{s}\right). \tag*{(86)} P ( w t = 0 ∣ w ⋆ = 0 ) ≤ s + 3 655360 δ c ⋆ s log ( s 1 ) + log ( 2 T ) log 3 ( s 1 ) l o g ( s 1 ) + l o g ( 2 T ) ≥ l o g ( s 1 ) ≤ s + 3 655360 δ c ⋆ s log 2 ( s 1 ) . (86)
Moreover, P ( w t ≠ w t − 1 ) ≤ s + P ( w t ≠ 0 ∣ w ⋆ = 0 ) \mathbb{P}(w_t\ne w_{t-1})\le s+\mathbb{P}(w_t\ne0\mid w^{\star}=0) P ( w t = w t − 1 ) ≤ s + P ( w t = 0 ∣ w ⋆ = 0 ) , because an update can occur either on an active coordinate, whose probability is s s s , or on a null coordinate that is nonzero at time t t t ({ w t ≠ w t − 1 , w ⋆ = 0 } ⊆ { w t ≠ 0 , w ⋆ = 0 } \{w_t\ne w_{t-1},w^{\star}=0\}\subseteq\{w_t\ne0,w^{\star}=0\} { w t = w t − 1 , w ⋆ = 0 } ⊆ { w t = 0 , w ⋆ = 0 } , almost surely). From the lower bound in Eq. (78) ,
c ⋆ s 6 log 3 ( 1 s ) r 2 ν 2 φ ( 1 ) ( log ( 1 s ) + log ( 2 T ) ) 2 ≥ ( λ s ( 1 − 2 ρ ) h ( ν ) log ( 1 s ) 16 ( log ( 2 T ) + log ( 1 s ) ) ) 3 c_{\star}s\frac{6\log^{3}\left(\frac{1}{s}\right)}{r^{2}\nu^{2}\varphi(1)\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^{2}}\ge\left(\frac{\lambda\sqrt{s}}{(1-2\rho)h(\nu)}\frac{\log\left(\frac{1}{s}\right)}{16\left(\log(2T)+\log\left(\frac{1}{s}\right)\right)}\right)^{3} c ⋆ s r 2 ν 2 φ ( 1 ) ( log ( s 1 ) + log ( 2 T ) ) 2 6 log 3 ( s 1 ) ≥ ( ( 1 − 2 ρ ) h ( ν ) λ s 16 ( log ( 2 T ) + log ( s 1 ) ) log ( s 1 ) ) 3
we obtain
λ s ≤ ( 1 − 2 ρ ) h ( ν ) ( 49152 ν r 2 φ ( 1 ) ) 1 3 1 c ⋆ 1 3 [ s ( log ( 1 s ) + log ( 2 T ) ) ] 1 3 . (87) \lambda\sqrt{s}\le(1-2\rho)h(\nu)\left(\frac{49152\nu}{r^{2}\varphi(1)}\right)^{\frac{1}{3}}\frac{1}{c_{\star}^{\frac{1}{3}}}\left[s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)\right]^{\frac{1}{3}}. \tag*{(87)} λ s ≤ ( 1 − 2 ρ ) h ( ν ) ( r 2 φ ( 1 ) 49152 ν ) 3 1 c ⋆ 3 1 1 [ s ( log ( s 1 ) + log ( 2 T ) ) ] 3 1 . (87)
Since in the present case s ( log ( 1 s ) + log ( 2 T ) ) < 1 s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)<1 s ( log ( s 1 ) + log ( 2 T ) ) < 1 . Eq. (87) implies that λ s ≲ c ⋆ 1 3 \lambda\sqrt{s}\lesssim c_{\star}^{\frac{1}{3}} λ s ≲ c ⋆ 3 1 (up to constants depending on the problem parameters). Thus, by choosing c ⋆ c_{\star} c ⋆ sufficiently small, λ s \lambda\sqrt{s} λ s is uniformly as small as needed. We henceforth take s s s sufficiently small that,
s ( 2 + 655360 3 δ c ⋆ log 2 ( 1 s ) ) ≤ min { 1 , δ } Φ − 1 ( 3 4 ) h ( ν ) 8 δ . \sqrt{s}\left(2+\frac{655360}{3}\delta c_{\star}\log^{2}\left(\frac{1}{s}\right)\right)\le\min\{1,\delta\}\Phi^{-1}\left(\frac{3}{4}\right)\sqrt{\frac{h(\nu)}{8\delta}}. s ( 2 + 3 655360 δ c ⋆ log 2 ( s 1 ) ) ≤ min { 1 , δ } Φ − 1 ( 4 3 ) 8 δ h ( ν ) .
Then,
P ( w t ≠ w t − 1 ) ≤ s ( 2 + 655360 3 δ c ⋆ log 2 ( 1 s ) ) ≤ λ s . (88) \mathbb{P}(w_t\ne w_{t-1})\le s\left(2+\frac{655360}{3}\delta c_{\star}\log^{2}\left(\frac{1}{s}\right)\right)\le\lambda\sqrt{s}. \tag*{(88)} P ( w t = w t − 1 ) ≤ s ( 2 + 3 655360 δ c ⋆ log 2 ( s 1 ) ) ≤ λ s . (88)
We also proved in Eqs. (72) and (73) ,
0 ≤ ( 1 − 2 ρ ) h ( ν ) − θ t ≤ 2 ( 1 − 2 ρ ) δ P ( w t ≠ w t − 1 ) ≤ 2 ( 1 − 2 ρ ) λ s , (89) 0\le(1-2\rho)h(\nu)-\theta_t\le\frac{2(1-2\rho)}{\delta}\mathbb{P}(w_t\ne w_{t-1})\le2(1-2\rho)\lambda\sqrt{s}, \tag*{(89)} 0 ≤ ( 1 − 2 ρ ) h ( ν ) − θ t ≤ δ 2 ( 1 − 2 ρ ) P ( w t = w t − 1 ) ≤ 2 ( 1 − 2 ρ ) λ s , (89)
and therefore since P ( w t ≠ w t − 1 ) ≤ λ s \mathbb{P}(w_t\ne w_{t-1})\le\lambda\sqrt{s} P ( w t = w t − 1 ) ≤ λ s by Eq. (88) so using Eq. (80) ,
∣ E [ w ⋆ w t ] ν 2 σ t τ t − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ ( 2 π 1 ν + 2 ( 1 − 2 ρ ) ) λ s . (90) \left|\frac{\mathbb{E}[w^{\star}w_t]}{\nu^{2}\sigma_t\tau_t}-(1-2\rho)h(\nu)\right|\le\left(\sqrt{\frac{2}{\pi}}\frac{1}{\nu}+2(1-2\rho)\right)\lambda\sqrt{s}. \tag*{(90)} ν 2 σ t τ t E [ w ⋆ w t ] − ( 1 − 2 ρ ) h ( ν ) ≤ ( π 2 ν 1 + 2 ( 1 − 2 ρ ) ) λ s . (90)
Using Eqs. (85) and (86) , we obtain,
σ t 2 ( σ t τ t ) 2 ≤ 2 s ( 2 + 655360 3 δ c ⋆ log 3 ( 1 s ) log ( 1 s ) + log ( 2 T ) ) ( λ 2 + 1 δ ) = ( 2 + 655360 3 δ c ⋆ log 3 ( 1 s ) log ( 1 s ) + log ( 2 T ) ) ( 2 ( λ s ) 2 + 2 s δ ) ≤ 4 λ s . (91) \begin{aligned}
\frac{\sigma_t^{2}}{(\sigma_t\tau_t)^{2}}\le2s\left(2+\frac{655360}{3}\delta c_{\star}\frac{\log^{3}\left(\frac{1}{s}\right)}{\log\left(\frac{1}{s}\right)+\log(2T)}\right)\left(\lambda^{2}+\frac{1}{\delta}\right) \\
&=\left(2+\frac{655360}{3}\delta c_{\star}\frac{\log^{3}\left(\frac{1}{s}\right)}{\log\left(\frac{1}{s}\right)+\log(2T)}\right)\left(2(\lambda\sqrt{s})^{2}+\frac{2s}{\delta}\right)\le4\lambda\sqrt{s}. \tag*{(91)}
\end{aligned} ( σ t τ t ) 2 σ t 2 ≤ 2 s ( 2 + 3 655360 δ c ⋆ log ( s 1 ) + log ( 2 T ) log 3 ( s 1 ) ) ( λ 2 + δ 1 ) = ( 2 + 3 655360 δ c ⋆ log ( s 1 ) + log ( 2 T ) log 3 ( s 1 ) ) ( 2 ( λ s ) 2 + δ 2 s ) ≤ 4 λ s . (91)
Here we chose λ s ( 2 + 655360 3 δ c ⋆ log 3 ( 1 s ) log ( 1 s ) + log ( 2 T ) ) ≤ 1 \lambda\sqrt{s}\left(2+\frac{655360}{3}\delta_{c_\star}\frac{\log^3\left(\frac{1}{s}\right)}{\log\left(\frac{1}{s}\right)+\log(2T)}\right)\le1 λ s ( 2 + 3 655360 δ c ⋆ l o g ( s 1 ) + l o g ( 2 T ) l o g 3 ( s 1 ) ) ≤ 1 and s ( 2 + 655360 3 δ c ⋆ log 3 ( 1 s ) log ( 1 s ) + log ( 2 T ) ) δ ≤ λ s \frac{s\left(2+\frac{655360}{3}\delta_{c_\star}\frac{\log^3\left(\frac{1}{s}\right)}{\log\left(\frac{1}{s}\right)+\log(2T)}\right)}{\delta}\le\lambda\sqrt{s} δ s ( 2 + 3 655360 δ c ⋆ l o g ( s 1 ) + l o g ( 2 T ) l o g 3 ( s 1 ) ) ≤ λ s for a sufficiently small s s s . We now prove a uniform upper bound on σ t τ t \sigma_t\tau_t σ t τ t . The lower bound follows directly from σ t τ t r t ≥ 1 \sigma_t\tau_t r_t\ge1 σ t τ t r t ≥ 1 (see below Eq. (50) ) and r t ≤ 1 r_t\le1 r t ≤ 1 (see conclusion beneath Eq. (48) ), and therefore σ t τ t ≥ 1 \sigma_t\tau_t\ge1 σ t τ t ≥ 1 . Since ( 1 − 2 ρ ) h ( ν ) = r h ( ν r ) (1-2\rho)h(\nu)=rh(\nu r) ( 1 − 2 ρ ) h ( ν ) = r h ( ν r ) by Lemma 16 and
ν r h ( ν r ) = ν r ∫ R ϕ ′ ′ ( ν r z ) φ ( z ) d z = u = ν r z 1 2 π ∫ R ϕ ′ ′ ( u ) e − u 2 2 ν 2 r 2 d u < 1 2 π ∫ R ϕ ′ ′ ( u ) d u = 1 2 π , \nu r h(\nu r)=\nu r\int_{\mathbb{R}}\phi''(\nu r z)\varphi(z)\,\mathrm{d}z\underset{u=\nu r z}{=}\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}\phi''(u)e^{-\frac{u^2}{2\nu^2r^2}}\,\mathrm{d}u<\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}\phi''(u)\,\mathrm{d}u=\frac{1}{\sqrt{2\pi}}, ν r h ( ν r ) = ν r ∫ R ϕ ′′ ( ν r z ) φ ( z ) d z u = ν r z = 2 π 1 ∫ R ϕ ′′ ( u ) e − 2 ν 2 r 2 u 2 d u < 2 π 1 ∫ R ϕ ′′ ( u ) d u = 2 π 1 ,
we have 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π > 0 1-\nu(1-2\rho)h(\nu)\sqrt{2\pi}>0 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π > 0 . For simplicity let Q 0 = max { 1 , 32 log ( 2 ) ν ( 1 − 2 ρ ) ( ν ( 1 − 2 ρ ) 2 2 π + 2 π e ) ( 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π ) ( 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π ) ν 2 ( 1 − 2 ρ ) h ( ν ) 2 } Q_0=\max\left\{1,\frac{32\log(2)}{\nu(1-2\rho)}\frac{\left(\frac{\nu(1-2\rho)}{2\sqrt{2\pi}}+\sqrt{\frac{2}{\pi e}}\right)(1+\nu(1-2\rho)h(\nu)\sqrt{2\pi})}{(1-\nu(1-2\rho)h(\nu)\sqrt{2\pi})\nu^2(1-2\rho)h(\nu)^2}\right\} Q 0 = max { 1 , ν ( 1 − 2 ρ ) 32 l o g ( 2 ) ( 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π ) ν 2 ( 1 − 2 ρ ) h ( ν ) 2 ( 2 2 π ν ( 1 − 2 ρ ) + π e 2 ) ( 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π ) } .
We take s s s still smaller, if necessary, so that
( 2 π ν + 2 ( 1 − 2 ρ ) ) λ s ≤ min { ( 1 − 2 ρ ) h ( ν ) 2 , 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π 4 ν 2 π } . (92) \left(\frac{\sqrt{\frac{2}{\pi}}}{\nu}+2(1-2\rho)\right)\lambda\sqrt{s}\le\min\left\{\frac{(1-2\rho)h(\nu)}{2},\frac{1-\nu(1-2\rho)h(\nu)\sqrt{2\pi}}{4\nu\sqrt{2\pi}}\right\}. \tag*{(92)} ν π 2 + 2 ( 1 − 2 ρ ) λ s ≤ min { 2 ( 1 − 2 ρ ) h ( ν ) , 4 ν 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π } . (92)
λ s ≤ min { ( 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π ) 2 128 π , ν 2 ( 1 − 2 ρ ) 2 h ( ν ) 2 8 ( ν ( 1 − 2 ρ ) 2 2 π + 2 π e ) 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π } . (93) \lambda\sqrt{s}\le\min\left\{\frac{\left(1-\nu(1-2\rho)h(\nu)\sqrt{2\pi}\right)^2}{128\pi},\frac{\nu^2(1-2\rho)^2h(\nu)^2}{8\left(\frac{\nu(1-2\rho)}{2\sqrt{2\pi}}+\sqrt{\frac{2}{\pi e}}\right)}\frac{1-\nu(1-2\rho)h(\nu)\sqrt{2\pi}}{1+\nu(1-2\rho)h(\nu)\sqrt{2\pi}}\right\}. \tag*{(93)} λ s ≤ min ⎩ ⎨ ⎧ 128 π ( 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π ) 2 , 8 ( 2 2 π ν ( 1 − 2 ρ ) + π e 2 ) ν 2 ( 1 − 2 ρ ) 2 h ( ν ) 2 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π ⎭ ⎬ ⎫ . (93)
These conditions are possible because we may pick c ⋆ c_\star c ⋆ in Eq. (87) arbitrarily small (depending on the problem parameters). Multiplying the sixth scalar equation by σ t τ t \sigma_t\tau_t σ t τ t and using E [ 2 π ρ ( ν Z 1 ) − 1 ] = 0 \mathbb{E}[2\pi_\rho(\nu Z_1)-1]=0 E [ 2 π ρ ( ν Z 1 ) − 1 ] = 0 gives,
1 = σ t τ t E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . (94) 1=\sigma_t\tau_t\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}\right]. \tag*{(94)} 1 = σ t τ t E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . (94)
Condition on Z 2 = z 2 Z_2=z_2 Z 2 = z 2 and make the change of variables ν α t z 1 + σ t z 2 = u + γ t ϕ ′ ( u ) \nu\alpha_tz_1+\sigma_tz_2=u+\gamma_t\phi'(u) ν α t z 1 + σ t z 2 = u + γ t ϕ ′ ( u ) . Then d z 1 = 1 + γ t ϕ ′ ′ ( u ) ν α t d u \mathrm{d}z_1=\frac{1+\gamma_t\phi''(u)}{\nu\alpha_t}\,\mathrm{d}u d z 1 = ν α t 1 + γ t ϕ ′′ ( u ) d u , and hence the Jacobian exactly cancels the denominator in Eq. (94) . The function z ↦ 2 ( 1 − π ρ ( ν z ) ) φ ( z ) z\mapsto2(1-\pi_\rho(\nu z))\varphi(z) z ↦ 2 ( 1 − π ρ ( ν z )) φ ( z ) is Lipschitz with constant ν ( 1 − 2 ρ ) 2 2 π + 2 π e \frac{\nu(1-2\rho)}{2\sqrt{2\pi}}+\sqrt{\frac{2}{\pi e}} 2 2 π ν ( 1 − 2 ρ ) + π e 2 . Indeed,
∣ d d z [ 2 ( 1 − π ρ ( ν z ) ) φ ( z ) ] ∣ = ∣ − 2 ν π ρ ′ ( ν z ) ⏟ = ( 1 − 2 ρ ) ϕ ′ ′ ( ν z ) φ ( z ) − 2 ( 1 − π ρ ( ν z ) ) z φ ( z ) ∣ ≤ ν ( 1 − 2 ρ ) 2 φ ( 0 ) + 2 sup z ∈ R ∣ z ∣ φ ( z ) , \left|\frac{\mathrm{d}}{\mathrm{d}z}\left[2(1-\pi_\rho(\nu z))\varphi(z)\right]\right|=\left|-2\nu\underbrace{\pi'_\rho(\nu z)}_{=(1-2\rho)\phi''(\nu z)}\varphi(z)-2(1-\pi_\rho(\nu z))z\varphi(z)\right|\le\frac{\nu(1-2\rho)}{2}\varphi(0)+2\sup_{z\in\mathbb{R}}|z|\varphi(z), d z d [ 2 ( 1 − π ρ ( ν z )) φ ( z ) ] = − 2 ν = ( 1 − 2 ρ ) ϕ ′′ ( ν z ) π ρ ′ ( ν z ) φ ( z ) − 2 ( 1 − π ρ ( ν z )) z φ ( z ) ≤ 2 ν ( 1 − 2 ρ ) φ ( 0 ) + 2 z ∈ R sup ∣ z ∣ φ ( z ) ,
and φ ( 0 ) = 1 2 π \varphi(0)=\frac{1}{\sqrt{2\pi}} φ ( 0 ) = 2 π 1 while sup z ∣ z ∣ φ ( z ) = 1 2 π e \sup_{z}|z|\varphi(z)=\frac{1}{\sqrt{2\pi e}} sup z ∣ z ∣ φ ( z ) = 2 π e 1 ; we focus on z > 0 z>0 z > 0 due to the symmetry of ∣ z ∣ φ ( z ) |z|\varphi(z) ∣ z ∣ φ ( z ) . Its derivative is given by d d z z φ ( z ) = 1 2 π e − z 2 2 ( 1 − z 2 ) \frac{\mathrm{d}}{\mathrm{d}z}z\varphi(z)=\frac{1}{\sqrt{2\pi}}e^{-\frac{z^2}{2}}(1-z^2) d z d z φ ( z ) = 2 π 1 e − 2 z 2 ( 1 − z 2 ) , and a stationary point is z = 1 z=1 z = 1 , which is also the maximum. Using this Lipschitz bound after the preceding change of variables gives
∣ σ t τ t E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] − 1 2 π [ ν 2 ( α t σ t ) 2 + σ t 2 ( σ t τ t ) 2 ] ∣ ≤ σ t τ t ν α t ∣ ∫ R 2 [ 2 ( 1 − π ρ ( u + γ t ϕ ′ ( u ) − σ t z 2 α t ) ) φ ( u + γ t ϕ ′ ( u ) − σ t z 2 ν α t ) − 2 ( 1 − π ρ ( − σ t z 2 α t ) ) φ ( − σ t z 2 ν α t ) ] ϕ ′ ′ ( u ) φ ( z 2 ) d u d z 2 ∣ ≤ σ t τ t ν α t ( ν ( 1 − 2 ρ ) 2 2 π + 2 π e ) ∫ R 2 ∣ u + γ t ϕ ′ ( u ) ν α t ∣ ϕ ′ ′ ( u ) φ ( z 2 ) d u d z 2 ≤ σ t τ t ν 2 α t 2 ( ν ( 1 − 2 ρ ) 2 2 π + 2 π e ) ∫ R 2 ( ∣ u ∣ + γ t ϕ ′ ( u ) ) ϕ ′ ′ ( u ) φ ( z 2 ) d u d z 2 = σ t τ t ν 2 α t 2 ( ν ( 1 − 2 ρ ) 2 2 π + 2 π e ) [ ∫ R ∣ u φ ′ ′ ( u ) ∣ d u + γ t ∫ R φ ′ ( u ) φ ′ ′ ( u ) d u ] = σ t τ t ν 2 α t 2 ( ν ( 1 − 2 ρ ) 2 2 π + 2 π e ) ( 2 log 2 + γ t 2 ) = ν ( 1 − 2 ρ ) 2 2 π + 2 π e ν 2 2 ( α t σ t τ t ) 2 [ 2 log 2 σ t τ t + 1 2 γ t σ t τ t ] . (95) \begin{aligned}
\left|\sigma_t\tau_t\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}\right]-\frac{1}{\sqrt{2\pi\left[\nu^2\left(\frac{\alpha_t}{\sigma_t}\right)^2+\frac{\sigma_t^2}{(\sigma_t\tau_t)^2}\right]}}\right| \\
&\le\frac{\sigma_t\tau_t}{\nu\alpha_t}\left|\int_{\mathbb{R}^2}\left[2\left(1-\pi_\rho\left(\frac{u+\gamma_t\phi'(u)-\sigma_tz_2}{\alpha_t}\right)\right)\varphi\left(\frac{u+\gamma_t\phi'(u)-\sigma_tz_2}{\nu\alpha_t}\right)-2\left(1-\pi_\rho\left(-\frac{\sigma_tz_2}{\alpha_t}\right)\right)\varphi\left(-\frac{\sigma_tz_2}{\nu\alpha_t}\right)\right]\phi''(u)\varphi(z_2)\,\mathrm{d}u\,\mathrm{d}z_2\right| \\
&\le\frac{\sigma_t\tau_t}{\nu\alpha_t}\left(\frac{\nu(1-2\rho)}{2\sqrt{2\pi}}+\sqrt{\frac{2}{\pi e}}\right)\int_{\mathbb{R}^2}\left|\frac{u+\gamma_t\phi'(u)}{\nu\alpha_t}\right|\phi''(u)\varphi(z_2)\,\mathrm{d}u\,\mathrm{d}z_2 \\
&\le\frac{\sigma_t\tau_t}{\nu^2\alpha_t^2}\left(\frac{\nu(1-2\rho)}{2\sqrt{2\pi}}+\sqrt{\frac{2}{\pi e}}\right)\int_{\mathbb{R}^2}\left(|u|+\gamma_t\phi'(u)\right)\phi''(u)\varphi(z_2)\,\mathrm{d}u\,\mathrm{d}z_2 \\
&= \frac{\sigma_t\tau_t}{\nu^2\alpha_t^2}\left(\frac{\nu(1-2\rho)}{2\sqrt{2\pi}}+\sqrt{\frac{2}{\pi e}}\right)\left[\int_{\mathbb{R}}|u\varphi^{\prime\prime}(u)|\,du+\gamma_t\int_{\mathbb{R}}\varphi^\prime(u)\varphi^{\prime\prime}(u)\,du\right] \\
&= \frac{\sigma_t\tau_t}{\nu^2\alpha_t^2}\left(\frac{\nu(1-2\rho)}{2\sqrt{2\pi}}+\sqrt{\frac{2}{\pi e}}\right)\left(2\log2+\frac{\gamma_t}{2}\right) \\
&= \frac{\frac{\nu(1-2\rho)}{2\sqrt{2\pi}}+\sqrt{\frac{2}{\pi e}}}{\nu^2}\frac{2}{\left(\frac{\alpha_t}{\sigma_t\tau_t}\right)^2}\left[\frac{2\log2}{\sigma_t\tau_t}+\frac{1}{2}\frac{\gamma_t}{\sigma_t\tau_t}\right]. \tag*{(95)}
\end{aligned} σ t τ t E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] − 2 π [ ν 2 ( σ t α t ) 2 + ( σ t τ t ) 2 σ t 2 ] 1 ≤ ν α t σ t τ t ∫ R 2 [ 2 ( 1 − π ρ ( α t u + γ t ϕ ′ ( u ) − σ t z 2 ) ) φ ( ν α t u + γ t ϕ ′ ( u ) − σ t z 2 ) − 2 ( 1 − π ρ ( − α t σ t z 2 ) ) φ ( − ν α t σ t z 2 ) ] ϕ ′′ ( u ) φ ( z 2 ) d u d z 2 ≤ ν α t σ t τ t ( 2 2 π ν ( 1 − 2 ρ ) + π e 2 ) ∫ R 2 ν α t u + γ t ϕ ′ ( u ) ϕ ′′ ( u ) φ ( z 2 ) d u d z 2 ≤ ν 2 α t 2 σ t τ t ( 2 2 π ν ( 1 − 2 ρ ) + π e 2 ) ∫ R 2 ( ∣ u ∣ + γ t ϕ ′ ( u ) ) ϕ ′′ ( u ) φ ( z 2 ) d u d z 2 = ν 2 α t 2 σ t τ t ( 2 2 π ν ( 1 − 2 ρ ) + π e 2 ) [ ∫ R ∣ u φ ′′ ( u ) ∣ d u + γ t ∫ R φ ′ ( u ) φ ′′ ( u ) d u ] = ν 2 α t 2 σ t τ t ( 2 2 π ν ( 1 − 2 ρ ) + π e 2 ) ( 2 log 2 + 2 γ t ) = ν 2 2 2 π ν ( 1 − 2 ρ ) + π e 2 ( σ t τ t α t ) 2 2 [ σ t τ t 2 log 2 + 2 1 σ t τ t γ t ] . (95)
where [1] follows by the change of variables ν α t z 1 + σ t z 2 = u + γ t φ ′ ( u ) \nu\alpha_t z_1+\sigma_t z_2=u+\gamma_t\varphi^\prime(u) ν α t z 1 + σ t z 2 = u + γ t φ ′ ( u ) , and from symmetry and π ρ ( − u ) + π ρ ( u ) = 1 \pi_\rho(-u)+\pi_\rho(u)=1 π ρ ( − u ) + π ρ ( u ) = 1 ,
∫ R 2 ( 1 − π ρ ( − σ t z 2 α t ) ) φ ( σ t z 2 ν α t ) φ ( z 2 ) d z 2 = ∫ R φ ( σ t z 2 ν α t ) φ ( z 2 ) d z 2 \int_{\mathbb{R}}2\left(1-\pi_\rho\left(-\frac{\sigma_t z_2}{\alpha_t}\right)\right)\varphi\left(\frac{\sigma_t z_2}{\nu\alpha_t}\right)\varphi(z_2)\,dz_2=\int_{\mathbb{R}}\varphi\left(\frac{\sigma_t z_2}{\nu\alpha_t}\right)\varphi(z_2)\,dz_2 ∫ R 2 ( 1 − π ρ ( − α t σ t z 2 ) ) φ ( ν α t σ t z 2 ) φ ( z 2 ) d z 2 = ∫ R φ ( ν α t σ t z 2 ) φ ( z 2 ) d z 2
Now since ∫ R φ ′ ′ ( u ) d u = 1 \int_{\mathbb{R}}\varphi^{\prime\prime}(u)\,du=1 ∫ R φ ′′ ( u ) d u = 1 and ∫ R φ ( σ t z 2 ν α t ) φ ( z 2 ) d z 2 = ν α t 2 π ( ν 2 α t 2 + σ t 2 ) \int_{\mathbb{R}}\varphi\left(\frac{\sigma_t z_2}{\nu\alpha_t}\right)\varphi(z_2)\,dz_2=\frac{\nu\alpha_t}{\sqrt{2\pi(\nu^2\alpha_t^2+\sigma_t^2)}} ∫ R φ ( ν α t σ t z 2 ) φ ( z 2 ) d z 2 = 2 π ( ν 2 α t 2 + σ t 2 ) ν α t , this explains the transition. By Eqs. (90) , (92) and (93) with the fact that E [ w ⋆ w t ] = ν 2 α t \mathbb{E}[w^\star w_t]=\nu^2\alpha_t E [ w ⋆ w t ] = ν 2 α t , imply
2 π [ ν 2 ( α t σ t τ t ) 2 + ( σ t σ t τ t ) 2 ] ≤ 2 π ν ∣ α t σ t τ t ∣ + 2 π σ t σ t τ t ≤ ν ( 1 − 2 ρ ) h ( ν ) 2 π + ( 2 + 2 ν 2 π ( 1 − 2 ρ ) ) λ s + 2 π σ t σ t τ t ≤ ν ( 1 − 2 ρ ) h ( ν ) 2 π + 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π 2 = 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π 2 . \begin{aligned}
\sqrt{2\pi\left[\nu^2\left(\frac{\alpha_t}{\sigma_t\tau_t}\right)^2+\left(\frac{\sigma_t}{\sigma_t\tau_t}\right)^2\right]}\leq\sqrt{2\pi}\nu\left|\frac{\alpha_t}{\sigma_t\tau_t}\right|+\sqrt{2\pi}\frac{\sigma_t}{\sigma_t\tau_t} \\
&\leq\nu(1-2\rho)h(\nu)\sqrt{2\pi}+\left(2+2\nu\sqrt{2\pi}(1-2\rho)\right)\lambda\sqrt{s}+\sqrt{2\pi}\frac{\sigma_t}{\sigma_t\tau_t} \\
&\leq\nu(1-2\rho)h(\nu)\sqrt{2\pi}+\frac{1-\nu(1-2\rho)h(\nu)\sqrt{2\pi}}{2}=\frac{1+\nu(1-2\rho)h(\nu)\sqrt{2\pi}}{2}.
\end{aligned} 2 π [ ν 2 ( σ t τ t α t ) 2 + ( σ t τ t σ t ) 2 ] ≤ 2 π ν σ t τ t α t + 2 π σ t τ t σ t ≤ ν ( 1 − 2 ρ ) h ( ν ) 2 π + ( 2 + 2 ν 2 π ( 1 − 2 ρ ) ) λ s + 2 π σ t τ t σ t ≤ ν ( 1 − 2 ρ ) h ( ν ) 2 π + 2 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π = 2 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π .
Therefore,
1 2 π [ ν 2 ( α t σ t τ t ) 2 + ( σ t σ t τ t ) 2 ] ≥ 2 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π = 1 + 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π . (96) \frac{1}{\sqrt{2\pi\left[\nu^2\left(\frac{\alpha_t}{\sigma_t\tau_t}\right)^2+\left(\frac{\sigma_t}{\sigma_t\tau_t}\right)^2\right]}}\geq\frac{2}{1+\nu(1-2\rho)h(\nu)\sqrt{2\pi}}=1+\frac{1-\nu(1-2\rho)h(\nu)\sqrt{2\pi}}{1+\nu(1-2\rho)h(\nu)\sqrt{2\pi}}. \tag*{(96)} 2 π [ ν 2 ( σ t τ t α t ) 2 + ( σ t τ t σ t ) 2 ] 1 ≥ 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π 2 = 1 + 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π . (96)
Furthermore by Eq. (88) , γ t σ t τ t = 1 δ P ( w t ≠ w t − 1 ) ≤ λ s \frac{\gamma_t}{\sigma_t\tau_t}=\frac{1}{\delta}\mathbb{P}(w_t\ne w_{t-1})\leq\lambda\sqrt{s} σ t τ t γ t = δ 1 P ( w t = w t − 1 ) ≤ λ s . Thus, if σ t τ t ≥ Q 0 \sigma_t\tau_t\geq Q_0 σ t τ t ≥ Q 0 , then the lower bound of α t σ t τ t ≥ 1 2 ( 1 − 2 ρ ) h ( ν ) \frac{\alpha_t}{\sigma_t\tau_t}\geq\frac{1}{2}(1-2\rho)h(\nu) σ t τ t α t ≥ 2 1 ( 1 − 2 ρ ) h ( ν ) (from the lower bound established, see Eq. (58) and conclusions below it), the estimate γ t σ t τ t ≤ λ s \frac{\gamma_t}{\sigma_t\tau_t}\leq\lambda\sqrt{s} σ t τ t γ t ≤ λ s , the definition of Q 0 Q_0 Q 0 , Eq. (93) imply that the absolute error in Eq. (95) is at most 1 2 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π \frac{1}{2}\frac{1-\nu(1-2\rho)h(\nu)\sqrt{2\pi}}{1+\nu(1-2\rho)h(\nu)\sqrt{2\pi}} 2 1 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π . Together with Eq. (96) , the right-hand side of Eq. (94) would then be at least 1 + 1 2 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π > 1 1+\frac{1}{2}\frac{1-\nu(1-2\rho)h(\nu)\sqrt{2\pi}}{1+\nu(1-2\rho)h(\nu)\sqrt{2\pi}}>1 1 + 2 1 1 + ν ( 1 − 2 ρ ) h ( ν ) 2 π 1 − ν ( 1 − 2 ρ ) h ( ν ) 2 π > 1 , contradicting Eq. (94) . Hence 1 ≤ σ t τ t ≤ Q 0 1\leq\sigma_t\tau_t\leq Q_0 1 ≤ σ t τ t ≤ Q 0 . From symmetry of ( Z 1 , Z 2 ) (Z_1,Z_2) ( Z 1 , Z 2 ) , evenness of φ ′ ′ \varphi^{\prime\prime} φ ′′ , and π ρ ( u ) + π ρ ( − u ) = 1 \pi_\rho(u)+\pi_\rho(-u)=1 π ρ ( u ) + π ρ ( − u ) = 1 ,
2 E [ ( 1 − π ρ ( ν Z 1 ) ) φ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ φ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] = h ( ν 2 α t 2 + σ t 2 ) . (97) 2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\varphi^{\prime\prime}(\nu\alpha_tZ_1+\sigma_tZ_2)\right]=\mathbb{E}\left[\varphi^{\prime\prime}(\nu\alpha_tZ_1+\sigma_tZ_2)\right]=h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right). \tag*{(97)} 2 E [ ( 1 − π ρ ( ν Z 1 )) φ ′′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ φ ′′ ( ν α t Z 1 + σ t Z 2 ) ] = h ( ν 2 α t 2 + σ t 2 ) . (97)
Therefore the sixth scalar (see Eq. (94) ) with E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ] = 1 \mathbb{E}[2(1-\pi_\rho(\nu Z_1))]=1 E [ 2 ( 1 − π ρ ( ν Z 1 ))] = 1 equation gives,
∣ 1 σ t τ t − h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ∣ φ ′ ′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) − φ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ∣ ] + E [ 2 ( 1 − π ρ ( ν Z 1 ) ) φ ′ ′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) γ t φ ′ ′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t φ ′ ′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) ] . \begin{aligned}
\left|\frac{1}{\sigma_t\tau_t}-h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)\right|\leq\mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))\left|\varphi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\varphi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)-\varphi^{\prime\prime}(\nu\alpha_tZ_1+\sigma_tZ_2)\right|\right] \\
&+\mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))\varphi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\varphi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)\frac{\gamma_t\varphi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\varphi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}{1+\gamma_t\varphi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\varphi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}\right].
\end{aligned} σ t τ t 1 − h ( ν 2 α t 2 + σ t 2 ) ≤ E [ 2 ( 1 − π ρ ( ν Z 1 )) ∣ φ ′′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) − φ ′′ ( ν α t Z 1 + σ t Z 2 ) ∣ ] + E [ 2 ( 1 − π ρ ( ν Z 1 )) φ ′′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t φ ′′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) γ t φ ′′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) ] .
Since φ ′ ′ ( u ) = φ ′ ( u ) ( 1 − φ ′ ( u ) ) \varphi^{\prime\prime}(u)=\varphi^\prime(u)(1-\varphi^\prime(u)) φ ′′ ( u ) = φ ′ ( u ) ( 1 − φ ′ ( u )) , the map p ↦ p ( 1 − p ) p\mapsto p(1-p) p ↦ p ( 1 − p ) is 1-Lipschitz on [ 0 , 1 ] [0,1] [ 0 , 1 ] , and from the Lipschitzness property we have ∣ φ ′ ′ ( u ) − φ ′ ′ ( v ) ∣ ≤ ∣ φ ′ ( u ) − φ ′ ( v ) ∣ |\varphi^{\prime\prime}(u)-\varphi^{\prime\prime}(v)|\leq|\varphi^\prime(u)-\varphi^\prime(v)| ∣ φ ′′ ( u ) − φ ′′ ( v ) ∣ ≤ ∣ φ ′ ( u ) − φ ′ ( v ) ∣ . Using Eq. (71) ,
0 ≤ φ ′ ( ν α t Z 1 + σ t Z 2 ) − φ ′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) ≤ 2 γ t φ ′ ′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t φ ′ ′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) , 0\leq\varphi^\prime(\nu\alpha_tZ_1+\sigma_tZ_2)-\varphi^\prime\left(\operatorname{Prox}_{\gamma_t\varphi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)\leq\frac{2\gamma_t\varphi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\varphi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}{1+\gamma_t\varphi^{\prime\prime}\left(\operatorname{Prox}_{\gamma_t\varphi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}, 0 ≤ φ ′ ( ν α t Z 1 + σ t Z 2 ) − φ ′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) ≤ 1 + γ t φ ′′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) 2 γ t φ ′′ ( Prox γ t φ ( ν α t Z 1 + σ t Z 2 ) ) ,
together with ϕ ′ ′ ≤ 1 4 \phi'' \le\frac{1}{4} ϕ ′′ ≤ 4 1 , gives,
∣ 1 σ t τ t ( ν 2 α t 2 + σ t 2 ) ∣ ≤ 9 4 E [ 2 ( 1 − π ν ( z 1 ) ) γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ≤ 9 4 γ t σ t τ t P ( w t ≠ w t − 1 ) ≤ 9 4 λ s . (98) \begin{aligned}
\left|\frac{1}{\sigma_t\tau_t}\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)\right|
\le\frac{9}{4}\mathbb{E}\left[2(1-\pi_\nu(z_1))\frac{\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}\right] \\
&\le\frac{9}{4}\frac{\gamma_t}{\sigma_t\tau_t}\mathbb{P}(w_t\ne w_{t-1})\le\frac{9}{4}\lambda\sqrt{s}. \tag*{(98)}
\end{aligned} σ t τ t 1 ( ν 2 α t 2 + σ t 2 ) ≤ 4 9 E [ 2 ( 1 − π ν ( z 1 )) 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] ≤ 4 9 σ t τ t γ t P ( w t = w t − 1 ) ≤ 4 9 λ s . (98)
Next by (90) and (91), α t σ t τ t ≥ 1 2 ( 1 − 2 ρ ) h ( ν ) \frac{\alpha_t}{\sigma_t\tau_t}\ge\frac{1}{2}(1-2\rho)h(\nu) σ t τ t α t ≥ 2 1 ( 1 − 2 ρ ) h ( ν ) , so by (80) and (88) to (90),
∣ ν 2 α t 2 + σ t 2 σ t τ t − ν ( 1 − 2 ρ ) h ( ν ) ∣ ≤ ∣ ν 2 α t 2 ( σ t τ t ) 2 + σ t 2 ( σ t τ t ) 2 − ν α t σ t τ t ∣ + ν ∣ α t σ t τ t − θ t ∣ + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = σ t 2 ( σ t τ t ) 2 ν 2 α t 2 ( σ t τ t ) 2 + σ t 2 ( σ t τ t ) 2 + ν α t σ t τ t + ν ∣ α t σ t τ t − θ t ∣ + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ σ t 2 ( σ t τ t ) 2 ν α t σ t τ t + ν ∣ α t σ t τ t − θ t ∣ + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ 2 ν ( 1 − 2 ρ ) h ( ν ) σ t 2 ( σ t τ t ) 2 + ν ∣ α t σ t τ t − θ t ∣ + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ [ 8 ν ( 1 − 2 ρ ) h ( ν ) + 2 π + 2 ( 1 − 2 ρ ) ν ] λ s . \begin{aligned}
\left|\frac{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}{\sigma_t\tau_t}-\nu(1-2\rho)h(\nu)\right|
&\le\left|\sqrt{\nu^2\frac{\alpha_t^2}{(\sigma_t\tau_t)^2}+\frac{\sigma_t^2}{(\sigma_t\tau_t)^2}}-\nu\frac{\alpha_t}{\sigma_t\tau_t}\right|+\nu\left|\frac{\alpha_t}{\sigma_t\tau_t}-\theta_t\right|+\nu|\theta_t-(1-2\rho)h(\nu)|\\
&=\frac{\frac{\sigma_t^2}{(\sigma_t\tau_t)^2}}{\sqrt{\nu^2\frac{\alpha_t^2}{(\sigma_t\tau_t)^2}+\frac{\sigma_t^2}{(\sigma_t\tau_t)^2}}+\nu\frac{\alpha_t}{\sigma_t\tau_t}}+\nu\left|\frac{\alpha_t}{\sigma_t\tau_t}-\theta_t\right|+\nu|\theta_t-(1-2\rho)h(\nu)|\\
&\le\frac{\frac{\sigma_t^2}{(\sigma_t\tau_t)^2}}{\nu\frac{\alpha_t}{\sigma_t\tau_t}}+\nu\left|\frac{\alpha_t}{\sigma_t\tau_t}-\theta_t\right|+\nu|\theta_t-(1-2\rho)h(\nu)|\\
&\le\frac{2}{\nu(1-2\rho)h(\nu)}\frac{\sigma_t^2}{(\sigma_t\tau_t)^2}+\nu\left|\frac{\alpha_t}{\sigma_t\tau_t}-\theta_t\right|+\nu|\theta_t-(1-2\rho)h(\nu)|\\
&\le\left[\frac{8}{\nu(1-2\rho)h(\nu)}+\sqrt{\frac{2}{\pi}}+2(1-2\rho)\nu\right]\lambda\sqrt{s}.
\end{aligned} σ t τ t ν 2 α t 2 + σ t 2 − ν ( 1 − 2 ρ ) h ( ν ) ≤ ν 2 ( σ t τ t ) 2 α t 2 + ( σ t τ t ) 2 σ t 2 − ν σ t τ t α t + ν σ t τ t α t − θ t + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ = ν 2 ( σ t τ t ) 2 α t 2 + ( σ t τ t ) 2 σ t 2 + ν σ t τ t α t ( σ t τ t ) 2 σ t 2 + ν σ t τ t α t − θ t + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ ν σ t τ t α t ( σ t τ t ) 2 σ t 2 + ν σ t τ t α t − θ t + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ ν ( 1 − 2 ρ ) h ( ν ) 2 ( σ t τ t ) 2 σ t 2 + ν σ t τ t α t − θ t + ν ∣ θ t − ( 1 − 2 ρ ) h ( ν ) ∣ ≤ [ ν ( 1 − 2 ρ ) h ( ν ) 8 + π 2 + 2 ( 1 − 2 ρ ) ν ] λ s .
Using σ t τ t ≤ Q 0 \sigma_t\tau_t\le Q_0 σ t τ t ≤ Q 0 ,
∣ ν 2 α t 2 + σ t 2 − ν ( 1 − 2 ρ ) h ( ν ) σ t τ t ∣ ≤ Q 0 [ 2 π + 2 ( 1 − 2 ρ ) ν + 8 ν ( 1 − 2 ρ ) h ( ν ) ] λ s . (99) \left|\sqrt{\nu^2\alpha_t^2+\sigma_t^2}-\nu(1-2\rho)h(\nu)\sigma_t\tau_t\right|
\le Q_0\left[\sqrt{\frac{2}{\pi}}+2(1-2\rho)\nu+\frac{8}{\nu(1-2\rho)h(\nu)}\right]\lambda\sqrt{s}. \tag*{(99)} ν 2 α t 2 + σ t 2 − ν ( 1 − 2 ρ ) h ( ν ) σ t τ t ≤ Q 0 [ π 2 + 2 ( 1 − 2 ρ ) ν + ν ( 1 − 2 ρ ) h ( ν ) 8 ] λ s . (99)
The function h h h is Lipschitz because ∣ h ′ ( x ) ∣ = ∣ E [ Z ϕ ′ ′ ( x Z ) ] ∣ ≤ 1 4 E ∣ Z ∣ = 1 4 2 π |h'(x)|=|\mathbb{E}[Z\phi''(xZ)]|\le\frac{1}{4}\mathbb{E}|Z|=\frac{1}{4}\sqrt{\frac{2}{\pi}} ∣ h ′ ( x ) ∣ = ∣ E [ Z ϕ ′′ ( x Z )] ∣ ≤ 4 1 E ∣ Z ∣ = 4 1 π 2 . Thus (98) and (98), and σ t τ t ≤ Q 0 \sigma_t\tau_t\le Q_0 σ t τ t ≤ Q 0 , give,
∣ 1 − σ t τ t h ( ν ( 1 − 2 ρ ) h ( ν ) σ t τ t ) ∣ ≤ σ t τ t ∣ h ( ν 2 α t 2 + σ t 2 ) − h ( ν ( 1 − 2 ρ ) h ( ν ) σ t τ t ) ∣ + σ t τ t ∣ 1 σ t τ t − h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ σ t τ t 4 2 π ∣ ν 2 α t 2 + σ t 2 − ν ( 1 − 2 ρ ) h ( ν ) σ t τ t ∣ + σ t τ t ∣ 1 σ t τ t − h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ [ 9 Q 0 4 + Q 0 2 4 2 π ( 2 π + 2 ( 1 − 2 ρ ) ν + 8 ν ( 1 − 2 ρ ) h ( ν ) ) ] λ s . (100) \begin{aligned}
|1-\sigma_t\tau_t h(\nu(1-2\rho)h(\nu)\sigma_t\tau_t)|
&\le\sigma_t\tau_t\left|h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)-h\left(\nu(1-2\rho)h(\nu)\sigma_t\tau_t\right)\right|+\sigma_t\tau_t\left|\frac{1}{\sigma_t\tau_t}-h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)\right|\\
&\le\frac{\sigma_t\tau_t}{4}\sqrt{\frac{2}{\pi}}\left|\sqrt{\nu^2\alpha_t^2+\sigma_t^2}-\nu(1-2\rho)h(\nu)\sigma_t\tau_t\right|+\sigma_t\tau_t\left|\frac{1}{\sigma_t\tau_t}-h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)\right|\\
&\le\left[\frac{9Q_0}{4}+\frac{Q_0^2}{4}\sqrt{\frac{2}{\pi}}\left(\sqrt{\frac{2}{\pi}}+2(1-2\rho)\nu+\frac{8}{\nu(1-2\rho)h(\nu)}\right)\right]\lambda\sqrt{s}. \tag*{(100)}
\end{aligned} ∣1 − σ t τ t h ( ν ( 1 − 2 ρ ) h ( ν ) σ t τ t ) ∣ ≤ σ t τ t h ( ν 2 α t 2 + σ t 2 ) − h ( ν ( 1 − 2 ρ ) h ( ν ) σ t τ t ) + σ t τ t σ t τ t 1 − h ( ν 2 α t 2 + σ t 2 ) ≤ 4 σ t τ t π 2 ν 2 α t 2 + σ t 2 − ν ( 1 − 2 ρ ) h ( ν ) σ t τ t + σ t τ t σ t τ t 1 − h ( ν 2 α t 2 + σ t 2 ) ≤ [ 4 9 Q 0 + 4 Q 0 2 π 2 ( π 2 + 2 ( 1 − 2 ρ ) ν + ν ( 1 − 2 ρ ) h ( ν ) 8 ) ] λ s . (100)
Now the function a ↦ a h ( ν ( 1 − 2 ρ ) h ( ν ) a ) a\mapsto ah(\nu(1-2\rho)h(\nu)a) a ↦ ah ( ν ( 1 − 2 ρ ) h ( ν ) a ) is strictly increasing since,
d d a [ a h ( ν ( 1 − 2 ρ ) h ( ν ) a ) ] = E [ Z 2 ϕ ′ ′ ( ν ( 1 − 2 ρ ) h ( ν ) a Z ) ] > 0. \frac{\mathrm{d}}{\mathrm{d}a}[ah(\nu(1-2\rho)h(\nu)a)]
=\mathbb{E}\left[Z^2\phi''\left(\nu(1-2\rho)h(\nu)aZ\right)\right]>0. d a d [ ah ( ν ( 1 − 2 ρ ) h ( ν ) a )] = E [ Z 2 ϕ ′′ ( ν ( 1 − 2 ρ ) h ( ν ) a Z ) ] > 0.
Moreover, because ( 1 − 2 ρ ) h ( ν ) 16 9 = r h ( ν r ) (1-2\rho)h(\nu)\frac{16}{9}=rh(\nu r) ( 1 − 2 ρ ) h ( ν ) 9 16 = r h ( ν r ) , then r h ( ν r ) ( 1 − 2 ρ ) h ( ν ) = 1 \frac{rh(\nu r)}{(1-2\rho)h(\nu)}=1 ( 1 − 2 ρ ) h ( ν ) r h ( ν r ) = 1 . Notice also that r ( 1 − 2 ρ ) h ( ν ) = 1 h ( ν r ) ≥ 4 \frac{r}{(1-2\rho)h(\nu)}=\frac{1}{h(\nu r)}\ge4 ( 1 − 2 ρ ) h ( ν ) r = h ( ν r ) 1 ≥ 4 , because h ( ν r ) = E [ ϕ ′ ′ ( ν r Z ) ] ≤ 1 4 h(\nu r)=\mathbb{E}[\phi''(\nu rZ)]\le\frac{1}{4} h ( ν r ) = E [ ϕ ′′ ( ν r Z )] ≤ 4 1 , while σ t τ t ≥ 1 \sigma_t\tau_t\ge1 σ t τ t ≥ 1 . Therefore the interval between σ t τ t \sigma_t\tau_t σ t τ t and r ( 1 − 2 ρ ) h ( ν ) \frac{r}{(1-2\rho)h(\nu)} ( 1 − 2 ρ ) h ( ν ) r is contained in [ 1 , max { Q 0 , r ( 1 − 2 ρ ) h ( ν ) } ] \left[1,\max\left\{Q_0,\frac{r}{(1-2\rho)h(\nu)}\right\}\right] [ 1 , max { Q 0 , ( 1 − 2 ρ ) h ( ν ) r } ] . For every a a a in this interval by (59) ,
d d a [ a h ( ν ( 1 − 2 ρ ) h ( ν ) a ) ] = d d x [ x h ( ν x ) ] ∣ x = ( 1 − 2 ρ ) h ( ν ) a ≥ ( 1 + ν ( 1 − 2 ρ ) h ( ν ) max { Q 0 , r ( 1 − 2 ρ ) h ( ν ) } ) − 3 . \frac{\mathrm{d}}{\mathrm{d}a}[ah(\nu(1-2\rho)h(\nu)a)]
=\left.\frac{\mathrm{d}}{\mathrm{d}x}[xh(\nu x)]\right|_{x=(1-2\rho)h(\nu)a}
\ge\left(1+\nu(1-2\rho)h(\nu)\max\left\{Q_0,\frac{r}{(1-2\rho)h(\nu)}\right\}\right)^{-3}. d a d [ ah ( ν ( 1 − 2 ρ ) h ( ν ) a )] = d x d [ x h ( ν x )] x = ( 1 − 2 ρ ) h ( ν ) a ≥ ( 1 + ν ( 1 − 2 ρ ) h ( ν ) max { Q 0 , ( 1 − 2 ρ ) h ( ν ) r } ) − 3 .
The mean-value theorem and (100) therefore give,
∣ σ t τ t − r ( 1 − 2 ρ ) h ( ν ) ∣ ≤ 9 Q 0 4 + Q 0 2 4 2 π ( 2 π + 2 ( 1 − 2 ρ ) ν + 8 ν ( 1 − 2 ρ ) h ( ν ) ) ( 1 + ν ( 1 − 2 ρ ) h ( ν ) max { Q 0 , r ( 1 − 2 ρ ) h ( ν ) } ) − 3 λ s . \left|\sigma_t\tau_t-\frac{r}{(1-2\rho)h(\nu)}\right|
\le
\frac{\frac{9Q_0}{4}+\frac{Q_0^2}{4}\sqrt{\frac{2}{\pi}}\left(\sqrt{\frac{2}{\pi}}+2(1-2\rho)\nu+\frac{8}{\nu(1-2\rho)h(\nu)}\right)}
{\left(1+\nu(1-2\rho)h(\nu)\max\left\{Q_0,\frac{r}{(1-2\rho)h(\nu)}\right\}\right)^{-3}}
\lambda\sqrt{s}. σ t τ t − ( 1 − 2 ρ ) h ( ν ) r ≤ ( 1 + ν ( 1 − 2 ρ ) h ( ν ) max { Q 0 , ( 1 − 2 ρ ) h ( ν ) r } ) − 3 4 9 Q 0 + 4 Q 0 2 π 2 ( π 2 + 2 ( 1 − 2 ρ ) ν + ν ( 1 − 2 ρ ) h ( ν ) 8 ) λ s .
Finally by Eq. (89) and,
∣ σ J τ J θ J − r ∣ ≤ σ J τ J ∣ θ J − ( 1 − 2 ρ ) h ( ν ) ∣ + ( 1 − 2 ρ ) h ( ν ) ∣ σ J τ J − r ( 1 − 2 ρ ) h ( ν ) ∣ , |\sigma_J\tau_J\theta_J-r| \le\sigma_J\tau_J|\theta_J-(1-2\rho)h(\nu)|+(1-2\rho)h(\nu)\left|\sigma_J\tau_J-\frac{r}{(1-2\rho)h(\nu)}\right|, ∣ σ J τ J θ J − r ∣ ≤ σ J τ J ∣ θ J − ( 1 − 2 ρ ) h ( ν ) ∣ + ( 1 − 2 ρ ) h ( ν ) σ J τ J − ( 1 − 2 ρ ) h ( ν ) r ,
therefore,
∣ σ J τ J θ J − r ∣ ≤ [ 2 ( 1 − 2 ρ ) Q 0 + ( 1 − 2 ρ ) h ( ν ) [ 9 Q 0 4 + Q 0 2 4 2 π ( 2 π + 2 ( 1 − 2 ρ ) + 8 ν ( 1 − 2 ρ ) h ( ν ) ) ] ( 1 + ν ( 1 − 2 ρ ) h ( ν ) max { Q 0 , r ( 1 − 2 ρ ) h ( ν ) } ) − 3 ] λ s . (101) |\sigma_J\tau_J\theta_J-r| \le\left[2(1-2\rho)Q_0+\frac{(1-2\rho)h(\nu)\left[\frac{9Q_0}{4}+\frac{Q_0^2}{4}\sqrt{\frac{2}{\pi}}\left(\sqrt{\frac{2}{\pi}}+2(1-2\rho)+\frac{8}{\sqrt{\nu(1-2\rho)h(\nu)}}\right)\right]}{\left(1+\nu(1-2\rho)h(\nu)\max\left\{Q_0,\frac{r}{(1-2\rho)h(\nu)}\right\}\right)^{-3}}\right]\lambda\sqrt{s}. \tag*{(101)} ∣ σ J τ J θ J − r ∣ ≤ 2 ( 1 − 2 ρ ) Q 0 + ( 1 + ν ( 1 − 2 ρ ) h ( ν ) max { Q 0 , ( 1 − 2 ρ ) h ( ν ) r } ) − 3 ( 1 − 2 ρ ) h ( ν ) [ 4 9 Q 0 + 4 Q 0 2 π 2 ( π 2 + 2 ( 1 − 2 ρ ) + ν ( 1 − 2 ρ ) h ( ν ) 8 ) ] λ s . (101)
We now return to the additive Gaussian event Eq. (75) . Let J ≤ T J\le T J ≤ T be the last time at which the active coordinate is updated. Then w T = w J w_T=w_J w T = w J , and since an update places w J w_J w J at one of the two endpoints of the temporal interval,
w T − r w ∗ = ( σ J τ J θ J − r ) w ∗ + σ J τ J ( r J δ Z J − ϵ J λ ) , w_T-rw^*=(\sigma_J\tau_J\theta_J-r)w^*+\sigma_J\tau_J\left(\frac{r_J}{\sqrt{\delta}}Z_J-\epsilon_J\lambda\right), w T − r w ∗ = ( σ J τ J θ J − r ) w ∗ + σ J τ J ( δ r J Z J − ϵ J λ ) ,
where ϵ J ∈ { − 1 , 1 } \epsilon_J\in\{-1,1\} ϵ J ∈ { − 1 , 1 } . Hence, by the reverse triangle inequality (twice),
∣ w T − r w ∗ ∣ ≥ σ J τ J ( λ − r J δ ∣ Z J ∣ ) − ∣ σ J τ J θ J − r ∣ ∣ w ∗ ∣ . |w_T-rw^*|\ge\sigma_J\tau_J\left(\lambda-\frac{r_J}{\sqrt{\delta}}|Z_J|\right)-|\sigma_J\tau_J\theta_J-r||w^*|. ∣ w T − r w ∗ ∣ ≥ σ J τ J ( λ − δ r J ∣ Z J ∣ ) − ∣ σ J τ J θ J − r ∣∣ w ∗ ∣.
On the event (74), which has probability at least 1 2 \frac{1}{2} 2 1 , we have simultaneously for every j ≤ T j\le T j ≤ T ,
∣ Z j ∣ ≤ λ δ r j − log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] 4 . |Z_j|\le\frac{\lambda\sqrt{\delta}}{r_j}-\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt[4]{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}. ∣ Z j ∣ ≤ r j λ δ − 4 log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] .
Therefore,
λ − r j δ ∣ Z j ∣ ≥ r j 4 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ≥ h ( ν ) 4 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] , \lambda-\frac{r_j}{\sqrt{\delta}}|Z_j|\ge\frac{r_j}{4\sqrt{\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}\ge\frac{\sqrt{h(\nu)}}{4\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}, λ − δ r j ∣ Z j ∣ ≥ 4 δ r j log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ≥ 4 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ,
where we used r j ≥ h ( ν ) 8 r_j\ge\sqrt{\frac{h(\nu)}{8}} r j ≥ 8 h ( ν ) (see Eq. (74) ). Since σ J τ J ≥ 1 \sigma_J\tau_J\ge1 σ J τ J ≥ 1 , it follows that the gap,
σ J τ J ( λ − r j δ ∣ Z j ∣ ) ≥ h ( ν ) 4 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] . \sigma_J\tau_J\left(\lambda-\frac{r_j}{\sqrt{\delta}}|Z_j|\right)\ge\frac{\sqrt{h(\nu)}}{4\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}. σ J τ J ( λ − δ r j ∣ Z j ∣ ) ≥ 4 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] .
By Eq. (101) , ∣ σ J τ J θ J − r ∣ ≤ K λ s |\sigma_J\tau_J\theta_J-r|\le K\lambda\sqrt{s} ∣ σ J τ J θ J − r ∣ ≤ K λ s . Hence, if we restrict to active coordinates satisfying,
0 < ∣ w ∗ ∣ ≤ h ( ν ) 16 K λ 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] , 0<|w^*|\le\frac{\sqrt{h(\nu)}}{16K\lambda\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}, 0 < ∣ w ∗ ∣ ≤ 16 K λ 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ,
then,
∣ σ J τ J θ J − r ∣ ∣ w ∗ ∣ ≤ h ( ν ) 16 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] . |\sigma_J\tau_J\theta_J-r||w^*|\le\frac{\sqrt{h(\nu)}}{16\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}. ∣ σ J τ J θ J − r ∣∣ w ∗ ∣ ≤ 16 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] .
Thus the calibration term consumes at most one quarter of the gap. Combining the estimates gives,
∣ w T − r w ⋆ ∣ ≥ h ( ν ) 4 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] = h ( ν ) 16 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] = 3 h ( ν ) 16 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] . \begin{aligned}
|w_T-rw^\star| \ge\frac{\sqrt{h(\nu)}}{4\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}=\frac{\sqrt{h(\nu)}}{16\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}} \\
&=\frac{3\sqrt{h(\nu)}}{16\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}.
\end{aligned} ∣ w T − r w ⋆ ∣ ≥ 4 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] = 16 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] = 16 8 δ 3 h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] .
The same bound holds if the coordinate never updates. Indeed, since r j ≥ h ( ν ) 8 r_j\ge\sqrt{\frac{h(\nu)}{8}} r j ≥ 8 h ( ν ) (Eq. (74) ), we have
log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ≤ log [ ( 2 Φ ( − λ 8 δ h ( ν ) ) ) − 1 ] ≤ log [ ( 2 π exp ( − 1 2 ( λ 8 δ h ( ν ) + 1 ) 2 ) ) − 1 ] = 1 2 ( λ 8 δ h ( ν ) + 1 ) 2 + 1 2 log π 2 ≤ 8 λ 2 δ h ( ν ) [ 1 2 ( 1 + 1 Φ − 1 ( 3 4 ) ) 2 + 1 2 ( Φ − 1 ( 3 4 ) ) 2 log π 2 ] . \begin{aligned}
\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]\le\log\left[\left(2\Phi\left(-\lambda\sqrt{\frac{8\delta}{h(\nu)}}\right)\right)^{-1}\right] \\
&\le\log\left[\left(\sqrt{\frac{2}{\pi}}\exp\left(-\frac{1}{2}\left(\lambda\sqrt{\frac{8\delta}{h(\nu)}}+1\right)^2\right)\right)^{-1}\right] \\
&=\frac{1}{2}\left(\lambda\sqrt{\frac{8\delta}{h(\nu)}}+1\right)^2+\frac{1}{2}\log\frac{\pi}{2} \\
&\le\frac{8\lambda^2\delta}{h(\nu)}\left[\frac{1}{2}\left(1+\frac{1}{\Phi^{-1}\left(\frac{3}{4}\right)}\right)^2+\frac{1}{2\left(\Phi^{-1}\left(\frac{3}{4}\right)\right)^2}\log\frac{\pi}{2}\right].
\end{aligned} log ( k = 1 ∑ T 2Φ ( − r k λ δ ) ) − 1 ≤ log ( 2Φ ( − λ h ( ν ) 8 δ ) ) − 1 ≤ log π 2 exp − 2 1 ( λ h ( ν ) 8 δ + 1 ) 2 − 1 = 2 1 ( λ h ( ν ) 8 δ + 1 ) 2 + 2 1 log 2 π ≤ h ( ν ) 8 λ 2 δ 2 1 ( 1 + Φ − 1 ( 4 3 ) 1 ) 2 + 2 ( Φ − 1 ( 4 3 ) ) 2 1 log 2 π .
where in the last inequality we used λ 8 δ h ( ν ) ≥ Φ − 1 ( 3 4 ) \lambda\sqrt{\frac{8\delta}{h(\nu)}}\ge\Phi^{-1}\left(\frac{3}{4}\right) λ h ( ν ) 8 δ ≥ Φ − 1 ( 4 3 ) (recall we have shown it must hold λ → ∞ \lambda\to\infty λ → ∞ ). Equivalently, since conditionally on being active w ⋆ = ν s G w^\star=\frac{\nu}{\sqrt{s}}G w ⋆ = s ν G , the preceding lower bound holds for every updated active coordinate satisfying
∣ G ∣ ≤ h ( ν ) 16 K λ ν 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] . |G|\le\frac{\sqrt{h(\nu)}}{16K\lambda\nu\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}. ∣ G ∣ ≤ 16 K λ ν 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] .
On the other hand, if the coordinate never updates, then w T = 0 w_T=0 w T = 0 and ∣ w T − r w ⋆ ∣ = r ν ∣ G ∣ s |w_T-rw^\star|=\frac{r\nu|G|}{\sqrt{s}} ∣ w T − r w ⋆ ∣ = s r ν ∣ G ∣ . We confine ∣ G ∣ |G| ∣ G ∣ to a subinterval so that,
3 h ( ν ) s 16 r ν 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ≤ ∣ G ∣ ≤ h ( ν ) 16 K λ ν 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] . \frac{3\sqrt{h(\nu)}\sqrt{s}}{16r\nu\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}\le|G|\le\frac{\sqrt{h(\nu)}}{16K\lambda\nu\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}. 16 r ν 8 δ 3 h ( ν ) s log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ≤ ∣ G ∣ ≤ 16 K λ ν 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] .
By the contradicting assumption (Eq. (64) ), ν 2 ( α t − r ) 2 ≤ c ⋆ s log ( 1 s ) ≤ 1 4 r 2 ν 2 \nu^2(\alpha_t-r)^2\le c_\star s\log\left(\frac{1}{s}\right)\le\frac{1}{4}r^2\nu^2 ν 2 ( α t − r ) 2 ≤ c ⋆ s log ( s 1 ) ≤ 4 1 r 2 ν 2 , after choosing s s s sufficiently small and hence α t ≤ 3 r 2 \alpha_t\le\frac{3r}{2} α t ≤ 2 3 r . Therefore, σ t τ t = α t σ t τ t ≤ 3 r ( 1 − 2 ρ ) h ( ν ) = 3 h ( ν r ) ≤ 1 + ν \sigma_t\tau_t=\frac{\alpha_t}{\sigma_t\tau_t}\le\frac{3r}{(1-2\rho)h(\nu)}=\frac{3}{h(\nu r)}\le1+\nu σ t τ t = σ t τ t α t ≤ ( 1 − 2 ρ ) h ( ν ) 3 r = h ( ν r ) 3 ≤ 1 + ν , which follows from α t σ t τ t ≥ 1 2 ( 1 − 2 ρ ) h ( ν ) \frac{\alpha_t}{\sigma_t\tau_t}\ge\frac{1}{2}(1-2\rho)h(\nu) σ t τ t α t ≥ 2 1 ( 1 − 2 ρ ) h ( ν ) as we already established for sufficiently small s s s . With this upper bound we have K ≲ ( 1 + ν ) 6 ν K\lesssim\frac{(1+\nu)^6}{\nu} K ≲ ν ( 1 + ν ) 6 , we decrease in Eq. (87) c ⋆ c_\star c ⋆ so that G min G max = 3 K λ s 8 ≤ 1 2 \frac{G_{\min}}{G_{\max}}=\frac{3K\lambda\sqrt{s}}{8}\le\frac{1}{2} G m a x G m i n = 8 3 K λ s ≤ 2 1 , the interval G min ≤ ∣ G ∣ ≤ G max G_{\min}\le|G|\le G_{\max} G m i n ≤ ∣ G ∣ ≤ G m a x has one-sided length at least G max 2 \frac{G_{\max}}{2} 2 G m a x , namely
h ( ν ) 32 K λ ν 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] . \frac{\sqrt{h(\nu)}}{32K\lambda\nu\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}. 32 K λ ν 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] .
Moreover, the preceding upper bound on the logarithm implies that the upper endpoint of this interval is at most
1 16 K ν 1 2 ( 1 + 1 Φ − 1 ( 3 4 ) ) 2 + 1 2 ( Φ − 1 ( 3 4 ) ) 2 log π 2 . \frac{1}{16K\nu}\sqrt{\frac{1}{2}\left(1+\frac{1}{\Phi^{-1}\left(\frac{3}{4}\right)}\right)^2+\frac{1}{2\left(\Phi^{-1}\left(\frac{3}{4}\right)\right)^2}\log\frac{\pi}{2}}. 16 K ν 1 2 1 ( 1 + Φ − 1 ( 4 3 ) 1 ) 2 + 2 ( Φ − 1 ( 4 3 ) ) 2 1 log 2 π .
Hence the standard Gaussian density throughout this interval is bounded below by
φ ( 1 16 K ν 1 2 ( 1 + 1 Φ − 1 ( 3 4 ) ) 2 + 1 2 ( Φ − 1 ( 3 4 ) ) 2 log π 2 ) . \varphi\left(\frac{1}{16K\nu}\sqrt{\frac{1}{2}\left(1+\frac{1}{\Phi^{-1}\left(\frac{3}{4}\right)}\right)^2+\frac{1}{2\left(\Phi^{-1}\left(\frac{3}{4}\right)\right)^2}\log\frac{\pi}{2}}\right). φ 16 K ν 1 2 1 ( 1 + Φ − 1 ( 4 3 ) 1 ) 2 + 2 ( Φ − 1 ( 4 3 ) ) 2 1 log 2 π .
Therefore, integrating over these active coordinates,
ν 2 ( α T − r ) 2 + σ T 2 = E [ ( w T − r w ⋆ ) 2 ] ≥ s ( 3 h ( ν ) 16 8 δ ) 2 log 2 [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] × h ( ν ) 16 K λ ν 8 δ log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] φ ( 1 16 K ν 1 2 ( 1 + 1 Φ − 1 ( 3 4 ) ) 2 + 1 2 ( Φ − 1 ( 3 4 ) ) 2 log π 2 ) \begin{aligned}
\nu^2(\alpha_T-r)^2+\sigma_T^2=\mathbb{E}[(w_T-rw^\star)^2]\ge s\left(\frac{3\sqrt{h(\nu)}}{16\sqrt{8\delta}}\right)^2\frac{\log^2\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]} \\
&\times\frac{\sqrt{h(\nu)}}{16K\lambda\nu\sqrt{8\delta}}\frac{\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\sqrt{\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}}\varphi\left(\frac{1}{16K\nu}\sqrt{\frac{1}{2}\left(1+\frac{1}{\Phi^{-1}\left(\frac{3}{4}\right)}\right)^2+\frac{1}{2\left(\Phi^{-1}\left(\frac{3}{4}\right)\right)^2}\log\frac{\pi}{2}}\right)
\end{aligned} ν 2 ( α T − r ) 2 + σ T 2 = E [( w T − r w ⋆ ) 2 ] ≥ s ( 16 8 δ 3 h ( ν ) ) 2 log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log 2 [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] × 16 K λ ν 8 δ h ( ν ) log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] φ 16 K ν 1 2 1 ( 1 + Φ − 1 ( 4 3 ) 1 ) 2 + 2 ( Φ − 1 ( 4 3 ) ) 2 1 log 2 π
Here we used both signs of G G G . To eliminate λ \lambda λ , some j ≤ T j\le T j ≤ T satisfies 2 Φ ( − λ δ r j ) ≥ 1 T ∑ k = 1 T 2 Φ ( − λ δ r k ) 2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\ge\frac{1}{T}\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right) 2Φ ( − r j λ δ ) ≥ T 1 ∑ k = 1 T 2Φ ( − r k λ δ ) . Therefore, using 2 Φ ( − x ) ≤ e − x 2 / 2 2\Phi(-x)\le e^{-x^2/2} 2Φ ( − x ) ≤ e − x 2 /2 and r j ≤ 1 r_j\le1 r j ≤ 1 ,
1 T ∑ k = 1 T 2 Φ ( − λ δ r k ) ≤ 2 Φ ( − λ δ r j ) ≤ exp ( − λ 2 δ 2 r j 2 ) ≤ exp ( − λ 2 δ 2 ) , \frac{1}{T}\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\le2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\le\exp\left(-\frac{\lambda^2\delta}{2r_j^2}\right)\le\exp\left(-\frac{\lambda^2\delta}{2}\right), T 1 k = 1 ∑ T 2Φ ( − r k λ δ ) ≤ 2Φ ( − r j λ δ ) ≤ exp ( − 2 r j 2 λ 2 δ ) ≤ exp ( − 2 λ 2 δ ) ,
and hence λ δ ≤ 2 ( log ( T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ) \lambda\sqrt{\delta}\le\sqrt{2\left(\log(T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]\right)} λ δ ≤ 2 ( log ( T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ) ,
ν 2 ( α T − r ) 2 + σ T 2 ≥ s ( 3 h ( ν ) 16 8 δ ) 2 log 2 [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ( log ( 2 T ) + log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ) 2 × h ( ν ) 16 K λ ν 16 φ ( 1 16 K ν 1 2 ( 1 + 1 Φ − 1 ( 3 4 ) ) 2 + 1 2 ( Φ − 1 ( 3 4 ) ) 2 log π 2 ) . \begin{aligned}
\nu^2(\alpha_T-r)^2+\sigma_T^2\ge s\left(\frac{3\sqrt{h(\nu)}}{16\sqrt{8\delta}}\right)^2\frac{\log^2\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]}{\left(\log(2T)+\log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]\right)^2} \\
&\times\frac{\sqrt{h(\nu)}}{16K\lambda\nu\sqrt{16}}\varphi\left(\frac{1}{16K\nu}\sqrt{\frac{1}{2}\left(1+\frac{1}{\Phi^{-1}\left(\frac{3}{4}\right)}\right)^2+\frac{1}{2\left(\Phi^{-1}\left(\frac{3}{4}\right)\right)^2}\log\frac{\pi}{2}}\right).
\end{aligned} ν 2 ( α T − r ) 2 + σ T 2 ≥ s ( 16 8 δ 3 h ( ν ) ) 2 ( log ( 2 T ) + log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ) 2 log 2 [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] × 16 K λ ν 16 h ( ν ) φ 16 K ν 1 2 1 ( 1 + Φ − 1 ( 4 3 ) 1 ) 2 + 2 ( Φ − 1 ( 4 3 ) ) 2 1 log 2 π .
By Eq. (70) , log [ ( ∑ k = 1 T 2 Φ ( − λ δ r k ) ) − 1 ] ≥ 1 2 log ( 1 s ) \log\left[\left(\sum_{k=1}^{T}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)^{-1}\right]\ge\frac{1}{2}\log\left(\frac{1}{s}\right) log [ ( ∑ k = 1 T 2Φ ( − r k λ δ ) ) − 1 ] ≥ 2 1 log ( s 1 ) . Since x ↦ x 3 ( log ( 2 T ) + x ) 2 x\mapsto\frac{x^3}{(\log(2T)+x)^2} x ↦ ( l o g ( 2 T ) + x ) 2 x 3 is strictly increasing for x > 0 x>0 x > 0 ,
ν 2 ( α t − r ) 2 + σ t 2 ≥ 9 h ( ν ) 3 2 1048576 K ν δ φ ( 1 16 K ν 1 2 ( 1 + 1 Φ − 1 ( 3 4 ) ) 2 + 1 2 ( Φ − 1 ( 3 4 ) ) 2 log π 2 ) s log 3 ( 1 s ) ( log ( 2 T ) + 1 2 log ( 1 s ) ) 2 . \begin{aligned}
\nu^2(\alpha_t-r)^2+\sigma_t^2 \\
&\ge\frac{9h(\nu)^{\frac{3}{2}}}{1048576K\nu\delta}\varphi\left(\frac{1}{16K\nu}\sqrt{\frac{1}{2}\left(1+\frac{1}{\Phi^{-1}\left(\frac{3}{4}\right)}\right)^2+\frac{1}{2\left(\Phi^{-1}\left(\frac{3}{4}\right)\right)^2}\log\frac{\pi}{2}}\right)s\frac{\log^3\left(\frac{1}{s}\right)}{\left(\log(2T)+\frac{1}{2}\log\left(\frac{1}{s}\right)\right)^2}.
\end{aligned} ν 2 ( α t − r ) 2 + σ t 2 ≥ 1048576 K ν δ 9 h ( ν ) 2 3 φ 16 K ν 1 2 1 ( 1 + Φ − 1 ( 4 3 ) 1 ) 2 + 2 ( Φ − 1 ( 4 3 ) ) 2 1 log 2 π s ( log ( 2 T ) + 2 1 log ( s 1 ) ) 2 log 3 ( s 1 ) .
Thus Eq. (64) is contradicted whenever c ⋆ < 9 h ( ν ) 3 2 1048576 K ν δ φ ( 1 16 K ν 1 2 ( 1 + 1 Φ − 1 ( 3 4 ) ) 2 + 1 2 ( Φ − 1 ( 3 4 ) ) 2 log π 2 ) c_\star<\frac{9h(\nu)^{\frac{3}{2}}}{1048576K\nu\delta}\varphi\left(\frac{1}{16K\nu}\sqrt{\frac{1}{2}\left(1+\frac{1}{\Phi^{-1}\left(\frac{3}{4}\right)}\right)^2+\frac{1}{2\left(\Phi^{-1}\left(\frac{3}{4}\right)\right)^2}\log\frac{\pi}{2}}\right) c ⋆ < 1048576 K ν δ 9 h ( ν ) 2 3 φ 16 K ν 1 2 1 ( 1 + Φ − 1 ( 4 3 ) 1 ) 2 + 2 ( Φ − 1 ( 4 3 ) ) 2 1 log 2 π . Taking the minimum of this positive constant and the constants obtained in the preceding cases, the proof yields
ν 2 ( α t − r ) 2 + σ t 2 ≥ c ′ ′ s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 2 \nu^2(\alpha_t-r)^2+\sigma_t^2\ge c''s\frac{\log^3\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^2} ν 2 ( α t − r ) 2 + σ t 2 ≥ c ′′ s ( log ( s 1 ) + log ( 2 T ) ) 2 log 3 ( s 1 )
for c ′ ′ = min { 9 h ( ν ) 3 2 1048576 K ν δ φ ( 1 16 K ν 1 2 ( 1 + 1 Φ − 1 ( 3 4 ) ) 2 + 1 2 ( Φ − 1 ( 3 4 ) ) 2 log π 2 ) , r 2 φ ( 1 ) 384 8 δ 2 ( 1 − 2 ρ ) 3 } c''=\min\left\{\frac{9h(\nu)^{\frac{3}{2}}}{1048576K\nu\delta}\varphi\left(\frac{1}{16K\nu}\sqrt{\frac{1}{2}\left(1+\frac{1}{\Phi^{-1}\left(\frac{3}{4}\right)}\right)^2+\frac{1}{2\left(\Phi^{-1}\left(\frac{3}{4}\right)\right)^2}\log\frac{\pi}{2}}\right),\frac{r^2\varphi(1)}{384\sqrt{8\delta}^{\,2}(1-2\rho)^3}\right\} c ′′ = min ⎩ ⎨ ⎧ 1048576 K ν δ 9 h ( ν ) 2 3 φ 16 K ν 1 2 1 ( 1 + Φ − 1 ( 4 3 ) 1 ) 2 + 2 ( Φ − 1 ( 4 3 ) ) 2 1 log 2 π , 384 8 δ 2 ( 1 − 2 ρ ) 3 r 2 φ ( 1 ) ⎭ ⎬ ⎫ . Instead of choosing K K K as we chose, so that ∣ σ T J T θ J − r ∣ ≤ K λ s |\sigma_TJ_T\theta_J-r|\le K\lambda\sqrt{s} ∣ σ T J T θ J − r ∣ ≤ K λ s , and thereby risking an exponential dependence on the parameters due to the behavior of φ \varphi φ , to make sure it remains uniformly bounded from below, we set
K = max { K , 1 16 ν 1 2 ( 1 + 1 Φ − 1 ( 3 4 ) ) 2 + 1 2 ( Φ − 1 ( 3 4 ) ) 2 log π 2 } . K=\max\left\{K,\frac{1}{16\nu}\sqrt{\frac{1}{2}\left(1+\frac{1}{\Phi^{-1}\left(\frac{3}{4}\right)}\right)^2+\frac{1}{2\left(\Phi^{-1}\left(\frac{3}{4}\right)\right)^2}\log\frac{\pi}{2}}\right\}. K = max ⎩ ⎨ ⎧ K , 16 ν 1 2 1 ( 1 + Φ − 1 ( 4 3 ) 1 ) 2 + 2 ( Φ − 1 ( 4 3 ) ) 2 1 log 2 π ⎭ ⎬ ⎫ .
With this choice, the argument of φ \varphi φ is below 1, so it remains uniformly bounded from below by φ ( 1 ) \varphi(1) φ ( 1 ) and the same lower-bound argument applies. Using Eq. (61) h ( ν ) ≳ 1 1 + ν h(\nu) \gtrsim\frac{1}{1+\nu} h ( ν ) ≳ 1 + ν 1 , and by the relation from Lemma 16 r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) rh(\nu_r)=(1-2\rho)h(\nu) r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) we get r ≳ 1 − 2 ρ 1 + ν r \gtrsim\frac{1-2\rho}{1+\nu} r ≳ 1 + ν 1 − 2 ρ , d d a ( a h ( ν a ) ) ≳ ( 1 + ν a ) − 3 \frac{d}{da}(ah(\nu_a))\gtrsim(1+\nu_a)^{-3} d a d ( ah ( ν a )) ≳ ( 1 + ν a ) − 3 by Eq. (59) , we obtain Q 0 ≲ ( 1 + ν ) 5 ν 2 ( 1 − 2 ρ ) 2 Q_0 \lesssim\frac{(1+\nu)^5}{\nu^2(1-2\rho)^2} Q 0 ≲ ν 2 ( 1 − 2 ρ ) 2 ( 1 + ν ) 5 and K ≲ ν 7 ( 1 − 2 ρ ) 7 ν 8 ( 1 − 2 ρ ) 7 K\lesssim\frac{\nu^7(1-2\rho)^7}{\nu^8(1-2\rho)^7} K ≲ ν 8 ( 1 − 2 ρ ) 7 ν 7 ( 1 − 2 ρ ) 7 . Consequently, c ′ ′ ≳ ν 7 ( 1 − 2 ρ ) 7 ( 1 + ν ) 26 ( 1 + δ ) 3 / 2 c^{\prime\prime}\gtrsim\frac{\nu^7(1-2\rho)^7}{(1+\nu)^{26}(1+\delta)^{3/2}} c ′′ ≳ ( 1 + ν ) 26 ( 1 + δ ) 3/2 ν 7 ( 1 − 2 ρ ) 7 (this choice is also sufficient to make 3 K λ s r ≤ 1 2 \frac{3K\lambda\sqrt{s}}{r}\le\frac{1}{2} r 3 K λ s ≤ 2 1 by r ≳ 1 − 2 ρ 1 + ν r\gtrsim\frac{1-2\rho}{1+\nu} r ≳ 1 + ν 1 − 2 ρ ). By Proposition 22 ,
L ( w T ) − inf w L ( w ) ≥ C ν 7 ( 1 − 2 ρ ) 7 ( 1 + ν ) 3 ( 1 + δ ) 3 / 2 s log 3 ( 1 s ) ( log ( 1 s ) + log ( 2 T ) ) 2 . \mathcal{L}(w_T)-\inf_w\mathcal{L}(w)\ge C\frac{\nu^7(1-2\rho)^7}{(1+\nu)^3(1+\delta)^{3/2}}\frac{s\log^3\left(\frac{1}{s}\right)}{\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)^2}. L ( w T ) − w inf L ( w ) ≥ C ( 1 + ν ) 3 ( 1 + δ ) 3/2 ν 7 ( 1 − 2 ρ ) 7 ( log ( s 1 ) + log ( 2 T ) ) 2 s log 3 ( s 1 ) .
for some universal constant C > 0 C>0 C > 0 . Collecting all the conditions in which s s s was required to be sufficiently small, it is enough to require, for universal constants C 1 , C 2 , C 3 , C 4 > 0 C_1,C_2,C_3,C_4>0 C 1 , C 2 , C 3 , C 4 > 0 ,
s ≤ C 1 , s log 3 ( 1 s ) ≤ C 2 , (102) s\le C_1,\qquad s\log^3\left(\frac{1}{s}\right)\le C_2, \tag*{(102)} s ≤ C 1 , s log 3 ( s 1 ) ≤ C 2 , (102)
s ( 2 + 655360 3 δ c ⋆ log 2 ( 1 s ) ) ≤ δ h ( ν ) 4 , (103) s\left(2+\frac{655360}{3}\delta c_{\star}\log^2\left(\frac{1}{s}\right)\right)\le\frac{\delta h(\nu)}{4}, \tag*{(103)} s ( 2 + 3 655360 δ c ⋆ log 2 ( s 1 ) ) ≤ 4 δ h ( ν ) , (103)
s ( 2 + 655360 3 δ c ⋆ log 2 ( 1 s ) ) ≤ min { 1 , δ } Φ − 1 ( 3 4 ) h ( ν ) 8 δ , (104) \sqrt{s}\left(2+\frac{655360}{3}\delta c_{\star}\log^2\left(\frac{1}{s}\right)\right)\le\min\{1,\delta\}\Phi^{-1}\left(\frac{3}{4}\right)\sqrt{\frac{h(\nu)}{8\delta}}, \tag*{(104)} s ( 2 + 3 655360 δ c ⋆ log 2 ( s 1 ) ) ≤ min { 1 , δ } Φ − 1 ( 4 3 ) 8 δ h ( ν ) , (104)
c ⋆ s log ( 1 s ) ≤ C 3 r 2 ν 2 , (105) c_{\star}s\log\left(\frac{1}{s}\right)\le C_3r^2\nu^2, \tag*{(105)} c ⋆ s log ( s 1 ) ≤ C 3 r 2 ν 2 , (105)
λ s ( 2 + 655360 3 δ c ⋆ log 3 ( 1 s ) log ( 1 s ) + log ( 2 T ) ) ≤ 1 , (106) \lambda\sqrt{s}\left(2+\frac{655360}{3}\delta c_{\star}\frac{\log^3\left(\frac{1}{s}\right)}{\log\left(\frac{1}{s}\right)+\log(2T)}\right)\le1, \tag*{(106)} λ s ( 2 + 3 655360 δ c ⋆ log ( s 1 ) + log ( 2 T ) log 3 ( s 1 ) ) ≤ 1 , (106)
Q 3 s ν 2 δ ≤ 1 4 ( 1 − 2 ρ ) h ( ν ) . (107) Q\sqrt{\frac{3s}{\nu^2\delta}}\le\frac{1}{4}(1-2\rho)h(\nu). \tag*{(107)} Q ν 2 δ 3 s ≤ 4 1 ( 1 − 2 ρ ) h ( ν ) . (107)
The remaining smallness conditions on λ s \lambda\sqrt{s} λ s in Eqs. (92) to (93) are satisfied by taking c ⋆ c_{\star} c ⋆ sufficiently small and therefore impose no additional restriction on s 0 s_0 s 0 . Using Eq. (61) h ( ν ) ≳ 1 1 + ν h(\nu)\gtrsim\frac{1}{1+\nu} h ( ν ) ≳ 1 + ν 1 and c ⋆ ≲ ν 7 ( 1 − 2 ρ ) 7 ( 1 + ν ) 26 ( 1 + δ ) 3 / 2 c_{\star}\lesssim\frac{\nu^7(1-2\rho)^7}{(1+\nu)^{26}(1+\delta)^{3/2}} c ⋆ ≲ ( 1 + ν ) 26 ( 1 + δ ) 3/2 ν 7 ( 1 − 2 ρ ) 7 , we have δ c ⋆ ≲ 1 \delta c_{\star}\lesssim1 δ c ⋆ ≲ 1 and c ⋆ ≲ r 2 ν 2 c_{\star}\lesssim r^2\nu^2 c ⋆ ≲ r 2 ν 2 , so (S1) and (S4) follow by taking s s s smaller than a universal constant. Thus it is sufficient to require
s log 2 ( 1 s ) ≲ δ 1 + ν , s log 4 ( 1 s ) ≲ 1 δ ( 1 + ν ) . s\log^2\left(\frac{1}{s}\right)\lesssim\frac{\delta}{1+\nu},\qquad s\log^4\left(\frac{1}{s}\right)\lesssim\frac{1}{\delta(1+\nu)}. s log 2 ( s 1 ) ≲ 1 + ν δ , s log 4 ( s 1 ) ≲ δ ( 1 + ν ) 1 .
For (S6) we need to require s ≤ C 4 ν 2 δ r 2 h ( ν r ) 4 s\le C_4\nu^2\delta r^2h(\nu_r)^4 s ≤ C 4 ν 2 δ r 2 h ( ν r ) 4 . Consequently, for a sufficiently small universal constant c > 0 c>0 c > 0 , it is enough to take
s ≤ s 0 = c min { δ ( 1 + ν ) log + 2 ( e ( 1 + ν ) δ ) , 1 δ ( 1 + ν ) log + 4 ( e δ ( 1 + ν ) ) , δ ν 2 ( 1 − 2 ρ ) 2 ( 1 + ν ) 4 } , s\le s_0=c\min\left\{\frac{\delta}{(1+\nu)\log_+^2\left(\frac{e(1+\nu)}{\delta}\right)},\frac{1}{\delta(1+\nu)\log_+^4\left(e\delta(1+\nu)\right)},\frac{\delta\nu^2(1-2\rho)^2}{(1+\nu)^4}\right\}, s ≤ s 0 = c min ⎩ ⎨ ⎧ ( 1 + ν ) log + 2 ( δ e ( 1 + ν ) ) δ , δ ( 1 + ν ) log + 4 ( eδ ( 1 + ν ) ) 1 , ( 1 + ν ) 4 δ ν 2 ( 1 − 2 ρ ) 2 ⎭ ⎬ ⎫ ,
with the convention that each logarithmic denominator is replaced by a universal constant whenever its argument is bounded.
Proof of Lemma 4
Full Lemma 4. There exist universal constants C , ν > 0 C, \nu> 0 C , ν > 0 , a constant c ν , δ = c ( ν , δ ) > 0 c_{\nu,\delta}=c(\nu,\delta)>0 c ν , δ = c ( ν , δ ) > 0 and
λ = ν [ ( 1 + ν ) 7 δ ( 1 + ν 2 + 1 δ ) + ( 1 + ν ) 10 δ 2 ] + φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) log ( 2 T ) , \lambda=\sqrt{\nu\left[\frac{(1+\nu)^7}{\delta}\left(1+\nu^2+\frac{1}{\delta}\right)+\frac{(1+\nu)^{10}}{\delta^2}\right]+\frac{\varphi(1)}{2(1+e)^{2\delta(1+\nu)}}\log(2T)}, λ = ν [ δ ( 1 + ν ) 7 ( 1 + ν 2 + δ 1 ) + δ 2 ( 1 + ν ) 10 ] + 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) log ( 2 T ) , such that, for all T ≥ 1 T\ge1 T ≥ 1 and s ∈ ( 0 , 1 ) s\in(0,1) s ∈ ( 0 , 1 ) ,
L ( w T ) − inf w L ( w ) ≤ C ( 1 + ν ) 11 δ log ( 2 T ) + C ( 1 + ν ) 30 log 2 ( 2 T ) ( 1 + ν 2 + 1 δ ) 2 + C ( 1 + ν ) 8 exp ( − c ν , δ T log ( 2 T ) ) . \mathcal{L}(w_T)-\inf_w\mathcal{L}(w)\le\frac{C(1+\nu)^{11}}{\delta\log(2T)}+\frac{C(1+\nu)^{30}}{\log^2(2T)}\left(1+\nu^2+\frac{1}{\delta}\right)^2+C(1+\nu)^8\exp\left(-c_{\nu,\delta}\sqrt{\frac{T}{\log(2T)}}\right). L ( w T ) − w inf L ( w ) ≤ δ log ( 2 T ) C ( 1 + ν ) 11 + log 2 ( 2 T ) C ( 1 + ν ) 30 ( 1 + ν 2 + δ 1 ) 2 + C ( 1 + ν ) 8 exp ( − c ν , δ log ( 2 T ) T ) .
Proof. Let w t w_t w t follow the temporal L1 regularization, by Lemma 20,
w t = Π [ σ t τ t ( θ t w ⋆ + r t δ Z t − λ ) , σ t τ t ( θ t w ⋆ + r t δ Z t + λ ) ] ( w t − 1 ) , w_t=\Pi_{\left[\sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t-\lambda\right),\sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t+\lambda\right)\right]}(w_{t-1}), w t = Π [ σ t τ t ( θ t w ⋆ + δ r t Z t − λ ) , σ t τ t ( θ t w ⋆ + δ r t Z t + λ ) ] ( w t − 1 ) ,
We divide the proof into several steps. Recall γ t σ t τ t = 1 δ P ( w t ≠ w t − 1 ) \frac{\gamma_t}{\sigma_t\tau_t}=\frac{1}{\delta}\mathbb{P}(w_t\ne w_{t-1}) σ t τ t γ t = δ 1 P ( w t = w t − 1 ) . The equation for θ t \theta_t θ t and the proximal relation (Eq. (65) ) give
θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . \theta_t=2\mathbb{E}\left[\pi_\rho'(\nu Z_1)\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)\right]. θ t = 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] .
Consequently, 0 < θ t ≤ 2 E [ π ρ ′ ( ν Z 1 ) ] = 2 ( 1 − 2 ρ ) h ( ν ) ≤ 1 2 0<\theta_t\le2\mathbb{E}[\pi_\rho'(\nu Z_1)]=2(1-2\rho)h(\nu)\le\frac{1}{2} 0 < θ t ≤ 2 E [ π ρ ′ ( ν Z 1 )] = 2 ( 1 − 2 ρ ) h ( ν ) ≤ 2 1 , as well as Eq. (98) gives,
∣ 1 σ t τ t − h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ γ t σ t τ t . (108) \left|\frac{1}{\sigma_t\tau_t}-h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)\right|\le\frac{\gamma_t}{\sigma_t\tau_t}. \tag*{(108)} σ t τ t 1 − h ( ν 2 α t 2 + σ t 2 ) ≤ σ t τ t γ t . (108)
The fourth scalar equation with the proximal relation (Eq. (65) ) give
r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] ≤ 0 < ϕ ′ ( u ) ≤ 1 2 E [ 1 − π ρ ( ν Z 1 ) ] = 1. r_t^2=2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)^2\right]\underset{0<\phi'(u)\le1}{\le}2\mathbb{E}[1-\pi_\rho(\nu Z_1)]=1. r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] 0 < ϕ ′ ( u ) ≤ 1 ≤ 2 E [ 1 − π ρ ( ν Z 1 )] = 1.
The sixth scalar equation can be written as
1 σ t τ t = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . (109) \frac{1}{\sigma_t\tau_t}=\mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))\frac{\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}\right]. \tag*{(109)} σ t τ t 1 = E [ 2 ( 1 − π ρ ( ν Z 1 )) 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . (109)
Since 1 − π ρ ( ν Z 1 ) ≤ 1 1-\pi_\rho(\nu Z_1)\le1 1 − π ρ ( ν Z 1 ) ≤ 1 , we obtain
1 σ t τ t ≤ 2 E [ ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . \frac{1}{\sigma_t\tau_t}\le2\mathbb{E}\left[\frac{\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}\right]. σ t τ t 1 ≤ 2 E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] .
Since ν α t Z 1 + σ t Z 2 ∼ N ( 0 , ν 2 α t 2 + σ t 2 ) \nu\alpha_tZ_1+\sigma_tZ_2\sim\mathcal{N}(0,\nu^2\alpha_t^2+\sigma_t^2) ν α t Z 1 + σ t Z 2 ∼ N ( 0 , ν 2 α t 2 + σ t 2 ) , the change of variables x = u + γ t ϕ ′ ( u ) x=u+\gamma_t\phi'(u) x = u + γ t ϕ ′ ( u ) , d x = ( 1 + γ t ϕ ′ ′ ( u ) ) d u dx=(1+\gamma_t\phi''(u))\,du d x = ( 1 + γ t ϕ ′′ ( u )) d u , gives
E [ ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] = ∫ R ϕ ′ ′ ( u ) 2 π ( ν 2 α t 2 + σ t 2 ) exp ( − ( u + γ t ϕ ′ ( u ) ) 2 2 ( ν 2 α t 2 + σ t 2 ) ) d u . \mathbb{E}\left[\frac{\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_tZ_1+\sigma_tZ_2)\right)}\right]
=\int_{\mathbb{R}}\frac{\phi''(u)}{\sqrt{2\pi(\nu^2\alpha_t^2+\sigma_t^2)}}\exp\left(-\frac{(u+\gamma_t\phi'(u))^2}{2(\nu^2\alpha_t^2+\sigma_t^2)}\right)\,du. E [ 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] = ∫ R 2 π ( ν 2 α t 2 + σ t 2 ) ϕ ′′ ( u ) exp ( − 2 ( ν 2 α t 2 + σ t 2 ) ( u + γ t ϕ ′ ( u ) ) 2 ) d u .
Using ∫ R ϕ ′ ′ ( u ) d u = 1 \int_{\mathbb{R}}\phi''(u)\,du=1 ∫ R ϕ ′′ ( u ) d u = 1 , we obtain 1 σ t τ t ≤ 2 2 π ( ν 2 α t 2 + σ t 2 ) \frac{1}{\sigma_t\tau_t}\le\frac{2}{\sqrt{2\pi(\nu^2\alpha_t^2+\sigma_t^2)}} σ t τ t 1 ≤ 2 π ( ν 2 α t 2 + σ t 2 ) 2 , and hence
ν 2 α t 2 + σ t 2 σ t τ t ≤ 2 π . (110) \frac{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}{\sigma_t\tau_t}\le\sqrt{\frac{2}{\pi}}. \tag*{(110)} σ t τ t ν 2 α t 2 + σ t 2 ≤ π 2 . (110)
The inequality is trivial when ν 2 α t 2 + σ t 2 = 0 \sqrt{\nu^2\alpha_t^2+\sigma_t^2}=0 ν 2 α t 2 + σ t 2 = 0 . The temporal projection update can be written as
w t = w t − 1 + S σ t τ t λ ( σ t τ t ( θ t w ⋆ + r t δ Z t ) − w t − 1 ) . w_t=w_{t-1}+\mathcal{S}_{\sigma_t\tau_t\lambda}\left(\sigma_t\tau_t\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t\right)-w_{t-1}\right). w t = w t − 1 + S σ t τ t λ ( σ t τ t ( θ t w ⋆ + δ r t Z t ) − w t − 1 ) .
by Lemma 20. On the update event U t = { w t ≠ w t − 1 } U_t=\{w_t\ne w_{t-1}\} U t = { w t = w t − 1 } , we therefore have exactly
w t σ t τ t = θ t w ⋆ + r t δ Z t − λ sign ( θ t w ⋆ + r t δ Z t − w t − 1 σ t τ t ) . (111) \frac{w_t}{\sigma_t\tau_t}=\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t-\lambda\operatorname{sign}\left(\theta_tw^\star+\frac{r_t}{\sqrt{\delta}}Z_t-\frac{w_{t-1}}{\sigma_t\tau_t}\right). \tag*{(111)} σ t τ t w t = θ t w ⋆ + δ r t Z t − λ sign ( θ t w ⋆ + δ r t Z t − σ t τ t w t − 1 ) . (111)
Hence, on U t U_t U t we have ∣ θ t w ⋆ + r t δ Z t − w t − 1 σ t τ t ∣ > λ \left|\theta_t w^\star+ \frac{r_t}{\sqrt{\delta}}Z_t - \frac{w_{t-1}}{\sigma_t\tau_t}\right| > \lambda θ t w ⋆ + δ r t Z t − σ t τ t w t − 1 > λ . If ∣ θ t w ⋆ + r t δ Z t ∣ ≤ λ 2 \left|\theta_t w^\star+ \frac{r_t}{\sqrt{\delta}}Z_t\right| \le\frac{\lambda}{2} θ t w ⋆ + δ r t Z t ≤ 2 λ , then ∣ w t σ t τ t ∣ ≥ λ 2 \left|\frac{w_t}{\sigma_t\tau_t}\right| \ge\frac{\lambda}{2} σ t τ t w t ≥ 2 λ an vice versa. Consequently,
U t ⊆ { ∣ θ t w ⋆ + r t δ Z t ∣ > λ 2 } ∪ { ∣ w t σ t τ t ∣ > λ 2 } . (112) U_t \subseteq\left\{\left|\theta_t w^\star+ \frac{r_t}{\sqrt{\delta}}Z_t\right| > \frac{\lambda}{2}\right\} \cup\left\{\left|\frac{w_t}{\sigma_t\tau_t}\right| > \frac{\lambda}{2}\right\}. \tag*{(112)} U t ⊆ { θ t w ⋆ + δ r t Z t > 2 λ } ∪ { σ t τ t w t > 2 λ } . (112)
Using θ t ≤ 1 2 \theta_t \le\frac{1}{2} θ t ≤ 2 1 , r t ≤ 1 r_t \le1 r t ≤ 1 (see conclusion beneath Eq. (48) ), and Eq. (110) , we have
E [ ( θ t w ⋆ + r t δ Z t ) 2 ] ≤ ν 2 4 + 1 δ . \mathbb{E}\left[\left(\theta_t w^\star+ \frac{r_t}{\sqrt{\delta}}Z_t\right)^2\right] \le\frac{\nu^2}{4} + \frac{1}{\delta}. E [ ( θ t w ⋆ + δ r t Z t ) 2 ] ≤ 4 ν 2 + δ 1 .
and
E [ ( w t σ t τ t ) 2 ] = ν 2 α t 2 + σ t 2 ( σ t τ t ) 2 ≤ 2 π . \mathbb{E}\left[\left(\frac{w_t}{\sigma_t\tau_t}\right)^2\right] = \frac{\nu^2\alpha_t^2 + \sigma_t^2}{(\sigma_t\tau_t)^2} \le\frac{2}{\pi}. E [ ( σ t τ t w t ) 2 ] = ( σ t τ t ) 2 ν 2 α t 2 + σ t 2 ≤ π 2 .
Applying Chebyshev’s inequality to Eq. (112) therefore gives P ( U t ) ≤ 4 λ 2 ( ν 2 4 + 1 δ + 2 π ) \mathbb{P}(U_t) \le\frac{4}{\lambda^2}\left(\frac{\nu^2}{4} + \frac{1}{\delta} + \frac{2}{\pi}\right) P ( U t ) ≤ λ 2 4 ( 4 ν 2 + δ 1 + π 2 ) . Consequently,
γ t σ t τ t = 1 δ P ( U t ) ≤ 4 δ ( ν 2 4 + 1 δ + 2 π ) λ 2 . (113) \frac{\gamma_t}{\sigma_t\tau_t} = \frac{1}{\delta}\mathbb{P}(U_t) \le\frac{\frac{4}{\delta}\left(\frac{\nu^2}{4} + \frac{1}{\delta} + \frac{2}{\pi}\right)}{\lambda^2}. \tag*{(113)} σ t τ t γ t = δ 1 P ( U t ) ≤ λ 2 δ 4 ( 4 ν 2 + δ 1 + π 2 ) . (113)
Applying Eq. (51) with q = σ t τ t q = \sigma_t\tau_t q = σ t τ t , q H t = ν α t Z 1 + σ t Z 2 qH_t = \nu\alpha_t Z_1 + \sigma_t Z_2 q H t = ν α t Z 1 + σ t Z 2 , η = γ t σ t τ t \eta= \frac{\gamma_t}{\sigma_t\tau_t} η = σ t τ t γ t , and R = ν 2 α t 2 + σ t 2 R = \sqrt{\nu^2\alpha_t^2 + \sigma_t^2} R = ν 2 α t 2 + σ t 2 gives,
1 σ t τ t ≥ e − 1 2 16 2 π ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 + γ t ) . \frac{1}{\sigma_t\tau_t} \ge\frac{e^{-\frac{1}{2}}}{16\sqrt{2\pi}(1+\nu)\left(1+\sqrt{\nu^2\alpha_t^2+\sigma_t^2}+\gamma_t\right)}. σ t τ t 1 ≥ 16 2 π ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 + γ t ) e − 2 1 .
Thus
σ t τ t ≤ 16 2 π e 1 2 ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 + γ t ) . \sigma_t\tau_t \le16\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu)\left(1+\sqrt{\nu^2\alpha_t^2+\sigma_t^2}+\gamma_t\right). σ t τ t ≤ 16 2 π e 2 1 ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 + γ t ) .
Using Eq. (113) ,
σ t τ t ≤ 16 2 π e 1 2 ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 ) + 16 2 π e 1 2 ( 1 + ν ) 4 δ ( ν 2 4 + 1 δ + 2 π ) λ 2 σ t τ t . \sigma_t\tau_t \le16\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu)\left(1+\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right) + 16\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu)\frac{\frac{4}{\delta}\left(\frac{\nu^2}{4}+\frac{1}{\delta}+\frac{2}{\pi}\right)}{\lambda^2}\sigma_t\tau_t. σ t τ t ≤ 16 2 π e 2 1 ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 ) + 16 2 π e 2 1 ( 1 + ν ) λ 2 δ 4 ( 4 ν 2 + δ 1 + π 2 ) σ t τ t .
Hence, if λ 2 ≥ 128 2 π e 1 2 1 + ν δ ( ν 2 4 + 1 δ + 2 π ) \lambda^2 \ge128\sqrt{2\pi}e^{\frac{1}{2}}\frac{1+\nu}{\delta}\left(\frac{\nu^2}{4}+\frac{1}{\delta}+\frac{2}{\pi}\right) λ 2 ≥ 128 2 π e 2 1 δ 1 + ν ( 4 ν 2 + δ 1 + π 2 ) , then
σ t τ t ≤ 32 2 π e 1 2 ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 ) . (114) \sigma_t\tau_t \le32\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu)\left(1+\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right). \tag*{(114)} σ t τ t ≤ 32 2 π e 2 1 ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 ) . (114)
uniformly in t t t and ρ \rho ρ . Consequently,
γ t ≤ 128 2 π e 1 2 1 + ν δ ( ν 2 4 + 1 δ + 2 π ) 1 + ν 2 α t 2 + σ t 2 λ 2 . (115) \gamma_t \le128\sqrt{2\pi}e^{\frac{1}{2}}\frac{1+\nu}{\delta}\left(\frac{\nu^2}{4}+\frac{1}{\delta}+\frac{2}{\pi}\right)\frac{1+\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}{\lambda^2}. \tag*{(115)} γ t ≤ 128 2 π e 2 1 δ 1 + ν ( 4 ν 2 + δ 1 + π 2 ) λ 2 1 + ν 2 α t 2 + σ t 2 . (115)
We next obtain a lower bound on r t r_t r t . Suppose first that ν 2 α t 2 + σ t 2 ≥ 1 \sqrt{\nu^2\alpha_t^2+\sigma_t^2} \ge1 ν 2 α t 2 + σ t 2 ≥ 1 . For λ 2 ≥ 512 2 π e 1 2 ( 1 + ν ) 2 δ ( ν 2 4 + 1 δ + 2 π ) \lambda^2 \ge512\sqrt{2\pi}e^{\frac{1}{2}}\frac{(1+\nu)^2}{\delta}\left(\frac{\nu^2}{4}+\frac{1}{\delta}+\frac{2}{\pi}\right) λ 2 ≥ 512 2 π e 2 1 δ ( 1 + ν ) 2 ( 4 ν 2 + δ 1 + π 2 ) , Eq. (115) gives γ t ν 2 α t 2 + σ t 2 ≤ 1 2 ( 1 + ν ) \frac{\gamma_t}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}} \le\frac{1}{2(1+\nu)} ν 2 α t 2 + σ t 2 γ t ≤ 2 ( 1 + ν ) 1 . Conditionally on ν α t Z 1 + σ t Z 2 = x \nu\alpha_t Z_1+\sigma_t Z_2=x ν α t Z 1 + σ t Z 2 = x , we have Z 1 = d ν 2 α t 2 + σ t 2 x + 1 − ν 2 α t 2 ν 2 α t 2 + σ t 2 G Z_1 = \frac{d}{\nu^2\alpha_t^2+\sigma_t^2}x+\sqrt{1-\frac{\nu^2\alpha_t^2}{\nu^2\alpha_t^2+\sigma_t^2}}G Z 1 = ν 2 α t 2 + σ t 2 d x + 1 − ν 2 α t 2 + σ t 2 ν 2 α t 2 G , where G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) . By symmetry,
E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ∣ ν α t Z 1 + σ t Z 2 = 0 ] = 1. \mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))\mid\nu\alpha_t Z_1+\sigma_t Z_2=0\right]=1. E [ 2 ( 1 − π ρ ( ν Z 1 )) ∣ ν α t Z 1 + σ t Z 2 = 0 ] = 1.
Moreover,
∣ d d x E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ∣ ν α t Z 1 + σ t Z 2 = x ] ∣ ≤ 2 ν ∣ ν α t ∣ ν 2 α t 2 + σ t 2 sup u ∈ R π ρ ′ ( u ) ≤ ν 2 ν 2 α t 2 + σ t 2 . (116) \left|\frac{d}{dx}\mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))\mid\nu\alpha_t Z_1+\sigma_t Z_2=x\right]\right| \le\frac{2\nu|\nu\alpha_t|}{\nu^2\alpha_t^2+\sigma_t^2}\sup_{u\in\mathbb{R}}\pi_\rho'(u) \le\frac{\nu}{2\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}. \tag*{(116)} d x d E [ 2 ( 1 − π ρ ( ν Z 1 )) ∣ ν α t Z 1 + σ t Z 2 = x ] ≤ ν 2 α t 2 + σ t 2 2 ν ∣ ν α t ∣ u ∈ R sup π ρ ′ ( u ) ≤ 2 ν 2 α t 2 + σ t 2 ν . (116)
where we used ∣ ν α t ∣ ≤ ν 2 α t 2 + σ t 2 |\nu\alpha_t| \le\sqrt{\nu^2\alpha_t^2+\sigma_t^2} ∣ ν α t ∣ ≤ ν 2 α t 2 + σ t 2 and sup u ∈ R π ρ ′ ( u ) ≤ 1 4 \sup_{u\in\mathbb{R}}\pi'_{\rho}(u) \le\frac{1}{4} sup u ∈ R π ρ ′ ( u ) ≤ 4 1 . Therefore, whenever ∣ x ∣ ≤ ν 2 α t 2 + σ t 2 1 + ν ≤ ν 2 α t 2 + σ t 2 1 + ν |x| \le\frac{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}{1+\nu} \le\frac{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}{1+\nu} ∣ x ∣ ≤ 1 + ν ν 2 α t 2 + σ t 2 ≤ 1 + ν ν 2 α t 2 + σ t 2 ,
E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ∣ ν α t Z 1 + σ t Z 2 = x ] ≥ 1 2 . \mathbb{E}\left[2(1-\pi_{\rho}(\nu Z_1))\mid\nu\alpha_t Z_1+\sigma_t Z_2=x\right]\ge\frac{1}{2}. E [ 2 ( 1 − π ρ ( ν Z 1 )) ∣ ν α t Z 1 + σ t Z 2 = x ] ≥ 2 1 .
Consider the event
ν 2 α t 2 + σ t 2 2 ( 1 + ν ) ≤ ν α t Z 1 + σ t Z 2 ≤ ν 2 α t 2 + σ t 2 1 + ν . \frac{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}{2(1+\nu)}\le\nu\alpha_t Z_1+\sigma_t Z_2\le\frac{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}{1+\nu}. 2 ( 1 + ν ) ν 2 α t 2 + σ t 2 ≤ ν α t Z 1 + σ t Z 2 ≤ 1 + ν ν 2 α t 2 + σ t 2 .
this event has probability
P ( 1 2 ( 1 + ν ) ≤ ν α t Z 1 + σ t Z 2 ν 2 α t 2 + σ t 2 ≤ 1 1 + ν ) = Φ ( 1 1 + ν ) − Φ ( 1 2 ( 1 + ν ) ) . \mathbb{P}\left(\frac{1}{2(1+\nu)}\le\frac{\nu\alpha_t Z_1+\sigma_t Z_2}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\le\frac{1}{1+\nu}\right)=\Phi\left(\frac{1}{1+\nu}\right)-\Phi\left(\frac{1}{2(1+\nu)}\right). P ( 2 ( 1 + ν ) 1 ≤ ν 2 α t 2 + σ t 2 ν α t Z 1 + σ t Z 2 ≤ 1 + ν 1 ) = Φ ( 1 + ν 1 ) − Φ ( 2 ( 1 + ν ) 1 ) .
Moreover, the proximal relation (Eq. (65) ) on this event yields,
Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ≥ ν α t Z 1 + σ t Z 2 − γ t ≥ 0 , \operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\ge\nu\alpha_t Z_1+\sigma_t Z_2-\gamma_t\ge0, Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ≥ ν α t Z 1 + σ t Z 2 − γ t ≥ 0 ,
and hence ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≥ 1 2 \phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\ge\frac{1}{2} ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≥ 2 1 . Therefore,
r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] ≥ 1 8 [ Φ ( 1 1 + ν ) − Φ ( 1 2 ( 1 + ν ) ) ] ≥ φ ( 1 ) 16 ( 1 + ν ) . r_t^2=2\mathbb{E}\left[(1-\pi_{\rho}(\nu Z_1))\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))^2\right]\ge\frac{1}{8}\left[\Phi\left(\frac{1}{1+\nu}\right)-\Phi\left(\frac{1}{2(1+\nu)}\right)\right]\ge\frac{\varphi(1)}{16(1+\nu)}. r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] ≥ 8 1 [ Φ ( 1 + ν 1 ) − Φ ( 2 ( 1 + ν ) 1 ) ] ≥ 16 ( 1 + ν ) φ ( 1 ) .
Suppose now that ν 2 α t 2 + σ t 2 < 1 \sqrt{\nu^2\alpha_t^2+\sigma_t^2}<1 ν 2 α t 2 + σ t 2 < 1 . Under the same condition on λ \lambda λ , Eq. (115) gives γ t ≤ 1 \gamma_t\le1 γ t ≤ 1 . On the event 0 ≤ ν α t Z 1 + σ t Z 2 ≤ ν 2 α t 2 + σ t 2 1 + ν 0\le\nu\alpha_t Z_1+\sigma_t Z_2\le\frac{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}{1+\nu} 0 ≤ ν α t Z 1 + σ t Z 2 ≤ 1 + ν ν 2 α t 2 + σ t 2 , we have E [ 2 ( 1 − π ρ ( ν Z 1 ) ) ∣ ν α t Z 1 + σ t Z 2 ] ≥ 1 2 \mathbb{E}[2(1-\pi_{\rho}(\nu Z_1))\mid\nu\alpha_t Z_1+\sigma_t Z_2]\ge\frac{1}{2} E [ 2 ( 1 − π ρ ( ν Z 1 )) ∣ ν α t Z 1 + σ t Z 2 ] ≥ 2 1 , while Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ≥ − γ t ≥ − 1 \operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\ge-\gamma_t\ge-1 Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ≥ − γ t ≥ − 1 . Thus ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≥ ϕ ′ ( − 1 ) = 1 1 + e \phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\ge\phi'(-1)=\frac{1}{1+e} ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ≥ ϕ ′ ( − 1 ) = 1 + e 1 . Consequently,
r t 2 ≥ 1 2 ( 1 + e ) 2 [ Φ ( 1 1 + ν ) − 1 2 ] ≥ φ ( 1 ) 2 ( 1 + e ) 2 ( 1 + ν ) . r_t^2\ge\frac{1}{2(1+e)^2}\left[\Phi\left(\frac{1}{1+\nu}\right)-\frac{1}{2}\right]\ge\frac{\varphi(1)}{2(1+e)^2(1+\nu)}. r t 2 ≥ 2 ( 1 + e ) 2 1 [ Φ ( 1 + ν 1 ) − 2 1 ] ≥ 2 ( 1 + e ) 2 ( 1 + ν ) φ ( 1 ) .
Since φ ( 1 ) 2 ( 1 + e ) 2 ( 1 + ν ) ≤ φ ( 1 ) 16 ( 1 + ν ) \frac{\varphi(1)}{2(1+e)^2(1+\nu)}\le\frac{\varphi(1)}{16(1+\nu)} 2 ( 1 + e ) 2 ( 1 + ν ) φ ( 1 ) ≤ 16 ( 1 + ν ) φ ( 1 ) , we have uniformly in t t t and ρ \rho ρ , φ ( 1 ) 2 ( 1 + e ) 2 ( 1 + ν ) ≤ r t ≤ 1 \sqrt{\frac{\varphi(1)}{2(1+e)^2(1+\nu)}}\le r_t\le1 2 ( 1 + e ) 2 ( 1 + ν ) φ ( 1 ) ≤ r t ≤ 1 . We next refine the estimate on θ t \theta_t θ t . From symmetry, with ϕ ′ ( x ) = 1 − ϕ ′ ( − x ) \phi'(x)=1-\phi'(-x) ϕ ′ ( x ) = 1 − ϕ ′ ( − x ) ,
2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = ( 1 − 2 ρ ) h ( ν ) = r h ( ν r ) . 2\mathbb{E}\left[\pi'_{\rho}(\nu Z_1)\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)\right]=(1-2\rho)h(\nu)=rh(\nu r). 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = ( 1 − 2 ρ ) h ( ν ) = r h ( ν r ) .
From the proximal relation Prox γ t ϕ ( x ) = x − γ t ϕ ′ ( u ) ≤ x \operatorname{Prox}_{\gamma_t\phi}(x)=x-\gamma_t\phi'(u)\le x Prox γ t ϕ ( x ) = x − γ t ϕ ′ ( u ) ≤ x and ϕ ′ \phi' ϕ ′ is increasing, as a consequence of these identities we have 0 ≤ r h ( ν r ) − θ t 0\le rh(\nu r)-\theta_t 0 ≤ r h ( ν r ) − θ t . Moreover,
r h ( ν r ) − θ t = 2 E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ) ] . rh(\nu r)-\theta_t=2\mathbb{E}\left[\pi'_{\rho}(\nu Z_1)\left(\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)-\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\right)\right]. r h ( ν r ) − θ t = 2 E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ) ] .
If ν 2 α t 2 + σ t 2 ≤ 1 \sqrt{\nu^2\alpha_t^2+\sigma_t^2}\le1 ν 2 α t 2 + σ t 2 ≤ 1 , then, since ϕ ′ \phi' ϕ ′ is 1 4 \frac{1}{4} 4 1 -Lipschitz and the proximal relation (Eq. (65) ),
0 ≤ r h ( ν r ) − θ t ≤ γ t 2 ( 1 − 2 ρ ) h ( ν ) ≤ γ t 8 . 0\le rh(\nu r)-\theta_t\le\frac{\gamma_t}{2}(1-2\rho)h(\nu)\le\frac{\gamma_t}{8}. 0 ≤ r h ( ν r ) − θ t ≤ 2 γ t ( 1 − 2 ρ ) h ( ν ) ≤ 8 γ t .
Using Eq. (115) ,
r h ( ν r ) − θ t ≤ 32 2 π e 1 2 1 + ν δ ( ν 2 4 + 1 δ + 2 π ) 1 λ 2 . rh(\nu r)-\theta_t\le32\sqrt{2\pi}e^{\frac{1}{2}}\frac{1+\nu}{\delta}\left(\frac{\nu^2}{4}+\frac{1}{\delta}+\frac{2}{\pi}\right)\frac{1}{\lambda^2}. r h ( ν r ) − θ t ≤ 32 2 π e 2 1 δ 1 + ν ( 4 ν 2 + δ 1 + π 2 ) λ 2 1 .
If ν 2 α t 2 + σ t 2 ≥ 1 \sqrt{\nu^2\alpha_t^2+\sigma_t^2}\ge1 ν 2 α t 2 + σ t 2 ≥ 1 , then using Eq. (138) , ∫ R [ ϕ ′ ( x ) − ϕ ′ ( Prox γ t ϕ ( x ) ) ] d x = γ t 2 \int_{\mathbb{R}}[\phi'(x)-\phi'(\operatorname{Prox}_{\gamma_t\phi}(x))]\,\mathrm{d}x=\frac{\gamma_t}{2} ∫ R [ ϕ ′ ( x ) − ϕ ′ ( Prox γ t ϕ ( x ))] d x = 2 γ t . Conditioning on ν α t Z 1 + σ t Z 2 = x \nu\alpha_t Z_1+\sigma_t Z_2=x ν α t Z 1 + σ t Z 2 = x , using π ρ ′ ( u ) ≤ 1 4 \pi'_{\rho}(u)\le\frac{1}{4} π ρ ′ ( u ) ≤ 4 1 and the upper bound on the Gaussian density, gives
0 ≤ r h ( ν r ) − θ t ≤ 1 2 2 π ( ν 2 α t 2 + σ t 2 ) ∫ R [ ϕ ′ ( x ) − ϕ ′ ( Prox γ t ϕ ( x ) ) ] d x = γ t 4 2 π ( ν 2 α t 2 + σ t 2 ) ≤ 32 e 1 2 1 + ν δ ( ν 2 4 + 1 δ + 2 π ) 1 λ 2 . \begin{aligned}
0\le rh(\nu r)-\theta_t\le\frac{1}{2\sqrt{2\pi(\nu^2\alpha_t^2+\sigma_t^2)}}\int_{\mathbb{R}}[\phi'(x)-\phi'(\operatorname{Prox}_{\gamma_t\phi}(x))]\,\mathrm{d}x \\
&=\frac{\gamma_t}{4\sqrt{2\pi(\nu^2\alpha_t^2+\sigma_t^2)}}\le32e^{\frac{1}{2}}\frac{1+\nu}{\delta}\left(\frac{\nu^2}{4}+\frac{1}{\delta}+\frac{2}{\pi}\right)\frac{1}{\lambda^2}.
\end{aligned} 0 ≤ r h ( ν r ) − θ t ≤ 2 2 π ( ν 2 α t 2 + σ t 2 ) 1 ∫ R [ ϕ ′ ( x ) − ϕ ′ ( Prox γ t ϕ ( x ))] d x = 4 2 π ( ν 2 α t 2 + σ t 2 ) γ t ≤ 32 e 2 1 δ 1 + ν ( 4 ν 2 + δ 1 + π 2 ) λ 2 1 .
Thus, in both cases,
0 ≤ r h ( ν r ) − θ t ≤ 32 2 π e 1 2 1 + ν δ ( ν 2 4 + 1 δ + 2 π ) 1 λ 2 . (117) 0\le rh(\nu r)-\theta_t\le32\sqrt{2\pi}e^{\frac{1}{2}}\frac{1+\nu}{\delta}\left(\frac{\nu^2}{4}+\frac{1}{\delta}+\frac{2}{\pi}\right)\frac{1}{\lambda^2}. \tag*{(117)} 0 ≤ r h ( ν r ) − θ t ≤ 32 2 π e 2 1 δ 1 + ν ( 4 ν 2 + δ 1 + π 2 ) λ 2 1 . (117)
We first consider bounded ν 2 α t 2 + σ t 2 \sqrt{\nu^2\alpha_t^2+\sigma_t^2} ν 2 α t 2 + σ t 2 . From Eq. (97) ,
2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ] = h ( ν 2 α t 2 + σ t 2 ) , 2\mathbb{E}\left[(1-\pi_{\rho}(\nu Z_1))\phi''(\nu\alpha_t Z_1+\sigma_t Z_2)\right]=h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right), 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′′ ( ν α t Z 1 + σ t Z 2 ) ] = h ( ν 2 α t 2 + σ t 2 ) ,
the proximal relation, together with sup u ∣ ϕ ′ ′ ′ ( u ) ∣ = 1 6 3 \sup_u|\phi'''(u)|=\frac{1}{6\sqrt{3}} sup u ∣ ϕ ′′′ ( u ) ∣ = 6 3 1 , sup u ϕ ′ ′ ( u ) = 1 4 \sup_u\phi''(u)=\frac{1}{4} sup u ϕ ′′ ( u ) = 4 1 , gives
∣ ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) − ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ∣ ≤ ∣ ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) − ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) ∣ + \begin{aligned}
\left|\frac{\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}-\phi''(\nu\alpha_t Z_1+\sigma_t Z_2)\right| \\
&\leq\left|\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)-\phi''(\nu\alpha_t Z_1+\sigma_t Z_2)\right|+
\end{aligned} 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) − ϕ ′′ ( ν α t Z 1 + σ t Z 2 ) ≤ ∣ ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) − ϕ ′′ ( ν α t Z 1 + σ t Z 2 ) ∣ +
∣ ϕ ′ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ ≤ sup u ϕ ′ ′ ′ ( u ) ∣ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) − ν α t Z 1 + σ t Z 2 ∣ + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) \begin{aligned}
\left|\phi''(\nu\alpha_t Z_1+\sigma_t Z_2)-\frac{\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}\right| \\
&\leq\sup_u\phi'''(u)\left|\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)-\nu\alpha_t Z_1+\sigma_t Z_2\right|+\frac{\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)^2}{1+\gamma_t\phi''\left(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2)\right)}
\end{aligned} ϕ ′′ ( ν α t Z 1 + σ t Z 2 ) − 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ≤ u sup ϕ ′′′ ( u ) ∣ Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) − ν α t Z 1 + σ t Z 2 ∣ + 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2
Thus,
∣ 1 σ t τ t − h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ ( 1 6 3 + 1 16 ) γ t . \left|\frac{1}{\sigma_t\tau_t}-h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)\right|\leq\left(\frac{1}{6\sqrt{3}}+\frac{1}{16}\right)\gamma_t. σ t τ t 1 − h ( ν 2 α t 2 + σ t 2 ) ≤ ( 6 3 1 + 16 1 ) γ t .
Consequently, whenever ν 2 α t 2 + σ t 2 ≤ C ( 1 + ν ) 3 \sqrt{\nu^2\alpha_t^2+\sigma_t^2}\leq C(1+\nu)^3 ν 2 α t 2 + σ t 2 ≤ C ( 1 + ν ) 3 for a sufficiently large universal constant C C C , we have h ( ν 2 α t 2 + σ t 2 ) ≥ c ( 1 + ν ) 3 h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)\geq\frac{c}{(1+\nu)^3} h ( ν 2 α t 2 + σ t 2 ) ≥ ( 1 + ν ) 3 c for a universal c > 0 c>0 c > 0 (see Eq. (61) ). Using Eq. (115) , if λ 2 ≥ C ( 1 + ν ) 7 δ ( 1 + ν 2 + 1 δ ) \lambda^2\geq C\frac{(1+\nu)^7}{\delta}(1+\nu^2+\frac{1}{\delta}) λ 2 ≥ C δ ( 1 + ν ) 7 ( 1 + ν 2 + δ 1 ) , then 1 σ t τ t ≥ c 2 ( 1 + ν ) 3 \frac{1}{\sigma_t\tau_t}\geq\frac{c}{2(1+\nu)^3} σ t τ t 1 ≥ 2 ( 1 + ν ) 3 c from Eq. (115) . Therefore,
∣ σ t τ t − 1 h ( ν 2 α t 2 + σ t 2 ) ∣ = σ t τ t h ( ν 2 α t 2 + σ t 2 ) ∣ 1 σ t τ t − h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ C ( 1 + ν ) 10 δ λ 2 ( 1 + ν 2 + 1 δ ) . \left|\sigma_t\tau_t-\frac{1}{h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)}\right|=\frac{\sigma_t\tau_t}{h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)}\left|\frac{1}{\sigma_t\tau_t}-h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)\right|\leq\frac{C(1+\nu)^{10}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right). σ t τ t − h ( ν 2 α t 2 + σ t 2 ) 1 = h ( ν 2 α t 2 + σ t 2 ) σ t τ t σ t τ t 1 − h ( ν 2 α t 2 + σ t 2 ) ≤ δ λ 2 C ( 1 + ν ) 10 ( 1 + ν 2 + δ 1 ) .
Combining this with Eq. (117) ,
∣ σ t τ t θ t − r h ( ν r ) h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ σ t τ t ∣ θ t − r h ( ν r ) ∣ + r h ( ν r ) h ( ν 2 α t 2 + σ t 2 ) ∣ σ t τ t − 1 h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ C ( 1 + ν ) 10 δ λ 2 ( 1 + ν 2 + 1 δ ) . (118) \begin{aligned}
\left|\sigma_t\tau_t\theta_t-\frac{r h(\nu r)}{h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)}\right|\leq\sigma_t\tau_t\left|\theta_t-rh(\nu r)\right|+\frac{rh(\nu r)}{h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)}\left|\sigma_t\tau_t-\frac{1}{h\left(\sqrt{\nu^2\alpha_t^2+\sigma_t^2}\right)}\right| \\
&\leq\frac{C(1+\nu)^{10}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right). \tag*{(118)}
\end{aligned} σ t τ t θ t − h ( ν 2 α t 2 + σ t 2 ) r h ( ν r ) ≤ σ t τ t ∣ θ t − r h ( ν r ) ∣ + h ( ν 2 α t 2 + σ t 2 ) r h ( ν r ) σ t τ t − h ( ν 2 α t 2 + σ t 2 ) 1 ≤ δ λ 2 C ( 1 + ν ) 10 ( 1 + ν 2 + δ 1 ) . (118)
The function h h h is strictly decreasing on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) because ϕ ′ ′ \phi'' ϕ ′′ is strictly decreasing on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) . Moreover, d d a ( a h ( a ) ) = E [ Z 2 ϕ ′ ′ ( a Z ) ] > 0 \frac{\mathrm{d}}{\mathrm{d}a}(ah(a))=\mathbb{E}[Z^2\phi''(aZ)]>0 d a d ( ah ( a )) = E [ Z 2 ϕ ′′ ( a Z )] > 0 . Also, d 2 d a 2 ( a h ( a ) ) = E [ Z 3 ϕ ′ ′ ′ ( a Z ) ] ≤ 0 \frac{\mathrm{d}^2}{\mathrm{d}a^2}(ah(a))=\mathbb{E}[Z^3\phi'''(aZ)]\leq0 d a 2 d 2 ( ah ( a )) = E [ Z 3 ϕ ′′′ ( a Z )] ≤ 0 , so a h ( a ) ah(a) ah ( a ) is concave. For every a ≥ 0 a\geq0 a ≥ 0 , since min { 1 , 1 a } ≥ 1 1 + a \min\{1,\frac{1}{a}\}\geq\frac{1}{1+a} min { 1 , a 1 } ≥ 1 + a 1 , we obtain
E [ Z 2 ϕ ′ ′ ( a Z ) ] ≥ ϕ ′ ′ ( 1 ) E [ Z 2 1 ∣ Z ∣ ≤ min { 1 , 1 a } ] = 2 ϕ ′ ′ ( 1 ) ∫ 0 b z 2 φ ( z ) d z ≥ 2 φ ( 1 ) ϕ ′ ′ ( 1 ) ∫ 0 b z 2 d z = 2 φ ( 1 ) ϕ ′ ′ ( 1 ) 1 3 ( 1 + a ) 3 ≥ 2 φ ( 1 ) ϕ ′ ′ ( 1 ) 3 ( 1 + a ) 3 . (119) \begin{aligned}
\mathbb{E}[Z^2\phi''(aZ)]\geq\phi''(1)\mathbb{E}[Z^2\mathbf{1}_{|Z|\leq\min\{1,\frac{1}{a}\}}]=2\phi''(1)\int_0^b z^2\varphi(z)\,\mathrm{d}z\geq2\varphi(1)\phi''(1)\int_0^b z^2\,\mathrm{d}z \\
&=2\varphi(1)\phi''(1)\frac{1}{3(1+a)^3}\geq\frac{2\varphi(1)\phi''(1)}{3(1+a)^3}. \tag*{(119)}
\end{aligned} E [ Z 2 ϕ ′′ ( a Z )] ≥ ϕ ′′ ( 1 ) E [ Z 2 1 ∣ Z ∣ ≤ m i n { 1 , a 1 } ] = 2 ϕ ′′ ( 1 ) ∫ 0 b z 2 φ ( z ) d z ≥ 2 φ ( 1 ) ϕ ′′ ( 1 ) ∫ 0 b z 2 d z = 2 φ ( 1 ) ϕ ′′ ( 1 ) 3 ( 1 + a ) 3 1 ≥ 3 ( 1 + a ) 3 2 φ ( 1 ) ϕ ′′ ( 1 ) . (119)
Now fix any x ∈ [ 0 , ν ] x\in[0,\nu] x ∈ [ 0 , ν ] because h ( u ) h(u) h ( u ) is decreasing while u h ( u ) uh(u) u h ( u ) is increasing (Lemma 15 ). For R < x R<x R < x , R < x h ( x ) h ( R ) < x R<\frac{xh(x)}{h(R)}<x R < h ( R ) x h ( x ) < x , while for R > x R>x R > x , x < x h ( x ) h ( R ) < R x<\frac{xh(x)}{h(R)}<R x < h ( R ) x h ( x ) < R . Moreover, because x h ( x ) h ( R ) \frac{xh(x)}{h(R)} h ( R ) x h ( x ) is between R R R and x x x ,
1 − ∣ x h ( x ) h ( R ) − x ∣ ∣ R − x ∣ = 1 − ∣ x h ( x ) h ( R ) − x ∣ ∣ R − x h ( x ) h ( R ) ∣ + ∣ x h ( x ) h ( R ) − x ∣ = ∣ R h ( R ) − x h ( x ) ∣ h ( R ) ∣ R − x ∣ . 1-\frac{\left|\frac{xh(x)}{h(R)}-x\right|}{|R-x|}=1-\frac{\left|\frac{xh(x)}{h(R)}-x\right|}{\left|R-\frac{xh(x)}{h(R)}\right|+\left|\frac{xh(x)}{h(R)}-x\right|}=\frac{|Rh(R)-xh(x)|}{h(R)|R-x|}. 1 − ∣ R − x ∣ h ( R ) x h ( x ) − x = 1 − R − h ( R ) x h ( x ) + h ( R ) x h ( x ) − x h ( R ) x h ( x ) − x = h ( R ) ∣ R − x ∣ ∣ R h ( R ) − x h ( x ) ∣ .
If R ≤ 2 ( 1 + ν ) R\leq2(1+\nu) R ≤ 2 ( 1 + ν ) , the mean-value theorem, h ( R ) ≤ 1 4 h(R)\leq\frac{1}{4} h ( R ) ≤ 4 1 , and Eq. (119) give 1 − ∣ x h ( x ) h ( R ) − x ∣ ∣ R − x ∣ ≥ 8 φ ( 1 ) ϕ ′ ′ ( 1 ) 81 ( 1 + ν ) 3 1-\frac{\left|\frac{xh(x)}{h(R)}-x\right|}{|R-x|}\geq\frac{8\varphi(1)\phi''(1)}{81(1+\nu)^3} 1 − ∣ R − x ∣ ∣ h ( R ) x h ( x ) − x ∣ ≥ 81 ( 1 + ν ) 3 8 φ ( 1 ) ϕ ′′ ( 1 ) . If instead R ≥ 2 ( 1 + ν ) R\geq2(1+\nu) R ≥ 2 ( 1 + ν ) , since x ≤ ν < R x\leq\nu<R x ≤ ν < R and x h ( x ) h ( R ) \frac{xh(x)}{h(R)} h ( R ) x h ( x ) is between R R R and x x x , then
∣ x h ( x ) h ( R ) − x ∣ R − x ≤ x h ( x ) R h ( R ) ≤ ν h ( ν ) 2 ( 1 + ν ) h ( 2 ( 1 + ν ) ) . \frac{\left|\frac{xh(x)}{h(R)}-x\right|}{R-x}\leq\frac{xh(x)}{Rh(R)}\leq\frac{\nu h(\nu)}{2(1+\nu)h(2(1+\nu))}. R − x h ( R ) x h ( x ) − x ≤ R h ( R ) x h ( x ) ≤ 2 ( 1 + ν ) h ( 2 ( 1 + ν )) ν h ( ν ) .
Using again Eq. (119) , since q h ( q ) = ∫ R ϕ ′ ′ ( u ) φ ( u q ) d u ≤ φ ( 0 ) ∫ R ϕ ′ ′ ( u ) d u = 1 2 π qh(q)=\int_{\mathbb{R}}\phi^{\prime\prime}(u)\varphi\left(\frac{u}{q}\right)\,\mathrm{d}u\leq\varphi(0)\int_{\mathbb{R}}\phi^{\prime\prime}(u)\,\mathrm{d}u=\frac{1}{\sqrt{2\pi}} q h ( q ) = ∫ R ϕ ′′ ( u ) φ ( q u ) d u ≤ φ ( 0 ) ∫ R ϕ ′′ ( u ) d u = 2 π 1 , and because q h ( q ) qh(q) q h ( q ) is increasing we have 2 ( 1 + ν ) h ( 2 ( 1 + ν ) ) − ν h ( ν ) > 0 2(1+\nu)h(2(1+\nu))-\nu h(\nu)>0 2 ( 1 + ν ) h ( 2 ( 1 + ν )) − ν h ( ν ) > 0 , therefore,
1 − ν h ( ν ) 2 ( 1 + ν ) h ( 2 ( 1 + ν ) ) ≥ 2 π ( 2 ( 1 + ν ) h ( 2 ( 1 + ν ) ) − ν h ( ν ) ) ≥ 2 2 π φ ( 1 ) ϕ ′ ′ ( 1 ) 81 ( 1 + ν ) 2 . 1-\frac{\nu h(\nu)}{2(1+\nu)h(2(1+\nu))}\geq\sqrt{2\pi}\left(2(1+\nu)h(2(1+\nu))-\nu h(\nu)\right)\geq\frac{2\sqrt{2\pi}\varphi(1)\phi^{\prime\prime}(1)}{81(1+\nu)^2}. 1 − 2 ( 1 + ν ) h ( 2 ( 1 + ν )) ν h ( ν ) ≥ 2 π ( 2 ( 1 + ν ) h ( 2 ( 1 + ν )) − ν h ( ν ) ) ≥ 81 ( 1 + ν ) 2 2 2 π φ ( 1 ) ϕ ′′ ( 1 ) .
Thus, for every x ∈ [ 0 , ν ] x\in[0,\nu] x ∈ [ 0 , ν ] and every R ≥ 0 R\geq0 R ≥ 0 ,
∣ x h ( x ) h ( R ) − x ∣ ≤ ( 1 − 2 2 π φ ( 1 ) ϕ ′ ′ ( 1 ) 81 ( 1 + ν ) 3 ) ∣ R − x ∣ . (120) \left|\frac{xh(x)}{h(R)}-x\right|\leq\left(1-\frac{2\sqrt{2\pi}\varphi(1)\phi^{\prime\prime}(1)}{81(1+\nu)^3}\right)|R-x|. \tag*{(120)} h ( R ) x h ( x ) − x ≤ ( 1 − 81 ( 1 + ν ) 3 2 2 π φ ( 1 ) ϕ ′′ ( 1 ) ) ∣ R − x ∣. (120)
Since ν r ≤ ν \nu r\leq\nu ν r ≤ ν , taking x = ν r x=\nu r x = ν r gives
∣ ν r h ( ν r ) h ( R ) − ν r ∣ ≤ ( 1 − 2 2 π φ ( 1 ) ϕ ′ ′ ( 1 ) 81 ( 1 + ν ) 3 ) ∣ R − ν r ∣ . (121) \left|\frac{\nu r h(\nu r)}{h(R)}-\nu r\right|\leq\left(1-\frac{2\sqrt{2\pi}\varphi(1)\phi^{\prime\prime}(1)}{81(1+\nu)^3}\right)|R-\nu r|. \tag*{(121)} h ( R ) ν r h ( ν r ) − ν r ≤ ( 1 − 81 ( 1 + ν ) 3 2 2 π φ ( 1 ) ϕ ′′ ( 1 ) ) ∣ R − ν r ∣. (121)
Combining Eqs. (118) and (121) , whenever ν 2 α t 2 + σ t 2 ≤ C ( 1 + ν ) 3 \sqrt{\nu^2\alpha_t^2+\sigma_t^2}\leq C(1+\nu)^3 ν 2 α t 2 + σ t 2 ≤ C ( 1 + ν ) 3 , we obtain
ν ∣ σ t τ t θ t − r ∣ ≤ ( 1 − 2 2 π φ ( 1 ) ϕ ′ ′ ( 1 ) 81 ( 1 + ν ) 3 ) ∣ ν 2 α t 2 + σ t 2 − ν r ∣ + C ( 1 + ν ) 11 δ λ 2 ( 1 + ν 2 + 1 δ ) . (122) \nu|\sigma_t\tau_t\theta_t-r|\leq\left(1-\frac{2\sqrt{2\pi}\varphi(1)\phi^{\prime\prime}(1)}{81(1+\nu)^3}\right)\left|\sqrt{\nu^2\alpha_t^2+\sigma_t^2}-\nu r\right|+\frac{C(1+\nu)^{11}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right). \tag*{(122)} ν ∣ σ t τ t θ t − r ∣ ≤ ( 1 − 81 ( 1 + ν ) 3 2 2 π φ ( 1 ) ϕ ′′ ( 1 ) ) ν 2 α t 2 + σ t 2 − ν r + δ λ 2 C ( 1 + ν ) 11 ( 1 + ν 2 + δ 1 ) . (122)
We next control the large-ν 2 α t 2 + σ t 2 \sqrt{\nu^2\alpha_t^2+\sigma_t^2} ν 2 α t 2 + σ t 2 regime. On ν α t Z 1 + σ t Z 2 ν 2 α t 2 + σ t 2 = x \frac{\nu\alpha_t Z_1+\sigma_t Z_2}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}=x ν 2 α t 2 + σ t 2 ν α t Z 1 + σ t Z 2 = x , we have Z 1 = d ν α t ν 2 α t 2 + σ t 2 x + 1 − ν 2 α t 2 ν 2 α t 2 + σ t 2 G Z_1\mathrel{\mathop{=}^{\mathrm{d}}}\frac{\nu\alpha_t}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}x+\sqrt{1-\frac{\nu^2\alpha_t^2}{\nu^2\alpha_t^2+\sigma_t^2}}G Z 1 = d ν 2 α t 2 + σ t 2 ν α t x + 1 − ν 2 α t 2 + σ t 2 ν 2 α t 2 G . Let
ω t ( x ) = E G [ 2 ( 1 − π ρ ( ν ( ν α t ν 2 α t 2 + σ t 2 x + 1 − ν 2 α t 2 ν 2 α t 2 + σ t 2 G ) ) ) ] , \omega_t(x)=\mathbb{E}_G\left[2\left(1-\pi_\rho\left(\nu\left(\frac{\nu\alpha_t}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}x+\sqrt{1-\frac{\nu^2\alpha_t^2}{\nu^2\alpha_t^2+\sigma_t^2}}G\right)\right)\right)\right], ω t ( x ) = E G [ 2 ( 1 − π ρ ( ν ( ν 2 α t 2 + σ t 2 ν α t x + 1 − ν 2 α t 2 + σ t 2 ν 2 α t 2 G ) ) ) ] ,
from symmetry, ω t ( 0 ) = 1 \omega_t(0)=1 ω t ( 0 ) = 1 . Moreover, the derivative in x x x has absolute value at most ν 2 \frac{\nu}{2} 2 ν (see Eq. (116) ). Therefore ∣ ω t ( x ) − 1 ∣ ≤ ν 2 ∣ x ∣ |\omega_t(x)-1|\leq\frac{\nu}{2}|x| ∣ ω t ( x ) − 1∣ ≤ 2 ν ∣ x ∣ . Conditioning the sixth equation on ν α t Z 1 + σ t Z 2 \nu\alpha_t Z_1+\sigma_t Z_2 ν α t Z 1 + σ t Z 2 and making the exact change of variables x = u + γ t ϕ ′ ( u ) x=u+\gamma_t\phi^{\prime}(u) x = u + γ t ϕ ′ ( u ) with d x = ( 1 + γ t ϕ ′ ′ ( u ) ) d u \mathrm{d}x=(1+\gamma_t\phi^{\prime\prime}(u))\,\mathrm{d}u d x = ( 1 + γ t ϕ ′′ ( u )) d u , gives
1 = σ t τ t 2 π ( ν 2 α t 2 + σ t 2 ) ∫ R ϕ ′ ′ ( u ) ω t ( u + γ t ϕ ′ ( u ) ) exp ( − 1 2 ( u + γ t ϕ ′ ( u ) ν 2 α t 2 + σ t 2 ) 2 ) d u . (123) 1=\frac{\sigma_t\tau_t}{\sqrt{2\pi}(\nu^2\alpha_t^2+\sigma_t^2)}\int_{\mathbb{R}}\phi^{\prime\prime}(u)\omega_t\left(u+\gamma_t\phi^{\prime}(u)\right)\exp\left(-\frac{1}{2}\left(\frac{u+\gamma_t\phi^{\prime}(u)}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\right)^2\right)\,\mathrm{d}u. \tag*{(123)} 1 = 2 π ( ν 2 α t 2 + σ t 2 ) σ t τ t ∫ R ϕ ′′ ( u ) ω t ( u + γ t ϕ ′ ( u ) ) exp − 2 1 ( ν 2 α t 2 + σ t 2 u + γ t ϕ ′ ( u ) ) 2 d u . (123)
Using ∫ R ∣ u ∣ ϕ ′ ′ ( u ) d u = 2 log 2 \int_{\mathbb{R}}|u|\phi^{\prime\prime}(u)\,\mathrm{d}u=2\log2 ∫ R ∣ u ∣ ϕ ′′ ( u ) d u = 2 log 2 , ∫ R ϕ ′ ( u ) ϕ ′ ′ ( u ) d u = 1 2 \int_{\mathbb{R}}\phi^{\prime}(u)\phi^{\prime\prime}(u)\,\mathrm{d}u=\frac{1}{2} ∫ R ϕ ′ ( u ) ϕ ′′ ( u ) d u = 2 1 , we obtain
∣ ∫ R ϕ ′ ′ ( u ) ω t ( u + γ t ϕ ′ ( u ) ) exp ( − 1 2 ( u + γ t ϕ ′ ( u ) ν 2 α t 2 + σ t 2 ) 2 ) d u − 1 ∣ = ∣ ∫ R ϕ ′ ′ ( u ) ( ω t ( u + γ t ϕ ′ ( u ) ) exp ( − 1 2 ( u + γ t ϕ ′ ( u ) ν 2 α t 2 + σ t 2 ) 2 ) − 1 ) d u ∣ ≤ ∣ ∫ R ϕ ′ ′ ( u ) ( exp ( − 1 2 ( u + γ t ϕ ′ ( u ) ν 2 α t 2 + σ t 2 ) 2 ) ∣ ω t ( u + γ t ϕ ′ ( u ) ) − ω t ( 0 ) ∣ + ∣ exp ( − 1 2 ( u + γ t ϕ ′ ( u ) ν 2 α t 2 + σ t 2 ) 2 ) − 1 ∣ ) d u ∣ ≤ 1 1 − e − ν 2 2 ∣ x ∣ ≤ 1 ∫ R ϕ ′ ′ ( u ) ( ν 2 ν 2 α t 2 + σ t 2 ∣ u + γ t ϕ ′ ( u ) ∣ + ∣ u + γ t ϕ ′ ( u ) ∣ ν 2 α t 2 + σ t 2 ) d u ≤ 1 + ν 2 ν 2 α t 2 + σ t 2 ∫ R ϕ ′ ′ ( u ) ∣ u + γ t ϕ ′ ( u ) ∣ d u ≤ 1 + ν 2 ν 2 α t 2 + σ t 2 ( 2 log 2 + γ t 2 ) . \begin{aligned}
\left|\int_{\mathbb{R}}\phi^{\prime\prime}(u)\omega_t\left(u+\gamma_t\phi^{\prime}(u)\right)\exp\left(-\frac{1}{2}\left(\frac{u+\gamma_t\phi^{\prime}(u)}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\right)^2\right)\,\mathrm{d}u-1\right| \\
&=\left|\int_{\mathbb{R}}\phi^{\prime\prime}(u)\left(\omega_t\left(u+\gamma_t\phi^{\prime}(u)\right)\exp\left(-\frac{1}{2}\left(\frac{u+\gamma_t\phi^{\prime}(u)}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\right)^2\right)-1\right)\,\mathrm{d}u\right| \\
&\leq\left|\int_{\mathbb{R}}\phi^{\prime\prime}(u)\left(\exp\left(-\frac{1}{2}\left(\frac{u+\gamma_t\phi^{\prime}(u)}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\right)^2\right)\left|\omega_t\left(u+\gamma_t\phi^{\prime}(u)\right)-\omega_t(0)\right|+\left|\exp\left(-\frac{1}{2}\left(\frac{u+\gamma_t\phi^{\prime}(u)}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\right)^2\right)-1\right|\right)\,\mathrm{d}u\right| \\
&\leq\frac{1}{1-\mathrm{e}^{-\frac{\nu^2}{2}}|x|\leq1}\int_{\mathbb{R}}\phi^{\prime\prime}(u)\left(\frac{\nu}{2\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}|u+\gamma_t\phi^{\prime}(u)|+\frac{|u+\gamma_t\phi^{\prime}(u)|}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\right)\,\mathrm{d}u \\
&\leq\frac{1+\frac{\nu}{2}}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\int_{\mathbb{R}}\phi^{\prime\prime}(u)|u+\gamma_t\phi^{\prime}(u)|\,\mathrm{d}u \\
&\leq\frac{1+\frac{\nu}{2}}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\left(2\log2+\frac{\gamma_t}{2}\right).
\end{aligned} ∫ R ϕ ′′ ( u ) ω t ( u + γ t ϕ ′ ( u ) ) exp − 2 1 ( ν 2 α t 2 + σ t 2 u + γ t ϕ ′ ( u ) ) 2 d u − 1 = ∫ R ϕ ′′ ( u ) ω t ( u + γ t ϕ ′ ( u ) ) exp − 2 1 ( ν 2 α t 2 + σ t 2 u + γ t ϕ ′ ( u ) ) 2 − 1 d u ≤ ∫ R ϕ ′′ ( u ) exp − 2 1 ( ν 2 α t 2 + σ t 2 u + γ t ϕ ′ ( u ) ) 2 ∣ ω t ( u + γ t ϕ ′ ( u ) ) − ω t ( 0 ) ∣ + exp − 2 1 ( ν 2 α t 2 + σ t 2 u + γ t ϕ ′ ( u ) ) 2 − 1 d u ≤ 1 − e − 2 ν 2 ∣ x ∣ ≤ 1 1 ∫ R ϕ ′′ ( u ) ( 2 ν 2 α t 2 + σ t 2 ν ∣ u + γ t ϕ ′ ( u ) ∣ + ν 2 α t 2 + σ t 2 ∣ u + γ t ϕ ′ ( u ) ∣ ) d u ≤ ν 2 α t 2 + σ t 2 1 + 2 ν ∫ R ϕ ′′ ( u ) ∣ u + γ t ϕ ′ ( u ) ∣ d u ≤ ν 2 α t 2 + σ t 2 1 + 2 ν ( 2 log 2 + 2 γ t ) .
Recall that
1 − 2 π ν h ( ν ) = ∫ R φ ′ ′ ( u ) ( 1 − e − u 2 2 ν 2 ) d u . 1-\sqrt{2\pi}\nu h(\nu)=\int_{\mathbb{R}}\varphi''(u)\left(1-e^{-\frac{u^2}{2\nu^2}}\right)\,\mathrm{d}u. 1 − 2 π ν h ( ν ) = ∫ R φ ′′ ( u ) ( 1 − e − 2 ν 2 u 2 ) d u .
Using ∣ u ∣ ≥ 1 |u|\ge1 ∣ u ∣ ≥ 1 , 1 − 2 π ν h ( ν ) ≥ 2 ( 1 + e ) ( 1 + 2 ν 2 ) 1-\sqrt{2\pi}\nu h(\nu)\ge\frac{2}{(1+e)(1+2\nu^2)} 1 − 2 π ν h ( ν ) ≥ ( 1 + e ) ( 1 + 2 ν 2 ) 2 . Since 2 π ν h ( ν r ) = 16 9 2 π ν ( 1 − 2 ρ ) h ( ν ) ≤ 2 π ν h ( ν ) \sqrt{2\pi}\nu h(\nu r)=\frac{16}{9}\sqrt{2\pi}\nu(1-2\rho)h(\nu)\le\sqrt{2\pi}\nu h(\nu) 2 π ν h ( ν r ) = 9 16 2 π ν ( 1 − 2 ρ ) h ( ν ) ≤ 2 π ν h ( ν ) , so,
1 − 2 π ν r h ( ν r ) ≥ 2 ( 1 + e ) ( 1 + 2 ν 2 ) . (124) 1-\sqrt{2\pi}\nu r h(\nu r)\ge\frac{2}{(1+e)(1+2\nu^2)}. \tag*{(124)} 1 − 2 π ν r h ( ν r ) ≥ ( 1 + e ) ( 1 + 2 ν 2 ) 2 . (124)
Hence, whenever ν 2 α t 2 + σ t 2 > C ( 1 + ν ) 3 \sqrt{\nu^2\alpha_t^2+\sigma_t^2}>C(1+\nu)^3 ν 2 α t 2 + σ t 2 > C ( 1 + ν ) 3 and λ 2 > C ( 1 + ν ) 7 δ ( 1 + ν 2 + 1 δ ) \lambda^2>C\frac{(1+\nu)^7}{\delta}\left(1+\nu^2+\frac{1}{\delta}\right) λ 2 > C δ ( 1 + ν ) 7 ( 1 + ν 2 + δ 1 ) for a sufficiently large universal constant C C C , Eq. (115) gives γ t ν 2 α t 2 + σ t 2 ≤ C 1 + ν δ ( 1 + ν 2 + 1 δ ) 1 ( 1 + ν ) 6 \frac{\gamma_t}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\le\frac{C\frac{1+\nu}{\delta}\left(1+\nu^2+\frac{1}{\delta}\right)}{\frac{1}{(1+\nu)^6}} ν 2 α t 2 + σ t 2 γ t ≤ ( 1 + ν ) 6 1 C δ 1 + ν ( 1 + ν 2 + δ 1 ) . Consequently, after increasing C C C if necessary,
1 + ν 2 ν 2 α t 2 + σ t 2 ( 2 log 2 + γ t 2 ) ≤ 1 ( 1 + e ) ( 1 + 2 ν 2 ) . \frac{1+\frac{\nu}{2}}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\left(2\log2+\frac{\gamma_t}{2}\right)\le\frac{1}{(1+e)(1+2\nu^2)}. ν 2 α t 2 + σ t 2 1 + 2 ν ( 2 log 2 + 2 γ t ) ≤ ( 1 + e ) ( 1 + 2 ν 2 ) 1 .
Therefore, denoting the integral in Eq. (123) by I t I_t I t , we have I t ≥ 1 − 1 ( 1 + e ) ( 1 + 2 ν 2 ) I_t\ge1-\frac{1}{(1+e)(1+2\nu^2)} I t ≥ 1 − ( 1 + e ) ( 1 + 2 ν 2 ) 1 . Using Eq. (123) , θ t ≤ r h ( ν r ) \theta_t\le rh(\nu r) θ t ≤ r h ( ν r ) (it follows from the fifth equation where θ t ≤ 2 E [ π ρ ′ ( ν Z 1 ) φ ′ ( ν α t Z 1 + σ t Z 2 ) ] = E [ π ρ ′ ( ν Z 1 ) ] = ( 1 − 2 ρ ) h ( ν ) = r h ( ν r ) \theta_t\le2\mathbb{E}[\pi'_\rho(\nu Z_1)\varphi'(\nu\alpha_t Z_1+\sigma_t Z_2)]=\mathbb{E}[\pi'_\rho(\nu Z_1)]=(1-2\rho)h(\nu)=rh(\nu r) θ t ≤ 2 E [ π ρ ′ ( ν Z 1 ) φ ′ ( ν α t Z 1 + σ t Z 2 )] = E [ π ρ ′ ( ν Z 1 )] = ( 1 − 2 ρ ) h ( ν ) = r h ( ν r ) ) and Eq. (124) , we obtain
ν σ t π t θ t ν 2 α t 2 + σ t 2 = 2 π ν θ t I t ≤ 2 π ν r h ( ν r ) 1 − 1 ( 1 + e ) ( 1 + 2 ν 2 ) ≤ 1 − 2 ( 1 + e ) ( 1 + 2 ν 2 ) 1 − 1 ( 1 + e ) ( 1 + 2 ν 2 ) < 1 ≤ 1 − 1 2 ( 1 + e ) ( 1 + 2 ν 2 ) . \frac{\nu\sigma_t\pi_t\theta_t}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}=\frac{\sqrt{2\pi}\nu\theta_t}{I_t}\le\frac{\sqrt{2\pi}\nu r h(\nu r)}{1-\frac{1}{(1+e)(1+2\nu^2)}}\le\frac{1-\frac{2}{(1+e)(1+2\nu^2)}}{1-\frac{1}{(1+e)(1+2\nu^2)}}<1\le1-\frac{1}{2(1+e)(1+2\nu^2)}. ν 2 α t 2 + σ t 2 ν σ t π t θ t = I t 2 π ν θ t ≤ 1 − ( 1 + e ) ( 1 + 2 ν 2 ) 1 2 π ν r h ( ν r ) ≤ 1 − ( 1 + e ) ( 1 + 2 ν 2 ) 1 1 − ( 1 + e ) ( 1 + 2 ν 2 ) 2 < 1 ≤ 1 − 2 ( 1 + e ) ( 1 + 2 ν 2 ) 1 .
which gives
ν σ t π t θ t ν 2 α t 2 + σ t 2 ≤ 1 − 1 2 ( 1 + e ) ( 1 + 2 ν 2 ) . \frac{\nu\sigma_t\pi_t\theta_t}{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}\le1-\frac{1}{2(1+e)(1+2\nu^2)}. ν 2 α t 2 + σ t 2 ν σ t π t θ t ≤ 1 − 2 ( 1 + e ) ( 1 + 2 ν 2 ) 1 .
Since ν r ≤ ν \nu r\le\nu ν r ≤ ν (Lemma 16) and ν 2 α t 2 + σ t 2 ≥ C ( 1 + ν ) 3 \sqrt{\nu^2\alpha_t^2+\sigma_t^2}\ge C(1+\nu)^3 ν 2 α t 2 + σ t 2 ≥ C ( 1 + ν ) 3 , increasing the universal constant C C C if necessary gives
ν ∣ σ t π t θ t − r ∣ ≤ ( 1 − 1 2 ( 1 + e ) ( 1 + 2 ν 2 ) ) ∣ ν 2 α t 2 + σ t 2 − ν r ∣ ≤ ( 1 − 2 2 π φ ( 1 ) φ ′ ′ ( 1 ) 81 ( 1 + ν ) 3 ) ∣ ν 2 α t 2 + σ t 2 − ν r ∣ . (125) \nu\left|\sigma_t\pi_t\theta_t-r\right|\le\left(1-\frac{1}{2(1+e)(1+2\nu^2)}\right)\left|\sqrt{\nu^2\alpha_t^2+\sigma_t^2}-\nu r\right|\le\left(1-\frac{2\sqrt{2\pi}\varphi(1)\varphi''(1)}{81(1+\nu)^3}\right)\left|\sqrt{\nu^2\alpha_t^2+\sigma_t^2}-\nu r\right|. \tag*{(125)} ν ∣ σ t π t θ t − r ∣ ≤ ( 1 − 2 ( 1 + e ) ( 1 + 2 ν 2 ) 1 ) ν 2 α t 2 + σ t 2 − ν r ≤ ( 1 − 81 ( 1 + ν ) 3 2 2 π φ ( 1 ) φ ′′ ( 1 ) ) ν 2 α t 2 + σ t 2 − ν r . (125)
Combining Eqs. (122) and (125) , we conclude that, provided
λ 2 ≥ C [ ( 1 + ν ) 7 δ ( 1 + ν 2 + 1 δ ) + ( 1 + ν ) 10 δ 2 ] (126) \lambda^2\ge C\left[\frac{(1+\nu)^7}{\delta}\left(1+\nu^2+\frac{1}{\delta}\right)+\frac{(1+\nu)^{10}}{\delta^2}\right] \tag*{(126)} λ 2 ≥ C [ δ ( 1 + ν ) 7 ( 1 + ν 2 + δ 1 ) + δ 2 ( 1 + ν ) 10 ] (126)
for a sufficiently large universal constant C C C , then for every t t t ,
ν ∣ σ t π t θ t − r ∣ ≤ ( 1 − 2 2 π φ ( 1 ) φ ′ ′ ( 1 ) 81 ( 1 + ν ) 3 ) ∣ ν 2 α t 2 + σ t 2 − ν r ∣ + C ( 1 + ν ) 11 δ λ 2 ( 1 + ν 2 + 1 δ ) . (127) \nu\left|\sigma_t\pi_t\theta_t-r\right|\le\left(1-\frac{2\sqrt{2\pi}\varphi(1)\varphi''(1)}{81(1+\nu)^3}\right)\left|\sqrt{\nu^2\alpha_t^2+\sigma_t^2}-\nu r\right|+\frac{C(1+\nu)^{11}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right). \tag*{(127)} ν ∣ σ t π t θ t − r ∣ ≤ ( 1 − 81 ( 1 + ν ) 3 2 2 π φ ( 1 ) φ ′′ ( 1 ) ) ν 2 α t 2 + σ t 2 − ν r + δ λ 2 C ( 1 + ν ) 11 ( 1 + ν 2 + δ 1 ) . (127)
Finally, by the reverse triangle inequality,
∣ ν 2 α t 2 + σ t 2 − ν r ∣ = ∣ ∥ ( ν α t , σ t ) ∥ − ∥ ( ν r , 0 ) ∥ ∣ ≤ ν 2 ( α t − r ) 2 + σ t 2 = E [ ( w t − r w ⋆ ) 2 ] . (128) \left|\sqrt{\nu^2\alpha_t^2+\sigma_t^2}-\nu r\right|=\left|\left\|(\nu\alpha_t,\sigma_t)\right\|-\left\|(\nu r,0)\right\|\right|\le\sqrt{\nu^2(\alpha_t-r)^2+\sigma_t^2}=\sqrt{\mathbb{E}[(w_t-rw^\star)^2]}. \tag*{(128)} ν 2 α t 2 + σ t 2 − ν r = ∣ ∥ ( ν α t , σ t ) ∥ − ∥ ( ν r , 0 ) ∥ ∣ ≤ ν 2 ( α t − r ) 2 + σ t 2 = E [( w t − r w ⋆ ) 2 ] . (128)
Hence,
ν ∣ σ t π t θ t − r ∣ ≤ ( 1 − 2 2 π φ ( 1 ) φ ′ ′ ( 1 ) 81 ( 1 + ν ) 3 ) E [ ( w t − r w ⋆ ) 2 ] + C ( 1 + ν ) 11 δ λ 2 ( 1 + ν 2 + 1 δ ) . (129) \nu\left|\sigma_t\pi_t\theta_t-r\right|\le\left(1-\frac{2\sqrt{2\pi}\varphi(1)\varphi''(1)}{81(1+\nu)^3}\right)\sqrt{\mathbb{E}[(w_t-rw^\star)^2]}+\frac{C(1+\nu)^{11}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right). \tag*{(129)} ν ∣ σ t π t θ t − r ∣ ≤ ( 1 − 81 ( 1 + ν ) 3 2 2 π φ ( 1 ) φ ′′ ( 1 ) ) E [( w t − r w ⋆ ) 2 ] + δ λ 2 C ( 1 + ν ) 11 ( 1 + ν 2 + δ 1 ) . (129)
We now show the following identity. Let a , b > 0 a,b>0 a , b > 0 and fixed x x x , then we have
E [ Π [ a ( Z − b ) , a ( Z + b ) ] ( x ) 2 ] ≤ ( 1 − 2 Φ ( − b ) ) x 2 + C a 2 2 Φ ( − b ) b 2 . (130) \mathbb{E}\left[\Pi_{[a(Z-b),a(Z+b)]}(x)^2\right]\leq(1-2\Phi(-b))x^2+C\frac{a^2 2\Phi(-b)}{b^2}. \tag*{(130)} E [ Π [ a ( Z − b ) , a ( Z + b )] ( x ) 2 ] ≤ ( 1 − 2Φ ( − b )) x 2 + C b 2 a 2 2Φ ( − b ) . (130)
After rescaling it suffices to take a = 1 a=1 a = 1 . The function g b ( x ) = E [ Π [ Z − b , Z + b ] ( x ) 2 ] g_b(x)=\mathbb{E}\left[\Pi_{[Z-b,Z+b]}(x)^2\right] g b ( x ) = E [ Π [ Z − b , Z + b ] ( x ) 2 ] is even and, for x ≥ 0 x\geq0 x ≥ 0 ,
g b ′ ( x ) = 2 x P ( ∣ Z − x ∣ ≤ b ) ≤ 2 x ( 1 − 2 Φ ( − b ) ) , g_b'(x)=2x\mathbb{P}(|Z-x|\leq b)\leq2x(1-2\Phi(-b)), g b ′ ( x ) = 2 x P ( ∣ Z − x ∣ ≤ b ) ≤ 2 x ( 1 − 2Φ ( − b )) ,
since a Gaussian interval of fixed length has maximal probability when centered at zero. Hence g b ( x ) ≤ g b ( 0 ) + ( 1 − 2 Φ ( − b ) ) x 2 g_b(x)\leq g_b(0)+(1-2\Phi(-b))x^2 g b ( x ) ≤ g b ( 0 ) + ( 1 − 2Φ ( − b )) x 2 . Moreover,
g b ( 0 ) = E [ ( ∣ Z ∣ − b ) + 2 ] = 2 ∫ b ∞ ( z − b ) 2 φ ( z ) d z = 2 φ ( b ) ∫ 0 ∞ y 2 e − b y − y 2 2 d y ≤ 2 φ ( b ) ∫ 0 ∞ y 2 e − b y d y = 4 φ ( b ) b 3 , g_b(0)=\mathbb{E}\left[(|Z|-b)_+^2\right]=2\int_b^\infty(z-b)^2\varphi(z)\,\mathrm{d}z=2\varphi(b)\int_0^\infty y^2e^{-by-\frac{y^2}{2}}\,\mathrm{d}y\leq2\varphi(b)\int_0^\infty y^2e^{-by}\,\mathrm{d}y=\frac{4\varphi(b)}{b^3}, g b ( 0 ) = E [ ( ∣ Z ∣ − b ) + 2 ] = 2 ∫ b ∞ ( z − b ) 2 φ ( z ) d z = 2 φ ( b ) ∫ 0 ∞ y 2 e − b y − 2 y 2 d y ≤ 2 φ ( b ) ∫ 0 ∞ y 2 e − b y d y = b 3 4 φ ( b ) ,
and hence, for b ≥ 1 b\geq1 b ≥ 1 , by Mills inequality, φ ( b ) ≤ ( b + 1 b ) Φ ( − b ) ≤ 2 b Φ ( − b ) \varphi(b)\leq(b+\frac{1}{b})\Phi(-b)\leq2b\Phi(-b) φ ( b ) ≤ ( b + b 1 ) Φ ( − b ) ≤ 2 b Φ ( − b ) . This proves Eq. (130) . Now suppose X ′ = Π [ d + a ( Z − b ) , d + a ( Z + b ) ] ( X ) X'=\Pi_{[d+a(Z-b),d+a(Z+b)]}(X) X ′ = Π [ d + a ( Z − b ) , d + a ( Z + b )] ( X ) , where ∣ d ∣ ≤ D |d|\leq D ∣ d ∣ ≤ D . We claim that
E [ ( ∣ X ′ ∣ − D ) + 2 ∣ X ] ≤ ( 1 − 2 Φ ( − b ) ) ( ∣ X ∣ − D ) + 2 + C a 2 2 Φ ( − b ) b 2 . (131) \mathbb{E}\left[(|X'|-D)_+^2\mid X\right]\leq(1-2\Phi(-b))(|X|-D)_+^2+C\frac{a^2 2\Phi(-b)}{b^2}. \tag*{(131)} E [ ( ∣ X ′ ∣ − D ) + 2 ∣ X ] ≤ ( 1 − 2Φ ( − b )) ( ∣ X ∣ − D ) + 2 + C b 2 a 2 2Φ ( − b ) . (131)
To prove the claim, first assume X > D X>D X > D . Note that
X ′ = d + Π [ a ( Z − b ) , a ( Z + b ) ] ( X − d ) . X'=d+\Pi_{[a(Z-b),a(Z+b)]}(X-d). X ′ = d + Π [ a ( Z − b ) , a ( Z + b )] ( X − d ) .
Since Π [ a ( Z − b ) , a ( Z + b ) ] ( ⋅ ) \Pi_{[a(Z-b),a(Z+b)]}(\cdot) Π [ a ( Z − b ) , a ( Z + b )] ( ⋅ ) is increasing and 1-Lipschitz,
( X ′ − D ) + = ( Π [ a ( Z − b ) , a ( Z + b ) ] ( X − d ) − Π [ a ( Z − b ) , a ( Z + b ) ] ( X − D ) + Π [ a ( Z − b ) , a ( Z + b ) ] ( X − D ) − ( D − d ) ) + ≤ ( Π [ a ( Z − b ) , a ( Z + b ) ] ( X − D ) ) + . \begin{aligned}
(X'-D)_+=\left(\Pi_{[a(Z-b),a(Z+b)]}(X-d)-\Pi_{[a(Z-b),a(Z+b)]}(X-D)+\Pi_{[a(Z-b),a(Z+b)]}(X-D)-(D-d)\right)_+ \\
&\leq\left(\Pi_{[a(Z-b),a(Z+b)]}(X-D)\right)_+.
\end{aligned} ( X ′ − D ) + = ( Π [ a ( Z − b ) , a ( Z + b )] ( X − d ) − Π [ a ( Z − b ) , a ( Z + b )] ( X − D ) + Π [ a ( Z − b ) , a ( Z + b )] ( X − D ) − ( D − d ) ) + ≤ ( Π [ a ( Z − b ) , a ( Z + b )] ( X − D ) ) + .
Likewise,
( − D − X ′ ) + ≤ ( − Π [ a ( Z − b ) , a ( Z + b ) ] ( X − D ) ) + , (-D-X')_+\leq\left(-\Pi_{[a(Z-b),a(Z+b)]}(X-D)\right)_+, ( − D − X ′ ) + ≤ ( − Π [ a ( Z − b ) , a ( Z + b )] ( X − D ) ) + ,
and hence
( ∣ X ′ ∣ − D ) + ≤ ∣ Π [ a ( Z − b ) , a ( Z + b ) ] ( X − D ) ∣ . (|X'|-D)_+\leq\left|\Pi_{[a(Z-b),a(Z+b)]}(X-D)\right|. ( ∣ X ′ ∣ − D ) + ≤ Π [ a ( Z − b ) , a ( Z + b )] ( X − D ) .
The case X < − D X<-D X < − D follows by reflection.
Finally, suppose ∣ X ∣ ≤ D |X|\leq D ∣ X ∣ ≤ D . Since − D − d ≤ X − d ≤ D − d -D-d\leq X-d\leq D-d − D − d ≤ X − d ≤ D − d , monotonicity and the 1-Lipschitz property give ( X ′ − D ) + ≤ ( Π [ a ( Z − b ) , a ( Z + b ) ] ( 0 ) ) + (X'-D)_+\leq\left(\Pi_{[a(Z-b),a(Z+b)]}(0)\right)_+ ( X ′ − D ) + ≤ ( Π [ a ( Z − b ) , a ( Z + b )] ( 0 ) ) + , and ( − D − X ′ ) + ≤ ( − Π [ a ( Z − b ) , a ( Z + b ) ] ( 0 ) ) + (-D-X')_+\leq\left(-\Pi_{[a(Z-b),a(Z+b)]}(0)\right)_+ ( − D − X ′ ) + ≤ ( − Π [ a ( Z − b ) , a ( Z + b )] ( 0 ) ) + . Thus ( ∣ X ′ ∣ − D ) + ≤ ∣ Π [ a ( Z − b ) , a ( Z + b ) ] ( 0 ) ∣ (|X'|-D)_+\leq\left|\Pi_{[a(Z-b),a(Z+b)]}(0)\right| ( ∣ X ′ ∣ − D ) + ≤ Π [ a ( Z − b ) , a ( Z + b )] ( 0 ) . Applying Eq. (130) proves Eq. (131) .
Fix integers u < t u<t u < t . By Eq. (129) ,
ν max u < j ≤ t ∣ σ j τ j θ j − r ∣ ≤ ( 1 − 2 2 π φ ( 1 ) ϕ ′ ′ ( 1 ) 81 ( 1 + ν ) 3 ) max u ≤ j ≤ t E [ ( w j − r w ⋆ ) 2 ] + C ( 1 + ν ) 11 δ λ 2 ( 1 + ν 2 + 1 δ ) . (132) \nu\max_{u<j\leq t}|\sigma_j\tau_j\theta_j-r|\leq\left(1-\frac{2\sqrt{2\pi}\varphi(1)\phi''(1)}{81(1+\nu)^3}\right)\max_{u\leq j\leq t}\sqrt{\mathbb{E}[(w_j-rw^\star)^2]}+\frac{C(1+\nu)^{11}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right). \tag*{(132)} ν u < j ≤ t max ∣ σ j τ j θ j − r ∣ ≤ ( 1 − 81 ( 1 + ν ) 3 2 2 π φ ( 1 ) ϕ ′′ ( 1 ) ) u ≤ j ≤ t max E [( w j − r w ⋆ ) 2 ] + δ λ 2 C ( 1 + ν ) 11 ( 1 + ν 2 + δ 1 ) . (132)
The scalar proximal update is written as,
w j − r w ⋆ = Π [ σ j τ j ( θ j w ⋆ + r j δ z j ) − λ σ j τ j − r w ⋆ , σ j τ j ( θ j w ⋆ + r j δ z j ) + λ σ j τ j − r w ⋆ ] ( w j − 1 − r w ⋆ ) w_j-rw^\star=\Pi_{\left[\sigma_j\tau_j\left(\theta_jw^\star+\frac{r_j}{\sqrt{\delta}}z_j\right)-\lambda\sigma_j\tau_j-rw^\star,\ \sigma_j\tau_j\left(\theta_jw^\star+\frac{r_j}{\sqrt{\delta}}z_j\right)+\lambda\sigma_j\tau_j-rw^\star\right]}(w_{j-1}-rw^\star) w j − r w ⋆ = Π [ σ j τ j ( θ j w ⋆ + δ r j z j ) − λ σ j τ j − r w ⋆ , σ j τ j ( θ j w ⋆ + δ r j z j ) + λ σ j τ j − r w ⋆ ] ( w j − 1 − r w ⋆ )
Each projection center belongs to [ − ∣ w ∣ max u < j ≤ t ∣ σ j τ j θ j − r ∣ , ∣ w ∣ max u < j ≤ t ∣ σ j τ j θ j − r ∣ ] \left[-|w|\max_{u<j\leq t}|\sigma_j\tau_j\theta_j-r|,|w|\max_{u<j\leq t}|\sigma_j\tau_j\theta_j-r|\right] [ − ∣ w ∣ max u < j ≤ t ∣ σ j τ j θ j − r ∣ , ∣ w ∣ max u < j ≤ t ∣ σ j τ j θ j − r ∣ ] . By φ ( 1 ) 2 ( 1 + e ) 2 ( 1 + ν ) ≤ r j \sqrt{\frac{\varphi(1)}{2(1+e)^2(1+\nu)}}\leq r_j 2 ( 1 + e ) 2 ( 1 + ν ) φ ( 1 ) ≤ r j ,
2 Φ ( − λ δ r j ) ≥ 2 Φ ( − λ 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) ) . 2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\geq2\Phi\left(-\lambda\sqrt{\frac{2(1+e)^2\delta(1+\nu)}{\varphi(1)}}\right). 2Φ ( − r j λ δ ) ≥ 2Φ ( − λ φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) ) .
Using Eq. (128) ,
ν 2 α j 2 + σ j 2 ≤ ν r + E [ ( w j − r w ⋆ ) 2 ] , \sqrt{\nu^2\alpha_j^2+\sigma_j^2}\leq\nu r+\sqrt{\mathbb{E}[(w_j-rw^\star)^2]}, ν 2 α j 2 + σ j 2 ≤ ν r + E [( w j − r w ⋆ ) 2 ] ,
and therefore from Eq. (114) and r ≤ 1 r \le1 r ≤ 1 ,
σ j τ j ≤ 32 2 π e 1 2 ( 1 + ν ) ( 1 + ν + E [ ( w j − r w ⋆ ) 2 ] ) ≤ 32 2 π e 1 2 ( 1 + ν ) 2 ( 1 + max u ≤ k ≤ t E [ ( w k − r w ⋆ ) 2 ] ) . \sigma_j\tau_j \le32\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu)\left(1+\nu+\sqrt{\mathbb{E}\left[(w_j-rw^\star)^2\right]}\right)\le32\sqrt{2\pi}e^{\frac{1}{2}}(1+\nu)^2\left(1+\max_{u\le k\le t}\sqrt{\mathbb{E}\left[(w_k-rw^\star)^2\right]}\right). σ j τ j ≤ 32 2 π e 2 1 ( 1 + ν ) ( 1 + ν + E [ ( w j − r w ⋆ ) 2 ] ) ≤ 32 2 π e 2 1 ( 1 + ν ) 2 ( 1 + u ≤ k ≤ t max E [ ( w k − r w ⋆ ) 2 ] ) .
Consequently using r j ≤ 1 r_j\le1 r j ≤ 1 ,
( σ j τ j r j δ ) 2 ( λ δ r j ) 2 = ( σ j τ j ) 2 r j 4 δ 2 λ 2 ≤ C ( 1 + ν ) 4 δ 2 λ 2 ( 1 + max u ≤ k ≤ t E [ ( w k − r w ⋆ ) 2 ] ) 2 . (133) \frac{\left(\frac{\sigma_j\tau_jr_j}{\sqrt{\delta}}\right)^2}{\left(\frac{\lambda\sqrt{\delta}}{r_j}\right)^2}=\frac{(\sigma_j\tau_j)^2r_j^4}{\delta^2\lambda^2}\le\frac{C(1+\nu)^4}{\delta^2\lambda^2}\left(1+\max_{u\le k\le t}\sqrt{\mathbb{E}\left[(w_k-rw^\star)^2\right]}\right)^2. \tag*{(133)} ( r j λ δ ) 2 ( δ σ j τ j r j ) 2 = δ 2 λ 2 ( σ j τ j ) 2 r j 4 ≤ δ 2 λ 2 C ( 1 + ν ) 4 ( 1 + u ≤ k ≤ t max E [ ( w k − r w ⋆ ) 2 ] ) 2 . (133)
Applying Eq. (131) successively over the block, and using the telescopic identity
∑ j = u + 1 t 2 Φ ( − λ δ r j ) ∏ k = j + 1 t ( 1 − 2 Φ ( − λ δ r k ) ) = 1 − ∏ j = u + 1 t ( 1 − 2 Φ ( − λ δ r j ) ) ≤ 1 , \sum_{j=u+1}^{t}2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\prod_{k=j+1}^{t}\left(1-2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_k}\right)\right)=1-\prod_{j=u+1}^{t}\left(1-2\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_j}\right)\right)\le1, j = u + 1 ∑ t 2Φ ( − r j λ δ ) k = j + 1 ∏ t ( 1 − 2Φ ( − r k λ δ ) ) = 1 − j = u + 1 ∏ t ( 1 − 2Φ ( − r j λ δ ) ) ≤ 1 ,
with d = ( σ j τ j θ j − r ) w ⋆ d=(\sigma_j\tau_j\theta_j-r)w^\star d = ( σ j τ j θ j − r ) w ⋆ , D = ∣ w ⋆ ∣ max u < k ≤ t ∣ σ k τ k θ k − r ∣ D=|w^\star|\max_{u<k\le t}|\sigma_k\tau_k\theta_k-r| D = ∣ w ⋆ ∣ max u < k ≤ t ∣ σ k τ k θ k − r ∣ , X = w j − 1 − r w ⋆ X=w_{j-1}-rw^\star X = w j − 1 − r w ⋆ , X ′ = w j − r w ⋆ X'=w_j-rw^\star X ′ = w j − r w ⋆ , gives
E [ ( ∣ w t − r w ⋆ ∣ − ∣ w ⋆ ∣ max u < k ≤ t ∣ σ k τ k θ k − r ∣ ) + 2 ] ≤ exp ( − 2 Φ ( − λ 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) ) ( t − u ) ) E [ ( w u − r w ⋆ ) 2 ] + C ( 1 + ν ) 4 δ 2 λ 2 ( 1 + max u ≤ j ≤ t E [ ( w j − r w ⋆ ) 2 ] ) 2 . \mathbb{E}\left[\left(|w_t-rw^\star|-|w^\star|\max_{u<k\le t}|\sigma_k\tau_k\theta_k-r|\right)_+^2\right]\le\exp\left(-2\Phi\left(-\lambda\sqrt{\frac{2(1+e)^2\delta(1+\nu)}{\varphi(1)}}\right)(t-u)\right)\mathbb{E}\left[(w_u-rw^\star)^2\right]
+\frac{C(1+\nu)^4}{\delta^2\lambda^2}\left(1+\max_{u\le j\le t}\sqrt{\mathbb{E}\left[(w_j-rw^\star)^2\right]}\right)^2. E [ ( ∣ w t − r w ⋆ ∣ − ∣ w ⋆ ∣ u < k ≤ t max ∣ σ k τ k θ k − r ∣ ) + 2 ] ≤ exp ( − 2Φ ( − λ φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) ) ( t − u ) ) E [ ( w u − r w ⋆ ) 2 ] + δ 2 λ 2 C ( 1 + ν ) 4 ( 1 + u ≤ j ≤ t max E [ ( w j − r w ⋆ ) 2 ] ) 2 .
From Minkowski inequality, ∣ w t − r w ⋆ ∣ ≤ ( ∣ w t − r w ⋆ ∣ − ∣ w ⋆ ∣ max u < k ≤ t ∣ σ k τ k θ k − r ∣ ) + + ∣ w ⋆ ∣ max u < k ≤ t ∣ σ k τ k θ k − r ∣ |w_t-rw^\star|\le(|w_t-rw^\star|-|w^\star|\max_{u<k\le t}|\sigma_k\tau_k\theta_k-r|)_++|w^\star|\max_{u<k\le t}|\sigma_k\tau_k\theta_k-r| ∣ w t − r w ⋆ ∣ ≤ ( ∣ w t − r w ⋆ ∣ − ∣ w ⋆ ∣ max u < k ≤ t ∣ σ k τ k θ k − r ∣ ) + + ∣ w ⋆ ∣ max u < k ≤ t ∣ σ k τ k θ k − r ∣ ,
E [ ( w t − r w ⋆ ) 2 ] ≤ E [ ( ∣ w t − r w ⋆ ∣ − ∣ w ⋆ ∣ max u < k ≤ t ∣ σ k τ k θ k − r ∣ ) + 2 ] + ν max u < k ≤ t ∣ σ k τ k θ k − r ∣ . \sqrt{\mathbb{E}[(w_t-rw^\star)^2]}\le\sqrt{\mathbb{E}\left[\left(|w_t-rw^\star|-|w^\star|\max_{u<k\le t}|\sigma_k\tau_k\theta_k-r|\right)_+^2\right]}+\nu\max_{u<k\le t}|\sigma_k\tau_k\theta_k-r|. E [( w t − r w ⋆ ) 2 ] ≤ E [ ( ∣ w t − r w ⋆ ∣ − ∣ w ⋆ ∣ u < k ≤ t max ∣ σ k τ k θ k − r ∣ ) + 2 ] + ν u < k ≤ t max ∣ σ k τ k θ k − r ∣.
Combining with Eq. (132) yields
E [ ( w t − r w ⋆ ) 2 ] ≤ exp ( − Φ ( − λ 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) ) ( t − u ) ) E [ ( w u − r w ⋆ ) 2 ] + ( 1 − 2 2 π φ ( 1 ) φ ′ ′ ( 1 ) 81 ( 1 + ν ) 3 + C ( 1 + ν ) 2 δ λ ) max u ≤ j ≤ t E [ ( w j − r w ⋆ ) 2 ] + C ( 1 + ν ) 2 δ λ + C ( 1 + ν ) 11 δ λ 2 ( 1 + ν 2 + 1 δ ) . (134) \sqrt{\mathbb{E}[(w_t-rw^\star)^2]}\le\exp\left(-\Phi\left(-\lambda\sqrt{\frac{2(1+e)^2\delta(1+\nu)}{\varphi(1)}}\right)(t-u)\right)\sqrt{\mathbb{E}[(w_u-rw^\star)^2]}
+\left(1-\frac{2\sqrt{2}\pi\varphi(1)\varphi''(1)}{81(1+\nu)^3}+\frac{C(1+\nu)^2}{\delta\lambda}\right)\max_{u\le j\le t}\sqrt{\mathbb{E}[(w_j-rw^\star)^2]}
+\frac{C(1+\nu)^2}{\delta\lambda}+\frac{C(1+\nu)^{11}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right). \tag*{(134)} E [( w t − r w ⋆ ) 2 ] ≤ exp ( − Φ ( − λ φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) ) ( t − u ) ) E [( w u − r w ⋆ ) 2 ] + ( 1 − 81 ( 1 + ν ) 3 2 2 π φ ( 1 ) φ ′′ ( 1 ) + δ λ C ( 1 + ν ) 2 ) u ≤ j ≤ t max E [( w j − r w ⋆ ) 2 ] + δ λ C ( 1 + ν ) 2 + δ λ 2 C ( 1 + ν ) 11 ( 1 + ν 2 + δ 1 ) . (134)
By Eq. (126) , after increasing the universal constant C C C if necessary, C ( 1 + ν ) 2 δ λ ≤ 2 π φ ( 1 ) φ ′ ′ ( 1 ) 324 ( 1 + ν ) 3 \frac{C(1+\nu)^2}{\delta\lambda}\le\frac{\sqrt{2}\pi\varphi(1)\varphi''(1)}{324(1+\nu)^3} δ λ C ( 1 + ν ) 2 ≤ 324 ( 1 + ν ) 3 2 π φ ( 1 ) φ ′′ ( 1 ) . Partition time into consecutive blocks of length
⌈ 1 Φ ( − λ 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) ) log ( 324 ( 1 + ν ) 3 2 π φ ( 1 ) φ ′ ′ ( 1 ) ) ⌉ . \left\lceil\frac{1}{\Phi\left(-\lambda\sqrt{\frac{2(1+e)^2\delta(1+\nu)}{\varphi(1)}}\right)}\log\left(\frac{324(1+\nu)^3}{\sqrt{2}\pi\varphi(1)\varphi''(1)}\right)\right\rceil. Φ ( − λ φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) ) 1 log ( 2 π φ ( 1 ) φ ′′ ( 1 ) 324 ( 1 + ν ) 3 ) .
Then the exponential factor across a full block is at most 2 π φ ( 1 ) φ ′ ′ ( 1 ) 324 ( 1 + ν ) 3 \frac{\sqrt{2}\pi\varphi(1)\varphi''(1)}{324(1+\nu)^3} 324 ( 1 + ν ) 3 2 π φ ( 1 ) φ ′′ ( 1 ) . Applying Eq. (134) to a point attaining the maximum on each block gives two cases. If the maximum on the current block does not exceed the maximum on the previous block, then
max c u r r . b l o c k E [ ( w t − r w ⋆ ) 2 ] ≤ ( 1 − c ( 1 + ν ) 3 ) max p r e v . b l o c k E [ ( w t − r w ⋆ ) 2 ] + C ( 1 + ν ) 2 δ λ + C ( 1 + ν ) 11 δ λ 2 ( 1 + ν 2 + 1 δ ) . \max_{\mathrm{curr.\ block}}\sqrt{\mathbb{E}[(w_t-rw^\star)^2]}\le\left(1-\frac{c}{(1+\nu)^3}\right)\max_{\mathrm{prev.\ block}}\sqrt{\mathbb{E}[(w_t-rw^\star)^2]}+\frac{C(1+\nu)^2}{\delta\lambda}+\frac{C(1+\nu)^{11}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right). curr. block max E [( w t − r w ⋆ ) 2 ] ≤ ( 1 − ( 1 + ν ) 3 c ) prev. block max E [( w t − r w ⋆ ) 2 ] + δ λ C ( 1 + ν ) 2 + δ λ 2 C ( 1 + ν ) 11 ( 1 + ν 2 + δ 1 ) .
If the maximum on the current block exceeds the maximum on the previous block, moving the current-block maximum appearing on the right-hand side of Eq. (134) to the left gives
max c u r r . b l o c k E [ ( w t − r w ⋆ ) 2 ] ≤ 1 6 max p r e v . b l o c k E [ ( w t − r w ⋆ ) 2 ] + C ( 1 + ν ) 3 [ ( 1 + ν ) 2 δ λ + ( 1 + ν ) 11 δ λ 2 ( 1 + ν 2 + 1 δ ) ] . \max_{\mathrm{curr.\ block}}\sqrt{\mathbb{E}[(w_t-rw^\star)^2]}\le\frac{1}{6}\max_{\mathrm{prev.\ block}}\sqrt{\mathbb{E}[(w_t-rw^\star)^2]}+C(1+\nu)^3\left[\frac{(1+\nu)^2}{\delta\lambda}+\frac{(1+\nu)^{11}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right)\right]. curr. block max E [( w t − r w ⋆ ) 2 ] ≤ 6 1 prev. block max E [( w t − r w ⋆ ) 2 ] + C ( 1 + ν ) 3 [ δ λ ( 1 + ν ) 2 + δ λ 2 ( 1 + ν ) 11 ( 1 + ν 2 + δ 1 ) ] .
Consequently, after subtracting C ( 1 + ν ) 5 δ λ + C ( 1 + ν ) 14 δ λ 2 ( 1 + ν 2 + 1 δ ) \frac{C(1+\nu)^5}{\delta\lambda}+\frac{C(1+\nu)^{14}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right) δ λ C ( 1 + ν ) 5 + δ λ 2 C ( 1 + ν ) 14 ( 1 + ν 2 + δ 1 ) , the maxima over consecutive blocks contract by a factor at most 1 − 1 + ν ( 1 + ν ) 3 > 1 6 1-\frac{1+\nu}{(1+\nu)^3}>\frac{1}{6} 1 − ( 1 + ν ) 3 1 + ν > 6 1 . The first block is bounded by C ( 1 + ν ) 4 C(1+\nu)^4 C ( 1 + ν ) 4 under Eq. (126) . Iterating the block inequality therefore gives
E [ ( w t − r w ⋆ ) 2 ] ≤ C ( 1 + ν ) 4 exp ( − c t Φ ( − λ 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) ) ( 1 + ν ) 3 log ( C ( 1 + ν ) ) ) + C ( 1 + ν ) 5 δ λ + C ( 1 + ν ) 14 δ λ 2 ( 1 + ν 2 + 1 δ ) . (135) \sqrt{\mathbb{E}[(w_t-rw^\star)^2]}\le C(1+\nu)^4\exp\left(-\frac{ct\Phi\left(-\lambda\sqrt{\frac{2(1+e)^2\delta(1+\nu)}{\varphi(1)}}\right)}{(1+\nu)^3\log(C(1+\nu))}\right)+\frac{C(1+\nu)^5}{\delta\lambda}+\frac{C(1+\nu)^{14}}{\delta\lambda^2}\left(1+\nu^2+\frac{1}{\delta}\right). \tag*{(135)} E [( w t − r w ⋆ ) 2 ] ≤ C ( 1 + ν ) 4 exp − ( 1 + ν ) 3 log ( C ( 1 + ν )) c t Φ ( − λ φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) ) + δ λ C ( 1 + ν ) 5 + δ λ 2 C ( 1 + ν ) 14 ( 1 + ν 2 + δ 1 ) . (135)
Squaring and adjusting universal constants gives
E [ ( w t − r w ⋆ ) 2 ] ≤ C ( 1 + ν ) 8 exp ( − c t Φ ( − λ 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) ) ( 1 + ν ) 3 log ( C ( 1 + ν ) ) ) + C ( 1 + ν ) 10 δ 2 λ 2 + C ( 1 + ν ) 28 δ 2 λ 4 ( 1 + ν 2 + 1 δ ) 2 . (136) \mathbb{E}[(w_t-rw^\star)^2]\le C(1+\nu)^8\exp\left(-\frac{ct\Phi\left(-\lambda\sqrt{\frac{2(1+e)^2\delta(1+\nu)}{\varphi(1)}}\right)}{(1+\nu)^3\log(C(1+\nu))}\right)+\frac{C(1+\nu)^{10}}{\delta^2\lambda^2}+\frac{C(1+\nu)^{28}}{\delta^2\lambda^4}\left(1+\nu^2+\frac{1}{\delta}\right)^2. \tag*{(136)} E [( w t − r w ⋆ ) 2 ] ≤ C ( 1 + ν ) 8 exp − ( 1 + ν ) 3 log ( C ( 1 + ν )) c t Φ ( − λ φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) ) + δ 2 λ 2 C ( 1 + ν ) 10 + δ 2 λ 4 C ( 1 + ν ) 28 ( 1 + ν 2 + δ 1 ) 2 . (136)
Choose λ 2 = C [ ( 1 + ν ) 7 δ ( 1 + ν 2 + 1 δ ) + ( 1 + ν ) 10 δ 2 ] + φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) log ( 2 T ) \lambda^2=C\left[\frac{(1+\nu)^7}{\delta}\left(1+\nu^2+\frac{1}{\delta}\right)+\frac{(1+\nu)^{10}}{\delta^2}\right]+\frac{\varphi(1)}{2(1+e)^2\delta(1+\nu)}\log(2T) λ 2 = C [ δ ( 1 + ν ) 7 ( 1 + ν 2 + δ 1 ) + δ 2 ( 1 + ν ) 10 ] + 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) log ( 2 T ) at all tasks, where C C C is a sufficiently large universal constant. Then Eq. (126) holds for every T ≥ 1 T\ge1 T ≥ 1 , and λ − 2 ≤ 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) log ( 2 T ) \lambda^{-2}\le\frac{2(1+e)^2\delta(1+\nu)}{\varphi(1)\log(2T)} λ − 2 ≤ φ ( 1 ) l o g ( 2 T ) 2 ( 1 + e ) 2 δ ( 1 + ν ) . Consequently, C ( 1 + ν ) 10 δ 2 λ 2 ≤ C ( 1 + ν ) 11 δ log ( 2 T ) \frac{C(1+\nu)^{10}}{\delta^2\lambda^2}\le\frac{C(1+\nu)^{11}}{\delta\log(2T)} δ 2 λ 2 C ( 1 + ν ) 10 ≤ δ l o g ( 2 T ) C ( 1 + ν ) 11 , and
C ( 1 + ν ) 28 δ 2 λ 4 ( 1 + ν 2 + 1 δ ) 2 ≤ C ( 1 + ν ) 30 log 2 ( 2 T ) ( 1 + ν 2 + 1 δ ) 2 . \frac{C(1+\nu)^{28}}{\delta^2\lambda^4}\left(1+\nu^2+\frac{1}{\delta}\right)^2\le\frac{C(1+\nu)^{30}}{\log^2(2T)}\left(1+\nu^2+\frac{1}{\delta}\right)^2. δ 2 λ 4 C ( 1 + ν ) 28 ( 1 + ν 2 + δ 1 ) 2 ≤ log 2 ( 2 T ) C ( 1 + ν ) 30 ( 1 + ν 2 + δ 1 ) 2 .
Moreover,
( λ 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) ) 2 = log ( 2 T ) + C 2 ( 1 + e ) 2 φ ( 1 ) [ ( 1 + ν ) 8 ( 1 + ν 2 + 1 δ ) + ( 1 + ν ) 11 δ ] . \left(\lambda\sqrt{\frac{2(1+e)^2\delta(1+\nu)}{\varphi(1)}}\right)^2=\log(2T)+C\frac{2(1+e)^2}{\varphi(1)}\left[(1+\nu)^8\left(1+\nu^2+\frac{1}{\delta}\right)+\frac{(1+\nu)^{11}}{\delta}\right]. ( λ φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) ) 2 = log ( 2 T ) + C φ ( 1 ) 2 ( 1 + e ) 2 [ ( 1 + ν ) 8 ( 1 + ν 2 + δ 1 ) + δ ( 1 + ν ) 11 ] .
By the Gaussian Mills lower bound,
Φ ( − λ 2 ( 1 + e ) 2 δ ( 1 + ν ) φ ( 1 ) ) ≥ c exp ( − C [ ( 1 + ν ) 8 ( 1 + ν 2 + 1 δ ) + ( 1 + ν ) 11 δ ] ) log ( 2 T ) + C [ ( 1 + ν ) 8 ( 1 + ν 2 + 1 δ ) + ( 1 + ν ) 11 δ ] T − 1 2 . \Phi\left(-\lambda\sqrt{\frac{2(1+e)^2\delta(1+\nu)}{\varphi(1)}}\right)\ge\frac{c\exp\left(-C\left[(1+\nu)^8\left(1+\nu^2+\frac{1}{\delta}\right)+\frac{(1+\nu)^{11}}{\delta}\right]\right)}{\sqrt{\log(2T)+C\left[(1+\nu)^8\left(1+\nu^2+\frac{1}{\delta}\right)+\frac{(1+\nu)^{11}}{\delta}\right]}}T^{-\frac{1}{2}}. Φ ( − λ φ ( 1 ) 2 ( 1 + e ) 2 δ ( 1 + ν ) ) ≥ log ( 2 T ) + C [ ( 1 + ν ) 8 ( 1 + ν 2 + δ 1 ) + δ ( 1 + ν ) 11 ] c exp ( − C [ ( 1 + ν ) 8 ( 1 + ν 2 + δ 1 ) + δ ( 1 + ν ) 11 ] ) T − 2 1 .
Substituting this into Eq. (136) gives, for every T ≥ 1 T\ge1 T ≥ 1 ,
E [ ( w T − r w ⋆ ) 2 ] ≤ C ( 1 + ν ) 8 exp ( − c exp ( − C [ ( 1 + ν ) 8 ( 1 + ν 2 + 1 δ ) + ( 1 + ν ) 11 δ ] ) T ( 1 + ν ) 3 log ( C ( 1 + ν ) ) log ( 2 T ) + C [ ( 1 + ν ) 8 ( 1 + ν 2 + 1 δ ) + ( 1 + ν ) 11 δ ] ) + C ( 1 + ν ) 11 δ log ( 2 T ) + C ( 1 + ν ) 30 log 2 ( 2 T ) ( 1 + ν 2 + 1 δ ) 2 . \mathbb{E}[(w_T-rw^\star)^2]\le C(1+\nu)^8\exp\left(-\frac{c\exp\left(-C\left[(1+\nu)^8\left(1+\nu^2+\frac{1}{\delta}\right)+\frac{(1+\nu)^{11}}{\delta}\right]\right)\sqrt{T}}{(1+\nu)^3\log(C(1+\nu))\sqrt{\log(2T)+C\left[(1+\nu)^8\left(1+\nu^2+\frac{1}{\delta}\right)+\frac{(1+\nu)^{11}}{\delta}\right]}}\right)+\frac{C(1+\nu)^{11}}{\delta\log(2T)}+\frac{C(1+\nu)^{30}}{\log^2(2T)}\left(1+\nu^2+\frac{1}{\delta}\right)^2. E [( w T − r w ⋆ ) 2 ] ≤ C ( 1 + ν ) 8 exp − ( 1 + ν ) 3 log ( C ( 1 + ν )) log ( 2 T ) + C [ ( 1 + ν ) 8 ( 1 + ν 2 + δ 1 ) + δ ( 1 + ν ) 11 ] c exp ( − C [ ( 1 + ν ) 8 ( 1 + ν 2 + δ 1 ) + δ ( 1 + ν ) 11 ] ) T + δ log ( 2 T ) C ( 1 + ν ) 11 + log 2 ( 2 T ) C ( 1 + ν ) 30 ( 1 + ν 2 + δ 1 ) 2 .
Finally, Proposition 22 closes the proof.
Proof for Lemma 3
Full Lemma 3. There exist universal constants c , C > 0 c, C > 0 c , C > 0 , such that the following holds. If
s ( log ( 1 s ) + log ( 2 T ) ) ≤ c δ ( 1 + ν ) 18 s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)\leq c\frac{\delta}{(1+\nu)^{18}} s ( log ( s 1 ) + log ( 2 T ) ) ≤ c ( 1 + ν ) 18 δ
and we choose the same regularization parameter at every task as λ = 2 δ log ( 1 s ) + log ( 2 T ) \lambda=\frac{2}{\sqrt{\delta}}\sqrt{\log\left(\frac{1}{s}\right)+\log(2T)} λ = δ 2 log ( s 1 ) + log ( 2 T ) , then
L ( w T ) − inf w L ( w ) ≤ C s ( 1 + ν ) 12 δ ( log ( 1 s ) + log ( 2 T ) ) . \mathcal{L}(w_T)-\inf_w\mathcal{L}(w)\leq C\frac{s(1+\nu)^{12}}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right). L ( w T ) − w inf L ( w ) ≤ C δ s ( 1 + ν ) 12 ( log ( s 1 ) + log ( 2 T ) ) .
Proof. The temporal L1 regularization, by Lemma 20, follows
w t = Π [ σ t τ t ( θ t w ⋆ + r t δ Z t − λ ) , σ t τ t ( θ t w ⋆ + r t δ Z t + λ ) ] ( w t − 1 ) . w_t=\Pi_{\left[\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z_t-\lambda\right),\sigma_t\tau_t\left(\theta_t w^\star+\frac{r_t}{\sqrt{\delta}}Z_t+\lambda\right)\right]}(w_{t-1}). w t = Π [ σ t τ t ( θ t w ⋆ + δ r t Z t − λ ) , σ t τ t ( θ t w ⋆ + δ r t Z t + λ ) ] ( w t − 1 ) .
We first control the probability of an update. On { w ⋆ = 0 } \{w^\star=0\} { w ⋆ = 0 } , as long as the coordinate has not moved from zero, an update at task j j j requires ∣ r j δ Z j ∣ > λ \left|\frac{r_j}{\sqrt{\delta}}Z_j\right|>\lambda δ r j Z j > λ . From the fourth scalar equation (see Eq. (44) and derivations above it),
r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] . r_t^2=2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))^2\right]. r t 2 = 2 E [ ( 1 − π ρ ( ν Z 1 )) ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 2 ] .
Since 0 < ϕ ′ ( u ) < 1 0<\phi'(u)<1 0 < ϕ ′ ( u ) < 1 and 2 E [ 1 − π ρ ( ν Z 1 ) ] = 1 2\mathbb{E}[1-\pi_\rho(\nu Z_1)]=1 2 E [ 1 − π ρ ( ν Z 1 )] = 1 , we have 0 < r t ≤ 1 0<r_t\leq1 0 < r t ≤ 1 (see conclusion beneath Eq. (48) ). Then almost surely (unless one of the endpoints in the projection becomes exactly zero),
P ( w t ≠ 0 ∣ w ⋆ = 0 ) = P ( ∃ j : ∣ r j δ Z j ∣ > λ ) ≤ 2 T Φ ( − λ δ ) ≤ T e − δ λ 2 2 = s 2 4 T , λ 2 = 4 δ ( log ( 1 s ) + log ( 2 T ) ) . \mathbb{P}(w_t\neq0\mid w^\star=0)=\mathbb{P}\left(\exists j:\left|\frac{r_j}{\sqrt{\delta}}Z_j\right|>\lambda\right)\leq2T\Phi(-\lambda\sqrt{\delta})\leq Te^{-\frac{\delta\lambda^2}{2}}=\frac{s^2}{4T},
\qquad\lambda^2=\frac{4}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right). P ( w t = 0 ∣ w ⋆ = 0 ) = P ( ∃ j : δ r j Z j > λ ) ≤ 2 T Φ ( − λ δ ) ≤ T e − 2 δ λ 2 = 4 T s 2 , λ 2 = δ 4 ( log ( s 1 ) + log ( 2 T ) ) .
Consequently, because an update can occur either on an active coordinate, whose probability is s s s , or on a null coordinate that is nonzero at time t t t , as { w t ≠ w t − 1 , w ⋆ = 0 } ⊆ { w t ≠ 0 , w ⋆ = 0 } \{w_t\neq w_{t-1},w^\star=0\}\subseteq\{w_t\neq0,w^\star=0\} { w t = w t − 1 , w ⋆ = 0 } ⊆ { w t = 0 , w ⋆ = 0 } almost surely, P ( w t ≠ w t − 1 ) ≤ s + s 2 4 T ≤ 2 s \mathbb{P}(w_t\neq w_{t-1})\leq s+\frac{s^2}{4T}\leq2s P ( w t = w t − 1 ) ≤ s + 4 T s 2 ≤ 2 s . Using γ t σ t τ t = γ t δ P ( w t ≠ w t − 1 ) \frac{\gamma_t}{\sigma_t\tau_t}=\frac{\gamma_t}{\delta}\mathbb{P}(w_t\neq w_{t-1}) σ t τ t γ t = δ γ t P ( w t = w t − 1 ) we obtain uniformly for t ≤ T t\leq T t ≤ T , γ t σ t τ t ≤ 2 s δ \frac{\gamma_t}{\sigma_t\tau_t}\leq\frac{2s}{\delta} σ t τ t γ t ≤ δ 2 s . We next control θ t \theta_t θ t . From Eq. (54) ,
2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = ( 1 − 2 ρ ) h ( ν ) = 16 r h ( ν r ) . 2\mathbb{E}\left[\pi'_\rho(\nu Z_1)\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)\right]=(1-2\rho)h(\nu)\overset{16}{=}rh(\nu r). 2 E [ π ρ ′ ( ν Z 1 ) ϕ ′ ( ν α t Z 1 + σ t Z 2 ) ] = ( 1 − 2 ρ ) h ( ν ) = 16 r h ( ν r ) .
Therefore using the equation for θ t \theta_t θ t (see Eq. (50) ),
0 ≤ r h ( ν r ) − θ t = 2 E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ) ] . 0\leq rh(\nu r)-\theta_t=2\mathbb{E}\left[\pi'_\rho(\nu Z_1)\left(\phi'(\nu\alpha_t Z_1+\sigma_t Z_2)-\phi'(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))\right)\right]. 0 ≤ r h ( ν r ) − θ t = 2 E [ π ρ ′ ( ν Z 1 ) ( ϕ ′ ( ν α t Z 1 + σ t Z 2 ) − ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ) ] .
We will also use the sixth scalar equation. By Eq. (45) ,
1 σ t τ t = 2 E [ ( 1 − π ρ ( ν Z 1 ) ) ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) 1 + γ t ϕ ′ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ] . \frac{1}{\sigma_t\tau_t}=2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\frac{\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}{1+\gamma_t\phi''(\operatorname{Prox}_{\gamma_t\phi}(\nu\alpha_t Z_1+\sigma_t Z_2))}\right]. σ t τ t 1 = 2 E [ ( 1 − π ρ ( ν Z 1 )) 1 + γ t ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ϕ ′′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ] .
The change of variables u = Prox γ t ϕ ( x ) u=\operatorname{Prox}_{\gamma_t\phi}(x) u = Prox γ t ϕ ( x ) which gives x = u + γ t ϕ ′ ( u ) x=u+\gamma_t\phi'(u) x = u + γ t ϕ ′ ( u ) and hence d x = ( 1 + γ t ϕ ′ ′ ( u ) ) d u \mathrm{d}x=(1+\gamma_t\phi''(u))\mathrm{d}u d x = ( 1 + γ t ϕ ′′ ( u )) d u , so,
1 σ t τ t = 2 2 π ( ν 2 α t 2 + σ t 2 ) ∫ R ϕ ′ ′ ( u ) exp ( − ( u + γ t ϕ ′ ( u ) ) 2 2 ( ν 2 α t 2 + σ t 2 ) ) d u ≤ 2 2 π ( ν 2 α t 2 + σ t 2 ) ∫ R ϕ ′ ′ ( u ) d u . \frac{1}{\sigma_t\tau_t}=\frac{2}{\sqrt{2\pi(\nu^2\alpha_t^2+\sigma_t^2)}}\int_{\mathbb{R}}\phi''(u)\exp\left(-\frac{(u+\gamma_t\phi'(u))^2}{2(\nu^2\alpha_t^2+\sigma_t^2)}\right)\mathrm{d}u\leq\frac{2}{\sqrt{2\pi(\nu^2\alpha_t^2+\sigma_t^2)}}\int_{\mathbb{R}}\phi''(u)\mathrm{d}u. σ t τ t 1 = 2 π ( ν 2 α t 2 + σ t 2 ) 2 ∫ R ϕ ′′ ( u ) exp ( − 2 ( ν 2 α t 2 + σ t 2 ) ( u + γ t ϕ ′ ( u ) ) 2 ) d u ≤ 2 π ( ν 2 α t 2 + σ t 2 ) 2 ∫ R ϕ ′′ ( u ) d u .
Note that we integrate over the random variable ν α t Z 1 + σ t Z 2 ∼ N ( 0 , ν 2 α t 2 + σ t 2 ) \nu\alpha_t Z_1+\sigma_t Z_2\sim\mathcal{N}(0,\sqrt{\nu^2\alpha_t^2+\sigma_t^2}) ν α t Z 1 + σ t Z 2 ∼ N ( 0 , ν 2 α t 2 + σ t 2 ) therefore ν 2 α t 2 + σ t 2 σ t τ t ≤ 2 π \frac{\sqrt{\nu^2\alpha_t^2+\sigma_t^2}}{\sigma_t\tau_t}\leq\sqrt{\frac{2}{\pi}} σ t τ t ν 2 α t 2 + σ t 2 ≤ π 2 .
Applying Eq. (51) with q = σ t τ t q=\sigma_t\tau_t q = σ t τ t , q H t = ν α t Z 1 + σ t Z 2 qH_t=\nu\alpha_t Z_1+\sigma_t Z_2 q H t = ν α t Z 1 + σ t Z 2 , η = γ t σ t τ t \eta=\frac{\gamma_t}{\sigma_t\tau_t} η = σ t τ t γ t , and R = ν 2 α t 2 + σ t 2 R=\sqrt{\nu^2\alpha_t^2+\sigma_t^2} R = ν 2 α t 2 + σ t 2 gives,
1 σ t τ t ≥ c ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 + γ t ) (137) \frac{1}{\sigma_t\tau_t}\geq\frac{c}{(1+\nu)\left(1+\sqrt{\nu^2\alpha_t^2+\sigma_t^2}+\gamma_t\right)} \tag*{(137)} σ t τ t 1 ≥ ( 1 + ν ) ( 1 + ν 2 α t 2 + σ t 2 + γ t ) c (137)
for a universal constant c > 0 c > 0 c > 0 . If ν 2 α t 2 + σ t 2 ≤ 1 \sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}} \le1 ν 2 α t 2 + σ t 2 ≤ 1 , combining Eq. (137) with γ t σ t τ t ≤ 2 s δ \frac{\gamma_{t}}{\sigma_{t}\tau_{t}} \le\frac{2s}{\delta} σ t τ t γ t ≤ δ 2 s , we obtain σ t τ t ≤ C ( 1 + ν ) \sigma_{t}\tau_{t} \le C(1+\nu) σ t τ t ≤ C ( 1 + ν ) , provided s δ ≤ c 1 + ν \frac{s}{\delta} \le\frac{c}{1+\nu} δ s ≤ 1 + ν c . Since ϕ ′ \phi' ϕ ′ is 1 4 \frac{1}{4} 4 1 -Lipschitz, ∣ π ϕ ′ ( u ) ∣ = ∣ ( 1 − 2 ρ ) σ ( u ) ( 1 − σ ( u ) ) ∣ ≤ 1 4 |\pi_{\phi}'(u)| = |(1-2\rho)\sigma(u)(1-\sigma(u))| \le\frac{1}{4} ∣ π ϕ ′ ( u ) ∣ = ∣ ( 1 − 2 ρ ) σ ( u ) ( 1 − σ ( u )) ∣ ≤ 4 1 and
∣ ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ∣ = ∣ γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ) ∣ ≤ γ t . |\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}-\operatorname{Prox}_{\gamma_{t}\phi}(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2})|=|\gamma_{t}\phi'(\operatorname{Prox}_{\gamma_{t}\phi}(\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}))| \le\gamma_{t}. ∣ ν α t Z 1 + σ t Z 2 − Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 ) ∣ = ∣ γ t ϕ ′ ( Prox γ t ϕ ( ν α t Z 1 + σ t Z 2 )) ∣ ≤ γ t .
therefore,
r h ( ν r ) − θ t ≤ γ t 8 ≤ C ( 1 + ν ) s δ . rh(\nu r)-\theta_{t} \le\frac{\gamma_{t}}{8} \le C(1+\nu)\frac{s}{\delta}. r h ( ν r ) − θ t ≤ 8 γ t ≤ C ( 1 + ν ) δ s .
If ν 2 α t 2 + σ t 2 ≥ 1 \sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}} \ge1 ν 2 α t 2 + σ t 2 ≥ 1 , then by Eq. (137)
γ t ν 2 α t 2 + σ t 2 = γ t σ t τ t σ t τ t ν 2 α t 2 + σ t 2 ≤ 2 s δ σ t τ t ν 2 α t 2 + σ t 2 , \frac{\gamma_{t}}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}=\frac{\gamma_{t}}{\sigma_{t}\tau_{t}}\frac{\sigma_{t}\tau_{t}}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}\le\frac{2s}{\delta}\frac{\sigma_{t}\tau_{t}}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}, ν 2 α t 2 + σ t 2 γ t = σ t τ t γ t ν 2 α t 2 + σ t 2 σ t τ t ≤ δ 2 s ν 2 α t 2 + σ t 2 σ t τ t ,
again provided s δ ≤ c 1 + ν \frac{s}{\delta}\le\frac{c}{1+\nu} δ s ≤ 1 + ν c (we take such sufficiently small s s s ) so σ t τ t ν 2 α t 2 + σ t 2 ≤ C ( 1 + ν ) \frac{\sigma_{t}\tau_{t}}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}\le C(1+\nu) ν 2 α t 2 + σ t 2 σ t τ t ≤ C ( 1 + ν ) . Moreover let f ( x ) = ϕ ′ ( x ) f(x)=\phi'(x) f ( x ) = ϕ ′ ( x ) and g ( x ) = ϕ ′ ( Prox γ t ϕ ( x ) ) g(x)=\phi'(\operatorname{Prox}_{\gamma_{t}\phi}(x)) g ( x ) = ϕ ′ ( Prox γ t ϕ ( x )) . Since f − 1 ( y ) = ( ϕ ′ − 1 ( y ) ) f^{-1}(y)=(\phi'^{-1}(y)) f − 1 ( y ) = ( ϕ ′ − 1 ( y )) , while the proximal identity x = u + γ t ϕ ′ ( u ) x=u+\gamma_{t}\phi'(u) x = u + γ t ϕ ′ ( u ) gives g − 1 ( y ) = ( ϕ ′ ) − 1 ( y ) + γ t y g^{-1}(y)=(\phi')^{-1}(y)+\gamma_{t}y g − 1 ( y ) = ( ϕ ′ ) − 1 ( y ) + γ t y , we obtain
∫ R [ ϕ ′ ( x ) − ϕ ′ ( Prox γ t ϕ ( x ) ) ] d x = ∫ 0 1 [ g − 1 ( y ) − f − 1 ( y ) ] d y = ∫ 0 1 γ t y d y = γ t 2 . (138) \int_{\mathbb{R}}[\phi'(x)-\phi'(\operatorname{Prox}_{\gamma_{t}\phi}(x))]\,\mathrm{d}x=\int_{0}^{1}[g^{-1}(y)-f^{-1}(y)]\,\mathrm{d}y=\int_{0}^{1}\gamma_{t}y\,\mathrm{d}y=\frac{\gamma_{t}}{2}. \tag*{(138)} ∫ R [ ϕ ′ ( x ) − ϕ ′ ( Prox γ t ϕ ( x ))] d x = ∫ 0 1 [ g − 1 ( y ) − f − 1 ( y )] d y = ∫ 0 1 γ t y d y = 2 γ t . (138)
Conditioning on ν α t Z 1 + σ t Z 2 = x \nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}=x ν α t Z 1 + σ t Z 2 = x and using π ϕ ′ ( u ) ≤ 1 4 \pi_{\phi}'(u)\le\frac{1}{4} π ϕ ′ ( u ) ≤ 4 1 therefore gives
r h ( ν r ) − θ t ≤ C γ t ν 2 α t 2 + σ t 2 ≤ C ( 1 + ν ) s δ . rh(\nu r)-\theta_{t}\le C\frac{\gamma_{t}}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}\le C(1+\nu)\frac{s}{\delta}. r h ( ν r ) − θ t ≤ C ν 2 α t 2 + σ t 2 γ t ≤ C ( 1 + ν ) δ s .
Thus, uniformly in t ≤ T t\le T t ≤ T and uniformly in ρ \rho ρ ,
∣ θ t − r h ( ν r ) ∣ ≤ C ( 1 + ν ) s δ . (139) |\theta_{t}-rh(\nu r)|\le C(1+\nu)\frac{s}{\delta}. \tag*{(139)} ∣ θ t − r h ( ν r ) ∣ ≤ C ( 1 + ν ) δ s . (139)
Also, from Eq. (108) and γ t σ t τ t ≤ 2 s δ \frac{\gamma_{t}}{\sigma_{t}\tau_{t}}\le\frac{2s}{\delta} σ t τ t γ t ≤ δ 2 s ,
∣ 1 σ t τ t − h ( ν 2 α t 2 + σ t 2 ) ∣ ≤ C s δ . (140) \left|\frac{1}{\sigma_{t}\tau_{t}}-h\left(\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}\right)\right|\le C\frac{s}{\delta}. \tag*{(140)} σ t τ t 1 − h ( ν 2 α t 2 + σ t 2 ) ≤ C δ s . (140)
Since the projection output belongs to its defining interval, ∣ w t σ t τ t − θ t w ⋆ − r t δ Z t ∣ ≤ λ \left|\frac{w_{t}}{\sigma_{t}\tau_{t}}-\theta_{t}w^{\star}-\frac{r_{t}}{\sqrt{\delta}}Z_{t}\right|\le\lambda σ t τ t w t − θ t w ⋆ − δ r t Z t ≤ λ . Multiplying by ∣ w ⋆ ∣ |w^{\star}| ∣ w ⋆ ∣ and taking expectations, using E ∣ w ⋆ ∣ = ν 2 s π \mathbb{E}|w^{\star}|=\nu\sqrt{\frac{2s}{\pi}} E ∣ w ⋆ ∣ = ν π 2 s , and E [ w ⋆ w t ] = ν 2 α t \mathbb{E}[w^{\star}w_{t}]=\nu^{2}\alpha_{t} E [ w ⋆ w t ] = ν 2 α t , yields
∣ α t σ t τ t − θ t ∣ ≤ λ ν 2 s π ≤ C ν δ s ( log ( 1 s ) + log ( 2 T ) ) . \left|\frac{\alpha_{t}}{\sigma_{t}\tau_{t}}-\theta_{t}\right|\le\frac{\lambda}{\nu}\sqrt{\frac{2s}{\pi}}\le\frac{C}{\nu\sqrt{\delta}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}. σ t τ t α t − θ t ≤ ν λ π 2 s ≤ ν δ C s ( log ( s 1 ) + log ( 2 T ) ) .
Combining this with Eq. (139) ,
∣ α t σ t τ t − r h ( ν r ) ∣ ≤ C ν δ s ( log ( 1 s ) + log ( 2 T ) ) + C ( 1 + ν ) s δ . (141) \left|\frac{\alpha_{t}}{\sigma_{t}\tau_{t}}-rh(\nu r)\right|\le\frac{C}{\nu\sqrt{\delta}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}+C(1+\nu)\frac{s}{\delta}. \tag*{(141)} σ t τ t α t − r h ( ν r ) ≤ ν δ C s ( log ( s 1 ) + log ( 2 T ) ) + C ( 1 + ν ) δ s . (141)
Moreover, by Eq. (81) ,
ν 2 α t 2 + σ t 2 ( σ t τ t ) 2 − ν 2 ( α t σ t τ t ) 2 = σ t 2 ( σ t τ t ) 2 ≤ E [ ( w t σ t τ t − θ t w ⋆ ) 2 ] . \frac{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}{(\sigma_{t}\tau_{t})^{2}}-\nu^{2}\left(\frac{\alpha_{t}}{\sigma_{t}\tau_{t}}\right)^{2}=\frac{\sigma_{t}^{2}}{(\sigma_{t}\tau_{t})^{2}}\le\mathbb{E}\left[\left(\frac{w_{t}}{\sigma_{t}\tau_{t}}-\theta_{t}w^{\star}\right)^{2}\right]. ( σ t τ t ) 2 ν 2 α t 2 + σ t 2 − ν 2 ( σ t τ t α t ) 2 = ( σ t τ t ) 2 σ t 2 ≤ E [ ( σ t τ t w t − θ t w ⋆ ) 2 ] .
On { w ⋆ ≠ 0 } \{w^{\star}\ne0\} { w ⋆ = 0 } because r t ≤ 1 r_{t}\le1 r t ≤ 1 and the projection output belongs to its defining interval,
E [ ( w t σ t τ t − θ t w ⋆ ) 2 1 { w ⋆ ≠ 0 } ] ≤ s E [ ( λ + ∣ Z t ∣ δ ) 2 ] ≤ C s δ ( log ( 1 s ) + log ( 2 T ) ) . \mathbb{E}\left[\left(\frac{w_{t}}{\sigma_{t}\tau_{t}}-\theta_{t}w^{\star}\right)^{2}\mathbf{1}_{\{w^{\star}\ne0\}}\right]\le s\mathbb{E}\left[\left(\lambda+\frac{|Z_{t}|}{\sqrt{\delta}}\right)^{2}\right]\le C\frac{s}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right). E [ ( σ t τ t w t − θ t w ⋆ ) 2 1 { w ⋆ = 0 } ] ≤ s E [ ( λ + δ ∣ Z t ∣ ) 2 ] ≤ C δ s ( log ( s 1 ) + log ( 2 T ) ) .
On { w ⋆ = 0 } \{w^{\star}=0\} { w ⋆ = 0 } the coordinate is zero except on an event of probability at most σ t 2 4 T \frac{\sigma_{t}^{2}}{4T} 4 T σ t 2 ; using the same tail bound as above shows that the null contribution is smaller than the same right-hand side. Hence
σ t 2 ( σ t τ t ) 2 ≤ C s δ ( log ( 1 s ) + log ( 2 T ) ) . (142) \frac{\sigma_{t}^{2}}{(\sigma_{t}\tau_{t})^{2}}\le C\frac{s}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right). \tag*{(142)} ( σ t τ t ) 2 σ t 2 ≤ C δ s ( log ( s 1 ) + log ( 2 T ) ) . (142)
Consequently, by Eqs. (141) and (142) , and using s δ ( log ( 1 s ) + log ( 2 T ) ) ≤ c ( 1 + ν ) 18 \frac{s}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)\leq\frac{c}{(1+\nu)^{18}} δ s ( log ( s 1 ) + log ( 2 T ) ) ≤ ( 1 + ν ) 18 c , and the reverse triangle inequality
∣ ν 2 α t 2 + σ t 2 σ t 2 τ t 2 − ν r h ( ν r ) ∥ ( ν α t σ t τ t , σ t σ t τ t ) ∥ ∣ ≤ ( ν α t σ t τ t − ν r h ( ν r ) ) 2 + σ t 2 ( σ t τ t ) 2 ≤ O ( s δ ( log ( 1 s ) + log ( 2 T ) ) ) \left|\sqrt{\frac{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}{\sigma_{t}^{2}\tau_{t}^{2}}}-\frac{\nu r h(\nu r)}{\left\Vert\left(\frac{\nu\alpha_{t}}{\sigma_{t}\tau_{t}},\frac{\sigma_{t}}{\sigma_{t}\tau_{t}}\right)\right\Vert}\right|\leq\sqrt{\left(\frac{\nu\alpha_{t}}{\sigma_{t}\tau_{t}}-\nu r h(\nu r)\right)^{2}+\frac{\sigma_{t}^{2}}{(\sigma_{t}\tau_{t})^{2}}}\leq O\left(\sqrt{\frac{s}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}\right) σ t 2 τ t 2 ν 2 α t 2 + σ t 2 − ( σ t τ t ν α t , σ t τ t σ t ) ν r h ( ν r ) ≤ ( σ t τ t ν α t − ν r h ( ν r ) ) 2 + ( σ t τ t ) 2 σ t 2 ≤ O ( δ s ( log ( s 1 ) + log ( 2 T ) ) )
where the constant in O ( ⋅ ) O(\cdot) O ( ⋅ ) is universal, then we have
ν 2 α t 2 + σ t 2 σ t τ t = ν r h ( ν r ) + O ( s δ ( log ( 1 s ) + log ( 2 T ) ) ) , (143) \frac{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}{\sigma_{t}\tau_{t}}=\nu r h(\nu r)+O\left(\sqrt{\frac{s}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}\right), \tag*{(143)} σ t τ t ν 2 α t 2 + σ t 2 = ν r h ( ν r ) + O ( δ s ( log ( s 1 ) + log ( 2 T ) ) ) , (143)
and σ t τ t ≤ C ( 1 + ν ) 3 \sigma_{t}\tau_{t}\leq C(1+\nu)^{3} σ t τ t ≤ C ( 1 + ν ) 3 uniformly for t ≤ T t\leq T t ≤ T . Indeed, 1 2 π − ν h ( ν ) = ∫ R ϕ ′ ′ ( u ) ( 1 − e − u 2 2 ν 2 ) d u ≥ c ( 1 + ν ) 2 \frac{1}{\sqrt{2\pi}}-\nu h(\nu)=\int_{\mathbb{R}}\phi''(u)\left(1-e^{-\frac{u^{2}}{2\nu^{2}}}\right)\,du\geq\frac{c}{(1+\nu)^{2}} 2 π 1 − ν h ( ν ) = ∫ R ϕ ′′ ( u ) ( 1 − e − 2 ν 2 u 2 ) d u ≥ ( 1 + ν ) 2 c . By the sixth scalar equation, ν 2 α t 2 + σ t 2 σ t τ t = 1 2 π ∫ R F t ( x ) e − x 2 2 ( ν 2 α t 2 + σ t 2 ) d d x ϕ ′ ( Prox γ t ϕ ( x ) ) d x \frac{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}{\sigma_{t}\tau_{t}}=\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}F_{t}(x)e^{-\frac{x^{2}}{2(\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2})}}\frac{d}{dx}\phi'(\operatorname{Prox}_{\gamma_{t}\phi}(x))\,dx σ t τ t ν 2 α t 2 + σ t 2 = 2 π 1 ∫ R F t ( x ) e − 2 ( ν 2 α t 2 + σ t 2 ) x 2 d x d ϕ ′ ( Prox γ t ϕ ( x )) d x , and so
∣ ν 2 α t 2 + σ t 2 σ t τ t − 1 2 π ∣ ≤ 1 2 π ∫ R ∣ F t ( x ) − 1 ∣ ∣ d d x ϕ ′ ( Prox γ t ϕ ( x ) ) ∣ d x \left|\frac{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}{\sigma_{t}\tau_{t}}-\frac{1}{\sqrt{2\pi}}\right|\leq\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}|F_{t}(x)-1|\left|\frac{d}{dx}\phi'(\operatorname{Prox}_{\gamma_{t}\phi}(x))\right|dx σ t τ t ν 2 α t 2 + σ t 2 − 2 π 1 ≤ 2 π 1 ∫ R ∣ F t ( x ) − 1∣ d x d ϕ ′ ( Prox γ t ϕ ( x )) d x
+ 1 2 π ∫ R F t ( x ) ∣ d d x ϕ ′ ( Prox γ t ϕ ( x ) ) ∣ ∣ e − x 2 2 ( ν 2 α t 2 + σ t 2 ) − 1 ∣ d x ≤ C ( 1 + ν ) ( 1 + γ t ) ν 2 α t 2 + σ t 2 , \qquad+\frac{1}{\sqrt{2\pi}}\int_{\mathbb{R}}F_{t}(x)\left|\frac{d}{dx}\phi'(\operatorname{Prox}_{\gamma_{t}\phi}(x))\right|\left|e^{-\frac{x^{2}}{2(\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2})}}-1\right|dx\leq C\frac{(1+\nu)(1+\gamma_{t})}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}, + 2 π 1 ∫ R F t ( x ) d x d ϕ ′ ( Prox γ t ϕ ( x )) e − 2 ( ν 2 α t 2 + σ t 2 ) x 2 − 1 d x ≤ C ν 2 α t 2 + σ t 2 ( 1 + ν ) ( 1 + γ t ) ,
where F t ( x ) = E [ 1 − 2 π ρ ( ν Z 1 ) ∣ ν α t Z 1 + σ t Z 2 = x ] F_{t}(x)=\mathbb{E}[1-2\pi\rho(\nu Z_{1})\mid\nu\alpha_{t}Z_{1}+\sigma_{t}Z_{2}=x] F t ( x ) = E [ 1 − 2 π ρ ( ν Z 1 ) ∣ ν α t Z 1 + σ t Z 2 = x ] and ∫ R d d x ϕ ′ ( Prox γ t ϕ ( x ) ) d x = 1 \int_{\mathbb{R}}\frac{d}{dx}\phi'(\operatorname{Prox}_{\gamma_{t}\phi}(x))\,dx=1 ∫ R d x d ϕ ′ ( Prox γ t ϕ ( x )) d x = 1 together with the conditional estimate above and ∣ e − u − 1 ∣ ≤ ∣ u ∣ |e^{-u}-1|\leq|u| ∣ e − u − 1∣ ≤ ∣ u ∣ with
∫ R ∣ x ∣ ∣ d d x ϕ ′ ( Prox γ t ϕ ( x ) ) ∣ d x = ∫ R ∣ u + γ t ϕ ′ ( u ) ∣ ϕ ′ ′ ( u ) d u ≤ ∫ R ∣ u ∣ ϕ ′ ′ ( u ) d u ⏟ = 2 ∫ 0 ∞ 1 1 + e − u d u = 2 log 2 + γ t ∫ R ϕ ′ ( u ) ϕ ′ ′ ( u ) d u ⏟ = 1 2 ∫ R d d u ( ϕ ′ ( u ) ) 2 d u = 1 2 , \int_{\mathbb{R}}|x|\left|\frac{d}{dx}\phi'(\operatorname{Prox}_{\gamma_{t}\phi}(x))\right|dx=\int_{\mathbb{R}}|u+\gamma_{t}\phi'(u)|\phi''(u)\,du\leq\underbrace{\int_{\mathbb{R}}|u|\phi''(u)\,du}_{=2\int_{0}^{\infty}\frac{1}{1+e^{-u}}\,du=2\log2}+\gamma_{t}\underbrace{\int_{\mathbb{R}}\phi'(u)\phi''(u)\,du}_{=\frac{1}{2}\int_{\mathbb{R}}\frac{d}{du}(\phi'(u))^{2}\,du=\frac{1}{2}}, ∫ R ∣ x ∣ d x d ϕ ′ ( Prox γ t ϕ ( x )) d x = ∫ R ∣ u + γ t ϕ ′ ( u ) ∣ ϕ ′′ ( u ) d u ≤ = 2 ∫ 0 ∞ 1 + e − u 1 d u = 2 l o g 2 ∫ R ∣ u ∣ ϕ ′′ ( u ) d u + γ t = 2 1 ∫ R d u d ( ϕ ′ ( u ) ) 2 d u = 2 1 ∫ R ϕ ′ ( u ) ϕ ′′ ( u ) d u ,
because γ t σ t τ t ≤ 2 s δ \frac{\gamma_{t}}{\sigma_{t}\tau_{t}}\leq\frac{2s}{\delta} σ t τ t γ t ≤ δ 2 s , σ t τ t ≤ C ( 1 + ν ) \sigma_{t}\tau_{t}\leq C(1+\nu) σ t τ t ≤ C ( 1 + ν ) thus ( 1 + ν ) ( 1 + γ t ) ν 2 α t 2 + σ t 2 ≤ C 1 + ν ν 2 α t 2 + σ t 2 + C ( 1 + ν ) 2 ν 2 α t 2 + σ t 2 s δ \frac{(1+\nu)(1+\gamma_{t})}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}\leq C\frac{1+\nu}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}+C\frac{(1+\nu)^{2}}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}\frac{s}{\delta} ν 2 α t 2 + σ t 2 ( 1 + ν ) ( 1 + γ t ) ≤ C ν 2 α t 2 + σ t 2 1 + ν + C ν 2 α t 2 + σ t 2 ( 1 + ν ) 2 δ s and if ν 2 α t 2 + σ t 2 ≥ C ( 1 + ν ) 3 \sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}\geq C(1+\nu)^{3} ν 2 α t 2 + σ t 2 ≥ C ( 1 + ν ) 3 ,
ν 2 α t 2 + σ t 2 σ t τ t = 1 2 π + O ( 1 + ν ν 2 α t 2 + σ t 2 + 1 1 + ν s δ ) . \frac{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}{\sigma_{t}\tau_{t}}=\frac{1}{\sqrt{2\pi}}+O\left(\frac{1+\nu}{\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}}+\frac{1}{1+\nu}\frac{s}{\delta}\right). σ t τ t ν 2 α t 2 + σ t 2 = 2 π 1 + O ( ν 2 α t 2 + σ t 2 1 + ν + 1 + ν 1 δ s ) .
For a sufficiently large universal C C C and sufficiently small s δ ( log ( 1 s ) + log ( 2 T ) ) \frac{s}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right) δ s ( log ( s 1 ) + log ( 2 T ) ) , this contradicts Eq. (143) , since
ν r h ( ν r ) = ν ( 1 − 2 ρ ) h ( ν ) ≤ ν h ( ν ) ≤ 1 2 π − c ( 1 + ν ) 2 . \nu r h(\nu r)=\nu(1-2\rho)h(\nu)\leq\nu h(\nu)\leq\frac{1}{\sqrt{2\pi}}-\frac{c}{(1+\nu)^{2}}. ν r h ( ν r ) = ν ( 1 − 2 ρ ) h ( ν ) ≤ ν h ( ν ) ≤ 2 π 1 − ( 1 + ν ) 2 c .
Hence ν 2 α t 2 + σ t 2 ≤ C ( 1 + ν ) 3 \sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}\leq C(1+\nu)^{3} ν 2 α t 2 + σ t 2 ≤ C ( 1 + ν ) 3 . Using Eq. (140) and h ( x ) ≥ c 1 + x h(x)\geq\frac{c}{1+x} h ( x ) ≥ 1 + x c (see Eq. (61) ) we conclude that σ t τ t ≤ C ( 1 + ν ) 3 \sigma_{t}\tau_{t}\leq C(1+\nu)^{3} σ t τ t ≤ C ( 1 + ν ) 3 uniformly for t ≤ T t\leq T t ≤ T , after choosing s δ \frac{s}{\delta} δ s sufficiently small. It follows that
∣ ν 2 α t 2 + σ t 2 − ν r h ( ν r ) σ t τ t ∣ ≤ C ( 1 + ν ) 3 δ s ( log ( 1 s ) + log ( 2 T ) ) . \left|\sqrt{\nu^{2}\alpha_{t}^{2}+\sigma_{t}^{2}}-\nu r h(\nu r)\sigma_{t}\tau_{t}\right|\leq C\frac{(1+\nu)^{3}}{\sqrt{\delta}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}. ν 2 α t 2 + σ t 2 − ν r h ( ν r ) σ t τ t ≤ C δ ( 1 + ν ) 3 s ( log ( s 1 ) + log ( 2 T ) ) .
Using Eq. (140) and since h h h is Lipschitz as sup u ∣ ϕ ′ ′ ′ ( u ) ∣ = 1 6 3 \sup_{u}|\phi'''(u)|=\frac{1}{6\sqrt{3}} sup u ∣ ϕ ′′′ ( u ) ∣ = 6 3 1 , we may write
∣ σ t τ t h ( ν r h ( ν r ) σ t τ t ) − 1 ∣ ≤ C ( 1 + ν ) 6 δ s ( log ( 1 s ) + log ( 2 T ) ) . (144) \left|\sigma_{t}\tau_{t}h(\nu r h(\nu r)\sigma_{t}\tau_{t})-1\right|\leq C\frac{(1+\nu)^{6}}{\sqrt{\delta}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}. \tag*{(144)} ∣ σ t τ t h ( ν r h ( ν r ) σ t τ t ) − 1 ∣ ≤ C δ ( 1 + ν ) 6 s ( log ( s 1 ) + log ( 2 T ) ) . (144)
The function x ↦ x h ( ν r h ( ν r ) x ) x\mapsto xh(\nu r h(\nu r)x) x ↦ x h ( ν r h ( ν r ) x ) is strictly increasing, and by Eq. (59)
d d x [ x h ( ν r h ( ν r ) x ) ] = E [ Z 2 ϕ ′ ′ ( ν r h ( ν r ) x Z ) ] ≥ c ( 1 + ν r h ( ν r ) x ) 3 . (145) \frac{d}{dx}\left[xh\left(\nu r h(\nu r)x\right)\right]=\mathbb{E}\left[Z^{2}\phi''\left(\nu r h(\nu r)xZ\right)\right]\geq\frac{c}{(1+\nu r h(\nu r)x)^{3}}. \tag*{(145)} d x d [ x h ( ν r h ( ν r ) x ) ] = E [ Z 2 ϕ ′′ ( ν r h ( ν r ) x Z ) ] ≥ ( 1 + ν r h ( ν r ) x ) 3 c . (145)
and it equals one at x = 1 h ( ν r ) x = \frac{1}{h(\nu r)} x = h ( ν r ) 1 . On 1 2 h ( ν r ) ≤ x ≤ 2 h ( ν r ) \frac{1}{2h(\nu r)} \le x \le\frac{2}{h(\nu r)} 2 h ( ν r ) 1 ≤ x ≤ h ( ν r ) 2 we have ν r h ( ν r ) x ≤ 2 ν r ≤ 2 ν \nu r h(\nu r)x \le2\nu r \le2\nu ν r h ( ν r ) x ≤ 2 ν r ≤ 2 ν (because r ≤ 1 r \le1 r ≤ 1 see Lemma 16). Integrating (145) from 1 h ( ν r ) \frac{1}{h(\nu r)} h ( ν r ) 1 to either 1 2 h ( ν r ) \frac{1}{2h(\nu r)} 2 h ( ν r ) 1 or 2 h ( ν r ) \frac{2}{h(\nu r)} h ( ν r ) 2 shows that
∣ χ ( ν r h ( ν r ) x ) − 1 ∣ ≥ c ( 1 + ν ) 3 \left| \chi(\nu r h(\nu r)x) - 1 \right| \ge\frac{c}{(1+\nu)^3} ∣ χ ( ν r h ( ν r ) x ) − 1 ∣ ≥ ( 1 + ν ) 3 c
at both endpoints. Therefore (144) , together with s δ ( log ( 1 s ) + log ( 2 T ) ) ≤ c ( 1 + ν ) 18 \frac{s}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right) \le\frac{c}{(1+\nu)^{18}} δ s ( log ( s 1 ) + log ( 2 T ) ) ≤ ( 1 + ν ) 18 c , implies the inequality 1 2 h ( ν r ) ≤ σ T t T ≤ 2 h ( ν r ) \frac{1}{2h(\nu r)} \le\sigma_T t_T \le\frac{2}{h(\nu r)} 2 h ( ν r ) 1 ≤ σ T t T ≤ h ( ν r ) 2 . Since ν r ≤ ν \nu r \le\nu ν r ≤ ν and h ( ν ) ≥ c 1 + ν h(\nu) \ge\frac{c}{1+\nu} h ( ν ) ≥ 1 + ν c (see (61) ), we have in particular
σ T t T ≤ C ( 1 + ν ) . (146) \sigma_T t_T \le C(1+\nu). \tag*{(146)} σ T t T ≤ C ( 1 + ν ) . (146)
Returning to (140) and (136), now using the sharper bound (146) , gives
∣ σ T t T h ( ν r h ( ν r ) σ T t T ) − 1 ∣ ≤ C ( 1 + ν ) 2 δ s ( log ( 1 s ) + log ( 2 T ) ) . (147) \left|\sigma_T t_T h(\nu r h(\nu r)\sigma_T t_T)-1\right| \le C\frac{(1+\nu)^2}{\sqrt{\delta}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}. \tag*{(147)} ∣ σ T t T h ( ν r h ( ν r ) σ T t T ) − 1 ∣ ≤ C δ ( 1 + ν ) 2 s ( log ( s 1 ) + log ( 2 T ) ) . (147)
Applying (145) and the mean-value theorem,
∣ σ T t T − 1 h ( ν r ) ∣ ≤ C ( 1 + ν ) 5 δ s ( log ( 1 s ) + log ( 2 T ) ) . \left|\sigma_T t_T-\frac{1}{h(\nu r)}\right| \le C\frac{(1+\nu)^5}{\sqrt{\delta}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}. σ T t T − h ( ν r ) 1 ≤ C δ ( 1 + ν ) 5 s ( log ( s 1 ) + log ( 2 T ) ) .
Finally, using
σ T t T θ T − r = σ T t T ( θ T − r h ( ν r ) ) + r h ( ν r ) ( σ T t T − 1 h ( ν r ) ) , \sigma_T t_T\theta_T-r=\sigma_T t_T(\theta_T-rh(\nu r))+rh(\nu r)\left(\sigma_T t_T-\frac{1}{h(\nu r)}\right), σ T t T θ T − r = σ T t T ( θ T − r h ( ν r )) + r h ( ν r ) ( σ T t T − h ( ν r ) 1 ) ,
together with (139) and (139), we obtain
∣ σ T t T θ T − r ∣ ≤ C ( 1 + ν ) 5 δ s ( log ( 1 s ) + log ( 2 T ) ) . (148) \left|\sigma_T t_T\theta_T-r\right| \le C\frac{(1+\nu)^5}{\sqrt{\delta}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}. \tag*{(148)} ∣ σ T t T θ T − r ∣ ≤ C δ ( 1 + ν ) 5 s ( log ( s 1 ) + log ( 2 T ) ) . (148)
Now, we split into cases. If the coordinate w 0 w_0 w 0 never updates, then ∣ σ T t T θ T w ⋆ + σ T t T r T δ Z t ∣ ≤ σ T t T λ \left|\sigma_T t_T\theta_T w^\star+\sigma_T t_T\frac{r_T}{\sqrt{\delta}}Z_t\right|\le\sigma_T t_T\lambda σ T t T θ T w ⋆ + σ T t T δ r T Z t ≤ σ T t T λ , for all t t t . Thus w T = w 0 = 0 w_T=w_0=0 w T = w 0 = 0 ,
∣ w T − r w ⋆ ∣ = r ∣ w ⋆ ∣ ≤ ∣ r − σ T t T θ 1 ∣ ∣ w ⋆ ∣ + σ T t T ∣ θ 1 w ⋆ ∣ ≤ C ( 1 + ν ) 5 δ s ( log ( 1 s ) + log ( 2 T ) ) ∣ w ⋆ ∣ + C ( 1 + ν ) ( λ + max j ≤ T ∣ Z j ∣ δ ) . \begin{aligned}
|w_T-rw^\star|=r|w^\star|\le|r-\sigma_T t_T\theta_1||w^\star|+\sigma_T t_T|\theta_1w^\star| \\
&\le C\frac{(1+\nu)^5}{\sqrt{\delta}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}|w^\star|+C(1+\nu)\left(\lambda+\frac{\max_{j\le T}|Z_j|}{\sqrt{\delta}}\right).
\end{aligned} ∣ w T − r w ⋆ ∣ = r ∣ w ⋆ ∣ ≤ ∣ r − σ T t T θ 1 ∣∣ w ⋆ ∣ + σ T t T ∣ θ 1 w ⋆ ∣ ≤ C δ ( 1 + ν ) 5 s ( log ( s 1 ) + log ( 2 T ) ) ∣ w ⋆ ∣ + C ( 1 + ν ) ( λ + δ max j ≤ T ∣ Z j ∣ ) .
Otherwise, let J ≤ T J\le T J ≤ T be the last update. Since an update has been made, w J = w t w_J=w_t w J = w t has to be one of the endpoints in the interval, so w T = w J = σ T t T ( θ T w ⋆ + r J δ Z J ± λ ) w_T=w_J=\sigma_Tt_T\left(\theta_Tw^\star+\frac{r_J}{\sqrt{\delta}}Z_J\pm\lambda\right) w T = w J = σ T t T ( θ T w ⋆ + δ r J Z J ± λ ) , and then
∣ w T − r w ⋆ ∣ ≤ ∣ σ T t T θ T − r ∣ ∣ w ⋆ ∣ + σ T t T r J δ ∣ Z J ∣ + σ T t T λ ≤ C ( 1 + ν ) 5 δ s ( log ( 1 s ) + log ( 2 T ) ) ∣ w ⋆ ∣ + C ( 1 + ν ) ( λ + max j ≤ T ∣ Z j ∣ δ ) . \begin{aligned}
|w_T-rw^\star|\le|\sigma_Tt_T\theta_T-r||w^\star|+\sigma_Tt_T\frac{r_J}{\sqrt{\delta}}|Z_J|+\sigma_Tt_T\lambda\\
&\le C\frac{(1+\nu)^5}{\sqrt{\delta}}\sqrt{s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right)}|w^\star|+C(1+\nu)\left(\lambda+\frac{\max_{j\le T}|Z_j|}{\sqrt{\delta}}\right).
\end{aligned} ∣ w T − r w ⋆ ∣ ≤ ∣ σ T t T θ T − r ∣∣ w ⋆ ∣ + σ T t T δ r J ∣ Z J ∣ + σ T t T λ ≤ C δ ( 1 + ν ) 5 s ( log ( s 1 ) + log ( 2 T ) ) ∣ w ⋆ ∣ + C ( 1 + ν ) ( λ + δ max j ≤ T ∣ Z j ∣ ) .
Because P ( max j ≤ T ∣ Z j ∣ > u ) ≤ 2 T e − u 2 / 2 \mathbb{P}\left(\max_{j\le T}|Z_j|>u\right)\le2Te^{-u^2/2} P ( max j ≤ T ∣ Z j ∣ > u ) ≤ 2 T e − u 2 /2 , we have E [ max j ≤ T Z j 2 ] ≤ 2 log ( 2 T ) + 2 \mathbb{E}[\max_{j\le T}Z_j^2]\le2\log(2T)+2 E [ max j ≤ T Z j 2 ] ≤ 2 log ( 2 T ) + 2 . On the event { w ⋆ ≠ 0 } \{w^\star\ne0\} { w ⋆ = 0 } , recall that P ( w ⋆ ≠ 0 ) = s \mathbb{P}(w^\star\ne0)=s P ( w ⋆ = 0 ) = s and E [ ( w ⋆ ) 2 ] = ν 2 \mathbb{E}[(w^\star)^2]=\nu^2 E [( w ⋆ ) 2 ] = ν 2 . Hence
E [ ( w T − r w ⋆ ) 2 1 { w ⋆ ≠ 0 } ] ≤ C ( 1 + ν ) 12 δ s ( log ( 1 s ) + log ( 2 T ) ) . (149) \mathbb{E}\left[(w_T-rw^\star)^2\mathbf{1}_{\{w^\star\ne0\}}\right]\le C\frac{(1+\nu)^{12}}{\delta}s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right). \tag*{(149)} E [ ( w T − r w ⋆ ) 2 1 { w ⋆ = 0 } ] ≤ C δ ( 1 + ν ) 12 s ( log ( s 1 ) + log ( 2 T ) ) . (149)
Otherwise, if { w ⋆ = 0 } \{w^\star=0\} { w ⋆ = 0 } , we distinguish between coordinates that were already nonzero at time T − 1 T-1 T − 1 and newly activated coordinates. If w T − 1 ≠ 0 w_{T-1}\ne0 w T − 1 = 0 , then the event { w ⋆ = 0 , w T − 1 ≠ 0 } \{w^\star=0,w_{T-1}\ne0\} { w ⋆ = 0 , w T − 1 = 0 } depends only on the previous Gaussian variables and is therefore independent of the fresh Z t Z_t Z t . By r t ≤ 1 r_t\le1 r t ≤ 1 , ∣ w t σ t t T ∣ ≤ λ + ∣ Z t ∣ δ \left|\frac{w_t}{\sigma_t t_T}\right|\le\lambda+\frac{|Z_t|}{\sqrt{\delta}} σ t t T w t ≤ λ + δ ∣ Z t ∣ , and hence
E [ ( w T σ T t T ) 2 1 { w ⋆ = 0 , w T − 1 ≠ 0 } ] ≤ 2 ( 1 − s ) P ( w T − 1 ≠ 0 ∣ w ⋆ = 0 ) ( λ 2 + 1 δ ) ≤ C s 2 T ( λ 2 + 1 δ ) , \mathbb{E}\left[\left(\frac{w_T}{\sigma_Tt_T}\right)^2\mathbf{1}_{\{w^\star=0,w_{T-1}\ne0\}}\right]\le2(1-s)\mathbb{P}(w_{T-1}\ne0\mid w^\star=0)\left(\lambda^2+\frac{1}{\delta}\right)\le C\frac{s^2}{T}\left(\lambda^2+\frac{1}{\delta}\right), E [ ( σ T t T w T ) 2 1 { w ⋆ = 0 , w T − 1 = 0 } ] ≤ 2 ( 1 − s ) P ( w T − 1 = 0 ∣ w ⋆ = 0 ) ( λ 2 + δ 1 ) ≤ C T s 2 ( λ 2 + δ 1 ) ,
where we used
P ( w T − 1 ≠ 0 ∣ w ⋆ = 0 ) ≤ 2 T Φ ( − λ δ ) ≤ s 2 4 T . \mathbb{P}(w_{T-1}\ne0\mid w^\star=0)\le2T\Phi(-\lambda\sqrt{\delta})\le\frac{s^2}{4T}. P ( w T − 1 = 0 ∣ w ⋆ = 0 ) ≤ 2 T Φ ( − λ δ ) ≤ 4 T s 2 .
If instead w T − 1 = 0 w_{T-1}=0 w T − 1 = 0 , then on a null coordinate the temporal soft-thresholding update gives exactly the equality
w T σ T τ T = sign ( Z T ) ( r T δ ∣ Z T ∣ − λ ) + . \frac{w_T}{\sigma_T\tau_T}=\operatorname{sign}(Z_T)\left(\frac{r_T}{\sqrt{\delta}}\lvert Z_T\rvert-\lambda\right)_+. σ T τ T w T = sign ( Z T ) ( δ r T ∣ Z T ∣ − λ ) + .
Therefore
E [ ( w T σ T τ T ) 2 1 { w ⋆ = 0 , w T − 1 = 0 } ] ≤ 2 δ Φ ( − λ δ ) ≤ C s 2 δ T 2 . \mathbb{E}\left[\left(\frac{w_T}{\sigma_T\tau_T}\right)^2\mathbf{1}_{\{w^\star=0,w_{T-1}=0\}}\right]\le\frac{2}{\delta}\Phi(-\lambda\sqrt{\delta})\le C\frac{s^2}{\delta T^2}. E [ ( σ T τ T w T ) 2 1 { w ⋆ = 0 , w T − 1 = 0 } ] ≤ δ 2 Φ ( − λ δ ) ≤ C δ T 2 s 2 .
where we used r t ≤ 1 r_t\le1 r t ≤ 1 (see conclusion beneath Eq. (48) ) and
∫ x ∞ ( z − x ) 2 φ ( z ) d z ≤ ∫ x ∞ z ( z − x ) φ ( z ) d z = ∫ x ∞ z 2 φ ( z ) d z − ∫ x ∞ z x φ ( z ) d z = ( x φ ( x ) + Φ ( − x ) ) − x φ ( − x ) = Φ ( − x ) . \int_x^\infty(z-x)^2\varphi(z)\,dz\le\int_x^\infty z(z-x)\varphi(z)\,dz=\int_x^\infty z^2\varphi(z)\,dz-\int_x^\infty zx\varphi(z)\,dz=(x\varphi(x)+\Phi(-x))-x\varphi(-x)=\Phi(-x). ∫ x ∞ ( z − x ) 2 φ ( z ) d z ≤ ∫ x ∞ z ( z − x ) φ ( z ) d z = ∫ x ∞ z 2 φ ( z ) d z − ∫ x ∞ z x φ ( z ) d z = ( x φ ( x ) + Φ ( − x )) − x φ ( − x ) = Φ ( − x ) .
Combining the two cases and using Eq. (139)},
E [ w T 2 1 { w ⋆ = 0 } ] ≤ C ( 1 + ν ) 2 δ s ( log ( 1 s ) + log ( 2 T ) ) . \mathbb{E}\left[w_T^2\mathbf{1}_{\{w^\star=0\}}\right]\le C\frac{(1+\nu)^2}{\delta}s\left(\log\left(\frac{1}{s}\right)+\log(2T)\right). E [ w T 2 1 { w ⋆ = 0 } ] ≤ C δ ( 1 + ν ) 2 s ( log ( s 1 ) + log ( 2 T ) ) .
where we used s ≤ 1 s\le1 s ≤ 1 and T ≥ 1 T\ge1 T ≥ 1 . Combining this with Eq. (149) , we obtain
E [ ( w T − r w ⋆ ) 2 ] ≤ C s ( 1 + ν ) 12 δ ( log ( 1 s ) + log ( 2 T ) ) . \mathbb{E}\left[(w_T-rw^\star)^2\right]\le C\frac{s(1+\nu)^{12}}{\delta}\left(\log\left(\frac{1}{s}\right)+\log(2T)\right). E [ ( w T − r w ⋆ ) 2 ] ≤ C δ s ( 1 + ν ) 12 ( log ( s 1 ) + log ( 2 T ) ) .
The constant C C C is universal and the bound is uniform over ρ ∈ [ 0 , 1 2 ) \rho\in[0,\frac{1}{2}) ρ ∈ [ 0 , 2 1 ) . Finally, Proposition 22 closes the proof.
Proof of lower bound for temporal L1 when T = 1 T=1 T = 1
Theorem 27. There exist universal constants c , C > 0 c,C>0 c , C > 0 such that, for all s ≤ c δ ν 2 ( 1 − 2 ρ ) 2 ( 1 + ν ) 2 log ( 1 + ν δ ν 2 ( 1 − 2 ρ ) 2 ) s\le c\frac{\delta\nu^2(1-2\rho)^2}{(1+\nu)^2\log\left(\frac{1+\nu}{\delta\nu^2(1-2\rho)^2}\right)} s ≤ c ( 1 + ν ) 2 l o g ( δ ν 2 ( 1 − 2 ρ ) 2 1 + ν ) δ ν 2 ( 1 − 2 ρ ) 2 and λ > 0 \lambda>0 λ > 0 ,
L ( w 1 ) − inf w L ( w ) ≥ C ( 1 + ν ) 6 s δ log ( 1 s ) . \mathcal{L}(\mathbf{w}_1)-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\ge\frac{C}{(1+\nu)^6}\frac{s}{\delta}\log\left(\frac{1}{s}\right). L ( w 1 ) − w inf L ( w ) ≥ ( 1 + ν ) 6 C δ s log ( s 1 ) .
Proof. Since w 0 = 0 w_0=0 w 0 = 0 , the first temporal L1 iterate by Lemma 20 follows,
w 1 = σ 1 τ 1 S λ ( θ 1 w ⋆ + r 1 δ Z ) . w_1=\sigma_1\tau_1S_\lambda\left(\theta_1w^\star+\frac{r_1}{\sqrt{\delta}}Z\right). w 1 = σ 1 τ 1 S λ ( θ 1 w ⋆ + δ r 1 Z ) .
Recall that E [ ( w 1 − r w ⋆ ) 2 ] = ν 2 ( α 1 − r ) 2 + σ 1 2 \mathbb{E}[(w_1-rw^\star)^2]=\nu^2(\alpha_1-r)^2+\sigma_1^2 E [( w 1 − r w ⋆ ) 2 ] = ν 2 ( α 1 − r ) 2 + σ 1 2 by Eq. (40) . We argue by contradiction. Suppose that, for a sufficiently small universal constant c ⋆ > 0 c_\star>0 c ⋆ > 0 ,
ν 2 ( α 1 − r ) 2 + σ 1 2 < c ⋆ s δ log ( 1 s ) . \nu^2(\alpha_1-r)^2+\sigma_1^2<c_\star\frac{s}{\delta}\log\left(\frac{1}{s}\right). ν 2 ( α 1 − r ) 2 + σ 1 2 < c ⋆ δ s log ( s 1 ) .
Since r > 0 r>0 r > 0 (see Lemma 16) for every fixed ρ < 1 2 \rho<\frac{1}{2} ρ < 2 1 and s log ( 1 s ) → 0 s\log\left(\frac{1}{s}\right)\to0 s log ( s 1 ) → 0 , by decreasing s 0 s_0 s 0 if necessary, r 2 ≤ α 1 ≤ 3 r 2 \frac{r}{2}\le\alpha_1\le\frac{3r}{2} 2 r ≤ α 1 ≤ 2 3 r . We first note that ( σ 1 τ 1 ) r 1 ≥ 1 (\sigma_1\tau_1)r_1\ge1 ( σ 1 τ 1 ) r 1 ≥ 1 . Indeed, the sixth scalar equation with E [ 1 − 2 π ρ ( ν Z 1 ) ] = 0 \mathbb{E}[1-2\pi_\rho(\nu Z_1)]=0 E [ 1 − 2 π ρ ( ν Z 1 )] = 0 gives,
1 σ 1 τ 1 = E [ 2 ( 1 − π ρ ( ν Z 1 ) ) φ ′ ′ ( Prox γ 1 φ ( ν α 1 Z 1 + σ 1 Z 2 ) ) 1 + γ 1 φ ′ ′ ( Prox γ 1 φ ( ν α 1 Z 1 + σ 1 Z 2 ) ) ] . \frac{1}{\sigma_1\tau_1}=\mathbb{E}\left[\frac{2(1-\pi_\rho(\nu Z_1))\varphi''\left(\operatorname{Prox}_{\gamma_1\varphi}(\nu\alpha_1Z_1+\sigma_1Z_2)\right)}{1+\gamma_1\varphi''\left(\operatorname{Prox}_{\gamma_1\varphi}(\nu\alpha_1Z_1+\sigma_1Z_2)\right)}\right]. σ 1 τ 1 1 = E [ 1 + γ 1 φ ′′ ( Prox γ 1 φ ( ν α 1 Z 1 + σ 1 Z 2 ) ) 2 ( 1 − π ρ ( ν Z 1 )) φ ′′ ( Prox γ 1 φ ( ν α 1 Z 1 + σ 1 Z 2 ) ) ] .
Since φ ′ ′ ( u ) ≤ φ ′ ( u ) \varphi''(u)\le\varphi'(u) φ ′′ ( u ) ≤ φ ′ ( u ) and φ ′ ′ ( u ) ≥ 0 \varphi''(u)\ge0 φ ′′ ( u ) ≥ 0 ,
1 σ 1 τ 1 ≤ E [ 2 ( 1 − π ρ ( ν Z 1 ) ) φ ′ ( Prox γ 1 φ ( ν α 1 Z 1 + σ 1 Z 2 ) ) ] . \frac{1}{\sigma_1\tau_1}\le\mathbb{E}\left[2(1-\pi_\rho(\nu Z_1))\varphi'\left(\operatorname{Prox}_{\gamma_1\varphi}(\nu\alpha_1Z_1+\sigma_1Z_2)\right)\right]. σ 1 τ 1 1 ≤ E [ 2 ( 1 − π ρ ( ν Z 1 )) φ ′ ( Prox γ 1 φ ( ν α 1 Z 1 + σ 1 Z 2 ) ) ] .
By Cauchy–Schwarz and 2 E [ 1 − π ρ ( ν Z 1 ) ] = 1 2\mathbb{E}[1-\pi_\rho(\nu Z_1)]=1 2 E [ 1 − π ρ ( ν Z 1 )] = 1 and the proximal optimality condition (Eq. (65) ), using the fourth scalar equation, gives,
1 σ 1 τ 1 ≤ 2 E [ ( 1 − π ρ ( ν Z 1 ) ) φ ′ ( Prox γ 1 φ ( ν α 1 Z 1 + σ 1 Z 2 ) ) 2 ] = r 1 . \frac{1}{\sigma_1\tau_1}\le\sqrt{2\mathbb{E}\left[(1-\pi_\rho(\nu Z_1))\varphi'\left(\operatorname{Prox}_{\gamma_1\varphi}(\nu\alpha_1Z_1+\sigma_1Z_2)\right)^2\right]}=r_1. σ 1 τ 1 1 ≤ 2 E [ ( 1 − π ρ ( ν Z 1 )) φ ′ ( Prox γ 1 φ ( ν α 1 Z 1 + σ 1 Z 2 ) ) 2 ] = r 1 .
Hence ( σ 1 τ 1 ) r 1 ≥ 1 (\sigma_1\tau_1)r_1 \ge1 ( σ 1 τ 1 ) r 1 ≥ 1 . We next show that the threshold must be large on null coordinates. Let Z ∼ N ( 0 , 1 ) Z \sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) be independent of w ⋆ w^\star w ⋆ , E [ Z ( w 1 − r w ⋆ ) ] = E [ Z w 1 ] \mathbb{E}[Z(w_1-rw^\star)]=\mathbb{E}[Zw_1] E [ Z ( w 1 − r w ⋆ )] = E [ Z w 1 ] . Stein’s Lemma gives
E [ Z w 1 ] = σ 1 τ 1 r 1 δ P ( ∣ θ 1 w ⋆ + r 1 δ Z ∣ > λ ) . \mathbb{E}[Zw_1]=\frac{\sigma_1\tau_1r_1}{\sqrt{\delta}}\mathbb{P}\left(\left|\theta_1w^\star+\frac{r_1}{\sqrt{\delta}}Z\right|>\lambda\right). E [ Z w 1 ] = δ σ 1 τ 1 r 1 P ( θ 1 w ⋆ + δ r 1 Z > λ ) .
Restricting to { w ⋆ = 0 } \{w^\star=0\} { w ⋆ = 0 } and using ( σ 1 τ 1 ) r 1 ≥ 1 (\sigma_1\tau_1)r_1 \ge1 ( σ 1 τ 1 ) r 1 ≥ 1 ,
E [ Z w 1 ] ≥ 2 ( 1 − s ) δ Φ ( − λ δ r 1 ) . \mathbb{E}[Zw_1]\ge\frac{2(1-s)}{\sqrt{\delta}}\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_1}\right). E [ Z w 1 ] ≥ δ 2 ( 1 − s ) Φ ( − r 1 λ δ ) .
Therefore, by Cauchy–Schwarz,
ν 2 ( α 1 − r ) 2 + σ 1 2 = E [ ( w 1 − r w ⋆ ) 2 ] ≥ E [ Z ( w 1 − r w ⋆ ) ] 2 ≥ 4 ( 1 − s ) 2 δ Φ ( − λ δ r 1 ) 2 . \nu^2(\alpha_1-r)^2+\sigma_1^2=\mathbb{E}[(w_1-rw^\star)^2]\ge\mathbb{E}[Z(w_1-rw^\star)]^2\ge\frac{4(1-s)^2}{\delta}\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_1}\right)^2. ν 2 ( α 1 − r ) 2 + σ 1 2 = E [( w 1 − r w ⋆ ) 2 ] ≥ E [ Z ( w 1 − r w ⋆ ) ] 2 ≥ δ 4 ( 1 − s ) 2 Φ ( − r 1 λ δ ) 2 .
Thus, for s ≤ 1 2 s\le\frac{1}{2} s ≤ 2 1 , E [ ( w 1 − r w ⋆ ) 2 ] ≥ 1 δ Φ ( − λ δ r 1 ) 2 \mathbb{E}[(w_1-rw^\star)^2]\ge\frac{1}{\delta}\Phi\left(-\frac{\lambda\sqrt{\delta}}{r_1}\right)^2 E [( w 1 − r w ⋆ ) 2 ] ≥ δ 1 Φ ( − r 1 λ δ ) 2 . If λ 2 δ r 1 2 ≤ 1 2 log ( 1 s ) \frac{\lambda^2\delta}{r_1^2}\le\frac{1}{2}\log\left(\frac{1}{s}\right) r 1 2 λ 2 δ ≤ 2 1 log ( s 1 ) , then the lower Mills bound Φ ( − x ) ≥ c e − x 2 / 2 1 + x 2 \Phi(-x)\ge c\frac{e^{-x^2/2}}{1+x^2} Φ ( − x ) ≥ c 1 + x 2 e − x 2 /2 gives Φ ( − λ δ r 1 ) 2 ≥ c s 1 + log ( 1 s ) \Phi\left(-\frac{\lambda\sqrt{\delta}}{r_1}\right)^2\ge c\frac{\sqrt{s}}{1+\log\left(\frac{1}{s}\right)} Φ ( − r 1 λ δ ) 2 ≥ c 1 + l o g ( s 1 ) s . Since s 1 + log ( 1 s ) ≫ s log ( 1 s ) \frac{\sqrt{s}}{1+\log\left(\frac{1}{s}\right)}\gg s\log\left(\frac{1}{s}\right) 1 + l o g ( s 1 ) s ≫ s log ( s 1 ) , this contradicts the assumed upper bound for a sufficiently small s s s . Consequently, λ 2 δ r 1 2 ≥ 1 2 log ( 1 s ) \frac{\lambda^2\delta}{r_1^2}\ge\frac{1}{2}\log\left(\frac{1}{s}\right) r 1 2 λ 2 δ ≥ 2 1 log ( s 1 ) .
We now use the active coordinates. Conditionally on { w ⋆ ≠ 0 } \{w^\star\ne0\} { w ⋆ = 0 } , write w ⋆ = ν s G w^\star=\frac{\nu}{\sqrt{s}}G w ⋆ = s ν G , with G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) independent of Z Z Z . Then
θ 1 w ⋆ + r 1 δ Z = θ 1 ν s G + r 1 δ Z . \theta_1w^\star+\frac{r_1}{\sqrt{\delta}}Z=\frac{\theta_1\nu}{\sqrt{s}}G+\frac{r_1}{\sqrt{\delta}}Z. θ 1 w ⋆ + δ r 1 Z = s θ 1 ν G + δ r 1 Z .
Let p = 2 Φ ( − λ θ 1 2 ν 2 s + r 1 2 δ ) p=2\Phi\left(-\frac{\lambda}{\sqrt{\frac{\theta_1^2\nu^2}{s}+\frac{r_1^2}{\delta}}}\right) p = 2Φ − s θ 1 2 ν 2 + δ r 1 2 λ . Stein’s Lemma gives the exact identity α 1 = ( σ 1 τ 1 ) θ 1 p \alpha_1=(\sigma_1\tau_1)\theta_1p α 1 = ( σ 1 τ 1 ) θ 1 p . Indeed,
ν 2 α 1 = E [ w ⋆ w 1 ] = s ν s ( σ 1 τ 1 ) E [ G S λ ( θ 1 ν s G + r 1 δ Z ) ] = s ν s ( σ 1 τ 1 ) E [ S λ ′ ( θ 1 ν s G + r 1 δ Z ) ] = s ν s ( σ 1 τ 1 ) E [ 1 { ∣ θ 1 ν s G + r 1 δ Z ∣ > λ } ] = ( σ 1 τ 1 ) θ 1 ν 2 p . \begin{aligned}
\nu^2\alpha_1=\mathbb{E}[w^\star w_1]=s\frac{\nu}{\sqrt{s}}(\sigma_1\tau_1)\mathbb{E}\left[G S_\lambda\left(\frac{\theta_1\nu}{\sqrt{s}}G+\frac{r_1}{\sqrt{\delta}}Z\right)\right]=s\frac{\nu}{\sqrt{s}}(\sigma_1\tau_1)\mathbb{E}\left[S'_\lambda\left(\frac{\theta_1\nu}{\sqrt{s}}G+\frac{r_1}{\sqrt{\delta}}Z\right)\right] \\
&=s\frac{\nu}{\sqrt{s}}(\sigma_1\tau_1)\mathbb{E}\left[\mathbf{1}_{\left\{\left|\frac{\theta_1\nu}{\sqrt{s}}G+\frac{r_1}{\sqrt{\delta}}Z\right|>\lambda\right\}}\right]=(\sigma_1\tau_1)\theta_1\nu^2p.
\end{aligned} ν 2 α 1 = E [ w ⋆ w 1 ] = s s ν ( σ 1 τ 1 ) E [ G S λ ( s θ 1 ν G + δ r 1 Z ) ] = s s ν ( σ 1 τ 1 ) E [ S λ ′ ( s θ 1 ν G + δ r 1 Z ) ] = s s ν ( σ 1 τ 1 ) E [ 1 { s θ 1 ν G + δ r 1 Z > λ } ] = ( σ 1 τ 1 ) θ 1 ν 2 p .
We next lower bound the residual on active coordinates. Put y = λ θ 1 2 ν 2 s + r 1 2 δ y=\frac{\lambda}{\sqrt{\frac{\theta_1^2\nu^2}{s}+\frac{r_1^2}{\delta}}} y = s θ 1 2 ν 2 + δ r 1 2 λ , and let U ∼ N ( 0 , 1 ) U\sim\mathcal{N}(0,1) U ∼ N ( 0 , 1 ) . Since
S λ ( θ 1 ν s G + r 1 δ Z ) = d θ 1 2 ν 2 s + r 1 2 δ s y ( U ) , S_\lambda\left(\frac{\theta_1\nu}{\sqrt{s}}G+\frac{r_1}{\sqrt{\delta}}Z\right)\overset{d}{=}\sqrt{\frac{\theta_1^2\nu^2}{s}+\frac{r_1^2}{\delta}}s_y(U), S λ ( s θ 1 ν G + δ r 1 Z ) = d s θ 1 2 ν 2 + δ r 1 2 s y ( U ) ,
we obtain
σ 1 2 = E [ ( w 1 − α 1 w ⋆ ) 2 ] ≥ s E [ ( σ 1 τ 1 S λ ( θ 1 ν s G + r 1 δ Z ) − ν α 1 s G ) 2 | w ⋆ ≠ 0 ] ≥ α 1 2 ν 2 ( E [ s y ( U ) 2 ] p 2 − 1 ) . \sigma_1^2=\mathbb{E}[(w_1-\alpha_1w^\star)^2]\ge s\mathbb{E}\left[\left(\sigma_1\tau_1S_\lambda\left(\frac{\theta_1\nu}{\sqrt{s}}G+\frac{r_1}{\sqrt{\delta}}Z\right)-\frac{\nu\alpha_1}{\sqrt{s}}G\right)^2\middle|w^\star\ne0\right]\ge\alpha_1^2\nu^2\left(\frac{\mathbb{E}[s_y(U)^2]}{p^2}-1\right). σ 1 2 = E [( w 1 − α 1 w ⋆ ) 2 ] ≥ s E [ ( σ 1 τ 1 S λ ( s θ 1 ν G + δ r 1 Z ) − s ν α 1 G ) 2 w ⋆ = 0 ] ≥ α 1 2 ν 2 ( p 2 E [ s y ( U ) 2 ] − 1 ) .
Here we used p = P ( ∣ U ∣ > y ) = E [ U s y ( U ) ] p=\mathbb{P}(|U|>y)=\mathbb{E}[Us_y(U)] p = P ( ∣ U ∣ > y ) = E [ U s y ( U )] by Stein’s Lemma, and
( σ 1 τ 1 ) 2 ( θ 1 2 ν 2 s + r 1 2 δ ) ≥ ( σ 1 τ 1 ) 2 θ 1 2 ν 2 s = α 1 2 ν 2 s p 2 , (\sigma_1\tau_1)^2\left(\frac{\theta_1^2\nu^2}{s}+\frac{r_1^2}{\delta}\right)\ge(\sigma_1\tau_1)^2\frac{\theta_1^2\nu^2}{s}=\frac{\alpha_1^2\nu^2}{sp^2}, ( σ 1 τ 1 ) 2 ( s θ 1 2 ν 2 + δ r 1 2 ) ≥ ( σ 1 τ 1 ) 2 s θ 1 2 ν 2 = s p 2 α 1 2 ν 2 ,
from α 1 = ( σ 1 τ 1 ) θ 1 p \alpha_1=(\sigma_1\tau_1)\theta_1p α 1 = ( σ 1 τ 1 ) θ 1 p by the first scalar equation. For y ≥ 1 y\ge1 y ≥ 1 ,
E [ s y ( U ) 2 ] p 2 − 1 = E [ ( s y ( U ) − p U ) 2 ] p 2 ≥ E [ U 2 1 { ∣ U ∣ ≤ 1 } ] > 0. \frac{\mathbb{E}[s_y(U)^2]}{p^2}-1=\frac{\mathbb{E}[(s_y(U)-pU)^2]}{p^2}\ge\mathbb{E}[U^2\mathbf{1}_{\{|U|\le1\}}]>0. p 2 E [ s y ( U ) 2 ] − 1 = p 2 E [( s y ( U ) − p U ) 2 ] ≥ E [ U 2 1 { ∣ U ∣ ≤ 1 } ] > 0.
Hence y ≥ 1 y \ge1 y ≥ 1 would imply σ 1 2 ≥ c α 1 2 ν 2 ≥ c r 2 ν 2 \sigma_1^2 \ge c\alpha_1^2\nu^2 \ge cr^2\nu^2 σ 1 2 ≥ c α 1 2 ν 2 ≥ c r 2 ν 2 , contradicting σ 1 2 → 0 \sigma_1^2 \to0 σ 1 2 → 0 . For 0 ≤ y ≤ 1 0 \le y \le1 0 ≤ y ≤ 1 ,
E [ S y ( U ) 2 ] − p 2 = E [ ( S y ( U ) − p U ) 2 ] . \mathbb{E}[S_y(U)^2]-p^2=\mathbb{E}[(S_y(U)-pU)^2]. E [ S y ( U ) 2 ] − p 2 = E [( S y ( U ) − p U ) 2 ] .
Moreover, writing 1 − p = P ( ∣ U ∣ ≤ y ) = 2 ∫ 0 y φ ( u ) d u ≤ 2 φ ( 0 ) y ≤ 2 π y 1-p=\mathbb{P}(|U|\le y)=2\int_0^y\varphi(u)\,\mathrm{d}u\le2\varphi(0)y\le\sqrt{\frac{2}{\pi}}y 1 − p = P ( ∣ U ∣ ≤ y ) = 2 ∫ 0 y φ ( u ) d u ≤ 2 φ ( 0 ) y ≤ π 2 y , by Mill’s ratio bound. On the event 1 ≤ U ≤ 9 8 1\le U\le\frac{9}{8} 1 ≤ U ≤ 8 9 ,
∣ S y ( U ) − p U ∣ = ∣ U − y − p U ∣ = ∣ y − ( 1 − p ) U ∣ ≥ ( 1 − 9 8 2 π ) y . |S_y(U)-pU|=|U-y-pU|=|y-(1-p)U|\ge\left(1-\frac{9}{8}\sqrt{\frac{2}{\pi}}\right)y. ∣ S y ( U ) − p U ∣ = ∣ U − y − p U ∣ = ∣ y − ( 1 − p ) U ∣ ≥ ( 1 − 8 9 π 2 ) y .
Therefore there exists a universal constant c > 0 c>0 c > 0 such that E [ S y ( U ) 2 ] p 2 − 1 ≥ c y 2 \frac{\mathbb{E}[S_y(U)^2]}{p^2}-1\ge cy^2 p 2 E [ S y ( U ) 2 ] − 1 ≥ c y 2 , and hence σ 1 2 ≥ c α 1 2 ν 2 y 2 ≥ c r 2 ν 2 y 2 \sigma_1^2\ge c\alpha_1^2\nu^2y^2\ge cr^2\nu^2y^2 σ 1 2 ≥ c α 1 2 ν 2 y 2 ≥ c r 2 ν 2 y 2 , where we use s s s sufficiently small so that α 1 ≥ r 2 \alpha_1\ge\frac{r}{2} α 1 ≥ 2 r . It remains to lower bound y y y . Using α 1 = ( σ 1 τ 1 ) θ 1 p \alpha_1=(\sigma_1\tau_1)^{\theta_1}p α 1 = ( σ 1 τ 1 ) θ 1 p ,
( σ 1 τ 1 ) θ 1 ν s = α 1 ν p s ≤ C r ν s , (\sigma_1\tau_1)\frac{\theta_1\nu}{\sqrt{s}}=\frac{\alpha_1\nu}{p\sqrt{s}}\le C\frac{r\nu}{\sqrt{s}}, ( σ 1 τ 1 ) s θ 1 ν = p s α 1 ν ≤ C s r ν ,
where we used α 1 ≤ 3 r 2 \alpha_1\le\frac{3r}{2} α 1 ≤ 2 3 r for a sufficiently small s s s and p ≥ 2 Φ ( − 1 ) p\ge2\Phi(-1) p ≥ 2Φ ( − 1 ) . Also, ( σ 1 τ 1 ) r 1 δ ≥ 1 δ (\sigma_1\tau_1)\frac{r_1}{\sqrt{\delta}}\ge\frac{1}{\sqrt{\delta}} ( σ 1 τ 1 ) δ r 1 ≥ δ 1 . Since u ↦ u 2 a 2 + u 2 u\mapsto\frac{u^2}{a^2+u^2} u ↦ a 2 + u 2 u 2 is increasing for u > 0 u>0 u > 0 ,
y 2 = ( σ 1 τ 1 ) 2 λ 2 ( σ 1 τ 1 ) 2 θ 1 2 ν 2 s + ( σ 1 τ 1 ) 2 r 1 2 δ = λ 2 δ r 1 2 ( ( σ 1 τ 1 ) r 1 δ ) 2 ( σ 1 τ 1 ) 2 θ 1 2 ν 2 s + ( ( σ 1 τ 1 ) r 1 δ ) 2 ≥ 1 δ λ 2 δ r 1 2 C r 2 ν 2 s + 1 δ ≥ c s δ r 2 ν 2 λ 2 δ r 1 2 , y^2=\frac{(\sigma_1\tau_1)^2\lambda^2}{(\sigma_1\tau_1)^2\frac{\theta_1^2\nu^2}{s}+(\sigma_1\tau_1)^2\frac{r_1^2}{\delta}}
=\frac{\lambda^2\delta}{r_1^2}\frac{\left((\sigma_1\tau_1)\frac{r_1}{\sqrt{\delta}}\right)^2}{(\sigma_1\tau_1)^2\frac{\theta_1^2\nu^2}{s}+\left((\sigma_1\tau_1)\frac{r_1}{\sqrt{\delta}}\right)^2}
\ge\frac{\frac{1}{\delta}\frac{\lambda^2\delta}{r_1^2}}{C\frac{r^2\nu^2}{s}+\frac{1}{\delta}}
\ge c\frac{s}{\delta r^2\nu^2}\frac{\lambda^2\delta}{r_1^2}, y 2 = ( σ 1 τ 1 ) 2 s θ 1 2 ν 2 + ( σ 1 τ 1 ) 2 δ r 1 2 ( σ 1 τ 1 ) 2 λ 2 = r 1 2 λ 2 δ ( σ 1 τ 1 ) 2 s θ 1 2 ν 2 + ( ( σ 1 τ 1 ) δ r 1 ) 2 ( ( σ 1 τ 1 ) δ r 1 ) 2 ≥ C s r 2 ν 2 + δ 1 δ 1 r 1 2 λ 2 δ ≥ c δ r 2 ν 2 s r 1 2 λ 2 δ ,
where in the last step we decreased s 0 s_0 s 0 so that s ≤ δ r 2 ν 2 s\le\delta r^2\nu^2 s ≤ δ r 2 ν 2 . Using the threshold lower bound obtained above,
y 2 ≥ c δ r 2 ν 2 log ( 1 s ) . y^2\ge\frac{c}{\delta r^2\nu^2}\log\left(\frac{1}{s}\right). y 2 ≥ δ r 2 ν 2 c log ( s 1 ) .
Consequently,
ν 2 ( α 1 − r ) 2 + σ 1 2 ≥ σ 1 2 ≥ c r 2 ν 2 y 2 ≥ c δ s log ( 1 s ) , \nu^2(\alpha_1-r)^2+\sigma_1^2\ge\sigma_1^2\ge cr^2\nu^2y^2\ge\frac{c}{\delta}s\log\left(\frac{1}{s}\right), ν 2 ( α 1 − r ) 2 + σ 1 2 ≥ σ 1 2 ≥ c r 2 ν 2 y 2 ≥ δ c s log ( s 1 ) ,
contradicting the assumption for c ⋆ > 0 c_\star>0 c ⋆ > 0 sufficiently small.
For some universal constants C 1 , C 2 , C 3 > 0 C_1,C_2,C_3>0 C 1 , C 2 , C 3 > 0 , a sufficiently small s < s 0 s<s_0 s < s 0 would need to make
r 2 ≤ α 1 ≤ 3 r 2 ⟹ s log ( 1 s ) ≤ C 1 δ ν 2 r 2 \frac{r}{2}\le\alpha_1\le\frac{3r}{2}\quad\Longrightarrow\quad s\log\left(\frac{1}{s}\right)\le C_1\delta\nu^2r^2 2 r ≤ α 1 ≤ 2 3 r ⟹ s log ( s 1 ) ≤ C 1 δ ν 2 r 2
s ≤ C 2 δ r 2 ν 2 . s\le C_2\delta r^2\nu^2. s ≤ C 2 δ r 2 ν 2 .
By Lemma 16, r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) rh(\nu r)=(1-2\rho)h(\nu) r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) therefore since h ( ν r ) ≤ h ( 0 ) = 1 4 h(\nu r)\le h(0)=\frac{1}{4} h ( ν r ) ≤ h ( 0 ) = 4 1 so r ≥ 4 ( 1 − 2 ρ ) h ( ν ) r\ge4(1-2\rho)h(\nu) r ≥ 4 ( 1 − 2 ρ ) h ( ν ) , by solving s 0 log ( 1 s 0 ) ≍ δ ν 2 ( 1 − 2 ρ ) 2 h ( ν ) 2 s_0\log\left(\frac{1}{s_0}\right)\asymp\delta\nu^2(1-2\rho)^2h(\nu)^2 s 0 log ( s 0 1 ) ≍ δ ν 2 ( 1 − 2 ρ ) 2 h ( ν ) 2 we get, that there exists C > 0 C>0 C > 0 so that, s 0 ≤ C δ ν 2 ( 1 − 2 ρ ) 2 ( 1 + ν ) 2 log + ( 1 + ν δ ν 2 ( 1 − 2 ρ ) 2 ) s_0\le C\frac{\delta\nu^2(1-2\rho)^2}{(1+\nu)^2\log_+\left(\frac{1+\nu}{\delta\nu^2(1-2\rho)^2}\right)} s 0 ≤ C ( 1 + ν ) 2 l o g + ( δ ν 2 ( 1 − 2 ρ ) 2 1 + ν ) δ ν 2 ( 1 − 2 ρ ) 2 , as
h ( ν ) = 1 ν ∫ R φ ′ ′ ( u ) φ ( u ν ) d u ≤ min { 1 4 , 1 2 π ν } ≤ C 3 1 1 + ν . h(\nu)=\frac{1}{\nu}\int_{\mathbb{R}}\varphi''(u)\varphi\left(\frac{u}{\nu}\right)\,\mathrm{d}u\le\min\left\{\frac{1}{4},\frac{1}{\sqrt{2\pi}\nu}\right\}\le C_3\frac{1}{1+\nu}. h ( ν ) = ν 1 ∫ R φ ′′ ( u ) φ ( ν u ) d u ≤ min { 4 1 , 2 π ν 1 } ≤ C 3 1 + ν 1 .
Combining with Proposition 22 we get
L 1 − inf w L ( w ) ≥ C 1 ( 1 + ν ) 6 s δ log ( 1 s ) , \mathcal{L}_1-\inf_{\mathbf{w}}\mathcal{L}(\mathbf{w})\ge C\frac{1}{(1+\nu)^6}\frac{s}{\delta}\log\left(\frac{1}{s}\right), L 1 − w inf L ( w ) ≥ C ( 1 + ν ) 6 1 δ s log ( s 1 ) ,
for some universal constant C > 0 C>0 C > 0 .
Proof of prop:deep-equivalence
Recall Proposition 5. Denote by w ( θ ) \mathbf{w}(\theta) w ( θ ) the linear predictor induced by a depth-h h h DLN parameterized by θ \theta θ . Then, for any predictor w ∈ R d \mathbf{w} \in\mathbb{R}^{d} w ∈ R d and weighting function c ( ⋅ , ⋅ ) > 0 c(\cdot,\cdot)>0 c ( ⋅ , ⋅ ) > 0 ,
∑ i = 1 d c i ∣ w i ∣ p / h = min w 1 , … , w h ∈ R d F ( w 1 , … , w h ) = w 1 h ∑ i = 1 d c i ∑ ℓ = 1 h ∣ ( w ℓ ) i ∣ p \sum_{i=1}^{d} c_i |w_i|^{p/h}
=
\min_{\substack{\mathbf{w}_1,\ldots,\mathbf{w}_h\in\mathbb{R}^{d}\\ F(\mathbf{w}_1,\ldots,\mathbf{w}_h)=\mathbf{w}}}
\frac{1}{h}\sum_{i=1}^{d}c_i\sum_{\ell=1}^{h}|(\mathbf{w}_{\ell})_i|^p i = 1 ∑ d c i ∣ w i ∣ p / h = w 1 , … , w h ∈ R d F ( w 1 , … , w h ) = w min h 1 i = 1 ∑ d c i ℓ = 1 ∑ h ∣ ( w ℓ ) i ∣ p
Proof. Since w i = ∏ ℓ = 1 h ( w ℓ ) i w_i=\prod_{\ell=1}^{h}(\mathbf{w}_{\ell})_i w i = ∏ ℓ = 1 h ( w ℓ ) i , the constraint and the objective separate over the coordinates. For every i i i , ∣ w i ∣ p = ∏ ℓ = 1 h ∣ ( w ℓ ) i ∣ p |w_i|^p=\prod_{\ell=1}^{h}|(\mathbf{w}_{\ell})_i|^p ∣ w i ∣ p = ∏ ℓ = 1 h ∣ ( w ℓ ) i ∣ p . Applying the arithmetic–geometric mean inequality to the h h h nonnegative quantities ∣ ( w 1 ) i ∣ p , … , ∣ ( w h ) i ∣ p |(\mathbf{w}_1)_i|^p,\ldots,|(\mathbf{w}_h)_i|^p ∣ ( w 1 ) i ∣ p , … , ∣ ( w h ) i ∣ p and multiplying by c i c_i c i gives
c i h ∑ ℓ = 1 h ∣ ( w ℓ ) i ∣ p ≥ c i ( ∏ ℓ = 1 h ∣ ( w ℓ ) i ∣ p ) 1 / h = c i ∣ w i ∣ p / h . \frac{c_i}{h}\sum_{\ell=1}^{h}|(\mathbf{w}_{\ell})_i|^p
\ge
c_i\left(\prod_{\ell=1}^{h}|(\mathbf{w}_{\ell})_i|^p\right)^{1/h}
=
c_i|w_i|^{p/h}. h c i ℓ = 1 ∑ h ∣ ( w ℓ ) i ∣ p ≥ c i ( ℓ = 1 ∏ h ∣ ( w ℓ ) i ∣ p ) 1/ h = c i ∣ w i ∣ p / h .
When w i ≠ 0 w_i\ne0 w i = 0 , equality is attained by choosing ( w 1 ) i = sign ( w i ) ∣ w i ∣ 1 / h (\mathbf{w}_1)_i=\operatorname{sign}(w_i)|w_i|^{1/h} ( w 1 ) i = sign ( w i ) ∣ w i ∣ 1/ h and ( w ℓ ) i = ∣ w i ∣ 1 / h (\mathbf{w}_{\ell})_i=|w_i|^{1/h} ( w ℓ ) i = ∣ w i ∣ 1/ h for every ℓ ∈ { 2 , … , h } \ell\in\{2,\ldots,h\} ℓ ∈ { 2 , … , h } . Indeed, these choices satisfy ∏ ℓ = 1 h ( w ℓ ) i = w i \prod_{\ell=1}^{h}(\mathbf{w}_{\ell})_i=w_i ∏ ℓ = 1 h ( w ℓ ) i = w i , and all h h h quantities in the arithmetic–geometric mean inequality equal ∣ w i ∣ p / h |w_i|^{p/h} ∣ w i ∣ p / h . Because c i c_i c i multiplies all h h h terms associated with coordinate i i i , it does not alter the equality condition. If w i = 0 w_i=0 w i = 0 , equality is attained by taking ( w ℓ ) i = 0 (\mathbf{w}_{\ell})_i=0 ( w ℓ ) i = 0 for every ℓ \ell ℓ . Summing over i i i proves the result. □
Theorem 28 (Information-theoretic lower bound). Assume x ∼ N ( 0 , 1 d I d ) \mathbf{x} \sim\mathcal{N}(0,\frac{1}{d}\mathbf{I}_d) x ∼ N ( 0 , d 1 I d ) and, conditionally on x \mathbf{x} x ,
P ( y = 1 ∣ x ) = ρ + ( 1 − 2 ρ ) e x T w ⋆ 1 + e x T w ⋆ . \mathbb{P}(y=1\mid\mathbf{x})=\rho+(1-2\rho)\frac{e^{\mathbf{x}^{\mathsf{T}}\mathbf{w}_{\star}}}{1+e^{\mathbf{x}^{\mathsf{T}}\mathbf{w}_{\star}}}. P ( y = 1 ∣ x ) = ρ + ( 1 − 2 ρ ) 1 + e x T w ⋆ e x T w ⋆ . where ρ ∈ [ 0 , 1 2 ) \rho\in[0,\frac{1}{2}) ρ ∈ [ 0 , 2 1 ) , and ∥ w ⋆ ∥ 2 2 d = ν 2 \frac{\lVert\mathbf{w}_{\star}\rVert_2^2}{d}=\nu^2 d ∥ w ⋆ ∥ 2 2 = ν 2 , with ν > 0 \nu>0 ν > 0 . Let r > 0 r>0 r > 0 be the solution of r = ( 1 − 2 ρ ) E [ ϕ ′ ′ ( ν Z ) ] E [ ϕ ′ ′ ( ν r Z ) ] r=\frac{(1-2\rho)\mathbb{E}[\phi^{\prime\prime}(\nu Z)]}{\mathbb{E}[\phi^{\prime\prime}(\nu rZ)]} r = E [ ϕ ′′ ( ν r Z )] ( 1 − 2 ρ ) E [ ϕ ′′ ( ν Z )] , where Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) . Then, there exists c ρ , ν > 0 c_{\rho,\nu}>0 c ρ , ν > 0 (depending on ρ , ν \rho,\nu ρ , ν ) such that
inf w ^ sup ∥ w ⋆ ∥ 2 = ν d = ν E [ L ( w ^ ) − L ( r w ⋆ ) ] ≥ c ρ , ν min { 1 , 1 δ T } . \inf_{\widehat{\mathbf{w}}}\ \sup_{\lVert\mathbf{w}_{\star}\rVert_2=\nu\sqrt{d}}=\nu\mathbb{E}\left[\mathcal{L}(\widehat{\mathbf{w}})-\mathcal{L}(r\mathbf{w}_{\star})\right]\geq c_{\rho,\nu}\min\left\{1,\frac{1}{\delta T}\right\}. w inf ∥ w ⋆ ∥ 2 = ν d sup = ν E [ L ( w ) − L ( r w ⋆ ) ] ≥ c ρ , ν min { 1 , δ T 1 } .
Proof. We allow our estimator w ^ \widehat{\mathbf{w}} w to depend jointly on all n T nT n T samples. Thus any lower bound applies to every sequential learner. Recall that
L ( w ) = E [ ϕ ( x T w ) − ( ρ + ( 1 − 2 ρ ) ϕ ′ ( x T w ⋆ ) ) x T w ] . \mathcal{L}(\mathbf{w})=\mathbb{E}\left[\phi(\mathbf{x}^{\mathsf{T}}\mathbf{w})-\left(\rho+(1-2\rho)\phi^{\prime}(\mathbf{x}^{\mathsf{T}}\mathbf{w}_{\star})\right)\mathbf{x}^{\mathsf{T}}\mathbf{w}\right]. L ( w ) = E [ ϕ ( x T w ) − ( ρ + ( 1 − 2 ρ ) ϕ ′ ( x T w ⋆ ) ) x T w ] .
Since E [ x ] = 0 \mathbb{E}[\mathbf{x}]=0 E [ x ] = 0 , the ρ \rho ρ -term vanishes. By Stein’s Lemma, E [ ϕ ′ ( x T w ⋆ ) x ] = h ( ν ) d w ⋆ \mathbb{E}[\phi^{\prime}(\mathbf{x}^{\mathsf{T}}\mathbf{w}_{\star})\mathbf{x}]=\frac{h(\nu)}{d}\mathbf{w}_{\star} E [ ϕ ′ ( x T w ⋆ ) x ] = d h ( ν ) w ⋆ . Therefore,
L ( w ) = E [ ϕ ( x T w ) ] − ( 1 − 2 ρ ) h ( ν ) d w ⋆ T w . \mathcal{L}(\mathbf{w})=\mathbb{E}[\phi(\mathbf{x}^{\mathsf{T}}\mathbf{w})]-\frac{(1-2\rho)h(\nu)}{d}\mathbf{w}_{\star}^{\mathsf{T}}\mathbf{w}. L ( w ) = E [ ϕ ( x T w )] − d ( 1 − 2 ρ ) h ( ν ) w ⋆ T w .
For fixed ∥ w ∥ 2 \lVert\mathbf{w}\rVert_2 ∥ w ∥ 2 , the first term depends only on ∥ w ∥ 2 \lVert\mathbf{w}\rVert_2 ∥ w ∥ 2 , because x T w ∼ N ( 0 , ∥ w ∥ 2 2 d ) \mathbf{x}^{\mathsf{T}}\mathbf{w}\sim\mathcal{N}(0,\frac{\lVert\mathbf{w}\rVert_2^2}{d}) x T w ∼ N ( 0 , d ∥ w ∥ 2 2 ) . The second term is minimized by maximizing w ⋆ T w \mathbf{w}_{\star}^{\mathsf{T}}\mathbf{w} w ⋆ T w . By Cauchy–Schwarz, this happens when w \mathbf{w} w is a nonnegative multiple of w ⋆ \mathbf{w}_{\star} w ⋆ . Thus the minimizer has the form w = a w ⋆ \mathbf{w}=a\mathbf{w}_{\star} w = a w ⋆ , with a ≥ 0 a\geq0 a ≥ 0 . For such we have w , x T ( a w ⋆ ) ∼ N ( 0 , a 2 ν 2 ) \mathbf{w},\mathbf{x}^{\mathsf{T}}(a\mathbf{w}_{\star})\sim\mathcal{N}(0,a^2\nu^2) w , x T ( a w ⋆ ) ∼ N ( 0 , a 2 ν 2 ) , and
d d a L ( a w ⋆ ) = ν 2 [ a h ( ν a ) − ( 1 − 2 ρ ) h ( ν ) ] . \frac{\mathrm{d}}{\mathrm{d}a}\mathcal{L}(a\mathbf{w}_{\star})=\nu^2\left[ah(\nu a)-(1-2\rho)h(\nu)\right]. d a d L ( a w ⋆ ) = ν 2 [ ah ( ν a ) − ( 1 − 2 ρ ) h ( ν ) ] .
Moreover, d d q ( q h ( q ) ) = E [ Z 2 ϕ ′ ′ ( q Z ) ] > 0 \frac{\mathrm{d}}{\mathrm{d}q}(qh(q))=\mathbb{E}[Z^2\phi^{\prime\prime}(qZ)]>0 d q d ( q h ( q )) = E [ Z 2 ϕ ′′ ( q Z )] > 0 . Hence the unique minimizer is r w ⋆ r\mathbf{w}_{\star} r w ⋆ , where r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) rh(\nu r)=(1-2\rho)h(\nu) r h ( ν r ) = ( 1 − 2 ρ ) h ( ν ) . The Hessian is ∇ 2 L ( w ) = E [ ϕ ′ ′ ( x T w ) x x T ] \nabla^2\mathcal{L}(\mathbf{w})=\mathbb{E}[\phi^{\prime\prime}(\mathbf{x}^{\mathsf{T}}\mathbf{w})\mathbf{x}\mathbf{x}^{\mathsf{T}}] ∇ 2 L ( w ) = E [ ϕ ′′ ( x T w ) x x T ] . Thus, for every u ∈ R d \mathbf{u}\in\mathbb{R}^d u ∈ R d ,
u T ∇ 2 L ( w ) u = E [ ϕ ′ ′ ( x T w ) ( x T u ) 2 ] . \mathbf{u}^{\mathsf{T}}\nabla^2\mathcal{L}(\mathbf{w})\mathbf{u}=\mathbb{E}[\phi^{\prime\prime}(\mathbf{x}^{\mathsf{T}}\mathbf{w})(\mathbf{x}^{\mathsf{T}}\mathbf{u})^2]. u T ∇ 2 L ( w ) u = E [ ϕ ′′ ( x T w ) ( x T u ) 2 ] .
By rotational invariance, we take w = ∥ w ∥ 2 e 1 \mathbf{w}=\lVert\mathbf{w}\rVert_2\mathbf{e}_1 w = ∥ w ∥ 2 e 1 . Write x = d − 1 2 ( Z 1 , … , Z d ) \mathbf{x}=d^{-\frac{1}{2}}(Z_1,\ldots,Z_d) x = d − 2 1 ( Z 1 , … , Z d ) and u = u ∥ e 1 + u ⊥ \mathbf{u}=u_{\parallel}\mathbf{e}_1+\mathbf{u}_{\perp} u = u ∥ e 1 + u ⊥ ,
E [ ϕ ′ ′ ( x T w ) ( x T u ) 2 ] = 1 d ( u ∥ 2 E [ ϕ ′ ′ ( ∥ w ∥ 2 d Z ) Z 2 ] + ∥ u ⊥ ∥ 2 2 E [ ϕ ′ ′ ( ∥ w ∥ 2 d Z ) ] ) . \mathbb{E}[\phi^{\prime\prime}(\mathbf{x}^{\mathsf{T}}\mathbf{w})(\mathbf{x}^{\mathsf{T}}\mathbf{u})^2]=\frac{1}{d}\left(u_{\parallel}^2\mathbb{E}\left[\phi^{\prime\prime}\left(\frac{\lVert\mathbf{w}\rVert_2}{\sqrt{d}}Z\right)Z^2\right]+\lVert\mathbf{u}_{\perp}\rVert_2^2\mathbb{E}\left[\phi^{\prime\prime}\left(\frac{\lVert\mathbf{w}\rVert_2}{\sqrt{d}}Z\right)\right]\right). E [ ϕ ′′ ( x T w ) ( x T u ) 2 ] = d 1 ( u ∥ 2 E [ ϕ ′′ ( d ∥ w ∥ 2 Z ) Z 2 ] + ∥ u ⊥ ∥ 2 2 E [ ϕ ′′ ( d ∥ w ∥ 2 Z ) ] ) .
Let c = min ∣ a ∣ ≤ ν { min { E [ ϕ ′ ′ ( a Z ) Z 2 ] , E [ ϕ ′ ′ ( a Z ) ] } } > 0 c=\min_{|a|\leq\nu}\{\min\{\mathbb{E}[\phi^{\prime\prime}(aZ)Z^2],\mathbb{E}[\phi^{\prime\prime}(aZ)]\}\}>0 c = min ∣ a ∣ ≤ ν { min { E [ ϕ ′′ ( a Z ) Z 2 ] , E [ ϕ ′′ ( a Z )]}} > 0 , then E [ ϕ ′ ′ ( x T w ) ( x T u ) 2 ] ≥ c d ∥ u ∥ 2 2 \mathbb{E}[\phi^{\prime\prime}(\mathbf{x}^{\mathsf{T}}\mathbf{w})(\mathbf{x}^{\mathsf{T}}\mathbf{u})^2]\geq\frac{c}{d}\lVert\mathbf{u}\rVert_2^2 E [ ϕ ′′ ( x T w ) ( x T u ) 2 ] ≥ d c ∥ u ∥ 2 2 . On every set where ∥ w ∥ 2 d \frac{\lVert\mathbf{w}\rVert_2}{\sqrt{d}} d ∥ w ∥ 2 is bounded by a constant depending only on ν \nu ν , Gaussian rotational invariance and ϕ ′ ′ > 0 \phi^{\prime\prime}>0 ϕ ′′ > 0 imply ∇ 2 L ( w ) ⪰ c d − 1 I d \nabla^2\mathcal{L}(\mathbf{w})\succeq cd^{-1}\mathbf{I}_d ∇ 2 L ( w ) ⪰ c d − 1 I d . Therefore, by Taylor’s theorem, if ∥ w − r w ⋆ ∥ 2 ≤ c d \lVert\mathbf{w}-r\mathbf{w}_{\star}\rVert_2\leq c\sqrt{d} ∥ w − r w ⋆ ∥ 2 ≤ c d ,
L ( w ) − L ( r w ⋆ ) ≥ c ρ , ν ∥ w − r w ⋆ ∥ 2 2 d . \mathcal{L}(\mathbf{w})-\mathcal{L}(r\mathbf{w}_{\star})\geq c_{\rho,\nu}\frac{\lVert\mathbf{w}-r\mathbf{w}_{\star}\rVert_2^2}{d}. L ( w ) − L ( r w ⋆ ) ≥ c ρ , ν d ∥ w − r w ⋆ ∥ 2 2 .
Moreover the risk is continuous, has the unique minimizer r w ⋆ r\mathbf{w}_{\star} r w ⋆ , and is coercive, E [ ϕ ( x T w ) ] ≥ E [ ( x T w ) + ] = ∥ w ∥ 2 2 π \mathbb{E}[\phi(\mathbf{x}^{\mathsf{T}}\mathbf{w})]\geq\mathbb{E}[(\mathbf{x}^{\mathsf{T}}\mathbf{w})_+]=\frac{\lVert\mathbf{w}\rVert_2}{\sqrt{2\pi}} E [ ϕ ( x T w )] ≥ E [( x T w ) + ] = 2 π ∥ w ∥ 2 and − ( 1 − 2 ρ ) h ( ν ) w ⋆ T w ≥ − ( 1 − 2 ρ ) ν h ( ν ) ∥ w ∥ -(1-2\rho)h(\nu)\mathbf{w}_{\star}^{\mathsf{T}}\mathbf{w}\geq-(1-2\rho)\nu h(\nu)\lVert\mathbf{w}\rVert − ( 1 − 2 ρ ) h ( ν ) w ⋆ T w ≥ − ( 1 − 2 ρ ) ν h ( ν ) ∥ w ∥ . Therefore,
L ( d w ) = [ 1 2 π − ( 1 − 2 ρ ) ν h ( ν ) ] ∥ w ∥ ⟶ ∥ w ∥ → ∞ ∞ \mathcal{L}(\sqrt{d}\mathbf{w})=\left[\frac{1}{\sqrt{2\pi}}-(1-2\rho)\nu h(\nu)\right]\lVert\mathbf{w}\rVert\underset{\lVert\mathbf{w}\rVert\to\infty}{\longrightarrow}\infty L ( d w ) = [ 2 π 1 − ( 1 − 2 ρ ) ν h ( ν ) ] ∥ w ∥ ∥ w ∥ → ∞ ⟶ ∞
since 1 2 π − ν h ( ν ) = ∫ R ϕ ′ ′ ( u ) ( 1 − e − u 2 2 ν 2 ) d u ≥ c ( 1 + ν ) 2 \frac{1}{\sqrt{2\pi}}-\nu h(\nu)=\int_{\mathbb{R}}\phi^{\prime\prime}(u)\left(1-e^{-\frac{u^2}{2\nu^2}}\right)\mathrm{d}u\geq\frac{c}{(1+\nu)^2} 2 π 1 − ν h ( ν ) = ∫ R ϕ ′′ ( u ) ( 1 − e − 2 ν 2 u 2 ) d u ≥ ( 1 + ν ) 2 c . Hence, after decreasing c ρ , ν c_{\rho,\nu} c ρ , ν if necessary, for all w ∈ R d \mathbf{w}\in\mathbb{R}^d w ∈ R d ,
L ( w ) − L ( r w ⋆ ) ≥ c ρ , ν min { 1 , ∥ w − r w ⋆ ∥ 2 2 d } . (150) \mathcal{L}(\mathbf{w})-\mathcal{L}(r\mathbf{w}_{\star})\geq c_{\rho,\nu}\min\left\{1,\frac{\lVert\mathbf{w}-r\mathbf{w}_{\star}\rVert_2^2}{d}\right\}. \tag*{(150)} L ( w ) − L ( r w ⋆ ) ≥ c ρ , ν min { 1 , d ∥ w − r w ⋆ ∥ 2 2 } . (150)
We now use Assouad’s Lemma. Assouad’s Lemma provides a methodological way to give an uniform lower bound of the objective. Set b 2 = c 0 d n T ∨ d b^2=c_0\frac{d}{nT\vee d} b 2 = c 0 n T ∨ d d , where c 0 > 0 c_0>0 c 0 > 0 is sufficiently small depending only on ν \nu ν . For every v ∈ { − 1 , + 1 } d − 1 \mathbf{v}\in\{-1,+1\}^{d-1} v ∈ { − 1 , + 1 } d − 1 , define
w v = ν 2 d − ( d − 1 ) b 2 e 1 + b ∑ j = 1 d − 1 v j e j + 1 . \mathbf{w}_{\mathbf{v}}=\sqrt{\nu^2d-(d-1)b^2}\mathbf{e}_1+b\sum_{j=1}^{d-1}v_j\mathbf{e}_{j+1}. w v = ν 2 d − ( d − 1 ) b 2 e 1 + b j = 1 ∑ d − 1 v j e j + 1 .
Then ∥ w v ∥ 2 2 = ν 2 d \|\mathbf{w}_{\mathbf{v}}\|_2^2=\nu^2d ∥ w v ∥ 2 2 = ν 2 d . For two parameters v , v ′ \mathbf{v},\mathbf{v}' v , v ′ differing only in coordinate j j j , ∥ w v − w v ′ ∥ 2 2 = 4 b 2 \|\mathbf{w}_{\mathbf{v}}-\mathbf{w}_{\mathbf{v}'}\|_2^2=4b^2 ∥ w v − w v ′ ∥ 2 2 = 4 b 2 . For the ordinary logistic model, its KL divergence is given by,
D K L ( Ber ( ϕ ′ ( u ) ) ∥ Ber ( ϕ ′ ( v ) ) ) = ϕ ( v ) − ϕ ( u ) − ϕ ′ ( u ) ( v − u ) ≤ 1 8 ( u − v ) 2 . D_{\mathrm{KL}}\bigl(\operatorname{Ber}(\phi'(u))\Vert\operatorname{Ber}(\phi'(v))\bigr)=\phi(v)-\phi(u)-\phi'(u)(v-u)\leq\frac{1}{8}(u-v)^2. D KL ( Ber ( ϕ ′ ( u )) ∥ Ber ( ϕ ′ ( v )) ) = ϕ ( v ) − ϕ ( u ) − ϕ ′ ( u ) ( v − u ) ≤ 8 1 ( u − v ) 2 .
because ϕ ′ ′ ≤ 1 4 \phi''\leq\frac{1}{4} ϕ ′′ ≤ 4 1 , and Taylor’s Theorem. The noisy model with parameter ρ \rho ρ is obtained by passing the logistic label through an independent binary symmetric channel with flip probability ρ \rho ρ . The data-processing inequality for KL says that applying the same stochastic channel to two distributions cannot increase their KL divergence,
D K L ( Ber ( ρ + ( 1 − 2 ρ ) ϕ ′ ( u ) ) ∥ Ber ( ρ + ( 1 − 2 ρ ) ϕ ′ ( v ) ) ) ≤ 1 8 ( u − v ) 2 . D_{\mathrm{KL}}\bigl(\operatorname{Ber}(\rho+(1-2\rho)\phi'(u))\Vert\operatorname{Ber}(\rho+(1-2\rho)\phi'(v))\bigr)\leq\frac{1}{8}(u-v)^2. D KL ( Ber ( ρ + ( 1 − 2 ρ ) ϕ ′ ( u )) ∥ Ber ( ρ + ( 1 − 2 ρ ) ϕ ′ ( v )) ) ≤ 8 1 ( u − v ) 2 .
Thus, for one observation,
D K L ( P v ∥ P v ′ ) ≤ 1 8 d E [ ( x T ( w v − w v ′ ) ) 2 ] = 1 8 d ∥ w v − w v ′ ∥ 2 2 = b 2 2 d = c 0 2 ( n T ∨ d ) . D_{\mathrm{KL}}(P_{\mathbf{v}}\Vert P_{\mathbf{v}'})\leq\frac{1}{8d}\mathbb{E}\left[(\mathbf{x}^{\mathsf T}(\mathbf{w}_{\mathbf{v}}-\mathbf{w}_{\mathbf{v}'}))^2\right]=\frac{1}{8d}\|\mathbf{w}_{\mathbf{v}}-\mathbf{w}_{\mathbf{v}'}\|_2^2=\frac{b^2}{2d}=\frac{c_0}{2(nT\vee d)}. D KL ( P v ∥ P v ′ ) ≤ 8 d 1 E [ ( x T ( w v − w v ′ ) ) 2 ] = 8 d 1 ∥ w v − w v ′ ∥ 2 2 = 2 d b 2 = 2 ( n T ∨ d ) c 0 .
For n T nT n T observations, D K L ( P v ⊗ n T ∥ P v ′ ⊗ n T ) ≤ c 0 2 D_{\mathrm{KL}}(P_{\mathbf{v}}^{\otimes nT}\Vert P_{\mathbf{v}'}^{\otimes nT})\leq\frac{c_0}{2} D KL ( P v ⊗ n T ∥ P v ′ ⊗ n T ) ≤ 2 c 0 . Choosing c 0 c_0 c 0 sufficiently small, Assouad’s lemma implies that for every estimator v ^ ∈ { − 1 , + 1 } d − 1 \widehat{\mathbf{v}}\in\{-1,+1\}^{d-1} v ∈ { − 1 , + 1 } d − 1 ,
sup v ∈ { − 1 , + 1 } d − 1 E v [ d H ( v ^ , v ) ] ≥ c ( d − 1 ) . \sup_{\mathbf{v}\in\{-1,+1\}^{d-1}}\mathbb{E}_{\mathbf{v}}[d_H(\widehat{\mathbf{v}},\mathbf{v})]\geq c(d-1). v ∈ { − 1 , + 1 } d − 1 sup E v [ d H ( v , v )] ≥ c ( d − 1 ) .
where d H ( v , v ′ ) = ∑ j = 1 d − 1 1 { v ^ j ≠ v j } d_H(\mathbf{v},\mathbf{v}')=\sum_{j=1}^{d-1}\mathbf{1}\{\widehat{v}_j\neq v_j\} d H ( v , v ′ ) = ∑ j = 1 d − 1 1 { v j = v j } is the Hamming distance. Given an arbitrary estimator w ^ \widehat{\mathbf{w}} w , define its induced sign estimator by v ^ j = sign ( w ^ j + 1 ) \widehat{v}_j=\operatorname{sign}(\widehat{w}_{j+1}) v j = sign ( w j + 1 ) , with an arbitrary convention when w ^ j + 1 = 0 \widehat{w}_{j+1}=0 w j + 1 = 0 . If v ^ j ≠ v j \widehat{v}_j\neq v_j v j = v j , then ∣ w ^ j + 1 − r v j ∣ ≥ r |\widehat{w}_{j+1}-r v_j|\geq r ∣ w j + 1 − r v j ∣ ≥ r . Therefore
∥ w ^ − r w v ∥ 2 2 ≥ r 2 b 2 d H ( v ^ , v ) . \|\widehat{\mathbf{w}}-r\mathbf{w}_{\mathbf{v}}\|_2^2\geq r^2b^2d_H(\widehat{\mathbf{v}},\mathbf{v}). ∥ w − r w v ∥ 2 2 ≥ r 2 b 2 d H ( v , v ) .
Using equation 143 and choosing c 0 c_0 c 0 smaller if necessary so that r 2 c 0 ≤ 1 r^2c_0\leq1 r 2 c 0 ≤ 1 ,
L w v ( w ^ ) − L w v ( r w v ) ≥ c ρ , ν r 2 b 2 d d H ( v ^ , v ) . \mathcal{L}_{\mathbf{w}_{\mathbf{v}}}(\widehat{\mathbf{w}})-\mathcal{L}_{\mathbf{w}_{\mathbf{v}}}(r\mathbf{w}_{\mathbf{v}})\geq c_{\rho,\nu}\frac{r^2b^2}{d}d_H(\widehat{\mathbf{v}},\mathbf{v}). L w v ( w ) − L w v ( r w v ) ≥ c ρ , ν d r 2 b 2 d H ( v , v ) .
Taking expectations and applying Assouad’s lemma gives
inf w ^ sup ∥ w ∗ ∥ 2 2 = ν d E [ L ( w ^ ) − L ( r w ∗ ) ] ≥ c ρ , ν b 2 d inf v ^ sup v ∈ { − 1 , + 1 } d − 1 E v [ d H ( v ^ , v ) ] ≥ c ρ , ν b 2 d ( d − 1 ) ≥ c ρ , ν d n T ∨ d = c ρ , ν min { 1 , d n T } . \begin{aligned}
\inf_{\widehat{\mathbf{w}}}\sup_{\|\mathbf{w}_*\|_2^2=\nu\sqrt{d}}\mathbb{E}\left[\mathcal{L}(\widehat{\mathbf{w}})-\mathcal{L}(r\mathbf{w}_*)\right]\geq c_{\rho,\nu}\frac{b^2}{d}\inf_{\widehat{\mathbf{v}}}\sup_{\mathbf{v}\in\{-1,+1\}^{d-1}}\mathbb{E}_{\mathbf{v}}[d_H(\widehat{\mathbf{v}},\mathbf{v})] \\
&\geq c_{\rho,\nu}\frac{b^2}{d}(d-1)\geq c_{\rho,\nu}\frac{d}{nT\vee d}=c_{\rho,\nu}\min\left\{1,\frac{d}{nT}\right\}.
\end{aligned} w inf ∥ w ∗ ∥ 2 2 = ν d sup E [ L ( w ) − L ( r w ∗ ) ] ≥ c ρ , ν d b 2 v inf v ∈ { − 1 , + 1 } d − 1 sup E v [ d H ( v , v )] ≥ c ρ , ν d b 2 ( d − 1 ) ≥ c ρ , ν n T ∨ d d = c ρ , ν min { 1 , n T d } .
Finally, since n d = δ ∈ ( 0 , ∞ ) \frac{n}{d}=\delta\in(0,\infty) d n = δ ∈ ( 0 , ∞ ) , and d n T = 1 T ( n / d ) = 1 δ T \frac{d}{nT}=\frac{1}{T(n/d)}=\frac{1}{\delta T} n T d = T ( n / d ) 1 = δ T 1 . Therefore,
inf w ^ sup ∥ w ∗ ∥ 2 2 = ν d E [ L ( w ^ ) − L ( r w ∗ ) ] ≥ c ρ , ν min { 1 , 1 δ T } . \inf_{\widehat{\mathbf{w}}}\sup_{\|\mathbf{w}_*\|_2^2=\nu\sqrt{d}}\mathbb{E}\left[\mathcal{L}(\widehat{\mathbf{w}})-\mathcal{L}(r\mathbf{w}_*)\right]\geq c_{\rho,\nu}\min\left\{1,\frac{1}{\delta T}\right\}. w inf ∥ w ∗ ∥ 2 2 = ν d sup E [ L ( w ) − L ( r w ∗ ) ] ≥ c ρ , ν min { 1 , δ T 1 } .