Introduction
A recurring theme in learning theory is that provable guarantees can be highly sensitive to the structure of label noise. The classical random classification noise model (RCN), in which every label is independently flipped at a fixed rate, is straightforward to analyze, and has played a foundational role in the theory of noisy halfspace learning [2 ] . However, even mild departures from this model can change the statistical and computational behavior of classification problems. For halfspaces, allowing the corruption rate to vary with the instance, or with the margin to the decision boundary, has led to substantially different algorithms and strong statistical-query lower bounds [13 , 31 , 17 ] . In this work we revisit a particularly sharp instance of this phenomenon: learning an affine Gaussian halfspace under label noise in the large-bias regime.
Concretely, we observe i.i.d. labeled examples ( X , Y ) (X,Y) ( X , Y ) with X ∼ N ( 0 , I d ) X \sim\mathcal{N}(0,I_d) X ∼ N ( 0 , I d ) , Y ∈ { + 1 } Y \in\{+1\} Y ∈ { + 1 } , and an unknown target separator f ⋆ ( x ) = sign ( m ( x ) ) f^\star(x) = \operatorname{sign}(m(x)) f ⋆ ( x ) = sign ( m ( x )) , where m ( x ) = ⟨ w ⋆ , x ⟩ + t ⋆ m(x) = \langle w^\star,x\rangle+ t^\star m ( x ) = ⟨ w ⋆ , x ⟩ + t ⋆ is the signed margin, and w ⋆ ∈ S d − 1 w^\star\in S^{d-1} w ⋆ ∈ S d − 1 , t ⋆ ∈ R t^\star\in\mathbb{R} t ⋆ ∈ R . Our goal is to output a halfspace whose excess risk over f ⋆ f^\star f ⋆ is at most a target accuracy ε \varepsilon ε .
For ease of exposition, assume the threshold t ⋆ > 0 t^\star> 0 t ⋆ > 0 (the same ideas apply to t ⋆ < 0 t^\star< 0 t ⋆ < 0 by symmetry). Then, the halfspace has bias p = min { P ( f ⋆ ( X ) = − 1 ) , P ( f ⋆ ( X ) = + 1 ) } = Φ ( − t ⋆ ) p = \min\{\mathbb{P}(f^\star(X) = -1), \mathbb{P}(f^\star(X) = +1)\} = \Phi(-t^\star) p = min { P ( f ⋆ ( X ) = − 1 ) , P ( f ⋆ ( X ) = + 1 )} = Φ ( − t ⋆ ) , where Φ ( ⋅ ) \Phi(\cdot) Φ ( ⋅ ) is the usual Gaussian CDF. Conditioned on X = x X = x X = x , the observed label satisfies Y = f ⋆ ( x ) Y = f^\star(x) Y = f ⋆ ( x ) with probability 1 − g ( ∣ m ( x ) ∣ ) 1-g(|m(x)|) 1 − g ( ∣ m ( x ) ∣ ) and Y = − f ⋆ ( x ) Y = -f^\star(x) Y = − f ⋆ ( x ) with probability g ( ∣ m ( x ) ∣ ) g(|m(x)|) g ( ∣ m ( x ) ∣ ) , where g : [ 0 , ∞ ) → [ 0 , 1 / 2 ] g : [0,\infty) \to[0,1/2] g : [ 0 , ∞ ) → [ 0 , 1/2 ] is nonincreasing. This margin-dependent noise profile, going back to Bylander’s [9 ] monotonic-noise model for linear threshold functions, interpolates between the flat η \eta η -RCN model, where g ( ⋅ ) = η g(\cdot) = \eta g ( ⋅ ) = η and every label is flipped at the same rate, and boundary-local noise, where corruption is concentrated near the decision boundary and decays away from it. The latter describes a realistic annotation regime, in which labels far from the boundary are more reliable, while labels near the boundary may be close to random. Crowdsourced labels, for example from Amazon Mechanical Turk (AMT), behave this way. Annotators tend to agree on prototypical instances, which we can think of as points far from the decision boundary, and their labels become nearly random on ambiguous instances close to the separator [36 , 6 ] . In contrast, η \eta η -RCN describes errors that do not depend on the instance at all, such as labels from annotators who answer without looking at the samples [32 ] .
The large-bias regime is where the distinction between flat and boundary-local noise becomes most pronounced. When t ⋆ > 0 t^\star> 0 t ⋆ > 0 is large, the true negative class has mass p = Φ ( − t ⋆ ) ≪ 1 p = \Phi(-t^\star) \ll1 p = Φ ( − t ⋆ ) ≪ 1 , and most samples lie deep in the positive region. Conditioning on the positive side removes only a set of mass p p p , so a positive sample looks almost like a draw from N ( 0 , I d ) \mathcal{N}(0,I_d) N ( 0 , I d ) and says little about the orientation of the separator. The negative samples, on the other hand, all come from the thin Gaussian tail cut off by the separator, so where they lie is tied directly to the orientation of the halfspace. In the absence of noise, conditioning on this tail is extremely favorable: if X = Z w ⋆ + X ⊥ X = Z w^\star+ X_\perp X = Z w ⋆ + X ⊥ with Z = ⟨ w ⋆ , X ⟩ ∼ N ( 0 , 1 ) Z = \langle w^\star, X\rangle\sim\mathcal{N}(0,1) Z = ⟨ w ⋆ , X ⟩ ∼ N ( 0 , 1 ) and X ⊥ ∼ N ( 0 , I d − w ⋆ w ⋆ T ) X_\perp\sim\mathcal{N}(0,I_d-w^\star{w^\star}^{T}) X ⊥ ∼ N ( 0 , I d − w ⋆ w ⋆ T ) , then the event Y = − 1 Y=-1 Y = − 1 is simply { Z < − t ⋆ } \{Z<-t^\star\} { Z < − t ⋆ } . Conditioning on this event then shifts the Z Z Z coordinate, but leaves the orthogonal directions unchanged in distribution. Consequently, the signal E [ X ∣ Y = − 1 ] \mathbb{E}[X \mid Y=-1] E [ X ∣ Y = − 1 ] is collinear with − w ⋆ -w^\star − w ⋆ and has large norm Θ ( t ⋆ ) \Theta(t^\star) Θ ( t ⋆ ) relative to the unit variance noise in every direction.
This suggests an extremely simple estimator: average the examples with the empirically rarer label and orient the resulting vector according to that label. We call this the rare-tail direction estimator (Algorithm 1). This is essentially a variant of Servedio’s classical Averaging algorithm [33 ] , with the labels centered at the majority label, so that the majority samples, which carry noise but no additional signal, drop out of the average. The difference is small for homogeneous halfspaces, but striking for affine ones: in the noiseless biased benchmark discussed above, this rare-tail statistic reaches O ( 1 / t ⋆ ) O(1/t^{\star}) O ( 1/ t ⋆ ) angular precision at the information-theoretically optimal O ~ ( d / p ) \widetilde{O}(d/p) O ( d / p ) sample rate, whereas the global Average statistic requires O ~ ( d / p 2 ) \widetilde{O}(d/p^{2}) O ( d / p 2 ) samples to reach the same precision (Figure 1 ).
Figure 1. A 2D cartoon of the noiseless biased regime. The solid black line is the true boundary and the point − t ⋆ w ⋆ -t^\star w^\star − t ⋆ w ⋆ is its closest point to the origin. The dashed gray axes depict the decomposition X = Z w ⋆ + X ⊥ X = Z w^\star+ X_\perp X = Z w ⋆ + X ⊥ , and the colored dashed separators are drawn through the same point − t ⋆ w ⋆ -t^\star w^\star − t ⋆ w ⋆ to isolate directional error. Panel (a) uses the empirical Average moment, while panel (b) uses the empirical minority mean. For the same sample budget n = O ~ ( d / p ) n = \widetilde{O}(d/p) n = O ( d / p ) , the minority mean typically achieves O ~ ( 1 / t ⋆ ) \widetilde{O}(1/t^\star) O ( 1/ t ⋆ ) -scale angular distance to w ⋆ w^\star w ⋆ , while the Average estimator typically only reaches O ~ ( 1 / ( t ⋆ p ) ) \widetilde{O}(1/(t^\star\sqrt{p})) O ( 1/ ( t ⋆ p )) -scale, which explains the larger angle in (a) compared to (b). The signal magnitude of the Average moment is smaller by a 2 p 2p 2 p factor, hence the shorter arrow in (a), while its orthogonal noise is smaller only by a p \sqrt{p} p factor, since it also sums over the majority samples.
It remains to understand whether this geometric intuition survives when the labels are unreliable. We find that the answer depends on the shape of the noise profile, as opposed to just the amount of noise there is. In particular, constant-rate flat noise profiles destroy the rare-tail advantage. Under η \eta η -RCN, points far inside the majority region are flipped into the observed negative class at the same rate as points near the boundary. In the large-bias regime, this creates a constant-rate stream of false negatives from the Gaussian bulk that dilute the signal, since the observed negative class is no longer a rare tail event, but a mixture of the true tail and mislabeled majority points. Boundary-local noise can produce just as many false negatives, even far more than the true tail, but these are drawn preferentially from near the separator, which, in the large-bias regime, lies far from the origin, in the direction − w ⋆ -w^{\star} − w ⋆ . Therefore, the observed negative class is no longer the true tail, yet the false negatives themselves pull its mean along − w ⋆ -w^{\star} − w ⋆ , thereby turning label noise into signal. The obstruction is therefore not inherent to label noise itself, but to the flatness of the noise profile.
The central question of the paper is whether this one-shot statistic retains its geometric advantage when the noise is not flat, but is otherwise as severe as possible.
We answer this question in the affirmative for a broad class of noise profiles that satisfy a mild regularity condition. This condition, which we call regular variation, rules out RCN-like flat plateaus, but includes noise models that may force random labels close to the decision boundary, but have very slow decay with distance to it. In particular, it includes canonical boundary-local models such as probit noise, which permits the flip probability near the boundary to be arbitrarily close to 1 / 2 1/2 1/2 . Our main theorem shows that, under this condition, the rare-tail direction estimator recovers w ⋆ w^{\star} w ⋆ to O ( 1 / t ⋆ ) O(1/t^{\star}) O ( 1/ t ⋆ ) angular accuracy using the same leading sample scale as the noiseless regime. Moreover, this is exactly the angular scale needed in the large-bias regime. Indeed, when p = Θ ( ε ) p=\Theta(\varepsilon) p = Θ ( ε ) , a direction within O ( 1 / t ⋆ ) O(1/t^{\star}) O ( 1/ t ⋆ ) and a correctly chosen threshold already yield ε \varepsilon ε excess risk. A one-dimensional threshold search along the recovered direction (Algorithm 2) then gives a complete learner that runs in nearly linear time.
Isolating the shape of the noise profile as the obstruction also sheds light on a known computational barrier for learning biased Gaussian halfspaces under RCN. Diakonikolas et al. [20 ] give efficient algorithms for η \eta η -RCN, but in the critical regime p = Θ ( ε ) p=\Theta(\varepsilon) p = Θ ( ε ) , their sample complexity has a leading O ~ ( d / ε 2 ) \widetilde{O}(d/\varepsilon^{2}) O ( d / ε 2 ) term, while the information-theoretic rate is O ~ ( d / ε ) \widetilde{O}(d/\varepsilon) O ( d / ε ) . They also give a statistical-query lower bound, suggesting that this quadratic dependence is unavoidable for efficient learners that only see RCN samples. Our analysis suggests where this barrier comes from, namely, the search for a warm start in the presence of constant-rate flat noise. As discussed above, flat noise buries much of the signal, while the same halfspace is easy to warm-start from boundary-local noisy labels. This is also the only part where flatness meaningfully affects learning: given a direction within O ( 1 / t ⋆ ) O(1/t^{\star}) O ( 1/ t ⋆ ) of w ⋆ w^{\star} w ⋆ , the threshold is a one-dimensional learning problem, and the local refinement procedure of Diakonikolas et al. [20 ] from this warm-start is sample-efficient, even under RCN.
This suggests a way around the barrier: obtain the warm start from labels whose noise is not flat, and use the RCN samples only for the remaining steps. Strategies of this kind, in which a learner sees data in stages, starting from samples where the signal is easier to extract, have long been used in empirical machine learning under the name of curriculum learning [7 ] , including for training on noisy labels, where the curriculum favors examples whose labels are more likely to be correct [24 , 27 ] . Our two-stage approach serves as a mathematical model of curriculum learning for noisy halfspaces, where both stages share the Gaussian marginal and Bayes-optimal halfspace, but differ only in their label noise. The boundary-local source stage is not a uniformly easier version of the target, since near the boundary its labels may be even noisier than the target’s. Instead, it resembles careful annotators, whose mistakes concentrate on ambiguous instances, as in the crowdsourcing example above. In contrast, the η \eta η -RCN target stage resembles a pool in which each label, with probability 2 η 2\eta 2 η , comes from an annotator who answers at random, without looking at the instance. Our curriculum learner Algorithm 1 RARE-TAIL DIRECTION ESTIMATOR
Require: Samples S = { ( X i , Y i ) } i = 1 N S = \{(X_i,Y_i)\}_{i=1}^N S = {( X i , Y i ) } i = 1 N with X i ∈ R d X_i \in\mathbb{R}^d X i ∈ R d and Y i ∈ { ± 1 } Y_i \in\{\pm1\} Y i ∈ { ± 1 }
Ensure: A unit vector w ^ ∈ S d − 1 \widehat{w} \in\mathbb{S}^{d-1} w ∈ S d − 1 and empirical rare label y ^ r t \widehat{y}_{\mathrm{rt}} y rt
1: N + ← ∑ i = 1 N 1 { Y i = + 1 } , N − ← ∑ i = 1 N 1 { Y i = − 1 } N_+ \leftarrow\sum_{i=1}^N \mathbf{1}\{Y_i=+1\},\quad N_- \leftarrow\sum_{i=1}^N \mathbf{1}\{Y_i=-1\} N + ← ∑ i = 1 N 1 { Y i = + 1 } , N − ← ∑ i = 1 N 1 { Y i = − 1 } 2: y ^ r t ← − 1 \widehat{y}_{\mathrm{rt}} \leftarrow-1 y rt ← − 1 if N − ≤ N + N_- \leq N_+ N − ≤ N + , and y ^ r t ← + 1 \widehat{y}_{\mathrm{rt}} \leftarrow+1 y rt ← + 1 otherwise 3: I r t ← { i ∈ [ N ] : Y i = y ^ r t } I_{\mathrm{rt}} \leftarrow\{i \in[N] : Y_i=\widehat{y}_{\mathrm{rt}}\} I rt ← { i ∈ [ N ] : Y i = y rt } 4: if I r t = ∅ I_{\mathrm{rt}}=\varnothing I rt = ∅ then 5: return any fixed unit vector and y ^ r t \widehat{y}_{\mathrm{rt}} y rt 6: end if 7: μ ^ r t ← ∣ I r t ∣ − 1 ∑ i ∈ I r t X i \widehat{\mu}_{\mathrm{rt}} \leftarrow|I_{\mathrm{rt}}|^{-1}\sum_{i\in I_{\mathrm{rt}}}X_i μ rt ← ∣ I rt ∣ − 1 ∑ i ∈ I rt X i 8: if μ ^ r t = 0 \widehat{\mu}_{\mathrm{rt}}=0 μ rt = 0 then 9: return any fixed unit vector and y ^ r t \widehat{y}_{\mathrm{rt}} y rt 10: end if 11: w ^ ← y ^ r t μ ^ r t / ∥ μ ^ r t ∥ 2 \widehat{w} \leftarrow\widehat{y}_{\mathrm{rt}}\widehat{\mu}_{\mathrm{rt}}/\|\widehat{\mu}_{\mathrm{rt}}\|_2 w ← y rt μ rt /∥ μ rt ∥ 2 12: return ( w ^ , y ^ r t ) (\widehat{w},\widehat{y}_{\mathrm{rt}}) ( w , y rt )
runs the rare-tail estimator on the source stage to obtain a warm start, and then uses target RCN samples to estimate the threshold and refine the direction locally, following Diakonikolas et al. [20 ] . Since the learner also sees the source stage, the statistical-query lower bound does not apply, even though the final guarantee is measured on the RCN target.
Our Results
We summarize the main guarantees. Throughout, p = Φ ( − t ⋆ ) p=\Phi(-t^\star) p = Φ ( − t ⋆ ) denotes the bias of the halfspace under the convention t ⋆ > 0 t^\star>0 t ⋆ > 0 , and our primary regime is the critical large-bias regime p = Θ ( ε ) p=\Theta(\varepsilon) p = Θ ( ε ) .
Our first result shows that the rare-tail direction estimator obtains the same leading sample scale as in the noiseless benchmark for a broad class of noise profiles that satisfy a mild regularity condition, which we call regular-variation (Definition 2.2 ).
Theorem 1.1 (Informal: rare-tail direction recovery). Under regularly-varying margin-dependent noise, Algorithm 1 returns a direction w ^ \widehat{w} w with ∠ ( w ^ , w ⋆ ) ≤ O ( 1 / t ⋆ ) \angle(\widehat{w},w^\star)\leq O(1/t^\star) ∠ ( w , w ⋆ ) ≤ O ( 1/ t ⋆ ) with high probability using O ~ ( ( d + log ( 1 / δ ) ) / p ) \widetilde{O}((d+\log(1/\delta))/p) O (( d + log ( 1/ δ )) / p ) samples.
The angular scale O ( 1 / t ⋆ ) O(1/t^\star) O ( 1/ t ⋆ ) is exactly the scale needed for excess risk in the critical regime. To estimate the threshold, we use the threshold ERM of Algorithm 2 with the following guarantee:
Theorem 1.2 (Informal: one-dimensional threshold ERM). For any fixed direction w ^ \widehat{w} w , Algorithm 2 uses O ~ ( 1 / ε 2 ) \widetilde{O}(1/\varepsilon^2) O ( 1/ ε 2 ) independent validation samples and returns a threshold t ^ \widehat{t} t whose risk is within ε \varepsilon ε of the best threshold along w ^ \widehat{w} w .
This leads us to the standalone sample-optimal halfspace learning guarantee.
Theorem 1.3 (Informal: standalone halfspace learning). In the critical regime p = Θ ( ε ) p=\Theta(\varepsilon) p = Θ ( ε ) , Algorithm 1 followed by Algorithm 2 returns a halfspace f ^ \widehat{f} f with excess risk ≤ ε \leq\varepsilon ≤ ε , with probability at least 1 − δ 1-\delta 1 − δ , using O ~ ( ( d + log ( 1 / δ ) ) / ε ) + O ~ ( 1 / ε 2 ) \widetilde{O}((d+\log(1/\delta))/\varepsilon)+\widetilde{O}(1/\varepsilon^2) O (( d + log ( 1/ δ )) / ε ) + O ( 1/ ε 2 ) samples, and runs in time O ~ ( d N ) \widetilde{O}(dN) O ( d N ) .
In high-dimensions, the dimension-free threshold-validation term is lower order, so the sample complexity is governed by the rare-tail direction-estimation term. As a concrete specialization of the general theory, we study the probit noise profile. The resulting learning guarantee is in Corollary 3.4.
Application to curriculum learning for RCN. The guarantees above do not cover RCN, since its noise is flat. Next, we use the rare-tail estimator as the warm start of the two-stage curriculum described earlier, to circumvent the statistical-query lower bound of Diakonikolas et al. [20 ] for affine Gaussian halfspaces under RCN.
Theorem 1.4 (Informal: curriculum learning for an RCN target). For p = Ω η ( ε ) p=\Omega_\eta(\varepsilon) p = Ω η ( ε ) , there is a curriculum learning algorithm that learns the target halfspace to excess error ε \varepsilon ε using O ~ ( d / p ) \widetilde{O}(d/p) O ( d / p ) source samples and O ~ η ( ( d / ε ) + 1 / ε 2 ) \widetilde{O}_\eta((d/\varepsilon)+1/\varepsilon^2) O η (( d / ε ) + 1/ ε 2 ) target RCN samples, with high probability.
In high dimensions, the leading target cost thus matches the information-theoretic rate, whereas the best known target-only learner [20 ] uses O ~ ( d / ε + d / max ( p , ε ) 2 ) \widetilde{O}(d/\varepsilon+d/\max(p,\varepsilon)^2) O ( d / ε + d / max ( p , ε ) 2 ) samples.
Technical Preliminaries
Notation. For d ∈ N d \in\mathbb{N} d ∈ N , let I d I_d I d denote the d × d d \times d d × d identity. For u , v ∈ R d u,v \in\mathbb{R}^d u , v ∈ R d , write ⟨ u , v ⟩ \langle u,v\rangle ⟨ u , v ⟩ for the Euclidean inner product and ∥ u ∥ 2 \|u\|_2 ∥ u ∥ 2 for the Euclidean norm. Let S d − 1 = { w ∈ R d : ∥ w ∥ 2 = 1 } S^{d-1}=\{w \in\mathbb{R}^d:\|w\|_2=1\} S d − 1 = { w ∈ R d : ∥ w ∥ 2 = 1 } . For unit vectors u , v ∈ S d − 1 u,v \in S^{d-1} u , v ∈ S d − 1 , write ∠ ( u , v ) = arccos ( ⟨ u , v ⟩ ) ∈ [ 0 , π ] \angle(u,v)=\arccos(\langle u,v\rangle) \in[0,\pi] ∠ ( u , v ) = arccos (⟨ u , v ⟩) ∈ [ 0 , π ] . We write 1 { ⋅ } \mathbf{1}\{\cdot\} 1 { ⋅ } for indicator functions. All logarithms are natural. We use O ~ ( ⋅ ) \widetilde{O}(\cdot) O ( ⋅ ) and Ω ~ ( ⋅ ) \widetilde{\Omega}(\cdot) Ω ( ⋅ ) to suppress polylogarithmic factors in the relevant parameters (typically d , 1 / ε , 1 / δ , t d,1/\varepsilon,1/\delta,t d , 1/ ε , 1/ δ , t ). Throughout, all statements involving P ( ⋅ ∣ X = x ) , E [ ⋅ ∣ X = x ] \mathbb{P}(\cdot\mid X=x),\mathbb{E}[\cdot\mid X=x] P ( ⋅ ∣ X = x ) , E [ ⋅ ∣ X = x ] hold P X P_X P X -almost surely. We write ϕ \phi ϕ and Φ \Phi Φ to denote the standard normal density and CDF. We also use λ ( t ) = ϕ ( t ) / Φ ( − t ) \lambda(t)=\phi(t)/\Phi(-t) λ ( t ) = ϕ ( t ) /Φ ( − t ) for the inverse Mills ratio.
Gaussian halfspace model. Fix an unknown w ⋆ ∈ S d − 1 w^\star\in S^{d-1} w ⋆ ∈ S d − 1 and threshold t ⋆ ∈ R t^\star\in\mathbb{R} t ⋆ ∈ R . We draw X ∼ N ( 0 , I d ) X \sim\mathcal{N}(0,I_d) X ∼ N ( 0 , I d ) and define the signed margin m ( x ) = ⟨ w ⋆ , x ⟩ + t ⋆ m(x)=\langle w^\star,x\rangle+t^\star m ( x ) = ⟨ w ⋆ , x ⟩ + t ⋆ and the corresponding halfspace f ⋆ ( x ) = sign ( m ( x ) ) f^\star(x)=\operatorname{sign}(m(x)) f ⋆ ( x ) = sign ( m ( x )) , where sign ( u ) = + 1 \operatorname{sign}(u)=+1 sign ( u ) = + 1 for u ≥ 0 u \ge0 u ≥ 0 and sign ( u ) = − 1 \operatorname{sign}(u)=-1 sign ( u ) = − 1 otherwise.
Risk and excess risk. For a distribution D \mathcal{D} D over R d × { ± 1 } \mathbb{R}^d \times\{\pm1\} R d × { ± 1 } and a classifier f : R d → { ± 1 } f:\mathbb{R}^d \to\{\pm1\} f : R d → { ± 1 } , define the 0 0 0 –1 1 1 risk
err D ( f ) = P ( X , Y ) ∼ D ( f ( X ) ≠ Y ) \operatorname{err}_{\mathcal{D}}(f)=\mathbb{P}_{(X,Y)\sim\mathcal{D}}(f(X)\ne Y) err D ( f ) = P ( X , Y ) ∼ D ( f ( X ) = Y )
Given a reference classifier f ⋆ f^\star f ⋆ , the excess risk of f f f with respect to f ⋆ f^\star f ⋆ is err D ( f ) − err D ( f ⋆ ) \operatorname{err}_{\mathcal{D}}(f)-\operatorname{err}_{\mathcal{D}}(f^\star) err D ( f ) − err D ( f ⋆ ) . When the distribution is clear from context, we write simply err ( f ) \operatorname{err}(f) err ( f ) . In the high bias regime, the following directional guarantee suffices for an ε \varepsilon ε excess risk guarantee. The proof is deferred to Appendix § 9 .§ 2 .
Lemma 2.1 (Angular error controls excess risk). In the Gaussian halfspace model described above, fix t ⋆ ≥ 1 t^\star\ge1 t ⋆ ≥ 1 , and let w ∈ S d − 1 w \in S^{d-1} w ∈ S d − 1 be arbitrary, with induced halfspace f w ( x ) = sign ( ⟨ w , x ⟩ + t ⋆ ) f_w(x)=\operatorname{sign}(\langle w,x\rangle+t^\star) f w ( x ) = sign (⟨ w , x ⟩ + t ⋆ ) . There exist universal constants c 0 , C 0 > 0 c_0,C_0>0 c 0 , C 0 > 0 such that, whenever θ = ∠ ( w , w ⋆ ) ≤ c 0 / t ⋆ \theta=\angle(w,w^\star)\le c_0/t^\star θ = ∠ ( w , w ⋆ ) ≤ c 0 / t ⋆ , we have
P ( f w ( X ) ≠ f ⋆ ( X ) ) ≤ C 0 θ ϕ ( t ⋆ ) \mathbb{P}(f_w(X)\ne f^\star(X))\le C_0\theta\phi(t^\star) P ( f w ( X ) = f ⋆ ( X )) ≤ C 0 θ ϕ ( t ⋆ ) In particular, in the critical high-bias regime p = Θ ( ε ) p=\Theta(\varepsilon) p = Θ ( ε ) , there is a sufficiently small universal constant c > 0 c>0 c > 0 such that ∠ ( w , w ⋆ ) ≤ c / t ⋆ \angle(w,w^\star)\le c/t^\star ∠ ( w , w ⋆ ) ≤ c / t ⋆ implies err ( f w ) − err ( f ⋆ ) ≤ ε \operatorname{err}(f_w)-\operatorname{err}(f^\star)\le\varepsilon err ( f w ) − err ( f ⋆ ) ≤ ε .
Noise Families
General margin-dependent label noise. Let g : [ 0 , ∞ ) → [ 0 , 1 / 2 ] g:[0,\infty)\to[0,1/2] g : [ 0 , ∞ ) → [ 0 , 1/2 ] be measurable. We say ( X , Y ) (X,Y) ( X , Y ) follows margin-dependent noise with profile g g g if
P ( Y ≠ f ⋆ ( X ) ∣ X = x ) = g ( ∣ m ( x ) ∣ ) \mathbb{P}(Y\ne f^\star(X)\mid X=x)=g(|m(x)|) P ( Y = f ⋆ ( X ) ∣ X = x ) = g ( ∣ m ( x ) ∣ )
If g g g is nonincreasing, we call this margin-monotone noise . For a profile g g g , we write D g \mathcal{D}^g D g for the induced distribution over ( X , Y ) (X,Y) ( X , Y ) under the current target halfspace; in particular, D η \mathcal{D}^\eta D η denotes the η \eta η -RCN distribution, where g ( u ) = η g(u)=\eta g ( u ) = η , ∀ u ∈ R \forall u \in\mathbb{R} ∀ u ∈ R , and η ∈ [ 0 , 1 / 2 ) \eta\in[0,1/2) η ∈ [ 0 , 1/2 ) .
Flat noise profiles. Throughout this paper, when we refer to a noise profile as flat , we mean RCN-like flat profiles where g ( ⋅ ) g(\cdot) g ( ⋅ ) is a positive constant. The degenerate noiseless profile g ( ⋅ ) = 0 g(\cdot)=0 g ( ⋅ ) = 0 is a flat function, but it does not create the obstruction studied here: it introduces no false negatives, and the rare-tail mean estimator achieves its strongest signal in precisely this case.
Probit noise. Fix σ > 0 \sigma>0 σ > 0 and let G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) be independent of X X X . In the probit model,
Y = sign ( ⟨ w ⋆ , X ⟩ + t ⋆ + σ G ) Y=\operatorname{sign}(\langle w^\star,X\rangle+t^\star+\sigma G) Y = sign (⟨ w ⋆ , X ⟩ + t ⋆ + σ G )
This is margin-monotone with g ( u ) = Φ ( − u / σ ) g(u)=\Phi(-u/\sigma) g ( u ) = Φ ( − u / σ ) (with the convention Φ ( − u / 0 ) = 0 \Phi(-u/0)=0 Φ ( − u /0 ) = 0 for u > 0 u>0 u > 0 ).
Definition 2.2 (Regularly varying margin noise). Fix constants c > 0 c>0 c > 0 and C ≥ 0 C\ge0 C ≥ 0 . A margin-monotone noise function g g g has regular margin variation if g g g is absolutely continuous, g ( u ) → 0 g(u)\to0 g ( u ) → 0 as u → ∞ u\to\infty u → ∞ , and for almost every u ≥ 1 u\ge1 u ≥ 1 with g ( u ) > 0 g(u)>0 g ( u ) > 0 ,
− g ′ ( u ) ≥ ( c ( 1 + u ) ( log ( e + u ) ) C ) g ( u ) -g'(u)\ge\left(\frac{c}{(1+u)(\log(e+u))^C}\right)g(u) − g ′ ( u ) ≥ ( ( 1 + u ) ( log ( e + u ) ) C c ) g ( u )
This condition controls the relative decay rate of the noise profile, while placing no restrictions on the noise level near the boundary, where g ( 0 ) g(0) g ( 0 ) may be arbitrarily close, or even equal, to 1 / 2 1/2 1/2 . It says that the flip probability cannot remain essentially flat over large margin scales. On a multiplicative margin scale, such as from u u u to 2 u 2u 2 u , it forces an inverse-polylogarithmic decrease in the current value of g g g . This excludes constant RCN and long flat plateaus, but admits very slowly decaying noise functions, including logarithmic and sub-polynomial tails. Polynomial, exponential, logistic, probit, and sub-Gaussian noise all satisfy the condition.
Overview of Techniques
To establish the core ideas behind our main results, we first formalize the intuition that, in the biased regime, the typical samples are not the most informative ones. It is perhaps most apparent in the absence of noise, where Y = f ⋆ ( X ) Y=f^{\star}(X) Y = f ⋆ ( X ) . For t ⋆ > 0 t^{\star}>0 t ⋆ > 0 , the rare-tail event corresponds to the negative side of the halfspace. When 0 ≤ t ⋆ ≤ 1 0\le t^{\star}\le1 0 ≤ t ⋆ ≤ 1 , this tail has constant mass, so Algorithm 1 only needs O ~ ( d + log ( 1 / δ ) ) \widetilde{O}(d+\log(1/\delta)) O ( d + log ( 1/ δ )) samples. Our results are therefore stated for nontrivial t ⋆ > 1 t^{\star}>1 t ⋆ > 1 .
Noiseless benchmark. The relevant geometry becomes visible in two dimensions. When t ⋆ t^{\star} t ⋆ is large, the separator is shifted away from the origin, so the Gaussian mass is heavily concentrated on one side of the boundary. Thus, a typical sample lies deep in the majority region and carries only weak directional information about the orientation of the halfspace. In contrast, the minority samples come from the thin tail cut off by the separator, and are therefore much more tightly coupled to the true normal.
Figure 1 illustrates this distinction. Both panels show the same set of samples from a two-dimensional Gaussian with the same separator. In (a), the empirical Average vector induces a normal whose angle with w ⋆ w^{\star} w ⋆ is relatively large, and hence the corresponding separator has a noticeable directional error. In (b), we instead use the direction induced by the rare-tail mean, namely, the normalized vector − μ ^ − -\hat{\mu}_{-} − μ ^ − . Its angle to w ⋆ w^{\star} w ⋆ is smaller, as a consequence of Gaussian structure. In d d d dimensions, we repeat the earlier decomposition X = Z w ⋆ + X ⊥ X=Zw^{\star}+X_{\perp} X = Z w ⋆ + X ⊥ . In the absence of noise, the minority event is exactly { Z < − t ⋆ } \{Z<-t^{\star}\} { Z < − t ⋆ } , and conditioning on this event only affects the coordinate Z = ⟨ w ⋆ , x ⟩ Z=\langle w^{\star},x\rangle Z = ⟨ w ⋆ , x ⟩ , with E [ X ∣ Y = − 1 ] = E [ Z ∣ Z < − t ⋆ ] w ⋆ = − ( ϕ ( t ⋆ ) / Φ ( − t ⋆ ) ) w ⋆ E[X\mid Y=-1]=E[Z\mid Z<-t^{\star}]w^{\star}=-(\phi(t^{\star})/\Phi(-t^{\star}))w^{\star} E [ X ∣ Y = − 1 ] = E [ Z ∣ Z < − t ⋆ ] w ⋆ = − ( ϕ ( t ⋆ ) /Φ ( − t ⋆ )) w ⋆ , and the population minority mean is therefore collinear with w ⋆ w^{\star} w ⋆ . Moreover, the norm is the inverse Mills ratio, which is of the order t ⋆ t^{\star} t ⋆ in the large bias regime, so conditioning on the rare tail event actually amplifies the signal in the w ⋆ w^{\star} w ⋆ direction by a factor of Θ ( t ⋆ ) \Theta(t^{\star}) Θ ( t ⋆ ) .
The Average estimator enjoys the same symmetry, but not the same signal strength. Indeed, if we write μ a v g = E [ Y X ] \mu_{\mathrm{avg}}=E[Y X] μ avg = E [ Y X ] , then μ a v g = − 2 E [ X 1 { Y = − 1 } ] = − 2 P ( Y = − 1 ) E [ X ∣ Y = − 1 ] = − 2 p μ − \mu_{\mathrm{avg}}=-2E[X1\{Y=-1\}]=-2\mathbb{P}(Y=-1)E[X\mid Y=-1]=-2p\mu_{-} μ avg = − 2 E [ X 1 { Y = − 1 }] = − 2 P ( Y = − 1 ) E [ X ∣ Y = − 1 ] = − 2 p μ − , i.e., the Average vector has the same direction as the minority mean by a factor of 2 p 2p 2 p . Both estimators incur orthogonal noise of typical size Θ ( d / N ) \Theta(\sqrt{d/N}) Θ ( d / N ) for Average and Θ ( d / ( N p ) ) \Theta(\sqrt{d/(Np)}) Θ ( d / ( N p ) ) for the minority mean. Consequently, at the same sample budget N = O ~ ( d / p ) N=\widetilde{O}(d/p) N = O ( d / p ) , the minority mean typically reaches O ( 1 / t ⋆ ) O(1/t^{\star}) O ( 1/ t ⋆ ) -scale angular error, while Average remains at the larger O ( 1 / ( t ⋆ p ) ) O(1/(t^{\star}\sqrt{p})) O ( 1/ ( t ⋆ p )) scale.
As it turns out, what matters for this benchmark is not exact noiselessness, but that the observed rare class remains tail-like, i.e. p − = O ( p ) p_{-}=O(p) p − = O ( p ) . Under η \eta η -RCN, p − = p + ( 1 − 2 η ) p p_{-}=p+(1-2\eta)p p − = p + ( 1 − 2 η ) p , so this condition holds whenever η = O ( p ) \eta=O(p) η = O ( p ) . The rare-tail mean therefore retains the O ~ ( d / p ) \widetilde{O}(d/p) O ( d / p ) rate even, after exact linear separability is destroyed. This contrasts with separability-based methods, which rely on the absence of label flips. By Lemma 2.1, two Gaussian halfspaces, sharing the same threshold, with normals at angle θ \theta θ , disagree on at most O ( θ t ⋆ ) O(\theta t^{\star}) O ( θ t ⋆ ) mass. Therefore, when p = Φ ( − t ⋆ ) = Θ ( ε ) p=\Phi(-t^{\star})=\Theta(\varepsilon) p = Φ ( − t ⋆ ) = Θ ( ε ) , taking θ = O ( 1 / t ⋆ ) \theta=O(1/t^{\star}) θ = O ( 1/ t ⋆ ) gives O ( p ) = O ( ε ) O(p)=O(\varepsilon) O ( p ) = O ( ε ) disagreement. Since excess risk under our noise models is bounded by disagreement with the Bayes halfspace, a correct threshold along such a direction suffices for ε \varepsilon ε excess risk.
Robustness beyond η = O ( p ) \eta=O(p) η = O ( p ) . The signal amplification in the benchmark above extends to a broad and classical family of margin-dependent corruption models, in which the label flip probability depends only on the distance to the decision boundary. Consider the regularly-varying margin noise profiles of Definition 2.2. Under this weak condition, the same one-pass estimator achieves the O ( 1 / t ⋆ ) O(1/t^{\star}) O ( 1/ t ⋆ ) -scale angular error with the same sample complexity as in the noiseless regime.
Before stating the result, we note a small notational mismatch between the algorithm and the analysis. The proof uses the convention t ⋆ > 0 t^{\star}>0 t ⋆ > 0 and is written for the conditional mean μ − = E [ X ∣ Y = \mu_{-}=E[X\mid Y= μ − = E [ X ∣ Y = −1, while Algorithm 1 first identifies the empirical rare label and then orients the corresponding conditional mean. In practice, this is how one would go about implementing this estimator, since the rare label cannot be known apriori, but this difference in notation does not affect the stated sample-complexity bounds. In particular, if p − = P ( Y = − 1 ) p_{-} = \mathbb{P}(Y = -1) p − = P ( Y = − 1 ) is small, the empirical rare label is − 1 -1 − 1 with high probability, but if p − p_{-} p − is instead bounded below by a constant, either label-conditional mean gives the same population direction up to a constant-factor in the sample complexity, since E [ X ] = 0 \mathbb{E}[X] = 0 E [ X ] = 0 implies p − μ − + p + μ + = 0 p_{-}\mu_{-} + p_{+}\mu_{+} = 0 p − μ − + p + μ + = 0 . The formal reconciliation is discussed in Appendix § 9 .11.
Theorem 3.1 (Direction recovery under regular variation). Assume t ⋆ ≥ 1 t^{\star} \ge1 t ⋆ ≥ 1 , and let p = Φ ( − t ⋆ ) p = \Phi(-t^{\star}) p = Φ ( − t ⋆ ) , p − = P ( Y = − 1 ) p_{-} = \mathbb{P}(Y = -1) p − = P ( Y = − 1 ) . If the labels follow a ( c , C ) (c, C) ( c , C ) -regularly varying margin-monotone noise profile then there is a sufficiently small universal constant c d i r > 0 c_{\mathrm{dir}} > 0 c dir > 0 such that Algorithm 1 returns a direction w ^ \widehat{w} w satisfying ∠ ( w ^ , w ⋆ ) ≤ c d i r / t ⋆ \angle(\widehat{w}, w^{\star}) \le c_{\mathrm{dir}}/t^{\star} ∠ ( w , w ⋆ ) ≤ c dir / t ⋆ with probability at least 1 − δ 1 - \delta 1 − δ , using N d i r N_{\mathrm{dir}} N dir samples, where
N d i r = O ~ c , C ( ( d + log ( 1 / δ ) ) max { 1 p , ( t ⋆ ) 4 ( log ( e + t ⋆ ) ) 2 C p − } ) N_{\mathrm{dir}} = \widetilde{O}_{c,C}\left((d + \log(1/\delta)) \max\left\{\frac{1}{p}, \frac{(t^{\star})^{4}(\log(\mathrm{e} + t^{\star}))^{2C}}{p_{-}}\right\}\right) N dir = O c , C ( ( d + log ( 1/ δ )) max { p 1 , p − ( t ⋆ ) 4 ( log ( e + t ⋆ ) ) 2 C } )
Proof sketch. Since this result is for direction recovery, we set t = t ⋆ t = t^{\star} t = t ⋆ for readability. Let τ \tau τ denote the target angle. Since the label depends on X X X only through the margin coordinate Z + t Z + t Z + t , the population conditional mean μ − \mu_{-} μ − is of the form
μ − = E [ X ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 ] w ⋆ , \mu_{-} = \mathbb{E}[X \mid Y = -1] = \mathbb{E}[Z \mid Y = -1]w^{\star}, μ − = E [ X ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 ] w ⋆ ,
and is therefore aligned with the target w ⋆ w^{\star} w ⋆ . It remains to lower bound the norm of this conditional mean for empirical concentration. In Theorem B.13, we show that the empirical rare-tail mean is within angle τ \tau τ of w ⋆ w^{\star} w ⋆ once there are O ~ ( ( d + log ( 1 / δ ) ) / ( ∥ μ − ∥ 2 2 sin 2 ( τ / 2 ) ) ) \widetilde{O}((d + \log(1/\delta))/(\lVert\mu_{-}\rVert_{2}^{2}\sin^{2}(\tau/2))) O (( d + log ( 1/ δ )) / (∥ μ − ∥ 2 2 sin 2 ( τ /2 ))) observed negatives, where the logarithmic factor comes from conditioning on a rare label. Since N − ≥ p − N / 2 N_{-} \ge p_{-}N/2 N − ≥ p − N /2 with high probability and τ = c d i r / t \tau= c_{\mathrm{dir}}/t τ = c dir / t , it suffices that
N = O ~ ( t 2 ( d + log ( 1 / δ ) ) p − ∥ μ − ∥ 2 2 ) (1) N = \widetilde{O}\left(\frac{t^{2}(d + \log(1/\delta))}{p_{-}\lVert\mu_{-}\rVert_{2}^{2}}\right) \tag*{(1)} N = O ( p − ∥ μ − ∥ 2 2 t 2 ( d + log ( 1/ δ )) ) (1)
Let q ( z ) = P ( Y = − 1 ∣ Z = z ) q(z) = \mathbb{P}(Y = -1 \mid Z = z) q ( z ) = P ( Y = − 1 ∣ Z = z ) and U = ∣ Z + t ∣ U = \lvert Z + t\rvert U = ∣ Z + t ∣ . Since g g g is nonincreasing and g ≤ 1 / 2 g \le1/2 g ≤ 1/2 , q q q is nonincreasing, with a downward jump of 1 − 2 g ( 0 ) 1 - 2g(0) 1 − 2 g ( 0 ) at z = − t z = -t z = − t and derivative q ′ ( z ) = g ′ ( ∣ z + t ∣ ) q'(z) = g'(\lvert z + t\rvert) q ′ ( z ) = g ′ (∣ z + t ∣) for almost every z ≠ − t z \ne-t z = − t . Gaussian integration by parts applied to p − ∥ μ − ∥ 2 2 = − E [ Z q ( Z ) ] p_{-}\lVert\mu_{-}\rVert_{2}^{2} = -\mathbb{E}[Zq(Z)] p − ∥ μ − ∥ 2 2 = − E [ Z q ( Z )] then gives the following signal identity that captures the intuition as to why non-flat profiles amplify the signal
p − ∥ μ − ∥ 2 2 = ( 1 − 2 g ( 0 ) ) ϕ ( t ) + E [ − g ′ ( U ) ] . (2) p_{-}\lVert\mu_{-}\rVert_{2}^{2} = (1 - 2g(0))\phi(t) + \mathbb{E}[-g'(U)]. \tag*{(2)} p − ∥ μ − ∥ 2 2 = ( 1 − 2 g ( 0 )) ϕ ( t ) + E [ − g ′ ( U )] . (2)
Notice that the first term is due to the jump of q q q at the boundary, which is the only part of the signal that survives in flat profiles, and is at most of order t p tp tp . The second term, involving the expected derivative, captures signal variation away from the boundary. This additional signal is precisely the edge that boundary-local profiles have over flat profiles. To facilitate a closed-form analysis of this term, we will use the regular variation condition from Definition 2.2, although Eq. (2) is more general and applies to all absolutely continuous margin-monotone profiles for this problem.
The observed negatives come from two groups, namely samples from the clean tail { Z < − t } \{Z < -t\} { Z < − t } , and false negatives from { Z ≥ − t } \{Z \ge-t\} { Z ≥ − t } . Since g ≤ 1 / 2 g \le1/2 g ≤ 1/2 , every clean sample keeps its label − 1 -1 − 1 with probability at least 1 / 2 1/2 1/2 , so the first group alone has probability at least p / 2 p/2 p /2 , and p − ≥ p / 2 p_{-} \ge p/2 p − ≥ p /2 always.
We first consider the tail-like regime p − ≤ 2 p p_{-} \le2p p − ≤ 2 p , where the observed negatives have a distribution closer to the true negatives. The first group then makes up at least a p / 2 p − ≥ 1 / 4 p/2p_{-} \ge1/4 p /2 p − ≥ 1/4 fraction of the observed negatives, and all of these lie at Z < − t Z < -t Z < − t . The false negatives make up the rest, but because g g g is nonincreasing, flips are more likely close to the boundary, and false negatives have mean at most E [ Z ∣ Z ≥ − t ] ≤ 3 / 10 \mathbb{E}[Z \mid Z \ge-t] \le3/10 E [ Z ∣ Z ≥ − t ] ≤ 3/10 . Together, this gives ∥ μ − ∥ 2 = − E [ Z ∣ Y = − 1 ] ≥ t / 4 − 9 / 40 ≥ t / 40 \lVert\mu_{-}\rVert_{2} = -\mathbb{E}[Z \mid Y = -1] \ge t/4 - 9/40 \ge t/40 ∥ μ − ∥ 2 = − E [ Z ∣ Y = − 1 ] ≥ t /4 − 9/40 ≥ t /40 , and since p − ≥ p / 2 p_{-} \ge p/2 p − ≥ p /2 , applying (1) gives the first term for N d i r N_{\mathrm{dir}} N dir . For this tail-like regime, regular variation is unnecessary.
The main technical difficulty is in the inflated regime p − > 2 p p_{-} > 2p p − > 2 p , where false negatives make up more than half of the observed negatives. They increase p − p_{-} p − without adding to the boundary-jump term, which on its own only gives ∥ μ − ∥ 2 ≥ ( 1 − 2 g ( 0 ) ) ϕ ( t ) / p − \lVert\mu_{-}\rVert_{2} \ge(1 - 2g(0))\phi(t)/p_{-} ∥ μ − ∥ 2 ≥ ( 1 − 2 g ( 0 )) ϕ ( t ) / p − , of order ( 1 − 2 g ( 0 ) ) t p / p − (1 - 2g(0))tp/p_{-} ( 1 − 2 g ( 0 )) tp / p − . This degrades as p − / p p_{-}/p p − / p grows, as in η \eta η -RCN where p − ≈ η p_{-} \approx\eta p − ≈ η , and vanishes when g ( 0 ) = 1 / 2 g(0) = 1/2 g ( 0 ) = 1/2 , as for probit noise. The boundary-jump term alone need not provide enough signal, and we shift focus to the variation term instead. For a suitable constant L L L , Definition 2.2 gives − g ′ ( u ) ≥ c 1 g ( u ) / ( t ( log ( e + t ) ) C ) -g'(u) \ge c_{1}g(u)/(t(\log(\mathrm{e} + t))^{C}) − g ′ ( u ) ≥ c 1 g ( u ) / ( t ( log ( e + t ) ) C ) for almost every u ∈ [ 1 , L t ] u \in[1,Lt] u ∈ [ 1 , L t ] , where c 1 = c 1 ( c , C ) > 0 c_1=c_1(c,C)>0 c 1 = c 1 ( c , C ) > 0 . In other words, on margins up to order t t t , the flip probability decays at a relative rate of at least 1 / t 1/t 1/ t , up to logarithmic factors. Since the condition only constrains margins u ≥ 1 u \ge1 u ≥ 1 , we split according to where these flips happen.
Suppose first that a constant fraction of the flips happen away from the boundary, say E [ g ( U ) 1 { U ≥ 1 } ] ≥ p − / 4 \mathbb{E}[g(U)1\{U\ge1\}]\ge p_-/4 E [ g ( U ) 1 { U ≥ 1 }] ≥ p − /4 . For a suitable universal constant L L L , P ( U > L t ) ≤ p / 2 \mathbb{P}(U>Lt)\le p/2 P ( U > L t ) ≤ p /2 . Since g ≤ 1 / 2 g\le1/2 g ≤ 1/2 ,
E [ g ( U ) 1 { U > L t } ] ≤ 1 2 P ( U > L t ) ≤ p 4 < p − 8 \mathbb{E}[g(U)1\{U>Lt\}]\le\frac{1}{2}\mathbb{P}(U>Lt)\le\frac{p}{4}<\frac{p_-}{8} E [ g ( U ) 1 { U > L t }] ≤ 2 1 P ( U > L t ) ≤ 4 p < 8 p −
and at least half of these flips lie in the window [ 1 , L t ] [1,Lt] [ 1 , L t ] . Consequently,
E [ − g ′ ( U ) ] ≥ c 1 E [ g ( U ) 1 { 1 ≤ U ≤ L t } ] t ( log ( e + t ) ) C ≥ c 1 p − 8 t ( log ( e + t ) ) C . \mathbb{E}[-g'(U)]\ge\frac{c_1\mathbb{E}[g(U)1\{1\le U\le Lt\}]}{t(\log(e+t))^C}\ge\frac{c_1p_-}{8t(\log(e+t))^C}. E [ − g ′ ( U )] ≥ t ( log ( e + t ) ) C c 1 E [ g ( U ) 1 { 1 ≤ U ≤ L t }] ≥ 8 t ( log ( e + t ) ) C c 1 p − .
Since the boundary-jump term is nonnegative, eq. (2) gives ∥ μ − ∥ 2 = Ω c , C ( 1 / ( t ( log ( e + t ) ) C ) ) \|\mu_-\|_2=\Omega_{c,C}(1/(t(\log(e+t))^C)) ∥ μ − ∥ 2 = Ω c , C ( 1/ ( t ( log ( e + t ) ) C )) , and applying (1) gives the second term of N d i r N_{\rm dir} N dir .
The other possibility is that E [ g ( U ) 1 { U ≥ 1 } ] < p − / 4 \mathbb{E}[g(U)1\{U\ge1\}]<p_-/4 E [ g ( U ) 1 { U ≥ 1 }] < p − /4 , so most flips happen within distance 1 1 1 of the boundary. The observed negatives with Z > 1 − t Z>1-t Z > 1 − t are flips at margin larger than 1 1 1 , so they have probability less than p − / 4 p_-/4 p − /4 , and at least three quarters of the observed negatives lie at Z ≤ 1 − t ≤ 0 Z\le1-t\le0 Z ≤ 1 − t ≤ 0 . Set b = 1 − t b=1-t b = 1 − t . On each side of b b b , conditioning on Y = − 1 Y=-1 Y = − 1 reweights the truncated Gaussian by the nonincreasing function q q q . Then the two conditional means are at most E [ Z ∣ Z ≤ b ] ≤ − 2 / π \mathbb{E}[Z\mid Z\le b]\le-\sqrt{2/\pi} E [ Z ∣ Z ≤ b ] ≤ − 2/ π and E [ Z ∣ Z > b ] ≤ 2 / π \mathbb{E}[Z\mid Z>b]\le\sqrt{2/\pi} E [ Z ∣ Z > b ] ≤ 2/ π , respectively. Combining their contributions gives E [ Z ∣ Y = − 1 ] ≤ − 1 / 2 2 / π \mathbb{E}[Z\mid Y=-1]\le-1/2\sqrt{2/\pi} E [ Z ∣ Y = − 1 ] ≤ − 1/2 2/ π . Therefore, ∥ μ − ∥ 2 = Ω ( 1 ) \|\mu_-\|_2=\Omega(1) ∥ μ − ∥ 2 = Ω ( 1 ) , and (1) requires only O ~ ( ( d + log ( 1 / δ ) ) t 2 / p − ) \widetilde{O}((d+\log(1/\delta))t^2/p_-) O (( d + log ( 1/ δ )) t 2 / p − ) samples, which is weaker than the second term of N d i r N_{\rm dir} N dir .
Combining the three cases proves the theorem. Finally, since p − ≥ p / 2 p_-\ge p/2 p − ≥ p /2 and t 2 ≤ 2 log ( 1 / p ) t^2\le2\log(1/p) t 2 ≤ 2 log ( 1/ p ) , the bound simplifies to O ~ c , C ( ( d + log ( 1 / δ ) ) / p ) \widetilde{O}_{c,C}((d+\log(1/\delta))/p) O c , C (( d + log ( 1/ δ )) / p ) after suppressing polylogarithmic factors in 1 / p 1/p 1/ p , which then matches the noiseless rate. □
Threshold estimation. Once the direction is accurate, estimating the threshold is a one-dimensional problem. We use Algorithm 2, which projects the validation samples onto the recovered direction, sorts the projected values, and scans the induced threshold intervals.
Theorem 3.2 (Threshold ERM along a fixed direction). Fix a unit vector w ^ \widehat{w} w independent of a validation sample of size k k k , and let t ^ \widehat{t} t be the threshold returned by Algorithm 2. For any ε , δ ∈ ( 0 , 1 ) \varepsilon,\delta\in(0,1) ε , δ ∈ ( 0 , 1 ) , with k = O ~ ( ε − 2 log ( 1 / δ ) ) k=\widetilde{O}(\varepsilon^{-2}\log(1/\delta)) k = O ( ε − 2 log ( 1/ δ )) ,
err sign ( ⟨ w ^ , X ⟩ + t ^ ) ≤ inf b ∈ R err ( sign ( ⟨ w ^ , X ⟩ + b ) ) + ε \operatorname{err}_{\operatorname{sign}(\langle\widehat{w},X\rangle+\widehat{t})}\le\inf_{b\in\mathbb{R}}\operatorname{err}(\operatorname{sign}(\langle\widehat{w},X\rangle+b))+\varepsilon err sign (⟨ w , X ⟩ + t ) ≤ b ∈ R inf err ( sign (⟨ w , X ⟩ + b )) + ε with probability at least 1 − δ 1-\delta 1 − δ . Moreover, Algorithm 2 runs in time O ( d k + k log k ) O(dk+k\log k) O ( d k + k log k ) .
The proof uses the standard VC bound for thresholds on the real line. For the final learner, we apply Theorems 3.2 and 3.1 with accuracy parameter set to a sufficiently small constant multiple of the target ε \varepsilon ε , and study the failure probability via a union bound. The full proof is in Appendix B.12.
Theorem 3.3 (Halfspace learning under regular variation). Assume the setting of Theorem 3.1, and suppose p = Φ ( − t ⋆ ) = Θ ( ε ) p=\Phi(-t^\star)=\Theta(\varepsilon) p = Φ ( − t ⋆ ) = Θ ( ε ) . There is an algorithm that, given
N = O ~ c , C ( ( d + log ( 1 / δ ) ) max { 1 p , ( t ⋆ ) 4 ( log ( e + t ⋆ ) ) 2 C p − } + 1 ε 2 ) N=\widetilde{O}_{c,C}\left((d+\log(1/\delta))\max\left\{\frac{1}{p},\frac{(t^\star)^4(\log(e+t^\star))^{2C}}{p_-}\right\}+\frac{1}{\varepsilon^2}\right) N = O c , C ( ( d + log ( 1/ δ )) max { p 1 , p − ( t ⋆ ) 4 ( log ( e + t ⋆ ) ) 2 C } + ε 2 1 ) independent labeled samples, returns a halfspace f ^ = ( w ^ , t ^ ) \widehat{f}=(\widehat{w},\widehat{t}) f = ( w , t ) such that err ( f ^ ) − err ( f ⋆ ) ≤ ε \operatorname{err}(\widehat{f})-\operatorname{err}(f^\star)\le\varepsilon err ( f ) − err ( f ⋆ ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ . The algorithm runs in time O ~ ( d N ) \widetilde{O}(dN) O ( d N ) .
A canonical specialization: probit noise. The probit model is a special case of regularly-varying margin-dependent noise profiles, and the following learning guarantee is almost immediate from the proof of Theorem 3.1. The proof is deferred to Appendix C.
Corollary 3.4 (Excess-risk guarantee under probit noise). Assume the probit model with σ > 0 \sigma>0 σ > 0 and t ⋆ ≥ 1 t^\star\ge1 t ⋆ ≥ 1 . Let p = Θ ( ε ) p=\Theta(\varepsilon) p = Θ ( ε ) , p σ = Φ ( − t ⋆ / 1 + σ 2 ) p_\sigma=\Phi(-t^\star/\sqrt{1+\sigma^2}) p σ = Φ ( − t ⋆ / 1 + σ 2 ) . There is an algorithm that, given
N = O ~ ( ( d + log ( 1 / δ ) ) ( 1 + σ 2 ) 2 p σ + 1 ε 2 ) N=\widetilde{O}\left(\frac{(d+\log(1/\delta))(1+\sigma^2)^2}{p_\sigma}+\frac{1}{\varepsilon^2}\right) N = O ( p σ ( d + log ( 1/ δ )) ( 1 + σ 2 ) 2 + ε 2 1 ) independent probit samples, returns a halfspace f ^ \widehat{f} f such that err ( f ^ ) − err ( f ⋆ ) ≤ ε \operatorname{err}(\widehat{f})-\operatorname{err}(f^\star)\le\varepsilon err ( f ) − err ( f ⋆ ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ . The algorithm runs in time O ~ ( d N ) \widetilde{O}(dN) O ( d N ) .
Applications to Curriculum Learning
The guarantees of § 3 need the noise to decay away from the boundary, so they do not cover flat profiles such as η \eta η -RCN [2 ] . For biased Gaussian halfspaces, this is also where efficient learning appears to be expensive, as both the best known efficient learner and the statistical-query lower bound of Diakonikolas et al. [20 ] depend quadratically on 1 / p 1/p 1/ p . We first identify where this cost comes from, and then show that one additional stage of samples with boundary-local noise removes it.
The cost. Under η \eta η -RCN with t ⋆ > 0 t^\star> 0 t ⋆ > 0 , the label − 1 -1 − 1 is observed with probability 1 − η 1-\eta 1 − η on the tail side of the separator and η \eta η on the other, so the label distribution jumps at the separator and is constant elsewhere. A first-moment statistic, such as the rare-tail mean or the Average vector E [ Y X ] \mathbb{E}[Y X] E [ Y X ] , therefore sees the direction of w ⋆ w^\star w ⋆ only through this jump, weighted by the Gaussian density at the separator, ϕ ( t ⋆ ) ≈ t ⋆ p \phi(t^\star) \approx t^\star p ϕ ( t ⋆ ) ≈ t ⋆ p . This is the boundary-jump term ( 1 − 2 η ) ϕ ( t ⋆ ) (1-2\eta)\phi(t^\star) ( 1 − 2 η ) ϕ ( t ⋆ ) of eq. (2) , which is all that survives under flat noise, and since it is of order t ⋆ p t^\star p t ⋆ p , such statistics need on the order of d / p 2 d/p^2 d / p 2 samples to reach angle O ( 1 / t ⋆ ) O(1/t^\star) O ( 1/ t ⋆ ) . The best known efficient learner pays the same price in its warm start. Diakonikolas et al. [20 ] initialize their algorithm with the Average vector, and this accounts for the only quadratic term in their sample complexity O ~ ( d / ε + d / max { p , ε 2 } ) \widetilde{O}(d/\varepsilon+ d/\max\{p,\varepsilon^2\}) O ( d / ε + d / max { p , ε 2 }) , since once a direction within angle 1 / ( 5 t ⋆ ) 1/(5t^\star) 1/ ( 5 t ⋆ ) is available, the remaining steps use the information-theoretically optimal O ~ ( d / ε ) \widetilde{O}(d/\varepsilon) O ( d / ε ) samples [30 ] . Their lower bound suggests that efficient methods cannot avoid the quadratic dependence on 1 / p 1/p 1/ p with RCN samples alone. Efficient statistical-query (SQ) algorithms [29 ] , which access the data only through averages of bounded functions, as Algorithm 1 and gradient methods do, need Ω ~ ( d / max { p , ε 2 } ) \widetilde{\Omega}(\sqrt{d}/\max\{p,\varepsilon^2\}) Ω ( d / max { p , ε 2 }) samples, and so do low-degree polynomial tests. Since the later steps are cheap, whenever this lower bound exceeds d / ε d/\varepsilon d / ε , it is in effect a lower bound on finding the warm start from RCN samples.
A simple curriculum. If no average of RCN samples reveals the direction at the information-theoretic rate, we can instead compute the same average on samples from a slightly different distribution. Cornacchia and Mossel [12 ] prove an analogous separation for parities, where a curriculum that starts from a biased product distribution lets a two-layer network learn k k k -parities at poly ( d ) \operatorname{poly}(d) poly ( d ) cost, while noisy-gradient training on the uniform distribution alone requires d Ω ( k ) d^{\Omega(k)} d Ω ( k ) samples. Their curriculum changes the input distribution between the stages, but ours does not need to. Our source stage keeps the Gaussian marginal and the Bayes halfspace, and differs from the target only in its noise profile, which needs to decay away from the boundary (Definition 2.2 ). This is the least intrusive way to add information, since it requires no control over which instances are labeled, only over who labels them. Careful human annotators, whose mistakes concentrate on ambiguous instances [36 , 6 ] , are a natural source of such labels.
Definition 4.1 (Two-stage curriculum learning for Gaussian halfspaces). Fix stage lengths ( n 1 , n 2 ) (n_1,n_2) ( n 1 , n 2 ) , a source profile g 1 : [ 0 , ∞ ) → [ 0 , 1 / 2 ] g_1:[0,\infty)\to[0,1/2] g 1 : [ 0 , ∞ ) → [ 0 , 1/2 ] , and a target noise rate η < 1 / 2 \eta<1/2 η < 1/2 . Nature fixes an unknown target halfspace f ⋆ = f w ⋆ , t ⋆ f^\star=f_{w^\star,t^\star} f ⋆ = f w ⋆ , t ⋆ , and defines
D 1 = D w ⋆ , t ⋆ g 1 , D 2 = D w ⋆ , t ⋆ η D_1=D^{g_1}_{w^\star,t^\star}, \qquad D_2=D^\eta_{w^\star,t^\star} D 1 = D w ⋆ , t ⋆ g 1 , D 2 = D w ⋆ , t ⋆ η Both stages share the same Gaussian marginal and Bayes halfspace, but have different label noise profiles. The learner first receives source samples S 1 ∼ D 1 n 1 S_1\sim D_1^{n_1} S 1 ∼ D 1 n 1 and computes an intermediate state M 1 M_1 M 1 , such as an initialization, before the target samples are revealed. It then receives independent target samples S 2 ∼ D 2 n 2 S_2\sim D_2^{n_2} S 2 ∼ D 2 n 2 and, using M 1 M_1 M 1 and S 2 S_2 S 2 , outputs a classifier f ^ : R d → { ± 1 } \hat f:\mathbb{R}^d\to\{\pm1\} f ^ : R d → { ± 1 } . The target-stage procedure accesses the source samples only through M 1 M_1 M 1 . The goal is to ensure the excess risk with respect to D 2 D_2 D 2 satisfies err 2 ( f ^ ) − err 2 ( f ⋆ ) ≤ ε \operatorname{err}_2(\hat f)-\operatorname{err}_2(f^\star)\le\varepsilon err 2 ( f ^ ) − err 2 ( f ⋆ ) ≤ ε .
Theorem 4.2 (Two-stage curriculum guarantee for an η \eta η -RCN target). Assume the two-stage model of Definition 4.1. The source stage satisfies the assumptions of Theorem 3.1 with p ( 1 ) = P D 1 ( Y = − 1 ) p^{(1)}=\mathbb{P}_{D_1}(Y=-1) p ( 1 ) = P D 1 ( Y = − 1 ) , and the target stage is η \eta η -RCN with known constant η < 1 / 2 \eta<1/2 η < 1/2 . For t ⋆ ≥ 1 t^\star\ge1 t ⋆ ≥ 1 and p = Φ ( − t ⋆ ) ≥ C η ε p=\Phi(-t^\star)\ge C_\eta\varepsilon p = Φ ( − t ⋆ ) ≥ C η ε , if
n 1 = O ~ c , C ( ( d + log ( 1 / δ ) ) max { 1 p , ( t ⋆ ) 4 ( log ( e + t ⋆ ) ) 2 C ( p ( 1 ) ) 2 } ) n_1=\widetilde{O}_{c,C}\left((d+\log(1/\delta))\max\left\{\frac{1}{p},\frac{(t^\star)^4(\log(e+t^\star))^{2C}}{(p^{(1)})^2}\right\}\right) n 1 = O c , C ( ( d + log ( 1/ δ )) max { p 1 , ( p ( 1 ) ) 2 ( t ⋆ ) 4 ( log ( e + t ⋆ ) ) 2 C } ) source samples and
n 2 = O ~ η ( d + log ( 1 / δ ) ε + 1 ε 2 ) n_2=\widetilde{O}_\eta\left(\frac{d+\log(1/\delta)}{\varepsilon}+\frac{1}{\varepsilon^2}\right) n 2 = O η ( ε d + log ( 1/ δ ) + ε 2 1 )
target RCN samples are used, then Algorithm 5 returns a halfspace f ^ \hat{f} f ^ satisfying err D 2 ( f ^ ) − err D 2 ( f ⋆ ) ≤ ε \operatorname{err}_{D_2}(\hat{f})-\operatorname{err}_{D_2}(f^\star)\le\varepsilon err D 2 ( f ^ ) − err D 2 ( f ⋆ ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ . The runtime is O ~ η ( d ( n 1 + n 2 ) ) \widetilde{O}_{\eta}(d(n_1+n_2)) O η ( d ( n 1 + n 2 )) .
Remark 4.3 (Comparison with target-only learning). Table 1 compares Theorem 4.2 with learning from target samples alone. When d ≳ 1 / ε d\gtrsim1/\varepsilon d ≳ 1/ ε , the curriculum learner uses O ~ ( d / ε ) \widetilde{O}(d/\varepsilon) O ( d / ε ) samples in every nontrivial bias regime, which is the information-theoretic rate. This does not contradict the lower bound of Diakonikolas et al. [20 ] , since the warm start now comes from source samples, and with a flat source the quadratic cost would return. When p > ε p>\sqrt{\varepsilon} p > ε , the target-only algorithm is already sample-optimal, and the gain is in runtime. We defer proofs to Appendix § 11 .
Limitations and Open Questions
Like much of the algorithmic literature on learning halfspaces with label noise, our results assume Gaussian covariates. In particular, this is the setting of the η \eta η -RCN baseline and of its SQ lower bound [20 ] , and, to our knowledge, the only one in which affine halfspaces with label noise are known to be learnable to excess risk ε \varepsilon ε at near-optimal rates in polynomial time (see Appendix § 8 .§ 1 ). Nonetheless, our analysis is not particularly tied to Gaussians: the alignment of the minority mean with w ⋆ w^\star w ⋆ only uses rotational symmetry about w ⋆ w^\star w ⋆ , and the signal identity of eq. (2) keeps its form for any such law, with ϕ \phi ϕ replaced by Ψ ( z ) = ∫ z ∞ u ρ ( u ) d u \Psi(z)=\int_z^\infty u\rho(u)\,du Ψ ( z ) = ∫ z ∞ u ρ ( u ) d u , where ρ \rho ρ is the density of ⟨ w ⋆ , X ⟩ \langle w^\star,X\rangle ⟨ w ⋆ , X ⟩ . The assumption appears in the tail and concentration bounds, and is a strict requirement only for the target-stage requirement of § 4 , where it is inherited from [20 ] . We therefore expect the direction-recovery guarantees to carry over to rotationally invariant laws with comparable tails. Rotational symmetry, however, is harder to remove. For an isotropic law, E [ X ∣ ⟨ w , X ⟩ ] \mathbb{E}[X\mid\langle w,X\rangle] E [ X ∣ ⟨ w , X ⟩] is parallel to w w w for every direction w w w only if the law is spherically symmetric [23 ] . On other isotropic marginals, including log-concave ones, the minority mean can point away from w ⋆ w^\star w ⋆ , and the estimator would need some form of correction.
Consistent with the literature discussed in Appendix § 8 .§ 1 , our curriculum result assumes that the source samples arrive in a separate stage before the target RCN samples, and it does not cover the case when a single pool of samples is provided to the learner, but of mixed annotation quality. A pool with an α \alpha α fraction of source samples has the margin-monotone profile α q 1 + ( 1 − α ) η \alpha q_1+(1-\alpha)\eta α q 1 + ( 1 − α ) η , and since the boundary-jump term of eq. (2) is at most of order t ⋆ p t^\star p t ⋆ p , Algorithm 1 run directly on the pool still gains from the variation term α E [ − g ^ 1 ′ ( U ) ] \alpha\mathbb{E}[-\widehat{g}_1'(U)] α E [ − g 1 ′ ( U )] whenever the source noise varies with the margin. Perhaps surprisingly, this would fail if the source samples were noiseless, since the pool is then RCN with the smaller rate ( 1 − α ) η (1-\alpha)\eta ( 1 − α ) η , and unless this rate is O ( p ) O(p) O ( p ) , the clean source samples have to be identified first. When each instance is labeled several times by known annotators, as is common in crowdsourcing, spammer scores that detect annotators who label without looking can provide a simple way to do this, and designing such filters under minimal assumptions is a natural next step.
Acknowledgments and Disclosure of Funding
This work is supported in part by NSF award # 2217058 (EnCORE Institute).
Statement on AI Use
All ideas and technical contributions in this paper are the authors’ own. ChatGPT 5.5 Plus and, later, Opus 5.5 were used for language editing, proofreading, and transcribing handwritten proofs to LaTeX. ChatGPT 5.5 Plus was also used to transcribe a hand-drawn version of Figure 1 into TikZ code. The authors bear sole responsibility for the content and correctness of this manuscript.
Technical appendices and supplementary material
Related Work
Noisy halfspace learning. Learning from noisy labels has been central to computational learning theory since Angluin and Laird [2 ] introduced random classification noise (RCN). Kearns [29 ] then showed that any algorithm that accesses the data only through statistical queries (SQ), i.e., averages of bounded functions, can be simulated from RCN samples, and for halfspaces, this led to polynomial-time learners that make no assumption on the marginal distribution [8 , 11 ] . Massart noise [30 ] sits between RCN and adversarial noise. Here, the flip probability may depend on the instance arbitrarily, but can never exceed some η < 1 / 2 \eta< 1/2 η < 1/2 . For homogeneous halfspaces, error opt + ε + \varepsilon + ε can be reached in polynomial time under structured marginals, for instance under the uniform distribution on the unit ball, but only for very small η < 1.8 × 10 − 6 \eta< 1.8 \times10^{-6} η < 1.8 × 1 0 − 6 [4 ] . Similar results exist for a broader family of marginals under Massart noise [16 ] and Tsybakov noise [34 , 17 ] , which allows the flip probability to approach 1 / 2 1/2 1/2 on a fraction of the instances. For affine halfspaces under Gaussian marginals, the algorithm of Diakonikolas et al. [18 ] runs in time d O ( log ( 1 / γ ) ) poly ( 1 / ε ) d^{O(\log(1/\gamma))}\operatorname{poly}(1/\varepsilon) d O ( l o g ( 1/ γ )) poly ( 1/ ε ) , where γ = max { p , ε } \gamma= \max\{p,\varepsilon\} γ = max { p , ε } , which is quasi-polynomial when p p p is of order ε \varepsilon ε . Margin-monotone noise with g ( 0 ) < 1 / 2 g(0) < 1/2 g ( 0 ) < 1/2 is a special case of Massart noise, but since their algorithm is designed for the worst case over all Massart noise, the resulting guarantee is no better on these profiles. In contrast, for the regularly varying profiles of Definition 2.2, our learner runs in polynomial time with O ~ ( d / p ) \widetilde{O}(d/p) O ( d / p ) samples for the direction. In this sense, our result is a beyond-worst-case counterpart of theirs [5 ] .
Margin-dependent noise. Noise that depends on an instance only through its distance to the decision boundary, and does not increase with it, goes back to the monotonic noise of Bylander [9 ] . Servedio [33 ] showed that a very simple algorithm, which he called Averaging and which estimates the degree-one Chow parameters E [ Y X ] \mathbb{E}[YX] E [ Y X ] by summing the examples multiplied by their labels, learns homogeneous halfspaces under this noise when the marginal is uniform on the unit sphere. Our rare-tail mean is a variant of this sum, with the labels centered at the majority label. The two behave similarly for homogeneous halfspaces, but with a bias p p p , the sum needs O ~ ( d / p 2 ) \widetilde{O}(d/p^2) O ( d / p 2 ) samples to reach the angular accuracy that the rare-tail mean reaches with O ~ ( d / p ) \widetilde{O}(d/p) O ( d / p ) , even without noise. The testable learner of Diakonikolas et al. [21 ] also averages the minority-label points, but only to locate a point near the boundary, around which the problem becomes nearly homogeneous. Under Gaussian marginals and logistic noise, Hsu and Mazumdar [26 ] determine, up to logarithmic factors, the sample complexity of estimating the direction of an homogeneous halfspace, but they know of efficient estimators that achieve it only at high noise levels and without noise. In the biased regime, where an angle of order 1 / t ∗ 1/t^* 1/ t ∗ suffices, the rare-tail mean is efficient for every regularly varying profile, including the logistic and probit ones. These have g ( 0 ) = 1 / 2 g(0) = 1/2 g ( 0 ) = 1/2 , so they are not Massart, and although they satisfy a Tsybakov condition, the polynomial-time Tsybakov results above are for homogeneous halfspaces, while ours have a bias.
The choice of marginal. For general halfspaces with label noise, the algorithms closest to ours are all stated for Gaussian marginals. They reach excess risk ε \varepsilon ε under RCN [20 ] , run in quasi-polynomial time under Massart noise [18 ] , and reach error O ( opt + ε ) O(\operatorname{opt} + \varepsilon) O ( opt + ε ) under adversarial noise [14 , 19 ] , which, under RCN in the biased regime, the constant classifier already does. Without assumptions on the marginal, learners pay for their generality. Under RCN, their sample complexity is polynomial in d d d , 1 / ε 1/\varepsilon 1/ ε , 1 / ( 1 − 2 η ) 1/(1-2\eta) 1/ ( 1 − 2 η ) and the number b b b of bits used to write each example [8 ] , while O ~ ( d / ε ) \widetilde{O}(d/\varepsilon) O ( d / ε ) samples suffice information-theoretically, independently of b b b [30 ] . Under Massart noise, they reach only error η + ε \eta+ \varepsilon η + ε [15 , 10 ] , which can be far from opt + ε \operatorname{opt} + \varepsilon opt + ε , a gap that efficient SQ algorithms cannot close [10 , 13 , 31 ] . Under log-concave marginals, polynomial regression reaches error opt + ε \operatorname{opt} + \varepsilon opt + ε for any label noise, which covers the setting of this paper, but with running time and samples of the form d f ( 1 / ε ) d^{f(1/\varepsilon)} d f ( 1/ ε ) for a growing function f f f [28 ] . With Massart noise, the only efficient learners we know of that reach excess risk ε \varepsilon ε for general halfspaces beyond Gaussian marginals use comparison queries, a stronger oracle than labels alone [25 ] . Since Gaussian distributions are log-concave, the SQ lower bounds below also apply to any SQ algorithm that succeeds for all log-concave marginals.
Lower bounds and data access. SQ lower bounds are the standard evidence that these problems are computationally hard, and they apply to every algorithm that, like Algorithm 1 and gradient methods, accesses the data only through averages. Under Gaussian marginals, SQ algorithms for general halfspaces need complexity d Ω ( log ( 1 / γ ) ) d^{\Omega(\log(1/\gamma))} d Ω ( l o g ( 1/ γ )) under Massart noise [18 ] , matching the algorithm above. Under RCN, efficient SQ algorithms and low-degree polynomial tests need Ω ( d / max { p , ε } 2 ) \Omega(\sqrt{d}/\max\{p,\varepsilon\}^{2}) Ω ( d / max { p , ε } 2 ) samples, with a quadratic dependence on 1 / p 1/p 1/ p that the information-theoretic rate does not have [20 ] . These lower bounds, however, concern the distribution that the queries are answered from, so they do not apply to learners with additional access. For general Gaussian halfspaces, Diakonikolas et al. [22 ] show that label queries on a pool of unlabeled points do not help much unless the pool is exponentially large in d d d , while membership queries reduce the label complexity to O ~ ( min { 1 / p , 1 / ε } + d polylog ( 1 / ε ) ) \widetilde{O}(\min\{1/p,1/\varepsilon\}+d\operatorname{polylog}(1/\varepsilon)) O ( min { 1/ p , 1/ ε } + d polylog ( 1/ ε )) for the weaker target O ( opt ) + ε O(\operatorname{opt})+\varepsilon O ( opt ) + ε . Membership queries, however, ask for labels of points that the learner constructs, and as Awasthi et al. [3 ] note, querying such unnatural points increases the noise of human labelers. Our curriculum needs no such queries, only some labels from annotators whose mistakes concentrate near the boundary.
Curriculum learning. Curriculum learning was popularized by Bengio et al. [7 ] as the principle that learning can benefit from examples being presented to a learner in a meaningful order. Curricula that move from reliable to noisy labels have been among the most effective strategies on noisy-label benchmarks [24 , 27 ] , and a controlled study finds that curricula help primarily when the labels are noisy or training budget is limited [37 ] . Theoretical models of curriculum learning are fewer. For learning parities under the uniform distribution, which is another problem where there is a proven gap for SQ methods, they often change the input distribution between the stages. Cornacchia and Mossel [12 ] , for instance, move from a biased product distribution to the uniform one, and Abbe et al. [1 ] from sparse inputs to dense ones. Moreover, their mathematical modeling of curriculum learning is specific to the training dynamics of a neural network, through SGD or noisy gradient methods. Our formulation is instead algorithm-agnostic. In Appendix D.1, we give a general definition of curriculum learning for an arbitrary binary hypothesis class and any finite number of stages, and then specialize to the two-stage model described in § 4 for noisy Gaussian halfspaces.
Learning Under Regular Variation Margin Monotone Profiles
Throughout, X ∼ N ( 0 , I d ) X \sim\mathcal{N}(0,I_d) X ∼ N ( 0 , I d ) , the target halfspace is f ⋆ ( x ) = sign ( ⟨ w ⋆ , x ⟩ + t ) f^\star(x)=\operatorname{sign}(\langle w^\star,x\rangle+t) f ⋆ ( x ) = sign (⟨ w ⋆ , x ⟩ + t ) with w ⋆ ∈ S d − 1 w^\star\in S^{d-1} w ⋆ ∈ S d − 1 and t > 0 t>0 t > 0 , and we write Z = ⟨ w ⋆ , X ⟩ ∼ N ( 0 , 1 ) Z=\langle w^\star,X\rangle\sim\mathcal{N}(0,1) Z = ⟨ w ⋆ , X ⟩ ∼ N ( 0 , 1 ) and M = Z + t ∼ N ( t , 1 ) M=Z+t\sim\mathcal{N}(t,1) M = Z + t ∼ N ( t , 1 ) . We denote the standard normal density and CDF by
ϕ ( u ) = ( 2 π ) − 1 / 2 e − u 2 / 2 , Φ ( u ) = ∫ − ∞ u ϕ ( s ) d s \phi(u)=(2\pi)^{-1/2}e^{-u^2/2},\qquad\Phi(u)=\int_{-\infty}^{u}\phi(s)\,ds ϕ ( u ) = ( 2 π ) − 1/2 e − u 2 /2 , Φ ( u ) = ∫ − ∞ u ϕ ( s ) d s
and write the halfspace bias as p = P ( f ⋆ ( X ) = − 1 ) = Φ ( − t ) p=\mathbb{P}(f^\star(X)=-1)=\Phi(-t) p = P ( f ⋆ ( X ) = − 1 ) = Φ ( − t ) and the inverse Mills ratio as λ ( t ) = ϕ ( t ) / Φ ( − t ) \lambda(t)=\phi(t)/\Phi(-t) λ ( t ) = ϕ ( t ) /Φ ( − t ) .
Gaussian identities
Lemma B.1 (One-dimensional decomposition). Let X ∼ N ( 0 , I d ) X\sim\mathcal{N}(0,I_d) X ∼ N ( 0 , I d ) and w ⋆ ∈ S d − 1 w^\star\in S^{d-1} w ⋆ ∈ S d − 1 . Define Z = ⟨ w ⋆ , X ⟩ Z=\langle w^\star,X\rangle Z = ⟨ w ⋆ , X ⟩ and X ⊥ = X − Z w ⋆ X_\perp=X-Zw^\star X ⊥ = X − Z w ⋆ . Then Z ∼ N ( 0 , 1 ) Z\sim\mathcal{N}(0,1) Z ∼ N ( 0 , 1 ) , X ⊥ ∼ N ( 0 , I d − w ⋆ w ⋆ T ) X_\perp\sim\mathcal{N}(0,I_d-w^\star{w^\star}^{\mathsf{T}}) X ⊥ ∼ N ( 0 , I d − w ⋆ w ⋆ T ) , and Z ⊥ X ⊥ Z\perp X_\perp Z ⊥ X ⊥ . Consequently, for any integrable ψ : R → R \psi:\mathbb{R}\to\mathbb{R} ψ : R → R ,
E [ X ψ ( Z ) ] = w ⋆ E [ Z ψ ( Z ) ] \mathbb{E}[X\psi(Z)]=w^\star\mathbb{E}[Z\psi(Z)] E [ X ψ ( Z )] = w ⋆ E [ Z ψ ( Z )]
Proof. Rotational invariance gives that ( Z , X ⊥ ) (Z,X_\perp) ( Z , X ⊥ ) is jointly Gaussian with E [ Z ] = 0 \mathbb{E}[Z]=0 E [ Z ] = 0 , E [ X ⊥ ] = 0 \mathbb{E}[X_\perp]=0 E [ X ⊥ ] = 0 , Var ( Z ) = 1 \operatorname{Var}(Z)=1 Var ( Z ) = 1 , Cov ( Z , X ⊥ ) = 0 \operatorname{Cov}(Z,X_\perp)=0 Cov ( Z , X ⊥ ) = 0 , and Cov ( X ⊥ ) = I d − w ⋆ w ⋆ T \operatorname{Cov}(X_\perp)=I_d-w^\star{w^\star}^{\mathsf{T}} Cov ( X ⊥ ) = I d − w ⋆ w ⋆ T . Hence Z ⊥ X ⊥ Z\perp X_\perp Z ⊥ X ⊥ and X = Z w ⋆ + X ⊥ X=Zw^\star+X_\perp X = Z w ⋆ + X ⊥ . The displayed identity follows by conditioning on Z Z Z and using E [ X ⊥ ∣ Z ] = 0 \mathbb{E}[X_\perp\mid Z]=0 E [ X ⊥ ∣ Z ] = 0 .
:::
Lemma B.2 (Truncation and Mills bounds). For every t > 0 t>0 t > 0 ,
E [ Z 1 { Z < − t } ] = − ϕ ( t ) , E [ Z ∣ Z < − t ] = − λ ( t ) \mathbb{E}[Z\mathbf{1}\{Z<-t\}]=-\phi(t),\qquad\mathbb{E}[Z\mid Z<-t]=-\lambda(t) E [ Z 1 { Z < − t }] = − ϕ ( t ) , E [ Z ∣ Z < − t ] = − λ ( t ) Moreover,
t t 2 + 1 ϕ ( t ) ≤ Φ ( − t ) ≤ ϕ ( t ) t , t ≤ λ ( t ) ≤ t + 1 t \frac{t}{t^2+1}\phi(t)\leq\Phi(-t)\leq\frac{\phi(t)}{t},\qquad t\leq\lambda(t)\leq t+\frac{1}{t} t 2 + 1 t ϕ ( t ) ≤ Φ ( − t ) ≤ t ϕ ( t ) , t ≤ λ ( t ) ≤ t + t 1
Proof. Integrating by parts,
E [ Z 1 { Z < − t } ] = ∫ − ∞ − t z ϕ ( z ) d z = [ − ϕ ( z ) ] − ∞ − t = − ϕ ( t ) \mathbb{E}[Z\mathbf{1}\{Z < -t\}] = \int_{-\infty}^{-t} z\phi(z)\,\mathrm{d}z = \left[-\phi(z)\right]_{-\infty}^{-t} = -\phi(t) E [ Z 1 { Z < − t }] = ∫ − ∞ − t z ϕ ( z ) d z = [ − ϕ ( z ) ] − ∞ − t = − ϕ ( t )
so E [ Z ∣ Z < − t ] = − ϕ ( t ) / Φ ( − t ) = − λ ( t ) \mathbb{E}[Z \mid Z < -t] = -\phi(t)/\Phi(-t) = -\lambda(t) E [ Z ∣ Z < − t ] = − ϕ ( t ) /Φ ( − t ) = − λ ( t ) The Mills bounds are the standard integration-by-parts inequalities
From fixed-threshold angular error to excess risk
Lemma B.3 (Restatement of Lemma 2.1). In the Gaussian halfspace model described above, fix t ≥ 1 t \ge1 t ≥ 1 , and let w ∈ S d − 1 w \in\mathbb{S}^{d-1} w ∈ S d − 1 be arbitrary, with induced halfspace f w ( x ) = sign ( ⟨ w , x ⟩ + t ) f_w(x) = \operatorname{sign}(\langle w,x\rangle+ t) f w ( x ) = sign (⟨ w , x ⟩ + t ) . There exist universal constants c 0 , C 0 > 0 c_0, C_0 > 0 c 0 , C 0 > 0 such that, whenever θ = ∠ ( w , w ⋆ ) ≤ c 0 t \theta= \angle(w,w^\star) \le\frac{c_0}{t} θ = ∠ ( w , w ⋆ ) ≤ t c 0 , we have
P ( f w ( X ) ≠ f ⋆ ( X ) ) ≤ C 0 θ ϕ ( t ) \mathbb{P}(f_w(X) \ne f^\star(X)) \le C_0\theta\phi(t) P ( f w ( X ) = f ⋆ ( X )) ≤ C 0 θ ϕ ( t ) In particular, in the critical high-bias regime p = Θ ( ε ) p = \Theta(\varepsilon) p = Θ ( ε ) , there is a sufficiently small universal constant c > 0 c > 0 c > 0 such that ∠ ( w , w ⋆ ) ≤ c t \angle(w,w^\star) \le\frac{c}{t} ∠ ( w , w ⋆ ) ≤ t c implies
err ( f w ) − err ( f ⋆ ) ≤ ε \operatorname{err}(f_w) - \operatorname{err}(f^\star) \le\varepsilon err ( f w ) − err ( f ⋆ ) ≤ ε
Proof. We write f w ( x ) = sign ( ⟨ w , x ⟩ + t ) f_w(x) = \operatorname{sign}(\langle w,x\rangle+ t) f w ( x ) = sign (⟨ w , x ⟩ + t ) . If d = 1 d = 1 d = 1 , the angle hypothesis forces θ = 0 \theta= 0 θ = 0 after taking c 0 < π c_0 < \pi c 0 < π , and the claim is immediate. We will therefore assume d ≥ 2 d \ge2 d ≥ 2 .
By rotational invariance, we may assume, without loss of generality,
w ⋆ = e 1 , w = cos θ e 1 + sin θ e 2 w^\star= e_1,\qquad w = \cos\theta\,e_1 + \sin\theta\,e_2 w ⋆ = e 1 , w = cos θ e 1 + sin θ e 2
Let G 1 , G 2 G_1, G_2 G 1 , G 2 be independent standard Gaussians. Then
⟨ w ⋆ , X ⟩ + t = G 1 + t \langle w^\star, X\rangle+ t = G_1 + t ⟨ w ⋆ , X ⟩ + t = G 1 + t
and
⟨ w , X ⟩ + t = cos θ ( G 1 + t ) + sin θ G 2 + t ( 1 − cos θ ) \langle w, X\rangle+ t = \cos\theta\,(G_1+t) + \sin\theta\,G_2 + t(1-\cos\theta) ⟨ w , X ⟩ + t = cos θ ( G 1 + t ) + sin θ G 2 + t ( 1 − cos θ )
If the two halfspaces disagree, these two signed margins have opposite signs. Since cos θ > 0 \cos\theta> 0 cos θ > 0 for θ ≤ c 0 / t \theta\le c_0/t θ ≤ c 0 / t and c 0 c_0 c 0 sufficiently small, this implies
∣ G 1 + t ∣ ≤ ∣ sin θ ∣ cos θ ∣ G 2 ∣ + t ( 1 − cos θ ) cos θ \lvert G_1+t\rvert\le\frac{\lvert\sin\theta\rvert}{\cos\theta}\lvert G_2\rvert+ \frac{t(1-\cos\theta)}{\cos\theta} ∣ G 1 + t ∣ ≤ cos θ ∣ sin θ ∣ ∣ G 2 ∣ + cos θ t ( 1 − cos θ )
For θ ≤ c 0 / t \theta\le c_0/t θ ≤ c 0 / t , after choosing c 0 c_0 c 0 sufficiently small, we have
∣ sin θ ∣ cos θ ≤ 2 θ , t ( 1 − cos θ ) cos θ ≤ 2 t θ 2 \frac{\lvert\sin\theta\rvert}{\cos\theta} \le2\theta,\qquad\frac{t(1-\cos\theta)}{\cos\theta} \le2t\theta^2 cos θ ∣ sin θ ∣ ≤ 2 θ , cos θ t ( 1 − cos θ ) ≤ 2 t θ 2
Therefore
P ( f w ( X ) ≠ f ⋆ ( X ) ) ≤ P ( ∣ G 1 + t ∣ ≤ 2 θ ∣ G 2 ∣ + 2 t θ 2 ) \mathbb{P}(f_w(X) \ne f^\star(X)) \le\mathbb{P}(\lvert G_1+t\rvert\le2\theta\lvert G_2\rvert+ 2t\theta^2) P ( f w ( X ) = f ⋆ ( X )) ≤ P (∣ G 1 + t ∣ ≤ 2 θ ∣ G 2 ∣ + 2 t θ 2 )
Conditioning on G 2 G_2 G 2 , and using the mean value theorem, for every r ≥ 0 r \ge0 r ≥ 0 ,
P ( ∣ G 1 + t ∣ ≤ r ) ≤ 2 r sup u ∈ [ t − r , t + r ] ϕ ( u ) ≤ 2 r ϕ ( t ) exp ( t r ) \mathbb{P}(\lvert G_1+t\rvert\le r) \le2r\sup_{u\in[t-r,t+r]}\phi(u) \le2r\phi(t)\exp(tr) P (∣ G 1 + t ∣ ≤ r ) ≤ 2 r u ∈ [ t − r , t + r ] sup ϕ ( u ) ≤ 2 r ϕ ( t ) exp ( t r )
Applying this with r = 2 θ ∣ G 2 ∣ + 2 t θ 2 r = 2\theta\lvert G_2\rvert+ 2t\theta^2 r = 2 θ ∣ G 2 ∣ + 2 t θ 2 gives
P ( f w ( X ) ≠ f ⋆ ( X ) ) ≤ 2 ϕ ( t ) E [ ( 2 θ ∣ G 2 ∣ + 2 t θ 2 ) exp ( t ( 2 θ ∣ G 2 ∣ + 2 t θ 2 ) ) ] \mathbb{P}(f_w(X) \ne f^\star(X)) \le2\phi(t)\mathbb{E}\left[(2\theta\lvert G_2\rvert+ 2t\theta^2)\exp\left(t(2\theta\lvert G_2\rvert+ 2t\theta^2)\right)\right] P ( f w ( X ) = f ⋆ ( X )) ≤ 2 ϕ ( t ) E [ ( 2 θ ∣ G 2 ∣ + 2 t θ 2 ) exp ( t ( 2 θ ∣ G 2 ∣ + 2 t θ 2 ) ) ]
Since t θ ≤ c 0 t\theta\le c_0 tθ ≤ c 0 , the expectation is at most C θ C\theta C θ for a universal constant C > 0 C > 0 C > 0 . Indeed,
2 θ ∣ G 2 ∣ + 2 t θ 2 = θ ( 2 ∣ G 2 ∣ + 2 t θ ) 2\theta\lvert G_2\rvert+ 2t\theta^2 = \theta(2\lvert G_2\rvert+ 2t\theta) 2 θ ∣ G 2 ∣ + 2 t θ 2 = θ ( 2 ∣ G 2 ∣ + 2 tθ )
and
t ( 2 θ ∣ G 2 ∣ + 2 t θ 2 ) = 2 ( t θ ) ∣ G 2 ∣ + 2 ( t θ ) 2 t(2\theta\lvert G_2\rvert+ 2t\theta^2) = 2(t\theta)\lvert G_2\rvert+ 2(t\theta)^2 t ( 2 θ ∣ G 2 ∣ + 2 t θ 2 ) = 2 ( tθ ) ∣ G 2 ∣ + 2 ( tθ ) 2
which has bounded exponential moment for all t θ ≤ c 0 t\theta\le c_0 tθ ≤ c 0 , after reducing c 0 c_0 c 0 if necessary. Thus
P ( f w ( X ) ≠ f ∗ ( X ) ) ≤ C 0 θ ϕ ( t ) \mathbb{P}(f_w(X) \ne f^*(X)) \le C_0\theta\phi(t) P ( f w ( X ) = f ∗ ( X )) ≤ C 0 θ ϕ ( t )
for a universal constant C 0 > 0 C_0 > 0 C 0 > 0 .
We now prove the excess-risk implication. Under the margin-dependent noise model,
err ( h ) − err ( f ∗ ) = E [ ( 1 − 2 g ( ∣ ⟨ w ∗ , X ⟩ + t ∣ ) ) 1 { h ( X ) ≠ f ∗ ( X ) } ] \operatorname{err}(h) - \operatorname{err}(f^*) = \mathbb{E}[(1 - 2g(|\langle w^*,X\rangle+ t|))\mathbb{1}\{h(X) \ne f^*(X)\}] err ( h ) − err ( f ∗ ) = E [( 1 − 2 g ( ∣ ⟨ w ∗ , X ⟩ + t ∣ )) 1 { h ( X ) = f ∗ ( X )}]
Since 0 ≤ g ≤ 1 / 2 0 \le g \le1/2 0 ≤ g ≤ 1/2 , applying this identity to h = f w h=f_w h = f w gives
err ( f w ) − err ( f ∗ ) ≤ P ( f w ( X ) ≠ f ∗ ( X ) ) ≤ C 0 θ ϕ ( t ) \operatorname{err}(f_w) - \operatorname{err}(f^*) \le\mathbb{P}(f_w(X) \ne f^*(X)) \le C_0\theta\phi(t) err ( f w ) − err ( f ∗ ) ≤ P ( f w ( X ) = f ∗ ( X )) ≤ C 0 θ ϕ ( t )
Finally, for t ≥ 1 t \ge1 t ≥ 1 , Mills’ ratio gives
ϕ ( t ) ≤ 2 t Φ ( − t ) = 2 t p \phi(t) \le2t\Phi(-t) = 2tp ϕ ( t ) ≤ 2 t Φ ( − t ) = 2 tp
If p = Θ ( ε ) p=\Theta(\varepsilon) p = Θ ( ε ) , then there is a constant A > 0 A>0 A > 0 such that p ≤ A ε p \le A\varepsilon p ≤ A ε . Therefore, whenever θ ≤ c / t \theta\le c/t θ ≤ c / t ,
err ( f w ) − err ( f ∗ ) ≤ 2 C 0 c p ≤ 2 C 0 c A ε \operatorname{err}(f_w) - \operatorname{err}(f^*) \le2C_0cp \le2C_0cA\varepsilon err ( f w ) − err ( f ∗ ) ≤ 2 C 0 c p ≤ 2 C 0 c A ε
Choosing c ≤ ( 2 C 0 A ) − 1 c \le(2C_0A)^{-1} c ≤ ( 2 C 0 A ) − 1 completes the proof.
A standard ε \varepsilon ε -net tool
Lemma B.4 (ε \varepsilon ε -net on the sphere). [35 Chapter 5] Fix d ≥ 1 d \ge1 d ≥ 1 and ε ∈ ( 0 , 1 ) \varepsilon\in(0,1) ε ∈ ( 0 , 1 ) . There exists a finite set N ε ⊂ S d − 1 \mathcal{N}_{\varepsilon} \subset S^{d-1} N ε ⊂ S d − 1 such that ∀ u ∈ S d − 1 ∃ v ∈ N ε : ∥ u − v ∥ 2 ≤ ε \forall u \in S^{d-1}\ \exists v \in\mathcal{N}_{\varepsilon}: \lVert u-v\rVert_2 \le\varepsilon ∀ u ∈ S d − 1 ∃ v ∈ N ε : ∥ u − v ∥ 2 ≤ ε and
∣ N ε ∣ ≤ ( 1 + 2 ε ) d |\mathcal{N}_{\varepsilon}| \le\left(1+\frac{2}{\varepsilon}\right)^d ∣ N ε ∣ ≤ ( 1 + ε 2 ) d Moreover, for every u ∈ R d u \in\mathbb{R}^d u ∈ R d ,
∥ u ∥ 2 ≤ 1 1 − ε max v ∈ N ε ∣ ⟨ v , u ⟩ ∣ \lVert u\rVert_2 \le\frac{1}{1-\varepsilon}\max_{v\in\mathcal{N}_{\varepsilon}}|\langle v,u\rangle| ∥ u ∥ 2 ≤ 1 − ε 1 v ∈ N ε max ∣ ⟨ v , u ⟩ ∣
Chernoff lower tail for binomials
Lemma B.5 (Lower-tail Chernoff for a binomial). [35 Chapter 2] Let B ∼ Bin ( N , q ) B\sim\operatorname{Bin}(N,q) B ∼ Bin ( N , q ) with q ∈ ( 0 , 1 ) q\in(0,1) q ∈ ( 0 , 1 ) and μ = E [ B ] = N q \mu=\mathbb{E}[B]=Nq μ = E [ B ] = N q . Then for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ,
P ( B ≤ ( 1 − δ ) μ ) ≤ exp ( − δ 2 μ 2 ) \mathbb{P}(B\le(1-\delta)\mu)\le\exp\left(-\frac{\delta^2\mu}{2}\right) P ( B ≤ ( 1 − δ ) μ ) ≤ exp ( − 2 δ 2 μ ) In particular,
P ( B ≤ μ 2 ) ≤ exp ( − μ 8 ) \mathbb{P}\left(B\le\frac{\mu}{2}\right)\le\exp\left(-\frac{\mu}{8}\right) P ( B ≤ 2 μ ) ≤ exp ( − 8 μ )
Margin-monotone noise and the rare-tail mean estimator
We recall the margin-dependent model: a measurable g : [ 0 , ∞ ) → [ 0 , 1 / 2 ] g:[0,\infty)\to[0,1/2] g : [ 0 , ∞ ) → [ 0 , 1/2 ] induces labels via
P ( Y ≠ f ∗ ( X ) ∣ X = x ) = g ( ∣ m ( x ) ∣ ) where m ( x ) = ⟨ w ∗ , x ⟩ + t \mathbb{P}(Y\ne f^*(X)\mid X=x)=g(|m(x)|) \qquad\text{where } m(x)=\langle w^*,x\rangle+t P ( Y = f ∗ ( X ) ∣ X = x ) = g ( ∣ m ( x ) ∣ ) where m ( x ) = ⟨ w ∗ , x ⟩ + t
If g g g is nonincreasing we call this margin-monotone noise.
Lemma B.6 (Orthogonal independence and direction of μ − \mu_- μ − ). Under any margin-monotone noise model, Y Y Y depends on X X X only through Z = ⟨ w ∗ , X ⟩ Z=\langle w^*,X\rangle Z = ⟨ w ∗ , X ⟩ . Hence X ⊥ Y X\perp Y X ⊥ Y and
μ − = E [ X ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 ] w ∗ \mu_-=\mathbb{E}[X\mid Y=-1]=\mathbb{E}[Z\mid Y=-1]w^* μ − = E [ X ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 ] w ∗ In particular, if μ − ≠ 0 \mu_-\ne0 μ − = 0 then w ∗ = − μ − / ∥ μ − ∥ 2 w^*=-\mu_-/\lVert\mu_-\rVert_2 w ∗ = − μ − / ∥ μ − ∥ 2
Proof. By Lemma B.1 , X = Z w ⋆ + X ⊥ X = Zw^\star+ X_\perp X = Z w ⋆ + X ⊥ , where X ⊥ X_\perp X ⊥ is independent of Z Z Z and has mean zero. Since the conditional law of Y Y Y given X X X depends only on Z Z Z , X ⊥ X_\perp X ⊥ is also independent of Y Y Y . Therefore,
μ − = E [ X ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 ] w ⋆ \mu_- = \mathbb{E}[X \mid Y=-1] = \mathbb{E}[Z \mid Y=-1]w^\star μ − = E [ X ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 ] w ⋆
For the sign, let q ( z ) = Pr ( Y = − 1 ∣ Z = z ) q(z)=\Pr(Y=-1 \mid Z=z) q ( z ) = Pr ( Y = − 1 ∣ Z = z ) . We have q ( z ) = 1 − g ( − z − t ) q(z)=1-g(-z-t) q ( z ) = 1 − g ( − z − t ) for z < − t z<-t z < − t and q ( z ) = g ( z + t ) q(z)=g(z+t) q ( z ) = g ( z + t ) for z ≥ − t z\geq-t z ≥ − t . Since g g g is nonincreasing and g ≤ 1 / 2 g\leq1/2 g ≤ 1/2 , q q q is nonincreasing. Therefore, z ( q ( z ) − q ( 0 ) ) ≤ 0 z(q(z)-q(0))\leq0 z ( q ( z ) − q ( 0 )) ≤ 0 for every z z z , and since E [ Z ] = 0 \mathbb{E}[Z]=0 E [ Z ] = 0 , we have
E [ Z ∣ Y = − 1 ] = E [ Z q ( Z ) ] p − = E [ Z ( q ( Z ) − q ( 0 ) ) ] p − ≤ 0 \mathbb{E}[Z \mid Y=-1]=\frac{\mathbb{E}[Zq(Z)]}{p_-}=\frac{\mathbb{E}[Z(q(Z)-q(0))]}{p_-}\leq0 E [ Z ∣ Y = − 1 ] = p − E [ Z q ( Z )] = p − E [ Z ( q ( Z ) − q ( 0 ))] ≤ 0
If μ − ≠ 0 \mu_-\neq0 μ − = 0 , this conditional mean is nonzero and strictly negative, which gives w ⋆ = − μ − / ∥ μ − ∥ 2 w^\star=-\mu_-/\|\mu_-\|_2 w ⋆ = − μ − /∥ μ − ∥ 2 .
Lemma B.7 (Integral and tanh representation). Let M ∼ N ( t , 1 ) M\sim\mathcal{N}(t,1) M ∼ N ( t , 1 ) with t > 0 t>0 t > 0 . Let U = ∣ M ∣ U=|M| U = ∣ M ∣ . Then for every measurable g : [ 0 , ∞ ) → R g:[0,\infty)\to\mathbb{R} g : [ 0 , ∞ ) → R satisfying E ∣ g ( U ) ∣ < ∞ \mathbb{E}|g(U)|<\infty E ∣ g ( U ) ∣ < ∞ ,
E [ g ( ∣ M ∣ ) sign ( M ) ] = ∫ 0 ∞ g ( u ) ( ϕ ( u − t ) − ϕ ( u + t ) ) d u \mathbb{E}[g(|M|)\operatorname{sign}(M)]=\int_0^\infty g(u)\bigl(\phi(u-t)-\phi(u+t)\bigr)\,\mathrm{d}u E [ g ( ∣ M ∣ ) sign ( M )] = ∫ 0 ∞ g ( u ) ( ϕ ( u − t ) − ϕ ( u + t ) ) d u = E [ g ( U ) tanh ( t U ) ] =\mathbb{E}[g(U)\tanh(tU)] = E [ g ( U ) tanh ( t U )]
Proof. Let ϕ ( m − t ) \phi(m-t) ϕ ( m − t ) be the density function of M M M . Splitting the integral at m = 0 m=0 m = 0 gives
E [ g ( ∣ M ∣ ) sign ( M ) ] = ∫ 0 ∞ g ( m ) ϕ ( m − t ) d m − ∫ − ∞ 0 g ( − m ) ϕ ( m − t ) d m \mathbb{E}[g(|M|)\operatorname{sign}(M)]=\int_0^\infty g(m)\phi(m-t)\,\mathrm{d}m-\int_{-\infty}^0 g(-m)\phi(m-t)\,\mathrm{d}m E [ g ( ∣ M ∣ ) sign ( M )] = ∫ 0 ∞ g ( m ) ϕ ( m − t ) d m − ∫ − ∞ 0 g ( − m ) ϕ ( m − t ) d m
In the second integral substitute m = − u m=-u m = − u (u > 0 u>0 u > 0 ), d m = − d u \mathrm{d}m=-\mathrm{d}u d m = − d u :
∫ − ∞ 0 g ( − m ) ϕ ( m − t ) d m = ∫ 0 ∞ g ( u ) ϕ ( − u − t ) d u = ∫ 0 ∞ g ( u ) ϕ ( u + t ) d u , \int_{-\infty}^0 g(-m)\phi(m-t)\,\mathrm{d}m=\int_0^\infty g(u)\phi(-u-t)\,\mathrm{d}u=\int_0^\infty g(u)\phi(u+t)\,\mathrm{d}u, ∫ − ∞ 0 g ( − m ) ϕ ( m − t ) d m = ∫ 0 ∞ g ( u ) ϕ ( − u − t ) d u = ∫ 0 ∞ g ( u ) ϕ ( u + t ) d u ,
by symmetry of ϕ \phi ϕ . Subtracting gives the first result.
Next, note that for u > 0 u>0 u > 0 the density of U = ∣ M ∣ U=|M| U = ∣ M ∣ is f U ( u ) = ϕ ( u − t ) + ϕ ( u + t ) f_U(u)=\phi(u-t)+\phi(u+t) f U ( u ) = ϕ ( u − t ) + ϕ ( u + t ) , and
ϕ ( u − t ) − ϕ ( u + t ) ϕ ( u − t ) + ϕ ( u + t ) = e u t − e − u t e u t + e − u t = tanh ( u t ) . \frac{\phi(u-t)-\phi(u+t)}{\phi(u-t)+\phi(u+t)}=\frac{e^{ut}-e^{-ut}}{e^{ut}+e^{-ut}}=\tanh(ut). ϕ ( u − t ) + ϕ ( u + t ) ϕ ( u − t ) − ϕ ( u + t ) = e u t + e − u t e u t − e − u t = tanh ( u t ) .
Replacing the integrand with g ( u ) f U ( u ) tanh ( t u ) g(u)f_U(u)\tanh(tu) g ( u ) f U ( u ) tanh ( t u ) gives the claimed result.
Concentration for the empirical minority mean
Lemma B.8 (Conditioning inflation for Gaussian projections). Assume only that X ∼ N ( 0 , I d ) X\sim\mathcal{N}(0,I_d) X ∼ N ( 0 , I d ) and Y ∈ { ± 1 } Y\in\{\pm1\} Y ∈ { ± 1 } is arbitrary with p − = P ( Y = − 1 ) > 0 p_-=\mathbb{P}(Y=-1)>0 p − = P ( Y = − 1 ) > 0 . Fix u ∈ S d − 1 u\in\mathbb{S}^{d-1} u ∈ S d − 1 and set V = ⟨ u , X ⟩ ∼ N ( 0 , 1 ) V=\langle u,X\rangle\sim\mathcal{N}(0,1) V = ⟨ u , X ⟩ ∼ N ( 0 , 1 ) . Define
κ 2 = 2 log ( 2 p − ) \kappa^2=2\log\left(\frac{2}{p_-}\right) κ 2 = 2 log ( p − 2 ) Then for every $s\geq 0,
P ( ∣ V ∣ ≥ s ∣ Y = − 1 ) ≤ 2 exp ( − s 2 2 κ 2 ) \mathbb{P}(|V|\geq s\mid Y=-1)\leq2\exp\left(-\frac{s^2}{2\kappa^2}\right) P ( ∣ V ∣ ≥ s ∣ Y = − 1 ) ≤ 2 exp ( − 2 κ 2 s 2 )
Proof. Since 1 { Y = − 1 } ≤ 1 \mathbf{1}\{Y=-1\}\leq1 1 { Y = − 1 } ≤ 1 ,
P ( ∣ V ∣ ≥ s ∣ Y = − 1 ) = P ( ∣ V ∣ ≥ s , Y = − 1 ) p − ≤ P ( ∣ V ∣ ≥ s ) p − ≤ min { 1 , 2 e − s 2 / 2 p − } . \mathbb{P}(|V|\geq s\mid Y=-1)=\frac{\mathbb{P}(|V|\geq s,Y=-1)}{p_-}\leq\frac{\mathbb{P}(|V|\geq s)}{p_-}\leq\min\left\{1,\frac{2e^{-s^2/2}}{p_-}\right\}. P ( ∣ V ∣ ≥ s ∣ Y = − 1 ) = p − P ( ∣ V ∣ ≥ s , Y = − 1 ) ≤ p − P ( ∣ V ∣ ≥ s ) ≤ min { 1 , p − 2 e − s 2 /2 } .
If s ≤ κ s\leq\kappa s ≤ κ then 2 exp ( − s 2 / ( 2 κ 2 ) ) ≥ 2 e − 1 / 2 > 1 2\exp(-s^2/(2\kappa^2))\geq2e^{-1/2}>1 2 exp ( − s 2 / ( 2 κ 2 )) ≥ 2 e − 1/2 > 1 , so the desired bound holds. If s ≥ κ s\geq\kappa s ≥ κ then 2 e − s 2 / 2 / p − ≤ 2 e − κ 2 / 2 / p − = 1 2e^{-s^2/2}/p_-\leq2e^{-\kappa^2/2}/p_-=1 2 e − s 2 /2 / p − ≤ 2 e − κ 2 /2 / p − = 1 , so the minimum equals 2 e − s 2 / 2 / p − 2e^{-s^2/2}/p_- 2 e − s 2 /2 / p − . It remains to show
2 e − s 2 / 2 p − ≤ 2 exp ( − s 2 2 κ 2 ) \frac{2e^{-s^2/2}}{p_-}\leq2\exp\left(-\frac{s^2}{2\kappa^2}\right) p − 2 e − s 2 /2 ≤ 2 exp ( − 2 κ 2 s 2 )
Equivalently,
log ( 1 p − ) ≤ s 2 2 ( 1 − 1 κ − 2 ) \log\left(\frac{1}{p_{-}}\right) \le\frac{s^{2}}{2}\left(1-\frac{1}{\kappa_{-}^{2}}\right) log ( p − 1 ) ≤ 2 s 2 ( 1 − κ − 2 1 )
For s > κ − s>\kappa_{-} s > κ − , the right-hand side is at least ( κ − 2 − 1 ) / 2 = log ( 2 / p − ) − 1 / 2 ≥ log ( 1 / p − ) (\kappa_{-}^{2}-1)/2=\log(2/p_{-})-1/2\ge\log(1/p_{-}) ( κ − 2 − 1 ) /2 = log ( 2/ p − ) − 1/2 ≥ log ( 1/ p − ) since log 2 ≥ 1 / 2 \log2\ge1/2 log 2 ≥ 1/2 .
Lemma B.9 (One-dimensional mean concentration under conditioning). Assume N − = b ≥ 1 N_{-}=b\ge1 N − = b ≥ 1 . Fix v ∈ S d − 1 v\in\mathbb{S}^{d-1} v ∈ S d − 1 and define W = ⟨ v , X ⟩ − E [ ⟨ v , X ⟩ ∣ Y = − 1 ] W=\langle v,X\rangle-\mathbb{E}[\langle v,X\rangle\mid Y=-1] W = ⟨ v , X ⟩ − E [⟨ v , X ⟩ ∣ Y = − 1 ] under the conditional law ( ⋅ ∣ Y = − 1 ) (\cdot\mid Y=-1) ( ⋅ ∣ Y = − 1 ) . Then W W W satisfies the tail bound of Lemma B.8 with κ = κ − \kappa=\kappa_{-} κ = κ − . Consequently, conditional on N − = b N_{-}=b N − = b , for all s ≥ 0 s\ge0 s ≥ 0 ,
P ( ∣ ⟨ v , μ ^ − − μ − ⟩ ∣ ≥ s ∣ N − = b ) ≤ 2 exp ( − b s 2 16 κ − 2 ) \mathbb{P}\left(\left|\langle v,\widehat{\mu}_{-}-\mu_{-}\rangle\right|\ge s\mid N_{-}=b\right)\le2\exp\left(-\frac{bs^{2}}{16\kappa_{-}^{2}}\right) P ( ∣ ⟨ v , μ − − μ − ⟩ ∣ ≥ s ∣ N − = b ) ≤ 2 exp ( − 16 κ − 2 b s 2 )
Proof. Under Y = − 1 Y=-1 Y = − 1 , Lemma B.8 gives P ( ∣ ⟨ v , X ⟩ ∣ ≥ s ∣ Y = − 1 ) ≤ 2 e − s 2 / ( 2 κ − 2 ) \mathbb{P}(|\langle v,X\rangle|\ge s\mid Y=-1)\le2e^{-s^{2}/(2\kappa_{-}^{2})} P ( ∣ ⟨ v , X ⟩ ∣ ≥ s ∣ Y = − 1 ) ≤ 2 e − s 2 / ( 2 κ − 2 ) . Centering changes tails only by constant factors, so W W W obeys the same functional form with the same κ − \kappa_{-} κ − . The two-sided subgaussian tail bound implies the mgf bound E [ exp ( λ W ) ∣ Y = − 1 ] ≤ exp ( 2 κ − 2 λ 2 ) \mathbb{E}[\exp(\lambda W)\mid Y=-1]\le\exp(2\kappa_{-}^{2}\lambda^{2}) E [ exp ( λW ) ∣ Y = − 1 ] ≤ exp ( 2 κ − 2 λ 2 ) for all λ ∈ R \lambda\in\mathbb{R} λ ∈ R . Conditional on N − = b N_{-}=b N − = b , ⟨ v , μ ^ − − μ − ⟩ \langle v,\widehat{\mu}_{-}-\mu_{-}\rangle ⟨ v , μ − − μ − ⟩ is the average of W W W . Thus its mgf is at most exp ( 2 κ − 2 λ 2 / b ) \exp(2\kappa_{-}^{2}\lambda^{2}/b) exp ( 2 κ − 2 λ 2 / b ) . Chernoff’s method 1 yields, for λ > 0 \lambda>0 λ > 0 ,
P ( ⟨ v , μ ^ − − μ − ⟩ ≥ s ∣ N − = b ) ≤ exp ( − λ s + 2 κ − 2 λ 2 b ) \mathbb{P}\left(\langle v,\widehat{\mu}_{-}-\mu_{-}\rangle\ge s\mid N_{-}=b\right)\le\exp\left(-\lambda s+\frac{2\kappa_{-}^{2}\lambda^{2}}{b}\right) P ( ⟨ v , μ − − μ − ⟩ ≥ s ∣ N − = b ) ≤ exp ( − λ s + b 2 κ − 2 λ 2 )
Optimizing at λ = b s / ( 4 κ − 2 ) \lambda=bs/(4\kappa_{-}^{2}) λ = b s / ( 4 κ − 2 ) gives exp ( − b s 2 / ( 8 κ − 2 ) ) \exp(-bs^{2}/(8\kappa_{-}^{2})) exp ( − b s 2 / ( 8 κ − 2 )) . Applying the same bound to − ⟨ v , μ ^ − − μ − ⟩ -\langle v,\widehat{\mu}_{-}-\mu_{-}\rangle − ⟨ v , μ − − μ − ⟩ and union bounding gives the stated inequality (with a conservative constant 16).
Lemma B.10 (Vector mean concentration given N − N_{-} N − ). Fix ϵ ∈ ( 0 , 1 ) \epsilon\in(0,1) ϵ ∈ ( 0 , 1 ) and let N ϵ \mathcal{N}_{\epsilon} N ϵ be an ϵ \epsilon ϵ -net as in Lemma B.4. Conditional on N − = b ≥ 1 N_{-}=b\ge1 N − = b ≥ 1 , for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ,
P ( ∥ μ ^ − − μ − ∥ 2 ≤ κ − 1 − ϵ 32 ( log ∣ N ϵ ∣ + log ( 2 / δ ) ) b ∣ N − = b ) ≥ 1 − δ \mathbb{P}\left(\left.\|\widehat{\mu}_{-}-\mu_{-}\|_{2}\le\frac{\kappa_{-}}{1-\epsilon}\sqrt{\frac{32(\log|\mathcal{N}_{\epsilon}|+\log(2/\delta))}{b}}\right|N_{-}=b\right)\ge1-\delta P ( ∥ μ − − μ − ∥ 2 ≤ 1 − ϵ κ − b 32 ( log ∣ N ϵ ∣ + log ( 2/ δ )) N − = b ) ≥ 1 − δ In particular, using ∣ N 1 / 2 ∣ ≤ 5 d |\mathcal{N}_{1/2}|\le5^{d} ∣ N 1/2 ∣ ≤ 5 d ,
P ( ∥ μ ^ − − μ − ∥ 2 ≤ 2 κ − 32 ( d log 5 + log ( 2 / δ ) ) b ∣ N − = b ) ≥ 1 − δ \mathbb{P}\left(\left.\|\widehat{\mu}_{-}-\mu_{-}\|_{2}\le2\kappa_{-}\sqrt{\frac{32(d\log5+\log(2/\delta))}{b}}\right|N_{-}=b\right)\ge1-\delta P ( ∥ μ − − μ − ∥ 2 ≤ 2 κ − b 32 ( d log 5 + log ( 2/ δ )) N − = b ) ≥ 1 − δ Proof. By Lemma B.9, for each fixed v ∈ N ϵ v\in\mathcal{N}_{\epsilon} v ∈ N ϵ ,
P ( ∣ ⟨ v , μ ^ − − μ − ⟩ ∣ ≥ s ∣ N − = b ) ≤ 2 exp ( − b s 2 16 κ − 2 ) \mathbb{P}\left(\left|\langle v,\widehat{\mu}_{-}-\mu_{-}\rangle\right|\ge s\mid N_{-}=b\right)\le2\exp\left(-\frac{bs^{2}}{16\kappa_{-}^{2}}\right) P ( ∣ ⟨ v , μ − − μ − ⟩ ∣ ≥ s ∣ N − = b ) ≤ 2 exp ( − 16 κ − 2 b s 2 ) A union bound over v ∈ N ϵ v\in\mathcal{N}_{\epsilon} v ∈ N ϵ gives
P ( max v ∈ N ϵ ∣ ⟨ v , μ ^ − − μ − ⟩ ∣ ≥ s ∣ N − = b ) ≤ 2 ∣ N ϵ ∣ exp ( − b s 2 16 κ − 2 ) \mathbb{P}\left(\max_{v\in\mathcal{N}_{\epsilon}}\left|\langle v,\widehat{\mu}_{-}-\mu_{-}\rangle\right|\ge s\mid N_{-}=b\right)\le2|\mathcal{N}_{\epsilon}|\exp\left(-\frac{bs^{2}}{16\kappa_{-}^{2}}\right) P ( v ∈ N ϵ max ∣ ⟨ v , μ − − μ − ⟩ ∣ ≥ s ∣ N − = b ) ≤ 2∣ N ϵ ∣ exp ( − 16 κ − 2 b s 2 ) Set the right-hand side equal to δ \delta δ and solve for s s s . On the complementary event, the norm result of Lemma B.4 implies
∥ μ ^ − − μ − ∥ 2 ≤ 1 1 − ϵ max v ∈ N ϵ ∣ ⟨ v , μ ^ − − μ − ⟩ ∣ ≤ s 1 − ϵ \|\widehat{\mu}_{-}-\mu_{-}\|_{2}\le\frac{1}{1-\epsilon}\max_{v\in\mathcal{N}_{\epsilon}}\left|\langle v,\widehat{\mu}_{-}-\mu_{-}\rangle\right|\le\frac{s}{1-\epsilon} ∥ μ − − μ − ∥ 2 ≤ 1 − ϵ 1 v ∈ N ϵ max ∣ ⟨ v , μ − − μ − ⟩ ∣ ≤ 1 − ϵ s
Lemma B.11 (Reweighting by nonincreasing function lowers mean). Let ν \nu ν be a probability distribution on R \mathbb{R} R , and let Z ∼ ν Z \sim\nu Z ∼ ν . Let h : R → [ 0 , ∞ ) h:\mathbb{R}\to[0,\infty) h : R → [ 0 , ∞ ) be nonincreasing, with 0 < E ν [ h ( Z ) ] < ∞ 0 < \mathbb{E}_{\nu}[h(Z)] < \infty 0 < E ν [ h ( Z )] < ∞ . Assume also that E ν [ ∣ Z ∣ ] < ∞ \mathbb{E}_{\nu}[|Z|] < \infty E ν [ ∣ Z ∣ ] < ∞ and E ν [ ∣ Z ∣ h ( Z ) ] < ∞ \mathbb{E}_{\nu}[|Z|h(Z)] < \infty E ν [ ∣ Z ∣ h ( Z )] < ∞ . Then
E ν [ Z h ( Z ) ] ≤ E ν [ Z ] E ν [ h ( Z ) ] \mathbb{E}_{\nu}[Zh(Z)] \le\mathbb{E}_{\nu}[Z]\mathbb{E}_{\nu}[h(Z)] E ν [ Z h ( Z )] ≤ E ν [ Z ] E ν [ h ( Z )] Equivalently,
E ν [ Z h ( Z ) ] E ν [ h ( Z ) ] ≤ E ν [ Z ] \frac{\mathbb{E}_{\nu}[Zh(Z)]}{\mathbb{E}_{\nu}[h(Z)]} \le\mathbb{E}_{\nu}[Z] E ν [ h ( Z )] E ν [ Z h ( Z )] ≤ E ν [ Z ] Proof. Let Z 1 , Z 2 Z_1, Z_2 Z 1 , Z 2 be independent random variables with law ν \nu ν . Since h h h is nonincreasing, for every z 1 , z 2 ∈ R z_1,z_2 \in\mathbb{R} z 1 , z 2 ∈ R ,
( z 1 − z 2 ) ( h ( z 1 ) − h ( z 2 ) ) ≤ 0 (z_1-z_2)(h(z_1)-h(z_2)) \le0 ( z 1 − z 2 ) ( h ( z 1 ) − h ( z 2 )) ≤ 0 Taking expectation gives
0 ≥ E [ ( Z 1 − Z 2 ) ( h ( Z 1 ) − h ( Z 2 ) ) ] = 2 E ν [ Z h ( Z ) ] − 2 E ν [ Z ] E ν [ h ( Z ) ] \begin{aligned}
0 &\ge\mathbb{E}\left[(Z_1-Z_2)(h(Z_1)-h(Z_2))\right] \\
&= 2\mathbb{E}_{\nu}[Zh(Z)]-2\mathbb{E}_{\nu}[Z]\mathbb{E}_{\nu}[h(Z)]
\end{aligned} 0 ≥ E [ ( Z 1 − Z 2 ) ( h ( Z 1 ) − h ( Z 2 )) ] = 2 E ν [ Z h ( Z )] − 2 E ν [ Z ] E ν [ h ( Z )] Rearranging proves
E ν [ Z h ( Z ) ] ≤ E ν [ Z ] E ν [ h ( Z ) ] \mathbb{E}_{\nu}[Zh(Z)] \le\mathbb{E}_{\nu}[Z]\mathbb{E}_{\nu}[h(Z)] E ν [ Z h ( Z )] ≤ E ν [ Z ] E ν [ h ( Z )] Since E ν [ h ( Z ) ] > 0 \mathbb{E}_{\nu}[h(Z)] > 0 E ν [ h ( Z )] > 0 , dividing by E ν [ h ( Z ) ] \mathbb{E}_{\nu}[h(Z)] E ν [ h ( Z )] gives the equivalent weighted-mean form.
Angle recovery
Lemma B.12 (Angle from a mean-estimation error). Let u , v ∈ R d u,v \in\mathbb{R}^{d} u , v ∈ R d be nonzero, and define u ˉ = u / ∥ u ∥ 2 \bar{u}=u/\lVert u\rVert_{2} u ˉ = u / ∥ u ∥ 2 , v ˉ = v / ∥ v ∥ 2 \bar{v}=v/\lVert v\rVert_{2} v ˉ = v / ∥ v ∥ 2 . Let θ = ∠ ( u ˉ , v ˉ ) ∈ [ 0 , π ] \theta=\angle(\bar{u},\bar{v})\in[0,\pi] θ = ∠ ( u ˉ , v ˉ ) ∈ [ 0 , π ] . Then
sin ( θ / 2 ) ≤ ∥ u − v ∥ 2 ∥ v ∥ 2 \sin(\theta/2) \le\frac{\lVert u-v\rVert_{2}}{\lVert v\rVert_{2}} sin ( θ /2 ) ≤ ∥ v ∥ 2 ∥ u − v ∥ 2 Proof. For unit vectors, ∥ u ˉ − v ˉ ∥ 2 2 = 2 − 2 ⟨ u ˉ , v ˉ ⟩ = 4 sin 2 ( θ / 2 ) \lVert\bar{u}-\bar{v}\rVert_{2}^{2}=2-2\langle\bar{u},\bar{v}\rangle=4\sin^{2}(\theta/2) ∥ u ˉ − v ˉ ∥ 2 2 = 2 − 2 ⟨ u ˉ , v ˉ ⟩ = 4 sin 2 ( θ /2 ) . Thus, sin ( θ / 2 ) = ∥ u ˉ − v ˉ ∥ 2 / 2 \sin(\theta/2)=\lVert\bar{u}-\bar{v}\rVert_{2}/2 sin ( θ /2 ) = ∥ u ˉ − v ˉ ∥ 2 /2 . By the triangle inequality,
∥ u ˉ − v ˉ ∥ 2 = ∥ u ∥ u ∥ 2 − v ∥ v ∥ 2 ∥ 2 ≤ ∥ u ∥ u ∥ 2 − u ∥ v ∥ 2 ∥ 2 + ∥ u ∥ v ∥ 2 − v ∥ v ∥ 2 ∥ 2 ≤ 2 ∥ u − v ∥ 2 ∥ v ∥ 2 \lVert\bar{u}-\bar{v}\rVert_{2}
= \left\lVert\frac{u}{\lVert u\rVert_{2}}-\frac{v}{\lVert v\rVert_{2}}\right\rVert_{2}
\le\left\lVert\frac{u}{\lVert u\rVert_{2}}-\frac{u}{\lVert v\rVert_{2}}\right\rVert_{2}
+\left\lVert\frac{u}{\lVert v\rVert_{2}}-\frac{v}{\lVert v\rVert_{2}}\right\rVert_{2}
\le\frac{2\lVert u-v\rVert_{2}}{\lVert v\rVert_{2}} ∥ u ˉ − v ˉ ∥ 2 = ∥ u ∥ 2 u − ∥ v ∥ 2 v 2 ≤ ∥ u ∥ 2 u − ∥ v ∥ 2 u 2 + ∥ v ∥ 2 u − ∥ v ∥ 2 v 2 ≤ ∥ v ∥ 2 2 ∥ u − v ∥ 2 Combining gives the claim.
Theorem B.13 (Sample size for angle recovery via the minority mean). Assume margin-monotone noise with t > 0 t>0 t > 0 and suppose μ − ≠ 0 \mu_{-}\ne0 μ − = 0 . Let w ^ \widehat{w} w be the minority-mean estimator of Algorithm 1 from N N N i.i.d. samples. Fix τ ∈ ( 0 , π ) \tau\in(0,\pi) τ ∈ ( 0 , π ) and δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) . Let κ − 2 = 2 log ( 2 / p − ) \kappa_{-}^{2}=2\log(2/p_{-}) κ − 2 = 2 log ( 2/ p − ) . If
N ≥ max { 256 κ − 2 ( d log 5 + log ( 4 / δ ) ) p − ∥ μ − ∥ 2 2 sin 2 ( τ / 2 ) , 8 p − log ( 2 δ ) } N \ge\max\left\{\frac{256\kappa_{-}^{2}(d\log5+\log(4/\delta))}{p_{-}\lVert\mu_{-}\rVert_{2}^{2}\sin^{2}(\tau/2)},\frac{8}{p_{-}}\log\left(\frac{2}{\delta}\right)\right\} N ≥ max { p − ∥ μ − ∥ 2 2 sin 2 ( τ /2 ) 256 κ − 2 ( d log 5 + log ( 4/ δ )) , p − 8 log ( δ 2 ) } then with probability at least 1 − δ 1-\delta 1 − δ we have N − ≥ 1 N_{-}\ge1 N − ≥ 1 and ∠ ( w ^ , w ⋆ ) ≤ τ \angle(\widehat{w},w^{\star})\le\tau ∠ ( w , w ⋆ ) ≤ τ .
Proof. Note that N − ∼ Bin ( N , p − ) N_{-}\sim\operatorname{Bin}(N,p_{-}) N − ∼ Bin ( N , p − ) . Applying Lemma B.5 with δ = 1 / 2 \delta=1/2 δ = 1/2 and μ = N p − \mu=Np_{-} μ = N p − , if N ≥ 8 p − log ( 2 / δ ) N\ge\frac{8}{p_{-}}\log(2/\delta) N ≥ p − 8 log ( 2/ δ ) then
P ( N − < N p − 2 ) ≤ δ 2 \mathbb{P}\left(N_{-}<\frac{Np_{-}}{2}\right)\le\frac{\delta}{2} P ( N − < 2 N p − ) ≤ 2 δ
Define
b 0 = ⌈ 128 κ − 2 ( d log 5 + log ( 4 / δ ) ) ∥ μ − ∥ 2 2 sin 2 ( τ / 2 ) ⌉ b_0=\left\lceil\frac{128\kappa_-^2(d\log5+\log(4/\delta))}{\|\mu_-\|_2^2\sin^2(\tau/2)}\right\rceil b 0 = ⌈ ∥ μ − ∥ 2 2 sin 2 ( τ /2 ) 128 κ − 2 ( d log 5 + log ( 4/ δ )) ⌉
Under the stated sample bound, N p − / 2 > b 0 − 1 Np_-/2>b_0-1 N p − /2 > b 0 − 1 . On the event N − ≥ N p − / 2 N_-\geq Np_-/2 N − ≥ N p − /2 , integrality therefore gives N − ≥ b 0 N_-\geq b_0 N − ≥ b 0 . Consequently, P ( N − ≥ b 0 ) ≥ 1 − δ / 2 \mathbb{P}(N_-\geq b_0)\geq1-\delta/2 P ( N − ≥ b 0 ) ≥ 1 − δ /2 .
Next, we condition on the event N − = b ≥ b 0 N_-=b\geq b_0 N − = b ≥ b 0 . Applying Lemma B.10 with ε = 1 / 2 \varepsilon=1/2 ε = 1/2 and confidence δ / 2 \delta/2 δ /2 gives, with conditional probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 ,
∥ μ ^ − − μ − ∥ 2 ≤ 2 κ − 32 ( d log 5 + log ( 4 / δ ) ) b ≤ ∥ μ − ∥ 2 sin ( τ / 2 ) \|\widehat{\mu}_--\mu_-\|_2\leq2\kappa_-\sqrt{\frac{32(d\log5+\log(4/\delta))}{b}}\leq\|\mu_-\|_2\sin(\tau/2) ∥ μ − − μ − ∥ 2 ≤ 2 κ − b 32 ( d log 5 + log ( 4/ δ )) ≤ ∥ μ − ∥ 2 sin ( τ /2 )
where the last inequality uses b ≥ b 0 b\geq b_0 b ≥ b 0 . On this event, μ ^ − ≠ 0 \widehat{\mu}_-\neq0 μ − = 0 since ∥ μ ^ − ∥ 2 ≥ ∥ μ − ∥ 2 − ∥ μ ^ − − μ − ∥ 2 > 0 \|\widehat{\mu}_-\|_2\geq\|\mu_-\|_2-\|\widehat{\mu}_--\mu_-\|_2>0 ∥ μ − ∥ 2 ≥ ∥ μ − ∥ 2 − ∥ μ − − μ − ∥ 2 > 0 . Moreover, Lemma B.6 gives w ⋆ = − μ − / ∥ μ − ∥ 2 w^\star=-\mu_-/\|\mu_-\|_2 w ⋆ = − μ − /∥ μ − ∥ 2 , hence the angle between w ^ \widehat{w} w and w ⋆ w^\star w ⋆ equals the angle between the normalized vectors μ ^ − / ∥ μ ^ − ∥ 2 \widehat{\mu}_-/\|\widehat{\mu}_-\|_2 μ − /∥ μ − ∥ 2 and μ − / ∥ μ − ∥ 2 \mu_-/\|\mu_-\|_2 μ − /∥ μ − ∥ 2 . Lemma B.12 yields
sin ( ∠ ( w ^ , w ⋆ ) 2 ) ≤ ∥ μ ^ − − μ − ∥ 2 ∥ μ − ∥ 2 ≤ sin ( τ / 2 ) \sin\left(\frac{\angle(\widehat{w},w^\star)}{2}\right)\leq\frac{\|\widehat{\mu}_--\mu_-\|_2}{\|\mu_-\|_2}\leq\sin(\tau/2) sin ( 2 ∠ ( w , w ⋆ ) ) ≤ ∥ μ − ∥ 2 ∥ μ − − μ − ∥ 2 ≤ sin ( τ /2 )
Since sin \sin sin is increasing on [ 0 , π / 2 ] [0,\pi/2] [ 0 , π /2 ] , this implies ∠ ( w ^ , w ⋆ ) ≤ τ \angle(\widehat{w},w^\star)\leq\tau ∠ ( w , w ⋆ ) ≤ τ . A union bound over the two failure events gives overall success probability at least 1 − δ 1-\delta 1 − δ .
Lower bound on p − p_- p − under regular variation
Lemma B.14 (Observed negative mass is never below the clean tail scale). Assume t > 0 t>0 t > 0 and margin-dependent noise with profile g : [ 0 , ∞ ) → [ 0 , 1 / 2 ] g:[0,\infty)\to[0,1/2] g : [ 0 , ∞ ) → [ 0 , 1/2 ] . Let p = Φ ( − t ) p=\Phi(-t) p = Φ ( − t ) and p − = P ( Y = − 1 ) p_-=\mathbb{P}(Y=-1) p − = P ( Y = − 1 ) . Then
p − ≥ p 2 p_-\geq\frac{p}{2} p − ≥ 2 p
Consequently, for every fixed C ≥ 0 C\geq0 C ≥ 0 ,
t 4 ( log ( e + t ) ) 2 C p − = O ~ C ( 1 p ) \frac{t^4(\log(e+t))^{2C}}{p_-}=\widetilde{O}_C\left(\frac{1}{p}\right) p − t 4 ( log ( e + t ) ) 2 C = O C ( p 1 )
Proof. On the clean negative tail { Z < − t } \{Z<-t\} { Z < − t } , we have f ⋆ ( X ) = − 1 f^\star(X)=-1 f ⋆ ( X ) = − 1 , and therefore
P ( Y = − 1 ∣ Z ) ≥ 1 2 \mathbb{P}(Y=-1\mid Z)\geq\frac{1}{2} P ( Y = − 1 ∣ Z ) ≥ 2 1
because g ≤ 1 / 2 g\leq1/2 g ≤ 1/2 . Hence
p − ≥ P ( Y = − 1 , Z < − t ) ≥ 1 2 P ( Z < − t ) = p 2 p_-\geq\mathbb{P}(Y=-1,Z<-t)\geq\frac{1}{2}\mathbb{P}(Z<-t)=\frac{p}{2} p − ≥ P ( Y = − 1 , Z < − t ) ≥ 2 1 P ( Z < − t ) = 2 p
For t ≥ 1 t\geq1 t ≥ 1 , the Gaussian tail bound p = Φ ( − t ) ≤ exp ( − t 2 / 2 ) p=\Phi(-t)\leq\exp(-t^2/2) p = Φ ( − t ) ≤ exp ( − t 2 /2 ) implies t 2 ≤ 2 log ( 1 / p ) t^2\leq2\log(1/p) t 2 ≤ 2 log ( 1/ p ) . Thus t 4 ( log ( e + t ) ) 2 C t^4(\log(e+t))^{2C} t 4 ( log ( e + t ) ) 2 C is polylogarithmic in 1 / p 1/p 1/ p , and the conclusion follows.
Lower bound on ∥ μ − ∥ 2 \|\mu_-\|_2 ∥ μ − ∥ 2 under regular variation
Assume g g g satisfies the regular variation condition of Definition 2.2. Define q t ( z ) = P ( Y = − 1 ∣ Z = z ) q_t(z)=\mathbb{P}(Y=-1\mid Z=z) q t ( z ) = P ( Y = − 1 ∣ Z = z ) . Under margin-monotone noise,
q t ( z ) = { 1 − g ( − ( z + t ) ) z < − t , g ( z + t ) z > − t . q_t(z)=
\begin{cases}
1-g(-(z+t)) & z<-t,\\
g(z+t) & z>-t.
\end{cases} q t ( z ) = { 1 − g ( − ( z + t )) g ( z + t ) z < − t , z > − t .
Lemma B.15 (Signal decomposition). Let U = ∣ Z + t ∣ U=|Z+t| U = ∣ Z + t ∣ . Then
p − ∥ μ − ∥ 2 = ( 1 − 2 g ( 0 ) ) ϕ ( t ) + E [ − g ′ ( U ) ] p_-\|\mu_-\|_2=(1-2g(0))\phi(t)+\mathbb{E}[-g'(U)] p − ∥ μ − ∥ 2 = ( 1 − 2 g ( 0 )) ϕ ( t ) + E [ − g ′ ( U )]
Proof. By Lemma B.6, p − ∥ μ − ∥ 2 = − E [ Z 1 { Y = − 1 } ] = − E [ Z q t ( Z ) ] p_{-}\left\|\mu_{-}\right\|_{2}=-\mathbb{E}[Z\mathbf{1}\{Y=-1\}]=-\mathbb{E}[Zq_{t}(Z)] p − ∥ μ − ∥ 2 = − E [ Z 1 { Y = − 1 }] = − E [ Z q t ( Z )] . Using ϕ ′ ( z ) = − z ϕ ( z ) \phi'(z)=-z\phi(z) ϕ ′ ( z ) = − z ϕ ( z ) ,
E [ Z q t ( Z ) ] = ∫ R z q t ( z ) ϕ ( z ) d z = − ∫ R q t ( z ) d ϕ ( z ) = ∫ R ϕ ( z ) d q t ( z ) \begin{aligned}
\mathbb{E}[Zq_{t}(Z)]=\int_{\mathbb{R}}zq_{t}(z)\phi(z)\,dz \\
&=-\int_{\mathbb{R}}q_{t}(z)\,d\phi(z) \\
&=\int_{\mathbb{R}}\phi(z)\,dq_{t}(z)
\end{aligned} E [ Z q t ( Z )] = ∫ R z q t ( z ) ϕ ( z ) d z = − ∫ R q t ( z ) d ϕ ( z ) = ∫ R ϕ ( z ) d q t ( z )
The function q t q_{t} q t has a single jump at z = − t z=-t z = − t of size ( 1 − g ( 0 ) ) − g ( 0 ) = 1 − 2 g ( 0 ) (1-g(0))-g(0)=1-2g(0) ( 1 − g ( 0 )) − g ( 0 ) = 1 − 2 g ( 0 ) . Therefore, d q t dq_{t} d q t has an atom of mass − ( 1 − 2 g ( 0 ) ) -(1-2g(0)) − ( 1 − 2 g ( 0 )) at − t -t − t . On ( − t , ∞ ) (-t,\infty) ( − t , ∞ ) , d q t ( z ) = g ′ ( z + t ) d z dq_{t}(z)=g'(z+t)\,dz d q t ( z ) = g ′ ( z + t ) d z , and on ( − ∞ , − t ) (-\infty,-t) ( − ∞ , − t ) , d q t ( z ) = g ′ ( − ( z + t ) ) d z dq_{t}(z)=g'(-(z+t))\,dz d q t ( z ) = g ′ ( − ( z + t )) d z . This gives
− E [ Z q t ( Z ) ] = − ∫ R ϕ ( z ) d q t ( z ) = ( 1 − 2 g ( 0 ) ) ϕ ( t ) + ∫ − t ∞ ϕ ( z ) ( − g ′ ( z + t ) ) d z + ∫ − ∞ − t ϕ ( z ) ( − g ′ ( − ( z + t ) ) ) d z \begin{aligned}
-\mathbb{E}[Zq_{t}(Z)]=-\int_{\mathbb{R}}\phi(z)\,dq_{t}(z) \\
&=(1-2g(0))\phi(t)+\int_{-t}^{\infty}\phi(z)(-g'(z+t))\,dz+\int_{-\infty}^{-t}\phi(z)(-g'(-(z+t)))\,dz
\end{aligned} − E [ Z q t ( Z )] = − ∫ R ϕ ( z ) d q t ( z ) = ( 1 − 2 g ( 0 )) ϕ ( t ) + ∫ − t ∞ ϕ ( z ) ( − g ′ ( z + t )) d z + ∫ − ∞ − t ϕ ( z ) ( − g ′ ( − ( z + t ))) d z
Change variables u = z + t u=z+t u = z + t in the first integral and u = − ( z + t ) u=-(z+t) u = − ( z + t ) in the second to obtain
− E [ Z q t ( Z ) ] = ( 1 − 2 g ( 0 ) ) ϕ ( t ) + ∫ 0 ∞ ( ϕ ( u − t ) + ϕ ( u + t ) ) ( − g ′ ( u ) ) d u -\mathbb{E}[Zq_{t}(Z)]=(1-2g(0))\phi(t)+\int_{0}^{\infty}\bigl(\phi(u-t)+\phi(u+t)\bigr)(-g'(u))\,du − E [ Z q t ( Z )] = ( 1 − 2 g ( 0 )) ϕ ( t ) + ∫ 0 ∞ ( ϕ ( u − t ) + ϕ ( u + t ) ) ( − g ′ ( u )) d u
Since U = ∣ Z + t ∣ U=|Z+t| U = ∣ Z + t ∣ has density f U ( u ) = ϕ ( u − t ) + ϕ ( u + t ) f_{U}(u)=\phi(u-t)+\phi(u+t) f U ( u ) = ϕ ( u − t ) + ϕ ( u + t ) on ( 0 , ∞ ) (0,\infty) ( 0 , ∞ ) , the integral equals E [ − g ′ ( U ) ] \mathbb{E}[-g'(U)] E [ − g ′ ( U )] .
Theorem B.16 (Direction estimation under regular variation (restatement of Theorem 3.1 ).) Assume t ≥ 1 t\ge1 t ≥ 1 and ( c , C ) (c,C) ( c , C ) -regular variation margin-monotone noise (Definition 2.2 ). Let w ^ \widehat{w} w be the minority-mean estimator of Algorithm 1 built from N N N i.i.d. samples. Then for every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , if
N ≥ O ~ ( ( d + log ( 1 / δ ) ) ⋅ max { 1 p − , t 4 ( log ( e + t ) ) 2 C p − } ) , N\ge\widetilde{O}\left((d+\log(1/\delta))\cdot\max\left\{\frac{1}{p_{-}},\frac{t^{4}(\log(e+t))^{2C}}{p_{-}}\right\}\right), N ≥ O ( ( d + log ( 1/ δ )) ⋅ max { p − 1 , p − t 4 ( log ( e + t ) ) 2 C } ) , then ∠ ( w ^ , w ⋆ ) ≤ c d i r / t \angle(\widehat{w},w^{\star})\le c_{\mathrm{dir}}/t ∠ ( w , w ⋆ ) ≤ c dir / t with probability at least 1 − δ 1-\delta 1 − δ .
Proof. We first analyze the estimator formed from the Y = − 1 Y=-1 Y = − 1 samples. Apply Theorem B.13 with target angle τ = c d i r / t \tau=c_{\mathrm{dir}}/t τ = c dir / t and confidence parameter δ / 2 \delta/2 δ /2 . Here c d i r ∈ ( 0 , 1 ] c_{\mathrm{dir}}\in(0,1] c dir ∈ ( 0 , 1 ] is the fixed constant in Theorem 3.1 . Since t ≥ 1 t\ge1 t ≥ 1 , the inequality sin x ≥ 2 x / π \sin x\ge2x/\pi sin x ≥ 2 x / π for x ∈ [ 0 , π / 2 ] x\in[0,\pi/2] x ∈ [ 0 , π /2 ] gives
sin ( τ / 2 ) ≥ c d i r π t \sin(\tau/2)\ge\frac{c_{\mathrm{dir}}}{\pi t} sin ( τ /2 ) ≥ π t c dir
Thus it suffices to lower bound ∥ μ − ∥ 2 \left\|\mu_{-}\right\|_{2} ∥ μ − ∥ 2 .
Write the decomposition from Lemma B.15:
p − ∥ μ − ∥ 2 = ( 1 − 2 g ( 0 ) ) ϕ ( t ) + E [ − g ′ ( U ) ] , p_{-}\left\|\mu_{-}\right\|_{2}=(1-2g(0))\phi(t)+\mathbb{E}[-g'(U)], p − ∥ μ − ∥ 2 = ( 1 − 2 g ( 0 )) ϕ ( t ) + E [ − g ′ ( U )] ,
where U = ∣ Z + t ∣ U=|Z+t| U = ∣ Z + t ∣ .
Case 1: tail-like observed negatives. Assume p − ≤ 2 p p_{-}\le2p p − ≤ 2 p . On the event { Z < − t } \{Z<-t\} { Z < − t } , we have q t ( Z ) = 1 − g ( ∣ Z + t ∣ ) ≥ 1 − g ( 0 ) ≥ 1 / 2 q_{t}(Z)=1-g(|Z+t|)\ge1-g(0)\ge1/2 q t ( Z ) = 1 − g ( ∣ Z + t ∣ ) ≥ 1 − g ( 0 ) ≥ 1/2 , hence
P ( Z < − t ∣ Y = − 1 ) = E [ q t ( Z ) 1 { Z < − t } ] p − ≥ ( 1 / 2 ) P ( Z < − t ) p − ≥ ( 1 / 2 ) p 2 p = 1 4 \mathbb{P}(Z<-t\mid Y=-1)=\frac{\mathbb{E}[q_{t}(Z)\mathbf{1}\{Z<-t\}]}{p_{-}}\ge\frac{(1/2)\mathbb{P}(Z<-t)}{p_{-}}\ge\frac{(1/2)p}{2p}=\frac{1}{4} P ( Z < − t ∣ Y = − 1 ) = p − E [ q t ( Z ) 1 { Z < − t }] ≥ p − ( 1/2 ) P ( Z < − t ) ≥ 2 p ( 1/2 ) p = 4 1
In addition, q t q_{t} q t is nonincreasing in z z z , so when Z ≥ − t Z\ge-t Z ≥ − t , the weighting by q t q_{t} q t can only decrease the conditional mean. Thus
E [ Z ∣ Y = − 1 , Z ≥ − t ] = E [ Z 1 { Y = − 1 , Z ≥ − t } ] P ( Y = − 1 , Z ≥ − t ) = E [ Z q t ( Z ) 1 { Z ≥ − t } ] E [ q t ( Z ) 1 { Z ≥ − t } ] \begin{aligned}
\mathbb{E}[Z\mid Y=-1,Z\ge-t]=\frac{\mathbb{E}[Z\mathbf{1}\{Y=-1,Z\ge-t\}]}{\mathbb{P}(Y=-1,Z\ge-t)} \\
&=\frac{\mathbb{E}[Zq_{t}(Z)\mathbf{1}\{Z\ge-t\}]}{\mathbb{E}[q_{t}(Z)\mathbf{1}\{Z\ge-t\}]}
\end{aligned} E [ Z ∣ Y = − 1 , Z ≥ − t ] = P ( Y = − 1 , Z ≥ − t ) E [ Z 1 { Y = − 1 , Z ≥ − t }] = E [ q t ( Z ) 1 { Z ≥ − t }] E [ Z q t ( Z ) 1 { Z ≥ − t }]
From Lemma B.11 , this is bounded above by E [ Z ∣ Z ≥ − t ] ≤ ϕ ( t ) / Φ ( t ) \mathbb{E}[Z \mid Z \ge-t] \le\phi(t)/\Phi(t) E [ Z ∣ Z ≥ − t ] ≤ ϕ ( t ) /Φ ( t ) . For t ≥ 0 t \ge0 t ≥ 0 , ϕ ( t ) / Φ ( t ) \phi(t)/\Phi(t) ϕ ( t ) /Φ ( t ) is decreasing, and in the regime t ≥ 1 t \ge1 t ≥ 1 , ϕ ( t ) / Φ ( t ) ≤ ϕ ( 1 ) / Φ ( 1 ) ≤ 3 / 10 \phi(t)/\Phi(t) \le\phi(1)/\Phi(1) \le3/10 ϕ ( t ) /Φ ( t ) ≤ ϕ ( 1 ) /Φ ( 1 ) ≤ 3/10 , where we used that ϕ ( t ) / Φ ( t ) \phi(t)/\Phi(t) ϕ ( t ) /Φ ( t ) is decreasing for t ≥ 0 t \ge0 t ≥ 0 . Also, E [ Z ∣ Y = − 1 , Z < − t ] ≤ E [ Z ∣ Z < − t ] = − λ ( t ) ≤ − t \mathbb{E}[Z \mid Y=-1,Z<-t] \le\mathbb{E}[Z \mid Z<-t]=-\lambda(t)\le-t E [ Z ∣ Y = − 1 , Z < − t ] ≤ E [ Z ∣ Z < − t ] = − λ ( t ) ≤ − t . Therefore,
E [ Z ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 , Z < − t ] P ( Z < − t ∣ Y = − 1 ) + E [ Z ∣ Y = − 1 , Z ≥ − t ] P ( Z ≥ − t ∣ Y = − 1 ) ≤ ( − t ) ⋅ 1 4 + 3 10 ⋅ 3 4 = − 10 t − 9 40 ≤ − t 40 , \begin{aligned}
\mathbb{E}[Z \mid Y=-1] &= \mathbb{E}[Z \mid Y=-1,Z<-t]\mathbb{P}(Z<-t \mid Y=-1) \\
&\quad+ \mathbb{E}[Z \mid Y=-1,Z\ge-t]\mathbb{P}(Z\ge-t \mid Y=-1) \\
&\le(-t)\cdot\frac{1}{4}+\frac{3}{10}\cdot\frac{3}{4}=-\frac{10t-9}{40}\le-\frac{t}{40},
\end{aligned} E [ Z ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 , Z < − t ] P ( Z < − t ∣ Y = − 1 ) + E [ Z ∣ Y = − 1 , Z ≥ − t ] P ( Z ≥ − t ∣ Y = − 1 ) ≤ ( − t ) ⋅ 4 1 + 10 3 ⋅ 4 3 = − 40 10 t − 9 ≤ − 40 t ,
where the last inequality holds for all t ≥ 1 t \ge1 t ≥ 1 . Thus, in this case
∥ μ − ∥ 2 = − E [ Z ∣ Y = − 1 ] ≥ t 40 \|\mu_-\|_2=-\mathbb{E}[Z \mid Y=-1]\ge\frac{t}{40} ∥ μ − ∥ 2 = − E [ Z ∣ Y = − 1 ] ≥ 40 t
Plugging ∥ μ − ∥ 2 ≥ t / 40 \|\mu_-\|_2 \ge t/40 ∥ μ − ∥ 2 ≥ t /40 and sin ( τ / 2 ) ≥ c d i r / ( π t ) \sin(\tau/2)\ge c_{\mathrm{dir}}/(\pi t) sin ( τ /2 ) ≥ c dir / ( π t ) into Theorem B.13 shows that the estimation term is at most O ~ ( ( d + log ( 1 / δ ) ) / p − ) \widetilde{O}((d+\log(1/\delta))/p_-) O (( d + log ( 1/ δ )) / p − ) . Since p − ≥ p p_-\ge p p − ≥ p , this is at most O ~ ( ( d + log ( 1 / δ ) ) / p ) \widetilde{O}((d+\log(1/\delta))/p) O (( d + log ( 1/ δ )) / p ) .
Case 2: inflated observed negatives. Assume p − > 2 p p_->2p p − > 2 p . We begin with a deterministic lower bound on the average flip probability. From Lemma B.7 , we have p − − p = E [ g ( U ) tanh ( t U ) ] ≤ E [ g ( U ) ] p_- - p=\mathbb{E}[g(U)\tanh(tU)]\le\mathbb{E}[g(U)] p − − p = E [ g ( U ) tanh ( t U )] ≤ E [ g ( U )] , since tanh ( t U ) ≤ 1 \tanh(tU)\le1 tanh ( t U ) ≤ 1 for t U ≥ 0 tU\ge0 t U ≥ 0 . Moreover,
E [ g ( U ) ] ≥ E [ g ( U ) 1 { M > 0 } ] − E [ g ( U ) 1 { M < 0 } ] = p − − p ≥ p − 2 . \begin{aligned}
\mathbb{E}[g(U)] &\ge\mathbb{E}[g(U)\mathbf{1}\{M>0\}]-\mathbb{E}[g(U)\mathbf{1}\{M<0\}] \\
&=p_- - p \\
&\ge\frac{p_-}{2}.
\end{aligned} E [ g ( U )] ≥ E [ g ( U ) 1 { M > 0 }] − E [ g ( U ) 1 { M < 0 }] = p − − p ≥ 2 p − .
We claim that in this regime, either the variation term E [ − g ′ ( U ) ] \mathbb{E}[-g'(U)] E [ − g ′ ( U )] is already large enough, or we fall back to the 1 / p 1/p 1/ p term in the theorem statement. To prove this, we first split according to whether U U U lies above or below 1 1 1
E [ g ( U ) ] = E [ g ( U ) 1 { U < 1 } ] + E [ g ( U ) 1 { U ≥ 1 } ] \mathbb{E}[g(U)]=\mathbb{E}[g(U)\mathbf{1}\{U<1\}]+\mathbb{E}[g(U)\mathbf{1}\{U\ge1\}] E [ g ( U )] = E [ g ( U ) 1 { U < 1 }] + E [ g ( U ) 1 { U ≥ 1 }]
Subcase 2(a): E [ g ( U ) 1 { U ≥ 1 } ] ≥ p − / 4 \mathbb{E}[g(U)\mathbf{1}\{U\ge1\}]\ge p_-/4 E [ g ( U ) 1 { U ≥ 1 }] ≥ p − /4 .
Fix a universal constant L = 3 L=3 L = 3 and consider the truncation event { U ≤ L t } \{U\le Lt\} { U ≤ L t } . We first bound the Gaussian tail of U U U in terms of p = Φ ( − t ) p=\Phi(-t) p = Φ ( − t ) . Since U = ∣ Z + t ∣ U=|Z+t| U = ∣ Z + t ∣ and t ≥ 1 t\ge1 t ≥ 1 ,
P ( U > L t ) = P ( Z > ( L − 1 ) t ) + P ( Z < − ( L + 1 ) t ) ≤ 2 Φ ( − ( L − 1 ) t ) \begin{aligned}
\mathbb{P}(U>Lt) &= \mathbb{P}(Z>(L-1)t)+\mathbb{P}(Z<-(L+1)t) \\
&\le2\Phi(-(L-1)t)
\end{aligned} P ( U > L t ) = P ( Z > ( L − 1 ) t ) + P ( Z < − ( L + 1 ) t ) ≤ 2Φ ( − ( L − 1 ) t )
Using Mills bounds Φ ( − a ) ≤ ϕ ( a ) / a \Phi(-a)\le\phi(a)/a Φ ( − a ) ≤ ϕ ( a ) / a and Φ ( − t ) ≥ t t 2 + 1 ϕ ( t ) \Phi(-t)\ge\frac{t}{t^2+1}\phi(t) Φ ( − t ) ≥ t 2 + 1 t ϕ ( t ) , we obtain for t ≥ 1 t\ge1 t ≥ 1 and a = ( L − 1 ) t a=(L-1)t a = ( L − 1 ) t ,
Φ ( − ( L − 1 ) t ) Φ ( − t ) ≤ ϕ ( ( L − 1 ) t ) ( L − 1 ) t ⋅ t 2 + 1 t ϕ ( t ) ≤ 2 L − 1 exp ( − ( ( L − 1 ) 2 − 1 ) t 2 2 ) ≤ 2 L − 1 exp ( − ( L − 1 ) 2 − 1 2 ) = : c L \frac{\Phi(-(L-1)t)}{\Phi(-t)}
\le\frac{\phi((L-1)t)}{(L-1)t}\cdot\frac{t^2+1}{t\phi(t)}
\le\frac{2}{L-1}\exp\left(-\frac{((L-1)^2-1)t^2}{2}\right)
\le\frac{2}{L-1}\exp\left(-\frac{(L-1)^2-1}{2}\right)=:c_L Φ ( − t ) Φ ( − ( L − 1 ) t ) ≤ ( L − 1 ) t ϕ (( L − 1 ) t ) ⋅ tϕ ( t ) t 2 + 1 ≤ L − 1 2 exp ( − 2 (( L − 1 ) 2 − 1 ) t 2 ) ≤ L − 1 2 exp ( − 2 ( L − 1 ) 2 − 1 ) =: c L
For L = 3 L=3 L = 3 , c L = exp ( − 3 / 2 ) c_L=\exp(-3/2) c L = exp ( − 3/2 ) is an absolute constant smaller than 1 / 4 1/4 1/4 . Therefore,
P ( U > L t ) ≤ 2 c L p ≤ c L p − , \mathbb{P}(U>Lt)\le2c_Lp\le c_Lp_-, P ( U > L t ) ≤ 2 c L p ≤ c L p − ,
where the last inequality uses p − > 2 p p_->2p p − > 2 p . Since g ( ⋅ ) ≤ 1 / 2 g(\cdot)\le1/2 g ( ⋅ ) ≤ 1/2 ,
E [ g ( U ) 1 { U > L t } ] ≤ 1 2 P ( U > L t ) ≤ c L 2 p − \mathbb{E}[g(U)\mathbf{1}\{U>Lt\}]\le\frac{1}{2}\mathbb{P}(U>Lt)\le\frac{c_L}{2}p_- E [ g ( U ) 1 { U > L t }] ≤ 2 1 P ( U > L t ) ≤ 2 c L p −
Combining with the subcase assumption gives
E [ g ( U ) 1 { 1 ≤ U ≤ L t } ] ≥ E [ g ( U ) 1 { U ≥ 1 } ] − E [ g ( U ) 1 { U > L t } ] ≥ ( 1 4 − c L 2 ) p − ≥ 1 8 p − \mathbb{E}[g(U)\mathbf{1}\{1 \le U \le Lt\}] \ge\mathbb{E}[g(U)\mathbf{1}\{U \ge1\}] - \mathbb{E}[g(U)\mathbf{1}\{U > Lt\}] \ge\left(\frac{1}{4} - \frac{c_L}{2}\right)p_- \ge\frac{1}{8}p_- E [ g ( U ) 1 { 1 ≤ U ≤ L t }] ≥ E [ g ( U ) 1 { U ≥ 1 }] − E [ g ( U ) 1 { U > L t }] ≥ ( 4 1 − 2 c L ) p − ≥ 8 1 p −
By regular variation, for almost every u ≥ 1 u \ge1 u ≥ 1 with g ( u ) > 0 g(u)>0 g ( u ) > 0 , we have
− g ′ ( u ) ≥ c ( 1 + u ) ( log ( e + u ) ) C g ( u ) -g'(u) \ge\frac{c}{(1+u)(\log(e+u))^C}g(u) − g ′ ( u ) ≥ ( 1 + u ) ( log ( e + u ) ) C c g ( u )
On the interval 1 ≤ u ≤ L t 1 \le u \le Lt 1 ≤ u ≤ L t , we have
1 + u ≤ ( L + 1 ) t 1+u \le(L+1)t 1 + u ≤ ( L + 1 ) t
and, since L L L is an absolute constant,
log ( e + u ) ≤ log ( e + L t ) ≤ C L ′ log ( e + t ) \log(e+u) \le\log(e+Lt) \le C'_L\log(e+t) log ( e + u ) ≤ log ( e + L t ) ≤ C L ′ log ( e + t )
for a constant C L ′ C'_L C L ′ depending only on L L L . Hence, for almost every u ∈ [ 1 , L t ] u \in[1,Lt] u ∈ [ 1 , L t ] ,
− g ′ ( u ) ≥ c ′ t ( log ( e + t ) ) C g ( u ) -g'(u) \ge\frac{c'}{t(\log(e+t))^C}g(u) − g ′ ( u ) ≥ t ( log ( e + t ) ) C c ′ g ( u )
where c ′ = c ′ ( c , C ) > 0 c'=c'(c,C)>0 c ′ = c ′ ( c , C ) > 0 depends only on the regular-variation constants and on the fixed choice of L L L .
Taking expectations and restricting to ${1 \le U \le Lt} yields
E [ − g ′ ( U ) ] ≥ c ′ t ( log ( e + t ) ) C E [ g ( U ) 1 { 1 ≤ U ≤ L t } ] ≥ c ′ 8 ⋅ p − t ( log ( e + t ) ) C \mathbb{E}[-g'(U)] \ge\frac{c'}{t(\log(e+t))^C}\mathbb{E}[g(U)\mathbf{1}\{1 \le U \le Lt\}] \ge\frac{c'}{8}\cdot\frac{p_-}{t(\log(e+t))^C} E [ − g ′ ( U )] ≥ t ( log ( e + t ) ) C c ′ E [ g ( U ) 1 { 1 ≤ U ≤ L t }] ≥ 8 c ′ ⋅ t ( log ( e + t ) ) C p −
Using p − ∥ μ − ∥ 2 ≥ E [ − g ′ ( U ) ] p_-\lVert\mu_-\rVert_2 \ge\mathbb{E}[-g'(U)] p − ∥ μ − ∥ 2 ≥ E [ − g ′ ( U )] gives
∥ μ − ∥ 2 ≥ c ′ ′ t ( log ( e + t ) ) C \lVert\mu_-\rVert_2 \ge\frac{c''}{t(\log(e+t))^C} ∥ μ − ∥ 2 ≥ t ( log ( e + t ) ) C c ′′
for a constant $c''=c''(c,C)>0. Plugging this lower bound and sin ( τ / 2 ) ≥ c d i r / ( π t ) \sin(\tau/2) \ge c_{\mathrm{dir}}/(\pi t) sin ( τ /2 ) ≥ c dir / ( π t ) into Theorem B.13 gives the sample requirement
N ≳ O ~ ( ( d + log ( 1 / δ ) ) ⋅ t 4 ( log ( e + t ) ) 2 C p − ) , N \gtrsim\widetilde{O}\left((d+\log(1/\delta))\cdot\frac{t^4(\log(e+t))^{2C}}{p_-}\right), N ≳ O ( ( d + log ( 1/ δ )) ⋅ p − t 4 ( log ( e + t ) ) 2 C ) ,
as desired.
Subcase 2(b): E [ g ( U ) 1 { U ≥ 1 } ] < p − / 4 \mathbb{E}[g(U)\mathbf{1}\{U \ge1\}] < p_-/4 E [ g ( U ) 1 { U ≥ 1 }] < p − /4 .
Recall that in Case 2 we have E [ g ( U ) ] ≥ p − / 2 \mathbb{E}[g(U)] \ge p_-/2 E [ g ( U )] ≥ p − /2 . Hence
E [ g ( U ) 1 { U < 1 } ] = E [ g ( U ) ] − E [ g ( U ) 1 { U ≥ 1 } ] ≥ p − 4 \mathbb{E}[g(U)\mathbf{1}\{U<1\}] = \mathbb{E}[g(U)]-\mathbb{E}[g(U)\mathbf{1}\{U \ge1\}] \ge\frac{p_-}{4} E [ g ( U ) 1 { U < 1 }] = E [ g ( U )] − E [ g ( U ) 1 { U ≥ 1 }] ≥ 4 p −
Moreover, q t ( Z ) ≥ g ( U ) q_t(Z) \ge g(U) q t ( Z ) ≥ g ( U ) for all Z Z Z : if Z > − t Z>-t Z > − t , then q t ( Z ) = g ( Z + t ) = g ( U ) q_t(Z)=g(Z+t)=g(U) q t ( Z ) = g ( Z + t ) = g ( U ) , while if Z < − t Z<-t Z < − t , then q t ( Z ) = 1 − g ( U ) ≥ g ( U ) q_t(Z)=1-g(U)\ge g(U) q t ( Z ) = 1 − g ( U ) ≥ g ( U ) because g ( U ) ≤ 1 / 2 g(U)\le1/2 g ( U ) ≤ 1/2 . Therefore
P ( Y = − 1 , U < 1 ) = E [ q t ( Z ) 1 { U < 1 } ] ≥ E [ g ( U ) 1 { U < 1 } ] ≥ p − 4 \mathbb{P}(Y=-1,U<1)=\mathbb{E}[q_t(Z)\mathbf{1}\{U<1\}] \ge\mathbb{E}[g(U)\mathbf{1}\{U<1\}] \ge\frac{p_-}{4} P ( Y = − 1 , U < 1 ) = E [ q t ( Z ) 1 { U < 1 }] ≥ E [ g ( U ) 1 { U < 1 }] ≥ 4 p −
Let
A = { Z > 1 − t } A=\{Z>1-t\} A = { Z > 1 − t }
On A A A , we have Z > − t Z>-t Z > − t and U = Z + t ≥ 1 U=Z+t\ge1 U = Z + t ≥ 1 , so
P ( Y = − 1 , A ) = E [ g ( U ) 1 { A } ] ≤ E [ g ( U ) 1 { U ≥ 1 } ] < p − 4 \mathbb{P}(Y=-1,A)=\mathbb{E}[g(U)\mathbf{1}\{A\}] \le\mathbb{E}[g(U)\mathbf{1}\{U\ge1\}] < \frac{p_-}{4} P ( Y = − 1 , A ) = E [ g ( U ) 1 { A }] ≤ E [ g ( U ) 1 { U ≥ 1 }] < 4 p −
Consequently,
P ( A ∣ Y = − 1 ) < 1 4 and P ( A c ∣ Y = − 1 ) > 3 4 \mathbb{P}(A\mid Y=-1)<\frac{1}{4}
\qquad\text{and}\qquad
\mathbb{P}(A^{c}\mid Y=-1)>\frac{3}{4} P ( A ∣ Y = − 1 ) < 4 1 and P ( A c ∣ Y = − 1 ) > 4 3
Set a = 1 − t ≤ 0 a=1-t\le0 a = 1 − t ≤ 0 , so that A = { Z > a } A=\{Z>a\} A = { Z > a } and A c = { Z ≤ a } A^{c}=\{Z\le a\} A c = { Z ≤ a } . On both intervals A A A and A c A^{c} A c , the function q t ( z ) q_t(z) q t ( z ) is nonincreasing in z z z . If P ( Y = − 1 , A ) > 0 \mathbb{P}(Y=-1,A)>0 P ( Y = − 1 , A ) > 0 , applying Lemma B.11 to the law of Z Z Z conditioned on A A A , with h ( z ) = q t ( z ) h(z)=q_t(z) h ( z ) = q t ( z ) , gives
E [ Z ∣ Y = − 1 , A ] = E [ Z q t ( Z ) 1 { A } ] E [ q t ( Z ) 1 { A } ] ≤ E [ Z ∣ A ] = E [ Z ∣ Z > a ] \mathbb{E}[Z\mid Y=-1,A]=\frac{\mathbb{E}[Zq_t(Z)\mathbf{1}\{A\}]}{\mathbb{E}[q_t(Z)\mathbf{1}\{A\}]}\le\mathbb{E}[Z\mid A]=\mathbb{E}[Z\mid Z>a] E [ Z ∣ Y = − 1 , A ] = E [ q t ( Z ) 1 { A }] E [ Z q t ( Z ) 1 { A }] ≤ E [ Z ∣ A ] = E [ Z ∣ Z > a ]
If P ( Y = − 1 , A ) = 0 \mathbb{P}(Y=-1,A)=0 P ( Y = − 1 , A ) = 0 , this conditional expectation is irrelevant in the decomposition below. Similarly, applying Lemma B.11 to the law of Z Z Z conditioned on A c A^{c} A c gives
E [ Z ∣ Y = − 1 , A c ] = E [ Z q t ( Z ) 1 { A c } ] E [ q t ( Z ) 1 { A c } ] ≤ E [ Z ∣ A c ] = E [ Z ∣ Z ≤ a ] \mathbb{E}[Z\mid Y=-1,A^{c}]=\frac{\mathbb{E}[Zq_t(Z)\mathbf{1}\{A^{c}\}]}{\mathbb{E}[q_t(Z)\mathbf{1}\{A^{c}\}]}\le\mathbb{E}[Z\mid A^{c}]=\mathbb{E}[Z\mid Z\le a] E [ Z ∣ Y = − 1 , A c ] = E [ q t ( Z ) 1 { A c }] E [ Z q t ( Z ) 1 { A c }] ≤ E [ Z ∣ A c ] = E [ Z ∣ Z ≤ a ]
For a ≤ 0 a\le0 a ≤ 0 ,
E [ Z ∣ Z > a ] = ϕ ( a ) Φ ( − a ) ≤ ϕ ( 0 ) Φ ( 0 ) = 2 π \mathbb{E}[Z\mid Z>a]=\frac{\phi(a)}{\Phi(-a)}\le\frac{\phi(0)}{\Phi(0)}=\sqrt{\frac{2}{\pi}} E [ Z ∣ Z > a ] = Φ ( − a ) ϕ ( a ) ≤ Φ ( 0 ) ϕ ( 0 ) = π 2
and
E [ Z ∣ Z ≤ a ] = − ϕ ( a ) Φ ( a ) ≤ − ϕ ( 0 ) Φ ( 0 ) = − 2 π \mathbb{E}[Z\mid Z\le a]=-\frac{\phi(a)}{\Phi(a)}\le-\frac{\phi(0)}{\Phi(0)}=-\sqrt{\frac{2}{\pi}} E [ Z ∣ Z ≤ a ] = − Φ ( a ) ϕ ( a ) ≤ − Φ ( 0 ) ϕ ( 0 ) = − π 2
Combining, we have
E [ Z ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 , A c ] P ( A c ∣ Y = − 1 ) + E [ Z ∣ Y = − 1 , A ] P ( A ∣ Y = − 1 ) ≤ − 2 π ⋅ 3 4 + 2 π ⋅ 1 4 = − 1 2 2 π \begin{aligned}
\mathbb{E}[Z\mid Y=-1]&=\mathbb{E}[Z\mid Y=-1,A^{c}]\mathbb{P}(A^{c}\mid Y=-1)+\mathbb{E}[Z\mid Y=-1,A]\mathbb{P}(A\mid Y=-1)\\
&\le-\sqrt{\frac{2}{\pi}}\cdot\frac{3}{4}+\sqrt{\frac{2}{\pi}}\cdot\frac{1}{4}\\
&=-\frac{1}{2}\sqrt{\frac{2}{\pi}}
\end{aligned} E [ Z ∣ Y = − 1 ] = E [ Z ∣ Y = − 1 , A c ] P ( A c ∣ Y = − 1 ) + E [ Z ∣ Y = − 1 , A ] P ( A ∣ Y = − 1 ) ≤ − π 2 ⋅ 4 3 + π 2 ⋅ 4 1 = − 2 1 π 2
Therefore
∥ μ − ∥ 2 = − E [ Z ∣ Y = − 1 ] ≥ c 0 \lVert\mu_{-}\rVert_{2}=-\mathbb{E}[Z\mid Y=-1]\ge c_{0} ∥ μ − ∥ 2 = − E [ Z ∣ Y = − 1 ] ≥ c 0
for a universal constant c 0 > 0 c_{0}>0 c 0 > 0 .
Applying Theorem B.13 with target angle τ = 1 / t \tau=1/t τ = 1/ t , and using sin ( τ / 2 ) ≥ c d i r / ( π t ) \sin(\tau/2)\ge c_{\mathrm{dir}}/(\pi t) sin ( τ /2 ) ≥ c dir / ( π t ) , the required sample size in this subcase is
N ≥ O ~ ( ( d + log ( 1 / δ ) ) t 2 p − ) N\ge\widetilde{O}\left((d+\log(1/\delta))\frac{t^{2}}{p_{-}}\right) N ≥ O ( ( d + log ( 1/ δ )) p − t 2 )
This is implied by the stated bound, since for every t ≥ 1 t\ge1 t ≥ 1 ,
t 2 p − ≤ t 4 ( log ( e + t ) ) 2 C p − \frac{t^{2}}{p_{-}}\le\frac{t^{4}(\log(e+t))^{2C}}{p_{-}} p − t 2 ≤ p − t 4 ( log ( e + t ) ) 2 C
Rare-label reconciliation: Algorithm 1 versus the Y = − 1 Y=-1 Y = − 1 analysis
Lemma B.17 (Reconciling empirical rare-label selection with the Y = − 1 Y=-1 Y = − 1 analysis). Assume t > 0 t>0 t > 0 and margin-dependent noise with profile g : [ 0 , ∞ ) → [ 0 , 1 / 2 ] g:[0,\infty)\to[0,1/2] g : [ 0 , ∞ ) → [ 0 , 1/2 ] . Then p − ≤ 1 / 2 p_{-}\le1/2 p − ≤ 1/2 . Recall Algorithm 1 breaks ties in favor of − 1 -1 − 1 .
If p − ≤ 1 / 4 p_{-}\le1/4 p − ≤ 1/4 , then
P ( y ^ r t ≠ − 1 ) ≤ exp ( − N / 8 ) \mathbb{P}(\widehat{y}_{\mathrm{rt}}\ne-1)\le\exp(-N/8) P ( y rt = − 1 ) ≤ exp ( − N /8 ) In particular, if N ≥ 8 log ( 2 / δ ) N\ge8\log(2/\delta) N ≥ 8 log ( 2/ δ ) , then
P ( y ^ r t ≠ − 1 ) ≤ δ / 2 \mathbb{P}(\widehat{y}_{\mathrm{rt}}\ne-1)\le\delta/2 P ( y rt = − 1 ) ≤ δ /2
If p − ≥ 1 / 4 p_{-} \ge1/4 p − ≥ 1/4 and μ − ≠ 0 \mu_{-} \ne0 μ − = 0 , then using the Y = + 1 Y=+1 Y = + 1 class instead of the Y = − 1 Y=-1 Y = − 1 class changes the leading conditional-mean sample complexity term by at most a universal constant factor. More precisely,
1 / ( p + ∥ μ + ∥ 2 2 ) 1 / ( p − ∥ μ − ∥ 2 2 ) = p + p − ≤ 3 \frac{1/(p_{+}\lVert\mu_{+}\rVert_{2}^{2})}{1/(p_{-}\lVert\mu_{-}\rVert_{2}^{2})} = \frac{p_{+}}{p_{-}} \le3 1/ ( p − ∥ μ − ∥ 2 2 ) 1/ ( p + ∥ μ + ∥ 2 2 ) = p − p + ≤ 3
Proof. Let M = Z + t M=Z+t M = Z + t and h ( u ) = 1 / 2 − g ( u ) ≥ 0 h(u)=1/2-g(u)\ge0 h ( u ) = 1/2 − g ( u ) ≥ 0 . Since
p − − 1 2 = − E [ h ( ∣ M ∣ ) sign ( M ) ] p_{-}-\frac{1}{2}=-\mathbb{E}[h(|M|)\operatorname{sign}(M)] p − − 2 1 = − E [ h ( ∣ M ∣ ) sign ( M )]
and
E [ h ( ∣ M ∣ ) sign ( M ) ] = ∫ 0 ∞ h ( u ) ( ϕ ( u − t ) − ϕ ( u + t ) ) d u \mathbb{E}[h(|M|)\operatorname{sign}(M)] = \int_{0}^{\infty} h(u)\bigl(\phi(u-t)-\phi(u+t)\bigr)\,du E [ h ( ∣ M ∣ ) sign ( M )] = ∫ 0 ∞ h ( u ) ( ϕ ( u − t ) − ϕ ( u + t ) ) d u
while ϕ ( u − t ) ≥ ϕ ( u + t ) \phi(u-t)\ge\phi(u+t) ϕ ( u − t ) ≥ ϕ ( u + t ) for all u ≥ 0 u\ge0 u ≥ 0 and t > 0 t>0 t > 0 , we get
p − − 1 2 ≤ 0 p_{-}-\frac{1}{2}\le0 p − − 2 1 ≤ 0
Therefore, p − ≤ 1 / 2 p_{-}\le1/2 p − ≤ 1/2 .
For the first claim, let N − = ∑ i = 1 N 1 { Y i = − 1 } N_{-}=\sum_{i=1}^{N}\mathbb{1}\{Y_i=-1\} N − = ∑ i = 1 N 1 { Y i = − 1 } . Since ties are broken in favor of − 1 -1 − 1 ,
{ y ^ r t ≠ − 1 } = { N − > N / 2 } \{\widehat{y}_{\mathrm{rt}}\ne-1\}=\{N_{-}>N/2\} { y rt = − 1 } = { N − > N /2 }
Since N − ∼ Bin ( N , p − ) N_{-}\sim\operatorname{Bin}(N,p_{-}) N − ∼ Bin ( N , p − ) , Hoeffding’s inequality gives
P ( N − > N / 2 ) = P ( N − N − p − > 1 2 − p − ) ≤ exp ( − 2 N ( 1 2 − p − ) 2 ) \mathbb{P}(N_{-}>N/2)=\mathbb{P}\left(\frac{N_{-}}{N}-p_{-}>\frac{1}{2}-p_{-}\right)\le\exp\left(-2N\left(\frac{1}{2}-p_{-}\right)^{2}\right) P ( N − > N /2 ) = P ( N N − − p − > 2 1 − p − ) ≤ exp ( − 2 N ( 2 1 − p − ) 2 )
If p − ≤ 1 / 4 p_{-}\le1/4 p − ≤ 1/4 , then 1 / 2 − p − ≥ 1 / 4 1/2-p_{-}\ge1/4 1/2 − p − ≥ 1/4 , so
P ( y ^ r t ≠ − 1 ) ≤ exp ( − N / 8 ) \mathbb{P}(\widehat{y}_{\mathrm{rt}}\ne-1)\le\exp(-N/8) P ( y rt = − 1 ) ≤ exp ( − N /8 )
For the second claim, E [ X ] = 0 \mathbb{E}[X]=0 E [ X ] = 0 gives
p − μ − + p + μ + = 0 p_{-}\mu_{-}+p_{+}\mu_{+}=0 p − μ − + p + μ + = 0
and hence
μ + = − p − p + μ − , ∥ μ + ∥ 2 = p − p + ∥ μ − ∥ 2 \mu_{+}=-\frac{p_{-}}{p_{+}}\mu_{-},\qquad\lVert\mu_{+}\rVert_{2}=\frac{p_{-}}{p_{+}}\lVert\mu_{-}\rVert_{2} μ + = − p + p − μ − , ∥ μ + ∥ 2 = p + p − ∥ μ − ∥ 2
Therefore
1 p + ∥ μ + ∥ 2 2 = p + p − ⋅ 1 p − ∥ μ − ∥ 2 2 \frac{1}{p_{+}\lVert\mu_{+}\rVert_{2}^{2}}=\frac{p_{+}}{p_{-}}\cdot\frac{1}{p_{-}\lVert\mu_{-}\rVert_{2}^{2}} p + ∥ μ + ∥ 2 2 1 = p − p + ⋅ p − ∥ μ − ∥ 2 2 1
Since p − ≥ 1 / 4 p_{-}\ge1/4 p − ≥ 1/4 and p + ≤ 3 / 4 p_{+}\le3/4 p + ≤ 3/4 , we have p + / p − ≤ 3 p_{+}/p_{-}\le3 p + / p − ≤ 3 . □
Threshold estimation
The rare-tail mean estimator returns a direction. Given an independent validation sample, we estimate the threshold by empirical risk minimization over one-dimensional thresholds along this direction. Throughout this subsection, we use the convention sign ( 0 ) = + 1 \operatorname{sign}(0)=+1 sign ( 0 ) = + 1 .
Note that the algorithm is finite, despite the optimization being over all real thresholds. If s i = ⟨ w , X ^ i ⟩ s_i=\langle w,\widehat{X}_i\rangle s i = ⟨ w , X i ⟩ , then sign ( s i + b ) = − 1 \operatorname{sign}(s_i+b)=-1 sign ( s i + b ) = − 1 implies s i < − b s_i<-b s i < − b . As the cutoff − b -b − b moves from left to right, predictions change only when the cutoff crosses one of the projected values s i s_i s i . The algorithm sorts the distinct projected values and scans the K + 1 K+1 K + 1 possible threshold regions. Initially, the cutoff lies to the left of all projected values, so all points are predicted + 1 +1 + 1 and the number of mistakes is equal to the number of negative labels. When the scan crosses a group G j G_j G j , all points in G j G_j G j switch prediction from + 1 +1 + 1 to − 1 -1 − 1 , i.e., each negative label in G j G_j G j removes exactly one mistake, and each positive label in G j G_j G j adds exactly one mistake.
Algorithm 2 THRESHOLD ERM ALONG A FIXED DIRECTION
Require: Samples $S_{\mathrm{thr}} = \{(X_i,Y_i)\}_{i=1}^{k}$ with $X_i \in \mathbb{R}^{d}$ and $Y_i \in \{\pm 1\}$, and a unit vector $\widehat{w} \in \mathbb{S}^{d-1}$
Ensure: A threshold estimate $\widehat{t} \in \mathbb{R}$
1: for $i = 1,\ldots,k$ do
2: $s_i \leftarrow \langle \widehat{w},X_i\rangle$
3: end for
4: Let $r_1 < r_2 < \cdots < r_K$ be the $K$ distinct values among $\{s_i\}_{i=1}^{k}$
5: $\operatorname{err} \leftarrow \sum_{i=1}^{k}\mathbf{1}\{Y_i=-1\}$, $\operatorname{err}_{\min} \leftarrow \operatorname{err}$, $j^\star \leftarrow 0$
6: for $j = 1,\ldots,K$ do
7: $G_j \leftarrow \{i \in [k] : s_i = r_j\}$
8: $\operatorname{err} \leftarrow \operatorname{err} - \sum_{i\in G_j}\mathbf{1}\{Y_i=-1\} + \sum_{i\in G_j}\mathbf{1}\{Y_i=+1\}$
9: if $\operatorname{err} < \operatorname{err}_{\min}$ then
10: $\operatorname{err}_{\min} \leftarrow \operatorname{err}$, $j^\star \leftarrow j$
11: end if
12: end for
13: if $j^\star = 0$ then
14: $\widehat{t} \leftarrow -r_1$
15: else if $j^\star = K$ then
16: $\widehat{t} \leftarrow -r_K-1$
17: else
18: $\widehat{t} \leftarrow -\frac{1}{2}(r_{j^\star}+r_{j^\star+1})$
19: end if
20: return $\widehat{t}$
Lemma B.18 (Sample complexity of one-dimensional threshold ERM). Fix any unit vector w ∈ S d − 1 w \in\mathbb{S}^{d-1} w ∈ S d − 1 and consider the one-parameter threshold class
H w = { h b ( x ) = sign ( ⟨ w , x ⟩ + b ) ∣ b ∈ R } \mathcal{H}_w = \{h_b(x) = \operatorname{sign}(\langle w,x\rangle+b) \mid b \in\mathbb{R}\} H w = { h b ( x ) = sign (⟨ w , x ⟩ + b ) ∣ b ∈ R } Let S t h r = { ( X i , Y i ) } i = 1 k S_{\mathrm{thr}} = \{(X_i,Y_i)\}_{i=1}^{k} S thr = {( X i , Y i ) } i = 1 k be i.i.d. samples from an arbitrary distribution over R d × { ± 1 } \mathbb{R}^{d} \times\{\pm1\} R d × { ± 1 } . For h : R d → { ± 1 } h:\mathbb{R}^{d}\to\{\pm1\} h : R d → { ± 1 } , define
err ( h ) = P ( h ( X ) ≠ Y ) \operatorname{err}(h) = \mathbb{P}(h(X) \ne Y) err ( h ) = P ( h ( X ) = Y ) err ^ ( h ) = 1 k ∑ i = 1 k 1 { h ( X i ) ≠ Y i } \widehat{\operatorname{err}}(h) = \frac{1}{k}\sum_{i=1}^{k}\mathbf{1}\{h(X_i)\ne Y_i\} err ( h ) = k 1 i = 1 ∑ k 1 { h ( X i ) = Y i } Let b ^ \widehat{b} b be any empirical risk minimizer over H w \mathcal{H}_w H w ,
b ^ ∈ arg min b ∈ R err ^ ( h b ) \widehat{b} \in\underset{b\in\mathbb{R}}{\arg\min}\ \widehat{\operatorname{err}}(h_b) b ∈ b ∈ R arg min err ( h b ) Then there is a universal constant C > 0 C>0 C > 0 such that, for every ε t h r , δ t h r ∈ ( 0 , 1 ) \varepsilon_{\mathrm{thr}},\delta_{\mathrm{thr}}\in(0,1) ε thr , δ thr ∈ ( 0 , 1 ) , if
k ≥ C ε t h r 2 ( log ( 1 ε t h r ) + log ( 1 δ t h r ) ) k \ge\frac{C}{\varepsilon_{\mathrm{thr}}^{2}}\left(\log\left(\frac{1}{\varepsilon_{\mathrm{thr}}}\right)+\log\left(\frac{1}{\delta_{\mathrm{thr}}}\right)\right) k ≥ ε thr 2 C ( log ( ε thr 1 ) + log ( δ thr 1 ) ) then, with probability at least 1 − δ t h r 1-\delta_{\mathrm{thr}} 1 − δ thr ,
err ( h b ^ ) ≤ inf b ∈ R err ( h b ) + ε t h r \operatorname{err}(h_{\widehat{b}}) \le\inf_{b\in\mathbb{R}}\operatorname{err}(h_b)+\varepsilon_{\mathrm{thr}} err ( h b ) ≤ b ∈ R inf err ( h b ) + ε thr
Proof. Consider the loss class
L w = { ( x , y ) ↦ 1 { h b ( x ) ≠ y } : b ∈ R } \mathcal{L}_w = \{(x,y)\mapsto\mathbf{1}\{h_b(x)\ne y\}:b\in\mathbb{R}\} L w = {( x , y ) ↦ 1 { h b ( x ) = y } : b ∈ R }
Each classifier h b h_b h b depends on x x x only through the scalar projection ⟨ w , x ⟩ \langle w,x\rangle ⟨ w , x ⟩ . Thus, after setting s i = ⟨ w , X i ⟩ s_i=\langle w,X_i\rangle s i = ⟨ w , X i ⟩ , the class H w \mathcal{H}_w H w induces exactly the same labelings on X 1 , … , X k X_1,\ldots,X_k X 1 , … , X k as ordinary thresholds on the real numbers s 1 , … , s k s_1,\ldots,s_k s 1 , … , s k . In particular, after sorting the s i s_i s i ’s, a threshold can only choose an initial segment to label − 1 -1 − 1 and the remaining points to label + 1 +1 + 1 , so there are at most k + 1 k+1 k + 1 distinct labelings. The same is true for the loss class L w \mathcal{L}_w L w , because the labels Y i Y_i Y i are fixed when counting empirical labelings. Thus L w \mathcal{L}_w L w has VC dimension at most 1. Next, we apply the standard VC uniform convergence theorem from Wainwright [35 Chapter 4] to L w \mathcal{L}_{w} L w . There is a universal constant C > 0 C > 0 C > 0 such that, if
k ≥ C ε t h r 2 ( log ( 1 ε t h r ) + log ( 1 δ t h r ) ) k \ge\frac{C}{\varepsilon_{\mathrm{thr}}^{2}}\left(\log\left(\frac{1}{\varepsilon_{\mathrm{thr}}}\right)+\log\left(\frac{1}{\delta_{\mathrm{thr}}}\right)\right) k ≥ ε thr 2 C ( log ( ε thr 1 ) + log ( δ thr 1 ) )
then, with probability at least 1 − δ t h r 1-\delta_{\mathrm{thr}} 1 − δ thr ,
sup b ∈ R ∣ err ( h b ) − err ^ ( h b ) ∣ ≤ ε t h r 2 \sup_{b\in\mathbb{R}}\left|\operatorname{err}(h_b)-\widehat{\operatorname{err}}(h_b)\right|\le\frac{\varepsilon_{\mathrm{thr}}}{2} b ∈ R sup ∣ err ( h b ) − err ( h b ) ∣ ≤ 2 ε thr
On this event,
err ( h ^ b ) ≤ err ^ ( h ^ b ) + ε t h r 2 = inf b ∈ R err ^ ( h b ) + ε t h r 2 ≤ inf b ∈ R ( err ( h b ) + ε t h r 2 ) + ε t h r 2 = inf b ∈ R err ( h b ) + ε t h r \begin{aligned}
\operatorname{err}(\widehat{h}_b)\le\widehat{\operatorname{err}}(\widehat{h}_b)+\frac{\varepsilon_{\mathrm{thr}}}{2} \\
&= \inf_{b\in\mathbb{R}}\widehat{\operatorname{err}}(h_b)+\frac{\varepsilon_{\mathrm{thr}}}{2} \\
&\le\inf_{b\in\mathbb{R}}\left(\operatorname{err}(h_b)+\frac{\varepsilon_{\mathrm{thr}}}{2}\right)+\frac{\varepsilon_{\mathrm{thr}}}{2} \\
&= \inf_{b\in\mathbb{R}}\operatorname{err}(h_b)+\varepsilon_{\mathrm{thr}}
\end{aligned} err ( h b ) ≤ err ( h b ) + 2 ε thr = b ∈ R inf err ( h b ) + 2 ε thr ≤ b ∈ R inf ( err ( h b ) + 2 ε thr ) + 2 ε thr = b ∈ R inf err ( h b ) + ε thr
This proves the claim. □
Theorem B.19. (Excess risk after one-dimensional threshold ERM ). Assume the margin-dependent noise model with Bayes halfspace f ⋆ ( x ) = sign ( ⟨ w ⋆ , x ⟩ + t ) f^{\star}(x)=\operatorname{sign}(\langle w^{\star},x\rangle+t) f ⋆ ( x ) = sign (⟨ w ⋆ , x ⟩ + t ) , where t ≥ 1 t\ge1 t ≥ 1 and p = Φ ( − t ) p=\Phi(-t) p = Φ ( − t ) . Fix ε , δ t h r ∈ ( 0 , 1 ) \varepsilon,\delta_{\mathrm{thr}}\in(0,1) ε , δ thr ∈ ( 0 , 1 ) , and assume the critical high-bias scaling p ≤ A ε p\le A\varepsilon p ≤ A ε for some constant A > 0 A>0 A > 0 . Let w ^ \widehat{w} w be independent of the threshold sample and suppose
∠ ( w ^ , w ⋆ ) ≤ c t \angle(\widehat{w},w^{\star})\le\frac{c}{t} ∠ ( w , w ⋆ ) ≤ t c for a sufficiently small constant c = c ( A ) > 0 c=c(A)>0 c = c ( A ) > 0 . Let t ^ \widehat{t} t be the output of Algorithm 2 run with direction w ^ \widehat{w} w on an independent threshold sample of size k k k . If
k ≥ C ε − 2 ( log ( 1 ε ) + log ( 1 δ t h r ) ) k\ge C\varepsilon^{-2}\left(\log\left(\frac{1}{\varepsilon}\right)+\log\left(\frac{1}{\delta_{\mathrm{thr}}}\right)\right) k ≥ C ε − 2 ( log ( ε 1 ) + log ( δ thr 1 ) ) for a sufficiently large universal constant C > 0 C>0 C > 0 , and if
h ^ ( x ) = sign ( ⟨ w ^ , x ⟩ + t ^ ) \widehat{h}(x)=\operatorname{sign}(\langle\widehat{w},x\rangle+\widehat{t}) h ( x ) = sign (⟨ w , x ⟩ + t ) then, with probability at least 1 − δ t h r 1-\delta_{\mathrm{thr}} 1 − δ thr over the threshold sample,
err ( h ^ ) − err ( f ⋆ ) ≤ ε \operatorname{err}(\widehat{h})-\operatorname{err}(f^{\star})\le\varepsilon err ( h ) − err ( f ⋆ ) ≤ ε
Proof. Condition on the direction w ^ \widehat{w} w . Since the threshold sample is independent of the direction-estimation sample, Lemma B.18 applies to the fixed one-dimensional threshold class
H w ^ = { x ↦ sign ( ⟨ w ^ , x ⟩ + b ) : b ∈ R } \mathcal{H}_{\widehat{w}}=\{x\mapsto\operatorname{sign}(\langle\widehat{w},x\rangle+b):b\in\mathbb{R}\} H w = { x ↦ sign (⟨ w , x ⟩ + b ) : b ∈ R }
Apply Lemma B.18 with ε t h r = ε / 2 \varepsilon_{\mathrm{thr}}=\varepsilon/2 ε thr = ε /2 . Under the stated lower bound on k k k , increasing the universal constant C C C if necessary, with probability at least 1 − δ t h r 1-\delta_{\mathrm{thr}} 1 − δ thr we have
err ( h ^ ) ≤ inf b ∈ R err ( sign ( ⟨ w ^ , X ⟩ + b ) ) + ε 2 \operatorname{err}(\widehat{h})\le\inf_{b\in\mathbb{R}}\operatorname{err}(\operatorname{sign}(\langle\widehat{w},X\rangle+b))+\frac{\varepsilon}{2} err ( h ) ≤ b ∈ R inf err ( sign (⟨ w , X ⟩ + b )) + 2 ε
The threshold b = t b=t b = t is feasible in the infimum, hence
err ( h ^ ) − err ( f ⋆ ) ≤ err ( sign ( ⟨ w ^ , X ⟩ + t ) ) − err ( f ⋆ ) + ε 2 \operatorname{err}(\widehat{h})-\operatorname{err}(f^{\star})\le\operatorname{err}(\operatorname{sign}(\langle\widehat{w},X\rangle+t))-\operatorname{err}(f^{\star})+\frac{\varepsilon}{2} err ( h ) − err ( f ⋆ ) ≤ err ( sign (⟨ w , X ⟩ + t )) − err ( f ⋆ ) + 2 ε
By Lemma 2.1, since ∠ ( w ^ , w ⋆ ) ≤ c / t \angle(\widehat{w},w^{\star})\le c/t ∠ ( w , w ⋆ ) ≤ c / t , we have
err ( sign ( ⟨ w ^ , X ⟩ + t ) ) − err ( f ⋆ ) ≤ C 0 c t ϕ ( t ) \operatorname{err}(\operatorname{sign}(\langle\widehat{w},X\rangle+t))-\operatorname{err}(f^{\star})\le C_{0}\frac{c}{t}\phi(t) err ( sign (⟨ w , X ⟩ + t )) − err ( f ⋆ ) ≤ C 0 t c ϕ ( t )
For t ≥ 1 t\ge1 t ≥ 1 , Mills’ ratio gives
ϕ ( t ) ≤ 2 t Φ ( − t ) = 2 t p \phi(t)\le2t\Phi(-t)=2tp ϕ ( t ) ≤ 2 t Φ ( − t ) = 2 tp
Therefore
C 0 c t ϕ ( t ) ≤ 2 C 0 c p ≤ 2 C 0 c A ε C_0\frac{c}{t}\phi(t) \le2C_0cp \le2C_0cA\varepsilon C 0 t c ϕ ( t ) ≤ 2 C 0 c p ≤ 2 C 0 c A ε
Choosing c ≤ ( 4 C 0 A ) − 1 c \le(4C_0A)^{-1} c ≤ ( 4 C 0 A ) − 1 gives
C 0 c t ϕ ( t ) ≤ ε 2 C_0\frac{c}{t}\phi(t) \le\frac{\varepsilon}{2} C 0 t c ϕ ( t ) ≤ 2 ε
which then yields the final bound
err ( h ^ ) − err ( f ⋆ ) ≤ ε \operatorname{err}(\widehat{h})-\operatorname{err}(f^\star) \le\varepsilon err ( h ) − err ( f ⋆ ) ≤ ε
as claimed.
Finally, we record the running time of the standalone halfspace learner.
Proposition B.20 (Runtime of the final learner). Algorithm 1 followed by Algorithm 2 can be implemented in time
O ( d N d i r + d k + k log k ) O(dN_{\mathrm{dir}}+dk+k\log k) O ( d N dir + d k + k log k ) and memory
where N d i r N_{\mathrm{dir}} N dir is the number of samples used for direction estimation and m m m is the number of samples used for threshold estimation.
Proof. The direction estimation step counts the number of samples per label and computes the empirical mean of the rare class. This takes one pass over N d i r N_{\mathrm{dir}} N dir samples and costs O ( d N d i r ) O(dN_{\mathrm{dir}}) O ( d N dir ) arithmetic operations.
For threshold estimation, computing the projections s i = ⟨ w ^ , X i ⟩ s_i=\langle\widehat{w},X_i\rangle s i = ⟨ w , X i ⟩ costs O ( d k ) O(dk) O ( d k ) , while sorting the projected values costs O ( k log k ) O(k\log k) O ( k log k ) .
Post sorting, the scan over the distinct projected values is linear in k k k , since each sample enters only one group G j G_j G j and changes the tracked error count just once. This gives a total running time of
O ( d N d i r + d k + k log k ) O(dN_{\mathrm{dir}}+dk+k\log k) O ( d N dir + d k + k log k ) The memory bound follows from storing the d d d directions and the k k k projected scalar values.
Learning under probit noise
The natural first step is to formalize probit as an instance of regularly-varying margin-monotone profiles. Recall from Section 2.1
Definition C.1 (Probit noise). Fix σ > 0 \sigma>0 σ > 0 and let G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) be independent of X X X . In the probit model,
Y = sign ( ⟨ w ⋆ , X ⟩ + t + σ G ) Y=\operatorname{sign}(\langle w^\star,X\rangle+t+\sigma G) Y = sign (⟨ w ⋆ , X ⟩ + t + σ G )
Probit is regularly varying margin-monotone noise
Lemma C.2 (Probit is margin-monotone and regularly varying). For σ > 0 \sigma>0 σ > 0 , the probit model is margin-monotone with profile
g σ ( u ) = Φ ( − u / σ ) g_\sigma(u)=\Phi(-u/\sigma) g σ ( u ) = Φ ( − u / σ )
Moreover, g σ g_\sigma g σ obeys the regular-variation condition of Definition 2.2 with constants c σ = 1 / σ 2 c_\sigma=1/\sigma^2 c σ = 1/ σ 2 and C σ = 0 C_\sigma=0 C σ = 0 in the sense that, for every u ≥ 1 u\ge1 u ≥ 1 ,
− g σ ′ ( u ) ≥ c σ 1 + u g σ ( u ) -g_\sigma'(u)\ge\frac{c_\sigma}{1+u}g_\sigma(u) − g σ ′ ( u ) ≥ 1 + u c σ g σ ( u )
Proof. Let M = ⟨ w ∗ , X ⟩ + t M = \langle w^*, X\rangle+ t M = ⟨ w ∗ , X ⟩ + t . Conditioned on X X X , the clean label is sign ( M ) \operatorname{sign}(M) sign ( M ) , while the probit label is sign ( M + σ G ) \operatorname{sign}(M + \sigma G) sign ( M + σ G ) .
Since σ > 0 \sigma> 0 σ > 0 , if M > 0 M > 0 M > 0 , a label flip occurs only when
M + σ G < 0 M + \sigma G < 0 M + σ G < 0
which has probability Φ ( − M / σ ) \Phi(-M/\sigma) Φ ( − M / σ ) . If M < 0 M < 0 M < 0 , a label flip occurs only when
M + σ G ≥ 0 M + \sigma G \ge0 M + σ G ≥ 0
which has probability P ( G ≥ ∣ M ∣ / σ ) = Φ ( − ∣ M ∣ / σ ) \mathbb{P}(G \ge|M|/\sigma) = \Phi(-|M|/\sigma) P ( G ≥ ∣ M ∣/ σ ) = Φ ( − ∣ M ∣/ σ ) . In both cases,
P ( Y ≠ f ∗ ( X ) ∣ X ) = Φ ( − ∣ M ∣ / σ ) \mathbb{P}(Y \ne f^*(X) \mid X) = \Phi(-|M|/\sigma) P ( Y = f ∗ ( X ) ∣ X ) = Φ ( − ∣ M ∣/ σ )
and the profile therefore has the form g σ ( u ) = Φ ( − u / σ ) g_\sigma(u) = \Phi(-u/\sigma) g σ ( u ) = Φ ( − u / σ ) .
The function g σ g_\sigma g σ is nonincreasing, absolutely continuous, and satisfies g σ ( u ) → 0 g_\sigma(u) \to0 g σ ( u ) → 0 as u → ∞ u \to\infty u → ∞ . Moreover,
g σ ′ ( u ) = − 1 σ ϕ ( u / σ ) g_\sigma'(u) = -\frac{1}{\sigma}\phi(u/\sigma) g σ ′ ( u ) = − σ 1 ϕ ( u / σ )
and the relative decay is
− g σ ′ ( u ) g σ ( u ) = 1 σ ϕ ( u / σ ) Φ ( − u / σ ) = 1 σ λ ( u / σ ) -\frac{g_\sigma'(u)}{g_\sigma(u)} = \frac{1}{\sigma}\frac{\phi(u/\sigma)}{\Phi(-u/\sigma)} = \frac{1}{\sigma}\lambda(u/\sigma) − g σ ( u ) g σ ′ ( u ) = σ 1 Φ ( − u / σ ) ϕ ( u / σ ) = σ 1 λ ( u / σ )
By Mills’ lower bound, λ ( a ) ≥ a \lambda(a) \ge a λ ( a ) ≥ a for a > 0 a > 0 a > 0 . Therefore, for u ≥ 1 u \ge1 u ≥ 1 ,
− g σ ′ ( u ) g σ ( u ) ≥ u σ 2 ≥ 1 σ 2 ( 1 + u ) -\frac{g_\sigma'(u)}{g_\sigma(u)} \ge\frac{u}{\sigma^2} \ge\frac{1}{\sigma^2(1+u)} − g σ ( u ) g σ ′ ( u ) ≥ σ 2 u ≥ σ 2 ( 1 + u ) 1
which completes the proof.
Direction Recovery and Excess Risk
Estimating the direction is a direct consequence of the general Theorem B.13 .
Lemma C.3 (Direction estimation under probit). Assume the probit model with σ > 0 \sigma> 0 σ > 0 and t ≥ 1 t \ge1 t ≥ 1 , and let p σ = Φ ( − t / 1 + σ 2 ) p_\sigma= \Phi(-t/\sqrt{1+\sigma^2}) p σ = Φ ( − t / 1 + σ 2 ) . Fix a sufficiently small universal constant c > 0 c > 0 c > 0 . For every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , Algorithm 1 returns w ^ \widehat{w} w satisfying ∠ ( w ^ , w ∗ ) ≤ c / t \angle(\widehat{w}, w^*) \le c/t ∠ ( w , w ∗ ) ≤ c / t with probability at least 1 − δ 1-\delta 1 − δ , provided
N ≥ C ( 1 + σ 2 ) 2 log ( 2 / p σ ) ( d + log ( 1 / δ ) ) p σ N \ge C(1+\sigma^2)^2 \frac{\log(2/p_\sigma)(d+\log(1/\delta))}{p_\sigma} N ≥ C ( 1 + σ 2 ) 2 p σ log ( 2/ p σ ) ( d + log ( 1/ δ )) for a sufficiently large universal constant C > 0 C > 0 C > 0 .
Proof. Let H = ⟨ w ∗ , X ⟩ + σ G H = \langle w^*, X\rangle+ \sigma G H = ⟨ w ∗ , X ⟩ + σ G . Then H ∼ N ( 0 , 1 + σ 2 ) H \sim\mathcal{N}(0,1+\sigma^2) H ∼ N ( 0 , 1 + σ 2 ) , and Y = − 1 Y=-1 Y = − 1 precisely when H < − t H<-t H < − t , so p − = P ( Y = − 1 ) = p σ p_-=\mathbb{P}(Y=-1)=p_\sigma p − = P ( Y = − 1 ) = p σ . Since ( X , H ) (X,H) ( X , H ) is jointly Gaussian with Cov ( X , H ) = w ∗ \operatorname{Cov}(X,H)=w^* Cov ( X , H ) = w ∗ , we have
E [ ⟨ w ∗ , X ⟩ ∣ H ] = Cov ( ⟨ w ∗ , X ⟩ , H ) Var ( H ) H = H 1 + σ 2 \mathbb{E}[\langle w^*,X\rangle\mid H] = \frac{\operatorname{Cov}(\langle w^*,X\rangle,H)}{\operatorname{Var}(H)}H = \frac{H}{1+\sigma^2} E [⟨ w ∗ , X ⟩ ∣ H ] = Var ( H ) Cov (⟨ w ∗ , X ⟩ , H ) H = 1 + σ 2 H
where Cov ( ⟨ w ∗ , X ⟩ , H ) = ⟨ w ∗ , Cov ( X , H ) ⟩ = 1 \operatorname{Cov}(\langle w^*,X\rangle,H)=\langle w^*,\operatorname{Cov}(X,H)\rangle=1 Cov (⟨ w ∗ , X ⟩ , H ) = ⟨ w ∗ , Cov ( X , H )⟩ = 1 . The component of X X X orthogonal to w ∗ w^* w ∗ is independent of H H H and has mean zero. Conditioning further on H < − t H<-t H < − t and using that H / 1 + σ 2 H/\sqrt{1+\sigma^2} H / 1 + σ 2 is standard Gaussian, we obtain
μ − = E [ H ∣ H < − t ] 1 + σ 2 w ∗ = w ∗ p σ 1 + σ 2 ∫ − ∞ − t / 1 + σ 2 u ϕ ( u ) d u = − ϕ ( t / 1 + σ 2 ) p σ 1 + σ 2 w ∗ = − λ ( t / 1 + σ 2 ) 1 + σ 2 w ∗ \begin{aligned}
\mu_- = \frac{\mathbb{E}[H \mid H<-t]}{1+\sigma^2}w^* \\
&= \frac{w^*}{p_\sigma\sqrt{1+\sigma^2}}\int_{-\infty}^{-t/\sqrt{1+\sigma^2}}u\phi(u)\,du \\
&= -\frac{\phi(t/\sqrt{1+\sigma^2})}{p_\sigma\sqrt{1+\sigma^2}}w^*
= -\frac{\lambda(t/\sqrt{1+\sigma^2})}{\sqrt{1+\sigma^2}}w^*
\end{aligned} μ − = 1 + σ 2 E [ H ∣ H < − t ] w ∗ = p σ 1 + σ 2 w ∗ ∫ − ∞ − t / 1 + σ 2 u ϕ ( u ) d u = − p σ 1 + σ 2 ϕ ( t / 1 + σ 2 ) w ∗ = − 1 + σ 2 λ ( t / 1 + σ 2 ) w ∗
where the integral is evaluated using ϕ ′ ( u ) = − u ϕ ( u ) \phi'(u)=-u\phi(u) ϕ ′ ( u ) = − u ϕ ( u ) . From Mills’ lower bound, λ ( a ) ≥ a \lambda(a)\ge a λ ( a ) ≥ a for a > 0 a>0 a > 0 , which then gives ∥ μ − ∥ 2 ≥ t / ( 1 + σ 2 ) \|\mu_-\|_2\ge t/(1+\sigma^2) ∥ μ − ∥ 2 ≥ t / ( 1 + σ 2 ) .
All that remains is to apply Theorem B.13 with target angle τ = c / t \tau=c/t τ = c / t . Since t ≥ 1 t\ge1 t ≥ 1 , sin ( τ / 2 ) ≥ c / ( π t ) \sin(\tau/2)\ge c/(\pi t) sin ( τ /2 ) ≥ c / ( π t ) , and
∥ μ − ∥ 2 2 sin 2 ( τ / 2 ) ≥ t 2 ( 1 + σ 2 ) 2 c 2 π 2 t 2 = c 2 π 2 ( 1 + σ 2 ) 2 \|\mu_-\|_2^2\sin^2(\tau/2)\ge\frac{t^2}{(1+\sigma^2)^2}\frac{c^2}{\pi^2t^2}=\frac{c^2}{\pi^2(1+\sigma^2)^2} ∥ μ − ∥ 2 2 sin 2 ( τ /2 ) ≥ ( 1 + σ 2 ) 2 t 2 π 2 t 2 c 2 = π 2 ( 1 + σ 2 ) 2 c 2
Substituting this bound and p − = p σ p_-=p_\sigma p − = p σ into Theorem B.13 completes the proof.
It remains to reconcile direction recovery with the threshold ERM of Algorithm 2.
Theorem C.4 (Excess-risk guarantee under probit noise). Assume the probit model with σ > 0 \sigma>0 σ > 0 and t ≥ 1 t\ge1 t ≥ 1 . Let p = Φ ( − t ) = Θ ( ϵ ) p=\Phi(-t)=\Theta(\epsilon) p = Φ ( − t ) = Θ ( ϵ ) , p σ = Φ ( − t 1 + σ 2 ) p_\sigma=\Phi\left(-\frac{t}{\sqrt{1+\sigma^2}}\right) p σ = Φ ( − 1 + σ 2 t ) . There is an algorithm which, given
N = O ~ ( ( d + log ( 1 / δ ) ) ( 1 + σ 2 ) 2 p σ + 1 ϵ 2 ) N=\widetilde{O}\left(\frac{(d+\log(1/\delta))(1+\sigma^2)^2}{p_\sigma}+\frac{1}{\epsilon^2}\right) N = O ( p σ ( d + log ( 1/ δ )) ( 1 + σ 2 ) 2 + ϵ 2 1 ) independent probit samples, returns a halfspace h ^ \widehat{h} h such that
err ( h ^ ) − err ( f ⋆ ) ≤ ϵ \operatorname{err}(\widehat{h})-\operatorname{err}(f^\star)\le\epsilon err ( h ) − err ( f ⋆ ) ≤ ϵ with probability at least 1 − δ 1-\delta 1 − δ . The algorithm runs in time O ~ ( d N ) \widetilde{O}(dN) O ( d N )
Proof. As in proof of Theorem 3.3, we will first split the samples into two independent sets, and use the first for direction estimation via Algorithm 1 and the second for threshold estimation via Algorithm 2.
From Lemma C.3, applied with confidence parameter δ / 2 \delta/2 δ /2 and a sufficiently small constant c > 0 c>0 c > 0 such that Lemma 2.1 holds, we have that
N d i r = O ~ ( ( d + log ( 1 / δ ) ) ( 1 + σ 2 ) 2 p σ ) N_{\mathrm{dir}}=\widetilde{O}\left(\frac{(d+\log(1/\delta))(1+\sigma^2)^2}{p_\sigma}\right) N dir = O ( p σ ( d + log ( 1/ δ )) ( 1 + σ 2 ) 2 )
samples suffice to ensure ∠ ( w ^ , w ⋆ ) ≤ c t \angle(\widehat{w},w^\star)\le\frac{c}{t} ∠ ( w , w ⋆ ) ≤ t c with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 .
Conditioned on this event, Theorem B.19 applies with confidence parameter δ / 2 \delta/2 δ /2 . If the accuracy to which the threshold is to be computed is a sufficiently small constant multiple of ϵ \epsilon ϵ , then
k = O ~ ( 1 ϵ 2 ) k=\widetilde{O}\left(\frac{1}{\epsilon^2}\right) k = O ( ϵ 2 1 )
samples ensures
err ( h ^ ) − err ( f ⋆ ) ≤ ϵ \operatorname{err}(\widehat{h})-\operatorname{err}(f^\star)\le\epsilon err ( h ) − err ( f ⋆ ) ≤ ϵ
with probability at least 1 − δ / 2 1-\delta/2 1 − δ /2 . A union bound completes the proof for the sample complexity, and the runtime argument from the proof of Proposition B.20 applies verbatim. \hfill□ \square □
Results for Curriculum Learning
A General Learning Model
Although Section 4 defines a two-stage curriculum, for the sake of completeness, we record the following general formalization of curriculum learning and its instantiation in the Gaussian halfspace setting.
Definition D.1 (r r r -stage curriculum learning). Let X \mathcal{X} X be an instance space, let Y = { ± 1 } \mathcal{Y}=\{\pm1\} Y = { ± 1 } , and let H ⊆ Y X \mathcal{H}\subseteq\mathcal{Y}^{\mathcal{X}} H ⊆ Y X be a hypothesis class. Fix an integer r ≥ 1 r\ge1 r ≥ 1 and stage lengths n = ( n 1 , … , n r ) ∈ N r n=(n_1,\ldots,n_r)\in\mathbb{N}^r n = ( n 1 , … , n r ) ∈ N r .
An r r r -stage curriculum family is a collection of stage-indexed distribution families
D = ( { D h ( j ) : h ∈ H } ) j = 1 r (3) \mathcal{D}=\left(\left\{D_h^{(j)}:h\in\mathcal{H}\right\}\right)_{j=1}^{r} \tag*{(3)} D = ( { D h ( j ) : h ∈ H } ) j = 1 r (3)
where each D h ( j ) D_h^{(j)} D h ( j ) is a distribution over X × Y \mathcal{X} \times\mathcal{Y} X × Y . We say that the family is task-consistent across stages if, for every h ∈ H h \in\mathcal{H} h ∈ H and every stage j ∈ [ r ] j \in[r] j ∈ [ r ] , if ( X , Y ) ∼ D h ( j ) (X,Y) \sim D_h^{(j)} ( X , Y ) ∼ D h ( j ) and P X , h ( j ) P_{X,h}^{(j)} P X , h ( j ) denotes the X X X -marginal of D h ( j ) D_h^{(j)} D h ( j ) , then for P X , h ( j ) P_{X,h}^{(j)} P X , h ( j ) -almost every x ∈ X x \in\mathcal{X} x ∈ X ,
P ( Y = h ( x ) ∣ X = x ) ≥ 1 2 \mathbb{P}(Y = h(x) \mid X = x) \ge\frac{1}{2} P ( Y = h ( x ) ∣ X = x ) ≥ 2 1
An r r r -stage curriculum proceeds as follows: an unknown target h ⋆ ∈ H h^\star\in\mathcal{H} h ⋆ ∈ H is fixed. The learner receives r r r mutually independent stages of i.i.d. labeled samples
S j = { ( X i ( j ) , Y i ( j ) ) } i = 1 n j i.i.d. ∼ D h ⋆ ( j ) for j ∈ [ r ] S_j = \left\{(X_i^{(j)},Y_i^{(j)})\right\}_{i=1}^{n_j} \quad\text{i.i.d.} \sim D_{h^\star}^{(j)} \quad\text{for } j \in[r] S j = { ( X i ( j ) , Y i ( j ) ) } i = 1 n j i.i.d. ∼ D h ⋆ ( j ) for j ∈ [ r ]
in order. At stage j j j , it uses S j S_j S j and the state retained from the preceding stage to compute
M j = A j ( M j − 1 , S j ) M_j = \mathcal{A}_j(M_{j-1},S_j) M j = A j ( M j − 1 , S j )
where M 0 M_0 M 0 is fixed and the procedures A j \mathcal{A}_j A j may be randomized. This computation is completed before the next stage is revealed, and every subsequent stage can access earlier samples only through the retained state. After the final stage, the learner outputs a classifier h ^ = B ( M r ) \widehat{h} = \mathcal{B}(M_r) h = B ( M r ) , proper or improper.
We refer to the last-stage distribution D h ⋆ ( r ) D_{h^\star}^{(r)} D h ⋆ ( r ) as the target distribution, and to ( D h ⋆ ( 1 ) , … , D h ⋆ ( r − 1 ) ) (D_{h^\star}^{(1)},\ldots,D_{h^\star}^{(r-1)}) ( D h ⋆ ( 1 ) , … , D h ⋆ ( r − 1 ) ) as the curriculum distributions. Performance is measured by excess 0 − 1 0-1 0 − 1 risk on the target stage, i.e.,
err D h ⋆ ( r ) ( h ^ ) − err D h ⋆ ( r ) ( h ⋆ ) \operatorname{err}_{D_{h^\star}^{(r)}}(\widehat{h})-\operatorname{err}_{D_{h^\star}^{(r)}}(h^\star) err D h ⋆ ( r ) ( h ) − err D h ⋆ ( r ) ( h ⋆ )
Definition D.2 (Gaussian halfspace instantiation). Recall the Gaussian halfspace setting. For ( w , t ) ∈ S d − 1 × R (w,t) \in\mathbb{S}^{d-1} \times\mathbb{R} ( w , t ) ∈ S d − 1 × R , let
m w , t ( x ) = ⟨ w , x ⟩ + t , f w , t ( x ) = sign ( m w , t ( x ) ) m_{w,t}(x) = \langle w,x\rangle+ t,\qquad f_{w,t}(x) = \operatorname{sign}(m_{w,t}(x)) m w , t ( x ) = ⟨ w , x ⟩ + t , f w , t ( x ) = sign ( m w , t ( x ))
Fix an integer r ≥ 1 r \ge1 r ≥ 1 and choose stage-dependent noise profiles g 1 , … , g r : [ 0 , ∞ ) → [ 0 , 1 / 2 ] g_1,\ldots,g_r : [0,\infty) \to[0,1/2] g 1 , … , g r : [ 0 , ∞ ) → [ 0 , 1/2 ] .
For each stage j ∈ [ r ] j \in[r] j ∈ [ r ] and each halfspace f w , t f_{w,t} f w , t , let D w , t ( j ) D_{w,t}^{(j)} D w , t ( j ) be the distribution of ( X , Y ) (X,Y) ( X , Y ) obtained by drawing X ∼ N ( 0 , I d ) X \sim\mathcal{N}(0,I_d) X ∼ N ( 0 , I d ) and then generating Y Y Y according to
P ( Y ≠ f w , t ( X ) ∣ X = x ) = g j ( ∣ m w , t ( x ) ∣ ) \mathbb{P}(Y \ne f_{w,t}(X) \mid X = x) = g_j(\lvert m_{w,t}(x)\rvert) P ( Y = f w , t ( X ) ∣ X = x ) = g j (∣ m w , t ( x )∣)
Equivalently,
E [ Y ∣ X = x ] = ( 1 − 2 g j ( ∣ m w , t ( x ) ∣ ) ) f w , t ( x ) \mathbb{E}[Y \mid X = x] = \left(1 - 2g_j(\lvert m_{w,t}(x)\rvert)\right) f_{w,t}(x) E [ Y ∣ X = x ] = ( 1 − 2 g j (∣ m w , t ( x )∣) ) f w , t ( x )
Since g j ≤ 1 / 2 g_j \le1/2 g j ≤ 1/2 , the halfspace f w , t f_{w,t} f w , t is Bayes optimal for every stage, and this construction satisfies the task-consistency condition in Definition D.1.
A two-stage curriculum learner for η \eta η -RCN targets
Throughout this subsection, the source stage has regularly-varying margin-monotone noise (Definition 2.2) and the target stage is η \eta η -RCN with known constant η < 1 / 2 \eta< 1/2 η < 1/2 . As in the rest of the appendix, the analysis is written under the convention t > 0 t > 0 t > 0 without loss of generality.
Recall from Section 4 that, under η \eta η -RCN, the algorithm of Diakonikolas et al. [20 ] is expensive only in its first step. Finding a warm start within angle 1 / ( 5 t ) 1/(5t) 1/ ( 5 t ) of w ⋆ w^\star w ⋆ from the Average vector takes O ~ ( d / max { p , ε } 2 ) \widetilde{O}(d/\max\{p,\varepsilon\}^2) O ( d / max { p , ε } 2 ) samples, whereas turning this warm start into a halfspace with excess risk ε \varepsilon ε takes only O ~ ( d / ε ) \widetilde{O}(d/\varepsilon) O ( d / ε ) . Our curriculum learner (Algorithm 5) therefore computes the warm start from the source samples, where Algorithm 1 needs only O ~ ( d / p ) \widetilde{O}(d/p) O ( d / p ) of them, and leaves two tasks to the target samples: fitting the threshold, and shrinking the angle from O ( 1 / t ) O(1/t) O ( 1/ t ) to O n ( ε exp ( t 2 / 2 ) ) O_n(\varepsilon\exp(t^2/2)) O n ( ε exp ( t 2 /2 )) . This is a reduction by a factor of order p / ε p/\varepsilon p / ε , down to the scale at which Lemma 2.1 bounds the disagreement with f ⋆ f^\star f ⋆ by O ( ε ) O(\varepsilon) O ( ε ) . For this second task, we follow the Optimization procedure of Diakonikolas et al. [20 Algorithm 2] . We first describe this procedure, and then the changes that we make to it.
The band update. The refinement procedure of Diakonikolas et al. [20 ] uses target samples in a thin band next to its current guess of the separator. Given a direction w ∈ S d − 1 w \in\mathbb{S}^{d-1} w ∈ S d − 1 , a threshold guess t 0 t_0 t 0 , and a width γ 0 > 0 \gamma_0 > 0 γ 0 > 0 , the band is
E ( w , t 0 ) = { x : − t 0 ≤ ⟨ w , x ⟩ ≤ − t 0 + γ 0 } (4) \mathcal{E}(w,t_0) = \{x : -t_0 \le\langle w,x\rangle\le-t_0 + \gamma_0\} \tag*{(4)} E ( w , t 0 ) = { x : − t 0 ≤ ⟨ w , x ⟩ ≤ − t 0 + γ 0 } (4)
which lies on the side that the current classifier labels + 1 +1 + 1 . Each sample ( x , y ) (x,y) ( x , y ) in the band contributes the vector
g ( w ; x , y ) = 1 2 ( ( 1 − 2 η ) sign ( ⟨ w , x ⟩ + t 0 ) − y ) proj w ⊥ ( x ) (5) g(w;x,y) = \frac{1}{2}\left((1-2\eta)\operatorname{sign}(\langle w,x\rangle+t_0)-y\right)\operatorname{proj}_{w^\perp}(x) \tag*{(5)} g ( w ; x , y ) = 2 1 ( ( 1 − 2 η ) sign (⟨ w , x ⟩ + t 0 ) − y ) proj w ⊥ ( x ) (5)
where proj w ⊥ ( x ) = x − ⟨ w , x ⟩ w \operatorname{proj}_{w^\perp}(x)=x-\langle w,x\rangle w proj w ⊥ ( x ) = x − ⟨ w , x ⟩ w . The procedure estimates the conditional mean of these vectors on the band, takes a small step against this estimate, and normalizes, keeping t 0 t_0 t 0 fixed.
The reason for this update is that, within the range of the local descent lemma, the population mean is orthogonal to w w w and negatively correlated with w ∗ w^* w ∗ . We refer to Diakonikolas et al. [20 ] , Section 2.2 for the geometric argument, and establish the corresponding bound for our threshold guess in Lemma D.4 . We use the band width
γ 0 = ε 2 ( 1 − 2 η ) exp ( t 0 2 / 2 ) \gamma_0 = \frac{\varepsilon}{2(1-2\eta)}\exp(t_0^2/2) γ 0 = 2 ( 1 − 2 η ) ε exp ( t 0 2 /2 )
and describe our changes to the procedure below.
Our changes. In Diakonikolas et al. [20 ] , the band update is one of three components of the learner (their Algorithm 1). The warm start is the normalized Average vector, which is where the quadratic cost arises. The optimization is then run once for every threshold guess on a grid of spacing ε 2 / ( 8 ( 1 − 2 η ) 2 ) \varepsilon^2/(8(1-2\eta)^2) ε 2 / ( 8 ( 1 − 2 η ) 2 ) , which is the accuracy that their descent lemma requires of t 0 t_0 t 0 , and a final testing step selects one of the resulting O ( 1 / ε 2 ) O(1/\varepsilon^2) O ( 1/ ε 2 ) halfspaces. We keep the band update and replace the other two components. The warm start is the output of Algorithm 1 on the source samples. For the threshold, we show that the update keeps its direction whenever ∣ t 0 − t ∣ ≤ γ 0 |t_0-t|\leq\gamma_0 ∣ t 0 − t ∣ ≤ γ 0 (Lemma D.4 ). This is a much weaker requirement, since γ 0 ≥ ε / ( 2 ( 1 − 2 η ) ) \gamma_0\geq\varepsilon/(2(1-2\eta)) γ 0 ≥ ε / ( 2 ( 1 − 2 η )) , and it is met by a guess computed directly from the frequency of the label − 1 -1 − 1 , which equals η + ( 1 − 2 η ) p \eta+(1-2\eta)p η + ( 1 − 2 η ) p under η \eta η -RCN (Algorithm 3 and Lemma D.3 ). The refinement is therefore run only once, which is the source of the runtime gain in Table 1 , and the final threshold is fitted by Algorithm 2 along the refined direction.
Table 1. Bias-regime comparison for learning an η \eta η -RCN target. We suppress polylogarithmic factors in d d d , 1 / ε 1/\varepsilon 1/ ε , 1 / δ 1/\delta 1/ δ , t t t , and constants depending on fixed η < 1 / 2 \eta<1/2 η < 1/2 , and display only leading high-dimensional terms. Our curriculum learner uses source and target samples, while Diakonikolas et al. [20 ] use target samples only. For p < ε p<\varepsilon p < ε , a constant classifier suffices, and in the nontrivial regimes, the information-theoretic target sample rate is Θ ~ ( d / ε ) \widetilde{\Theta}(d/\varepsilon) Θ ( d / ε ) [30 ] .
We also change two details of the procedure itself, at points where we could not verify the analysis of Diakonikolas et al. [20 ] as stated. The first is the step size. Their schedule starts at μ 0 = ( 1 − 4 ρ ) 2 π / ( 16 ( 1 − 2 η ) ) \mu_0=(1-4\rho)\sqrt{2\pi}/(16(1-2\eta)) μ 0 = ( 1 − 4 ρ ) 2 π / ( 16 ( 1 − 2 η )) with ρ = 0.00098 \rho=0.00098 ρ = 0.00098 , which is calibrated for an initial angle of constant order, whereas the descent lemma, theirs as well as ours, holds only within angle 1 / ( 5 t ) 1/(5t) 1/ ( 5 t ) of w ∗ w^* w ∗ . With this schedule, the first steps can rotate the direction by angles larger than 1 / ( 5 t ) 1/(5t) 1/ ( 5 t ) once t ≥ 4 t\geq4 t ≥ 4 , and the induction of [20 Lemma 2.8] does not show that the iterates stay in the region where the descent lemma applies. We start the same schedule at the angle of the warm start instead (line 1 of Algorithm 4). The induction then goes through as before, and also shows that every iterate stays within angle 1 / ( 5 t ) 1/(5t) 1/ ( 5 t ) of w ∗ w^* w ∗ .
The second is the use of the samples. In Diakonikolas et al. [20 ] , every iteration uses the same sample, so each iterate depends on the sample that updates it, and the empirical update has to be close to its mean for all directions at once. Their concentration bound [20 Lemma 2.5] is proved for a fixed direction and extended by a union bound over a cover, but do not address directions outside the cover. It is also stated only for the angles covered by the descent lemma, while the induction applies it to iterates that are already closer to w ∗ w^* w ∗ . We instead split the sample into one batch per iteration, so that each iterate is independent of the batch that updates it, and prove a bound for a fixed direction that holds at every angle (Lemma D.5 ). Since there are O ( log ( 1 / ε ) ) O(\log(1/\varepsilon)) O ( log ( 1/ ε )) iterations, the refinement stage still uses O ~ η ( d + log ( 1 / δ ) ) / ε \widetilde{O}_\eta(d+\log(1/\delta))/\varepsilon O η ( d + log ( 1/ δ )) / ε target samples.
We begin the analysis with the threshold guess. Since the label − 1 -1 − 1 has frequency η + ( 1 − 2 η ) p \eta+(1-2\eta)p η + ( 1 − 2 η ) p under η \eta η -RCN, estimating this frequency to accuracy of order ε \varepsilon ε determines p p p , and hence t t t , up to a fraction of the band width.
Lemma D.3 (Band-scale threshold estimation from target RCN labels). Assume the target stage is η \eta η -RCN with known constant η < 1 / 2 \eta<1/2 η < 1/2 . Suppose t ≥ 1 t\geq1 t ≥ 1 and p = Φ ( − t ) ≥ C η ε p=\Phi(-t)\geq C_\eta\varepsilon p = Φ ( − t ) ≥ C η ε for a sufficiently large constant C η > 0 C_\eta>0 C η > 0 depending only on η \eta η . Define
γ ∗ = ε 2 ( 1 − 2 η ) exp ( t 2 / 2 ) \gamma_*=\frac{\varepsilon}{2(1-2\eta)}\exp(t^2/2) γ ∗ = 2 ( 1 − 2 η ) ε exp ( t 2 /2 )
Algorithm 3 RCN BAND-SCALE THRESHOLD FROM LABELS
Require: Target RCN samples S = { ( X i , Y i ) } i = 1 k S = \{(X_i, Y_i)\}_{i=1}^k S = {( X i , Y i ) } i = 1 k , known noise rate η < 1 / 2 \eta< 1/2 η < 1/2
Ensure: A threshold estimate t ^ \hat{t} t ^
N − ← ∑ i = 1 k 1 { Y i = − 1 } , N + ← ∑ i = 1 k 1 { Y i = + 1 } N_- \gets\sum_{i=1}^k \mathbf{1}\{Y_i=-1\},\quad N_+ \gets\sum_{i=1}^k \mathbf{1}\{Y_i=+1\} N − ← ∑ i = 1 k 1 { Y i = − 1 } , N + ← ∑ i = 1 k 1 { Y i = + 1 }
y ^ r t ← − 1 \hat{y}_{\mathrm{rt}} \gets-1 y ^ rt ← − 1 if N − ≤ N + N_- \le N_+ N − ≤ N + , and y ^ r t ← + 1 \hat{y}_{\mathrm{rt}} \gets+1 y ^ rt ← + 1 otherwise
q ^ r t ← k − 1 ∑ i = 1 k 1 { Y i = y ^ r t } \hat{q}_{\mathrm{rt}} \gets k^{-1}\sum_{i=1}^k \mathbf{1}\{Y_i=\hat{y}_{\mathrm{rt}}\} q ^ rt ← k − 1 ∑ i = 1 k 1 { Y i = y ^ rt }
p ^ r a w ← ( q ^ r t − η ) / ( 1 − 2 η ) \hat{p}_{\mathrm{raw}} \gets(\hat{q}_{\mathrm{rt}}-\eta)/(1-2\eta) p ^ raw ← ( q ^ rt − η ) / ( 1 − 2 η )
p ^ ← Π [ 0 , 1 / 2 ] ( p ^ r a w ) \hat{p} \gets\Pi_{[0,1/2]}(\hat{p}_{\mathrm{raw}}) p ^ ← Π [ 0 , 1/2 ] ( p ^ raw ) (clip to the valid bias range)
if p ^ = 0 \hat{p}=0 p ^ = 0 then
a ^ ← 1 \hat{a}\gets1 a ^ ← 1
else
a ^ ← Φ − 1 ( 1 − p ^ ) \hat{a}\gets\Phi^{-1}(1-\hat{p}) a ^ ← Φ − 1 ( 1 − p ^ )
end if
t ^ ← − y ^ r t a ^ \hat{t}\gets-\hat{y}_{\mathrm{rt}}\hat{a} t ^ ← − y ^ rt a ^
return t ^ \hat{t} t ^
For every constant a ∈ ( 0 , 1 / 2 ] a \in(0,1/2] a ∈ ( 0 , 1/2 ] , Algorithm 3 returns t ^ \hat{t} t ^ satisfying
∣ t ^ − t ∣ ≤ a γ ⋆ |\hat{t}-t| \le a\gamma_\star ∣ t ^ − t ∣ ≤ a γ ⋆
with probability at least 1 − δ 1-\delta 1 − δ , using
k = O η ( 1 a 2 ε 2 log ( 1 / δ ) ) k=O_\eta\left(\frac{1}{a^2\varepsilon^2}\log(1/\delta)\right) k = O η ( a 2 ε 2 1 log ( 1/ δ ) )
target RCN samples.
Proof. Since t > 0 t>0 t > 0 , the clean rare label is − 1 -1 − 1 . Under η \eta η -RCN,
q = P ( Y = − 1 ) = η + ( 1 − 2 η ) p q=\mathbb{P}(Y=-1)=\eta+(1-2\eta)p q = P ( Y = − 1 ) = η + ( 1 − 2 η ) p
and we write q ^ = N − / k \hat{q}=N_-/k q ^ = N − / k for the empirical frequency of the label − 1 -1 − 1 .
We first record that the band is narrow in the regime p ≥ C η ε p\ge C_\eta\varepsilon p ≥ C η ε . Since exp ( t 2 / 2 ) = 1 / ( 2 π ϕ ( t ) ) \exp(t^2/2)=1/(\sqrt{2\pi}\phi(t)) exp ( t 2 /2 ) = 1/ ( 2 π ϕ ( t )) and Φ ( − t ) ≤ ϕ ( t ) / t \Phi(-t)\le\phi(t)/t Φ ( − t ) ≤ ϕ ( t ) / t ,
t γ ⋆ = t ε 2 ( 1 − 2 η ) 2 π ϕ ( t ) ≤ ε 2 2 π ( 1 − 2 η ) p ≤ 1 2 2 π ( 1 − 2 η ) C η ≤ 1 40 (6) t\gamma_\star=\frac{t\varepsilon}{2(1-2\eta)\sqrt{2\pi}\phi(t)}
\le\frac{\varepsilon}{2\sqrt{2\pi}(1-2\eta)p}
\le\frac{1}{2\sqrt{2\pi}(1-2\eta)C_\eta}
\le\frac{1}{40} \tag*{(6)} t γ ⋆ = 2 ( 1 − 2 η ) 2 π ϕ ( t ) tε ≤ 2 2 π ( 1 − 2 η ) p ε ≤ 2 2 π ( 1 − 2 η ) C η 1 ≤ 40 1 (6)
where the last inequality holds once C η ≥ 8 / ( 1 − 2 η ) C_\eta\ge8/(1-2\eta) C η ≥ 8/ ( 1 − 2 η ) . Since t ≥ 1 t\ge1 t ≥ 1 , this also gives γ ⋆ ≤ 1 / 40 \gamma_\star\le1/40 γ ⋆ ≤ 1/40 .
Identifying the rare label. Since t ≥ 1 t\ge1 t ≥ 1 , we have p ≤ Φ ( − 1 ) < 1 / 4 p\le\Phi(-1)<1/4 p ≤ Φ ( − 1 ) < 1/4 , which gives
1 2 − q = ( 1 − 2 η ) ( 1 2 − p ) ≥ 1 − 2 η 4 \frac{1}{2}-q=(1-2\eta)\left(\frac{1}{2}-p\right)\ge\frac{1-2\eta}{4} 2 1 − q = ( 1 − 2 η ) ( 2 1 − p ) ≥ 4 1 − 2 η
Algorithm 3 sets y ^ r t = − 1 \hat{y}_{\mathrm{rt}}=-1 y ^ rt = − 1 unless N − > N + N_->N_+ N − > N + , that is, unless q ^ > 1 / 2 \hat{q}>1/2 q ^ > 1/2 . By Hoeffding’s inequality,
P ( y ^ r t ≠ − 1 ) ≤ P ( q ^ − q > 1 2 − q ) ≤ exp ( − 2 k ( 1 2 − q ) 2 ) ≤ exp ( − k ( 1 − 2 η ) 2 8 ) \mathbb{P}(\hat{y}_{\mathrm{rt}}\ne-1)
\le\mathbb{P}\left(\hat{q}-q>\frac{1}{2}-q\right)
\le\exp\left(-2k\left(\frac{1}{2}-q\right)^2\right)
\le\exp\left(-\frac{k(1-2\eta)^2}{8}\right) P ( y ^ rt = − 1 ) ≤ P ( q ^ − q > 2 1 − q ) ≤ exp ( − 2 k ( 2 1 − q ) 2 ) ≤ exp ( − 8 k ( 1 − 2 η ) 2 )
which is at most δ / 2 \delta/2 δ /2 once k ≥ 8 log ( 2 / δ ) / ( 1 − 2 η ) 2 k\ge8\log(2/\delta)/(1-2\eta)^2 k ≥ 8 log ( 2/ δ ) / ( 1 − 2 η ) 2 .
From the label frequency to the threshold. Let τ = a γ ⋆ ϕ ( t + a γ ⋆ ) \tau=a\gamma_\star\phi(t+a\gamma_\star) τ = a γ ⋆ ϕ ( t + a γ ⋆ ) . We show that, on the event where y ^ r t = − 1 \hat{y}_{\mathrm{rt}}=-1 y ^ rt = − 1 and ∣ q ^ − q ∣ ≤ ( 1 − 2 η ) τ |\hat{q}-q|\le(1-2\eta)\tau ∣ q ^ − q ∣ ≤ ( 1 − 2 η ) τ , the output satisfies ∣ t ^ − t ∣ ≤ a γ ⋆ |\hat{t}-t|\le a\gamma_\star ∣ t ^ − t ∣ ≤ a γ ⋆ . On this event, q ^ r t = q ^ \hat{q}_{\mathrm{rt}}=\hat{q} q ^ rt = q ^ , and therefore
∣ p ^ r a w − p ∣ = ∣ q ^ − q ∣ 1 − 2 η ≤ τ |\hat{p}_{\mathrm{raw}}-p|=\frac{|\hat{q}-q|}{1-2\eta}\le\tau ∣ p ^ raw − p ∣ = 1 − 2 η ∣ q ^ − q ∣ ≤ τ
Algorithm 4 Local direction refinement, adapted from Diakonikolas et al. [20, Algorithm 2]
Require: Initial direction w 0 ∈ S d − 1 w_0 \in S^{d-1} w 0 ∈ S d − 1 , threshold estimate t 0 t_0 t 0 , band width γ 0 \gamma_0 γ 0 , known noise rate η < 1 / 2 \eta< 1/2 η < 1/2 , target RCN samples S = { ( X i , Y i ) } i = 1 N 2 S = \{(X_i,Y_i)\}_{i=1}^{N_2} S = {( X i , Y i ) } i = 1 N 2
Ensure: A refined direction if t 0 < 1 / 2 t_0 < 1/2 t 0 < 1/2 then return w 0 w_0 w 0
Set ρ l o c ← 0.00098 \rho_{\mathrm{loc}} \leftarrow0.00098 ρ loc ← 0.00098 , θ ˉ ← 1 / ( 6 t 0 ) \bar{\theta} \leftarrow1/(6t_0) θ ˉ ← 1/ ( 6 t 0 ) , and
μ 0 ← ( 1 − 4 ρ l o c ) 2 π sin ( θ ˉ / 2 ) 16 ( 1 − 2 η ) , K ← ⌈ log ( 3 sin ( θ ˉ / 2 ) / ( 16 γ 0 ) ) log ( 1 / ( 1 − ρ l o c ) ) ⌉ \mu_0 \leftarrow\frac{(1-4\rho_{\mathrm{loc}})\sqrt{2\pi}\sin(\bar{\theta}/2)}{16(1-2\eta)}, \qquad K \leftarrow\left\lceil\frac{\log(3\sin(\bar{\theta}/2)/(16\gamma_0))}{\log(1/(1-\rho_{\mathrm{loc}}))}\right\rceil μ 0 ← 16 ( 1 − 2 η ) ( 1 − 4 ρ loc ) 2 π sin ( θ ˉ /2 ) , K ← ⌈ log ( 1/ ( 1 − ρ loc )) log ( 3 sin ( θ ˉ /2 ) / ( 16 γ 0 )) ⌉
if K < 0 K < 0 K < 0 then return w 0 w_0 w 0
Set
P ( t 0 , γ 0 ) ← P Z ∼ N ( 0 , 1 ) [ − t 0 ≤ Z ≤ − t 0 + γ 0 ] P(t_0,\gamma_0) \leftarrow\mathbb{P}_{Z\sim\mathcal{N}(0,1)}[-t_0 \le Z \le-t_0+\gamma_0] P ( t 0 , γ 0 ) ← P Z ∼ N ( 0 , 1 ) [ − t 0 ≤ Z ≤ − t 0 + γ 0 ]
Split S S S into K + 1 K+1 K + 1 batches S ( 0 ) , … , S ( K ) S^{(0)},\ldots,S^{(K)} S ( 0 ) , … , S ( K ) of equal size m = ⌊ N 2 / ( K + 1 ) ⌋ m=\lfloor N_2/(K+1)\rfloor m = ⌊ N 2 / ( K + 1 )⌋
for k = 0 , 1 , … , K k=0,1,\ldots,K k = 0 , 1 , … , K do
Define the current band
E k = { x : − t 0 ≤ ⟨ w k , x ⟩ ≤ − t 0 + γ 0 } \mathcal{E}_k=\{x:-t_0\le\langle w_k,x\rangle\le-t_0+\gamma_0\} E k = { x : − t 0 ≤ ⟨ w k , x ⟩ ≤ − t 0 + γ 0 }
For each ( X i , Y i ) ∈ S ( k ) (X_i,Y_i)\in S^{(k)} ( X i , Y i ) ∈ S ( k ) , set
G k ( X i , Y i ) = 1 2 ( ( 1 − 2 η ) sign ( ⟨ w k , X i ⟩ + t 0 ) − Y i ) proj w k ⊥ ( X i ) G_k(X_i,Y_i)=\frac{1}{2}\left((1-2\eta)\operatorname{sign}(\langle w_k,X_i\rangle+t_0)-Y_i\right)\operatorname{proj}_{w_k^\perp}(X_i) G k ( X i , Y i ) = 2 1 ( ( 1 − 2 η ) sign (⟨ w k , X i ⟩ + t 0 ) − Y i ) proj w k ⊥ ( X i )
Form the empirical localized update from the batch S ( k ) S^{(k)} S ( k )
G ^ k = 1 m ∑ ( X i , Y i ) ∈ S ( k ) G k ( X i , Y i ) 1 { X i ∈ E k } P ( t 0 , γ 0 ) \widehat{G}_k=\frac{1}{m}\sum_{(X_i,Y_i)\in S^{(k)}}G_k(X_i,Y_i)\frac{\mathbf{1}\{X_i\in\mathcal{E}_k\}}{P(t_0,\gamma_0)} G k = m 1 ( X i , Y i ) ∈ S ( k ) ∑ G k ( X i , Y i ) P ( t 0 , γ 0 ) 1 { X i ∈ E k }
μ k ← μ 0 ( 1 − ρ l o c ) k \mu_k\leftarrow\mu_0(1-\rho_{\mathrm{loc}})^k μ k ← μ 0 ( 1 − ρ loc ) k
Update
w k + 1 ← w k − μ k G ^ k ∥ w k − μ k G ^ k ∥ 2 w_{k+1}\leftarrow\frac{w_k-\mu_k\widehat{G}_k}{\lVert w_k-\mu_k\widehat{G}_k\rVert_2} w k + 1 ← ∥ w k − μ k G k ∥ 2 w k − μ k G k
end for
return w K + 1 w_{K+1} w K + 1
Since p ∈ [ 0 , 1 / 2 ] p\in[0,1/2] p ∈ [ 0 , 1/2 ] , clipping p ^ r a w \widehat{p}_{\mathrm{raw}} p raw to this interval cannot increase its distance to p p p , and ∣ p ^ − p ∣ ≤ τ |\widehat{p}-p|\le\tau ∣ p − p ∣ ≤ τ .
Next, ϕ \phi ϕ is decreasing on [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) and t − a γ ⋆ ≥ 0 t-a\gamma_\star\ge0 t − a γ ⋆ ≥ 0 by eq. (5), hence
Φ ( − ( t − a γ ⋆ ) ) − p = ∫ t − a γ ⋆ t ϕ ( u ) d u ≥ a γ ⋆ ϕ ( t ) ≥ τ \Phi(-(t-a\gamma_\star))-p=\int_{t-a\gamma_\star}^{t}\phi(u)\,\mathrm{d}u\ge a\gamma_\star\phi(t)\ge\tau Φ ( − ( t − a γ ⋆ )) − p = ∫ t − a γ ⋆ t ϕ ( u ) d u ≥ a γ ⋆ ϕ ( t ) ≥ τ
p − Φ ( − ( t + a γ ⋆ ) ) = ∫ t t + a γ ⋆ ϕ ( u ) d u ≥ a γ ⋆ ϕ ( t + a γ ⋆ ) = τ p-\Phi(-(t+a\gamma_\star))=\int_{t}^{t+a\gamma_\star}\phi(u)\,\mathrm{d}u\ge a\gamma_\star\phi(t+a\gamma_\star)=\tau p − Φ ( − ( t + a γ ⋆ )) = ∫ t t + a γ ⋆ ϕ ( u ) d u ≥ a γ ⋆ ϕ ( t + a γ ⋆ ) = τ
Combining these bounds with ∣ p ^ − p ∣ ≤ τ |\widehat{p}-p|\le\tau ∣ p − p ∣ ≤ τ gives
0 < Φ ( − ( t + a γ ⋆ ) ) ≤ p ^ ≤ Φ ( − ( t − a γ ⋆ ) ) 0<\Phi(-(t+a\gamma_\star))\le\widehat{p}\le\Phi(-(t-a\gamma_\star)) 0 < Φ ( − ( t + a γ ⋆ )) ≤ p ≤ Φ ( − ( t − a γ ⋆ ))
In particular p ^ > 0 \widehat{p}>0 p > 0 , so the algorithm sets a ^ = Φ − 1 ( 1 − p ^ ) \widehat{a}=\Phi^{-1}(1-\widehat{p}) a = Φ − 1 ( 1 − p ) , which is the same as Φ ( − a ^ ) = p ^ \Phi(-\widehat{a})=\widehat{p} Φ ( − a ) = p . Since Φ \Phi Φ is increasing, this gives t − a γ ⋆ ≤ a ^ ≤ t + a γ ⋆ t-a\gamma_\star\le\widehat{a}\le t+a\gamma_\star t − a γ ⋆ ≤ a ≤ t + a γ ⋆ . Finally, y ^ r t = − 1 \widehat{y}_{\mathrm{rt}}=-1 y rt = − 1 gives t ^ = a ^ \widehat{t}=\widehat{a} t = a , and ∣ t ^ − t ∣ ≤ a γ ⋆ |\widehat{t}-t|\le a\gamma_\star ∣ t − t ∣ ≤ a γ ⋆ .
Sample size. It remains to bound the probability that ∣ q ^ − q ∣ > ( 1 − 2 η ) τ |\widehat{q}-q|>(1-2\eta)\tau ∣ q − q ∣ > ( 1 − 2 η ) τ . By definition of γ ⋆ \gamma_\star γ ⋆ ,
γ ⋆ ϕ ( t ) = ε 2 ( 1 − 2 η ) exp ( t 2 / 2 ) ⋅ 1 2 π exp ( − t 2 / 2 ) = ε 2 ( 1 − 2 η ) 2 π \gamma_\star\phi(t)=\frac{\varepsilon}{2(1-2\eta)}\exp(t^2/2)\cdot\frac{1}{\sqrt{2\pi}}\exp(-t^2/2)=\frac{\varepsilon}{2(1-2\eta)\sqrt{2\pi}} γ ⋆ ϕ ( t ) = 2 ( 1 − 2 η ) ε exp ( t 2 /2 ) ⋅ 2 π 1 exp ( − t 2 /2 ) = 2 ( 1 − 2 η ) 2 π ε
and by eq. (5) and a ≤ 1 / 2 a\le1/2 a ≤ 1/2 ,
ϕ ( t + a γ ⋆ ) ϕ ( t ) = exp ( − a t γ ⋆ − a 2 γ ⋆ 2 2 ) ≥ exp ( − 1 40 ) ≥ 1 2 \frac{\phi(t+a\gamma_\star)}{\phi(t)}=\exp\left(-at\gamma_\star-\frac{a^2\gamma_\star^2}{2}\right)\ge\exp\left(-\frac{1}{40}\right)\ge\frac{1}{2} ϕ ( t ) ϕ ( t + a γ ⋆ ) = exp ( − a t γ ⋆ − 2 a 2 γ ⋆ 2 ) ≥ exp ( − 40 1 ) ≥ 2 1
Algorithm 5 RARE-TAIL CURRICULUM LEARNER FOR η \eta η -RCN TARGETS
Require: Source samples S 1 S_1 S 1 , target RCN samples S 2 S_2 S 2 , accuracy ε \varepsilon ε , failure probability δ \delta δ , known noise rate η < 1 / 2 \eta< 1/2 η < 1/2
Ensure: A halfspace h ^ \hat{h} h ^
Split S 2 S_2 S 2 into independent parts S 2 , b a n d S_{2,\mathrm{band}} S 2 , band , S 2 , r e f S_{2,\mathrm{ref}} S 2 , ref , and S 2 , t h r S_{2,\mathrm{thr}} S 2 , thr
( w ^ 0 , y ^ r t ) ← (\hat{w}_0,\hat{y}_{\mathrm{rt}}) \leftarrow ( w ^ 0 , y ^ rt ) ← RARE-TAIL DIRECTION ESTIMATOR( S 1 ) (S_1) ( S 1 )
t ^ 0 ← \hat{t}_0 \leftarrow t ^ 0 ← RCN BAND-SCALE THRESHOLD FROM LABELS( S 2 , b a n d , η ) (S_{2,\mathrm{band}},\eta) ( S 2 , band , η )
Set
γ ^ ← ε 2 ( 1 − 2 η ) exp ( t ^ 0 2 / 2 ) \hat{\gamma} \leftarrow\frac{\varepsilon}{2(1-2\eta)}\exp(\hat{t}_0^2/2) γ ^ ← 2 ( 1 − 2 η ) ε exp ( t ^ 0 2 /2 )
w ^ r e f ← \hat{w}_{\mathrm{ref}} \leftarrow w ^ ref ← LOCAL DIRECTION REFINEMENT( w ^ 0 , t ^ 0 , γ ^ , η , S 2 , r e f ) (\hat{w}_0,\hat{t}_0,\hat{\gamma},\eta,S_{2,\mathrm{ref}}) ( w ^ 0 , t ^ 0 , γ ^ , η , S 2 , ref )
t ^ f i n a l ← \hat{t}_{\mathrm{final}} \leftarrow t ^ final ← THRESHOLD-ERM( S 2 , t h r , w ^ r e f ) (S_{2,\mathrm{thr}},\hat{w}_{\mathrm{ref}}) ( S 2 , thr , w ^ ref )
return h ^ ( x ) = sign ( ⟨ w ^ r e f , x ⟩ + t ^ f i n a l ) \hat{h}(x)=\operatorname{sign}(\langle\hat{w}_{\mathrm{ref}},x\rangle+\hat{t}_{\mathrm{final}}) h ^ ( x ) = sign (⟨ w ^ ref , x ⟩ + t ^ final )
Therefore ( 1 − 2 η ) τ ≥ a ε / ( 4 2 π ) (1-2\eta)\tau\ge a\varepsilon/(4\sqrt{2\pi}) ( 1 − 2 η ) τ ≥ a ε / ( 4 2 π ) , and Hoeffding’s inequality gives
P ( ∣ q ^ − q ∣ > ( 1 − 2 η ) τ ) ≤ 2 exp ( − 2 k ( 1 − 2 η ) 2 τ 2 ) ≤ 2 exp ( − k a 2 ε 2 16 π τ ) \mathbb{P}\left(\lvert\hat{q}-q\rvert>(1-2\eta)\tau\right)\le2\exp\left(-2k(1-2\eta)^2\tau^2\right)\le2\exp\left(-\frac{ka^2\varepsilon^2}{16\pi\tau}\right) P ( ∣ q ^ − q ∣ > ( 1 − 2 η ) τ ) ≤ 2 exp ( − 2 k ( 1 − 2 η ) 2 τ 2 ) ≤ 2 exp ( − 16 π τ k a 2 ε 2 )
which is at most δ / 2 \delta/2 δ /2 once k ≥ 16 π log ( 4 / δ ) / ( a 2 ε 2 ) k\ge16\pi\log(4/\delta)/(a^2\varepsilon^2) k ≥ 16 π log ( 4/ δ ) / ( a 2 ε 2 ) . A union bound over the two events completes the proof.
The next lemma is the population statement behind the band update, and our counterpart to [20 Lemmas 2.3 and 2.4] . Their threshold-accuracy assumption, written in our notation, is
∣ t 0 − t ∣ ≤ ε 2 8 ( 1 − 2 η ) 2 \lvert t_0-t\rvert\le\frac{\varepsilon^2}{8(1-2\eta)^2} ∣ t 0 − t ∣ ≤ 8 ( 1 − 2 η ) 2 ε 2
where we have substituted their ε ′ = ε / ( 1 − 2 η ) \varepsilon'=\varepsilon/(1-2\eta) ε ′ = ε / ( 1 − 2 η ) . We show that the population descent bound holds under the weaker requirement
∣ t 0 − t ∣ ≤ γ 0 , \lvert t_0-t\rvert\le\gamma_0, ∣ t 0 − t ∣ ≤ γ 0 ,
provided 8 γ 0 ≤ θ ≤ 1 / ( 5 t ) 8\gamma_0\le\theta\le1/(5t) 8 γ 0 ≤ θ ≤ 1/ ( 5 t ) .
Lemma D.4 (Population descent under a band-accurate threshold guess). Assume the target stage is η \eta η -RCN with known constant η < 1 / 2 \eta<1/2 η < 1/2 , and suppose t ≥ 1 t\ge1 t ≥ 1 . Let w ∈ S d − 1 w\in S^{d-1} w ∈ S d − 1 , t 0 ∈ R t_0\in\mathbb{R} t 0 ∈ R , and γ 0 > 0 \gamma_0>0 γ 0 > 0 , let E ( w , t 0 ) \mathcal{E}(w,t_0) E ( w , t 0 ) be the band of eq. (4) and g ( w ; x , y ) g(w;x,y) g ( w ; x , y ) the vector of eq. (5) , and define the population localized update
G ( w ) = E [ g ( w ; X , Y ) ∣ X ∈ E ( w , t 0 ) ] G(w)=\mathbb{E}[g(w;X,Y)\mid X\in\mathcal{E}(w,t_0)] G ( w ) = E [ g ( w ; X , Y ) ∣ X ∈ E ( w , t 0 )] Then
∥ G ( w ) ∥ 2 ≤ C 1 = 1 − 2 η 2 π \lVert G(w)\rVert_2\le C_1=\frac{1-2\eta}{\sqrt{2\pi}} ∥ G ( w ) ∥ 2 ≤ C 1 = 2 π 1 − 2 η Moreover, if θ = ∠ ( w , w ⋆ ) \theta=\angle(w,w^\star) θ = ∠ ( w , w ⋆ ) satisfies
8 γ 0 ≤ θ ≤ 1 5 t and ∣ t 0 − t ∣ ≤ γ 0 8\gamma_0\le\theta\le\frac{1}{5t}
\qquad\text{and}\qquad
\lvert t_0-t\rvert\le\gamma_0 8 γ 0 ≤ θ ≤ 5 t 1 and ∣ t 0 − t ∣ ≤ γ 0 then
G ( w ) ⋅ w ⋆ ≤ − ϕ ( 7 / 10 ) ( 1 − 2 η ) sin θ ≤ − C 1 2 sin θ G(w)\cdot w^\star\le-\phi(7/10)(1-2\eta)\sin\theta\le-\frac{C_1}{2}\sin\theta G ( w ) ⋅ w ⋆ ≤ − ϕ ( 7/10 ) ( 1 − 2 η ) sin θ ≤ − 2 C 1 sin θ
Proof. On the band event X ∈ E ( w , t 0 ) X\in\mathcal{E}(w,t_0) X ∈ E ( w , t 0 ) , we have sign ( ⟨ w , X ⟩ + t 0 ) = + 1 \operatorname{sign}(\langle w,X\rangle+t_0)=+1 sign (⟨ w , X ⟩ + t 0 ) = + 1 . Also, under η \eta η -RCN,
E [ Y ∣ X ] = ( 1 − 2 η ) f ⋆ ( X ) \mathbb{E}[Y\mid X]=(1-2\eta)f^\star(X) E [ Y ∣ X ] = ( 1 − 2 η ) f ⋆ ( X )
Therefore
G ( w ) = 1 2 E [ ( ( 1 − 2 η ) − ( 1 − 2 η ) f ⋆ ( X ) ) proj w ⊥ ( X ) ∣ X ∈ E ( w , t 0 ) ] = ( 1 − 2 η ) E [ 1 { f ⋆ ( X ) = − 1 } proj w ⊥ ( X ) ∣ X ∈ E ( w , t 0 ) ] \begin{aligned}
G(w) = \frac{1}{2}\mathbb{E}\left[((1-2\eta)-(1-2\eta)f^\star(X))\operatorname{proj}_{w^\perp}(X)\mid X\in\mathcal{E}(w,t_0)\right] \\
&= (1-2\eta)\mathbb{E}\left[\mathbf{1}\{f^\star(X)=-1\}\operatorname{proj}_{w^\perp}(X)\mid X\in\mathcal{E}(w,t_0)\right]
\end{aligned} G ( w ) = 2 1 E [ (( 1 − 2 η ) − ( 1 − 2 η ) f ⋆ ( X )) proj w ⊥ ( X ) ∣ X ∈ E ( w , t 0 ) ] = ( 1 − 2 η ) E [ 1 { f ⋆ ( X ) = − 1 } proj w ⊥ ( X ) ∣ X ∈ E ( w , t 0 ) ]
If w = ± w ⋆ w=\pm w^\star w = ± w ⋆ , then both f ⋆ ( X ) f^\star(X) f ⋆ ( X ) and the band event are determined by ⟨ w , X ⟩ \langle w,X\rangle ⟨ w , X ⟩ , which is independent of proj w ⊥ ( X ) \operatorname{proj}_{w^\perp}(X) proj w ⊥ ( X ) , so G ( w ) = 0 G(w)=0 G ( w ) = 0 and the norm bound is trivial. Otherwise θ ∈ ( 0 , π ) \theta\in(0,\pi) θ ∈ ( 0 , π ) , and by rotational invariance we may assume w = e 1 w=e_1 w = e 1 and
w ⋆ = cos θ e 1 + sin θ e 2 w^\star=\cos\theta\,e_1+\sin\theta\,e_2 w ⋆ = cos θ e 1 + sin θ e 2
Let Z = ⟨ w , X ⟩ Z=\langle w,X\rangle Z = ⟨ w , X ⟩ and U = ⟨ e 2 , X ⟩ U=\langle e_2,X\rangle U = ⟨ e 2 , X ⟩ . Then Z , U Z,U Z , U are independent standard Gaussians. The event f ⋆ ( X ) = − 1 f^\star(X)=-1 f ⋆ ( X ) = − 1 is
cos θ Z + sin θ U + t < 0 \cos\theta\,Z+\sin\theta\,U+t<0 cos θ Z + sin θ U + t < 0
or equivalently U < a ( Z ) U<a(Z) U < a ( Z ) , where
a ( z ) = − t − cos θ z sin θ a(z)=\frac{-t-\cos\theta\,z}{\sin\theta} a ( z ) = sin θ − t − cos θ z
The coordinates of X X X orthogonal to span { e 1 , e 2 } \operatorname{span}\{e_1,e_2\} span { e 1 , e 2 } are independent of ( Z , U ) (Z,U) ( Z , U ) and have mean zero, so G ( w ) G(w) G ( w ) is collinear with e 2 e_2 e 2 . Since E [ U 1 { U < a } ] = − ϕ ( a ) \mathbb{E}[U\mathbf{1}\{U<a\}]=-\phi(a) E [ U 1 { U < a }] = − ϕ ( a ) for every a ∈ R a\in\mathbb{R} a ∈ R , we get
G ( w ) ⋅ e 2 = ( 1 − 2 η ) E [ U 1 { U < a ( Z ) } ∣ Z ∈ [ − t 0 , − t 0 + γ 0 ] ] = − ( 1 − 2 η ) E [ ϕ ( a ( Z ) ) ∣ Z ∈ [ − t 0 , − t 0 + γ 0 ] ] \begin{aligned}
G(w)\cdot e_2=(1-2\eta)\mathbb{E}[U\mathbf{1}\{U<a(Z)\}\mid Z\in[-t_0,-t_0+\gamma_0]] \\
&=-(1-2\eta)\mathbb{E}[\phi(a(Z))\mid Z\in[-t_0,-t_0+\gamma_0]]
\end{aligned} G ( w ) ⋅ e 2 = ( 1 − 2 η ) E [ U 1 { U < a ( Z )} ∣ Z ∈ [ − t 0 , − t 0 + γ 0 ]] = − ( 1 − 2 η ) E [ ϕ ( a ( Z )) ∣ Z ∈ [ − t 0 , − t 0 + γ 0 ]]
Since 0 < ϕ ≤ ϕ ( 0 ) = 1 / 2 π 0<\phi\leq\phi(0)=1/\sqrt{2\pi} 0 < ϕ ≤ ϕ ( 0 ) = 1/ 2 π , this proves the first claim,
∥ G ( w ) ∥ 2 = ∣ G ( w ) ⋅ e 2 ∣ ≤ 1 − 2 η 2 π \lVert G(w)\rVert_2=|G(w)\cdot e_2|\leq\frac{1-2\eta}{\sqrt{2\pi}} ∥ G ( w ) ∥ 2 = ∣ G ( w ) ⋅ e 2 ∣ ≤ 2 π 1 − 2 η
Moreover, w ⋆ ⋅ e 2 = sin θ w^\star\cdot e_2=\sin\theta w ⋆ ⋅ e 2 = sin θ , and
G ( w ) ⋅ w ⋆ = − ( 1 − 2 η ) sin θ E [ ϕ ( a ( Z ) ) ∣ Z ∈ [ − t 0 , − t 0 + γ 0 ] ] G(w)\cdot w^\star=-(1-2\eta)\sin\theta\mathbb{E}[\phi(a(Z))\mid Z\in[-t_0,-t_0+\gamma_0]] G ( w ) ⋅ w ⋆ = − ( 1 − 2 η ) sin θ E [ ϕ ( a ( Z )) ∣ Z ∈ [ − t 0 , − t 0 + γ 0 ]]
For the second claim, it remains to lower bound ϕ ( a ( z ) ) \phi(a(z)) ϕ ( a ( z )) uniformly over the band. Write z = − t 0 + s z=-t_0+s z = − t 0 + s with s ∈ [ 0 , γ 0 ] s\in[0,\gamma_0] s ∈ [ 0 , γ 0 ] and Δ = t 0 − t \Delta=t_0-t Δ = t 0 − t . Then
− t − cos θ z = t ( cos θ − 1 ) + cos θ Δ − cos θ s -t-\cos\theta\,z=t(\cos\theta-1)+\cos\theta\,\Delta-\cos\theta\,s − t − cos θ z = t ( cos θ − 1 ) + cos θ Δ − cos θ s
which then gives
∣ a ( z ) ∣ ≤ t ( 1 − cos θ ) sin θ + ∣ Δ ∣ sin θ + s sin θ |a(z)|\leq\frac{t(1-\cos\theta)}{\sin\theta}+\frac{|\Delta|}{\sin\theta}+\frac{s}{\sin\theta} ∣ a ( z ) ∣ ≤ sin θ t ( 1 − cos θ ) + sin θ ∣Δ∣ + sin θ s
Since t ≥ 1 t\geq1 t ≥ 1 , we have θ ≤ 1 / ( 5 t ) ≤ 1 \theta\leq1/(5t)\leq1 θ ≤ 1/ ( 5 t ) ≤ 1 . Using 1 − cos θ ≤ θ 2 / 2 1-\cos\theta\leq\theta^2/2 1 − cos θ ≤ θ 2 /2 and sin θ ≥ θ / 2 \sin\theta\geq\theta/2 sin θ ≥ θ /2 for θ ∈ ( 0 , 1 ] \theta\in(0,1] θ ∈ ( 0 , 1 ] , together with θ ≤ 1 / ( 5 t ) \theta\leq1/(5t) θ ≤ 1/ ( 5 t ) , θ ≥ 8 γ 0 \theta\geq8\gamma_0 θ ≥ 8 γ 0 , ∣ Δ ∣ ≤ γ 0 |\Delta|\leq\gamma_0 ∣Δ∣ ≤ γ 0 , and s ≤ γ 0 s\leq\gamma_0 s ≤ γ 0 , we have
∣ a ( z ) ∣ ≤ t θ + 2 γ 0 θ + 2 γ 0 θ ≤ 1 5 + 1 4 + 1 4 = 7 10 |a(z)|\leq t\theta+2\frac{\gamma_0}{\theta}+2\frac{\gamma_0}{\theta}\leq\frac{1}{5}+\frac{1}{4}+\frac{1}{4}=\frac{7}{10} ∣ a ( z ) ∣ ≤ tθ + 2 θ γ 0 + 2 θ γ 0 ≤ 5 1 + 4 1 + 4 1 = 10 7
Hence ϕ ( a ( z ) ) ≥ ϕ ( 7 / 10 ) \phi(a(z))\geq\phi(7/10) ϕ ( a ( z )) ≥ ϕ ( 7/10 ) throughout the band, and
G ( w ) ⋅ w ⋆ ≤ − ϕ ( 7 / 10 ) ( 1 − 2 η ) sin θ G(w)\cdot w^\star\leq-\phi(7/10)(1-2\eta)\sin\theta G ( w ) ⋅ w ⋆ ≤ − ϕ ( 7/10 ) ( 1 − 2 η ) sin θ
Finally, ϕ ( 7 / 10 ) = e − 49 / 200 / 2 π ≥ 1 / ( 2 2 π ) \phi(7/10)=e^{-49/200}/\sqrt{2\pi}\geq1/(2\sqrt{2\pi}) ϕ ( 7/10 ) = e − 49/200 / 2 π ≥ 1/ ( 2 2 π ) , which gives the bound in terms of C 1 C_1 C 1 .
The empirical update of Algorithm 4 averages the vectors of eq. (5) over the samples of one batch that fall in the band. For a direction that does not depend on the batch, this average is close to G ( w ) G(w) G ( w ) , whatever the angle between w w w and w ⋆ w^\star w ⋆ .
Lemma D.5 (Concentration of the localized update for a fixed direction). Assume the target stage is η \eta η -RCN with known constant η < 1 / 2 \eta<1/2 η < 1/2 . Fix w ∈ S d − 1 w\in S^{d-1} w ∈ S d − 1 , t 0 ∈ R t_0\in\mathbb{R} t 0 ∈ R , and γ 0 > 0 \gamma_0>0 γ 0 > 0 , and let
P = P Z ∼ N ( 0 , 1 ) [ − t 0 ≤ Z ≤ − t 0 + γ 0 ] P=\mathbb{P}_{Z\sim\mathcal{N}(0,1)}[-t_0\leq Z\leq-t_0+\gamma_0] P = P Z ∼ N ( 0 , 1 ) [ − t 0 ≤ Z ≤ − t 0 + γ 0 ]
be the Gaussian mass of the band E ( w , t 0 ) \mathcal{E}(w,t_0) E ( w , t 0 ) . Let ( X 1 , Y 1 ) , … , ( X m , Y m ) (X_1,Y_1),\ldots,(X_m,Y_m) ( X 1 , Y 1 ) , … , ( X m , Y m ) be i.i.d. target RCN samples, and define
G ^ ( w ) = 1 m P ∑ i = 1 m g ( w ; X i , Y i ) 1 { X i ∈ E ( w , t 0 ) } \widehat{G}(w)=\frac{1}{mP}\sum_{i=1}^{m}g(w;X_i,Y_i)1\{X_i\in\mathcal{E}(w,t_0)\} G ( w ) = m P 1 i = 1 ∑ m g ( w ; X i , Y i ) 1 { X i ∈ E ( w , t 0 )}
For every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) , if
m ≥ 2048 π ( 1 − 2 η ) 2 P ( d log 5 + log ( 2 / δ ) ) m\geq\frac{2048\pi}{(1-2\eta)^2P}\left(d\log5+\log(2/\delta)\right) m ≥ ( 1 − 2 η ) 2 P 2048 π ( d log 5 + log ( 2/ δ ) )
then, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ G ^ ( w ) − G ( w ) ∥ 2 ≤ C 1 4 = 1 − 2 η 4 2 π \|\widehat{G}(w)-G(w)\|_2\leq\frac{C_1}{4}=\frac{1-2\eta}{4\sqrt{2\pi}} ∥ G ( w ) − G ( w ) ∥ 2 ≤ 4 C 1 = 4 2 π 1 − 2 η
Proof. Let B i = 1 { X i ∈ E ( w , t 0 ) } B_i=1\{X_i\in\mathcal{E}(w,t_0)\} B i = 1 { X i ∈ E ( w , t 0 )} , so that P ( B i = 1 ) = P \mathbb{P}(B_i=1)=P P ( B i = 1 ) = P . On the band, sign ( ⟨ w , X i ⟩ + t 0 ) = + 1 \operatorname{sign}(\langle w,X_i\rangle+t_0)=+1 sign (⟨ w , X i ⟩ + t 0 ) = + 1 , and therefore
B i g ( w ; X i , Y i ) = B i c i proj w ⊥ ( X i ) , c i = 1 2 ( ( 1 − 2 η ) − Y i ) ∈ { − η , 1 − η } B_i g(w;X_i,Y_i)=B_i c_i\operatorname{proj}_{w^\perp}(X_i),\qquad c_i=\frac{1}{2}\left((1-2\eta)-Y_i\right)\in\{-\eta,1-\eta\} B i g ( w ; X i , Y i ) = B i c i proj w ⊥ ( X i ) , c i = 2 1 ( ( 1 − 2 η ) − Y i ) ∈ { − η , 1 − η }
In particular, ∣ c i ∣ ≤ 1 |c_i|\leq1 ∣ c i ∣ ≤ 1 .
Fix a unit vector v ∈ S d − 1 v\in S^{d-1} v ∈ S d − 1 , and let h i = ⟨ v , proj w ⊥ ( X i ) ⟩ = ⟨ proj w ⊥ ( v ) , X i ⟩ h_i=\langle v,\operatorname{proj}_{w^\perp}(X_i)\rangle=\langle\operatorname{proj}_{w^\perp}(v),X_i\rangle h i = ⟨ v , proj w ⊥ ( X i )⟩ = ⟨ proj w ⊥ ( v ) , X i ⟩ . This is a centered Gaussian with variance σ v 2 = ∥ proj w ⊥ ( v ) ∥ 2 2 ≤ 1 \sigma_v^2=\|\operatorname{proj}_{w^\perp}(v)\|_2^2\leq1 σ v 2 = ∥ proj w ⊥ ( v ) ∥ 2 2 ≤ 1 , and it is independent of ⟨ w , X i ⟩ \langle w,X_i\rangle ⟨ w , X i ⟩ , and, therefore, of B i B_i B i . Let
ξ i = B i c i h i = ⟨ v , B i g ( w ; X i , Y i ) ⟩ \xi_i=B_ic_ih_i=\langle v,B_i g(w;X_i,Y_i)\rangle ξ i = B i c i h i = ⟨ v , B i g ( w ; X i , Y i )⟩
so that ⟨ v , G ^ ( w ) ⟩ = ( m P ) − 1 ∑ i = 1 m ξ i \langle v,\widehat{G}(w)\rangle=(mP)^{-1}\sum_{i=1}^{m}\xi_i ⟨ v , G ( w )⟩ = ( m P ) − 1 ∑ i = 1 m ξ i , and E [ ξ i ] = P ⟨ v , G ( w ) ⟩ \mathbb{E}[\xi_i]=P\langle v,G(w)\rangle E [ ξ i ] = P ⟨ v , G ( w )⟩ by the definition of G ( w ) G(w) G ( w ) . We bound the moment generating function of ξ i \xi_i ξ i . For every x ∈ R x\in\mathbb{R} x ∈ R , Taylor’s theorem gives e x ≤ 1 + x + x 2 2 e ∣ x ∣ e^x\leq1+x+\frac{x^2}{2}e^{|x|} e x ≤ 1 + x + 2 x 2 e ∣ x ∣ . We apply this with x = λ ξ i x=\lambda\xi_i x = λ ξ i for ∣ λ ∣ ≤ 1 |\lambda|\leq1 ∣ λ ∣ ≤ 1 . Since ∣ ξ i ∣ ≤ B i ∣ h i ∣ |\xi_i|\leq B_i|h_i| ∣ ξ i ∣ ≤ B i ∣ h i ∣ , and since B i B_i B i and h i h_i h i are independent,
E [ e λ ξ i ] ≤ 1 + λ E [ ξ i ] + λ 2 2 E [ B i h i 2 e ∣ h i ∣ ] = 1 + λ P ⟨ v , G ( w ) ⟩ + λ 2 P 2 E [ h i 2 e ∣ h i ∣ ] \mathbb{E}\left[e^{\lambda\xi_i}\right]\leq1+\lambda\mathbb{E}[\xi_i]+\frac{\lambda^2}{2}\mathbb{E}\left[B_i h_i^2e^{|h_i|}\right]=1+\lambda P\langle v,G(w)\rangle+\frac{\lambda^2P}{2}\mathbb{E}\left[h_i^2e^{|h_i|}\right] E [ e λ ξ i ] ≤ 1 + λ E [ ξ i ] + 2 λ 2 E [ B i h i 2 e ∣ h i ∣ ] = 1 + λ P ⟨ v , G ( w )⟩ + 2 λ 2 P E [ h i 2 e ∣ h i ∣ ]
A centered Gaussian h h h with variance σ 2 ≤ 1 \sigma^2\leq1 σ 2 ≤ 1 satisfies E [ h 2 e h ] = ( σ 2 + σ 4 ) e σ 2 / 2 ≤ 2 e \mathbb{E}[h^2e^h]=(\sigma^2+\sigma^4)e^{\sigma^2/2}\leq2\sqrt{e} E [ h 2 e h ] = ( σ 2 + σ 4 ) e σ 2 /2 ≤ 2 e , and e ∣ h ∣ ≤ e h + e − h e^{|h|}\leq e^h+e^{-h} e ∣ h ∣ ≤ e h + e − h , so the last expectation is at most 4 e ≤ 8 4\sqrt{e}\leq8 4 e ≤ 8 . Using 1 + x ≤ e x 1+x\leq e^x 1 + x ≤ e x , we obtain
E [ exp ( λ ( ξ i − E [ ξ i ] ) ) ] ≤ exp ( 4 P λ 2 ) for all ∣ λ ∣ ≤ 1 \mathbb{E}\left[\exp\left(\lambda(\xi_i-\mathbb{E}[\xi_i])\right)\right]\leq\exp(4P\lambda^2)\qquad\text{for all }|\lambda|\leq1 E [ exp ( λ ( ξ i − E [ ξ i ]) ) ] ≤ exp ( 4 P λ 2 ) for all ∣ λ ∣ ≤ 1
The samples are independent, so for every s ∈ [ 0 , 8 ] s\in[0,8] s ∈ [ 0 , 8 ] , the Chernoff bound with λ = s / 8 \lambda=s/8 λ = s /8 gives
P ( ⟨ v , G ^ ( w ) − G ( w ) ⟩ ≥ s ) = P ( ∑ i = 1 m ( ξ i − E [ ξ i ] ) ≥ m P s ) ≤ exp ( − λ m P s + 4 m P λ 2 ) = exp ( − m P s 2 16 ) \begin{aligned}
\mathbb{P}\left(\langle v,\widehat{G}(w)-G(w)\rangle\geq s\right)=\mathbb{P}\left(\sum_{i=1}^{m}(\xi_i-\mathbb{E}[\xi_i])\geq mPs\right) \\
&\leq\exp\left(-\lambda mPs+4mP\lambda^2\right)=\exp\left(-\frac{mPs^2}{16}\right)
\end{aligned} P ( ⟨ v , G ( w ) − G ( w )⟩ ≥ s ) = P ( i = 1 ∑ m ( ξ i − E [ ξ i ]) ≥ m P s ) ≤ exp ( − λm P s + 4 m P λ 2 ) = exp ( − 16 m P s 2 )
Applying the same bound to − v -v − v shows that ∣ ⟨ v , G ^ ( w ) − G ( w ) ⟩ ∣ ≥ s |\langle v,\widehat{G}(w)-G(w)\rangle|\geq s ∣ ⟨ v , G ( w ) − G ( w )⟩ ∣ ≥ s with probability at most 2 exp ( − m P s 2 / 16 ) 2\exp(-mPs^2/16) 2 exp ( − m P s 2 /16 ) .
Let N 1 / 2 \mathcal{N}_{1/2} N 1/2 be a 1 / 2 1/2 1/2 -net of S d − 1 S^{d-1} S d − 1 with ∣ N 1 / 2 ∣ ≤ 5 d |\mathcal{N}_{1/2}|\leq5^d ∣ N 1/2 ∣ ≤ 5 d , as in Lemma B.4, so that every u ∈ R d u\in\mathbb{R}^d u ∈ R d satisfies
∥ u ∥ 2 ≤ 2 max v ∈ N 1 / 2 ∣ ⟨ v , u ⟩ ∣ \|u\|_2\leq2\max_{v\in\mathcal{N}_{1/2}}|\langle v,u\rangle| ∥ u ∥ 2 ≤ 2 v ∈ N 1/2 max ∣ ⟨ v , u ⟩ ∣
Taking s = C 1 / 8 s=C_1/8 s = C 1 /8 and a union bound over the net,
P ( ∥ G ^ ( w ) − G ( w ) ∥ 2 > C 1 4 ) ≤ 2 ⋅ 5 d exp ( − m P C 1 2 1024 ) = 2 ⋅ 5 d exp ( − m P ( 1 − 2 η ) 2 2048 π ) \mathbb{P}\left(\|\widehat{G}(w)-G(w)\|_2>\frac{C_1}{4}\right)\leq2\cdot5^d\exp\left(-\frac{mPC_1^2}{1024}\right)=2\cdot5^d\exp\left(-\frac{mP(1-2\eta)^2}{2048\pi}\right) P ( ∥ G ( w ) − G ( w ) ∥ 2 > 4 C 1 ) ≤ 2 ⋅ 5 d exp ( − 1024 m P C 1 2 ) = 2 ⋅ 5 d exp ( − 2048 π m P ( 1 − 2 η ) 2 )
which is at most δ \delta δ under the stated lower bound on m m m .
We can now analyze Algorithm 4. The proof follows the induction of [20 Lemma 2.8] , started at the angle of the warm start.
Theorem D.6 (Local refinement from a band-accurate threshold guess). Assume the target stage is η \eta η -RCN with known constant η < 1 / 2 \eta< 1/2 η < 1/2 . Suppose t ≥ 1 t \ge1 t ≥ 1 and p = Φ ( − t ) ≥ C η ε p = \Phi(-t) \ge C_{\eta}\varepsilon p = Φ ( − t ) ≥ C η ε . Define
γ ⋆ = ε 2 ( 1 − 2 η ) exp ( t 2 / 2 ) \gamma_{\star} = \frac{\varepsilon}{2(1-2\eta)}\exp(t^{2}/2) γ ⋆ = 2 ( 1 − 2 η ) ε exp ( t 2 /2 ) Assume the initialization satisfies
∠ ( w 0 , w ⋆ ) ≤ 1 7 t , ∣ t 0 − t ∣ ≤ a γ ⋆ \angle(w_{0}, w^{\star}) \le\frac{1}{7t}, \qquad|t_{0}-t| \le a\gamma_{\star} ∠ ( w 0 , w ⋆ ) ≤ 7 t 1 , ∣ t 0 − t ∣ ≤ a γ ⋆ for a constant a ∈ ( 0 , 1 / 2 ] a \in(0,1/2] a ∈ ( 0 , 1/2 ] . Set
γ 0 = ε 2 ( 1 − 2 η ) exp ( t 0 2 / 2 ) \gamma_{0} = \frac{\varepsilon}{2(1-2\eta)}\exp(t_{0}^{2}/2) γ 0 = 2 ( 1 − 2 η ) ε exp ( t 0 2 /2 ) Then Algorithm 4, initialized at ( w 0 , t 0 , γ 0 ) (w_{0},t_{0},\gamma_{0}) ( w 0 , t 0 , γ 0 ) and run on
N 2 = O ~ η ( d + log ( 1 / δ ) ε ) N_{2} = \widetilde{O}_{\eta}\left(\frac{d+\log(1/\delta)}{\varepsilon}\right) N 2 = O η ( ε d + log ( 1/ δ ) ) target RCN samples that are independent of ( w 0 , t 0 ) (w_{0},t_{0}) ( w 0 , t 0 ) , returns a direction w ^ r e f \widehat{w}_{\mathrm{ref}} w ref satisfying
∠ ( w ^ r e f , w ⋆ ) ≤ 11 γ 0 ≤ 12 γ ⋆ = 6 1 − 2 η ε exp ( t 2 / 2 ) \angle(\widehat{w}_{\mathrm{ref}},w^{\star}) \le11\gamma_{0} \le12\gamma_{\star} = \frac{6}{1-2\eta}\varepsilon\exp(t^{2}/2) ∠ ( w ref , w ⋆ ) ≤ 11 γ 0 ≤ 12 γ ⋆ = 1 − 2 η 6 ε exp ( t 2 /2 ) with probability at least 1 − δ 1-\delta 1 − δ . The algorithm runs in time O ( d N 2 ) O(dN_{2}) O ( d N 2 ) .
Proof. The band width. By eq. (6) , t γ ⋆ ≤ 1 / 40 t\gamma_{\star} \le1/40 t γ ⋆ ≤ 1/40 and γ ⋆ ≤ 1 / 40 \gamma_{\star} \le1/40 γ ⋆ ≤ 1/40 , so ∣ t 0 − t ∣ ≤ a γ ⋆ ≤ 1 / 80 |t_{0}-t| \le a\gamma_{\star} \le1/80 ∣ t 0 − t ∣ ≤ a γ ⋆ ≤ 1/80 . Hence
∣ t 0 2 − t 2 ∣ = ∣ t 0 − t ∣ ∣ t 0 + t ∣ ≤ a γ ⋆ ( 2 t + a γ ⋆ ) ≤ 1 30 . |t_{0}^{2}-t^{2}| = |t_{0}-t||t_{0}+t| \le a\gamma_{\star}(2t+a\gamma_{\star}) \le\frac{1}{30}. ∣ t 0 2 − t 2 ∣ = ∣ t 0 − t ∣∣ t 0 + t ∣ ≤ a γ ⋆ ( 2 t + a γ ⋆ ) ≤ 30 1 .
and since γ 0 / γ ⋆ = exp ( ( t 0 2 − t 2 ) / 2 ) \gamma_{0}/\gamma_{\star}=\exp((t_{0}^{2}-t^{2})/2) γ 0 / γ ⋆ = exp (( t 0 2 − t 2 ) /2 ) , we get e − 1 / 60 γ ⋆ ≤ γ 0 ≤ e 1 / 60 γ ⋆ e^{-1/60}\gamma_{\star} \le\gamma_{0} \le e^{1/60}\gamma_{\star} e − 1/60 γ ⋆ ≤ γ 0 ≤ e 1/60 γ ⋆ . In particular,
∣ t 0 − t ∣ ≤ γ ⋆ 2 ≤ γ 0 and 11 γ 0 ≤ 12 γ ⋆ . |t_{0}-t| \le\frac{\gamma_{\star}}{2} \le\gamma_{0}
\qquad\text{and}\qquad
11\gamma_{0} \le12\gamma_{\star}. ∣ t 0 − t ∣ ≤ 2 γ ⋆ ≤ γ 0 and 11 γ 0 ≤ 12 γ ⋆ .
Moreover, ∣ t 0 − t ∣ ≤ 1 / 80 ≤ t / 6 |t_{0}-t| \le1/80 \le t/6 ∣ t 0 − t ∣ ≤ 1/80 ≤ t /6 gives 5 t / 6 ≤ t 0 ≤ 7 t / 6 5t/6 \le t_{0} \le7t/6 5 t /6 ≤ t 0 ≤ 7 t /6 , so the angle θ ˉ = 1 / ( 6 t 0 ) \bar{\theta}=1/(6t_{0}) θ ˉ = 1/ ( 6 t 0 ) of Algorithm 4 satisfies
1 7 t ≤ θ ˉ ≤ 1 5 t . \frac{1}{7t} \le\bar{\theta} \le\frac{1}{5t}. 7 t 1 ≤ θ ˉ ≤ 5 t 1 .
The empirical updates. Write ρ = ρ l o c \rho=\rho_{\mathrm{loc}} ρ = ρ loc , θ k = ∠ ( w k , w ⋆ ) \theta_{k}=\angle(w_{k},w^{\star}) θ k = ∠ ( w k , w ⋆ ) , and ψ k = sin ( θ ˉ / 2 ) ( 1 − ρ ) k \psi_{k}=\sin(\bar{\theta}/2)(1-\rho)^{k} ψ k = sin ( θ ˉ /2 ) ( 1 − ρ ) k , so that μ k = ( 1 − 4 ρ ) ψ k / ( 16 C 1 ) \mu_{k}=(1-4\rho)\psi_{k}/(16C_{1}) μ k = ( 1 − 4 ρ ) ψ k / ( 16 C 1 ) with C 1 = ( 1 − 2 η ) / 2 π C_{1}=(1-2\eta)/\sqrt{2\pi} C 1 = ( 1 − 2 η ) / 2 π . Suppose first that K ≥ 0 K\ge0 K ≥ 0 . The iterate w k w_{k} w k is a function of w 0 w_{0} w 0 and of the batches S ( 0 ) , … , S ( k − 1 ) S^{(0)},\ldots,S^{(k-1)} S ( 0 ) , … , S ( k − 1 ) , and is therefore independent of the batch S ( k ) S^{(k)} S ( k ) . Conditioned on earlier batches, we can apply Lemma D.5 to the fixed direction w k w_{k} w k and the batch S ( k ) S^{(k)} S ( k ) , with confidence parameter δ / ( K + 1 ) \delta/(K+1) δ / ( K + 1 ) . A union bound over k = 0 , … , K k=0,\ldots,K k = 0 , … , K shows that, with probability at least 1 − δ 1-\delta 1 − δ ,
∥ G ^ k − G ( w k ) ∥ 2 ≤ C 1 4 for every k ∈ { 0 , … , K } (7) \lVert\widehat{G}_{k}-G(w_{k})\rVert_{2}\le\frac{C_{1}}{4}
\qquad\text{for every }k\in\{0,\ldots,K\} \tag*{(7)} ∥ G k − G ( w k ) ∥ 2 ≤ 4 C 1 for every k ∈ { 0 , … , K } (7)
provided that each batch has size
m ≥ 2048 π ( 1 − 2 η ) 2 P ( t 0 , γ 0 ) ( d log 5 + log 2 ( K + 1 ) δ ) (8) m\ge\frac{2048\pi}{(1-2\eta)^{2}P(t_{0},\gamma_{0})}
\left(d\log5+\log\frac{2(K+1)}{\delta}\right) \tag*{(8)} m ≥ ( 1 − 2 η ) 2 P ( t 0 , γ 0 ) 2048 π ( d log 5 + log δ 2 ( K + 1 ) ) (8)
We verify eq. (8) at the end of the proof, and work on the event of eq. (7) until then. On this event, the norm bound of Lemma D.4, which holds for every direction, gives
∥ G ^ k ∥ 2 ≤ ∥ G ( w k ) ∥ 2 + C 1 4 ≤ 2 C 1 for every k ∈ { 0 , … , K } (9) \lVert\widehat{G}_{k}\rVert_{2}\le\lVert G(w_{k})\rVert_{2}+\frac{C_{1}}{4}\le2C_{1}
\qquad\text{for every }k\in\{0,\ldots,K\} \tag*{(9)} ∥ G k ∥ 2 ≤ ∥ G ( w k ) ∥ 2 + 4 C 1 ≤ 2 C 1 for every k ∈ { 0 , … , K } (9)
The induction. We show that sin ( θ k / 2 ) ≤ ψ k \sin(\theta_k/2) \le\psi_k sin ( θ k /2 ) ≤ ψ k for k = 0 , … , K + 1 k = 0,\ldots,K+1 k = 0 , … , K + 1 . The base case holds because θ 0 ≤ 1 / ( 7 t ) ≤ θ ˉ \theta_0 \le1/(7t) \le\bar{\theta} θ 0 ≤ 1/ ( 7 t ) ≤ θ ˉ . Suppose that sin ( θ k / 2 ) ≤ ψ k \sin(\theta_k/2) \le\psi_k sin ( θ k /2 ) ≤ ψ k for some k ≤ K k \le K k ≤ K . Every summand of G ^ k \widehat{G}_k G k is a multiple of proj w k ⊥ ( X i ) \operatorname{proj}_{w_k^\perp}(X_i) proj w k ⊥ ( X i ) , so G ^ k \widehat{G}_k G k is orthogonal to w k w_k w k , and ∥ w k − μ k G ^ k ∥ 2 ≥ 1 \lVert w_k-\mu_k\widehat{G}_k\rVert_2 \ge1 ∥ w k − μ k G k ∥ 2 ≥ 1 . The normalization in the update is therefore the Euclidean projection onto the unit ball, which is non-expansive and fixes both w k w_k w k and w ⋆ w^\star w ⋆ . Consequently,
∥ w k + 1 − w k ∥ 2 ≤ μ k ∥ G ^ k ∥ 2 and ∥ w k + 1 − w ⋆ ∥ 2 ≤ ∥ w k − μ k G ^ k − w ⋆ ∥ 2 (10) \lVert w_{k+1}-w_k\rVert_2 \le\mu_k\lVert\widehat{G}_k\rVert_2
\qquad\text{and}\qquad
\lVert w_{k+1}-w^\star\rVert_2 \le\lVert w_k-\mu_k\widehat{G}_k-w^\star\rVert_2 \tag*{(10)} ∥ w k + 1 − w k ∥ 2 ≤ μ k ∥ G k ∥ 2 and ∥ w k + 1 − w ⋆ ∥ 2 ≤ ∥ w k − μ k G k − w ⋆ ∥ 2 (10)
We also recall that ∥ u − w ⋆ ∥ 2 = 2 sin ( ∠ ( u , w ⋆ ) / 2 ) \lVert u-w^\star\rVert_2=2\sin(\angle(u,w^\star)/2) ∥ u − w ⋆ ∥ 2 = 2 sin ( ∠ ( u , w ⋆ ) /2 ) for every unit vector u u u . We distinguish two cases.
Suppose first that sin ( θ k / 2 ) ≤ 3 4 ψ k \sin(\theta_k/2) \le\frac{3}{4}\psi_k sin ( θ k /2 ) ≤ 4 3 ψ k . By the triangle inequality, eq. (10) and eq. (9) ,
2 sin ( θ k + 1 / 2 ) ≤ 2 sin ( θ k / 2 ) + 2 μ k C 1 ≤ 3 2 ψ k + 1 − 4 ρ 8 ψ k 2\sin(\theta_{k+1}/2) \le2\sin(\theta_k/2)+2\mu_kC_1 \le\frac{3}{2}\psi_k+\frac{1-4\rho}{8}\psi_k 2 sin ( θ k + 1 /2 ) ≤ 2 sin ( θ k /2 ) + 2 μ k C 1 ≤ 2 3 ψ k + 8 1 − 4 ρ ψ k
and therefore sin ( θ k + 1 / 2 ) ≤ 13 − 4 ρ 16 ψ k ≤ ( 1 − ρ ) ψ k = ψ k + 1 \sin(\theta_{k+1}/2) \le\frac{13-4\rho}{16}\psi_k \le(1-\rho)\psi_k=\psi_{k+1} sin ( θ k + 1 /2 ) ≤ 16 13 − 4 ρ ψ k ≤ ( 1 − ρ ) ψ k = ψ k + 1 .
Suppose now that sin ( θ k / 2 ) > 3 4 ψ k \sin(\theta_k/2)>\frac{3}{4}\psi_k sin ( θ k /2 ) > 4 3 ψ k . The choice of K K K in Algorithm 4 gives 3 4 ψ k ≥ 3 4 ψ K ≥ 4 γ 0 \frac{3}{4}\psi_k\ge\frac{3}{4}\psi_K\ge4\gamma_0 4 3 ψ k ≥ 4 3 ψ K ≥ 4 γ 0 , so that θ k ≥ 2 sin ( θ k / 2 ) > 8 γ 0 \theta_k\ge2\sin(\theta_k/2)>8\gamma_0 θ k ≥ 2 sin ( θ k /2 ) > 8 γ 0 . On the other hand, sin ( θ k / 2 ) ≤ ψ k ≤ sin ( θ ˉ / 2 ) \sin(\theta_k/2)\le\psi_k\le\sin(\bar{\theta}/2) sin ( θ k /2 ) ≤ ψ k ≤ sin ( θ ˉ /2 ) gives θ k ≤ θ ˉ ≤ 1 / ( 5 t ) \theta_k\le\bar{\theta}\le1/(5t) θ k ≤ θ ˉ ≤ 1/ ( 5 t ) . Together with ∣ t 0 − t ∣ ≤ γ 0 \lvert t_0-t\rvert\le\gamma_0 ∣ t 0 − t ∣ ≤ γ 0 , this places w k w_k w k in the range of Lemma D.4, and thus
G ( w k ) ⋅ w ⋆ ≤ − C 1 2 sin θ k G(w_k)\cdot w^\star\le-\frac{C_1}{2}\sin\theta_k G ( w k ) ⋅ w ⋆ ≤ − 2 C 1 sin θ k
The vector G ^ k − G ( w k ) \widehat{G}_k-G(w_k) G k − G ( w k ) is orthogonal to w k w_k w k , and the component of w ⋆ w^\star w ⋆ orthogonal to w k w_k w k has norm sin θ k \sin\theta_k sin θ k . Eq. (7) therefore gives ∣ ( G ^ k − G ( w k ) ) ⋅ w ⋆ ∣ ≤ C 1 4 sin θ k \lvert(\widehat{G}_k-G(w_k))\cdot w^\star\rvert\le\frac{C_1}{4}\sin\theta_k ∣( G k − G ( w k )) ⋅ w ⋆ ∣ ≤ 4 C 1 sin θ k , and hence
G ^ k ⋅ w ⋆ ≤ − C 1 4 sin θ k \widehat{G}_k\cdot w^\star\le-\frac{C_1}{4}\sin\theta_k G k ⋅ w ⋆ ≤ − 4 C 1 sin θ k
Expanding the square on the right-hand side of eq. (10) , and using G ^ k ⋅ w k = 0 \widehat{G}_k\cdot w_k=0 G k ⋅ w k = 0 and eq. (9) , we obtain the one-step inequality of [20 Claim 2.7] ,
4 sin 2 ( θ k + 1 / 2 ) ≤ ∥ w k − w ⋆ ∥ 2 2 + 2 μ k G ^ k ⋅ w ⋆ + μ k 2 ∥ G ^ k ∥ 2 2 4\sin^2(\theta_{k+1}/2)\le\lVert w_k-w^\star\rVert_2^2+2\mu_k\widehat{G}_k\cdot w^\star+\mu_k^2\lVert\widehat{G}_k\rVert_2^2 4 sin 2 ( θ k + 1 /2 ) ≤ ∥ w k − w ⋆ ∥ 2 2 + 2 μ k G k ⋅ w ⋆ + μ k 2 ∥ G k ∥ 2 2
≤ 4 sin 2 ( θ k / 2 ) − C 1 2 μ k sin θ k + 4 C 1 2 μ k 2 \le4\sin^2(\theta_k/2)-\frac{C_1}{2}\mu_k\sin\theta_k+4C_1^2\mu_k^2 ≤ 4 sin 2 ( θ k /2 ) − 2 C 1 μ k sin θ k + 4 C 1 2 μ k 2
Since θ k ≤ 1 / 5 \theta_k\le1/5 θ k ≤ 1/5 , we have sin θ k ≥ sin ( θ k / 2 ) > 3 4 ψ k \sin\theta_k\ge\sin(\theta_k/2)>\frac{3}{4}\psi_k sin θ k ≥ sin ( θ k /2 ) > 4 3 ψ k . Substituting this bound, the induction hypothesis, and μ k = ( 1 − 4 ρ ) ψ k / ( 16 C 1 ) \mu_k=(1-4\rho)\psi_k/(16C_1) μ k = ( 1 − 4 ρ ) ψ k / ( 16 C 1 ) gives
4 sin 2 ( θ k + 1 / 2 ) ≤ 4 ψ k 2 − 3 ( 1 − 4 ρ ) 128 ψ k 2 + ( 1 − 4 ρ ) 2 64 ψ k 2 = 4 ψ k 2 ( 1 − ( 1 − 4 ρ ) ( 1 + 8 ρ ) 512 ) 4\sin^2(\theta_{k+1}/2)\le4\psi_k^2-\frac{3(1-4\rho)}{128}\psi_k^2+\frac{(1-4\rho)^2}{64}\psi_k^2
=4\psi_k^2\left(1-\frac{(1-4\rho)(1+8\rho)}{512}\right) 4 sin 2 ( θ k + 1 /2 ) ≤ 4 ψ k 2 − 128 3 ( 1 − 4 ρ ) ψ k 2 + 64 ( 1 − 4 ρ ) 2 ψ k 2 = 4 ψ k 2 ( 1 − 512 ( 1 − 4 ρ ) ( 1 + 8 ρ ) )
The value ρ = 0.00098 \rho=0.00098 ρ = 0.00098 satisfies 32 ρ 2 + 1020 ρ ≤ 1 32\rho^2+1020\rho\le1 32 ρ 2 + 1020 ρ ≤ 1 , which is equivalent to ( 1 − 4 ρ ) ( 1 + 8 ρ ) / 512 ≥ 2 ρ (1-4\rho)(1+8\rho)/512\ge2\rho ( 1 − 4 ρ ) ( 1 + 8 ρ ) /512 ≥ 2 ρ . Hence
sin ( θ k + 1 / 2 ) ≤ ψ k 1 − 2 ρ ≤ ( 1 − ρ ) ψ k = ψ k + 1 \sin(\theta_{k+1}/2)\le\psi_k\sqrt{1-2\rho}\le(1-\rho)\psi_k=\psi_{k+1} sin ( θ k + 1 /2 ) ≤ ψ k 1 − 2 ρ ≤ ( 1 − ρ ) ψ k = ψ k + 1
which completes the induction.
The final angle. Let θ ^ = ∠ ( w ^ r e f , w ⋆ ) \hat{\theta}=\angle(\widehat{w}_{\mathrm{ref}},w^\star) θ ^ = ∠ ( w ref , w ⋆ ) . If K ≥ 0 K\ge0 K ≥ 0 , the algorithm returns w ^ r e f = w K + 1 \widehat{w}_{\mathrm{ref}}=w_{K+1} w ref = w K + 1 , and the induction gives sin ( θ ^ / 2 ) ≤ ψ K + 1 \sin(\hat{\theta}/2)\le\psi_{K+1} sin ( θ ^ /2 ) ≤ ψ K + 1 , where ψ K + 1 < 16 γ 0 / 3 \psi_{K+1}<16\gamma_0/3 ψ K + 1 < 16 γ 0 /3 by the choice of K K K . If K < 0 K<0 K < 0 , the algorithm returns w ^ r e f = w 0 \widehat{w}_{\mathrm{ref}}=w_0 w ref = w 0 , and sin ( θ ^ / 2 ) ≤ sin ( θ ˉ / 2 ) < 16 γ 0 / 3 \sin(\hat{\theta}/2)\le\sin(\bar{\theta}/2)<16\gamma_0/3 sin ( θ ^ /2 ) ≤ sin ( θ ˉ /2 ) < 16 γ 0 /3 , again by the definition of K K K . In both cases, sin ( θ ^ / 2 ) ≤ sin ( θ ˉ / 2 ) \sin(\hat{\theta}/2)\le\sin(\bar{\theta}/2) sin ( θ ^ /2 ) ≤ sin ( θ ˉ /2 ) , so that θ ^ / 2 ≤ θ ˉ / 2 ≤ 1 / 10 \hat{\theta}/2\le\bar{\theta}/2\le1/10 θ ^ /2 ≤ θ ˉ /2 ≤ 1/10 . Since x ≤ 201 200 sin x x\le\frac{201}{200}\sin x x ≤ 200 201 sin x for x ∈ [ 0 , 1 / 10 ] x\in[0,1/10] x ∈ [ 0 , 1/10 ] , we conclude that
∠ ( w ^ r e f , w ⋆ ) ≤ 201 100 ⋅ 16 3 γ 0 ≤ 11 γ 0 ≤ 12 γ ⋆ \angle(\widehat{w}_{\mathrm{ref}},w^\star)\le\frac{201}{100}\cdot\frac{16}{3}\gamma_0\le11\gamma_0\le12\gamma_\star ∠ ( w ref , w ⋆ ) ≤ 100 201 ⋅ 3 16 γ 0 ≤ 11 γ 0 ≤ 12 γ ⋆
Sample size and runtime. It remains to verify eq. (8) . We have γ 0 ≤ 2 γ ⋆ ≤ 1 / 20 \gamma_0\le2\gamma_\star\le1/20 γ 0 ≤ 2 γ ⋆ ≤ 1/20 and t 0 ≥ 5 / 6 t_0\ge5/6 t 0 ≥ 5/6 , so the interval [ t 0 − γ 0 , t 0 ] [t_0-\gamma_0,t_0] [ t 0 − γ 0 , t 0 ] lies in [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) , where ϕ \phi ϕ is decreasing. Hence
P ( t 0 , γ 0 ) = ∫ t 0 − γ 0 t 0 ϕ ( u ) d u ≥ γ 0 ϕ ( t 0 ) = ϵ 2 2 π ( 1 − 2 η ) P(t_0,\gamma_0)=\int_{t_0-\gamma_0}^{t_0}\phi(u)\,du\ge\gamma_0\phi(t_0)=\frac{\epsilon}{2\sqrt{2\pi}(1-2\eta)} P ( t 0 , γ 0 ) = ∫ t 0 − γ 0 t 0 ϕ ( u ) d u ≥ γ 0 ϕ ( t 0 ) = 2 2 π ( 1 − 2 η ) ϵ
Moreover, log ( 1 / ( 1 − ρ ) ) ≥ ρ \log(1/(1-\rho)) \ge\rho log ( 1/ ( 1 − ρ )) ≥ ρ and 3 sin ( θ ˉ / 2 ) / ( 16 γ 0 ) ≤ 1 / γ 0 ≤ 2 / ε 3\sin(\bar{\theta}/2)/(16\gamma_0) \le1/\gamma_0 \le2/\varepsilon 3 sin ( θ ˉ /2 ) / ( 16 γ 0 ) ≤ 1/ γ 0 ≤ 2/ ε , which gives K + 1 ≤ 1 + log ( 2 / ε ) / ρ = O ( log ( 1 / ε ) ) K+1 \le1+\log(2/\varepsilon)/\rho= O(\log(1/\varepsilon)) K + 1 ≤ 1 + log ( 2/ ε ) / ρ = O ( log ( 1/ ε )) . Eq. (8) therefore holds with batches of size m = O ( ( d + log ( 1 / δ ) + log log ( 1 / ε ) ) / ( ( 1 − 2 η ) ε ) ) m = O((d+\log(1/\delta)+\log\log(1/\varepsilon))/((1-2\eta)\varepsilon)) m = O (( d + log ( 1/ δ ) + log log ( 1/ ε )) / (( 1 − 2 η ) ε )) , for a total of
N 2 = ( K + 1 ) m = O ( log ( 1 / ε ) ( d + log ( 1 / δ ) + log log ( 1 / ε ) ) ( 1 − 2 η ) ε ) = O ~ η ( d + log ( 1 / δ ) ε ) . N_2=(K+1)m=O\left(\frac{\log(1/\varepsilon)\left(d+\log(1/\delta)+\log\log(1/\varepsilon)\right)}{(1-2\eta)\varepsilon}\right)=\widetilde{O}_{\eta}\left(\frac{d+\log(1/\delta)}{\varepsilon}\right). N 2 = ( K + 1 ) m = O ( ( 1 − 2 η ) ε log ( 1/ ε ) ( d + log ( 1/ δ ) + log log ( 1/ ε ) ) ) = O η ( ε d + log ( 1/ δ ) ) .
samples. These bounds on K K K and on P ( t 0 , γ 0 ) P(t_0,\gamma_0) P ( t 0 , γ 0 ) do not depend on the inputs, so N 2 N_2 N 2 can be fixed before w 0 w_0 w 0 and t 0 t_0 t 0 are computed. Finally, iteration k k k computes one inner product and at most one vector addition for each sample of the batch S ( k ) S^{(k)} S ( k ) , and every sample belongs to one batch. The runtime is therefore O ( d N 2 ) O(dN_2) O ( d N 2 ) .
Combining the warm start, the threshold guess, and the refinement with a final threshold ERM proves Theorem 4.2 , which we restate below.
Theorem D.7 (Restatement of Theorem 4.2). Assume the two-stage model of Definition 4.1 . The source stage satisfies the assumptions of Theorem 3.1 with p − ( 1 ) = P D 1 ( Y = − 1 ) p_-^{(1)} = P_{D_1}(Y=-1) p − ( 1 ) = P D 1 ( Y = − 1 ) , and the target stage is η \eta η -RCN with known constant η < 1 / 2 \eta< 1/2 η < 1/2 . Suppose t ≥ 1 t \ge1 t ≥ 1 and p = Φ ( − t ) ≥ C η ε p=\Phi(-t)\ge C_\eta\varepsilon p = Φ ( − t ) ≥ C η ε . If
n 1 = O ~ c , C ( ( d + log ( 1 / δ ) ) max { 1 p , t 4 ( log ( e + t ) ) 2 C p − ( 1 ) } ) n_1=\widetilde{O}_{\mathord{c},C}\left((d+\log(1/\delta))\max\left\{\frac{1}{p},\frac{t^4(\log(e+t))^{2C}}{p_-^{(1)}}\right\}\right) n 1 = O c , C ( ( d + log ( 1/ δ )) max { p 1 , p − ( 1 ) t 4 ( log ( e + t ) ) 2 C } ) source samples and
n 2 = O ~ η ( d + log ( 1 / δ ) ε + 1 ε 2 ) n_2=\widetilde{O}_{\eta}\left(\frac{d+\log(1/\delta)}{\varepsilon}+\frac{1}{\varepsilon^2}\right) n 2 = O η ( ε d + log ( 1/ δ ) + ε 2 1 ) target RCN samples are used, then Algorithm 5 returns a halfspace h ^ \widehat{h} h satisfying
err D 2 ( h ^ ) − err D 2 ( f ⋆ ) ≤ ε \operatorname{err}_{D_2}(\widehat{h})-\operatorname{err}_{D_2}(f^\star)\le\varepsilon err D 2 ( h ) − err D 2 ( f ⋆ ) ≤ ε with probability at least 1 − δ 1-\delta 1 − δ . The runtime is O ~ η ( d ( n 1 + n 2 ) ) \widetilde{O}_{\eta}(d(n_1+n_2)) O η ( d ( n 1 + n 2 )) .
Proof. Split the target samples as in Algorithm 5. We first show that the excess risk is at most C ⋆ ε C_\star\varepsilon C ⋆ ε for a universal constant C ⋆ C_\star C ⋆ , and rescale ε \varepsilon ε at the end.
Warm start. Apply Theorem 3.1 to the source sample with confidence parameter δ / 4 \delta/4 δ /4 . Increasing the hidden constant in n 1 n_1 n 1 if necessary gives
∠ ( w ^ 0 , w ⋆ ) ≤ 1 7 t \angle(\widehat{w}_0,w^\star)\le\frac{1}{7t} ∠ ( w 0 , w ⋆ ) ≤ 7 t 1
with probability at least 1 − δ / 4 1-\delta/4 1 − δ /4 .
Threshold guess. Apply Lemma D.3 to S 2 , b a n d S_{2,\mathrm{band}} S 2 , band with a = 1 / 2 a=1/2 a = 1/2 and confidence parameter δ / 4 \delta/4 δ /4 . With
∣ S 2 , b a n d ∣ = O η ( 1 ε 2 log ( 1 / δ ) ) |S_{2,\mathrm{band}}|=O_{\eta}\left(\frac{1}{\varepsilon^2}\log(1/\delta)\right) ∣ S 2 , band ∣ = O η ( ε 2 1 log ( 1/ δ ) )
we get
∣ t ^ 0 − t ∣ ≤ γ ⋆ 2 |\widehat{t}_0-t|\le\frac{\gamma_\star}{2} ∣ t 0 − t ∣ ≤ 2 γ ⋆
with probability at least 1 − δ / 4 1-\delta/4 1 − δ /4 .
Refinement. On the intersection of these two events, the pair ( w ^ 0 , t ^ 0 ) (\widehat{w}_0,\widehat{t}_0) ( w 0 , t 0 ) satisfies the hypotheses of Theorem D.6 , and it is independent of S 2 , r e f S_{2,\mathrm{ref}} S 2 , ref . Applying that theorem to S 2 , r e f S_{2,\mathrm{ref}} S 2 , ref with confidence parameter δ / 4 \delta/4 δ /4 and
∣ S 2 , r e f ∣ = O ~ η ( d + log ( 1 / δ ) ε ) |S_{2,\mathrm{ref}}|=\widetilde{O}_{\eta}\left(\frac{d+\log(1/\delta)}{\varepsilon}\right) ∣ S 2 , ref ∣ = O η ( ε d + log ( 1/ δ ) )
gives
∠ ( w ^ r e f , w ⋆ ) ≤ 12 γ ⋆ \angle(\widehat{w}_{\mathrm{ref}},w^\star)\le12\gamma_\star ∠ ( w ref , w ⋆ ) ≤ 12 γ ⋆
with probability at least 1 − δ / 4 1-\delta/4 1 − δ /4 .
Final threshold. Algorithm 2 is run on the independent split S 2 , t h r S_{2,\mathrm{thr}} S 2 , thr along w ^ r e f \widehat{w}_{\mathrm{ref}} w ref , and returns an empirical risk minimizer over the thresholds along this direction. We apply Lemma B.18 conditionally on w ^ r e f \widehat{w}_{\mathrm{ref}} w ref , with accuracy ε \varepsilon ε and confidence parameter δ / 4 \delta/4 δ /4 . With
∣ S 2 , t h r ∣ = O ( 1 ε 2 ( log ( 1 / ε ) + log ( 1 / δ ) ) ) |S_{2,\mathrm{thr}}| = O\left(\frac{1}{\varepsilon^2}\left(\log(1/\varepsilon)+\log(1/\delta)\right)\right) ∣ S 2 , thr ∣ = O ( ε 2 1 ( log ( 1/ ε ) + log ( 1/ δ ) ) )
samples, it gives, with probability at least 1 − δ / 4 1-\delta/4 1 − δ /4 ,
err D 2 ( h ^ ) ≤ inf b ∈ R err D 2 ( sign ( ⟨ w ^ r e f , ⋅ ⟩ + b ) ) + ε ≤ err D 2 ( h t ) + ε \operatorname{err}_{D_2}(\widehat{h}) \le\inf_{b\in\mathbb{R}}\operatorname{err}_{D_2}\left(\operatorname{sign}\left(\langle\widehat{w}_{\mathrm{ref}},\cdot\rangle+b\right)\right)+\varepsilon\le\operatorname{err}_{D_2}(h_t)+\varepsilon err D 2 ( h ) ≤ b ∈ R inf err D 2 ( sign ( ⟨ w ref , ⋅ ⟩ + b ) ) + ε ≤ err D 2 ( h t ) + ε
where h t ( x ) = sign ( ⟨ w ^ r e f , x ⟩ + t ) h_t(x)=\operatorname{sign}(\langle\widehat{w}_{\mathrm{ref}},x\rangle+t) h t ( x ) = sign (⟨ w ref , x ⟩ + t ) is the halfspace with direction w ^ r e f \widehat{w}_{\mathrm{ref}} w ref and the true threshold.
Excess risk. It remains to bound the excess risk of h t h_t h t . Under η \eta η -RCN, every classifier h h h satisfies
err D 2 ( h ) − err D 2 ( f ⋆ ) = ( 1 − 2 η ) P ( h ( X ) ≠ f ⋆ ( X ) ) \operatorname{err}_{D_2}(h)-\operatorname{err}_{D_2}(f^\star)=(1-2\eta)\mathbb{P}(h(X)\ne f^\star(X)) err D 2 ( h ) − err D 2 ( f ⋆ ) = ( 1 − 2 η ) P ( h ( X ) = f ⋆ ( X ))
By eq. (6) , after increasing C η C_\eta C η if necessary, 12 γ ⋆ ≤ c 0 / t 12\gamma_\star\le c_0/t 12 γ ⋆ ≤ c 0 / t for the constant c 0 c_0 c 0 of Lemma 2.1. This lemma then gives
P ( h t ( X ) ≠ f ⋆ ( X ) ) ≤ C 0 ∠ ( w ^ r e f , w ⋆ ) ϕ ( t ) ≤ 12 C 0 γ ⋆ ϕ ( t ) = 6 C 0 ε 2 π ( 1 − 2 η ) \mathbb{P}(h_t(X)\ne f^\star(X))\le C_0\angle(\widehat{w}_{\mathrm{ref}},w^\star)\phi(t)\le12C_0\gamma_\star\phi(t)=\frac{6C_0\varepsilon}{\sqrt{2\pi}(1-2\eta)} P ( h t ( X ) = f ⋆ ( X )) ≤ C 0 ∠ ( w ref , w ⋆ ) ϕ ( t ) ≤ 12 C 0 γ ⋆ ϕ ( t ) = 2 π ( 1 − 2 η ) 6 C 0 ε
since γ ⋆ ϕ ( t ) = ε / ( 2 ( 1 − 2 η ) 2 π ) \gamma_\star\phi(t)=\varepsilon/(2(1-2\eta)\sqrt{2\pi}) γ ⋆ ϕ ( t ) = ε / ( 2 ( 1 − 2 η ) 2 π ) . Hence err D 2 ( h t ) − err D 2 ( f ⋆ ) ≤ 6 C 0 ε / 2 π \operatorname{err}_{D_2}(h_t)-\operatorname{err}_{D_2}(f^\star)\le6C_0\varepsilon/\sqrt{2\pi} err D 2 ( h t ) − err D 2 ( f ⋆ ) ≤ 6 C 0 ε / 2 π , and
err D 2 ( h ^ ) − err D 2 ( f ⋆ ) ≤ ( 6 C 0 2 π + 1 ) ε = : C ⋆ ε \operatorname{err}_{D_2}(\widehat{h})-\operatorname{err}_{D_2}(f^\star)\le\left(\frac{6C_0}{\sqrt{2\pi}}+1\right)\varepsilon=:C_\star\varepsilon err D 2 ( h ) − err D 2 ( f ⋆ ) ≤ ( 2 π 6 C 0 + 1 ) ε =: C ⋆ ε
A union bound over the warm start, the threshold guess, the refinement, and the final threshold gives success probability at least 1 − δ 1-\delta 1 − δ . Running the target stage with accuracy parameter ε / C ⋆ \varepsilon/C_\star ε / C ⋆ in place of ε \varepsilon ε gives excess risk at most ε \varepsilon ε . Since p ≥ C η ε ≥ C η ε / C ⋆ p\ge C_\eta\varepsilon\ge C_\eta\varepsilon/C_\star p ≥ C η ε ≥ C η ε / C ⋆ , the assumptions remain satisfied, and n 2 n_2 n 2 changes only by a constant factor.
Sample size and runtime. The sample bound follows by summing the three parts of the target sample. Algorithm 1 makes one pass over the source samples, at cost O ( d n 1 ) O(dn_1) O ( d n 1 ) . The threshold guess uses only the target labels, at cost O ( n 2 ) O(n_2) O ( n 2 ) . The refinement costs O ( d ∣ S 2 , r e f ∣ ) O(d|S_{2,\mathrm{ref}}|) O ( d ∣ S 2 , ref ∣ ) by Theorem D.6, and the final threshold ERM costs O ( d ∣ S 2 , t h r ∣ + ∣ S 2 , t h r ∣ log ∣ S 2 , t h r ∣ ) O(d|S_{2,\mathrm{thr}}|+|S_{2,\mathrm{thr}}|\log|S_{2,\mathrm{thr}}|) O ( d ∣ S 2 , thr ∣ + ∣ S 2 , thr ∣ log ∣ S 2 , thr ∣ ) . Taken together, the runtime is O ~ η ( d ( n 1 + n 2 ) ) \widetilde{O}_\eta(d(n_1+n_2)) O η ( d ( n 1 + n 2 )) .