*New York University (g@csail.mit.edu). Supported by an NSERC Discovery Grant and an Ontario Early Researcher Award.
†Massachusetts Institute of Technology (mahbod@mit.edu).
‡Massachusetts Institute of Technology (moitra@mit.edu). Supported in part by DARPA expMath, a Microsoft Trustworthy AI Grant, NSF-CCF 2430381, an ONR grant, and a David and Lucile Packard Fellowship.
§University of Texas at Austin and University of Waterloo (argymouz@cs.utexas.edu). Supported by an Ontario Early Researcher Award and a David R. Cheriton Graduate Scholarship.
¶Northeastern University (jullman@ccs.neu.edu). Supported by NSF CNS-2232692 and CNS-2247484.
Introduction
Federated learning [MMR$^{+}$17, KMA$^{+}$21] is motivated by the tension between two important goals in modern machine learning:
to train machine learning models from large and diverse data sets, and
while doing so, to preserve the data privacy, security, and ownership of the data, since data is increasingly distributed across a network of devices.
Motivated by such considerations, Qiao and Valiant [QV18] introduced the following problem which they called learning from untrusted batches . There exists an unknown discrete distribution μ \mu μ [ k ] [k] [ k ] n n n m m m μ \mu μ α \alpha α
n = Θ ( k + log ( 1 / β ) α 2 m ) , n = \Theta\left(\frac{k + \log(1/\beta)}{\alpha^{2}m}\right), n = Θ ( α 2 m k + log ( 1/ β ) ) , where β \beta β n m nm nm [14 ] .
However, in federated learning, we can rarely assume that all the batches are sampled i.i.d. from μ \mu μ η \eta η n m nm nm η \eta η O ( η / m ) O(\eta/\sqrt{m}) O ( η / m ) μ \mu μ μ ~ \tilde{\mu} μ ~ γ \gamma γ γ \gamma γ μ ⊗ m \mu^{\otimes m} μ ⊗ m μ ~ ⊗ m \tilde{\mu}^{\otimes m} μ ~ ⊗ m Ω ( min { 1 , γ m } ) \Omega(\min\{1,\gamma\sqrt{m}\}) Ω ( min { 1 , γ m })
Qiao and Valiant gave two algorithms that solve the learning from untrusted batches problem, running in time exponential in k k k m m m [19 ] gave similar algorithms and explored extensions to more complex settings, including improving sample complexity when there are shape constraints on μ \mu μ
Qiao and Valiant motivated their model by the need to learn from batches supplied by potentially malicious agents. Privacy, communication, and computational costs are also central concerns in federated learning. Keeping raw data local does not by itself ensure privacy: shared gradients can still leak private training data [ZLH19].
In this work, we revisit the learning from batches problem using the lens of differential privacy [30 ] . There is an unknown discrete distribution μ \mu μ [ k ] [k] [ k ] m m m batch-level differentially private1
We pursue three goals. First, we require the strong guarantee of pure batch-level differential privacy, formally ( ε , 0 ) (\varepsilon, 0) ( ε , 0 ) 2 α \alpha α
n = O ~ ( k α 2 m + k α m ε + k ε ) . n = \widetilde{O}\left(\frac{k}{\alpha^{2}m}+\frac{k}{\alpha\sqrt{m}\varepsilon}+\frac{k}{\varepsilon}\right). n = O ( α 2 m k + α m ε k + ε k ) . Our final goal is to attain this bound in polynomial time.
Several groups have considered private learning from batches, but none give an algorithm satisfying all three goals. Liu et al. [? ] introduced the problem, giving an efficient algorithm achieving the weaker notion of approximate differential privacy, or, formally, ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) ( ε , 0 ) (\varepsilon,0) ( ε , 0 ) [31 ] , but doing so would increase the required number of batches by a poly ( k ) \operatorname{poly}(k) poly ( k ) ? ] presented a pure differentially private algorithm with the information-theoretically optimal sample complexity, but their stated construction applies private hypothesis selection over an exponentially large cover and is not polynomial-time.
We use the efficient robustness-to-privacy reduction of Hopkins, Kamath, Majid, and Narayanan [? ]. Our contribution is to identify the robust certificate that makes this reduction work for discrete batches. It is a single degree-6 6 6
Results
Our certificate is satisfied by clean data at the right scale, forces every feasible pseudo-mean to be close to the truth, remains stable when batches are discarded, and admits polynomial-time optimization. It yields the robust estimator in Theorem 1.1 and, through the reduction of Hopkins et al. [? ], the private algorithms in Theorem 1.3.
Theorem 1.1 (Robust SoS Estimation from Discrete Batches). Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 β ∈ ( 0 , 1 ) \beta\in(0,1) β ∈ ( 0 , 1 ) m ∈ N m \in\mathbb{N} m ∈ N η > 0 \eta> 0 η > 0 6 6 6 n n n m m m η \eta η
B i ⋆ = ( U i , 1 ⋆ , … , U i , m ⋆ ) , U i , 1 ⋆ , … , U i , m ⋆ ∼ i i d μ , B_i^\star= (U_{i,1}^\star,\ldots,U_{i,m}^\star), \qquad U_{i,1}^\star,\ldots,U_{i,m}^\star\mathrel{\overset{\mathrm{iid}}{\sim}} \mu, B i ⋆ = ( U i , 1 ⋆ , … , U i , m ⋆ ) , U i , 1 ⋆ , … , U i , m ⋆ ∼ iid μ , and outputs an estimate μ ^ \widehat{\mu} μ 1 − β 1-\beta 1 − β
∥ μ ^ − μ ∥ T V ≤ O ( η log ( 1 / η ) m ) \lVert\widehat{\mu}-\mu\rVert_{\mathrm{TV}} \le O\left(\eta\sqrt{\frac{\log(1/\eta)}{m}}\right) ∥ μ − μ ∥ TV ≤ O ( η m log ( 1/ η ) ) provided that
n = Ω ( k + log ( 1 / β ) α 2 m ) n=\Omega\left(\frac{k+\log(1/\beta)}{\alpha^{2}m}\right) n = Ω ( α 2 m k + log ( 1/ β ) ) for α = Θ ( η log ( 1 / η ) / m ) \alpha=\Theta\left(\eta\sqrt{\log(1/\eta)/m}\right) α = Θ ( η log ( 1/ η ) / m )
The diagonal-majorizer certificate defines a low-sensitivity, quasi-convex score (Definition 4.5 ), so the efficient robustness-to-privacy reduction gives the private guarantees stated next.
Theorem 1.3 (Main Result, Private Estimation from Discrete Batches). Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 [ k ] [k] [ k ] α , β ∈ ( 0 , 1 ) \alpha,\beta\in(0,1) α , β ∈ ( 0 , 1 ) ε > 0 \varepsilon>0 ε > 0 m ∈ N m\in\mathbb{N} m ∈ N n n n
B i = ( U i , 1 , … , U i , m ) , U i , 1 , … , U i , m ∼ i i d μ . B_i=(U_{i,1},\ldots,U_{i,m}),\qquad U_{i,1},\ldots,U_{i,m}\mathrel{\overset{\mathrm{iid}}{\sim}}\mu. B i = ( U i , 1 , … , U i , m ) , U i , 1 , … , U i , m ∼ iid μ . Then the following guarantees hold.
(i) There is a polynomial-time ( ε , 0 ) (\varepsilon,0) ( ε , 0 ) μ ^ \widehat{\mu} μ ∥ μ ^ − μ ∥ T V ≤ α \lVert\widehat{\mu}-\mu\rVert_{\mathrm{TV}}\le\alpha ∥ μ − μ ∥ TV ≤ α 1 − β 1-\beta 1 − β
n = O ~ ( k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α m ε + k + log ( 1 / β ) ε ) . n=\widetilde{O}\left(\frac{k+\log(1/\beta)}{\alpha^{2}m}+\frac{k+\log(1/\beta)}{\alpha\sqrt{m}\,\varepsilon}+\frac{k+\log(1/\beta)}{\varepsilon}\right). n = O ( α 2 m k + log ( 1/ β ) + α m ε k + log ( 1/ β ) + ε k + log ( 1/ β ) ) . (ii) For every δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ( ε , δ ) (\varepsilon,\delta) ( ε , δ )
n = O ~ ( k + k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α m ε + log ( 1 / δ ) + log ( 1 / β ) ε ) . n=\widetilde{O}\left(k+\frac{k+\log(1/\beta)}{\alpha^{2}m}+\frac{k+\log(1/\beta)}{\alpha\sqrt{m}\,\varepsilon}+\frac{\log(1/\delta)+\log(1/\beta)}{\varepsilon}\right). n = O ( k + α 2 m k + log ( 1/ β ) + α m ε k + log ( 1/ β ) + ε log ( 1/ δ ) + log ( 1/ β ) ) . The O ~ \widetilde{O} O 1 / α 1/\alpha 1/ α Theorem B.5 gives the dependence on the failure probability explicitly.
Prior polynomial-time approximate-DP results cover different user regimes. Suppressing logarithmic factors other than those displayed below, the relevant privacy terms and user-count requirements are
Liu et al. [LSY+20]:
k log ( 1 / δ ) α ε m , n ≳ k log ( 1 / δ ) ε . \frac{k\sqrt{\log(1/\delta)}}{\alpha\varepsilon\sqrt{m}},\qquad n\gtrsim\frac{\sqrt{k\log(1/\delta)}}{\varepsilon}. α ε m k log ( 1/ δ ) , n ≳ ε k log ( 1/ δ ) . (few users) Esfandiari et al. [32 ] :
k log ( k / δ ) log ( 1 / δ ) log ( k n / β ) α ε m , n ≳ log ( 1 / ( β δ ) ) ε . \frac{k\log(k/\delta)\sqrt{\log(1/\delta)\log(kn/\beta)}}{\alpha\varepsilon\sqrt{m}},\qquad n\gtrsim\frac{\log(1/(\beta\delta))}{\varepsilon}. α ε m k log ( k / δ ) log ( 1/ δ ) log ( k n / β ) , n ≳ ε log ( 1/ ( β δ )) . Esfandiari et al.’s few-user guarantee removes the dimension-dependent user floor required by Liu et al. Their all-regime result combines this guarantee with earlier algorithms for the high-user regime; its larger logarithmic factor is the cost of reaching the few-user regime. For negligible approximate-privacy parameters δ = 2 − k Ω ( 1 ) \delta= 2^{-k^{\Omega(1)}} δ = 2 − k Ω ( 1 ) α , ε , m \alpha,\varepsilon,m α , ε , m k k k log ( 1 / δ ) \log(1/\delta) log ( 1/ δ ) O ~ ( k ) \widetilde{O}(k) O ( k ) δ = 2 − Θ ( k ) \delta= 2^{-\Theta(k)} δ = 2 − Θ ( k ) O ~ ( k ) \widetilde{O}(k) O ( k ) α , ε , m \alpha,\varepsilon,m α , ε , m
Dependence on the batch size. For constant failure probability and 0 < ε ≤ 1 0 < \varepsilon\le1 0 < ε ≤ 1 α m ≤ 1 \alpha\sqrt{m} \le1 α m ≤ 1 k / ( α m ε ) k/(\alpha\sqrt{m}\,\varepsilon) k / ( α m ε ) k / ε k/\varepsilon k / ε α m ≥ 1 \alpha\sqrt{m} \ge1 α m ≥ 1 α \alpha α O ~ ( k / ε ) \widetilde{O}(k/\varepsilon) O ( k / ε )
The additional k k k η \eta η Theorem 1.1 guarantees error O ( η log ( 1 / η ) / m ) = O ( 1 / m ) O(\eta\sqrt{\log(1/\eta)/m}) = O(1/\sqrt{m}) O ( η log ( 1/ η ) / m ) = O ( 1/ m ) Ω ( k ) \Omega(k) Ω ( k ) Ω ( k / ( α 2 m ) ) \Omega(k/(\alpha^{2}m)) Ω ( k / ( α 2 m )) Ω ( k ) \Omega(k) Ω ( k ) α = Θ ( η log ( 1 / η ) / m ) \alpha= \Theta(\eta\sqrt{\log(1/\eta)/m}) α = Θ ( η log ( 1/ η ) / m ) k / ε k/\varepsilon k / ε k k k
For larger batches, the target error α \alpha α 1 / m 1/\sqrt{m} 1/ m k k k [32 ] . Exploiting this regime through robustness-to-privacy requires a robust guarantee adapted to the coarser target accuracy. Remark 4.17 describes how splitting batches reduces the user requirement while retaining additive dependence on log ( 1 / δ ) \log(1/\delta) log ( 1/ δ )
On optimality under efficient computation. Our polynomial-time robust algorithm achieves error O ( η log ( 1 / η ) / m ) O(\eta\sqrt{\log(1/\eta)/m}) O ( η log ( 1/ η ) / m ) O ( η / m ) O(\eta/\sqrt{m}) O ( η / m ) log ( 1 / η ) \sqrt{\log(1/\eta)} log ( 1/ η ) [28 ] and Gaussians [27 ] . In § 5 , we prove a lower bound directly for categorical batches. For every fixed integer t ≥ 1 t \ge1 t ≥ 1 c t > 0 c_t > 0 c t > 0
η min { 1 , log ( 1 / η ) m } . \eta\min\left\{1,\sqrt{\frac{\log(1/\eta)}{m}}\right\}. η min { 1 , m log ( 1/ η ) } . For all sufficiently large k k k t t t 2 k − ( t + 1 ) / 8 2k^{-(t+1)/8} 2 k − ( t + 1 ) /8 exp ( Ω ( k ) ) \exp(\Omega(\sqrt{k})) exp ( Ω ( k )) k , m , 1 / η k,m,1/\eta k , m , 1/ η m m m η \eta η m ≥ log ( 1 / η ) m \ge\log(1/\eta) m ≥ log ( 1/ η ) O ( η ) O(\eta) O ( η ) [28 ] , with a discrete polynomial correction and a balanced subset of the original alphabet. If this SQ lower bound captures the computational threshold for robust estimation, then the privacy-to-robustness connection of Georgiev and Hopkins [GH22] predicts a corresponding threshold for private estimation with high success probability. For 0 < ε ≤ 1 0 < \varepsilon\le1 0 < ε ≤ 1 β = exp ( − Θ ( k ) ) \beta= \exp(-\Theta(k)) β = exp ( − Θ ( k )) η = Θ ( k / ( n ε ) ) \eta= \Theta(k/(n\varepsilon)) η = Θ ( k / ( n ε )) α m \alpha\sqrt{m} α m m ≥ log ( 1 / η ) m \ge\log(1/\eta) m ≥ log ( 1/ η )
k log ( 1 / ( α m ) ) α m ε . \frac{k\sqrt{\log(1/(\alpha\sqrt{m}))}}{\alpha\sqrt{m}\,\varepsilon}. α m ε k log ( 1/ ( α m )) . This agrees with the explicit privacy term in Remark 4.16, including the square-root logarithm. Georgiev and Hopkins derive an analogous conditional square-root logarithmic gap for Gaussian mean estimation [GH22, full version, Corollary 4.2]. This interpretation also applies to approximate DP for δ ≤ ε exp ( − C k ) \delta\le\varepsilon\exp(-Ck) δ ≤ ε exp ( − C k ) C C C
Techniques
We first describe the subset-variance condition that identifies the unknown distribution, then encode it by a polynomial-size semidefinite constraint. We define the score used by the privacy reduction as the minimum number of batches that must be discarded to satisfy this constraint.
Subset-variance control. Let y i ∈ Δ k − 1 y_i \in\Delta^{k-1} y i ∈ Δ k − 1 i i i S ⊆ [ k ] S \subseteq[k] S ⊆ [ k ] s ∈ { 0 , 1 } k s \in\{0,1\}^k s ∈ { 0 , 1 } k ⟨ y i , s ⟩ \langle y_i,s\rangle ⟨ y i , s ⟩ i i i S S S μ ( S ) ( 1 − μ ( S ) ) / m \mu(S)(1-\mu(S))/m μ ( S ) ( 1 − μ ( S )) / m V ( ν , s ) ≔ ⟨ ν , s ⟩ ( 1 − ⟨ ν , s ⟩ ) / m V(\nu,s) \coloneqq\langle\nu,s\rangle(1-\langle\nu,s\rangle)/m V ( ν , s ) : = ⟨ ν , s ⟩ ( 1 − ⟨ ν , s ⟩) / m ν \nu ν
A statistically natural goal is to find a set T ⊆ [ n ] T \subseteq[n] T ⊆ [ n ] ∣ T ∣ ≥ ( 1 − η ) n |T| \ge(1-\eta)n ∣ T ∣ ≥ ( 1 − η ) n μ ˉ T ≔ ∣ T ∣ − 1 ∑ i ∈ T y i \bar{\mu}_T \coloneqq|T|^{-1}\sum_{i\in T}y_i μ ˉ T : = ∣ T ∣ − 1 ∑ i ∈ T y i
∣ 1 ∣ T ∣ ∑ i ∈ T ⟨ y i − μ ˉ T , s ⟩ 2 − V ( μ ˉ T , s ) ∣ ≤ O ( η log ( 1 / η ) m ) (1) \left|\frac{1}{|T|}\sum_{i\in T}\langle y_i-\bar{\mu}_T,s\rangle^2 - V(\bar{\mu}_T,s)\right| \le O\left(\frac{\eta\log(1/\eta)}{m}\right) \tag*{(1)} ∣ T ∣ 1 i ∈ T ∑ ⟨ y i − μ ˉ T , s ⟩ 2 − V ( μ ˉ T , s ) ≤ O ( m η log ( 1/ η ) ) (1) for every s ∈ { 0 , 1 } k s \in\{0,1\}^k s ∈ { 0 , 1 } k
This condition says that, in every subset direction, the empirical variance should agree with the multinomial variance predicted by the empirical mean μ ˉ T \bar{\mu}_T μ ˉ T
Representing this condition poses a problem: there are exponentially many subset directions, and listing all of them does not give the succinct, low-sensitivity certificate needed for privacy. Earlier robust algorithms do not directly provide such a certificate. Jain and Orlitsky [JO20b] obtain the right rate through filtering and randomized rounding. Their guarantee does not directly supply the efficiently computable, sensitivity-one, quasi-convex score needed by the privacy reduction. Chen, Li, and Moitra [19 ] give an SoS formulation. Bounding only second moments in their argument gives O ( n / m ) O(\sqrt{n/m}) O ( n / m ) n n n O ( n 1 − 1 / t t ) O(n^{1-1/t}\sqrt{t}) O ( n 1 − 1/ t t ) O ( t ) O(t) O ( t ) O ( η log ( 1 / η ) / m ) O(\eta\sqrt{\log(1/\eta)/m}) O ( η log ( 1/ η ) / m ) t = Θ ( log ( 1 / η ) ) t=\Theta(\log(1/\eta)) t = Θ ( log ( 1/ η ))
Compressing the subset constraints. The subset-variance condition (1) contains one inequality for each of the 2 k 2^k 2 k [ k ] [k] [ k ]
Two facts justify the replacement. First, clean data admit such a majorizer; proving this uses semidefinite duality and the stronger symmetric Grothendieck inequality of Friedland and Lim [33 ] . Second, testing any feasible majorizer on sign vectors recovers the required upper variance bound in every subset direction. Thus the majorizer is a polynomial-size encoding of the one-sided subset control we need.
Robust SoS program. We define the relaxation in § 3.1 . Its constraints have the following schematic interpretation: retain almost all batches, repair the discarded ones, let μ ′ \mu' μ ′ E i z i : = n − 1 ∑ i = 1 n z i \mathbb{E}_i z_i := n^{-1}\sum_{i=1}^n z_i E i z i := n − 1 ∑ i = 1 n z i w i w_i w i y i ′ y_i' y i ′ μ ′ \mu' μ ′ λ \lambda λ M M M
w i 2 = w i , w i y i ′ = w i y i , ∑ i = 1 n w i ≥ ( 1 − η ) n , w_i^2 = w_i,\qquad w_i y_i' = w_i y_i,\qquad\sum_{i=1}^{n} w_i \ge(1-\eta)n, w i 2 = w i , w i y i ′ = w i y i , i = 1 ∑ n w i ≥ ( 1 − η ) n , y i ′ ∈ Δ k − 1 for all i , μ ′ ∈ Δ k − 1 , μ ′ = E i y i ′ , y_i' \in\Delta^{k-1}\text{ for all }i,\qquad\mu' \in\Delta^{k-1},\qquad\mu' = \mathbb{E}_i y_i', y i ′ ∈ Δ k − 1 for all i , μ ′ ∈ Δ k − 1 , μ ′ = E i y i ′ , Σ ^ ′ + M M T = Σ ( μ ′ ) + Diag ( λ ) , ∑ j = 1 k λ j ≤ τ c o v , \widehat{\Sigma}' + MM^{\mathsf{T}} = \Sigma(\mu') + \operatorname{Diag}(\lambda),\qquad\sum_{j=1}^{k}\lambda_j \le\tau_{\mathrm{cov}}, Σ ′ + M M T = Σ ( μ ′ ) + Diag ( λ ) , j = 1 ∑ k λ j ≤ τ cov , where
Σ ^ ′ : = E i [ ( y i ′ − μ ′ ) ( y i ′ − μ ′ ) T ] , Σ ( ν ) : = 1 m ( diag ( ν ) − ν ν T ) , τ c o v ≍ η log ( 1 / η ) m . \widehat{\Sigma}' := \mathbb{E}_i[(y_i'-\mu')(y_i'-\mu')^{\mathsf{T}}],\qquad\Sigma(\nu) := \frac{1}{m}\left(\operatorname{diag}(\nu)-\nu\nu^{\mathsf{T}}\right),\qquad\tau_{\mathrm{cov}} \asymp\frac{\eta\log(1/\eta)}{m}. Σ ′ := E i [( y i ′ − μ ′ ) ( y i ′ − μ ′ ) T ] , Σ ( ν ) := m 1 ( diag ( ν ) − ν ν T ) , τ cov ≍ m η log ( 1/ η ) . We denote this system by A η \mathcal{A}_{\eta} A η s s s s T Σ ( ν ) s = V ( ν , s ) s^{\mathsf{T}}\Sigma(\nu)s = V(\nu,s) s T Σ ( ν ) s = V ( ν , s ) (1) for the repaired data. Here M M M M M T MM^{\mathsf{T}} M M T Σ ^ ′ − Σ ( μ ′ ) ⪯ Diag ( λ ) \widehat{\Sigma}'-\Sigma(\mu') \preceq\operatorname{Diag}(\lambda) Σ ′ − Σ ( μ ′ ) ⪯ Diag ( λ )
The program repairs discarded batches rather than removing and renormalizing them. On the clean witness, a discarded batch is filled in by its latent clean value; in the soundness proof, the influence of all repaired batches is controlled through their second moment. This represents the upper bound in (1) by a polynomial system over all n n n
We solve the degree-6 relaxation and use its pseudo-mean E ~ μ ′ \widetilde{\mathbb{E}}\mu' E μ ′ ℓ 1 \ell_1 ℓ 1
Clean-data feasibility. We assign the selector, repaired batches, and candidate mean directly from the latent clean data. Let r i = 1 [ y i = y i ∗ ] r_i = \mathbf{1}[y_i = y_i^*] r i = 1 [ y i = y i ∗ ] μ ˉ = E i y i ∗ \bar{\mu} = \mathbb{E}_i y_i^* μ ˉ = E i y i ∗
w i = r i , y i ′ = y i ∗ , μ ′ = μ ˉ . w_i = r_i,\qquad y_i' = y_i^*,\qquad\mu' = \bar{\mu}. w i = r i , y i ′ = y i ∗ , μ ′ = μ ˉ . These assignments satisfy the selector and simplex constraints by inspection. It remains to construct λ \lambda λ
Σ ^ ⋆ : = 1 m ∑ i ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) T , Δ c o v ⋆ : = Σ ^ ⋆ − Σ ( μ ˉ ) . \widehat{\Sigma}^{\star} := \frac{1}{m}\sum_i (y_i^\star-\bar{\mu})(y_i^\star-\bar{\mu})^{\mathsf{T}}, \qquad\Delta_{\mathrm{cov}}^\star:= \widehat{\Sigma}^\star-\Sigma(\bar{\mu}). Σ ⋆ := m 1 i ∑ ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) T , Δ cov ⋆ := Σ ⋆ − Σ ( μ ˉ ) . If Δ c o v ⋆ \Delta_{\mathrm{cov}}^\star Δ cov ⋆ τ c o v \tau_{\mathrm{cov}} τ cov M M T = Diag ( λ ) − Δ c o v ⋆ MM^{\mathsf{T}}=\operatorname{Diag}(\lambda)-\Delta_{\mathrm{cov}}^\star M M T = Diag ( λ ) − Δ cov ⋆
The only probabilistic input is the η \eta η Theorem 1.1 , Corollary 3.3 guarantees this event with the required probability. For feasibility, we use only the following two consequences:
sup s ∈ { 0 , 1 } k ∣ s T Δ c o v ⋆ s ∣ ≤ O ( η log ( 1 / η ) m ) , Δ c o v ⋆ 1 = 0. \sup_{s\in\{0,1\}^k} \left|s^{\mathsf{T}}\Delta_{\mathrm{cov}}^\star s\right| \le O\left(\frac{\eta\log(1/\eta)}{m}\right), \qquad\Delta_{\mathrm{cov}}^\star\mathbf{1}=0. s ∈ { 0 , 1 } k sup s T Δ cov ⋆ s ≤ O ( m η log ( 1/ η ) ) , Δ cov ⋆ 1 = 0. We prove the subset-discrepancy bound in Lemma 3.4 , and the identity Δ c o v ⋆ 1 = 0 \Delta_{\mathrm{cov}}^\star\mathbf{1}=0 Δ cov ⋆ 1 = 0 Lemma 3.5 . From this point on, the feasibility argument is deterministic.
From subset bounds to a diagonal majorizer. Let Δ \Delta Δ Δ 1 = 0 \Delta\mathbf{1}=0 Δ 1 = 0 [33 ] , and the substitution σ = 2 s − 1 \sigma=2s-1 σ = 2 s − 1
min λ ∈ R k : Diag ( λ ) ⪰ Δ ∑ j = 1 k λ j = max X ⪰ 0 diag ( X ) = 1 ⟨ Δ , X ⟩ ≤ max X ⪰ 0 diag ( X ) = 1 ∣ ⟨ Δ , X ⟩ ∣ ≤ C F L max σ ∈ { ± 1 } k ∣ σ T Δ σ ∣ = 4 C F L max s ∈ { 0 , 1 } k ∣ s T Δ s ∣ . \begin{aligned}
\min_{\lambda\in\mathbb{R}^k:\operatorname{Diag}(\lambda)\succeq\Delta}\sum_{j=1}^{k}\lambda_j
&=\max_{\substack{X\succeq0\\\operatorname{diag}(X)=1}}\langle\Delta,X\rangle\\
&\le\max_{\substack{X\succeq0\\\operatorname{diag}(X)=1}}\left|\langle\Delta,X\rangle\right|\\
&\le C_{\mathrm{FL}}\max_{\sigma\in\{\pm1\}^k}\left|\sigma^{\mathsf{T}}\Delta\sigma\right|
=4C_{\mathrm{FL}}\max_{s\in\{0,1\}^k}\left|s^{\mathsf{T}}\Delta s\right|.
\end{aligned} λ ∈ R k : Diag ( λ ) ⪰ Δ min j = 1 ∑ k λ j = X ⪰ 0 diag ( X ) = 1 max ⟨ Δ , X ⟩ ≤ X ⪰ 0 diag ( X ) = 1 max ∣ ⟨ Δ , X ⟩ ∣ ≤ C FL σ ∈ { ± 1 } k max σ T Δ σ = 4 C FL s ∈ { 0 , 1 } k max s T Δ s . where C F L C_{\mathrm{FL}} C FL σ = 2 s − 1 \sigma=2s-1 σ = 2 s − 1 Δ 1 = 0 \Delta\mathbf{1}=0 Δ 1 = 0 λ \lambda λ Lemma 2.11 and corollaries 2.14 and 2.16 for the details.
Applying this chain to Δ c o v ⋆ \Delta_{\mathrm{cov}}^\star Δ cov ⋆ O ( η log ( 1 / η ) / m ) O(\eta\log(1/\eta)/m) O ( η log ( 1/ η ) / m ) τ c o v \tau_{\mathrm{cov}} τ cov
Recovering the subset bounds. To recover the subset bounds, let Δ c o v ′ : = Σ ^ ′ − Σ ( μ ′ ) \Delta_{\mathrm{cov}}' := \widehat{\Sigma}'-\Sigma(\mu') Δ cov ′ := Σ ′ − Σ ( μ ′ ) 1 \mathbf{1} 1 Δ c o v ′ 1 = 0 \Delta_{\mathrm{cov}}'\mathbf{1}=0 Δ cov ′ 1 = 0 s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k σ = 2 s − 1 \sigma=2s-1 σ = 2 s − 1 σ \sigma σ
4 s T Δ c o v ′ s = σ T Δ c o v ′ σ ≤ σ T Diag ( λ ) σ = ∑ j = 1 k λ j . 4s^{\mathsf{T}}\Delta_{\mathrm{cov}}'s=\sigma^{\mathsf{T}}\Delta_{\mathrm{cov}}'\sigma\le\sigma^{\mathsf{T}}\operatorname{Diag}(\lambda)\sigma=\sum_{j=1}^{k}\lambda_j. 4 s T Δ cov ′ s = σ T Δ cov ′ σ ≤ σ T Diag ( λ ) σ = j = 1 ∑ k λ j . Thus testing the majorizer on the sign vector associated with s s s s T Σ ( μ ′ ) s = V ( μ ′ , s ) s^{\mathsf{T}}\Sigma(\mu')s=V(\mu',s) s T Σ ( μ ′ ) s = V ( μ ′ , s )
A η ∣ 6 { ⟨ y i ′ − μ ′ , s ⟩ 2 ≤ V ( μ ′ , s ) + O ( η log ( 1 / η ) m ) } . \mathcal{A}_{\eta\mid6}\left\{\left\langle y_i'-\mu',s\right\rangle^2\le V(\mu',s)+O\left(\frac{\eta\log(1/\eta)}{m}\right)\right\}. A η ∣ 6 { ⟨ y i ′ − μ ′ , s ⟩ 2 ≤ V ( μ ′ , s ) + O ( m η log ( 1/ η ) ) } . This is the upper bound in (1) , applied to the repaired data. The two-sided subset-discrepancy bound in Lemma 3.4 was needed to construct the majorizer. For identifiability, the program uses only the repaired-variance upper bound, together with the clean-retained second-moment lower bound in (9) .
Low-degree SoS identifiability proof. We argue in one dimension. Fix a subset direction s ∈ { 0 , 1 } k s \in\{0,1\}^{k} s ∈ { 0 , 1 } k A η ∣ 6 { ⋯ } \mathcal{A}_{\eta}\vert_{6}\{\cdots\} A η ∣ 6 { ⋯ }
On the η \eta η 2 η 2\eta 2 η μ ′ = E i y i ′ \mu' = \mathbb{E}_{i} y'_i μ ′ = E i y i ′ y i ′ − μ = ( y i ′ − μ ′ ) + ( μ ′ − μ ) y'_i-\mu=(y'_i-\mu')+(\mu'-\mu) y i ′ − μ = ( y i ′ − μ ′ ) + ( μ ′ − μ ) V ( μ ′ , s ) V(\mu',s) V ( μ ′ , s ) V ( μ , s ) V(\mu,s) V ( μ , s )
A η ∣ 6 { ⟨ μ ′ − μ , s ⟩ 2 ≤ O ( η 2 log ( 1 / η ) m ) + 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) + 4 η ⟨ μ ′ − μ , s ⟩ 2 } . (2) \mathcal{A}_{\eta}\vert_{6}\left\{\langle\mu'-\mu,s\rangle^{2}\leq O\left(\frac{\eta^{2}\log(1/\eta)}{m}\right)+4\eta\bigl(V(\mu',s)-V(\mu,s)\bigr)+4\eta\langle\mu'-\mu,s\rangle^{2}\right\}. \tag*{(2)} A η ∣ 6 { ⟨ μ ′ − μ , s ⟩ 2 ≤ O ( m η 2 log ( 1/ η ) ) + 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) + 4 η ⟨ μ ′ − μ , s ⟩ 2 } . (2) We give the full calculation in § 3.2 . In (2) , the O ( η 2 log ( 1 / η ) / m ) O(\eta^{2}\log(1/\eta)/m) O ( η 2 log ( 1/ η ) / m ) 4 η ⟨ μ ′ − μ , s ⟩ 2 4\eta\langle\mu'-\mu,s\rangle^{2} 4 η ⟨ μ ′ − μ , s ⟩ 2 V ( μ ′ , s ) − V ( μ , s ) V(\mu',s)-V(\mu,s) V ( μ ′ , s ) − V ( μ , s )
To control the variance difference, we keep the covariance predicted by μ ′ \mu' μ ′ ν \nu ν s T Σ ( ν ) s = V ( ν , s ) = ⟨ ν , s ⟩ ( 1 − ⟨ ν , s ⟩ ) / m s^{\mathsf{T}}\Sigma(\nu)s=V(\nu,s)=\langle\nu,s\rangle(1-\langle\nu,s\rangle)/m s T Σ ( ν ) s = V ( ν , s ) = ⟨ ν , s ⟩ ( 1 − ⟨ ν , s ⟩) / m V ( μ ′ , s ) − V ( μ , s ) V(\mu',s)-V(\mu,s) V ( μ ′ , s ) − V ( μ , s ) ⟨ μ ′ − μ , s ⟩ \langle\mu'-\mu,s\rangle ⟨ μ ′ − μ , s ⟩ V ( μ ′ , s ) ≤ 1 / ( 4 m ) V(\mu',s)\leq1/(4m) V ( μ ′ , s ) ≤ 1/ ( 4 m ) O ( η / m ) O(\eta/m) O ( η / m ) O ( η / m ) O(\sqrt{\eta/m}) O ( η / m ) [19 ] encounters this limitation; their higher-moment constraints improve the rate.
To absorb the term 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) 4\eta(V(\mu',s)-V(\mu,s)) 4 η ( V ( μ ′ , s ) − V ( μ , s )) (2) , Lemma 3.10 bounds the square of the variance difference by the squared mean error, as shown in the first line below. Apply Young’s inequality from Fact 2.8 with a = V ( μ ′ , s ) − V ( μ , s ) a=V(\mu',s)-V(\mu,s) a = V ( μ ′ , s ) − V ( μ , s ) b = 4 η b=4\eta b = 4 η λ = m 2 / 16 \lambda=m^{2}/16 λ = m 2 /16
A η ∣ 6 { ( V ( μ ′ , s ) − V ( μ , s ) ) 2 ≤ 4 m 2 ⟨ μ ′ − μ , s ⟩ 2 , 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) ≤ m 2 16 ( V ( μ ′ , s ) − V ( μ , s ) ) 2 + 64 η 2 m 2 ≤ 1 4 ⟨ μ ′ − μ , s ⟩ 2 + 64 η 2 m 2 . \mathcal{A}_{\eta}\vert_{6}\left\{
\begin{aligned}
\bigl(V(\mu',s)-V(\mu,s)\bigr)^{2} &\leq\frac{4}{m^{2}}\langle\mu'-\mu,s\rangle^{2},\\
4\eta\bigl(V(\mu',s)-V(\mu,s)\bigr) &\leq\frac{m^{2}}{16}\bigl(V(\mu',s)-V(\mu,s)\bigr)^{2}+\frac{64\eta^{2}}{m^{2}}\\
&\leq\frac{1}{4}\langle\mu'-\mu,s\rangle^{2}+\frac{64\eta^{2}}{m^{2}}.
\end{aligned}
\right. A η ∣ 6 ⎩ ⎨ ⎧ ( V ( μ ′ , s ) − V ( μ , s ) ) 2 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) ≤ m 2 4 ⟨ μ ′ − μ , s ⟩ 2 , ≤ 16 m 2 ( V ( μ ′ , s ) − V ( μ , s ) ) 2 + m 2 64 η 2 ≤ 4 1 ⟨ μ ′ − μ , s ⟩ 2 + m 2 64 η 2 . Substituting into (2) leaves ( 1 / 4 + 4 η ) ⟨ μ ′ − μ , s ⟩ 2 (1/4+4\eta)\langle\mu'-\mu,s\rangle^{2} ( 1/4 + 4 η ) ⟨ μ ′ − μ , s ⟩ 2 η \eta η 64 η 2 / m 2 = O ( η 2 log ( 1 / η ) / m ) 64\eta^{2}/m^{2}=O(\eta^{2}\log(1/\eta)/m) 64 η 2 / m 2 = O ( η 2 log ( 1/ η ) / m )
A η ∣ 6 { ⟨ μ ′ − μ , s ⟩ 2 ≤ O ( η 2 log ( 1 / η ) m ) } . \mathcal{A}_{\eta}\vert_{6}\left\{\langle\mu'-\mu,s\rangle^{2}\leq O\left(\frac{\eta^{2}\log(1/\eta)}{m}\right)\right\}. A η ∣ 6 { ⟨ μ ′ − μ , s ⟩ 2 ≤ O ( m η 2 log ( 1/ η ) ) } . Finally, pseudoexpectation Cauchy–Schwarz and total-variation duality turn this directional certificate into ∥ μ ^ − μ ∥ T V ≤ O ( η log ( 1 / η ) / m ) \lVert\hat{\mu}-\mu\rVert_{\mathrm{TV}}\leq O(\eta\sqrt{\log(1/\eta)/m}) ∥ μ ^ − μ ∥ TV ≤ O ( η log ( 1/ η ) / m ) s s s μ ^ − μ \hat{\mu}-\mu μ ^ − μ
From the robust certificate to a private score. Clean data admit a polynomial-size certificate, and every certificate with a small discard budget is accurate. For privacy, we retain these two facts and allow the discard budget T T T ν \nu ν T T T A η \mathcal{A}_{\eta} A η ∑ i w i ≥ n − T \sum_i w_i \ge n-T ∑ i w i ≥ n − T ℓ 1 \ell_1 ℓ 1 α \alpha α ν \nu ν ν \nu ν
The score satisfies the conditions of the robustness-to-privacy reduction: changing one batch costs one additional discard, so sensitivity is one; combining certificates gives quasi-convexity; and the polynomial-size semidefinite formulation gives efficient computation. One further ingredient is needed: the low-score sets must have controlled volume.
ℓ 1 \ell_1 ℓ 1 ℓ 1 \ell_1 ℓ 1 α \alpha α η ∗ \eta^* η ∗ η ∗ n \eta^* n η ∗ n α + O ( 1 / m ) \alpha+ O(1/\sqrt{m}) α + O ( 1/ m ) Lemma 4.12 ). Since the volume of an ℓ 1 \ell_1 ℓ 1 k k k Lemma 4.12 gives a sharper radius bound at each score level.
Approximate DP requires the analogous comparison at score levels 0.8 η ∗ n 0.8\eta^* n 0.8 η ∗ n η ∗ n \eta^* n η ∗ n ℓ 1 \ell_1 ℓ 1
∥ x ∥ 1 = max σ ∈ { ± 1 } k ⟨ x , σ ⟩ . \lVert x\rVert_1 = \max_{\sigma\in\{\pm1\}^k} \langle x,\sigma\rangle. ∥ x ∥ 1 = σ ∈ { ± 1 } k max ⟨ x , σ ⟩ . If the pseudo-means of two certificates with discard budget at most η ∗ n \eta^* n η ∗ n O ( 1 / m ) O(1/\sqrt{m}) O ( 1/ m ) ℓ 1 \ell_1 ℓ 1 O ( 1 / m ) O(1/\sqrt{m}) O ( 1/ m ) ℓ 1 \ell_1 ℓ 1 ℓ 2 \ell_2 ℓ 2 ℓ 1 \ell_1 ℓ 1 k \sqrt{k} k α \alpha α O ( k log ( 1 + C / ( α m ) ) ) O(k\log(1+C/(\alpha\sqrt{m}))) O ( k log ( 1 + C / ( α m ))) α \alpha α Lemma 4.13 , is needed when α m \alpha\sqrt{m} α m n = Ω ~ ( k ) n=\widetilde{\Omega}(k) n = Ω ( k )
Related Work
Private learning discrete distributions from batches was introduced by Liu et al. [LSY+20], and was subsequently studied by others [32 GKK+23b] . This problem also fits into the broader literature on user-level privacy [LSA+21], which studies adjacent tasks including mean estimation [34 , 32 , 18 GRST24, ZLS+24, 2 ] , PAC learning [GKM21], and convex optimization [GKK+23a, BS23, AL24, GKLM26]. There is likewise an extensive literature on private estimation in the item-level setting, corresponding to m = 1 m=1 m = 1 [ k ] [k] [ k ] ( ε , 0 ) (\varepsilon,0) ( ε , 0 ) [22 ] . This estimator, followed by ℓ 1 \ell_1 ℓ 1 η \eta η O ( η + k / n + k / ( n ε ) ) O(\eta+ \sqrt{k}/n + k/(n\varepsilon)) O ( η + k / n + k / ( n ε )) η \eta η
Qiao and Valiant [QV18] introduced the batch-robust setting, which was subsequently studied by several groups [JO20b, CLM20a], as well as in related settings including structured distributions [JO20a, JO21, CLM20b], linear regression [23 JSK+24, 26 ] , and local differential privacy [21 , 6 ] . Our work, together with many of these results, also fits into the broader line of algorithmic robust statistics, which began with work on robust mean estimation [25 LRV16] and has since expanded substantially; see [24 ] for a textbook treatment. As in the private setting, the interesting algorithmic issues for learning discrete distributions arise because of the batch structure: when m = 1 m = 1 m = 1
Our private algorithm uses the robustness-to-privacy framework introduced by Hopkins, Kamath, Majid, and Narayanan [HKMN23] and independently by Asi, Ullman, and Zakynthinou [10 ] , which converts robust estimators into private ones. Efficient versions of this framework have already yielded near-optimal private and robust algorithms for Gaussian estimation [HKMN23], parameter estimation in random graphs [16 ] , and linear regression [1 ] . Related sum-of-squares exponential mechanisms also give private algorithms for graphon estimation [15 ] . Robustness-to-privacy methods also underlie private Bayesian estimation [17 ] and improved sample bounds for privately learning Gaussian mixtures [9 ] . To our knowledge, this is the first polynomial-time application of this framework that converts batch-level robustness into user-level privacy for learning discrete distributions. Georgiev and Hopkins [GH22] prove a partial converse, showing settings in which privacy automatically yields robustness.
Starting from the original work of Dwork and Lei [29 ] , there has been substantial work on connections between robustness and privacy [12 KSU20, LKO22, BGS+21, RC21, KMV22, HKM22, 3 ] . Some of these papers give alternative general methods for converting robust estimators into private ones [LKO22, KMV22]. Other work from the statistics community studies these connections from a different perspective [7 , 8 RJC25, SM22] , including in the local model [LBY23]. A recent line of work studies estimators that are simultaneously private and robust for high-dimensional problems [12 GKMN21, LKKO21, 32 , 4 KMV22, TCK+22, LKO22, LJK+23] . See the survey of Kamath [Kam24] for a broader discussion of the robustness–privacy interface.
Paper roadmap. § 2 shows how subset control produces a diagonal majorizer. § 3 uses the majorizer in both directions—first for clean-data feasibility and then for degree-6 SoS identifiability. § 4 turns this certificate into a score function and proves the categorical private theorems. § 5 proves the SQ lower bound for robust learning from discrete batches. The appendices give the approximate-DP ℓ 1 \ell_1 ℓ 1
Preliminaries
Problem 2.1 (Robust Learning from Discrete Batches). Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 [ k ] [k] [ k ] m ∈ N m \in\mathbb{N} m ∈ N η \eta η { B i ∗ } i ∈ [ n ] \{B_i^*\}_{i \in[n]} { B i ∗ } i ∈ [ n ] n n n
B i ∗ = { U i , 1 ∗ , … , U i , m ∗ } B_i^* = \{U^*_{i,1}, \ldots, U^*_{i,m}\} B i ∗ = { U i , 1 ∗ , … , U i , m ∗ } and each U i , j ∗ ∼ μ U^{*}_{i,j} \sim\mu U i , j ∗ ∼ μ μ \mu μ η \eta η { B i } i ∈ [ n ] \{B_i\}_{i\in[n]} { B i } i ∈ [ n ] S ⊆ [ n ] S \subseteq[n] S ⊆ [ n ] ∣ S ∣ ≤ η n |S| \le\eta n ∣ S ∣ ≤ η n B i = B i ∗ B_i = B_i^{*} B i = B i ∗ i ∉ S i \notin S i ∈ / S { B i } i ∈ [ n ] \{B_i\}_{i\in[n]} { B i } i ∈ [ n ] μ ^ ∈ Δ k − 1 \hat{\mu} \in\Delta^{k-1} μ ^ ∈ Δ k − 1
∥ μ ^ − μ ∥ T V ≤ α , \lVert\hat{\mu} - \mu\rVert_{\mathrm{TV}} \le\alpha, ∥ μ ^ − μ ∥ TV ≤ α , where α ≥ η log ( 1 / η ) / m \alpha\ge\eta\sqrt{\log(1/\eta)/m} α ≥ η log ( 1/ η ) / m
Data Model and Notation
We use calligraphic letters such as B \mathcal{B} B Y \mathcal{Y} Y
For a batch B i = { U i , 1 , … , U i , m } B_i = \{U_{i,1},\ldots,U_{i,m}\} B i = { U i , 1 , … , U i , m } [ k ] [k] [ k ]
y i : = 1 m ∑ j = 1 m e U i , j ∈ Δ k − 1 , y_i := \frac{1}{m}\sum_{j=1}^{m} e_{U_{i,j}} \in\Delta^{k-1}, y i := m 1 j = 1 ∑ m e U i , j ∈ Δ k − 1 , where e 1 , … , e k e_1,\ldots,e_k e 1 , … , e k R k \mathbb{R}^k R k B i ∗ B_i^{*} B i ∗
y i ∗ : = 1 m ∑ j = 1 m e U i , j ∗ ∈ Δ k − 1 . y_i^{*} := \frac{1}{m}\sum_{j=1}^{m} e_{U_{i,j}^{*}} \in\Delta^{k-1}. y i ∗ := m 1 j = 1 ∑ m e U i , j ∗ ∈ Δ k − 1 . We write
B ∗ = { B i ∗ } i ∈ [ n ] , B = { B i } i ∈ [ n ] , Y ∗ = { y i ∗ } i ∈ [ n ] , Y = { y i } i ∈ [ n ] . \mathcal{B}^{*} = \{B_i^{*}\}_{i\in[n]}, \qquad\mathcal{B} = \{B_i\}_{i\in[n]}, \qquad\mathcal{Y}^{*} = \{y_i^{*}\}_{i\in[n]}, \qquad\mathcal{Y} = \{y_i\}_{i\in[n]}. B ∗ = { B i ∗ } i ∈ [ n ] , B = { B i } i ∈ [ n ] , Y ∗ = { y i ∗ } i ∈ [ n ] , Y = { y i } i ∈ [ n ] . From now on, we work with the averaged vectors Y ∗ \mathcal{Y}^{*} Y ∗ Y \mathcal{Y} Y y i ′ y_i' y i ′ μ ′ \mu' μ ′ μ ^ \hat{\mu} μ ^ ν \nu ν
Probability Simplex
We will repeatedly make use of the probability simplex of distributions over [ k ] [k] [ k ]
Definition 2.1 (Probability Simplex Δ k − 1 \Delta^{k-1} Δ k − 1 Δ k − 1 \Delta^{k-1} Δ k − 1
Δ k − 1 : = { x ∣ ⟨ 1 , x ⟩ = 1 , x ∈ R ≥ 0 k } . \Delta^{k-1} := \{x \mid\langle1,x\rangle= 1,\ x \in\mathbb{R}_{\ge0}^{k}\}. Δ k − 1 := { x ∣ ⟨ 1 , x ⟩ = 1 , x ∈ R ≥ 0 k } . Fact 2.3 (Diagonal minus rank-one is PSD on the simplex). Let x ∈ R ≥ 0 k x \in\mathbb{R}_{\ge0}^{k} x ∈ R ≥ 0 k s : = ∑ i = 1 k x i ≤ 1 s := \sum_{i=1}^{k}x_i \le1 s := ∑ i = 1 k x i ≤ 1 x ∈ Δ k − 1 x \in\Delta^{k-1} x ∈ Δ k − 1 s = 1 s=1 s = 1
M ( x ) : = diag ( x ) − x x T ∈ R k × k . M(x) := \operatorname{diag}(x) - xx^{\mathsf{T}} \in\mathbb{R}^{k\times k}. M ( x ) := diag ( x ) − x x T ∈ R k × k . Then M ( x ) ⪰ 0 M(x) \succeq0 M ( x ) ⪰ 0 supp ( x ) \operatorname{supp}(x) supp ( x )
M ( x ) ⪰ ( 1 − s ) diag ( x ) . M(x) \succeq(1-s)\operatorname{diag}(x). M ( x ) ⪰ ( 1 − s ) diag ( x ) . Proof. Let D : = diag ( x ) D := \operatorname{diag}(x) D := diag ( x ) y : = D 1 / 2 1 y := D^{1/2}\mathbf{1} y := D 1/2 1 M ( x ) M(x) M ( x )
M ( x ) = D 1 / 2 ( I − y y T ) D 1 / 2 , ∥ y ∥ 2 = 1 T D 1 = s ≤ 1. M(x) = D^{1/2}(I-yy^{\mathsf{T}})D^{1/2}, \qquad\lVert y\rVert^2 = \mathbf{1}^{\mathsf{T}}D\mathbf{1} = s \le1. M ( x ) = D 1/2 ( I − y y T ) D 1/2 , ∥ y ∥ 2 = 1 T D 1 = s ≤ 1. Here I − y y T I-yy^{\mathsf{T}} I − y y T 1 1 1 k − 1 k-1 k − 1 y ⊥ y^\perp y ⊥ 1 − ∥ y ∥ 2 = 1 − s ≥ 0 1-\lVert y\rVert^2=1-s\ge0 1 − ∥ y ∥ 2 = 1 − s ≥ 0 y y y I − y y T ⪰ 0 I-yy^{\mathsf{T}}\succeq0 I − y y T ⪰ 0 D 1 / 2 D^{1/2} D 1/2 M ( x ) ⪰ 0 M(x)\succeq0 M ( x ) ⪰ 0 M ( x ) ⪰ ( 1 − s ) diag ( x ) M(x)\succeq(1-s)\operatorname{diag}(x) M ( x ) ⪰ ( 1 − s ) diag ( x )
u T M ( x ) u = ∥ D 1 / 2 u ∥ 2 − ⟨ D 1 / 2 u , y ⟩ 2 ≥ ( 1 − ∥ y ∥ 2 ) ∥ D 1 / 2 u ∥ 2 = ( 1 − s ) u T D u , u^{\mathsf{T}}M(x)u=\lVert D^{1/2}u\rVert^2-\langle D^{1/2}u,y\rangle^2\ge(1-\lVert y\rVert^2)\lVert D^{1/2}u\rVert^2=(1-s)u^{\mathsf{T}}Du, u T M ( x ) u = ∥ D 1/2 u ∥ 2 − ⟨ D 1/2 u , y ⟩ 2 ≥ ( 1 − ∥ y ∥ 2 ) ∥ D 1/2 u ∥ 2 = ( 1 − s ) u T D u , by Cauchy–Schwarz. In the special case s = 1 s=1 s = 1 I − y y T I-yy^{\mathsf{T}} I − y y T M ( x ) M(x) M ( x ) k − 1 k-1 k − 1 1 \mathbf{1} 1
u T M ( x ) u = 1 2 ∑ i , j x i x j ( u i − u j ) 2 ≥ 0. u^{\mathsf{T}}M(x)u=\frac{1}{2}\sum_{i,j}x_i x_j(u_i-u_j)^2\ge0. u T M ( x ) u = 2 1 i , j ∑ x i x j ( u i − u j ) 2 ≥ 0. Fact 2.4 (Mean and Covariance of a Clean Batch Average). Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 [ k ] [k] [ k ] m ∈ N m\in\mathbb{N} m ∈ N
U 1 , … , U m ∼ i i d μ , Y : = 1 m ∑ ℓ = 1 m e U ℓ ∈ Δ k − 1 , U_1,\ldots,U_m\mathrel{\overset{\mathrm{iid}}{\sim}}\mu,\qquad Y:=\frac{1}{m}\sum_{\ell=1}^{m}e_{U_\ell}\in\Delta^{k-1}, U 1 , … , U m ∼ iid μ , Y := m 1 ℓ = 1 ∑ m e U ℓ ∈ Δ k − 1 , and write
Σ ( μ ) : = 1 m ( diag ( μ ) − μ μ T ) . \Sigma(\mu):=\frac{1}{m}\left(\operatorname{diag}(\mu)-\mu\mu^{\mathsf{T}}\right). Σ ( μ ) := m 1 ( diag ( μ ) − μ μ T ) . Then
E [ Y ] = μ , \mathbb{E}[Y]=\mu, E [ Y ] = μ , E [ Y Y T ] − E [ Y ] E [ Y ] T = Σ ( μ ) . \mathbb{E}[YY^{\mathsf{T}}]-\mathbb{E}[Y]\mathbb{E}[Y]^{\mathsf{T}}=\Sigma(\mu). E [ Y Y T ] − E [ Y ] E [ Y ] T = Σ ( μ ) . Proof. Linearity of expectation gives the mean. For the second moment, we have
E [ Y Y T ] = E [ 1 m 2 ∑ ℓ , t ∈ [ m ] e U ℓ e U t T ] = 1 m 2 ( ∑ ℓ ∈ [ m ] E [ e U ℓ e U ℓ T ] + ∑ ℓ ≠ t E [ e U ℓ ] E [ e U t ] T ) = 1 m 2 ( m diag ( μ ) + m ( m − 1 ) μ μ T ) . \begin{aligned}
\mathbb{E}[YY^{\mathsf{T}}]=\mathbb{E}\left[\frac{1}{m^2}\sum_{\ell,t\in[m]}e_{U_\ell}e_{U_t}^{\mathsf{T}}\right] \\
&=\frac{1}{m^2}\left(\sum_{\ell\in[m]}\mathbb{E}[e_{U_\ell}e_{U_\ell}^{\mathsf{T}}]+\sum_{\ell\ne t}\mathbb{E}[e_{U_\ell}]\mathbb{E}[e_{U_t}]^{\mathsf{T}}\right) \\
&=\frac{1}{m^2}\left(m\operatorname{diag}(\mu)+m(m-1)\mu\mu^{\mathsf{T}}\right).
\end{aligned} E [ Y Y T ] = E m 2 1 ℓ , t ∈ [ m ] ∑ e U ℓ e U t T = m 2 1 ℓ ∈ [ m ] ∑ E [ e U ℓ e U ℓ T ] + ℓ = t ∑ E [ e U ℓ ] E [ e U t ] T = m 2 1 ( m diag ( μ ) + m ( m − 1 ) μ μ T ) . Thus,
E [ Y Y T ] − E [ Y ] E [ Y ] T = 1 m 2 ( m diag ( μ ) + m ( m − 1 ) μ μ T ) − μ μ T = 1 m ( diag ( μ ) − μ μ T ) . \begin{aligned}
\mathbb{E}[YY^{\mathsf{T}}]-\mathbb{E}[Y]\mathbb{E}[Y]^{\mathsf{T}}
=\frac{1}{m^2}\left(m\operatorname{diag}(\mu)+m(m-1)\mu\mu^{\mathsf{T}}\right)-\mu\mu^{\mathsf{T}} \\
&=\frac{1}{m}\left(\operatorname{diag}(\mu)-\mu\mu^{\mathsf{T}}\right).
\end{aligned} E [ Y Y T ] − E [ Y ] E [ Y ] T = m 2 1 ( m diag ( μ ) + m ( m − 1 ) μ μ T ) − μ μ T = m 1 ( diag ( μ ) − μ μ T ) . We will also repeatedly use the variance of a clean batch average along a direction s ∈ { 0 , 1 } k s \in\{0,1\}^{k} s ∈ { 0 , 1 } k
Definition 2.5 (Subset-Direction Mean and Variance). Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 s s s k k k { 0 , 1 } k \{0,1\}^{k} { 0 , 1 } k s s s
μ ( s ) : = ⟨ s , μ ⟩ , \mu(s) := \langle s,\mu\rangle, μ ( s ) := ⟨ s , μ ⟩ , V ( μ , s ) : = ⟨ s , μ ⟩ ( 1 − ⟨ s , μ ⟩ ) m . V(\mu,s) := \frac{\langle s,\mu\rangle(1-\langle s,\mu\rangle)}{m}. V ( μ , s ) := m ⟨ s , μ ⟩ ( 1 − ⟨ s , μ ⟩) . Concentration Bounds
We use a version of the clean-sample condition in [JO20b] with an explicit failure probability.
Fact 2.6 (cf. Lemma 3 in [JO20b]). Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 η , β ∈ ( 0 , 1 / 2 ) \eta,\beta\in(0,1/2) η , β ∈ ( 0 , 1/2 ) m ∈ N m \in\mathbb{N} m ∈ N
B i ∗ = ( U i , 1 ∗ , … , U i , m ∗ ) , U i , 1 ∗ , … , U i , m ∗ ∼ i i d μ , y i ∗ : = 1 m ∑ j = 1 m e U i , j ∗ , B_i^* = (U_{i,1}^*,\ldots,U_{i,m}^*), \qquad U_{i,1}^*,\ldots,U_{i,m}^* \overset{\mathrm{iid}}{\sim} \mu, \qquad y_i^* := \frac{1}{m}\sum_{j=1}^{m} e_{U_{i,j}^*}, B i ∗ = ( U i , 1 ∗ , … , U i , m ∗ ) , U i , 1 ∗ , … , U i , m ∗ ∼ iid μ , y i ∗ := m 1 j = 1 ∑ m e U i , j ∗ , for each i ∈ [ n ] i \in[n] i ∈ [ n ]
n ≥ Ω ( k + log ( 1 / β ) η 2 log ( 1 / η ) ) , n \ge\Omega\left(\frac{k+\log(1/\beta)}{\eta^2\log(1/\eta)}\right), n ≥ Ω ( η 2 log ( 1/ η ) k + log ( 1/ β ) ) , with probability 1 − β 1-\beta 1 − β T ⊆ [ n ] T \subseteq[n] T ⊆ [ n ] ∣ T ∣ ≥ ( 1 − η ) n |T| \ge(1-\eta)n ∣ T ∣ ≥ ( 1 − η ) n s ∈ { 0 , 1 } k s \in\{0,1\}^{k} s ∈ { 0 , 1 } k
∣ 1 ∣ T ∣ ∑ i ∈ T ⟨ y i ∗ , s ⟩ − ⟨ μ , s ⟩ ∣ ≤ O ( η log ( 1 / η ) / m ) , \left|\frac{1}{|T|}\sum_{i\in T}\langle y_i^*,s\rangle-\langle\mu,s\rangle\right| \le O\left(\eta\sqrt{\log(1/\eta)/m}\right), ∣ T ∣ 1 i ∈ T ∑ ⟨ y i ∗ , s ⟩ − ⟨ μ , s ⟩ ≤ O ( η log ( 1/ η ) / m ) , and
∣ 1 ∣ T ∣ ∑ i ∈ T ⟨ s , y i ∗ − μ ⟩ 2 − V ( μ , s ) ∣ ≤ O ( η log ( 1 / η ) / m ) . \left|\frac{1}{|T|}\sum_{i\in T}\langle s,y_i^*-\mu\rangle^2-V(\mu,s)\right| \le O\left(\eta\log(1/\eta)/m\right). ∣ T ∣ 1 i ∈ T ∑ ⟨ s , y i ∗ − μ ⟩ 2 − V ( μ , s ) ≤ O ( η log ( 1/ η ) / m ) . Proof. Write ξ i ( s ) = ⟨ y i ∗ − μ , s ⟩ \xi_i(s)=\langle y_i^*-\mu,s\rangle ξ i ( s ) = ⟨ y i ∗ − μ , s ⟩ s s s P [ ∣ ξ i ( s ) ∣ > t ] ≤ 2 exp ( − 2 m t 2 ) \mathbb{P}[|\xi_i(s)|>t]\le2\exp(-2mt^2) P [ ∣ ξ i ( s ) ∣ > t ] ≤ 2 exp ( − 2 m t 2 ) ξ i ( s ) 2 \xi_i(s)^2 ξ i ( s ) 2 O ( 1 / m ) O(1/m) O ( 1/ m ) 2 k 2^k 2 k s s s 1 − β / 2 1-\beta/2 1 − β /2
∣ E i ξ i ( s ) ∣ ≤ O ( k + log ( 2 / β ) n m ) , \left|\mathbb{E}_i\,\xi_i(s)\right| \le O\left(\sqrt{\frac{k+\log(2/\beta)}{nm}}\right), ∣ E i ξ i ( s ) ∣ ≤ O ( nm k + log ( 2/ β ) ) , ∣ E i ξ i ( s ) 2 − V ( μ , s ) ∣ ≤ O ( 1 ) m ( k + log ( 2 / β ) n + k + log ( 2 / β ) n ) \left|\mathbb{E}_i\,\xi_i(s)^2-V(\mu,s)\right| \le\frac{O(1)}{m}\left(\sqrt{\frac{k+\log(2/\beta)}{n}}+\frac{k+\log(2/\beta)}{n}\right) E i ξ i ( s ) 2 − V ( μ , s ) ≤ m O ( 1 ) ( n k + log ( 2/ β ) + n k + log ( 2/ β ) ) simultaneously for all s s s η n ≥ 1 \eta n \ge1 η n ≥ 1 U U U ⌈ η n ⌉ \lceil\eta n\rceil ⌈ η n ⌉ ∑ i ∈ U ξ i ( s ) 2 ≤ O ( ( ∣ U ∣ + t ) / m ) \sum_{i\in U}\xi_i(s)^2 \le O((|U|+t)/m) ∑ i ∈ U ξ i ( s ) 2 ≤ O (( ∣ U ∣ + t ) / m ) e − t e^{-t} e − t s s s ( n ⌈ η n ⌉ ) ≤ ( e n / ⌈ η n ⌉ ) ⌈ η n ⌉ \binom{n}{\lceil\eta n\rceil}\le(en/\lceil\eta n\rceil)^{\lceil\eta n\rceil} ( ⌈ η n ⌉ n ) ≤ ( e n / ⌈ η n ⌉ ) ⌈ η n ⌉ 1 − β / 2 1-\beta/2 1 − β /2
max ∣ U ∣ ≤ η n 1 n ∑ i ∈ U ξ i ( s ) 2 ≤ O ( 1 ) m ( η log ( 1 / η ) + k + log ( 2 / β ) n ) ≤ O ( η log ( 1 / η ) m ) \max_{|U|\le\eta n}\frac{1}{n}\sum_{i\in U}\xi_i(s)^2 \le\frac{O(1)}{m}\left(\eta\log(1/\eta)+\frac{k+\log(2/\beta)}{n}\right)\le O\left(\frac{\eta\log(1/\eta)}{m}\right) ∣ U ∣ ≤ η n max n 1 i ∈ U ∑ ξ i ( s ) 2 ≤ m O ( 1 ) ( η log ( 1/ η ) + n k + log ( 2/ β ) ) ≤ O ( m η log ( 1/ η ) ) for every s s s T = [ n ] ∖ U T=[n]\setminus U T = [ n ] ∖ U ∣ n − 1 ∑ i ∈ U ξ i ( s ) ∣ |n^{-1}\sum_{i\in U}\xi_i(s)| ∣ n − 1 ∑ i ∈ U ξ i ( s ) ∣ ( ∣ U ∣ / n ) ( n − 1 ∑ i ∈ U ξ i ( s ) 2 ) \sqrt{(|U|/n)(n^{-1}\sum_{i\in U}\xi_i(s)^2)} ( ∣ U ∣/ n ) ( n − 1 ∑ i ∈ U ξ i ( s ) 2 ) U U U ∣ T ∣ / n ≥ 1 / 2 |T|/n\ge1/2 ∣ T ∣/ n ≥ 1/2 T T T 2 η V ( μ , s ) ≤ η / ( 2 m ) 2\eta V(\mu,s)\le\eta/(2m) 2 η V ( μ , s ) ≤ η / ( 2 m )
Sum-of-Squares
Fact 2.7 (Triangle inequality). Let a a a b b b
{ ( a + b ) 2 ≤ 2 a 2 + 2 b 2 } 2 a , b . \left\{(a+b)^2\le2a^2+2b^2\right\}_{2}^{a,b}. { ( a + b ) 2 ≤ 2 a 2 + 2 b 2 } 2 a , b . Fact 2.8 (Young’s inequality). Let a a a b b b λ > 0 \lambda>0 λ > 0
{ a b ≤ λ a 2 + b 2 4 λ } 2 a , b . \left\{ab\le\lambda a^2+\frac{b^2}{4\lambda}\right\}_{2}^{a,b}. { ab ≤ λ a 2 + 4 λ b 2 } 2 a , b . Proof. We can write
{ λ a 2 + b 2 4 λ − a b = ( λ a − b 2 λ ) 2 ≥ 0 } 2 a , b , \left\{\lambda a^2+\frac{b^2}{4\lambda}-ab=
\left(\sqrt{\lambda}a-\frac{b}{2\sqrt{\lambda}}\right)^2\ge0\right\}_{2}^{a,b}, { λ a 2 + 4 λ b 2 − ab = ( λ a − 2 λ b ) 2 ≥ 0 } 2 a , b , as desired.
Lemma 2.9 (SoS Selector Lemma [1 ] ). Let B , a 1 , … , a n ∈ R B,a_1,\ldots,a_n\in\mathbb{R} B , a 1 , … , a n ∈ R k ∈ N k\in\mathbb{N} k ∈ N ∑ i ∈ T a i ≤ B \sum_{i\in T}a_i\le B ∑ i ∈ T a i ≤ B T ⊆ [ n ] T\subseteq[n] T ⊆ [ n ] ∣ T ∣ ≤ k |T|\le k ∣ T ∣ ≤ k
{ ∀ i ∈ [ n ] : 0 ≤ b i ≤ 1 , ∑ i = 1 n b i = k } ⊢ 2 b 1 , … , b n { ∑ i = 1 n b i a i ≤ B } . \left\{\forall i\in[n]:0\le b_i\le1,\ \sum_{i=1}^{n}b_i=k\right\}\vdash_{2}^{b_1,\ldots,b_n}\left\{\sum_{i=1}^{n}b_i a_i\le B\right\}. { ∀ i ∈ [ n ] : 0 ≤ b i ≤ 1 , i = 1 ∑ n b i = k } ⊢ 2 b 1 , … , b n { i = 1 ∑ n b i a i ≤ B } . Additionally, if the a i a_i a i ∑ i ∈ [ n ] b i = k \sum_{i\in[n]}b_i=k ∑ i ∈ [ n ] b i = k ∑ i ∈ [ n ] b i ≤ k \sum_{i\in[n]}b_i\le k ∑ i ∈ [ n ] b i ≤ k 0 ≤ b i ≤ 1 0\le b_i\le1 0 ≤ b i ≤ 1 i i i t ∈ [ 0 , n ] t\in[0,n] t ∈ [ 0 , n ] a i ≥ 0 a_i\ge0 a i ≥ 0 ∑ i ∈ T a i ≤ B \sum_{i\in T}a_i\le B ∑ i ∈ T a i ≤ B ∣ T ∣ ≤ t |T|\le t ∣ T ∣ ≤ t
{ ∀ i : b i 2 = b i , ∑ i b i ≤ t } ⊢ 4 { ∑ i a i b i ≤ 2 B } . \left\{\forall i:b_i^2=b_i,\ \sum_i b_i\le t\right\}\vdash_{4}\left\{\sum_i a_i b_i\le2B\right\}. { ∀ i : b i 2 = b i , i ∑ b i ≤ t } ⊢ 4 { i ∑ a i b i ≤ 2 B } . Proof of the real-budget extension. For t ≥ 1 t\ge1 t ≥ 1 a 1 ≥ ⋯ ≥ a n a_1\ge\cdots\ge a_n a 1 ≥ ⋯ ≥ a n 0 ≤ b i ≤ 1 0\le b_i\le1 0 ≤ b i ≤ 1
∑ i a i b i ≤ a ⌊ t ⌋ ∑ i b i + ∑ i ≤ ⌊ t ⌋ ( a i − a ⌊ t ⌋ ) ≤ ∑ i ≤ ⌊ t ⌋ a i + ( t − ⌊ t ⌋ ) a ⌊ t ⌋ ≤ t ⌊ t ⌋ ∑ i ≤ ⌊ t ⌋ a i ≤ 2 B . \begin{aligned}
\sum_i a_i b_i\le a_{\lfloor t\rfloor}\sum_i b_i+\sum_{i\le\lfloor t\rfloor}(a_i-a_{\lfloor t\rfloor}) \\
&\le\sum_{i\le\lfloor t\rfloor}a_i+(t-\lfloor t\rfloor)a_{\lfloor t\rfloor} \\
&\le\frac{t}{\lfloor t\rfloor}\sum_{i\le\lfloor t\rfloor}a_i\le2B.
\end{aligned} i ∑ a i b i ≤ a ⌊ t ⌋ i ∑ b i + i ≤ ⌊ t ⌋ ∑ ( a i − a ⌊ t ⌋ ) ≤ i ≤ ⌊ t ⌋ ∑ a i + ( t − ⌊ t ⌋) a ⌊ t ⌋ ≤ ⌊ t ⌋ t i ≤ ⌊ t ⌋ ∑ a i ≤ 2 B . These are degree-2 SoS inequalities. If t < 1 t<1 t < 1
( t − 1 ) b i = b i 2 ( t − ∑ j b j ) + ∑ j ≠ i ( b i b j ) 2 ≥ 0. (t-1)b_i=b_i^2\left(t-\sum_j b_j\right)+\sum_{j\ne i}(b_i b_j)^2\ge0. ( t − 1 ) b i = b i 2 ( t − j ∑ b j ) + j = i ∑ ( b i b j ) 2 ≥ 0. Thus b i ≤ 0 b_i\le0 b i ≤ 0 ∑ i a i b i ≤ 0 ≤ 2 B \sum_i a_i b_i\le0\le2B ∑ i a i b i ≤ 0 ≤ 2 B
Definition 2.10 (Approximately Satisfying Linear Functionals). Let n ∈ N n \in\mathbb{N} n ∈ N T ∈ [ 0 , n ] T \in[0,n] T ∈ [ 0 , n ] A T \mathcal{A}_T A T T T T w = ( w 1 , … , w n ) w=(w_1,\ldots,w_n) w = ( w 1 , … , w n ) z z z
A T = { ∀ i ∈ [ m ≥ ] : q i ( z , w ) ≥ 0 } ∪ { ∀ j ∈ [ m e q ] : r j ( z , w ) = 0 } ∪ { ∑ i ∈ [ n ] w i ≥ n − T } . \mathcal{A}_T=\left\{\forall i\in[m_{\geq}]:q_i(z,w)\geq0\right\}\cup\left\{\forall j\in[m_{\mathrm{eq}}]:r_j(z,w)=0\right\}\cup\left\{\sum_{i\in[n]}w_i\geq n-T\right\}. A T = { ∀ i ∈ [ m ≥ ] : q i ( z , w ) ≥ 0 } ∪ { ∀ j ∈ [ m eq ] : r j ( z , w ) = 0 } ∪ ⎩ ⎨ ⎧ i ∈ [ n ] ∑ w i ≥ n − T ⎭ ⎬ ⎫ . Let R \mathcal{R} R τ ≥ 0 \tau\geq0 τ ≥ 0 L \mathcal{L} L τ \tau τ A T \mathcal{A}_T A T D D D
L 1 = 1 \mathcal{L}1=1 L 1 = 1
For all degree at most D / 2 D/2 D /2 p p p ∥ R ( p ) ∥ 2 ≤ 1 \|\mathcal{R}(p)\|_2\leq1 ∥ R ( p ) ∥ 2 ≤ 1 L p 2 ≥ − τ T \mathcal{L}p^2\geq-\tau T L p 2 ≥ − τ T
For all i ∈ [ m ≥ ] i\in[m_{\geq}] i ∈ [ m ≥ ] p p p deg ( p 2 q i ) ≤ D \deg(p^2q_i)\leq D deg ( p 2 q i ) ≤ D ∥ R ( p ) ∥ 2 ≤ 1 \|\mathcal{R}(p)\|_2\leq1 ∥ R ( p ) ∥ 2 ≤ 1 L p 2 q i ≥ − τ T \mathcal{L}p^2q_i\geq-\tau T L p 2 q i ≥ − τ T
For all j ∈ [ m e q ] j\in[m_{\mathrm{eq}}] j ∈ [ m eq ] p p p deg ( p 2 r j ) ≤ D \deg(p^2r_j)\leq D deg ( p 2 r j ) ≤ D ∥ R ( p ) ∥ 2 ≤ 1 \|\mathcal{R}(p)\|_2\leq1 ∥ R ( p ) ∥ 2 ≤ 1 ∣ L p 2 r j ∣ ≤ τ T \lvert\mathcal{L}p^2r_j\rvert\leq\tau T ∣ L p 2 r j ∣ ≤ τ T
For all polynomials p p p deg ( p 2 ( ∑ i w i − n + T ) ) ≤ D \deg\left(p^2\left(\sum_i w_i-n+T\right)\right)\leq D deg ( p 2 ( ∑ i w i − n + T ) ) ≤ D ∥ R ( p ) ∥ 2 ≤ 1 \|\mathcal{R}(p)\|_2\leq1 ∥ R ( p ) ∥ 2 ≤ 1 L [ p 2 ( ∑ i w i − n + T ) ] ≥ − 5 τ T n \mathcal{L}\left[p^2\left(\sum_i w_i-n+T\right)\right]\geq-5\tau Tn L [ p 2 ( ∑ i w i − n + T ) ] ≥ − 5 τ T n
Subset-Variance Control and the Diagonal-Majorizer SDP
We isolate the abstract bridge from subset-direction control of a covariance-gap matrix to a trace-bounded diagonal majorizer. Let Δ ∈ R k × k \Delta\in\mathbb{R}^{k\times k} Δ ∈ R k × k s T Δ s s^\mathsf{T}\Delta s s T Δ s s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k
Lemma 2.11 (Subset-to-sign reduction under a nullspace condition). Let Δ ∈ R k × k \Delta\in\mathbb{R}^{k\times k} Δ ∈ R k × k Δ 1 = 0 \Delta\mathbf{1}=0 Δ 1 = 0 s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k σ = 2 s − 1 \sigma=2s-1 σ = 2 s − 1
∣ σ T Δ σ ∣ = 4 ∣ s T Δ s ∣ . \lvert\sigma^\mathsf{T}\Delta\sigma\rvert=4\lvert s^\mathsf{T}\Delta s\rvert. ∣ σ T Δ σ ∣ = 4 ∣ s T Δ s ∣ . Proof.
σ T Δ σ = ( 2 s − 1 ) T Δ ( 2 s − 1 ) = 4 s T Δ s \sigma^\mathsf{T}\Delta\sigma=(2s-1)^\mathsf{T}\Delta(2s-1)=4s^\mathsf{T}\Delta s σ T Δ σ = ( 2 s − 1 ) T Δ ( 2 s − 1 ) = 4 s T Δ s because Δ 1 = 0 \Delta\mathbf{1}=0 Δ 1 = 0
Definition 2.12 (Elliptope). Define
E k ≔ { X ∈ R k × k ∣ X ⪰ 0 , diag ( X ) = 1 } . \mathcal{E}_k\coloneqq\left\{X\in\mathbb{R}^{k\times k}\mid X\succeq0,\ \operatorname{diag}(X)=1\right\}. E k : = { X ∈ R k × k ∣ X ⪰ 0 , diag ( X ) = 1 } . Each rank-one matrix σ σ T \sigma\sigma^\mathsf{T} σ σ T σ ∈ { ± 1 } k \sigma\in\{\pm1\}^k σ ∈ { ± 1 } k E k \mathcal{E}_k E k
Theorem 2.13 (Symmetric Grothendieck inequality, real γ \gamma γ [33 ] ). Let A = ( a i j ) ∈ R k × k A=(a_{ij})\in\mathbb{R}^{k\times k} A = ( a ij ) ∈ R k × k d ≥ k d\ge k d ≥ k
max x 1 , … , x k ∈ R d ∥ x i ∥ 2 = 1 ∀ i ∣ ∑ i , j = 1 k a i j ⟨ x i , x j ⟩ ∣ ≤ sinh ( π / 2 ) max σ ∈ { ± 1 } k ∣ ∑ i , j = 1 k a i j σ i σ j ∣ . \max_{\substack{x_1,\ldots,x_k\in\mathbb{R}^d\\ \lVert x_i\rVert_2=1\ \forall i}}\left|\sum_{i,j=1}^{k}a_{ij}\langle x_i,x_j\rangle\right|\le\sinh(\pi/2)\max_{\sigma\in\{\pm1\}^{k}}\left|\sum_{i,j=1}^{k}a_{ij}\sigma_i\sigma_j\right|. x 1 , … , x k ∈ R d ∥ x i ∥ 2 = 1 ∀ i max i , j = 1 ∑ k a ij ⟨ x i , x j ⟩ ≤ sinh ( π /2 ) σ ∈ { ± 1 } k max i , j = 1 ∑ k a ij σ i σ j . Equivalently, this is the real symmetric γ \gamma γ
Corollary 2.14 (Correlation-matrix form). Let A ∈ R k × k A\in\mathbb{R}^{k\times k} A ∈ R k × k
max X ⪰ 0 , diag ( X ) = 1 ∣ ⟨ A , X ⟩ ∣ ≤ sinh ( π / 2 ) max σ ∈ { ± 1 } k ∣ σ T A σ ∣ . \max_{\substack{X\succeq0,\ \operatorname{diag}(X)=1}}|\langle A,X\rangle|\le\sinh(\pi/2)\max_{\sigma\in\{\pm1\}^{k}}|\sigma^{\mathsf T}A\sigma|. X ⪰ 0 , diag ( X ) = 1 max ∣ ⟨ A , X ⟩ ∣ ≤ sinh ( π /2 ) σ ∈ { ± 1 } k max ∣ σ T A σ ∣. Proof. Every feasible X X X x 1 , … , x k ∈ R r x_1,\ldots,x_k\in\mathbb{R}^r x 1 , … , x k ∈ R r r ≤ k r\le k r ≤ k
X i j = ⟨ x i , x j ⟩ for all i , j ∈ [ k ] . X_{ij}=\langle x_i,x_j\rangle\qquad\text{for all }i,j\in[k]. X ij = ⟨ x i , x j ⟩ for all i , j ∈ [ k ] . Therefore
⟨ A , X ⟩ = ∑ i , j = 1 k a i j X i j = ∑ i , j = 1 k a i j ⟨ x i , x j ⟩ , \langle A,X\rangle=\sum_{i,j=1}^{k}a_{ij}X_{ij}=\sum_{i,j=1}^{k}a_{ij}\langle x_i,x_j\rangle, ⟨ A , X ⟩ = i , j = 1 ∑ k a ij X ij = i , j = 1 ∑ k a ij ⟨ x i , x j ⟩ , and the claim follows from Theorem 2.13 .
We use the real symmetric indefinite absolute-value form from Friedland–Lim because Δ \Delta Δ
Fact 2.15 (Semidefinite duality with equality constraints). Let C , A 1 , … , A m ∈ R s y m d × d C,A_1,\ldots,A_m\in\mathbb{R}_{\mathrm{sym}}^{d\times d} C , A 1 , … , A m ∈ R sym d × d b ∈ R m b\in\mathbb{R}^m b ∈ R m
(P) max X ⪰ 0 ⟨ C , X ⟩ such that ⟨ A t , X ⟩ = b t for all t ∈ [ m ] , (D) min y ∈ R m ∑ t = 1 m b t y t such that ∑ t = 1 m y t A t − C ⪰ 0. \begin{aligned}
\text{(P)}\quad&\max_{X\succeq0}\ \langle C,X\rangle&&\text{such that }\langle A_t,X\rangle=b_t\text{ for all }t\in[m],\\
\text{(D)}\quad&\min_{y\in\mathbb{R}^m}\ \sum_{t=1}^{m}b_ty_t &&\text{such that }\sum_{t=1}^{m}y_tA_t-C\succeq0.
\end{aligned} (P) (D) X ⪰ 0 max ⟨ C , X ⟩ y ∈ R m min t = 1 ∑ m b t y t such that ⟨ A t , X ⟩ = b t for all t ∈ [ m ] , such that t = 1 ∑ m y t A t − C ⪰ 0. If both programs are strictly feasible, then strong duality holds and the two optimal values are equal.
Corollary 2.16 (Diagonal-majorizer SDP duality). Let Δ ∈ R k × k \Delta\in\mathbb{R}^{k\times k} Δ ∈ R k × k
max X ⪰ 0 , diag ( X ) = 1 ⟨ Δ , X ⟩ = min λ ∈ R k { ∑ j ∈ [ k ] λ j | Diag ( λ ) − Δ ⪰ 0 } . \max_{\substack{X\succeq0,\ \operatorname{diag}(X)=1}}\langle\Delta,X\rangle=\min_{\lambda\in\mathbb{R}^k}\left\{\sum_{j\in[k]}\lambda_j\ \middle|\ \operatorname{Diag}(\lambda)-\Delta\succeq0\right\}. X ⪰ 0 , diag ( X ) = 1 max ⟨ Δ , X ⟩ = λ ∈ R k min ⎩ ⎨ ⎧ j ∈ [ k ] ∑ λ j Diag ( λ ) − Δ ⪰ 0 ⎭ ⎬ ⎫ . Moreover, both programs are strictly feasible.
Proof. Apply Fact 2.15 with A j = e j e j T A_j=e_je_j^{\mathsf T} A j = e j e j T b j = 1 b_j=1 b j = 1 C = Δ C=\Delta C = Δ X = I k X=I_k X = I k λ = C 1 \lambda=C\mathbf{1} λ = C 1 C C C
Proposition 2.17 (Subset Control Implies Diagonal-Majorizer Feasibility). Let Δ ∈ R k × k \Delta\in\mathbb{R}^{k \times k} Δ ∈ R k × k
Δ 1 = 0 , sup s ∈ { 0 , 1 } k ∣ s T Δ s ∣ ≤ ρ . \Delta\mathbf{1}=0,\qquad\sup_{s\in\{0,1\}^{k}} \left|s^{\mathsf{T}}\Delta s\right| \le\rho. Δ 1 = 0 , s ∈ { 0 , 1 } k sup s T Δ s ≤ ρ . Then there exists λ ∈ R k \lambda\in\mathbb{R}^{k} λ ∈ R k
Diag ( λ ) − Δ ⪰ 0 , ∑ j ∈ [ k ] λ j ≤ O ( ρ ) . \operatorname{Diag}(\lambda)-\Delta\succeq0,\qquad\sum_{j\in[k]} \lambda_j \le O(\rho). Diag ( λ ) − Δ ⪰ 0 , j ∈ [ k ] ∑ λ j ≤ O ( ρ ) . Proof. By Lemma 2.11,
max σ ∈ { ± 1 } k ∣ σ T Δ σ ∣ ≤ 4 ρ . \max_{\sigma\in\{\pm1\}^{k}}\left|\sigma^{\mathsf{T}}\Delta\sigma\right| \le4\rho. σ ∈ { ± 1 } k max σ T Δ σ ≤ 4 ρ . By Corollary 2.14,
max X ⪰ 0 , diag ( X ) = 1 ⟨ Δ , X ⟩ ≤ max X ⪰ 0 , diag ( X ) = 1 ∣ ⟨ Δ , X ⟩ ∣ ≤ sinh ( π / 2 ) max σ ∈ { ± 1 } k ∣ σ T Δ σ ∣ ≤ O ( ρ ) . \max_{X\succeq0,\ \operatorname{diag}(X)=1} \langle\Delta,X\rangle
\le\max_{X\succeq0,\ \operatorname{diag}(X)=1} \left|\langle\Delta,X\rangle\right|
\le\sinh(\pi/2)\max_{\sigma\in\{\pm1\}^{k}}\left|\sigma^{\mathsf{T}}\Delta\sigma\right|
\le O(\rho). X ⪰ 0 , diag ( X ) = 1 max ⟨ Δ , X ⟩ ≤ X ⪰ 0 , diag ( X ) = 1 max ∣ ⟨ Δ , X ⟩ ∣ ≤ sinh ( π /2 ) σ ∈ { ± 1 } k max σ T Δ σ ≤ O ( ρ ) . Applying Corollary 2.16 yields the claim. □
Robustness to Privacy
We apply the following theorem in order to transform robust algorithms into private ones.
Theorem 2.18 ([HKMN23]). Let ε > 0 \varepsilon>0 ε > 0 β ∈ ( 0 , 1 ) \beta\in(0,1) β ∈ ( 0 , 1 ) 0 < η , r < 1 < R 0<\eta,r<1<R 0 < η , r < 1 < R S ( θ , Y ) ∈ [ 0 , n ] \mathcal{S}(\theta,\mathcal{Y})\in[0,n] S ( θ , Y ) ∈ [ 0 , n ] Y = { y 1 , … , y n } \mathcal{Y}=\{y_1,\ldots,y_n\} Y = { y 1 , … , y n } θ ∈ Θ ⊂ B ( R ) d \theta\in\Theta\subset B(R)^d θ ∈ Θ ⊂ B ( R ) d Θ \Theta Θ R R R
(Bounded Sensitivity) For any two adjacent datasets Y , Y ′ \mathcal{Y},\mathcal{Y}' Y , Y ′ θ ∈ Θ \theta\in\Theta θ ∈ Θ ∣ S ( θ , Y ) − S ( θ , Y ′ ) ∣ ≤ 1 \left|\mathcal{S}(\theta,\mathcal{Y})-\mathcal{S}(\theta,\mathcal{Y}')\right|\le1 ∣ S ( θ , Y ) − S ( θ , Y ′ ) ∣ ≤ 1
(Quasi-Convexity) For any fixed dataset Y \mathcal{Y} Y θ , θ ′ ∈ Θ \theta,\theta'\in\Theta θ , θ ′ ∈ Θ 0 ≤ λ ≤ 1 0\le\lambda\le1 0 ≤ λ ≤ 1 S ( λ θ + ( 1 − λ ) θ ′ , Y ) ≤ max ( S ( θ , Y ) , S ( θ ′ , Y ) ) \mathcal{S}(\lambda\theta+(1-\lambda)\theta',\mathcal{Y})\le\max(\mathcal{S}(\theta,\mathcal{Y}),\mathcal{S}(\theta',\mathcal{Y})) S ( λ θ + ( 1 − λ ) θ ′ , Y ) ≤ max ( S ( θ , Y ) , S ( θ ′ , Y ))
(Efficiently Computable) For any given θ ∈ Θ \theta\in\Theta θ ∈ Θ Y \mathcal{Y} Y S ( θ , Y ) \mathcal{S}(\theta,\mathcal{Y}) S ( θ , Y ) γ \gamma γ poly ( n , d , log ( R / r ) , log γ − 1 ) \operatorname{poly}(n,d,\log(R/r),\log\gamma^{-1}) poly ( n , d , log ( R / r ) , log γ − 1 ) γ > 0 \gamma>0 γ > 0
(Robust algorithm finds low-scoring point) For a given dataset Y \mathcal{Y} Y T = min θ 0 ∈ Θ S ( θ 0 , Y ) T=\min_{\theta_0\in\Theta}\mathcal{S}(\theta_0,\mathcal{Y}) T = min θ 0 ∈ Θ S ( θ 0 , Y ) θ \theta θ θ ′ \theta' θ ′ r r r θ \theta θ S ( θ ′ , Y ) ≤ T + 1 \mathcal{S}(\theta',\mathcal{Y})\le T+1 S ( θ ′ , Y ) ≤ T + 1 poly ( n , d , log ( R / r ) ) \operatorname{poly}(n,d,\log(R/r)) poly ( n , d , log ( R / r ))
(Volume) For any given dataset Y \mathcal{Y} Y η ′ ≥ η \eta'\ge\eta η ′ ≥ η V η ′ ( Y ) V_{\eta'}(\mathcal{Y}) V η ′ ( Y ) d d d θ ∈ Θ ⊂ R d \theta\in\Theta\subset\mathbb{R}^{d} θ ∈ Θ ⊂ R d η ′ n \eta'n η ′ n V 1 ( Y ) V_1(\mathcal{Y}) V 1 ( Y ) Θ \Theta Θ
Then, we have a pure ε \varepsilon ε A \mathcal{A} A n n n poly ( n , d , log ( R / r ) ) \operatorname{poly}(n,d,\log(R/r)) poly ( n , d , log ( R / r )) Y \mathcal{Y} Y θ \theta θ S ( θ , Y ) ≤ η n \mathcal{S}(\theta,\mathcal{Y})\le\eta n S ( θ , Y ) ≤ η n
n ≥ Ω ( max η ′ : η ≤ η ′ ≤ 1 log ( V η ′ ( Y ) / V η ( Y ) ) + log ( 1 / ( β ⋅ η ) ) ε ⋅ η ′ ) , n\ge\Omega\left(\max_{\eta':\eta\le\eta'\le1}\frac{\log\left(V_{\eta'}(\mathcal{Y})/V_{\eta}(\mathcal{Y})\right)+\log(1/(\beta\cdot\eta))}{\varepsilon\cdot\eta'}\right), n ≥ Ω ( η ′ : η ≤ η ′ ≤ 1 max ε ⋅ η ′ log ( V η ′ ( Y ) / V η ( Y ) ) + log ( 1/ ( β ⋅ η )) ) , then A ( Y ) \mathcal{A}(\mathcal{Y}) A ( Y ) θ ∈ Θ \theta\in\Theta θ ∈ Θ 2 η n 2\eta n 2 η n 1 − β 1-\beta 1 − β
For context we also record the corresponding approximate-DP reduction theorem. Relative to Theorem 2.18, the only extra input is the intermediate volume-ratio condition at levels 0.8 η ∗ 0.8\eta^* 0.8 η ∗ η ∗ \eta^* η ∗
Theorem 2.19 ([HKMN23]). Let ε > 0 \varepsilon> 0 ε > 0 δ , β ∈ ( 0 , 1 ) \delta,\beta\in(0,1) δ , β ∈ ( 0 , 1 ) 0 < η < 0.1 0 < \eta< 0.1 0 < η < 0.1 0 < r < 1 < R 0 < r < 1 < R 0 < r < 1 < R S ( θ , Y ) ∈ [ 0 , ∞ ) \mathcal{S}(\theta,\mathcal{Y}) \in[0,\infty) S ( θ , Y ) ∈ [ 0 , ∞ ) Y = { y 1 , … , y n } \mathcal{Y} = \{y_1,\ldots,y_n\} Y = { y 1 , … , y n } θ ∈ Θ ⊂ R d \theta\in\Theta\subset\mathbb{R}^{d} θ ∈ Θ ⊂ R d Θ \Theta Θ R R R η ∗ ∈ [ 10 η , 1 ] \eta^{*} \in[10\eta,1] η ∗ ∈ [ 10 η , 1 ]
n ≥ Ω ( log ( 1 / δ ) + log ( V η ∗ ( Y ) / V 0.8 η ∗ ( Y ) ) ε ⋅ η ∗ ) (3) n \ge\Omega\left(\frac{\log(1/\delta) + \log(V_{\eta^{*}}(\mathcal{Y})/V_{0.8\eta^{*}}(\mathcal{Y}))}{\varepsilon\cdot\eta^{*}}\right) \tag*{(3)} n ≥ Ω ( ε ⋅ η ∗ log ( 1/ δ ) + log ( V η ∗ ( Y ) / V 0.8 η ∗ ( Y )) ) (3) for all Y \mathcal{Y} Y θ \theta θ S ( θ , Y ) ≤ 0.7 η ∗ n \mathcal{S}(\theta,\mathcal{Y}) \le0.7\eta^{*}n S ( θ , Y ) ≤ 0.7 η ∗ n ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) A \mathcal{A} A poly ( n , d , log ( R / r ) ) \operatorname{poly}(n,d,\log(R/r)) poly ( n , d , log ( R / r )) Y \mathcal{Y} Y θ \theta θ S ( θ , Y ) ≤ η n \mathcal{S}(\theta,\mathcal{Y}) \le\eta n S ( θ , Y ) ≤ η n
n ≥ Ω ( max η : η ≤ η ′ ≤ η ∗ log ( V η ′ ( Y ) / V η ( Y ) ) + log ( 1 / ( β ⋅ η ) ) ε ⋅ η ′ ) , (4) n \ge\Omega\left(\max_{\eta:\eta\le\eta'\le\eta^{*}}\frac{\log(V_{\eta'}(\mathcal{Y})/V_{\eta}(\mathcal{Y}))+\log(1/(\beta\cdot\eta))}{\varepsilon\cdot\eta'}\right), \tag*{(4)} n ≥ Ω ( η : η ≤ η ′ ≤ η ∗ max ε ⋅ η ′ log ( V η ′ ( Y ) / V η ( Y )) + log ( 1/ ( β ⋅ η )) ) , (4) then A ( Y ) \mathcal{A}(\mathcal{Y}) A ( Y ) θ ∈ Θ \theta\in\Theta θ ∈ Θ S ( θ , Y ) ≤ 2 η n \mathcal{S}(\theta,\mathcal{Y}) \le2\eta n S ( θ , Y ) ≤ 2 η n 1 − β 1-\beta 1 − β
Lemma 2.20 (A refined utility bound). Suppose 0 < ε ≤ 1 0 < \varepsilon\le1 0 < ε ≤ 1 η n \eta n η n r r r Theorem 2.18 and in (4) , the term log ( 1 / ( β η ) ) \log(1/(\beta\eta)) log ( 1/ ( β η )) log ( 2 / β ) \log(2/\beta) log ( 2/ β ) (3) for approximate DP.
Proof. We refine the tail calculation for the samplers of [HKMN23, full version, Lemmas 4.7 and 4.8]. These samplers assign weights proportional to exp ( − c ε S ) \exp(-c\varepsilon\mathcal{S}) exp ( − c ε S ) 0.9 η ∗ n + O ( 1 ) 0.9\eta^{*}n + O(1) 0.9 η ∗ n + O ( 1 ) η \eta η
Each nonempty shell with score in [ 2 j η n , 2 j + 1 η n ) [2^{j}\eta n,2^{j+1}\eta n) [ 2 j η n , 2 j + 1 η n ) j ≥ 1 j \ge1 j ≥ 1
A exp ( − c ε n ( 2 j − 1 ) η ) { V min { 2 j + 1 η , 1 } ( Y ) V η ( Y ) , for pure DP , V min { 2 j + 1 η , η ∗ } ( Y ) V η ( Y ) , for approximate DP , A\exp(-c\varepsilon n(2^{j}-1)\eta)
\begin{cases}
\dfrac{V_{\min\{2^{j+1}\eta,1\}}(\mathcal{Y})}{V_{\eta}(\mathcal{Y})}, & \text{for pure DP},\\[6pt]
\dfrac{V_{\min\{2^{j+1}\eta,\eta^{*}\}}(\mathcal{Y})}{V_{\eta}(\mathcal{Y})}, & \text{for approximate DP},
\end{cases} A exp ( − c ε n ( 2 j − 1 ) η ) ⎩ ⎨ ⎧ V η ( Y ) V m i n { 2 j + 1 η , 1 } ( Y ) , V η ( Y ) V m i n { 2 j + 1 η , η ∗ } ( Y ) , for pure DP , for approximate DP , for absolute constants A , c > 0 A,c>0 A , c > 0 O ( 1 ) O(1) O ( 1 ) A A A ε ≤ 1 \varepsilon\le1 ε ≤ 1
n ≥ { C max η ≤ u ≤ 1 log ( V u ( Y ) / V η ( Y ) ) + log ( 2 / β ) ε u , for pure DP , C max η ≤ u ≤ η ∗ log ( V u ( Y ) / V η ( Y ) ) + log ( 2 / β ) ε u , for approximate DP , (5) n \ge
\begin{cases}
C\max_{\eta\le u\le1}\frac{\log(V_{u}(\mathcal{Y})/V_{\eta}(\mathcal{Y}))+\log(2/\beta)}{\varepsilon u}, & \text{for pure DP},\\[10pt]
C\max_{\eta\le u\le\eta^{*}}\frac{\log(V_{u}(\mathcal{Y})/V_{\eta}(\mathcal{Y}))+\log(2/\beta)}{\varepsilon u}, & \text{for approximate DP},
\end{cases} \tag*{(5)} n ≥ ⎩ ⎨ ⎧ C max η ≤ u ≤ 1 ε u l o g ( V u ( Y ) / V η ( Y )) + l o g ( 2/ β ) , C max η ≤ u ≤ η ∗ ε u l o g ( V u ( Y ) / V η ( Y )) + l o g ( 2/ β ) , for pure DP , for approximate DP , (5) then the capped volume levels are at most 2 j + 1 η ≤ 4 ( 2 j − 1 ) η 2^{j+1}\eta\le4(2^{j}-1)\eta 2 j + 1 η ≤ 4 ( 2 j − 1 ) η C C C u = η u=\eta u = η (5) also gives ε n η ≥ C log ( 2 / β ) \varepsilon n\eta\ge C\log(2/\beta) ε n η ≥ C log ( 2/ β )
A ∑ j ≥ 1 exp ( − c 2 ε n ( 2 j − 1 ) η ) ≤ β / 2. A\sum_{j\ge1}\exp\left(-\frac{c}{2}\varepsilon n(2^{j}-1)\eta\right)\le\beta/2. A j ≥ 1 ∑ exp ( − 2 c ε n ( 2 j − 1 ) η ) ≤ β /2. For approximate DP, the reduction’s test based on the minimum score accepts whenever that minimum is at most η n \eta n η n η ∗ ≥ 10 η \eta^{*}\ge10\eta η ∗ ≥ 10 η (5) gives η n ≥ C log ( 2 / β ) / ε \eta n \ge C\log(2/\beta)/\varepsilon η n ≥ C log ( 2/ β ) / ε η ∗ n \eta^{*}n η ∗ n 0.9 η ∗ n + O ( 1 ) < η ∗ n 0.9\eta^{*}n+O(1)<\eta^{*}n 0.9 η ∗ n + O ( 1 ) < η ∗ n min { β / 2 , c 0 δ } \min\{\beta/2,c_{0}\delta\} min { β /2 , c 0 δ } c 0 > 0 c_{0}>0 c 0 > 0 □ \square □
Robust Estimation for Discrete Batches
In this section, we design a robust estimation algorithm. In § 4 we will show how to use this robust algorithm (Theorem 3.1 ) and Theorem 2.18 to obtain a private algorithm. For the technical arguments in this section, fix a sufficiently small absolute constant η a b s ∈ ( 0 , 1 / 20 ) \eta_{\mathrm{abs}} \in(0,1/20) η abs ∈ ( 0 , 1/20 )
Theorem 3.1 (Robust Estimation from Discrete Batches). Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 η ∈ ( 0 , η a b s ) \eta\in(0,\eta_{\mathrm{abs}}) η ∈ ( 0 , η abs ) β ∈ ( 0 , 1 ) \beta\in(0,1) β ∈ ( 0 , 1 ) m ∈ N m \in\mathbb{N} m ∈ N n n n m m m η \eta η
B i ⋆ = ( U i , 1 ⋆ , … , U i , m ⋆ ) , U i , 1 ⋆ , … , U i , m ⋆ ∼ i i d μ , B_i^\star= (U_{i,1}^\star,\ldots,U_{i,m}^\star), \qquad U_{i,1}^\star,\ldots,U_{i,m}^\star\overset{\mathrm{iid}}{\sim} \mu, B i ⋆ = ( U i , 1 ⋆ , … , U i , m ⋆ ) , U i , 1 ⋆ , … , U i , m ⋆ ∼ iid μ , and outputs an estimate μ ^ \widehat{\mu} μ ∥ μ ^ − μ ∥ T V ≤ α \lVert\widehat{\mu}-\mu\rVert_{\mathrm{TV}} \le\alpha ∥ μ − μ ∥ TV ≤ α 1 − β 1-\beta 1 − β α = O ( η log ( 1 / η ) / m ) \alpha= O(\eta\sqrt{\log(1/\eta)/m}) α = O ( η log ( 1/ η ) / m )
n = Ω ( k + log ( 1 / β ) α 2 m ) . n = \Omega\left(\frac{k+\log(1/\beta)}{\alpha^2m}\right). n = Ω ( α 2 m k + log ( 1/ β ) ) . We will view the robust constraints through the variance discrepancy for a kept subset with empirical mean μ ˉ T \bar{\mu}_T μ ˉ T Σ ^ T \widehat{\Sigma}_T Σ T
⟨ s s ⊤ , Σ ^ T ⟩ − 1 m ( ⟨ μ ˉ T , s ⟩ − ⟨ μ ˉ T , s ⟩ 2 ) , s ∈ { 0 , 1 } k . \langle ss^\top,\widehat{\Sigma}_T\rangle-\frac{1}{m}\left(\langle\bar{\mu}_T,s\rangle-\langle\bar{\mu}_T,s\rangle^2\right), \qquad s\in\{0,1\}^k. ⟨ s s ⊤ , Σ T ⟩ − m 1 ( ⟨ μ ˉ T , s ⟩ − ⟨ μ ˉ T , s ⟩ 2 ) , s ∈ { 0 , 1 } k . This is the object optimized by the SDP-based algorithms of Jain and Orlitsky [JO20b] and Chen, Li, and Moitra [20 ] . Our succinct SoS certificate will be the dual object: a trace-bounded diagonal majorizer.
We isolate the deterministic properties of the clean samples needed in the robust analysis.
Definition 3.2 (η \eta η μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 η ∈ ( 0 , 1 / 2 ) \eta\in(0,1/2) η ∈ ( 0 , 1/2 ) m ∈ N m \in\mathbb{N} m ∈ N y 1 ⋆ , … , y n ⋆ ∈ R k y_1^\star,\ldots,y_n^\star\in\mathbb{R}^k y 1 ⋆ , … , y n ⋆ ∈ R k s ∈ { 0 , 1 } k s \in\{0,1\}^k s ∈ { 0 , 1 } k
ξ i ( s ) ≔ ⟨ y i ⋆ − μ , s ⟩ . \xi_i(s) \coloneqq\langle y_i^\star-\mu,s\rangle. ξ i ( s ) : = ⟨ y i ⋆ − μ , s ⟩ . We say that y 1 ⋆ , … , y n ⋆ y_1^\star,\ldots,y_n^\star y 1 ⋆ , … , y n ⋆ η \eta η s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k
∣ E i ξ i ( s ) ∣ ≤ O ( η log ( 1 / η ) / m ) \left|\mathbb{E}_i \xi_i(s)\right| \le O(\eta\sqrt{\log(1/\eta)/m}) ∣ E i ξ i ( s ) ∣ ≤ O ( η log ( 1/ η ) / m )
∣ E i ξ i ( s ) 2 − V ( μ , s ) ∣ ≤ O ( η log ( 1 / η ) / m ) \left|\mathbb{E}_i \xi_i(s)^2-V(\mu,s)\right| \le O(\eta\log(1/\eta)/m) E i ξ i ( s ) 2 − V ( μ , s ) ≤ O ( η log ( 1/ η ) / m )
For every subset T ⊆ [ n ] T\subseteq[n] T ⊆ [ n ] ∣ T ∣ ≤ 2 η n |T|\le2\eta n ∣ T ∣ ≤ 2 η n
1 n ∑ i ∈ T ξ i ( s ) 2 ≤ O ( η log ( 1 / η ) / m ) . \frac{1}{n}\sum_{i\in T}\xi_i(s)^2 \le O(\eta\log(1/\eta)/m). n 1 i ∈ T ∑ ξ i ( s ) 2 ≤ O ( η log ( 1/ η ) / m ) . Clean batch-average vectors are η \eta η
Corollary 3.3 (True Samples Are η \eta η μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 ) β ∈ ( 0 , 1 / 2 ) \beta\in(0,1/2) β ∈ ( 0 , 1/2 ) m ∈ N m\in\mathbb{N} m ∈ N
B i ⋆ = ( U i , 1 ⋆ , … , U i , m ⋆ ) , U i , 1 ⋆ , … , U i , m ⋆ ∼ i i d μ , y i ⋆ ≔ 1 m ∑ j = 1 m e U i , j ⋆ B_i^\star=(U_{i,1}^\star,\ldots,U_{i,m}^\star), \qquad U_{i,1}^\star,\ldots,U_{i,m}^\star\overset{\mathrm{iid}}{\sim}\mu, \qquad y_i^\star\coloneqq\frac{1}{m}\sum_{j=1}^m e_{U_{i,j}^\star} B i ⋆ = ( U i , 1 ⋆ , … , U i , m ⋆ ) , U i , 1 ⋆ , … , U i , m ⋆ ∼ iid μ , y i ⋆ : = m 1 j = 1 ∑ m e U i , j ⋆ for each i ∈ [ n ] i\in[n] i ∈ [ n ] n ≥ Ω ( ( k + log ( 1 / β ) ) / ( η 2 log ( 1 / η ) ) ) n\ge\Omega((k+\log(1/\beta))/(\eta^2\log(1/\eta))) n ≥ Ω (( k + log ( 1/ β )) / ( η 2 log ( 1/ η ))) 1 − β 1-\beta 1 − β y 1 ⋆ , … , y n ⋆ y_1^\star,\ldots,y_n^\star y 1 ⋆ , … , y n ⋆ η \eta η Definition 3.2 .
Proof. Apply Fact 2.6 with parameter 2 η 2\eta 2 η Corollary 3.3 , up to absolute constants, so the concentration event holds with probability 1 − β 1-\beta 1 − β T = [ n ] T=[n] T = [ n ] Definition 3.2 .
For the third property, fix a subset T ⊆ [ n ] T\subseteq[n] T ⊆ [ n ] ∣ T ∣ ≤ 2 η n |T|\leq2\eta n ∣ T ∣ ≤ 2 η n S : = [ n ] ∖ T S:=[n]\setminus T S := [ n ] ∖ T ∣ S ∣ ≥ ( 1 − 2 η ) n |S|\geq(1-2\eta)n ∣ S ∣ ≥ ( 1 − 2 η ) n
1 n ∑ i ∈ T ξ i ( s ) 2 ≤ E ξ i ( s ) 2 − ∣ S ∣ n ⋅ 1 ∣ S ∣ ∑ i ∈ S ξ i ( s ) 2 . \frac{1}{n}\sum_{i\in T}\xi_i(s)^2 \leq\mathbb{E}\,\xi_i(s)^2-\frac{|S|}{n}\cdot\frac{1}{|S|}\sum_{i\in S}\xi_i(s)^2. n 1 i ∈ T ∑ ξ i ( s ) 2 ≤ E ξ i ( s ) 2 − n ∣ S ∣ ⋅ ∣ S ∣ 1 i ∈ S ∑ ξ i ( s ) 2 . Therefore,
1 n ∑ i ∈ T ξ i ( s ) 2 ≤ ∣ E i ξ i ( s ) 2 − V ( μ , s ) ∣ + ∣ S ∣ n ∣ 1 ∣ S ∣ ∑ i ∈ S ξ i ( s ) 2 − V ( μ , s ) ∣ + ∣ T ∣ n V ( μ , s ) ≤ O ( η log ( 1 / η ) m ) + O ( η log ( 1 / η ) m ) + 2 η ⋅ 1 4 m ≤ O ( η log ( 1 / η ) m ) . \begin{aligned}
\frac{1}{n}\sum_{i\in T}\xi_i(s)^2 \leq\left|\mathbb{E}_i\xi_i(s)^2-V(\mu,s)\right|+\frac{|S|}{n}\left|\frac{1}{|S|}\sum_{i\in S}\xi_i(s)^2-V(\mu,s)\right|+\frac{|T|}{n}V(\mu,s) \\
&\leq O\left(\frac{\eta\log(1/\eta)}{m}\right)+O\left(\frac{\eta\log(1/\eta)}{m}\right)+2\eta\cdot\frac{1}{4m} \\
&\leq O\left(\frac{\eta\log(1/\eta)}{m}\right).
\end{aligned} n 1 i ∈ T ∑ ξ i ( s ) 2 ≤ E i ξ i ( s ) 2 − V ( μ , s ) + n ∣ S ∣ ∣ S ∣ 1 i ∈ S ∑ ξ i ( s ) 2 − V ( μ , s ) + n ∣ T ∣ V ( μ , s ) ≤ O ( m η log ( 1/ η ) ) + O ( m η log ( 1/ η ) ) + 2 η ⋅ 4 m 1 ≤ O ( m η log ( 1/ η ) ) . Here we used V ( μ , s ) ≤ 1 / ( 4 m ) V(\mu,s)\leq1/(4m) V ( μ , s ) ≤ 1/ ( 4 m ) Definition 3.2 , and hence the samples are η \eta η
Sum-of-Squares Based Robust Algorithm
Let
B i ∗ = ( U i , 1 ∗ , … , U i , m ∗ ) , U i , 1 ∗ , … , U i , m ∗ ∼ i i d μ , y i ∗ : = 1 m ∑ j = 1 m e U i , j ∗ . B_i^*=(U_{i,1}^*,\ldots,U_{i,m}^*),\qquad U_{i,1}^*,\ldots,U_{i,m}^*\overset{\mathrm{iid}}{\sim}\mu,\qquad y_i^*:=\frac{1}{m}\sum_{j=1}^{m}e_{U_{i,j}^*}. B i ∗ = ( U i , 1 ∗ , … , U i , m ∗ ) , U i , 1 ∗ , … , U i , m ∗ ∼ iid μ , y i ∗ := m 1 j = 1 ∑ m e U i , j ∗ . for each i ∈ [ n ] i\in[n] i ∈ [ n ] { y i } i = 1 n \{y_i\}_{i=1}^n { y i } i = 1 n η \eta η { y i ′ } i = 1 n \{y_i'\}_{i=1}^n { y i ′ } i = 1 n
Σ ^ ′ : = E i ( y i ′ − μ ′ ) ( y i ′ − μ ′ ) T , Σ ( v ) : = 1 m ( diag ( v ) − v v T ) , τ c o v : = C η log ( 1 / η ) m , \widehat{\Sigma}':=\mathbb{E}_i(y_i'-\mu')(y_i'-\mu')^{\mathsf{T}},\qquad\Sigma(v):=\frac{1}{m}\left(\operatorname{diag}(v)-vv^{\mathsf{T}}\right),\qquad\tau_{\mathrm{cov}}:=C\frac{\eta\log(1/\eta)}{m}, Σ ′ := E i ( y i ′ − μ ′ ) ( y i ′ − μ ′ ) T , Σ ( v ) := m 1 ( diag ( v ) − v v T ) , τ cov := C m η log ( 1/ η ) , for a sufficiently large universal constant C C C λ ∈ R k \lambda\in\mathbb{R}^k λ ∈ R k M ∈ R k × k M\in\mathbb{R}^{k\times k} M ∈ R k × k
A η = { selector/data consistency: ∀ i ∈ [ n ] : w i 2 = w i , w i y i ′ = w i y i , ∑ i = 1 n w i ≥ ( 1 − η ) n , simplex constraints: ∀ i , j ∈ [ n ] × [ k ] : y i , j ′ ≥ 0 , ∀ i ∈ [ n ] : ∑ j ∈ [ k ] y i , j ′ = 1 , ∀ j ∈ [ k ] : μ j ′ ≥ 0 , ∑ j ∈ [ k ] μ j ′ = 1 , covariance-majorizer constraints: Σ ^ ′ + M M T = Σ ( μ ′ ) + Diag ( λ ) , ∑ j ∈ [ k ] λ j ≤ τ c o v , where μ ′ = E i y i ′ . \mathcal{A}_{\eta}=\left\{
\begin{aligned}
&\textbf{selector/data consistency:} &&\forall i\in[n]:\ w_i^2=w_i,\quad w_i y_i'=w_i y_i,\quad\sum_{i=1}^{n}w_i\geq(1-\eta)n,\\
&\textbf{simplex constraints:} &&\forall i,j\in[n]\times[k]:\ y_{i,j}'\geq0,\quad\forall i\in[n]:\ \sum_{j\in[k]}y_{i,j}'=1,\\
&&&\forall j\in[k]:\ \mu_j'\geq0,\quad\sum_{j\in[k]}\mu_j'=1,\\
&\textbf{covariance-majorizer constraints:} &&\widehat{\Sigma}'+MM^{\mathsf{T}}=\Sigma(\mu')+\operatorname{Diag}(\lambda),\quad\sum_{j\in[k]}\lambda_j\leq\tau_{\mathrm{cov}},\\
&&&\text{where }\mu'=\mathbb{E}_i y_i'.
\end{aligned}
\right. A η = ⎩ ⎨ ⎧ selector/data consistency: simplex constraints: covariance-majorizer constraints: ∀ i ∈ [ n ] : w i 2 = w i , w i y i ′ = w i y i , i = 1 ∑ n w i ≥ ( 1 − η ) n , ∀ i , j ∈ [ n ] × [ k ] : y i , j ′ ≥ 0 , ∀ i ∈ [ n ] : j ∈ [ k ] ∑ y i , j ′ = 1 , ∀ j ∈ [ k ] : μ j ′ ≥ 0 , j ∈ [ k ] ∑ μ j ′ = 1 , Σ ′ + M M T = Σ ( μ ′ ) + Diag ( λ ) , j ∈ [ k ] ∑ λ j ≤ τ cov , where μ ′ = E i y i ′ . Selector/data consistency determines which observed batch averages are kept and ties the repaired points to the observed data on those indices. Simplex constraints keep both the repaired points and the pseudo-mean inside the simplex. To replace all subset-direction variance checks succinctly, the covariance-majorizer block uses a diagonal slack that upper-bounds the difference between the empirical and multinomial covariance simultaneously in every subset direction.
On the η \eta η r i = 1 [ y i = y i ⋆ ] r_i=\mathbb{1}[y_i=y_i^\star] r i = 1 [ y i = y i ⋆ ] μ ˉ = E i y i ⋆ \bar{\mu}=\mathbb{E}_i y_i^\star μ ˉ = E i y i ⋆ w i = r i w_i=r_i w i = r i y i ′ = y i ⋆ y_i'=y_i^\star y i ′ = y i ⋆ μ ′ = μ ˉ \mu'=\bar{\mu} μ ′ = μ ˉ Δ c o v ⋆ \Delta^\star_{\mathrm{cov}} Δ cov ⋆
Lemma 3.4 (Clean Covariance Gap Has Small Subset Discrepancy). Let y 1 ⋆ , … , y n ⋆ y_1^\star,\ldots,y_n^\star y 1 ⋆ , … , y n ⋆ η \eta η Definition 3.2 . Let
μ ˉ : = E i y i ⋆ , Δ c o v ⋆ : = E i ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) T − 1 m ( diag ( μ ˉ ) − μ ˉ μ ˉ T ) . \bar{\mu}:=\mathbb{E}_i y_i^\star,\qquad\Delta^\star_{\mathrm{cov}}:=\mathbb{E}_i\left(y_i^\star-\bar{\mu}\right)\left(y_i^\star-\bar{\mu}\right)^{\mathsf{T}}-\frac{1}{m}\left(\operatorname{diag}(\bar{\mu})-\bar{\mu}\bar{\mu}^{\mathsf{T}}\right). μ ˉ := E i y i ⋆ , Δ cov ⋆ := E i ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) T − m 1 ( diag ( μ ˉ ) − μ ˉ μ ˉ T ) . Then for every s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k
∣ s T Δ c o v ⋆ s ∣ ≤ O ( η log ( 1 / η ) m ) . \left|s^{\mathsf{T}}\Delta^\star_{\mathrm{cov}}s\right|\le O\left(\frac{\eta\log(1/\eta)}{m}\right). s T Δ cov ⋆ s ≤ O ( m η log ( 1/ η ) ) . Proof. Let h : = μ ˉ − μ h:=\bar{\mu}-\mu h := μ ˉ − μ s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k ξ i ( s ) : = ⟨ y i ⋆ − μ , s ⟩ \xi_i(s):=\langle y_i^\star-\mu,s\rangle ξ i ( s ) := ⟨ y i ⋆ − μ , s ⟩
s T Δ c o v ⋆ s = E i ⟨ y i ⋆ − μ ˉ , s ⟩ 2 − V ( μ , s ) = E i ⟨ ξ i ( s ) − ⟨ h , s ⟩ ⟩ 2 − V ( μ ˉ , s ) = E i ξ i ( s ) 2 − ⟨ h , s ⟩ 2 − V ( μ ˉ , s ) = ( E i ξ i ( s ) 2 − V ( μ , s ) ) − ⟨ h , s ⟩ 2 + ( V ( μ , s ) − V ( μ ˉ , s ) ) . \begin{aligned}
s^{\mathsf{T}}\Delta^\star_{\mathrm{cov}}s=\mathbb{E}_i\langle y_i^\star-\bar{\mu},s\rangle^2-V(\mu,s) \\
&=\mathbb{E}_i\langle\xi_i(s)-\langle h,s\rangle\rangle^2-V(\bar{\mu},s) \\
&=\mathbb{E}_i\xi_i(s)^2-\langle h,s\rangle^2-V(\bar{\mu},s) \\
&=\left(\mathbb{E}_i\xi_i(s)^2-V(\mu,s)\right)-\langle h,s\rangle^2+\left(V(\mu,s)-V(\bar{\mu},s)\right).
\end{aligned} s T Δ cov ⋆ s = E i ⟨ y i ⋆ − μ ˉ , s ⟩ 2 − V ( μ , s ) = E i ⟨ ξ i ( s ) − ⟨ h , s ⟩ ⟩ 2 − V ( μ ˉ , s ) = E i ξ i ( s ) 2 − ⟨ h , s ⟩ 2 − V ( μ ˉ , s ) = ( E i ξ i ( s ) 2 − V ( μ , s ) ) − ⟨ h , s ⟩ 2 + ( V ( μ , s ) − V ( μ ˉ , s ) ) . By Property 1 of Definition 3.2 ,
∣ ⟨ h , s ⟩ ∣ = ∣ E i ξ i ( s ) ∣ ≤ O ( η log ( 1 / η ) m ) . |\langle h,s\rangle|=\left|\mathbb{E}_i\xi_i(s)\right|\le O\left(\eta\sqrt{\frac{\log(1/\eta)}{m}}\right). ∣ ⟨ h , s ⟩ ∣ = ∣ E i ξ i ( s ) ∣ ≤ O ( η m log ( 1/ η ) ) . Since x ↦ x ( 1 − x ) x\mapsto x(1-x) x ↦ x ( 1 − x ) [ 0 , 1 ] [0,1] [ 0 , 1 ]
∣ V ( μ , s ) − V ( μ ˉ , s ) ∣ ≤ ∣ ⟨ h , s ⟩ ∣ m ≤ O ( η log ( 1 / η ) m 3 / 2 ) ≤ O ( η log ( 1 / η ) m ) . |V(\mu,s)-V(\bar{\mu},s)|\le\frac{|\langle h,s\rangle|}{m}\le O\left(\frac{\eta\sqrt{\log(1/\eta)}}{m^{3/2}}\right)\le O\left(\frac{\eta\log(1/\eta)}{m}\right). ∣ V ( μ , s ) − V ( μ ˉ , s ) ∣ ≤ m ∣ ⟨ h , s ⟩ ∣ ≤ O ( m 3/2 η log ( 1/ η ) ) ≤ O ( m η log ( 1/ η ) ) . Therefore
∣ s T Δ c o v ⋆ s ∣ ≤ ∣ E i ξ i ( s ) 2 − V ( μ , s ) ∣ + ∣ ⟨ h , s ⟩ ∣ 2 + ∣ V ( μ , s ) − V ( μ ˉ , s ) ∣ ≤ O ( η log ( 1 / η ) m ) + O ( η 2 log ( 1 / η ) m ) + O ( η log ( 1 / η ) m ) ≤ O ( η log ( 1 / η ) m ) . \begin{aligned}
\left|s^{\mathsf{T}}\Delta^\star_{\mathrm{cov}}s\right|\le\left|\mathbb{E}_i\xi_i(s)^2-V(\mu,s)\right|+|\langle h,s\rangle|^2+|V(\mu,s)-V(\bar{\mu},s)| \\
&\le O\left(\frac{\eta\log(1/\eta)}{m}\right)+O\left(\frac{\eta^2\log(1/\eta)}{m}\right)+O\left(\frac{\eta\log(1/\eta)}{m}\right) \\
&\le O\left(\frac{\eta\log(1/\eta)}{m}\right).
\end{aligned} s T Δ cov ⋆ s ≤ E i ξ i ( s ) 2 − V ( μ , s ) + ∣ ⟨ h , s ⟩ ∣ 2 + ∣ V ( μ , s ) − V ( μ ˉ , s ) ∣ ≤ O ( m η log ( 1/ η ) ) + O ( m η 2 log ( 1/ η ) ) + O ( m η log ( 1/ η ) ) ≤ O ( m η log ( 1/ η ) ) . Lemma 3.5 (Clean Covariance Gap Kills the All-Ones Direction). Let y 1 ⋆ , … , y n ⋆ y_1^\star,\ldots,y_n^\star y 1 ⋆ , … , y n ⋆
μ ˉ ≔ E i y i ⋆ , Δ c o v ⋆ ≔ E i ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) ⊤ − 1 m ( diag ( μ ˉ ) − μ ˉ μ ˉ ⊤ ) . \bar{\mu} \coloneqq\mathbb{E}_i y_i^\star,\qquad\Delta^\star_{\mathrm{cov}} \coloneqq\mathbb{E}_i (y_i^\star-\bar{\mu})(y_i^\star-\bar{\mu})^\top-\frac{1}{m}\left(\operatorname{diag}(\bar{\mu})-\bar{\mu}\bar{\mu}^{\top}\right). μ ˉ : = E i y i ⋆ , Δ cov ⋆ : = E i ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) ⊤ − m 1 ( diag ( μ ˉ ) − μ ˉ μ ˉ ⊤ ) . Then
Δ c o v ⋆ 1 = 0. \Delta^\star_{\mathrm{cov}}\mathbf{1}=0. Δ cov ⋆ 1 = 0. Proof. Since each y i ⋆ y_i^\star y i ⋆ μ ˉ \bar{\mu} μ ˉ
⟨ y i ⋆ − μ ˉ , 1 ⟩ = 0 for all i ∈ [ n ] , \langle y_i^\star-\bar{\mu},\mathbf{1}\rangle=0\qquad\text{for all }i\in[n], ⟨ y i ⋆ − μ ˉ , 1 ⟩ = 0 for all i ∈ [ n ] , and hence
E i ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) ⊤ 1 = 0. \mathbb{E}_i (y_i^\star-\bar{\mu})(y_i^\star-\bar{\mu})^\top\mathbf{1}=0. E i ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) ⊤ 1 = 0. Also,
( diag ( μ ˉ ) − μ ˉ μ ˉ ⊤ ) 1 = μ ˉ − μ ˉ ⟨ μ ˉ , 1 ⟩ = 0. \left(\operatorname{diag}(\bar{\mu})-\bar{\mu}\bar{\mu}^{\top}\right)\mathbf{1}=\bar{\mu}-\bar{\mu}\langle\bar{\mu},\mathbf{1}\rangle=0. ( diag ( μ ˉ ) − μ ˉ μ ˉ ⊤ ) 1 = μ ˉ − μ ˉ ⟨ μ ˉ , 1 ⟩ = 0. Corollary 3.6 (Diagonal-Majorizer Feasibility). Let y 1 ⋆ , … , y n ⋆ y_1^\star,\ldots,y_n^\star y 1 ⋆ , … , y n ⋆ η \eta η Definition 3.2 . Let
μ ˉ ≔ E i y i ⋆ , Δ c o v ⋆ ≔ E i ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) ⊤ − 1 m ( diag ( μ ˉ ) − μ ˉ μ ˉ ⊤ ) . \bar{\mu} \coloneqq\mathbb{E}_i y_i^\star,\qquad\Delta^\star_{\mathrm{cov}} \coloneqq\mathbb{E}_i (y_i^\star-\bar{\mu})(y_i^\star-\bar{\mu})^\top-\frac{1}{m}\left(\operatorname{diag}(\bar{\mu})-\bar{\mu}\bar{\mu}^{\top}\right). μ ˉ : = E i y i ⋆ , Δ cov ⋆ : = E i ( y i ⋆ − μ ˉ ) ( y i ⋆ − μ ˉ ) ⊤ − m 1 ( diag ( μ ˉ ) − μ ˉ μ ˉ ⊤ ) . Then there exists λ ∈ R k \lambda\in\mathbb{R}^k λ ∈ R k
Diag ( λ ) − Δ c o v ⋆ ⪰ 0 , ∑ j ∈ [ k ] λ j ≤ O ( η log ( 1 / η ) m ) . \operatorname{Diag}(\lambda)-\Delta^\star_{\mathrm{cov}}\succeq0,\qquad\sum_{j\in[k]}\lambda_j\le O\left(\frac{\eta\log(1/\eta)}{m}\right). Diag ( λ ) − Δ cov ⋆ ⪰ 0 , j ∈ [ k ] ∑ λ j ≤ O ( m η log ( 1/ η ) ) . Proof. By Proposition 2.17 , it remains only to verify the subset-discrepancy and nullspace hypotheses. Those are exactly Lemmas 3.4 and 3.5 applied to Δ = Δ c o v ⋆ \Delta=\Delta^\star_{\mathrm{cov}} Δ = Δ cov ⋆
This proves feasibility of the covariance-majorizer block for the clean witness. If Δ ′ 1 = 0 \Delta'\mathbf{1}=0 Δ ′ 1 = 0 Diag ( λ ) − Δ ′ ⪰ 0 \operatorname{Diag}(\lambda)-\Delta'\succeq0 Diag ( λ ) − Δ ′ ⪰ 0 s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k σ = 2 s − 1 \sigma=2s-1 σ = 2 s − 1
4 s ⊤ Δ ′ s = σ ⊤ Δ ′ σ ≤ σ ⊤ Diag ( λ ) σ = ∑ j ∈ [ k ] λ j . 4s^\top\Delta's=\sigma^\top\Delta'\sigma\le\sigma^\top\operatorname{Diag}(\lambda)\sigma=\sum_{j\in[k]}\lambda_j. 4 s ⊤ Δ ′ s = σ ⊤ Δ ′ σ ≤ σ ⊤ Diag ( λ ) σ = j ∈ [ k ] ∑ λ j . Thus one matrix majorizer simultaneously controls every subset direction. We next give the SoS version of this calculation for Δ c o v ′ = Σ ^ ′ − Σ ( μ ′ ) \Delta'_{\mathrm{cov}}=\widehat{\Sigma}'-\Sigma(\mu') Δ cov ′ = Σ ′ − Σ ( μ ′ )
Lemma 3.7 (Diagonal Majorizer Implies Program-Side Subset-Direction Variance Control). Under A η \mathcal{A}_\eta A η { ∀ j ∈ [ k ] : s j 2 = s j } \{\forall j\in[k]:s_j^2=s_j\} { ∀ j ∈ [ k ] : s j 2 = s j }
A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ⊢ 6 { s ⊤ ( Σ ^ ′ − Σ ( μ ′ ) ) s ≤ 1 4 ∑ j ∈ [ k ] λ j } . \mathcal{A}_\eta\cup\{\forall j\in[k]:s_j^2=s_j\}\vdash_6\left\{s^\top\left(\widehat{\Sigma}'-\Sigma(\mu')\right)s\le\frac14\sum_{j\in[k]}\lambda_j\right\}. A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ⊢ 6 ⎩ ⎨ ⎧ s ⊤ ( Σ ′ − Σ ( μ ′ ) ) s ≤ 4 1 j ∈ [ k ] ∑ λ j ⎭ ⎬ ⎫ . Consequently,
A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ⊢ 6 { E i ⟨ y i ′ − μ ′ , s ⟩ 2 ≤ ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩ ) m + O ( η log ( 1 / η ) m ) } . \mathcal{A}_\eta\cup\{\forall j\in[k]:s_j^2=s_j\}\vdash_6\left\{\mathbb{E}_i\langle y_i'-\mu',s\rangle^2\le\frac{\langle\mu',s\rangle(1-\langle\mu',s\rangle)}{m}+O\left(\frac{\eta\log(1/\eta)}{m}\right)\right\}. A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ⊢ 6 { E i ⟨ y i ′ − μ ′ , s ⟩ 2 ≤ m ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩) + O ( m η log ( 1/ η ) ) } . Proof. Write
Δ c o v ′ ≔ Σ ^ ′ − Σ ( μ ′ ) , σ ≔ 2 s − 1. \Delta'_{\mathrm{cov}} \coloneqq\widehat{\Sigma}'-\Sigma(\mu'), \qquad\sigma\coloneqq2s-1. Δ cov ′ : = Σ ′ − Σ ( μ ′ ) , σ : = 2 s − 1. We first show that A η ∣ 6 { Δ c o v ′ 1 = 0 } \mathcal{A}_\eta\mid_6 \{\Delta'_{\mathrm{cov}}\mathbf{1}=0\} A η ∣ 6 { Δ cov ′ 1 = 0 } i ∈ [ n ] i\in[n] i ∈ [ n ]
A η ∣ 6 { ⟨ y i ′ − μ ′ , 1 ⟩ = ∑ j ∈ [ k ] y i , j ′ − ∑ j ∈ [ k ] μ j ′ = 0 } . \mathcal{A}_\eta\mid_6 \left\{\langle y'_i-\mu',\mathbf{1}\rangle=\sum_{j\in[k]}y'_{i,j}-\sum_{j\in[k]}\mu'_j=0\right\}. A η ∣ 6 ⎩ ⎨ ⎧ ⟨ y i ′ − μ ′ , 1 ⟩ = j ∈ [ k ] ∑ y i , j ′ − j ∈ [ k ] ∑ μ j ′ = 0 ⎭ ⎬ ⎫ . Therefore
A η ∣ 6 { Σ ^ ′ 1 = E i ⟨ y i ′ − μ ′ ⟩ ⟨ y i ′ − μ ′ , 1 ⟩ = 0 } . \mathcal{A}_\eta\mid_6 \left\{\widehat{\Sigma}'\mathbf{1}=\mathbb{E}_i\langle y'_i-\mu'\rangle\langle y'_i-\mu',\mathbf{1}\rangle=0\right\}. A η ∣ 6 { Σ ′ 1 = E i ⟨ y i ′ − μ ′ ⟩ ⟨ y i ′ − μ ′ , 1 ⟩ = 0 } . Also,
A η ∣ 6 { Σ ( μ ′ ) 1 = 1 m ( μ ′ − μ ′ ⟨ μ ′ , 1 ⟩ ) = 0 } . \mathcal{A}_\eta\mid_6 \left\{\Sigma(\mu')\mathbf{1}=\frac{1}{m}\left(\mu'-\mu'\langle\mu',\mathbf{1}\rangle\right)=0\right\}. A η ∣ 6 { Σ ( μ ′ ) 1 = m 1 ( μ ′ − μ ′ ⟨ μ ′ , 1 ⟩ ) = 0 } . and hence Δ c o v ′ 1 = 0 \Delta'_{\mathrm{cov}}\mathbf{1}=0 Δ cov ′ 1 = 0
Using this identity, we have
A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { σ T Δ c o v ′ σ = ( 2 s − 1 ) T Δ c o v ′ ( 2 s − 1 ) = 4 s T Δ c o v ′ s } . \mathcal{A}_\eta\cup\left\{\forall j\in[k]:s_j^2=s_j\right\}\mid_6\left\{\sigma^{\mathsf T}\Delta'_{\mathrm{cov}}\sigma=(2s-1)^{\mathsf T}\Delta'_{\mathrm{cov}}(2s-1)=4s^{\mathsf T}\Delta'_{\mathrm{cov}}s\right\}. A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { σ T Δ cov ′ σ = ( 2 s − 1 ) T Δ cov ′ ( 2 s − 1 ) = 4 s T Δ cov ′ s } . Now the diagonal-majorizer constraint in A η \mathcal{A}_\eta A η
A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { σ T Δ c o v ′ σ ≤ σ T Diag ( λ ) σ } . \mathcal{A}_\eta\cup\left\{\forall j\in[k]:s_j^2=s_j\right\}\mid_6\left\{\sigma^{\mathsf T}\Delta'_{\mathrm{cov}}\sigma\leq\sigma^{\mathsf T}\operatorname{Diag}(\lambda)\sigma\right\}. A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { σ T Δ cov ′ σ ≤ σ T Diag ( λ ) σ } . since
σ T ( Diag ( λ ) − Δ c o v ′ ) σ = ∥ M T σ ∥ 2 2 ≥ 0. \sigma^{\mathsf T}\left(\operatorname{Diag}(\lambda)-\Delta'_{\mathrm{cov}}\right)\sigma=\lVert M^{\mathsf T}\sigma\rVert_2^2\geq0. σ T ( Diag ( λ ) − Δ cov ′ ) σ = ∥ M T σ ∥ 2 2 ≥ 0. Under the Boolean constraints on s s s
{ ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { σ j 2 = ( 2 s j − 1 ) 2 = 1 } \left\{\forall j\in[k]:s_j^2=s_j\right\}\mid_6\left\{\sigma_j^2=(2s_j-1)^2=1\right\} { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { σ j 2 = ( 2 s j − 1 ) 2 = 1 } for every j ∈ [ k ] j\in[k] j ∈ [ k ]
A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { σ T Diag ( λ ) σ = ∑ j ∈ [ k ] λ j } . \mathcal{A}_\eta\cup\left\{\forall j\in[k]:s_j^2=s_j\right\}\mid_6\left\{\sigma^{\mathsf T}\operatorname{Diag}(\lambda)\sigma=\sum_{j\in[k]}\lambda_j\right\}. A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 ⎩ ⎨ ⎧ σ T Diag ( λ ) σ = j ∈ [ k ] ∑ λ j ⎭ ⎬ ⎫ . Substituting σ T Δ c o v ′ σ = 4 s T Δ c o v ′ s \sigma^{\mathsf T}\Delta'_{\mathrm{cov}}\sigma=4s^{\mathsf T}\Delta'_{\mathrm{cov}}s σ T Δ cov ′ σ = 4 s T Δ cov ′ s σ T Diag ( λ ) σ = ∑ j λ j \sigma^{\mathsf T}\operatorname{Diag}(\lambda)\sigma=\sum_j\lambda_j σ T Diag ( λ ) σ = ∑ j λ j
For the variance bound, observe that
{ ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { s T Σ ^ ′ s = E i ⟨ y i ′ − μ ′ , s ⟩ 2 } \left\{\forall j\in[k]:s_j^2=s_j\right\}\mid_6\left\{s^{\mathsf T}\widehat{\Sigma}'s=\mathbb{E}_i\langle y'_i-\mu',s\rangle^2\right\} { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { s T Σ ′ s = E i ⟨ y i ′ − μ ′ , s ⟩ 2 } and
A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { s T Σ ( μ ′ ) s = ⟨ μ ′ , s ⟩ − ⟨ μ ′ , s ⟩ 2 m = ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩ ) m } . \mathcal{A}_\eta\cup\left\{\forall j\in[k]:s_j^2=s_j\right\}\mid_6\left\{s^{\mathsf T}\Sigma(\mu')s=\frac{\langle\mu',s\rangle-\langle\mu',s\rangle^2}{m}=\frac{\langle\mu',s\rangle(1-\langle\mu',s\rangle)}{m}\right\}. A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { s T Σ ( μ ′ ) s = m ⟨ μ ′ , s ⟩ − ⟨ μ ′ , s ⟩ 2 = m ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩) } . Using the bound s T Δ c o v ′ s ≤ 1 4 ∑ j λ j s^{\mathsf T}\Delta'_{\mathrm{cov}}s\leq\frac{1}{4}\sum_j\lambda_j s T Δ cov ′ s ≤ 4 1 ∑ j λ j ∑ j λ j ≤ τ c o v \sum_j\lambda_j\leq\tau_{\mathrm{cov}} ∑ j λ j ≤ τ cov
A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { E i ⟨ y i ′ − μ ′ , s ⟩ 2 ≤ ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩ ) m + 1 4 τ c o v } . \mathcal{A}_\eta\cup\left\{\forall j\in[k]:s_j^2=s_j\right\}\mid_6\left\{\mathbb{E}_i\langle y'_i-\mu',s\rangle^2\leq\frac{\langle\mu',s\rangle(1-\langle\mu',s\rangle)}{m}+\frac{1}{4}\tau_{\mathrm{cov}}\right\}. A η ∪ { ∀ j ∈ [ k ] : s j 2 = s j } ∣ 6 { E i ⟨ y i ′ − μ ′ , s ⟩ 2 ≤ m ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩) + 4 1 τ cov } . which is the desired bound.
Algorithm 3.1 (Robust Estimation from Discrete Batches).
Input: An η \eta η { B i } i ∈ [ n ] \{B_i\}_{i\in[n]} { B i } i ∈ [ n ] m m m
Output: An estimate μ ^ \hat{\mu} μ ^ μ \mu μ
Operations:
Convert each observed batch to its average vector
y i ≔ 1 m ∑ j = 1 m e U i , j ∈ Δ k − 1 for all i ∈ [ n ] . y_i \coloneqq\frac{1}{m}\sum_{j=1}^{m} e_{U_{i,j}} \in\Delta^{k-1} \qquad\text{for all } i\in[n]. y i : = m 1 j = 1 ∑ m e U i , j ∈ Δ k − 1 for all i ∈ [ n ] . Form the SoS constraint system A η \mathcal{A}_{\eta} A η § 3.1 , with variables ( w i ) i ∈ [ n ] (w_i)_{i\in[n]} ( w i ) i ∈ [ n ] ( y i ′ ) i ∈ [ n ] (y'_i)_{i\in[n]} ( y i ′ ) i ∈ [ n ] μ ′ \mu' μ ′ λ \lambda λ M M M
Find a degree-6 pseudoexpectation E ~ \widetilde{\mathbb{E}} E A η \mathcal{A}_{\eta} A η
Output an ℓ 1 \ell_1 ℓ 1 E ~ μ ′ \widetilde{\mathbb{E}}\mu' E μ ′ Δ k − 1 \Delta^{k-1} Δ k − 1 μ ^ = E ~ μ ′ \hat{\mu}=\widetilde{\mathbb{E}}\mu' μ ^ = E μ ′
We can now state our main technical lemma.
Lemma 3.8 (Main SoS Lemma). Suppose Y ∗ = { y i ∗ } i ∈ [ n ] \mathcal{Y}^*=\{y_i^*\}_{i\in[n]} Y ∗ = { y i ∗ } i ∈ [ n ] η \eta η Definition 3.2 ). Then for any fixed binary vector s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k
A η ⊢ 6 { ⟨ μ ′ − μ , s ⟩ 2 ≤ O ( ( η log ( 1 / η ) m ) 2 ) } . \mathcal{A}_{\eta}\vdash_6 \left\{\langle\mu'-\mu,s\rangle^2\leq O\left(\left(\eta\sqrt{\frac{\log(1/\eta)}{m}}\right)^2\right)\right\}. A η ⊢ 6 ⎩ ⎨ ⎧ ⟨ μ ′ − μ , s ⟩ 2 ≤ O ( η m log ( 1/ η ) ) 2 ⎭ ⎬ ⎫ . Algorithm 3.1 summarizes the robust estimator promised by Theorem 3.1 . We use exact pseudoexpectation notation in the analysis. For computation, we may restrict the SDP to a ball of radius poly ( n , k ) \operatorname{poly}(n,k) poly ( n , k ) w i w_i w i y i , j ′ y'_{i,j} y i , j ′ μ j ′ ∈ [ 0 , 1 ] \mu'_j\in[0,1] μ j ′ ∈ [ 0 , 1 ] ( Δ c o v ) j j ≥ − 1 / ( 4 m ) (\Delta_{\mathrm{cov}})_{jj}\geq-1/(4m) ( Δ cov ) j j ≥ − 1/ ( 4 m )
− 1 4 m ≤ λ j ≤ τ c o v + k − 1 4 m , ∥ M ∥ F 2 = ∑ j λ j − tr ( Δ c o v ∗ ) ≤ τ c o v + k 4 m . -\frac{1}{4m}\leq\lambda_j\leq\tau_{\mathrm{cov}}+\frac{k-1}{4m}, \qquad\lVert M\rVert_F^2=\sum_j\lambda_j-\operatorname{tr}(\Delta_{\mathrm{cov}}^*)\leq\tau_{\mathrm{cov}}+\frac{k}{4m}. − 4 m 1 ≤ λ j ≤ τ cov + 4 m k − 1 , ∥ M ∥ F 2 = j ∑ λ j − tr ( Δ cov ∗ ) ≤ τ cov + 4 m k . Thus the point-evaluation functional at this witness has polynomially bounded degree-6 moments. In the standard bit model, solve the bounded SDP to an inverse-polynomial feasibility tolerance τ \tau τ
τ poly ( n , k , m ) = o ( η 2 log ( 1 / η ) m ) . \tau\operatorname{poly}(n,k,m)=o\left(\frac{\eta^2\log(1/\eta)}{m}\right). τ poly ( n , k , m ) = o ( m η 2 log ( 1/ η ) ) . Since the degree-6 certificates in the proof of Lemma 3.8 have coefficient norms bounded by poly ( n , k , m ) \operatorname{poly}(n,k,m) poly ( n , k , m ) τ poly ( n , k , m ) \tau\operatorname{poly}(n,k,m) τ poly ( n , k , m ) ℓ 1 \ell_1 ℓ 1 Theorem 3.1 and polynomial running time, as in Lemma 4.11 .
We now derive Theorem 3.1 from Lemma 3.8.
Proof of Theorem 3.1 . Apply Corollary 3.3 with failure probability min { β , 1 / 4 } \min\{\beta,1/4\} min { β , 1/4 } 1 − β 1-\beta 1 − β Definition 3.2 . Conditioned on this event, Lemma 3.8 applies. Let E ~ \widetilde{\mathbb{E}} E A η \mathcal{A}_\eta A η μ ^ = E ~ μ ′ \hat{\mu}=\widetilde{\mathbb{E}}\mu' μ ^ = E μ ′ s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k s i = 1 [ μ ^ i > μ i ] s_i=\mathbb{1}[\hat{\mu}_i>\mu_i] s i = 1 [ μ ^ i > μ i ]
∥ μ ^ − μ ∥ T V 2 = ⟨ μ ^ − μ , s ⟩ 2 = ⟨ E ~ μ ′ − μ , s ⟩ 2 ≤ E ~ ⟨ μ ′ − μ , s ⟩ 2 ≤ O ( ( η log ( 1 / η ) m ) 2 ) . \begin{aligned}
\lVert\hat{\mu}-\mu\rVert_{\mathrm{TV}}^2
&=\langle\hat{\mu}-\mu,s\rangle^2\\
&=\langle\widetilde{\mathbb{E}}\mu'-\mu,s\rangle^2\\
&\le\widetilde{\mathbb{E}}\langle\mu'-\mu,s\rangle^2\\
&\le O\left(\left(\eta\sqrt{\frac{\log(1/\eta)}{m}}\right)^2\right).
\end{aligned} ∥ μ ^ − μ ∥ TV 2 = ⟨ μ ^ − μ , s ⟩ 2 = ⟨ E μ ′ − μ , s ⟩ 2 ≤ E ⟨ μ ′ − μ , s ⟩ 2 ≤ O ( η m log ( 1/ η ) ) 2 . Therefore,
∥ μ ^ − μ ∥ T V ≤ O ( η log ( 1 / η ) m ) , \lVert\hat{\mu}-\mu\rVert_{\mathrm{TV}}\le O\left(\eta\sqrt{\frac{\log(1/\eta)}{m}}\right), ∥ μ ^ − μ ∥ TV ≤ O ( η m log ( 1/ η ) ) , as desired. □
Proof of the Main SoS Lemma
Proof. We proceed in three steps: first control the clean-kept and corrupted/discarded contributions in a fixed direction; then use the diagonal-majorizer certificate to upper bound the repaired second moment; finally control and absorb the remaining variance difference using the covariance formula V ( μ , s ) = ⟨ μ , s ⟩ ( 1 − ⟨ μ , s ⟩ ) / m V(\mu,s)=\langle\mu,s\rangle(1-\langle\mu,s\rangle)/m V ( μ , s ) = ⟨ μ , s ⟩ ( 1 − ⟨ μ , s ⟩) / m
Let r i r_i r i r i = 1 [ y i ∗ = y i ] r_i=\mathbb{1}[y_i^*=y_i] r i = 1 [ y i ∗ = y i ] s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k
Δ s : = ⟨ μ ′ − μ , s ⟩ , ξ i : = ⟨ y i ∗ − μ , s ⟩ , ξ i ′ : = ⟨ y i ′ − μ , s ⟩ . \Delta_s:=\langle\mu'-\mu,s\rangle,\qquad\xi_i:=\langle y_i^*-\mu,s\rangle,\qquad\xi_i':=\langle y_i'-\mu,s\rangle. Δ s := ⟨ μ ′ − μ , s ⟩ , ξ i := ⟨ y i ∗ − μ , s ⟩ , ξ i ′ := ⟨ y i ′ − μ , s ⟩ . Multiplying by r i w i r_iw_i r i w i r i ( 1 − w i ) r_i(1-w_i) r i ( 1 − w i ) 1 − r i w i 1-r_iw_i 1 − r i w i
Split into the clean-kept part and its complement. Applying SoS triangle inequality we can write
A η ⊢ Δ s 2 ≤ 2 ( E i r i w i ξ i ′ ) 2 ⏟ clean-kept contribution + 2 ( E i ( 1 − r i w i ) ξ i ′ ) 2 ⏟ corrupted-or-discarded contribution . (6) \mathcal{A}_\eta\vdash\Delta_s^2\le2\underbrace{\left(\mathbb{E}_i r_iw_i\xi_i'\right)^2}_{\text{clean-kept contribution}}+2\underbrace{\left(\mathbb{E}_i(1-r_iw_i)\xi_i'\right)^2}_{\text{corrupted-or-discarded contribution}}. \tag*{(6)} A η ⊢ Δ s 2 ≤ 2 clean-kept contribution ( E i r i w i ξ i ′ ) 2 + 2 corrupted-or-discarded contribution ( E i ( 1 − r i w i ) ξ i ′ ) 2 . (6) Clean-kept contribution. Observing that r i w i y i ∗ = r i w i y i ′ r_iw_iy_i^*=r_iw_iy_i' r i w i y i ∗ = r i w i y i ′
A η ⊢ { ( E i r i w i ξ i ′ ) 2 = ( E i r i w i ξ i ) 2 ≤ 2 ( E i r i ξ i ) 2 + 2 ( E i r i ( 1 − w i ) ξ i ) 2 . \mathcal{A}_\eta\vdash
\left\{
\begin{aligned}
\left(\mathbb{E}_i r_iw_i\xi_i'\right)^2
&=\left(\mathbb{E}_i r_iw_i\xi_i\right)^2\\
&\le2\left(\mathbb{E}_i r_i\xi_i\right)^2+2\left(\mathbb{E}_i r_i(1-w_i)\xi_i\right)^2.
\end{aligned}
\right. A η ⊢ { ( E i r i w i ξ i ′ ) 2 = ( E i r i w i ξ i ) 2 ≤ 2 ( E i r i ξ i ) 2 + 2 ( E i r i ( 1 − w i ) ξ i ) 2 . Now
( E i r i ξ i ) 2 = ( E i ξ i − E i ( 1 − r i ) ξ i ) 2 ≤ 2 ( E i ξ i ) 2 + 2 ( E i ( 1 − r i ) ξ i ) 2 ≤ O ( η 2 log ( 1 / η ) m ) + 2 E i ( 1 − r i ) ⋅ E i ( 1 − r i ) ξ i 2 ≤ O ( η 2 log ( 1 / η ) m ) . \begin{aligned}
\left(\mathbb{E}_{i} r_i\xi_i\right)^2
&= \left(\mathbb{E}_{i}\xi_i-\mathbb{E}_{i}(1-r_i)\xi_i\right)^2 \\
&\leq2\left(\mathbb{E}_{i}\xi_i\right)^2+2\left(\mathbb{E}_{i}(1-r_i)\xi_i\right)^2 \\
&\leq O\left(\frac{\eta^2\log(1/\eta)}{m}\right)+2\mathbb{E}_{i}(1-r_i)\cdot\mathbb{E}_{i}(1-r_i)\xi_i^2 \\
&\leq O\left(\frac{\eta^2\log(1/\eta)}{m}\right).
\end{aligned} ( E i r i ξ i ) 2 = ( E i ξ i − E i ( 1 − r i ) ξ i ) 2 ≤ 2 ( E i ξ i ) 2 + 2 ( E i ( 1 − r i ) ξ i ) 2 ≤ O ( m η 2 log ( 1/ η ) ) + 2 E i ( 1 − r i ) ⋅ E i ( 1 − r i ) ξ i 2 ≤ O ( m η 2 log ( 1/ η ) ) . Here we used Property 1 of Definition 3.2 in the third line and Property 3 in the last line, since the corrupted indices form a subset of size at most m η m\eta m η
A η ∣ 6 ( E i r i w i ξ i ′ ) 2 ≤ O ( η 2 log ( 1 / η ) m ) + 2 ( E i r i ( 1 − w i ) ξ i ) 2 ⏟ discarded-clean contribution . (7) \mathcal{A}_{\eta}\mid6\left(\mathbb{E}_{i}r_iw_i\xi_i'\right)^2
\leq O\left(\frac{\eta^2\log(1/\eta)}{m}\right)
+2\underbrace{\left(\mathbb{E}_{i}r_i(1-w_i)\xi_i\right)^2}_{\text{discarded-clean contribution}}. \tag*{(7)} A η ∣ 6 ( E i r i w i ξ i ′ ) 2 ≤ O ( m η 2 log ( 1/ η ) ) + 2 discarded-clean contribution ( E i r i ( 1 − w i ) ξ i ) 2 . (7) We next bound the discarded-clean contribution. We have
A η ∣ 6 { ( E i r i ( 1 − w i ) ξ i ) 2 ≤ ( E i r i ( 1 − w i ) ) ⋅ ( E i r i ( 1 − w i ) ξ i 2 ) ≤ η ⋅ ( E i ( 1 − w i ) ξ i 2 ) } . \mathcal{A}_{\eta}\mid6\left\{
\left(\mathbb{E}_{i}r_i(1-w_i)\xi_i\right)^2
\leq
\left(\mathbb{E}_{i}r_i(1-w_i)\right)\cdot
\left(\mathbb{E}_{i}r_i(1-w_i)\xi_i^2\right)
\leq
\eta\cdot\left(\mathbb{E}_{i}(1-w_i)\xi_i^2\right)
\right\}. A η ∣ 6 { ( E i r i ( 1 − w i ) ξ i ) 2 ≤ ( E i r i ( 1 − w i ) ) ⋅ ( E i r i ( 1 − w i ) ξ i 2 ) ≤ η ⋅ ( E i ( 1 − w i ) ξ i 2 ) } . Property 3 of Definition 3.2 together with Lemma 2.9 implies
A η ∣ 6 { ( E i r i ( 1 − w i ) ξ i ) 2 ≤ η ⋅ O ( η log ( 1 / η ) / m ) = O ( η 2 log ( 1 / η ) / m ) } . \mathcal{A}_{\eta}\mid6\left\{
\left(\mathbb{E}_{i}r_i(1-w_i)\xi_i\right)^2
\leq\eta\cdot O\left(\eta\log(1/\eta)/m\right)
=O\left(\eta^2\log(1/\eta)/m\right)
\right\}. A η ∣ 6 { ( E i r i ( 1 − w i ) ξ i ) 2 ≤ η ⋅ O ( η log ( 1/ η ) / m ) = O ( η 2 log ( 1/ η ) / m ) } . Therefore,
A η ∣ 6 { ( E i r i w i ξ i ′ ) 2 ≤ O ( η 2 log ( 1 / η ) m ) } . \mathcal{A}_{\eta}\mid6\left\{
\left(\mathbb{E}_{i}r_iw_i\xi_i'\right)^2
\leq O\left(\frac{\eta^2\log(1/\eta)}{m}\right)
\right\}. A η ∣ 6 { ( E i r i w i ξ i ′ ) 2 ≤ O ( m η 2 log ( 1/ η ) ) } . Corrupted-or-discarded contribution. We can write
A η ⊢ { ( E i ( 1 − r i w i ) ξ i ′ ) 2 ≤ ( E i ( 1 − r i w i ) ) ( E i ( 1 − r i w i ) ξ i ′ 2 ) ≤ 2 η ( E i ( 1 − r i w i ) ξ i ′ 2 ) = 2 η ( E i ξ i ′ 2 ⏟ repaired second moment − E i r i w i ξ i ′ 2 ⏟ clean-kept second moment ) (8) \mathcal{A}_{\eta} \vdash\left\{
\begin{aligned}
\left(\mathbb{E}_{i}(1-r_iw_i)\xi_i'\right)^2 &\le\left(\mathbb{E}_{i}(1-r_iw_i)\right)\left(\mathbb{E}_{i}(1-r_iw_i)\xi_i'^2\right) \\
&\le2\eta\left(\mathbb{E}_{i}(1-r_iw_i)\xi_i'^2\right) \\
&=2\eta\left(\underbrace{\mathbb{E}_{i}\xi_i'^2}_{\text{repaired second moment}}-\underbrace{\mathbb{E}_{i}r_iw_i\xi_i'^2}_{\text{clean-kept second moment}}\right)
\end{aligned}
\right. \tag*{(8)} A η ⊢ ⎩ ⎨ ⎧ ( E i ( 1 − r i w i ) ξ i ′ ) 2 ≤ ( E i ( 1 − r i w i ) ) ( E i ( 1 − r i w i ) ξ i ′2 ) ≤ 2 η ( E i ( 1 − r i w i ) ξ i ′2 ) = 2 η repaired second moment E i ξ i ′2 − clean-kept second moment E i r i w i ξ i ′2 (8) For completeness, the weighted Cauchy–Schwarz step in (8) is the degree-6 bottleneck. With b i : = 1 − r i w i b_i:=1-r_iw_i b i := 1 − r i w i z i : = ξ i ′ z_i:=\xi_i' z i := ξ i ′ b i 2 = b i b_i^2=b_i b i 2 = b i
A η ⊢ { ( E i b i ) ( E i b i z i 2 ) − ( E i b i z i ) 2 = 1 2 n 2 ∑ i , j b i b j ( z i − z j ) 2 = 1 2 n 2 ∑ i , j ( b i b j ( z i − z j ) ) 2 ≥ 0 \mathcal{A}_{\eta}\vdash\left\{
\begin{aligned}
\left(\mathbb{E}_{i}b_i\right)\left(\mathbb{E}_{i}b_iz_i^2\right)-\left(\mathbb{E}_{i}b_iz_i\right)^2
&=\frac{1}{2n^2}\sum_{i,j}b_ib_j(z_i-z_j)^2 \\
&=\frac{1}{2n^2}\sum_{i,j}\left(b_ib_j(z_i-z_j)\right)^2 \\
&\ge0
\end{aligned}
\right. A η ⊢ ⎩ ⎨ ⎧ ( E i b i ) ( E i b i z i 2 ) − ( E i b i z i ) 2 = 2 n 2 1 i , j ∑ b i b j ( z i − z j ) 2 = 2 n 2 1 i , j ∑ ( b i b j ( z i − z j ) ) 2 ≥ 0 Since the polynomials being squared have degree 3, this certificate has degree 6. We first bound the repaired second moment. Since
E i ξ i ′ 2 = E i [ y i ′ − μ ′ , s ) 2 + Δ s 2 , \mathbb{E}_{i}\xi_i'^2=\mathbb{E}_{i}[y_i'-\mu',s)^2+\Delta_s^2, E i ξ i ′2 = E i [ y i ′ − μ ′ , s ) 2 + Δ s 2 , Lemma 3.7 with subset direction s s s
A η ⊢ { E i ξ i ′ 2 ≤ ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩ ) m + O ( η log ( 1 / η ) m ) + Δ s 2 = V ( μ ′ , s ) + O ( η log ( 1 / η ) m ) + Δ s 2 \mathcal{A}_{\eta}\vdash\left\{
\begin{aligned}
\mathbb{E}_{i}\xi_i'^2 &\le\frac{\langle\mu',s\rangle(1-\langle\mu',s\rangle)}{m}+O\left(\frac{\eta\log(1/\eta)}{m}\right)+\Delta_s^2 \\
&=V(\mu',s)+O\left(\frac{\eta\log(1/\eta)}{m}\right)+\Delta_s^2
\end{aligned}
\right. A η ⊢ ⎩ ⎨ ⎧ E i ξ i ′2 ≤ m ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩) + O ( m η log ( 1/ η ) ) + Δ s 2 = V ( μ ′ , s ) + O ( m η log ( 1/ η ) ) + Δ s 2 Next we lower bound the clean-kept second moment. Since r i w i ξ i ′ 2 = r i w i ξ i 2 r_iw_i\xi_i'^2=r_iw_i\xi_i^2 r i w i ξ i ′2 = r i w i ξ i 2
A η ⊢ { E i r i w i ξ i ′ 2 = E i r i w i ξ i 2 = E i ξ i 2 − E i ( 1 − r i w i ) ξ i ′ 2 ≥ V ( μ , s ) − O ( η log ( 1 / η ) / m ) − E i ( 1 − r i w i ) ξ i ′ 2 ⏟ missing clean second moment (9) \mathcal{A}_{\eta}\vdash\left\{
\begin{aligned}
\mathbb{E}_{i}r_iw_i\xi_i'^2&=\mathbb{E}_{i}r_iw_i\xi_i^2 \\
&=\mathbb{E}_{i}\xi_i^2-\mathbb{E}_{i}(1-r_iw_i)\xi_i'^2 \\
&\ge V(\mu,s)-O(\eta\log(1/\eta)/m)-\underbrace{\mathbb{E}_{i}(1-r_iw_i)\xi_i'^2}_{\text{missing clean second moment}}
\end{aligned}
\right. \tag*{(9)} A η ⊢ ⎩ ⎨ ⎧ E i r i w i ξ i ′2 = E i r i w i ξ i 2 = E i ξ i 2 − E i ( 1 − r i w i ) ξ i ′2 ≥ V ( μ , s ) − O ( η log ( 1/ η ) / m ) − missing clean second moment E i ( 1 − r i w i ) ξ i ′2 (9) Here the first inequality follows from Property 2 of Definition 3.2 , and the remaining error comes from the missing clean second moment. Since
∑ i ( 1 − r i w i ) ≤ ∑ i ( 1 − r i ) + ∑ i r i ( 1 − w i ) ≤ 2 η n , \sum_i(1-r_iw_i)\le\sum_i(1-r_i)+\sum_i r_i(1-w_i)\le2\eta n, i ∑ ( 1 − r i w i ) ≤ i ∑ ( 1 − r i ) + i ∑ r i ( 1 − w i ) ≤ 2 η n , Property 3 of Definition 3.2 together with Lemma 2.9 gives
A η ⊢ E i ( 1 − r i w i ) ξ i ′ 2 ≤ O ( η log ( 1 / η ) / m ) . \mathcal{A}_{\eta}\vdash\mathbb{E}_{i}(1-r_iw_i)\xi_i'^2\le O(\eta\log(1/\eta)/m). A η ⊢ E i ( 1 − r i w i ) ξ i ′2 ≤ O ( η log ( 1/ η ) / m ) . Directional identifiability inequality. Apply the clean-kept contribution bound to (6), and use (8) for the complementary contribution. Combining the upper bound on E i ξ i 2 \mathbb{E}_i \xi_i^2 E i ξ i 2 E i r i w i ξ i 2 \mathbb{E}_i r_i w_i \xi_i^2 E i r i w i ξ i 2
A η ∣ 6 { Δ s 2 ≤ O ( η 2 log ( 1 / η ) m ) + 4 η ( E i ξ i 2 − E i r i w i ξ i 2 ) ≤ O ( η 2 log ( 1 / η ) m ) + 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) + 4 η Δ s 2 (10) \mathcal{A}_{\eta\mid6}\left\{
\begin{aligned}
\Delta_s^2 &\le O\left(\frac{\eta^2\log(1/\eta)}{m}\right)+4\eta\left(\mathbb{E}_i \xi_i^2-\mathbb{E}_i r_iw_i\xi_i^2\right)\\
&\le O\left(\frac{\eta^2\log(1/\eta)}{m}\right)+4\eta\left(V(\mu',s)-V(\mu,s)\right)+4\eta\Delta_s^2
\end{aligned}
\right. \tag*{(10)} A η ∣ 6 ⎩ ⎨ ⎧ Δ s 2 ≤ O ( m η 2 log ( 1/ η ) ) + 4 η ( E i ξ i 2 − E i r i w i ξ i 2 ) ≤ O ( m η 2 log ( 1/ η ) ) + 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) + 4 η Δ s 2 (10) Absorbing the variance difference. The term 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) 4\eta(V(\mu',s)-V(\mu,s)) 4 η ( V ( μ ′ , s ) − V ( μ , s )) a = V ( μ ′ , s ) − V ( μ , s ) a=V(\mu',s)-V(\mu,s) a = V ( μ ′ , s ) − V ( μ , s ) b = 4 η b=4\eta b = 4 η λ = m 2 / 16 \lambda=m^2/16 λ = m 2 /16
A η ∣ 4 { 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) ≤ m 2 16 ( V ( μ ′ , s ) − V ( μ , s ) ) 2 + ( 4 η ) 2 4 ( m 2 / 16 ) = m 2 16 ( V ( μ ′ , s ) − V ( μ , s ) ) 2 + 64 η 2 m 2 \mathcal{A}_{\eta\mid4}\left\{
\begin{aligned}
4\eta\left(V(\mu',s)-V(\mu,s)\right)
&\le\frac{m^2}{16}\left(V(\mu',s)-V(\mu,s)\right)^2+\frac{(4\eta)^2}{4(m^2/16)}\\
&= \frac{m^2}{16}\left(V(\mu',s)-V(\mu,s)\right)^2+\frac{64\eta^2}{m^2}
\end{aligned}
\right. A η ∣ 4 ⎩ ⎨ ⎧ 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) ≤ 16 m 2 ( V ( μ ′ , s ) − V ( μ , s ) ) 2 + 4 ( m 2 /16 ) ( 4 η ) 2 = 16 m 2 ( V ( μ ′ , s ) − V ( μ , s ) ) 2 + m 2 64 η 2 We chose λ = m 2 / 16 \lambda=m^2/16 λ = m 2 /16 Δ s 2 / 4 \Delta_s^2/4 Δ s 2 /4 V V V
A η ∣ 6 { m 2 16 ( V ( μ ′ , s ) − V ( μ , s ) ) 2 = 1 16 ( ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩ ) − ⟨ μ , s ⟩ ( 1 − ⟨ μ , s ⟩ ) ) 2 ≤ 1 4 Δ s 2 \mathcal{A}_{\eta\mid6}\left\{
\begin{aligned}
\frac{m^2}{16}\left(V(\mu',s)-V(\mu,s)\right)^2
&=\frac{1}{16}\left(\langle\mu',s\rangle(1-\langle\mu',s\rangle)-\langle\mu,s\rangle(1-\langle\mu,s\rangle)\right)^2\\
&\le\frac{1}{4}\Delta_s^2
\end{aligned}
\right. A η ∣ 6 ⎩ ⎨ ⎧ 16 m 2 ( V ( μ ′ , s ) − V ( μ , s ) ) 2 = 16 1 ( ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩) − ⟨ μ , s ⟩ ( 1 − ⟨ μ , s ⟩) ) 2 ≤ 4 1 Δ s 2 Combining these two inequalities, we obtain
A η ∣ 6 { 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) ≤ 1 4 Δ s 2 + 64 η 2 m 2 } . (11) \mathcal{A}_{\eta\mid6}\left\{4\eta\left(V(\mu',s)-V(\mu,s)\right)\le\frac{1}{4}\Delta_s^2+\frac{64\eta^2}{m^2}\right\}. \tag*{(11)} A η ∣ 6 { 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) ≤ 4 1 Δ s 2 + m 2 64 η 2 } . (11) Substituting (11) into (10) gives
A η ∣ 6 { Δ s 2 ≤ O ( η 2 log ( 1 / η ) m ) + ( 1 4 + 4 η ) Δ s 2 + 64 η 2 m 2 } . \mathcal{A}_{\eta\mid6}\left\{\Delta_s^2\le O\left(\frac{\eta^2\log(1/\eta)}{m}\right)+\left(\frac{1}{4}+4\eta\right)\Delta_s^2+\frac{64\eta^2}{m^2}\right\}. A η ∣ 6 { Δ s 2 ≤ O ( m η 2 log ( 1/ η ) ) + ( 4 1 + 4 η ) Δ s 2 + m 2 64 η 2 } . Since η ≤ η a b s \eta\le\eta_{\mathrm{abs}} η ≤ η abs η a b s \eta_{\mathrm{abs}} η abs
A η ∣ 6 { Δ s 2 ≤ O ( η 2 log ( 1 / η ) m ) + O ( η 2 m 2 ) = O ( η 2 log ( 1 / η ) m ) } , \mathcal{A}_{\eta\mid6}\left\{\Delta_s^2\le O\left(\frac{\eta^2\log(1/\eta)}{m}\right)+O\left(\frac{\eta^2}{m^2}\right)=O\left(\frac{\eta^2\log(1/\eta)}{m}\right)\right\}, A η ∣ 6 { Δ s 2 ≤ O ( m η 2 log ( 1/ η ) ) + O ( m 2 η 2 ) = O ( m η 2 log ( 1/ η ) ) } , as desired. The weighted Cauchy–Schwarz step in (8) has degree 6, and every other step has degree at most 6. Hence the entire proof has degree 6. □ \square □
Covariance Closeness Guarantees
We first prove the following lemma that will help us in bounding the differences of covariances by the differences of the means inside SoS.
Lemma 3.9. Let x x x a a a [ 0 , 1 ] [0,1] [ 0 , 1 ]
{ 0 ≤ x ≤ 1 } ∣ ∣ 6 { ( x ( 1 − x ) − a ( 1 − a ) ) 2 ≤ 4 ( x − a ) 2 } \{0 \le x \le1\}\ \left|\vphantom{\big|}\right._6\left\{(x(1-x)-a(1-a))^2 \le4(x-a)^2\right\} { 0 ≤ x ≤ 1 } 6 { ( x ( 1 − x ) − a ( 1 − a ) ) 2 ≤ 4 ( x − a ) 2 } Proof. We have
4 ( x − a ) 2 − ( x ( 1 − x ) − a ( 1 − a ) ) 2 = 4 ( x − a ) 2 − ( x − a ) 2 ( x + a − 1 ) 2 = 3 ( x − a ) 2 + ( x − a ) 2 ( 1 − ( x + a − 1 ) 2 ) = 3 ( x − a ) 2 + ( 1 − ( a − 1 ) 2 ) ( 1 − x ) ( x − a ) 2 + ( 1 − a 2 ) x ( x − a ) 2 + x [ ( x − a ) ( 1 − x ) ] 2 + ( 1 − x ) [ x ( x − a ) ] 2 ≥ 0. \begin{aligned}
4(x-a)^2-(x(1-x)-a(1-a))^2
&=4(x-a)^2-(x-a)^2(x+a-1)^2\\
&=3(x-a)^2+(x-a)^2(1-(x+a-1)^2)\\
&=3(x-a)^2+(1-(a-1)^2)(1-x)(x-a)^2+(1-a^2)x(x-a)^2\\
&\quad+x\left[(x-a)(1-x)\right]^2+(1-x)\left[x(x-a)\right]^2\\
&\ge0.
\end{aligned} 4 ( x − a ) 2 − ( x ( 1 − x ) − a ( 1 − a ) ) 2 = 4 ( x − a ) 2 − ( x − a ) 2 ( x + a − 1 ) 2 = 3 ( x − a ) 2 + ( x − a ) 2 ( 1 − ( x + a − 1 ) 2 ) = 3 ( x − a ) 2 + ( 1 − ( a − 1 ) 2 ) ( 1 − x ) ( x − a ) 2 + ( 1 − a 2 ) x ( x − a ) 2 + x [ ( x − a ) ( 1 − x ) ] 2 + ( 1 − x ) [ x ( x − a ) ] 2 ≥ 0. where every term in the last two lines is a square times either 1 1 1 x x x 1 − x 1-x 1 − x 1 − ( a − 1 ) 2 ≥ 0 1-(a-1)^2 \ge0 1 − ( a − 1 ) 2 ≥ 0 1 − a 2 ≥ 0 1-a^2 \ge0 1 − a 2 ≥ 0
{ 0 ≤ x ≤ 1 } ∣ ∣ 6 { ( x ( 1 − x ) − a ( 1 − a ) ) 2 ≤ 4 ( x − a ) 2 } . \{0 \le x \le1\}\ \left|\vphantom{\big|}\right._6\left\{(x(1-x)-a(1-a))^2 \le4(x-a)^2\right\}. { 0 ≤ x ≤ 1 } 6 { ( x ( 1 − x ) − a ( 1 − a ) ) 2 ≤ 4 ( x − a ) 2 } . Now using this lemma we can bound the difference between the covariances inside SoS.
Lemma 3.10. Let s s s { 0 , 1 } k \{0,1\}^k { 0 , 1 } k
A η ∣ ∣ 6 { ( ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩ ) − ⟨ μ , s ⟩ ( 1 − ⟨ μ , s ⟩ ) ) 2 ≤ 4 ⟨ μ ′ − μ , s ⟩ 2 } . \mathcal{A}_\eta\left|\vphantom{\big|}\right._6\left\{(\langle\mu',s\rangle(1-\langle\mu',s\rangle)-\langle\mu,s\rangle(1-\langle\mu,s\rangle))^2 \le4\langle\mu'-\mu,s\rangle^2\right\}. A η 6 { (⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩) − ⟨ μ , s ⟩ ( 1 − ⟨ μ , s ⟩) ) 2 ≤ 4 ⟨ μ ′ − μ , s ⟩ 2 } . Proof. Note that
A η ∣ ∣ 6 { ⟨ μ ′ , s ⟩ = ∑ j ∈ [ k ] μ j ′ s j ≥ 0 } , \mathcal{A}_\eta\left|\vphantom{\big|}\right._6\left\{
\begin{aligned}
\langle\mu',s\rangle&=\sum_{j\in[k]}\mu'_js_j\\
&\ge0
\end{aligned}
\right\}, A η 6 ⎩ ⎨ ⎧ ⟨ μ ′ , s ⟩ = j ∈ [ k ] ∑ μ j ′ s j ≥ 0 ⎭ ⎬ ⎫ , where the inequality comes from A η ∣ ∣ 6 { ∀ j ∈ [ k ] : μ j ′ ≥ 0 } \mathcal{A}_\eta\left|\vphantom{\big|}\right._6\{\forall j\in[k]:\mu'_j\ge0\} A η 6 { ∀ j ∈ [ k ] : μ j ′ ≥ 0 } s j s_j s j
A η ∣ ∣ 6 { ⟨ μ ′ , s ⟩ = ∑ j ∈ [ k ] μ j ′ s j ≤ 1 } , \mathcal{A}_\eta\left|\vphantom{\big|}\right._6\left\{
\begin{aligned}
\langle\mu',s\rangle&=\sum_{j\in[k]}\mu'_js_j\\
&\le1
\end{aligned}
\right\}, A η 6 ⎩ ⎨ ⎧ ⟨ μ ′ , s ⟩ = j ∈ [ k ] ∑ μ j ′ s j ≤ 1 ⎭ ⎬ ⎫ , where the inequality comes from the fact that s j s_j s j { 0 , 1 } \{0,1\} { 0 , 1 } A η ∣ ∣ 6 { ∀ j ∈ [ k ] : μ j ′ ≥ 0 , ∑ j μ j ′ = 1 } \mathcal{A}_\eta\left|\vphantom{\big|}\right._6\{\forall j\in[k]:\mu'_j\ge0,\sum_j\mu'_j=1\} A η 6 { ∀ j ∈ [ k ] : μ j ′ ≥ 0 , ∑ j μ j ′ = 1 }
A η ∣ ∣ 6 { 0 ≤ ⟨ μ ′ , s ⟩ ≤ 1 } , \mathcal{A}_\eta\left|\vphantom{\big|}\right._6\{0\le\langle\mu',s\rangle\le1\}, A η 6 { 0 ≤ ⟨ μ ′ , s ⟩ ≤ 1 } , Applying Lemma 3.9 with x = ⟨ μ ′ , s ⟩ x=\langle\mu',s\rangle x = ⟨ μ ′ , s ⟩ a = ⟨ μ , s ⟩ ∈ [ 0 , 1 ] a=\langle\mu,s\rangle\in[0,1] a = ⟨ μ , s ⟩ ∈ [ 0 , 1 ]
A η ∣ ∣ 6 { ( ⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩ ) − ⟨ μ , s ⟩ ( 1 − ⟨ μ , s ⟩ ) ) 2 ≤ 4 ⟨ μ ′ − μ , s ⟩ 2 } , \mathcal{A}_\eta\left|\vphantom{\big|}\right._6\left\{(\langle\mu',s\rangle(1-\langle\mu',s\rangle)-\langle\mu,s\rangle(1-\langle\mu,s\rangle))^2\le4\langle\mu'-\mu,s\rangle^2\right\}, A η 6 { (⟨ μ ′ , s ⟩ ( 1 − ⟨ μ ′ , s ⟩) − ⟨ μ , s ⟩ ( 1 − ⟨ μ , s ⟩) ) 2 ≤ 4 ⟨ μ ′ − μ , s ⟩ 2 } , as desired.
Heterogeneous Clean Batches
Corollary 3.11. Suppose the clean batches are independent and batch i i i m m m μ i ∈ Δ k − 1 \mu_i \in\Delta^{k-1} μ i ∈ Δ k − 1 ∥ μ i − μ ∥ T V ≤ ω \lVert\mu_i-\mu\rVert_{\mathrm{TV}} \le\omega ∥ μ i − μ ∥ TV ≤ ω i i i ω ∈ [ 0 , 1 ] \omega\in[0,1] ω ∈ [ 0 , 1 ] η ∈ ( 0 , η a b s ) \eta\in(0,\eta_{\mathrm{abs}}) η ∈ ( 0 , η abs ) β ∈ ( 0 , 1 ) \beta\in(0,1) β ∈ ( 0 , 1 )
τ c o v = C ( η log ( 1 / η ) m + ω m + ω 2 ) . \tau_{\mathrm{cov}}=C\left(\frac{\eta\log(1/\eta)}{m}+\frac{\omega}{m}+\omega^2\right). τ cov = C ( m η log ( 1/ η ) + m ω + ω 2 ) . for a sufficiently large absolute constant C C C n ≥ Ω ( ( k + log ( 1 / β ) ) / ( η 2 log ( 1 / η ) ) ) n \ge\Omega((k+\log(1/\beta))/(\eta^2\log(1/\eta))) n ≥ Ω (( k + log ( 1/ β )) / ( η 2 log ( 1/ η ))) η \eta η 1 − β 1-\beta 1 − β
∥ μ ^ − μ ∥ T V ≤ O ( η log ( 1 / η ) m + ω ) . \lVert\widehat{\mu}-\mu\rVert_{\mathrm{TV}}\le O\left(\eta\sqrt{\frac{\log(1/\eta)}{m}}+\omega\right). ∥ μ − μ ∥ TV ≤ O ( η m log ( 1/ η ) + ω ) . Proof. For a fixed s ∈ { 0 , 1 } k s\in\{0,1\}^k s ∈ { 0 , 1 } k d i = ⟨ μ i − μ , s ⟩ d_i=\langle\mu_i-\mu,s\rangle d i = ⟨ μ i − μ , s ⟩ Z i = ⟨ y i ∗ − μ i , s ⟩ Z_i=\langle y_i^*-\mu_i,s\rangle Z i = ⟨ y i ∗ − μ i , s ⟩ ∣ d i ∣ ≤ ω \lvert d_i\rvert\le\omega ∣ d i ∣ ≤ ω ξ i = Z i + d i \xi_i=Z_i+d_i ξ i = Z i + d i Z i Z_i Z i O ( 1 / m ) O(1/m) O ( 1/ m ) Fact 2.6 applies to these variables even though their distributions may differ. In addition, the weighted average E i d i Z i \mathbb{E}_i d_iZ_i E i d i Z i O ( ω 2 / ( n m ) ) O(\omega^2/(nm)) O ( ω 2 / ( nm )) s s s ∣ V ( μ i , s ) − V ( μ , s ) ∣ ≤ ω / m \lvert V(\mu_i,s)-V(\mu,s)\rvert\le\omega/m ∣ V ( μ i , s ) − V ( μ , s )∣ ≤ ω / m ( Z i + d i ) 2 ≤ 2 Z i 2 + 2 d i 2 (Z_i+d_i)^2\le2Z_i^2+2d_i^2 ( Z i + d i ) 2 ≤ 2 Z i 2 + 2 d i 2 1 − β 1-\beta 1 − β
∣ ∑ i ξ i ∣ ≤ O ( η log ( 1 / η ) m + ω ) , \left\lvert\sum_i \xi_i\right\rvert\le O\left(\eta\sqrt{\frac{\log(1/\eta)}{m}}+\omega\right), i ∑ ξ i ≤ O ( η m log ( 1/ η ) + ω ) , ∣ ∑ i ξ i 2 − V ( μ , s ) ∣ ≤ O ( η log ( 1 / η ) m + ω m + ω 2 ) , \left\lvert\sum_i \xi_i^2-V(\mu,s)\right\rvert\le O\left(\frac{\eta\log(1/\eta)}{m}+\frac{\omega}{m}+\omega^2\right), i ∑ ξ i 2 − V ( μ , s ) ≤ O ( m η log ( 1/ η ) + m ω + ω 2 ) , max ∣ T ∣ ≤ 2 η n 1 n ∑ i ∈ T ξ i 2 ≤ O ( η log ( 1 / η ) m + η ω 2 ) . \max_{\lvert T\rvert\le2\eta n}\frac{1}{n}\sum_{i\in T}\xi_i^2\le O\left(\frac{\eta\log(1/\eta)}{m}+\eta\omega^2\right). ∣ T ∣ ≤ 2 η n max n 1 i ∈ T ∑ ξ i 2 ≤ O ( m η log ( 1/ η ) + η ω 2 ) . Here the cross term in the second-moment estimate is at most O ( ω η log ( 1 / η ) / m ) ≤ O ( ω 2 + η 2 log ( 1 / η ) / m ) O(\omega\eta\sqrt{\log(1/\eta)/m})\le O(\omega^2+\eta^2\log(1/\eta)/m) O ( ω η log ( 1/ η ) / m ) ≤ O ( ω 2 + η 2 log ( 1/ η ) / m ) ω 2 \omega^2 ω 2 η 2 log ( 1 / η ) / m \eta^2\log(1/\eta)/m η 2 log ( 1/ η ) / m Lemma 3.4 bounds the clean covariance discrepancy by O ( η log ( 1 / η ) / m + ω / m + ω 2 ) O(\eta\log(1/\eta)/m+\omega/m+\omega^2) O ( η log ( 1/ η ) / m + ω / m + ω 2 ) Δ c o v ⋆ \Delta_{\mathrm{cov}}^\star Δ cov ⋆ Δ c o v ⋆ 1 = 0 \Delta_{\mathrm{cov}}^\star\mathbf{1}=0 Δ cov ⋆ 1 = 0 y i ∗ y_i^* y i ∗ μ ˉ = E i y i ∗ \bar{\mu}=\mathbb{E}_i y_i^* μ ˉ = E i y i ∗ Proposition 2.17 gives a diagonal majorizer with trace at most τ c o v \tau_{\mathrm{cov}} τ cov
For soundness, the clean-kept contribution in the proof of Lemma 3.8 is now O ( η 2 log ( 1 / η ) / m + ω 2 ) O(\eta^2\log(1/\eta)/m+\omega^2) O ( η 2 log ( 1/ η ) / m + ω 2 ) (10) , gives
A η ∣ 6 { Δ s 2 ≤ O ( η 2 log ( 1 / η ) m + ω 2 + η ω m ) + 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) + 4 η Δ s 2 } . \mathcal{A}_{\eta\mid6}\left\{\Delta_s^2\le O\left(\frac{\eta^2\log(1/\eta)}{m}+\omega^2+\frac{\eta\omega}{m}\right)+4\eta\left(V(\mu',s)-V(\mu,s)\right)+4\eta\Delta_s^2\right\}. A η ∣ 6 { Δ s 2 ≤ O ( m η 2 log ( 1/ η ) + ω 2 + m η ω ) + 4 η ( V ( μ ′ , s ) − V ( μ , s ) ) + 4 η Δ s 2 } . By Young’s inequality, η ω / m \eta\omega/m η ω / m ( ω 2 + η 2 / m 2 ) / 2 (\omega^2+\eta^2/m^2)/2 ( ω 2 + η 2 / m 2 ) /2 (11) and absorbing ( 1 / 4 + 4 η ) Δ s 2 (1/4+4\eta)\Delta_s^2 ( 1/4 + 4 η ) Δ s 2 Δ s 2 ≤ O ( η 2 log ( 1 / η ) / m + ω 2 ) \Delta_s^2\le O(\eta^2\log(1/\eta)/m+\omega^2) Δ s 2 ≤ O ( η 2 log ( 1/ η ) / m + ω 2 )
Private Estimation for Discrete Batches
This section instantiates the reduction of Hopkins, Kamath, Majid, and Narayanan [HKMN23] on the batch-average dataset Y = { y i } i ∈ [ n ] \mathcal{Y} = \{y_i\}_{i\in[n]} Y = { y i } i ∈ [ n ] § 3 , implemented here through low-degree pseudoexpectations for the diagonal-majorizer system.
Theorem 4.1 (Pure-DP Estimation from Discrete Batches). Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 α , β ∈ ( 0 , 1 ) \alpha,\beta\in(0,1) α , β ∈ ( 0 , 1 ) ε > 0 \varepsilon> 0 ε > 0 m ∈ N m \in\mathbb{N} m ∈ N ( ε , 0 ) (\varepsilon,0) ( ε , 0 ) n n n
B i = ( U i , 1 , … , U i , m ) , U i , 1 , … , U i , m ∼ i i d μ , B_i = (U_{i,1},\ldots,U_{i,m}), \qquad U_{i,1},\ldots,U_{i,m} \mathrel{\overset{\mathrm{iid}}{\sim}} \mu, B i = ( U i , 1 , … , U i , m ) , U i , 1 , … , U i , m ∼ iid μ , and outputs an estimate μ ^ \hat{\mu} μ ^ ∥ μ ^ − μ ∥ T V ≤ α \lVert\hat{\mu}-\mu\rVert_{\mathrm{TV}} \le\alpha ∥ μ ^ − μ ∥ TV ≤ α 1 − β 1-\beta 1 − β
n ≥ O ~ ( k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α m ε + k + log ( 1 / β ) ε ) . n \ge\widetilde{O}\left(\frac{k+\log(1/\beta)}{\alpha^2m}+\frac{k+\log(1/\beta)}{\alpha\sqrt{m}\varepsilon}+\frac{k+\log(1/\beta)}{\varepsilon}\right). n ≥ O ( α 2 m k + log ( 1/ β ) + α m ε k + log ( 1/ β ) + ε k + log ( 1/ β ) ) . Theorem 4.2 (Approximate-DP Estimation from Discrete Batches). Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 α , β , δ ∈ ( 0 , 1 ) \alpha,\beta,\delta\in(0,1) α , β , δ ∈ ( 0 , 1 ) ε > 0 \varepsilon> 0 ε > 0 m ∈ N m \in\mathbb{N} m ∈ N ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) n n n
B i = ( U i , 1 , … , U i , m ) , U i , 1 , … , U i , m ∼ i i d μ , B_i = (U_{i,1},\ldots,U_{i,m}), \qquad U_{i,1},\ldots,U_{i,m} \mathrel{\overset{\mathrm{iid}}{\sim}} \mu, B i = ( U i , 1 , … , U i , m ) , U i , 1 , … , U i , m ∼ iid μ , and outputs an estimate μ ^ \hat{\mu} μ ^ ∥ μ ^ − μ ∥ T V ≤ α \lVert\hat{\mu}-\mu\rVert_{\mathrm{TV}} \le\alpha ∥ μ ^ − μ ∥ TV ≤ α 1 − β 1-\beta 1 − β
n ≥ O ~ ( k + k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α m ε + log ( 1 / δ ) + log ( 1 / β ) ε ) . n \ge\widetilde{O}\left(k+\frac{k+\log(1/\beta)}{\alpha^2m}+\frac{k+\log(1/\beta)}{\alpha\sqrt{m}\varepsilon}+\frac{\log(1/\delta)+\log(1/\beta)}{\varepsilon}\right). n ≥ O ( k + α 2 m k + log ( 1/ β ) + α m ε k + log ( 1/ β ) + ε log ( 1/ δ ) + log ( 1/ β ) ) . Both privacy arguments use one score function. For the approximate-DP theorem, we also need a worst-case bound on the intermediate ratio V η ∗ ( Y ) / V 0.8 η ∗ ( Y ) V_{\eta^*}(\mathcal{Y})/V_{0.8\eta^*}(\mathcal{Y}) V η ∗ ( Y ) / V 0.8 η ∗ ( Y ) ℓ 1 \ell_1 ℓ 1 η ∗ n \eta^*n η ∗ n
Our score function reuses the succinct diagonal-majorizer system introduced in § 3.1 . For a dataset Y = { y i } i ∈ [ n ] \mathcal{Y} = \{y_i\}_{i\in[n]} Y = { y i } i ∈ [ n ] T T T B T ( Y ) \mathcal{B}_T(\mathcal{Y}) B T ( Y ) § 3.1 on the observed batch averages Y \mathcal{Y} Y A η \mathcal{A}_\eta A η
∑ i = 1 n w i ≥ n − T . \sum_{i=1}^{n} w_i \ge n-T. i = 1 ∑ n w i ≥ n − T . Relative to A η \mathcal{A}_\eta A η T T T
Σ ^ ′ : = E i [ ( y i ′ − μ ′ ) ( y i ′ − μ ′ ) T ] , Σ ( ν ) : = 1 m ( diag ( ν ) − ν ν T ) , τ c o v : = C η log ( 1 / η ) m , \widehat{\Sigma}' := \mathbb{E}_i[(y'_i-\mu')(y'_i-\mu')^{\mathsf T}], \qquad\Sigma(\nu) := \frac{1}{m}\left(\operatorname{diag}(\nu)-\nu\nu^{\mathsf T}\right), \qquad\tau_{\mathrm{cov}} := C\frac{\eta\log(1/\eta)}{m}, Σ ′ := E i [( y i ′ − μ ′ ) ( y i ′ − μ ′ ) T ] , Σ ( ν ) := m 1 ( diag ( ν ) − ν ν T ) , τ cov := C m η log ( 1/ η ) , for the same sufficiently large universal constant C C C § 3.1 . The selector/data-consistency block decides which observed batch averages are kept and ties the repaired points to the observed data on those indices. Simplex constraints keep the repaired points and the pseudo-mean inside the simplex, while the covariance-majorizer block enforces the subset-direction variance control in Lemma 3.7 .
Given a query point ν \nu ν
L ~ = B T ( Y ) \widetilde{\mathcal{L}} = \mathcal{B}_{T}(\mathcal{Y}) L = B T ( Y )
Closeness constraint: ∥ μ ~ ′ − v ∥ 1 ≤ α \lVert\widetilde{\mu}' - v\rVert_{1} \le\alpha ∥ μ ′ − v ∥ 1 ≤ α
Informally, the score of a candidate mean is the smallest trimming budget for which there exists a low-degree certificate consistent with that candidate. We first define certifiable means, then use them to define the score.
Definition 4.3 ((α , τ , T \alpha,\tau,T α , τ , T α , τ ≥ 0 \alpha,\tau\ge0 α , τ ≥ 0 k , n ∈ N k,n \in\mathbb{N} k , n ∈ N T ∈ [ 0 , n ] T \in[0,n] T ∈ [ 0 , n ] Y = { y i } i ∈ [ n ] ⊆ Δ k − 1 \mathcal{Y} = \{y_i\}_{i \in[n]} \subseteq\Delta^{k-1} Y = { y i } i ∈ [ n ] ⊆ Δ k − 1 v ∈ Δ k − 1 + B 1 k ( τ n + α ) ⊆ R k v \in\Delta^{k-1} + \mathbb{B}_{1}^{k}(\tau n + \alpha) \subseteq\mathbb{R}^{k} v ∈ Δ k − 1 + B 1 k ( τ n + α ) ⊆ R k ( α , τ , T ) (\alpha,\tau,T) ( α , τ , T ) Y \mathcal{Y} Y L \mathcal{L} L τ \tau τ Definition 2.10 ) satisfies B T ( Y ) \mathcal{B}_{T}(\mathcal{Y}) B T ( Y ) μ L : = L μ ′ \mu_{\mathcal{L}} := \mathcal{L}\mu' μ L := L μ ′
∥ R ( L ) ∥ R ≤ R ′ + τ \lVert\mathcal{R}(\mathcal{L})\rVert_{\mathbb{R}} \le R' + \tau ∥ R ( L ) ∥ R ≤ R ′ + τ R ′ = poly ( n , k ) R' = \operatorname{poly}(n,k) R ′ = poly ( n , k ) R ( L ) \mathcal{R}(\mathcal{L}) R ( L ) L \mathcal{L} L
The query point v v v L \mathcal{L} L
∥ μ L − v ∥ 1 ≤ α + τ T . \lVert\mu_{\mathcal{L}} - v\rVert_{1} \le\alpha+ \tau T . ∥ μ L − v ∥ 1 ≤ α + τ T . Moreover, we will call such a functional L \mathcal{L} L ( α , τ , T ) (\alpha,\tau,T) ( α , τ , T ) ( Y , v ) (\mathcal{Y},v) ( Y , v )
Lemma 4.4 (Approximate Certificates Are Close to the Simplex). Let L \mathcal{L} L τ \tau τ B T ( Y ) \mathcal{B}_{T}(\mathcal{Y}) B T ( Y )
x : = L μ ′ ∈ R k . x := \mathcal{L}\mu' \in\mathbb{R}^{k}. x := L μ ′ ∈ R k . Then for every j ∈ [ k ] j \in[k] j ∈ [ k ]
x j ≥ − τ T , ∣ ∑ j = 1 k x j − 1 ∣ ≤ τ T . x_j \ge-\tau T,\qquad\left\lvert\sum_{j=1}^{k}x_j - 1\right\rvert\le\tau T . x j ≥ − τ T , j = 1 ∑ k x j − 1 ≤ τ T . Consequently, there exists a point Π ( x ) ∈ Δ k − 1 \Pi(x) \in\Delta^{k-1} Π ( x ) ∈ Δ k − 1
∥ x − Π ( x ) ∥ 1 ≤ C k τ T \lVert x - \Pi(x)\rVert_{1} \le Ck\tau T ∥ x − Π ( x ) ∥ 1 ≤ C k τ T for an absolute constant C C C
Proof. For the coordinate lower bound, apply item 3 of Definition 2.10 to the constraint μ j ′ ≥ 0 \mu'_j \ge0 μ j ′ ≥ 0 p = 1 p = 1 p = 1
x j = L μ j ′ ≥ − τ T x_j = \mathcal{L}\mu'_j \ge-\tau T x j = L μ j ′ ≥ − τ T for every j ∈ [ k ] j \in[k] j ∈ [ k ] ∑ j ∈ [ k ] μ j ′ − 1 = 0 \sum_{j \in[k]}\mu'_j - 1 = 0 ∑ j ∈ [ k ] μ j ′ − 1 = 0 Definition 2.10 , with p = 1 p = 1 p = 1 L 1 = 1 \mathcal{L}1 = 1 L 1 = 1
∣ ∑ j = 1 k x j − 1 ∣ = ∣ L [ ∑ j = 1 k μ j ′ − 1 ] ∣ ≤ τ T . \left\lvert\sum_{j=1}^{k}x_j - 1\right\rvert
= \left\lvert\mathcal{L}\left[\sum_{j=1}^{k}\mu'_j - 1\right]\right\rvert
\le\tau T . j = 1 ∑ k x j − 1 = L [ j = 1 ∑ k μ j ′ − 1 ] ≤ τ T . Set y j : = max { x j , 0 } y_j := \max\{x_j,0\} y j := max { x j , 0 } j ∈ [ k ] j \in[k] j ∈ [ k ]
∥ x − y ∥ 1 = ∑ j = 1 k max { − x j , 0 } ≤ k τ T . \lVert x-y\rVert_{1} = \sum_{j=1}^{k}\max\{-x_j,0\} \le k\tau T . ∥ x − y ∥ 1 = j = 1 ∑ k max { − x j , 0 } ≤ k τ T . Also,
1 − τ T ≤ ∑ j = 1 k x j ≤ ∑ j = 1 k y j ≤ ∑ j = 1 k x j + k τ T ≤ 1 + ( k + 1 ) τ T . 1-\tau T \le\sum_{j=1}^{k}x_j \le\sum_{j=1}^{k}y_j \le\sum_{j=1}^{k}x_j+k\tau T \le1+(k+1)\tau T. 1 − τ T ≤ j = 1 ∑ k x j ≤ j = 1 ∑ k y j ≤ j = 1 ∑ k x j + k τ T ≤ 1 + ( k + 1 ) τ T . Let s ≔ ∑ j = 1 k y j s\coloneqq\sum_{j=1}^{k}y_j s : = ∑ j = 1 k y j
Π ( x ) ≔ { y / s if s > 0 , e 1 if s = 0 , \Pi(x)\coloneqq
\begin{cases}
y/s & \text{if }s>0,\\
e_1 & \text{if }s=0,
\end{cases} Π ( x ) : = { y / s e 1 if s > 0 , if s = 0 , where e 1 e_1 e 1 Π ( x ) ∈ Δ k − 1 \Pi(x)\in\Delta^{k-1} Π ( x ) ∈ Δ k − 1 s > 0 s>0 s > 0
∥ y − Π ( x ) ∥ 1 = ∑ j = 1 k y j ∣ 1 − 1 s ∣ = ∣ s − 1 ∣ ≤ ( k + 1 ) τ T . \lVert y-\Pi(x)\rVert_1=\sum_{j=1}^{k}y_j\left|1-\frac{1}{s}\right|=|s-1|\le(k+1)\tau T. ∥ y − Π ( x ) ∥ 1 = j = 1 ∑ k y j 1 − s 1 = ∣ s − 1∣ ≤ ( k + 1 ) τ T . If s = 0 s=0 s = 0 y = 0 y=0 y = 0 x j ≤ 0 x_j\le0 x j ≤ 0 j j j ∣ ∑ j x j − 1 ∣ ≤ τ T \left|\sum_j x_j-1\right|\le\tau T ∑ j x j − 1 ≤ τ T τ T ≥ 1 \tau T\ge1 τ T ≥ 1
∥ x − Π ( x ) ∥ 1 = ∥ x − e 1 ∥ 1 ≤ 1 + k τ T ≤ 2 k τ T . \lVert x-\Pi(x)\rVert_1=\lVert x-e_1\rVert_1\le1+k\tau T\le2k\tau T. ∥ x − Π ( x ) ∥ 1 = ∥ x − e 1 ∥ 1 ≤ 1 + k τ T ≤ 2 k τ T . For s > 0 s>0 s > 0
∥ x − Π ( x ) ∥ 1 ≤ ∥ x − y ∥ 1 + ∥ y − Π ( x ) ∥ 1 ≤ ( 2 k + 1 ) τ T ≤ 3 k τ T . \lVert x-\Pi(x)\rVert_1\le\lVert x-y\rVert_1+\lVert y-\Pi(x)\rVert_1\le(2k+1)\tau T\le3k\tau T. ∥ x − Π ( x ) ∥ 1 ≤ ∥ x − y ∥ 1 + ∥ y − Π ( x ) ∥ 1 ≤ ( 2 k + 1 ) τ T ≤ 3 k τ T . So the claim holds with C = 3 C=3 C = 3
Henceforth choose
0 < τ ≤ min { α , 1 / m } 100 C k n . (12) 0<\tau\le\frac{\min\{\alpha,1/\sqrt{m}\}}{100Ckn}. \tag*{(12)} 0 < τ ≤ 100 C k n min { α , 1/ m } . (12) This bound permits an inverse-polynomial choice of τ \tau τ
Definition 4.5 (Score Function). Let Δ k − 1 \Delta^{k-1} Δ k − 1 Δ k − 1 + B 1 k ( τ n + α ) ⊆ R k \Delta^{k-1}+B_1^k(\tau n+\alpha)\subseteq\mathbb{R}^k Δ k − 1 + B 1 k ( τ n + α ) ⊆ R k Δ k − 1 \Delta^{k-1} Δ k − 1 ℓ 1 \ell_1 ℓ 1 τ n + α \tau n+\alpha τ n + α k k k α , τ ≥ 0 \alpha,\tau\ge0 α , τ ≥ 0 Y = { y i } i ∈ [ n ] ⊆ Δ k − 1 \mathcal{Y}=\{y_i\}_{i\in[n]}\subseteq\Delta^{k-1} Y = { y i } i ∈ [ n ] ⊆ Δ k − 1 ν ∈ Δ k − 1 + B 1 k ( τ n + α ) \nu\in\Delta^{k-1}+B_1^k(\tau n+\alpha) ν ∈ Δ k − 1 + B 1 k ( τ n + α ) S : Δ k − 1 + B 1 k ( τ n + α ) → R ≥ 0 \mathcal{S}:\Delta^{k-1}+B_1^k(\tau n+\alpha)\to\mathbb{R}_{\ge0} S : Δ k − 1 + B 1 k ( τ n + α ) → R ≥ 0 ν \nu ν
S ( ν ; Y , α , τ ) ≔ min T ≥ 0 such that ν is a ( α , τ , T ) certifiable mean for Y . \mathcal{S}(\nu;\mathcal{Y},\alpha,\tau)\coloneqq\min_{T\ge0}\text{ such that }\nu\text{ is a }(\alpha,\tau,T)\text{ certifiable mean for }\mathcal{Y}. S ( ν ; Y , α , τ ) : = T ≥ 0 min such that ν is a ( α , τ , T ) certifiable mean for Y . For the rest of the section, we fix α \alpha α τ \tau τ Y \mathcal{Y} Y
S Y ( ν ) ≔ S ( ν ; Y , α , τ ) . \mathcal{S}_{\mathcal{Y}}(\nu)\coloneqq\mathcal{S}(\nu;\mathcal{Y},\alpha,\tau). S Y ( ν ) : = S ( ν ; Y , α , τ ) . We next verify the score properties required by Hopkins, Kamath, Majid, and Narayanan [HKMN23]: well-definedness, sensitivity, quasi-convexity, efficient computation, a low-scoring neighborhood, and bounds on sublevel-set volumes.
Properties of the Score
Well-Definedness of the Score Function
Lemma 4.6 (score function upper bound). Let α , τ ≥ 0 \alpha,\tau\ge0 α , τ ≥ 0 Y = { y i } i ∈ [ n ] \mathcal{Y}=\{y_i\}_{i\in[n]} Y = { y i } i ∈ [ n ] ν ∈ Δ k − 1 + B 1 k ( τ n + α ) \nu\in\Delta^{k-1} + B_1^k(\tau n+\alpha) ν ∈ Δ k − 1 + B 1 k ( τ n + α ) S Y ( ν ) ≤ n \mathcal{S}_{\mathcal{Y}}(\nu) \le n S Y ( ν ) ≤ n S \mathcal{S} S Definition 4.5 .
Proof. Let ν \nu ν Δ k − 1 + B 1 k ( τ n + α ) \Delta^{k-1}+B_1^k(\tau n+\alpha) Δ k − 1 + B 1 k ( τ n + α ) ( α , τ , T ) (\alpha,\tau,T) ( α , τ , T ) L \mathcal{L} L ( Y , ν ) (\mathcal{Y},\nu) ( Y , ν ) T = n T=n T = n
Let μ 0 : = Proj ℓ 1 ( ν ; Δ k − 1 ) \mu_0 := \operatorname{Proj}_{\ell_1}(\nu;\Delta^{k-1}) μ 0 := Proj ℓ 1 ( ν ; Δ k − 1 )
∥ ν − μ 0 ∥ 1 ≤ α + τ n = α + τ T \|\nu-\mu_0\|_1 \le\alpha+\tau n=\alpha+\tau T ∥ ν − μ 0 ∥ 1 ≤ α + τ n = α + τ T because ν ∈ Δ k − 1 + B 1 k ( α + τ n ) \nu\in\Delta^{k-1}+B_1^k(\alpha+\tau n) ν ∈ Δ k − 1 + B 1 k ( α + τ n ) L \mathcal{L} L
∀ i ∈ [ n ] : w i = 0 , y i ′ = μ 0 , μ ′ = μ 0 , λ = 0. \forall i\in[n]:\qquad w_i=0,\qquad y_i'=\mu_0,\qquad\mu'=\mu_0,\qquad\lambda=0. ∀ i ∈ [ n ] : w i = 0 , y i ′ = μ 0 , μ ′ = μ 0 , λ = 0. All constraints except the diagonal-majorizer equality are then immediate. Since every y i ′ = μ 0 y_i'=\mu_0 y i ′ = μ 0 Σ ^ ′ = 0 \widehat{\Sigma}'=0 Σ ′ = 0 μ 0 ∈ Δ k − 1 \mu_0\in\Delta^{k-1} μ 0 ∈ Δ k − 1 Fact 2.3 implies that Σ ( μ 0 ) \Sigma(\mu_0) Σ ( μ 0 ) M M M
M M T = Σ ( μ 0 ) . MM^{\mathsf{T}}=\Sigma(\mu_0). M M T = Σ ( μ 0 ) . Thus
Σ ^ ′ + M M T = Σ ( μ 0 ) = Σ ( μ ′ ) + Diag ( λ ) , ∑ j ∈ [ k ] λ j = 0 ≤ τ c o v . \widehat{\Sigma}'+MM^{\mathsf{T}}=\Sigma(\mu_0)=\Sigma(\mu')+\operatorname{Diag}(\lambda),\qquad\sum_{j\in[k]}\lambda_j=0\le\tau_{\mathrm{cov}}. Σ ′ + M M T = Σ ( μ 0 ) = Σ ( μ ′ ) + Diag ( λ ) , j ∈ [ k ] ∑ λ j = 0 ≤ τ cov . Since the entries of μ 0 \mu_0 μ 0 M M M R ′ R' R ′ S Y ( ν ) ≤ n \mathcal{S}_{\mathcal{Y}}(\nu)\le n S Y ( ν ) ≤ n ( T , L ) (T,\mathcal{L}) ( T , L ) □ \square □
Bounded Sensitivity
Lemma 4.7 (Bounded Sensitivity). For every fixed ν ∈ Δ k − 1 + B 1 k ( τ n + α ) \nu\in\Delta^{k-1}+B_1^k(\tau n+\alpha) ν ∈ Δ k − 1 + B 1 k ( τ n + α ) Y ↦ S ( ν ; Y , α , τ ) \mathcal{Y}\mapsto S(\nu;\mathcal{Y},\alpha,\tau) Y ↦ S ( ν ; Y , α , τ ) Y = { y i } i ∈ [ n ] \mathcal{Y}=\{y_i\}_{i\in[n]} Y = { y i } i ∈ [ n ]
Proof. Suppose Y = { y 1 , … , y n } \mathcal{Y}=\{y_1,\ldots,y_n\} Y = { y 1 , … , y n } Y ′ = { y ˉ 1 , y 2 , … , y n } \mathcal{Y}'=\{\bar{y}_1,y_2,\ldots,y_n\} Y ′ = { y ˉ 1 , y 2 , … , y n } ν ∈ Δ k − 1 + B 1 k ( τ n + α ) \nu\in\Delta^{k-1}+B_1^k(\tau n+\alpha) ν ∈ Δ k − 1 + B 1 k ( τ n + α ) S Y ( ν ) = T \mathcal{S}_{\mathcal{Y}}(\nu)=T S Y ( ν ) = T ( α , τ , T ) (\alpha,\tau,T) ( α , τ , T ) L \mathcal{L} L ( Y , ν ) (\mathcal{Y},\nu) ( Y , ν ) ( α , τ , T + 1 ) (\alpha,\tau,T+1) ( α , τ , T + 1 ) L ′ \mathcal{L}' L ′ ( Y ′ , ν ) (\mathcal{Y}',\nu) ( Y ′ , ν )
We define L ′ \mathcal{L}' L ′ p p p w 1 w_1 w 1 L ′ ( p ) = L ( p ) \mathcal{L}'(p)=\mathcal{L}(p) L ′ ( p ) = L ( p ) p p p w 1 w_1 w 1 L ′ ( p ) = 0 \mathcal{L}'(p)=0 L ′ ( p ) = 0
If T + 1 ≥ n T+1\ge n T + 1 ≥ n T + 1 < n T+1<n T + 1 < n ∥ R ( L ) ∥ F \|\mathcal{R}(\mathcal{L})\|_{\mathrm{F}} ∥ R ( L ) ∥ F p p p ∥ R ( p ) ∥ 2 ≤ 1 \|\mathcal{R}(p)\|_2\le1 ∥ R ( p ) ∥ 2 ≤ 1 q = p ∣ w 1 = 0 q=p|_{w_1=0} q = p ∣ w 1 = 0 ∥ R ( q ) ∥ 2 ≤ 1 \|\mathcal{R}(q)\|_2\le1 ∥ R ( q ) ∥ 2 ≤ 1 s s s w 1 w_1 w 1
L ′ [ s p 2 ] = L [ s q 2 ] . \mathcal{L}'[sp^2]=\mathcal{L}[sq^2]. L ′ [ s p 2 ] = L [ s q 2 ] . ::: Thus the constraints independent of w 1 w_1 w 1 T T T w 1 w_1 w 1 Definition 2.10 ,
L ′ [ ( ∑ i = 1 n w i − n + T + 1 ) p 2 ] = L [ ( ∑ i ≠ 1 n w i − n + T + 1 ) q 2 ] = L [ ( ∑ i = 1 n w i − n + T ) q 2 ] + L [ ( 1 − w 1 ) 2 q 2 ] + L [ w 1 ( 1 − w 1 ) q 2 ] ≥ − 5 τ T n − 4 τ T − τ T ≥ − 5 τ ( T + 1 ) n . \mathcal{L}'\left[\left(\sum_{i=1}^{n}w_i-n+T+1\right)p^2\right]=\mathcal{L}\left[\left(\sum_{i\ne1}^{n}w_i-n+T+1\right)q^2\right]
=\mathcal{L}\left[\left(\sum_{i=1}^{n}w_i-n+T\right)q^2\right]+\mathcal{L}[(1-w_1)^2q^2]+\mathcal{L}[w_1(1-w_1)q^2]
\ge-5\tau Tn-4\tau T-\tau T
\ge-5\tau(T+1)n. L ′ [ ( i = 1 ∑ n w i − n + T + 1 ) p 2 ] = L i = 1 ∑ n w i − n + T + 1 q 2 = L [ ( i = 1 ∑ n w i − n + T ) q 2 ] + L [( 1 − w 1 ) 2 q 2 ] + L [ w 1 ( 1 − w 1 ) q 2 ] ≥ − 5 τ T n − 4 τ T − τ T ≥ − 5 τ ( T + 1 ) n . Here deg q ≤ 2 \deg q\le2 deg q ≤ 2 L [ ( 1 − w 1 ) 2 q 2 ] \mathcal{L}[(1-w_1)^2q^2] L [( 1 − w 1 ) 2 q 2 ] − 4 τ T -4\tau T − 4 τ T ∥ R ( ( 1 − w 1 ) q ) ∥ 2 ≤ 2 \|\mathcal{R}((1-w_1)q)\|_2\le2 ∥ R (( 1 − w 1 ) q ) ∥ 2 ≤ 2 L [ w 1 ( 1 − w 1 ) q 2 ] \mathcal{L}[w_1(1-w_1)q^2] L [ w 1 ( 1 − w 1 ) q 2 ] − τ T -\tau T − τ T Definition 2.10 , applied to w 1 2 − w 1 = 0 w_1^2-w_1=0 w 1 2 − w 1 = 0 Y \mathcal{Y} Y Y ′ \mathcal{Y}' Y ′
Quasi Convexity
Lemma 4.8 (Quasi Convexity). For every fixed dataset Y \mathcal{Y} Y ν ↦ S Y ( ν ) \nu\mapsto\mathcal{S}_{\mathcal{Y}}(\nu) ν ↦ S Y ( ν ) Δ k − 1 + B 1 k ( τ n + α ) \Delta^{k-1}+B_1^k(\tau n+\alpha) Δ k − 1 + B 1 k ( τ n + α )
Proof. Let ν 1 , ν 2 ∈ Δ k − 1 + B 1 k ( τ n + α ) \nu_1,\nu_2\in\Delta^{k-1}+B_1^k(\tau n+\alpha) ν 1 , ν 2 ∈ Δ k − 1 + B 1 k ( τ n + α ) T 1 , T 2 ∈ [ 0 , n ] T_1,T_2\in[0,n] T 1 , T 2 ∈ [ 0 , n ] S Y ( ν 1 ) = T 1 \mathcal{S}_{\mathcal{Y}}(\nu_1)=T_1 S Y ( ν 1 ) = T 1 S Y ( ν 2 ) = T 2 \mathcal{S}_{\mathcal{Y}}(\nu_2)=T_2 S Y ( ν 2 ) = T 2 λ ∈ [ 0 , 1 ] \lambda\in[0,1] λ ∈ [ 0 , 1 ] T = S Y ( λ ν 1 + ( 1 − λ ) ν 2 ) T=\mathcal{S}_{\mathcal{Y}}(\lambda\nu_1+(1-\lambda)\nu_2) T = S Y ( λ ν 1 + ( 1 − λ ) ν 2 ) T ≤ max { T 1 , T 2 } T\le\max\{T_1,T_2\} T ≤ max { T 1 , T 2 } Definition 4.5 , we get that ν 1 \nu_1 ν 1 ( α , τ , T 1 ) (\alpha,\tau,T_1) ( α , τ , T 1 ) Y \mathcal{Y} Y ν 2 \nu_2 ν 2 L 1 , L 2 \mathcal{L}_1,\mathcal{L}_2 L 1 , L 2 ( Y , ν 1 ) (\mathcal{Y},\nu_1) ( Y , ν 1 ) ( Y , ν 2 ) (\mathcal{Y},\nu_2) ( Y , ν 2 ) T 1 , T 2 T_1,T_2 T 1 , T 2 Definition 4.3 . We construct L = λ L 1 + ( 1 − λ ) L 2 \mathcal{L}=\lambda\mathcal{L}_1+(1-\lambda)\mathcal{L}_2 L = λ L 1 + ( 1 − λ ) L 2 λ ν 1 + ( 1 − λ ) ν 2 \lambda\nu_1+(1-\lambda)\nu_2 λ ν 1 + ( 1 − λ ) ν 2 max { T 1 , T 2 } \max\{T_1,T_2\} max { T 1 , T 2 } T T T L \mathcal{L} L T T T Definition 2.10 . Let p p p ∥ R ( p ) ∥ 2 ≤ 1 \|\mathcal{R}(p)\|_2\le1 ∥ R ( p ) ∥ 2 ≤ 1 T 1 ≤ T 2 T_1\le T_2 T 1 ≤ T 2
L [ ( ∑ i = 1 n w i + T 2 − n ) p 2 ] = λ L 1 [ ( ∑ i = 1 n w i + T 1 + ( T 2 − T 1 ) − n ) p 2 ] + ( 1 − λ ) L 2 [ ( ∑ i = 1 n w i + T 2 − n ) p 2 ] = λ L 1 [ ( ∑ i = 1 n w i + T 1 − n ) p 2 ] + λ ( T 2 − T 1 ) L 1 p 2 + ( 1 − λ ) L 2 [ ( ∑ i = 1 n w i + T 2 − n ) p 2 ] ≥ λ ( − 5 τ T 1 n ) + λ ( T 2 − T 1 ) ( − 5 τ T 1 ) + ( 1 − λ ) ( − 5 τ T 2 n ) ≥ λ ( − 5 τ T 1 n ) + λ [ − 5 τ n ( T 2 − T 1 ) ] + ( 1 − λ ) ( − 5 τ T 2 n ) ≥ − 5 τ T 2 n , \begin{aligned}
\mathcal{L}\left[\left(\sum_{i=1}^{n}w_i+T_2-n\right)p^2\right]
=\lambda\mathcal{L}_1\left[\left(\sum_{i=1}^{n}w_i+T_1+(T_2-T_1)-n\right)p^2\right]+(1-\lambda)\mathcal{L}_2\left[\left(\sum_{i=1}^{n}w_i+T_2-n\right)p^2\right] \\
&=\lambda\mathcal{L}_1\left[\left(\sum_{i=1}^{n}w_i+T_1-n\right)p^2\right]+\lambda(T_2-T_1)\mathcal{L}_1p^2+(1-\lambda)\mathcal{L}_2\left[\left(\sum_{i=1}^{n}w_i+T_2-n\right)p^2\right] \\
&\ge\lambda(-5\tau T_1n)+\lambda(T_2-T_1)(-5\tau T_1)+(1-\lambda)(-5\tau T_2n) \\
&\ge\lambda(-5\tau T_1n)+\lambda[-5\tau n(T_2-T_1)]+(1-\lambda)(-5\tau T_2n) \\
&\ge-5\tau T_2n,
\end{aligned} L [ ( i = 1 ∑ n w i + T 2 − n ) p 2 ] = λ L 1 [ ( i = 1 ∑ n w i + T 1 + ( T 2 − T 1 ) − n ) p 2 ] + ( 1 − λ ) L 2 [ ( i = 1 ∑ n w i + T 2 − n ) p 2 ] = λ L 1 [ ( i = 1 ∑ n w i + T 1 − n ) p 2 ] + λ ( T 2 − T 1 ) L 1 p 2 + ( 1 − λ ) L 2 [ ( i = 1 ∑ n w i + T 2 − n ) p 2 ] ≥ λ ( − 5 τ T 1 n ) + λ ( T 2 − T 1 ) ( − 5 τ T 1 ) + ( 1 − λ ) ( − 5 τ T 2 n ) ≥ λ ( − 5 τ T 1 n ) + λ [ − 5 τ n ( T 2 − T 1 )] + ( 1 − λ ) ( − 5 τ T 2 n ) ≥ − 5 τ T 2 n , where we used that T 1 ≤ n T_1\le n T 1 ≤ n
Efficient Computability
Lemma 4.9 (Efficient Computation of Score Function). Let ν \nu ν Δ k − 1 + B 1 k ( τ n + α ) \Delta^{k-1}+B_1^k(\tau n+\alpha) Δ k − 1 + B 1 k ( τ n + α ) T = S Y ( ν ) T=\mathcal{S}_{\mathcal{Y}}(\nu) T = S Y ( ν ) T T T γ \gamma γ n n n k k k log ( 1 / γ ) \log(1/\gamma) log ( 1/ γ )
log ( 1 / τ ) \log(1/\tau) log ( 1/ τ )
Proof. There are poly ( n , k ) \operatorname{poly}(n,k) poly ( n , k ) T T T ℓ 1 \ell_1 ℓ 1 T T T
For completeness, after fixing the normalization L 1 = 1 \mathcal{L}1=1 L 1 = 1 T T T γ \gamma γ τ γ \tau\gamma τ γ τ n γ \tau n\gamma τ nγ T ≤ n T \le n T ≤ n L p 2 ≥ − τ T \mathcal{L}p^2 \ge-\tau T L p 2 ≥ − τ T Lemma 4.6 . Taking R ′ R' R ′ τ γ / poly ( n , k ) \tau\gamma/\operatorname{poly}(n,k) τ γ / poly ( n , k ) τ γ / poly ( n , k ) \tau\gamma/\operatorname{poly}(n,k) τ γ / poly ( n , k ) T T T γ \gamma γ log ( 1 / τ ) \log(1/\tau) log ( 1/ τ ) τ \tau τ
Problem-Specific Consequences
Efficiently Finding Low Scoring Points
Lemma 4.10. There exists an algorithm, running in time polynomial in n n n k k k log ( 1 / τ ) \log(1/\tau) log ( 1/ τ ) μ ^ \hat{\mu} μ ^
S Y ( μ ^ ) ≤ min ν ∈ Δ k − 1 + B 1 k ( τ n + α ) S Y ( ν ) + 1 , \mathcal{S}_{\mathcal{Y}}(\hat{\mu}) \le\min_{\nu\in\Delta^{k-1}+B_1^k(\tau n+\alpha)} \mathcal{S}_{\mathcal{Y}}(\nu)+1, S Y ( μ ^ ) ≤ ν ∈ Δ k − 1 + B 1 k ( τ n + α ) min S Y ( ν ) + 1 , and every point within ℓ 1 \ell_1 ℓ 1 α / 2 \alpha/2 α /2 μ ^ \hat{\mu} μ ^
Proof. Consider the certificate constraints in Definition 4.3 with the closeness constraint in item 2 omitted. Without the closeness constraint, we may still apply the weak-feasibility argument in Lemma 4.9. Binary search, with an additive budget allowance smaller than 1, returns a feasible budget T min T_{\min} T m i n L \mathcal{L} L
T min ≤ min ν ∈ Δ k − 1 + B 1 k ( τ n + α ) S Y ( ν ) + 1. T_{\min} \le\min_{\nu\in\Delta^{k-1}+B_1^k(\tau n+\alpha)} \mathcal{S}_{\mathcal{Y}}(\nu)+1. T m i n ≤ ν ∈ Δ k − 1 + B 1 k ( τ n + α ) min S Y ( ν ) + 1. Indeed, dropping the closeness constraint can only decrease the optimum. Define
μ L : = L μ ′ , μ ^ : = Π ( μ L ) . \mu_{\mathcal{L}} := \mathcal{L}\mu', \qquad\hat{\mu} := \Pi(\mu_{\mathcal{L}}). μ L := L μ ′ , μ ^ := Π ( μ L ) . By Lemma 4.4 and (12) , we have
∥ μ ^ − μ L ∥ 1 ≤ C k τ T min ≤ C k τ n ≤ α / 100 ≤ α / 10. \lVert\hat{\mu}-\mu_{\mathcal{L}}\rVert_1 \le C k \tau T_{\min} \le C k \tau n \le\alpha/100 \le\alpha/10. ∥ μ ^ − μ L ∥ 1 ≤ C k τ T m i n ≤ C k τ n ≤ α /100 ≤ α /10. In particular, μ ^ ∈ Δ k − 1 ⊆ Δ k − 1 + B 1 k ( τ n + α ) \hat{\mu} \in\Delta^{k-1} \subseteq\Delta^{k-1}+B_1^k(\tau n+\alpha) μ ^ ∈ Δ k − 1 ⊆ Δ k − 1 + B 1 k ( τ n + α ) L \mathcal{L} L μ ^ \hat{\mu} μ ^ T min T_{\min} T m i n
∥ μ ^ − μ L ∥ 1 ≤ α / 10 ≤ α + τ T min . \lVert\hat{\mu}-\mu_{\mathcal{L}}\rVert_1 \le\alpha/10 \le\alpha+\tau T_{\min}. ∥ μ ^ − μ L ∥ 1 ≤ α /10 ≤ α + τ T m i n . Now let ν \nu ν ∥ ν − μ ^ ∥ 1 ≤ α / 2 \|\nu-\hat{\mu}\|_1\leq\alpha/2 ∥ ν − μ ^ ∥ 1 ≤ α /2
∥ ν − μ L ∥ 1 ≤ ∥ ν − μ ^ ∥ 1 + ∥ μ ^ − μ L ∥ 1 ≤ α 2 + α 10 < α . \|\nu-\mu_{\mathcal L}\|_1\leq\|\nu-\hat{\mu}\|_1+\|\hat{\mu}-\mu_{\mathcal L}\|_1\leq\frac{\alpha}{2}+\frac{\alpha}{10}<\alpha. ∥ ν − μ L ∥ 1 ≤ ∥ ν − μ ^ ∥ 1 + ∥ μ ^ − μ L ∥ 1 ≤ 2 α + 10 α < α . Hence L \mathcal L L ν \nu ν T min T_{\min} T m i n T min ≤ min ν ∈ Δ k − 1 + B 1 ( τ n + α ) S Y ( ν ) + 1 T_{\min}\leq\min_{\nu\in\Delta^{k-1}+\mathbb{B}_1(\tau n+\alpha)}\mathcal{S}_{\mathcal Y}(\nu)+1 T m i n ≤ min ν ∈ Δ k − 1 + B 1 ( τ n + α ) S Y ( ν ) + 1
Lemma 4.11 (Soundness of Low-Trimming Certificates). Let η 0 ∈ ( 0 , η a b s ) \eta_0\in(0,\eta_{\mathrm{abs}}) η 0 ∈ ( 0 , η abs ) τ c o v ≤ C η 0 log ( 1 / η 0 ) / m \tau_{\mathrm{cov}}\leq C\eta_0\log(1/\eta_0)/m τ cov ≤ C η 0 log ( 1/ η 0 ) / m Y ∗ = { y i ∗ } i ∈ [ n ] \mathcal{Y}^*=\{y_i^*\}_{i\in[n]} Y ∗ = { y i ∗ } i ∈ [ n ] η 0 \eta_0 η 0 Y = { y i } i ∈ [ n ] \mathcal{Y}=\{y_i\}_{i\in[n]} Y = { y i } i ∈ [ n ] η 0 \eta_0 η 0 Y ∗ \mathcal{Y}^* Y ∗ L \mathcal L L ( α , τ , T ) (\alpha,\tau,T) ( α , τ , T ) ( Y , ν ) (\mathcal{Y},\nu) ( Y , ν ) T ≤ η 0 n T\leq\eta_0n T ≤ η 0 n μ L : = L μ ′ \mu_{\mathcal L}:=\mathcal L\mu' μ L := L μ ′
∥ μ L − μ ∥ 1 ≤ O ( η 0 log ( 1 / η 0 ) m ) . \|\mu_{\mathcal L}-\mu\|_1\leq O\left(\eta_0\sqrt{\frac{\log(1/\eta_0)}{m}}\right). ∥ μ L − μ ∥ 1 ≤ O ( η 0 m log ( 1/ η 0 ) ) . Proof. For exact certificates, this is exactly the deterministic soundness proof from lemma 3.8 and the proof of Theorem 3.1 , with trimming budget T ≤ η 0 n T\leq\eta_0n T ≤ η 0 n τ poly ( n , k , m , η 0 − 1 ) T \tau\operatorname{poly}(n,k,m,\eta_0^{-1})T τ poly ( n , k , m , η 0 − 1 ) T τ \tau τ
τ poly ( n , k , m , η 0 − 1 ) T = o ( η 0 2 log ( 1 / η 0 ) m ) . \tau\operatorname{poly}(n,k,m,\eta_0^{-1})T=o\left(\frac{\eta_0^2\log(1/\eta_0)}{m}\right). τ poly ( n , k , m , η 0 − 1 ) T = o ( m η 0 2 log ( 1/ η 0 ) ) . Thus the perturbation is lower order than the squared directional-error bound in Lemma 3.8 . Applying that directional bound in both subset directions and using ∣ ∑ j ( μ L ) j − 1 ∣ ≤ τ T \left|\sum_j(\mu_{\mathcal L})_j-1\right|\leq\tau T ∑ j ( μ L ) j − 1 ≤ τ T Lemma 4.4 gives the claimed ℓ 1 \ell_1 ℓ 1
Volume
Lemma 4.12 (Volume). Assume n ≥ Ω ( ( k + log ( 1 / β ) ) / ( η 2 log ( 1 / η ) ) ) n\geq\Omega((k+\log(1/\beta))/(\eta^2\log(1/\eta))) n ≥ Ω (( k + log ( 1/ β )) / ( η 2 log ( 1/ η ))) η ∗ ≥ η \eta^*\geq\eta η ∗ ≥ η
B i ∗ = ( U i , 1 ∗ , … , U i , m ∗ ) , U i , 1 ∗ , … , U i , m ∗ ∼ i i d μ , y i ∗ : = 1 m ∑ j = 1 m e U i , j ∗ B_i^*=(U_{i,1}^*,\ldots,U_{i,m}^*),\qquad U_{i,1}^*,\ldots,U_{i,m}^*\overset{\mathrm{iid}}{\sim}\mu,\qquad y_i^*:=\frac{1}{m}\sum_{j=1}^{m}e_{U_{i,j}^*} B i ∗ = ( U i , 1 ∗ , … , U i , m ∗ ) , U i , 1 ∗ , … , U i , m ∗ ∼ iid μ , y i ∗ := m 1 j = 1 ∑ m e U i , j ∗ for each i ∈ [ n ] i\in[n] i ∈ [ n ] Y ∗ = { y 1 ∗ , … , y n ∗ } \mathcal{Y}^*=\{y_1^*,\ldots,y_n^*\} Y ∗ = { y 1 ∗ , … , y n ∗ } Y = { y i } i ∈ [ n ] \mathcal{Y}=\{y_i\}_{i\in[n]} Y = { y i } i ∈ [ n ] η \eta η Y ∗ \mathcal{Y}^* Y ∗ 1 − β 1-\beta 1 − β Y ∗ \mathcal{Y}^* Y ∗
There exists a point μ 0 \mu_0 μ 0 ℓ 1 \ell_1 ℓ 1 α \alpha α μ 0 \mu_0 μ 0 η n \eta n η n
All points which have score at most η ∗ n \eta^*n η ∗ n ℓ 1 \ell_1 ℓ 1 α + O ( 1 / m ) \alpha+O(1/\sqrt{m}) α + O ( 1/ m ) μ 0 \mu_0 μ 0
Simultaneously for every η ′ ∈ [ η , η ∗ ] \eta'\in[\eta,\eta^*] η ′ ∈ [ η , η ∗ ] η ′ n \eta'n η ′ n ℓ 1 \ell_1 ℓ 1 μ 0 \mu_0 μ 0
α + O ( η ′ log ( 1 / η ′ ) m ) . \alpha+O\left(\eta'\sqrt{\frac{\log(1/\eta')}{m}}\right). α + O ( η ′ m log ( 1/ η ′ ) ) . Proof. Take μ 0 \mu_0 μ 0 μ ˉ \bar{\mu} μ ˉ 0 ≤ j ≤ ⌈ log 2 ( η ∗ / η ) ⌉ 0 \le j \le\lceil\log_2(\eta^*/\eta)\rceil 0 ≤ j ≤ ⌈ log 2 ( η ∗ / η )⌉ Corollary 3.3 at min { 2 j η , η ∗ } \min\{2^j\eta,\eta^*\} min { 2 j η , η ∗ } β / 2 j + 2 \beta/2^{j+2} β / 2 j + 2 1 − β 1-\beta 1 − β
( 2 q ) 2 log ( 1 / ( 2 q ) ) ≥ 2 q 2 log ( 1 / q ) ( 0 < q ≤ 1 / 8 ) . (2q)^2\log(1/(2q)) \ge2q^2\log(1/q) \qquad(0<q\le1/8). ( 2 q ) 2 log ( 1/ ( 2 q )) ≥ 2 q 2 log ( 1/ q ) ( 0 < q ≤ 1/8 ) . For the final, possibly shorter step, we therefore obtain
min { 2 j η , η ∗ } 2 log ( 1 min { 2 j η , η ∗ } ) ≥ 2 j − 1 η 2 log ( 1 / η ) ( j ≥ 1 ) . \min\{2^j\eta,\eta^*\}^2\log\left(\frac{1}{\min\{2^j\eta,\eta^*\}}\right) \ge2^{j-1}\eta^2\log(1/\eta) \qquad(j\ge1). min { 2 j η , η ∗ } 2 log ( min { 2 j η , η ∗ } 1 ) ≥ 2 j − 1 η 2 log ( 1/ η ) ( j ≥ 1 ) . Since k + log ( 2 j + 2 / β ) = O ( k + log ( 1 / β ) + j ) k+\log(2^{j+2}/\beta)=O(k+\log(1/\beta)+j) k + log ( 2 j + 2 / β ) = O ( k + log ( 1/ β ) + j ) j + 1 = O ( 2 j ) j+1=O(2^j) j + 1 = O ( 2 j ) min { 2 j η , η ∗ } \min\{2^j\eta,\eta^*\} min { 2 j η , η ∗ } r i r_i r i Corollary 3.6 , there exists λ \lambda λ
Diag ( λ ) − ( E i [ ( y i ∗ − μ ˉ ) ( y i ∗ − μ ˉ ) T ] − 1 m ( diag ( μ ˉ ) − μ ˉ μ ˉ T ) ) ⪰ 0 \operatorname{Diag}(\lambda)-\left(\mathbb{E}_i[(y_i^*-\bar{\mu})(y_i^*-\bar{\mu})^{\mathsf{T}}]-\frac{1}{m}\left(\operatorname{diag}(\bar{\mu})-\bar{\mu}\bar{\mu}^{\mathsf{T}}\right)\right)\succeq0 Diag ( λ ) − ( E i [( y i ∗ − μ ˉ ) ( y i ∗ − μ ˉ ) T ] − m 1 ( diag ( μ ˉ ) − μ ˉ μ ˉ T ) ) ⪰ 0 and ∑ j λ j ≤ O ( η log ( 1 / η ) / m ) ≤ τ c o v \sum_j\lambda_j\le O(\eta\log(1/\eta)/m)\le\tau_{\mathrm{cov}} ∑ j λ j ≤ O ( η log ( 1/ η ) / m ) ≤ τ cov τ c o v \tau_{\mathrm{cov}} τ cov
w i = r i , y i ′ = y i ∗ , μ ′ = μ ˉ , w_i=r_i,\qquad y_i'=y_i^*,\qquad\mu'=\bar{\mu}, w i = r i , y i ′ = y i ∗ , μ ′ = μ ˉ , and letting M M M
Diag ( λ ) − ( E i [ ( y i ∗ − μ ˉ ) ( y i ∗ − μ ˉ ) T ] − 1 m ( diag ( μ ˉ ) − μ ˉ μ ˉ T ) ) , \operatorname{Diag}(\lambda)-\left(\mathbb{E}_i[(y_i^*-\bar{\mu})(y_i^*-\bar{\mu})^{\mathsf{T}}]-\frac{1}{m}\left(\operatorname{diag}(\bar{\mu})-\bar{\mu}\bar{\mu}^{\mathsf{T}}\right)\right), Diag ( λ ) − ( E i [( y i ∗ − μ ˉ ) ( y i ∗ − μ ˉ ) T ] − m 1 ( diag ( μ ˉ ) − μ ˉ μ ˉ T ) ) , gives a feasible point-mass functional. The diagonal entries of the majorizer imply λ j ≥ − 1 / ( 4 m ) \lambda_j\ge-1/(4m) λ j ≥ − 1/ ( 4 m ) λ j \lambda_j λ j ∥ M ∥ F \lVert M\rVert_F ∥ M ∥ F poly ( k ) \operatorname{poly}(k) poly ( k ) Definition 4.3 for sufficiently large R ′ R' R ′ μ ˉ \bar{\mu} μ ˉ ( α , τ , η n ) (\alpha,\tau,\eta n) ( α , τ , η n ) ν \nu ν
∥ ν − μ ˉ ∥ 1 ≤ α \lVert\nu-\bar{\mu}\rVert_1\le\alpha ∥ ν − μ ˉ ∥ 1 ≤ α also has score at most η n \eta n η n
For the third part, fix η ′ ∈ [ η , η ∗ ] \eta'\in[\eta,\eta^*] η ′ ∈ [ η , η ∗ ] j j j min { 2 j η , η ∗ } ≥ η ′ \min\{2^j\eta,\eta^*\}\ge\eta' min { 2 j η , η ∗ } ≥ η ′ min { 2 j η , η ∗ } ≤ 2 η ′ \min\{2^j\eta,\eta^*\}\le2\eta' min { 2 j η , η ∗ } ≤ 2 η ′
τ c o v = C η log ( 1 / η ) m ≤ C min { 2 j η , η ∗ } log ( 1 / min { 2 j η , η ∗ } ) m . \tau_{\mathrm{cov}}=C\frac{\eta\log(1/\eta)}{m}\le C\frac{\min\{2^j\eta,\eta^*\}\log(1/\min\{2^j\eta,\eta^*\})}{m}. τ cov = C m η log ( 1/ η ) ≤ C m min { 2 j η , η ∗ } log ( 1/ min { 2 j η , η ∗ }) . The dataset is a min { 2 j η , η ∗ } \min\{2^j\eta,\eta^*\} min { 2 j η , η ∗ } Lemma 4.11 applies to every certificate of score at most η ′ n \eta'n η ′ n
∥ L μ ′ − μ ∥ 1 ≤ O ( min { 2 j η , η ∗ } log ( 1 / min { 2 j η , η ∗ } ) m ) ≤ O ( η ′ log ( 1 / η ′ ) m ) . \lVert L\mu'-\mu\rVert_1\le O\left(\min\{2^j\eta,\eta^*\}\sqrt{\frac{\log(1/\min\{2^j\eta,\eta^*\})}{m}}\right)
\le O\left(\eta'\sqrt{\frac{\log(1/\eta')}{m}}\right). ∥ L μ ′ − μ ∥ 1 ≤ O ( min { 2 j η , η ∗ } m log ( 1/ min { 2 j η , η ∗ }) ) ≤ O ( η ′ m log ( 1/ η ′ ) ) . Choose τ \tau τ τ n ≤ η log ( 1 / η ) / m \tau n\le\eta\sqrt{\log(1/\eta)}/m τ n ≤ η log ( 1/ η ) / m η \eta η η \eta η ∥ μ ˉ − μ ∥ 1 \lVert\bar{\mu}-\mu\rVert_1 ∥ μ ˉ − μ ∥ 1 O ( η log ( 1 / η ) / m ) O(\eta\sqrt{\log(1/\eta)/m}) O ( η log ( 1/ η ) / m ) η ′ = η ∗ \eta'=\eta^* η ′ = η ∗
Lemma 4.13 (Lower Volume at Level 0.8 η ∗ n 0.8\eta^{*}n 0.8 η ∗ n Y = { y i } i ∈ [ n ] \mathcal{Y} = \{y_i\}_{i\in[n]} Y = { y i } i ∈ [ n ] η ∗ \eta^{*} η ∗ L 0 \mathcal{L}_0 L 0 ( α , τ , 0.7 η ∗ n ) (\alpha,\tau,0.7\eta^{*}n) ( α , τ , 0.7 η ∗ n )
μ ∘ = Π ( L 0 μ ′ ) . \mu^{\circ}=\Pi(\mathcal{L}_0\mu'). μ ∘ = Π ( L 0 μ ′ ) . Every point ν \nu ν ∥ ν − μ ∘ ∥ 1 ≤ α − C k τ n \lVert\nu-\mu^{\circ}\rVert_1\leq\alpha-Ck\tau n ∥ ν − μ ∘ ∥ 1 ≤ α − C k τ n 0.8 η ∗ n 0.8\eta^{*}n 0.8 η ∗ n C C C
Proof. The ball lies in Δ k − 1 + B 1 k ( τ n + α ) \Delta^{k-1}+B_1^k(\tau n+\alpha) Δ k − 1 + B 1 k ( τ n + α ) μ ∘ ∈ Δ k − 1 \mu^{\circ}\in\Delta^{k-1} μ ∘ ∈ Δ k − 1
∥ ν − L 0 μ ′ ∥ 1 ≤ ∥ ν − μ ∘ ∥ 1 + ∥ μ ∘ − L 0 μ ′ ∥ 1 ≤ α − C k τ n + C k τ n = α . \lVert\nu-\mathcal{L}_0\mu'\rVert_1\leq\lVert\nu-\mu^{\circ}\rVert_1+\lVert\mu^{\circ}-\mathcal{L}_0\mu'\rVert_1\leq\alpha-Ck\tau n+Ck\tau n=\alpha. ∥ ν − L 0 μ ′ ∥ 1 ≤ ∥ ν − μ ∘ ∥ 1 + ∥ μ ∘ − L 0 μ ′ ∥ 1 ≤ α − C k τ n + C k τ n = α . Thus this certificate gives score at most 0.7 η ∗ n 0.7\eta^{*}n 0.7 η ∗ n 0.8 η ∗ n 0.8\eta^{*}n 0.8 η ∗ n
Lemma 4.14 (Worst-Case ℓ 1 \ell_1 ℓ 1 η ∗ n \eta^{*}n η ∗ n η ∗ \eta^{*} η ∗ τ c o v ≥ 0 \tau_{\mathrm{cov}}\geq0 τ cov ≥ 0 ν 1 , ν 2 ∈ Δ k − 1 + B 1 k ( τ n + α ) \nu_1,\nu_2\in\Delta^{k-1}+B_1^k(\tau n+\alpha) ν 1 , ν 2 ∈ Δ k − 1 + B 1 k ( τ n + α ) η ∗ n \eta^{*}n η ∗ n
∥ ν 1 − ν 2 ∥ 1 ≤ O ( α + 1 m + τ c o v ) . \lVert\nu_1-\nu_2\rVert_1\leq O\left(\alpha+\sqrt{\frac{1}{m}+\tau_{\mathrm{cov}}}\right). ∥ ν 1 − ν 2 ∥ 1 ≤ O ( α + m 1 + τ cov ) . More precisely, any corresponding certificates L 1 , L 2 \mathcal{L}_1,\mathcal{L}_2 L 1 , L 2
∥ L 1 μ ′ − L 2 μ ′ ∥ 1 ≤ O ( 1 m + τ c o v ) . \lVert\mathcal{L}_1\mu'-\mathcal{L}_2\mu'\rVert_1\leq O\left(\sqrt{\frac{1}{m}+\tau_{\mathrm{cov}}}\right). ∥ L 1 μ ′ − L 2 μ ′ ∥ 1 ≤ O ( m 1 + τ cov ) . Proof. See Section A for the full proof.
Corollary 4.15 (Intermediate Volume Ratio). Let Y = { y i } i ∈ [ n ] \mathcal{Y}=\{y_i\}_{i\in[n]} Y = { y i } i ∈ [ n ] η ∗ \eta^{*} η ∗ τ c o v ≥ 0 \tau_{\mathrm{cov}}\geq0 τ cov ≥ 0 0.7 η ∗ n 0.7\eta^{*}n 0.7 η ∗ n
log ( V η ∗ ( Y ) V 0.8 η ∗ ( Y ) ) ≤ O ( k log ( 1 + 1 / m + τ c o v α ) ) . \log\left(\frac{V_{\eta^{*}}(\mathcal{Y})}{V_{0.8\eta^{*}}(\mathcal{Y})}\right)\leq O\left(k\log\left(1+\frac{\sqrt{1/m+\tau_{\mathrm{cov}}}}{\alpha}\right)\right). log ( V 0.8 η ∗ ( Y ) V η ∗ ( Y ) ) ≤ O ( k log ( 1 + α 1/ m + τ cov ) ) . Proof. Choose a certificate L 0 \mathcal{L}_0 L 0 0.7 η ∗ n 0.7\eta^{*}n 0.7 η ∗ n μ ∘ = Π ( L 0 μ ′ ) \mu^{\circ}=\Pi(\mathcal{L}_0\mu') μ ∘ = Π ( L 0 μ ′ ) C C C 0.8 η ∗ n 0.8\eta^{*}n 0.8 η ∗ n ℓ 1 \ell_1 ℓ 1 α − C k τ n \alpha-Ck\tau n α − C k τ n μ ∘ \mu^{\circ} μ ∘ ν \nu ν η ∗ n \eta^{*}n η ∗ n L \mathcal{L} L
∥ ν − μ ∘ ∥ 1 ≤ ∥ ν − L μ ′ ∥ 1 + ∥ L μ ′ − L 0 μ ′ ∥ 1 + ∥ L 0 μ ′ − μ ∘ ∥ 1 ≤ α + τ n + O ( 1 m + τ c o v ) + C k τ n . \begin{aligned}
\lVert\nu-\mu^{\circ}\rVert_1\leq\lVert\nu-\mathcal{L}\mu'\rVert_1+\lVert\mathcal{L}\mu'-\mathcal{L}_0\mu'\rVert_1+\lVert\mathcal{L}_0\mu'-\mu^{\circ}\rVert_1 \\
&\leq\alpha+\tau n+O\left(\sqrt{\frac{1}{m}+\tau_{\mathrm{cov}}}\right)+Ck\tau n.
\end{aligned} ∥ ν − μ ∘ ∥ 1 ≤ ∥ ν − L μ ′ ∥ 1 + ∥ L μ ′ − L 0 μ ′ ∥ 1 + ∥ L 0 μ ′ − μ ∘ ∥ 1 ≤ α + τ n + O ( m 1 + τ cov ) + C k τ n . Choosing τ \tau τ C k τ n ≤ min { α / 100 , 1 / ( 100 m ) } Ck\tau n\leq\min\{\alpha/100,1/(100\sqrt{m})\} C k τ n ≤ min { α /100 , 1/ ( 100 m )} τ n ≤ C k τ n \tau n\leq Ck\tau n τ n ≤ C k τ n ℓ 1 \ell_1 ℓ 1
V η ∗ ( Y ) V 0.8 η ∗ ( Y ) ≤ ( α + O ( 1 m + τ c o v ) + 2 C k τ n α − C k τ n ) k ≤ ( 1 + C ′ 1 / m + τ c o v α ) k \frac{V_{\eta^{*}}(\mathcal{Y})}{V_{0.8\eta^{*}}(\mathcal{Y})}\leq\left(\frac{\alpha+O\left(\sqrt{\frac{1}{m}+\tau_{\mathrm{cov}}}\right)+2Ck\tau n}{\alpha-Ck\tau n}\right)^k\leq\left(1+\frac{C'\sqrt{1/m+\tau_{\mathrm{cov}}}}{\alpha}\right)^k V 0.8 η ∗ ( Y ) V η ∗ ( Y ) ≤ α − C k τ n α + O ( m 1 + τ cov ) + 2 C k τ n k ≤ ( 1 + α C ′ 1/ m + τ cov ) k for an absolute constant C ′ C' C ′
Algorithm 4.1 (Private Estimation from Discrete Batches).
Input: User batches { B i } i ∈ [ n ] \{B_i\}_{i\in[n]} { B i } i ∈ [ n ] α \alpha α β \beta β ( ε , 0 ) (\varepsilon,0) ( ε , 0 ) ( ε , δ ) (\varepsilon,\delta) ( ε , δ )
Output: A private estimate μ ^ \hat{\mu} μ ^
Operations:
If ε > 1 \varepsilon> 1 ε > 1 § 4.2.3 .
If approximate DP is requested and δ > 1 − β / 2 \delta> 1-\beta/2 δ > 1 − β /2 n m nm nm δ \delta δ e 1 e_1 e 1
Replace α \alpha α α / 2 \alpha/2 α /2
Convert each observed batch to its average vector
y i : = 1 m ∑ j = 1 m e U i , j , Y : = { y i } i ∈ [ n ] . y_i := \frac{1}{m}\sum_{j=1}^{m} e_{U_{i,j}}, \qquad\mathcal{Y} := \{y_i\}_{i\in[n]}. y i := m 1 j = 1 ∑ m e U i , j , Y := { y i } i ∈ [ n ] . Define the score function S Y ( ν ) \mathcal{S}_{\mathcal{Y}}(\nu) S Y ( ν ) Definition 4.5 : the smallest trimming budget T T T ν \nu ν ( α , τ , T ) (\alpha,\tau,T) ( α , τ , T ) B T ( Y ) \mathcal{B}_T(\mathcal{Y}) B T ( Y )
Set the internal trimming parameter
η p r i v : = min { η a b s / 20 , Θ ~ ( α m ) } \eta_{\mathrm{priv}} := \min\{\eta_{\mathrm{abs}}/20,\widetilde{\Theta}(\alpha\sqrt{m})\} η priv := min { η abs /20 , Θ ( α m )} and instantiate the score system with
τ c o v : = C η p r i v log ( 1 / η p r i v ) m . \tau_{\mathrm{cov}} := C\frac{\eta_{\mathrm{priv}}\log(1/\eta_{\mathrm{priv}})}{m}. τ cov := C m η priv log ( 1/ η priv ) . Run the reduction from Theorem 2.18 (for pure DP) or Theorem 2.19 (for approximate DP) on the domain Θ = Δ k − 1 + B 1 k ( τ n + α ) \Theta=\Delta^{k-1}+B_1^k(\tau n+\alpha) Θ = Δ k − 1 + B 1 k ( τ n + α ) S Y \mathcal{S}_{\mathcal{Y}} S Y
Output a nearest point in Δ k − 1 \Delta^{k-1} Δ k − 1 ℓ 1 \ell_1 ℓ 1
Algorithm 4.1 summarizes the private estimator obtained from the robustness-to-privacy reduction.
For the proofs of Theorems 4.1 and 4.2 , fix η ∗ = η a b s / 2 \eta^*=\eta_{\mathrm{abs}}/2 η ∗ = η abs /2
η p r i v : = min { η a b s / 20 , Θ ~ ( α m ) } , τ c o v : = C η p r i v log ( 1 / η p r i v ) m , \eta_{\mathrm{priv}} := \min\{\eta_{\mathrm{abs}}/20,\widetilde{\Theta}(\alpha\sqrt{m})\}, \qquad\tau_{\mathrm{cov}} := C\frac{\eta_{\mathrm{priv}}\log(1/\eta_{\mathrm{priv}})}{m}, η priv := min { η abs /20 , Θ ( α m )} , τ cov := C m η priv log ( 1/ η priv ) , so that 10 η p r i v ≤ η ∗ 10\eta_{\mathrm{priv}}\leq\eta^* 10 η priv ≤ η ∗ Θ ~ ( α m ) \widetilde{\Theta}(\alpha\sqrt{m}) Θ ( α m ) η p r i v < η a b s / 20 \eta_{\mathrm{priv}}<\eta_{\mathrm{abs}}/20 η priv < η abs /20
O ( η p r i v log ( 1 / η p r i v ) m ) ≤ α / 10. O\left(\eta_{\mathrm{priv}}\sqrt{\frac{\log(1/\eta_{\mathrm{priv}})}{m}}\right)\leq\alpha/10. O ( η priv m log ( 1/ η priv ) ) ≤ α /10. Equivalently, after fixing the hidden constant once and for all, there is an absolute split threshold such that:
if α m \alpha\sqrt{m} α m η p r i v = Θ ~ ( α m ) \eta_{\mathrm{priv}} = \widetilde{\Theta}(\alpha\sqrt{m}) η priv = Θ ( α m )
if α m \alpha\sqrt{m} α m η p r i v = η a b s / 20 \eta_{\mathrm{priv}} = \eta_{\mathrm{abs}}/20 η priv = η abs /20 2 η p r i v < 1 / 10 2\eta_{\mathrm{priv}} < 1/10 2 η priv < 1/10
O ( η p r i v log ( 1 / η p r i v ) m ) = O ( 1 m ) ≤ α / 10. O\left(\eta_{\mathrm{priv}}\sqrt{\frac{\log(1/\eta_{\mathrm{priv}})}{m}}\right) = O\left(\frac{1}{\sqrt{m}}\right) \le\alpha/10. O ( η priv m log ( 1/ η priv ) ) = O ( m 1 ) ≤ α /10. Proof of the Pure-DP Theorem
First suppose 0 < ε ≤ 1 0 < \varepsilon\le1 0 < ε ≤ 1 β / 3 \beta/3 β /3 2 η p r i v 2\eta_{\mathrm{priv}} 2 η priv Lemma 4.12 , and the privacy mechanism; this changes only absolute constants and logarithmic factors. We apply Theorem 2.18 , using the score function defined in Definition 4.5 , with Θ = Δ k − 1 + B 1 k ( τ n + α ) \Theta= \Delta^{k-1} + B_1^k(\tau n + \alpha) Θ = Δ k − 1 + B 1 k ( τ n + α ) η p r i v \eta_{\mathrm{priv}} η priv α / ( 2 k ) \alpha/(2\sqrt{k}) α / ( 2 k ) ℓ 1 \ell_1 ℓ 1 Theorem 2.18 follow from Lemmas 4.6 to 4.10. It remains to verify the low-score witness, bound the volume term, and invoke soundness.
Both the good-sample event from Corollary 3.3 at parameter 2 η p r i v 2\eta_{\mathrm{priv}} 2 η priv Lemma 4.12 at parameter η p r i v \eta_{\mathrm{priv}} η priv
n ≥ Ω ( k + log ( 1 / β ) η p r i v 2 log ( 1 / η p r i v ) ) . (13) n \ge\Omega\left(\frac{k+\log(1/\beta)}{\eta_{\mathrm{priv}}^2\log(1/\eta_{\mathrm{priv}})}\right). \tag*{(13)} n ≥ Ω ( η priv 2 log ( 1/ η priv ) k + log ( 1/ β ) ) . (13) In the below-threshold regime η p r i v = Θ ~ ( α m ) \eta_{\mathrm{priv}} = \widetilde{\Theta}(\alpha\sqrt{m}) η priv = Θ ( α m ) (13) becomes
O ~ ( k + log ( 1 / β ) α 2 m ) . \widetilde{O}\left(\frac{k+\log(1/\beta)}{\alpha^2m}\right). O ( α 2 m k + log ( 1/ β ) ) . In the above-threshold regime η p r i v = η a b s / 20 \eta_{\mathrm{priv}} = \eta_{\mathrm{abs}}/20 η priv = η abs /20 O ( k + log ( 1 / β ) ) O(k+\log(1/\beta)) O ( k + log ( 1/ β )) O ~ ( ( k + log ( 1 / β ) ) / ε ) \widetilde{O}((k+\log(1/\beta))/\varepsilon) O (( k + log ( 1/ β )) / ε ) ε ≤ 1 \varepsilon\le1 ε ≤ 1
For the volume term, let μ 0 \mu_0 μ 0 Lemma 4.12 at parameter η p r i v \eta_{\mathrm{priv}} η priv μ 0 \mu_0 μ 0 ℓ 1 \ell_1 ℓ 1 α \alpha α μ 0 \mu_0 μ 0 η p r i v n \eta_{\mathrm{priv}}n η priv n Lemma 2.20 ; the α \alpha α ℓ 1 \ell_1 ℓ 1 μ 0 \mu_0 μ 0 α / ( 2 k ) \alpha/(2\sqrt{k}) α / ( 2 k ) (5) into η ′ ∈ [ η p r i v , η ∗ ] \eta' \in[\eta_{\mathrm{priv}},\eta^*] η ′ ∈ [ η priv , η ∗ ] η ′ ∈ [ η ∗ , 1 ] \eta' \in[\eta^*,1] η ′ ∈ [ η ∗ , 1 ] Lemma 4.12 gives
log ( V η ′ ( Y ) V η p r i v ( Y ) ) ≤ k log ( 1 + C η ′ log ( 1 / η ′ ) α m ) ≤ k log ( 1 + C ′ η ′ η p r i v ) . \log\left(\frac{V_{\eta'}(\mathcal{Y})}{V_{\eta_{\mathrm{priv}}}(\mathcal{Y})}\right)
\le k\log\left(1+C\frac{\eta'\sqrt{\log(1/\eta')}}{\alpha\sqrt{m}}\right)
\le k\log\left(1+C'\frac{\eta'}{\eta_{\mathrm{priv}}}\right). log ( V η priv ( Y ) V η ′ ( Y ) ) ≤ k log ( 1 + C α m η ′ log ( 1/ η ′ ) ) ≤ k log ( 1 + C ′ η priv η ′ ) . where the last step uses η p r i v log ( 1 / η p r i v ) = O ( α m ) \eta_{\mathrm{priv}}\sqrt{\log(1/\eta_{\mathrm{priv}})} = O(\alpha\sqrt{m}) η priv log ( 1/ η priv ) = O ( α m ) sup u ≥ 1 log ( 1 + C ′ u ) / u = O ( 1 ) \sup_{u\ge1}\log(1+C'u)/u = O(1) sup u ≥ 1 log ( 1 + C ′ u ) / u = O ( 1 )
max η p r i v ≤ η ′ ≤ η ∗ log ( V η ′ ( Y ) / V η p r i v ( Y ) ) + log ( 2 / β ) ε η ′ ≤ O ( k + log ( 2 / β ) ε η p r i v ) . (14) \max_{\eta_{\mathrm{priv}}\le\eta'\le\eta^*}
\frac{\log(V_{\eta'}(\mathcal{Y})/V_{\eta_{\mathrm{priv}}}(\mathcal{Y}))+\log(2/\beta)}
{\varepsilon\eta'}
\le O\left(\frac{k+\log(2/\beta)}{\varepsilon\eta_{\mathrm{priv}}}\right). \tag*{(14)} η priv ≤ η ′ ≤ η ∗ max ε η ′ log ( V η ′ ( Y ) / V η priv ( Y )) + log ( 2/ β ) ≤ O ( ε η priv k + log ( 2/ β ) ) . (14) The right-hand side of (14) is O ~ ( ( k + log ( 2 / β ) ) / ( α m ε ) ) \widetilde{O}((k+\log(2/\beta))/(\alpha\sqrt{m}\varepsilon)) O (( k + log ( 2/ β )) / ( α m ε )) O ( ( k + log ( 2 / β ) ) / ε ) O((k+\log(2/\beta))/\varepsilon) O (( k + log ( 2/ β )) / ε ) η ′ ≥ η ∗ \eta' \ge\eta^* η ′ ≥ η ∗ V η ′ V_{\eta'} V η ′ V 1 V_1 V 1
log ( V 1 ( Y ) / V η p r i v ( Y ) ) + log ( 2 / β ) ε ⋅ η ∗ ≤ O ~ ( k + log ( 1 / β ) ε ) . \frac{\log(V_1(\mathcal{Y})/V_{\eta_{\mathrm{priv}}}(\mathcal{Y}))+\log(2/\beta)}
{\varepsilon\cdot\eta^*}
\le\widetilde{O}\left(\frac{k+\log(1/\beta)}{\varepsilon}\right). ε ⋅ η ∗ log ( V 1 ( Y ) / V η priv ( Y )) + log ( 2/ β ) ≤ O ( ε k + log ( 1/ β ) ) . Therefore, as long as
n ≥ O ~ ( k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α m ε + k + log ( 1 / β ) ε ) , n \ge\widetilde{O}\left(\frac{k+\log(1/\beta)}{\alpha^{2}m}+\frac{k+\log(1/\beta)}{\alpha\sqrt{m\varepsilon}}+\frac{k+\log(1/\beta)}{\varepsilon}\right), n ≥ O ( α 2 m k + log ( 1/ β ) + α m ε k + log ( 1/ β ) + ε k + log ( 1/ β ) ) , the sampler of Theorem 2.18 , with utility analyzed in Lemma 2.20 , returns a point ν \nu ν 2 η p r i v n 2\eta_{\mathrm{priv}}n 2 η priv n 1 − β 1-\beta 1 − β
Let L \mathcal{L} L ( α , τ , 2 η p r i v n ) (\alpha,\tau,2\eta_{\mathrm{priv}}n) ( α , τ , 2 η priv n ) ( Y , ν ) (\mathcal{Y},\nu) ( Y , ν ) 2 η p r i v ≤ η a b s / 10 < η a b s 2\eta_{\mathrm{priv}}\le\eta_{\mathrm{abs}}/10<\eta_{\mathrm{abs}} 2 η priv ≤ η abs /10 < η abs Corollary 3.3 and Lemma 4.11 with parameter η 0 = 2 η p r i v \eta_0=2\eta_{\mathrm{priv}} η 0 = 2 η priv
∥ L μ ′ − μ ∥ 1 ≤ O ( η p r i v log ( 1 / η p r i v ) m ) ≤ α / 10. \lVert\mathcal{L}\mu^{\prime}-\mu\rVert_1\le O\left(\eta_{\mathrm{priv}}\sqrt{\frac{\log(1/\eta_{\mathrm{priv}})}{m}}\right)\le\alpha/10. ∥ L μ ′ − μ ∥ 1 ≤ O ( η priv m log ( 1/ η priv ) ) ≤ α /10. Using the closeness constraint and taking τ \tau τ
∥ ν − μ ∥ 1 ≤ ∥ ν − L μ ′ ∥ 1 + ∥ L μ ′ − μ ∥ 1 ≤ α + 2 τ η p r i v n + α / 10 ≤ 6 α / 5. \lVert\nu-\mu\rVert_1\le\lVert\nu-\mathcal{L}\mu^{\prime}\rVert_1+\lVert\mathcal{L}\mu^{\prime}-\mu\rVert_1\le\alpha+2\tau\eta_{\mathrm{priv}}n+\alpha/10\le6\alpha/5. ∥ ν − μ ∥ 1 ≤ ∥ ν − L μ ′ ∥ 1 + ∥ L μ ′ − μ ∥ 1 ≤ α + 2 τ η priv n + α /10 ≤ 6 α /5. If μ ^ \hat{\mu} μ ^ ν \nu ν Δ k − 1 \Delta^{k-1} Δ k − 1 ℓ 1 \ell_1 ℓ 1
∥ μ ^ − μ ∥ T V = 1 2 ∥ μ ^ − μ ∥ 1 ≤ ∥ ν − μ ∥ 1 ≤ 6 α / 5. (15) \lVert\hat{\mu}-\mu\rVert_{\mathrm{TV}}=\frac{1}{2}\lVert\hat{\mu}-\mu\rVert_1\le\lVert\nu-\mu\rVert_1\le6\alpha/5. \tag*{(15)} ∥ μ ^ − μ ∥ TV = 2 1 ∥ μ ^ − μ ∥ 1 ≤ ∥ ν − μ ∥ 1 ≤ 6 α /5. (15) Running the construction with internal accuracy α / 2 \alpha/2 α /2 α \alpha α ε > 1 \varepsilon>1 ε > 1 ε ≤ m \varepsilon\le m ε ≤ m ⌈ ε ⌉ \lceil\varepsilon\rceil ⌈ ε ⌉ ⌊ m / ⌈ ε ⌉ ⌋ \lfloor m/\lceil\varepsilon\rceil\rfloor ⌊ m / ⌈ ε ⌉⌋ n ⌈ ε ⌉ n\lceil\varepsilon\rceil n ⌈ ε ⌉ ε / ⌈ ε ⌉ ≤ 1 \varepsilon/\lceil\varepsilon\rceil\le1 ε / ⌈ ε ⌉ ≤ 1 ⌈ ε ⌉ \lceil\varepsilon\rceil ⌈ ε ⌉ ε \varepsilon ε ⌊ m / ⌈ ε ⌉ ⌋ ≥ m / ( 2 ⌈ ε ⌉ ) \lfloor m/\lceil\varepsilon\rceil\rfloor\ge m/(2\lceil\varepsilon\rceil) ⌊ m / ⌈ ε ⌉⌋ ≥ m / ( 2 ⌈ ε ⌉) ⌈ ε ⌉ ≤ 2 ε \lceil\varepsilon\rceil\le2\varepsilon ⌈ ε ⌉ ≤ 2 ε
O ~ ( k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α m ε + k + log ( 1 / β ) ε ) . \widetilde{O}\left(\frac{k+\log(1/\beta)}{\alpha^{2}m}+\frac{k+\log(1/\beta)}{\alpha\sqrt{m\varepsilon}}+\frac{k+\log(1/\beta)}{\varepsilon}\right). O ( α 2 m k + log ( 1/ β ) + α m ε k + log ( 1/ β ) + ε k + log ( 1/ β ) ) . By the arithmetic–geometric mean inequality, the middle term is at most the sum of the other two. If ε > m \varepsilon>m ε > m ℓ 1 \ell_1 ℓ 1 2 / n 2/n 2/ n
O ~ ( k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α ε ) \widetilde{O}\left(\frac{k+\log(1/\beta)}{\alpha^{2}m}+\frac{k+\log(1/\beta)}{\alpha\varepsilon}\right) O ( α 2 m k + log ( 1/ β ) + α ε k + log ( 1/ β ) ) users; its second term is absorbed by the first because ε > m \varepsilon>m ε > m α < 1 \alpha<1 α < 1 Theorem 4.1 .
Proof of the Approximate-DP Theorem
If δ > 1 − β / 2 \delta>1-\beta/2 δ > 1 − β /2 δ \delta δ ( 0 , δ ) (0,\delta) ( 0 , δ ) n = O ( ( k + log ( 1 / β ) ) / ( α 2 m ) ) n=O((k+\log(1/\beta))/(\alpha^{2}m)) n = O (( k + log ( 1/ β )) / ( α 2 m )) α \alpha α 1 − β / 2 1-\beta/2 1 − β /2 δ ( 1 − β / 2 ) > ( 1 − β / 2 ) 2 ≥ 1 − β \delta(1-\beta/2)>(1-\beta/2)^2\ge1-\beta δ ( 1 − β /2 ) > ( 1 − β /2 ) 2 ≥ 1 − β δ ≤ 1 − β / 2 \delta\le1-\beta/2 δ ≤ 1 − β /2 β δ ≤ 1 / 2 \beta\delta\le1/2 β δ ≤ 1/2 log ( 2 / ( β δ ) ) \log(2/(\beta\delta)) log ( 2/ ( β δ )) log ( 1 / δ ) + log ( 1 / β ) \log(1/\delta)+\log(1/\beta) log ( 1/ δ ) + log ( 1/ β ) 0 < ε ≤ 1 0 < \varepsilon\le1 0 < ε ≤ 1 Theorem 2.19 with the score, internal threshold η p r i v \eta_{\mathrm{priv}} η priv α / ( 2 k ) \alpha/(2\sqrt{k}) α / ( 2 k ) § 4.2.3 . That section verifies the oracle properties, and item 1 of Lemma 4.12 supplies the low-score witness.
To obtain the good-sample event used in § 4.2.3 , we need
n ≥ O ~ ( k + log ( 1 / β ) + k + log ( 1 / β ) α 2 m ) . (16) n \ge\widetilde{O}\left(k+\log(1/\beta)+\frac{k+\log(1/\beta)}{\alpha^2m}\right). \tag*{(16)} n ≥ O ( k + log ( 1/ β ) + α 2 m k + log ( 1/ β ) ) . (16) because η p r i v = min { η a b s / 20 , Θ ~ ( α m ) } \eta_{\mathrm{priv}}=\min\{\eta_{\mathrm{abs}}/20,\widetilde{\Theta}(\alpha\sqrt{m})\} η priv = min { η abs /20 , Θ ( α m )} (5) on this event, the balls in items 1 and 2 of Lemma 4.12 give
log ( V η ′ ( Y ) V η p r i v ( Y ) ) ≤ O ( k log ( 1 + 1 α m ) ) (17) \log\left(\frac{V_{\eta'}(\mathcal{Y})}{V_{\eta_{\mathrm{priv}}}(\mathcal{Y})}\right)\le O\left(k\log\left(1+\frac{1}{\alpha\sqrt{m}}\right)\right) \tag*{(17)} log ( V η priv ( Y ) V η ′ ( Y ) ) ≤ O ( k log ( 1 + α m 1 ) ) (17) (η p r i v ≤ η ′ ≤ η ∗ \eta_{\mathrm{priv}}\le\eta'\le\eta^* η priv ≤ η ′ ≤ η ∗
Apply Lemma 2.20 , using the inner ball in item 1 of Lemma 4.12 . Below the fixed split threshold, (14) gives O ( ( k + log ( 2 / β ) ) / ( ε η p r i v ) ) O((k+\log(2/\beta))/(\varepsilon\eta_{\mathrm{priv}})) O (( k + log ( 2/ β )) / ( ε η priv )) η p r i v \eta_{\mathrm{priv}} η priv log ( 1 + 1 / ( α m ) ) ≤ 1 / ( α m ) \log(1+1/(\alpha\sqrt{m}))\le1/(\alpha\sqrt{m}) log ( 1 + 1/ ( α m )) ≤ 1/ ( α m ) (17) bounds the utility requirement by
O ~ ( k α m ε + log ( 2 / β ) ε ) . \widetilde{O}\left(\frac{k}{\alpha\sqrt{m}\varepsilon}+\frac{\log(2/\beta)}{\varepsilon}\right). O ( α m ε k + ε log ( 2/ β ) ) . Thus (5) is implied by
n ≥ O ~ ( k + log ( 1 / β ) α m ε + log ( 2 / β ) ε ) . n\ge\widetilde{O}\left(\frac{k+\log(1/\beta)}{\alpha\sqrt{m}\varepsilon}+\frac{\log(2/\beta)}{\varepsilon}\right). n ≥ O ( α m ε k + log ( 1/ β ) + ε log ( 2/ β ) ) . For the worst-case volume condition (3) , choose η a b s \eta_{\mathrm{abs}} η abs m τ c o v ≤ 1 / 2 m\tau_{\mathrm{cov}}\le1/2 m τ cov ≤ 1/2
n ≥ O ( log ( 1 / δ ) + k log ( 1 + 1 / ( α m ) ) ε ) n\ge O\left(\frac{\log(1/\delta)+k\log(1+1/(\alpha\sqrt{m}))}{\varepsilon}\right) n ≥ O ( ε log ( 1/ δ ) + k log ( 1 + 1/ ( α m )) ) as a sufficient condition. Since log ( 1 + 1 / ( α m ) ) ≤ 1 / ( α m ) \log(1+1/(\alpha\sqrt{m}))\le1/(\alpha\sqrt{m}) log ( 1 + 1/ ( α m )) ≤ 1/ ( α m ) Theorem 4.2 . Finally, k + ( log ( 1 / δ ) + log ( 1 / β ) ) / ε ≥ k + log ( 1 / β ) k+(\log(1/\delta)+\log(1/\beta))/\varepsilon\ge k+\log(1/\beta) k + ( log ( 1/ δ ) + log ( 1/ β )) / ε ≥ k + log ( 1/ β ) ε ≤ 1 \varepsilon\le1 ε ≤ 1 (16) .
Allocating failure probability β / 3 \beta/3 β /3 ν \nu ν 2 η p r i v n 2\eta_{\mathrm{priv}}n 2 η priv n 1 − β 1-\beta 1 − β Lemma 4.11 at 2 η p r i v 2\eta_{\mathrm{priv}} 2 η priv (15) gives error at most α \alpha α α / 2 \alpha/2 α /2
If ε > 1 \varepsilon>1 ε > 1 Theorem 4.1 . Its last sample term satisfies
k + log ( 1 / β ) ε ≤ k + log ( 1 / δ ) + log ( 1 / β ) ε , \frac{k+\log(1/\beta)}{\varepsilon}\le k+\frac{\log(1/\delta)+\log(1/\beta)}{\varepsilon}, ε k + log ( 1/ β ) ≤ k + ε log ( 1/ δ ) + log ( 1/ β ) , so it also satisfies the sample bound in Theorem 4.2 and is ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) Theorem 4.2 .
Remark 4.16 (Explicit logarithmic factors). For 0 < ε ≤ 1 0<\varepsilon\le1 0 < ε ≤ 1 Theorems 4.1 and 4.2 give sufficient sample bounds with ordinary O O O
n = O ( k + log ( 2 / β ) α 2 m + ( k + log ( 2 / β ) ) log ( e + 1 / ( α m ) ) α m ε + k log ( e / α ) + log ( 2 / β ) ε ) . n=O\left(\frac{k+\log(2/\beta)}{\alpha^2m}+\frac{(k+\log(2/\beta))\sqrt{\log(e+1/(\alpha\sqrt{m}))}}{\alpha\sqrt{m}\varepsilon}+\frac{k\log(e/\alpha)+\log(2/\beta)}{\varepsilon}\right). n = O ( α 2 m k + log ( 2/ β ) + α m ε ( k + log ( 2/ β )) log ( e + 1/ ( α m )) + ε k log ( e / α ) + log ( 2/ β ) ) . For approximate DP, the bound is
n = O ( k + k + log ( 2 / β ) α 2 m + ( k + log ( 2 / β ) ) log ( e + 1 / ( α m ) ) α m ε + log ( 1 / δ ) + log ( 2 / β ) ε ) . n = O\left(k + \frac{k+\log(2/\beta)}{\alpha^{2}m} + \frac{(k+\log(2/\beta))\sqrt{\log(e+1/(\alpha\sqrt{m}))}}{\alpha\sqrt{m}\varepsilon} + \frac{\log(1/\delta)+\log(2/\beta)}{\varepsilon}\right). n = O ( k + α 2 m k + log ( 2/ β ) + α m ε ( k + log ( 2/ β )) log ( e + 1/ ( α m )) + ε log ( 1/ δ ) + log ( 2/ β ) ) . Indeed, choose
η p r i v = c min { 1 , α m log ( e + 1 / ( α m ) ) } . \eta_{\mathrm{priv}}=c\min\left\{1,\frac{\alpha\sqrt{m}}{\sqrt{\log(e+1/(\alpha\sqrt{m}))}}\right\}. η priv = c min { 1 , log ( e + 1/ ( α m )) α m } . with c > 0 c>0 c > 0 η p r i v 2 log ( 1 / η p r i v ) = Θ ( min { 1 , α 2 m } ) \eta_{\mathrm{priv}}^{2}\log(1/\eta_{\mathrm{priv}})=\Theta(\min\{1,\alpha^{2}m\}) η priv 2 log ( 1/ η priv ) = Θ ( min { 1 , α 2 m }) (13) requires O ( k + log ( 2 / β ) + ( k + log ( 2 / β ) ) / ( α 2 m ) ) O(k+\log(2/\beta)+(k+\log(2/\beta))/(\alpha^{2}m)) O ( k + log ( 2/ β ) + ( k + log ( 2/ β )) / ( α 2 m )) (14) requires
O ( k + log ( 2 / β ) ε η p r i v ) = O ( ( k + log ( 2 / β ) ) log ( e + 1 / ( α m ) ) α m ε ) . O\left(\frac{k+\log(2/\beta)}{\varepsilon\eta_{\mathrm{priv}}}\right)
=O\left(\frac{(k+\log(2/\beta))\sqrt{\log(e+1/(\alpha\sqrt{m}))}}{\alpha\sqrt{m}\varepsilon}\right). O ( ε η priv k + log ( 2/ β ) ) = O ( α m ε ( k + log ( 2/ β )) log ( e + 1/ ( α m )) ) . Above the split threshold, (17) gives O ( k / ( α m ε ) + log ( 2 / β ) / ε ) O(k/(\alpha\sqrt{m}\varepsilon)+\log(2/\beta)/\varepsilon) O ( k / ( α m ε ) + log ( 2/ β ) / ε ) O ( ( k + log ( 2 / β ) ) / ε ) O((k+\log(2/\beta))/\varepsilon) O (( k + log ( 2/ β )) / ε ) ( k log ( e / α ) + log ( 2 / β ) ) / ε (k\log(e/\alpha)+\log(2/\beta))/\varepsilon ( k log ( e / α ) + log ( 2/ β )) / ε ℓ 1 \ell_{1} ℓ 1 log ( V 1 / V η p r i v ) = O ( k log ( e / α ) ) \log(V_{1}/V_{\eta_{\mathrm{priv}}})=O(k\log(e/\alpha)) log ( V 1 / V η priv ) = O ( k log ( e / α )) (3) contributes
O ( log ( 1 / δ ) + k log ( 1 + 1 / ( α m ) ) ε ) . O\left(\frac{\log(1/\delta)+k\log(1+1/(\alpha\sqrt{m}))}{\varepsilon}\right). O ( ε log ( 1/ δ ) + k log ( 1 + 1/ ( α m )) ) . Its dimension-dependent term is absorbed by
( k + log ( 2 / β ) ) log ( e + 1 / ( α m ) ) α m ε . \frac{(k+\log(2/\beta))\sqrt{\log(e+1/(\alpha\sqrt{m}))}}{\alpha\sqrt{m}\varepsilon}. α m ε ( k + log ( 2/ β )) log ( e + 1/ ( α m )) . Thus the privacy term for small α m \alpha\sqrt{m} α m
discarding any remaining samples. The number of blocks per user is at most m m m α ε ≤ 1 \alpha\varepsilon\le1 α ε ≤ 1 Theorem 4.2 to the n ⌈ ( α m ε ) 2 / 3 ⌉ n\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil n ⌈( α m ε ) 2/3 ⌉
( ε ⌈ ( α m ε ) 2 / 3 ⌉ , δ e ε ⌈ ( α m ε ) 2 / 3 ⌉ ) . \left(\frac{\varepsilon}{\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil},\frac{\delta}{e^{\varepsilon}\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil}\right). ( ⌈( α m ε ) 2/3 ⌉ ε , e ε ⌈( α m ε ) 2/3 ⌉ δ ) . Changing one original user changes at most ⌈ ( α m ε ) 2 / 3 ⌉ \lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil ⌈( α m ε ) 2/3 ⌉ ( ε , δ ) (\varepsilon,\delta) ( ε , δ )
δ e ε ⌈ ( α m ε ) 2 / 3 ⌉ ∑ j = 0 ⌈ ( α m ε ) 2 / 3 ⌉ − 1 exp ( j ε ⌈ ( α m ε ) 2 / 3 ⌉ ) ≤ δ . \frac{\delta}{e^{\varepsilon}\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil}\sum_{j=0}^{\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil-1}\exp\left(\frac{j\varepsilon}{\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil}\right)\le\delta. e ε ⌈( α m ε ) 2/3 ⌉ δ j = 0 ∑ ⌈( α m ε ) 2/3 ⌉ − 1 exp ( ⌈( α m ε ) 2/3 ⌉ j ε ) ≤ δ . Each block has size at least m / ( 2 ⌈ ( α m ε ) 2 / 3 ⌉ ) m/(2\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil) m / ( 2 ⌈( α m ε ) 2/3 ⌉) ⌈ ( α m ε ) 2 / 3 ⌉ \lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil ⌈( α m ε ) 2/3 ⌉
n = O ~ ( k ⌈ ( α m ε ) 2 / 3 ⌉ + k + log ( 1 / β ) α 2 m + ( k + log ( 1 / β ) ) ⌈ ( α m ε ) 2 / 3 ⌉ α m ε + log ( 1 / δ ) + log ( 1 / β ) ε + log ⌈ ( α m ε ) 2 / 3 ⌉ ε ) . n=\widetilde{O}\left(\frac{k}{\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil}+\frac{k+\log(1/\beta)}{\alpha^{2}m}+\frac{(k+\log(1/\beta))\sqrt{\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil}}{\alpha\sqrt{m}\varepsilon}+\frac{\log(1/\delta)+\log(1/\beta)}{\varepsilon}+\frac{\log\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil}{\varepsilon}\right). n = O ( ⌈( α m ε ) 2/3 ⌉ k + α 2 m k + log ( 1/ β ) + α m ε ( k + log ( 1/ β )) ⌈( α m ε ) 2/3 ⌉ + ε log ( 1/ δ ) + log ( 1/ β ) + ε log ⌈( α m ε ) 2/3 ⌉ ) . Since α m ε > 1 \alpha\sqrt{m}\varepsilon>1 α m ε > 1 Theorem 4.2 applied to the original batches proves the claim. Here O ~ \widetilde{O} O log ⌈ ( α m ε ) 2 / 3 ⌉ \log\lceil(\alpha\sqrt{m}\varepsilon)^{2/3}\rceil log ⌈( α m ε ) 2/3 ⌉ δ \delta δ β \beta β α m ε > 1 \alpha\sqrt{m}\varepsilon>1 α m ε > 1 k k k k / ( α m ε ) 2 / 3 k/(\alpha\sqrt{m}\varepsilon)^{2/3} k / ( α m ε ) 2/3 Corollary 4.18 : splitting reduces the batch size that controls the corruption error.
For clean data, we may also choose between this estimator and that of Esfandiari et al. [32 ] using only the public parameters, obtaining the smaller of the two sample bounds without any privacy composition. In particular, for constant β \beta β m ≥ C k 2 / α 2 m\ge Ck^{2}/\alpha^{2} m ≥ C k 2 / α 2 n = O ( log ( 1 / δ ) / ε ) n=O(\log(1/\delta)/\varepsilon) n = O ( log ( 1/ δ ) / ε ) E ∥ y i − μ ∥ 2 2 ≤ 1 / m \mathbb{E}\lVert y_i-\mu\rVert_{2}^{2}\le1/m E ∥ y i − μ ∥ 2 2 ≤ 1/ m O ( 1 / m ) O(1/\sqrt{m}) O ( 1/ m ) μ \mu μ ℓ 2 \ell_{2} ℓ 2 [32 Theorem 3.3] , followed by Euclidean projection onto the simplex, gives ℓ 2 \ell_{2} ℓ 2 O ( k / m ) O(\sqrt{k/m}) O ( k / m ) O ( k / m ) O(k/\sqrt{m}) O ( k / m ) 1 / δ 1/\delta 1/ δ [32 Theorem 4.1] .
Corollary 4.18 (Robustness of the Private Estimators). There are absolute constants c , C > 0 c,C>0 c , C > 0 0 < ε ≤ 1 0<\varepsilon\le1 0 < ε ≤ 1 0 ≤ η ≤ c 0\le\eta\le c 0 ≤ η ≤ c α , β ∈ ( 0 , 1 ) \alpha,\beta\in(0,1) α , β ∈ ( 0 , 1 ) i i i m m m μ i \mu_i μ i ∥ μ i − μ ∥ T V ≤ ω \lVert\mu_i-\mu\rVert_{\mathrm{TV}}\le\omega ∥ μ i − μ ∥ TV ≤ ω i i i ω ∈ [ 0 , 1 ] \omega\in[0,1] ω ∈ [ 0 , 1 ] ε \varepsilon ε η \eta η μ ^ ∈ Δ k − 1 \widehat{\mu}\in\Delta^{k-1} μ ∈ Δ k − 1 ∥ μ ^ − μ ∥ T V ≤ α \lVert\widehat{\mu}-\mu\rVert_{\mathrm{TV}}\le\alpha ∥ μ − μ ∥ TV ≤ α 1 − β 1-\beta 1 − β
α ≥ C ( η log ( 1 / η ) m + ω ) \alpha\ge C\left(\eta\sqrt{\frac{\log(1/\eta)}{m}}+\omega\right) α ≥ C ( η m log ( 1/ η ) + ω ) and
n ≥ O ~ ( k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α m ε + k + log ( 1 / β ) ε ) . n \ge\widetilde{O}\left(\frac{k+\log(1/\beta)}{\alpha^2m}+\frac{k+\log(1/\beta)}{\alpha\sqrt{m}\,\varepsilon}+\frac{k+\log(1/\beta)}{\varepsilon}\right). n ≥ O ( α 2 m k + log ( 1/ β ) + α m ε k + log ( 1/ β ) + ε k + log ( 1/ β ) ) . The term involving η \eta η η = 0 \eta=0 η = 0 δ ∈ ( 0 , 1 ) \delta\in(0,1) δ ∈ ( 0 , 1 ) ( ε , δ ) (\varepsilon,\delta) ( ε , δ )
n ≥ O ~ ( k + k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α m ε + k ω / α + log ( 1 / δ ) + log ( 1 / β ) ε ) . n \ge\widetilde{O}\left(k+\frac{k+\log(1/\beta)}{\alpha^2m}+\frac{k+\log(1/\beta)}{\alpha\sqrt{m}\,\varepsilon}+\frac{k\omega/\alpha+\log(1/\delta)+\log(1/\beta)}{\varepsilon}\right). n ≥ O ( k + α 2 m k + log ( 1/ β ) + α m ε k + log ( 1/ β ) + ε k ω / α + log ( 1/ δ ) + log ( 1/ β ) ) . This recovers the sample bound of Theorem 4.2 when ω = O ( 1 / m + α ε ) \omega=O(1/\sqrt{m}+\alpha\varepsilon) ω = O ( 1/ m + α ε ) α m ≤ 1 \alpha\sqrt{m}\le1 α m ≤ 1 ε \varepsilon ε
Proof. We first prove the pure-DP claim. Use the score construction from § 4 , with internal accuracy c 1 α c_1\alpha c 1 α c 1 > 0 c_1>0 c 1 > 0 η p r i v ≤ η a b s / 20 \eta_{\mathrm{priv}}\le\eta_{\mathrm{abs}}/20 η priv ≤ η abs /20 § 4.2.3 , so that
η p r i v = min { η a b s / 20 , Θ ~ ( α m ) } , O ( η p r i v log ( 1 / η p r i v ) m ) ≤ α / 10. \eta_{\mathrm{priv}}=\min\left\{\eta_{\mathrm{abs}}/20,\widetilde{\Theta}(\alpha\sqrt{m})\right\},\qquad O\left(\eta_{\mathrm{priv}}\sqrt{\frac{\log(1/\eta_{\mathrm{priv}})}{m}}\right)\le\alpha/10. η priv = min { η abs /20 , Θ ( α m ) } , O ( η priv m log ( 1/ η priv ) ) ≤ α /10. The constants C C C c c c η ≤ c \eta\le c η ≤ c η ≤ η p r i v \eta\le\eta_{\mathrm{priv}} η ≤ η priv
τ c o v = C 0 ( η p r i v log ( 1 / η p r i v ) m + ω m + ω 2 ) , (18) \tau_{\mathrm{cov}}=C_0\left(\frac{\eta_{\mathrm{priv}}\log(1/\eta_{\mathrm{priv}})}{m}+\frac{\omega}{m}+\omega^2\right), \tag*{(18)} τ cov = C 0 ( m η priv log ( 1/ η priv ) + m ω + ω 2 ) , (18) where C 0 C_0 C 0 Lemmas 4.7 to 4.10 apply without modification. The witness bounds remain polynomial because ω ≤ 1 \omega\le1 ω ≤ 1
We verify the statistical properties needed by the pure-DP reduction. The concentration estimates in the proof of Corollary 3.11 hold simultaneously at parameters η p r i v \eta_{\mathrm{priv}} η priv 2 η p r i v 2\eta_{\mathrm{priv}} 2 η priv η ∗ = η a b s / 2 \eta^*=\eta_{\mathrm{abs}}/2 η ∗ = η abs /2 β / 2 \beta/2 β /2
n ≥ Ω ( k + log ( 1 / β ) η p r i v 2 log ( 1 / η p r i v ) ) . n\ge\Omega\left(\frac{k+\log(1/\beta)}{\eta_{\mathrm{priv}}^2\log(1/\eta_{\mathrm{priv}})}\right). n ≥ Ω ( η priv 2 log ( 1/ η priv ) k + log ( 1/ β ) ) . For the low-score witness, take w i w_i w i y i ′ = y i ∗ y_i'=y_i^* y i ′ = y i ∗ μ ′ = μ ˉ = E i y i ∗ \mu'=\bar{\mu}=\mathbb{E}_i y_i^* μ ′ = μ ˉ = E i y i ∗ η n ≤ η p r i v n \eta n\le\eta_{\mathrm{priv}}n η n ≤ η priv n η p r i v n \eta_{\mathrm{priv}}n η priv n μ ˉ \bar{\mu} μ ˉ ℓ 1 \ell_1 ℓ 1 c 1 α c_1\alpha c 1 α μ ˉ \bar{\mu} μ ˉ
For η 0 ∈ { 2 η p r i v , η ∗ } \eta_0\in\{2\eta_{\mathrm{priv}},\eta^*\} η 0 ∈ { 2 η priv , η ∗ } C 0 ( η 0 log ( 1 / η 0 ) / m + ω / m + ω 2 ) C_0(\eta_0\log(1/\eta_0)/m+\omega/m+\omega^2) C 0 ( η 0 log ( 1/ η 0 ) / m + ω / m + ω 2 ) Lemma 4.11 , shows that every certificate with trimming budget at most η 0 n \eta_0n η 0 n
∥ L μ ′ − μ ∥ 1 ≤ O ( η 0 log ( 1 / η 0 ) m + ω ) . (19) \left\|\mathcal{L}\mu'-\mu\right\|_1\le O\left(\eta_0\sqrt{\frac{\log(1/\eta_0)}{m}}+\omega\right). \tag*{(19)} ∥ L μ ′ − μ ∥ 1 ≤ O ( η 0 m log ( 1/ η 0 ) + ω ) . (19) In particular, the score-η ∗ n \eta^*n η ∗ n ℓ 1 \ell_1 ℓ 1 μ \mu μ O ( c 1 α + 1 / m + ω ) O(c_1\alpha+1/\sqrt{m}+\omega) O ( c 1 α + 1/ m + ω ) ω ≤ α / C = O ( c 1 α ) \omega\le\alpha/C=O(c_1\alpha) ω ≤ α / C = O ( c 1 α ) O ( c 1 α + 1 / m ) O(c_1\alpha+1/\sqrt{m}) O ( c 1 α + 1/ m ) η p r i v n \eta_{\mathrm{priv}}n η priv n c 1 α c_1\alpha c 1 α μ ˉ \bar{\mu} μ ˉ
log ( V η ′ ( Y ) V η p r i v ( Y ) ) ≤ O ( k log ( 2 + 1 c 1 α m ) ) ( η p r i v ≤ η ′ ≤ η ∗ ) . \log\left(\frac{V_{\eta'}(\mathcal{Y})}{V_{\eta_{\mathrm{priv}}}(\mathcal{Y})}\right) \le O\left(k\log\left(2+\frac{1}{c_1\alpha\sqrt{m}}\right)\right) \qquad(\eta_{\mathrm{priv}} \le\eta' \le\eta^*). log ( V η priv ( Y ) V η ′ ( Y ) ) ≤ O ( k log ( 2 + c 1 α m 1 ) ) ( η priv ≤ η ′ ≤ η ∗ ) . The full score domain is still Δ k − 1 + B 1 k ( c 1 α + τ n ) \Delta^{k-1}+\mathbb{B}_1^k(c_1\alpha+\tau_n) Δ k − 1 + B 1 k ( c 1 α + τ n ) V 1 / V η p r i v V_1/V_{\eta_{\mathrm{priv}}} V 1 / V η priv § 4.2.3 also applies.
It follows from Theorem 2.18 that the sample bound in Corollary 4.18 returns a point with score at most 2 η p r i v n 2\eta_{\mathrm{priv}}n 2 η priv n 1 − β / 2 1-\beta/2 1 − β /2 n = Ω ( k + log ( 1 / β ) ) n=\Omega(k+\log(1/\beta)) n = Ω ( k + log ( 1/ β )) ( k + log ( 1 / β ) ) / ε (k+\log(1/\beta))/\varepsilon ( k + log ( 1/ β )) / ε ε ≤ 1 \varepsilon\le1 ε ≤ 1 (19) with η 0 = 2 η p r i v \eta_0=2\eta_{\mathrm{priv}} η 0 = 2 η priv Definition 4.3 , and project onto the simplex in ℓ 1 \ell_1 ℓ 1 c 1 c_1 c 1 C C C α \alpha α β \beta β
For approximate DP, use the heterogeneous covariance allowance (18) . If δ > 1 − β / 2 \delta>1-\beta/2 δ > 1 − β /2 δ \delta δ § 4.2.4 to the robust estimator of Corollary 3.11, with trimming parameter η p r i v \eta_{\mathrm{priv}} η priv β / 2 \beta/2 β /2
O ~ ( k + log ( 1 / β ) + k + log ( 1 / β ) α 2 m ) , \widetilde{O}\left(k+\log(1/\beta)+\frac{k+\log(1/\beta)}{\alpha^2m}\right), O ( k + log ( 1/ β ) + α 2 m k + log ( 1/ β ) ) , which the stated approximate-DP bound supplies since ε ≤ 1 \varepsilon\le1 ε ≤ 1 δ ( 1 − β / 2 ) > 1 − β \delta(1-\beta/2)>1-\beta δ ( 1 − β /2 ) > 1 − β
Otherwise, use the approximate-DP reduction with internal accuracy c 1 α c_1\alpha c 1 α η p r i v \eta_{\mathrm{priv}} η priv ω ≤ 1 \omega\le1 ω ≤ 1 (18) gives
1 / m + τ c o v ≤ O ( 1 / m + ω ) . \sqrt{1/m+\tau_{\mathrm{cov}}}\le O(1/\sqrt{m}+\omega). 1/ m + τ cov ≤ O ( 1/ m + ω ) . Thus Corollary 4.15 bounds the worst-case intermediate log-volume ratio by
log ( V η ∗ ( Y ) V 0.8 η ∗ ( Y ) ) ≤ O ( k log ( 1 + 1 c 1 α m + ω c 1 α ) ) . (20) \log\left(\frac{V_{\eta^*}(\mathcal{Y})}{V_{0.8\eta^*}(\mathcal{Y})}\right)\le O\left(k\log\left(1+\frac{1}{c_1\alpha\sqrt{m}}+\frac{\omega}{c_1\alpha}\right)\right). \tag*{(20)} log ( V 0.8 η ∗ ( Y ) V η ∗ ( Y ) ) ≤ O ( k log ( 1 + c 1 α m 1 + c 1 α ω ) ) . (20) Using log ( 1 + x ) ≤ x \log(1+x)\le x log ( 1 + x ) ≤ x (3) is therefore implied by
n ≥ O ( k α m ε + k ω / α + log ( 1 / δ ) ε ) . n\ge O\left(\frac{k}{\alpha\sqrt{m}\,\varepsilon}+\frac{k\omega/\alpha+\log(1/\delta)}{\varepsilon}\right). n ≥ O ( α m ε k + ε k ω / α + log ( 1/ δ ) ) . On the concentration event used for pure DP, the score-η p r i v n \eta_{\mathrm{priv}}n η priv n c 1 α c_1\alpha c 1 α μ ˉ \bar{\mu} μ ˉ (19) at η 0 = η ∗ \eta_0=\eta^* η 0 = η ∗ η ∗ n \eta^*n η ∗ n c 1 α + O ( 1 / m + ω ) c_1\alpha+O(1/\sqrt{m}+\omega) c 1 α + O ( 1/ m + ω ) μ ˉ \bar{\mu} μ ˉ log ( V η ′ ( Y ) / V η p r i v ( Y ) ) \log(V_{\eta'}(\mathcal{Y})/V_{\eta_{\mathrm{priv}}}(\mathcal{Y})) log ( V η ′ ( Y ) / V η priv ( Y )) (20) for η p r i v ≤ η ′ ≤ η ∗ \eta_{\mathrm{priv}}\le\eta'\le\eta^* η priv ≤ η ′ ≤ η ∗ α m \alpha\sqrt{m} α m § 4.2.3 , ω ≤ α / C \omega\le\alpha/C ω ≤ α / C O ~ ( ( k + log ( 1 / β ) ) / ( α m ε ) ) \widetilde{O}((k+\log(1/\beta))/(\alpha\sqrt{m}\,\varepsilon)) O (( k + log ( 1/ β )) / ( α m ε )) η p r i v \eta_{\mathrm{priv}} η priv Lemma 2.20 requires only
n ≥ O ( k α m ε + k ω / α + log ( 2 / β ) ε ) . n\ge O\left(\frac{k}{\alpha\sqrt{m}\,\varepsilon}+\frac{k\omega/\alpha+\log(2/\beta)}{\varepsilon}\right). n ≥ O ( α m ε k + ε k ω / α + log ( 2/ β ) ) . Both requirements are covered by the stated sample bound, which also supplies the concentration event. The reduction returns score at most 2 η p r i v n 2\eta_{\mathrm{priv}}n 2 η priv n (19) and projecting as in (15) gives accuracy α \alpha α β / 2 \beta/2 β /2 β / 2 \beta/2 β /2
Statistical Query Lower Bound for Discrete Batches
In this section, we prove a statistical query (SQ) lower bound for robust learning from discrete batches. We show that no SQ algorithm with polynomially many queries of inverse-polynomial tolerance achieves error o ( η min { 1 , log ( 1 / η ) / m } ) o(\eta\min\{1,\sqrt{\log(1/\eta)/m}\}) o ( η min { 1 , log ( 1/ η ) / m }) [28 ] to categorical batches in [ k ] m [k]^m [ k ] m
Testing Problem
Let μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 Q Q Q [ k ] m [k]^m [ k ] m μ ⊗ m \mu^{\otimes m} μ ⊗ m η \eta η
∥ Q − μ ⊗ m ∥ T V ≤ η . \lVert Q-\mu^{\otimes m}\rVert_{\mathrm{TV}}\leq\eta. ∥ Q − μ ⊗ m ∥ TV ≤ η . An SQ query to Q Q Q ϕ : [ k ] m → [ − 1 , 1 ] \phi:[k]^m\to[-1,1] ϕ : [ k ] m → [ − 1 , 1 ] τ > 0 \tau>0 τ > 0 τ \tau τ E B ∼ Q [ ϕ ( B ) ] \mathbb{E}_{B\sim Q}[\phi(B)] E B ∼ Q [ ϕ ( B )]
We give the construction and proof for even k k k k k k μ 0 \mu_0 μ 0 [ k ] [k] [ k ] P = μ 0 ⊗ m P=\mu_0^{\otimes m} P = μ 0 ⊗ m { 1 , 2 } , { 3 , 4 } , … , { k − 1 , k } \{1,2\},\{3,4\},\ldots,\{k-1,k\} { 1 , 2 } , { 3 , 4 } , … , { k − 1 , k } S S S ∣ S ∣ = k / 2 |S|=k/2 ∣ S ∣ = k /2 0 < Δ < 1 / 4 0<\Delta<1/4 0 < Δ < 1/4 S S S
μ S ( j ) ≔ { ( 1 + 2 Δ ) / k , j ∈ S , ( 1 − 2 Δ ) / k , j ∉ S , ∥ μ S − μ 0 ∥ T V = Δ . \mu_S(j)\coloneqq
\begin{cases}
(1+2\Delta)/k, & j\in S,\\
(1-2\Delta)/k, & j\notin S,
\end{cases}
\qquad
\lVert\mu_S-\mu_0\rVert_{\mathrm{TV}}=\Delta. μ S ( j ) : = { ( 1 + 2Δ ) / k , ( 1 − 2Δ ) / k , j ∈ S , j ∈ / S , ∥ μ S − μ 0 ∥ TV = Δ. For a batch x = ( x 1 , … , x m ) ∈ [ k ] m x=(x_1,\ldots,x_m)\in[k]^m x = ( x 1 , … , x m ) ∈ [ k ] m
N S ( x ) ≔ ∑ i = 1 m 1 [ x i ∈ S ] N_S(x)\coloneqq\sum_{i=1}^{m}\mathbf{1}[x_i\in S] N S ( x ) : = i = 1 ∑ m 1 [ x i ∈ S ] be the number of its elements that lie in the hidden subset. A single draw lies in S S S 1 / 2 1/2 1/2 μ 0 \mu_0 μ 0 1 / 2 + Δ 1/2+\Delta 1/2 + Δ μ S \mu_S μ S p s ≔ Bin ( m , 1 / 2 + s ) p_s\coloneqq\operatorname{Bin}(m,1/2+s) p s : = Bin ( m , 1/2 + s ) ∣ s ∣ < 1 / 2 |s|<1/2 ∣ s ∣ < 1/2
N S ( X ) ∼ { p 0 , X ∼ P , p Δ , X ∼ μ S ⊗ m . N_S(X)\sim
\begin{cases}
p_0, & X\sim P,\\
p_\Delta, & X\sim\mu_S^{\otimes m}.
\end{cases} N S ( X ) ∼ { p 0 , p Δ , X ∼ P , X ∼ μ S ⊗ m . Conditioned on the count N S N_S N S
To compare different hidden subsets, encode the choice from each pair by σ ∈ { − 1 , 1 } k / 2 \sigma\in\{-1,1\}^{k/2} σ ∈ { − 1 , 1 } k /2 2 u − 1 2u-1 2 u − 1 S σ S_\sigma S σ σ u = 1 \sigma_u=1 σ u = 1 2 u 2u 2 u S σ S_\sigma S σ σ u = − 1 \sigma_u=-1 σ u = − 1 μ σ = μ S σ \mu_\sigma=\mu_{S_\sigma} μ σ = μ S σ N σ = N S σ N_\sigma=N_{S_\sigma} N σ = N S σ
ν σ ( j ) ≔ 21 [ j ∈ S σ ] − 1 , ν σ = ( σ 1 , − σ 1 , … , σ k / 2 , − σ k / 2 ) \nu_\sigma(j)\coloneqq2\mathbf{1}[j\in S_\sigma]-1,\qquad
\nu_\sigma=(\sigma_1,-\sigma_1,\ldots,\sigma_{k/2},-\sigma_{k/2}) ν σ ( j ) : = 2 1 [ j ∈ S σ ] − 1 , ν σ = ( σ 1 , − σ 1 , … , σ k /2 , − σ k /2 ) satisfies μ σ ( j ) = ( 1 + 2 Δ v σ ( j ) ) / k \mu_\sigma(j) = (1 + 2\Delta v_\sigma(j))/k μ σ ( j ) = ( 1 + 2Δ v σ ( j )) / k
H 0 : ( μ , Q ) = ( μ 0 , P ) , H 1 : μ = μ σ for some σ ∈ { − 1 , 1 } k / 2 , ∥ Q − μ σ ⊗ m ∥ T V ≤ η / 20. (21) \begin{aligned}
H_0 &: \quad(\mu,Q) = (\mu_0,P),\\
H_1 &: \quad\mu= \mu_\sigma\ \text{for some }\sigma\in\{-1,1\}^{k/2}, \qquad\lVert Q-\mu_\sigma^{\otimes m}\rVert_{\mathrm{TV}} \le\eta/20. \tag*{(21)}
\end{aligned} H 0 H 1 : ( μ , Q ) = ( μ 0 , P ) , : μ = μ σ for some σ ∈ { − 1 , 1 } k /2 , ∥ Q − μ σ ⊗ m ∥ TV ≤ η /20. (21) Given SQ access to Q Q Q
SQ Hardness
Theorem 5.1 (SQ Lower Bound for Testing). There is an absolute constant c > 0 c > 0 c > 0 t ≥ 1 t \ge1 t ≥ 1 c t > 0 c_t > 0 c t > 0 k 0 ( t ) k_0(t) k 0 ( t ) k ≥ k 0 ( t ) k \ge k_0(t) k ≥ k 0 ( t ) m ∈ N m \in\mathbb{N} m ∈ N η ∈ ( 0 , 1 / 4 ] \eta\in(0,1/4] η ∈ ( 0 , 1/4 ] Δ = c t η min { 1 , log ( 1 / η ) / m } \Delta= c_t\eta\min\{1,\sqrt{\log(1/\eta)/m}\} Δ = c t η min { 1 , log ( 1/ η ) / m } (21) must either make a query of tolerance less than
2 k − ( t + 1 ) / 8 2k^{-(t+1)/8} 2 k − ( t + 1 ) /8 or make at least exp ( c k ) \exp(c\sqrt{k}) exp ( c k )
Proof roadmap. For each hidden subset S σ S_\sigma S σ Q σ Q_\sigma Q σ f σ = d Q σ / d P − 1 f_\sigma= \mathrm{d}Q_\sigma/\mathrm{d}P - 1 f σ = d Q σ / d P − 1
∥ f σ ∥ L 2 ( P ) 2 ≤ 2 , ∣ ⟨ f σ , f ω ⟩ L 2 ( P ) ∣ ≤ 2 ∣ ρ ( σ , ω ) ∣ t + 1 . \lVert f_\sigma\rVert_{L^2(P)}^2 \le2,\qquad\left|\langle f_\sigma,f_\omega\rangle_{L^2(P)}\right| \le2|\rho(\sigma,\omega)|^{t+1}. ∥ f σ ∥ L 2 ( P ) 2 ≤ 2 , ⟨ f σ , f ω ⟩ L 2 ( P ) ≤ 2∣ ρ ( σ , ω ) ∣ t + 1 . Here ρ ( σ , ω ) : = ( 2 / k ) ∑ u = 1 k / 2 σ u ω u \rho(\sigma,\omega) := (2/k)\sum_{u=1}^{k/2}\sigma_u\omega_u ρ ( σ , ω ) := ( 2/ k ) ∑ u = 1 k /2 σ u ω u Q σ Q_\sigma Q σ Q σ Q_\sigma Q σ 1 , … , t 1,\ldots,t 1 , … , t ∣ ρ ∣ |\rho| ∣ ρ ∣ § 5.3 reduces robust estimation to this testing problem. The binomial construction is proved in § 5.4 .
Lemma 5.2 (Binomial Moment Matching). For every integer t ≥ 1 t \ge1 t ≥ 1 c t > 0 c_t > 0 c t > 0 m ∈ N m \in\mathbb{N} m ∈ N η ∈ ( 0 , 1 / 4 ] \eta\in(0,1/4] η ∈ ( 0 , 1/4 ]
Δ : = c t η min { 1 , log ( 1 / η ) m } , p s : = Bin ( m , 1 / 2 + s ) . \Delta:= c_t\eta\sqrt{\min\left\{1,\frac{\log(1/\eta)}{m}\right\}},\qquad p_s := \operatorname{Bin}(m,1/2+s). Δ := c t η min { 1 , m log ( 1/ η ) } , p s := Bin ( m , 1/2 + s ) . There is a probability distribution p ~ \widetilde p p { 0 , … , m } \{0,\ldots,m\} { 0 , … , m }
E N ∼ p ~ [ N i ] = E N ∼ p 0 [ N i ] ( 0 ≤ i ≤ t ) , ∥ p ~ − p Δ ∥ T V ≤ η / 20 , χ 2 ( p ~ ∥ p 0 ) ≤ 2. (22) \mathbb{E}_{N\sim\widetilde p}[N^i] = \mathbb{E}_{N\sim p_0}[N^i]\quad(0\le i\le t),\qquad\lVert\widetilde p-p_\Delta\rVert_{\mathrm{TV}}\le\eta/20,\qquad\chi^2(\widetilde p\mathbin\Vert p_0)\le2. \tag*{(22)} E N ∼ p [ N i ] = E N ∼ p 0 [ N i ] ( 0 ≤ i ≤ t ) , ∥ p − p Δ ∥ TV ≤ η /20 , χ 2 ( p ∥ p 0 ) ≤ 2. (22) Here χ 2 ( p ~ ∥ p ) : = ∑ j ( p ~ ( j ) − p ( j ) ) 2 / p ( j ) \chi^2(\widetilde p\mathbin\Vert p) := \sum_j(\widetilde p(j)-p(j))^2/p(j) χ 2 ( p ∥ p ) := ∑ j ( p ( j ) − p ( j ) ) 2 / p ( j ) [28 Lemma 11] . We prove this lemma after Theorem 5.1.
Lemma 5.3 (SQ Testing from Correlation Tails). Let P P P { Q θ : θ ∈ Θ } \{Q_\theta:\theta\in\Theta\} { Q θ : θ ∈ Θ } X \mathcal{X} X Θ \Theta Θ Q θ ≪ P Q_\theta\ll P Q θ ≪ P f θ = d Q θ / d P − 1 f_\theta=\mathrm{d}Q_\theta/\mathrm{d}P-1 f θ = d Q θ / d P − 1 B > 0 B>0 B > 0 γ , ξ ≥ 0 \gamma,\xi\ge0 γ , ξ ≥ 0 θ ∈ Θ \theta\in\Theta θ ∈ Θ
∥ f θ ∥ L 2 ( P ) 2 ≤ B , P ω ∼ Unif ( Θ ) [ ∣ ⟨ f θ , f ω ⟩ L 2 ( P ) ∣ > γ ] ≤ ξ . \|f_\theta\|_{L^2(P)}^2\le B,\qquad\mathop{\mathbb{P}}_{\omega\sim\operatorname{Unif}(\Theta)}\left[\left|\langle f_\theta,f_\omega\rangle_{L^2(P)}\right|>\gamma\right]\le\xi. ∥ f θ ∥ L 2 ( P ) 2 ≤ B , P ω ∼ Unif ( Θ ) [ ⟨ f θ , f ω ⟩ L 2 ( P ) > γ ] ≤ ξ . Let τ > 0 \tau>0 τ > 0 τ 2 > γ \tau^2>\gamma τ 2 > γ P P P Q θ Q_\theta Q θ 2 / 3 2/3 2/3 q q q τ \tau τ
q B ξ ≥ τ 2 − γ 3 . qB\xi\ge\frac{\tau^2-\gamma}{3}. q B ξ ≥ 3 τ 2 − γ . Proof of Theorem 5.1. Take p ~ \widetilde p p σ ∈ { − 1 , 1 } k / 2 \sigma\in\{-1,1\}^{k/2} σ ∈ { − 1 , 1 } k /2 Q σ Q_\sigma Q σ ∥ Q σ − μ σ ⊗ m ∥ T V ≤ η / 20 \|Q_\sigma-\mu_\sigma^{\otimes m}\|_{\mathrm{TV}}\le\eta/20 ∥ Q σ − μ σ ⊗ m ∥ TV ≤ η /20
Hard distributions. To draw a batch from Q σ Q_\sigma Q σ N σ N_\sigma N σ p ~ \widetilde p p
F σ ( x ) ≔ d Q σ d P ( x ) = p ~ ( N σ ( x ) ) p 0 ( N σ ( x ) ) , f σ ≔ F σ − 1. F_\sigma(x)\coloneqq\frac{\mathrm{d}Q_\sigma}{\mathrm{d}P}(x)=\frac{\widetilde p(N_\sigma(x))}{p_0(N_\sigma(x))},\qquad f_\sigma\coloneqq F_\sigma-1. F σ ( x ) : = d P d Q σ ( x ) = p 0 ( N σ ( x )) p ( N σ ( x )) , f σ : = F σ − 1. Under P P P N σ N_\sigma N σ p 0 p_0 p 0 Q σ Q_\sigma Q σ μ σ ⊗ m \mu_\sigma^{\otimes m} μ σ ⊗ m p Δ ( N σ ) / p 0 ( N σ ) p_\Delta(N_\sigma)/p_0(N_\sigma) p Δ ( N σ ) / p 0 ( N σ ) P P P
∥ Q σ − μ σ ⊗ m ∥ T V = ∥ p ~ − p Δ ∥ T V ≤ η / 20 , ∥ f σ ∥ L 2 ( P ) 2 = χ 2 ( p ~ ∥ p 0 ) ≤ 2. (23) \|Q_\sigma-\mu_\sigma^{\otimes m}\|_{\mathrm{TV}}=\|\widetilde p-p_\Delta\|_{\mathrm{TV}}\le\eta/20,\qquad\|f_\sigma\|_{L^2(P)}^2=\chi^2(\widetilde p\|p_0)\le2. \tag*{(23)} ∥ Q σ − μ σ ⊗ m ∥ TV = ∥ p − p Δ ∥ TV ≤ η /20 , ∥ f σ ∥ L 2 ( P ) 2 = χ 2 ( p ∥ p 0 ) ≤ 2. (23) Bounding correlations. For z ∈ { − 1 , 1 } m z\in\{-1,1\}^m z ∈ { − 1 , 1 } m N ( z ) N(z) N ( z )
g ( z ) ≔ p ~ ( N ( z ) ) p 0 ( N ( z ) ) = ∑ A ⊆ [ m ] g ^ ( A ) ∏ i ∈ A z i . g(z)\coloneqq\frac{\widetilde p(N(z))}{p_0(N(z))}=\sum_{A\subseteq[m]}\widehat g(A)\prod_{i\in A}z_i. g ( z ) : = p 0 ( N ( z )) p ( N ( z )) = A ⊆ [ m ] ∑ g ( A ) i ∈ A ∏ z i . The batch likelihood ratios satisfy F σ ( x ) = g ( v σ ( x 1 ) , … , v σ ( x m ) ) F_\sigma(x)=g(v_\sigma(x_1),\ldots,v_\sigma(x_m)) F σ ( x ) = g ( v σ ( x 1 ) , … , v σ ( x m )) g ^ ( ∅ ) = 1 \widehat g(\varnothing)=1 g ( ∅ ) = 1 g ^ ( A ) = 0 \widehat g(A)=0 g ( A ) = 0 1 ≤ ∣ A ∣ ≤ t 1\le|A|\le t 1 ≤ ∣ A ∣ ≤ t ∏ i ∈ A z i \prod_{i\in A}z_i ∏ i ∈ A z i N ( z ) = j N(z)=j N ( z ) = j
( m ∣ A ∣ ) − 1 ∑ r = 0 ∣ A ∣ ( − 1 ) ∣ A ∣ − r ( j r ) ( m − j ∣ A ∣ − r ) . \binom{m}{|A|}^{-1}\sum_{r=0}^{|A|}(-1)^{|A|-r}\binom{j}{r}\binom{m-j}{|A|-r}. ( ∣ A ∣ m ) − 1 r = 0 ∑ ∣ A ∣ ( − 1 ) ∣ A ∣ − r ( r j ) ( ∣ A ∣ − r m − j ) . This is a polynomial in j j j ∣ A ∣ |A| ∣ A ∣ p ~ \widetilde p p p 0 p_0 p 0
For U U U [ k ] [k] [ k ] v σ ( U ) v_\sigma(U) v σ ( U ) v ω ( U ) v_\omega(U) v ω ( U ) E [ v σ ( U ) v ω ( U ) ] = ρ ( σ , ω ) \mathbb{E}[v_\sigma(U)v_\omega(U)]=\rho(\sigma,\omega) E [ v σ ( U ) v ω ( U )] = ρ ( σ , ω ) A A A ρ ( σ , ω ) ∣ A ∣ \rho(\sigma,\omega)^{|A|} ρ ( σ , ω ) ∣ A ∣
⟨ f σ , f ω ⟩ L 2 ( P ) = ∑ ∣ A ∣ > t g ^ ( A ) 2 ρ ( σ , ω ) ∣ A ∣ , ∣ ⟨ f σ , f ω ⟩ L 2 ( P ) ∣ ≤ 2 ∣ ρ ( σ , ω ) ∣ t + 1 . (24) \langle f_\sigma,f_\omega\rangle_{L^2(P)}=\sum_{|A|>t}\widehat g(A)^2\rho(\sigma,\omega)^{|A|},\qquad\left|\langle f_\sigma,f_\omega\rangle_{L^2(P)}\right|\le2|\rho(\sigma,\omega)|^{t+1}. \tag*{(24)} ⟨ f σ , f ω ⟩ L 2 ( P ) = ∣ A ∣ > t ∑ g ( A ) 2 ρ ( σ , ω ) ∣ A ∣ , ⟨ f σ , f ω ⟩ L 2 ( P ) ≤ 2∣ ρ ( σ , ω ) ∣ t + 1 . (24) Parseval’s identity and (23) give the inequality. If t ≥ m t\ge m t ≥ m f σ = 0 f_\sigma=0 f σ = 0 σ \sigma σ
Applying the SQ lemma. For each fixed σ \sigma σ
Pr ω ∼ Unif ( { ± 1 } k / 2 ) [ ∣ ρ ( σ , ω ) ∣ > k − 1 / 4 ] ≤ 2 e − k / 8 . \Pr_{\omega\sim\operatorname{Unif}(\{\pm1\}^{k/2})}\left[|\rho(\sigma,\omega)|>k^{-1/4}\right]\le2e^{-\sqrt{k}/8}. ω ∼ Unif ({ ± 1 } k /2 ) Pr [ ∣ ρ ( σ , ω ) ∣ > k − 1/4 ] ≤ 2 e − k /8 . Together with (23) and (24) , this verifies the hypotheses of Lemma 5.3 with
B = 2 , γ = 2 k − ( t + 1 ) / 4 , ξ = 2 e − k / 8 , τ = 2 k − ( t + 1 ) / 8 . B=2,\qquad\gamma=2k^{-(t+1)/4},\qquad\xi=2e^{-\sqrt{k}/8},\qquad\tau=2k^{-(t+1)/8}. B = 2 , γ = 2 k − ( t + 1 ) /4 , ξ = 2 e − k /8 , τ = 2 k − ( t + 1 ) /8 . If all query tolerances are at least τ \tau τ
q ≥ τ 2 − γ 3 B ξ = 1 6 k − ( t + 1 ) / 4 e k / 8 ≥ e k / 24 . q\ge\frac{\tau^2-\gamma}{3B\xi}=\frac{1}{6}k^{-(t+1)/4}e^{\sqrt{k}/8}\ge e^{\sqrt{k}/24}. q ≥ 3 B ξ τ 2 − γ = 6 1 k − ( t + 1 ) /4 e k /8 ≥ e k /24 . for sufficiently large k k k t t t c = 1 / 24 c=1/24 c = 1/24 □ \square □
Proof of Lemma 5.3. All inner products and norms below are in L 2 ( P ) L^2(P) L 2 ( P ) Θ \Theta Θ
One query. Fix ϕ : X → [ − 1 , 1 ] \phi:\mathcal{X}\to[-1,1] ϕ : X → [ − 1 , 1 ]
T ϕ : = { θ : ∣ ⟨ ϕ , f θ ⟩ ∣ > τ } , p : = Pr θ [ θ ∈ T ϕ ] , G : = E θ [ 1 [ θ ∈ T ϕ ] sign ( ⟨ ϕ , f θ ⟩ ) f θ ] . \mathcal{T}_{\phi}:=\{\theta:|\langle\phi,f_{\theta}\rangle|>\tau\},\qquad p:=\Pr_{\theta}[\theta\in\mathcal{T}_{\phi}],\qquad G:=\mathbb{E}_{\theta}\left[\mathbf{1}[\theta\in\mathcal{T}_{\phi}]\operatorname{sign}(\langle\phi,f_{\theta}\rangle)f_{\theta}\right]. T ϕ := { θ : ∣ ⟨ ϕ , f θ ⟩ ∣ > τ } , p := θ Pr [ θ ∈ T ϕ ] , G := E θ [ 1 [ θ ∈ T ϕ ] sign (⟨ ϕ , f θ ⟩) f θ ] . If p > 0 p>0 p > 0 ∥ ϕ ∥ ≤ 1 \|\phi\|\le1 ∥ ϕ ∥ ≤ 1
p 2 τ 2 < ⟨ ϕ , G ⟩ 2 ≤ ∥ G ∥ 2 2 ≤ γ p 2 + B ξ p . p^2\tau^2<\langle\phi,G\rangle^2\le\|G\|_2^2\le\gamma p^2+B\xi p. p 2 τ 2 < ⟨ ϕ , G ⟩ 2 ≤ ∥ G ∥ 2 2 ≤ γ p 2 + B ξ p . For the last inequality, pairs with ∣ ⟨ f θ , f ω ⟩ ∣ ≤ γ |\langle f_{\theta},f_{\omega}\rangle|\le\gamma ∣ ⟨ f θ , f ω ⟩ ∣ ≤ γ γ p 2 \gamma p^2 γ p 2 θ ∈ T ϕ \theta\in\mathcal{T}_{\phi} θ ∈ T ϕ ω \omega ω ξ \xi ξ ∣ ⟨ f θ , f ω ⟩ ∣ ≤ B |\langle f_{\theta},f_{\omega}\rangle|\le B ∣ ⟨ f θ , f ω ⟩ ∣ ≤ B
p ≤ B ξ τ 2 − γ . (25) p\le\frac{B\xi}{\tau^2-\gamma}. \tag*{(25)} p ≤ τ 2 − γ B ξ . (25) This bound also holds when p = 0 p=0 p = 0 E Q θ ϕ − E P ϕ = ⟨ ϕ , f θ ⟩ \mathbb{E}_{Q_{\theta}}\phi-\mathbb{E}_{P}\phi=\langle\phi,f_{\theta}\rangle E Q θ ϕ − E P ϕ = ⟨ ϕ , f θ ⟩ τ \tau τ
Adaptive queries. Run the tester against the oracle returning exact expectations under P P P q q q (25) and a union bound, these answers are all valid for a uniform Q θ Q_{\theta} Q θ q B ξ / ( τ 2 − γ ) qB\xi/(\tau^2-\gamma) q B ξ / ( τ 2 − γ )
For each θ \theta θ E P ϕ \mathbb{E}_{P}\phi E P ϕ E Q θ ϕ \mathbb{E}_{Q_{\theta}}\phi E Q θ ϕ E Q θ ϕ \mathbb{E}_{Q_{\theta}}\phi E Q θ ϕ q B ξ / ( τ 2 − γ ) qB\xi/(\tau^2-\gamma) q B ξ / ( τ 2 − γ ) θ \theta θ 2 / 3 2/3 2/3 1 / 3 1/3 1/3
q B ξ τ 2 − γ ≥ 1 3 . \frac{qB\xi}{\tau^2-\gamma}\ge\frac{1}{3}. τ 2 − γ q B ξ ≥ 3 1 . □ \square □
From Estimation to Testing
Corollary 5.4 (SQ Lower Bound for Robust Estimation). For every integer t ≥ 1 t \ge1 t ≥ 1 c t c_t c t k 0 ( t ) k_0(t) k 0 ( t ) Theorem 5.1 . For all k ≥ k 0 ( t ) k \ge k_0(t) k ≥ k 0 ( t ) m ∈ N m \in\mathbb{N} m ∈ N η ∈ ( 0 , 1 / 4 ] \eta\in(0,1/4] η ∈ ( 0 , 1/4 ] μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 Q Q Q ∥ Q − μ ⊗ m ∥ T V ≤ η \lVert Q-\mu^{\otimes m}\rVert_{\mathrm{TV}} \le\eta ∥ Q − μ ⊗ m ∥ TV ≤ η
∥ μ ^ − μ ∥ T V ≤ c t 4 η min { 1 , log ( 1 / η ) m } \lVert\hat{\mu}-\mu\rVert_{\mathrm{TV}} \le\frac{c_t}{4}\eta\min\left\{1,\sqrt{\frac{\log(1/\eta)}{m}}\right\} ∥ μ ^ − μ ∥ TV ≤ 4 c t η min { 1 , m log ( 1/ η ) } with probability at least 2 / 3 2/3 2/3 2 k − ( t + 1 ) / 8 2k^{-(t+1)/8} 2 k − ( t + 1 ) /8 exp ( c k ) \exp(c\sqrt{k}) exp ( c k )
Consequently, no SQ algorithm with fixed polynomial bounds on query count and inverse tolerance in k , m , 1 / η k,m,1/\eta k , m , 1/ η o ( η min { 1 , log ( 1 / η ) / m } ) o\left(\eta\min\left\{1,\sqrt{\log(1/\eta)/m}\right\}\right) o ( η min { 1 , log ( 1/ η ) / m } ) η = 1 / k \eta=1/k η = 1/ k m = ⌈ ( log k ) 2 ⌉ m=\lceil(\log k)^2\rceil m = ⌈( log k ) 2 ⌉ k → ∞ k\to\infty k → ∞
Proof. Run the estimator and declare H 1 H_1 H 1 ∥ μ ^ − μ 0 ∥ T V > Δ / 2 \lVert\hat{\mu}-\mu_0\rVert_{\mathrm{TV}}>\Delta/2 ∥ μ ^ − μ 0 ∥ TV > Δ/2 Δ / 4 \Delta/4 Δ/4 H 0 H_0 H 0 3 Δ / 4 3\Delta/4 3Δ/4 H 1 H_1 H 1 (21) , and Theorem 5.1 applies.
For the asymptotic conclusion, set η = 1 / k \eta=1/k η = 1/ k m = ⌈ ( log k ) 2 ⌉ m=\lceil(\log k)^2\rceil m = ⌈( log k ) 2 ⌉ q ≤ k a q\le k^a q ≤ k a k − b k^{-b} k − b a , b a,b a , b k k k t t t ( t + 1 ) / 8 > b (t+1)/8>b ( t + 1 ) /8 > b o ( η min { 1 , log ( 1 / η ) / m } ) o\left(\eta\min\left\{1,\sqrt{\log(1/\eta)/m}\right\}\right) o ( η min { 1 , log ( 1/ η ) / m } ) c t η min { 1 , log ( 1 / η ) / m } / 4 c_t\eta\min\left\{1,\sqrt{\log(1/\eta)/m}\right\}/4 c t η min { 1 , log ( 1/ η ) / m } /4 k − b ≥ 2 k − ( t + 1 ) / 8 k^{-b}\ge2k^{-(t+1)/8} k − b ≥ 2 k − ( t + 1 ) /8 k a < exp ( k / 24 ) k^a<\exp(\sqrt{k}/24) k a < exp ( k /24 ) Theorem 5.1 . □
With sufficiently many batches, the polynomial-time estimator of Jain and Orlitsky [JO20b], together with the pooled histogram for m ≤ log ( 1 / η ) m\le\log(1/\eta) m ≤ log ( 1/ η ) O ( η min { 1 , log ( 1 / η ) / m } ) O\left(\eta\min\left\{1,\sqrt{\log(1/\eta)/m}\right\}\right) O ( η min { 1 , log ( 1/ η ) / m } ) Θ ( η / m ) \Theta(\eta/\sqrt{m}) Θ ( η / m ) Theorem 5.1 , the error scale matches the dependence on m m m η \eta η c t c_t c t
One can also obtain the distributions used in the SQ lower bound, up to exponentially small total-variation error, by replacing at most ⌊ η n ⌋ \lfloor\eta n\rfloor ⌊ η n ⌋ n n n Q σ Q_\sigma Q σ ⌊ η n ⌋ \lfloor\eta n\rfloor ⌊ η n ⌋ ∥ Q σ − μ ⊗ m ∥ T V ≤ η / 20 \lVert Q_\sigma-\mu^{\otimes m}\rVert_{\mathrm{TV}}\le\eta/20 ∥ Q σ − μ ⊗ m ∥ TV ≤ η /20 exp ( − c ′ η n ) \exp(-c'\eta n) exp ( − c ′ η n ) Q σ ⊗ n Q_\sigma^{\otimes n} Q σ ⊗ n c ′ > 0 c'>0 c ′ > 0
Binomial Moment Construction
We follow the polynomial-correction method of Diakonikolas, Kane, and Sun [28 Proposition 25] . For each fixed moment order t t t
Proof of Lemma 5.2. Constants denoted C t C_t C t t t t min { m , log ( 1 / η ) } ≥ R t \min\{m,\log(1/\eta)\}\ge R_t min { m , log ( 1/ η )} ≥ R t R t > t + 1 R_t>t+1 R t > t + 1 t t t
I : = { j ∈ { 0 , … , m } : ∣ j − m / 2 ∣ ≤ m min { m , log ( 1 / η ) } 16 } I:=\left\{j\in\{0,\ldots,m\}: \lvert j-m/2\rvert\le\frac{\sqrt{m\min\{m,\log(1/\eta)\}}}{16}\right\} I := { j ∈ { 0 , … , m } : ∣ j − m /2 ∣ ≤ 16 m min { m , log ( 1/ η )} } z j ≔ 16 ( j − m / 2 ) m min { m , log ( 1 / η ) } . z_j \coloneqq\frac{16(j-m/2)}{\sqrt{m\min\{m,\log(1/\eta)\}}}. z j : = m min { m , log ( 1/ η )} 16 ( j − m /2 ) . Since min { m , log ( 1 / η ) } ≤ m \min\{m,\log(1/\eta)\} \le m min { m , log ( 1/ η )} ≤ m
1 16 min { m , log ( 1 / η ) } ≤ m min { m , log ( 1 / η ) } 16 ≤ m / 16 \frac{1}{16}\min\{m,\log(1/\eta)\} \le\frac{\sqrt{m\min\{m,\log(1/\eta)\}}}{16} \le m/16 16 1 min { m , log ( 1/ η )} ≤ 16 m min { m , log ( 1/ η )} ≤ m /16 and ∣ I ∣ = Θ ( m min { m , log ( 1 / η ) } / 16 ) |I| = \Theta(\sqrt{m\min\{m,\log(1/\eta)\}}/16) ∣ I ∣ = Θ ( m min { m , log ( 1/ η )} /16 ) ( t + 1 ) × ( t + 1 ) (t+1) \times(t+1) ( t + 1 ) × ( t + 1 )
M u v ≔ 1 ∣ I ∣ ∑ j ∈ I z j u + v , 0 ≤ u , v ≤ t . M_{uv} \coloneqq\frac{1}{|I|}\sum_{j\in I} z_j^{u+v}, \qquad0 \le u,v \le t. M uv : = ∣ I ∣ 1 j ∈ I ∑ z j u + v , 0 ≤ u , v ≤ t . These Riemann sums converge to
H u v = 1 2 ∫ − 1 1 x u + v d x as m min { m , log ( 1 / η ) } 16 → ∞ , H_{uv} = \frac{1}{2}\int_{-1}^{1} x^{u+v}\,dx \qquad\text{as } \frac{\sqrt{m\min\{m,\log(1/\eta)\}}}{16} \to\infty, H uv = 2 1 ∫ − 1 1 x u + v d x as 16 m min { m , log ( 1/ η )} → ∞ , with error O t ( 16 / m min { m , log ( 1 / η ) } ) O_t(16/\sqrt{m\min\{m,\log(1/\eta)\}}) O t ( 16/ m min { m , log ( 1/ η )} ) H H H R t R_t R t
∥ M − 1 ∥ o p ≤ C t . (26) \|M^{-1}\|_{\mathrm{op}} \le C_t . \tag*{(26)} ∥ M − 1 ∥ op ≤ C t . (26) The bound is uniform over m m m η \eta η m m m
Correcting the moments. Let b i ≔ E N ∼ p 0 [ z N i ] − E N ∼ p Δ [ z N i ] b_i \coloneqq\mathbb{E}_{N\sim p_0}[z_N^i] - \mathbb{E}_{N\sim p_\Delta}[z_N^i] b i : = E N ∼ p 0 [ z N i ] − E N ∼ p Δ [ z N i ]
∣ b i ∣ ≤ C t 16 m Δ m min { m , log ( 1 / η ) } ≤ C t c t η , 0 ≤ i ≤ t . (27) |b_i| \le C_t\frac{16m\Delta}{\sqrt{m\min\{m,\log(1/\eta)\}}} \le C_t c_t\eta,\qquad0 \le i \le t. \tag*{(27)} ∣ b i ∣ ≤ C t m min { m , log ( 1/ η )} 16 m Δ ≤ C t c t η , 0 ≤ i ≤ t . (27) Differentiating the binomial formula gives, for any function f f f { 0 , … , m } \{0,\ldots,m\} { 0 , … , m }
d d s E N ∼ p s [ f ( N ) ] = m E Y ∼ Bin ( m − 1 , 1 / 2 + s ) [ f ( Y + 1 ) − f ( Y ) ] . \frac{d}{ds}\mathbb{E}_{N\sim p_s}[f(N)] = m\mathbb{E}_{Y\sim\operatorname{Bin}(m-1,1/2+s)}[f(Y+1)-f(Y)]. d s d E N ∼ p s [ f ( N )] = m E Y ∼ Bin ( m − 1 , 1/2 + s ) [ f ( Y + 1 ) − f ( Y )] . For 1 ≤ i ≤ t 1 \le i \le t 1 ≤ i ≤ t f ( j ) = z j i f(j)=z_j^i f ( j ) = z j i
16 i m min { m , log ( 1 / η ) } ∫ 0 1 ( 16 ( Y + u − m / 2 ) m min { m , log ( 1 / η ) } ) i − 1 d u . \frac{16i}{\sqrt{m\min\{m,\log(1/\eta)\}}}\int_0^1\left(\frac{16(Y+u-m/2)}{\sqrt{m\min\{m,\log(1/\eta)\}}}\right)^{i-1}du. m min { m , log ( 1/ η )} 16 i ∫ 0 1 ( m min { m , log ( 1/ η )} 16 ( Y + u − m /2 ) ) i − 1 d u . For 0 ≤ s ≤ Δ 0 \le s \le\Delta 0 ≤ s ≤ Δ 0 ≤ u ≤ 1 0 \le u \le1 0 ≤ u ≤ 1 16 ( Y + u − m / 2 ) / m min { m , log ( 1 / η ) } 16(Y+u-m/2)/\sqrt{m\min\{m,\log(1/\eta)\}} 16 ( Y + u − m /2 ) / m min { m , log ( 1/ η )} 16 / min { m , log ( 1 / η ) } ≤ 16 16/\sqrt{\min\{m,\log(1/\eta)\}} \le16 16/ min { m , log ( 1/ η )} ≤ 16 16 ( m Δ + 1 ) / m min { m , log ( 1 / η ) } ≤ 2 16(m\Delta+1)/\sqrt{m\min\{m,\log(1/\eta)\}} \le2 16 ( m Δ + 1 ) / m min { m , log ( 1/ η )} ≤ 2 c t ≤ 1 / 16 c_t \le1/16 c t ≤ 1/16 R t R_t R t E [ ∣ Y − E Y ∣ r ] ≤ C r m r / 2 \mathbb{E}[|Y-\mathbb{E}Y|^r]\le C_r m^{r/2} E [ ∣ Y − E Y ∣ r ] ≤ C r m r /2 [ 0 , Δ ] [0,\Delta] [ 0 , Δ ] (27) ; also b 0 = 0 b_0=0 b 0 = 0
Solve M λ = b M\lambda=b M λ = b
h ( z ) ≔ ∑ i = 0 t λ i z i , d j ≔ { h ( z j ) / ∣ I ∣ , j ∈ I , 0 , j ∉ I , p ~ ( j ) ≔ p Δ ( j ) + d j . h(z)\coloneqq\sum_{i=0}^{t}\lambda_i z^i,\qquad d_j\coloneqq
\begin{cases}
h(z_j)/|I|, & j\in I,\\
0, & j\notin I,
\end{cases}
\qquad\widetilde p(j)\coloneqq p_\Delta(j)+d_j. h ( z ) : = i = 0 ∑ t λ i z i , d j : = { h ( z j ) /∣ I ∣ , 0 , j ∈ I , j ∈ / I , p ( j ) : = p Δ ( j ) + d j . Then ∑ j d j z j i = b i \sum_j d_j z_j^i=b_i ∑ j d j z j i = b i 0 ≤ i ≤ t 0\le i\le t 0 ≤ i ≤ t (22) and total mass one. By (26) and (27) ,
sup ∣ z ∣ ≤ 1 ∣ h ( z ) ∣ ≤ C t c t η , ∑ j ∣ d j ∣ ≤ C t c t η , max j ∣ d j ∣ ≤ 16 C t c t η m min { m , log ( 1 / η ) } . (28) \sup_{|z|\le1}|h(z)|\le C_t c_t\eta,\qquad\sum_j|d_j|\le C_t c_t\eta,\qquad\max_j|d_j|\le\frac{16C_t c_t\eta}{\sqrt{m\min\{m,\log(1/\eta)\}}}. \tag*{(28)} ∣ z ∣ ≤ 1 sup ∣ h ( z ) ∣ ≤ C t c t η , j ∑ ∣ d j ∣ ≤ C t c t η , j max ∣ d j ∣ ≤ m min { m , log ( 1/ η )} 16 C t c t η . (28) Nonnegativity and distance bounds. For j ∈ I j \in I j ∈ I j / m j/m j / m 1 / 2 + Δ 1/2 + \Delta 1/2 + Δ [ 1 / 4 , 3 / 4 ] [1/4, 3/4] [ 1/4 , 3/4 ]
∣ j − m ( 1 / 2 + Δ ) ∣ ≤ m min { m , log ( 1 / η ) } 8 . \lvert j - m(1/2 + \Delta) \rvert\le\frac{\sqrt{m\min\{m,\log(1/\eta)\}}}{8}. ∣ j − m ( 1/2 + Δ )∣ ≤ 8 m min { m , log ( 1/ η )} . since m Δ ≤ m min { m , log ( 1 / η ) } / 16 m\Delta\le\sqrt{m\min\{m,\log(1/\eta)\}}/16 m Δ ≤ m min { m , log ( 1/ η )} /16
p Δ ( j ) ≥ c m exp ( − m D ( j / m ∥ 1 / 2 + Δ ) ) , p_{\Delta}(j) \ge\frac{c}{\sqrt{m}}\exp\left(-mD(j/m \mathbin\Vert1/2+\Delta)\right), p Δ ( j ) ≥ m c exp ( − m D ( j / m ∥ 1/2 + Δ ) ) , where D ( ⋅ ∥ ⋅ ) D(\cdot\mathbin\Vert\cdot) D ( ⋅ ∥ ⋅ ) log x ≤ x − 1 \log x \le x-1 log x ≤ x − 1
D ( j / m ∥ 1 / 2 + Δ ) ≤ ( j / m − 1 / 2 − Δ ) 2 ( 1 / 2 + Δ ) ( 1 / 2 − Δ ) ≤ 16 3 ( j / m − 1 / 2 − Δ ) 2 . D(j/m \mathbin\Vert1/2+\Delta) \le\frac{(j/m-1/2-\Delta)^2}{(1/2+\Delta)(1/2-\Delta)} \le\frac{16}{3}(j/m-1/2-\Delta)^2. D ( j / m ∥ 1/2 + Δ ) ≤ ( 1/2 + Δ ) ( 1/2 − Δ ) ( j / m − 1/2 − Δ ) 2 ≤ 3 16 ( j / m − 1/2 − Δ ) 2 . Therefore
p Δ ( j ) ≥ c m e − min { m , log ( 1 / η ) } / 12 ≥ c η m . p_{\Delta}(j) \ge\frac{c}{\sqrt{m}}e^{-\min\{m,\log(1/\eta)\}/12} \ge\frac{c\sqrt{\eta}}{\sqrt{m}}. p Δ ( j ) ≥ m c e − m i n { m , l o g ( 1/ η )} /12 ≥ m c η . Consequently,
∣ d j ∣ p Δ ( j ) ≤ C t c t η min { m , log ( 1 / η ) } ≤ 1 2 \frac{\lvert d_j\rvert}{p_{\Delta}(j)} \le C_t c_t\frac{\sqrt{\eta}}{\sqrt{\min\{m,\log(1/\eta)\}}} \le\frac{1}{2} p Δ ( j ) ∣ d j ∣ ≤ C t c t min { m , log ( 1/ η )} η ≤ 2 1 after reducing c t c_t c t 0 ≤ p ~ ≤ 3 2 p Δ 0 \le\widetilde p \le\frac{3}{2}p_{\Delta} 0 ≤ p ≤ 2 3 p Δ (28) gives ∥ p ~ − p Δ ∥ T V ≤ η / 20 \lVert\widetilde p-p_{\Delta}\rVert_{\mathrm{TV}} \le\eta/20 ∥ p − p Δ ∥ TV ≤ η /20
By the binomial theorem,
∑ j = 0 m p Δ ( j ) 2 p 0 ( j ) = ∑ j = 0 m ( m j ) [ 2 ( 1 / 2 + Δ ) 2 ] j [ 2 ( 1 / 2 − Δ ) 2 ] m − j = ( 1 + 4 Δ 2 ) m . (29) \sum_{j=0}^{m}\frac{p_{\Delta}(j)^2}{p_0(j)}
=\sum_{j=0}^{m}\binom{m}{j}\left[2(1/2+\Delta)^2\right]^j\left[2(1/2-\Delta)^2\right]^{m-j}
=(1+4\Delta^2)^m. \tag*{(29)} j = 0 ∑ m p 0 ( j ) p Δ ( j ) 2 = j = 0 ∑ m ( j m ) [ 2 ( 1/2 + Δ ) 2 ] j [ 2 ( 1/2 − Δ ) 2 ] m − j = ( 1 + 4 Δ 2 ) m . (29) Hence
1 + χ 2 ( p ~ ∥ p 0 ) ≤ 9 4 ( 1 + 4 Δ 2 ) m ≤ 3. 1+\chi^2(\widetilde p\mathbin\Vert p_0) \le\frac{9}{4}(1+4\Delta^2)^m \le3. 1 + χ 2 ( p ∥ p 0 ) ≤ 4 9 ( 1 + 4 Δ 2 ) m ≤ 3. The last inequality holds for sufficiently small c t c_t c t m Δ 2 ≤ c t 2 η 2 log ( 1 / η ) m\Delta^2 \le c_t^2\eta^2\log(1/\eta) m Δ 2 ≤ c t 2 η 2 log ( 1/ η )
Remaining parameter range. If min { m , log ( 1 / η ) } < R t \min\{m,\log(1/\eta)\}<R_t min { m , log ( 1/ η )} < R t p ~ = p 0 \widetilde p=p_0 p = p 0 (29) and Cauchy–Schwarz,
∥ p 0 − p Δ ∥ T V ≤ 1 2 ( 1 + 4 Δ 2 ) m − 1 ≤ C Δ m ≤ C c t η R t ≤ η / 20. \lVert p_0-p_{\Delta}\rVert_{\mathrm{TV}} \le\frac{1}{2}\sqrt{(1+4\Delta^2)^m-1} \le C\Delta\sqrt{m} \le Cc_t\eta\sqrt{R_t} \le\eta/20. ∥ p 0 − p Δ ∥ TV ≤ 2 1 ( 1 + 4 Δ 2 ) m − 1 ≤ C Δ m ≤ C c t η R t ≤ η /20. All moments match and the chi-square divergence is zero. Every choice of constants depends only on t t t
AI Disclosure. We used Codex and GPT Pro to assist with technical development and exposition, including the diagonal-majorizer and semidefinite-duality formulation, the application of the symmetric Grothendieck inequality, the approximate-DP obstruction argument, and the SQ lower-bound proof. The authors are responsible for the final manuscript.
References
Details for the Approximate-DP Section
This appendix proves the obstruction argument of Hopkins, Kamath, Majid, and Narayanan [HKMN23] in sign directions σ ∈ { ± 1 } k \sigma\in\{\pm1\}^{k} σ ∈ { ± 1 } k ℓ 1 \ell_1 ℓ 1 ℓ 1 \ell_1 ℓ 1 ℓ 2 \ell_2 ℓ 2 ℓ 1 \ell_1 ℓ 1
Lemma A.1 (One-dimensional obstruction for low-variance certificates). Let z 1 , … , z n ∈ R z_1,\ldots,z_n \in\mathbb{R} z 1 , … , z n ∈ R n / 4 n/4 n /4 z i ′ z_i' z i ′ L \mathcal{L} L { w i , z i ′ } i ∈ [ n ] \{w_i,z_i'\}_{i\in[n]} { w i , z i ′ } i ∈ [ n ]
w i 2 = w i w_i^2 = w_i w i 2 = w i i ∈ [ n ] i \in[n] i ∈ [ n ]
∑ i = 1 n w i ≥ 0.99 n \sum_{i=1}^{n} w_i \ge0.99n ∑ i = 1 n w i ≥ 0.99 n
w i ( z i ′ − z i ) = 0 w_i(z_i' - z_i) = 0 w i ( z i ′ − z i ) = 0 i ∈ [ n ] i \in[n] i ∈ [ n ]
L [ 1 n ∑ i = 1 n z i ′ ] = 0 \mathcal{L}\left[\frac{1}{n}\sum_{i=1}^{n} z_i'\right] = 0 L [ n 1 ∑ i = 1 n z i ′ ] = 0
Then
L [ 1 n ∑ i = 1 n ( z i ′ − 1 n ∑ j = 1 n z j ′ ) 2 ] ≥ 2. \mathcal{L}\left[\frac{1}{n}\sum_{i=1}^{n}\left(z_i' - \frac{1}{n}\sum_{j=1}^{n}z_j'\right)^2\right] \ge2. L n 1 i = 1 ∑ n ( z i ′ − n 1 j = 1 ∑ n z j ′ ) 2 ≥ 2. Proof. Repeating each index four times does not change the hypotheses or conclusion, so we may assume that n n n z ˉ ′ = 1 n ∑ i = 1 n z i ′ \bar{z}' = \frac{1}{n}\sum_{i=1}^{n} z_i' z ˉ ′ = n 1 ∑ i = 1 n z i ′
L [ 1 n ∑ i = 1 n ( z i ′ − z ˉ ′ ) 2 ] < 2. (30) \mathcal{L}\left[\frac{1}{n}\sum_{i=1}^{n}(z_i' - \bar{z}')^2\right] < 2. \tag*{(30)} L [ n 1 i = 1 ∑ n ( z i ′ − z ˉ ′ ) 2 ] < 2. (30) Because w i 2 = w i w_i^2 = w_i w i 2 = w i ( 1 − w i ) 2 = 1 − w i (1-w_i)^2 = 1-w_i ( 1 − w i ) 2 = 1 − w i 0 ≤ L [ w i ] ≤ 1 0 \le\mathcal{L}[w_i] \le1 0 ≤ L [ w i ] ≤ 1 i i i
w i w j − w i − w j + 1 = ( 1 − w i ) ( 1 − w j ) w_iw_j - w_i - w_j + 1 = (1-w_i)(1-w_j) w i w j − w i − w j + 1 = ( 1 − w i ) ( 1 − w j ) is a square modulo the Boolean equalities, so L [ w i w j ] ≥ L [ w i ] + L [ w j ] − 1 \mathcal{L}[w_iw_j] \ge\mathcal{L}[w_i] + \mathcal{L}[w_j] - 1 L [ w i w j ] ≥ L [ w i ] + L [ w j ] − 1 0 ≤ L [ w i w j ] ≤ 1 0 \le\mathcal{L}[w_iw_j] \le1 0 ≤ L [ w i w j ] ≤ 1 z ( 1 ) ≤ ⋯ ≤ z ( n ) z_{(1)} \le\cdots\le z_{(n)} z ( 1 ) ≤ ⋯ ≤ z ( n ) C 1 = z ( n / 4 ) C_1 = z_{(n/4)} C 1 = z ( n /4 ) C 2 = z ( 3 n / 4 + 1 ) C_2 = z_{(3n/4+1)} C 2 = z ( 3 n /4 + 1 ) C 2 − C 1 > 8 C_2 - C_1 > 8 C 2 − C 1 > 8 S + , S − = [ n ] S_+, S_- = [n] S + , S − = [ n ] ∣ S + ∣ = ∣ S − ∣ = n / 4 |S_+| = |S_-| = n/4 ∣ S + ∣ = ∣ S − ∣ = n /4 z i − z j ≥ 8 z_i - z_j \ge8 z i − z j ≥ 8 i ∈ S + i \in S_+ i ∈ S + j ∈ S − j \in S_- j ∈ S −
L [ ∑ i , j = 1 n ( z i ′ − z j ′ ) 2 ] ≥ L [ ∑ i , j = 1 n w i w j ( z i ′ − z j ′ ) 2 ] = ∑ i , j = 1 n ( z i − z j ) 2 L [ w i w j ] ≥ 2 ∑ i ∈ S + , j ∈ S − ( z i − z j ) 2 L [ w i w j ] ≥ 2 ⋅ 8 2 ∑ i ∈ S + , j ∈ S − L [ w i w j ] ≥ 2 ⋅ 8 2 ∑ i ∈ S + , j ∈ S − ( L [ w i ] + L [ w j ] − 1 ) . \begin{aligned}
\mathcal{L}\left[\sum_{i,j=1}^{n}(z_i' - z_j')^2\right]
&\ge\mathcal{L}\left[\sum_{i,j=1}^{n}w_iw_j(z_i' - z_j')^2\right] \\
&= \sum_{i,j=1}^{n}(z_i-z_j)^2\mathcal{L}[w_iw_j] \\
&\ge2\sum_{i\in S_+,j\in S_-}(z_i-z_j)^2\mathcal{L}[w_iw_j] \\
&\ge2\cdot8^2\sum_{i\in S_+,j\in S_-}\mathcal{L}[w_iw_j] \\
&\ge2\cdot8^2\sum_{i\in S_+,j\in S_-}\left(\mathcal{L}[w_i]+\mathcal{L}[w_j]-1\right).
\end{aligned} L [ i , j = 1 ∑ n ( z i ′ − z j ′ ) 2 ] ≥ L [ i , j = 1 ∑ n w i w j ( z i ′ − z j ′ ) 2 ] = i , j = 1 ∑ n ( z i − z j ) 2 L [ w i w j ] ≥ 2 i ∈ S + , j ∈ S − ∑ ( z i − z j ) 2 L [ w i w j ] ≥ 2 ⋅ 8 2 i ∈ S + , j ∈ S − ∑ L [ w i w j ] ≥ 2 ⋅ 8 2 i ∈ S + , j ∈ S − ∑ ( L [ w i ] + L [ w j ] − 1 ) . Since ∣ S + ∣ = ∣ S − ∣ = n / 4 |S_{+}|=|S_{-}|=n/4 ∣ S + ∣ = ∣ S − ∣ = n /4 ∑ i L [ w i ] ≥ 0.99 n \sum_i \mathcal{L}[w_i] \ge0.99n ∑ i L [ w i ] ≥ 0.99 n S + ∪ S − S_{+}\cup S_{-} S + ∪ S − 0.49 n 0.49n 0.49 n
∑ i ∈ S + , j ∈ S − ( L [ w i ] + L [ w j ] − 1 ) = n 4 ∑ i ∈ S + L [ w i ] + n 4 ∑ j ∈ S − L [ w j ] − n 2 16 ≥ n 4 ( 0.49 n ) − n 2 16 > 0.06 n 2 . \sum_{\substack{i\in S_{+},j\in S_{-}}}\left(\mathcal{L}[w_i]+\mathcal{L}[w_j]-1\right)=\frac{n}{4}\sum_{i\in S_{+}}\mathcal{L}[w_i]+\frac{n}{4}\sum_{j\in S_{-}}\mathcal{L}[w_j]-\frac{n^2}{16}
\ge\frac{n}{4}(0.49n)-\frac{n^2}{16}>0.06n^2. i ∈ S + , j ∈ S − ∑ ( L [ w i ] + L [ w j ] − 1 ) = 4 n i ∈ S + ∑ L [ w i ] + 4 n j ∈ S − ∑ L [ w j ] − 16 n 2 ≥ 4 n ( 0.49 n ) − 16 n 2 > 0.06 n 2 . Thus L [ ∑ i , j ( z i ′ − z j ′ ) 2 ] > 6 n 2 \mathcal{L}[\sum_{i,j}(z'_i-z'_j)^2]>6n^2 L [ ∑ i , j ( z i ′ − z j ′ ) 2 ] > 6 n 2
∑ i , j = 1 n ( z i ′ − z j ′ ) 2 = 2 n ∑ i = 1 n ( z i ′ − z ˉ ′ ) 2 , \sum_{i,j=1}^{n}(z'_i-z'_j)^2=2n\sum_{i=1}^{n}(z'_i-\bar z')^2, i , j = 1 ∑ n ( z i ′ − z j ′ ) 2 = 2 n i = 1 ∑ n ( z i ′ − z ˉ ′ ) 2 , we conclude that
L [ 1 n ∑ i = 1 n ( z i ′ − z ˉ ′ ) 2 ] > 3 , \mathcal{L}\left[\frac{1}{n}\sum_{i=1}^{n}(z'_i-\bar z')^2\right]>3, L [ n 1 i = 1 ∑ n ( z i ′ − z ˉ ′ ) 2 ] > 3 , contradicting (30) . Hence we may assume that C 2 − C 1 ≤ 8 C_2-C_1\le8 C 2 − C 1 ≤ 8
Let S S S n / 2 n/2 n /2 z i ′ z'_i z i ′ C ≥ 16 C\ge16 C ≥ 16
z i ′ ∈ [ C − 4 , C + 4 ] for all i ∈ S , z'_i\in[C-4,C+4]\qquad\text{for all }i\in S, z i ′ ∈ [ C − 4 , C + 4 ] for all i ∈ S , because at least n / 4 n/4 n /4 z i ′ z'_i z i ′ ∑ i ∈ S L [ w i ] ≥ 0.49 n \sum_{i\in S}\mathcal{L}[w_i]\ge0.49n ∑ i ∈ S L [ w i ] ≥ 0.49 n
L [ ∑ i ∈ S w i z i ′ ] = ∑ i ∈ S z i L [ w i ] ≥ ( C − 4 ) ∑ i ∈ S L [ w i ] ≥ 0.49 ( C − 4 ) n . (31) \mathcal{L}\left[\sum_{i\in S}w_i z'_i\right]=\sum_{i\in S}z_i\mathcal{L}[w_i]\ge(C-4)\sum_{i\in S}\mathcal{L}[w_i]\ge0.49(C-4)n. \tag*{(31)} L [ i ∈ S ∑ w i z i ′ ] = i ∈ S ∑ z i L [ w i ] ≥ ( C − 4 ) i ∈ S ∑ L [ w i ] ≥ 0.49 ( C − 4 ) n . (31) Likewise,
L [ ( ∑ i ∈ S w i z i ′ ) 2 ] = ∑ i , j ∈ S z i z j L [ w i w j ] ≤ ( C + 4 ) 2 ( n 2 ) 2 . (32) \mathcal{L}\left[\left(\sum_{i\in S}w_i z'_i\right)^2\right]=\sum_{i,j\in S}z_i z_j\mathcal{L}[w_iw_j]\le(C+4)^2\left(\frac{n}{2}\right)^2. \tag*{(32)} L ( i ∈ S ∑ w i z i ′ ) 2 = i , j ∈ S ∑ z i z j L [ w i w j ] ≤ ( C + 4 ) 2 ( 2 n ) 2 . (32) Next, write
z ˉ S ′ : = 1 ∣ S ∣ ∑ i ∈ S z i ′ . \bar z'_S:=\frac{1}{|S|}\sum_{i\in S}z'_i. z ˉ S ′ := ∣ S ∣ 1 i ∈ S ∑ z i ′ . Because ∣ S ∣ = n / 2 |S|=n/2 ∣ S ∣ = n /2
∑ i ∈ S ( z i ′ ) 2 = ∑ i ∈ S ( z i ′ − z ˉ S ′ ) 2 + 1 ∣ S ∣ ( ∑ i ∈ S z i ′ ) 2 . \sum_{i\in S}(z'_i)^2=\sum_{i\in S}(z'_i-\bar z'_S)^2+\frac{1}{|S|}\left(\sum_{i\in S}z'_i\right)^2. i ∈ S ∑ ( z i ′ ) 2 = i ∈ S ∑ ( z i ′ − z ˉ S ′ ) 2 + ∣ S ∣ 1 ( i ∈ S ∑ z i ′ ) 2 . Since z ˉ S ′ \bar z'_S z ˉ S ′ ∑ i ∈ S ( z i ′ − c ) 2 \sum_{i\in S}(z'_i-c)^2 ∑ i ∈ S ( z i ′ − c ) 2 c c c
∑ i ∈ S ( z i ′ − z ˉ S ′ ) 2 ≤ ∑ i ∈ S ( z i ′ − z ˉ ′ ) 2 ≤ ∑ i = 1 n ( z i ′ − z ˉ ′ ) 2 . \sum_{i\in S}(z'_i-\bar z'_S)^2\le\sum_{i\in S}(z'_i-\bar z')^2\le\sum_{i=1}^{n}(z'_i-\bar z')^2. i ∈ S ∑ ( z i ′ − z ˉ S ′ ) 2 ≤ i ∈ S ∑ ( z i ′ − z ˉ ′ ) 2 ≤ i = 1 ∑ n ( z i ′ − z ˉ ′ ) 2 . Therefore,
∑ i ∈ S ( z i ′ ) 2 ≤ ∑ i = 1 n ( z i ′ − z ˉ ′ ) 2 + 2 n ( ∑ i ∈ S z i ′ ) 2 . \sum_{i\in S}(z'_i)^2\le\sum_{i=1}^{n}(z'_i-\bar z')^2+\frac{2}{n}\left(\sum_{i\in S}z'_i\right)^2. i ∈ S ∑ ( z i ′ ) 2 ≤ i = 1 ∑ n ( z i ′ − z ˉ ′ ) 2 + n 2 ( i ∈ S ∑ z i ′ ) 2 . Taking L \mathcal{L} L (30) yields
L [ ∑ i ∈ S ( z i ′ ) 2 ] ≤ 2 n + 2 n L [ ( ∑ i ∈ S z i ′ ) 2 ] . (33) \mathcal{L}\left[\sum_{i\in S}(z'_i)^2\right]\le2n+\frac{2}{n}\mathcal{L}\left[\left(\sum_{i\in S}z'_i\right)^2\right]. \tag*{(33)} L [ i ∈ S ∑ ( z i ′ ) 2 ] ≤ 2 n + n 2 L ( i ∈ S ∑ z i ′ ) 2 . (33) Define
E d r o p : = L [ ( ∑ i ∈ S ( 1 − w i ) z i ′ ) 2 ] . E_{\mathrm{drop}}:=\mathcal{L}\left[\left(\sum_{i\in S}(1-w_i)z'_i\right)^2\right]. E drop := L ( i ∈ S ∑ ( 1 − w i ) z i ′ ) 2 . Apply the degree-6 polynomial weighted Cauchy–Schwarz certificate from the proof of Lemma 3.8 with b i = 1 − w i b_i=1-w_i b i = 1 − w i b i b_i b i
E d r o p ≤ L [ ( ∑ i ∈ S ( 1 − w i ) ) ( ∑ i ∈ S ( 1 − w i ) ( z i ′ ) 2 ) ] E_{\mathrm{drop}}\le\mathcal{L}\left[\left(\sum_{i\in S}(1-w_i)\right)\left(\sum_{i\in S}(1-w_i)(z'_i)^2\right)\right] E drop ≤ L [ ( i ∈ S ∑ ( 1 − w i ) ) ( i ∈ S ∑ ( 1 − w i ) ( z i ′ ) 2 ) ] ≤ 0.01 n L [ ∑ i ∈ S ( 1 − w i ) ( z i ′ ) 2 ] \le0.01n\mathcal{L}\left[\sum_{i\in S}(1-w_i)(z'_i)^2\right] ≤ 0.01 n L [ i ∈ S ∑ ( 1 − w i ) ( z i ′ ) 2 ] ≤ 0.01 n L [ ∑ i ∈ S ( z i ′ ) 2 ] . \le0.01n\mathcal{L}\left[\sum_{i\in S}(z'_i)^2\right]. ≤ 0.01 n L [ i ∈ S ∑ ( z i ′ ) 2 ] . Combining this with (33) , and then using
∑ i ∈ S z i ′ = ∑ i ∈ S ( 1 − w i ) z i ′ + ∑ i ∈ S w i z i ′ , \sum_{i\in S}z'_i=\sum_{i\in S}(1-w_i)z'_i+\sum_{i\in S}w_i z'_i, i ∈ S ∑ z i ′ = i ∈ S ∑ ( 1 − w i ) z i ′ + i ∈ S ∑ w i z i ′ , followed by ( a + b ) 2 ≤ 2 a 2 + 2 b 2 (a+b)^2\le2a^2+2b^2 ( a + b ) 2 ≤ 2 a 2 + 2 b 2
E d r o p ≤ 0.01 n [ 2 n + 2 n L [ ( ∑ i ∈ S z i ′ ) 2 ] ] E_{\mathrm{drop}}\le0.01n\left[2n+\frac{2}{n}\mathcal{L}\left[\left(\sum_{i\in S}z'_i\right)^2\right]\right] E drop ≤ 0.01 n 2 n + n 2 L ( i ∈ S ∑ z i ′ ) 2 ≤ 0.02 n 2 + 0.04 E d r o p + 0.04 L [ ( ∑ i ∈ S w i z i ′ ) 2 ] . \le0.02n^2+0.04E_{\mathrm{drop}}+0.04\mathcal{L}\left[\left(\sum_{i\in S}w_i z'_i\right)^2\right]. ≤ 0.02 n 2 + 0.04 E drop + 0.04 L ( i ∈ S ∑ w i z i ′ ) 2 . Rearranging and applying (32) ,
E d r o p ≤ 1 24 n 2 + 1 96 ( C + 4 ) 2 n 2 ≤ 0.02 C 2 n 2 , E_{\mathrm{drop}}\le\frac{1}{24}n^2+\frac{1}{96}(C+4)^2n^2\le0.02C^2n^2, E drop ≤ 24 1 n 2 + 96 1 ( C + 4 ) 2 n 2 ≤ 0.02 C 2 n 2 , where the last inequality uses C ≥ 16 C\ge16 C ≥ 16
∣ L [ ∑ i ∈ S ( 1 − w i ) z i ′ ] ∣ ≤ E d r o p ≤ 0.15 C n . (34) \left|\mathcal{L}\left[\sum_{i\in S}(1-w_i)z'_i\right]\right|\le\sqrt{E_{\mathrm{drop}}}\le0.15Cn. \tag*{(34)} L [ i ∈ S ∑ ( 1 − w i ) z i ′ ] ≤ E drop ≤ 0.15 C n . (34) From (31) and (34) , and using C ≥ 16 C\ge16 C ≥ 16
L [ ∑ i ∈ S z i ′ ] = L [ ∑ i ∈ S w i z i ′ ] + L [ ∑ i ∈ S ( 1 − w i ) z i ′ ] ≥ 0.35 C n − 0.15 C n ≥ 0.2 C n . \mathcal{L}\left[\sum_{i\in S}z'_i\right]=\mathcal{L}\left[\sum_{i\in S}w_i z'_i\right]+\mathcal{L}\left[\sum_{i\in S}(1-w_i)z'_i\right]\ge0.35Cn-0.15Cn\ge0.2Cn. L [ i ∈ S ∑ z i ′ ] = L [ i ∈ S ∑ w i z i ′ ] + L [ i ∈ S ∑ ( 1 − w i ) z i ′ ] ≥ 0.35 C n − 0.15 C n ≥ 0.2 C n . Condition 4 says L [ ∑ i = 1 n z i ′ ] = 0 \mathcal{L}[\sum_{i=1}^{n} z_i']=0 L [ ∑ i = 1 n z i ′ ] = 0
L [ ∑ i ∈ S z i ′ − 1 2 ∑ i = 1 n z i ′ ] ≥ 0.2 C n . \mathcal{L}\left[\sum_{i\in S} z_i'-\frac{1}{2}\sum_{i=1}^{n}z_i'\right]\ge0.2Cn. L [ i ∈ S ∑ z i ′ − 2 1 i = 1 ∑ n z i ′ ] ≥ 0.2 C n . Another application of Cauchy–Schwarz yields
L [ ( ∑ i ∈ S z i ′ − 1 2 ∑ i = 1 n z i ′ ) 2 ] ≥ 0.04 C 2 n 2 . (35) \mathcal{L}\left[\left(\sum_{i\in S}z_i'-\frac{1}{2}\sum_{i=1}^{n}z_i'\right)^2\right]\ge0.04C^2n^2. \tag*{(35)} L ( i ∈ S ∑ z i ′ − 2 1 i = 1 ∑ n z i ′ ) 2 ≥ 0.04 C 2 n 2 . (35) Let R = [ n ] ∖ S R=[n]\setminus S R = [ n ] ∖ S ∣ R ∣ = n / 2 |R|=n/2 ∣ R ∣ = n /2
∑ i ∈ S z i ′ − 1 2 ∑ i = 1 n z i ′ = 1 2 ( ∑ i ∈ S z i ′ − ∑ j ∈ R z j ′ ) = 1 n ∑ i ∈ S , j ∈ R ( z i ′ − z j ′ ) . \sum_{i\in S}z_i'-\frac{1}{2}\sum_{i=1}^{n}z_i'=\frac{1}{2}\left(\sum_{i\in S}z_i'-\sum_{j\in R}z_j'\right)=\frac{1}{n}\sum_{i\in S,j\in R}(z_i'-z_j'). i ∈ S ∑ z i ′ − 2 1 i = 1 ∑ n z i ′ = 2 1 i ∈ S ∑ z i ′ − j ∈ R ∑ z j ′ = n 1 i ∈ S , j ∈ R ∑ ( z i ′ − z j ′ ) . Applying Cauchy–Schwarz once more,
( 1 n ∑ i ∈ S , j ∈ R ( z i ′ − z j ′ ) ) 2 ≤ 1 4 ∑ i ∈ S , j ∈ R ( z i ′ − z j ′ ) 2 . \left(\frac{1}{n}\sum_{i\in S,j\in R}(z_i'-z_j')\right)^2\le\frac{1}{4}\sum_{i\in S,j\in R}(z_i'-z_j')^2. n 1 i ∈ S , j ∈ R ∑ ( z i ′ − z j ′ ) 2 ≤ 4 1 i ∈ S , j ∈ R ∑ ( z i ′ − z j ′ ) 2 . Combining this with (35) shows
L [ ∑ i ∈ S , j ∈ R ( z i ′ − z j ′ ) 2 ] ≥ 0.16 C 2 n 2 . \mathcal{L}\left[\sum_{i\in S,j\in R}(z_i'-z_j')^2\right]\ge0.16C^2n^2. L i ∈ S , j ∈ R ∑ ( z i ′ − z j ′ ) 2 ≥ 0.16 C 2 n 2 . Since the left-hand side is at most L [ ∑ i , j ( z i ′ − z j ′ ) 2 ] \mathcal{L}[\sum_{i,j}(z_i'-z_j')^2] L [ ∑ i , j ( z i ′ − z j ′ ) 2 ]
2 n ⋅ L [ ∑ i = 1 n ( z i ′ − z i ′ ′ ) 2 ] = L [ ∑ i , j = 1 n ( z i ′ − z j ′ ) 2 ] ≥ 0.16 C 2 n 2 > 4 n 2 , 2n\cdot\mathcal{L}\left[\sum_{i=1}^{n}(z_i'-z_i'')^2\right]=\mathcal{L}\left[\sum_{i,j=1}^{n}(z_i'-z_j')^2\right]\ge0.16C^2n^2>4n^2, 2 n ⋅ L [ i = 1 ∑ n ( z i ′ − z i ′′ ) 2 ] = L [ i , j = 1 ∑ n ( z i ′ − z j ′ ) 2 ] ≥ 0.16 C 2 n 2 > 4 n 2 , again contradicting (30) . This contradiction proves the lemma. □ \square □
Full proof of Lemma 4.14 . Let L 1 , L 2 \mathcal{L}_1,\mathcal{L}_2 L 1 , L 2 ( α , τ , η ∗ n ) (\alpha,\tau,\eta^*n) ( α , τ , η ∗ n ) ν 1 , ν 2 \nu_1,\nu_2 ν 1 , ν 2 μ L j : = L j μ j ′ \mu_{\mathcal{L}_j}:=\mathcal{L}_j\mu'_j μ L j := L j μ j ′ j ∈ { 1 , 2 } j\in\{1,2\} j ∈ { 1 , 2 } L j \mathcal{L}_j L j Y \mathcal{Y} Y y i ′ y'_i y i ′ B η ∗ n ( Y ) \mathcal{B}_{\eta^*n}(\mathcal{Y}) B η ∗ n ( Y ) σ ∈ { ± 1 } k \sigma\in\{\pm1\}^k σ ∈ { ± 1 } k
∣ ⟨ μ L 1 − μ L 2 , σ ⟩ ∣ ≤ O ( 1 m + τ c o v ) . (36) \left|\langle\mu_{\mathcal{L}_1}-\mu_{\mathcal{L}_2},\sigma\rangle\right|\le O\left(\sqrt{\frac{1}{m}+\tau_{\mathrm{cov}}}\right). \tag*{(36)} ∣ ⟨ μ L 1 − μ L 2 , σ ⟩ ∣ ≤ O ( m 1 + τ cov ) . (36) Fix j ∈ { 1 , 2 } j\in\{1,2\} j ∈ { 1 , 2 } σ \sigma σ
a i : = 1 1 / m + τ c o v ⟨ y i − μ L j , σ ⟩ and a i ′ : = 1 1 / m + τ c o v ⟨ y i ′ − μ L j , σ ⟩ . a_i:=\frac{1}{\sqrt{1/m+\tau_{\mathrm{cov}}}}\langle y_i-\mu_{\mathcal{L}_j},\sigma\rangle
\qquad\text{and}\qquad
a'_i:=\frac{1}{\sqrt{1/m+\tau_{\mathrm{cov}}}}\langle y'_i-\mu_{\mathcal{L}_j},\sigma\rangle. a i := 1/ m + τ cov 1 ⟨ y i − μ L j , σ ⟩ and a i ′ := 1/ m + τ cov 1 ⟨ y i ′ − μ L j , σ ⟩ . We first treat the exact case τ = 0 \tau=0 τ = 0 L j \mathcal{L}_j L j B η ∗ n ( Y ) \mathcal{B}_{\eta^*n}(\mathcal{Y}) B η ∗ n ( Y )
w i 2 = w i w_i^2=w_i w i 2 = w i
∑ i = 1 n w i ≥ ( 1 − η ∗ ) n \sum_{i=1}^n w_i\ge(1-\eta^*)n ∑ i = 1 n w i ≥ ( 1 − η ∗ ) n
w i ( a i ′ − a i ) = ( 1 / m + τ c o v ) − 1 / 2 σ T w i ( y i ′ − y i ) = 0 w_i(a'_i-a_i)=(1/m+\tau_{\mathrm{cov}})^{-1/2}\sigma^\mathsf{T}w_i(y'_i-y_i)=0 w i ( a i ′ − a i ) = ( 1/ m + τ cov ) − 1/2 σ T w i ( y i ′ − y i ) = 0
L j [ n − 1 ∑ i = 1 n a i ′ ] = ( 1 / m + τ c o v ) − 1 / 2 ⟨ L j μ ′ − μ L j , σ ⟩ = 0 \mathcal{L}_j[n^{-1}\sum_{i=1}^n a'_i]=(1/m+\tau_{\mathrm{cov}})^{-1/2}\langle\mathcal{L}_j\mu'-\mu_{\mathcal{L}_j},\sigma\rangle=0 L j [ n − 1 ∑ i = 1 n a i ′ ] = ( 1/ m + τ cov ) − 1/2 ⟨ L j μ ′ − μ L j , σ ⟩ = 0
Choosing η ∗ \eta^* η ∗ n / 4 n/4 n /4 a i ′ a'_i a i ′
L j [ 1 n ∑ i = 1 n ( a i ′ − 1 n ∑ ℓ = 1 n a ℓ ′ ) 2 ] ≥ 2. (37) \mathcal{L}_j\left[\frac{1}{n}\sum_{i=1}^n\left(a'_i-\frac{1}{n}\sum_{\ell=1}^n a'_\ell\right)^2\right]\ge2. \tag*{(37)} L j n 1 i = 1 ∑ n ( a i ′ − n 1 ℓ = 1 ∑ n a ℓ ′ ) 2 ≥ 2. (37) By definition of a i ′ a'_i a i ′ μ L j \mu_{\mathcal{L}_j} μ L j
1 1 / m + τ c o v L j [ 1 n ∑ i = 1 n ⟨ y i ′ − μ ′ , σ ⟩ 2 ] . \frac{1}{1/m+\tau_{\mathrm{cov}}}\mathcal{L}_j\left[\frac{1}{n}\sum_{i=1}^n\langle y'_i-\mu',\sigma\rangle^2\right]. 1/ m + τ cov 1 L j [ n 1 i = 1 ∑ n ⟨ y i ′ − μ ′ , σ ⟩ 2 ] . Applying the covariance constraint from B η ∗ n ( Y ) \mathcal{B}_{\eta^*n}(\mathcal{Y}) B η ∗ n ( Y )
1 1 / m + τ c o v L j [ 1 n ∑ i = 1 n ⟨ y i ′ − μ ′ , σ ⟩ 2 ] ≤ L j [ σ T ( diag ( μ ′ ) − μ ′ μ ′ T ) σ ] + m L j [ ∑ ℓ = 1 k λ ℓ ] 1 + m τ c o v ≤ L j [ 1 − ⟨ μ ′ , σ ⟩ 2 ] + m τ c o v 1 + m τ c o v ≤ 1 , \begin{aligned}
\frac{1}{1/m+\tau_{\mathrm{cov}}}\mathcal{L}_j\left[\frac{1}{n}\sum_{i=1}^n\langle y'_i-\mu',\sigma\rangle^2\right]
\le\frac{\mathcal{L}_j[\sigma^\mathsf{T}(\operatorname{diag}(\mu')-\mu'\mu'^\mathsf{T})\sigma]+m\mathcal{L}_j[\sum_{\ell=1}^k\lambda_\ell]}{1+m\tau_{\mathrm{cov}}} \\
&\le\frac{\mathcal{L}_j[1-\langle\mu',\sigma\rangle^2]+m\tau_{\mathrm{cov}}}{1+m\tau_{\mathrm{cov}}}\le1,
\end{aligned} 1/ m + τ cov 1 L j [ n 1 i = 1 ∑ n ⟨ y i ′ − μ ′ , σ ⟩ 2 ] ≤ 1 + m τ cov L j [ σ T ( diag ( μ ′ ) − μ ′ μ ′ T ) σ ] + m L j [ ∑ ℓ = 1 k λ ℓ ] ≤ 1 + m τ cov L j [ 1 − ⟨ μ ′ , σ ⟩ 2 ] + m τ cov ≤ 1 , since μ ′ \mu' μ ′ σ ℓ 2 = 1 \sigma_\ell^2=1 σ ℓ 2 = 1 ℓ ∈ [ k ] \ell\in[k] ℓ ∈ [ k ] (37) . Hence fewer than n / 4 n/4 n /4 a i ′ a'_i a i ′ − σ -\sigma − σ n / 4 n/4 n /4 a i ′ a'_i a i ′ − 20 -20 − 20 { ⟨ y i , σ ⟩ } i ∈ [ n ] \{\langle y_i,\sigma\rangle\}_{i\in[n]} {⟨ y i , σ ⟩ } i ∈ [ n ] 20 1 / m + τ c o v 20\sqrt{1/m+\tau_{\mathrm{cov}}} 20 1/ m + τ cov ⟨ μ L j , σ ⟩ \langle\mu_{\mathcal{L}_j},\sigma\rangle ⟨ μ L j , σ ⟩
By Remark A.2 , the bound on the distance from ⟨ μ L j , σ ⟩ \langle\mu_{\mathcal{L}_j},\sigma\rangle ⟨ μ L j , σ ⟩
Both projected certificate means ⟨ μ L 1 , σ ⟩ \langle\mu_{\mathcal{L}_1,\sigma}\rangle ⟨ μ L 1 , σ ⟩ ⟨ μ L 2 , σ ⟩ \langle\mu_{\mathcal{L}_2,\sigma}\rangle ⟨ μ L 2 , σ ⟩ O ( 1 / m + τ c o v ) O(\sqrt{1/m}+\tau_{\mathrm{cov}}) O ( 1/ m + τ cov ) { ⟨ y i , σ ⟩ } i ∈ [ n ] \{\langle y_i,\sigma\rangle\}_{i\in[n]} {⟨ y i , σ ⟩ } i ∈ [ n ] (36) . By exact ℓ 1 \ell_1 ℓ 1 (36) implies
∥ μ L 1 − μ L 2 ∥ 1 = max σ ∈ { ± 1 } k ⟨ μ L 1 − μ L 2 , σ ⟩ ≤ O ( 1 m + τ c o v ) . \lVert\mu_{\mathcal{L}_1}-\mu_{\mathcal{L}_2}\rVert_1=\max_{\sigma\in\{\pm1\}^k}\langle\mu_{\mathcal{L}_1}-\mu_{\mathcal{L}_2},\sigma\rangle\le O\left(\sqrt{\frac{1}{m}+\tau_{\mathrm{cov}}}\right). ∥ μ L 1 − μ L 2 ∥ 1 = σ ∈ { ± 1 } k max ⟨ μ L 1 − μ L 2 , σ ⟩ ≤ O ( m 1 + τ cov ) . Finally, since ν j \nu_j ν j L j \mathcal{L}_j L j
∥ ν j − μ L j ∥ 1 ≤ α + τ η ∗ n = O ( α ) for each j ∈ { 1 , 2 } . \lVert\nu_j-\mu_{\mathcal{L}_j}\rVert_1\le\alpha+\tau\eta^*n=O(\alpha)\qquad\text{for each }j\in\{1,2\}. ∥ ν j − μ L j ∥ 1 ≤ α + τ η ∗ n = O ( α ) for each j ∈ { 1 , 2 } . Hence
∥ ν 1 − ν 2 ∥ 1 ≤ O ( α + 1 m + τ c o v ) . \lVert\nu_1-\nu_2\rVert_1\le O\left(\alpha+\sqrt{\frac{1}{m}+\tau_{\mathrm{cov}}}\right). ∥ ν 1 − ν 2 ∥ 1 ≤ O ( α + m 1 + τ cov ) . This proves Lemma 4.14. □
Pure DP Lower Bounds
This appendix proves the pure-DP lower bound in Theorem B.5. Its three terms come from three different arguments: classical nonprivate estimation, a centered code packing around the uniform distribution, and a packing inside the simplex in ( k − 1 ) (k-1) ( k − 1 )
Projection Notation for Lower Bounds
We work in ( k − 1 ) (k-1) ( k − 1 )
Definition B.1 (Nonnegative ℓ 1 \ell_1 ℓ 1 R k − 1 \mathbb{R}^{k-1} R k − 1 Δ ≥ 0 k − 1 \Delta_{\geq0}^{k-1} Δ ≥ 0 k − 1
Δ ≥ 0 k − 1 : = { x ∣ ⟨ 1 , x ⟩ ≤ 1 , x ∈ R ≥ 0 k − 1 } . \Delta_{\geq0}^{k-1}:=\{x\mid\langle1,x\rangle\le1,\ x\in\mathbb{R}_{\geq0}^{k-1}\}. Δ ≥ 0 k − 1 := { x ∣ ⟨ 1 , x ⟩ ≤ 1 , x ∈ R ≥ 0 k − 1 } . Definition B.2 (Restriction and Extension Operators). We define Res : R k → R k − 1 \operatorname{Res}:\mathbb{R}^k\to\mathbb{R}^{k-1} Res : R k → R k − 1 and Ext : R k − 1 → R k \operatorname{Ext}:\mathbb{R}^{k-1}\to\mathbb{R}^k Ext : R k − 1 → R k
Res ( v ) : = v [ k − 1 ] \operatorname{Res}(v):=v_{[k-1]} Res ( v ) := v [ k − 1 ] Ext ( v ) : = ( v , 1 − ∑ i ∈ [ k − 1 ] v i ) . \operatorname{Ext}(v):=\left(v,1-\sum_{i\in[k-1]}v_i\right). Ext ( v ) := v , 1 − i ∈ [ k − 1 ] ∑ v i . In words, Res restricts a vector to its first k − 1 k-1 k − 1
Fact B.3 (ℓ 1 \ell_1 ℓ 1 For any u , v ∈ R k − 1 u,v\in\mathbb{R}^{k-1} u , v ∈ R k − 1
∥ Ext ( u ) − Ext ( v ) ∥ 1 = ∥ u − v ∥ 1 + ∣ ∑ i = 1 k − 1 ( u i − v i ) ∣ ∈ [ ∥ u − v ∥ 1 , 2 ∥ u − v ∥ 1 ] . \lVert\operatorname{Ext}(u)-\operatorname{Ext}(v)\rVert_1=\lVert u-v\rVert_1+\left\lvert\sum_{i=1}^{k-1}(u_i-v_i)\right\rvert\in[\lVert u-v\rVert_1,2\lVert u-v\rVert_1]. ∥ Ext ( u ) − Ext ( v ) ∥ 1 = ∥ u − v ∥ 1 + i = 1 ∑ k − 1 ( u i − v i ) ∈ [∥ u − v ∥ 1 , 2 ∥ u − v ∥ 1 ] . Moreover, for any x , y ∈ R k x,y\in\mathbb{R}^k x , y ∈ R k with ∑ i = 1 k x i = ∑ i = 1 k y i \sum_{i=1}^k x_i=\sum_{i=1}^k y_i ∑ i = 1 k x i = ∑ i = 1 k y i (for example x , y ∈ Δ k − 1 x,y\in\Delta^{k-1} x , y ∈ Δ k − 1 ),
∥ Res ( x ) − Res ( y ) ∥ 1 ≤ ∥ x − y ∥ 1 ≤ 2 ∥ Res ( x ) − Res ( y ) ∥ 1 . \lVert\operatorname{Res}(x)-\operatorname{Res}(y)\rVert_1\le\lVert x-y\rVert_1\le2\lVert\operatorname{Res}(x)-\operatorname{Res}(y)\rVert_1. ∥ Res ( x ) − Res ( y ) ∥ 1 ≤ ∥ x − y ∥ 1 ≤ 2 ∥ Res ( x ) − Res ( y ) ∥ 1 . Proof. For the extension inequalities, we can write
Ext ( u ) − Ext ( v ) = ( u − v , 1 − ∑ i = 1 k − 1 u i − ( 1 − ∑ i = 1 k − 1 v i ) ) = ( u − v , − ∑ i = 1 k − 1 ( u i − v i ) ) . \operatorname{Ext}(u)-\operatorname{Ext}(v)=\left(u-v,\,1-\sum_{i=1}^{k-1}u_i-\left(1-\sum_{i=1}^{k-1}v_i\right)\right)=\left(u-v,\,-\sum_{i=1}^{k-1}(u_i-v_i)\right). Ext ( u ) − Ext ( v ) = ( u − v , 1 − i = 1 ∑ k − 1 u i − ( 1 − i = 1 ∑ k − 1 v i ) ) = ( u − v , − i = 1 ∑ k − 1 ( u i − v i ) ) . so
∥ Ext ( u ) − Ext ( v ) ∥ 1 = ∑ i = 1 k − 1 ∣ u i − v i ∣ + ∣ ∑ i = 1 k − 1 ( u i − v i ) ∣ ∈ [ ∥ u − v ∥ 1 , 2 ∥ u − v ∥ 1 ] . \left\|\operatorname{Ext}(u)-\operatorname{Ext}(v)\right\|_1=\sum_{i=1}^{k-1}|u_i-v_i|+\left|\sum_{i=1}^{k-1}(u_i-v_i)\right|\in\left[\left\|u-v\right\|_1,\,2\left\|u-v\right\|_1\right]. ∥ Ext ( u ) − Ext ( v ) ∥ 1 = i = 1 ∑ k − 1 ∣ u i − v i ∣ + i = 1 ∑ k − 1 ( u i − v i ) ∈ [ ∥ u − v ∥ 1 , 2 ∥ u − v ∥ 1 ] . by the triangle inequality. For the restriction inequalities, the lower bound is immediate since restriction can only decrease ℓ 1 \ell_1 ℓ 1 ∑ i x i = ∑ i y i \sum_i x_i=\sum_i y_i ∑ i x i = ∑ i y i x k − y k = − ∑ i = 1 k − 1 ( x i − y i ) x_k-y_k=-\sum_{i=1}^{k-1}(x_i-y_i) x k − y k = − ∑ i = 1 k − 1 ( x i − y i )
∥ x − y ∥ 1 = ∑ i = 1 k − 1 ∣ x i − y i ∣ + ∣ ∑ i = 1 k − 1 ( x i − y i ) ∣ ≤ 2 ∥ Res ( x ) − Res ( y ) ∥ 1 . \left\|x-y\right\|_1=\sum_{i=1}^{k-1}|x_i-y_i|+\left|\sum_{i=1}^{k-1}(x_i-y_i)\right|\leq2\left\|\operatorname{Res}(x)-\operatorname{Res}(y)\right\|_1. ∥ x − y ∥ 1 = i = 1 ∑ k − 1 ∣ x i − y i ∣ + i = 1 ∑ k − 1 ( x i − y i ) ≤ 2 ∥ Res ( x ) − Res ( y ) ∥ 1 . □
We invoke the following result.
Theorem B.4 (Packing Lower Bounds [HKM22]). Let T ≥ 2 T\geq2 T ≥ 2 0 < β ≤ 1 / 3 0<\beta\leq1/3 0 < β ≤ 1/3 P = { P 1 , … , P T } \mathcal{P}=\{P_1,\ldots,P_T\} P = { P 1 , … , P T } P 0 P_0 P 0 P i ∈ P P_i\in\mathcal{P} P i ∈ P ∥ P i − P 0 ∥ T V ≤ γ \left\|P_i-P_0\right\|_{\mathrm{TV}}\leq\gamma ∥ P i − P 0 ∥ TV ≤ γ G = { G 1 , … , G T } \mathcal{G}=\{G_1,\ldots,G_T\} G = { G 1 , … , G T } Y \mathcal{Y} Y ε \varepsilon ε M M M P X ∼ P i [ M ( X ) ∈ G i ] ≥ 1 − β \mathbb{P}_{X\sim P_i}[M(X)\in G_i]\geq1-\beta P X ∼ P i [ M ( X ) ∈ G i ] ≥ 1 − β i ∈ [ T ] i\in[T] i ∈ [ T ]
n ≥ Ω ( log T + log ( 1 / β ) γ ( e 2 ε − 1 ) ) . n\geq\Omega\left(\frac{\log T+\log(1/\beta)}{\gamma(e^{2\varepsilon}-1)}\right). n ≥ Ω ( γ ( e 2 ε − 1 ) log T + log ( 1/ β ) ) . Note that for the usual regime ε ≤ 1 \varepsilon\leq1 ε ≤ 1 e 2 ε − 1 e^{2\varepsilon}-1 e 2 ε − 1 ε \varepsilon ε
We use Theorem B.4 to show the following lower bound under pure DP. Similar lower bounds already exist in [LSY+20]. Here we focus on the dependence on the failure probability β \beta β
Theorem B.5. Assume k ≥ 2 k\geq2 k ≥ 2 0 < α ≤ 1 / 8 0<\alpha\leq1/8 0 < α ≤ 1/8 0 < β ≤ 1 / 3 0<\beta\leq1/3 0 < β ≤ 1/3 0 < ε ≤ 1 0<\varepsilon\leq1 0 < ε ≤ 1 ε \varepsilon ε μ ∈ Δ k − 1 \mu\in\Delta^{k-1} μ ∈ Δ k − 1 n n n m m m μ \mu μ μ ^ \hat{\mu} μ ^ ∥ μ ^ − μ ∥ 1 ≤ α \left\|\hat{\mu}-\mu\right\|_1\leq\alpha ∥ μ ^ − μ ∥ 1 ≤ α 1 − β 1-\beta 1 − β
n = Ω ( k + log ( 1 / β ) α 2 m + k + log ( 1 / β ) α m ε + k log ( 1 / α ) + log ( 1 / β ) ε ) . n=\Omega\left(\frac{k+\log(1/\beta)}{\alpha^2m}+\frac{k+\log(1/\beta)}{\alpha\sqrt{m}\varepsilon}+\frac{k\log(1/\alpha)+\log(1/\beta)}{\varepsilon}\right). n = Ω ( α 2 m k + log ( 1/ β ) + α m ε k + log ( 1/ β ) + ε k log ( 1/ α ) + log ( 1/ β ) ) . Proof. First term: nonprivate estimation. Classical nonprivate estimation of a distribution on [ k ] [k] [ k ] ℓ 1 \ell_1 ℓ 1 α \alpha α β \beta β Ω ( ( k + log ( 1 / β ) ) / α 2 ) \Omega((k+\log(1/\beta))/\alpha^2) Ω (( k + log ( 1/ β )) / α 2 ) [14 ] . Since there are n m nm nm m m m
Third term: packing inside the simplex. From Fact B.3 we have
∥ Res ( μ ) − Res ( μ ′ ) ∥ 1 ≤ ∥ μ − μ ′ ∥ 1 ≤ 2 ∥ Res ( μ ) − Res ( μ ′ ) ∥ 1 . \left\|\operatorname{Res}(\mu)-\operatorname{Res}(\mu')\right\|_1\leq\left\|\mu-\mu'\right\|_1\leq2\left\|\operatorname{Res}(\mu)-\operatorname{Res}(\mu')\right\|_1. ∥ Res ( μ ) − Res ( μ ′ ) ∥ 1 ≤ ∥ μ − μ ′ ∥ 1 ≤ 2 ∥ Res ( μ ) − Res ( μ ′ ) ∥ 1 . If we find some points v 1 , … , v T ∈ Δ ≤ k − 1 v_1,\ldots,v_T\in\Delta_{\leq}^{k-1} v 1 , … , v T ∈ Δ ≤ k − 1 Definition B.1 ) such that ∥ v i − v j ∥ 1 ≥ 3 α \left\|v_i-v_j\right\|_1\geq3\alpha ∥ v i − v j ∥ 1 ≥ 3 α i ≠ j i\neq j i = j μ i = Ext ( v i ) \mu_i=\operatorname{Ext}(v_i) μ i = Ext ( v i )
P i : = μ i ⊗ m for i ∈ [ T ] , P 0 : = μ 1 ⊗ m . P_i:=\mu_i^{\otimes m}\qquad\text{for }i\in[T],\qquad P_0:=\mu_1^{\otimes m}. P i := μ i ⊗ m for i ∈ [ T ] , P 0 := μ 1 ⊗ m . Since total variation distance between any two probability distributions is at most 1, we have ∥ P i − P 0 ∥ T V ≤ 1 \lVert P_i-P_0\rVert_{\mathrm{TV}} \le1 ∥ P i − P 0 ∥ TV ≤ 1 i ∈ [ T ] i \in[T] i ∈ [ T ] γ = 1 \gamma=1 γ = 1 { P i } i ∈ [ T ] \{P_i\}_{i\in[T]} { P i } i ∈ [ T ]
G i = { μ ^ ∣ ∥ μ ^ − μ i ∥ 1 ≤ α } . G_i=\{\hat{\mu}\mid\lVert\hat{\mu}-\mu_i\rVert_1\le\alpha\}. G i = { μ ^ ∣ ∥ μ ^ − μ i ∥ 1 ≤ α } . To construct the 3 α 3\alpha 3 α v 1 , … , v T v_1,\ldots,v_T v 1 , … , v T Δ ≤ k − 1 \Delta_{\le}^{k-1} Δ ≤ k − 1 C s , k = { x ∣ ∥ x ∥ 1 ≤ 1 , s i x i ≥ 0 } C_{s,k}=\{x\mid\lVert x\rVert_1\le1,\ s_i x_i\ge0\} C s , k = { x ∣ ∥ x ∥ 1 ≤ 1 , s i x i ≥ 0 } s ∈ { ± 1 } k − 1 s\in\{\pm1\}^{k-1} s ∈ { ± 1 } k − 1 ℓ 1 \ell_1 ℓ 1 R k − 1 \mathbb{R}^{k-1} R k − 1 T ′ T' T ′ 3 α 3\alpha 3 α T = 2 − k + 1 T ′ T=2^{-k+1}T' T = 2 − k + 1 T ′ Δ ≤ k − 1 \Delta_{\le}^{k-1} Δ ≤ k − 1 ℓ 1 \ell_1 ℓ 1 T ′ ≥ ( 1 / ( 3 α ) ) k − 1 T'\ge(1/(3\alpha))^{k-1} T ′ ≥ ( 1/ ( 3 α ) ) k − 1 T ≥ ( 1 / ( 6 α ) ) k − 1 T\ge(1/(6\alpha))^{k-1} T ≥ ( 1/ ( 6 α ) ) k − 1 α ≤ 1 / 8 \alpha\le1/8 α ≤ 1/8
n ≥ Ω ( k log ( 1 / α ) + log ( 1 / β ) ε ) . n\ge\Omega\left(\frac{k\log(1/\alpha)+\log(1/\beta)}{\varepsilon}\right). n ≥ Ω ( ε k log ( 1/ α ) + log ( 1/ β ) ) . Second term: centered code packing around the uniform distribution. Let μ o \mu_o μ o 2 ⌊ k / 2 ⌋ 2\lfloor k/2\rfloor 2 ⌊ k /2 ⌋ [ k ] [k] [ k ] x ∈ { ± 1 } ⌊ k / 2 ⌋ x\in\{\pm1\}^{\lfloor k/2\rfloor} x ∈ { ± 1 } ⌊ k /2 ⌋ s ( x ) ∈ { − 1 , 0 , + 1 } k s(x)\in\{-1,0,+1\}^k s ( x ) ∈ { − 1 , 0 , + 1 } k
s ( x ) 2 u − 1 = x u , s ( x ) 2 u = − x u for u ∈ [ ⌊ k / 2 ⌋ ] , s(x)_{2u-1}=x_u,\qquad s(x)_{2u}=-x_u\qquad\text{for }u\in[\lfloor k/2\rfloor], s ( x ) 2 u − 1 = x u , s ( x ) 2 u = − x u for u ∈ [⌊ k /2 ⌋] , and if k k k s ( x ) s(x) s ( x )
μ ( x ) : = μ o + 4 α 2 ⌊ k / 2 ⌋ s ( x ) . \mu(x):=\mu_o+\frac{4\alpha}{2\lfloor k/2\rfloor}s(x). μ ( x ) := μ o + 2 ⌊ k /2 ⌋ 4 α s ( x ) . Since ∑ i = 1 k s ( x ) i = 0 \sum_{i=1}^k s(x)_i=0 ∑ i = 1 k s ( x ) i = 0 4 α ≤ 1 / 2 4\alpha\le1/2 4 α ≤ 1/2 α ≤ 1 / 8 \alpha\le1/8 α ≤ 1/8 μ ( x ) ∈ Δ k − 1 \mu(x)\in\Delta^{k-1} μ ( x ) ∈ Δ k − 1 x x x x x x
KL ( μ ( x ) ∥ μ o ) = ∑ i = 1 2 ⌊ k / 2 ⌋ μ ( x ) ( i ) log ( μ ( x ) ( i ) μ o ( i ) ) = 1 2 ⌊ k / 2 ⌋ ∑ i = 1 2 ⌊ k / 2 ⌋ ( 1 + 4 α s ( x ) i ) log ( 1 + 4 α s ( x ) i ) . \begin{aligned}
\operatorname{KL}(\mu(x)\mathbin\Vert\mu_o)=\sum_{i=1}^{2\lfloor k/2\rfloor}\mu(x)(i)\log\left(\frac{\mu(x)(i)}{\mu_o(i)}\right) \\
&=\frac{1}{2\lfloor k/2\rfloor}\sum_{i=1}^{2\lfloor k/2\rfloor}(1+4\alpha s(x)_i)\log(1+4\alpha s(x)_i).
\end{aligned} KL ( μ ( x ) ∥ μ o ) = i = 1 ∑ 2 ⌊ k /2 ⌋ μ ( x ) ( i ) log ( μ o ( i ) μ ( x ) ( i ) ) = 2 ⌊ k /2 ⌋ 1 i = 1 ∑ 2 ⌊ k /2 ⌋ ( 1 + 4 α s ( x ) i ) log ( 1 + 4 α s ( x ) i ) . For ∣ t ∣ ≤ 1 / 2 |t|\le1/2 ∣ t ∣ ≤ 1/2 ( 1 + t ) log ( 1 + t ) ≤ t + 2 t 2 (1+t)\log(1+t)\le t+2t^2 ( 1 + t ) log ( 1 + t ) ≤ t + 2 t 2 t = 4 α s ( x ) i t=4\alpha s(x)_i t = 4 α s ( x ) i ∑ i s ( x ) i = 0 \sum_i s(x)_i=0 ∑ i s ( x ) i = 0
KL ( μ ( x ) ∥ μ o ) ≤ 32 α 2 2 ⌊ k / 2 ⌋ ∑ i = 1 2 ⌊ k / 2 ⌋ s ( x ) i 2 ≤ 32 α 2 = O ( α 2 ) . \operatorname{KL}(\mu(x)\mathbin\Vert\mu_o)\le\frac{32\alpha^2}{2\lfloor k/2\rfloor}\sum_{i=1}^{2\lfloor k/2\rfloor}s(x)_i^2\le32\alpha^2=O(\alpha^2). KL ( μ ( x ) ∥ μ o ) ≤ 2 ⌊ k /2 ⌋ 32 α 2 i = 1 ∑ 2 ⌊ k /2 ⌋ s ( x ) i 2 ≤ 32 α 2 = O ( α 2 ) . By tensorization of KL and Pinsker’s inequality,
∥ μ ( x ) ⊗ m − μ o ⊗ m ∥ T V ≤ O ( α m ) . \lVert\mu(x)^{\otimes m}-\mu_o^{\otimes m}\rVert_{\mathrm{TV}}\le O(\alpha\sqrt{m}). ∥ μ ( x ) ⊗ m − μ o ⊗ m ∥ TV ≤ O ( α m ) . A greedy binary-code packing gives x 1 , … , x M ∈ { ± 1 } ⌊ k / 2 ⌋ x_1,\ldots,x_M\in\{\pm1\}^{\lfloor k/2\rfloor} x 1 , … , x M ∈ { ± 1 } ⌊ k /2 ⌋ M = 2 Ω ( ⌊ k / 2 ⌋ ) = 2 Ω ( k ) M=2^{\Omega(\lfloor k/2\rfloor)}=2^{\Omega(k)} M = 2 Ω (⌊ k /2 ⌋) = 2 Ω ( k ) ⌊ k / 2 ⌋ / 4 \lfloor k/2\rfloor/4 ⌊ k /2 ⌋ /4
∑ j = 0 ⌊ ⌊ k / 2 ⌋ / 4 ⌋ ( ⌊ k / 2 ⌋ j ) ≤ 2 H 2 ( 1 / 4 ) ⌊ k / 2 ⌋ . \sum_{j=0}^{\lfloor\lfloor k/2\rfloor/4\rfloor}\binom{\lfloor k/2\rfloor}{j}\le2^{H_2(1/4)\lfloor k/2\rfloor}. j = 0 ∑ ⌊⌊ k /2 ⌋ /4 ⌋ ( j ⌊ k /2 ⌋ ) ≤ 2 H 2 ( 1/4 ) ⌊ k /2 ⌋ . words, where H 2 ( 1 / 4 ) < 1 H_2(1/4) < 1 H 2 ( 1/4 ) < 1 i ≠ j i \ne j i = j
∥ μ ( x i ) − μ ( x j ) ∥ 1 = 8 α ⌊ k / 2 ⌋ d H ( x i , x j ) > 4 α 2 = 2 α . \lVert\mu(x_i) - \mu(x_j) \rVert_1 = \frac{8\alpha}{\lfloor k/2 \rfloor} d_H(x_i, x_j) > \frac{4\alpha}{2} = 2\alpha. ∥ μ ( x i ) − μ ( x j ) ∥ 1 = ⌊ k /2 ⌋ 8 α d H ( x i , x j ) > 2 4 α = 2 α . Thus the closed ℓ 1 \ell_1 ℓ 1 α \alpha α Theorem B.4 with γ = min { 1 , O ( α m ) } \gamma= \min\{1, O(\alpha\sqrt{m})\} γ = min { 1 , O ( α m )}
n ≥ Ω ( k + log ( 1 / β ) α m ε ) . n \ge\Omega\left(\frac{k + \log(1/\beta)}{\alpha\sqrt{m}\varepsilon}\right). n ≥ Ω ( α m ε k + log ( 1/ β ) ) . 