Introduction
Determining whether a collection of covariates is associated with a response is a classical problem in statistical inference. In high-dimensional regression, this question can be formulated as the global hypothesis testing problem
H 0 : β = 0 against H 1 : ∥ β ∥ 2 ≥ r , H_0:\ \beta=0 \qquad\text{against} \qquad H_1:\ \lVert\beta\rVert_2 \ge r, H 0 : β = 0 against H 1 : ∥ β ∥ 2 ≥ r , where β ∈ R d \beta\in\mathbb{R}^d β ∈ R d r r r d = o ( n ) d=o(n) d = o ( n ) ℓ 2 \ell_2 ℓ 2 d / n \sqrt{d/n} d / n d 1 / 4 / n d^{1/4}/\sqrt{n} d 1/4 / n [26 ] .
The gap between testing and estimation makes signal detection particularly relevant in high-dimensional applications, many of which involve sensitive data such as genomic records or consumer transactions. This motivates studying how hypothesis testing can be carried out while protecting individual-level data. A leading framework for privacy is differential privacy (DP) [17 ] , which has been widely used in both industry (e.g. [16 ] ) and government applications (e.g. Abowd and Hawes [1 ] ). At a high level, differential privacy requires the distribution of a released output to be stable under replacement of a single observation.
Intuitively, stronger privacy guarantees require larger signals to reliably distinguish between the null and alternative hypotheses. We study how the minimax separation rate depends on the sample size, dimension, sparsity level, and privacy parameters, as well as on the regression model and its nuisance parameters.
Contributions
We characterise, up to polylogarithmic factors, the minimax separation rates for private signal detection across sparsity levels in linear regression with an unknown error distribution and generalised linear models (GLMs) with an unknown intercept.
In the dense regime, the rates for linear regression and GLMs with two response support points match the private Gaussian mean testing rate established by Narayanan [36 ] . We attain this rate through a transformation that reduces regression testing to mean testing and accommodates the unknown nuisance parameters. In particular, for linear regression, no moment, symmetry or smoothness assumptions on the errors are needed.
Our lower bounds identify when nuisance adjustment can absorb changes in the response marginal caused by a regression signal. This affects both the non-private and private detection rates. For GLMs with more than two response support points, the response marginal retains second-order information about the signal despite the unknown intercept, allowing faster detection of sufficiently small signals when d ≳ n d \gtrsim n d ≳ n
To exploit sparsity, we test for non-zero covariances between bounded transformations of the response and sparse covariate projections. Combining the sparse covariance test with the mean-reduction and response-only tests yields a procedure that adapts to the unknown sparsity level. As a byproduct, we characterise the minimax separation rates for private sparse mean testing of sub-Gaussian vectors up to polylogarithmic factors. To our knowledge, this characterisation under central DP is new and we present it in Appendix § 12 as a result of independent interest.
We also construct polynomial-time tests that match the minimax rates up to polylogarithmic factors when ϵ ≳ n − 1 / 2 \epsilon\gtrsim n^{-1/2} ϵ ≳ n − 1/2
Related work
Non-private regression testing. Without privacy constraints, global hypothesis testing in high-dimensional regression has been studied extensively, so we only list a few examples. Ingster et al. [26 ] established the asymptotic detection boundaries for Gaussian linear regression across sparsity levels, distinguishing known and unknown error variance, while Carpentier et al. (2022) obtained non-asymptotic separation rates under sub-Gaussian errors with unknown variance. Mukherjee et al. [35 ] derived analogous detection boundaries in the context of binary regression. For GLMs, Goeman et al. [20 ] developed asymptotic theory for a low dimensional null against a high dimensional alternative under canonical link functions, allowing for nuisance parameters; this framework was extended in Guo and Chen [23 ] to accommodate diverging dimension and a broader class of link functions.
Private regression inference. Early work developed private procedures for ordinary least squares (OLS), including confidence intervals and approximations to classical t t t [41 , 7 ] . Alabi and Vadhan [4 ] constructed differentially private F F F [32 ] . For M M M [5 ] developed asymptotic theory for private analogues of the Wald, score and likelihood ratio tests. Bayesian approaches for private inference have been developed for linear regression [8 ] and for logistic and Poisson regression [30 ] . Hung and Yu [25 ] consider private hypothesis testing with censored survival data in the Cox regression model. For high-dimensional sparse linear regression, Cai et al. [10 ] developed differentially private methods for debiased inference on regression coefficients and multiple testing with false discovery rate control. These contributions primarily study private analogues of classical inferential procedures and their validity. Recent work also studies minimax support recovery in sparse linear regression with bounded covariates (Gu and Cai, 2026).
Optimality in private hypothesis testing. Another line of work develops finite-sample private tests together with optimality results. Canonne et al. [11 ] characterised the sample complexity of private testing between simple hypotheses, while Awan and Slavković [6 ] derived uniformly most powerful private tests for binomial data. For distribution testing, Acharya et al. [2 ] established optimal rates for identity testing of discrete distributions and closeness testing in the sparse regime. Canonne et al. [12 ] developed identity tests for high-dimensional product Bernoulli distributions and Gaussians with known covariance, and the Gaussian sample complexity bounds were sharpened by Narayanan [36 ] . More recently, private nonparametric testing has been studied through minimax analysis of permutation tests [29 ] and smooth two-sample testing [28 ] . The central DP rate for the white-noise-with-drift model follows as a special case of the federated DP framework of Cai et al. [9 ] . In the local model of DP, minimax testing has also been studied for simple hypotheses (e.g. Pensia et al., [39 ] ) and for goodness-of-fit testing in discrete and continuous settings (e.g. Lam-Weil et al., [31 ] ).
Hence, to the best of our knowledge, minimax separation rates for private signal detection in regression remain largely unresolved.
Organisation and notation
Section 2 introduces the models and states the main results. Sections 3 and 4 develop the lower bounds and corresponding tests. Section 5 presents polynomial-time implementations and numerical experiments, and Section 6 discusses future directions. Additional technical details are deferred to the appendices.
We use the following notation. For a positive integer n n n [ n ] : = { 1 , 2 , … , n } [n] := \{1,2,\ldots,n\} [ n ] := { 1 , 2 , … , n } x ∈ R x \in\mathbb{R} x ∈ R a ≤ b a \le b a ≤ b [ x ] a b : = min { b , max { a , x } } [x]^b_a := \min\{b,\max\{a,x\}\} [ x ] a b := min { b , max { a , x }} v ∈ R d v \in\mathbb{R}^d v ∈ R d ∥ v ∥ 2 \lVert v\rVert_2 ∥ v ∥ 2 ∥ v ∥ 0 : = ∣ { j ∈ [ d ] : v j ≠ 0 } ∣ \lVert v\rVert_0 := |\{j \in[d] : v_j \ne0\}| ∥ v ∥ 0 := ∣ { j ∈ [ d ] : v j = 0 } ∣ X ∈ R X \in\mathbb{R} X ∈ R ∥ X ∥ ψ 2 : = inf { c > 0 : E [ exp ( X 2 / c 2 ) ] ≤ 2 } < ∞ \lVert X\rVert_{\psi_2} := \inf\{c > 0 : \mathbb{E}[\exp(X^2/c^2)] \le2\} < \infty ∥ X ∥ ψ 2 := inf { c > 0 : E [ exp ( X 2 / c 2 )] ≤ 2 } < ∞ X ∈ R d X \in\mathbb{R}^d X ∈ R d ∥ X ∥ ψ 2 : = sup u : ∥ u ∥ 2 ≤ 1 ∥ u ⊤ X ∥ ψ 2 < ∞ \lVert X\rVert_{\psi_2} := \sup_{u:\lVert u\rVert_2 \le1}\lVert u^\top X\rVert_{\psi_2} < \infty ∥ X ∥ ψ 2 := sup u : ∥ u ∥ 2 ≤ 1 ∥ u ⊤ X ∥ ψ 2 < ∞ P P P P ⊗ n P^{\otimes n} P ⊗ n n n n C C C c c c c 1 , … c_1,\ldots c 1 , … n n n d d d s ∗ s^* s ∗ ϵ \epsilon ϵ δ \delta δ { a n } \{a_n\} { a n } { b n } \{b_n\} { b n } a n ≲ b n a_n \lesssim b_n a n ≲ b n C > 0 C > 0 C > 0 a n ≤ C b n a_n \le Cb_n a n ≤ C b n n n n a n ≍ b n a_n \asymp b_n a n ≍ b n a n ≲ b n a_n \lesssim b_n a n ≲ b n b n ≲ a n b_n \lesssim a_n b n ≲ a n ≲ log \lesssim_{\log} ≲ l o g ≍ log \asymp_{\log} ≍ l o g a n = O ( b n ) a_n = O(b_n) a n = O ( b n ) ∣ a n ∣ ≲ b n |a_n| \lesssim b_n ∣ a n ∣ ≲ b n a n = o ( b n ) a_n = o(b_n) a n = o ( b n ) a n / b n → 0 a_n/b_n \to0 a n / b n → 0 n → ∞ n \to\infty n → ∞ b n > 0 b_n > 0 b n > 0 Lap ( b ) \operatorname{Lap}(b) Lap ( b ) b b b D T V D_{\mathrm{TV}} D TV D K L D_{\mathrm{KL}} D KL D H a m D_{\mathrm{Ham}} D Ham
Problem setup and main results
We observe n n n { ( X i , Y i ) } i = 1 n \{(X_i,Y_i)\}_{i=1}^n {( X i , Y i ) } i = 1 n R d × Y \mathbb{R}^d \times\mathcal{Y} R d × Y Y Y Y X = x X = x X = x x T β x^{\mathsf T}\beta x T β ν ∈ N \nu\in\mathcal{N} ν ∈ N β ∈ R d \beta\in\mathbb{R}^d β ∈ R d ν \nu ν
H 0 : β = 0 against H 1 ( r , R ) : r ≤ ∥ β ∥ 2 ≤ R (1) H_0 : \beta= 0 \qquad\text{against} \qquad H_1(r,R) : r \le\lVert\beta\rVert_2 \le R \tag*{(1)} H 0 : β = 0 against H 1 ( r , R ) : r ≤ ∥ β ∥ 2 ≤ R (1) for 0 < r ≤ R ≤ ∞ 0 < r \le R \le\infty 0 < r ≤ R ≤ ∞ R = r 0 R = r_0 R = r 0 r 0 > 0 r_0 > 0 r 0 > 0
Assumption 1 (Covariate distribution). Fix constants 0 < λ ‾ ≤ λ ‾ 0 < \underline{\lambda} \le\overline{\lambda} 0 < λ ≤ λ C X ≥ 8 λ / 3 C_X \ge\sqrt{8\lambda/3} C X ≥ 8 λ /3 E [ X ] = 0 \mathbb{E}[X] = 0 E [ X ] = 0 λ ‾ I d ⪯ E [ X X T ] ⪯ λ ‾ I d \underline{\lambda} I_d \preceq\mathbb{E}[XX^{\mathsf T}] \preceq\overline{\lambda} I_d λ I d ⪯ E [ X X T ] ⪯ λ I d ∥ X ∥ ψ 2 ≤ C X \lVert X\rVert_{\psi_2} \le C_X ∥ X ∥ ψ 2 ≤ C X
The restriction on C X C_X C X N ( 0 , λ ‾ I d ) N(0,\underline{\lambda} I_d) N ( 0 , λ I d )
Model classes
We focus on two classes of regression models, which have different nuisance parameters.
Linear regression. We consider the model
Y = X T β + ξ , ξ ⊥ X (2) Y = X^{\mathsf T}\beta+ \xi,\qquad\xi\perp X \tag*{(2)} Y = X T β + ξ , ξ ⊥ X (2) where the distribution of ξ \xi ξ
Assumption 2 (Linear regression error). There exist fixed constants M ξ > 0 M_\xi> 0 M ξ > 0 p ξ ∈ ( 0 , 1 ) p_\xi\in(0,1) p ξ ∈ ( 0 , 1 ) P ( ∣ ξ ∣ ≤ M ξ ) ≥ p ξ \mathbb{P}(|\xi| \le M_\xi) \ge p_\xi P ( ∣ ξ ∣ ≤ M ξ ) ≥ p ξ
Assumption 2 only requires that errors have bounded magnitude with a fixed positive probability. In particular, it imposes no smoothness, symmetry, or tail assumptions on the error distribution.
Generalised linear models. We consider GLMs based on a fixed non-degenerate, one-parameter natural exponential family, with open natural parameter space Θ \Theta Θ X = x X = x X = x
p β , γ ( y ∣ x ) = h ( y ) exp { y ϑ ( γ + x T β ) − b ( ϑ ( γ + x T β ) ) } . (3) p_{\beta,\gamma}(y \mid x) = h(y)\exp\left\{y\vartheta(\gamma+ x^{\mathsf T}\beta) - b\left(\vartheta(\gamma+ x^{\mathsf T}\beta)\right)\right\}. \tag*{(3)} p β , γ ( y ∣ x ) = h ( y ) exp { y ϑ ( γ + x T β ) − b ( ϑ ( γ + x T β ) ) } . (3) Here, h : Y → R ≥ 0 h : \mathcal{Y} \to\mathbb{R}_{\ge0} h : Y → R ≥ 0 ϑ : R → Θ \vartheta: \mathbb{R} \to\Theta ϑ : R → Θ b b b γ ∈ Γ \gamma\in\Gamma γ ∈ Γ
Assumption 3 (GLM regularity). We assume that ϑ \vartheta ϑ R \mathbb{R} R Γ \Gamma Γ U ⊃ Γ U \supset\Gamma U ⊃ Γ ϑ ∈ C 3 ( U ) \vartheta\in C^3(U) ϑ ∈ C 3 ( U ) inf a ∈ U ∣ ϑ ′ ( a ) ∣ > 0 \inf_{a \in U} |\vartheta'(a)| > 0 inf a ∈ U ∣ ϑ ′ ( a ) ∣ > 0
The conditions in Assumption 3 hold for Bernoulli regression with logit or probit links, and Poisson and exponential regression with log links, among others. The global monotonicity condition ensures that the effect of the linear predictor does not reverse outside the intercept region. The remaining conditions provide the smoothness needed for local likelihood expansions and ensure that the response distribution remains locally sensitive to changes in the linear predictor.
Preliminaries on differential privacy
Definition 1 ((ϵ , δ \epsilon, \delta ϵ , δ ϵ > 0 \epsilon> 0 ϵ > 0 δ ≥ 0 \delta\ge0 δ ≥ 0 M M M ( ϵ , δ ) (\epsilon, \delta) ( ϵ , δ ) R \mathcal{R} R
M ( R ∈ A ∣ D ) ≤ e ϵ M ( R ∈ A ∣ D ′ ) + δ M(R \in A \mid D) \le e^\epsilon M(R \in A \mid D') + \delta M ( R ∈ A ∣ D ) ≤ e ϵ M ( R ∈ A ∣ D ′ ) + δ for any measurable set A A A σ \sigma σ R \mathcal{R} R ( D , D ′ ) (D, D') ( D , D ′ )
A test ϕ \phi ϕ { 0 , 1 } \{0,1\} { 0 , 1 } ϕ = 1 \phi= 1 ϕ = 1
Lemma 1 (Post-processing). Let M : D → R M : \mathcal{D} \to\mathcal{R} M : D → R ( ϵ , δ ) (\epsilon, \delta) ( ϵ , δ ) f : R → S f : \mathcal{R} \to\mathcal{S} f : R → S M ( D ) M(D) M ( D ) f ∘ M f \circ M f ∘ M ( ϵ , δ ) (\epsilon, \delta) ( ϵ , δ )
Many of our tests will be based on privatising test statistics using the Laplace mechanism.
Lemma 2 (Laplace mechanism). Let Δ T : = sup D , D ′ : D H a m ( D , D ′ ) = 1 ∣ T ( D ) − T ( D ′ ) ∣ \Delta_T := \sup_{D,D':D_{\mathrm{Ham}}(D,D')=1} |T(D)-T(D')| Δ T := sup D , D ′ : D Ham ( D , D ′ ) = 1 ∣ T ( D ) − T ( D ′ ) ∣ T : D → R T : \mathcal{D} \to\mathbb{R} T : D → R
M ( D ) = T ( D ) + W , W ∼ Lap ( Δ ϵ ) , M(D) = T(D) + W, \qquad W \sim\operatorname{Lap}\left(\frac{\Delta}{\epsilon}\right), M ( D ) = T ( D ) + W , W ∼ Lap ( ϵ Δ ) , is ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) Δ ≥ Δ T \Delta\ge\Delta_T Δ ≥ Δ T τ ∈ R \tau\in\mathbb{R} τ ∈ R ϕ ( D ) : = 1 { M ( D ) > τ } \phi(D) := \mathbb{1}\{M(D) > \tau\} ϕ ( D ) := 1 { M ( D ) > τ } ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 )
Minimax separation rates
For a class of joint distributions P = { P β , ν : β ∈ R d , ν ∈ N } \mathcal{P} = \{P_{\beta,\nu} : \beta\in\mathbb{R}^d, \nu\in\mathcal{N}\} P = { P β , ν : β ∈ R d , ν ∈ N } ν \nu ν 1 ≤ s ∗ ≤ d 1 \le s^* \le d 1 ≤ s ∗ ≤ d 0 < r < R ≤ ∞ 0 < r < R \le\infty 0 < r < R ≤ ∞
P 0 : = { P 0 , ν : ν ∈ N } , P 1 ( s ∗ , r , R ) : = { P β , ν : r ≤ ∥ β ∥ 2 ≤ R , ∥ β ∥ 0 ≤ s ∗ , ν ∈ N } . \mathcal{P}_0 := \{P_{0,\nu} : \nu\in\mathcal{N}\}, \qquad\mathcal{P}_1(s^*,r,R) := \{P_{\beta,\nu} : r \le\lVert\beta\rVert_2 \le R,\ \lVert\beta\rVert_0 \le s^*,\ \nu\in\mathcal{N}\}. P 0 := { P 0 , ν : ν ∈ N } , P 1 ( s ∗ , r , R ) := { P β , ν : r ≤ ∥ β ∥ 2 ≤ R , ∥ β ∥ 0 ≤ s ∗ , ν ∈ N } . We define the minimax separation radius as
r ∗ ( n , d , s ∗ , ϵ , δ , P , R ) : = inf { r ∈ ( 0 , R ) : ∃ ( ϵ , δ ) -DP test ϕ such that r^*(n,d,s^*,\epsilon,\delta,\mathcal{P},R) := \inf\{r \in(0,R) : \exists\;(\epsilon,\delta)\text{-DP test }\phi\text{ such that} r ∗ ( n , d , s ∗ , ϵ , δ , P , R ) := inf { r ∈ ( 0 , R ) : ∃ ( ϵ , δ ) -DP test ϕ such that sup P ∈ P 0 P ⊗ n ( ϕ = 1 ) + sup P ∈ P 1 ( s ∗ , r , R ) P ⊗ n ( ϕ = 0 ) ≤ 1 3 } . \sup_{P\in\mathcal{P}_0}P^{\otimes n}(\phi=1)+\sup_{P\in\mathcal{P}_1(s^*,r,R)}P^{\otimes n}(\phi=0)\leq\frac{1}{3}\}. P ∈ P 0 sup P ⊗ n ( ϕ = 1 ) + P ∈ P 1 ( s ∗ , r , R ) sup P ⊗ n ( ϕ = 0 ) ≤ 3 1 } . where the probabilities are taken over the randomness of the samples and the private test. The threshold 1 / 3 1/3 1/3 ( 0 , 1 ) (0,1) ( 0 , 1 )
Let P L i n \mathcal{P}_{\mathrm{Lin}} P Lin ( X , Y ) (X,Y) ( X , Y ) β \beta β F \mathcal{F} F P G L M ( F ) \mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P GLM ( F ) β \beta β γ \gamma γ
ϱ s ∗ , D : = min { D 1 / 4 n 1 / 2 + D 1 / 4 n 3 / 4 ϵ 1 / 2 , s ∗ n } + 1 n ϵ and ℓ n , d , ϵ : = 1 + log { ( n + 1 ) ( d + 1 ) ( 1 + ϵ − 1 ) } . (4) \varrho_{s^*,D}:=\min\left\{\frac{D^{1/4}}{n^{1/2}}+\frac{D^{1/4}}{n^{3/4}\epsilon^{1/2}},\sqrt{\frac{s^*}{n}}\right\}+\frac{1}{n\epsilon}
\quad\text{and}\quad
\ell_{n,d,\epsilon}:=1+\log\{(n+1)(d+1)(1+\epsilon^{-1})\}. \tag*{(4)} ϱ s ∗ , D := min { n 1/2 D 1/4 + n 3/4 ϵ 1/2 D 1/4 , n s ∗ } + n ϵ 1 and ℓ n , d , ϵ := 1 + log {( n + 1 ) ( d + 1 ) ( 1 + ϵ − 1 )} . (4) The following theorem gives the minimax separation rate for linear regression and GLMs with two-point response support, and the local minimax separation rate for GLMs with larger response support.
Theorem 3. Fix ϵ > 0 \epsilon>0 ϵ > 0 0 ≤ δ < 1 0\leq\delta<1 0 ≤ δ < 1 δ ≤ c δ ϵ \delta\leq c_\delta\epsilon δ ≤ c δ ϵ c δ > 0 c_\delta>0 c δ > 0 F \mathcal{F} F
For P = P L i n \mathcal{P}=\mathcal{P}_{\mathrm{Lin}} P = P Lin P = P G L M ( F ) \mathcal{P}=\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P = P GLM ( F ) c P > 0 c_{\mathcal{P}}>0 c P > 0 ℓ n , d , ϵ 3 ϱ s ∗ , d ≤ c P \ell_{n,d,\epsilon}^{3}\varrho_{s^*,d}\leq c_{\mathcal{P}} ℓ n , d , ϵ 3 ϱ s ∗ , d ≤ c P
ϱ s ∗ , d ≲ r ∗ ( n , d , s ∗ , ϵ , δ , P , ∞ ) ≲ ℓ n , d , ϵ 3 ϱ s ∗ , d . (5) \varrho_{s^*,d}\lesssim r^*(n,d,s^*,\epsilon,\delta,\mathcal{P},\infty)\lesssim\ell_{n,d,\epsilon}^{3}\varrho_{s^*,d}. \tag*{(5)} ϱ s ∗ , d ≲ r ∗ ( n , d , s ∗ , ϵ , δ , P , ∞ ) ≲ ℓ n , d , ϵ 3 ϱ s ∗ , d . (5) For P G L M ( F ) \mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P GLM ( F ) r 0 , c P > 0 r_0,c_{\mathcal{P}}>0 r 0 , c P > 0 ℓ n , d , ϵ 3 ϱ s ∗ , d ∧ n ≤ c P \ell_{n,d,\epsilon}^{3}\varrho_{s^*,d\wedge n}\leq c_{\mathcal{P}} ℓ n , d , ϵ 3 ϱ s ∗ , d ∧ n ≤ c P
ϱ s ∗ , d ∧ n ≲ r ∗ ( n , d , s ∗ , ϵ , δ , P G L M ( F ) , r 0 ) ≲ ℓ n , d , ϵ 3 ϱ s ∗ , d ∧ n . (6) \varrho_{s^*,d\wedge n}\lesssim r^*(n,d,s^*,\epsilon,\delta,\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}),r_0)\lesssim\ell_{n,d,\epsilon}^{3}\varrho_{s^*,d\wedge n}. \tag*{(6)} ϱ s ∗ , d ∧ n ≲ r ∗ ( n , d , s ∗ , ϵ , δ , P GLM ( F ) , r 0 ) ≲ ℓ n , d , ϵ 3 ϱ s ∗ , d ∧ n . (6) The upper bounds are attained by pure-DP (δ = 0 \delta=0 δ = 0 δ ≤ c δ ϵ \delta\leq c_\delta\epsilon δ ≤ c δ ϵ ( n ϵ ) − 1 (n\epsilon)^{-1} ( n ϵ ) − 1 d 1 / 4 d^{1/4} d 1/4 s ∗ / ( n ϵ ) s^*/(n\epsilon) s ∗ / ( n ϵ ) ℓ 2 \ell_2 ℓ 2 [e.g. 19 ] .
For GLMs with more than two points in their response distribution, the effective dimension is d ∧ n d\wedge n d ∧ n d d d d ≥ n d\geq n d ≥ n s ∗ ≳ max { n , ϵ − 1 } s^{*} \gtrsim\max\{\sqrt{n}, \epsilon^{-1}\} s ∗ ≳ max { n , ϵ − 1 }
Lower bounds
In this section, we establish the lower bounds in Theorem 3. We begin with a general bound for private testing based on divergences between distributions.
Proposition 4. Fix ϵ , δ ≥ 0 \epsilon,\delta\ge0 ϵ , δ ≥ 0 X \mathcal{X} X σ \sigma σ P 0 \mathcal{P}_0 P 0 P 1 \mathcal{P}_1 P 1 X \mathcal{X} X P 0 ∈ P 0 P_0 \in\mathcal{P}_0 P 0 ∈ P 0 P 1 ∈ P 1 P_1 \in\mathcal{P}_1 P 1 ∈ P 1 π \pi π P β ∈ P 1 P_{\beta} \in\mathcal{P}_1 P β ∈ P 1 β ∈ supp ( π ) \beta\in\operatorname{supp}(\pi) β ∈ supp ( π ) P ‾ ( n ) : = ∫ P β ⊗ n π ( d β ) \overline{P}^{(n)} := \int P_{\beta}^{\otimes n}\pi(\mathrm{d}\beta) P ( n ) := ∫ P β ⊗ n π ( d β ) α , τ , ω ≥ 0 \alpha,\tau,\omega\ge0 α , τ , ω ≥ 0
(a) we have D T V ( P 0 , P 1 ) ≤ α D_{\mathrm{TV}}(P_0,P_1) \le\alpha D TV ( P 0 , P 1 ) ≤ α
(b) for each β ∈ supp ( π ) \beta\in\operatorname{supp}(\pi) β ∈ supp ( π ) Q β Q_{\beta} Q β D T V ( P β , Q β ) ≤ τ D_{\mathrm{TV}}(P_{\beta},Q_{\beta}) \le\tau D TV ( P β , Q β ) ≤ τ
(c) we can bound D K L ( Q ‾ ( n ) ∥ P 0 ⊗ n ) ≤ n ω 2 D_{\mathrm{KL}}(\overline{Q}^{(n)} \mathbin\Vert P_0^{\otimes n}) \le n\omega^{2} D KL ( Q ( n ) ∥ P 0 ⊗ n ) ≤ n ω 2 Q ‾ ( n ) : = ∫ Q β ⊗ n π ( d β ) \overline{Q}^{(n)} := \int Q_{\beta}^{\otimes n}\pi(\mathrm{d}\beta) Q ( n ) := ∫ Q β ⊗ n π ( d β )
Then for any ( ϵ , δ ) (\epsilon,\delta) ( ϵ , δ ) ϕ \phi ϕ
sup P ∈ P 0 P ⊗ n ( ϕ = 1 ) + sup P ∈ P 1 P ⊗ n ( ϕ = 0 ) ≥ 1 2 max { 1 − n ω − n τ , 0.9 exp ( − 10 ϵ n { α ∧ ( ω + τ ) } ) − 10 δ n { α ∧ ( ω + τ ) } } . (7) \sup_{P \in\mathcal{P}_0} P^{\otimes n}(\phi= 1) + \sup_{P \in\mathcal{P}_1} P^{\otimes n}(\phi= 0)
\ge\frac{1}{2}\max\left\{1-\sqrt{n}\omega-n\tau,\;0.9\exp\left(-10\epsilon n\{\alpha\wedge(\omega+\tau)\}\right)-10\delta n\{\alpha\wedge(\omega+\tau)\}\right\}. \tag*{(7)} P ∈ P 0 sup P ⊗ n ( ϕ = 1 ) + P ∈ P 1 sup P ⊗ n ( ϕ = 0 ) ≥ 2 1 max { 1 − n ω − n τ , 0.9 exp ( − 10 ϵ n { α ∧ ( ω + τ )} ) − 10 δ n { α ∧ ( ω + τ )} } . (7) The two terms in (7) give the non-private and privacy-constrained minimax lower bounds for testing, respectively. A related transport inequality for ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) [28 ] . Our formulation differs in two ways. First, to accommodate approximate ( δ > 0 ) (\delta> 0) ( δ > 0 ) [3 ] in applying group privacy constraints to datasets generated from a coupling of the null and alternative mixture distributions. This allows us to control the difference in rejection probabilities through the Hamming distance between these datasets. We then use the transport inequality of Marton [33 ] to bound the expected Hamming distance in terms of the KL divergence. Second, when a direct KL comparison between the original mixture and the null is difficult to bound, we introduce proxy distributions for the mixture components. The proof is given in Appendix § 7 .§ 1 . Applying Proposition 4 to our regression setting then requires suitable alternatives and proxy distributions.
Role of the nuisance intercept
The different rates in Theorem 3 reflect how much of the change in the response marginal distribution can be explained by perturbing the nuisance parameter. For GLMs, averaging over the centred covariate distribution removes the first-order marginal perturbation, leaving a leading change of order ∥ β ∥ 2 \lVert\beta\rVert^{2} ∥ β ∥ 2 (3) , a perturbation of the intercept on the same scale can absorb part of this remaining signal.
For a ∈ Γ a \in\Gamma a ∈ Γ P a P_a P a a a a p a p_a p a E a \mathbb{E}_a E a P a P_a P a S a : = ( ∂ a p a ) / p a S_a := (\partial_a p_a)/p_a S a := ( ∂ a p a ) / p a H a : = ( ∂ a 2 p a ) / p a H_a := (\partial_a^2 p_a)/p_a H a := ( ∂ a 2 p a ) / p a R a R_a R a H a H_a H a S a S_a S a L 2 ( P a ) L_2(P_a) L 2 ( P a )
E a [ R a 2 ] = inf c ∈ R E a [ ( H a + 2 c S a ) 2 ] . (8) \mathbb{E}_a[R_a^2] = \inf_{c \in\mathbb{R}} \mathbb{E}_a\left[(H_a + 2cS_a)^2\right]. \tag*{(8)} E a [ R a 2 ] = c ∈ R inf E a [ ( H a + 2 c S a ) 2 ] . (8) We provide an explicit form for R a R_a R a Lemma 17 .
Lemma 5. Under Assumption 3, for every a ∈ Γ a \in\Gamma a ∈ Γ E a [ R a 2 ] = 0 \mathbb{E}_a[R_a^2] = 0 E a [ R a 2 ] = 0 ∣ supp ( Y ) ∣ = 2 |\operatorname{supp}(Y)| = 2 ∣ supp ( Y ) ∣ = 2 E a [ R a 2 ] \mathbb{E}_a[R_a^2] E a [ R a 2 ]
Lemma 5 explains why the separation rates in Theorem 3 differ across response families. When the response takes only two possible values, its distribution is determined by the probability of either value. The effect of a small signal on this marginal probability can be cancelled by adjusting the intercept. Similarly, in the Gaussian linear model, the error variance can be adjusted to match the response marginal. These least favourable constructions leave the dependence between X X X Y Y Y X ⊤ β X^\top\beta X ⊤ β Y ∼ Exp ( exp ( a ) ) Y \sim\operatorname{Exp}(\exp(a)) Y ∼ Exp ( exp ( a )) R a ( Y ) = exp ( 2 a ) Y 2 − 4 exp ( a ) Y + 2 R_a(Y) = \exp(2a)Y^2 - 4\exp(a)Y + 2 R a ( Y ) = exp ( 2 a ) Y 2 − 4 exp ( a ) Y + 2 E a [ R a 2 ] = 4 \mathbb{E}_a[R_a^2] = 4 E a [ R a 2 ] = 4
To apply Proposition 4 , we construct mixtures over sparse coefficient vectors with randomly chosen supports and signs, and bound the divergence between the corresponding proxy mixture and a fixed null. The directional contribution to the divergence is governed by the overlap between two independent draws from this prior, capturing uncertainty about both the support and direction of the signal. The residual marginal contribution has leading order n E a [ R a 2 ] r 4 n\mathbb{E}_a[R_a^2]r^4 n E a [ R a 2 ] r 4 1 / ( n ϵ ) 1/(n\epsilon) 1/ ( n ϵ ) § 7 .
Upper bounds
Guided by the lower bounds, we use two sources of information about the regression signal: dependence between the covariates and response, and changes in the response marginal.
To detect dependence, a natural starting point is E [ X Y ] \mathbb{E}[XY] E [ X Y ] (2) , we have E [ X Y ] = E [ X X ⊤ ] β \mathbb{E}[XY] = \mathbb{E}[XX^\top]\beta E [ X Y ] = E [ X X ⊤ ] β E [ ∣ ξ ∣ ] < ∞ \mathbb{E}[|\xi|] < \infty E [ ∣ ξ ∣ ] < ∞ X Y XY X Y Y Y Y tanh ( Y ) \tanh(Y) tanh ( Y ) ∣ tanh ∣ ≤ 1 |\tanh| \le1 ∣ tanh ∣ ≤ 1 X tanh ( Y ) X\tanh(Y) X tanh ( Y ) tanh \tanh tanh
Lemma 6. *Suppose ( X , Y ) ∼ P (X,Y) \sim P ( X , Y ) ∼ P P ∈ P L i n P \in\mathcal{P}_{\mathrm{Lin}} P ∈ P Lin P ∈ P G L M ( F ) P \in\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P ∈ P GLM ( F ) F \mathcal{F} F
response family satisfying Assumption 3. Set Z : = X tanh ( Y ) Z := X \tanh(Y) Z := X tanh ( Y )
E [ Z Z ⊤ ] ⪯ E [ X X ⊤ ] , ∥ E [ Z ] ∥ 2 ≤ λ , ∥ Z − E [ Z ] ∥ ψ 2 ≲ C X . \mathbb{E}[ZZ^{\top}] \preceq\mathbb{E}[XX^{\top}], \qquad\|\mathbb{E}[Z]\|_{2} \le\sqrt{\lambda}, \qquad\|Z-\mathbb{E}[Z]\|_{\psi_{2}} \lesssim C_{X}. E [ Z Z ⊤ ] ⪯ E [ X X ⊤ ] , ∥ E [ Z ] ∥ 2 ≤ λ , ∥ Z − E [ Z ] ∥ ψ 2 ≲ C X . Under the null, E [ Z ] = 0 \mathbb{E}[Z]=0 E [ Z ] = 0 c μ > 0 c_{\mu}>0 c μ > 0
∥ E [ Z ] ∥ 2 ≥ c μ min { ∥ β ∥ 2 , 1 } . (9) \|\mathbb{E}[Z]\|_{2} \ge c_{\mu}\min\{\|\beta\|_{2},1\}. \tag*{(9)} ∥ E [ Z ] ∥ 2 ≥ c μ min { ∥ β ∥ 2 , 1 } . (9) Lemma 6 reduces regression signal detection to private mean testing, giving the first test in § 4.1 . In § 4.2 , we consider sparse alternatives by constructing an adaptive test based on empirical covariances between tanh ( v ⊤ X ) \tanh(v^{\top}X) tanh ( v ⊤ X ) tanh ( Y ) \tanh(Y) tanh ( Y ) v v v § 4.3 constructs response-only tests for GLMs with more than two response support points, which target the second-order marginal change (8) identified in the lower bounds.
A mean-reduction test
We start by setting up some notation to define the mean test. For a transformed dataset z 1 : n : = ( z 1 , … , z n ) ∈ ( R d ) n z_{1:n}:=(z_{1},\ldots,z_{n})\in(\mathbb{R}^{d})^{n} z 1 : n := ( z 1 , … , z n ) ∈ ( R d ) n
u i ( z 1 : n ) : = ∑ j ≠ i z i ⊤ z j , Q S ( z 1 : n ) : = ∑ i , j ∈ S , i ≠ j z i ⊤ z j . u_i(z_{1:n}) := \sum_{j\ne i} z_i^{\top}z_j,\qquad Q_S(z_{1:n}) := \sum_{i,j\in S,\ i\ne j} z_i^{\top}z_j. u i ( z 1 : n ) := j = i ∑ z i ⊤ z j , Q S ( z 1 : n ) := i , j ∈ S , i = j ∑ z i ⊤ z j . For s , L > 0 s,L>0 s , L > 0 D n , s , L \mathcal{D}_{n,s,L} D n , s , L z 1 : n ∈ ( R d ) n z_{1:n}\in(\mathbb{R}^{d})^{n} z 1 : n ∈ ( R d ) n
∣ u i ( z 1 : n ) ∣ ≤ L { n d + s n } , i ∈ [ n ] , (10) |u_i(z_{1:n})| \le L\{\sqrt{nd}+sn\},\qquad i\in[n], \tag*{(10)} ∣ u i ( z 1 : n ) ∣ ≤ L { n d + s n } , i ∈ [ n ] , (10) ∣ Q S ( z 1 : n ) ∣ ≤ L { ∣ S ∣ ∣ S ∣ d + ∣ S ∣ 2 } , S ⊆ [ n ] . (11) |Q_S(z_{1:n})| \le L\{|S|\sqrt{|S|d}+|S|^2\},\qquad S\subseteq[n]. \tag*{(11)} ∣ Q S ( z 1 : n ) ∣ ≤ L { ∣ S ∣ ∣ S ∣ d + ∣ S ∣ 2 } , S ⊆ [ n ] . (11) For the mean testing problem, a natural statistic is Q [ n ] ( Z 1 : n ) : = ∑ i ≠ j Z i ⊤ Z j Q_{[n]}(Z_{1:n}):=\sum_{i\ne j}Z_i^{\top}Z_j Q [ n ] ( Z 1 : n ) := ∑ i = j Z i ⊤ Z j n ( n − 1 ) ∥ E [ Z ] ∥ 2 2 n(n-1)\|\mathbb{E}[Z]\|_2^2 n ( n − 1 ) ∥ E [ Z ] ∥ 2 2 [36 ] is that although Q [ n ] Q_{[n]} Q [ n ] D n , s , L \mathcal{D}_{n,s,L} D n , s , L
Lemma 7. If z 1 : n , z 1 : n ′ ∈ D n , s , L z_{1:n},z'_{1:n}\in\mathcal{D}_{n,s,L} z 1 : n , z 1 : n ′ ∈ D n , s , L K K K
∣ Q [ n ] ( z 1 : n ) − Q [ n ] ( z 1 : n ′ ) ∣ ≤ 6 K L { n d + n s + K } . |Q_{[n]}(z_{1:n})-Q_{[n]}(z'_{1:n})| \le6KL\{\sqrt{nd}+ns+K\}. ∣ Q [ n ] ( z 1 : n ) − Q [ n ] ( z 1 : n ′ ) ∣ ≤ 6 K L { n d + n s + K } . Comparing this bound with the expectation of Q [ n ] ( Z 1 : n ) Q_{[n]}(Z_{1:n}) Q [ n ] ( Z 1 : n ) s s s (5) : the n d \sqrt{nd} n d n s ns n s 1 / ( n ϵ ) 1/(n\epsilon) 1/ ( n ϵ )
To apply this stability property to regression testing, we establish that the transformed observations Z 1 : n Z_{1:n} Z 1 : n D n , s , L \mathcal{D}_{n,s,L} D n , s , L Q [ n ] ( Z 1 : n ) Q_{[n]}(Z_{1:n}) Q [ n ] ( Z 1 : n ) § 8 .§ 3 . Together with our transformation in Lemma 6 , these concentration properties allow us to adapt the Lipschitz extension approach of Canonne et al. [12 ] and Narayanan [36 ] to the regression models considered here.
For n ≥ 2 n\ge2 n ≥ 2 ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0
G ( z 1 : n ) : = [ Q [ n ] ( z 1 : n ) ] 0 B n , B n : = s 2 n ( n − 1 ) , D : = ϵ 0 B n 8 log ( 2 / η ) . (12) G(z_{1:n}) := [Q_{[n]}(z_{1:n})]_{0}^{B_n},\qquad B_n:=s^2n(n-1),\qquad D:=\frac{\epsilon_0B_n}{8\log(2/\eta)}. \tag*{(12)} G ( z 1 : n ) := [ Q [ n ] ( z 1 : n ) ] 0 B n , B n := s 2 n ( n − 1 ) , D := 8 log ( 2/ η ) ϵ 0 B n . (12) For the choices of s s s L L L G G G D D D D n , s , L \mathcal{D}_{n,s,L} D n , s , L G ^ \widehat{G} G [34 ] )
G ^ ( z 1 : n ) : = inf z ~ 1 : n ∈ D n , s , L { G ( z ~ 1 : n ) + D D H a m ( z 1 : n , z ~ 1 : n ) } . (13) \widehat{G}(z_{1:n}) := \inf_{\widetilde{z}_{1:n}\in\mathcal{D}_{n,s,L}} \left\{G(\widetilde{z}_{1:n}) + D D_{\mathrm{Ham}}(z_{1:n},\widetilde{z}_{1:n})\right\}. \tag*{(13)} G ( z 1 : n ) := z 1 : n ∈ D n , s , L inf { G ( z 1 : n ) + D D Ham ( z 1 : n , z 1 : n ) } . (13) By construction, G ^ \widehat{G} G D D D G G G D n , s , L \mathcal{D}_{n,s,L} D n , s , L
ϕ s ( Z 1 : n ) : = 1 { G ^ ( Z 1 : n ) + W > B n 2 } , W ∼ Lap ( D ϵ 0 ) . (14) \phi_s(Z_{1:n}) := \mathbb{1}\left\{\widehat{G}(Z_{1:n}) + W > \frac{B_n}{2}\right\}, \qquad W \sim\operatorname{Lap}\left(\frac{D}{\epsilon_0}\right). \tag*{(14)} ϕ s ( Z 1 : n ) := 1 { G ( Z 1 : n ) + W > 2 B n } , W ∼ Lap ( ϵ 0 D ) . (14) The next lemma gives the privacy and testing guarantees of ϕ s \phi_s ϕ s s s s
Lemma 8 (Single-scale mean test). Let { ( X i , Y i ) } i = 1 n \{(X_i,Y_i)\}_{i=1}^n {( X i , Y i ) } i = 1 n P ∈ P L i n P \in\mathcal{P}_{\mathrm{Lin}} P ∈ P Lin P ∈ P G L M ( F ) P \in\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P ∈ P GLM ( F ) F \mathcal{F} F Z i : = X i tanh ( Y i ) Z_i := X_i\tanh(Y_i) Z i := X i tanh ( Y i ) i ∈ [ n ] i \in[n] i ∈ [ n ] 0 < s ≤ λ 0 < s \le\sqrt{\lambda} 0 < s ≤ λ 0 < η ≤ 1 / 4 0 < \eta\le1/4 0 < η ≤ 1/4 L L L (57) with failure probability η / 2 \eta/2 η /2 G ^ \widehat{G} G ϕ s \phi_s ϕ s (14) is ( ϵ 0 , 0 ) (\epsilon_0,0) ( ϵ 0 , 0 ) C C C
s ≥ ρ ( n , d , ϵ 0 , η ) : = C { l n , η d 1 / 4 n + l n , η d 1 / 4 n 3 / 4 ϵ 0 1 / 2 + l n , η 2 n ϵ 0 } , l n , η : = 1 + log ( 24 ( n + 1 ) η ) , (15) s \ge\rho(n,d,\epsilon_0,\eta) := C\left\{\frac{l_{n,\eta}d^{1/4}}{\sqrt{n}}+\frac{l_{n,\eta}d^{1/4}}{n^{3/4}\epsilon_0^{1/2}}+\frac{l_{n,\eta}^2}{n\epsilon_0}\right\}, \qquad l_{n,\eta} := 1+\log\left(\frac{24(n+1)}{\eta}\right), \tag*{(15)} s ≥ ρ ( n , d , ϵ 0 , η ) := C { n l n , η d 1/4 + n 3/4 ϵ 0 1/2 l n , η d 1/4 + n ϵ 0 l n , η 2 } , l n , η := 1 + log ( η 24 ( n + 1 ) ) , (15) the type-I and type-II errors of ϕ s \phi_s ϕ s
H 0 : E [ Z ] = 0 against H 1 ( s ) : s ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s H_0:\mathbb{E}[Z]=0 \qquad\text{against} \qquad H_1(s):s\le\|\mathbb{E}[Z]\|_2\le2s H 0 : E [ Z ] = 0 against H 1 ( s ) : s ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s are each at most η \eta η
Lemma 6 transfers the mean testing guarantee to regression signal detection. With c μ > 0 c_\mu>0 c μ > 0 (9) , every alternative with ∥ β ∥ 2 ≥ r ~ / c μ \|\beta\|_2\ge\widetilde{r}/c_\mu ∥ β ∥ 2 ≥ r / c μ ∥ E [ Z ] ∥ 2 ≥ r ~ \|\mathbb{E}[Z]\|_2\ge\widetilde{r} ∥ E [ Z ] ∥ 2 ≥ r r ~ ≤ c μ \widetilde{r}\le c_\mu r ≤ c μ ∥ E [ Z ] ∥ 2 \|\mathbb{E}[Z]\|_2 ∥ E [ Z ] ∥ 2
To cover the full alternative, we apply Lemma 8 over a dyadic grid of signal strengths. Splitting the privacy budget across scales and controlling the errors by a union bound only inflates the detection boundary by logarithmic factors; see Appendix § 8 .§ 5 for details. Combining this multiscale construction with Lemma 6 gives the dense directional component in the minimax rate.
Computing the extension involves optimisation over typical datasets D n , s , L \mathcal{D}_{n,s,L} D n , s , L § 5 , we construct a polynomial-time test attaining the same directional rate, up to logarithmic factors, when ϵ ≳ n − 1 / 2 \epsilon\gtrsim n^{-1/2} ϵ ≳ n − 1/2
Sparse upper bound
The test in the previous section uses the transformed mean E [ X tanh ( Y ) ] \mathbb{E}[X\tanh(Y)] E [ X tanh ( Y )] β \beta β β \beta β tanh \tanh tanh Cov ( tanh ( v T X ) , tanh ( Y ) ) = 0 \operatorname{Cov}(\tanh(v^{\mathsf{T}}X),\tanh(Y))=0 Cov ( tanh ( v T X ) , tanh ( Y )) = 0
∣ Cov ( tanh ( v T X ) , tanh ( Y ) ) ∣ ≳ min { ∥ β ∥ 2 , 1 } , v : = β ∥ β ∥ 2 . \left|\operatorname{Cov}\left(\tanh(v^{\mathsf{T}}X),\tanh(Y)\right)\right|\gtrsim\min\{\|\beta\|_2,1\}, \qquad v:=\frac{\beta}{\|\beta\|_2}. Cov ( tanh ( v T X ) , tanh ( Y ) ) ≳ min { ∥ β ∥ 2 , 1 } , v := ∥ β ∥ 2 β . This signal persists for sufficiently close directions, allowing us to restrict attention to a fixed set of candidate directions.
To adapt to unknown sparsity, we combine tests over the dyadic grid K : = { 2 j : 0 ≤ j ≤ ⌊ log 2 d ⌋ } ∪ { d } \mathcal{K} := \{2^j : 0 \le j \le\lfloor\log_2 d \rfloor\} \cup\{d\} K := { 2 j : 0 ≤ j ≤ ⌊ log 2 d ⌋} ∪ { d } η n e t ∈ ( 0 , 1 ) \eta_{\mathrm{net}} \in(0,1) η net ∈ ( 0 , 1 ) S ⊆ [ d ] S \subseteq[d] S ⊆ [ d ] ∣ S ∣ ∈ K |S| \in\mathcal{K} ∣ S ∣ ∈ K η n e t \eta_{\mathrm{net}} η net V S \mathcal{V}_S V S { v ∈ R d : ∥ v ∥ 2 = 1 , supp ( v ) ⊆ S } \{v \in\mathbb{R}^d : \|v\|_2 = 1, \operatorname{supp}(v) \subseteq S\} { v ∈ R d : ∥ v ∥ 2 = 1 , supp ( v ) ⊆ S } ∣ V S ∣ ≤ ( 1 + 2 / η n e t ) ∣ S ∣ |\mathcal{V}_S| \le(1+2/\eta_{\mathrm{net}})^{|S|} ∣ V S ∣ ≤ ( 1 + 2/ η net ) ∣ S ∣ [43 ] . For each k ∈ K k \in\mathcal{K} k ∈ K V k : = ⋃ S ⊆ [ d ] , ∣ S ∣ = k V S \mathcal{V}_k := \bigcup_{S\subseteq[d],|S|=k}\mathcal{V}_S V k := ⋃ S ⊆ [ d ] , ∣ S ∣ = k V S n ≥ 2 n \ge2 n ≥ 2
T k : = max v ∈ V k 1 2 m ∑ j = 1 m { tanh ( v ⊤ X 2 j − 1 ) − tanh ( v ⊤ X 2 j ) } { tanh ( Y 2 j − 1 ) − tanh ( Y 2 j ) } , m : = ⌊ n / 2 ⌋ . T_k := \max_{v\in\mathcal{V}_k} \frac{1}{2m}\sum_{j=1}^{m}\{\tanh(v^\top X_{2j-1})-\tanh(v^\top X_{2j})\}\{\tanh(Y_{2j-1})-\tanh(Y_{2j})\},\quad m:=\lfloor n/2\rfloor. T k := v ∈ V k max 2 m 1 j = 1 ∑ m { tanh ( v ⊤ X 2 j − 1 ) − tanh ( v ⊤ X 2 j )} { tanh ( Y 2 j − 1 ) − tanh ( Y 2 j )} , m := ⌊ n /2 ⌋ . For each fixed v v v 4 / m 4/m 4/ m
ϕ s p a r s e : = 1 { max k ∈ K { T k − 8 log ( 2 ∣ K ∣ ∣ V k ∣ / η ) m } + 4 W m ϵ > 4 log ( 1 / η ) m ϵ } , W ∼ Lap ( 1 ) . (16) \phi_{\mathrm{sparse}} := \mathbb{1}\left\{\max_{k\in\mathcal{K}}\left\{T_k-\sqrt{\frac{8\log(2|\mathcal{K}||\mathcal{V}_k|/\eta)}{m}}\right\}+\frac{4W}{m\epsilon}>\frac{4\log(1/\eta)}{m\epsilon}\right\},\quad W\sim\operatorname{Lap}(1). \tag*{(16)} ϕ sparse := 1 { k ∈ K max { T k − m 8 log ( 2∣ K ∣∣ V k ∣/ η ) } + m ϵ 4 W > m ϵ 4 log ( 1/ η ) } , W ∼ Lap ( 1 ) . (16) Proposition 9. Let { ( X i , Y i ) } i = 1 n \{(X_i,Y_i)\}_{i=1}^n {( X i , Y i ) } i = 1 n P ∈ P L i n P\in\mathcal{P}_{\mathrm{Lin}} P ∈ P Lin P ∈ P G L M ( F ) P\in\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P ∈ P GLM ( F ) F \mathcal{F} F η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 ) (16) is ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) η \eta η η n e t ∈ ( 0 , 1 ) \eta_{\mathrm{net}}\in(0,1) η net ∈ ( 0 , 1 ) c > 0 c>0 c > 0 ϕ s p a r s e \phi_{\mathrm{sparse}} ϕ sparse η \eta η ∥ β ∥ 0 ≤ s ∗ \|\beta\|_0\le s^* ∥ β ∥ 0 ≤ s ∗
s ∗ log ( e d / s ∗ ) + log ( ∣ K ∣ / η ) n + log ( 1 / η ) n ϵ ≤ c min { ∥ β ∥ 2 , 1 } . (17) \sqrt{\frac{s^*\log(ed/s^*)+\log(|\mathcal{K}|/\eta)}{n}}+\frac{\log(1/\eta)}{n\epsilon}\le c\min\{\|\beta\|_2,1\}. \tag*{(17)} n s ∗ log ( e d / s ∗ ) + log ( ∣ K ∣/ η ) + n ϵ log ( 1/ η ) ≤ c min { ∥ β ∥ 2 , 1 } . (17) The test ϕ s p a r s e \phi_{\mathrm{sparse}} ϕ sparse § 9 . Checking all candidate directions to adapt over sparsity levels up to s ∗ = d s^*=d s ∗ = d O ( n d ( 2 + 2 / η n e t ) d ) O(nd(2+2/\eta_{\mathrm{net}})^d) O ( n d ( 2 + 2/ η net ) d ) k ≤ s max k\le s_{\max} k ≤ s m a x s max s_{\max} s m a x O ( n d s max ) O(nd^{s_{\max}}) O ( n d s m a x )
Response marginal test
For GLMs with more than two response support points, the lower bounds identify second-order information in the response marginal that cannot be absorbed by adjusting the nuisance intercept. This residual information, quantified by E a [ R a 2 ] \mathbb{E}_a[R_a^2] E a [ R a 2 ] (8) , motivates a test based on the responses alone.
We construct a bounded statistic that preserves a signal of order ∥ β ∥ 2 2 \|\beta\|_2^2 ∥ β ∥ 2 2 ∥ β ∥ 2 \|\beta\|_2 ∥ β ∥ 2 R a ( Y ) R_a(Y) R a ( Y ) (8) , centre the resulting function under the null, and then re-orthogonalise it to the nuisance score. The resulting functions have the following properties.
Lemma 10. Let F \mathcal{F} F Γ 1 \Gamma_1 Γ 1 Γ ⊂ int ( Γ 1 ) ⊂ Γ 1 ⊂ U \Gamma\subset\operatorname{int}(\Gamma_1)\subset\Gamma_1\subset U Γ ⊂ int ( Γ 1 ) ⊂ Γ 1 ⊂ U
measurable functions { g a } a ∈ Γ 1 \{g_a\}_{a\in\Gamma_1} { g a } a ∈ Γ 1 c g , B g > 0 c_g, B_g > 0 c g , B g > 0 a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1
E a [ g a ( Y ) ] = 0 , (18) \mathbb{E}_a[g_a(Y)] = 0, \tag*{(18)} E a [ g a ( Y )] = 0 , (18) E a [ g a ( Y ) S a ( Y ) ] = 0 , (19) \mathbb{E}_a[g_a(Y)S_a(Y)] = 0, \tag*{(19)} E a [ g a ( Y ) S a ( Y )] = 0 , (19) E a [ g a ( Y ) H a ( Y ) ] ≥ c g , (20) \mathbb{E}_a[g_a(Y)H_a(Y)] \ge c_g, \tag*{(20)} E a [ g a ( Y ) H a ( Y )] ≥ c g , (20) sup y ∣ g a ( y ) ∣ ≤ B g . (21) \sup_y |g_a(y)| \le B_g. \tag*{(21)} y sup ∣ g a ( y ) ∣ ≤ B g . (21) Property (18) ensures that the statistic has mean zero under the null hypothesis, when evaluated at the true intercept, while (19) gives nuisance orthogonality. In particular, since
d d t E a + t [ g a ( Y ) ] ∣ t = 0 = E a [ g a ( Y ) S a ( Y ) ] = 0 , \left.\frac{\mathrm{d}}{\mathrm{d}t}\mathbb{E}_{a+t}[g_a(Y)]\right|_{t=0} = \mathbb{E}_a[g_a(Y)S_a(Y)] = 0, d t d E a + t [ g a ( Y )] t = 0 = E a [ g a ( Y ) S a ( Y )] = 0 , the statistic is locally insensitive to errors in estimating the nuisance parameter, in the spirit of Neyman’s C ( α ) C(\alpha) C ( α ) [21 ] ). The condition in (20) ensures a positive second-order signal, while the uniform bound in (21) controls the global sensitivity of the empirical mean of g a ( Y ) g_a(Y) g a ( Y ) g a g_a g a c g c_g c g B g B_g B g
The testing procedure consists of two stages, which are described in Algorithm 1. In the first stage, we use the first half of the samples to obtain a private estimate of the intercept γ ^ \widehat{\gamma} γ E [ g γ ^ ( Y ) ] \mathbb{E}[g_{\widehat{\gamma}}(Y)] E [ g γ ( Y )]
Algorithm 1 Response-only test
Input: Data { Y i } i = 1 n \{Y_i\}_{i=1}^n { Y i } i = 1 n n ≥ 2 n \ge2 n ≥ 2 ϵ > 0 \epsilon> 0 ϵ > 0 η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 ) c T > 0 c_T > 0 c T > 0
Let n 1 = ⌊ n / 2 ⌋ n_1 = \lfloor n/2\rfloor n 1 = ⌊ n /2 ⌋ n 2 = n − n 1 n_2 = n-n_1 n 2 = n − n 1
Set
m ~ : = 1 n 1 ∑ i = 1 n 1 tanh ( Y i ) + W 1 , W 1 ∼ Lap ( 2 n 1 ϵ ) (22) \widetilde{m} := \frac{1}{n_1}\sum_{i=1}^{n_1}\tanh(Y_i) + W_1,\qquad W_1 \sim\operatorname{Lap}\left(\frac{2}{n_1\epsilon}\right) \tag*{(22)} m := n 1 1 i = 1 ∑ n 1 tanh ( Y i ) + W 1 , W 1 ∼ Lap ( n 1 ϵ 2 ) (22) Obtain the private intercept estimate
γ ^ : = arg min a ∈ Γ 1 ∣ m ( a ) − m ~ ∣ , m ( a ) : = E a [ tanh ( Y ) ] . (23) \widehat{\gamma} := \underset{a\in\Gamma_1}{\arg\min}\, |m(a)-\widetilde{m}|,\qquad m(a):=\mathbb{E}_a[\tanh(Y)]. \tag*{(23)} γ := a ∈ Γ 1 arg min ∣ m ( a ) − m ∣ , m ( a ) := E a [ tanh ( Y )] . (23) Set the private test statistic
T ~ : = 1 n 2 ∑ i = n 1 + 1 n g γ ^ ( Y i ) + W 2 , W 2 ∼ Lap ( 2 B g n 2 ϵ ) , W 2 ⊥ W 1 (24) \widetilde{T} := \frac{1}{n_2}\sum_{i=n_1+1}^{n}g_{\widehat{\gamma}}(Y_i)+W_2,\qquad W_2\sim\operatorname{Lap}\left(\frac{2B_g}{n_2\epsilon}\right),\quad W_2\perp W_1 \tag*{(24)} T := n 2 1 i = n 1 + 1 ∑ n g γ ( Y i ) + W 2 , W 2 ∼ Lap ( n 2 ϵ 2 B g ) , W 2 ⊥ W 1 (24) Set a n ( ϵ , η ) : = log ( 8 / η ) / n + log ( 8 / η ) / ( n ϵ ) a_n(\epsilon,\eta):=\sqrt{\log(8/\eta)/n+\log(8/\eta)/(n\epsilon)} a n ( ϵ , η ) := log ( 8/ η ) / n + log ( 8/ η ) / ( n ϵ )
return 1 { T ~ ≥ c T a n ( ϵ , η ) } \mathbb{1}\{\widetilde{T}\ge c_Ta_n(\epsilon,\eta)\} 1 { T ≥ c T a n ( ϵ , η )}
Algorithm 1 satisfies the following privacy and error guarantees.
Proposition 11. The test in Algorithm 1 is ( ϵ , 0 ) (\epsilon, 0) ( ϵ , 0 ) P ∈ P G L M ( F ) P \in\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P ∈ P GLM ( F ) F \mathcal{F} F η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 ) c T > 0 c_T > 0 c T > 0 C , r 0 , a ∗ > 0 C,r_0,a_* > 0 C , r 0 , a ∗ > 0 c T c_T c T η \eta η a n ( ϵ , η ) ≤ a ∗ a_n(\epsilon,\eta) \leq a_* a n ( ϵ , η ) ≤ a ∗ η \eta η
0 < r ≤ ∥ β ∥ 2 ≤ r 0 , r 2 ≥ C { log ( 8 / η ) n + log ( 8 / η ) n ϵ } . (25) 0 < r \leq\lVert\beta\rVert_2 \leq r_0,\qquad r^2 \geq C\left\{\sqrt{\frac{\log(8/\eta)}{n}}+\frac{\log(8/\eta)}{n\epsilon}\right\}. \tag*{(25)} 0 < r ≤ ∥ β ∥ 2 ≤ r 0 , r 2 ≥ C { n log ( 8/ η ) + n ϵ log ( 8/ η ) } . (25) Proposition 11 shows that, for GLMs with more than two response support points, the regression signal can be detected at rate
r ∗ ≲ log n − 1 / 4 + ( n ϵ ) − 1 / 2 r^* \lesssim_{\log} n^{-1/4}+(n\epsilon)^{-1/2} r ∗ ≲ l o g n − 1/4 + ( n ϵ ) − 1/2 using the responses alone. This gives the marginal contribution to the GLM minimax rate of Theorem 3. Unlike the mean-reduction test, which detects dependence between the covariates and response, the response-only test detects a second-order change in the response marginal without searching for the unknown signal direction. Its rate is therefore independent of d d d d ≳ n d \gtrsim n d ≳ n d ≍ n d \asymp n d ≍ n
Remark 1. The restriction ∥ β ∥ 2 ≤ r 0 \lVert\beta\rVert_2 \leq r_0 ∥ β ∥ 2 ≤ r 0 Proposition 11 comes from the local likelihood expansion used to bound the type-II error and carries over to the upper bound in Theorem 3. For some response families, invariance to the intercept allows us to avoid estimating it and to detect signals beyond this local neighbourhood. For example, in exponential regression with log rate γ + x ⊤ β \gamma+x^\top\beta γ + x ⊤ β Y 1 Y_1 Y 1 Y 2 Y_2 Y 2
h Exp ( y 1 , y 2 ) : = 1 { max ( y 1 , y 2 ) > 2 min ( y 1 , y 2 ) } − 2 / 3 (26) h_{\operatorname{Exp}}(y_1,y_2):=\mathbb{1}\{\max(y_1,y_2)>2\min(y_1,y_2)\}-2/3 \tag*{(26)} h Exp ( y 1 , y 2 ) := 1 { max ( y 1 , y 2 ) > 2 min ( y 1 , y 2 )} − 2/3 (26) is invariant under this common scaling, and is bounded and satisfies E 0 , γ [ h Exp ( Y 1 , Y 2 ) ] = 0 \mathbb{E}_{0,\gamma}[h_{\operatorname{Exp}}(Y_1,Y_2)]=0 E 0 , γ [ h Exp ( Y 1 , Y 2 )] = 0 E β , γ [ h Exp ( Y 1 , Y 2 ) ] ≳ min { ∥ β ∥ 2 2 , 1 } \mathbb{E}_{\beta,\gamma}[h_{\operatorname{Exp}}(Y_1,Y_2)]\gtrsim\min\{\lVert\beta\rVert_2^2,1\} E β , γ [ h Exp ( Y 1 , Y 2 )] ≳ min {∥ β ∥ 2 2 , 1 } γ ∈ R \gamma\in\mathbb{R} γ ∈ R U U U ∥ β ∥ 2 ≤ r 0 \lVert\beta\rVert_2\leq r_0 ∥ β ∥ 2 ≤ r 0 § 10 .§ 4 .
To complete the upper bound in Theorem 3, we combine the multiscale mean-reduction test (§ 4.1 ) and the sparse covariance test (§ 4.2 ), adding the response-only test for GLMs with more than two response support points. We omit any component whose separation condition cannot be met within the alternative class, allocate a fixed fraction of the privacy budget to each remaining test, and reject if any component rejects. The combined test is ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) Lemma 39 ). Its type-I error is controlled by a union bound and its type-II error is controlled whenever at least one component’s separation condition holds. The combined test therefore attains the minimum of the component rates, up to polylogarithmic factors, without requiring knowledge of the sparsity level. Full details are given in Appendix § 10 .§ 5 .
Computation and numerical experiments
The mean-reduction test in § 4.1 and the sparse covariance test in § 4.2 attain the dense directional and sparse rates in Theorem 3, respectively, up to logarithmic factors. However, the former relies on a Lipschitz extension with no polynomial-time guarantee, while the latter checks exponentially many directions. We now give computationally efficient alternatives and quantify their additional privacy costs, and then present numerical experiments in § 5.3 .
A polynomial-time mean-reduction test
We consider the same quadratic statistic as in § 4.1 , but use a clipped version of the inner product and a different privatisation mechanism. For some fixed constant C c l i p > 0 C_{\mathrm{clip}}>0 C clip > 0 α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) s s s
h s ( z 1 , z 2 ) : = [ z 1 T z 2 ] − R ( s ) R ( s ) , R ( s ) : = C c l i p ( d l s + l s ) , l s : = 1 + log { ( d + 1 ) ( n + 1 ) ( s − 1 + 1 ) / α } . (27) h_s(z_1,z_2):=\left[z_1^{\mathsf{T}}z_2\right]_{-R(s)}^{R(s)},\qquad R(s):=C_{\mathrm{clip}}\left(\sqrt{d l_s}+l_s\right),\qquad l_s:=1+\log\left\{(d+1)(n+1)(s^{-1}+1)/\alpha\right\}. \tag*{(27)} h s ( z 1 , z 2 ) := [ z 1 T z 2 ] − R ( s ) R ( s ) , R ( s ) := C clip ( d l s + l s ) , l s := 1 + log { ( d + 1 ) ( n + 1 ) ( s − 1 + 1 ) / α } . (27) We privatise the resulting U-statistic using the local Hájek mechanism of Chaudhuri et al. [14 ] , which downweights influential observations and calibrates noise to a smooth sensitivity bound. In Lemma 34, we bound the clipping bias, variance and deviations of the kernel averages associated with individual observations to establish the type-I and type-II error of the test. As in § 4.1 , we combine tests over a dyadic grid of signal strengths to cover the full alternative. The full testing procedure is specified in Algorithm 2.
Algorithm 2 Polynomial-time mean-reduction test
Input: Data { Z i } i = 1 n \{Z_i\}_{i=1}^{n} { Z i } i = 1 n n ≥ 2 n\ge2 n ≥ 2 0 < r ~ ≤ λ 0<\tilde r\le\sqrt{\lambda} 0 < r ~ ≤ λ ϵ > 0 \epsilon>0 ϵ > 0 η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 )
Set J ← max { 1 , ⌈ log 2 ( λ / r ~ ) ⌉ } J\leftarrow\max\{1,\lceil\log_2(\sqrt{\lambda}/\tilde r)\rceil\} J ← max { 1 , ⌈ log 2 ( λ / r ~ )⌉} α ← η / ( 3 J ) \alpha\leftarrow\eta/(3J) α ← η / ( 3 J )
for j = 0 , … , J − 1 j=0,\ldots,J-1 j = 0 , … , J − 1
Set s j ← 2 j r ~ s_j\leftarrow2^j\tilde r s j ← 2 j r ~ ξ s j ← C t o l l s j ( s j + d / n ) \xi_{s_j}\leftarrow C_{\mathrm{tol}}l_{s_j}\left(s_j+\sqrt{d/n}\right) ξ s j ← C tol l s j ( s j + d / n )
Set U ~ j ← L o c a l H a ˊ j e k M e a n ( Z 1 : n , h s j , 2 R ( s j ) , ξ s j , ϵ / J ) \widetilde U_j\leftarrow\operatorname{LocalHájekMean}(Z_{1:n},h_{s_j},2R(s_j),\xi_{s_j},\epsilon/J) U j ← LocalH a ˊ jekMean ( Z 1 : n , h s j , 2 R ( s j ) , ξ s j , ϵ / J )
if U ~ j > s j 2 / 2 \widetilde U_j>s_j^2/2 U j > s j 2 /2
return 1.
end if
end for
return 0.
The LocalHájekMean algorithm in Algorithm 2 can be found as Algorithm 3 in Appendix § 11 ; it specialises Algorithm 1 of Chaudhuri et al. [14 ] to U-statistics of order two. The test has the following guarantees.
Proposition 12. The test in Algorithm 2 is ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) O ( J d n 2 ) O(Jdn^2) O ( J d n 2 ) { ( X i , Y i ) } i = 1 n \{(X_i,Y_i)\}_{i=1}^{n} {( X i , Y i ) } i = 1 n P ∈ P L i n P\in\mathcal{P}_{\mathrm{Lin}} P ∈ P Lin P ∈ P G L M ( F ) P\in\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P ∈ P GLM ( F ) F \mathcal{F} F η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 ) C c l i p C_{\mathrm{clip}} C clip C t o l C_{\mathrm{tol}} C tol c , C , C 1 > 0 c,C,C_1>0 c , C , C 1 > 0
c η − 1 / 2 ℓ n , d , ϵ β 3 { d 1 / 4 n 1 / 2 + d 1 / 4 n 3 / 4 ϵ 1 / 2 + d 1 / 4 n ϵ + d 1 / 4 n 3 / 2 ϵ 3 / 2 } ≤ r ~ ≤ C , (28) c\eta^{-1/2}\ell_{n,d,\epsilon}^{\beta_3}\left\{\frac{d^{1/4}}{n^{1/2}}+\frac{d^{1/4}}{n^{3/4}\epsilon^{1/2}}+\frac{d^{1/4}}{n\epsilon}+\frac{d^{1/4}}{n^{3/2}\epsilon^{3/2}}\right\}\le\tilde r\le C, \tag*{(28)} c η − 1/2 ℓ n , d , ϵ β 3 { n 1/2 d 1/4 + n 3/4 ϵ 1/2 d 1/4 + n ϵ d 1/4 + n 3/2 ϵ 3/2 d 1/4 } ≤ r ~ ≤ C , (28) the sum of the type-I and type-II errors of Algorithm 2 for the hypothesis test in (1) with r = C 1 r ~ r=C_1\tilde r r = C 1 r ~ R = ∞ R=\infty R = ∞ 2 η 2\eta 2 η r ~ \tilde r r ~ O ( d n 2 log ( n + 1 ) ) O(dn^2\log(n+1)) O ( d n 2 log ( n + 1 ))
Proposition 12 shows that Algorithm 2 provides a polynomial-time alternative to the mean-reduction test in § 4.1 , at the cost of additional privacy-dependent terms. The final term in (28) is dominated by the third term whenever n ϵ ≳ 1 n\epsilon\gtrsim1 n ϵ ≳ 1 c 0 > 0 c_0 > 0 c 0 > 0 n ϵ < c 0 n\epsilon< c_0 n ϵ < c 0 1 / ( n ϵ ) 1/(n\epsilon) 1/ ( n ϵ ) d 1 / 4 / ( n ϵ ) d^{1/4}/(n\epsilon) d 1/4 / ( n ϵ ) (28) . The two rates match when we have ϵ ≳ n − 1 / 2 \epsilon\gtrsim n^{-1/2} ϵ ≳ n − 1/2
A coordinate test for sparse alternatives
We next consider a computationally efficient alternative to the sparse test in § 4.2 , based on the largest coordinate of the transformed sample mean from Lemma 6 . Although E [ X tanh ( Y ) ] \mathbb{E}[X\tanh(Y)] E [ X tanh ( Y )] § 11 .§ 3 that its largest coordinate still retains a signal
∥ E [ X tanh ( Y ) ] ∥ ∞ ≥ c μ s ∗ min { ∥ β ∥ 2 , 1 } . \left\|\mathbb{E}[X\tanh(Y)]\right\|_{\infty} \ge\frac{c_\mu}{\sqrt{s^*}}\min\{\|\beta\|_2,1\}. ∥ E [ X tanh ( Y )] ∥ ∞ ≥ s ∗ c μ min { ∥ β ∥ 2 , 1 } . We therefore target this quantity by clipping each covariate coordinate at B : = C X log ( 2 n ) B := C_X\sqrt{\log(2n)} B := C X log ( 2 n )
Proposition 13. Suppose that { ( X i , Y i ) } i = 1 n \{(X_i,Y_i)\}_{i=1}^n {( X i , Y i ) } i = 1 n P ∈ P L i n P \in\mathcal{P}_{\mathrm{Lin}} P ∈ P Lin P ∈ P G L M ( F ) P \in\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P ∈ P GLM ( F ) F \mathcal{F} F ϵ > 0 \epsilon> 0 ϵ > 0 η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 )
ϕ : = 1 { T + 2 B n ϵ W > b B + q B + 2 B log ( 1 / η ) n ϵ } , T : = max j ∈ [ d ] ∣ 1 n ∑ i = 1 n [ X i j ] − B B tanh ( Y i ) ∣ , \phi:= \mathbb{1}\left\{T+\frac{2B}{n\epsilon}W>b_B+q_B+\frac{2B\log(1/\eta)}{n\epsilon}\right\}, \qquad T:=\max_{j\in[d]}\left|\frac{1}{n}\sum_{i=1}^{n}[X_{ij}]_{-B}^{B}\tanh(Y_i)\right|, ϕ := 1 { T + n ϵ 2 B W > b B + q B + n ϵ 2 B log ( 1/ η ) } , T := j ∈ [ d ] max n 1 i = 1 ∑ n [ X ij ] − B B tanh ( Y i ) , with W ∼ Lap ( 1 ) W\sim\operatorname{Lap}(1) W ∼ Lap ( 1 ) ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) O ( n d ) O(nd) O ( n d ) b B : = C X 2 exp ( − B 2 / C X 2 ) / B b_B:=C_X^2\exp(-B^2/C_X^2)/B b B := C X 2 exp ( − B 2 / C X 2 ) / B q B : = 2 λ log ( 4 d / η ) / n + 4 B log ( 4 d / η ) / ( 3 n ) q_B:=\sqrt{2\lambda\log(4d/\eta)/n+4B\log(4d/\eta)/(3n)} q B := 2 λ log ( 4 d / η ) / n + 4 B log ( 4 d / η ) / ( 3 n ) η \eta η C > 0 C>0 C > 0 1 ≤ s ∗ ≤ d 1\le s^*\le d 1 ≤ s ∗ ≤ d η \eta η ∥ β ∥ 0 ≤ s ∗ \|\beta\|_0\le s^* ∥ β ∥ 0 ≤ s ∗
C s ∗ { log ( 4 d / η ) n + log ( 2 n ) log ( 4 d / η ) n + log ( 2 n ) log ( 1 / η ) n ϵ } ≤ min { ∥ β ∥ 2 , 1 } . (29) C\sqrt{s^*}\left\{\sqrt{\frac{\log(4d/\eta)}{n}}+\frac{\sqrt{\log(2n)}\log(4d/\eta)}{n}+\frac{\sqrt{\log(2n)}\log(1/\eta)}{n\epsilon}\right\}\le\min\{\|\beta\|_2,1\}. \tag*{(29)} C s ∗ { n log ( 4 d / η ) + n log ( 2 n ) log ( 4 d / η ) + n ϵ log ( 2 n ) log ( 1/ η ) } ≤ min { ∥ β ∥ 2 , 1 } . (29) The test in Proposition 13 does not require knowledge of s ∗ s^* s ∗ ϵ ≳ n − 1 / 2 \epsilon\gtrsim n^{-1/2} ϵ ≳ n − 1/2 s ∗ \sqrt{s^*} s ∗
The response-only test in Algorithm 1 also has an efficient Monte Carlo implementation for many commonly used response families. As long as we can efficiently sample from P a P_a P a S a ( y ) S_a(y) S a ( y ) R a ( y ) R_a(y) R a ( y ) Lemma 17 ) for every a ∈ Γ 1 a\in\Gamma_1 a ∈ Γ 1 O ( n 2 / η ) O(n^2/\eta) O ( n 2 / η ) § 10 .§ 3 for details. Combining Propositions 11, 12, and 13 then gives a polynomial-time procedure across all regimes. When ϵ ≳ n − 1 / 2 \epsilon\gtrsim n^{-1/2} ϵ ≳ n − 1/2 (28) and (29) reflect fundamental computational-statistical gaps or are an artefact of our test constructions to future work.
Numerical experiments
In this subsection, we evaluate the numerical performance of our computationally efficient tests. The code for these experiments can be found at https://github.com/EKHung/DP_regression_testing . For each experimental configuration, we use 250 repetitions under each of the null and alternative hypotheses and report the estimated sum of the type-I and type-II errors, with the error bars showing Monte Carlo standard errors.
We set the tuning parameters of Algorithm 2 to be C c l i p = 0.5 C_{\mathrm{clip}} = 0.5 C clip = 0.5 C t o l = 0.05 C_{\mathrm{tol}} = 0.05 C tol = 0.05 η = 0.05 \eta= 0.05 η = 0.05 r ~ \tilde{r} r ~ (28) , multiplied by C s e p = 5 C_{\mathrm{sep}} = 5 C sep = 5 C X C_X C X λ ˉ \bar{\lambda} λ ˉ § 13 suggest reasonable robustness to their misspecification. We also examine how the performance of Algorithm 2 depends on its tuning choices.
We first compare our tests with the following private procedures:
dpHSIC : a DP analogue of the permutation HSIC independence test, from Kim and Schrab [29 ] .
ToT-F : the test-of-tests framework of Kazan et al. [27 ] , based on the subsample-and-aggregate approach of Nissim et al. [37 ] . We use the regression F-test as the non-private test within each block.
PB-LRT : the private likelihood ratio test of Peña and Barrientos (2024) for Gaussian linear regression. This test also splits the sample into blocks, but then computes and clips the blockwise log-likelihood ratios, averages them, and adds Laplace noise.
Algorithm 2 and all the tests listed above satisfy pure differential privacy, i.e. δ = 0 \delta= 0 δ = 0 § 13 .
For the comparison, we vary n ∈ { 2000 , 2300 , … , 5000 } n \in\{2000, 2300, \ldots, 5000\} n ∈ { 2000 , 2300 , … , 5000 } ϵ = 1 \epsilon= 1 ϵ = 1 (2) , with independent covariates X i ∼ N ( 0 , I 500 ) X_i \sim\mathcal{N}(0, I_{500}) X i ∼ N ( 0 , I 500 ) ξ i ∼ N ( 0 , 1 ) \xi_i \sim\mathcal{N}(0,1) ξ i ∼ N ( 0 , 1 ) β = ( 1 , 0 , 0 , … , 0 ) \beta= (1, 0, 0, \ldots, 0) β = ( 1 , 0 , 0 , … , 0 )
Figure 1 shows that the coordinate test from Proposition 13 makes no error across the simulated repetitions. Since the alternative satisfies ∥ β ∥ 0 = 1 \lVert\beta\rVert_0 = 1 ∥ β ∥ 0 = 1 n = 2900 n = 2900 n = 2900 n = 3200 n = 3200 n = 3200
Figure 1. Comparison of private tests under Gaussian linear regression with d = 500 d = 500 d = 500 ϵ = 1 \epsilon= 1 ϵ = 1
We also consider two models with independent Rademacher covariate coordinates, and set β = ( 1 / d , 1 / d , … , 1 / d ) \beta= (1/\sqrt{d},1/\sqrt{d},\ldots,1/\sqrt{d}) β = ( 1/ d , 1/ d , … , 1/ d ) Figure 2 shows results for a linear regression model (2) with d = 500 d = 500 d = 500 Cauchy ( 0 , 1 ) \operatorname{Cauchy}(0,1) Cauchy ( 0 , 1 ) Figure 2 shows results for an exponential regression model, with d = 50 d = 50 d = 50 exp ( − ( 0.2 + X i T β ) ) \exp(-(0.2 + X_i^{\mathsf{T}}\beta)) exp ( − ( 0.2 + X i T β )) ϵ ∈ { 1 , 2 , 3 , 4 } \epsilon\in\{1,2,3,4\} ϵ ∈ { 1 , 2 , 3 , 4 }
Figure 2. Performance of the polynomial-time mean-reduction test under Cauchy linear regression with d = 500 d = 500 d = 500 d = 50 d = 50 d = 50 ϵ ∈ { 1 , 2 , 3 , 4 } \epsilon\in\{1, 2, 3, 4\} ϵ ∈ { 1 , 2 , 3 , 4 }
Next, we evaluate how our coordinate test from § 5.2 performs at four sparsity levels ∥ β ∥ 0 ∈ { 1 , 5 , 10 , 20 } \lVert\beta\rVert_0 \in\{1, 5, 10, 20\} ∥ β ∥ 0 ∈ { 1 , 5 , 10 , 20 } ϵ = 1 \epsilon= 1 ϵ = 1 ∥ β ∥ 2 = 1 \lVert\beta\rVert_2 = 1 ∥ β ∥ 2 = 1 1 / ∥ β ∥ 0 1/\sqrt{\lVert\beta\rVert_0} 1/ ∥ β ∥ 0 n ∈ { 300 , 600 , … , 3000 } n \in\{300, 600, \ldots, 3000\} n ∈ { 300 , 600 , … , 3000 } Figure 3 . Consistent with Proposition 13 , sparser alternatives require fewer observations to achieve low testing error, illustrating how the test benefits from sparsity without requiring knowledge of the sparsity level.
Figure 3. Estimated sum of type-I and type-II errors for the coordinate test under Gaussian linear regression with d = 500 d = 500 d = 500 ϵ = 1 \epsilon= 1 ϵ = 1 ∥ β ∥ 2 = 1 \lVert\beta\rVert_2 = 1 ∥ β ∥ 2 = 1 s ∗ ∈ { 1 , 5 , 10 , 20 } s^* \in\{1,5,10,20\} s ∗ ∈ { 1 , 5 , 10 , 20 }
To illustrate a response-only mechanism, we also consider the exponential response-only test described in Remark 1 and Appendix § 10 .§ 4 , setting η = 0.05 \eta= 0.05 η = 0.05 (78) . We use the same exponential regression model as above, fix n = 2000 n = 2000 n = 2000 d d d ϵ ∈ { 0.1 , 0.15 , 0.2 , 0.3 } \epsilon\in\{0.1, 0.15, 0.2, 0.3\} ϵ ∈ { 0.1 , 0.15 , 0.2 , 0.3 } Figure 4 shows that testing error remains broadly stable as the dimension increases, including beyond the sample size.
Figure 4. Estimated sum of type-I and type-II errors from numerical experiments using the test in Remark 1, with n = 2000 n = 2000 n = 2000 ∥ β ∥ 2 = 1 \lVert\beta\rVert_2 = 1 ∥ β ∥ 2 = 1 ϵ ∈ { 0.1 , 0.15 , 0.2 , 0.3 } \epsilon\in\{0.1,0.15,0.2,0.3\} ϵ ∈ { 0.1 , 0.15 , 0.2 , 0.3 }
Discussion
We established minimax separation rates for private signal detection in linear regression and GLMs with unknown nuisance parameters. The rates depend on whether nuisance adjustment can absorb the change in the response marginal: when it cannot, response-only testing can improve detection in high dimensions. We also characterised how sparsity improves the separation rates and constructed tests that attain these rates up to polylogarithmic factors without knowing the sparsity level.
There are several directions for future work. First, it remains open whether polynomial-time tests can attain the minimax separation rates in the very high privacy regime ϵ ≲ n − 1 / 2 \epsilon\lesssim n^{-1/2} ϵ ≲ n − 1/2 [24 ] , is to investigate whether sum-of-squares methods can approximate the test in (14) while preserving its privacy and power guarantees. Second, we could consider testing a subset of regression coefficients, treating the remaining coefficients as potentially high-dimensional nuisance parameters. The nuisance orthogonality used in our response-only test motivates exploring orthogonal test statistics for this setting, drawing on related ideas from double machine learning (Chernozhukov et al., [15 ] ). A central question is how nuisance complexity and privacy constraints jointly affect the minimax separation rates.
Acknowledgements. The author is supported by the Chancellors’ Scholarship scheme and the Statistics Centre for Doctoral Training at the University of Warwick. The author thanks Benedict Risebrow for helpful comments.
Use of AI tools. OpenAI’s ChatGPT was used to assist with mathematical derivations, refining proof arguments, simulation code development, and language polishing. The author reviewed and revised the resulting material and takes full responsibility for the work.
Technical details for 3
Appendix A.1 provides the proof of our general lower bound. Appendices A.2 and A.3 develop the constructions for obtaining the signal detection lower bounds. The proof of the lower bounds in Theorem 3 is in Appendix § 7 .§ 4 .
Proof of 4
Proof. By the triangle inequality, we have
D T V ( P 0 ⊗ n , P ‾ ( n ) ) ≤ D T V ( P 0 ⊗ n , Q ‾ ( n ) ) + D T V ( Q ‾ ( n ) , P ‾ ( n ) ) ≤ 1 2 n ω 2 + n τ . \begin{aligned}
D_{\mathrm{TV}}\left(P_0^{\otimes n},\overline{P}^{(n)}\right) \le D_{\mathrm{TV}}\left(P_0^{\otimes n},\overline{Q}^{(n)}\right) + D_{\mathrm{TV}}\left(\overline{Q}^{(n)},\overline{P}^{(n)}\right) \\
&\le\sqrt{\frac{1}{2}n\omega^2} + n\tau.
\end{aligned} D TV ( P 0 ⊗ n , P ( n ) ) ≤ D TV ( P 0 ⊗ n , Q ( n ) ) + D TV ( Q ( n ) , P ( n ) ) ≤ 2 1 n ω 2 + n τ . The second inequality is due to conditions (b) and (c) of Proposition 4, Pinsker’s inequality, and convexity and tensorisation of the total variation distance. The non-private rate then follows from Le Cam’s two point method.
To obtain the private rate, we consider two couplings. First, since D T V ( P 0 , P 1 ) ≤ α D_{\mathrm{TV}}(P_0,P_1) \le\alpha D TV ( P 0 , P 1 ) ≤ α P 0 ⊗ n P_0^{\otimes n} P 0 ⊗ n P 1 ⊗ n P_1^{\otimes n} P 1 ⊗ n n D T V ( P 0 , P 1 ) ≤ n α nD_{\mathrm{TV}}(P_0,P_1) \le n\alpha n D TV ( P 0 , P 1 ) ≤ n α
We next consider a coupling for the mixture distribution to the fixed null. First sample β ∼ π \beta\sim\pi β ∼ π β \beta β i ∈ [ n ] i \in[n] i ∈ [ n ] ( D i , D ~ i ) (D_i,\widetilde{D}_i) ( D i , D i ) P β , Q β P_\beta, Q_\beta P β , Q β β \beta β
E [ D H a m ( D , D ~ ) ] = ∑ i = 1 n P ( D i ≠ D ~ i ) ≤ n τ , \mathbb{E}[D_{\mathrm{Ham}}(D,\widetilde{D})] = \sum_{i=1}^{n}\mathbb{P}(D_i \ne\widetilde{D}_i) \le n\tau, E [ D Ham ( D , D )] = i = 1 ∑ n P ( D i = D i ) ≤ n τ , where D : = ( D 1 , … , D n ) D := (D_1,\ldots,D_n) D := ( D 1 , … , D n ) D ~ : = ( D ~ 1 , … , D ~ n ) \widetilde{D} := (\widetilde{D}_1,\ldots,\widetilde{D}_n) D := ( D 1 , … , D n )
Next, by Lemma 14 , there exists a coupling between Q ‾ ( n ) \overline{Q}^{(n)} Q ( n ) P 0 ⊗ n P_0^{\otimes n} P 0 ⊗ n
E [ D H a m ( D ~ , D 0 ) ] ≤ n 2 D K L ( Q ‾ ( n ) ∥ P 0 ⊗ n ) ≤ n ω 2 , \mathbb{E}[D_{\mathrm{Ham}}(\widetilde{D},D_0)] \le\sqrt{\frac{n}{2}D_{\mathrm{KL}}\left(\overline{Q}^{(n)} \mathbin\Vert P_0^{\otimes n}\right)} \le\frac{n\omega}{\sqrt{2}}, E [ D Ham ( D , D 0 )] ≤ 2 n D KL ( Q ( n ) ∥ P 0 ⊗ n ) ≤ 2 nω , where the final inequality is due to condition (c) of Proposition 4.
We can realise both couplings on a common probability space by first sampling D ~ \widetilde{D} D D , D 0 D,D_0 D , D 0 P 0 ⊗ n P_0^{\otimes n} P 0 ⊗ n P ‾ ( n ) \overline{P}^{(n)} P ( n )
E [ D H a m ( D 0 , D ) ] ≤ E [ D H a m ( D 0 , D ~ ) ] + E [ D H a m ( D ~ , D ) ] ≤ n ( τ + ω ) . \mathbb{E}[D_{\mathrm{Ham}}(D_0,D)] \le\mathbb{E}[D_{\mathrm{Ham}}(D_0,\widetilde{D})] + \mathbb{E}[D_{\mathrm{Ham}}(\widetilde{D},D)] \le n(\tau+\omega). E [ D Ham ( D 0 , D )] ≤ E [ D Ham ( D 0 , D )] + E [ D Ham ( D , D )] ≤ n ( τ + ω ) . Applying Lemma 15 with the smaller of the Hamming-cost bounds concludes the proof.
Lemma 14 (Hamming cost; Marton [33 ] ). Let X \mathcal{X} X σ \sigma σ P ( X ) \mathcal{P}(\mathcal{X}) P ( X ) X \mathcal{X} X Q ∈ P ( X ) Q \in\mathcal{P}(\mathcal{X}) Q ∈ P ( X ) P ∈ P ( X n ) P \in\mathcal{P}(\mathcal{X}^n) P ∈ P ( X n ) ( D , D ′ ) (D,D') ( D , D ′ ) D ∼ Q ⊗ n D \sim Q^{\otimes n} D ∼ Q ⊗ n D ′ ∼ P D' \sim P D ′ ∼ P
E [ D H a m ( D , D ′ ) ] ≤ n 2 D K L ( P ∥ Q ⊗ n ) . \mathbb{E}[D_{\mathrm{Ham}}(D,D')] \le\sqrt{\frac{n}{2}D_{\mathrm{KL}}\left(P \mathbin\Vert Q^{\otimes n}\right)}. E [ D Ham ( D , D ′ )] ≤ 2 n D KL ( P ∥ Q ⊗ n ) . Proof. For completeness, we recall the sequential coupling argument. Construct ( D , D ′ ) (D,D') ( D , D ′ ) ( D < i , D < i ′ ) (D_{<i},D'_{<i}) ( D < i , D < i ′ ) Q Q Q P i ( ⋅ ∣ D < i ′ ) P_i(\cdot\mid D'_{<i}) P i ( ⋅ ∣ D < i ′ ) D ∼ Q ⊗ n D \sim Q^{\otimes n} D ∼ Q ⊗ n D ′ ∼ P D' \sim P D ′ ∼ P
P ( D i ≠ D i ′ ∣ D < i ′ ) = D T V ( P i ( ⋅ ∣ D < i ′ ) , Q ) . \mathbb{P}(D_i \ne D'_i \mid D'_{<i}) = D_{\mathrm{TV}}\left(P_i(\cdot\mid D'_{<i}), Q\right). P ( D i = D i ′ ∣ D < i ′ ) = D TV ( P i ( ⋅ ∣ D < i ′ ) , Q ) . Hence, by Pinsker’s inequality and Cauchy–Schwarz,
E [ D H a m ( D , D ′ ) ] = ∑ i = 1 n E D T V ( P i ( ⋅ ∣ D < i ′ ) , Q ) ≤ ∑ i = 1 n 1 2 E D K L ( P i ( ⋅ ∣ D < i ′ ) ∥ Q ) ≤ n 2 ∑ i = 1 n E D K L ( P i ( ⋅ ∣ D < i ′ ) ∥ Q ) = n 2 D K L ( P ∥ Q ⊗ n ) . \begin{aligned}
\mathbb{E}[D_{\mathrm{Ham}}(D,D')] = \sum_{i=1}^{n} \mathbb{E}D_{\mathrm{TV}}\left(P_i(\cdot\mid D'_{<i}),Q\right) \\
&\le\sum_{i=1}^{n}\sqrt{\frac{1}{2}\mathbb{E}D_{\mathrm{KL}}\left(P_i(\cdot\mid D'_{<i}) \,\|\, Q\right)} \\
&\le\sqrt{\frac{n}{2}\sum_{i=1}^{n}\mathbb{E}D_{\mathrm{KL}}\left(P_i(\cdot\mid D'_{<i}) \,\|\, Q\right)}
= \sqrt{\frac{n}{2}D_{\mathrm{KL}}\left(P \,\|\, Q^{\otimes n}\right)}.
\end{aligned} E [ D Ham ( D , D ′ )] = i = 1 ∑ n E D TV ( P i ( ⋅ ∣ D < i ′ ) , Q ) ≤ i = 1 ∑ n 2 1 E D KL ( P i ( ⋅ ∣ D < i ′ ) ∥ Q ) ≤ 2 n i = 1 ∑ n E D KL ( P i ( ⋅ ∣ D < i ′ ) ∥ Q ) = 2 n D KL ( P ∥ Q ⊗ n ) . Lemma 15 (Theorem 1 of Acharya et al. [3 ] ). Let P 0 P_0 P 0 P 1 P_1 P 1 X n \mathcal{X}^n X n P 0 ∈ conv ( P 0 ) P_0 \in\operatorname{conv}(P_0) P 0 ∈ conv ( P 0 ) P 1 ∈ conv ( P 1 ) P_1 \in\operatorname{conv}(P_1) P 1 ∈ conv ( P 1 ) ( D , D ′ ) (D,D') ( D , D ′ ) P 0 P_0 P 0 P 1 P_1 P 1 H : = E [ D H a m ( D , D ′ ) ] H := \mathbb{E}[D_{\mathrm{Ham}}(D,D')] H := E [ D Ham ( D , D ′ )] ( ϵ , δ ) (\epsilon,\delta) ( ϵ , δ ) ϕ \phi ϕ
max i ∈ { 0 , 1 } sup P ∈ P i P ( ϕ ( X ) ≠ i ∣ X ∼ P ) ≥ 1 2 max { 1 − D T V ( P 0 , P 1 ) , 0.9 exp ( − 10 ϵ H ) − 10 H δ } . (30) \max_{i\in\{0,1\}}\sup_{P\in\mathcal{P}_i}\mathbb{P}(\phi(X)\ne i\mid X\sim P)
\ge\frac{1}{2}\max\left\{1-D_{\mathrm{TV}}(P_0,P_1),0.9\exp(-10\epsilon H)-10H\delta\right\}. \tag*{(30)} i ∈ { 0 , 1 } max P ∈ P i sup P ( ϕ ( X ) = i ∣ X ∼ P ) ≥ 2 1 max { 1 − D TV ( P 0 , P 1 ) , 0.9 exp ( − 10 ϵH ) − 10 H δ } . (30) Local likelihood expansions and nuisance adjustment
Notation. For a linear predictor a a a Γ \Gamma Γ P a P_a P a η a : = ϑ ( a ) \eta_a := \vartheta(a) η a := ϑ ( a )
Z a : = Y − b ′ ( η a ) , κ j ( a ) : = b ( j ) ( η a ) , j = 2 , 3 , 4. Z_a := Y-b'(\eta_a), \qquad\kappa_j(a):=b^{(j)}(\eta_a), \quad j=2,3,4. Z a := Y − b ′ ( η a ) , κ j ( a ) := b ( j ) ( η a ) , j = 2 , 3 , 4. We denote the first two likelihood derivatives by
S a : = ϑ ′ ( a ) Z a , H a : = ϑ ′ ′ ( a ) Z a + ϑ ′ ( a ) 2 { Z a 2 − κ 2 ( a ) } . S_a := \vartheta'(a)Z_a, \qquad H_a := \vartheta''(a)Z_a+\vartheta'(a)^2\{Z_a^2-\kappa_2(a)\}. S a := ϑ ′ ( a ) Z a , H a := ϑ ′′ ( a ) Z a + ϑ ′ ( a ) 2 { Z a 2 − κ 2 ( a )} . Let c ∗ ( a ) c^*(a) c ∗ ( a ) R a : = H a + 2 c ∗ ( a ) S a R_a := H_a+2c^*(a)S_a R a := H a + 2 c ∗ ( a ) S a E a [ R a 2 ] \mathbb{E}_a[R_a^2] E a [ R a 2 ] L 2 ( P a ) L_2(P_a) L 2 ( P a ) ϑ − 1 \vartheta^{-1} ϑ − 1 ϑ \vartheta ϑ U U U
We now collect some useful properties that will be used in the lower and upper bound proofs.
Lemma 16 (Local likelihood expansion). Under Assumption 3 , there exist compact intervals
Γ ⊂ int ( Γ 1 ) , Γ 1 ⊂ int ( Γ 2 ) , Γ 2 ⊂ U , (31) \Gamma\subset\operatorname{int}(\Gamma_1), \qquad\Gamma_1 \subset\operatorname{int}(\Gamma_2), \qquad\Gamma_2 \subset U, \tag*{(31)} Γ ⊂ int ( Γ 1 ) , Γ 1 ⊂ int ( Γ 2 ) , Γ 2 ⊂ U , (31) a compact interval K + ⊂ ϑ ( U ) K^+ \subset\vartheta(U) K + ⊂ ϑ ( U ) ρ L > 0 \rho_L>0 ρ L > 0 a ∈ Γ 1 a\in\Gamma_1 a ∈ Γ 1 ∣ t 1 ∣ , ∣ t 2 ∣ ≤ ρ L |t_1|,|t_2|\le\rho_L ∣ t 1 ∣ , ∣ t 2 ∣ ≤ ρ L
a + t 1 , a + t 2 ∈ Γ 2 , ϑ ( a + t 1 ) + ϑ ( a + t 2 ) − ϑ ( a ) ∈ K + . (32) a+t_1,a+t_2\in\Gamma_2, \qquad\vartheta(a+t_1)+\vartheta(a+t_2)-\vartheta(a)\in K^+. \tag*{(32)} a + t 1 , a + t 2 ∈ Γ 2 , ϑ ( a + t 1 ) + ϑ ( a + t 2 ) − ϑ ( a ) ∈ K + . (32) Uniformly over a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1
L a , t ( Y ) : = d P a + t d P a ( Y ) = 1 + t S a ( Y ) + t 2 2 H a ( Y ) + O L 2 ( P a ) ( ∣ t ∣ 3 ) , ∣ t ∣ ≤ ρ L . (33) L_{a,t}(Y) := \frac{\mathrm{d}P_{a+t}}{\mathrm{d}P_a}(Y) = 1 + tS_a(Y) + \frac{t^2}{2}H_a(Y) + O_{L_2(P_a)}(|t|^3), \qquad|t| \le\rho_L. \tag*{(33)} L a , t ( Y ) := d P a d P a + t ( Y ) = 1 + t S a ( Y ) + 2 t 2 H a ( Y ) + O L 2 ( P a ) ( ∣ t ∣ 3 ) , ∣ t ∣ ≤ ρ L . (33) Moreover the first three derivatives of t ↦ L a , t t \mapsto L_{a,t} t ↦ L a , t L 2 ( P a ) L_2(P_a) L 2 ( P a ) L 2 ( P a ) L_2(P_a) L 2 ( P a ) γ 0 ∈ int ( Γ ) \gamma_0 \in\operatorname{int}(\Gamma) γ 0 ∈ int ( Γ ) κ > 0 \kappa> 0 κ > 0 γ 0 + [ − 3 κ , 3 κ ] ⊂ Γ 2 \gamma_0 + [-3\kappa,3\kappa] \subset\Gamma_2 γ 0 + [ − 3 κ , 3 κ ] ⊂ Γ 2
ϑ ( γ 0 + u 1 ) + ϑ ( γ 0 + u 2 ) − ϑ ( γ 0 ) ∈ K + , ∣ u 1 ∣ , ∣ u 2 ∣ ≤ 3 κ . (34) \vartheta(\gamma_0+u_1)+\vartheta(\gamma_0+u_2)-\vartheta(\gamma_0) \in K^+, \qquad|u_1|,|u_2| \le3\kappa. \tag*{(34)} ϑ ( γ 0 + u 1 ) + ϑ ( γ 0 + u 2 ) − ϑ ( γ 0 ) ∈ K + , ∣ u 1 ∣ , ∣ u 2 ∣ ≤ 3 κ . (34) Proof. Since Γ \Gamma Γ Γ ⊂ U \Gamma\subset U Γ ⊂ U Γ 1 , Γ 2 \Gamma_1,\Gamma_2 Γ 1 , Γ 2 (31) holds. The map
( a , t 1 , t 2 ) ↦ ϑ ( a + t 1 ) + ϑ ( a + t 2 ) − ϑ ( a ) (a,t_1,t_2) \mapsto\vartheta(a+t_1)+\vartheta(a+t_2)-\vartheta(a) ( a , t 1 , t 2 ) ↦ ϑ ( a + t 1 ) + ϑ ( a + t 2 ) − ϑ ( a ) is continuous and equals ϑ ( a ) \vartheta(a) ϑ ( a ) ( t 1 , t 2 ) = ( 0 , 0 ) (t_1,t_2)=(0,0) ( t 1 , t 2 ) = ( 0 , 0 ) ϑ ( Γ 1 ) \vartheta(\Gamma_1) ϑ ( Γ 1 ) Θ \Theta Θ ρ L > 0 \rho_L>0 ρ L > 0 K + ⊂ ϑ ( U ) K^+ \subset\vartheta(U) K + ⊂ ϑ ( U ) (32) holds. Decreasing ρ L \rho_L ρ L a + t j ∈ Γ 2 a+t_j \in\Gamma_2 a + t j ∈ Γ 2 γ 0 ∈ int ( Γ ) \gamma_0 \in\operatorname{int}(\Gamma) γ 0 ∈ int ( Γ ) κ \kappa κ 3 κ ≤ ρ L 3\kappa\le\rho_L 3 κ ≤ ρ L γ 0 + [ − 3 κ , 3 κ ] ⊂ Γ 2 \gamma_0+[-3\kappa,3\kappa] \subset\Gamma_2 γ 0 + [ − 3 κ , 3 κ ] ⊂ Γ 2 (34) . All derivatives of ϑ \vartheta ϑ Γ 2 \Gamma_2 Γ 2
Let η a , t : = ϑ ( a + t ) \eta_{a,t}:=\vartheta(a+t) η a , t := ϑ ( a + t )
ℓ a , t ( Y ) : = log L a , t ( Y ) = Y ( η a , t − η a ) − { b ( η a , t ) − b ( η a ) } . \ell_{a,t}(Y):=\log L_{a,t}(Y)=Y(\eta_{a,t}-\eta_a)-\{b(\eta_{a,t})-b(\eta_a)\}. ℓ a , t ( Y ) := log L a , t ( Y ) = Y ( η a , t − η a ) − { b ( η a , t ) − b ( η a )} . The derivatives of ℓ a , t \ell_{a,t} ℓ a , t Y Y Y a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1 ∣ t ∣ ≤ ρ L |t| \le\rho_L ∣ t ∣ ≤ ρ L L a , t = e ℓ a , t L_{a,t}=e^{\ell_{a,t}} L a , t = e ℓ a , t L a , t L_{a,t} L a , t Y Y Y
Fix any polynomial P P P
E a [ L a , t 2 ( Y ) P ( Y ) 2 ] = ∫ P ( y ) 2 exp { 2 ( ( η a , t − η a ) y − b ( η a , t ) + b ( η a ) ) } d P a ( y ) = exp { b ( 2 η a , t − η a ) − 2 b ( η a , t ) + b ( η a ) } E ϑ − 1 ( 2 η a , t − η a ) [ P ( Y ) 2 ] . \begin{aligned}
\mathbb{E}_a[L_{a,t}^2(Y)P(Y)^2]
&=\int P(y)^2\exp\{2((\eta_{a,t}-\eta_a)y-b(\eta_{a,t})+b(\eta_a))\}\,\mathrm{d}P_a(y)\\
&=\exp\{b(2\eta_{a,t}-\eta_a)-2b(\eta_{a,t})+b(\eta_a)\}\mathbb{E}_{\vartheta^{-1}(2\eta_{a,t}-\eta_a)}[P(Y)^2].
\end{aligned} E a [ L a , t 2 ( Y ) P ( Y ) 2 ] = ∫ P ( y ) 2 exp { 2 (( η a , t − η a ) y − b ( η a , t ) + b ( η a ))} d P a ( y ) = exp { b ( 2 η a , t − η a ) − 2 b ( η a , t ) + b ( η a )} E ϑ − 1 ( 2 η a , t − η a ) [ P ( Y ) 2 ] . We see that squaring the likelihood ratio tilts the natural parameter from η a \eta_a η a 2 η a , t − η a 2\eta_{a,t}-\eta_a 2 η a , t − η a K + K^+ K + K + K^+ K + Θ \Theta Θ K + K^+ K + sup η ∈ K + E ϑ − 1 ( η ) [ P ( Y ) 2 ] < ∞ \sup_{\eta\in K^+}\mathbb{E}_{\vartheta^{-1}(\eta)}[P(Y)^2]<\infty sup η ∈ K + E ϑ − 1 ( η ) [ P ( Y ) 2 ] < ∞
Since the first three derivatives of L a , t L_{a,t} L a , t L a , t L_{a,t} L a , t Y Y Y (32) and the preceding display imply that their L 2 ( P a ) L_2(P_a) L 2 ( P a ) a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1 ∣ t ∣ ≤ ρ L |t| \le\rho_L ∣ t ∣ ≤ ρ L L 2 ( P a ) L_2(P_a) L 2 ( P a )
L a , t = L a , 0 + t L a , 0 ′ + t 2 2 L a , 0 ′ ′ + O L 2 ( P a ) ( ∣ t ∣ 3 ) . L_{a,t}=L_{a,0}+tL'_{a,0}+\frac{t^2}{2}L''_{a,0}+O_{L_2(P_a)}(|t|^3). L a , t = L a , 0 + t L a , 0 ′ + 2 t 2 L a , 0 ′′ + O L 2 ( P a ) ( ∣ t ∣ 3 ) . Noting that L a , 0 ′ = ℓ a , 0 ′ = ϑ ′ ( a ) Z a L'_{a,0}=\ell'_{a,0}=\vartheta'(a)Z_a L a , 0 ′ = ℓ a , 0 ′ = ϑ ′ ( a ) Z a
L a , 0 ′ ′ = ( ℓ a , 0 ′ ) 2 + ℓ a , 0 ′ ′ = ϑ ′ ′ ( a ) Z a + ϑ ′ ( a ) 2 { Z a 2 − κ 2 ( a ) } L''_{a,0}=(\ell'_{a,0})^2+\ell''_{a,0}=\vartheta''(a)Z_a+\vartheta'(a)^2\{Z_a^2-\kappa_2(a)\} L a , 0 ′′ = ( ℓ a , 0 ′ ) 2 + ℓ a , 0 ′′ = ϑ ′′ ( a ) Z a + ϑ ′ ( a ) 2 { Z a 2 − κ 2 ( a )} proves (33) .
Our next lemma provides an explicit form for E a [ R a 2 ] \mathbb{E}_a[R_a^2] E a [ R a 2 ] (8) .
Lemma 17 (Least favourable nuisance direction). For a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1 define
c ∗ ( a ) : = − ϑ ′ ′ ( a ) 2 ϑ ′ ( a ) − ϑ ′ ( a ) κ 3 ( a ) 2 κ 2 ( a ) c^*(a) := -\frac{\vartheta''(a)}{2\vartheta'(a)}-\frac{\vartheta'(a)\kappa_3(a)}{2\kappa_2(a)} c ∗ ( a ) := − 2 ϑ ′ ( a ) ϑ ′′ ( a ) − 2 κ 2 ( a ) ϑ ′ ( a ) κ 3 ( a ) and
R a : = H a + 2 c ∗ ( a ) S a = ϑ ′ ( a ) 2 [ Z a 2 − κ 2 ( a ) − κ 3 ( a ) κ 2 ( a ) Z a ] . (35) R_a := H_a+2c^*(a)S_a=\vartheta'(a)^2\left[Z_a^2-\kappa_2(a)-\frac{\kappa_3(a)}{\kappa_2(a)}Z_a\right]. \tag*{(35)} R a := H a + 2 c ∗ ( a ) S a = ϑ ′ ( a ) 2 [ Z a 2 − κ 2 ( a ) − κ 2 ( a ) κ 3 ( a ) Z a ] . (35) Then E a [ R a ] = 0 \mathbb{E}_a[R_a]=0 E a [ R a ] = 0 E a [ R a S a ] = 0 \mathbb{E}_a[R_aS_a]=0 E a [ R a S a ] = 0
E a [ R a 2 ] = ϑ ′ ( a ) 4 { κ 4 ( a ) + 2 κ 2 ( a ) 2 − κ 3 ( a ) 2 κ 2 ( a ) } . (36) \mathbb{E}_a[R_a^2]=\vartheta'(a)^4\left\{\kappa_4(a)+2\kappa_2(a)^2-\frac{\kappa_3(a)^2}{\kappa_2(a)}\right\}. \tag*{(36)} E a [ R a 2 ] = ϑ ′ ( a ) 4 { κ 4 ( a ) + 2 κ 2 ( a ) 2 − κ 2 ( a ) κ 3 ( a ) 2 } . (36) Proof. The exponential family cumulant identities give
E a Z a = 0 , E a Z a 2 = κ 2 ( a ) , E a Z a 3 = κ 3 ( a ) , E a Z a 4 = κ 4 ( a ) + 3 κ 2 ( a ) 2 . \mathbb{E}_aZ_a=0,\qquad\mathbb{E}_aZ_a^2=\kappa_2(a),\qquad\mathbb{E}_aZ_a^3=\kappa_3(a),\qquad\mathbb{E}_aZ_a^4=\kappa_4(a)+3\kappa_2(a)^2. E a Z a = 0 , E a Z a 2 = κ 2 ( a ) , E a Z a 3 = κ 3 ( a ) , E a Z a 4 = κ 4 ( a ) + 3 κ 2 ( a ) 2 . Expanding H a S a = ϑ ′ ( a ) ϑ ′ ′ ( a ) Z a 2 + ϑ ′ ( a ) 3 Z a 3 − ϑ ′ ( a ) 3 κ 2 ( a ) Z a H_aS_a=\vartheta'(a)\vartheta''(a)Z_a^2+\vartheta'(a)^3Z_a^3-\vartheta'(a)^3\kappa_2(a)Z_a H a S a = ϑ ′ ( a ) ϑ ′′ ( a ) Z a 2 + ϑ ′ ( a ) 3 Z a 3 − ϑ ′ ( a ) 3 κ 2 ( a ) Z a
E a [ H a S a ] = ϑ ′ ( a ) ϑ ′ ′ ( a ) κ 2 ( a ) + ϑ ′ ( a ) 3 κ 3 ( a ) , \mathbb{E}_a[H_aS_a]=\vartheta'(a)\vartheta''(a)\kappa_2(a)+\vartheta'(a)^3\kappa_3(a), E a [ H a S a ] = ϑ ′ ( a ) ϑ ′′ ( a ) κ 2 ( a ) + ϑ ′ ( a ) 3 κ 3 ( a ) , and we also have that E a [ S a 2 ] = ϑ ′ ( a ) 2 κ 2 ( a ) \mathbb{E}_a[S_a^2]=\vartheta'(a)^2\kappa_2(a) E a [ S a 2 ] = ϑ ′ ( a ) 2 κ 2 ( a ) E a [ R a S a ] = E a [ H a S a ] + 2 c ∗ E a [ S a 2 ] = 0 \mathbb{E}_a[R_aS_a]=\mathbb{E}_a[H_aS_a]+2c^*\mathbb{E}_a[S_a^2]=0 E a [ R a S a ] = E a [ H a S a ] + 2 c ∗ E a [ S a 2 ] = 0
2 c ∗ ( a ) = − E a [ H a S a ] E a [ S a 2 ] = − ϑ ′ ′ ( a ) ϑ ′ ( a ) − ϑ ′ ( a ) κ 3 ( a ) κ 2 ( a ) , 2c^*(a)=-\frac{\mathbb{E}_a[H_aS_a]}{\mathbb{E}_a[S_a^2]}=-\frac{\vartheta''(a)}{\vartheta'(a)}-\vartheta'(a)\frac{\kappa_3(a)}{\kappa_2(a)}, 2 c ∗ ( a ) = − E a [ S a 2 ] E a [ H a S a ] = − ϑ ′ ( a ) ϑ ′′ ( a ) − ϑ ′ ( a ) κ 2 ( a ) κ 3 ( a ) , which proves the orthogonality and (35) . Squaring (35) and applying the cumulant identities shows (36) . □ \square □
Finally, we provide the proof that E a [ R a 2 ] \mathbb{E}_a[R_a^2] E a [ R a 2 ]
Proof of Lemma 5. If E a [ R a 2 ] = 0 \mathbb{E}_a[R_a^2]=0 E a [ R a 2 ] = 0
Z a 2 − κ 2 ( a ) = κ 3 ( a ) κ 2 ( a ) Z a , P a -almost surely , (37) Z_a^2-\kappa_2(a)=\frac{\kappa_3(a)}{\kappa_2(a)}Z_a,\qquad P_a\text{-almost surely}, \tag*{(37)} Z a 2 − κ 2 ( a ) = κ 2 ( a ) κ 3 ( a ) Z a , P a -almost surely , (37) so Z a Z_a Z a b ′ ′ ( η a ) = Var a ( Y ) > 0 b''(\eta_a)=\operatorname{Var}_a(Y)>0 b ′′ ( η a ) = Var a ( Y ) > 0
For the converse statement, suppose that Y Y Y Z a ∈ { z 1 , z 2 } Z_a\in\{z_1,z_2\} Z a ∈ { z 1 , z 2 } Z a 2 − ( z 1 + z 2 ) Z a + z 1 z 2 = 0 Z_a^2-(z_1+z_2)Z_a+z_1z_2=0 Z a 2 − ( z 1 + z 2 ) Z a + z 1 z 2 = 0 P a P_a P a E a [ Z a ] = 0 \mathbb{E}_a[Z_a]=0 E a [ Z a ] = 0 z 1 z 2 = − κ 2 ( a ) z_1z_2=-\kappa_2(a) z 1 z 2 = − κ 2 ( a ) Z a Z_a Z a (37) , which implies E a [ R a 2 ] = 0 \mathbb{E}_a[R_a^2]=0 E a [ R a 2 ] = 0
Since each of the functions that make up (36) is continuous and κ 2 ( a ) > 0 \kappa_2(a)>0 κ 2 ( a ) > 0 a ↦ E a [ R a 2 ] a\mapsto\mathbb{E}_a[R_a^2] a ↦ E a [ R a 2 ] U U U ∣ supp ( Y ) ∣ > 2 |\operatorname{supp}(Y)|>2 ∣ supp ( Y ) ∣ > 2 Γ \Gamma Γ γ ∗ ∈ Γ \gamma^*\in\Gamma γ ∗ ∈ Γ
inf γ ∈ Γ E γ [ R γ 2 ] = E γ ∗ [ R γ ∗ 2 ] > 0. \inf_{\gamma\in\Gamma}\mathbb{E}_\gamma[R_\gamma^2]=\mathbb{E}_{\gamma^*}[R_{\gamma^*}^2]>0. γ ∈ Γ inf E γ [ R γ 2 ] = E γ ∗ [ R γ ∗ 2 ] > 0. Distribution constructions and divergence bounds
Throughout this subsection, we impose Assumption 3 for the GLM constructions. We can take X ∼ N ( 0 , λ I d ) X \sim\mathcal{N}(0,\lambda I_d) X ∼ N ( 0 , λ I d ) X X X β \beta β X / λ X/\sqrt{\lambda} X / λ λ β \sqrt{\lambda}\beta λ β
The notation described next will be used throughout this subsection, unless stated otherwise. Fix some γ 0 ∈ int ( Γ ) \gamma_0 \in\operatorname{int}(\Gamma) γ 0 ∈ int ( Γ ) P γ 0 P_{\gamma_0} P γ 0 P 0 P_0 P 0 c ∗ : = c ∗ ( γ 0 ) c^* := c^*(\gamma_0) c ∗ := c ∗ ( γ 0 ) R : = R γ 0 = H γ 0 + 2 c ∗ S γ 0 R := R_{\gamma_0} = H_{\gamma_0} + 2c^*S_{\gamma_0} R := R γ 0 = H γ 0 + 2 c ∗ S γ 0 Lemma 17 .
We choose a cutoff function by fixing an even smooth function w : R → [ 0 , 1 ] w : \mathbb{R} \to[0,1] w : R → [ 0 , 1 ]
w ( u ) = 1 for ∣ u ∣ ≤ κ , w ( u ) = 0 for ∣ u ∣ ≥ 2 κ , w(u)=1 \quad\text{for } |u|\le\kappa,\qquad w(u)=0 \quad\text{for } |u|\ge2\kappa, w ( u ) = 1 for ∣ u ∣ ≤ κ , w ( u ) = 0 for ∣ u ∣ ≥ 2 κ , where κ \kappa κ Lemma 16 . For β ∈ R d \beta\in\mathbb{R}^d β ∈ R d γ r \gamma_r γ r
Q β , γ r ( d y ∣ X = x ) : = w ( x T β ) P β , γ r ( d y ∣ X = x ) + { 1 − w ( x T β ) } P 0 ( d y ∣ X = x ) , (38) Q_{\beta,\gamma_r}(\mathrm{d}y \mid X=x) := w(x^\mathsf{T}\beta)P_{\beta,\gamma_r}(\mathrm{d}y \mid X=x) + \{1-w(x^\mathsf{T}\beta)\}P_0(\mathrm{d}y \mid X=x), \tag*{(38)} Q β , γ r ( d y ∣ X = x ) := w ( x T β ) P β , γ r ( d y ∣ X = x ) + { 1 − w ( x T β )} P 0 ( d y ∣ X = x ) , (38) where we write P β , γ r ( d y ∣ X = x ) = P γ r + x T β ( d y ) P_{\beta,\gamma_r}(\mathrm{d}y \mid X=x)=P_{\gamma_r+x^\mathsf{T}\beta}(\mathrm{d}y) P β , γ r ( d y ∣ X = x ) = P γ r + x T β ( d y )
Let π r , s ∗ \pi_{r,s^*} π r , s ∗ s ∗ s^* s ∗ ± r / s ∗ \pm r/\sqrt{s^*} ± r / s ∗
Q ‾ r , s ∗ ( n ) : = ∫ Q β , γ r ⊗ n π r , s ∗ ( d β ) . (39) \overline{Q}_{r,s^*}^{(n)} := \int Q_{\beta,\gamma_r}^{\otimes n}\,\pi_{r,s^*}(\mathrm{d}\beta). \tag*{(39)} Q r , s ∗ ( n ) := ∫ Q β , γ r ⊗ n π r , s ∗ ( d β ) . (39) Let
Λ u : = d P γ 0 + u d P γ 0 , L r ( u , y ) : = 1 + w ( u ) { Λ u + γ r − γ 0 ( y ) − 1 } , \Lambda_u := \frac{\mathrm{d}P_{\gamma_0+u}}{\mathrm{d}P_{\gamma_0}},\qquad L_r(u,y):=1+w(u)\{\Lambda_{u+\gamma_r-\gamma_0}(y)-1\}, Λ u := d P γ 0 d P γ 0 + u , L r ( u , y ) := 1 + w ( u ) { Λ u + γ r − γ 0 ( y ) − 1 } , so that for X ∼ N ( 0 , I d ) X \sim\mathcal{N}(0,I_d) X ∼ N ( 0 , I d ) ∥ β ∥ 2 = r \|\beta\|_2=r ∥ β ∥ 2 = r
d Q β , γ r d P 0 ( X , Y ) = L r ( X T β , Y ) . \frac{\mathrm{d}Q_{\beta,\gamma_r}}{\mathrm{d}P_0}(X,Y)=L_r(X^\mathsf{T}\beta,Y). d P 0 d Q β , γ r ( X , Y ) = L r ( X T β , Y ) . For β , β ′ ∈ R d \beta,\beta' \in\mathbb{R}^d β , β ′ ∈ R d ∥ β ∥ 2 = ∥ β ′ ∥ 2 = r \|\beta\|_2=\|\beta'\|_2=r ∥ β ∥ 2 = ∥ β ′ ∥ 2 = r ρ : = β T β ′ \rho:=\beta^\mathsf{T}\beta' ρ := β T β ′ U : = X T β U:=X^\mathsf{T}\beta U := X T β V : = X T β ′ V:=X^\mathsf{T}\beta' V := X T β ′
( U , V ) ∼ N ( 0 , ( r 2 ρ ρ r 2 ) ) . (U,V)\sim\mathcal{N}\left(0,\begin{pmatrix}r^2&\rho\\\rho&r^2\end{pmatrix}\right). ( U , V ) ∼ N ( 0 , ( r 2 ρ ρ r 2 ) ) . We define the pairwise likelihood ratio overlap by
M r ( ρ ) : = E ρ E γ 0 [ L r ( U , Y ) L r ( V , Y ) ] , (40) M_r(\rho):=\mathbb{E}_{\rho}\mathbb{E}_{\gamma_0}[L_r(U,Y)L_r(V,Y)], \tag*{(40)} M r ( ρ ) := E ρ E γ 0 [ L r ( U , Y ) L r ( V , Y )] , (40) where E ρ \mathbb{E}_{\rho} E ρ E γ 0 \mathbb{E}_{\gamma_0} E γ 0 Y ∼ P γ 0 Y\sim P_{\gamma_0} Y ∼ P γ 0 ( U , V ) (U,V) ( U , V ) P γ 0 P_{\gamma_0} P γ 0
Lemma 18. Let M r ( ρ ) M_r(\rho) M r ( ρ ) (40) . Suppose that, for some fixed C > 0 C>0 C > 0 0 < r ≤ r 0 0<r\le r_0 0 < r ≤ r 0 a r ∈ R a_r\in\mathbb{R} a r ∈ R B r ≥ 0 B_r\ge0 B r ≥ 0 ∣ a r ∣ ≤ C |a_r|\le C ∣ a r ∣ ≤ C
log M r ( ρ ) ≤ a r ρ + C ρ 2 + B r , ∣ ρ ∣ ≤ r 2 . (41) \log M_r(\rho)\le a_r\rho+C\rho^2+B_r,\qquad|\rho|\le r^2. \tag*{(41)} log M r ( ρ ) ≤ a r ρ + C ρ 2 + B r , ∣ ρ ∣ ≤ r 2 . (41) There exists a constant L L L C , r 0 C,r_0 C , r 0 Q ˉ r , s ∗ ( n ) \bar{Q}_{r,s^*}^{(n)} Q ˉ r , s ∗ ( n ) (41) , we have
D K L ( Q ˉ r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≤ n B r + ( s ∗ ) 2 d { cosh ( L n r 2 s ∗ ) − 1 } , 0 < r ≤ r 0 . \mathrm{D}_{\mathrm{KL}}\left(\bar{Q}_{r,s^*}^{(n)} \middle\| P_0^{\otimes n}\right) \le nB_r+\frac{(s^*)^2}{d}\left\{\cosh\left(\frac{Lnr^2}{s^*}\right)-1\right\}, \qquad0<r\le r_0. D KL ( Q ˉ r , s ∗ ( n ) P 0 ⊗ n ) ≤ n B r + d ( s ∗ ) 2 { cosh ( s ∗ L n r 2 ) − 1 } , 0 < r ≤ r 0 . Proof. For independent β , β ′ ∼ π r , s ∗ \beta,\beta'\sim\pi_{r,s^*} β , β ′ ∼ π r , s ∗ K : = ∣ S ∩ S ′ ∣ K:=|S\cap S'| K := ∣ S ∩ S ′ ∣ S , S ′ S,S' S , S ′ ρ : = β T β ′ \rho:=\beta^\mathsf{T}\beta' ρ := β T β ′ S S S S ′ S' S ′ ρ \rho ρ ( r 2 / s ∗ ) ∑ j = 1 K s j (r^2/s^*)\sum_{j=1}^K s_j ( r 2 / s ∗ ) ∑ j = 1 K s j s j s_j s j
D K L ( Q ˉ r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≤ log { 1 + χ 2 ( Q ˉ r , s ∗ ( n ) ∥ P 0 ⊗ n ) } = log E β , β ′ [ M r ( β T β ′ ) n ] ≤ log E [ exp ( n a r ρ + C n ρ 2 ) ] + n B r . (42) \begin{aligned}
\mathrm{D}_{\mathrm{KL}}\left(\bar{Q}_{r,s^*}^{(n)} \middle\| P_0^{\otimes n}\right)\le\log\left\{1+\chi^2\left(\bar{Q}_{r,s^*}^{(n)}\middle\|P_0^{\otimes n}\right)\right\} \\
&=\log\mathbb{E}_{\beta,\beta'}\left[M_r(\beta^\mathsf{T}\beta')^n\right] \\
&\le\log\mathbb{E}\left[\exp(na_r\rho+Cn\rho^2)\right]+nB_r. \tag*{(42)}
\end{aligned} D KL ( Q ˉ r , s ∗ ( n ) P 0 ⊗ n ) ≤ log { 1 + χ 2 ( Q ˉ r , s ∗ ( n ) P 0 ⊗ n ) } = log E β , β ′ [ M r ( β T β ′ ) n ] ≤ log E [ exp ( n a r ρ + C n ρ 2 ) ] + n B r . (42) where the first inequality is due to Theorem 2.7 of Tsybakov [42 ] and the second is due to (41) . Since ρ \rho ρ
E [ exp ( n a r ρ + C n ρ 2 ) ] = E [ exp ( C n ρ 2 ) { exp ( n a r ρ ) + exp ( − n a r ρ ) 2 } ] = E [ exp ( C n ρ 2 ) cosh ( n a r ρ ) ] . \begin{aligned}
\mathbb{E}[\exp(na_r\rho+Cn\rho^2)]=\mathbb{E}\left[\exp(Cn\rho^2)\left\{\frac{\exp(na_r\rho)+\exp(-na_r\rho)}{2}\right\}\right] \\
&=\mathbb{E}[\exp(Cn\rho^2)\cosh(na_r\rho)].
\end{aligned} E [ exp ( n a r ρ + C n ρ 2 )] = E [ exp ( C n ρ 2 ) { 2 exp ( n a r ρ ) + exp ( − n a r ρ ) } ] = E [ exp ( C n ρ 2 ) cosh ( n a r ρ )] . We claim that, for L ≥ max { 1 , 2 C , 2 C / tanh ( 1 ) , 2 C r 0 2 / tanh ( 1 ) } L\ge\max\{1,2C,2\sqrt{C}/\tanh(1),2Cr_0^2/\tanh(1)\} L ≥ max { 1 , 2 C , 2 C / tanh ( 1 ) , 2 C r 0 2 / tanh ( 1 )}
exp ( C n x 2 ) cosh ( η ∣ a x ∣ ) ≤ cosh ( L n x ) , ∣ α ∣ ≤ C , ∣ x ∣ ≤ r 0 2 . (43) \exp(Cnx^2)\cosh(\eta|ax|)\le\cosh(Lnx),\qquad|\alpha|\le C,\quad|x|\le r_0^2. \tag*{(43)} exp ( C n x 2 ) cosh ( η ∣ a x ∣ ) ≤ cosh ( L n x ) , ∣ α ∣ ≤ C , ∣ x ∣ ≤ r 0 2 . (43) We can see this from
log cosh ( L n ∣ x ∣ ) − log cosh ( η ∣ a x ∣ ) ≥ ∫ L n ∣ x ∣ / 2 L n ∣ x ∣ tanh ( t ) d t ≥ L n ∣ x ∣ 2 tanh ( L n ∣ x ∣ / 2 ) ≥ tanh ( 1 ) min { L 2 n 2 x 2 4 , L n ∣ x ∣ 2 } ≥ C n x 2 \begin{aligned}
\log\cosh(Ln|x|)-\log\cosh(\eta|ax|)\ge\int_{Ln|x|/2}^{Ln|x|}\tanh(t)\,\mathrm{d}t\ge\frac{Ln|x|}{2}\tanh(Ln|x|/2) \\
&\ge\tanh(1)\min\left\{\frac{L^2n^2x^2}{4},\frac{Ln|x|}{2}\right\}\ge Cnx^2
\end{aligned} log cosh ( L n ∣ x ∣ ) − log cosh ( η ∣ a x ∣ ) ≥ ∫ L n ∣ x ∣/2 L n ∣ x ∣ tanh ( t ) d t ≥ 2 L n ∣ x ∣ tanh ( L n ∣ x ∣/2 ) ≥ tanh ( 1 ) min { 4 L 2 n 2 x 2 , 2 L n ∣ x ∣ } ≥ C n x 2 where the first inequality is due to η ∣ a x ∣ ≤ L n ∣ x ∣ / 2 \eta|ax|\le Ln|x|/2 η ∣ a x ∣ ≤ L n ∣ x ∣/2 tanh ( t ) ≥ tanh ( 1 ) min ( t , 1 ) \tanh(t)\ge\tanh(1)\min(t,1) tanh ( t ) ≥ tanh ( 1 ) min ( t , 1 ) t ≥ 0 t\ge0 t ≥ 0 (43) . Since ∣ ρ ∣ ≤ r 2 ≤ r 0 2 |\rho|\le r^2\le r_0^2 ∣ ρ ∣ ≤ r 2 ≤ r 0 2
E [ exp ( n a r ρ + C n ρ 2 ) ] ≤ E [ cosh ( L n ρ ) ] = E [ { cosh ( L n r 2 s ∗ ) } K ] . (44) \mathbb{E}[\exp(na_r\rho+Cn\rho^2)]\le\mathbb{E}[\cosh(Ln\rho)]=\mathbb{E}\left[\left\{\cosh\left(\frac{Lnr^2}{s^*}\right)\right\}^K\right]. \tag*{(44)} E [ exp ( n a r ρ + C n ρ 2 )] ≤ E [ cosh ( L n ρ )] = E [ { cosh ( s ∗ L n r 2 ) } K ] . (44) For fixed k ∈ { 0 , 1 , … , s ∗ } k\in\{0,1,\ldots,s^*\} k ∈ { 0 , 1 , … , s ∗ }
E [ ( K k ) ] = E S { ∑ A ⊆ S ∣ A ∣ = k P ( A ⊆ S ′ ∣ S ) } = E S { E [ ∑ A ⊆ S ∣ A ∣ = k 1 { A ⊆ S ′ } ∣ S ] } = ( s ∗ k ) ( d − k s ∗ − k ) ( d s ∗ ) = ( s ∗ k ) ∏ i = 0 k − 1 s ∗ − i d − i ≤ ( s ∗ k ) ( s ∗ d ) k , \begin{aligned}
\mathbb{E}\left[\binom{K}{k}\right]=\mathbb{E}_S\left\{\sum_{\substack{A\subseteq S\\|A|=k}}\mathbb{P}(A\subseteq S'\mid S)\right\}=\mathbb{E}_S\left\{\mathbb{E}\left[\left.\sum_{\substack{A\subseteq S\\|A|=k}}\mathbf{1}\{A\subseteq S'\}\right|S\right]\right\} \\
&=\binom{s^*}{k}\frac{\binom{d-k}{s^*-k}}{\binom{d}{s^*}}=\binom{s^*}{k}\prod_{i=0}^{k-1}\frac{s^*-i}{d-i}\le\binom{s^*}{k}\left(\frac{s^*}{d}\right)^k,
\end{aligned} E [ ( k K ) ] = E S ⎩ ⎨ ⎧ A ⊆ S ∣ A ∣ = k ∑ P ( A ⊆ S ′ ∣ S ) ⎭ ⎬ ⎫ = E S ⎩ ⎨ ⎧ E A ⊆ S ∣ A ∣ = k ∑ 1 { A ⊆ S ′ } S ⎭ ⎬ ⎫ = ( k s ∗ ) ( s ∗ d ) ( s ∗ − k d − k ) = ( k s ∗ ) i = 0 ∏ k − 1 d − i s ∗ − i ≤ ( k s ∗ ) ( d s ∗ ) k , where for k = 0 k=0 k = 0 z K = ∑ k = 0 s ∗ ( s ∗ k ) ( z − 1 ) k z^K=\sum_{k=0}^{s^*}\binom{s^*}{k}(z-1)^k z K = ∑ k = 0 s ∗ ( k s ∗ ) ( z − 1 ) k z = cosh ( L n r 2 / s ∗ ) z=\cosh(Lnr^2/s^*) z = cosh ( L n r 2 / s ∗ )
E [ { cosh ( L n r 2 / s ∗ ) } K ] ≤ ∑ k = 0 s ∗ ( cosh ( L n r 2 / s ∗ ) − 1 ) k ( s ∗ k ) ( s ∗ d ) k = { 1 + s ∗ d ( cosh ( L n r 2 / s ∗ ) − 1 ) } s ∗ ≤ exp { ( s ∗ ) 2 d ( cosh ( L n r 2 / s ∗ ) − 1 ) } . \begin{aligned}
\mathbb{E}\left[\left\{\cosh(Lnr^2/s^*)\right\}^K\right]\leq\sum_{k=0}^{s^*}\left(\cosh(Lnr^2/s^*)-1\right)^k\binom{s^*}{k}\left(\frac{s^*}{d}\right)^k \\
&=\left\{1+\frac{s^*}{d}\left(\cosh(Lnr^2/s^*)-1\right)\right\}^{s^*} \\
&\leq\exp\left\{\frac{(s^*)^2}{d}\left(\cosh(Lnr^2/s^*)-1\right)\right\}.
\end{aligned} E [ { cosh ( L n r 2 / s ∗ ) } K ] ≤ k = 0 ∑ s ∗ ( cosh ( L n r 2 / s ∗ ) − 1 ) k ( k s ∗ ) ( d s ∗ ) k = { 1 + d s ∗ ( cosh ( L n r 2 / s ∗ ) − 1 ) } s ∗ ≤ exp { d ( s ∗ ) 2 ( cosh ( L n r 2 / s ∗ ) − 1 ) } . Substituting the final expression into (44) concludes the proof. □
Lemma 19. Under Assumption 3, there exist constants r 0 , C > 0 r_0,C>0 r 0 , C > 0 0 < r ≤ r 0 0<r\leq r_0 0 < r ≤ r 0 γ r = γ 0 + c ∗ r 2 \gamma_r=\gamma_0+c^*r^2 γ r = γ 0 + c ∗ r 2
D K L ( Q ‾ r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≤ C n r 4 + ( s ∗ ) 2 d { cosh ( L n r 2 s ∗ ) − 1 } . (45) D_{\mathrm{KL}}\left(\overline{Q}_{r,s^*}^{(n)}\|P_0^{\otimes n}\right)\leq Cnr^4+\frac{(s^*)^2}{d}\left\{\cosh\left(\frac{Lnr^2}{s^*}\right)-1\right\}. \tag*{(45)} D KL ( Q r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≤ C n r 4 + d ( s ∗ ) 2 { cosh ( s ∗ L n r 2 ) − 1 } . (45) and, for each fixed alternative component,
D T V ( P β , γ r , Q β , γ r ) ≲ 2 exp { − κ 2 2 r 2 } . (46) D_{\mathrm{TV}}\left(P_{\beta,\gamma_r},Q_{\beta,\gamma_r}\right)\lesssim2\exp\left\{-\frac{\kappa^2}{2r^2}\right\}. \tag*{(46)} D TV ( P β , γ r , Q β , γ r ) ≲ 2 exp { − 2 r 2 κ 2 } . (46) Moreover, we have that
D T V ( P β , γ r , P 0 ) ≲ r + exp { − κ 2 2 r 2 } . (47) D_{\mathrm{TV}}\left(P_{\beta,\gamma_r},P_0\right)\lesssim r+\exp\left\{-\frac{\kappa^2}{2r^2}\right\}. \tag*{(47)} D TV ( P β , γ r , P 0 ) ≲ r + exp { − 2 r 2 κ 2 } . (47) Proof. By Lemma 16 , uniformly where ∣ u ∣ ≤ κ |u|\leq\kappa ∣ u ∣ ≤ κ
Λ u = 1 + u S γ 0 + u 2 2 H γ 0 + O L 2 ( P 0 ) ( ∣ u ∣ 3 ) . \Lambda_u=1+uS_{\gamma_0}+\frac{u^2}{2}H_{\gamma_0}+O_{L_2(P_0)}(|u|^3). Λ u = 1 + u S γ 0 + 2 u 2 H γ 0 + O L 2 ( P 0 ) ( ∣ u ∣ 3 ) . Let A : = U + c ∗ r 2 A:=U+c^*r^2 A := U + c ∗ r 2 U ∼ N ( 0 , r 2 ) U\sim\mathcal{N}(0,r^2) U ∼ N ( 0 , r 2 ) κ \kappa κ r 0 r_0 r 0 3 κ ≤ ρ L 3\kappa\leq\rho_L 3 κ ≤ ρ L ∣ c ∗ ∣ r 0 2 ≤ κ |c^*|r_0^2\leq\kappa ∣ c ∗ ∣ r 0 2 ≤ κ Lemma 16 we have uniform L 2 ( P 0 ) L_2(P_0) L 2 ( P 0 ) L r ( ⋅ , Y ) L_r(\cdot,Y) L r ( ⋅ , Y ) w ( U ) ≠ 1 w(U)\ne1 w ( U ) = 1 ∣ U ∣ > κ |U|>\kappa ∣ U ∣ > κ O ( e − c / r 2 ) O(e^{-c/r^2}) O ( e − c / r 2 )
E U [ L r ( U , Y ) ] = 1 + E [ A ] S γ 0 + E [ A 2 ] 2 H γ 0 + o L 2 ( r 2 ) = 1 + r 2 2 R + o L 2 ( r 2 ) . \begin{aligned}
\mathbb{E}_U[L_r(U,Y)]=1+\mathbb{E}[A]S_{\gamma_0}+\frac{\mathbb{E}[A^2]}{2}H_{\gamma_0}+o_{L_2}(r^2) \\
&=1+\frac{r^2}{2}R+o_{L_2}(r^2).
\end{aligned} E U [ L r ( U , Y )] = 1 + E [ A ] S γ 0 + 2 E [ A 2 ] H γ 0 + o L 2 ( r 2 ) = 1 + 2 r 2 R + o L 2 ( r 2 ) . Recalling (40) , independence of U U U V V V ρ = 0 \rho=0 ρ = 0
M r ( 0 ) − 1 = E γ 0 [ { E U L r ( U , Y ) } 2 − 1 ] = E γ 0 [ { E U L r ( U , Y ) − 1 } 2 ] = r 4 4 E γ 0 [ R 2 ] + o ( r 4 ) = E γ 0 [ R 2 ] 4 r 4 + o ( r 4 ) , \begin{aligned}
M_r(0)-1=\mathbb{E}_{\gamma_0}\left[\{\mathbb{E}_U L_r(U,Y)\}^2-1\right] \\
&=\mathbb{E}_{\gamma_0}\left[\{\mathbb{E}_U L_r(U,Y)-1\}^2\right] \\
&=\frac{r^4}{4}\mathbb{E}_{\gamma_0}[R^2]+o(r^4)=\frac{\mathbb{E}_{\gamma_0}[R^2]}{4}r^4+o(r^4),
\end{aligned} M r ( 0 ) − 1 = E γ 0 [ { E U L r ( U , Y ) } 2 − 1 ] = E γ 0 [ { E U L r ( U , Y ) − 1 } 2 ] = 4 r 4 E γ 0 [ R 2 ] + o ( r 4 ) = 4 E γ 0 [ R 2 ] r 4 + o ( r 4 ) , where the second equality uses E γ 0 E U [ L r ( U , Y ) ] = 1 \mathbb{E}_{\gamma_0}\mathbb{E}_U[L_r(U,Y)] = 1 E γ 0 E U [ L r ( U , Y )] = 1
By the Cauchy–Schwarz inequality and Lemma 16 , we have ∣ E 0 [ ∂ u L r ( u , Y ) ∂ v L r ( v , Y ) ] ∣ ≤ C \left|\mathbb{E}_0[\partial_u L_r(u,Y)\,\partial_v L_r(v,Y)]\right| \le C ∣ E 0 [ ∂ u L r ( u , Y ) ∂ v L r ( v , Y )] ∣ ≤ C C C C [44 ] ), we have for ∣ ρ ∣ < r 2 |\rho| < r^2 ∣ ρ ∣ < r 2
M r ′ ( ρ ) = E ρ E γ 0 [ ∂ u L r ( U , Y ) ∂ v L r ( V , Y ) ] , M_r'(\rho) = \mathbb{E}_{\rho}\mathbb{E}_{\gamma_0}[\partial_u L_r(U,Y)\partial_v L_r(V,Y)], M r ′ ( ρ ) = E ρ E γ 0 [ ∂ u L r ( U , Y ) ∂ v L r ( V , Y )] , so averaging over ( U , V ) (U,V) ( U , V ) ∣ M r ′ ( ρ ) ∣ ≤ C |M_r'(\rho)| \le C ∣ M r ′ ( ρ ) ∣ ≤ C ∣ M r ′ ′ ( ρ ) ∣ ≤ C |M_r''(\rho)| \le C ∣ M r ′′ ( ρ ) ∣ ≤ C M r ( 0 ) = 1 + O ( r 4 ) M_r(0) = 1 + O(r^4) M r ( 0 ) = 1 + O ( r 4 ) M r ( ρ ) M_r(\rho) M r ( ρ ) ∣ ρ ∣ < r 2 |\rho| < r^2 ∣ ρ ∣ < r 2 r r r
sup ∣ ρ ∣ < r 2 ∣ ( log M r ) ′ ′ ( ρ ) ∣ = sup ∣ ρ ∣ < r 2 ∣ M r ′ ′ ( ρ ) M r ( ρ ) − { M r ′ ( ρ ) M r ( ρ ) } 2 ∣ ≤ C , \sup_{|\rho|<r^2}|(\log M_r)''(\rho)| = \sup_{|\rho|<r^2}\left|\frac{M_r''(\rho)}{M_r(\rho)} - \left\{\frac{M_r'(\rho)}{M_r(\rho)}\right\}^2\right| \le C, ∣ ρ ∣ < r 2 sup ∣ ( log M r ) ′′ ( ρ ) ∣ = ∣ ρ ∣ < r 2 sup M r ( ρ ) M r ′′ ( ρ ) − { M r ( ρ ) M r ′ ( ρ ) } 2 ≤ C , so taking a Taylor expansion at ρ = 0 \rho= 0 ρ = 0
log M r ( ρ ) ≤ a r ρ + C ρ 2 + C r 4 , a r : = ( log M r ) ′ ( 0 ) , ∣ a r ∣ ≤ C . (48) \log M_r(\rho) \le a_r\rho+ C\rho^2 + Cr^4,\qquad a_r := (\log M_r)'(0),\qquad|a_r| \le C. \tag*{(48)} log M r ( ρ ) ≤ a r ρ + C ρ 2 + C r 4 , a r := ( log M r ) ′ ( 0 ) , ∣ a r ∣ ≤ C . (48) The choice of smooth cutoff function w w w Lemma 16 imply that ( u , v ) ↦ E γ 0 [ L r ( u , Y ) L r ( v , Y ) ] (u,v) \mapsto\mathbb{E}_{\gamma_0}[L_r(u,Y)L_r(v,Y)] ( u , v ) ↦ E γ 0 [ L r ( u , Y ) L r ( v , Y )] ( U , V ) (U,V) ( U , V ) ρ → ± r 2 \rho\to\pm r^2 ρ → ± r 2 M r ( ρ ) M_r(\rho) M r ( ρ ) (48) holds for ∣ ρ ∣ ≤ r 2 |\rho| \le r^2 ∣ ρ ∣ ≤ r 2 Lemma 18 with B r = C r 4 B_r = Cr^4 B r = C r 4 (45) .
Since P β , γ r P_{\beta,\gamma_r} P β , γ r Q β , γ r Q_{\beta,\gamma_r} Q β , γ r X X X
D T V ( P β , γ r , Q β , γ r ) ≤ E X [ D T V ( P β , γ r ( ⋅ ∣ X ) , Q β , γ r ( ⋅ ∣ X ) ) ] ≤ E X [ 1 − w ( X T β ) ] ≤ P X ( ∣ X T β ∣ > κ ) \begin{aligned}
D_{\mathrm{TV}}(P_{\beta,\gamma_r},Q_{\beta,\gamma_r}) \le\mathbb{E}_X[D_{\mathrm{TV}}(P_{\beta,\gamma_r}(\cdot\mid X),Q_{\beta,\gamma_r}(\cdot\mid X))] \\
&\le\mathbb{E}_X[1-w(X^\mathsf{T}\beta)] \\
&\le\mathbb{P}_X(|X^\mathsf{T}\beta|>\kappa)
\end{aligned} D TV ( P β , γ r , Q β , γ r ) ≤ E X [ D TV ( P β , γ r ( ⋅ ∣ X ) , Q β , γ r ( ⋅ ∣ X ))] ≤ E X [ 1 − w ( X T β )] ≤ P X ( ∣ X T β ∣ > κ ) ≤ 2 exp ( − κ 2 / 2 r 2 ) . \le2\exp(-\kappa^2/2r^2). ≤ 2 exp ( − κ 2 /2 r 2 ) . where in the third inequality we used that w ( u ) = 1 w(u) = 1 w ( u ) = 1 ∣ u ∣ ≤ κ |u| \le\kappa ∣ u ∣ ≤ κ
Finally, to show the total variation distance bound in (47) , we start with the triangle inequality
D T V ( P β , γ r , P 0 ) ≤ D T V ( P β , γ r , Q β , γ r ) + D T V ( Q β , γ r , P 0 ) . (49) D_{\mathrm{TV}}(P_{\beta,\gamma_r},P_0) \le D_{\mathrm{TV}}(P_{\beta,\gamma_r},Q_{\beta,\gamma_r}) + D_{\mathrm{TV}}(Q_{\beta,\gamma_r},P_0). \tag*{(49)} D TV ( P β , γ r , P 0 ) ≤ D TV ( P β , γ r , Q β , γ r ) + D TV ( Q β , γ r , P 0 ) . (49) The first term can be bounded by (46) . To bound the second term, for t : = X T β + c ∗ r 2 t := X^\mathsf{T}\beta+ c^*r^2 t := X T β + c ∗ r 2 η t : = ϑ ( γ 0 + t ) \eta_t := \vartheta(\gamma_0+t) η t := ϑ ( γ 0 + t ) η 0 : = ϑ ( γ 0 ) \eta_0 := \vartheta(\gamma_0) η 0 := ϑ ( γ 0 ) ∣ t ∣ < 3 κ |t| < 3\kappa ∣ t ∣ < 3 κ
D K L ( P γ 0 + t ∥ P γ 0 ) = E γ 0 + t [ ( η t − η 0 ) Y − b ( η t ) − b ( η 0 ) ] = ( η t − η 0 ) b ′ ( η t ) − b ( η t ) − b ( η 0 ) = ∫ η 0 η t { b ′ ( η t ) − b ′ ( s ) } d s . \begin{aligned}
D_{\mathrm{KL}}(P_{\gamma_0+t}\mathbin\Vert P_{\gamma_0}) = \mathbb{E}_{\gamma_0+t}[(\eta_t-\eta_0)Y-b(\eta_t)-b(\eta_0)] \\
&= (\eta_t-\eta_0)b'(\eta_t)-b(\eta_t)-b(\eta_0) \\
&= \int_{\eta_0}^{\eta_t}\{b'(\eta_t)-b'(s)\}\,\mathrm{d}s.
\end{aligned} D KL ( P γ 0 + t ∥ P γ 0 ) = E γ 0 + t [( η t − η 0 ) Y − b ( η t ) − b ( η 0 )] = ( η t − η 0 ) b ′ ( η t ) − b ( η t ) − b ( η 0 ) = ∫ η 0 η t { b ′ ( η t ) − b ′ ( s )} d s . By the choice of κ \kappa κ γ 0 + t ∈ Γ 2 \gamma_0+t \in\Gamma_2 γ 0 + t ∈ Γ 2 ∣ t ∣ ≤ 3 κ |t| \le3\kappa ∣ t ∣ ≤ 3 κ ϑ ′ \vartheta' ϑ ′ b ′ ′ b'' b ′′ Γ 2 \Gamma_2 Γ 2 ϑ ( Γ 2 ) ⊂ Θ \vartheta(\Gamma_2) \subset\Theta ϑ ( Γ 2 ) ⊂ Θ Γ 2 \Gamma_2 Γ 2 Lemma 16 , the preceding identity and the mean value theorem give
D K L ( P γ 0 + t ∥ P γ 0 ) ≤ C ∣ η t − η 0 ∣ 2 ≤ C t 2 . D_{\mathrm{KL}}(P_{\gamma_0+t}\mathbin\Vert P_{\gamma_0}) \le C|\eta_t-\eta_0|^2 \le Ct^2. D KL ( P γ 0 + t ∥ P γ 0 ) ≤ C ∣ η t − η 0 ∣ 2 ≤ C t 2 . We can now bound
D K L ( Q β , γ r ∥ P 0 ) ≤ E X [ w ( X T β ) D K L ( P γ 0 + t ∥ P γ 0 ) ] ≤ C E X [ w ( X T β ) ( X T β + c ∗ r 2 ) 2 ] ≲ r 2 + ( c ∗ ) 2 r 4 ≲ r 2 . D_{\mathrm{KL}}(Q_{\beta,\gamma_r}\mathbin\Vert P_0) \leq\mathbb{E}_X\left[w(X^{\mathsf{T}}\beta)D_{\mathrm{KL}}(P_{\gamma_0+t}\mathbin\Vert P_{\gamma_0})\right]
\leq C\mathbb{E}_X\left[w(X^{\mathsf{T}}\beta)(X^{\mathsf{T}}\beta+c^*r^2)^2\right]
\lesssim r^2+(c^*)^2r^4\lesssim r^2. D KL ( Q β , γ r ∥ P 0 ) ≤ E X [ w ( X T β ) D KL ( P γ 0 + t ∥ P γ 0 ) ] ≤ C E X [ w ( X T β ) ( X T β + c ∗ r 2 ) 2 ] ≲ r 2 + ( c ∗ ) 2 r 4 ≲ r 2 . Therefore, using Pinsker’s inequality and substituting into (49) finishes the proof for (47) .
Lemma 20 (Construction for GLMs with two support points). Consider the Gaussian design and prior in Appendix § 7 .§ 3 . Suppose that the response family has two-point support and satisfies Assumption 3, and fix γ 0 ∈ int ( Γ ) \gamma_0 \in\operatorname{int}(\Gamma) γ 0 ∈ int ( Γ ) r > 0 r>0 r > 0 γ r \gamma_r γ r ∣ γ r − γ 0 ∣ ≲ r 2 |\gamma_r-\gamma_0|\lesssim r^2 ∣ γ r − γ 0 ∣ ≲ r 2 Q β , γ r Q_{\beta,\gamma_r} Q β , γ r (38) , formed using this intercept, has response marginal P γ 0 P_{\gamma_0} P γ 0 L > 0 L>0 L > 0 n ≥ 1 n\geq1 n ≥ 1 1 ≤ s ∗ ≤ d 1\leq s^*\leq d 1 ≤ s ∗ ≤ d
D K L ( Q ‾ r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≤ ( s ∗ ) 2 d { cosh ( L n r 2 s ∗ ) − 1 } . D_{\mathrm{KL}}\left(\overline{Q}_{r,s^*}^{(n)}\mathbin\Vert P_0^{\otimes n}\right)\leq\frac{(s^*)^2}{d}\left\{\cosh\left(\frac{Lnr^2}{s^*}\right)-1\right\}. D KL ( Q r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≤ d ( s ∗ ) 2 { cosh ( s ∗ L n r 2 ) − 1 } . Moreover, uniformly over ∥ β ∥ 2 = r \|\beta\|_2=r ∥ β ∥ 2 = r
D T V ( P β , γ r , Q β , γ r ) ≤ 2 exp ( − κ 2 2 r 2 ) , D T V ( P β , γ r , P 0 ) ≲ r . (50) D_{\mathrm{TV}}(P_{\beta,\gamma_r},Q_{\beta,\gamma_r})\leq2\exp\left(-\frac{\kappa^2}{2r^2}\right),\qquad D_{\mathrm{TV}}(P_{\beta,\gamma_r},P_0)\lesssim r. \tag*{(50)} D TV ( P β , γ r , Q β , γ r ) ≤ 2 exp ( − 2 r 2 κ 2 ) , D TV ( P β , γ r , P 0 ) ≲ r . (50) Proof. Since the support has two points, after a linear mapping, we may assume that Y ∈ { 0 , 1 } Y\in\{0,1\} Y ∈ { 0 , 1 } p ( a ) : = P a ( Y = 1 ) p(a):=P_a(Y=1) p ( a ) := P a ( Y = 1 ) p 0 : = p ( γ 0 ) p_0:=p(\gamma_0) p 0 := p ( γ 0 )
F ( r , γ ) : = E [ w ( r G ) { p ( γ + r G ) − p 0 } ] , G ∼ N ( 0 , 1 ) . F(r,\gamma):=\mathbb{E}\left[w(rG)\{p(\gamma+rG)-p_0\}\right],\qquad G\sim N(0,1). F ( r , γ ) := E [ w ( r G ) { p ( γ + r G ) − p 0 } ] , G ∼ N ( 0 , 1 ) . The proxy has the same response marginal as P γ 0 P_{\gamma_0} P γ 0 F ( r , γ ) = 0 F(r,\gamma)=0 F ( r , γ ) = 0 p p p w w w r r r G G G C ( 1 + ∣ G ∣ + G 2 ) C(1+|G|+G^2) C ( 1 + ∣ G ∣ + G 2 ) F F F ( 0 , γ 0 ) (0,\gamma_0) ( 0 , γ 0 ) r = 0 r=0 r = 0 F ( 0 , γ 0 ) = 0 F(0,\gamma_0)=0 F ( 0 , γ 0 ) = 0 ∂ γ F ( 0 , γ 0 ) = p ′ ( γ 0 ) ≠ 0 \partial_\gamma F(0,\gamma_0)=p'(\gamma_0)\ne0 ∂ γ F ( 0 , γ 0 ) = p ′ ( γ 0 ) = 0 γ ( r ) \gamma(r) γ ( r )
γ ( 0 ) = γ 0 , F ( r , γ ( r ) ) = 0. \gamma(0)=\gamma_0,\qquad F(r,\gamma(r))=0. γ ( 0 ) = γ 0 , F ( r , γ ( r )) = 0. By the symmetry of G G G
F ( − r , γ ) = E [ w ( − r G ) { p ( γ − r G ) − p 0 } ] = E [ w ( r G ) { p ( γ + r G ) − p 0 } ] = F ( r , γ ) , F(-r,\gamma)=\mathbb{E}\left[w(-rG)\{p(\gamma-rG)-p_0\}\right]=\mathbb{E}\left[w(rG)\{p(\gamma+rG)-p_0\}\right]=F(r,\gamma), F ( − r , γ ) = E [ w ( − r G ) { p ( γ − r G ) − p 0 } ] = E [ w ( r G ) { p ( γ + r G ) − p 0 } ] = F ( r , γ ) , and by local uniqueness, γ ( − r ) = γ ( r ) \gamma(-r)=\gamma(r) γ ( − r ) = γ ( r ) γ ′ ( 0 ) = 0 \gamma'(0)=0 γ ′ ( 0 ) = 0 γ r : = γ ( r ) \gamma_r:=\gamma(r) γ r := γ ( r ) ∣ γ r − γ 0 ∣ ≲ r 2 |\gamma_r-\gamma_0|\lesssim r^2 ∣ γ r − γ 0 ∣ ≲ r 2 γ 0 ∈ int ( Γ ) \gamma_0\in\operatorname{int}(\Gamma) γ 0 ∈ int ( Γ ) γ r \gamma_r γ r r r r
Let L r ( u , y ) : = 1 + w ( u ) { Λ u + γ r − γ 0 ( y ) − 1 } L_r(u,y):=1+w(u)\{\Lambda_{u+\gamma_r-\gamma_0}(y)-1\} L r ( u , y ) := 1 + w ( u ) { Λ u + γ r − γ 0 ( y ) − 1 } E U [ L r ( U , y ) ] = 1 \mathbb{E}_U[L_r(U,y)]=1 E U [ L r ( U , y )] = 1 ρ = 0 \rho=0 ρ = 0 (40) , independence of U U U V V V
M r ( 0 ) = E γ 0 [ { E U [ L r ( U , Y ) ] } 2 ] = 1. M_r(0)=\mathbb{E}_{\gamma_0}\left[\{\mathbb{E}_U[L_r(U,Y)]\}^2\right]=1. M r ( 0 ) = E γ 0 [ { E U [ L r ( U , Y )] } 2 ] = 1. The derivative bounds used to establish (48) hold uniformly for ∣ γ r − γ 0 ∣ ≲ r 2 |\gamma_r-\gamma_0| \lesssim r^2 ∣ γ r − γ 0 ∣ ≲ r 2 M r ( 0 ) = 1 M_r(0)=1 M r ( 0 ) = 1
log M r ( ρ ) ≤ a r ρ + C ρ 2 , ∣ a r ∣ ≤ C , ∣ ρ ∣ ≤ r 2 . \log M_r(\rho) \le a_r\rho+C\rho^2,\qquad|a_r|\le C,\qquad|\rho|\le r^2. log M r ( ρ ) ≤ a r ρ + C ρ 2 , ∣ a r ∣ ≤ C , ∣ ρ ∣ ≤ r 2 . Applying Lemma 18 with B r = 0 B_r=0 B r = 0 (50) follow from similar arguments to the proof of Lemma 19 with the specified alternative intercept.
Lemma 21 (Construction for linear regression). *Write P β , σ P_{\beta,\sigma} P β , σ X ∼ N ( 0 , I d ) X\sim\mathcal{N}(0,I_d) X ∼ N ( 0 , I d ) Y ∣ X = x ∼ N ( β ⊤ x , σ 2 ) Y\mid X=x\sim\mathcal{N}(\beta^\top x,\sigma^2) Y ∣ X = x ∼ N ( β ⊤ x , σ 2 ) σ 0 > 0 \sigma_0>0 σ 0 > 0 M ξ M_\xi M ξ p ξ p_\xi p ξ P ( ∣ σ 0 G ∣ ≤ M ξ ) ≥ p ξ \mathbb{P}(|\sigma_0G|\le M_\xi)\ge p_\xi P ( ∣ σ 0 G ∣ ≤ M ξ ) ≥ p ξ G ∼ N ( 0 , 1 ) G\sim\mathcal{N}(0,1) G ∼ N ( 0 , 1 ) σ r 2 : = σ 0 2 − r 2 \sigma_r^2:=\sigma_0^2-r^2 σ r 2 := σ 0 2 − r 2 r 2 < σ 0 2 / 2 r^2<\sigma_0^2/2 r 2 < σ 0 2 /2 N ( 0 , σ 0 2 ) \mathcal{N}(0,\sigma_0^2) N ( 0 , σ 0 2 ) N ( 0 , σ r 2 ) \mathcal{N}(0,\sigma_r^2) N ( 0 , σ r 2 ) P 0 , σ 0 P_{0,\sigma_0} P 0 , σ 0 n ≥ 1 n\ge1 n ≥ 1 1 ≤ s ∗ ≤ d 1\le s^*\le d 1 ≤ s ∗ ≤ d
P ‾ r , s ∗ ( n ) : = ∫ P β , σ r ⊗ n π r , s ∗ ( d β ) . \overline{P}_{r,s^*}^{(n)}:=\int P_{\beta,\sigma_r}^{\otimes n}\,\pi_{r,s^*}(d\beta). P r , s ∗ ( n ) := ∫ P β , σ r ⊗ n π r , s ∗ ( d β ) . There exists a constant L > 0 L>0 L > 0 σ 0 \sigma_0 σ 0
D K L ( P ‾ r , s ∗ ( n ) ∥ P 0 , σ 0 ⊗ n ) ≤ ( s ∗ ) 2 d { cosh ( L n r 2 s ∗ ) − 1 } , 0 < r 2 < σ 0 2 2 . (51) D_{\mathrm{KL}}\left(\overline{P}_{r,s^*}^{(n)}\middle\|P_{0,\sigma_0}^{\otimes n}\right)\le\frac{(s^*)^2}{d}\left\{\cosh\left(\frac{Lnr^2}{s^*}\right)-1\right\},\qquad0<r^2<\frac{\sigma_0^2}{2}. \tag*{(51)} D KL ( P r , s ∗ ( n ) P 0 , σ 0 ⊗ n ) ≤ d ( s ∗ ) 2 { cosh ( s ∗ L n r 2 ) − 1 } , 0 < r 2 < 2 σ 0 2 . (51) Moreover for every fixed β \beta β ∥ β ∥ 2 = r \|\beta\|_2=r ∥ β ∥ 2 = r D T V ( P β , σ r , P 0 , σ 0 ) ≲ r D_{\mathrm{TV}}(P_{\beta,\sigma_r},P_{0,\sigma_0})\lesssim r D TV ( P β , σ r , P 0 , σ 0 ) ≲ r
Proof. Since σ r ≤ σ 0 \sigma_r\le\sigma_0 σ r ≤ σ 0 σ ∈ { σ 0 , σ r } \sigma\in\{\sigma_0,\sigma_r\} σ ∈ { σ 0 , σ r }
P ( ∣ σ G ∣ ≤ M ξ ) ≥ P ( ∣ σ 0 G ∣ ≤ M ξ ) ≥ p ξ , \mathbb{P}(|\sigma G|\le M_\xi)\ge\mathbb{P}(|\sigma_0G|\le M_\xi)\ge p_\xi, P ( ∣ σ G ∣ ≤ M ξ ) ≥ P ( ∣ σ 0 G ∣ ≤ M ξ ) ≥ p ξ , which verifies Assumption 2.
For every ∥ β ∥ 2 = r \|\beta\|_2=r ∥ β ∥ 2 = r X ⊤ β ∼ N ( 0 , r 2 ) X^\top\beta\sim\mathcal{N}(0,r^2) X ⊤ β ∼ N ( 0 , r 2 ) P β , σ r P_{\beta,\sigma_r} P β , σ r
Y ∼ N ( 0 , r 2 + σ r 2 ) = N ( 0 , σ 0 2 ) , Y\sim\mathcal{N}(0,r^2+\sigma_r^2)=\mathcal{N}(0,\sigma_0^2), Y ∼ N ( 0 , r 2 + σ r 2 ) = N ( 0 , σ 0 2 ) , i.e. the response marginal agrees exactly with that under P 0 , σ 0 P_{0,\sigma_0} P 0 , σ 0 U = X ⊤ β U=X^\top\beta U = X ⊤ β V = X ⊤ β ′ V=X^\top\beta' V = X ⊤ β ′ ρ = β ⊤ β ′ \rho=\beta^\top\beta' ρ = β ⊤ β ′
E 0 [ d P β , σ r d P 0 , σ 0 d P β ′ , σ r d P 0 , σ 0 | X ] = σ 0 2 ( σ 0 2 − r 2 ) ( σ 0 2 + r 2 ) exp { ( U + V ) 2 4 ( σ 0 2 + r 2 ) − ( U − V ) 2 4 ( σ 0 2 − r 2 ) } . \mathbb{E}_0\left[\frac{dP_{\beta,\sigma_r}}{dP_{0,\sigma_0}}\frac{dP_{\beta',\sigma_r}}{dP_{0,\sigma_0}}\middle|X\right]
=\frac{\sigma_0^2}{\sqrt{(\sigma_0^2-r^2)(\sigma_0^2+r^2)}}\exp\left\{\frac{(U+V)^2}{4(\sigma_0^2+r^2)}-\frac{(U-V)^2}{4(\sigma_0^2-r^2)}\right\}. E 0 [ d P 0 , σ 0 d P β , σ r d P 0 , σ 0 d P β ′ , σ r X ] = ( σ 0 2 − r 2 ) ( σ 0 2 + r 2 ) σ 0 2 exp { 4 ( σ 0 2 + r 2 ) ( U + V ) 2 − 4 ( σ 0 2 − r 2 ) ( U − V ) 2 } . Taking expectations with respect to X X X
E 0 [ d P β , σ r d P 0 , σ 0 d P β ′ , σ r d P 0 , σ 0 ] = σ 0 2 ( σ 0 2 − r 2 ) ( σ 0 2 + r 2 ) ( σ 0 2 + r 2 σ 0 2 − ρ ) 1 / 2 ( σ 0 2 − r 2 σ 0 2 − ρ ) 1 / 2 = σ 0 2 σ 0 2 − ρ = 1 1 − ρ / σ 0 2 . (52) \begin{aligned}
\mathbb{E}_0\left[\frac{dP_{\beta,\sigma_r}}{dP_{0,\sigma_0}}\frac{dP_{\beta',\sigma_r}}{dP_{0,\sigma_0}}\right]
=\frac{\sigma_0^2}{\sqrt{(\sigma_0^2-r^2)(\sigma_0^2+r^2)}}\left(\frac{\sigma_0^2+r^2}{\sigma_0^2-\rho}\right)^{1/2}\left(\frac{\sigma_0^2-r^2}{\sigma_0^2-\rho}\right)^{1/2} \\
&=\frac{\sigma_0^2}{\sigma_0^2-\rho}=\frac{1}{1-\rho/\sigma_0^2}. \tag*{(52)}
\end{aligned} E 0 [ d P 0 , σ 0 d P β , σ r d P 0 , σ 0 d P β ′ , σ r ] = ( σ 0 2 − r 2 ) ( σ 0 2 + r 2 ) σ 0 2 ( σ 0 2 − ρ σ 0 2 + r 2 ) 1/2 ( σ 0 2 − ρ σ 0 2 − r 2 ) 1/2 = σ 0 2 − ρ σ 0 2 = 1 − ρ / σ 0 2 1 . (52) where we used that U + V U+V U + V U − V U-V U − V 2 ( r 2 + ρ ) 2(r^2+\rho) 2 ( r 2 + ρ ) 2 ( r 2 − ρ ) 2(r^2-\rho) 2 ( r 2 − ρ ) ∣ ρ ∣ ≤ r 2 < σ 0 2 / 2 |\rho|\le r^2<\sigma_0^2/2 ∣ ρ ∣ ≤ r 2 < σ 0 2 /2
log 1 1 − ρ / σ 0 2 ≤ ρ σ 0 2 + C ρ 2 . \log\frac{1}{1-\rho/\sigma_0^2}\le\frac{\rho}{\sigma_0^2}+C\rho^2. log 1 − ρ / σ 0 2 1 ≤ σ 0 2 ρ + C ρ 2 . The expectation in (52) defines the analogous version of M r ( ρ ) M_r(\rho) M r ( ρ ) Lemma 18 with Q β = P β , σ r Q_\beta= P_{\beta,\sigma_r} Q β = P β , σ r P 0 = P 0 , σ 0 P_0 = P_{0,\sigma_0} P 0 = P 0 , σ 0 B r = 0 B_r = 0 B r = 0 (51) .
Finally, for any fixed β \beta β ∥ β ∥ 2 = r \lVert\beta\rVert_2 = r ∥ β ∥ 2 = r
D K L ( P β , σ r ∥ P 0 , σ 0 ) = 1 2 log ( σ 0 2 σ 0 2 − r 2 ) ≲ r 2 , D_{\mathrm{KL}}(P_{\beta,\sigma_r}\mathbin\Vert P_{0,\sigma_0}) = \frac{1}{2}\log\left(\frac{\sigma_0^2}{\sigma_0^2-r^2}\right) \lesssim r^2, D KL ( P β , σ r ∥ P 0 , σ 0 ) = 2 1 log ( σ 0 2 − r 2 σ 0 2 ) ≲ r 2 , so by Pinsker's inequality, we have D T V ( P β , σ r , P 0 , σ 0 ) ≲ r D_{\mathrm{TV}}(P_{\beta,\sigma_r},P_{0,\sigma_0}) \lesssim r D TV ( P β , σ r , P 0 , σ 0 ) ≲ r
Proof of lower bounds
Proof of the lower bounds in Theorem 3. Our constructions in Appendix § 7 .§ 3 lie in the model class after a rescaling of the Gaussian covariates and a corresponding inverse scaling of the regression coefficients, which only changes the separation radius by a fixed constant.
First, suppose 0 < ϵ ≤ 1 0 < \epsilon\le1 0 < ϵ ≤ 1
r n p : = { s ∗ n arcosh ( 1 + d ( s ∗ ) 2 ) } 1 / 2 r p r i v : = { s ∗ n arcosh ( 1 + d n ϵ 2 ( s ∗ ) 2 ) } 1 / 2 . (53) r_{\mathrm{np}} := \left\{\frac{s^*}{n}\operatorname{arcosh}\left(1+\frac{d}{(s^*)^2}\right)\right\}^{1/2}
\qquad
r_{\mathrm{priv}} := \left\{\frac{s^*}{n}\operatorname{arcosh}\left(1+\frac{d}{n\epsilon^2(s^*)^2}\right)\right\}^{1/2}. \tag*{(53)} r np := { n s ∗ arcosh ( 1 + ( s ∗ ) 2 d ) } 1/2 r priv := { n s ∗ arcosh ( 1 + n ϵ 2 ( s ∗ ) 2 d ) } 1/2 . (53) For 0 ≤ λ ≤ 1 0 \le\lambda\le1 0 ≤ λ ≤ 1 cosh \cosh cosh cosh ( λ x ) − 1 ≤ λ 2 { cosh ( x ) − 1 } \cosh(\lambda x)-1 \le\lambda^2\{\cosh(x)-1\} cosh ( λ x ) − 1 ≤ λ 2 { cosh ( x ) − 1 } c > 0 c > 0 c > 0 Lemmas 20 and 21 imply for linear regression and two-point GLMs that
D K L ( Q ‾ r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≲ { c 4 , r = c r n p , c 4 / ( n ϵ 2 ) , r = c r p r i v , D_{\mathrm{KL}}\left(\overline{Q}_{r,s^*}^{(n)}\mathbin\Vert P_0^{\otimes n}\right) \lesssim
\begin{cases}
c^4, & r = c r_{\mathrm{np}},\\
c^4/(n\epsilon^2), & r = c r_{\mathrm{priv}},
\end{cases} D KL ( Q r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≲ { c 4 , c 4 / ( n ϵ 2 ) , r = c r np , r = c r priv , where for linear regression we take Q β = P β , σ r Q_\beta= P_{\beta,\sigma_r} Q β = P β , σ r P 0 = P 0 , σ 0 P_0 = P_{0,\sigma_0} P 0 = P 0 , σ 0 (39) for the two-point GLM.
Recalling the notation of (4) , note that we can bound
min { d 1 / 4 n , s ∗ n } ≲ r n p ≲ min { d 1 / 4 n , s ∗ ℓ n , d , ϵ n } . \min\left\{\frac{d^{1/4}}{\sqrt{n}},\sqrt{\frac{s^*}{n}}\right\} \lesssim r_{\mathrm{np}} \lesssim\min\left\{\frac{d^{1/4}}{\sqrt{n}},\sqrt{\frac{s^*\ell_{n,d,\epsilon}}{n}}\right\}. min { n d 1/4 , n s ∗ } ≲ r np ≲ min { n d 1/4 , n s ∗ ℓ n , d , ϵ } . The lower bound follows from arcosh ( 1 + x ) ≳ min { x , 1 } \operatorname{arcosh}(1+x) \gtrsim\min\{\sqrt{x},1\} arcosh ( 1 + x ) ≳ min { x , 1 }
r n p 2 ≤ s ∗ n 2 d ( s ∗ ) 2 = 2 d n , r_{\mathrm{np}}^2 \le\frac{s^*}{n}\sqrt{\frac{2d}{(s^*)^2}} = \frac{\sqrt{2d}}{n}, r np 2 ≤ n s ∗ ( s ∗ ) 2 2 d = n 2 d , where we used cosh ( t ) ≥ 1 + t 2 / 2 \cosh(t) \ge1+t^2/2 cosh ( t ) ≥ 1 + t 2 /2 arcosh ( 1 + x ) ≤ 2 x \operatorname{arcosh}(1+x) \le\sqrt{2x} arcosh ( 1 + x ) ≤ 2 x
Secondly, we have that
s ∗ n arcosh ( 1 + d ( s ∗ ) 2 ) = s ∗ n log { 1 + d ( s ∗ ) 2 + d ( s ∗ ) 2 ( 2 + d ( s ∗ ) 2 ) } ≤ s ∗ n log { 2 ( 1 + d ( s ∗ ) 2 ) } ≤ s ∗ n log { 2 ( 1 + d ) } ≲ s ∗ ℓ n , d , ϵ n . \begin{aligned}
\frac{s^*}{n}\operatorname{arcosh}\left(1+\frac{d}{(s^*)^2}\right)
= \frac{s^*}{n}\log\left\{1+\frac{d}{(s^*)^2}+\sqrt{\frac{d}{(s^*)^2}\left(2+\frac{d}{(s^*)^2}\right)}\right\} \\
&\le\frac{s^*}{n}\log\left\{2\left(1+\frac{d}{(s^*)^2}\right)\right\} \\
&\le\frac{s^*}{n}\log\{2(1+d)\} \lesssim\frac{s^*\ell_{n,d,\epsilon}}{n}.
\end{aligned} n s ∗ arcosh ( 1 + ( s ∗ ) 2 d ) = n s ∗ log { 1 + ( s ∗ ) 2 d + ( s ∗ ) 2 d ( 2 + ( s ∗ ) 2 d ) } ≤ n s ∗ log { 2 ( 1 + ( s ∗ ) 2 d ) } ≤ n s ∗ log { 2 ( 1 + d )} ≲ n s ∗ ℓ n , d , ϵ . Similar arguments show that
min { d 1 / 4 n 3 / 4 ϵ 1 / 2 , s ∗ n } ≲ r p r i v ≲ min { d 1 / 4 n 3 / 4 ϵ 1 / 2 , s ∗ ℓ n , d , ϵ n } . \min\left\{\frac{d^{1/4}}{n^{3/4}\epsilon^{1/2}},\sqrt{\frac{s^*}{n}}\right\}\lesssim r_{\mathrm{priv}}\lesssim\min\left\{\frac{d^{1/4}}{n^{3/4}\epsilon^{1/2}},\sqrt{\frac{s^*\ell_{n,d,\epsilon}}{n}}\right\}. min { n 3/4 ϵ 1/2 d 1/4 , n s ∗ } ≲ r priv ≲ min { n 3/4 ϵ 1/2 d 1/4 , n s ∗ ℓ n , d , ϵ } . Therefore, for two-point GLMs and linear regression, by Lemma 20 and Lemma 21 respectively, there exist distributions such that we can bound the quantities in Proposition 4 by
α ≲ r , τ ≲ exp ( − κ 2 2 r 2 ) , ω ≲ { c 2 / n , r = c r n p , c 2 / ( n ϵ ) , r = c r p r i v . \alpha\lesssim r,\qquad\tau\lesssim\exp\left(-\frac{\kappa^2}{2r^2}\right),\qquad\omega\lesssim\begin{cases}c^2/\sqrt{n},&r=cr_{\mathrm{np}},\\c^2/(n\epsilon),&r=cr_{\mathrm{priv}}.\end{cases} α ≲ r , τ ≲ exp ( − 2 r 2 κ 2 ) , ω ≲ { c 2 / n , c 2 / ( n ϵ ) , r = c r np , r = c r priv . The bounds on r n p r_{\mathrm{np}} r np r p r i v r_{\mathrm{priv}} r priv c P c_P c P r 2 log ( n + 1 ) r^2\log(n+1) r 2 log ( n + 1 ) n τ n\tau n τ
Taking r = c 1 r n p r=c_1r_{\mathrm{np}} r = c 1 r np c 1 c_1 c 1
1 2 ( 1 − n ω − n τ ) > 1 3 , \frac{1}{2}\left(1-\sqrt{n}\omega-n\tau\right)>\frac{1}{3}, 2 1 ( 1 − n ω − n τ ) > 3 1 , proving the lower bound term r n p r_{\mathrm{np}} r np
For the private contributions, Proposition 4 depends on α ∧ ( ω + τ ) \alpha\wedge(\omega+\tau) α ∧ ( ω + τ ) r = c 2 r p r i v r=c_2r_{\mathrm{priv}} r = c 2 r priv n ϵ ω n\epsilon\omega n ϵ ω ϵ ≤ 1 \epsilon\leq1 ϵ ≤ 1 n τ n\tau n τ n ϵ ( ω + τ ) n\epsilon(\omega+\tau) n ϵ ( ω + τ ) r = c 3 / ( n ϵ ) r=c_3/(n\epsilon) r = c 3 / ( n ϵ ) α ≲ r \alpha\lesssim r α ≲ r n ϵ α n\epsilon\alpha n ϵ α δ ≤ c δ ϵ \delta\leq c_\delta\epsilon δ ≤ c δ ϵ n δ { α ∧ ( ω + τ ) } n\delta\{\alpha\wedge(\omega+\tau)\} n δ { α ∧ ( ω + τ )} max { r p r i v , ( n ϵ ) − 1 } \max\{r_{\mathrm{priv},(n\epsilon)^{-1}\}} max { r priv , ( n ϵ ) − 1 }
Combining these three lower bounds, we have that
r ∗ ( n , d , s ∗ , ϵ , δ , P , ∞ ) ≳ max { d 1 / 4 n ∧ s ∗ n , d 1 / 4 n 3 / 4 ϵ 1 / 2 ∧ s ∗ n , 1 n ϵ } . r^*(n,d,s^*,\epsilon,\delta,\mathcal{P},\infty)\gtrsim\max\left\{\frac{d^{1/4}}{\sqrt{n}}\wedge\sqrt{\frac{s^*}{n}},\frac{d^{1/4}}{n^{3/4}\epsilon^{1/2}}\wedge\sqrt{\frac{s^*}{n}},\frac{1}{n\epsilon}\right\}. r ∗ ( n , d , s ∗ , ϵ , δ , P , ∞ ) ≳ max { n d 1/4 ∧ n s ∗ , n 3/4 ϵ 1/2 d 1/4 ∧ n s ∗ , n ϵ 1 } . For GLMs with more than two response support points, by Lemma 19, there exist distributions such that we can bound the quantities in Proposition 4 by
α ≲ r , τ ≲ exp ( − κ 2 2 r 2 ) , ω ≲ { c 2 / n , r = c min { r n p , n − 1 / 4 } , c 2 / ( n ϵ ) , r = c min { r p r i v , ( n ϵ ) − 1 / 2 } . \alpha\lesssim r,\qquad\tau\lesssim\exp\left(-\frac{\kappa^2}{2r^2}\right),\qquad\omega\lesssim\begin{cases}c^2/\sqrt{n},&r=c\min\{r_{\mathrm{np}},n^{-1/4}\},\\c^2/(n\epsilon),&r=c\min\{r_{\mathrm{priv}},(n\epsilon)^{-1/2}\}.\end{cases} α ≲ r , τ ≲ exp ( − 2 r 2 κ 2 ) , ω ≲ { c 2 / n , c 2 / ( n ϵ ) , r = c min { r np , n − 1/4 } , r = c min { r priv , ( n ϵ ) − 1/2 } . The same testing arguments therefore yield
r ∗ ( n , d , s ∗ , ϵ , δ , P G L M ( F ) , r 0 ) ≳ max { r n p ∧ n − 1 / 4 , r p r i v ∧ ( n ϵ ) − 1 / 2 , 1 n ϵ } . r^*(n,d,s^*,\epsilon,\delta,\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}),r_0)\gtrsim\max\left\{r_{\mathrm{np}}\wedge n^{-1/4},r_{\mathrm{priv}}\wedge(n\epsilon)^{-1/2},\frac{1}{n\epsilon}\right\}. r ∗ ( n , d , s ∗ , ϵ , δ , P GLM ( F ) , r 0 ) ≳ max { r np ∧ n − 1/4 , r priv ∧ ( n ϵ ) − 1/2 , n ϵ 1 } . We scale r 0 r_0 r 0 r 0 r_0 r 0 n τ n\tau n τ
Finally, if ϵ > 1 \epsilon>1 ϵ > 1 D ≥ 1 D\geq1 D ≥ 1
D 1 / 4 n 3 / 4 ϵ 1 / 2 ≤ D 1 / 4 n , 1 n ϵ ≤ min { D 1 / 4 n , s ∗ n } . \frac{D^{1/4}}{n^{3/4}\epsilon^{1/2}}\leq\frac{D^{1/4}}{\sqrt{n}},\qquad\frac{1}{n\epsilon}\leq\min\left\{\frac{D^{1/4}}{\sqrt{n}},\sqrt{\frac{s^*}{n}}\right\}. n 3/4 ϵ 1/2 D 1/4 ≤ n D 1/4 , n ϵ 1 ≤ min { n D 1/4 , n s ∗ } . Consequently, the non-private lower bound alone gives the claimed rate, with D = d D=d D = d D = d ∧ n D=d\wedge n D = d ∧ n
Technical details of the mean-reduction test
This appendix contains the proof of Lemma 6 and further details of § 4.1 .
Proof of 6
We first establish properties of the conditional response mean. Let s ϑ = 1 s_{\vartheta}=1 s ϑ = 1 (3) with non-decreasing ϑ \vartheta ϑ s ϑ = − 1 s_{\vartheta}=-1 s ϑ = − 1
m ν ( t ) : = s ϑ E β , ν [ tanh ( Y ) ∣ X T β = t ] , (54) m_{\nu}(t):=s_{\vartheta}\mathbb{E}_{\beta,\nu}[\tanh(Y)\mid X^{\mathsf{T}}\beta=t], \tag*{(54)} m ν ( t ) := s ϑ E β , ν [ tanh ( Y ) ∣ X T β = t ] , (54) where ν \nu ν (2) or the nuisance intercept in (3) . The conditional distribution is defined through the model response law at predictor t t t
Lemma 22. Let ( X , Y ) ∼ P (X,Y)\sim P ( X , Y ) ∼ P P ∈ P L i n P\in\mathcal{P}_{\mathrm{Lin}} P ∈ P Lin P ∈ P G L M ( F ) P\in\mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P ∈ P GLM ( F ) F \mathcal{F} F ν \nu ν ∣ m ν ∣ ≤ 1 |m_{\nu}|\leq1 ∣ m ν ∣ ≤ 1 m ν m_{\nu} m ν R \mathbb{R} R t 0 , c m , C m > 0 t_{0},c_{m},C_{m}>0 t 0 , c m , C m > 0
c m ( t − s ) ≤ m ν ( t ) − m ν ( s ) ≤ C m ( t − s ) , − t 0 ≤ s ≤ t ≤ t 0 . c_{m}(t-s)\leq m_{\nu}(t)-m_{\nu}(s)\leq C_{m}(t-s),\qquad-t_{0}\leq s\leq t\leq t_{0}. c m ( t − s ) ≤ m ν ( t ) − m ν ( s ) ≤ C m ( t − s ) , − t 0 ≤ s ≤ t ≤ t 0 . Proof. Linear regression transformation. Here, m ν ( t ) : = E [ tanh ( t + ξ ) ] m_{\nu}(t):=\mathbb{E}[\tanh(t+\xi)] m ν ( t ) := E [ tanh ( t + ξ )] tanh \tanh tanh
m ν ′ ( t ) = E [ sech 2 ( t + ξ ) ] ≤ 1. m_{\nu}'(t)=\mathbb{E}[\operatorname{sech}^{2}(t+\xi)]\leq1. m ν ′ ( t ) = E [ sech 2 ( t + ξ )] ≤ 1. Therefore, for ∣ t ∣ ≤ t 0 |t|\leq t_{0} ∣ t ∣ ≤ t 0
m ν ′ ( t ) ≥ sech 2 ( M ξ + t 0 ) P ( ∣ ξ ∣ ≤ M ξ ) ≥ p ξ sech 2 ( M ξ + t 0 ) = : C ξ > 0. m_{\nu}'(t)\geq\operatorname{sech}^{2}(M_{\xi}+t_{0})\mathbb{P}(|\xi|\leq M_{\xi})
\geq p_{\xi}\operatorname{sech}^{2}(M_{\xi}+t_{0})=:C_{\xi}>0. m ν ′ ( t ) ≥ sech 2 ( M ξ + t 0 ) P ( ∣ ξ ∣ ≤ M ξ ) ≥ p ξ sech 2 ( M ξ + t 0 ) =: C ξ > 0. It follows that m ν m_{\nu} m ν R \mathbb{R} R
GLM transformation. Fix η ∈ Θ \eta\in\Theta η ∈ Θ Y Y Y η \eta η Θ \Theta Θ η \eta η
d d η E [ tanh ( Y ) ] = ∫ tanh ( y ) { y − b ′ ( η ) } exp { η y − b ( η ) } h ( y ) d y = E [ Y tanh ( Y ) ] − b ′ ( η ) E [ tanh ( Y ) ] = E [ Y tanh ( Y ) ] − E [ Y ] E [ tanh ( Y ) ] = Cov ( tanh ( Y ) , Y ) = 1 2 E [ ( tanh ( Y ) − tanh ( Y ′ ) ) ( Y − Y ′ ) ] > 0 , \begin{aligned}
\frac{\mathrm{d}}{\mathrm{d}\eta}\mathbb{E}[\tanh(Y)]
&=\int\tanh(y)\{y-b'(\eta)\}\exp\{\eta y-b(\eta)\}h(y)\,\mathrm{d}y\\
&=\mathbb{E}[Y\tanh(Y)]-b'(\eta)\mathbb{E}[\tanh(Y)]\\
&=\mathbb{E}[Y\tanh(Y)]-\mathbb{E}[Y]\mathbb{E}[\tanh(Y)]\\
&=\operatorname{Cov}(\tanh(Y),Y)\\
&=\frac{1}{2}\mathbb{E}[(\tanh(Y)-\tanh(Y'))(Y-Y')]>0,
\end{aligned} d η d E [ tanh ( Y )] = ∫ tanh ( y ) { y − b ′ ( η )} exp { η y − b ( η )} h ( y ) d y = E [ Y tanh ( Y )] − b ′ ( η ) E [ tanh ( Y )] = E [ Y tanh ( Y )] − E [ Y ] E [ tanh ( Y )] = Cov ( tanh ( Y ) , Y ) = 2 1 E [( tanh ( Y ) − tanh ( Y ′ )) ( Y − Y ′ )] > 0 , where in the final line Y ′ Y' Y ′ Y Y Y ϑ \vartheta ϑ s ϑ s_{\vartheta} s ϑ m γ m_{\gamma} m γ R \mathbb{R} R Γ 2 \Gamma_2 Γ 2 Lemma 16 , we have that
m γ ′ ( t ) = s ϑ ϑ ′ ( γ + t ) Cov γ + t ( tanh ( Y ) , Y ) = ∣ ϑ ′ ( γ + t ) ∣ Cov γ + t ( tanh ( Y ) , Y ) . m'_{\gamma}(t) = s_{\vartheta}\vartheta'(\gamma+t)\operatorname{Cov}_{\gamma+t}(\tanh(Y),Y)
= |\vartheta'(\gamma+t)|\operatorname{Cov}_{\gamma+t}(\tanh(Y),Y). m γ ′ ( t ) = s ϑ ϑ ′ ( γ + t ) Cov γ + t ( tanh ( Y ) , Y ) = ∣ ϑ ′ ( γ + t ) ∣ Cov γ + t ( tanh ( Y ) , Y ) . Since both factors are continuous and strictly positive, compactness of Γ 2 \Gamma_2 Γ 2 c m , C m > 0 c_m,C_m>0 c m , C m > 0
c m ≤ m γ ′ ( t ) ≤ C m ∀ ( γ , t ) with γ + t ∈ Γ 2 . (55) c_m \le m'_{\gamma}(t) \le C_m \qquad\forall(\gamma,t)\text{ with }\gamma+t\in\Gamma_2. \tag*{(55)} c m ≤ m γ ′ ( t ) ≤ C m ∀ ( γ , t ) with γ + t ∈ Γ 2 . (55) Choose t 0 > 0 t_0>0 t 0 > 0 Γ + [ − t 0 , t 0 ] ⊆ Γ 2 \Gamma+[-t_0,t_0]\subseteq\Gamma_2 Γ + [ − t 0 , t 0 ] ⊆ Γ 2 t 0 , c m , C m > 0 t_0,c_m,C_m>0 t 0 , c m , C m > 0 ∣ m ν ∣ ≤ 1 |m_{\nu}|\le1 ∣ m ν ∣ ≤ 1 m ν m_{\nu} m ν R \mathbb{R} R
c m ( t − s ) ≤ m ν ( t ) − m ν ( s ) ≤ C m ( t − s ) , − t 0 ≤ s ≤ t ≤ t 0 , c_m(t-s)\le m_{\nu}(t)-m_{\nu}(s)\le C_m(t-s),\qquad-t_0\le s\le t\le t_0, c m ( t − s ) ≤ m ν ( t ) − m ν ( s ) ≤ C m ( t − s ) , − t 0 ≤ s ≤ t ≤ t 0 , uniformly over the allowed nuisance parameters.
We can now proceed to prove Lemma 6 .
Proof of Lemma 6 . Since ∣ tanh ( Y ) ∣ ≤ 1 |\tanh(Y)|\le1 ∣ tanh ( Y ) ∣ ≤ 1 Z Z ⊤ ⪯ X X ⊤ ZZ^{\top}\preceq XX^{\top} Z Z ⊤ ⪯ X X ⊤ E [ Z Z ⊤ ] ⪯ Σ : = E [ X X ⊤ ] \mathbb{E}[ZZ^{\top}]\preceq\Sigma:=\mathbb{E}[XX^{\top}] E [ Z Z ⊤ ] ⪯ Σ := E [ X X ⊤ ] u u u
∣ u ⊤ E [ Z ] ∣ ≤ { E [ ( u ⊤ Z ) 2 ] } 1 / 2 ≤ u ⊤ Σ u ≤ λ . |u^{\top}\mathbb{E}[Z]|\le\{\mathbb{E}[(u^{\top}Z)^2]\}^{1/2}\le\sqrt{u^{\top}\Sigma u}\le\sqrt{\lambda}. ∣ u ⊤ E [ Z ] ∣ ≤ { E [( u ⊤ Z ) 2 ] } 1/2 ≤ u ⊤ Σ u ≤ λ . Taking the supremum over u u u ∥ E [ Z ] ∥ 2 ≤ λ \|\mathbb{E}[Z]\|_2\le\sqrt{\lambda} ∥ E [ Z ] ∥ 2 ≤ λ ∣ tanh ( Y ) ∣ ≤ 1 |\tanh(Y)|\le1 ∣ tanh ( Y ) ∣ ≤ 1 [43 ] .
Under the null, independence of X X X Y Y Y
E [ Z ] = E [ X E [ tanh ( Y ) ∣ X ] ] = s ϑ m ν ( 0 ) E [ X ] = 0. \mathbb{E}[Z]=\mathbb{E}[X\mathbb{E}[\tanh(Y)\mid X]]=s_{\vartheta}m_{\nu}(0)\mathbb{E}[X]=0. E [ Z ] = E [ X E [ tanh ( Y ) ∣ X ]] = s ϑ m ν ( 0 ) E [ X ] = 0. To show (9) , let u : = β / ∥ β ∥ 2 u:=\beta/\|\beta\|_2 u := β /∥ β ∥ 2 E [ u ⊤ X ] = 0 \mathbb{E}[u^{\top}X]=0 E [ u ⊤ X ] = 0
s ϑ u ⊤ E [ Z ] = E [ u ⊤ X { m ν ( ∥ β ∥ 2 u ⊤ X ) − m ν ( 0 ) } ] ≥ c m ∥ β ∥ 2 E [ ( u ⊤ X ) 2 1 { ∥ β ∥ 2 ∣ u ⊤ X ∣ ≤ t 0 } ] ≥ c m ∥ β ∥ 2 { u ⊤ Σ u − ∥ β ∥ 2 2 t 0 2 E [ ( u ⊤ X ) 4 ] } ≥ c m ∥ β ∥ 2 { λ − C ∥ β ∥ 2 2 C X 4 t 0 2 } . (56) \begin{aligned}
s_{\vartheta}u^{\top}\mathbb{E}[Z]=\mathbb{E}\left[u^{\top}X\{m_{\nu}(\|\beta\|_2u^{\top}X)-m_{\nu}(0)\}\right] \\
&\ge c_m\|\beta\|_2\mathbb{E}\left[(u^{\top}X)^2\mathbf{1}\{\|\beta\|_2|u^{\top}X|\le t_0\}\right] \\
&\ge c_m\|\beta\|_2\left\{u^{\top}\Sigma u-\frac{\|\beta\|_2^2}{t_0^2}\mathbb{E}[(u^{\top}X)^4]\right\} \\
&\ge c_m\|\beta\|_2\left\{\lambda-\frac{C\|\beta\|_2^2C_X^4}{t_0^2}\right\}. \tag*{(56)}
\end{aligned} s ϑ u ⊤ E [ Z ] = E [ u ⊤ X { m ν ( ∥ β ∥ 2 u ⊤ X ) − m ν ( 0 )} ] ≥ c m ∥ β ∥ 2 E [ ( u ⊤ X ) 2 1 { ∥ β ∥ 2 ∣ u ⊤ X ∣ ≤ t 0 } ] ≥ c m ∥ β ∥ 2 { u ⊤ Σ u − t 0 2 ∥ β ∥ 2 2 E [( u ⊤ X ) 4 ] } ≥ c m ∥ β ∥ 2 { λ − t 0 2 C ∥ β ∥ 2 2 C X 4 } . (56) The first inequality is due to Lemma 22 and the final inequality follows from the sub-Gaussian assumption on the covariates. Thus, for a sufficiently small absolute constant c 0 > 0 c_0>0 c 0 > 0 s ϑ u ⊤ E [ Z ] ≥ c m λ / 2 ∥ β ∥ 2 s_{\vartheta}u^{\top}\mathbb{E}[Z]\ge c_m\lambda/2\|\beta\|_2 s ϑ u ⊤ E [ Z ] ≥ c m λ /2∥ β ∥ 2 ∥ β ∥ 2 ≤ c 0 t 0 λ / C X 2 \|\beta\|_2\le c_0t_0\sqrt{\lambda}/C_X^2 ∥ β ∥ 2 ≤ c 0 t 0 λ / C X 2 m ν m_\nu m ν t ↦ u ⊤ x { m ν ( t u ⊤ x ) − m ν ( 0 ) } t \mapsto u^\top x\{m_\nu(tu^\top x)-m_\nu(0)\} t ↦ u ⊤ x { m ν ( t u ⊤ x ) − m ν ( 0 )} x x x ∥ β ∥ 2 > c 0 t 0 λ / C X 2 \|\beta\|_2 > c_0t_0\sqrt{\lambda}/C_X^2 ∥ β ∥ 2 > c 0 t 0 λ / C X 2
∥ E [ Z ] ∥ 2 ≥ s ϑ u ⊤ E [ Z ] ≥ c min { ∥ β ∥ 2 , t 0 λ C X 2 } ≥ c μ min { ∥ β ∥ 2 , 1 } . \|\mathbb{E}[Z]\|_2 \ge s_\vartheta u^\top\mathbb{E}[Z] \ge c\min\left\{\|\beta\|_2,\frac{t_0\sqrt{\lambda}}{C_X^2}\right\} \ge c_\mu\min\{\|\beta\|_2,1\}. ∥ E [ Z ] ∥ 2 ≥ s ϑ u ⊤ E [ Z ] ≥ c min { ∥ β ∥ 2 , C X 2 t 0 λ } ≥ c μ min { ∥ β ∥ 2 , 1 } . where c μ > 0 c_\mu> 0 c μ > 0
Proof of Lemma 7
Proof (adapted from Lemma 44 of Narayanan, [36 ] ). Let S ⊆ [ n ] S \subseteq[n] S ⊆ [ n ]
Q [ n ] ( z 1 : n ) = Q S c ( z 1 : n ) + 2 ∑ i ∈ S , j ∈ S c z i ⊤ z j + Q S ( z 1 : n ) = Q S c ( z 1 : n ) + 2 ∑ i ∈ S u i ( z 1 : n ) − Q S ( z 1 : n ) . \begin{aligned}
Q_{[n]}(z_{1:n})=Q_{S^c}(z_{1:n})+2\sum_{i\in S,j\in S^c}z_i^\top z_j+Q_S(z_{1:n}) \\
&=Q_{S^c}(z_{1:n})+2\sum_{i\in S}u_i(z_{1:n})-Q_S(z_{1:n}).
\end{aligned} Q [ n ] ( z 1 : n ) = Q S c ( z 1 : n ) + 2 i ∈ S , j ∈ S c ∑ z i ⊤ z j + Q S ( z 1 : n ) = Q S c ( z 1 : n ) + 2 i ∈ S ∑ u i ( z 1 : n ) − Q S ( z 1 : n ) . This can be used to bound
∣ Q [ n ] ( z 1 : n ) − Q [ n ] ( z 1 : n ′ ) ∣ ≤ 2 ∑ i ∈ S { ∣ u i ( z 1 : n ) ∣ + ∣ u i ( z 1 : n ′ ) ∣ } + ∣ Q S ( z 1 : n ) ∣ + ∣ Q S ( z 1 : n ′ ) ∣ \left|Q_{[n]}(z_{1:n})-Q_{[n]}(z'_{1:n})\right|
\le2\sum_{i\in S}\{|u_i(z_{1:n})|+|u_i(z'_{1:n})|\}+|Q_S(z_{1:n})|+|Q_S(z'_{1:n})| Q [ n ] ( z 1 : n ) − Q [ n ] ( z 1 : n ′ ) ≤ 2 i ∈ S ∑ { ∣ u i ( z 1 : n ) ∣ + ∣ u i ( z 1 : n ′ ) ∣ } + ∣ Q S ( z 1 : n ) ∣ + ∣ Q S ( z 1 : n ′ ) ∣ ≤ 4 K L { n d + n s } + 2 L { K K d + K 2 } \le4KL\{\sqrt{nd}+ns\}+2L\{K\sqrt{Kd}+K^2\} ≤ 4 K L { n d + n s } + 2 L { K K d + K 2 } ≤ 6 K L { n d + n s + K } , \le6KL\{\sqrt{nd}+ns+K\}, ≤ 6 K L { n d + n s + K } , where K ≤ n K\le n K ≤ n
Proof of typical set probability and concentration
Lemma 23. Let ( X 1 , Y 1 ) , … , ( X n , Y n ) (X_1,Y_1),\ldots,(X_n,Y_n) ( X 1 , Y 1 ) , … , ( X n , Y n ) P P P P L i n P_{\mathrm{Lin}} P Lin P G L M ( F ) P_{\mathrm{GLM}}(\mathcal{F}) P GLM ( F ) F \mathcal{F} F 0 < η ≤ 1 / 4 0<\eta\le1/4 0 < η ≤ 1/4 E [ Z ] = 0 \mathbb{E}[Z]=0 E [ Z ] = 0 s ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s s\le\|\mathbb{E}[Z]\|_2\le2s s ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s 0 < s ≤ λ 0<s\le\sqrt{\lambda} 0 < s ≤ λ C C C
L = C { 1 + log ( 12 ( n + 1 ) η ) } , (57) L=C\left\{1+\log\left(\frac{12(n+1)}{\eta}\right)\right\}, \tag*{(57)} L = C { 1 + log ( η 12 ( n + 1 ) ) } , (57) we have, with probability at least 1 − η 1-\eta 1 − η ( Z 1 , … , Z n ) ∈ D n , s , L (Z_1,\ldots,Z_n)\in D_{n,s,L} ( Z 1 , … , Z n ) ∈ D n , s , L
∣ Q [ n ] ( Z ) − n ( n − 1 ) ∥ E [ Z ] ∥ 2 2 ∣ ≤ L { n d + n 3 / 2 s } . \left|Q_{[n]}(Z)-n(n-1)\|\mathbb{E}[Z]\|_2^2\right|\le L\{\sqrt{n}d+n^{3/2}s\}. Q [ n ] ( Z ) − n ( n − 1 ) ∥ E [ Z ] ∥ 2 2 ≤ L { n d + n 3/2 s } . Proof. Denoting R i : = Z i − E [ Z ] R_i:=Z_i-\mathbb{E}[Z] R i := Z i − E [ Z ]
u i = ( n − 1 ) ∥ E [ Z ] ∥ 2 2 + ( n − 1 ) ⟨ R i , E [ Z ] ⟩ + E [ Z ] ⊤ ∑ j ≠ i R j + R i ⊤ ∑ j ≠ i R j . u_i=(n-1)\|\mathbb{E}[Z]\|_2^2+(n-1)\langle R_i,\mathbb{E}[Z]\rangle+\mathbb{E}[Z]^\top\sum_{j\ne i}R_j+R_i^\top\sum_{j\ne i}R_j. u i = ( n − 1 ) ∥ E [ Z ] ∥ 2 2 + ( n − 1 ) ⟨ R i , E [ Z ]⟩ + E [ Z ] ⊤ j = i ∑ R j + R i ⊤ j = i ∑ R j . Since R i R_i R i ∑ j ≠ i R j \sum_{j\ne i}R_j ∑ j = i R j ∥ R i ∥ ψ 2 ≤ c \|R_i\|_{\psi_2}\le c ∥ R i ∥ ψ 2 ≤ c ∥ ∑ j ≠ i R j ∥ ψ 2 ≤ c n \left\|\sum_{j\ne i}R_j\right\|_{\psi_2}\le c\sqrt{n} ∑ j = i R j ψ 2 ≤ c n Lemma 37 that
P ( ∣ u i − ( n − 1 ) ∥ E [ Z ] ∥ 2 2 ∣ > C { ∥ E Z ∥ 2 n t + ∥ E Z ∥ 2 n t + n ( d t + t ) } ) ≤ 3 e − t . \mathbb{P}\left(\left|u_i-(n-1)\|\mathbb{E}[Z]\|_2^2\right|>C\left\{\|\mathbb{E}Z\|_2 n\sqrt{t}+\|\mathbb{E}Z\|_2\sqrt{nt}+\sqrt{n}(\sqrt{dt}+t)\right\}\right)\leq3e^{-t}. P ( u i − ( n − 1 ) ∥ E [ Z ] ∥ 2 2 > C { ∥ E Z ∥ 2 n t + ∥ E Z ∥ 2 n t + n ( d t + t ) } ) ≤ 3 e − t . Taking t = log ( 12 n / η ) t=\log(12n/\eta) t = log ( 12 n / η ) i ∈ [ n ] i\in[n] i ∈ [ n ] (10) holds for every i ∈ [ n ] i\in[n] i ∈ [ n ] 1 − η / 4 1-\eta/4 1 − η /4
We next consider the probability of the event in (11) . For any fixed set S S S ∣ S ∣ = k |S|=k ∣ S ∣ = k
Q S = k ( k − 1 ) ∥ E [ Z ] ∥ 2 2 + 2 ( k − 1 ) E [ Z ] T ∑ i ∈ S R i + ∑ i , j ∈ S , i ≠ j R i T R j (58) Q_S=k(k-1)\|\mathbb{E}[Z]\|_2^2+2(k-1)\mathbb{E}[Z]^\mathsf{T}\sum_{i\in S}R_i+\sum_{i,j\in S,\ i\ne j}R_i^\mathsf{T}R_j \tag*{(58)} Q S = k ( k − 1 ) ∥ E [ Z ] ∥ 2 2 + 2 ( k − 1 ) E [ Z ] T i ∈ S ∑ R i + i , j ∈ S , i = j ∑ R i T R j (58) We introduce a sequence of Bernoulli random variables ζ : = { ζ i : i ∈ S } \zeta:=\{\zeta_i:i\in S\} ζ := { ζ i : i ∈ S } ζ i ∼ i . i . d . Bernoulli ( 1 / 2 ) \zeta_i\mathrel{\overset{\mathrm{i.i.d.}}{\sim}}\operatorname{Bernoulli}(1/2) ζ i ∼ i.i.d. Bernoulli ( 1/2 ) (58) . Let
C ζ : = A ζ T B ζ : = ⟨ ∑ i ∈ S ζ i R i , ∑ j ∈ S ( 1 − ζ j ) R j ⟩ ; C_\zeta:=A_\zeta^\mathsf{T}B_\zeta:=\left\langle\sum_{i\in S}\zeta_iR_i,\sum_{j\in S}(1-\zeta_j)R_j\right\rangle; C ζ := A ζ T B ζ := ⟨ i ∈ S ∑ ζ i R i , j ∈ S ∑ ( 1 − ζ j ) R j ⟩ ; note that we have
D : = ∑ i , j ∈ S , i ≠ j R i T R j = 4 E ζ [ C ζ ∣ { R i : i ∈ S } ] . D:=\sum_{i,j\in S,\ i\ne j}R_i^\mathsf{T}R_j=4\mathbb{E}_\zeta[C_\zeta\mid\{R_i:i\in S\}]. D := i , j ∈ S , i = j ∑ R i T R j = 4 E ζ [ C ζ ∣ { R i : i ∈ S }] . Fix a realisation of ζ \zeta ζ A ζ A_\zeta A ζ B ζ B_\zeta B ζ ζ \zeta ζ [43 ] gives
∥ A ζ ∥ ψ 2 ≤ C ( ∑ i ∈ S : ζ i = 1 ∥ R i ∥ ψ 2 2 ) 1 / 2 ≤ C C X k , \|A_\zeta\|_{\psi_2}\leq C\left(\sum_{i\in S:\zeta_i=1}\|R_i\|_{\psi_2}^2\right)^{1/2}\leq CC_X\sqrt{k}, ∥ A ζ ∥ ψ 2 ≤ C i ∈ S : ζ i = 1 ∑ ∥ R i ∥ ψ 2 2 1/2 ≤ C C X k , and similarly ∥ B ζ ∥ ψ 2 ≤ C C X k \|B_\zeta\|_{\psi_2}\leq CC_X\sqrt{k} ∥ B ζ ∥ ψ 2 ≤ C C X k Lemma 37 therefore implies, for every p ≥ 1 p\geq1 p ≥ 1
∥ C ζ ∥ L p ( R ∣ ζ ) ≤ C 1 k ( d p + p ) , \|C_\zeta\|_{L^p(R\mid\zeta)}\leq C_1k(\sqrt{dp}+p), ∥ C ζ ∥ L p ( R ∣ ζ ) ≤ C 1 k ( d p + p ) , uniformly over ζ \zeta ζ
E R [ ∣ D ∣ p ] = 4 p E R ∣ E ζ [ C ζ ∣ { R i : i ∈ S } ] ∣ p ≤ 4 p E R E ζ [ ∣ C ζ ∣ p ∣ { R i : i ∈ S } ] = 4 p E ζ E R ∣ C ζ ∣ p ≤ { C 1 k ( d p + p ) } p . \begin{aligned}
\mathbb{E}_R[|D|^p]
&=4^p\mathbb{E}_R\left|\mathbb{E}_\zeta[C_\zeta\mid\{R_i:i\in S\}]\right|^p\\
&\leq4^p\mathbb{E}_R\mathbb{E}_\zeta\left[|C_\zeta|^p\mid\{R_i:i\in S\}\right]\\
&=4^p\mathbb{E}_\zeta\mathbb{E}_R|C_\zeta|^p\\
&\leq\left\{C_1k(\sqrt{dp}+p)\right\}^p.
\end{aligned} E R [ ∣ D ∣ p ] = 4 p E R ∣ E ζ [ C ζ ∣ { R i : i ∈ S }] ∣ p ≤ 4 p E R E ζ [ ∣ C ζ ∣ p ∣ { R i : i ∈ S } ] = 4 p E ζ E R ∣ C ζ ∣ p ≤ { C 1 k ( d p + p ) } p . Applying Markov’s inequality for t > 1 t>1 t > 1 ∥ D ∥ L t ( R ) ≤ C 1 k ( d t + t ) \|D\|_{L^t(R)}\leq C_1k(\sqrt{dt}+t) ∥ D ∥ L t ( R ) ≤ C 1 k ( d t + t )
P ( ∣ ∑ i , j ∈ S : i ≠ j R i T R j ∣ > C 2 k { d t + t } ) ≤ e − t . (59) \mathbb{P}\left(\left|\sum_{i,j\in S:i\ne j}R_i^\mathsf{T}R_j\right|>C_2k\{\sqrt{dt}+t\}\right)\leq e^{-t}. \tag*{(59)} P i , j ∈ S : i = j ∑ R i T R j > C 2 k { d t + t } ≤ e − t . (59) The second term in (58) has sub-Gaussian norm at most C ∥ E [ Z ] ∥ 2 k 3 / 2 C\|\mathbb{E}[Z]\|_2k^{3/2} C ∥ E [ Z ] ∥ 2 k 3/2 ∥ E [ Z ] ∥ 2 ≤ 2 s \|\mathbb{E}[Z]\|_2\leq2s ∥ E [ Z ] ∥ 2 ≤ 2 s (59) gives
P ( ∣ Q S − k ( k − 1 ) ∥ E [ Z ] ∥ 2 2 ∣ > C { k d t + k t + s k 3 / 2 t } ) ≤ 3 e − t , t > 1. (60) \mathbb{P}\left(\left|Q_S-k(k-1)\|\mathbb{E}[Z]\|_2^2\right|>C\{k\sqrt{d t}+kt+sk^{3/2}\sqrt{t}\}\right)\leq3e^{-t},\qquad t>1. \tag*{(60)} P ( Q S − k ( k − 1 ) ∥ E [ Z ] ∥ 2 2 > C { k d t + k t + s k 3/2 t } ) ≤ 3 e − t , t > 1. (60) Adding the deterministic term k ( k − 1 ) ∥ E [ Z ] ∥ 2 2 ≤ 4 k 2 s 2 k(k-1)\|\mathbb{E}[Z]\|_2^2\leq4k^2s^2 k ( k − 1 ) ∥ E [ Z ] ∥ 2 2 ≤ 4 k 2 s 2
P ( ∣ Q S ∣ ≤ c 3 { k d t + k t + s k 3 / 2 t + k 2 s 2 } ) ≥ 1 − 3 e − t (61) \mathbb{P}\left(\left|Q_S\right|\leq c_3\{k\sqrt{d t}+kt+sk^{3/2}\sqrt{t}+k^2s^2\}\right)\geq1-3e^{-t} \tag*{(61)} P ( ∣ Q S ∣ ≤ c 3 { k d t + k t + s k 3/2 t + k 2 s 2 } ) ≥ 1 − 3 e − t (61) for all t ≥ 1 t\geq1 t ≥ 1
Now for each k k k ( n k ) ≤ ( e n / k ) k \binom{n}{k}\leq(en/k)^k ( k n ) ≤ ( e n / k ) k [ n ] [n] [ n ] k k k t k : = k log ( e n / k ) + log ( 12 n / η ) t_k:=k\log(en/k)+\log(12n/\eta) t k := k log ( e n / k ) + log ( 12 n / η ) t k ≲ k { 1 + log ( 12 ( n + 1 ) / η ) } t_k\lesssim k\{1+\log(12(n+1)/\eta)\} t k ≲ k { 1 + log ( 12 ( n + 1 ) / η )} s ≤ λ s\leq\sqrt{\lambda} s ≤ λ (61) is therefore at most L { k k d + k 2 } L\{k\sqrt{kd}+k^2\} L { k k d + k 2 } C C C
P ( A c ) ≤ 3 ∑ k = 1 n ( e n k ) k e − t k ≤ η 4 , \mathbb{P}(\mathcal{A}^{\mathrm{c}})\leq3\sum_{k=1}^{n}\binom{en}{k}^{k}e^{-t_k}\leq\frac{\eta}{4}, P ( A c ) ≤ 3 k = 1 ∑ n ( k e n ) k e − t k ≤ 4 η , where A \mathcal{A} A (11) holds for every S ⊆ [ n ] S\subseteq[n] S ⊆ [ n ]
Finally, applying (60) with S = [ n ] S=[n] S = [ n ] t = log ( 12 / η ) t=\log(12/\eta) t = log ( 12/ η )
∣ Q [ n ] − n ( n − 1 ) ∥ E [ Z ] ∥ 2 2 ∣ ≤ C { n d t + n t + s n 3 / 2 t } ≤ L { n d + n 3 / 2 s } \left|Q_{[n]}-n(n-1)\|\mathbb{E}[Z]\|_2^2\right|\leq C\{n\sqrt{d t}+nt+sn^{3/2}\sqrt{t}\}\leq L\{n\sqrt{d}+n^{3/2}s\} Q [ n ] − n ( n − 1 ) ∥ E [ Z ] ∥ 2 2 ≤ C { n d t + n t + s n 3/2 t } ≤ L { n d + n 3/2 s } except on an event of probability at most η / 4 \eta/4 η /4
Proof of Lemma 8
Proof. Privacy. For any z , z ′ z,z' z , z ′
G ^ ( z ) = inf z ~ ∈ D n , s , L { G ( z ~ ) + D D H a m ( z , z ~ ) } ≤ inf z ~ ∈ D n , s , L { G ( z ~ ) + D D H a m ( z , z ′ ) + D D H a m ( z ′ , z ~ ) } = G ^ ( z ′ ) + D D H a m ( z , z ′ ) . \begin{aligned}
\widehat{G}(z)&=\inf_{\widetilde{z}\in\mathcal{D}_{n,s,L}}\{G(\widetilde{z})+D D_{\mathrm{Ham}}(z,\widetilde{z})\}\\
&\leq\inf_{\widetilde{z}\in\mathcal{D}_{n,s,L}}\{G(\widetilde{z})+D D_{\mathrm{Ham}}(z,z')+D D_{\mathrm{Ham}}(z',\widetilde{z})\}\\
&=\widehat{G}(z')+D D_{\mathrm{Ham}}(z,z').
\end{aligned} G ( z ) = z ∈ D n , s , L inf { G ( z ) + D D Ham ( z , z )} ≤ z ∈ D n , s , L inf { G ( z ) + D D Ham ( z , z ′ ) + D D Ham ( z ′ , z )} = G ( z ′ ) + D D Ham ( z , z ′ ) . Interchanging the roles of z z z z ′ z' z ′ ∣ G ^ ( z ) − G ^ ( z ′ ) ∣ ≤ D D H a m ( z , z ′ ) |\widehat{G}(z)-\widehat{G}(z')|\leq D D_{\mathrm{Ham}}(z,z') ∣ G ( z ) − G ( z ′ ) ∣ ≤ D D Ham ( z , z ′ )
Error guarantee. We will first show that G G G D D D D n , s , L \mathcal{D}_{n,s,L} D n , s , L
Note that the definition of D D D s s s (15) , implies we have
6 L ( n d + s n + 8 log ( 2 / η ) / ϵ 0 ) ≤ D . (62) 6L(\sqrt{nd}+sn+8\log(2/\eta)/\epsilon_0)\leq D. \tag*{(62)} 6 L ( n d + s n + 8 log ( 2/ η ) / ϵ 0 ) ≤ D . (62) Suppose two datasets z 1 : n , z 1 : n ′ ∈ D n , s , L z_{1:n},z'_{1:n}\in\mathcal{D}_{n,s,L} z 1 : n , z 1 : n ′ ∈ D n , s , L K K K
∣ Q ( z 1 : n ) − Q ( z 1 : n ′ ) ∣ ≤ 6 K L ( n d + s n + K ) . |Q(z_{1:n})-Q(z'_{1:n})|\leq6KL(\sqrt{nd}+sn+K). ∣ Q ( z 1 : n ) − Q ( z 1 : n ′ ) ∣ ≤ 6 K L ( n d + s n + K ) . First suppose K ≤ ( 8 log ( 2 / η ) ) / ϵ 0 K \le(8\log(2/\eta))/\epsilon_0 K ≤ ( 8 log ( 2/ η )) / ϵ 0 (62) that
∣ G ( z 1 : n ) − G ( z 1 : n ′ ) ∣ ≤ ∣ Q ( z 1 : n ) − Q ( z 1 : n ′ ) ∣ ≤ K D . \lvert G(z_{1:n})-G(z'_{1:n})\rvert\le\lvert Q(z_{1:n})-Q(z'_{1:n})\rvert\le KD. ∣ G ( z 1 : n ) − G ( z 1 : n ′ )∣ ≤ ∣ Q ( z 1 : n ) − Q ( z 1 : n ′ )∣ ≤ K D . Otherwise, if K > ( 8 log ( 2 / η ) ) / ϵ 0 K > (8\log(2/\eta))/\epsilon_0 K > ( 8 log ( 2/ η )) / ϵ 0
∣ G ( z 1 : n ) − G ( z 1 : n ′ ) ∣ ≤ B n ≤ K D . \lvert G(z_{1:n})-G(z'_{1:n})\rvert\le B_n \le KD. ∣ G ( z 1 : n ) − G ( z 1 : n ′ )∣ ≤ B n ≤ K D . Combining both cases, we have that ∣ G ( z 1 : n ) − G ( z 1 : n ′ ) ∣ ≤ D D H a m ( z 1 : n , z 1 : n ′ ) \lvert G(z_{1:n})-G(z'_{1:n})\rvert\le D D_{\mathrm{Ham}}(z_{1:n},z'_{1:n}) ∣ G ( z 1 : n ) − G ( z 1 : n ′ )∣ ≤ D D Ham ( z 1 : n , z 1 : n ′ ) z 1 : n , z 1 : n ′ ∈ D n , s , L z_{1:n},z'_{1:n} \in\mathcal{D}_{n,s,L} z 1 : n , z 1 : n ′ ∈ D n , s , L
Since D n , s , L \mathcal{D}_{n,s,L} D n , s , L Lemma 38 ), G ^ \widehat{G} G (13) satisfies G ^ ( z 1 : n ) = G ( z 1 : n ) \widehat{G}(z_{1:n})=G(z_{1:n}) G ( z 1 : n ) = G ( z 1 : n ) z 1 : n ∈ D n , s , L z_{1:n}\in\mathcal{D}_{n,s,L} z 1 : n ∈ D n , s , L D D D
By Lemma 23 , under either hypothesis, we have that Z ∈ D n , s , L Z\in\mathcal{D}_{n,s,L} Z ∈ D n , s , L 1 − η / 2 1-\eta/2 1 − η /2
∣ Q ( Z 1 : n ) − n ( n − 1 ) ∥ E [ Z ] ∥ 2 2 ∣ ≤ L ( n d + n 3 / 2 s ) . \lvert Q(Z_{1:n})-n(n-1)\lVert\mathbb{E}[Z]\rVert_2^2\rvert\le L\left(n\sqrt{d}+n^{3/2}s\right). ∣ Q ( Z 1 : n ) − n ( n − 1 ) ∥ E [ Z ] ∥ 2 2 ∣ ≤ L ( n d + n 3/2 s ) . Therefore, on this typical event under H 0 H_0 H 0 G ( Z 1 : n ) ≤ B n / 4 G(Z_{1:n})\le B_n/4 G ( Z 1 : n ) ≤ B n /4 H 1 ( s ) H_1(s) H 1 ( s ) G ( Z 1 : n ) ≥ 3 B n / 4 G(Z_{1:n})\ge3B_n/4 G ( Z 1 : n ) ≥ 3 B n /4 s s s (15) . In addition, we have that
P ( ∣ W ∣ ≥ B n / 4 ) = exp { − ϵ 0 B n / ( 4 D ) } = ( η / 2 ) 2 ≤ η / 2 , \mathbb{P}(\lvert W\rvert\ge B_n/4)=\exp\{-\epsilon_0B_n/(4D)\}=(\eta/2)^2\le\eta/2, P (∣ W ∣ ≥ B n /4 ) = exp { − ϵ 0 B n / ( 4 D )} = ( η /2 ) 2 ≤ η /2 , so a union bound proves the error guarantee.
Multiscale test
Lemma 24 (Multiscale test). Let P \mathcal{P} P E [ Z ] ( P ) ∈ R d \mathbb{E}[Z](P)\in\mathbb{R}^d E [ Z ] ( P ) ∈ R d ∥ E [ Z ] ( P ) ∥ 2 ≤ B \lVert\mathbb{E}[Z](P)\rVert_2\le B ∥ E [ Z ] ( P ) ∥ 2 ≤ B P ∈ P P\in\mathcal{P} P ∈ P B > 0 B>0 B > 0 s ∈ ( 0 , B ] s\in(0,B] s ∈ ( 0 , B ] 0 < ϵ 0 0<\epsilon_0 0 < ϵ 0 0 < η 0 ≤ 1 / 4 0<\eta_0\le1/4 0 < η 0 ≤ 1/4 ( ϵ 0 , 0 ) (\epsilon_0,0) ( ϵ 0 , 0 ) ϕ s , ϵ 0 , η 0 \phi_{s,\epsilon_0,\eta_0} ϕ s , ϵ 0 , η 0
H 0 : E [ Z ] ( P ) = 0 against H 1 ( s ) : s ≤ ∥ E [ Z ] ( P ) ∥ 2 ≤ 2 s H_0:\mathbb{E}[Z](P)=0 \qquad\text{against}\qquad H_1(s):s\le\lVert\mathbb{E}[Z](P)\rVert_2\le2s H 0 : E [ Z ] ( P ) = 0 against H 1 ( s ) : s ≤ ∥ E [ Z ] ( P ) ∥ 2 ≤ 2 s such that, whenever s ≥ ρ ( n , d , ϵ 0 , η 0 ) s\ge\rho(n,d,\epsilon_0,\eta_0) s ≥ ρ ( n , d , ϵ 0 , η 0 )
sup P ∈ P E [ Z ] ( P ) = 0 P P ( ϕ s , ϵ 0 , η 0 = 1 ) ≤ η 0 , sup P ∈ P s ≤ ∥ E [ Z ] ( P ) ∥ 2 ≤ 2 s P P ( ϕ s , ϵ 0 , η 0 = 0 ) ≤ η 0 . \sup_{\substack{P\in\mathcal{P}\\ \mathbb{E}[Z](P)=0}}\mathbb{P}_P(\phi_{s,\epsilon_0,\eta_0}=1)\le\eta_0,\qquad
\sup_{\substack{P\in\mathcal{P}\\ s\le\lVert\mathbb{E}[Z](P)\rVert_2\le2s}}\mathbb{P}_P(\phi_{s,\epsilon_0,\eta_0}=0)\le\eta_0. P ∈ P E [ Z ] ( P ) = 0 sup P P ( ϕ s , ϵ 0 , η 0 = 1 ) ≤ η 0 , P ∈ P s ≤ ∥ E [ Z ] ( P ) ∥ 2 ≤ 2 s sup P P ( ϕ s , ϵ 0 , η 0 = 0 ) ≤ η 0 . Fix 0 < r ≤ B 0<r\le B 0 < r ≤ B ϵ > 0 \epsilon>0 ϵ > 0 0 < η ≤ 1 / 4 0<\eta\le1/4 0 < η ≤ 1/4
r ≥ ρ ( n , d , ϵ J , η J ) , J : = max { 1 , ⌈ log 2 ( B r ) ⌉ } , (63) r\ge\rho\left(n,d,\frac{\epsilon}{J},\frac{\eta}{J}\right),\qquad J:=\max\left\{1,\left\lceil\log_2\left(\frac{B}{r}\right)\right\rceil\right\}, \tag*{(63)} r ≥ ρ ( n , d , J ϵ , J η ) , J := max { 1 , ⌈ log 2 ( r B ) ⌉ } , (63) there exists an ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) Φ r \Phi_r Φ r
H 0 : E [ Z ] = 0 against H 1 : ∥ E [ Z ] ∥ 2 ≥ r H_0:\mathbb{E}[Z]=0 \qquad\text{against}\qquad H_1:\lVert\mathbb{E}[Z]\rVert_2\ge r H 0 : E [ Z ] = 0 against H 1 : ∥ E [ Z ] ∥ 2 ≥ r whose type-I and type-II errors are each at most η \eta η
Proof. Set s j : = 2 j r s_j:=2^jr s j := 2 j r ϵ j : = ϵ / J \epsilon_j:=\epsilon/J ϵ j := ϵ / J η j : = η / J \eta_j:=\eta/J η j := η / J j = 0 , … , J − 1 j=0,\ldots,J-1 j = 0 , … , J − 1 ϕ s j , ϵ j , η j \phi_{s_j,\epsilon_j,\eta_j} ϕ s j , ϵ j , η j Lemma 39 ), the test is ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) ∑ j η j = η \sum_j \eta_j=\eta ∑ j η j = η ∥ E [ Z ] ∥ 2 ≥ r \|\mathbb{E}[Z]\|_2 \ge r ∥ E [ Z ] ∥ 2 ≥ r j ∈ { 0 , … , J − 1 } j \in\{0,\ldots,J-1\} j ∈ { 0 , … , J − 1 }
s j ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s j , s_j \le\|\mathbb{E}[Z]\|_2 \le2s_j, s j ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s j , because ∥ E [ Z ] ∥ 2 ≤ B \|\mathbb{E}[Z]\|_2 \le B ∥ E [ Z ] ∥ 2 ≤ B s j ≥ r s_j \ge r s j ≥ r (63) holds, the corresponding single-scale test has type-II error at most η j ≤ η \eta_j \le\eta η j ≤ η
Technical details of the sparse upper bound
Proof of Proposition 9. We take the choice of η n e t : = min { 1 / 2 , c c / ( 2 C c ) } \eta_{\mathrm{net}} := \min\{1/2, c_c/(2C_c)\} η net := min { 1/2 , c c / ( 2 C c )} c c , C c c_c, C_c c c , C c Lemma 25 . For v ∈ V k v \in\mathcal{V}_k v ∈ V k
K j , v : = 1 2 { tanh ( v ⊤ X 2 j − 1 ) − tanh ( v ⊤ X 2 j ) } { tanh ( Y 2 j − 1 ) − tanh ( Y 2 j ) } , K ‾ v : = 1 m ∑ j = 1 m K j , v , K_{j,v} := \frac{1}{2}\left\{\tanh(v^\top X_{2j-1})-\tanh(v^\top X_{2j})\right\}\left\{\tanh(Y_{2j-1})-\tanh(Y_{2j})\right\}, \qquad\overline{K}_v := \frac{1}{m}\sum_{j=1}^{m}K_{j,v}, K j , v := 2 1 { tanh ( v ⊤ X 2 j − 1 ) − tanh ( v ⊤ X 2 j ) } { tanh ( Y 2 j − 1 ) − tanh ( Y 2 j ) } , K v := m 1 j = 1 ∑ m K j , v , and
A ( D ) : = max k ∈ K { max v ∈ V k K ‾ v − q k } , q k : = 8 log ( 2 ∣ K ∣ ∣ V k ∣ / η ) m . A(D) := \max_{k\in\mathcal{K}}\left\{\max_{v\in\mathcal{V}_k}\overline{K}_v-q_k\right\}, \qquad q_k := \sqrt{\frac{8\log(2|\mathcal{K}||\mathcal{V}_k|/\eta)}{m}}. A ( D ) := k ∈ K max { v ∈ V k max K v − q k } , q k := m 8 log ( 2∣ K ∣∣ V k ∣/ η ) . Privacy. Since K j , v ∈ [ − 2 , 2 ] K_{j,v} \in[-2,2] K j , v ∈ [ − 2 , 2 ] K ‾ v \overline{K}_v K v 4 / m 4/m 4/ m v v v ( k ∗ , v ∗ ) (k_*,v_*) ( k ∗ , v ∗ ) A ( D ) A(D) A ( D ) D , D ′ D,D' D , D ′
A ( D ) = K ‾ v ∗ ( D ) − q k ∗ ≤ K ‾ v ∗ ( D ′ ) − q k ∗ + 4 m ≤ A ( D ′ ) + 4 m . A(D)=\overline{K}_{v_*}(D)-q_{k_*}\le\overline{K}_{v_*}(D')-q_{k_*}+\frac{4}{m}\le A(D')+\frac{4}{m}. A ( D ) = K v ∗ ( D ) − q k ∗ ≤ K v ∗ ( D ′ ) − q k ∗ + m 4 ≤ A ( D ′ ) + m 4 . Interchanging D D D D ′ D' D ′ ∣ A ( D ) − A ( D ′ ) ∣ ≤ 4 / m |A(D)-A(D')|\le4/m ∣ A ( D ) − A ( D ′ ) ∣ ≤ 4/ m ϕ s p a r s e \phi_{\mathrm{sparse}} ϕ sparse ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 )
Type-I error. By independence of the observations, we have that
E [ K j , v ] = Cov { tanh ( v ⊤ X ) , tanh ( Y ) } . \mathbb{E}[K_{j,v}]=\operatorname{Cov}\{\tanh(v^\top X),\tanh(Y)\}. E [ K j , v ] = Cov { tanh ( v ⊤ X ) , tanh ( Y )} . Under the null, this expectation is zero, so Hoeffding’s inequality and a union bound give
P ( A ( D ) > 0 ) ≤ ∑ k ∈ K ∣ V k ∣ exp ( − m q k 2 8 ) = η 2 . \mathbb{P}(A(D)>0)\le\sum_{k\in\mathcal{K}}|\mathcal{V}_k|\exp\left(-\frac{m q_k^2}{8}\right)=\frac{\eta}{2}. P ( A ( D ) > 0 ) ≤ k ∈ K ∑ ∣ V k ∣ exp ( − 8 m q k 2 ) = 2 η . Since P ( W > log ( 1 / η ) ) = η / 2 \mathbb{P}(W>\log(1/\eta))=\eta/2 P ( W > log ( 1/ η )) = η /2 η \eta η
Type-II error. Suppose that ∥ β ∥ 2 > 0 \|\beta\|_2>0 ∥ β ∥ 2 > 0 ∥ β ∥ 0 ≤ s ∗ \|\beta\|_0\le s^* ∥ β ∥ 0 ≤ s ∗ k ∗ ∈ K k^*\in\mathcal{K} k ∗ ∈ K s ∗ ≤ k ∗ ≤ 2 s ∗ s^*\le k^*\le2s^* s ∗ ≤ k ∗ ≤ 2 s ∗ Lemma 25 , one of u ∗ = β / ∥ β ∥ 2 u^*=\beta/\|\beta\|_2 u ∗ = β /∥ β ∥ 2 u ∗ = − β / ∥ β ∥ 2 u^*=-\beta/\|\beta\|_2 u ∗ = − β /∥ β ∥ 2
Cov ( tanh ( ( u ∗ ) ⊤ X ) , tanh ( Y ) ) ≥ c c min { ∥ β ∥ 2 , 1 } . \operatorname{Cov}\left(\tanh((u^*)^\top X),\tanh(Y)\right)\ge c_c\min\{\|\beta\|_2,1\}. Cov ( tanh (( u ∗ ) ⊤ X ) , tanh ( Y ) ) ≥ c c min { ∥ β ∥ 2 , 1 } . By construction, there exists v ∗ ∈ V k ∗ v^* \in V_{k^*} v ∗ ∈ V k ∗ ∥ v ∗ − u ∗ ∥ 2 ≤ η n e t \lVert v^* - u^* \rVert_2 \le\eta_{\mathrm{net}} ∥ v ∗ − u ∗ ∥ 2 ≤ η net Lemma 25 , we have that
E [ K 1 , v ∗ ] = Cov ( tanh ( ( v ∗ ) ⊤ X ) , tanh ( Y ) ) ≥ Cov ( tanh ( ( u ∗ ) ⊤ X ) , tanh ( Y ) ) − C c ∥ v ∗ − u ∗ ∥ 2 min { ∥ β ∥ 2 , 1 } ≥ ( c c − C c η n e t ) min { ∥ β ∥ 2 , 1 } ≥ c c 2 min { ∥ β ∥ 2 , 1 } , (64) \begin{aligned}
\mathbb{E}[K_{1,v^*}] &= \operatorname{Cov}\left(\tanh((v^*)^\top X),\tanh(Y)\right) \\
&\ge\operatorname{Cov}\left(\tanh((u^*)^\top X),\tanh(Y)\right)-C_c\lVert v^*-u^*\rVert_2\min\{\lVert\beta\rVert_2,1\} \\
&\ge(c_c-C_c\eta_{\mathrm{net}})\min\{\lVert\beta\rVert_2,1\} \\
&\ge\frac{c_c}{2}\min\{\lVert\beta\rVert_2,1\}, \tag*{(64)}
\end{aligned} E [ K 1 , v ∗ ] = Cov ( tanh (( v ∗ ) ⊤ X ) , tanh ( Y ) ) ≥ Cov ( tanh (( u ∗ ) ⊤ X ) , tanh ( Y ) ) − C c ∥ v ∗ − u ∗ ∥ 2 min {∥ β ∥ 2 , 1 } ≥ ( c c − C c η net ) min {∥ β ∥ 2 , 1 } ≥ 2 c c min {∥ β ∥ 2 , 1 } , (64) by noting that since tanh \tanh tanh (68) applies around either sign of β / ∥ β ∥ 2 \beta/\lVert\beta\rVert_2 β / ∥ β ∥ 2
Hoeffding’s inequality implies that
P β ( K ‾ v ∗ < E [ K 1 , v ∗ ] − q k ∗ ) ≤ exp ( − m q k ∗ 2 8 ) = η 2 ∣ K ∣ ∣ V k ∗ ∣ . (65) \mathbb{P}_{\beta}\left(\overline{K}_{v^*}<\mathbb{E}[K_{1,v^*}]-q_{k^*}\right)\le\exp\left(-\frac{m q_{k^*}^{2}}{8}\right)=\frac{\eta}{2|\mathcal{K}||V_{k^*}|}. \tag*{(65)} P β ( K v ∗ < E [ K 1 , v ∗ ] − q k ∗ ) ≤ exp ( − 8 m q k ∗ 2 ) = 2∣ K ∣∣ V k ∗ ∣ η . (65) By construction of the nets V s \mathcal{V}_s V s
log ∣ V k ∗ ∣ ≤ k ∗ log ( e d / k ∗ ) + k ∗ log ( 1 + 2 / η n e t ) ≤ 2 s ∗ log ( e d / s ∗ ) + 2 s ∗ log ( 1 + 2 / η n e t ) ≲ s ∗ log ( e d / s ∗ ) . (66) \begin{aligned}
\log|V_{k^*}| &\le k^*\log(ed/k^*)+k^*\log(1+2/\eta_{\mathrm{net}}) \\
&\le2s^*\log(ed/s^*)+2s^*\log(1+2/\eta_{\mathrm{net}})\lesssim s^*\log(ed/s^*). \tag*{(66)}
\end{aligned} log ∣ V k ∗ ∣ ≤ k ∗ log ( e d / k ∗ ) + k ∗ log ( 1 + 2/ η net ) ≤ 2 s ∗ log ( e d / s ∗ ) + 2 s ∗ log ( 1 + 2/ η net ) ≲ s ∗ log ( e d / s ∗ ) . (66) Applying the separation condition in (17) with c c c (64) therefore gives
E [ K 1 , v ∗ ] > 2 q k ∗ + 8 log ( 1 / η ) m c . \mathbb{E}[K_{1,v^*}]>2q_{k^*}+\frac{8\log(1/\eta)}{mc}. E [ K 1 , v ∗ ] > 2 q k ∗ + m c 8 log ( 1/ η ) . Thus, combining (65) and that P ( W < − log ( 1 / η ) ) = η / 2 \mathbb{P}(W<-\log(1/\eta))=\eta/2 P ( W < − log ( 1/ η )) = η /2 η \eta η
Lemma 25. Suppose ( X , Y ) (X,Y) ( X , Y ) P L i n \mathcal{P}_{\mathrm{Lin}} P Lin P G L M ( F ) \mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P GLM ( F ) F \mathcal{F} F c c , C c > 0 c_c,C_c>0 c c , C c > 0 β = ∥ β ∥ 2 u \beta=\lVert\beta\rVert_2u β = ∥ β ∥ 2 u ∥ β ∥ 2 > 0 \lVert\beta\rVert_2>0 ∥ β ∥ 2 > 0 ∥ u ∥ 2 = 1 \lVert u\rVert_2=1 ∥ u ∥ 2 = 1 v v v
∣ Cov ( tanh ( u ⊤ X ) , tanh ( Y ) ) ∣ ≥ c c min { ∥ β ∥ 2 , 1 } (67) \left|\operatorname{Cov}\left(\tanh(u^\top X),\tanh(Y)\right)\right|\ge c_c\min\{\lVert\beta\rVert_2,1\} \tag*{(67)} Cov ( tanh ( u ⊤ X ) , tanh ( Y ) ) ≥ c c min {∥ β ∥ 2 , 1 } (67) and
∣ Cov ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) , tanh ( Y ) ) ∣ ≤ C c ∥ v − u ∥ 2 min { ∥ β ∥ 2 , 1 } . (68) \left|\operatorname{Cov}\left(\tanh(u^\top X)-\tanh(v^\top X),\tanh(Y)\right)\right|\le C_c\lVert v-u\rVert_2\min\{\lVert\beta\rVert_2,1\}. \tag*{(68)} Cov ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) , tanh ( Y ) ) ≤ C c ∥ v − u ∥ 2 min {∥ β ∥ 2 , 1 } . (68) Proof. Let V = u ⊤ X V=u^\top X V = u ⊤ X c m , C m , t 0 c_m,C_m,t_0 c m , C m , t 0 Lemma 22 . By Assumption 1, we have that E [ V ] = 0 \mathbb{E}[V]=0 E [ V ] = 0 λ ≤ E [ V 2 ] ≤ λ ‾ \lambda\le\mathbb{E}[V^2]\le\overline{\lambda} λ ≤ E [ V 2 ] ≤ λ E [ V 4 ] ≤ M 4 \mathbb{E}[V^4]\le M_4 E [ V 4 ] ≤ M 4 M 4 M_4 M 4 E [ V ] = 0 \mathbb{E}[V]=0 E [ V ] = 0
E [ { V } + ] = E [ { V } − ] = 1 2 E [ ∣ V ∣ ] ≥ 1 2 E [ V 2 ] 3 E [ V 4 ] ≥ c V : = λ 3 / 2 2 M 4 > 0. \mathbb{E}[\{V\}_+]=\mathbb{E}[\{V\}_-]=\frac{1}{2}\mathbb{E}[|V|]\ge\frac{1}{2}\sqrt{\frac{\mathbb{E}[V^2]^3}{\mathbb{E}[V^4]}}\ge c_V:=\frac{\lambda^{3/2}}{2\sqrt{M_4}}>0. E [{ V } + ] = E [{ V } − ] = 2 1 E [ ∣ V ∣ ] ≥ 2 1 E [ V 4 ] E [ V 2 ] 3 ≥ c V := 2 M 4 λ 3/2 > 0. We can bound
E [ { V } + 1 { V ≥ c V / 2 } ] = E [ { V } + ] − E [ { V } + 1 { V < c V / 2 } ] ≥ c V / 2 , \mathbb{E}\left[\{V\}_{+}\mathbf{1}\{V\ge c_V/2\}\right]=\mathbb{E}[\{V\}_{+}]-\mathbb{E}\left[\{V\}_{+}\mathbf{1}\{V<c_V/2\}\right]\ge c_V/2, E [ { V } + 1 { V ≥ c V /2 } ] = E [{ V } + ] − E [ { V } + 1 { V < c V /2 } ] ≥ c V /2 , and by Cauchy–Schwarz, we have that
E [ { V } + 1 { V ≥ c V / 2 } ] 2 ≤ E [ { V } + 2 ] P ( V ≥ c V / 2 ) , \mathbb{E}\left[\{V\}_{+}\mathbf{1}\{V\ge c_V/2\}\right]^2\le\mathbb{E}[\{V\}_{+}^2]\mathbb{P}(V\ge c_V/2), E [ { V } + 1 { V ≥ c V /2 } ] 2 ≤ E [{ V } + 2 ] P ( V ≥ c V /2 ) , which implies that
P ( V ≥ c V / 2 ) ≥ p V , p V : = λ 3 16 M 4 λ . \mathbb{P}(V\ge c_V/2)\ge p_V,\qquad p_V:=\frac{\lambda^3}{16M_4\lambda}. P ( V ≥ c V /2 ) ≥ p V , p V := 16 M 4 λ λ 3 . Likewise, we have P ( V ≤ − c V / 2 ) ≥ p V \mathbb{P}(V\le-c_V/2)\ge p_V P ( V ≤ − c V /2 ) ≥ p V
With V ′ V' V ′ V V V m ν m_\nu m ν (54) , we have
Cov ( tanh ( V ) , m ν ( ∥ β ∥ 2 V ) ) = 1 2 E [ ( tanh ( V ) − tanh ( V ′ ) ) ( m ν ( ∥ β ∥ 2 V ) − m ν ( ∥ β ∥ 2 V ′ ) ) ] \operatorname{Cov}(\tanh(V),m_\nu(\|\beta\|_2V))=\frac{1}{2}\mathbb{E}\left[(\tanh(V)-\tanh(V'))(m_\nu(\|\beta\|_2V)-m_\nu(\|\beta\|_2V'))\right] Cov ( tanh ( V ) , m ν ( ∥ β ∥ 2 V )) = 2 1 E [ ( tanh ( V ) − tanh ( V ′ )) ( m ν ( ∥ β ∥ 2 V ) − m ν ( ∥ β ∥ 2 V ′ )) ] ≥ 2 p V 2 tanh ( c V / 2 ) { m ν ( ∥ β ∥ 2 c V / 2 ) − m ν ( − ∥ β ∥ 2 c V / 2 ) } \ge2p_V^2\tanh(c_V/2)\{m_\nu(\|\beta\|_2c_V/2)-m_\nu(-\|\beta\|_2c_V/2)\} ≥ 2 p V 2 tanh ( c V /2 ) { m ν ( ∥ β ∥ 2 c V /2 ) − m ν ( − ∥ β ∥ 2 c V /2 )} ≥ 4 p V 2 c m tanh ( c V / 2 ) min { ∥ β ∥ 2 c V / 2 , t 0 } ≳ min ( ∥ β ∥ 2 , 1 ) , \ge4p_V^2c_m\tanh(c_V/2)\min\{\|\beta\|_2c_V/2,t_0\}\gtrsim\min(\|\beta\|_2,1), ≥ 4 p V 2 c m tanh ( c V /2 ) min { ∥ β ∥ 2 c V /2 , t 0 } ≳ min ( ∥ β ∥ 2 , 1 ) , where the final inequality is due to Lemma 22. Conditioning on X X X
Cov ( tanh ( V ) , m ν ( ∥ β ∥ 2 V ) ) = s ϑ Cov ( tanh ( u ⊤ X ) , tanh ( Y ) ) \operatorname{Cov}(\tanh(V),m_\nu(\|\beta\|_2V))=s_\vartheta\operatorname{Cov}(\tanh(u^\top X),\tanh(Y)) Cov ( tanh ( V ) , m ν ( ∥ β ∥ 2 V )) = s ϑ Cov ( tanh ( u ⊤ X ) , tanh ( Y )) which proves (67) .
Conditioning on X X X
∣ Cov ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) , tanh ( Y ) ) ∣ = ∣ Cov ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) , m ν ( ∥ β ∥ 2 V ) ) ∣ ≤ Var ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) ) 1 / 2 Var ( m ν ( ∥ β ∥ 2 V ) ) 1 / 2 . (69) \begin{aligned}
\left|\operatorname{Cov}\left(\tanh(u^\top X)-\tanh(v^\top X),\tanh(Y)\right)\right|
=
\left|\operatorname{Cov}\left(\tanh(u^\top X)-\tanh(v^\top X),m_\nu(\|\beta\|_2V)\right)\right| \\
&\le\operatorname{Var}\left(\tanh(u^\top X)-\tanh(v^\top X)\right)^{1/2}\operatorname{Var}\left(m_\nu(\|\beta\|_2V)\right)^{1/2}. \tag*{(69)}
\end{aligned} Cov ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) , tanh ( Y ) ) = Cov ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) , m ν ( ∥ β ∥ 2 V ) ) ≤ Var ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) ) 1/2 Var ( m ν ( ∥ β ∥ 2 V ) ) 1/2 . (69) Since tanh \tanh tanh
Var ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) ) ≤ E [ ( ( u − v ) ⊤ X ) 2 ] ≤ λ ˉ ∥ u − v ∥ 2 2 . \operatorname{Var}\left(\tanh(u^\top X)-\tanh(v^\top X)\right)\le\mathbb{E}\left[((u-v)^\top X)^2\right]\le\bar{\lambda}\|u-v\|_2^2. Var ( tanh ( u ⊤ X ) − tanh ( v ⊤ X ) ) ≤ E [ (( u − v ) ⊤ X ) 2 ] ≤ λ ˉ ∥ u − v ∥ 2 2 . It remains to show that Var ( m ν ( ∥ β ∥ 2 V ) ) ≲ min { 1 , ∥ β ∥ 2 2 } \operatorname{Var}(m_\nu(\|\beta\|_2V))\lesssim\min\{1,\|\beta\|_2^2\} Var ( m ν ( ∥ β ∥ 2 V )) ≲ min { 1 , ∥ β ∥ 2 2 } ∥ β ∥ 2 ≤ 1 \|\beta\|_2\le1 ∥ β ∥ 2 ≤ 1
E ∣ m ν ( ∥ β ∥ 2 V ) − m ν ( 0 ) ∣ 2 ≤ C m 2 ∥ β ∥ 2 2 E [ V 2 ] + 4 P ( ∥ β ∥ 2 ∣ V ∣ > t 0 ) ≤ ( C m 2 + 4 / t 0 2 ) λ ∥ β ∥ 2 2 . \begin{aligned}
\mathbb{E}|m_\nu(\|\beta\|_2V)-m_\nu(0)|^2\le C_m^2\|\beta\|_2^2\mathbb{E}[V^2]+4\mathbb{P}(\|\beta\|_2|V|>t_0) \\
&\le(C_m^2+4/t_0^2)\lambda\|\beta\|_2^2.
\end{aligned} E ∣ m ν ( ∥ β ∥ 2 V ) − m ν ( 0 ) ∣ 2 ≤ C m 2 ∥ β ∥ 2 2 E [ V 2 ] + 4 P ( ∥ β ∥ 2 ∣ V ∣ > t 0 ) ≤ ( C m 2 + 4/ t 0 2 ) λ ∥ β ∥ 2 2 . Otherwise, if ∥ β ∥ 2 ≥ 1 \|\beta\|_2\ge1 ∥ β ∥ 2 ≥ 1 ∣ m ν ∣ ≤ 1 |m_\nu|\le1 ∣ m ν ∣ ≤ 1
Var ( m ν ( ∥ β ∥ 2 V ) ) ≤ E ∣ m ν ( ∥ β ∥ 2 V ) − m ν ( 0 ) ∣ 2 , \operatorname{Var}(m_\nu(\|\beta\|_2V))\le\mathbb{E}|m_\nu(\|\beta\|_2V)-m_\nu(0)|^2, Var ( m ν ( ∥ β ∥ 2 V )) ≤ E ∣ m ν ( ∥ β ∥ 2 V ) − m ν ( 0 ) ∣ 2 , combining the two cases yields the second variance bound in (69) , which proves (68) .
Technical details on response-only tests
Appendix D.1 states the precise form of the g a g_a g a
Proof of 10
For a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1 Γ 1 \Gamma_1 Γ 1 Lemma 16 , we define our test function as follows. Let
A a , M ( y ) : = [ R a ( y ) ] − M M , M : = max { 1 , 2 K 4 Δ min 2 , 8 K S K R K 4 c m Δ min 2 } , (70) A_{a,M}(y) := [R_a(y)]_{-M}^{M}, \qquad M := \max\left\{1, 2\sqrt{\frac{K_4}{\Delta_{\min}^2}}, \frac{8K_SK_R\sqrt{K_4}}{c_m\Delta_{\min}^2}\right\}, \tag*{(70)} A a , M ( y ) := [ R a ( y ) ] − M M , M := max { 1 , 2 Δ m i n 2 K 4 , c m Δ m i n 2 8 K S K R K 4 } , (70) where
K 4 : = sup a ∈ Γ 1 E a [ R a 4 ] , K R : = sup a ∈ Γ 1 ∥ R a ∥ L 2 ( P a ) , K S : = sup a ∈ Γ 1 ∥ S a ∥ L 2 ( P a ) , K_4 := \sup_{a\in\Gamma_1}\mathbb{E}_a[R_a^4], \qquad K_R := \sup_{a\in\Gamma_1}\|R_a\|_{L_2(P_a)}, \qquad K_S := \sup_{a\in\Gamma_1}\|S_a\|_{L_2(P_a)}, K 4 := a ∈ Γ 1 sup E a [ R a 4 ] , K R := a ∈ Γ 1 sup ∥ R a ∥ L 2 ( P a ) , K S := a ∈ Γ 1 sup ∥ S a ∥ L 2 ( P a ) , and, defining m ( a ) : = E a [ tanh ( Y ) ] m(a) := \mathbb{E}_a[\tanh(Y)] m ( a ) := E a [ tanh ( Y )] ψ ˉ a ( y ) : = tanh ( y ) − m ( a ) \bar{\psi}_a(y) := \tanh(y)-m(a) ψ ˉ a ( y ) := tanh ( y ) − m ( a )
c m : = inf a ∈ Γ 1 ∣ m ′ ( a ) ∣ > 0 , Δ min 2 : = inf a ∈ Γ 1 E a [ R a 2 ] > 0 , c_m := \inf_{a\in\Gamma_1}|m'(a)| > 0, \qquad\Delta_{\min}^2 := \inf_{a\in\Gamma_1}\mathbb{E}_a[R_a^2] > 0, c m := a ∈ Γ 1 inf ∣ m ′ ( a ) ∣ > 0 , Δ m i n 2 := a ∈ Γ 1 inf E a [ R a 2 ] > 0 , where Γ 1 \Gamma_1 Γ 1 Lemma 16 . The first three constants are well-defined due to Lemma 26. The positivity of c m c_m c m (55) . The positivity of Δ min 2 \Delta_{\min}^2 Δ m i n 2 Lemma 5 applied on the compact interval Γ 1 ⊂ U \Gamma_1 \subset U Γ 1 ⊂ U
Finally, define
λ a , M : = E a [ A a , M ( Y ) S a ( Y ) ] m ′ ( a ) , g a ( y ) : = A a , M ( y ) − E a [ A a , M ( Y ) ] − λ a , M ψ ˉ a ( y ) . \lambda_{a,M} := \frac{\mathbb{E}_a[A_{a,M}(Y)S_a(Y)]}{m'(a)}, \qquad g_a(y) := A_{a,M}(y)-\mathbb{E}_a[A_{a,M}(Y)]-\lambda_{a,M}\bar{\psi}_a(y). λ a , M := m ′ ( a ) E a [ A a , M ( Y ) S a ( Y )] , g a ( y ) := A a , M ( y ) − E a [ A a , M ( Y )] − λ a , M ψ ˉ a ( y ) . Proof. We have that E a [ g a ( Y ) ] = 0 \mathbb{E}_a[g_a(Y)] = 0 E a [ g a ( Y )] = 0 E a [ ψ ˉ a ( Y ) ] = 0 \mathbb{E}_a[\bar{\psi}_a(Y)] = 0 E a [ ψ ˉ a ( Y )] = 0
m ′ ( a ) = ∫ tanh ( y ) p a ( y ) ∂ a log ( p a ( y ) ) d y = E a [ { tanh ( Y ) − m ( a ) } S a ( Y ) ] + m ( a ) E a [ S a ( Y ) ] = E a [ ψ ˉ a ( Y ) S a ( Y ) ] , \begin{aligned}
m'(a) = \int\tanh(y)p_a(y)\partial_a\log(p_a(y))\,\mathrm{d}y \\
&= \mathbb{E}_a[\{\tanh(Y)-m(a)\}S_a(Y)] + m(a)\mathbb{E}_a[S_a(Y)] \\
&= \mathbb{E}_a[\bar{\psi}_a(Y)S_a(Y)],
\end{aligned} m ′ ( a ) = ∫ tanh ( y ) p a ( y ) ∂ a log ( p a ( y )) d y = E a [{ tanh ( Y ) − m ( a )} S a ( Y )] + m ( a ) E a [ S a ( Y )] = E a [ ψ ˉ a ( Y ) S a ( Y )] , we see that the orthogonality claim in (19) follows from
E a [ g a ( Y ) S a ( Y ) ] = E a [ A a , M ( Y ) S a ( Y ) ] − λ a , M E a [ ψ ˉ a ( Y ) S a ( Y ) ] = 0. \mathbb{E}_a[g_a(Y)S_a(Y)] = \mathbb{E}_a[A_{a,M}(Y)S_a(Y)]-\lambda_{a,M}\mathbb{E}_a[\bar{\psi}_a(Y)S_a(Y)] = 0. E a [ g a ( Y ) S a ( Y )] = E a [ A a , M ( Y ) S a ( Y )] − λ a , M E a [ ψ ˉ a ( Y ) S a ( Y )] = 0. We now show that this choice of g a g_a g a Lemma 17 that R a = H a + 2 c ∗ ( a ) S a R_a = H_a + 2c^*(a)S_a R a = H a + 2 c ∗ ( a ) S a E a [ g a ( Y ) S a ( Y ) ] = 0 \mathbb{E}_a[g_a(Y)S_a(Y)] = 0 E a [ g a ( Y ) S a ( Y )] = 0
E a [ g a ( Y ) H a ( Y ) ] = E a [ g a ( Y ) R a ( Y ) ] = E a [ R a 2 ] + E a [ ( g a − R a ) R a ] = E a [ R a 2 ] + E a [ ( A a , M − R a ) R a ] − λ a , M E a [ ψ ˉ a R a ] ≥ Δ min 2 + E a [ ( A a , M − R a ) R a ] − λ a , M E a [ ψ ˉ a R a ] , \begin{aligned}
\mathbb{E}_a[g_a(Y)H_a(Y)] &= \mathbb{E}_a[g_a(Y)R_a(Y)] = \mathbb{E}_a[R_a^2] + \mathbb{E}_a[(g_a-R_a)R_a] \\
&= \mathbb{E}_a[R_a^2] + \mathbb{E}_a[(A_{a,M}-R_a)R_a] - \lambda_{a,M}\mathbb{E}_a[\bar{\psi}_aR_a] \\
&\ge\Delta_{\min}^2 + \mathbb{E}_a[(A_{a,M}-R_a)R_a] - \lambda_{a,M}\mathbb{E}_a[\bar{\psi}_aR_a],
\end{aligned} E a [ g a ( Y ) H a ( Y )] = E a [ g a ( Y ) R a ( Y )] = E a [ R a 2 ] + E a [( g a − R a ) R a ] = E a [ R a 2 ] + E a [( A a , M − R a ) R a ] − λ a , M E a [ ψ ˉ a R a ] ≥ Δ m i n 2 + E a [( A a , M − R a ) R a ] − λ a , M E a [ ψ ˉ a R a ] , where the second equality is due to E a [ R a ] = 0 \mathbb{E}_a[R_a] = 0 E a [ R a ] = 0
The truncation error satisfies
E a [ ( A a , M − R a ) R a ] ≥ − E a [ R a 2 1 { ∣ R a ∣ > M } ] ≥ − E a [ R a 4 ] M 2 ≥ − K 4 M 2 . \mathbb{E}_a[(A_{a,M} - R_a)R_a] \ge-\mathbb{E}_a[R_a^2\mathbf{1}\{|R_a| > M\}]
\ge-\frac{\mathbb{E}_a[R_a^4]}{M^2} \ge-\frac{K_4}{M^2}. E a [( A a , M − R a ) R a ] ≥ − E a [ R a 2 1 { ∣ R a ∣ > M }] ≥ − M 2 E a [ R a 4 ] ≥ − M 2 K 4 . For the correction term, we can bound
∣ λ a , M ∣ ∣ E a [ ψ ˉ a R a ] ∣ ≤ ∣ E a [ A a , M S a ] ∣ ∣ m ′ ( a ) ∣ × 2 E a [ ∣ R a ∣ ] |\lambda_{a,M}||\mathbb{E}_a[\bar{\psi}_aR_a]| \le\frac{|\mathbb{E}_a[A_{a,M}S_a]|}{|m'(a)|} \times2\mathbb{E}_a[|R_a|] ∣ λ a , M ∣∣ E a [ ψ ˉ a R a ] ∣ ≤ ∣ m ′ ( a ) ∣ ∣ E a [ A a , M S a ] ∣ × 2 E a [ ∣ R a ∣ ] ≤ 2 K R ∣ E a [ ( A a , M − R a ) S a ] ∣ c m ≤ 2 K R K S c m E a [ ( A a , M − R a ) 2 ] 1 / 2 ≤ 2 K R K S K 4 c m M . \begin{aligned}
\le2K_R\frac{|\mathbb{E}_a[(A_{a,M}-R_a)S_a]|}{c_m} \\
&\le\frac{2K_RK_S}{c_m}\mathbb{E}_a[(A_{a,M}-R_a)^2]^{1/2} \le\frac{2K_RK_S\sqrt{K_4}}{c_mM}.
\end{aligned} ≤ 2 K R c m ∣ E a [( A a , M − R a ) S a ] ∣ ≤ c m 2 K R K S E a [( A a , M − R a ) 2 ] 1/2 ≤ c m M 2 K R K S K 4 . where the first inequality is from ∣ tanh ∣ ≤ 1 |\tanh| \le1 ∣ tanh ∣ ≤ 1 ∣ ψ ˉ a ∣ ≤ 2 |\bar{\psi}_a| \le2 ∣ ψ ˉ a ∣ ≤ 2 E a [ R a S a ] = 0 \mathbb{E}_a[R_aS_a] = 0 E a [ R a S a ] = 0
E a [ ( A a , M − R a ) 2 ] 1 / 2 ≤ E a [ R a 2 1 { ∣ R a ∣ > M } ] 1 / 2 ≤ E a [ R a 4 / M 2 ] 1 / 2 ≤ K 4 / M . \begin{aligned}
\mathbb{E}_a[(A_{a,M}-R_a)^2]^{1/2} \le\mathbb{E}_a[R_a^2\mathbf{1}\{|R_a| > M\}]^{1/2} \\
&\le\mathbb{E}_a[R_a^4/M^2]^{1/2} \le\sqrt{K_4}/M.
\end{aligned} E a [( A a , M − R a ) 2 ] 1/2 ≤ E a [ R a 2 1 { ∣ R a ∣ > M } ] 1/2 ≤ E a [ R a 4 / M 2 ] 1/2 ≤ K 4 / M . We therefore have
E a [ g a H a ] ≥ Δ min 2 − K 4 M 2 − 2 K R K S K 4 c m M ≥ Δ min 2 2 . \mathbb{E}_a[g_aH_a] \ge\Delta_{\min}^2-\frac{K_4}{M^2}-\frac{2K_RK_S\sqrt{K_4}}{c_mM}\ge\frac{\Delta_{\min}^2}{2}. E a [ g a H a ] ≥ Δ m i n 2 − M 2 K 4 − c m M 2 K R K S K 4 ≥ 2 Δ m i n 2 . uniformly over a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1 M M M (70) , which proves (20) with c g : = Δ min 2 / 2 c_g := \Delta_{\min}^2/2 c g := Δ m i n 2 /2
Finally, we have
∣ g a ( y ) ∣ ≤ ∣ A a , M ( y ) ∣ + ∣ E a [ A a , M ( Y ) ] ∣ + 2 ∣ λ a , M ∣ |g_a(y)| \le|A_{a,M}(y)|+|\mathbb{E}_a[A_{a,M}(Y)]|+2|\lambda_{a,M}| ∣ g a ( y ) ∣ ≤ ∣ A a , M ( y ) ∣ + ∣ E a [ A a , M ( Y )] ∣ + 2∣ λ a , M ∣ ≤ 2 M + 2 K S K 4 c m M \le2M+\frac{2K_S\sqrt{K_4}}{c_mM} ≤ 2 M + c m M 2 K S K 4 uniformly over y y y a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1 B g B_g B g (21) . □ \square □
Lemma 26. Under Assumption 3, and with Γ 2 \Gamma_2 Γ 2 as defined in Lemma 16, there exists s 0 > 0 s_0 > 0 s 0 > 0 such that
sup a ∈ Γ 2 E a exp { s 0 ∣ Y ∣ } < ∞ . (71) \sup_{a\in\Gamma_2}\mathbb{E}_a\exp\{s_0|Y|\}<\infty. \tag*{(71)} a ∈ Γ 2 sup E a exp { s 0 ∣ Y ∣ } < ∞. (71) Consequently, for every fixed integer k ≥ 1 k \ge1 k ≥ 1 sup a ∈ Γ 2 E a ∣ Y ∣ k < ∞ \sup_{a\in\Gamma_2}\mathbb{E}_a|Y|^k < \infty sup a ∈ Γ 2 E a ∣ Y ∣ k < ∞
Proof. The compact set K 0 : = ϑ ( Γ 2 ) K_0 := \vartheta(\Gamma_2) K 0 := ϑ ( Γ 2 ) Θ \Theta Θ s 0 > 0 s_0 > 0 s 0 > 0 K 0 + [ − s 0 , s 0 ] ⊂ Θ K_0 + [-s_0,s_0] \subset\Theta K 0 + [ − s 0 , s 0 ] ⊂ Θ Y ∼ P ϑ − 1 ( η ) Y \sim P_{\vartheta^{-1}(\eta)} Y ∼ P ϑ − 1 ( η ) η ∈ K 0 \eta\in K_0 η ∈ K 0
E ϑ − 1 ( η ) e s 0 Y = exp { b ( η + s 0 ) − b ( η ) } , E ϑ − 1 ( η ) e − s 0 Y = exp { b ( η − s 0 ) − b ( η ) } . \mathbb{E}_{\vartheta^{-1}(\eta)}e^{s_0Y}=\exp\{b(\eta+s_0)-b(\eta)\},\qquad\mathbb{E}_{\vartheta^{-1}(\eta)}e^{-s_0Y}=\exp\{b(\eta-s_0)-b(\eta)\}. E ϑ − 1 ( η ) e s 0 Y = exp { b ( η + s 0 ) − b ( η )} , E ϑ − 1 ( η ) e − s 0 Y = exp { b ( η − s 0 ) − b ( η )} . The two right-hand sides are uniformly bounded because b b b
set K 0 + [ − s 0 , s 0 ] K_0 + [-s_0,s_0] K 0 + [ − s 0 , s 0 ] exp ( s 0 ∣ Y ∣ ) ≤ exp ( s 0 Y ) + exp ( − s 0 Y ) \exp(s_0|Y|) \le\exp(s_0Y) + \exp(-s_0Y) exp ( s 0 ∣ Y ∣ ) ≤ exp ( s 0 Y ) + exp ( − s 0 Y ) (71) . The polynomial moment bound follows from ∣ y ∣ k ≤ C k , s 0 exp ( s 0 ∣ y ∣ ) |y|^k \le C_{k,s_0}\exp(s_0|y|) ∣ y ∣ k ≤ C k , s 0 exp ( s 0 ∣ y ∣ )
Proof of 11
Lemma 27. Let γ ^ \widehat{\gamma} γ (23) . There exist constants C γ , r 0 > 0 C_\gamma,r_0 > 0 C γ , r 0 > 0 ∥ β ∥ 2 < r 0 \|\beta\|_2 < r_0 ∥ β ∥ 2 < r 0 η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 )
P ( ∣ γ ^ − γ ∣ > C γ { ∥ β ∥ 2 2 + a n ( ϵ , η ) } ) ≤ η 2 , a n ( ϵ , η ) : = log ( 8 / η ) n + log ( 8 / η ) n ϵ . \mathbb{P}\left(\left|\widehat{\gamma}-\gamma\right| > C_\gamma\left\{\|\beta\|_2^2+a_n(\epsilon,\eta)\right\}\right) \le\frac{\eta}{2}, \qquad a_n(\epsilon,\eta) := \sqrt{\frac{\log(8/\eta)}{n}}+\frac{\log(8/\eta)}{n\epsilon}. P ( ∣ γ − γ ∣ > C γ { ∥ β ∥ 2 2 + a n ( ϵ , η ) } ) ≤ 2 η , a n ( ϵ , η ) := n log ( 8/ η ) + n ϵ log ( 8/ η ) . Proof. By Lemma 22 , m ( a ) m(a) m ( a ) ∣ m ′ ∣ ≥ c m > 0 |m'| \ge c_m > 0 ∣ m ′ ∣ ≥ c m > 0 Γ 1 \Gamma_1 Γ 1 m ( γ ^ ) m(\widehat{\gamma}) m ( γ ) m ~ \widetilde{m} m m ( Γ 1 ) m(\Gamma_1) m ( Γ 1 )
∣ γ ^ − γ ∣ ≤ c m − 1 ∣ m ( γ ^ ) − m ( γ ) ∣ ≤ c m − 1 ∣ m ~ − m ( γ ) ∣ . |\widehat{\gamma}-\gamma| \le c_m^{-1}|m(\widehat{\gamma})-m(\gamma)| \le c_m^{-1}|\widetilde{m}-m(\gamma)|. ∣ γ − γ ∣ ≤ c m − 1 ∣ m ( γ ) − m ( γ ) ∣ ≤ c m − 1 ∣ m − m ( γ ) ∣. It therefore suffices to obtain a high probability bound on
∣ m ~ − m ( γ ) ∣ ≤ ∣ m ~ − E β , γ [ tanh ( Y ) ] ∣ + ∣ E β , γ [ tanh ( Y ) ] − m ( γ ) ∣ , (72) |\widetilde{m}-m(\gamma)| \le|\widetilde{m}-\mathbb{E}_{\beta,\gamma}[\tanh(Y)]|+|\mathbb{E}_{\beta,\gamma}[\tanh(Y)]-m(\gamma)|, \tag*{(72)} ∣ m − m ( γ ) ∣ ≤ ∣ m − E β , γ [ tanh ( Y )] ∣ + ∣ E β , γ [ tanh ( Y )] − m ( γ ) ∣ , (72) where we recall the definition of m ~ \widetilde{m} m (22) . We first bound the bias term. By Lemma 16 , we have, over a ∈ Γ a \in\Gamma a ∈ Γ ∣ u ∣ ≤ t 0 |u| \le t_0 ∣ u ∣ ≤ t 0 t 0 > 0 t_0 > 0 t 0 > 0
m ( a + u ) = E a [ tanh ( Y ) L a , u ( Y ) ] = E a [ tanh ( Y ) ] + u E a [ tanh ( Y ) S a ( Y ) ] + u 2 2 E a [ tanh ( Y ) H a ( Y ) ] + o ( u 2 ) = m ( a ) + m ′ ( a ) u + O ( u 2 ) . \begin{aligned}
m(a+u) &= \mathbb{E}_a[\tanh(Y)L_{a,u}(Y)] \\
&= \mathbb{E}_a[\tanh(Y)]+u\mathbb{E}_a[\tanh(Y)S_a(Y)]+\frac{u^2}{2}\mathbb{E}_a[\tanh(Y)H_a(Y)]+o(u^2) \\
&= m(a)+m'(a)u+O(u^2).
\end{aligned} m ( a + u ) = E a [ tanh ( Y ) L a , u ( Y )] = E a [ tanh ( Y )] + u E a [ tanh ( Y ) S a ( Y )] + 2 u 2 E a [ tanh ( Y ) H a ( Y )] + o ( u 2 ) = m ( a ) + m ′ ( a ) u + O ( u 2 ) . This remainder bound extends to all u ∈ R u \in\mathbb{R} u ∈ R ∣ u ∣ > t 0 |u| > t_0 ∣ u ∣ > t 0 m m m ∣ m ′ ∣ |m'| ∣ m ′ ∣ Γ \Gamma Γ
∣ m ( γ + u ) − m ( γ ) − m ′ ( γ ) u ∣ ≤ 2 + C ∣ u ∣ ≤ C ′ u 2 . |m(\gamma+u)-m(\gamma)-m'(\gamma)u| \le2+C|u| \le C'u^2. ∣ m ( γ + u ) − m ( γ ) − m ′ ( γ ) u ∣ ≤ 2 + C ∣ u ∣ ≤ C ′ u 2 . Thus, uniformly over γ ∈ Γ \gamma\in\Gamma γ ∈ Γ C ′ u 2 C'u^2 C ′ u 2 u ∈ R u \in\mathbb{R} u ∈ R E [ X ⊤ β ] = 0 \mathbb{E}[X^\top\beta]=0 E [ X ⊤ β ] = 0
∣ E β , γ [ tanh ( Y ) ] − m ( γ ) ∣ = ∣ E X [ m ( γ + X ⊤ β ) − m ( γ ) − m ′ ( γ ) X ⊤ β ] ∣ ≤ C E X [ ( X ⊤ β ) 2 ] ≲ ∥ β ∥ 2 2 . \begin{aligned}
|\mathbb{E}_{\beta,\gamma}[\tanh(Y)]-m(\gamma)|
&= \left|\mathbb{E}_X\left[m(\gamma+X^\top\beta)-m(\gamma)-m'(\gamma)X^\top\beta\right]\right| \\
&\le C\mathbb{E}_X[(X^\top\beta)^2] \lesssim\|\beta\|_2^2.
\end{aligned} ∣ E β , γ [ tanh ( Y )] − m ( γ ) ∣ = E X [ m ( γ + X ⊤ β ) − m ( γ ) − m ′ ( γ ) X ⊤ β ] ≤ C E X [( X ⊤ β ) 2 ] ≲ ∥ β ∥ 2 2 . By Hoeffding’s inequality, we have that
P ( ∣ 1 n 1 ∑ i = 1 n 1 tanh ( Y i ) − E β , γ [ tanh ( Y ) ] ∣ > 2 log ( 8 / η ) n 1 ) ≤ η 4 . \mathbb{P}\left(\left|\frac{1}{n_1}\sum_{i=1}^{n_1}\tanh(Y_i)-\mathbb{E}_{\beta,\gamma}[\tanh(Y)]\right|>\sqrt{\frac{2\log(8/\eta)}{n_1}}\right)\le\frac{\eta}{4}. P n 1 1 i = 1 ∑ n 1 tanh ( Y i ) − E β , γ [ tanh ( Y )] > n 1 2 log ( 8/ η ) ≤ 4 η . We also have that P ( ∣ W 1 ∣ > 2 log ( 4 / η ) / ( n 1 ϵ ) ) = η / 4 \mathbb{P}(|W_1|>2\log(4/\eta)/(n_1\epsilon))=\eta/4 P ( ∣ W 1 ∣ > 2 log ( 4/ η ) / ( n 1 ϵ )) = η /4 (72) by C a n ( ϵ , η ) Ca_n(\epsilon,\eta) C a n ( ϵ , η ) 1 − η / 2 1-\eta/2 1 − η /2
Proof of Proposition 11. Privacy guarantee. The first stage is ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) tanh ( Y ) \tanh(Y) tanh ( Y ) 2 / n 1 2/n_1 2/ n 1 a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1 2 B g / n 2 2B_g/n_2 2 B g / n 2 ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 )
Concentration result. Let F 1 \mathcal{F}_1 F 1 F 1 \mathcal{F}_1 F 1 g γ ^ ( Y i ) g_{\widehat{\gamma}}(Y_i) g γ ( Y i ) i ∈ { n 1 + 1 , … , n } i \in\{n_1+1,\ldots,n\} i ∈ { n 1 + 1 , … , n } ∣ g γ ^ ( Y i ) ∣ ≤ B g |g_{\widehat{\gamma}}(Y_i)| \le B_g ∣ g γ ( Y i ) ∣ ≤ B g C μ > 0 C_\mu>0 C μ > 0
P ( ∣ T ~ − μ ( γ ^ ) ∣ > C μ a n ( ϵ , η ) ∣ F 1 ) ≤ η / 2 , μ ( a ) : = E β , γ [ g a ( Y ) ] . (73) \mathbb{P}\left(\left|\widetilde{T}-\mu(\widehat{\gamma})\right|>C_\mu a_n(\epsilon,\eta)\mid\mathcal{F}_1\right)\le\eta/2,\qquad\mu(a):=\mathbb{E}_{\beta,\gamma}[g_a(Y)]. \tag*{(73)} P ( T − μ ( γ ) > C μ a n ( ϵ , η ) ∣ F 1 ) ≤ η /2 , μ ( a ) := E β , γ [ g a ( Y )] . (73) Type-I error. Under the null, we have by Lemma 27 that ∣ γ − γ ^ ∣ ≤ C γ a n ( ϵ , η ) |\gamma-\widehat{\gamma}|\le C_\gamma a_n(\epsilon,\eta) ∣ γ − γ ∣ ≤ C γ a n ( ϵ , η ) 1 − η / 2 1-\eta/2 1 − η /2 Lemma 28 gives ∣ μ ( γ ^ ) ∣ ≲ a n ( ϵ , η ) 2 |\mu(\widehat{\gamma})|\lesssim a_n(\epsilon,\eta)^2 ∣ μ ( γ ) ∣ ≲ a n ( ϵ , η ) 2 (73) , we have, for c T c_T c T P 0 , γ { T ~ > c T a n ( ϵ , η ) } ≤ η \mathbb{P}_{0,\gamma}\{\widetilde{T}>c_Ta_n(\epsilon,\eta)\}\le\eta P 0 , γ { T > c T a n ( ϵ , η )} ≤ η
Type-II error. Suppose that C 1 a n ( ϵ , η ) ≤ ∥ β ∥ 2 2 ≤ r 0 2 C_1a_n(\epsilon,\eta)\le\|\beta\|_2^2\le r_0^2 C 1 a n ( ϵ , η ) ≤ ∥ β ∥ 2 2 ≤ r 0 2 Lemma 27 , with probability at least 1 − η / 2 1-\eta/2 1 − η /2
∣ γ − γ ^ ∣ ≲ ∥ β ∥ 2 2 + a n ( ϵ , η ) ≲ ∥ β ∥ 2 2 (74) |\gamma-\widehat{\gamma}|\lesssim\|\beta\|_2^2+a_n(\epsilon,\eta)\lesssim\|\beta\|_2^2 \tag*{(74)} ∣ γ − γ ∣ ≲ ∥ β ∥ 2 2 + a n ( ϵ , η ) ≲ ∥ β ∥ 2 2 (74) where the last inequality holds provided that C 1 C_1 C 1 v 0 v_0 v 0 Lemma 28 satisfies C v 0 ≤ c g / 4 C v_0\le c_g/4 C v 0 ≤ c g /4 (74) and take r 0 r_0 r 0 ∣ γ − γ ^ ∣ ≤ v 0 / 2 |\gamma-\widehat{\gamma}|\le v_0/2 ∣ γ − γ ∣ ≤ v 0 /2 Lemma 28 gives
E γ + X T β [ g γ ^ ( Y ) ] ≥ c g 4 ( γ − γ ^ + X T β ) 2 , ∣ γ − γ ^ + X T β ∣ ≤ v 0 . \mathbb{E}_{\gamma+X^\mathsf{T}\beta}\left[g_{\widehat{\gamma}}(Y)\right]\ge\frac{c_g}{4}\left(\gamma-\widehat{\gamma}+X^\mathsf{T}\beta\right)^2,\qquad|\gamma-\widehat{\gamma}+X^\mathsf{T}\beta|\le v_0. E γ + X T β [ g γ ( Y ) ] ≥ 4 c g ( γ − γ + X T β ) 2 , ∣ γ − γ + X T β ∣ ≤ v 0 . We can therefore bound
μ ( γ ^ ) : = E X E γ + X T β [ g γ ^ ( Y ) ] ≥ c g 4 E X [ ( γ − γ ^ + X T β ) 2 1 { ∣ γ − γ ^ + X T β ∣ ≤ v 0 } ] − B g P ( ∣ γ − γ ^ + X T β ∣ > v 0 ) (75) \begin{aligned}
\mu(\widehat{\gamma})&:=\mathbb{E}_X\mathbb{E}_{\gamma+X^\mathsf{T}\beta}[g_{\widehat{\gamma}}(Y)]\\
&\ge\frac{c_g}{4}\mathbb{E}_X\left[(\gamma-\widehat{\gamma}+X^\mathsf{T}\beta)^2\mathbf{1}\{|\gamma-\widehat{\gamma}+X^\mathsf{T}\beta|\le v_0\}\right]-B_g\mathbb{P}(|\gamma-\widehat{\gamma}+X^\mathsf{T}\beta|>v_0) \tag*{(75)}
\end{aligned} μ ( γ ) := E X E γ + X T β [ g γ ( Y )] ≥ 4 c g E X [ ( γ − γ + X T β ) 2 1 { ∣ γ − γ + X T β ∣ ≤ v 0 } ] − B g P ( ∣ γ − γ + X T β ∣ > v 0 ) (75) where the inequality is due to the fact that ∣ g γ ^ ( y ) ∣ ≤ B g |g_{\widehat{\gamma}}(y)|\le B_g ∣ g γ ( y ) ∣ ≤ B g
We have that
E X [ ( γ − γ ^ + X T β ) 2 ] = ( γ − γ ^ ) 2 + β T E [ X X T ] β ≥ λ ∥ β ∥ 2 2 , \mathbb{E}_X\left[(\gamma-\widehat{\gamma}+X^\mathsf{T}\beta)^2\right]=(\gamma-\widehat{\gamma})^2+\beta^\mathsf{T}\mathbb{E}[XX^\mathsf{T}]\beta\ge\lambda\|\beta\|_2^2, E X [ ( γ − γ + X T β ) 2 ] = ( γ − γ ) 2 + β T E [ X X T ] β ≥ λ ∥ β ∥ 2 2 , and
P X ( ∣ γ − γ ^ + X T β ∣ > v 0 ) ≤ P ( ∣ X T β ∣ > v 0 / 2 ) ≲ exp ( − c / ∥ β ∥ 2 2 ) . \mathbb{P}_X(|\gamma-\widehat{\gamma}+X^\mathsf{T}\beta|>v_0)\le\mathbb{P}(|X^\mathsf{T}\beta|>v_0/2)\lesssim\exp(-c/\|\beta\|_2^2). P X ( ∣ γ − γ + X T β ∣ > v 0 ) ≤ P ( ∣ X T β ∣ > v 0 /2 ) ≲ exp ( − c /∥ β ∥ 2 2 ) . Therefore, applying Cauchy–Schwarz to bound
E X [ ( γ − γ ^ + X T β ) 2 1 { ∣ γ − γ ^ + X T β ∣ > v 0 } ] ≲ ∥ β ∥ 2 2 exp ( − c / ∥ β ∥ 2 2 ) \mathbb{E}_X\left[(\gamma-\widehat{\gamma}+X^\mathsf{T}\beta)^2\mathbf{1}\{|\gamma-\widehat{\gamma}+X^\mathsf{T}\beta|>v_0\}\right]\lesssim\|\beta\|_2^2\exp(-c/\|\beta\|_2^2) E X [ ( γ − γ + X T β ) 2 1 { ∣ γ − γ + X T β ∣ > v 0 } ] ≲ ∥ β ∥ 2 2 exp ( − c /∥ β ∥ 2 2 ) and substituting into (75) , we obtain
μ ( γ ^ ) ≥ c g 4 { λ ∥ β ∥ 2 2 − C ∥ β ∥ 2 2 exp ( − c / ∥ β ∥ 2 2 ) } − C B g exp ( − c / ∥ β ∥ 2 2 ) ≳ ∥ β ∥ 2 2 . \mu(\widehat{\gamma}) \ge\frac{c_g}{4}\left\{\lambda\|\beta\|_2^2-C\|\beta\|_2^2\exp\left(-c/\|\beta\|_2^2\right)\right\}-CB_g\exp\left(-c/\|\beta\|_2^2\right) \gtrsim\|\beta\|_2^2. μ ( γ ) ≥ 4 c g { λ ∥ β ∥ 2 2 − C ∥ β ∥ 2 2 exp ( − c /∥ β ∥ 2 2 ) } − C B g exp ( − c /∥ β ∥ 2 2 ) ≳ ∥ β ∥ 2 2 . Since ∥ β ∥ 2 2 ≥ C 1 a n ( ϵ , η ) \|\beta\|_2^2 \ge C_1a_n(\epsilon,\eta) ∥ β ∥ 2 2 ≥ C 1 a n ( ϵ , η ) C 1 C_1 C 1 μ ( γ ^ ) − C μ a n ( ϵ , η ) ≥ c T a n ( ϵ , η ) \mu(\widehat{\gamma})-C\mu a_n(\epsilon,\eta)\ge c_Ta_n(\epsilon,\eta) μ ( γ ) − C μ a n ( ϵ , η ) ≥ c T a n ( ϵ , η ) (73) together with the probability bound 1 − η / 2 1-\eta/2 1 − η /2 (74) gives P β , γ ( T ~ ≤ c T a n ) ≤ η \mathbb{P}_{\beta,\gamma}(\widetilde{T}\le c_Ta_n)\le\eta P β , γ ( T ≤ c T a n ) ≤ η
Lemma 28. Fix a ∈ Γ 1 a\in\Gamma_1 a ∈ Γ 1 Γ 1 \Gamma_1 Γ 1 Lemma 16 , and let g a g_a g a v 0 , C > 0 v_0,C>0 v 0 , C > 0 a ∈ Γ 1 a\in\Gamma_1 a ∈ Γ 1 ∣ v ∣ ≤ v 0 |v|\le v_0 ∣ v ∣ ≤ v 0
∣ E a + v [ g a ( Y ) ] − v 2 2 E a [ g a ( Y ) H a ( Y ) ] ∣ ≤ C ∣ v ∣ 3 , E a [ g a ( Y ) H a ( Y ) ] ≥ c g > 0. \left|\mathbb{E}_{a+v}[g_a(Y)]-\frac{v^2}{2}\mathbb{E}_a[g_a(Y)H_a(Y)]\right|\le C|v|^3,\qquad\mathbb{E}_a[g_a(Y)H_a(Y)]\ge c_g>0. E a + v [ g a ( Y )] − 2 v 2 E a [ g a ( Y ) H a ( Y )] ≤ C ∣ v ∣ 3 , E a [ g a ( Y ) H a ( Y )] ≥ c g > 0. Proof. The uniformly bounded third L 2 ( P a ) L_2(P_a) L 2 ( P a ) Lemma 16 gives
L a , v ( Y ) : = d P a + v d P a = 1 + v S a + v 2 2 H a + R a , v , ∥ R a , v ∥ L 2 ( P a ) ≤ C ∣ v ∣ 3 L_{a,v}(Y):=\frac{\mathrm{d}P_{a+v}}{\mathrm{d}P_a}=1+vS_a+\frac{v^2}{2}H_a+R_{a,v},\qquad\|R_{a,v}\|_{L_2(P_a)}\le C|v|^3 L a , v ( Y ) := d P a d P a + v = 1 + v S a + 2 v 2 H a + R a , v , ∥ R a , v ∥ L 2 ( P a ) ≤ C ∣ v ∣ 3 on a common neighbourhood of zero. Since E a + v [ g a ( Y ) ] = E a [ g a L a , v ] \mathbb{E}_{a+v}[g_a(Y)]=\mathbb{E}_a[g_aL_{a,v}] E a + v [ g a ( Y )] = E a [ g a L a , v ] (18) and (19) give
E a + v [ g a ] = v 2 2 E a [ g a H a ] + E a [ g a R a , v ] , \mathbb{E}_{a+v}[g_a]=\frac{v^2}{2}\mathbb{E}_a[g_aH_a]+\mathbb{E}_a[g_aR_{a,v}], E a + v [ g a ] = 2 v 2 E a [ g a H a ] + E a [ g a R a , v ] , where by (21) , we have ∣ E a ( g a R a , v ) ∣ ≤ B g ∥ R a , v ∥ L 2 ( P a ) ≤ C B g ∣ v ∣ 3 |\mathbb{E}_a(g_aR_{a,v})|\le B_g\|R_{a,v}\|_{L_2(P_a)}\le CB_g|v|^3 ∣ E a ( g a R a , v ) ∣ ≤ B g ∥ R a , v ∥ L 2 ( P a ) ≤ C B g ∣ v ∣ 3
Monte Carlo approximations
We will show that the inversion step in Algorithm 1 and the relevant g a g_a g a
Fix a grid A n ⊂ Γ 1 \mathcal{A}_n\subset\Gamma_1 A n ⊂ Γ 1 n − 1 / 2 n^{-1/2} n − 1/2 Γ 1 \Gamma_1 Γ 1 O ( n ) O(\sqrt{n}) O ( n ) a ∈ A n a\in\mathcal{A}_n a ∈ A n N = ⌈ C n ∣ A n ∣ / η ⌉ N=\lceil Cn|\mathcal{A}_n|/\eta\rceil N = ⌈ C n ∣ A n ∣/ η ⌉ V a , 1 , … , V a , N ∼ P a V_{a,1},\ldots,V_{a,N}\sim P_a V a , 1 , … , V a , N ∼ P a
m ^ ( a ) = 1 N ∑ ℓ = 1 N tanh ( V a , ℓ ) , d ^ ( a ) = 1 N ∑ ℓ = 1 N tanh ( V a , ℓ ) S a ( V a , ℓ ) , \widehat{m}(a)=\frac{1}{N}\sum_{\ell=1}^{N}\tanh(V_{a,\ell}),\qquad\widehat{d}(a)=\frac{1}{N}\sum_{\ell=1}^{N}\tanh(V_{a,\ell})S_a(V_{a,\ell}), m ( a ) = N 1 ℓ = 1 ∑ N tanh ( V a , ℓ ) , d ( a ) = N 1 ℓ = 1 ∑ N tanh ( V a , ℓ ) S a ( V a , ℓ ) , c ^ ( a ) = 1 N ∑ ℓ = 1 N A a , M ( V a , ℓ ) , q ^ ( a ) = 1 N ∑ ℓ = 1 N A a , M ( V a , ℓ ) S a ( V a , ℓ ) , \widehat{c}(a)=\frac{1}{N}\sum_{\ell=1}^{N}A_{a,M}(V_{a,\ell}),\qquad\widehat{q}(a)=\frac{1}{N}\sum_{\ell=1}^{N}A_{a,M}(V_{a,\ell})S_a(V_{a,\ell}), c ( a ) = N 1 ℓ = 1 ∑ N A a , M ( V a , ℓ ) , q ( a ) = N 1 ℓ = 1 ∑ N A a , M ( V a , ℓ ) S a ( V a , ℓ ) , where the draws are independent between the different a a a m m m d d d c c c q q q
γ ^ M C ∈ arg min a ∈ A n ∣ m ^ ( a ) − m ~ ∣ , \widehat{\gamma}_{\mathrm{MC}}\in\arg\min_{a\in\mathcal{A}_n}|\widehat{m}(a)-\widetilde{m}|, γ MC ∈ arg a ∈ A n min ∣ m ( a ) − m ∣ , recalling the definition of m ~ \widetilde{m} m (22) . Further define
g ^ γ ^ M C ( y ) = [ A γ ^ M C , M ( y ) − c ^ ( γ ^ M C ) − q ^ ( γ ^ M C ) d ^ ( γ ^ M C ) { tanh ( y ) − m ^ ( γ ^ M C ) } ] − B g B g . \widehat{g}_{\widehat{\gamma}_{\mathrm{MC}}}(y)=\left[A_{\widehat{\gamma}_{\mathrm{MC}},M}(y)-\widehat{c}(\widehat{\gamma}_{\mathrm{MC}})-\frac{\widehat{q}(\widehat{\gamma}_{\mathrm{MC}})}{\widehat{d}(\widehat{\gamma}_{\mathrm{MC}})}\{\tanh(y)-\widehat{m}(\widehat{\gamma}_{\mathrm{MC}})\}\right]_{-B_g}^{B_g}. g γ MC ( y ) = [ A γ MC , M ( y ) − c ( γ MC ) − d ( γ MC ) q ( γ MC ) { tanh ( y ) − m ( γ MC )} ] − B g B g . if ∣ d ^ ( γ M C ) ∣ ≥ c m / 2 |\widehat{d}(\gamma_{\mathrm{MC}})| \ge c_m/2 ∣ d ( γ MC ) ∣ ≥ c m /2 c m c_m c m Lemma 22 , and set g ^ γ M C ≡ 0 \widehat{g}_{\gamma_{\mathrm{MC}}} \equiv0 g γ MC ≡ 0
Lemma 29. Suppose that independent draws from P a P_a P a S a ( y ) S_a(y) S a ( y ) R a ( y ) R_a(y) R a ( y ) a ∈ Γ 1 a \in\Gamma_1 a ∈ Γ 1 y ∈ Y y \in\mathcal{Y} y ∈ Y γ ^ M C \widehat{\gamma}_{\mathrm{MC}} γ MC g ^ γ M C \widehat{g}_{\gamma_{\mathrm{MC}}} g γ MC γ ^ \widehat{\gamma} γ g ^ γ \widehat{g}_{\gamma} g γ Proposition 11 and has a runtime of O ( n 2 / η ) O(n^2/\eta) O ( n 2 / η )
Proof. For the privacy guarantee, we may condition on the simulation draws, which are independent of the data, so that the release of γ ^ M C \widehat{\gamma}_{\mathrm{MC}} γ MC m ~ \widetilde{m} m B g B_g B g g ^ \widehat{g} g A n \mathcal{A}_n A n O ( ∣ A n ∣ N ) = O ( n 2 / η ) O(|\mathcal{A}_n|N)=O(n^2/\eta) O ( ∣ A n ∣ N ) = O ( n 2 / η ) O ( ∣ A n ∣ + n ) = O ( n ) O(|\mathcal{A}_n|+n)=O(n) O ( ∣ A n ∣ + n ) = O ( n )
Let G \mathcal{G} G
G : = { max a ∈ A n max h ∈ { m , d , c , q } ∣ h ^ ( a ) − h ( a ) ∣ ≤ c 0 n − 1 / 2 } , c 0 : = min { 1 , c m / 4 } . \mathcal{G}:=\left\{\max_{a\in\mathcal{A}_n}\max_{h\in\{m,d,c,q\}}|\widehat{h}(a)-h(a)|\le c_0n^{-1/2}\right\},\qquad c_0:=\min\{1,c_m/4\}. G := { a ∈ A n max h ∈ { m , d , c , q } max ∣ h ( a ) − h ( a ) ∣ ≤ c 0 n − 1/2 } , c 0 := min { 1 , c m /4 } . Since tanh \tanh tanh A a , M A_{a,M} A a , M sup a ∈ Γ 1 E a [ S a ( Y ) 2 ] < ∞ \sup_{a\in\Gamma_1}\mathbb{E}_a[S_a(Y)^2]<\infty sup a ∈ Γ 1 E a [ S a ( Y ) 2 ] < ∞ Y ∼ P a Y\sim P_a Y ∼ P a a ∈ Γ 1 a\in\Gamma_1 a ∈ Γ 1 C C C N N N P ( G ) ≥ 1 − η / 4 \mathbb{P}(\mathcal{G})\ge1-\eta/4 P ( G ) ≥ 1 − η /4
We work on G \mathcal{G} G γ ~ \widetilde{\gamma} γ (23) , and choose a γ ^ ∈ A n a_{\widehat{\gamma}}\in\mathcal{A}_n a γ ∈ A n ∣ a γ ^ − γ ~ ∣ ≤ n − 1 / 2 |a_{\widehat{\gamma}}-\widetilde{\gamma}|\le n^{-1/2} ∣ a γ − γ ∣ ≤ n − 1/2 m ( γ ^ ) m(\widehat{\gamma}) m ( γ ) m ~ \widetilde{m} m m ( Γ 1 ) m(\Gamma_1) m ( Γ 1 ) 0 < c m ≤ ∣ m ′ ∣ ≤ C m < ∞ 0<c_m\le|m'|\le C_m<\infty 0 < c m ≤ ∣ m ′ ∣ ≤ C m < ∞ Γ 1 \Gamma_1 Γ 1 γ M C \gamma_{\mathrm{MC}} γ MC
∣ γ ^ M C − γ ^ ∣ ≤ 1 c m { ∣ m ( γ ^ M C ) − m ~ ∣ − ∣ m ~ − m ( γ ^ ) ∣ } ≤ 1 c m { ∣ m ( γ ^ M C ) − m ^ ( γ ^ M C ) ∣ + ∣ m ^ ( γ ^ M C ) − m ~ ∣ − ∣ m ~ − m ( γ ^ ) ∣ } ≤ 1 c m { sup u ∈ Γ 1 ∣ m ′ ( u ) ∣ + 2 c 0 } n − 1 / 2 ≲ n − 1 / 2 . \begin{aligned}
|\widehat{\gamma}_{\mathrm{MC}}-\widehat{\gamma}|\le\frac{1}{c_m}\{|m(\widehat{\gamma}_{\mathrm{MC}})-\widetilde{m}|-|\widetilde{m}-m(\widehat{\gamma})|\} \\
&\le\frac{1}{c_m}\{|m(\widehat{\gamma}_{\mathrm{MC}})-\widehat{m}(\widehat{\gamma}_{\mathrm{MC}})|+|\widehat{m}(\widehat{\gamma}_{\mathrm{MC}})-\widetilde{m}|-|\widetilde{m}-m(\widehat{\gamma})|\} \\
&\le\frac{1}{c_m}\left\{\sup_{u\in\Gamma_1}|m'(u)|+2c_0\right\}n^{-1/2}\lesssim n^{-1/2}.
\end{aligned} ∣ γ MC − γ ∣ ≤ c m 1 { ∣ m ( γ MC ) − m ∣ − ∣ m − m ( γ ) ∣ } ≤ c m 1 { ∣ m ( γ MC ) − m ( γ MC ) ∣ + ∣ m ( γ MC ) − m ∣ − ∣ m − m ( γ ) ∣ } ≤ c m 1 { u ∈ Γ 1 sup ∣ m ′ ( u ) ∣ + 2 c 0 } n − 1/2 ≲ n − 1/2 . Let a ∈ A n a\in\mathcal{A}_n a ∈ A n y ∈ Y y\in\mathcal{Y} y ∈ Y
∣ A a , M ( y ) − c ^ ( a ) − q ^ ( a ) d ^ ( a ) { tanh ( y ) − m ^ ( a ) } − g a ( y ) ∣ ≤ ∣ c ( a ) − c ^ ( a ) ∣ + ∣ { q ^ ( a ) d ^ ( a ) − q ( a ) d ( a ) } { tanh ( y ) − m ^ ( a ) } ∣ + ∣ q ( a ) d ( a ) { m ^ ( a ) − m ( a ) } ∣ . \begin{aligned}
\left|A_{a,M}(y)-\widehat{c}(a)-\frac{\widehat{q}(a)}{\widehat{d}(a)}\{\tanh(y)-\widehat{m}(a)\}-g_a(y)\right| \\
&\le|c(a)-\widehat{c}(a)|+\left|\left\{\frac{\widehat{q}(a)}{\widehat{d}(a)}-\frac{q(a)}{d(a)}\right\}\{\tanh(y)-\widehat{m}(a)\}\right|+\left|\frac{q(a)}{d(a)}\{\widehat{m}(a)-m(a)\}\right|.
\end{aligned} A a , M ( y ) − c ( a ) − d ( a ) q ( a ) { tanh ( y ) − m ( a )} − g a ( y ) ≤ ∣ c ( a ) − c ( a ) ∣ + { d ( a ) q ( a ) − d ( a ) q ( a ) } { tanh ( y ) − m ( a )} + d ( a ) q ( a ) { m ( a ) − m ( a )} . On G \mathcal{G} G n − 1 / 2 n^{-1/2} n − 1/2
∣ q ^ ( a ) d ^ ( a ) − q ( a ) d ( a ) ∣ ≤ ∣ q ^ ( a ) − q ( a ) d ^ ( a ) ∣ + ∣ q ( a ) { d ( a ) − d ^ ( a ) } d ^ ( a ) d ( a ) ∣ ≤ 4 3 c m ∣ q ^ ( a ) − q ( a ) ∣ + 4 ∣ q ( a ) ∣ 3 c m 2 ∣ d ^ ( a ) − d ( a ) ∣ ≲ n − 1 / 2 . \begin{aligned}
\left|\frac{\widehat{q}(a)}{\widehat{d}(a)}-\frac{q(a)}{d(a)}\right|\le\left|\frac{\widehat{q}(a)-q(a)}{\widehat{d}(a)}\right|+\left|\frac{q(a)\{d(a)-\widehat{d}(a)\}}{\widehat{d}(a)d(a)}\right| \\
&\le\frac{4}{3c_m}|\widehat{q}(a)-q(a)|+\frac{4|q(a)|}{3c_m^2}|\widehat{d}(a)-d(a)|\lesssim n^{-1/2}.
\end{aligned} d ( a ) q ( a ) − d ( a ) q ( a ) ≤ d ( a ) q ( a ) − q ( a ) + d ( a ) d ( a ) q ( a ) { d ( a ) − d ( a )} ≤ 3 c m 4 ∣ q ( a ) − q ( a ) ∣ + 3 c m 2 4∣ q ( a ) ∣ ∣ d ( a ) − d ( a ) ∣ ≲ n − 1/2 . where the second inequality is due to
∣ d ^ ( a ) ∣ ≥ ∣ d ( a ) ∣ − ∣ d ^ ( a ) − d ( a ) ∣ ≥ c m − c 0 n − 1 / 2 ≥ 3 c m / 4. |\widehat{d}(a)| \ge|d(a)| - |\widehat{d}(a)-d(a)| \ge c_m-c_0n^{-1/2} \ge3c_m/4. ∣ d ( a ) ∣ ≥ ∣ d ( a ) ∣ − ∣ d ( a ) − d ( a ) ∣ ≥ c m − c 0 n − 1/2 ≥ 3 c m /4. In particular, the bounds hold for a = γ ^ M C a=\widehat{\gamma}_{\mathrm{MC}} a = γ MC g a ( y ) ∈ [ − B g , B g ] g_a(y)\in[-B_g,B_g] g a ( y ) ∈ [ − B g , B g ]
sup y ∈ Y ∣ g ^ γ ^ M C ( y ) − g γ ^ M C ( y ) ∣ ≲ n − 1 / 2 . (76) \sup_{y\in\mathcal{Y}}|\widehat{g}_{\widehat{\gamma}_{\mathrm{MC}}}(y)-g_{\widehat{\gamma}_{\mathrm{MC}}}(y)|\lesssim n^{-1/2}. \tag*{(76)} y ∈ Y sup ∣ g γ MC ( y ) − g γ MC ( y ) ∣ ≲ n − 1/2 . (76) By a union bound considering the event G \mathcal{G} G Lemma 27 , we have that
∣ γ ^ M C − γ ∣ ≤ ∣ γ ^ M C − γ ^ ∣ + ∣ γ ^ − γ ∣ |\widehat{\gamma}_{\mathrm{MC}}-\gamma|\leq|\widehat{\gamma}_{\mathrm{MC}}-\widehat{\gamma}|+|\widehat{\gamma}-\gamma| ∣ γ MC − γ ∣ ≤ ∣ γ MC − γ ∣ + ∣ γ − γ ∣ ≲ n − 1 / 2 + ∥ β ∥ 2 2 + a n ( ϵ , η / 2 ) ≲ ∥ β ∥ 2 2 + a n ( ϵ , η ) \lesssim n^{-1/2}+\|\beta\|_2^2+a_n(\epsilon,\eta/2)\lesssim\|\beta\|_2^2+a_n(\epsilon,\eta) ≲ n − 1/2 + ∥ β ∥ 2 2 + a n ( ϵ , η /2 ) ≲ ∥ β ∥ 2 2 + a n ( ϵ , η ) with probability at least 1 − η / 2 1-\eta/2 1 − η /2 (76) , we can apply the same concentration argument in (73) . Following the remaining arguments for the type-I and type-II error bounds in the proof of Proposition 11 gives the separation rate, with constants adjusted.
Technical details of the exponential regression test
In this subsection, we give a generic U U U Lemma 31 , we show that the kernel construction in Remark 1 satisfies the requirements needed for this test.
Proposition 30. Let h h h H H H c > 0 c>0 c > 0
E 0 [ h ( Y 1 , Y 2 ) ] = 0 and E β [ h ( Y 1 , Y 2 ) ] ≥ c min { ∥ β ∥ 2 2 , 1 } (77) \mathbb{E}_0[h(Y_1,Y_2)]=0\quad\text{and}\quad\mathbb{E}_\beta[h(Y_1,Y_2)]\geq c\min\{\|\beta\|_2^2,1\} \tag*{(77)} E 0 [ h ( Y 1 , Y 2 )] = 0 and E β [ h ( Y 1 , Y 2 )] ≥ c min { ∥ β ∥ 2 2 , 1 } (77) uniformly over the nuisance class. Define U : = ( n 2 ) − 1 ∑ i < j h ( Y i , Y j ) U:=\binom{n}{2}^{-1}\sum_{i<j}h(Y_i,Y_j) U := ( 2 n ) − 1 ∑ i < j h ( Y i , Y j )
ϕ : = 1 { U + 2 H n ϵ W ≥ H 2 log ( 2 / η ) n + 2 H log ( 1 / η ) n ϵ } , W ∼ Lap ( 1 ) (78) \phi:=\mathbf{1}\left\{U+\frac{2H}{n\epsilon}W\geq H\sqrt{\frac{2\log(2/\eta)}{n}}+\frac{2H\log(1/\eta)}{n\epsilon}\right\},\qquad W\sim\operatorname{Lap}(1) \tag*{(78)} ϕ := 1 { U + n ϵ 2 H W ≥ H n 2 log ( 2/ η ) + n ϵ 2 H log ( 1/ η ) } , W ∼ Lap ( 1 ) (78) is ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 )
c min { ∥ β ∥ 2 2 , 1 } ≥ 4 H { log ( 1 / η ) n + log ( 1 / η ) n ϵ } , c\min\{\|\beta\|_2^2,1\}\geq4H\left\{\sqrt{\frac{\log(1/\eta)}{n}}+\frac{\log(1/\eta)}{n\epsilon}\right\}, c min { ∥ β ∥ 2 2 , 1 } ≥ 4 H { n log ( 1/ η ) + n ϵ log ( 1/ η ) } , then the sum of the type-I and type-II errors is upper-bounded by 2 η 2\eta 2 η
Proof. Privacy. If two datasets differ in one observation, then at most n − 1 n-1 n − 1 U U U h h h H H H H H H U U U 2 H / n 2H/n 2 H / n ϕ \phi ϕ ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 )
Type-I error. We next control the type-I error. Changing one observation changes U U U 2 H / n 2H/n 2 H / n
P ( U − E [ U ] ≥ t ) ≤ exp ( − n t 2 2 H 2 ) . \mathbb{P}\left(U-\mathbb{E}[U]\ge t\right)\le\exp\left(-\frac{nt^2}{2H^2}\right). P ( U − E [ U ] ≥ t ) ≤ exp ( − 2 H 2 n t 2 ) . Under the null, E [ U ] = 0 \mathbb{E}[U]=0 E [ U ] = 0
P ( U ≥ H 2 log ( 2 / η ) n ) ≤ η 2 . \mathbb{P}\left(U\ge H\sqrt{\frac{2\log(2/\eta)}{n}}\right)\le\frac{\eta}{2}. P ( U ≥ H n 2 log ( 2/ η ) ) ≤ 2 η . Moreover, since W ∼ Lap ( 1 ) W\sim\operatorname{Lap}(1) W ∼ Lap ( 1 )
P ( 2 H n ϵ W ≥ 2 H log ( 1 / η ) n ϵ ) = η 2 . \mathbb{P}\left(\frac{2H}{n\epsilon}W\ge\frac{2H\log(1/\eta)}{n\epsilon}\right)=\frac{\eta}{2}. P ( n ϵ 2 H W ≥ n ϵ 2 H log ( 1/ η ) ) = 2 η . A union bound therefore gives P ( ϕ = 1 ) ≤ η \mathbb{P}(\phi=1)\le\eta P ( ϕ = 1 ) ≤ η
Type-II error. Applying McDiarmid’s inequality to the lower tail gives
P ( U − E [ U ] ≤ − H 2 log ( 2 / η ) n ) ≤ η 2 . \mathbb{P}\left(U-\mathbb{E}[U]\le-H\sqrt{\frac{2\log(2/\eta)}{n}}\right)\le\frac{\eta}{2}. P ( U − E [ U ] ≤ − H n 2 log ( 2/ η ) ) ≤ 2 η . Combining the above with the Laplace tail bound, we have that the type-II error is at most η \eta η
E [ U ] ≥ 2 H 2 log ( 2 / η ) n + 4 H log ( 1 / η ) n ϵ , \mathbb{E}[U]\ge2H\sqrt{\frac{2\log(2/\eta)}{n}}+\frac{4H\log(1/\eta)}{n\epsilon}, E [ U ] ≥ 2 H n 2 log ( 2/ η ) + n ϵ 4 H log ( 1/ η ) , and recalling the assumption in (77) concludes the proof.
Given Proposition 30 , it suffices to show that the kernel in (26) satisfies the properties in (77) for the exponential regression model.
Lemma 31. Suppose that ( X 1 , Y 1 ) (X_1,Y_1) ( X 1 , Y 1 ) and ( X 2 , Y 2 ) (X_2,Y_2) ( X 2 , Y 2 ) are independent copies of ( X , Y ) (X,Y) ( X , Y ) where X ∼ P X X\sim P_X X ∼ P X for some P X P_X P X satisfying Assumption 1 and
Y ∣ X = x ∼ Exp ( exp { γ + x T β } ) , Y\mid X=x\sim\operatorname{Exp}\left(\exp\{\gamma+x^{\mathsf{T}}\beta\}\right), Y ∣ X = x ∼ Exp ( exp { γ + x T β } ) , where Exp ( λ ) \operatorname{Exp}(\lambda) Exp ( λ ) denotes the exponential distribution with rate λ \lambda λ Recall that
h Exp ( y 1 , y 2 ) : = 1 { max ( y 1 , y 2 ) > 2 min ( y 1 , y 2 ) } − 2 3 . h_{\operatorname{Exp}}(y_1,y_2):=\mathbf{1}\{\max(y_1,y_2)>2\min(y_1,y_2)\}-\frac{2}{3}. h Exp ( y 1 , y 2 ) := 1 { max ( y 1 , y 2 ) > 2 min ( y 1 , y 2 )} − 3 2 . Then, for all γ ∈ R \gamma\in\mathbb{R} γ ∈ R we have that
E 0 , γ [ h Exp ( Y 1 , Y 2 ) ] = 0. (79) \mathbb{E}_{0,\gamma}[h_{\operatorname{Exp}}(Y_1,Y_2)]=0. \tag*{(79)} E 0 , γ [ h Exp ( Y 1 , Y 2 )] = 0. (79) and
E β , γ [ h Exp ( Y 1 , Y 2 ) ] ≳ min { ∥ β ∥ 2 2 , 1 } . (80) \mathbb{E}_{\beta,\gamma}[h_{\operatorname{Exp}}(Y_1,Y_2)]\gtrsim\min\{\|\beta\|_2^2,1\}. \tag*{(80)} E β , γ [ h Exp ( Y 1 , Y 2 )] ≳ min { ∥ β ∥ 2 2 , 1 } . (80) Proof. Write λ i : = exp { γ + X i T β } \lambda_i:=\exp\{\gamma+X_i^{\mathsf{T}}\beta\} λ i := exp { γ + X i T β } i = 1 , 2 i=1,2 i = 1 , 2 ( X 1 , X 2 ) (X_1,X_2) ( X 1 , X 2 )
Pr ( Y 1 > 2 Y 2 ∣ X 1 , X 2 ) = ∫ 0 ∞ exp ( − 2 λ 1 y ) λ 2 exp ( − λ 2 y ) d y = λ 2 2 λ 1 + λ 2 . \Pr(Y_1 > 2Y_2 \mid X_1,X_2) = \int_{0}^{\infty} \exp(-2\lambda_1 y)\lambda_2\exp(-\lambda_2 y)\,\mathrm{d}y = \frac{\lambda_2}{2\lambda_1+\lambda_2}. Pr ( Y 1 > 2 Y 2 ∣ X 1 , X 2 ) = ∫ 0 ∞ exp ( − 2 λ 1 y ) λ 2 exp ( − λ 2 y ) d y = 2 λ 1 + λ 2 λ 2 . Since the events { Y 1 > 2 Y 2 } \{Y_1 > 2Y_2\} { Y 1 > 2 Y 2 } { Y 2 > 2 Y 1 } \{Y_2 > 2Y_1\} { Y 2 > 2 Y 1 } u : = β T ( X 1 − X 2 ) u := \beta^{\mathsf{T}}(X_1-X_2) u := β T ( X 1 − X 2 )
E [ h Exp ( Y 1 , Y 2 ) ∣ X 1 , X 2 ] = 1 1 + 2 e u + e u e u + 2 − 2 3 = 2 ( e u − 1 ) 2 3 ( 1 + 2 e u ) ( e u + 2 ) = 2 e u ( e u + e − u − 2 ) 3 e u ( 2 e u + 5 + 2 e − u ) = 4 ( cosh ( u ) − 1 ) 3 ( 4 cosh ( u ) + 5 ) = : f ( u ) . \begin{aligned}
\mathbb{E}[h_{\operatorname{Exp}}(Y_1,Y_2)\mid X_1,X_2] = \frac{1}{1+2e^u}+\frac{e^u}{e^u+2}-\frac{2}{3} \\
&= \frac{2(e^u-1)^2}{3(1+2e^u)(e^u+2)}=\frac{2e^u(e^u+e^{-u}-2)}{3e^u(2e^u+5+2e^{-u})} \\
&= \frac{4(\cosh(u)-1)}{3(4\cosh(u)+5)}=:f(u).
\end{aligned} E [ h Exp ( Y 1 , Y 2 ) ∣ X 1 , X 2 ] = 1 + 2 e u 1 + e u + 2 e u − 3 2 = 3 ( 1 + 2 e u ) ( e u + 2 ) 2 ( e u − 1 ) 2 = 3 e u ( 2 e u + 5 + 2 e − u ) 2 e u ( e u + e − u − 2 ) = 3 ( 4 cosh ( u ) + 5 ) 4 ( cosh ( u ) − 1 ) =: f ( u ) . Under the null, u = 0 u=0 u = 0 f ( 0 ) = 0 f(0)=0 f ( 0 ) = 0 (79) .
Since the map x ↦ 4 x / ( 27 + 12 x ) x\mapsto4x/(27+12x) x ↦ 4 x / ( 27 + 12 x ) [ 0 , ∞ ) [0,\infty) [ 0 , ∞ ) cosh ( u ) − 1 ≥ u 2 / 2 \cosh(u)-1\ge u^2/2 cosh ( u ) − 1 ≥ u 2 /2
f ( u ) = 4 ( cosh ( u ) − 1 ) 27 + 12 ( cosh ( u ) − 1 ) ≥ 2 u 2 27 + 6 u 2 . f(u)=\frac{4(\cosh(u)-1)}{27+12(\cosh(u)-1)}\ge\frac{2u^2}{27+6u^2}. f ( u ) = 27 + 12 ( cosh ( u ) − 1 ) 4 ( cosh ( u ) − 1 ) ≥ 27 + 6 u 2 2 u 2 . By the Cauchy–Schwarz inequality, we have that
( E [ u 2 ] ) 2 ≤ E [ u 2 27 + 6 u 2 ] E [ u 2 ( 27 + 6 u 2 ) ] . \left(\mathbb{E}[u^2]\right)^2\le\mathbb{E}\left[\frac{u^2}{27+6u^2}\right]\mathbb{E}\left[u^2(27+6u^2)\right]. ( E [ u 2 ] ) 2 ≤ E [ 27 + 6 u 2 u 2 ] E [ u 2 ( 27 + 6 u 2 ) ] . For β \beta β ∥ β ∥ 2 > 0 \lVert\beta\rVert_2>0 ∥ β ∥ 2 > 0 2 λ ∥ β ∥ 2 2 ≤ E [ u 2 ] ≤ 2 λ ˉ ∥ β ∥ 2 2 2\lambda\lVert\beta\rVert_2^2\le\mathbb{E}[u^2]\le2\bar{\lambda}\lVert\beta\rVert_2^2 2 λ ∥ β ∥ 2 2 ≤ E [ u 2 ] ≤ 2 λ ˉ ∥ β ∥ 2 2 E [ u 4 ] ≲ C X 4 ∥ β ∥ 2 4 \mathbb{E}[u^4]\lesssim C_X^4\lVert\beta\rVert_2^4 E [ u 4 ] ≲ C X 4 ∥ β ∥ 2 4
E β , γ [ h Exp ( Y 1 , Y 2 ) ] = E [ f ( u ) ] ≥ 2 ( E [ u 2 ] ) 2 27 E [ u 2 ] + 6 E [ u 4 ] \mathbb{E}_{\beta,\gamma}[h_{\operatorname{Exp}}(Y_1,Y_2)]=\mathbb{E}[f(u)]\ge\frac{2(\mathbb{E}[u^2])^2}{27\mathbb{E}[u^2]+6\mathbb{E}[u^4]} E β , γ [ h Exp ( Y 1 , Y 2 )] = E [ f ( u )] ≥ 27 E [ u 2 ] + 6 E [ u 4 ] 2 ( E [ u 2 ] ) 2 ≳ ∥ β ∥ 2 2 1 + ∥ β ∥ 2 2 ≳ min { ∥ β ∥ 2 2 , 1 } , \gtrsim\frac{\lVert\beta\rVert_2^2}{1+\lVert\beta\rVert_2^2}\gtrsim\min\{\lVert\beta\rVert_2^2,1\}, ≳ 1 + ∥ β ∥ 2 2 ∥ β ∥ 2 2 ≳ min {∥ β ∥ 2 2 , 1 } , which proves (80) .
Proof of the upper bounds in Theorem 3
Proof. Run the multiscale mean test and sparse covariance test, together with the response-only test for GLMs with more than two response support points, using independent randomisation, privacy budget ϵ / 3 \epsilon/3 ϵ /3 η = 1 / 12 \eta=1/12 η = 1/12 Lemma 39 , the resulting test is ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) 1 / 4 1/4 1/4 1 / 12 1/12 1/12
For the mean test, choose
r ~ = c ℓ n , d , ϵ 3 { d 1 / 4 n + d 1 / 4 n 3 / 4 ϵ 1 / 2 + 1 n ϵ } . \tilde r=c\ell_{n,d,\epsilon}^3\left\{\frac{d^{1/4}}{\sqrt n}+\frac{d^{1/4}}{n^{3/4}\epsilon^{1/2}}+\frac{1}{n\epsilon}\right\}. r ~ = c ℓ n , d , ϵ 3 { n d 1/4 + n 3/4 ϵ 1/2 d 1/4 + n ϵ 1 } . with sufficiently large fixed constant c > 0 c > 0 c > 0 r ~ ≤ min { c μ , λ } \tilde{r} \le\min\{c\mu, \sqrt{\lambda}\} r ~ ≤ min { c μ , λ } r ~ ≳ n − 1 / 2 \tilde{r} \gtrsim n^{-1/2} r ~ ≳ n − 1/2 J ≲ ℓ n , d , ϵ J \lesssim\ell_{n,d,\epsilon} J ≲ ℓ n , d , ϵ (15) , we have
ρ ( n , d , ϵ 3 J , 1 12 J ) ≲ ℓ n , d , ϵ d 1 / 4 n + ℓ n , d , ϵ 3 / 2 d 1 / 4 n 3 / 4 ϵ 1 / 2 + ℓ n , d , ϵ 3 n ϵ ≤ r ~ . \rho\left(n,d,\frac{\epsilon}{3J},\frac{1}{12J}\right) \lesssim\frac{\ell_{n,d,\epsilon}d^{1/4}}{\sqrt{n}}+\frac{\ell_{n,d,\epsilon}^{3/2}d^{1/4}}{n^{3/4}\epsilon^{1/2}}+\frac{\ell_{n,d,\epsilon}^{3}}{n\epsilon}\leq\tilde{r}. ρ ( n , d , 3 J ϵ , 12 J 1 ) ≲ n ℓ n , d , ϵ d 1/4 + n 3/4 ϵ 1/2 ℓ n , d , ϵ 3/2 d 1/4 + n ϵ ℓ n , d , ϵ 3 ≤ r ~ . By Lemmas 6 and 24 , this component detects alternatives with ∥ β ∥ 2 ≥ r ~ / c μ \lVert\beta\rVert_2 \geq\tilde{r}/c_\mu ∥ β ∥ 2 ≥ r ~ / c μ C { ℓ n , d , ϵ 1 / 2 s ∗ / n + ( n ϵ ) − 1 } C\{\ell_{n,d,\epsilon}^{1/2}\sqrt{s^*/n}+(n\epsilon)^{-1}\} C { ℓ n , d , ϵ 1/2 s ∗ / n + ( n ϵ ) − 1 } C ℓ n , d , ϵ 3 ϱ s ∗ , d C\ell_{n,d,\epsilon}^{3}\varrho_{s^*,d} C ℓ n , d , ϵ 3 ϱ s ∗ , d d ≤ n d \leq n d ≤ n
For larger-support GLMs, when d > n d > n d > n Proposition 11 gives the separation C { n − 1 / 4 + ( n ϵ ) − 1 / 2 } C\{n^{-1/4}+(n\epsilon)^{-1/2}\} C { n − 1/4 + ( n ϵ ) − 1/2 } ∥ β ∥ 2 ≤ r 0 \lVert\beta\rVert_2 \leq r_0 ∥ β ∥ 2 ≤ r 0 C ℓ n , d , ϵ 3 ϱ s ∗ , n C\ell_{n,d,\epsilon}^{3}\varrho_{s^*,n} C ℓ n , d , ϵ 3 ϱ s ∗ , n c P c_P c P r 0 r_0 r 0 1 / 3 1/3 1/3 s ∗ s^* s ∗
Technical details on the polynomial-time implementations
Appendix E.1 contains all the technical details on the privacy guarantees of the subroutine in Algorithm 2, and its error guarantees are proven in Appendix E.2. The proof of the polynomial-time sparse test from Proposition 13 is in Appendix E.3.
Local Hájek mean algorithm with explicit implementation
Algorithm 3 is based on Algorithm 1 of Chaudhuri et al. [14 ] , specialised to U-statistics of order two and with the additional wrapper step omitted. We use their local Hájek stabilisation construction, while deriving below the explicit sensitivity bound and privacy calibration used in our implementation.
Remark 2. The maximisation in Step 6 of Algorithm 3 does not require checking O ( 1 / ϵ ′ ) O(1/\epsilon') O ( 1/ ϵ ′ ) G ( v ) = a 0 + a 1 v + a 2 v 2 G(v)=a_0+a_1v+a_2v^2 G ( v ) = a 0 + a 1 v + a 2 v 2 0 0 0 ⌈ 2 / ϵ ′ ⌉ \lceil2/\epsilon'\rceil ⌈ 2/ ϵ ′ ⌉
a 1 + 2 a 2 ( L + u ) − ϵ ′ { a 0 + a 1 ( L + u ) + a 2 ( L + u ) 2 } = 0. a_1+2a_2(L+u)-\epsilon'\{a_0+a_1(L+u)+a_2(L+u)^2\}=0. a 1 + 2 a 2 ( L + u ) − ϵ ′ { a 0 + a 1 ( L + u ) + a 2 ( L + u ) 2 } = 0. The following lemma gives the privacy and accuracy guarantees of Algorithm 3, following the local Hájek analysis of Chaudhuri et al. [14 ] . We specialise their sensitivity argument to order-two U-statistics and track the constants explicitly to obtain the bound in Lemma 33 and the calibration ϵ ′ = ϵ 0 / 6 \epsilon'=\epsilon_0/6 ϵ ′ = ϵ 0 /6
Lemma 32. Algorithm 3 is ( ϵ 0 , 0 ) (\epsilon_0,0) ( ϵ 0 , 0 ) ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 θ h : = E [ h ( Z 1 , Z 2 ) ] \theta_h:=\mathbb{E}[h(Z_1,Z_2)] θ h := E [ h ( Z 1 , Z 2 )] α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 )
P { max i ∣ h ^ i − U h ∣ ≤ ξ } ≥ 1 − α . (81) \mathbb{P}\left\{\max_i\lvert\hat{h}_i-U_h\rvert\leq\xi\right\}\geq1-\alpha. \tag*{(81)} P { i max ∣ h ^ i − U h ∣ ≤ ξ } ≥ 1 − α . (81) Algorithm 3 LocalHájekMean
Input: Data z 1 : n z_{1:n} z 1 : n n ≥ 2 n \ge2 n ≥ 2 h h h sup h − inf h ≤ C h \sup h-\inf h \le C_h sup h − inf h ≤ C h C h > 0 C_h>0 C h > 0 ξ ≥ 0 \xi\ge0 ξ ≥ 0 ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0
Set ϵ ′ ← ϵ 0 / 6 \epsilon' \leftarrow\epsilon_0/6 ϵ ′ ← ϵ 0 /6
U h ← ( n 2 ) − 1 ∑ i < j h ( z i , z j ) , h ^ i ← 1 n − 1 ∑ j ≠ i h ( z i , z j ) , i ∈ [ n ] . U_h \leftarrow\binom{n}{2}^{-1}\sum_{i<j}h(z_i,z_j),\qquad\widehat{h}_i \leftarrow\frac{1}{n-1}\sum_{j\ne i}h(z_i,z_j),\quad i\in[n]. U h ← ( 2 n ) − 1 i < j ∑ h ( z i , z j ) , h i ← n − 1 1 j = i ∑ h ( z i , z j ) , i ∈ [ n ] . Set
L ← min { ℓ ∈ [ n ] : # { i ∈ [ n ] : ∣ h ^ i − U h ∣ > ξ + 12 C h ℓ n } ≤ ℓ } . L \leftarrow\min\left\{\ell\in[n]:\#\left\{i\in[n]:\left|\widehat{h}_i-U_h\right|>\xi+\frac{12C_h\ell}{n}\right\}\le\ell\right\}. L ← min { ℓ ∈ [ n ] : # { i ∈ [ n ] : h i − U h > ξ + n 12 C h ℓ } ≤ ℓ } . For i ∈ [ n ] i\in[n] i ∈ [ n ]
w i ← [ 1 − n ϵ ′ 12 C h ( ∣ h ^ i − U h ∣ − ξ − 12 C h L n ) ] + . w_i\leftarrow\left[1-\frac{n\epsilon'}{12C_h}\left(\left|\widehat{h}_i-U_h\right|-\xi-\frac{12C_hL}{n}\right)\right]_+. w i ← [ 1 − 12 C h n ϵ ′ ( h i − U h − ξ − n 12 C h L ) ] + . Compute
A ← U h + ( n 2 ) − 1 ∑ i < j min { w i , w j } { h ( z i , z j ) − U h } . (82) A\leftarrow U_h+\binom{n}{2}^{-1}\sum_{i<j}\min\{w_i,w_j\}\{h(z_i,z_j)-U_h\}. \tag*{(82)} A ← U h + ( 2 n ) − 1 i < j ∑ min { w i , w j } { h ( z i , z j ) − U h } . (82) Define
G ( v ) : = 4 ξ n ( 1 + ϵ ′ v ) + 57 C h n 2 { v + ( ϵ ′ + 1 n ) v 2 + 1 ϵ ′ } , v ≥ 1. (83) G(v):=\frac{4\xi}{n}(1+\epsilon'v)+\frac{57C_h}{n^2}\left\{v+\left(\epsilon'+\frac{1}{n}\right)v^2+\frac{1}{\epsilon'}\right\},\qquad v\ge1. \tag*{(83)} G ( v ) := n 4 ξ ( 1 + ϵ ′ v ) + n 2 57 C h { v + ( ϵ ′ + n 1 ) v 2 + ϵ ′ 1 } , v ≥ 1. (83) Set S ← max u ∈ { 0 , … , ⌈ 2 / ϵ ′ ⌉ } exp ( − ϵ ′ u ) G ( L + u ) S\leftarrow\max_{u\in\{0,\ldots,\lceil2/\epsilon'\rceil\}}\exp(-\epsilon'u)G(L+u) S ← max u ∈ { 0 , … , ⌈ 2/ ϵ ′ ⌉} exp ( − ϵ ′ u ) G ( L + u )
Draw Q Q Q q ( t ) : = 2 / { π ( 1 + t 4 ) } q(t):=\sqrt{2}/\{\pi(1+t^4)\} q ( t ) := 2 / { π ( 1 + t 4 )}
return A + ( S / ϵ ′ ) Q A+(S/\epsilon')Q A + ( S / ϵ ′ ) Q
then its output U ~ h \widetilde{U}_h U h
P { ∣ U ~ h − θ h ∣ > C [ Var ( U h ) α + α − 1 / 3 { ξ n ( 1 + ϵ 0 − 1 ) + C h n 2 ( 1 + ϵ 0 − 2 ) + C h n 3 ( 1 + ϵ 0 − 3 ) } ] } ≤ 3 α , (84) \mathbb{P}\left\{\left|\widetilde{U}_h-\theta_h\right|>C\left[\sqrt{\frac{\operatorname{Var}(U_h)}{\alpha}}+\alpha^{-1/3}\left\{\frac{\xi}{n}(1+\epsilon_0^{-1})+\frac{C_h}{n^2}(1+\epsilon_0^{-2})+\frac{C_h}{n^3}(1+\epsilon_0^{-3})\right\}\right]\right\}\le3\alpha, \tag*{(84)} P { U h − θ h > C [ α Var ( U h ) + α − 1/3 { n ξ ( 1 + ϵ 0 − 1 ) + n 2 C h ( 1 + ϵ 0 − 2 ) + n 3 C h ( 1 + ϵ 0 − 3 ) } ] } ≤ 3 α , (84) where C > 0 C>0 C > 0
Proof. Privacy. Let
S ~ ( D ) : = max u ∈ Z ≥ 0 exp ( − ϵ ′ u ) G ( L ( D ) + u ) . \widetilde{S}(D):=\max_{u\in\mathbb{Z}_{\ge0}}\exp(-\epsilon'u)G(L(D)+u). S ( D ) := u ∈ Z ≥ 0 max exp ( − ϵ ′ u ) G ( L ( D ) + u ) . By Lemma 33 , we have that L S A ( D ) ≤ G ( L ( D ) ) LS_A(D)\le G(L(D)) L S A ( D ) ≤ G ( L ( D )) ∣ L ( D ) − L ( D ′ ) ∣ ≤ 1 |L(D)-L(D')|\le1 ∣ L ( D ) − L ( D ′ ) ∣ ≤ 1 D , D ′ D,D' D , D ′ G G G
S ~ ( D ) ≤ max u ≥ 0 exp ( − ϵ ′ u ) G ( L ( D ′ ) + u + 1 ) = exp ( ϵ ′ ) max v ≥ 1 exp ( − ϵ ′ v ) G ( L ( D ′ ) + v ) ≤ exp ( ϵ ′ ) S ~ ( D ′ ) . \begin{aligned}
\widetilde{S}(D)\le\max_{u\ge0}\exp(-\epsilon'u)G(L(D')+u+1) \\
&=\exp(\epsilon')\max_{v\ge1}\exp(-\epsilon'v)G(L(D')+v) \\
&\le\exp(\epsilon')\widetilde{S}(D').
\end{aligned} S ( D ) ≤ u ≥ 0 max exp ( − ϵ ′ u ) G ( L ( D ′ ) + u + 1 ) = exp ( ϵ ′ ) v ≥ 1 max exp ( − ϵ ′ v ) G ( L ( D ′ ) + v ) ≤ exp ( ϵ ′ ) S ( D ′ ) . This implies that S ~ ( D ) \widetilde{S}(D) S ( D ) ϵ ′ \epsilon' ϵ ′ L S A ( D ) LS_A(D) L S A ( D ) [37 ] . Since v G ′ ( v ) ≤ 2 G ( v ) vG'(v) \leq2G(v) v G ′ ( v ) ≤ 2 G ( v ) v > 0 v > 0 v > 0 L ( D ) ≥ 0 L(D) \geq0 L ( D ) ≥ 0
d d u [ e − ϵ ′ u G ( L ( D ) + u ) ] ≤ e − ϵ ′ u G ( L ( D ) + u ) { 2 L ( D ) + u − ϵ ′ } ≤ 0 , u ≥ 2 ϵ ′ , \frac{\mathrm{d}}{\mathrm{d}u}\left[e^{-\epsilon' u}G(L(D)+u)\right] \leq e^{-\epsilon' u}G(L(D)+u)\left\{\frac{2}{L(D)+u}-\epsilon'\right\} \leq0,\qquad u \geq\frac{2}{\epsilon'}, d u d [ e − ϵ ′ u G ( L ( D ) + u ) ] ≤ e − ϵ ′ u G ( L ( D ) + u ) { L ( D ) + u 2 − ϵ ′ } ≤ 0 , u ≥ ϵ ′ 2 , so the expression for S ( D ) S(D) S ( D ) S ~ ( D ) \widetilde{S}(D) S ( D )
The next part of the proof follows the arguments of Lemma 2.7 of Nissim et al. [37 ] , with q q q σ ( D ) : = S ( D ) / ϵ ′ \sigma(D):=S(D)/\epsilon' σ ( D ) := S ( D ) / ϵ ′
p D ( x ) : = 1 σ ( D ) q ( x − A ( D ) σ ( D ) ) , p ~ ( x ) : = 1 σ ( D ′ ) q ( x − A ( D ) σ ( D ′ ) ) , p_D(x):=\frac{1}{\sigma(D)}q\left(\frac{x-A(D)}{\sigma(D)}\right),\qquad\widetilde{p}(x):=\frac{1}{\sigma(D')}q\left(\frac{x-A(D)}{\sigma(D')}\right), p D ( x ) := σ ( D ) 1 q ( σ ( D ) x − A ( D ) ) , p ( x ) := σ ( D ′ ) 1 q ( σ ( D ′ ) x − A ( D ) ) , so that p D ( x ) p_D(x) p D ( x ) p ~ ( x ) \widetilde{p}(x) p ( x )
∣ log p D ( x ) p D ′ ( x ) ∣ ≤ ∣ log p D ( x ) p ~ ( x ) ∣ + ∣ log p ~ ( x ) p D ′ ( x ) ∣ . (85) \left|\log\frac{p_D(x)}{p_{D'}(x)}\right|\leq\left|\log\frac{p_D(x)}{\widetilde{p}(x)}\right|+\left|\log\frac{\widetilde{p}(x)}{p_{D'}(x)}\right|. \tag*{(85)} log p D ′ ( x ) p D ( x ) ≤ log p ( x ) p D ( x ) + log p D ′ ( x ) p ( x ) . (85) We bound the first term of (85) , corresponding to the scale change. Considering the change of variable t = ( x − A ( D ) ) / σ t=(x-A(D))/\sigma t = ( x − A ( D )) / σ
∣ ∂ ∂ log σ log { 1 σ q ( x − A ( D ) σ ) } ∣ = ∣ − 1 − t ( log q ) ′ ( t ) ∣ = ∣ 1 − 3 t 4 1 + t 4 ∣ ≤ 3. \begin{aligned}
\left|\frac{\partial}{\partial\log\sigma}\log\left\{\frac{1}{\sigma}q\left(\frac{x-A(D)}{\sigma}\right)\right\}\right|=\left|-1-t(\log q)'(t)\right| \\
&=\left|\frac{1-3t^4}{1+t^4}\right|\leq3.
\end{aligned} ∂ log σ ∂ log { σ 1 q ( σ x − A ( D ) ) } = ∣ − 1 − t ( log q ) ′ ( t ) ∣ = 1 + t 4 1 − 3 t 4 ≤ 3. Thus, by the mean value theorem, we can bound the first term of (85) by
∣ log p D ( x ) p ~ ( x ) ∣ ≤ 3 ∣ log ( σ ( D ) ) − log ( σ ( D ′ ) ) ∣ ≤ 3 ϵ ′ . \left|\log\frac{p_D(x)}{\widetilde{p}(x)}\right|\leq3|\log(\sigma(D))-\log(\sigma(D'))|\leq3\epsilon'. log p ( x ) p D ( x ) ≤ 3∣ log ( σ ( D )) − log ( σ ( D ′ )) ∣ ≤ 3 ϵ ′ . Since we have that
∣ ∂ ∂ μ log { 1 σ ( D ′ ) q ( x − μ σ ( D ′ ) ) } ∣ = ∣ − 1 σ ( D ′ ) ( log q ) ′ ( x − μ σ ( D ′ ) ) ∣ , \left|\frac{\partial}{\partial\mu}\log\left\{\frac{1}{\sigma(D')}q\left(\frac{x-\mu}{\sigma(D')}\right)\right\}\right|=\left|-\frac{1}{\sigma(D')}(\log q)'\left(\frac{x-\mu}{\sigma(D')}\right)\right|, ∂ μ ∂ log { σ ( D ′ ) 1 q ( σ ( D ′ ) x − μ ) } = − σ ( D ′ ) 1 ( log q ) ′ ( σ ( D ′ ) x − μ ) , we can similarly use
∣ ( log q ) ′ ( t ) ∣ ≤ ∣ − 4 t 3 1 + t 4 ∣ ≤ 3 |(\log q)'(t)|\leq\left|\frac{-4t^3}{1+t^4}\right|\leq3 ∣ ( log q ) ′ ( t ) ∣ ≤ 1 + t 4 − 4 t 3 ≤ 3 and the mean value theorem to bound
∣ log p ~ ( x ) p D ′ ( x ) ∣ ≤ 3 ∣ A ( D ) − A ( D ′ ) ∣ σ ( D ′ ) ≤ 3 S ( D ′ ) S ( D ′ ) / ϵ ′ = 3 ϵ ′ . \left|\log\frac{\widetilde{p}(x)}{p_{D'}(x)}\right|\leq\frac{3|A(D)-A(D')|}{\sigma(D')}\leq\frac{3S(D')}{S(D')/\epsilon'}=3\epsilon'. log p D ′ ( x ) p ( x ) ≤ σ ( D ′ ) 3∣ A ( D ) − A ( D ′ ) ∣ ≤ S ( D ′ ) / ϵ ′ 3 S ( D ′ ) = 3 ϵ ′ . Substituting these bounds into (85) gives ∣ log { p D ( x ) / p D ′ ( x ) } ∣ ≤ 6 ϵ ′ = ϵ 0 |\log\{p_D(x)/p_{D'}(x)\}|\leq6\epsilon'=\epsilon_0 ∣ log { p D ( x ) / p D ′ ( x )} ∣ ≤ 6 ϵ ′ = ϵ 0
Accuracy. For the non-private component, we have by Chebyshev’s inequality that
P ( ∣ U h − θ h ∣ > Var ( U h ) / α ) ≤ α . (86) \mathbb{P}\left(|U_h-\theta_h|>\sqrt{\operatorname{Var}(U_h)/\alpha}\right)\leq\alpha. \tag*{(86)} P ( ∣ U h − θ h ∣ > Var ( U h ) / α ) ≤ α . (86) On the event in (81) , we have that L ( D ) = 1 L(D)=1 L ( D ) = 1 w i = 1 w_i=1 w i = 1 i ∈ [ n ] i \in[n] i ∈ [ n ] A ( D ) = U h A(D)=U_h A ( D ) = U h L ( D ) = 1 L(D)=1 L ( D ) = 1 S ( D ) S(D) S ( D )
S ( D ) ≤ sup u ≥ 0 exp ( − ϵ ′ u ) G ( 1 + u ) . S(D) \le\sup_{u \ge0} \exp(-\epsilon' u)G(1+u). S ( D ) ≤ u ≥ 0 sup exp ( − ϵ ′ u ) G ( 1 + u ) . Using
sup u ≥ 0 exp ( − ϵ ′ u ) ( 1 + u ) k ≤ 2 k − 1 { 1 + sup u ≥ 0 u k exp ( − ϵ ′ u ) } = 2 k − 1 { 1 + ( k e ϵ ′ ) k } ≲ 1 + 1 ( ϵ ′ ) k , k = 1 , 2 , \begin{aligned}
\sup_{u \ge0}\exp(-\epsilon' u)(1+u)^k \le2^{k-1}\left\{1+\sup_{u \ge0}u^k\exp(-\epsilon' u)\right\} \\
&=2^{k-1}\left\{1+\left(\frac{k}{e\epsilon'}\right)^k\right\}\lesssim1+\frac{1}{(\epsilon')^k},\qquad k=1,2,
\end{aligned} u ≥ 0 sup exp ( − ϵ ′ u ) ( 1 + u ) k ≤ 2 k − 1 { 1 + u ≥ 0 sup u k exp ( − ϵ ′ u ) } = 2 k − 1 { 1 + ( e ϵ ′ k ) k } ≲ 1 + ( ϵ ′ ) k 1 , k = 1 , 2 , we obtain
S ( D ) ϵ ′ ≲ ξ n ( 1 + 1 ϵ ′ ) + C h n 2 ( 1 + 1 ϵ ′ + 1 ( ϵ ′ ) 2 ) + C h n 3 ( 1 ϵ ′ + 1 ( ϵ ′ ) 2 + 1 ( ϵ ′ ) 3 ) \frac{S(D)}{\epsilon'}\lesssim\frac{\xi}{n}\left(1+\frac{1}{\epsilon'}\right)+\frac{C_h}{n^2}\left(1+\frac{1}{\epsilon'}+\frac{1}{(\epsilon')^2}\right)+\frac{C_h}{n^3}\left(\frac{1}{\epsilon'}+\frac{1}{(\epsilon')^2}+\frac{1}{(\epsilon')^3}\right) ϵ ′ S ( D ) ≲ n ξ ( 1 + ϵ ′ 1 ) + n 2 C h ( 1 + ϵ ′ 1 + ( ϵ ′ ) 2 1 ) + n 3 C h ( ϵ ′ 1 + ( ϵ ′ ) 2 1 + ( ϵ ′ ) 3 1 ) ≲ ξ n ( 1 + ϵ 0 − 1 ) + C h n 2 ( 1 + ϵ 0 − 2 ) + C h n 3 ( 1 + ϵ 0 − 3 ) . \lesssim\frac{\xi}{n}(1+\epsilon_0^{-1})+\frac{C_h}{n^2}(1+\epsilon_0^{-2})+\frac{C_h}{n^3}(1+\epsilon_0^{-3}). ≲ n ξ ( 1 + ϵ 0 − 1 ) + n 2 C h ( 1 + ϵ 0 − 2 ) + n 3 C h ( 1 + ϵ 0 − 3 ) . We also have, for Q ∼ q Q\sim q Q ∼ q
P ( ∣ Q ∣ > u ) = 2 2 π ∫ u ∞ 1 1 + t 4 d t ≤ 2 2 3 π u 3 , u > 0. (87) \mathbb{P}(|Q|>u)=\frac{2\sqrt{2}}{\pi}\int_u^\infty\frac{1}{1+t^4}\,\mathrm{d}t\le\frac{2\sqrt{2}}{3\pi u^3},\qquad u>0. \tag*{(87)} P ( ∣ Q ∣ > u ) = π 2 2 ∫ u ∞ 1 + t 4 1 d t ≤ 3 π u 3 2 2 , u > 0. (87) Combining (86) , the event in (81) , and (87) with u = ( 2 2 / ( 3 π α ) ) 1 / 3 u=(2\sqrt{2}/(3\pi\alpha))^{1/3} u = ( 2 2 / ( 3 π α ) ) 1/3 (84) . □
Lemma 33. For every ϵ ′ > 0 \epsilon' > 0 ϵ ′ > 0 D , D ′ D,D' D , D ′ ∣ L ( D ) − L ( D ′ ) ∣ ≤ 1 |L(D)-L(D')|\le1 ∣ L ( D ) − L ( D ′ ) ∣ ≤ 1 L S A ( D ) ≤ G ( L ( D ) ) \mathrm{LS}_A(D)\le G(L(D)) LS A ( D ) ≤ G ( L ( D )) G G G (83) .
Proof. Suppose D , D ′ D,D' D , D ′ k k k D ′ D' D ′
∣ U h − U h ′ ∣ = ∣ 2 n ( n − 1 ) ∑ j ≠ k { h ( z j , z k ) − h ( z j , z k ′ ) } ∣ ≤ 2 C h n . (88) |U_h-U'_h|=\left|\frac{2}{n(n-1)}\sum_{j\ne k}\{h(z_j,z_k)-h(z_j,z'_k)\}\right|\le\frac{2C_h}{n}. \tag*{(88)} ∣ U h − U h ′ ∣ = n ( n − 1 ) 2 j = k ∑ { h ( z j , z k ) − h ( z j , z k ′ )} ≤ n 2 C h . (88) and for i ≠ k i\ne k i = k
∣ h ^ i − U h − ( h ^ i ′ − U h ′ ) ∣ = ∣ n − 2 n ( n − 1 ) { h ( z i , z k ) − h ( z i , z k ′ ) } − 2 n ( n − 1 ) ∑ j ≠ i , k { h ( z j , z k ) − h ( z j , z k ′ ) } ∣ ≤ 3 C h ( n − 2 ) n ( n − 1 ) . (89) \begin{aligned}
\left|\hat{h}_i-U_h-(\hat{h}'_i-U'_h)\right|=\left|\frac{n-2}{n(n-1)}\{h(z_i,z_k)-h(z_i,z'_k)\}-\frac{2}{n(n-1)}\sum_{j\ne i,k}\{h(z_j,z_k)-h(z_j,z'_k)\}\right| \\
&\le\frac{3C_h(n-2)}{n(n-1)}. \tag*{(89)}
\end{aligned} h ^ i − U h − ( h ^ i ′ − U h ′ ) = n ( n − 1 ) n − 2 { h ( z i , z k ) − h ( z i , z k ′ )} − n ( n − 1 ) 2 j = i , k ∑ { h ( z j , z k ) − h ( z j , z k ′ )} ≤ n ( n − 1 ) 3 C h ( n − 2 ) . (89) Therefore, ∣ h ^ i − U h ∣ ≤ ξ + 12 C h L ( D ) / n |\hat{h}_i-U_h|\le\xi+12C_hL(D)/n ∣ h ^ i − U h ∣ ≤ ξ + 12 C h L ( D ) / n
∣ h ^ i ′ − U h ′ ∣ ≤ ξ + 12 C h ( L ( D ) + 1 ) n . |\hat{h}'_i-U'_h|\le\xi+\frac{12C_h(L(D)+1)}{n}. ∣ h ^ i ′ − U h ′ ∣ ≤ ξ + n 12 C h ( L ( D ) + 1 ) . There are at most L ( D ) L(D) L ( D ) D D D L ( D ′ ) ≤ L ( D ) + 1 L(D') \le L(D)+1 L ( D ′ ) ≤ L ( D ) + 1 D , D ′ D,D' D , D ′ ∣ L ( D ) − L ( D ′ ) ∣ ≤ 1 |L(D)-L(D')| \le1 ∣ L ( D ) − L ( D ′ ) ∣ ≤ 1
For T ≥ 1 T \ge1 T ≥ 1
F T ( x ) : = x min { 1 , n ϵ ′ 12 C h [ ∣ x ∣ − ξ − 12 C h T n ] + } . F_T(x) := x \min\left\{1,\frac{n\epsilon'}{12C_h}\left[|x|-\xi-\frac{12C_hT}{n}\right]_+ \right\}. F T ( x ) := x min { 1 , 12 C h n ϵ ′ [ ∣ x ∣ − ξ − n 12 C h T ] + } . so that F L ( D ) ( h ^ i − U h ) = ( h ^ i − U h ) ( 1 − w i ) F_{L(D)}(\hat{h}_i-U_h)=(\hat{h}_i-U_h)(1-w_i) F L ( D ) ( h ^ i − U h ) = ( h ^ i − U h ) ( 1 − w i )
∣ x − F T ( x ) ∣ ≤ ξ + 12 C h T n + 3 C h n ϵ ′ ( 1 − ϵ ′ T ) + 2 . (90) |x-F_T(x)| \le\xi+\frac{12C_hT}{n}+\frac{3C_h}{n\epsilon'}(1-\epsilon'T)_+^2. \tag*{(90)} ∣ x − F T ( x ) ∣ ≤ ξ + n 12 C h T + n ϵ ′ 3 C h ( 1 − ϵ ′ T ) + 2 . (90) ∣ F T ( x ) − F T ( y ) ∣ ≤ 3 C h n { 2 + ϵ ′ ( n ξ 12 C h + T ) } , ∣ x − y ∣ ≤ 3 C h n . (91) |F_T(x)-F_T(y)| \le\frac{3C_h}{n}\left\{2+\epsilon'\left(\frac{n\xi}{12C_h}+T\right)\right\},\qquad|x-y|\le\frac{3C_h}{n}. \tag*{(91)} ∣ F T ( x ) − F T ( y ) ∣ ≤ n 3 C h { 2 + ϵ ′ ( 12 C h n ξ + T ) } , ∣ x − y ∣ ≤ n 3 C h . (91) ∣ F T ( x ) − F T + 1 ( y ) ∣ ≤ 12 C h n [ 5 ϵ ′ 4 ( n ξ 12 C h + T − 1 4 ) + 3 2 ] , ∣ x − y ∣ ≤ 3 C h n . (92) |F_T(x)-F_{T+1}(y)| \le\frac{12C_h}{n}\left[\frac{5\epsilon'}{4}\left(\frac{n\xi}{12C_h}+T-\frac{1}{4}\right)+\frac{3}{2}\right],\qquad|x-y|\le\frac{3C_h}{n}. \tag*{(92)} ∣ F T ( x ) − F T + 1 ( y ) ∣ ≤ n 12 C h [ 4 5 ϵ ′ ( 12 C h n ξ + T − 4 1 ) + 2 3 ] , ∣ x − y ∣ ≤ n 3 C h . (92) We defer showing these bounds to the end of this proof. In particular, (90) implies that
∣ ( h ^ i − U h ) w i ∣ ≤ ξ + C h n { 6 L ( D ) + 3 ϵ ′ + 3 ϵ ′ L ( D ) 2 } . (93) \left|(\hat{h}_i-U_h)w_i\right| \le\xi+\frac{C_h}{n}\left\{6L(D)+\frac{3}{\epsilon'}+3\epsilon'L(D)^2\right\}. \tag*{(93)} ( h ^ i − U h ) w i ≤ ξ + n C h { 6 L ( D ) + ϵ ′ 3 + 3 ϵ ′ L ( D ) 2 } . (93) We now use these bounds to control the local sensitivity of A A A (82) as
A = U h − ( n 2 ) − 1 ∑ i < j { ( 1 − w i ) + ( 1 − w j ) } { h ( z i , z j ) − U h } + R = U h − ( n 2 ) − 1 ∑ i ( 1 − w i ) ∑ j ≠ i { h ( z i , z j ) − U h } + R = U h − 2 n ∑ i ( 1 − w i ) ( h ^ i − U h ) + R = U h − 2 n ∑ i F L ( D ) ( h ^ i − U h ) + R , \begin{aligned}
A=U_h-\binom{n}{2}^{-1}\sum_{i<j}\{(1-w_i)+(1-w_j)\}\{h(z_i,z_j)-U_h\}+R \\
&=U_h-\binom{n}{2}^{-1}\sum_i(1-w_i)\sum_{j\ne i}\{h(z_i,z_j)-U_h\}+R \\
&=U_h-\frac{2}{n}\sum_i(1-w_i)(\hat{h}_i-U_h)+R
=U_h-\frac{2}{n}\sum_iF_{L(D)}(\hat{h}_i-U_h)+R,
\end{aligned} A = U h − ( 2 n ) − 1 i < j ∑ {( 1 − w i ) + ( 1 − w j )} { h ( z i , z j ) − U h } + R = U h − ( 2 n ) − 1 i ∑ ( 1 − w i ) j = i ∑ { h ( z i , z j ) − U h } + R = U h − n 2 i ∑ ( 1 − w i ) ( h ^ i − U h ) + R = U h − n 2 i ∑ F L ( D ) ( h ^ i − U h ) + R , where we denote
R : = ( n 2 ) − 1 ∑ i < j min { 1 − w i , 1 − w j } { h ( z i , z j ) − U h } . R:=\binom{n}{2}^{-1}\sum_{i<j}\min\{1-w_i,1-w_j\}\{h(z_i,z_j)-U_h\}. R := ( 2 n ) − 1 i < j ∑ min { 1 − w i , 1 − w j } { h ( z i , z j ) − U h } . The first equality follows from
min { w i , w j } = 1 − ( 1 − w i ) − ( 1 − w j ) + min { 1 − w i , 1 − w j } \min\{w_i,w_j\}=1-(1-w_i)-(1-w_j)+\min\{1-w_i,1-w_j\} min { w i , w j } = 1 − ( 1 − w i ) − ( 1 − w j ) + min { 1 − w i , 1 − w j } and that ∑ i < j { h ( z i , z j ) − U h } = 0 \sum_{i<j}\{h(z_i,z_j)-U_h\}=0 ∑ i < j { h ( z i , z j ) − U h } = 0 ∑ j ≠ i { h ( z i , z j ) − U h } = ( n − 1 ) ( h ^ i − U h ) \sum_{j\ne i}\{h(z_i,z_j)-U_h\}=(n-1)(\hat{h}_i-U_h) ∑ j = i { h ( z i , z j ) − U h } = ( n − 1 ) ( h ^ i − U h )
With this form of A A A k k k
A − A ′ = U h − U h ′ − 2 n { ( h ^ k − U h ) − ( h ^ k ′ − U h ′ ) } + 2 n { ( h ^ k − U h ) w k − ( h ^ k ′ − U h ′ ) w k ′ } − 2 n ∑ i ≠ k { F L ( D ) ( h ^ i − U h ) − F L ( D ′ ) ( h ^ i ′ − U h ′ ) } + R − R ′ = 2 n [ U h − U h ′ + ( h ^ k − U h ) w k − ( h ^ k ′ − U h ′ ) w k ′ − ∑ i ≠ k { F L ( D ) ( h ^ i − U h ) − F L ( D ′ ) ( h ^ i ′ − U h ′ ) } ] + R − R ′ . (94) \begin{aligned}
A-A'={}&U_h-U_h'-\frac{2}{n}\{(\hat{h}_k-U_h)-(\hat{h}_k'-U_h')\}+\frac{2}{n}\{(\hat{h}_k-U_h)w_k-(\hat{h}_k'-U_h')w_k'\}\\
&-\frac{2}{n}\sum_{i\ne k}\{F_{L(D)}(\hat{h}_i-U_h)-F_{L(D')}(\hat{h}_i'-U_h')\}+R-R'\\
={}&\frac{2}{n}\left[U_h-U_h'+(\hat{h}_k-U_h)w_k-(\hat{h}_k'-U_h')w_k' \right.\\
&\left.\qquad-\sum_{i\ne k}\{F_{L(D)}(\hat{h}_i-U_h)-F_{L(D')}(\hat{h}_i'-U_h')\}\right]+R-R'. \tag*{(94)}
\end{aligned} A − A ′ = = U h − U h ′ − n 2 {( h ^ k − U h ) − ( h ^ k ′ − U h ′ )} + n 2 {( h ^ k − U h ) w k − ( h ^ k ′ − U h ′ ) w k ′ } − n 2 i = k ∑ { F L ( D ) ( h ^ i − U h ) − F L ( D ′ ) ( h ^ i ′ − U h ′ )} + R − R ′ n 2 [ U h − U h ′ + ( h ^ k − U h ) w k − ( h ^ k ′ − U h ′ ) w k ′ − i = k ∑ { F L ( D ) ( h ^ i − U h ) − F L ( D ′ ) ( h ^ i ′ − U h ′ )} + R − R ′ . (94) The second equality is from noting that
( h ^ k − h ^ k ′ ) − ( U h − U h ′ ) = 1 n − 1 ∑ j ≠ k { h ( z k , z j ) − h ( z k ′ , z j ) } − ( U h − U h ′ ) = n − 2 2 ( U h − U h ′ ) . \begin{aligned}
(\hat{h}_k-\hat{h}_k')-(U_h-U_h')=\frac{1}{n-1}\sum_{j\ne k}\{h(z_k,z_j)-h(z_k',z_j)\}-(U_h-U_h') \\
&=\frac{n-2}{2}(U_h-U_h').
\end{aligned} ( h ^ k − h ^ k ′ ) − ( U h − U h ′ ) = n − 1 1 j = k ∑ { h ( z k , z j ) − h ( z k ′ , z j )} − ( U h − U h ′ ) = 2 n − 2 ( U h − U h ′ ) . Since ∣ L ( D ) − L ( D ′ ) ∣ ≤ 1 |L(D)-L(D')|\le1 ∣ L ( D ) − L ( D ′ ) ∣ ≤ 1 (94) in the two different cases.
Case 1: L ( D ) = L ( D ′ ) L(D)=L(D') L ( D ) = L ( D ′ ) (88) and (93) , we have
∣ A − A ′ ∣ ≤ 2 n { 2 C h n + 2 [ ε + C h n { 6 L ( D ) + 3 ε ′ + 3 ε ′ L ( D ) 2 } ] } + 2 n ∣ ∑ i ≠ k { F L ( D ) ( h ^ i − U h ) − F L ( D ′ ) ( h ^ i ′ − U h ′ ) } ∣ + ∣ R − R ′ ∣ . |A-A'|\le\frac{2}{n}\left\{\frac{2C_h}{n}+2\left[\varepsilon+\frac{C_h}{n}\left\{6L(D)+\frac{3}{\varepsilon'}+3\varepsilon'L(D)^2\right\}\right]\right\}
+\frac{2}{n}\left|\sum_{i\ne k}\{F_{L(D)}(\hat{h}_i-U_h)-F_{L(D')}(\hat{h}_i'-U_h')\}\right|+|R-R'|. ∣ A − A ′ ∣ ≤ n 2 { n 2 C h + 2 [ ε + n C h { 6 L ( D ) + ε ′ 3 + 3 ε ′ L ( D ) 2 } ] } + n 2 i = k ∑ { F L ( D ) ( h ^ i − U h ) − F L ( D ′ ) ( h ^ i ′ − U h ′ )} + ∣ R − R ′ ∣. In the summation, at most 2 L ( D ) 2L(D) 2 L ( D ) (91) we can bound
∣ F L ( D ) ( h ^ i − U h ) − F L ( D ) ( h ^ i ′ − U h ′ ) ∣ ≤ 3 C h n { 2 + ε ′ ( n ε 12 C h + L ( D ) ) } . |F_{L(D)}(\hat{h}_i-U_h)-F_{L(D)}(\hat{h}_i'-U_h')|\le\frac{3C_h}{n}\left\{2+\varepsilon'\left(\frac{n\varepsilon}{12C_h}+L(D)\right)\right\}. ∣ F L ( D ) ( h ^ i − U h ) − F L ( D ) ( h ^ i ′ − U h ′ ) ∣ ≤ n 3 C h { 2 + ε ′ ( 12 C h n ε + L ( D ) ) } . Let B : = { i : 1 − w i > 0 } B:=\{i:1-w_i>0\} B := { i : 1 − w i > 0 } B ′ B' B ′ ∣ B ∣ , ∣ B ′ ∣ ≤ L ( D ) |B|,|B'|\le L(D) ∣ B ∣ , ∣ B ′ ∣ ≤ L ( D ) i i i i ≠ k i\ne k i = k (89) that
∣ w i − w i ′ ∣ ≤ n ε ′ 12 C h ∣ h ^ i − U h − ( h ^ i ′ − U h ′ ) ∣ ≤ ε ′ 4 . |w_i-w_i'|\le\frac{n\varepsilon'}{12C_h}|\hat{h}_i-U_h-(\hat{h}_i'-U_h')|\le\frac{\varepsilon'}{4}. ∣ w i − w i ′ ∣ ≤ 12 C h n ε ′ ∣ h ^ i − U h − ( h ^ i ′ − U h ′ ) ∣ ≤ 4 ε ′ . We can therefore write
∣ R − R ′ ∣ ≤ ( n 2 ) − 1 { ∑ i < j , i , j ≠ k i , j ∈ B ′ min { 1 − w i ′ , 1 − w j ′ } ∣ U h ′ − U h ∣ + ∑ i < j , i , j ≠ k i , j ∈ B ∪ B ′ ∣ min { 1 − w i , 1 − w j } − min { 1 − w i ′ , 1 − w j ′ } ∣ ∣ h ( z i , z j ) − U h ∣ + ∑ j ≠ k j ∈ B ∪ B ′ ∣ min { 1 − w k , 1 − w j } { h ( z k , z j ) − U h } − min { 1 − w k ′ , 1 − w j ′ } { h ( z k ′ , z j ) − U h ′ } ∣ } ≤ ( n 2 ) − 1 { C h ϵ ′ 4 ( 2 L ( D ) 2 ) + 2 C h n ( L ( D ) 2 ) + 4 L ( D ) C h } ≤ C h n 2 { 16 L ( D ) + 2 ϵ ′ L ( D ) 2 − ϵ ′ L ( D ) + 4 ( L ( D ) 2 − L ( D ) ) n } . \begin{aligned}
|R-R'| &\leq\binom{n}{2}^{-1}\Biggl\{
\sum_{\substack{i<j,\ i,j\ne k\\ i,j\in B'}} \min\{1-w_i',1-w_j'\}|U_h'-U_h| \\
&\quad+\sum_{\substack{i<j,\ i,j\ne k\\ i,j\in B\cup B'}} \left|\min\{1-w_i,1-w_j\}-\min\{1-w_i',1-w_j'\}\right||h(z_i,z_j)-U_h| \\
&\quad+\sum_{\substack{j\ne k\\ j\in B\cup B'}} \left|\min\{1-w_k,1-w_j\}\{h(z_k,z_j)-U_h\}-\min\{1-w_k',1-w_j'\}\{h(z_k',z_j)-U_h'\}\right|
\Biggr\} \\
&\leq\binom{n}{2}^{-1}\left\{\frac{C_h\epsilon'}{4}\binom{2L(D)}{2}+\frac{2C_h}{n}\binom{L(D)}{2}+4L(D)C_h\right\}\\
&\leq\frac{C_h}{n^2}\left\{16L(D)+2\epsilon'L(D)^2-\epsilon'L(D)+\frac{4(L(D)^2-L(D))}{n}\right\}.
\end{aligned} ∣ R − R ′ ∣ ≤ ( 2 n ) − 1 { i < j , i , j = k i , j ∈ B ′ ∑ min { 1 − w i ′ , 1 − w j ′ } ∣ U h ′ − U h ∣ + i < j , i , j = k i , j ∈ B ∪ B ′ ∑ min { 1 − w i , 1 − w j } − min { 1 − w i ′ , 1 − w j ′ } ∣ h ( z i , z j ) − U h ∣ + j = k j ∈ B ∪ B ′ ∑ min { 1 − w k , 1 − w j } { h ( z k , z j ) − U h } − min { 1 − w k ′ , 1 − w j ′ } { h ( z k ′ , z j ) − U h ′ } } ≤ ( 2 n ) − 1 { 4 C h ϵ ′ ( 2 2 L ( D ) ) + n 2 C h ( 2 L ( D ) ) + 4 L ( D ) C h } ≤ n 2 C h { 16 L ( D ) + 2 ϵ ′ L ( D ) 2 − ϵ ′ L ( D ) + n 4 ( L ( D ) 2 − L ( D )) } . Hence, we can bound
∣ A − A ′ ∣ ≤ ξ n ( 4 + ϵ ′ L ( D ) ) + C h n 2 { 64 L ( D ) + 4 + 26 ϵ ′ L ( D ) 2 − ϵ ′ L ( D ) + 4 ( L ( D ) 2 − L ( D ) ) n + 12 ϵ ′ } . |A-A'|\leq\frac{\xi}{n}(4+\epsilon'L(D))+\frac{C_h}{n^2}\left\{64L(D)+4+26\epsilon'L(D)^2-\epsilon'L(D)+\frac{4(L(D)^2-L(D))}{n}+\frac{12}{\epsilon'}\right\}. ∣ A − A ′ ∣ ≤ n ξ ( 4 + ϵ ′ L ( D )) + n 2 C h { 64 L ( D ) + 4 + 26 ϵ ′ L ( D ) 2 − ϵ ′ L ( D ) + n 4 ( L ( D ) 2 − L ( D )) + ϵ ′ 12 } . Case 2: ∣ L ( D ′ ) − L ( D ) ∣ = 1 |L(D')-L(D)|=1 ∣ L ( D ′ ) − L ( D ) ∣ = 1 L ( D ′ ) = L ( D ) + 1 L(D')=L(D)+1 L ( D ′ ) = L ( D ) + 1 (88) and (93) , we have
∣ A − A ′ ∣ ≤ 2 n { 2 C h n + 2 ξ + C h n [ 6 L ( D ) + 6 ( L ( D ) + 1 ) + 6 ϵ ′ + 3 ϵ ′ { L ( D ) 2 + ( L ( D ) + 1 ) 2 } ] } + 2 n ∣ ∑ i ≠ k { F L ( D ) ( h ^ i − U h ) − F L ( D ) + 1 ( h ^ i ′ − U h ′ ) } ∣ + ∣ R − R ′ ∣ . \begin{aligned}
|A-A'| &\leq\frac{2}{n}\left\{\frac{2C_h}{n}+2\xi+\frac{C_h}{n}\left[6L(D)+6(L(D)+1)+\frac{6}{\epsilon'}+3\epsilon'\{L(D)^2+(L(D)+1)^2\}\right]\right\}\\
&\quad+\frac{2}{n}\left|\sum_{i\ne k}\{F_{L(D)}(\hat h_i-U_h)-F_{L(D)+1}(\hat h_i'-U_h')\}\right|+|R-R'|.
\end{aligned} ∣ A − A ′ ∣ ≤ n 2 { n 2 C h + 2 ξ + n C h [ 6 L ( D ) + 6 ( L ( D ) + 1 ) + ϵ ′ 6 + 3 ϵ ′ { L ( D ) 2 + ( L ( D ) + 1 ) 2 } ] } + n 2 i = k ∑ { F L ( D ) ( h ^ i − U h ) − F L ( D ) + 1 ( h ^ i ′ − U h ′ )} + ∣ R − R ′ ∣. For i ≠ k i\ne k i = k i ∈ B ′ i\in B' i ∈ B ′ ∣ h ^ i ′ − U h ′ ∣ > ξ + 12 C h ( L ( D ) + 1 ) / n |\hat h_i'-U_h'|>\xi+12C_h(L(D)+1)/n ∣ h ^ i ′ − U h ′ ∣ > ξ + 12 C h ( L ( D ) + 1 ) / n (89) , implies ∣ h ^ i − U h ∣ > ξ + 12 C h L ( D ) / n |\hat h_i-U_h|>\xi+12C_hL(D)/n ∣ h ^ i − U h ∣ > ξ + 12 C h L ( D ) / n B ′ ∖ { k } ⊆ B ∖ { k } B'\setminus\{k\}\subseteq B\setminus\{k\} B ′ ∖ { k } ⊆ B ∖ { k } L ( D ) L(D) L ( D ) (92) gives
∣ F L ( D ) ( h ^ i − U h ) − F L ( D ) + 1 ( h ^ i ′ − U h ′ ) ∣ ≤ 12 C h n { 5 ϵ ′ 4 ( n ξ 12 C h + L ( D ) − 1 4 ) + 3 2 } . |F_{L(D)}(\hat h_i-U_h)-F_{L(D)+1}(\hat h_i'-U_h')|
\leq\frac{12C_h}{n}\left\{\frac{5\epsilon'}{4}\left(\frac{n\xi}{12C_h}+L(D)-\frac{1}{4}\right)+\frac{3}{2}\right\}. ∣ F L ( D ) ( h ^ i − U h ) − F L ( D ) + 1 ( h ^ i ′ − U h ′ ) ∣ ≤ n 12 C h { 4 5 ϵ ′ ( 12 C h n ξ + L ( D ) − 4 1 ) + 2 3 } . Further, we have
∣ w i − w i ′ ∣ ≤ n ϵ ′ 12 C h ∣ ( ∣ h ^ i − U h ∣ − ξ − 12 C h L ( D ) n ) − ( ∣ h ^ i ′ − U h ′ ∣ − ξ − 12 C h L ( D ′ ) n ) ∣ ≤ n ϵ ′ 12 C h ( 3 C h n + 12 C h n ) = 5 ϵ ′ 4 , i ≠ k . \begin{aligned}
|w_i-w_i'|&\leq\frac{n\epsilon'}{12C_h}\left|\left(|\hat h_i-U_h|-\xi-\frac{12C_hL(D)}{n}\right)-\left(|\hat h_i'-U_h'|-\xi-\frac{12C_hL(D')}{n}\right)\right|\\
&\leq\frac{n\epsilon'}{12C_h}\left(\frac{3C_h}{n}+\frac{12C_h}{n}\right)=\frac{5\epsilon'}{4},\qquad i\ne k.
\end{aligned} ∣ w i − w i ′ ∣ ≤ 12 C h n ϵ ′ ( ∣ h ^ i − U h ∣ − ξ − n 12 C h L ( D ) ) − ( ∣ h ^ i ′ − U h ′ ∣ − ξ − n 12 C h L ( D ′ ) ) ≤ 12 C h n ϵ ′ ( n 3 C h + n 12 C h ) = 4 5 ϵ ′ , i = k . Using the same grouping argument as in Case 1 leads to
∣ R − R ′ ∣ ≤ 4 n 2 { 2 C h n ( L ( D ) 2 ) + 5 ϵ ′ C h 4 ( L ( D ) 2 ) + 2 C h L ( D ) } = C h n 2 { 8 L ( D ) + 5 ϵ ′ 2 ( L ( D ) 2 − L ( D ) ) + 4 ( L ( D ) 2 − L ( D ) ) n } . \lvert R-R'\rvert\le\frac{4}{n^2}\left\{\frac{2C_h}{n}\binom{L(D)}{2}+\frac{5\epsilon'C_h}{4}\binom{L(D)}{2}+2C_hL(D)\right\}
= \frac{C_h}{n^2}\left\{8L(D)+\frac{5\epsilon'}{2}\left(L(D)^2-L(D)\right)+\frac{4\left(L(D)^2-L(D)\right)}{n}\right\}. ∣ R − R ′ ∣ ≤ n 2 4 { n 2 C h ( 2 L ( D ) ) + 4 5 ϵ ′ C h ( 2 L ( D ) ) + 2 C h L ( D ) } = n 2 C h { 8 L ( D ) + 2 5 ϵ ′ ( L ( D ) 2 − L ( D ) ) + n 4 ( L ( D ) 2 − L ( D ) ) } . We therefore have that
∣ A − A ′ ∣ ≤ ξ n ( 4 + 5 2 ϵ ′ L ( D ) ) + C h n 2 { 68 L ( D ) + 16 + 89 2 ϵ ′ L ( D ) 2 + 2 ϵ ′ L ( D ) + 6 ϵ ′ + 4 ( L ( D ) 2 − L ( D ) ) n + 12 ϵ ′ } . \lvert A-A'\rvert\le\frac{\xi}{n}\left(4+\frac{5}{2}\epsilon'L(D)\right)
+\frac{C_h}{n^2}\left\{68L(D)+16+\frac{89}{2}\epsilon'L(D)^2+2\epsilon'L(D)+6\epsilon'+\frac{4\left(L(D)^2-L(D)\right)}{n}+\frac{12}{\epsilon'}\right\}. ∣ A − A ′ ∣ ≤ n ξ ( 4 + 2 5 ϵ ′ L ( D ) ) + n 2 C h { 68 L ( D ) + 16 + 2 89 ϵ ′ L ( D ) 2 + 2 ϵ ′ L ( D ) + 6 ϵ ′ + n 4 ( L ( D ) 2 − L ( D ) ) + ϵ ′ 12 } . If L ( D ′ ) = L ( D ) − 1 L(D')=L(D)-1 L ( D ′ ) = L ( D ) − 1 D D D D ′ D' D ′ ∣ A ( D ) − A ( D ′ ) ∣ ≤ G ( L ( D ′ ) ) ≤ G ( L ( D ) ) \lvert A(D)-A(D')\rvert\le G(L(D'))\le G(L(D)) ∣ A ( D ) − A ( D ′ )∣ ≤ G ( L ( D ′ )) ≤ G ( L ( D )) G G G D , D ′ D,D' D , D ′
Bound for both cases. Using L ( D ) ≥ 1 L(D)\ge1 L ( D ) ≥ 1 16 ≤ 16 L ( D ) 16\le16L(D) 16 ≤ 16 L ( D ) 2 ϵ ′ L ( D ) + 6 ϵ ′ ≤ 8 ϵ ′ L ( D ) 2 2\epsilon'L(D)+6\epsilon'\le8\epsilon'L(D)^2 2 ϵ ′ L ( D ) + 6 ϵ ′ ≤ 8 ϵ ′ L ( D ) 2
∣ A − A ′ ∣ ≤ 4 ξ n ( 1 + ϵ ′ L ( D ) ) + C h n 2 { 84 L ( D ) + 105 2 ϵ ′ L ( D ) 2 + 4 L ( D ) 2 n + 12 ϵ ′ } ≤ 4 ξ n ( 1 + ϵ ′ L ( D ) ) + C h n 2 { 57 L ( D ) + 57 ϵ ′ L ( D ) 2 + 4 L ( D ) 2 n + 105 2 ϵ ′ } . \begin{aligned}
\lvert A-A'\rvert\le\frac{4\xi}{n}(1+\epsilon'L(D))+\frac{C_h}{n^2}\left\{84L(D)+\frac{105}{2}\epsilon'L(D)^2+\frac{4L(D)^2}{n}+\frac{12}{\epsilon'}\right\} \\
&\le\frac{4\xi}{n}(1+\epsilon'L(D))+\frac{C_h}{n^2}\left\{57L(D)+57\epsilon'L(D)^2+\frac{4L(D)^2}{n}+\frac{105}{2\epsilon'}\right\}.
\end{aligned} ∣ A − A ′ ∣ ≤ n 4 ξ ( 1 + ϵ ′ L ( D )) + n 2 C h { 84 L ( D ) + 2 105 ϵ ′ L ( D ) 2 + n 4 L ( D ) 2 + ϵ ′ 12 } ≤ n 4 ξ ( 1 + ϵ ′ L ( D )) + n 2 C h { 57 L ( D ) + 57 ϵ ′ L ( D ) 2 + n 4 L ( D ) 2 + 2 ϵ ′ 105 } . where the second inequality is from noting that 27 L ( D ) ≤ ( 9 / 2 ) ϵ ′ L ( D ) 2 + 81 / ( 2 ϵ ′ ) 27L(D)\le(9/2)\epsilon'L(D)^2+81/(2\epsilon') 27 L ( D ) ≤ ( 9/2 ) ϵ ′ L ( D ) 2 + 81/ ( 2 ϵ ′ ) ∣ A − A ′ ∣ ≤ G ( L ( D ) ) \lvert A-A'\rvert\le G(L(D)) ∣ A − A ′ ∣ ≤ G ( L ( D )) D ′ D' D ′ LSA ( D ) ≤ G ( L ( D ) ) \operatorname{LSA}(D)\le G(L(D)) LSA ( D ) ≤ G ( L ( D ))
Showing the bounds in (90) , (91) , and (92) . To show (90) , we first consider the interval where the clipping factor is strictly between 0 and 1, i.e.
ξ + 12 C h T n < ∣ x ∣ < ξ + 12 C h T n + 12 C h n ϵ ′ (95) \xi+\frac{12C_hT}{n}<\lvert x\rvert<\xi+\frac{12C_hT}{n}+\frac{12C_h}{n\epsilon'} \tag*{(95)} ξ + n 12 C h T < ∣ x ∣ < ξ + n 12 C h T + n ϵ ′ 12 C h (95) When x x x (95) , writing s = ∣ x ∣ − ξ − 12 C h T / n s=\lvert x\rvert-\xi-12C_hT/n s = ∣ x ∣ − ξ − 12 C h T / n
∣ x − F T ( x ) ∣ = ξ + 12 C h T n + s ( 1 − ϵ ′ T − n ϵ ′ ξ 12 C h − n ϵ ′ s 12 C h ) , 0 ≤ s ≤ 12 C h n ϵ ′ ≤ ξ + 12 C h T n + 3 C h n ϵ ′ ( 1 − ϵ ′ T ) 2 . \lvert x-F_T(x)\rvert=\xi+\frac{12C_hT}{n}+s\left(1-\epsilon'T-\frac{n\epsilon'\xi}{12C_h}-\frac{n\epsilon's}{12C_h}\right),\qquad0\le s\le\frac{12C_h}{n\epsilon'}
\le\xi+\frac{12C_hT}{n}+\frac{3C_h}{n\epsilon'}(1-\epsilon'T)^2. ∣ x − F T ( x )∣ = ξ + n 12 C h T + s ( 1 − ϵ ′ T − 12 C h n ϵ ′ ξ − 12 C h n ϵ ′ s ) , 0 ≤ s ≤ n ϵ ′ 12 C h ≤ ξ + n 12 C h T + n ϵ ′ 3 C h ( 1 − ϵ ′ T ) 2 . The last inequality uses ξ ≥ 0 \xi\ge0 ξ ≥ 0 s ≥ 0 s\ge0 s ≥ 0 x x x (95) , we have that ∣ x − F T ( x ) ∣ ≤ ξ + 12 C h T / n \lvert x-F_T(x)\rvert\le\xi+12C_hT/n ∣ x − F T ( x )∣ ≤ ξ + 12 C h T / n
Next, to show (91) , note that on (95) , we have
F T ′ ( x ) = n ϵ ′ 12 C h ( 2 ∣ x ∣ − ξ − 12 C h T n ) ≤ 2 + ϵ ′ ( n ξ 12 C h + T ) . (96) F_T'(x)=\frac{n\epsilon'}{12C_h}\left(2\lvert x\rvert-\xi-\frac{12C_hT}{n}\right)\le2+\epsilon'\left(\frac{n\xi}{12C_h}+T\right). \tag*{(96)} F T ′ ( x ) = 12 C h n ϵ ′ ( 2 ∣ x ∣ − ξ − n 12 C h T ) ≤ 2 + ϵ ′ ( 12 C h n ξ + T ) . (96) Outside of (95) , the derivative of F T ( x ) F_T(x) F T ( x ) F T F_T F T (96) is a global Lipschitz bound on F T F_T F T (91) .
Finally, we show (92) . Since F T F_T F T F T + 1 F_{T+1} F T + 1 0 ≤ F T + 1 ( r ) ≤ F T ( r ) 0 \leq F_{T+1}(r) \leq F_T(r) 0 ≤ F T + 1 ( r ) ≤ F T ( r ) r ≥ 0 r \geq0 r ≥ 0 x , y x,y x , y
∣ F T ( x ) − F T + 1 ( y ) ∣ ≤ sup r ≥ 0 { F T ( r + 3 C h n ) − F T + 1 ( r ) } , ∣ x − y ∣ ≤ 3 C h n . \lvert F_T(x)-F_{T+1}(y)\rvert\leq\sup_{r\geq0}\left\{F_T\left(r+\frac{3C_h}{n}\right)-F_{T+1}(r)\right\},\quad\lvert x-y\rvert\leq\frac{3C_h}{n}. ∣ F T ( x ) − F T + 1 ( y )∣ ≤ r ≥ 0 sup { F T ( r + n 3 C h ) − F T + 1 ( r ) } , ∣ x − y ∣ ≤ n 3 C h . Substituting u : = n r / ( 12 C h ) u:=nr/(12C_h) u := n r / ( 12 C h )
12 C h n [ ( u + 1 4 ) min { 1 , ϵ ′ ( u + 1 / 4 − b ) + } − u min { 1 , ϵ ′ ( u − b − 1 ) + } ] , (97) \frac{12C_h}{n}\left[\left(u+\frac{1}{4}\right)\min\{1,\epsilon'(u+1/4-b)_+\}-u\min\{1,\epsilon'(u-b-1)_+\}\right], \tag*{(97)} n 12 C h [ ( u + 4 1 ) min { 1 , ϵ ′ ( u + 1/4 − b ) + } − u min { 1 , ϵ ′ ( u − b − 1 ) + } ] , (97) where b : = n ξ / ( 12 C h ) + T b:=n\xi/(12C_h)+T b := n ξ / ( 12 C h ) + T
If ϵ ′ ≤ 4 / 5 \epsilon' \leq4/5 ϵ ′ ≤ 4/5 (97) is maximised at u = b + ( ϵ ′ ) − 1 − 1 / 4 u=b+(\epsilon')^{-1}-1/4 u = b + ( ϵ ′ ) − 1 − 1/4 3 / 2 + 5 ϵ ′ / 4 ( b − 1 / 4 ) 3/2+5\epsilon'/4(b-1/4) 3/2 + 5 ϵ ′ /4 ( b − 1/4 ) ϵ ′ ≥ 4 / 5 \epsilon' \geq4/5 ϵ ′ ≥ 4/5 (97) is maximised at u = b + 1 u=b+1 u = b + 1 b + 5 / 4 b+5/4 b + 5/4
b + 5 4 ≤ 3 2 + 5 ϵ ′ 4 ( b − 1 4 ) b+\frac{5}{4}\leq\frac{3}{2}+\frac{5\epsilon'}{4}\left(b-\frac{1}{4}\right) b + 4 5 ≤ 2 3 + 4 5 ϵ ′ ( b − 4 1 ) since the difference is ( 5 ϵ ′ / 4 − 1 ) ( b − 1 / 4 ) ≥ 0 (5\epsilon'/4-1)(b-1/4)\geq0 ( 5 ϵ ′ /4 − 1 ) ( b − 1/4 ) ≥ 0
It remains to consider when x , y x,y x , y
∣ F T ( x ) − F T + 1 ( y ) ∣ ≤ ∣ x ∣ + ∣ y ∣ = ∣ x − y ∣ ≤ 3 C h n . \lvert F_T(x)-F_{T+1}(y)\rvert\leq\lvert x\rvert+\lvert y\rvert=\lvert x-y\rvert\leq\frac{3C_h}{n}. ∣ F T ( x ) − F T + 1 ( y )∣ ≤ ∣ x ∣ + ∣ y ∣ = ∣ x − y ∣ ≤ n 3 C h . This concludes the proof.
Proof of 12
Proof. The test at each scale uses privacy budget ϵ / J \epsilon/J ϵ / J Lemma 32 , sequential composition and post-processing give ( ϵ , 0 ) (\epsilon,0) ( ϵ , 0 ) O ( d n 2 ) O(dn^2) O ( d n 2 ) O ( J d n 2 ) O(Jdn^2) O ( J d n 2 )
Write θ s j : = E [ h s j ( Z 1 , Z 2 ) ] \theta_{s_j}:=\mathbb{E}[h_{s_j}(Z_1,Z_2)] θ s j := E [ h s j ( Z 1 , Z 2 )] α : = η / ( 3 J ) \alpha:=\eta/(3J) α := η / ( 3 J ) E [ Z ] = 0 \mathbb{E}[Z]=0 E [ Z ] = 0 s j s_j s j s j ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s j s_j\leq\lVert\mathbb{E}[Z]\rVert_2\leq2s_j s j ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s j Lemma 34 , the event in (81) holds with probability at least 1 − α 1-\alpha 1 − α ξ = ξ s j \xi=\xi_{s_j} ξ = ξ s j Lemma 32 with ϵ 0 = ϵ / J \epsilon_0=\epsilon/J ϵ 0 = ϵ / J C h = 2 R ( s j ) C_h=2R(s_j) C h = 2 R ( s j )
∣ U ~ j − θ s j ∣ ≲ s j 2 / n + d / n 2 α + α − 1 / 3 { l s j ( s j + d / n ) n ( 1 + J ϵ ) + R ( s j ) n 2 ( 1 + J 2 ϵ 2 ) + R ( s j ) n 3 ( 1 + J 3 ϵ 3 ) } \lvert\widetilde{U}_j-\theta_{s_j}\rvert\lesssim
\sqrt{\frac{s_j^2/n+d/n^2}{\alpha}}
+\alpha^{-1/3}\left\{
\frac{l_{s_j}(s_j+\sqrt{d/n})}{n}\left(1+\frac{J}{\epsilon}\right)
+\frac{R(s_j)}{n^2}\left(1+\frac{J^2}{\epsilon^2}\right)
+\frac{R(s_j)}{n^3}\left(1+\frac{J^3}{\epsilon^3}\right)
\right\} ∣ U j − θ s j ∣ ≲ α s j 2 / n + d / n 2 + α − 1/3 { n l s j ( s j + d / n ) ( 1 + ϵ J ) + n 2 R ( s j ) ( 1 + ϵ 2 J 2 ) + n 3 R ( s j ) ( 1 + ϵ 3 J 3 ) } with probability at least 1 − 3 α = 1 − η / J 1-3\alpha=1-\eta/J 1 − 3 α = 1 − η / J
Under the condition in (28) with sufficiently large constant and polylogarithmic factors, we thus have with probability at least 1 − η / J 1-\eta/J 1 − η / J
∣ U ~ j − θ s j ∣ ≲ η − 1 / 2 { ℓ n , d , ϵ 4 / 3 ( s j n + d n ) + ℓ n , d , ϵ 7 / 3 ( s j n ϵ + d n 3 / 2 ϵ ) + ℓ n , d , ϵ 10 / 3 d n 2 ϵ 2 + ℓ n , d , ϵ 13 / 3 d n 2 ϵ 3 } ≤ s j 2 4 . |\widetilde U_j-\theta_{s_j}| \lesssim\eta^{-1/2}\left\{\ell_{n,d,\epsilon}^{4/3}\left(\frac{s_j}{\sqrt{n}}+\frac{\sqrt{d}}{n}\right)+\ell_{n,d,\epsilon}^{7/3}\left(\frac{s_j}{n\epsilon}+\frac{\sqrt{d}}{n^{3/2}\epsilon}\right)+\frac{\ell_{n,d,\epsilon}^{10/3}\sqrt{d}}{n^2\epsilon^2}+\frac{\ell_{n,d,\epsilon}^{13/3}\sqrt{d}}{n^2\epsilon^3}\right\}\leq\frac{s_j^2}{4}. ∣ U j − θ s j ∣ ≲ η − 1/2 { ℓ n , d , ϵ 4/3 ( n s j + n d ) + ℓ n , d , ϵ 7/3 ( n ϵ s j + n 3/2 ϵ d ) + n 2 ϵ 2 ℓ n , d , ϵ 10/3 d + n 2 ϵ 3 ℓ n , d , ϵ 13/3 d } ≤ 4 s j 2 . Under the null, these bounds apply at every scale. By (98) and a union bound, with probability at least 1 − η 1-\eta 1 − η
U ~ j ≤ s j 2 / 16 + s j 2 / 4 = 5 s j 2 / 16 < s j 2 / 2 \widetilde U_j\leq s_j^2/16+s_j^2/4=5s_j^2/16<s_j^2/2 U j ≤ s j 2 /16 + s j 2 /4 = 5 s j 2 /16 < s j 2 /2 for all j j j
Under an alternative with r ~ ≤ ∥ E [ Z ] ∥ 2 ≤ λ \widetilde r\leq\|\mathbb{E}[Z]\|_2\leq\sqrt{\lambda} r ≤ ∥ E [ Z ] ∥ 2 ≤ λ k k k s k ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s k s_k\leq\|\mathbb{E}[Z]\|_2\leq2s_k s k ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s k 1 − η / J 1-\eta/J 1 − η / J
U ~ k ≥ ∥ E [ Z ] ∥ 2 2 − s k 2 / 16 − s k 2 / 4 ≥ 11 s k 2 / 16 > s k 2 / 2. \widetilde U_k\geq\|\mathbb{E}[Z]\|_2^2-s_k^2/16-s_k^2/4\geq11s_k^2/16>s_k^2/2. U k ≥ ∥ E [ Z ] ∥ 2 2 − s k 2 /16 − s k 2 /4 ≥ 11 s k 2 /16 > s k 2 /2. The test therefore rejects with probability at least 1 − η / J ≥ 1 − η 1-\eta/J\geq1-\eta 1 − η / J ≥ 1 − η c μ > 0 c_\mu>0 c μ > 0 r ~ ≤ c μ \widetilde r\leq c_\mu r ≤ c μ C 1 ≥ 1 / c μ C_1\geq1/c_\mu C 1 ≥ 1/ c μ ∥ β ∥ 2 ≥ C 1 r ~ \|\beta\|_2\geq C_1\widetilde r ∥ β ∥ 2 ≥ C 1 r ∥ E [ Z ] ∥ 2 ≥ r ~ \|\mathbb{E}[Z]\|_2\geq\widetilde r ∥ E [ Z ] ∥ 2 ≥ r ∥ E [ Z ] ∥ 2 ≤ λ \|\mathbb{E}[Z]\|_2\leq\sqrt{\lambda} ∥ E [ Z ] ∥ 2 ≤ λ ∣ tanh ∣ ≤ 1 |\tanh|\leq1 ∣ tanh ∣ ≤ 1 E [ X X ⊤ ] ⪯ λ I \mathbb{E}[XX^\top]\preceq\lambda I E [ X X ⊤ ] ⪯ λ I E [ Z ] \mathbb{E}[Z] E [ Z ] β \beta β
The following lemma bounds the clipping bias and the non-private variance of the proposed kernel, and provides the local stability required by the private U-statistic mechanism.
Lemma 34. Suppose we have i.i.d. { Z i : = X i tanh ( Y i ) } i = 1 n \{Z_i:=X_i\tanh(Y_i)\}_{i=1}^n { Z i := X i tanh ( Y i ) } i = 1 n P L i n \mathcal{P}_{\mathrm{Lin}} P Lin P G L M ( F ) \mathcal{P}_{\mathrm{GLM}}(\mathcal{F}) P GLM ( F ) F \mathcal{F} F α ∈ ( 0 , 1 ) \alpha\in(0,1) α ∈ ( 0 , 1 ) 0 < s ≤ λ 0<s\leq\sqrt{\lambda} 0 < s ≤ λ E [ Z ] = 0 \mathbb{E}[Z]=0 E [ Z ] = 0 s ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s s\leq\|\mathbb{E}[Z]\|_2\leq2s s ≤ ∥ E [ Z ] ∥ 2 ≤ 2 s
h ^ i ( s ) : = 1 n − 1 ∑ j ≠ i h s ( Z i , Z j ) , U s : = 1 n ∑ i = 1 n h ^ i ( s ) . \widehat h_i^{(s)}:=\frac{1}{n-1}\sum_{j\neq i}h_s(Z_i,Z_j),\qquad U_s:=\frac{1}{n}\sum_{i=1}^n\widehat h_i^{(s)}. h i ( s ) := n − 1 1 j = i ∑ h s ( Z i , Z j ) , U s := n 1 i = 1 ∑ n h i ( s ) . For C c l i p C_{\mathrm{clip}} C clip h s h_s h s (27) satisfies
∣ E [ h s ( Z 1 , Z 2 ) ] − ∥ E [ Z ] ∥ 2 2 ∣ ≤ s 2 16 (98) \left|\mathbb{E}[h_s(Z_1,Z_2)]-\|\mathbb{E}[Z]\|_2^2\right|\leq\frac{s^2}{16} \tag*{(98)} E [ h s ( Z 1 , Z 2 )] − ∥ E [ Z ] ∥ 2 2 ≤ 16 s 2 (98) and
Var ( U s ( Z 1 : n ) ) ≲ s 2 n + d n 2 . (99) \operatorname{Var}(U_s(Z_{1:n}))\lesssim\frac{s^2}{n}+\frac{d}{n^2}. \tag*{(99)} Var ( U s ( Z 1 : n )) ≲ n s 2 + n 2 d . (99) Moreover, there exists a constant C > 0 C>0 C > 0
P ( max i ∈ [ n ] ∣ h ^ i ( s ) − U s ( Z 1 : n ) ∣ ≤ C ℓ s { s + d n } ) ≥ 1 − α . (100) \mathbb{P}\left(\max_{i\in[n]}\left|\widehat h_i^{(s)}-U_s(Z_{1:n})\right|\leq C\ell_s\left\{s+\sqrt{\frac{d}{n}}\right\}\right)\geq1-\alpha. \tag*{(100)} P ( i ∈ [ n ] max h i ( s ) − U s ( Z 1 : n ) ≤ C ℓ s { s + n d } ) ≥ 1 − α . (100) Proof. For i ∈ [ n ] i\in[n] i ∈ [ n ] R i : = Z i − E [ Z ] R_i:=Z_i-\mathbb{E}[Z] R i := Z i − E [ Z ]
Z i ⊤ Z j = ∥ E [ Z ] ∥ 2 2 + E [ Z ] ⊤ ( R i + R j ) + R i ⊤ R j . Z_i^\top Z_j=\|\mathbb{E}[Z]\|_2^2+\mathbb{E}[Z]^\top(R_i+R_j)+R_i^\top R_j. Z i ⊤ Z j = ∥ E [ Z ] ∥ 2 2 + E [ Z ] ⊤ ( R i + R j ) + R i ⊤ R j . By sub-Gaussianity of the second term and Lemma 37 , we have that
P ( ∣ Z i T Z j ∣ ≥ C C X , λ { d t + t } ) ≤ e − t (101) \mathbb{P}\left(\left|Z_i^{\mathsf{T}}Z_j\right| \ge C_{CX,\lambda}\{\sqrt{d}t+t\}\right) \le e^{-t} \tag*{(101)} P ( Z i T Z j ≥ C C X , λ { d t + t } ) ≤ e − t (101) for all t ≥ 1 t \ge1 t ≥ 1 u = C C X , λ { d t + t } u=C_{CX,\lambda}\{\sqrt{d}t+t\} u = C C X , λ { d t + t } C C X , λ { d ( 2 k t ) + 2 k t } ≤ 2 k C C X , λ { d t + t } = 2 k u C_{CX,\lambda}\{\sqrt{d(2^k t)+2^k t}\}\le2^k C_{CX,\lambda}\{\sqrt{d}t+t\}=2^k u C C X , λ { d ( 2 k t ) + 2 k t } ≤ 2 k C C X , λ { d t + t } = 2 k u (101) give
E [ ( Z 1 T Z 2 ) 2 1 { ∣ Z 1 T Z 2 ∣ > u } ] ≤ ∑ k = 0 ∞ ( 2 k + 1 u ) 2 P { ∣ Z 1 T Z 2 ∣ > 2 k u } \mathbb{E}\left[(Z_1^{\mathsf{T}}Z_2)^2\mathbb{1}\{|Z_1^{\mathsf{T}}Z_2|>u\}\right]\le\sum_{k=0}^{\infty}(2^{k+1}u)^2\mathbb{P}\{|Z_1^{\mathsf{T}}Z_2|>2^k u\} E [ ( Z 1 T Z 2 ) 2 1 { ∣ Z 1 T Z 2 ∣ > u } ] ≤ k = 0 ∑ ∞ ( 2 k + 1 u ) 2 P { ∣ Z 1 T Z 2 ∣ > 2 k u } ≲ 24 u 2 ∑ k = 0 ∞ 4 k exp ( − 2 k t ) \lesssim24u^2\sum_{k=0}^{\infty}4^k\exp(-2^k t) ≲ 24 u 2 k = 0 ∑ ∞ 4 k exp ( − 2 k t ) ≲ ( d t + t 2 ) exp ( − t ) . \lesssim(dt+t^2)\exp(-t). ≲ ( d t + t 2 ) exp ( − t ) . Since l s ≥ 1 + log { ( d + 1 ) / s } l_s\ge1+\log\{(d+1)/s\} l s ≥ 1 + log {( d + 1 ) / s } t = c 1 l s t=c_1l_s t = c 1 l s c 1 c_1 c 1 C c l i p C_{\mathrm{clip}} C clip (27) ensures that R ( s ) ≥ u R(s)\ge u R ( s ) ≥ u
E [ ( h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ) 2 ] ≤ E [ ( Z 1 T Z 2 ) 2 1 { ∣ Z 1 T Z 2 ∣ > R ( s ) } ] ≤ s 4 256 . (102) \mathbb{E}\left[(h_s(Z_1,Z_2)-Z_1^{\mathsf{T}}Z_2)^2\right]\le\mathbb{E}\left[(Z_1^{\mathsf{T}}Z_2)^2\mathbb{1}\{|Z_1^{\mathsf{T}}Z_2|>R(s)\}\right]\le\frac{s^4}{256}. \tag*{(102)} E [ ( h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ) 2 ] ≤ E [ ( Z 1 T Z 2 ) 2 1 { ∣ Z 1 T Z 2 ∣ > R ( s )} ] ≤ 256 s 4 . (102) Since E [ Z 1 T Z 2 ] = ∥ E [ Z ] ∥ 2 2 \mathbb{E}[Z_1^{\mathsf{T}}Z_2]=\|\mathbb{E}[Z]\|_2^2 E [ Z 1 T Z 2 ] = ∥ E [ Z ] ∥ 2 2
∣ E [ h s ( Z 1 , Z 2 ) ] − ∥ E [ Z ] ∥ 2 2 ∣ ≤ { E [ ( h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ) 2 ] } 1 / 2 ≤ s 2 16 . \left|\mathbb{E}[h_s(Z_1,Z_2)]-\|\mathbb{E}[Z]\|_2^2\right|\le\left\{\mathbb{E}\left[(h_s(Z_1,Z_2)-Z_1^{\mathsf{T}}Z_2)^2\right]\right\}^{1/2}\le\frac{s^2}{16}. E [ h s ( Z 1 , Z 2 )] − ∥ E [ Z ] ∥ 2 2 ≤ { E [ ( h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ) 2 ] } 1/2 ≤ 16 s 2 . This proves the clipping bias in (98) .
To obtain the bound on the variance in (99) , let
v R ( z ) : = E [ h s ( z , Z 2 ) ] , v ( z ) : = E [ z T Z 2 ] = z T E [ Z ] . v_R(z):=\mathbb{E}[h_s(z,Z_2)],\qquad v(z):=\mathbb{E}[z^{\mathsf{T}}Z_2]=z^{\mathsf{T}}\mathbb{E}[Z]. v R ( z ) := E [ h s ( z , Z 2 )] , v ( z ) := E [ z T Z 2 ] = z T E [ Z ] . Since Z 2 Z_2 Z 2 Z 1 Z_1 Z 1
v R ( Z 1 ) − v ( Z 1 ) = E [ h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ∣ Z 1 ] , v_R(Z_1)-v(Z_1)=\mathbb{E}[h_s(Z_1,Z_2)-Z_1^{\mathsf{T}}Z_2\mid Z_1], v R ( Z 1 ) − v ( Z 1 ) = E [ h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ∣ Z 1 ] , so by a conditional Jensen’s inequality and (102) , we have
E [ ( v R ( Z 1 ) − v ( Z 1 ) ) 2 ] ≤ E [ E [ ( h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ) 2 ∣ Z 1 ] ] = E [ ( h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ) 2 ] ≤ s 4 256 . \begin{aligned}
\mathbb{E}\left[(v_R(Z_1)-v(Z_1))^2\right]\le\mathbb{E}\left[\mathbb{E}\left[(h_s(Z_1,Z_2)-Z_1^{\mathsf{T}}Z_2)^2\mid Z_1\right]\right] \\
&=\mathbb{E}\left[(h_s(Z_1,Z_2)-Z_1^{\mathsf{T}}Z_2)^2\right]\le\frac{s^4}{256}.
\end{aligned} E [ ( v R ( Z 1 ) − v ( Z 1 ) ) 2 ] ≤ E [ E [ ( h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ) 2 ∣ Z 1 ] ] = E [ ( h s ( Z 1 , Z 2 ) − Z 1 T Z 2 ) 2 ] ≤ 256 s 4 . Writing Ω : = E [ Z 1 Z 1 T ] \Omega:=\mathbb{E}[Z_1Z_1^{\mathsf{T}}] Ω := E [ Z 1 Z 1 T ] Ω ⪯ E [ X X T ] \Omega\preceq\mathbb{E}[XX^{\mathsf{T}}] Ω ⪯ E [ X X T ] Z 1 = X 1 tanh ( Y 1 ) Z_1=X_1\tanh(Y_1) Z 1 = X 1 tanh ( Y 1 ) ∣ tanh ( Y 1 ) ∣ ≤ 1 |\tanh(Y_1)|\le1 ∣ tanh ( Y 1 ) ∣ ≤ 1
Var { v ( Z 1 ) } ≤ E [ ( Z 1 T E [ Z ] ) 2 ] = E [ Z ] T Ω E [ Z ] ≤ λ ∥ E [ Z ] ∥ 2 2 ≤ 4 λ s 2 . \operatorname{Var}\{v(Z_1)\}\le\mathbb{E}[(Z_1^{\mathsf{T}}\mathbb{E}[Z])^2]=\mathbb{E}[Z]^{\mathsf{T}}\Omega\mathbb{E}[Z]\le\lambda\|\mathbb{E}[Z]\|_2^2\le4\lambda s^2. Var { v ( Z 1 )} ≤ E [( Z 1 T E [ Z ] ) 2 ] = E [ Z ] T Ω E [ Z ] ≤ λ ∥ E [ Z ] ∥ 2 2 ≤ 4 λ s 2 . Using Var ( A + B ) ≤ 2 Var ( A ) + 2 E [ B 2 ] \operatorname{Var}(A+B)\le2\operatorname{Var}(A)+2\mathbb{E}[B^2] Var ( A + B ) ≤ 2 Var ( A ) + 2 E [ B 2 ] s ≤ λ s\le\sqrt{\lambda} s ≤ λ
Var { v R ( Z 1 ) } ≤ 2 Var { v ( Z 1 ) } + 2 E [ ( v R ( Z 1 ) − v ( Z 1 ) ) 2 ] ≲ s 2 . \operatorname{Var}\{v_R(Z_1)\}\le2\operatorname{Var}\{v(Z_1)\}+2\mathbb{E}[(v_R(Z_1)-v(Z_1))^2]\lesssim s^2. Var { v R ( Z 1 )} ≤ 2 Var { v ( Z 1 )} + 2 E [( v R ( Z 1 ) − v ( Z 1 ) ) 2 ] ≲ s 2 . Moreover, independence of Z 1 Z_1 Z 1 Z 2 Z_2 Z 2
E [ ( Z 1 T Z 2 ) 2 ] = E [ tr ( Z 1 Z 1 T Z 2 Z 2 T ) ] = tr ( E [ Z 1 Z 1 T ] E [ Z 2 Z 2 T ] ) ≲ λ 2 d . \mathbb{E}[(Z_1^{\mathsf{T}}Z_2)^2] = \mathbb{E}\left[\operatorname{tr}(Z_1Z_1^{\mathsf{T}}Z_2Z_2^{\mathsf{T}})\right]
= \operatorname{tr}\left(\mathbb{E}[Z_1Z_1^{\mathsf{T}}]\mathbb{E}[Z_2Z_2^{\mathsf{T}}]\right) \lesssim\lambda^2 d. E [( Z 1 T Z 2 ) 2 ] = E [ tr ( Z 1 Z 1 T Z 2 Z 2 T ) ] = tr ( E [ Z 1 Z 1 T ] E [ Z 2 Z 2 T ] ) ≲ λ 2 d . Since ∣ h s ( Z 1 , Z 2 ) ∣ ≤ ∣ Z 1 T Z 2 ∣ |h_s(Z_1,Z_2)| \leq|Z_1^{\mathsf{T}}Z_2| ∣ h s ( Z 1 , Z 2 ) ∣ ≤ ∣ Z 1 T Z 2 ∣ Var { h s ( Z 1 , Z 2 ) } ≲ d \operatorname{Var}\{h_s(Z_1,Z_2)\} \lesssim d Var { h s ( Z 1 , Z 2 )} ≲ d [40 Chapter 5] ) gives
Var ( U s ) ≤ 4 ( n − 2 ) n ( n − 1 ) Var ( v R ( Z 1 ) ) + 2 n ( n − 1 ) Var ( h s ( Z 1 , Z 2 ) ) ≲ ( s 2 n + d n 2 ) , \operatorname{Var}(U_s) \leq\frac{4(n-2)}{n(n-1)}\operatorname{Var}(v_R(Z_1))+\frac{2}{n(n-1)}\operatorname{Var}(h_s(Z_1,Z_2))\lesssim\left(\frac{s^2}{n}+\frac{d}{n^2}\right), Var ( U s ) ≤ n ( n − 1 ) 4 ( n − 2 ) Var ( v R ( Z 1 )) + n ( n − 1 ) 2 Var ( h s ( Z 1 , Z 2 )) ≲ ( n s 2 + n 2 d ) , which proves (99) .
Finally, we prove (100) . The concentration argument in the proof of the row sum in Lemma 23 , with t = log ( 6 n / α ) t=\log(6n/\alpha) t = log ( 6 n / α ) i i i
P ( max i ∈ [ n ] ∣ 1 n − 1 ∑ j ≠ i Z i T Z j − ∥ E [ Z ] ∥ 2 ∣ ≤ C l s ( s + d n ) ) ≥ 1 − α / 2. \mathbb{P}\left(\max_{i\in[n]}\left|\frac{1}{n-1}\sum_{j\ne i}Z_i^{\mathsf{T}}Z_j-\|\mathbb{E}[Z]\|^2\right|\leq C l_s\left(s+\sqrt{\frac{d}{n}}\right)\right)\geq1-\alpha/2. P i ∈ [ n ] max n − 1 1 j = i ∑ Z i T Z j − ∥ E [ Z ] ∥ 2 ≤ C l s ( s + n d ) ≥ 1 − α /2. Taking t = c l s t=c l_s t = c l s (101) and choosing c c c C c l i p C_{\mathrm{clip}} C clip
P ( max i < j ∣ Z i T Z j ∣ > R ( s ) ) ≤ ( n 2 ) exp ( − c l s ) ≤ α / 2. \mathbb{P}\left(\max_{i<j}|Z_i^{\mathsf{T}}Z_j|>R(s)\right)\leq\binom{n}{2}\exp(-c l_s)\leq\alpha/2. P ( i < j max ∣ Z i T Z j ∣ > R ( s ) ) ≤ ( 2 n ) exp ( − c l s ) ≤ α /2. On the intersection of these events, no inner product is clipped, so h ^ i ( s ) = u i / ( n − 1 ) \widehat{h}_i^{(s)}=u_i/(n-1) h i ( s ) = u i / ( n − 1 ) U s U_s U s
max i ∣ h ^ i ( s ) − U s ∣ ≤ 2 max i ∣ h ^ i ( s ) − ∥ E [ Z ] ∥ 2 ∣ ≤ C l s ( s + d / n ) . \max_i|\widehat{h}_i^{(s)}-U_s|\leq2\max_i|\widehat{h}_i^{(s)}-\|\mathbb{E}[Z]\|^2|\leq C l_s\left(s+\sqrt{d/n}\right). i max ∣ h i ( s ) − U s ∣ ≤ 2 i max ∣ h i ( s ) − ∥ E [ Z ] ∥ 2 ∣ ≤ C l s ( s + d / n ) . A union bound argument then proves (100) .
Proof of 13
Proof. Privacy and computation. Let D , D ′ D,D' D , D ′ k k k
∣ T ( D ) − T ( D ′ ) ∣ = ∣ max j ∈ [ d ] ∣ 1 n ∑ i = 1 n [ X i j ] − B B tanh ( Y i ) ∣ − max j ∈ [ d ] ∣ 1 n ∑ i = 1 n [ X i j ′ ] − B B tanh ( Y i ′ ) ∣ ∣ ≤ max j ∈ [ d ] ∣ 1 n ∑ i = 1 n { [ X i j ] − B B tanh ( Y i ) − [ X i j ′ ] − B B tanh ( Y i ′ ) } ∣ = 1 n max j ∈ [ d ] ∣ [ X k j ] − B B tanh ( Y k ) − [ X k j ′ ] − B B tanh ( Y k ′ ) ∣ ≤ 2 B n . \begin{aligned}
|T(D)-T(D')|=\left|\max_{j\in[d]}\left|\frac{1}{n}\sum_{i=1}^{n}[X_{ij}]_{-B}^{B}\tanh(Y_i)\right|-\max_{j\in[d]}\left|\frac{1}{n}\sum_{i=1}^{n}[X'_{ij}]_{-B}^{B}\tanh(Y'_i)\right|\right| \\
&\leq\max_{j\in[d]}\left|\frac{1}{n}\sum_{i=1}^{n}\left\{[X_{ij}]_{-B}^{B}\tanh(Y_i)-[X'_{ij}]_{-B}^{B}\tanh(Y'_i)\right\}\right| \\
&=\frac{1}{n}\max_{j\in[d]}\left|[X_{kj}]_{-B}^{B}\tanh(Y_k)-[X'_{kj}]_{-B}^{B}\tanh(Y'_k)\right| \\
&\leq\frac{2B}{n}.
\end{aligned} ∣ T ( D ) − T ( D ′ ) ∣ = j ∈ [ d ] max n 1 i = 1 ∑ n [ X ij ] − B B tanh ( Y i ) − j ∈ [ d ] max n 1 i = 1 ∑ n [ X ij ′ ] − B B tanh ( Y i ′ ) ≤ j ∈ [ d ] max n 1 i = 1 ∑ n { [ X ij ] − B B tanh ( Y i ) − [ X ij ′ ] − B B tanh ( Y i ′ ) } = n 1 j ∈ [ d ] max [ X k j ] − B B tanh ( Y k ) − [ X k j ′ ] − B B tanh ( Y k ′ ) ≤ n 2 B . Since the sensitivity of T T T 2 B / n 2B/n 2 B / n
Computing the clipped coordinate means and taking the maximum over d d d O ( n d ) O(nd) O ( n d )
Clipping bias and concentration. We have that
∣ E [ [ X j ] − B B tanh ( Y ) ] − E [ X j tanh ( Y ) ] ∣ ≤ E [ { ∣ X j ∣ − B } + ] = ∫ B ∞ P ( ∣ X j ∣ > t ) d t ≤ 2 ∫ B ∞ exp ( − t 2 / C X 2 ) d t ≤ C X 2 B exp ( − B 2 / C X 2 ) = b B (103) \left|\mathbb{E}\left[[X_j]^B_{-B}\tanh(Y)\right]-\mathbb{E}\left[X_j\tanh(Y)\right]\right| \leq\mathbb{E}\left[\{|X_j|-B\}_{+}\right]
= \int_B^\infty\mathbb{P}(|X_j|>t)\,\mathrm{d}t \leq2\int_B^\infty\exp\left(-t^2/C_X^2\right)\,\mathrm{d}t
\leq\frac{C_X^2}{B}\exp\left(-B^2/C_X^2\right)=b_B \tag*{(103)} E [ [ X j ] − B B tanh ( Y ) ] − E [ X j tanh ( Y ) ] ≤ E [ { ∣ X j ∣ − B } + ] = ∫ B ∞ P ( ∣ X j ∣ > t ) d t ≤ 2 ∫ B ∞ exp ( − t 2 / C X 2 ) d t ≤ B C X 2 exp ( − B 2 / C X 2 ) = b B (103) where the tail bound in the second line is due to Assumption 1.
Since
Var ( [ X j ] − B B tanh ( Y ) ) ≤ E [ ( [ X j ] − B B tanh ( Y ) ) 2 ] ≤ E [ X j 2 ] ≤ λ ‾ , \operatorname{Var}\left([X_j]^B_{-B}\tanh(Y)\right)\leq\mathbb{E}\left[([X_j]^B_{-B}\tanh(Y))^2\right]\leq\mathbb{E}[X_j^2]\leq\overline{\lambda}, Var ( [ X j ] − B B tanh ( Y ) ) ≤ E [ ([ X j ] − B B tanh ( Y ) ) 2 ] ≤ E [ X j 2 ] ≤ λ , by Bernstein’s inequality and a union bound over the d d d
P ( max j ∈ [ d ] ∣ 1 n ∑ i = 1 n [ X i j ] − B B tanh ( Y i ) − E [ [ X j ] − B B tanh ( Y ) ] ∣ > q B ) ≤ 2 d exp ( − n q B 2 2 { λ ‾ + ( 2 B / 3 ) q B } ) ≤ η 2 . (104) \mathbb{P}\left(\max_{j\in[d]}\left|\frac{1}{n}\sum_{i=1}^{n}[X_{ij}]^B_{-B}\tanh(Y_i)-\mathbb{E}\left[[X_j]^B_{-B}\tanh(Y)\right]\right|>q_B\right)\leq2d\exp\left(-\frac{nq_B^2}{2\{\overline{\lambda}+(2B/3)q_B\}}\right)\leq\frac{\eta}{2}. \tag*{(104)} P ( j ∈ [ d ] max n 1 i = 1 ∑ n [ X ij ] − B B tanh ( Y i ) − E [ [ X j ] − B B tanh ( Y ) ] > q B ) ≤ 2 d exp ( − 2 { λ + ( 2 B /3 ) q B } n q B 2 ) ≤ 2 η . (104) Type-I error. Under the null, we have max j ∈ [ d ] ∣ E [ [ X j ] − B B tanh ( Y ) ] ∣ ≤ b B \max_{j\in[d]}\left|\mathbb{E}\left[[X_j]^B_{-B}\tanh(Y)\right]\right|\leq b_B max j ∈ [ d ] E [ [ X j ] − B B tanh ( Y ) ] ≤ b B (103) . By (104) , we have that T ≤ b B + q B T\leq b_B+q_B T ≤ b B + q B 1 − η / 2 1-\eta/2 1 − η /2 P ( W > log ( 1 / η ) ) = η / 2 \mathbb{P}(W>\log(1/\eta))=\eta/2 P ( W > log ( 1/ η )) = η /2 η \eta η
Type-II error. Suppose that ∥ β ∥ 2 ≠ 0 \lVert\beta\rVert_2\ne0 ∥ β ∥ 2 = 0 β = ∥ β ∥ 2 u \beta=\lVert\beta\rVert_2u β = ∥ β ∥ 2 u V : = u ⊤ X V:=u^\top X V := u ⊤ X m ν m_\nu m ν s ϑ s_\vartheta s ϑ Lemma 22 , it implies that
max j ∈ [ d ] ∣ E [ [ X j ] − B B tanh ( Y ) ] ∣ ≥ ∥ E [ X tanh ( Y ) ] ∥ ∞ − b B ≥ ∣ u ⊤ E [ X tanh ( Y ) ] ∣ ∥ u ∥ 1 − b B ≥ c μ s ∗ min { ∥ β ∥ 2 , 1 } − b B . \max_{j\in[d]}\left|\mathbb{E}\left[[X_j]^B_{-B}\tanh(Y)\right]\right|\geq\left\lVert\mathbb{E}[X\tanh(Y)]\right\rVert_\infty-b_B
\geq\frac{\left|u^\top\mathbb{E}[X\tanh(Y)]\right|}{\lVert u\rVert_1}-b_B
\geq\frac{c_\mu}{\sqrt{s^*}}\min\{\lVert\beta\rVert_2,1\}-b_B. j ∈ [ d ] max E [ [ X j ] − B B tanh ( Y ) ] ≥ ∥ E [ X tanh ( Y )] ∥ ∞ − b B ≥ ∥ u ∥ 1 u ⊤ E [ X tanh ( Y )] − b B ≥ s ∗ c μ min {∥ β ∥ 2 , 1 } − b B . where the first inequality is due to (103) and the third uses (56) . By a union bound with (104) and P ( W < − log ( 1 / η ) ) = η / 2 \mathbb{P}(W<-\log(1/\eta))=\eta/2 P ( W < − log ( 1/ η )) = η /2
T + 2 B n ϵ W ≥ c μ s ∗ min { ∥ β ∥ 2 , 1 } − b B − q B − 2 B log ( 1 / η ) n ϵ (105) T+\frac{2B}{n\epsilon}W\geq\frac{c_\mu}{\sqrt{s^*}}\min\{\lVert\beta\rVert_2,1\}-b_B-q_B-\frac{2B\log(1/\eta)}{n\epsilon} \tag*{(105)} T + n ϵ 2 B W ≥ s ∗ c μ min {∥ β ∥ 2 , 1 } − b B − q B − n ϵ 2 B log ( 1/ η ) (105) with probability at least 1 − η 1-\eta 1 − η B B B b B b_B b B q B q_B q B
b B + q B + 2 B log ( 1 / η ) n ϵ ≲ log ( 4 d / η ) n + log ( 2 n ) log ( 4 d / η ) n + log ( 2 n ) log ( 1 / η ) n ϵ . b_B+q_B+\frac{2B\log(1/\eta)}{n\epsilon}\lesssim\sqrt{\frac{\log(4d/\eta)}{n}}+\frac{\sqrt{\log(2n)}\log(4d/\eta)}{n}+\frac{\sqrt{\log(2n)}\log(1/\eta)}{n\epsilon}. b B + q B + n ϵ 2 B log ( 1/ η ) ≲ n log ( 4 d / η ) + n log ( 2 n ) log ( 4 d / η ) + n ϵ log ( 2 n ) log ( 1/ η ) . Under (29) with a sufficiently large constant C > 0 C>0 C > 0 (105) exceeds the rejection threshold, and hence the test has type-II error at most η \eta η
Minimax rates for sparse mean testing
We can obtain the minimax rates for sparse mean testing by making some modifications to our existing arguments. Fix C Z ∈ ( 0 , ∞ ) C_Z \in(0,\infty) C Z ∈ ( 0 , ∞ )
P m e a n : = { P : ∥ E P [ Z ] ∥ 2 ≤ 1 , ∥ Z − E P [ Z ] ∥ ψ 2 ≤ C Z } . \mathcal{P}_{\mathrm{mean}} := \left\{P : \left\|\mathbb{E}_P[Z]\right\|_2 \le1,\ \left\|Z-\mathbb{E}_P[Z]\right\|_{\psi_2} \le C_Z\right\}. P mean := { P : ∥ E P [ Z ] ∥ 2 ≤ 1 , ∥ Z − E P [ Z ] ∥ ψ 2 ≤ C Z } . For 1 ≤ s ∗ ≤ d 1 \le s^* \le d 1 ≤ s ∗ ≤ d 0 < r < 1 0 < r < 1 0 < r < 1
P 0 : = { P ∈ P m e a n : E P [ Z ] = 0 } , P 1 ( s ∗ , r , 1 ) : = { P ∈ P m e a n : ∥ E P [ Z ] ∥ 2 ≥ r , ∥ E P [ Z ] ∥ 0 ≤ s ∗ } . \mathcal{P}_0 := \left\{P \in\mathcal{P}_{\mathrm{mean}} : \mathbb{E}_P[Z]=0\right\},\qquad\mathcal{P}_1(s^*,r,1) := \left\{P \in\mathcal{P}_{\mathrm{mean}} : \left\|\mathbb{E}_P[Z]\right\|_2 \ge r,\ \left\|\mathbb{E}_P[Z]\right\|_0 \le s^*\right\}. P 0 := { P ∈ P mean : E P [ Z ] = 0 } , P 1 ( s ∗ , r , 1 ) := { P ∈ P mean : ∥ E P [ Z ] ∥ 2 ≥ r , ∥ E P [ Z ] ∥ 0 ≤ s ∗ } . The bound ∥ E P [ Z ] ∥ 2 ≤ 1 \left\|\mathbb{E}_P[Z]\right\|_2 \le1 ∥ E P [ Z ] ∥ 2 ≤ 1 ∥ E P [ Z ] ∥ 2 ≤ M \left\|\mathbb{E}_P[Z]\right\|_2 \le M ∥ E P [ Z ] ∥ 2 ≤ M M > 0 M>0 M > 0 M M M
We define the minimax separation radius as
r ∗ ( n , d , s ∗ , ϵ , δ , P m e a n , 1 ) : = inf { r ∈ ( 0 , 1 ) : ∃ ( ϵ , δ ) -DP test ϕ such that sup P ∈ P 0 P ⊗ n ( ϕ = 1 ) + sup P ∈ P 1 ( s ∗ , r , 1 ) P ⊗ n ( ϕ = 0 ) ≤ 1 3 } . r^*(n,d,s^*,\epsilon,\delta,\mathcal{P}_{\mathrm{mean}},1):=\inf\left\{r\in(0,1):\exists\ (\epsilon,\delta)\text{-DP test }\phi\text{ such that }\sup_{P\in\mathcal{P}_0}P^{\otimes n}(\phi=1)+\sup_{P\in\mathcal{P}_1(s^*,r,1)}P^{\otimes n}(\phi=0)\le\frac{1}{3}\right\}. r ∗ ( n , d , s ∗ , ϵ , δ , P mean , 1 ) := inf { r ∈ ( 0 , 1 ) : ∃ ( ϵ , δ ) -DP test ϕ such that P ∈ P 0 sup P ⊗ n ( ϕ = 1 ) + P ∈ P 1 ( s ∗ , r , 1 ) sup P ⊗ n ( ϕ = 0 ) ≤ 3 1 } . Recall the definitions of ϱ s ∗ , d \varrho_{s^*,d} ϱ s ∗ , d ℓ n , d , ϵ \ell_{n,d,\epsilon} ℓ n , d , ϵ (4) .
Proposition 35. There exist constants c P c_P c P c δ c_\delta c δ C Z C_Z C Z s ∗ ∈ [ d ] s^*\in[d] s ∗ ∈ [ d ] ℓ n , d , ϵ 3 ϱ s ∗ , d ≤ c P \ell_{n,d,\epsilon}^3\varrho_{s^*,d}\le c_P ℓ n , d , ϵ 3 ϱ s ∗ , d ≤ c P 0 ≤ δ ≤ c δ ϵ 0\le\delta\le c_\delta\epsilon 0 ≤ δ ≤ c δ ϵ
ϱ s ∗ , d ≲ r ∗ ( n , d , s ∗ , ϵ , δ , P m e a n , 1 ) ≲ ℓ n , d , ϵ 3 ϱ s ∗ , d . \varrho_{s^*,d}\lesssim r^*(n,d,s^*,\epsilon,\delta,\mathcal{P}_{\mathrm{mean}},1)\lesssim\ell_{n,d,\epsilon}^3\varrho_{s^*,d}. ϱ s ∗ , d ≲ r ∗ ( n , d , s ∗ , ϵ , δ , P mean , 1 ) ≲ ℓ n , d , ϵ 3 ϱ s ∗ , d . The minimax rates for sparse mean testing are therefore the same as those for signal detection in linear and binary regression in (5) .
Our test for the sparse regime is based on testing the sample mean of clipped sparse projections of the observations. Let K : = { 2 j : 0 ≤ j ≤ ⌊ log 2 d ⌋ } ∪ { d } \mathcal{K}:=\{2^j:0\le j\le\lfloor\log_2 d\rfloor\}\cup\{d\} K := { 2 j : 0 ≤ j ≤ ⌊ log 2 d ⌋} ∪ { d } k ∈ K k\in\mathcal{K} k ∈ K V k V_k V k 1 / 4 1/4 1/4 k k k ∣ V k ∣ ≤ ( 9 e d / k ) k \lvert V_k\rvert\le(9ed/k)^k ∣ V k ∣ ≤ ( 9 e d / k ) k η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 ) B : = 1 + C Z 2 log ( 2 n 2 ) B:=1+C_Z\sqrt{2\log(2n^2)} B := 1 + C Z 2 log ( 2 n 2 ) ϵ 0 \epsilon_0 ϵ 0
1 { A ( D ) + 2 B n ϵ 0 W > 2 B log ( 1 / η ) n ϵ 0 } , A ( D ) : = max k ∈ K { max v ∈ V k ∣ 1 n ∑ i = 1 n [ v ⊤ Z i ] B ∣ − p B − q k } , (106) \mathbf{1}\left\{A(D)+\frac{2B}{n\epsilon_0}W>\frac{2B\log(1/\eta)}{n\epsilon_0}\right\},\qquad A(D):=\max_{k\in\mathcal{K}}\left\{\max_{v\in V_k}\left|\frac{1}{n}\sum_{i=1}^n [v^\top Z_i]_B\right|-p_B-q_k\right\}, \tag*{(106)} 1 { A ( D ) + n ϵ 0 2 B W > n ϵ 0 2 B log ( 1/ η ) } , A ( D ) := k ∈ K max { v ∈ V k max n 1 i = 1 ∑ n [ v ⊤ Z i ] B − p B − q k } , (106) where W ∼ Lap ( 1 ) W\sim\operatorname{Lap}(1) W ∼ Lap ( 1 )
p B : = 2 C Z 2 B − 1 exp { − ( B − 1 ) 2 2 C Z 2 } , q k : = B 2 log ( 4 ∣ K ∣ ∣ V k ∣ / η ) n . p_B:=\frac{2C_Z^2}{B-1}\exp\left\{-\frac{(B-1)^2}{2C_Z^2}\right\},\qquad q_k:=B\sqrt{\frac{2\log(4|\mathcal{K}||V_k|/\eta)}{n}}. p B := B − 1 2 C Z 2 exp { − 2 C Z 2 ( B − 1 ) 2 } , q k := B n 2 log ( 4∣ K ∣∣ V k ∣/ η ) . Lemma 36. Suppose Z 1 , … , Z n Z_1,\ldots,Z_n Z 1 , … , Z n P ∈ P m e a n P\in\mathcal{P}_{\mathrm{mean}} P ∈ P mean μ \mu μ n ≥ 2 n\ge2 n ≥ 2 ϵ 0 > 0 \epsilon_0>0 ϵ 0 > 0 η ∈ ( 0 , 1 / 4 ) \eta\in(0,1/4) η ∈ ( 0 , 1/4 ) (106) is ( ϵ 0 , 0 ) (\epsilon_0,0) ( ϵ 0 , 0 ) η \eta η
universal constant C > 0 C>0 C > 0 ∥ μ ∥ 0 ≤ s ∗ \|\mu\|_0\le s^* ∥ μ ∥ 0 ≤ s ∗
∥ μ ∥ 2 ≥ C { p B + B s ∗ log ( e d / s ∗ ) + log ( ∣ K ∣ / η ) n + B log ( 1 / η ) n ϵ 0 } , (107) \|\mu\|_2\ge C\left\{p_B+B\sqrt{\frac{s^*\log(ed/s^*)+\log(|\mathcal{K}|/\eta)}{n}}+\frac{B\log(1/\eta)}{n\epsilon_0}\right\}, \tag*{(107)} ∥ μ ∥ 2 ≥ C { p B + B n s ∗ log ( e d / s ∗ ) + log ( ∣ K ∣/ η ) + n ϵ 0 B log ( 1/ η ) } , (107) its type-II error is at most η \eta η
Proof of the lower bound
Proof. We consider the set of distributions P μ = N ( μ , σ 0 2 I d ) P_\mu=\mathcal{N}(\mu,\sigma_0^2I_d) P μ = N ( μ , σ 0 2 I d ) ∥ μ ∥ 2 ≤ 1 \|\mu\|_2\le1 ∥ μ ∥ 2 ≤ 1 σ 0 : = C Z 3 / 8 \sigma_0:=C_Z\sqrt{3/8} σ 0 := C Z 3/8 P m e a n P_{\mathrm{mean}} P mean μ \mu μ s ∗ s^* s ∗ ± r / s ∗ \pm r/\sqrt{s^*} ± r / s ∗ P ‾ r , s ∗ ( n ) \overline{P}_{r,s^*}^{(n)} P r , s ∗ ( n )
For ∥ μ ∥ 2 = ∥ μ ′ ∥ 2 = r \|\mu\|_2=\|\mu'\|_2=r ∥ μ ∥ 2 = ∥ μ ′ ∥ 2 = r
E 0 [ d P μ d P 0 ( Z ) d P μ ′ d P 0 ( Z ) ] = E 0 [ exp { ( μ + μ ′ ) ⊤ Z − r 2 σ 0 2 } ] = exp { 1 2 σ 0 2 ∥ μ + μ ′ ∥ 2 2 − r 2 σ 0 2 } = exp { μ ⊤ μ ′ σ 0 2 } . \mathbb{E}_0\left[\frac{\mathrm{d}P_\mu}{\mathrm{d}P_0}(Z)\frac{\mathrm{d}P_{\mu'}}{\mathrm{d}P_0}(Z)\right]
=\mathbb{E}_0\left[\exp\left\{\frac{(\mu+\mu')^\top Z-r^2}{\sigma_0^2}\right\}\right]
=\exp\left\{\frac{1}{2\sigma_0^2}\|\mu+\mu'\|_2^2-\frac{r^2}{\sigma_0^2}\right\}
=\exp\left\{\frac{\mu^\top\mu'}{\sigma_0^2}\right\}. E 0 [ d P 0 d P μ ( Z ) d P 0 d P μ ′ ( Z ) ] = E 0 [ exp { σ 0 2 ( μ + μ ′ ) ⊤ Z − r 2 } ] = exp { 2 σ 0 2 1 ∥ μ + μ ′ ∥ 2 2 − σ 0 2 r 2 } = exp { σ 0 2 μ ⊤ μ ′ } . By similar arguments to the proof of Lemma 18 , we thus have that
D K L ( P ‾ r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≤ ( s ∗ ) 2 d { cosh ( n r 2 σ 0 2 s ∗ ) − 1 } . (108) D_{\mathrm{KL}}\left(\overline{P}_{r,s^*}^{(n)}\middle\|P_0^{\otimes n}\right)
\le\frac{(s^*)^2}{d}\left\{\cosh\left(\frac{nr^2}{\sigma_0^2s^*}\right)-1\right\}. \tag*{(108)} D KL ( P r , s ∗ ( n ) P 0 ⊗ n ) ≤ d ( s ∗ ) 2 { cosh ( σ 0 2 s ∗ n r 2 ) − 1 } . (108) Let r n p r_{\mathrm{np}} r np r p r i v r_{\mathrm{priv}} r priv (53) . Similar calculations give, for sufficiently small c > 0 c>0 c > 0
D K L ( P ‾ r , s ∗ ( n ) ∥ P 0 ⊗ n ) ≤ { c 4 , r = c min { σ 0 r n p , 1 } , c 4 / ( n ϵ 2 ) , r = c min { σ 0 r p r i v , 1 } . D_{\mathrm{KL}}\left(\overline{P}_{r,s^*}^{(n)}\middle\|P_0^{\otimes n}\right)
\le
\begin{cases}
c^4, & r=c\min\{\sigma_0r_{\mathrm{np}},1\},\\
c^4/(n\epsilon^2), & r=c\min\{\sigma_0r_{\mathrm{priv}},1\}.
\end{cases} D KL ( P r , s ∗ ( n ) P 0 ⊗ n ) ≤ { c 4 , c 4 / ( n ϵ 2 ) , r = c min { σ 0 r np , 1 } , r = c min { σ 0 r priv , 1 } . By considering P 0 P_0 P 0 P r ( 1 , 0 , … , 0 ) P_{r(1,0,\ldots,0)} P r ( 1 , 0 , … , 0 ) r 2 / ( 2 σ 0 2 ) r^2/(2\sigma_0^2) r 2 / ( 2 σ 0 2 ) Proposition 4 with
α ≲ r , τ = 0 , ω ≲ { c 2 / n , r = c min { σ 0 r n p , 1 } , c 2 / ( n ϵ ) , r = c min { σ 0 r p r i v , 1 } \alpha\lesssim r,\qquad\tau=0,\qquad\omega\lesssim
\begin{cases}
c^2/\sqrt{n}, & r=c\min\{\sigma_0r_{\mathrm{np}},1\},\\
c^2/(n\epsilon), & r=c\min\{\sigma_0r_{\mathrm{priv}},1\}
\end{cases} α ≲ r , τ = 0 , ω ≲ { c 2 / n , c 2 / ( n ϵ ) , r = c min { σ 0 r np , 1 } , r = c min { σ 0 r priv , 1 } and taking sufficiently small constants to satisfy the error bounds as in Appendix § 7 .§ 4 gives the lower bound. □ \square □
Proof of the upper bound
Proof. We can construct a test for the dense regime using Lemma 8 and applying it over a dyadic grid of scales as in Lemma 24 , since its proof and that of Lemma 23 only require Z Z Z ∥ E [ Z ] ∥ 2 \|\mathbb{E}[Z]\|_2 ∥ E [ Z ] ∥ 2 ( ϵ 0 , 0 ) (\epsilon_0,0) ( ϵ 0 , 0 )
C ℓ n , d , ϵ 0 3 / 4 ( d 1 / 4 n + d 1 / 4 n 3 / 4 ϵ 0 1 / 2 + 1 n ϵ 0 ) . (109) C\ell_{n,d,\epsilon_0}^{3/4}\left(\frac{d^{1/4}}{\sqrt{n}}+\frac{d^{1/4}}{n^{3/4}\epsilon_0^{1/2}}+\frac{1}{n\epsilon_0}\right). \tag*{(109)} C ℓ n , d , ϵ 0 3/4 ( n d 1/4 + n 3/4 ϵ 0 1/2 d 1/4 + n ϵ 0 1 ) . (109) We can therefore run this test and the test in (106) each with privacy budget ϵ 0 = ϵ / 2 \epsilon_0 = \epsilon/2 ϵ 0 = ϵ /2 η = 1 / 9 \eta= 1/9 η = 1/9 2 η 2\eta 2 η η \eta η (109) or (107) . Taking the minimum of the two separation conditions gives a test with rate C ℓ n , d , ϵ 3 ϱ s ∗ , d C\ell_{n,d,\epsilon}^{3}\varrho_{s^*,d} C ℓ n , d , ϵ 3 ϱ s ∗ , d 1 / 3 1/3 1/3
Proof of Lemma 36. Privacy. Replacing one observation changes Z ˉ v : = n − 1 ∑ i = 1 n [ v ⊤ Z i ] − B B \bar{Z}_v := n^{-1}\sum_{i=1}^{n}[v^\top Z_i]_{-B}^{B} Z ˉ v := n − 1 ∑ i = 1 n [ v ⊤ Z i ] − B B 2 B / n 2B/n 2 B / n (16) , taking absolute values, subtracting fixed thresholds and maximising over ( k , v ) (k,v) ( k , v ) ∣ A ( D ) − A ( D ′ ) ∣ ≤ 2 B / n |A(D)-A(D')| \leq2B/n ∣ A ( D ) − A ( D ′ ) ∣ ≤ 2 B / n (106) is ( ϵ 0 , 0 ) (\epsilon_0,0) ( ϵ 0 , 0 )
Clipping bias and concentration. Since ∣ v ⊤ μ ∣ ≤ 1 |v^\top\mu| \leq1 ∣ v ⊤ μ ∣ ≤ 1 (103) gives
∣ E [ [ v ⊤ Z ] − B B ] − v ⊤ μ ∣ ≤ ∫ B ∞ P ( ∣ v ⊤ Z ∣ > t ) d t ≤ 2 ∫ B ∞ exp { − ( t − 1 ) 2 2 C Z 2 } d t ≤ p B ≲ n − 2 . (110) \left|\mathbb{E}\left[[v^\top Z]_{-B}^{B}\right]-v^\top\mu\right| \leq\int_B^\infty\mathbb{P}(|v^\top Z|>t)\,\mathrm{d}t \leq2\int_B^\infty\exp\left\{-\frac{(t-1)^2}{2C_Z^2}\right\}\,\mathrm{d}t \leq p_B \lesssim n^{-2}. \tag*{(110)} E [ [ v ⊤ Z ] − B B ] − v ⊤ μ ≤ ∫ B ∞ P ( ∣ v ⊤ Z ∣ > t ) d t ≤ 2 ∫ B ∞ exp { − 2 C Z 2 ( t − 1 ) 2 } d t ≤ p B ≲ n − 2 . (110) Hoeffding’s inequality and a union bound, give
P ( ∃ k ∈ K , v ∈ V k : ∣ 1 n ∑ i = 1 n [ v ⊤ Z i ] − B B − E [ [ v ⊤ Z ] − B B ] ∣ > q k ) ≤ ∑ k ∈ K 2 ∣ V k ∣ exp ( − n q k 2 2 B 2 ) = η 2 . (111) \mathbb{P}\left(\exists k\in\mathcal{K},\ v\in\mathcal{V}_k:\left|\frac{1}{n}\sum_{i=1}^{n}[v^\top Z_i]_{-B}^{B}-\mathbb{E}\left[[v^\top Z]_{-B}^{B}\right]\right|>q_k\right)\leq\sum_{k\in\mathcal{K}}2|\mathcal{V}_k|\exp\left(-\frac{nq_k^2}{2B^2}\right)=\frac{\eta}{2}. \tag*{(111)} P ( ∃ k ∈ K , v ∈ V k : n 1 i = 1 ∑ n [ v ⊤ Z i ] − B B − E [ [ v ⊤ Z ] − B B ] > q k ) ≤ k ∈ K ∑ 2∣ V k ∣ exp ( − 2 B 2 n q k 2 ) = 2 η . (111) Type-I error. Under the null, (110) and (111) imply that A ( D ) ≤ 0 A(D)\leq0 A ( D ) ≤ 0 η / 2 \eta/2 η /2 P ( W > log ( 1 / η ) ) = η / 2 \mathbb{P}(W>\log(1/\eta))=\eta/2 P ( W > log ( 1/ η )) = η /2 (106) is at most η \eta η
Type-II error. Suppose ∥ μ ∥ 2 > 0 \|\mu\|_2>0 ∥ μ ∥ 2 > 0 ∥ μ ∥ 0 ≤ s ∗ \|\mu\|_0\leq s^* ∥ μ ∥ 0 ≤ s ∗ k ∗ ∈ K k^*\in\mathcal{K} k ∗ ∈ K s ∗ ≤ k ∗ ≤ 2 s ∗ s^*\leq k^*\leq2s^* s ∗ ≤ k ∗ ≤ 2 s ∗ v ∗ ∈ V k ∗ v^*\in\mathcal{V}_{k^*} v ∗ ∈ V k ∗ ∥ v ∗ − ( μ / ∥ μ ∥ 2 ) ∥ 2 ≤ 1 / 4 \|v^*-(\mu/\|\mu\|_2)\|_2\leq1/4 ∥ v ∗ − ( μ /∥ μ ∥ 2 ) ∥ 2 ≤ 1/4 v ∗ v^* v ∗ ∣ ( v ∗ ) ⊤ μ ∣ ≥ 3 ∥ μ ∥ 2 / 4 |(v^*)^\top\mu|\geq3\|\mu\|_2/4 ∣ ( v ∗ ) ⊤ μ ∣ ≥ 3∥ μ ∥ 2 /4 (111) , we have
A ( D ) ≥ ∣ 1 n ∑ i = 1 n [ ( v ∗ ) ⊤ Z i ] − B B ∣ − p B − q k ∗ ≥ 3 4 ∥ μ ∥ 2 − 2 p B − 2 q k ∗ . A(D)\geq\left|\frac{1}{n}\sum_{i=1}^{n}[(v^*)^\top Z_i]_{-B}^{B}\right|-p_B-q_{k^*}\geq\frac{3}{4}\|\mu\|_2-2p_B-2q_{k^*}. A ( D ) ≥ n 1 i = 1 ∑ n [( v ∗ ) ⊤ Z i ] − B B − p B − q k ∗ ≥ 4 3 ∥ μ ∥ 2 − 2 p B − 2 q k ∗ . Since P ( W < − log ( 1 / η ) ) = η / 2 \mathbb{P}(W<-\log(1/\eta))=\eta/2 P ( W < − log ( 1/ η )) = η /2 η \eta η
∥ μ ∥ 2 ≥ 4 ( p B + q k ∗ + 2 B log ( 1 / η ) n ϵ 0 ) . (112) \|\mu\|_2\geq4\left(p_B+q_{k^*}+\frac{2B\log(1/\eta)}{n\epsilon_0}\right). \tag*{(112)} ∥ μ ∥ 2 ≥ 4 ( p B + q k ∗ + n ϵ 0 2 B log ( 1/ η ) ) . (112) The bound in (66) implies that log ∣ V k ∗ ∣ ≲ s ∗ log ( e d / s ∗ ) \log|\mathcal{V}_{k^*}|\lesssim s^*\log(ed/s^*) log ∣ V k ∗ ∣ ≲ s ∗ log ( e d / s ∗ ) (112) is satisfied under (107) . □
Additional details on the numerical experiments
In this appendix, we give further details on sensitivity to tuning parameters and implementation of the different methods during the numerical experiments.
Sensitivity of Algorithm 2 to tuning parameters. The implementation in Algorithm 2 involves three tuning constants: C s e p C_{\mathrm{sep}} C sep C c l i p C_{\mathrm{clip}} C clip C t o l C_{\mathrm{tol}} C tol C s e p = 5 C_{\mathrm{sep}}=5 C sep = 5 C c l i p = 0.5 C_{\mathrm{clip}}=0.5 C clip = 0.5 C t o l = 0.05 C_{\mathrm{tol}}=0.05 C tol = 0.05 C s e p ∈ { 3 , 5 , 7 } C_{\mathrm{sep}} \in\{3,5,7\} C sep ∈ { 3 , 5 , 7 } C c l i p ∈ { 0.25 , 0.5 , 0.75 } C_{\mathrm{clip}} \in\{0.25,0.5,0.75\} C clip ∈ { 0.25 , 0.5 , 0.75 } C t o l ∈ { 0.025 , 0.05 , 0.075 } C_{\mathrm{tol}} \in\{0.025,0.05,0.075\} C tol ∈ { 0.025 , 0.05 , 0.075 } n = 4000 n=4000 n = 4000 d = 500 d=500 d = 500 ϵ = 1 \epsilon=1 ϵ = 1 d = 50 d=50 d = 50 ϵ = 4 \epsilon=4 ϵ = 4 Table 1 . In these pilot experiments, performance is relatively stable across the values of C t o l C_{\mathrm{tol}} C tol C s e p C_{\mathrm{sep}} C sep C c l i p C_{\mathrm{clip}} C clip C s e p C_{\mathrm{sep}} C sep
Table 1. Sensitivity of Algorithm 2 to the tuning parameters C s e p C_{\mathrm{sep}} C sep C c l i p C_{\mathrm{clip}} C clip C t o l C_{\mathrm{tol}} C tol n = 4000 n=4000 n = 4000 d = 500 d=500 d = 500 ϵ = 1 \epsilon=1 ϵ = 1 n = 4000 n=4000 n = 4000 d = 50 d=50 d = 50 ϵ = 4 \epsilon=4 ϵ = 4
Sensitivity of the coordinate test. To quantify the robustness of the coordinate test in Proposition 13 to misspecification in the covariate population quantities C X C_X C X λ ˉ \bar{\lambda} λ ˉ C X = c B 8 / 3 C_X=c_B\sqrt{8/3} C X = c B 8/3 λ ˉ = c λ \bar{\lambda}=c_\lambda λ ˉ = c λ 8 / 3 \sqrt{8/3} 8/3 Table 2 suggests some robustness to scaling by 0.1 to 5 in both parameters.
Table 2. Sensitivity of the coordinate test, for Gaussian linear regression with n = 4000 n=4000 n = 4000 d = 500 d=500 d = 500 ϵ = 1 \epsilon=1 ϵ = 1 ∥ β ∥ 0 = 1 \|\beta\|_0=1 ∥ β ∥ 0 = 1 C X = c B 8 / 3 C_X=c_B\sqrt{8/3} C X = c B 8/3 λ ˉ = c λ \bar{\lambda}=c_\lambda λ ˉ = c λ
Implementation of dpHSIC. For dpHSIC, we use Gaussian kernels with bandwidth $\sigma_X=\sqrt{d} for the covariates, and σ Y = 1 \sigma_Y = 1 σ Y = 1 [29 ] . We use 99 permutations. Table 3 reports sensitivity to bandwidth choice at n = 4000 n = 4000 n = 4000
Table 3. Sensitivity of dpHSIC to kernel bandwidths, for Gaussian linear regression with n = 4000 n = 4000 n = 4000 d = 500 d = 500 d = 500 ϵ = 1 \epsilon= 1 ϵ = 1
Implementation of ToT-F. We randomly partition the observations into M = ⌊ n / ( d + 1 ) ⌋ M = \lfloor n/(d + 1) \rfloor M = ⌊ n / ( d + 1 )⌋ d d d β = 0 \beta= 0 β = 0 [27 ] , we add Tulap (truncated-uniform-Laplace) noise to this count and reject when the noisy count exceeds the 0.95 quantile of the convolution of the Binomial(M M M Table 4 supports using the largest feasible number of blocks in this setting.
Table 4. Sensitivity of ToT-F and PB-LRT to the number of aggregation blocks, for Gaussian linear regression with n = 4000 n = 4000 n = 4000 d = 500 d = 500 d = 500 ϵ = 1 \epsilon= 1 ϵ = 1
Implementation of PB-LRT. Following Peña and Barrientos (2024), we compute the Gaussian regression log-likelihood ratio within each block, for M M M [ 0 , U ] [0, U] [ 0 , U ] U / ( M ϵ ) U/(M\epsilon) U / ( M ϵ ) U U U
Auxiliary lemmas
Lemma 37 (Inner products of independent sub-Gaussian vectors). Let U , V ∈ R d U,V\in\mathbb{R}^d U , V ∈ R d ∥ U ∥ ψ 2 ≤ a \|U\|_{\psi_2}\le a ∥ U ∥ ψ 2 ≤ a ∥ V ∥ ψ 2 ≤ b \|V\|_{\psi_2}\le b ∥ V ∥ ψ 2 ≤ b C > 0 C>0 C > 0 p ≥ 1 p\ge1 p ≥ 1 t ≥ 1 t\ge1 t ≥ 1
∥ U ⊤ V ∥ L p ≤ C a b ( d p + p ) , \|U^\top V\|_{L^p}\le Cab(\sqrt{dp}+p), ∥ U ⊤ V ∥ L p ≤ C ab ( d p + p ) , and consequently
P ( ∣ U ⊤ V ∣ > C a b ( d t + t ) ) ≤ e − t . \mathbb{P}\left(\lvert U^\top V\rvert>Cab(\sqrt{dt}+t)\right)\le e^{-t}. P ( ∣ U ⊤ V ∣ > C ab ( d t + t ) ) ≤ e − t . Proof. Conditional on V V V U ⊤ V U^\top V U ⊤ V ∥ U ⊤ V ∥ ψ 2 ∣ V ≤ a ∥ V ∥ 2 \|U^\top V\|_{\psi_2\mid V}\le a\|V\|_2 ∥ U ⊤ V ∥ ψ 2 ∣ V ≤ a ∥ V ∥ 2
∥ U ⊤ V ∥ L p ( U ∣ V ) ≤ C a p ∥ V ∥ 2 . (113) \|U^\top V\|_{L^p(U\mid V)}\le Ca\sqrt{p}\|V\|_2. \tag*{(113)} ∥ U ⊤ V ∥ L p ( U ∣ V ) ≤ C a p ∥ V ∥ 2 . (113) By Proposition 6.2.1 of Vershynin [43 ] ,
P { ∥ V ∥ 2 > C b ( d + s ) } ≤ 2 e − s 2 , s ≥ 0. \mathbb{P}\{\|V\|_2>Cb(\sqrt{d}+s)\}\le2e^{-s^2},\qquad s\ge0. P { ∥ V ∥ 2 > C b ( d + s )} ≤ 2 e − s 2 , s ≥ 0. Taking the L p L^p L p V V V
∥ U ⊤ V ∥ L p ≤ C a p ∥ ∥ V ∥ 2 ∥ L p ≤ C a b p ( d + p ) = C a b ( d p + p ) . \begin{aligned}
\|U^\top V\|_{L^p}&\le Ca\sqrt{p}\|\|V\|_2\|_{L^p}\\
&\le Cab\sqrt{p}(\sqrt{d}+\sqrt{p})=Cab(\sqrt{dp}+p).
\end{aligned} ∥ U ⊤ V ∥ L p ≤ C a p ∥∥ V ∥ 2 ∥ L p ≤ C ab p ( d + p ) = C ab ( d p + p ) . The tail bound follows from Markov’s inequality:
P ( ∣ U ⊤ V ∣ > e C a b ( d t + t ) ) ≤ e − t . \mathbb{P}\left(\lvert U^\top V\rvert>eCab(\sqrt{dt}+t)\right)\le e^{-t}. P ( ∣ U ⊤ V ∣ > e C ab ( d t + t ) ) ≤ e − t . Lemma 38 (McShane [34 ] ). Let ( X , d ) (\mathcal{X},d) ( X , d ) ∅ ≠ C ⊆ X \varnothing\ne C\subseteq\mathcal{X} ∅ = C ⊆ X f : C → R f:C\to\mathbb{R} f : C → R L L L
∣ f ( z ) − f ( z ′ ) ∣ ≤ L d ( z , z ′ ) z , z ′ ∈ C . \lvert f(z)-f(z')\rvert\le Ld(z,z')\qquad z,z'\in C. ∣ f ( z ) − f ( z ′ )∣ ≤ L d ( z , z ′ ) z , z ′ ∈ C . Then
f ^ ( x ) : = inf z ∈ C { f ( z ) + L d ( x , z ) } , x ∈ X , \widehat{f}(x):=\inf_{z\in C}\{f(z)+Ld(x,z)\},\qquad x\in\mathcal{X}, f ( x ) := z ∈ C inf { f ( z ) + L d ( x , z )} , x ∈ X , is an L L L f f f X \mathcal{X} X f ^ ( x ) = f ( x ) \widehat{f}(x)=f(x) f ( x ) = f ( x ) x ∈ C x\in C x ∈ C
∣ f ^ ( x ) − f ^ ( x ′ ) ∣ ≤ L d ( x , x ′ ) , x , x ′ ∈ X . \lvert\widehat{f}(x)-\widehat{f}(x')\rvert\le Ld(x,x'),\qquad x,x'\in\mathcal{X}. ∣ f ( x ) − f ( x ′ )∣ ≤ L d ( x , x ′ ) , x , x ′ ∈ X . Lemma 39 (Sequential composition of DP; e.g. [18 Theorem 3.16] ). Let ϵ 1 , ϵ 2 > 0 \epsilon_1,\epsilon_2>0 ϵ 1 , ϵ 2 > 0 δ 1 , δ 2 ≥ 0 \delta_1,\delta_2\ge0 δ 1 , δ 2 ≥ 0 M 1 M_1 M 1 M 2 M_2 M 2 ( ϵ 1 , δ 1 ) (\epsilon_1,\delta_1) ( ϵ 1 , δ 1 ) ( ϵ 2 , δ 2 ) (\epsilon_2,\delta_2) ( ϵ 2 , δ 2 ) ( ϵ 1 + ϵ 2 , δ 1 + δ 2 ) (\epsilon_1+\epsilon_2,\delta_1+\delta_2) ( ϵ 1 + ϵ 2 , δ 1 + δ 2 )
