Introduction
Modern applications increasingly generate complex data with multiway structures. Examples include facial images indexed by person, lighting condition, and pixel location [38 ] ; recommendation data indexed by user, item, and context [37 ] ; and gene-expression data indexed by individual mouse, tissue type, and gene [1 ] . The resulting tensors are often high-dimensional, making it desirable to represent the data parsimoniously while preserving the multiway structure as each tensor mode may exhibit a distinct source of variation. The Tucker tensor factor model provides such a representation through decomposing the data into a low-rank component driven by a core factor tensor, and an idiosyncratic noise, under which it is of interest to learn the low-dimensional spaces spanned by mode-specific loading matrices that describe the main patterns of variation along each mode [20 , 18 , 3 , 6 ] .
In many applications, tensor-valued observations are held by multiple clients, and privacy regulations or institutional restrictions may prevent them from being pooled for centralized analysis; consider e.g., patient imaging data held by hospitals, gene-expression measurements stored by research centers, and user–item data retained by platforms in different jurisdictions. Federated learning allows these clients to collaborate by transmitting locally computed model updates or summary statistics to a central server for aggregation, while retaining the raw observations locally [42 ] . However, keeping the raw data in local clients does not by itself does not provide a formal privacy guarantee, since transmitted outputs may still reveal information about individual records. Federated differential privacy (FDP) addresses this risk by requiring that any output released by each client to satisfy record-level differential privacy (DP), so that its distribution remains stable under the replacement of any single local record [23 , 24 ] .
Despite the growing literature on FDP estimation, methods and statistical theory specific for tensor data are underdeveloped, particularly on the trade-off between privacy and statistical efficiency. We propose to close this gap by studying the problem of estimating mode-wise loading spaces in Tucker tensor factor models under FDP, which comes with worst-case record-level FDP under arbitrary record replacements.
Related literature
Our work lies at the intersection of tensor factor models, differentially private principal component analysis (PCA), and FDP.
Tensor factor models. Low-rank tensor models are commonly formulated through Tucker or canonical polyadic (CP) decompositions [20 , 38 ] . Building on the Tucker decomposition, statistical tensor factor models represent tensor-valued observations through latent core tensors and mode-specific loading matrices. Generalizing from matrix factor models by Wang et al. [50 ] , the seminal work by Chen et al. [18 ] considered a Tucker-type model for high-dimensional tensor time series. Barigozzi et al. [6 ] studied a Tucker tensor factor model and developed an iterative projection estimator, deriving convergence rates and asymptotic distributions for the loading matrices and common factors. Babii et al. [3 ] showed that tensor PCA is rate-optimal under strong factors, while alternating least squares improves the convergence rate under weak factors. More recent literature has seen a wide range of developments (e.g. [19 , 30 , 16 , 5 , 56 ] ; see a recent review in Bolivar et al. [8 ] . All these studies consider tensor factor models without privacy constraints.
Differentially private PCA. PCA provides a low-dimensional representation of multivariate data by identifying orthogonal directions that capture the dominant variation [35 , 36 ] and has popularly been adopted for estimation of factor models and other low-rank structures. Early work on differentially private PCA studied fixed, bounded datasets without making distributional assumptions [7 , 17 , 25 ] . Subsequent work considered private PCA in distributed settings under boundedness assumptions. Wang and Xu [51 ] studied non-interactive locally private PCA and obtained optimal or near-optimal rates in low-dimensional and sparse high-dimensional settings. Grammenos et al. [27 ] developed a differentially private federated PCA for bounded streaming data under homogeneous sample sizes and privacy budgets. More recent work has considered unbounded observations under distributional assumptions. Liu et al. [41 ] established nearly optimal rates for rank-one PCA under sub-Gaussian distributions. For general-rank PCA, Cai et al. [14 ] obtained nearly minimax-optimal rates under the spiked covariance model, but their privacy guarantee holds only with high probability and for independent-copy replacements rather than arbitrary record replacements. Düngler and Sanyal [21 ] subsequently developed an iterative algorithm for general-rank PCA with worst-case privacy guarantees and established nearly matching upper and lower error bounds. In all above, the focus lies in the study of PCA-based methods for a sequence of random vectors, and their guarantees do not extend to tensor-valued data without careful investigation into the multiarray structure.
Federated differential privacy. This framework requires each client’s complete release to satisfy DP with respect to its local dataset, while the central server has access only to these privatized releases [52 ] (Li et al. 2024b). FDP has been studied in a range of statistical problems, including mean estimation and linear regression (Li et al., 2024b), nonparametric regression and classification [11 , 2 ] , functional data analysis [10 , 53 ] , and Cox regression [34 ] . In these works, minimax framework has been adopted for the quantification of the cost of privacy, establishing minimax lower bounds on the rates of estimation accompanied by matching upper bounds up to logarithmic factors. Minimax-optimal rates have been established in these settings, up to logarithmic factors. Most closely related to our setting, Li et al. (2024a) studied federated estimation of a common principal subspace and covariance matrix under the spiked covariance model. They obtained an upper bound matching the federated minimax lower bound up to logarithmic factors but as in Cai et al. [14 ] , their privacy guarantee is limited in its formulation.
List of contributions
The main contributions of this paper are summarized as follows.
Firstly, we develop a novel loading-space estimator for Tucker tensor factor models under FDP, allowing for heterogeneous sample sizes, privacy budgets, and factor strengths. The proposed algorithm combines a private stability test performed on blockwise loading-space projectors, a Huber loss-based aggregation strategy, and Gaussian perturbations for constructing a private local projector. Then, the central server aggregates these local projectors using client-specific weights. Through these steps, the algorithm guarantees worst-case record-level FDP under arbitrary record replacements, in contrast to existing studies which only provide high probability guarantees and permit independent-copy replacements [14 ] (Li et al. 2024a). Secondly, we establish the non-asymptotic upper bound for the oracle procedure whose Huber radii, noise scales, and aggregation weights depend on the unknown noise level and client-specific factor strengths. Our results show that these rates are governed by a scaled harmonic mean of the squared local error rates, which is no larger than the smallest squared local error rate and can vanish as the number of clients increases even when none of the local rates does. We derive a federated minimax lower bound with the same harmonic structure, establishing minimax optimality up to logarithmic factors under strong factors with fixed ranks and noise variance. Our upper bound is attained under the same worst-case FDP requirement as the lower bound and, to the best of our knowledge, this is the first minimax-optimality result for low-rank structure estimation for tensor-valued data under the worst-case FDP.
Thirdly, we develop a private calibration procedure that adaptively selects the local tuning parameters and aggregation weights without knowledge of the noise level or client-specific factor strengths. In Li et al. (2024b), clients share a common covariance matrix, so the common signal and noise parameters cancel from the normalized PCA aggregation weights. In our setting, we allow factor strengths to vary across clients, so the optimal weights additionally depend on unknown client-specific signal parameters. In addition, both their procedure and our oracle procedure require knowledge of the noise level and signal strength to calibrate the Gaussian perturbation added for privacy. To address both sources of dependence, each client uses a separate calibration sample and a private threshold search to estimate the dispersion of its blockwise loading-space projectors. The resulting scale estimate determines the Huber radii, the scale of perturbations, and the estimated local error rate used to compute the aggregation weights. Our adaptive estimator retains the same worst-case privacy guarantee and, under the additional signal condition, attains the same error rate as the oracle estimator.
Notation and organization
We use bold uppercase letters (e.g. X \mathbf{X} X ) for tensors, plain uppercase letters (e.g. A A A ) for matrices, bold lowercase letters (e.g. x \mathbf{x} x ) for vectors, and plain lowercase letters (e.g. t , λ t,\lambda t , λ ) for scalars. For any positive integer p p p , let [ p ] = { 1 , … , p } [p]=\{1,\ldots,p\} [ p ] = { 1 , … , p } . For real numbers a , b a,b a , b , write a ∨ b = max { a , b } a\vee b=\max\{a,b\} a ∨ b = max { a , b } and a ∧ b = min { a , b } a\wedge b=\min\{a,b\} a ∧ b = min { a , b } . For a matrix A ∈ R p 1 × p 2 A\in\mathbb{R}^{p_1\times p_2} A ∈ R p 1 × p 2 , denote by ∥ A ∥ \lVert A\rVert ∥ A ∥ its spectral norm, ∥ A ∥ F \lVert A\rVert_{\mathrm{F}} ∥ A ∥ F its Frobenius norm, and λ 1 ( A ) ≥ ⋯ ≥ λ p 1 ∧ p 2 ( A ) ≥ 0 \lambda_1(A)\geq\cdots\geq\lambda_{p_1\wedge p_2}(A)\geq0 λ 1 ( A ) ≥ ⋯ ≥ λ p 1 ∧ p 2 ( A ) ≥ 0 its singular values. For j ∈ [ p 1 ∧ p 2 ] j\in[p_1\wedge p_2] j ∈ [ p 1 ∧ p 2 ] , let SVD j ( A ) \operatorname{SVD}_j(A) SVD j ( A ) denote the matrix whose columns are the leading j j j left singular vectors of A A A ; we fix a measurable selection rule whenever there is a tie at the singular values. For symmetric matrices A , B A,B A , B of the same dimension, write A ⪯ B A\preceq B A ⪯ B if B − A B-A B − A is positive semidefinite. For integers p ≥ r p\geq r p ≥ r , define the Stiefel manifold O p × r = { U ∈ R p × r : U T U = I r } \mathbb{O}_{p\times r}=\{U\in\mathbb{R}^{p\times r}:U^{\mathsf{T}}U=I_r\} O p × r = { U ∈ R p × r : U T U = I r } . For a matrix U ∈ O p × r U\in\mathbb{O}_{p\times r} U ∈ O p × r , let Π ( U ) = U U T \Pi(U)=UU^{\mathsf{T}} Π ( U ) = U U T denote the orthogonal projector onto its column space. For any order-q q q tensor M ∈ R p 1 × ⋯ × p q \mathbf{M}\in\mathbb{R}^{p_1\times\cdots\times p_q} M ∈ R p 1 × ⋯ × p q and s ∈ [ q ] s\in[q] s ∈ [ q ] , let M s ( M ) ∈ R p s × ∏ s ′ ≠ s p s ′ \mathcal{M}_s(\mathbf{M})\in\mathbb{R}^{p_s\times\prod_{s'\neq s}p_{s'}} M s ( M ) ∈ R p s × ∏ s ′ = s p s ′ denote its mode-s s s matricization. For B ∈ R p s × p s ′ B\in\mathbb{R}^{p_s\times p_s'} B ∈ R p s × p s ′ , the mode-s s s product of M \mathbf{M} M and B B B is denoted by M × s B \mathbf{M}\times_s B M × s B . We write ⨂ j = 1 q A j \bigotimes_{j=1}^q A_j ⨂ j = 1 q A j for the Kronecker product of the matrices A 1 , … , A q A_1,\ldots,A_q A 1 , … , A q . For probability measures P , Q \mathbb{P},\mathbb{Q} P , Q , TV ( P , Q ) \operatorname{TV}(\mathbb{P},\mathbb{Q}) TV ( P , Q ) denotes the total variation distance.
The rest of this paper is organized as follows. § 2 introduces the federated model, formalizes the DP constraints, and presents the estimation procedure together with its privacy guarantee. In § 3 , we state the model assumptions and derive the non-asymptotic upper bound as well as establishing the matching minimax lower bound. § 4 develops the private calibration procedure and establishes that the adaptive estimator preserves the privacy guarantee and attains the oracle rate. Monte Carlo experiments and a real data application are presented in § 5 , followed by a discussion of possible extensions in § 6 . Proofs and auxiliary results are deferred to the supplementary material.
Methodology
In this section, we develop a federated estimator of a common mode-wise loading space from tensor-valued observations and establish its FDP guarantee. We first formulate the federated Tucker tensor factor model and define the relevant notions of DP in § 2.1 . The proposed estimator is presented in § 2.2 , and its privacy guarantee is established through sensitivity analysis in § 2.3 .
Federated model and differential privacy
We consider a federated setting with m m m local clients. Client j ∈ [ m ] j \in[m] j ∈ [ m ] observes n j n_j n j order-K K K tensors, collected in the dataset D j = ( X 1 ( j ) , … , X n j ( j ) ) ∈ Z n j D_j = (X_1^{(j)}, \ldots, X_{n_j}^{(j)}) \in\mathcal{Z}^{n_j} D j = ( X 1 ( j ) , … , X n j ( j ) ) ∈ Z n j , where Z = R p 1 × ⋯ × p K \mathcal{Z} = \mathbb{R}^{p_1 \times\cdots\times p_K} Z = R p 1 × ⋯ × p K . For each j ∈ [ m ] j \in[m] j ∈ [ m ] and t ∈ [ n j ] t \in[n_j] t ∈ [ n j ] , we assume that the data follow a Tucker tensor factor model
X t ( j ) = S t ( j ) + E t ( j ) , where S t ( j ) = F t ( j ) × 1 A 1 × 2 ⋯ × K A K . (1) X_t^{(j)} = S_t^{(j)} + E_t^{(j)}, \qquad\text{where } S_t^{(j)} = F_t^{(j)} \times_1 A_1 \times_2 \cdots\times_K A_K. \tag*{(1)} X t ( j ) = S t ( j ) + E t ( j ) , where S t ( j ) = F t ( j ) × 1 A 1 × 2 ⋯ × K A K . (1)
Here, F t ( j ) ∈ R r 1 × ⋯ × r K F_t^{(j)} \in\mathbb{R}^{r_1 \times\cdots\times r_K} F t ( j ) ∈ R r 1 × ⋯ × r K is the client-specific core factor tensor, A k ∈ R p k × r k A_k \in\mathbb{R}^{p_k \times r_k} A k ∈ R p k × r k is the mode-k k k loading matrix, and E t ( j ) E_t^{(j)} E t ( j ) is the idiosyncratic noise. We assume that all clients share the same mode-wise loading spaces, while their latent factors and factor strengths may differ. Such an assumption is a natural one when clients collect comparable tensor-valued observations with aligned modes, as in the multi-hospital, multi-center, and multi-platform applications discussed in § 1 . The common loading spaces capture shared mode-wise structures, whereas the client-specific latent factors accommodate heterogeneity across clients.
Even when the loading matrices differ across the clients as S t ( j ) = F t ( j ) × 1 A 1 ( j ) × 2 ⋯ × K A K ( j ) S_t^{(j)} = F_t^{(j)} \times_1 A_1^{(j)} \times_2 \cdots\times_K A_K^{(j)} S t ( j ) = F t ( j ) × 1 A 1 ( j ) × 2 ⋯ × K A K ( j ) , if there exist A k A_k A k and B k ( j ) ∈ R r k × r k B_k^{(j)} \in\mathbb{R}^{r_k \times r_k} B k ( j ) ∈ R r k × r k satisfying A k ( j ) = A k B k ( j ) A_k^{(j)} = A_k B_k^{(j)} A k ( j ) = A k B k ( j ) for all j ∈ [ m ] j \in[m] j ∈ [ m ] , the cross-client heterogeneity can be absorbed into the client-specific core factor, by noting that
S t ( j ) = ( F t ( j ) × 1 B 1 ( j ) × 2 ⋯ × K B K ( j ) ) × 1 A 1 × 2 ⋯ × K A K . S_t^{(j)} = \left(F_t^{(j)} \times_1 B_1^{(j)} \times_2 \cdots\times_K B_K^{(j)}\right) \times_1 A_1 \times_2 \cdots\times_K A_K. S t ( j ) = ( F t ( j ) × 1 B 1 ( j ) × 2 ⋯ × K B K ( j ) ) × 1 A 1 × 2 ⋯ × K A K .
It also implies that F t ( j ) F_t^{(j)} F t ( j ) and A k A_k A k , k ∈ [ K ] k \in[K] k ∈ [ K ] , are identifiable up to an invertible transform only, which is an inherent feature of the Tucker tensor factor models. On the other hand, the mode-wise loading spaces denoted by span ( A k ) \operatorname{span}(A_k) span ( A k ) , are well-defined, and thus serve as our estimation target.
We require each client’s complete release to satisfy record-level DP, with each tensor-valued observation treated as a single record. For a positive integer N N N , two datasets D , D ′ ∈ Z N D, D' \in\mathcal{Z}^N D , D ′ ∈ Z N are neighbors, written as D ∼ D ′ D \sim D' D ∼ D ′ , if they differ by the replacement of at most one record. The replacement may be arbitrary and need not follow the identical model in (1). We first define central differential privacy (CDP) for a single client.
Definition 1 ((ε , δ \varepsilon,\delta ε , δ )-central differential privacy). Let ε > 0 \varepsilon> 0 ε > 0 and δ ∈ [ 0 , 1 ) \delta\in[0,1) δ ∈ [ 0 , 1 ) . A randomized mechanism Q : Z N → Y \mathcal{Q} : \mathcal{Z}^N \to\mathcal{Y} Q : Z N → Y is ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -centrally differentially private if, for every pair D ∼ D ′ D \sim D' D ∼ D ′ and every measurable set S ⊆ Y S \subseteq\mathcal{Y} S ⊆ Y ,
P Q { Q ( D ) ∈ S } ≤ e ε P Q { Q ( D ′ ) ∈ S } + δ , \mathbb{P}_{\mathcal{Q}}\{\mathcal{Q}(D) \in S\} \leq e^\varepsilon\mathbb{P}_{\mathcal{Q}}\{\mathcal{Q}(D') \in S\} + \delta, P Q { Q ( D ) ∈ S } ≤ e ε P Q { Q ( D ′ ) ∈ S } + δ , where the probability is taken over the internal randomness of Q \mathcal{Q} Q .
We extend this requirement to the federated setting by imposing a separate privacy constraint on each client’s dataset.
Definition 2 (( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -federated differential privacy). Let ε = ( ε 1 , … , ε m ) T ∈ ( 0 , ∞ ) m \boldsymbol{\varepsilon}=(\varepsilon_1,\ldots,\varepsilon_m)^{\mathsf{T}}\in(0,\infty)^m ε = ( ε 1 , … , ε m ) T ∈ ( 0 , ∞ ) m and δ = ( δ 1 , … , δ m ) T ∈ [ 0 , 1 ) m \boldsymbol{\delta}=(\delta_1,\ldots,\delta_m)^{\mathsf{T}}\in[0,1)^m δ = ( δ 1 , … , δ m ) T ∈ [ 0 , 1 ) m . A randomized mechanism M : Z n 1 × ⋯ × Z n m → Y \mathcal{M}:\mathcal{Z}^{n_1}\times\cdots\times\mathcal{Z}^{n_m}\to\mathcal{Y} M : Z n 1 × ⋯ × Z n m → Y is ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -federated differentially private if, for every j ∈ [ m ] j\in[m] j ∈ [ m ] , every pair D j ∼ D j ′ \mathcal{D}_j\sim\mathcal{D}'_j D j ∼ D j ′ , every collection D − j = { D ℓ : ℓ ≠ j } \mathcal{D}_{-j}=\{\mathcal{D}_{\ell}:\ell\ne j\} D − j = { D ℓ : ℓ = j } , and every measurable set S ⊆ Y S\subseteq\mathcal{Y} S ⊆ Y ,
P M { M ( D j , D − j ) ∈ S } ≤ e ε j P M { M ( D j ′ , D − j ) ∈ S } + δ j , \mathbb{P}_{\mathcal{M}}\{\mathcal{M}(\mathcal{D}_j,\mathcal{D}_{-j})\in S\}\le e^{\varepsilon_j}\mathbb{P}_{\mathcal{M}}\{\mathcal{M}(\mathcal{D}'_j,\mathcal{D}_{-j})\in S\}+\delta_j, P M { M ( D j , D − j ) ∈ S } ≤ e ε j P M { M ( D j ′ , D − j ) ∈ S } + δ j , where the probability is taken over the internal randomness of M \mathcal{M} M .
Throughout, we consider noninteractive protocols in which each client j j j sends the server a single message T j = Q j ( D j ) T_j=Q_j(\mathcal{D}_j) T j = Q j ( D j ) computed from its local data. The local mechanisms use mutually independent randomness and generate their messages without data-dependent feedback from the server or other clients. If each Q j Q_j Q j satisfies ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -CDP, then the joint release ( T 1 , … , T m ) (T_1,\ldots,T_m) ( T 1 , … , T m ) satisfies ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -FDP. The server constructs its estimate from these messages alone, so the estimate also satisfies ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -FDP by the post-processing property [24 ] . We focus on estimating span ( A k ) \operatorname{span}(A_k) span ( A k ) under FDP for a fixed target mode k ∈ [ K ] k\in[K] k ∈ [ K ] . To estimate loading spaces for multiple modes, we allocate each client’s privacy budget across modes and apply sequential composition; see Appendix § 8 .§ 1 for a detailed discussion.
Federated private loading-space estimation
In this subsection, for a fixed mode k ∈ [ K ] k\in[K] k ∈ [ K ] , we construct a federated estimator of the common loading space span ( A k ) \operatorname{span}(A_k) span ( A k ) under FDP. Each client uses blockwise estimation, a private stability test, and Gaussian perturbation of a Huber aggregate to compute a private local loading-space estimate, which it transmits to the server. The server then aggregates the corresponding projection matrices using client-specific weights. The procedure is summarized in Algorithm 1.
Specifically, client j j j randomly partitions its observations into disjoint blocks and computes a loading-space estimate from each block. Blocking ensures that replacing one observation affects at most one block estimate. The client represents these estimates by their orthogonal projectors Π ^ j , k [ ℓ ] \widehat{\Pi}^{[\ell]}_{j,k} Π j , k [ ℓ ] and computes a consistency score q j , k q_{j,k} q j , k , a normalized count of projector pairs within Frobenius distance R j , k / 8 R_{j,k}/8 R j , k /8 of each other. A sufficiently high score ensures that the Huber aggregate satisfies the sensitivity bound in Lemma 1 below under an arbitrary replacement of one observation. We refer to the following requirement on the score as the stability condition:
q j , k ≥ ( 7 B j ) / 8. (2) q_{j,k}\ge(7B_j)/8. \tag*{(2)} q j , k ≥ ( 7 B j ) /8. (2)
The client performs a private stability test by adding Laplace noise to the consistency score and comparing the perturbed score with the prescribed threshold. The added noise protects the pass/fail decision, which could itself reveal information about individual observations.
If the test passes, the client forms the Huber aggregate H ^ j , k \widehat{H}_{j,k} H j , k using the Huber loss, defined for R > 0 R>0 R > 0 by
ρ R ( u ) = { u 2 / 2 , 0 ≤ u ≤ R , R u − R 2 / 2 , u > R . (3) \rho_R(u)=
\begin{cases}
u^2/2, & 0\le u\le R,\\
Ru-R^2/2, & u>R.
\end{cases} \tag*{(3)} ρ R ( u ) = { u 2 /2 , R u − R 2 /2 , 0 ≤ u ≤ R , u > R . (3)
The quadratic part pools nearby block estimates, while the linear part limits the influence of extreme estimates on the aggregate. The client then adds symmetric Gaussian noise to H ^ j , k \widehat{H}_{j,k} H j , k and extracts the
Algorithm 1 Federated private estimation of the mode-k k k loading space
Input: Datasets { D j } j = 1 m \{D_j\}_{j=1}^m { D j } j = 1 m ; rank r k r_k r k ; privacy parameters ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) ; block sizes { b j } j = 1 m \{b_j\}_{j=1}^m { b j } j = 1 m ; Huber radii { R j , k } j = 1 m \{R_{j,k}\}_{j=1}^m { R j , k } j = 1 m ; and weights { w j , k } j = 1 m \{w_{j,k}\}_{j=1}^m { w j , k } j = 1 m .
for j = 1 j=1 j = 1 to m m m do
Set B j = ⌊ n j / b j ⌋ B_j=\lfloor n_j/b_j\rfloor B j = ⌊ n j / b j ⌋ . Randomly permute [ n j ] [n_j] [ n j ] , independently of the observations, and partition the resulting sequence into blocks I j , 1 , … , I j , B j \mathcal{I}_{j,1},\ldots,\mathcal{I}_{j,B_j} I j , 1 , … , I j , B j with the first B j − 1 B_j-1 B j − 1 blocks containing b j b_j b j indices each and the last block containing the remaining indices.
For each ℓ ∈ [ B j ] \ell\in[B_j] ℓ ∈ [ B j ] , compute
U ^ j , k [ ℓ ] : = SVD r k { 1 ∣ I j , ℓ ∣ ∑ t ∈ I j , ℓ M k ( X t ( j ) ) M k ( X t ( j ) ) T } and Π ^ j , k [ ℓ ] : = Π ( U ^ j , k [ ℓ ] ) . \widehat{U}_{j,k}^{[\ell]}:=\operatorname{SVD}_{r_k}\left\{\frac{1}{|\mathcal{I}_{j,\ell}|}\sum_{t\in\mathcal{I}_{j,\ell}}\mathcal{M}_k(\mathbf{X}^{(j)}_t)\mathcal{M}_k(\mathbf{X}^{(j)}_t)^{\mathsf{T}}\right\}
\quad\text{and}\quad
\widehat{\Pi}_{j,k}^{[\ell]}:=\Pi\left(\widehat{U}_{j,k}^{[\ell]}\right). U j , k [ ℓ ] := SVD r k ⎩ ⎨ ⎧ ∣ I j , ℓ ∣ 1 t ∈ I j , ℓ ∑ M k ( X t ( j ) ) M k ( X t ( j ) ) T ⎭ ⎬ ⎫ and Π j , k [ ℓ ] := Π ( U j , k [ ℓ ] ) .
Compute
q j , k : = 1 B j 2 ∑ ℓ = 1 B j ∑ h = 1 B j 1 { ∥ Π ^ j , k [ ℓ ] − Π ^ j , k [ h ] ∥ F ≤ R j , k / 8 } . q_{j,k}:=\frac{1}{B_j^2}\sum_{\ell=1}^{B_j}\sum_{h=1}^{B_j}\mathbb{1}\left\{\left\|\widehat{\Pi}_{j,k}^{[\ell]}-\widehat{\Pi}_{j,k}^{[h]}\right\|_{\mathrm{F}}\le R_{j,k}/8\right\}. q j , k := B j 2 1 ℓ = 1 ∑ B j h = 1 ∑ B j 1 { Π j , k [ ℓ ] − Π j , k [ h ] F ≤ R j , k /8 } .
Independently draw W j , k ∼ Lap ( 4 / ε j ) W_{j,k}\sim\operatorname{Lap}(4/\varepsilon_j) W j , k ∼ Lap ( 4/ ε j ) .
if q j , k + W j , k < ( 7 B j ) / 8 + 2 + ( 4 / ε j ) log ( 2 / δ j ) q_{j,k}+W_{j,k}<(7B_j)/8+2+(4/\varepsilon_j)\log(2/\delta_j) q j , k + W j , k < ( 7 B j ) /8 + 2 + ( 4/ ε j ) log ( 2/ δ j ) then
Draw U ~ j , k \widetilde{U}_{j,k} U j , k independently of the data from the Haar distribution on O p k × r k \mathbb{O}_{p_k\times r_k} O p k × r k .
else
Form the Huber aggregate
H ^ j , k : = arg min H = H T ∑ ℓ = 1 B j ρ R j , k ( ∥ H − Π ^ j , k [ ℓ ] ∥ F ) . \widehat{H}_{j,k}:=\underset{H=H^{\mathsf{T}}}{\arg\min}\sum_{\ell=1}^{B_j}\rho_{R_{j,k}}\left(\left\|H-\widehat{\Pi}_{j,k}^{[\ell]}\right\|_{\mathrm{F}}\right). H j , k := H = H T arg min ℓ = 1 ∑ B j ρ R j , k ( H − Π j , k [ ℓ ] F ) .
See (3) for the definition of ρ R j , k \rho_{R_{j,k}} ρ R j , k .
Independently draw G j , k ∈ R p k × p k G_{j,k}\in\mathbb{R}^{p_k\times p_k} G j , k ∈ R p k × p k with i.i.d. N ( 0 , 1 ) \mathcal{N}(0,1) N ( 0 , 1 ) entries, and set
τ j , k 2 : = 64 R j , k 2 B j 2 ε j 2 log ( 2.5 δ j ) , Z j , k : = τ j , k 2 ( G j , k + G j , k T ) , U ~ j , k : = SVD r k ( H ^ j , k + Z j , k ) . \tau_{j,k}^2:=\frac{64R_{j,k}^2}{B_j^2\varepsilon_j^2}\log\left(\frac{2.5}{\delta_j}\right),\qquad
Z_{j,k}:=\frac{\tau_{j,k}}{\sqrt{2}}\left(G_{j,k}+G_{j,k}^{\mathsf{T}}\right),\qquad
\widetilde{U}_{j,k}:=\operatorname{SVD}_{r_k}\left(\widehat{H}_{j,k}+Z_{j,k}\right). τ j , k 2 := B j 2 ε j 2 64 R j , k 2 log ( δ j 2.5 ) , Z j , k := 2 τ j , k ( G j , k + G j , k T ) , U j , k := SVD r k ( H j , k + Z j , k ) .
end if
Transmit U ~ j , k \widetilde{U}_{j,k} U j , k to the server.
end for
Server: Output A ^ k f e d : = SVD r k { ∑ j = 1 m w j , k U ~ j , k U ~ j , k T } \widehat{A}^{\mathrm{fed}}_k:=\operatorname{SVD}_{r_k}\left\{\sum_{j=1}^m w_{j,k}\widetilde{U}_{j,k}\widetilde{U}_{j,k}^{\mathsf{T}}\right\} A k fed := SVD r k { ∑ j = 1 m w j , k U j , k U j , k T } .
leading r k r_k r k left singular vectors of the perturbed matrix to obtain U ~ j , k \widetilde{U}_{j,k} U j , k . The choice of Gaussian noise scale involving τ j , k \tau_{j,k} τ j , k is calibrated using the sensitivity bound in Lemma 1. If the test fails, the client instead draws U ~ j , k \widetilde{U}_{j,k} U j , k independently of the data from the Haar distribution on O p k × r k \mathbb{O}_{p_k\times r_k} O p k × r k . In either case, it transmits U ~ j , k \widetilde{U}_{j,k} U j , k to the server. The server forms a weighted average of the corresponding projectors U ~ j , k U ~ j , k T \widetilde{U}_{j,k}\widetilde{U}_{j,k}^{\mathsf{T}} U j , k U j , k T with client-specific weights, and takes its leading r k r_k r k -dimensional eigenspace as the federated estimate.
Several ingredients of Algorithm 1 have been studied in the literature on DP estimation. Specifically, blockwise subspace estimation has been used for private PCA [46 ] . Our stability test follows the propose-test-release principle of Dwork and Lei [22 ] . Its pairwise agreement score has the same form as that used by Tsfadia et al. [47 ] for private mean estimation. Huber aggregation with Gaussian perturbation was studied for user-level private mean estimation [57 ] , while weighted aggregation of privatized projectors has been studied for federated PCA (Li et al., 2024a). Our contribution is to combine a private stability test based on pairwise projector agreement with Huber aggregation. The resulting mechanism ensures DP for every pair of neighboring datasets in the full, unbounded sample space, which in turn carries over to the worst-case FDP of the final output, all the while permitting arbitrary-copy replacement.
Sensitivity analysis and privacy guarantees
In this subsection, we establish the privacy guarantee for Algorithm 1. A standard approach to releasing statistics under DP is output perturbation, i.e., adding noise calibrated to the sensitivity of the released statistics [23 , 24 ] .
Recall that D ∼ D ′ D \sim D' D ∼ D ′ denotes neighboring datasets. Then for a vector-valued function f f f , we define its ℓ 1 \ell_1 ℓ 1 - and ℓ 2 \ell_2 ℓ 2 -sensitivities by
ω f , 1 : = sup D ∼ D ′ ∥ f ( D ) − f ( D ′ ) ∥ 1 , and ω f , 2 : = sup D ∼ D ′ ∥ f ( D ) − f ( D ′ ) ∥ 2 , \omega_{f,1} := \sup_{D\sim D'} \|f(D)-f(D')\|_1,\quad\text{and}\quad\omega_{f,2} := \sup_{D\sim D'} \|f(D)-f(D')\|_2, ω f , 1 := D ∼ D ′ sup ∥ f ( D ) − f ( D ′ ) ∥ 1 , and ω f , 2 := D ∼ D ′ sup ∥ f ( D ) − f ( D ′ ) ∥ 2 ,
respectively. For any ε > 0 \varepsilon>0 ε > 0 , the Laplace mechanism adds independent Lap ( ω f , 1 / ε ) \operatorname{Lap}(\omega_{f,1}/\varepsilon) Lap ( ω f , 1 / ε ) noise to each coordinate, ensuring ( ε , 0 ) (\varepsilon,0) ( ε , 0 ) -CDP [24 Theorem 3.6] . For ε , δ ∈ ( 0 , 1 ) \varepsilon,\delta\in(0,1) ε , δ ∈ ( 0 , 1 ) , the Gaussian mechanism adds independent centered Gaussian noise with variance 2 ω f , 2 2 ε − 2 log ( 1.25 / δ ) 2\omega_{f,2}^2\varepsilon^{-2}\log(1.25/\delta) 2 ω f , 2 2 ε − 2 log ( 1.25/ δ ) to each coordinate, ensuring ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -CDP [24 Theorem A.1] . For a matrix-valued function f f f , the identity ∥ vec ( M ) ∥ 2 = ∥ M ∥ F \|\operatorname{vec}(M)\|_2=\|M\|_F ∥ vec ( M ) ∥ 2 = ∥ M ∥ F implies that its Frobenius-norm sensitivity,
ω f , F : = sup D ∼ D ′ ∥ f ( D ) − f ( D ′ ) ∥ F , \omega_{f,F} := \sup_{D\sim D'} \|f(D)-f(D')\|_F, ω f , F := D ∼ D ′ sup ∥ f ( D ) − f ( D ′ ) ∥ F ,
is equivalent to the ℓ 2 \ell_2 ℓ 2 -sensitivity of the vectorized statistic. Thus, Gaussian perturbation can be calibrated directly to the Frobenius-norm sensitivity of matrix-valued statistics.
At each client j j j , Algorithm 1 first applies a private stability test to the consistency score q j , k q_{j,k} q j , k . If the test fails, the client draws U ~ j , k \widetilde{U}_{j,k} U j , k independently of the data from the Haar distribution on O p k × r k \mathbb{O}_{p_k\times r_k} O p k × r k ; this draw itself satisfies ( 0 , 0 ) (0,0) ( 0 , 0 ) -CDP. If the test passes, the client forms the Huber aggregate H ^ j , k \widehat{H}_{j,k} H j , k and adds symmetric Gaussian noise before extracting the local loading-space estimate. The following lemma provides a uniform sensitivity bound required for the scalar-valued consistency score q j , k q_{j,k} q j , k , and for the matrix-valued Huber aggregate H ^ j , k \widehat{H}_{j,k} H j , k .
Lemma 1. Let B ≥ 8 B \ge8 B ≥ 8 , R > 0 R > 0 R > 0 , and let P [ 1 ] , … , P [ B ] P^{[1]},\ldots,P^{[B]} P [ 1 ] , … , P [ B ] be a list of symmetric matrices of the same dimension. Define
q : = 1 B ∑ ℓ = 1 B ∑ h = 1 B 1 { ∥ P [ ℓ ] − P [ h ] ∥ F ≤ R / 8 } and H ^ : = arg min H = H T ∑ ℓ = 1 B ρ R ( ∥ H − P [ ℓ ] ∥ F ) , (4) q := \frac{1}{B}\sum_{\ell=1}^{B}\sum_{h=1}^{B}\mathbf{1}\left\{\left\|P^{[\ell]}-P^{[h]}\right\|_{\mathrm{F}}\le R/8\right\}
\quad\text{and}\quad
\widehat{H}:=\underset{H=H^{\mathsf{T}}}{\arg\min}\sum_{\ell=1}^{B}\rho_R\left(\left\|H-P^{[\ell]}\right\|_{\mathrm{F}}\right), \tag*{(4)} q := B 1 ℓ = 1 ∑ B h = 1 ∑ B 1 { P [ ℓ ] − P [ h ] F ≤ R /8 } and H := H = H T arg min ℓ = 1 ∑ B ρ R ( H − P [ ℓ ] F ) , (4) where ρ R \rho_R ρ R is the Huber loss defined in (3) . For a second list of symmetric matrices P [ 1 ] ′ , … , P [ B ] ′ P^{[1]\prime},\ldots,P^{[B]\prime} P [ 1 ] ′ , … , P [ B ] ′ differing from the original list in at most one index, define q ′ q' q ′ and H ^ ′ \widehat{H}' H ′ analogously. Then
∣ q − q ′ ∣ ≤ 2. (5) \left|q-q'\right|\le2. \tag*{(5)} ∣ q − q ′ ∣ ≤ 2. (5) Moreover, if q ≥ 7 B / 8 q\ge7B/8 q ≥ 7 B /8 , then
∥ H ^ − H ^ ′ ∥ F ≤ 4 R B . (6) \left\|\widehat{H}-\widehat{H}'\right\|_{\mathrm{F}}\le\frac{4R}{B}. \tag*{(6)} H − H ′ F ≤ B 4 R . (6)
Lemma 1 shows that the consistency score q j , k q_{j,k} q j , k has sensitivity at most two uniformly over all neighboring datasets. Allocating privacy budget ε j / 2 \varepsilon_j/2 ε j /2 to the private stability test therefore gives W j , k ∼ Lap ( 4 / ε j ) W_{j,k}\sim\operatorname{Lap}(4/\varepsilon_j) W j , k ∼ Lap ( 4/ ε j ) , so the test satisfies ( ε j / 2 , 0 ) (\varepsilon_j/2,0) ( ε j /2 , 0 ) -CDP. Under the stability condition in (2) , the Huber aggregate H ^ j , k \widehat{H}_{j,k} H j , k has Frobenius-norm sensitivity at most 4 R j , k / B j 4R_{j,k}/B_j 4 R j , k / B j . We allocate privacy budget of ( ε j / 2 , δ j / 2 ) (\varepsilon_j/2,\delta_j/2) ( ε j /2 , δ j /2 ) to its Gaussian perturbation. The symmetric Gaussian noise in Algorithm 1 is obtained by symmetrizing a matrix with independent N ( 0 , 2 τ j , k 2 ) \mathcal{N}(0,2\tau_{j,k}^2) N ( 0 , 2 τ j , k 2 ) entries. The Gaussian calibration therefore gives
τ j , k 2 = ( 4 R j , k / B j ) 2 ( ε j / 2 ) 2 log ( 1.25 δ j / 2 ) = 64 R j , k 2 B j 2 ε j 2 log ( 2.5 δ j ) . (7) \tau_{j,k}^{2}=\frac{(4R_{j,k}/B_j)^2}{(\varepsilon_j/2)^2}\log\left(\frac{1.25}{\delta_j/2}\right)=\frac{64R_{j,k}^{2}}{B_j^{2}\varepsilon_j^{2}}\log\left(\frac{2.5}{\delta_j}\right). \tag*{(7)} τ j , k 2 = ( ε j /2 ) 2 ( 4 R j , k / B j ) 2 log ( δ j /2 1.25 ) = B j 2 ε j 2 64 R j , k 2 log ( δ j 2.5 ) . (7)
The remaining δ j / 2 \delta_j/2 δ j /2 accounts for the possibility that the private stability test passes when the stability condition fails. The threshold is chosen to control the probability of such events. Combining this control with the calibrated Laplace and Gaussian perturbations and the data-independent Haar draw when the test fails yields the following privacy guarantee for Algorithm 1.
Proposition 1. Fix a mode k ∈ [ K ] k\in[K] k ∈ [ K ] . Suppose B j ≥ 8 B_j\ge8 B j ≥ 8 and ε j , δ j ∈ ( 0 , 1 ) \varepsilon_j,\delta_j\in(0,1) ε j , δ j ∈ ( 0 , 1 ) for every j ∈ [ m ] j\in[m] j ∈ [ m ] . Assume that the partition law, block sizes b j b_j b j , Huber radii R j , k R_{j,k} R j , k , and server weights w j , k w_{j,k} w j , k are fixed independently of the raw observations. Then for every j ∈ [ m ] j\in[m] j ∈ [ m ] , the local mechanism in Algorithm 1 satisfies ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -CDP. Hence the federated estimator A ^ k f e d \widehat{A}^{\mathrm{fed}}_k A k fed also satisfies ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -FDP.
Proposition 1 establishes a standard worst-case privacy guarantee for every pair of neighboring datasets in the full, unbounded sample space, without distributional assumptions on the observations. In contrast, the privacy guarantees in Cai et al. [14 ] and Li et al. (2024a) hold only with high probability under the sampling model and for independent-copy replacements. This is due to that their analyses calibrate the Gaussian mechanism using high-probability sensitivity bounds for spectral projectors computed from sample covariance matrices, which do not provide uniform sensitivity control under arbitrary observation replacement.
Main theoretical results
Having established the privacy guarantees in § 2.3 , we now derive the non-asymptotic upper bound and minimax lower bound for loading-space estimation in § 3.1 and § 3.2 , respectively. We first introduce the notation and assumptions used throughout this section.
Let n : = ∑ j = 1 m n j n := \sum_{j=1}^{m} n_j n := ∑ j = 1 m n j . For the fixed target mode k ∈ [ K ] k \in[K] k ∈ [ K ] , define
p : = ∏ ℓ = 1 K p ℓ , p − k : = ∏ ℓ ∈ [ K ] ∖ { k } p ℓ , and r − k : = ∏ ℓ ∈ [ K ] ∖ { k } r ℓ . p := \prod_{\ell=1}^{K} p_\ell,\qquad p_{-k} := \prod_{\ell\in[K]\setminus\{k\}} p_\ell,\qquad\text{and}\qquad r_{-k} := \prod_{\ell\in[K]\setminus\{k\}} r_\ell. p := ℓ = 1 ∏ K p ℓ , p − k := ℓ ∈ [ K ] ∖ { k } ∏ p ℓ , and r − k := ℓ ∈ [ K ] ∖ { k } ∏ r ℓ .
with the convention p − k = r − k = 1 p_{-k}=r_{-k}=1 p − k = r − k = 1 when K = 1 K=1 K = 1 . For each client j ∈ [ m ] j \in[m] j ∈ [ m ] , define the empirical mode-k k k second-moment matrix of the core factors by
Λ j , ( k ) : = 1 n j ∑ t = 1 n j M k ( F t ( j ) ) M k ( F t ( j ) ) ⊤ . (8) \Lambda_{j,(k)} := \frac{1}{n_j}\sum_{t=1}^{n_j}\mathcal{M}_k(\mathbf{F}^{(j)}_t)\mathcal{M}_k(\mathbf{F}^{(j)}_t)^\top. \tag*{(8)} Λ j , ( k ) := n j 1 t = 1 ∑ n j M k ( F t ( j ) ) M k ( F t ( j ) ) ⊤ . (8)
Define the largest and smallest eigenvalues of Λ j , ( k ) \Lambda_{j,(k)} Λ j , ( k ) and the maximum mode-k k k factor strength over time by
λ j , 1 ( k ) : = ∥ Λ j , ( k ) ∥ , λ j , r k ( k ) : = λ r k ( Λ j , ( k ) ) , and λ j , max ( k ) : = max t ∈ [ n j ] ∥ M k ( F t ( j ) ) M k ( F t ( j ) ) ⊤ ∥ , (9) \lambda^{(k)}_{j,1} := \|\Lambda_{j,(k)}\|,\qquad\lambda^{(k)}_{j,r_k} := \lambda_{r_k}(\Lambda_{j,(k)}),\qquad\text{and}\qquad\lambda^{(k)}_{j,\max} := \max_{t\in[n_j]}\|\mathcal{M}_k(\mathbf{F}^{(j)}_t)\mathcal{M}_k(\mathbf{F}^{(j)}_t)^\top\|, \tag*{(9)} λ j , 1 ( k ) := ∥ Λ j , ( k ) ∥ , λ j , r k ( k ) := λ r k ( Λ j , ( k ) ) , and λ j , m a x ( k ) := t ∈ [ n j ] max ∥ M k ( F t ( j ) ) M k ( F t ( j ) ) ⊤ ∥ , (9)
respectively.
We impose the following assumptions on the loading matrices, latent factors, and idiosyncratic noise.
Assumption 1 (Loading matrices). For every ℓ ∈ [ K ] \ell\in[K] ℓ ∈ [ K ] , A ℓ ∈ O p ℓ × r ℓ A_\ell\in\mathbb{O}_{p_\ell\times r_\ell} A ℓ ∈ O p ℓ × r ℓ and p ℓ > r ℓ p_\ell> r_\ell p ℓ > r ℓ .
Assumption 2 (Latent factors). The sequences { F t ( j ) } t ∈ [ n j ] \{\mathbf{F}^{(j)}_t\}_{t\in[n_j]} { F t ( j ) } t ∈ [ n j ] , j ∈ [ m ] j \in[m] j ∈ [ m ] , are deterministic. There exists an absolute constant C λ > 0 C_\lambda> 0 C λ > 0 such that 0 < λ j , r k ( k ) ≤ λ j , 1 ( k ) ≤ C λ λ j , r k ( k ) 0 < \lambda^{(k)}_{j,r_k} \le\lambda^{(k)}_{j,1} \le C_\lambda\lambda^{(k)}_{j,r_k} 0 < λ j , r k ( k ) ≤ λ j , 1 ( k ) ≤ C λ λ j , r k ( k ) , and λ j , max ( k ) ≤ C λ λ j , 1 ( k ) \lambda^{(k)}_{j,\max} \le C_\lambda\lambda^{(k)}_{j,1} λ j , m a x ( k ) ≤ C λ λ j , 1 ( k ) .
Assumption 3 (Idiosyncratic noise). The entries of { E t ( j ) : j ∈ [ m ] , t ∈ [ n j ] } \{\mathbf{E}^{(j)}_t:j\in[m],t\in[n_j]\} { E t ( j ) : j ∈ [ m ] , t ∈ [ n j ]} are independent and identically distributed as N ( 0 , σ 2 ) \mathcal{N}(0,\sigma^2) N ( 0 , σ 2 ) .
Assumption 1 adopts the standard orthonormal normalization ([14 , 56 , 3 ] ), with loading scales absorbed into the core factors. Assumption 2 treats the core factors as deterministic, as in [4 , 43 , 5 ] (); He et al. 2023). The assumption requires comparable factor strengths within each client and prevents a single time point from dominating the empirical second moment of the core factor. Assumption 3 imposes independent Gaussian idiosyncratic noise with a common variance, as in [32 ] (Gu and Shen 2020). Under Assumptions 1–3, we may define the population mode-k k k covariance matrix of the observations as
Σ k ( j ) : = E { 1 n j ∑ t = 1 n j M k ( X t ( j ) ) M k ( X t ( j ) ) ⊤ } = A k Λ j , ( k ) A k ⊤ + σ 2 p − k I p k . \boldsymbol{\Sigma}^{(j)}_k := \mathbb{E}\left\{\frac{1}{n_j}\sum_{t=1}^{n_j}\mathcal{M}_k(\mathbf{X}^{(j)}_t)\mathcal{M}_k(\mathbf{X}^{(j)}_t)^\top\right\}=A_k\Lambda_{j,(k)}A_k^\top+\sigma^2p_{-k}I_{p_k}. Σ k ( j ) := E { n j 1 t = 1 ∑ n j M k ( X t ( j ) ) M k ( X t ( j ) ) ⊤ } = A k Λ j , ( k ) A k ⊤ + σ 2 p − k I p k .
Its leading r k r_k r k -dimensional eigenspace is span ( A k ) \operatorname{span}(A_k) span ( A k ) , and the corresponding eigengap is λ j , r k ( k ) \lambda^{(k)}_{j,r_k} λ j , r k ( k ) .
Non-asymptotic upper bound
We first derive an error bound for Algorithm 1 under the oracle tuning, where the optimal choices of tuning parameters such as Huber radii and client-specific weights depend on unknown quantities.
Theorem 1. Fix a mode k ∈ [ K ] k \in[K] k ∈ [ K ] . Suppose that Assumptions 1–3 hold. For each client j ∈ [ m ] j \in[m] j ∈ [ m ] , let ε j , δ j ∈ ( 0 , 1 ) \varepsilon_j,\delta_j \in(0,1) ε j , δ j ∈ ( 0 , 1 ) . In Algorithm 1, choose b j = ⌈ C b { r k + log ( n ∨ p ) } ⌉ b_j = \lceil C_b\{r_k+\log(n\vee p)\}\rceil b j = ⌈ C b { r k + log ( n ∨ p )}⌉ and Huber radii
R j , k = C R r k [ σ λ j , 1 ( k ) p k + log ( n ∨ p ) b j + σ 2 λ j , 1 ( k ) { p + p − k log ( n ∨ p ) b j + p k + log ( n ∨ p ) b j } ] , (10) R_{j,k}=C_R\sqrt{r_k}\left[\frac{\sigma}{\sqrt{\lambda_{j,1}^{(k)}}}\sqrt{\frac{p_k+\log(n\vee p)}{b_j}}+\frac{\sigma^2}{\lambda_{j,1}^{(k)}}\left\{\sqrt{\frac{p+p_{-k}\log(n\vee p)}{b_j}}+\frac{p_k+\log(n\vee p)}{b_j}\right\}\right], \tag*{(10)} R j , k = C R r k λ j , 1 ( k ) σ b j p k + log ( n ∨ p ) + λ j , 1 ( k ) σ 2 { b j p + p − k log ( n ∨ p ) + b j p k + log ( n ∨ p ) } , (10) where C b , C R > 0 C_b,C_R>0 C b , C R > 0 are sufficiently large absolute constants. Let B j = ⌊ n j / b j ⌋ B_j=\lfloor n_j/b_j\rfloor B j = ⌊ n j / b j ⌋ and define the local error rate and aggregation weights by
Ψ j , k l o c : = R j , k r k log ( n ∨ p ) B j + τ j , k p k + log ( n ∨ p ) , and w j , k = ( Ψ j , k l o c ) − 2 ∑ ℓ = 1 m ( Ψ ℓ , k l o c ) − 2 , (11) \Psi_{j,k}^{\mathrm{loc}}:=\frac{R_{j,k}}{\sqrt{r_k}}\sqrt{\frac{\log(n\vee p)}{B_j}}+\tau_{j,k}\sqrt{p_k+\log(n\vee p)},\qquad\text{and}\qquad w_{j,k}=\frac{(\Psi_{j,k}^{\mathrm{loc}})^{-2}}{\sum_{\ell=1}^{m}(\Psi_{\ell,k}^{\mathrm{loc}})^{-2}}, \tag*{(11)} Ψ j , k loc := r k R j , k B j log ( n ∨ p ) + τ j , k p k + log ( n ∨ p ) , and w j , k = ∑ ℓ = 1 m ( Ψ ℓ , k loc ) − 2 ( Ψ j , k loc ) − 2 , (11) respectively, where τ j , k \tau_{j,k} τ j , k is defined in (7) . Suppose that, for every j ∈ [ m ] j \in[m] j ∈ [ m ] ,
λ j , 1 ( k ) / σ 2 ≥ C 0 ( p k + p ) , (12) \lambda_{j,1}^{(k)}/\sigma^2 \ge C_0(p_k+\sqrt{p}), \tag*{(12)} λ j , 1 ( k ) / σ 2 ≥ C 0 ( p k + p ) , (12) and
n j ε j ≥ C 0 { r k + log ( n ∨ p ) } [ log { ( n ∨ p ) / δ j } + r k { p k + log ( n ∨ p ) } { 1 + log ( 1 / δ j ) } ] , (13) n_j\varepsilon_j \ge C_0\{r_k+\log(n\vee p)\}\left[\log\{(n\vee p)/\delta_j\}+\sqrt{r_k\{p_k+\log(n\vee p)\}\{1+\log(1/\delta_j)\}}\right], \tag*{(13)} n j ε j ≥ C 0 { r k + log ( n ∨ p )} [ log {( n ∨ p ) / δ j } + r k { p k + log ( n ∨ p )} { 1 + log ( 1/ δ j )} ] , (13) where C 0 > 0 C_0>0 C 0 > 0 is a sufficiently large absolute constant. Then, there exist absolute constants C , c > 0 C,c>0 C , c > 0 such that, with probability at least 1 − ( n ∨ p ) − c 1-(n\vee p)^{-c} 1 − ( n ∨ p ) − c ,
∥ A ^ k f e d ( A ^ k f e d ) ⊤ − A k A k ⊤ ∥ F 2 ≤ C r k log ( n ∨ p ) ∑ j = 1 m ( Ψ j , k l o c ) − 2 ∧ ( 2 r k ) . (14) \left\|\widehat A_k^{\mathrm{fed}}(\widehat A_k^{\mathrm{fed}})^\top-A_kA_k^\top\right\|_F^2\le\frac{Cr_k\log(n\vee p)}{\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-2}}\wedge(2r_k). \tag*{(14)} A k fed ( A k fed ) ⊤ − A k A k ⊤ F 2 ≤ ∑ j = 1 m ( Ψ j , k loc ) − 2 C r k log ( n ∨ p ) ∧ ( 2 r k ) . (14)
In Theorem 1, condition (12) can be viewed as a signal-to-noise ratio condition, while (13) imposes a sample-size condition under privacy constraints. Similar conditions arise in the analysis of differentially private PCA (Cai et al., 2024c; Li et al., 2024a). The Huber radius R j , k R_{j,k} R j , k is chosen to match the Frobenius-norm error rate of the blockwise projectors. The signal-to-noise ratio condition ensures R j , k ≲ 1 R_{j,k}\lesssim1 R j , k ≲ 1 , so that the blockwise projectors concentrate within a sufficiently small neighborhood of the true loading-space projector. Under this condition, the sample-size condition in (13) implies
B j ≳ 1 ε j log { ( n ∨ p ) / δ j } and r k Ψ j , k l o c ≲ 1. (15) B_j\gtrsim\frac{1}{\varepsilon_j}\log\{(n\vee p)/\delta_j\}\qquad\text{and}\qquad\sqrt{r_k}\Psi_{j,k}^{\mathrm{loc}}\lesssim1. \tag*{(15)} B j ≳ ε j 1 log {( n ∨ p ) / δ j } and r k Ψ j , k loc ≲ 1. (15)
The first bound provides enough blocks for the private stability test to pass with high probability, while the second controls the accuracy of the privatized local estimators.
When there exists a single client (m = 1 m=1 m = 1 ), Proposition 1 guarantees ( ε 1 , δ 1 ) (\varepsilon_1,\delta_1) ( ε 1 , δ 1 ) -CDP, while Theorem 1 gives a Frobenius-norm error bound of order r k log ( n ∨ p ) Ψ 1 , k l o c ∧ r k \sqrt{r_k\log(n\vee p)}\Psi_{1,k}^{\mathrm{loc}}\wedge\sqrt{r_k} r k log ( n ∨ p ) Ψ 1 , k loc ∧ r k . For fixed factor ranks and p k ≳ log ( n ∨ p ) p_k\gtrsim\log(n\vee p) p k ≳ log ( n ∨ p ) , this bound simplifies in the nontrivial regime to
( σ p k λ 1 , 1 ( k ) + σ 2 p λ 1 , 1 ( k ) ) 1 n 1 + ( σ p k λ 1 , 1 ( k ) + σ 2 p λ 1 , 1 ( k ) ) p k log ( 2.5 / δ 1 ) n 1 ε 1 . (16) \left(\frac{\sigma\sqrt{p_k}}{\sqrt{\lambda_{1,1}^{(k)}}}+\frac{\sigma^2\sqrt{p}}{\lambda_{1,1}^{(k)}}\right)\frac{1}{\sqrt{n_1}}+\left(\frac{\sigma\sqrt{p_k}}{\sqrt{\lambda_{1,1}^{(k)}}}+\frac{\sigma^2\sqrt{p}}{\lambda_{1,1}^{(k)}}\right)\frac{\sqrt{p_k\log(2.5/\delta_1)}}{n_1\varepsilon_1}. \tag*{(16)} λ 1 , 1 ( k ) σ p k + λ 1 , 1 ( k ) σ 2 p n 1 1 + λ 1 , 1 ( k ) σ p k + λ 1 , 1 ( k ) σ 2 p n 1 ε 1 p k log ( 2.5/ δ 1 ) . (16)
up to logarithmic factors. The two terms represent the non-private estimation error and the privacy cost, respectively. To compare this rate with existing differentially private PCA results, set K = 1 K=1 K = 1 , so that p − k = 1 p_{-k}=1 p − k = 1 . For r k = 1 r_k=1 r k = 1 , the rate in (16) matches the bounds derived for differentially private PCA methods proposed by Liu et al. [41 ] and Düngler and Sanyal [21 ] in their signal-plus-noise setting, up to logarithmic factors. For fixed rank r k ≥ 1 r_k \ge1 r k ≥ 1 , it also matches the rate of Cai et al. [14 ] under the spiked covariance model, up to logarithmic factors.
For general K ≥ 1 K \ge1 K ≥ 1 , suppose further that σ \sigma σ is fixed and that the factors are strong, with λ 1 , 1 ( k ) ≍ p \lambda_{1,1}^{(k)} \asymp p λ 1 , 1 ( k ) ≍ p . The error bound in (16) then simplifies, up to logarithmic factors, to
1 n 1 p − k + p k log ( 2.5 / δ 1 ) n 1 ε 1 p − k . (17) \frac{1}{\sqrt{n_1p_{-k}}}+\frac{\sqrt{p_k\log(2.5/\delta_1)}}{n_1\varepsilon_1\sqrt{p_{-k}}}. \tag*{(17)} n 1 p − k 1 + n 1 ε 1 p − k p k log ( 2.5/ δ 1 ) . (17)
The first term matches the minimax rate for non-private tensor loading-space estimation under fixed ranks and strong factors (Han et al. [30 ] , Theorem 3.5). For comparison, consider estimating the principal subspace of the vectorized observations in R p \mathbb{R}^{p} R p , where p = p k p − k p=p_kp_{-k} p = p k p − k , without exploiting the Tucker structure. Using the PCA rate of Cai et al. [14 ] as a benchmark, the corresponding error bound under fixed ranks, fixed noise variance, and spikes of order p p p is
1 n 1 + p log ( 2.5 / δ 1 ) n 1 ε 1 \frac{1}{\sqrt{n_1}}+\frac{\sqrt{p\log(2.5/\delta_1)}}{n_1\varepsilon_1} n 1 1 + n 1 ε 1 p log ( 2.5/ δ 1 )
up to logarithmic factors. Compared with this benchmark, the two terms in (17) are smaller by factors of p − k \sqrt{p_{-k}} p − k and p − k p_{-k} p − k , respectively, illustrating the benefit of exploiting the Tucker structure.
For multiple clients (m ≥ 2 m \ge2 m ≥ 2 ), the weights in (11) minimize ∑ j = 1 m w j , k 2 ( Ψ j , k l o c ) 2 \sum_{j=1}^{m}w_{j,k}^2(\Psi_{j,k}^{\mathrm{loc}})^2 ∑ j = 1 m w j , k 2 ( Ψ j , k loc ) 2 over nonnegative weights summing to one. The aggregation term in (14) , i.e., 1 / ∑ j = 1 m ( Ψ j , k l o c ) − 2 1/\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-2} 1/ ∑ j = 1 m ( Ψ j , k loc ) − 2 , equals the harmonic mean of the squared local rates divided by m m m and satisfies
1 ∑ j = 1 m ( Ψ j , k l o c ) − 2 ≤ min { min j ∈ [ m ] ( Ψ j , k l o c ) 2 , 1 m 2 ∑ j = 1 m ( Ψ j , k l o c ) 2 , 2 m median j ∈ [ m ] ( Ψ j , k l o c ) 2 } . \frac{1}{\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-2}}\le\min\left\{\min_{j\in[m]}(\Psi_{j,k}^{\mathrm{loc}})^2,\frac{1}{m^2}\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^2,\frac{2}{m}\operatorname{median}_{j\in[m]}(\Psi_{j,k}^{\mathrm{loc}})^2\right\}. ∑ j = 1 m ( Ψ j , k loc ) − 2 1 ≤ min { j ∈ [ m ] min ( Ψ j , k loc ) 2 , m 2 1 j = 1 ∑ m ( Ψ j , k loc ) 2 , m 2 median j ∈ [ m ] ( Ψ j , k loc ) 2 } .
Under the conditions of Theorem 1, the federated bound therefore vanishes whenever any single-client bound vanishes. It can also vanish as m m m increases, even when none of the local bounds does. The harmonic aggregation structure is consistent with the federated PCA results of Li et al. (2024a). For K = 1 K=1 K = 1 , fixed ranks, and a common factor strength across clients, our bound recovers their rate up to logarithmic factors.
Suppose that the factor ranks and σ \sigma σ are fixed, and consider homogeneous clients with n j ≍ n / m n_j\asymp n/m n j ≍ n / m , λ j , 1 ( k ) ≍ p \lambda_{j,1}^{(k)}\asymp p λ j , 1 ( k ) ≍ p , ε j ≍ ε \varepsilon_j\asymp\varepsilon ε j ≍ ε , and δ j ≍ δ \delta_j\asymp\delta δ j ≍ δ . In the nontrivial regime, the squared error bound in (14) is of order
1 n p − k + m p k log ( 2.5 / δ ) n 2 ε 2 p − k , \frac{1}{np_{-k}}+\frac{mp_k\log(2.5/\delta)}{n^2\varepsilon^2p_{-k}}, n p − k 1 + n 2 ε 2 p − k m p k log ( 2.5/ δ ) ,
up to logarithmic factors. The first term is the non-private rate based on all n n n observations, and the second is the privacy cost. At fixed local sample sizes, aggregation reduces both terms by a factor of m m m , whereas at fixed total sample size, the privacy cost increases with m m m .
The radii R j , k R_{j,k} R j , k , Gaussian noise scales τ j , k \tau_{j,k} τ j , k , and aggregation weights w j , k w_{j,k} w j , k depend on the unknown noise level σ \sigma σ and client-specific factor strengths λ j , 1 ( k ) \lambda_{j,1}^{(k)} λ j , 1 ( k ) . Under the common covariance model in Li et al. (2024a), the common signal and noise parameters get canceled out due to the normalization in (11) for w j , k w_{j,k} w j , k , but they are still needed to calibrate the Gaussian perturbation added for privacy. In our heterogeneous setting, the normalized weights also depend on unknown client-specific signal parameters. To address both sources of dependence, we develop in Section 4 a private calibration procedure that selects these quantities without knowing σ \sigma σ or λ j , 1 ( k ) \lambda^{(k)}_{j,1} λ j , 1 ( k ) . Under the conditions stated there, the resulting adaptive estimator preserves the same privacy guarantee and attains the oracle high-probability error rate.
Minimax lower bound
We next establish a minimax lower bound under FDP to assess the optimality of the upper bound in Theorem 1.
Theorem 2. *Fix a mode k ∈ [ K ] k \in[K] k ∈ [ K ] . For λ , σ 2 > 0 \lambda,\sigma^2 > 0 λ , σ 2 > 0 , let P k f e d ( λ , σ 2 ) \mathcal{P}^{\mathrm{fed}}_k(\lambda,\sigma^2) P k fed ( λ , σ 2 ) denote the class of joint distributions generated by (1) under Assumptions 1–3, with c λ I r k ⪯ Λ j , ( k ) ⪯ C λ I r k c\lambda I_{r_k} \preceq\Lambda_{j,(k)} \preceq C\lambda I_{r_k} c λ I r k ⪯ Λ j , ( k ) ⪯ C λ I r k for every j ∈ [ m ] j \in[m] j ∈ [ m ] , where C ≥ c > 0 C \ge c > 0 C ≥ c > 0 are absolute constants. Let n : = ( n 1 , … , n m ) ⊤ \mathbf{n}:=(n_1,\ldots,n_m)^\top n := ( n 1 , … , n m ) ⊤ , and U n , ϵ , δ f e d , ( k ) \mathcal{U}^{\mathrm{fed},(k)}_{\mathbf{n},\epsilon,\delta} U n , ϵ , δ fed , ( k ) denote the class of all O p k × r k \mathbb{O}_{p_k \times r_k} O p k × r k -valued estimators satisfying ( ϵ , δ ) (\epsilon,\delta) ( ϵ , δ ) -federated differential privacy within the noninteractive framework. Suppose that the factor ranks are fixed and p k ≥ 2 r k p_k \ge2r_k p k ≥ 2 r k . For every j ∈ [ m ] j \in[m] j ∈ [ m ] , assume that ϵ j ∈ ( 0 , 1 ) \epsilon_j \in(0,1) ϵ j ∈ ( 0 , 1 ) , δ j ∈ ( 0 , e − 1 ] \delta_j \in(0,e^{-1}] δ j ∈ ( 0 , e − 1 ] ,
n j r − k ≥ r k , c 0 n j ϵ j 2 ≥ r k ( p k − r k ) δ j log ( 1 / δ j ) , n_j r_{-k} \ge r_k,\qquad c_0\sqrt{n_j}\epsilon_j^2 \ge r_k(p_k-r_k)\delta_j\log(1/\delta_j), n j r − k ≥ r k , c 0 n j ϵ j 2 ≥ r k ( p k − r k ) δ j log ( 1/ δ j ) , where c 0 > 0 c_0 > 0 c 0 > 0 is a sufficiently small constant. Then there exists a constant c 1 > 0 c_1 > 0 c 1 > 0 such that
inf A ^ k ∈ U n , ϵ , δ f e d , ( k ) sup P ∈ P k f e d ( λ , σ 2 ) E P ∥ A ^ k A ^ k ⊤ − A k A k ⊤ ∥ F 2 ≥ c 1 { r k ∑ j = 1 m ( Ψ j , k l o c ) − 2 ∧ ( 2 r k ) } . (18) \inf_{\widehat A_k\in\mathcal{U}^{\mathrm{fed},(k)}_{\mathbf{n},\epsilon,\delta}}\sup_{P\in\mathcal{P}^{\mathrm{fed}}_k(\lambda,\sigma^2)}
\mathbb{E}_{P}\left\|\widehat A_k\widehat A_k^\top-A_kA_k^\top\right\|_{\mathrm{F}}^2
\ge c_1\left\{\frac{r_k}{\sum_{j=1}^{m}(\Psi^{\mathrm{loc}}_{j,k})^{-2}}\wedge(2r_k)\right\}. \tag*{(18)} A k ∈ U n , ϵ , δ fed , ( k ) inf P ∈ P k fed ( λ , σ 2 ) sup E P A k A k ⊤ − A k A k ⊤ F 2 ≥ c 1 { ∑ j = 1 m ( Ψ j , k loc ) − 2 r k ∧ ( 2 r k ) } . (18) where
( Ψ j , k l o c ) 2 : = σ 2 λ ( p k n j + p k 2 r k n j 2 ϵ j 2 ) , j ∈ [ m ] . (19) (\Psi^{\mathrm{loc}}_{j,k})^2:=\frac{\sigma^2}{\lambda}\left(\frac{p_k}{n_j}+\frac{p_k^2r_k}{n_j^2\epsilon_j^2}\right),\qquad j\in[m]. \tag*{(19)} ( Ψ j , k loc ) 2 := λ σ 2 ( n j p k + n j 2 ϵ j 2 p k 2 r k ) , j ∈ [ m ] . (19)
The lower bound in (18) has the same harmonic aggregation structure as the upper bound in (14) . For fixed factor ranks and σ \sigma σ , with strong factors λ ≍ p \lambda\asymp p λ ≍ p , the local lower rates Ψ j , k l o c \Psi^{\mathrm{loc}}_{j,k} Ψ j , k loc in (19) match the corresponding upper rates Ψ j , k l o c \Psi^{\mathrm{loc}}_{j,k} Ψ j , k loc in (11) , up to logarithmic factors, and thus Algorithm 1 attains the near-optimal rate for estimating the common loading space under FDP in this regime. When K = 1 K=1 K = 1 , the lower bound matches that derived in Cai et al. [14 ] under CDP when m = 1 m=1 m = 1 , and Li et al. (2024a) under FDP when m ≥ 2 m\ge2 m ≥ 2 . For K > 1 K>1 K > 1 , both the non-private and privacy terms of the squared local rates contain the factor p − k − 1 p_{-k}^{-1} p − k − 1 , reflecting the contribution of the dimensions of the other modes.
Adaptive tuning
The oracle choices of the Huber radii R j , k R_{j,k} R j , k , Gaussian noise scales τ j , k \tau_{j,k} τ j , k and aggregation weights w j , k w_{j,k} w j , k in Theorem 1 depend on the unknown noise level σ \sigma σ and client-specific factor strengths λ j , 1 ( k ) \lambda^{(k)}_{j,1} λ j , 1 ( k ) . To address this, we develop a private calibration procedure that selects these quantities from the data. Section 4.1 presents this procedure and the resulting adaptive version of Algorithm 1, and Section 4.2 establishes its FDP guarantee and estimation error bound.
Algorithm 2 Adaptive federated private loading-space estimation
Input: Datasets { D j } j = 1 m \{\mathcal{D}_j\}_{j=1}^{m} { D j } j = 1 m ; rank r k r_k r k ; privacy parameters ( ϵ , δ ) (\epsilon,\delta) ( ϵ , δ ) ; block sizes { b j } j = 1 m \{b_j\}_{j=1}^{m} { b j } j = 1 m ; public grid R k = { d s } s = 0 S k \mathcal{R}_k=\{d_s\}_{s=0}^{S_k} R k = { d s } s = 0 S k ; constants C A , c 0 C_A,c_0 C A , c 0 .
for j = 1 j=1 j = 1 to m m m do
Randomly split D j \mathcal{D}_j D j , independently of the observations, into calibration and main samples of sizes n j ( 0 ) = ⌊ n j / 2 ⌋ n_j^{(0)}=\lfloor n_j/2\rfloor n j ( 0 ) = ⌊ n j /2 ⌋ and n j ( 1 ) = n j − n j ( 0 ) n_j^{(1)}=n_j-n_j^{(0)} n j ( 1 ) = n j − n j ( 0 ) .
Form B j ( a ) = ⌊ n j ( a ) / b j ⌋ B_j^{(a)}=\lfloor n_j^{(a)}/b_j\rfloor B j ( a ) = ⌊ n j ( a ) / b j ⌋ blocks in each sample, a ∈ { 0 , 1 } a\in\{0,1\} a ∈ { 0 , 1 } , as in Algorithm 1.
For each calibration block ℓ ∈ [ B j ( 0 ) ] \ell\in[B_j^{(0)}] ℓ ∈ [ B j ( 0 ) ] , compute U ^ j , k ( 0 ) , [ ℓ ] \widehat{U}_{j,k}^{(0),[\ell]} U j , k ( 0 ) , [ ℓ ] as in Algorithm 1 and set Π ^ j , k ( 0 ) , [ ℓ ] = Π ( U ^ j , k ( 0 ) , [ ℓ ] ) \widehat{\Pi}_{j,k}^{(0),[\ell]}=\Pi(\widehat{U}_{j,k}^{(0),[\ell]}) Π j , k ( 0 ) , [ ℓ ] = Π ( U j , k ( 0 ) , [ ℓ ] ) .
For s = 0 , … , S k s=0,\ldots,S_k s = 0 , … , S k , compute
q j , k ( 0 ) ( d s ) : = 1 B j ( 0 ) ∑ ℓ = 1 B j ( 0 ) ∑ h = 1 B j ( 0 ) 1 { ∥ Π ^ j , k ( 0 ) , [ ℓ ] − Π ^ j , k ( 0 ) , [ h ] ∥ F ≤ d s } . q_{j,k}^{(0)}(d_s):=\frac{1}{B_j^{(0)}}\sum_{\ell=1}^{B_j^{(0)}}\sum_{h=1}^{B_j^{(0)}}\mathbb{1}\left\{\left\|\widehat{\Pi}_{j,k}^{(0),[\ell]}-\widehat{\Pi}_{j,k}^{(0),[h]}\right\|_{\mathrm{F}}\le d_s\right\}. q j , k ( 0 ) ( d s ) := B j ( 0 ) 1 ℓ = 1 ∑ B j ( 0 ) h = 1 ∑ B j ( 0 ) 1 { Π j , k ( 0 ) , [ ℓ ] − Π j , k ( 0 ) , [ h ] F ≤ d s } .
Independently draw Y j ∼ Lap ( 4 / ϵ j ) Y_j\sim\operatorname{Lap}(4/\epsilon_j) Y j ∼ Lap ( 4/ ϵ j ) and Y j , s ∼ Lap ( 8 / ϵ j ) Y_{j,s}\sim\operatorname{Lap}(8/\epsilon_j) Y j , s ∼ Lap ( 8/ ϵ j ) , s = 0 , … , S k s=0,\ldots,S_k s = 0 , … , S k . Set
s ^ j , k : = min { s : q j , k ( 0 ) ( d s ) + Y j , s ≥ ( 7 B j ( 0 ) ) / 8 + Y j } . (20) \widehat{s}_{j,k}:=\min\left\{s:q_{j,k}^{(0)}(d_s)+Y_{j,s}\ge(7B_j^{(0)})/8+Y_j\right\}. \tag*{(20)} s j , k := min { s : q j , k ( 0 ) ( d s ) + Y j , s ≥ ( 7 B j ( 0 ) ) /8 + Y j } . (20)
taking s ^ j , k = S k \widehat{s}_{j,k}=S_k s j , k = S k if the set is empty.
Compute
R ^ j , k : = C A d s ^ j , k , τ ^ j , k 2 : = 64 R ^ j , k 2 ( B j ( 1 ) ) 2 ϵ j 2 log ( 2.5 δ j ) and Ψ ^ j , k : = R ^ j , k r k log ( n ∨ p ) B j ( 1 ) + τ ^ j , k p k . (21) \widehat{R}_{j,k}:=C_A d_{\widehat{s}_{j,k}},\qquad\widehat{\tau}_{j,k}^{\,2}:=\frac{64\widehat{R}_{j,k}^{\,2}}{(B_j^{(1)})^2\epsilon_j^2}\log\left(\frac{2.5}{\delta_j}\right)
\qquad\text{and}\qquad
\widehat{\Psi}_{j,k}:=\frac{\widehat{R}_{j,k}}{\sqrt{r_k}}\sqrt{\frac{\log(n\vee p)}{B_j^{(1)}}}+\widehat{\tau}_{j,k}\sqrt{p_k}. \tag*{(21)} R j , k := C A d s j , k , τ j , k 2 := ( B j ( 1 ) ) 2 ϵ j 2 64 R j , k 2 log ( δ j 2.5 ) and Ψ j , k := r k R j , k B j ( 1 ) log ( n ∨ p ) + τ j , k p k . (21)
Set ι ^ j , k = 1 \widehat{\iota}_{j,k}=1 ι j , k = 1 if
R ^ j , k ≤ c 0 , r k Ψ ^ j , k ≤ c 0 , and B j ( 1 ) ≥ 16 + 32 ϵ j log ( 8 m ( n ∨ p ) c 0 δ j ) , (22) \widehat{R}_{j,k}\le c_0,\qquad\sqrt{r_k}\widehat{\Psi}_{j,k}\le c_0,\qquad\text{and}\qquad B_j^{(1)}\ge16+\frac{32}{\epsilon_j}\log\left(\frac{8m(n\vee p)^{c_0}}{\delta_j}\right), \tag*{(22)} R j , k ≤ c 0 , r k Ψ j , k ≤ c 0 , and B j ( 1 ) ≥ 16 + ϵ j 32 log ( δ j 8 m ( n ∨ p ) c 0 ) , (22)
and set ι ^ j , k = 0 \widehat{\iota}_{j,k}=0 ι j , k = 0 otherwise.
if ι ^ j , k = 1 \widehat{\iota}_{j,k}=1 ι j , k = 1 then
Using the main-sample blocks formed above, apply the local estimation and release steps of Algorithm 1, with Huber radius R ^ j , k \widehat{R}_{j,k} R j , k and privacy parameters ( ϵ j , δ j ) (\epsilon_j,\delta_j) ( ϵ j , δ j ) , to obtain U ~ j , k \widetilde{U}_{j,k} U j , k .
else
Draw U ~ j , k \widetilde{U}_{j,k} U j , k independently of the data from the Haar distribution on O p k × r k \mathcal{O}_{p_k\times r_k} O p k × r k .
end if
Transmit ( ι ^ j , k , Ψ ^ j , k , U ~ j , k ) (\widehat{\iota}_{j,k},\widehat{\Psi}_{j,k},\widetilde{U}_{j,k}) ( ι j , k , Ψ j , k , U j , k ) to the server.
end for
Server: Set G ^ k : = { j ∈ [ m ] : ι ^ j , k = 1 } \widehat{G}_k:=\{j\in[m]:\widehat{\iota}_{j,k}=1\} G k := { j ∈ [ m ] : ι j , k = 1 } .
if G ^ k ≠ ∅ \widehat{G}_k\ne\varnothing G k = ∅ then
Set w ^ j , k = Ψ ^ j , k − 2 / ∑ ℓ ∈ G ^ k Ψ ^ ℓ , k − 2 \widehat{w}_{j,k}=\widehat{\Psi}_{j,k}^{-2}/\sum_{\ell\in\widehat{G}_k}\widehat{\Psi}_{\ell,k}^{-2} w j , k = Ψ j , k − 2 / ∑ ℓ ∈ G k Ψ ℓ , k − 2 for j ∈ G ^ k j\in\widehat{G}_k j ∈ G k , and output
A ^ k a d : = SVD r k { ∑ j ∈ G ^ k w ^ j , k U ~ j , k U ~ j , k T } . \widehat{A}_k^{\mathrm{ad}}:=\operatorname{SVD}_{r_k}\left\{\sum_{j\in\widehat{G}_k}\widehat{w}_{j,k}\widetilde{U}_{j,k}\widetilde{U}_{j,k}^{\mathsf{T}}\right\}. A k ad := SVD r k ⎩ ⎨ ⎧ j ∈ G k ∑ w j , k U j , k U j , k T ⎭ ⎬ ⎫ .
else
Output A ^ k a d \widehat{A}_k^{\mathrm{ad}} A k ad drawn independently of the data from the Haar distribution on O p k × r k \mathcal{O}_{p_k\times r_k} O p k × r k .
end if
Adaptive federated estimation
In this subsection, we develop an adaptive version of Algorithm 1. In short, each client selects the Huber radius through a private threshold search using blockwise loading-space projectors from a calibration sample. The selected radius determines the Gaussian noise scale and the estimated local error rate. Clients that pass the screening conditions and use these calibrated parameters to compute private local loading-space estimates from separate main samples. The server aggregates the corresponding projectors using client-specific weights based on the estimated local error rates. The procedure is summarized in Algorithm 2.
Particularly, note that the oracle radius R j , k R_{j,k} R j , k in (10) reflects the scale of client j j j ’s blockwise projector errors ∥ Π ^ j , k [ ℓ ] − A k A k ⊤ ∥ F \lVert\widehat{\Pi}_{j,k}^{[\ell]}-A_kA_k^\top\rVert_{\mathrm{F}} ∥ Π j , k [ ℓ ] − A k A k ⊤ ∥ F . As A k A k ⊤ A_kA_k^\top A k A k ⊤ is unknown, we instead use pairwise distances between the calibration projectors Π ^ j , k ( 0 ) , [ ℓ ] \widehat{\Pi}_{j,k}^{(0),[\ell]} Π j , k ( 0 ) , [ ℓ ] . Under the conditions of Theorem 3, the empirical 7 / 8 7/8 7/8 quantile of these distances is comparable to the oracle radius R j , k R_{j,k} R j , k with high probability. To approximate this quantile, given a constant γ ≥ 1 \gamma\ge1 γ ≥ 1 , let d min : = ( n ∨ p ) − γ d_{\min}:=(n\vee p)^{-\gamma} d m i n := ( n ∨ p ) − γ , and we define the public grid
R k : = { d s = 2 s d min : s = 0 , … , S k } , where S k : = ⌈ log 2 ( d min − 1 2 r k ) ⌉ . (23) \mathcal{R}_k:=\{d_s=2^s d_{\min}:s=0,\ldots,S_k\},\qquad\text{where }S_k:=\left\lceil\log_2(d_{\min}^{-1}\sqrt{2}r_k)\right\rceil. \tag*{(23)} R k := { d s = 2 s d m i n : s = 0 , … , S k } , where S k := ⌈ log 2 ( d m i n − 1 2 r k ) ⌉ . (23)
The grid contains O { log ( n ∨ p ) } O\{\log(n\vee p)\} O { log ( n ∨ p )} points, and its upper endpoint d S k ≥ 2 r k d_{S_k}\ge\sqrt{2}r_k d S k ≥ 2 r k bounds all possible Frobenius distances between rank-r k r_k r k orthogonal projectors. The ratio q j , k ( 0 ) ( d s ) / B j ( 0 ) q_{j,k}^{(0)}(d_s)/B_j^{(0)} q j , k ( 0 ) ( d s ) / B j ( 0 ) is the proportion of the ( B j ( 0 ) ) 2 (B_j^{(0)})^2 ( B j ( 0 ) ) 2 ordered projector pairs whose distance does not exceed d s d_s d s . Therefore, the first grid point satisfying q j , k ( 0 ) ( d s ) ≥ ( 7 B j ( 0 ) ) / 8 q_{j,k}^{(0)}(d_s)\ge(7B_j^{(0)})/8 q j , k ( 0 ) ( d s ) ≥ ( 7 B j ( 0 ) ) /8 , which is akin to the stability condition in (2) , rounds the empirical 7 / 8 7/8 7/8 quantile up to the nearest grid point.
To protect this data-dependent selection, we use a private threshold search based on the AboveThreshold mechanism (Dwork and Roth, [24 ] , Algorithm 1). In detail, the client adds independent Laplace noise to each consistency score and to the threshold, reusing the same noisy threshold across all grid points. It then selects the first index satisfying the comparison in (20) . The selected grid point determines R ^ j , k \widehat{R}_{j,k} R j , k and τ ^ j , k \widehat{\tau}_{j,k} τ j , k through (21) , where C A > 0 C_A>0 C A > 0 is a sufficiently large absolute constant. These quantities in turn determine the estimated local error rate Ψ ^ j , k \widehat{\Psi}_{j,k} Ψ j , k in (21) , constructed analogously to the oracle local error rate Ψ j , k l o c \Psi_{j,k}^{\mathrm{loc}} Ψ j , k loc in (11) .
The client then uses the calibrated radius R ^ j , k \widehat{R}_{j,k} R j , k , estimated local error rate Ψ ^ j , k \widehat{\Psi}_{j,k} Ψ j , k , and main-sample block count B j ( 1 ) B_j^{(1)} B j ( 1 ) to check the screening conditions in (22) , where c 0 > 0 c_0>0 c 0 > 0 is a sufficiently small absolute constant. These conditions are computable counterparts of the radius bound and the requirements in (15) . If ζ ^ j , k = 1 \widehat{\zeta}_{j,k}=1 ζ j , k = 1 , the client applies the local steps of Algorithm 1 to its main sample with Huber radius R ^ j , k \widehat{R}_{j,k} R j , k and Gaussian noise scale τ ^ j , k \widehat{\tau}_{j,k} τ j , k . Otherwise, it draws a data-independent random subspace from the Haar distribution. Each client transmits its screening indicator ζ ^ j , k \widehat{\zeta}_{j,k} ζ j , k , estimated local error rate Ψ ^ j , k \widehat{\Psi}_{j,k} Ψ j , k , and private local loading-space estimate U ~ j , k \widetilde{U}_{j,k} U j , k to the server. The server aggregates the corresponding projectors from clients with ζ ^ j , k = 1 \widehat{\zeta}_{j,k}=1 ζ j , k = 1 , using weights proportional to Ψ ^ j , k − 2 \widehat{\Psi}_{j,k}^{-2} Ψ j , k − 2 . If all screening indicators are zero, it instead outputs a data-independent random subspace drawn from the Haar distribution.
Theoretical guarantees
We next show that Algorithm 2 satisfies ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -FDP and attains the oracle rate in Theorem 1.
Proposition 2. Fix a mode k ∈ [ K ] k \in[K] k ∈ [ K ] . Suppose B j ( 0 ) ≥ 1 B_j^{(0)} \ge1 B j ( 0 ) ≥ 1 and ε j , δ j ∈ ( 0 , 1 ) \varepsilon_j,\delta_j \in(0,1) ε j , δ j ∈ ( 0 , 1 ) for every j ∈ [ m ] j \in[m] j ∈ [ m ] . Assume that the split and partition laws, block sizes b j b_j b j , and public grid R k \mathcal{R}_k R k are fixed independently of the raw observations. Then each complete local release ( ι ^ j , k , Ψ ^ j , k , U ^ j , k ) (\widehat{\iota}_{j,k},\widehat{\Psi}_{j,k},\widehat{U}_{j,k}) ( ι j , k , Ψ j , k , U j , k ) produced by Algorithm 2 is ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -centrally differentially private. Hence, the adaptive estimator A ^ k a d \widehat{A}^{\mathrm{ad}}_k A k ad satisfies ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -federated differential privacy.
Proposition 2 provides the same worst-case privacy guarantee as Proposition 1, under arbitrary record replacements and without distributional assumptions on the observations. For each client j ∈ [ m ] j \in[m] j ∈ [ m ] , the private threshold search satisfies ( ε j , 0 ) (\varepsilon_j,0) ( ε j , 0 ) -CDP. The tuning parameters R ^ j , k \widehat{R}_{j,k} R j , k and τ ^ j , k \widehat{\tau}_{j,k} τ j , k , screening indicator ι ^ j , k \widehat{\iota}_{j,k} ι j , k , and estimated local error rate Ψ ^ j , k \widehat{\Psi}_{j,k} Ψ j , k are obtained from its output by post-processing. For every fixed calibration output, the main-sample procedure satisfies ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -CDP. Since the split is independent of the observations and the two samples are disjoint, parallel composition preserves ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -CDP for the complete local release, even though each stage uses ε j \varepsilon_j ε j . These CDP guarantees for the complete local releases imply ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -FDP for their joint release. The server weights and final estimator inherit this guarantee by post-processing.
We next establish an error bound for the adaptive estimator.
Theorem 3. Fix a mode k ∈ [ K ] k \in[K] k ∈ [ K ] . Suppose that Assumptions 1–3 hold. Consider Algorithm 2 with block sizes b j = ⌈ C b [ r k + log ( n ∨ p ) ] ⌉ b_j = \lceil C_b[r_k + \log(n \vee p)]\rceil b j = ⌈ C b [ r k + log ( n ∨ p )]⌉ , j ∈ [ m ] j \in[m] j ∈ [ m ] , and the public grid in (23) for a fixed constant γ ≥ 1 \gamma\ge1 γ ≥ 1 . Suppose that p k ≥ 2 r k p_k \ge2r_k p k ≥ 2 r k , p k ≥ C p log ( n ∨ p ) p_k \ge C_p\log(n \vee p) p k ≥ C p log ( n ∨ p ) , and for every j ∈ [ m ] j \in[m] j ∈ [ m ] , ε j , δ j ∈ ( 0 , 1 ) \varepsilon_j,\delta_j \in(0,1) ε j , δ j ∈ ( 0 , 1 ) ,
C 0 ( p k + p ) ≤ λ j , 1 ( k ) / σ 2 ≤ c 0 ( n ∨ p ) 2 γ r k p k r k + log ( n ∨ p ) , (24) C_0(p_k+\sqrt{p}) \le\lambda_{j,1}^{(k)}/\sigma^2 \le c_0(n \vee p)^{2\gamma}\frac{r_kp_k}{r_k+\log(n \vee p)}, \tag*{(24)} C 0 ( p k + p ) ≤ λ j , 1 ( k ) / σ 2 ≤ c 0 ( n ∨ p ) 2 γ r k + log ( n ∨ p ) r k p k , (24) and
n j ε j ≥ C 0 { r k + log ( n ∨ p ) } { log { ( n ∨ p ) / δ j } + r k p k { 1 + log ( 1 / δ j ) } } , ∗ (25) n_j\varepsilon_j \ge C_0\{r_k+\log(n \vee p)\}\left\{\log\{(n \vee p)/\delta_j\}+\sqrt{r_kp_k\{1+\log(1/\delta_j)\}}\right\},* \tag*{(25)} n j ε j ≥ C 0 { r k + log ( n ∨ p )} { log {( n ∨ p ) / δ j } + r k p k { 1 + log ( 1/ δ j )} } , ∗ (25) where C b C_b C b , C p C_p C p , C 0 > 0 C_0 > 0 C 0 > 0 are sufficiently large absolute constants, and c 0 > 0 c_0 > 0 c 0 > 0 is the same constant as in the screening conditions (22) of Algorithm 2. Then there exist absolute constants C , c > 0 C,c > 0 C , c > 0 such that, with probability at least 1 − ( n ∨ p ) − c 1-(n \vee p)^{-c} 1 − ( n ∨ p ) − c , we have
∥ A ^ k a d ( A ^ k a d ) T − A k A k T ∥ F 2 ≤ C r k log ( n ∨ p ) ∑ j = 1 m ( Ψ j , k l o c ) − 2 ∧ ( 2 r k ) , \left\|\widehat{A}^{\mathrm{ad}}_k(\widehat{A}^{\mathrm{ad}}_k)^{\mathsf T}-A_kA_k^{\mathsf T}\right\|_{\mathrm{F}}^2
\le\frac{Cr_k\log(n \vee p)}{\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-2}}\wedge(2r_k), A k ad ( A k ad ) T − A k A k T F 2 ≤ ∑ j = 1 m ( Ψ j , k loc ) − 2 C r k log ( n ∨ p ) ∧ ( 2 r k ) , where Ψ j , k l o c \Psi_{j,k}^{\mathrm{loc}} Ψ j , k loc is defined in (11) .
The lower bound of the signal-to-noise ratio in (24) has the same form as (12) in Theorem 1. Since p k ≥ C p log ( n ∨ p ) p_k \ge C_p\log(n \vee p) p k ≥ C p log ( n ∨ p ) , the sample-size condition in (25) also has the same order as its oracle counterpart in (13) . The upper bound in (24) ensures that the lower endpoint d min = ( n ∨ p ) − γ d_{\min}=(n \vee p)^{-\gamma} d m i n = ( n ∨ p ) − γ of the public grid R k \mathcal{R}_k R k lies below the oracle radius R j , k R_{j,k} R j , k in (10) . Under these conditions, the proof of Theorem 3 shows that, with high probability, the calibrated Huber radii, Gaussian noise scales and aggregation weights satisfy
R ^ j , k ≍ R j , k , τ ^ j , k ≍ τ j , k , and w ^ j , k ≍ w j , k , ∀ j ∈ [ m ] , \widehat{R}_{j,k}\asymp R_{j,k},\quad\widehat{\tau}_{j,k}\asymp\tau_{j,k},\quad\text{and}\quad\widehat{w}_{j,k}\asymp w_{j,k},\quad\forall j\in[m], R j , k ≍ R j , k , τ j , k ≍ τ j , k , and w j , k ≍ w j , k , ∀ j ∈ [ m ] ,
and all clients satisfy (22) . The adaptive estimator therefore attains the oracle rate in Theorem 1 without the knowledge of the noise level or client-specific factor strengths.
Numerical analysis
In this section, we investigate the finite-sample performance of the proposed estimators. We first present simulation results in § 5.1 , followed by an application to the meditation traits electroencephalography (EEG) dataset in § 5.2 .
Simulation studies
We begin by describing the data generating process and the competing methods used throughout.
Data generating process. We first consider model (1) with order K = 2 K = 2 K = 2 , dimensions p 1 = p 2 = 15 p_1 = p_2 = 15 p 1 = p 2 = 15 , ranks r 1 = r 2 = 2 r_1 = r_2 = 2 r 1 = r 2 = 2 , and m = 3 m = 3 m = 3 clients. Each client has n j ∈ { 2000 , 4000 , 6000 , 8000 } n_j \in\{2000, 4000, 6000, 8000\} n j ∈ { 2000 , 4000 , 6000 , 8000 } observations, with equal sample sizes across the clients in each experiment. The target is the mode-1 loading space. For each mode ℓ \ell ℓ , we generate an orthonormal loading matrix A ℓ ∈ O p ℓ × r ℓ A_\ell\in\mathcal{O}_{p_\ell\times r_\ell} A ℓ ∈ O p ℓ × r ℓ from a Gaussian matrix via QR decomposition and use the same loading matrices for all clients. The entries of the core factors are independent N ( 0 , λ ) \mathcal{N}(0, \lambda) N ( 0 , λ ) variables, with λ = ∏ ℓ = 1 K p ℓ \lambda= \prod_{\ell=1}^{K} p_\ell λ = ∏ ℓ = 1 K p ℓ , and the idiosyncratic noise entries are independent N ( 0 , 1 ) \mathcal{N}(0, 1) N ( 0 , 1 ) variables. Factors and noise are independent across the observations and clients. We assign heterogeneous privacy budgets ε = ( 0.6 , 0.8 , 1 ) ⊤ \boldsymbol{\varepsilon} = (0.6, 0.8, 1)^\top ε = ( 0.6 , 0.8 , 1 ) ⊤ and δ = ( 0.2 , 0.2 , 0.2 ) ⊤ \boldsymbol{\delta} = (0.2, 0.2, 0.2)^\top δ = ( 0.2 , 0.2 , 0.2 ) ⊤ , to the three clients. Unless otherwise specified, every experiment is repeated 100 times.
Competing methods. We compare the following estimators:
ORACLE: Algorithm 1 with the oracle radius R j , k R_{j,k} R j , k in (10) and the oracle weights w j , k w_{j,k} w j , k in (11) , both computed from the true σ \sigma σ and λ j , 1 ( k ) \lambda_{j,1}^{(k)} λ j , 1 ( k ) .
ADA: Algorithm 2, using adaptively chosen Huber Radii, Gaussian noise scale and weights.
RADIUS: Algorithm 1 with the oracle weights w j , k w_{j,k} w j , k and a randomized radius R j , k = U j , k R j , k ∗ R_{j,k} = U_{j,k}R_{j,k}^{*} R j , k = U j , k R j , k ∗ , where R j , k ∗ R_{j,k}^{*} R j , k ∗ is the oracle radius in (10) and U j , k ∼ Unif ( 0.5 , 1.5 ) U_{j,k} \sim\operatorname{Unif}(0.5, 1.5) U j , k ∼ Unif ( 0.5 , 1.5 ) is drawn independently for each client.
WEIGHT: Algorithm 1 with the oracle radius R j , k R_{j,k} R j , k in (10) and uniform weights w j , k = 1 / m w_{j,k} = 1/m w j , k = 1/ m .
PRACTICAL: Algorithm 1 with the randomized radius as in RADIUS and uniform weights as in WEIGHT. This method uses no oracle quantity other than the scale of R j , k ∗ R_{j,k}^{*} R j , k ∗ .
HP-VEC: the high-probability privatized algorithm by Li et al. (2024a) applied to the vectorized observations vec ( X t ( j ) ) \operatorname{vec}(\mathbf{X}_t^{(j)}) vec ( X t ( j ) ) (detailed below).
The first two methods are those proposed in this paper. In particular, ORACLE with known σ \sigma σ and λ j , 1 ( k ) \lambda_{j,1}^{(k)} λ j , 1 ( k ) provides a benchmark, while ADA does not require the knowledge of either quantity. Methods RADIUS, WEIGHT, and PRACTICAL are variants of ORACLE. Note that none of these variants are genuinely feasible in practice, as even PRACTICAL still relies partially on the oracle radius. Methods ORACLE, ADA, RADIUS, WEIGHT, and PRACTICAL all directly consider the tensor-valued observations X t ( j ) \mathbf{X}_t^{(j)} X t ( j ) and target the mode-1 loading space span ( A 1 ) \operatorname{span}(A_1) span ( A 1 ) . In contrast, HP-VEC considers the vectorized observations vec ( X t ( j ) ) ∈ R p \operatorname{vec}(\mathbf{X}_t^{(j)}) \in\mathbb{R}^{p} vec ( X t ( j ) ) ∈ R p , where p = ∏ ℓ = 1 K p ℓ p = \prod_{\ell=1}^{K} p_\ell p = ∏ ℓ = 1 K p ℓ , and targets the resulting loading space span ( A 2 ⊗ A 1 ) ⊂ R p \operatorname{span}(A_2 \otimes A_1) \subset\mathbb{R}^{p} span ( A 2 ⊗ A 1 ) ⊂ R p of the vectorized model. For all methods, we set the target rank to r 1 r_1 r 1 on tensor-valued observations and to r = ∏ ℓ = 1 K r ℓ r = \prod_{\ell=1}^{K} r_\ell r = ∏ ℓ = 1 K r ℓ on vectorized observations. Additionally, we define methods ORACLE-M1 and ADA-M1 as applying ORACLE and ADA to client 1 alone, to assess the benefit of federated aggregation.
Evaluation and results. For evaluation, we report the empirical mean and standard deviation of the projector distance of the estimated projectors to their respective targets, i.e., ∥ A ^ 1 A ^ 1 ⊤ − A 1 A 1 ⊤ ∥ F \left\|\widehat A_1\widehat A_1^\top-A_1A_1^\top\right\|_F A 1 A 1 ⊤ − A 1 A 1 ⊤ F for methods (i)–(iv), and ∥ A ^ A ^ ⊤ − ( A 2 ⊗ A 1 ) ( A 2 ⊗ A 1 ) ⊤ ∥ F \left\|\widehat A\widehat A^\top-(A_2\otimes A_1)(A_2\otimes A_1)^\top\right\|_F A A ⊤ − ( A 2 ⊗ A 1 ) ( A 2 ⊗ A 1 ) ⊤ F for method (v). To implement Algorithm 1, we fix the block size at b j = 4 b_j = 4 b j = 4 , and C R = 15 C_R = 15 C R = 15 in (10) . The same parameters are used in Algorithm 2, and we additionally set C A = 8 C_A = 8 C A = 8 and d min = ( n ∨ p ) − 2 d_{\min} = (n \lor p)^{-2} d m i n = ( n ∨ p ) − 2 in (23) , where n = ∑ j n j n = \sum_j n_j n = ∑ j n j and p = ∑ ℓ p ℓ p = \sum_\ell p_\ell p = ∑ ℓ p ℓ . We perform additional sensitivity analysis on b j b_j b j and C R C_R C R in Section A.3 of the supplementary material.
Table 1 reports the results for K = 2 K = 2 K = 2 . The method WEIGHT performs comparably to ORACLE, suggesting that uniform weighting incurs little loss in this setting with equal sample sizes and homogeneous factor strengths. Thus, we consider heterogeneous factor strengths at the end of this subsection to assess the benefit of the proposed aggregation weights. By contrast, RADIUS and PRACTICAL have larger mean errors and greater variability than ORACLE for n j ≥ 4000 n_j \ge4000 n j ≥ 4000 , illustrating the sensitivity to the choice of radius. The reported errors of HP-VEC are smaller than those of ORACLE at the first three sample sizes, but slightly larger at n j = 8000 n_j = 8000 n j = 8000 . This may be accounted for by its weaker privacy requirement: HP-VEC directly perturbs local empirical projectors with Gaussian noise and provides a high-probability privacy guarantee under independent-copy replacements. Our procedure additionally uses Huber aggregation and private stability testing, with a data-independent fallback when the test fails. These safeguards ensure worst-case privacy under arbitrary record replacements, but can incur additional estimation error when local sample sizes are small. Moreover, HP-VEC concerns the full vectorized loading space rather than the model-1 loading space.
Table 1. Mean Frobenius projector distance over 100 realizations for K = 2 K = 2 K = 2 , p ℓ = 15 p_\ell= 15 p ℓ = 15 , r ℓ = 2 r_\ell= 2 r ℓ = 2 , m = 3 m = 3 m = 3 , ε = ( 0.6 , 0.8 , 1 ) ⊤ \varepsilon= (0.6, 0.8, 1)^\top ε = ( 0.6 , 0.8 , 1 ) ⊤ , and δ = ( 0.2 , 0.2 , 0.2 ) ⊤ \delta= (0.2, 0.2, 0.2)^\top δ = ( 0.2 , 0.2 , 0.2 ) ⊤ , with standard deviations in parentheses.
Among the procedures compared here, only ADA requires no knowledge of the noise level or factor strengths for either local estimation or server aggregation. Nevertheless, ADA performs reasonably well relative to ORACLE across the reported sample sizes, illustrating the effectiveness of private calibration. Both federated estimators also have smaller mean errors than their corresponding client-1 versions at every sample size, consistent with the aggregation benefit described in Theorem 1.
We next repeat the experiment with order K = 3 K = 3 K = 3 , dimensions p ℓ = 10 p_\ell= 10 p ℓ = 10 , and ranks r ℓ = 2 r_\ell= 2 r ℓ = 2 for every mode. All other numerical choices are unchanged. Table 2 shows that both ORACLE and ADA outperform most of the other methods overall, with ORACLE attaining the smallest mean error at every sample size. In particular, HP-VEC has substantially larger reported errors than both of the proposed estimators. This difference is consistent with the benefit of exploiting the multiway structure: vectorization leads to a higher-dimensional estimation problem, whereas our procedures retain the tensor structure and directly estimate the mode-wise loading spaces.
Table 2. Mean Frobenius projector distance over 100 realizations for K = 3 K = 3 K = 3 , p ℓ = 10 p_\ell= 10 p ℓ = 10 , r ℓ = 2 r_\ell= 2 r ℓ = 2 , m = 3 m = 3 m = 3 , ε = ( 0.6 , 0.8 , 1 ) ⊤ \varepsilon= (0.6, 0.8, 1)^\top ε = ( 0.6 , 0.8 , 1 ) ⊤ , and δ = ( 0.2 , 0.2 , 0.2 ) ⊤ \delta= (0.2, 0.2, 0.2)^\top δ = ( 0.2 , 0.2 , 0.2 ) ⊤ , with standard deviations in parentheses.
Heterogeneous factor strengths. We next examine how heterogeneity in factor strength affects estimation and aggregation across clients. We consider the setting with K = 2 K = 2 K = 2 , p 1 = p 2 = 15 p_1 = p_2 = 15 p 1 = p 2 = 15 , and r 1 = r 2 = 2 r_1 = r_2 = 2 r 1 = r 2 = 2 . There are three clients, each with sample size n j = 12000 n_j = 12000 n j = 12000 and privacy parameters ( ε j , δ j ) = ( 0.8 , 0.2 ) (\varepsilon_j, \delta_j) = (0.8, 0.2) ( ε j , δ j ) = ( 0.8 , 0.2 ) . The entries of the core factor at client j j j have variance λ j \lambda_j λ j . With p = p 1 p 2 = 225 p = p_1p_2 = 225 p = p 1 p 2 = 225 , we consider four configurations:
S 1 : ( λ 1 , λ 2 , λ 3 ) = ( p , p 0.6 , p 0.6 ) , S 2 : q u a d ( λ 1 , λ 2 , λ 3 ) = ( p 0.6 , p , p ) , S 3 : ( λ 1 , λ 2 , λ 3 ) = ( p 0.6 , p 0.6 , p 0.6 ) , S 4 : ( λ 1 , λ 2 , λ 3 ) = ( p , p , p ) . \begin{aligned}
\mathrm{S1}: \quad&(\lambda_1,\lambda_2,\lambda_3) = (p,p^{0.6},p^{0.6}), \qquad\mathrm{S2}:quad (\lambda_1,\lambda_2,\lambda_3) = (p^{0.6},p,p), \\
\mathrm{S3}: \quad&(\lambda_1,\lambda_2,\lambda_3) = (p^{0.6},p^{0.6},p^{0.6}), \qquad\mathrm{S4}: \quad(\lambda_1,\lambda_2,\lambda_3) = (p,p,p).
\end{aligned} S1 : S3 : ( λ 1 , λ 2 , λ 3 ) = ( p , p 0.6 , p 0.6 ) , S2 : q u a d ( λ 1 , λ 2 , λ 3 ) = ( p 0.6 , p , p ) , ( λ 1 , λ 2 , λ 3 ) = ( p 0.6 , p 0.6 , p 0.6 ) , S4 : ( λ 1 , λ 2 , λ 3 ) = ( p , p , p ) .
S1 and S2 represent heterogeneous configurations with one and two strong-factor clients, respectively, while S3 and S4 consider homogeneous weak- and strong-factor clients, respectively. All other implementation choices remain unchanged.
Table 3 reports the results for all the aforementioned methods. The results illustrate the importance of accounting for heterogeneous factor strengths when choosing aggregation weights. In the heterogeneous settings S1 and S2, ORACLE achieves substantially smaller errors than WEIGHT, which assigns equal weights to all clients, whereas the two methods perform similarly in S3 and S4. In S1, where client 1 is the only client with strong factors, ORACLE and ORACLE-M1 have similar mean errors. In S2, where client 1 has weak factors and the other two clients have strong factors, ORACLE substantially improves on ORACLE-M1. In both heterogeneous settings, ADA also achieves smaller mean errors than WEIGHT, RADIUS, and PRACTICAL, despite requiring no knowledge of the noise level or factor strengths. The large mean error of ADA in S3 highlights the difficulty of adaptive estimation when all clients have weak factors. Although the procedure does not require factor strengths as inputs, weaker factors can increase the dispersion of the blockwise calibration projectors, leading to larger estimated radii R ^ j , k \widehat R_{j,k} R j , k and local error rates Ψ ^ j , k \widehat\Psi_{j,k} Ψ j , k through (21) . Larger values make it harder to meet the screening conditions in (22) ; if no client passes screening, the server returns a Haar random subspace, which can lead to a large estimation error.
Table 3. Mean Frobenius projector distance over 100 realizations under factor-strength configurations S1–S4 for K = 2 K = 2 K = 2 , p ℓ = 15 p_\ell= 15 p ℓ = 15 , r ℓ = 2 r_\ell= 2 r ℓ = 2 , m = 3 m = 3 m = 3 , n j = 12000 n_j = 12000 n j = 12000 , and ( ε j , δ j ) = ( 0.8 , 0.2 ) (\varepsilon_j, \delta_j) = (0.8, 0.2) ( ε j , δ j ) = ( 0.8 , 0.2 ) , with standard deviations in parentheses.
Real data analysis
We analyze the meditation traits dataset studied by Braboszcz et al. [9 ] . The dataset consists of tensor-valued EEG observations with three anatomical regions, six frequency bands, and three time frames per window, giving a tensor X t ( j ) ∈ R 3 × 6 × 3 \mathbf{X}_t^{(j)} \in\mathbb{R}^{3 \times6 \times3} X t ( j ) ∈ R 3 × 6 × 3 for each observation. In total, it contains 13740 windows from 29 participants across four groups: a control group and three practitioner groups (Vipassana, Himalayan Yoga, and Isha Shoonya). We first demean each feature within each participant, and then form m = 4 m = 4 m = 4 clients by grouping the windows according to the group membership, giving local sample sizes ( n 1 , n 2 , n 3 , n 4 ) = ( 2380 , 3880 , 3780 , 3700 ) (n_1, n_2, n_3, n_4) = (2380, 3880, 3780, 3700) ( n 1 , n 2 , n 3 , n 4 ) = ( 2380 , 3880 , 3780 , 3700 ) . The target parameter is the mode-1 loading space, corresponding to the anatomical regions. We fix the factor ranks at ( r 1 , r 2 , r 3 ) = ( 2 , 2 , 2 ) (r_1, r_2, r_3) = (2, 2, 2) ( r 1 , r 2 , r 3 ) = ( 2 , 2 , 2 ) , and set δ j = 0.2 \delta_j = 0.2 δ j = 0.2 for all clients.
We compare three estimators: ADA, PLUG, and HP-VEC. Specifically, ADA applies Algorithm 2. Also for each j ∈ [ 4 ] j \in[4] j ∈ [ 4 ] , ADA-MJ applies the same adaptive procedure to client j j j alone, using its local sample and privacy parameters. These single-client versions allow us to assess the benefit of federated aggregation. Second, we introduce PLUG which applies Algorithm 1 with the following plug-in estimates. For each client j j j , let Σ ^ k ( j ) \widehat{\boldsymbol{\Sigma}}_k^{(j)} Σ k ( j ) denote its sample mode-k k k covariance matrix, with eigenvalues e ^ j , 1 ≥ ⋯ ≥ e ^ j , p k \widehat e_{j,1} \ge\cdots\ge\widehat e_{j,p_k} e j , 1 ≥ ⋯ ≥ e j , p k . Under the tensor factor model, the population covariance satisfies Σ k ( j ) = A k Λ j , ( k ) A k T + σ p k 2 I p k \boldsymbol{\Sigma}_k^{(j)} = A_k\Lambda_{j,(k)}A_k^{\mathsf{T}} + \sigma_{p_k}^{2}I_{p_k} Σ k ( j ) = A k Λ j , ( k ) A k T + σ p k 2 I p k . Its smallest p k − r k p_k-r_k p k − r k eigenvalues therefore equal σ p k 2 \sigma_{p_k}^{2} σ p k 2 , so we set
σ ^ j , 0 2 : = median { e ^ j , r k + 1 , … , e ^ j , p k } , σ ^ j : = σ ^ j , 0 2 / p k , λ ^ j , 1 ( k ) : = e ^ j , 1 − σ ^ j , 0 2 . \widehat{\sigma}_{j,0}^{2} := \operatorname{median}\{\widehat e_{j,r_k+1},\ldots,\widehat e_{j,p_k}\}, \quad\widehat{\sigma}_j := \sqrt{\widehat{\sigma}_{j,0}^{2}/p_k}, \quad\widehat{\lambda}_{j,1}^{(k)} := \widehat e_{j,1} - \widehat{\sigma}_{j,0}^{2}. σ j , 0 2 := median { e j , r k + 1 , … , e j , p k } , σ j := σ j , 0 2 / p k , λ j , 1 ( k ) := e j , 1 − σ j , 0 2 .
The plug-in radius R ^ j , k \widehat R_{j,k} R j , k is obtained by replacing σ \sigma σ and λ j , 1 ( k ) \lambda_{j,1}^{(k)} λ j , 1 ( k ) in (10) with σ ^ j \widehat{\sigma}_j σ j and λ ^ j , 1 ( k ) \widehat{\lambda}_{j,1}^{(k)} λ j , 1 ( k ) , respectively.
The remaining tuning parameters and aggregation weights are computed using the same plug-in estimates. Since these plug-in estimates are not privatized, PLUG is not covered by the privacy guarantee in Proposition 1 and is therefore included only as an empirical benchmark. Lastly, HP-VEC applies the algorithm of Li et al. (2024a) to the vectorized observations vec ( X t ( j ) ) \operatorname{vec}(X_t^{(j)}) vec ( X t ( j ) ) with target rank r = 8 r = 8 r = 8 . Its plug-in estimates are computed similarly to those of PLUG, but using the sample covariance matrix of the vectorized data. The original high-probability privacy guarantee does not directly cover this plug-in implementation.
We evaluate all tensor-based estimators, including the single-client versions, against the same non-private mode-1 reference constructed from all four clients. For HP-VEC, we use a separate non-private reference for the full vectorized loading space. Both references are defined in Section A.2 of the supplementary material. Table 4 reports the corresponding Frobenius projector distance, which measures the degree of agreement with empirical loading spaces rather than error relative to unknown population loading spaces. The method ADA has smaller distances than PLUG in every reported configuration and achieves the smallest distance among the tensor-based estimators in several configurations. These results support the effectiveness of private adaptive calibration compared with direct plug-in tuning in this application.
Table 4. Frobenius projector distance to the corresponding non-private reference on the meditation traits dataset for K = 3 K = 3 K = 3 , ( p 1 , p 2 , p 3 ) = ( 3 , 6 , 3 ) (p_1,p_2,p_3) = (3, 6, 3) ( p 1 , p 2 , p 3 ) = ( 3 , 6 , 3 ) , r ℓ = 2 r_\ell= 2 r ℓ = 2 , m = 4 m = 4 m = 4 , n j ∈ { 2380 , 3880 , 3780 , 3700 } n_j \in\{2380, 3880, 3780, 3700\} n j ∈ { 2380 , 3880 , 3780 , 3700 } , ε j = ε \varepsilon_j = \varepsilon ε j = ε , and δ j = 0.2 \delta_j = 0.2 δ j = 0.2 .
We note that the performance varies across block sizes and privacy budgets. In particular, no method achieves the smallest distance in every configuration. The federated ADA estimator does not uniformly improve on the best single-client result as it occasionally reports somewhat larger distances with b j = 6 b_j = 6 b j = 6 or 8 8 8 . This may partly reflect the trade-off associated with block size: increasing the block size provides more observations for each blockwise estimate but reduces the number of blocks available for calibration, testing, and local aggregation. This trade-off, together with randomness of the algorithm, can affect the final estimate.
To assess variability due to algorithmic randomness, we repeat the analysis 100 times on the same dataset. Specifically, we consider block sizes b j ∈ { 4 , 6 , 8 } b_j \in\{4,6,8\} b j ∈ { 4 , 6 , 8 } and privacy parameters ε j = ε ∈ { 0.4 , 0.6 , 0.8 , 1 , 1.2 , 1.5 , 2 } \varepsilon_j = \varepsilon\in\{0.4,0.6,0.8,1,1.2,1.5,2\} ε j = ε ∈ { 0.4 , 0.6 , 0.8 , 1 , 1.2 , 1.5 , 2 } , with δ j = 0.2 \delta_j = 0.2 δ j = 0.2 for all clients. The block size applies only to the tensor methods. We regenerate the random partitions and the randomness used in calibration, testing, and release in each run, while holding the dataset and deterministic tuning choices fixed.
Figure 1 shows that, for b j = 4 b_j = 4 b j = 4 , ADA outperforms the other methods across all examined privacy budgets. Its mean distance decreases steadily as ε \varepsilon ε increases, and its variability is substantially smaller at larger values of ε \varepsilon ε . These results show that its favorable performance with this block size persists across repeated randomized runs. We additionally examine the frequency of complete fallback, defined as the event that no client passes the local stability test. For PLUG, this event occurs in nearly all runs across the examined settings, which explains the large distances between its outputs and the non-private reference and limited improvement with increasing ε \varepsilon ε . Complete fallback is rare for ADA. Substantial variability nevertheless remains, particularly at b j = 6 b_j = 6 b j = 6 , suggesting that complete fallback alone does not explain the variation across the runs. The reported mean distances of HP-VEC decrease modestly with ε \varepsilon ε and then largely level off, with comparatively little variability at larger budgets.
Figure 1. Mean Frobenius projector distance to the corresponding non-private reference over 100 randomized runs on the meditation traits dataset for K = 3 K = 3 K = 3 , ( p 1 , p 2 , p 3 ) = ( 3 , 6 , 3 ) (p_1,p_2,p_3) = (3,6,3) ( p 1 , p 2 , p 3 ) = ( 3 , 6 , 3 ) , r ℓ = 2 r_{\ell} = 2 r ℓ = 2 , m = 4 m = 4 m = 4 , n j ∈ { 2380 , 3880 , 3780 , 3700 } n_j \in\{2380,3880,3780,3700\} n j ∈ { 2380 , 3880 , 3780 , 3700 } , ε j = ε \varepsilon_j = \varepsilon ε j = ε , and δ j = 0.2 \delta_j = 0.2 δ j = 0.2 , with shaded bands indicating the 10th–90th percentiles. Columns correspond to b j = 4 , 6 , 8 b_j = 4,6,8 b j = 4 , 6 , 8 .
Discussion
We have developed a federated estimator of common loading spaces in Tucker tensor factor models under the worst-case, record-level FDP. Our method combines private stability testing, Huber aggregation and Gaussian perturbation at each client, with weighted aggregation at the server. We establish its non-asymptotic upper bound governed by a scaled harmonic mean of the squared local error rates, together with the minimax lower bound, and show that the former match the latter up to logarithmic terms under strong factors, fixed ranks and noise variance. We further propose a private calibration procedure that selects the Huber radii, Gaussian noise scales and aggregation weights without requiring any knowledge of the noise level or client-specific factor strengths. The resulting adaptive estimator preserves the privacy guarantee and, under the stated conditions, attains the oracle high-probability error rate.
Several extensions remain open for future research. Firstly, we assume independent Gaussian idiosyncratic noise. Extending our results to heavy-tailed noise may require robust estimation within blocks and new error bounds for the resulting projector estimates. While it is possible to relax the independence assumption to allow for temporally dependent noise, such temporal dependence raises additional privacy concerns in addition to requiring new estimation error bounds. In particular, our record-level FDP guarantee remains valid under temporal dependence, but information about an observation may still be inferred from other correlated records. Addressing such inference calls for suitable privacy notions and the development of methods satisfying them, together with corresponding estimation error bounds.
Secondly, we assume that the factor ranks are known. For unknown ranks, Han et al. [31 ] and Chen and Lam [19 ] , among others, proposed information criteria or eigenvalue-ratio methods in the non-private setting. Adapting these methods to our federated procedure would require establishing conditions for consistent rank selection and, more importantly, developing a private rank-selection mechanism to ensure FDP for the complete release.
Lastly, note that all clients are assumed to share the same loading spaces. We could extend the model to include both shared and client-specific loading components. In the non-private setting, Yang and Ma [54 ] first estimated each dataset’s full signal space, including both shared and dataset-specific components. They then aggregated the corresponding projection matrices and took the leading eigenspace as an estimate of the shared space. We could adapt this approach under FDP by aggregating privatized local projectors to estimate the shared space, then estimating client-specific spaces by projecting out the estimated shared space. We leave the development of such an extension, including suitable identifiability conditions and estimation error bounds under FDP, to future work.
Bibliography
Additional details and discussion
Estimation of multiple loading matrices
Suppose that our interest lies in estimating a subset of modes K ⊂ [ K ] \mathcal{K} \subset[K] K ⊂ [ K ] , and each client j j j has a total privacy budget ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) . Client j j j allocates budgets ( ε j , k , δ j , k ) (\varepsilon_{j,k},\delta_{j,k}) ( ε j , k , δ j , k ) for k ∈ K k \in\mathcal{K} k ∈ K such that
∑ k ∈ K ε j , k ≤ ε j , ∑ k ∈ K δ j , k ≤ δ j . \sum_{k\in\mathcal{K}}\varepsilon_{j,k}\leq\varepsilon_j,\qquad\sum_{k\in\mathcal{K}}\delta_{j,k}\leq\delta_j. k ∈ K ∑ ε j , k ≤ ε j , k ∈ K ∑ δ j , k ≤ δ j .
For each mode k ∈ K k \in\mathcal{K} k ∈ K , we run our procedure in the main text with budgets ( ε j , k , δ j , k ) (\varepsilon_{j,k},\delta_{j,k}) ( ε j , k , δ j , k ) and with independently generated noise matrices.
By the composition theorem for DP [24 Theorem 3.16] , the vector of all mode-wise outputs released by client j j j is ( ∑ k ∈ K ε j , k , ∑ k ∈ K δ j , k ) (\sum_{k\in\mathcal{K}}\varepsilon_{j,k},\sum_{k\in\mathcal{K}}\delta_{j,k}) ( ∑ k ∈ K ε j , k , ∑ k ∈ K δ j , k ) -CDP, and hence ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -CDP. The server then aggregates the privatized projectors separately for each mode. This final aggregation step is post-processing of the released client outputs, and hence it does not incur additional privacy loss.
More generally, the budgets ( ε j , k , δ j , k ) (\varepsilon_{j,k},\delta_{j,k}) ( ε j , k , δ j , k ) for every client-mode pair ( j , k ) (j,k) ( j , k ) can be directly specified. This motivates us to extend the notion of FDP as follows.
Definition 3 (Multi-mode federated differential privacy). Let E = ( ε j , k ) j ∈ [ m ] , k ∈ [ K ] E=(\varepsilon_{j,k})_{j\in[m],k\in[K]} E = ( ε j , k ) j ∈ [ m ] , k ∈ [ K ] and D = ( δ j , k ) j ∈ [ m ] , k ∈ [ K ] D=(\delta_{j,k})_{j\in[m],k\in[K]} D = ( δ j , k ) j ∈ [ m ] , k ∈ [ K ] be the matrices of privacy budgets. A randomized mechanism
M : Z n 1 × ⋯ × Z n m → Z ~ 1 × ⋯ × Z ~ K \mathcal{M}:\mathbb{Z}^{n_1}\times\cdots\times\mathbb{Z}^{n_m}\to\widetilde{\mathbb{Z}}_1\times\cdots\times\widetilde{\mathbb{Z}}_K M : Z n 1 × ⋯ × Z n m → Z 1 × ⋯ × Z K is called ( E , D ) (E,D) ( E , D ) -multi-mode federated differentially private if, for every client j ∈ [ m ] j\in[m] j ∈ [ m ] , the combined output ( M j , 1 , … , M j , K ) (\mathcal{M}_{j,1},\ldots,\mathcal{M}_{j,K}) ( M j , 1 , … , M j , K ) over all modes is ( ∑ k = 1 K ε j , k , ∑ k = 1 K δ j , k ) (\sum_{k=1}^{K}\varepsilon_{j,k},\sum_{k=1}^{K}\delta_{j,k}) ( ∑ k = 1 K ε j , k , ∑ k = 1 K δ j , k ) -centrally differentially private with respect to client j j j ’s local dataset.
Applying our federated estimation mode-by-mode with independently generated noise yields an ( E , D ) (E,D) ( E , D ) -multi-mode FDP mechanism. In particular, if
∑ k = 1 K ε j , k ≤ ε j , ∑ k = 1 K δ j , k ≤ δ j , \sum_{k=1}^{K}\varepsilon_{j,k}\leq\varepsilon_j,\qquad\sum_{k=1}^{K}\delta_{j,k}\leq\delta_j, k = 1 ∑ K ε j , k ≤ ε j , k = 1 ∑ K δ j , k ≤ δ j ,
for all j ∈ [ m ] j\in[m] j ∈ [ m ] , then the resulting mechanism is also ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -federated differentially private in the sense of Definition 2, with
ε = ( ∑ k = 1 K ε 1 , k , … , ∑ k = 1 K ε m , k ) ⊤ , δ = ( ∑ k = 1 K δ 1 , k , … , ∑ k = 1 K δ m , k ) ⊤ . \varepsilon=\left(\sum_{k=1}^{K}\varepsilon_{1,k},\ldots,\sum_{k=1}^{K}\varepsilon_{m,k}\right)^\top,\qquad
\delta=\left(\sum_{k=1}^{K}\delta_{1,k},\ldots,\sum_{k=1}^{K}\delta_{m,k}\right)^\top. ε = ( k = 1 ∑ K ε 1 , k , … , k = 1 ∑ K ε m , k ) ⊤ , δ = ( k = 1 ∑ K δ 1 , k , … , k = 1 ∑ K δ m , k ) ⊤ .
The non-private estimators are computed on the same dataset as the private estimators. To compute the non-private estimator for ADA and PLUG, for each client j j j , let
Σ ^ 1 ( j ) , f u l l : = n j − 1 ∑ t = 1 n j M 1 ( X t ( j ) ) M 1 ( X t ( j ) ) ⊤ \widehat{\Sigma}_1^{(j),\mathrm{full}}:=n_j^{-1}\sum_{t=1}^{n_j}\mathcal{M}_1(\mathbf{X}_t^{(j)})\mathcal{M}_1(\mathbf{X}_t^{(j)})^\top Σ 1 ( j ) , full := n j − 1 t = 1 ∑ n j M 1 ( X t ( j ) ) M 1 ( X t ( j ) ) ⊤
denote the full-sample mode-1 covariance matrix, and U ^ j n p : = SVD r 1 ( Σ ^ 1 ( j ) , f u l l ) \widehat{U}_{j}^{\mathrm{np}} := \operatorname{SVD}_{r_{1}}\left(\widehat{\Sigma}_{1}^{(j),\mathrm{full}}\right) U j np := SVD r 1 ( Σ 1 ( j ) , full ) the matrix of its top r 1 r_{1} r 1 eigenvectors. Then we compute
M ^ 1 n p : = ∑ j = 1 m n j ∑ ℓ = 1 m n ℓ U ^ j n p ( U ^ j n p ) T , \widehat{M}_{1}^{\mathrm{np}} := \sum_{j=1}^{m} \frac{n_j}{\sum_{\ell=1}^{m} n_\ell} \widehat{U}_{j}^{\mathrm{np}}\left(\widehat{U}_{j}^{\mathrm{np}}\right)^{\mathsf{T}}, M 1 np := j = 1 ∑ m ∑ ℓ = 1 m n ℓ n j U j np ( U j np ) T ,
and the non-private reference is set as A ^ 1 n p : = SVD r 1 ( M ^ 1 n p ) \widehat{A}_{1}^{\mathrm{np}} := \operatorname{SVD}_{r_{1}}\left(\widehat{M}_{1}^{\mathrm{np}}\right) A 1 np := SVD r 1 ( M 1 np ) . The weights { n j / ∑ ℓ n ℓ } j \{n_j/\sum_{\ell}n_{\ell}\}_j { n j / ∑ ℓ n ℓ } j are the federated analogue of the pooling, in that if the clients shared the same mode-1 loading space, the pooled sample covariance would be the same as the weighted average of the local sample covariances up to the isotropic noise.
The non-private reference for HP-VEC is computed analogously as above, except that all estimations are based on the vectorized data. In detail, its non-private reference consists of the leading r = ∏ k = 1 K r k = 8 r=\prod_{k=1}^{K}r_k=8 r = ∏ k = 1 K r k = 8 eigenvectors of
Σ ^ v e c r e f : = 1 m ∑ j = 1 m { 1 n j ∑ t = 1 n j vec ( X t ( j ) ) vec ( X t ( j ) ) T } . \widehat{\Sigma}_{\mathrm{vec}}^{\mathrm{ref}} := \frac{1}{m}\sum_{j=1}^{m}\left\{\frac{1}{n_j}\sum_{t=1}^{n_j}\operatorname{vec}\left(X_t^{(j)}\right)\operatorname{vec}\left(X_t^{(j)}\right)^{\mathsf{T}}\right\}. Σ vec ref := m 1 j = 1 ∑ m { n j 1 t = 1 ∑ n j vec ( X t ( j ) ) vec ( X t ( j ) ) T } .
Additional numerical results
Sensitivity. We further examine the sensitivity of the estimators to the choice of b j b_j b j and C R C_R C R . Continuing with the data generating process in § 5.1 , we fix K = 2 K=2 K = 2 , p 1 = p 2 = 15 p_1=p_2=15 p 1 = p 2 = 15 , r 1 = r 2 = 2 r_1=r_2=2 r 1 = r 2 = 2 , and n j = 4000 n_j=4000 n j = 4000 , retaining the privacy budgets and other settings of the preceding experiment. We vary b j ∈ { 4 , 5 , 6 , 7 } b_j\in\{4,5,6,7\} b j ∈ { 4 , 5 , 6 , 7 } while holding C R = 15 C_R=15 C R = 15 , and then vary C R ∈ { 15 , 16 , 17 , 18 } C_R\in\{15,16,17,18\} C R ∈ { 15 , 16 , 17 , 18 } while holding b j = 4 b_j=4 b j = 4 . We experiment oracle, ada, radius, weight, and practical which involve b j b_j b j ; similarly for C R C_R C R except that ada calibrates its radius and does not depend on C R C_R C R . We use the same simulated datasets across all settings within each replication, and the results are shown in Figure A1 .
Figure A1. Mean Frobenius projector distance for K = 2 K = 2 K = 2 , p ℓ = 15 p_{\ell} = 15 p ℓ = 15 , r ℓ = 2 r_{\ell} = 2 r ℓ = 2 , m = 3 m = 3 m = 3 , n j = 4000 n_j = 4000 n j = 4000 , ε = ( 0.6 , 0.8 , 1 ) T \varepsilon= (0.6, 0.8, 1)^{\mathsf{T}} ε = ( 0.6 , 0.8 , 1 ) T , and δ = ( 0.2 , 0.2 , 0.2 ) T \delta= (0.2, 0.2, 0.2)^{\mathsf{T}} δ = ( 0.2 , 0.2 , 0.2 ) T , with error bars indicating one standard deviation across replications. The left panel varies b j b_j b j with C R = 15 C_R = 15 C R = 15 , while the right panel varies C R C_R C R with b j = 4 b_j = 4 b j = 4 . Markers are slightly offset horizontally for clarity. The method ada does not depend on C R C_R C R , so its results are unchanged in the right panel.
For oracle and weight, increasing the block size b j b_j b j from 4 to 5 reduces the mean error and its variability, while further increasing its value produces modest changes. We note that the errors of ada increases with the block size, which is mainly due to the screening rule in (22) , as client 1 (with stronger privacy constraint tends to fail the screening when b j = 6 b_j=6 b j = 6 or 7. Analogous patterns are observed when varying C R C_R C R , except that the performance of ada is unchanged because its calibrated radius and weights do not depend on C R C_R C R . Overall, these results mark a range of relatively stable performance for oracle, while indicating considerable sensitivity of ada to the block size, and for either, the privacy is always guaranteed as established in our theoretical results.
Proofs for the main results
Proofs on privacy guarantee
Proof of Lemma 1. If the two lists are identical, both conclusions are immediate. Otherwise, let i 0 ∈ [ B ] i_0\in[B] i 0 ∈ [ B ] be the unique index at which they differ, so that P [ ℓ ] ′ = P [ ℓ ] P^{[\ell]\prime}=P^{[\ell]} P [ ℓ ] ′ = P [ ℓ ] for every ℓ ≠ i 0 \ell\ne i_0 ℓ = i 0 . Define the two objectives
F ( H ) : = ∑ ℓ = 1 B ρ R ( ∥ H − P [ ℓ ] ∥ F ) , F ′ ( H ) : = ∑ ℓ = 1 B ρ R ( ∥ H − P [ ℓ ] ′ ∥ F ) . (26) F(H):=\sum_{\ell=1}^{B}\rho_R\left(\lVert H-P^{[\ell]}\rVert_F\right),\qquad F'(H):=\sum_{\ell=1}^{B}\rho_R\left(\lVert H-P^{[\ell]\prime}\rVert_F\right). \tag*{(26)} F ( H ) := ℓ = 1 ∑ B ρ R ( ∥ H − P [ ℓ ] ∥ F ) , F ′ ( H ) := ℓ = 1 ∑ B ρ R ( ∥ H − P [ ℓ ] ′ ∥ F ) . (26)
By (4) , H ^ \widehat{H} H and H ^ ′ \widehat{H}' H ′ are the minimizers of F F F and F ′ F' F ′ over all compatible symmetric matrices, respectively. Equation (5) is direct. To see this, in the double sum in (4) , only the ordered pairs ( i 0 , h ) (i_0,h) ( i 0 , h ) and ( ℓ , i 0 ) (\ell,i_0) ( ℓ , i 0 ) , where h , ℓ ≠ i 0 h,\ell\ne i_0 h , ℓ = i 0 , can change; the indicator corresponding to ( i 0 , i 0 ) (i_0,i_0) ( i 0 , i 0 ) remains equal to one. Hence at most 2 ( B − 1 ) 2(B-1) 2 ( B − 1 ) indicators can change, and division by B B B gives (5) .
We show (6) in the remainder of this proof. Suppose q ≥ 7 B / 8 q \ge7B/8 q ≥ 7 B /8 . Then note that for each h ∈ [ B ] h \in[B] h ∈ [ B ] , the sum over ℓ \ell ℓ in (4) is the number of matrices within the R / 8 R/8 R /8 -neighborhood (in terms of Frobenius norm) of P [ h ] P^{[h]} P [ h ] . Since the average of these B B B neighborhood sizes is q q q , there must exist h 0 ∈ [ B ] h_0 \in[B] h 0 ∈ [ B ] such that
∣ { ℓ ∈ [ B ] : ∥ P [ ℓ ] − P [ h 0 ] ∥ F ≤ R / 8 } ∣ ≥ 7 B / 8 , \left|\left\{\ell\in[B] : \left\|P^{[\ell]} - P^{[h_0]}\right\|_{\mathrm{F}} \le R/8\right\}\right| \ge7B/8, { ℓ ∈ [ B ] : P [ ℓ ] − P [ h 0 ] F ≤ R /8 } ≥ 7 B /8 ,
otherwise q < 7 B / 8 q < 7B/8 q < 7 B /8 . Define the set of block indices
I : = { ℓ ∈ [ B ] ∖ { i 0 } : ∥ P [ ℓ ] − P [ h 0 ] ∥ F ≤ R / 8 } . (27) \mathcal{I} := \left\{\ell\in[B]\setminus\{i_0\} : \left\|P^{[\ell]} - P^{[h_0]}\right\|_{\mathrm{F}} \le R/8\right\}. \tag*{(27)} I := { ℓ ∈ [ B ] ∖ { i 0 } : P [ ℓ ] − P [ h 0 ] F ≤ R /8 } . (27)
Then by the above and recall B ≥ 8 B \ge8 B ≥ 8 , we have
∣ I ∣ ≥ 7 B / 8 − 1 ≥ 3 B / 4. (28) |\mathcal{I}| \ge7B/8 - 1 \ge3B/4. \tag*{(28)} ∣ I ∣ ≥ 7 B /8 − 1 ≥ 3 B /4. (28)
Moreover, by construction, P [ ℓ ] ′ = P [ ℓ ] P^{[\ell]'} = P^{[\ell]} P [ ℓ ] ′ = P [ ℓ ] and ∥ P [ ℓ ] − P [ h 0 ] ∥ F ≤ R / 8 \left\|P^{[\ell]} - P^{[h_0]}\right\|_{\mathrm{F}} \le R/8 P [ ℓ ] − P [ h 0 ] F ≤ R /8 for every ℓ ∈ I \ell\in\mathcal{I} ℓ ∈ I , so that the two objectives in (26) contain at least 3 B / 4 3B/4 3 B /4 common terms with P [ ℓ ] P^{[\ell]} P [ ℓ ] , P [ ℓ ] ′ P^{[\ell]'} P [ ℓ ] ′ lie in the R / 8 R/8 R /8 -neighborhood of P [ h 0 ] P^{[h_0]} P [ h 0 ] .
We next show that the minimizers of both objectives are close to P [ h 0 ] P^{[h_0]} P [ h 0 ] . Consider first F F F in (26) . Let H H H be any symmetric matrix satisfying s : = ∥ H − P [ h 0 ] ∥ F ≥ R / 2 s := \left\|H - P^{[h_0]}\right\|_{\mathrm{F}} \ge R/2 s := H − P [ h 0 ] F ≥ R /2 , and set V : = ( H − P [ h 0 ] ) / s V := (H - P^{[h_0]})/s V := ( H − P [ h 0 ] ) / s . Then V = V ⊤ V = V^{\top} V = V ⊤ , ∥ V ∥ F = 1 \|V\|_{\mathrm{F}} = 1 ∥ V ∥ F = 1 , and H = P [ h 0 ] + s V H = P^{[h_0]} + sV H = P [ h 0 ] + s V . By the Huber loss definition in (3) , we have
∇ H ρ R ( ∥ H − P ∥ F ) = ( H − P ) min { 1 , R ∥ H − P ∥ F } (29) \nabla_H \rho_R(\|H-P\|_{\mathrm{F}}) = (H-P)\min\left\{1,\frac{R}{\|H-P\|_{\mathrm{F}}}\right\} \tag*{(29)} ∇ H ρ R ( ∥ H − P ∥ F ) = ( H − P ) min { 1 , ∥ H − P ∥ F R } (29)
where the gradient is defined to be zero when H = P H = P H = P . In particular, the gradient’s Frobenius norm is at most R R R . For every ℓ ∈ I \ell\in\mathcal{I} ℓ ∈ I , (27) gives
⟨ H − P [ ℓ ] , V ⟩ = ⟨ H − P [ ℓ ] , ( H − P [ h 0 ] ) / s ⟩ = ⟨ ( H − P [ h 0 ] ) + ( P [ h 0 ] − P [ ℓ ] ) , ( H − P [ h 0 ] ) / s ⟩ = s − 1 ∥ H − P [ h 0 ] ∥ F 2 + s − 1 ⟨ P [ h 0 ] − P [ ℓ ] , H − P [ h 0 ] ⟩ ≥ s − s − 1 ∥ P [ h 0 ] − P [ ℓ ] ∥ F ∥ H − P [ h 0 ] ∥ F ≥ s − R / 8 , \begin{aligned}
\langle H-P^{[\ell]},V\rangle
&=\langle H-P^{[\ell]},(H-P^{[h_0]})/s\rangle\\
&=\langle(H-P^{[h_0]})+(P^{[h_0]}-P^{[\ell]}),(H-P^{[h_0]})/s\rangle\\
&=s^{-1}\|H-P^{[h_0]}\|_{\mathrm{F}}^2+s^{-1}\langle P^{[h_0]}-P^{[\ell]},H-P^{[h_0]}\rangle\\
&\ge s-s^{-1}\|P^{[h_0]}-P^{[\ell]}\|_{\mathrm{F}}\|H-P^{[h_0]}\|_{\mathrm{F}}\ge s-R/8,
\end{aligned} ⟨ H − P [ ℓ ] , V ⟩ = ⟨ H − P [ ℓ ] , ( H − P [ h 0 ] ) / s ⟩ = ⟨( H − P [ h 0 ] ) + ( P [ h 0 ] − P [ ℓ ] ) , ( H − P [ h 0 ] ) / s ⟩ = s − 1 ∥ H − P [ h 0 ] ∥ F 2 + s − 1 ⟨ P [ h 0 ] − P [ ℓ ] , H − P [ h 0 ] ⟩ ≥ s − s − 1 ∥ P [ h 0 ] − P [ ℓ ] ∥ F ∥ H − P [ h 0 ] ∥ F ≥ s − R /8 ,
and similarly, ∥ H − P [ ℓ ] ∥ F ≤ s + R / 8 \|H-P^{[\ell]}\|_{\mathrm{F}}\le s+R/8 ∥ H − P [ ℓ ] ∥ F ≤ s + R /8 . Consequently, with (29) , we have
⟨ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) , V ⟩ ≥ ( s − R / 8 ) min { 1 , R s + R / 8 } ≥ 3 R / 8 , \left\langle\nabla_H\rho_R(\|H-P^{[\ell]}\|_{\mathrm{F}}),V\right\rangle\ge(s-R/8)\min\left\{1,\frac{R}{s+R/8}\right\}\ge3R/8, ⟨ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) , V ⟩ ≥ ( s − R /8 ) min { 1 , s + R /8 R } ≥ 3 R /8 ,
where the last inequality is direct to verify by considering s ≤ 7 R / 8 s\le7R/8 s ≤ 7 R /8 and s > 7 R / 8 s>7R/8 s > 7 R /8 , respectively. For every ℓ ∉ I \ell\notin\mathcal{I} ℓ ∈ / I , (29) and the Cauchy–Schwarz inequality give
⟨ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) , V ⟩ ≥ − ∥ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) ∥ F ∥ V ∥ F ≥ − R , \left\langle\nabla_H\rho_R(\|H-P^{[\ell]}\|_{\mathrm{F}}),V\right\rangle\ge-\|\nabla_H\rho_R(\|H-P^{[\ell]}\|_{\mathrm{F}})\|_{\mathrm{F}}\|V\|_{\mathrm{F}}\ge-R, ⟨ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) , V ⟩ ≥ − ∥ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) ∥ F ∥ V ∥ F ≥ − R ,
which, combined with (28) , we obtain
⟨ ∇ F ( H ) , V ⟩ = ∑ ℓ ∈ I ⟨ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) , V ⟩ + ∑ ℓ ∉ I ⟨ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) , V ⟩ ≥ ∣ I ∣ ⋅ 3 R 8 + ( B − ∣ I ∣ ) ⋅ ( − R ) ≥ 3 B 4 3 R 8 − B 4 R = B R 32 > 0. (30) \begin{aligned}
\langle\nabla F(H),V\rangle
&=\sum_{\ell\in\mathcal{I}}\left\langle\nabla_H\rho_R(\|H-P^{[\ell]}\|_{\mathrm{F}}),V\right\rangle
+\sum_{\ell\notin\mathcal{I}}\left\langle\nabla_H\rho_R(\|H-P^{[\ell]}\|_{\mathrm{F}}),V\right\rangle\\
&\ge|\mathcal{I}|\cdot\frac{3R}{8}+(B-|\mathcal{I}|)\cdot(-R)
\ge\frac{3B}{4}\frac{3R}{8}-\frac{B}{4}R
=\frac{BR}{32}>0. \tag*{(30)}
\end{aligned} ⟨ ∇ F ( H ) , V ⟩ = ℓ ∈ I ∑ ⟨ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) , V ⟩ + ℓ ∈ / I ∑ ⟨ ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) , V ⟩ ≥ ∣ I ∣ ⋅ 8 3 R + ( B − ∣ I ∣ ) ⋅ ( − R ) ≥ 4 3 B 8 3 R − 4 B R = 32 B R > 0. (30)
For every fixed symmetric matrix V V V with ∥ V ∥ F = 1 \|V\|_{\mathrm{F}}=1 ∥ V ∥ F = 1 , we have
d d s F ( P [ h 0 ] + s V ) = ⟨ ∇ F ( P [ h 0 ] + s V ) , V ⟩ . \frac{d}{ds}F(P^{[h_0]}+sV)=\left\langle\nabla F(P^{[h_0]}+sV),V\right\rangle. d s d F ( P [ h 0 ] + s V ) = ⟨ ∇ F ( P [ h 0 ] + s V ) , V ⟩ .
Hence (30) implies that s ↦ F ( P [ h 0 ] + s V ) s\mapsto F(P^{[h_0]}+sV) s ↦ F ( P [ h 0 ] + s V ) is strictly increasing for s ≥ R / 2 s\ge R/2 s ≥ R /2 , and no minimizer of F F F can lie outside or on the boundary (recall the Huber loss is a C 1 C^1 C 1 function and hence the function s ↦ F ( P [ h 0 ] + s V ) s\mapsto F(P^{[h_0]}+sV) s ↦ F ( P [ h 0 ] + s V ) is differentiable at R / 2 R/2 R /2 ) of the R / 2 R/2 R /2 -ball centered at P [ h 0 ] P^{[h_0]} P [ h 0 ] . The same argument applies to F ′ F' F ′ as its terms indexed by I \mathcal{I} I coincide with those of F F F , while every remaining Huber term still has gradient norm at most R R R . Altogether, we can conclude
∥ H ^ − P [ h 0 ] ∥ F < R / 2 , ∥ H ^ ′ − P [ h 0 ] ∥ F < R / 2. (31) \|\widehat{H}-P^{[h_0]}\|_{\mathrm{F}}<R/2,\qquad\|\widehat{H}'-P^{[h_0]}\|_{\mathrm{F}}<R/2. \tag*{(31)} ∥ H − P [ h 0 ] ∥ F < R /2 , ∥ H ′ − P [ h 0 ] ∥ F < R /2. (31)
It remains to compare the two minimizers. For t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] , let H t : = ( 1 − t ) H ^ + t H ^ ′ H_t:=(1-t)\widehat{H}+t\widehat{H}' H t := ( 1 − t ) H + t H ′ . By (27) and (31) , for every ℓ ∈ I \ell\in\mathcal{I} ℓ ∈ I ,
∥ H t − P [ ℓ ] ∥ F ≤ ∥ H t − P [ h 0 ] ∥ F + ∥ P [ h 0 ] − P [ ℓ ] ∥ F < R / 2 + R / 8 < R . \|H_t-P^{[\ell]}\|_{\mathrm{F}}\le\|H_t-P^{[h_0]}\|_{\mathrm{F}}+\|P^{[h_0]}-P^{[\ell]}\|_{\mathrm{F}}<R/2+R/8<R. ∥ H t − P [ ℓ ] ∥ F ≤ ∥ H t − P [ h 0 ] ∥ F + ∥ P [ h 0 ] − P [ ℓ ] ∥ F < R /2 + R /8 < R .
Then by the Huber loss function (3) , the terms indexed by I \mathcal{I} I (i.e., ρ R ( ∥ H ^ ′ − P [ ℓ ] ∥ F ) \rho_R(\|\widehat{H}'-P^{[\ell]}\|_{\mathrm{F}}) ρ R ( ∥ H ′ − P [ ℓ ] ∥ F ) and ρ R ( ∥ H ^ − P [ ℓ ] ∥ F ) \rho_R(\|\widehat{H}-P^{[\ell]}\|_{\mathrm{F}}) ρ R ( ∥ H − P [ ℓ ] ∥ F ) ) admit the quadratic loss function form throughout the segment joining H ^ \widehat{H} H and H ^ ′ \widehat{H}' H ′ . Each such term hence satisfies
∇ H ρ R ( ∥ H ^ ′ − P [ ℓ ] ∥ F ) − ∇ H ρ R ( ∥ H ^ − P [ ℓ ] ∥ F ) = H ^ ′ − H ^ . \nabla_H\rho_R(\|\widehat{H}'-P^{[\ell]}\|_{\mathrm{F}})-\nabla_H\rho_R(\|\widehat{H}-P^{[\ell]}\|_{\mathrm{F}})=\widehat{H}'-\widehat{H}. ∇ H ρ R ( ∥ H ′ − P [ ℓ ] ∥ F ) − ∇ H ρ R ( ∥ H − P [ ℓ ] ∥ F ) = H ′ − H .
For every ℓ ∉ I \ell\notin\mathcal{I} ℓ ∈ / I , convexity of the Huber loss function gives
⟨ ∇ H ρ R ( ∥ H ^ ′ − P [ ℓ ] ∥ F ) − ∇ H ρ R ( ∥ H ^ − P [ ℓ ] ∥ F ) , H ^ ′ − H ^ ⟩ ≥ 0. \left\langle\nabla H_{\rho R}\left(\lVert\widehat{H}' - P^{[\ell]}\rVert_{\mathrm{F}}\right) - \nabla H_{\rho R}\left(\lVert\widehat{H} - P^{[\ell]}\rVert_{\mathrm{F}}\right), \widehat{H}' - \widehat{H} \right\rangle\ge0. ⟨ ∇ H ρR ( ∥ H ′ − P [ ℓ ] ∥ F ) − ∇ H ρR ( ∥ H − P [ ℓ ] ∥ F ) , H ′ − H ⟩ ≥ 0.
Summing over all terms and using (28) , we obtain
3 B 4 ∥ H ^ ′ − H ^ ∥ F 2 ≤ ∣ I ∣ ⟨ H ^ ′ − H ^ , H ^ ′ − H ^ ⟩ ≤ ∑ ℓ ∈ I ⟨ ∇ H ρ R ( ∥ H ^ ′ − P [ ℓ ] ∥ F ) − ∇ H ρ R ( ∥ H ^ − P [ ℓ ] ∥ F ) , H ^ ′ − H ^ ⟩ (32) \begin{aligned}
\frac{3B}{4}\lVert\widehat{H}' - \widehat{H}\rVert_{\mathrm{F}}^2 \le|\mathcal{I}|\langle\widehat{H}' - \widehat{H}, \widehat{H}' - \widehat{H}\rangle\\
&\le\sum_{\ell\in\mathcal{I}} \left\langle\nabla H_{\rho R}\left(\lVert\widehat{H}' - P^{[\ell]}\rVert_{\mathrm{F}}\right) - \nabla H_{\rho R}\left(\lVert\widehat{H} - P^{[\ell]}\rVert_{\mathrm{F}}\right), \widehat{H}' - \widehat{H} \right\rangle \tag*{(32)}
\end{aligned} 4 3 B ∥ H ′ − H ∥ F 2 ≤ ∣ I ∣ ⟨ H ′ − H , H ′ − H ⟩ ≤ ℓ ∈ I ∑ ⟨ ∇ H ρR ( ∥ H ′ − P [ ℓ ] ∥ F ) − ∇ H ρR ( ∥ H − P [ ℓ ] ∥ F ) , H ′ − H ⟩ (32)
≤ ⟨ ∇ F ( H ^ ′ ) − ∇ F ( H ^ ) , H ^ ′ − H ^ ⟩ . (33) \le\left\langle\nabla F(\widehat{H}') - \nabla F(\widehat{H}), \widehat{H}' - \widehat{H} \right\rangle. \tag*{(33)} ≤ ⟨ ∇ F ( H ′ ) − ∇ F ( H ) , H ′ − H ⟩ . (33)
Since two objectives F F F and F ′ F' F ′ are differentiable and are minimized over the linear space of symmetric matrices, their first-order optimality conditions are
∇ F ( H ^ ) = 0 , ∇ F ′ ( H ^ ′ ) = 0. (34) \nabla F(\widehat{H}) = 0,\qquad\nabla F'(\widehat{H}') = 0. \tag*{(34)} ∇ F ( H ) = 0 , ∇ F ′ ( H ′ ) = 0. (34)
By (26) , F F F and F ′ F' F ′ differ only in the term indexed by i 0 i_0 i 0 . Thus, (34) imply
∥ ∇ F ( H ^ ′ ) ∥ F = ∥ ∇ F ( H ^ ′ ) − ∇ F ′ ( H ^ ′ ) ∥ F = ∥ ∇ H ρ R ( ∥ H ^ ′ − P [ i 0 ] ∥ F ) − ∇ H ρ R ( ∥ H ^ ′ − P [ i 0 ] ′ ∥ F ) ∥ F ≤ 2 R , (35) \begin{aligned}
\lVert\nabla F(\widehat{H}')\rVert_{\mathrm{F}} = \lVert\nabla F(\widehat{H}') - \nabla F'(\widehat{H}')\rVert_{\mathrm{F}} \\
&= \left\lVert\nabla H_{\rho R}\left(\lVert\widehat{H}' - P^{[i_0]}\rVert_{\mathrm{F}}\right) - \nabla H_{\rho R}\left(\lVert\widehat{H}' - P^{[i_0]'}\rVert_{\mathrm{F}}\right) \right\rVert_{\mathrm{F}} \le2R, \tag*{(35)}
\end{aligned} ∥ ∇ F ( H ′ ) ∥ F = ∥ ∇ F ( H ′ ) − ∇ F ′ ( H ′ ) ∥ F = ∇ H ρR ( ∥ H ′ − P [ i 0 ] ∥ F ) − ∇ H ρR ( ∥ H ′ − P [ i 0 ] ′ ∥ F ) F ≤ 2 R , (35)
where the last inequality used the gradient upper bounded by R R R as mentioned below (29) . Combining (33) , (34) , (35) , and the Cauchy–Schwarz inequality, we obtain
3 B 4 ∥ H ^ ′ − H ^ ∥ F 2 ≤ ∥ ∇ F ( H ^ ′ ) − 0 ∥ F ⋅ ∥ H ^ ′ − H ^ ∥ F ≤ 2 R ∥ H ^ ′ − H ^ ∥ F . \frac{3B}{4}\lVert\widehat{H}' - \widehat{H}\rVert_{\mathrm{F}}^2 \le\lVert\nabla F(\widehat{H}') - 0\rVert_{\mathrm{F}} \cdot\lVert\widehat{H}' - \widehat{H}\rVert_{\mathrm{F}} \le2R\lVert\widehat{H}' - \widehat{H}\rVert_{\mathrm{F}}. 4 3 B ∥ H ′ − H ∥ F 2 ≤ ∥ ∇ F ( H ′ ) − 0 ∥ F ⋅ ∥ H ′ − H ∥ F ≤ 2 R ∥ H ′ − H ∥ F .
If H ^ ′ = H ^ \widehat{H}' = \widehat{H} H ′ = H , (6) is immediate. Otherwise, dividing both sides by ∥ H ^ ′ − H ^ ∥ F \lVert\widehat{H}' - \widehat{H}\rVert_{\mathrm{F}} ∥ H ′ − H ∥ F yields
∥ H ^ ′ − H ^ ∥ F ≤ 8 R 3 B ≤ 4 R B , \lVert\widehat{H}' - \widehat{H}\rVert_{\mathrm{F}} \le\frac{8R}{3B} \le\frac{4R}{B}, ∥ H ′ − H ∥ F ≤ 3 B 8 R ≤ B 4 R ,
which proves (6) . This completes the proof of the lemma. □
Proof of Proposition 1. For notational convenience, we suppress the client and mode indices (i.e., j j j and k k k in Algorithm 1) below. Fix a client and condition on a realization of its data-independent random partition. Fix any pair of neighboring datasets and denote their block-projector lists, consistency scores, and Huber aggregates by
{ P [ ℓ ] } ℓ = 1 B , q , H ^ , and { P [ ℓ ] ′ } ℓ = 1 B , q ′ , H ^ ′ , \{P^{[\ell]}\}_{\ell=1}^{B},\quad q,\quad\widehat{H},\quad\text{and}\quad\{P^{[\ell]'}\}_{\ell=1}^{B},\quad q',\quad\widehat{H}', { P [ ℓ ] } ℓ = 1 B , q , H , and { P [ ℓ ] ′ } ℓ = 1 B , q ′ , H ′ ,
respectively. A replacement of one tensor observation changes at most one block projector, so Lemma 1 applies.
For a dataset with consistency score q q q (i.e., as q j , k q_{j,k} q j , k computed in the procedure in Algorithm 1), define the outcome
T : = 1 { q + W ≥ 7 B 8 + 2 + 4 ε log ( 2 δ ) } , W ∼ Lap ( 4 / ε ) . (36) T := \mathbb{1}\left\{q + W \ge\frac{7B}{8} + 2 + \frac{4}{\varepsilon}\log\left(\frac{2}{\delta}\right)\right\},\qquad W \sim\operatorname{Lap}(4/\varepsilon). \tag*{(36)} T := 1 { q + W ≥ 8 7 B + 2 + ε 4 log ( δ 2 ) } , W ∼ Lap ( 4/ ε ) . (36)
The corresponding outcome for the neighboring dataset is defined analogously using q ′ q' q ′ . The Gaussian and Haar randomness used after the test are independent of W W W . By (5) , q q q has sensitivity at most two. The Laplace mechanism [24 Theorem 3.6] therefore shows that q + W q+W q + W , and hence its post-processing T T T , is ( ε / 2 , 0 ) (\varepsilon/2,0) ( ε /2 , 0 ) -CDP. To explicitly account for the branch selected by the stability test T T T in (36) , we prove privacy for the augmented local output ( T , U ~ ) (T,\widetilde{U}) ( T , U ) . The actual local estimator U ~ \widetilde{U} U is a post-processing of this augmented output.
We first consider the calibration of the data under the Gaussian mechanism, i.e., when T = 1 T=1 T = 1 . For each pair ( a , b ) ∈ [ p k ] × [ p k ] (a,b)\in[p_k]\times[p_k] ( a , b ) ∈ [ p k ] × [ p k ] , let E a b E_{ab} E ab denote the p k × p k p_k\times p_k p k × p k matrix with a one in position ( a , b ) (a,b) ( a , b ) and zeros elsewhere. With respect to the Frobenius inner product, an orthonormal basis of the symmetric matrices is
E a a , E a b + E b a 2 , 1 ≤ a < b ≤ p k . E_{aa},\qquad\frac{E_{ab}+E_{ba}}{\sqrt{2}},\qquad1\leq a<b\leq p_k. E aa , 2 E ab + E ba , 1 ≤ a < b ≤ p k .
It is direct to check that under this basis, the coordinates of Z = τ ( G + G T ) / 2 Z=\tau(G+G^\mathsf{T})/\sqrt{2} Z = τ ( G + G T ) / 2 are mutually independent N ( 0 , 2 τ 2 ) \mathcal{N}(0,2\tau^2) N ( 0 , 2 τ 2 ) random variables. This follows from that the diagonal entry Z a a = 2 τ G a a Z_{aa}=\sqrt{2}\tau G_{aa} Z aa = 2 τ G aa has variance 2 τ 2 2\tau^2 2 τ 2 , while each off-diagonal entry Z a b Z_{ab} Z ab with a ≠ b a\ne b a = b has variance τ 2 \tau^2 τ 2 ; the basis element ( E a b + E b a ) / 2 (E_{ab}+E_{ba})/\sqrt{2} ( E ab + E ba ) / 2 combines the symmetric pair ( Z a b , Z b a ) (Z_{ab},Z_{ba}) ( Z ab , Z ba ) with the factor 1 / 2 1/\sqrt{2} 1/ 2 , giving the same variance 2 τ 2 2\tau^2 2 τ 2 in the orthonormal basis. By the definition of τ \tau τ in Algorithm 1,
2 τ 2 = 2 ( 4 R / B ) 2 ( ε / 2 ) 2 log ( 2.5 δ ) . (37) 2\tau^2=\frac{2(4R/B)^2}{(\varepsilon/2)^2}\log\left(\frac{2.5}{\delta}\right). \tag*{(37)} 2 τ 2 = ( ε /2 ) 2 2 ( 4 R / B ) 2 log ( δ 2.5 ) . (37)
Consequently, whenever two Huber aggregates are at Frobenius distance at most 4 R / B 4R/B 4 R / B , the corresponding sensitivity is 4 R / B 4R/B 4 R / B , the Gaussian mechanism [24 Theorem A.1] applied to them is ( ε / 2 , δ / 2 ) (\varepsilon/2,\delta/2) ( ε /2 , δ /2 ) -CDP, and applying singular value decomposition afterward preserves this guarantee.
We now consider two exhaustive cases for the fixed neighboring pair. First, suppose q ≥ 7 B / 8 q\geq7B/8 q ≥ 7 B /8 and q ′ ≥ 7 B / 8 q'\geq7B/8 q ′ ≥ 7 B /8 . Then by (6) in Lemma 1,
∥ H ^ − H ^ ′ ∥ F ≤ 4 R / B . \left\|\widehat{H}-\widehat{H}'\right\|_{\mathrm{F}}\leq4R/B. H − H ′ F ≤ 4 R / B .
Conditional on T = 0 T=0 T = 0 , the output of Algorithm 1 is drawn from the same data-independent Haar distribution under both datasets and is therefore ( 0 , 0 ) (0,0) ( 0 , 0 ) -CDP. Conditional on T = 1 T=1 T = 1 , the output of the local estimation in Algorithm 1 is SVD r k ( H ^ + Z ) \operatorname{SVD}_{r_k}(\widehat{H}+Z) SVD r k ( H + Z ) , which, by (37) , is ( ε / 2 , δ / 2 ) (\varepsilon/2,\delta/2) ( ε /2 , δ /2 ) -CDP for this neighboring pair. Hence for either possible value of T T T , Step 2(b) is ( ε / 2 , δ / 2 ) (\varepsilon/2,\delta/2) ( ε /2 , δ /2 ) -CDP. Together with the ( ε / 2 , 0 ) (\varepsilon/2,0) ( ε /2 , 0 ) -CDP score q q q , sequential composition [24 Theorem 3.16] implies that the augmented output ( T , U ~ ) (T,\widetilde{U}) ( T , U ) is ( ε , δ / 2 ) (\varepsilon,\delta/2) ( ε , δ /2 ) -CDP.
Second, suppose that at least one of the scores q q q and q ′ q' q ′ is smaller than 7 B / 8 7B/8 7 B /8 . By (5) , both q q q and q ′ q' q ′ are smaller than 7 B / 8 + 2 7B/8+2 7 B /8 + 2 . It follows from (36) that, under either dataset, passing the test with T = 1 T=1 T = 1 at least requires
W ≥ 4 ε log ( 2 δ ) . W\geq\frac{4}{\varepsilon}\log\left(\frac{2}{\delta}\right). W ≥ ε 4 log ( δ 2 ) .
Since W ∼ Lap ( 4 / ε ) W\sim\operatorname{Lap}(4/\varepsilon) W ∼ Lap ( 4/ ε ) , we have
P ( T = 1 ) ≤ P ( W ≥ 4 ε log ( 2 δ ) ) = 1 2 exp { − ( 4 / ε ) log ( 2 / δ ) 4 / ε } = δ 4 . (38) \mathbb{P}(T=1)\leq\mathbb{P}\left(W\geq\frac{4}{\varepsilon}\log\left(\frac{2}{\delta}\right)\right)=\frac{1}{2}\exp\left\{-\frac{(4/\varepsilon)\log(2/\delta)}{4/\varepsilon}\right\}=\frac{\delta}{4}. \tag*{(38)} P ( T = 1 ) ≤ P ( W ≥ ε 4 log ( δ 2 ) ) = 2 1 exp { − 4/ ε ( 4/ ε ) log ( 2/ δ ) } = 4 δ . (38)
Let a a a and a ′ a' a ′ be the probabilities of passing the test with T = 1 T = 1 T = 1 under the two neighboring datasets. Then (38) gives a , a ′ ≤ δ / 4 a,a' \le\delta/4 a , a ′ ≤ δ /4 . Let Q Q Q be the common distribution of the augmented output ( T , U ~ ) (T,\widetilde{U}) ( T , U ) when the test fails, so that by construction, under Q Q Q , it holds that T = 0 T = 0 T = 0 and U ~ \widetilde{U} U has a Haar distribution which is data-independent. Let Q 1 Q_1 Q 1 and Q 1 ′ Q'_1 Q 1 ′ be the conditional distributions of the augmented output when the test passes under the two datasets, and these two conditional distributions may be arbitrary. Then note that the two unconditional output distributions, denoted by μ \mu μ and μ ′ \mu' μ ′ , satisfy
μ = ( 1 − a ) Q + a Q 1 , μ ′ = ( 1 − a ′ ) Q + a ′ Q 1 ′ . (39) \mu= (1-a)Q + aQ_1,\qquad\mu' = (1-a')Q + a'Q'_1. \tag*{(39)} μ = ( 1 − a ) Q + a Q 1 , μ ′ = ( 1 − a ′ ) Q + a ′ Q 1 ′ . (39)
Using (39) and the fact that the total variation distance between two probability measures is at most one, we have
d T V ( μ , μ ′ ) = sup A ∣ μ ( A ) − μ ′ ( A ) ∣ = sup A ∣ a [ Q 1 ( A ) − Q ( A ) ] + a ′ [ Q ( A ) − Q 1 ′ ( A ) ] ∣ ≤ a ⋅ d T V ( Q 1 , Q ) + a ′ ⋅ d T V ( Q 1 ′ , Q ) ≤ a + a ′ ≤ δ / 2. \begin{aligned}
d_{\mathrm{TV}}(\mu,\mu') &= \sup_A \left|\mu(A)-\mu'(A)\right| \\
&= \sup_A \left|a[Q_1(A)-Q(A)] + a'[Q(A)-Q'_1(A)]\right| \\
&\le a \cdot d_{\mathrm{TV}}(Q_1,Q) + a' \cdot d_{\mathrm{TV}}(Q'_1,Q) \le a+a' \le\delta/2.
\end{aligned} d TV ( μ , μ ′ ) = A sup ∣ μ ( A ) − μ ′ ( A ) ∣ = A sup ∣ a [ Q 1 ( A ) − Q ( A )] + a ′ [ Q ( A ) − Q 1 ′ ( A )] ∣ ≤ a ⋅ d TV ( Q 1 , Q ) + a ′ ⋅ d TV ( Q 1 ′ , Q ) ≤ a + a ′ ≤ δ /2.
Hence, for every measurable set A A A in the augmented output space,
μ ( A ) ≤ μ ′ ( A ) + δ / 2 = e 0 μ ′ ( A ) + δ / 2. \mu(A) \le\mu'(A) + \delta/2 = e^0\mu'(A) + \delta/2. μ ( A ) ≤ μ ′ ( A ) + δ /2 = e 0 μ ′ ( A ) + δ /2.
which implies that the augmented output ( T , U ~ ) (T,\widetilde{U}) ( T , U ) is ( 0 , δ / 2 ) (0,\delta/2) ( 0 , δ /2 ) -CDP for this neighboring pair. This case covers all configurations in which at least one of q q q and q ′ q' q ′ is below 7 B / 8 7B/8 7 B /8 , including the case where both are below and the mixed case where exactly one is below. In all such configurations, the sensitivity bound ensures that both scores are below 7 B / 8 + 2 7B/8+2 7 B /8 + 2 , so the pass-probability bound a , a ′ ≤ δ / 4 a,a' \le\delta/4 a , a ′ ≤ δ /4 remains valid.
Finally, the above discussion covers every fixed neighboring pair. In the first case, where both q ≥ 7 B / 8 q \ge7B/8 q ≥ 7 B /8 and q ′ ≥ 7 B / 8 q' \ge7B/8 q ′ ≥ 7 B /8 , the augmented local output ( T , U ~ ) (T,\widetilde{U}) ( T , U ) is ( ε , δ / 2 ) (\varepsilon,\delta/2) ( ε , δ /2 ) -CDP. In the second case, where at least one of q q q and q ′ q' q ′ is below 7 B / 8 7B/8 7 B /8 , it is ( 0 , δ / 2 ) (0,\delta/2) ( 0 , δ /2 ) -CDP, which is stronger than ( ε , δ / 2 ) (\varepsilon,\delta/2) ( ε , δ /2 ) -CDP since e 0 = 1 ≤ e ε e^0=1\le e^\varepsilon e 0 = 1 ≤ e ε . Hence in either case the augmented local output satisfies ( ε , δ / 2 ) (\varepsilon,\delta/2) ( ε , δ /2 ) -CDP. This conclusion holds for every fixed realization of the data-independent partition with the same privacy parameters, and averaging over the random partition preserves DP. Reintroducing the index j j j , the local release of client j j j is ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -CDP with respect to D j D_j D j . For fixed data from all other clients, their messages and every server-side operation are post-processing relative to D j D_j D j . Thus the final federated estimator satisfies ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -FDP. □
Auxiliary lemmas with proofs
We first establish three auxiliary results: (i) the first shows that random partitioning preserves the factor strength in every block, with high probability; (ii) the second controls the blockwise empirical projectors; and (iii) the third gives the local stochastic bound used in the federated aggregation.
Lemma 2 (Factor balance under random partitioning). Let Assumption 2 holds. Fix j ∈ [ m ] j \in[m] j ∈ [ m ] and k ∈ [ K ] k \in[K] k ∈ [ K ] . Let Λ j , ( k ) \Lambda_{j,(k)} Λ j , ( k ) be the full-sample mode-k k k factor covariance defined in (8) , let b j b_j b j and B j B_j B j be set as in Theorem 1, and let I j , 1 , … , I j , B j \mathcal{I}_{j,1},\ldots,\mathcal{I}_{j,B_j} I j , 1 , … , I j , B j be the random blocks constructed in Algorithm 1. Then with probability at least
1 − 2 r k B j exp { − c b j C χ 2 } . 1-2r_kB_j\exp\left\{-\frac{cb_j}{C_\chi^2}\right\}. 1 − 2 r k B j exp { − C χ 2 c b j } .
it holds that
1 2 Λ j , ( k ) ⪯ 1 ∣ I j , ℓ ∣ ∑ t ∈ I j , ℓ M k ( F t ( j ) ) M k ( F t ( j ) ) T ⪯ 3 2 Λ j , ( k ) , ∀ ℓ ∈ [ B j ] , (40) \frac{1}{2}\Lambda_{j,(k)} \preceq\frac{1}{|\mathcal{I}_{j,\ell}|}\sum_{t\in\mathcal{I}_{j,\ell}}\mathcal{M}_k(\mathbf{F}^{(j)}_t)\mathcal{M}_k(\mathbf{F}^{(j)}_t)^{\mathsf{T}} \preceq\frac{3}{2}\Lambda_{j,(k)}, \quad\forall\ell\in[B_j], \tag*{(40)} 2 1 Λ j , ( k ) ⪯ ∣ I j , ℓ ∣ 1 t ∈ I j , ℓ ∑ M k ( F t ( j ) ) M k ( F t ( j ) ) T ⪯ 2 3 Λ j , ( k ) , ∀ ℓ ∈ [ B j ] , (40)
where A ⪯ B A\preceq B A ⪯ B for matrices A , B A,B A , B means B − A B-A B − A is positive semi-definite; the probability is taken over the random partition; C λ > 0 C_\lambda>0 C λ > 0 is the absolute constant in Assumption 2; and c > 0 c>0 c > 0 is an absolute constant.
Proof of Lemma 2. By Assumption 2, we have λ j , r k ( k ) > 0 \lambda_{j,r_k}^{(k)}>0 λ j , r k ( k ) > 0 and hence Λ j , ( k ) \Lambda_{j,(k)} Λ j , ( k ) is positive definite. For each t ∈ [ n j ] t\in[n_j] t ∈ [ n j ] , define the normalized mode-k k k factor matrix as
Y j , k , t : = Λ j , ( k ) − 1 / 2 M k ( F t ( j ) ) M k ( F t ( j ) ) T Λ j , ( k ) − 1 / 2 . (41) Y_{j,k,t}:=\Lambda_{j,(k)}^{-1/2}\mathcal{M}_k(\mathbf{F}^{(j)}_t)\mathcal{M}_k(\mathbf{F}^{(j)}_t)^{\mathsf{T}}\Lambda_{j,(k)}^{-1/2}. \tag*{(41)} Y j , k , t := Λ j , ( k ) − 1/2 M k ( F t ( j ) ) M k ( F t ( j ) ) T Λ j , ( k ) − 1/2 . (41)
By the definition of Λ j , ( k ) \Lambda_{j,(k)} Λ j , ( k ) in (8) ,
1 n j ∑ t = 1 n j Y j , k , t = I r k . (42) \frac{1}{n_j}\sum_{t=1}^{n_j}Y_{j,k,t}=I_{r_k}. \tag*{(42)} n j 1 t = 1 ∑ n j Y j , k , t = I r k . (42)
Moreover, the definition of λ j , max ( k ) \lambda_{j,\max}^{(k)} λ j , m a x ( k ) in (9) gives
0 ⪯ M k ( F t ( j ) ) M k ( F t ( j ) ) T ⪯ λ j , max ( k ) I r k . 0\preceq\mathcal{M}_k(\mathbf{F}^{(j)}_t)\mathcal{M}_k(\mathbf{F}^{(j)}_t)^{\mathsf{T}}\preceq\lambda_{j,\max}^{(k)}I_{r_k}. 0 ⪯ M k ( F t ( j ) ) M k ( F t ( j ) ) T ⪯ λ j , m a x ( k ) I r k .
Hence by Assumption 2, we have
0 ⪯ Y j , k , t ⪯ λ j , max ( k ) Λ j , ( k ) − 1 ⪯ λ j , max ( k ) λ j , r k ( k ) I r k ⪯ C λ 2 I r k . (43) 0\preceq Y_{j,k,t}\preceq\lambda_{j,\max}^{(k)}\Lambda_{j,(k)}^{-1}\preceq\frac{\lambda_{j,\max}^{(k)}}{\lambda_{j,r_k}^{(k)}}I_{r_k}\preceq C_\lambda^2I_{r_k}. \tag*{(43)} 0 ⪯ Y j , k , t ⪯ λ j , m a x ( k ) Λ j , ( k ) − 1 ⪯ λ j , r k ( k ) λ j , m a x ( k ) I r k ⪯ C λ 2 I r k . (43)
Fix ℓ ∈ [ B j ] \ell\in[B_j] ℓ ∈ [ B j ] . Then conditional on the deterministic factors, the matrices Y j , k , 1 , … , Y j , k , n j Y_{j,k,1},\ldots,Y_{j,k,n_j} Y j , k , 1 , … , Y j , k , n j form a fixed finite population, while the block I j , ℓ \mathcal{I}_{j,\ell} I j , ℓ is random. For every t ∈ [ n j ] t\in[n_j] t ∈ [ n j ] , define the inclusion indicator Z j , ℓ , t p a r t : = 1 { t ∈ I j , ℓ } Z_{j,\ell,t}^{\mathrm{part}}:=\mathbb{1}\{t\in\mathcal{I}_{j,\ell}\} Z j , ℓ , t part := 1 { t ∈ I j , ℓ } . By construction via random permutation, I j , ℓ \mathcal{I}_{j,\ell} I j , ℓ is a uniformly sampled subset of [ n j ] [n_j] [ n j ] of size ∣ I j , ℓ ∣ |\mathcal{I}_{j,\ell}| ∣ I j , ℓ ∣ . Hence we have
P π j { Z j , ℓ , t p a r t = 1 } = ( n j − 1 ∣ I j , ℓ ∣ − 1 ) ( n j ∣ I j , ℓ ∣ ) = ∣ I j , ℓ ∣ n j , (44) \mathbb{P}_{\pi_j}\{Z_{j,\ell,t}^{\mathrm{part}}=1\}=\frac{\binom{n_j-1}{|\mathcal{I}_{j,\ell}|-1}}{\binom{n_j}{|\mathcal{I}_{j,\ell}|}}=\frac{|\mathcal{I}_{j,\ell}|}{n_j}, \tag*{(44)} P π j { Z j , ℓ , t part = 1 } = ( ∣ I j , ℓ ∣ n j ) ( ∣ I j , ℓ ∣ − 1 n j − 1 ) = n j ∣ I j , ℓ ∣ , (44)
where P π j \mathbb{P}_{\pi_j} P π j denotes probability over the random permutation. Since the matrices Y j , k , t Y_{j,k,t} Y j , k , t are fixed conditional on the factors, (42) and (44) give
E π j { 1 ∣ I j , ℓ ∣ ∑ t ∈ I j , ℓ Y j , k , t } = 1 ∣ I j , ℓ ∣ ∑ t = 1 n j E π j ( Z j , ℓ , t p a r t ) Y j , k , t = 1 n j ∑ t = 1 n j Y j , k , t = I r k , (45) \mathbb{E}_{\pi_j}\left\{\frac{1}{|\mathcal{I}_{j,\ell}|}\sum_{t\in\mathcal{I}_{j,\ell}}Y_{j,k,t}\right\}
=\frac{1}{|\mathcal{I}_{j,\ell}|}\sum_{t=1}^{n_j}\mathbb{E}_{\pi_j}(Z_{j,\ell,t}^{\mathrm{part}})Y_{j,k,t}
=\frac{1}{n_j}\sum_{t=1}^{n_j}Y_{j,k,t}=I_{r_k}, \tag*{(45)} E π j ⎩ ⎨ ⎧ ∣ I j , ℓ ∣ 1 t ∈ I j , ℓ ∑ Y j , k , t ⎭ ⎬ ⎫ = ∣ I j , ℓ ∣ 1 t = 1 ∑ n j E π j ( Z j , ℓ , t part ) Y j , k , t = n j 1 t = 1 ∑ n j Y j , k , t = I r k , (45)
where E π j \mathbb{E}_{\pi_j} E π j denotes expectation over the random permutation. Note that the indicators Z j , ℓ , t p a r t Z_{j,\ell,t}^{\mathrm{part}} Z j , ℓ , t part are dependent with each other, since the sampling is without replacement. We address this below. Conditional on the factors, the matrices indexed by I j , ℓ \mathcal{I}_{j,\ell} I j , ℓ constitute a uniform sample without replacement from a finite population whose mean is I r k I_{r_k} I r k and whose elements satisfy (43) . To justify the use of the matrix Chernoff inequality (see e.g. Chapter 5.1 of [45 ] , set s = ∣ I j , ℓ ∣ s = |\mathcal{I}_{j,\ell}| s = ∣ I j , ℓ ∣ and let ξ 1 , … , ξ s ∈ [ n j ] \xi_1,\ldots,\xi_s \in[n_j] ξ 1 , … , ξ s ∈ [ n j ] be independent and uniformly distributed on [ n j ] [n_j] [ n j ] . Then Y j , k , ξ 1 , … , Y j , k , ξ s Y_{j,k,\xi_1},\ldots,Y_{j,k,\xi_s} Y j , k , ξ 1 , … , Y j , k , ξ s form the corresponding samples with replacement. Recall (42) and (43) which remain to hold, such that
E ( Y j , k , ξ a ) = I r k , 0 ⪯ Y j , k , ξ a ⪯ C λ 2 I r k . \mathbb{E}(Y_{j,k,\xi_a}) = I_{r_k}, \qquad0 \preceq Y_{j,k,\xi_a} \preceq C_\lambda^2 I_{r_k}. E ( Y j , k , ξ a ) = I r k , 0 ⪯ Y j , k , ξ a ⪯ C λ 2 I r k .
Applying Lemma 5.4.1 in Tropp [45 ] gives, for every θ ∈ R \theta\in\mathbb{R} θ ∈ R ,
E [ tr { exp ( θ ∑ a = 1 s Y j , k , ξ a ) } ] ≤ r k exp { s C λ 2 ( e θ C λ 2 − 1 ) } . (46) \mathbb{E}\left[\operatorname{tr}\left\{\exp\left(\theta\sum_{a=1}^{s}Y_{j,k,\xi_a}\right)\right\}\right]\leq r_k\exp\left\{\frac{s}{C_\lambda^2}\left(e^{\theta C_\lambda^2}-1\right)\right\}. \tag*{(46)} E [ tr { exp ( θ a = 1 ∑ s Y j , k , ξ a ) } ] ≤ r k exp { C λ 2 s ( e θ C λ 2 − 1 ) } . (46)
By the arguments at the end of Gross and Nesme [28 ] , we have
E Π j [ tr { exp ( θ ∑ t ∈ I j , ℓ Y j , k , t ) } ] ≤ E [ tr { exp ( θ ∑ a = 1 s Y j , k , ξ a ) } ] , θ ∈ R . (47) \mathbb{E}_{\Pi_j}\left[\operatorname{tr}\left\{\exp\left(\theta\sum_{t\in\mathcal{I}_{j,\ell}}Y_{j,k,t}\right)\right\}\right]\leq\mathbb{E}\left[\operatorname{tr}\left\{\exp\left(\theta\sum_{a=1}^{s}Y_{j,k,\xi_a}\right)\right\}\right],\qquad\theta\in\mathbb{R}. \tag*{(47)} E Π j tr ⎩ ⎨ ⎧ exp θ t ∈ I j , ℓ ∑ Y j , k , t ⎭ ⎬ ⎫ ≤ E [ tr { exp ( θ a = 1 ∑ s Y j , k , ξ a ) } ] , θ ∈ R . (47)
so that it suffices to consider the bounds under the samples with replacement, which we constructed earlier. Consequently, although the matrices selected without replacement are dependent, (46) and (47) give their sum the same trace moment-generating-function upper bound as the independent sum used by Lemma 5.4.1 in Tropp [45 ] .
Then for every u ≥ 0 u\geq0 u ≥ 0 , Proposition 3.2.1 of Tropp [45 ] gives
P Π j { λ max ( ∑ t ∈ I j , ℓ Y j , k , t ) ≥ ( 1 + u ) s } ≤ inf θ > 0 e − θ ( 1 + u ) s E Π j [ tr { exp ( θ ∑ t ∈ I j , ℓ Y j , k , t ) } ] ≤ r k { e u ( 1 + u ) 1 + u } s / C λ 2 . (48) \mathbb{P}_{\Pi_j}\left\{\lambda_{\max}\left(\sum_{t\in\mathcal{I}_{j,\ell}}Y_{j,k,t}\right)\geq(1+u)s\right\}
\leq\inf_{\theta>0}e^{-\theta(1+u)s}\mathbb{E}_{\Pi_j}\left[\operatorname{tr}\left\{\exp\left(\theta\sum_{t\in\mathcal{I}_{j,\ell}}Y_{j,k,t}\right)\right\}\right]
\leq r_k\left\{\frac{e^u}{(1+u)^{1+u}}\right\}^{s/C_\lambda^2}. \tag*{(48)} P Π j ⎩ ⎨ ⎧ λ m a x t ∈ I j , ℓ ∑ Y j , k , t ≥ ( 1 + u ) s ⎭ ⎬ ⎫ ≤ θ > 0 inf e − θ ( 1 + u ) s E Π j tr ⎩ ⎨ ⎧ exp θ t ∈ I j , ℓ ∑ Y j , k , t ⎭ ⎬ ⎫ ≤ r k { ( 1 + u ) 1 + u e u } s / C λ 2 . (48)
where the last inequality used (46) and (47) by taking θ = log ( 1 + u ) / C λ 2 \theta=\log(1+u)/C_\lambda^2 θ = log ( 1 + u ) / C λ 2 . Similarly, for every u ∈ [ 0 , 1 ) u\in[0,1) u ∈ [ 0 , 1 ) , applying the same argument with − θ -\theta − θ and taking θ = − log ( 1 − u ) / C λ 2 \theta=-\log(1-u)/C_\lambda^2 θ = − log ( 1 − u ) / C λ 2 gives
P Π j { λ min ( ∑ t ∈ I j , ℓ Y j , k , t ) ≤ ( 1 − u ) s } ≤ r k { e − u ( 1 − u ) 1 − u } s / C λ 2 . (49) \mathbb{P}_{\Pi_j}\left\{\lambda_{\min}\left(\sum_{t\in\mathcal{I}_{j,\ell}}Y_{j,k,t}\right)\leq(1-u)s\right\}\leq r_k\left\{\frac{e^{-u}}{(1-u)^{1-u}}\right\}^{s/C_\lambda^2}. \tag*{(49)} P Π j ⎩ ⎨ ⎧ λ m i n t ∈ I j , ℓ ∑ Y j , k , t ≤ ( 1 − u ) s ⎭ ⎬ ⎫ ≤ r k { ( 1 − u ) 1 − u e − u } s / C λ 2 . (49)
Thus Lemma 5.4.1 in Tropp [45 ] applies under the above events. Taking u = 1 / 2 u=1/2 u = 1/2 in (48) and (49) and combining the two tails gives
P Π j { ∥ 1 ∣ I j , ℓ ∣ ∑ t ∈ I j , ℓ Y j , k , t − I r k ∥ > 1 2 } ≤ 2 r k exp { − c s C λ 2 } ≤ 2 r k exp { − c b j C λ 2 } . (50) \mathbb{P}_{\Pi_j}\left\{\left\|\frac{1}{|\mathcal{I}_{j,\ell}|}\sum_{t\in\mathcal{I}_{j,\ell}}Y_{j,k,t}-I_{r_k}\right\|>\frac{1}{2}\right\}\leq2r_k\exp\left\{-\frac{cs}{C_\lambda^2}\right\}\leq2r_k\exp\left\{-\frac{cb_j}{C_\lambda^2}\right\}. \tag*{(50)} P Π j ⎩ ⎨ ⎧ ∣ I j , ℓ ∣ 1 t ∈ I j , ℓ ∑ Y j , k , t − I r k > 2 1 ⎭ ⎬ ⎫ ≤ 2 r k exp { − C λ 2 cs } ≤ 2 r k exp { − C λ 2 c b j } . (50)
where c > 0 c>0 c > 0 is an absolute constant and the last inequality follows from setting s = ∣ I j , ℓ ∣ ≥ b j s=|\mathcal{I}_{j,\ell}|\geq b_j s = ∣ I j , ℓ ∣ ≥ b j . This completes the proof of this lemma.
::: Lemma 3 (Uniform block-projector concentration). *Suppose that Assumptions 1–3 and the conditions of Theorem 1 hold. Condition on any realization of the random partitions for which (40) holds for every client and block. Let R j , k R_{j,k} R j , k be defined in (10) , and suppose R j , k ≤ c r a d R_{j,k}\leq c_{\mathrm{rad}} R j , k ≤ c rad for every j ∈ [ m ] j\in[m] j ∈ [ m ] , where c r a d > 0 c_{\mathrm{rad}}>0 c rad > 0 is a sufficiently small absolute constant. Then for a sufficiently large choice of C R C_R C R , there exist absolute constants C , c > 0 C,c>0 C , c > 0 such that, with conditional probability at least 1 − C ( n ∨ p ) − c 1-C(n\vee p)^{-c} 1 − C ( n ∨ p ) − c ,
max j ∈ [ m ] max ℓ ∈ [ B j ] ∥ Π ^ j , k [ ℓ ] − A k A k T ∥ F R j , k ≤ 1 16 . (51) \max_{j\in[m]}\max_{\ell\in[B_j]}\frac{\left\|\widehat{\Pi}^{[\ell]}_{j,k}-A_kA_k^{\mathsf T}\right\|_{\mathrm F}}{R_{j,k}}\leq\frac{1}{16}. \tag*{(51)} j ∈ [ m ] max ℓ ∈ [ B j ] max R j , k Π j , k [ ℓ ] − A k A k T F ≤ 16 1 . (51)
Proof of Lemma 3. Fix j ∈ [ m ] j\in[m] j ∈ [ m ] and ℓ ∈ [ B j ] \ell\in[B_j] ℓ ∈ [ B j ] , and condition on the random partition and the factors. For notational simplicity within this proof, write I = I j , ℓ \mathcal{I}=\mathcal{I}_{j,\ell} I = I j , ℓ , s = ∣ I ∣ s=|\mathcal{I}| s = ∣ I ∣ , and A − k = ⨂ h ∈ [ K ] ∖ { k } A h A_{-k}=\bigotimes_{h\in[K]\setminus\{k\}}A_h A − k = ⨂ h ∈ [ K ] ∖ { k } A h (where the Kronecker product is always taken in descending order unless otherwise specified). Also, for each t ∈ I t\in\mathcal{I} t ∈ I , define D t : = M k ( F t ( j ) ) D_t:=\mathcal{M}_k(F_t^{(j)}) D t := M k ( F t ( j ) ) , E t : = M k ( E t ( j ) ) E_t:=\mathcal{M}_k(E_t^{(j)}) E t := M k ( E t ( j ) ) , and Y t : = M k ( X t ( j ) ) Y_t:=\mathcal{M}_k(X_t^{(j)}) Y t := M k ( X t ( j ) ) . Let block sample covariance used to construct Π ^ j , k [ ℓ ] \widehat{\Pi}^{[\ell]}_{j,k} Π j , k [ ℓ ] be
Σ ^ j , k [ ℓ ] : = 1 s ∑ t ∈ I Y t Y t T . (52) \widehat{\Sigma}^{[\ell]}_{j,k}:=\frac{1}{s}\sum_{t\in\mathcal{I}}Y_tY_t^{\mathsf T}. \tag*{(52)} Σ j , k [ ℓ ] := s 1 t ∈ I ∑ Y t Y t T . (52)
Also define S D : = s − 1 ∑ t ∈ I D t D t T S_D:=s^{-1}\sum_{t\in\mathcal{I}}D_tD_t^{\mathsf T} S D := s − 1 ∑ t ∈ I D t D t T . Then expanding each summand in (52), as the mode-k k k matricization of X t ( j ) X_t^{(j)} X t ( j ) is Y t = A k D t A − k T + E t Y_t=A_kD_tA_{-k}^{\mathsf T}+E_t Y t = A k D t A − k T + E t , and by A − k T A − k = ⨂ h ∈ [ K ] ∖ { k } ( A h T A h ) = I r − k A_{-k}^{\mathsf T}A_{-k}=\bigotimes_{h\in[K]\setminus\{k\}}(A_h^{\mathsf T}A_h)=I_{r_{-k}} A − k T A − k = ⨂ h ∈ [ K ] ∖ { k } ( A h T A h ) = I r − k from Assumption 1, we obtain
Σ ^ j , k [ ℓ ] = A k S D A k T + 1 s ∑ t ∈ I ( A k D t A − k T E t T + E t A − k D t T A k T ) + 1 s ∑ t ∈ I E t E t T . (53) \widehat{\Sigma}^{[\ell]}_{j,k}=A_kS_DA_k^{\mathsf T}+\frac{1}{s}\sum_{t\in\mathcal{I}}\left(A_kD_tA_{-k}^{\mathsf T}E_t^{\mathsf T}+E_tA_{-k}D_t^{\mathsf T}A_k^{\mathsf T}\right)+\frac{1}{s}\sum_{t\in\mathcal{I}}E_tE_t^{\mathsf T}. \tag*{(53)} Σ j , k [ ℓ ] = A k S D A k T + s 1 t ∈ I ∑ ( A k D t A − k T E t T + E t A − k D t T A k T ) + s 1 t ∈ I ∑ E t E t T . (53)
As the entries of E t E_t E t are independent N ( 0 , σ 2 ) \mathcal{N}(0,\sigma^2) N ( 0 , σ 2 ) random variables, we have E ( E t E t T ) = σ 2 p − k I p k \mathbb{E}(E_tE_t^{\mathsf T})=\sigma^2p_{-k}I_{p_k} E ( E t E t T ) = σ 2 p − k I p k and E ( A k D t A − k T E t T ∣ D t ) = 0 \mathbb{E}(A_kD_tA_{-k}^{\mathsf T}E_t^{\mathsf T}\mid D_t)=0 E ( A k D t A − k T E t T ∣ D t ) = 0 . Hence conditional on the factors and the partition, we have
Σ ∘ : = E [ Σ ^ j , k [ ℓ ] ∣ { D t } t ∈ I , I ] = A k S D A k T + σ 2 p − k I p k . (54) \Sigma^\circ:=\mathbb{E}\left[\widehat{\Sigma}^{[\ell]}_{j,k}\mid\{D_t\}_{t\in\mathcal{I}},\mathcal{I}\right]=A_kS_DA_k^{\mathsf T}+\sigma^2p_{-k}I_{p_k}. \tag*{(54)} Σ ∘ := E [ Σ j , k [ ℓ ] ∣ { D t } t ∈ I , I ] = A k S D A k T + σ 2 p − k I p k . (54)
Define the centered perturbation Δ : = Σ ^ j , k [ ℓ ] − Σ ∘ \Delta:=\widehat{\Sigma}^{[\ell]}_{j,k}-\Sigma^\circ Δ := Σ j , k [ ℓ ] − Σ ∘ , which, by (53) and (54), reduces to
Δ = 1 s ∑ t ∈ I ( A k D t A − k T E t T + E t A − k D t T A k T ) + 1 s ∑ t ∈ I ( E t E t T − σ 2 p − k I p k ) = C + C T + W , \Delta=\frac{1}{s}\sum_{t\in\mathcal{I}}\left(A_kD_tA_{-k}^{\mathsf T}E_t^{\mathsf T}+E_tA_{-k}D_t^{\mathsf T}A_k^{\mathsf T}\right)+\frac{1}{s}\sum_{t\in\mathcal{I}}\left(E_tE_t^{\mathsf T}-\sigma^2p_{-k}I_{p_k}\right)=C+C^{\mathsf T}+W, Δ = s 1 t ∈ I ∑ ( A k D t A − k T E t T + E t A − k D t T A k T ) + s 1 t ∈ I ∑ ( E t E t T − σ 2 p − k I p k ) = C + C T + W ,
where
C : = 1 s ∑ t ∈ I A k D t A − k T E t T , W : = 1 s ∑ t ∈ I ( E t E t T − σ 2 p − k I p k ) . C:=\frac{1}{s}\sum_{t\in\mathcal{I}}A_kD_tA_{-k}^{\mathsf T}E_t^{\mathsf T},\qquad W:=\frac{1}{s}\sum_{t\in\mathcal{I}}\left(E_tE_t^{\mathsf T}-\sigma^2p_{-k}I_{p_k}\right). C := s 1 t ∈ I ∑ A k D t A − k T E t T , W := s 1 t ∈ I ∑ ( E t E t T − σ 2 p − k I p k ) .
Note that the triangle inequality gives
∥ Δ ∥ ≤ 2 ∥ C ∥ + ∥ W ∥ . (55) \|\Delta\|\leq2\|C\|+\|W\|. \tag*{(55)} ∥Δ∥ ≤ 2∥ C ∥ + ∥ W ∥. (55)
Consider C C C first. Define E ~ t : = E t A − k \widetilde{E}_t:=E_tA_{-k} E t := E t A − k , so that C = s − 1 A k ∑ t ∈ I D t E ~ t T C=s^{-1}A_k\sum_{t\in\mathcal{I}}D_t\widetilde{E}_t^{\mathsf T} C = s − 1 A k ∑ t ∈ I D t E t T . Since A − k A_{-k} A − k has orthonormal columns, the entries of E ~ t ∈ R p k × r − k \widetilde{E}_t\in\mathbb{R}^{p_k\times r_{-k}} E t ∈ R p k × r − k are independent N ( 0 , σ 2 ) \mathcal{N}(0,\sigma^2) N ( 0 , σ 2 ) random variables. Hence conditional on the factors and the partition, the columns of s − 1 ∑ t ∈ I D t E ~ t T s^{-1}\sum_{t\in\mathcal{I}}D_t\widetilde{E}_t^{\mathsf T} s − 1 ∑ t ∈ I D t E t T are independent Gaussian vectors with mean zero and covariance matrix
σ 2 s 2 ∑ t ∈ I D t D t T = σ 2 s S D . \frac{\sigma^2}{s^2}\sum_{t\in\mathcal{I}}D_tD_t^{\mathsf T}=\frac{\sigma^2}{s}S_D. s 2 σ 2 t ∈ I ∑ D t D t T = s σ 2 S D .
Altogether, we have
C = d σ s A k S D 1 / 2 G 1 , (56) \mathcal{C} \overset{d}{=} \frac{\sigma}{\sqrt{s}} A_k S_D^{1/2}G_1, \tag*{(56)} C = d s σ A k S D 1/2 G 1 , (56)
where G 1 ∈ R r k × p k G_1 \in\mathbb{R}^{r_k \times p_k} G 1 ∈ R r k × p k has independent standard Gaussian entries. For every u > 0 u>0 u > 0 , the standard Gaussian spectral-norm bound (e.g. [48 Theorem 4.4.5] ) gives
∥ G 1 ∥ ≤ p k + r k + 2 u , \lVert G_1\rVert\le\sqrt{p_k}+\sqrt{r_k}+\sqrt{2u}, ∥ G 1 ∥ ≤ p k + r k + 2 u ,
with probability at least 1 − exp ( − u ) 1-\exp(-u) 1 − exp ( − u ) . In addition, (40) in Lemma 2 implies ∥ S D ∥ ≤ ( 3 / 2 ) λ j , 1 ( k ) \lVert S_D\rVert\le(3/2)\lambda_{j,1}^{(k)} ∥ S D ∥ ≤ ( 3/2 ) λ j , 1 ( k ) . Therefore, it holds with probability at least 1 − exp ( − u ) 1-\exp(-u) 1 − exp ( − u ) that
∥ C ∥ ≤ C σ λ j , 1 ( k ) p k + r k + u s . (57) \lVert\mathcal{C}\rVert\le C\sigma\sqrt{\lambda_{j,1}^{(k)}}\sqrt{\frac{p_k+r_k+u}{s}}. \tag*{(57)} ∥ C ∥ ≤ C σ λ j , 1 ( k ) s p k + r k + u . (57)
Next, consider W \mathcal{W} W . Stack the noise matrices column-wise as E : = [ E t : t ∈ I ] ∈ R p k × s p − k \mathcal{E}:=[E_t:t\in\mathcal{I}]\in\mathbb{R}^{p_k\times sp_{-k}} E := [ E t : t ∈ I ] ∈ R p k × s p − k . Note that we can write E = σ G 2 \mathcal{E}=\sigma G_2 E = σ G 2 , where G 2 ∈ R p k × s p − k G_2\in\mathbb{R}^{p_k\times sp_{-k}} G 2 ∈ R p k × s p − k has independent standard Gaussian entries. Hence,
W = σ 2 s ( G 2 G 2 T − s p − k I p k ) . \mathcal{W}=\frac{\sigma^2}{s}\left(G_2G_2^{\mathsf{T}}-sp_{-k}I_{p_k}\right). W = s σ 2 ( G 2 G 2 T − s p − k I p k ) .
The Gaussian sample-covariance bound in Vershynin [48 ] (see Equation (4.22) below Theorem 4.6.1 within) gives, with probability at least 1 − 2 exp ( − u ) 1-2\exp(-u) 1 − 2 exp ( − u ) ,
∥ W ∥ ≤ C σ 2 s ( s p − k ) { p k s p − k + u s p − k + p k s p − k + u s p − k } = C σ 2 { p − k ( p k + u ) s + p k + u s } . (58) \begin{aligned}
\lVert\mathcal{W}\rVert\le\frac{C\sigma^2}{s}(sp_{-k})\left\{\sqrt{\frac{p_k}{sp_{-k}}}+\frac{\sqrt{u}}{\sqrt{sp_{-k}}}+\frac{p_k}{sp_{-k}}+\frac{u}{sp_{-k}}\right\} \\
&= C\sigma^2\left\{\sqrt{\frac{p_{-k}(p_k+u)}{s}}+\frac{p_k+u}{s}\right\}. \tag*{(58)}
\end{aligned} ∥ W ∥ ≤ s C σ 2 ( s p − k ) { s p − k p k + s p − k u + s p − k p k + s p − k u } = C σ 2 { s p − k ( p k + u ) + s p k + u } . (58)
Combining (55) , (57) , and (58) , we obtain that, with probability at least 1 − 3 exp ( − u ) 1-3\exp(-u) 1 − 3 exp ( − u ) ,
∥ Δ ∥ ≤ C [ σ λ j , 1 ( k ) p k + r k + u s + σ 2 { p − k ( p k + u ) s + p k + u s } ] . (59) \lVert\Delta\rVert\le C\left[\sigma\sqrt{\lambda_{j,1}^{(k)}}\sqrt{\frac{p_k+r_k+u}{s}}+\sigma^2\left\{\sqrt{\frac{p_{-k}(p_k+u)}{s}}+\frac{p_k+u}{s}\right\}\right]. \tag*{(59)} ∥ Δ ∥ ≤ C [ σ λ j , 1 ( k ) s p k + r k + u + σ 2 { s p − k ( p k + u ) + s p k + u } ] . (59)
It remains to convert the above bound into a projector perturbation bound. Recall Lemma 2, it holds with high probability that
1 2 Λ j , ( k ) ⪯ S D ⪯ 3 2 Λ j , ( k ) . \frac{1}{2}\Lambda_{j,(k)}\preceq S_D\preceq\frac{3}{2}\Lambda_{j,(k)}. 2 1 Λ j , ( k ) ⪯ S D ⪯ 2 3 Λ j , ( k ) .
It follows from (54) that the leading r k r_k r k -dimensional eigenspace of Σ ∘ \Sigma^\circ Σ ∘ is span ( A k ) \operatorname{span}(A_k) span ( A k ) , with spectral projector P k = A k A k T P_k=A_kA_k^{\mathsf{T}} P k = A k A k T . Consequently,
λ r k ( Σ ∘ ) − λ r k + 1 ( Σ ∘ ) = λ r k ( S D ) ≥ 1 2 λ j , r k ( k ) . (60) \lambda_{r_k}(\Sigma^\circ)-\lambda_{r_k+1}(\Sigma^\circ)=\lambda_{r_k}(S_D)\ge\frac{1}{2}\lambda_{j,r_k}^{(k)}. \tag*{(60)} λ r k ( Σ ∘ ) − λ r k + 1 ( Σ ∘ ) = λ r k ( S D ) ≥ 2 1 λ j , r k ( k ) . (60)
As Π ^ j , k [ ℓ ] \widehat{\Pi}_{j,k}^{[\ell]} Π j , k [ ℓ ] is the spectral projector onto the leading r k r_k r k -dimensional eigenspace of Σ ^ j , k [ ℓ ] = Σ ∘ + Δ \widehat{\Sigma}_{j,k}^{[\ell]}=\Sigma^\circ+\Delta Σ j , k [ ℓ ] = Σ ∘ + Δ , the Davis–Kahan theorem (Yu et al. (2015, Theorem 2)) and (60) give
∥ Π ^ j , k [ ℓ ] − A k A k T ∥ F ≤ C r k ∥ Δ ∥ λ r k ( S D ) ≤ C r k λ j , r k ( k ) ( 2 ∥ C ∥ + ∥ W ∥ ) . (61) \left\lVert\widehat{\Pi}_{j,k}^{[\ell]}-A_kA_k^{\mathsf{T}}\right\rVert_F\le C\sqrt{r_k}\frac{\lVert\Delta\rVert}{\lambda_{r_k}(S_D)}\le\frac{C\sqrt{r_k}}{\lambda_{j,r_k}^{(k)}}(2\lVert\mathcal{C}\rVert+\lVert\mathcal{W}\rVert). \tag*{(61)} Π j , k [ ℓ ] − A k A k T F ≤ C r k λ r k ( S D ) ∥ Δ ∥ ≤ λ j , r k ( k ) C r k ( 2 ∥ C ∥ + ∥ W ∥) . (61)
Then substituting (57) and (58) into (61) yields
∥ Π ^ j , k [ ℓ ] − A k A k T ∥ F ≤ C r k [ σ λ j , 1 ( k ) λ j , r k ( k ) p k + r k + u s + σ 2 λ j , r k ( k ) { p − k ( p k + u ) s + p k + u s } ] . (62) \left\|\widehat{\Pi}^{[\ell]}_{j,k}-A_kA_k^{\mathsf T}\right\|_{\mathrm F}\le C\sqrt{r_k}\left[\frac{\sigma\sqrt{\lambda^{(k)}_{j,1}}}{\lambda^{(k)}_{j,r_k}}\sqrt{\frac{p_k+r_k+u}{s}}+\frac{\sigma^2}{\lambda^{(k)}_{j,r_k}}\left\{\sqrt{\frac{p_{-k}(p_k+u)}{s}}+\frac{p_k+u}{s}\right\}\right]. \tag*{(62)} Π j , k [ ℓ ] − A k A k T F ≤ C r k λ j , r k ( k ) σ λ j , 1 ( k ) s p k + r k + u + λ j , r k ( k ) σ 2 { s p − k ( p k + u ) + s p k + u } . (62)
Fix an absolute constant c > 0 c>0 c > 0 and, for each client j ∈ [ m ] j\in[m] j ∈ [ m ] , take u j = log { 16 m n j ( n ∨ p ) c } u_j=\log\{16mn_j(n\vee p)^c\} u j = log { 16 m n j ( n ∨ p ) c } . Since m ≤ n m\le n m ≤ n , n j ≤ n n_j\le n n j ≤ n , and n ∨ p ≥ 2 n\vee p\ge2 n ∨ p ≥ 2 , it holds that c log ( n ∨ p ) ≤ u j ≤ ( c + 6 ) log ( n ∨ p ) c\log(n\vee p)\le u_j\le(c+6)\log(n\vee p) c log ( n ∨ p ) ≤ u j ≤ ( c + 6 ) log ( n ∨ p ) , and hence u j ≍ log ( n ∨ p ) u_j\asymp\log(n\vee p) u j ≍ log ( n ∨ p ) , with comparison constants depending only on the fixed constant c c c . Also, as s ≥ b j s\ge b_j s ≥ b j (just by construction in Algorithm 1), with a sufficiently large choice of C R C_R C R , the definition of R j , k R_{j,k} R j , k in (10) implies
∥ Π ^ j , k [ ℓ ] − A k A k T ∥ F ≤ R j , k 16 \left\|\widehat{\Pi}^{[\ell]}_{j,k}-A_kA_k^{\mathsf T}\right\|_{\mathrm F}\le\frac{R_{j,k}}{16} Π j , k [ ℓ ] − A k A k T F ≤ 16 R j , k
for this fixed block, except on an event of probability at most 3 exp ( − u j ) 3\exp(-u_j) 3 exp ( − u j ) . Recall the condition R j , k ≤ c r a d R_{j,k}\le c_{\mathrm{rad}} R j , k ≤ c rad , for sufficiently small c r a d c_{\mathrm{rad}} c rad , Finally, as B j ≤ n j B_j\le n_j B j ≤ n j , a union bound over all clients and blocks gives
P { max j ∈ [ m ] max ℓ ∈ [ B j ] ∥ Π ^ j , k [ ℓ ] − A k A k T ∥ F R j , k > 1 16 } ≤ ∑ j = 1 m 3 B j exp ( − u j ) ≤ ∑ j = 1 m 3 n j 16 m n j ( n ∨ p ) c ≤ C ( n ∨ p ) − c , \mathbb{P}\left\{\max_{j\in[m]}\max_{\ell\in[B_j]}\frac{\left\|\widehat{\Pi}^{[\ell]}_{j,k}-A_kA_k^{\mathsf T}\right\|_{\mathrm F}}{R_{j,k}}>\frac{1}{16}\right\}\le\sum_{j=1}^{m}3B_j\exp(-u_j)\le\sum_{j=1}^{m}\frac{3n_j}{16mn_j(n\vee p)^c}\le C(n\vee p)^{-c}, P ⎩ ⎨ ⎧ j ∈ [ m ] max ℓ ∈ [ B j ] max R j , k Π j , k [ ℓ ] − A k A k T F > 16 1 ⎭ ⎬ ⎫ ≤ j = 1 ∑ m 3 B j exp ( − u j ) ≤ j = 1 ∑ m 16 m n j ( n ∨ p ) c 3 n j ≤ C ( n ∨ p ) − c ,
as desired. This shows (51) and thus concludes the proof of this lemma. □
Lemma 4 (Local ideal release). Let all conditions in Theorem 1 hold. Then for every client j j j , there exists a random projector Π ~ j , k \widetilde{\Pi}_{j,k} Π j , k with the following properties:
(i) the projectors Π ~ 1 , k , … , Π ~ m , k \widetilde{\Pi}_{1,k},\ldots,\widetilde{\Pi}_{m,k} Π 1 , k , … , Π m , k are independent;
(ii) It holds with probability at least 1 − C ( n ∨ p ) − c 1-C(n\vee p)^{-c} 1 − C ( n ∨ p ) − c that Π ~ j , k = U ~ j , k U ~ j , k T \widetilde{\Pi}_{j,k}=\widetilde{U}_{j,k}\widetilde{U}_{j,k}^{\mathsf T} Π j , k = U j , k U j , k T for every j ∈ [ m ] j\in[m] j ∈ [ m ] ;
(iii) ( I p k − A k A k T ) E ( Π ~ j , k ) A k A k T = 0 (I_{p_k}-A_kA_k^{\mathsf T})\mathbb{E}(\widetilde{\Pi}_{j,k})A_kA_k^{\mathsf T}=0 ( I p k − A k A k T ) E ( Π j , k ) A k A k T = 0 ;
(iv) ∥ E ( Π ~ j , k ) − A k A k T ∥ ≤ c \|\mathbb{E}(\widetilde{\Pi}_{j,k})-A_kA_k^{\mathsf T}\|\le c ∥ E ( Π j , k ) − A k A k T ∥ ≤ c for a sufficiently small absolute constant c c c ;
(v) it holds almost surely that ∥ Π ~ j , k − E ( Π ~ j , k ) ∥ F ≤ C r k Ψ j , k l o c \|\widetilde{\Pi}_{j,k}-\mathbb{E}(\widetilde{\Pi}_{j,k})\|_{\mathrm F}\le C\sqrt{r_k}\Psi^{\mathrm{loc}}_{j,k} ∥ Π j , k − E ( Π j , k ) ∥ F ≤ C r k Ψ j , k loc .
Here the expectations in parts (iii)–(v) are taken over the observation noises, the data-independent random partitions, and the auxiliary Gaussian randomness used in the construction.
Proof of Lemma 4. Fix any j ∈ [ m ] j\in[m] j ∈ [ m ] and write P : = A k A k T P:=A_kA_k^{\mathsf T} P := A k A k T for the target projector. Recall from our procedure in Algorithm 1 that I j , 1 , … , I j , B j \mathcal I_{j,1},\ldots,\mathcal I_{j,B_j} I j , 1 , … , I j , B j are the random blocks, Π ^ j , k [ ℓ ] \widehat{\Pi}^{[\ell]}_{j,k} Π j , k [ ℓ ] is the empirical projector computed from block I j , ℓ \mathcal I_{j,\ell} I j , ℓ , and Z j , k Z_{j,k} Z j , k is the Gaussian orthogonal ensemble perturbation with scale τ j , k \tau_{j,k} τ j , k . Without loss of generality, we may generate Z j , k Z_{j,k} Z j , k before the private test and ignore it when the test fails, and note that this coupling does not change the distribution of the released estimator. Let P j : = ( I j , 1 , … , I j , B j ) \mathcal P_j:=(\mathcal I_{j,1},\ldots,\mathcal I_{j,B_j}) P j := ( I j , 1 , … , I j , B j ) denote the random partition and let B j \mathcal B_j B j denote the event that (40) holds for all ℓ ∈ [ B j ] \ell\in[B_j] ℓ ∈ [ B j ] . Define the truncated block projector
Π ‾ j , k [ ℓ ] : = { Π ^ j , k [ ℓ ] , B j holds and ∥ Π ^ j , k [ ℓ ] − P ∥ F ≤ R j , k / 16 , P , otherwise . (63) \overline{\Pi}^{[\ell]}_{j,k}:=
\begin{cases}
\widehat{\Pi}^{[\ell]}_{j,k}, & \mathcal B_j\text{ holds and }\left\|\widehat{\Pi}^{[\ell]}_{j,k}-P\right\|_{\mathrm F}\le R_{j,k}/16,\\
P, & \text{otherwise}.
\end{cases} \tag*{(63)} Π j , k [ ℓ ] := { Π j , k [ ℓ ] , P , B j holds and Π j , k [ ℓ ] − P F ≤ R j , k /16 , otherwise . (63)
It is noteworthy that this construction will be used only in the proof as it depends on the unknown P P P . By (63) , it holds almost surely that
∥ Π ‾ j , k [ ℓ ] − P ∥ F ≤ R j , k / 16. (64) \left\|\overline{\Pi}_{j,k}^{[\ell]}-P\right\|_{\mathrm{F}} \le R_{j,k}/16. \tag*{(64)} Π j , k [ ℓ ] − P F ≤ R j , k /16. (64)
Moreover, if B j \mathcal{B}_j B j and (51) hold, then Π ‾ j , k [ ℓ ] = Π ^ j , k [ ℓ ] \overline{\Pi}_{j,k}^{[\ell]}=\widehat{\Pi}_{j,k}^{[\ell]} Π j , k [ ℓ ] = Π j , k [ ℓ ] for every ℓ ∈ [ B j ] \ell\in[\mathcal{B}_j] ℓ ∈ [ B j ] . Conditional on P j \mathcal{P}_j P j , the blocks are disjoint. Assumption 3 therefore implies that Π ‾ j , k [ 1 ] , … , Π ‾ j , k [ B j ] \overline{\Pi}_{j,k}^{[1]},\ldots,\overline{\Pi}_{j,k}^{[\mathcal{B}_j]} Π j , k [ 1 ] , … , Π j , k [ B j ] are conditionally independent. Define their conditional average and centered summands by
M ‾ j , k : = E { 1 B j ∑ ℓ = 1 B j Π ‾ j , k [ ℓ ] | P j } , X j , k [ ℓ ] : = Π ‾ j , k [ ℓ ] − E { Π ‾ j , k [ ℓ ] | P j } , (65) \overline{M}_{j,k}:=\mathbb{E}\left\{\frac{1}{B_j}\sum_{\ell=1}^{B_j}\overline{\Pi}_{j,k}^{[\ell]}\,\middle|\,\mathcal{P}_j\right\},\qquad X_{j,k}^{[\ell]}:=\overline{\Pi}_{j,k}^{[\ell]}-\mathbb{E}\left\{\overline{\Pi}_{j,k}^{[\ell]}\,\middle|\,\mathcal{P}_j\right\}, \tag*{(65)} M j , k := E ⎩ ⎨ ⎧ B j 1 ℓ = 1 ∑ B j Π j , k [ ℓ ] P j ⎭ ⎬ ⎫ , X j , k [ ℓ ] := Π j , k [ ℓ ] − E { Π j , k [ ℓ ] P j } , (65)
respectively. The matrices X j , k [ ℓ ] X_{j,k}^{[\ell]} X j , k [ ℓ ] are conditionally independent and centered and also, by (64) and Jensen’s inequality:
∥ X j , k [ ℓ ] ∥ F ≤ ∥ Π ‾ j , k [ ℓ ] − P ∥ F + E { ∥ Π ‾ j , k [ ℓ ] − P ∥ F | P j } ≤ R j , k / 8. (66) \left\|X_{j,k}^{[\ell]}\right\|_{\mathrm{F}}\le\left\|\overline{\Pi}_{j,k}^{[\ell]}-P\right\|_{\mathrm{F}}+\mathbb{E}\left\{\left\|\overline{\Pi}_{j,k}^{[\ell]}-P\right\|_{\mathrm{F}}\,\middle|\,\mathcal{P}_j\right\}\le R_{j,k}/8. \tag*{(66)} X j , k [ ℓ ] F ≤ Π j , k [ ℓ ] − P F + E { Π j , k [ ℓ ] − P F P j } ≤ R j , k /8. (66)
The symmetric p k × p k p_k\times p_k p k × p k matrices equipped with the Frobenius inner product form a Hilbert space. Hence applying Theorem 3.5 of Pinelis [44 ] , conditionally on P j \mathcal{P}_j P j to (65) and (66) , gives for every u > 0 u>0 u > 0 that
P { ∥ 1 B j ∑ ℓ = 1 B j X j , k [ ℓ ] ∥ F > C R j , k u B j | P j } ≤ 2 e − u . (67) \mathbb{P}\left\{\left\|\frac{1}{B_j}\sum_{\ell=1}^{B_j}X_{j,k}^{[\ell]}\right\|_{\mathrm{F}}>CR_{j,k}\sqrt{\frac{u}{B_j}}\,\middle|\,\mathcal{P}_j\right\}\le2e^{-u}. \tag*{(67)} P ⎩ ⎨ ⎧ B j 1 ℓ = 1 ∑ B j X j , k [ ℓ ] F > C R j , k B j u P j ⎭ ⎬ ⎫ ≤ 2 e − u . (67)
Since m ≤ n ≤ n ∨ p m\le n\le n\vee p m ≤ n ≤ n ∨ p , taking u = ( c + 1 ) log ( n ∨ p ) u=(c+1)\log(n\vee p) u = ( c + 1 ) log ( n ∨ p ) in (67) and then averaging over P j \mathcal{P}_j P j , we have
∥ 1 B j ∑ ℓ = 1 B j Π ‾ j , k [ ℓ ] − M ‾ j , k ∥ F ≤ C R j , k log ( n ∨ p ) B j , (68) \left\|\frac{1}{B_j}\sum_{\ell=1}^{B_j}\overline{\Pi}_{j,k}^{[\ell]}-\overline{M}_{j,k}\right\|_{\mathrm{F}}\le CR_{j,k}\sqrt{\frac{\log(n\vee p)}{B_j}}, \tag*{(68)} B j 1 ℓ = 1 ∑ B j Π j , k [ ℓ ] − M j , k F ≤ C R j , k B j log ( n ∨ p ) , (68)
with probability at least 1 − C ( n ∨ p ) − c / m 1-C(n\vee p)^{-c/m} 1 − C ( n ∨ p ) − c / m . Also, the standard Gaussian spectral-norm bound [48 Equation (4.22] gives
∥ Z j , k ∥ ≤ C τ j , k p k + log ( n ∨ p ) , (69) \left\|Z_{j,k}\right\|\le C\tau_{j,k}\sqrt{p_k+\log(n\vee p)}, \tag*{(69)} ∥ Z j , k ∥ ≤ C τ j , k p k + log ( n ∨ p ) , (69)
with probability at least 1 − C ( n ∨ p ) − c / m 1-C(n\vee p)^{-c/m} 1 − C ( n ∨ p ) − c / m . Let E j , k \mathcal{E}_{j,k} E j , k denote the intersection of the events in (68) and (69) , so that
P ( E j , k c ) ≤ C ( n ∨ p ) − c / m . (70) \mathbb{P}(\mathcal{E}_{j,k}^{c})\le C(n\vee p)^{-c/m}. \tag*{(70)} P ( E j , k c ) ≤ C ( n ∨ p ) − c / m . (70)
We next identify the leading eigenspace of the conditional mean M ‾ j , k \overline{M}_{j,k} M j , k . Let O : = 2 P − I p k O:=2P-I_{p_k} O := 2 P − I p k . As P P P is an orthogonal projector, O T O = I p k O^{\mathsf{T}}O=I_{p_k} O T O = I p k , O P = P O = P OP=PO=P O P = P O = P , ( I p k − P ) O = − ( I p k − P ) (I_{p_k}-P)O=-(I_{p_k}-P) ( I p k − P ) O = − ( I p k − P ) , and O A k = A k OA_k=A_k O A k = A k . By (1) , left multiplication of the mode-k k k matricization by O O O leaves its signal component unchanged, and hence with Assumption 3, we have
O M k ( E t ( j ) ) = d M k ( E t ( j ) ) , OM_k(E_t^{(j)})\mathrel{\overset{d}{=}}M_k(E_t^{(j)}), O M k ( E t ( j ) ) = d M k ( E t ( j ) ) ,
so that conditional on P j \mathcal{P}_j P j , O Π ‾ j , k [ ℓ ] O = d Π ‾ j , k [ ℓ ] O\overline{\Pi}_{j,k}^{[\ell]}O\mathrel{\overset{d}{=}}\overline{\Pi}_{j,k}^{[\ell]} O Π j , k [ ℓ ] O = d Π j , k [ ℓ ] . By O P O = P OPO=P O P O = P and the fact that the Frobenius norm is invariant to orthogonal transformation, the truncation in (63) is equivariant under conjugation by O O O . We then arrive at
O Π ‾ j , k [ ℓ ] O = d Π ‾ j , k [ ℓ ] , O M ‾ j , k O = M ‾ j , k . (71) O\overline{\Pi}_{j,k}^{[\ell]}O\mathrel{\overset{d}{=}}\overline{\Pi}_{j,k}^{[\ell]},\qquad O\overline{M}_{j,k}O=\overline{M}_{j,k}. \tag*{(71)} O Π j , k [ ℓ ] O = d Π j , k [ ℓ ] , O M j , k O = M j , k . (71)
By O P = P OP = P O P = P and ( I p k − P ) O = − ( I p k − P ) (I_{p_k} - P)O = -(I_{p_k} - P) ( I p k − P ) O = − ( I p k − P ) in (71) , we have
( I p k − P ) M ‾ j , k P = − ( I p k − P ) M ‾ j , k P = 0. (72) (I_{p_k} - P)\overline{M}_{j,k}P = -(I_{p_k} - P)\overline{M}_{j,k}P = 0. \tag*{(72)} ( I p k − P ) M j , k P = − ( I p k − P ) M j , k P = 0. (72)
Hence M ‾ j , k \overline{M}_{j,k} M j , k is block diagonal with respect to span ( A k ) ⊕ span ( A k ) ⊥ \operatorname{span}(A_k) \oplus\operatorname{span}(A_k)^\perp span ( A k ) ⊕ span ( A k ) ⊥ . Furthermore, (64) and Jensen’s inequality imply
∥ M ‾ j , k − P ∥ ≤ ∥ M ‾ j , k − P ∥ F ≤ R j , k / 16. (73) \lVert\overline{M}_{j,k} - P \rVert\le\lVert\overline{M}_{j,k} - P \rVert_F \le R_{j,k}/16. \tag*{(73)} ∥ M j , k − P ∥ ≤ ∥ M j , k − P ∥ F ≤ R j , k /16. (73)
Combining (72) and (73) , the leading r k r_k r k -dimensional eigenspace of M ‾ j , k \overline{M}_{j,k} M j , k is span ( A k ) \operatorname{span}(A_k) span ( A k ) and its eigengap is at least 1 − R j , k / 8 1 - R_{j,k}/8 1 − R j , k /8 . Note that the lower bound in (12) and our choice of block size choice ensure R j , k R_{j,k} R j , k is sufficiently small, so this eigengap is bounded below by an absolute positive constant. Next, define the ideal loading matrix and its projector by
U j , k ∘ : = SVD r k { 1 B j ∑ ℓ = 1 B j Π ‾ j , k [ ℓ ] + Z j , k } , Π j , k ∘ : = U j , k ∘ ( U j , k ∘ ) ⊤ . (74) U^\circ_{j,k} := \operatorname{SVD}_{r_k}\left\{ \frac{1}{B_j}\sum_{\ell=1}^{B_j}\overline{\Pi}^{[\ell]}_{j,k} + Z_{j,k} \right\}, \qquad\Pi^\circ_{j,k} := U^\circ_{j,k}(U^\circ_{j,k})^\top. \tag*{(74)} U j , k ∘ := SVD r k ⎩ ⎨ ⎧ B j 1 ℓ = 1 ∑ B j Π j , k [ ℓ ] + Z j , k ⎭ ⎬ ⎫ , Π j , k ∘ := U j , k ∘ ( U j , k ∘ ) ⊤ . (74)
On E j , k \mathcal{E}_{j,k} E j , k , write the matrix inside the SVD above as M ‾ j , k + Δ j , k \overline{M}_{j,k} + \Delta_{j,k} M j , k + Δ j , k , where
Δ j , k : = 1 B j ∑ ℓ = 1 B j Π ‾ j , k [ ℓ ] − M ‾ j , k + Z j , k . \Delta_{j,k} := \frac{1}{B_j}\sum_{\ell=1}^{B_j}\overline{\Pi}^{[\ell]}_{j,k} - \overline{M}_{j,k} + Z_{j,k}. Δ j , k := B j 1 ℓ = 1 ∑ B j Π j , k [ ℓ ] − M j , k + Z j , k .
Combining (68) , (69) , and (11) , we have ∥ Δ j , k ∥ ≤ C r k Ψ j , k l o c \lVert\Delta_{j,k}\rVert\le C\sqrt{r_k}\Psi^{\mathrm{loc}}_{j,k} ∥ Δ j , k ∥ ≤ C r k Ψ j , k loc . Then by Davis–Kahan theorem (Yu et al. (2015), Theorem 2), we have
∥ Π j , k ∘ − P ∥ F ≤ C ∥ Δ j , k P ∥ F ≤ C { ∥ 1 B j ∑ ℓ = 1 B j Π ‾ j , k [ ℓ ] − M ‾ j , k ∥ F + r k ∥ Z j , k ∥ } ≤ C { R j , k log ( n ∨ p ) B j + r k τ j , k p k + log ( n ∨ p ) } = C r k Ψ j , k l o c . (75) \begin{aligned}
\lVert\Pi^\circ_{j,k} - P\rVert_F
&\le C\lVert\Delta_{j,k}P\rVert_F \le C\left\{\left\lVert\frac{1}{B_j}\sum_{\ell=1}^{B_j}\overline{\Pi}^{[\ell]}_{j,k} - \overline{M}_{j,k}\right\rVert_F + \sqrt{r_k}\lVert Z_{j,k}\rVert\right\}\\
&\le C\left\{R_{j,k}\sqrt{\frac{\log(n \vee p)}{B_j}}+\sqrt{r_k}\tau_{j,k}\sqrt{p_k+\log(n \vee p)}\right\}=C\sqrt{r_k}\Psi^{\mathrm{loc}}_{j,k}. \tag*{(75)}
\end{aligned} ∥ Π j , k ∘ − P ∥ F ≤ C ∥ Δ j , k P ∥ F ≤ C ⎩ ⎨ ⎧ B j 1 ℓ = 1 ∑ B j Π j , k [ ℓ ] − M j , k F + r k ∥ Z j , k ∥ ⎭ ⎬ ⎫ ≤ C { R j , k B j log ( n ∨ p ) + r k τ j , k p k + log ( n ∨ p ) } = C r k Ψ j , k loc . (75)
Now define projector
Π ~ j , k : = { Π j , k ∘ , E j , k holds , P , E j , k c holds . (76) \widetilde{\Pi}_{j,k} := \begin{cases}
\Pi^\circ_{j,k}, & \mathcal{E}_{j,k}\text{ holds},\\
P, & \mathcal{E}_{j,k}^{c}\text{ holds}.
\end{cases} \tag*{(76)} Π j , k := { Π j , k ∘ , P , E j , k holds , E j , k c holds . (76)
We next verify parts (iii)–(v) in Lemma 4. The GOE distribution satisfies O Z j , k O = d Z j , k OZ_{j,k}O \overset{d}{=} Z_{j,k} O Z j , k O = d Z j , k . Together with (71) , this shows that O Π j , k ∘ O = d Π j , k ∘ O\Pi^\circ_{j,k}O \overset{d}{=} \Pi^\circ_{j,k} O Π j , k ∘ O = d Π j , k ∘ conditional on P j \mathcal{P}_j P j . The event E j , k \mathcal{E}_{j,k} E j , k is also invariant under this conjugation because it is defined by the Frobenius and spectral norms in (68) and (69) . Since O P O = P OPO = P O P O = P , (76) gives O Π ~ j , k O = d Π ~ j , k O\widetilde{\Pi}_{j,k}O \overset{d}{=} \widetilde{\Pi}_{j,k} O Π j , k O = d Π j , k . Taking expectations and using O P = P OP = P O P = P and ( I p k − P ) O = − ( I p k − P ) (I_{p_k} - P)O = -(I_{p_k} - P) ( I p k − P ) O = − ( I p k − P ) , we obtain
( I p k − P ) E ( Π ~ j , k ) P = 0 , (77) (I_{p_k} - P)\mathbb{E}(\widetilde{\Pi}_{j,k})P = 0, \tag*{(77)} ( I p k − P ) E ( Π j , k ) P = 0 , (77)
which is part (iii). By (75) and (76) , it holds almost surely that
∥ Π ~ j , k − P ∥ F ≤ C r k Ψ j , k l o c . (78) \lVert\widetilde{\Pi}_{j,k} - P\rVert_F \le C\sqrt{r_k}\Psi^{\mathrm{loc}}_{j,k}. \tag*{(78)} ∥ Π j , k − P ∥ F ≤ C r k Ψ j , k loc . (78)
By Jensen’s inequality, (78) , and the last condition in (15) , we have
∥ E ( Π ~ j , k ) − P ∥ ≤ E ∥ Π ~ j , k − P ∥ F ≤ C r k Ψ j , k l o c ≤ c , \left\|\mathbb{E}(\widetilde{\Pi}_{j,k})-P\right\| \leq\mathbb{E}\left\|\widetilde{\Pi}_{j,k}-P\right\|_{F} \leq C\sqrt{r_k}\Psi^{\mathrm{loc}}_{j,k} \leq c, E ( Π j , k ) − P ≤ E Π j , k − P F ≤ C r k Ψ j , k loc ≤ c ,
after choosing the constant C 0 C_0 C 0 in (12) and (13) sufficiently large. This proves part (iv). Also by the triangle and Jensen inequalities, we have
∥ Π ~ j , k − E ( Π ~ j , k ) ∥ F ≤ ∥ Π ~ j , k − P ∥ F + ∥ E ( Π ~ j , k ) − P ∥ F ≤ C r k Ψ j , k l o c , \left\|\widetilde{\Pi}_{j,k}-\mathbb{E}(\widetilde{\Pi}_{j,k})\right\|_{F} \leq\left\|\widetilde{\Pi}_{j,k}-P\right\|_{F}+\left\|\mathbb{E}(\widetilde{\Pi}_{j,k})-P\right\|_{F} \leq C\sqrt{r_k}\Psi^{\mathrm{loc}}_{j,k}, Π j , k − E ( Π j , k ) F ≤ Π j , k − P F + E ( Π j , k ) − P F ≤ C r k Ψ j , k loc ,
which proves part (v).
We next prove part (i) in Lemma 4 . For different clients, the data noises are independent by Assumption 3, and the random partitions and GOE matrices are generated independently by construction. Note that all constructions in (63) –(76) use only the corresponding client’s random variables and the fixed matrix P P P . Hence Π ~ 1 , k , … , Π ~ m , k \widetilde{\Pi}_{1,k},\ldots,\widetilde{\Pi}_{m,k} Π 1 , k , … , Π m , k are independent.
It remains to establish the simultaneous coupling in part (ii) of Lemma 4 . On the intersection of B j B_j B j and the event in (51) , the definition (63) gives Π ~ j , k = Π ^ j , k [ ℓ ] \widetilde{\Pi}_{j,k}=\widehat{\Pi}^{[\ell]}_{j,k} Π j , k = Π j , k [ ℓ ] for every ℓ ∈ [ B j ] \ell\in[B_j] ℓ ∈ [ B j ] . Moreover, for any ℓ , h ∈ [ B j ] \ell,h\in[B_j] ℓ , h ∈ [ B j ] ,
∥ Π ^ j , k [ ℓ ] − Π ^ j , k [ h ] ∥ F ≤ ∥ Π ^ j , k [ ℓ ] − P ∥ F + ∥ Π ^ j , k [ h ] − P ∥ F ≤ R j , k / 8. \left\|\widehat{\Pi}^{[\ell]}_{j,k}-\widehat{\Pi}^{[h]}_{j,k}\right\|_{F}\leq\left\|\widehat{\Pi}^{[\ell]}_{j,k}-P\right\|_{F}+\left\|\widehat{\Pi}^{[h]}_{j,k}-P\right\|_{F}\leq R_{j,k}/8. Π j , k [ ℓ ] − Π j , k [ h ] F ≤ Π j , k [ ℓ ] − P F + Π j , k [ h ] − P F ≤ R j , k /8.
Hence, the score defined in Algorithm 1 satisfies q j , k = B j q_{j,k}=B_j q j , k = B j . Let H j , k 0 : = B j − 1 ∑ ℓ = 1 B j Π ^ j , k [ ℓ ] H^0_{j,k}:=B_j^{-1}\sum_{\ell=1}^{B_j}\widehat{\Pi}^{[\ell]}_{j,k} H j , k 0 := B j − 1 ∑ ℓ = 1 B j Π j , k [ ℓ ] . For every ℓ ∈ [ B j ] \ell\in[B_j] ℓ ∈ [ B j ] , we have
∥ H j , k 0 − Π ^ j , k [ ℓ ] ∥ F ≤ 1 B j ∑ h = 1 B j ∥ Π ^ j , k [ h ] − Π ^ j , k [ ℓ ] ∥ F ≤ R j , k / 8 < R j , k . \left\|H^0_{j,k}-\widehat{\Pi}^{[\ell]}_{j,k}\right\|_{F}\leq\frac{1}{B_j}\sum_{h=1}^{B_j}\left\|\widehat{\Pi}^{[h]}_{j,k}-\widehat{\Pi}^{[\ell]}_{j,k}\right\|_{F}\leq R_{j,k}/8<R_{j,k}. H j , k 0 − Π j , k [ ℓ ] F ≤ B j 1 h = 1 ∑ B j Π j , k [ h ] − Π j , k [ ℓ ] F ≤ R j , k /8 < R j , k .
Thus, every Huber term is quadratic at H j , k 0 H^0_{j,k} H j , k 0 , and the gradient of the Huber objective at this point is ∑ ℓ = 1 B j ( H j , k 0 − Π ^ j , k [ ℓ ] ) = 0 \sum_{\ell=1}^{B_j}(H^0_{j,k}-\widehat{\Pi}^{[\ell]}_{j,k})=0 ∑ ℓ = 1 B j ( H j , k 0 − Π j , k [ ℓ ] ) = 0 . The convexity of the objective in the minimization problem of Algorithm 1 and its strict convexity in a neighborhood of H j , k 0 H^0_{j,k} H j , k 0 imply that its selected minimizer is
H ^ j , k = H j , k 0 = 1 B j ∑ ℓ = 1 B j Π ^ j , k [ ℓ ] . (79) \widehat{H}_{j,k}=H^0_{j,k}=\frac{1}{B_j}\sum_{\ell=1}^{B_j}\widehat{\Pi}^{[\ell]}_{j,k}. \tag*{(79)} H j , k = H j , k 0 = B j 1 ℓ = 1 ∑ B j Π j , k [ ℓ ] . (79)
When q j , k = B j q_{j,k}=B_j q j , k = B j , the test in Algorithm 1 fails only if
W j , k < 2 + 4 ε j log ( 2 δ j ) − B j 8 . W_{j,k}<2+\frac{4}{\varepsilon_j}\log\left(\frac{2}{\delta_j}\right)-\frac{B_j}{8}. W j , k < 2 + ε j 4 log ( δ j 2 ) − 8 B j .
Note that the first condition in (15) makes the right-hand side at most − 4 log { 4 m ( n ∨ p ) − c } / ε j -4\log\{4m(n\vee p)^{-c}\}/\varepsilon_j − 4 log { 4 m ( n ∨ p ) − c } / ε j . Since W j , k ∼ Lap ( 4 / ε j ) W_{j,k}\sim\operatorname{Lap}(4/\varepsilon_j) W j , k ∼ Lap ( 4/ ε j ) ,
P { the test of client j fails ∣ q j , k = B j } ≤ 1 8 m ( n ∨ p ) c ≤ ( n ∨ p ) − c 4 m . (80) \mathbb{P}\{\text{the test of client }j\text{ fails}\mid q_{j,k}=B_j\}\leq\frac{1}{8m(n\vee p)^c}\leq\frac{(n\vee p)^{-c}}{4m}. \tag*{(80)} P { the test of client j fails ∣ q j , k = B j } ≤ 8 m ( n ∨ p ) c 1 ≤ 4 m ( n ∨ p ) − c . (80)
Under B j B_j B j , (51) , E j , k \mathcal{E}_{j,k} E j , k , and the event that the test passes, then (74) , (76) , and (79) altogether give Π ~ j , k = Π j , k ∘ = U ~ j , k U ~ j , k ⊤ \widetilde{\Pi}_{j,k}=\Pi^\circ_{j,k}=\widetilde{U}_{j,k}\widetilde{U}_{j,k}^{\top} Π j , k = Π j , k ∘ = U j , k U j , k ⊤ . By Lemma 2 and our choice of b j b_j b j as in Theorem 1, a sufficiently large C b C_b C b makes the probability that B j B_j B j fails for at least one client at most C ( n ∨ p ) − c C(n\vee p)^{-c} C ( n ∨ p ) − c . Finally, note that Lemma 3 controls (51) , and also, (70) and a union bound control ⋂ j = 1 m E j , k \bigcap_{j=1}^{m}\mathcal{E}_{j,k} ⋂ j = 1 m E j , k , and note that (80) controls the private tests. Altogether, we conclude
P { U ~ j , k = U ~ j , k U ~ j , k ⊤ , ∀ j ∈ [ m ] } ≥ 1 − C ( n ∨ p ) − c , \mathbb{P}\left\{\widetilde{U}_{j,k}=\widetilde{U}_{j,k}\widetilde{U}_{j,k}^{\top},\ \forall j\in[m]\right\}\geq1-C(n\vee p)^{-c}, P { U j , k = U j , k U j , k ⊤ , ∀ j ∈ [ m ] } ≥ 1 − C ( n ∨ p ) − c ,
which proves part (ii) of this lemma and thus completes this proof.
Lemma 5 (Record-level private information contraction for Gaussian models). Let Θ ⊆ R d \Theta\subseteq\mathbb{R}^{d} Θ ⊆ R d be open and let D = ( Y 1 , … , Y N ) \mathcal{D}=(Y_{1},\ldots,Y_{N}) D = ( Y 1 , … , Y N ) consist of independent records satisfying
Y t ∼ N { μ t ( θ ) , σ 2 I q } , t ∈ [ N ] , Y_t\sim\mathcal{N}\{\mu_t(\theta),\sigma^{2}I_q\},\qquad t\in[N], Y t ∼ N { μ t ( θ ) , σ 2 I q } , t ∈ [ N ] , where θ ∈ Θ \theta\in\Theta θ ∈ Θ , σ 2 > 0 \sigma^{2}>0 σ 2 > 0 , and each μ t : Θ ↦ R q \mu_t:\Theta\mapsto\mathbb{R}^{q} μ t : Θ ↦ R q is continuously differentiable. Write B t ( θ ) : = D θ μ t ( θ ) ∈ R q × d B_t(\theta):=D_\theta\mu_t(\theta)\in\mathbb{R}^{q\times d} B t ( θ ) := D θ μ t ( θ ) ∈ R q × d for its Jacobian. Suppose that for some λ > 0 \lambda>0 λ > 0 and absolute constants C ≥ c > 0 C\geq c>0 C ≥ c > 0 , uniformly over θ ∈ Θ \theta\in\Theta θ ∈ Θ , that
c N λ I d ⪯ ∑ t = 1 N B t ( θ ) ⊤ B t ( θ ) ⪯ C N λ I d and max t ∈ [ N ] ∥ B t ( θ ) ∥ 2 ≤ C λ . (81) cN\lambda I_d\preceq\sum_{t=1}^{N}B_t(\theta)^{\top}B_t(\theta)\preceq CN\lambda I_d\qquad\text{and}\qquad\max_{t\in[N]}\|B_t(\theta)\|^{2}\leq C\lambda. \tag*{(81)} c N λ I d ⪯ t = 1 ∑ N B t ( θ ) ⊤ B t ( θ ) ⪯ C N λ I d and t ∈ [ N ] max ∥ B t ( θ ) ∥ 2 ≤ C λ . (81) Let Z = M ( D ) Z=\mathcal{M}(\mathcal{D}) Z = M ( D ) , where M \mathcal{M} M is a parameter-independent ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -centrally differentially private mechanism under replacement of one record, taking values in a standard Borel space. Suppose that there exists an absolute constant ε 0 , c 0 > 0 \varepsilon_0,c_0>0 ε 0 , c 0 > 0 such that 0 < ε ≤ ε 0 0<\varepsilon\leq\varepsilon_0 0 < ε ≤ ε 0 , 0 < δ ≤ exp ( − 1 ) 0<\delta\leq\exp(-1) 0 < δ ≤ exp ( − 1 ) , and
d δ log ( 1 / δ ) ≤ c 0 N ε 2 . (82) d\delta\log(1/\delta)\leq c_0\sqrt{N}\varepsilon^{2}. \tag*{(82)} d δ log ( 1/ δ ) ≤ c 0 N ε 2 . (82) Let I Z ( θ ) I_Z(\theta) I Z ( θ ) denote the Fisher information matrix for θ \theta θ in the induced distribution of Z Z Z . Then uniformly over θ ∈ Θ \theta\in\Theta θ ∈ Θ , there exists an absolute constant C ′ > 0 C'>0 C ′ > 0 such that
tr { I Z ( θ ) } ≤ C ′ λ σ 2 { ( N d ) ∧ ( N 2 ε 2 ) } . (83) \operatorname{tr}\{I_Z(\theta)\}\leq C'\frac{\lambda}{\sigma^{2}}\{(Nd)\wedge(N^{2}\varepsilon^{2})\}. \tag*{(83)} tr { I Z ( θ )} ≤ C ′ σ 2 λ {( N d ) ∧ ( N 2 ε 2 )} . (83)
Proof of Lemma 5. We divide the proof into four steps.
Step 1. Complete-data information. Let p t , θ p_{t,\theta} p t , θ be the density of Y t Y_t Y t . Its score is
S t ( θ ) : = ∇ θ log ( p t , θ ( Y t ) ) = 1 σ 2 B t ( θ ) ⊤ { Y t − μ t ( θ ) } . S_t(\theta):=\nabla_\theta\log(p_{t,\theta}(Y_t))=\frac{1}{\sigma^{2}}B_t(\theta)^{\top}\{Y_t-\mu_t(\theta)\}. S t ( θ ) := ∇ θ log ( p t , θ ( Y t )) = σ 2 1 B t ( θ ) ⊤ { Y t − μ t ( θ )} .
Thus, S t ( θ ) S_t(\theta) S t ( θ ) is centered Gaussian with covariance σ − 2 B t ( θ ) ⊤ B t ( θ ) \sigma^{-2}B_t(\theta)^{\top}B_t(\theta) σ − 2 B t ( θ ) ⊤ B t ( θ ) . Since the records are independent, the complete-data score is ∑ t = 1 N S t ( θ ) \sum_{t=1}^{N}S_t(\theta) ∑ t = 1 N S t ( θ ) and its Fisher information is
J ( θ ) : = E θ { ( ∑ t = 1 N S t ( θ ) ) ( ∑ t = 1 N S t ( θ ) ) ⊤ } = 1 σ 2 ∑ t = 1 N B t ( θ ) ⊤ B t ( θ ) , (84) J(\theta):=\mathbb{E}_{\theta}\left\{\left(\sum_{t=1}^{N}S_t(\theta)\right)\left(\sum_{t=1}^{N}S_t(\theta)\right)^{\top}\right\}=\frac{1}{\sigma^{2}}\sum_{t=1}^{N}B_t(\theta)^{\top}B_t(\theta), \tag*{(84)} J ( θ ) := E θ ⎩ ⎨ ⎧ ( t = 1 ∑ N S t ( θ ) ) ( t = 1 ∑ N S t ( θ ) ) ⊤ ⎭ ⎬ ⎫ = σ 2 1 t = 1 ∑ N B t ( θ ) ⊤ B t ( θ ) , (84)
where the last equality follows from the fact that the record scores are independent and centered. Combining (81) and (84) , we have
c N λ σ 2 I d ⪯ J ( θ ) ⪯ C N λ σ 2 I d . (85) cN\frac{\lambda}{\sigma^{2}}I_d\preceq J(\theta)\preceq CN\frac{\lambda}{\sigma^{2}}I_d. \tag*{(85)} c N σ 2 λ I d ⪯ J ( θ ) ⪯ C N σ 2 λ I d . (85)
Step 2. Score of the private output. Let p θ ( y ) = ∏ t = 1 N p t , θ ( y t ) p_\theta(\mathbf{y}) = \prod_{t=1}^{N} p_{t,\theta}(y_t) p θ ( y ) = ∏ t = 1 N p t , θ ( y t ) . Write k ( z ∣ y ) k(z \mid\mathbf{y}) k ( z ∣ y ) for the conditional density or mass function of the output given D = y \mathcal{D} = \mathbf{y} D = y , and let q θ q_\theta q θ be the density or mass function of Z Z Z . Since the mechanism does not depend on θ \theta θ , neither does k ( z ∣ y ) k(z \mid\mathbf{y}) k ( z ∣ y ) , and
q θ ( z ) = ∫ k ( z ∣ y ) p θ ( y ) d y . q_\theta(z) = \int k(z \mid\mathbf{y})p_\theta(\mathbf{y})\mathrm{d}\mathbf{y}. q θ ( z ) = ∫ k ( z ∣ y ) p θ ( y ) d y .
Differentiating under the integral gives that
∇ θ log ( q θ ( z ) ) = 1 q θ ( z ) ∫ k ( z ∣ y ) ∇ θ p θ ( y ) d y = ∫ ∑ t = 1 N ∇ θ log ( p t , θ ( y t ) ) k ( z ∣ y ) p θ ( y ) q θ ( z ) d y = E θ { ∑ t = 1 N S t ( θ ) ∣ Z = z } , (86) \begin{aligned}
\nabla_\theta\log(q_\theta(z)) &= \frac{1}{q_\theta(z)}\int k(z \mid\mathbf{y})\nabla_\theta p_\theta(\mathbf{y})\mathrm{d}\mathbf{y}\\
&= \int\sum_{t=1}^{N}\nabla_\theta\log(p_{t,\theta}(y_t))\frac{k(z \mid\mathbf{y})p_\theta(\mathbf{y})}{q_\theta(z)}\mathrm{d}\mathbf{y}
= \mathbb{E}_\theta\left\{\left.\sum_{t=1}^{N}S_t(\theta)\right|Z=z\right\}, \tag*{(86)}
\end{aligned} ∇ θ log ( q θ ( z )) = q θ ( z ) 1 ∫ k ( z ∣ y ) ∇ θ p θ ( y ) d y = ∫ t = 1 ∑ N ∇ θ log ( p t , θ ( y t )) q θ ( z ) k ( z ∣ y ) p θ ( y ) d y = E θ { t = 1 ∑ N S t ( θ ) Z = z } , (86)
where the second equality follows from
∇ θ p θ ( y ) = p θ ( y ) ∑ t = 1 N ∇ θ log ( p t , θ ( y t ) ) , \nabla_\theta p_\theta(\mathbf{y}) = p_\theta(\mathbf{y})\sum_{t=1}^{N}\nabla_\theta\log(p_{t,\theta}(y_t)), ∇ θ p θ ( y ) = p θ ( y ) t = 1 ∑ N ∇ θ log ( p t , θ ( y t )) ,
and the last follows since k ( z ∣ y ) p θ ( y ) / q θ ( z ) k(z \mid\mathbf{y})p_\theta(\mathbf{y})/q_\theta(z) k ( z ∣ y ) p θ ( y ) / q θ ( z ) is the conditional density of D \mathcal{D} D given Z = z Z=z Z = z . Consequently, (86) gives
I Z ( θ ) = E θ [ E θ { ∑ t = 1 N S t ( θ ) ∣ Z } E θ { ∑ t = 1 N S t ( θ ) ∣ Z } T ] . (87) I_Z(\theta) = \mathbb{E}_\theta\left[\mathbb{E}_\theta\left\{\left.\sum_{t=1}^{N}S_t(\theta)\right|Z\right\}\mathbb{E}_\theta\left\{\left.\sum_{t=1}^{N}S_t(\theta)\right|Z\right\}^{\mathsf{T}}\right]. \tag*{(87)} I Z ( θ ) = E θ E θ { t = 1 ∑ N S t ( θ ) Z } E θ { t = 1 ∑ N S t ( θ ) Z } T . (87)
Step 3. Private-score contraction. We apply Cai and Li [12 ] to bound the normalized Fisher information tr { J ( θ ) − 1 I Z ( θ ) } \operatorname{tr}\{J(\theta)^{-1}I_Z(\theta)\} tr { J ( θ ) − 1 I Z ( θ )} retained in the privatized output Z Z Z . Note that (85) implies J ( θ ) ≻ 0 J(\theta) \succ0 J ( θ ) ≻ 0 . Since the complete-data score is centered Gaussian with covariance J ( θ ) J(\theta) J ( θ ) ,
J ( θ ) − 1 / 2 ∑ t = 1 N S t ( θ ) ∼ N ( 0 , I d ) . J(\theta)^{-1/2}\sum_{t=1}^{N}S_t(\theta) \sim\mathcal{N}(0,I_d). J ( θ ) − 1/2 t = 1 ∑ N S t ( θ ) ∼ N ( 0 , I d ) .
For every ∥ v ∥ 2 = 1 \|v\|_2 = 1 ∥ v ∥ 2 = 1 , we have
Var θ { v T J ( θ ) − 1 / 2 S t ( θ ) } = 1 σ 2 v T J ( θ ) − 1 / 2 B t ( θ ) T B t ( θ ) J ( θ ) − 1 / 2 v ≤ C C N , \operatorname{Var}_\theta\{v^{\mathsf{T}}J(\theta)^{-1/2}S_t(\theta)\} = \frac{1}{\sigma^2}v^{\mathsf{T}}J(\theta)^{-1/2}B_t(\theta)^{\mathsf{T}}B_t(\theta)J(\theta)^{-1/2}v \le\frac{C}{CN}, Var θ { v T J ( θ ) − 1/2 S t ( θ )} = σ 2 1 v T J ( θ ) − 1/2 B t ( θ ) T B t ( θ ) J ( θ ) − 1/2 v ≤ C N C ,
where the inequality follows from (81) and (85) . Since this random variable is Gaussian, its sub-Gaussian norm is bounded by a constant multiple of its standard deviation. Hence there exists an absolute constant C 1 > 0 C_1 > 0 C 1 > 0 such that
sup ∥ v ∥ 2 = 1 ∥ v T J ( θ ) − 1 / 2 ∑ t = 1 N S t ( θ ) ∥ ψ 2 ≤ C 1 , sup ∥ v ∥ 2 = 1 ∥ v T J ( θ ) − 1 / 2 S t ( θ ) ∥ ψ 2 ≤ C 1 N , ∀ t ∈ [ N ] . (88) \sup_{\|v\|_2=1}\left\|v^{\mathsf{T}}J(\theta)^{-1/2}\sum_{t=1}^{N}S_t(\theta)\right\|_{\psi_2}\le C_1,\qquad\sup_{\|v\|_2=1}\left\|v^{\mathsf{T}}J(\theta)^{-1/2}S_t(\theta)\right\|_{\psi_2}\le\frac{C_1}{\sqrt{N}},\quad\forall t\in[N]. \tag*{(88)} ∥ v ∥ 2 = 1 sup v T J ( θ ) − 1/2 t = 1 ∑ N S t ( θ ) ψ 2 ≤ C 1 , ∥ v ∥ 2 = 1 sup v T J ( θ ) − 1/2 S t ( θ ) ψ 2 ≤ N C 1 , ∀ t ∈ [ N ] . (88)
By Cai and Li [12 ] , together with (88) and rank { J ( θ ) } = d \operatorname{rank}\{J(\theta)\}=d rank { J ( θ )} = d , there exists an absolute constant C 2 > 0 C_2 > 0 C 2 > 0 such that
tr { J ( θ ) − 1 I Z ( θ ) } ≤ C 2 { N ( e ε − 1 ) 2 + C 1 δ d N log ( 1 / δ ) } . (89) \operatorname{tr}\{J(\theta)^{-1}I_Z(\theta)\}\le C_2\left\{N(\mathrm{e}^{\varepsilon}-1)^2+C_1\delta d\sqrt{N}\log(1/\delta)\right\}. \tag*{(89)} tr { J ( θ ) − 1 I Z ( θ )} ≤ C 2 { N ( e ε − 1 ) 2 + C 1 δ d N log ( 1/ δ ) } . (89)
Since 0 < ε ≤ ε 0 0<\varepsilon\le\varepsilon_0 0 < ε ≤ ε 0 , we have
( e ε − 1 ) 2 = ( ∫ 0 ε e t d t ) 2 ≤ e 2 ε 0 ε 2 . (90) (e^\varepsilon-1)^2=\left(\int_0^\varepsilon e^t\,\mathrm{d}t\right)^2\le e^{2\varepsilon_0}\varepsilon^2. \tag*{(90)} ( e ε − 1 ) 2 = ( ∫ 0 ε e t d t ) 2 ≤ e 2 ε 0 ε 2 . (90)
Combining (89) and (90) implies
tr { J ( θ ) − 1 I Z ( θ ) } ≤ C 2 { N e 2 ε 0 ε 2 + C 1 δ d N log ( 1 / δ ) } ≤ C 3 N ε 2 , (91) \operatorname{tr}\{J(\theta)^{-1}I_Z(\theta)\}\le C_2\left\{Ne^{2\varepsilon_0}\varepsilon^2+C_1\delta d\sqrt{N}\log(1/\delta)\right\}\le C_3N\varepsilon^2, \tag*{(91)} tr { J ( θ ) − 1 I Z ( θ )} ≤ C 2 { N e 2 ε 0 ε 2 + C 1 δ d N log ( 1/ δ ) } ≤ C 3 N ε 2 , (91)
where C 3 = C 2 e 2 ε 0 + C 2 C 1 c 0 C_3=C_2e^{2\varepsilon_0}+C_2C_1c_0 C 3 = C 2 e 2 ε 0 + C 2 C 1 c 0 is an absolute constant and the last inequality follows from (82) .
Step 4. Completion of the proof. Since I Z ( θ ) ⪰ 0 I_Z(\theta)\succeq0 I Z ( θ ) ⪰ 0 , J ( θ ) − 1 / 2 I Z ( θ ) J ( θ ) − 1 / 2 ⪰ 0 J(\theta)^{-1/2}I_Z(\theta)J(\theta)^{-1/2}\succeq0 J ( θ ) − 1/2 I Z ( θ ) J ( θ ) − 1/2 ⪰ 0 . Hence combining (85) and (91) , we have
tr { I Z ( θ ) } = tr { J ( θ ) J ( θ ) − 1 / 2 I Z ( θ ) J ( θ ) − 1 / 2 } ≤ λ 1 { J ( θ ) } tr { J ( θ ) − 1 I Z ( θ ) } ≤ C C 3 N 2 ε 2 λ σ 2 . (92) \begin{aligned}
\operatorname{tr}\{I_Z(\theta)\}=\operatorname{tr}\{J(\theta)J(\theta)^{-1/2}I_Z(\theta)J(\theta)^{-1/2}\} \\
&\le\lambda_1\{J(\theta)\}\operatorname{tr}\{J(\theta)^{-1}I_Z(\theta)\}\le CC_3\frac{N^2\varepsilon^2\lambda}{\sigma^2}. \tag*{(92)}
\end{aligned} tr { I Z ( θ )} = tr { J ( θ ) J ( θ ) − 1/2 I Z ( θ ) J ( θ ) − 1/2 } ≤ λ 1 { J ( θ )} tr { J ( θ ) − 1 I Z ( θ )} ≤ C C 3 σ 2 N 2 ε 2 λ . (92)
For the non-private bound, combining (84) and (87) , we have
E θ [ Var θ { ∑ t = 1 N S t ( θ ) ∣ Z } ] = E θ { ( ∑ t = 1 N S t ( θ ) ) ( ∑ t = 1 N S t ( θ ) ) T } \mathbb{E}_{\theta}\left[\operatorname{Var}_{\theta}\left\{\left.\sum_{t=1}^{N}S_t(\theta)\right|Z\right\}\right]
=\mathbb{E}_{\theta}\left\{\left(\sum_{t=1}^{N}S_t(\theta)\right)\left(\sum_{t=1}^{N}S_t(\theta)\right)^{\mathsf{T}}\right\} E θ [ Var θ { t = 1 ∑ N S t ( θ ) Z } ] = E θ ⎩ ⎨ ⎧ ( t = 1 ∑ N S t ( θ ) ) ( t = 1 ∑ N S t ( θ ) ) T ⎭ ⎬ ⎫
− E θ [ E θ { ∑ t = 1 N S t ( θ ) ∣ Z } E θ { ∑ t = 1 N S t ( θ ) ∣ Z } T ] = J ( θ ) − I Z ( θ ) ⪰ 0. -\mathbb{E}_{\theta}\left[\mathbb{E}_{\theta}\left\{\left.\sum_{t=1}^{N}S_t(\theta)\right|Z\right\}\mathbb{E}_{\theta}\left\{\left.\sum_{t=1}^{N}S_t(\theta)\right|Z\right\}^{\mathsf{T}}\right]
=J(\theta)-I_Z(\theta)\succeq0. − E θ E θ { t = 1 ∑ N S t ( θ ) Z } E θ { t = 1 ∑ N S t ( θ ) Z } T = J ( θ ) − I Z ( θ ) ⪰ 0.
Thus, it follows that
tr { I Z ( θ ) } ≤ tr { J ( θ ) } ≤ C N d λ σ 2 , (93) \operatorname{tr}\{I_Z(\theta)\}\le\operatorname{tr}\{J(\theta)\}\le C\frac{Nd\lambda}{\sigma^2}, \tag*{(93)} tr { I Z ( θ )} ≤ tr { J ( θ )} ≤ C σ 2 N d λ , (93)
where the last inequality follows from (85) . Combining (92) and (93) , we conclude (83) , which completes this proof.
Proofs of theorems and corollaries
Proof of Theorem 1. Let P : = A k A k T P:=A_kA_k^{\mathsf{T}} P := A k A k T and let Π ~ j , k \widetilde{\Pi}_{j,k} Π j , k be the projector constructed in Lemma 4. Recall from (11) that w j , k ≥ 0 w_{j,k}\ge0 w j , k ≥ 0 and ∑ j = 1 m w j , k = 1 \sum_{j=1}^{m}w_{j,k}=1 ∑ j = 1 m w j , k = 1 . Define the population and random aggregation matrices as
M k : = ∑ j = 1 m w j , k E ( Π ~ j , k ) , M ~ k : = ∑ j = 1 m w j , k Π ~ j , k . (94) M_k:=\sum_{j=1}^{m}w_{j,k}\mathbb{E}(\widetilde{\Pi}_{j,k}),\qquad\widetilde{M}_k:=\sum_{j=1}^{m}w_{j,k}\widetilde{\Pi}_{j,k}. \tag*{(94)} M k := j = 1 ∑ m w j , k E ( Π j , k ) , M k := j = 1 ∑ m w j , k Π j , k . (94)
We first identify the leading eigenspace of M k M_k M k . By Lemma 4 (iii), we have
( I p k − P ) M k P = ∑ j = 1 m w j , k ( I p k − P ) E ( Π ~ j , k ) P = 0. (95) (I_{p_k}-P)M_kP=\sum_{j=1}^{m}w_{j,k}(I_{p_k}-P)\mathbb{E}(\widetilde{\Pi}_{j,k})P=0. \tag*{(95)} ( I p k − P ) M k P = j = 1 ∑ m w j , k ( I p k − P ) E ( Π j , k ) P = 0. (95)
Since M k M_k M k is symmetric, (95) also gives P M k ( I p k − P ) = 0 PM_k(I_{p_k}-P)=0 P M k ( I p k − P ) = 0 . Hence M k M_k M k is block diagonal with respect to span ( A k ) ⊕ span ( A k ) ⊥ \operatorname{span}(A_k)\oplus\operatorname{span}(A_k)^\perp span ( A k ) ⊕ span ( A k ) ⊥ . Also, by Lemma 4 (iv), we have
∥ M k − P ∥ ≤ ∑ j = 1 m w j , k ∥ E ( Π ~ j , k ) − P ∥ ≤ c . (96) \lVert M_k-P\rVert\le\sum_{j=1}^{m}w_{j,k}\lVert\mathbb{E}(\widetilde{\Pi}_{j,k})-P\rVert\le c. \tag*{(96)} ∥ M k − P ∥ ≤ j = 1 ∑ m w j , k ∥ E ( Π j , k ) − P ∥ ≤ c . (96)
For every unit vector u ∈ span ( A k ) u \in\operatorname{span}(A_k) u ∈ span ( A k ) and every unit vector v ∈ span ( A k ) ⊥ v \in\operatorname{span}(A_k)^\perp v ∈ span ( A k ) ⊥ , (96) gives u ⊤ M k u ≥ 1 − c u^\top M_k u \ge1-c u ⊤ M k u ≥ 1 − c and v ⊤ M k v ≤ c v^\top M_kv \le c v ⊤ M k v ≤ c . Consequently, if the absolute constant c c c in Lemma 4 (iv) is sufficiently small, the leading r k r_k r k -dimensional eigenspace of M k M_k M k is exactly span ( A k ) \operatorname{span}(A_k) span ( A k ) and
λ r k ( M k ) − λ r k + 1 ( M k ) ≥ 1 − 2 c ≥ 1 2 . (97) \lambda_{r_k}(M_k)-\lambda_{r_k+1}(M_k) \ge1-2c \ge\frac{1}{2}. \tag*{(97)} λ r k ( M k ) − λ r k + 1 ( M k ) ≥ 1 − 2 c ≥ 2 1 . (97)
Consider M ~ k − M k \widetilde{M}_k-M_k M k − M k next. For j ∈ [ m ] j \in[m] j ∈ [ m ] , set Y j , k : = w j , k { Π ~ j , k − E ( Π ~ j , k ) } Y_{j,k}:=w_{j,k}\{\widetilde{\Pi}_{j,k}-\mathbb{E}(\widetilde{\Pi}_{j,k})\} Y j , k := w j , k { Π j , k − E ( Π j , k )} . Then by Lemma 4 (i), the matrices Y 1 , k , … , Y m , k Y_{1,k},\ldots,Y_{m,k} Y 1 , k , … , Y m , k are independent, and E ( Y j , k ) = 0 \mathbb{E}(Y_{j,k})=0 E ( Y j , k ) = 0 . Lemma 4 (v) gives the almost-sure bound such that
∥ Y j , k ∥ F ≤ C w j , k r k Ψ j , k l o c . (98) \lVert Y_{j,k}\rVert_F \le Cw_{j,k}\sqrt{r_k}\Psi_{j,k}^{\mathrm{loc}}. \tag*{(98)} ∥ Y j , k ∥ F ≤ C w j , k r k Ψ j , k loc . (98)
By similar arguments for (67) and (68) , applying Theorem 3.5 of Pinelis [44 ] in the space of symmetric matrices equipped with the Frobenius norm to (98) , for every x > 0 x>0 x > 0 , we obtain
P { ∥ M ~ k − M k ∥ F 2 > C r k x ∑ j = 1 m w j , k 2 ( Ψ j , k l o c ) 2 } ≤ 2 e − x . (99) \mathbb{P}\left\{\lVert\widetilde{M}_k-M_k\rVert_F^2>Cr_kx\sum_{j=1}^{m}w_{j,k}^2(\Psi_{j,k}^{\mathrm{loc}})^2\right\}\le2e^{-x}. \tag*{(99)} P { ∥ M k − M k ∥ F 2 > C r k x j = 1 ∑ m w j , k 2 ( Ψ j , k loc ) 2 } ≤ 2 e − x . (99)
By the definition of the weights in (11) , we have
∑ j = 1 m w j , k 2 ( Ψ j , k l o c ) 2 = ∑ j = 1 m ( Ψ j , k l o c ) − 4 ( Ψ j , k l o c ) 2 { ∑ j = 1 m ( Ψ j , k l o c ) − 2 } 2 = 1 ∑ j = 1 m ( Ψ j , k l o c ) − 2 . (100) \sum_{j=1}^{m}w_{j,k}^2(\Psi_{j,k}^{\mathrm{loc}})^2
=\frac{\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-4}(\Psi_{j,k}^{\mathrm{loc}})^2}{\left\{\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-2}\right\}^2}
=\frac{1}{\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-2}}. \tag*{(100)} j = 1 ∑ m w j , k 2 ( Ψ j , k loc ) 2 = { ∑ j = 1 m ( Ψ j , k loc ) − 2 } 2 ∑ j = 1 m ( Ψ j , k loc ) − 4 ( Ψ j , k loc ) 2 = ∑ j = 1 m ( Ψ j , k loc ) − 2 1 . (100)
Setting x = log ( n ∨ p ) x=\log(n\vee p) x = log ( n ∨ p ) in (99) , together with (100) , it then holds with probability at least 1 − 2 ( n ∨ p ) − 1 1-2(n\vee p)^{-1} 1 − 2 ( n ∨ p ) − 1 that
∥ M ~ k − M k ∥ F 2 ≤ C r k log ( n ∨ p ) ∑ j = 1 m ( Ψ j , k l o c ) − 2 . (101) \lVert\widetilde{M}_k-M_k\rVert_F^2\le\frac{Cr_k\log(n\vee p)}{\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-2}}. \tag*{(101)} ∥ M k − M k ∥ F 2 ≤ ∑ j = 1 m ( Ψ j , k loc ) − 2 C r k log ( n ∨ p ) . (101)
Let A ~ k : = SVD r k ( M ~ k ) \widetilde{A}_k:=\operatorname{SVD}_{r_k}(\widetilde{M}_k) A k := SVD r k ( M k ) and P ~ k : = A ~ k A ~ k ⊤ \widetilde{P}_k:=\widetilde{A}_k\widetilde{A}_k^\top P k := A k A k ⊤ . The population eigengap in (97) and the Davis–Kahan theorem (Yu et al., 2015, Theorem 2) imply
∥ P ~ k − P ∥ F 2 ≤ C ∥ M ~ k − M k ∥ F 2 . (102) \lVert\widetilde{P}_k-P\rVert_F^2\le C\lVert\widetilde{M}_k-M_k\rVert_F^2. \tag*{(102)} ∥ P k − P ∥ F 2 ≤ C ∥ M k − M k ∥ F 2 . (102)
Combining (101) and (102) , it holds with probability at least 1 − 2 ( n ∨ p ) − 1 1-2(n\vee p)^{-1} 1 − 2 ( n ∨ p ) − 1 that
∥ P ~ k − P ∥ F 2 ≤ C r k log ( n ∨ p ) ∑ j = 1 m ( Ψ j , k l o c ) − 2 . (103) \lVert\widetilde{P}_k-P\rVert_F^2\le\frac{Cr_k\log(n\vee p)}{\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-2}}. \tag*{(103)} ∥ P k − P ∥ F 2 ≤ ∑ j = 1 m ( Ψ j , k loc ) − 2 C r k log ( n ∨ p ) . (103)
In addition, note that for any two rank-r k r_k r k orthogonal projectors P 1 P_1 P 1 and P 2 P_2 P 2 , we have ∥ P 1 − P 2 ∥ F 2 = 2 r k − 2 tr ( P 1 P 2 ) ≤ 2 r k \lVert P_1-P_2\rVert_F^2=2r_k-2\operatorname{tr}(P_1P_2)\le2r_k ∥ P 1 − P 2 ∥ F 2 = 2 r k − 2 tr ( P 1 P 2 ) ≤ 2 r k . Hence the right-hand side of (103) may be replaced by its minimum with 2 r k 2r_k 2 r k . It remains to connect the above result to our procedure in Algorithm 1. To this end, define the matrix aggregated by the server as M ~ k : = ∑ j = 1 m w j , k U ~ j , k U ~ j , k ⊤ \widetilde{M}_k:=\sum_{j=1}^{m}w_{j,k}\widetilde{U}_{j,k}\widetilde{U}_{j,k}^{\top} M k := ∑ j = 1 m w j , k U j , k U j , k ⊤ . By Lemma 4 (ii), it holds with probability at least 1 − C ( n ∨ p ) − c 1-C(n\vee p)^{-c} 1 − C ( n ∨ p ) − c that
Π ~ j , k = U ~ j , k U ~ j , k ⊤ , ∀ j ∈ [ m ] . \widetilde{\Pi}_{j,k}=\widetilde{U}_{j,k}\widetilde{U}_{j,k}^{\top},\qquad\forall j\in[m]. Π j , k = U j , k U j , k ⊤ , ∀ j ∈ [ m ] .
Note that on this event, M ~ k = M ^ k \widetilde{M}_k=\widehat{M}_k M k = M k . Hence by the fixed tie-breaking rule stated in Section 1.3, P ~ k = A ^ k f e d ( A ^ k f e d ) ⊤ \widetilde{P}_k=\widehat{A}_k^{\mathrm{fed}}(\widehat{A}_k^{\mathrm{fed}})^\top P k = A k fed ( A k fed ) ⊤ . Combining this event with (103) by a union bound, we obtain that, with probability at least 1 − 2 ( n ∨ p ) − 1 − C ( n ∨ p ) − c 1-2(n\vee p)^{-1}-C(n\vee p)^{-c} 1 − 2 ( n ∨ p ) − 1 − C ( n ∨ p ) − c ,
∥ A ^ k f e d ( A ^ k f e d ) ⊤ − A k A k ⊤ ∥ F 2 ≤ C r k log ( n ∨ p ) ∑ j = 1 m ( Ψ j , k l o c ) − 2 ∧ ( 2 r k ) , \left\lVert\widehat{A}_k^{\mathrm{fed}}(\widehat{A}_k^{\mathrm{fed}})^\top-A_kA_k^\top\right\rVert_F^2
\le\frac{Cr_k\log(n\vee p)}{\sum_{j=1}^{m}(\Psi_{j,k}^{\mathrm{loc}})^{-2}}\wedge(2r_k), A k fed ( A k fed ) ⊤ − A k A k ⊤ F 2 ≤ ∑ j = 1 m ( Ψ j , k loc ) − 2 C r k log ( n ∨ p ) ∧ ( 2 r k ) ,
which is (14) . This completes the proof of this theorem.
Proof of Theorem 2. Our proof proceeds in three steps. First, we construct an r k ( p k − r k ) r_k(p_k-r_k) r k ( p k − r k ) -dimensional Gaussian sub-model in which only the mode-k k k loading space varies. Second, we bound the Fisher information retained in each private client message and aggregate these bounds over all clients. Finally, we apply the multivariate van Trees inequality and derive the desired minimax lower bound.
Step 1. In this step, we construct a r k ( p k − r k ) r_k(p_k-r_k) r k ( p k − r k ) -dimensional Gaussian sub-model that remains inside P k f e d ( λ , σ 2 ) \mathcal{P}^{\mathrm{fed}}_k(\lambda,\sigma^2) P k fed ( λ , σ 2 ) . For each h ∈ [ K ] ∖ { k } h \in[K]\setminus\{k\} h ∈ [ K ] ∖ { k } , fix A h ∈ O p h × r h A_h \in\mathbb{O}_{p_h\times r_h} A h ∈ O p h × r h satisfying Assumption 1, and recall A − k = ⨂ h ∈ [ K ] ∖ { k } A h A_{-k}=\bigotimes_{h\in[K]\setminus\{k\}} A_h A − k = ⨂ h ∈ [ K ] ∖ { k } A h . Since A h T A h = I r h A_h^{\mathsf{T}}A_h=I_{r_h} A h T A h = I r h for every h ∈ [ K ] h\in[K] h ∈ [ K ] , we have
A − k T A − k = ⨂ h ∈ [ K ] ∖ { k } A h T A h = I r − k . A_{-k}^{\mathsf{T}}A_{-k}=\bigotimes_{h\in[K]\setminus\{k\}}A_h^{\mathsf{T}}A_h=I_{r_{-k}}. A − k T A − k = h ∈ [ K ] ∖ { k } ⨂ A h T A h = I r − k .
We next construct a local family of mode-k k k loading matrices. For any H ∈ R ( p k − r k ) × r k H\in\mathbb{R}^{(p_k-r_k)\times r_k} H ∈ R ( p k − r k ) × r k , define
A k ( H ) : = ( I r k H ) ( I r k + H T H ) − 1 / 2 , (104) A_k(H):=
\begin{pmatrix}
I_{r_k}\\
H
\end{pmatrix}
\left(I_{r_k}+H^{\mathsf{T}}H\right)^{-1/2}, \tag*{(104)} A k ( H ) := ( I r k H ) ( I r k + H T H ) − 1/2 , (104)
Since A k ( H ) T A k ( H ) = I r k A_k(H)^{\mathsf{T}}A_k(H)=I_{r_k} A k ( H ) T A k ( H ) = I r k , we have that A k ( H ) ∈ O p k × r k A_k(H)\in\mathbb{O}_{p_k\times r_k} A k ( H ) ∈ O p k × r k for every H ∈ R ( p k − r k ) × r k H\in\mathbb{R}^{(p_k-r_k)\times r_k} H ∈ R ( p k − r k ) × r k . Let P k ( H ) = A k ( H ) A k ( H ) T P_k(H)=A_k(H)A_k(H)^{\mathsf{T}} P k ( H ) = A k ( H ) A k ( H ) T and d k = r k ( p k − r k ) d_k=r_k(p_k-r_k) d k = r k ( p k − r k ) , where d k d_k d k is the number of free coordinates in H H H .
Step 1.1. In this sub-step, we show that there exist absolute constants C 1 ′ ≥ c 1 ′ > 0 C'_1\ge c'_1>0 C 1 ′ ≥ c 1 ′ > 0 such that, for all H , H ′ ∈ R ( p k − r k ) × r k H,H'\in\mathbb{R}^{(p_k-r_k)\times r_k} H , H ′ ∈ R ( p k − r k ) × r k satisfying ∥ H ∥ F , ∥ H ′ ∥ F ≤ 1 \|H\|_{\mathrm{F}},\|H'\|_{\mathrm{F}}\le1 ∥ H ∥ F , ∥ H ′ ∥ F ≤ 1 ,
c 1 ′ ∥ H − H ′ ∥ F ≤ ∥ P k ( H ) − P k ( H ′ ) ∥ F ≤ C 1 ′ ∥ H − H ′ ∥ F . (105) c'_1\|H-H'\|_{\mathrm{F}}\le\|P_k(H)-P_k(H')\|_{\mathrm{F}}\le C'_1\|H-H'\|_{\mathrm{F}}. \tag*{(105)} c 1 ′ ∥ H − H ′ ∥ F ≤ ∥ P k ( H ) − P k ( H ′ ) ∥ F ≤ C 1 ′ ∥ H − H ′ ∥ F . (105)
Fix H , H ′ ∈ R ( p k − r k ) × r k H,H'\in\mathbb{R}^{(p_k-r_k)\times r_k} H , H ′ ∈ R ( p k − r k ) × r k with ∥ H ∥ F , ∥ H ′ ∥ F ≤ 1 \|H\|_{\mathrm{F}},\|H'\|_{\mathrm{F}}\le1 ∥ H ∥ F , ∥ H ′ ∥ F ≤ 1 . Let
X ( H ) : = ( I r k H ) and R ( H ) : = ( I r k + H T H ) − 1 / 2 . X(H):=
\begin{pmatrix}
I_{r_k}\\
H
\end{pmatrix}
\quad\text{and}\quad
R(H):=\left(I_{r_k}+H^{\mathsf{T}}H\right)^{-1/2}. X ( H ) := ( I r k H ) and R ( H ) := ( I r k + H T H ) − 1/2 .
Then A k ( H ) = X ( H ) R ( H ) A_k(H)=X(H)R(H) A k ( H ) = X ( H ) R ( H ) and
P k ( H ) = ( R ( H ) 2 R ( H ) 2 H T H R ( H ) 2 H R ( H ) 2 H T ) . (106) P_k(H)=
\begin{pmatrix}
R(H)^2 & R(H)^2H^{\mathsf{T}}\\
HR(H)^2 & HR(H)^2H^{\mathsf{T}}
\end{pmatrix}. \tag*{(106)} P k ( H ) = ( R ( H ) 2 H R ( H ) 2 R ( H ) 2 H T H R ( H ) 2 H T ) . (106)
Since R ( H ) 2 = ( I r k + H T H ) − 1 R(H)^2=(I_{r_k}+H^{\mathsf{T}}H)^{-1} R ( H ) 2 = ( I r k + H T H ) − 1 and R ( H ′ ) 2 = ( I r k + H ′ T H ′ ) − 1 R(H')^2=(I_{r_k}+H'^{\mathsf{T}}H')^{-1} R ( H ′ ) 2 = ( I r k + H ′ T H ′ ) − 1 ,
∥ R ( H ) − 2 ∥ ≤ 2 , ∥ R ( H ′ ) − 2 ∥ ≤ 2 , ∥ R ( H ) 2 ∥ ≤ 1 , ∥ H ′ R ( H ′ ) 2 ∥ ≤ ∥ H ′ ∥ ∥ R ( H ′ ) 2 ∥ ≤ 1. (107) \|R(H)^{-2}\|\le2,\qquad
\|R(H')^{-2}\|\le2,\qquad
\|R(H)^2\|\le1,\qquad
\|H'R(H')^2\|\le\|H'\|\|R(H')^2\|\le1. \tag*{(107)} ∥ R ( H ) − 2 ∥ ≤ 2 , ∥ R ( H ′ ) − 2 ∥ ≤ 2 , ∥ R ( H ) 2 ∥ ≤ 1 , ∥ H ′ R ( H ′ ) 2 ∥ ≤ ∥ H ′ ∥∥ R ( H ′ ) 2 ∥ ≤ 1. (107)
Moreover,
∥ R ( H ) − 2 − R ( H ′ ) − 2 ∥ F = ∥ R ( H ) − 2 { R ( H ′ ) 2 − R ( H ) 2 } R ( H ′ ) − 2 ∥ F ≤ ∥ R ( H ) − 2 ∥ ∥ R ( H ′ ) 2 − R ( H ) 2 ∥ F ∥ R ( H ′ ) − 2 ∥ ≤ 4 ∥ R ( H ′ ) 2 − R ( H ) 2 ∥ F , (108) \begin{aligned}
\|R(H)^{-2}-R(H')^{-2}\|_{\mathrm{F}}
&=\|R(H)^{-2}\{R(H')^2-R(H)^2\}R(H')^{-2}\|_{\mathrm{F}}\\
&\le\|R(H)^{-2}\|\|R(H')^2-R(H)^2\|_{\mathrm{F}}\|R(H')^{-2}\|\\
&\le4\|R(H')^2-R(H)^2\|_{\mathrm{F}}, \tag*{(108)}
\end{aligned} ∥ R ( H ) − 2 − R ( H ′ ) − 2 ∥ F = ∥ R ( H ) − 2 { R ( H ′ ) 2 − R ( H ) 2 } R ( H ′ ) − 2 ∥ F ≤ ∥ R ( H ) − 2 ∥∥ R ( H ′ ) 2 − R ( H ) 2 ∥ F ∥ R ( H ′ ) − 2 ∥ ≤ 4∥ R ( H ′ ) 2 − R ( H ) 2 ∥ F , (108)
where the last inequality follows from (107). Therefore, we have
∥ H − H ′ ∥ F = ∥ ( H R ( H ) 2 − H ′ R ( H ′ ) 2 ) R ( H ) − 2 + H ′ R ( H ′ ) 2 { R ( H ) − 2 − R ( H ′ ) − 2 } ∥ F ≤ ∥ R ( H ) − 2 ∥ ∥ H R ( H ) 2 − H ′ R ( H ′ ) 2 ∥ F + ∥ H ′ R ( H ′ ) 2 ∥ ∥ R ( H ) − 2 − R ( H ′ ) − 2 ∥ F ≤ 2 ∥ H R ( H ) 2 − H ′ R ( H ′ ) 2 ∥ F + 4 ∥ R ( H ) 2 − R ( H ′ ) 2 ∥ F ≤ 6 ∥ P k ( H ) − P k ( H ′ ) ∥ F . (109) \begin{aligned}
\|H-H'\|_{\mathrm{F}}
&=\|(HR(H)^2-H'R(H')^2)R(H)^{-2}+H'R(H')^2\{R(H)^{-2}-R(H')^{-2}\}\|_{\mathrm{F}}\\
&\le\|R(H)^{-2}\|\|HR(H)^2-H'R(H')^2\|_{\mathrm{F}}+\|H'R(H')^2\|\|R(H)^{-2}-R(H')^{-2}\|_{\mathrm{F}}\\
&\le2\|HR(H)^2-H'R(H')^2\|_{\mathrm{F}}+4\|R(H)^2-R(H')^2\|_{\mathrm{F}}\\
&\le6\|P_k(H)-P_k(H')\|_{\mathrm{F}}. \tag*{(109)}
\end{aligned} ∥ H − H ′ ∥ F = ∥ ( H R ( H ) 2 − H ′ R ( H ′ ) 2 ) R ( H ) − 2 + H ′ R ( H ′ ) 2 { R ( H ) − 2 − R ( H ′ ) − 2 } ∥ F ≤ ∥ R ( H ) − 2 ∥∥ H R ( H ) 2 − H ′ R ( H ′ ) 2 ∥ F + ∥ H ′ R ( H ′ ) 2 ∥∥ R ( H ) − 2 − R ( H ′ ) − 2 ∥ F ≤ 2∥ H R ( H ) 2 − H ′ R ( H ′ ) 2 ∥ F + 4∥ R ( H ) 2 − R ( H ′ ) 2 ∥ F ≤ 6∥ P k ( H ) − P k ( H ′ ) ∥ F . (109)
where the second inequality follows from (107) and (108) , and the last inequality follows from (106) .
For any positive definite matrices U , V ⪰ I r k U, V \succeq I_{r_k} U , V ⪰ I r k , the integral representation of the inverse square root gives
∥ U − 1 / 2 − V − 1 / 2 ∥ F = ∥ 1 π ∫ 0 ∞ t − 1 / 2 ( U + t I r k ) − 1 d t − 1 π ∫ 0 ∞ t − 1 / 2 ( V + t I r k ) − 1 d t ∥ F = ∥ 1 π ∫ 0 ∞ t − 1 / 2 ( U + t I r k ) − 1 ( V − U ) ( V + t I r k ) − 1 d t ∥ F ≤ 1 π ∫ 0 ∞ t − 1 / 2 ∥ ( U + t I r k ) − 1 ∥ ∥ V − U ∥ F ∥ ( V + t I r k ) − 1 ∥ d t ≤ 1 π ∫ 0 ∞ t − 1 / 2 ( 1 + t ) 2 d t ∥ U − V ∥ F = 1 2 ∥ U − V ∥ F . (110) \begin{aligned}
\lVert U^{-1/2}-V^{-1/2}\rVert_{\mathrm{F}}
&=\left\lVert\frac{1}{\pi}\int_{0}^{\infty}t^{-1/2}(U+tI_{r_k})^{-1}\,dt-\frac{1}{\pi}\int_{0}^{\infty}t^{-1/2}(V+tI_{r_k})^{-1}\,dt\right\rVert_{\mathrm{F}}\\
&=\left\lVert\frac{1}{\pi}\int_{0}^{\infty}t^{-1/2}(U+tI_{r_k})^{-1}(V-U)(V+tI_{r_k})^{-1}\,dt\right\rVert_{\mathrm{F}}\\
&\leq\frac{1}{\pi}\int_{0}^{\infty}t^{-1/2}\lVert(U+tI_{r_k})^{-1}\rVert\lVert V-U\rVert_{\mathrm{F}}\lVert(V+tI_{r_k})^{-1}\rVert\,dt\\
&\leq\frac{1}{\pi}\int_{0}^{\infty}\frac{t^{-1/2}}{(1+t)^2}\,dt\lVert U-V\rVert_{\mathrm{F}}=\frac{1}{2}\lVert U-V\rVert_{\mathrm{F}}. \tag*{(110)}
\end{aligned} ∥ U − 1/2 − V − 1/2 ∥ F = π 1 ∫ 0 ∞ t − 1/2 ( U + t I r k ) − 1 d t − π 1 ∫ 0 ∞ t − 1/2 ( V + t I r k ) − 1 d t F = π 1 ∫ 0 ∞ t − 1/2 ( U + t I r k ) − 1 ( V − U ) ( V + t I r k ) − 1 d t F ≤ π 1 ∫ 0 ∞ t − 1/2 ∥( U + t I r k ) − 1 ∥ ∥ V − U ∥ F ∥( V + t I r k ) − 1 ∥ d t ≤ π 1 ∫ 0 ∞ ( 1 + t ) 2 t − 1/2 d t ∥ U − V ∥ F = 2 1 ∥ U − V ∥ F . (110)
where the second inequality follows from U , V ⪰ I r k U,V\succeq I_{r_k} U , V ⪰ I r k , which implies that
∥ ( U + t I r k ) − 1 ∥ ≤ 1 1 + t and ∥ ( V + t I r k ) − 1 ∥ ≤ 1 1 + t . \lVert(U+tI_{r_k})^{-1}\rVert\leq\frac{1}{1+t}\quad\text{and}\quad\lVert(V+tI_{r_k})^{-1}\rVert\leq\frac{1}{1+t}. ∥( U + t I r k ) − 1 ∥ ≤ 1 + t 1 and ∥( V + t I r k ) − 1 ∥ ≤ 1 + t 1 .
Applying (110) with U = I r k + H T H U=I_{r_k}+H^{\mathsf{T}}H U = I r k + H T H and V = I r k + H ′ T H ′ V=I_{r_k}+H'^{\mathsf{T}}H' V = I r k + H ′ T H ′ gives
∥ R ( H ) − R ( H ′ ) ∥ F ≤ 1 2 ∥ H T H − H ′ T H ′ ∥ F = 1 2 ∥ H T ( H − H ′ ) + ( H − H ′ ) T H ′ ∥ F ≤ 1 2 { ∥ H ∥ + ∥ H ′ ∥ } ∥ H − H ′ ∥ F ≤ ∥ H − H ′ ∥ F , (111) \begin{aligned}
\lVert R(H)-R(H')\rVert_{\mathrm{F}}
&\leq\frac{1}{2}\lVert H^{\mathsf{T}}H-H'^{\mathsf{T}}H'\rVert_{\mathrm{F}}\\
&=\frac{1}{2}\lVert H^{\mathsf{T}}(H-H')+(H-H')^{\mathsf{T}}H'\rVert_{\mathrm{F}}\\
&\leq\frac{1}{2}\{\lVert H\rVert+\lVert H'\rVert\}\lVert H-H'\rVert_{\mathrm{F}}\leq\lVert H-H'\rVert_{\mathrm{F}}, \tag*{(111)}
\end{aligned} ∥ R ( H ) − R ( H ′ ) ∥ F ≤ 2 1 ∥ H T H − H ′ T H ′ ∥ F = 2 1 ∥ H T ( H − H ′ ) + ( H − H ′ ) T H ′ ∥ F ≤ 2 1 {∥ H ∥ + ∥ H ′ ∥} ∥ H − H ′ ∥ F ≤ ∥ H − H ′ ∥ F , (111)
where the last inequality follows from ∥ H ∥ , ∥ H ′ ∥ ≤ 1 \lVert H\rVert,\lVert H'\rVert\leq1 ∥ H ∥ , ∥ H ′ ∥ ≤ 1 . Moreover, since
X ( H ) − X ( H ′ ) = ( 0 H − H ′ ) , X(H)-X(H')=\begin{pmatrix}0\\H-H'\end{pmatrix}, X ( H ) − X ( H ′ ) = ( 0 H − H ′ ) ,
we have
∥ X ( H ) − X ( H ′ ) ∥ F = ∥ H − H ′ ∥ F , (112) \lVert X(H)-X(H')\rVert_{\mathrm{F}}=\lVert H-H'\rVert_{\mathrm{F}}, \tag*{(112)} ∥ X ( H ) − X ( H ′ ) ∥ F = ∥ H − H ′ ∥ F , (112)
∥ X ( H ′ ) ∥ 2 = ∥ X ( H ′ ) T X ( H ′ ) ∥ = ∥ I r k + H ′ T H ′ ∥ ≤ 2. (113) \lVert X(H')\rVert^2=\lVert X(H')^{\mathsf{T}}X(H')\rVert=\lVert I_{r_k}+H'^{\mathsf{T}}H'\rVert\leq2. \tag*{(113)} ∥ X ( H ′ ) ∥ 2 = ∥ X ( H ′ ) T X ( H ′ )∥ = ∥ I r k + H ′ T H ′ ∥ ≤ 2. (113)
Hence altogether, we arrive at
∥ P k ( H ) − P k ( H ′ ) ∥ F = ∥ { A k ( H ) − A k ( H ′ ) } A k ( H ) T + A k ( H ′ ) { A k ( H ) − A k ( H ′ ) } T ∥ F ≤ ( ∥ A k ( H ) ∥ + ∥ A k ( H ′ ) ∥ ) ∥ A k ( H ) − A k ( H ′ ) ∥ F = 2 ∥ A k ( H ) − A k ( H ′ ) ∥ F = 2 ∥ { X ( H ) − X ( H ′ ) } R ( H ) + X ( H ′ ) { R ( H ) − R ( H ′ ) } ∥ F ≤ 2 ∥ X ( H ) − X ( H ′ ) ∥ F ∥ R ( H ) ∥ + 2 ∥ X ( H ′ ) ∥ ∥ R ( H ) − R ( H ′ ) ∥ F ≤ 2 ( 1 + 2 ) ∥ H − H ′ ∥ F , (114) \begin{aligned}
\lVert P_k(H)-P_k(H')\rVert_{\mathrm{F}}
&=\lVert\{A_k(H)-A_k(H')\}A_k(H)^{\mathsf{T}}+A_k(H')\{A_k(H)-A_k(H')\}^{\mathsf{T}}\rVert_{\mathrm{F}}\\
&\leq(\lVert A_k(H)\rVert+\lVert A_k(H')\rVert)\lVert A_k(H)-A_k(H')\rVert_{\mathrm{F}}=2\lVert A_k(H)-A_k(H')\rVert_{\mathrm{F}}\\
&=2\lVert\{X(H)-X(H')\}R(H)+X(H')\{R(H)-R(H')\}\rVert_{\mathrm{F}}\\
&\leq2\lVert X(H)-X(H')\rVert_{\mathrm{F}}\lVert R(H)\rVert+2\lVert X(H')\rVert\lVert R(H)-R(H')\rVert_{\mathrm{F}}\\
&\leq2(1+\sqrt{2})\lVert H-H'\rVert_{\mathrm{F}}, \tag*{(114)}
\end{aligned} ∥ P k ( H ) − P k ( H ′ ) ∥ F = ∥{ A k ( H ) − A k ( H ′ )} A k ( H ) T + A k ( H ′ ) { A k ( H ) − A k ( H ′ ) } T ∥ F ≤ (∥ A k ( H )∥ + ∥ A k ( H ′ )∥) ∥ A k ( H ) − A k ( H ′ ) ∥ F = 2 ∥ A k ( H ) − A k ( H ′ ) ∥ F = 2 ∥{ X ( H ) − X ( H ′ )} R ( H ) + X ( H ′ ) { R ( H ) − R ( H ′ )} ∥ F ≤ 2 ∥ X ( H ) − X ( H ′ ) ∥ F ∥ R ( H )∥ + 2 ∥ X ( H ′ )∥ ∥ R ( H ) − R ( H ′ ) ∥ F ≤ 2 ( 1 + 2 ) ∥ H − H ′ ∥ F , (114)
where the second equality follows from A k ( H ) , A k ( H ′ ) ∈ O p k × r k A_k(H),A_k(H')\in\mathbb{O}_{p_k\times r_k} A k ( H ) , A k ( H ′ ) ∈ O p k × r k , and the last inequality follows from (111) , (112) , (113) and ∥ R ( H ) ∥ = ∥ ( I r k + H T H ) − 1 / 2 ∥ ≤ 1 \lVert R(H)\rVert=\lVert(I_{r_k}+H^{\mathsf{T}}H)^{-1/2}\rVert\leq1 ∥ R ( H )∥ = ∥( I r k + H T H ) − 1/2 ∥ ≤ 1 . Combining (109) and (114) , we complete the proof of (105) with constants c 1 ′ = 1 / 6 c'_1=1/6 c 1 ′ = 1/6 and C 1 ′ = 2 ( 1 + 2 ) C'_1=2(1+\sqrt{2}) C 1 ′ = 2 ( 1 + 2 ) .
Step 1.2. For H , G ∈ R ( p k − r k ) × r k H,G\in\mathbb{R}^{(p_k-r_k)\times r_k} H , G ∈ R ( p k − r k ) × r k , define the directional derivatives
A ˙ k ( H ) [ G ] : = d d s A k ( H + s G ) ∣ s = 0 and P ˙ k ( H ) [ G ] : = d d s P k ( H + s G ) ∣ s = 0 . \dot{A}_k(H)[G]:=\left.\frac{d}{ds}A_k(H+sG)\right|_{s=0}
\quad\text{and}\quad
\dot{P}_k(H)[G]:=\left.\frac{d}{ds}P_k(H+sG)\right|_{s=0}. A ˙ k ( H ) [ G ] := d s d A k ( H + s G ) s = 0 and P ˙ k ( H ) [ G ] := d s d P k ( H + s G ) s = 0 .
In this sub-step, we show that there exist absolute constants C 2 ′ ≥ c 2 ′ > 0 C'_2\ge c'_2>0 C 2 ′ ≥ c 2 ′ > 0 such that, for every ∥ H ∥ F ≤ 1 / 2 \lVert H\rVert_F\le1/2 ∥ H ∥ F ≤ 1/2 and every G ∈ R ( p k − r k ) × r k G\in\mathbb{R}^{(p_k-r_k)\times r_k} G ∈ R ( p k − r k ) × r k ,
c 2 ′ ∥ G ∥ F ≤ ∥ A ˙ k ( H ) [ G ] ∥ F ≤ C 2 ′ ∥ G ∥ F . (115) c'_2\lVert G\rVert_F\le\lVert\dot{A}_k(H)[G]\rVert_F\le C'_2\lVert G\rVert_F. \tag*{(115)} c 2 ′ ∥ G ∥ F ≤ ∥ A ˙ k ( H ) [ G ] ∥ F ≤ C 2 ′ ∥ G ∥ F . (115)
Fix ∥ H ∥ F ≤ 1 / 2 \lVert H\rVert_F\le1/2 ∥ H ∥ F ≤ 1/2 and G ∈ R ( p k − r k ) × r k G\in\mathbb{R}^{(p_k-r_k)\times r_k} G ∈ R ( p k − r k ) × r k . For all sufficiently small ∣ s ∣ |s| ∣ s ∣ , both H H H and H + s G H+sG H + s G belong to the unit ball. Therefore, (105) gives
c 1 ′ ∣ s ∣ ∥ G ∥ F ≤ ∥ P k ( H + s G ) − P k ( H ) ∥ F . c'_1|s|\lVert G\rVert_F\le\lVert P_k(H+sG)-P_k(H)\rVert_F. c 1 ′ ∣ s ∣ ∥ G ∥ F ≤ ∥ P k ( H + s G ) − P k ( H ) ∥ F .
Moreover, the calculation leading to (114) gives
∥ A k ( H + s G ) − A k ( H ) ∥ F ≤ ( 1 + 2 ) ∣ s ∣ ∥ G ∥ F . \lVert A_k(H+sG)-A_k(H)\rVert_F\le(1+\sqrt{2})|s|\lVert G\rVert_F. ∥ A k ( H + s G ) − A k ( H ) ∥ F ≤ ( 1 + 2 ) ∣ s ∣ ∥ G ∥ F .
Dividing these inequalities by ∣ s ∣ |s| ∣ s ∣ and letting s → 0 s\to0 s → 0 gives
c 1 ′ ∥ G ∥ F ≤ ∥ P ˙ k ( H ) [ G ] ∥ F , and ∥ A ˙ k ( H ) [ G ] ∥ F ≤ ( 1 + 2 ) ∥ G ∥ F . (116) c'_1\lVert G\rVert_F\le\lVert\dot{P}_k(H)[G]\rVert_F,
\quad\text{and}\quad
\lVert\dot{A}_k(H)[G]\rVert_F\le(1+\sqrt{2})\lVert G\rVert_F. \tag*{(116)} c 1 ′ ∥ G ∥ F ≤ ∥ P ˙ k ( H ) [ G ] ∥ F , and ∥ A ˙ k ( H ) [ G ] ∥ F ≤ ( 1 + 2 ) ∥ G ∥ F . (116)
Differentiating P k ( H ) = A k ( H ) A k ( H ) ⊤ P_k(H)=A_k(H)A_k(H)^\top P k ( H ) = A k ( H ) A k ( H ) ⊤ in the direction G G G gives
P ˙ k ( H ) [ G ] = A ˙ k ( H ) [ G ] A k ( H ) ⊤ + A k ( H ) A ˙ k ( H ) [ G ] ⊤ . \dot{P}_k(H)[G]=\dot{A}_k(H)[G]A_k(H)^\top+A_k(H)\dot{A}_k(H)[G]^\top. P ˙ k ( H ) [ G ] = A ˙ k ( H ) [ G ] A k ( H ) ⊤ + A k ( H ) A ˙ k ( H ) [ G ] ⊤ .
Since ∥ A k ( H ) ∥ = 1 \lVert A_k(H)\rVert=1 ∥ A k ( H )∥ = 1 , we have
∥ P ˙ k ( H ) [ G ] ∥ F ≤ 2 ∥ A ˙ k ( H ) [ G ] ∥ F . (117) \lVert\dot{P}_k(H)[G]\rVert_F\le2\lVert\dot{A}_k(H)[G]\rVert_F. \tag*{(117)} ∥ P ˙ k ( H ) [ G ] ∥ F ≤ 2 ∥ A ˙ k ( H ) [ G ] ∥ F . (117)
Combining (116) and (117) , we have
c 1 ′ 2 ∥ G ∥ F ≤ ∥ A ˙ k ( H ) [ G ] ∥ F ≤ ( 1 + 2 ) ∥ G ∥ F . (118) \frac{c'_1}{2}\lVert G\rVert_F\le\lVert\dot{A}_k(H)[G]\rVert_F\le(1+\sqrt{2})\lVert G\rVert_F. \tag*{(118)} 2 c 1 ′ ∥ G ∥ F ≤ ∥ A ˙ k ( H ) [ G ] ∥ F ≤ ( 1 + 2 ) ∥ G ∥ F . (118)
Thus, (115) holds with c 2 ′ = c 1 ′ / 2 c'_2=c'_1/2 c 2 ′ = c 1 ′ /2 and C 2 ′ = 1 + 2 C'_2=1+\sqrt{2} C 2 ′ = 1 + 2 .
Step 1.3. In this sub-step, we construct core factors. Fix j ∈ [ m ] j\in[m] j ∈ [ m ] . We construct its core factors as follows. For each ℓ ∈ [ n j r − k ] \ell\in[n_jr_{-k}] ℓ ∈ [ n j r − k ] , define
i ℓ : = 1 + ( ( ℓ − 1 ) m o d r k ) , a i : = ∣ { ℓ ∈ [ n j r − k ] : i ℓ = i } ∣ , i ∈ [ r k ] . i_\ell:=1+((\ell-1)\bmod r_k),\qquad
a_i:=\left|\{\ell\in[n_jr_{-k}]:i_\ell=i\}\right|,\qquad i\in[r_k]. i ℓ := 1 + (( ℓ − 1 ) mod r k ) , a i := ∣ { ℓ ∈ [ n j r − k ] : i ℓ = i } ∣ , i ∈ [ r k ] .
The indices i ℓ i_\ell i ℓ cycle through [ r k ] [r_k] [ r k ] , and hence
a i ∈ { ⌊ n j r − k r k ⌋ , ⌈ n j r − k r k ⌉ } . a_i\in\left\{\left\lfloor\frac{n_jr_{-k}}{r_k}\right\rfloor,\left\lceil\frac{n_jr_{-k}}{r_k}\right\rceil\right\}. a i ∈ { ⌊ r k n j r − k ⌋ , ⌈ r k n j r − k ⌉ } .
Since n j r − k ≥ r k n_jr_{-k}\ge r_k n j r − k ≥ r k , we have
a i ≥ ⌊ n j r − k r k ⌋ ≥ n j r − k 2 r k . (119) a_i\ge\left\lfloor\frac{n_jr_{-k}}{r_k}\right\rfloor\ge\frac{n_jr_{-k}}{2r_k}. \tag*{(119)} a i ≥ ⌊ r k n j r − k ⌋ ≥ 2 r k n j r − k . (119)
Let e i e_i e i denote the i i i th canonical basis vector of R r k \mathbb{R}^{r_k} R r k and define
u ℓ : = ( n j λ a i ℓ ) 1 / 2 e i ℓ , ∀ ℓ ∈ [ n j r − k ] . u_\ell:= \left(\frac{n_j\lambda}{a_{i_\ell}}\right)^{1/2}e_{i_\ell},\quad\forall\ell\in[n_jr_{-k}]. u ℓ := ( a i ℓ n j λ ) 1/2 e i ℓ , ∀ ℓ ∈ [ n j r − k ] .
For each j ∈ [ m ] j \in[m] j ∈ [ m ] and t ∈ [ n j ] t \in[n_j] t ∈ [ n j ] , define
D j , t : = [ u ( t − 1 ) r − k + 1 , … , u t r − k ] ∈ R r k × r − k . D_{j,t} := [u_{(t-1)r_{-k}+1},\ldots,u_{tr_{-k}}] \in\mathbb{R}^{r_k \times r_{-k}}. D j , t := [ u ( t − 1 ) r − k + 1 , … , u t r − k ] ∈ R r k × r − k .
Since each coordinate i ∈ [ r k ] i \in[r_k] i ∈ [ r k ] occurs exactly a i a_i a i times, we have
1 n j ∑ t = 1 n j D j , t D j , t T = 1 n j ∑ ℓ = 1 n j r − k u ℓ u ℓ T = 1 n j ∑ i = 1 r k a i n j λ a i e i e i T = λ I r k . (120) \frac{1}{n_j}\sum_{t=1}^{n_j}D_{j,t}D_{j,t}^{\mathsf{T}} = \frac{1}{n_j}\sum_{\ell=1}^{n_jr_{-k}}u_\ell u_\ell^{\mathsf{T}} = \frac{1}{n_j}\sum_{i=1}^{r_k}a_i\frac{n_j\lambda}{a_i}e_ie_i^{\mathsf{T}} = \lambda I_{r_k}. \tag*{(120)} n j 1 t = 1 ∑ n j D j , t D j , t T = n j 1 ℓ = 1 ∑ n j r − k u ℓ u ℓ T = n j 1 i = 1 ∑ r k a i a i n j λ e i e i T = λ I r k . (120)
Moreover, for every t ∈ [ n j ] t \in[n_j] t ∈ [ n j ] , we obtain
∥ D j , t ∥ 2 ≤ ∥ D j , t ∥ F 2 = ∑ ℓ = ( t − 1 ) r − k + 1 t r − k ∥ u ℓ ∥ 2 = ∑ ℓ = ( t − 1 ) r − k + 1 t r − k n j λ a i ℓ ≤ n j r − k λ min i ∈ [ r k ] a i ≤ 2 r k λ = C r λ , (121) \lVert D_{j,t}\rVert^2 \leq\lVert D_{j,t}\rVert_{\mathrm{F}}^2 = \sum_{\ell=(t-1)r_{-k}+1}^{tr_{-k}}\lVert u_\ell\rVert^2 = \sum_{\ell=(t-1)r_{-k}+1}^{tr_{-k}}\frac{n_j\lambda}{a_{i_\ell}} \leq\frac{n_jr_{-k}\lambda}{\min_{i\in[r_k]}a_i} \leq2r_k\lambda= C_{r}\lambda, \tag*{(121)} ∥ D j , t ∥ 2 ≤ ∥ D j , t ∥ F 2 = ℓ = ( t − 1 ) r − k + 1 ∑ t r − k ∥ u ℓ ∥ 2 = ℓ = ( t − 1 ) r − k + 1 ∑ t r − k a i ℓ n j λ ≤ min i ∈ [ r k ] a i n j r − k λ ≤ 2 r k λ = C r λ , (121)
where the last inequality follows from (119) , and C r = 2 r k > 0 C_r=2r_k>0 C r = 2 r k > 0 is a constant since the factor ranks are fixed.
For each j ∈ [ m ] j \in[m] j ∈ [ m ] and t ∈ [ n j ] t \in[n_j] t ∈ [ n j ] , let F t ( j ) F_t^{(j)} F t ( j ) be the core tensor satisfying M k ( F t ( j ) ) = D j , t \mathcal{M}_k(F_t^{(j)})=D_{j,t} M k ( F t ( j ) ) = D j , t . Since mode-k k k matricization is a bijection, this core tensor is uniquely determined. For each H H H satisfying ∥ H ∥ F < 1 / 2 \lVert H\rVert_{\mathrm{F}}<1/2 ∥ H ∥ F < 1/2 , define
S t ( j ) : = F t ( j ) × 1 A 1 × 2 ⋯ × k A k ( H ) × k + 1 ⋯ × K A K , X t ( j ) : = S t ( j ) + E t ( j ) , S_t^{(j)} := F_t^{(j)} \times_1 A_1 \times_2 \cdots\times_k A_k(H) \times_{k+1}\cdots\times_K A_K,\qquad X_t^{(j)} := S_t^{(j)}+E_t^{(j)}, S t ( j ) := F t ( j ) × 1 A 1 × 2 ⋯ × k A k ( H ) × k + 1 ⋯ × K A K , X t ( j ) := S t ( j ) + E t ( j ) ,
where the entries of E t ( j ) E_t^{(j)} E t ( j ) are independent N ( 0 , σ 2 ) \mathcal{N}(0,\sigma^2) N ( 0 , σ 2 ) random variables. For notational convenience, write Y j , t : = M k ( X t ( j ) ) Y_{j,t}:=\mathcal{M}_k(X_t^{(j)}) Y j , t := M k ( X t ( j ) ) and E j , t : = M k ( E t ( j ) ) E_{j,t}:=\mathcal{M}_k(E_t^{(j)}) E j , t := M k ( E t ( j ) ) . Then we have
Y j , t = A k ( H ) D j , t A − k T + E j , t . (122) Y_{j,t}=A_k(H)D_{j,t}A_{-k}^{\mathsf{T}}+E_{j,t}. \tag*{(122)} Y j , t = A k ( H ) D j , t A − k T + E j , t . (122)
Moreover, (120) and (121) imply Λ j , ( k ) = λ I r k \Lambda_{j,(k)}=\lambda I_{r_k} Λ j , ( k ) = λ I r k and max t ∥ D j , t ∥ 2 ≤ C r λ \max_t\lVert D_{j,t}\rVert^2\leq C_r\lambda max t ∥ D j , t ∥ 2 ≤ C r λ . Hence the constructed model is a subclass of P k f e d ( λ , σ 2 ) \mathcal{P}_k^{\mathrm{fed}}(\lambda,\sigma^2) P k fed ( λ , σ 2 ) .
Step 2. In this step, we first fix a client j ∈ [ m ] j\in[m] j ∈ [ m ] , verify the conditions of Lemma 5 , and apply the lemma to bound the Fisher information in its private message. We then aggregate these information bounds over all clients.
Step 2.1. In this sub-step, we derive the score and Fisher information of one record. Fix j ∈ [ m ] j\in[m] j ∈ [ m ] . Since Lemma 5 is formulated for a vector-valued parameter, for every H ∈ R ( p k − r k ) × r k H\in\mathbb{R}^{(p_k-r_k)\times r_k} H ∈ R ( p k − r k ) × r k , write θ : = vec ( H ) ∈ R d k \theta:=\operatorname{vec}(H)\in\mathbb{R}^{d_k} θ := vec ( H ) ∈ R d k . Note that ∥ θ ∥ 2 = ∥ H ∥ F \lVert\theta\rVert_2=\lVert H\rVert_{\mathrm{F}} ∥ θ ∥ 2 = ∥ H ∥ F . For every t ∈ [ n j ] t\in[n_j] t ∈ [ n j ] , define the mean vector
μ j , t ( H ) : = vec { A k ( H ) D j , t A − k T } ∈ R p k p − k . \mu_{j,t}(H):=\operatorname{vec}\{A_k(H)D_{j,t}A_{-k}^{\mathsf{T}}\}\in\mathbb{R}^{p_kp_{-k}}. μ j , t ( H ) := vec { A k ( H ) D j , t A − k T } ∈ R p k p − k .
It follows from (122) that
vec ( Y j , t ) ∼ N { μ j , t ( H ) , σ 2 I p k p − k } , ∀ t ∈ [ n j ] . (123) \operatorname{vec}(Y_{j,t})\sim\mathcal{N}\{\mu_{j,t}(H),\sigma^2I_{p_kp_{-k}}\},\quad\forall t\in[n_j]. \tag*{(123)} vec ( Y j , t ) ∼ N { μ j , t ( H ) , σ 2 I p k p − k } , ∀ t ∈ [ n j ] . (123)
and these Gaussian records are independent across t t t .
Define the Jacobian of μ j , t ( H ) \mu_{j,t}(H) μ j , t ( H ) with respect to θ = vec ( H ) \theta= \operatorname{vec}(H) θ = vec ( H ) by
B j , t ( H ) = D θ μ j , t ( H ) = ( ∂ μ j , t , a ( H ) ∂ θ b ) a ∈ [ p k p − k ] , b ∈ [ d k ] ∈ R p k p − k × d k . B_{j,t}(H)=D_{\theta}\mu_{j,t}(H)=\left(\frac{\partial\mu_{j,t,a}(H)}{\partial\theta_b}\right)_{a\in[p_kp_{-k}],b\in[d_k]}\in\mathbb{R}^{p_kp_{-k}\times d_k}. B j , t ( H ) = D θ μ j , t ( H ) = ( ∂ θ b ∂ μ j , t , a ( H ) ) a ∈ [ p k p − k ] , b ∈ [ d k ] ∈ R p k p − k × d k .
For any G ∈ R ( p k − r k ) × r k G\in\mathbb{R}^{(p_k-r_k)\times r_k} G ∈ R ( p k − r k ) × r k , let g : = vec ( G ) g:=\operatorname{vec}(G) g := vec ( G ) . By the definition of the directional derivative, we have
B j , t ( H ) g = d d s μ j , t ( H + s G ) ∣ s = 0 = vec { A ˙ k ( H ) [ G ] D j , t A k T } . (124) B_{j,t}(H)g=\left.\frac{d}{ds}\mu_{j,t}(H+sG)\right|_{s=0}=\operatorname{vec}\left\{\dot{A}_k(H)[G]D_{j,t}A_k^{\mathsf T}\right\}. \tag*{(124)} B j , t ( H ) g = d s d μ j , t ( H + s G ) s = 0 = vec { A ˙ k ( H ) [ G ] D j , t A k T } . (124)
Let f j , t , H f_{j,t,H} f j , t , H denote the density of vec ( Y j , t ) \operatorname{vec}(Y_{j,t}) vec ( Y j , t ) under the parameter H H H . We have
log ( f j , t , H { vec ( Y j , t ) } ) = − 1 2 σ 2 ∥ vec ( Y j , t ) − μ j , t ( H ) ∥ 2 2 . \log\left(f_{j,t,H}\{\operatorname{vec}(Y_{j,t})\}\right)=-\frac{1}{2\sigma^2}\left\|\operatorname{vec}(Y_{j,t})-\mu_{j,t}(H)\right\|_2^2. log ( f j , t , H { vec ( Y j , t )} ) = − 2 σ 2 1 ∥ vec ( Y j , t ) − μ j , t ( H ) ∥ 2 2 .
Correspondingly, the score of the t t t th record with respect to θ = vec ( H ) \theta=\operatorname{vec}(H) θ = vec ( H ) is
S j , t ( H ) = ∇ θ log ( f j , t , H { vec ( Y j , t ) } ) = 1 σ 2 B j , t ( H ) T { vec ( Y j , t ) − μ j , t ( H ) } = 1 σ 2 B j , t ( H ) T vec ( E j , t ) ∈ R d k . (125) \begin{aligned}
S_{j,t}(H)=\nabla_\theta\log\left(f_{j,t,H}\{\operatorname{vec}(Y_{j,t})\}\right)=\frac{1}{\sigma^2}B_{j,t}(H)^{\mathsf T}\{\operatorname{vec}(Y_{j,t})-\mu_{j,t}(H)\} \\
&=\frac{1}{\sigma^2}B_{j,t}(H)^{\mathsf T}\operatorname{vec}(E_{j,t})\in\mathbb{R}^{d_k}. \tag*{(125)}
\end{aligned} S j , t ( H ) = ∇ θ log ( f j , t , H { vec ( Y j , t )} ) = σ 2 1 B j , t ( H ) T { vec ( Y j , t ) − μ j , t ( H )} = σ 2 1 B j , t ( H ) T vec ( E j , t ) ∈ R d k . (125)
Since the entries of E j , t E_{j,t} E j , t are independent N ( 0 , σ 2 ) \mathcal{N}(0,\sigma^2) N ( 0 , σ 2 ) random variables, S j , t ( H ) S_{j,t}(H) S j , t ( H ) is a centered Gaussian vector. Moreover,
E H { S j , t ( H ) S j , t ( H ) T } = 1 σ 4 B j , t ( H ) T E H { vec ( E j , t ) vec ( E j , t ) T } B j , t ( H ) = 1 σ 2 B j , t ( H ) T B j , t ( H ) . (126) \begin{aligned}
\mathbb{E}_H\{S_{j,t}(H)S_{j,t}(H)^{\mathsf T}\}=\frac{1}{\sigma^4}B_{j,t}(H)^{\mathsf T}\mathbb{E}_H\{\operatorname{vec}(E_{j,t})\operatorname{vec}(E_{j,t})^{\mathsf T}\}B_{j,t}(H) \\
&=\frac{1}{\sigma^2}B_{j,t}(H)^{\mathsf T}B_{j,t}(H). \tag*{(126)}
\end{aligned} E H { S j , t ( H ) S j , t ( H ) T } = σ 4 1 B j , t ( H ) T E H { vec ( E j , t ) vec ( E j , t ) T } B j , t ( H ) = σ 2 1 B j , t ( H ) T B j , t ( H ) . (126)
Step 2.2. In this sub-step, we verify the conditions in Lemma 5 . For any g = vec ( G ) ∈ R d k g=\operatorname{vec}(G)\in\mathbb{R}^{d_k} g = vec ( G ) ∈ R d k , we have
g T { ∑ t = 1 n j B j , t ( H ) T B j , t ( H ) } g = ∑ t = 1 n j ∥ B j , t ( H ) g ∥ 2 2 = ∑ t = 1 n j ∥ A ˙ k ( H ) [ G ] D j , t A k T ∥ F 2 = ∑ t = 1 n j ∥ A ˙ k ( H ) [ G ] D j , t ∥ F 2 = tr [ A ˙ k ( H ) [ G ] T A ˙ k ( H ) [ G ] { ∑ t = 1 n j D j , t D j , t T } ] = n j λ ∥ A ˙ k ( H ) [ G ] ∥ F 2 . (127) \begin{aligned}
g^{\mathsf T}\left\{\sum_{t=1}^{n_j}B_{j,t}(H)^{\mathsf T}B_{j,t}(H)\right\}g=\sum_{t=1}^{n_j}\|B_{j,t}(H)g\|_2^2=\sum_{t=1}^{n_j}\|\dot{A}_k(H)[G]D_{j,t}A_k^{\mathsf T}\|_{\mathrm F}^2 \\
&=\sum_{t=1}^{n_j}\|\dot{A}_k(H)[G]D_{j,t}\|_{\mathrm F}^2=\operatorname{tr}\left[\dot{A}_k(H)[G]^{\mathsf T}\dot{A}_k(H)[G]\left\{\sum_{t=1}^{n_j}D_{j,t}D_{j,t}^{\mathsf T}\right\}\right]=n_j\lambda\|\dot{A}_k(H)[G]\|_{\mathrm F}^2. \tag*{(127)}
\end{aligned} g T { t = 1 ∑ n j B j , t ( H ) T B j , t ( H ) } g = t = 1 ∑ n j ∥ B j , t ( H ) g ∥ 2 2 = t = 1 ∑ n j ∥ A ˙ k ( H ) [ G ] D j , t A k T ∥ F 2 = t = 1 ∑ n j ∥ A ˙ k ( H ) [ G ] D j , t ∥ F 2 = tr [ A ˙ k ( H ) [ G ] T A ˙ k ( H ) [ G ] { t = 1 ∑ n j D j , t D j , t T } ] = n j λ ∥ A ˙ k ( H ) [ G ] ∥ F 2 . (127)
where the first second equality follows from (124) , and the last equality follows from (120) . Since ∥ g ∥ 2 = ∥ G ∥ F \|g\|_2=\|G\|_{\mathrm F} ∥ g ∥ 2 = ∥ G ∥ F , combining (115) and (127) gives
( c 2 ′ ) 2 n j λ ∥ g ∥ 2 2 ≤ g T { ∑ t = 1 n j B j , t ( H ) T B j , t ( H ) } g ≤ ( C 2 ′ ) 2 n j λ ∥ g ∥ 2 2 . (c'_2)^2n_j\lambda\|g\|_2^2\leq g^{\mathsf T}\left\{\sum_{t=1}^{n_j}B_{j,t}(H)^{\mathsf T}B_{j,t}(H)\right\}g\leq(C'_2)^2n_j\lambda\|g\|_2^2. ( c 2 ′ ) 2 n j λ ∥ g ∥ 2 2 ≤ g T { t = 1 ∑ n j B j , t ( H ) T B j , t ( H ) } g ≤ ( C 2 ′ ) 2 n j λ ∥ g ∥ 2 2 .
Since this holds for every g ∈ R d k g\in\mathbb{R}^{d_k} g ∈ R d k , we have that
( c 2 ′ ) 2 n j λ I d k ⪯ ∑ t = 1 n j B j , t ( H ) T B j , t ( H ) ⪯ ( C 2 ′ ) 2 n j λ I d k . (128) (c'_2)^2n_j\lambda I_{d_k}\preceq\sum_{t=1}^{n_j}B_{j,t}(H)^{\mathsf T}B_{j,t}(H)\preceq(C'_2)^2n_j\lambda I_{d_k}. \tag*{(128)} ( c 2 ′ ) 2 n j λ I d k ⪯ t = 1 ∑ n j B j , t ( H ) T B j , t ( H ) ⪯ ( C 2 ′ ) 2 n j λ I d k . (128)
We also note that
∥ B j , t ( H ) ∥ 2 = sup ∥ g ∥ 2 = 1 ∥ B j , t ( H ) g ∥ 2 2 = sup ∥ G ∥ F = 1 ∥ A ˙ k ( H ) [ G ] D j , t A − k T ∥ F 2 = sup ∥ G ∥ F = 1 ∥ A ˙ k ( H ) [ G ] D j , t ∥ F 2 ≤ sup ∥ G ∥ F = 1 ∥ A ˙ k ( H ) [ G ] ∥ F 2 ∥ D j , t ∥ 2 ≤ ( C 2 ′ ) 2 C r λ . \begin{aligned}
\|B_{j,t}(H)\|^2
&= \sup_{\|g\|_2=1}\|B_{j,t}(H)g\|_2^2
= \sup_{\|G\|_F=1}\|\dot{A}_k(H)[G]D_{j,t}A_{-k}^{\mathsf T}\|_F^2 \\
&= \sup_{\|G\|_F=1}\|\dot{A}_k(H)[G]D_{j,t}\|_F^2
\leq\sup_{\|G\|_F=1}\|\dot{A}_k(H)[G]\|_F^2\|D_{j,t}\|^2
\leq(C'_2)^2 C_r\lambda.
\end{aligned} ∥ B j , t ( H ) ∥ 2 = ∥ g ∥ 2 = 1 sup ∥ B j , t ( H ) g ∥ 2 2 = ∥ G ∥ F = 1 sup ∥ A ˙ k ( H ) [ G ] D j , t A − k T ∥ F 2 = ∥ G ∥ F = 1 sup ∥ A ˙ k ( H ) [ G ] D j , t ∥ F 2 ≤ ∥ G ∥ F = 1 sup ∥ A ˙ k ( H ) [ G ] ∥ F 2 ∥ D j , t ∥ 2 ≤ ( C 2 ′ ) 2 C r λ .
where the second equality follows from (124) , and the last inequality follows from (115) and (121) . Consequently, we obtain
max t ∈ [ n j ] ∥ B j , t ( H ) ∥ 2 ≤ ( C 2 ′ ) 2 C r λ . (129) \max_{t\in[n_j]}\|B_{j,t}(H)\|^2 \leq(C'_2)^2 C_r\lambda. \tag*{(129)} t ∈ [ n j ] max ∥ B j , t ( H ) ∥ 2 ≤ ( C 2 ′ ) 2 C r λ . (129)
Step 2.3. In this sub-step, we apply Lemma 5 to the private message released by client j j j . Let D j : = ( X 1 ( j ) , … , X n j ( j ) ) \mathcal{D}_j := (X_1^{(j)},\ldots,X_{n_j}^{(j)}) D j := ( X 1 ( j ) , … , X n j ( j ) ) be the complete dataset of client j j j , and let Z j : = M j ( D j ) Z_j := \mathcal{M}_j(\mathcal{D}_j) Z j := M j ( D j ) denote its private message. We now verify the assumptions of Lemma 5. First,
{ vec ( H ) : ∥ H ∥ F < 1 2 } \left\{\operatorname{vec}(H):\|H\|_F<\frac{1}{2}\right\} { vec ( H ) : ∥ H ∥ F < 2 1 }
is an open subset of R d k \mathbb{R}^{d_k} R d k . Moreover, H ↦ A k ( H ) H\mapsto A_k(H) H ↦ A k ( H ) is continuously differentiable. Hence H ↦ μ j , t ( H ) H\mapsto\mu_{j,t}(H) H ↦ μ j , t ( H ) is continuously differentiable. Second, (123) shows that the observations are independent Gaussian vectors with covariance σ 2 I p k p − k \sigma^2 I_{p_kp_{-k}} σ 2 I p k p − k . Equations (128) and (129) verify the two Jacobian conditions of Lemma 5, uniformly over ∥ H ∥ F < 1 / 2 \|H\|_F<1/2 ∥ H ∥ F < 1/2 . Third, by the definition of FDP, M j \mathcal{M}_j M j is ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -centrally differentially private. Finally, the condition on δ \delta δ in Lemma 5 becomes
d k δ j log ( 1 / δ j ) ≤ c 0 n j ε j 2 . d_k\delta_j\log(1/\delta_j)\leq c_0\sqrt{n_j}\varepsilon_j^2. d k δ j log ( 1/ δ j ) ≤ c 0 n j ε j 2 .
which follows from the condition in the statement of Theorem 2. Therefore, Lemma 5 gives that there exists an absolute constant C ′ > 0 C'>0 C ′ > 0 such that
tr { I Z j ( H ) } ≤ C ′ λ σ 2 ( ( n j d k ) ∧ ( n j 2 ε j 2 ) ) , (130) \operatorname{tr}\{I_{Z_j}(H)\}\leq C'\frac{\lambda}{\sigma^2}\left((n_jd_k)\wedge(n_j^2\varepsilon_j^2)\right), \tag*{(130)} tr { I Z j ( H )} ≤ C ′ σ 2 λ ( ( n j d k ) ∧ ( n j 2 ε j 2 ) ) , (130)
uniformly over ∥ H ∥ F < 1 / 2 \|H\|_F<1/2 ∥ H ∥ F < 1/2 , where I Z j ( H ) I_{Z_j}(H) I Z j ( H ) denotes the Fisher information matrix for θ = vec ( H ) \theta=\operatorname{vec}(H) θ = vec ( H ) in the induced distribution of Z j Z_j Z j .
Step 2.4. In this sub-step, we aggregate the Fisher information bounds over all clients. Let Z : = ( Z 1 , … , Z m ) \boldsymbol{Z}:=(Z_1,\ldots,Z_m) Z := ( Z 1 , … , Z m ) denote the collection of private messages received by the server. Conditional on H H H , the client datasets are independent, and Z j Z_j Z j depends only on D j \mathcal{D}_j D j and the local randomization of client j j j . Therefore, Z 1 , … , Z m Z_1,\ldots,Z_m Z 1 , … , Z m are independent conditional on H H H .
Let q j , H q_{j,H} q j , H denote the density of Z j Z_j Z j under the parameter H H H . The joint density of Z \boldsymbol{Z} Z is
q H ( z 1 , … , z m ) = ∏ j = 1 m q j , H ( z j ) . q_H(z_1,\ldots,z_m)=\prod_{j=1}^{m}q_{j,H}(z_j). q H ( z 1 , … , z m ) = j = 1 ∏ m q j , H ( z j ) .
Consequently, the score of Z \boldsymbol{Z} Z with respect to θ = vec ( H ) \theta=\operatorname{vec}(H) θ = vec ( H ) is
S Z ( H ) = ∇ θ log ( q H ( Z 1 , … , Z m ) ) = ∑ j = 1 m ∇ θ log ( q j , H ( Z j ) ) . S_{\boldsymbol{Z}}(H)=\nabla_\theta\log\bigl(q_H(Z_1,\ldots,Z_m)\bigr)=\sum_{j=1}^{m}\nabla_\theta\log\bigl(q_{j,H}(Z_j)\bigr). S Z ( H ) = ∇ θ log ( q H ( Z 1 , … , Z m ) ) = j = 1 ∑ m ∇ θ log ( q j , H ( Z j ) ) .
Since the individual message scores are independent and centered under H H H , the Fisher information matrix for θ = vec ( H ) \theta= \operatorname{vec}(H) θ = vec ( H ) in the induced distribution of Z Z Z , denoted by I Z ( H ) \mathbf{I}_Z(H) I Z ( H ) , satisfies
I Z ( H ) = ∑ j = 1 m I Z j ( H ) . (131) \mathbf{I}_Z(H) = \sum_{j=1}^{m} \mathbf{I}_{Z_j}(H). \tag*{(131)} I Z ( H ) = j = 1 ∑ m I Z j ( H ) . (131)
Taking the trace in (131) and applying (130) , we obtain
tr { I Z ( H ) } ≤ C ′ λ σ 2 ∑ j = 1 m { ( n j d k ) ∧ ( n j 2 ε j 2 ) } . (132) \operatorname{tr}\{\mathbf{I}_Z(H)\} \le C^{\prime}\frac{\lambda}{\sigma^2}\sum_{j=1}^{m}\left\{(n_jd_k)\wedge(n_j^2\varepsilon_j^2)\right\}. \tag*{(132)} tr { I Z ( H )} ≤ C ′ σ 2 λ j = 1 ∑ m { ( n j d k ) ∧ ( n j 2 ε j 2 ) } . (132)
Step 3. In this final step, we first reduce the projector loss to the squared Euclidean loss for the local coordinate. We then construct a product cosine prior, apply the multivariate van Trees inequality, and express the resulting lower bound in terms of Ψ j , k l o c \Psi_{j,k}^{\mathrm{loc}} Ψ j , k loc .
Step 3.1. In this sub-step, we reduce estimation of the loading projector to estimation of the local coordinate H H H .
Fix 0 < ρ < 1 / 2 0<\rho<1/2 0 < ρ < 1/2 and let A ^ k \widehat{A}_k A k be any admissible estimator based on Z Z Z . Define H ^ \widehat{H} H as a measurable selection from
arg min ∥ H ′ ∥ F ≤ ρ ∥ A ^ k A ^ k T − P k ( H ′ ) ∥ F . \mathop{\arg\min}_{\|H'\|_{\mathrm{F}}\leq\rho}\left\|\widehat{A}_k\widehat{A}_k^{\mathsf{T}}-P_k(H')\right\|_{\mathrm{F}}. arg min ∥ H ′ ∥ F ≤ ρ A k A k T − P k ( H ′ ) F .
Such a minimizer exists as the set { H ′ : ∥ H ′ ∥ F ≤ ρ } \{H':\|H'\|_{\mathrm{F}}\leq\rho\} { H ′ : ∥ H ′ ∥ F ≤ ρ } is compact and P k ( H ′ ) P_k(H') P k ( H ′ ) is continuous in H ′ H' H ′ .
For every H H H satisfying ∥ H ∥ F ≤ ρ \|H\|_{\mathrm{F}}\leq\rho ∥ H ∥ F ≤ ρ , both H H H and H ^ \widehat{H} H belong to the unit ball. By (105) , the triangle inequality, and the fact that H ^ \widehat{H} H is the minimizer, we have
c 1 ′ ∥ H ^ − H ∥ F ≤ ∥ P k ( H ^ ) − P k ( H ) ∥ F ≤ ∥ P k ( H ^ ) − A ^ k A ^ k T ∥ F + ∥ A ^ k A ^ k T − P k ( H ) ∥ F ≤ 2 ∥ A ^ k A ^ k T − P k ( H ) ∥ F . \begin{aligned}
c'_1\|\widehat{H}-H\|_{\mathrm{F}}\leq\|P_k(\widehat{H})-P_k(H)\|_{\mathrm{F}} \\
&\leq\|P_k(\widehat{H})-\widehat{A}_k\widehat{A}_k^{\mathsf{T}}\|_{\mathrm{F}}+\|\widehat{A}_k\widehat{A}_k^{\mathsf{T}}-P_k(H)\|_{\mathrm{F}}\leq2\|\widehat{A}_k\widehat{A}_k^{\mathsf{T}}-P_k(H)\|_{\mathrm{F}}.
\end{aligned} c 1 ′ ∥ H − H ∥ F ≤ ∥ P k ( H ) − P k ( H ) ∥ F ≤ ∥ P k ( H ) − A k A k T ∥ F + ∥ A k A k T − P k ( H ) ∥ F ≤ 2∥ A k A k T − P k ( H ) ∥ F .
Consequently, we have that
( c 1 ′ ) 2 4 ∥ H ^ − H ∥ F 2 ≤ ∥ A ^ k A ^ k T − P k ( H ) ∥ F 2 . (133) \frac{(c'_1)^2}{4}\|\widehat{H}-H\|_{\mathrm{F}}^2\leq\|\widehat{A}_k\widehat{A}_k^{\mathsf{T}}-P_k(H)\|_{\mathrm{F}}^2. \tag*{(133)} 4 ( c 1 ′ ) 2 ∥ H − H ∥ F 2 ≤ ∥ A k A k T − P k ( H ) ∥ F 2 . (133)
Step 3.2. In this sub-step, we construct a prior for θ = vec ( H ) \theta=\operatorname{vec}(H) θ = vec ( H ) and calculate its Fisher information.
Let a : = ρ / d k a:=\rho/\sqrt{d_k} a := ρ / d k and define the density
π a ( u ) : = 1 a cos 2 ( π u 2 a ) 1 { ∣ u ∣ ≤ a } . \pi_a(u):=\frac{1}{a}\cos^2\left(\frac{\pi u}{2a}\right)\mathbb{1}\{|u|\leq a\}. π a ( u ) := a 1 cos 2 ( 2 a π u ) 1 { ∣ u ∣ ≤ a } .
Since
1 a ∫ − a a cos 2 ( π u 2 a ) d u = 1 , \frac{1}{a}\int_{-a}^{a}\cos^2\left(\frac{\pi u}{2a}\right)\,du=1, a 1 ∫ − a a cos 2 ( 2 a π u ) d u = 1 ,
it is a probability density. Place independent priors with density π a \pi_a π a on the d k d_k d k coordinates of θ = vec ( H ) \theta=\operatorname{vec}(H) θ = vec ( H ) , and denote the resulting product prior by π ρ \pi_\rho π ρ . If θ \theta θ belongs to the support of π ρ \pi_\rho π ρ , then
∥ H ∥ F 2 = ∥ θ ∥ 2 2 ≤ d k a 2 = ρ 2 . \|H\|_{\mathrm{F}}^2=\|\theta\|_2^2\leq d_ka^2=\rho^2. ∥ H ∥ F 2 = ∥ θ ∥ 2 2 ≤ d k a 2 = ρ 2 .
Thus, π ρ \pi_\rho π ρ is supported on { H : ∥ H ∥ F ≤ ρ } \{H : \lVert H\rVert_{\mathrm{F}} \le\rho\} { H : ∥ H ∥ F ≤ ρ } . For ∣ u ∣ < a |u| < a ∣ u ∣ < a , the score of the one-dimensional prior is
d d u log ( π a ( u ) ) = − π a tan ( π u 2 a ) . \frac{d}{du}\log(\pi_a(u))=-\frac{\pi}{a}\tan\left(\frac{\pi u}{2a}\right). d u d log ( π a ( u )) = − a π tan ( 2 a π u ) .
Hence its Fisher information is
∫ − a a { d d u log ( π a ( u ) ) } 2 π a ( u ) d u = π 2 a 3 ∫ − a a sin 2 ( π u 2 a ) d u = π 2 a 2 . (134) \int_{-a}^{a}\left\{\frac{d}{du}\log(\pi_a(u))\right\}^{2}\pi_a(u)du=\frac{\pi^2}{a^3}\int_{-a}^{a}\sin^2\left(\frac{\pi u}{2a}\right)du=\frac{\pi^2}{a^2}. \tag*{(134)} ∫ − a a { d u d log ( π a ( u )) } 2 π a ( u ) d u = a 3 π 2 ∫ − a a sin 2 ( 2 a π u ) d u = a 2 π 2 . (134)
Let I π ρ I_{\pi_\rho} I π ρ denote the Fisher information matrix of the product prior. Since its coordinates are independent, by (134) , we have
tr { I π ρ } = d k π 2 a 2 = π 2 d k 2 ρ 2 . (135) \operatorname{tr}\{I_{\pi_\rho}\}=d_k\frac{\pi^2}{a^2}=\frac{\pi^2d_k^2}{\rho^2}. \tag*{(135)} tr { I π ρ } = d k a 2 π 2 = ρ 2 π 2 d k 2 . (135)
Step 3.3. In this sub-step, we apply the classical multivariate van Trees inequality (Gill and Levit, [26 ] ), in the form stated in Lemma D.2 in Cai and Li [13 ] .
We first verify the required conditions. The joint sub-model obtained by stacking the Gaussian records in (123) has a smooth mean and a fixed positive definite covariance matrix. Hence, it satisfies the differentiability conditions in Assumption D.1 of Cai and Li [13 ] . Since Z Z Z is obtained through a parameter-independent Markov kernel, the induced experiment is also differentiable in quadratic mean. Moreover, the product cosine prior π ρ \pi_\rho π ρ is continuously differentiable, vanishes on the boundary of its support, and has finite Fisher information. Thus, the conditions of Cai and Li [13 ] are satisfied.
Applying Lemma D.2 in Cai and Li [13 ] and using ∥ vec ( H ^ ) − vec ( H ) ∥ 2 = ∥ H ^ − H ∥ F \lVert\operatorname{vec}(\widehat{H})-\operatorname{vec}(H)\rVert_2=\lVert\widehat{H}-H\rVert_{\mathrm{F}} ∥ vec ( H ) − vec ( H ) ∥ 2 = ∥ H − H ∥ F , we obtain
∫ E H ∥ H ^ − H ∥ F 2 d π ρ ( H ) ≥ d k 2 ∫ tr { I Z ( H ) } d π ρ ( H ) + tr { I π ρ } ≥ d k 2 C ′ σ 2 ∑ j = 1 m { ( n j d k ) ∧ ( n j 2 ε j 2 ) } + ( π 2 d k 2 ) / ρ 2 . (136) \begin{aligned}
\int\mathbb{E}_{H}\lVert\widehat{H}-H\rVert_{\mathrm{F}}^2d\pi_\rho(H)\ge\frac{d_k^2}{\int\operatorname{tr}\{I_Z(H)\}d\pi_\rho(H)+\operatorname{tr}\{I_{\pi_\rho}\}} \\
&\ge\frac{d_k^2}{\frac{C'}{\sigma^2}\sum_{j=1}^{m}\{(n_jd_k)\wedge(n_j^2\varepsilon_j^2)\}+(\pi^2d_k^2)/\rho^2}. \tag*{(136)}
\end{aligned} ∫ E H ∥ H − H ∥ F 2 d π ρ ( H ) ≥ ∫ tr { I Z ( H )} d π ρ ( H ) + tr { I π ρ } d k 2 ≥ σ 2 C ′ ∑ j = 1 m {( n j d k ) ∧ ( n j 2 ε j 2 )} + ( π 2 d k 2 ) / ρ 2 d k 2 . (136)
where the last inequality follows from (132) and (135) . Choose
ρ 2 = c ρ ( σ 2 d k 2 λ ∑ j = 1 m { ( n j d k ) ∧ ( n j 2 ε j 2 ) } ∧ 1 ) , (137) \rho^2=c_\rho\left(\frac{\sigma^2d_k^2}{\lambda\sum_{j=1}^{m}\{(n_jd_k)\wedge(n_j^2\varepsilon_j^2)\}}\wedge1\right), \tag*{(137)} ρ 2 = c ρ ( λ ∑ j = 1 m {( n j d k ) ∧ ( n j 2 ε j 2 )} σ 2 d k 2 ∧ 1 ) , (137)
where c ρ = 1 / 16 c_\rho=1/16 c ρ = 1/16 . Since u − 1 ≤ ( u ∧ 1 ) − 1 u^{-1}\leq(u\wedge1)^{-1} u − 1 ≤ ( u ∧ 1 ) − 1 for every u > 0 u>0 u > 0 , combining (136) and (137) gives
∫ E H ∥ H ^ − H ∥ F 2 d π ρ ( H ) ≥ C 2 ( σ 2 d k 2 λ ∑ j = 1 m { ( n j d k ) ∧ ( n j 2 ε j 2 ) } ∧ 1 ) ≥ C 2 2 { 1 ∑ j = 1 m ( σ 2 d k λ n j + σ 2 d k 2 n j 2 ε j 2 ) − 1 ∧ 2 } , (138) \begin{aligned}
\int\mathbb{E}_{H}\lVert\widehat{H}-H\rVert_{\mathrm{F}}^2d\pi_\rho(H)\ge C_2\left(\frac{\sigma^2d_k^2}{\lambda\sum_{j=1}^{m}\{(n_jd_k)\wedge(n_j^2\varepsilon_j^2)\}}\wedge1\right) \\
&\ge\frac{C_2}{2}\left\{\frac{1}{\sum_{j=1}^{m}\left(\frac{\sigma^2d_k}{\lambda n_j}+\frac{\sigma^2d_k^2}{n_j^2\varepsilon_j^2}\right)^{-1}}\wedge2\right\}, \tag*{(138)}
\end{aligned} ∫ E H ∥ H − H ∥ F 2 d π ρ ( H ) ≥ C 2 ( λ ∑ j = 1 m {( n j d k ) ∧ ( n j 2 ε j 2 )} σ 2 d k 2 ∧ 1 ) ≥ 2 C 2 ⎩ ⎨ ⎧ ∑ j = 1 m ( λ n j σ 2 d k + n j 2 ε j 2 σ 2 d k 2 ) − 1 1 ∧ 2 ⎭ ⎬ ⎫ , (138)
where C 2 = 1 / ( C ′ + 16 π 2 ) > 0 C_2=1/(C'+16\pi^2)>0 C 2 = 1/ ( C ′ + 16 π 2 ) > 0 is an absolute constant, and the last inequality follows from the fact that for any x , y > 0 x,y>0 x , y > 0 ,
x ∧ y ≤ 2 ( x − 1 + y − 1 ) − 1 . x\wedge y\leq2(x^{-1}+y^{-1})^{-1}. x ∧ y ≤ 2 ( x − 1 + y − 1 ) − 1 .
Moreover, since p k ≥ 2 r k p_k \ge2r_k p k ≥ 2 r k ,
d k = r k ( p k − r k ) ≥ r k p k 2 d_k = r_k(p_k-r_k) \ge\frac{r_kp_k}{2} d k = r k ( p k − r k ) ≥ 2 r k p k
It follows from the definition of Ψ j , k l o c \Psi^{\mathrm{loc}}_{j,k} Ψ j , k loc that
σ 2 d k λ n j + σ 2 d k 2 λ n j 2 ε j 2 ≥ r k ( Ψ j , k l o c ) 2 4 . (139) \frac{\sigma^2d_k}{\lambda n_j}+\frac{\sigma^2d_k^2}{\lambda n_j^2\varepsilon_j^2}\ge\frac{r_k(\Psi^{\mathrm{loc}}_{j,k})^2}{4}. \tag*{(139)} λ n j σ 2 d k + λ n j 2 ε j 2 σ 2 d k 2 ≥ 4 r k ( Ψ j , k loc ) 2 . (139)
Combining (133) , (138) , and (139) , we obtain
∫ E H ∥ A ^ k A ^ k T − P k ( H ) ∥ F 2 d π ρ ( H ) ≥ C 2 ( c 1 ′ ) 2 32 { r k ∑ j = 1 m ( Ψ j , k l o c ) − 2 ∧ 8 } ≥ c 2 { r k ∑ j = 1 m ( Ψ j , k l o c ) − 2 ∧ ( 2 r k ) } . (140) \int\mathbb{E}_H\left\|\widehat A_k\widehat A_k^{\mathsf T}-P_k(H)\right\|_{\mathrm F}^2d\pi_\rho(H)\ge\frac{C_2(c'_1)^2}{32}\left\{\frac{r_k}{\sum_{j=1}^m(\Psi^{\mathrm{loc}}_{j,k})^{-2}}\wedge8\right\}
\ge c_2\left\{\frac{r_k}{\sum_{j=1}^m(\Psi^{\mathrm{loc}}_{j,k})^{-2}}\wedge(2r_k)\right\}. \tag*{(140)} ∫ E H A k A k T − P k ( H ) F 2 d π ρ ( H ) ≥ 32 C 2 ( c 1 ′ ) 2 { ∑ j = 1 m ( Ψ j , k loc ) − 2 r k ∧ 8 } ≥ c 2 { ∑ j = 1 m ( Ψ j , k loc ) − 2 r k ∧ ( 2 r k ) } . (140)
where the last inequality follows since under the fixed-rank assumption, r k ≤ r max r_k\le r_{\max} r k ≤ r m a x for some fixed absolute constant r max r_{\max} r m a x and, for every x ≥ 0 x\ge0 x ≥ 0 ,
x ∧ 8 ≥ min { 1 , 4 r max } ( x ∧ ( 2 r k ) ) . x\wedge8\ge\min\left\{1,\frac{4}{r_{\max}}\right\}(x\wedge(2r_k)). x ∧ 8 ≥ min { 1 , r m a x 4 } ( x ∧ ( 2 r k )) .
with
c 2 = C 2 ( c 1 ′ ) 2 min { 1 , 4 / r max } 32 . c_2=\frac{C_2(c'_1)^2\min\{1,4/r_{\max}\}}{32}. c 2 = 32 C 2 ( c 1 ′ ) 2 min { 1 , 4/ r m a x } .
The support of π ρ \pi_\rho π ρ is contained in the Gaussian subclass constructed in Step 1 . Hence for every admissible estimator,
sup P ∈ P k f e d ( λ , σ 2 ) E P ∥ A ^ k A ^ k T − A k A k T ∥ F 2 ≥ ∫ E H ∥ A ^ k A ^ k T − P k ( H ) ∥ F 2 d π ρ ( H ) ≥ c 2 { r k ∑ j = 1 m ( Ψ j , k l o c ) − 2 ∧ ( 2 r k ) } , \sup_{P\in\mathcal P_{k}^{\mathrm{fed}}(\lambda,\sigma^2)}\mathbb E_P\left\|\widehat A_k\widehat A_k^{\mathsf T}-A_kA_k^{\mathsf T}\right\|_{\mathrm F}^2\ge\int\mathbb E_H\left\|\widehat A_k\widehat A_k^{\mathsf T}-P_k(H)\right\|_{\mathrm F}^2d\pi_\rho(H)
\ge c_2\left\{\frac{r_k}{\sum_{j=1}^m(\Psi^{\mathrm{loc}}_{j,k})^{-2}}\wedge(2r_k)\right\}, P ∈ P k fed ( λ , σ 2 ) sup E P A k A k T − A k A k T F 2 ≥ ∫ E H A k A k T − P k ( H ) F 2 d π ρ ( H ) ≥ c 2 { ∑ j = 1 m ( Ψ j , k loc ) − 2 r k ∧ ( 2 r k ) } ,
which completes the proof.
Proofs for adaptive tuning
Auxiliary lemmas with proofs
For the adaptive calibration arguments, define
ϱ j , k : = r k [ σ λ j , 1 ( k ) p k b j + σ 2 λ j , 1 ( k ) { p b j + p k b j } ] . (141) \varrho_{j,k}:=\sqrt{r_k}\left[\frac{\sigma}{\sqrt{\lambda_{j,1}^{(k)}}}\sqrt{\frac{p_k}{b_j}}+\frac{\sigma^2}{\lambda_{j,1}^{(k)}}\left\{\sqrt{\frac{p}{b_j}}+\frac{p_k}{b_j}\right\}\right]. \tag*{(141)} ϱ j , k := r k λ j , 1 ( k ) σ b j p k + λ j , 1 ( k ) σ 2 { b j p + b j p k } . (141)
Let C p > 0 C_p>0 C p > 0 be the fixed dimension constant used in Theorem 3 , and define C c m p : = 1 + C p − 1 C_{\mathrm{cmp}}:=1+C_p^{-1} C cmp := 1 + C p − 1 . Under p k ≥ C p log ( n ∨ p ) p_k\ge C_p\log(n\vee p) p k ≥ C p log ( n ∨ p ) , the radius in (10) satisfies
C R ϱ j , k ≤ R j , k ≤ C c m p C R ϱ j , k . (142) C_R\varrho_{j,k}\le R_{j,k}\le C_{\mathrm{cmp}}C_R\varrho_{j,k}. \tag*{(142)} C R ϱ j , k ≤ R j , k ≤ C cmp C R ϱ j , k . (142)
Lemma 6. Fix j ∈ [ m ] j \in[m] j ∈ [ m ] , and let P j ( 0 ) \mathcal{P}^{(0)}_j P j ( 0 ) be the sigma-algebra generated by its pilot partition and deterministic factors. Define
G j , k ( d ) : = 1 ( B j ( 0 ) ) 2 ∑ ℓ = 1 B j ( 0 ) ∑ h = 1 B j ( 0 ) P { ∥ Π ^ j , k ( 0 ) , [ ℓ ] − Π ^ j , k ( 0 ) , [ h ] ∥ F ≤ d | P j ( 0 ) } . (143) G_{j,k}(d) := \frac{1}{(B_j^{(0)})^2}\sum_{\ell=1}^{B_j^{(0)}}\sum_{h=1}^{B_j^{(0)}} \mathbb{P}\left\{\left\|\widehat{\Pi}^{(0),[\ell]}_{j,k}-\widehat{\Pi}^{(0),[h]}_{j,k}\right\|_{\mathrm{F}}\le d \,\middle|\, \mathcal{P}^{(0)}_j\right\}. \tag*{(143)} G j , k ( d ) := ( B j ( 0 ) ) 2 1 ℓ = 1 ∑ B j ( 0 ) h = 1 ∑ B j ( 0 ) P { Π j , k ( 0 ) , [ ℓ ] − Π j , k ( 0 ) , [ h ] F ≤ d P j ( 0 ) } . (143) Suppose that Assumptions 1–3 hold. Let
b j = ⌈ C b { r k + log ( n ∨ p ) } ⌉ , p k ≥ 2 r k , p k ≥ C p log ( n ∨ p ) , λ j , 1 ( k ) σ 2 ≥ C 0 ( p k + p ) . b_j=\left\lceil C_b\{r_k+\log(n\vee p)\}\right\rceil,\qquad p_k\ge2r_k,\qquad p_k\ge C_p\log(n\vee p),\qquad\frac{\lambda^{(k)}_{j,1}}{\sigma^2}\ge C_0(p_k+\sqrt{p}). b j = ⌈ C b { r k + log ( n ∨ p )} ⌉ , p k ≥ 2 r k , p k ≥ C p log ( n ∨ p ) , σ 2 λ j , 1 ( k ) ≥ C 0 ( p k + p ) . Also assume B j ( 0 ) ≥ 8 B_j^{(0)}\ge8 B j ( 0 ) ≥ 8 and (40) holds for every pilot block. Then for sufficiently large absolute constants C b C_b C b , C p C_p C p , and C 0 C_0 C 0 , there exist absolute constants 0 < c − < c + < ∞ 0<c_-<c_+<\infty 0 < c − < c + < ∞ such that
G j , k ( c − ϱ j , k ) ≤ 3 4 , G j , k ( c + ϱ j , k ) ≥ 15 16 . (144) G_{j,k}(c_-\varrho_{j,k})\le\frac{3}{4},\qquad G_{j,k}(c_+\varrho_{j,k})\ge\frac{15}{16}. \tag*{(144)} G j , k ( c − ϱ j , k ) ≤ 4 3 , G j , k ( c + ϱ j , k ) ≥ 16 15 . (144)
Proof of Lemma 6. Condition on P j ( 0 ) \mathcal{P}^{(0)}_j P j ( 0 ) , and we suppress the indices j j j and k k k throughout this proof. Write P : = A k A k T P:=A_kA_k^{\mathsf{T}} P := A k A k T and q k : = p k − r k q_k:=p_k-r_k q k := p k − r k and choose U k ∈ O p k × q k U_k\in\mathbb{O}_{p_k\times q_k} U k ∈ O p k × q k such that U k U k T = I p k − P U_kU_k^{\mathsf{T}}=I_{p_k}-P U k U k T = I p k − P . Within this proof, we write I ℓ : = I j , ℓ ( 0 ) \mathcal{I}_{\ell}:=\mathcal{I}^{(0)}_{j,\ell} I ℓ := I j , ℓ ( 0 ) for the ℓ \ell ℓ th pilot block, suppressing the client index and the pilot superscript. Then for a pilot block I ℓ \mathcal{I}_{\ell} I ℓ , set
s ℓ = ∣ I ℓ ∣ , S ℓ = 1 s ℓ ∑ t ∈ I ℓ D t D t T , s_{\ell}=|\mathcal{I}_{\ell}|,\qquad S_{\ell}=\frac{1}{s_{\ell}}\sum_{t\in\mathcal{I}_{\ell}}D_tD_t^{\mathsf{T}}, s ℓ = ∣ I ℓ ∣ , S ℓ = s ℓ 1 t ∈ I ℓ ∑ D t D t T ,
where D t : = M k ( F t ( j ) ) D_t:=\mathcal{M}_k(F_t^{(j)}) D t := M k ( F t ( j ) ) . By (40) , we have
1 2 Λ j , ( k ) ⪯ S ℓ ⪯ 3 2 Λ j , ( k ) , b j ≤ s ℓ ≤ 2 b j . (145) \frac{1}{2}\Lambda_{j,(k)}\preceq S_{\ell}\preceq\frac{3}{2}\Lambda_{j,(k)},\qquad b_j\le s_{\ell}\le2b_j. \tag*{(145)} 2 1 Λ j , ( k ) ⪯ S ℓ ⪯ 2 3 Λ j , ( k ) , b j ≤ s ℓ ≤ 2 b j . (145)
Step 1. In this step, we first establish an upper bound for the distance between two pilot block projectors. Apply (62) with a fixed sufficiently large tail parameter. Assumption 2, (145) , and p k ≳ log ( n ∨ p ) p_k\gtrsim\log(n\vee p) p k ≳ log ( n ∨ p ) give
P { ∥ Π ^ j , k ( 0 ) , [ ℓ ] − P ∥ F > C ϱ j , k | P j ( 0 ) } ≤ 1 64 . (146) \mathbb{P}\left\{\left\|\widehat{\Pi}^{(0),[\ell]}_{j,k}-P\right\|_{\mathrm{F}}>C\varrho_{j,k}\,\middle|\,\mathcal{P}^{(0)}_j\right\}\le\frac{1}{64}. \tag*{(146)} P { Π j , k ( 0 ) , [ ℓ ] − P F > C ϱ j , k P j ( 0 ) } ≤ 64 1 . (146)
For two distinct blocks, the triangle inequality and (146) therefore give, after choosing c + c_+ c + sufficiently large,
P { ∥ Π ^ j , k ( 0 ) , [ ℓ ] − Π ^ j , k ( 0 ) , [ h ] ∥ F ≤ c + ϱ j , k | P j ( 0 ) } ≥ 31 32 . (147) \mathbb{P}\left\{\left\|\widehat{\Pi}^{(0),[\ell]}_{j,k}-\widehat{\Pi}^{(0),[h]}_{j,k}\right\|_{\mathrm{F}}\le c_+\varrho_{j,k}\,\middle|\,\mathcal{P}^{(0)}_j\right\}\ge\frac{31}{32}. \tag*{(147)} P { Π j , k ( 0 ) , [ ℓ ] − Π j , k ( 0 ) , [ h ] F ≤ c + ϱ j , k P j ( 0 ) } ≥ 32 31 . (147)
Step 2. Next, we derive a first-order expansion of each pilot-block spectral projector around the true loading-space projector. For a pilot block I ℓ \mathcal{I}_{\ell} I ℓ , let
Σ ^ ℓ : = 1 s ℓ ∑ t ∈ I ℓ M k ( X t ( j ) ) M k ( X t ( j ) ) T . \widehat{\Sigma}_{\ell}:=\frac{1}{s_{\ell}}\sum_{t\in\mathcal{I}_{\ell}}\mathcal{M}_k(X_t^{(j)})\mathcal{M}_k(X_t^{(j)})^{\mathsf{T}}. Σ ℓ := s ℓ 1 t ∈ I ℓ ∑ M k ( X t ( j ) ) M k ( X t ( j ) ) T .
Conditional on P j ( 0 ) \mathcal{P}^{(0)}_j P j ( 0 ) , we have
Σ ℓ ⋆ : = E [ Σ ^ ℓ ] = A k S ℓ A k T + σ 2 p − k I p k . \Sigma_{\ell}^{\star}:=\mathbb{E}[\widehat{\Sigma}_{\ell}]=A_kS_{\ell}A_k^{\mathsf{T}}+\sigma^2p_{-k}I_{p_k}. Σ ℓ ⋆ := E [ Σ ℓ ] = A k S ℓ A k T + σ 2 p − k I p k .
Next, set Δ ℓ = Σ ^ ℓ − Σ ℓ ⋆ \Delta_\ell= \widehat{\Sigma}_\ell- \Sigma^\star_\ell Δ ℓ = Σ ℓ − Σ ℓ ⋆ and P = A k A k ⊤ P = A_k A_k^\top P = A k A k ⊤ . Since A k ⊤ A k = I r k A_k^\top A_k = I_{r_k} A k ⊤ A k = I r k and U k U k ⊤ = I p k − P U_k U_k^\top= I_{p_k} - P U k U k ⊤ = I p k − P , we have
( A k U k ) ⊤ Σ ℓ ⋆ ( A k U k ) = ( S ℓ + σ p − k 2 I r k 0 0 σ p − k 2 I q k ) . \left(A_k\ \ U_k\right)^\top\Sigma^\star_\ell\left(A_k\ \ U_k\right) =
\begin{pmatrix}
S_\ell+ \sigma^2_{p-k} I_{r_k} & 0 \\
0 & \sigma^2_{p-k} I_{q_k}
\end{pmatrix}. ( A k U k ) ⊤ Σ ℓ ⋆ ( A k U k ) = ( S ℓ + σ p − k 2 I r k 0 0 σ p − k 2 I q k ) .
Thus P P P is the leading rank-r k r_k r k spectral projector of Σ ℓ ⋆ \Sigma^\star_\ell Σ ℓ ⋆ , with eigengap λ r k ( S ℓ ) \lambda_{r_k}(S_\ell) λ r k ( S ℓ ) .
For u ∈ [ 0 , 1 ] u \in[0,1] u ∈ [ 0 , 1 ] , let Π ℓ ( u ) \Pi_\ell(u) Π ℓ ( u ) be the leading rank-r k r_k r k spectral projector of Σ ℓ ⋆ + u Δ ℓ \Sigma^\star_\ell+ u\Delta_\ell Σ ℓ ⋆ + u Δ ℓ . For the remainder of the proof, we work on the event
E g a p : = { ∥ Δ ℓ ∥ ≤ 1 4 λ r k ( S ℓ ) } . (148) \mathcal{E}_{\mathrm{gap}} := \left\{ \lVert\Delta_\ell\rVert\le\frac{1}{4}\lambda_{r_k}(S_\ell) \right\}. \tag*{(148)} E gap := { ∥ Δ ℓ ∥ ≤ 4 1 λ r k ( S ℓ ) } . (148)
Then Weyl’s inequality gives
λ r k ( Σ ℓ ⋆ + u Δ ℓ ) − λ r k + 1 ( Σ ℓ ⋆ + u Δ ℓ ) ≥ λ r k ( S ℓ ) − 2 u ∥ Δ ℓ ∥ ≥ 1 2 λ r k ( S ℓ ) , u ∈ [ 0 , 1 ] . (149) \lambda_{r_k}(\Sigma^\star_\ell+ u\Delta_\ell) - \lambda_{r_k+1}(\Sigma^\star_\ell+ u\Delta_\ell) \ge\lambda_{r_k}(S_\ell) - 2u\lVert\Delta_\ell\rVert\ge\frac{1}{2}\lambda_{r_k}(S_\ell), \quad u \in[0,1]. \tag*{(149)} λ r k ( Σ ℓ ⋆ + u Δ ℓ ) − λ r k + 1 ( Σ ℓ ⋆ + u Δ ℓ ) ≥ λ r k ( S ℓ ) − 2 u ∥ Δ ℓ ∥ ≥ 2 1 λ r k ( S ℓ ) , u ∈ [ 0 , 1 ] . (149)
The uniform eigengap ensures that Π ℓ ( u ) \Pi_\ell(u) Π ℓ ( u ) is twice differentiable along the path, and
Π ℓ ( 0 ) = P , Π ℓ ( 1 ) = Π ^ j , k ( 0 ) , [ ℓ ] . \Pi_\ell(0) = P,\quad\Pi_\ell(1) = \widehat{\Pi}^{(0),[\ell]}_{j,k}. Π ℓ ( 0 ) = P , Π ℓ ( 1 ) = Π j , k ( 0 ) , [ ℓ ] .
We first calculate Π ˙ ℓ ( 0 ) \dot{\Pi}_\ell(0) Π ˙ ℓ ( 0 ) . Differentiating
( Σ ℓ ⋆ + u Δ ℓ ) Π ℓ ( u ) = Π ℓ ( u ) ( Σ ℓ ⋆ + u Δ ℓ ) , Π ℓ ( u ) 2 = Π ℓ ( u ) , (150) (\Sigma^\star_\ell+ u\Delta_\ell)\Pi_\ell(u) = \Pi_\ell(u)(\Sigma^\star_\ell+ u\Delta_\ell),\quad\Pi_\ell(u)^2 = \Pi_\ell(u), \tag*{(150)} ( Σ ℓ ⋆ + u Δ ℓ ) Π ℓ ( u ) = Π ℓ ( u ) ( Σ ℓ ⋆ + u Δ ℓ ) , Π ℓ ( u ) 2 = Π ℓ ( u ) , (150)
at u = 0 u = 0 u = 0 gives
Δ ℓ P + Σ ℓ ⋆ Π ˙ ℓ ( 0 ) = Π ˙ ℓ ( 0 ) Σ ℓ ⋆ + P Δ ℓ , Π ˙ ℓ ( 0 ) P + P Π ˙ ℓ ( 0 ) = Π ˙ ℓ ( 0 ) . (151) \Delta_\ell P + \Sigma^\star_\ell\dot{\Pi}_\ell(0) = \dot{\Pi}_\ell(0)\Sigma^\star_\ell+ P\Delta_\ell,\quad\dot{\Pi}_\ell(0)P + P\dot{\Pi}_\ell(0) = \dot{\Pi}_\ell(0). \tag*{(151)} Δ ℓ P + Σ ℓ ⋆ Π ˙ ℓ ( 0 ) = Π ˙ ℓ ( 0 ) Σ ℓ ⋆ + P Δ ℓ , Π ˙ ℓ ( 0 ) P + P Π ˙ ℓ ( 0 ) = Π ˙ ℓ ( 0 ) . (151)
The second identity implies
P Π ˙ ℓ ( 0 ) P = 0 , ( I p k − P ) Π ˙ ℓ ( 0 ) ( I p k − P ) = 0. (152) P\dot{\Pi}_\ell(0)P = 0,\quad(I_{p_k} - P)\dot{\Pi}_\ell(0)(I_{p_k} - P) = 0. \tag*{(152)} P Π ˙ ℓ ( 0 ) P = 0 , ( I p k − P ) Π ˙ ℓ ( 0 ) ( I p k − P ) = 0. (152)
To determine the off-diagonal blocks, multiply the first differentiated identity in (151) on the left by I p k − P I_{p_k} - P I p k − P and on the right by P P P . The term containing P Δ ℓ P\Delta_\ell P Δ ℓ vanishes because ( I p k − P ) P = 0 (I_{p_k} - P)P = 0 ( I p k − P ) P = 0 . Hence we have
( I p k − P ) Δ ℓ P + ( I p k − P ) Σ ℓ ⋆ Π ˙ ℓ ( 0 ) P = ( I p k − P ) Π ˙ ℓ ( 0 ) Σ ℓ ⋆ P . (I_{p_k} - P)\Delta_\ell P + (I_{p_k} - P)\Sigma^\star_\ell\dot{\Pi}_\ell(0)P = (I_{p_k} - P)\dot{\Pi}_\ell(0)\Sigma^\star_\ell P. ( I p k − P ) Δ ℓ P + ( I p k − P ) Σ ℓ ⋆ Π ˙ ℓ ( 0 ) P = ( I p k − P ) Π ˙ ℓ ( 0 ) Σ ℓ ⋆ P .
By Σ ℓ ⋆ = A k S ℓ A k ⊤ + σ p − k 2 I p k \Sigma^\star_\ell= A_k S_\ell A_k^\top+ \sigma^2_{p-k}I_{p_k} Σ ℓ ⋆ = A k S ℓ A k ⊤ + σ p − k 2 I p k , P A k = A k PA_k = A_k P A k = A k , and ( I p k − P ) A k = 0 (I_{p_k} - P)A_k = 0 ( I p k − P ) A k = 0 , we have
( I p k − P ) Σ ℓ ⋆ = σ p − k 2 ( I p k − P ) , Σ ℓ ⋆ P = A k S ℓ A k ⊤ + σ p − k 2 P . (I_{p_k} - P)\Sigma^\star_\ell= \sigma^2_{p-k}(I_{p_k} - P),\quad\Sigma^\star_\ell P = A_k S_\ell A_k^\top+ \sigma^2_{p-k}P. ( I p k − P ) Σ ℓ ⋆ = σ p − k 2 ( I p k − P ) , Σ ℓ ⋆ P = A k S ℓ A k ⊤ + σ p − k 2 P .
Substituting these expressions into the preceding identity gives
( I p k − P ) Δ ℓ P + σ p − k 2 ( I p k − P ) Π ˙ ℓ ( 0 ) P = ( I p k − P ) Π ˙ ℓ ( 0 ) A k S ℓ A k ⊤ + σ p − k 2 ( I p k − P ) Π ˙ ℓ ( 0 ) P . (I_{p_k} - P)\Delta_\ell P + \sigma^2_{p-k}(I_{p_k} - P)\dot{\Pi}_\ell(0)P = (I_{p_k} - P)\dot{\Pi}_\ell(0)A_kS_\ell A_k^\top+ \sigma^2_{p-k}(I_{p_k} - P)\dot{\Pi}_\ell(0)P. ( I p k − P ) Δ ℓ P + σ p − k 2 ( I p k − P ) Π ˙ ℓ ( 0 ) P = ( I p k − P ) Π ˙ ℓ ( 0 ) A k S ℓ A k ⊤ + σ p − k 2 ( I p k − P ) Π ˙ ℓ ( 0 ) P .
Hence
( I p k − P ) Δ ℓ P = ( I p k − P ) Π ˙ ℓ ( 0 ) A k S ℓ A k ⊤ . (I_{p_k} - P)\Delta_\ell P = (I_{p_k} - P)\dot{\Pi}_\ell(0)A_kS_\ell A_k^\top. ( I p k − P ) Δ ℓ P = ( I p k − P ) Π ˙ ℓ ( 0 ) A k S ℓ A k ⊤ .
Right-multiplying both sides by A k S ℓ − 1 A k ⊤ A_kS_\ell^{-1}A_k^{\top} A k S ℓ − 1 A k ⊤ and using P A k = A k PA_k=A_k P A k = A k and A k ⊤ A k = I r k A_k^{\top}A_k=I_{r_k} A k ⊤ A k = I r k gives
( I p k − P ) Δ ℓ A k S ℓ − 1 A k ⊤ = ( I p k − P ) Π ˙ ℓ ( 0 ) P . (I_{p_k}-P)\Delta_\ell A_kS_\ell^{-1}A_k^{\top}=(I_{p_k}-P)\dot{\Pi}_\ell(0)P. ( I p k − P ) Δ ℓ A k S ℓ − 1 A k ⊤ = ( I p k − P ) Π ˙ ℓ ( 0 ) P .
Each Π ℓ ( u ) \Pi_\ell(u) Π ℓ ( u ) is a symmetric projector, so Π ˙ ℓ ( 0 ) \dot{\Pi}_\ell(0) Π ˙ ℓ ( 0 ) is symmetric. As Δ ℓ \Delta_\ell Δ ℓ is also symmetric, transposing the last equality gives
P Π ˙ ℓ ( 0 ) ( I p k − P ) = A k S ℓ − 1 A k ⊤ Δ ℓ ( I p k − P ) . P\dot{\Pi}_\ell(0)(I_{p_k}-P)=A_kS_\ell^{-1}A_k^{\top}\Delta_\ell(I_{p_k}-P). P Π ˙ ℓ ( 0 ) ( I p k − P ) = A k S ℓ − 1 A k ⊤ Δ ℓ ( I p k − P ) .
Finally, the two diagonal blocks of Π ˙ ℓ ( 0 ) \dot{\Pi}_\ell(0) Π ˙ ℓ ( 0 ) were shown above to vanish. Therefore
Π ˙ ℓ ( 0 ) = ( I p k − P ) Π ˙ ℓ ( 0 ) P + P Π ˙ ℓ ( 0 ) ( I p k − P ) = ( I p k − P ) Δ ℓ A k S ℓ − 1 A k ⊤ + A k S ℓ − 1 A k ⊤ Δ ℓ ( I p k − P ) . \begin{aligned}
\dot{\Pi}_\ell(0)=(I_{p_k}-P)\dot{\Pi}_\ell(0)P+P\dot{\Pi}_\ell(0)(I_{p_k}-P) \\
&=(I_{p_k}-P)\Delta_\ell A_kS_\ell^{-1}A_k^{\top}+A_kS_\ell^{-1}A_k^{\top}\Delta_\ell(I_{p_k}-P).
\end{aligned} Π ˙ ℓ ( 0 ) = ( I p k − P ) Π ˙ ℓ ( 0 ) P + P Π ˙ ℓ ( 0 ) ( I p k − P ) = ( I p k − P ) Δ ℓ A k S ℓ − 1 A k ⊤ + A k S ℓ − 1 A k ⊤ Δ ℓ ( I p k − P ) .
Using I p k − P = U k U k ⊤ I_{p_k}-P=U_kU_k^{\top} I p k − P = U k U k ⊤ , define
K ℓ : = U k ⊤ Δ ℓ A k S ℓ − 1 , L ℓ : = Π ˙ ℓ ( 0 ) = U k K ℓ A k ⊤ + A k K ℓ ⊤ U k ⊤ , R ℓ : = Π ^ j , k ( 0 ) , [ ℓ ] − P − L ℓ . (153) K_\ell:=U_k^{\top}\Delta_\ell A_kS_\ell^{-1},\quad L_\ell:=\dot{\Pi}_\ell(0)=U_kK_\ell A_k^{\top}+A_kK_\ell^{\top}U_k^{\top},\quad R_\ell:=\widehat{\Pi}^{(0),[\ell]}_{j,k}-P-L_\ell. \tag*{(153)} K ℓ := U k ⊤ Δ ℓ A k S ℓ − 1 , L ℓ := Π ˙ ℓ ( 0 ) = U k K ℓ A k ⊤ + A k K ℓ ⊤ U k ⊤ , R ℓ := Π j , k ( 0 ) , [ ℓ ] − P − L ℓ . (153)
Then we can write
Π ^ j , k ( 0 ) , [ ℓ ] − P = L ℓ + R ℓ . (154) \widehat{\Pi}^{(0),[\ell]}_{j,k}-P=L_\ell+R_\ell. \tag*{(154)} Π j , k ( 0 ) , [ ℓ ] − P = L ℓ + R ℓ . (154)
Step 3. We establish uniform operator- and Frobenius-norm bounds for Π ˙ ℓ ( u ) \dot{\Pi}_\ell(u) Π ˙ ℓ ( u ) along the perturbation path. These bounds will be used to control the remainder R ℓ R_\ell R ℓ .
Fix u ∈ [ 0 , 1 ] u\in[0,1] u ∈ [ 0 , 1 ] . Differentiating (150) gives
Π ℓ ( u ) Π ˙ ℓ ( u ) + Π ˙ ℓ ( u ) Π ℓ ( u ) = Π ˙ ℓ ( u ) , (155) \Pi_\ell(u)\dot{\Pi}_\ell(u)+\dot{\Pi}_\ell(u)\Pi_\ell(u)=\dot{\Pi}_\ell(u), \tag*{(155)} Π ℓ ( u ) Π ˙ ℓ ( u ) + Π ˙ ℓ ( u ) Π ℓ ( u ) = Π ˙ ℓ ( u ) , (155)
Δ ℓ Π ℓ ( u ) + ( Σ ℓ ⋆ + u Δ ℓ ) Π ˙ ℓ ( u ) = Π ˙ ℓ ( u ) ( Σ ℓ ⋆ + u Δ ℓ ) + Π ℓ ( u ) Δ ℓ . (156) \Delta_\ell\Pi_\ell(u)+(\Sigma_\ell^\star+u\Delta_\ell)\dot{\Pi}_\ell(u)=\dot{\Pi}_\ell(u)(\Sigma_\ell^\star+u\Delta_\ell)+\Pi_\ell(u)\Delta_\ell. \tag*{(156)} Δ ℓ Π ℓ ( u ) + ( Σ ℓ ⋆ + u Δ ℓ ) Π ˙ ℓ ( u ) = Π ˙ ℓ ( u ) ( Σ ℓ ⋆ + u Δ ℓ ) + Π ℓ ( u ) Δ ℓ . (156)
Multiplying (155) on both sides by Π ℓ ( u ) \Pi_\ell(u) Π ℓ ( u ) gives
Π ℓ ( u ) Π ˙ ℓ ( u ) Π ℓ ( u ) = 0. (157) \Pi_\ell(u)\dot{\Pi}_\ell(u)\Pi_\ell(u)=0. \tag*{(157)} Π ℓ ( u ) Π ˙ ℓ ( u ) Π ℓ ( u ) = 0. (157)
Similarly, multiplying (155) on both sides by I p k − Π ℓ ( u ) I_{p_k}-\Pi_\ell(u) I p k − Π ℓ ( u ) gives
( I p k − Π ℓ ( u ) ) Π ˙ ℓ ( u ) ( I p k − Π ℓ ( u ) ) = 0. (158) (I_{p_k}-\Pi_\ell(u))\dot{\Pi}_\ell(u)(I_{p_k}-\Pi_\ell(u))=0. \tag*{(158)} ( I p k − Π ℓ ( u )) Π ˙ ℓ ( u ) ( I p k − Π ℓ ( u )) = 0. (158)
Thus the two diagonal blocks of Π ˙ ℓ ( u ) \dot{\Pi}_\ell(u) Π ˙ ℓ ( u ) , relative to the range of Π ℓ ( u ) \Pi_\ell(u) Π ℓ ( u ) and its orthogonal complement, are zero. Since Π ˙ ℓ ( u ) \dot{\Pi}_\ell(u) Π ˙ ℓ ( u ) is symmetric, its two remaining off-diagonal blocks are transposes of each other.
Fix u ∈ [ 0 , 1 ] u\in[0,1] u ∈ [ 0 , 1 ] . After taking the derivatives above, choose an orthonormal eigen-basis of Σ ℓ ⋆ + u Δ ℓ \Sigma_\ell^\star+u\Delta_\ell Σ ℓ ⋆ + u Δ ℓ , with its leading r k r_k r k eigenvectors first. All matrix entries below are read in this basis, which is held fixed. In this basis, Σ ℓ ⋆ + u Δ ℓ \Sigma_\ell^\star+u\Delta_\ell Σ ℓ ⋆ + u Δ ℓ is diagonal. Also, Π ℓ ( u ) \Pi_\ell(u) Π ℓ ( u ) has diagonal entries one in the first r k r_k r k positions and zero in the remaining positions. Equations (157) and (158) show that the two diagonal blocks of Π ˙ ℓ ( u ) \dot{\Pi}_\ell(u) Π ˙ ℓ ( u ) are zero. As Π ℓ ( u ) \Pi_\ell(u) Π ℓ ( u ) is symmetric for every u u u , Π ˙ ℓ ( u ) \dot{\Pi}_\ell(u) Π ˙ ℓ ( u ) is symmetric as well.
Now take row a > r k a>r_k a > r k and column i ≤ r k i\le r_k i ≤ r k . Note that
[ Δ ℓ Π ℓ ( u ) ] a i = [ Δ ℓ ] a i , [ ( Σ ℓ ⋆ + u Δ ℓ ) Π ˙ ℓ ( u ) ] a i = λ a ( Σ ℓ ⋆ + u Δ ℓ ) [ Π ˙ ℓ ( u ) ] a i , [\Delta_\ell\Pi_\ell(u)]_{ai}=[\Delta_\ell]_{ai},\quad[(\Sigma_\ell^\star+u\Delta_\ell)\dot{\Pi}_\ell(u)]_{ai}=\lambda_a(\Sigma_\ell^\star+u\Delta_\ell)[\dot{\Pi}_\ell(u)]_{ai}, [ Δ ℓ Π ℓ ( u ) ] ai = [ Δ ℓ ] ai , [( Σ ℓ ⋆ + u Δ ℓ ) Π ˙ ℓ ( u ) ] ai = λ a ( Σ ℓ ⋆ + u Δ ℓ ) [ Π ˙ ℓ ( u ) ] ai ,
[ Π ˙ ℓ ( u ) ( Σ ℓ ⋆ + u Δ ℓ ) ] a i = [ Π ˙ ℓ ( u ) ] a i λ i ( Σ ℓ ⋆ + u Δ ℓ ) , [ Π ℓ ( u ) Δ ℓ ] a i = 0 , [\dot{\Pi}_\ell(u)(\Sigma_\ell^\star+u\Delta_\ell)]_{ai}=[\dot{\Pi}_\ell(u)]_{ai}\lambda_i(\Sigma_\ell^\star+u\Delta_\ell),\quad[\Pi_\ell(u)\Delta_\ell]_{ai}=0, [ Π ˙ ℓ ( u ) ( Σ ℓ ⋆ + u Δ ℓ ) ] ai = [ Π ˙ ℓ ( u ) ] ai λ i ( Σ ℓ ⋆ + u Δ ℓ ) , [ Π ℓ ( u ) Δ ℓ ] ai = 0 ,
where the first equality holds because, in the chosen eigen-basis, right multiplication by Π ℓ ( u ) \Pi_\ell(u) Π ℓ ( u ) leaves the i i i th column unchanged, the second and third equalities hold because left and right multiplication by the diagonal matrix Σ ℓ ⋆ + u Δ ℓ \Sigma_\ell^\star+u\Delta_\ell Σ ℓ ⋆ + u Δ ℓ scale the a a a th row and the i i i th column, respectively, by their corresponding eigenvalues, and the last equality holds because the a a a th row of Π ℓ ( u ) \Pi_\ell(u) Π ℓ ( u ) is zero. Substituting these four entries into (156) gives
[ Δ ℓ ] a i + λ a ( Σ ℓ ⋆ + u Δ ℓ ) [ Π ˙ ℓ ( u ) ] a i = [ Π ˙ ℓ ( u ) ] a i λ i ( Σ ℓ ⋆ + u Δ ℓ ) . [\Delta_\ell]_{ai}+\lambda_a(\Sigma_\ell^\star+u\Delta_\ell)[\dot{\Pi}_\ell(u)]_{ai}=[\dot{\Pi}_\ell(u)]_{ai}\lambda_i(\Sigma_\ell^\star+u\Delta_\ell). [ Δ ℓ ] ai + λ a ( Σ ℓ ⋆ + u Δ ℓ ) [ Π ˙ ℓ ( u ) ] ai = [ Π ˙ ℓ ( u ) ] ai λ i ( Σ ℓ ⋆ + u Δ ℓ ) .
Hence we have
[ Π ˙ ℓ ( u ) ] a i = [ Δ ℓ ] a i λ i ( Σ ℓ ⋆ + u Δ ℓ ) − λ a ( Σ ℓ ⋆ + u Δ ℓ ) . [\dot{\Pi}_\ell(u)]_{ai}=\frac{[\Delta_\ell]_{ai}}{\lambda_i(\Sigma_\ell^\star+u\Delta_\ell)-\lambda_a(\Sigma_\ell^\star+u\Delta_\ell)}. [ Π ˙ ℓ ( u ) ] ai = λ i ( Σ ℓ ⋆ + u Δ ℓ ) − λ a ( Σ ℓ ⋆ + u Δ ℓ ) [ Δ ℓ ] ai .
As i ≤ r k < a i\le r_k<a i ≤ r k < a , (149) gives
∣ [ Π ˙ ℓ ( u ) ] a i ∣ ≤ 2 ∣ [ Δ ℓ ] a i ∣ λ r k ( S ℓ ) . (159) |[\dot{\Pi}_\ell(u)]_{ai}|\le\frac{2|[\Delta_\ell]_{ai}|}{\lambda_{r_k}(S_\ell)}. \tag*{(159)} ∣ [ Π ˙ ℓ ( u ) ] ai ∣ ≤ λ r k ( S ℓ ) 2∣ [ Δ ℓ ] ai ∣ . (159)
Hence we have
∥ Π ˙ ℓ ( u ) ∥ F 2 = 2 ∑ i ≤ r k < a ∣ [ Π ˙ ℓ ( u ) ] a i ∣ 2 ≤ 8 λ r k ( S ℓ ) 2 ∑ i ≤ r k < a ∣ [ Δ ℓ ] a i ∣ 2 = 8 λ r k ( S ℓ ) 2 ∥ ( I p k − Π ℓ ( u ) ) Δ ℓ Π ℓ ( u ) ∥ F 2 ≤ 8 λ r k ( S ℓ ) 2 ∥ Δ ℓ ∥ 2 ∥ Π ℓ ( u ) ∥ F 2 = 8 r k ∥ Δ ℓ ∥ 2 λ r k ( S ℓ ) 2 , (160) \begin{aligned}
\|\dot{\Pi}_\ell(u)\|_F^2=2\sum_{i\le r_k<a}|[\dot{\Pi}_\ell(u)]_{ai}|^2\le\frac{8}{\lambda_{r_k}(S_\ell)^2}\sum_{i\le r_k<a}|[\Delta_\ell]_{ai}|^2 \\
&=\frac{8}{\lambda_{r_k}(S_\ell)^2}\|(I_{p_k}-\Pi_\ell(u))\Delta_\ell\Pi_\ell(u)\|_F^2\le\frac{8}{\lambda_{r_k}(S_\ell)^2}\|\Delta_\ell\|^2\|\Pi_\ell(u)\|_F^2=\frac{8r_k\|\Delta_\ell\|^2}{\lambda_{r_k}(S_\ell)^2}, \tag*{(160)}
\end{aligned} ∥ Π ˙ ℓ ( u ) ∥ F 2 = 2 i ≤ r k < a ∑ ∣ [ Π ˙ ℓ ( u ) ] ai ∣ 2 ≤ λ r k ( S ℓ ) 2 8 i ≤ r k < a ∑ ∣ [ Δ ℓ ] ai ∣ 2 = λ r k ( S ℓ ) 2 8 ∥ ( I p k − Π ℓ ( u )) Δ ℓ Π ℓ ( u ) ∥ F 2 ≤ λ r k ( S ℓ ) 2 8 ∥ Δ ℓ ∥ 2 ∥ Π ℓ ( u ) ∥ F 2 = λ r k ( S ℓ ) 2 8 r k ∥ Δ ℓ ∥ 2 , (160)
where the first inequality used (159) , the second equality follows from the fact that ( I p k − Π ℓ ( u ) ) Δ ℓ Π ℓ ( u ) (I_{p_k}-\Pi_\ell(u))\Delta_\ell\Pi_\ell(u) ( I p k − Π ℓ ( u )) Δ ℓ Π ℓ ( u ) selects exactly the entries with a > r k a>r_k a > r k and i ≤ r k i\le r_k i ≤ r k , and the last equality follows from ∥ Π ℓ ( u ) ∥ F 2 = rank { Π ℓ ( u ) } = r k \|\Pi_\ell(u)\|_F^2=\operatorname{rank}\{\Pi_\ell(u)\}=r_k ∥ Π ℓ ( u ) ∥ F 2 = rank { Π ℓ ( u )} = r k .
For the operator norm, a similar argument yields
∥ Π ˙ ℓ ( u ) ∥ ≤ 2 ∥ Δ ℓ ∥ λ r k ( S ℓ ) . (161) \|\dot{\Pi}_\ell(u)\|\le\frac{2\|\Delta_\ell\|}{\lambda_{r_k}(S_\ell)}. \tag*{(161)} ∥ Π ˙ ℓ ( u ) ∥ ≤ λ r k ( S ℓ ) 2∥ Δ ℓ ∥ . (161)
Combining (160) and (161) , and using the invariance of both norms under orthogonal changes of basis, on the event E g a p \mathcal{E}_{\mathrm{gap}} E gap , we obtain the following bounds in the original basis, uniformly over u ∈ [ 0 , 1 ] u\in[0,1] u ∈ [ 0 , 1 ] :
∥ Π ˙ ℓ ( u ) ∥ ≤ 2 ∥ Δ ℓ ∥ λ r k ( S ℓ ) , ∥ Π ˙ ℓ ( u ) ∥ F ≤ 2 r k ∥ Δ ℓ ∥ λ r k ( S ℓ ) . (162) \|\dot{\Pi}_\ell(u)\|\le\frac{2\|\Delta_\ell\|}{\lambda_{r_k}(S_\ell)},\qquad\|\dot{\Pi}_\ell(u)\|_F\le\frac{2\sqrt{r_k}\|\Delta_\ell\|}{\lambda_{r_k}(S_\ell)}. \tag*{(162)} ∥ Π ˙ ℓ ( u ) ∥ ≤ λ r k ( S ℓ ) 2∥ Δ ℓ ∥ , ∥ Π ˙ ℓ ( u ) ∥ F ≤ λ r k ( S ℓ ) 2 r k ∥ Δ ℓ ∥ . (162)
Step 4. In this step, we show that, with high conditional probability, the second-order projector remainder R ℓ \mathcal{R}_\ell R ℓ is a small fraction of the intrinsic radius ϱ j , k \varrho_{j,k} ϱ j , k . Differentiating (155) and (156) once more gives
Π ℓ ( u ) Π ¨ ℓ ( u ) + Π ¨ ℓ ( u ) Π ℓ ( u ) + 2 Π ˙ ℓ ( u ) 2 = Π ¨ ℓ ( u ) , \Pi_\ell(u)\ddot{\Pi}_\ell(u)+\ddot{\Pi}_\ell(u)\Pi_\ell(u)+2\dot{\Pi}_\ell(u)^2=\ddot{\Pi}_\ell(u), Π ℓ ( u ) Π ¨ ℓ ( u ) + Π ¨ ℓ ( u ) Π ℓ ( u ) + 2 Π ˙ ℓ ( u ) 2 = Π ¨ ℓ ( u ) ,
( Σ ℓ ⋆ + u Δ ℓ ) Π ¨ ℓ ( u ) − Π ¨ ℓ ( u ) ( Σ ℓ ⋆ + u Δ ℓ ) = 2 { Π ˙ ℓ ( u ) Δ ℓ − Δ ℓ Π ˙ ℓ ( u ) } . (\Sigma_\ell^\star+u\Delta_\ell)\ddot{\Pi}_\ell(u)-\ddot{\Pi}_\ell(u)(\Sigma_\ell^\star+u\Delta_\ell)=2\{\dot{\Pi}_\ell(u)\Delta_\ell-\Delta_\ell\dot{\Pi}_\ell(u)\}. ( Σ ℓ ⋆ + u Δ ℓ ) Π ¨ ℓ ( u ) − Π ¨ ℓ ( u ) ( Σ ℓ ⋆ + u Δ ℓ ) = 2 { Π ˙ ℓ ( u ) Δ ℓ − Δ ℓ Π ˙ ℓ ( u )} .
Applying the same blockwise argument as Step 3 , with the first identity controlling the diagonal blocks and the second controlling the off-diagonal blocks through the eigengap, together with (162) yields that
∥ Π ¨ ℓ ( u ) ∥ F ≤ C 2 r k ∥ Δ ℓ ∥ 2 λ r k ( S ℓ ) 2 , u ∈ [ 0 , 1 ] , \|\ddot{\Pi}_\ell(u)\|_F\le C_2\sqrt{r_k}\frac{\|\Delta_\ell\|^2}{\lambda_{r_k}(S_\ell)^2},\qquad u\in[0,1], ∥ Π ¨ ℓ ( u ) ∥ F ≤ C 2 r k λ r k ( S ℓ ) 2 ∥ Δ ℓ ∥ 2 , u ∈ [ 0 , 1 ] ,
where C 2 > 0 C_2 > 0 C 2 > 0 is an absolute constant. Since Π ℓ ( 0 ) = P \Pi_\ell(0) = P Π ℓ ( 0 ) = P , Π ℓ ( 1 ) = Π ^ j , k ( 0 ) , [ ℓ ] \Pi_\ell(1) = \widehat{\Pi}_{j,k}^{(0),[\ell]} Π ℓ ( 1 ) = Π j , k ( 0 ) , [ ℓ ] , and Π ˙ ℓ ( 0 ) = L ℓ \dot{\Pi}_\ell(0) = L_\ell Π ˙ ℓ ( 0 ) = L ℓ , Taylor’s formula yields
R ℓ = ∫ 0 1 ( 1 − u ) Π ¨ ℓ ( u ) d u . \mathcal{R}_\ell= \int_0^1 (1-u)\ddot{\Pi}_\ell(u)\,du. R ℓ = ∫ 0 1 ( 1 − u ) Π ¨ ℓ ( u ) d u .
Hence, on the event E g a p \mathcal{E}_{\mathrm{gap}} E gap , it holds that
∥ R ℓ ∥ F ≤ C r k ∥ Δ ℓ ∥ 2 λ r k ( S ℓ ) 2 . (163) \|\mathcal{R}_\ell\|_{\mathrm{F}} \le C\sqrt{r_k}\frac{\|\Delta_\ell\|^2}{\lambda_{r_k}(S_\ell)^2}. \tag*{(163)} ∥ R ℓ ∥ F ≤ C r k λ r k ( S ℓ ) 2 ∥ Δ ℓ ∥ 2 . (163)
It remains to verify the event E g a p \mathcal{E}_{\mathrm{gap}} E gap defined in (148) . Define
ζ j , k : = σ λ j , 1 ( k ) λ j , r k ( k ) p k b j + σ 2 λ j , r k ( k ) { p b j + p k b j } . \zeta_{j,k} := \frac{\sigma\sqrt{\lambda_{j,1}^{(k)}}}{\lambda_{j,r_k}^{(k)}}\sqrt{\frac{p_k}{b_j}}+\frac{\sigma^2}{\lambda_{j,r_k}^{(k)}}\left\{\sqrt{\frac{p}{b_j}}+\frac{p_k}{b_j}\right\}. ζ j , k := λ j , r k ( k ) σ λ j , 1 ( k ) b j p k + λ j , r k ( k ) σ 2 { b j p + b j p k } .
By Assumption 2 and the condition stated in Lemma 6, we have
ζ j , k ≤ C 3 ( C 0 − 1 / 2 + C 0 − 1 ) , \zeta_{j,k} \le C_3\left(C_0^{-1/2}+C_0^{-1}\right), ζ j , k ≤ C 3 ( C 0 − 1/2 + C 0 − 1 ) ,
where C 3 > 0 C_3 > 0 C 3 > 0 is an absolute constant. By (59) , there exists an absolute constant C 4 > 0 C_4 > 0 C 4 > 0 such that
P { ∥ Δ ℓ ∥ λ r k ( S ℓ ) > C 4 ζ j , k | P j ( 0 ) } ≤ 1 96 . (164) \mathbb{P}\left\{\frac{\|\Delta_\ell\|}{\lambda_{r_k}(S_\ell)}>C_4\zeta_{j,k}\,\middle|\,\mathcal{P}_j^{(0)}\right\}\le\frac{1}{96}. \tag*{(164)} P { λ r k ( S ℓ ) ∥ Δ ℓ ∥ > C 4 ζ j , k P j ( 0 ) } ≤ 96 1 . (164)
We choose C 0 C_0 C 0 sufficiently large that C 4 ζ j , k ≤ 1 / 4 C_4\zeta_{j,k}\le1/4 C 4 ζ j , k ≤ 1/4 . On the complement of the exceptional event in (164) ,
∥ Δ ℓ ∥ λ r k ( S ℓ ) ≤ C 4 ζ j , k ≤ 1 4 . \frac{\|\Delta_\ell\|}{\lambda_{r_k}(S_\ell)}\le C_4\zeta_{j,k}\le\frac{1}{4}. λ r k ( S ℓ ) ∥ Δ ℓ ∥ ≤ C 4 ζ j , k ≤ 4 1 .
Hence, conditional on P j ( 0 ) \mathcal{P}_j^{(0)} P j ( 0 ) , the event E g a p \mathcal{E}_{\mathrm{gap}} E gap defined in (148) holds with probability at least 95 / 96 95/96 95/96 . On this event, (163) gives
∥ R ℓ ∥ F ≤ C 2 r k { ∥ Δ ℓ ∥ λ r k ( S ℓ ) } 2 ≤ C 5 r k ζ j , k 2 , (165) \|\mathcal{R}_\ell\|_{\mathrm{F}}\le C_2\sqrt{r_k}\left\{\frac{\|\Delta_\ell\|}{\lambda_{r_k}(S_\ell)}\right\}^{2}\le C_5\sqrt{r_k}\zeta_{j,k}^{2}, \tag*{(165)} ∥ R ℓ ∥ F ≤ C 2 r k { λ r k ( S ℓ ) ∥ Δ ℓ ∥ } 2 ≤ C 5 r k ζ j , k 2 , (165)
where C 5 > 0 C_5 > 0 C 5 > 0 is an absolute constant. Note that
r k ζ j , k = λ j , 1 ( k ) λ j , r k ( k ) ϱ j , k . (166) \sqrt{r_k}\zeta_{j,k}=\frac{\lambda_{j,1}^{(k)}}{\lambda_{j,r_k}^{(k)}}\varrho_{j,k}. \tag*{(166)} r k ζ j , k = λ j , r k ( k ) λ j , 1 ( k ) ϱ j , k . (166)
Combining Assumption 2, (165) , and (166) , on the event E g a p \mathcal{E}_{\mathrm{gap}} E gap , we have
∥ R ℓ ∥ F ≤ C 6 ζ j , k ϱ j , k , \|\mathcal{R}_\ell\|_{\mathrm{F}}\le C_6\zeta_{j,k}\varrho_{j,k}, ∥ R ℓ ∥ F ≤ C 6 ζ j , k ϱ j , k ,
where C 6 > 0 C_6 > 0 C 6 > 0 is an absolute constant.
Fix a sufficiently small absolute constant c 1 > 0 c_1 > 0 c 1 > 0 . Since ζ j , k \zeta_{j,k} ζ j , k can be made uniformly small by increasing C 0 C_0 C 0 , choose C 0 C_0 C 0 further so that C 6 ζ j , k ≤ c 1 C_6\zeta_{j,k}\le c_1 C 6 ζ j , k ≤ c 1 . Hence we have
P { ∥ R ℓ ∥ F > c 1 ϱ j , k | P j ( 0 ) } ≤ 1 96 . (167) \mathbb{P}\left\{\|\mathcal{R}_\ell\|_{\mathrm{F}}>c_1\varrho_{j,k}\,\middle|\,\mathcal{P}_j^{(0)}\right\}\le\frac{1}{96}. \tag*{(167)} P { ∥ R ℓ ∥ F > c 1 ϱ j , k P j ( 0 ) } ≤ 96 1 . (167)
Step 5. In this step, we establish a lower bound of order ϱ j , k \varrho_{j,k} ϱ j , k for the distance between projectors from any two distinct pilot blocks, with probability at least 2 / 3 2/3 2/3 conditional on P j ( 0 ) \mathcal{P}^{(0)}_j P j ( 0 ) .
Fix two distinct pilot blocks I ℓ \mathcal{I}_{\ell} I ℓ and I h \mathcal{I}_{h} I h . By the definition of L ℓ \mathcal{L}_{\ell} L ℓ in (153) , we have
L ℓ − L h = U k ( K ℓ − K h ) A k T + A k ( K ℓ − K h ) T U k T . \mathcal{L}_{\ell}-\mathcal{L}_{h}=U_k(K_{\ell}-K_h)A_k^{\mathsf{T}}+A_k(K_{\ell}-K_h)^{\mathsf{T}}U_k^{\mathsf{T}}. L ℓ − L h = U k ( K ℓ − K h ) A k T + A k ( K ℓ − K h ) T U k T .
Hence
∥ L ℓ − L h ∥ F 2 = ∥ U k ( K ℓ − K h ) A k T + A k ( K ℓ − K h ) T U k T ∥ F 2 = 2 ∥ K ℓ − K h ∥ F 2 . \left\|\mathcal{L}_{\ell}-\mathcal{L}_{h}\right\|_{\mathrm{F}}^2=\left\|U_k(K_{\ell}-K_h)A_k^{\mathsf{T}}+A_k(K_{\ell}-K_h)^{\mathsf{T}}U_k^{\mathsf{T}}\right\|_{\mathrm{F}}^2=2\left\|K_{\ell}-K_h\right\|_{\mathrm{F}}^2. ∥ L ℓ − L h ∥ F 2 = U k ( K ℓ − K h ) A k T + A k ( K ℓ − K h ) T U k T F 2 = 2 ∥ K ℓ − K h ∥ F 2 .
Moreover, (154) gives
Π ^ j , k ( 0 ) , [ ℓ ] − Π ^ j , k ( 0 ) , [ h ] = ( L ℓ − L h ) + ( R ℓ − R h ) , \widehat{\Pi}^{(0),[\ell]}_{j,k}-\widehat{\Pi}^{(0),[h]}_{j,k}=(\mathcal{L}_{\ell}-\mathcal{L}_{h})+(\mathcal{R}_{\ell}-\mathcal{R}_{h}), Π j , k ( 0 ) , [ ℓ ] − Π j , k ( 0 ) , [ h ] = ( L ℓ − L h ) + ( R ℓ − R h ) ,
and therefore
∥ Π ^ j , k ( 0 ) , [ ℓ ] − Π ^ j , k ( 0 ) , [ h ] ∥ F ≥ 2 ∥ K ℓ − K h ∥ F − ∥ R ℓ ∥ F − ∥ R h ∥ F . (168) \left\|\widehat{\Pi}^{(0),[\ell]}_{j,k}-\widehat{\Pi}^{(0),[h]}_{j,k}\right\|_{\mathrm{F}}\ge\sqrt{2}\left\|K_{\ell}-K_h\right\|_{\mathrm{F}}-\left\|\mathcal{R}_{\ell}\right\|_{\mathrm{F}}-\left\|\mathcal{R}_{h}\right\|_{\mathrm{F}}. \tag*{(168)} Π j , k ( 0 ) , [ ℓ ] − Π j , k ( 0 ) , [ h ] F ≥ 2 ∥ K ℓ − K h ∥ F − ∥ R ℓ ∥ F − ∥ R h ∥ F . (168)
The remainders ∥ R ℓ ∥ F \left\|\mathcal{R}_{\ell}\right\|_{\mathrm{F}} ∥ R ℓ ∥ F and ∥ R h ∥ F \left\|\mathcal{R}_{h}\right\|_{\mathrm{F}} ∥ R h ∥ F are controlled by (167) . Hence it remains to argue that ∥ K ℓ − K h ∥ F \left\|K_{\ell}-K_h\right\|_{\mathrm{F}} ∥ K ℓ − K h ∥ F is unlikely to be much smaller than ϱ j , k \varrho_{j,k} ϱ j , k .
To this end, we derive a representation of K ℓ K_{\ell} K ℓ , and the same argument applies to K h K_h K h . Recall that
M k ( X t ( j ) ) = A k D t A − k T + E t , A − k = ⨂ u ∈ [ K ] ∖ { k } A u . \mathcal{M}_k(X_t^{(j)})=A_kD_tA_{-k}^{\mathsf{T}}+E_t,\qquad A_{-k}=\bigotimes_{u\in[K]\setminus\{k\}}A_u. M k ( X t ( j ) ) = A k D t A − k T + E t , A − k = u ∈ [ K ] ∖ { k } ⨂ A u .
Using s ℓ − 1 ∑ t ∈ I ℓ D t D t T = S ℓ s_{\ell}^{-1}\sum_{t\in\mathcal{I}_{\ell}}D_tD_t^{\mathsf{T}}=S_{\ell} s ℓ − 1 ∑ t ∈ I ℓ D t D t T = S ℓ , we can write the covariance perturbation as
Δ ℓ = 1 s ℓ ∑ t ∈ I ℓ { A k D t A − k T A − k D t T A k T + E t A − k D t T A k T + A k D t A − k T E t T + E t E t T − σ 2 p − k I p k } . \Delta_{\ell}=\frac{1}{s_{\ell}}\sum_{t\in\mathcal{I}_{\ell}}\left\{A_kD_tA_{-k}^{\mathsf{T}}A_{-k}D_t^{\mathsf{T}}A_k^{\mathsf{T}}+E_tA_{-k}D_t^{\mathsf{T}}A_k^{\mathsf{T}}+A_kD_tA_{-k}^{\mathsf{T}}E_t^{\mathsf{T}}+E_tE_t^{\mathsf{T}}-\sigma^2p_{-k}I_{p_k}\right\}. Δ ℓ = s ℓ 1 t ∈ I ℓ ∑ { A k D t A − k T A − k D t T A k T + E t A − k D t T A k T + A k D t A − k T E t T + E t E t T − σ 2 p − k I p k } .
Multiplying on the left by U k T U_k^{\mathsf{T}} U k T and on the right by A k A_k A k , the first term vanishes as U k T A k = 0 U_k^{\mathsf{T}}A_k=0 U k T A k = 0 . The constant part of the last term vanishes for the same reason. The remaining terms give
U k T Δ ℓ A k = 1 s ℓ ∑ t ∈ I ℓ U k T E t ( A − k D t T + E t T A k ) . U_k^{\mathsf{T}}\Delta_{\ell}A_k=\frac{1}{s_{\ell}}\sum_{t\in\mathcal{I}_{\ell}}U_k^{\mathsf{T}}E_t\left(A_{-k}D_t^{\mathsf{T}}+E_t^{\mathsf{T}}A_k\right). U k T Δ ℓ A k = s ℓ 1 t ∈ I ℓ ∑ U k T E t ( A − k D t T + E t T A k ) .
Define
Z t : = U k T E t ∈ R q k × p − k , H t : = A − k D t T + E t T A k ∈ R p − k × r k . Z_t:=U_k^{\mathsf{T}}E_t\in\mathbb{R}^{q_k\times p_{-k}},\qquad H_t:=A_{-k}D_t^{\mathsf{T}}+E_t^{\mathsf{T}}A_k\in\mathbb{R}^{p_{-k}\times r_k}. Z t := U k T E t ∈ R q k × p − k , H t := A − k D t T + E t T A k ∈ R p − k × r k .
As K ℓ = U k T Δ ℓ A k S ℓ − 1 K_{\ell}=U_k^{\mathsf{T}}\Delta_{\ell}A_kS_{\ell}^{-1} K ℓ = U k T Δ ℓ A k S ℓ − 1 , we have
K ℓ = 1 s ℓ ∑ t ∈ I ℓ Z t H t S ℓ − 1 . (169) K_{\ell}=\frac{1}{s_{\ell}}\sum_{t\in\mathcal{I}_{\ell}}Z_tH_tS_{\ell}^{-1}. \tag*{(169)} K ℓ = s ℓ 1 t ∈ I ℓ ∑ Z t H t S ℓ − 1 . (169)
We next obtain the conditional distribution of K ℓ − K h K_{\ell}-K_h K ℓ − K h . Conditional on P j ( 0 ) \mathcal{P}^{(0)}_j P j ( 0 ) , the D t D_t D t are fixed. The Gaussian projections U k T E t U_k^{\mathsf{T}}E_t U k T E t and A k T E t A_k^{\mathsf{T}}E_t A k T E t have cross-covariance
Cov { vec ( U k T E t ) , vec ( A k T E t ) } = σ 2 ( I p − k ⊗ U k T A k ) = 0. \operatorname{Cov}\left\{\operatorname{vec}(U_k^{\mathsf{T}}E_t),\operatorname{vec}(A_k^{\mathsf{T}}E_t)\right\}=\sigma^2\left(I_{p_{-k}}\otimes U_k^{\mathsf{T}}A_k\right)=0. Cov { vec ( U k T E t ) , vec ( A k T E t ) } = σ 2 ( I p − k ⊗ U k T A k ) = 0.
They are therefore independent. In particular, Z t Z_t Z t is independent of H t H_t H t . Conditional on P j ( 0 ) \mathcal{P}^{(0)}_j P j ( 0 ) and the H t H_t H t in block I ℓ \mathcal{I}_\ell I ℓ , each row of (169) is a centered Gaussian vector with covariance
V ℓ : = σ 2 s ℓ 2 S ℓ − 1 ( ∑ t ∈ I ℓ H t T H t ) S ℓ − 1 . (170) V_\ell:= \frac{\sigma^2}{s_\ell^2}S_\ell^{-1}\left(\sum_{t\in\mathcal{I}_\ell}H_t^{\mathsf{T}}H_t\right)S_\ell^{-1}. \tag*{(170)} V ℓ := s ℓ 2 σ 2 S ℓ − 1 ( t ∈ I ℓ ∑ H t T H t ) S ℓ − 1 . (170)
The rows are independent. The same statement holds for K h K_h K h , with row covariance V h V_h V h . Since the two blocks are disjoint, K ℓ K_\ell K ℓ and K h K_h K h are conditionally independent given P j ( 0 ) \mathcal{P}^{(0)}_j P j ( 0 ) and the H t H_t H t in both blocks. Thus K ℓ − K h K_\ell-K_h K ℓ − K h has independent centered Gaussian rows, each with covariance V ℓ + V h V_\ell+V_h V ℓ + V h . Since it has q k q_k q k rows,
v ℓ h 2 : = E { ∥ K ℓ − K h ∥ F 2 ∣ P j ( 0 ) , { H t : t ∈ I ℓ ∪ I h } } = q k tr ( V ℓ + V h ) . (171) v_{\ell h}^2 := \mathbb{E}\left\{\left\|K_\ell-K_h\right\|_{\mathrm{F}}^2 \mid\mathcal{P}^{(0)}_j,\{H_t:t\in\mathcal{I}_\ell\cup\mathcal{I}_h\}\right\}=q_k\operatorname{tr}(V_\ell+V_h). \tag*{(171)} v ℓ h 2 := E { ∥ K ℓ − K h ∥ F 2 ∣ P j ( 0 ) , { H t : t ∈ I ℓ ∪ I h } } = q k tr ( V ℓ + V h ) . (171)
We now show that this conditional second moment is of order ϱ j , k 2 \varrho_{j,k}^2 ϱ j , k 2 . For one block, define
Q ℓ : = ∑ t ∈ I ℓ ∥ H t S ℓ − 1 ∥ F 2 . Q_\ell:= \sum_{t\in\mathcal{I}_\ell}\left\|H_tS_\ell^{-1}\right\|_{\mathrm{F}}^2. Q ℓ := t ∈ I ℓ ∑ H t S ℓ − 1 F 2 .
Taking the trace in (170) gives
tr ( V ℓ ) = σ 2 s ℓ 2 Q ℓ . (172) \operatorname{tr}(V_\ell)=\frac{\sigma^2}{s_\ell^2}Q_\ell. \tag*{(172)} tr ( V ℓ ) = s ℓ 2 σ 2 Q ℓ . (172)
Thus it suffices to determine the size of Q ℓ Q_\ell Q ℓ .
Conditional on P j ( 0 ) \mathcal{P}^{(0)}_j P j ( 0 ) , the matrices D t D_t D t and S ℓ S_\ell S ℓ are fixed, while the matrices E t T A k E_t^{\mathsf{T}}A_k E t T A k have independent N ( 0 , σ 2 ) \mathcal{N}(0,\sigma^2) N ( 0 , σ 2 ) entries and are independent across t t t . Since H t = A − k D t T + E t T A k H_t=A_{-k}D_t^{\mathsf{T}}+E_t^{\mathsf{T}}A_k H t = A − k D t T + E t T A k , expanding ∥ H t S ℓ − 1 ∥ F 2 \left\|H_tS_\ell^{-1}\right\|_{\mathrm{F}}^2 H t S ℓ − 1 F 2 gives
E ( ∥ H t S ℓ − 1 ∥ F 2 ∣ P j ( 0 ) ) = ∥ A − k D t T S ℓ − 1 ∥ F 2 + p − k σ 2 tr ( S ℓ − 2 ) , \mathbb{E}\left(\left\|H_tS_\ell^{-1}\right\|_{\mathrm{F}}^2\mid\mathcal{P}^{(0)}_j\right)=\left\|A_{-k}D_t^{\mathsf{T}}S_\ell^{-1}\right\|_{\mathrm{F}}^2+p_{-k}\sigma^2\operatorname{tr}(S_\ell^{-2}), E ( H t S ℓ − 1 F 2 ∣ P j ( 0 ) ) = A − k D t T S ℓ − 1 F 2 + p − k σ 2 tr ( S ℓ − 2 ) ,
Var ( ∥ H t S ℓ − 1 ∥ F 2 ∣ P j ( 0 ) ) = 4 σ 2 ∥ A − k D t T S ℓ − 2 ∥ F 2 + 2 p − k σ 4 tr ( S ℓ − 4 ) . \operatorname{Var}\left(\left\|H_tS_\ell^{-1}\right\|_{\mathrm{F}}^2\mid\mathcal{P}^{(0)}_j\right)=4\sigma^2\left\|A_{-k}D_t^{\mathsf{T}}S_\ell^{-2}\right\|_{\mathrm{F}}^2+2p_{-k}\sigma^4\operatorname{tr}(S_\ell^{-4}). Var ( H t S ℓ − 1 F 2 ∣ P j ( 0 ) ) = 4 σ 2 A − k D t T S ℓ − 2 F 2 + 2 p − k σ 4 tr ( S ℓ − 4 ) .
Moreover, A − k T A − k = I A_{-k}^{\mathsf{T}}A_{-k}=I A − k T A − k = I and ∑ t ∈ I ℓ D t D t T = s ℓ S ℓ \sum_{t\in\mathcal{I}_\ell}D_tD_t^{\mathsf{T}}=s_\ell S_\ell ∑ t ∈ I ℓ D t D t T = s ℓ S ℓ imply
∑ t ∈ I ℓ ∥ A − k D t T S ℓ − 1 ∥ F 2 = s ℓ tr ( S ℓ − 1 ) , ∑ t ∈ I ℓ ∥ A − k D t T S ℓ − 2 ∥ F 2 = s ℓ tr ( S ℓ − 3 ) . \sum_{t\in\mathcal{I}_\ell}\left\|A_{-k}D_t^{\mathsf{T}}S_\ell^{-1}\right\|_{\mathrm{F}}^2=s_\ell\operatorname{tr}(S_\ell^{-1}),\qquad
\sum_{t\in\mathcal{I}_\ell}\left\|A_{-k}D_t^{\mathsf{T}}S_\ell^{-2}\right\|_{\mathrm{F}}^2=s_\ell\operatorname{tr}(S_\ell^{-3}). t ∈ I ℓ ∑ A − k D t T S ℓ − 1 F 2 = s ℓ tr ( S ℓ − 1 ) , t ∈ I ℓ ∑ A − k D t T S ℓ − 2 F 2 = s ℓ tr ( S ℓ − 3 ) .
Summing the preceding mean and variance identities over the independent observations t ∈ I ℓ t\in\mathcal{I}_\ell t ∈ I ℓ yields
E ( Q ℓ ∣ P j ( 0 ) ) = s ℓ tr ( S ℓ − 1 ) + s ℓ p − k σ 2 tr ( S ℓ − 2 ) , (173) \mathbb{E}(Q_\ell\mid\mathcal{P}^{(0)}_j)=s_\ell\operatorname{tr}(S_\ell^{-1})+s_\ell p_{-k}\sigma^2\operatorname{tr}(S_\ell^{-2}), \tag*{(173)} E ( Q ℓ ∣ P j ( 0 ) ) = s ℓ tr ( S ℓ − 1 ) + s ℓ p − k σ 2 tr ( S ℓ − 2 ) , (173)
Var ( Q ℓ ∣ P j ( 0 ) ) = 4 s ℓ σ 2 tr ( S ℓ − 3 ) + 2 s ℓ p − k σ 4 tr ( S ℓ − 4 ) . (174) \operatorname{Var}(Q_\ell\mid\mathcal{P}^{(0)}_j)=4s_\ell\sigma^2\operatorname{tr}(S_\ell^{-3})+2s_\ell p_{-k}\sigma^4\operatorname{tr}(S_\ell^{-4}). \tag*{(174)} Var ( Q ℓ ∣ P j ( 0 ) ) = 4 s ℓ σ 2 tr ( S ℓ − 3 ) + 2 s ℓ p − k σ 4 tr ( S ℓ − 4 ) . (174)
By (172) and (173) , we have
q k E { tr ( V ℓ ) ∣ P j ( 0 ) } = q k σ 2 s ℓ { tr ( S ℓ − 1 ) + p − k σ 2 tr ( S ℓ − 2 ) } . q_k\mathbb{E}\{\operatorname{tr}(V_\ell)\mid\mathcal{P}^{(0)}_j\}=\frac{q_k\sigma^2}{s_\ell}\left\{\operatorname{tr}(S_\ell^{-1})+p_{-k}\sigma^2\operatorname{tr}(S_\ell^{-2})\right\}. q k E { tr ( V ℓ ) ∣ P j ( 0 ) } = s ℓ q k σ 2 { tr ( S ℓ − 1 ) + p − k σ 2 tr ( S ℓ − 2 ) } .
By (145) and Assumption 2,
2 r k 3 λ j , 1 ( k ) ≤ tr ( S ℓ − 1 ) ≤ 2 C χ r k λ j , 1 ( k ) , 4 r k 9 ( λ j , 1 ( k ) ) 2 ≤ tr ( S ℓ − 2 ) ≤ 4 C 2 χ 2 r k ( λ j , 1 ( k ) ) 2 . \frac{2r_k}{3\lambda_{j,1}^{(k)}}\leq\operatorname{tr}(S_\ell^{-1})\leq\frac{2C\chi r_k}{\lambda_{j,1}^{(k)}},\qquad
\frac{4r_k}{9(\lambda_{j,1}^{(k)})^2}\leq\operatorname{tr}(S_\ell^{-2})\leq\frac{4C^2\chi^2r_k}{(\lambda_{j,1}^{(k)})^2}. 3 λ j , 1 ( k ) 2 r k ≤ tr ( S ℓ − 1 ) ≤ λ j , 1 ( k ) 2 C χ r k , 9 ( λ j , 1 ( k ) ) 2 4 r k ≤ tr ( S ℓ − 2 ) ≤ ( λ j , 1 ( k ) ) 2 4 C 2 χ 2 r k .
Since p k / 2 ≤ q k ≤ p k p_k/2 \le q_k \le p_k p k /2 ≤ q k ≤ p k and b j ≤ s ℓ ≤ 2 b j b_j \le s_\ell\le2b_j b j ≤ s ℓ ≤ 2 b j , it follows that
r k 9 { σ 2 p k b j λ j , 1 ( k ) + σ 4 p b j ( λ j , 1 ( k ) ) 2 } ≤ q k E { tr ( V ℓ ) ∣ P j ( 0 ) } ≤ 4 C λ 2 r k { σ 2 p k b j λ j , 1 ( k ) + σ 4 p b j ( λ j , 1 ( k ) ) 2 } . (175) \frac{r_k}{9}\left\{\frac{\sigma^2p_k}{b_j\lambda_{j,1}^{(k)}}+\frac{\sigma^4p}{b_j(\lambda_{j,1}^{(k)})^2}\right\}\le q_k\mathbb{E}\{\operatorname{tr}(V_\ell)\mid\mathcal{P}_j^{(0)}\}\le4C_\lambda^2r_k\left\{\frac{\sigma^2p_k}{b_j\lambda_{j,1}^{(k)}}+\frac{\sigma^4p}{b_j(\lambda_{j,1}^{(k)})^2}\right\}. \tag*{(175)} 9 r k { b j λ j , 1 ( k ) σ 2 p k + b j ( λ j , 1 ( k ) ) 2 σ 4 p } ≤ q k E { tr ( V ℓ ) ∣ P j ( 0 ) } ≤ 4 C λ 2 r k { b j λ j , 1 ( k ) σ 2 p k + b j ( λ j , 1 ( k ) ) 2 σ 4 p } . (175)
To compare these bounds with the intrinsic radius, recall that
ϱ j , k r k = σ λ j , 1 ( k ) p k b j + σ 2 λ j , 1 ( k ) p b j + σ 2 p k b j λ j , 1 ( k ) . (176) \frac{\varrho_{j,k}}{\sqrt{r_k}}=\frac{\sigma}{\sqrt{\lambda_{j,1}^{(k)}}}\sqrt{\frac{p_k}{b_j}}+\frac{\sigma^2}{\sqrt{\lambda_{j,1}^{(k)}}}\sqrt{\frac{p}{b_j}}+\frac{\sigma^2p_k}{b_j\lambda_{j,1}^{(k)}}. \tag*{(176)} r k ϱ j , k = λ j , 1 ( k ) σ b j p k + λ j , 1 ( k ) σ 2 b j p + b j λ j , 1 ( k ) σ 2 p k . (176)
By the condition stated in Lemma 6 , we have
σ 2 p k b j λ j , 1 ( k ) ≤ 1 C 0 C b ≤ 1. \frac{\sigma^2p_k}{b_j\lambda_{j,1}^{(k)}}\le\frac{1}{C_0C_b}\le1. b j λ j , 1 ( k ) σ 2 p k ≤ C 0 C b 1 ≤ 1.
The last summand in ϱ j , k / r k \varrho_{j,k}/\sqrt{r_k} ϱ j , k / r k is the square of the first and is therefore no larger than the first. Squaring (176) gives
r k { σ 2 p k b j λ j , 1 ( k ) + σ 4 p b j ( λ j , 1 ( k ) ) 2 } ≤ ϱ j , k 2 ≤ 5 r k { σ 2 p k b j λ j , 1 ( k ) + σ 4 p b j ( λ j , 1 ( k ) ) 2 } . (177) r_k\left\{\frac{\sigma^2p_k}{b_j\lambda_{j,1}^{(k)}}+\frac{\sigma^4p}{b_j(\lambda_{j,1}^{(k)})^2}\right\}\le\varrho_{j,k}^2\le5r_k\left\{\frac{\sigma^2p_k}{b_j\lambda_{j,1}^{(k)}}+\frac{\sigma^4p}{b_j(\lambda_{j,1}^{(k)})^2}\right\}. \tag*{(177)} r k { b j λ j , 1 ( k ) σ 2 p k + b j ( λ j , 1 ( k ) ) 2 σ 4 p } ≤ ϱ j , k 2 ≤ 5 r k { b j λ j , 1 ( k ) σ 2 p k + b j ( λ j , 1 ( k ) ) 2 σ 4 p } . (177)
Combining (175) and (177) , we obtain
1 45 ϱ j , k 2 ≤ q k E { tr ( V ℓ ) ∣ P j ( 0 ) } ≤ 4 C λ 2 ϱ j , k 2 . (178) \frac{1}{45}\varrho_{j,k}^2\le q_k\mathbb{E}\{\operatorname{tr}(V_\ell)\mid\mathcal{P}_j^{(0)}\}\le4C_\lambda^2\varrho_{j,k}^2. \tag*{(178)} 45 1 ϱ j , k 2 ≤ q k E { tr ( V ℓ ) ∣ P j ( 0 ) } ≤ 4 C λ 2 ϱ j , k 2 . (178)
Next, we check that Q ℓ Q_\ell Q ℓ is close to its mean. By (173) , (174) , (145) , and Assumption 2, we have
Var ( Q ℓ ∣ P j ( 0 ) ) { E ( Q ℓ ∣ P j ( 0 ) ) } 2 ≤ 24 C 7 σ 2 s ℓ r k λ j , r k ( k ) ≤ C 7 C 0 p k r k . \frac{\operatorname{Var}(Q_\ell\mid\mathcal{P}_j^{(0)})}{\{\mathbb{E}(Q_\ell\mid\mathcal{P}_j^{(0)})\}^2}\le\frac{24C_7\sigma^2}{s_\ell r_k\lambda_{j,r_k}^{(k)}}\le\frac{C_7}{C_0p_kr_k}. { E ( Q ℓ ∣ P j ( 0 ) ) } 2 Var ( Q ℓ ∣ P j ( 0 ) ) ≤ s ℓ r k λ j , r k ( k ) 24 C 7 σ 2 ≤ C 0 p k r k C 7 .
where C 7 > 0 C_7>0 C 7 > 0 is an absolute constant and the last inequality used the condition stated in Lemma 6 . Hence after increasing C 0 C_0 C 0 if necessary, by Chebyshev’s inequality we have
P { ∣ Q ℓ − E ( Q ℓ ∣ P j ( 0 ) ) ∣ ≥ 1 2 E ( Q ℓ ∣ P j ( 0 ) ) ∣ P j ( 0 ) } ≤ 4 Var ( Q ℓ ∣ P j ( 0 ) ) { E ( Q ℓ ∣ P j ( 0 ) ) } 2 ≤ 1 96 . (179) \mathbb{P}\left\{\left|Q_\ell-\mathbb{E}(Q_\ell\mid\mathcal{P}_j^{(0)})\right|\ge\frac{1}{2}\mathbb{E}(Q_\ell\mid\mathcal{P}_j^{(0)})\mid\mathcal{P}_j^{(0)}\right\}\le\frac{4\operatorname{Var}(Q_\ell\mid\mathcal{P}_j^{(0)})}{\{\mathbb{E}(Q_\ell\mid\mathcal{P}_j^{(0)})\}^2}\le\frac{1}{96}. \tag*{(179)} P { Q ℓ − E ( Q ℓ ∣ P j ( 0 ) ) ≥ 2 1 E ( Q ℓ ∣ P j ( 0 ) ) ∣ P j ( 0 ) } ≤ { E ( Q ℓ ∣ P j ( 0 ) ) } 2 4 Var ( Q ℓ ∣ P j ( 0 ) ) ≤ 96 1 . (179)
Since q k tr ( V ℓ ) = q k σ 2 Q ℓ / s ℓ 2 q_k\operatorname{tr}(V_\ell)=q_k\sigma^2Q_\ell/s_\ell^2 q k tr ( V ℓ ) = q k σ 2 Q ℓ / s ℓ 2 , (178) shows that, outside this exceptional event in (179) ,
1 90 ϱ j , k 2 ≤ q k tr ( V ℓ ) ≤ 6 C λ 2 ϱ j , k 2 . \frac{1}{90}\varrho_{j,k}^2\le q_k\operatorname{tr}(V_\ell)\le6C_\lambda^2\varrho_{j,k}^2. 90 1 ϱ j , k 2 ≤ q k tr ( V ℓ ) ≤ 6 C λ 2 ϱ j , k 2 .
The same argument applies to block h h h . A union bound over the two blocks gives
P ( E ℓ h ∣ P j ( 0 ) ) ≥ 47 48 , \mathbb{P}(\mathcal{E}_{\ell h}\mid\mathcal{P}_j^{(0)})\ge\frac{47}{48}, P ( E ℓ h ∣ P j ( 0 ) ) ≥ 48 47 ,
where E ℓ h : = { c ′ ′ ϱ j , k 2 ≤ 2 q k tr ( V ℓ + V h ) ≤ C ′ ′ ϱ j , k 2 } \mathcal{E}_{\ell h}:=\{c^{\prime\prime}\varrho_{j,k}^2\le2q_k\operatorname{tr}(V_\ell+V_h)\le C^{\prime\prime}\varrho_{j,k}^2\} E ℓ h := { c ′′ ϱ j , k 2 ≤ 2 q k tr ( V ℓ + V h ) ≤ C ′′ ϱ j , k 2 } , (180)
for some absolute constants C ′ ′ ≥ c ′ ′ > 0 C^{\prime\prime}\ge c^{\prime\prime}>0 C ′′ ≥ c ′′ > 0 . In view of (171) , this establishes the typical conditional second-moment scale of K ℓ − K h K_\ell-K_h K ℓ − K h .
If G G G is a centered Gaussian vector with v 2 = E ∥ G ∥ 2 2 > 0 v^2 = \mathbb{E}\lVert G\rVert_2^2 > 0 v 2 = E ∥ G ∥ 2 2 > 0 , Theorem 8 in Carbery and Wright [15 ] , applied to the nonnegative quadratic form ∥ G ∥ 2 2 \lVert G\rVert_2^2 ∥ G ∥ 2 2 , gives
P { ∥ G ∥ 2 2 ≤ u v } ≤ C 8 u , u ∈ ( 0 , 1 ) , (180) \mathbb{P}\{\lVert G\rVert_2^2 \le uv\} \le C_8u,\quad u \in(0,1), \tag*{(180)} P {∥ G ∥ 2 2 ≤ uv } ≤ C 8 u , u ∈ ( 0 , 1 ) , (180)
for an absolute constant C 8 > 0 C_8 > 0 C 8 > 0 . On the event E ℓ h \mathcal{E}_{\ell h} E ℓ h in (180), (171) implies
v ℓ h ≥ c ′ ′ 2 ϱ j , k . v_{\ell h} \ge\sqrt{\frac{c^{\prime\prime}}{2}}\varrho_{j,k}. v ℓ h ≥ 2 c ′′ ϱ j , k .
Fix u ∈ ( 0 , 1 ) u \in(0,1) u ∈ ( 0 , 1 ) and set c 3 = u c ′ ′ / 2 c_3 = u\sqrt{c^{\prime\prime}/2} c 3 = u c ′′ /2 . Conditional on P j ( 0 ) \mathcal{P}^{(0)}_j P j ( 0 ) and the H t H_t H t in the two blocks, apply (180) to K ℓ − K h K_\ell-K_h K ℓ − K h . On the event E ℓ h \mathcal{E}_{\ell h} E ℓ h in (180), this gives
P { ∥ K ℓ − K h ∥ F ≤ c 3 ϱ j , k | P j ( 0 ) , { H t : t ∈ I ℓ ∪ I h } } ≤ C 8 u . \mathbb{P}\left\{\lVert K_\ell-K_h\rVert_F \le c_3\varrho_{j,k}\,\middle|\,\mathcal{P}^{(0)}_j,\{H_t:t\in\mathcal{I}_\ell\cup\mathcal{I}_h\}\right\}\le C_8u. P { ∥ K ℓ − K h ∥ F ≤ c 3 ϱ j , k P j ( 0 ) , { H t : t ∈ I ℓ ∪ I h } } ≤ C 8 u .
Hence we have
P { ∥ K ℓ − K h ∥ F ≤ c 3 ϱ j , k | P j ( 0 ) } ≤ C 8 u P ( E ℓ h ∣ P j ( 0 ) ) + P ( E ℓ h c ∣ P j ( 0 ) ) ≤ C 8 u + 1 48 . (181) \mathbb{P}\left\{\lVert K_\ell-K_h\rVert_F\le c_3\varrho_{j,k}\,\middle|\,\mathcal{P}^{(0)}_j\right\}\le C_8u\mathbb{P}\left(\mathcal{E}_{\ell h}\mid\mathcal{P}^{(0)}_j\right)+\mathbb{P}\left(\mathcal{E}_{\ell h}^{c}\mid\mathcal{P}^{(0)}_j\right)\le C_8u+\frac{1}{48}. \tag*{(181)} P { ∥ K ℓ − K h ∥ F ≤ c 3 ϱ j , k P j ( 0 ) } ≤ C 8 u P ( E ℓ h ∣ P j ( 0 ) ) + P ( E ℓ h c ∣ P j ( 0 ) ) ≤ C 8 u + 48 1 . (181)
Choose u ∈ ( 0 , 1 ) u\in(0,1) u ∈ ( 0 , 1 ) sufficiently small such that C 8 u + 1 / 24 ≤ 1 / 3 C_8u+1/24\le1/3 C 8 u + 1/24 ≤ 1/3 . Choose c 1 < c 3 / 2 c_1<c_3/\sqrt{2} c 1 < c 3 / 2 and increase C 0 C_0 C 0 as required by (167) . Then
c − : = 2 c 3 − 2 c 1 > 0. c_-:=\sqrt{2}c_3-2c_1>0. c − := 2 c 3 − 2 c 1 > 0.
If ∥ K ℓ − K h ∥ F > c 3 ϱ j , k \lVert K_\ell-K_h\rVert_F>c_3\varrho_{j,k} ∥ K ℓ − K h ∥ F > c 3 ϱ j , k and both remainders have Frobenius norm at most c 1 ϱ j , k c_1\varrho_{j,k} c 1 ϱ j , k in (167) , then (168) gives
∥ Π ^ j , k ( 0 ) , [ ℓ ] − Π ^ j , k ( 0 ) , [ h ] ∥ F > c − ϱ j , k . \left\lVert\widehat{\Pi}^{(0),[\ell]}_{j,k}-\widehat{\Pi}^{(0),[h]}_{j,k}\right\rVert_F>c_-\varrho_{j,k}. Π j , k ( 0 ) , [ ℓ ] − Π j , k ( 0 ) , [ h ] F > c − ϱ j , k .
Hence, by (167) and (181) , we have
P { ∥ Π ^ j , k ( 0 ) , [ ℓ ] − Π ^ j , k ( 0 ) , [ h ] ∥ F ≤ c − ϱ j , k | P j ( 0 ) } ≤ P { ∥ K ℓ − K h ∥ F ≤ c 3 ϱ j , k | P j ( 0 ) } + P { ∥ R ℓ ∥ F > c 1 ϱ j , k | P j ( 0 ) } + P { ∥ R h ∥ F > c 1 ϱ j , k | P j ( 0 ) } ≤ C 8 u + 1 48 + 2 96 = C 8 u + 1 24 ≤ 1 3 . \begin{aligned}
&\mathbb{P}\left\{\left\lVert\widehat{\Pi}^{(0),[\ell]}_{j,k}-\widehat{\Pi}^{(0),[h]}_{j,k}\right\rVert_F\le c_-\varrho_{j,k}\,\middle|\,\mathcal{P}^{(0)}_j\right\}\\
&\le\mathbb{P}\left\{\lVert K_\ell-K_h\rVert_F\le c_3\varrho_{j,k}\,\middle|\,\mathcal{P}^{(0)}_j\right\}+\mathbb{P}\left\{\lVert R_\ell\rVert_F>c_1\varrho_{j,k}\,\middle|\,\mathcal{P}^{(0)}_j\right\}+\mathbb{P}\left\{\lVert R_h\rVert_F>c_1\varrho_{j,k}\,\middle|\,\mathcal{P}^{(0)}_j\right\}\\
&\le C_8u+\frac{1}{48}+\frac{2}{96}=C_8u+\frac{1}{24}\le\frac{1}{3}.
\end{aligned} P { Π j , k ( 0 ) , [ ℓ ] − Π j , k ( 0 ) , [ h ] F ≤ c − ϱ j , k P j ( 0 ) } ≤ P { ∥ K ℓ − K h ∥ F ≤ c 3 ϱ j , k P j ( 0 ) } + P { ∥ R ℓ ∥ F > c 1 ϱ j , k P j ( 0 ) } + P { ∥ R h ∥ F > c 1 ϱ j , k P j ( 0 ) } ≤ C 8 u + 48 1 + 96 2 = C 8 u + 24 1 ≤ 3 1 .
Consequently, for every pair of distinct blocks,
P { ∥ Π ^ j , k ( 0 ) , [ ℓ ] − Π ^ j , k ( 0 ) , [ h ] ∥ F > c − ϱ j , k | P j ( 0 ) } ≥ 2 3 . (182) \mathbb{P}\left\{\left\lVert\widehat{\Pi}^{(0),[\ell]}_{j,k}-\widehat{\Pi}^{(0),[h]}_{j,k}\right\rVert_F>c_-\varrho_{j,k}\,\middle|\,\mathcal{P}^{(0)}_j\right\}\ge\frac{2}{3}. \tag*{(182)} P { Π j , k ( 0 ) , [ ℓ ] − Π j , k ( 0 ) , [ h ] F > c − ϱ j , k P j ( 0 ) } ≥ 3 2 . (182)
Step 6. Finally, we conclude this proof as follows. Since every diagonal distance is zero, the definition of G j , k G_{j,k} G j , k gives, for d ≥ 0 d\ge0 d ≥ 0 ,
G j , k ( d ) = 1 B j ( 0 ) + 1 ( B j ( 0 ) ) 2 ∑ ℓ , h ∈ [ B j ( 0 ) ] ℓ ≠ h P { ∥ Π ^ j , k ( 0 ) , [ ℓ ] − Π ^ j , k ( 0 ) , [ h ] ∥ F ≤ d | P j ( 0 ) } . G_{j,k}(d)=\frac{1}{B_j^{(0)}}+\frac{1}{(B_j^{(0)})^2}\sum_{\substack{\ell,h\in[B_j^{(0)}]\\\ell\ne h}}\mathbb{P}\left\{\left\lVert\widehat{\Pi}^{(0),[\ell]}_{j,k}-\widehat{\Pi}^{(0),[h]}_{j,k}\right\rVert_F\le d\,\middle|\,\mathcal{P}^{(0)}_j\right\}. G j , k ( d ) = B j ( 0 ) 1 + ( B j ( 0 ) ) 2 1 ℓ , h ∈ [ B j ( 0 ) ] ℓ = h ∑ P { Π j , k ( 0 ) , [ ℓ ] − Π j , k ( 0 ) , [ h ] F ≤ d P j ( 0 ) } .
For each off-diagonal pair, (182) bounds the probability in this sum by 1 / 3 1/3 1/3 when d = c − ⋅ ϱ j , k d=c_{-}\cdot\varrho_{j,k} d = c − ⋅ ϱ j , k . Thus, using B j ( 0 ) ≥ 8 B_j^{(0)}\ge8 B j ( 0 ) ≥ 8 , we have
G j , k ( c − ϱ j , k ) ≤ 1 B j ( 0 ) + ( 1 − 1 B j ( 0 ) ) 1 3 ≤ 5 12 ≤ 3 4 . (183) G_{j,k}(c_{-}\varrho_{j,k})\le\frac{1}{B_j^{(0)}}+\left(1-\frac{1}{B_j^{(0)}}\right)\frac{1}{3}\le\frac{5}{12}\le\frac{3}{4}. \tag*{(183)} G j , k ( c − ϱ j , k ) ≤ B j ( 0 ) 1 + ( 1 − B j ( 0 ) 1 ) 3 1 ≤ 12 5 ≤ 4 3 . (183)
Similarly, (147) bounds each off-diagonal probability below by 31 / 32 31/32 31/32 when d = c + ⋅ ϱ j , k d=c_{+}\cdot\varrho_{j,k} d = c + ⋅ ϱ j , k . Therefore,
G j , k ( c + ϱ j , k ) ≥ 1 B j ( 0 ) + ( 1 − 1 B j ( 0 ) ) 31 32 ≥ 31 32 ≥ 15 16 . (184) G_{j,k}(c_{+}\varrho_{j,k})\ge\frac{1}{B_j^{(0)}}+\left(1-\frac{1}{B_j^{(0)}}\right)\frac{31}{32}\ge\frac{31}{32}\ge\frac{15}{16}. \tag*{(184)} G j , k ( c + ϱ j , k ) ≥ B j ( 0 ) 1 + ( 1 − B j ( 0 ) 1 ) 32 31 ≥ 32 31 ≥ 16 15 . (184)
Combining (183) and (184) , we complete the proof of (144) , which concludes the proof of this lemma. □
Lemma 7. Suppose that Assumptions 1–3 hold. We use the block sizes and public grid specified in Theorem 3. Suppose also p k ≥ 2 r k p_k\ge2r_k p k ≥ 2 r k , p k ≥ C p log ( n ∨ p ) p_k\ge C_p\log(n\vee p) p k ≥ C p log ( n ∨ p ) , and (24) holds. Let c − c_{-} c − and c + c_{+} c + be the constants in Lemma 6 , and let C c m p C_{\mathrm{cmp}} C cmp be defined in (142) . Choose the fixed calibration constant C A C_A C A and the constant c 0 c_0 c 0 in the upper signal bound so that
C A ≥ C c m p C R c − , 0 < c 0 ≤ c − 2 C b + 1 . C_A\ge\frac{C_{\mathrm{cmp}}C_R}{c_{-}},\qquad0<c_0\le\frac{c_{-}^{2}}{C_b+1}. C A ≥ c − C cmp C R , 0 < c 0 ≤ C b + 1 c − 2 . Then there exist absolute constants C 1 , C ′ , C , c > 0 C_1,C',C,c>0 C 1 , C ′ , C , c > 0 such that, if ε j ∈ ( 0 , 1 ] \varepsilon_j\in(0,1] ε j ∈ ( 0 , 1 ] and
n j ε j ≥ C 1 { r k + log ( n ∨ p ) } log { m ( n ∨ p ) c + 1 } , j ∈ [ m ] , (185) n_j\varepsilon_j\ge C_1\{r_k+\log(n\vee p)\}\log\{m(n\vee p)^{c+1}\},\qquad j\in[m], \tag*{(185)} n j ε j ≥ C 1 { r k + log ( n ∨ p )} log { m ( n ∨ p ) c + 1 } , j ∈ [ m ] , (185) then it holds that
P { R j , k ≤ R ^ j , k ≤ C ′ R j , k , for all j ∈ [ m ] } ≥ 1 − C ( n ∨ p ) − c . (186) \mathbb{P}\{R_{j,k}\le\widehat{R}_{j,k}\le C'R_{j,k},\ \text{for all }j\in[m]\}\ge1-C(n\vee p)^{-c}. \tag*{(186)} P { R j , k ≤ R j , k ≤ C ′ R j , k , for all j ∈ [ m ]} ≥ 1 − C ( n ∨ p ) − c . (186)
Proof of Lemma 7. Define
E b a l ( 0 ) : = ⋂ j = 1 m ⋂ ℓ = 1 B j ( 0 ) { 1 2 Λ j , ( k ) ( 0 ) ⪯ 1 ∣ I j , ℓ ( 0 ) ∣ ∑ t ∈ I j , ℓ ( 0 ) M k ( F t ( j ) ) M k ( F t ( j ) ) T ⪯ 3 2 Λ j , ( k ) ( 0 ) } . \mathcal{E}_{\mathrm{bal}}^{(0)}:=\bigcap_{j=1}^{m}\bigcap_{\ell=1}^{B_j^{(0)}}\left\{\frac{1}{2}\Lambda_{j,(k)}^{(0)}\preceq\frac{1}{\lvert\mathcal{I}_{j,\ell}^{(0)}\rvert}\sum_{t\in\mathcal{I}_{j,\ell}^{(0)}}\mathcal{M}_k(\mathbf{F}_t^{(j)})\mathcal{M}_k(\mathbf{F}_t^{(j)})^{\mathsf{T}}\preceq\frac{3}{2}\Lambda_{j,(k)}^{(0)}\right\}. E bal ( 0 ) := j = 1 ⋂ m ℓ = 1 ⋂ B j ( 0 ) ⎩ ⎨ ⎧ 2 1 Λ j , ( k ) ( 0 ) ⪯ ∣ I j , ℓ ( 0 ) ∣ 1 t ∈ I j , ℓ ( 0 ) ∑ M k ( F t ( j ) ) M k ( F t ( j ) ) T ⪯ 2 3 Λ j , ( k ) ( 0 ) ⎭ ⎬ ⎫ .
Each pilot block is a uniformly selected subset of the corresponding client’s records and has size at least b j b_j b j . Let c b a l > 0 c_{\mathrm{bal}}>0 c bal > 0 denote the absolute constant in the exponential bound of Lemma 2. By that lemma and a union bound, we have
P { ( E b a l ( 0 ) ) c } ≤ 2 r k ∑ j = 1 m B j ( 0 ) exp { − c b a l b j C λ 2 } \mathbb{P}\left\{\left(\mathcal{E}_{\mathrm{bal}}^{(0)}\right)^{c}\right\}\le2r_k\sum_{j=1}^{m}B_j^{(0)}\exp\left\{-\frac{c_{\mathrm{bal}}b_j}{C_{\lambda}^{2}}\right\} P { ( E bal ( 0 ) ) c } ≤ 2 r k j = 1 ∑ m B j ( 0 ) exp { − C λ 2 c bal b j }
≤ 2 r k ( n ∨ p ) − c b a l C b / C λ 2 ⋅ ∑ j = 1 m B j ( 0 ) ≤ 2 ( n ∨ p ) 2 − c b a l C b / C λ 2 ≤ 2 ( n ∨ p ) − c , (187) \le2r_k(n\vee p)^{-c_{\mathrm{bal}}C_b/C_{\lambda}^{2}}\cdot\sum_{j=1}^{m}B_j^{(0)}\le2(n\vee p)^{2-c_{\mathrm{bal}}C_b/C_{\lambda}^{2}}\le2(n\vee p)^{-c}, \tag*{(187)} ≤ 2 r k ( n ∨ p ) − c bal C b / C λ 2 ⋅ j = 1 ∑ m B j ( 0 ) ≤ 2 ( n ∨ p ) 2 − c bal C b / C λ 2 ≤ 2 ( n ∨ p ) − c , (187)
where C λ C_\lambda C λ is the constant in Assumption 2 and c > 0 c > 0 c > 0 is the fixed probability exponent in this lemma; the second inequality used our choice of block size choice in Theorem 1, which is also used by the adaptive procedure; the third used r k ≤ p r_k \le p r k ≤ p and ∑ j = 1 m B j ( 0 ) ≤ n \sum_{j=1}^{m} B_j^{(0)} \le n ∑ j = 1 m B j ( 0 ) ≤ n ; and the last holds provided
C b ≥ ( c + 2 ) C λ 2 c b a l . C_b \ge\frac{(c+2)C_\lambda^2}{c_{\mathrm{bal}}}. C b ≥ c bal ( c + 2 ) C λ 2 .
We now work on the event E b a l ( 0 ) \mathcal{E}_{\mathrm{bal}}^{(0)} E bal ( 0 ) . Fix j ∈ [ m ] j \in[m] j ∈ [ m ] , condition on P j ( 0 ) \mathcal{P}_j^{(0)} P j ( 0 ) , and write B : = B j ( 0 ) B := B_j^{(0)} B := B j ( 0 ) . By (23) and r k ≤ p r_k \le p r k ≤ p , we have
S k + 1 ≤ C γ log ( n ∨ p ) , (188) S_k+1 \le C_\gamma\log(n \vee p), \tag*{(188)} S k + 1 ≤ C γ log ( n ∨ p ) , (188)
where C γ > 0 C_\gamma> 0 C γ > 0 is an absolute constant. Also, note that
B ≥ n j ( 0 ) b j − 1 ≥ n j 2 b j − 2. (189) B \ge\frac{n_j^{(0)}}{b_j}-1 \ge\frac{n_j}{2b_j}-2. \tag*{(189)} B ≥ b j n j ( 0 ) − 1 ≥ 2 b j n j − 2. (189)
Using b j ≤ ( C b + 1 ) { r k + log ( n ∨ p ) } b_j \le(C_b+1)\{r_k+\log(n \vee p)\} b j ≤ ( C b + 1 ) { r k + log ( n ∨ p )} and (189) , we obtain
B ε j ≥ n ε j 2 ( C b + 1 ) { r k + log ( n ∨ p ) } − 2 ε j . (190) B\varepsilon_j \ge\frac{n\varepsilon_j}{2(C_b+1)\{r_k+\log(n \vee p)\}}-2\varepsilon_j. \tag*{(190)} B ε j ≥ 2 ( C b + 1 ) { r k + log ( n ∨ p )} n ε j − 2 ε j . (190)
Since ε j ≤ 1 \varepsilon_j \le1 ε j ≤ 1 , combining (185) and (190) gives
B ε j ≥ C 2 log { m ( n ∨ p ) c + 1 } , B ≥ C 2 log { m ( n ∨ p ) c + 1 } , (191) B\varepsilon_j \ge C_2\log\{m(n \vee p)^{c+1}\}, \qquad B \ge C_2\log\{m(n \vee p)^{c+1}\}, \tag*{(191)} B ε j ≥ C 2 log { m ( n ∨ p ) c + 1 } , B ≥ C 2 log { m ( n ∨ p ) c + 1 } , (191)
where 0 < C 2 ≤ C 1 / { 2 ( C b + 1 ) } − 2 / log ( 2 ) 0 < C_2 \le C_1/\{2(C_b+1)\}-2/\log(2) 0 < C 2 ≤ C 1 / { 2 ( C b + 1 )} − 2/ log ( 2 ) is a large enough absolute constant. In particular, B ≥ 8 B \ge8 B ≥ 8 .
Define
E s c o r e ( 0 ) : = ⋂ j = 1 m { max 0 ≤ s ≤ S k ∣ q j , k ( 0 ) ( d s ) − B j ( 0 ) G j , k ( d s ) ∣ ≤ B j ( 0 ) 64 } , \mathcal{E}_{\mathrm{score}}^{(0)} := \bigcap_{j=1}^{m}\left\{\max_{0\le s\le S_k}\left|q_{j,k}^{(0)}(d_s)-B_j^{(0)}G_{j,k}(d_s)\right|\le\frac{B_j^{(0)}}{64}\right\}, E score ( 0 ) := j = 1 ⋂ m { 0 ≤ s ≤ S k max q j , k ( 0 ) ( d s ) − B j ( 0 ) G j , k ( d s ) ≤ 64 B j ( 0 ) } ,
E n o i s e ( 0 ) : = ⋂ j = 1 m { max { ∣ Y j ∣ , ∣ Y j , 0 ∣ , … , ∣ Y j , S k ∣ } ≤ B j ( 0 ) 64 } . \mathcal{E}_{\mathrm{noise}}^{(0)} := \bigcap_{j=1}^{m}\left\{\max\{|Y_j|,|Y_{j,0}|,\ldots,|Y_{j,S_k}|\}\le\frac{B_j^{(0)}}{64}\right\}. E noise ( 0 ) := j = 1 ⋂ m { max { ∣ Y j ∣ , ∣ Y j , 0 ∣ , … , ∣ Y j , S k ∣ } ≤ 64 B j ( 0 ) } .
By (143) ,
E { q j , k ( 0 ) ( d s ) ∣ P j ( 0 ) } = B j ( 0 ) G j , k ( d s ) . \mathbb{E}\{q_{j,k}^{(0)}(d_s)\mid\mathcal{P}_j^{(0)}\}=B_j^{(0)}G_{j,k}(d_s). E { q j , k ( 0 ) ( d s ) ∣ P j ( 0 ) } = B j ( 0 ) G j , k ( d s ) .
Conditional on the pilot partitions, block projectors computed from distinct blocks are independent, and changing one block projector changes q j , k ( 0 ) ( d s ) q_{j,k}^{(0)}(d_s) q j , k ( 0 ) ( d s ) by at most two. The bounded differences inequality (e.g., Corollary 2.21 in [49 ] ) and a union bound over the grid give, for an absolute constant c 1 > 0 c_1>0 c 1 > 0 ,
P { ( E s c o r e ( 0 ) ) c ∣ P 1 ( 0 ) , … , P m ( 0 ) } ≤ 2 ( S k + 1 ) ∑ j = 1 m e − c 1 B j ( 0 ) . (192) \mathbb{P}\left\{(\mathcal{E}_{\mathrm{score}}^{(0)})^c\mid\mathcal{P}_1^{(0)},\ldots,\mathcal{P}_m^{(0)}\right\}\le2(S_k+1)\sum_{j=1}^{m}e^{-c_1B_j^{(0)}}. \tag*{(192)} P { ( E score ( 0 ) ) c ∣ P 1 ( 0 ) , … , P m ( 0 ) } ≤ 2 ( S k + 1 ) j = 1 ∑ m e − c 1 B j ( 0 ) . (192)
The Laplace tail bound and a union bound over the grid give, for an absolute constant c 2 > 0 c_2>0 c 2 > 0 ,
P { ( E n o i s e ( 0 ) ) c ∣ P 1 ( 0 ) , … , P m ( 0 ) } ≤ 2 ( S k + 1 ) ∑ j = 1 m e − c 2 B j ( 0 ) ε j . (193) \mathbb{P}\left\{(\mathcal{E}_{\mathrm{noise}}^{(0)})^c\mid\mathcal{P}_1^{(0)},\ldots,\mathcal{P}_m^{(0)}\right\}\le2(S_k+1)\sum_{j=1}^{m}e^{-c_2B_j^{(0)}\varepsilon_j}. \tag*{(193)} P { ( E noise ( 0 ) ) c ∣ P 1 ( 0 ) , … , P m ( 0 ) } ≤ 2 ( S k + 1 ) j = 1 ∑ m e − c 2 B j ( 0 ) ε j . (193)
Combining (188) , (191) , (192) , and (193) , we have
P { ( E s c o r e ( 0 ) ∩ E n o i s e ( 0 ) ) c | P 1 ( 0 ) , … , P m ( 0 ) } ≤ 4 C γ log ( n ∨ p ) ( n ∨ p ) c + 1 ≤ 4 C γ ( n ∨ p ) − c . (194) \mathbb{P}\left\{\left(\mathcal{E}_{\mathrm{score}}^{(0)} \cap\mathcal{E}_{\mathrm{noise}}^{(0)}\right)^c \,\middle|\, \mathcal{P}_1^{(0)},\ldots,\mathcal{P}_m^{(0)}\right\} \le\frac{4C_\gamma\log(n \vee p)}{(n \vee p)^{c+1}} \le4C_\gamma(n \vee p)^{-c}. \tag*{(194)} P { ( E score ( 0 ) ∩ E noise ( 0 ) ) c P 1 ( 0 ) , … , P m ( 0 ) } ≤ ( n ∨ p ) c + 1 4 C γ log ( n ∨ p ) ≤ 4 C γ ( n ∨ p ) − c . (194)
Next, by the upper bound in (24) , the definition (141) , and b j ≤ ( C b + 1 ) { r k + log ( n ∨ p ) } b_j \le(C_b+1)\{r_k+\log(n \vee p)\} b j ≤ ( C b + 1 ) { r k + log ( n ∨ p )} , we have
ϱ j , k ≥ r k σ λ j , 1 2 p k b j ≥ ( n ∨ p ) − γ r k + log ( n ∨ p ) c 0 b j ≥ ( n ∨ p ) − γ c 0 ( C b + 1 ) . \varrho_{j,k} \ge\sqrt{r_k}\frac{\sigma}{\sqrt{\lambda_{j,1}^2}}\sqrt{\frac{p_k}{b_j}} \ge(n \vee p)^{-\gamma}\sqrt{\frac{r_k+\log(n \vee p)}{c_0b_j}} \ge\frac{(n \vee p)^{-\gamma}}{\sqrt{c_0(C_b+1)}}. ϱ j , k ≥ r k λ j , 1 2 σ b j p k ≥ ( n ∨ p ) − γ c 0 b j r k + log ( n ∨ p ) ≥ c 0 ( C b + 1 ) ( n ∨ p ) − γ .
so that by the choice c 0 ≤ c − 2 / ( C b + 1 ) c_0 \le c_-^2/(C_b+1) c 0 ≤ c − 2 / ( C b + 1 ) , we conclude d min = ( n ∨ p ) − γ ≤ c − ϱ j , k d_{\min}=(n \vee p)^{-\gamma}\le c_-\varrho_{j,k} d m i n = ( n ∨ p ) − γ ≤ c − ϱ j , k .
We now work on the event E b a l ( 0 ) ∩ E s c o r e ( 0 ) ∩ E n o i s e ( 0 ) \mathcal{E}_{\mathrm{bal}}^{(0)}\cap\mathcal{E}_{\mathrm{score}}^{(0)}\cap\mathcal{E}_{\mathrm{noise}}^{(0)} E bal ( 0 ) ∩ E score ( 0 ) ∩ E noise ( 0 ) . If d s ≤ c − ϱ j , k d_s\le c_-\varrho_{j,k} d s ≤ c − ϱ j , k , monotonicity of G j , k G_{j,k} G j , k and Lemma 6 give G j , k ( d s ) ≤ 3 / 4 G_{j,k}(d_s)\le3/4 G j , k ( d s ) ≤ 3/4 . On E s c o r e ( 0 ) \mathcal{E}_{\mathrm{score}}^{(0)} E score ( 0 ) ,
q j , k ( 0 ) ( d s ) ≤ B G j , k ( d s ) + B 64 ≤ 49 B 64 . q_{j,k}^{(0)}(d_s)\le BG_{j,k}(d_s)+\frac{B}{64}\le\frac{49B}{64}. q j , k ( 0 ) ( d s ) ≤ B G j , k ( d s ) + 64 B ≤ 64 49 B .
On E n o i s e ( 0 ) \mathcal{E}_{\mathrm{noise}}^{(0)} E noise ( 0 ) , we have Y j , s ≤ B / 64 Y_{j,s}\le B/64 Y j , s ≤ B /64 and Y j ≥ − B / 64 Y_j\ge-B/64 Y j ≥ − B /64 . Consequently,
q j , k ( 0 ) ( d s ) + Y j , s ≤ 50 B 64 < 55 B 64 = 7 B 8 − B 64 ≤ 7 B 8 + Y j . q_{j,k}^{(0)}(d_s)+Y_{j,s}\le\frac{50B}{64}<\frac{55B}{64}=\frac{7B}{8}-\frac{B}{64}\le\frac{7B}{8}+Y_j. q j , k ( 0 ) ( d s ) + Y j , s ≤ 64 50 B < 64 55 B = 8 7 B − 64 B ≤ 8 7 B + Y j .
Thus no grid point at or below c − ⋅ ϱ j , k c_-\cdot\varrho_{j,k} c − ⋅ ϱ j , k crosses the noisy threshold.
To bound the selected grid point from above, first suppose that c + ⋅ ϱ j , k ≤ d S k c_+\cdot\varrho_{j,k}\le d_{S_k} c + ⋅ ϱ j , k ≤ d S k , and let d s ∗ d_{s_*} d s ∗ be the first grid point satisfying d s ∗ ≥ c + ⋅ ϱ j , k d_{s_*}\ge c_+\cdot\varrho_{j,k} d s ∗ ≥ c + ⋅ ϱ j , k . By monotonicity of G j , k G_{j,k} G j , k and Lemma 6 , we have
G j , k ( d s ∗ ) ≥ 15 16 = 60 64 . G_{j,k}(d_{s_*})\ge\frac{15}{16}=\frac{60}{64}. G j , k ( d s ∗ ) ≥ 16 15 = 64 60 .
On E s c o r e ( 0 ) \mathcal{E}_{\mathrm{score}}^{(0)} E score ( 0 ) , this gives
q j , k ( 0 ) ( d s ∗ ) ≥ B G j , k ( d s ∗ ) − B 64 ≥ 59 B 64 . q_{j,k}^{(0)}(d_{s_*})\ge BG_{j,k}(d_{s_*})-\frac{B}{64}\ge\frac{59B}{64}. q j , k ( 0 ) ( d s ∗ ) ≥ B G j , k ( d s ∗ ) − 64 B ≥ 64 59 B .
On E n o i s e ( 0 ) \mathcal{E}_{\mathrm{noise}}^{(0)} E noise ( 0 ) , we have Y j , s ∗ ≥ − B / 64 Y_{j,s_*}\ge-B/64 Y j , s ∗ ≥ − B /64 and Y j ≤ B / 64 Y_j\le B/64 Y j ≤ B /64 . Consequently,
q j , k ( 0 ) ( d s ∗ ) + Y j , s ∗ ≥ 58 B 64 > 57 B 64 = 7 B 8 + B 64 ≥ 7 B 8 + Y j . q_{j,k}^{(0)}(d_{s_*})+Y_{j,s_*}\ge\frac{58B}{64}>\frac{57B}{64}=\frac{7B}{8}+\frac{B}{64}\ge\frac{7B}{8}+Y_j. q j , k ( 0 ) ( d s ∗ ) + Y j , s ∗ ≥ 64 58 B > 64 57 B = 8 7 B + 64 B ≥ 8 7 B + Y j .
Thus d s ∗ d_{s_*} d s ∗ crosses the noisy threshold, so the search stops no later than s ∗ s_* s ∗ .
Finally, d 0 = d min ≤ c − ⋅ ϱ j , k < c + ⋅ ϱ j , k d_0=d_{\min}\le c_-\cdot\varrho_{j,k}<c_+\cdot\varrho_{j,k} d 0 = d m i n ≤ c − ⋅ ϱ j , k < c + ⋅ ϱ j , k , so s ∗ ≥ 1 s_*\ge1 s ∗ ≥ 1 . By the construction of s ∗ s_* s ∗ , d s ∗ − 1 < c + ⋅ ϱ j , k d_{s_*-1}<c_+\cdot\varrho_{j,k} d s ∗ − 1 < c + ⋅ ϱ j , k . Since consecutive grid points differ by a factor of two, it holds that
d s ^ j , k ≤ d s ∗ = 2 d s ∗ − 1 < 2 c + ⋅ ϱ j , k . d_{\widehat{s}_{j,k}}\le d_{s_*}=2d_{s_*-1}<2c_+\cdot\varrho_{j,k}. d s j , k ≤ d s ∗ = 2 d s ∗ − 1 < 2 c + ⋅ ϱ j , k .
If instead d S k < c + ⋅ ϱ j , k d_{S_k}<c_+\cdot\varrho_{j,k} d S k < c + ⋅ ϱ j , k , then d S k ≥ 2 r k d_{S_k}\ge\sqrt{2r_k} d S k ≥ 2 r k by (23) . The Frobenius distance between any two rank-r k r_k r k orthogonal projectors is at most 2 r k \sqrt{2r_k} 2 r k . Hence every ordered block pair is counted at d S k d_{S_k} d S k , and q j , k ( 0 ) ( d S k ) = B q_{j,k}^{(0)}(d_{S_k})=B q j , k ( 0 ) ( d S k ) = B .
On E n o i s e ( 0 ) \mathcal{E}_{\mathrm{noise}}^{(0)} E noise ( 0 ) , we have Y j , S k ≥ − B / 64 Y_{j,S_k} \ge-B/64 Y j , S k ≥ − B /64 and Y j ≤ B / 64 Y_j \le B/64 Y j ≤ B /64 . Therefore,
q j , k ( 0 ) ( d S k ) + Y j , S k ≥ 63 B 64 > 57 B 64 = 7 B 8 + B 64 ≥ 7 B 8 + Y j . q_{j,k}^{(0)}(d_{S_k}) + Y_{j,S_k} \ge\frac{63B}{64} > \frac{57B}{64} = \frac{7B}{8} + \frac{B}{64} \ge\frac{7B}{8} + Y_j. q j , k ( 0 ) ( d S k ) + Y j , S k ≥ 64 63 B > 64 57 B = 8 7 B + 64 B ≥ 8 7 B + Y j .
The terminal grid point crosses the noisy threshold. The search thus selects a grid point satisfying
d s ^ j , k ≤ d S k < c + ⋅ ϱ j , k . d_{\widehat{s}_{j,k}} \le d_{S_k} < c_+ \cdot\varrho_{j,k}. d s j , k ≤ d S k < c + ⋅ ϱ j , k .
We have already shown that no grid point at or below c − ⋅ ϱ j , k c_- \cdot\varrho_{j,k} c − ⋅ ϱ j , k crosses the noisy threshold. Combining this lower bound with the upper bound from either case, we obtain, on E b a l ( 0 ) ∩ E s c o r e ( 0 ) ∩ E n o i s e ( 0 ) \mathcal{E}_{\mathrm{bal}}^{(0)} \cap\mathcal{E}_{\mathrm{score}}^{(0)} \cap\mathcal{E}_{\mathrm{noise}}^{(0)} E bal ( 0 ) ∩ E score ( 0 ) ∩ E noise ( 0 ) , that
c − ⋅ ϱ j , k < d s ^ j , k ≤ 2 c + ⋅ ϱ j , k , j ∈ [ m ] . (195) c_- \cdot\varrho_{j,k} < d_{\widehat{s}_{j,k}} \le2c_+ \cdot\varrho_{j,k}, \qquad j \in[m]. \tag*{(195)} c − ⋅ ϱ j , k < d s j , k ≤ 2 c + ⋅ ϱ j , k , j ∈ [ m ] . (195)
The conditional probability bound for E s c o r e ( 0 ) ∩ E n o i s e ( 0 ) \mathcal{E}_{\mathrm{score}}^{(0)} \cap\mathcal{E}_{\mathrm{noise}}^{(0)} E score ( 0 ) ∩ E noise ( 0 ) obtained above is uniform over the pilot partitions. Together with (187) and (194) , it shows that (195) holds with probability at least 1 − C ( n ∨ p ) − c 1 - C(n \vee p)^{-c} 1 − C ( n ∨ p ) − c .
Finally, (142) gives R j , k ≍ ϱ j , k R_{j,k} \asymp\varrho_{j,k} R j , k ≍ ϱ j , k . Since R ^ j , k = C A d s ^ j , k \widehat{R}_{j,k} = C_A d_{\widehat{s}_{j,k}} R j , k = C A d s j , k , choose C A C_A C A sufficiently large that C A c − ⋅ ϱ j , k ≥ R j , k C_Ac_- \cdot\varrho_{j,k} \ge R_{j,k} C A c − ⋅ ϱ j , k ≥ R j , k for all j j j . Whenever (195) holds,
R j , k ≤ R ^ j , k ≤ 2 C A c + ϱ j , k ≤ C ′ R j , k , j ∈ [ m ] . R_{j,k} \le\widehat{R}_{j,k} \le2C_Ac_+ \varrho_{j,k} \le C'R_{j,k}, \qquad j \in[m]. R j , k ≤ R j , k ≤ 2 C A c + ϱ j , k ≤ C ′ R j , k , j ∈ [ m ] .
This proves (186) and thus completes the proof of this lemma.
Proofs of propositions and theorems
Proof of Proposition 2. Fix j ∈ [ m ] j \in[m] j ∈ [ m ] and a realization of the data-independent split and block partitions. Consider any two neighboring datasets of client j j j , with all other clients’ data held fixed. As the split is fixed, the record on which the two datasets differ belongs to either the pilot half or the main half.
Case 1: the differing record belongs to the pilot half. Changing this record changes at most one pilot block projector. It hence changes each query in the definition of q j , k ( 0 ) ( d s ) q_{j,k}^{(0)}(d_s) q j , k ( 0 ) ( d s ) (in Algorithm 2) by at most two. With the noises Y j Y_j Y j and Y j , s Y_{j,s} Y j , s (also in Algorithm 2), the output s ^ j , k \widehat{s}_{j,k} s j , k is ( ε j , 0 ) (\varepsilon_j,0) ( ε j , 0 ) -CDP.
The main observations are identical in the two datasets. Once s ^ j , k \widehat{s}_{j,k} s j , k is given, the quantities R ^ j , k \widehat{R}_{j,k} R j , k , τ ^ j , k \widehat{\tau}_{j,k} τ j , k , Ψ ^ j , k \widehat{\Psi}_{j,k} Ψ j , k , and ι ^ j , k \widehat{\iota}_{j,k} ι j , k are fixed, and the remaining output is generated from the same main observations under both datasets. Thus the complete local message is a randomized post-processing of s ^ j , k \widehat{s}_{j,k} s j , k . It is ( ε j , 0 ) (\varepsilon_j,0) ( ε j , 0 ) -CDP in this case.
Case 2: the differing record belongs to the main half. The pilot observations are identical in the two datasets, so s ^ j , k \widehat{s}_{j,k} s j , k has the same distribution under both. Condition on any value of s ^ j , k \widehat{s}_{j,k} s j , k . The calibration parameters, certificate, and flag are now fixed. If ι ^ j , k = 1 \widehat{\iota}_{j,k}=1 ι j , k = 1 , Proposition 1 gives ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -CDP for the augmented main output. If ι ^ j , k = 0 \widehat{\iota}_{j,k}=0 ι j , k = 0 , the main output is drawn independently of the data. Hence for every fixed pilot output, the complete local message is ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -CDP with respect to the main observations. Averaging over the common distribution of the pilot output preserves this guarantee.
These two cases cover every pair of neighboring datasets. Thus, for each fixed split and block partition, the local message is ( ε j , δ j ) (\varepsilon_j,\delta_j) ( ε j , δ j ) -CDP. The split and block partitions are generated independently of the data, so averaging the privacy inequality over their randomness gives the same guarantee for Algorithm 2 without conditioning on a particular partition.
Finally, the server weights and final singular value decomposition are post-processing of the client messages. Applying the argument to each client proves ( ε , δ ) (\varepsilon,\delta) ( ε , δ ) -FDP.
Proof of Theorem 3. For notational convenience, write P : = A k A k ⊤ P := A_kA_k^\top P := A k A k ⊤ throughout this proof. Our proof proceeds in three steps, detailed as follows.
Step 1. Pilot calibration and screening. Apply Lemma 2 to all pilot and main blocks. The same union-bound argument as in its proof and a sufficiently large C b C_b C b show that simultaneous factor balance fails with probability at most C ( n ∨ p ) − c C(n \vee p)^{-c} C ( n ∨ p ) − c . On the factor-balance event, apply Lemma 7. Define
( τ j , k ( 1 ) ) 2 : = 64 R j , k 2 ( B j ( 1 ) ) 2 ε j 2 log ( 2.5 δ j ) , Ψ j , k ( 1 ) : = R j , k r k log ( n ∨ p ) B j ( 1 ) + τ j , k ( 1 ) p k + log ( n ∨ p ) . (\tau_{j,k}^{(1)})^2 := \frac{64R_{j,k}^2}{(B_j^{(1)})^2\varepsilon_j^2}\log\left(\frac{2.5}{\delta_j}\right), \qquad\Psi_{j,k}^{(1)} := \frac{R_{j,k}}{\sqrt{r_k}}\sqrt{\frac{\log(n \vee p)}{B_j^{(1)}}}+\tau_{j,k}^{(1)}\sqrt{p_k+\log(n \vee p)}. ( τ j , k ( 1 ) ) 2 := ( B j ( 1 ) ) 2 ε j 2 64 R j , k 2 log ( δ j 2.5 ) , Ψ j , k ( 1 ) := r k R j , k B j ( 1 ) log ( n ∨ p ) + τ j , k ( 1 ) p k + log ( n ∨ p ) .
Then simultaneously for all clients, we have
R j , k ≤ R ^ j , k ≤ C R j , k , c Ψ j , k ( 1 ) ≤ Ψ ^ j , k ≤ C Ψ j , k ( 1 ) , j ∈ [ m ] , (196) R_{j,k} \le\widehat R_{j,k} \le CR_{j,k}, \qquad c\Psi_{j,k}^{(1)} \le\widehat\Psi_{j,k} \le C\Psi_{j,k}^{(1)}, \qquad j\in[m], \tag*{(196)} R j , k ≤ R j , k ≤ C R j , k , c Ψ j , k ( 1 ) ≤ Ψ j , k ≤ C Ψ j , k ( 1 ) , j ∈ [ m ] , (196)
By the lower bound in (24) , the block-size choice, and (196) , R ^ j , k \widehat R_{j,k} R j , k is uniformly small when C 0 C_0 C 0 is sufficiently large. The condition (25) also ensures r k Ψ ^ j , k ≤ c 0 \sqrt{r_k}\widehat\Psi_{j,k}\le c_0 r k Ψ j , k ≤ c 0 and the requirement in (22) , after increasing C 0 C_0 C 0 if necessary. Hence we conclude G ^ k = [ m ] \widehat G_k=[m] G k = [ m ] on the calibration event.
Step 2. Conditional local ideal releases. Let T k \mathcal T_k T k be the sigma-algebra generated by the data-independent split and partitions, the pilot observations, and the pilot mechanism. Conditional on T k \mathcal T_k T k , all adaptive radii, certificates, and server weights are fixed, while the main observation noises remain independent across clients and blocks.
For each main block, repeat all the construction in (63) –(76) , replacing
B j , R j , k , τ j , k , Ψ j , k l o c B_j,\qquad R_{j,k},\qquad\tau_{j,k},\qquad\Psi_{j,k}^{\mathrm{loc}} B j , R j , k , τ j , k , Ψ j , k loc
by
B j ( 1 ) , R ^ j , k , τ ^ j , k , Ψ ^ j , k , B_j^{(1)},\qquad\widehat R_{j,k},\qquad\widehat\tau_{j,k},\qquad\widehat\Psi_{j,k}, B j ( 1 ) , R j , k , τ j , k , Ψ j , k ,
respectively. Denote the resulting projector by Π ~ j , k a d \widetilde\Pi_{j,k}^{\mathrm{ad}} Π j , k ad . The first inequality in (196) and the main-block version of Lemma 3 show that the truncation agrees with every empirical main-block projector except on an event of probability C ( n ∨ p ) − c C(n \vee p)^{-c} C ( n ∨ p ) − c .
The conditional Hilbert-space Hoeffding bound in (67) and the GOE bound in (69) remain valid after these replacements. Since p k ≳ log ( n ∨ p ) p_k\gtrsim\log(n\vee p) p k ≳ log ( n ∨ p ) , their right-hand side is bounded by C r k Ψ ^ j , k C\sqrt{r_k}\widehat\Psi_{j,k} C r k Ψ j , k . The reflection argument in (71) –(77) also remains valid conditionally on T k \mathcal T_k T k . Consequently,
( I p k − P ) E ( Π ~ j , k a d ∣ T k ) P = 0 , (197) (I_{p_k}-P)\mathbb E(\widetilde\Pi_{j,k}^{\mathrm{ad}}\mid\mathcal T_k)P=0, \tag*{(197)} ( I p k − P ) E ( Π j , k ad ∣ T k ) P = 0 , (197)
∥ E ( Π ~ j , k a d ∣ T k ) − P ∥ ≤ c , (198) \left\|\mathbb E(\widetilde\Pi_{j,k}^{\mathrm{ad}}\mid\mathcal T_k)-P\right\|\le c, \tag*{(198)} E ( Π j , k ad ∣ T k ) − P ≤ c , (198)
∥ Π ~ j , k a d − E ( Π ~ j , k a d ∣ T k ) ∥ F ≤ C r k Ψ ^ j , k almost surely . (199) \left\|\widetilde\Pi_{j,k}^{\mathrm{ad}}-\mathbb E(\widetilde\Pi_{j,k}^{\mathrm{ad}}\mid\mathcal T_k)\right\|_{\mathrm F}\le C\sqrt{r_k}\widehat\Psi_{j,k}\quad\text{almost surely}. \tag*{(199)} Π j , k ad − E ( Π j , k ad ∣ T k ) F ≤ C r k Ψ j , k almost surely . (199)
The projectors are conditionally independent across j j j .
It remains to verify the coupling. On the simultaneous block concentration event, every pair of main-block projectors is at distance at most R j , k / 8 ≤ R ^ j , k / 8 R_{j,k}/8\le\widehat R_{j,k}/8 R j , k /8 ≤ R j , k /8 . Hence, exactly as in (79) , the Huber aggregate equals the arithmetic block average. Note that B j ( 1 ) ≥ n j / ( 2 b j ) − 1 B_j^{(1)}\ge n_j/(2b_j)-1 B j ( 1 ) ≥ n j / ( 2 b j ) − 1 . Hence (25) and our choice of block size imply
B j ( 1 ) ε j ≥ C t e s t log { n ∨ p δ j } . B_j^{(1)}\varepsilon_j\ge C_{\mathrm{test}}\log\left\{\frac{n\vee p}{\delta_j}\right\}. B j ( 1 ) ε j ≥ C test log { δ j n ∨ p } .
where C t e s t > 0 C_{\mathrm{test}}>0 C test > 0 is an absolute constant that can be made sufficiently large by increasing C 0 C_0 C 0 . Conditional on the simultaneous block concentration event, the main stability score equals B j ( 1 ) B_j^{(1)} B j ( 1 ) for every client, so that by (80) and taking a union bound over m ≤ n m\le n m ≤ n clients, we conclude that every main stability test passes, except on an event of conditional probability at most C ( n ∨ p ) − c C(n\vee p)^{-c} C ( n ∨ p ) − c . Hence we have
P { Π ~ j , k a d = U ~ j , k U ~ j , k ⊤ , ∀ j ∈ [ m ] ∣ T k } ≥ 1 − C ( n ∨ p ) − c . (200) \mathbb{P}\left\{\widetilde{\Pi}_{j,k}^{\mathrm{ad}}=\widetilde{U}_{j,k}\widetilde{U}_{j,k}^{\top},\ \forall j\in[m]\mid\mathcal{T}_k\right\}\ge1-C(n\vee p)^{-c}. \tag*{(200)} P { Π j , k ad = U j , k U j , k ⊤ , ∀ j ∈ [ m ] ∣ T k } ≥ 1 − C ( n ∨ p ) − c . (200)
Step 3. Adaptive server aggregation. Conditional on T k \mathcal{T}_k T k , define
M k a d : = ∑ j = 1 m w ^ j , k E ( Π ~ j , k a d ∣ T k ) , M ~ k a d : = ∑ j = 1 m w ^ j , k Π ~ j , k a d . M_k^{\mathrm{ad}}:=\sum_{j=1}^{m}\widehat{w}_{j,k}\mathbb{E}(\widetilde{\Pi}_{j,k}^{\mathrm{ad}}\mid\mathcal{T}_k),\qquad\widetilde{M}_k^{\mathrm{ad}}:=\sum_{j=1}^{m}\widehat{w}_{j,k}\widetilde{\Pi}_{j,k}^{\mathrm{ad}}. M k ad := j = 1 ∑ m w j , k E ( Π j , k ad ∣ T k ) , M k ad := j = 1 ∑ m w j , k Π j , k ad .
Then (197) and (198) show that M k a d M_k^{\mathrm{ad}} M k ad is block diagonal with respect to span ( A k ) ⊕ span ( A k ) ⊥ \operatorname{span}(A_k)\oplus\operatorname{span}(A_k)^\perp span ( A k ) ⊕ span ( A k ) ⊥ , its leading r k r_k r k -dimensional eigenspace is span ( A k ) \operatorname{span}(A_k) span ( A k ) , and its eigengap is at least an absolute positive constant. Conditional independence and (199) , followed by the Hilbert-space Hoeffding inequality, give, for every u > 0 u>0 u > 0 ,
P { ∥ M ~ k a d − M k a d ∥ F 2 > C r k u ∑ j = 1 m w ^ j , k 2 Ψ ^ j , k 2 | T k } ≤ 2 e − u . (201) \mathbb{P}\left\{\left\|\widetilde{M}_k^{\mathrm{ad}}-M_k^{\mathrm{ad}}\right\|_{\mathrm{F}}^2>C r_k u\sum_{j=1}^{m}\widehat{w}_{j,k}^{2}\widehat{\Psi}_{j,k}^{2}\,\middle|\,\mathcal{T}_k\right\}\le2e^{-u}. \tag*{(201)} P { M k ad − M k ad F 2 > C r k u j = 1 ∑ m w j , k 2 Ψ j , k 2 T k } ≤ 2 e − u . (201)
By the definition of w ^ j , k \widehat{w}_{j,k} w j , k in Algorithm 2, we have
∑ j = 1 m w ^ j , k 2 Ψ ^ j , k 2 = 1 ∑ j = 1 m Ψ ^ j , k − 2 . (202) \sum_{j=1}^{m}\widehat{w}_{j,k}^{2}\widehat{\Psi}_{j,k}^{2}=\frac{1}{\sum_{j=1}^{m}\widehat{\Psi}_{j,k}^{-2}}. \tag*{(202)} j = 1 ∑ m w j , k 2 Ψ j , k 2 = ∑ j = 1 m Ψ j , k − 2 1 . (202)
Take u = log ( n ∨ p ) u=\log(n\vee p) u = log ( n ∨ p ) in (201) . By the Davis–Kahan theorem (Yu et al., 2015, Theorem 2), (200) , and (203), we have
∥ A ^ k a d ( A ^ k a d ) ⊤ − P ∥ F 2 ≤ C r k log ( n ∨ p ) ∑ j = 1 m Ψ ^ j , k − 2 ∧ ( 2 r k ) . \left\|\widehat{A}_k^{\mathrm{ad}}(\widehat{A}_k^{\mathrm{ad}})^{\top}-P\right\|_{\mathrm{F}}^{2}\le\frac{C r_k\log(n\vee p)}{\sum_{j=1}^{m}\widehat{\Psi}_{j,k}^{-2}}\wedge(2r_k). A k ad ( A k ad ) ⊤ − P F 2 ≤ ∑ j = 1 m Ψ j , k − 2 C r k log ( n ∨ p ) ∧ ( 2 r k ) .
By (196) , we have
∑ j = 1 m Ψ ^ j , k − 2 ≥ 1 C ∑ j = 1 m ( Ψ j , k ( 1 ) ) − 2 . \sum_{j=1}^{m}\widehat{\Psi}_{j,k}^{-2}\ge\frac{1}{C}\sum_{j=1}^{m}\left(\Psi_{j,k}^{(1)}\right)^{-2}. j = 1 ∑ m Ψ j , k − 2 ≥ C 1 j = 1 ∑ m ( Ψ j , k ( 1 ) ) − 2 .
Moreover, the lower bound on B j ( 1 ) B_j^{(1)} B j ( 1 ) implies B j ( 1 ) ≍ B j B_j^{(1)}\asymp B_j B j ( 1 ) ≍ B j , as the main sample contains a fixed fraction of the observations. Together with p k ≳ log ( n ∨ p ) p_k\gtrsim\log(n\vee p) p k ≳ log ( n ∨ p ) , this gives
Ψ j , k ( 1 ) ≤ C Ψ j , k l o c . \Psi_{j,k}^{(1)}\le C\Psi_{j,k}^{\mathrm{loc}}. Ψ j , k ( 1 ) ≤ C Ψ j , k loc .
Altogether, we conclude the desired upper bound, thus concluding the stated probability guarantee and this proof.