Introduction
Network data provide a natural framework for representing pairwise relationships among a collection of entities. A single network is commonly encoded by an adjacency matrix, whose entries record whether, or to what extent, pairs of nodes are connected. Examples include social networks, where nodes represent individuals and edges represent friendship or communication ties [8 ] , and structural brain networks, where nodes represent brain regions and edges represent anatomical connections estimated from diffusion MRI tractography [19 ] . Multilayer network data, represented as collections of adjacency matrices on a common set of nodes, have become increasingly common in modern applications [32 , 11 ] . Such layers may arise from repeated measurements over time [42 ] , different experimental or cognitive conditions [38 ] , or distinct types of relations among the same entities [50 ] . They may also arise from multimodal measurements. For example, in neuroimaging, a common anatomical parcellation can be used to define a shared set of nodes, while different layers may encode distinct forms of structural or functional connectivity derived from different imaging modalities [14 ] .
A central task in network data analysis is to uncover latent community or membership structure from the observed pattern of edges [17 ] . The stochastic block model (SBM) provides a canonical probabilistic framework for this task [23 ] . Its extensions include the mixed-membership stochastic block model (MMSBM), which allows nodes to participate in multiple communities simultaneously [3 ] , the degree-corrected stochastic block model (DCBM), which accounts for node-specific degree heterogeneity [29 ] , and the degree-corrected mixed-membership model (DCMM), which combines these two features [56 , 26 ] . Because the DCMM jointly encodes degree effects, mixed memberships, and block connectivity in the same probability matrix, identifiability is a nontrivial issue. A wide range of methods have been developed for community detection, including likelihood-based and Bayesian procedures, modularity maximization, matrix factorization approaches, and spectral methods. The latter are especially relevant to the present work, as they provide computationally efficient estimators based on low-rank structure in the population adjacency matrix. The random dot product graph (RDPG) [54 , 7 ] framework provides a latent-position interpretation of such embeddings by representing edge probabilities through inner products of latent node positions.
For multilayer network data, block models are often extended by assuming that the layers share a latent membership structure [45 ] . For example, the multilayer degree-corrected stochastic block model (ML-DCBM) extends the DCBM by assuming common non-overlapping community assignments across layers, together with layer-specific degree-correction parameters and block connectivity matrices. The multilayer degree-corrected mixed-membership model (ML-DCMM), a multilayer extension of the single-layer DCMM, assumes the layers share a common mixed-membership matrix while the degree-correction parameters and block connectivity matrices are allowed to vary across layers. From the perspective of membership estimation, it is important to use information across layers without collapsing the multilayer structure into a single averaged adjacency matrix. While averaging can improve signal strength, it may also discard layer-specific connectivity patterns and degree heterogeneity when the layers are heterogeneous. This has motivated joint multilayer approaches that retain the layer index, including tensor-based methods that stack adjacency matrices into a third-order array and use tensor-decomposition techniques to extract shared node-level structure across layers [27 , 2 ] .
In this paper, we study the ML-DCMM, provide identifiability conditions and propose algorithms for community detection under this model. The main contributions of this paper are as follows.
We establish identifiability conditions for the ML-DCMM, clarifying how the shared mixed-membership matrix can be separated from layer-specific degree corrections and block connectivity matrices. We also show that other latent space models for multilayer network data can be described as a special case of this model, showing that this model provides a general framework with interpretable latent structure.
We propose an efficient spectral algorithm for estimation of the common node memberships via a corner estimation procedure. This method consists of the concatenation of normalized adjacency spectral embeddings (CNASE) followed by vertex hunting. We also establish a con-
nection of these embeddings with the ParaTuck-2 tensor factorization and propose a refinement of the estimator based on this decomposition.
We establish error bounds for spectral estimation of node memberships in terms of the two-to-infinity norm of the mixed membership matrix, providing row-wise control of the community-detection error. We also establish a minimax lower bound for membership estimation that does not improve with increasing number of layers. This lower bound match the error bounds of the proposed method up to logarithmic factors under homogeneous degree-correction parameters, showing that our method is nearly minimax optimal.
Related work
The stochastic block model (SBM) assumes that nodes of a graph are divided into K K K communities, and edges are independently generated according to a block-structured probability matrix. In matrix form, this probability matrix can be written as P = Z B Z T \mathbf{P} = \mathbf{Z}\mathbf{B}\mathbf{Z}^{\mathsf{T}} P = ZB Z T , where Z \mathbf{Z} Z is an n × K n \times K n × K binary membership matrix whose i i i th row is the canonical basis vector corresponding to the community label of node i i i , and B ∈ [ 0 , 1 ] K × K \mathbf{B} \in[0,1]^{K \times K} B ∈ [ 0 , 1 ] K × K specifies the connection probabilities between communities [23 ] . A large body of work has studied estimation and theory under the SBM, including likelihood-and modularity-based approaches [10 ] and spectral methods based on graph Laplacians [47 ] , adjacency spectral embeddings [49 ] , and adjacency spectral clustering [35 ] . While this parsimonious model can approximate a wide range of random networks, it has notable limitations. First, nodes assigned to the same community are stochastically equivalent so the model cannot capture node-level degree heterogeneity. Karrer and Newman [ [29 ] proposed the DCBM to overcome this limitation by introducing node-specific degree-correction parameters. Several spectral methods have been proposed including regularized graph Laplacian [46 ] , SCORE [24 ] as well as theoretical works including consistency [57 ] and minimax optimality [18 ] . Second, each node is assigned to a single community, which can be restrictive when communities overlap or when nodes participate in multiple communities simultaneously. The MMSBM relaxes this restriction by allowing overlapping communities for each node [3 ] . Anandkumar et al. [ [4 ] proposed a tensor spectral method and Mao et al. [ [39 ] connected MMSBM estimation to symmetric nonnegative matrix factorization. Mao et al. [ [41 ] exploited the row-wise simplex geometry of the leading population eigenspace under the MMSBM, where pure nodes correspond to simplex vertices and mixed nodes are represented as convex combinations of these vertices.
The degree-corrected mixed-membership model (DCMM) allows both node-specific degree correction and mixed memberships. Existing simplex-based spectral methods for DCMM-type models exploit a normalized spectral embedding in which degree effects are factored out and membership information is encoded by a simplex geometry. Estimating the corners of this simplex provides the anchor points needed for membership reconstruction. Examples of these methods include sketched vertex search or successive projection on the ratios of eigenvectors [24 ] , and k k k -medians clustering or one-class SVM on row-wise ℓ 2 \ell_2 ℓ 2 normalized spectral embeddings [56 , 40 ] .
For multilayer networks, most existing work on community detection focuses on extensions that typically assume a shared community structure across layers, with the intention of capturing joint structure. In the multilayer SBM, estimators of shared community structure includes likelihood-based approaches [20 , 44 ] , least-squares [36 ] , and spectral methods [9 , 6 , 34 ] , and nonparametric Bayesian methods [45 ] . Extensions to other multilayer models are often non-trivial due to the specific identifiability and estimation constraints that arise. For instance, identifiability the multilayer degree-corrected mixed-membership model (ML-DCMM) is more challenging than in the single-layer DCMM because the model introduces additional layer-specific parameters. Xie and Huang [ [53 ] developed a spectral-assisted Gaussian variational inference method for the ML-DCMM under a direct layer-wise extension of the single-layer DCMM identifiability condition.
Spectral methods are a particularly appealing approach in modern data science due to their computational scalability and theoretical guarantees [13 ] . In multilayer community detection, these methods require constructing a joint spectral embedding matrix that captures community information across the layers. While several of these approaches have been developed for the ML-SBM, these ideas do not generalize to other models due to the more complex geometry of the embedding spaces. For instance, Agterberg et al. [ [1 ] proposed a spectral clustering method for the ML-DCBM based on a joint spectral embedding obtained from the singular value decomposition of concatenated spectral embeddings. In this paper, we employ a similar concatenation idea, but in the ML-DCMM model with mixed memberships, this approach does not yield consistent estimators of the node memberships due to heterogeneity across the layer, motivating the use of a different embedding matrix to estimate the corners of the simplex that identify the communities of the model.
Tensor-based approaches have also been proposed for multilayer community detection, applying Tucker-type or nonnegative Tucker decompositions [27 , 2 ] . These tensor decompositions are natural in the SBM and some of its extensions as the parameters of these models can be expressed as low-rank tensors. However, the ML-DCMM structure does not allow us to apply these decompositions to recover the model parameters. Instead, we establish a connection between the parameters of the ML-DCMM and the ParaTuck-2 decomposition [22 ] . Specifically, for a third-order tensor U ∈ R I × J × L U \in\mathbb{R}^{I \times J \times L} U ∈ R I × J × L , let U ⋅ , ⋅ , l ∈ R I × J U_{\cdot,\cdot,l} \in\mathbb{R}^{I \times J} U ⋅ , ⋅ , l ∈ R I × J be its l l l -th frontal slice. The ParaTuck-2 decomposition represents the l l l -th slice as
U ⋅ , ⋅ , l = A D A ( l ) H D B ( l ) B T , (1) U_{\cdot,\cdot,l} = A D_A^{(l)} H D_B^{(l)} B^\mathsf{T}, \tag*{(1)} U ⋅ , ⋅ , l = A D A ( l ) H D B ( l ) B T , (1)
where A ∈ R I × R 1 A \in\mathbb{R}^{I \times R_1} A ∈ R I × R 1 and B ∈ R J × R 2 B \in\mathbb{R}^{J \times R_2} B ∈ R J × R 2 are shared loading matrices independent of l l l , D A ( l ) ∈ R R 1 × R 1 D_A^{(l)} \in\mathbb{R}^{R_1 \times R_1} D A ( l ) ∈ R R 1 × R 1 and D B ( l ) ∈ R R 2 × R 2 D_B^{(l)} \in\mathbb{R}^{R_2 \times R_2} D B ( l ) ∈ R R 2 × R 2 are diagonal slice-specific scaling matrices, and H ∈ R R 1 × R 2 H \in\mathbb{R}^{R_1 \times R_2} H ∈ R R 1 × R 2 captures the interactions between the two factor spaces. In particular, when A A A and B B B are the identity matrix, the entry ( i , j , l ) (i,j,l) ( i , j , l ) of this tensor is U i j l = C i l H i j D j l U_{ijl}=C_{il}H_{ij}D_{jl} U ij l = C i l H ij D j l for some matrices C ∈ R I × L C \in\mathbb{R}^{I \times L} C ∈ R I × L , D ∈ R J × L D \in\mathbb{R}^{J \times L} D ∈ R J × L , which is known as the no three-way interaction model [Daconis and Sturmfels, 1998], and we show that it correspond to a factorization of the parameters of the ML-DCMM. Several algorithms to fit this decomposition have been proposed, including alternating least squares (ALS) [12 ] , with additional stochastic ALS step [Znyied and de Almeida, 2025] and a lifting-based algebraic solver [51 ] .
Notation
Let a a a , a \mathbf{a} a , and A \mathbf{A} A denote a scalar, column vector, and matrix, respectively. For any matrix A ∈ R m × n \mathbf{A} \in\mathbb{R}^{m \times n} A ∈ R m × n , A i , ⋅ \mathbf{A}_{i,\cdot} A i , ⋅ , A ⋅ , j \mathbf{A}_{\cdot,j} A ⋅ , j , and A i j A_{ij} A ij denote the i i i -th row, j j j -th column of A \mathbf{A} A and ( i , j ) (i,j) ( i , j ) -th entry of A \mathbf{A} A , respectively. For a positive integer m m m , let [ m ] : = { 1 , … , m } [m] := \{1,\ldots,m\} [ m ] := { 1 , … , m } . Let 1 K \mathbf{1}_K 1 K and I K \mathbf{I}_K I K denote the vector of ones in R K \mathbb{R}^K R K and identity matrix in R K × K \mathbb{R}^{K \times K} R K × K , respectively. Let I p \mathbf{I}_p I p denote the diagonal matrix whose diagonal entries consist of p p p ones followed by q q q negative ones. Let Q K \mathcal{Q}_K Q K denote the set of K × K K \times K K × K permutation matrices. Let λ max ( A ) \lambda_{\max}(\mathbf{A}) λ m a x ( A ) , λ min ( A ) \lambda_{\min}(\mathbf{A}) λ m i n ( A ) , σ max ( A ) \sigma_{\max}(\mathbf{A}) σ m a x ( A ) , σ min ( A ) \sigma_{\min}(\mathbf{A}) σ m i n ( A ) denote the maximum/minimum eigenvalues/singular values of A \mathbf{A} A , respectively. Define O ( K ) \mathcal{O}(K) O ( K ) be the set of K × K K \times K K × K orthogonal matrices and O ( p , q ) = { A ∈ R K × K : A I p , q A T = I p , q , p + q = K } \mathcal{O}(p,q) = \{\mathbf{A} \in\mathbb{R}^{K \times K}: \mathbf{A}\mathbf{I}_{p,q}\mathbf{A}^{\mathsf T} = \mathbf{I}_{p,q}, p+q=K\} O ( p , q ) = { A ∈ R K × K : A I p , q A T = I p , q , p + q = K } be the indefinite orthogonal group. For any symmetric positive definite matrix A \mathbf{A} A , let A = V Λ V T \mathbf{A} = \mathbf{V}\mathbf{\Lambda}\mathbf{V}^{\mathsf T} A = VΛ V T be a spectral decomposition and define A ± 1 / 2 : = V Λ ± 1 / 2 V T \mathbf{A}^{\pm1/2} := \mathbf{V}\mathbf{\Lambda}^{\pm1/2}\mathbf{V}^{\mathsf T} A ± 1/2 := V Λ ± 1/2 V T , where, for Λ : = diag ( λ 1 , … , λ d ) \mathbf{\Lambda} := \operatorname{diag}(\lambda_1,\ldots,\lambda_d) Λ := diag ( λ 1 , … , λ d ) , Λ ± 1 / 2 : = diag ( λ 1 ± 1 / 2 , … , λ d ± 1 / 2 ) \mathbf{\Lambda}^{\pm1/2} := \operatorname{diag}(\lambda_1^{\pm1/2},\ldots,\lambda_d^{\pm1/2}) Λ ± 1/2 := diag ( λ 1 ± 1/2 , … , λ d ± 1/2 ) . Let Δ K − 1 : = { x ∈ R K : ∑ i = 1 K x i = 1 , x i ≥ 0 } \Delta^{K-1} := \{\mathbf{x} \in\mathbb{R}^K : \sum_{i=1}^{K} x_i = 1, x_i \ge0\} Δ K − 1 := { x ∈ R K : ∑ i = 1 K x i = 1 , x i ≥ 0 } denote the standard simplex in R K \mathbb{R}^K R K . Let S K − 1 : = { x ∈ R K : ∥ x ∥ 2 = 1 } \mathbb{S}^{K-1} := \{\mathbf{x} \in\mathbb{R}^K : \|\mathbf{x}\|_2 = 1\} S K − 1 := { x ∈ R K : ∥ x ∥ 2 = 1 } denote the unit ℓ 2 \ell_2 ℓ 2 -norm ball in R K \mathbb{R}^K R K . For any matrix A ∈ R n × K \mathbf{A} \in\mathbb{R}^{n \times K} A ∈ R n × K and an index set J ⊂ [ n ] \mathcal{J} \subset[n] J ⊂ [ n ] with ∣ J ∣ = K |\mathcal{J}| = K ∣ J ∣ = K such that A J , ⋅ \mathbf{A}_{\mathcal{J},\cdot} A J , ⋅ is nonsingular, let S ( A , J ) : = { ∑ j ∈ J a j A j , ⋅ ∥ ∑ j ∈ J a j A j , ⋅ ∥ 2 : ∑ j ∈ J a j = 1 , a j ≥ 0 } \mathcal{S}(\mathbf{A},\mathcal{J}) := \left\{ \frac{\sum_{j \in\mathcal{J}} a_j \mathbf{A}_{j,\cdot}}{\left\|\sum_{j \in\mathcal{J}} a_j \mathbf{A}_{j,\cdot}\right\|_2} : \sum_{j \in\mathcal{J}} a_j = 1, a_j \ge0 \right\} S ( A , J ) := { ∥ ∑ j ∈ J a j A j , ⋅ ∥ 2 ∑ j ∈ J a j A j , ⋅ : ∑ j ∈ J a j = 1 , a j ≥ 0 } denote the spherical simplex generated by the rows of A \mathbf{A} A indexed by J \mathcal{J} J .
Methodology
The Multilayer Degree-Corrected Mixed Membership model
In this paper, we consider the degree-corrected mixed-membership stochastic block model in the multilayer setting (ML-DCMM). Concretely, consider a collection of L L L matrices P ( 1 ) , … , P ( L ) ∈ R n × n \mathbf{P}^{(1)},\ldots,\mathbf{P}^{(L)} \in\mathbb{R}^{n\times n} P ( 1 ) , … , P ( L ) ∈ R n × n . From these matrices, we observe L L L adjacency matrices A ( 1 ) , … , A ( L ) \mathbf{A}^{(1)},\ldots,\mathbf{A}^{(L)} A ( 1 ) , … , A ( L ) , where A i j ( l ) ∼ i n d e p Bernoulli ( P i j ( l ) ) A_{ij}^{(l)} \mathrel{\overset{\mathrm{indep}}{\sim}} \operatorname{Bernoulli}(P_{ij}^{(l)}) A ij ( l ) ∼ indep Bernoulli ( P ij ( l ) ) . For simplicity of the analysis, we assume undirected, binary graphs allowing self-loops. Thus, each A ( l ) \mathbf{A}^{(l)} A ( l ) is symmetric, and its diagonal entries are treated in the same way as the off-diagonal entries. This convention is mainly technical, as it avoids a separate treatment of the diagonal elements. The loopless setting can be handled by setting the diagonal entries to zero; this modification does not affect the main methodology, although it would require minor notational changes in the analysis. In the ML-DCMM, we assume P ( l ) \mathbf{P}^{(l)} P ( l ) of the form
P ( l ) = Θ ( l ) Z B ( l ) Z T Θ ( l ) , 1 ≤ l ≤ L , (2) \mathbf{P}^{(l)} = \boldsymbol{\Theta}^{(l)}\mathbf{Z}\mathbf{B}^{(l)}\mathbf{Z}^{\mathsf{T}}\boldsymbol{\Theta}^{(l)}, \qquad1 \le l \le L, \tag*{(2)} P ( l ) = Θ ( l ) Z B ( l ) Z T Θ ( l ) , 1 ≤ l ≤ L , (2)
where Θ ( l ) = diag ( θ ( l ) ) ∈ R n × n \boldsymbol{\Theta}^{(l)} = \operatorname{diag}(\boldsymbol{\theta}^{(l)}) \in\mathbb{R}^{n\times n} Θ ( l ) = diag ( θ ( l ) ) ∈ R n × n is a diagonal matrix that consists of positive degree-correction parameters θ ( l ) = ( θ 1 ( l ) , … , θ n ( l ) ) \boldsymbol{\theta}^{(l)} = (\theta_1^{(l)},\ldots,\theta_n^{(l)}) θ ( l ) = ( θ 1 ( l ) , … , θ n ( l ) ) , Z ∈ R n × K \mathbf{Z} \in\mathbb{R}^{n\times K} Z ∈ R n × K is a membership matrix with ∥ Z i , ⋅ ∥ 1 = 1 \lVert\mathbf{Z}_{i,\cdot}\rVert_1 = 1 ∥ Z i , ⋅ ∥ 1 = 1 and Z i j ≥ 0 Z_{ij} \ge0 Z ij ≥ 0 , and B ( l ) ∈ R K × K \mathbf{B}^{(l)} \in\mathbb{R}^{K\times K} B ( l ) ∈ R K × K is a full-rank symmetric encoding the connectivity across blocks. Specifically, the element in the i i i -th row and j j j -th column of P ( l ) \mathbf{P}^{(l)} P ( l ) is computed as P i j ( l ) = θ i ( l ) θ j ( l ) Z i , ⋅ B ( l ) Z j , ⋅ T P_{ij}^{(l)} = \theta_i^{(l)}\theta_j^{(l)}\mathbf{Z}_{i,\cdot}\mathbf{B}^{(l)}\mathbf{Z}_{j,\cdot}^{\mathsf{T}} P ij ( l ) = θ i ( l ) θ j ( l ) Z i , ⋅ B ( l ) Z j , ⋅ T . Note that the membership matrix Z \mathbf{Z} Z is shared across all layers, indicating that the community structure of a given node is the same across layers, whereas ( θ ( l ) , B ( l ) ) (\boldsymbol{\theta}^{(l)},\mathbf{B}^{(l)}) ( θ ( l ) , B ( l ) ) are layer-specific. This modeling structure is natural in many real-world multilayer networks, where the latent roles of nodes are relatively stable but the edge formation patterns may vary across layers. For example, in transportation networks, a station or location may have a stable role in the system, while traffic intensity and connectivity patterns may change across time periods, days, or operating conditions. The shared matrix Z \mathbf{Z} Z represents these stable node-level roles, whereas the layer-specific parameters θ ( l ) \boldsymbol{\theta}^{(l)} θ ( l ) and B ( l ) \mathbf{B}^{(l)} B ( l ) capture variation in node activity levels and block-level interaction strengths across layers.
The model in Equation (2) extends several existing community detection and low-rank latent space models to multilayer networks by allowing a common mixed-membership matrix and layer-specific degree-correction and block-connectivity parameters. In particular, when L = 1 L = 1 L = 1 , it reduces to the single-layer DCMM [56 , 26 ] . When each row of Z \mathbf{Z} Z is a canonical basis vector in R K \mathbb{R}^{K} R K , it reduces to the ML-DCBM, as in Agterberg et al. [ [1 ] . When both restrictions hold, it reduces to the DCBM of Karrer and Newman [ [29 ] . When the matrices B ( l ) \mathbf{B}^{(l)} B ( l ) have unit diagonal, it reduces to the version of this model studied in Xie and Huang [ [53 ] .
More generally, the ML-DCMM model extends network models with a common invariant subspace shared across the parameters of all the layers [6 , 28 , 52 ] , which extend the notion of latent spaces to multilayer settings. The next proposition formalizes this result.
Proposition 2.1. Consider a collection of symmetric matrices P ( 1 ) , … , P ( L ) ∈ [ 0 , 1 ] n × n \mathbf{P}^{(1)},\ldots,\mathbf{P}^{(L)} \in[0,1]^{n\times n} P ( 1 ) , … , P ( L ) ∈ [ 0 , 1 ] n × n such that P ( l ) = V R ( l ) V T \mathbf{P}^{(l)} = \mathbf{V}\mathbf{R}^{(l)}\mathbf{V}^{\mathsf{T}} P ( l ) = V R ( l ) V T for full-rank matrices V ∈ R n × d \mathbf{V} \in\mathbb{R}^{n\times d} V ∈ R n × d and symmetric R ( l ) ∈ R d × d \mathbf{R}^{(l)} \in\mathbb{R}^{d\times d} R ( l ) ∈ R d × d , l ∈ [ L ] l \in[L] l ∈ [ L ] . Then, there exist a diagonal matrix Θ ∈ R n × n \boldsymbol{\Theta} \in\mathbb{R}^{n\times n} Θ ∈ R n × n , full-rank square symmetric matrices B ( l ) \mathbf{B}^{(l)} B ( l ) , l ∈ [ L ] l \in[L] l ∈ [ L ] and a membership matrix Z \mathbf{Z} Z with ∥ Z i , ⋅ ∥ 1 = 1 \lVert\mathbf{Z}_{i,\cdot}\rVert_1 = 1 ∥ Z i , ⋅ ∥ 1 = 1 and Z i j ≥ 0 Z_{ij} \ge0 Z ij ≥ 0 , i ∈ [ n ] , j ∈ [ d ] i \in[n], j \in[d] i ∈ [ n ] , j ∈ [ d ] such that P ( l ) = Θ Z B ( l ) Z T Θ \mathbf{P}^{(l)} = \boldsymbol{\Theta}\mathbf{Z}\mathbf{B}^{(l)}\mathbf{Z}^{\mathsf{T}}\boldsymbol{\Theta} P ( l ) = Θ Z B ( l ) Z T Θ for l ∈ [ L ] l \in[L] l ∈ [ L ] .
The previous proposition establishes the connection between a shared latent space across the layers encoded in the matrix V \mathbf{V} V and a common mixed-membership matrix Z \mathbf{Z} Z . While the latter provides a more interpretable representation of the latent factors through cluster memberships, the matrix Z \mathbf{Z} Z is generally not unique, as additional conditions are needed to ensure that the memberships are identifiable. We provide some sufficient conditions in § 2.3 through a latent space representation of the model.
Latent Space Geometry of the ML-DCMM
We now examine the latent geometry induced by the ML-DCMM. This geometry links the common membership matrix to the population spectral embedding and therefore motivates our use of spectral methods to fit this model. To this end, consider the eigendecomposition of P ( l ) = V ( l ) Λ ( l ) V ( l ) ⊤ \mathbf{P}^{(l)} = \mathbf{V}^{(l)}\mathbf{\Lambda}^{(l)}\mathbf{V}^{(l)\top} P ( l ) = V ( l ) Λ ( l ) V ( l ) ⊤ , where V ( l ) = [ v 1 ( l ) , … , v K ( l ) ] ∈ R n × K \mathbf{V}^{(l)} = [\mathbf{v}^{(l)}_1,\ldots,\mathbf{v}^{(l)}_K] \in\mathbb{R}^{n \times K} V ( l ) = [ v 1 ( l ) , … , v K ( l ) ] ∈ R n × K contains orthonormal eigenvectors corresponding to the nonzero eigenvalues of P ( l ) \mathbf{P}^{(l)} P ( l ) , and Λ ( l ) = diag { λ 1 ( l ) , … , λ K ( l ) } ∈ R K × K \mathbf{\Lambda}^{(l)} = \operatorname{diag}\{\lambda^{(l)}_1,\ldots,\lambda^{(l)}_K\} \in\mathbb{R}^{K \times K} Λ ( l ) = diag { λ 1 ( l ) , … , λ K ( l ) } ∈ R K × K . For each l ∈ [ L ] l \in[L] l ∈ [ L ] , let p ( l ) p^{(l)} p ( l ) and q ( l ) q^{(l)} q ( l ) denote the numbers of positive and negative eigenvalues of P ( l ) \mathbf{P}^{(l)} P ( l ) , respectively, so that p ( l ) + q ( l ) = K p^{(l)} + q^{(l)} = K p ( l ) + q ( l ) = K . We order the eigenvalues so that λ 1 ( l ) ≥ ⋯ ≥ λ p ( l ) ( l ) > 0 \lambda^{(l)}_1 \ge\cdots\ge\lambda^{(l)}_{p^{(l)}} > 0 λ 1 ( l ) ≥ ⋯ ≥ λ p ( l ) ( l ) > 0 and ∣ λ p ( l ) + 1 ( l ) ∣ ≥ ⋯ ≥ ∣ λ K ( l ) ∣ > 0 |\lambda^{(l)}_{p^{(l)}+1}| \ge\cdots\ge|\lambda^{(l)}_K| > 0 ∣ λ p ( l ) + 1 ( l ) ∣ ≥ ⋯ ≥ ∣ λ K ( l ) ∣ > 0 , where λ p ( l ) + 1 ( l ) , … , λ K ( l ) < 0 \lambda^{(l)}_{p^{(l)}+1},\ldots,\lambda^{(l)}_K < 0 λ p ( l ) + 1 ( l ) , … , λ K ( l ) < 0 .
The generalized random dot product graph (GRDPG) model [54 , 48 ] provides a model-based interpretation of adjacency spectral embedding as latent positions in a low-dimensional space. The following proposition formalizes the connection between spectral embeddings in the GRDPG and the ML-DCMM.
Proposition 2.2 (Spectral representation of the ML-DCMM). Consider the ML-DCMM model in Equation (2) . Let X ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1 / 2 \mathbf{X}^{(l)} = \mathbf{V}^{(l)}|\mathbf{\Lambda}^{(l)}|^{1/2} X ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1/2 be the population spectral embedding of P ( l ) \mathbf{P}^{(l)} P ( l ) , and let I p ( l ) , q ( l ) = sgn ( Λ ( l ) ) \mathbf{I}_{p^{(l)},q^{(l)}} = \operatorname{sgn}(\mathbf{\Lambda}^{(l)}) I p ( l ) , q ( l ) = sgn ( Λ ( l ) ) be the signature matrix associated with the ordered eigendecomposition above. Then there exists a matrix R ( l ) ∈ R K × K \mathbf{R}^{(l)} \in\mathbb{R}^{K \times K} R ( l ) ∈ R K × K such that
X ( l ) = Θ ( l ) Z R ( l ) (3) \mathbf{X}^{(l)} = \mathbf{\Theta}^{(l)}\mathbf{Z}\mathbf{R}^{(l)} \tag*{(3)} X ( l ) = Θ ( l ) Z R ( l ) (3) Moreover, with the same choice of R ( l ) \mathbf{R}^{(l)} R ( l ) , it follows that
B ( l ) = R ( l ) I p ( l ) , q ( l ) R ( l ) ⊤ . (4) \mathbf{B}^{(l)} = \mathbf{R}^{(l)}\mathbf{I}_{p^{(l)},q^{(l)}}\mathbf{R}^{(l)\top}. \tag*{(4)} B ( l ) = R ( l ) I p ( l ) , q ( l ) R ( l ) ⊤ . (4)
Remark 2.1. Proposition 2.2 establishes two useful identities relating the population spectral embeddings to the ML-DCMM model parameters. Equation (2.2) shows that the spectral embedding, although defined through the eigendecomposition of P ( l ) \mathbf{P}^{(l)} P ( l ) , can be written exactly in the model-based form. The matrix R ( l ) \mathbf{R}^{(l)} R ( l ) is not an arbitrary auxiliary factor introduced only for this representation: equation (2.3) shows that the same R ( l ) \mathbf{R}^{(l)} R ( l ) also encodes the layer-specific block connectivity matrix B ( l ) \mathbf{B}^{(l)} B ( l ) through the indefinite inner product.
Observe that each row of X ( l ) \mathbf{X}^{(l)} X ( l ) can be written as X i , ⋅ ( l ) = θ i ( l ) R ( l ) ⊤ Z i , ⋅ \mathbf{X}^{(l)}_{i,\cdot} = \theta^{(l)}_i\mathbf{R}^{(l)\top}\mathbf{Z}_{i,\cdot} X i , ⋅ ( l ) = θ i ( l ) R ( l ) ⊤ Z i , ⋅ . That is, each row of X ( l ) \mathbf{X}^{(l)} X ( l ) is a convex combination of the rows of R ( l ) \mathbf{R}^{(l)} R ( l ) , with coefficients Z i , ⋅ \mathbf{Z}_{i,\cdot} Z i , ⋅ shared across all layers, multiplied by its degree-correction parameter θ i ( l ) \theta^{(l)}_i θ i ( l ) . In particular, for a pure node i i i in community k k k , we have X i , ⋅ ( l ) = θ i ( l ) R k , ⋅ ( l ) \mathbf{X}^{(l)}_{i,\cdot} = \theta^{(l)}_i\mathbf{R}^{(l)}_{k,\cdot} X i , ⋅ ( l ) = θ i ( l ) R k , ⋅ ( l ) . Therefore, the rows of R ( l ) \mathbf{R}^{(l)} R ( l ) determine the community-specific directions of the population embedding, while θ i ( l ) \theta^{(l)}_i θ i ( l ) controls the radial position along the corresponding ray. This fact motivates estimation approaches for Z \mathbf{Z} Z via row-normalizing X ( l ) \mathbf{X}^{(l)} X ( l ) [26 , 56 ] , which removes the radial effect of the degree-correction parameter and reveals a geometry that contains information about the communities. However, in the multilayer setup, the different parameters on each layer have a different effect on the normalized embeddings from each layer, complicating estimation of these memberships directly from the normalized embeddings. We thus handle the different layers separately in order to construct an appropriate estimator.
Identifiability of multilayer mixed memberships with degree corrections
Identifiability of the DCMM parameters generally consists of two problems: global scale and community membership-specific scaling. In the single-layer model, the former issue can be resolved by assuming ∥ Z i , ⋅ ∥ 1 = 1 \|\mathbf{Z}_{i,\cdot}\|_1 = 1 ∥ Z i , ⋅ ∥ 1 = 1 for all i ∈ [ n ] i \in[n] i ∈ [ n ] , and an additional constraint to distinguish the degree parameters from the global connectivity, such as ∑ i = 1 n θ i = n \sum_{i=1}^{n}\theta_i = n ∑ i = 1 n θ i = n or ∥ B ∥ max = 1 \|\mathbf{B}\|_{\max} = 1 ∥ B ∥ m a x = 1 . The second identifiability problem implies that for any positive diagonal matrix D ∈ R K × K \mathbf{D} \in\mathbb{R}^{K \times K} D ∈ R K × K , there exists a positive diagonal matrix N D ∈ R n × n \mathbf{N}_{D} \in\mathbb{R}^{n \times n} N D ∈ R n × n such that ( Θ , Z , B ) (\boldsymbol{\Theta}, \mathbf{Z}, \mathbf{B}) ( Θ , Z , B ) and ( Θ N D − 1 , N D Z D , D − 1 B D − 1 ) (\boldsymbol{\Theta}\mathbf{N}_{D}^{-1}, \mathbf{N}_{D}\mathbf{Z}\mathbf{D}, \mathbf{D}^{-1}\mathbf{B}\mathbf{D}^{-1}) ( Θ N D − 1 , N D ZD , D − 1 B D − 1 ) yield the same probability matrix P \mathbf{P} P [26 ] . One can address this nonidentifiability issue by imposing a constraint on the entries of B \mathbf{B} B , such as diag ( B ) = 1 K \operatorname{diag}(\mathbf{B}) = \mathbf{1}_{K} diag ( B ) = 1 K . Note that this nonidentifiability of the membership matrix is specific to mixed-membership models and does not arise under non-overlapping memberships.
Moving onto multilayer models, identifiability conditions must allow sufficient flexibility on each layer to capture heterogeneous patterns. Directly imposing the identifiability conditions for single-layer block models on every layer can unnecessarily restrict the ML-DCMM parameter space. For example, imposing a unit-diagonal constraint on each B ( l ) \mathbf{B}^{(l)} B ( l ) generally requires layer-specific rescalings of the membership matrix, which need not preserve a common Z \mathbf{Z} Z across layers (see Appendix § 11 .5.2). Instead, for the ML-DCMM, we designate one layer as a reference layer to fix the parameterization of the shared membership matrix, and represent the remaining layers in terms of this reference. Without loss of generality, we take layer 1 to be the reference layer. We state identifiability conditions of the ML-DCMM.
Condition 1 (Identifiability conditions). The following conditions hold.
(a) Membership matrix. Each row of Z \mathbf{Z} Z has unit ℓ 1 \ell_{1} ℓ 1 -norm, and there exists at least one pure node for each community.
(b) Community-overlap connectivity. Define a community graph G Z = ( [ K ] , E Z ) \mathcal{G}_{Z} = ([K], \mathcal{E}_{Z}) G Z = ([ K ] , E Z ) , where
{ a , b } ∈ E Z ⟺ ∃ i ∈ [ n ] such that Z i a > 0 , Z i b > 0 , a ≠ b . \{a,b\} \in\mathcal{E}_{Z} \Longleftrightarrow\exists i \in[n]\ \text{such that}\ Z_{ia} > 0, Z_{ib} > 0,\quad a \ne b. { a , b } ∈ E Z ⟺ ∃ i ∈ [ n ] such that Z ia > 0 , Z ib > 0 , a = b . Then, G Z \mathcal{G}_{Z} G Z is connected.
(c) Block connectivity matrices. The matrix B ( l ) \mathbf{B}^{(l)} B ( l ) is full rank for every 1 ≤ l ≤ L 1 \le l \le L 1 ≤ l ≤ L . Moreover, for the reference layer l = 1 l = 1 l = 1 the matrix R ( 1 ) \mathbf{R}^{(1)} R ( 1 ) defined in Equation (2.3) satisfies ∥ ( R ( 1 ) ) k , ⋅ ∥ 2 = 1 \lVert(\mathbf{R}^{(1)})_{k,\cdot}\rVert_{2} = 1 ∥( R ( 1 ) ) k , ⋅ ∥ 2 = 1 for all k ∈ [ K ] k \in[K] k ∈ [ K ] . Lastly, ∥ B ( l ) ∥ max = 1 \lVert\mathbf{B}^{(l)}\rVert_{\max} = 1 ∥ B ( l ) ∥ m a x = 1 for all l ≠ 1 l \ne1 l = 1 .
Under these conditions, the ML-DCMM parameters are identifiable up to a common permutation of the community labels as shown next.
Proposition 2.3 (Identifiability of ML-DCMM). Suppose that two parameterizations { Θ ( l ) } l = 1 L \{\boldsymbol{\Theta}^{(l)}\}_{l=1}^{L} { Θ ( l ) } l = 1 L , Z \mathbf{Z} Z , { B ( l ) } l = 1 L \{\mathbf{B}^{(l)}\}_{l=1}^{L} { B ( l ) } l = 1 L and { Θ ~ ( l ) } l = 1 L \{\widetilde{\boldsymbol{\Theta}}^{(l)}\}_{l=1}^{L} { Θ ( l ) } l = 1 L , Z ~ \widetilde{\mathbf{Z}} Z , { B ~ ( l ) } l = 1 L \{\widetilde{\mathbf{B}}^{(l)}\}_{l=1}^{L} { B ( l ) } l = 1 L both satisfy Condition 1 and generate the same population probability matrices, that is, P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) = Θ ~ ( l ) Z ~ B ~ ( l ) Z ~ ⊤ Θ ~ ( l ) \mathbf{P}^{(l)} = \boldsymbol{\Theta}^{(l)}\mathbf{Z}\mathbf{B}^{(l)}\mathbf{Z}^{\top}\boldsymbol{\Theta}^{(l)} = \widetilde{\boldsymbol{\Theta}}^{(l)}\widetilde{\mathbf{Z}}\widetilde{\mathbf{B}}^{(l)}\widetilde{\mathbf{Z}}^{\top}\widetilde{\boldsymbol{\Theta}}^{(l)} P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) for all l ∈ [ L ] l \in[L] l ∈ [ L ] . Then the model parameters are identifiable up to a common permutation of the community labels. More precisely, there exists a K × K K \times K K × K permutation matrix Π \mathbf{\Pi} Π such that, for every 1 ≤ l ≤ L 1 \le l \le L 1 ≤ l ≤ L ,
Θ ~ ( l ) = Θ ( l ) , Z ~ = Z Π , B ~ ( l ) = Π ⊤ B ( l ) Π . \widetilde{\boldsymbol{\Theta}}^{(l)} = \boldsymbol{\Theta}^{(l)},\qquad\widetilde{\mathbf{Z}} = \mathbf{Z}\mathbf{\Pi},\qquad\widetilde{\mathbf{B}}^{(l)} = \mathbf{\Pi}^{\top}\mathbf{B}^{(l)}\mathbf{\Pi}. Θ ( l ) = Θ ( l ) , Z = ZΠ , B ( l ) = Π ⊤ B ( l ) Π .
Proposed estimation procedure
Estimation of the mixed memberships in the model can be done by identifying the indices of the pure nodes, which characterize the communities. From a latent space point of view, this reduces to estimating the corners of the space in which the latent positions lie. Single-layer methods typically resolve this problem by normalizing the latent positions to remove the degree heterogeneity effect, followed by a corner estimation procedure [56 , 26 , 40 ] . In the multilayer setup, these methods are not directly applicable due to the potential heterogeneity across layers that needs to be individually handled.
To remove the degree heterogeneity effect, in this paper we use an ℓ 2 \ell_2 ℓ 2 normalization of the latent positions for each layer. At the population level, consider the matrices X ( l ) \mathbf{X}^{(l)} X ( l ) and R ( l ) \mathbf{R}^{(l)} R ( l ) from Proposition 2.2. Define a diagonal matrix M ( l ) : = diag ( m 1 ( l ) , … , m K ( l ) ) ∈ R K × K \mathbf{M}^{(l)}:=\operatorname{diag}(m_1^{(l)},\ldots,m_K^{(l)})\in\mathbb{R}^{K\times K} M ( l ) := diag ( m 1 ( l ) , … , m K ( l ) ) ∈ R K × K with m k ( l ) : = ∥ R k , ⋅ ( l ) ∥ 2 m_k^{(l)}:=\|\mathbf{R}_{k,\cdot}^{(l)}\|_2 m k ( l ) := ∥ R k , ⋅ ( l ) ∥ 2 so that S ( l ) : = ( M ( l ) ) − 1 R ( l ) \mathbf{S}^{(l)}:=(\mathbf{M}^{(l)})^{-1}\mathbf{R}^{(l)} S ( l ) := ( M ( l ) ) − 1 R ( l ) denotes the ℓ 2 \ell_2 ℓ 2 -norm normalized corner matrix. Let Y ( l ) ∈ R n × K \mathbf{Y}^{(l)}\in\mathbb{R}^{n\times K} Y ( l ) ∈ R n × K denote the ℓ 2 \ell_2 ℓ 2 -norm normalized spectral embedding matrix defined by
Y i , ⋅ ( l ) = X i , ⋅ ( l ) ∥ X i , ⋅ ( l ) ∥ 2 = ( Z i , ⋅ T M ( l ) S ( l ) ) T ∥ ( Z i , ⋅ T M ( l ) S ( l ) ) T ∥ 2 ∈ S S ( S ( l ) , [ K ] ) , 1 ≤ i ≤ n . (5) \mathbf{Y}_{i,\cdot}^{(l)}=\frac{\mathbf{X}_{i,\cdot}^{(l)}}{\|\mathbf{X}_{i,\cdot}^{(l)}\|_2}=\frac{(\mathbf{Z}_{i,\cdot}^{\mathsf{T}}\mathbf{M}^{(l)}\mathbf{S}^{(l)})^{\mathsf{T}}}{\|(\mathbf{Z}_{i,\cdot}^{\mathsf{T}}\mathbf{M}^{(l)}\mathbf{S}^{(l)})^{\mathsf{T}}\|_2}\in\mathcal{S}\mathcal{S}(\mathbf{S}^{(l)},[K]),\quad1\leq i\leq n. \tag*{(5)} Y i , ⋅ ( l ) = ∥ X i , ⋅ ( l ) ∥ 2 X i , ⋅ ( l ) = ∥ ( Z i , ⋅ T M ( l ) S ( l ) ) T ∥ 2 ( Z i , ⋅ T M ( l ) S ( l ) ) T ∈ SS ( S ( l ) , [ K ]) , 1 ≤ i ≤ n . (5)
Consequently, the rows of Y ( l ) \mathbf{Y}^{(l)} Y ( l ) lie on a spherical simplex in S K − 1 \mathbb{S}^{K-1} S K − 1 , whose corners are given by the rows of S ( l ) \mathbf{S}^{(l)} S ( l ) . Moreover, Y i , ⋅ ( l ) = S k , ⋅ ( l ) \mathbf{Y}_{i,\cdot}^{(l)}=\mathbf{S}_{k,\cdot}^{(l)} Y i , ⋅ ( l ) = S k , ⋅ ( l ) if and only if node i i i is pure in community k k k , which implies that the corners of the spherical simplex correspond precisely to pure nodes in each community, and these are the same across every layer. This observation motivates our approach for clustering.
Our proposed method for corner estimation constructs a matrix that concatenates row-normalized adjacency spectral embeddings (CNASE) across layers to jointly utilize layer-wise information. For each l ∈ [ L ] l\in[L] l ∈ [ L ] , let V ^ ( l ) ∈ R n × K \widehat{\mathbf{V}}^{(l)}\in\mathbb{R}^{n\times K} V ( l ) ∈ R n × K contain the leading eigenvectors of A ( l ) \mathbf{A}^{(l)} A ( l ) corresponding to its K K K largest eigenvalues in magnitude, and let Λ ^ ( l ) \widehat{\boldsymbol{\Lambda}}^{(l)} Λ ( l ) be the diagonal matrix containing the corresponding eigenvalues. The rank-K K K adjacency spectral embedding [49 , 48 ] is defined as X ^ ( l ) : = V ^ ( l ) ∣ Λ ^ ( l ) ∣ 1 / 2 \widehat{\mathbf{X}}^{(l)}:=\widehat{\mathbf{V}}^{(l)}|\widehat{\boldsymbol{\Lambda}}^{(l)}|^{1/2} X ( l ) := V ( l ) ∣ Λ ( l ) ∣ 1/2 . We then estimate the shared corner indices using the concatenated normalized embeddings Y ^ ( l ) \widehat{\mathbf{Y}}^{(l)} Y ( l ) computed as in Equation (5) , which combine the L L L layer-wise spherical-simplex structures. This procedure is summarized in Algorithm 1.
Algorithm 1 Corner and membership estimation on concatenated normalized adjacency spectral embeddings (CNASE)
Input: Adjacency matrices { A ( 1 ) , … , A ( L ) } \{\mathbf{A}^{(1)},\ldots,\mathbf{A}^{(L)}\} { A ( 1 ) , … , A ( L ) } and number of communities K K K .
Ensure: Membership estimate Z ^ 0 \widehat{\mathbf{Z}}_0 Z 0 and matrices { U ^ ( 1 ) , … , U ^ ( L ) } \{\widehat{\mathbf{U}}^{(1)},\ldots,\widehat{\mathbf{U}}^{(L)}\} { U ( 1 ) , … , U ( L ) }
for l = 1 l=1 l = 1 to L L L do
Obtain the adjacency spectral embedding X ^ ( l ) \widehat{\mathbf{X}}^{(l)} X ( l ) of A ( l ) \mathbf{A}^{(l)} A ( l ) .
Normalize the rows of X ^ ( l ) \widehat{\mathbf{X}}^{(l)} X ( l ) to obtain Y ^ ( l ) \widehat{\mathbf{Y}}^{(l)} Y ( l ) , where Y ^ i , ⋅ ( l ) = X ^ i , ⋅ ( l ) / ∥ X ^ i , ⋅ ( l ) ∥ 2 \widehat{\mathbf{Y}}_{i,\cdot}^{(l)}=\widehat{\mathbf{X}}_{i,\cdot}^{(l)}/\|\widehat{\mathbf{X}}_{i,\cdot}^{(l)}\|_2 Y i , ⋅ ( l ) = X i , ⋅ ( l ) /∥ X i , ⋅ ( l ) ∥ 2 for all i ∈ [ n ] i\in[n] i ∈ [ n ] .
end for
Concatenate the normalized adjacency spectral embeddings and form Y ^ = [ Y ^ ( 1 ) ⋯ Y ^ ( L ) ] \widehat{\mathbf{Y}}=[\widehat{\mathbf{Y}}^{(1)}\cdots\widehat{\mathbf{Y}}^{(L)}] Y = [ Y ( 1 ) ⋯ Y ( L ) ] .
Apply a corner estimation method to the rows of Y ^ \widehat{\mathbf{Y}} Y and obtain a set of K K K corner indices I ^ \widehat{\mathcal{I}} I .
For each l ∈ [ L ] l\in[L] l ∈ [ L ] , set S ^ ( l ) = Y ^ I ^ , ⋅ ( l ) \widehat{\mathbf{S}}^{(l)}=\widehat{\mathbf{Y}}_{\widehat{\mathcal{I}},\cdot}^{(l)} S ( l ) = Y I , ⋅ ( l ) , and form S ^ = [ S ^ ( 1 ) ⋯ S ^ ( L ) ] ∈ R K × K L \widehat{\mathbf{S}}=[\widehat{\mathbf{S}}^{(1)}\cdots\widehat{\mathbf{S}}^{(L)}]\in\mathbb{R}^{K\times KL} S = [ S ( 1 ) ⋯ S ( L ) ] ∈ R K × K L .
for l = 1 l=1 l = 1 to L L L do
Obtain U ^ ( l ) = X ^ ( l ) ( S ^ ( l ) ) − 1 \widehat{\mathbf{U}}^{(l)}=\widehat{\mathbf{X}}^{(l)}(\widehat{\mathbf{S}}^{(l)})^{-1} U ( l ) = X ( l ) ( S ( l ) ) − 1 .
end for
For the reference layer, normalize U ^ ( 1 ) \widehat{\mathbf{U}}^{(1)} U ( 1 ) by U ~ ( 1 ) = diag ( U ^ ( 1 ) 1 K ) − 1 U ^ ( 1 ) \widetilde{\mathbf{U}}^{(1)}=\operatorname{diag}(\widehat{\mathbf{U}}^{(1)}\mathbf{1}_K)^{-1}\widehat{\mathbf{U}}^{(1)} U ( 1 ) = diag ( U ( 1 ) 1 K ) − 1 U ( 1 ) and obtain membership estimate Z ^ 0 \widehat{\mathbf{Z}}_0 Z 0 by ( Z ^ 0 ) i , ⋅ = Π Δ K − 1 ( U ~ i , ⋅ ( 1 ) ) (\widehat{\mathbf{Z}}_0)_{i,\cdot}=\Pi_{\Delta_{K-1}}(\widetilde{\mathbf{U}}_{i,\cdot}^{(1)}) ( Z 0 ) i , ⋅ = Π Δ K − 1 ( U i , ⋅ ( 1 ) ) .
The corner-estimation step in Algorithm 1 can be treated as a plug-in step. That is, the proposed procedure is not tied to a particular vertex-hunting routine; any suitable simplex corner-finding method can be used. For any corner candidate set I ⊆ [ n ] \mathcal{I} \subseteq[n] I ⊆ [ n ] with ∣ I ∣ ≥ K |\mathcal{I}| \ge K ∣ I ∣ ≥ K , define the averaged vertex-hunting objective
Φ ~ L , I ( J ) : = max i ∈ I 1 L ∑ l = 1 L d ( Y ^ i , ⋅ ( l ) , SS ( Y ^ ( l ) , J ) ) , J ⊆ I , ∣ J ∣ = K . (6) \widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J}) := \max_{i\in\mathcal{I}} \frac{1}{L}\sum_{l=1}^{L} d\left(\widehat{Y}_{i,\cdot}^{(l)},\operatorname{SS}\left(\widehat{Y}^{(l)},\mathcal{J}\right)\right),\quad\mathcal{J}\subseteq\mathcal{I},\ |\mathcal{J}|=K. \tag*{(6)} Φ L , I ( J ) := i ∈ I max L 1 l = 1 ∑ L d ( Y i , ⋅ ( l ) , SS ( Y ( l ) , J ) ) , J ⊆ I , ∣ J ∣ = K . (6)
Here, d ( ⋅ , ⋅ ) d(\cdot,\cdot) d ( ⋅ , ⋅ ) denotes the Euclidean distance between a point and a set. We use a variant of Sketched Vertex Search [26 ] equipped with the k k k -medoids algorithm for the corner estimation as follows:
Denoise step (optional ). For a value m > K m>K m > K , apply the k k k -medoids algorithm with m m m medoids to the rows of the concatenated normalized embedding Y ^ \widehat{Y} Y from Algorithm 1, and let M ^ m \widehat{\mathcal{M}}_m M m denote the resulting index set of medoids.
Corner search step. Select J ^ ∈ arg min J ⊆ M ^ m ∣ J ∣ = K Φ ~ L , M ^ m ( J ) \widehat{\mathcal{J}}\in\underset{\substack{\mathcal{J}\subseteq\widehat{\mathcal{M}}_m\\|\mathcal{J}|=K}}{\arg\min}\ \widetilde{\Phi}_{L,\widehat{\mathcal{M}}_m}(\mathcal{J}) J ∈ J ⊆ M m ∣ J ∣ = K arg min Φ L , M m ( J ) .
The first denoise step is optional but recommended for computational efficiency. Otherwise, one can find corners directly from the set of n n n concatenated adjacency spectral embeddings instead of m m m medoids. In practice, we choose the number of medoids m m m in a data-driven manner (see Appendix § 9 for details).
Connection with a tensor decomposition
Several approaches for multilayer clustering and joint embedding often rely in joint spectral decompositions that recover informative signal about the communities or the shared latent space [44 , 34 , 1 ] . In contrast, the structure of the ML-DCMM does not admit a joint spectral decomposition to recover the mixed-membership matrix due to individual-specific parameters. However, here we show that the parameters of the model can be expressed as the solution of the ParaTuck-2 decomposition [22 ] . This connection motivate us to propose an algorithmic refinement of the parameter estimates based on computing this decomposition.
Recall that the ParaTuck-2 decomposition (1) with A = I n \mathbf{A}=\mathbf{I}_n A = I n and B = I K \mathbf{B}=\mathbf{I}_K B = I K induces two diagonal matrices C ( l ) ∈ R n × n \mathbf{C}^{(l)}\in\mathbb{R}^{n\times n} C ( l ) ∈ R n × n and D ( l ) ∈ R K × K \mathbf{D}^{(l)}\in\mathbb{R}^{K\times K} D ( l ) ∈ R K × K such that the l l l -th frontal slice of U \mathcal{U} U can be written as U ( l ) = C ( l ) H D ( l ) \mathbf{U}^{(l)}=\mathbf{C}^{(l)}\mathbf{H}\mathbf{D}^{(l)} U ( l ) = C ( l ) H D ( l ) for l ∈ [ L ] l\in[L] l ∈ [ L ] . Under the ML-DCMM, define a corner-coordinate matrix U ( l ) : = X ( l ) ( S ( l ) ) − 1 \mathbf{U}^{(l)}:=\mathbf{X}^{(l)}(\mathbf{S}^{(l)})^{-1} U ( l ) := X ( l ) ( S ( l ) ) − 1 . By the definition of S ( l ) \mathbf{S}^{(l)} S ( l ) , we obtain the exact ParaTuck-2 decomposition for the corner-coordinate matrix U ( l ) = Θ ( l ) Z M ( l ) \mathbf{U}^{(l)}=\mathbf{\Theta}^{(l)}\mathbf{Z}\mathbf{M}^{(l)} U ( l ) = Θ ( l ) Z M ( l ) . Note that, by Condition 1(c), we have M ( 1 ) = I K \mathbf{M}^{(1)}=\mathbf{I}_K M ( 1 ) = I K and hence U ( 1 ) = Θ ( 1 ) Z \mathbf{U}^{(1)}=\mathbf{\Theta}^{(1)}\mathbf{Z} U ( 1 ) = Θ ( 1 ) Z . Consequently, row-wise ℓ 1 \ell_1 ℓ 1 -norm normalization of U ( 1 ) \mathbf{U}^{(1)} U ( 1 ) recovers the common membership matrix Z \mathbf{Z} Z . Note that applying the same normalization to non-reference layers does not necessarily recover Z \mathbf{Z} Z . Additional Euclidean projection onto the standard simplex Δ K − 1 \Delta^{K-1} Δ K − 1 , denoted by Π Δ K − 1 \Pi_{\Delta^{K-1}} Π Δ K − 1 in Algorithm 1, ensures the membership estimates to lie on the standard simplex.
This equivalent expression motivates the following refinement step: Construct U ^ ( l ) : = X ^ ( l ) ( S ^ ( l ) ) − 1 \widehat{\mathbf{U}}^{(l)}:=\widehat{\mathbf{X}}^{(l)}(\widehat{\mathbf{S}}^{(l)})^{-1} U ( l ) := X ( l ) ( S ( l ) ) − 1 for all l ∈ [ L ] l\in[L] l ∈ [ L ] , with X ^ ( l ) \widehat{\mathbf{X}}^{(l)} X ( l ) and S ^ ( l ) \widehat{\mathbf{S}}^{(l)} S ( l ) as defined in Algorithm 1. Define a mean square error loss L ( { Θ ( l ) } l = 1 L , Z , { M ( l ) } l = 1 L ) : = ∑ l = 1 L ∥ U ^ ( l ) − Θ ( l ) Z M ( l ) ∥ F 2 \mathcal{L}(\{ \mathbf{\Theta}^{(l)}\}_{l=1}^{L},\mathbf{Z},\{\mathbf{M}^{(l)}\}_{l=1}^{L}):=\sum_{l=1}^{L}\left\|\widehat{\mathbf{U}}^{(l)}-\mathbf{\Theta}^{(l)}\mathbf{Z}\mathbf{M}^{(l)}\right\|_{F}^{2} L ({ Θ ( l ) } l = 1 L , Z , { M ( l ) } l = 1 L ) := ∑ l = 1 L U ( l ) − Θ ( l ) Z M ( l ) F 2 . Implement ALS [12 ] with random initiates { Θ ^ 0 ( l ) , M 0 ( l ) } l = 1 L \{\widehat{\Theta}_0^{(l)},\mathbf{M}_0^{(l)}\}_{l=1}^{L} { Θ 0 ( l ) , M 0 ( l ) } l = 1 L and spectral membership estimates Z ^ 0 \widehat{\mathbf{Z}}_0 Z 0 defined in algorithm 1. We can obtain a refined membership estimate Z ^ \widehat{\mathbf{Z}} Z as the output of the refinement step. See Appendix § 9 for the algorithm.
Theory
In this section, we establish the non-asymptotic properties of mixed-membership estimation following our proposed method. We state regularity assumptions for theoretical analysis. For each layer l l l , define the normalized degree-weighted membership Gram matrix G ( l ) ∈ R K × K \mathbf{G}^{(l)} \in\mathbb{R}^{K \times K} G ( l ) ∈ R K × K as G ( l ) : = K ∥ θ ( l ) ∥ 2 [ Z ⊤ ( Θ ( l ) ) 2 Z ] 1 / 2 \mathbf{G}^{(l)} := \frac{\sqrt{K}}{\lVert\boldsymbol{\theta}^{(l)}\rVert_2}\left[\mathbf{Z}^{\top}(\boldsymbol{\Theta}^{(l)})^2\mathbf{Z}\right]^{1/2} G ( l ) := ∥ θ ( l ) ∥ 2 K [ Z ⊤ ( Θ ( l ) ) 2 Z ] 1/2 .
Assumption 1. There exists a constant C G > 0 C_G > 0 C G > 0 , independent of l l l , such that
max { ∥ G ( l ) ∥ o p , ∥ ( G ( l ) ) − 1 ∥ o p } ≤ C G , 1 ≤ l ≤ L . \max\left\{\lVert\mathbf{G}^{(l)}\rVert_{\mathrm{op}},\lVert(\mathbf{G}^{(l)})^{-1}\rVert_{\mathrm{op}}\right\} \le C_G,\quad1 \le l \le L. max { ∥ G ( l ) ∥ op , ∥( G ( l ) ) − 1 ∥ op } ≤ C G , 1 ≤ l ≤ L .
Assumption Assumption 1 requires the eigenvalues of G ( l ) \mathbf{G}^{(l)} G ( l ) to be uniformly bounded above and away from zero across layers. This implies the degree-weighted membership vectors ( Θ ( l ) Z ) i , ⋅ (\boldsymbol{\Theta}^{(l)}\mathbf{Z})_{i,\cdot} ( Θ ( l ) Z ) i , ⋅ to remain sufficiently diverse across all community directions. In particular, each community must carry a nonvanishing amount of degree-weighted squared membership mass.
Assumption 2. For each layer l l l , let σ B ( l ) ∈ ( 0 , 1 ) \sigma_B^{(l)} \in(0,1) σ B ( l ) ∈ ( 0 , 1 ) be a constant independent of n n n , with inf l ≥ 1 σ B ( l ) = σ B , inf > 0 \inf_{l\ge1}\sigma_B^{(l)} = \sigma_{B,\inf} > 0 inf l ≥ 1 σ B ( l ) = σ B , i n f > 0 , such that σ min ( B ( l ) ) ≥ σ B ( l ) \sigma_{\min}(\mathbf{B}^{(l)}) \ge\sigma_B^{(l)} σ m i n ( B ( l ) ) ≥ σ B ( l ) for all l ∈ [ L ] l \in[L] l ∈ [ L ] .
Assumption Assumption 2 prevents the layer-specific block connectivity matrices from becoming asymptotically singular. Next, for each layer l l l , define
β n ( l ) = ( θ max ( l ) θ min ( l ) ) K 8 θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ( σ B ( l ) ) 3 ∥ θ ( l ) ∥ 2 4 . \beta_n^{(l)} = \left(\frac{\theta_{\max}^{(l)}}{\theta_{\min}^{(l)}}\right)\frac{K^8\theta_{\max}^{(l)}\lVert\boldsymbol{\theta}^{(l)}\rVert_1\log(n)}{(\sigma_B^{(l)})^3\lVert\boldsymbol{\theta}^{(l)}\rVert_2^4}. β n ( l ) = ( θ m i n ( l ) θ m a x ( l ) ) ( σ B ( l ) ) 3 ∥ θ ( l ) ∥ 2 4 K 8 θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) .
We refer to β n ( l ) \beta_n^{(l)} β n ( l ) as the layer-wise spectral difficulty parameter, which is analogous to the corresponding quantity in the analysis of multilayer degree-corrected block models [1 ] .
Assumption 3. There exist constants c d e g > 0 c_{\mathrm{deg}} > 0 c deg > 0 and c β > 0 c_{\beta} > 0 c β > 0 , independent of l l l , where c d e g c_{\mathrm{deg}} c deg is sufficiently large and c β c_{\beta} c β is sufficiently small, such that θ min ( l ) ∥ θ ( l ) ∥ 1 ≥ c d e g log ( n ) \theta_{\min}^{(l)}\lVert\boldsymbol{\theta}^{(l)}\rVert_1 \ge c_{\mathrm{deg}}\log(n) θ m i n ( l ) ∥ θ ( l ) ∥ 1 ≥ c deg log ( n ) and max 1 ≤ l ≤ L β n ( l ) ≤ c β \max_{1\le l\le L}\beta_n^{(l)} \le c_{\beta} max 1 ≤ l ≤ L β n ( l ) ≤ c β for all l ∈ [ L ] l \in[L] l ∈ [ L ] .
For the theoretical analysis, we work with aligned empirical embeddings. For each layer l l l , let W ∗ ( l ) \mathbf{W}_*^{(l)} W ∗ ( l ) be the orthogonal Procrustes alignment matrix between the empirical and population embeddings and define corresponding aligned embeddings X ~ ( l ) : = X ^ ( l ) W ∗ ( l ) \widetilde{\mathbf{X}}^{(l)} := \widehat{\mathbf{X}}^{(l)}\mathbf{W}_*^{(l)} X ( l ) := X ( l ) W ∗ ( l ) and Y ~ i , ⋅ ( l ) : = X ~ i , ⋅ ( l ) ∥ X ~ i , ⋅ ( l ) ∥ 2 \widetilde{\mathbf{Y}}_{i,\cdot}^{(l)} := \frac{\widetilde{\mathbf{X}}_{i,\cdot}^{(l)}}{\lVert\widetilde{\mathbf{X}}_{i,\cdot}^{(l)}\rVert_2} Y i , ⋅ ( l ) := ∥ X i , ⋅ ( l ) ∥ 2 X i , ⋅ ( l ) . Since W ∗ ( l ) \mathbf{W}_*^{(l)} W ∗ ( l ) is orthogonal, we have Y ~ ( l ) = Y ^ ( l ) W ∗ ( l ) \widetilde{\mathbf{Y}}^{(l)} = \widehat{\mathbf{Y}}^{(l)}\mathbf{W}_*^{(l)} Y ( l ) = Y ( l ) W ∗ ( l ) . Let Y ~ : = [ Y ~ ( 1 ) ⋯ Y ~ ( L ) ] \widetilde{\mathbf{Y}} := [\widetilde{\mathbf{Y}}^{(1)}\cdots\widetilde{\mathbf{Y}}^{(L)}] Y := [ Y ( 1 ) ⋯ Y ( L ) ] denote the matrix of concatenated normalized adjacency spectral embeddings. The matrices X ~ ( l ) \widetilde{\mathbf{X}}^{(l)} X ( l ) and Y ~ ( l ) \widetilde{\mathbf{Y}}^{(l)} Y ( l ) are used only for analysis; the estimator in Algorithm 1 is computed from the observable embeddings X ^ ( l ) \widehat{\mathbf{X}}^{(l)} X ( l ) and Y ^ ( l ) \widehat{\mathbf{Y}}^{(l)} Y ( l ) . It can be verified that this alignment does not change the vertex-hunting objective. Indeed, for any candidate set I ⊂ [ n ] \mathcal{I}\subset[n] I ⊂ [ n ] and any J ⊂ I \mathcal{J}\subset\mathcal{I} J ⊂ I , d ( Y ^ i , ⋅ ( l ) , S S ( Y ^ ( l ) , J ) ) = d ( Y ~ i , ⋅ ( l ) , S S ( Y ~ ( l ) , J ) ) d(\widehat{\mathbf{Y}}_{i,\cdot}^{(l)},SS(\widehat{\mathbf{Y}}^{(l)},\mathcal{J}))=d(\widetilde{\mathbf{Y}}_{i,\cdot}^{(l)},SS(\widetilde{\mathbf{Y}}^{(l)},\mathcal{J})) d ( Y i , ⋅ ( l ) , S S ( Y ( l ) , J )) = d ( Y i , ⋅ ( l ) , S S ( Y ( l ) , J )) for all i i i , l l l , and J \mathcal{J} J . Hence the minimizers of the observable objective based on Y ^ \widehat{\mathbf{Y}} Y are identical to those of the aligned objective based on Y ~ \widetilde{\mathbf{Y}} Y .
Geometry and Stability of the Spherical Normalization
We introduce the layer-wise population transformations T ( l ) : Δ K − 1 → S K − 1 T^{(l)}:\Delta^{K-1}\to\mathbb{S}^{K-1} T ( l ) : Δ K − 1 → S K − 1 defined by T ( l ) ( z ) = ( R ( l ) ) ⊤ z ∥ ( R ( l ) ) ⊤ z ∥ 2 T^{(l)}(\mathbf{z})=\frac{(\mathbf{R}^{(l)})^{\top}\mathbf{z}}{\lVert(\mathbf{R}^{(l)})^{\top}\mathbf{z}\rVert_2} T ( l ) ( z ) = ∥( R ( l ) ) ⊤ z ∥ 2 ( R ( l ) ) ⊤ z for all l ∈ [ L ] l\in[L] l ∈ [ L ] . For a node with membership vector Z i , ⋅ \mathbf{Z}_{i,\cdot} Z i , ⋅ , this gives the population row-normalized spectral embedding in layer l l l by T ( l ) ( Z i , ⋅ ) = Y i , ⋅ ( l ) T^{(l)}(\mathbf{Z}_{i,\cdot})=\mathbf{Y}_{i,\cdot}^{(l)} T ( l ) ( Z i , ⋅ ) = Y i , ⋅ ( l ) . We then introduce the concatenated transformation T : Δ K − 1 → ∏ l = 1 L S K − 1 T:\Delta^{K-1}\to\prod_{l=1}^{L}\mathbb{S}^{K-1} T : Δ K − 1 → ∏ l = 1 L S K − 1 defined by T ( z ) = ( T ( 1 ) ( z ) ⊤ , … , T ( L ) ( z ) ⊤ ) ⊤ T(\mathbf{z})=\left(T^{(1)}(\mathbf{z})^{\top},\ldots,T^{(L)}(\mathbf{z})^{\top}\right)^{\top} T ( z ) = ( T ( 1 ) ( z ) ⊤ , … , T ( L ) ( z ) ⊤ ) ⊤ . The transformation T T T maps membership vectors to concatenated row-normalized spectral embeddings by T ( Z i , ⋅ ) = ( ( Y i , ⋅ ( 1 ) ) ⊤ , … , ( Y i , ⋅ ( L ) ) ⊤ ) ⊤ = : Y i , ⋅ T(Z_{i,\cdot})=\left((Y_{i,\cdot}^{(1)})^{\top},\ldots,(Y_{i,\cdot}^{(L)})^{\top}\right)^{\top}=:Y_{i,\cdot} T ( Z i , ⋅ ) = ( ( Y i , ⋅ ( 1 ) ) ⊤ , … , ( Y i , ⋅ ( L ) ) ⊤ ) ⊤ =: Y i , ⋅ . The following theorem establishes bi-Lipschitz bounds for the concatenated population map T T T , showing that the concatenated row-normalized spectral embedding preserves distances on the membership simplex up to layer-dependent constants.
Theorem 3.1 (Bi-Lipschitz geometry of the concatenated row-normalized spectral embedding). For any z 1 , z 2 ∈ Δ K − 1 \mathbf{z}_{1},\mathbf{z}_{2}\in\Delta^{K-1} z 1 , z 2 ∈ Δ K − 1 , there exist constants c L i p , C L i p > 0 c_{\mathrm{Lip}},C_{\mathrm{Lip}}>0 c Lip , C Lip > 0 such that
c L i p ( ∑ l = 1 L ( σ B ( l ) ) 2 K 3 ) 1 / 2 ∥ z 2 − z 1 ∥ 2 ≤ ∥ T ( z 2 ) − T ( z 1 ) ∥ 2 ≤ C L i p K ( ∑ l = 1 L 1 σ B ( l ) ) 1 / 2 ∥ z 2 − z 1 ∥ 2 . c_{\mathrm{Lip}}\left(\sum_{l=1}^{L}\frac{(\sigma_{B}^{(l)})^{2}}{K^{3}}\right)^{1/2}\|\mathbf{z}_{2}-\mathbf{z}_{1}\|_{2}\leq\|T(\mathbf{z}_{2})-T(\mathbf{z}_{1})\|_{2}\leq C_{\mathrm{Lip}}K\left(\sum_{l=1}^{L}\frac{1}{\sigma_{B}^{(l)}}\right)^{1/2}\|\mathbf{z}_{2}-\mathbf{z}_{1}\|_{2}. c Lip ( l = 1 ∑ L K 3 ( σ B ( l ) ) 2 ) 1/2 ∥ z 2 − z 1 ∥ 2 ≤ ∥ T ( z 2 ) − T ( z 1 ) ∥ 2 ≤ C Lip K ( l = 1 ∑ L σ B ( l ) 1 ) 1/2 ∥ z 2 − z 1 ∥ 2 .
Consequently, since σ B ( l ) > 0 \sigma_{B}^{(l)}>0 σ B ( l ) > 0 for all 1 ≤ l ≤ L 1\leq l\leq L 1 ≤ l ≤ L , the map T T T is bi-Lipschitz on Δ K − 1 \Delta^{K-1} Δ K − 1 for each fixed finite L L L . Moreover, under Assumption 2 and fixed K K K , the rescaled map L − 1 / 2 T L^{-1/2}T L − 1/2 T is bi-Lipschitz with constants independent of L L L . We next state maximum perturbation bounds for both layer-wise and concatenated row-normalized spectral embeddings. This result extends the analysis of Agterberg et al. [ [1 ] from the ML-DCBM to the mixed membership setup.
Theorem 3.2 (Layer-wise and concatenated row-wise perturbation bounds). Suppose that Assumptions 1, 2, and 3, together with Condition 1, hold. Then there exist constants c E , c Y > 0 c_{E},c_{Y}>0 c E , c Y > 0 , independent of n n n , L L L , and l l l , such that, for each l ∈ [ L ] l\in[L] l ∈ [ L ] , the following bound holds with probability at least 1 − c E n − 18 1-c_{E}n^{-18} 1 − c E n − 18 :
h ^ ( l ) : = max 1 ≤ i ≤ n ∥ Y ~ i , ⋅ ( l ) − Y i , ⋅ ( l ) ∥ 2 ≤ c Y β n ( l ) . (7) \widehat{h}^{(l)}:=\max_{1\leq i\leq n}\left\|\widetilde{Y}_{i,\cdot}^{(l)}-Y_{i,\cdot}^{(l)}\right\|_{2}\leq c_{Y}\sqrt{\beta_{n}^{(l)}}. \tag*{(7)} h ( l ) := 1 ≤ i ≤ n max Y i , ⋅ ( l ) − Y i , ⋅ ( l ) 2 ≤ c Y β n ( l ) . (7) Consequently, by the union bound, with probability at least 1 − c E L n − 18 1-c_{E}Ln^{-18} 1 − c E L n − 18 ,
max 1 ≤ i ≤ n 1 L ∥ Y ~ i , ⋅ − Y i , ⋅ ∥ 2 ≤ ( c Y L ∑ l = 1 L β n ( l ) ) 1 / 2 . \max_{1\leq i\leq n}\frac{1}{\sqrt{L}}\left\|\widetilde{Y}_{i,\cdot}-Y_{i,\cdot}\right\|_{2}\leq\left(\frac{c_{Y}}{L}\sum_{l=1}^{L}\beta_{n}^{(l)}\right)^{1/2}. 1 ≤ i ≤ n max L 1 Y i , ⋅ − Y i , ⋅ 2 ≤ ( L c Y l = 1 ∑ L β n ( l ) ) 1/2 .
To illustrate the upper bound in the theorem, consider the homogeneous degree correction parameters θ i ( l ) ≲ ρ n \theta_{i}^{(l)}\lesssim\sqrt{\rho_{n}} θ i ( l ) ≲ ρ n . In this case, Assumption 1 and Assumption 3 require n k ≳ n / K n_{k}\gtrsim n/K n k ≳ n / K for all k ∈ [ K ] k\in[K] k ∈ [ K ] and ρ n ≳ log ( n ) n \rho_{n}\gtrsim\frac{\log(n)}{n} ρ n ≳ n l o g ( n ) , respectively. Moreover, Equation (7) implies that max i ∥ Y ~ i , ⋅ ( l ) − Y i , ⋅ ( l ) ∥ 2 ≲ β n ( l ) ≲ log n n ρ n \max_{i}\|\widetilde{Y}_{i,\cdot}^{(l)}-Y_{i,\cdot}^{(l)}\|_{2}\lesssim\sqrt{\beta_{n}^{(l)}}\lesssim\sqrt{\frac{\log n}{n\rho_{n}}} max i ∥ Y i , ⋅ ( l ) − Y i , ⋅ ( l ) ∥ 2 ≲ β n ( l ) ≲ n ρ n l o g n , which is also the rate of the scaled concatenated embedding.
Near-Pure Recovery via an Averaged Vertex-Hunting Objective
We now show that the nodes recovered by the corner estimation procedure are nearly-pure, and this corner-recovery error is controlled by the average layerwise embedding error h ˉ : = ∑ l = 1 L h ^ ( l ) / L \bar{h}:=\sum_{l=1}^{L}\widehat{h}^{(l)}/L h ˉ := ∑ l = 1 L h ( l ) / L , reflecting the averaging structure of the vertex-hunting objective defined in (6) . This result uses similar arguments as the vertex-hunting scheme of Jin et al. [ [26 ] , but the use of a different normalization strategy and the heterogeneity across layers require special care. In particular, for denoising we use k k k -medoids instead of k k k -means, which ensures the recovery of a point near the spherical simplex.
We first state a high-probability result on the recovery of pure nodes via corner estimation. Independence across layers allows us to obtain a high-probability bound on h ˉ \bar{h} h ˉ with a concentration margin that decreases as L L L increases. For a deterministic version of this result, see Appendix E. Let
r n , L ( x ) : = C V H ( 1 L ∑ l = 1 L β n ( l ) + n − 18 + x L ) r_{n,L}(x):=C_{\mathrm{VH}}\left(\frac{1}{L}\sum_{l=1}^{L}\sqrt{\beta_{n}^{(l)}}+n^{-18}+\frac{x}{L}\right) r n , L ( x ) := C VH ( L 1 l = 1 ∑ L β n ( l ) + n − 18 + L x )
denote a radius of the estimated corner from every pure node, where C V H C_{\mathrm{VH}} C VH is a constant independent of n n n and L L L . We state the corner estimation theorem that allows a general choice of x x x in the probability.
Theorem 3.3 (High-probability almost-pure recovery). Suppose that K K K is fixed, the layers are independent, and Assumptions 1, 2, 3 and Condition 1 hold. For any candidate set I ⊆ [ n ] \mathcal{I}\subseteq[n] I ⊆ [ n ] containing at least one pure node from each community, let J ^ ( I ) ∈ arg min J ⊆ I ∣ J ∣ = K Φ ~ L , I ( J ) \widehat{\mathcal{J}}(\mathcal{I})\in\mathop{\arg\min}_{\substack{\mathcal{J}\subseteq\mathcal{I}\\|\mathcal{J}|=K}}\widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J}) J ( I ) ∈ arg min J ⊆ I ∣ J ∣ = K Φ L , I ( J ) . Then for any x > 0 x>0 x > 0 , with probability at least 1 − e − x 1-e^{-x} 1 − e − x , every candidate set I ⊆ [ n ] \mathcal{I}\subseteq[n] I ⊆ [ n ] containing at least one pure node from each community satisfies min k ∈ [ K ] max j ∈ J ^ ( I ) Z j k ≥ 1 − r n , L ( x ) \min_{k\in[K]}\max_{j\in\widehat{\mathcal{J}}(\mathcal{I})}Z_{jk}\geq1-r_{n,L}(x) min k ∈ [ K ] max j ∈ J ( I ) Z j k ≥ 1 − r n , L ( x ) .
Since Theorem 3.3 applies to any corner candidate set containing at least one pure node from each community, it can be naturally applied to the corner-search step without denoising by taking I = [ n ] \mathcal{I}=[n] I = [ n ] under Condition 1. When the denoise step is performed, the same guarantee also applies with I = M ^ m \mathcal{I}=\widehat{\mathcal{M}}_m I = M m if M ^ m ∩ P k ≠ ∅ \widehat{\mathcal{M}}_m\cap\mathcal{P}_k\neq\varnothing M m ∩ P k = ∅ for all k ∈ [ K ] k\in[K] k ∈ [ K ] , where P k : = { i ∈ [ n ] : Z i , ⋅ = e k } \mathcal{P}_k:=\{i\in[n]:\mathbf{Z}_{i,\cdot}=\mathbf{e}_k\} P k := { i ∈ [ n ] : Z i , ⋅ = e k } denote the set of pure nodes in community k k k . We next introduce an additional conditions under which this property holds with high probability.
Condition 2. The following conditions hold.
(a) Pure node abundance. There exists a constant p p u r e > 0 p_{\mathrm{pure}}>0 p pure > 0 , independent of n n n and L L L , such that min k ∈ [ K ] ∣ P k ∣ n ≥ p p u r e \min_{k\in[K]}\frac{|\mathcal{P}_k|}{n}\geq p_{\mathrm{pure}} min k ∈ [ K ] n ∣ P k ∣ ≥ p pure .
(b) Mixed node separation. There exists a constant δ m i x ∈ ( 0 , 1 ) \delta_{\mathrm{mix}}\in(0,1) δ mix ∈ ( 0 , 1 ) , independent of n n n and L L L , such that for every mixed node i i i , min k ∈ [ K ] ∥ Z i , ⋅ − e k ∥ 2 ≥ δ m i x \min_{k\in[K]}\|\mathbf{Z}_{i,\cdot}-\mathbf{e}_k\|_2\geq\delta_{\mathrm{mix}} min k ∈ [ K ] ∥ Z i , ⋅ − e k ∥ 2 ≥ δ mix .
Condition 2 ensures that each community contains a nonvanishing fraction of pure nodes and that mixed nodes are separated from the pure vertices. We provide a sufficient assumption for the one-swap local minimizer m m m -medoids set M ^ m \widehat{\mathcal{M}}_m M m to contain a pure node from each community.
Proposition 3.4. Suppose Assumptions 1, 2, 3 and Condition 1, 2 are satisfied. Let M ^ m \widehat{\mathcal{M}}_m M m be a one-swap local minimizer of the m m m -medoids objective, where K ≤ m ≤ n K\leq m\leq n K ≤ m ≤ n . Define the population separation Δ 0 = 1 L min k ∈ [ K ] min i ∈ [ n ] ∖ P k ∥ T ( Z i , ⋅ ) − T ( e k ) ∥ 2 \Delta_0=\frac{1}{\sqrt{L}}\min_{k\in[K]}\min_{i\in[n]\setminus\mathcal{P}_k}\|T(\mathbf{Z}_{i,\cdot})-T(\mathbf{e}_k)\|_2 Δ 0 = L 1 min k ∈ [ K ] min i ∈ [ n ] ∖ P k ∥ T ( Z i , ⋅ ) − T ( e k ) ∥ 2 . There exists a constant C > 0 C>0 C > 0 , independent of n n n , L L L , m m m , x x x , such that, for any x > 0 x>0 x > 0 , if
4 C ( 1 L ∑ l = 1 L β n ( l ) + n − 1 / 8 + x L ) 1 / 2 + 2 m p p u r e < Δ 0 , 4C\left(\frac{1}{L}\sum_{l=1}^{L}\sqrt{\beta_n^{(l)}}+n^{-1/8}+\frac{x}{L}\right)^{1/2}+\frac{2}{mp_{\mathrm{pure}}}<\Delta_0, 4 C ( L 1 l = 1 ∑ L β n ( l ) + n − 1/8 + L x ) 1/2 + m p pure 2 < Δ 0 , then M ^ m ∩ P k ≠ ∅ \widehat{\mathcal{M}}_m\cap\mathcal{P}_k\neq\varnothing M m ∩ P k = ∅ for all k ∈ [ K ] k\in[K] k ∈ [ K ] with probability at least 1 − e − x 1-e^{-x} 1 − e − x .
Theorem 3.3 guarantees near-pure corner recovery for any candidate set containing at least one pure node from each community. Under the mixed-node separation condition, this guarantee can be strengthened to exact pure-node recovery when the recovery radius r n , L ( x ) r_{n,L}(x) r n , L ( x ) is sufficiently small.
Corollary 3.5 (Exact pure-node recovery). Suppose that the assumptions of Theorem 3.3 and Condition 2(b) hold. Fix x > 0 x > 0 x > 0 such that r n , L ( x ) < δ m i x 2 r_{n,L}(x) < \frac{\delta_{\mathrm{mix}}}{\sqrt{2}} r n , L ( x ) < 2 δ mix . Then, with probability at least 1 − e − x 1-e^{-x} 1 − e − x , J ^ ( I ) \widehat{\mathcal{J}}(\mathcal{I}) J ( I ) contains exactly one pure node from each community for any corner candidate set I \mathcal{I} I with I ∩ P k ≠ ∅ \mathcal{I} \cap\mathcal{P}_k \ne\varnothing I ∩ P k = ∅ for all k ∈ [ K ] k \in[K] k ∈ [ K ] .
In particular, the corollary applies directly to I = [ n ] \mathcal{I} = [n] I = [ n ] . It also applies to the m m m -medoid candidate set I = M ^ m \mathcal{I} = \widehat{\mathcal{M}}_{\mathrm{m}} I = M m if the separation condition in Proposition 3.4 holds for the same x x x , together with r n , L ( x ) < δ m i x / 2 r_{n,L}(x) < \delta_{\mathrm{mix}}/\sqrt{2} r n , L ( x ) < δ mix / 2 .
Rate of the Membership Estimator
Given the estimated corners of the L L L spherical simplices, we construct a membership estimator using the reference layer, as described in Algorithm 1. The following corollary establishes its estimation error bound.
Corollary 3.6 (Rate of the spectral membership estimator). Suppose the conditions of Theorem 3.3 hold. Let I ⊆ [ n ] \mathcal{I} \subseteq[n] I ⊆ [ n ] be any corner candidate set containing at least one pure node from each community. Let Z ^ 0 \widehat{Z}_0 Z 0 be the membership estimator obtained from Algorithm 1. Then there exist constants C > 0 C > 0 C > 0 , c x > 0 c_x > 0 c x > 0 and n 0 ≥ 1 n_0 \ge1 n 0 ≥ 1 such that, for every n ≥ n 0 n \ge n_0 n ≥ n 0 and 0 < x ≤ c x L 0 < x \le c_xL 0 < x ≤ c x L , with probability at least 1 − e − x − c E n − 18 1-e^{-x}-c_E n^{-18} 1 − e − x − c E n − 18 , there exists a permutation matrix Π ∈ Q K \Pi\in\mathcal{Q}_K Π ∈ Q K satisfying
1 n ∥ Z ^ 0 Π − Z ∥ F ≤ ∥ Z ^ 0 Π − Z ∥ 2 , ∞ ≤ C ( β n ( 1 ) + r n , L ( x ) ) . \frac{1}{\sqrt{n}}\|\widehat{Z}_0\Pi-Z\|_F \le\|\widehat{Z}_0\Pi-Z\|_{2,\infty} \le C\left(\sqrt{\beta_n^{(1)}}+r_{n,L}(x)\right). n 1 ∥ Z 0 Π − Z ∥ F ≤ ∥ Z 0 Π − Z ∥ 2 , ∞ ≤ C ( β n ( 1 ) + r n , L ( x ) ) .
Moreover, if Condition 2(b) holds with r n , L ( x ) < δ m i x / 2 r_{n,L}(x) < \delta_{\mathrm{mix}}/\sqrt{2} r n , L ( x ) < δ mix / 2 , then on the same event
1 n ∥ Z ^ 0 Π − Z ∥ F ≤ ∥ Z ^ 0 Π − Z ∥ 2 , ∞ ≤ C β n ( 1 ) . \frac{1}{\sqrt{n}}\|\widehat{Z}_0\Pi-Z\|_F \le\|\widehat{Z}_0\Pi-Z\|_{2,\infty} \le C\sqrt{\beta_n^{(1)}}. n 1 ∥ Z 0 Π − Z ∥ F ≤ ∥ Z 0 Π − Z ∥ 2 , ∞ ≤ C β n ( 1 ) .
The above corollary implies that consistency of the membership estimator is ensured when the reference-layer error β n ( 1 ) \beta_n^{(1)} β n ( 1 ) and the average layer-wise error rate 1 L ∑ l = 1 L β n ( l ) \frac{1}{L}\sum_{l=1}^{L}\sqrt{\beta_n^{(l)}} L 1 ∑ l = 1 L β n ( l ) both vanish, without requiring β n ( l ) → 0 \beta_n^{(l)} \to0 β n ( l ) → 0 for every non-reference layer.
Minimax lower bound for membership estimation
An important feature of the ML-DCMM is that additional layers need not always provide additional information for membership estimation. Under the normalization imposed by Condition 1, community-specific rescaling can alter the membership vectors of mixed nodes while its effect is absorbed by the layer-specific Θ ( l ) \Theta^{(l)} Θ ( l ) and B ( l ) B^{(l)} B ( l ) in non-reference layers. Consequently, there exist distinct membership matrices that induce identical edge-probability matrices P ( l ) P^{(l)} P ( l ) in all non-reference layers. Therefore, even as additional non-reference layers are observed, they provide no further information for distinguishing between such membership matrices. In such configurations, the distinguishability of the membership matrices, and hence a component of the membership estimation error, is determined entirely by the reference layer and cannot be improved by increasing L L L . This particular reparametrization phenomenon does not arise in the analogous non-overlapping membership model, since community-specific rescaling followed by row normalization leaves pure membership vectors unchanged. A concrete construction illustrating this bottleneck is provided in Appendix § 15 .§ 1 . The following theorem formalizes its statistical consequence.
Theorem 3.7 (Minimax lower bound for membership estimation). *Fix K ≥ 2 K \ge2 K ≥ 2 . Suppose that n ≥ n 0 ( K ) n \ge n_0(K) n ≥ n 0 ( K ) and ρ n ∈ ( 0 , 1 ] \rho_n \in(0,1] ρ n ∈ ( 0 , 1 ] satisfy n ρ n ≥ C ρ ( K ) log n n\rho_n \ge C_\rho(K)\log n n ρ n ≥ C ρ ( K ) log n , where n 0 ( K ) n_0(K) n 0 ( K ) and C ρ ( K ) > 0 C_\rho(K) > 0 C ρ ( K ) > 0 are sufficiently large constants. Let F n , K , L \mathcal{F}_{n,K,L} F n , K , L denote the parameter space of the ML-DCMM with n n n nodes, K K K communities
and L L L layers satisfying Assumptions 1, 2, 3 and Condition 1. Then there exists a universal constant c > 0 c>0 c > 0 , independent of n n n , K K K , L L L , and ρ n \rho_n ρ n , such that
inf Z ^ sup ϑ ∈ F n , K , L sup Π ∈ Q K E ϑ [ min Π ∈ Q K ∥ Z ^ Π − Z ∥ 2 , ∞ ] ≥ c n ρ n . (8) \inf_{\widehat{Z}}\sup_{\vartheta\in\mathcal{F}_{n,K,L}}\sup_{\Pi\in\mathcal{Q}_K}\mathbb{E}_{\vartheta}\left[\min_{\Pi\in\mathcal{Q}_K}\|\widehat{Z}\Pi-Z\|_{2,\infty}\right]\geq\frac{c}{\sqrt{n\rho_n}}. \tag*{(8)} Z inf ϑ ∈ F n , K , L sup Π ∈ Q K sup E ϑ [ Π ∈ Q K min ∥ Z Π − Z ∥ 2 , ∞ ] ≥ n ρ n c . (8)
Remark 3.2. Consider homogeneous degree-correction parameter setting in the reference layer θ i ( 1 ) ≳ ρ n \theta_i^{(1)}\gtrsim\sqrt{\rho_n} θ i ( 1 ) ≳ ρ n . When K K K is fixed, our minimax lower bound of ML-DCMM (3.2) has the same order as the minimax lower bound of DCMM in Ke and Wang [ [31 ] . Moreover, if r n , L ( x ) < δ m i x / 2 r_{n,L}(x)<\delta_{\mathrm{mix}}/\sqrt{2} r n , L ( x ) < δ mix / 2 , Corollary 3.6 gives min Π ∈ Q K ∥ Z ^ 0 Π − Z ∥ 2 , ∞ = O p ( log n n ρ n ) \min_{\Pi\in\mathcal{Q}_K}\|\widehat{Z}_0\Pi-Z\|_{2,\infty}=O_p\left(\sqrt{\frac{\log n}{n\rho_n}}\right) min Π ∈ Q K ∥ Z 0 Π − Z ∥ 2 , ∞ = O p ( n ρ n l o g n ) . Combining this upper bound with Theorem 3.7, the proposed spectral membership estimator is minimax optimal up to a factor of log n \sqrt{\log n} log n under the 2 → ∞ 2\to\infty 2 → ∞ loss.
These results also highlight the distinct roles of additional layers in our procedure. Increasing L L L can improve corner recovery through information aggregation across layers, whereas the subsequent membership estimation rate can remain bottlenecked by the reference layer once the corners are sufficiently well estimated.
Simulations
We conduct simulation studies to evaluate the proposed method. The first experiment studies the effect of the number of layers on the performance of the proposed methodology to validate our theoretical results. The second experiment compares the performance in estimating the node memberships across different data-generating settings and other alternative methodologies.
Experiment 1 (Corner recovery). We investigate whether observing more layers improves the recovery of spherical simplex corners. Fix ( n , n 0 , K , L max ) = ( 300 , 30 , 3 , 30 ) (n,n_0,K,L_{\max})=(300,30,3,30) ( n , n 0 , K , L m a x ) = ( 300 , 30 , 3 , 30 ) , where n 0 n_0 n 0 denotes the number of pure nodes per community. The remaining n − K n 0 n-Kn_0 n − K n 0 mixed nodes are independently generated from Dirichlet ( 1 K ) \operatorname{Dirichlet}(1_K) Dirichlet ( 1 K ) . We set Θ ( l ) = I n \Theta^{(l)}=I_n Θ ( l ) = I n for all l l l , and use a common block connectivity matrix B ( l ) = B B^{(l)}=B B ( l ) = B with B k k = 1 B_{kk}=1 B k k = 1 and B j k = 0.2 B_{jk}=0.2 B j k = 0.2 for j ≠ k j\ne k j = k for all layers. The probability matrices are scaled so that the expected average degree is 30. We generate L max = 30 L_{\max}=30 L m a x = 30 independent layers from this model.
We apply Algorithm 1 to a collection of layers and report the result of step 6 (corner estimation) using the first L ′ L' L ′ layers for each L ′ ∈ { 2 , 4 , 6 , 8 , 10 , 15 , 20 , 30 } L'\in\{2,4,6,8,10,15,20,30\} L ′ ∈ { 2 , 4 , 6 , 8 , 10 , 15 , 20 , 30 } . Let J ^ ( L ′ ) \widehat{\mathcal{J}}(L') J ( L ′ ) denote the estimated corner index set obtained from these L ′ L' L ′ layers. For each community k k k , define the recovered purity q k ( L ′ ) : = max j ∈ J ^ ( L ′ ) Z j k q_k^{(L')}:=\max_{j\in\widehat{\mathcal{J}}(L')}Z_{jk} q k ( L ′ ) := max j ∈ J ( L ′ ) Z j k . Observe that larger values of q k ( L ′ ) q_k^{(L')} q k ( L ′ ) indicate that the true membership of a node from community k k k is closer to a pure node with Z j k = 1 Z_{jk}=1 Z j k = 1 . We summarize corner recovery by the minimum purity score Q ( L ′ ) : = min 1 ≤ k ≤ K q k ( L ′ ) = min 1 ≤ k ≤ K max j ∈ J ^ ( L ′ ) Z j k Q(L'):=\min_{1\leq k\leq K}q_k^{(L')}=\min_{1\leq k\leq K}\max_{j\in\widehat{\mathcal{J}}(L')}Z_{jk} Q ( L ′ ) := min 1 ≤ k ≤ K q k ( L ′ ) = min 1 ≤ k ≤ K max j ∈ J ( L ′ ) Z j k , which is only equal to 1 when each community is represented by a pure node on the estimated set of corners. For r 0 ∈ { 0.05 , 0.1 } r_0\in\{0.05,0.1\} r 0 ∈ { 0.05 , 0.1 } , we also define the success indicator 1 [ Q ( L ′ ) ≥ 1 − r 0 ] \mathbf{1}\left[Q(L')\geq1-r_0\right] 1 [ Q ( L ′ ) ≥ 1 − r 0 ] . We report the empirical averages of Q ( L ′ ) Q(L') Q ( L ′ ) and the success indicators over 100 independent repetitions.
Figure 4.1 reports the corner recovery results. The left panel shows that the average minimum purity score Q ( L ′ ) Q(L') Q ( L ′ ) improves as the number of layers used for vertex hunting increases. The right panel shows that the success proportion also increases with L ′ L' L ′ for both r 0 = 0.05 r_0=0.05 r 0 = 0.05 and r 0 = 0.10 r_0=0.10 r 0 = 0.10 . These empirical trends are consistent with our theoretical results, suggesting that additional informative layers improve the reliability of corner recovery.
Figure 4.1. Vertex-hunting performance as the number of layers L ′ L' L ′ increases. Left: average minimum purity score Q ( L ′ ) Q(L') Q ( L ′ ) . Right: success proportion for r 0 = 0.05 r_0 = 0.05 r 0 = 0.05 and r 0 = 0.10 r_0 = 0.10 r 0 = 0.10 . Error bars indicate one standard error over 100 independent repetitions.
Experiment 2 (Membership recovery). We compare the proposed method with competing approaches across six simulation settings, formed by varying the layer-specific block connectivity matrices and the purity of mixed memberships. Fix ( n , n 0 , K , L ) = ( 300 , 30 , 3 , 10 ) (n,n_0,K,L)=(300,30,3,10) ( n , n 0 , K , L ) = ( 300 , 30 , 3 , 10 ) . The six settings are obtained by combining one of the following three block-connectivity regimes with one of the following two membership-purity regimes.
Different connectivity: the matrices B ( l ) \mathbf{B}^{(l)} B ( l ) vary across layers. Each B ( l ) \mathbf{B}^{(l)} B ( l ) has unit diagonal entries B k k ( l ) = 1 B_{kk}^{(l)} = 1 B k k ( l ) = 1 ; for j < k j < k j < k , the off-diagonal entries are generated independently as B j k ( l ) ∼ U ( 0.1 , 0.3 ) B_{jk}^{(l)} \sim U(0.1, 0.3) B j k ( l ) ∼ U ( 0.1 , 0.3 ) , and we set B k j ( l ) = B j k ( l ) B_{kj}^{(l)} = B_{jk}^{(l)} B k j ( l ) = B j k ( l ) .
Alternating connectivity: odd layers have an assortative community structure, with B k k ( l ) = 1 B_{kk}^{(l)} = 1 B k k ( l ) = 1 and B j k ( l ) = 0.2 B_{jk}^{(l)} = 0.2 B j k ( l ) = 0.2 for j ≠ k j \ne k j = k , whereas even layers have a disassortative structure, with B k k ( l ) = 0.2 B_{kk}^{(l)} = 0.2 B k k ( l ) = 0.2 and B j k ( l ) = 1 B_{jk}^{(l)} = 1 B j k ( l ) = 1 for j ≠ k j \ne k j = k .
For the membership vectors, the remaining n − K n 0 n-Kn_0 n − K n 0 mixed nodes are evenly split among the following six membership patterns supported on pairs of communities:
( 1 − r , r , 0 ) , ( r , 1 − r , 0 ) , ( 1 − r , 0 , r ) , ( r , 0 , 1 − r ) , ( 0 , 1 − r , r ) , ( 0 , r , 1 − r ) . (1-r,r,0),\quad(r,1-r,0),\quad(1-r,0,r),\quad(r,0,1-r),\quad(0,1-r,r),\quad(0,r,1-r). ( 1 − r , r , 0 ) , ( r , 1 − r , 0 ) , ( 1 − r , 0 , r ) , ( r , 0 , 1 − r ) , ( 0 , 1 − r , r ) , ( 0 , r , 1 − r ) .
We consider two values of r r r : r = 0.2 r = 0.2 r = 0.2 , corresponding to higher-purity mixed memberships, and r = 0.4 r = 0.4 r = 0.4 , corresponding to lower-purity mixed memberships. We use the same degree-correction parameters across layers Θ ( l ) = Θ = diag ( θ 1 , … , θ n ) \boldsymbol{\Theta}^{(l)} = \boldsymbol{\Theta} = \operatorname{diag}(\theta_1,\ldots,\theta_n) Θ ( l ) = Θ = diag ( θ 1 , … , θ n ) , 1 ≤ l ≤ L 1 \le l \le L 1 ≤ l ≤ L , where 1 / θ i ∼ i i d U ( 1 , 3 ) 1/\theta_i \mathrel{\overset{\mathrm{iid}}{\sim}} U(1,3) 1/ θ i ∼ iid U ( 1 , 3 ) . For each target density ρ ∈ { 0.04 , 0.06 , 0.08 , 0.10 , 0.12 , 0.14 , 0.16 } \rho\in\{0.04, 0.06, 0.08, 0.10, 0.12, 0.14, 0.16\} ρ ∈ { 0.04 , 0.06 , 0.08 , 0.10 , 0.12 , 0.14 , 0.16 } , the probability matrices are scaled so that the expected average degree is ( n − 1 ) ρ (n-1)\rho ( n − 1 ) ρ .
We compare the proposed method with three types of baselines. First, we include Mixed-SCORE (Avg), an aggregation-based baseline that applies the single-layer Mixed-SCORE method of Jin et al. [ [26 ] to the average adjacency matrix L − 1 ∑ l = 1 L A ( l ) L^{-1}\sum_{l=1}^{L} A^{(l)} L − 1 ∑ l = 1 L A ( l ) . This baseline represents a natural way to extend a single-layer mixed-membership method to multilayer data by network aggregation. Second, we include DC-MASE of Agterberg et al. [ [1 ] and Bias-adjusted SoS of Lei and Lin [ [34 ] , two multilayer methods modeling non-overlapping community structure. This comparison separates two sources of improvement: the benefit of using multilayer information and the benefit of explicitly modeling mixed memberships. For each method, we report the membership estimation error Err ( Z ^ , Z ) : = min Q ∈ P K 1 n ∥ Z − Z ^ Q ∥ F 2 \operatorname{Err}(\widehat{\mathbf{Z}}, \mathbf{Z}) := \min_{\mathbf{Q}\in\mathcal{P}_K} \frac{1}{n}\lVert\mathbf{Z} - \widehat{\mathbf{Z}}\mathbf{Q}\rVert_F^2 Err ( Z , Z ) := min Q ∈ P K n 1 ∥ Z − Z Q ∥ F 2 averaged over 50 independent repetitions.
Figure 4.2 shows the membership estimation errors across the six simulation settings. The relative performance depends strongly on both the layer-specific block connectivity structure and the purity of the membership vectors. In the Same B \mathbf{B} B and Different B \mathbf{B} B settings, Mixed-SCORE (Avg) performs very well, and in several cases it outperforms CNASE. This behavior is expected because the layers share a common assortative structure, or have block connectivity matrices that are similar in expectation, so averaging the adjacency matrices preserves most of the membership signal. The picture changes substantially in the Alternating B setting. In this case, the assortative and disassortative layer-specific connectivity patterns can cancel each other when the adjacency matrices are averaged, leading to poor performance of Mixed-SCORE (Avg). By contrast, CNASE preserves the layer-specific connectivity information, and its error decreases rapidly as the target density ρ \rho ρ increases. This illustrates the advantage of using the multilayer structure when the connectivity patterns differ substantially across layers.
Figure 4.2. Membership estimation error across six simulation settings. Columns correspond to the three block-connectivity regimes, and rows correspond to the two membership-purity regimes. Each curve reports the average membership error over 50 independent repetitions as the target density ρ \rho ρ varies. Lower values indicate better membership recovery.
The comparison with the multilayer hard-clustering methods, DCMASE and Bias-adjusted SoS, highlights a different aspect. These methods can use information across layers, but they target hard community assignments rather than mixed memberships. As a result, they are competitive in some higher-purity settings, where most nodes are close to pure, but their errors remain bounded away from zero in the lower-purity settings. In contrast, CNASE explicitly models mixed memberships and therefore achieves substantially smaller errors in the lower-purity regimes once the networks are sufficiently dense. Overall, these results suggest that CNASE is particularly beneficial when multilayer networks exhibit substantial layer-to-layer variation in connectivity patterns together with non-negligible mixed membership structure.
Data analysis
We analyze London bicycle-sharing trip data originally provided by Transport for London (TfL) and preprocessed and distributed through the bikecommclust GitHub repository by Munoz-Mendez et al. [ [43 ] . The data contain 1,469,851 bike trips from June 22, 2014 to July 31, 2014 among n = 750 n = 750 n = 750 bike stations.
We construct four layers by crossing day type (weekday versus weekend) with time period (AM versus PM), where AM and PM refer to trips starting before and after noon, respectively. Throughout this analysis, the layers are ordered as weekday AM, weekday PM, weekend AM, and weekend PM for l = 1 , 2 , 3 , 4 l = 1, 2, 3, 4 l = 1 , 2 , 3 , 4 , respectively. For each layer, we first symmetrize the directed trip counts by summing trips in the two directions between each pair of stations, and then divide by the number of days in the corresponding day-type category. Let W ( l ) ∈ R 750 × 750 \mathbf{W}^{(l)} \in\mathbb{R}^{750 \times750} W ( l ) ∈ R 750 × 750 denote the resulting symmetric weighted adjacency matrix, where W i j ( l ) W_{ij}^{(l)} W ij ( l ) is the average number of trips per day between stations i i i and j j j in layer l ∈ { 1 , 2 , 3 , 4 } l \in\{1,2,3,4\} l ∈ { 1 , 2 , 3 , 4 } . We then threshold these weighted matrices to obtain binary symmetric adjacency matrices A ( 1 ) , … , A ( 4 ) A^{(1)}, \ldots, A^{(4)} A ( 1 ) , … , A ( 4 ) , defined by A i j ( l ) = 1 { W i j ( l ) ≥ 0.1 } A_{ij}^{(l)} = \mathbf{1}\{W_{ij}^{(l)} \ge0.1\} A ij ( l ) = 1 { W ij ( l ) ≥ 0.1 } for every i < j i < j i < j with A i i ( l ) = 0 A_{ii}^{(l)} = 0 A ii ( l ) = 0 . Summary statistics for the four thresholded layers are reported in Table 5.1 . The layer densities vary substantially, suggesting non-negligible heterogeneity across the four temporal layers.
Table 5.1. Summary statistics for the four thresholded layers of the London bike-sharing network.
To evaluate the model and the proposed ML-DCMM-based membership estimator, we compare our methodology with other baseline models and membership estimation methods. The competing models include DCBM, DCMM and ML-DCBM, while the estimation methods include SCORE [24 ] , Mixed-SCORE [26 ] and DC-MASE [1 ] . For the ML-DCMM, we include Mixed-SCORE (Avg), introduced in § 4 , to compare estimation performance of the proposed CNASE under the same model. Note that these baselines include both overlapping and non-overlapping communities, as well as same or different connectivity parameters across layers. See Table H.1 in Appendix § 16 .§ 1 for a summary of the compared models and methods. Following the node-pair cross-validation scheme of Li et al. [ [37 ] , we use a 10-fold cross-validation procedure adapted to multilayer networks. See Appendix § 16 .§ 2 for details. Table 5.2 summarizes the 10-fold cross-validation errors.
Table 5.2. 10-fold node-pair cross-validation errors for the London bike-sharing network. Entries are mean CV errors across 10 folds, with empirical standard errors in parentheses. For each K K K , the smallest mean CV error is highlighted in bold. Each column corresponds to a different model, with the method to fit the communities indicated in parentheses.
The proposed CNASE method achieves the smallest error for every K ∈ { 4 , 5 , 6 , 7 , 8 } K \in\{4,5,6,7,8\} K ∈ { 4 , 5 , 6 , 7 , 8 } , with the minimum attained at K = 7 K = 7 K = 7 . The comparison of cross-validation errors under the DCMM and ML-DCMM using the same Mixed-SCORE method is informative. Both procedures estimate the shared membership matrix from the aggregated network, but the ML-DCMM additionally refits the layer-specific degree-correction parameters and block connectivity matrices. Across all values of K K K , this refitting step substantially reduces the cross-validation error relative to the DCMM. This suggests that layer-specific degree effects and connectivity patterns play an important role in explaining the bike-sharing networks. At the same time, Mixed-SCORE still has larger cross-validation errors than CNASE for every K K K . Since both procedures allow layer-specific Θ ( l ) \Theta^{(l)} Θ ( l ) and B ( l ) B^{(l)} B ( l ) in the reconstruction step, this remaining gap points to the estimation of the shared membership structure. In particular, estimating the membership matrix from the concatenated multilayer embeddings appears to yield a more predictive shared community representation than estimating it from the averaged adjacency matrix alone. Figure 5.1 shows dominant estimated community structure by our method assuming $K = 7 communities. The map suggests a spatially coherent community structure, in which stations assigned to the same dominant community are geographically clustered. The station coordinates are not used in fitting the model, so this spatial coherence provides an external validation of the estimated community structure. Stations located near transition regions between neighboring estimated communities are expected to have more ambiguous community assignments. Consistent with this interpretation, many stations with maximum estimated membership at most 0.6 appear near such transition regions. In contrast to hard-clustering models such as ML-DCBM, which assign each station to a single community, ML-DCMM allows these boundary stations to retain non-negligible memberships in multiple communities. This provides a more flexible representation of the spatial community structure.
Figure 5.1. Estimated community structure of the London bike-sharing stations. Stations are colored by their dominant estimated community, argmax k Z ^ i k \operatorname{argmax}_k \widehat{Z}_{ik} argmax k Z ik . Black outlines indicate stations with maximum membership at most 0.6, corresponding to relatively mixed-membership stations.
Discussion
In this work, we studied the multilayer degree-corrected mixed-membership model. We established identifiability conditions and a minimax lower bound for estimating the membership matrix under the ML-DCMM. We developed a spectral method for estimating the shared membership matrix, and we established a connection with the ParaTuck-2 tensor decomposition. Unlike approaches that collapse the multilayer structure into a single averaged network, the proposed method jointly uses information across layers while retaining layer-specific variation. Accurate corner estimation is crucial for membership estimation, and our theory shows that incorporating more layers can improve corner recovery. Simulation studies show that the proposed method performs well, and it is particularly suitable when the layers are heterogeneous. The bike-sharing network analysis further demonstrates that the method can produce meaningful mixed-membership estimates and improve predictive performance in real multilayer network data, suggesting that the ML-DCMM model equipped with CNASE for membership estimation provides a better fit to the real data.
One limitation is that our theoretical analysis assumes the number of communities K K K is known, a common simplifying assumption in the community detection literature. In practice, however, K K K is typically unknown and data-driven estimation requires additional substantial work. Extension of existing methods under several single-layer block models can address this limitation [25 , 21 , 16 ] . A second limitation is algorithmic. The ParaTuck-2 refinement step leads to a structured nonconvex optimization problem, and our implementation uses an alternating least-squares procedure because it is easy to implement and performs well empirically. Nevertheless, a more systematic study of initialization, optimization strategies, and convergence behavior for this constrained tensor decomposition would strengthen the practical reliability of the method.
Future work could further extend the proposed framework in several directions. A particularly interesting direction is to develop methods that combine complementary partial signals across layers, so that the full community structure can be recovered even when individual layers are insufficient. It would also be useful to study dependent multilayer networks, including temporal networks, and to extend the framework to directed or weighted multilayer networks.
Acknowledgment
Jesús Arroyo acknowledges support from the National Science Foundation under grant DMS-2413553. The authors also acknowledge that portions of this research were conducted with the advanced computing resources provided by Texas A&M Department of Statistics Arseven Computing Cluster.
Declaration of generative AI use
The authors used ChatGPT 5.6 to partially assist with language editing, constructing examples in Theorem 3.7 and improving computational efficiency of the code implementation. All AI-assisted outputs were reviewed by the authors, who take full responsibility for the final manuscript.
Algorithmic details
In this section, we provide details of the proposed estimation procedure, including choosing the number of medoids in the corner search step and random initialization in the refinement step.
Choosing the number of medoids
The number of medoids is selected in a data-driven manner from the candidate set M m e d : = { 2 K , 2 K + 1 , … , 3 K } \mathcal{M}_{\mathrm{med}} := \{2K,2K+1,\ldots,3K\} M med := { 2 K , 2 K + 1 , … , 3 K } . For each m ∈ M m e d m \in\mathcal{M}_{\mathrm{med}} m ∈ M med , we run the vertex-hunting procedure with m m m medoids and obtain a candidate index set J ^ m \widehat{\mathcal{J}}_m J m . We then compute the corresponding objective value Φ ~ L , M ^ m ( J ^ m ) \widetilde{\Phi}_{L,\widehat{\mathcal{M}}_m}(\widehat{\mathcal{J}}_m) Φ L , M m ( J m ) . Let m ∗ ∈ arg min m ∈ M m e d Φ ~ L , M ^ m ( J ^ m ) m^* \in\arg\min_{m\in\mathcal{M}_{\mathrm{med}}}\widetilde{\Phi}_{L,\widehat{\mathcal{M}}_m}(\widehat{\mathcal{J}}_m) m ∗ ∈ arg min m ∈ M med Φ L , M m ( J m ) . Rather than selecting m ∗ m^* m ∗ directly, we allow a small tolerance around the minimum objective value. Specifically, for a fixed tolerance parameter τ > 0 \tau> 0 τ > 0 , we define
M τ : = { m ∈ M m e d : Φ ~ L , M ^ m ( J ^ m ) ≤ ( 1 + τ ) Φ ~ L , M ^ m ∗ ( J ^ m ∗ ) } . \mathcal{M}_{\tau} := \left\{m \in\mathcal{M}_{\mathrm{med}} : \widetilde{\Phi}_{L,\widehat{\mathcal{M}}_m}(\widehat{\mathcal{J}}_m) \le(1+\tau)\widetilde{\Phi}_{L,\widehat{\mathcal{M}}_{m^*}}(\widehat{\mathcal{J}}_{m^*})\right\}. M τ := { m ∈ M med : Φ L , M m ( J m ) ≤ ( 1 + τ ) Φ L , M m ∗ ( J m ∗ ) } .
Among these near-optimal choices, we select the value that gives the best-conditioned estimated corner matrices:
m ^ ∈ arg max m ∈ M τ min 1 ≤ l ≤ L σ K ( S ^ m ( l ) ) , \widehat{m} \in\arg\max_{m\in\mathcal{M}_{\tau}}\min_{1\le l\le L}\sigma_K\left(\widehat{S}^{(l)}_m\right), m ∈ arg m ∈ M τ max 1 ≤ l ≤ L min σ K ( S m ( l ) ) ,
where S ^ m ( l ) = Y ^ J ^ m , ⋅ ( l ) \widehat{S}^{(l)}_m=\widehat{Y}^{(l)}_{\widehat{\mathcal{J}}_m,\cdot} S m ( l ) = Y J m , ⋅ ( l ) is the corner matrix in layer l l l formed by the K K K medoids selected from the m m m candidates. In our numerical studies, we set τ = 0.1 \tau=0.1 τ = 0.1 . This aims to obtain numerically stable inverse matrix ( S ^ m ( l ) ) − 1 \left(\widehat{S}^{(l)}_m\right)^{-1} ( S m ( l ) ) − 1 among possible candidates in M τ \mathcal{M}_{\tau} M τ .
If max m ∈ M m e d ( m K ) = ( 3 K K ) ≤ 5000 \max_{m\in\mathcal{M}_{\mathrm{med}}}\binom{m}{K}=\binom{3K}{K}\le5000 max m ∈ M med ( K m ) = ( K 3 K ) ≤ 5000 , we enumerate all K K K -subsets and obtain an exact minimizer of Φ ~ L , M ^ m \widetilde{\Phi}_{L,\widehat{\mathcal{M}}_m} Φ L , M m for every m ∈ M m e d m\in\mathcal{M}_{\mathrm{med}} m ∈ M med . Otherwise, we apply the following same greedy search procedure for every candidate m m m , followed by a local-swap refinement. Our greedy search is inspired by the successive projection algorithm [5 ] . For each m ∈ M m e d m\in\mathcal{M}_{\mathrm{med}} m ∈ M med ,
Choose two farthest medoids and let i ^ 1 , i ^ 2 \widehat{i}_1,\widehat{i}_2 i 1 , i 2 denote the index of the medoids.
For each 3 ≤ b < K 3\le b<K 3 ≤ b < K , let I ^ b − 1 = { i ^ 1 , … , i ^ b − 1 } \widehat{\mathcal{I}}_{b-1}=\{\widehat{i}_1,\ldots,\widehat{i}_{b-1}\} I b − 1 = { i 1 , … , i b − 1 } denote the selected b − 1 b-1 b − 1 medoids index set. Let P I ^ b − 1 ( l ) P^{(l)}_{\widehat{\mathcal{I}}_{b-1}} P I b − 1 ( l ) denote the orthogonal projection matrix onto the linear span of the medoids indexed by I ^ b − 1 \widehat{\mathcal{I}}_{b-1} I b − 1 in layer l l l . Select
i ^ b ∈ arg max j ∈ M m e d ∖ I ^ b − 1 1 L ∑ l = 1 L ∥ ( I − P I ^ b − 1 ( l ) ) y j ( l ) ∥ 2 2 , \widehat{i}_b \in\arg\max_{j\in\mathcal{M}_{\mathrm{med}}\setminus\widehat{\mathcal{I}}_{b-1}}\frac{1}{L}\sum_{l=1}^{L}\left\|\left(I-P^{(l)}_{\widehat{\mathcal{I}}_{b-1}}\right)y_j^{(l)}\right\|_2^2, i b ∈ arg j ∈ M med ∖ I b − 1 max L 1 l = 1 ∑ L ( I − P I b − 1 ( l ) ) y j ( l ) 2 2 ,
and set I ^ b = I ^ b − 1 ∪ { i ^ b } \widehat{\mathcal{I}}_b=\widehat{\mathcal{I}}_{b-1}\cup\{\widehat{i}_b\} I b = I b − 1 ∪ { i b } .
For b = K b=K b = K , select
i ^ K ∈ arg min j ∈ M m e d ∖ I ^ K − 1 Φ ~ L , M m e d ( I ^ K − 1 ∪ { j } ) , \widehat{i}_K \in\arg\min_{j\in\mathcal{M}_{\mathrm{med}}\setminus\widehat{\mathcal{I}}_{K-1}}\widetilde{\Phi}_{L,\mathcal{M}_{\mathrm{med}}}\left(\widehat{\mathcal{I}}_{K-1}\cup\{j\}\right), i K ∈ arg j ∈ M med ∖ I K − 1 min Φ L , M med ( I K − 1 ∪ { j } ) ,
and set I ^ K = I ^ K − 1 ∪ { i ^ K } \widehat{\mathcal{I}}_K=\widehat{\mathcal{I}}_{K-1}\cup\{\widehat{i}_K\} I K = I K − 1 ∪ { i K } . That is, we select i ^ K \widehat{i}_K i K that minimize the averaged vertex-hunting objective defined in (6) .
Refine the selected medoid set I ^ K \widehat{\mathcal{I}}_K I K to one-swap local optimality.
The resulting solution need not be a global minimizer of Φ ~ L , M ^ m \widetilde{\Phi}_{L,\widehat{\mathcal{M}}_m} Φ L , M m ; consequently, our current recovery theory, which is stated for global minimizers of the corner-search objective, does not directly apply to this scalable implementation.
Estimation refinement via a tensor decomposition
Following the spectral membership estimation, the second step refines Z ^ 0 \widehat{Z}_0 Z 0 through the ParaTuck-2 representation. We estimate the ParaTuck-2 factors from { U ^ ( l ) } l = 1 L \{\widehat{U}^{(l)}\}_{l=1}^{L} { U ( l ) } l = 1 L . The ParaTuck-2 decomposition may be computed by different numerical or algebraic algorithms; throughout our experiments, we use alternating least squares (ALS). The refinement step is summarized in Algorithm 2.
Algorithm 2 ParaTuck-2 refinement
Input: List of matrices { U ^ ( 1 ) , … , U ^ ( L ) } \{\widehat{\mathbf{U}}^{(1)},\ldots,\widehat{\mathbf{U}}^{(L)}\} { U ( 1 ) , … , U ( L ) } , number of clusters K K K , spectral membership estimates Z ^ 0 \widehat{\mathbf{Z}}_0 Z 0 , maximum number of iterations T max T_{\max} T m a x , convergence criteria ϵ \epsilon ϵ
Ensure: Estimated membership matrix Z ^ \widehat{\mathbf{Z}} Z
Obtain random initialization { Θ ^ 0 ( 1 ) , … , Θ ^ 0 ( L ) } \{\widehat{\boldsymbol{\Theta}}_0^{(1)},\ldots,\widehat{\boldsymbol{\Theta}}_0^{(L)}\} { Θ 0 ( 1 ) , … , Θ 0 ( L ) } and { M ^ 0 ( 1 ) , … , M ^ 0 ( L ) } \{\widehat{\mathbf{M}}_0^{(1)},\ldots,\widehat{\mathbf{M}}_0^{(L)}\} { M 0 ( 1 ) , … , M 0 ( L ) } by ( Θ ^ 0 ( l ) ) i i ∼ U ( 0.5 , 1.5 ) (\widehat{\boldsymbol{\Theta}}_0^{(l)})_{ii}\sim U(0.5,1.5) ( Θ 0 ( l ) ) ii ∼ U ( 0.5 , 1.5 ) and ( M ^ 0 ) i i ∼ U ( 0.1 , 0.5 ) (\widehat{\mathbf{M}}_0)_{ii}\sim U(0.1,0.5) ( M 0 ) ii ∼ U ( 0.1 , 0.5 ) .
for t = 1 t=1 t = 1 to T max T_{\max} T m a x do
Perform one ALS iteration to update { Θ ^ ( l ) } l = 1 L \{\widehat{\boldsymbol{\Theta}}^{(l)}\}_{l=1}^{L} { Θ ( l ) } l = 1 L , Z ^ \widehat{\mathbf{Z}} Z , and { M ^ ( l ) } l = 1 L \{\widehat{\mathbf{M}}^{(l)}\}_{l=1}^{L} { M ( l ) } l = 1 L .
if the convergence criterion is satisfied then
break
end if
end for
Spectral Embedding Error Bounds
Let f : R d ∖ { 0 } ↦ R d f:\mathbb{R}^{d}\setminus\{0\}\mapsto\mathbb{R}^{d} f : R d ∖ { 0 } ↦ R d be the ℓ 2 \ell_2 ℓ 2 normalization function defined as:
f ( x ) = x ∥ x ∥ 2 . f(\mathbf{x})=\frac{\mathbf{x}}{\|\mathbf{x}\|_2}. f ( x ) = ∥ x ∥ 2 x .
In this section, we derive row-wise error bounds for the adjacency spectral embedding X ^ i , : ( l ) \widehat{\mathbf{X}}^{(l)}_{i,:} X i , : ( l ) relative to its population counterpart X i , : ( l ) \mathbf{X}^{(l)}_{i,:} X i , : ( l ) . Using the elementary bound,
∥ f ( x ) − f ( y ) ∥ ≤ 2 ∥ x − y ∥ ∥ y ∥ . \|f(\mathbf{x})-f(\mathbf{y})\|\leq\frac{2\|\mathbf{x}-\mathbf{y}\|}{\|\mathbf{y}\|}. ∥ f ( x ) − f ( y ) ∥ ≤ ∥ y ∥ 2∥ x − y ∥ .
we further derive an upper bound for ∥ ( Y ^ ( l ) W ∗ ( l ) ) i , : − Y i , : ( l ) ∥ \|(\widehat{\mathbf{Y}}^{(l)}\mathbf{W}^{(l)}_*)_{i,:}-\mathbf{Y}^{(l)}_{i,:}\| ∥ ( Y ( l ) W ∗ ( l ) ) i , : − Y i , : ( l ) ∥ . To this end, we use the following decomposition from Agterberg et al. [ [1 ] :
X ^ ( l ) W ∗ ( l ) − X ( l ) = ( A ( l ) − E A ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) + R X ( l ) , (9) \widehat{\mathbf{X}}^{(l)}\mathbf{W}^{(l)}_*-\mathbf{X}^{(l)}=(\mathbf{A}^{(l)}-\mathbb{E}\mathbf{A}^{(l)})\mathbf{V}^{(l)}\left|\boldsymbol{\Lambda}^{(l)}\right|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}+\mathbf{R}^{(l)}_{\mathbf{X}}, \tag*{(9)} X ( l ) W ∗ ( l ) − X ( l ) = ( A ( l ) − E A ( l ) ) V ( l ) Λ ( l ) − 2 1 I p , q ( l ) + R X ( l ) , (9)
where the residual term R X ( l ) \mathbf{R}^{(l)}_{\mathbf{X}} R X ( l ) consists of the following 6 terms:
R X ( l ) = R 1 ( l ) W ∗ ( l ) + R 2 ( l ) W ∗ ( l ) + R 3 ( l ) W ∗ ( l ) + R 4 ( l ) + R 5 ( l ) + R 6 ( l ) , \mathbf{R}^{(l)}_{\mathbf{X}}=\mathbf{R}^{(l)}_1\mathbf{W}^{(l)}_*+\mathbf{R}^{(l)}_2\mathbf{W}^{(l)}_*+\mathbf{R}^{(l)}_3\mathbf{W}^{(l)}_*+\mathbf{R}^{(l)}_4+\mathbf{R}^{(l)}_5+\mathbf{R}^{(l)}_6, R X ( l ) = R 1 ( l ) W ∗ ( l ) + R 2 ( l ) W ∗ ( l ) + R 3 ( l ) W ∗ ( l ) + R 4 ( l ) + R 5 ( l ) + R 6 ( l ) ,
R 1 ( l ) = − V ( l ) ( ( V ( l ) ) T ( A ( l ) − E A ( l ) ) V ^ ( l ) ) ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ; \mathbf{R}^{(l)}_1=-\mathbf{V}^{(l)}((\mathbf{V}^{(l)})^\mathsf{T}(\mathbf{A}^{(l)}-\mathbb{E}\mathbf{A}^{(l)})\widehat{\mathbf{V}}^{(l)})|\widehat{\boldsymbol{\Lambda}}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}; R 1 ( l ) = − V ( l ) (( V ( l ) ) T ( A ( l ) − E A ( l ) ) V ( l ) ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ;
R 2 ( l ) = V ( l ) ( ( V ( l ) ) T V ^ ( l ) ∣ Λ ^ ( l ) ∣ 1 2 − ∣ Λ ( l ) ∣ 1 2 ( V ( l ) ) T V ^ ( l ) ) ; \mathbf{R}^{(l)}_2=\mathbf{V}^{(l)}\left((\mathbf{V}^{(l)})^\mathsf{T}\widehat{\mathbf{V}}^{(l)}|\widehat{\boldsymbol{\Lambda}}^{(l)}|^{\frac{1}{2}}-|\boldsymbol{\Lambda}^{(l)}|^{\frac{1}{2}}(\mathbf{V}^{(l)})^\mathsf{T}\widehat{\mathbf{V}}^{(l)}\right); R 2 ( l ) = V ( l ) ( ( V ( l ) ) T V ( l ) ∣ Λ ( l ) ∣ 2 1 − ∣ Λ ( l ) ∣ 2 1 ( V ( l ) ) T V ( l ) ) ;
R 3 ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1 2 ( ( V ( l ) ) T V ^ ( l ) − ( W ∗ ( l ) ) T ) ; \mathbf{R}^{(l)}_3=\mathbf{V}^{(l)}|\boldsymbol{\Lambda}^{(l)}|^{\frac{1}{2}}\left((\mathbf{V}^{(l)})^\mathsf{T}\widehat{\mathbf{V}}^{(l)}-(\mathbf{W}^{(l)}_*)^\mathsf{T}\right); R 3 ( l ) = V ( l ) ∣ Λ ( l ) ∣ 2 1 ( ( V ( l ) ) T V ( l ) − ( W ∗ ( l ) ) T ) ;
R 4 ( l ) = ( A ( l ) − E A ( l ) ) ( V ^ ( l ) ( V ^ ( l ) ) T V ( l ) − V ( l ) ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) ; \mathbf{R}^{(l)}_4=(\mathbf{A}^{(l)}-\mathbb{E}\mathbf{A}^{(l)})(\widehat{\mathbf{V}}^{(l)}(\widehat{\mathbf{V}}^{(l)})^\mathsf{T}\mathbf{V}^{(l)}-\mathbf{V}^{(l)})|\boldsymbol{\Lambda}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}; R 4 ( l ) = ( A ( l ) − E A ( l ) ) ( V ( l ) ( V ( l ) ) T V ( l ) − V ( l ) ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ;
R 5 ( l ) = − ( A ( l ) − E A ( l ) ) V ^ ( l ) ( ( V ^ ( l ) ) T V ( l ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) − ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ( V ^ ( l ) ) T V ( l ) ) ; \mathbf{R}^{(l)}_5=-(\mathbf{A}^{(l)}-\mathbb{E}\mathbf{A}^{(l)})\widehat{\mathbf{V}}^{(l)}\left((\widehat{\mathbf{V}}^{(l)})^\mathsf{T}\mathbf{V}^{(l)}|\boldsymbol{\Lambda}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}-|\widehat{\boldsymbol{\Lambda}}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}(\widehat{\mathbf{V}}^{(l)})^\mathsf{T}\mathbf{V}^{(l)}\right); R 5 ( l ) = − ( A ( l ) − E A ( l ) ) V ( l ) ( ( V ( l ) ) T V ( l ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) − ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ( V ( l ) ) T V ( l ) ) ;
R 6 ( l ) = ( A ( l ) − E A ( l ) ) V ^ ( l ) ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ( W ∗ ( l ) − ( V ^ ( l ) ) T V ( l ) ) . \mathbf{R}^{(l)}_6=(\mathbf{A}^{(l)}-\mathbb{E}\mathbf{A}^{(l)})\widehat{\mathbf{V}}^{(l)}|\widehat{\boldsymbol{\Lambda}}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}\left(\mathbf{W}^{(l)}_*-(\widehat{\mathbf{V}}^{(l)})^\mathsf{T}\mathbf{V}^{(l)}\right). R 6 ( l ) = ( A ( l ) − E A ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ( W ∗ ( l ) − ( V ( l ) ) T V ( l ) ) .
We will bound the leading term and each residual term following the approach of Agterberg et al. [ [1 ] .
Preliminary lemmas
We begin by presenting several lemmas used in the proof.
Lemma B.1. Suppose Assumption 1 and 2 hold, and let σ r ( P ( l ) ) \sigma_r(\mathbf{P}^{(l)}) σ r ( P ( l ) ) denote the r r r -th singular value of P ( l ) \mathbf{P}^{(l)} P ( l ) and let σ r ( B ( l ) ) \sigma_r(\mathbf{B}^{(l)}) σ r ( B ( l ) ) denote the r r r -th singular value of B ( l ) \mathbf{B}^{(l)} B ( l ) in descending order. Then for all 1 ≤ r ≤ K 1 \le r \le K 1 ≤ r ≤ K and 1 ≤ i ≤ n 1 \le i \le n 1 ≤ i ≤ n ,
θ i ( l ) σ B ( l ) K ≲ ∥ X i , ⋅ ( l ) ∥ ≲ θ i ( l ) K ; (10) \theta_i^{(l)} \frac{\sigma_B^{(l)}}{K} \lesssim\lVert\mathbf{X}_{i,\cdot}^{(l)} \rVert\lesssim\theta_i^{(l)} \sqrt{K}; \tag*{(10)} θ i ( l ) K σ B ( l ) ≲ ∥ X i , ⋅ ( l ) ∥ ≲ θ i ( l ) K ; (10) ∥ V i ( l ) ∥ ≤ C G K θ i ( l ) ∥ θ ( l ) ∥ ; (11) \lVert\mathbf{V}_i^{(l)} \rVert\le C_G \sqrt{K}\frac{\theta_i^{(l)}}{\lVert\boldsymbol{\theta}^{(l)} \rVert}; \tag*{(11)} ∥ V i ( l ) ∥ ≤ C G K ∥ θ ( l ) ∥ θ i ( l ) ; (11) σ r ( P ( l ) ) ≍ ∥ θ ( l ) ∥ 2 K σ r ( B ( l ) ) . (12) \sigma_r(\mathbf{P}^{(l)}) \asymp\frac{\lVert\boldsymbol{\theta}^{(l)} \rVert^2}{K}\sigma_r(\mathbf{B}^{(l)}). \tag*{(12)} σ r ( P ( l ) ) ≍ K ∥ θ ( l ) ∥ 2 σ r ( B ( l ) ) . (12)
Proof. First, we prove (B.4). Define
Q ( l ) : = Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 / 2 . \mathbf{Q}^{(l)} := \boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^2\mathbf{Z}\right)^{-1/2}. Q ( l ) := Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1/2 .
Then,
( Q ( l ) ) T Q ( l ) = I K , (\mathbf{Q}^{(l)})^{\mathsf{T}}\mathbf{Q}^{(l)} = I_K, ( Q ( l ) ) T Q ( l ) = I K ,
so Q ( l ) \mathbf{Q}^{(l)} Q ( l ) has orthonormal columns. By the definition of G ( l ) = K ∥ θ ( l ) ∥ ( Z T ( Θ ( l ) ) 2 Z ) 1 / 2 \mathbf{G}^{(l)} = \frac{\sqrt{K}}{\lVert\boldsymbol{\theta}^{(l)} \rVert}\left(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^2\mathbf{Z}\right)^{1/2} G ( l ) = ∥ θ ( l ) ∥ K ( Z T ( Θ ( l ) ) 2 Z ) 1/2 , we have
( Z T ( Θ ( l ) ) 2 Z ) 1 / 2 = ∥ θ ( l ) ∥ K G ( l ) . \left(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^2\mathbf{Z}\right)^{1/2} = \frac{\lVert\boldsymbol{\theta}^{(l)} \rVert}{\sqrt{K}}\mathbf{G}^{(l)}. ( Z T ( Θ ( l ) ) 2 Z ) 1/2 = K ∥ θ ( l ) ∥ G ( l ) .
Therefore,
P ( l ) = ∥ θ ( l ) ∥ 2 K Q ( l ) G ( l ) B ( l ) G ( l ) ( Q ( l ) ) T . \mathbf{P}^{(l)} = \frac{\lVert\boldsymbol{\theta}^{(l)} \rVert^2}{K}\mathbf{Q}^{(l)}\mathbf{G}^{(l)}\mathbf{B}^{(l)}\mathbf{G}^{(l)}(\mathbf{Q}^{(l)})^{\mathsf{T}}. P ( l ) = K ∥ θ ( l ) ∥ 2 Q ( l ) G ( l ) B ( l ) G ( l ) ( Q ( l ) ) T .
Since Q ( l ) \mathbf{Q}^{(l)} Q ( l ) has orthonormal columns, multiplication by Q ( l ) \mathbf{Q}^{(l)} Q ( l ) and ( Q ( l ) ) T (\mathbf{Q}^{(l)})^{\mathsf{T}} ( Q ( l ) ) T preserves the nonzero singular values. Hence, for every 1 ≤ r ≤ K 1 \le r \le K 1 ≤ r ≤ K ,
σ r ( P ( l ) ) = ∥ θ ( l ) ∥ 2 K σ r ( G ( l ) B ( l ) G ( l ) ) . \sigma_r(\mathbf{P}^{(l)}) = \frac{\lVert\boldsymbol{\theta}^{(l)} \rVert^2}{K}\sigma_r\left(\mathbf{G}^{(l)}\mathbf{B}^{(l)}\mathbf{G}^{(l)}\right). σ r ( P ( l ) ) = K ∥ θ ( l ) ∥ 2 σ r ( G ( l ) B ( l ) G ( l ) ) .
Using the standard singular-value inequality and Assumption 1,
σ r ( G ( l ) B ( l ) G ( l ) ) ≤ ∥ G ( l ) ∥ 2 σ r ( B ( l ) ) ≤ C G 2 σ r ( B ( l ) ) . \sigma_r\left(\mathbf{G}^{(l)}\mathbf{B}^{(l)}\mathbf{G}^{(l)}\right) \le\lVert\mathbf{G}^{(l)} \rVert^2 \sigma_r(\mathbf{B}^{(l)}) \le C_G^2\sigma_r(\mathbf{B}^{(l)}). σ r ( G ( l ) B ( l ) G ( l ) ) ≤ ∥ G ( l ) ∥ 2 σ r ( B ( l ) ) ≤ C G 2 σ r ( B ( l ) ) .
Conversely,
B ( l ) = ( G ( l ) ) − 1 { G ( l ) B ( l ) G ( l ) } ( G ( l ) ) − 1 , \mathbf{B}^{(l)} = (\mathbf{G}^{(l)})^{-1}\left\{\mathbf{G}^{(l)}\mathbf{B}^{(l)}\mathbf{G}^{(l)}\right\}(\mathbf{G}^{(l)})^{-1}, B ( l ) = ( G ( l ) ) − 1 { G ( l ) B ( l ) G ( l ) } ( G ( l ) ) − 1 ,
and hence
σ r ( B ( l ) ) ≤ ∥ ( G ( l ) ) − 1 ∥ 2 σ r ( G ( l ) B ( l ) G ( l ) ) ≤ C G 2 σ r ( G ( l ) B ( l ) G ( l ) ) . \sigma_r(\mathbf{B}^{(l)}) \le\lVert(\mathbf{G}^{(l)})^{-1}\rVert^2\sigma_r\left(\mathbf{G}^{(l)}\mathbf{B}^{(l)}\mathbf{G}^{(l)}\right) \le C_G^2\sigma_r\left(\mathbf{G}^{(l)}\mathbf{B}^{(l)}\mathbf{G}^{(l)}\right). σ r ( B ( l ) ) ≤ ∥( G ( l ) ) − 1 ∥ 2 σ r ( G ( l ) B ( l ) G ( l ) ) ≤ C G 2 σ r ( G ( l ) B ( l ) G ( l ) ) .
It follows that
C G − 2 σ r ( B ( l ) ) ≤ σ r ( G ( l ) B ( l ) G ( l ) ) ≤ C G 2 σ r ( B ( l ) ) . C_G^{-2}\sigma_r(\mathbf{B}^{(l)}) \le\sigma_r\left(\mathbf{G}^{(l)}\mathbf{B}^{(l)}\mathbf{G}^{(l)}\right) \le C_G^2\sigma_r(\mathbf{B}^{(l)}). C G − 2 σ r ( B ( l ) ) ≤ σ r ( G ( l ) B ( l ) G ( l ) ) ≤ C G 2 σ r ( B ( l ) ) .
Therefore,
∥ θ ( l ) ∥ 2 K C G 2 σ r ( B ( l ) ) ≤ σ r ( P ( l ) ) ≤ C G 2 ∥ θ ( l ) ∥ 2 K σ r ( B ( l ) ) . \frac{\lVert\boldsymbol{\theta}^{(l)} \rVert^2}{K C_G^2}\sigma_r(\mathbf{B}^{(l)}) \le\sigma_r(\mathbf{P}^{(l)}) \le\frac{C_G^2\lVert\boldsymbol{\theta}^{(l)} \rVert^2}{K}\sigma_r(\mathbf{B}^{(l)}). K C G 2 ∥ θ ( l ) ∥ 2 σ r ( B ( l ) ) ≤ σ r ( P ( l ) ) ≤ K C G 2 ∥ θ ( l ) ∥ 2 σ r ( B ( l ) ) .
Thus,
σ r ( P ( l ) ) ≍ ∥ θ ( l ) ∥ 2 K σ r ( B ( l ) ) . \sigma_r(\mathbf{P}^{(l)}) \asymp\frac{\lVert\boldsymbol{\theta}^{(l)} \rVert^2}{K}\sigma_r(\mathbf{B}^{(l)}). σ r ( P ( l ) ) ≍ K ∥ θ ( l ) ∥ 2 σ r ( B ( l ) ) .
which proves (B.4).
Next, we show (B.3). First, we show P ( l ) \mathbf{P}^{(l)} P ( l ) and ( Z T ( Θ ( l ) ) 2 Z ) 1 2 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 1 2 \left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}}\mathbf{B}^{(l)}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}} ( Z T ( Θ ( l ) ) 2 Z ) 2 1 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 2 1 have the same non-zero eigenvalues. We can rewrite P ( l ) \mathbf{P}^{(l)} P ( l ) in terms of ( Z T ( Θ ( l ) ) 2 Z ) 1 2 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 1 2 \left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}}\mathbf{B}^{(l)}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}} ( Z T ( Θ ( l ) ) 2 Z ) 2 1 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 2 1 by following:
P ( l ) = Θ ( l ) Z B ( l ) Z T Θ ( l ) = Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 ( Z T ( Θ ( l ) ) 2 Z ) 1 2 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 1 2 ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 Z T Θ ( l ) . \begin{aligned}
\mathbf{P}^{(l)}=\boldsymbol{\Theta}^{(l)}\mathbf{Z}\mathbf{B}^{(l)}\mathbf{Z}^{\mathsf{T}}\boldsymbol{\Theta}^{(l)} \\
&=\boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}}\mathbf{B}^{(l)}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\mathbf{Z}^{\mathsf{T}}\boldsymbol{\Theta}^{(l)}.
\end{aligned} P ( l ) = Θ ( l ) Z B ( l ) Z T Θ ( l ) = Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 ( Z T ( Θ ( l ) ) 2 Z ) 2 1 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 2 1 ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 Z T Θ ( l ) .
Since Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 \boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}} Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 is full column rank matrix with orthonormal columns, P ( l ) \mathbf{P}^{(l)} P ( l ) and ( Z T ( Θ ( l ) ) 2 Z ) 1 2 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 1 2 \left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}}\mathbf{B}^{(l)}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}} ( Z T ( Θ ( l ) ) 2 Z ) 2 1 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 2 1 have the same non-zero eigenvalues. Using the same non-zero eigenvalues, let
( Z T ( Θ ( l ) ) 2 Z ) 1 2 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 1 2 = U ~ ( l ) Λ ( l ) U ~ ( l ) T (13) \left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}}\mathbf{B}^{(l)}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}}=\widetilde{\mathbf{U}}^{(l)}\boldsymbol{\Lambda}^{(l)}\widetilde{\mathbf{U}}^{(l)\mathsf{T}} \tag*{(13)} ( Z T ( Θ ( l ) ) 2 Z ) 2 1 B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 2 1 = U ( l ) Λ ( l ) U ( l ) T (13)
be an eigendecomposition, where the columns of U ~ ( l ) \widetilde{\mathbf{U}}^{(l)} U ( l ) are ordered consistently with the diagonal entries of Λ ( l ) \boldsymbol{\Lambda}^{(l)} Λ ( l ) . Then it holds that
P ( l ) = Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) Λ ( l ) ( U ~ ( l ) ) T ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 Z T Θ ( l ) . (14) \mathbf{P}^{(l)}=\boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\boldsymbol{\Lambda}^{(l)}\left(\widetilde{\mathbf{U}}^{(l)}\right)^{\mathsf{T}}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\mathbf{Z}^{\mathsf{T}}\boldsymbol{\Theta}^{(l)}. \tag*{(14)} P ( l ) = Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) Λ ( l ) ( U ( l ) ) T ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 Z T Θ ( l ) . (14)
Note that the matrix Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) \boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)} Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) has orthonormal columns. That is,
( Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) ) T Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) = ( U ~ ( l ) ) T ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 Z T Θ ( l ) Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) = ( U ~ ( l ) ) T U ~ ( l ) = I K . \begin{aligned}
\left(\boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\right)^{\mathsf{T}}\boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)} \\
&=\left(\widetilde{\mathbf{U}}^{(l)}\right)^{\mathsf{T}}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\mathbf{Z}^{\mathsf{T}}\boldsymbol{\Theta}^{(l)}\boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)} \\
&=\left(\widetilde{\mathbf{U}}^{(l)}\right)^{\mathsf{T}}\widetilde{\mathbf{U}}^{(l)}=\mathbf{I}_{K}.
\end{aligned} ( Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) ) T Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) = ( U ( l ) ) T ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 Z T Θ ( l ) Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) = ( U ( l ) ) T U ( l ) = I K .
Therefore, the decomposition (B.6) is also an eigen-decomposition of P ( l ) \mathbf{P}^{(l)} P ( l ) . That is, V ( l ) \mathbf{V}^{(l)} V ( l ) can be written as
V ( l ) = Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) O ( l ) . (15) \mathbf{V}^{(l)}=\boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\mathbf{O}^{(l)}. \tag*{(15)} V ( l ) = Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) O ( l ) . (15)
up to orthogonal transformation O ( l ) \mathbf{O}^{(l)} O ( l ) satisfying O ( l ) Λ ( l ) O ( l ) T = Λ ( l ) \mathbf{O}^{(l)}\boldsymbol{\Lambda}^{(l)}\mathbf{O}^{(l)\mathsf{T}}=\boldsymbol{\Lambda}^{(l)} O ( l ) Λ ( l ) O ( l ) T = Λ ( l ) . Now, we can obtain the upper bound of ∥ V i , ⋅ ( l ) ∥ \left\|\mathbf{V}^{(l)}_{i,\cdot}\right\| V i , ⋅ ( l ) as
∥ V i , ⋅ ( l ) ∥ = ∥ e i T Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) ∥ = θ i ( l ) ∥ Z i , ⋅ T ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) ∥ ≤ θ i ( l ) ∥ ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 ∥ ∥ U ~ ( l ) ∥ ∥ Z i , ⋅ ∥ ≤ C G θ i ( l ) K ∥ θ ( l ) ∥ . \begin{aligned}
\left\|\mathbf{V}^{(l)}_{i,\cdot}\right\|=\left\|\mathbf{e}_i^{\mathsf{T}}\boldsymbol{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\right\| \\
&=\theta_i^{(l)}\left\|\mathbf{Z}_{i,\cdot}^{\mathsf{T}}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\right\| \\
&\leq\theta_i^{(l)}\left\|\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\right\|\left\|\widetilde{\mathbf{U}}^{(l)}\right\|\left\|\mathbf{Z}_{i,\cdot}\right\| \\
&\leq C_G\frac{\theta_i^{(l)}\sqrt{K}}{\left\|\boldsymbol{\theta}^{(l)}\right\|}.
\end{aligned} V i , ⋅ ( l ) = e i T Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) = θ i ( l ) Z i , ⋅ T ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) ≤ θ i ( l ) ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) ∥ Z i , ⋅ ∥ ≤ C G θ ( l ) θ i ( l ) K .
where the last inequality comes from the assumption 1,
∥ ( G ( l ) ) − 1 ∥ = ∥ θ ( l ) ∥ K ∥ ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 ∥ ≤ C G . \left\Vert\left(G^{(l)}\right)^{-1}\right\Vert=\frac{\left\Vert\theta^{(l)}\right\Vert}{\sqrt{K}}\left\Vert\left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{-\frac{1}{2}}\right\Vert\leq C_G. ( G ( l ) ) − 1 = K θ ( l ) ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 ≤ C G .
This proves (B.3).
Lastly, we show (B.2). The upper bound of ∥ X i , : ( l ) ∥ \left\Vert X_{i,:}^{(l)}\right\Vert X i , : ( l ) can be obtained by
∥ X i , : ( l ) ∥ = ∥ V i , : ( l ) ∣ Λ ( l ) ∣ 1 2 ∥ ≤ ∥ ∣ Λ ( l ) ∣ 1 2 ∥ ∥ V i , : ( l ) ∥ = σ 1 ( P ( l ) ) ∥ V i , : ( l ) ∥ ≤ C G σ 1 ( P ( l ) ) K θ i ( l ) ∥ θ ( l ) ∥ ≤ ∥ θ ( l ) ∥ K σ 1 ( B ( l ) ) K θ i ( l ) ∥ θ ( l ) ∥ = σ 1 ( B ( l ) ) θ i ( l ) ≤ K θ i ( l ) , \begin{aligned}
\left\Vert X_{i,:}^{(l)}\right\Vert=\left\Vert V_{i,:}^{(l)}\left|\Lambda^{(l)}\right|^{\frac{1}{2}}\right\Vert\\
&\leq\left\Vert\left|\Lambda^{(l)}\right|^{\frac{1}{2}}\right\Vert\left\Vert V_{i,:}^{(l)}\right\Vert\\
&=\sqrt{\sigma_1\left(P^{(l)}\right)}\left\Vert V_{i,:}^{(l)}\right\Vert\\
&\leq C_G\sqrt{\sigma_1\left(P^{(l)}\right)}\sqrt{K}\frac{\theta_i^{(l)}}{\left\Vert\theta^{(l)}\right\Vert} \\
&\leq\frac{\left\Vert\theta^{(l)}\right\Vert}{\sqrt{K}}\sqrt{\sigma_1\left(B^{(l)}\right)}\sqrt{K}\frac{\theta_i^{(l)}}{\left\Vert\theta^{(l)}\right\Vert} \\
&=\sqrt{\sigma_1\left(B^{(l)}\right)}\theta_i^{(l)} \\
&\leq\sqrt{K}\theta_i^{(l)},
\end{aligned} X i , : ( l ) = V i , : ( l ) Λ ( l ) 2 1 ≤ Λ ( l ) 2 1 V i , : ( l ) = σ 1 ( P ( l ) ) V i , : ( l ) ≤ C G σ 1 ( P ( l ) ) K θ ( l ) θ i ( l ) ≤ K θ ( l ) σ 1 ( B ( l ) ) K θ ( l ) θ i ( l ) = σ 1 ( B ( l ) ) θ i ( l ) ≤ K θ i ( l ) ,
where the last inequality comes from the equivalence property between max norm and spectral norm of a matrix. Next, we derive a lower bound for ∥ X i , : ( l ) ∥ \left\Vert X_{i,:}^{(l)}\right\Vert X i , : ( l ) . Using the expression of V ( l ) V^{(l)} V ( l ) by (B.7), X ( l ) X^{(l)} X ( l ) can be written as
X ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1 2 = Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) ∣ Λ ( l ) ∣ 1 2 . X^{(l)}=V^{(l)}\left|\Lambda^{(l)}\right|^{\frac{1}{2}}=\Theta^{(l)}Z\left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{-\frac{1}{2}}\widetilde{U}^{(l)}\left|\Lambda^{(l)}\right|^{\frac{1}{2}}. X ( l ) = V ( l ) Λ ( l ) 2 1 = Θ ( l ) Z ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) Λ ( l ) 2 1 .
Recall that by (B.5), we have
( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) = B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 1 2 U ~ ( l ) ( Λ ( l ) ) − 1 , \left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{-\frac{1}{2}}\widetilde{U}^{(l)}=B^{(l)}\left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{\frac{1}{2}}\widetilde{U}^{(l)}\left(\Lambda^{(l)}\right)^{-1}, ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) = B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 2 1 U ( l ) ( Λ ( l ) ) − 1 ,
which implies that
V ( l ) ∣ Λ ( l ) ∣ 1 2 = Θ ( l ) Z B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 1 2 U ~ ( l ) ( Λ ( l ) ) − 1 ∣ Λ ( l ) ∣ 1 2 = Θ ( l ) Z B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 1 2 U ~ ( l ) ∣ Λ ( l ) ∣ − 1 2 W ( l ) , \begin{aligned}
V^{(l)}\left|\Lambda^{(l)}\right|^{\frac{1}{2}}=\Theta^{(l)}ZB^{(l)}\left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{\frac{1}{2}}\widetilde{U}^{(l)}\left(\Lambda^{(l)}\right)^{-1}\left|\Lambda^{(l)}\right|^{\frac{1}{2}} \\
&=\Theta^{(l)}ZB^{(l)}\left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{\frac{1}{2}}\widetilde{U}^{(l)}\left|\Lambda^{(l)}\right|^{-\frac{1}{2}}W^{(l)},
\end{aligned} V ( l ) Λ ( l ) 2 1 = Θ ( l ) Z B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 2 1 U ( l ) ( Λ ( l ) ) − 1 Λ ( l ) 2 1 = Θ ( l ) Z B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 2 1 U ( l ) Λ ( l ) − 2 1 W ( l ) ,
where W ( l ) W^{(l)} W ( l ) is the diagonal matrix of sign of Λ ( l ) \Lambda^{(l)} Λ ( l ) . Then,
∥ X i , : ( l ) ∥ = θ i ( l ) ∥ Z i , : B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 1 2 U ~ ( l ) ∣ Λ ( l ) ∣ − 1 2 W ( l ) ∥ ≥ θ i ( l ) ∥ Z i , : B ( l ) ∥ σ min ( ( Z T ( Θ ( l ) ) 2 Z ) 1 2 U ~ ( l ) ∣ Λ ( l ) ∣ − 1 2 ) ≥ θ i ( l ) ∥ Z i , : ∥ σ min ( B ( l ) ) σ min ( ( Z T ( Θ ( l ) ) 2 Z ) 1 2 U ~ ( l ) ∣ Λ ( l ) ∣ − 1 2 ) ≥ θ i ( l ) 1 K σ min ( B ( l ) ) σ min ( ( Z T ( Θ ( l ) ) 2 Z ) 1 2 ) σ min ( ∣ Λ ( l ) ∣ − 1 2 ) , \begin{aligned}
\left\Vert X_{i,:}^{(l)}\right\Vert=\theta_i^{(l)}\left\Vert Z_{i,:}B^{(l)}\left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{\frac{1}{2}}\widetilde{U}^{(l)}\left|\Lambda^{(l)}\right|^{-\frac{1}{2}}W^{(l)}\right\Vert\\
&\geq\theta_i^{(l)}\left\Vert Z_{i,:}B^{(l)}\right\Vert\sigma_{\min}\left(\left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{\frac{1}{2}}\widetilde{U}^{(l)}\left|\Lambda^{(l)}\right|^{-\frac{1}{2}}\right) \\
&\geq\theta_i^{(l)}\left\Vert Z_{i,:}\right\Vert\sigma_{\min}\left(B^{(l)}\right)\sigma_{\min}\left(\left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{\frac{1}{2}}\widetilde{U}^{(l)}\left|\Lambda^{(l)}\right|^{-\frac{1}{2}}\right) \\
&\geq\theta_i^{(l)}\frac{1}{\sqrt{K}}\sigma_{\min}\left(B^{(l)}\right)\sigma_{\min}\left(\left(Z^{\mathsf{T}}\left(\Theta^{(l)}\right)^2Z\right)^{\frac{1}{2}}\right)\sigma_{\min}\left(\left|\Lambda^{(l)}\right|^{-\frac{1}{2}}\right),
\end{aligned} X i , : ( l ) = θ i ( l ) Z i , : B ( l ) ( Z T ( Θ ( l ) ) 2 Z ) 2 1 U ( l ) Λ ( l ) − 2 1 W ( l ) ≥ θ i ( l ) Z i , : B ( l ) σ m i n ( ( Z T ( Θ ( l ) ) 2 Z ) 2 1 U ( l ) Λ ( l ) − 2 1 ) ≥ θ i ( l ) ∥ Z i , : ∥ σ m i n ( B ( l ) ) σ m i n ( ( Z T ( Θ ( l ) ) 2 Z ) 2 1 U ( l ) Λ ( l ) − 2 1 ) ≥ θ i ( l ) K 1 σ m i n ( B ( l ) ) σ m i n ( ( Z T ( Θ ( l ) ) 2 Z ) 2 1 ) σ m i n ( Λ ( l ) − 2 1 ) ,
where the last inequality comes from σ min ( A B ) ≥ σ min ( A ) σ min ( B ) \sigma_{\min}(\mathbf{A}\mathbf{B}) \ge\sigma_{\min}(\mathbf{A})\sigma_{\min}(\mathbf{B}) σ m i n ( AB ) ≥ σ m i n ( A ) σ m i n ( B ) for any invertible matrix A , B \mathbf{A},\mathbf{B} A , B and σ min ( W ) = 1 \sigma_{\min}(\mathbf{W})=1 σ m i n ( W ) = 1 for any orthogonal matrix W \mathbf{W} W . We provide a lower bound for each minimum singular value. We know σ min ( ( Z T ( Θ ( l ) ) 2 Z ) 1 2 ) = 1 σ max ( ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 ) \sigma_{\min}\left(\left(\mathbf{Z}^{\mathsf{T}}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}}\right)=\frac{1}{\sigma_{\max}\left(\left(\mathbf{Z}^{\mathsf{T}}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\right)} σ m i n ( ( Z T ( Θ ( l ) ) 2 Z ) 2 1 ) = σ m a x ( ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 ) 1 and by the assumption 1,
∥ ( G ( l ) ) − 1 ∥ = ∥ θ ( l ) ∥ K ∥ ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 ∥ ≤ C G ⇔ σ max ( ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 ) ≤ C G K ∥ θ ( l ) ∥ ⇔ 1 σ max ( ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 ) ≥ ∥ θ ( l ) ∥ C G K . \begin{aligned}
\left\|\left(\mathbf{G}^{(l)}\right)^{-1}\right\|=\frac{\left\|\mathbf{\theta}^{(l)}\right\|}{\sqrt{K}}\left\|\left(\mathbf{Z}^{\mathsf{T}}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\right\|\le C_G \\
&\Leftrightarrow\sigma_{\max}\left(\left(\mathbf{Z}^{\mathsf{T}}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\right)\le C_G\frac{\sqrt{K}}{\left\|\mathbf{\theta}^{(l)}\right\|} \\
&\Leftrightarrow\frac{1}{\sigma_{\max}\left(\left(\mathbf{Z}^{\mathsf{T}}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\right)}\ge\frac{\left\|\mathbf{\theta}^{(l)}\right\|}{C_G\sqrt{K}}.
\end{aligned} ( G ( l ) ) − 1 = K θ ( l ) ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 ≤ C G ⇔ σ m a x ( ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 ) ≤ C G θ ( l ) K ⇔ σ m a x ( ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 ) 1 ≥ C G K θ ( l ) .
Also, it can be shown that
σ min ( ∣ Λ ( l ) ∣ − 1 2 ) = λ min ( ∣ Λ ( l ) ∣ − 1 2 ) = 1 λ max ( ∣ Λ ( l ) ∣ 1 2 ) = 1 σ 1 ( P ( l ) ) . \sigma_{\min}\left(\left|\mathbf{\Lambda}^{(l)}\right|^{-\frac{1}{2}}\right)=\lambda_{\min}\left(\left|\mathbf{\Lambda}^{(l)}\right|^{-\frac{1}{2}}\right)=\frac{1}{\lambda_{\max}\left(\left|\mathbf{\Lambda}^{(l)}\right|^{\frac{1}{2}}\right)}=\frac{1}{\sqrt{\sigma_1\left(\mathbf{P}^{(l)}\right)}}. σ m i n ( Λ ( l ) − 2 1 ) = λ m i n ( Λ ( l ) − 2 1 ) = λ m a x ( Λ ( l ) 2 1 ) 1 = σ 1 ( P ( l ) ) 1 .
Therefore,
∥ X i , : ( l ) ∥ ≥ θ i ( l ) 1 K σ min ( B ( l ) ) σ min ( ( Z T ( Θ ( l ) ) 2 Z ) 1 2 ) σ min ( ∣ Λ ( l ) ∣ − 1 2 ) ≥ θ i ( l ) 1 K σ min ( B ( l ) ) ∥ θ ( l ) ∥ C G K 1 σ 1 ( P ( l ) ) ≥ θ i ( l ) 1 K σ min ( B ( l ) ) ∥ θ ( l ) ∥ K ∥ θ ( l ) ∥ K ∥ θ ( l ) ∥ 1 σ 1 ( B ( l ) ) ≥ θ i ( l ) 1 K 1 K σ min ( B ( l ) ) . \begin{aligned}
\left\|\mathbf{X}_{i,:}^{(l)}\right\|\ge\theta_i^{(l)}\frac{1}{\sqrt{K}}\sigma_{\min}\left(\mathbf{B}^{(l)}\right)\sigma_{\min}\left(\left(\mathbf{Z}^{\mathsf{T}}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{\frac{1}{2}}\right)\sigma_{\min}\left(\left|\mathbf{\Lambda}^{(l)}\right|^{-\frac{1}{2}}\right) \\
&\ge\theta_i^{(l)}\frac{1}{\sqrt{K}}\sigma_{\min}\left(\mathbf{B}^{(l)}\right)\frac{\left\|\mathbf{\theta}^{(l)}\right\|}{C_G\sqrt{K}}\frac{1}{\sqrt{\sigma_1\left(\mathbf{P}^{(l)}\right)}} \\
&\ge\theta_i^{(l)}\frac{1}{\sqrt{K}}\sigma_{\min}\left(\mathbf{B}^{(l)}\right)\frac{\left\|\mathbf{\theta}^{(l)}\right\|}{\sqrt{K}\left\|\mathbf{\theta}^{(l)}\right\|}\frac{\sqrt{K}}{\left\|\mathbf{\theta}^{(l)}\right\|}\frac{1}{\sqrt{\sigma_1\left(\mathbf{B}^{(l)}\right)}} \\
&\ge\theta_i^{(l)}\frac{1}{\sqrt{K}}\frac{1}{\sqrt{K}}\sigma_{\min}\left(\mathbf{B}^{(l)}\right).
\end{aligned} X i , : ( l ) ≥ θ i ( l ) K 1 σ m i n ( B ( l ) ) σ m i n ( ( Z T ( Θ ( l ) ) 2 Z ) 2 1 ) σ m i n ( Λ ( l ) − 2 1 ) ≥ θ i ( l ) K 1 σ m i n ( B ( l ) ) C G K θ ( l ) σ 1 ( P ( l ) ) 1 ≥ θ i ( l ) K 1 σ m i n ( B ( l ) ) K θ ( l ) θ ( l ) θ ( l ) K σ 1 ( B ( l ) ) 1 ≥ θ i ( l ) K 1 K 1 σ m i n ( B ( l ) ) .
We use the following key lemmas from Agterberg et al. [ [1 ] (Lemmas C.3–C.7) without proof, which are crucial for our proof. The proofs of Lemmas C.3 – C.7 depend only on concentration bounds for A ( l ) = P ( l ) \mathbf{A}^{(l)}=\mathbf{P}^{(l)} A ( l ) = P ( l ) , the eigendecomposition of P ( l ) \mathbf{P}^{(l)} P ( l ) , and algebraic identities involving V ( l ) \mathbf{V}^{(l)} V ( l ) and V ^ ( l ) \widehat{\mathbf{V}}^{(l)} V ( l ) . In particular, they do not use the hard-clustering structure of Z \mathbf{Z} Z . Hence the same arguments extend to the MLD-CM setting.
Lemma B.2. When θ max ( l ) ∥ θ ( l ) ∥ 1 ≥ log ( n ) \theta_{\max}^{(l)}\left\|\boldsymbol{\theta}^{(l)}\right\|_1\ge\log(n) θ m a x ( l ) θ ( l ) 1 ≥ log ( n ) , it holds that
∥ A ( l ) − P ( l ) ∥ ≲ θ max ( l ) ∥ θ ( l ) ∥ 1 \left\|\mathbf{A}^{(l)}-\mathbf{P}^{(l)}\right\|\lesssim\sqrt{\theta_{\max}^{(l)}\left\|\boldsymbol{\theta}^{(l)}\right\|_1} A ( l ) − P ( l ) ≲ θ m a x ( l ) θ ( l ) 1 ∥ ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) ∥ ≲ K + log ( n ) , \left\|\left(\mathbf{V}^{(l)}\right)^{\mathsf{T}}\left(\mathbf{A}^{(l)}-\mathbf{P}^{(l)}\right)\mathbf{V}^{(l)}\right\|\lesssim\sqrt{K}+\sqrt{\log(n)}, ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) ≲ K + log ( n ) , with probability at least 1 − O ( n − 20 ) 1-O(n^{-20}) 1 − O ( n − 20 ) .
Lemma B.3. When σ K ( P ( l ) ) ≥ θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) \sigma_K\left(\mathbf{P}^{(l)}\right)\ge\sqrt{\theta_{\max}^{(l)}}\left\|\boldsymbol{\theta}^{(l)}\right\|_1\log(n) σ K ( P ( l ) ) ≥ θ m a x ( l ) θ ( l ) 1 log ( n ) and min i θ i ( l ) ∥ θ ( l ) ∥ 1 ≳ log ( n ) \min_i\theta_i^{(l)}\left\|\boldsymbol{\theta}^{(l)}\right\|_1\gtrsim\log(n) min i θ i ( l ) θ ( l ) 1 ≳ log ( n ) , the following
bounds hold with probability at least 1 − O ( n − 20 ) 1-O(n^{-20}) 1 − O ( n − 20 ) :
∥ ( V ( l ) ) ⊤ V ^ ( l ) − ( W ∗ ( l ) ) ⊤ ∥ ≲ θ max ( l ) ∥ θ ( l ) ∥ 1 ( σ K ( P ( l ) ) ) 2 \left\|(\mathbf{V}^{(l)})^\top\widehat{\mathbf{V}}^{(l)}-(\mathbf{W}^{(l)}_*)^\top\right\| \lesssim\frac{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{(\sigma_K(\mathbf{P}^{(l)}))^2} ( V ( l ) ) ⊤ V ( l ) − ( W ∗ ( l ) ) ⊤ ≲ ( σ K ( P ( l ) ) ) 2 θ m a x ( l ) ∥ θ ( l ) ∥ 1
∥ ( V ^ ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ 1 2 − ∣ Λ ^ ( l ) ∣ 1 2 ( V ^ ( l ) ) ⊤ V ( l ) ∥ ≲ K 2 ( σ K ( P ( l ) ) ) 1 2 ( K log ( n ) + θ max ( l ) ∥ θ ( l ) ∥ 1 σ K ( P ( l ) ) ) \left\|(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}|\boldsymbol{\Lambda}^{(l)}|^{\frac{1}{2}}-|\widehat{\boldsymbol{\Lambda}}^{(l)}|^{\frac{1}{2}}(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}\right\| \lesssim\frac{K^2}{(\sigma_K(\mathbf{P}^{(l)}))^{\frac{1}{2}}}\left(\sqrt{K}\log(n)+\frac{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\sigma_K(\mathbf{P}^{(l)})}\right) ( V ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ 2 1 − ∣ Λ ( l ) ∣ 2 1 ( V ( l ) ) ⊤ V ( l ) ≲ ( σ K ( P ( l ) ) ) 2 1 K 2 ( K log ( n ) + σ K ( P ( l ) ) θ m a x ( l ) ∥ θ ( l ) ∥ 1 )
∥ ( V ^ ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) − ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ( V ^ ( l ) ) ⊤ V ( l ) ∥ ≲ K 2 ( σ K ( P ( l ) ) ) 3 2 ( K log ( n ) + θ max ( l ) ∥ θ ( l ) ∥ 1 σ K ( P ( l ) ) ) . \left\|(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}|\boldsymbol{\Lambda}^{(l)}|^{-\frac{1}{2}}\mathbf{I}_{p,q}^{(l)}-\left|\widehat{\boldsymbol{\Lambda}}^{(l)}\right|^{-\frac{1}{2}}\mathbf{I}_{p,q}^{(l)}(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}\right\| \lesssim\frac{K^2}{(\sigma_K(\mathbf{P}^{(l)}))^{\frac{3}{2}}}\left(\sqrt{K}\log(n)+\frac{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\sigma_K(\mathbf{P}^{(l)})}\right). ( V ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) − Λ ( l ) − 2 1 I p , q ( l ) ( V ( l ) ) ⊤ V ( l ) ≲ ( σ K ( P ( l ) ) ) 2 3 K 2 ( K log ( n ) + σ K ( P ( l ) ) θ m a x ( l ) ∥ θ ( l ) ∥ 1 ) .
Lemma B.4. When σ K ( P ( l ) ) ≳ θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) \sigma_K(\mathbf{P}^{(l)})\gtrsim\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}\log(n) σ K ( P ( l ) ) ≳ θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) and min i θ i ( l ) ∥ θ ( l ) ∥ 1 ≳ log ( n ) \min_i\theta_i^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\gtrsim\log(n) min i θ i ( l ) ∥ θ ( l ) ∥ 1 ≳ log ( n ) , the following bound holds with probability at least 1 − O ( n − 20 ) 1-O(n^{-20}) 1 − O ( n − 20 ) :
∥ e i ⊤ ( A ( l ) − E A ( l ) ) V ^ ( l ) ∥ ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ^ ( l ) ∥ 2 , ∞ . \left\|\mathbf{e}_i^\top(\mathbf{A}^{(l)}-\mathbb{E}\mathbf{A}^{(l)})\widehat{\mathbf{V}}^{(l)}\right\|\lesssim\sqrt{\theta_i^{(l)}}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)\left\|\widehat{\mathbf{V}}^{(l)}\right\|_{2,\infty}. e i ⊤ ( A ( l ) − E A ( l ) ) V ( l ) ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) V ( l ) 2 , ∞ .
Lemma B.5. When σ K ( P ( l ) ) ≳ θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) \sigma_K(\mathbf{P}^{(l)})\gtrsim\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}\log(n) σ K ( P ( l ) ) ≳ θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) and min i θ i ( l ) ∥ θ ( l ) ∥ 1 ≳ log ( n ) \min_i\theta_i^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\gtrsim\log(n) min i θ i ( l ) ∥ θ ( l ) ∥ 1 ≳ log ( n ) , the following bounds holds with probability at least 1 − O ( n − 19 ) 1-O(n^{-19}) 1 − O ( n − 19 ) :
∥ V ^ ( l ) − V ( l ) ( V ( l ) ) ⊤ V ^ ( l ) ∥ 2 , ∞ ≲ θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) σ K ( P ( l ) ) ∥ V ( l ) ∥ 2 , ∞ \left\|\widehat{\mathbf{V}}^{(l)}-\mathbf{V}^{(l)}(\mathbf{V}^{(l)})^\top\widehat{\mathbf{V}}^{(l)}\right\|_{2,\infty}\lesssim\frac{\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)}}{\sigma_K(\mathbf{P}^{(l)})}\left\|\mathbf{V}^{(l)}\right\|_{2,\infty} V ( l ) − V ( l ) ( V ( l ) ) ⊤ V ( l ) 2 , ∞ ≲ σ K ( P ( l ) ) θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) V ( l ) 2 , ∞ ∥ V ^ ( l ) W ∗ ( l ) − V ( l ) ∥ 2 , ∞ ≲ θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) σ K ( P ( l ) ) ∥ V ( l ) ∥ 2 , ∞ \left\|\widehat{\mathbf{V}}^{(l)}\mathbf{W}^{(l)}_*-\mathbf{V}^{(l)}\right\|_{2,\infty}\lesssim\frac{\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)}}{\sigma_K(\mathbf{P}^{(l)})}\left\|\mathbf{V}^{(l)}\right\|_{2,\infty} V ( l ) W ∗ ( l ) − V ( l ) 2 , ∞ ≲ σ K ( P ( l ) ) θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) V ( l ) 2 , ∞ ∥ V ^ ( l ) ( V ^ ( l ) ) ⊤ V ( l ) − V ( l ) ∥ 2 , ∞ ≲ θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) σ K ( P ( l ) ) ∥ V ( l ) ∥ 2 , ∞ . \left\|\widehat{\mathbf{V}}^{(l)}(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}-\mathbf{V}^{(l)}\right\|_{2,\infty}\lesssim\frac{\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)}}{\sigma_K(\mathbf{P}^{(l)})}\left\|\mathbf{V}^{(l)}\right\|_{2,\infty}. V ( l ) ( V ( l ) ) ⊤ V ( l ) − V ( l ) 2 , ∞ ≲ σ K ( P ( l ) ) θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) V ( l ) 2 , ∞ .
Lemma B.6. When σ K ( P ( l ) ) ≳ θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) \sigma_K(\mathbf{P}^{(l)})\gtrsim\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}\log(n) σ K ( P ( l ) ) ≳ θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) and min i θ i ( l ) ∥ θ ( l ) ∥ 1 ≳ log ( n ) \min_i\theta_i^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\gtrsim\log(n) min i θ i ( l ) ∥ θ ( l ) ∥ 1 ≳ log ( n ) , with probability at least 1 − O ( n − 19 ) 1-O(n^{-19}) 1 − O ( n − 19 ) , it holds that
∥ e i ⊤ ( A ( l ) − E A ( l ) ) ( V ^ ( l ) ( V ^ ( l ) ) ⊤ V ( l ) − V ( l ) ) ∥ ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) σ K ( P ( l ) ) ∥ V ( l ) ∥ 2 , ∞ . \left\|\mathbf{e}_i^\top(\mathbf{A}^{(l)}-\mathbb{E}\mathbf{A}^{(l)})(\widehat{\mathbf{V}}^{(l)}(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}-\mathbf{V}^{(l)})\right\|\lesssim\sqrt{\theta_i^{(l)}}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)\frac{\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)}}{\sigma_K(\mathbf{P}^{(l)})}\left\|\mathbf{V}^{(l)}\right\|_{2,\infty}. e i ⊤ ( A ( l ) − E A ( l ) ) ( V ( l ) ( V ( l ) ) ⊤ V ( l ) − V ( l ) ) ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) σ K ( P ( l ) ) θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) V ( l ) 2 , ∞ .
Lastly, we derive useful bounds relating Λ ^ ( l ) \widehat{\boldsymbol{\Lambda}}^{(l)} Λ ( l ) and Λ ( l ) \boldsymbol{\Lambda}^{(l)} Λ ( l ) as a consequence of Weyl’s inequality.
Lemma B.7 (Weyl consequence). Suppose the conditions of Lemma B.2 hold. In addition, assume that
σ K ( P ( l ) ) ≳ θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) . \sigma_K(\mathbf{P}^{(l)})\gtrsim\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}\log(n). σ K ( P ( l ) ) ≳ θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) . Then, on the event of Lemma B.2, for all sufficiently large n n n ,
∥ A ( l ) − P ( l ) ∥ ≤ 1 2 σ K ( P ( l ) ) . \|\mathbf{A}^{(l)}-\mathbf{P}^{(l)}\|\leq\frac{1}{2}\sigma_K(\mathbf{P}^{(l)}). ∥ A ( l ) − P ( l ) ∥ ≤ 2 1 σ K ( P ( l ) ) . Consequently,
σ K ( A ( l ) ) ≥ 1 2 σ K ( P ( l ) ) \sigma_K(\mathbf{A}^{(l)})\geq\frac{1}{2}\sigma_K(\mathbf{P}^{(l)}) σ K ( A ( l ) ) ≥ 2 1 σ K ( P ( l ) ) and
σ 1 ( A ( l ) ) ≤ 3 2 σ 1 ( P ( l ) ) . \sigma_1(\mathbf{A}^{(l)})\leq\frac{3}{2}\sigma_1(\mathbf{P}^{(l)}). σ 1 ( A ( l ) ) ≤ 2 3 σ 1 ( P ( l ) ) .
Therefore,
∥ Λ ^ ( l ) − 1 ∥ ≤ 2 ∥ Λ ( l ) − 1 ∥ , \left\|\widehat{\Lambda}^{(l)-1}\right\| \le2\left\|\Lambda^{(l)-1}\right\|, Λ ( l ) − 1 ≤ 2 Λ ( l ) − 1 ,
and
∥ Λ ^ ( l ) − 1 / 2 ∥ ≤ 2 ∥ Λ ( l ) − 1 / 2 ∥ , \left\|\widehat{\Lambda}^{(l)-1/2}\right\| \le\sqrt{2}\left\|\Lambda^{(l)-1/2}\right\|, Λ ( l ) − 1/2 ≤ 2 Λ ( l ) − 1/2 ,
and
∥ Λ ^ ( l ) 1 / 2 ∥ ≤ 3 2 ∥ Λ ( l ) 1 / 2 ∥ . \left\|\widehat{\Lambda}^{(l)1/2}\right\| \le\sqrt{\frac{3}{2}}\left\|\Lambda^{(l)1/2}\right\|. Λ ( l ) 1/2 ≤ 2 3 Λ ( l ) 1/2 .
Proof. On the event of Lemma B.2, there exists a constant C > 0 C > 0 C > 0 such that
∥ A ( l ) − P ( l ) ∥ ≤ C θ max ( l ) ∥ θ ( l ) ∥ 1 . (16) \left\|A^{(l)}-P^{(l)}\right\| \le C\sqrt{\theta_{\max}^{(l)}}\left\|\theta^{(l)}\right\|_1. \tag*{(16)} A ( l ) − P ( l ) ≤ C θ m a x ( l ) θ ( l ) 1 . (16)
Moreover, by assumption, we have
σ K ( P ( l ) ) ≳ θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) . (17) \sigma_K\left(P^{(l)}\right) \gtrsim\sqrt{\theta_{\max}^{(l)}}\left\|\theta^{(l)}\right\|_1\log(n). \tag*{(17)} σ K ( P ( l ) ) ≳ θ m a x ( l ) θ ( l ) 1 log ( n ) . (17)
By combining (B.8) and (B.9), we have
∥ A ( l ) − P ( l ) ∥ σ K ( P ( l ) ) ≲ 1 log n → 0. \frac{\left\|A^{(l)}-P^{(l)}\right\|}{\sigma_K\left(P^{(l)}\right)} \lesssim\frac{1}{\sqrt{\log n}} \to0. σ K ( P ( l ) ) A ( l ) − P ( l ) ≲ log n 1 → 0.
Hence, for all sufficiently large n n n ,
∥ A ( l ) − P ( l ) ∥ ≤ 1 2 σ K ( P ( l ) ) . \left\|A^{(l)}-P^{(l)}\right\| \le\frac{1}{2}\sigma_K\left(P^{(l)}\right). A ( l ) − P ( l ) ≤ 2 1 σ K ( P ( l ) ) .
By Weyl’s inequality,
σ K ( A ( l ) ) ≥ σ K ( P ( l ) ) − ∥ A ( l ) − P ( l ) ∥ ≥ 1 2 σ K ( P ( l ) ) . \sigma_K\left(A^{(l)}\right) \ge\sigma_K\left(P^{(l)}\right)-\left\|A^{(l)}-P^{(l)}\right\| \ge\frac{1}{2}\sigma_K\left(P^{(l)}\right). σ K ( A ( l ) ) ≥ σ K ( P ( l ) ) − A ( l ) − P ( l ) ≥ 2 1 σ K ( P ( l ) ) .
Similarly,
σ 1 ( A ( l ) ) ≤ σ 1 ( P ( l ) ) + ∥ A ( l ) − P ( l ) ∥ ≤ 3 2 σ 1 ( P ( l ) ) , \sigma_1\left(A^{(l)}\right) \le\sigma_1\left(P^{(l)}\right)+\left\|A^{(l)}-P^{(l)}\right\| \le\frac{3}{2}\sigma_1\left(P^{(l)}\right), σ 1 ( A ( l ) ) ≤ σ 1 ( P ( l ) ) + A ( l ) − P ( l ) ≤ 2 3 σ 1 ( P ( l ) ) ,
because σ K ( P ( l ) ) ≤ σ 1 ( P ( l ) ) \sigma_K\left(P^{(l)}\right) \le\sigma_1\left(P^{(l)}\right) σ K ( P ( l ) ) ≤ σ 1 ( P ( l ) ) . The displayed bounds for ∣ Λ ^ ( l ) ∣ − 1 \left|\widehat{\Lambda}^{(l)}\right|^{-1} Λ ( l ) − 1 , ∣ Λ ^ ( l ) ∣ − 1 / 2 \left|\widehat{\Lambda}^{(l)}\right|^{-1/2} Λ ( l ) − 1/2 , and ∣ Λ ^ ( l ) ∣ 1 / 2 \left|\widehat{\Lambda}^{(l)}\right|^{1/2} Λ ( l ) 1/2 follow directly from these two singular-value inequalities.
Bound of the residual terms R x ( l ) \mathbf{R}_{x}^{(l)} R x ( l )
For each l ∈ [ L ] l \in[L] l ∈ [ L ] and i ∈ [ n ] i \in[n] i ∈ [ n ] , let E r e s , i ( l ) \mathcal{E}_{\mathrm{res},i}^{(l)} E res , i ( l ) denote the event on which the conclusions of Lemmas B.2–B.6, the Weyl consequence in Lemma B.7, and the Davis–Kahan bound
∥ ( V ( l ) ) ⊤ V ^ ( l ) ∥ ≤ C ∥ A ( l ) − P ( l ) ∥ σ K ( P ( l ) ) \left\|\left(V^{(l)}\right)^{\top}\widehat{V}^{(l)}\right\| \le C\frac{\left\|A^{(l)}-P^{(l)}\right\|}{\sigma_K\left(P^{(l)}\right)} ( V ( l ) ) ⊤ V ( l ) ≤ C σ K ( P ( l ) ) A ( l ) − P ( l )
hold simultaneously. By a union bound, there exists a constant C > 0 C > 0 C > 0 , independent of l l l , such that
P ( E r e s , i ( l ) ) ≥ 1 − C n − 19 . \mathbb{P}\left(\mathcal{E}_{\mathrm{res},i}^{(l)}\right) \ge1-Cn^{-19}. P ( E res , i ( l ) ) ≥ 1 − C n − 19 .
Define
E r e s ( l ) : = ⋂ i = 1 n E r e s , i ( l ) . \mathcal{E}_{\mathrm{res}}^{(l)}:=\bigcap_{i=1}^{n}\mathcal{E}_{\mathrm{res},i}^{(l)}. E res ( l ) := i = 1 ⋂ n E res , i ( l ) .
Then
P ( E r e s ( l ) ) ≥ 1 − C n − 18 . \mathbb{P}\left(\mathcal{E}_{\mathrm{res}}^{(l)}\right) \ge1-Cn^{-18}. P ( E res ( l ) ) ≥ 1 − C n − 18 .
In the residual-bound subsection, all row-wise inequalities for layer l l l are understood to hold on E r e s ( l ) \mathcal{E}_{\mathrm{res}}^{(l)} E res ( l ) . We provide an upper bound of the residual term in the decomposition (B.1). We will apply the lemmas from Section B.1 to their corresponding terms throughout the proof.
The term R 1 ( l ) \mathbf{R}_{1}^{(l)} R 1 ( l )
We start with the first term R 1 ( l ) = − V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ^ ( l ) ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) \mathbf{R}_{1}^{(l)}=-\mathbf{V}^{(l)}(\mathbf{V}^{(l)})^{\mathsf{T}}(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\widehat{\mathbf{V}}^{(l)}\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{-\frac{1}{2}}\mathbf{I}_{p,q}^{(l)} R 1 ( l ) = − V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) Λ ( l ) − 2 1 I p , q ( l ) .
∥ e i T R 1 ( l ) ∥ = ∥ e i T V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ^ ( l ) ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ∥ = ∥ e i T V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) ( V ( l ) ( V ( l ) ) T + I − V ( l ) ( V ( l ) ) T ) V ^ ( l ) ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ∥ ≤ ∥ e i T V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) ( V ( l ) ) T V ^ ( l ) ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ∥ + ∥ e i T V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ⊥ ( l ) ( V ⊥ ( l ) ) T V ^ ( l ) ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ∥ ≤ ∥ e i T V ( l ) ∥ ∥ ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) ∥ ∥ ∣ Λ ^ ( l ) ∣ − 1 2 ∥ + ∥ e i T V ( l ) ∥ ∥ A ( l ) − P ( l ) ∥ ∥ ( V ⊥ ( l ) ) T V ^ ( l ) ∥ ∥ ∣ Λ ^ ( l ) ∣ − 1 2 ∥ ≲ ∥ e i T V ( l ) ∥ ( σ K ( P ( l ) ) ) 1 2 ( ∥ ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) ∥ + ∥ A ( l ) − P ( l ) ∥ 2 σ K ( P ( l ) ) ) ≲ ∥ e i T V ( l ) ∥ ( σ K ( P ( l ) ) ) 1 2 ( K + log ( n ) + θ max ( l ) σ K ( P ( l ) ) ∥ θ ( l ) ∥ 1 ) ≲ K θ i ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 1 K ( σ B ( l ) ) 1 2 ( K log ( n ) + θ max ( l ) ∥ θ ( l ) ∥ 1 2 σ B ( l ) ∥ θ ( l ) ∥ 1 ) ≲ K θ i ( l ) ∥ θ ( l ) ∥ 1 2 ( σ B ( l ) ) 1 2 ( K log ( n ) + K θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 1 2 σ B ( l ) ) . \begin{aligned}
\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{R}_{1}^{(l)}\right\|
&=\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}(\mathbf{V}^{(l)})^{\mathsf{T}}(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\widehat{\mathbf{V}}^{(l)}\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{-\frac{1}{2}}\mathbf{I}_{p,q}^{(l)}\right\|\\
&=\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}(\mathbf{V}^{(l)})^{\mathsf{T}}(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\left(\mathbf{V}^{(l)}(\mathbf{V}^{(l)})^{\mathsf{T}}+\mathbf{I}-\mathbf{V}^{(l)}(\mathbf{V}^{(l)})^{\mathsf{T}}\right)\widehat{\mathbf{V}}^{(l)}\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{-\frac{1}{2}}\mathbf{I}_{p,q}^{(l)}\right\|\\
&\leq\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}(\mathbf{V}^{(l)})^{\mathsf{T}}(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\mathbf{V}^{(l)}(\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{-\frac{1}{2}}\mathbf{I}_{p,q}^{(l)}\right\|\\
&\quad+\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}(\mathbf{V}^{(l)})^{\mathsf{T}}(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\mathbf{V}_{\perp}^{(l)}(\mathbf{V}_{\perp}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{-\frac{1}{2}}\mathbf{I}_{p,q}^{(l)}\right\|\\
&\leq\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}\right\|\left\|(\mathbf{V}^{(l)})^{\mathsf{T}}(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\mathbf{V}^{(l)}\right\|\left\|\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{-\frac{1}{2}}\right\|\\
&\quad+\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}\right\|\left\|\mathbf{A}^{(l)}-\mathbf{P}^{(l)}\right\|\left\|(\mathbf{V}_{\perp}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}\right\|\left\|\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{-\frac{1}{2}}\right\|\\
&\lesssim\frac{\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}\right\|}{\left(\sigma_{K}(\mathbf{P}^{(l)})\right)^{\frac{1}{2}}}\left(\left\|(\mathbf{V}^{(l)})^{\mathsf{T}}(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\mathbf{V}^{(l)}\right\|+\frac{\left\|\mathbf{A}^{(l)}-\mathbf{P}^{(l)}\right\|^{2}}{\sigma_{K}(\mathbf{P}^{(l)})}\right)\\
&\lesssim\frac{\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}\right\|}{\left(\sigma_{K}(\mathbf{P}^{(l)})\right)^{\frac{1}{2}}}\left(\sqrt{K}+\sqrt{\log(n)}+\frac{\theta_{\max}^{(l)}}{\sigma_{K}(\mathbf{P}^{(l)})}\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}\right)\\
&\lesssim\frac{\sqrt{K}\frac{\theta_{i}^{(l)}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}}}{\frac{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}}{\sqrt{K}}\left(\sigma_{B}^{(l)}\right)^{\frac{1}{2}}}\left(\sqrt{K\log(n)}+\frac{\theta_{\max}^{(l)}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}^{2}\sigma_{B}^{(l)}}\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}\right)\\
&\lesssim\frac{K\theta_{i}^{(l)}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}^{2}\left(\sigma_{B}^{(l)}\right)^{\frac{1}{2}}}\left(\sqrt{K\log(n)}+\frac{K\theta_{\max}^{(l)}\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}^{2}\sigma_{B}^{(l)}}\right).
\end{aligned} e i T R 1 ( l ) = e i T V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) Λ ( l ) − 2 1 I p , q ( l ) = e i T V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) ( V ( l ) ( V ( l ) ) T + I − V ( l ) ( V ( l ) ) T ) V ( l ) Λ ( l ) − 2 1 I p , q ( l ) ≤ e i T V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) ( V ( l ) ) T V ( l ) Λ ( l ) − 2 1 I p , q ( l ) + e i T V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ⊥ ( l ) ( V ⊥ ( l ) ) T V ( l ) Λ ( l ) − 2 1 I p , q ( l ) ≤ e i T V ( l ) ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) Λ ( l ) − 2 1 + e i T V ( l ) A ( l ) − P ( l ) ( V ⊥ ( l ) ) T V ( l ) Λ ( l ) − 2 1 ≲ ( σ K ( P ( l ) ) ) 2 1 e i T V ( l ) ( ( V ( l ) ) T ( A ( l ) − P ( l ) ) V ( l ) + σ K ( P ( l ) ) A ( l ) − P ( l ) 2 ) ≲ ( σ K ( P ( l ) ) ) 2 1 e i T V ( l ) ( K + log ( n ) + σ K ( P ( l ) ) θ m a x ( l ) θ ( l ) 1 ) ≲ K ∥ θ ( l ) ∥ 1 ( σ B ( l ) ) 2 1 K ∥ θ ( l ) ∥ 1 θ i ( l ) K log ( n ) + θ ( l ) 1 2 σ B ( l ) θ m a x ( l ) θ ( l ) 1 ≲ θ ( l ) 1 2 ( σ B ( l ) ) 2 1 K θ i ( l ) K log ( n ) + θ ( l ) 1 2 σ B ( l ) K θ m a x ( l ) θ ( l ) 1 .
The term R 2 ( l ) \mathbf{R}_{2}^{(l)} R 2 ( l )
Restate the term R 2 ( l ) = V ( l ) ( ( V ( l ) ) T V ^ ( l ) ∣ Λ ^ ( l ) ∣ 1 2 − ∣ Λ ( l ) ∣ 1 2 ( V ( l ) ) T V ^ ( l ) ) \mathbf{R}_{2}^{(l)}=\mathbf{V}^{(l)}\left((\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{\frac{1}{2}}-\left|\mathbf{\Lambda}^{(l)}\right|^{\frac{1}{2}}(\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}\right) R 2 ( l ) = V ( l ) ( ( V ( l ) ) T V ( l ) Λ ( l ) 2 1 − Λ ( l ) 2 1 ( V ( l ) ) T V ( l ) ) for convenience.
∥ e i T R 2 ( l ) ∥ = ∥ e i T V ( l ) ( ( V ( l ) ) T V ^ ( l ) ∣ Λ ^ ( l ) ∣ 1 2 − ∣ Λ ( l ) ∣ 1 2 ( V ( l ) ) T V ^ ( l ) ) ∥ ≤ ∥ e i T V ( l ) ∥ ∥ ( V ( l ) ) T V ^ ( l ) ∣ Λ ^ ( l ) ∣ 1 2 − ∣ Λ ( l ) ∣ 1 2 ( V ( l ) ) T V ^ ( l ) ∥ ≲ K θ i ( l ) ∥ θ ( l ) ∥ 1 K 2 ( σ K ( P ( l ) ) ) 1 2 ( K log ( n ) + θ max ( l ) σ K ( P ( l ) ) ∥ θ ( l ) ∥ 1 ) ≲ K θ i ( l ) ∥ θ ( l ) ∥ 1 K 3 2 ∥ θ ( l ) ∥ 1 ( σ B ( l ) ) 1 2 ( K log ( n ) + K θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 1 2 σ B ( l ) ) ≲ θ i ( l ) K 3 ∥ θ ( l ) ∥ 1 2 ( σ B ( l ) ) 1 2 ( K log ( n ) + K θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 1 2 σ B ( l ) ) . \begin{aligned}
\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{R}_{2}^{(l)}\right\|
&=\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}\left((\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{\frac{1}{2}}-\left|\mathbf{\Lambda}^{(l)}\right|^{\frac{1}{2}}(\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}\right)\right\|\\
&\leq\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}\right\|\left\|(\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}\left|\widehat{\mathbf{\Lambda}}^{(l)}\right|^{\frac{1}{2}}-\left|\mathbf{\Lambda}^{(l)}\right|^{\frac{1}{2}}(\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}\right\|\\
&\lesssim\sqrt{K}\frac{\theta_{i}^{(l)}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}}\frac{K^{2}}{\left(\sigma_{K}(\mathbf{P}^{(l)})\right)^{\frac{1}{2}}}\left(\sqrt{K\log(n)}+\frac{\theta_{\max}^{(l)}}{\sigma_{K}(\mathbf{P}^{(l)})}\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}\right)\\
&\lesssim\sqrt{K}\frac{\theta_{i}^{(l)}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}}\frac{K^{\frac{3}{2}}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}\left(\sigma_{B}^{(l)}\right)^{\frac{1}{2}}}\left(\sqrt{K\log(n)}+\frac{K\theta_{\max}^{(l)}\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}^{2}\sigma_{B}^{(l)}}\right)\\
&\lesssim\frac{\theta_{i}^{(l)}K^{3}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}^{2}\left(\sigma_{B}^{(l)}\right)^{\frac{1}{2}}}\left(\sqrt{K\log(n)}+\frac{K\theta_{\max}^{(l)}\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}}{\left\|\boldsymbol{\theta}^{(l)}\right\|_{1}^{2}\sigma_{B}^{(l)}}\right).
\end{aligned} e i T R 2 ( l ) = e i T V ( l ) ( ( V ( l ) ) T V ( l ) Λ ( l ) 2 1 − Λ ( l ) 2 1 ( V ( l ) ) T V ( l ) ) ≤ e i T V ( l ) ( V ( l ) ) T V ( l ) Λ ( l ) 2 1 − Λ ( l ) 2 1 ( V ( l ) ) T V ( l ) ≲ K θ ( l ) 1 θ i ( l ) ( σ K ( P ( l ) ) ) 2 1 K 2 ( K log ( n ) + σ K ( P ( l ) ) θ m a x ( l ) θ ( l ) 1 ) ≲ K θ ( l ) 1 θ i ( l ) θ ( l ) 1 ( σ B ( l ) ) 2 1 K 2 3 K log ( n ) + θ ( l ) 1 2 σ B ( l ) K θ m a x ( l ) θ ( l ) 1 ≲ θ ( l ) 1 2 ( σ B ( l ) ) 2 1 θ i ( l ) K 3 K log ( n ) + θ ( l ) 1 2 σ B ( l ) K θ m a x ( l ) θ ( l ) 1 .
The term R 3 ( l ) \mathbf{R}_{3}^{(l)} R 3 ( l )
Restate the term R 3 ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1 2 ( ( V ( l ) ) T V ^ ( l ) − ( W ∗ ( l ) ) T ) \mathbf{R}_{3}^{(l)}=\mathbf{V}^{(l)}\lvert\mathbf{\Lambda}^{(l)}\rvert^{\frac{1}{2}}\left((\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}-(\mathbf{W}_{*}^{(l)})^{\mathsf{T}}\right) R 3 ( l ) = V ( l ) ∣ Λ ( l ) ∣ 2 1 ( ( V ( l ) ) T V ( l ) − ( W ∗ ( l ) ) T ) for convenience.
∥ e i T R 3 ( l ) ∥ = ∥ e i T V ( l ) ∣ Λ ( l ) ∣ 1 2 ( ( V ( l ) ) T V ^ ( l ) − ( W ∗ ( l ) ) T ) ∥ ≲ ∥ e i T X ( l ) ∥ ∥ ( V ( l ) ) T V ^ ( l ) − ( W ∗ ( l ) ) T ∥ ≲ K θ i ( l ) θ max ( l ) ∥ θ ( l ) ∥ 1 ( σ K ( P ( l ) ) ) 1 2 ≲ K 5 2 θ i ( l ) θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 1 4 ( σ B ( l ) ) 2 . \begin{aligned}
\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{R}_{3}^{(l)}\right\|
&=\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{V}^{(l)}\lvert\mathbf{\Lambda}^{(l)}\rvert^{\frac{1}{2}}\left((\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}-(\mathbf{W}_{*}^{(l)})^{\mathsf{T}}\right)\right\|\\
&\lesssim\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{X}^{(l)}\right\|\left\|(\mathbf{V}^{(l)})^{\mathsf{T}}\widehat{\mathbf{V}}^{(l)}-(\mathbf{W}_{*}^{(l)})^{\mathsf{T}}\right\|\\
&\lesssim\sqrt{K}\theta_{i}^{(l)}\frac{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_{1}}{(\sigma_{K}(\mathbf{P}^{(l)}))^{\frac{1}{2}}}\\
&\lesssim\frac{K^{\frac{5}{2}}\theta_{i}^{(l)}\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_{1}}{\|\boldsymbol{\theta}^{(l)}\|_{1}^{4}(\sigma_{B}^{(l)})^{2}}.
\end{aligned} e i T R 3 ( l ) = e i T V ( l ) ∣ Λ ( l ) ∣ 2 1 ( ( V ( l ) ) T V ( l ) − ( W ∗ ( l ) ) T ) ≲ e i T X ( l ) ( V ( l ) ) T V ( l ) − ( W ∗ ( l ) ) T ≲ K θ i ( l ) ( σ K ( P ( l ) ) ) 2 1 θ m a x ( l ) ∥ θ ( l ) ∥ 1 ≲ ∥ θ ( l ) ∥ 1 4 ( σ B ( l ) ) 2 K 2 5 θ i ( l ) θ m a x ( l ) ∥ θ ( l ) ∥ 1 .
The term R 4 ( l ) \mathbf{R}_{4}^{(l)} R 4 ( l )
Restate the term R 4 ( l ) = ( A ( l ) − P ( l ) ) ( V ^ ( l ) ( V ^ ( l ) ) T V ( l ) − V ( l ) ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) \mathbf{R}_{4}^{(l)}=(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})(\widehat{\mathbf{V}}^{(l)}(\widehat{\mathbf{V}}^{(l)})^{\mathsf{T}}\mathbf{V}^{(l)}-\mathbf{V}^{(l)})\lvert\mathbf{\Lambda}^{(l)}\rvert^{-\frac{1}{2}}\mathbf{I}_{p,q}^{(l)} R 4 ( l ) = ( A ( l ) − P ( l ) ) ( V ( l ) ( V ( l ) ) T V ( l ) − V ( l ) ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) for convenience.
∥ e i T R 4 ( l ) ∥ = ∥ e i T ( A ( l ) − P ( l ) ) ( V ^ ( l ) ( V ^ ( l ) ) T V ( l ) − V ( l ) ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) ∥ ≲ ∥ e i T ( A ( l ) − P ( l ) ) ( V ^ ( l ) ( V ^ ( l ) ) T V ( l ) − V ( l ) ) ∥ 1 ( σ K ( P ( l ) ) ) 1 2 ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) σ K ( P ( l ) ) ∥ V ( l ) ∥ 2 , ∞ 1 ( σ K ( P ( l ) ) ) 1 2 ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) K θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 1 2 σ B ( l ) K θ max ( l ) ∥ θ ( l ) ∥ 1 K ∥ θ ( l ) ∥ 1 ( σ B ( l ) ) 1 2 ≲ K 2 ( θ i ( l ) ) 1 2 ( θ max ( l ) ) 3 2 ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 1 4 ( σ B ( l ) ) 3 2 . \begin{aligned}
\left\|\mathbf{e}_{i}^{\mathsf{T}}\mathbf{R}_{4}^{(l)}\right\|
&=\left\|\mathbf{e}_{i}^{\mathsf{T}}(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})(\widehat{\mathbf{V}}^{(l)}(\widehat{\mathbf{V}}^{(l)})^{\mathsf{T}}\mathbf{V}^{(l)}-\mathbf{V}^{(l)})\lvert\mathbf{\Lambda}^{(l)}\rvert^{-\frac{1}{2}}\mathbf{I}_{p,q}^{(l)}\right\|\\
&\lesssim\left\|\mathbf{e}_{i}^{\mathsf{T}}(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})(\widehat{\mathbf{V}}^{(l)}(\widehat{\mathbf{V}}^{(l)})^{\mathsf{T}}\mathbf{V}^{(l)}-\mathbf{V}^{(l)})\right\|\frac{1}{(\sigma_{K}(\mathbf{P}^{(l)}))^{\frac{1}{2}}}\\
&\lesssim\sqrt{\theta_{i}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_{1}\log(n)}\frac{\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_{1}\log(n)}}{\sigma_{K}(\mathbf{P}^{(l)})}\left\|\mathbf{V}^{(l)}\right\|_{2,\infty}\frac{1}{(\sigma_{K}(\mathbf{P}^{(l)}))^{\frac{1}{2}}}\\
&\lesssim\sqrt{\theta_{i}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_{1}\log(n)}\frac{K\sqrt{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_{1}\log(n)}}{\|\boldsymbol{\theta}^{(l)}\|_{1}^{2}\sigma_{B}^{(l)}}\sqrt{K}\frac{\theta_{\max}^{(l)}}{\|\boldsymbol{\theta}^{(l)}\|_{1}}\frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|_{1}(\sigma_{B}^{(l)})^{\frac{1}{2}}}\\
&\lesssim\frac{K^{2}(\theta_{i}^{(l)})^{\frac{1}{2}}(\theta_{\max}^{(l)})^{\frac{3}{2}}\|\boldsymbol{\theta}^{(l)}\|_{1}\log(n)}{\|\boldsymbol{\theta}^{(l)}\|_{1}^{4}(\sigma_{B}^{(l)})^{\frac{3}{2}}}.
\end{aligned} e i T R 4 ( l ) = e i T ( A ( l ) − P ( l ) ) ( V ( l ) ( V ( l ) ) T V ( l ) − V ( l ) ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ≲ e i T ( A ( l ) − P ( l ) ) ( V ( l ) ( V ( l ) ) T V ( l ) − V ( l ) ) ( σ K ( P ( l ) ) ) 2 1 1 ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) σ K ( P ( l ) ) θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) V ( l ) 2 , ∞ ( σ K ( P ( l ) ) ) 2 1 1 ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 1 2 σ B ( l ) K θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) K ∥ θ ( l ) ∥ 1 θ m a x ( l ) ∥ θ ( l ) ∥ 1 ( σ B ( l ) ) 2 1 K ≲ ∥ θ ( l ) ∥ 1 4 ( σ B ( l ) ) 2 3 K 2 ( θ i ( l ) ) 2 1 ( θ m a x ( l ) ) 2 3 ∥ θ ( l ) ∥ 1 log ( n ) .
The term R 5 ( l ) \mathbf{R}^{(l)}_5 R 5 ( l )
Restate the term R 5 ( l ) = − ( A ( l ) − P ( l ) ) V ^ ( l ) ( ( V ^ ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) − ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ( V ^ ( l ) ) ⊤ V ( l ) ) \mathbf{R}^{(l)}_5 = -(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\widehat{\mathbf{V}}^{(l)}\left((\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}|\mathbf{\Lambda}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}-|\widehat{\mathbf{\Lambda}}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}\right) R 5 ( l ) = − ( A ( l ) − P ( l ) ) V ( l ) ( ( V ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) − ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ( V ( l ) ) ⊤ V ( l ) ) for convenience.
∥ e i ⊤ R 5 ( l ) ∥ = ∥ e i ⊤ ( A ( l ) − P ( l ) ) V ^ ( l ) ( ( V ^ ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) − ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ( V ^ ( l ) ) ⊤ V ( l ) ) ∥ ≤ ∥ e i ⊤ ( A ( l ) − P ( l ) ) V ^ ( l ) ∥ ∥ ( V ^ ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) − ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ( V ^ ( l ) ) ⊤ V ( l ) ∥ \begin{aligned}
\left\|\mathbf{e}_i^\top\mathbf{R}^{(l)}_5\right\|=\left\|\mathbf{e}_i^\top(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\widehat{\mathbf{V}}^{(l)}\left((\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}|\mathbf{\Lambda}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}-|\widehat{\mathbf{\Lambda}}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}\right)\right\| \\
&\leq\left\|\mathbf{e}_i^\top(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\widehat{\mathbf{V}}^{(l)}\right\|\left\|(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}|\mathbf{\Lambda}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}-|\widehat{\mathbf{\Lambda}}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}\right\|
\end{aligned} e i ⊤ R 5 ( l ) = e i ⊤ ( A ( l ) − P ( l ) ) V ( l ) ( ( V ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) − ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ( V ( l ) ) ⊤ V ( l ) ) ≤ e i ⊤ ( A ( l ) − P ( l ) ) V ( l ) ( V ( l ) ) ⊤ V ( l ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) − ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ( V ( l ) ) ⊤ V ( l )
≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ^ ( l ) ∥ 2 , ∞ K 2 ( σ K ( P ( l ) ) ) 3 2 ( K log ( n ) + θ max ( l ) ∥ θ ( l ) ∥ 1 σ K ( P ( l ) ) ) \lesssim\sqrt{\theta_i^{(l)}}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)\|\widehat{\mathbf{V}}^{(l)}\|_{2,\infty}\frac{K^2}{(\sigma_K(\mathbf{P}^{(l)}))^{\frac{3}{2}}}\left(\sqrt{K\log(n)}+\frac{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\sigma_K(\mathbf{P}^{(l)})}\right) ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ( l ) ∥ 2 , ∞ ( σ K ( P ( l ) ) ) 2 3 K 2 ( K log ( n ) + σ K ( P ( l ) ) θ m a x ( l ) ∥ θ ( l ) ∥ 1 )
≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ( l ) ∥ 2 , ∞ K 7 2 ∥ θ ( l ) ∥ 3 ( σ B ( l ) ) 3 2 ( K log ( n ) + K θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 2 σ B ( l ) ) \lesssim\sqrt{\theta_i^{(l)}}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)\|\mathbf{V}^{(l)}\|_{2,\infty}\frac{K^{\frac{7}{2}}}{\|\boldsymbol{\theta}^{(l)}\|^3(\sigma_B^{(l)})^{\frac{3}{2}}}\left(\sqrt{K\log(n)}+\frac{K\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\|\boldsymbol{\theta}^{(l)}\|^2\sigma_B^{(l)}}\right) ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ( l ) ∥ 2 , ∞ ∥ θ ( l ) ∥ 3 ( σ B ( l ) ) 2 3 K 2 7 ( K log ( n ) + ∥ θ ( l ) ∥ 2 σ B ( l ) K θ m a x ( l ) ∥ θ ( l ) ∥ 1 )
≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) K θ max ( l ) ∥ θ ( l ) ∥ K 7 2 ∥ θ ( l ) ∥ 3 ( σ B ( l ) ) 3 2 ( K log ( n ) + K θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 2 σ B ( l ) ) \lesssim\sqrt{\theta_i^{(l)}}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)\sqrt{K}\frac{\theta_{\max}^{(l)}}{\|\boldsymbol{\theta}^{(l)}\|}\frac{K^{\frac{7}{2}}}{\|\boldsymbol{\theta}^{(l)}\|^3(\sigma_B^{(l)})^{\frac{3}{2}}}\left(\sqrt{K\log(n)}+\frac{K\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\|\boldsymbol{\theta}^{(l)}\|^2\sigma_B^{(l)}}\right) ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) K ∥ θ ( l ) ∥ θ m a x ( l ) ∥ θ ( l ) ∥ 3 ( σ B ( l ) ) 2 3 K 2 7 ( K log ( n ) + ∥ θ ( l ) ∥ 2 σ B ( l ) K θ m a x ( l ) ∥ θ ( l ) ∥ 1 )
≲ K 4 θ max ( l ) θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 3 2 ( K log ( n ) + K θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 2 σ B ( l ) ) . \lesssim\frac{K^4\theta_{\max}^{(l)}\sqrt{\theta_i^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)}}{\|\boldsymbol{\theta}^{(l)}\|^4(\sigma_B^{(l)})^{\frac{3}{2}}}\left(\sqrt{K\log(n)}+\frac{K\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\|\boldsymbol{\theta}^{(l)}\|^2\sigma_B^{(l)}}\right). ≲ ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 2 3 K 4 θ m a x ( l ) θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ( K log ( n ) + ∥ θ ( l ) ∥ 2 σ B ( l ) K θ m a x ( l ) ∥ θ ( l ) ∥ 1 ) .
The term R 6 ( l ) \mathbf{R}^{(l)}_6 R 6 ( l )
Restate the term R 6 ( l ) = ( A ( l ) − P ( l ) ) V ^ ( l ) ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ( W ∗ ( l ) − ( V ^ ( l ) ) ⊤ V ( l ) ) \mathbf{R}^{(l)}_6 = (\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\widehat{\mathbf{V}}^{(l)}|\widehat{\mathbf{\Lambda}}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}\left(\mathbf{W}^{(l)}_*-(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}\right) R 6 ( l ) = ( A ( l ) − P ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ( W ∗ ( l ) − ( V ( l ) ) ⊤ V ( l ) ) for convenience.
∥ e i ⊤ R 6 ( l ) ∥ = ∥ e i ⊤ ( A ( l ) − P ( l ) ) V ^ ( l ) ∣ Λ ^ ( l ) ∣ − 1 2 I p , q ( l ) ( W ∗ ( l ) − ( V ^ ( l ) ) ⊤ V ( l ) ) ∥ ≤ ∥ e i ⊤ ( A ( l ) − P ( l ) ) V ^ ( l ) ∥ ∥ ∣ Λ ^ ( l ) ∣ − 1 2 ∥ ∥ W ∗ ( l ) − ( V ^ ( l ) ) ⊤ V ( l ) ∥ \begin{aligned}
\left\|\mathbf{e}_i^\top\mathbf{R}^{(l)}_6\right\|=\left\|\mathbf{e}_i^\top(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\widehat{\mathbf{V}}^{(l)}|\widehat{\mathbf{\Lambda}}^{(l)}|^{-\frac{1}{2}}\mathbf{I}^{(l)}_{p,q}\left(\mathbf{W}^{(l)}_*-(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}\right)\right\| \\
&\leq\left\|\mathbf{e}_i^\top(\mathbf{A}^{(l)}-\mathbf{P}^{(l)})\widehat{\mathbf{V}}^{(l)}\right\|\left\||\widehat{\mathbf{\Lambda}}^{(l)}|^{-\frac{1}{2}}\right\|\left\|\mathbf{W}^{(l)}_*-(\widehat{\mathbf{V}}^{(l)})^\top\mathbf{V}^{(l)}\right\|
\end{aligned} e i ⊤ R 6 ( l ) = e i ⊤ ( A ( l ) − P ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 2 1 I p , q ( l ) ( W ∗ ( l ) − ( V ( l ) ) ⊤ V ( l ) ) ≤ e i ⊤ ( A ( l ) − P ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 2 1 W ∗ ( l ) − ( V ( l ) ) ⊤ V ( l )
≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ^ ( l ) ∥ 2 , ∞ 1 ( σ K ( P ( l ) ) ) 1 2 θ max ( l ) ∥ θ ( l ) ∥ 1 ( σ K ( P ( l ) ) ) 2 \lesssim\sqrt{\theta_i^{(l)}}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)\|\widehat{\mathbf{V}}^{(l)}\|_{2,\infty}\frac{1}{(\sigma_K(\mathbf{P}^{(l)}))^{\frac{1}{2}}}\frac{\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{(\sigma_K(\mathbf{P}^{(l)}))^2} ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ( l ) ∥ 2 , ∞ ( σ K ( P ( l ) ) ) 2 1 1 ( σ K ( P ( l ) ) ) 2 θ m a x ( l ) ∥ θ ( l ) ∥ 1
≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ( l ) ∥ 2 , ∞ K 5 2 θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 5 ( σ B ( l ) ) 5 2 \lesssim\sqrt{\theta_i^{(l)}}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)\|\mathbf{V}^{(l)}\|_{2,\infty}\frac{K^{\frac{5}{2}}\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\|\boldsymbol{\theta}^{(l)}\|^5(\sigma_B^{(l)})^{\frac{5}{2}}} ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ( l ) ∥ 2 , ∞ ∥ θ ( l ) ∥ 5 ( σ B ( l ) ) 2 5 K 2 5 θ m a x ( l ) ∥ θ ( l ) ∥ 1
≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) K θ max ( l ) ∥ θ ( l ) ∥ K 5 2 θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 5 ( σ B ( l ) ) 5 2 \lesssim\sqrt{\theta_i^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)}\sqrt{K}\frac{\theta_{\max}^{(l)}}{\|\boldsymbol{\theta}^{(l)}\|}\frac{K^{\frac{5}{2}}\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\|\boldsymbol{\theta}^{(l)}\|^5(\sigma_B^{(l)})^{\frac{5}{2}}} ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) K ∥ θ ( l ) ∥ θ m a x ( l ) ∥ θ ( l ) ∥ 5 ( σ B ( l ) ) 2 5 K 2 5 θ m a x ( l ) ∥ θ ( l ) ∥ 1
≲ θ i ( l ) log ( n ) ∥ θ ( l ) ∥ 3 2 ( θ max ( l ) ) 2 K 3 ∥ θ ( l ) ∥ 6 ( σ B ( l ) ) 5 2 . \lesssim\frac{\sqrt{\theta_i^{(l)}\log(n)}\|\boldsymbol{\theta}^{(l)}\|^{\frac{3}{2}}(\theta_{\max}^{(l)})^2K^3}{\|\boldsymbol{\theta}^{(l)}\|^6(\sigma_B^{(l)})^{\frac{5}{2}}}. ≲ ∥ θ ( l ) ∥ 6 ( σ B ( l ) ) 2 5 θ i ( l ) log ( n ) ∥ θ ( l ) ∥ 2 3 ( θ m a x ( l ) ) 2 K 3 .
Combining the bounds for R 1 ( l ) , … , R 6 ( l ) \mathbf{R}^{(l)}_1,\ldots,\mathbf{R}^{(l)}_6 R 1 ( l ) , … , R 6 ( l ) , on the event E l r e s \mathcal{E}^{\mathrm{res}}_l E l res , we obtain
∥ e i ⊤ R X ( l ) ∥ ≲ θ i ( l ) K 5 2 θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 2 + θ i ( l ) ( θ max ( l ) θ min ( l ) ) 1 2 ( K 9 2 θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 3 2 ) . (18) \left\|\mathbf{e}_i^\top\mathbf{R}^{(l)}_{\mathsf{X}}\right\|\lesssim\theta_i^{(l)}\frac{K^{\frac{5}{2}}\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\|\boldsymbol{\theta}^{(l)}\|^4(\sigma_B^{(l)})^2}+\theta_i^{(l)}\left(\frac{\theta_{\max}^{(l)}}{\theta_{\min}^{(l)}}\right)^{\frac{1}{2}}\left(\frac{K^{\frac{9}{2}}\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)}{\|\boldsymbol{\theta}^{(l)}\|^4(\sigma_B^{(l)})^{\frac{3}{2}}}\right). \tag*{(18)} e i ⊤ R X ( l ) ≲ θ i ( l ) ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 2 K 2 5 θ m a x ( l ) ∥ θ ( l ) ∥ 1 + θ i ( l ) ( θ m i n ( l ) θ m a x ( l ) ) 2 1 ( ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 2 3 K 2 9 θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ) . (18)
Denote
α R ( l ) = K 5 2 θ max ( l ) ∥ θ ( l ) ∥ 1 ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 2 + ( θ max ( l ) θ min ( l ) ) 1 2 ( K 9 2 θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 3 2 ) . \alpha_{\mathbf{R}}^{(l)}=\frac{K^{\frac{5}{2}}\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1}{\|\boldsymbol{\theta}^{(l)}\|^4(\sigma_B^{(l)})^2}+\left(\frac{\theta_{\max}^{(l)}}{\theta_{\min}^{(l)}}\right)^{\frac{1}{2}}\left(\frac{K^{\frac{9}{2}}\theta_{\max}^{(l)}\|\boldsymbol{\theta}^{(l)}\|_1\log(n)}{\|\boldsymbol{\theta}^{(l)}\|^4(\sigma_B^{(l)})^{\frac{3}{2}}}\right). α R ( l ) = ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 2 K 2 5 θ m a x ( l ) ∥ θ ( l ) ∥ 1 + ( θ m i n ( l ) θ m a x ( l ) ) 2 1 ( ∥ θ ( l ) ∥ 4 ( σ B ( l ) ) 2 3 K 2 9 θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ) .
Then, ∥ e i T R X ( l ) ∥ 2 ≲ θ i ( l ) α R ( l ) \lVert\mathbf{e}_i^{\mathsf{T}}\mathbf{R}_{X}^{(l)}\rVert_2 \lesssim\theta_i^{(l)}\alpha_R^{(l)} ∥ e i T R X ( l ) ∥ 2 ≲ θ i ( l ) α R ( l ) .
Closeness of X ^ ( l ) \widehat{X}^{(l)} X ( l ) to X ( l ) X^{(l)} X ( l )
For each l ∈ [ L ] l \in[L] l ∈ [ L ] and i ∈ [ n ] i \in[n] i ∈ [ n ] , let E X , i ( l ) \mathcal{E}_{X,i}^{(l)} E X , i ( l ) denote the event
{ ∥ e i T ( A ( l ) − P ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 1 / 2 I p , q ( l ) ∥ 2 ≤ C θ i ( l ) ∥ θ ( l ) ∥ 1 log n σ K ( P ( l ) ) ∥ V ( l ) ∥ 2 , ∞ } . \left\{\left\lVert\mathbf{e}_i^{\mathsf{T}}\left(A^{(l)}-P^{(l)}\right)V^{(l)}\left|\Lambda^{(l)}\right|^{-1/2}I_{p,q}^{(l)}\right\rVert_2 \le C\frac{\sqrt{\theta_i^{(l)}}\lVert\boldsymbol{\theta}^{(l)}\rVert_1\log n}{\sqrt{\sigma_K(P^{(l)})}}\lVert V^{(l)}\rVert_{2,\infty}\right\}. ⎩ ⎨ ⎧ e i T ( A ( l ) − P ( l ) ) V ( l ) Λ ( l ) − 1/2 I p , q ( l ) 2 ≤ C σ K ( P ( l ) ) θ i ( l ) ∥ θ ( l ) ∥ 1 log n ∥ V ( l ) ∥ 2 , ∞ ⎭ ⎬ ⎫ .
By a Bernstein inequality for sums of independent random vectors,
P ( E X , i ( l ) ) ≥ 1 − C n − 20 . \mathbb{P}\left(\mathcal{E}_{X,i}^{(l)}\right)\ge1-Cn^{-20}. P ( E X , i ( l ) ) ≥ 1 − C n − 20 .
Define
E X ( l ) : = ⋂ i = 1 n E X , i ( l ) . \mathcal{E}_{X}^{(l)}:=\bigcap_{i=1}^{n}\mathcal{E}_{X,i}^{(l)}. E X ( l ) := i = 1 ⋂ n E X , i ( l ) .
Then
P ( E X ( l ) ) ≥ 1 − C n − 19 . \mathbb{P}\left(\mathcal{E}_{X}^{(l)}\right)\ge1-Cn^{-19}. P ( E X ( l ) ) ≥ 1 − C n − 19 .
Let E ( l ) : = E r e s ( l ) ∩ E X ( l ) \mathcal{E}^{(l)}:=\mathcal{E}_{\mathrm{res}}^{(l)}\cap\mathcal{E}_{X}^{(l)} E ( l ) := E res ( l ) ∩ E X ( l ) . Then
P ( E ( l ) ) ≥ 1 − C n − 18 . \mathbb{P}\left(\mathcal{E}^{(l)}\right)\ge1-Cn^{-18}. P ( E ( l ) ) ≥ 1 − C n − 18 .
In the rest of this subsection and in the normalized-embedding subsection, all inequalities are understood to hold on E ( l ) \mathcal{E}^{(l)} E ( l ) .
Note that
e i T ( X ^ ( l ) W ∗ ( l ) − X ( l ) ) = e i T ( A ( l ) − P ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 1 / 2 I p , q ( l ) + e i T R X ( l ) . (19) \mathbf{e}_i^{\mathsf{T}}\left(\widehat{X}^{(l)}W_*^{(l)}-X^{(l)}\right)
=\mathbf{e}_i^{\mathsf{T}}\left(A^{(l)}-P^{(l)}\right)V^{(l)}
\left|\Lambda^{(l)}\right|^{-1/2}I_{p,q}^{(l)}
+\mathbf{e}_i^{\mathsf{T}}R_X^{(l)}. \tag*{(19)} e i T ( X ( l ) W ∗ ( l ) − X ( l ) ) = e i T ( A ( l ) − P ( l ) ) V ( l ) Λ ( l ) − 1/2 I p , q ( l ) + e i T R X ( l ) . (19)
and we have ∥ e i T R X ( l ) ∥ 2 ≲ θ i ( l ) α R ( l ) \lVert\mathbf{e}_i^{\mathsf{T}}R_X^{(l)}\rVert_2\lesssim\theta_i^{(l)}\alpha_R^{(l)} ∥ e i T R X ( l ) ∥ 2 ≲ θ i ( l ) α R ( l ) from Section B.2. Now, we bound the main term e i T ( A ( l ) − P ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 1 / 2 I p , q ( l ) \mathbf{e}_i^{\mathsf{T}}\left(A^{(l)}-P^{(l)}\right)V^{(l)}\left|\Lambda^{(l)}\right|^{-1/2}I_{p,q}^{(l)} e i T ( A ( l ) − P ( l ) ) V ( l ) Λ ( l ) − 1/2 I p , q ( l ) . Matrix Bernstein inequality gives
∥ e i T ( A ( l ) − P ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 1 / 2 I p , q ( l ) ∥ 2 = ∥ ∑ j = 1 n ( A i j ( l ) − P i j ( l ) ) ( V ( l ) ∣ Λ ( l ) ∣ − 1 / 2 I p , q ( l ) ) j ⋅ ∥ 2 ≲ θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ( σ K ( P ( l ) ) ) 1 / 2 ∥ V ( l ) ∥ 2 , ∞ ≲ K θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 1 ( σ B ( l ) ) 1 / 2 K θ max ( l ) ∥ θ ( l ) ∥ ≲ K θ max ( l ) θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 2 ( σ B ( l ) ) 1 / 2 ≲ θ i ( l ) ( θ max ( l ) θ i ( l ) ) 1 / 2 K θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 2 ( σ B ( l ) ) 1 / 2 . (20) \begin{aligned}
\left\lVert\mathbf{e}_i^{\mathsf{T}}\left(A^{(l)}-P^{(l)}\right)V^{(l)}\left|\Lambda^{(l)}\right|^{-1/2}I_{p,q}^{(l)}\right\rVert_2
&=\left\lVert\sum_{j=1}^{n}\left(A_{ij}^{(l)}-P_{ij}^{(l)}\right)\left(V^{(l)}\left|\Lambda^{(l)}\right|^{-1/2}I_{p,q}^{(l)}\right)_{j\cdot}\right\rVert_2\\
&\lesssim\frac{\sqrt{\theta_i^{(l)}}\lVert\boldsymbol{\theta}^{(l)}\rVert_1\log(n)}{\left(\sigma_K(P^{(l)})\right)^{1/2}}\lVert V^{(l)}\rVert_{2,\infty}\\
&\lesssim\frac{\sqrt{K\theta_i^{(l)}}\lVert\boldsymbol{\theta}^{(l)}\rVert_1\log(n)}{\lVert\boldsymbol{\theta}^{(l)}\rVert_1\left(\sigma_B^{(l)}\right)^{1/2}}\sqrt{K}\frac{\theta_{\max}^{(l)}}{\lVert\boldsymbol{\theta}^{(l)}\rVert}\\
&\lesssim\frac{K\theta_{\max}^{(l)}\sqrt{\theta_i^{(l)}}\lVert\boldsymbol{\theta}^{(l)}\rVert_1\log(n)}{\lVert\boldsymbol{\theta}^{(l)}\rVert^2\left(\sigma_B^{(l)}\right)^{1/2}}\\
&\lesssim\theta_i^{(l)}\left(\frac{\theta_{\max}^{(l)}}{\theta_i^{(l)}}\right)^{1/2}
\frac{K\sqrt{\theta_{\max}^{(l)}}\lVert\boldsymbol{\theta}^{(l)}\rVert_1\log(n)}
{\lVert\boldsymbol{\theta}^{(l)}\rVert^2\left(\sigma_B^{(l)}\right)^{1/2}}. \tag*{(20)}
\end{aligned} e i T ( A ( l ) − P ( l ) ) V ( l ) Λ ( l ) − 1/2 I p , q ( l ) 2 = j = 1 ∑ n ( A ij ( l ) − P ij ( l ) ) ( V ( l ) Λ ( l ) − 1/2 I p , q ( l ) ) j ⋅ 2 ≲ ( σ K ( P ( l ) ) ) 1/2 θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ V ( l ) ∥ 2 , ∞ ≲ ∥ θ ( l ) ∥ 1 ( σ B ( l ) ) 1/2 K θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) K ∥ θ ( l ) ∥ θ m a x ( l ) ≲ ∥ θ ( l ) ∥ 2 ( σ B ( l ) ) 1/2 K θ m a x ( l ) θ i ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ≲ θ i ( l ) ( θ i ( l ) θ m a x ( l ) ) 1/2 ∥ θ ( l ) ∥ 2 ( σ B ( l ) ) 1/2 K θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) . (20)
Combining (B.10) and (B.12),
∥ e i T ( X ^ ( l ) W ∗ ( l ) − X ( l ) ) ∥ 2 = ∥ e i T ( A ( l ) − P ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) + e i T R X ( l ) ∥ 2 ≤ ∥ e i T ( A ( l ) − P ( l ) ) V ( l ) ∣ Λ ( l ) ∣ − 1 2 I p , q ( l ) ∥ 2 + ∥ e i T R X ( l ) ∥ 2 ≲ θ i ( l ) ( ( θ max ( l ) θ i ( l ) ) 1 2 K θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 2 2 ( σ B ( l ) ) 1 2 + α R ( l ) ) ≲ θ i ( l ) σ B ( l ) K ( ( θ max ( l ) θ i ( l ) ) 1 2 K 2 θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ∥ θ ( l ) ∥ 2 2 ( σ B ( l ) ) 3 2 + K σ B ( l ) α R ( l ) ) . (21) \begin{aligned}
\left\|e_i^\mathsf{T}\left(\widehat{X}^{(l)}W_*^{(l)}-X^{(l)}\right)\right\|_2
&=\left\|e_i^\mathsf{T}\left(A^{(l)}-P^{(l)}\right)V^{(l)}\left|\Lambda^{(l)}\right|^{-\frac{1}{2}}I_{p,q}^{(l)}+e_i^\mathsf{T}R_X^{(l)}\right\|_2\\
&\leq\left\|e_i^\mathsf{T}\left(A^{(l)}-P^{(l)}\right)V^{(l)}\left|\Lambda^{(l)}\right|^{-\frac{1}{2}}I_{p,q}^{(l)}\right\|_2+\left\|e_i^\mathsf{T}R_X^{(l)}\right\|_2\\
&\lesssim\theta_i^{(l)}\left(\left(\frac{\theta_{\max}^{(l)}}{\theta_i^{(l)}}\right)^{\frac{1}{2}}\frac{K\sqrt{\theta_{\max}^{(l)}}\|\theta^{(l)}\|_1\log(n)}{\|\theta^{(l)}\|_2^2(\sigma_B^{(l)})^{\frac{1}{2}}}+\alpha_R^{(l)}\right)\\
&\lesssim\theta_i^{(l)}\frac{\sigma_B^{(l)}}{K}\left(\left(\frac{\theta_{\max}^{(l)}}{\theta_i^{(l)}}\right)^{\frac{1}{2}}\frac{K^2\sqrt{\theta_{\max}^{(l)}}\|\theta^{(l)}\|_1\log(n)}{\|\theta^{(l)}\|_2^2(\sigma_B^{(l)})^{\frac{3}{2}}}+\frac{K}{\sigma_B^{(l)}}\alpha_R^{(l)}\right). \tag*{(21)}
\end{aligned} e i T ( X ( l ) W ∗ ( l ) − X ( l ) ) 2 = e i T ( A ( l ) − P ( l ) ) V ( l ) Λ ( l ) − 2 1 I p , q ( l ) + e i T R X ( l ) 2 ≤ e i T ( A ( l ) − P ( l ) ) V ( l ) Λ ( l ) − 2 1 I p , q ( l ) 2 + e i T R X ( l ) 2 ≲ θ i ( l ) ( θ i ( l ) θ m a x ( l ) ) 2 1 ∥ θ ( l ) ∥ 2 2 ( σ B ( l ) ) 2 1 K θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) + α R ( l ) ≲ θ i ( l ) K σ B ( l ) ( θ i ( l ) θ m a x ( l ) ) 2 1 ∥ θ ( l ) ∥ 2 2 ( σ B ( l ) ) 2 3 K 2 θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) + σ B ( l ) K α R ( l ) . (21)
We rewrite (B.13) in terms of β n ( l ) \beta_n^{(l)} β n ( l ) . Recall that
β n ( l ) : = ( θ max ( l ) θ min ( l ) ) K 8 θ max ( l ) ∥ θ ( l ) ∥ 1 log ( n ) ( σ B ( l ) ) 3 ∥ θ ( l ) ∥ 1 . \beta_n^{(l)}:=\left(\frac{\theta_{\max}^{(l)}}{\theta_{\min}^{(l)}}\right)\frac{K^8\theta_{\max}^{(l)}\|\theta^{(l)}\|_1\log(n)}{(\sigma_B^{(l)})^3\|\theta^{(l)}\|_1}. β n ( l ) := ( θ m i n ( l ) θ m a x ( l ) ) ( σ B ( l ) ) 3 ∥ θ ( l ) ∥ 1 K 8 θ m a x ( l ) ∥ θ ( l ) ∥ 1 log ( n ) .
We bound the first term in (B.13):
( θ max ( l ) θ i ( l ) ) 1 2 K 2 θ max ( l ) ∥ θ ( l ) ∥ 1 log n ∥ θ ( l ) ∥ 2 2 ( σ B ( l ) ) 3 / 2 ≤ 1 K 2 β n ( l ) . (22) \left(\frac{\theta_{\max}^{(l)}}{\theta_i^{(l)}}\right)^{\frac{1}{2}}\frac{K^2\sqrt{\theta_{\max}^{(l)}}\|\theta^{(l)}\|_1\log n}{\|\theta^{(l)}\|_2^2(\sigma_B^{(l)})^{3/2}}\leq\frac{1}{K^2}\sqrt{\beta_n^{(l)}}. \tag*{(22)} ( θ i ( l ) θ m a x ( l ) ) 2 1 ∥ θ ( l ) ∥ 2 2 ( σ B ( l ) ) 3/2 K 2 θ m a x ( l ) ∥ θ ( l ) ∥ 1 log n ≤ K 2 1 β n ( l ) . (22)
Next, we bound the second term in (B.13):
K σ B ( l ) α R ( l ) = β n ( l ) [ ( θ min ( l ) θ max ( l ) ) 1 K 9 / 2 log n + ( θ min ( l ) θ max ( l ) ) 1 / 2 ( σ B ( l ) ) 1 / 2 K 5 / 2 ] ≲ β n ( l ) . (23) \frac{K}{\sigma_B^{(l)}}\alpha_R^{(l)}=\beta_n^{(l)}\left[\left(\frac{\theta_{\min}^{(l)}}{\theta_{\max}^{(l)}}\right)\frac{1}{K^{9/2}\log n}+\left(\frac{\theta_{\min}^{(l)}}{\theta_{\max}^{(l)}}\right)^{1/2}\frac{(\sigma_B^{(l)})^{1/2}}{K^{5/2}}\right]\lesssim\beta_n^{(l)}. \tag*{(23)} σ B ( l ) K α R ( l ) = β n ( l ) ( θ m a x ( l ) θ m i n ( l ) ) K 9/2 log n 1 + ( θ m a x ( l ) θ m i n ( l ) ) 1/2 K 5/2 ( σ B ( l ) ) 1/2 ≲ β n ( l ) . (23)
Substituting (B.14) and (B.15 into (B.13), we obtain
∥ e i T ( X ^ ( l ) W ∗ ( l ) − X ( l ) ) ∥ 2 ≲ θ i ( l ) σ B ( l ) K ( β n ( l ) + β n ( l ) ) . \left\|e_i^\mathsf{T}\left(\widehat{X}^{(l)}W_*^{(l)}-X^{(l)}\right)\right\|_2\lesssim\theta_i^{(l)}\frac{\sigma_B^{(l)}}{K}\left(\sqrt{\beta_n^{(l)}}+\beta_n^{(l)}\right). e i T ( X ( l ) W ∗ ( l ) − X ( l ) ) 2 ≲ θ i ( l ) K σ B ( l ) ( β n ( l ) + β n ( l ) ) .
If, in addition,
max 1 ≤ l ≤ L β n ( l ) ≤ C β \max_{1\leq l\leq L}\beta_n^{(l)}\leq C_\beta 1 ≤ l ≤ L max β n ( l ) ≤ C β
for some fixed constant C β > 0 C_\beta>0 C β > 0 , then
β n ( l ) ≤ C β β n ( l ) , \beta_n^{(l)}\leq\sqrt{C_\beta}\sqrt{\beta_n^{(l)}}, β n ( l ) ≤ C β β n ( l ) ,
and hence
β n ( l ) + β n ( l ) ≤ ( 1 + C β ) β n ( l ) . \sqrt{\beta_n^{(l)}}+\beta_n^{(l)}\leq(1+\sqrt{C_\beta})\sqrt{\beta_n^{(l)}}. β n ( l ) + β n ( l ) ≤ ( 1 + C β ) β n ( l ) .
Therefore, on E l \mathcal{E}_l E l , there exists a constant C X > 0 C_X>0 C X > 0 , depending only on C β C_\beta C β and fixed model constants, such that for all i ∈ [ n ] i\in[n] i ∈ [ n ] ,
∥ e i T ( X ^ ( l ) W ∗ ( l ) − X ( l ) ) ∥ 2 ≤ C X θ i ( l ) σ B ( l ) K β n ( l ) . (24) \left\|e_i^\mathsf{T}\left(\widehat{X}^{(l)}W_*^{(l)}-X^{(l)}\right)\right\|_2\leq C_X\theta_i^{(l)}\frac{\sigma_B^{(l)}}{K}\sqrt{\beta_n^{(l)}}. \tag*{(24)} e i T ( X ( l ) W ∗ ( l ) − X ( l ) ) 2 ≤ C X θ i ( l ) K σ B ( l ) β n ( l ) . (24)
Closeness of Y ^ ( l ) \widehat{\mathbf{Y}}^{(l)} Y ( l ) to Y ( l ) \mathbf{Y}^{(l)} Y ( l )
Recall that
Y i , ⋅ ( l ) = X i , ⋅ ( l ) ∥ X i , ⋅ ( l ) ∥ 2 , Y ^ i , ⋅ ( l ) = X ^ i , ⋅ ( l ) ∥ X ^ i , ⋅ ( l ) ∥ 2 . \mathbf{Y}_{i,\cdot}^{(l)} = \frac{\mathbf{X}_{i,\cdot}^{(l)}}{\left\lVert\mathbf{X}_{i,\cdot}^{(l)} \right\rVert_2}, \qquad\widehat{\mathbf{Y}}_{i,\cdot}^{(l)} = \frac{\widehat{\mathbf{X}}_{i,\cdot}^{(l)}}{\left\lVert\widehat{\mathbf{X}}_{i,\cdot}^{(l)} \right\rVert_2}. Y i , ⋅ ( l ) = X i , ⋅ ( l ) 2 X i , ⋅ ( l ) , Y i , ⋅ ( l ) = X i , ⋅ ( l ) 2 X i , ⋅ ( l ) .
Let
f ( x ) : = x ∥ x ∥ 2 , x ∈ R K ∖ { 0 } . f(\mathbf{x}) := \frac{\mathbf{x}}{\left\lVert\mathbf{x} \right\rVert_2}, \qquad\mathbf{x} \in\mathbb{R}^{K} \setminus\{0\}. f ( x ) := ∥ x ∥ 2 x , x ∈ R K ∖ { 0 } .
Since W ∗ ( l ) \mathbf{W}_{*}^{(l)} W ∗ ( l ) is orthogonal, we have
f ( x W ∗ ( l ) ) = f ( x ) W ∗ ( l ) f(\mathbf{x}\mathbf{W}_{*}^{(l)}) = f(\mathbf{x})\mathbf{W}_{*}^{(l)} f ( x W ∗ ( l ) ) = f ( x ) W ∗ ( l )
for every x ∈ R K ∖ { 0 } \mathbf{x} \in\mathbb{R}^{K} \setminus\{0\} x ∈ R K ∖ { 0 } . Hence
( Y ^ ( l ) W ∗ ( l ) ) i , ⋅ = f ( ( X ^ ( l ) W ∗ ( l ) ) i , ⋅ ) . \left(\widehat{\mathbf{Y}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot} = f\left(\left(\widehat{\mathbf{X}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot}\right). ( Y ( l ) W ∗ ( l ) ) i , ⋅ = f ( ( X ( l ) W ∗ ( l ) ) i , ⋅ ) .
We use the elementary inequality
∥ f ( x ^ ) − f ( x ) ∥ 2 ≤ 2 ∥ x ^ − x ∥ 2 ∥ x ∥ 2 , x , x ^ ∈ R K ∖ { 0 } . (25) \left\lVert f(\widehat{\mathbf{x}})-f(\mathbf{x})\right\rVert_2 \leq\frac{2\left\lVert\widehat{\mathbf{x}}-\mathbf{x}\right\rVert_2}{\left\lVert\mathbf{x}\right\rVert_2}, \qquad\mathbf{x},\widehat{\mathbf{x}} \in\mathbb{R}^{K} \setminus\{0\}. \tag*{(25)} ∥ f ( x ) − f ( x ) ∥ 2 ≤ ∥ x ∥ 2 2 ∥ x − x ∥ 2 , x , x ∈ R K ∖ { 0 } . (25)
Applying (B.17) with
x = X i , ⋅ ( l ) , x ^ = ( X ^ ( l ) W ∗ ( l ) ) i , ⋅ , \mathbf{x} = \mathbf{X}_{i,\cdot}^{(l)}, \qquad\widehat{\mathbf{x}} = \left(\widehat{\mathbf{X}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot}, x = X i , ⋅ ( l ) , x = ( X ( l ) W ∗ ( l ) ) i , ⋅ ,
we obtain
∥ ( Y ^ ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) ∥ 2 ≤ 2 ∥ ( X ^ ( l ) W ∗ ( l ) ) i , ⋅ − X i , ⋅ ( l ) ∥ 2 ∥ X i , ⋅ ( l ) ∥ 2 . (26) \left\lVert\left(\widehat{\mathbf{Y}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot} - \mathbf{Y}_{i,\cdot}^{(l)} \right\rVert_2 \leq\frac{2\left\lVert\left(\widehat{\mathbf{X}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot} - \mathbf{X}_{i,\cdot}^{(l)} \right\rVert_2}{\left\lVert\mathbf{X}_{i,\cdot}^{(l)} \right\rVert_2}. \tag*{(26)} ( Y ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) 2 ≤ X i , ⋅ ( l ) 2 2 ( X ( l ) W ∗ ( l ) ) i , ⋅ − X i , ⋅ ( l ) 2 . (26)
By Lemma B.1 , there exists a constant c X > 0 c_X > 0 c X > 0 such that
∥ X i , ⋅ ( l ) ∥ 2 ≥ c X θ i ( l ) σ B ( l ) K . (27) \left\lVert\mathbf{X}_{i,\cdot}^{(l)} \right\rVert_2 \geq c_X \theta_i^{(l)} \frac{\sigma_B^{(l)}}{K}. \tag*{(27)} X i , ⋅ ( l ) 2 ≥ c X θ i ( l ) K σ B ( l ) . (27)
Moreover, by (B.16), there exists a constant C X > 0 C_X > 0 C X > 0 such that
∥ ( X ^ ( l ) W ∗ ( l ) ) i , ⋅ − X i , ⋅ ( l ) ∥ 2 ≤ C X θ i ( l ) σ B ( l ) K β n ( l ) . \left\lVert\left(\widehat{\mathbf{X}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot} - \mathbf{X}_{i,\cdot}^{(l)} \right\rVert_2 \leq C_X \theta_i^{(l)} \frac{\sigma_B^{(l)}}{K}\sqrt{\beta_n^{(l)}}. ( X ( l ) W ∗ ( l ) ) i , ⋅ − X i , ⋅ ( l ) 2 ≤ C X θ i ( l ) K σ B ( l ) β n ( l ) .
Substituting this and (B.19) into (B.18), we obtain
∥ ( Y ^ ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) ∥ 2 ≤ 2 C X c X β n ( l ) . \left\lVert\left(\widehat{\mathbf{Y}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot} - \mathbf{Y}_{i,\cdot}^{(l)} \right\rVert_2 \leq\frac{2C_X}{c_X}\sqrt{\beta_n^{(l)}}. ( Y ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) 2 ≤ c X 2 C X β n ( l ) .
Define
C Y : = 2 C X c X . C_Y := \frac{2C_X}{c_X}. C Y := c X 2 C X .
Then
∥ ( Y ^ ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) ∥ 2 ≤ C Y β n ( l ) . (28) \left\lVert\left(\widehat{\mathbf{Y}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot} - \mathbf{Y}_{i,\cdot}^{(l)} \right\rVert_2 \leq C_Y\sqrt{\beta_n^{(l)}}. \tag*{(28)} ( Y ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) 2 ≤ C Y β n ( l ) . (28)
Since Y i , ⋅ ( l ) \mathbf{Y}_{i,\cdot}^{(l)} Y i , ⋅ ( l ) and ( Y ^ ( l ) W ∗ ( l ) ) i , ⋅ \left(\widehat{\mathbf{Y}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot} ( Y ( l ) W ∗ ( l ) ) i , ⋅ both lie on the unit sphere,
∥ ( Y ^ ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) ∥ 2 ≤ 2. \left\lVert\left(\widehat{\mathbf{Y}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot} - \mathbf{Y}_{i,\cdot}^{(l)} \right\rVert_2 \leq2. ( Y ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) 2 ≤ 2.
Combining this with (B.20), on the same event E ( l ) \mathcal{E}^{(l)} E ( l ) , we conclude that
∥ ( Y ^ ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) ∥ 2 ≤ min { C Y β n ( l ) , 2 } . (29) \left\lVert\left(\widehat{\mathbf{Y}}^{(l)}\mathbf{W}_{*}^{(l)}\right)_{i,\cdot} - \mathbf{Y}_{i,\cdot}^{(l)} \right\rVert_2 \leq\min\left\{C_Y\sqrt{\beta_n^{(l)}},\,2\right\}. \tag*{(29)} ( Y ( l ) W ∗ ( l ) ) i , ⋅ − Y i , ⋅ ( l ) 2 ≤ min { C Y β n ( l ) , 2 } . (29)
Population Embedding Representation and Identifiability
This section collects auxiliary results for the identifiability analysis of the MLDCMM model. We begin with Proposition 2.2, which expresses the population spectral embeddings in terms of the model parameters. Next, we show that the reference-layer ℓ 2 \ell_2 ℓ 2 -unit-row-norm condition of R ( r e f ) \mathbf{R}^{(\mathrm{ref})} R ( ref ) in Condition 1 can always be achieved through a canonical reparametrization, and hence should be viewed as a normalization rather than an additional model restriction. With these results, we prove the identifiability of MLDCMM (Proposition 2.3).
Model-Based Representation of Population Embeddings
Although the population spectral embedding is defined as
X ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1 / 2 , \mathbf{X}^{(l)}=\mathbf{V}^{(l)}\lvert\mathbf{\Lambda}^{(l)}\rvert^{1/2}, X ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1/2 ,
via the eigendecomposition of P ( l ) \mathbf{P}^{(l)} P ( l ) , it can also be represented directly in terms of the MLDCMM parameters. Specifically, we show that
X ( l ) = Θ ( l ) Z R ( l ) . \mathbf{X}^{(l)}=\mathbf{\Theta}^{(l)}\mathbf{Z}\mathbf{R}^{(l)}. X ( l ) = Θ ( l ) Z R ( l ) .
for some invertible matrix R ( l ) ∈ R K × K \mathbf{R}^{(l)}\in\mathbb{R}^{K\times K} R ( l ) ∈ R K × K . This model-based representation links the spectral embedding to the membership matrix, and plays a key role both in the identifiability argument and in characterizing the spherical simplex structure induced by the embeddings. We prove this representation in Proposition 2.2.
Proposition 2.2 (Spectral representation of the ML-DCMM). Consider the ML-DCMM model in Equation (2) . Let X ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1 / 2 \mathbf{X}^{(l)}=\mathbf{V}^{(l)}\lvert\mathbf{\Lambda}^{(l)}\rvert^{1/2} X ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1/2 be the population spectral embedding of P ( l ) \mathbf{P}^{(l)} P ( l ) , and let I p ( l ) , q ( l ) = sgn ( Λ ( l ) ) \mathbf{I}_{p^{(l)},q^{(l)}}=\operatorname{sgn}(\mathbf{\Lambda}^{(l)}) I p ( l ) , q ( l ) = sgn ( Λ ( l ) ) be the signature matrix associated with the ordered eigendecomposition above. Then there exists a matrix R ( l ) ∈ R K × K \mathbf{R}^{(l)}\in\mathbb{R}^{K\times K} R ( l ) ∈ R K × K such that
X ( l ) = Θ ( l ) Z R ( l ) (30) \mathbf{X}^{(l)}=\mathbf{\Theta}^{(l)}\mathbf{Z}\mathbf{R}^{(l)} \tag*{(30)} X ( l ) = Θ ( l ) Z R ( l ) (30) Moreover, with the same choice of R ( l ) \mathbf{R}^{(l)} R ( l ) , it follows that
B ( l ) = R ( l ) I p ( l ) , q ( l ) R ( l ) ⊤ . (31) \mathbf{B}^{(l)}=\mathbf{R}^{(l)}\mathbf{I}_{p^{(l)},q^{(l)}}\mathbf{R}^{(l)\top}. \tag*{(31)} B ( l ) = R ( l ) I p ( l ) , q ( l ) R ( l ) ⊤ . (31)
Proof. By the eigenvector representation in (B.7), there exists an orthogonal matrix O ( l ) \mathbf{O}^{(l)} O ( l ) satisfying
O ( l ) Λ ( l ) O ( l ) ⊤ = Λ ( l ) \mathbf{O}^{(l)}\mathbf{\Lambda}^{(l)}\mathbf{O}^{(l)\top}=\mathbf{\Lambda}^{(l)} O ( l ) Λ ( l ) O ( l ) ⊤ = Λ ( l )
such that
V ( l ) = Θ ( l ) Z ( Z ⊤ ( Θ ( l ) ) 2 Z ) − 1 / 2 U ~ ( l ) O ( l ) . \mathbf{V}^{(l)}=\mathbf{\Theta}^{(l)}\mathbf{Z}\left(\mathbf{Z}^{\top}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-1/2}\widetilde{\mathbf{U}}^{(l)}\mathbf{O}^{(l)}. V ( l ) = Θ ( l ) Z ( Z ⊤ ( Θ ( l ) ) 2 Z ) − 1/2 U ( l ) O ( l ) .
Define
R ( l ) : = ( Z ⊤ ( Θ ( l ) ) 2 Z ) − 1 / 2 U ~ ( l ) O ( l ) ∣ Λ ( l ) ∣ 1 / 2 . \mathbf{R}^{(l)}:=\left(\mathbf{Z}^{\top}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-1/2}\widetilde{\mathbf{U}}^{(l)}\mathbf{O}^{(l)}\lvert\mathbf{\Lambda}^{(l)}\rvert^{1/2}. R ( l ) := ( Z ⊤ ( Θ ( l ) ) 2 Z ) − 1/2 U ( l ) O ( l ) ∣ Λ ( l ) ∣ 1/2 .
Then
X ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1 / 2 = Θ ( l ) Z R ( l ) . \mathbf{X}^{(l)}=\mathbf{V}^{(l)}\lvert\mathbf{\Lambda}^{(l)}\rvert^{1/2}=\mathbf{\Theta}^{(l)}\mathbf{Z}\mathbf{R}^{(l)}. X ( l ) = V ( l ) ∣ Λ ( l ) ∣ 1/2 = Θ ( l ) Z R ( l ) .
Now we show B ( l ) = R ( l ) I p ( l ) , q ( l ) R ( l ) ⊤ \mathbf{B}^{(l)}=\mathbf{R}^{(l)}\mathbf{I}_{p^{(l)},q^{(l)}}\mathbf{R}^{(l)\top} B ( l ) = R ( l ) I p ( l ) , q ( l ) R ( l ) ⊤ using the following relation from (B.5):
( Z ⊤ ( Θ ( l ) ) 2 Z ) 1 / 2 B ( l ) ( Z ⊤ ( Θ ( l ) ) 2 Z ) 1 / 2 = U ~ ( l ) Λ ( l ) U ~ ( l ) ⊤ . \left(\mathbf{Z}^{\top}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{1/2}\mathbf{B}^{(l)}\left(\mathbf{Z}^{\top}\left(\mathbf{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{1/2}=\widetilde{\mathbf{U}}^{(l)}\mathbf{\Lambda}^{(l)}\widetilde{\mathbf{U}}^{(l)\top}. ( Z ⊤ ( Θ ( l ) ) 2 Z ) 1/2 B ( l ) ( Z ⊤ ( Θ ( l ) ) 2 Z ) 1/2 = U ( l ) Λ ( l ) U ( l ) ⊤ .
Since
Λ ( l ) = ∣ Λ ( l ) ∣ 1 / 2 I p ( l ) , q ( l ) ∣ Λ ( l ) ∣ 1 / 2 , \mathbf{\Lambda}^{(l)}=\lvert\mathbf{\Lambda}^{(l)}\rvert^{1/2}\mathbf{I}_{p^{(l)},q^{(l)}}\lvert\mathbf{\Lambda}^{(l)}\rvert^{1/2}, Λ ( l ) = ∣ Λ ( l ) ∣ 1/2 I p ( l ) , q ( l ) ∣ Λ ( l ) ∣ 1/2 ,
we have
R ( l ) I p , q ( l ) R ( l ) T = ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) O ( l ) Λ ( l ) O ( l ) T U ~ ( l ) T ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 = ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) Λ ( l ) U ~ ( l ) T ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 = B ( l ) . \mathbf{R}^{(l)} I_{p,q}^{(l)} \mathbf{R}^{(l)\mathsf{T}} = \left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\mathbf{O}^{(l)}\boldsymbol{\Lambda}^{(l)}\mathbf{O}^{(l)\mathsf{T}}\widetilde{\mathbf{U}}^{(l)\mathsf{T}}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}
= \left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\boldsymbol{\Lambda}^{(l)}\widetilde{\mathbf{U}}^{(l)\mathsf{T}}\left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}
= \mathbf{B}^{(l)}. R ( l ) I p , q ( l ) R ( l ) T = ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) O ( l ) Λ ( l ) O ( l ) T U ( l ) T ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 = ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) Λ ( l ) U ( l ) T ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 = B ( l ) .
Singular-Value Relations Between R and B
The model-based representation in Proposition 2.2 introduces the direction matrix R ( l ) \mathbf{R}^{(l)} R ( l ) by
R ( l ) = ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) O ( l ) ∣ Λ ( l ) ∣ 1 2 . \mathbf{R}^{(l)} = \left(\mathbf{Z}^{\mathsf{T}}\left(\boldsymbol{\Theta}^{(l)}\right)^2\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\mathbf{O}^{(l)}\lvert\boldsymbol{\Lambda}^{(l)}\rvert^{\frac{1}{2}}. R ( l ) = ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) O ( l ) ∣ Λ ( l ) ∣ 2 1 .
The direction matrices R ( l ) \mathbf{R}^{(l)} R ( l ) map membership vectors in Δ K − 1 \Delta^{K-1} Δ K − 1 to population spectral embeddings under the special case Θ ( l ) = I n \boldsymbol{\Theta}^{(l)}=\mathbf{I}_n Θ ( l ) = I n . After row normalization, these linear images form spherical simplices whose hemisphere poles are determined by ( R ( l ) ) − 1 1 K (\mathbf{R}^{(l)})^{-1}\mathbf{1}_K ( R ( l ) ) − 1 1 K . Thus, the maximum and minimum singular values of R ( l ) \mathbf{R}^{(l)} R ( l ) are central to the bi-Lipschitz analysis of the induced embedding map. See Section D for details.
Since our assumptions are stated in terms of the block connectivity matrix B ( l ) \mathbf{B}^{(l)} B ( l ) , we study relationships between the singular values of B ( l ) \mathbf{B}^{(l)} B ( l ) and R ( l ) \mathbf{R}^{(l)} R ( l ) . Note that
σ k ( B ( l ) ) ≠ ( σ k ( R ( l ) ) ) 2 \sigma_k\left(\mathbf{B}^{(l)}\right) \ne\left(\sigma_k\left(\mathbf{R}^{(l)}\right)\right)^2 σ k ( B ( l ) ) = ( σ k ( R ( l ) ) ) 2
unless I p , q ( l ) = I K ( l ) I_{p,q}^{(l)}=I_K^{(l)} I p , q ( l ) = I K ( l ) . Nevertheless, standard singular-value inequalities yield the following bounds:
σ max ( B ( l ) ) = σ max ( R ( l ) I p , q ( l ) R ( l ) T ) ≤ σ max ( R ( l ) ) σ max ( I p , q ( l ) ) σ max ( R ( l ) T ) = ( σ max ( R ( l ) ) ) 2 , (32) \begin{aligned}
\sigma_{\max}\left(\mathbf{B}^{(l)}\right) &= \sigma_{\max}\left(\mathbf{R}^{(l)}I_{p,q}^{(l)}\mathbf{R}^{(l)\mathsf{T}}\right) \\
&\le\sigma_{\max}\left(\mathbf{R}^{(l)}\right)\sigma_{\max}\left(I_{p,q}^{(l)}\right)\sigma_{\max}\left(\mathbf{R}^{(l)\mathsf{T}}\right) \\
&= \left(\sigma_{\max}\left(\mathbf{R}^{(l)}\right)\right)^2, \tag*{(32)}
\end{aligned} σ m a x ( B ( l ) ) = σ m a x ( R ( l ) I p , q ( l ) R ( l ) T ) ≤ σ m a x ( R ( l ) ) σ m a x ( I p , q ( l ) ) σ m a x ( R ( l ) T ) = ( σ m a x ( R ( l ) ) ) 2 , (32)
σ min ( B ( l ) ) = σ min ( R ( l ) I p , q ( l ) R ( l ) T ) ≥ σ min ( R ( l ) ) σ min ( I p , q ( l ) ) σ min ( R ( l ) T ) = ( σ min ( R ( l ) ) ) 2 . (33) \begin{aligned}
\sigma_{\min}\left(\mathbf{B}^{(l)}\right) &= \sigma_{\min}\left(\mathbf{R}^{(l)}I_{p,q}^{(l)}\mathbf{R}^{(l)\mathsf{T}}\right) \\
&\ge\sigma_{\min}\left(\mathbf{R}^{(l)}\right)\sigma_{\min}\left(I_{p,q}^{(l)}\right)\sigma_{\min}\left(\mathbf{R}^{(l)\mathsf{T}}\right) \\
&= \left(\sigma_{\min}\left(\mathbf{R}^{(l)}\right)\right)^2. \tag*{(33)}
\end{aligned} σ m i n ( B ( l ) ) = σ m i n ( R ( l ) I p , q ( l ) R ( l ) T ) ≥ σ m i n ( R ( l ) ) σ m i n ( I p , q ( l ) ) σ m i n ( R ( l ) T ) = ( σ m i n ( R ( l ) ) ) 2 . (33)
The bounds in (C.1) and (C.2) control the singular values of R ( l ) \mathbf{R}^{(l)} R ( l ) in terms of those of B ( l ) \mathbf{B}^{(l)} B ( l ) . We now derive bounds in the opposite direction, controlling σ max ( R ( l ) ) \sigma_{\max}(\mathbf{R}^{(l)}) σ m a x ( R ( l ) ) from above and σ min ( R ( l ) ) \sigma_{\min}(\mathbf{R}^{(l)}) σ m i n ( R ( l ) ) from below in terms of the singular values of B ( l ) \mathbf{B}^{(l)} B ( l ) .
Upper bound of σ max ( R ( l ) ) \sigma_{\max}(\mathbf{R}^{(l)}) σ m a x ( R ( l ) )
We provide an upper bound for σ max ( R ( l ) ) \sigma_{\max}(\mathbf{R}^{(l)}) σ m a x ( R ( l ) ) . Recall that ∥ G ( l ) ∥ = ∥ K ∥ θ ( l ) ∥ ( Z T ( Θ ( l ) ) 2 Z ) 1 2 ∥ ≤ C \|\mathbf{G}^{(l)}\| = \left\| \frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|} \left(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^{2}\mathbf{Z}\right)^{\frac{1}{2}} \right\| \le C ∥ G ( l ) ∥ = ∥ θ ( l ) ∥ K ( Z T ( Θ ( l ) ) 2 Z ) 2 1 ≤ C and ∥ ( G ( l ) ) − 1 ∥ = ∥ ∥ θ ( l ) ∥ K ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 ∥ ≤ C \|(\mathbf{G}^{(l)})^{-1}\| = \left\| \frac{\|\boldsymbol{\theta}^{(l)}\|}{\sqrt{K}} \left(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^{2}\mathbf{Z}\right)^{-\frac{1}{2}} \right\| \le C ∥ ( G ( l ) ) − 1 ∥ = K ∥ θ ( l ) ∥ ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 ≤ C by assumption 1.
σ max ( R ( l ) ) = σ max ( ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) O ( l ) ∣ Λ ( l ) ∣ 1 2 ) = σ max ( ∥ θ ( l ) ∥ K ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 K ∥ θ ( l ) ∥ U ~ ( l ) O ( l ) ∣ Λ ( l ) ∣ 1 2 ) = K ∥ θ ( l ) ∥ σ max ( ( G ( l ) ) − 1 U ~ ( l ) O ( l ) ∣ Λ ( l ) ∣ 1 2 ) ≤ K ∥ θ ( l ) ∥ σ max ( ( G ( l ) ) − 1 ) σ max ( U ~ ( l ) ) σ max ( O ( l ) ) σ max ( ∣ Λ ( l ) ∣ 1 2 ) ≤ C K ∥ θ ( l ) ∥ σ max ( P ( l ) ) ≲ C K ∥ θ ( l ) ∥ ∥ θ ( l ) ∥ K σ max ( B ( l ) ) = C σ max ( B ( l ) ) . (34) \begin{aligned}
\sigma_{\max}(\mathbf{R}^{(l)}) &= \sigma_{\max}\left(\left(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^{2}\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\mathbf{O}^{(l)}\left|\boldsymbol{\Lambda}^{(l)}\right|^{\frac{1}{2}}\right) \\
&= \sigma_{\max}\left(\frac{\|\boldsymbol{\theta}^{(l)}\|}{\sqrt{K}}\left(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^{2}\mathbf{Z}\right)^{-\frac{1}{2}}\frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|}\widetilde{\mathbf{U}}^{(l)}\mathbf{O}^{(l)}\left|\boldsymbol{\Lambda}^{(l)}\right|^{\frac{1}{2}}\right) \\
&= \frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|}\sigma_{\max}\left((\mathbf{G}^{(l)})^{-1}\widetilde{\mathbf{U}}^{(l)}\mathbf{O}^{(l)}\left|\boldsymbol{\Lambda}^{(l)}\right|^{\frac{1}{2}}\right) \\
&\le\frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|}\sigma_{\max}\left((\mathbf{G}^{(l)})^{-1}\right)\sigma_{\max}\left(\widetilde{\mathbf{U}}^{(l)}\right)\sigma_{\max}\left(\mathbf{O}^{(l)}\right)\sigma_{\max}\left(\left|\boldsymbol{\Lambda}^{(l)}\right|^{\frac{1}{2}}\right) \\
&\le C\frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|}\sqrt{\sigma_{\max}(\mathbf{P}^{(l)})} \\
&\lesssim C\frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|}\frac{\|\boldsymbol{\theta}^{(l)}\|}{\sqrt{K}}\sqrt{\sigma_{\max}(\mathbf{B}^{(l)})} \\
&= C\sqrt{\sigma_{\max}(\mathbf{B}^{(l)})}. \tag*{(34)}
\end{aligned} σ m a x ( R ( l ) ) = σ m a x ( ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) O ( l ) Λ ( l ) 2 1 ) = σ m a x ( K ∥ θ ( l ) ∥ ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 ∥ θ ( l ) ∥ K U ( l ) O ( l ) Λ ( l ) 2 1 ) = ∥ θ ( l ) ∥ K σ m a x ( ( G ( l ) ) − 1 U ( l ) O ( l ) Λ ( l ) 2 1 ) ≤ ∥ θ ( l ) ∥ K σ m a x ( ( G ( l ) ) − 1 ) σ m a x ( U ( l ) ) σ m a x ( O ( l ) ) σ m a x ( Λ ( l ) 2 1 ) ≤ C ∥ θ ( l ) ∥ K σ m a x ( P ( l ) ) ≲ C ∥ θ ( l ) ∥ K K ∥ θ ( l ) ∥ σ m a x ( B ( l ) ) = C σ m a x ( B ( l ) ) . (34)
Lower bound of σ min ( R ( l ) ) \sigma_{\min}(\mathbf{R}^{(l)}) σ m i n ( R ( l ) )
Similarly, we provide a lower bound of σ min ( R ( l ) ) \sigma_{\min}(\mathbf{R}^{(l)}) σ m i n ( R ( l ) ) .
σ min ( R ( l ) ) = σ min ( ( Z T ( Θ ( l ) ) 2 Z ) − 1 2 U ~ ( l ) O ( l ) ∣ Λ ( l ) ∣ 1 2 ) = 1 σ max ( ∣ Λ ( l ) ∣ − 1 2 ( O ( l ) ) T ( U ~ ( l ) ) T ( Z T ( Θ ( l ) ) 2 Z ) 1 2 ) = 1 ∥ θ ( l ) ∥ K σ max ( ∣ Λ ( l ) ∣ − 1 2 ( O ( l ) ) T ( U ~ ( l ) ) T K ∥ θ ( l ) ∥ ( Z T ( Θ ( l ) ) 2 Z ) 1 2 ) ≥ K ∥ θ ( l ) ∥ 1 σ max ( ∣ Λ ( l ) ∣ − 1 2 ) 1 ∥ G ( l ) ∥ ≥ K ∥ θ ( l ) ∥ σ min ( ∣ Λ ( l ) ∣ 1 2 ) 1 C ≳ K ∥ θ ( l ) ∥ ∥ θ ( l ) ∥ K σ min ( B ( l ) ) ≲ σ min ( B ( l ) ) . (35) \begin{aligned}
\sigma_{\min}(\mathbf{R}^{(l)}) &= \sigma_{\min}\left(\left(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^{2}\mathbf{Z}\right)^{-\frac{1}{2}}\widetilde{\mathbf{U}}^{(l)}\mathbf{O}^{(l)}\left|\boldsymbol{\Lambda}^{(l)}\right|^{\frac{1}{2}}\right) \\
&= \frac{1}{\sigma_{\max}\left(\left|\boldsymbol{\Lambda}^{(l)}\right|^{-\frac{1}{2}}(\mathbf{O}^{(l)})^{\mathsf{T}}(\widetilde{\mathbf{U}}^{(l)})^{\mathsf{T}}(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^{2}\mathbf{Z})^{\frac{1}{2}}\right)} \\
&= \frac{1}{\frac{\|\boldsymbol{\theta}^{(l)}\|}{\sqrt{K}}\sigma_{\max}\left(\left|\boldsymbol{\Lambda}^{(l)}\right|^{-\frac{1}{2}}(\mathbf{O}^{(l)})^{\mathsf{T}}(\widetilde{\mathbf{U}}^{(l)})^{\mathsf{T}}\frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|}(\mathbf{Z}^{\mathsf{T}}(\boldsymbol{\Theta}^{(l)})^{2}\mathbf{Z})^{\frac{1}{2}}\right)} \\
&\ge\frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|}\frac{1}{\sigma_{\max}\left(\left|\boldsymbol{\Lambda}^{(l)}\right|^{-\frac{1}{2}}\right)}\frac{1}{\|\mathbf{G}^{(l)}\|} \\
&\ge\frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|}\sigma_{\min}\left(\left|\boldsymbol{\Lambda}^{(l)}\right|^{\frac{1}{2}}\right)\frac{1}{C} \\
&\gtrsim\frac{\sqrt{K}}{\|\boldsymbol{\theta}^{(l)}\|}\frac{\|\boldsymbol{\theta}^{(l)}\|}{\sqrt{K}}\sqrt{\sigma_{\min}(\mathbf{B}^{(l)})} \\
&\lesssim\sqrt{\sigma_{\min}(\mathbf{B}^{(l)})}. \tag*{(35)}
\end{aligned} σ m i n ( R ( l ) ) = σ m i n ( ( Z T ( Θ ( l ) ) 2 Z ) − 2 1 U ( l ) O ( l ) Λ ( l ) 2 1 ) = σ m a x ( Λ ( l ) − 2 1 ( O ( l ) ) T ( U ( l ) ) T ( Z T ( Θ ( l ) ) 2 Z ) 2 1 ) 1 = K ∥ θ ( l ) ∥ σ m a x ( Λ ( l ) − 2 1 ( O ( l ) ) T ( U ( l ) ) T ∥ θ ( l ) ∥ K ( Z T ( Θ ( l ) ) 2 Z ) 2 1 ) 1 ≥ ∥ θ ( l ) ∥ K σ m a x ( Λ ( l ) − 2 1 ) 1 ∥ G ( l ) ∥ 1 ≥ ∥ θ ( l ) ∥ K σ m i n ( Λ ( l ) 2 1 ) C 1 ≳ ∥ θ ( l ) ∥ K K ∥ θ ( l ) ∥ σ m i n ( B ( l ) ) ≲ σ m i n ( B ( l ) ) . (35)
Reparametrization of MLDCMM model
We restate the identifiability conditions for convenience.
Condition 1 (Identifiability conditions). The following conditions hold. (a) Membership matrix. Each row of Z Z Z has unit ℓ 1 \ell_1 ℓ 1 -norm, and there exists at least one pure node for each community.
(b) Community-overlap connectivity. Define a community graph G Z = ( [ K ] , E Z ) G_Z=([K],\mathcal{E}_Z) G Z = ([ K ] , E Z ) , where
{ a , b } ∈ E Z ⟺ ∃ i ∈ [ n ] such that Z i a > 0 , Z i b > 0 , a ≠ b . \{a,b\}\in\mathcal{E}_Z \Longleftrightarrow\exists i\in[n]\ \text{such that}\ Z_{ia}>0,\ Z_{ib}>0,\quad a\ne b. { a , b } ∈ E Z ⟺ ∃ i ∈ [ n ] such that Z ia > 0 , Z ib > 0 , a = b .
Then, G Z G_Z G Z is connected.
(c) Block connectivity matrices. The matrix B ( l ) \mathbf{B}^{(l)} B ( l ) is full rank for every 1 ≤ l ≤ L 1\le l\le L 1 ≤ l ≤ L . Moreover, for the reference layer l = 1 l=1 l = 1 the matrix R ( 1 ) \mathbf{R}^{(1)} R ( 1 ) defined in Equation (2.3) satisfies ∥ ( R ( 1 ) ) k , ⋅ ∥ 2 = 1 \lVert(\mathbf{R}^{(1)})_{k,\cdot}\rVert_2=1 ∥( R ( 1 ) ) k , ⋅ ∥ 2 = 1 for all k ∈ [ K ] k\in[K] k ∈ [ K ] . Lastly, ∥ B ( l ) ∥ max = 1 \lVert\mathbf{B}^{(l)}\rVert_{\max}=1 ∥ B ( l ) ∥ m a x = 1 for all l ≠ 1 l\ne1 l = 1 .
Condition 1 includes a reference-layer normalization: there exists a layer, denoted by ref, such that
∥ R k , ⋅ ( r e f ) ∥ 2 = 1 , 1 ≤ k ≤ K . \lVert\mathbf{R}^{(\mathrm{ref})}_{k,\cdot}\rVert_2=1,\quad1\le k\le K. ∥ R k , ⋅ ( ref ) ∥ 2 = 1 , 1 ≤ k ≤ K .
We show that this condition should be viewed as a normalization rather than a substantive model restriction. Specifically, any parametrization satisfying the remaining structural identifiability conditions can be reparametrized into a canonical form
( { Θ c a n ( l ) } l = 1 L , Z c a n , { B c a n ( l ) } l = 1 L ) \left(\{\boldsymbol{\Theta}_{\mathrm{can}}^{(l)}\}_{l=1}^{L},Z_{\mathrm{can}},\{\mathbf{B}_{\mathrm{can}}^{(l)}\}_{l=1}^{L}\right) ( { Θ can ( l ) } l = 1 L , Z can , { B can ( l ) } l = 1 L )
that yields the same population matrices and satisfies the required normalization conditions. Without loss of generality, assume that the first layer is the reference layer. Let D 1 ( 1 ) ∈ R K × K \mathbf{D}_1^{(1)}\in\mathbb{R}^{K\times K} D 1 ( 1 ) ∈ R K × K be a diagonal matrix with ( D 1 ( 1 ) ) k k = ∥ ( R ( 1 ) ) k , ⋅ ∥ 2 (\mathbf{D}_1^{(1)})_{kk}=\lVert(\mathbf{R}^{(1)})_{k,\cdot}\rVert_2 ( D 1 ( 1 ) ) k k = ∥( R ( 1 ) ) k , ⋅ ∥ 2 for 1 ≤ k ≤ K 1\le k\le K 1 ≤ k ≤ K . Then, we obtain row-normalized matrix R c a n ( 1 ) \mathbf{R}_{\mathrm{can}}^{(1)} R can ( 1 ) :
P ( 1 ) = Θ ( 1 ) Z B ( 1 ) Z T Θ ( 1 ) = Θ ( 1 ) Z D 1 ( 1 ) ( D 1 ( 1 ) ) − 1 R ( 1 ) I p , q ( R ( 1 ) ) R ( 1 ) T ( D 1 ( 1 ) ) − 1 D 1 ( 1 ) Z T Θ ( 1 ) ≡ Θ ( 1 ) Z † R c a n ( 1 ) I p , q ( R c a n ( 1 ) ) R c a n ( 1 ) T Z † T Θ ( 1 ) , \begin{aligned}
\mathbf{P}^{(1)}&=\boldsymbol{\Theta}^{(1)}Z\mathbf{B}^{(1)}Z^\mathsf{T}\boldsymbol{\Theta}^{(1)}\\
&=\boldsymbol{\Theta}^{(1)}Z\mathbf{D}_1^{(1)}(\mathbf{D}_1^{(1)})^{-1}\mathbf{R}^{(1)}\mathbf{I}_{p,q}(\mathbf{R}^{(1)})\mathbf{R}^{(1)\mathsf{T}}(\mathbf{D}_1^{(1)})^{-1}\mathbf{D}_1^{(1)}Z^\mathsf{T}\boldsymbol{\Theta}^{(1)}\\
&\equiv\boldsymbol{\Theta}^{(1)}Z_{\dagger}\mathbf{R}_{\mathrm{can}}^{(1)}\mathbf{I}_{p,q}(\mathbf{R}_{\mathrm{can}}^{(1)})\mathbf{R}_{\mathrm{can}}^{(1)\mathsf{T}}Z_{\dagger}^{\mathsf{T}}\boldsymbol{\Theta}^{(1)},
\end{aligned} P ( 1 ) = Θ ( 1 ) Z B ( 1 ) Z T Θ ( 1 ) = Θ ( 1 ) Z D 1 ( 1 ) ( D 1 ( 1 ) ) − 1 R ( 1 ) I p , q ( R ( 1 ) ) R ( 1 ) T ( D 1 ( 1 ) ) − 1 D 1 ( 1 ) Z T Θ ( 1 ) ≡ Θ ( 1 ) Z † R can ( 1 ) I p , q ( R can ( 1 ) ) R can ( 1 ) T Z † T Θ ( 1 ) ,
where
Z † : = Z D 1 ( 1 ) , R c a n ( 1 ) : = ( D 1 ( 1 ) ) − 1 R ( 1 ) . Z_{\dagger}:=Z\mathbf{D}_1^{(1)},\qquad\mathbf{R}_{\mathrm{can}}^{(1)}:=(\mathbf{D}_1^{(1)})^{-1}\mathbf{R}^{(1)}. Z † := Z D 1 ( 1 ) , R can ( 1 ) := ( D 1 ( 1 ) ) − 1 R ( 1 ) .
By construction, R c a n ( 1 ) \mathbf{R}_{\mathrm{can}}^{(1)} R can ( 1 ) has unit ℓ 2 \ell_2 ℓ 2 -norm rows, while Z † Z_{\dagger} Z † may not have unit ℓ 1 \ell_1 ℓ 1 -norm rows. Next, let D 2 ( 1 ) ∈ R n × n \mathbf{D}_2^{(1)}\in\mathbb{R}^{n\times n} D 2 ( 1 ) ∈ R n × n be the diagonal matrix with ( D 2 ( 1 ) ) i i = ∥ ( Z † ) i , ⋅ ∥ 1 (\mathbf{D}_2^{(1)})_{ii}=\lVert(Z_{\dagger})_{i,\cdot}\rVert_1 ( D 2 ( 1 ) ) ii = ∥( Z † ) i , ⋅ ∥ 1 for 1 ≤ i ≤ n 1\le i\le n 1 ≤ i ≤ n . Then we obtain a matrix with unit ℓ 1 \ell_1 ℓ 1 -norm rows Z c a n Z_{\mathrm{can}} Z can :
P ( 1 ) = Θ ( 1 ) Z † R c a n ( 1 ) I p , q ( R c a n ( 1 ) ) R c a n ( 1 ) T Z † T Θ ( 1 ) = Θ ( 1 ) D 2 ( 1 ) ( D 2 ( 1 ) ) − 1 Z † R c a n ( 1 ) I p , q ( R c a n ( 1 ) ) R c a n ( 1 ) T Z † T ( D 2 ( 1 ) ) − 1 D 2 ( 1 ) Θ ( 1 ) ≡ Θ c a n ( 1 ) Z c a n R c a n ( 1 ) I p , q ( R c a n ( 1 ) ) R c a n ( 1 ) T Z c a n T Θ c a n ( 1 ) , \begin{aligned}
\mathbf{P}^{(1)}&=\boldsymbol{\Theta}^{(1)}Z_{\dagger}\mathbf{R}_{\mathrm{can}}^{(1)}\mathbf{I}_{p,q}(\mathbf{R}_{\mathrm{can}}^{(1)})\mathbf{R}_{\mathrm{can}}^{(1)\mathsf{T}}Z_{\dagger}^{\mathsf{T}}\boldsymbol{\Theta}^{(1)}\\
&=\boldsymbol{\Theta}^{(1)}\mathbf{D}_2^{(1)}(\mathbf{D}_2^{(1)})^{-1}Z_{\dagger}\mathbf{R}_{\mathrm{can}}^{(1)}\mathbf{I}_{p,q}(\mathbf{R}_{\mathrm{can}}^{(1)})\mathbf{R}_{\mathrm{can}}^{(1)\mathsf{T}}Z_{\dagger}^{\mathsf{T}}(\mathbf{D}_2^{(1)})^{-1}\mathbf{D}_2^{(1)}\boldsymbol{\Theta}^{(1)}\\
&\equiv\boldsymbol{\Theta}_{\mathrm{can}}^{(1)}Z_{\mathrm{can}}\mathbf{R}_{\mathrm{can}}^{(1)}\mathbf{I}_{p,q}(\mathbf{R}_{\mathrm{can}}^{(1)})\mathbf{R}_{\mathrm{can}}^{(1)\mathsf{T}}Z_{\mathrm{can}}^\mathsf{T}\boldsymbol{\Theta}_{\mathrm{can}}^{(1)},
\end{aligned} P ( 1 ) = Θ ( 1 ) Z † R can ( 1 ) I p , q ( R can ( 1 ) ) R can ( 1 ) T Z † T Θ ( 1 ) = Θ ( 1 ) D 2 ( 1 ) ( D 2 ( 1 ) ) − 1 Z † R can ( 1 ) I p , q ( R can ( 1 ) ) R can ( 1 ) T Z † T ( D 2 ( 1 ) ) − 1 D 2 ( 1 ) Θ ( 1 ) ≡ Θ can ( 1 ) Z can R can ( 1 ) I p , q ( R can ( 1 ) ) R can ( 1 ) T Z can T Θ can ( 1 ) ,
where
Z c a n : = ( D 2 ( 1 ) ) − 1 Z † , Θ c a n ( 1 ) : = Θ ( 1 ) D 2 ( 1 ) . Z_{\mathrm{can}}:=(\mathbf{D}_2^{(1)})^{-1}Z_{\dagger},\qquad\boldsymbol{\Theta}_{\mathrm{can}}^{(1)}:=\boldsymbol{\Theta}^{(1)}\mathbf{D}_2^{(1)}. Z can := ( D 2 ( 1 ) ) − 1 Z † , Θ can ( 1 ) := Θ ( 1 ) D 2 ( 1 ) .
Equivalently, the reference layer can be written as
P ( 1 ) = Θ c a n ( 1 ) Z c a n B c a n ( 1 ) Z c a n T Θ c a n ( 1 ) , \mathbf{P}^{(1)}=\boldsymbol{\Theta}_{\mathrm{can}}^{(1)}Z_{\mathrm{can}}\mathbf{B}_{\mathrm{can}}^{(1)}Z_{\mathrm{can}}^\mathsf{T}\boldsymbol{\Theta}_{\mathrm{can}}^{(1)}, P ( 1 ) = Θ can ( 1 ) Z can B can ( 1 ) Z can T Θ can ( 1 ) ,
where
B c a n ( 1 ) : = R c a n ( 1 ) I p , q ( R c a n ( 1 ) ) R c a n ( 1 ) T . \mathbf{B}_{\mathrm{can}}^{(1)}:=\mathbf{R}_{\mathrm{can}}^{(1)}\mathbf{I}_{p,q}(\mathbf{R}_{\mathrm{can}}^{(1)})\mathbf{R}_{\mathrm{can}}^{(1)\mathsf{T}}. B can ( 1 ) := R can ( 1 ) I p , q ( R can ( 1 ) ) R can ( 1 ) T .
This representation satisfies the membership row-normalization and the reference-layer unit-row-norm condition.
Now, we represent P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) \mathbf{P}^{(l)} = \mathbf{\Theta}^{(l)}\mathbf{Z}\mathbf{B}^{(l)}\mathbf{Z}^{\top}\mathbf{\Theta}^{(l)} P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) in terms of Z c a n \mathbf{Z}_{\mathrm{can}} Z can for l ≥ 2 l \ge2 l ≥ 2 .
P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) = Θ ( l ) D 2 ( 1 ) ( D 2 ( 1 ) ) − 1 Z D 1 ( 1 ) ( D 1 ( 1 ) ) − 1 B ( l ) ( D 1 ( 1 ) ) − 1 D 1 ( 1 ) Z ⊤ ( D 2 ( 1 ) ) − 1 D 2 ( 1 ) Θ ( l ) ≡ Θ † ( l ) Z c a n B † ( l ) Z c a n ⊤ Θ † ( l ) , \begin{aligned}
\mathbf{P}^{(l)} = \mathbf{\Theta}^{(l)}\mathbf{Z}\mathbf{B}^{(l)}\mathbf{Z}^{\top}\mathbf{\Theta}^{(l)} \\
&= \mathbf{\Theta}^{(l)}\mathbf{D}_{2}^{(1)}\left(\mathbf{D}_{2}^{(1)}\right)^{-1}\mathbf{Z}\mathbf{D}_{1}^{(1)}\left(\mathbf{D}_{1}^{(1)}\right)^{-1}\mathbf{B}^{(l)}\left(\mathbf{D}_{1}^{(1)}\right)^{-1}\mathbf{D}_{1}^{(1)}\mathbf{Z}^{\top}\left(\mathbf{D}_{2}^{(1)}\right)^{-1}\mathbf{D}_{2}^{(1)}\mathbf{\Theta}^{(l)} \\
&\equiv\mathbf{\Theta}_{\dagger}^{(l)}\mathbf{Z}_{\mathrm{can}}\mathbf{B}_{\dagger}^{(l)}\mathbf{Z}_{\mathrm{can}}^{\top}\mathbf{\Theta}_{\dagger}^{(l)},
\end{aligned} P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) = Θ ( l ) D 2 ( 1 ) ( D 2 ( 1 ) ) − 1 Z D 1 ( 1 ) ( D 1 ( 1 ) ) − 1 B ( l ) ( D 1 ( 1 ) ) − 1 D 1 ( 1 ) Z ⊤ ( D 2 ( 1 ) ) − 1 D 2 ( 1 ) Θ ( l ) ≡ Θ † ( l ) Z can B † ( l ) Z can ⊤ Θ † ( l ) ,
where Θ † ( l ) : = Θ ( l ) D 2 ( 1 ) \mathbf{\Theta}_{\dagger}^{(l)} := \mathbf{\Theta}^{(l)}\mathbf{D}_{2}^{(1)} Θ † ( l ) := Θ ( l ) D 2 ( 1 ) and B † ( l ) : = ( D 1 ( 1 ) ) − 1 B ( l ) ( D 1 ( 1 ) ) − 1 \mathbf{B}_{\dagger}^{(l)} := \left(\mathbf{D}_{1}^{(1)}\right)^{-1}\mathbf{B}^{(l)}\left(\mathbf{D}_{1}^{(1)}\right)^{-1} B † ( l ) := ( D 1 ( 1 ) ) − 1 B ( l ) ( D 1 ( 1 ) ) − 1 . Note that B † ( l ) \mathbf{B}_{\dagger}^{(l)} B † ( l ) need not satisfy
∥ B † ( l ) ∥ max = 1. \left\|\mathbf{B}_{\dagger}^{(l)}\right\|_{\max} = 1. B † ( l ) m a x = 1.
To this end, let b ( l ) = ∥ B † ( l ) ∥ max b^{(l)} = \left\|\mathbf{B}_{\dagger}^{(l)}\right\|_{\max} b ( l ) = B † ( l ) m a x . Then,
P ( l ) = Θ † ( l ) Z c a n B † ( l ) Z c a n ⊤ Θ † ( l ) = ( b ( l ) Θ † ( l ) ) Z c a n B † ( l ) b ( l ) Z c a n ⊤ ( b ( l ) Θ † ( l ) ) ≡ Θ c a n ( l ) Z c a n B c a n ( l ) Z c a n ⊤ Θ c a n ( l ) , \begin{aligned}
\mathbf{P}^{(l)} = \mathbf{\Theta}_{\dagger}^{(l)}\mathbf{Z}_{\mathrm{can}}\mathbf{B}_{\dagger}^{(l)}\mathbf{Z}_{\mathrm{can}}^{\top}\mathbf{\Theta}_{\dagger}^{(l)} \\
&= \left(\sqrt{b^{(l)}}\mathbf{\Theta}_{\dagger}^{(l)}\right)\mathbf{Z}_{\mathrm{can}}\frac{\mathbf{B}_{\dagger}^{(l)}}{b^{(l)}}\mathbf{Z}_{\mathrm{can}}^{\top}\left(\sqrt{b^{(l)}}\mathbf{\Theta}_{\dagger}^{(l)}\right) \\
&\equiv\mathbf{\Theta}_{\mathrm{can}}^{(l)}\mathbf{Z}_{\mathrm{can}}\mathbf{B}_{\mathrm{can}}^{(l)}\mathbf{Z}_{\mathrm{can}}^{\top}\mathbf{\Theta}_{\mathrm{can}}^{(l)},
\end{aligned} P ( l ) = Θ † ( l ) Z can B † ( l ) Z can ⊤ Θ † ( l ) = ( b ( l ) Θ † ( l ) ) Z can b ( l ) B † ( l ) Z can ⊤ ( b ( l ) Θ † ( l ) ) ≡ Θ can ( l ) Z can B can ( l ) Z can ⊤ Θ can ( l ) ,
where Θ c a n ( l ) = b ( l ) Θ † ( l ) \mathbf{\Theta}_{\mathrm{can}}^{(l)} = \sqrt{b^{(l)}}\mathbf{\Theta}_{\dagger}^{(l)} Θ can ( l ) = b ( l ) Θ † ( l ) , and B c a n ( l ) = B † ( l ) b ( l ) \mathbf{B}_{\mathrm{can}}^{(l)} = \frac{\mathbf{B}_{\dagger}^{(l)}}{b^{(l)}} B can ( l ) = b ( l ) B † ( l ) . Since D 1 ( 1 ) \mathbf{D}_{1}^{(1)} D 1 ( 1 ) and D 2 ( 1 ) \mathbf{D}_{2}^{(1)} D 2 ( 1 ) are positive diagonal matrices, this reparametrization preserves the support pattern of the membership vectors. Since D 1 ( 1 ) \mathbf{D}_{1}^{(1)} D 1 ( 1 ) and D 2 ( 1 ) \mathbf{D}_{2}^{(1)} D 2 ( 1 ) are positive diagonal matrices, this reparametrization preserves the support pattern of the membership vectors. In particular, pure nodes remain pure, and the community-overlap graph G Z G_Z G Z is unchanged. Hence the community-overlap connectivity condition is preserved. Moreover, full rank of the block connectivity matrices is preserved under multiplication by nonsingular diagonal matrices. Finally, by construction,
∥ ( R c a n ( 1 ) ) k , ⋅ ∥ 2 = 1 , 1 ≤ k ≤ K , \left\|\left(\mathbf{R}_{\mathrm{can}}^{(1)}\right)_{k,\cdot}\right\|_2 = 1,\quad1 \le k \le K, ( R can ( 1 ) ) k , ⋅ 2 = 1 , 1 ≤ k ≤ K ,
and
∥ B c a n ( l ) ∥ max = 1 , l ≥ 2. \left\|\mathbf{B}_{\mathrm{can}}^{(l)}\right\|_{\max} = 1,\quad l \ge2. B can ( l ) m a x = 1 , l ≥ 2.
Therefore, the canonical parametrization
( { Θ c a n ( l ) } l = 1 L , Z c a n , { B c a n ( l ) } l = 1 L ) \left(\left\{\mathbf{\Theta}_{\mathrm{can}}^{(l)}\right\}_{l=1}^{L},\mathbf{Z}_{\mathrm{can}},\left\{\mathbf{B}_{\mathrm{can}}^{(l)}\right\}_{l=1}^{L}\right) ( { Θ can ( l ) } l = 1 L , Z can , { B can ( l ) } l = 1 L )
satisfies all required identifiability conditions and yields the same population matrices { P ( l ) } l = 1 L \left\{\mathbf{P}^{(l)}\right\}_{l=1}^{L} { P ( l ) } l = 1 L .
Identifiability of MLDCMM
We now establish identifiability of the MLDCMM model under Condition 1. Compared with the single-layer DCMM setting, the main additional difficulty is to identify the layer-specific scaling in the non-reference layers. The reference layer identifies the shared membership matrix Z \mathbf{Z} Z up to a common permutation of community labels. For each non-reference layer l ≥ 2 l \ge2 l ≥ 2 , the mixed-node richness condition ensures that the mixed nodes link the community coordinates through overlapping membership supports; these overlap constraints determine the relative row norms of R ( l ) \mathbf{R}^{(l)} R ( l ) . Once these relative row-norm ratios are identified, the remaining layer-specific parameters can be recovered. We restate Proposition 2.3 for convenience.
Proposition 2.3 (Identifiability of ML-DCMM). Suppose that two parameterizations { Θ ( l ) } l = 1 L \left\{\mathbf{\Theta}^{(l)}\right\}_{l=1}^{L} { Θ ( l ) } l = 1 L , Z \mathbf{Z} Z , { B ( l ) } l = 1 L \left\{\mathbf{B}^{(l)}\right\}_{l=1}^{L} { B ( l ) } l = 1 L and { Θ ~ ( l ) } l = 1 L \left\{\widetilde{\mathbf{\Theta}}^{(l)}\right\}_{l=1}^{L} { Θ ( l ) } l = 1 L , Z ~ \widetilde{\mathbf{Z}} Z , { B ~ ( l ) } l = 1 L \left\{\widetilde{\mathbf{B}}^{(l)}\right\}_{l=1}^{L} { B ( l ) } l = 1 L both satisfy Condition 1 1 1 and generate the same population probability matrices, that is, P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) = Θ ~ ( l ) Z ~ B ~ ( l ) Z ~ ⊤ Θ ~ ( l ) \mathbf{P}^{(l)} = \mathbf{\Theta}^{(l)}\mathbf{Z}\mathbf{B}^{(l)}\mathbf{Z}^{\top}\mathbf{\Theta}^{(l)} = \widetilde{\mathbf{\Theta}}^{(l)}\widetilde{\mathbf{Z}}\widetilde{\mathbf{B}}^{(l)}\widetilde{\mathbf{Z}}^{\top}\widetilde{\mathbf{\Theta}}^{(l)} P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) for all l ∈ [ L ] l \in[L] l ∈ [ L ] . Then the model parameters are identifiable up to a common permutation of the community labels. More precisely, there exists a K × K K \times K K × K permutation matrix Π \mathbf{\Pi} Π such that, for every 1 ≤ l ≤ L 1 \le l \le L 1 ≤ l ≤ L ,
Θ ~ ( l ) = Θ ( l ) , Z ~ = Z Π , B ~ ( l ) = Π ⊤ B ( l ) Π . \widetilde{\mathbf{\Theta}}^{(l)} = \mathbf{\Theta}^{(l)},\quad\widetilde{\mathbf{Z}} = \mathbf{Z}\mathbf{\Pi},\quad\widetilde{\mathbf{B}}^{(l)} = \mathbf{\Pi}^{\top}\mathbf{B}^{(l)}\mathbf{\Pi}. Θ ( l ) = Θ ( l ) , Z = ZΠ , B ( l ) = Π ⊤ B ( l ) Π .
Proof. Assume the permutation of the communities is fixed. Let J 1 \mathcal{J}_1 J 1 and J 2 \mathcal{J}_2 J 2 be pure node index sets such that Z J 1 = Z ~ J 2 = I K Z_{\mathcal{J}_1} = \widetilde{Z}_{\mathcal{J}_2} = I_K Z J 1 = Z J 2 = I K . By the proof of Lemma B.1 in Mao et al. [ [40 ] , it can be shown that pure nodes match up. By Proposition 2.2, we can define two population spectral embeddings X ( l ) X^{(l)} X ( l ) and X ~ ( l ) \widetilde{X}^{(l)} X ( l ) such that
X ( l ) = Θ ( l ) Z R ( l ) and X ~ ( l ) = Θ ~ ( l ) Z ~ R ~ ( l ) . X^{(l)} = \Theta^{(l)} Z R^{(l)} \quad\text{and} \quad\widetilde{X}^{(l)} = \widetilde{\Theta}^{(l)} \widetilde{Z} \widetilde{R}^{(l)}. X ( l ) = Θ ( l ) Z R ( l ) and X ( l ) = Θ ( l ) Z R ( l ) .
Since the two parametrizations generate the same P ( l ) P^{(l)} P ( l ) , we have
X ( l ) ( X ( l ) ) T = X ~ ( l ) ( X ~ ( l ) ) T ⇒ Θ ( l ) Z R ( l ) ( R ( l ) ) T Z T Θ ( l ) = Θ ~ ( l ) Z ~ R ~ ( l ) ( R ~ ( l ) ) T Z ~ T Θ ~ ( l ) . \begin{aligned}
X^{(l)}(X^{(l)})^{\mathsf{T}} = \widetilde{X}^{(l)}(\widetilde{X}^{(l)})^{\mathsf{T}} \\
&\Rightarrow\Theta^{(l)} Z R^{(l)}(R^{(l)})^{\mathsf{T}} Z^{\mathsf{T}}\Theta^{(l)} = \widetilde{\Theta}^{(l)}\widetilde{Z}\widetilde{R}^{(l)}(\widetilde{R}^{(l)})^{\mathsf{T}}\widetilde{Z}^{\mathsf{T}}\widetilde{\Theta}^{(l)}.
\end{aligned} X ( l ) ( X ( l ) ) T = X ( l ) ( X ( l ) ) T ⇒ Θ ( l ) Z R ( l ) ( R ( l ) ) T Z T Θ ( l ) = Θ ( l ) Z R ( l ) ( R ( l ) ) T Z T Θ ( l ) .
Selecting the rows and columns corresponding to the indices in J 1 \mathcal{J}_1 J 1 on both sides,
Θ J 1 ( 1 ) R ( 1 ) ( R ( 1 ) ) T Θ J 1 ( 1 ) = Θ ~ J 1 ( 1 ) Π ( 1 ) R ~ ( 1 ) ( R ~ ( 1 ) ) T ( Π ( 1 ) ) T Θ ~ J 1 ( 1 ) , \Theta_{\mathcal{J}_1}^{(1)}R^{(1)}(R^{(1)})^{\mathsf{T}}\Theta_{\mathcal{J}_1}^{(1)} = \widetilde{\Theta}_{\mathcal{J}_1}^{(1)}\Pi^{(1)}\widetilde{R}^{(1)}(\widetilde{R}^{(1)})^{\mathsf{T}}(\Pi^{(1)})^{\mathsf{T}}\widetilde{\Theta}_{\mathcal{J}_1}^{(1)}, Θ J 1 ( 1 ) R ( 1 ) ( R ( 1 ) ) T Θ J 1 ( 1 ) = Θ J 1 ( 1 ) Π ( 1 ) R ( 1 ) ( R ( 1 ) ) T ( Π ( 1 ) ) T Θ J 1 ( 1 ) ,
where Π ( 1 ) : = Z ~ J 1 \Pi^{(1)} := \widetilde{Z}_{\mathcal{J}_1} Π ( 1 ) := Z J 1 is a permutation matrix. Note that R ( 1 ) ( R ( 1 ) ) T R^{(1)}(R^{(1)})^{\mathsf{T}} R ( 1 ) ( R ( 1 ) ) T and R ~ ( 1 ) ( R ~ ( 1 ) ) T \widetilde{R}^{(1)}(\widetilde{R}^{(1)})^{\mathsf{T}} R ( 1 ) ( R ( 1 ) ) T have unit diagonals since R ( 1 ) R^{(1)} R ( 1 ) and R ~ ( 1 ) \widetilde{R}^{(1)} R ( 1 ) have unit ℓ 2 \ell_2 ℓ 2 -norm rows. By comparing diagonal elements of both sides, it can be shown that
Θ J 1 ( 1 ) = Θ ~ J 1 ( 1 ) . (36) \Theta_{\mathcal{J}_1}^{(1)} = \widetilde{\Theta}_{\mathcal{J}_1}^{(1)}. \tag*{(36)} Θ J 1 ( 1 ) = Θ J 1 ( 1 ) . (36)
We use the following key result from Lemma 2.1. in Mao et al. [ [40 ] ,
V ( l ) = Θ ( l ) Z ( Θ P ( l ) ) − 1 V P ( l ) . (37) V^{(l)} = \Theta^{(l)}Z(\Theta_{\mathcal{P}}^{(l)})^{-1}V_{\mathcal{P}}^{(l)}. \tag*{(37)} V ( l ) = Θ ( l ) Z ( Θ P ( l ) ) − 1 V P ( l ) . (37)
Substituting P \mathcal{P} P with J 1 \mathcal{J}_1 J 1 and multiplying ( V J 1 ( 1 ) ) − 1 (V_{\mathcal{J}_1}^{(1)})^{-1} ( V J 1 ( 1 ) ) − 1 to (C.6) give
Θ ( 1 ) Z ( Θ J 1 ( 1 ) ) − 1 = Θ ~ ( 1 ) Z ~ ( Θ ~ J 1 ( 1 ) ) − 1 . (38) \Theta^{(1)}Z(\Theta_{\mathcal{J}_1}^{(1)})^{-1} = \widetilde{\Theta}^{(1)}\widetilde{Z}(\widetilde{\Theta}_{\mathcal{J}_1}^{(1)})^{-1}. \tag*{(38)} Θ ( 1 ) Z ( Θ J 1 ( 1 ) ) − 1 = Θ ( 1 ) Z ( Θ J 1 ( 1 ) ) − 1 . (38)
By combining (C.5) and (C.7), we have
Θ ( 1 ) Z = Θ ~ ( 1 ) Z ~ . (39) \Theta^{(1)}Z = \widetilde{\Theta}^{(1)}\widetilde{Z}. \tag*{(39)} Θ ( 1 ) Z = Θ ( 1 ) Z . (39)
By comparing the row-wise ℓ 1 \ell_1 ℓ 1 -norms on both sides of (C.8), we have Θ ( 1 ) = Θ ~ ( 1 ) \Theta^{(1)} = \widetilde{\Theta}^{(1)} Θ ( 1 ) = Θ ( 1 ) . By pre-multiplying ( Θ ( 1 ) ) − 1 (\Theta^{(1)})^{-1} ( Θ ( 1 ) ) − 1 to (C.8), we have Z = Z ~ Z = \widetilde{Z} Z = Z . Note that Θ ( 1 ) Z \Theta^{(1)}Z Θ ( 1 ) Z is full column rank matrix. By multiplying ( Z T Θ ( 1 ) Θ ( 1 ) Z ) − 1 Z T Θ ( 1 ) (Z^{\mathsf{T}}\Theta^{(1)}\Theta^{(1)}Z)^{-1}Z^{\mathsf{T}}\Theta^{(1)} ( Z T Θ ( 1 ) Θ ( 1 ) Z ) − 1 Z T Θ ( 1 ) on the left and its transpose on the right of P ( 1 ) P^{(1)} P ( 1 ) , we prove B ( 1 ) = B ~ ( 1 ) B^{(1)} = \widetilde{B}^{(1)} B ( 1 ) = B ( 1 ) .
Now we show Θ ( l ) = Θ ~ ( l ) \Theta^{(l)} = \widetilde{\Theta}^{(l)} Θ ( l ) = Θ ( l ) , and B ( l ) = B ~ ( l ) B^{(l)} = \widetilde{B}^{(l)} B ( l ) = B ( l ) for all 2 ≤ l ≤ L 2 \le l \le L 2 ≤ l ≤ L . Let J = { i 1 , ⋯ , i K } \mathcal{J} = \{i_1, \cdots, i_K\} J = { i 1 , ⋯ , i K } be a set of pure nodes such that Z J = I K Z_{\mathcal{J}} = I_K Z J = I K . Using the relationship (C.6), we have the following:
Θ ( l ) Z ( Θ J ( l ) ) − 1 = Θ ~ ( l ) Z ~ ( Θ ~ J ( l ) ) − 1 . (40) \Theta^{(l)}Z(\Theta_{\mathcal{J}}^{(l)})^{-1} = \widetilde{\Theta}^{(l)}\widetilde{Z}(\widetilde{\Theta}_{\mathcal{J}}^{(l)})^{-1}. \tag*{(40)} Θ ( l ) Z ( Θ J ( l ) ) − 1 = Θ ( l ) Z ( Θ J ( l ) ) − 1 . (40)
The i i i -th row of both sides are calculated by
θ i ( l ) ( Z i 1 θ i 1 ( l ) , ⋯ , Z i K θ i K ( l ) ) = θ ~ i ( l ) ( Z ~ i 1 θ ~ i 1 ( l ) , ⋯ , Z ~ i K θ ~ i K ( l ) ) . \theta_i^{(l)}\left(\frac{Z_{i_1}}{\theta_{i_1}^{(l)}}, \cdots, \frac{Z_{i_K}}{\theta_{i_K}^{(l)}}\right) = \widetilde{\theta}_i^{(l)}\left(\frac{\widetilde{Z}_{i_1}}{\widetilde{\theta}_{i_1}^{(l)}}, \cdots, \frac{\widetilde{Z}_{i_K}}{\widetilde{\theta}_{i_K}^{(l)}}\right). θ i ( l ) ( θ i 1 ( l ) Z i 1 , ⋯ , θ i K ( l ) Z i K ) = θ i ( l ) ( θ i 1 ( l ) Z i 1 , ⋯ , θ i K ( l ) Z i K ) .
By the community-overlap connectivity condition, the ratios are propagated across all communities. Hence
θ i 1 ( l ) θ ~ i 1 ( l ) = ⋯ = θ i K ( l ) θ ~ i K ( l ) = : c ( l ) . \frac{\theta_{i_1}^{(l)}}{\widetilde{\theta}_{i_1}^{(l)}} = \cdots= \frac{\theta_{i_K}^{(l)}}{\widetilde{\theta}_{i_K}^{(l)}} =: c^{(l)}. θ i 1 ( l ) θ i 1 ( l ) = ⋯ = θ i K ( l ) θ i K ( l ) =: c ( l ) .
Thus
Θ J ( l ) = c ( l ) Θ ~ J ( l ) . \Theta_{\mathcal{J}}^{(l)} = c^{(l)}\widetilde{\Theta}_{\mathcal{J}}^{(l)}. Θ J ( l ) = c ( l ) Θ J ( l ) .
Multiplying (C.9) on the right by Θ J ( l ) \Theta_{\mathcal{J}}^{(l)} Θ J ( l ) , we obtain
Θ ( l ) Z = c ( l ) Θ ~ ( l ) Z . \Theta^{(l)}Z = c^{(l)}\widetilde{\Theta}^{(l)}Z. Θ ( l ) Z = c ( l ) Θ ( l ) Z .
Since each row of Z Z Z has unit ℓ 1 \ell_{1} ℓ 1 -norm, comparing the row-wise ℓ 1 \ell_{1} ℓ 1 -norms gives
Θ ( l ) = c ( l ) Θ ~ ( l ) . \Theta^{(l)} = c^{(l)}\widetilde{\Theta}^{(l)}. Θ ( l ) = c ( l ) Θ ( l ) .
Substituting this relation into the equality of the probability matrices,
Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) = Θ ~ ( l ) Z B ~ ( l ) Z ⊤ Θ ~ ( l ) , \Theta^{(l)}ZB^{(l)}Z^{\top}\Theta^{(l)} = \widetilde{\Theta}^{(l)}Z\widetilde{B}^{(l)}Z^{\top}\widetilde{\Theta}^{(l)}, Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) ,
yields
( c ( l ) ) 2 Θ ~ ( l ) Z B ( l ) Z ⊤ Θ ~ ( l ) = Θ ~ ( l ) Z B ~ ( l ) Z ⊤ Θ ~ ( l ) . \left(c^{(l)}\right)^{2}\widetilde{\Theta}^{(l)}ZB^{(l)}Z^{\top}\widetilde{\Theta}^{(l)} = \widetilde{\Theta}^{(l)}Z\widetilde{B}^{(l)}Z^{\top}\widetilde{\Theta}^{(l)}. ( c ( l ) ) 2 Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) .
Since Θ ~ ( l ) Z \widetilde{\Theta}^{(l)}Z Θ ( l ) Z has full column rank, it follows that
B ~ ( l ) = ( c ( l ) ) 2 B ( l ) . \widetilde{B}^{(l)} = \left(c^{(l)}\right)^{2}B^{(l)}. B ( l ) = ( c ( l ) ) 2 B ( l ) .
Both parametrizations satisfy
∥ B ( l ) ∥ max = ∥ B ~ ( l ) ∥ max = 1 \lVert B^{(l)}\rVert_{\max} = \lVert\widetilde{B}^{(l)}\rVert_{\max} = 1 ∥ B ( l ) ∥ m a x = ∥ B ( l ) ∥ m a x = 1
for every non-reference layer l l l . Therefore,
1 = ∥ B ~ ( l ) ∥ max = ( c ( l ) ) 2 ∥ B ( l ) ∥ max = ( c ( l ) ) 2 . 1 = \lVert\widetilde{B}^{(l)}\rVert_{\max} = \left(c^{(l)}\right)^{2}\lVert B^{(l)}\rVert_{\max} = \left(c^{(l)}\right)^{2}. 1 = ∥ B ( l ) ∥ m a x = ( c ( l ) ) 2 ∥ B ( l ) ∥ m a x = ( c ( l ) ) 2 .
Since c ( l ) > 0 c^{(l)} > 0 c ( l ) > 0 , we have c ( l ) = 1 c^{(l)} = 1 c ( l ) = 1 . Hence
Θ ( l ) = Θ ~ ( l ) and B ( l ) = B ~ ( l ) . \Theta^{(l)} = \widetilde{\Theta}^{(l)} \quad\text{and}\quad B^{(l)} = \widetilde{B}^{(l)}. Θ ( l ) = Θ ( l ) and B ( l ) = B ( l ) .
Illustrative Examples of the Identifiability Conditions
We provide two illustrative examples showing the roles of community-overlap connectivity and unit ℓ 2 \ell_{2} ℓ 2 -norm condition on ∥ R k , ⋅ ( 1 ) ∥ 2 \lVert R_{k,\cdot}^{(1)}\rVert_{2} ∥ R k , ⋅ ( 1 ) ∥ 2 only on the reference layer. Specifically, we show how nonidentifiability can arise when community-overlap connectivity condition fails and how imposing the unit-norm constraint ∥ R k , ⋅ ( l ) ∥ 2 \lVert R_{k,\cdot}^{(l)}\rVert_{2} ∥ R k , ⋅ ( l ) ∥ 2 on every layer can be restrictive.
We provide an illustrative example of the role of the Community-overlap connectivity condition in Condition 1 when K = 3 K = 3 K = 3 . A set of mixed nodes satisfying the condition determines the relative ratio of ∥ R k , ⋅ ( l ) ∥ 2 \lVert R_{k,\cdot}^{(l)}\rVert_{2} ∥ R k , ⋅ ( l ) ∥ 2 for k ∈ [ K ] k \in[K] k ∈ [ K ] . Specifically, once the membership matrix Z Z Z is identified in the reference layer, we can recover the relative ratio of ∥ R k , ⋅ ( l ) ∥ 2 \lVert R_{k,\cdot}^{(l)}\rVert_{2} ∥ R k , ⋅ ( l ) ∥ 2 by solving X ( l ) = Θ ( l ) Z R ( l ) X^{(l)} = \Theta^{(l)}ZR^{(l)} X ( l ) = Θ ( l ) Z R ( l ) . We can fix the relative ratio by assuming ∥ B ( l ) ∥ max = 1 \lVert B^{(l)}\rVert_{\max} = 1 ∥ B ( l ) ∥ m a x = 1 and then determine the remaining Θ ( l ) \Theta^{(l)} Θ ( l ) for every non-reference layer l l l . If the condition is not satisfied, we have a nonidentifiability between Θ ( l ) \Theta^{(l)} Θ ( l ) and R ( l ) R^{(l)} R ( l ) even though Z Z Z is identified from the reference layer. Figure C.1 illustrates the phenomenon. In (a), given the membership of the black point node is identified from the reference layer, for example, z = ( 1 / 3 , 1 / 3 , 1 / 3 ) ⊤ z = (1/3,1/3,1/3)^{\top} z = ( 1/3 , 1/3 , 1/3 ) ⊤ , and x = θ z ⊤ R x = \theta z^{\top}R x = θ z ⊤ R , we can identify θ \theta θ and R R R . In (b), however, since community 3 is not connected to the other communities, we have a scale nonidentifiability between θ \theta θ and R R R . Therefore, one can represent the same white point node x x x by assuming ( θ , R 3 , ⋅ ) = ( 1 , 2 a R 3 , ⋅ ) = ( 2 , a R 3 , ⋅ ) (\theta, R_{3,\cdot}) = (1, 2aR_{3,\cdot}) = (2, aR_{3,\cdot}) ( θ , R 3 , ⋅ ) = ( 1 , 2 a R 3 , ⋅ ) = ( 2 , a R 3 , ⋅ ) .
We illustrate how imposing unit-diagonal constraints on B ( l ) B^{(l)} B ( l ) across all layers can affect the shared membership structure. Consider a two-layer ML-DCMM with the following parameters. Let P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) ∈ R n × n P^{(l)} = \Theta^{(l)}ZB^{(l)}Z^{\top}\Theta^{(l)} \in\mathbb{R}^{n \times n} P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) ∈ R n × n be given with Θ ( l ) = I n \Theta^{(l)} = I_n Θ ( l ) = I n and
B ( 1 ) = ( 1 0.1 0.1 1 ) , B ( 2 ) = ( 1 0.14 0.14 0.49 ) B^{(1)} =
\begin{pmatrix}
1 & 0.1 \\
0.1 & 1
\end{pmatrix},
\qquad
B^{(2)} =
\begin{pmatrix}
1 & 0.14 \\
0.14 & 0.49
\end{pmatrix} B ( 1 ) = ( 1 0.1 0.1 1 ) , B ( 2 ) = ( 1 0.14 0.14 0.49 )
for 1 ≤ l ≤ 2 1 \le l \le2 1 ≤ l ≤ 2 . Assume Z 1 , ⋅ = ( 2 3 , 1 3 ) T \mathbf{Z}_{1,\cdot} = \left(\frac{2}{3},\frac{1}{3}\right)^{\mathsf{T}} Z 1 , ⋅ = ( 3 2 , 3 1 ) T , Z 2 , ⋅ = ( 1 3 , 2 3 ) T \mathbf{Z}_{2,\cdot} = \left(\frac{1}{3},\frac{2}{3}\right)^{\mathsf{T}} Z 2 , ⋅ = ( 3 1 , 3 2 ) T . Although the unit-diagonal constraint can be imposed on each layer through reparameterization, the resulting membership vectors differ across layers. See Figure C.2. Thus, applying the normalization separately to each layer need not preserve the shared membership structure. This motivates imposing the normalization only on the reference layer.
Figure C.1. Geometric intuition for the community-overlap connectivity condition. This figure is drawn in the unnormalized embedding X = Θ Z R \mathbf{X} = \mathbf{\Theta}\mathbf{Z}\mathbf{R} X = ΘZR . (a) In the connected case, mixed-membership nodes link the community-specific rays, making their relative scales identifiable. (b) In the disconnected case, mixed nodes are supported only on communities 1 and 2, so the scale of the third ray is not identified.
Figure C.2. Illustration of the effect of imposing a unit-diagonal constraint across layers. The left panel shows the latent positions of two nodes in Layer 1. In Layer 2, allowing non-unit diagonal entries in B ( 2 ) \mathbf{B}^{(2)} B ( 2 ) preserves the original memberships (upper right), whereas enforcing unit diagonal requires a different reparametrization and hence different memberships for the same latent positions (lower right).
Proof of Proposition 2.1
We restate Proposition 2.1 for convenience.
Proposition 2.1. Consider a collection of symmetric matrices P ( 1 ) , … , P ( L ) ∈ [ 0 , 1 ] n × n \mathbf{P}^{(1)},\ldots,\mathbf{P}^{(L)} \in[0,1]^{n \times n} P ( 1 ) , … , P ( L ) ∈ [ 0 , 1 ] n × n such that P ( l ) = V R ( l ) V T \mathbf{P}^{(l)} = \mathbf{V}\mathbf{R}^{(l)}\mathbf{V}^{\mathsf{T}} P ( l ) = V R ( l ) V T for full-rank matrices V ∈ R n × d \mathbf{V} \in\mathbb{R}^{n \times d} V ∈ R n × d and symmetric R ( l ) ∈ R d × d \mathbf{R}^{(l)} \in\mathbb{R}^{d \times d} R ( l ) ∈ R d × d , l ∈ [ L ] l \in[L] l ∈ [ L ] . Then, there exist a diagonal matrix Θ ∈ R n × n \boldsymbol{\Theta} \in\mathbb{R}^{n \times n} Θ ∈ R n × n , full-rank square symmetric matrices B ( l ) \mathbf{B}^{(l)} B ( l ) , l ∈ [ L ] l \in[L] l ∈ [ L ] and a membership matrix Z \mathbf{Z} Z with ∥ Z i , ⋅ ∥ 1 = 1 \|\mathbf{Z}_{i,\cdot}\|_1 = 1 ∥ Z i , ⋅ ∥ 1 = 1 and Z i j ≥ 0 Z_{ij} \ge0 Z ij ≥ 0 , i ∈ [ n ] i \in[n] i ∈ [ n ] , j ∈ [ d ] j \in[d] j ∈ [ d ] such that P ( l ) = Θ Z B ( l ) Z T Θ \mathbf{P}^{(l)} = \boldsymbol{\Theta}\mathbf{Z}\mathbf{B}^{(l)}\mathbf{Z}^{\mathsf{T}}\boldsymbol{\Theta} P ( l ) = Θ Z B ( l ) Z T Θ for l ∈ [ L ] l \in[L] l ∈ [ L ] .
Proof. Without loss of generality, we fix layer 1 as the reference layer. Let U ( 1 ) Λ ( 1 ) U ( 1 ) T \mathbf{U}^{(1)}\boldsymbol{\Lambda}^{(1)}\mathbf{U}^{(1)\mathsf{T}} U ( 1 ) Λ ( 1 ) U ( 1 ) T denote a spectral decomposition of R ( 1 ) \mathbf{R}^{(1)} R ( 1 ) . Following the GRDPG framework [48 ] , let X ( 1 ) : = V U ( 1 ) ∣ Λ ( 1 ) ∣ 1 / 2 ∈ R n × d \mathbf{X}^{(1)} := \mathbf{V}\mathbf{U}^{(1)}|\boldsymbol{\Lambda}^{(1)}|^{1/2} \in\mathbb{R}^{n \times d} X ( 1 ) := V U ( 1 ) ∣ Λ ( 1 ) ∣ 1/2 ∈ R n × d denote a population spectral embedding such that X ( 1 ) I p ( 1 ) , q ( 1 ) X ( 1 ) T = P ( 1 ) \mathbf{X}^{(1)}\mathbf{I}_{p^{(1)},q^{(1)}}\mathbf{X}^{(1)\mathsf{T}} = \mathbf{P}^{(1)} X ( 1 ) I p ( 1 ) , q ( 1 ) X ( 1 ) T = P ( 1 ) , where p ( 1 ) p^{(1)} p ( 1 ) and q ( 1 ) q^{(1)} q ( 1 ) denote the number of positive and negative eigenvalues of R ( 1 ) \mathbf{R}^{(1)} R ( 1 ) . Since rank ( V ) = d \operatorname{rank}(\mathbf{V}) = d rank ( V ) = d , we can define J \mathcal{J} J an index set of d d d independent rows of V \mathbf{V} V . Then W : = P ⋅ , J ( 1 ) = X ( 1 ) I p ( 1 ) , q ( 1 ) X J , ⋅ ( 1 ) T ∈ R n × d \mathbf{W} := \mathbf{P}_{\cdot,\mathcal{J}}^{(1)} = \mathbf{X}^{(1)}\mathbf{I}_{p^{(1)},q^{(1)}}\mathbf{X}_{\mathcal{J},\cdot}^{(1)\mathsf{T}} \in\mathbb{R}^{n \times d} W := P ⋅ , J ( 1 ) = X ( 1 ) I p ( 1 ) , q ( 1 ) X J , ⋅ ( 1 ) T ∈ R n × d is a nonnegative matrix since P i j ( 1 ) ≥ 0 P_{ij}^{(1)} \ge0 P ij ( 1 ) ≥ 0 for all i i i and j j j . Moreover, we have the following relationships:
P ( 1 ) = V R ( 1 ) V T = X ( 1 ) I p ( 1 ) , q ( 1 ) X ( 1 ) T = W ( X J , ⋅ ( 1 ) ) − T I p ( 1 ) , q ( 1 ) ( X J , ⋅ ( 1 ) ) − T W T . (41) \mathbf{P}^{(1)} = \mathbf{V}\mathbf{R}^{(1)}\mathbf{V}^{\mathsf{T}} = \mathbf{X}^{(1)}\mathbf{I}_{p^{(1)},q^{(1)}}\mathbf{X}^{(1)\mathsf{T}} = \mathbf{W}(\mathbf{X}_{\mathcal{J},\cdot}^{(1)})^{-\mathsf{T}}\mathbf{I}_{p^{(1)},q^{(1)}}(\mathbf{X}_{\mathcal{J},\cdot}^{(1)})^{-\mathsf{T}}\mathbf{W}^{\mathsf{T}}. \tag*{(41)} P ( 1 ) = V R ( 1 ) V T = X ( 1 ) I p ( 1 ) , q ( 1 ) X ( 1 ) T = W ( X J , ⋅ ( 1 ) ) − T I p ( 1 ) , q ( 1 ) ( X J , ⋅ ( 1 ) ) − T W T . (41)
Let Θ : = diag ( W 1 d ) \boldsymbol{\Theta} := \operatorname{diag}(\mathbf{W}\mathbf{1}_d) Θ := diag ( W 1 d ) and Z : = Θ − 1 W \mathbf{Z} := \boldsymbol{\Theta}^{-1}\mathbf{W} Z := Θ − 1 W . Then we define a diagonal matrix Θ \boldsymbol{\Theta} Θ and a membership matrix Z \mathbf{Z} Z such that W = Θ Z \mathbf{W} = \boldsymbol{\Theta}\mathbf{Z} W = Θ Z . Substituting W \mathbf{W} W with Θ Z \boldsymbol{\Theta}\mathbf{Z} Θ Z in (C.10), we have
P ( 1 ) = Θ Z ( X J , ⋅ ( 1 ) ) − T I p ( 1 ) , q ( 1 ) ( X J , ⋅ ( 1 ) ) − T Z T Θ . \mathbf{P}^{(1)} = \boldsymbol{\Theta}\mathbf{Z}(\mathbf{X}_{\mathcal{J},\cdot}^{(1)})^{-\mathsf{T}}\mathbf{I}_{p^{(1)},q^{(1)}}(\mathbf{X}_{\mathcal{J},\cdot}^{(1)})^{-\mathsf{T}}\mathbf{Z}^{\mathsf{T}}\boldsymbol{\Theta}. P ( 1 ) = Θ Z ( X J , ⋅ ( 1 ) ) − T I p ( 1 ) , q ( 1 ) ( X J , ⋅ ( 1 ) ) − T Z T Θ .
By taking B ( 1 ) : = ( X J , ⋅ ( 1 ) ) − T I p ( 1 ) , q ( 1 ) ( X J , ⋅ ( 1 ) ) − T \mathbf{B}^{(1)} := (\mathbf{X}_{\mathcal{J},\cdot}^{(1)})^{-\mathsf{T}}\mathbf{I}_{p^{(1)},q^{(1)}}(\mathbf{X}_{\mathcal{J},\cdot}^{(1)})^{-\mathsf{T}} B ( 1 ) := ( X J , ⋅ ( 1 ) ) − T I p ( 1 ) , q ( 1 ) ( X J , ⋅ ( 1 ) ) − T , we have an equivalent expression for the reference layer.
For non-reference layers, let l > 1 l > 1 l > 1 be given. Then P ( l ) = V R ( l ) V T \mathbf{P}^{(l)} = \mathbf{V}\mathbf{R}^{(l)}\mathbf{V}^{\mathsf{T}} P ( l ) = V R ( l ) V T . By definition of X ( 1 ) \mathbf{X}^{(1)} X ( 1 ) , it follows that
W = X ( 1 ) I p ( 1 ) , q ( 1 ) X J , ⋅ ( 1 ) T = V U ( 1 ) ∣ Λ ( 1 ) ∣ 1 / 2 I p ( 1 ) , q ( 1 ) X J , ⋅ ( 1 ) T . \mathbf{W} = \mathbf{X}^{(1)}\mathbf{I}_{p^{(1)},q^{(1)}}\mathbf{X}_{\mathcal{J},\cdot}^{(1)\mathsf{T}} = \mathbf{V}\mathbf{U}^{(1)}|\boldsymbol{\Lambda}^{(1)}|^{1/2}\mathbf{I}_{p^{(1)},q^{(1)}}\mathbf{X}_{\mathcal{J},\cdot}^{(1)\mathsf{T}}. W = X ( 1 ) I p ( 1 ) , q ( 1 ) X J , ⋅ ( 1 ) T = V U ( 1 ) ∣ Λ ( 1 ) ∣ 1/2 I p ( 1 ) , q ( 1 ) X J , ⋅ ( 1 ) T .
Since U ( 1 ) \mathbf{U}^{(1)} U ( 1 ) , Λ ( 1 ) \boldsymbol{\Lambda}^{(1)} Λ ( 1 ) , I p ( 1 ) , q ( 1 ) \mathbf{I}_{p^{(1)},q^{(1)}} I p ( 1 ) , q ( 1 ) and X J , ⋅ ( 1 ) T \mathbf{X}_{\mathcal{J},\cdot}^{(1)\mathsf{T}} X J , ⋅ ( 1 ) T are nonsingular, we have col ( W ) = col ( V ) \operatorname{col}(\mathbf{W}) = \operatorname{col}(\mathbf{V}) col ( W ) = col ( V ) and hence there exists a nonsingular matrix C \mathbf{C} C such that V = W C \mathbf{V} = \mathbf{W}\mathbf{C} V = WC . By substituting V \mathbf{V} V with W C \mathbf{W}\mathbf{C} WC , we have
P ( l ) = V R ( l ) V T = W C R ( l ) C T W T = Θ Z C R ( l ) C T Z T Θ . \mathbf{P}^{(l)} = \mathbf{V}\mathbf{R}^{(l)}\mathbf{V}^{\mathsf{T}} = \mathbf{W}\mathbf{C}\mathbf{R}^{(l)}\mathbf{C}^{\mathsf{T}}\mathbf{W}^{\mathsf{T}} = \boldsymbol{\Theta}\mathbf{Z}\mathbf{C}\mathbf{R}^{(l)}\mathbf{C}^{\mathsf{T}}\mathbf{Z}^{\mathsf{T}}\boldsymbol{\Theta}. P ( l ) = V R ( l ) V T = WC R ( l ) C T W T = Θ ZC R ( l ) C T Z T Θ .
By taking B ( l ) : = C R ( l ) C T \mathbf{B}^{(l)} := \mathbf{C}\mathbf{R}^{(l)}\mathbf{C}^{\mathsf{T}} B ( l ) := C R ( l ) C T , we obtain an equivalent expression of the common subspace model. This completes the proof.
Bi-Lipschitz Continuity of the Concatenated Embedding Map
The error bounds in Section B control the norm of difference between the population and empirical row-normalized spectral embeddings. To translate these embedding-space bounds into guarantees for membership recovery, we need to relate distances in the standard simplex to distances between the corresponding normalized embeddings. This section establishes such a relationship through a concatenated embedding map.
For z ∈ Δ K − 1 \mathbf{z} \in\Delta^{K-1} z ∈ Δ K − 1 , define
T ( z ) : = ( T ( 1 ) ( z ) T , … , T ( L ) ( z ) T ) T ∈ R K L , T(\mathbf{z}) := \left(T^{(1)}(\mathbf{z})^{\mathsf{T}},\ldots,T^{(L)}(\mathbf{z})^{\mathsf{T}}\right)^{\mathsf{T}} \in\mathbb{R}^{KL}, T ( z ) := ( T ( 1 ) ( z ) T , … , T ( L ) ( z ) T ) T ∈ R K L ,
where
T ( l ) ( z ) : = R ( l ) T z ∥ R ( l ) T z ∥ 2 . T^{(l)}(\mathbf{z}) := \frac{\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}}{\|\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}\|_2}. T ( l ) ( z ) := ∥ R ( l ) T z ∥ 2 R ( l ) T z .
In particular, for the membership vector Z i , ⋅ \mathbf{Z}_{i,\cdot} Z i , ⋅ of node i i i , this map satisfies
T ( Z i , ⋅ ) = ( ( Y i , ⋅ ( 1 ) ) ⊤ , … , ( Y i , ⋅ ( L ) ) ⊤ ) ⊤ . T(\mathbf{Z}_{i,\cdot}) = \left(\left(\mathbf{Y}_{i,\cdot}^{(1)}\right)^{\top},\ldots,\left(\mathbf{Y}_{i,\cdot}^{(L)}\right)^{\top}\right)^{\top}. T ( Z i , ⋅ ) = ( ( Y i , ⋅ ( 1 ) ) ⊤ , … , ( Y i , ⋅ ( L ) ) ⊤ ) ⊤ .
The goal of this section is to establish the bi-Lipschitz continuity of T T T on Δ K − 1 \Delta^{K-1} Δ K − 1 , which allows us to convert spectral embedding error bounds into membership recovery guarantees and to analyze the vertex hunting step. The argument proceeds in two steps: we first obtain bi-Lipschitz bounds for each layer-wise map T ( l ) T^{(l)} T ( l ) , and then combine the layer-wise bounds to obtain the desired result for the concatenated map T T T .
Definition of the layer-wise map T ( l ) T^{(l)} T ( l ) . For each l ∈ [ L ] l \in[L] l ∈ [ L ] , define
C R ( l ) : = conv { R 1 , ⋅ ( l ) , R 2 , ⋅ ( l ) , … , R K , ⋅ ( l ) } ⊂ R K . \mathcal{C}_{R}^{(l)} := \operatorname{conv}\left\{\mathbf{R}_{1,\cdot}^{(l)},\mathbf{R}_{2,\cdot}^{(l)},\ldots,\mathbf{R}_{K,\cdot}^{(l)}\right\} \subset\mathbb{R}^{K}. C R ( l ) := conv { R 1 , ⋅ ( l ) , R 2 , ⋅ ( l ) , … , R K , ⋅ ( l ) } ⊂ R K .
When R ( l ) \mathbf{R}^{(l)} R ( l ) is nonsingular, C R ( l ) \mathcal{C}_{R}^{(l)} C R ( l ) is a ( K − 1 ) (K-1) ( K − 1 ) -simplex whose vertices are the rows of R ( l ) \mathbf{R}^{(l)} R ( l ) . Then, there exists a normal vector
n ( l ) = ( R ( l ) ) − 1 1 K (42) \mathbf{n}^{(l)} = \left(\mathbf{R}^{(l)}\right)^{-1}\mathbf{1}_{K} \tag*{(42)} n ( l ) = ( R ( l ) ) − 1 1 K (42)
such that C R ( l ) \mathcal{C}_{R}^{(l)} C R ( l ) is a subset of the hyperplane of H R ( l ) \mathcal{H}_{R}^{(l)} H R ( l ) defined by
H R ( l ) = { ( x 1 , x 2 , ⋯ , x K ) ∈ R K : ⟨ n ( l ) , x ⟩ = 1 } . (43) \mathcal{H}_{R}^{(l)} = \left\{(x_{1},x_{2},\cdots,x_{K}) \in\mathbb{R}^{K} : \langle\mathbf{n}^{(l)},\mathbf{x}\rangle= 1\right\}. \tag*{(43)} H R ( l ) = { ( x 1 , x 2 , ⋯ , x K ) ∈ R K : ⟨ n ( l ) , x ⟩ = 1 } . (43)
Define a linear transformation
T 1 ( l ) : Δ K − 1 → C R ( l ) by T 1 ( x ) = ( R ( l ) ) ⊤ x . T_{1}^{(l)}:\Delta^{K-1}\to\mathcal{C}_{R}^{(l)} \text{ by } T_{1}(\mathbf{x})=\left(\mathbf{R}^{(l)}\right)^{\top}\mathbf{x}. T 1 ( l ) : Δ K − 1 → C R ( l ) by T 1 ( x ) = ( R ( l ) ) ⊤ x .
Also define the ℓ 2 \ell_{2} ℓ 2 -normalized image of C R ( l ) \mathcal{C}_{R}^{(l)} C R ( l ) by
S R ( l ) = { ( x 1 , x 2 , ⋯ , x K ) ∈ R K : x = y ∥ y ∥ 2 , y ∈ C R ( l ) } ⊆ S K − 1 . (44) \mathcal{S}_{R}^{(l)} = \left\{(x_{1},x_{2},\cdots,x_{K}) \in\mathbb{R}^{K} : \mathbf{x} = \frac{\mathbf{y}}{\|\mathbf{y}\|_{2}},\ \mathbf{y} \in\mathcal{C}_{R}^{(l)}\right\} \subseteq\mathbb{S}^{K-1}. \tag*{(44)} S R ( l ) = { ( x 1 , x 2 , ⋯ , x K ) ∈ R K : x = ∥ y ∥ 2 y , y ∈ C R ( l ) } ⊆ S K − 1 . (44)
Let T 2 ( l ) : C R ( l ) → S R ( l ) T_{2}^{(l)}:\mathcal{C}_{R}^{(l)}\to\mathcal{S}_{R}^{(l)} T 2 ( l ) : C R ( l ) → S R ( l ) denote the ℓ 2 \ell_{2} ℓ 2 -norm normalization function. Then we can define the layer-wise map T ( l ) T^{(l)} T ( l ) by following:
T ( l ) = T 2 ( l ) ∘ T 1 ( l ) . T^{(l)} = T_{2}^{(l)} \circ T_{1}^{(l)}. T ( l ) = T 2 ( l ) ∘ T 1 ( l ) .
We derive lower and upper Lipschitz bounds for T 1 ( l ) T_{1}^{(l)} T 1 ( l ) and T 2 ( l ) T_{2}^{(l)} T 2 ( l ) separately. These bounds imply corresponding bi-Lipschitz bounds for T ( l ) T^{(l)} T ( l ) by composition. Finally, the bi-Lipschitz constants for the concatenated map T T T are obtained by combining the layer-wise bounds across l = 1 , … , L l=1,\ldots,L l = 1 , … , L .
Bi-Lipschitz continuity of T ( l ) T^{(l)} T ( l ) . Note that T 1 ( l ) ( x ) = ( R ( l ) ) ⊤ x T_{1}^{(l)}(\mathbf{x})=\left(\mathbf{R}^{(l)}\right)^{\top}\mathbf{x} T 1 ( l ) ( x ) = ( R ( l ) ) ⊤ x , where R ( l ) \mathbf{R}^{(l)} R ( l ) is a full rank matrix in R K × K \mathbb{R}^{K\times K} R K × K . Then, for any x , y ∈ R K \mathbf{x},\mathbf{y}\in\mathbb{R}^{K} x , y ∈ R K ,
σ min ( ( R ( l ) ) ⊤ ) ∥ x − y ∥ 2 ≤ ∥ T 1 ( l ) ( x ) − T 1 ( l ) ( y ) ∥ 2 ≤ σ max ( ( R ( l ) ) ⊤ ) ∥ x − y ∥ 2 . (45) \sigma_{\min}\left(\left(\mathbf{R}^{(l)}\right)^{\top}\right)\|\mathbf{x}-\mathbf{y}\|_{2}
\leq
\left\|T_{1}^{(l)}(\mathbf{x})-T_{1}^{(l)}(\mathbf{y})\right\|_{2}
\leq
\sigma_{\max}\left(\left(\mathbf{R}^{(l)}\right)^{\top}\right)\|\mathbf{x}-\mathbf{y}\|_{2}. \tag*{(45)} σ m i n ( ( R ( l ) ) ⊤ ) ∥ x − y ∥ 2 ≤ T 1 ( l ) ( x ) − T 1 ( l ) ( y ) 2 ≤ σ m a x ( ( R ( l ) ) ⊤ ) ∥ x − y ∥ 2 . (45)
Since R ( l ) \mathbf{R}^{(l)} R ( l ) is a full rank matrix, σ min ( R ( l ) ) > 0 \sigma_{\min}(\mathbf{R}^{(l)})>0 σ m i n ( R ( l ) ) > 0 . As shown in Section C.2, we have the following:
σ B ( l ) ∥ x − y ∥ 2 ≲ ∥ T 1 ( l ) ( x ) − T 1 ( l ) ( y ) ∥ 2 ≲ K ∥ x − y ∥ 2 . \sqrt{\sigma_{B}^{(l)}}\|\mathbf{x}-\mathbf{y}\|_{2}
\lesssim
\left\|T_{1}^{(l)}(\mathbf{x})-T_{1}^{(l)}(\mathbf{y})\right\|_{2}
\lesssim
\sqrt{K}\|\mathbf{x}-\mathbf{y}\|_{2}. σ B ( l ) ∥ x − y ∥ 2 ≲ T 1 ( l ) ( x ) − T 1 ( l ) ( y ) 2 ≲ K ∥ x − y ∥ 2 .
Bi-Lipschitz continuity of T 2 ( l ) T_2^{(l)} T 2 ( l ) . First, we show the transformation T 2 ( l ) T_2^{(l)} T 2 ( l ) is C 1 C^1 C 1 on R K ∖ { 0 } \mathbb{R}^K \setminus\{0\} R K ∖ { 0 } . It is known that T 2 ( l ) T_2^{(l)} T 2 ( l ) is differentiable at a point x \mathbf{x} x if and only if the component functions
( T 2 ( l ) ( x ) ) k = x k ∥ x ∥ 2 \left(T_2^{(l)}(\mathbf{x})\right)_k = \frac{x_k}{\lVert\mathbf{x}\rVert_2} ( T 2 ( l ) ( x ) ) k = ∥ x ∥ 2 x k
is differentiable for all 1 ≤ k ≤ K 1 \leq k \leq K 1 ≤ k ≤ K . For each component functions, we obtain partial derivatives by
∂ ( T 2 ( l ) ( x ) ) k ∂ x j = { − x k x j ∥ x ∥ 3 if k ≠ j , 1 ∥ x ∥ − x j 2 ∥ x ∥ 3 if k = j . \frac{\partial\left(T_2^{(l)}(\mathbf{x})\right)_k}{\partial x_j} =
\begin{cases}
-\dfrac{x_kx_j}{\lVert\mathbf{x}\rVert^3} & \text{if } k \ne j,\\
\dfrac{1}{\lVert\mathbf{x}\rVert}-\dfrac{x_j^2}{\lVert\mathbf{x}\rVert^3} & \text{if } k = j.
\end{cases} ∂ x j ∂ ( T 2 ( l ) ( x ) ) k = ⎩ ⎨ ⎧ − ∥ x ∥ 3 x k x j ∥ x ∥ 1 − ∥ x ∥ 3 x j 2 if k = j , if k = j .
Since all partial derivatives are well-defined and continuous, T 2 ( l ) T_2^{(l)} T 2 ( l ) is C 1 C^1 C 1 on R K ∖ { 0 } \mathbb{R}^K \setminus\{0\} R K ∖ { 0 } . Next, we show Lipschitz continuity of T 2 ( l ) T_2^{(l)} T 2 ( l ) by finding a constant L 2 , U ( l ) > 0 L_{2,U}^{(l)} > 0 L 2 , U ( l ) > 0 such that
∥ T 2 ( l ) ( x ) − T 2 ( l ) ( y ) ∥ ≤ L 2 , U ( l ) ∥ x − y ∥ . \left\lVert T_2^{(l)}(\mathbf{x})-T_2^{(l)}(\mathbf{y})\right\rVert\leq L_{2,U}^{(l)}\lVert\mathbf{x}-\mathbf{y}\rVert. T 2 ( l ) ( x ) − T 2 ( l ) ( y ) ≤ L 2 , U ( l ) ∥ x − y ∥ .
Recall one useful equation (D.5) from the fundamental theorem of calculus:
T 2 ( l ) ( y ) − T 2 ( l ) ( x ) = ∫ 0 1 D T 2 ( l ) ( x + t ( y − x ) ) ( y − x ) d t , (46) T_2^{(l)}(\mathbf{y})-T_2^{(l)}(\mathbf{x})=\int_0^1DT_2^{(l)}(\mathbf{x}+t(\mathbf{y}-\mathbf{x}))(\mathbf{y}-\mathbf{x})\mathrm{d}t, \tag*{(46)} T 2 ( l ) ( y ) − T 2 ( l ) ( x ) = ∫ 0 1 D T 2 ( l ) ( x + t ( y − x )) ( y − x ) d t , (46)
where
D T 2 ( l ) ( x ) = I K ∥ x ∥ − x x T ∥ x ∥ 3 . DT_2^{(l)}(\mathbf{x})=\frac{\mathbf{I}_K}{\lVert\mathbf{x}\rVert}-\frac{\mathbf{x}\mathbf{x}^{\mathsf{T}}}{\lVert\mathbf{x}\rVert^3}. D T 2 ( l ) ( x ) = ∥ x ∥ I K − ∥ x ∥ 3 x x T .
is the Jacobian matrix of the transformation T 2 ( l ) T_2^{(l)} T 2 ( l ) at the point x \mathbf{x} x . Let
σ U ( l ) = sup x ∈ C R ( l ) σ max ( D T 2 ( l ) ( x ) ) . \sigma_U^{(l)}=\sup_{\mathbf{x}\in\mathcal{C}_R^{(l)}}\sigma_{\max}\left(DT_2^{(l)}(\mathbf{x})\right). σ U ( l ) = x ∈ C R ( l ) sup σ m a x ( D T 2 ( l ) ( x ) ) .
and x t = x + t ( y − x ) \mathbf{x}_t=\mathbf{x}+t(\mathbf{y}-\mathbf{x}) x t = x + t ( y − x ) for t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] . Note that σ U ( l ) \sigma_U^{(l)} σ U ( l ) is finite by the extreme value theorem. Now we bound σ U ( l ) \sigma_U^{(l)} σ U ( l ) :
σ U ( l ) = sup x ∈ C R ( l ) σ max ( D T 2 ( l ) ( x ) ) = sup x ∈ C R ( l ) σ max ( I K ∥ x ∥ − x x T ∥ x ∥ 3 ) = sup x ∈ C R ( l ) 1 ∥ x ∥ = 1 inf x ∈ C R ( l ) ∥ x ∥ . \begin{aligned}
\sigma_U^{(l)}
&=\sup_{\mathbf{x}\in\mathcal{C}_R^{(l)}}\sigma_{\max}\left(DT_2^{(l)}(\mathbf{x})\right)\\
&=\sup_{\mathbf{x}\in\mathcal{C}_R^{(l)}}\sigma_{\max}\left(\frac{\mathbf{I}_K}{\lVert\mathbf{x}\rVert}-\frac{\mathbf{x}\mathbf{x}^{\mathsf{T}}}{\lVert\mathbf{x}\rVert^3}\right)\\
&=\sup_{\mathbf{x}\in\mathcal{C}_R^{(l)}}\frac{1}{\lVert\mathbf{x}\rVert}\\
&=\frac{1}{\inf_{\mathbf{x}\in\mathcal{C}_R^{(l)}}\lVert\mathbf{x}\rVert}.
\end{aligned} σ U ( l ) = x ∈ C R ( l ) sup σ m a x ( D T 2 ( l ) ( x ) ) = x ∈ C R ( l ) sup σ m a x ( ∥ x ∥ I K − ∥ x ∥ 3 x x T ) = x ∈ C R ( l ) sup ∥ x ∥ 1 = inf x ∈ C R ( l ) ∥ x ∥ 1 .
For any x ∈ C R ( l ) \mathbf{x}\in\mathcal{C}_R^{(l)} x ∈ C R ( l ) , by Cauchy-Schwarz inequality, we obtain
⟨ x , n ( l ) ⟩ ≤ ∥ x ∥ 2 ∥ n ( l ) ∥ 2 ⟺ 1 ∥ x ∥ 2 ≤ ∥ n ( l ) ∥ 2 ⟨ x , n ( l ) ⟩ = ∥ n ( l ) ∥ 2 . \langle\mathbf{x},\mathbf{n}^{(l)}\rangle\leq\lVert\mathbf{x}\rVert_2\lVert\mathbf{n}^{(l)}\rVert_2\Longleftrightarrow\frac{1}{\lVert\mathbf{x}\rVert_2}\leq\frac{\lVert\mathbf{n}^{(l)}\rVert_2}{\langle\mathbf{x},\mathbf{n}^{(l)}\rangle}=\lVert\mathbf{n}^{(l)}\rVert_2. ⟨ x , n ( l ) ⟩ ≤ ∥ x ∥ 2 ∥ n ( l ) ∥ 2 ⟺ ∥ x ∥ 2 1 ≤ ⟨ x , n ( l ) ⟩ ∥ n ( l ) ∥ 2 = ∥ n ( l ) ∥ 2 .
Hence, by (C.4), we obtain
∥ n ( l ) ∥ = ∥ ( R ( l ) ) − 1 1 K ∥ ≤ ∥ ( R ( l ) ) − 1 ∥ ∥ 1 K ∥ 2 ≲ K σ B ( l ) . (47) \begin{aligned}
\lVert\mathbf{n}^{(l)}\rVert
&=\left\lVert\left(R^{(l)}\right)^{-1}\mathbf{1}_K\right\rVert\\
&\leq\left\lVert\left(R^{(l)}\right)^{-1}\right\rVert\lVert\mathbf{1}_K\rVert_2\\
&\lesssim\frac{\sqrt{K}}{\sqrt{\sigma_B^{(l)}}}. \tag*{(47)}
\end{aligned} ∥ n ( l ) ∥ = ( R ( l ) ) − 1 1 K ≤ ( R ( l ) ) − 1 ∥ 1 K ∥ 2 ≲ σ B ( l ) K . (47)
We show T 2 ( l ) T_2^{(l)} T 2 ( l ) is Lipschitz continuous. Since C R ( l ) C_R^{(l)} C R ( l ) is convex and 0 ∉ C R ( l ) 0 \notin C_R^{(l)} 0 ∈ / C R ( l ) , the line segment between any two points of C R ( l ) C_R^{(l)} C R ( l ) lies in the domain of T 2 ( l ) T_2^{(l)} T 2 ( l ) . Hence, by the fundamental theorem of calculus and (D.6),
∥ T 2 ( l ) ( y ) − T 2 ( l ) ( x ) ∥ = ∥ ∫ 0 1 D T 2 ( l ) ( x + t ( y − x ) ) ( y − x ) d t ∥ ≤ ∫ 0 1 ∥ D T 2 ( l ) ( x + t ( y − x ) ) ( y − x ) ∥ d t ≤ ∫ 0 1 σ max ( D T 2 ( l ) ( x + t ( y − x ) ) ) ∥ y − x ∥ 2 d t ≤ ∫ 0 1 σ U ( l ) ∥ y − x ∥ 2 d t = σ U ( l ) ∥ y − x ∥ 2 ≲ K σ B ( l ) ∥ y − x ∥ 2 (48) \begin{aligned}
\left\|T_2^{(l)}(\mathbf{y})-T_2^{(l)}(\mathbf{x})\right\|
&=\left\|\int_0^1 DT_2^{(l)}(\mathbf{x}+t(\mathbf{y}-\mathbf{x}))(\mathbf{y}-\mathbf{x})\,dt\right\|\\
&\leq\int_0^1\left\|DT_2^{(l)}(\mathbf{x}+t(\mathbf{y}-\mathbf{x}))(\mathbf{y}-\mathbf{x})\right\|\,dt\\
&\leq\int_0^1\sigma_{\max}\left(DT_2^{(l)}(\mathbf{x}+t(\mathbf{y}-\mathbf{x}))\right)\|\mathbf{y}-\mathbf{x}\|_2\,dt\\
&\leq\int_0^1\sigma_U^{(l)}\|\mathbf{y}-\mathbf{x}\|_2\,dt\\
&=\sigma_U^{(l)}\|\mathbf{y}-\mathbf{x}\|_2\\
&\lesssim\frac{\sqrt{K}}{\sqrt{\sigma_B^{(l)}}}\|\mathbf{y}-\mathbf{x}\|_2 \tag*{(48)}
\end{aligned} T 2 ( l ) ( y ) − T 2 ( l ) ( x ) = ∫ 0 1 D T 2 ( l ) ( x + t ( y − x )) ( y − x ) d t ≤ ∫ 0 1 D T 2 ( l ) ( x + t ( y − x )) ( y − x ) d t ≤ ∫ 0 1 σ m a x ( D T 2 ( l ) ( x + t ( y − x )) ) ∥ y − x ∥ 2 d t ≤ ∫ 0 1 σ U ( l ) ∥ y − x ∥ 2 d t = σ U ( l ) ∥ y − x ∥ 2 ≲ σ B ( l ) K ∥ y − x ∥ 2 (48)
Similarly, we show ( T 2 ( l ) ) − 1 (T_2^{(l)})^{-1} ( T 2 ( l ) ) − 1 is Lipschitz continuous. Let x ∈ C R ( l ) \mathbf{x}\in C_R^{(l)} x ∈ C R ( l ) and y = x ∥ x ∥ 2 \mathbf{y}=\frac{\mathbf{x}}{\|\mathbf{x}\|_2} y = ∥ x ∥ 2 x be given. Since x ∈ C R ( l ) ⊂ H R ( l ) \mathbf{x}\in C_R^{(l)}\subset\mathcal{H}_R^{(l)} x ∈ C R ( l ) ⊂ H R ( l ) , we have ⟨ n ( l ) , x ⟩ = 1 \langle\mathbf{n}^{(l)},\mathbf{x}\rangle=1 ⟨ n ( l ) , x ⟩ = 1 . Hence,
⟨ n ( l ) , y ⟩ = 1 ∥ x ∥ 2 ⟨ n ( l ) , x ⟩ = 1 ∥ x ∥ 2 . (49) \langle\mathbf{n}^{(l)},\mathbf{y}\rangle=\frac{1}{\|\mathbf{x}\|_2}\langle\mathbf{n}^{(l)},\mathbf{x}\rangle=\frac{1}{\|\mathbf{x}\|_2}. \tag*{(49)} ⟨ n ( l ) , y ⟩ = ∥ x ∥ 2 1 ⟨ n ( l ) , x ⟩ = ∥ x ∥ 2 1 . (49)
We obtain an explicit form of ( T 2 ( l ) ) − 1 ( y ) (T_2^{(l)})^{-1}(\mathbf{y}) ( T 2 ( l ) ) − 1 ( y ) by following:
( T 2 ( l ) ) − 1 ( y ) = x = ∥ x ∥ 2 y = y ⟨ n ( l ) , y ⟩ . (T_2^{(l)})^{-1}(\mathbf{y})=\mathbf{x}=\|\mathbf{x}\|_2\mathbf{y}=\frac{\mathbf{y}}{\langle\mathbf{n}^{(l)},\mathbf{y}\rangle}. ( T 2 ( l ) ) − 1 ( y ) = x = ∥ x ∥ 2 y = ⟨ n ( l ) , y ⟩ y .
Note that, however, we cannot directly use the fundamental theorem of calculus since a line segment between two points in S R ( l ) S_R^{(l)} S R ( l ) is not guaranteed to be included in S R ( l ) S_R^{(l)} S R ( l ) . Therefore, we define an open half-space U ( l ) \mathcal{U}^{(l)} U ( l ) and a transformation F ( l ) F^{(l)} F ( l ) on U ( l ) \mathcal{U}^{(l)} U ( l ) by following:
U ( l ) = { u ∈ R K : ⟨ n ( l ) , u ⟩ > 0 } \mathcal{U}^{(l)}=\left\{\mathbf{u}\in\mathbb{R}^{K}:\langle\mathbf{n}^{(l)},\mathbf{u}\rangle>0\right\} U ( l ) = { u ∈ R K : ⟨ n ( l ) , u ⟩ > 0 }
and
F ( l ) ( u ) = u ⟨ n ( l ) , u ⟩ . F^{(l)}(\mathbf{u})=\frac{\mathbf{u}}{\langle\mathbf{n}^{(l)},\mathbf{u}\rangle}. F ( l ) ( u ) = ⟨ n ( l ) , u ⟩ u .
It is clear that S R ( l ) ⊂ U ( l ) S_R^{(l)}\subset\mathcal{U}^{(l)} S R ( l ) ⊂ U ( l ) and F ( l ) = ( T 2 ( l ) ) − 1 F^{(l)}=(T_2^{(l)})^{-1} F ( l ) = ( T 2 ( l ) ) − 1 on S R ( l ) S_R^{(l)} S R ( l ) . Moreover, F ( l ) ∈ C 1 ( U ( l ) , R K ) F^{(l)}\in C^1(\mathcal{U}^{(l)},\mathbb{R}^{K}) F ( l ) ∈ C 1 ( U ( l ) , R K ) since both numerator and denominator are C 1 C^1 C 1 . Hence we can show Lipschitz continuity of F ( l ) F^{(l)} F ( l ) by the fundamental theorem of calculus.
Let u 1 , u 2 ∈ S R ( l ) \mathbf{u}_1,\mathbf{u}_2\in S_R^{(l)} u 1 , u 2 ∈ S R ( l ) be given. By definition of S R ( l ) S_R^{(l)} S R ( l ) , there exist x 1 , x 2 ∈ C R ( l ) \mathbf{x}_1,\mathbf{x}_2\in C_R^{(l)} x 1 , x 2 ∈ C R ( l ) such that
u j = x j ∥ x j ∥ 2 , j = 1 , 2. \mathbf{u}_j=\frac{\mathbf{x}_j}{\|\mathbf{x}_j\|_2},\qquad j=1,2. u j = ∥ x j ∥ 2 x j , j = 1 , 2.
We first bound the operator norm of the Jacobian D F ( l ) DF^{(l)} D F ( l ) along the line segment between u 1 \mathbf{u}_1 u 1 and u 2 \mathbf{u}_2 u 2 . For t ∈ [ 0 , 1 ] t\in[0,1] t ∈ [ 0 , 1 ] , let
u t : = u 1 + t ( u 2 − u 1 ) , \mathbf{u}_t:=\mathbf{u}_1+t(\mathbf{u}_2-\mathbf{u}_1), u t := u 1 + t ( u 2 − u 1 ) ,
which parametrizes the line segment between u 1 \mathbf{u}_1 u 1 and u 2 \mathbf{u}_2 u 2 . Using the triangle inequality and the identity ∥ u 1 u 2 ⊤ ∥ = ∥ u 1 ∥ 2 ∥ u 2 ∥ 2 \lVert\mathbf{u}_1\mathbf{u}_2^{\top}\rVert=\lVert\mathbf{u}_1\rVert_2\lVert\mathbf{u}_2\rVert_2 ∥ u 1 u 2 ⊤ ∥ = ∥ u 1 ∥ 2 ∥ u 2 ∥ 2 , we obtain
∥ D F ( l ) ( u t ) ∥ = ∥ I K ⟨ n ( l ) , u t ⟩ − u t n ( l ) ⊤ ⟨ n ( l ) , u t ⟩ 2 ∥ ≤ ∥ I K ⟨ n ( l ) , u t ⟩ ∥ + ∥ u t n ( l ) ⊤ ⟨ n ( l ) , u t ⟩ 2 ∥ ≤ 1 ⟨ n ( l ) , u t ⟩ + ∥ u t ∥ 2 ∥ n ( l ) ∥ 2 ⟨ n ( l ) , u t ⟩ 2 . (50) \begin{aligned}
\lVert DF^{(l)}(\mathbf{u}_t)\rVert
&=\left\lVert\frac{I_K}{\langle\mathbf{n}^{(l)},\mathbf{u}_t\rangle}-\frac{\mathbf{u}_t\mathbf{n}^{(l)\top}}{\langle\mathbf{n}^{(l)},\mathbf{u}_t\rangle^2}\right\rVert\\
&\le\left\lVert\frac{I_K}{\langle\mathbf{n}^{(l)},\mathbf{u}_t\rangle}\right\rVert+\left\lVert\frac{\mathbf{u}_t\mathbf{n}^{(l)\top}}{\langle\mathbf{n}^{(l)},\mathbf{u}_t\rangle^2}\right\rVert\\
&\le\frac{1}{\langle\mathbf{n}^{(l)},\mathbf{u}_t\rangle}+\frac{\lVert\mathbf{u}_t\rVert_2\lVert\mathbf{n}^{(l)}\rVert_2}{\langle\mathbf{n}^{(l)},\mathbf{u}_t\rangle^2}. \tag*{(50)}
\end{aligned} ∥ D F ( l ) ( u t )∥ = ⟨ n ( l ) , u t ⟩ I K − ⟨ n ( l ) , u t ⟩ 2 u t n ( l ) ⊤ ≤ ⟨ n ( l ) , u t ⟩ I K + ⟨ n ( l ) , u t ⟩ 2 u t n ( l ) ⊤ ≤ ⟨ n ( l ) , u t ⟩ 1 + ⟨ n ( l ) , u t ⟩ 2 ∥ u t ∥ 2 ∥ n ( l ) ∥ 2 . (50)
We bound each term in (D.9). By triangle inequality, we have
∥ u t ∥ = ∥ ( 1 − t ) u 1 + t u 2 ∥ ≤ ( 1 − t ) ∥ u 1 ∥ + t ∥ u 2 ∥ = 1. (51) \lVert\mathbf{u}_t\rVert=\lVert(1-t)\mathbf{u}_1+t\mathbf{u}_2\rVert\le(1-t)\lVert\mathbf{u}_1\rVert+t\lVert\mathbf{u}_2\rVert=1. \tag*{(51)} ∥ u t ∥ = ∥( 1 − t ) u 1 + t u 2 ∥ ≤ ( 1 − t ) ∥ u 1 ∥ + t ∥ u 2 ∥ = 1. (51)
By linearity of inner product,
⟨ n ( l ) , u t ⟩ = ⟨ n ( l ) , ( 1 − t ) u 1 + t u 2 ⟩ = ( 1 − t ) ⟨ n ( l ) , u 1 ⟩ + t ⟨ n ( l ) , u 2 ⟩ = ( 1 − t ) 1 ∥ x 1 ∥ 2 + t 1 ∥ x 2 ∥ 2 ≥ 1 sup x ∈ C R ( l ) ∥ x ∥ = 1 max 1 ≤ k ≤ K ∥ R k ( l ) ∥ 2 ≥ 1 ∥ R ( l ) ∥ . \begin{aligned}
\langle\mathbf{n}^{(l)},\mathbf{u}_t\rangle
&=\langle\mathbf{n}^{(l)},(1-t)\mathbf{u}_1+t\mathbf{u}_2\rangle\\
&=(1-t)\langle\mathbf{n}^{(l)},\mathbf{u}_1\rangle+t\langle\mathbf{n}^{(l)},\mathbf{u}_2\rangle\\
&=(1-t)\frac{1}{\lVert\mathbf{x}_1\rVert_2}+t\frac{1}{\lVert\mathbf{x}_2\rVert_2}\\
&\ge\frac{1}{\sup_{\mathbf{x}\in C_{\mathbb{R}^{(l)}}}\lVert\mathbf{x}\rVert}\\
&=\frac{1}{\max_{1\le k\le K}\lVert\mathbf{R}^{(l)}_k\rVert_2}\\
&\ge\frac{1}{\lVert\mathbf{R}^{(l)}\rVert}.
\end{aligned} ⟨ n ( l ) , u t ⟩ = ⟨ n ( l ) , ( 1 − t ) u 1 + t u 2 ⟩ = ( 1 − t ) ⟨ n ( l ) , u 1 ⟩ + t ⟨ n ( l ) , u 2 ⟩ = ( 1 − t ) ∥ x 1 ∥ 2 1 + t ∥ x 2 ∥ 2 1 ≥ sup x ∈ C R ( l ) ∥ x ∥ 1 = max 1 ≤ k ≤ K ∥ R k ( l ) ∥ 2 1 ≥ ∥ R ( l ) ∥ 1 .
Moreover, by (C.3), we have
⟨ n ( l ) , u t ⟩ ≳ 1 K . (52) \langle\mathbf{n}^{(l)},\mathbf{u}_t\rangle\gtrsim\frac{1}{\sqrt{K}}. \tag*{(52)} ⟨ n ( l ) , u t ⟩ ≳ K 1 . (52)
By substituting each bound (D.6), (D.10), and (D.11) to (D.9)`, we have
sup t ∈ [ 0 , 1 ] ∥ D F ( l ) ( u t ) ∥ ≲ K + K 3 / 2 σ B ( l ) ≲ K 3 / 2 σ B ( l ) . (53) \sup_{t\in[0,1]}\lVert DF^{(l)}(\mathbf{u}_t)\rVert\lesssim\sqrt{K}+\frac{K^{3/2}}{\sqrt{\sigma_B^{(l)}}}\lesssim\frac{K^{3/2}}{\sqrt{\sigma_B^{(l)}}}. \tag*{(53)} t ∈ [ 0 , 1 ] sup ∥ D F ( l ) ( u t )∥ ≲ K + σ B ( l ) K 3/2 ≲ σ B ( l ) K 3/2 . (53)
We show ( T 2 ( l ) ) − 1 (T_2^{(l)})^{-1} ( T 2 ( l ) ) − 1 is Lipschitz continuous. By the fundamental theorem of calculus and (D.12), we have
∥ ( T 2 ( l ) ) − 1 ( u 2 ) − ( T 2 ( l ) ) − 1 ( u 1 ) ∥ 2 = ∥ F ( l ) ( u 2 ) − F ( l ) ( u 1 ) ∥ 2 = ∥ ∫ 0 1 D F ( l ) ( u t ) ( u 2 − u 1 ) d t ∥ 2 ≤ ∫ 0 1 ∥ D F ( l ) ( u t ) ( u 2 − u 1 ) ∥ d t ≤ ∫ 0 1 ∥ D F ( l ) ( u t ) ∥ ∥ u 2 − u 1 ∥ 2 d t ≲ ∫ 0 1 K 3 / 2 σ B ( l ) ∥ u 2 − u 1 ∥ 2 d t = K 3 / 2 σ B ( l ) ∥ u 2 − u 1 ∥ 2 . (54) \begin{aligned}
\lVert(T_2^{(l)})^{-1}(\mathbf{u}_2)-(T_2^{(l)})^{-1}(\mathbf{u}_1)\rVert_2
&=\lVert F^{(l)}(\mathbf{u}_2)-F^{(l)}(\mathbf{u}_1)\rVert_2\\
&=\left\lVert\int_0^1DF^{(l)}(\mathbf{u}_t)(\mathbf{u}_2-\mathbf{u}_1)\,dt\right\rVert_2\\
&\le\int_0^1\lVert DF^{(l)}(\mathbf{u}_t)(\mathbf{u}_2-\mathbf{u}_1)\rVert\,dt\\
&\le\int_0^1\lVert DF^{(l)}(\mathbf{u}_t)\rVert\lVert\mathbf{u}_2-\mathbf{u}_1\rVert_2\,dt\\
&\lesssim\int_0^1\frac{K^{3/2}}{\sqrt{\sigma_B^{(l)}}}\lVert\mathbf{u}_2-\mathbf{u}_1\rVert_2\,dt\\
&=\frac{K^{3/2}}{\sqrt{\sigma_B^{(l)}}}\lVert\mathbf{u}_2-\mathbf{u}_1\rVert_2. \tag*{(54)}
\end{aligned} ∥( T 2 ( l ) ) − 1 ( u 2 ) − ( T 2 ( l ) ) − 1 ( u 1 ) ∥ 2 = ∥ F ( l ) ( u 2 ) − F ( l ) ( u 1 ) ∥ 2 = ∫ 0 1 D F ( l ) ( u t ) ( u 2 − u 1 ) d t 2 ≤ ∫ 0 1 ∥ D F ( l ) ( u t ) ( u 2 − u 1 )∥ d t ≤ ∫ 0 1 ∥ D F ( l ) ( u t )∥ ∥ u 2 − u 1 ∥ 2 d t ≲ ∫ 0 1 σ B ( l ) K 3/2 ∥ u 2 − u 1 ∥ 2 d t = σ B ( l ) K 3/2 ∥ u 2 − u 1 ∥ 2 . (54)
By (D.7) and (D.13), we have
σ B ( l ) K 3 / 2 ∥ x 2 − x 1 ∥ 2 ≲ ∥ T 2 ( l ) ( x 2 ) − T 2 ( l ) ( x 1 ) ∥ ≲ K σ B ( l ) ∥ x 2 − x 1 ∥ 2 . (55) \frac{\sqrt{\sigma_B^{(l)}}}{K^{3/2}}\lVert x_2-x_1\rVert_2 \lesssim\lVert T_2^{(l)}(x_2)-T_2^{(l)}(x_1)\rVert\lesssim\frac{\sqrt{K}}{\sqrt{\sigma_B^{(l)}}}\lVert x_2-x_1\rVert_2. \tag*{(55)} K 3/2 σ B ( l ) ∥ x 2 − x 1 ∥ 2 ≲ ∥ T 2 ( l ) ( x 2 ) − T 2 ( l ) ( x 1 )∥ ≲ σ B ( l ) K ∥ x 2 − x 1 ∥ 2 . (55)
Bi-Lipschitz continuity of T ( l ) T^{(l)} T ( l ) . Let z 1 , z 2 ∈ Δ K − 1 z_1,z_2\in\Delta^{K-1} z 1 , z 2 ∈ Δ K − 1 be given, and set
x j : = T 1 ( l ) ( z j ) = R ( l ) ⊤ z j , j = 1 , 2. x_j:=T_1^{(l)}(z_j)=R^{(l)\top}z_j,\qquad j=1,2. x j := T 1 ( l ) ( z j ) = R ( l ) ⊤ z j , j = 1 , 2.
Then x 1 , x 2 ∈ C R ( l ) x_1,x_2\in\mathcal{C}_R^{(l)} x 1 , x 2 ∈ C R ( l ) , and
T ( l ) ( z j ) = T 2 ( l ) ( x j ) , j = 1 , 2. T^{(l)}(z_j)=T_2^{(l)}(x_j),\qquad j=1,2. T ( l ) ( z j ) = T 2 ( l ) ( x j ) , j = 1 , 2.
By (D.4) and the singular-value bounds for R ( l ) R^{(l)} R ( l ) in Section C.2,
σ B ( l ) ∥ z 2 − z 1 ∥ 2 ≲ ∥ x 2 − x 1 ∥ 2 ≲ K ∥ z 2 − z 1 ∥ 2 . \sqrt{\sigma_B^{(l)}}\lVert z_2-z_1\rVert_2\lesssim\lVert x_2-x_1\rVert_2\lesssim\sqrt{K}\lVert z_2-z_1\rVert_2. σ B ( l ) ∥ z 2 − z 1 ∥ 2 ≲ ∥ x 2 − x 1 ∥ 2 ≲ K ∥ z 2 − z 1 ∥ 2 .
Combining this with (D.14), we obtain
∥ T ( l ) ( z 2 ) − T ( l ) ( z 1 ) ∥ 2 = ∥ T 2 ( l ) ( x 2 ) − T 2 ( l ) ( x 1 ) ∥ 2 ≳ σ B ( l ) K 3 / 2 ∥ x 2 − x 1 ∥ 2 ≳ σ B ( l ) K 3 / 2 ∥ z 2 − z 1 ∥ 2 , (56) \begin{aligned}
\lVert T^{(l)}(z_2)-T^{(l)}(z_1)\rVert_2
&=\lVert T_2^{(l)}(x_2)-T_2^{(l)}(x_1)\rVert_2\\
&\gtrsim\frac{\sqrt{\sigma_B^{(l)}}}{K^{3/2}}\lVert x_2-x_1\rVert_2
\gtrsim\frac{\sigma_B^{(l)}}{K^{3/2}}\lVert z_2-z_1\rVert_2, \tag*{(56)}
\end{aligned} ∥ T ( l ) ( z 2 ) − T ( l ) ( z 1 ) ∥ 2 = ∥ T 2 ( l ) ( x 2 ) − T 2 ( l ) ( x 1 ) ∥ 2 ≳ K 3/2 σ B ( l ) ∥ x 2 − x 1 ∥ 2 ≳ K 3/2 σ B ( l ) ∥ z 2 − z 1 ∥ 2 , (56)
and similarly,
∥ T ( l ) ( z 2 ) − T ( l ) ( z 1 ) ∥ 2 = ∥ T 2 ( l ) ( x 2 ) − T 2 ( l ) ( x 1 ) ∥ 2 ≲ K σ B ( l ) ∥ x 2 − x 1 ∥ 2 ≲ K σ B ( l ) ∥ z 2 − z 1 ∥ 2 . (57) \begin{aligned}
\lVert T^{(l)}(z_2)-T^{(l)}(z_1)\rVert_2
&=\lVert T_2^{(l)}(x_2)-T_2^{(l)}(x_1)\rVert_2\\
&\lesssim\frac{\sqrt{K}}{\sqrt{\sigma_B^{(l)}}}\lVert x_2-x_1\rVert_2
\lesssim\frac{K}{\sqrt{\sigma_B^{(l)}}}\lVert z_2-z_1\rVert_2. \tag*{(57)}
\end{aligned} ∥ T ( l ) ( z 2 ) − T ( l ) ( z 1 ) ∥ 2 = ∥ T 2 ( l ) ( x 2 ) − T 2 ( l ) ( x 1 ) ∥ 2 ≲ σ B ( l ) K ∥ x 2 − x 1 ∥ 2 ≲ σ B ( l ) K ∥ z 2 − z 1 ∥ 2 . (57)
Therefore,
σ B ( l ) K 3 / 2 ∥ z 2 − z 1 ∥ 2 ≲ ∥ T ( l ) ( z 2 ) − T ( l ) ( z 1 ) ∥ 2 ≲ K σ B ( l ) ∥ z 2 − z 1 ∥ 2 . (58) \frac{\sigma_B^{(l)}}{K^{3/2}}\lVert z_2-z_1\rVert_2\lesssim\lVert T^{(l)}(z_2)-T^{(l)}(z_1)\rVert_2\lesssim\frac{K}{\sqrt{\sigma_B^{(l)}}}\lVert z_2-z_1\rVert_2. \tag*{(58)} K 3/2 σ B ( l ) ∥ z 2 − z 1 ∥ 2 ≲ ∥ T ( l ) ( z 2 ) − T ( l ) ( z 1 ) ∥ 2 ≲ σ B ( l ) K ∥ z 2 − z 1 ∥ 2 . (58)
Thus, each layer-wise map T ( l ) T^{(l)} T ( l ) is bi-Lipschitz on Δ K − 1 \Delta^{K-1} Δ K − 1 .
Bi-Lipschitz continuity of T T T . We now combine the layer-wise bounds. For any z 1 , z 2 ∈ Δ K − 1 z_1,z_2\in\Delta^{K-1} z 1 , z 2 ∈ Δ K − 1 , by the definition of the concatenated map,
∥ T ( z 2 ) − T ( z 1 ) ∥ 2 2 = ∑ l = 1 L ∥ T ( l ) ( z 2 ) − T ( l ) ( z 1 ) ∥ 2 2 . \lVert T(z_2)-T(z_1)\rVert_2^2=\sum_{l=1}^{L}\lVert T^{(l)}(z_2)-T^{(l)}(z_1)\rVert_2^2. ∥ T ( z 2 ) − T ( z 1 ) ∥ 2 2 = l = 1 ∑ L ∥ T ( l ) ( z 2 ) − T ( l ) ( z 1 ) ∥ 2 2 .
Applying (D.17) to each layer yields
∥ T ( z 2 ) − T ( z 1 ) ∥ 2 ≳ ( ∑ l = 1 L ( σ B ( l ) K 3 / 2 ) 2 ) 1 / 2 ∥ z 2 − z 1 ∥ 2 , \lVert T(z_2)-T(z_1)\rVert_2\gtrsim\left(\sum_{l=1}^{L}\left(\frac{\sigma_B^{(l)}}{K^{3/2}}\right)^2\right)^{1/2}\lVert z_2-z_1\rVert_2, ∥ T ( z 2 ) − T ( z 1 ) ∥ 2 ≳ l = 1 ∑ L ( K 3/2 σ B ( l ) ) 2 1/2 ∥ z 2 − z 1 ∥ 2 ,
∥ T ( z 2 ) − T ( z 1 ) ∥ 2 ≲ ( ∑ l = 1 L K 2 σ B ( l ) ) 1 / 2 ∥ z 2 − z 1 ∥ 2 . (59) \lVert T(z_2)-T(z_1)\rVert_2\lesssim\left(\sum_{l=1}^{L}\frac{K^2}{\sigma_B^{(l)}}\right)^{1/2}\lVert z_2-z_1\rVert_2. \tag*{(59)} ∥ T ( z 2 ) − T ( z 1 ) ∥ 2 ≲ ( l = 1 ∑ L σ B ( l ) K 2 ) 1/2 ∥ z 2 − z 1 ∥ 2 . (59)
Equivalently,
( ∑ l = 1 L ( σ B ( l ) ) 2 K 3 ) 1 / 2 ∥ z 2 − z 1 ∥ 2 ≲ ∥ T ( z 2 ) − T ( z 1 ) ∥ 2 ≲ K ( ∑ l = 1 L 1 σ B ( l ) ) 1 / 2 ∥ z 2 − z 1 ∥ 2 . (60) \left(\sum_{l=1}^{L}\frac{(\sigma_B^{(l)})^2}{K^3}\right)^{1/2}\lVert z_2-z_1\rVert_2\lesssim\lVert T(z_2)-T(z_1)\rVert_2\lesssim K\left(\sum_{l=1}^{L}\frac{1}{\sigma_B^{(l)}}\right)^{1/2}\lVert z_2-z_1\rVert_2. \tag*{(60)} ( l = 1 ∑ L K 3 ( σ B ( l ) ) 2 ) 1/2 ∥ z 2 − z 1 ∥ 2 ≲ ∥ T ( z 2 ) − T ( z 1 ) ∥ 2 ≲ K ( l = 1 ∑ L σ B ( l ) 1 ) 1/2 ∥ z 2 − z 1 ∥ 2 . (60)
Since σ B ( l ) > 0 \sigma_B^{(l)}>0 σ B ( l ) > 0 for all l ∈ [ L ] l\in[L] l ∈ [ L ] , this proves that T T T is bi-Lipschitz on Δ K − 1 \Delta^{K-1} Δ K − 1 .
Corner estimation
Successful corner estimation is a key task of the proposed method. Under the regularity assumptions stated in § 3 , Theorem 3.3 establishes that increasing the number of observed layers can improve the probability of accurate corner recovery. Moreover, Corollary 3.5 states that the estimated corner contains exactly one pure node from each community with high probability under the condition 2(b). In this section, we provide a deterministic error radius of the corner estimation, proof of Theorem 3.3, Corollary 3.5 and required preliminary lemmas.
Throughout this section, we use the following notion to describe whether a candidate index set contains a sufficiently pure representative from each community. Let I k ( r ) = { i ∈ [ n ] : Z i k ≥ 1 − r } \mathcal{I}_k(r) = \{i \in[n] : Z_{ik} \ge1-r\} I k ( r ) = { i ∈ [ n ] : Z ik ≥ 1 − r } for any r ∈ [ 0 , 1 ) r \in[0,1) r ∈ [ 0 , 1 ) . Given an index set J ⊆ [ n ] J \subseteq[n] J ⊆ [ n ] , we say that J J J is r r r -good if
J ∩ I k ( r ) ≠ ∅ for every 1 ≤ k ≤ K . J \cap\mathcal{I}_k(r) \ne\varnothing\quad\text{for every } 1 \le k \le K. J ∩ I k ( r ) = ∅ for every 1 ≤ k ≤ K .
Preliminary lemmas
We present the lemmas required for the corner estimation and the proof of the main theorem. First, the following lemma provides a convenient way to compute the Euclidean distance from a point to a spherical simplex defined in (6) .
Lemma E.1 (Computing the Euclidean distance to a spherical simplex). Let v 1 , … , v K ∈ S K − 1 \mathbf{v}_1,\ldots,\mathbf{v}_K \in S^{K-1} v 1 , … , v K ∈ S K − 1 , and define
C : = cone { v 1 , … , v K } , S : = C ∩ S K − 1 . C := \operatorname{cone}\{\mathbf{v}_1,\ldots,\mathbf{v}_K\}, \qquad S := C \cap S^{K-1}. C := cone { v 1 , … , v K } , S := C ∩ S K − 1 . For y ∈ S K − 1 \mathbf{y} \in S^{K-1} y ∈ S K − 1 , let p : = Proj C ( y ) \mathbf{p} := \operatorname{Proj}_C(\mathbf{y}) p := Proj C ( y ) . If p ≠ 0 \mathbf{p} \ne0 p = 0 , then Proj S ( y ) = p ∥ p ∥ 2 \operatorname{Proj}_S(\mathbf{y}) = \frac{\mathbf{p}}{\lVert\mathbf{p}\rVert_2} Proj S ( y ) = ∥ p ∥ 2 p . If p = 0 \mathbf{p} = 0 p = 0 , then at least one generating vertex is a Euclidean projection of y \mathbf{y} y onto S S S . In particular, any
j ∗ ∈ arg min 1 ≤ j ≤ K ∥ y − v j ∥ 2 j^* \in\underset{1 \le j \le K}{\operatorname{arg\,min}}\ \lVert\mathbf{y}-\mathbf{v}_j\rVert_2 j ∗ ∈ 1 ≤ j ≤ K arg min ∥ y − v j ∥ 2 satisfies
v j ∗ ∈ arg min u ∈ S ∥ y − u ∥ 2 . \mathbf{v}_{j^*} \in\underset{\mathbf{u}\in S}{\operatorname{arg\,min}}\ \lVert\mathbf{y}-\mathbf{u}\rVert_2. v j ∗ ∈ u ∈ S arg min ∥ y − u ∥ 2 . Consequently,
d ( y , S ) = { 2 − 2 ∥ p ∥ 2 , p ≠ 0 , min 1 ≤ j ≤ K ∥ y − v j ∥ 2 , p = 0. d(\mathbf{y},S)=
\begin{cases}
\sqrt{2-2\lVert\mathbf{p}\rVert_2}, & \mathbf{p}\ne0,\\
\underset{1\le j\le K}{\min}\ \lVert\mathbf{y}-\mathbf{v}_j\rVert_2, & \mathbf{p}=0.
\end{cases} d ( y , S ) = { 2 − 2 ∥ p ∥ 2 , 1 ≤ j ≤ K min ∥ y − v j ∥ 2 , p = 0 , p = 0.
Lemma E.1 shows that the distance between a point on the unit sphere and a spherical simplex can be computed by first projecting the point onto the generating cone. If the cone projection is nonzero, we normalize it to unit length; otherwise, it suffices to compare the distances to the K K K generating vertices.
Proof. Consider the first case p ≠ 0 \mathbf{p} \ne0 p = 0 . For any u ∈ S \mathbf{u} \in S u ∈ S , the squared Euclidean distance between y \mathbf{y} y and u \mathbf{u} u can be written as
∥ y − u ∥ 2 2 = ∥ y ∥ 2 2 + ∥ u ∥ 2 2 − 2 ⟨ y , u ⟩ = 2 − 2 ⟨ y , u ⟩ . \lVert\mathbf{y}-\mathbf{u}\rVert_2^2 = \lVert\mathbf{y}\rVert_2^2 + \lVert\mathbf{u}\rVert_2^2 - 2\langle\mathbf{y},\mathbf{u}\rangle= 2 - 2\langle\mathbf{y},\mathbf{u}\rangle. ∥ y − u ∥ 2 2 = ∥ y ∥ 2 2 + ∥ u ∥ 2 2 − 2 ⟨ y , u ⟩ = 2 − 2 ⟨ y , u ⟩ .
Hence we aim to find the minimizer u ∈ S \mathbf{u} \in S u ∈ S such that u ∈ arg max x ∈ S ⟨ y , x ⟩ \mathbf{u} \in\operatorname{arg\,max}_{\mathbf{x}\in S}\langle\mathbf{y},\mathbf{x}\rangle u ∈ arg max x ∈ S ⟨ y , x ⟩ . By the first-order optimality of projection, for any x ∈ C \mathbf{x} \in C x ∈ C , we have
⟨ y − p , x − p ⟩ ≤ 0. (61) \langle\mathbf{y}-\mathbf{p},\mathbf{x}-\mathbf{p}\rangle\le0. \tag*{(61)} ⟨ y − p , x − p ⟩ ≤ 0. (61)
By definition of a cone, we know 0 , 2 p ∈ C 0,2\mathbf{p} \in C 0 , 2 p ∈ C . By substituting x = 0 , 2 p \mathbf{x}=0,2\mathbf{p} x = 0 , 2 p in (E.1), we have ⟨ y − p , p ⟩ = 0 \langle\mathbf{y}-\mathbf{p},\mathbf{p}\rangle= 0 ⟨ y − p , p ⟩ = 0 and ⟨ y − p , x ⟩ ≤ 0 \langle\mathbf{y}-\mathbf{p},\mathbf{x}\rangle\le0 ⟨ y − p , x ⟩ ≤ 0 . Then for any t ∈ S ⊆ C \mathbf{t} \in S \subseteq C t ∈ S ⊆ C , we have the following upper bound.
⟨ y , t ⟩ = ⟨ y − p , t ⟩ + ⟨ p , t ⟩ ≤ ⟨ p , t ⟩ ≤ ∥ p ∥ 2 ∥ t ∥ 2 = ∥ p ∥ 2 . \langle\mathbf{y},\mathbf{t}\rangle= \langle\mathbf{y}-\mathbf{p},\mathbf{t}\rangle+ \langle\mathbf{p},\mathbf{t}\rangle\le\langle\mathbf{p},\mathbf{t}\rangle\le\lVert\mathbf{p}\rVert_2\lVert\mathbf{t}\rVert_2 = \lVert\mathbf{p}\rVert_2. ⟨ y , t ⟩ = ⟨ y − p , t ⟩ + ⟨ p , t ⟩ ≤ ⟨ p , t ⟩ ≤ ∥ p ∥ 2 ∥ t ∥ 2 = ∥ p ∥ 2 .
Let u = p / ∥ p ∥ 2 ∈ S ⊆ C \mathbf{u}=\mathbf{p}/\|\mathbf{p}\|_{2}\in S\subseteq\mathcal{C} u = p /∥ p ∥ 2 ∈ S ⊆ C . Then,
⟨ y , u ⟩ = ⟨ y , p ∥ p ∥ 2 ⟩ = 1 ∥ p ∥ 2 ⟨ y , p ⟩ = 1 ∥ p ∥ 2 ( ⟨ y − p , p ⟩ + ⟨ p , p ⟩ ) = ∥ p ∥ 2 . \langle\mathbf{y},\mathbf{u}\rangle=\left\langle\mathbf{y},\frac{\mathbf{p}}{\|\mathbf{p}\|_{2}}\right\rangle=\frac{1}{\|\mathbf{p}\|_{2}}\langle\mathbf{y},\mathbf{p}\rangle=\frac{1}{\|\mathbf{p}\|_{2}}\left(\langle\mathbf{y}-\mathbf{p},\mathbf{p}\rangle+\langle\mathbf{p},\mathbf{p}\rangle\right)=\|\mathbf{p}\|_{2}. ⟨ y , u ⟩ = ⟨ y , ∥ p ∥ 2 p ⟩ = ∥ p ∥ 2 1 ⟨ y , p ⟩ = ∥ p ∥ 2 1 ( ⟨ y − p , p ⟩ + ⟨ p , p ⟩ ) = ∥ p ∥ 2 .
That is, u = p / ∥ p ∥ 2 ∈ arg max x ∈ S ⟨ y , x ⟩ \mathbf{u}=\mathbf{p}/\|\mathbf{p}\|_{2}\in\operatorname{arg\,max}_{\mathbf{x}\in S}\langle\mathbf{y},\mathbf{x}\rangle u = p /∥ p ∥ 2 ∈ arg max x ∈ S ⟨ y , x ⟩ and
d ( y , S ) = ∥ y − p ∥ p ∥ 2 ∥ 2 = 2 − 2 ⟨ y , p ∥ p ∥ 2 ⟩ = 2 − 2 ∥ p ∥ 2 ( ⟨ y − p + p , p ⟩ ) = 2 − 2 ∥ p ∥ 2 . d(\mathbf{y},S)=\left\|\mathbf{y}-\frac{\mathbf{p}}{\|\mathbf{p}\|_{2}}\right\|_{2}=\sqrt{2-2\left\langle\mathbf{y},\frac{\mathbf{p}}{\|\mathbf{p}\|_{2}}\right\rangle}=\sqrt{2-\frac{2}{\|\mathbf{p}\|_{2}}\left(\langle\mathbf{y}-\mathbf{p}+\mathbf{p},\mathbf{p}\rangle\right)}=\sqrt{2-2\|\mathbf{p}\|_{2}}. d ( y , S ) = y − ∥ p ∥ 2 p 2 = 2 − 2 ⟨ y , ∥ p ∥ 2 p ⟩ = 2 − ∥ p ∥ 2 2 ( ⟨ y − p + p , p ⟩ ) = 2 − 2∥ p ∥ 2 .
Next, consider the second case p = 0 \mathbf{p}=\mathbf{0} p = 0 . Similarly, we aim to find u ∈ S \mathbf{u}\in S u ∈ S such that u ∈ arg max x ∈ S ⟨ y , x ⟩ \mathbf{u}\in\operatorname{arg\,max}_{\mathbf{x}\in S}\langle\mathbf{y},\mathbf{x}\rangle u ∈ arg max x ∈ S ⟨ y , x ⟩ . We show ⟨ y , v j ⟩ ≤ 0 \langle\mathbf{y},\mathbf{v}_{j}\rangle\leq0 ⟨ y , v j ⟩ ≤ 0 for all j ∈ [ K ] j\in[K] j ∈ [ K ] . Fix j ∈ [ K ] j\in[K] j ∈ [ K ] . For any t > 0 t>0 t > 0 , it is known that t v j ∈ C t\mathbf{v}_{j}\in\mathcal{C} t v j ∈ C . Since p = 0 ∈ C \mathbf{p}=\mathbf{0}\in\mathcal{C} p = 0 ∈ C is the projected point of y \mathbf{y} y onto C \mathcal{C} C , we have
∥ y ∥ 2 2 ≤ ∥ y − t v j ∥ 2 2 = ∥ y ∥ 2 2 + t 2 ∥ v j ∥ 2 2 − 2 t ⟨ y , v j ⟩ ⟺ ⟨ y , v j ⟩ ≤ t 2 . \|\mathbf{y}\|_{2}^{2}\leq\|\mathbf{y}-t\mathbf{v}_{j}\|_{2}^{2}=\|\mathbf{y}\|_{2}^{2}+t^{2}\|\mathbf{v}_{j}\|_{2}^{2}-2t\langle\mathbf{y},\mathbf{v}_{j}\rangle\Longleftrightarrow\langle\mathbf{y},\mathbf{v}_{j}\rangle\leq\frac{t}{2}. ∥ y ∥ 2 2 ≤ ∥ y − t v j ∥ 2 2 = ∥ y ∥ 2 2 + t 2 ∥ v j ∥ 2 2 − 2 t ⟨ y , v j ⟩ ⟺ ⟨ y , v j ⟩ ≤ 2 t .
By taking t → 0 t\to0 t → 0 , we show ⟨ y , v j ⟩ ≤ 0 \langle\mathbf{y},\mathbf{v}_{j}\rangle\leq0 ⟨ y , v j ⟩ ≤ 0 for all j ∈ [ K ] j\in[K] j ∈ [ K ] . Let t ∈ S \mathbf{t}\in S t ∈ S be given. Then there exists a constant c = ( c 1 , ⋯ , c K ) ⊤ ∈ R K \mathbf{c}=(c_{1},\cdots,c_{K})^{\top}\in\mathbb{R}^{K} c = ( c 1 , ⋯ , c K ) ⊤ ∈ R K with ∑ i = 1 K c i = 1 \sum_{i=1}^{K}c_{i}=1 ∑ i = 1 K c i = 1 and c i ≥ 0 c_{i}\geq0 c i ≥ 0 for all i ∈ [ K ] i\in[K] i ∈ [ K ] such that t = ∑ i = 1 K c i v i ∥ ∑ i = 1 K c i v i ∥ 2 \mathbf{t}=\frac{\sum_{i=1}^{K}c_{i}\mathbf{v}_{i}}{\left\|\sum_{i=1}^{K}c_{i}\mathbf{v}_{i}\right\|_{2}} t = ∥ ∑ i = 1 K c i v i ∥ 2 ∑ i = 1 K c i v i . Let m : = max 1 ≤ i ≤ K ⟨ y , v i ⟩ m:=\max_{1\leq i\leq K}\langle\mathbf{y},\mathbf{v}_{i}\rangle m := max 1 ≤ i ≤ K ⟨ y , v i ⟩ . Then,
⟨ y , t ⟩ = ⟨ y , ∑ i = 1 K c i v i ∥ ∑ i = 1 K c i v i ∥ 2 ⟩ = 1 ∥ ∑ i = 1 K c i v i ∥ 2 ∑ i = 1 K c i ⟨ y , v i ⟩ ≤ m ∥ ∑ i = 1 K c i v i ∥ 2 . \langle\mathbf{y},\mathbf{t}\rangle=\left\langle\mathbf{y},\frac{\sum_{i=1}^{K}c_{i}\mathbf{v}_{i}}{\left\|\sum_{i=1}^{K}c_{i}\mathbf{v}_{i}\right\|_{2}}\right\rangle=\frac{1}{\left\|\sum_{i=1}^{K}c_{i}\mathbf{v}_{i}\right\|_{2}}\sum_{i=1}^{K}c_{i}\langle\mathbf{y},\mathbf{v}_{i}\rangle\leq\frac{m}{\left\|\sum_{i=1}^{K}c_{i}\mathbf{v}_{i}\right\|_{2}}. ⟨ y , t ⟩ = ⟨ y , ∑ i = 1 K c i v i 2 ∑ i = 1 K c i v i ⟩ = ∑ i = 1 K c i v i 2 1 i = 1 ∑ K c i ⟨ y , v i ⟩ ≤ ∑ i = 1 K c i v i 2 m .
By triangle inequality, ∥ ∑ i = 1 K c i v i ∥ 2 ≤ ∑ i = 1 K c i ∥ v i ∥ 2 = 1 \left\|\sum_{i=1}^{K}c_{i}\mathbf{v}_{i}\right\|_{2}\leq\sum_{i=1}^{K}c_{i}\|\mathbf{v}_{i}\|_{2}=1 ∑ i = 1 K c i v i 2 ≤ ∑ i = 1 K c i ∥ v i ∥ 2 = 1 . Hence we obtain the following upper bound of ⟨ y , t ⟩ \langle\mathbf{y},\mathbf{t}\rangle ⟨ y , t ⟩ : since m ≤ 0 m\leq0 m ≤ 0 ,
⟨ y , t ⟩ ≤ m ∥ ∑ i = 1 K c i v i ∥ 2 ≤ m . \langle\mathbf{y},\mathbf{t}\rangle\leq\frac{m}{\left\|\sum_{i=1}^{K}c_{i}\mathbf{v}_{i}\right\|_{2}}\leq m. ⟨ y , t ⟩ ≤ ∑ i = 1 K c i v i 2 m ≤ m .
Let i ∗ ∈ arg max i ∈ [ K ] ⟨ y , v i ⟩ i^{*}\in\operatorname{arg\,max}_{i\in[K]}\langle\mathbf{y},\mathbf{v}_{i}\rangle i ∗ ∈ arg max i ∈ [ K ] ⟨ y , v i ⟩ . Then u = v i ∗ ∈ arg max x ∈ S ⟨ y , x ⟩ \mathbf{u}=\mathbf{v}_{i^{*}}\in\operatorname{arg\,max}_{\mathbf{x}\in S}\langle\mathbf{y},\mathbf{x}\rangle u = v i ∗ ∈ arg max x ∈ S ⟨ y , x ⟩ and
d ( y , S ) = ∥ y − v i ∗ ∥ 2 = min 1 ≤ j ≤ K ∥ y − v j ∥ 2 . d(\mathbf{y},S)=\|\mathbf{y}-\mathbf{v}_{i^{*}}\|_{2}=\min_{1\leq j\leq K}\|\mathbf{y}-\mathbf{v}_{j}\|_{2}. d ( y , S ) = ∥ y − v i ∗ ∥ 2 = 1 ≤ j ≤ K min ∥ y − v j ∥ 2 .
□
Lemma E.2 (Pullback representation of a spherical simplex). Fix 1 ≤ l ≤ L 1\leq l\leq L 1 ≤ l ≤ L , and let J = { j 1 , … , j K } ⊂ [ n ] \mathcal{J}=\{j_{1},\ldots,j_{K}\}\subset[n] J = { j 1 , … , j K } ⊂ [ n ] . For each j ∈ J j\in\mathcal{J} j ∈ J , define
z j : = Z j , ⋅ ∈ Δ K − 1 , y j ( l ) : = Y j , ⋅ ( l ) = T ( l ) ( z j ) = R ( l ) ⊤ z j ∥ R ( l ) ⊤ z j ∥ 2 . \mathbf{z}_{j}:=Z_{j,\cdot}\in\Delta^{K-1},\qquad\mathbf{y}_{j}^{(l)}:=Y_{j,\cdot}^{(l)}=T^{(l)}(\mathbf{z}_{j})=\frac{R^{(l)\top}\mathbf{z}_{j}}{\|R^{(l)\top}\mathbf{z}_{j}\|_{2}}. z j := Z j , ⋅ ∈ Δ K − 1 , y j ( l ) := Y j , ⋅ ( l ) = T ( l ) ( z j ) = ∥ R ( l ) ⊤ z j ∥ 2 R ( l ) ⊤ z j . Let
C J : = conv { z j : j ∈ J } ⊂ Δ K − 1 . C_{\mathcal{J}}:=\operatorname{conv}\{\mathbf{z}_{j}:j\in\mathcal{J}\}\subset\Delta^{K-1}. C J := conv { z j : j ∈ J } ⊂ Δ K − 1 . Then
S S ( Y ( l ) , J ) = T ( l ) ( C J ) . SS(\mathbf{Y}^{(l)},\mathcal{J})=T^{(l)}(C_{\mathcal{J}}). S S ( Y ( l ) , J ) = T ( l ) ( C J ) . This lemma shows that the spherical simplex can be equivalently represented via its pullback to the convex set C J ⊂ Δ K − 1 C_{\mathcal{J}}\subset\Delta^{K-1} C J ⊂ Δ K − 1 , allowing us to analyze its geometry in the simplex domain.
Proof. We first show that T ( l ) ( C J ) ⊂ S S ( Y ( l ) , J ) T^{(l)}(C_{\mathcal{J}})\subset SS(\mathbf{Y}^{(l)},\mathcal{J}) T ( l ) ( C J ) ⊂ S S ( Y ( l ) , J ) . Let y ( l ) ∈ T ( l ) ( C J ) \mathbf{y}^{(l)}\in T^{(l)}(C_{\mathcal{J}}) y ( l ) ∈ T ( l ) ( C J ) . Then there exists
z = ∑ j ∈ J u j z j ∈ C J , u j ≥ 0 , ∑ j ∈ J u j = 1 , \mathbf{z}=\sum_{j\in\mathcal{J}}u_{j}\mathbf{z}_{j}\in C_{\mathcal{J}},\qquad u_{j}\geq0,\qquad\sum_{j\in\mathcal{J}}u_{j}=1, z = j ∈ J ∑ u j z j ∈ C J , u j ≥ 0 , j ∈ J ∑ u j = 1 ,
such that
y ( l ) = T ( l ) ( z ) = R ( l ) T z ∥ R ( l ) T z ∥ 2 . \mathbf{y}^{(l)}=T^{(l)}(\mathbf{z})=\frac{\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}}{\left\|\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}\right\|_2}. y ( l ) = T ( l ) ( z ) = R ( l ) T z 2 R ( l ) T z .
Since
R ( l ) T z = ∑ j ∈ J u j R ( l ) T z j = ∑ j ∈ J u j ∥ R ( l ) T z j ∥ 2 y j ( l ) , \mathbf{R}^{(l)\mathsf{T}}\mathbf{z}=\sum_{j\in\mathcal{J}}u_j\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}_j=\sum_{j\in\mathcal{J}}u_j\left\|\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}_j\right\|_2\mathbf{y}_j^{(l)}, R ( l ) T z = j ∈ J ∑ u j R ( l ) T z j = j ∈ J ∑ u j R ( l ) T z j 2 y j ( l ) ,
define
d j : = u j ∥ R ( l ) T z j ∥ 2 ∑ m ∈ J u m ∥ R ( l ) T z m ∥ 2 , j ∈ J . d_j:=\frac{u_j\left\|\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}_j\right\|_2}{\sum_{m\in\mathcal{J}}u_m\left\|\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}_m\right\|_2},\qquad j\in\mathcal{J}. d j := ∑ m ∈ J u m R ( l ) T z m 2 u j R ( l ) T z j 2 , j ∈ J .
Then d j ≥ 0 d_j\ge0 d j ≥ 0 and ∑ j ∈ J d j = 1 \sum_{j\in\mathcal{J}}d_j=1 ∑ j ∈ J d j = 1 , and hence
y ( l ) = ∑ j ∈ J d j y j ( l ) ∥ ∑ j ∈ J d j y j ( l ) ∥ 2 ∈ S S ( Y ( l ) , J ) . \mathbf{y}^{(l)}=\frac{\sum_{j\in\mathcal{J}}d_j\mathbf{y}_j^{(l)}}{\left\|\sum_{j\in\mathcal{J}}d_j\mathbf{y}_j^{(l)}\right\|_2}\in SS(\mathbf{Y}^{(l)},\mathcal{J}). y ( l ) = ∑ j ∈ J d j y j ( l ) 2 ∑ j ∈ J d j y j ( l ) ∈ S S ( Y ( l ) , J ) .
This proves T ( l ) ( C J ) ⊂ S S ( Y ( l ) , J ) T^{(l)}(C_{\mathcal{J}})\subset SS(\mathbf{Y}^{(l)},\mathcal{J}) T ( l ) ( C J ) ⊂ S S ( Y ( l ) , J ) .
Next, we show that S S ( Y ( l ) , J ) ⊂ T ( l ) ( C J ) SS(\mathbf{Y}^{(l)},\mathcal{J})\subset T^{(l)}(C_{\mathcal{J}}) S S ( Y ( l ) , J ) ⊂ T ( l ) ( C J ) . Let y ( l ) ∈ S S ( Y ( l ) , J ) \mathbf{y}^{(l)}\in SS(\mathbf{Y}^{(l)},\mathcal{J}) y ( l ) ∈ S S ( Y ( l ) , J ) . Then there exists
u = ( u j ) j ∈ J ∈ Δ K − 1 \mathbf{u}=(u_j)_{j\in\mathcal{J}}\in\Delta^{K-1} u = ( u j ) j ∈ J ∈ Δ K − 1
such that
y ( l ) = ∑ j ∈ J u j y j ( l ) ∥ ∑ j ∈ J u j y j ( l ) ∥ 2 . \mathbf{y}^{(l)}=\frac{\sum_{j\in\mathcal{J}}u_j\mathbf{y}_j^{(l)}}{\left\|\sum_{j\in\mathcal{J}}u_j\mathbf{y}_j^{(l)}\right\|_2}. y ( l ) = ∑ j ∈ J u j y j ( l ) 2 ∑ j ∈ J u j y j ( l ) .
Define
γ : = ∑ j ∈ J u j ∥ R ( l ) T z j ∥ 2 , d j : = u j / ∥ R ( l ) T z j ∥ 2 γ , j ∈ J , \gamma:=\sum_{j\in\mathcal{J}}\frac{u_j}{\left\|\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}_j\right\|_2},\qquad d_j:=\frac{u_j/\left\|\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}_j\right\|_2}{\gamma},\qquad j\in\mathcal{J}, γ := j ∈ J ∑ R ( l ) T z j 2 u j , d j := γ u j / R ( l ) T z j 2 , j ∈ J ,
and let
z : = ∑ j ∈ J d j z j ∈ C J . \mathbf{z}:=\sum_{j\in\mathcal{J}}d_j\mathbf{z}_j\in C_{\mathcal{J}}. z := j ∈ J ∑ d j z j ∈ C J .
Then
R ( l ) T z = ∑ j ∈ J d j R ( l ) T z j = ∑ j ∈ J d j ∥ R ( l ) T z j ∥ 2 y j ( l ) = 1 γ ∑ j ∈ J u j y j ( l ) . \mathbf{R}^{(l)\mathsf{T}}\mathbf{z}=\sum_{j\in\mathcal{J}}d_j\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}_j=\sum_{j\in\mathcal{J}}d_j\left\|\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}_j\right\|_2\mathbf{y}_j^{(l)}=\frac{1}{\gamma}\sum_{j\in\mathcal{J}}u_j\mathbf{y}_j^{(l)}. R ( l ) T z = j ∈ J ∑ d j R ( l ) T z j = j ∈ J ∑ d j R ( l ) T z j 2 y j ( l ) = γ 1 j ∈ J ∑ u j y j ( l ) .
Therefore,
T ( l ) ( z ) = R ( l ) T z ∥ R ( l ) T z ∥ 2 = ∑ j ∈ J u j y j ( l ) ∥ ∑ j ∈ J u j y j ( l ) ∥ 2 = y ( l ) . T^{(l)}(\mathbf{z})=\frac{\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}}{\left\|\mathbf{R}^{(l)\mathsf{T}}\mathbf{z}\right\|_2}=\frac{\sum_{j\in\mathcal{J}}u_j\mathbf{y}_j^{(l)}}{\left\|\sum_{j\in\mathcal{J}}u_j\mathbf{y}_j^{(l)}\right\|_2}=\mathbf{y}^{(l)}. T ( l ) ( z ) = R ( l ) T z 2 R ( l ) T z = ∑ j ∈ J u j y j ( l ) 2 ∑ j ∈ J u j y j ( l ) = y ( l ) .
Hence y ( l ) ∈ T ( l ) ( C J ) \mathbf{y}^{(l)}\in T^{(l)}(C_{\mathcal{J}}) y ( l ) ∈ T ( l ) ( C J ) , which proves
S S ( Y ( l ) , J ) ⊂ T ( l ) ( C J ) . SS(\mathbf{Y}^{(l)},\mathcal{J})\subset T^{(l)}(C_{\mathcal{J}}). S S ( Y ( l ) , J ) ⊂ T ( l ) ( C J ) .
Combining the two inclusions completes the proof.
Lemma E.3 (Open hemisphere containment of embeddings). For each 1 ≤ l ≤ L 1\le l\le L 1 ≤ l ≤ L , there exists a unit vector u ( l ) ∈ S K − 1 \mathbf{u}^{(l)}\in S^{K-1} u ( l ) ∈ S K − 1 such that
⟨ y i ( l ) , u ( l ) ⟩ ≥ c s t a b σ B ( l ) K , ∀ i ∈ [ n ] , \left\langle\mathbf{y}_i^{(l)},\mathbf{u}^{(l)}\right\rangle\ge\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{K},\qquad\forall i\in[n], ⟨ y i ( l ) , u ( l ) ⟩ ≥ K c stab σ B ( l ) , ∀ i ∈ [ n ] , where u ( l ) \mathbf{u}^{(l)} u ( l ) is given by
u ( l ) = n ( l ) ∥ n ( l ) ∥ 2 , n ( l ) = ( R ( l ) ) − 1 1 K . \mathbf{u}^{(l)}=\frac{\mathbf{n}^{(l)}}{\left\|\mathbf{n}^{(l)}\right\|_2},\qquad\mathbf{n}^{(l)}=\left(\mathbf{R}^{(l)}\right)^{-1}\mathbf{1}_K. u ( l ) = n ( l ) 2 n ( l ) , n ( l ) = ( R ( l ) ) − 1 1 K .
Equivalently, all embeddings { y ~ i ( l ) } i = 1 n \{\widetilde{y}^{(l)}_i\}_{i=1}^n { y i ( l ) } i = 1 n lie in an open hemisphere and are separated from its boundary by at least c s t a b σ B ( l ) K \frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{K} K c stab σ B ( l ) . Moreover, if h ^ ( l ) ≤ c s t a b σ B ( l ) 2 K \hat{h}^{(l)} \le\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K} h ^ ( l ) ≤ 2 K c stab σ B ( l ) , then
⟨ y ~ i ( l ) , u ( l ) ⟩ ≥ c s t a b σ B ( l ) 2 K , ∀ i ∈ [ n ] , \langle\widetilde{y}^{(l)}_i,\mathbf{u}^{(l)}\rangle\ge\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K}, \qquad\forall i \in[n], ⟨ y i ( l ) , u ( l ) ⟩ ≥ 2 K c stab σ B ( l ) , ∀ i ∈ [ n ] ,
where y ~ i ( l ) : = Y ~ i , : ( l ) \widetilde{y}^{(l)}_i := \widetilde{Y}^{(l)}_{i,:} y i ( l ) := Y i , : ( l ) . In particular, the empirical embeddings also lie in the same open hemisphere with margin at least c s t a b σ B ( l ) 2 K \frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K} 2 K c stab σ B ( l ) .
This lemma gives the geometric interpretation of the event G l G_l G l used in the main text. On G l G_l G l , the empirical row-normalized embeddings { y ~ i ( l ) } i = 1 n \{\widetilde{y}^{(l)}_i\}_{i=1}^n { y i ( l ) } i = 1 n preserve the population hemisphere margin: they remain in the same open hemisphere, with margin at least c s t a b σ B ( l ) 2 K \frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K} 2 K c stab σ B ( l ) .
Proof. For any i ∈ [ n ] i \in[n] i ∈ [ n ] , since T 1 ( l ) ( z i ) = ( R ( l ) ) ⊤ z i ∈ C R ( l ) T_1^{(l)}(z_i) = (R^{(l)})^\top z_i \in C_R^{(l)} T 1 ( l ) ( z i ) = ( R ( l ) ) ⊤ z i ∈ C R ( l ) and
⟨ n ( l ) , T 1 ( l ) ( z i ) ⟩ = ⟨ ( R ( l ) ) − 1 1 K , ( R ( l ) ) ⊤ z i ⟩ = 1 K ⊤ z i = 1 , \left\langle n^{(l)},T_1^{(l)}(z_i)\right\rangle
=
\left\langle(R^{(l)})^{-1}\mathbf{1}_K,(R^{(l)})^\top z_i\right\rangle
=
\mathbf{1}_K^\top z_i
=
1, ⟨ n ( l ) , T 1 ( l ) ( z i ) ⟩ = ⟨ ( R ( l ) ) − 1 1 K , ( R ( l ) ) ⊤ z i ⟩ = 1 K ⊤ z i = 1 ,
we have
⟨ n ( l ) ∥ n ( l ) ∥ 2 , y i ( l ) ⟩ = 1 ∥ n ( l ) ∥ 2 ∥ T 1 ( l ) ( z i ) ∥ 2 ≥ 1 ∥ n ( l ) ∥ 2 sup x ∈ C R ( l ) ∥ x ∥ 2 = 1 ∥ n ( l ) ∥ 2 max 1 ≤ k ≤ K ∥ R k ( l ) ∥ 2 ≥ 1 ∥ n ( l ) ∥ 2 σ max ( R ( l ) ) ≥ c s t a b σ B ( l ) K . \left\langle\frac{n^{(l)}}{\|n^{(l)}\|_2},y_i^{(l)}\right\rangle
=
\frac{1}{\|n^{(l)}\|_2\|T_1^{(l)}(z_i)\|_2}
\ge
\frac{1}{\|n^{(l)}\|_2\sup_{x\in C_R^{(l)}}\|x\|_2}
=
\frac{1}{\|n^{(l)}\|_2\max_{1\le k\le K}\|R_k^{(l)}\|_2}
\ge
\frac{1}{\|n^{(l)}\|_2\sigma_{\max}(R^{(l)})}
\ge
\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{K}. ⟨ ∥ n ( l ) ∥ 2 n ( l ) , y i ( l ) ⟩ = ∥ n ( l ) ∥ 2 ∥ T 1 ( l ) ( z i ) ∥ 2 1 ≥ ∥ n ( l ) ∥ 2 sup x ∈ C R ( l ) ∥ x ∥ 2 1 = ∥ n ( l ) ∥ 2 max 1 ≤ k ≤ K ∥ R k ( l ) ∥ 2 1 ≥ ∥ n ( l ) ∥ 2 σ m a x ( R ( l ) ) 1 ≥ K c stab σ B ( l ) .
Next, assume h ^ ( l ) ≤ c s t a b σ B ( l ) 2 K \hat{h}^{(l)} \le\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K} h ^ ( l ) ≤ 2 K c stab σ B ( l ) . Then, we have
⟨ n ( l ) ∥ n ( l ) ∥ 2 , y ~ i ( l ) ⟩ = ⟨ n ( l ) ∥ n ( l ) ∥ 2 , y i ( l ) − y ~ i ( l ) + y ~ i ( l ) ⟩ = ⟨ n ( l ) ∥ n ( l ) ∥ 2 , y i ( l ) ⟩ + ⟨ n ( l ) ∥ n ( l ) ∥ 2 , y ~ i ( l ) − y i ( l ) ⟩ ≥ c s t a b σ B ( l ) K − ∥ n ( l ) ∥ n ( l ) ∥ 2 ∥ 2 ∥ y ~ i ( l ) − y i ( l ) ∥ 2 ≥ c s t a b σ B ( l ) K − h ^ ( l ) ≥ c s t a b σ B ( l ) 2 K . \begin{aligned}
\left\langle\frac{n^{(l)}}{\|n^{(l)}\|_2},\widetilde{y}^{(l)}_i\right\rangle
=
\left\langle\frac{n^{(l)}}{\|n^{(l)}\|_2},y_i^{(l)}-\widetilde{y}^{(l)}_i+\widetilde{y}^{(l)}_i\right\rangle\\
&=
\left\langle\frac{n^{(l)}}{\|n^{(l)}\|_2},y_i^{(l)}\right\rangle
+
\left\langle\frac{n^{(l)}}{\|n^{(l)}\|_2},\widetilde{y}^{(l)}_i-y_i^{(l)}\right\rangle
\ge
\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{K}
-
\left\|\frac{n^{(l)}}{\|n^{(l)}\|_2}\right\|_2\left\|\widetilde{y}^{(l)}_i-y_i^{(l)}\right\|_2 \\
&\ge
\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{K}
-
\hat{h}^{(l)}
\ge
\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K}.
\end{aligned} ⟨ ∥ n ( l ) ∥ 2 n ( l ) , y i ( l ) ⟩ = ⟨ ∥ n ( l ) ∥ 2 n ( l ) , y i ( l ) − y i ( l ) + y i ( l ) ⟩ = ⟨ ∥ n ( l ) ∥ 2 n ( l ) , y i ( l ) ⟩ + ⟨ ∥ n ( l ) ∥ 2 n ( l ) , y i ( l ) − y i ( l ) ⟩ ≥ K c stab σ B ( l ) − ∥ n ( l ) ∥ 2 n ( l ) 2 y i ( l ) − y i ( l ) 2 ≥ K c stab σ B ( l ) − h ^ ( l ) ≥ 2 K c stab σ B ( l ) .
□ \square □
Lemma E.4 (Stability of point-to-set distance). Let u 1 , u 2 ∈ R K \mathbf{u}_1,\mathbf{u}_2 \in\mathbb{R}^K u 1 , u 2 ∈ R K , and let A , B ⊂ R K A,B \subset\mathbb{R}^K A , B ⊂ R K be nonempty sets. Then
∣ d ( u 1 , A ) − d ( u 2 , B ) ∣ ≤ ∥ u 1 − u 2 ∥ 2 + d H ( A , B ) , \left|d(\mathbf{u}_1,A)-d(\mathbf{u}_2,B)\right| \le\|\mathbf{u}_1-\mathbf{u}_2\|_2+d_H(A,B), ∣ d ( u 1 , A ) − d ( u 2 , B ) ∣ ≤ ∥ u 1 − u 2 ∥ 2 + d H ( A , B ) ,
where
d ( u , A ) : = inf a ∈ A ∥ u − a ∥ 2 d(\mathbf{u}, A) := \inf_{\mathbf{a}\in A} \|\mathbf{u}-\mathbf{a}\|_2 d ( u , A ) := a ∈ A inf ∥ u − a ∥ 2
denotes the point-to-set distance, and d H ( A , B ) d_H(A,B) d H ( A , B ) denotes the Hausdorff distance between A A A and B B B .
Proof. By the triangle inequality,
∣ d ( u 1 , A ) − d ( u 2 , B ) ∣ ≤ ∣ d ( u 1 , A ) − d ( u 1 , B ) ∣ + ∣ d ( u 1 , B ) − d ( u 2 , B ) ∣ . |d(\mathbf{u}_1,A)-d(\mathbf{u}_2,B)| \leq|d(\mathbf{u}_1,A)-d(\mathbf{u}_1,B)|+|d(\mathbf{u}_1,B)-d(\mathbf{u}_2,B)|. ∣ d ( u 1 , A ) − d ( u 2 , B ) ∣ ≤ ∣ d ( u 1 , A ) − d ( u 1 , B ) ∣ + ∣ d ( u 1 , B ) − d ( u 2 , B ) ∣.
It therefore suffices to show that
∣ d ( u 1 , A ) − d ( u 1 , B ) ∣ ≤ d H ( A , B ) and ∣ d ( u 1 , B ) − d ( u 2 , B ) ∣ ≤ ∥ u 1 − u 2 ∥ 2 . |d(\mathbf{u}_1,A)-d(\mathbf{u}_1,B)| \leq d_H(A,B) \quad\text{and}\quad|d(\mathbf{u}_1,B)-d(\mathbf{u}_2,B)| \leq\|\mathbf{u}_1-\mathbf{u}_2\|_2. ∣ d ( u 1 , A ) − d ( u 1 , B ) ∣ ≤ d H ( A , B ) and ∣ d ( u 1 , B ) − d ( u 2 , B ) ∣ ≤ ∥ u 1 − u 2 ∥ 2 .
We first prove that
∣ d ( u 1 , A ) − d ( u 1 , B ) ∣ ≤ d H ( A , B ) . |d(\mathbf{u}_1,A)-d(\mathbf{u}_1,B)| \leq d_H(A,B). ∣ d ( u 1 , A ) − d ( u 1 , B ) ∣ ≤ d H ( A , B ) .
Let ϵ > 0 \epsilon> 0 ϵ > 0 . Choose b ϵ ∈ B \mathbf{b}_{\epsilon} \in B b ϵ ∈ B such that
∥ u 1 − b ϵ ∥ 2 ≤ d ( u 1 , B ) + ϵ . \|\mathbf{u}_1-\mathbf{b}_{\epsilon}\|_2 \leq d(\mathbf{u}_1,B)+\epsilon. ∥ u 1 − b ϵ ∥ 2 ≤ d ( u 1 , B ) + ϵ .
Then
d ( b ϵ , A ) ≤ sup b ∈ B d ( b , A ) ≤ d H ( A , B ) . d(\mathbf{b}_{\epsilon},A) \leq\sup_{\mathbf{b}\in B}d(\mathbf{b},A) \leq d_H(A,B). d ( b ϵ , A ) ≤ b ∈ B sup d ( b , A ) ≤ d H ( A , B ) .
Hence, there exists a ϵ ∈ A \mathbf{a}_{\epsilon} \in A a ϵ ∈ A such that
∥ b ϵ − a ϵ ∥ 2 ≤ d H ( A , B ) + ϵ . \|\mathbf{b}_{\epsilon}-\mathbf{a}_{\epsilon}\|_2 \leq d_H(A,B)+\epsilon. ∥ b ϵ − a ϵ ∥ 2 ≤ d H ( A , B ) + ϵ .
Therefore,
d ( u 1 , A ) ≤ ∥ u 1 − a ϵ ∥ 2 ≤ ∥ u 1 − b ϵ ∥ 2 + ∥ b ϵ − a ϵ ∥ 2 ≤ d ( u 1 , B ) + d H ( A , B ) + 2 ϵ . d(\mathbf{u}_1,A) \leq\|\mathbf{u}_1-\mathbf{a}_{\epsilon}\|_2 \leq\|\mathbf{u}_1-\mathbf{b}_{\epsilon}\|_2+\|\mathbf{b}_{\epsilon}-\mathbf{a}_{\epsilon}\|_2 \leq d(\mathbf{u}_1,B)+d_H(A,B)+2\epsilon. d ( u 1 , A ) ≤ ∥ u 1 − a ϵ ∥ 2 ≤ ∥ u 1 − b ϵ ∥ 2 + ∥ b ϵ − a ϵ ∥ 2 ≤ d ( u 1 , B ) + d H ( A , B ) + 2 ϵ .
Letting ϵ → 0 \epsilon\to0 ϵ → 0 yields
d ( u 1 , A ) ≤ d ( u 1 , B ) + d H ( A , B ) . d(\mathbf{u}_1,A) \leq d(\mathbf{u}_1,B)+d_H(A,B). d ( u 1 , A ) ≤ d ( u 1 , B ) + d H ( A , B ) .
Interchanging the roles of A A A and B B B , we also obtain
d ( u 1 , B ) ≤ d ( u 1 , A ) + d H ( A , B ) . d(\mathbf{u}_1,B) \leq d(\mathbf{u}_1,A)+d_H(A,B). d ( u 1 , B ) ≤ d ( u 1 , A ) + d H ( A , B ) .
Combining the two inequalities gives
∣ d ( u 1 , A ) − d ( u 1 , B ) ∣ ≤ d H ( A , B ) . |d(\mathbf{u}_1,A)-d(\mathbf{u}_1,B)| \leq d_H(A,B). ∣ d ( u 1 , A ) − d ( u 1 , B ) ∣ ≤ d H ( A , B ) .
Next, for any b ∈ B \mathbf{b} \in B b ∈ B , the triangle inequality gives
∥ u 1 − b ∥ 2 ≤ ∥ u 1 − u 2 ∥ 2 + ∥ u 2 − b ∥ 2 . \|\mathbf{u}_1-\mathbf{b}\|_2 \leq\|\mathbf{u}_1-\mathbf{u}_2\|_2+\|\mathbf{u}_2-\mathbf{b}\|_2. ∥ u 1 − b ∥ 2 ≤ ∥ u 1 − u 2 ∥ 2 + ∥ u 2 − b ∥ 2 .
Taking the infimum over b ∈ B \mathbf{b} \in B b ∈ B yields
d ( u 1 , B ) ≤ ∥ u 1 − u 2 ∥ 2 + d ( u 2 , B ) . d(\mathbf{u}_1,B) \leq\|\mathbf{u}_1-\mathbf{u}_2\|_2+d(\mathbf{u}_2,B). d ( u 1 , B ) ≤ ∥ u 1 − u 2 ∥ 2 + d ( u 2 , B ) .
Interchanging u 1 \mathbf{u}_1 u 1 and u 2 \mathbf{u}_2 u 2 gives
d ( u 2 , B ) ≤ ∥ u 1 − u 2 ∥ 2 + d ( u 1 , B ) . d(\mathbf{u}_2,B) \leq\|\mathbf{u}_1-\mathbf{u}_2\|_2+d(\mathbf{u}_1,B). d ( u 2 , B ) ≤ ∥ u 1 − u 2 ∥ 2 + d ( u 1 , B ) .
Hence,
∣ d ( u 1 , B ) − d ( u 2 , B ) ∣ ≤ ∥ u 1 − u 2 ∥ 2 . |d(\mathbf{u}_1,B)-d(\mathbf{u}_2,B)| \leq\|\mathbf{u}_1-\mathbf{u}_2\|_2. ∣ d ( u 1 , B ) − d ( u 2 , B ) ∣ ≤ ∥ u 1 − u 2 ∥ 2 .
This completes the proof.
Lemma E.5 (Hausdorff bound under a shared parametrization). Let U U U be a nonempty set, and let f , g : U → R K f,g:U\to\mathbb{R}^{K} f , g : U → R K be two functions. Define
A : = f ( U ) and B : = g ( U ) . A:=f(U) \qquad\text{and}\qquad B:=g(U). A := f ( U ) and B := g ( U ) . Then,
d H ( A , B ) ≤ sup u ∈ U ∥ f ( u ) − g ( u ) ∥ 2 . d_H(A,B)\leq\sup_{u\in U}\lVert f(\mathbf{u})-g(\mathbf{u})\rVert_2. d H ( A , B ) ≤ u ∈ U sup ∥ f ( u ) − g ( u ) ∥ 2 . where d H ( A , B ) d_H(A,B) d H ( A , B ) denotes the Hausdorff distance between A A A and B B B .
Proof. By definition of the Hausdorff distance, it suffices to show that
sup a ∈ A d ( a , B ) ≤ sup u ∈ U ∥ f ( u ) − g ( u ) ∥ 2 and sup b ∈ B d ( b , A ) ≤ sup u ∈ U ∥ f ( u ) − g ( u ) ∥ 2 . \sup_{\mathbf{a}\in A}d(\mathbf{a},B)\leq\sup_{u\in U}\lVert f(\mathbf{u})-g(\mathbf{u})\rVert_2
\qquad\text{and}\qquad
\sup_{\mathbf{b}\in B}d(\mathbf{b},A)\leq\sup_{u\in U}\lVert f(\mathbf{u})-g(\mathbf{u})\rVert_2. a ∈ A sup d ( a , B ) ≤ u ∈ U sup ∥ f ( u ) − g ( u ) ∥ 2 and b ∈ B sup d ( b , A ) ≤ u ∈ U sup ∥ f ( u ) − g ( u ) ∥ 2 . We first prove the first inequality. Let a ∈ A \mathbf{a}\in A a ∈ A be arbitrary. Then there exists some u ∈ U u\in U u ∈ U such that a = f ( u ) \mathbf{a}=f(\mathbf{u}) a = f ( u ) . Since g ( u ) ∈ B g(\mathbf{u})\in B g ( u ) ∈ B , we have
d ( a , B ) = inf b ∈ B ∥ a − b ∥ 2 ≤ ∥ a − g ( u ) ∥ 2 = ∥ f ( u ) − g ( u ) ∥ 2 . d(\mathbf{a},B)=\inf_{\mathbf{b}\in B}\lVert\mathbf{a}-\mathbf{b}\rVert_2\leq\lVert\mathbf{a}-g(\mathbf{u})\rVert_2=\lVert f(\mathbf{u})-g(\mathbf{u})\rVert_2. d ( a , B ) = b ∈ B inf ∥ a − b ∥ 2 ≤ ∥ a − g ( u ) ∥ 2 = ∥ f ( u ) − g ( u ) ∥ 2 . Taking the supremum over a ∈ A \mathbf{a}\in A a ∈ A yields
sup a ∈ A d ( a , B ) ≤ sup u ∈ U ∥ f ( u ) − g ( u ) ∥ 2 . \sup_{\mathbf{a}\in A}d(\mathbf{a},B)\leq\sup_{u\in U}\lVert f(\mathbf{u})-g(\mathbf{u})\rVert_2. a ∈ A sup d ( a , B ) ≤ u ∈ U sup ∥ f ( u ) − g ( u ) ∥ 2 . The second inequality follows analogously. Indeed, for any b ∈ B \mathbf{b}\in B b ∈ B , there exists u ∈ U u\in U u ∈ U such that b = g ( u ) \mathbf{b}=g(\mathbf{u}) b = g ( u ) , and since f ( u ) ∈ A f(\mathbf{u})\in A f ( u ) ∈ A ,
d ( b , A ) = inf a ∈ A ∥ b − a ∥ 2 ≤ ∥ b − f ( u ) ∥ 2 = ∥ g ( u ) − f ( u ) ∥ 2 . d(\mathbf{b},A)=\inf_{\mathbf{a}\in A}\lVert\mathbf{b}-\mathbf{a}\rVert_2\leq\lVert\mathbf{b}-f(\mathbf{u})\rVert_2=\lVert g(\mathbf{u})-f(\mathbf{u})\rVert_2. d ( b , A ) = a ∈ A inf ∥ b − a ∥ 2 ≤ ∥ b − f ( u ) ∥ 2 = ∥ g ( u ) − f ( u ) ∥ 2 . Taking the supremum over b ∈ B \mathbf{b}\in B b ∈ B gives
sup b ∈ B d ( b , A ) ≤ sup u ∈ U ∥ f ( u ) − g ( u ) ∥ 2 . \sup_{\mathbf{b}\in B}d(\mathbf{b},A)\leq\sup_{u\in U}\lVert f(\mathbf{u})-g(\mathbf{u})\rVert_2. b ∈ B sup d ( b , A ) ≤ u ∈ U sup ∥ f ( u ) − g ( u ) ∥ 2 . Combining the two bounds proves the result.
Deterministic almost-pure recovery
For every l ∈ [ L ] l\in[L] l ∈ [ L ] , define an embedding-stability event
G l : = { h ^ ( l ) ≤ c s t a b σ B ( l ) 2 K } . G_l:=\left\{\widehat{h}^{(l)}\leq\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K}\right\}. G l := ⎩ ⎨ ⎧ h ( l ) ≤ 2 K c stab σ B ( l ) ⎭ ⎬ ⎫ .
We state a deterministic result of the corner estimation procedure. The following theorem is used in the proof of Theorem 3.3.
Theorem E.1 (Deterministic almost-pure recovery). Suppose that Assumptions 1 1 1 , 2 2 2 , and 3 3 3 , together with Condition 1 1 1 , hold. Define the layer-aggregated embedding-error quantities and the block-signal summary
h ˉ : = 1 L ∑ l = 1 L h ^ ( l ) , p ˉ : = 1 L ∑ l = 1 L 1 G l c , σ ˉ B : = 1 L ∑ l = 1 L σ B ( l ) . \bar{h}:=\frac{1}{L}\sum_{l=1}^{L}\widehat{h}^{(l)},\qquad
\bar{p}:=\frac{1}{L}\sum_{l=1}^{L}\mathbf{1}_{G_l^{c}},\qquad
\bar{\sigma}_B:=\frac{1}{L}\sum_{l=1}^{L}\sigma_B^{(l)}. h ˉ := L 1 l = 1 ∑ L h ( l ) , p ˉ := L 1 l = 1 ∑ L 1 G l c , σ ˉ B := L 1 l = 1 ∑ L σ B ( l ) . Moreover, let
ρ V H : = 2 K 3 / 2 c L i p σ ˉ B [ ( 1 + 4 K c s t a b σ B , inf ) h ˉ + 2 p ˉ ] \rho_{\mathrm{VH}}:=\frac{2K^{3/2}}{c_{\mathrm{Lip}}\bar{\sigma}_B}\left[\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\inf}}}\right)\bar{h}+2\bar{p}\right] ρ VH := c Lip σ ˉ B 2 K 3/2 [ ( 1 + c stab σ B , i n f 4 K ) h ˉ + 2 p ˉ ]
denote the vertex-hunting error radius. Suppose the corner candidate set I \mathcal{I} I contains at least one pure node from each community. Then, every minimizer
J ^ ∈ arg min ∣ J ∣ = K Φ ~ L , I ( J ) \widehat{\mathcal{J}} \in\mathop{\arg\min}_{|\mathcal{J}|=K} \widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J}) J ∈ arg min ∣ J ∣ = K Φ L , I ( J )
satisfies
min 1 ≤ k ≤ K max j ∈ J ^ Z j k ≥ 1 − ρ V H . \min_{1\leq k\leq K}\max_{j\in\widehat{\mathcal{J}}} Z_{jk} \geq1-\rho_{\mathrm{VH}}. 1 ≤ k ≤ K min j ∈ J max Z j k ≥ 1 − ρ VH .
Proof. Given any corner candidate set I ⊆ [ n ] \mathcal{I}\subseteq[n] I ⊆ [ n ] , we define the population averaged vertex-hunting objective by
Φ L , I ( J ) : = max i ∈ I 1 L ∑ l = 1 L d ( Y i , ⋅ ( l ) , S S ( Y ( l ) , J ) ) . \Phi_{L,\mathcal{I}}(\mathcal{J}) := \max_{i\in\mathcal{I}} \frac{1}{L}\sum_{l=1}^{L}d\left(\mathbf{Y}^{(l)}_{i,\cdot},SS(\mathbf{Y}^{(l)},\mathcal{J})\right). Φ L , I ( J ) := i ∈ I max L 1 l = 1 ∑ L d ( Y i , ⋅ ( l ) , S S ( Y ( l ) , J ) ) .
For each k ∈ [ K ] k\in[K] k ∈ [ K ] , choose a pure node i k i_k i k from community k k k , and define a pure node set
J ⋆ : = { i 1 , … , i K } . \mathcal{J}^{\star}:=\{i_1,\ldots,i_K\}. J ⋆ := { i 1 , … , i K } .
Let
J ^ ∈ arg min ∣ J ∣ = K Φ ~ L , I ( J ) \widehat{\mathcal{J}} \in\mathop{\arg\min}_{|\mathcal{J}|=K} \widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J}) J ∈ arg min ∣ J ∣ = K Φ L , I ( J )
be an arbitrary minimizer of the empirical averaged vertex-hunting objective defined in (6) . We argue by contradiction. Suppose there exists k ∈ [ K ] k\in[K] k ∈ [ K ] such that
Z j k < 1 − r for all j ∈ J ^ . Z_{jk}<1-r\qquad\text{for all }j\in\widehat{\mathcal{J}}. Z j k < 1 − r for all j ∈ J .
Under this assumption, we will show that
Φ ~ L , I ( J ⋆ ) < Φ ~ L , I ( J ^ ) , \widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J}^{\star})<\widetilde{\Phi}_{L,\mathcal{I}}(\widehat{\mathcal{J}}), Φ L , I ( J ⋆ ) < Φ L , I ( J ) ,
which contradicts the optimality of J ^ \widehat{\mathcal{J}} J . Fix k k k . First, we obtain a lower bound of Φ L , I ( J ^ ) \Phi_{L,\mathcal{I}}(\widehat{\mathcal{J}}) Φ L , I ( J ) . By Lemma E.2 ,
S S ( Y ( l ) , J ^ ) = T ( l ) ( conv { Z j , ⋅ : j ∈ J ^ } ) . SS(\mathbf{Y}^{(l)},\widehat{\mathcal{J}})=T^{(l)}\left(\operatorname{conv}\{Z_{j,\cdot}:j\in\widehat{\mathcal{J}}\}\right). S S ( Y ( l ) , J ) = T ( l ) ( conv { Z j , ⋅ : j ∈ J } ) .
Since the convex hull is compact and T ( l ) T^{(l)} T ( l ) is continuous on Δ K − 1 \Delta^{K-1} Δ K − 1 , the set S S ( Y ( l ) , J ^ ) SS(\mathbf{Y}^{(l)},\widehat{\mathcal{J}}) S S ( Y ( l ) , J ) is compact. Therefore, the infimum in d ( Y i k , ⋅ ( l ) , S S ( Y ( l ) , J ^ ) ) d(\mathbf{Y}^{(l)}_{i_k,\cdot},SS(\mathbf{Y}^{(l)},\widehat{\mathcal{J}})) d ( Y i k , ⋅ ( l ) , S S ( Y ( l ) , J )) is attained at some point in S S ( Y ( l ) , J ^ ) SS(\mathbf{Y}^{(l)},\widehat{\mathcal{J}}) S S ( Y ( l ) , J ) for every l ∈ [ L ] l\in[L] l ∈ [ L ] . Let y ⋆ ( l ) ∈ S S ( Y ( l ) , J ^ ) \mathbf{y}^{\star(l)}\in SS(\mathbf{Y}^{(l)},\widehat{\mathcal{J}}) y ⋆ ( l ) ∈ S S ( Y ( l ) , J ) such that
d ( Y i k , ⋅ ( l ) , S S ( Y ( l ) , J ^ ) ) = ∥ Y i k , ⋅ ( l ) − y ⋆ ( l ) ∥ 2 . d(\mathbf{Y}^{(l)}_{i_k,\cdot},SS(\mathbf{Y}^{(l)},\widehat{\mathcal{J}}))=\left\|\mathbf{Y}^{(l)}_{i_k,\cdot}-\mathbf{y}^{\star(l)}\right\|_2. d ( Y i k , ⋅ ( l ) , S S ( Y ( l ) , J )) = Y i k , ⋅ ( l ) − y ⋆ ( l ) 2 .
By Lemma E.2 , there exists z ⋆ ( l ) ∈ conv { z j : j ∈ J ^ } \mathbf{z}^{\star(l)}\in\operatorname{conv}\{ \mathbf{z}_j:j\in\widehat{\mathcal{J}}\} z ⋆ ( l ) ∈ conv { z j : j ∈ J } such that T ( l ) ( z ⋆ ( l ) ) = y ⋆ ( l ) T^{(l)}(\mathbf{z}^{\star(l)})=\mathbf{y}^{\star(l)} T ( l ) ( z ⋆ ( l ) ) = y ⋆ ( l ) . Note that
z k ⋆ ( l ) = ∑ j ∈ J ^ c j ( l ) z j ( k ) ≤ ∑ j ∈ J ^ c j ( l ) ( 1 − r ) = ( 1 − r ) . z_k^{\star(l)}=\sum_{j\in\widehat{\mathcal{J}}}c_j^{(l)}z_j(k)\leq\sum_{j\in\widehat{\mathcal{J}}}c_j^{(l)}(1-r)=(1-r). z k ⋆ ( l ) = j ∈ J ∑ c j ( l ) z j ( k ) ≤ j ∈ J ∑ c j ( l ) ( 1 − r ) = ( 1 − r ) .
By bi-Lipschitz continuity of T ( l ) T^{(l)} T ( l ) ,
d ( Y i k , ⋅ ( l ) , S S ( Y ( l ) , J ^ ) ) = ∥ T ( l ) ( z ⋆ ( l ) ) − T ( l ) ( e k ) ∥ 2 ≥ c L i p σ B ( l ) K 3 / 2 ∥ z ⋆ ( l ) − e k ∥ 2 ≥ c L i p σ B ( l ) K 3 / 2 r . d(\mathbf{Y}^{(l)}_{i_k,\cdot},SS(\mathbf{Y}^{(l)},\widehat{\mathcal{J}}))=\left\|T^{(l)}(\mathbf{z}^{\star(l)})-T^{(l)}(\mathbf{e}_k)\right\|_2\geq c_{\mathrm{Lip}}\frac{\sigma_B^{(l)}}{K^{3/2}}\left\|\mathbf{z}^{\star(l)}-\mathbf{e}_k\right\|_2\geq c_{\mathrm{Lip}}\frac{\sigma_B^{(l)}}{K^{3/2}}r. d ( Y i k , ⋅ ( l ) , S S ( Y ( l ) , J )) = T ( l ) ( z ⋆ ( l ) ) − T ( l ) ( e k ) 2 ≥ c Lip K 3/2 σ B ( l ) z ⋆ ( l ) − e k 2 ≥ c Lip K 3/2 σ B ( l ) r .
Therefore, we obtain a lower bound of Φ L ( J ^ ) \Phi_L(\widehat{\mathcal{J}}) Φ L ( J ) given by:
Φ L , I ( J ^ ) ≥ 1 L ∑ l = 1 L d ( Y i k , ⋅ ( l ) , S S ( Y ( l ) , J ^ ) ) = 1 L ∑ l = 1 L ∥ T ( l ) ( e k ) − T ( l ) ( z ∗ ( l ) ) ∥ 2 ≥ 1 L ∑ l = 1 L c L i p σ B ( l ) K 3 / 2 r = c L i p σ ‾ B K 3 / 2 r . (62) \begin{aligned}
\Phi_{L,I}(\widehat{\mathcal{J}}) &\ge\frac{1}{L}\sum_{l=1}^{L}d(\mathbf{Y}^{(l)}_{i_k,\cdot},SS(\mathbf{Y}^{(l)},\widehat{\mathcal{J}})) \\
&= \frac{1}{L}\sum_{l=1}^{L}\left\|T^{(l)}(\mathbf{e}_k)-T^{(l)}(\mathbf{z}^{*(l)})\right\|_2 \\
&\ge\frac{1}{L}\sum_{l=1}^{L}c_{\mathrm{Lip}}\frac{\sigma_B^{(l)}}{K^{3/2}}r \\
&=c_{\mathrm{Lip}}\frac{\overline{\sigma}_B}{K^{3/2}}r. \tag*{(62)}
\end{aligned} Φ L , I ( J ) ≥ L 1 l = 1 ∑ L d ( Y i k , ⋅ ( l ) , S S ( Y ( l ) , J )) = L 1 l = 1 ∑ L T ( l ) ( e k ) − T ( l ) ( z ∗ ( l ) ) 2 ≥ L 1 l = 1 ∑ L c Lip K 3/2 σ B ( l ) r = c Lip K 3/2 σ B r . (62)
On the other hand, since J ∗ \mathcal{J}^* J ∗ is a true pure node index set, we have d ( Y i , ⋅ ( l ) , S S ( Y ( l ) , J ∗ ) ) = 0 d(\mathbf{Y}^{(l)}_{i,\cdot},SS(\mathbf{Y}^{(l)},\mathcal{J}^*))=0 d ( Y i , ⋅ ( l ) , S S ( Y ( l ) , J ∗ )) = 0 for all i ∈ [ n ] i\in[n] i ∈ [ n ] . Hence we have
Φ L , I ( J ∗ ) = 0 and Φ L , I ( J ^ ) − Φ L , I ( J ∗ ) ≥ c L i p σ ‾ B K 3 / 2 r . \Phi_{L,I}(\mathcal{J}^*)=0 \quad\text{and}\quad\Phi_{L,I}(\widehat{\mathcal{J}})-\Phi_{L,I}(\mathcal{J}^*)\ge c_{\mathrm{Lip}}\frac{\overline{\sigma}_B}{K^{3/2}}r. Φ L , I ( J ∗ ) = 0 and Φ L , I ( J ) − Φ L , I ( J ∗ ) ≥ c Lip K 3/2 σ B r .
Next, we bound ∣ Φ L , I ( J ) − Φ ~ L , I ( J ) ∣ \left|\Phi_{L,I}(\mathcal{J})-\widetilde{\Phi}_{L,I}(\mathcal{J})\right| Φ L , I ( J ) − Φ L , I ( J ) for an arbitrary index set J = { j 1 , … , j K } \mathcal{J}=\{j_1,\ldots,j_K\} J = { j 1 , … , j K } with ∣ J ∣ = K |\mathcal{J}|=K ∣ J ∣ = K . Fix l ∈ [ L ] l\in[L] l ∈ [ L ] . By Lemma E.4,
∣ d ( Y i , ⋅ ( l ) , S S ( Y ( l ) , J ) ) − d ( Y ~ i , ⋅ ( l ) , S S ( Y ~ ( l ) , J ) ) ∣ ≤ ∥ Y i , ⋅ ( l ) − Y ~ i , ⋅ ( l ) ∥ 2 + d H ( S S ( Y ( l ) , J ) , S S ( Y ~ ( l ) , J ) ) . (63) \left|d(\mathbf{Y}^{(l)}_{i,\cdot},SS(\mathbf{Y}^{(l)},\mathcal{J}))-d(\widetilde{\mathbf{Y}}^{(l)}_{i,\cdot},SS(\widetilde{\mathbf{Y}}^{(l)},\mathcal{J}))\right|
\le\left\|\mathbf{Y}^{(l)}_{i,\cdot}-\widetilde{\mathbf{Y}}^{(l)}_{i,\cdot}\right\|_2+d_H(SS(\mathbf{Y}^{(l)},\mathcal{J}),SS(\widetilde{\mathbf{Y}}^{(l)},\mathcal{J})). \tag*{(63)} d ( Y i , ⋅ ( l ) , S S ( Y ( l ) , J )) − d ( Y i , ⋅ ( l ) , S S ( Y ( l ) , J )) ≤ Y i , ⋅ ( l ) − Y i , ⋅ ( l ) 2 + d H ( S S ( Y ( l ) , J ) , S S ( Y ( l ) , J )) . (63)
We bound the Hausdorff distance in (E.3). Define two matrices S J ( l ) , S ~ J ( l ) ∈ R K × K \mathbf{S}^{(l)}_{\mathcal{J}},\widetilde{\mathbf{S}}^{(l)}_{\mathcal{J}}\in\mathbb{R}^{K\times K} S J ( l ) , S J ( l ) ∈ R K × K by
( S J ( l ) ) k , ⋅ = ( y j k ( l ) ) ⊤ , ( S ~ J ( l ) ) k , ⋅ = ( y ~ j k ( l ) ) ⊤ , k ∈ [ K ] . (\mathbf{S}^{(l)}_{\mathcal{J}})_{k,\cdot}=(\mathbf{y}^{(l)}_{j_k})^\top,\qquad(\widetilde{\mathbf{S}}^{(l)}_{\mathcal{J}})_{k,\cdot}=(\widetilde{\mathbf{y}}^{(l)}_{j_k})^\top,\qquad k\in[K]. ( S J ( l ) ) k , ⋅ = ( y j k ( l ) ) ⊤ , ( S J ( l ) ) k , ⋅ = ( y j k ( l ) ) ⊤ , k ∈ [ K ] .
By Lemma E.5 , we have
d H ( S S ( Y ( l ) , J ) , S S ( l ) ( Y ~ ( l ) , J ) ) ≤ sup z ∈ Δ K − 1 ∥ ( S J ( l ) ) ⊤ z ∥ ( S J ( l ) ) ⊤ z ∥ 2 − ( S ~ J ( l ) ) ⊤ z ∥ ( S ~ J ( l ) ) ⊤ z ∥ 2 ∥ 2 . (64) d_H(SS(\mathbf{Y}^{(l)},\mathcal{J}),SS^{(l)}(\widetilde{\mathbf{Y}}^{(l)},\mathcal{J}))\le\sup_{\mathbf{z}\in\Delta^{K-1}}\left\|\frac{(\mathbf{S}^{(l)}_{\mathcal{J}})^\top\mathbf{z}}{\|(\mathbf{S}^{(l)}_{\mathcal{J}})^\top\mathbf{z}\|_2}-\frac{(\widetilde{\mathbf{S}}^{(l)}_{\mathcal{J}})^\top\mathbf{z}}{\|(\widetilde{\mathbf{S}}^{(l)}_{\mathcal{J}})^\top\mathbf{z}\|_2}\right\|_2. \tag*{(64)} d H ( S S ( Y ( l ) , J ) , S S ( l ) ( Y ( l ) , J )) ≤ z ∈ Δ K − 1 sup ∥ ( S J ( l ) ) ⊤ z ∥ 2 ( S J ( l ) ) ⊤ z − ∥ ( S J ( l ) ) ⊤ z ∥ 2 ( S J ( l ) ) ⊤ z 2 . (64)
First, we show the denominators in (E.4) is bounded away from 0 on the event G l G_l G l . For every z ∈ Δ K − 1 \mathbf{z}\in\Delta^{K-1} z ∈ Δ K − 1 ,
∥ ( S J ( l ) ) ⊤ z ∥ 2 ≥ ⟨ n ( l ) ∥ n ( l ) ∥ 2 , ( S J ( l ) ) ⊤ z ⟩ = ∑ m = 1 K z m ⟨ n ( l ) ∥ n ( l ) ∥ 2 , y j m ( l ) ⟩ ≥ c s t a b σ B ( l ) K , \begin{aligned}
\|(\mathbf{S}^{(l)}_{\mathcal{J}})^\top\mathbf{z}\|_2\ge\left\langle\frac{\mathbf{n}^{(l)}}{\|\mathbf{n}^{(l)}\|_2},(\mathbf{S}^{(l)}_{\mathcal{J}})^\top\mathbf{z}\right\rangle\\
&=\sum_{m=1}^{K}z_m\left\langle\frac{\mathbf{n}^{(l)}}{\|\mathbf{n}^{(l)}\|_2},\mathbf{y}^{(l)}_{j_m}\right\rangle\ge c_{\mathrm{stab}}\frac{\sqrt{\sigma_B^{(l)}}}{K},
\end{aligned} ∥ ( S J ( l ) ) ⊤ z ∥ 2 ≥ ⟨ ∥ n ( l ) ∥ 2 n ( l ) , ( S J ( l ) ) ⊤ z ⟩ = m = 1 ∑ K z m ⟨ ∥ n ( l ) ∥ 2 n ( l ) , y j m ( l ) ⟩ ≥ c stab K σ B ( l ) ,
and similarly,
∥ S ~ J ( l ) ⊤ z ∥ 2 ≥ ⟨ n ( l ) ∥ n ( l ) ∥ 2 , S ~ J ( l ) ⊤ z ⟩ = ∑ m = 1 K z m ⟨ n ( l ) ∥ n ( l ) ∥ 2 , y ~ j m ( l ) ⟩ ≥ c s t a b σ B ( l ) 2 K . \begin{aligned}
\|\widetilde{\mathbf{S}}^{(l)\top}_{\mathcal{J}}\mathbf{z}\|_2\ge\left\langle\frac{\mathbf{n}^{(l)}}{\|\mathbf{n}^{(l)}\|_2},\widetilde{\mathbf{S}}^{(l)\top}_{\mathcal{J}}\mathbf{z}\right\rangle\\
&=\sum_{m=1}^{K}z_m\left\langle\frac{\mathbf{n}^{(l)}}{\|\mathbf{n}^{(l)}\|_2},\widetilde{\mathbf{y}}^{(l)}_{j_m}\right\rangle\ge\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K}.
\end{aligned} ∥ S J ( l ) ⊤ z ∥ 2 ≥ ⟨ ∥ n ( l ) ∥ 2 n ( l ) , S J ( l ) ⊤ z ⟩ = m = 1 ∑ K z m ⟨ ∥ n ( l ) ∥ 2 n ( l ) , y j m ( l ) ⟩ ≥ 2 K c stab σ B ( l ) .
Hence,
min { ∥ ( S J ( l ) ) ⊤ z ∥ 2 , ∥ S ~ J ( l ) ⊤ z ∥ 2 } ≥ c s t a b σ B ( l ) 2 K . \min\left\{\|(\mathbf{S}^{(l)}_{\mathcal{J}})^\top\mathbf{z}\|_2,\|\widetilde{\mathbf{S}}^{(l)\top}_{\mathcal{J}}\mathbf{z}\|_2\right\}\ge\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K}. min { ∥ ( S J ( l ) ) ⊤ z ∥ 2 , ∥ S J ( l ) ⊤ z ∥ 2 } ≥ 2 K c stab σ B ( l ) .
Next, we bound numerator in (E.4), we have
∥ ( S J ( l ) − S ~ J ( l ) ) T z ∥ 2 = ∥ ∑ m = 1 K z m ( y j m ( l ) − y ~ j m ( l ) ) ∥ 2 ≤ ∑ m = 1 K z m ∥ y j m ( l ) − y ~ j m ( l ) ∥ 2 ≤ h ^ ( l ) . \begin{aligned}
\left\|\left(\mathbf{S}_{\mathcal{J}}^{(l)}-\widetilde{\mathbf{S}}_{\mathcal{J}}^{(l)}\right)^{\mathsf{T}}\mathbf{z}\right\|_{2}
=
\left\|\sum_{m=1}^{K}\mathbf{z}_{m}\left(\mathbf{y}_{j_m}^{(l)}-\widetilde{\mathbf{y}}_{j_m}^{(l)}\right)\right\|_{2} \\
&\leq\sum_{m=1}^{K}\mathbf{z}_{m}\left\|\mathbf{y}_{j_m}^{(l)}-\widetilde{\mathbf{y}}_{j_m}^{(l)}\right\|_{2} \\
&\leq\widehat{h}^{(l)}.
\end{aligned} ( S J ( l ) − S J ( l ) ) T z 2 = m = 1 ∑ K z m ( y j m ( l ) − y j m ( l ) ) 2 ≤ m = 1 ∑ K z m y j m ( l ) − y j m ( l ) 2 ≤ h ( l ) .
Therefore, by Lemma E.5,
d H ( S ( Y ( l ) , J ) , S ( l ) ( Y ~ ( l ) J ) ) ≤ sup z ∈ Δ K − 1 ∥ S J ( l ) T z ∥ S J ( l ) T z ∥ 2 − S ~ J ( l ) T z ∥ S ~ J ( l ) T z ∥ 2 ∥ 2 ≤ sup z ∈ Δ K − 1 2 ∥ ( S J ( l ) − S ~ J ( l ) ) T z ∥ 2 min { ∥ S J ( l ) T z ∥ 2 , ∥ S ~ J ( l ) T z ∥ 2 } ≤ 4 K h ^ ( l ) c s t a b σ B ( l ) . \begin{aligned}
d_{H}\left(\mathcal{S}\left(\mathbf{Y}^{(l)},\mathcal{J}\right),\mathcal{S}^{(l)}\left(\widetilde{\mathbf{Y}}^{(l)}\mathcal{J}\right)\right)
\leq
\sup_{\mathbf{z}\in\Delta^{K-1}}
\left\|
\frac{\mathbf{S}_{\mathcal{J}}^{(l)\mathsf{T}}\mathbf{z}}{\left\|\mathbf{S}_{\mathcal{J}}^{(l)\mathsf{T}}\mathbf{z}\right\|_{2}}
-
\frac{\widetilde{\mathbf{S}}_{\mathcal{J}}^{(l)\mathsf{T}}\mathbf{z}}{\left\|\widetilde{\mathbf{S}}_{\mathcal{J}}^{(l)\mathsf{T}}\mathbf{z}\right\|_{2}}
\right\|_{2} \\
&\leq
\sup_{\mathbf{z}\in\Delta^{K-1}}
\frac{2\left\|\left(\mathbf{S}_{\mathcal{J}}^{(l)}-\widetilde{\mathbf{S}}_{\mathcal{J}}^{(l)}\right)^{\mathsf{T}}\mathbf{z}\right\|_{2}}
{\min\left\{\left\|\mathbf{S}_{\mathcal{J}}^{(l)\mathsf{T}}\mathbf{z}\right\|_{2},\left\|\widetilde{\mathbf{S}}_{\mathcal{J}}^{(l)\mathsf{T}}\mathbf{z}\right\|_{2}\right\}} \\
&\leq\frac{4K\widehat{h}^{(l)}}{c_{\mathrm{stab}}\sqrt{\sigma_{B}^{(l)}}}.
\end{aligned} d H ( S ( Y ( l ) , J ) , S ( l ) ( Y ( l ) J ) ) ≤ z ∈ Δ K − 1 sup S J ( l ) T z 2 S J ( l ) T z − S J ( l ) T z 2 S J ( l ) T z 2 ≤ z ∈ Δ K − 1 sup min { S J ( l ) T z 2 , S J ( l ) T z 2 } 2 ( S J ( l ) − S J ( l ) ) T z 2 ≤ c stab σ B ( l ) 4 K h ( l ) .
Hence, on G l G_l G l ,
∣ d ( y i ( l ) , S ( Y ( l ) , J ) ) − d ( y ~ i ( l ) , S ( l ) ( Y ~ ( l ) J ) ) ∣ ≤ ( 1 + 4 K c s t a b σ B ( l ) ) h ^ ( l ) . (65) \left|d\left(\mathbf{y}_{i}^{(l)},\mathcal{S}\left(\mathbf{Y}^{(l)},\mathcal{J}\right)\right)-d\left(\widetilde{\mathbf{y}}_{i}^{(l)},\mathcal{S}^{(l)}\left(\widetilde{\mathbf{Y}}^{(l)}\mathcal{J}\right)\right)\right|
\leq
\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B}^{(l)}}}\right)\widehat{h}^{(l)}. \tag*{(65)} d ( y i ( l ) , S ( Y ( l ) , J ) ) − d ( y i ( l ) , S ( l ) ( Y ( l ) J ) ) ≤ 1 + c stab σ B ( l ) 4 K h ( l ) . (65)
On the complementary event G l c G_{l}^{c} G l c , both y i ( l ) \mathbf{y}_{i}^{(l)} y i ( l ) and y ~ i ( l ) \widetilde{\mathbf{y}}_{i}^{(l)} y i ( l ) lie on the unit sphere, while both S ( Y ( l ) , J ) \mathcal{S}\left(\mathbf{Y}^{(l)},\mathcal{J}\right) S ( Y ( l ) , J ) and S ( Y ~ ( l ) J ) \mathcal{S}\left(\widetilde{\mathbf{Y}}^{(l)}\mathcal{J}\right) S ( Y ( l ) J ) are nonempty subsets of the unit sphere. Therefore,
0 ≤ d ( y i ( l ) , S ( Y ( l ) , J ) ) ≤ 2 , 0 ≤ d ( y ~ i ( l ) , S ( Y ~ ( l ) J ) ) ≤ 2 , 0\leq d\left(\mathbf{y}_{i}^{(l)},\mathcal{S}\left(\mathbf{Y}^{(l)},\mathcal{J}\right)\right)\leq2,\qquad
0\leq d\left(\widetilde{\mathbf{y}}_{i}^{(l)},\mathcal{S}\left(\widetilde{\mathbf{Y}}^{(l)}\mathcal{J}\right)\right)\leq2, 0 ≤ d ( y i ( l ) , S ( Y ( l ) , J ) ) ≤ 2 , 0 ≤ d ( y i ( l ) , S ( Y ( l ) J ) ) ≤ 2 ,
and hence
∣ d ( y i ( l ) , S ( Y ( l ) , J ) ) − d ( y ~ i ( l ) , S ( Y ~ ( l ) J ) ) ∣ ≤ 2. (66) \left|d\left(\mathbf{y}_{i}^{(l)},\mathcal{S}\left(\mathbf{Y}^{(l)},\mathcal{J}\right)\right)-d\left(\widetilde{\mathbf{y}}_{i}^{(l)},\mathcal{S}\left(\widetilde{\mathbf{Y}}^{(l)}\mathcal{J}\right)\right)\right|\leq2. \tag*{(66)} d ( y i ( l ) , S ( Y ( l ) , J ) ) − d ( y i ( l ) , S ( Y ( l ) J ) ) ≤ 2. (66)
Combining (E.5) and (E.6) yields
∣ d ( y i ( l ) , S ( Y ( l ) , J ) ) − d ( y ~ i ( l ) , S ( Y ~ ( l ) J ) ) ∣ ≤ ( 1 + 4 K c s t a b σ B ( l ) ) h ^ ( l ) 1 G l + 21 G l c . \left|d\left(\mathbf{y}_{i}^{(l)},\mathcal{S}\left(\mathbf{Y}^{(l)},\mathcal{J}\right)\right)-d\left(\widetilde{\mathbf{y}}_{i}^{(l)},\mathcal{S}\left(\widetilde{\mathbf{Y}}^{(l)}\mathcal{J}\right)\right)\right|
\leq
\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B}^{(l)}}}\right)\widehat{h}^{(l)}\mathbf{1}_{G_l}+2\mathbf{1}_{G_l^c}. d ( y i ( l ) , S ( Y ( l ) , J ) ) − d ( y i ( l ) , S ( Y ( l ) J ) ) ≤ 1 + c stab σ B ( l ) 4 K h ( l ) 1 G l + 2 1 G l c .
Averaging over l l l and using σ B ( l ) ≥ σ B , min \sigma_{B}^{(l)}\geq\sigma_{B,\min} σ B ( l ) ≥ σ B , m i n gives
∣ 1 L ∑ l = 1 L d ( y i ( l ) , S ( Y ( l ) , J ) ) − 1 L ∑ l = 1 L d ( y ~ i ( l ) , S ( Y ~ ( l ) J ) ) ∣ ≤ 1 L ∑ l = 1 L [ ( 1 + 4 K c s t a b σ B ( l ) ) h ^ ( l ) 1 G l + 21 G l c ] ≤ ( 1 + 4 K c s t a b σ B , min ) h ‾ + 2 p ‾ . \begin{aligned}
\left|
\frac{1}{L}\sum_{l=1}^{L}d\left(\mathbf{y}_{i}^{(l)},\mathcal{S}\left(\mathbf{Y}^{(l)},\mathcal{J}\right)\right)
-
\frac{1}{L}\sum_{l=1}^{L}d\left(\widetilde{\mathbf{y}}_{i}^{(l)},\mathcal{S}\left(\widetilde{\mathbf{Y}}^{(l)}\mathcal{J}\right)\right)
\right| \\
&\leq
\frac{1}{L}\sum_{l=1}^{L}
\left[
\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B}^{(l)}}}\right)\widehat{h}^{(l)}\mathbf{1}_{G_l}
+2\mathbf{1}_{G_l^c}
\right] \\
&\leq
\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}\right)\overline{h}+2\overline{p}.
\end{aligned} L 1 l = 1 ∑ L d ( y i ( l ) , S ( Y ( l ) , J ) ) − L 1 l = 1 ∑ L d ( y i ( l ) , S ( Y ( l ) J ) ) ≤ L 1 l = 1 ∑ L 1 + c stab σ B ( l ) 4 K h ( l ) 1 G l + 2 1 G l c ≤ ( 1 + c stab σ B , m i n 4 K ) h + 2 p .
Since
∣ max i a i − max i b i ∣ ≤ max i ∣ a i − b i ∣ , (67) \left|\max_{i}a_i-\max_{i}b_i\right|\leq\max_{i}\left|a_i-b_i\right|, \tag*{(67)} i max a i − i max b i ≤ i max ∣ a i − b i ∣ , (67)
it follows that
∣ Φ L , I ( J ) − Φ ~ L , I ( J ) ∣ ≤ ( 1 + 4 K c s t a b σ B , min ) h ˉ + 2 p ˉ for every J ⊂ [ n ] with ∣ J ∣ = K . \left|\Phi_{L,\mathcal{I}}(\mathcal{J})-\widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J})\right| \le\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}\right)\bar{h}+2\bar{p}\quad\text{for every }\mathcal{J}\subset[n]\text{ with }|\mathcal{J}|=K. Φ L , I ( J ) − Φ L , I ( J ) ≤ ( 1 + c stab σ B , m i n 4 K ) h ˉ + 2 p ˉ for every J ⊂ [ n ] with ∣ J ∣ = K .
Applying this bound with J = J ^ \mathcal{J}=\widehat{\mathcal{J}} J = J and using (E.2), we obtain
Φ ~ L , I ( J ^ ) ≥ Φ L , I ( J ^ ) − ( 1 + 4 K c s t a b σ B , min ) h ˉ − 2 p ˉ ≥ c L i p σ ˉ B K 3 / 2 r − ( 1 + 4 K c s t a b σ B , min ) h ˉ − 2 p ˉ . \widetilde{\Phi}_{L,\mathcal{I}}(\widehat{\mathcal{J}}) \ge\Phi_{L,\mathcal{I}}(\widehat{\mathcal{J}})-\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}\right)\bar{h}-2\bar{p}\ge c_{\mathrm{Lip}}\frac{\bar{\sigma}_{B}}{K^{3/2}}r-\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}\right)\bar{h}-2\bar{p}. Φ L , I ( J ) ≥ Φ L , I ( J ) − ( 1 + c stab σ B , m i n 4 K ) h ˉ − 2 p ˉ ≥ c Lip K 3/2 σ ˉ B r − ( 1 + c stab σ B , m i n 4 K ) h ˉ − 2 p ˉ .
On the other hand, since J ∗ \mathcal{J}^{*} J ∗ is a pure node index set, Φ L , I ( J ∗ ) = 0 \Phi_{L,\mathcal{I}}(\mathcal{J}^{*})=0 Φ L , I ( J ∗ ) = 0 , and thus
Φ ~ L , I ( J ∗ ) ≤ Φ L , I ( J ∗ ) + ( 1 + 4 K c s t a b σ B , min ) h ˉ + 2 p ˉ = ( 1 + 4 K c s t a b σ B , min ) h ˉ + 2 p ˉ . \widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J}^{*})\le\Phi_{L,\mathcal{I}}(\mathcal{J}^{*})+\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}\right)\bar{h}+2\bar{p}=\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}\right)\bar{h}+2\bar{p}. Φ L , I ( J ∗ ) ≤ Φ L , I ( J ∗ ) + ( 1 + c stab σ B , m i n 4 K ) h ˉ + 2 p ˉ = ( 1 + c stab σ B , m i n 4 K ) h ˉ + 2 p ˉ .
Therefore, if
c L i p σ ˉ B K 3 / 2 r − ( 1 + 4 K c s t a b σ B , min ) h ˉ − 2 p ˉ > ( 1 + 4 K c s t a b σ B , min ) h ˉ + 2 p ˉ , c_{\mathrm{Lip}}\frac{\bar{\sigma}_{B}}{K^{3/2}}r-\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}\right)\bar{h}-2\bar{p}>\left(1+\frac{4K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}\right)\bar{h}+2\bar{p}, c Lip K 3/2 σ ˉ B r − ( 1 + c stab σ B , m i n 4 K ) h ˉ − 2 p ˉ > ( 1 + c stab σ B , m i n 4 K ) h ˉ + 2 p ˉ ,
then
Φ ~ L , I ( J ^ ) > Φ ~ L , I ( J ∗ ) , \widetilde{\Phi}_{L,\mathcal{I}}(\widehat{\mathcal{J}})>\widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J}^{*}), Φ L , I ( J ) > Φ L , I ( J ∗ ) ,
which contradicts the definition
J ^ ∈ arg min ∣ J ∣ = K Φ ~ L , I ( J ) . \widehat{\mathcal{J}}\in\underset{|\mathcal{J}|=K}{\arg\min}\ \widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J}). J ∈ ∣ J ∣ = K arg min Φ L , I ( J ) .
This completes the proof.
Proof of Theorem 3.3
We restate Theorem 3.3 for convenience.
Theorem 3.3 (High-probability almost-pure recovery). Suppose that K K K is fixed, the layers are independent, and Assumptions 1, 2, 3 and Condition 1 hold. For any candidate set I ⊆ [ n ] \mathcal{I}\subseteq[n] I ⊆ [ n ] containing at least one pure node from each community, let J ^ ( I ) ∈ arg min J ⊆ I ∣ J ∣ = K Φ ~ L , I ( J ) \widehat{\mathcal{J}}(\mathcal{I})\in\arg\min_{\substack{\mathcal{J}\subseteq\mathcal{I}\\|\mathcal{J}|=K}}\widetilde{\Phi}_{L,\mathcal{I}}(\mathcal{J}) J ( I ) ∈ arg min J ⊆ I ∣ J ∣ = K Φ L , I ( J ) . Then for any x > 0 x>0 x > 0 , with probability at least 1 − e − x 1-e^{-x} 1 − e − x , every candidate set I ⊆ [ n ] \mathcal{I}\subseteq[n] I ⊆ [ n ] containing at least one pure node from each community satisfies min k ∈ [ K ] max j ∈ J ^ ( I ) Z j k ≥ 1 − r n , L ( x ) \min_{k\in[K]}\max_{j\in\widehat{\mathcal{J}}(\mathcal{I})}Z_{jk}\ge1-r_{n,L}(x) min k ∈ [ K ] max j ∈ J ( I ) Z j k ≥ 1 − r n , L ( x ) .
Proof. By Theorem E.1, for the minimizer J ^ ( I ) \widehat{\mathcal{J}}(\mathcal{I}) J ( I ) , it follows that
min 1 ≤ k ≤ K max j ∈ J ^ ( I ) Z j k ≥ 1 − c ( h ˉ + p ˉ ) \min_{1\le k\le K}\max_{j\in\widehat{\mathcal{J}}(\mathcal{I})}Z_{jk}\ge1-c(\bar{h}+\bar{p}) 1 ≤ k ≤ K min j ∈ J ( I ) max Z j k ≥ 1 − c ( h ˉ + p ˉ )
for a universal constant c > 0 c>0 c > 0 . We show p ˉ = O ( h ˉ ) \bar{p}=O(\bar{h}) p ˉ = O ( h ˉ ) and obtain a high probability upper bound of h ˉ \bar{h} h ˉ using Bernstein’s inequality. On G l c G_l^c G l c , we have
h ^ ( l ) > c s t a b σ B ( l ) 2 K ≥ c s t a b σ B , min 2 K . \widehat{h}^{(l)}>\frac{c_{\mathrm{stab}}\sqrt{\sigma_B^{(l)}}}{2K}\ge\frac{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}{2K}. h ( l ) > 2 K c stab σ B ( l ) ≥ 2 K c stab σ B , m i n .
Hence,
p ˉ = 1 L ∑ l = 1 L 1 G l c ≤ 1 L ∑ l = 1 L 2 K h ^ ( l ) c s t a b σ B , min = 2 K c s t a b σ B , min h ˉ . \bar{p}=\frac{1}{L}\sum_{l=1}^{L}\mathbf{1}_{G_l^c}\le\frac{1}{L}\sum_{l=1}^{L}\frac{2K\widehat{h}^{(l)}}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}=\frac{2K}{c_{\mathrm{stab}}\sqrt{\sigma_{B,\min}}}\bar{h}. p ˉ = L 1 l = 1 ∑ L 1 G l c ≤ L 1 l = 1 ∑ L c stab σ B , m i n 2 K h ( l ) = c stab σ B , m i n 2 K h ˉ .
Under fixed K K K and assumption 2, we showed p ˉ = O ( h ˉ ) \bar{p}=O(\bar{h}) p ˉ = O ( h ˉ ) . Next, we obtain a high probability upper bound of h ˉ \bar{h} h ˉ using the Bernstein’s inequality. Recall that on the event E ( l ) \mathcal{E}^{(l)} E ( l ) defined in Appendix § 10 .§ 3 , we have h ^ ( l ) ≤ c Y β n ( l ) \hat{h}^{(l)} \le c_Y\sqrt{\beta_n^{(l)}} h ^ ( l ) ≤ c Y β n ( l ) with probability at least 1 − c E n − 18 1-c_E n^{-18} 1 − c E n − 18 . We decompose h ^ ( l ) \hat{h}^{(l)} h ^ ( l ) as below:
h ^ ( l ) = h ^ ( l ) 1 E ( l ) + h ^ ( l ) 1 ( E ( l ) ) c ≤ c Y β n ( l ) + 21 ( E ( l ) ) c . \hat{h}^{(l)}=\hat{h}^{(l)}\mathbf{1}_{\mathcal{E}^{(l)}}+\hat{h}^{(l)}\mathbf{1}_{(\mathcal{E}^{(l)})^c}\le c_Y\sqrt{\beta_n^{(l)}}+2\mathbf{1}_{(\mathcal{E}^{(l)})^c}. h ^ ( l ) = h ^ ( l ) 1 E ( l ) + h ^ ( l ) 1 ( E ( l ) ) c ≤ c Y β n ( l ) + 2 1 ( E ( l ) ) c .
Consequently, we have h ˉ ≤ c Y L ∑ l = 1 L β n ( l ) + 2 L ∑ l = 1 L 1 ( E ( l ) ) c \bar{h}\le\frac{c_Y}{L}\sum_{l=1}^{L}\sqrt{\beta_n^{(l)}}+\frac{2}{L}\sum_{l=1}^{L}\mathbf{1}_{(\mathcal{E}^{(l)})^c} h ˉ ≤ L c Y ∑ l = 1 L β n ( l ) + L 2 ∑ l = 1 L 1 ( E ( l ) ) c . Applying Bernstein’s inequality to ∑ l = 1 L 1 ( E ( l ) ) c \sum_{l=1}^{L}\mathbf{1}_{(\mathcal{E}^{(l)})^c} ∑ l = 1 L 1 ( E ( l ) ) c , we have
P ( ∑ l = 1 L 1 ( E ( l ) ) c − ∑ l = 1 L E [ 1 ( E ( l ) ) c ] > 2 L c E n − 18 x + 2 x 3 ) ≤ exp ( − x ) . \mathbb{P}\left(\sum_{l=1}^{L}\mathbf{1}_{(\mathcal{E}^{(l)})^c}-\sum_{l=1}^{L}\mathbb{E}[\mathbf{1}_{(\mathcal{E}^{(l)})^c}]>\sqrt{2Lc_E n^{-18}x}+\frac{2x}{3}\right)\le\exp(-x). P ( l = 1 ∑ L 1 ( E ( l ) ) c − l = 1 ∑ L E [ 1 ( E ( l ) ) c ] > 2 L c E n − 18 x + 3 2 x ) ≤ exp ( − x ) .
Hence, with probability at least 1 − exp ( − x ) 1-\exp(-x) 1 − exp ( − x ) , it follows that
1 L ∑ l = 1 L 1 ( E ( l ) ) c ≤ c E n − 18 + 2 c E x L n 18 + 2 x 3 L . (68) \frac{1}{L}\sum_{l=1}^{L}\mathbf{1}_{(\mathcal{E}^{(l)})^c}\le c_E n^{-18}+\sqrt{\frac{2c_E x}{Ln^{18}}}+\frac{2x}{3L}. \tag*{(68)} L 1 l = 1 ∑ L 1 ( E ( l ) ) c ≤ c E n − 18 + L n 18 2 c E x + 3 L 2 x . (68)
By AM-GM inequality, we have
1 L ∑ l = 1 L 1 ( E ( l ) ) c ≤ 3 2 c E n − 18 + 5 x 3 L . \frac{1}{L}\sum_{l=1}^{L}\mathbf{1}_{(\mathcal{E}^{(l)})^c}\le\frac{3}{2}c_E n^{-18}+\frac{5x}{3L}. L 1 l = 1 ∑ L 1 ( E ( l ) ) c ≤ 2 3 c E n − 18 + 3 L 5 x .
Therefore, there exists a universal constant C V H C_{\mathrm{VH}} C VH such that
h ˉ ≤ C V H ( 1 L ∑ l = 1 L β n ( l ) + n − 18 + x L ) (69) \bar{h}\le C_{\mathrm{VH}}\left(\frac{1}{L}\sum_{l=1}^{L}\sqrt{\beta_n^{(l)}}+n^{-18}+\frac{x}{L}\right) \tag*{(69)} h ˉ ≤ C VH ( L 1 l = 1 ∑ L β n ( l ) + n − 18 + L x ) (69)
with probability at least 1 − exp ( − x ) 1-\exp(-x) 1 − exp ( − x ) . Using p ˉ = O ( h ˉ ) \bar{p}=O(\bar{h}) p ˉ = O ( h ˉ ) and (E.8),
min 1 ≤ k ≤ K max j ∈ J ( I ) Z j k ≥ 1 − c ( h ˉ + p ˉ ) ≥ 1 − C h ˉ ≥ 1 − C V H ( 1 L ∑ l = 1 L β n ( l ) + n − 18 + x L ) \min_{1\le k\le K}\max_{j\in\mathcal{J}(\mathcal{I})}Z_{jk}\ge1-c(\bar{h}+\bar{p})\ge1-C\bar{h}\ge1-C_{\mathrm{VH}}\left(\frac{1}{L}\sum_{l=1}^{L}\sqrt{\beta_n^{(l)}}+n^{-18}+\frac{x}{L}\right) 1 ≤ k ≤ K min j ∈ J ( I ) max Z j k ≥ 1 − c ( h ˉ + p ˉ ) ≥ 1 − C h ˉ ≥ 1 − C VH ( L 1 l = 1 ∑ L β n ( l ) + n − 18 + L x )
with probability at least 1 − exp ( − x ) 1-\exp(-x) 1 − exp ( − x ) . This completes the proof. □
Proof of Corollary 3.5
We restate Corollary 3.5 for convenience.
Corollary 3.5 (Exact pure-node recovery). Suppose that the assumptions of Theorem 3.3 and Condition 2(b) hold. Fix x > 0 x>0 x > 0 such that r n , L ( x ) < δ m i x 2 r_{n,L}(x)<\frac{\delta_{\mathrm{mix}}}{\sqrt{2}} r n , L ( x ) < 2 δ mix . Then, with probability at least 1 − e − x 1-e^{-x} 1 − e − x , J ( I ) \mathcal{J}(\mathcal{I}) J ( I ) contains exactly one pure node from each community for any corner candidate set I \mathcal{I} I with I ∩ P k ≠ ∅ \mathcal{I}\cap\mathcal{P}_k\ne\varnothing I ∩ P k = ∅ for all k ∈ [ K ] k\in[K] k ∈ [ K ] .
Proof. For notational convenience, let r : = r n , L ( x ) r:=r_{n,L}(x) r := r n , L ( x ) . By definition of I k ( r ) = { i ∈ [ n ] : Z i k ≥ 1 − r } \mathcal{I}_k(r)=\{i\in[n]:Z_{ik}\ge1-r\} I k ( r ) = { i ∈ [ n ] : Z ik ≥ 1 − r } , it is straightforward to see P k ⊂ I k ( r ) \mathcal{P}_k\subset\mathcal{I}_k(r) P k ⊂ I k ( r ) for any r ≥ 0 r\ge0 r ≥ 0 . Therefore, we show I k ( r ) ⊂ P k \mathcal{I}_k(r)\subset\mathcal{P}_k I k ( r ) ⊂ P k with r < δ m i x 2 r<\frac{\delta_{\mathrm{mix}}}{\sqrt{2}} r < 2 δ mix . Let j ∉ P k j\notin\mathcal{P}_k j ∈ / P k be given. There are two possible cases for node j j j :
j ∈ P k ′ j\in\mathcal{P}_{k'} j ∈ P k ′ for some k ′ ≠ k k'\ne k k ′ = k .
j j j is a mixed node.
If j ∈ P k ′ j \in\mathcal{P}_{k'} j ∈ P k ′ for some k ′ ≠ k k' \ne k k ′ = k , then Z j k = 0 Z_{jk}=0 Z j k = 0 . Therefore, j ∉ I k ( r ) j \notin\mathcal{I}_k(r) j ∈ / I k ( r ) for any r < 1 r<1 r < 1 .
Assume j j j is a mixed node. Then by assumption, we have
∥ Z j , ⋅ − e k ∥ 2 ≥ δ m i x . \lVert\mathbf{Z}_{j,\cdot}-\mathbf{e}_k\rVert_2 \ge\delta_{\mathrm{mix}}. ∥ Z j , ⋅ − e k ∥ 2 ≥ δ mix .
Note that for any i ∈ I k ( r ) i \in\mathcal{I}_k(r) i ∈ I k ( r ) , it satisfies Z i k ≥ 1 − r Z_{ik} \ge1-r Z ik ≥ 1 − r and
∥ Z i , ⋅ − e k ∥ 2 2 = ( 1 − Z i k ) 2 + ∑ m ≠ k Z i m 2 ≤ ( 1 − Z i k ) 2 + ( ∑ m ≠ k Z i m ) 2 = 2 ( 1 − Z i k ) 2 \begin{aligned}
\lVert\mathbf{Z}_{i,\cdot}-\mathbf{e}_k\rVert_2^2=(1-Z_{ik})^2+\sum_{m\ne k}Z_{im}^2 \\
&\le(1-Z_{ik})^2+\left(\sum_{m\ne k}Z_{im}\right)^2 \\
&=2(1-Z_{ik})^2
\end{aligned} ∥ Z i , ⋅ − e k ∥ 2 2 = ( 1 − Z ik ) 2 + m = k ∑ Z im 2 ≤ ( 1 − Z ik ) 2 + m = k ∑ Z im 2 = 2 ( 1 − Z ik ) 2
Assume r < δ m i x 2 r<\frac{\delta_{\mathrm{mix}}}{\sqrt{2}} r < 2 δ mix . Since
min 1 ≤ k ≤ K ∥ Z j , ⋅ − e k ∥ 2 ≥ δ m i x > 2 r , \min_{1\le k\le K}\lVert\mathbf{Z}_{j,\cdot}-\mathbf{e}_k\rVert_2\ge\delta_{\mathrm{mix}}>\sqrt{2}r, 1 ≤ k ≤ K min ∥ Z j , ⋅ − e k ∥ 2 ≥ δ mix > 2 r ,
it is clear that j ∉ I k ( r ) j\notin\mathcal{I}_k(r) j ∈ / I k ( r ) . This completes the proof. □
Proof of Proposition 3.4
We restate Proposition 3.4 for convenience.
Proposition 3.4. Suppose Assumptions 1, 2, 3 and Condition 1, 2 are satisfied. Let M ^ m \widehat{\mathcal{M}}_m M m be a one-swap local minimizer of the m m m -medoids objective, where K ≤ m ≤ n K\le m\le n K ≤ m ≤ n . Define the population separation Δ 0 : = 1 L min k ∈ [ K ] min i ∈ [ n ] ∖ P k ∥ T ( Z i , ⋅ ) − T ( e k ) ∥ 2 \Delta_0:=\frac{1}{\sqrt{L}}\min_{k\in[K]}\min_{i\in[n]\setminus\mathcal{P}_k}\lVert T(Z_{i,\cdot})-T(\mathbf{e}_k)\rVert_2 Δ 0 := L 1 min k ∈ [ K ] min i ∈ [ n ] ∖ P k ∥ T ( Z i , ⋅ ) − T ( e k ) ∥ 2 . There exists a constant C > 0 C>0 C > 0 , independent of n n n , L L L , m m m , x x x , such that, for any x > 0 x>0 x > 0 , if
4 C ( 1 L ∑ l = 1 L β n ( l ) + n − 18 + x L ) 1 / 2 + 2 m p p u r e < Δ 0 , 4C\left(\frac{1}{L}\sum_{l=1}^{L}\beta_n^{(l)}+n^{-18}+\frac{x}{L}\right)^{1/2}+\frac{2}{m p_{\mathrm{pure}}}<\Delta_0, 4 C ( L 1 l = 1 ∑ L β n ( l ) + n − 18 + L x ) 1/2 + m p pure 2 < Δ 0 , then M ^ m ∩ P k ≠ ∅ \widehat{\mathcal{M}}_m\cap\mathcal{P}_k\ne\varnothing M m ∩ P k = ∅ for all k ∈ [ K ] k\in[K] k ∈ [ K ] with probability at least 1 − e − x 1-e^{-x} 1 − e − x .
Proof. Define the k k k -medoids objective with m m m medoids
F ( M ) : = 1 n ∑ i = 1 n min m ~ ∈ M 1 L ∥ y ~ i − y ~ m ~ ∥ 2 , ∣ M ∣ = m . F(\mathcal{M}):=\frac{1}{n}\sum_{i=1}^{n}\min_{\widetilde{m}\in\mathcal{M}}\frac{1}{\sqrt{L}}\lVert\widetilde{\mathbf{y}}_i-\widetilde{\mathbf{y}}_{\widetilde{m}}\rVert_2,\qquad|\mathcal{M}|=m. F ( M ) := n 1 i = 1 ∑ n m ∈ M min L 1 ∥ y i − y m ∥ 2 , ∣ M ∣ = m .
Let M ^ m \widehat{\mathcal{M}}_m M m be any one-swap local minimizer of the m m m -medoids objective used in the denoise step in § 2.4 . That is,
F ( M ^ m ) ≤ F ( ( M ^ m ∖ { m ^ } ) ∪ { j } ) , ∀ m ^ ∈ M ^ m , ∀ j ∉ M ^ m (70) F(\widehat{\mathcal{M}}_m)\le F((\widehat{\mathcal{M}}_m\setminus\{\widehat{m}\})\cup\{j\}),\qquad\forall\widehat{m}\in\widehat{\mathcal{M}}_m,\ \forall j\notin\widehat{\mathcal{M}}_m \tag*{(70)} F ( M m ) ≤ F (( M m ∖ { m }) ∪ { j }) , ∀ m ∈ M m , ∀ j ∈ / M m (70)
with ∣ M ^ m ∣ = m |\widehat{\mathcal{M}}_m|=m ∣ M m ∣ = m . Let r : = r n , L ( x ) r:=r_{n,L}(x) r := r n , L ( x ) . Then, to complete the proof, it suffices to show
M ^ m is r -good with r < δ m i x 2 (71) \widehat{\mathcal{M}}_m\text{ is }r\text{-good with }r<\frac{\delta_{\mathrm{mix}}}{\sqrt{2}} \tag*{(71)} M m is r -good with r < 2 δ mix (71)
since Corollary 3.5 can then be applied to m m m medoids indexed by M ^ m \widehat{\mathcal{M}}_m M m .
Now we prove (E.10) by contradiction. Suppose that M ^ B = { m ^ 1 , … , m ^ B } \widehat{\mathcal{M}}_B=\{\widehat{m}_1,\ldots,\widehat{m}_B\} M B = { m 1 , … , m B } is not r r r -good for some r < δ m i x 2 r<\frac{\delta_{\mathrm{mix}}}{\sqrt{2}} r < 2 δ mix . Then there exists k ∈ [ K ] k\in[K] k ∈ [ K ] such that
M ^ B ∩ I k ( r ) = ∅ . \widehat{\mathcal{M}}_B\cap\mathcal{I}_k(r)=\varnothing. M B ∩ I k ( r ) = ∅ .
For each i ∈ [ n ] i \in[n] i ∈ [ n ] , define
a ( i ) : = min { 1 ≤ b ≤ m : ∥ y ~ i − y ~ m b ∥ 2 = min 1 ≤ t ≤ m ∥ y ~ i − y ~ m t ∥ 2 } . a(i) := \min\left\{1 \le b \le m : \|\widetilde{y}_i-\widetilde{y}_{m_b}\|_2 = \min_{1\le t\le m}\|\widetilde{y}_i-\widetilde{y}_{m_t}\|_2\right\}. a ( i ) := min { 1 ≤ b ≤ m : ∥ y i − y m b ∥ 2 = 1 ≤ t ≤ m min ∥ y i − y m t ∥ 2 } .
that is, if the minimum distance is attained by multiple medoids, we assign i i i to the medoid with the smallest index. For each b ∈ [ m ] b \in[m] b ∈ [ m ] , define the corresponding Voronoi cell
C b : = { i ∈ [ n ] : a ( i ) = b } . C_b := \{i \in[n] : a(i)=b\}. C b := { i ∈ [ n ] : a ( i ) = b } .
Then { C b } b = 1 m \{C_b\}_{b=1}^{m} { C b } b = 1 m forms a partition of [ n ] [n] [ n ] , and hence
∑ b = 1 m ∣ C b ∣ = n . \sum_{b=1}^{m}|C_b|=n. b = 1 ∑ m ∣ C b ∣ = n .
Therefore, by an averaging argument, there exists b ∈ [ m ] b \in[m] b ∈ [ m ] such that
∣ C b ∣ ≤ n m . |C_b| \le\frac{n}{m}. ∣ C b ∣ ≤ m n .
Define another index set
M m ⋆ : = ( M ^ m ∖ { m ^ b } ) ∪ { i k } , \mathcal{M}_m^\star:= \left(\widehat{\mathcal{M}}_m \setminus\{\widehat{m}_b\}\right) \cup\{i_k\}, M m ⋆ := ( M m ∖ { m b } ) ∪ { i k } ,
where i k i_k i k is an index of a pure node in community k k k . Now we show F ( M B ⋆ ) < F ( M ^ B ) F(\mathcal{M}_B^\star) < F(\widehat{\mathcal{M}}_B) F ( M B ⋆ ) < F ( M B ) by showing
1 n ∑ i = 1 n 1 L ( min m ⋆ ∈ M m ⋆ ∥ y ~ i − y ~ m ⋆ ∥ 2 − min m ∈ M ^ m ∥ y ~ i − y ~ m ∥ 2 ) < 0. \frac{1}{n}\sum_{i=1}^{n}\frac{1}{\sqrt{L}}\left(\min_{m^\star\in\mathcal{M}_m^\star}\|\widetilde{y}_i-\widetilde{y}_{m^\star}\|_2-\min_{m\in\widehat{\mathcal{M}}_m}\|\widetilde{y}_i-\widetilde{y}_m\|_2\right)<0. n 1 i = 1 ∑ n L 1 ( m ⋆ ∈ M m ⋆ min ∥ y i − y m ⋆ ∥ 2 − m ∈ M m min ∥ y i − y m ∥ 2 ) < 0.
We partition the node set [ n ] [n] [ n ] into the following three disjoint groups:
P k \mathcal{P}_k P k ,
C b ∖ P k C_b \setminus\mathcal{P}_k C b ∖ P k ,
[ n ] ∖ ( P k ∪ C b ) [n] \setminus(\mathcal{P}_k \cup C_b) [ n ] ∖ ( P k ∪ C b ) .
We bound the contribution of each group separately. Let h ^ : = 1 L max i ∥ y i − y ~ i ∥ 2 \hat{h}:=\frac{1}{\sqrt{L}}\max_i\|y_i-\widetilde{y}_i\|_2 h ^ := L 1 max i ∥ y i − y i ∥ 2 . First, consider the first group. Note that i k ∈ M m ⋆ ∩ P k i_k \in\mathcal{M}_m^\star\cap\mathcal{P}_k i k ∈ M m ⋆ ∩ P k . For i ∈ P k i \in\mathcal{P}_k i ∈ P k ,
min m ⋆ ∈ M m ⋆ 1 L ∥ y ~ i − y ~ m ⋆ ∥ 2 ≤ min m ⋆ ∈ M m ⋆ 1 L ( ∥ y ~ i − y i ∥ 2 + ∥ y i − y m ⋆ ∥ 2 + ∥ y m ⋆ − y ~ m ⋆ ∥ 2 ) ≤ h ^ + min m ⋆ ∈ M m ⋆ 1 L ∥ y i − y m ⋆ ∥ 2 + h ^ = 2 h ^ . \begin{aligned}
\min_{m^\star\in\mathcal{M}_m^\star}\frac{1}{\sqrt{L}}\|\widetilde{y}_i-\widetilde{y}_{m^\star}\|_2
\le\min_{m^\star\in\mathcal{M}_m^\star}\frac{1}{\sqrt{L}}\left(\|\widetilde{y}_i-y_i\|_2+\|y_i-y_{m^\star}\|_2+\|y_{m^\star}-\widetilde{y}_{m^\star}\|_2\right) \\
&\le\hat{h}+\min_{m^\star\in\mathcal{M}_m^\star}\frac{1}{\sqrt{L}}\|y_i-y_{m^\star}\|_2+\hat{h} \\
&=2\hat{h}.
\end{aligned} m ⋆ ∈ M m ⋆ min L 1 ∥ y i − y m ⋆ ∥ 2 ≤ m ⋆ ∈ M m ⋆ min L 1 ( ∥ y i − y i ∥ 2 + ∥ y i − y m ⋆ ∥ 2 + ∥ y m ⋆ − y m ⋆ ∥ 2 ) ≤ h ^ + m ⋆ ∈ M m ⋆ min L 1 ∥ y i − y m ⋆ ∥ 2 + h ^ = 2 h ^ .
On the other hand, for i ∈ P k i \in\mathcal{P}_k i ∈ P k ,
min m ∈ M ^ m 1 L ∥ y ~ i − y ~ m ∥ 2 = min m ∈ M ^ m 1 L ( ∥ y ~ i − y i + y i − y m + y m − y ~ m ∥ 2 ) ≥ min m ∈ M ^ m 1 L ( ∥ y i − y m ∥ 2 − ∥ y ~ i − y i + y m − y ~ m ∥ 2 ) ≥ min m ∈ M ^ m 1 L ( ∥ y i − y m ∥ 2 − ( ∥ y ~ i − y i ∥ 2 + ∥ y m − y ~ m ∥ 2 ) ) ≥ min m ∈ M ^ m 1 L ∥ y i − y m ∥ 2 − 2 h ^ ≥ Δ 0 − 2 h ^ . \begin{aligned}
\min_{m\in\widehat{\mathcal{M}}_m}\frac{1}{\sqrt{L}}\|\widetilde{y}_i-\widetilde{y}_m\|_2
=\min_{m\in\widehat{\mathcal{M}}_m}\frac{1}{\sqrt{L}}\left(\|\widetilde{y}_i-y_i+y_i-y_m+y_m-\widetilde{y}_m\|_2\right) \\
&\ge\min_{m\in\widehat{\mathcal{M}}_m}\frac{1}{\sqrt{L}}\left(\|y_i-y_m\|_2-\|\widetilde{y}_i-y_i+y_m-\widetilde{y}_m\|_2\right) \\
&\ge\min_{m\in\widehat{\mathcal{M}}_m}\frac{1}{\sqrt{L}}\left(\|y_i-y_m\|_2-\left(\|\widetilde{y}_i-y_i\|_2+\|y_m-\widetilde{y}_m\|_2\right)\right) \\
&\ge\min_{m\in\widehat{\mathcal{M}}_m}\frac{1}{\sqrt{L}}\|y_i-y_m\|_2-2\hat{h} \\
&\ge\Delta_0-2\hat{h}.
\end{aligned} m ∈ M m min L 1 ∥ y i − y m ∥ 2 = m ∈ M m min L 1 ( ∥ y i − y i + y i − y m + y m − y m ∥ 2 ) ≥ m ∈ M m min L 1 ( ∥ y i − y m ∥ 2 − ∥ y i − y i + y m − y m ∥ 2 ) ≥ m ∈ M m min L 1 ( ∥ y i − y m ∥ 2 − ( ∥ y i − y i ∥ 2 + ∥ y m − y m ∥ 2 ) ) ≥ m ∈ M m min L 1 ∥ y i − y m ∥ 2 − 2 h ^ ≥ Δ 0 − 2 h ^ .
The last inequality follows from the definition of Δ 0 \Delta_0 Δ 0 . Indeed, since i ∈ P k i \in\mathcal{P}_k i ∈ P k , we have
y i = T ( e k ) . y_i = T(e_k). y i = T ( e k ) .
Moreover, M ^ m ∩ I k ( r ) = ∅ \widehat{\mathcal{M}}_m \cap\mathcal{I}_k(r) = \varnothing M m ∩ I k ( r ) = ∅ , and r < δ m i x / 2 r < \delta_{\mathrm{mix}}/\sqrt{2} r < δ mix / 2 implies I k ( r ) = P k \mathcal{I}_k(r) = \mathcal{P}_k I k ( r ) = P k by Corollary 3.5 . Hence, for every j ∈ [ m ] j \in[m] j ∈ [ m ] ,
m ^ j ∉ P k . \widehat{m}_j \notin\mathcal{P}_k. m j ∈ / P k .
Therefore, for each j ∈ [ m ] j \in[m] j ∈ [ m ] , either Z m ^ j , ⋅ Z_{\widehat{m}_j,\cdot} Z m j , ⋅ is a mixed node or Z m ^ j , ⋅ = e k ′ Z_{\widehat{m}_j,\cdot} = e_{k'} Z m j , ⋅ = e k ′ for some k ′ ≠ k k' \ne k k ′ = k . By definition of Δ 0 \Delta_0 Δ 0 , it follows that
1 L ∥ y i − y ~ m ^ j ∥ 2 = 1 L ∥ T ( e k ) − T ( Z m ^ j , ⋅ ) ∥ 2 ≥ Δ 0 . \frac{1}{\sqrt{L}}\lVert y_i-\widetilde{y}_{\widehat{m}_j}\rVert_2
=
\frac{1}{\sqrt{L}}\lVert T(e_k)-T(Z_{\widehat{m}_j,\cdot})\rVert_2
\ge\Delta_0. L 1 ∥ y i − y m j ∥ 2 = L 1 ∥ T ( e k ) − T ( Z m j , ⋅ ) ∥ 2 ≥ Δ 0 .
Consequently,
min j ∈ [ B ] 1 L ∥ y i − y ~ m ^ j ∥ 2 ≥ Δ 0 . \min_{j\in[B]}\frac{1}{\sqrt{L}}\lVert y_i-\widetilde{y}_{\widehat{m}_j}\rVert_2
\ge\Delta_0. j ∈ [ B ] min L 1 ∥ y i − y m j ∥ 2 ≥ Δ 0 .
Therefore, for the first group, we have
min m ⋆ ∈ M m ⋆ 1 L ∥ y ~ i − y ~ m ⋆ ∥ 2 − min m ∈ M ^ m 1 L ∥ y ~ i − y ~ m ∥ 2 ≤ 4 h ^ − Δ 0 . (72) \min_{m^\star\in\mathcal{M}_m^\star}\frac{1}{\sqrt{L}}\lVert\widetilde{y}_i-\widetilde{y}_{m^\star}\rVert_2
-
\min_{m\in\widehat{\mathcal{M}}_m}\frac{1}{\sqrt{L}}\lVert\widetilde{y}_i-\widetilde{y}_m\rVert_2
\le4\widehat{h}-\Delta_0. \tag*{(72)} m ⋆ ∈ M m ⋆ min L 1 ∥ y i − y m ⋆ ∥ 2 − m ∈ M m min L 1 ∥ y i − y m ∥ 2 ≤ 4 h − Δ 0 . (72)
Next, consider the second group. For each i ∈ C b ∖ P k i \in\mathcal{C}_b \setminus\mathcal{P}_k i ∈ C b ∖ P k , let
m i ⋆ : = arg min m ⋆ ∈ M m ⋆ ∥ y ~ i − y ~ m ⋆ ∥ 2 and m i : = arg min m ∈ M ^ m ∥ y ~ i − y ~ m ∥ 2 . m_i^\star:= \operatorname*{arg\,min}_{m^\star\in\mathcal{M}_m^\star}\lVert\widetilde{y}_i-\widetilde{y}_{m^\star}\rVert_2
\quad\text{and}\quad
m_i := \operatorname*{arg\,min}_{m\in\widehat{\mathcal{M}}_m}\lVert\widetilde{y}_i-\widetilde{y}_m\rVert_2. m i ⋆ := m ⋆ ∈ M m ⋆ arg min ∥ y i − y m ⋆ ∥ 2 and m i := m ∈ M m arg min ∥ y i − y m ∥ 2 .
Then
min m ⋆ ∈ M m ⋆ 1 L ∥ y ~ i − y ~ m ⋆ ∥ 2 − min m ∈ M ^ m 1 L ∥ y ~ i − y ~ m ∥ 2 = 1 L ∥ y ~ i − y ~ m i ⋆ ∥ 2 − 1 L ∥ y ~ i − y ~ m i ∥ 2 ≤ 1 L ∥ y ~ m i ⋆ − y ~ m i ∥ 2 ≤ 1 L ∥ y ~ m i ⋆ ∥ 2 + 1 L ∥ y ~ m i ∥ 2 = 2. (73) \begin{aligned}
&\min_{m^\star\in\mathcal{M}_m^\star}\frac{1}{\sqrt{L}}\lVert\widetilde{y}_i-\widetilde{y}_{m^\star}\rVert_2
-
\min_{m\in\widehat{\mathcal{M}}_m}\frac{1}{\sqrt{L}}\lVert\widetilde{y}_i-\widetilde{y}_m\rVert_2 \\
&\quad=
\frac{1}{\sqrt{L}}\lVert\widetilde{y}_i-\widetilde{y}_{m_i^\star}\rVert_2
-
\frac{1}{\sqrt{L}}\lVert\widetilde{y}_i-\widetilde{y}_{m_i}\rVert_2 \\
&\quad\le\frac{1}{\sqrt{L}}\lVert\widetilde{y}_{m_i^\star}-\widetilde{y}_{m_i}\rVert_2 \\
&\quad\le\frac{1}{\sqrt{L}}\lVert\widetilde{y}_{m_i^\star}\rVert_2
+
\frac{1}{\sqrt{L}}\lVert\widetilde{y}_{m_i}\rVert_2 \\
&\quad= 2. \tag*{(73)}
\end{aligned} m ⋆ ∈ M m ⋆ min L 1 ∥ y i − y m ⋆ ∥ 2 − m ∈ M m min L 1 ∥ y i − y m ∥ 2 = L 1 ∥ y i − y m i ⋆ ∥ 2 − L 1 ∥ y i − y m i ∥ 2 ≤ L 1 ∥ y m i ⋆ − y m i ∥ 2 ≤ L 1 ∥ y m i ⋆ ∥ 2 + L 1 ∥ y m i ∥ 2 = 2. (73)
Lastly, consider the third group i ∉ P k ∪ C b i \notin\mathcal{P}_k \cup\mathcal{C}_b i ∈ / P k ∪ C b . By the definition of C b \mathcal{C}_b C b , the removed medoid m ^ b \widehat{m}_b m b is not the assigned nearest medoid of i i i . Hence there exists some m i ∈ M ^ m ∖ { m ^ b } m_i \in\widehat{\mathcal{M}}_m \setminus\{\widehat{m}_b\} m i ∈ M m ∖ { m b } such that
∥ y ~ i − y ~ m i ∥ 2 = min m ∈ M ^ m ∥ y ~ i − y ~ m ∥ 2 . \lVert\widetilde{y}_i-\widetilde{y}_{m_i}\rVert_2
=
\min_{m\in\widehat{\mathcal{M}}_m}\lVert\widetilde{y}_i-\widetilde{y}_m\rVert_2. ∥ y i − y m i ∥ 2 = m ∈ M m min ∥ y i − y m ∥ 2 .
Since
m i ∈ M ^ m ∖ { m ^ b } ⊂ M m ⋆ , m_i \in\widehat{\mathcal{M}}_m \setminus\{\widehat{m}_b\} \subset\mathcal{M}_m^\star, m i ∈ M m ∖ { m b } ⊂ M m ⋆ ,
where M m ⋆ : = ( M ^ m ∖ { m ^ b } ) ∪ { i k } \mathcal{M}_m^\star:= (\widehat{\mathcal{M}}_m \setminus\{\widehat{m}_b\}) \cup\{i_k\} M m ⋆ := ( M m ∖ { m b }) ∪ { i k } , we obtain
min m ⋆ ∈ M m ⋆ ∥ y ~ i − y ~ m ⋆ ∥ 2 ≤ ∥ y ~ i − y ~ m i ∥ 2 = min m ∈ M ^ m ∥ y ~ i − y ~ m ∥ 2 . (74) \min_{m^\star\in\mathcal{M}_m^\star}\lVert\widetilde{y}_i-\widetilde{y}_{m^\star}\rVert_2
\le
\lVert\widetilde{y}_i-\widetilde{y}_{m_i}\rVert_2
=
\min_{m\in\widehat{\mathcal{M}}_m}\lVert\widetilde{y}_i-\widetilde{y}_m\rVert_2. \tag*{(74)} m ⋆ ∈ M m ⋆ min ∥ y i − y m ⋆ ∥ 2 ≤ ∥ y i − y m i ∥ 2 = m ∈ M m min ∥ y i − y m ∥ 2 . (74)
Therefore, replacing m ^ b \widehat{m}_b m b by i k i_k i k cannot increase the contribution of any node in the third group. Since M ^ m ∩ P k = ∅ \widehat{\mathcal{M}}_m \cap\mathcal{P}_k = \varnothing M m ∩ P k = ∅ and i k ∈ P k i_k \in\mathcal{P}_k i k ∈ P k , we have i k ∉ M ^ m i_k \notin\widehat{\mathcal{M}}_m i k ∈ / M m . Thus,
M m ⋆ = ( M ^ m ∖ { m ^ b } ) ∪ { i k } \mathcal{M}_m^\star= (\widehat{\mathcal{M}}_m \setminus\{\widehat{m}_b\}) \cup\{i_k\} M m ⋆ = ( M m ∖ { m b }) ∪ { i k }
is a feasible one-swap neighbor of M ^ m \widehat{\mathcal{M}}_m M m .
By (E.9), (E.11), (E.12), and (E.13), we have
0 ≤ F ( M m ⋆ ) − F ( M ^ m ) ≤ N k n ( 4 h ^ − Δ 0 ) + 2 ∣ C b ∖ P k ∣ n ≤ N k n ( 4 h ^ − Δ 0 ) + 2 m . \begin{aligned}
0 \leq F(M^{\star}_m) - F(\widehat{M}_m) \\
&\leq\frac{N_k}{n}(4\widehat{h}-\Delta_0) + \frac{2|C_b \setminus P_k|}{n} \\
&\leq\frac{N_k}{n}(4\widehat{h}-\Delta_0) + \frac{2}{m}.
\end{aligned} 0 ≤ F ( M m ⋆ ) − F ( M m ) ≤ n N k ( 4 h − Δ 0 ) + n 2∣ C b ∖ P k ∣ ≤ n N k ( 4 h − Δ 0 ) + m 2 .
where the first inequality follows from one-swap local optimality and the last inequality uses ∣ C b ∣ ≤ n / m |C_b| \leq n/m ∣ C b ∣ ≤ n / m . By condition 2(a), we derive a contradiction of one-swap local optimality of M ^ m \widehat{M}_m M m if
4 h ^ + 2 m p p u r e < Δ 0 . (75) 4\widehat{h} + \frac{2}{m p_{\mathrm{pure}}} < \Delta_0. \tag*{(75)} 4 h + m p pure 2 < Δ 0 . (75)
To this end, we derive a high probability upper bound of h ^ \widehat{h} h . Since h ^ 2 = 1 L max i ∥ y i − y ~ i ∥ 2 2 ≤ 1 L ∑ l = 1 L ( h ^ ( l ) ) 2 \widehat{h}^{2} = \frac{1}{L}\max_i \|y_i-\widetilde{y}_i\|_2^2 \leq\frac{1}{L}\sum_{l=1}^{L}(\widehat{h}^{(l)})^2 h 2 = L 1 max i ∥ y i − y i ∥ 2 2 ≤ L 1 ∑ l = 1 L ( h ( l ) ) 2 , we obtain a high probability upper bound of 1 L ∑ l = 1 L ( h ^ ( l ) ) 2 \frac{1}{L}\sum_{l=1}^{L}(\widehat{h}^{(l)})^2 L 1 ∑ l = 1 L ( h ( l ) ) 2 . Using a similar decomposition of h ^ ( l ) \widehat{h}^{(l)} h ( l ) in the proof of Theorem 3.3, we have
( h ^ ( l ) ) 2 = ( h ^ ( l ) ) 2 1 E ( l ) + ( h ^ ( l ) ) 2 1 ( E ( l ) ) c ≤ c Y 2 β n ( l ) + 41 ( E ( l ) ) c . (\widehat{h}^{(l)})^2 = (\widehat{h}^{(l)})^2 1_{\mathcal{E}^{(l)}} + (\widehat{h}^{(l)})^2 1_{(\mathcal{E}^{(l)})^c} \leq c_Y^2\beta_n^{(l)} + 4 1_{(\mathcal{E}^{(l)})^c}. ( h ( l ) ) 2 = ( h ( l ) ) 2 1 E ( l ) + ( h ( l ) ) 2 1 ( E ( l ) ) c ≤ c Y 2 β n ( l ) + 4 1 ( E ( l ) ) c .
Using the same upper bound defined in (E.7), it follows that
1 L ∑ l = 1 L ( h ^ ( l ) ) 2 ≤ c Y 2 L ∑ l = 1 L β n ( l ) + 4 ( c E n − 18 + 2 c E x L n 18 + 2 x 3 L ) ≤ c Y 2 L ∑ l = 1 L β n ( l ) + 4 ( c E n − 18 + c E 2 n 18 + x L + 2 x 3 L ) = c Y 2 L ∑ l = 1 L β n ( l ) + 6 c E n − 18 + 20 x 3 L , \begin{aligned}
\frac{1}{L}\sum_{l=1}^{L}(\widehat{h}^{(l)})^2 \leq\frac{c_Y^2}{L}\sum_{l=1}^{L}\beta_n^{(l)} + 4\left(c_E n^{-18}+\sqrt{\frac{2c_E x}{L n^{18}}}+\frac{2x}{3L}\right) \\
&\leq\frac{c_Y^2}{L}\sum_{l=1}^{L}\beta_n^{(l)} + 4\left(c_E n^{-18}+\frac{c_E}{2n^{18}}+\frac{x}{L}+\frac{2x}{3L}\right) \\
&= \frac{c_Y^2}{L}\sum_{l=1}^{L}\beta_n^{(l)} + 6c_E n^{-18}+\frac{20x}{3L},
\end{aligned} L 1 l = 1 ∑ L ( h ( l ) ) 2 ≤ L c Y 2 l = 1 ∑ L β n ( l ) + 4 ( c E n − 18 + L n 18 2 c E x + 3 L 2 x ) ≤ L c Y 2 l = 1 ∑ L β n ( l ) + 4 ( c E n − 18 + 2 n 18 c E + L x + 3 L 2 x ) = L c Y 2 l = 1 ∑ L β n ( l ) + 6 c E n − 18 + 3 L 20 x ,
with probability at least 1 − exp ( − x ) 1-\exp(-x) 1 − exp ( − x ) . Hence, on the same event, we have
h ^ ≤ 1 L ∑ l = 1 L ( h ^ ( l ) ) 2 ≤ ( c Y 2 L ∑ l = 1 L β n ( l ) + 6 c E n − 18 + 20 x 3 L ) 1 / 2 ≤ C ( 1 L ∑ l = 1 L β n ( l ) + n − 18 + x L ) 1 / 2 \widehat{h} \leq\sqrt{\frac{1}{L}\sum_{l=1}^{L}(\widehat{h}^{(l)})^2} \leq\left(\frac{c_Y^2}{L}\sum_{l=1}^{L}\beta_n^{(l)}+6c_E n^{-18}+\frac{20x}{3L}\right)^{1/2} \leq C\left(\frac{1}{L}\sum_{l=1}^{L}\beta_n^{(l)}+n^{-18}+\frac{x}{L}\right)^{1/2} h ≤ L 1 l = 1 ∑ L ( h ( l ) ) 2 ≤ ( L c Y 2 l = 1 ∑ L β n ( l ) + 6 c E n − 18 + 3 L 20 x ) 1/2 ≤ C ( L 1 l = 1 ∑ L β n ( l ) + n − 18 + L x ) 1/2
for a sufficiently large constant C C C . Therefore, if
4 C ( 1 L ∑ l = 1 L β n ( l ) + n − 18 + x L ) 1 / 2 + 2 m p p u r e < Δ 0 , 4C\left(\frac{1}{L}\sum_{l=1}^{L}\beta_n^{(l)}+n^{-18}+\frac{x}{L}\right)^{1/2}+\frac{2}{m p_{\mathrm{pure}}}<\Delta_0, 4 C ( L 1 l = 1 ∑ L β n ( l ) + n − 18 + L x ) 1/2 + m p pure 2 < Δ 0 ,
then (E.14) holds with probability at least 1 − exp ( − x ) 1-\exp(-x) 1 − exp ( − x ) and this completes the proof.
Proof of Corollary 3.6
We restate Corollary 3.6 for convenience.
Corollary 3.6 (Rate of the spectral membership estimator). Suppose the conditions of Theorem 3.3 hold. Let I ⊆ [ n ] I \subseteq[n] I ⊆ [ n ] be any corner candidate set containing at least one pure node from each community. Let Z ^ 0 \widehat{Z}_0 Z 0 be the membership estimator obtained from Algorithm 1. Then there exist constants C > 0 C>0 C > 0 ,
c x > 0 c_x>0 c x > 0 and n 0 ≥ 1 n_0\ge1 n 0 ≥ 1 such that, for every n ≥ n 0 n\ge n_0 n ≥ n 0 and 0 < x ≤ c x L 0<x\le c_xL 0 < x ≤ c x L , with probability at least 1 − e − x − c E n − 18 1-e^{-x}-c_E n^{-18} 1 − e − x − c E n − 18 , there exists a permutation matrix Π ∈ Q K \Pi\in\mathcal{Q}_K Π ∈ Q K satisfying
1 n ∥ Z ^ 0 Π − Z ∥ F ≤ ∥ Z ^ 0 Π − Z ∥ 2 , ∞ ≤ C ( β n ( 1 ) + r n , L ( x ) ) . \frac{1}{\sqrt{n}}\left\|\widehat{\mathbf{Z}}_0\Pi-\mathbf{Z}\right\|_F\le\left\|\widehat{\mathbf{Z}}_0\Pi-\mathbf{Z}\right\|_{2,\infty}\le C\left(\sqrt{\beta_n^{(1)}}+r_{n,L}(x)\right). n 1 Z 0 Π − Z F ≤ Z 0 Π − Z 2 , ∞ ≤ C ( β n ( 1 ) + r n , L ( x ) ) .
Moreover, if Condition 2 ( b ) 2(b) 2 ( b ) holds with r n , L ( x ) < δ m i x / 2 r_{n,L}(x)<\delta_{\mathrm{mix}}/\sqrt{2} r n , L ( x ) < δ mix / 2 , then on the same event
1 n ∥ Z ^ 0 Π − Z ∥ F ≤ ∥ Z ^ 0 Π − Z ∥ 2 , ∞ ≤ C β n ( 1 ) . \frac{1}{\sqrt{n}}\left\|\widehat{\mathbf{Z}}_0\Pi-\mathbf{Z}\right\|_F\le\left\|\widehat{\mathbf{Z}}_0\Pi-\mathbf{Z}\right\|_{2,\infty}\le C\sqrt{\beta_n^{(1)}}. n 1 Z 0 Π − Z F ≤ Z 0 Π − Z 2 , ∞ ≤ C β n ( 1 ) .
Proof. By relabeling the estimated communities if necessary, we assume throughout the proof that the estimated corner ordering agrees with the population community ordering. Thus, we suppress the permutation matrix Π \Pi Π in what follows.
Let E V H \mathcal{E}_{\mathrm{VH}} E VH denote the near-pure recovery event from Theorem 3.3, and let E X ( 1 ) \mathcal{E}_{X}^{(1)} E X ( 1 ) denote the rowwise spectral embedding event for the reference layer. By Theorem 3.3 and the rowwise embedding bound,
P ( E V H ) ≥ 1 − exp ( − x ) , P ( E X ( 1 ) ) ≥ 1 − c E n − 18 . \mathbb{P}(\mathcal{E}_{\mathrm{VH}})\ge1-\exp(-x),\qquad\mathbb{P}(\mathcal{E}_{X}^{(1)})\ge1-c_E n^{-18}. P ( E VH ) ≥ 1 − exp ( − x ) , P ( E X ( 1 ) ) ≥ 1 − c E n − 18 .
Hence,
P ( E V H ∩ E X ( 1 ) ) ≥ 1 − exp ( − x ) − c E n − 18 . \mathbb{P}\left(\mathcal{E}_{\mathrm{VH}}\cap\mathcal{E}_{X}^{(1)}\right)\ge1-\exp(-x)-c_E n^{-18}. P ( E VH ∩ E X ( 1 ) ) ≥ 1 − exp ( − x ) − c E n − 18 .
We work on this intersection event throughout the proof. Let J ^ \widehat{\mathcal{J}} J denote the estimated corner index set by the corner estimation method. Then the following statements hold:
min 1 ≤ k ≤ K max j ∈ J ^ Z j k ≥ 1 − r n , L ( x ) , \min_{1\le k\le K}\max_{j\in\widehat{\mathcal{J}}}Z_{jk}\ge1-r_{n,L}(x), 1 ≤ k ≤ K min j ∈ J max Z j k ≥ 1 − r n , L ( x ) ,
and, for all i ∈ [ n ] i\in[n] i ∈ [ n ] ,
∥ e i T ( X ^ ( 1 ) W ∗ ( 1 ) − X ( 1 ) ) ∥ 2 ≤ C X θ i ( 1 ) σ min ( 1 ) K β n ( 1 ) . \left\|\mathbf{e}_i^{\mathsf T}\left(\widehat{\mathbf{X}}^{(1)}\mathbf{W}_*^{(1)}-\mathbf{X}^{(1)}\right)\right\|_2
\le C_X\theta_i^{(1)}\frac{\sigma_{\min}^{(1)}}{K}\sqrt{\beta_n^{(1)}}. e i T ( X ( 1 ) W ∗ ( 1 ) − X ( 1 ) ) 2 ≤ C X θ i ( 1 ) K σ m i n ( 1 ) β n ( 1 ) .
Let
X ~ ( 1 ) = X ^ ( 1 ) W ∗ ( 1 ) , S ~ ( 1 ) = S ^ ( 1 ) W ∗ ( 1 ) , y ~ i ( 1 ) = y ^ i ( 1 ) W ∗ ( 1 ) . \widetilde{\mathbf{X}}^{(1)}=\widehat{\mathbf{X}}^{(1)}\mathbf{W}_*^{(1)},\qquad
\widetilde{\mathbf{S}}^{(1)}=\widehat{\mathbf{S}}^{(1)}\mathbf{W}_*^{(1)},\qquad
\widetilde{\mathbf{y}}_i^{(1)}=\widehat{\mathbf{y}}_i^{(1)}\mathbf{W}_*^{(1)}. X ( 1 ) = X ( 1 ) W ∗ ( 1 ) , S ( 1 ) = S ( 1 ) W ∗ ( 1 ) , y i ( 1 ) = y i ( 1 ) W ∗ ( 1 ) .
Since
X ^ ( 1 ) ( S ^ ( 1 ) ) − 1 = X ~ ( 1 ) ( S ~ ( 1 ) ) − 1 , \widehat{\mathbf{X}}^{(1)}\left(\widehat{\mathbf{S}}^{(1)}\right)^{-1}
=\widetilde{\mathbf{X}}^{(1)}\left(\widetilde{\mathbf{S}}^{(1)}\right)^{-1}, X ( 1 ) ( S ( 1 ) ) − 1 = X ( 1 ) ( S ( 1 ) ) − 1 ,
we analyze
U ^ ( 1 ) − U ( 1 ) = X ~ ( 1 ) ( S ~ ( 1 ) ) − 1 − X ( 1 ) ( S ( 1 ) ) − 1 . \widehat{\mathbf{U}}^{(1)}-\mathbf{U}^{(1)}
=\widetilde{\mathbf{X}}^{(1)}\left(\widetilde{\mathbf{S}}^{(1)}\right)^{-1}
-\mathbf{X}^{(1)}\left(\mathbf{S}^{(1)}\right)^{-1}. U ( 1 ) − U ( 1 ) = X ( 1 ) ( S ( 1 ) ) − 1 − X ( 1 ) ( S ( 1 ) ) − 1 .
First, we control the normalized embedding error on the reference layer. By the elementary normalization inequality,
∥ y ~ i ( 1 ) − y i ( 1 ) ∥ 2 ≤ 2 ∥ X ~ i , ⋅ ( 1 ) − X i , ⋅ ( 1 ) ∥ 2 ∥ X i , ⋅ ( 1 ) ∥ 2 . \left\|\widetilde{\mathbf{y}}_i^{(1)}-\mathbf{y}_i^{(1)}\right\|_2
\le
\frac{2\left\|\widetilde{\mathbf{X}}_{i,\cdot}^{(1)}-\mathbf{X}_{i,\cdot}^{(1)}\right\|_2}
{\left\|\mathbf{X}_{i,\cdot}^{(1)}\right\|_2}. y i ( 1 ) − y i ( 1 ) 2 ≤ X i , ⋅ ( 1 ) 2 2 X i , ⋅ ( 1 ) − X i , ⋅ ( 1 ) 2 .
By Lemma B.1 , we have
∥ X i , ⋅ ( 1 ) ∥ 2 ≳ θ i ( 1 ) σ min ( 1 ) K . \left\|\mathbf{X}_{i,\cdot}^{(1)}\right\|_2\gtrsim\theta_i^{(1)}\frac{\sigma_{\min}^{(1)}}{K}. X i , ⋅ ( 1 ) 2 ≳ θ i ( 1 ) K σ m i n ( 1 ) .
Therefore,
max 1 ≤ i ≤ n ∥ y ~ i ( 1 ) − y i ( 1 ) ∥ 2 ≤ C β n ( 1 ) . \max_{1\le i\le n}\left\|\widetilde{\mathbf{y}}_i^{(1)}-\mathbf{y}_i^{(1)}\right\|_2\le C\sqrt{\beta_n^{(1)}}. 1 ≤ i ≤ n max y i ( 1 ) − y i ( 1 ) 2 ≤ C β n ( 1 ) .
Next, we control the corner matrix error. By the almost-pure recovery event, for each community k k k , there exists i ^ k ∈ J ^ \widehat{i}_k\in\widehat{\mathcal{J}} i k ∈ J such that
Z i ^ k , k ≥ 1 − r n , L ( x ) . Z_{\widehat{i}_k,k}\ge1-r_{n,L}(x). Z i k , k ≥ 1 − r n , L ( x ) .
Since Z ^ i k , ⋅ ∈ Δ K − 1 \widehat{Z}_{i_k,\cdot} \in\Delta^{K-1} Z i k , ⋅ ∈ Δ K − 1 ,
∥ Z ^ i k , ⋅ − e k ∥ 2 ≤ 2 r n , L ( x ) . \lVert\widehat{Z}_{i_k,\cdot} - e_k \rVert_2 \le\sqrt{2}\,r_{n,L}(x). ∥ Z i k , ⋅ − e k ∥ 2 ≤ 2 r n , L ( x ) .
Let
Δ S : = S ~ ( 1 ) − S ( 1 ) . \Delta_S := \widetilde{S}^{(1)} - S^{(1)}. Δ S := S ( 1 ) − S ( 1 ) .
Then
∥ Δ S ∥ 2 ≤ ∥ Δ S ∥ F ≤ K ∥ Δ S ∥ 2 , ∞ . \lVert\Delta_S \rVert_2 \le\lVert\Delta_S \rVert_F \le\sqrt{K}\lVert\Delta_S \rVert_{2,\infty}. ∥ Δ S ∥ 2 ≤ ∥ Δ S ∥ F ≤ K ∥ Δ S ∥ 2 , ∞ .
For each k ∈ [ K ] k \in[K] k ∈ [ K ] , let i k ∈ P k i_k \in\mathcal{P}_k i k ∈ P k denote a pure node representing the k k k -th population corner, so that S k , ⋅ ( 1 ) = y i k ( 1 ) S^{(1)}_{k,\cdot} = y^{(1)}_{i_k} S k , ⋅ ( 1 ) = y i k ( 1 ) . Then
∥ S ~ k , ⋅ ( 1 ) − S k , ⋅ ( 1 ) ∥ 2 = ∥ y ~ i ^ k ( 1 ) − y i k ( 1 ) ∥ 2 ≤ ∥ y ~ i ^ k ( 1 ) − y i ^ k ( 1 ) ∥ 2 + ∥ y i ^ k ( 1 ) − y i k ( 1 ) ∥ 2 . (76) \begin{aligned}
\lVert\widetilde{S}^{(1)}_{k,\cdot} - S^{(1)}_{k,\cdot} \rVert_2
&= \lVert\widetilde{y}^{(1)}_{\widehat{i}_k} - y^{(1)}_{i_k} \rVert_2 \\
&\le\lVert\widetilde{y}^{(1)}_{\widehat{i}_k} - y^{(1)}_{\widehat{i}_k} \rVert_2 + \lVert y^{(1)}_{\widehat{i}_k} - y^{(1)}_{i_k} \rVert_2. \tag*{(76)}
\end{aligned} ∥ S k , ⋅ ( 1 ) − S k , ⋅ ( 1 ) ∥ 2 = ∥ y i k ( 1 ) − y i k ( 1 ) ∥ 2 ≤ ∥ y i k ( 1 ) − y i k ( 1 ) ∥ 2 + ∥ y i k ( 1 ) − y i k ( 1 ) ∥ 2 . (76)
The first term is bounded by C β n ( 1 ) C\sqrt{\beta_n^{(1)}} C β n ( 1 ) . For the second term, using the layerwise Lipschitz bound of T ( 1 ) T^{(1)} T ( 1 ) ,
∥ y i ^ k ( 1 ) − y i k ( 1 ) ∥ 2 = ∥ T ( 1 ) ( Z ^ i ^ k , ⋅ ) − T ( 1 ) ( e k ) ∥ 2 ≤ C K σ min ( 1 ) ∥ Z ^ i ^ k , ⋅ − e k ∥ 2 . \lVert y^{(1)}_{\widehat{i}_k} - y^{(1)}_{i_k} \rVert_2
= \lVert T^{(1)}(\widehat{Z}_{\widehat{i}_k,\cdot}) - T^{(1)}(e_k) \rVert_2
\le\frac{CK}{\sqrt{\sigma_{\min}^{(1)}}}\lVert\widehat{Z}_{\widehat{i}_k,\cdot} - e_k \rVert_2. ∥ y i k ( 1 ) − y i k ( 1 ) ∥ 2 = ∥ T ( 1 ) ( Z i k , ⋅ ) − T ( 1 ) ( e k ) ∥ 2 ≤ σ m i n ( 1 ) C K ∥ Z i k , ⋅ − e k ∥ 2 .
Since K K K is fixed and inf l σ min ( l ) ≥ c σ > 0 \inf_l \sigma_{\min}^{(l)} \ge c_\sigma> 0 inf l σ m i n ( l ) ≥ c σ > 0 , this gives
∥ y i ^ k ( 1 ) − y i k ( 1 ) ∥ 2 ≤ C r n , L ( x ) . \lVert y^{(1)}_{\widehat{i}_k} - y^{(1)}_{i_k} \rVert_2 \le C r_{n,L}(x). ∥ y i k ( 1 ) − y i k ( 1 ) ∥ 2 ≤ C r n , L ( x ) .
Hence
∥ Δ S ∥ 2 ≤ C ( β n ( 1 ) + r n , L ( x ) ) . (77) \lVert\Delta_S \rVert_2 \le C\left(\sqrt{\beta_n^{(1)}} + r_{n,L}(x)\right). \tag*{(77)} ∥ Δ S ∥ 2 ≤ C ( β n ( 1 ) + r n , L ( x ) ) . (77)
We now control the inverse terms. By the identifiability conditions, we have M ( 1 ) = I K \mathbf{M}^{(1)} = \mathbf{I}_K M ( 1 ) = I K . Since R ( l ) = M ( l ) S ( l ) \mathbf{R}^{(l)} = \mathbf{M}^{(l)}S^{(l)} R ( l ) = M ( l ) S ( l ) , we have S ( 1 ) = R ( 1 ) S^{(1)} = R^{(1)} S ( 1 ) = R ( 1 ) . Recall that σ min ( R ( l ) ) ≳ σ min ( B ( l ) ) \sigma_{\min}(\mathbf{R}^{(l)}) \gtrsim\sqrt{\sigma_{\min}(\mathbf{B}^{(l)})} σ m i n ( R ( l ) ) ≳ σ m i n ( B ( l ) ) by (C.4). We obtain an upper bound of ∥ ( S ( 1 ) ) − 1 ∥ 2 \lVert(S^{(1)})^{-1}\rVert_2 ∥( S ( 1 ) ) − 1 ∥ 2 .
∥ ( S ( 1 ) ) − 1 ∥ 2 = ∥ ( R ( 1 ) ) − 1 ∥ 2 = 1 σ min ( R ( 1 ) ) ≲ 1 σ min ( B ( 1 ) ) ≲ 1 σ min ( 1 ) . \begin{aligned}
\lVert(S^{(1)})^{-1}\rVert_2
&= \lVert(R^{(1)})^{-1}\rVert_2 \\
&= \frac{1}{\sigma_{\min}(R^{(1)})} \\
&\lesssim\frac{1}{\sqrt{\sigma_{\min}(B^{(1)})}} \\
&\lesssim\frac{1}{\sqrt{\sigma_{\min}^{(1)}}}.
\end{aligned} ∥( S ( 1 ) ) − 1 ∥ 2 = ∥( R ( 1 ) ) − 1 ∥ 2 = σ m i n ( R ( 1 ) ) 1 ≲ σ m i n ( B ( 1 ) ) 1 ≲ σ m i n ( 1 ) 1 .
Moreover, by Weyl’s inequality,
σ min ( S ~ ( 1 ) ) ≥ σ min ( S ( 1 ) ) − ∥ Δ S ∥ 2 . \sigma_{\min}(\widetilde{S}^{(1)}) \ge\sigma_{\min}(S^{(1)}) - \lVert\Delta_S \rVert_2. σ m i n ( S ( 1 ) ) ≥ σ m i n ( S ( 1 ) ) − ∥ Δ S ∥ 2 .
We show ∥ Δ S ∥ 2 ≤ 1 2 σ min ( S ( 1 ) ) \lVert\Delta_S \rVert_2 \le\frac{1}{2}\sigma_{\min}(S^{(1)}) ∥ Δ S ∥ 2 ≤ 2 1 σ m i n ( S ( 1 ) ) . By Assumption 3 and the definition of r n , L ( x ) r_{n,L}(x) r n , L ( x ) ,
β n ( 1 ) + r n , L ≤ C ( c β + n − 18 + x L ) . \sqrt{\beta_n^{(1)}} + r_{n,L} \le C\left(\sqrt{c_\beta} + n^{-18} + \frac{x}{L}\right). β n ( 1 ) + r n , L ≤ C ( c β + n − 18 + L x ) .
Since c β c_\beta c β is sufficiently small, we have C c β ≤ s 0 / 4 C\sqrt{c_\beta} \le s_0/4 C c β ≤ s 0 /4 , where s 0 s_0 s 0 is a uniform lower bound for σ min ( S ( 1 ) ) \sigma_{\min}(S^{(1)}) σ m i n ( S ( 1 ) ) . Moreover, since n − 18 → 0 n^{-18} \to0 n − 18 → 0 , there exist n 0 ∈ N n_0 \in\mathbb{N} n 0 ∈ N such that C n − 18 ≤ s 0 / 8 Cn^{-18} \le s_0/8 C n − 18 ≤ s 0 /8 for every n ≥ n 0 n \ge n_0 n ≥ n 0 . Lastly, take c x : = s 0 / ( 8 C ) c_x := s_0/(8C) c x := s 0 / ( 8 C ) so that C x / L ≤ C c x ≤ s 0 / 8 Cx/L \le Cc_x \le s_0/8 C x / L ≤ C c x ≤ s 0 /8 . Then, for every n ≥ n 0 n \ge n_0 n ≥ n 0 and 0 < x < c x L 0 < x < c_xL 0 < x < c x L , we have
∥ Δ S ∥ 2 ≤ 1 2 σ min ( S ( 1 ) ) . \lVert\Delta_S \rVert_2 \le\frac{1}{2}\sigma_{\min}(S^{(1)}). ∥ Δ S ∥ 2 ≤ 2 1 σ m i n ( S ( 1 ) ) .
Therefore,
σ min ( S ~ ( 1 ) ) ≥ 1 2 σ min ( S ( 1 ) ) , \sigma_{\min}(\widetilde{\mathbf{S}}^{(1)}) \ge\frac{1}{2}\sigma_{\min}(\mathbf{S}^{(1)}), σ m i n ( S ( 1 ) ) ≥ 2 1 σ m i n ( S ( 1 ) ) ,
and hence
∥ ( S ~ ( 1 ) ) − 1 ∥ 2 ≤ 2 ∥ ( S ( 1 ) ) − 1 ∥ 2 ≲ 1 σ min ( 1 ) . \left\|(\widetilde{\mathbf{S}}^{(1)})^{-1}\right\|_2 \le2\left\|(\mathbf{S}^{(1)})^{-1}\right\|_2 \lesssim\frac{1}{\sqrt{\sigma_{\min}^{(1)}}}. ( S ( 1 ) ) − 1 2 ≤ 2 ( S ( 1 ) ) − 1 2 ≲ σ m i n ( 1 ) 1 .
Also, by Lemma B.1 ,
∥ X i , ⋅ ( 1 ) ∥ 2 ≲ θ i ( 1 ) K \left\|\mathbf{X}_{i,\cdot}^{(1)}\right\|_2 \lesssim\theta_i^{(1)}\sqrt{K} X i , ⋅ ( 1 ) 2 ≲ θ i ( 1 ) K
and
∥ X ~ i , ⋅ ( 1 ) − X i , ⋅ ( 1 ) ∥ 2 ≤ C x θ i ( 1 ) σ min ( 1 ) K β n ( 1 ) . \left\|\widetilde{\mathbf{X}}_{i,\cdot}^{(1)}-\mathbf{X}_{i,\cdot}^{(1)}\right\|_2 \le Cx\theta_i^{(1)}\frac{\sigma_{\min}^{(1)}}{K}\sqrt{\beta_n^{(1)}}. X i , ⋅ ( 1 ) − X i , ⋅ ( 1 ) 2 ≤ C x θ i ( 1 ) K σ m i n ( 1 ) β n ( 1 ) .
Therefore, by triangle inequality,
∥ X ~ i , ⋅ ( 1 ) ∥ 2 ≤ C θ i ( 1 ) K \left\|\widetilde{\mathbf{X}}_{i,\cdot}^{(1)}\right\|_2 \le C\theta_i^{(1)}\sqrt{K} X i , ⋅ ( 1 ) 2 ≤ C θ i ( 1 ) K
uniformly over i i i .
Now decompose
U ^ i , ⋅ ( 1 ) − U i , ⋅ ( 1 ) = e i { X ~ ( 1 ) ( S ~ ( 1 ) ) − 1 − X ( 1 ) ( S ( 1 ) ) − 1 } = ( X ~ i , ⋅ ( 1 ) − X i , ⋅ ( 1 ) ) ( S ( 1 ) ) − 1 + X ~ i , ⋅ ( 1 ) { ( S ~ ( 1 ) ) − 1 − ( S ( 1 ) ) − 1 } . \begin{aligned}
\widehat{\mathbf{U}}_{i,\cdot}^{(1)}-\mathbf{U}_{i,\cdot}^{(1)}
= e_i\left\{\widetilde{\mathbf{X}}^{(1)}(\widetilde{\mathbf{S}}^{(1)})^{-1}-\mathbf{X}^{(1)}(\mathbf{S}^{(1)})^{-1}\right\} \\
&= (\widetilde{\mathbf{X}}_{i,\cdot}^{(1)}-\mathbf{X}_{i,\cdot}^{(1)})(\mathbf{S}^{(1)})^{-1}
+\widetilde{\mathbf{X}}_{i,\cdot}^{(1)}
\left\{(\widetilde{\mathbf{S}}^{(1)})^{-1}-(\mathbf{S}^{(1)})^{-1}\right\}.
\end{aligned} U i , ⋅ ( 1 ) − U i , ⋅ ( 1 ) = e i { X ( 1 ) ( S ( 1 ) ) − 1 − X ( 1 ) ( S ( 1 ) ) − 1 } = ( X i , ⋅ ( 1 ) − X i , ⋅ ( 1 ) ) ( S ( 1 ) ) − 1 + X i , ⋅ ( 1 ) { ( S ( 1 ) ) − 1 − ( S ( 1 ) ) − 1 } .
Using
( S ~ ( 1 ) ) − 1 − ( S ( 1 ) ) − 1 = ( S ~ ( 1 ) ) − 1 ( S ( 1 ) − S ~ ( 1 ) ) ( S ( 1 ) ) − 1 , (\widetilde{\mathbf{S}}^{(1)})^{-1}-(\mathbf{S}^{(1)})^{-1}
=(\widetilde{\mathbf{S}}^{(1)})^{-1}(\mathbf{S}^{(1)}-\widetilde{\mathbf{S}}^{(1)})(\mathbf{S}^{(1)})^{-1}, ( S ( 1 ) ) − 1 − ( S ( 1 ) ) − 1 = ( S ( 1 ) ) − 1 ( S ( 1 ) − S ( 1 ) ) ( S ( 1 ) ) − 1 ,
we obtain
∥ U ^ i , ⋅ ( 1 ) − U i , ⋅ ( 1 ) ∥ 2 ≤ ∥ X ~ i , ⋅ ( 1 ) − X i , ⋅ ( 1 ) ∥ 2 ∥ ( S ( 1 ) ) − 1 ∥ 2 + ∥ X ~ i , ⋅ ( 1 ) ∥ 2 ∥ ( S ~ ( 1 ) ) − 1 ∥ 2 ∥ Δ S ∥ 2 ∥ ( S ( 1 ) ) − 1 ∥ 2 (78) \left\|\widehat{\mathbf{U}}_{i,\cdot}^{(1)}-\mathbf{U}_{i,\cdot}^{(1)}\right\|_2
\le\left\|\widetilde{\mathbf{X}}_{i,\cdot}^{(1)}-\mathbf{X}_{i,\cdot}^{(1)}\right\|_2\left\|(\mathbf{S}^{(1)})^{-1}\right\|_2
+\left\|\widetilde{\mathbf{X}}_{i,\cdot}^{(1)}\right\|_2\left\|(\widetilde{\mathbf{S}}^{(1)})^{-1}\right\|_2\|\Delta S\|_2\left\|(\mathbf{S}^{(1)})^{-1}\right\|_2 \tag*{(78)} U i , ⋅ ( 1 ) − U i , ⋅ ( 1 ) 2 ≤ X i , ⋅ ( 1 ) − X i , ⋅ ( 1 ) 2 ( S ( 1 ) ) − 1 2 + X i , ⋅ ( 1 ) 2 ( S ( 1 ) ) − 1 2 ∥Δ S ∥ 2 ( S ( 1 ) ) − 1 2 (78)
≲ θ i ( 1 ) σ min ( 1 ) K β n ( 1 ) + θ i ( 1 ) K σ min ( 1 ) ( β n ( 1 ) + r n , L ) ≤ C U θ i ( 1 ) ( β n ( 1 ) + r n , L ( x ) ) ≤ C U ′ θ i ( 1 ) ( c β + n − 18 + x L ) . \begin{aligned}
\lesssim\theta_i^{(1)}\frac{\sqrt{\sigma_{\min}^{(1)}}}{K}\sqrt{\beta_n^{(1)}}+\theta_i^{(1)}\frac{\sqrt{K}}{\sigma_{\min}^{(1)}}\left(\sqrt{\beta_n^{(1)}}+r_{n,L}\right) \\
&\le C_U\theta_i^{(1)}\left(\sqrt{\beta_n^{(1)}}+r_{n,L}(x)\right) \\
&\le C_U'\theta_i^{(1)}\left(\sqrt{c_\beta}+n^{-18}+\frac{x}{L}\right).
\end{aligned} ≲ θ i ( 1 ) K σ m i n ( 1 ) β n ( 1 ) + θ i ( 1 ) σ m i n ( 1 ) K ( β n ( 1 ) + r n , L ) ≤ C U θ i ( 1 ) ( β n ( 1 ) + r n , L ( x ) ) ≤ C U ′ θ i ( 1 ) ( c β + n − 18 + L x ) .
Since c β c_\beta c β is sufficiently small, by choosing c β c_\beta c β and c x c_x c x sufficiently small and n 0 n_0 n 0 sufficiently large if necessary, for every n ≥ n 0 n\ge n_0 n ≥ n 0 and 0 < x ≤ c x L 0<x\le c_xL 0 < x ≤ c x L ,
∥ U ^ i , ⋅ ( 1 ) − U i , ⋅ ( 1 ) ∥ 2 ≤ C U ′ θ i ( 1 ) ( c β + n − 18 + x L ) ≤ θ i ( 1 ) 2 K . (79) \left\|\widehat{\mathbf{U}}_{i,\cdot}^{(1)}-\mathbf{U}_{i,\cdot}^{(1)}\right\|_2
\le C_U'\theta_i^{(1)}\left(\sqrt{c_\beta}+n^{-18}+\frac{x}{L}\right)
\le\frac{\theta_i^{(1)}}{2\sqrt{K}}. \tag*{(79)} U i , ⋅ ( 1 ) − U i , ⋅ ( 1 ) 2 ≤ C U ′ θ i ( 1 ) ( c β + n − 18 + L x ) ≤ 2 K θ i ( 1 ) . (79)
It remains to translate this U U U -error into membership error. Set
u i : = U i , ⋅ ( 1 ) , u ^ i : = U ^ i , ⋅ ( 1 ) . \mathbf{u}_i:=\mathbf{U}_{i,\cdot}^{(1)},\qquad\widehat{\mathbf{u}}_i:=\widehat{\mathbf{U}}_{i,\cdot}^{(1)}. u i := U i , ⋅ ( 1 ) , u i := U i , ⋅ ( 1 ) .
Since M ( 1 ) = I K \mathbf{M}^{(1)}=\mathbf{I}_K M ( 1 ) = I K ,
u i = θ i ( 1 ) Z i , ⋅ , 1 K T u i = θ i ( 1 ) . \mathbf{u}_i=\theta_i^{(1)}\mathbf{Z}_{i,\cdot},\qquad\mathbf{1}_K^\mathsf{T}\mathbf{u}_i=\theta_i^{(1)}. u i = θ i ( 1 ) Z i , ⋅ , 1 K T u i = θ i ( 1 ) .
Then
Z i , ⋅ = u i 1 K T u i = u i θ i ( 1 ) . \mathbf{Z}_{i,\cdot}=\frac{\mathbf{u}_i}{\mathbf{1}_K^\mathsf{T}\mathbf{u}_i}=\frac{\mathbf{u}_i}{\theta_i^{(1)}}. Z i , ⋅ = 1 K T u i u i = θ i ( 1 ) u i .
By norm equivalence and (F.4), we have
∥ u ^ i − u i ∥ 1 ≤ K ∥ u ^ i − u i ∥ 2 ≤ θ i ( 1 ) 2 . \left\|\widehat{\mathbf{u}}_i-\mathbf{u}_i\right\|_1
\le\sqrt{K}\left\|\widehat{\mathbf{u}}_i-\mathbf{u}_i\right\|_2
\le\frac{\theta_i^{(1)}}{2}. ∥ u i − u i ∥ 1 ≤ K ∥ u i − u i ∥ 2 ≤ 2 θ i ( 1 ) .
Hence we can obtain a lower bound of 1 K T u ^ i \mathbf{1}_K^{\mathsf{T}}\widehat{\mathbf{u}}_i 1 K T u i by following:
1 K T u ^ i = 1 K T u i + 1 K T ( u ^ i − u i ) ≥ 1 K T u i − ∣ 1 K T ( u ^ i − u i ) ∣ ≥ 1 K T u i − ∥ u ^ i − u i ∥ 1 ≥ θ i ( 1 ) − θ i ( 1 ) 2 = θ i ( 1 ) 2 > 0. \begin{aligned}
\mathbf{1}_K^{\mathsf{T}}\widehat{\mathbf{u}}_i
&= \mathbf{1}_K^{\mathsf{T}}\mathbf{u}_i+\mathbf{1}_K^{\mathsf{T}}(\widehat{\mathbf{u}}_i-\mathbf{u}_i) \\
&\ge\mathbf{1}_K^{\mathsf{T}}\mathbf{u}_i-\left|\mathbf{1}_K^{\mathsf{T}}(\widehat{\mathbf{u}}_i-\mathbf{u}_i)\right| \\
&\ge\mathbf{1}_K^{\mathsf{T}}\mathbf{u}_i-\|\widehat{\mathbf{u}}_i-\mathbf{u}_i\|_1 \\
&\ge\theta_i^{(1)}-\frac{\theta_i^{(1)}}{2}=\frac{\theta_i^{(1)}}{2}>0.
\end{aligned} 1 K T u i = 1 K T u i + 1 K T ( u i − u i ) ≥ 1 K T u i − 1 K T ( u i − u i ) ≥ 1 K T u i − ∥ u i − u i ∥ 1 ≥ θ i ( 1 ) − 2 θ i ( 1 ) = 2 θ i ( 1 ) > 0.
Therefore, the row-sum normalization is well-defined.
Now,
∥ u ^ i 1 K T u ^ i − u i 1 K T u i ∥ 1 ≤ ∥ u ^ i − u i ∥ 1 1 K T u ^ i + ∥ u i ∥ 1 ∣ 1 1 K T u ^ i − 1 1 K T u i ∣ ≤ 4 θ i ( 1 ) ∥ u ^ i − u i ∥ 1 . \begin{aligned}
\left\|\frac{\widehat{\mathbf{u}}_i}{\mathbf{1}_K^{\mathsf{T}}\widehat{\mathbf{u}}_i}-\frac{\mathbf{u}_i}{\mathbf{1}_K^{\mathsf{T}}\mathbf{u}_i}\right\|_1
\le\frac{\|\widehat{\mathbf{u}}_i-\mathbf{u}_i\|_1}{\mathbf{1}_K^{\mathsf{T}}\widehat{\mathbf{u}}_i}
+\|\mathbf{u}_i\|_1\left|\frac{1}{\mathbf{1}_K^{\mathsf{T}}\widehat{\mathbf{u}}_i}-\frac{1}{\mathbf{1}_K^{\mathsf{T}}\mathbf{u}_i}\right| \\
&\le\frac{4}{\theta_i^{(1)}}\|\widehat{\mathbf{u}}_i-\mathbf{u}_i\|_1.
\end{aligned} 1 K T u i u i − 1 K T u i u i 1 ≤ 1 K T u i ∥ u i − u i ∥ 1 + ∥ u i ∥ 1 1 K T u i 1 − 1 K T u i 1 ≤ θ i ( 1 ) 4 ∥ u i − u i ∥ 1 .
Therefore,
∥ u ^ i 1 K T u ^ i − Z i , ⋅ ∥ 2 ≤ 4 K θ i ( 1 ) ∥ u ^ i − u i ∥ 2 . \left\|\frac{\widehat{\mathbf{u}}_i}{\mathbf{1}_K^{\mathsf{T}}\widehat{\mathbf{u}}_i}-Z_{i,\cdot}\right\|_2
\le\frac{4\sqrt{K}}{\theta_i^{(1)}}\|\widehat{\mathbf{u}}_i-\mathbf{u}_i\|_2. 1 K T u i u i − Z i , ⋅ 2 ≤ θ i ( 1 ) 4 K ∥ u i − u i ∥ 2 .
Since Euclidean projection onto the simplex is non-expansive and Z i , ⋅ ∈ Δ K − 1 Z_{i,\cdot}\in\Delta^{K-1} Z i , ⋅ ∈ Δ K − 1 ,
∥ ( Z ^ 0 ) i , ⋅ − Z i , ⋅ ∥ 2 = ∥ Π Δ K − 1 ( u ^ i 1 K T u ^ i ) − Π Δ K − 1 ( Z i , ⋅ ) ∥ 2 ≤ ∥ u ^ i 1 K T u ^ i − Z i , ⋅ ∥ 2 ≤ 4 K θ i ( 1 ) ∥ u ^ i − u i ∥ 2 ≤ C Z ( β n ( 1 ) + r n , L ( x ) ) . \begin{aligned}
\|(\widehat{Z}_0)_{i,\cdot}-Z_{i,\cdot}\|_2
&=\left\|\Pi_{\Delta^{K-1}}\left(\frac{\widehat{\mathbf{u}}_i}{\mathbf{1}_K^{\mathsf{T}}\widehat{\mathbf{u}}_i}\right)-\Pi_{\Delta^{K-1}}(Z_{i,\cdot})\right\|_2 \\
&\le\left\|\frac{\widehat{\mathbf{u}}_i}{\mathbf{1}_K^{\mathsf{T}}\widehat{\mathbf{u}}_i}-Z_{i,\cdot}\right\|_2 \\
&\le\frac{4\sqrt{K}}{\theta_i^{(1)}}\|\widehat{\mathbf{u}}_i-\mathbf{u}_i\|_2 \\
&\le C_Z\left(\sqrt{\beta_n^{(1)}}+r_{n,L}(x)\right).
\end{aligned} ∥ ( Z 0 ) i , ⋅ − Z i , ⋅ ∥ 2 = Π Δ K − 1 ( 1 K T u i u i ) − Π Δ K − 1 ( Z i , ⋅ ) 2 ≤ 1 K T u i u i − Z i , ⋅ 2 ≤ θ i ( 1 ) 4 K ∥ u i − u i ∥ 2 ≤ C Z ( β n ( 1 ) + r n , L ( x ) ) .
Taking the maximum over i i i gives
∥ Z ^ 0 − Z ∥ 2 , ∞ ≤ C Z ( β n ( 1 ) + r n , L ( x ) ) . (80) \|\widehat{Z}_0-Z\|_{2,\infty}\le C_Z\left(\sqrt{\beta_n^{(1)}}+r_{n,L}(x)\right). \tag*{(80)} ∥ Z 0 − Z ∥ 2 , ∞ ≤ C Z ( β n ( 1 ) + r n , L ( x ) ) . (80)
Moreover,
1 n ∥ Z ^ 0 − Z ∥ F ≤ ∥ Z ^ 0 − Z ∥ 2 , ∞ . \frac{1}{\sqrt{n}}\|\widehat{Z}_0-Z\|_F\le\|\widehat{Z}_0-Z\|_{2,\infty}. n 1 ∥ Z 0 − Z ∥ F ≤ ∥ Z 0 − Z ∥ 2 , ∞ .
Substituting the definition of r n , L ( x ) r_{n,L}(x) r n , L ( x ) and restoring the label permutation Π \mathbf{\Pi} Π , we obtain
1 n ∥ Z ^ 0 Π − Z ∥ F ≤ ∥ Z ^ 0 Π − Z ∥ 2 , ∞ ≤ C ( β n ( 1 ) + 1 L ∑ l = 1 L β n ( l ) + n − 18 + x L ) . \frac{1}{\sqrt{n}}\|\widehat{Z}_0\mathbf{\Pi}-Z\|_F
\le\|\widehat{Z}_0\mathbf{\Pi}-Z\|_{2,\infty}
\le C\left(\sqrt{\beta_n^{(1)}}+\frac{1}{L}\sum_{l=1}^{L}\sqrt{\beta_n^{(l)}}+n^{-18}+\frac{x}{L}\right). n 1 ∥ Z 0 Π − Z ∥ F ≤ ∥ Z 0 Π − Z ∥ 2 , ∞ ≤ C ( β n ( 1 ) + L 1 l = 1 ∑ L β n ( l ) + n − 18 + L x ) .
This bound holds with probability at least 1 − e − x − O ( n − 18 ) 1-e^{-x}-O(n^{-18}) 1 − e − x − O ( n − 18 ) .
Moreover, suppose that r n , L ( x ) < δ m i x / 2 r_{n,L}(x)<\delta_{\mathrm{mix}}/\sqrt{2} r n , L ( x ) < δ mix / 2 . On the same event, Corollary 3.5 implies i ^ k ∈ P k \widehat{i}_k\in\mathcal{P}_k i k ∈ P k for every k ∈ [ K ] k\in[K] k ∈ [ K ] . Hence
y ^ i ^ k ( 1 ) = y i k ( 1 ) = s k , ⋅ ( 1 ) . \widehat{y}_{\widehat{i}_k}^{(1)}=y_{i_k}^{(1)}=s_{k,\cdot}^{(1)}. y i k ( 1 ) = y i k ( 1 ) = s k , ⋅ ( 1 ) .
Therefore, the second term in (F.1) vanishes, and
∥ Δ s ∥ 2 ≤ K max k ∈ [ K ] ∥ y ~ i ^ k ( 1 ) − y i k ( 1 ) ∥ 2 ≤ C β n ( 1 ) . \|\Delta s\|_2\le\sqrt{K}\max_{k\in[K]}\|\widetilde{y}_{\widehat{i}_k}^{(1)}-y_{i_k}^{(1)}\|_2\le C\sqrt{\beta_n^{(1)}}. ∥Δ s ∥ 2 ≤ K k ∈ [ K ] max ∥ y i k ( 1 ) − y i k ( 1 ) ∥ 2 ≤ C β n ( 1 ) .
Substituting this sharper corner-matrix bound into the inverse-perturbation argument (F.3) gives
∥ U ^ i , ⋅ ( 1 ) − U i , ⋅ ( 1 ) ∥ 2 ≤ C θ ( 1 ) β n ( 1 ) . \left\|\widehat{\mathbf{U}}^{(1)}_{i,\cdot}-\mathbf{U}^{(1)}_{i,\cdot}\right\|_{2}\le C\theta^{(1)}\sqrt{\beta_n^{(1)}}. U i , ⋅ ( 1 ) − U i , ⋅ ( 1 ) 2 ≤ C θ ( 1 ) β n ( 1 ) .
The same row-sum normalization and simplex-projection argument then yields, on the same event and with the same permutation,
1 n ∥ Z ^ 0 Π − Z ∥ F ≤ ∥ Z ^ 0 Π − Z ∥ 2 , ∞ ≤ C β n ( 1 ) . \frac{1}{\sqrt{n}}\left\|\widehat{\mathbf{Z}}_0\boldsymbol{\Pi}-\mathbf{Z}\right\|_{\mathrm{F}}\le\left\|\widehat{\mathbf{Z}}_0\boldsymbol{\Pi}-\mathbf{Z}\right\|_{2,\infty}\le C\sqrt{\beta_n^{(1)}}. n 1 Z 0 Π − Z F ≤ Z 0 Π − Z 2 , ∞ ≤ C β n ( 1 ) .
Minimax lower bound for membership estimation
Let F n , K , L \mathcal{F}_{n,K,L} F n , K , L denote the parameter space of MLDCMM with n n n nodes, K K K communities and L L L layers satisfying identifiability condition 1 and assumptions 1, 2 and 3. In this section, we provide a minimax lower bound of membership estimates under MLDCMM. That is, we obtain a lower bound of
inf Z ^ sup ϑ ∈ F n , K , L E ϑ [ min Π ∈ Q K ∥ Z ^ Π − Z ∥ 2 , ∞ ] . \inf_{\widehat{\mathbf{Z}}}\ \sup_{\vartheta\in\mathcal{F}_{n,K,L}}\mathbb{E}_{\vartheta}\left[\min_{\boldsymbol{\Pi}\in\mathcal{Q}_K}\left\|\widehat{\mathbf{Z}}\boldsymbol{\Pi}-\mathbf{Z}\right\|_{2,\infty}\right]. Z inf ϑ ∈ F n , K , L sup E ϑ [ Π ∈ Q K min Z Π − Z 2 , ∞ ] .
where Q K \mathcal{Q}_K Q K is the set of K × K K\times K K × K permutation matrices. We introduce a main challenge on multilayer mixed-membership model and provide a minimax lower bound for membership estimation.
A motivating example
In MLDCMM, given a parameter point ϑ \vartheta ϑ that satisfies the identifiability condition 1, one can construct another parameter point ϑ ′ \vartheta' ϑ ′ such that both ϑ \vartheta ϑ and ϑ ′ \vartheta' ϑ ′ yield the same edge probability matrix P ( l ) \mathbf{P}^{(l)} P ( l ) for all non-reference layers but different edge probability matrix for the reference layer. Without loss of generality, let l = 1 l=1 l = 1 denote the reference layer.
Consider the following parameter point ϑ 0 : = ( Θ 0 ( l ) , Z 0 , B 0 ( l ) ) l = 1 L \vartheta_0:=\left(\boldsymbol{\Theta}_0^{(l)},\mathbf{Z}_0,\mathbf{B}_0^{(l)}\right)_{l=1}^{L} ϑ 0 := ( Θ 0 ( l ) , Z 0 , B 0 ( l ) ) l = 1 L :
Θ 0 ( l ) = ρ n I n , B 0 ( l ) = ( 1 1 / 2 1 / 2 1 ) , ∀ l ∈ [ L ] , \boldsymbol{\Theta}_0^{(l)}=\sqrt{\rho_n}\mathbf{I}_n,\qquad
\mathbf{B}_0^{(l)}=
\begin{pmatrix}
1 & 1/2\\
1/2 & 1
\end{pmatrix},\quad\forall l\in[L], Θ 0 ( l ) = ρ n I n , B 0 ( l ) = ( 1 1/2 1/2 1 ) , ∀ l ∈ [ L ] ,
and Z 0 \mathbf{Z}_0 Z 0 be a mixed membership matrix satisfying the identifiability condition. Then ϑ 0 \vartheta_0 ϑ 0 satisfies the identifiability condition 1. We construct another such parameter point ϑ t : = ( Θ t ( l ) , Z t , B t ( l ) ) l = 1 L \vartheta_t:=\left(\boldsymbol{\Theta}_t^{(l)},\mathbf{Z}_t,\mathbf{B}_t^{(l)}\right)_{l=1}^{L} ϑ t := ( Θ t ( l ) , Z t , B t ( l ) ) l = 1 L . Suppose 0 < t ≤ 1 0<t\le1 0 < t ≤ 1 . Let D t : = diag ( e t , 1 ) \mathbf{D}_t:=\operatorname{diag}(e^t,1) D t := diag ( e t , 1 ) be a diagonal matrix that, when multiplied on the right of a membership matrix, changes the relative composition of mixed membership vectors while preserving the community direction of pure membership vectors. Since there exists at least one row with ∥ ( Z 0 D t ) i , ⋅ ∥ 1 ≠ 1 \|(\mathbf{Z}_0\mathbf{D}_t)_{i,\cdot}\|_1\ne1 ∥ ( Z 0 D t ) i , ⋅ ∥ 1 = 1 , we need another normalization matrix H t = diag ( Z 0 D t 1 2 ) \mathbf{H}_t=\operatorname{diag}(\mathbf{Z}_0\mathbf{D}_t\mathbf{1}_2) H t = diag ( Z 0 D t 1 2 ) so that Z t : = H t − 1 Z 0 D t \mathbf{Z}_t:=\mathbf{H}_t^{-1}\mathbf{Z}_0\mathbf{D}_t Z t := H t − 1 Z 0 D t has unit ℓ 1 \ell_1 ℓ 1 -norm rows. For l ≥ 2 l\ge2 l ≥ 2 , define Θ t ( l ) : = ρ n H t \boldsymbol{\Theta}_t^{(l)}:=\sqrt{\rho_n}\mathbf{H}_t Θ t ( l ) := ρ n H t and B t ( l ) : = D t − 1 B 0 ( l ) D t − 1 \mathbf{B}_t^{(l)}:=\mathbf{D}_t^{-1}\mathbf{B}_0^{(l)}\mathbf{D}_t^{-1} B t ( l ) := D t − 1 B 0 ( l ) D t − 1 . Then, for all non-reference layers l ≥ 2 l\ge2 l ≥ 2 ,
P t ( l ) = Θ t ( l ) Z t B t ( l ) Z t T Θ t ( l ) = ρ n H t Z t D t − 1 B 0 ( l ) D t − 1 Z t T H t = ρ n Z 0 B 0 ( l ) Z 0 T = P 0 ( l ) . \begin{aligned}
\mathbf{P}_t^{(l)}=\boldsymbol{\Theta}_t^{(l)}\mathbf{Z}_t\mathbf{B}_t^{(l)}\mathbf{Z}_t^{\mathsf{T}}\boldsymbol{\Theta}_t^{(l)} \\
&=\rho_n\mathbf{H}_t\mathbf{Z}_t\mathbf{D}_t^{-1}\mathbf{B}_0^{(l)}\mathbf{D}_t^{-1}\mathbf{Z}_t^{\mathsf{T}}\mathbf{H}_t \\
&=\rho_n\mathbf{Z}_0\mathbf{B}_0^{(l)}\mathbf{Z}_0^{\mathsf{T}}=\mathbf{P}_0^{(l)}.
\end{aligned} P t ( l ) = Θ t ( l ) Z t B t ( l ) Z t T Θ t ( l ) = ρ n H t Z t D t − 1 B 0 ( l ) D t − 1 Z t T H t = ρ n Z 0 B 0 ( l ) Z 0 T = P 0 ( l ) .
For the reference layer l = 1 l=1 l = 1 , define Θ t ( 1 ) : = ρ n I n \boldsymbol{\Theta}_t^{(1)}:=\sqrt{\rho_n}\mathbf{I}_n Θ t ( 1 ) := ρ n I n and B t ( 1 ) : = B 0 ( 1 ) \mathbf{B}_t^{(1)}:=\mathbf{B}_0^{(1)} B t ( 1 ) := B 0 ( 1 ) . Since Z 0 ≠ Z t \mathbf{Z}_0\ne\mathbf{Z}_t Z 0 = Z t , we have
P t ( 1 ) = ρ n Z t B 0 ( 1 ) Z t T ≠ ρ n Z 0 B 0 ( 1 ) Z 0 T = P 0 ( 1 ) . \mathbf{P}_t^{(1)}=\rho_n\mathbf{Z}_t\mathbf{B}_0^{(1)}\mathbf{Z}_t^{\mathsf{T}}\ne\rho_n\mathbf{Z}_0\mathbf{B}_0^{(1)}\mathbf{Z}_0^{\mathsf{T}}=\mathbf{P}_0^{(1)}. P t ( 1 ) = ρ n Z t B 0 ( 1 ) Z t T = ρ n Z 0 B 0 ( 1 ) Z 0 T = P 0 ( 1 ) .
Note that the two parameter points ϑ 0 \vartheta_0 ϑ 0 and ϑ t \vartheta_t ϑ t induce identical edge-probability matrices in all layers except the reference layer. Since both ϑ 0 \vartheta_0 ϑ 0 and ϑ t \vartheta_t ϑ t satisfy Condition 1, either collection of edge-probability matrices, ( P 0 ( l ) ) l = 1 L \left(P_0^{(l)}\right)_{l=1}^{L} ( P 0 ( l ) ) l = 1 L or ( P t ( l ) ) l = 1 L \left(P_t^{(l)}\right)_{l=1}^{L} ( P t ( l ) ) l = 1 L , uniquely identifies its corresponding parameter point. However, in this construction, all information that distinguishes Z 0 Z_0 Z 0 from Z t Z_t Z t is contained in the reference layer, since the two parameter points induce identical edge-probability matrices in every non-reference layer. Consequently, observing additional non-reference layers does not provide further information for distinguishing the two membership matrices. In this sense, membership estimation can be bottlenecked by a single reference layer even as the number of observed layers increases.
A minimax lower bound for membership estimation
We now show that, under the MLDCMM parameterization, the membership estimation error can be bottlenecked by a single reference layer. In particular, even when the number of layers increases, there exist parameter configurations for which the non-reference layers contain no information for distinguishing two nearby membership matrices. Throughout this subsection, we fix K = 2 K=2 K = 2 and let l = 1 l=1 l = 1 denote the reference layer. The lower bound below therefore also applies to any larger parameter space containing this class. We restate Theorem 3.7 for convenience.
Theorem 3.7 (Minimax lower bound for membership estimation). Fix K ≥ 2 K \ge2 K ≥ 2 . Suppose that n ≥ n 0 ( K ) n \ge n_0(K) n ≥ n 0 ( K ) and ρ n ∈ ( 0 , 1 ] \rho_n \in(0,1] ρ n ∈ ( 0 , 1 ] satisfy n ρ n ≥ C ρ ( K ) log n n\rho_n \ge C_\rho(K)\log n n ρ n ≥ C ρ ( K ) log n , where n 0 ( K ) n_0(K) n 0 ( K ) and C ρ ( K ) > 0 C_\rho(K)>0 C ρ ( K ) > 0 are sufficiently large constants. Let F n , K , L \mathcal{F}_{n,K,L} F n , K , L denote the parameter space of the ML-DCMM with n n n nodes, K K K communities and L L L layers satisfying Assumptions 1, 2, 3 and Condition 1. Then there exists a universal constant c > 0 c>0 c > 0 , independent of n n n , K K K , L L L , and ρ n \rho_n ρ n , such that
inf Z ^ sup ϑ ∈ F n , K , L E ϑ [ min Π ∈ Q K ∥ Z ^ Π − Z ∥ 2 , ∞ n ρ n ] ≥ c n ρ n . (81) \inf_{\widehat{Z}}\sup_{\vartheta\in\mathcal{F}_{n,K,L}}\mathbb{E}_{\vartheta}\left[\min_{\Pi\in\mathcal{Q}_K}\frac{\lVert\widehat{Z}\Pi-Z\rVert_{2,\infty}}{\sqrt{n\rho_n}}\right]\ge\frac{c}{\sqrt{n\rho_n}}. \tag*{(81)} Z inf ϑ ∈ F n , K , L sup E ϑ [ Π ∈ Q K min n ρ n ∥ Z Π − Z ∥ 2 , ∞ ] ≥ n ρ n c . (81)
Proof. We prove the result using Le Cam’s two-point method [33 ] (Yu 1997). We show the proof when K = 2 K=2 K = 2 and extend for K > 2 K>2 K > 2 at the end of the proof.
Step 1: Construction of two parameter points. In this step, we construct two parameter points in F n , 2 , L \mathcal{F}_{n,2,L} F n , 2 , L that satisfy the required conditions and assumptions. Fix ρ n ∈ ( 0 , 1 ] \rho_n\in(0,1] ρ n ∈ ( 0 , 1 ] such that n ρ n ≥ C ρ log n n\rho_n\ge C_\rho\log n n ρ n ≥ C ρ log n for a sufficiently large C ρ C_\rho C ρ to be specified below. We construct one point ϑ 0 : = ( Θ 0 ( l ) , Z 0 , B 0 ( l ) ) l = 1 L ∈ F n , 2 , L \vartheta_0 := \left(\Theta_0^{(l)},Z_0,B_0^{(l)}\right)_{l=1}^{L}\in\mathcal{F}_{n,2,L} ϑ 0 := ( Θ 0 ( l ) , Z 0 , B 0 ( l ) ) l = 1 L ∈ F n , 2 , L by following: let B 0 ( l ) : = B 0 = ( 1 1 / 2 1 / 2 1 ) B_0^{(l)}:=B_0=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix} B 0 ( l ) := B 0 = ( 1 1/2 1/2 1 ) and Θ 0 ( l ) : = ρ n I n \Theta_0^{(l)}:=\sqrt{\rho_n}I_n Θ 0 ( l ) := ρ n I n for all l ∈ [ L ] l\in[L] l ∈ [ L ] . Let Z 0 Z_0 Z 0 contain exactly one mixed node i ∗ i_* i ∗ satisfying ( Z 0 ) i ∗ , ⋅ = ( 1 / 2 , 1 / 2 ) (Z_0)_{i_*,\cdot}=(1/2,1/2) ( Z 0 ) i ∗ , ⋅ = ( 1/2 , 1/2 ) , and let all remaining nodes be n 1 n_1 n 1 pure nodes for the first community and n 2 n_2 n 2 pure nodes for the second community with n 1 + n 2 = n − 1 n_1+n_2=n-1 n 1 + n 2 = n − 1 and min ( n 1 , n 2 ) ≥ n 2 − 1 \min(n_1,n_2)\ge\frac{n}{2}-1 min ( n 1 , n 2 ) ≥ 2 n − 1 .
Next, we construct ϑ t : = ( Θ t ( l ) , Z t , B t ( l ) ) l = 1 L ∈ F n , 2 , L \vartheta_t:=\left(\Theta_t^{(l)},Z_t,B_t^{(l)}\right)_{l=1}^{L}\in\mathcal{F}_{n,2,L} ϑ t := ( Θ t ( l ) , Z t , B t ( l ) ) l = 1 L ∈ F n , 2 , L . For 0 < t ≤ 1 0<t\le1 0 < t ≤ 1 , define D t = diag ( t , 1 ) D_t=\operatorname{diag}(t,1) D t = diag ( t , 1 ) , H t = diag ( Z 0 D t 1 2 ) H_t=\operatorname{diag}(Z_0D_t1_2) H t = diag ( Z 0 D t 1 2 ) and Z t = H t − 1 Z 0 D t Z_t=H_t^{-1}Z_0D_t Z t = H t − 1 Z 0 D t . For every non-reference layer l ≥ 2 l\ge2 l ≥ 2 , set Θ t ( l ) = ρ n H t \Theta_t^{(l)}=\sqrt{\rho_n}H_t Θ t ( l ) = ρ n H t and B t ( l ) = D t − 1 B 0 D t − 1 B_t^{(l)}=D_t^{-1}B_0D_t^{-1} B t ( l ) = D t − 1 B 0 D t − 1 . For the reference layer, set Θ t ( 1 ) = ρ n I n \Theta_t^{(1)}=\sqrt{\rho_n}I_n Θ t ( 1 ) = ρ n I n , and B t ( 1 ) = B 0 B_t^{(1)}=B_0 B t ( 1 ) = B 0 . By construction, we have P 0 ( 1 ) ≠ P t ( 1 ) P_0^{(1)}\ne P_t^{(1)} P 0 ( 1 ) = P t ( 1 ) and P 0 ( l ) = P t ( l ) P_0^{(l)}=P_t^{(l)} P 0 ( l ) = P t ( l ) for all l ≥ 2 l\ge2 l ≥ 2 .
We show that the two parameter points ϑ 0 \vartheta_0 ϑ 0 and ϑ t \vartheta_t ϑ t satisfy Conditions 1 and 2, as well as Assumptions 1, 2, and 3.
Condition 1. Both Z 0 Z_0 Z 0 and Z t Z_t Z t have unit ℓ 1 \ell_1 ℓ 1 -norm rows and there exists at least n 2 − 1 \frac{n}{2}-1 2 n − 1 pure nodes for each community. Moreover, i ∗ i_* i ∗ -th node of Z 0 Z_0 Z 0 and Z t Z_t Z t satisfies the second condition of Condition 1. Lastly, we have the same block connectivity matrix for the reference layer: B 0 ( 1 ) = B t ( 1 ) = ( 1 1 / 2 1 / 2 1 ) B_0^{(1)}=B_t^{(1)}=\begin{pmatrix}1&1/2\\1/2&1\end{pmatrix} B 0 ( 1 ) = B t ( 1 ) = ( 1 1/2 1/2 1 ) . Since B 0 B_0 B 0 is positive definite, the condition ∥ ( R ( 1 ) ) k , ⋅ ∥ 2 = 1 \lVert(R^{(1)})_{k,\cdot}\rVert_2=1 ∥( R ( 1 ) ) k , ⋅ ∥ 2 = 1 for all k ∈ [ K ] k\in[K] k ∈ [ K ] is equivalent to diag ( B ( 1 ) ) = 1 K \operatorname{diag}(B^{(1)})=1_K diag ( B ( 1 ) ) = 1 K , which is true for both B 0 ( 1 ) B_0^{(1)} B 0 ( 1 ) and B t ( 1 ) B_t^{(1)} B t ( 1 ) . It is clear
that B 0 ( l ) \mathbf{B}^{(l)}_0 B 0 ( l ) is full rank and ∥ B 0 ( l ) ∥ max = 1 \|\mathbf{B}^{(l)}_0\|_{\max}=1 ∥ B 0 ( l ) ∥ m a x = 1 for all l ∈ [ L ] l\in[L] l ∈ [ L ] . For B t ( l ) \mathbf{B}^{(l)}_t B t ( l ) , we have
B t ( l ) = ( e − 2 t 1 2 e − t 1 2 e − t 1 ) , l ≥ 2. \mathbf{B}^{(l)}_t=\begin{pmatrix}e^{-2t}&\frac{1}{2}e^{-t}\\\frac{1}{2}e^{-t}&1\end{pmatrix},\quad l\ge2. B t ( l ) = ( e − 2 t 2 1 e − t 2 1 e − t 1 ) , l ≥ 2.
Therefore, B t ( l ) \mathbf{B}^{(l)}_t B t ( l ) is full rank and ∥ B t ( l ) ∥ max = 1 \|\mathbf{B}^{(l)}_t\|_{\max}=1 ∥ B t ( l ) ∥ m a x = 1 for all l ∈ [ L ] l\in[L] l ∈ [ L ] .
min ( n 1 , n 2 ) = { n 2 − 1 if n is even , n − 1 2 if n is odd . \min(n_1,n_2)=
\begin{cases}
\frac{n}{2}-1 & \text{if }n\text{ is even},\\
\frac{n-1}{2} & \text{if }n\text{ is odd}.
\end{cases} min ( n 1 , n 2 ) = { 2 n − 1 2 n − 1 if n is even , if n is odd .
Hence, when n ≥ 4 n\ge4 n ≥ 4 is even, min 1 ≤ k ≤ 2 n k n = 1 2 − 1 n ≥ 1 4 \min_{1\le k\le2}\frac{n_k}{n}=\frac{1}{2}-\frac{1}{n}\ge\frac{1}{4} min 1 ≤ k ≤ 2 n n k = 2 1 − n 1 ≥ 4 1 . When n ≥ 3 n\ge3 n ≥ 3 is odd, min 1 ≤ k ≤ 2 n k n = n − 1 2 n ≥ 1 3 > 1 4 \min_{1\le k\le2}\frac{n_k}{n}=\frac{n-1}{2n}\ge\frac{1}{3}>\frac{1}{4} min 1 ≤ k ≤ 2 n n k = 2 n n − 1 ≥ 3 1 > 4 1 . Therefore, we can fix p p u r e = 1 4 p_{\mathrm{pure}}=\frac{1}{4} p pure = 4 1 . It is clear that we can choose δ m i x = 1 2 \delta_{\mathrm{mix}}=\frac{1}{\sqrt{2}} δ mix = 2 1 for Z 0 \mathbf{Z}_0 Z 0 . The mixed node of Z t \mathbf{Z}_t Z t is computed as ( Z t ) i ∗ , ⋅ = ( e t 1 + e t , 1 1 + e t ) ⊤ (\mathbf{Z}_t)_{i_\ast,\cdot}=\left(\frac{e^t}{1+e^t},\frac{1}{1+e^t}\right)^\top ( Z t ) i ∗ , ⋅ = ( 1 + e t e t , 1 + e t 1 ) ⊤ . For any t ∈ ( 0 , 1 ] t\in(0,1] t ∈ ( 0 , 1 ] , we have
min k = 1 , 2 ∥ ( Z t ) i ∗ , ⋅ − e k ∥ 2 = 2 1 + e t ≥ 2 1 + e . \min_{k=1,2}\|(\mathbf{Z}_t)_{i_\ast,\cdot}-\mathbf{e}_k\|_2=\frac{\sqrt{2}}{1+e^t}\ge\frac{\sqrt{2}}{1+e}. k = 1 , 2 min ∥ ( Z t ) i ∗ , ⋅ − e k ∥ 2 = 1 + e t 2 ≥ 1 + e 2 .
Therefore, we can choose δ m i x = 2 1 + e \delta_{\mathrm{mix}}=\frac{\sqrt{2}}{1+e} δ mix = 1 + e 2 for Z t \mathbf{Z}_t Z t .
Assumption 1. Since G 0 ( l ) \mathbf{G}^{(l)}_0 G 0 ( l ) and G t ( l ) \mathbf{G}^{(l)}_t G t ( l ) are symmetric positive definite, we check Assumption 1 by showing boundedness of the eigenvalues of ( G 0 ( l ) ) 2 = K ∥ θ 0 ( l ) ∥ 2 2 Z 0 ⊤ ( Θ 0 ( l ) ) 2 Z 0 (\mathbf{G}^{(l)}_0)^2=\frac{K}{\|\boldsymbol{\theta}^{(l)}_0\|_2^2}\mathbf{Z}_0^\top(\boldsymbol{\Theta}^{(l)}_0)^2\mathbf{Z}_0 ( G 0 ( l ) ) 2 = ∥ θ 0 ( l ) ∥ 2 2 K Z 0 ⊤ ( Θ 0 ( l ) ) 2 Z 0 and ( G t ( l ) ) 2 = K ∥ θ t ( l ) ∥ 2 2 Z t ⊤ ( Θ t ( l ) ) 2 Z t (\mathbf{G}^{(l)}_t)^2=\frac{K}{\|\boldsymbol{\theta}^{(l)}_t\|_2^2}\mathbf{Z}_t^\top(\boldsymbol{\Theta}^{(l)}_t)^2\mathbf{Z}_t ( G t ( l ) ) 2 = ∥ θ t ( l ) ∥ 2 2 K Z t ⊤ ( Θ t ( l ) ) 2 Z t . Consider ( G 0 ( l ) ) 2 (\mathbf{G}^{(l)}_0)^2 ( G 0 ( l ) ) 2 . We have
( G 0 ( l ) ) 2 = 2 n ρ n Z 0 ⊤ ( Θ 0 ( l ) ) 2 Z 0 = 2 n Z 0 ⊤ Z 0 = 2 n ( ( n 1 0 0 n 2 ) + ( 1 / 4 1 / 4 1 / 4 1 / 4 ) ) = : 2 n C 0 ( l ) , ∀ l ∈ [ L ] . (\mathbf{G}^{(l)}_0)^2=\frac{2}{n\rho_n}\mathbf{Z}_0^\top(\boldsymbol{\Theta}^{(l)}_0)^2\mathbf{Z}_0=\frac{2}{n}\mathbf{Z}_0^\top\mathbf{Z}_0=\frac{2}{n}\left(\begin{pmatrix}n_1&0\\0&n_2\end{pmatrix}+\begin{pmatrix}1/4&1/4\\1/4&1/4\end{pmatrix}\right)=:\frac{2}{n}\mathbf{C}^{(l)}_0,\quad\forall l\in[L]. ( G 0 ( l ) ) 2 = n ρ n 2 Z 0 ⊤ ( Θ 0 ( l ) ) 2 Z 0 = n 2 Z 0 ⊤ Z 0 = n 2 ( ( n 1 0 0 n 2 ) + ( 1/4 1/4 1/4 1/4 ) ) =: n 2 C 0 ( l ) , ∀ l ∈ [ L ] .
Note that C 0 ( l ) \mathbf{C}^{(l)}_0 C 0 ( l ) is positive definite. Hence, λ max ( C 0 ( l ) ) ≤ tr ( C 0 ( l ) ) = n − 1 2 < n \lambda_{\max}(\mathbf{C}^{(l)}_0)\le\operatorname{tr}(\mathbf{C}^{(l)}_0)=n-\frac{1}{2}<n λ m a x ( C 0 ( l ) ) ≤ tr ( C 0 ( l ) ) = n − 2 1 < n and λ max ( ( G 0 ( l ) ) 2 ) = 2 n λ max ( C 0 ( l ) ) < 2 \lambda_{\max}((\mathbf{G}^{(l)}_0)^2)=\frac{2}{n}\lambda_{\max}(\mathbf{C}^{(l)}_0)<2 λ m a x (( G 0 ( l ) ) 2 ) = n 2 λ m a x ( C 0 ( l ) ) < 2 . On the other hand, by Weyl’s inequality, we have
λ min ( C 0 ( l ) ) ≥ λ min ( ( n 1 0 0 n 2 ) ) + λ min ( ( 1 / 4 1 / 4 1 / 4 1 / 4 ) ) = min ( n 1 , n 2 ) ≥ n 2 − 1. \lambda_{\min}(\mathbf{C}^{(l)}_0)\ge\lambda_{\min}\left(\begin{pmatrix}n_1&0\\0&n_2\end{pmatrix}\right)+\lambda_{\min}\left(\begin{pmatrix}1/4&1/4\\1/4&1/4\end{pmatrix}\right)=\min(n_1,n_2)\ge\frac{n}{2}-1. λ m i n ( C 0 ( l ) ) ≥ λ m i n ( ( n 1 0 0 n 2 ) ) + λ m i n ( ( 1/4 1/4 1/4 1/4 ) ) = min ( n 1 , n 2 ) ≥ 2 n − 1.
Therefore, λ min ( ( G 0 ( l ) ) 2 ) = 2 n λ min ( C 0 ( l ) ) ≥ 1 − 2 n ≥ 1 3 \lambda_{\min}((\mathbf{G}^{(l)}_0)^2)=\frac{2}{n}\lambda_{\min}(\mathbf{C}^{(l)}_0)\ge1-\frac{2}{n}\ge\frac{1}{3} λ m i n (( G 0 ( l ) ) 2 ) = n 2 λ m i n ( C 0 ( l ) ) ≥ 1 − n 2 ≥ 3 1 .
We show ( G t ( l ) ) 2 (\mathbf{G}^{(l)}_t)^2 ( G t ( l ) ) 2 satisfies the boundedness eigenvalue condition. Consider the reference layer l = 1 l=1 l = 1 . Let q t : = e t 1 + e t q_t:=\frac{e^t}{1+e^t} q t := 1 + e t e t so that ( Z t ) i ∗ , ⋅ = ( q t , 1 − q t ) ⊤ (\mathbf{Z}_t)_{i_\ast,\cdot}=(q_t,1-q_t)^\top ( Z t ) i ∗ , ⋅ = ( q t , 1 − q t ) ⊤ . Then
( G t ( 1 ) ) 2 = 2 n ρ n ρ n Z t ⊤ Z t = 2 n ( ( n 1 0 0 n 2 ) + ( Z t ) i ∗ , ⋅ ⊤ ( Z t ) i ∗ , ⋅ ) . (\mathbf{G}^{(1)}_t)^2=\frac{2}{n\rho_n}\rho_n\mathbf{Z}_t^\top\mathbf{Z}_t=\frac{2}{n}\left(\begin{pmatrix}n_1&0\\0&n_2\end{pmatrix}+(\mathbf{Z}_t)_{i_\ast,\cdot}^\top(\mathbf{Z}_t)_{i_\ast,\cdot}\right). ( G t ( 1 ) ) 2 = n ρ n 2 ρ n Z t ⊤ Z t = n 2 ( ( n 1 0 0 n 2 ) + ( Z t ) i ∗ , ⋅ ⊤ ( Z t ) i ∗ , ⋅ ) .
Similar to the case of ( G 0 ( l ) ) 2 (\mathbf{G}^{(l)}_0)^2 ( G 0 ( l ) ) 2 , one can obtain 1 3 ≤ λ min ( ( G t ( 1 ) ) 2 ) ≤ λ max ( ( G t ( 1 ) ) 2 ) < 2 \frac{1}{3}\le\lambda_{\min}((\mathbf{G}^{(1)}_t)^2)\le\lambda_{\max}((\mathbf{G}^{(1)}_t)^2)<2 3 1 ≤ λ m i n (( G t ( 1 ) ) 2 ) ≤ λ m a x (( G t ( 1 ) ) 2 ) < 2 . Next, consider l ≥ 2 l\ge2 l ≥ 2 . For non-reference layers, we have Θ t ( l ) = ρ n H t \boldsymbol{\Theta}^{(l)}_t=\sqrt{\rho_n}\mathbf{H}_t Θ t ( l ) = ρ n H t and Z t = H t − 1 Z 0 D t \mathbf{Z}_t=\mathbf{H}^{-1}_t\mathbf{Z}_0\mathbf{D}_t Z t = H t − 1 Z 0 D t . Hence,
Z t ⊤ ( Θ t ( l ) ) 2 Z t = ρ n D t Z 0 ⊤ Z 0 D t and ( G t ( l ) ) 2 = 2 ρ n ∥ θ t ( l ) ∥ 2 2 D t Z 0 ⊤ Z 0 D t . \mathbf{Z}_t^\top(\boldsymbol{\Theta}^{(l)}_t)^2\mathbf{Z}_t=\rho_n\mathbf{D}_t\mathbf{Z}_0^\top\mathbf{Z}_0\mathbf{D}_t
\quad\text{and}\quad
(\mathbf{G}^{(l)}_t)^2=\frac{2\rho_n}{\|\boldsymbol{\theta}^{(l)}_t\|_2^2}\mathbf{D}_t\mathbf{Z}_0^\top\mathbf{Z}_0\mathbf{D}_t. Z t ⊤ ( Θ t ( l ) ) 2 Z t = ρ n D t Z 0 ⊤ Z 0 D t and ( G t ( l ) ) 2 = ∥ θ t ( l ) ∥ 2 2 2 ρ n D t Z 0 ⊤ Z 0 D t .
Since 0 < t ≤ 1 0 < t \le1 0 < t ≤ 1 , 1 ≤ ( H t ) i i ≤ e 1 \le(H_t)_{ii} \le e 1 ≤ ( H t ) ii ≤ e and hence n ρ n ≤ ∥ θ t ( l ) ∥ 2 2 ≤ n ρ n e 2 n\rho_n \le\lVert\theta_t^{(l)}\rVert_2^2 \le n\rho_n e^2 n ρ n ≤ ∥ θ t ( l ) ∥ 2 2 ≤ n ρ n e 2 . Moreover, σ max ( D t ) = e t ≤ e \sigma_{\max}(D_t)=e^t \le e σ m a x ( D t ) = e t ≤ e and σ min ( D t ) = 1 \sigma_{\min}(D_t)=1 σ m i n ( D t ) = 1 . We obtain an upper bound of λ max ( ( G t ( l ) ) 2 ) \lambda_{\max}\left((G_t^{(l)})^2\right) λ m a x ( ( G t ( l ) ) 2 ) .
λ max ( ( G t ( l ) ) 2 ) = 2 ρ n ∥ θ t ( l ) ∥ 2 2 λ max ( D t Z 0 T Z 0 D t ) = 2 ρ n ∥ θ t ( l ) ∥ 2 2 ( σ max ( Z 0 D t ) ) 2 ≤ 2 ρ n ∥ θ t ( l ) ∥ 2 2 ( σ max ( Z 0 ) ) 2 ( σ max ( D t ) ) 2 = 2 ρ n ∥ θ t ( l ) ∥ 2 2 λ max ( Z 0 T Z 0 ) ( σ max ( D t ) ) 2 ≤ 2 ρ n n ρ n ( n − 1 2 ) e 2 ≤ 2 e 2 . \begin{aligned}
\lambda_{\max}\left((G_t^{(l)})^2\right)
&= \frac{2\rho_n}{\lVert\theta_t^{(l)}\rVert_2^2}\lambda_{\max}(D_t Z_0^\mathsf{T} Z_0 D_t) \\
&= \frac{2\rho_n}{\lVert\theta_t^{(l)}\rVert_2^2}\left(\sigma_{\max}(Z_0D_t)\right)^2 \\
&\le\frac{2\rho_n}{\lVert\theta_t^{(l)}\rVert_2^2}\left(\sigma_{\max}(Z_0)\right)^2\left(\sigma_{\max}(D_t)\right)^2 \\
&= \frac{2\rho_n}{\lVert\theta_t^{(l)}\rVert_2^2}\lambda_{\max}(Z_0^\mathsf{T}Z_0)\left(\sigma_{\max}(D_t)\right)^2 \\
&\le\frac{2\rho_n}{n\rho_n}\left(n-\frac{1}{2}\right)e^2 \\
&\le2e^2.
\end{aligned} λ m a x ( ( G t ( l ) ) 2 ) = ∥ θ t ( l ) ∥ 2 2 2 ρ n λ m a x ( D t Z 0 T Z 0 D t ) = ∥ θ t ( l ) ∥ 2 2 2 ρ n ( σ m a x ( Z 0 D t ) ) 2 ≤ ∥ θ t ( l ) ∥ 2 2 2 ρ n ( σ m a x ( Z 0 ) ) 2 ( σ m a x ( D t ) ) 2 = ∥ θ t ( l ) ∥ 2 2 2 ρ n λ m a x ( Z 0 T Z 0 ) ( σ m a x ( D t ) ) 2 ≤ n ρ n 2 ρ n ( n − 2 1 ) e 2 ≤ 2 e 2 .
We obtain a lower bound of λ min ( ( G t ( l ) ) 2 ) \lambda_{\min}\left((G_t^{(l)})^2\right) λ m i n ( ( G t ( l ) ) 2 ) . Since Z 0 Z_0 Z 0 is full column rank and D t D_t D t is invertible, σ min ( Z 0 D t ) ≥ σ min ( Z 0 ) σ min ( D t ) \sigma_{\min}(Z_0D_t)\ge\sigma_{\min}(Z_0)\sigma_{\min}(D_t) σ m i n ( Z 0 D t ) ≥ σ m i n ( Z 0 ) σ m i n ( D t ) . Therefore,
λ min ( ( G t ( l ) ) 2 ) = 2 ρ n ∥ θ t ( l ) ∥ 2 2 λ min ( D t Z 0 T Z 0 D t ) = 2 ρ n ∥ θ t ( l ) ∥ 2 2 ( σ min ( Z 0 D t ) ) 2 ≥ 2 ρ n ∥ θ t ( l ) ∥ 2 2 ( σ min ( Z 0 ) ) 2 ( σ min ( D t ) ) 2 = 2 ρ n ∥ θ t ( l ) ∥ 2 2 λ min ( Z 0 T Z 0 ) ( σ min ( D t ) ) 2 ≥ 2 ρ n n ρ n e 2 ( n / 2 − 1 ) 1 2 ≥ 1 e 2 − 2 n e 2 ≥ 1 3 e 2 ∀ n ≥ 3. \begin{aligned}
\lambda_{\min}\left((G_t^{(l)})^2\right)
&= \frac{2\rho_n}{\lVert\theta_t^{(l)}\rVert_2^2}\lambda_{\min}(D_t Z_0^\mathsf{T} Z_0 D_t) \\
&= \frac{2\rho_n}{\lVert\theta_t^{(l)}\rVert_2^2}\left(\sigma_{\min}(Z_0D_t)\right)^2 \\
&\ge\frac{2\rho_n}{\lVert\theta_t^{(l)}\rVert_2^2}\left(\sigma_{\min}(Z_0)\right)^2\left(\sigma_{\min}(D_t)\right)^2 \\
&= \frac{2\rho_n}{\lVert\theta_t^{(l)}\rVert_2^2}\lambda_{\min}(Z_0^\mathsf{T}Z_0)\left(\sigma_{\min}(D_t)\right)^2 \\
&\ge\frac{2\rho_n}{n\rho_n e^2}\left(n/2-1\right)1^2 \\
&\ge\frac{1}{e^2}-\frac{2}{ne^2} \\
&\ge\frac{1}{3e^2}\qquad\forall n\ge3.
\end{aligned} λ m i n ( ( G t ( l ) ) 2 ) = ∥ θ t ( l ) ∥ 2 2 2 ρ n λ m i n ( D t Z 0 T Z 0 D t ) = ∥ θ t ( l ) ∥ 2 2 2 ρ n ( σ m i n ( Z 0 D t ) ) 2 ≥ ∥ θ t ( l ) ∥ 2 2 2 ρ n ( σ m i n ( Z 0 ) ) 2 ( σ m i n ( D t ) ) 2 = ∥ θ t ( l ) ∥ 2 2 2 ρ n λ m i n ( Z 0 T Z 0 ) ( σ m i n ( D t ) ) 2 ≥ n ρ n e 2 2 ρ n ( n /2 − 1 ) 1 2 ≥ e 2 1 − n e 2 2 ≥ 3 e 2 1 ∀ n ≥ 3.
Consequently, Assumption 1 holds for both parameter points with the common constant C G = 3 e C_G=\sqrt{3}e C G = 3 e .
Assumption 2. Since σ min ( B 0 ( l ) ) = 0.5 \sigma_{\min}(B_0^{(l)})=0.5 σ m i n ( B 0 ( l ) ) = 0.5 , one can choose σ B t ( l ) = 0.5 \sigma_{B_t}^{(l)}=0.5 σ B t ( l ) = 0.5 for all l ∈ [ L ] l\in[L] l ∈ [ L ] . For the reference layer l = 1 l=1 l = 1 , it follows that B t ( l ) = B 0 B_t^{(l)}=B_0 B t ( l ) = B 0 . Hence one can choose σ B t ( l ) = 0.5 \sigma_{B_t}^{(l)}=0.5 σ B t ( l ) = 0.5 . For non-reference layers l ≥ 2 l\ge2 l ≥ 2 , we have B t ( l ) = D t − 1 B 0 D t − 1 B_t^{(l)}=D_t^{-1}B_0D_t^{-1} B t ( l ) = D t − 1 B 0 D t − 1 . Since D t − 1 D_t^{-1} D t − 1 and B 0 B_0 B 0 are invertible, it follows that σ min ( B t ( l ) ) = σ min ( D t − 1 B 0 D t − 1 ) ≥ ( σ min ( D t − 1 ) ) 2 σ min ( B 0 ) \sigma_{\min}(B_t^{(l)})=\sigma_{\min}(D_t^{-1}B_0D_t^{-1})\ge\left(\sigma_{\min}(D_t^{-1})\right)^2\sigma_{\min}(B_0) σ m i n ( B t ( l ) ) = σ m i n ( D t − 1 B 0 D t − 1 ) ≥ ( σ m i n ( D t − 1 ) ) 2 σ m i n ( B 0 ) . For 0 < t ≤ 1 0<t\le1 0 < t ≤ 1 , σ min ( D t − 1 ) = e − t ≥ e − 1 \sigma_{\min}(D_t^{-1})=e^{-t}\ge e^{-1} σ m i n ( D t − 1 ) = e − t ≥ e − 1 . Therefore,
σ min ( B t ( l ) ) ≥ ( σ min ( D t − 1 ) ) 2 σ min ( B 0 ) ≥ ( 1 e ) 2 1 2 = 1 2 e 2 . \sigma_{\min}(B_t^{(l)})\ge\left(\sigma_{\min}(D_t^{-1})\right)^2\sigma_{\min}(B_0)\ge\left(\frac{1}{e}\right)^2\frac{1}{2}=\frac{1}{2e^2}. σ m i n ( B t ( l ) ) ≥ ( σ m i n ( D t − 1 ) ) 2 σ m i n ( B 0 ) ≥ ( e 1 ) 2 2 1 = 2 e 2 1 .
Thus, one can choose σ B t ( l ) = 1 2 e 2 \sigma_{B_t}^{(l)}=\frac{1}{2e^2} σ B t ( l ) = 2 e 2 1 for all l ∈ [ L ] l\in[L] l ∈ [ L ] .
θ min , 0 ( l ) ∥ θ 0 ( l ) ∥ 1 = ρ n ⋅ n ρ n ≥ C ρ log ( n ) \theta_{\min,0}^{(l)}\lVert\theta_0^{(l)}\rVert_1=\sqrt{\rho_n}\cdot n\sqrt{\rho_n}\ge C_\rho\log(n) θ m i n , 0 ( l ) ∥ θ 0 ( l ) ∥ 1 = ρ n ⋅ n ρ n ≥ C ρ log ( n )
and
β n , 0 ( l ) = ( ρ n ρ n ) 2 8 ρ n n ρ n log ( n ) ( 1 / 2 ) 3 ( n ρ n ) 2 = 2 11 log ( n ) n ρ n ≤ 2 11 C ρ 2 . \beta_{n,0}^{(l)}=\left(\frac{\sqrt{\rho_n}}{\sqrt{\rho_n}}\right)\frac{2^8\sqrt{\rho_n}n\sqrt{\rho_n}\log(n)}{(1/2)^3(n\rho_n)^2}=\frac{2^{11}\log(n)}{n\rho_n}\le\frac{2^{11}}{C_\rho^2}. β n , 0 ( l ) = ( ρ n ρ n ) ( 1/2 ) 3 ( n ρ n ) 2 2 8 ρ n n ρ n log ( n ) = n ρ n 2 11 log ( n ) ≤ C ρ 2 2 11 .
Next, consider ϑ t \vartheta_t ϑ t . For the reference layer l = 1 l = 1 l = 1 , we have the same results as above. For non-reference layers l ≥ 2 l \ge2 l ≥ 2 , we have θ min , t ( l ) = ρ n \theta_{\min,t}^{(l)} = \sqrt{\rho_n} θ m i n , t ( l ) = ρ n and θ max , t ( l ) = e t ρ n \theta_{\max,t}^{(l)} = e^t\sqrt{\rho_n} θ m a x , t ( l ) = e t ρ n . Since 1 ≤ ( H t ) i i ≤ e t 1 \le(H_t)_{ii} \le e^t 1 ≤ ( H t ) ii ≤ e t , we have
n ρ n ≤ ∥ θ t ( l ) ∥ 1 ≤ e n ρ n and n ρ n ≤ ∥ θ t ( l ) ∥ 2 2 ≤ e 2 n ρ n . n\sqrt{\rho_n} \le\|\theta_t^{(l)}\|_1 \le en\sqrt{\rho_n} \quad\text{and} \quad n\rho_n \le\|\theta_t^{(l)}\|_2^2 \le e^2n\rho_n. n ρ n ≤ ∥ θ t ( l ) ∥ 1 ≤ e n ρ n and n ρ n ≤ ∥ θ t ( l ) ∥ 2 2 ≤ e 2 n ρ n .
For the first condition, it follows that
θ min , t ( l ) ∥ θ t ( l ) ∥ 1 ≥ ρ n ⋅ n ρ n ≥ C ρ log ( n ) . \theta_{\min,t}^{(l)}\|\theta_t^{(l)}\|_1 \ge\sqrt{\rho_n}\cdot n\sqrt{\rho_n} \ge C_\rho\log(n). θ m i n , t ( l ) ∥ θ t ( l ) ∥ 1 ≥ ρ n ⋅ n ρ n ≥ C ρ log ( n ) .
For the second condition, we have
β n , t ( l ) = ( e t ρ n ρ n ) 2 8 e t ρ n ∥ θ t ( l ) ∥ 1 log ( n ) ( 1 2 e 2 ) 3 ∥ θ t ( l ) ∥ 2 4 ≤ e t ⋅ 2 11 e 6 + t ρ n e n ρ n log ( n ) ( n ρ n ) 2 = 2 11 e 7 + 2 t log ( n ) n ρ n ≤ 2 11 e 7 + 2 t C ρ . \begin{aligned}
\beta_{n,t}^{(l)} = \left(\frac{e^t\sqrt{\rho_n}}{\sqrt{\rho_n}}\right)\frac{2^8e^t\sqrt{\rho_n}\|\theta_t^{(l)}\|_1\log(n)}{\left(\frac{1}{2e^2}\right)^3\|\theta_t^{(l)}\|_2^4} \\
&\le e^t\cdot\frac{2^{11}e^{6+t}\sqrt{\rho_n}en\sqrt{\rho_n}\log(n)}{(n\rho_n)^2} \\
&= \frac{2^{11}e^{7+2t}\log(n)}{n\rho_n} \\
&\le\frac{2^{11}e^{7+2t}}{C_\rho}.
\end{aligned} β n , t ( l ) = ( ρ n e t ρ n ) ( 2 e 2 1 ) 3 ∥ θ t ( l ) ∥ 2 4 2 8 e t ρ n ∥ θ t ( l ) ∥ 1 log ( n ) ≤ e t ⋅ ( n ρ n ) 2 2 11 e 6 + t ρ n e n ρ n log ( n ) = n ρ n 2 11 e 7 + 2 t log ( n ) ≤ C ρ 2 11 e 7 + 2 t .
For 0 < t ≤ 1 0 < t \le1 0 < t ≤ 1 , it follows that β n ( l ) ≤ 2 11 e 9 C ρ \beta_n^{(l)} \le\frac{2^{11}e^9}{C_\rho} β n ( l ) ≤ C ρ 2 11 e 9 . Let C ρ ≥ max { c d e g , 2 11 e 9 c β } C_\rho\ge\max\{c_{\mathrm{deg}}, \frac{2^{11}e^9}{c_\beta}\} C ρ ≥ max { c deg , c β 2 11 e 9 } . Then both ϑ 0 \vartheta_0 ϑ 0 and ϑ t \vartheta_t ϑ t satisfy the Assumption 3.
Step 2: Separation between the two membership matrices. The transformation leaves every pure membership vector unchanged. For the unique mixed node, we have ( Z t ) i ∗ , ⋅ = ( q t , 1 − q t ) (Z_t)_{i_*,\cdot} = (q_t,1-q_t) ( Z t ) i ∗ , ⋅ = ( q t , 1 − q t ) , where q t = e t 1 + e t q_t = \frac{e^t}{1+e^t} q t = 1 + e t e t . Therefore, Δ t : = ∥ Z t − Z 0 ∥ 2 , ∞ = 2 ( q t − 1 2 ) \Delta_t := \|Z_t-Z_0\|_{2,\infty} = \sqrt{2}\left(q_t-\frac{1}{2}\right) Δ t := ∥ Z t − Z 0 ∥ 2 , ∞ = 2 ( q t − 2 1 ) . Since q t − 1 2 = e t − 1 2 ( e t + 1 ) q_t-\frac{1}{2}=\frac{e^t-1}{2(e^t+1)} q t − 2 1 = 2 ( e t + 1 ) e t − 1 , for 0 < t ≤ 1 0<t\le1 0 < t ≤ 1 ,
t 2 ( e + 1 ) ≤ q t − 1 2 ≤ t 4 . (82) \frac{t}{2(e+1)} \le q_t-\frac{1}{2}\le\frac{t}{4}. \tag*{(82)} 2 ( e + 1 ) t ≤ q t − 2 1 ≤ 4 t . (82)
Consequently, Δ t ≍ t \Delta_t\asymp t Δ t ≍ t .
We next verify that label permutation does not reduce the separation for this two-point construction. There are only two permutations in P 2 \mathcal{P}_2 P 2 . For the identity permutation, the distance is Δ t \Delta_t Δ t . For the nontrivial permutation Π s w a p \Pi_{\mathrm{swap}} Π swap , consider any pure node in the first community. Since the transformation leaves this node pure in the first community, ( Z 0 ) i , ⋅ = ( Z t ) i , ⋅ = ( 1 , 0 ) (Z_0)_{i,\cdot}=(Z_t)_{i,\cdot}=(1,0) ( Z 0 ) i , ⋅ = ( Z t ) i , ⋅ = ( 1 , 0 ) . Thus,
∥ ( Z t Π s w a p ) i , ⋅ − ( Z 0 ) i , ⋅ ∥ 2 = 2 . \|(Z_t\Pi_{\mathrm{swap}})_{i,\cdot}-(Z_0)_{i,\cdot}\|_2=\sqrt{2}. ∥ ( Z t Π swap ) i , ⋅ − ( Z 0 ) i , ⋅ ∥ 2 = 2 .
Since Δ t < 2 \Delta_t<\sqrt{2} Δ t < 2 for 0 < t ≤ 1 0<t\le1 0 < t ≤ 1 , the identity permutation is the minimizer. Hence
min Π ∈ P 2 ∥ Z t Π − Z 0 ∥ 2 , ∞ = Δ t ≍ t . (83) \min_{\Pi\in\mathcal{P}_2}\|Z_t\Pi-Z_0\|_{2,\infty}=\Delta_t\asymp t. \tag*{(83)} Π ∈ P 2 min ∥ Z t Π − Z 0 ∥ 2 , ∞ = Δ t ≍ t . (83)
Step 3: Kullback–Leibler divergence. Let P t \mathbb{P}_t P t and P 0 \mathbb{P}_0 P 0 denote the joint distributions of the multilayer adjacency matrices under the two parameter points. Since P t ( l ) = P 0 ( l ) \mathbb{P}_t^{(l)}=\mathbb{P}_0^{(l)} P t ( l ) = P 0 ( l ) for all l ≥ 2 l\ge2 l ≥ 2 , only the reference layer contributes to the KL divergence.
Moreover, within the reference layer, only edges incident to the mixed node i ∗ i_* i ∗ are changed. For every j ≠ i ∗ j\ne i_* j = i ∗ , ( P 0 ( 1 ) ) i ∗ , j = 3 4 ρ n (P_0^{(1)})_{i_*,j}=\frac{3}{4}\rho_n ( P 0 ( 1 ) ) i ∗ , j = 4 3 ρ n . Regardless of whether j j j belongs to the first or the second community,
∣ ( P t ( 1 ) ) i ∗ , j − ( P 0 ( 1 ) ) i ∗ , j ∣ = ∣ ρ n ( 1 2 + q t 2 ) − 3 4 ρ n ∣ = ρ n 2 ∣ q t − 1 2 ∣ . \left|(P_t^{(1)})_{i_*,j}-(P_0^{(1)})_{i_*,j}\right|
=\left|\rho_n\left(\frac{1}{2}+\frac{q_t}{2}\right)-\frac{3}{4}\rho_n\right|
=\frac{\rho_n}{2}\left|q_t-\frac{1}{2}\right|. ( P t ( 1 ) ) i ∗ , j − ( P 0 ( 1 ) ) i ∗ , j = ρ n ( 2 1 + 2 q t ) − 4 3 ρ n = 2 ρ n q t − 2 1 .
For Bernoulli distributions with parameter p p p and q q q , we have
kl ( Bern ( p ) , Bern ( q ) ) ≤ ( p − q ) 2 q ( 1 − q ) . \operatorname{kl}(\operatorname{Bern}(p),\operatorname{Bern}(q)) \le\frac{(p-q)^2}{q(1-q)}. kl ( Bern ( p ) , Bern ( q )) ≤ q ( 1 − q ) ( p − q ) 2 .
Let p : = ( P t ( 1 ) ) i ∗ , j p := (\mathbf{P}^{(1)}_{t})_{i_*,j} p := ( P t ( 1 ) ) i ∗ , j and q : = ( P 0 ( 1 ) ) i ∗ , j q := (\mathbf{P}^{(1)}_{0})_{i_*,j} q := ( P 0 ( 1 ) ) i ∗ , j . Since q = 3 4 ρ n q = \frac{3}{4}\rho_n q = 4 3 ρ n and ρ n ≤ 1 \rho_n \le1 ρ n ≤ 1 ,
q ( 1 − q ) = 3 4 ρ n ( 1 − 3 4 ρ n ) ≥ 3 16 ρ n . (84) q(1-q) = \frac{3}{4}\rho_n\left(1-\frac{3}{4}\rho_n\right) \ge\frac{3}{16}\rho_n. \tag*{(84)} q ( 1 − q ) = 4 3 ρ n ( 1 − 4 3 ρ n ) ≥ 16 3 ρ n . (84)
By (G.1) and (G.3), it follows that
kl ( Bern ( ( P t ( 1 ) ) i ∗ , j ) , Bern ( ( P 0 ( 1 ) ) i ∗ , j ) ) ≤ 4 3 ρ n ( q t − 1 2 ) 2 ≤ 1 12 ρ n t 2 . \operatorname{kl}\left(\operatorname{Bern}\left((\mathbf{P}^{(1)}_{t})_{i_*,j}\right),\operatorname{Bern}\left((\mathbf{P}^{(1)}_{0})_{i_*,j}\right)\right) \le\frac{4}{3}\rho_n\left(q_t-\frac{1}{2}\right)^2 \le\frac{1}{12}\rho_n t^2. kl ( Bern ( ( P t ( 1 ) ) i ∗ , j ) , Bern ( ( P 0 ( 1 ) ) i ∗ , j ) ) ≤ 3 4 ρ n ( q t − 2 1 ) 2 ≤ 12 1 ρ n t 2 .
Summing over the n − 1 n-1 n − 1 affected edges yields
D K L ( P t ∥ P 0 ) ≤ n ρ n t 2 12 . D_{\mathrm{KL}}(\mathbb{P}_t\Vert\mathbb{P}_0) \le\frac{n\rho_n t^2}{12}. D KL ( P t ∥ P 0 ) ≤ 12 n ρ n t 2 .
Now choose t n : = 1 n ρ n t_n := \frac{1}{\sqrt{n\rho_n}} t n := n ρ n 1 . Then 0 < t n ≤ 1 0 < t_n \le1 0 < t n ≤ 1 and D K L ( P t n ∥ P 0 ) ≤ 1 12 D_{\mathrm{KL}}(\mathbb{P}_{t_n}\Vert\mathbb{P}_0) \le\frac{1}{12} D KL ( P t n ∥ P 0 ) ≤ 12 1 . By Pinsker’s inequality,
TV ( P t n , P 0 ) ≤ 1 2 D K L ( P t n ∥ P 0 ) < 1 4 . (85) \operatorname{TV}(\mathbb{P}_{t_n},\mathbb{P}_0) \le\sqrt{\frac{1}{2}D_{\mathrm{KL}}(\mathbb{P}_{t_n}\Vert\mathbb{P}_0)} < \frac{1}{4}. \tag*{(85)} TV ( P t n , P 0 ) ≤ 2 1 D KL ( P t n ∥ P 0 ) < 4 1 . (85)
Step 4: Le Cam’s two-point argument. In this step, we use Le Cam’s method Yu [1997, Lemma 1] to obtain a lower bound of max ϑ ∈ { ϑ 0 , ϑ t n } E ϑ [ d ( Z ^ , Z ϑ ) ] \max_{\vartheta\in\{\vartheta_0,\vartheta_{t_n}\}}\mathbb{E}_{\vartheta}\left[d(\widehat{\mathbf{Z}},\mathbf{Z}_{\vartheta})\right] max ϑ ∈ { ϑ 0 , ϑ t n } E ϑ [ d ( Z , Z ϑ ) ] . Recall that we defined the distance d ( Z ^ , Z ϑ ) : = min Π ∈ P 2 ∥ Z ^ Π − Z ϑ ∥ 2 , ∞ d(\widehat{\mathbf{Z}},\mathbf{Z}_{\vartheta}) := \min_{\Pi\in\mathcal{P}_2}\|\widehat{\mathbf{Z}}\Pi-\mathbf{Z}_{\vartheta}\|_{2,\infty} d ( Z , Z ϑ ) := min Π ∈ P 2 ∥ Z Π − Z ϑ ∥ 2 , ∞ . We show d d d is a pseudo-metric on the space of membership matrices. Nonnegativity and d ( Z , Z ) = 0 d(\mathbf{Z},\mathbf{Z})=0 d ( Z , Z ) = 0 are immediate, while symmetry follows from the fact that Π − 1 ∈ P 2 \Pi^{-1}\in\mathcal{P}_2 Π − 1 ∈ P 2 and from the invariance of the 2 → ∞ 2\to\infty 2 → ∞ norm under column permutations. It remains to verify the triangle inequality. For any Z 1 , Z 2 , Z 3 \mathbf{Z}_1,\mathbf{Z}_2,\mathbf{Z}_3 Z 1 , Z 2 , Z 3 , we show
d ( Z 1 , Z 3 ) ≤ d ( Z 1 , Z 2 ) + d ( Z 2 , Z 3 ) . d(\mathbf{Z}_1,\mathbf{Z}_3)\le d(\mathbf{Z}_1,\mathbf{Z}_2)+d(\mathbf{Z}_2,\mathbf{Z}_3). d ( Z 1 , Z 3 ) ≤ d ( Z 1 , Z 2 ) + d ( Z 2 , Z 3 ) .
Let Π 12 \Pi_{12} Π 12 and Π 23 \Pi_{23} Π 23 are minimizing permutations for d ( Z 1 , Z 2 ) d(\mathbf{Z}_1,\mathbf{Z}_2) d ( Z 1 , Z 2 ) and d ( Z 2 , Z 3 ) d(\mathbf{Z}_2,\mathbf{Z}_3) d ( Z 2 , Z 3 ) , respectively, then
d ( Z 1 , Z 3 ) ≤ ∥ Z 1 Π 12 Π 23 − Z 3 ∥ 2 , ∞ ≤ ∥ ( Z 1 Π 12 − Z 2 ) Π 23 ∥ 2 , ∞ + ∥ Z 2 Π 23 − Z 3 ∥ 2 , ∞ = d ( Z 1 , Z 2 ) + d ( Z 2 , Z 3 ) , \begin{aligned}
d(\mathbf{Z}_1,\mathbf{Z}_3) &\le\|\mathbf{Z}_1\Pi_{12}\Pi_{23}-\mathbf{Z}_3\|_{2,\infty}\\
&\le\|(\mathbf{Z}_1\Pi_{12}-\mathbf{Z}_2)\Pi_{23}\|_{2,\infty}+\|\mathbf{Z}_2\Pi_{23}-\mathbf{Z}_3\|_{2,\infty}\\
&=d(\mathbf{Z}_1,\mathbf{Z}_2)+d(\mathbf{Z}_2,\mathbf{Z}_3),
\end{aligned} d ( Z 1 , Z 3 ) ≤ ∥ Z 1 Π 12 Π 23 − Z 3 ∥ 2 , ∞ ≤ ∥ ( Z 1 Π 12 − Z 2 ) Π 23 ∥ 2 , ∞ + ∥ Z 2 Π 23 − Z 3 ∥ 2 , ∞ = d ( Z 1 , Z 2 ) + d ( Z 2 , Z 3 ) ,
where the last equality follows from the invariance of the 2 → ∞ 2\to\infty 2 → ∞ norm under column permutations. Moreover, by (G.2), the separation between the two membership matrices satisfies
min Π ∈ P 2 ∥ Z t n Π − Z 0 ∥ 2 , ∞ = Δ t n ≥ c 0 t n \min_{\Pi\in\mathcal{P}_2}\|\mathbf{Z}_{t_n}\Pi-\mathbf{Z}_0\|_{2,\infty}=\Delta_{t_n}\ge c_0t_n Π ∈ P 2 min ∥ Z t n Π − Z 0 ∥ 2 , ∞ = Δ t n ≥ c 0 t n
for some universal constant c 0 > 0 c_0>0 c 0 > 0 .
Following the convention
TV ( P , Q ) : = sup A ∣ P ( A ) − Q ( A ) ∣ = 1 2 ∥ P − Q ∥ 1 , \operatorname{TV}(P,Q):=\sup_A|P(A)-Q(A)|=\frac{1}{2}\|P-Q\|_1, TV ( P , Q ) := A sup ∣ P ( A ) − Q ( A ) ∣ = 2 1 ∥ P − Q ∥ 1 ,
the affinity appearing in Yu [1997, Lemma 1] satisfies ∥ P ∧ Q ∥ = 1 − TV ( P , Q ) \|P\wedge Q\|=1-\operatorname{TV}(P,Q) ∥ P ∧ Q ∥ = 1 − TV ( P , Q ) . Applying Le Cam’s method Yu [1997, Lemma 1] gives, for every estimator Z ^ \widehat{\mathbf{Z}} Z ,
max ϑ ∈ { ϑ 0 , ϑ t n } E ϑ [ d ( Z ^ , Z ϑ ) ] ≥ Δ t n 2 { 1 − TV ( P t n , P 0 ) } . \max_{\vartheta\in\{\vartheta_0,\vartheta_{t_n}\}}\mathbb{E}_{\vartheta}\left[d(\widehat{\mathbf{Z}},\mathbf{Z}_{\vartheta})\right]\ge\frac{\Delta_{t_n}}{2}\left\{1-\operatorname{TV}(\mathbb{P}_{t_n},\mathbb{P}_0)\right\}. ϑ ∈ { ϑ 0 , ϑ t n } max E ϑ [ d ( Z , Z ϑ ) ] ≥ 2 Δ t n { 1 − TV ( P t n , P 0 ) } .
Using the total-variation bound (G.4),
max ϑ ∈ { ϑ 0 , ϑ t n } E ϑ [ d ( Z ^ , Z ϑ ) ] ≥ 3 8 Δ t n ≳ 1 n ρ n . \max_{\vartheta\in\{\vartheta_0,\vartheta_{t_n}\}} \mathbb{E}_{\vartheta}\left[d(\widehat{Z},Z_{\vartheta})\right] \geq\frac{3}{8}\Delta_{t_n} \gtrsim\frac{1}{\sqrt{n\rho_n}}. ϑ ∈ { ϑ 0 , ϑ t n } max E ϑ [ d ( Z , Z ϑ ) ] ≥ 8 3 Δ t n ≳ n ρ n 1 .
Since { ϑ 0 , ϑ t n } ⊂ F n , 2 , L \{\vartheta_0,\vartheta_{t_n}\}\subset\mathcal{F}_{n,2,L} { ϑ 0 , ϑ t n } ⊂ F n , 2 , L , taking the supremum over F n , 2 , L \mathcal{F}_{n,2,L} F n , 2 , L and then the infimum over all estimators completes the proof when K = 2 K=2 K = 2 .
For K > 2 K>2 K > 2 , construct a membership matrix Z 0 \mathbf{Z}_0 Z 0 with K − 1 K-1 K − 1 mixed nodes indexed by { i 1 , ⋯ , i K − 1 } \{i_1,\cdots,i_{K-1}\} { i 1 , ⋯ , i K − 1 } such that ( Z 0 ) i k , ⋅ : = 1 2 e k + 1 2 e k + 1 (\mathbf{Z}_0)_{i_k,\cdot}:=\frac{1}{2}\mathbf{e}_k+\frac{1}{2}\mathbf{e}_{k+1} ( Z 0 ) i k , ⋅ := 2 1 e k + 2 1 e k + 1 and evenly distributed pure nodes. Then one can extend the same proof of K = 2 K=2 K = 2 with the same ( Θ 0 ( l ) , Z 0 , B 0 ( l ) ) l = 1 L (\mathbf{\Theta}_0^{(l)},\mathbf{Z}_0,\mathbf{B}_0^{(l)})_{l=1}^L ( Θ 0 ( l ) , Z 0 , B 0 ( l ) ) l = 1 L and ( Θ t ( l ) , Z t , B t ( l ) ) l = 1 L (\mathbf{\Theta}_t^{(l)},\mathbf{Z}_t,\mathbf{B}_t^{(l)})_{l=1}^L ( Θ t ( l ) , Z t , B t ( l ) ) l = 1 L .
Supplementary Details for the London Bike Data
In this section, we provide supplementary details for the London bike data analysis in § 5 . Specifically, we summarize the compared models and methods in data analysis. Next, we introduce the cross-validation procedure for the model evaluation.
Overview of Compared Models and Methods
In data analysis, we compare 4 models: DCBM, DCMM, ML-DCBM, ML-DCMM. Comparing non-overlapping and mixed-membership models in both single-layer and multilayer settings helps illustrate the effect of allowing mixed memberships. On the other hand, comparing single-layer and multilayer models under the same membership structure explains the difference between jointly leveraging layer-specific information and aggregating the layers into a single network L − 1 ∑ l = 1 L A ( l ) L^{-1}\sum_{l=1}^{L}\mathbf{A}^{(l)} L − 1 ∑ l = 1 L A ( l ) . We summarize the competing models and methods used in data analysis by the following table H.1.
Table H.1. Baseline and ablation methods organized by model class, layer structure, and membership structure.
Cross-validation
The cross-validation procedure used in the London bike-sharing network analysis requires reconstructing the layer-specific probability matrices beyond estimating the shared membership matrix. In this section, we describe the reconstruction procedure and evaluate its numerical accuracy at the population level. In particular, we show that the proposed procedure achieves small reconstruction errors even when the estimated membership matrix does not exactly recover the true memberships.
Cross-validation Procedure
Following the node-pair cross-validation scheme of Li et al. [ [37 ] , we use a 10-fold cross-validation procedure adapted to multilayer networks. Let D : = { ( i , j ) : 1 ≤ i < j ≤ n } \mathcal{D}:=\{(i,j):1\leq i<j\leq n\} D := {( i , j ) : 1 ≤ i < j ≤ n } denote the set of node pairs in the upper triangle, and randomly partition D \mathcal{D} D into 10 folds D 1 , … , D 10 \mathcal{D}_1,\ldots,\mathcal{D}_{10} D 1 , … , D 10 . For fold q q q , let Ω q = D ∖ D q \Omega_q=\mathcal{D}\setminus\mathcal{D}_q Ω q = D ∖ D q be the training node-pair set and let p q p_q p q be the training proportion. We write P Ω q \mathcal{P}_{\Omega_q} P Ω q for the masking operator that keeps entries in Ω q \Omega_q Ω q and sets the held-out entries to zero, symmetrically. For a symmetric matrix Q \mathbf{Q} Q , let S K ( Q ) S_K(\mathbf{Q}) S K ( Q ) denote its rank-K K K truncated spectral approximation.
For each fold q q q and candidate K K K , we construct both the layerwise completed training matrices
A ~ − q , K ( l ) : = S K ( 1 p q P Ω q A ( l ) ) , l = 1 , … , L , \widetilde{\mathbf{A}}_{-q,K}^{(l)} := S_K\left(\frac{1}{p_q}\mathcal{P}_{\Omega_q}\mathbf{A}^{(l)}\right), \qquad l=1,\ldots,L, A − q , K ( l ) := S K ( p q 1 P Ω q A ( l ) ) , l = 1 , … , L ,
and the corresponding averaged completed training matrix
A ~ − q , K a v g : = S K ( 1 L ∑ l = 1 L 1 p q P Ω q A ( l ) ) . \widetilde{\mathbf{A}}_{-q,K}^{\mathrm{avg}} := S_K\left(\frac{1}{L}\sum_{l=1}^{L}\frac{1}{p_q}\mathcal{P}_{\Omega_q}\mathbf{A}^{(l)}\right). A − q , K avg := S K ( L 1 l = 1 ∑ L p q 1 P Ω q A ( l ) ) .
Let Z ^ − q , K C N A S E \widehat{\mathbf{Z}}_{-q,K}^{\mathrm{CNASE}} Z − q , K CNASE denote the shared membership estimate produced by the proposed CNASE estimator, and let Z ^ − q , K M S \widehat{\mathbf{Z}}_{-q,K}^{\mathrm{MS}} Z − q , K MS denote the membership estimate obtained by applying Mixed-SCORE to A ~ − q , K a v g \widetilde{\mathbf{A}}_{-q,K}^{\mathrm{avg}} A − q , K avg . Similarly, let Z ^ − q , K D C M A S E \widehat{\mathbf{Z}}_{-q,K}^{\mathrm{DCMASE}} Z − q , K DCMASE and Z ^ − q , K S C O R E \widehat{\mathbf{Z}}_{-q,K}^{\mathrm{SCORE}} Z − q , K SCORE denote the membership estimates obtained from DCMASE and SCORE, respectively. The Mixed-SCORE (Avg) method uses Z ^ − q , K M S \widehat{\mathbf{Z}}_{-q,K}^{\mathrm{MS}} Z − q , K MS as its membership estimate.
For a training matrix Q \mathbf{Q} Q and a mixed-membership estimate Z ^ \widehat{\mathbf{Z}} Z , let R m i x ( Q , Z ^ ) \mathcal{R}_{\mathrm{mix}}(\mathbf{Q},\widehat{\mathbf{Z}}) R mix ( Q , Z ) denote probability matrix reconstruction process of mixed-membership models. Similarly, for a non-overlapping membership estimate Z ^ \widehat{\mathbf{Z}} Z , let R h a r d ( Q , Z ^ ) \mathcal{R}_{\mathrm{hard}}(\mathbf{Q},\widehat{\mathbf{Z}}) R hard ( Q , Z ) denote the standard DCBM plug-in reconstruction based on the corresponding binary membership matrix and closed-form estimates of the degree-correction and block parameters. Then the raw prediction matrices are defined as follows:
P ~ M L , − q , K ( l ) = R m i x ( A ~ − q , K ( l ) , Z ^ − q , K C N A S E ) , l = 1 , … , L , P ~ M S , − q , K ( l ) = R m i x ( A ~ − q , K a v g , Z ^ − q , K M S ) , l = 1 , … , L , P ~ D C M A S E , − q , K ( l ) = R h a r d ( A ~ − q , K ( l ) , Z ^ − q , K D C M A S E ) , l = 1 , … , L , P ~ S C O R E , − q , K ( l ) = R h a r d ( A ~ − q , K a v g , Z ^ − q , K S C O R E ) , l = 1 , … , L . \begin{aligned}
\widetilde{\mathbf{P}}_{\mathrm{ML},-q,K}^{(l)} &= \mathcal{R}_{\mathrm{mix}}\left(\widetilde{\mathbf{A}}_{-q,K}^{(l)},\widehat{\mathbf{Z}}_{-q,K}^{\mathrm{CNASE}}\right), && l=1,\ldots,L,\\
\widetilde{\mathbf{P}}_{\mathrm{MS},-q,K}^{(l)} &= \mathcal{R}_{\mathrm{mix}}\left(\widetilde{\mathbf{A}}_{-q,K}^{\mathrm{avg}},\widehat{\mathbf{Z}}_{-q,K}^{\mathrm{MS}}\right), && l=1,\ldots,L,\\
\widetilde{\mathbf{P}}_{\mathrm{DCMASE},-q,K}^{(l)} &= \mathcal{R}_{\mathrm{hard}}\left(\widetilde{\mathbf{A}}_{-q,K}^{(l)},\widehat{\mathbf{Z}}_{-q,K}^{\mathrm{DCMASE}}\right), && l=1,\ldots,L,\\
\widetilde{\mathbf{P}}_{\mathrm{SCORE},-q,K}^{(l)} &= \mathcal{R}_{\mathrm{hard}}\left(\widetilde{\mathbf{A}}_{-q,K}^{\mathrm{avg}},\widehat{\mathbf{Z}}_{-q,K}^{\mathrm{SCORE}}\right), && l=1,\ldots,L.
\end{aligned} P ML , − q , K ( l ) P MS , − q , K ( l ) P DCMASE , − q , K ( l ) P SCORE , − q , K ( l ) = R mix ( A − q , K ( l ) , Z − q , K CNASE ) , = R mix ( A − q , K avg , Z − q , K MS ) , = R hard ( A − q , K ( l ) , Z − q , K DCMASE ) , = R hard ( A − q , K avg , Z − q , K SCORE ) , l = 1 , … , L , l = 1 , … , L , l = 1 , … , L , l = 1 , … , L .
For prediction, we use
P ^ m , − q , K ( l ) = Π [ 0 , 1 ] ( P ~ m , − q , K ( l ) ) , \widehat{\mathbf{P}}_{m,-q,K}^{(l)}=\Pi_{[0,1]}\left(\widetilde{\mathbf{P}}_{m,-q,K}^{(l)}\right), P m , − q , K ( l ) = Π [ 0 , 1 ] ( P m , − q , K ( l ) ) ,
where Π [ 0 , 1 ] \Pi_{[0,1]} Π [ 0 , 1 ] denotes entrywise projection onto [ 0 , 1 ] [0,1] [ 0 , 1 ] with zero diagonal. This projection is used only for prediction and cross-validation. All procedures are evaluated on the same held-out node pairs. For procedure m m m , fold q q q , and candidate K K K , the held-out prediction error is
C V m , q ( K ) = 1 L ∣ D q ∣ ∑ l = 1 L ∑ ( i , j ) ∈ D q ( A i j ( l ) − P ^ m , − q , K , i j ( l ) ) 2 . \mathrm{CV}_{m,q}(K)=\frac{1}{L|\mathcal{D}_q|}\sum_{l=1}^{L}\sum_{(i,j)\in\mathcal{D}_q}\left(A_{ij}^{(l)}-\widehat{P}_{m,-q,K,ij}^{(l)}\right)^2. CV m , q ( K ) = L ∣ D q ∣ 1 l = 1 ∑ L ( i , j ) ∈ D q ∑ ( A ij ( l ) − P m , − q , K , ij ( l ) ) 2 .
We then compute
C V m ( K ) = 1 10 ∑ q = 1 10 C V m , q ( K ) , S E m ( K ) = sd { C V m , q ( K ) : q = 1 , … , 10 } 10 . \mathrm{CV}_m(K)=\frac{1}{10}\sum_{q=1}^{10}\mathrm{CV}_{m,q}(K), \qquad\mathrm{SE}_m(K)=\frac{\operatorname{sd}\{\mathrm{CV}_{m,q}(K):q=1,\ldots,10\}}{\sqrt{10}}. CV m ( K ) = 10 1 q = 1 ∑ 10 CV m , q ( K ) , SE m ( K ) = 10 sd { CV m , q ( K ) : q = 1 , … , 10 } .
For each procedure, K K K is selected by the one-standard-error rule: among the values of K K K whose cross-validation error is within one standard error of the minimum, we choose the smallest K K K .
Probability Matrix Reconstruction
Recall that the probability matrix can be written as P ( l ) = X ( l ) I p ( l ) , q ( l ) X ( l ) ⊤ \mathbf{P}^{(l)}=\mathbf{X}^{(l)}\mathbf{I}_{p^{(l)},q^{(l)}}\mathbf{X}^{(l)\top} P ( l ) = X ( l ) I p ( l ) , q ( l ) X ( l ) ⊤ , where X ( l ) = Θ ( l ) Z R ( l ) \mathbf{X}^{(l)}=\boldsymbol{\Theta}^{(l)}\mathbf{Z}\mathbf{R}^{(l)} X ( l ) = Θ ( l ) Z R ( l ) . The main idea for reconstruction is to estimate X ( l ) \mathbf{X}^{(l)} X ( l ) by X ~ ( l ) : = Θ ^ ( l ) Z ^ R ^ ( l ) \widetilde{\mathbf{X}}^{(l)}:=\widehat{\boldsymbol{\Theta}}^{(l)}\widehat{\mathbf{Z}}\widehat{\mathbf{R}}^{(l)} X ( l ) := Θ ( l ) Z R ( l ) and reconstruct the probability matrix by P ^ ( l ) : = X ~ ( l ) I p A ( l ) , q A ( l ) X ~ ( l ) ⊤ \widehat{\mathbf{P}}^{(l)} := \widetilde{\mathbf{X}}^{(l)}\mathbf{I}_{p_A^{(l)},q_A^{(l)}}\widetilde{\mathbf{X}}^{(l)\top} P ( l ) := X ( l ) I p A ( l ) , q A ( l ) X ( l ) ⊤ , where p A ( l ) p_A^{(l)} p A ( l ) and q A ( l ) q_A^{(l)} q A ( l ) is the number of positive and negative eigenvalues among largest K K K eigenvalues of A ( l ) \mathbf{A}^{(l)} A ( l ) in magnitude. Given the estimated membership matrix Z ^ \widehat{\mathbf{Z}} Z , we estimate the layer-specific degree matrices { Θ ( l ) } l = 1 L \{\boldsymbol{\Theta}^{(l)}\}_{l=1}^{L} { Θ ( l ) } l = 1 L and transformation matrices { R ( l ) } l = 1 L \{\mathbf{R}^{(l)}\}_{l=1}^{L} { R ( l ) } l = 1 L by minimizing the following least-squares objective:
L = ∑ l = 1 L ∥ X ^ ( l ) − Θ ( l ) Z ^ R ( l ) ∥ F 2 \mathcal{L} = \sum_{l=1}^{L} \left\|\widehat{\mathbf{X}}^{(l)}-\boldsymbol{\Theta}^{(l)}\widehat{\mathbf{Z}}\mathbf{R}^{(l)}\right\|_{F}^{2} L = l = 1 ∑ L X ( l ) − Θ ( l ) Z R ( l ) F 2
where X ^ ( l ) \widehat{\mathbf{X}}^{(l)} X ( l ) is the adjacency spectral embedding of A ( l ) \mathbf{A}^{(l)} A ( l ) . We employ an alternating least-squares (ALS) algorithm that iteratively updates R ( l ) \mathbf{R}^{(l)} R ( l ) and Θ ( l ) \boldsymbol{\Theta}^{(l)} Θ ( l ) , in that order, while holding the other parameter fixed at each update. The procedure is summarized in Algorithm 3.
Algorithm 3 Probability matrix reconstruction
Input: Adjacency matrices { A ( 1 ) , … , A ( L ) } \{\mathbf{A}^{(1)},\ldots,\mathbf{A}^{(L)}\} { A ( 1 ) , … , A ( L ) } , estimated membership matrix Z ^ \widehat{\mathbf{Z}} Z , maximum number of iterations T max T_{\max} T m a x , convergence criteria ϵ \epsilon ϵ
Ensure: Reconstructed probability matrices { P ^ ( 1 ) , … , P ^ ( L ) } \{\widehat{\mathbf{P}}^{(1)},\ldots,\widehat{\mathbf{P}}^{(L)}\} { P ( 1 ) , … , P ( L ) }
For l = 1 , … , L l=1,\ldots,L l = 1 , … , L , obtain the adjacency spectral embedding X ^ ( l ) \widehat{\mathbf{X}}^{(l)} X ( l ) .
For the initial points, set Θ ^ 0 ( l ) = I n = I n \widehat{\boldsymbol{\Theta}}_{0}^{(l)}=\mathbf{I}_{n}=\mathbf{I}_{n} Θ 0 ( l ) = I n = I n for all l ∈ [ L ] l\in[L] l ∈ [ L ] .
for t = 1 t=1 t = 1 to T max T_{\max} T m a x do
Perform one ALS iteration to update { R ^ ( l ) } l = 1 L \{\widehat{\mathbf{R}}^{(l)}\}_{l=1}^{L} { R ( l ) } l = 1 L and { Θ ^ ( l ) } l = 1 L \{\widehat{\boldsymbol{\Theta}}^{(l)}\}_{l=1}^{L} { Θ ( l ) } l = 1 L .
if the convergence criterion is satisfied then
break
end if
end for
Reconstruct P ~ ( l ) : = X ~ ( l ) I p A ( l ) , q A ( l ) X ~ ( l ) ⊤ \widetilde{\mathbf{P}}^{(l)}:=\widetilde{\mathbf{X}}^{(l)}\mathbf{I}_{p_A^{(l)},q_A^{(l)}}\widetilde{\mathbf{X}}^{(l)\top} P ( l ) := X ( l ) I p A ( l ) , q A ( l ) X ( l ) ⊤ , where X ~ ( l ) : = Θ ^ ( l ) Z ^ R ^ ( l ) \widetilde{\mathbf{X}}^{(l)}:=\widehat{\boldsymbol{\Theta}}^{(l)}\widehat{\mathbf{Z}}\widehat{\mathbf{R}}^{(l)} X ( l ) := Θ ( l ) Z R ( l ) for all l ∈ [ L ] l\in[L] l ∈ [ L ] .
Obtain P ^ ( l ) : = Π [ 0 , 1 ] ( P ~ ( l ) ) \widehat{\mathbf{P}}^{(l)}:=\Pi_{[0,1]}(\widetilde{\mathbf{P}}^{(l)}) P ( l ) := Π [ 0 , 1 ] ( P ( l ) ) , where Π [ 0 , 1 ] \Pi_{[0,1]} Π [ 0 , 1 ] denotes entrywise projection onto [ 0 , 1 ] [0,1] [ 0 , 1 ] with zero diagonal.
Simulation on the Population-Level Reconstruction
As an additional diagnostic, we apply the proposed estimation and reconstruction procedure directly to the population probability matrices rather than to sampled adjacency matrices. This experiment aims to assess the numerical accuracy of the full pipeline in the absence of sampling noise. In particular, we examine whether the procedure can accurately estimate the shared membership matrix and reconstruct the layer-specific probability matrices at the population level. In particular, it checks whether the proposed pipeline can recover the membership matrix and reconstruct the population matrices when sampling noise is absent.
We use the same six data-generating settings as in Experiment 2, with
( n , n 0 , K , L ) = ( 300 , 30 , 3 , 10 ) . (n,n_0,K,L)=(300,30,3,10). ( n , n 0 , K , L ) = ( 300 , 30 , 3 , 10 ) .
For each target density
ρ ∈ { 0.04 , 0.06 , 0.08 , 0.10 , 0.12 , 0.14 , 0.16 } , \rho\in\{0.04,0.06,0.08,0.10,0.12,0.14,0.16\}, ρ ∈ { 0.04 , 0.06 , 0.08 , 0.10 , 0.12 , 0.14 , 0.16 } ,
we apply the proposed method to the diagonal-included population matrices
P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) , 1 ≤ l ≤ L . \mathbf{P}^{(l)}=\boldsymbol{\Theta}^{(l)}\mathbf{Z}\mathbf{B}^{(l)}\mathbf{Z}^{\top}\boldsymbol{\Theta}^{(l)}, \qquad1\leq l\leq L. P ( l ) = Θ ( l ) Z B ( l ) Z ⊤ Θ ( l ) , 1 ≤ l ≤ L .
We report the maximum, over the target densities, of the average reconstruction errors across 100 repetitions and layers:
E Z : = max ρ 1 100 ∑ s = 1 100 ∥ Z ρ , s − Z ^ ρ , s ∥ F 2 n E_Z := \max_{\rho} \frac{1}{100} \sum_{s=1}^{100} \frac{\left\| \mathbf{Z}_{\rho,s} - \widehat{\mathbf{Z}}_{\rho,s} \right\|_{F}^{2}}{n} E Z := ρ max 100 1 s = 1 ∑ 100 n Z ρ , s − Z ρ , s F 2
E P : = max ρ 1 100 ⋅ 10 ∑ s = 1 100 ∑ l = 1 10 ∥ P ρ , s ( l ) − P ^ ρ , s ( l ) ∥ F 2 n 2 E_P := \max_{\rho} \frac{1}{100 \cdot10} \sum_{s=1}^{100} \sum_{l=1}^{10} \frac{\left\| \mathbf{P}_{\rho,s}^{(l)} - \widehat{\mathbf{P}}_{\rho,s}^{(l)} \right\|_{F}^{2}}{n^{2}} E P := ρ max 100 ⋅ 10 1 s = 1 ∑ 100 l = 1 ∑ 10 n 2 P ρ , s ( l ) − P ρ , s ( l ) F 2
E P , r e l : = max ρ 1 100 ⋅ 10 ∑ s = 1 100 ∑ l = 1 10 ∥ P ρ , s ( l ) − P ^ ρ , s ( l ) ∥ F ∥ P ρ , s ( l ) ∥ F E_{P,\mathrm{rel}} := \max_{\rho} \frac{1}{100 \cdot10} \sum_{s=1}^{100} \sum_{l=1}^{10} \frac{\left\| \mathbf{P}_{\rho,s}^{(l)} - \widehat{\mathbf{P}}_{\rho,s}^{(l)} \right\|_{F}}{\left\| \mathbf{P}_{\rho,s}^{(l)} \right\|_{F}} E P , rel := ρ max 100 ⋅ 10 1 s = 1 ∑ 100 l = 1 ∑ 10 P ρ , s ( l ) F P ρ , s ( l ) − P ρ , s ( l ) F
E B , r e l : = max ρ 1 100 ⋅ 10 ∑ s = 1 100 ∑ l = 1 10 ∥ B ρ , s ( l ) − B ^ ρ , s ( l ) ∥ F ∥ B ρ , s ( l ) ∥ F E_{B,\mathrm{rel}} := \max_{\rho} \frac{1}{100 \cdot10} \sum_{s=1}^{100} \sum_{l=1}^{10} \frac{\left\| \mathbf{B}_{\rho,s}^{(l)} - \widehat{\mathbf{B}}_{\rho,s}^{(l)} \right\|_{F}}{\left\| \mathbf{B}_{\rho,s}^{(l)} \right\|_{F}} E B , rel := ρ max 100 ⋅ 10 1 s = 1 ∑ 100 l = 1 ∑ 10 B ρ , s ( l ) F B ρ , s ( l ) − B ρ , s ( l ) F
Table H.2: Population-level reconstruction from oracle probability matrices. For each density ρ \rho ρ , the membership error is averaged over 100 repetitions. The P - and B -reconstruction errors are averaged over 100 repetitions and L = 10 L = 10 L = 10 layers. The table reports the maximum of these density-specific averages over ρ ∈ { 0.04 , 0.06 , … , 0.16 } \rho\in\{0.04, 0.06, \ldots, 0.16\} ρ ∈ { 0.04 , 0.06 , … , 0.16 } .
Table H.2.
Table H.2 demonstrates that the proposed procedure achieves highly accurate reconstruction at the population level. Across all purity levels and block-structure settings, the averaged probability matrix reconstruction error is on the order of 10 − 12 10^{-12} 1 0 − 12 or smaller after normalization by n 2 n^{2} n 2 , while the relative reconstruction error is on the order of 10 − 6 10^{-6} 1 0 − 6 . These results indicate that the proposed procedure can accurately reconstruct the population probability matrices even when the estimated membership matrix does not exactly match the true membership matrix.