Introduction
Conditional independence is a fundamental concept in statistics and provides the structural basis for a wide range of methods, including graphical models [23 ] , causal inference [38 ] , sufficient dimension reduction [11 , 25 ] , and latent-variable models [5 , 1 ] . However, most of the existing works on this problem are concerned with conditional independence between two variables given a third, and less attention has been given to conditional independence among several variables.
In latent-variable models, in particular multivariate settings are common. The dependence among several observed variables is assumed to be fully explained by a latent state Z Z Z . For observed variables X 1 , … , X d X_1,\ldots,X_d X 1 , … , X d , this assumption is expressed as
P X 1 , … , X d ∣ Z = ⨂ j = 1 d P X j ∣ Z P Z -almost surely . P_{X_1,\ldots,X_d\mid Z}=\bigotimes_{j=1}^{d}P_{X_j\mid Z}\quad P_Z\text{-almost surely}. P X 1 , … , X d ∣ Z = j = 1 ⨂ d P X j ∣ Z P Z -almost surely .
We refer to this property as multi-way conditional independence and write X 1 ⊥ ⋯ ⊥ X d ∣ Z X_1 \perp\cdots\perp X_d \mid Z X 1 ⊥ ⋯ ⊥ X d ∣ Z . This factorization is imposed as a key structural assumption in many latent-variable models. Under additional model-specific conditions, this factorization enables identification of the proportions and component distributions ([17 , 1 ] ) and supports their nonparametric or moment-based estimation ([17 , 6 , 3 ] ). It is therefore essential to test whether this condition actually holds. Residual dependence after conditioning on Z Z Z may indicate omitted latent structure, direct interactions, or model misspecification, thus undermining subsequent inference.
The question of whether a shared observed/latent state accounts for joint dependence is particularly relevant in genomics and neuroscience. Gene-expression latent-variable models represent dependence among genes through shared latent factors and observed covariates ([8 , 31 ] ). Neural population models similarly use shared latent states or common inputs to explain correlated activity ([32 ] (Yu et al. 2009)), while cortical recordings reveal higher-order correlations that persist after accounting for stimulus-dependent activity ([21 ] ). These examples motivate testing whether a given state Z Z Z fully explains joint dependence through the conditional factorization above.
Although model-specific diagnostics have been developed for certain parametric latent-class models ([16 , 36 , 30 ] ), a general nonparametric test for multi-way conditional independence has yet to be developed. In contrast, nonparametric testing of bivariate conditional independence has been extensively studied; see Li and Fan (2020) for a comprehensive review. Representative procedures are based on nonparametric Hellinger distances (Su and White, 2008), maximal nonlinear conditional correlation ([19 ] ), more general metric characterizations ([18 ] ), permutation-based reductions to two-sample test ([12 ] ), regression residuals under structural restrictions ([52 , 43 ] ), and distribution-free transformations ([7 ] ).
Among these nonparametric bivariate conditional independence tests, the operator-based approach in reproducing kernel Hilbert spaces provides the foundation for our method. Under suitable conditions on the kernels, bivariate conditional independence can be characterized by the vanishing of the conditional covariance operator ([13 , 14 ] ), which forms the basis of the kernel conditional independence test proposed by Zhang et al. [53 ] . Subsequent work has developed related operator-based methods for nonparametric graphical modeling and high-dimensional conditioning ([24 ] (Li and Kim 2024)), as well as extensions to functional data ([25 , 47 ] ). Alternative kernel formulations include kernel partial correlation and distance-based characterizations of conditional dependence ([20 , 44 ] ).
Building on this operator characterization, we extend the bivariate framework to multi-way conditional independence. The key observation is simple. For any fixed ordering of the variables, multi-way conditional independence can be decomposed into a sequence of pairwise conditional independence conditions:
X 1 ⊥ ⋯ ⊥ X d ∣ Z ⟺ X j ⊥ X { j + 1 , … , d } ∣ Z , for all j = 1 , … , d − 1. X_1 \perp\cdots\perp X_d \mid Z \quad\Longleftrightarrow\quad X_j \perp X_{\{j+1,\ldots,d\}} \mid Z,\quad\text{for all } j=1,\ldots,d-1. X 1 ⊥ ⋯ ⊥ X d ∣ Z ⟺ X j ⊥ X { j + 1 , … , d } ∣ Z , for all j = 1 , … , d − 1.
Thus, the multivariate null hypothesis can be expressed through d − 1 d-1 d − 1 bivariate conditional-independence restrictions. To construct the test statistic, we apply the conditional covariance operator characterization to each restriction and combine the resulting operators into a single omnibus test statistic of multi-way conditional independence. We further develop the asymptotic distribution of our test statistic under the null and local alternative distributions. To the best of our knowledge, this is the first general nonparametric test for multi-way conditional independence.
In particular, the multi-way unconditional independence test can be regarded as a special case by taking Z Z Z as constant, which also adds up to the existing literature. For example, joint independence based on the d d d -variable Hilbert–Schmidt independence criterion ([39 ] ) and distance-based measures ([33 , 9 ] ). We stay brief here as this unconditional degeneration is not the primary focus of the paper.
The remainder of the paper is organized as follows. § 2 reviews reproducing kernel Hilbert spaces and cross-covariance operators, which provide a framework for characterizing nonlinear dependence using linear operators. We construct the test statistics in § 3 and study their asymptotic properties in § 4 . The unconditional case then follows as a special case in § 5 . Finally, § 6 describes the numerical implementation, and § 7 presents simulation studies and applies our method to the OneK1K single-cell RNA-seq dataset.
Background
Background on Reproducing Kernel Hilbert Spaces
Let Ω X j = R p j \Omega_{X_j}=\mathbb{R}^{p_j} Ω X j = R p j for j = 1 , … , d j=1,\ldots,d j = 1 , … , d , and Ω Z = R p Z \Omega_Z=\mathbb{R}^{p_Z} Ω Z = R p Z .1 Let ( Ω , F , P ) (\Omega,\mathcal{F},\mathbb{P}) ( Ω , F , P ) be a probability space, and let X j : Ω → Ω X j X_j:\Omega\to\Omega_{X_j} X j : Ω → Ω X j , j = 1 , … , d j=1,\ldots,d j = 1 , … , d , and Z : Ω → Ω Z Z:\Omega\to\Omega_Z Z : Ω → Ω Z be random elements. For any subset J ⊂ { 1 , … , d } J \subset\{1,\ldots,d\} J ⊂ { 1 , … , d } , let X J = ( X j ) j ∈ J X_J=(X_j)_{j\in J} X J = ( X j ) j ∈ J denote the corresponding subvector. We use the pointwise product to define the kernel on the product space:
κ X J ( x J , x J ′ ) = ∏ ℓ ∈ J κ ℓ ( x ℓ , x ℓ ′ ) . \kappa_{X_J}(x_J,x'_J)=\prod_{\ell\in J}\kappa_\ell(x_\ell,x'_\ell). κ X J ( x J , x J ′ ) = ℓ ∈ J ∏ κ ℓ ( x ℓ , x ℓ ′ ) .
Let H X J \mathcal{H}_{X_J} H X J denote the reproducing kernel Hilbert space induced by κ X J \kappa_{X_J} κ X J .
We further define augmented variables X ¨ J = ( X J , Z ) \ddot{X}_J=(X_J,Z) X ¨ J = ( X J , Z ) . Here and throughout the paper, a double dot over a variable indicates that the variable has been augmented by adjoining Z Z Z . The associated kernel is then defined as
κ X ¨ J ( ( x J , z ) , ( x J ′ , z ′ ) ) = κ X J ( x J , x J ′ ) κ Z ( z , z ′ ) . \kappa_{\ddot{X}_J}((x_J,z),(x'_J,z'))=\kappa_{X_J}(x_J,x'_J)\kappa_Z(z,z'). κ X ¨ J (( x J , z ) , ( x J ′ , z ′ )) = κ X J ( x J , x J ′ ) κ Z ( z , z ′ ) .
Let H X ¨ J \mathcal{H}_{\ddot{X}_J} H X ¨ J denote the reproducing kernel Hilbert space induced by κ X ¨ J \kappa_{\ddot{X}_J} κ X ¨ J . Its inner product is also given by the product
⟨ f 1 g 1 , f 2 g 2 ⟩ H X ¨ J = ⟨ f 1 , f 2 ⟩ H X J ⟨ g 1 , g 2 ⟩ H Z . \langle f_1g_1,f_2g_2\rangle_{\mathcal{H}_{\ddot{X}_J}}=\langle f_1,f_2\rangle_{\mathcal{H}_{X_J}}\langle g_1,g_2\rangle_{\mathcal{H}_Z}. ⟨ f 1 g 1 , f 2 g 2 ⟩ H X ¨ J = ⟨ f 1 , f 2 ⟩ H X J ⟨ g 1 , g 2 ⟩ H Z .
where f 1 , f 2 ∈ H X J f_1,f_2\in\mathcal{H}_{X_J} f 1 , f 2 ∈ H X J , g 1 , g 2 ∈ H Z g_1,g_2\in\mathcal{H}_Z g 1 , g 2 ∈ H Z , and ( f g ) ( x J , z ) = f ( x J ) g ( z ) (fg)(x_J,z)=f(x_J)g(z) ( f g ) ( x J , z ) = f ( x J ) g ( z ) . This definition extends by bilinearity to finite linear combinations and then by continuity to the whole space.
Assumption 1. The kernels κ X J \kappa_{X_J} κ X J and κ Z \kappa_Z κ Z are continuous, bounded, and positive-definite. The conditions hold for all J ⊂ { 1 , … , d } J\subset\{1,\ldots,d\} J ⊂ { 1 , … , d } .
By the boundedness of κ X J \kappa_{X_J} κ X J , the expectation functional f ↦ E [ f ( X J ) ] f\mapsto\mathbb{E}[f(X_J)] f ↦ E [ f ( X J )] is a bounded linear functional on H X J \mathcal{H}_{X_J} H X J . Applying Riesz representation theorem, there exists a unique μ X J ∈ H X J \mu_{X_J}\in\mathcal{H}_{X_J} μ X J ∈ H X J such that
⟨ f , μ X J ⟩ H X J = E [ f ( X J ) ] . \langle f,\mu_{X_J}\rangle_{\mathcal{H}_{X_J}}=\mathbb{E}[f(X_J)]. ⟨ f , μ X J ⟩ H X J = E [ f ( X J )] .
We define μ X J \mu_{X_J} μ X J to be the kernel mean embedding of X J X_J X J . Similarly, we define the kernel mean embeddings μ Z ∈ H Z \mu_Z\in\mathcal{H}_Z μ Z ∈ H Z for Z Z Z and μ X ¨ J ∈ H X ¨ J \mu_{\ddot{X}_J}\in\mathcal{H}_{\ddot{X}_J} μ X ¨ J ∈ H X ¨ J for X ¨ J \ddot{X}_J X ¨ J , respectively. Then the corresponding empirical kernel mean embeddings are denoted by μ ^ X J = E n [ κ X J ( ⋅ , X J ) ] \hat{\mu}_{X_J}=\mathbb{E}_n[\kappa_{X_J}(\cdot,X_J)] μ ^ X J = E n [ κ X J ( ⋅ , X J )] , μ ^ Z = E n [ κ Z ( ⋅ , Z ) ] \hat{\mu}_Z=\mathbb{E}_n[\kappa_Z(\cdot,Z)] μ ^ Z = E n [ κ Z ( ⋅ , Z )] , and μ ^ X ¨ J = E n [ κ X ¨ J ( ⋅ , X ¨ J ) ] \hat{\mu}_{\ddot{X}_J}=\mathbb{E}_n[\kappa_{\ddot{X}_J}(\cdot,\ddot{X}_J)] μ ^ X ¨ J = E n [ κ X ¨ J ( ⋅ , X ¨ J )] , where E n \mathbb{E}_n E n stands for the sample average based on i.i.d. data { ( X 1 , i , … , X d , i , Z i ) ; i = 1 , … , n } \{(X_{1,i},\ldots,X_{d,i},Z_i); i=1,\ldots,n\} {( X 1 , i , … , X d , i , Z i ) ; i = 1 , … , n } .
We can then define the population-level and sample-level centralized kernels for κ X J ( ⋅ , X J ) \kappa_{X_J}(\cdot,X_J) κ X J ( ⋅ , X J ) as κ ~ X J ( ⋅ , X J ) = κ X J ( ⋅ , X J ) − μ X J \tilde{\kappa}_{X_J}(\cdot,X_J)=\kappa_{X_J}(\cdot,X_J)-\mu_{X_J} κ ~ X J ( ⋅ , X J ) = κ X J ( ⋅ , X J ) − μ X J and κ ^ X J = κ X J ( ⋅ , X J ) − E n [ κ X J ( ⋅ , X J ) ] \hat{\kappa}_{X_J}=\kappa_{X_J}(\cdot,X_J)-\mathbb{E}_n[\kappa_{X_J}(\cdot,X_J)] κ ^ X J = κ X J ( ⋅ , X J ) − E n [ κ X J ( ⋅ , X J )] , respectively. Similarly define κ ~ X ¨ J ( ⋅ , X ¨ J ) \tilde{\kappa}_{\ddot{X}_J}(\cdot,\ddot{X}_J) κ ~ X ¨ J ( ⋅ , X ¨ J ) and κ ^ X ¨ J ( ⋅ , X ¨ J ) \hat{\kappa}_{\ddot{X}_J}(\cdot,\ddot{X}_J) κ ^ X ¨ J ( ⋅ , X ¨ J ) , as well as κ ~ Z ( ⋅ , Z ) \tilde{\kappa}_Z(\cdot,Z) κ ~ Z ( ⋅ , Z ) and κ ^ Z ( ⋅ , Z ) \hat{\kappa}_Z(\cdot,Z) κ ^ Z ( ⋅ , Z ) . We denote the space of continuous real-valued functions on Ω \Omega Ω vanishing at infinity by C 0 ( Ω ) C_0(\Omega) C 0 ( Ω ) .
Assumption 2. Assume the following two conditions hold, for any J ⊂ { 1 , … , d } J\subset\{1,\ldots,d\} J ⊂ { 1 , … , d } :
E [ f ( X J ) ∣ Z = ⋅ ] ∈ H Z \mathbb{E}[f(X_J)\mid Z=\cdot]\in\mathcal{H}_Z E [ f ( X J ) ∣ Z = ⋅ ] ∈ H Z for all f ∈ H X J f\in\mathcal{H}_{X_J} f ∈ H X J , and E [ g ( X ¨ J ) ∣ Z = ⋅ ] ∈ H Z \mathbb{E}[g(\ddot{X}_J)\mid Z=\cdot]\in\mathcal{H}_Z E [ g ( X ¨ J ) ∣ Z = ⋅ ] ∈ H Z for all g ∈ H X ¨ J g\in\mathcal{H}_{\ddot{X}_J} g ∈ H X ¨ J ;
κ X J \kappa_{X_J} κ X J and κ Z \kappa_Z κ Z are c 0 c_0 c 0 -universal, i.e., H X J \mathcal{H}_{X_J} H X J and H Z \mathcal{H}_Z H Z are dense in C 0 ( Ω X J ) C_0(\Omega_{X_J}) C 0 ( Ω X J ) and C 0 ( Ω Z ) C_0(\Omega_Z) C 0 ( Ω Z ) under the supremum norm. Then H X ¨ J \mathcal{H}_{\ddot{X}_J} H X ¨ J is dense in C 0 ( Ω X J × Ω Z ) C_0(\Omega_{X_J}\times\Omega_Z) C 0 ( Ω X J × Ω Z ) .
Cross-Covariance Operators and Conditional Independence
We adopt the following convention for defining cross-covariance operators. As a representative example, consider the pair ( X J , Z ) (X_J, Z) ( X J , Z ) for some J ⊂ { 1 , … , d } J \subset\{1,\ldots,d\} J ⊂ { 1 , … , d } . The cross-covariance operator from H Z \mathcal{H}_Z H Z to H X J \mathcal{H}_{X_J} H X J is denoted by Σ X J Z : H Z → H X J \Sigma_{X_J Z}: \mathcal{H}_Z \to\mathcal{H}_{X_J} Σ X J Z : H Z → H X J and is defined through the identity
⟨ f , Σ X J Z g ⟩ H X J = E [ f ( X J ) g ( Z ) ] − E [ f ( X J ) ] E [ g ( Z ) ] , \langle f,\Sigma_{X_J Z}g\rangle_{\mathcal{H}_{X_J}} = \mathbb{E}[f(X_J)g(Z)]-\mathbb{E}[f(X_J)]\mathbb{E}[g(Z)], ⟨ f , Σ X J Z g ⟩ H X J = E [ f ( X J ) g ( Z )] − E [ f ( X J )] E [ g ( Z )] ,
for all f ∈ H X J f \in\mathcal{H}_{X_J} f ∈ H X J and g ∈ H Z g \in\mathcal{H}_Z g ∈ H Z , see Conway [10 ] and Li [25 ] . By Riesz representation theorem, Σ X J Z \Sigma_{X_J Z} Σ X J Z can be equivalently represented as
Σ X J Z = E [ κ ~ X J ( ⋅ , X ) ⊗ κ ~ Z ( ⋅ , Z ) ] . \Sigma_{X_J Z} = \mathbb{E}\left[\widetilde{\kappa}_{X_J}(\cdot,X) \otimes\widetilde{\kappa}_Z(\cdot,Z)\right]. Σ X J Z = E [ κ X J ( ⋅ , X ) ⊗ κ Z ( ⋅ , Z ) ] .
Here f ⊗ g f \otimes g f ⊗ g stands for the linear operator defined by ( f ⊗ g ) h = ⟨ g , h ⟩ f (f \otimes g)h = \langle g,h\rangle f ( f ⊗ g ) h = ⟨ g , h ⟩ f . At the sample level, the cross-covariance operator is estimated by
Σ ^ X J Z = E n [ κ ^ X J ( ⋅ , X J ) ⊗ κ ^ Z ( ⋅ , Z ) ] . \widehat{\Sigma}_{X_J Z} = \mathbb{E}_n\left[\widehat{\kappa}_{X_J}(\cdot,X_J) \otimes\widehat{\kappa}_Z(\cdot,Z)\right]. Σ X J Z = E n [ κ X J ( ⋅ , X J ) ⊗ κ Z ( ⋅ , Z ) ] .
The cross-covariance operators can then be defined for any other pair of random elements by replacing X J X_J X J and Z Z Z with the desired ones. For example, we define
Σ X ¨ J Z = E [ κ ~ X ¨ J ( ⋅ , X ¨ J ) ⊗ κ ~ Z ( ⋅ , Z ) ] and Σ Z Z = E [ κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] , (1) \Sigma_{\ddot{X}_J Z} = \mathbb{E}\left[\widetilde{\kappa}_{\ddot{X}_J}(\cdot,\ddot{X}_J) \otimes\widetilde{\kappa}_Z(\cdot,Z)\right]
\quad\text{and}\quad
\Sigma_{ZZ} = \mathbb{E}\left[\widetilde{\kappa}_Z(\cdot,Z) \otimes\widetilde{\kappa}_Z(\cdot,Z)\right], \tag*{(1)} Σ X ¨ J Z = E [ κ X ¨ J ( ⋅ , X ¨ J ) ⊗ κ Z ( ⋅ , Z ) ] and Σ Z Z = E [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] , (1)
and the corresponding sample estimators
Σ ^ X ¨ J Z = E n [ κ ^ X ¨ J ( ⋅ , X ¨ J ) ⊗ κ ^ Z ( ⋅ , Z ) ] and Σ ^ Z Z = E n [ κ ^ Z ( ⋅ , Z ) ⊗ κ ^ Z ( ⋅ , Z ) ] . \widehat{\Sigma}_{\ddot{X}_J Z} = \mathbb{E}_n\left[\widehat{\kappa}_{\ddot{X}_J}(\cdot,\ddot{X}_J) \otimes\widehat{\kappa}_Z(\cdot,Z)\right]
\quad\text{and}\quad
\widehat{\Sigma}_{ZZ} = \mathbb{E}_n\left[\widehat{\kappa}_Z(\cdot,Z) \otimes\widehat{\kappa}_Z(\cdot,Z)\right]. Σ X ¨ J Z = E n [ κ X ¨ J ( ⋅ , X ¨ J ) ⊗ κ Z ( ⋅ , Z ) ] and Σ Z Z = E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] .
In particular, we give the equivalent representation of Σ X ¨ J Z \Sigma_{\ddot{X}_J Z} Σ X ¨ J Z in (1) as
Σ X ¨ J Z = E [ ( κ X J ( ⋅ , X ) κ Z ( ⋅ , Z ) ) ⊗ κ ~ Z ( ⋅ , Z ) ] , \Sigma_{\ddot{X}_J Z} = \mathbb{E}\left[(\kappa_{X_J}(\cdot,X)\kappa_Z(\cdot,Z)) \otimes\widetilde{\kappa}_Z(\cdot,Z)\right], Σ X ¨ J Z = E [ ( κ X J ( ⋅ , X ) κ Z ( ⋅ , Z )) ⊗ κ Z ( ⋅ , Z ) ] ,
which will be repeatedly used in later sections.
Define the Moore–Penrose inverse Σ Z Z † \Sigma_{ZZ}^{\dagger} Σ Z Z † on dom ( Σ Z Z † ) = ran ( Σ Z Z ) ⊕ ker ( Σ Z Z ) \operatorname{dom}(\Sigma_{ZZ}^{\dagger}) = \operatorname{ran}(\Sigma_{ZZ}) \oplus\ker(\Sigma_{ZZ}) dom ( Σ Z Z † ) = ran ( Σ Z Z ) ⊕ ker ( Σ Z Z ) as follows. For g ∈ ran ( Σ Z Z ) g \in\operatorname{ran}(\Sigma_{ZZ}) g ∈ ran ( Σ Z Z ) , let Σ Z Z † g \Sigma_{ZZ}^{\dagger}g Σ Z Z † g be the unique element u ∈ ker ( Σ Z Z ) ⊥ u \in\ker(\Sigma_{ZZ})^\perp u ∈ ker ( Σ Z Z ) ⊥ satisfying Σ Z Z u = g \Sigma_{ZZ}u = g Σ Z Z u = g . For g ∈ ker ( Σ Z Z ) g \in\ker(\Sigma_{ZZ}) g ∈ ker ( Σ Z Z ) , set Σ Z Z † g = 0 \Sigma_{ZZ}^{\dagger}g = 0 Σ Z Z † g = 0 . The definition is then extended by linearity to the entire domain.
We then define the conditional cross-covariance operator following Fukumizu et al. [13 ] . Let J 1 J_1 J 1 and J 2 J_2 J 2 be two disjoint subsets of { 1 , … , d } \{1,\ldots,d\} { 1 , … , d } , we define Σ X J 1 X J 2 ∣ Z : H X J 2 → H X J 1 \Sigma_{X_{J_1}X_{J_2}\mid Z}: \mathcal{H}_{X_{J_2}} \to\mathcal{H}_{X_{J_1}} Σ X J 1 X J 2 ∣ Z : H X J 2 → H X J 1 as
Σ X J 1 X J 2 ∣ Z = Σ X J 1 X J 2 − Σ X J 1 Z Σ Z Z † Σ Z X J 2 . \Sigma_{X_{J_1}X_{J_2}\mid Z}
=
\Sigma_{X_{J_1}X_{J_2}}
-
\Sigma_{X_{J_1}Z}\Sigma_{ZZ}^{\dagger}\Sigma_{ZX_{J_2}}. Σ X J 1 X J 2 ∣ Z = Σ X J 1 X J 2 − Σ X J 1 Z Σ Z Z † Σ Z X J 2 .
Similarly, we define the conditional cross-covariance operator for the augmented variable:
Σ X ¨ J 1 X J 2 ∣ Z = Σ X ¨ J 1 X J 2 − Σ X ¨ J 1 Z Σ Z Z † Σ Z X J 2 . \Sigma_{\ddot{X}_{J_1}X_{J_2}\mid Z}
=
\Sigma_{\ddot{X}_{J_1}X_{J_2}}
-
\Sigma_{\ddot{X}_{J_1}Z}\Sigma_{ZZ}^{\dagger}\Sigma_{ZX_{J_2}}. Σ X ¨ J 1 X J 2 ∣ Z = Σ X ¨ J 1 X J 2 − Σ X ¨ J 1 Z Σ Z Z † Σ Z X J 2 .
Construction of Test Statistic
Regression Operator
We first express the conditional cross-covariance operators as cross-covariances of reproducing kernel Hilbert space regression residuals. These residuals are obtained by removing the components of the feature maps explained by Z Z Z . After mapping the variables into reproducing kernel Hilbert spaces, we transform nonlinear regression in the original variables into linear regression between their feature maps. To formalize this representation, we introduce the linear operators B X J ∣ Z B_{X_J\mid Z} B X J ∣ Z and B X ¨ J ∣ Z B_{\ddot{X}_J\mid Z} B X ¨ J ∣ Z , which are the adjoints of the reproducing kernel Hilbert space regression operators considered in Lee et al. [24 ] and Sang and Li [42 ] .
Definition 1. For any J ⊂ { 1 , … , d } J \subset\{1,\ldots,d\} J ⊂ { 1 , … , d } , we define linear operators B X J ∣ Z : H Z → H X J B_{X_J\mid Z}:\mathcal{H}_Z \to\mathcal{H}_{X_J} B X J ∣ Z : H Z → H X J and B X ¨ J ∣ Z : H Z → H X ¨ J B_{\ddot{X}_J\mid Z}:\mathcal{H}_Z \to\mathcal{H}_{\ddot{X}_J} B X ¨ J ∣ Z : H Z → H X ¨ J , respectively, as
B X J ∣ Z = Σ X J Z Σ Z Z † , B X ¨ J ∣ Z = Σ X ¨ J Z Σ Z Z † . B_{X_J\mid Z} = \Sigma_{X_J Z}\Sigma_{ZZ}^{\dagger}, \qquad B_{\ddot{X}_J\mid Z} = \Sigma_{\ddot{X}_J Z}\Sigma_{ZZ}^{\dagger}. B X J ∣ Z = Σ X J Z Σ Z Z † , B X ¨ J ∣ Z = Σ X ¨ J Z Σ Z Z † .
Note that both operators in Definition 1 should first be defined on dom ( Σ Z Z † ) = ran ( Σ Z Z ) ⊕ ker ( Σ Z Z ) \operatorname{dom}(\Sigma_{ZZ}^{\dagger})=\operatorname{ran}(\Sigma_{ZZ})\oplus\operatorname{ker}(\Sigma_{ZZ}) dom ( Σ Z Z † ) = ran ( Σ Z Z ) ⊕ ker ( Σ Z Z ) . Under Assumption 2, both operators are bounded, so their domains can be uniquely extended to the whole space H Z \mathcal{H}_Z H Z . Under Assumptions 1 and 2, Corollary 1 of Tang and Li [47 ] shows that these operators represent the corresponding conditional mean embeddings:
E [ κ ~ X J ( ⋅ , X J ) ∣ Z ] = B X J ∣ Z κ ~ Z ( ⋅ , Z ) , E [ κ ~ X ¨ J ( ⋅ , X ¨ J ) ∣ Z ] = B X ¨ J ∣ Z κ ~ Z ( ⋅ , Z ) , \mathbb{E}[\widetilde{\kappa}_{X_J}(\cdot,X_J)\mid Z]=B_{X_J\mid Z}\widetilde{\kappa}_Z(\cdot,Z), \qquad\mathbb{E}[\widetilde{\kappa}_{\ddot{X}_J}(\cdot,\ddot{X}_J)\mid Z]=B_{\ddot{X}_J\mid Z}\widetilde{\kappa}_Z(\cdot,Z), E [ κ X J ( ⋅ , X J ) ∣ Z ] = B X J ∣ Z κ Z ( ⋅ , Z ) , E [ κ X ¨ J ( ⋅ , X ¨ J ) ∣ Z ] = B X ¨ J ∣ Z κ Z ( ⋅ , Z ) ,
which hold almost surely. Accordingly, we can define the corresponding feature-space regression residuals by
ξ X J ∣ Z ( ⋅ , X J ; Z ) = κ ~ X J ( ⋅ , X J ) − B X J ∣ Z κ ~ Z ( ⋅ , Z ) , ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) = κ ~ X ¨ J ( ⋅ , X ¨ J ) − B X ¨ J ∣ Z κ ~ Z ( ⋅ , Z ) . \xi_{X_J\mid Z}(\cdot,X_J;Z)=\widetilde{\kappa}_{X_J}(\cdot,X_J)-B_{X_J\mid Z}\widetilde{\kappa}_Z(\cdot,Z), \qquad\xi_{\ddot{X}_J\mid Z}(\cdot,\ddot{X}_J;Z)=\widetilde{\kappa}_{\ddot{X}_J}(\cdot,\ddot{X}_J)-B_{\ddot{X}_J\mid Z}\widetilde{\kappa}_Z(\cdot,Z). ξ X J ∣ Z ( ⋅ , X J ; Z ) = κ X J ( ⋅ , X J ) − B X J ∣ Z κ Z ( ⋅ , Z ) , ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) = κ X ¨ J ( ⋅ , X ¨ J ) − B X ¨ J ∣ Z κ Z ( ⋅ , Z ) .
These are respectively H X J \mathcal{H}_{X_J} H X J -valued and H X ¨ J \mathcal{H}_{\ddot{X}_J} H X ¨ J -valued random elements, both measurable with respect to σ ( X J , Z ) \sigma(X_J,Z) σ ( X J , Z ) . By the preceding conditional mean identities, they are conditionally centered:
E [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ∣ Z ] = 0 and E [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ∣ Z ] = 0 almost surely . (2) \mathbb{E}[\xi_{X_J\mid Z}(\cdot,X_J;Z)\mid Z]=0 \quad\text{and}\quad\mathbb{E}[\xi_{\ddot{X}_J\mid Z}(\cdot,\ddot{X}_J;Z)\mid Z]=0 \quad\text{almost surely}. \tag*{(2)} E [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ∣ Z ] = 0 and E [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ∣ Z ] = 0 almost surely . (2)
For disjoint subsets J 1 J_1 J 1 and J 2 J_2 J 2 , the corresponding conditional cross-covariance operator is the cross-covariance of the residual feature maps. That is,
E [ ξ X ¨ J 1 ∣ Z ( ⋅ , X ¨ J 1 ; Z ) ⊗ ξ X J 2 ∣ Z ( ⋅ , X J 2 ; Z ) ] = E [ ( κ ~ X ¨ J 1 ( ⋅ , X ¨ J 1 ) − B X ¨ J 1 ∣ Z κ ~ Z ( ⋅ , Z ) ) ⊗ ( κ ~ X J 2 ( ⋅ , X J 2 ) − B X J 2 ∣ Z κ ~ Z ( ⋅ , Z ) ) ] = Σ X ¨ J 1 X J 2 − Σ X ¨ J 1 Z Σ Z Z † Σ Z X J 2 − Σ X ¨ J 1 Z Σ Z Z † Σ Z X J 2 + Σ X ¨ J 1 Z Σ Z Z † Σ Z Z Σ Z Z † Σ Z X J 2 = Σ X ¨ J 1 X J 2 ∣ Z . \begin{aligned}
\mathbb{E}[\xi_{\ddot{X}_{J_1}\mid Z}(\cdot,\ddot{X}_{J_1};Z)\otimes\xi_{X_{J_2}\mid Z}(\cdot,X_{J_2};Z)]
&=\mathbb{E}[(\widetilde{\kappa}_{\ddot{X}_{J_1}}(\cdot,\ddot{X}_{J_1})-B_{\ddot{X}_{J_1}\mid Z}\widetilde{\kappa}_Z(\cdot,Z))\\
&\qquad\otimes(\widetilde{\kappa}_{X_{J_2}}(\cdot,X_{J_2})-B_{X_{J_2}\mid Z}\widetilde{\kappa}_Z(\cdot,Z))]\\
&=\Sigma_{\ddot{X}_{J_1}X_{J_2}}-\Sigma_{\ddot{X}_{J_1}Z}\Sigma_{ZZ}^{\dagger}\Sigma_{ZX_{J_2}}-\Sigma_{\ddot{X}_{J_1}Z}\Sigma_{ZZ}^{\dagger}\Sigma_{ZX_{J_2}}\\
&\qquad+\Sigma_{\ddot{X}_{J_1}Z}\Sigma_{ZZ}^{\dagger}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\Sigma_{ZX_{J_2}}\\
&=\Sigma_{\ddot{X}_{J_1}X_{J_2}\mid Z}.
\end{aligned} E [ ξ X ¨ J 1 ∣ Z ( ⋅ , X ¨ J 1 ; Z ) ⊗ ξ X J 2 ∣ Z ( ⋅ , X J 2 ; Z )] = E [( κ X ¨ J 1 ( ⋅ , X ¨ J 1 ) − B X ¨ J 1 ∣ Z κ Z ( ⋅ , Z )) ⊗ ( κ X J 2 ( ⋅ , X J 2 ) − B X J 2 ∣ Z κ Z ( ⋅ , Z ))] = Σ X ¨ J 1 X J 2 − Σ X ¨ J 1 Z Σ Z Z † Σ Z X J 2 − Σ X ¨ J 1 Z Σ Z Z † Σ Z X J 2 + Σ X ¨ J 1 Z Σ Z Z † Σ Z Z Σ Z Z † Σ Z X J 2 = Σ X ¨ J 1 X J 2 ∣ Z .
The Tikhonov-regularized empirical counterparts of the regression operators are
B ^ X J ∣ Z = Σ ^ X J Z ( Σ ^ Z Z + ϵ n I ) − 1 , B ^ X ¨ J ∣ Z = Σ ^ X ¨ J Z ( Σ ^ Z Z + ϵ n I ) − 1 , \hat{B}_{X_J\mid Z}=\hat{\Sigma}_{X_JZ}(\hat{\Sigma}_{ZZ}+\epsilon_n I)^{-1},\qquad\hat{B}_{\ddot{X}_J\mid Z}=\hat{\Sigma}_{\ddot{X}_JZ}(\hat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}, B ^ X J ∣ Z = Σ ^ X J Z ( Σ ^ Z Z + ϵ n I ) − 1 , B ^ X ¨ J ∣ Z = Σ ^ X ¨ J Z ( Σ ^ Z Z + ϵ n I ) − 1 ,
where ϵ n > 0 \epsilon_n>0 ϵ n > 0 is a regularization parameter satisfying ϵ n ⟶ 0 \epsilon_n\longrightarrow0 ϵ n ⟶ 0 as n ⟶ ∞ n\longrightarrow\infty n ⟶ ∞ . Based on the residual representation, we define the empirical conditional cross-covariance operator as
Σ ^ X J 1 X J 2 ∣ Z = E n [ ( κ ^ X J 1 ( ⋅ , X J 1 ) − B ^ X J 1 ∣ Z κ ^ Z ( ⋅ , Z ) ) ⊗ ( κ ^ X J 2 ( ⋅ , X J 2 ) − B ^ X J 2 ∣ Z κ ^ Z ( ⋅ , Z ) ) ] = Σ ^ X J 1 X J 2 − B ^ X J 1 ∣ Z Σ ^ Z X J 2 − Σ ^ X J 1 Z B ^ X J 2 ∣ Z ∗ + B ^ X J 1 ∣ Z Σ ^ Z Z B ^ X J 2 ∣ Z ∗ = Σ ^ X J 1 X J 2 − Σ ^ X J 1 Z ( Σ ^ Z Z + ϵ n I ) − 1 Σ ^ Z X J 2 − ϵ n B ^ X J 1 ∣ Z B ^ X J 2 ∣ Z ∗ . \begin{aligned}
\hat{\Sigma}_{X_{J_1}X_{J_2}\mid Z}
&=\mathbb{E}_n\bigl[(\hat{\kappa}_{X_{J_1}}(\cdot,X_{J_1})-\hat{B}_{X_{J_1}\mid Z}\hat{\kappa}_Z(\cdot,Z))\otimes(\hat{\kappa}_{X_{J_2}}(\cdot,X_{J_2})-\hat{B}_{X_{J_2}\mid Z}\hat{\kappa}_Z(\cdot,Z))\bigr]\\
&=\hat{\Sigma}_{X_{J_1}X_{J_2}}-\hat{B}_{X_{J_1}\mid Z}\hat{\Sigma}_{ZX_{J_2}}-\hat{\Sigma}_{X_{J_1}Z}\hat{B}_{X_{J_2}\mid Z}^{*}+\hat{B}_{X_{J_1}\mid Z}\hat{\Sigma}_{ZZ}\hat{B}_{X_{J_2}\mid Z}^{*}\\
&=\hat{\Sigma}_{X_{J_1}X_{J_2}}-\hat{\Sigma}_{X_{J_1}Z}(\hat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\hat{\Sigma}_{ZX_{J_2}}-\epsilon_n\hat{B}_{X_{J_1}\mid Z}\hat{B}_{X_{J_2}\mid Z}^{*}.
\end{aligned} Σ ^ X J 1 X J 2 ∣ Z = E n [ ( κ ^ X J 1 ( ⋅ , X J 1 ) − B ^ X J 1 ∣ Z κ ^ Z ( ⋅ , Z )) ⊗ ( κ ^ X J 2 ( ⋅ , X J 2 ) − B ^ X J 2 ∣ Z κ ^ Z ( ⋅ , Z )) ] = Σ ^ X J 1 X J 2 − B ^ X J 1 ∣ Z Σ ^ Z X J 2 − Σ ^ X J 1 Z B ^ X J 2 ∣ Z ∗ + B ^ X J 1 ∣ Z Σ ^ Z Z B ^ X J 2 ∣ Z ∗ = Σ ^ X J 1 X J 2 − Σ ^ X J 1 Z ( Σ ^ Z Z + ϵ n I ) − 1 Σ ^ Z X J 2 − ϵ n B ^ X J 1 ∣ Z B ^ X J 2 ∣ Z ∗ .
Similarly, for the augmented variable, we define
Σ ^ X ¨ J 1 X J 2 ∣ Z = E n [ ( κ ^ X ¨ J 1 ( ⋅ , X ¨ J 1 ) − B ^ X ¨ J 1 ∣ Z κ ^ Z ( ⋅ , Z ) ) ⊗ ( κ ^ X J 2 ( ⋅ , X J 2 ) − B ^ X J 2 ∣ Z κ ^ Z ( ⋅ , Z ) ) ] . \hat{\Sigma}_{\ddot{X}_{J_1}X_{J_2}\mid Z}=\mathbb{E}_n\bigl[(\hat{\kappa}_{\ddot{X}_{J_1}}(\cdot,\ddot{X}_{J_1})-\hat{B}_{\ddot{X}_{J_1}\mid Z}\hat{\kappa}_Z(\cdot,Z))\otimes(\hat{\kappa}_{X_{J_2}}(\cdot,X_{J_2})-\hat{B}_{X_{J_2}\mid Z}\hat{\kappa}_Z(\cdot,Z))\bigr]. Σ ^ X ¨ J 1 X J 2 ∣ Z = E n [ ( κ ^ X ¨ J 1 ( ⋅ , X ¨ J 1 ) − B ^ X ¨ J 1 ∣ Z κ ^ Z ( ⋅ , Z )) ⊗ ( κ ^ X J 2 ( ⋅ , X J 2 ) − B ^ X J 2 ∣ Z κ ^ Z ( ⋅ , Z )) ] .
The residual-based estimator defined above is closely related to the conventional conditional cross-covariance operator, given by
Σ ^ X J 1 X J 2 − Σ ^ X J 1 Z ( Σ ^ Z Z + ϵ n I ) − 1 Σ ^ Z X J 2 . \hat{\Sigma}_{X_{J_1}X_{J_2}}-\hat{\Sigma}_{X_{J_1}Z}(\hat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\hat{\Sigma}_{ZX_{J_2}}. Σ ^ X J 1 X J 2 − Σ ^ X J 1 Z ( Σ ^ Z Z + ϵ n I ) − 1 Σ ^ Z X J 2 .
Compared with our construction, as illustrated in the next subsection, this formulation includes an additional term, ϵ n B ^ X J 1 ∣ Z B ^ X J 2 ∣ Z ∗ \epsilon_n\hat{B}_{X_{J_1}\mid Z}\hat{B}_{X_{J_2}\mid Z}^{*} ϵ n B ^ X J 1 ∣ Z B ^ X J 2 ∣ Z ∗ . This term converges to zero in Hilbert–Schmidt norm, ensuring that the two constructions share the same consistency limit. However, under our choice of ϵ n \epsilon_n ϵ n , this term need not be negligible at the n − 1 / 2 n^{-1/2} n − 1/2 scale. Moreover, the two constructions employ different coordinate mappings.
Construction of Test Statistic
We first state the theorem below to emphasize the key role played by (conditional) cross-covariance operators in discussing (conditional) independence.
Theorem 1. Under Assumptions 1–2, we have
(Gretton et al., [15 ] ) Σ X J 1 X J 2 = 0 ⟺ X J 1 ⊥ ⊥ X J 2 \Sigma_{X_{J_1}X_{J_2}}=0\Longleftrightarrow X_{J_1}\perp\!\!\perp X_{J_2} Σ X J 1 X J 2 = 0 ⟺ X J 1 ⊥ ⊥ X J 2 ;
(Fukumizu et al., [13 ] ) Σ X ¨ J 1 X J 2 ∣ Z = 0 ⟺ X J 1 ⊥ ⊥ X J 2 ∣ Z \Sigma_{\ddot{X}_{J_1}X_{J_2}\mid Z}=0\Longleftrightarrow X_{J_1}\perp\!\!\perp X_{J_2}\mid Z Σ X ¨ J 1 X J 2 ∣ Z = 0 ⟺ X J 1 ⊥ ⊥ X J 2 ∣ Z .
where X J 1 ⊥ ⊥ X J 2 ∣ Z X_{J_1}\perp\!\!\perp X_{J_2}\mid Z X J 1 ⊥ ⊥ X J 2 ∣ Z means P ( X J 1 , X J 2 ) ∣ Z = P X J 1 ∣ Z ⊗ P X J 2 ∣ Z P_{(X_{J_1},X_{J_2})\mid Z}=P_{X_{J_1}\mid Z}\otimes P_{X_{J_2}\mid Z} P ( X J 1 , X J 2 ) ∣ Z = P X J 1 ∣ Z ⊗ P X J 2 ∣ Z , P Z P_Z P Z -almost surely.
The cross-covariance operators offer a convenient tool to characterize conditional independence. Another useful fact is that the mutual independence between d d d variables can be equivalently derived by d − 1 d-1 d − 1 pairwise independence relations. For j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 , write j + = { j + 1 , … , d } j^+=\{j+1,\ldots,d\} j + = { j + 1 , … , d } . Then
X 1 ⊥ ⊥ X 2 ⊥ ⊥ … ⊥ ⊥ X d ∣ Z ⟺ X j ⊥ ⊥ X j + ∣ Z , for j = 1 , … , d − 1. (3) X_1\perp\!\!\perp X_2\perp\!\!\perp\ldots\perp\!\!\perp X_d\mid Z\Longleftrightarrow X_j\perp\!\!\perp X_{j^+}\mid Z,\quad\text{for }j=1,\ldots,d-1. \tag*{(3)} X 1 ⊥ ⊥ X 2 ⊥ ⊥ … ⊥ ⊥ X d ∣ Z ⟺ X j ⊥ ⊥ X j + ∣ Z , for j = 1 , … , d − 1. (3)
We combine the corresponding conditional cross-covariance operators through a Hilbert direct sum to obtain a unified characterization of multi-way conditional independence. Specifically, let
H S = ⨁ j = 1 d − 1 B 2 ( H X j + , H X ~ j ) . \mathbb{H}_{S} = \bigoplus_{j=1}^{d-1} \mathcal{B}_{2}(\mathcal{H}_{X_{j+}}, \mathcal{H}_{\tilde{X}_{j}}). H S = j = 1 ⨁ d − 1 B 2 ( H X j + , H X ~ j ) .
where B 2 ( H , G ) \mathcal{B}_{2}(\mathcal{H},\mathcal{G}) B 2 ( H , G ) denotes the space of Hilbert–Schmidt operators from H \mathcal{H} H to G \mathcal{G} G . A member of H S \mathbb{H}_{S} H S is a ( d − 1 ) (d-1) ( d − 1 ) -dimensional vector of linear operators in B 2 ( H X 1 + , H X ~ 1 ) × ⋯ × B 2 ( H X ( d − 1 ) + , H X ~ d − 1 ) \mathcal{B}_{2}(\mathcal{H}_{X_{1+}},\mathcal{H}_{\tilde{X}_{1}}) \times\cdots\times\mathcal{B}_{2}(\mathcal{H}_{X_{(d-1)+}},\mathcal{H}_{\tilde{X}_{d-1}}) B 2 ( H X 1 + , H X ~ 1 ) × ⋯ × B 2 ( H X ( d − 1 ) + , H X ~ d − 1 ) , with the inner product defined as follows: for any U = ( U 1 , … , U d − 1 ) U=(U_{1},\ldots,U_{d-1}) U = ( U 1 , … , U d − 1 ) and V = ( V 1 , … , V d − 1 ) V=(V_{1},\ldots,V_{d-1}) V = ( V 1 , … , V d − 1 ) in H S \mathbb{H}_{S} H S ,
⟨ U , V ⟩ H S = ∑ j = 1 d − 1 ⟨ U j , V j ⟩ H S = ∑ j = 1 d − 1 tr ( U j ∗ V j ) . \langle U,V\rangle_{\mathbb{H}_{S}} = \sum_{j=1}^{d-1}\langle U_{j},V_{j}\rangle_{\mathrm{HS}} = \sum_{j=1}^{d-1}\operatorname{tr}(U_{j}^{*}V_{j}). ⟨ U , V ⟩ H S = j = 1 ∑ d − 1 ⟨ U j , V j ⟩ HS = j = 1 ∑ d − 1 tr ( U j ∗ V j ) .
where U j ∗ U_{j}^{*} U j ∗ denotes the adjoint of U j U_{j} U j . The induced norm satisfies ∥ U ∥ H S 2 = ∑ j = 1 d − 1 ∥ U j ∥ H S 2 \lVert U\rVert_{\mathbb{H}_{S}}^{2}=\sum_{j=1}^{d-1}\lVert U_{j}\rVert_{\mathrm{HS}}^{2} ∥ U ∥ H S 2 = ∑ j = 1 d − 1 ∥ U j ∥ HS 2 . Following the idea in (3), we define S ∈ H S S\in\mathbb{H}_{S} S ∈ H S as
S = ( Σ ~ X 1 X 1 + ∣ Z , Σ ~ X 2 X 2 + ∣ Z , … , Σ ~ X d − 1 X d ∣ Z ) T . S = \left(\widetilde{\Sigma}_{X_{1}X_{1+}\mid Z},\widetilde{\Sigma}_{X_{2}X_{2+}\mid Z},\ldots,\widetilde{\Sigma}_{X_{d-1}X_{d\mid Z}}\right)^{\mathsf{T}}. S = ( Σ X 1 X 1 + ∣ Z , Σ X 2 X 2 + ∣ Z , … , Σ X d − 1 X d ∣ Z ) T .
Its squared norm is therefore
∥ S ∥ H S 2 = ∑ j = 1 d − 1 ∥ Σ ~ X j X j + ∣ Z ∥ H S 2 . \lVert S\rVert_{\mathbb{H}_{S}}^{2}=\sum_{j=1}^{d-1}\lVert\widetilde{\Sigma}_{X_{j}X_{j+}\mid Z}\rVert_{\mathrm{HS}}^{2}. ∥ S ∥ H S 2 = j = 1 ∑ d − 1 ∥ Σ X j X j + ∣ Z ∥ HS 2 .
The following theorem establishes that S S S is a well-defined element of H S \mathbb{H}_{S} H S and characterizes multi-way conditional independence through its vanishing.
Theorem 2. Under Assumptions 1 and 2, we have
S = 0 ⟺ X 1 ⊥ X 2 ⊥ ⋯ ⊥ X d ∣ Z . S=0 \quad\Longleftrightarrow\quad X_{1}\perp X_{2}\perp\cdots\perp X_{d}\mid Z. S = 0 ⟺ X 1 ⊥ X 2 ⊥ ⋯ ⊥ X d ∣ Z . Furthermore, by boundedness of kernels, we have
∥ S ∥ H S = 0 ⟺ X 1 ⊥ X 2 ⊥ ⋯ ⊥ X d ∣ Z . \lVert S\rVert_{\mathbb{H}_{S}}=0 \quad\Longleftrightarrow\quad X_{1}\perp X_{2}\perp\cdots\perp X_{d}\mid Z. ∥ S ∥ H S = 0 ⟺ X 1 ⊥ X 2 ⊥ ⋯ ⊥ X d ∣ Z .
This theorem provides the population foundation for the proposed test. It converts multi-way conditional independence into a single zero-norm condition in H S \mathbb{H}_{S} H S . Each component of S S S characterizes conditional independence between one variable and the joint block of variables that follow it. The direct-sum norm aggregates these restrictions and is strictly positive whenever at least one of them fails. At the sample level, we collect the regularized empirical operators into
S ^ = ( Σ ~ ^ X 1 X 1 + ∣ Z , Σ ~ ^ X 2 X 2 + ∣ Z , … , Σ ~ ^ X d − 1 X d ∣ Z ) T \widehat{S}=\left(\widehat{\widetilde{\Sigma}}_{X_{1}X_{1+}\mid Z},\widehat{\widetilde{\Sigma}}_{X_{2}X_{2+}\mid Z},\ldots,\widehat{\widetilde{\Sigma}}_{X_{d-1}X_{d\mid Z}}\right)^{\mathsf{T}} S = ( Σ X 1 X 1 + ∣ Z , Σ X 2 X 2 + ∣ Z , … , Σ X d − 1 X d ∣ Z ) T
and define the test statistic as
T n = n ∥ S ^ ∥ H S 2 = n ∑ j = 1 d − 1 ∥ Σ ^ X ~ j X j + 1 ∣ Z ∥ H S 2 . T_n = n\|\widehat{S}\|_{\mathrm{HS}}^2 = n\sum_{j=1}^{d-1}\|\widehat{\Sigma}_{\widetilde{X}_jX_{j+1}\mid Z}\|_{\mathrm{HS}}^2. T n = n ∥ S ∥ HS 2 = n j = 1 ∑ d − 1 ∥ Σ X j X j + 1 ∣ Z ∥ HS 2 .
The value of the proposed statistic may depend on that ordering. If symmetry is desired, one may consider another construction by averaging the statistic over all permutations, but this can entail substantial additional computation. A more economical alternative is to compare each X j X_j X j with the joint block X { 1 , … , d } ∖ { j } X_{\{1,\ldots,d\}\setminus\{j\}} X { 1 , … , d } ∖ { j } and aggregate the corresponding squared Hilbert–Schmidt norms over j = 1 , … , d j=1,\ldots,d j = 1 , … , d . It uses d d d components instead of d − 1 d-1 d − 1 , and common computations involving Z Z Z can be reused, so the additional computational burden is modest. Although we do not develop this alternative here, we could study it within the same framework and derive a similar characterization, adjusting the joint covariance operator and the resulting eigenvalue weights accordingly.
Asymptotic Properties
Convergence Rate of Regression Operators
We first study the convergence of the empirical regression operators. Our analysis requires smoothness conditions on the relevant cross-moment operators and an eigenvalue-decay condition on Σ Z Z \Sigma_{ZZ} Σ Z Z . These conditions allow us to derive convergence rates and control the estimation errors that enter the subsequent asymptotic analysis. For the augmented operator B ^ X ~ J ∣ Z \widehat{B}_{\widetilde{X}_J\mid Z} B X J ∣ Z , we first decompose Σ ^ X ~ J Z \widehat{\Sigma}_{\widetilde{X}_J Z} Σ X J Z so that different moment components can be treated separately. To express this decomposition, we introduce the following embedding operators.
For any subset J ⊂ { 1 , … , d } J\subset\{1,\ldots,d\} J ⊂ { 1 , … , d } , and any μ ∈ H X J \mu\in\mathcal{H}_{X_J} μ ∈ H X J , define the bounded linear operator T μ : H Z → H X ~ J T_\mu:\mathcal{H}_Z\to\mathcal{H}_{\widetilde{X}_J} T μ : H Z → H X J as
T μ f = μ f , for any f ∈ H Z . T_\mu f=\mu f,\qquad\text{for any }f\in\mathcal{H}_Z. T μ f = μ f , for any f ∈ H Z .
Notice that ∥ T μ ∥ o p = ∥ μ ∥ H X J \|T_\mu\|_{\mathrm{op}}=\|\mu\|_{\mathcal{H}_{X_J}} ∥ T μ ∥ op = ∥ μ ∥ H X J . Similarly, for any ν ∈ H Z \nu\in\mathcal{H}_Z ν ∈ H Z we can define R ν : H X J → H X ~ J R_\nu:\mathcal{H}_{X_J}\to\mathcal{H}_{\widetilde{X}_J} R ν : H X J → H X J as
R ν g = g ν , for any g ∈ H X J . R_\nu g=g\nu,\qquad\text{for any }g\in\mathcal{H}_{X_J}. R ν g = g ν , for any g ∈ H X J .
with ∥ R ν ∥ o p = ∥ ν ∥ H Z \|R_\nu\|_{\mathrm{op}}=\|\nu\|_{\mathcal{H}_Z} ∥ R ν ∥ op = ∥ ν ∥ H Z . Here, we slightly abuse notation by writing T μ T_\mu T μ and R ν R_\nu R ν without explicitly indicating their range/domain H X ~ J / H X J \mathcal{H}_{\widetilde{X}_J}/\mathcal{H}_{X_J} H X J / H X J , which varies with J J J and will be clear from the context.
Since E [ κ ~ Z ( ⋅ , Z ) ] = 0 \mathbb{E}[\widetilde{\kappa}_Z(\cdot,Z)]=0 E [ κ Z ( ⋅ , Z )] = 0 , we have
Σ ^ X ~ J Z = E [ κ X J ( ⋅ , X J ) κ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] = E [ ( μ X J + κ ~ X J ( ⋅ , X J ) ) ( μ Z + κ ~ Z ( ⋅ , Z ) ) ⊗ κ ~ Z ( ⋅ , Z ) ] = T μ X J Σ Z Z + R μ Z Σ X J Z + E [ κ ~ X J ( ⋅ , X J ) κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] , (4) \begin{aligned}
\widehat{\Sigma}_{\widetilde{X}_J Z}
&=\mathbb{E}\left[\kappa_{X_J}(\cdot,X_J)\kappa_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]\\
&=\mathbb{E}\left[\left(\mu_{X_J}+\widetilde{\kappa}_{X_J}(\cdot,X_J)\right)\left(\mu_Z+\widetilde{\kappa}_Z(\cdot,Z)\right)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]\\
&=T_{\mu_{X_J}}\Sigma_{ZZ}+R_{\mu_Z}\Sigma_{X_JZ}
+\mathbb{E}\left[\widetilde{\kappa}_{X_J}(\cdot,X_J)\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right], \tag*{(4)}
\end{aligned} Σ X J Z = E [ κ X J ( ⋅ , X J ) κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] = E [ ( μ X J + κ X J ( ⋅ , X J ) ) ( μ Z + κ Z ( ⋅ , Z ) ) ⊗ κ Z ( ⋅ , Z ) ] = T μ X J Σ Z Z + R μ Z Σ X J Z + E [ κ X J ( ⋅ , X J ) κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] , (4)
where the third equality comes from the fact that for any g ∈ H Z g \in\mathcal{H}_Z g ∈ H Z ,
E [ ( κ ~ X J ( ⋅ , X J ) μ Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] g = E [ ( κ ~ X J ( ⋅ , X J ) μ Z ) ( g ( Z ) − E [ g ( Z ) ] ) ] = ( Σ X J Z g ) μ Z = R μ Z Σ X J Z g . \mathbb{E}\left[(\widetilde{\kappa}_{X_J}(\cdot,X_J)\mu_Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]g=\mathbb{E}\left[(\widetilde{\kappa}_{X_J}(\cdot,X_J)\mu_Z)(g(Z)-\mathbb{E}[g(Z)])\right]=(\Sigma_{X_JZ}g)\mu_Z=R_{\mu_Z}\Sigma_{X_JZ}g. E [ ( κ X J ( ⋅ , X J ) μ Z ) ⊗ κ Z ( ⋅ , Z ) ] g = E [ ( κ X J ( ⋅ , X J ) μ Z ) ( g ( Z ) − E [ g ( Z )]) ] = ( Σ X J Z g ) μ Z = R μ Z Σ X J Z g .
Therefore, Σ ~ X J Z \widetilde{\Sigma}_{X_JZ} Σ X J Z consists of a term determined by the marginal covariance operator Σ Z Z \Sigma_{ZZ} Σ Z Z , a term determined by the cross-covariance operator Σ X J Z \Sigma_{X_JZ} Σ X J Z , and a genuinely centered third-order term, denoted by Λ X ~ J Z = E [ κ ~ X J ( ⋅ , X J ) κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] \Lambda_{\widetilde{X}_JZ}=\mathbb{E}\left[\widetilde{\kappa}_{X_J}(\cdot,X_J)\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right] Λ X J Z = E [ κ X J ( ⋅ , X J ) κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] . Similarly, we can derive the empirical version of the decomposition:
Σ ^ X ~ J Z = T μ ^ X J Σ ^ Z Z + R μ ^ Z Σ ^ X J Z + E n [ κ ^ X J ( ⋅ , X J ) κ ^ Z ( ⋅ , Z ) ⊗ κ ^ Z ( ⋅ , Z ) ] . \widehat{\Sigma}_{\widetilde{X}_JZ}=T_{\widehat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ}+R_{\widehat{\mu}_Z}\widehat{\Sigma}_{X_JZ}+\mathbb{E}_n\left[\widehat{\kappa}_{X_J}(\cdot,X_J)\widehat{\kappa}_Z(\cdot,Z)\otimes\widehat{\kappa}_Z(\cdot,Z)\right]. Σ X J Z = T μ X J Σ Z Z + R μ Z Σ X J Z + E n [ κ X J ( ⋅ , X J ) κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] .
Assumption 3. There exists some β > 0 \beta>0 β > 0 such that for any J ⊂ { 1 , … , d } J\subset\{1,\ldots,d\} J ⊂ { 1 , … , d } we have
Σ X J Z = S X J Z Σ Z Z 1 + β \Sigma_{X_JZ}=S_{X_JZ}\Sigma_{ZZ}^{1+\beta} Σ X J Z = S X J Z Σ Z Z 1 + β for some bounded linear operator S X J Z : H Z → H X J S_{X_JZ}:\mathcal{H}_Z\to\mathcal{H}_{X_J} S X J Z : H Z → H X J ;
Λ X ~ J Z = S X ~ J Z Σ Z Z 1 + β \Lambda_{\widetilde{X}_JZ}=S_{\widetilde{X}_JZ}\Sigma_{ZZ}^{1+\beta} Λ X J Z = S X J Z Σ Z Z 1 + β for some bounded linear operator S X ~ J Z : H Z → H X ~ J S_{\widetilde{X}_JZ}:\mathcal{H}_Z\to\mathcal{H}_{\widetilde{X}_J} S X J Z : H Z → H X J .
Assumption 4. Let γ 1 ≥ γ 2 ≥ … \gamma_1\ge\gamma_2\ge\ldots γ 1 ≥ γ 2 ≥ … be ordered eigenvalues of Σ Z Z \Sigma_{ZZ} Σ Z Z . Assume γ k ≍ k − α \gamma_k\asymp k^{-\alpha} γ k ≍ k − α for some α > 1 \alpha>1 α > 1 .
The preceding decomposition separates the component of Σ ~ X J Z \widetilde{\Sigma}_{X_JZ} Σ X J Z inherited from Σ Z Z \Sigma_{ZZ} Σ Z Z from the remaining components governed by the source condition. Together with the eigenvalue-decay assumption, this decomposition yields convergence rates for the empirical regression operator B ^ X J ∣ Z \widehat{B}_{X_J\mid Z} B X J ∣ Z and the corrected version of the augmented regression operator B ^ X ~ J ∣ Z \widehat{B}_{\widetilde{X}_J\mid Z} B X J ∣ Z , as stated in Theorem 3 and Proposition 1 below.
Theorem 3. Suppose Assumptions 1–4 hold. If ϵ n → 0 \epsilon_n\to0 ϵ n → 0 as n → ∞ n\to\infty n → ∞ , then
∥ B ^ X J ∣ Z − B X J ∣ Z ∥ o p = O P ( r n ) \left\|\widehat{B}_{X_J\mid Z}-B_{X_J\mid Z}\right\|_{\mathrm{op}}=O_P(r_n) B X J ∣ Z − B X J ∣ Z op = O P ( r n ) , and
∥ ( B ^ X ~ J ∣ Z − T μ ^ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 ) − ( B X ~ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ∥ o p = O P ( r n ) \left\|\left(\widehat{B}_{\widetilde{X}_J\mid Z}-T_{\widehat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right)-\left(B_{\widetilde{X}_J\mid Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)\right\|_{\mathrm{op}}=O_P(r_n) ( B X J ∣ Z − T μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 ) − ( B X J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) op = O P ( r n ) ,
where r n = n − 1 / 2 ϵ n β ∧ 1 − 1 + ϵ n β ∧ 1 + n − 1 ϵ n − 3 α + 1 2 α + n − 1 / 2 ϵ n − α + 1 2 α r_n=n^{-1/2}\epsilon_n^{\beta\wedge1-1}+\epsilon_n^{\beta\wedge1}+n^{-1}\epsilon_n^{-\frac{3\alpha+1}{2\alpha}}+n^{-1/2}\epsilon_n^{-\frac{\alpha+1}{2\alpha}} r n = n − 1/2 ϵ n β ∧ 1 − 1 + ϵ n β ∧ 1 + n − 1 ϵ n − 2 α 3 α + 1 + n − 1/2 ϵ n − 2 α α + 1 .
Proof. See Appendix. For assertion 1, the proof is the same as Sang and Li [42 ] . □
Proposition 1. Suppose the conditions of Theorem 3 hold and let ϵ n ≍ n − δ \epsilon_n\asymp n^{-\delta} ϵ n ≍ n − δ for some δ > 0 \delta>0 δ > 0 . The choice of δ \delta δ that optimizes the convergence rate r n r_n r n in Theorem 3 is as follows:
If β > α − 1 2 α \beta>\frac{\alpha-1}{2\alpha} β > 2 α α − 1 , then δ o p t = α 2 α ( β ∧ 1 ) + α + 1 \delta_{\mathrm{opt}}=\frac{\alpha}{2\alpha(\beta\wedge1)+\alpha+1} δ opt = 2 α ( β ∧ 1 ) + α + 1 α . Under this choice, the optimal rate of r n r_n r n is O P ( n − α ( β ∧ 1 ) 2 α ( β ∧ 1 ) + α + 1 ) O_P\left(n^{-\frac{\alpha(\beta\wedge1)}{2\alpha(\beta\wedge1)+\alpha+1}}\right) O P ( n − 2 α ( β ∧ 1 ) + α + 1 α ( β ∧ 1 ) ) ;
If 0 < β ≤ α − 1 2 α 0<\beta\le\frac{\alpha-1}{2\alpha} 0 < β ≤ 2 α α − 1 , then δ o p t = 1 2 \delta_{\mathrm{opt}}=\frac{1}{2} δ opt = 2 1 . Under this choice, the optimal rate of r n r_n r n is O P ( n − β / 2 ) O_P(n^{-\beta/2}) O P ( n − β /2 ) .
Proof. See Theorem 10 in Sang and Li [42 ] . □
In the first regime, the exponent in the optimal rate is − α ( β ∧ 1 ) 2 α ( β ∧ 1 ) + α + 1 -\frac{\alpha(\beta\wedge1)}{2\alpha(\beta\wedge1)+\alpha+1} − 2 α ( β ∧ 1 ) + α + 1 α ( β ∧ 1 ) . For fixed α \alpha α , this exponent is decreasing in β ∧ 1 \beta\wedge1 β ∧ 1 and attains its smallest value when β ≥ 1 \beta\ge1 β ≥ 1 . In this case, the convergence rate becomes O P ( n − α 3 α + 1 ) O_P(n^{-\frac{\alpha}{3\alpha+1}}) O P ( n − 3 α + 1 α ) . Since α > 1 \alpha>1 α > 1 ,
1 4 < α 3 α + 1 < 1 3 , α 3 α + 1 → 1 3 as α → ∞ . \frac{1}{4}<\frac{\alpha}{3\alpha+1}<\frac{1}{3},\qquad\frac{\alpha}{3\alpha+1}\to\frac{1}{3}\quad\text{as }\alpha\to\infty. 4 1 < 3 α + 1 α < 3 1 , 3 α + 1 α → 3 1 as α → ∞.
Thus, the fastest rate allowed by r n r_n r n can be arbitrarily close to n − 1 / 3 n^{-1/3} n − 1/3 . More generally, the rate is o P ( n − 1 / 4 ) o_P(n^{-1/4}) o P ( n − 1/4 ) whenever β ∧ 1 > α + 1 2 α \beta\wedge1>\frac{\alpha+1}{2\alpha} β ∧ 1 > 2 α α + 1 .
Assertion 2 of Theorem 3 controls the difference between the corrected regression operators, rather than B ^ X ~ J ∣ Z − B X ~ J ∣ Z \widehat{B}_{\widetilde{X}_J\mid Z}-B_{\widetilde{X}_J\mid Z} B X J ∣ Z − B X J ∣ Z itself. To isolate the remaining projection component, fix a subset J J J and define
R n = T μ ^ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 − T μ X J Σ Z Z Σ Z Z † . (5) R_n=T_{\widehat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}. \tag*{(5)} R n = T μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 − T μ X J Σ Z Z Σ Z Z † . (5)
The dependence of R n R_n R n on J J J is suppressed in the notation. Then
B ^ X ~ J ∣ Z − B X ~ J ∣ Z = [ ( B ^ X ~ J ∣ Z − T μ ^ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 ) − ( B X ~ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ] + R n . \widehat{B}_{\widetilde{X}_J\mid Z}-B_{\widetilde{X}_J\mid Z}=\left[\left(\widehat{B}_{\widetilde{X}_J\mid Z}-T_{\widehat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right)-\left(B_{\widetilde{X}_J\mid Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)\right]+R_n. B X J ∣ Z − B X J ∣ Z = [ ( B X J ∣ Z − T μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 ) − ( B X J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ] + R n .
The first term on the right-hand side is controlled by Theorem 3. In contrast, R n R_n R n need not converge to zero in operator norm, because the regularized empirical projection Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 \widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1} Σ Z Z ( Σ Z Z + ϵ n I ) − 1 need not converge to Σ Z Z Σ Z Z † \Sigma_{ZZ}\Sigma_{ZZ}^{\dagger} Σ Z Z Σ Z Z † in operator norm.
Nevertheless, R n R_n R n only appears after composition with a positive power of Σ Z Z \Sigma_{ZZ} Σ Z Z in the subsequent asymptotic expansion. The following results show that this smoothing operation is sufficient to make R n R_n R n asymptotically negligible, which are parallel to Tang and Li [47 ] .
Lemma 1. Suppose Assumptions 1–4 hold and ϵ n → 0 \epsilon_n\to0 ϵ n → 0 . Then, for every θ > 0 \theta>0 θ > 0 ,
∥ R n Σ Z Z θ ∥ o p = O P ( n − 1 / 2 ϵ n θ ∧ 1 − 1 + ϵ n θ ∧ 1 ) . (6) \left\|R_n\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}=O_P\left(n^{-1/2}\epsilon_n^{\theta\wedge1-1}+\epsilon_n^{\theta\wedge1}\right). \tag*{(6)} R n Σ Z Z θ op = O P ( n − 1/2 ϵ n θ ∧ 1 − 1 + ϵ n θ ∧ 1 ) . (6)
In particular, ∥ R n Σ Z Z ∥ o p = O P ( n − 1 / 2 + ϵ n ) \|R_n\Sigma_{ZZ}\|_{\mathrm{op}}=O_P(n^{-1/2}+\epsilon_n) ∥ R n Σ Z Z ∥ op = O P ( n − 1/2 + ϵ n ) .
Proposition 2. Suppose Assumptions 1–4 hold and ϵ n = n − α / ( 2 α ( β ∧ 1 ) + α + 1 ) \epsilon_n=n^{-\alpha/(2\alpha(\beta\wedge1)+\alpha+1)} ϵ n = n − α / ( 2 α ( β ∧ 1 ) + α + 1 ) , with β ∧ 1 > α + 1 2 α \beta\wedge1>\frac{\alpha+1}{2\alpha} β ∧ 1 > 2 α α + 1 and α > 2 \alpha>2 α > 2 . For each j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 , let R n R_n R n denote the operator in (5) with J = { j } J=\{j\} J = { j } . Then
n ∥ R n Σ Z Z ( B ^ X j ∣ Z − B X j ∣ Z ) ∗ ∥ H S = o P ( 1 ) . (7) \sqrt{n}\left\|R_n\Sigma_{ZZ}\left(\widehat{B}_{X_j\mid Z}-B_{X_j\mid Z}\right)^*\right\|_{\mathrm{HS}}=o_P(1). \tag*{(7)} n R n Σ Z Z ( B X j ∣ Z − B X j ∣ Z ) ∗ HS = o P ( 1 ) . (7)
Asymptotic Null Distribution
We now derive the asymptotic distribution of S ^ \widehat{S} S . Using the convergence results above, we show that the errors due to regression estimation and regularization contribute only an o P ( n − 1 / 2 ) o_P(n^{-1/2}) o P ( n − 1/2 ) remainder to S ^ \widehat{S} S under the null hypothesis. We first establish asymptotic normality for each component and then use these results to derive the joint asymptotic distribution of S ^ \widehat{S} S .
Lemma 2. Suppose Assumptions 1–4 hold. Choose ϵ n = n − α 2 α ( β ∧ 1 ) + α + 1 \epsilon_n = n^{-\frac{\alpha}{2\alpha(\beta\wedge1)+\alpha+1}} ϵ n = n − 2 α ( β ∧ 1 ) + α + 1 α according to Proposition 1, with β ∧ 1 > α + 1 2 α \beta\wedge1 > \frac{\alpha+1}{2\alpha} β ∧ 1 > 2 α α + 1 and α > 2 \alpha> 2 α > 2 . For j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 , define
ψ ( X ¨ j , X j + ; Z ) = ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) − Σ X ¨ j X j + ∣ Z . (8) \psi(\ddot{X}_j,X_{j+};Z)=\xi_{\ddot{X}_j\mid Z}(\cdot,\ddot{X}_j;Z)\otimes\xi_{X_{j+}\mid Z}(\cdot,X_{j+};Z)-\Sigma_{\ddot{X}_jX_{j+}\mid Z}. \tag*{(8)} ψ ( X ¨ j , X j + ; Z ) = ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) − Σ X ¨ j X j + ∣ Z . (8) Then Σ ^ X ¨ j X j + ∣ Z \widehat{\Sigma}_{\ddot{X}_jX_{j+}\mid Z} Σ X ¨ j X j + ∣ Z admits the asymptotic linear representation
n ( Σ ^ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) = 1 n ∑ i = 1 n ψ ( X ¨ j , i , X j + , i ; Z i ) + o P ( 1 ) , (9) \sqrt{n}\left(\widehat{\Sigma}_{\ddot{X}_jX_{j+}\mid Z}-\Sigma_{\ddot{X}_jX_{j+}\mid Z}\right)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi(\ddot{X}_{j,i},X_{j+,i};Z_i)+o_P(1), \tag*{(9)} n ( Σ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) = n 1 i = 1 ∑ n ψ ( X ¨ j , i , X j + , i ; Z i ) + o P ( 1 ) , (9) where the remainder is measured in the Hilbert–Schmidt norm. Consequently,
n ( Σ ^ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) ⟶ D N ( 0 , Γ X ¨ j X j + ∣ Z ) , \sqrt{n}\left(\widehat{\Sigma}_{\ddot{X}_jX_{j+}\mid Z}-\Sigma_{\ddot{X}_jX_{j+}\mid Z}\right)\overset{\mathcal{D}}{\longrightarrow}N(0,\Gamma_{\ddot{X}_jX_{j+}\mid Z}), n ( Σ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) ⟶ D N ( 0 , Γ X ¨ j X j + ∣ Z ) , where Γ X ¨ j X j + ∣ Z : H X ¨ j ⊗ H X j + → H X ¨ j ⊗ H X j + \Gamma_{\ddot{X}_jX_{j+}\mid Z}:\mathcal{H}_{\ddot{X}_j}\otimes\mathcal{H}_{X_{j+}}\to\mathcal{H}_{\ddot{X}_j}\otimes\mathcal{H}_{X_{j+}} Γ X ¨ j X j + ∣ Z : H X ¨ j ⊗ H X j + → H X ¨ j ⊗ H X j + is the covariance operator
Γ X ¨ j X j + ∣ Z = E [ ψ ( X ¨ j , X j + ; Z ) ⊗ 2 ] . (10) \Gamma_{\ddot{X}_jX_{j+}\mid Z}=\mathbb{E}\left[\psi(\ddot{X}_j,X_{j+};Z)^{\otimes2}\right]. \tag*{(10)} Γ X ¨ j X j + ∣ Z = E [ ψ ( X ¨ j , X j + ; Z ) ⊗ 2 ] . (10) Note that (9) indicates that ψ ( X ¨ j , X j + ; Z ) \psi(\ddot{X}_j,X_{j+};Z) ψ ( X ¨ j , X j + ; Z ) is the influence function of Σ ^ X ¨ j X j + ∣ Z \widehat{\Sigma}_{\ddot{X}_jX_{j+}\mid Z} Σ X ¨ j X j + ∣ Z . Once these influence functions have been identified, the asymptotic distribution of S ^ \widehat{S} S follows directly from its asymptotically linear representation. Combining the asymptotic linear representations of the individual components yields the following joint Gaussian limit:
Theorem 4. Under the same assumptions as Lemma 2, we have
n ( S ^ − S ) ⟶ D N ( 0 , Γ S ∣ Z ) (11) \sqrt{n}(\widehat{S}-S)\overset{\mathcal{D}}{\longrightarrow}N(0,\Gamma_{S\mid Z}) \tag*{(11)} n ( S − S ) ⟶ D N ( 0 , Γ S ∣ Z ) (11) where Γ S ∣ Z : H S → H S \Gamma_{S\mid Z}:\mathcal{H}_S\to\mathcal{H}_S Γ S ∣ Z : H S → H S is
( Γ S ∣ Z ) ( j , k ) = E [ ψ ( X ¨ j , X j + ; Z ) ⊗ ψ ( X ¨ k , X k + ; Z ) ] = E [ ψ j ⊗ ψ k ] . (\Gamma_{S\mid Z})_{(j,k)}=\mathbb{E}\left[\psi(\ddot{X}_j,X_{j+};Z)\otimes\psi(\ddot{X}_k,X_{k+};Z)\right]=\mathbb{E}\left[\psi_j\otimes\psi_k\right]. ( Γ S ∣ Z ) ( j , k ) = E [ ψ ( X ¨ j , X j + ; Z ) ⊗ ψ ( X ¨ k , X k + ; Z ) ] = E [ ψ j ⊗ ψ k ] .
The joint Gaussian limit determines the asymptotic null distribution of the test statistic. Under multi-way conditional independence S = 0 S=0 S = 0 , the continuous mapping theorem yields the following weighted chi-square limit for n ∥ S ^ ∥ H S 2 n\lVert\widehat{S}\rVert_{\mathcal{H}_S}^{2} n ∥ S ∥ H S 2 .
Corollary 1. Assume the null hypothesis S = 0 S=0 S = 0 holds. Then under the same assumptions of Theorem 4, we have
n ∥ S ^ ∥ H S 2 ⟶ D ∑ k = 1 ∞ λ k Z k 2 (12) n\lVert\widehat{S}\rVert_{\mathcal{H}_S}^{2}\overset{\mathcal{D}}{\longrightarrow}\sum_{k=1}^{\infty}\lambda_k Z_k^2 \tag*{(12)} n ∥ S ∥ H S 2 ⟶ D k = 1 ∑ ∞ λ k Z k 2 (12) where λ k \lambda_k λ k are the eigenvalues of Γ S ∣ Z \Gamma_{S\mid Z} Γ S ∣ Z in (11) with Σ X ¨ j X j + ∣ Z = 0 \Sigma_{\ddot{X}_jX_{j+}\mid Z}=0 Σ X ¨ j X j + ∣ Z = 0 , and Z k Z_k Z k are i.i.d. standard Gaussian variables.
Asymptotic Distribution under Local Alternatives
We next study the behavior of the proposed test under alternatives that converge to the null hypothesis as the sample size increases. Let P n P_n P n denote the distribution of each observation in row n n n , and attach a subscript n n n to population quantities computed under P n P_n P n . In particular, write S n S_n S n for the vector of conditional cross-covariance operators and Σ X J Z , n \Sigma_{X_JZ,n} Σ X J Z , n , Σ Z Z , n \Sigma_{ZZ,n} Σ Z Z , n for the cross-covariance operator under P n P_n P n ; Λ X ¨ J Z , n \Lambda_{\ddot{X}_JZ,n} Λ X ¨ J Z , n for the third-order term.
Assumption 5. For every n n n , assume Assumptions 1–4 hold under P n P_n P n with the same α \alpha α and β \beta β . The kernels and their associated reproducing kernel Hilbert spaces do not depend on n n n . Moreover, for every relevant subset J ⊂ { 1 , … , d } J \subset\{1,\ldots,d\} J ⊂ { 1 , … , d } , there exist uniformly bounded linear operators { S X J Z , n } n \{S_{X_JZ,n}\}_n { S X J Z , n } n and { S X ¨ J Z , n } n \{S_{\ddot{X}_JZ,n}\}_n { S X ¨ J Z , n } n , with
sup n ≥ 1 ∥ S X J Z , n ∥ o p < ∞ , sup n ≥ 1 ∥ S X ¨ J Z , n ∥ o p < ∞ , \sup_{n\geq1}\lVert S_{X_JZ,n}\rVert_{\mathrm{op}} < \infty,\qquad\sup_{n\geq1}\lVert S_{\ddot{X}_JZ,n}\rVert_{\mathrm{op}} < \infty, n ≥ 1 sup ∥ S X J Z , n ∥ op < ∞ , n ≥ 1 sup ∥ S X ¨ J Z , n ∥ op < ∞ , such that
Σ X J Z , n = S X J Z , n Σ Z Z , n 1 + β , Λ X ¨ J Z , n = S X ¨ J Z , n Σ Z Z , n 1 + β . \Sigma_{X_JZ,n}=S_{X_JZ,n}\Sigma_{ZZ,n}^{1+\beta},\qquad\Lambda_{\ddot{X}_JZ,n}=S_{\ddot{X}_JZ,n}\Sigma_{ZZ,n}^{1+\beta}. Σ X J Z , n = S X J Z , n Σ Z Z , n 1 + β , Λ X ¨ J Z , n = S X ¨ J Z , n Σ Z Z , n 1 + β . Let { γ k , n } k ≥ 1 \{\gamma_{k,n}\}_{k\geq1} { γ k , n } k ≥ 1 be the ordered eigenvalues of Σ Z Z , n \Sigma_{ZZ,n} Σ Z Z , n . We assume that γ k , n ≍ k − α \gamma_{k,n}\asymp k^{-\alpha} γ k , n ≍ k − α uniformly in n n n , with both the exponent α \alpha α and the implicit comparison constants independent of n n n .
Under P n P_n P n , define
ψ j , n = ξ X ¨ j ∣ Z , n ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z , n ( ⋅ , X j + ; Z ) − Σ X ¨ j X j + ∣ Z , n , j = 1 , … , d − 1 , \psi_{j,n}=\xi_{\ddot{X}_j|Z,n}(\cdot,\ddot{X}_j;Z)\otimes\xi_{X_{j+}|Z,n}(\cdot,X_{j+};Z)-\Sigma_{\ddot{X}_jX_{j+}|Z,n},\qquad j=1,\ldots,d-1, ψ j , n = ξ X ¨ j ∣ Z , n ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z , n ( ⋅ , X j + ; Z ) − Σ X ¨ j X j + ∣ Z , n , j = 1 , … , d − 1 , and let ψ n = ( ψ 1 , n , … , ψ d − 1 , n ) T \psi_n=(\psi_{1,n},\ldots,\psi_{d-1,n})^\mathsf{T} ψ n = ( ψ 1 , n , … , ψ d − 1 , n ) T , which serves as the influence function under the local alternative distribution. Also define Γ S ∣ Z , n = E P n [ ψ n ⊗ ψ n ] \Gamma_{S|Z,n}=E_{P_n}[\psi_n\otimes\psi_n] Γ S ∣ Z , n = E P n [ ψ n ⊗ ψ n ] . Indeed Γ S ∣ Z , n \Gamma_{S|Z,n} Γ S ∣ Z , n is a trace-class operator on H S 2 \mathcal{H}_S^2 H S 2 . The bounds on the kernels and regression operators imply that ψ n \psi_n ψ n is bounded, then
∥ Γ S ∣ Z , n ∥ t r ≤ E P n ∥ ψ n ⊗ ψ n ∥ t r = E P n ∥ ψ n ∥ H S 2 < ∞ . (13) \lVert\Gamma_{S|Z,n}\rVert_{\mathrm{tr}}\leq E_{P_n}\lVert\psi_n\otimes\psi_n\rVert_{\mathrm{tr}}=E_{P_n}\lVert\psi_n\rVert_{\mathcal{H}_S^2}<\infty. \tag*{(13)} ∥ Γ S ∣ Z , n ∥ tr ≤ E P n ∥ ψ n ⊗ ψ n ∥ tr = E P n ∥ ψ n ∥ H S 2 < ∞. (13) Therefore Γ S ∣ Z , n \Gamma_{S|Z,n} Γ S ∣ Z , n is a trace-class operator on H S \mathcal{H}_S H S by definition.
Theorem 5. Suppose Assumption 5 holds with α > 2 \alpha>2 α > 2 and β ∧ 1 > α + 1 2 α \beta\wedge1>\frac{\alpha+1}{2\alpha} β ∧ 1 > 2 α α + 1 . Set ϵ n = n − 2 α ( β ∧ 1 ) + α + 1 2 α \epsilon_n=n^{-\frac{2\alpha(\beta\wedge1)+\alpha+1}{2\alpha}} ϵ n = n − 2 α 2 α ( β ∧ 1 ) + α + 1 , as in Proposition 1. Assume further that
∥ Γ S ∣ Z , n − Γ S ∣ Z , 0 ∥ t r → 0 , (14) \lVert\Gamma_{S|Z,n}-\Gamma_{S|Z,0}\rVert_{\mathrm{tr}}\to0, \tag*{(14)} ∥ Γ S ∣ Z , n − Γ S ∣ Z , 0 ∥ tr → 0 , (14) where Γ S ∣ Z , 0 \Gamma_{S|Z,0} Γ S ∣ Z , 0 is a trace-class operator. Consider local alternatives satisfying
S n = n − 1 / 2 S h (15) S_n=n^{-1/2}S_h \tag*{(15)} S n = n − 1/2 S h (15)
for some fixed S h ∈ H S S_h \in\mathbb{H}_S S h ∈ H S . Then
n ( S ^ n − S n ) → D G , G ∼ N ( 0 , Γ S ∣ Z , 0 ) , (16) \sqrt{n}(\widehat{S}_n-S_n) \xrightarrow{\mathcal{D}} G,\qquad G\sim N(0,\Gamma_{S\mid Z,0}), \tag*{(16)} n ( S n − S n ) D G , G ∼ N ( 0 , Γ S ∣ Z , 0 ) , (16)
and
n ∥ S ^ n ∥ H S 2 → D ∥ G + S h ∥ H S 2 . (17) n\left\|\widehat{S}_n\right\|_{\mathbb{H}_S}^{2}\xrightarrow{\mathcal{D}}\left\|G+S_h\right\|_{\mathbb{H}_S}^{2}. \tag*{(17)} n S n H S 2 D ∥ G + S h ∥ H S 2 . (17)
Suppose additionally that S h ∈ ( ker ( Γ S ∣ Z , 0 ) ) ⊥ S_h\in\left(\ker\left(\Gamma_{S\mid Z,0}\right)\right)^\perp S h ∈ ( ker ( Γ S ∣ Z , 0 ) ) ⊥ . If we adopt the decomposition
Γ S ∣ Z , 0 = ∑ k = 1 ∞ λ k ( v k ⊗ v k ) , S h = ∑ k = 1 ∞ σ k v k , \Gamma_{S\mid Z,0}=\sum_{k=1}^{\infty}\lambda_k(v_k\otimes v_k),\qquad S_h=\sum_{k=1}^{\infty}\sigma_kv_k, Γ S ∣ Z , 0 = k = 1 ∑ ∞ λ k ( v k ⊗ v k ) , S h = k = 1 ∑ ∞ σ k v k ,
where λ k > 0 \lambda_k>0 λ k > 0 are the eigenvalues and v k v_k v k are the corresponding orthonormal eigenvectors, then
n ∥ S ^ n ∥ H S 2 → D ∑ k = 1 ∞ ( λ k Z k + σ k ) 2 , (18) n\left\|\widehat{S}_n\right\|_{\mathbb{H}_S}^{2}\xrightarrow{\mathcal{D}}\sum_{k=1}^{\infty}\left(\sqrt{\lambda_k}Z_k+\sigma_k\right)^2, \tag*{(18)} n S n H S 2 D k = 1 ∑ ∞ ( λ k Z k + σ k ) 2 , (18)
where Z 1 , Z 2 , … Z_1,Z_2,\ldots Z 1 , Z 2 , … are independent standard normal random variables.
Unconditional Independence
The unconditional case is a mathematical subproblem of the preceding conditional framework, by taking Z Z Z to be a degenerate random variable. Its asymptotic results are established separately below and do not require Assumptions 2–4 concerning regression on $Z. In the unconditional setting, we define H S u c \mathbb{H}^{\mathrm{uc}}_S H S uc
H S u c = ⨁ j = 1 d − 1 B 2 ( H X j + , H X j ) . \mathbb{H}^{\mathrm{uc}}_S=\bigoplus_{j=1}^{d-1}\mathcal{B}_2(\mathcal{H}_{X_{j+}},\mathcal{H}_{X_j}). H S uc = j = 1 ⨁ d − 1 B 2 ( H X j + , H X j ) .
In that case, the conditional cross-covariance operator degenerates to the unconditional one. Therefore, we define
S u c = ( Σ X 1 X 1 + , Σ X 2 X 2 + , … , Σ X d − 1 X d ) T S^{\mathrm{uc}}=\left(\Sigma_{X_1X_{1+}},\Sigma_{X_2X_{2+}},\ldots,\Sigma_{X_{d-1}X_d}\right)^{\mathsf{T}} S uc = ( Σ X 1 X 1 + , Σ X 2 X 2 + , … , Σ X d − 1 X d ) T
and its sample counterpart
S ^ u c = ( Σ ^ X 1 X 1 + , Σ ^ X 2 X 2 + , … , Σ ^ X d − 1 X d ) T . \widehat{S}^{\mathrm{uc}}=\left(\widehat{\Sigma}_{X_1X_{1+}},\widehat{\Sigma}_{X_2X_{2+}},\ldots,\widehat{\Sigma}_{X_{d-1}X_d}\right)^{\mathsf{T}}. S uc = ( Σ X 1 X 1 + , Σ X 2 X 2 + , … , Σ X d − 1 X d ) T .
For j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 , we can similarly show that
ψ j u c ( X j , X j + ) = κ ~ X j ( ⋅ , X j ) ⊗ κ ~ X j + ( ⋅ , X j + ) − Σ X j X j + \psi_j^{\mathrm{uc}}(X_j,X_{j+})=\widetilde{\kappa}_{X_j}(\cdot,X_j)\otimes\widetilde{\kappa}_{X_{j+}}(\cdot,X_{j+})-\Sigma_{X_jX_{j+}} ψ j uc ( X j , X j + ) = κ X j ( ⋅ , X j ) ⊗ κ X j + ( ⋅ , X j + ) − Σ X j X j +
is the influence function of Σ ^ X j X j + \widehat{\Sigma}_{X_jX_{j+}} Σ X j X j + , and define
ψ u c = ( ψ 1 u c , … , ψ d − 1 u c ) T , Γ S u c = E [ ψ u c ⊗ ψ u c ] . \psi^{\mathrm{uc}}=\left(\psi_1^{\mathrm{uc}},\ldots,\psi_{d-1}^{\mathrm{uc}}\right)^{\mathsf{T}},\qquad
\Gamma^{\mathrm{uc}}_S=\mathbb{E}\left[\psi^{\mathrm{uc}}\otimes\psi^{\mathrm{uc}}\right]. ψ uc = ( ψ 1 uc , … , ψ d − 1 uc ) T , Γ S uc = E [ ψ uc ⊗ ψ uc ] .
The corresponding results are proposed in the theorem below.
Theorem 6. Suppose for j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 , kernels κ X j \kappa_{X_j} κ X j and κ X j + \kappa_{X_{j+}} κ X j + are continuous, bounded, positive definite, and c 0 c_0 c 0 -universal. Then the following statements hold.
The random variables X 1 , … , X d X_1,\ldots,X_d X 1 , … , X d are mutually independent if and only if S u c = 0 S^{\mathrm{uc}}=0 S uc = 0 ; or equivalently Σ X 1 X 1 + = 0 , Σ X 2 X 2 + = 0 , … , Σ X d − 1 X d = 0 \Sigma_{X_1X_{1+}}=0,\Sigma_{X_2X_{2+}}=0,\ldots,\Sigma_{X_{d-1}X_d}=0 Σ X 1 X 1 + = 0 , Σ X 2 X 2 + = 0 , … , Σ X d − 1 X d = 0 .
We have
n ( S ^ u c − S u c ) → D N ( 0 , Γ S u c ) . \sqrt{n}(\widehat{S}^{\mathrm{uc}}-S^{\mathrm{uc}})\xrightarrow{\mathcal{D}}N(0,\Gamma_S^{\mathrm{uc}}). n ( S uc − S uc ) D N ( 0 , Γ S uc ) . In particular, if X 1 , … , X d X_1,\ldots,X_d X 1 , … , X d are mutually independent, then
n ∥ S ^ u c ∥ H S u c 2 → D ∑ k = 1 ∞ λ k Z k 2 , n\|\widehat{S}^{\mathrm{uc}}\|_{\mathcal{H}_S^{\mathrm{uc}}}^{2}\xrightarrow{\mathcal{D}}\sum_{k=1}^{\infty}\lambda_k Z_k^2, n ∥ S uc ∥ H S uc 2 D k = 1 ∑ ∞ λ k Z k 2 , where { λ k } k ≥ 1 \{\lambda_k\}_{k\geq1} { λ k } k ≥ 1 are the eigenvalues of Γ S u c \Gamma_S^{\mathrm{uc}} Γ S uc and Z 1 , Z 2 , … Z_1,Z_2,\ldots Z 1 , Z 2 , … are independent standard normal random variables.
We next consider local alternatives for the unconditional test. Let P n P_n P n denote the distribution of each observation in row n n n , and attach a subscript n n n to population quantities computed under P n P_n P n . In particular, define κ ~ X J , n ( ⋅ , X J ) = κ X J ( ⋅ , X J ) − μ X J , n \widetilde{\kappa}_{X_J,n}(\cdot,X_J)=\kappa_{X_J}(\cdot,X_J)-\mu_{X_J,n} κ X J , n ( ⋅ , X J ) = κ X J ( ⋅ , X J ) − μ X J , n . Then for j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 , let
ψ j , n u c = κ ~ X j , n ( ⋅ , X j ) ⊗ κ ~ X j + , n ( ⋅ , X j + ) − Σ X j X j + , n , (19) \psi_{j,n}^{\mathrm{uc}}=\widetilde{\kappa}_{X_j,n}(\cdot,X_j)\otimes\widetilde{\kappa}_{X_{j+},n}(\cdot,X_{j+})-\Sigma_{X_jX_{j+},n}, \tag*{(19)} ψ j , n uc = κ X j , n ( ⋅ , X j ) ⊗ κ X j + , n ( ⋅ , X j + ) − Σ X j X j + , n , (19)
ψ n u c = ( ψ 1 , n u c , … , ψ d − 1 , n u c ) T , Γ S , n u c = E P n [ ψ n u c ⊗ ψ n u c ] ∈ H S u c . (20) \psi_n^{\mathrm{uc}}=(\psi_{1,n}^{\mathrm{uc}},\ldots,\psi_{d-1,n}^{\mathrm{uc}})^{\mathsf{T}},\qquad\Gamma_{S,n}^{\mathrm{uc}}=\mathbb{E}_{P_n}\left[\psi_n^{\mathrm{uc}}\otimes\psi_n^{\mathrm{uc}}\right]\in\mathcal{H}_S^{\mathrm{uc}}. \tag*{(20)} ψ n uc = ( ψ 1 , n uc , … , ψ d − 1 , n uc ) T , Γ S , n uc = E P n [ ψ n uc ⊗ ψ n uc ] ∈ H S uc . (20)
By a similar argument as in (13), we know Γ S , n u c \Gamma_{S,n}^{\mathrm{uc}} Γ S , n uc is a trace-class operator.
Theorem 7. Suppose for j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 , kernels κ X j \kappa_{X_j} κ X j and κ X j + \kappa_{X_{j+}} κ X j + are continuous, bounded, positive definite, and c 0 c_0 c 0 -universal. The kernels and their associated reproducing kernel Hilbert spaces are fixed and do not depend on n n n . Assume that
∥ Γ S , n u c − Γ S , 0 u c ∥ t r → 0 , (21) \left\|\Gamma_{S,n}^{\mathrm{uc}}-\Gamma_{S,0}^{\mathrm{uc}}\right\|_{\mathrm{tr}}\to0, \tag*{(21)} Γ S , n uc − Γ S , 0 uc tr → 0 , (21) where Γ S , 0 u c \Gamma_{S,0}^{\mathrm{uc}} Γ S , 0 uc is a trace-class operator on H S u c \mathcal{H}_S^{\mathrm{uc}} H S uc . Consider local alternatives satisfying
S n u c = n − 1 / 2 S h u c (22) S_n^{\mathrm{uc}}=n^{-1/2}S_h^{\mathrm{uc}} \tag*{(22)} S n uc = n − 1/2 S h uc (22) for some fixed S h u c ∈ H S u c S_h^{\mathrm{uc}}\in\mathcal{H}_S^{\mathrm{uc}} S h uc ∈ H S uc . Then
n ( S ^ n u c − S n u c ) → D G , G ∼ N ( 0 , Γ S , 0 u c ) , (23) \sqrt{n}(\widehat{S}_n^{\mathrm{uc}}-S_n^{\mathrm{uc}})\xrightarrow{\mathcal{D}}G,\qquad G\sim N(0,\Gamma_{S,0}^{\mathrm{uc}}), \tag*{(23)} n ( S n uc − S n uc ) D G , G ∼ N ( 0 , Γ S , 0 uc ) , (23) and
n ∥ S ^ n u c ∥ H S u c 2 → D ∥ G + S h u c ∥ H S u c 2 . (24) n\|\widehat{S}_n^{\mathrm{uc}}\|_{\mathcal{H}_S^{\mathrm{uc}}}^{2}\xrightarrow{\mathcal{D}}\|G+S_h^{\mathrm{uc}}\|_{\mathcal{H}_S^{\mathrm{uc}}}^{2}. \tag*{(24)} n ∥ S n uc ∥ H S uc 2 D ∥ G + S h uc ∥ H S uc 2 . (24)
Suppose additionally that S h u c ∈ ker ( Γ S , 0 u c ) ⊥ S_h^{\mathrm{uc}} \in\ker(\Gamma_{S,0}^{\mathrm{uc}})^\perp S h uc ∈ ker ( Γ S , 0 uc ) ⊥ . If we adopt the decomposition
Γ S , 0 u c = ∑ k = 1 ∞ λ k ( v k ⊗ v k ) , S h u c = ∑ k = 1 ∞ σ k v k . \Gamma_{S,0}^{\mathrm{uc}} = \sum_{k=1}^{\infty} \lambda_k(v_k \otimes v_k), \qquad S_h^{\mathrm{uc}} = \sum_{k=1}^{\infty} \sigma_k v_k. Γ S , 0 uc = k = 1 ∑ ∞ λ k ( v k ⊗ v k ) , S h uc = k = 1 ∑ ∞ σ k v k .
where λ k > 0 \lambda_k > 0 λ k > 0 is the eigenvalue and v k v_k v k is the corresponding orthonormal eigenvectors, then
n ∥ S ^ n u c ∥ 2 ∥ Γ S u c ∥ → D ∑ k = 1 ∞ ( λ k Z k + σ k ) 2 . (25) n\frac{\lVert\hat{S}_n^{\mathrm{uc}}\rVert^2}{\lVert\Gamma_{S}^{\mathrm{uc}}\rVert} \xrightarrow{\mathcal{D}} \sum_{k=1}^{\infty} \left(\sqrt{\lambda_k}Z_k+\sigma_k\right)^2. \tag*{(25)} n ∥ Γ S uc ∥ ∥ S ^ n uc ∥ 2 D k = 1 ∑ ∞ ( λ k Z k + σ k ) 2 . (25)
where the Z k Z_k Z k are independent standard normal random variables.
Numerical Implementation
We now describe the numerical implementation of the proposed test. We first derive the coordinate mapping for the test statistics, then obtain a Gamma approximation to its null distribution, and finally discuss the selection of kernel bandwidths and the regularization parameter.
Coordinate Mapping for the Test Statistic
For j = 1 , … , d − 1 j = 1,\ldots,d-1 j = 1 , … , d − 1 , Let K X ¨ j ∈ R n × n K_{\ddot{X}_j} \in\mathbb{R}^{n\times n} K X ¨ j ∈ R n × n denote the Gram matrix with entries ( K X ¨ j ) i k = κ X ¨ j ( X ¨ j , i , X ¨ j , k ) (K_{\ddot{X}_j})_{ik} = \kappa_{\ddot{X}_j}(\ddot{X}_{j,i},\ddot{X}_{j,k}) ( K X ¨ j ) ik = κ X ¨ j ( X ¨ j , i , X ¨ j , k ) . Define H = I − n − 1 1 n 1 n ⊤ H = I-n^{-1}\mathbf{1}_n\mathbf{1}_n^\top H = I − n − 1 1 n 1 n ⊤ and K ^ X ¨ j = H K X ¨ j H \hat{K}_{\ddot{X}_j}=HK_{\ddot{X}_j}H K ^ X ¨ j = H K X ¨ j H . Similarly define K ^ X j + \hat{K}_{X_{j+}} K ^ X j + . Using the empirically centered feature maps, consider the spanning systems
B X ¨ j = { κ ^ X ¨ j ( ⋅ , X ¨ j , i ) : i = 1 , … , n } , B X j + = { κ ^ X j + ( ⋅ , X j + , i ) : i = 1 , … , n } . \mathcal{B}_{\ddot{X}_j}=\{\hat{\kappa}_{\ddot{X}_j}(\cdot,\ddot{X}_{j,i}):i=1,\ldots,n\}, \qquad\mathcal{B}_{X_{j+}}=\{\hat{\kappa}_{X_{j+}}(\cdot,X_{j+,i}):i=1,\ldots,n\}. B X ¨ j = { κ ^ X ¨ j ( ⋅ , X ¨ j , i ) : i = 1 , … , n } , B X j + = { κ ^ X j + ( ⋅ , X j + , i ) : i = 1 , … , n } .
Analogous notation is used for Z Z Z . Following Li [25 ] , the empirical covariance operators have the matrix representations
[ Σ ^ X ¨ j Z ] B X ¨ j B Z = n − 1 K ^ Z , [ Σ ^ Z Z ] B Z B Z = n − 1 K ^ Z , [ Σ ^ Z X j + ] B Z B X j + = n − 1 K ^ X j + . \left[\hat{\Sigma}_{\ddot{X}_j Z}\right]_{\mathcal{B}_{\ddot{X}_j}\mathcal{B}_Z}=n^{-1}\hat{K}_Z,\qquad
\left[\hat{\Sigma}_{ZZ}\right]_{\mathcal{B}_Z\mathcal{B}_Z}=n^{-1}\hat{K}_Z,\qquad
\left[\hat{\Sigma}_{ZX_{j+}}\right]_{\mathcal{B}_Z\mathcal{B}_{X_{j+}}}=n^{-1}\hat{K}_{X_{j+}}. [ Σ ^ X ¨ j Z ] B X ¨ j B Z = n − 1 K ^ Z , [ Σ ^ Z Z ] B Z B Z = n − 1 K ^ Z , [ Σ ^ Z X j + ] B Z B X j + = n − 1 K ^ X j + .
The same rules apply when either variable block is replaced by Z Z Z or an augmented block. Then the regularized regression operators therefore satisfy
[ B ^ X ¨ j ∣ Z ] B X ¨ j B Z = [ Σ ^ X ¨ j Z ] B X ¨ j B Z [ ( Σ ^ Z Z + ϵ n I ) − 1 ] B Z = 1 n K ^ Z ( 1 n K ^ Z + ϵ n I ) − 1 = K ^ Z ( K ^ Z + n ϵ n I ) − 1 . \left[\hat{B}_{\ddot{X}_j|Z}\right]_{\mathcal{B}_{\ddot{X}_j}\mathcal{B}_Z}
=
\left[\hat{\Sigma}_{\ddot{X}_j Z}\right]_{\mathcal{B}_{\ddot{X}_j}\mathcal{B}_Z}
\left[\left(\hat{\Sigma}_{ZZ}+\epsilon_n I\right)^{-1}\right]_{\mathcal{B}_Z}
=
\frac{1}{n}\hat{K}_Z\left(\frac{1}{n}\hat{K}_Z+\epsilon_n I\right)^{-1}
=
\hat{K}_Z(\hat{K}_Z+n\epsilon_n I)^{-1}. [ B ^ X ¨ j ∣ Z ] B X ¨ j B Z = [ Σ ^ X ¨ j Z ] B X ¨ j B Z [ ( Σ ^ Z Z + ϵ n I ) − 1 ] B Z = n 1 K ^ Z ( n 1 K ^ Z + ϵ n I ) − 1 = K ^ Z ( K ^ Z + n ϵ n I ) − 1 .
Similarly we get
[ B ^ X j + ∣ Z ∗ ] B Z B X j + = ( K ^ Z + n ϵ n I ) − 1 K ^ X j + . \left[\hat{B}_{X_{j+}|Z}^{*}\right]_{\mathcal{B}_Z\mathcal{B}_{X_{j+}}}
=
(\hat{K}_Z+n\epsilon_n I)^{-1}\hat{K}_{X_{j+}}. [ B ^ X j + ∣ Z ∗ ] B Z B X j + = ( K ^ Z + n ϵ n I ) − 1 K ^ X j + .
Applying them to the residual covariance formula gives
[ Σ ^ X ¨ j X j + ∣ Z ] B X ¨ j B X j + = [ Σ ^ X ¨ j X j + − B ^ X ¨ j ∣ Z Σ ^ Z X j + − Σ ^ X ¨ j Z B ^ X j + ∣ Z ∗ + B ^ X ¨ j ∣ Z Σ ^ Z Z B ^ X j + ∣ Z ∗ ] B X ¨ j B X j + = 1 n [ ( I − K ^ Z ( K ^ Z + n ϵ n I ) − 1 ) ] 2 K ^ X j + = 1 n R Z 2 K ^ X j + . \begin{aligned}
\left[\hat{\Sigma}_{\ddot{X}_jX_{j+}|Z}\right]_{\mathcal{B}_{\ddot{X}_j}\mathcal{B}_{X_{j+}}}
=
\left[\hat{\Sigma}_{\ddot{X}_jX_{j+}}-\hat{B}_{\ddot{X}_j|Z}\hat{\Sigma}_{ZX_{j+}}-\hat{\Sigma}_{\ddot{X}_jZ}\hat{B}_{X_{j+}|Z}^{*}+\hat{B}_{\ddot{X}_j|Z}\hat{\Sigma}_{ZZ}\hat{B}_{X_{j+}|Z}^{*}\right]_{\mathcal{B}_{\ddot{X}_j}\mathcal{B}_{X_{j+}}} \\
&=\frac{1}{n}\left[(I-\widehat{K}_Z(\widehat{K}_Z+n\epsilon_n I)^{-1})\right]^2\widehat{K}_{X_{j+}}=\frac{1}{n}R_Z^2\widehat{K}_{X_{j+}}.
\end{aligned} [ Σ ^ X ¨ j X j + ∣ Z ] B X ¨ j B X j + = [ Σ ^ X ¨ j X j + − B ^ X ¨ j ∣ Z Σ ^ Z X j + − Σ ^ X ¨ j Z B ^ X j + ∣ Z ∗ + B ^ X ¨ j ∣ Z Σ ^ Z Z B ^ X j + ∣ Z ∗ ] B X ¨ j B X j + = n 1 [ ( I − K Z ( K Z + n ϵ n I ) − 1 ) ] 2 K X j + = n 1 R Z 2 K X j + .
where R Z = I − K ^ Z ( K ^ Z + n ϵ n I ) − 1 = n ϵ n ( K ^ Z + n ϵ n I ) − 1 R_Z=I-\widehat{K}_Z\left(\widehat{K}_Z+n\epsilon_n I\right)^{-1}=n\epsilon_n\left(\widehat{K}_Z+n\epsilon_n I\right)^{-1} R Z = I − K Z ( K Z + n ϵ n I ) − 1 = n ϵ n ( K Z + n ϵ n I ) − 1 . Finally we obtain
∥ Σ ^ X j X j + ∣ Z ∥ H S 2 = 1 n 2 tr ( K ^ X j ∣ Z 1 2 R Z 2 K ^ X j + ∣ Z 1 2 K ^ X j + ∣ Z 1 2 R Z 2 K ^ X j ∣ Z 1 2 ) = 1 n 2 tr ( R Z K ^ X j R Z ⋅ R Z K ^ X j + R Z ) . \left\|\widehat{\Sigma}_{X_jX_{j+}|Z}\right\|_{\mathrm{HS}}^2=\frac{1}{n^2}\operatorname{tr}\left(\widehat{K}_{X_j|Z}^{\frac{1}{2}}R_Z^2\widehat{K}_{X_j+|Z}^{\frac{1}{2}}\widehat{K}_{X_j+|Z}^{\frac{1}{2}}R_Z^2\widehat{K}_{X_j|Z}^{\frac{1}{2}}\right)=\frac{1}{n^2}\operatorname{tr}\left(R_Z\widehat{K}_{X_j}R_Z\cdot R_Z\widehat{K}_{X_{j+}}R_Z\right). Σ X j X j + ∣ Z HS 2 = n 2 1 tr ( K X j ∣ Z 2 1 R Z 2 K X j + ∣ Z 2 1 K X j + ∣ Z 2 1 R Z 2 K X j ∣ Z 2 1 ) = n 2 1 tr ( R Z K X j R Z ⋅ R Z K X j + R Z ) .
We define K ^ X j ∣ Z = R Z K ^ X j R Z \widehat{K}_{X_j|Z}=R_Z\widehat{K}_{X_j}R_Z K X j ∣ Z = R Z K X j R Z and K ^ X j + ∣ Z = R Z K ^ X j + R Z \widehat{K}_{X_{j+}|Z}=R_Z\widehat{K}_{X_{j+}}R_Z K X j + ∣ Z = R Z K X j + R Z . Then we can conclude that
n ∥ S ^ ∥ H S 2 = n ∑ j = 1 d − 1 ∥ Σ ^ X j X j + ∣ Z ∥ H S 2 = 1 n ∑ j = 1 d − 1 tr ( K ^ X j ∣ Z ⋅ K ^ X j + ∣ Z ) . n\left\|\widehat{S}\right\|_{\mathrm{HS}}^2=n\sum_{j=1}^{d-1}\left\|\widehat{\Sigma}_{X_jX_{j+}|Z}\right\|_{\mathrm{HS}}^2=\frac{1}{n}\sum_{j=1}^{d-1}\operatorname{tr}\left(\widehat{K}_{X_j|Z}\cdot\widehat{K}_{X_{j+}|Z}\right). n S HS 2 = n j = 1 ∑ d − 1 Σ X j X j + ∣ Z HS 2 = n 1 j = 1 ∑ d − 1 tr ( K X j ∣ Z ⋅ K X j + ∣ Z ) .
Gamma Approximation to the Null Distribution
We now describe how to determine its rejection threshold under the null hypothesis. Under the null hypothesis, S = 0 S=0 S = 0 , the asymptotic null distribution of the test statistic is
T ^ = n ∥ S ^ ∥ H S 2 → D T = ∑ i = 1 ∞ λ i Z i 2 . \widehat{T}=n\left\|\widehat{S}\right\|_{\mathrm{HS}}^2\xrightarrow{\mathcal{D}}T=\sum_{i=1}^{\infty}\lambda_i Z_i^2. T = n S HS 2 D T = i = 1 ∑ ∞ λ i Z i 2 .
Although this representation characterizes the null distribution, it requires estimating the eigenvalues { λ i } i ≥ 1 \{\lambda_i\}_{i\geq1} { λ i } i ≥ 1 and then numerically evaluating the related weighted chi-square distribution. To obtain a simpler and computationally more efficient calibration, we follow Zhang et al. [53 ] and use a two-parameter Gamma distribution that matches the first two moments of T T T ; see also the equivalent Welch–Satterthwaite approximation in Tang and Li [47 ] . These moments only require tr ( Γ S ∣ Z ) \operatorname{tr}(\Gamma_{S|Z}) tr ( Γ S ∣ Z ) and tr ( Γ S ∣ Z 2 ) \operatorname{tr}(\Gamma_{S|Z}^2) tr ( Γ S ∣ Z 2 ) . To see this,
E [ T ] = tr ( Γ S ∣ Z ) , var ( T ) = 2 tr ( Γ S ∣ Z 2 ) . (26) \mathbb{E}[T]=\operatorname{tr}(\Gamma_{S|Z}),\qquad\operatorname{var}(T)=2\operatorname{tr}(\Gamma_{S|Z}^2). \tag*{(26)} E [ T ] = tr ( Γ S ∣ Z ) , var ( T ) = 2 tr ( Γ S ∣ Z 2 ) . (26)
Then we can use a Gamma distribution G a m m a ( α , β ) Gamma(\alpha,\beta) G amma ( α , β ) with shape parameter α = E [ T ] 2 var ( T ) \alpha=\frac{\mathbb{E}[T]^2}{\operatorname{var}(T)} α = var ( T ) E [ T ] 2 and rate parameter β = E [ T ] var ( T ) \beta=\frac{\mathbb{E}[T]}{\operatorname{var}(T)} β = var ( T ) E [ T ] to approximate the weighted sum of independent χ 2 \chi^2 χ 2 random variables as in Zhang et al. [53 ] .
The influence function ψ \psi ψ and its covariance operator Γ S ∣ Z \Gamma_{S|Z} Γ S ∣ Z are given in (8) and (10). Their empirical counterparts under the null hypothesis are
ψ ^ j , i = ξ ^ X j ∣ Z ( ⋅ , X j , i ; Z i ) ⊗ ξ ^ X j + ∣ Z ( ⋅ , X j + , i ; Z i ) , ψ ^ i = ( ψ ^ 1 , i , … , ψ ^ d − 1 , i ) T , \widehat{\psi}_{j,i}=\widehat{\xi}_{X_j|Z}(\cdot,X_{j,i};Z_i)\otimes\widehat{\xi}_{X_{j+}|Z}(\cdot,X_{j+,i};Z_i),\qquad\widehat{\psi}_i=(\widehat{\psi}_{1,i},\ldots,\widehat{\psi}_{d-1,i})^{\mathsf{T}}, ψ j , i = ξ X j ∣ Z ( ⋅ , X j , i ; Z i ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + , i ; Z i ) , ψ i = ( ψ 1 , i , … , ψ d − 1 , i ) T ,
Γ ^ S ∣ Z = E n [ ψ ^ ⊗ ψ ^ ] = 1 n ∑ i = 1 n ψ ^ i ⊗ ψ ^ i . \widehat{\Gamma}_{S|Z}=\mathbb{E}_n[\widehat{\psi}\otimes\widehat{\psi}]=\frac{1}{n}\sum_{i=1}^{n}\widehat{\psi}_i\otimes\widehat{\psi}_i. Γ S ∣ Z = E n [ ψ ⊗ ψ ] = n 1 i = 1 ∑ n ψ i ⊗ ψ i .
Indeed, Γ ^ S ∣ Z \widehat{\Gamma}_{S|Z} Γ S ∣ Z is a consistent estimator. We state this as a lemma below.
Lemma 3. Under the same assumptions as in Lemma 2 , we have
∥ Γ ^ S ∣ Z − Γ S ∣ Z ∥ t r = o P ( 1 ) . \left\|\widehat{\Gamma}_{S\mid Z}-\Gamma_{S\mid Z}\right\|_{\mathrm{tr}}=o_P(1). Γ S ∣ Z − Γ S ∣ Z tr = o P ( 1 ) .
Consequently, we obtain consistent estimates of the first two moments of T T T by replacing Γ S ∣ Z \Gamma_{S\mid Z} Γ S ∣ Z with Γ ^ S ∣ Z \widehat{\Gamma}_{S\mid Z} Γ S ∣ Z . The following proposition gives the corresponding matrix expressions, which are needed for Gamma approximation.
Proposition 3. Under the null hypothesis S = 0 S=0 S = 0 , the plug-in estimates of mean E [ T ] \mathbb{E}[T] E [ T ] and variance var ( T ) \operatorname{var}(T) var ( T ) in (26) are
tr ( Γ ^ S ∣ Z ) = 1 n ∑ j = 1 d − 1 tr ( K ^ X j ∣ Z ∘ K ^ X j + ∣ Z ) . \operatorname{tr}(\widehat{\Gamma}_{S\mid Z})=\frac{1}{n}\sum_{j=1}^{d-1}\operatorname{tr}\left(\widehat{K}_{X_j\mid Z}\circ\widehat{K}_{X_{j+}\mid Z}\right). tr ( Γ S ∣ Z ) = n 1 j = 1 ∑ d − 1 tr ( K X j ∣ Z ∘ K X j + ∣ Z ) . 2 tr ( Γ ^ S ∣ Z 2 ) = 2 n 2 ∑ j = 1 d − 1 ∑ k = 1 d − 1 tr [ ( K ^ X j ∣ Z ∘ K ^ X j + ∣ Z ) ( K ^ X k ∣ Z ∘ K ^ X k + ∣ Z ) ] . 2\operatorname{tr}(\widehat{\Gamma}_{S\mid Z}^{2})=\frac{2}{n^{2}}\sum_{j=1}^{d-1}\sum_{k=1}^{d-1}\operatorname{tr}\left[\left(\widehat{K}_{X_j\mid Z}\circ\widehat{K}_{X_{j+}\mid Z}\right)\left(\widehat{K}_{X_k\mid Z}\circ\widehat{K}_{X_{k+}\mid Z}\right)\right]. 2 tr ( Γ S ∣ Z 2 ) = n 2 2 j = 1 ∑ d − 1 k = 1 ∑ d − 1 tr [ ( K X j ∣ Z ∘ K X j + ∣ Z ) ( K X k ∣ Z ∘ K X k + ∣ Z ) ] .
Then the asymptotic distribution of T ^ \widehat{T} T can be approximated by a G a m m a ( α ^ , β ^ ) Gamma(\widehat{\alpha},\widehat{\beta}) G amma ( α , β ) distribution, where α ^ = tr ( Γ ^ S ∣ Z ) 2 2 tr ( Γ ^ S ∣ Z 2 ) \widehat{\alpha}=\frac{\operatorname{tr}(\widehat{\Gamma}_{S\mid Z})^{2}}{2\operatorname{tr}(\widehat{\Gamma}_{S\mid Z}^{2})} α = 2 tr ( Γ S ∣ Z 2 ) tr ( Γ S ∣ Z ) 2 , β ^ = tr ( Γ ^ S ∣ Z ) 2 tr ( Γ ^ S ∣ Z 2 ) \widehat{\beta}=\frac{\operatorname{tr}(\widehat{\Gamma}_{S\mid Z})}{2\operatorname{tr}(\widehat{\Gamma}_{S\mid Z}^{2})} β = 2 tr ( Γ S ∣ Z 2 ) tr ( Γ S ∣ Z ) .
Tuning
We use Gaussian radial basis functions as kernels, equipped with a bandwidth parameter h h h . For each X J X_J X J with J ⊂ { 1 , … , d } J\subset\{1,\ldots,d\} J ⊂ { 1 , … , d } and Z Z Z , we choose the bandwidth parameter according to the median pairwise distances among the observations. Specifically, letting { X J , i } i = 1 n \{X_{J,i}\}_{i=1}^{n} { X J , i } i = 1 n denote the sample of a generic variable X J X_J X J , the bandwidth is chosen as
h X J = median { ∥ X J , i − X J , j ∥ : 1 ≤ i < j ≤ n } . h_{X_J}=\operatorname{median}\left\{\left\|X_{J,i}-X_{J,j}\right\|:1\leq i<j\leq n\right\}. h X J = median { ∥ X J , i − X J , j ∥ : 1 ≤ i < j ≤ n } .
and the Gaussian kernel is
κ X J ( x , x ′ ) = exp ( − ∥ x − x ′ ∥ 2 2 h X J 2 ) . \kappa_{X_J}(x,x')=\exp\left(-\frac{\left\|x-x'\right\|^{2}}{2h_{X_J}^{2}}\right). κ X J ( x , x ′ ) = exp ( − 2 h X J 2 ∥ x − x ′ ∥ 2 ) .
As an analogy to Li and Solea [25 ] , we propose a GCV criterion based on the unaugmented feature maps of X j X_j X j and X j + X_{j+} X j + for the covariance operator Σ X j X j + ∣ Z \Sigma_{X_jX_{j+}\mid Z} Σ X j X j + ∣ Z , for j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 , as below. Denote the smoothing matrix
P Z ϵ n = K ^ Z ( K ^ Z + n ϵ n I ) − 1 . P_Z^{\epsilon_n}=\widehat{K}_Z\left(\widehat{K}_Z+n\epsilon_n I\right)^{-1}. P Z ϵ n = K Z ( K Z + n ϵ n I ) − 1 .
Using this matrix, we define
GCV X j X j + ( ϵ n ) = ∥ K ^ X j ∘ K ^ X j + − ( P Z ϵ n K ^ X j ) ∘ ( P Z ϵ n K ^ X j + ) ∥ F 2 ( 1 − n − 1 tr [ ( K ^ Z + n ϵ n I ) − 1 K ^ Z ] ) 2 . (27) \operatorname{GCV}_{X_jX_{j+}}(\epsilon_n)=\frac{\left\|\widehat{K}_{X_j}\circ\widehat{K}_{X_{j+}}-\left(P_Z^{\epsilon_n}\widehat{K}_{X_j}\right)\circ\left(P_Z^{\epsilon_n}\widehat{K}_{X_{j+}}\right)\right\|_{F}^{2}}{\left(1-n^{-1}\operatorname{tr}\left[(\widehat{K}_Z+n\epsilon_n I)^{-1}\widehat{K}_Z\right]\right)^{2}}. \tag*{(27)} GCV X j X j + ( ϵ n ) = ( 1 − n − 1 tr [ ( K Z + n ϵ n I ) − 1 K Z ] ) 2 K X j ∘ K X j + − ( P Z ϵ n K X j ) ∘ ( P Z ϵ n K X j + ) F 2 . (27)
Here we use ( P Z ε n K ^ X j ) ∘ ( P Z ε n K ^ X j + ) (P_{Z}^{\varepsilon_n}\widehat{K}_{X_j})\circ(P_{Z}^{\varepsilon_n}\widehat{K}_{X_{j+}}) ( P Z ε n K X j ) ∘ ( P Z ε n K X j + ) to represent the Z Z Z -explained component obtained by separately smoothing the two marginal feature maps. Hence for our S = ( Σ ^ X 1 X 1 + ∣ Z , … , Σ ^ X d − 1 X d − 1 + ∣ Z ) T S=(\widehat{\Sigma}_{X_1X_{1+}|Z},\ldots,\widehat{\Sigma}_{X_{d-1}X_{d-1+}|Z})^{\mathrm{T}} S = ( Σ X 1 X 1 + ∣ Z , … , Σ X d − 1 X d − 1 + ∣ Z ) T , we use the following practical tuning criterion:
GCV S ( ε n ) = ∑ j = 1 d − 1 GCV X j X j + ( ε n ) . \operatorname{GCV}_{S}(\varepsilon_n)=\sum_{j=1}^{d-1}\operatorname{GCV}_{X_jX_{j+}}(\varepsilon_n). GCV S ( ε n ) = j = 1 ∑ d − 1 GCV X j X j + ( ε n ) .
Then the regularization parameter ε n \varepsilon_n ε n is selected by minimizing GCV S \operatorname{GCV}_{S} GCV S .
Unconditional Case
In this section, we discuss the coordinate mapping and gamma approximation in the unconditional case. There is no Tikhonov regularization parameter ε n \varepsilon_n ε n involved, as we will see immediately in the coordinate mapping. Recall that
B X j [ Σ ^ X j X j + ] B X j + = 1 n K ^ X j + . \mathcal{B}_{X_j}\left[\widehat{\Sigma}_{X_jX_{j+}}\right]_{\mathcal{B}_{X_{j+}}}=\frac{1}{n}\widehat{K}_{X_{j+}}. B X j [ Σ X j X j + ] B X j + = n 1 K X j + .
Its Hilbert–Schmidt norm is
∥ Σ ^ X j X j + ∥ H S 2 = ∥ K ^ X j 1 2 [ Σ ^ X j X j + ] B X j K ^ X j + 1 2 ∥ F 2 = ∥ 1 n K ^ X j 1 2 K ^ X j + 1 2 ∥ F 2 = 1 n 2 tr ( K ^ X j 1 2 K ^ X j + 1 2 ⋅ K ^ X j + 1 2 K ^ X j 1 2 ) = 1 n 2 tr ( K ^ X j ⋅ K ^ X j + ) . \begin{aligned}
\left\|\widehat{\Sigma}_{X_jX_{j+}}\right\|_{\mathrm{HS}}^{2}
&=\left\|\widehat{K}_{X_j}^{\frac{1}{2}}\left[\widehat{\Sigma}_{X_jX_{j+}}\right]_{\mathcal{B}_{X_j}}\widehat{K}_{X_{j+}}^{\frac{1}{2}}\right\|_{\mathrm{F}}^{2}
=\left\|\frac{1}{n}\widehat{K}_{X_j}^{\frac{1}{2}}\widehat{K}_{X_{j+}}^{\frac{1}{2}}\right\|_{\mathrm{F}}^{2}\\
&=\frac{1}{n^{2}}\operatorname{tr}\left(\widehat{K}_{X_j}^{\frac{1}{2}}\widehat{K}_{X_{j+}}^{\frac{1}{2}}\cdot\widehat{K}_{X_{j+}}^{\frac{1}{2}}\widehat{K}_{X_j}^{\frac{1}{2}}\right)
=\frac{1}{n^{2}}\operatorname{tr}\left(\widehat{K}_{X_j}\cdot\widehat{K}_{X_{j+}}\right).
\end{aligned} Σ X j X j + HS 2 = K X j 2 1 [ Σ X j X j + ] B X j K X j + 2 1 F 2 = n 1 K X j 2 1 K X j + 2 1 F 2 = n 2 1 tr ( K X j 2 1 K X j + 2 1 ⋅ K X j + 2 1 K X j 2 1 ) = n 2 1 tr ( K X j ⋅ K X j + ) .
Then we get
n ∥ S ^ u c ∥ H S u c 2 = n ∑ j = 1 d − 1 ∥ Σ ^ X j X j + ∥ H S 2 = 1 n ∑ j = 1 d − 1 tr ( K ^ X j ⋅ K ^ X j + ) . (28) n\left\|\widehat{S}^{\mathrm{uc}}\right\|_{\mathcal{H}_{S}^{\mathrm{uc}}}^{2}
=n\sum_{j=1}^{d-1}\left\|\widehat{\Sigma}_{X_jX_{j+}}\right\|_{\mathrm{HS}}^{2}
=\frac{1}{n}\sum_{j=1}^{d-1}\operatorname{tr}\left(\widehat{K}_{X_j}\cdot\widehat{K}_{X_{j+}}\right). \tag*{(28)} n S uc H S uc 2 = n j = 1 ∑ d − 1 Σ X j X j + HS 2 = n 1 j = 1 ∑ d − 1 tr ( K X j ⋅ K X j + ) . (28)
We have shown that the influence function ψ u c \psi^{\mathrm{uc}} ψ uc and covariance Γ S u c \Gamma_{S}^{\mathrm{uc}} Γ S uc are given in (20) . Their empirical counterparts under the null hypothesis are
ψ ^ j , i u c = κ ^ X j ( ⋅ , X j , i ) ⊗ κ ^ X j + ( ⋅ , X j + , i ) , ψ ^ i u c = ( ψ ^ 1 , i u c , … , ψ ^ d − 1 , i u c ) T , \widehat{\psi}_{j,i}^{\mathrm{uc}}=\widehat{\kappa}_{X_j}(\cdot,X_{j,i})\otimes\widehat{\kappa}_{X_{j+}}(\cdot,X_{j+,i}),\qquad
\widehat{\psi}_{i}^{\mathrm{uc}}=\left(\widehat{\psi}_{1,i}^{\mathrm{uc}},\ldots,\widehat{\psi}_{d-1,i}^{\mathrm{uc}}\right)^{\mathrm{T}}, ψ j , i uc = κ X j ( ⋅ , X j , i ) ⊗ κ X j + ( ⋅ , X j + , i ) , ψ i uc = ( ψ 1 , i uc , … , ψ d − 1 , i uc ) T ,
Γ ^ S u c = E n [ ψ ^ u c ⊗ ψ ^ u c ] = 1 n ∑ i = 1 n ψ ^ i u c ⊗ ψ ^ i u c . \widehat{\Gamma}_{S}^{\mathrm{uc}}=\mathbb{E}_{n}\left[\widehat{\psi}^{\mathrm{uc}}\otimes\widehat{\psi}^{\mathrm{uc}}\right]
=\frac{1}{n}\sum_{i=1}^{n}\widehat{\psi}_{i}^{\mathrm{uc}}\otimes\widehat{\psi}_{i}^{\mathrm{uc}}. Γ S uc = E n [ ψ uc ⊗ ψ uc ] = n 1 i = 1 ∑ n ψ i uc ⊗ ψ i uc .
Under the null hypothesis, S u c = 0 S^{\mathrm{uc}}=0 S uc = 0 , the asymptotic null distribution of the test statistic is
T ^ u c = n ∥ S ^ u c ∥ H S u c 2 → D T u c = ∑ i = 1 ∞ λ i Z i 2 . \widehat{T}^{\mathrm{uc}}=n\left\|\widehat{S}^{\mathrm{uc}}\right\|_{\mathcal{H}_{S}^{\mathrm{uc}}}^{2}\xrightarrow{D}T^{\mathrm{uc}}=\sum_{i=1}^{\infty}\lambda_iZ_i^{2}. T uc = n S uc H S uc 2 D T uc = i = 1 ∑ ∞ λ i Z i 2 .
Again we use a Gamma distribution G a m m a ( α , β ) Gamma(\alpha,\beta) G amma ( α , β ) with α = E [ T u c ] 2 var ( T u c ) \alpha=\frac{\mathbb{E}[T^{\mathrm{uc}}]^{2}}{\operatorname{var}(T^{\mathrm{uc}})} α = var ( T uc ) E [ T uc ] 2 , β = E [ T u c ] var ( T u c ) \beta=\frac{\mathbb{E}[T^{\mathrm{uc}}]}{\operatorname{var}(T^{\mathrm{uc}})} β = var ( T uc ) E [ T uc ] to approximate the weighted sum of χ 2 \chi^{2} χ 2 distribution
E [ T u c ] = tr ( Γ S u c ) , var ( T u c ) = 2 tr ( ( Γ S u c ) 2 ) . \mathbb{E}[T^{\mathrm{uc}}]=\operatorname{tr}(\Gamma_{S}^{\mathrm{uc}}),\qquad
\operatorname{var}(T^{\mathrm{uc}})=2\operatorname{tr}\left((\Gamma_{S}^{\mathrm{uc}})^{2}\right). E [ T uc ] = tr ( Γ S uc ) , var ( T uc ) = 2 tr ( ( Γ S uc ) 2 ) .
Proposition 4. Under the null hypothesis S u c = 0 S^{\mathrm{uc}}=0 S uc = 0 , the plug-in estimates of mean E [ T ] \mathbb{E}[T] E [ T ] and variance var ( T ) \operatorname{var}(T) var ( T ) above are
tr ( Γ ^ S u c ) = 1 n ∑ j = 1 d − 1 tr ( K ^ X j ∘ K ^ X j + ) , \operatorname{tr}(\widehat{\Gamma}^{\mathrm{uc}}_S)=\frac{1}{n}\sum_{j=1}^{d-1}\operatorname{tr}\left(\widehat{K}_{X_j}\circ\widehat{K}_{X_{j+}}\right), tr ( Γ S uc ) = n 1 j = 1 ∑ d − 1 tr ( K X j ∘ K X j + ) , 2 tr ( ( Γ ^ S u c ) 2 ) = 2 n 2 ∑ j = 1 d − 1 ∑ k = 1 d − 1 tr [ ( K ^ X j ∘ K ^ X j + ) ( K ^ X k ∘ K ^ X k + ) ] . 2\operatorname{tr}\left((\widehat{\Gamma}^{\mathrm{uc}}_S)^2\right)=\frac{2}{n^2}\sum_{j=1}^{d-1}\sum_{k=1}^{d-1}\operatorname{tr}\left[\left(\widehat{K}_{X_j}\circ\widehat{K}_{X_{j+}}\right)\left(\widehat{K}_{X_k}\circ\widehat{K}_{X_{k+}}\right)\right]. 2 tr ( ( Γ S uc ) 2 ) = n 2 2 j = 1 ∑ d − 1 k = 1 ∑ d − 1 tr [ ( K X j ∘ K X j + ) ( K X k ∘ K X k + ) ] . where Γ ^ S u c \widehat{\Gamma}^{\mathrm{uc}}_S Γ S uc is the plug-in estimator of the covariance operator Γ S u c \Gamma^{\mathrm{uc}}_S Γ S uc under null hypothesis.
The proof is parallel to Proposition 3, and thus omitted. Then similarly to § 6.2 , the asymptotic distribution of T ^ u c \widehat{T}^{\mathrm{uc}} T uc can be approximated by a G a m m a ( α ^ , β ^ ) Gamma(\widehat{\alpha},\widehat{\beta}) G amma ( α , β ) distribution, where α ^ = tr ( ( Γ ^ S u c ) 2 ) 2 tr ( ( Γ ^ S u c ) 2 ) \widehat{\alpha}=\frac{\operatorname{tr}((\widehat{\Gamma}^{\mathrm{uc}}_S)^2)}{2\operatorname{tr}((\widehat{\Gamma}^{\mathrm{uc}}_S)^2)} α = 2 tr (( Γ S uc ) 2 ) tr (( Γ S uc ) 2 ) , β ^ = tr ( Γ ^ S u c ) 2 tr ( ( Γ ^ S u c ) 2 ) \widehat{\beta}=\frac{\operatorname{tr}(\widehat{\Gamma}^{\mathrm{uc}}_S)}{2\operatorname{tr}((\widehat{\Gamma}^{\mathrm{uc}}_S)^2)} β = 2 tr (( Γ S uc ) 2 ) tr ( Γ S uc ) .
Simulation and Application
Simulation with Linear and Nonlinear Models
We evaluate the finite-sample behavior of the proposed multi-way conditional independence test under linear and nonlinear models. Throughout, we take d = 4 d=4 d = 4 and let Z ∈ R 2 Z\in\mathbb{R}^2 Z ∈ R 2 . For each replication, we generate Z ∼ N ( 0 , I 2 ) Z\sim N(0,I_2) Z ∼ N ( 0 , I 2 ) and ϵ ∼ N ( 0 , Ψ ρ ) \epsilon\sim N(0,\Psi_\rho) ϵ ∼ N ( 0 , Ψ ρ ) independently, where ϵ = ( ϵ 1 , … , ϵ 4 ) T \epsilon=(\epsilon_1,\ldots,\epsilon_4)^{\mathsf{T}} ϵ = ( ϵ 1 , … , ϵ 4 ) T ,
( Ψ ρ ) j k = ρ ∣ j − k ∣ , j , k = 1 , … , 4. (\Psi_\rho)_{jk}=\rho^{|j-k|},\qquad j,k=1,\ldots,4. ( Ψ ρ ) j k = ρ ∣ j − k ∣ , j , k = 1 , … , 4.
Here ρ = 0 \rho=0 ρ = 0 corresponds to the conditional independence case, whereas increasing ρ \rho ρ produces stronger conditional dependence. We consider ρ ∈ { 0 , 0.2 , 0.4 , 0.6 , 0.8 } \rho\in\{0,0.2,0.4,0.6,0.8\} ρ ∈ { 0 , 0.2 , 0.4 , 0.6 , 0.8 } , n ∈ { 100 , 500 } n\in\{100,500\} n ∈ { 100 , 500 } , with 200 replications for each combination of model, sample size, and ρ \rho ρ .
The five models of the data-generating process are summarized in Table 1 . Polynomials of Z Z Z in Model 3 and product with ϵ \epsilon ϵ in Model 5 are taken component-wise. The coefficient matrices are given in Table 1 . We therefore examine whether the proposed test remains effective when the effect of Z Z Z is linear, nonlinear, or enters through the conditional variance rather than only through the conditional mean.
Table 1. Linear and nonlinear simulation models
For every generated data set, Gaussian radial basis function kernels are used. Kernel bandwidths are selected by the median pairwise-distance rule separately for each X j X_j X j , for Z Z Z , and for each grouped vector X j + X_{j+} X j + , where d = 4 d=4 d = 4 . For each replication, ϵ n \epsilon_n ϵ n is chosen by minimizing the GCV criterion discussed in § 6.3 over 51 logarithmically equally spaced values in [ 10 − 5 , 10 − 1 ] [10^{-5},10^{-1}] [ 1 0 − 5 , 1 0 − 1 ] . After conditional residualization, we compute n ∥ S ^ ∥ H S 2 n\|\widehat{S}\|_{\mathcal{H}_S}^2 n ∥ S ∥ H S 2 and approximate its null distribution by the moment-matched Gamma distribution. In most scenarios, the selected ϵ n \epsilon_n ϵ n values are not at the boundary, and the GCV curve is nearly flat when they are close to it. The resulting test rejects at significance level α = 0.05 \alpha=0.05 α = 0.05 .
Figure 1 displays the distributions of the resulting p p p -values. Under the null case ρ = 0 \rho= 0 ρ = 0 , the p p p -values are broadly spread over [ 0 , 1 ] [0, 1] [ 0 , 1 ] for all five models and for both sample sizes, with medians near 0.5. This behavior is consistent with the intended null calibration and, importantly, persists across all four nonlinear mechanisms, including the heteroscedastic Model 5. For n = 100 n = 100 n = 100 , weak residual dependence may be difficult to detect, whereas the test becomes substantially more reliable as the sample size increases.
Figure 1. p p p -value distributions for the five simulation models. Columns correspond to Models 1–5 and rows to n = 100 n = 100 n = 100 and n = 500 n = 500 n = 500 . Within each panel, boxplots are shown for ρ ∈ { 0 , 0.2 , 0.4 , 0.6 , 0.8 } \rho\in\{0, 0.2, 0.4, 0.6, 0.8\} ρ ∈ { 0 , 0.2 , 0.4 , 0.6 , 0.8 } . The red dashed line marks the nominal significance level α = 0.05 \alpha= 0.05 α = 0.05 . Each boxplot is based on 200 Monte Carlo replications.
Application to the OneK1K Single-Cell RNA-seq Data
Gene co-expression can arise from multiple, potentially overlapping sources. At the molecular level, coordinated gene expression may arise from regulation by common transcription factors [2 ] , coordinated downstream responses to common signaling pathways [37 ] , or shared or co-active cis-regulatory elements [41 ] . At the cellular level, observed co-expression may reflect heterogeneity in cell identity and cell state [22 ] . At the population level, genetic background, age, sex, and external exposures, and disease conditions can generate context- and cell-type-specific expression variation [40 , 37 , 50 ] (Su et al. 2023). Moreover, technical features of single-cell RNA sequencing, including sequencing-depth variation and measurement error, may create distortion (Su et al., 2023). Since these mechanisms are complex and only partially observed, latent factors are therefore commonly used to account for otherwise unexplained shared variation [4 , 49 ] .
The proposed test provides a practical tool for evaluating a hypothesized explanation: By conditioning on the biological, technical, or even latent covariates, it assesses whether any multi-way dependence among the gene-expression modules remains unexplained. In this paper, we only consider a very simple case. We compare two conditioning sets:
Z 1 = { batch , log ( 1 + nCount RNA ) , log ( 1 + nFeature RNA ) } ; Z_1 = \{\text{batch}, \log(1 + \text{nCount RNA}), \log(1 + \text{nFeature RNA})\}; Z 1 = { batch , log ( 1 + nCount RNA ) , log ( 1 + nFeature RNA )} ;
Z 2 = Z 1 ∪ { cell type } . Z_2 = Z_1 \cup\{\text{cell type}\}. Z 2 = Z 1 ∪ { cell type } .
Thus, Z 1 Z_1 Z 1 adjusts for major technical variation, while Z 2 Z_2 Z 2 additionally removes variation associated with cell-type heterogeneity.
We analyze the OneK1K data, a large population-scale single-cell RNA-sequencing study of peripheral blood mononuclear cells [50 ] . The data release used in the present analysis contains cells from 980 donors classified into 14 annotated cell types, together with cell-level technical measurements and donor-level covariates.
For each cell i i i and each predefined gene set G j G_j G j , we define the corresponding module score as
X i j = 1 ∣ G j ∣ ∑ g ∈ G j Y i g − Y ˉ g s g , j = 1 , 2 , 3 , X_{ij} = \frac{1}{\lvert G_j\rvert}\sum_{g \in G_j}\frac{Y_{ig} - \bar{Y}_g}{s_g}, \qquad j = 1, 2, 3, X ij = ∣ G j ∣ 1 g ∈ G j ∑ s g Y i g − Y ˉ g , j = 1 , 2 , 3 ,
where Y i g Y_{ig} Y i g is the normalized expression of gene g g g in cell i i i , and Y ˉ g \bar{Y}_g Y ˉ g and s g s_g s g are, respectively, the sample mean and sample standard deviation of that gene across the analyzed cells. Genes that were not detected or had zero sample variance were excluded before the scores were constructed.
We focus on the following three modules:
X 1 X_1 X 1 : classical-monocyte identity,
G 1 = { L Y Z , S 100 A 8 , S 100 A 9 , C T S S , L S T 1 , V C A N } ; \mathcal{G}_1 = \{\mathit{LYZ},\mathit{S100A8},\mathit{S100A9},\mathit{CTSS},\mathit{LST1},\mathit{VCAN}\}; G 1 = { LYZ , S100A8 , S100A9 , CTSS , LST1 , VCAN } ;
X 2 : inflammatory NF- κ B signaling , X_2 : \text{inflammatory NF-}\kappa\text{B signaling}, X 2 : inflammatory NF- κ B signaling ,
G 2 = { N F K B I A , R E L B , T N F A I P 3 , C X C L 8 , C X C L 2 , P T G S 2 , I E R 3 , B C L 3 } ; \mathcal{G}_2 = \{\mathit{NFKBIA},\mathit{RELB},\mathit{TNFAIP3},\mathit{CXCL8},\mathit{CXCL2},\mathit{PTGS2},\mathit{IER3},\mathit{BCL3}\}; G 2 = { NFKBIA , RELB , TNFAIP3 , CXCL8 , CXCL2 , PTGS2 , IER3 , BCL3 } ;
X 3 : oxidative-stress response , X_3 : \text{oxidative-stress response}, X 3 : oxidative-stress response ,
G 3 = { H M O X 1 , S O D 2 , P R D X 1 , T X N , N Q O 1 } . \mathcal{G}_3 = \{\mathit{HMOX1},\mathit{SOD2},\mathit{PRDX1},\mathit{TXN},\mathit{NQO1}\}. G 3 = { HMOX1 , SOD2 , PRDX1 , TXN , NQO1 } .
The first score represents classical-monocyte identity, whereas the other two capture inflammatory and oxidative-stress states that may vary within cell types. Classical monocytes exhibit prominent innate immune and inflammatory programs, for which NF-κ \kappa κ B is a key transcriptional regulator [48 , 35 ] . The inflammatory and oxidative-stress modules may interact because ROS can modulate NF-κ \kappa κ B activation, while NF-κ \kappa κ B regulates ROS production and antioxidant defenses [34 ] . For each n ∈ { 2000 , 4000 , 6000 } n \in\{2000, 4000, 6000\} n ∈ { 2000 , 4000 , 6000 } , cells are sampled stratified by cell type and batch.
Table 2. P-values for the multiway conditional independence test of the classical-monocyte, inflammatory NF-κ \kappa κ B, and oxidative-stress modules.
These results suggest that, for this triplet, the observed multi-way dependence is attributable to differences among cell types rather than to a stable pattern of coordinated within-cell-type variation. After additionally conditioning on cell type, the dependence is no longer significant. Consistent with the simulation results, the distinction between the two conditioning schemes becomes clearer as the sample size increases.
Appendix
Proofs for Section 4
Proof of Theorem 3. Recall we have the conditional mean-zero property of residual terms in (2) :
E [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ∣ Z ] = 0 almost surely , E [ ξ X ~ J ∣ Z ( ⋅ , X ~ J ; Z ) ∣ Z ] = 0 almost surely . \mathbb{E}\left[\xi_{X_J\mid Z}(\cdot,X_J;Z)\mid Z\right]=0\ \text{almost surely}, \qquad\mathbb{E}\left[\xi_{\widetilde{X}_J\mid Z}(\cdot,\widetilde{X}_J;Z)\mid Z\right]=0\ \text{almost surely}. E [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ∣ Z ] = 0 almost surely , E [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ∣ Z ] = 0 almost surely .
Consequently,
E [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] = 0. \mathbb{E}\left[\xi_{X_J\mid Z}(\cdot,X_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]=0. E [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ Z ( ⋅ , Z ) ] = 0.
Recall that κ ~ X J ( ⋅ , X J ) = B X J ∣ Z κ ~ Z ( ⋅ , Z ) + ξ X J ∣ Z ( ⋅ , X J ; Z ) \widetilde{\kappa}_{X_J}(\cdot,X_J)=B_{X_J\mid Z}\widetilde{\kappa}_Z(\cdot,Z)+\xi_{X_J\mid Z}(\cdot,X_J;Z) κ X J ( ⋅ , X J ) = B X J ∣ Z κ Z ( ⋅ , Z ) + ξ X J ∣ Z ( ⋅ , X J ; Z ) . It follows from the definitions of the sample-level covariance operators that
Σ ^ X J Z = B X J ∣ Z Σ ^ Z Z + E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] − E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ] ⊗ E n [ κ ~ Z ( ⋅ , Z ) ] . \widehat{\Sigma}_{X_JZ}=B_{X_J\mid Z}\widehat{\Sigma}_{ZZ}+\mathbb{E}_n\left[\xi_{X_J\mid Z}(\cdot,X_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]-\mathbb{E}_n\left[\xi_{X_J\mid Z}(\cdot,X_J;Z)\right]\otimes\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\right]. Σ X J Z = B X J ∣ Z Σ Z Z + E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ Z ( ⋅ , Z ) ] − E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ] ⊗ E n [ κ Z ( ⋅ , Z ) ] .
By the resolvent identity
( Σ ^ Z Z + ϵ n I ) − 1 − ( Σ Z Z + ϵ n I ) − 1 = ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ ^ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 , (29) (\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}-(\Sigma_{ZZ}+\epsilon_n I)^{-1}=(\Sigma_{ZZ}+\epsilon_n I)^{-1}(\Sigma_{ZZ}-\widehat{\Sigma}_{ZZ})(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}, \tag*{(29)} ( Σ Z Z + ϵ n I ) − 1 − ( Σ Z Z + ϵ n I ) − 1 = ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ Z Z ) ( Σ Z Z + ϵ n I ) − 1 , (29)
we have
B ^ X J ∣ Z − B X J ∣ Z = − ϵ n B X J ∣ Z ( Σ ^ Z Z + ϵ n I ) − 1 + E n [ ξ ~ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ ^ Z Z + ϵ n I ) − 1 − ( E n [ ξ ~ X J ∣ Z ( ⋅ , X J ; Z ) ] ⊗ E n [ κ ~ Z ( ⋅ , Z ) ] ) ( Σ ^ Z Z + ϵ n I ) − 1 = − ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 − ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ ^ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 + E n [ ξ ~ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 + E n [ ξ ~ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ ^ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 − ( E n [ ξ ~ X J ∣ Z ( ⋅ , X J ; Z ) ] ⊗ E n [ κ ~ Z ( ⋅ , Z ) ] ) ( Σ ^ Z Z + ϵ n I ) − 1 . (30) \begin{aligned}
\widehat{B}_{X_J\mid Z}-B_{X_J\mid Z}
={}&-\epsilon_n B_{X_J\mid Z}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}
+\mathbb{E}_n[\widetilde{\xi}_{X_J\mid Z}(\cdot,X_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)](\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
&-(\mathbb{E}_n[\widetilde{\xi}_{X_J\mid Z}(\cdot,X_J;Z)]\otimes\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)])(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
={}&-\epsilon_n B_{X_J\mid Z}(\Sigma_{ZZ}+\epsilon_n I)^{-1}
-\epsilon_n B_{X_J\mid Z}(\Sigma_{ZZ}+\epsilon_n I)^{-1}(\Sigma_{ZZ}-\widehat{\Sigma}_{ZZ})(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
&+\mathbb{E}_n[\widetilde{\xi}_{X_J\mid Z}(\cdot,X_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)](\Sigma_{ZZ}+\epsilon_n I)^{-1}\\
&+\mathbb{E}_n[\widetilde{\xi}_{X_J\mid Z}(\cdot,X_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)](\Sigma_{ZZ}+\epsilon_n I)^{-1}(\Sigma_{ZZ}-\widehat{\Sigma}_{ZZ})(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
&-(\mathbb{E}_n[\widetilde{\xi}_{X_J\mid Z}(\cdot,X_J;Z)]\otimes\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)])(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}. \tag*{(30)}
\end{aligned} B X J ∣ Z − B X J ∣ Z = = − ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 + E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ Z ( ⋅ , Z )] ( Σ Z Z + ϵ n I ) − 1 − ( E n [ ξ X J ∣ Z ( ⋅ , X J ; Z )] ⊗ E n [ κ Z ( ⋅ , Z )]) ( Σ Z Z + ϵ n I ) − 1 − ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 − ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ Z Z ) ( Σ Z Z + ϵ n I ) − 1 + E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ Z ( ⋅ , Z )] ( Σ Z Z + ϵ n I ) − 1 + E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ Z ( ⋅ , Z )] ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ Z Z ) ( Σ Z Z + ϵ n I ) − 1 − ( E n [ ξ X J ∣ Z ( ⋅ , X J ; Z )] ⊗ E n [ κ Z ( ⋅ , Z )]) ( Σ Z Z + ϵ n I ) − 1 . (30)
We control the five terms in (30) separately. For the first term, by Assumption 3,
B X J ∣ Z = Σ X J Z Σ Z Z † = S X J Z Σ Z Z 1 + β Σ Z Z † . B_{X_J\mid Z}=\Sigma_{X_JZ}\Sigma_{ZZ}^{\dagger}=S_{X_JZ}\Sigma_{ZZ}^{1+\beta}\Sigma_{ZZ}^{\dagger}. B X J ∣ Z = Σ X J Z Σ Z Z † = S X J Z Σ Z Z 1 + β Σ Z Z † .
where the operator product on the right-hand side is understood through its bounded extension. Since S X J Z S_{X_JZ} S X J Z is bounded,
∥ B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 ∥ o p ≤ ∥ S X J Z ∥ o p ∥ Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 ∥ o p . \left\|B_{X_J\mid Z}(\Sigma_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}
\leq\left\|S_{X_JZ}\right\|_{\mathrm{op}}
\left\|\Sigma_{ZZ}^{1+\beta}\Sigma_{ZZ}^{\dagger}(\Sigma_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}. B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 op ≤ ∥ S X J Z ∥ op Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 op .
Let { e k } k \{e_k\}_k { e k } k be the orthonormal eigen functions of Σ Z Z \Sigma_{ZZ} Σ Z Z , { γ k } k \{\gamma_k\}_k { γ k } k be the corresponding eigen values. Then
∥ Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 ∥ o p = sup k ⟨ Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 e k , e k ⟩ H Z = sup k γ k β γ k + ϵ n . \left\|\Sigma_{ZZ}^{1+\beta}\Sigma_{ZZ}^{\dagger}(\Sigma_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}
=\sup_k\left\langle\Sigma_{ZZ}^{1+\beta}\Sigma_{ZZ}^{\dagger}(\Sigma_{ZZ}+\epsilon_n I)^{-1}e_k,e_k\right\rangle_{\mathcal{H}_Z}
=\sup_k\frac{\gamma_k^\beta}{\gamma_k+\epsilon_n}. Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 op = k sup ⟨ Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 e k , e k ⟩ H Z = k sup γ k + ϵ n γ k β .
If 0 < β < 1 0<\beta<1 0 < β < 1 , γ k β γ k + ϵ n ≲ ϵ n β − 1 \frac{\gamma_k^\beta}{\gamma_k+\epsilon_n}\lesssim\epsilon_n^{\beta-1} γ k + ϵ n γ k β ≲ ϵ n β − 1 ; If β ≥ 1 \beta\geq1 β ≥ 1 , γ k β γ k + ϵ n ≤ γ k β − 1 ≤ ∥ Σ Z Z ∥ o p β − 1 < ∞ \frac{\gamma_k^\beta}{\gamma_k+\epsilon_n}\leq\gamma_k^{\beta-1}\leq\left\|\Sigma_{ZZ}\right\|_{\mathrm{op}}^{\beta-1}<\infty γ k + ϵ n γ k β ≤ γ k β − 1 ≤ ∥ Σ Z Z ∥ op β − 1 < ∞ . This holds for any k k k . This gives
∥ Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 ∥ o p ≲ ϵ n β ∧ 1 − 1 . (31) \left\|\Sigma_{ZZ}^{1+\beta}\Sigma_{ZZ}^{\dagger}(\Sigma_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}\lesssim\epsilon_n^{\beta\wedge1-1}. \tag*{(31)} Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 op ≲ ϵ n β ∧ 1 − 1 . (31)
Hence,
∥ ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 ∥ o p ≲ ϵ n β ∧ 1 . (32) \left\|\epsilon_n B_{X_J\mid Z}(\Sigma_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}\lesssim\epsilon_n^{\beta\wedge1}. \tag*{(32)} ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 op ≲ ϵ n β ∧ 1 . (32)
For the second term of (30), we have
∥ Σ ^ Z Z − Σ Z Z ∥ o p ≤ ∥ Σ ^ Z Z − Σ Z Z ∥ H S ≤ ∥ E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] − Σ Z Z ∥ H S + ∥ E n [ κ ~ Z ( ⋅ , Z ) ] ⊗ E n [ κ ~ Z ( ⋅ , Z ) ] ∥ H S = ∥ E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] − Σ Z Z ∥ H S + ∥ E n [ κ ~ Z ( ⋅ , Z ) ] ∥ H Z ∥ E n [ κ ~ Z ( ⋅ , Z ) ] ∥ H Z = O P ( n − 1 2 ) . (33) \begin{aligned}
\left\|\widehat{\Sigma}_{ZZ}-\Sigma_{ZZ}\right\|_{\mathrm{op}}
&\leq\left\|\widehat{\Sigma}_{ZZ}-\Sigma_{ZZ}\right\|_{\mathrm{HS}}\\
&\leq\left\|\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)]-\Sigma_{ZZ}\right\|_{\mathrm{HS}}
+\left\|\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)]\otimes\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)]\right\|_{\mathrm{HS}}\\
&=\left\|\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)]-\Sigma_{ZZ}\right\|_{\mathrm{HS}}
+\left\|\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)]\right\|_{\mathcal{H}_Z}
\left\|\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)]\right\|_{\mathcal{H}_Z} \\
&=O_P\left(n^{-\frac{1}{2}}\right). \tag*{(33)}
\end{aligned} Σ Z Z − Σ Z Z op ≤ Σ Z Z − Σ Z Z HS ≤ ∥ E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z )] − Σ Z Z ∥ HS + ∥ E n [ κ Z ( ⋅ , Z )] ⊗ E n [ κ Z ( ⋅ , Z )] ∥ HS = ∥ E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z )] − Σ Z Z ∥ HS + ∥ E n [ κ Z ( ⋅ , Z )] ∥ H Z ∥ E n [ κ Z ( ⋅ , Z )] ∥ H Z = O P ( n − 2 1 ) . (33)
And by the spectral theorem, we have
∥ ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ≤ ϵ n − 1 . (34) \left\|(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}\leq\epsilon_n^{-1}. \tag*{(34)} ( Σ Z Z + ϵ n I ) − 1 op ≤ ϵ n − 1 . (34)
Combining (32), (33) and (34) gives
∥ ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ ^ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ≤ ∥ ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 ∥ o p ∥ Σ Z Z − Σ ^ Z Z ∥ o p ∥ ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ≤ O P ( ϵ n β ∧ 1 ) O P ( n − 1 2 ) ϵ n − 1 = O P ( n − 1 2 ϵ n β ∧ 1 − 1 ) . (35) \begin{aligned}
&\left\|\epsilon_n B_{XJ\mid Z}(\Sigma_{ZZ}+\epsilon_n I)^{-1}(\Sigma_{ZZ}-\widehat{\Sigma}_{ZZ})(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}\\
&\leq\left\|\epsilon_n B_{XJ\mid Z}(\Sigma_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}\left\|\Sigma_{ZZ}-\widehat{\Sigma}_{ZZ}\right\|_{\mathrm{op}}\left\|(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}\\
&\leq O_P(\epsilon_n^{\beta\wedge1})O_P(n^{-\frac{1}{2}})\epsilon_n^{-1}\\
&=O_P\left(n^{-\frac{1}{2}}\epsilon_n^{\beta\wedge1-1}\right). \tag*{(35)}
\end{aligned} ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ Z Z ) ( Σ Z Z + ϵ n I ) − 1 op ≤ ϵ n B X J ∣ Z ( Σ Z Z + ϵ n I ) − 1 op Σ Z Z − Σ Z Z op ( Σ Z Z + ϵ n I ) − 1 op ≤ O P ( ϵ n β ∧ 1 ) O P ( n − 2 1 ) ϵ n − 1 = O P ( n − 2 1 ϵ n β ∧ 1 − 1 ) . (35)
We next consider the third term of (30). The boundedness of the kernels and B X J ∣ Z B_{XJ\mid Z} B X J ∣ Z implies the boundedness of the residual term ξ X J ∣ Z ( ⋅ , X J ; Z ) \xi_{XJ\mid Z}(\cdot,X_J;Z) ξ X J ∣ Z ( ⋅ , X J ; Z ) . By (2) and independence across observations, the cross terms in the second moment vanish. The boundedness of the residuals and the spectral decomposition of Σ Z Z \Sigma_{ZZ} Σ Z Z therefore give
E ∥ E n [ { ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) } ( Σ Z Z + ϵ n I ) − 1 ] ∥ H S 2 = 1 n E [ ∥ ξ X J ∣ Z ( ⋅ , X J ; Z ) ∥ H X J 2 ∥ ( Σ Z Z + ϵ n I ) − 1 κ ~ Z ( ⋅ , Z ) ∥ H Z 2 ] ≲ 1 n E [ ∥ ( Σ Z Z + ϵ n I ) − 1 κ ~ Z ( ⋅ , Z ) ∥ H Z 2 ] = 1 n ∑ k = 1 ∞ γ k ( γ k + ϵ n ) 2 . \begin{aligned}
\mathbb{E}\left\|\mathbb{E}_n\left[\{\xi_{XJ\mid Z}(\cdot,X_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\}(\Sigma_{ZZ}+\epsilon_n I)^{-1}\right]\right\|_{\mathrm{HS}}^2
&=\frac{1}{n}\mathbb{E}\left[\left\|\xi_{XJ\mid Z}(\cdot,X_J;Z)\right\|_{\mathcal{H}_{X_J}}^2\left\|(\Sigma_{ZZ}+\epsilon_n I)^{-1}\widetilde{\kappa}_Z(\cdot,Z)\right\|_{\mathcal{H}_Z}^2\right]\\
&\lesssim\frac{1}{n}\mathbb{E}\left[\left\|(\Sigma_{ZZ}+\epsilon_n I)^{-1}\widetilde{\kappa}_Z(\cdot,Z)\right\|_{\mathcal{H}_Z}^2\right]\\
&=\frac{1}{n}\sum_{k=1}^{\infty}\frac{\gamma_k}{(\gamma_k+\epsilon_n)^2}.
\end{aligned} E E n [ { ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ Z ( ⋅ , Z )} ( Σ Z Z + ϵ n I ) − 1 ] HS 2 = n 1 E [ ξ X J ∣ Z ( ⋅ , X J ; Z ) H X J 2 ( Σ Z Z + ϵ n I ) − 1 κ Z ( ⋅ , Z ) H Z 2 ] ≲ n 1 E [ ( Σ Z Z + ϵ n I ) − 1 κ Z ( ⋅ , Z ) H Z 2 ] = n 1 k = 1 ∑ ∞ ( γ k + ϵ n ) 2 γ k .
It remains to estimate the eigenvalue sum. Using γ k ≍ k − α \gamma_k\asymp k^{-\alpha} γ k ≍ k − α with α > 1 \alpha>1 α > 1 , we split the sum at ϵ n − 1 / α \epsilon_n^{-1/\alpha} ϵ n − 1/ α and apply standard bounds for sums of powers:
n − 1 ∑ k = 1 ∞ γ k ( γ k + ϵ n ) 2 ≍ n − 1 { ∑ k ≤ ϵ n − 1 / α γ k − 1 + ϵ n − 2 ∑ k > ϵ n − 1 / α γ k } ≍ n − 1 { ϵ n − ( α + 1 ) / α + ϵ n − 2 ϵ n ( α − 1 ) / α } ≍ n − 1 ϵ n − ( α + 1 ) / α . (36) \begin{aligned}
n^{-1}\sum_{k=1}^{\infty}\frac{\gamma_k}{(\gamma_k+\epsilon_n)^2}
&\asymp n^{-1}\left\{\sum_{k\leq\epsilon_n^{-1/\alpha}}\gamma_k^{-1}+\epsilon_n^{-2}\sum_{k>\epsilon_n^{-1/\alpha}}\gamma_k\right\}\\
&\asymp n^{-1}\left\{\epsilon_n^{-(\alpha+1)/\alpha}+\epsilon_n^{-2}\epsilon_n^{(\alpha-1)/\alpha}\right\}\\
&\asymp n^{-1}\epsilon_n^{-(\alpha+1)/\alpha}. \tag*{(36)}
\end{aligned} n − 1 k = 1 ∑ ∞ ( γ k + ϵ n ) 2 γ k ≍ n − 1 ⎩ ⎨ ⎧ k ≤ ϵ n − 1/ α ∑ γ k − 1 + ϵ n − 2 k > ϵ n − 1/ α ∑ γ k ⎭ ⎬ ⎫ ≍ n − 1 { ϵ n − ( α + 1 ) / α + ϵ n − 2 ϵ n ( α − 1 ) / α } ≍ n − 1 ϵ n − ( α + 1 ) / α . (36)
By Markov’s inequality, we have
∥ E n [ { ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) } ( Σ Z Z + ϵ n I ) − 1 ] ∥ H S = O P ( n − 1 2 ϵ n − α + 1 2 α ) . (37) \left\|\mathbb{E}_n\left[\{\xi_{XJ\mid Z}(\cdot,X_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\}(\Sigma_{ZZ}+\epsilon_n I)^{-1}\right]\right\|_{\mathrm{HS}}
=O_P\left(n^{-\frac{1}{2}}\epsilon_n^{-\frac{\alpha+1}{2\alpha}}\right). \tag*{(37)} E n [ { ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ Z ( ⋅ , Z )} ( Σ Z Z + ϵ n I ) − 1 ] HS = O P ( n − 2 1 ϵ n − 2 α α + 1 ) . (37)
It also helps with the fourth term of (30) as shown below:
∥ E n [ { ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) } ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ ^ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 ] ∥ H S ≤ ∥ E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ ^ Z Z + ϵ n I ) − 1 ∥ H S ∥ Σ Z Z − Σ ^ Z Z ∥ o p ∥ ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ≤ O P ( n − 1 2 ϵ n − α + 1 2 α ) O P ( n − 1 2 ) ϵ n − 1 ≤ O P ( n − 1 ϵ n − 3 α + 1 2 α ) . (38) \begin{aligned}
\left\|\mathbb{E}_n\left[\{\xi_{XJ\mid Z}(\cdot,X_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\}(\Sigma_{ZZ}+\epsilon_n I)^{-1}(\Sigma_{ZZ}-\widehat{\Sigma}_{ZZ})(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right]\right\|_{\mathrm{HS}} \\
&\leq\left\|\mathbb{E}_n\left[\xi_{X_J\mid Z}(\cdot, X_J; Z)\otimes\widetilde{\kappa}_Z(\cdot, Z)\right](\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{HS}}\left\|\Sigma_{ZZ}-\widehat{\Sigma}_{ZZ}\right\|_{\mathrm{op}}\left\|(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}
\\
\leq O_P\left(n^{-\frac{1}{2}}\epsilon_n^{-\frac{\alpha+1}{2\alpha}}\right)O_P\left(n^{-\frac{1}{2}}\right)\epsilon_n^{-1}
\\
\leq O_P\left(n^{-1}\epsilon_n^{-\frac{3\alpha+1}{2\alpha}}\right). \tag*{(38)}
\end{aligned} E n [ { ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ Z ( ⋅ , Z )} ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ Z Z ) ( Σ Z Z + ϵ n I ) − 1 ] HS ≤ O P ( n − 2 1 ϵ n − 2 α α + 1 ) O P ( n − 2 1 ) ϵ n − 1 ≤ O P ( n − 1 ϵ n − 2 α 3 α + 1 ) . ≤ E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ⊗ κ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 HS Σ Z Z − Σ Z Z op ( Σ Z Z + ϵ n I ) − 1 op (38)
Finally, for the fifth term of (30), by the Hilbert-space central limit theorem,
∥ E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ] ∥ H X J = O P ( n − 1 2 ) , ∥ E n [ κ ~ Z ( ⋅ , Z ) ] ∥ H Z = O P ( n − 1 2 ) . \left\|\mathbb{E}_n\left[\xi_{X_J\mid Z}(\cdot, X_J; Z)\right]\right\|_{\mathcal{H}_{X_J}}=O_P\left(n^{-\frac{1}{2}}\right), \qquad\left\|\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot, Z)\right]\right\|_{\mathcal{H}_Z}=O_P\left(n^{-\frac{1}{2}}\right). E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ] H X J = O P ( n − 2 1 ) , ∥ E n [ κ Z ( ⋅ , Z ) ] ∥ H Z = O P ( n − 2 1 ) .
Applying the above results and (34), we obtain
∥ ( E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ] ⊗ E n [ κ ~ Z ( ⋅ , Z ) ] ) ( Σ ^ Z Z + ϵ n I ) − 1 ∥ H S ≤ ϵ n − 1 ∥ E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ] ∥ H X J ∥ E n [ κ ~ Z ( ⋅ , Z ) ] ∥ H Z = O P ( n − 1 ϵ n − 1 ) . (39) \left\|\left(\mathbb{E}_n\left[\xi_{X_J\mid Z}(\cdot, X_J; Z)\right]\otimes\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot, Z)\right]\right)(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{HS}}
\\
\leq\epsilon_n^{-1}\left\|\mathbb{E}_n\left[\xi_{X_J\mid Z}(\cdot, X_J; Z)\right]\right\|_{\mathcal{H}_{X_J}}\left\|\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot, Z)\right]\right\|_{\mathcal{H}_Z}
\\
=O_P\left(n^{-1}\epsilon_n^{-1}\right). \tag*{(39)} ( E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ] ⊗ E n [ κ Z ( ⋅ , Z ) ] ) ( Σ Z Z + ϵ n I ) − 1 HS ≤ ϵ n − 1 E n [ ξ X J ∣ Z ( ⋅ , X J ; Z ) ] H X J ∥ E n [ κ Z ( ⋅ , Z ) ] ∥ H Z = O P ( n − 1 ϵ n − 1 ) . (39)
Since α > 1 \alpha>1 α > 1 and ϵ n → 0 \epsilon_n\to0 ϵ n → 0 ,
n − 1 ϵ n − 1 = o ( n − 1 ϵ n − 3 α + 1 2 α ) . n^{-1}\epsilon_n^{-1}=o\left(n^{-1}\epsilon_n^{-\frac{3\alpha+1}{2\alpha}}\right). n − 1 ϵ n − 1 = o ( n − 1 ϵ n − 2 α 3 α + 1 ) .
Combining (32), (35), (37), (38) and (39) gives
∥ B ^ X J ∣ Z − B X J ∣ Z ∥ o p = O P ( n − 1 2 ϵ n β ∧ 1 − 1 + ϵ n β ∧ 1 + n − 1 ϵ n − 3 α + 1 2 α + n − 1 2 ϵ n − α + 1 2 α ) , \left\|\widehat{B}_{X_J\mid Z}-B_{X_J\mid Z}\right\|_{\mathrm{op}}
=O_P\left(n^{-\frac{1}{2}}\epsilon_n^{\beta\wedge1-1}+\epsilon_n^{\beta\wedge1}+n^{-1}\epsilon_n^{-\frac{3\alpha+1}{2\alpha}}+n^{-\frac{1}{2}}\epsilon_n^{-\frac{\alpha+1}{2\alpha}}\right), B X J ∣ Z − B X J ∣ Z op = O P ( n − 2 1 ϵ n β ∧ 1 − 1 + ϵ n β ∧ 1 + n − 1 ϵ n − 2 α 3 α + 1 + n − 2 1 ϵ n − 2 α α + 1 ) ,
which proves assertion 1.
For assertion 2, recall that κ ^ Z ( ⋅ , Z ) = κ ~ Z ( ⋅ , Z ) − ( μ ^ Z − μ Z ) \widehat{\kappa}_Z(\cdot, Z)=\widetilde{\kappa}_Z(\cdot, Z)-(\widehat{\mu}_Z-\mu_Z) κ Z ( ⋅ , Z ) = κ Z ( ⋅ , Z ) − ( μ Z − μ Z ) and E n [ κ ^ Z ( ⋅ , Z ) ] = 0 \mathbb{E}_n[\widehat{\kappa}_Z(\cdot, Z)]=0 E n [ κ Z ( ⋅ , Z )] = 0 . Those facts give us
Σ ^ X J Z = E n [ κ ^ X ¨ J ( ⋅ , X ¨ J ) ⊗ κ ^ Z ( ⋅ , Z ) ] = E n [ κ ~ X ¨ J ( ⋅ , X ¨ J ) ⊗ κ ^ Z ( ⋅ , Z ) ] \widehat{\Sigma}_{X_J Z}=\mathbb{E}_n\left[\widehat{\kappa}_{\ddot{X}_J}(\cdot,\ddot{X}_J)\otimes\widehat{\kappa}_Z(\cdot,Z)\right]
=\mathbb{E}_n\left[\widetilde{\kappa}_{\ddot{X}_J}(\cdot,\ddot{X}_J)\otimes\widehat{\kappa}_Z(\cdot,Z)\right] Σ X J Z = E n [ κ X ¨ J ( ⋅ , X ¨ J ) ⊗ κ Z ( ⋅ , Z ) ] = E n [ κ X ¨ J ( ⋅ , X ¨ J ) ⊗ κ Z ( ⋅ , Z ) ]
and
Σ ^ Z Z = E n [ κ ^ Z ( ⋅ , Z ) ⊗ κ ^ Z ( ⋅ , Z ) ] = E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ^ Z ( ⋅ , Z ) ] \widehat{\Sigma}_{ZZ}=\mathbb{E}_n\left[\widehat{\kappa}_Z(\cdot,Z)\otimes\widehat{\kappa}_Z(\cdot,Z)\right]
=\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widehat{\kappa}_Z(\cdot,Z)\right] Σ Z Z = E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] = E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ]
immediately. By the definition of the residual term,
κ ~ X ¨ J ( ⋅ , X ¨ J ) = B X ¨ J ∣ Z κ ~ Z ( ⋅ , Z ) + ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) . \widetilde{\kappa}_{\ddot{X}_J}(\cdot,\ddot{X}_J)
=B_{\ddot{X}_J\mid Z}\widetilde{\kappa}_Z(\cdot,Z)+\xi_{\ddot{X}_J\mid Z}(\cdot,\ddot{X}_J;Z). κ X ¨ J ( ⋅ , X ¨ J ) = B X ¨ J ∣ Z κ Z ( ⋅ , Z ) + ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) .
Substituting this identity gives
Σ ^ X ¨ J Z = B X ¨ J ∣ Z E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ^ Z ( ⋅ , Z ) ] + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ^ Z ( ⋅ , Z ) ] = B X ¨ J ∣ Z Σ ^ Z Z + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ^ Z ( ⋅ , Z ) ] . \widehat{\Sigma}_{\ddot{X}_J Z}
=B_{\ddot{X}_J\mid Z}\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widehat{\kappa}_Z(\cdot,Z)\right]
+\mathbb{E}_n\left[\xi_{\ddot{X}_J\mid Z}(\cdot,\ddot{X}_J;Z)\otimes\widehat{\kappa}_Z(\cdot,Z)\right]
\\
=B_{\ddot{X}_J\mid Z}\widehat{\Sigma}_{ZZ}
+\mathbb{E}_n\left[\xi_{\ddot{X}_J\mid Z}(\cdot,\ddot{X}_J;Z)\otimes\widehat{\kappa}_Z(\cdot,Z)\right]. Σ X ¨ J Z = B X ¨ J ∣ Z E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ Z ( ⋅ , Z ) ] = B X ¨ J ∣ Z Σ Z Z + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ Z ( ⋅ , Z ) ] .
Subtracting T μ ^ X J Σ ^ Z Z T_{\hat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ} T μ ^ X J Σ Z Z from both sides gives
Σ ^ X ¨ J Z − T μ ^ X J Σ ^ Z Z = ( B X ¨ J ∣ Z − T μ ^ X J ) Σ ^ Z Z − T μ ^ X J − μ X J Σ ^ Z Z + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ^ Z ( ⋅ , Z ) ] . \widehat{\Sigma}_{\ddot{X}_JZ}-T_{\hat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ}=\left(B_{\ddot{X}_J|Z}-T_{\hat{\mu}_{X_J}}\right)\widehat{\Sigma}_{ZZ}-T_{\hat{\mu}_{X_J}-\mu_{X_J}}\widehat{\Sigma}_{ZZ}+\mathbb{E}_n\left[\xi_{\ddot{X}_J|Z}(\cdot,\ddot{X}_J;Z)\otimes\hat{\kappa}_Z(\cdot,Z)\right]. Σ X ¨ J Z − T μ ^ X J Σ Z Z = ( B X ¨ J ∣ Z − T μ ^ X J ) Σ Z Z − T μ ^ X J − μ X J Σ Z Z + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ^ Z ( ⋅ , Z ) ] .
Since ran ( Σ ^ Z Z ) ⊂ ran ( Σ ‾ Z Z ) \operatorname{ran}(\widehat{\Sigma}_{ZZ})\subset\operatorname{ran}(\overline{\Sigma}_{ZZ}) ran ( Σ Z Z ) ⊂ ran ( Σ Z Z ) almost surely, we have Σ Z Z Σ Z Z † Σ ^ Z Z = Σ ^ Z Z \Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\widehat{\Sigma}_{ZZ}=\widehat{\Sigma}_{ZZ} Σ Z Z Σ Z Z † Σ Z Z = Σ Z Z almost surely. Here, Σ Z Z Σ Z Z † \Sigma_{ZZ}\Sigma_{ZZ}^{\dagger} Σ Z Z Σ Z Z † , as mentioned above, denoting the unique bounded extension of Σ Z Z Σ Z Z † \Sigma_{ZZ}\Sigma_{ZZ}^{\dagger} Σ Z Z Σ Z Z † , is indeed the orthogonal projection operator onto ran ( Σ ‾ Z Z ) \operatorname{ran}(\overline{\Sigma}_{ZZ}) ran ( Σ Z Z ) .
Consequently,
( B X ¨ J ∣ Z − T μ X J ) Σ ^ Z Z = ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) Σ ^ Z Z . \left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\right)\widehat{\Sigma}_{ZZ}=\left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)\widehat{\Sigma}_{ZZ}. ( B X ¨ J ∣ Z − T μ X J ) Σ Z Z = ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) Σ Z Z .
Therefore, we have almost surely that
Σ ^ X ¨ J Z − T μ ^ X J Σ ^ Z Z = ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) Σ ^ Z Z + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] − E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ ^ Z − μ Z ) − T μ ^ X J − μ X J Σ ^ Z Z . (40) \begin{aligned}
\widehat{\Sigma}_{\ddot{X}_JZ}-T_{\hat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ}
={}&\left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)\widehat{\Sigma}_{ZZ}
+\mathbb{E}_n\left[\xi_{\ddot{X}_J|Z}(\cdot,\ddot{X}_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]\\
&-\mathbb{E}_n\left[\xi_{\ddot{X}_J|Z}(\cdot,\ddot{X}_J;Z)\right]\otimes(\hat{\mu}_Z-\mu_Z)-T_{\hat{\mu}_{X_J}-\mu_{X_J}}\widehat{\Sigma}_{ZZ}. \tag*{(40)}
\end{aligned} Σ X ¨ J Z − T μ ^ X J Σ Z Z = ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) Σ Z Z + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ Z ( ⋅ , Z ) ] − E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ ^ Z − μ Z ) − T μ ^ X J − μ X J Σ Z Z . (40)
We first rewrite the quantity of interest as
( B ^ X ¨ J ∣ Z − T μ ^ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 ) − ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) = ( Σ ^ X ¨ J Z − T μ ^ X J Σ ^ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 − ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) . \begin{aligned}
&\left(\widehat{B}_{\ddot{X}_J|Z}-T_{\hat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right)-\left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)\\
={}&\left(\widehat{\Sigma}_{\ddot{X}_JZ}-T_{\hat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ}\right)(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}-\left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right).
\end{aligned} = ( B X ¨ J ∣ Z − T μ ^ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 ) − ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ X ¨ J Z − T μ ^ X J Σ Z Z ) ( Σ Z Z + ϵ n I ) − 1 − ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) .
Substituting (40) into the right-hand side gives
( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ ^ Z Z + ϵ n I ) − 1 − ( E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ ^ Z − μ Z ) ) ( Σ ^ Z Z + ϵ n I ) − 1 − T μ ^ X J − μ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 − ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) . \begin{aligned}
&\left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
&+\mathbb{E}_n\left[\xi_{\ddot{X}_J|Z}(\cdot,\ddot{X}_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right](\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
&-\left(\mathbb{E}_n\left[\xi_{\ddot{X}_J|Z}(\cdot,\ddot{X}_J;Z)\right]\otimes(\hat{\mu}_Z-\mu_Z)\right)(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
&-T_{\hat{\mu}_{X_J}-\mu_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}
-\left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right).
\end{aligned} ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) Σ Z Z ( Σ Z Z + ϵ n I ) − 1 + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 − ( E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ ^ Z − μ Z ) ) ( Σ Z Z + ϵ n I ) − 1 − T μ ^ X J − μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 − ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) .
Combining the first and last terms, this expression reduces to
− ϵ n ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ ^ Z Z + ϵ n I ) − 1 + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ ^ Z Z + ϵ n I ) − 1 − ( E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ ^ Z − μ Z ) ) ( Σ ^ Z Z + ϵ n I ) − 1 − T μ ^ X J − μ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 . \begin{aligned}
&-\epsilon_n\left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
&+\mathbb{E}_n\left[\xi_{\ddot{X}_J|Z}(\cdot,\ddot{X}_J;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right](\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
&-\left(\mathbb{E}_n\left[\xi_{\ddot{X}_J|Z}(\cdot,\ddot{X}_J;Z)\right]\otimes(\hat{\mu}_Z-\mu_Z)\right)(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\\
&-T_{\hat{\mu}_{X_J}-\mu_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}.
\end{aligned} − ϵ n ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 + E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 − ( E n [ ξ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ ^ Z − μ Z ) ) ( Σ Z Z + ϵ n I ) − 1 − T μ ^ X J − μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 .
Finally, applying the resolvent identity (29) to the first two terms yields
( B ^ X ¨ J ∣ Z − T μ ^ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 ) − ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) = − ϵ n ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 \begin{aligned}
&\left(\widehat{B}_{\ddot{X}_J|Z}-T_{\hat{\mu}_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right)-\left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)\\
={}&-\epsilon_n\left(B_{\ddot{X}_J|Z}-T_{\mu_{X_J}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)(\Sigma_{ZZ}+\epsilon_n I)^{-1}
\end{aligned} = ( B X ¨ J ∣ Z − T μ ^ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 ) − ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) − ϵ n ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1
+ E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 − ( E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ ^ Z − μ Z ) ) ( Σ ^ Z Z + ϵ n I ) − 1 − T μ ^ X J − μ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 + ϵ n ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ ^ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 ( Σ ^ Z Z − Σ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 − E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 ( Σ ^ Z Z − Σ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 . (41) \begin{aligned}
&+\mathbb{E}_{n}\left[\tilde{\xi}_{\ddot{X}_{J|Z}}(\cdot,\ddot{X}_{J};Z)\otimes\tilde{\kappa}_{Z}(\cdot,Z)\right](\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\\
&-\left(\mathbb{E}_{n}\left[\tilde{\xi}_{\ddot{X}_{J|Z}}(\cdot,\ddot{X}_{J};Z)\right]\otimes(\hat{\mu}_{Z}-\mu_{Z})\right)(\hat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\\
&-T_{\hat{\mu}_{XJ}-\mu_{XJ}}\hat{\Sigma}_{ZZ}(\hat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\\
&+\epsilon_{n}\left(B_{\ddot{X}_{J|Z}}-T_{\mu_{XJ}}\Sigma_{ZZ}\hat{\Sigma}_{ZZ}^{\dagger}\right)(\Sigma_{ZZ}+\epsilon_{n}I)^{-1}(\hat{\Sigma}_{ZZ}-\Sigma_{ZZ})(\hat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\\
&-\mathbb{E}_{n}\left[\tilde{\xi}_{\ddot{X}_{J|Z}}(\cdot,\ddot{X}_{J};Z)\otimes\tilde{\kappa}_{Z}(\cdot,Z)\right](\Sigma_{ZZ}+\epsilon_{n}I)^{-1}(\hat{\Sigma}_{ZZ}-\Sigma_{ZZ})(\hat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}. \tag*{(41)}
\end{aligned} + E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 − ( E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ ^ Z − μ Z ) ) ( Σ ^ Z Z + ϵ n I ) − 1 − T μ ^ X J − μ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 + ϵ n ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ ^ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 ( Σ ^ Z Z − Σ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 − E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 ( Σ ^ Z Z − Σ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 . (41)
By Assumption 3, we have
B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † = ( R μ Z S X J Z + S X ¨ J Z ) Σ Z Z 1 + β Σ Z Z † B_{\ddot{X}_{J|Z}}-T_{\mu_{XJ}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}=(R_{\mu_{Z}}S_{XJZ}+S_{\ddot{X}_{J}Z})\Sigma_{ZZ}^{1+\beta}\Sigma_{ZZ}^{\dagger} B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † = ( R μ Z S X J Z + S X ¨ J Z ) Σ Z Z 1 + β Σ Z Z †
Then
∥ ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 ∥ o p ≤ ∥ R μ Z S X J Z + S X ¨ J Z ∥ o p ∥ Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 ∥ o p ≤ ( ∥ μ Z ∥ H Z ∥ S X J Z ∥ o p + ∥ S X ¨ J Z ∥ o p ) ∥ Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 ∥ o p ≲ ϵ n ( β ∧ 1 ) − 1 , \begin{aligned}
\left\|\left(B_{\ddot{X}_{J|Z}}-T_{\mu_{XJ}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)(\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}}
&\leq\left\|R_{\mu_{Z}}S_{XJZ}+S_{\ddot{X}_{J}Z}\right\|_{\mathrm{op}}\left\|\Sigma_{ZZ}^{1+\beta}\Sigma_{ZZ}^{\dagger}(\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}}\\
&\leq\left(\|\mu_{Z}\|_{\mathcal{H}_{Z}}\|S_{XJZ}\|_{\mathrm{op}}+\|S_{\ddot{X}_{J}Z}\|_{\mathrm{op}}\right)\left\|\Sigma_{ZZ}^{1+\beta}\Sigma_{ZZ}^{\dagger}(\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}}\\
&\lesssim\epsilon_{n}^{(\beta\wedge1)-1},
\end{aligned} ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 op ≤ R μ Z S X J Z + S X ¨ J Z op Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 op ≤ ( ∥ μ Z ∥ H Z ∥ S X J Z ∥ op + ∥ S X ¨ J Z ∥ op ) Σ Z Z 1 + β Σ Z Z † ( Σ Z Z + ϵ n I ) − 1 op ≲ ϵ n ( β ∧ 1 ) − 1 ,
where the last line holds because R μ Z R_{\mu_{Z}} R μ Z , S X J Z S_{XJZ} S X J Z and S X ¨ J Z S_{\ddot{X}_{J}Z} S X ¨ J Z are all bounded and (31) holds. Hence, the first term in (41) is O P ( ϵ n β ∧ 1 ) O_{P}(\epsilon_{n}^{\beta\wedge1}) O P ( ϵ n β ∧ 1 ) , which follows from
∥ ϵ n ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 ∥ o p ≲ ϵ n ϵ n ( β ∧ 1 ) − 1 ≲ ϵ n β ∧ 1 . (42) \left\|\epsilon_{n}\left(B_{\ddot{X}_{J|Z}}-T_{\mu_{XJ}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)(\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}}
\lesssim\epsilon_{n}\epsilon_{n}^{(\beta\wedge1)-1}\lesssim\epsilon_{n}^{\beta\wedge1}. \tag*{(42)} ϵ n ( B X ¨ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 op ≲ ϵ n ϵ n ( β ∧ 1 ) − 1 ≲ ϵ n β ∧ 1 . (42)
By (2), we have E [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] = 0 \mathbb{E}\left[\tilde{\xi}_{\ddot{X}_{J|Z}}(\cdot,\ddot{X}_{J};Z)\otimes\tilde{\kappa}_{Z}(\cdot,Z)\right]=0 E [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] = 0 . Moreover, the boundedness of the kernels and of B X ¨ J ∣ Z B_{\ddot{X}_{J|Z}} B X ¨ J ∣ Z implies the boundedness of ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) \tilde{\xi}_{\ddot{X}_{J|Z}}(\cdot,\ddot{X}_{J};Z) ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) under the Hilbert norm. We further use the independence across observations and obtain
E ∥ E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 ∥ H S 2 = 1 n E [ ∥ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ∥ H X ¨ J 2 ∥ ( Σ Z Z + ϵ n I ) − 1 κ ~ Z ( ⋅ , Z ) ∥ H Z 2 ] ≲ 1 n E [ ∥ ( Σ Z Z + ϵ n I ) − 1 κ ~ Z ( ⋅ , Z ) ∥ H Z 2 ] = 1 n ∑ k = 1 ∞ γ k ( γ k + ϵ n ) 2 ≍ n − 1 ϵ n − α + 1 α , \begin{aligned}
\mathbb{E}\left\|\mathbb{E}_{n}\left[\tilde{\xi}_{\ddot{X}_{J|Z}}(\cdot,\ddot{X}_{J};Z)\otimes\tilde{\kappa}_{Z}(\cdot,Z)\right](\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathcal{H}_{S}}^{2}
&=\frac{1}{n}\mathbb{E}\left[\left\|\tilde{\xi}_{\ddot{X}_{J|Z}}(\cdot,\ddot{X}_{J};Z)\right\|_{\mathcal{H}_{\ddot{X}_{J}}}^{2}\left\|(\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\tilde{\kappa}_{Z}(\cdot,Z)\right\|_{\mathcal{H}_{Z}}^{2}\right]\\
&\lesssim\frac{1}{n}\mathbb{E}\left[\left\|(\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\tilde{\kappa}_{Z}(\cdot,Z)\right\|_{\mathcal{H}_{Z}}^{2}\right]\\
&=\frac{1}{n}\sum_{k=1}^{\infty}\frac{\gamma_{k}}{(\gamma_{k}+\epsilon_{n})^{2}}\\
&\asymp n^{-1}\epsilon_{n}^{-\frac{\alpha+1}{\alpha}},
\end{aligned} E E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 H S 2 = n 1 E [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) H X ¨ J 2 ( Σ Z Z + ϵ n I ) − 1 κ ~ Z ( ⋅ , Z ) H Z 2 ] ≲ n 1 E [ ( Σ Z Z + ϵ n I ) − 1 κ ~ Z ( ⋅ , Z ) H Z 2 ] = n 1 k = 1 ∑ ∞ ( γ k + ϵ n ) 2 γ k ≍ n − 1 ϵ n − α α + 1 ,
Hence, by Markov’s inequality, we have the convergence rate for the second term in (41)
∥ E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 ∥ o p = O P ( n − 1 2 ϵ n − α + 1 2 α ) (43) \left\|\mathbb{E}_{n}\left[\tilde{\xi}_{\ddot{X}_{J|Z}}(\cdot,\ddot{X}_{J};Z)\otimes\tilde{\kappa}_{Z}(\cdot,Z)\right](\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}}
=O_{P}\left(n^{-\frac{1}{2}}\epsilon_{n}^{-\frac{\alpha+1}{2\alpha}}\right) \tag*{(43)} E n [ ξ ~ X ¨ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 op = O P ( n − 2 1 ϵ n − 2 α α + 1 ) (43)
And also for the sixth term in (41)
∥ E n [ ξ ~ X ˙ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ( Σ ^ Z Z + ϵ n I ) − 1 ( Σ ^ Z Z − Σ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p = O P ( n − 1 2 ϵ n − α + 1 2 α ) O P ( n − 1 2 ) O P ( ϵ n − 1 ) = O P ( n − 1 ϵ n − 3 α + 1 2 α ) . (44) \begin{aligned}
\left\|\mathbb{E}_{n}\left[\widetilde{\xi}_{\dot{X}_{J}\mid Z}(\cdot,\ddot{X}_{J};Z)\otimes\widetilde{\kappa}_{Z}(\cdot,Z)\right](\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}(\widehat{\Sigma}_{ZZ}-\Sigma_{ZZ})(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}} \\
&=O_{P}\left(n^{-\frac{1}{2}}\epsilon_{n}^{-\frac{\alpha+1}{2\alpha}}\right)O_{P}\left(n^{-\frac{1}{2}}\right)O_{P}\left(\epsilon_{n}^{-1}\right)=O_{P}\left(n^{-1}\epsilon_{n}^{-\frac{3\alpha+1}{2\alpha}}\right). \tag*{(44)}
\end{aligned} E n [ ξ X ˙ J ∣ Z ( ⋅ , X ¨ J ; Z ) ⊗ κ Z ( ⋅ , Z ) ] ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ Z Z ) ( Σ Z Z + ϵ n I ) − 1 op = O P ( n − 2 1 ϵ n − 2 α α + 1 ) O P ( n − 2 1 ) O P ( ϵ n − 1 ) = O P ( n − 1 ϵ n − 2 α 3 α + 1 ) . (44)
For the third term, since ( Σ ^ Z Z + ϵ n I ) − 1 (\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1} ( Σ Z Z + ϵ n I ) − 1 is self-adjoint, we have
∥ ( E n [ ξ ~ X ˙ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ ^ Z − μ Z ) ) ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p = ∥ E n [ ξ ~ X ˙ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ∥ H X J ∥ ( Σ ^ Z Z + ϵ n I ) − 1 ( μ ^ Z − μ Z ) ∥ H Z ≤ O P ( n − 1 2 ) ϵ n − 1 O P ( n − 1 2 ) = O P ( n − 1 ϵ n − 1 ) (45) \begin{aligned}
\left\|\left(\mathbb{E}_{n}\left[\widetilde{\xi}_{\dot{X}_{J}\mid Z}(\cdot,\ddot{X}_{J};Z)\right]\otimes(\widehat{\mu}_{Z}-\mu_{Z})\right)(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}} \\
&=\left\|\mathbb{E}_{n}\left[\widetilde{\xi}_{\dot{X}_{J}\mid Z}(\cdot,\ddot{X}_{J};Z)\right]\right\|_{\mathcal{H}_{X_{J}}}\left\|(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}(\widehat{\mu}_{Z}-\mu_{Z})\right\|_{\mathcal{H}_{Z}} \\
&\leq O_{P}\left(n^{-\frac{1}{2}}\right)\epsilon_{n}^{-1}O_{P}\left(n^{-\frac{1}{2}}\right) \\
&=O_{P}\left(n^{-1}\epsilon_{n}^{-1}\right) \tag*{(45)}
\end{aligned} ( E n [ ξ X ˙ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ⊗ ( μ Z − μ Z ) ) ( Σ Z Z + ϵ n I ) − 1 op = E n [ ξ X ˙ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] H X J ( Σ Z Z + ϵ n I ) − 1 ( μ Z − μ Z ) H Z ≤ O P ( n − 2 1 ) ϵ n − 1 O P ( n − 2 1 ) = O P ( n − 1 ϵ n − 1 ) (45)
which is absorbed by the sixth term because O P ( n − 1 ϵ n − 1 ) = o P ( n − 1 ϵ n − 3 α + 1 2 α ) O_{P}(n^{-1}\epsilon_{n}^{-1})=o_{P}\left(n^{-1}\epsilon_{n}^{-\frac{3\alpha+1}{2\alpha}}\right) O P ( n − 1 ϵ n − 1 ) = o P ( n − 1 ϵ n − 2 α 3 α + 1 ) . The inequality holds because
∥ E n [ ξ ~ X ˙ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] ∥ H X J = O P ( n − 1 2 ) and ∥ μ ^ Z − μ Z ∥ H Z = O P ( n − 1 2 ) \left\|\mathbb{E}_{n}\left[\widetilde{\xi}_{\dot{X}_{J}\mid Z}(\cdot,\ddot{X}_{J};Z)\right]\right\|_{\mathcal{H}_{X_{J}}}=O_{P}\left(n^{-\frac{1}{2}}\right)\quad\text{and}\quad\|\widehat{\mu}_{Z}-\mu_{Z}\|_{\mathcal{H}_{Z}}=O_{P}\left(n^{-\frac{1}{2}}\right) E n [ ξ X ˙ J ∣ Z ( ⋅ , X ¨ J ; Z ) ] H X J = O P ( n − 2 1 ) and ∥ μ Z − μ Z ∥ H Z = O P ( n − 2 1 )
by Hilbert space CLT; and ∥ ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ≤ ϵ n − 1 \|(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\|_{\mathrm{op}}\leq\epsilon_{n}^{-1} ∥ ( Σ Z Z + ϵ n I ) − 1 ∥ op ≤ ϵ n − 1 by spectrum calculation.
For the fourth term, by the definition of T μ ^ X J − μ X J T_{\widehat{\mu}_{X_{J}}-\mu_{X_{J}}} T μ X J − μ X J and ∥ Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ≤ 1 \|\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\|_{\mathrm{op}}\leq1 ∥ Σ Z Z ( Σ Z Z + ϵ n I ) − 1 ∥ op ≤ 1 , we have
∥ T μ ^ X J − μ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ≤ ∥ T μ ^ X J − μ X J ∥ o p ∥ Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ≤ ∥ μ ^ X J − μ X J ∥ H X J = O P ( n − 1 2 ) (46) \begin{aligned}
\left\|T_{\widehat{\mu}_{X_{J}}-\mu_{X_{J}}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}} \\
&\leq\left\|T_{\widehat{\mu}_{X_{J}}-\mu_{X_{J}}}\right\|_{\mathrm{op}}\left\|\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}} \\
&\leq\|\widehat{\mu}_{X_{J}}-\mu_{X_{J}}\|_{\mathcal{H}_{X_{J}}} \\
&=O_{P}\left(n^{-\frac{1}{2}}\right) \tag*{(46)}
\end{aligned} T μ X J − μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 op ≤ T μ X J − μ X J op Σ Z Z ( Σ Z Z + ϵ n I ) − 1 op ≤ ∥ μ X J − μ X J ∥ H X J = O P ( n − 2 1 ) (46)
For the fifth term,
∥ ϵ n ( B ^ X ˙ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 ( Σ ^ Z Z − Σ Z Z ) ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ≤ ϵ n ∥ ( B ^ X ˙ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 ∥ o p ∥ Σ ^ Z Z − Σ Z Z ∥ o p ∥ ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p \begin{aligned}
\left\|\epsilon_{n}\left(\widehat{B}_{\dot{X}_{J}\mid Z}-T_{\mu_{X_{J}}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)(\Sigma_{ZZ}+\epsilon_{n}I)^{-1}(\widehat{\Sigma}_{ZZ}-\Sigma_{ZZ})(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}} \\
&\leq\epsilon_{n}\left\|\left(\widehat{B}_{\dot{X}_{J}\mid Z}-T_{\mu_{X_{J}}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)(\Sigma_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}}\left\|\widehat{\Sigma}_{ZZ}-\Sigma_{ZZ}\right\|_{\mathrm{op}}\left\|(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\right\|_{\mathrm{op}}
\end{aligned} ϵ n ( B X ˙ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ Z Z ) ( Σ Z Z + ϵ n I ) − 1 op ≤ ϵ n ( B X ˙ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ( Σ Z Z + ϵ n I ) − 1 op Σ Z Z − Σ Z Z op ( Σ Z Z + ϵ n I ) − 1 op
≲ ϵ n ϵ n ( β ∧ 1 ) − 1 O P ( n − 1 2 ) ϵ n − 1 = O P ( n − 1 2 ϵ n ( β ∧ 1 ) − 1 ) (47) \begin{aligned}
\lesssim\epsilon_{n}\epsilon_{n}^{(\beta\wedge1)-1}O_{P}\left(n^{-\frac{1}{2}}\right)\epsilon_{n}^{-1} \\
&=O_{P}\left(n^{-\frac{1}{2}}\epsilon_{n}^{(\beta\wedge1)-1}\right) \tag*{(47)}
\end{aligned} ≲ ϵ n ϵ n ( β ∧ 1 ) − 1 O P ( n − 2 1 ) ϵ n − 1 = O P ( n − 2 1 ϵ n ( β ∧ 1 ) − 1 ) (47)
Therefore, we combine previous results (42)–(47) and conclude that
∥ ( B ^ X ˙ J ∣ Z − T μ ^ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 ) − ( B X ˙ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) ∥ o p = O P ( n − 1 2 ϵ n β ∧ 1 − 1 + ϵ n β ∧ 1 + n − 1 ϵ n − 3 α + 1 2 α + n − 1 2 ϵ n − 1 + α 2 α + n − 1 2 ) , \begin{aligned}
\left\|\left(\widehat{B}_{\dot{X}_{J}\mid Z}-T_{\widehat{\mu}_{X_{J}}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_{n}I)^{-1}\right)-\left(B_{\dot{X}_{J}\mid Z}-T_{\mu_{X_{J}}}\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right)\right\|_{\mathrm{op}} \\
&=O_{P}\left(n^{-\frac{1}{2}}\epsilon_{n}^{\beta\wedge1-1}+\epsilon_{n}^{\beta\wedge1}+n^{-1}\epsilon_{n}^{-\frac{3\alpha+1}{2\alpha}}+n^{-\frac{1}{2}}\epsilon_{n}^{-\frac{1+\alpha}{2\alpha}}+n^{-\frac{1}{2}}\right),
\end{aligned} ( B X ˙ J ∣ Z − T μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 ) − ( B X ˙ J ∣ Z − T μ X J Σ Z Z Σ Z Z † ) op = O P ( n − 2 1 ϵ n β ∧ 1 − 1 + ϵ n β ∧ 1 + n − 1 ϵ n − 2 α 3 α + 1 + n − 2 1 ϵ n − 2 α 1 + α + n − 2 1 ) ,
which proves assertion 2. □
Proof of Lemma 1. By the definition of R n R_n R n , adding and subtracting T μ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 T_{\mu_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1} T μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 we obtain
R n Σ Z Z θ = T μ ^ X J − μ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 Σ Z Z θ + T μ X J [ Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 − Σ Z Z Σ Z Z † ] Σ Z Z θ . R_n\Sigma_{ZZ}^{\theta}=T_{\widehat{\mu}_{X_J}-\mu_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta}+T_{\mu_{X_J}}\left[\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}-\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right]\Sigma_{ZZ}^{\theta}. R n Σ Z Z θ = T μ X J − μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ + T μ X J [ Σ Z Z ( Σ Z Z + ϵ n I ) − 1 − Σ Z Z Σ Z Z † ] Σ Z Z θ .
For the first term
∥ T μ ^ X J − μ X J Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 Σ Z Z θ ∥ o p ≤ ∥ μ ^ X J − μ X J ∥ H X J ∥ Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ∥ Σ Z Z θ ∥ o p = O P ( n − 1 / 2 ) (48) \left\|T_{\widehat{\mu}_{X_J}-\mu_{X_J}}\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}\leq\left\|\widehat{\mu}_{X_J}-\mu_{X_J}\right\|_{\mathcal{H}_{X_J}}\left\|\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}\left\|\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}=O_P(n^{-1/2}) \tag*{(48)} T μ X J − μ X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ op ≤ ∥ μ X J − μ X J ∥ H X J Σ Z Z ( Σ Z Z + ϵ n I ) − 1 op Σ Z Z θ op = O P ( n − 1/2 ) (48)
For the second term, since θ > 0 \theta>0 θ > 0 , we have Σ Z Z Σ Z Z † Σ Z Z θ = Σ Z Z θ \Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\Sigma_{ZZ}^{\theta}=\Sigma_{ZZ}^{\theta} Σ Z Z Σ Z Z † Σ Z Z θ = Σ Z Z θ . Then we obtain
[ Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 − Σ Z Z Σ Z Z † ] Σ Z Z θ = [ Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 − I ] Σ Z Z θ + [ I − Σ Z Z Σ Z Z † ] Σ Z Z θ = − ϵ n ( Σ ^ Z Z + ϵ n I ) − 1 Σ Z Z θ (49) \begin{aligned}
\left[\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}-\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right]\Sigma_{ZZ}^{\theta} \\
&=\left[\widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}-I\right]\Sigma_{ZZ}^{\theta}+\left[I-\Sigma_{ZZ}\Sigma_{ZZ}^{\dagger}\right]\Sigma_{ZZ}^{\theta} \\
&=-\epsilon_n(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta} \tag*{(49)}
\end{aligned} [ Σ Z Z ( Σ Z Z + ϵ n I ) − 1 − Σ Z Z Σ Z Z † ] Σ Z Z θ = [ Σ Z Z ( Σ Z Z + ϵ n I ) − 1 − I ] Σ Z Z θ + [ I − Σ Z Z Σ Z Z † ] Σ Z Z θ = − ϵ n ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ (49)
By the resolvent identity,
( Σ ^ Z Z + ϵ n I ) − 1 Σ Z Z θ = ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ + ( Σ ^ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ ^ Z Z ) ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ . (\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta}=(\Sigma_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta}+(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}(\Sigma_{ZZ}-\widehat{\Sigma}_{ZZ})(\Sigma_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta}. ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ = ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ + ( Σ Z Z + ϵ n I ) − 1 ( Σ Z Z − Σ Z Z ) ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ .
The spectral theorem gives ∥ ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ ∥ o p = O ( ϵ n θ ∧ 1 − 1 ) \left\|(\Sigma_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}=O(\epsilon_n^{\theta\wedge1-1}) ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ op = O ( ϵ n θ ∧ 1 − 1 ) . Combined with (33)
ϵ n ∥ ( Σ ^ Z Z + ϵ n I ) − 1 Σ Z Z θ ∥ o p ≤ ϵ n ∥ ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ ∥ o p + ϵ n ∥ ( Σ ^ Z Z + ϵ n I ) − 1 ∥ o p ∥ ( Σ Z Z − Σ ^ Z Z ) ∥ o p ∥ ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ ∥ o p ≤ ϵ n O ( ϵ n θ ∧ 1 − 1 ) + ϵ n ϵ n − 1 O P ( n − 1 / 2 ) O ( ϵ n θ ∧ 1 − 1 ) ≤ O P ( ϵ n θ ∧ 1 + n − 1 / 2 ϵ n θ ∧ 1 − 1 ) . (50) \begin{aligned}
\epsilon_n\left\|(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}
&\leq\epsilon_n\left\|(\Sigma_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}+\epsilon_n\left\|(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1}\right\|_{\mathrm{op}}\left\|(\Sigma_{ZZ}-\widehat{\Sigma}_{ZZ})\right\|_{\mathrm{op}}\left\|(\Sigma_{ZZ}+\epsilon_n I)^{-1}\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}\\
&\leq\epsilon_n O(\epsilon_n^{\theta\wedge1-1})+\epsilon_n\epsilon_n^{-1}O_P(n^{-1/2})O(\epsilon_n^{\theta\wedge1-1})\leq O_P(\epsilon_n^{\theta\wedge1}+n^{-1/2}\epsilon_n^{\theta\wedge1-1}). \tag*{(50)}
\end{aligned} ϵ n ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ op ≤ ϵ n ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ op + ϵ n ( Σ Z Z + ϵ n I ) − 1 op ( Σ Z Z − Σ Z Z ) op ( Σ Z Z + ϵ n I ) − 1 Σ Z Z θ op ≤ ϵ n O ( ϵ n θ ∧ 1 − 1 ) + ϵ n ϵ n − 1 O P ( n − 1/2 ) O ( ϵ n θ ∧ 1 − 1 ) ≤ O P ( ϵ n θ ∧ 1 + n − 1/2 ϵ n θ ∧ 1 − 1 ) . (50)
Since ∥ T μ X J ∥ o p = ∥ μ X J ∥ H X J < ∞ \|T_{\mu_{X_J}}\|_{\mathrm{op}}=\|\mu_{X_J}\|_{\mathcal{H}_{X_J}}<\infty ∥ T μ X J ∥ op = ∥ μ X J ∥ H X J < ∞ , equations (48) –(50) imply
∥ R n Σ Z Z θ ∥ o p = O P ( n − 1 / 2 + n − 1 / 2 ϵ n θ ∧ 1 − 1 + ϵ n θ ∧ 1 ) . \|R_n\Sigma_{ZZ}^{\theta}\|_{\mathrm{op}}=O_P(n^{-1/2}+n^{-1/2}\epsilon_n^{\theta\wedge1-1}+\epsilon_n^{\theta\wedge1}). ∥ R n Σ Z Z θ ∥ op = O P ( n − 1/2 + n − 1/2 ϵ n θ ∧ 1 − 1 + ϵ n θ ∧ 1 ) .
Finally, since ϵ n → 0 \epsilon_n\to0 ϵ n → 0 and θ ∧ 1 − 1 ≤ 0 \theta\wedge1-1\leq0 θ ∧ 1 − 1 ≤ 0 , we have n − 1 / 2 = O ( n − 1 / 2 ϵ n θ ∧ 1 − 1 ) n^{-1/2}=O(n^{-1/2}\epsilon_n^{\theta\wedge1-1}) n − 1/2 = O ( n − 1/2 ϵ n θ ∧ 1 − 1 ) . Therefore,
∥ R n Σ Z Z θ ∥ o p = O P ( n − 1 / 2 ϵ n θ ∧ 1 − 1 + ϵ n θ ∧ 1 ) , \|R_n\Sigma_{ZZ}^{\theta}\|_{\mathrm{op}}=O_P(n^{-1/2}\epsilon_n^{\theta\wedge1-1}+\epsilon_n^{\theta\wedge1}), ∥ R n Σ Z Z θ ∥ op = O P ( n − 1/2 ϵ n θ ∧ 1 − 1 + ϵ n θ ∧ 1 ) ,
which proves (6). □
Proof of Proposition 2. Fix j ∈ { 1 , … , d − 1 } j\in\{1,\ldots,d-1\} j ∈ { 1 , … , d − 1 } , and let R n R_n R n denote the operator in (5) with J = { j } J=\{j\} J = { j } . Since Σ Z Z \Sigma_{ZZ} Σ Z Z is a positive self-adjoint compact operator, its fractional power Σ Z Z 1 − θ \Sigma_{ZZ}^{1-\theta} Σ Z Z 1 − θ admits the spectral representation with eigenvalues { γ k 1 − θ } k \{\gamma_k^{1-\theta}\}_k { γ k 1 − θ } k . If we choose θ \theta θ such that
α + 1 2 α < θ < 1 − 1 2 α . \frac{\alpha+1}{2\alpha}<\theta<1-\frac{1}{2\alpha}. 2 α α + 1 < θ < 1 − 2 α 1 .
Such a θ \theta θ exists because α > 2 \alpha> 2 α > 2 . Moreover, the upper bound on θ \theta θ , together with Assumption 4, implies that
∥ Σ Z Z 1 − θ ∥ H S 2 = ∑ k = 1 ∞ γ k 2 ( 1 − θ ) ≲ ∑ k = 1 ∞ k − 2 α ( 1 − θ ) < ∞ . \left\|\Sigma_{ZZ}^{1-\theta}\right\|_{\mathrm{HS}}^2 = \sum_{k=1}^{\infty}\gamma_k^{2(1-\theta)} \lesssim\sum_{k=1}^{\infty} k^{-2\alpha(1-\theta)} < \infty. Σ Z Z 1 − θ HS 2 = k = 1 ∑ ∞ γ k 2 ( 1 − θ ) ≲ k = 1 ∑ ∞ k − 2 α ( 1 − θ ) < ∞.
It then follows from Lemma 1 and Proposition 1 that
n ∥ R n Σ Z Z ( B ^ X j + ∣ Z − B X j + ∣ Z ) ∗ ∥ H S ≤ n ∥ R n Σ Z Z θ ∥ o p ∥ Σ Z Z 1 − θ ∥ H S ∥ B ^ X j + ∣ Z − B X j + ∣ Z ∥ o p ≤ O P ( ε n θ − 1 n − α ( β ∧ 1 ) / ( 2 α ( β ∧ 1 ) + α + 1 ) + n ε n − α ( β ∧ 1 ) / ( 2 α ( β ∧ 1 ) + α + 1 ) ) ≤ O P ( n − α ( β ∧ 1 + θ − 1 ) / ( 2 α ( β ∧ 1 ) + α + 1 ) + n 1 / 2 − α ( β ∧ 1 + θ ) / ( 2 α ( β ∧ 1 ) + α + 1 ) ) = o P ( 1 ) . \begin{aligned}
\sqrt{n}\left\|R_n\Sigma_{ZZ}\left(\widehat{B}_{X_{j+}|Z}-B_{X_{j+}|Z}\right)^*\right\|_{\mathrm{HS}}
&\leq\sqrt{n}\left\|R_n\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}\left\|\Sigma_{ZZ}^{1-\theta}\right\|_{\mathrm{HS}}\left\|\widehat{B}_{X_{j+}|Z}-B_{X_{j+}|Z}\right\|_{\mathrm{op}}\\
&\leq O_P\left(\varepsilon_n^{\theta-1}n^{-\alpha(\beta\wedge1)/(2\alpha(\beta\wedge1)+\alpha+1)}+\sqrt{n}\varepsilon_n^{-\alpha(\beta\wedge1)/(2\alpha(\beta\wedge1)+\alpha+1)}\right)\\
&\leq O_P\left(n^{-\alpha(\beta\wedge1+\theta-1)/(2\alpha(\beta\wedge1)+\alpha+1)}+n^{1/2-\alpha(\beta\wedge1+\theta)/(2\alpha(\beta\wedge1)+\alpha+1)}\right)\\
&=o_P(1).
\end{aligned} n R n Σ Z Z ( B X j + ∣ Z − B X j + ∣ Z ) ∗ HS ≤ n R n Σ Z Z θ op Σ Z Z 1 − θ HS B X j + ∣ Z − B X j + ∣ Z op ≤ O P ( ε n θ − 1 n − α ( β ∧ 1 ) / ( 2 α ( β ∧ 1 ) + α + 1 ) + n ε n − α ( β ∧ 1 ) / ( 2 α ( β ∧ 1 ) + α + 1 ) ) ≤ O P ( n − α ( β ∧ 1 + θ − 1 ) / ( 2 α ( β ∧ 1 ) + α + 1 ) + n 1/2 − α ( β ∧ 1 + θ ) / ( 2 α ( β ∧ 1 ) + α + 1 ) ) = o P ( 1 ) .
Indeed, the exponent in the first term is negative because
β ∧ 1 + θ − 1 > α + 1 2 α + α + 1 2 α − 1 = 1 α > 0 , \beta\wedge1+\theta-1>\frac{\alpha+1}{2\alpha}+\frac{\alpha+1}{2\alpha}-1=\frac{1}{\alpha}>0, β ∧ 1 + θ − 1 > 2 α α + 1 + 2 α α + 1 − 1 = α 1 > 0 ,
whereas the exponent in the second term is negative because
1 2 − α ( β ∧ 1 + θ ) 2 α ( β ∧ 1 ) + α + 1 < 0 ⟺ θ > α + 1 2 α , \frac{1}{2}-\frac{\alpha(\beta\wedge1+\theta)}{2\alpha(\beta\wedge1)+\alpha+1}<0
\quad\Longleftrightarrow\quad
\theta>\frac{\alpha+1}{2\alpha}, 2 1 − 2 α ( β ∧ 1 ) + α + 1 α ( β ∧ 1 + θ ) < 0 ⟺ θ > 2 α α + 1 ,
which proves (7). □
Proof of Lemma 2. Fix j ∈ { 1 , … , d − 1 } j\in\{1,\ldots,d-1\} j ∈ { 1 , … , d − 1 } . Throughout this proof, R n R_n R n denotes the operator in (5) with J = { j } J=\{j\} J = { j } . Two intermediate operators are needed here. Let
Σ ~ X ¨ j X j + ∣ Z 1 = E n [ ξ ~ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] , \widetilde{\Sigma}^{1}_{\ddot X_jX_{j+}|Z}
=\mathbb{E}_n\left[\widetilde{\xi}_{\ddot X_j|Z}(\cdot,\ddot X_j;Z)\otimes\xi_{X_{j+}|Z}(\cdot,X_{j+};Z)\right], Σ X ¨ j X j + ∣ Z 1 = E n [ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] ,
Σ ~ X ¨ j X j + ∣ Z 2 = E n [ ( κ ~ X ¨ j ( ⋅ , X ¨ j ) − B ^ X ¨ j ∣ Z κ ~ Z ( ⋅ , Z ) ) ⊗ ( κ ~ X j + ( ⋅ , X j + ) − B ^ X j + ∣ Z κ ~ Z ( ⋅ , Z ) ) ] . \widetilde{\Sigma}^{2}_{\ddot X_jX_{j+}|Z}
=\mathbb{E}_n\left[\left(\widetilde{\kappa}_{\ddot X_j}(\cdot,\ddot X_j)-\widehat{B}_{\ddot X_j|Z}\widetilde{\kappa}_Z(\cdot,Z)\right)\otimes\left(\widetilde{\kappa}_{X_{j+}}(\cdot,X_{j+})-\widehat{B}_{X_{j+}|Z}\widetilde{\kappa}_Z(\cdot,Z)\right)\right]. Σ X ¨ j X j + ∣ Z 2 = E n [ ( κ X ¨ j ( ⋅ , X ¨ j ) − B X ¨ j ∣ Z κ Z ( ⋅ , Z ) ) ⊗ ( κ X j + ( ⋅ , X j + ) − B X j + ∣ Z κ Z ( ⋅ , Z ) ) ] .
Here Σ ^ X ¨ j X j + ∣ Z \widehat{\Sigma}_{\ddot X_jX_{j+}|Z} Σ X ¨ j X j + ∣ Z is the empirical cross-covariance of the fitted residuals, obtained from Σ ~ X ¨ j X j + ∣ Z 2 \widetilde{\Sigma}^{2}_{\ddot X_jX_{j+}|Z} Σ X ¨ j X j + ∣ Z 2 by replacing the population-centered kernels κ ~ \widetilde{\kappa} κ with the sample-centered kernels κ ^ \widehat{\kappa} κ . We first show that Σ ^ X ¨ j X j + ∣ Z \widehat{\Sigma}_{\ddot X_jX_{j+}|Z} Σ X ¨ j X j + ∣ Z and Σ ~ X ¨ j X j + ∣ Z 1 \widetilde{\Sigma}^{1}_{\ddot X_jX_{j+}|Z} Σ X ¨ j X j + ∣ Z 1 differ by o P ( n − 1 / 2 ) o_P(n^{-1/2}) o P ( n − 1/2 ) in Hilbert–Schmidt norm. We split it into two parts.
Let A = B ^ X ¨ j ∣ Z − B X ¨ j ∣ Z − R n A=\widehat{B}_{\ddot X_j|Z}-B_{\ddot X_j|Z}-R_n A = B X ¨ j ∣ Z − B X ¨ j ∣ Z − R n and C = B ^ X j + ∣ Z − B X j + ∣ Z C=\widehat{B}_{X_{j+}|Z}-B_{X_{j+}|Z} C = B X j + ∣ Z − B X j + ∣ Z . By Theorem 3 and Proposition 1,
∥ A ∥ o p + ∥ C ∥ o p = O P ( n − α ( β ∧ 1 ) / ( 2 α ( β ∧ 1 ) + α + 1 ) ) = o P ( n − 1 / 4 ) . (51) \|A\|_{\mathrm{op}}+\|C\|_{\mathrm{op}}
=O_P\left(n^{-\alpha(\beta\wedge1)/(2\alpha(\beta\wedge1)+\alpha+1)}\right)
=o_P(n^{-1/4}). \tag*{(51)} ∥ A ∥ op + ∥ C ∥ op = O P ( n − α ( β ∧ 1 ) / ( 2 α ( β ∧ 1 ) + α + 1 ) ) = o P ( n − 1/4 ) . (51)
Since the kernels are bounded and both Σ ^ Z Z ( Σ ^ Z Z + ϵ n I ) − 1 \widehat{\Sigma}_{ZZ}(\widehat{\Sigma}_{ZZ}+\epsilon_n I)^{-1} Σ Z Z ( Σ Z Z + ϵ n I ) − 1 and Σ Z Z Σ Z Z † \Sigma_{ZZ}\Sigma_{ZZ}^{\dagger} Σ Z Z Σ Z Z † have operator norm at most one,
∥ R n ∥ o p ≤ ∥ μ ^ X j ∥ H X j + ∥ μ X j ∥ H X j = O P ( 1 ) . \|R_n\|_{\mathrm{op}}\leq\|\widehat{\mu}_{X_j}\|_{\mathcal{H}_{X_j}}+\|\mu_{X_j}\|_{\mathcal{H}_{X_j}}=O_P(1). ∥ R n ∥ op ≤ ∥ μ X j ∥ H X j + ∥ μ X j ∥ H X j = O P ( 1 ) .
Together with the boundedness of the population regression operators, this gives
∥ B ^ X ¨ j ∣ Z ∥ o p + ∥ B ^ X j + ∣ Z ∥ o p = O P ( 1 ) . \left\|\widehat{B}_{\ddot{X}_j|Z}\right\|_{\mathrm{op}}+\left\|\widehat{B}_{X_{j+}|Z}\right\|_{\mathrm{op}}=O_P(1). B X ¨ j ∣ Z op + B X j + ∣ Z op = O P ( 1 ) .
We first consider the difference between Σ ^ X ¨ j X j + ∣ Z \widehat{\Sigma}_{\ddot{X}_jX_{j+}|Z} Σ X ¨ j X j + ∣ Z and Σ ~ X ¨ j X j + ∣ Z 2 \widetilde{\Sigma}^{2}_{\ddot{X}_jX_{j+}|Z} Σ X ¨ j X j + ∣ Z 2 . Since the kernels are bounded, ∥ μ ^ X − μ X ∥ H X = O P ( n − 1 / 2 ) \|\widehat{\mu}_X-\mu_X\|_{\mathcal{H}_X}=O_P(n^{-1/2}) ∥ μ X − μ X ∥ H X = O P ( n − 1/2 ) . Notice that κ ^ X ( ⋅ , X ) = κ ~ X ( ⋅ , X ) − ( μ ^ X − μ X ) \widehat{\kappa}_X(\cdot,X)=\widetilde{\kappa}_X(\cdot,X)-(\widehat{\mu}_X-\mu_X) κ X ( ⋅ , X ) = κ X ( ⋅ , X ) − ( μ X − μ X ) . Then by definition
Σ ^ X ¨ j X j + ∣ Z − Σ ~ X ¨ j X j + ∣ Z 2 = − [ ( μ ^ X ¨ j − μ X ¨ j ) − B ^ X ¨ j ∣ Z ( μ ^ Z − μ Z ) ] ⊗ [ ( μ ^ X j + − μ X j + ) − B ^ X j + ∣ Z ( μ ^ Z − μ Z ) ] . \widehat{\Sigma}_{\ddot{X}_jX_{j+}|Z}-\widetilde{\Sigma}^{2}_{\ddot{X}_jX_{j+}|Z}
=-\left[(\widehat{\mu}_{\ddot{X}_j}-\mu_{\ddot{X}_j})-\widehat{B}_{\ddot{X}_j|Z}(\widehat{\mu}_Z-\mu_Z)\right]\otimes\left[(\widehat{\mu}_{X_{j+}}-\mu_{X_{j+}})-\widehat{B}_{X_{j+}|Z}(\widehat{\mu}_Z-\mu_Z)\right]. Σ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z 2 = − [ ( μ X ¨ j − μ X ¨ j ) − B X ¨ j ∣ Z ( μ Z − μ Z ) ] ⊗ [ ( μ X j + − μ X j + ) − B X j + ∣ Z ( μ Z − μ Z ) ] .
Both factors are O P ( n − 1 / 2 ) O_P(n^{-1/2}) O P ( n − 1/2 ) in their respective reproducing kernel Hilbert space norms. Hence
n ∥ Σ ^ X ¨ j X j + ∣ Z − Σ ~ X ¨ j X j + ∣ Z 2 ∥ H S = O P ( n − 1 / 2 ) = o P ( 1 ) . \sqrt{n}\left\|\widehat{\Sigma}_{\ddot{X}_jX_{j+}|Z}-\widetilde{\Sigma}^{2}_{\ddot{X}_jX_{j+}|Z}\right\|_{\mathrm{HS}}=O_P(n^{-1/2})=o_P(1). n Σ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z 2 HS = O P ( n − 1/2 ) = o P ( 1 ) .
We next consider the difference between Σ ~ X ¨ j X j + ∣ Z 2 \widetilde{\Sigma}^{2}_{\ddot{X}_jX_{j+}|Z} Σ X ¨ j X j + ∣ Z 2 and Σ ~ X ¨ j X j + ∣ Z 1 \widetilde{\Sigma}^{1}_{\ddot{X}_jX_{j+}|Z} Σ X ¨ j X j + ∣ Z 1 . The boundedness of the kernels and the central limit theorem in Hilbert space give
∥ E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] − Σ Z Z ∥ H S = O P ( n − 1 / 2 ) , \left\|\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]-\Sigma_{ZZ}\right\|_{\mathrm{HS}}=O_P(n^{-1/2}), ∥ E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] − Σ Z Z ∥ HS = O P ( n − 1/2 ) ,
∥ E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ∥ H S = O P ( 1 ) . \left\|\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]\right\|_{\mathrm{HS}}=O_P(1). ∥ E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] ∥ HS = O P ( 1 ) .
We also know that
E [ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ∣ Z ] = 0 , E [ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ∣ Z ] = 0. \mathbb{E}\left[\xi_{\ddot{X}_j|Z}(\cdot,\ddot{X}_j;Z)\mid Z\right]=0,\qquad
\mathbb{E}\left[\xi_{X_{j+}|Z}(\cdot,X_{j+};Z)\mid Z\right]=0. E [ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ∣ Z ] = 0 , E [ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ∣ Z ] = 0.
Since the residuals are bounded, another application of the central limit theorem in Hilbert space gives
∥ E n [ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] ∥ H S = O P ( n − 1 / 2 ) , \left\|\mathbb{E}_n\left[\xi_{\ddot{X}_j|Z}(\cdot,\ddot{X}_j;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]\right\|_{\mathrm{HS}}=O_P(n^{-1/2}), E n [ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ κ Z ( ⋅ , Z ) ] HS = O P ( n − 1/2 ) ,
∥ E n [ κ ~ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] ∥ H S = O P ( n − 1 / 2 ) . \left\|\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\xi_{X_{j+}|Z}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}=O_P(n^{-1/2}). E n [ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] HS = O P ( n − 1/2 ) .
Notice that B ^ X ¨ j ∣ Z − B X ¨ j ∣ Z = A + R n \widehat{B}_{\ddot{X}_j|Z}-B_{\ddot{X}_j|Z}=A+R_n B X ¨ j ∣ Z − B X ¨ j ∣ Z = A + R n . Expanding the difference, we get
Σ ~ X ¨ j X j + ∣ Z 2 − Σ ~ X ¨ j X j + ∣ Z 1 = A E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] C ∗ − A E n [ κ ~ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] − E n [ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] C ∗ + R n E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] C ∗ − R n E n [ κ ~ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] . \begin{aligned}
\widetilde{\Sigma}^{2}_{\ddot{X}_jX_{j+}|Z}-\widetilde{\Sigma}^{1}_{\ddot{X}_jX_{j+}|Z}
={}&A\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]C^*\\
&-A\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\xi_{X_{j+}|Z}(\cdot,X_{j+};Z)\right]\\
&-\mathbb{E}_n\left[\xi_{\ddot{X}_j|Z}(\cdot,\ddot{X}_j;Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]C^*\\
&+R_n\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)\right]C^*\\
&-R_n\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\xi_{X_{j+}|Z}(\cdot,X_{j+};Z)\right].
\end{aligned} Σ X ¨ j X j + ∣ Z 2 − Σ X ¨ j X j + ∣ Z 1 = A E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] C ∗ − A E n [ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] − E n [ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ κ Z ( ⋅ , Z ) ] C ∗ + R n E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z ) ] C ∗ − R n E n [ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] .
By (51) and the preceding bounds, the first three terms are o P ( n − 1 / 2 ) o_P(n^{-1/2}) o P ( n − 1/2 ) in Hilbert–Schmidt norm. Now we only need to control the two terms involving R n R_n R n . By Proposition 2, n ∥ R n Σ Z Z C ∗ ∥ H S = o P ( 1 ) \sqrt{n}\lVert R_n\Sigma_{ZZ}C^*\rVert_{\mathrm{HS}}=o_P(1) n ∥ R n Σ Z Z C ∗ ∥ HS = o P ( 1 ) . Therefore,
n ∥ R n E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] C ∗ ∥ H S ≤ n ∥ R n Σ Z Z C ∗ ∥ H S + ∥ R n ∥ o p n ∥ E n [ κ ~ Z ( ⋅ , Z ) ⊗ κ ~ Z ( ⋅ , Z ) ] − Σ Z Z ∥ H S ∥ C ∥ o p = o P ( 1 ) + O P ( 1 ) O P ( 1 ) o P ( 1 ) = o P ( 1 ) . (52) \sqrt{n}\lVert R_n\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)]C^*\rVert_{\mathrm{HS}}
\leq\sqrt{n}\lVert R_n\Sigma_{ZZ}C^*\rVert_{\mathrm{HS}}+\lVert R_n\rVert_{\mathrm{op}}\sqrt{n}\lVert\mathbb{E}_n[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\kappa}_Z(\cdot,Z)]-\Sigma_{ZZ}\rVert_{\mathrm{HS}}\lVert C\rVert_{\mathrm{op}}
=o_P(1)+O_P(1)O_P(1)o_P(1)=o_P(1). \tag*{(52)} n ∥ R n E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z )] C ∗ ∥ HS ≤ n ∥ R n Σ Z Z C ∗ ∥ HS + ∥ R n ∥ op n ∥ E n [ κ Z ( ⋅ , Z ) ⊗ κ Z ( ⋅ , Z )] − Σ Z Z ∥ HS ∥ C ∥ op = o P ( 1 ) + O P ( 1 ) O P ( 1 ) o P ( 1 ) = o P ( 1 ) . (52)
For the remaining term, choose some constant θ \theta θ such that 0 < θ < α − 1 2 α 0<\theta<\frac{\alpha-1}{2\alpha} 0 < θ < 2 α α − 1 . Let { e k } k ≥ 1 \{e_k\}_{k\geq1} { e k } k ≥ 1 be orthonormal eigenfunctions of Σ Z Z \Sigma_{ZZ} Σ Z Z corresponding to its positive eigenvalues { γ k } k ≥ 1 \{\gamma_k\}_{k\geq1} { γ k } k ≥ 1 . Since κ ~ Z ( ⋅ , Z ) ∈ ker ( Σ Z Z ) ⊥ \widetilde{\kappa}_Z(\cdot,Z)\in\operatorname{ker}(\Sigma_{ZZ})^\perp κ Z ( ⋅ , Z ) ∈ ker ( Σ Z Z ) ⊥ almost surely, Parseval’s identity and Tonelli’s theorem give
E [ ∥ ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) ∥ H Z 2 ] = ∑ k = 1 ∞ γ k − 2 θ E [ ∣ ⟨ κ ~ Z ( ⋅ , Z ) , e k ⟩ H Z ∣ 2 ] = ∑ k = 1 ∞ γ k 1 − 2 θ ≲ ∑ k = 1 ∞ k − α ( 1 − 2 θ ) < ∞ . \mathbb{E}\left[\left\|(\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z)\right\|_{\mathcal{H}_Z}^2\right]
=\sum_{k=1}^{\infty}\gamma_k^{-2\theta}\mathbb{E}\left[\left|\left\langle\widetilde{\kappa}_Z(\cdot,Z),e_k\right\rangle_{\mathcal{H}_Z}\right|^2\right]
=\sum_{k=1}^{\infty}\gamma_k^{1-2\theta}\lesssim\sum_{k=1}^{\infty}k^{-\alpha(1-2\theta)}<\infty. E [ ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) H Z 2 ] = k = 1 ∑ ∞ γ k − 2 θ E [ ⟨ κ Z ( ⋅ , Z ) , e k ⟩ H Z 2 ] = k = 1 ∑ ∞ γ k 1 − 2 θ ≲ k = 1 ∑ ∞ k − α ( 1 − 2 θ ) < ∞.
In particular, ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) (\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z) ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) is well defined almost surely. Notice that Σ Z Z θ ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) = κ ~ Z ( ⋅ , Z ) \Sigma_{ZZ}^{\theta}(\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z)=\widetilde{\kappa}_Z(\cdot,Z) Σ Z Z θ ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) = κ Z ( ⋅ , Z ) almost surely. Then
n ∥ R n E n [ κ ~ Z ( ⋅ , Z ) ⊗ ξ ~ X j + ∣ Z ( ⋅ , X j + ; Z ) ] ∥ H S ≤ ∥ R n Σ Z Z θ ∥ o p n ∥ E n [ ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) ⊗ ξ ~ X j + ∣ Z ( ⋅ , X j + ; Z ) ] ∥ H S . (53) \sqrt{n}\left\|R_n\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\xi}_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}
\leq\left\|R_n\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}\sqrt{n}\left\|\mathbb{E}_n\left[(\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\xi}_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}. \tag*{(53)} n R n E n [ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] HS ≤ R n Σ Z Z θ op n E n [ ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] HS . (53)
Under the choice ϵ n = n − δ \epsilon_n=n^{-\delta} ϵ n = n − δ , where δ = α 2 α ( β ∧ 1 ) + α + 1 < 1 2 \delta=\frac{\alpha}{2\alpha(\beta\wedge1)+\alpha+1}<\frac{1}{2} δ = 2 α ( β ∧ 1 ) + α + 1 α < 2 1 , Lemma 1 gives
∥ R n Σ Z Z θ ∥ o p = O P ( n − 1 / 2 ϵ n θ − 1 + ϵ n θ ) = O P ( n − 1 / 2 + δ ( 1 − θ ) + n − δ θ ) = o P ( 1 ) . \left\|R_n\Sigma_{ZZ}^{\theta}\right\|_{\mathrm{op}}=O_P\left(n^{-1/2}\epsilon_n^{\theta-1}+\epsilon_n^\theta\right)
=O_P\left(n^{-1/2+\delta(1-\theta)}+n^{-\delta\theta}\right)=o_P(1). R n Σ Z Z θ op = O P ( n − 1/2 ϵ n θ − 1 + ϵ n θ ) = O P ( n − 1/2 + δ ( 1 − θ ) + n − δ θ ) = o P ( 1 ) .
Furthermore, the conditional mean-zero property of the residual gives
E [ ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) ⊗ ξ ~ X j + ∣ Z ( ⋅ , X j + ; Z ) ] = 0. \mathbb{E}\left[(\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\xi}_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right]=0. E [ ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] = 0.
Since ξ ~ X j + ∣ Z \widetilde{\xi}_{X_{j+}\mid Z} ξ X j + ∣ Z is bounded, we also have
E [ ∥ ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) ⊗ ξ ~ X j + ∣ Z ( ⋅ , X j + ; Z ) ∥ H S 2 ] ≲ E [ ∥ ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) ∥ H Z 2 ] < ∞ . \mathbb{E}\left[\left\|(\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\xi}_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right\|_{\mathrm{HS}}^2\right]\lesssim\mathbb{E}\left[\left\|(\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z)\right\|_{\mathcal{H}_Z}^2\right]<\infty. E [ ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) HS 2 ] ≲ E [ ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) H Z 2 ] < ∞.
It follows from independence across observations that
n E [ ∥ E n [ ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) ⊗ ξ ~ X j + ∣ Z ( ⋅ , X j + ; Z ) ] ∥ H S 2 ] = E [ ∥ ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) ⊗ ξ ~ X j + ∣ Z ( ⋅ , X j + ; Z ) ∥ H S 2 ] < ∞ . n\mathbb{E}\left[\left\|\mathbb{E}_n\left[(\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\xi}_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}^2\right]
=\mathbb{E}\left[\left\|(\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\xi}_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right\|_{\mathrm{HS}}^2\right]<\infty. n E [ E n [ ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] HS 2 ] = E [ ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) HS 2 ] < ∞.
Then by Chebyshev’s inequality,
n ∥ E n [ ( Σ Z Z † ) θ κ ~ Z ( ⋅ , Z ) ⊗ ξ ~ X j + ∣ Z ( ⋅ , X j + ; Z ) ] ∥ H S = O P ( 1 ) . \sqrt{n}\left\|\mathbb{E}_n\left[(\Sigma_{ZZ}^{\dagger})^\theta\widetilde{\kappa}_Z(\cdot,Z)\otimes\widetilde{\xi}_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}=O_P(1). n E n [ ( Σ Z Z † ) θ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] HS = O P ( 1 ) .
Now we come back to (53) and obtain
n ∥ R n E n [ κ ~ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] ∥ H S = o P ( 1 ) . (54) \sqrt{n}\left\|R_n\mathbb{E}_n\left[\widetilde{\kappa}_Z(\cdot,Z)\otimes\xi_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}=o_P(1). \tag*{(54)} n R n E n [ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ] HS = o P ( 1 ) . (54)
Combining (52) and (54) with the bounds for the first three terms proves
n ∥ Σ ~ X ¨ j X j + ∣ Z 2 − Σ ~ X ¨ j X j + ∣ Z 1 ∥ H S = o P ( 1 ) . \sqrt{n}\left\|\widetilde{\Sigma}^{2}_{\ddot{X}_jX_{j+}\mid Z}-\widetilde{\Sigma}^{1}_{\ddot{X}_jX_{j+}\mid Z}\right\|_{\mathrm{HS}}=o_P(1). n Σ X ¨ j X j + ∣ Z 2 − Σ X ¨ j X j + ∣ Z 1 HS = o P ( 1 ) .
Together with the first part, this gives
n ∥ Σ ^ X ¨ j X j + ∣ Z − Σ ~ X ¨ j X j + ∣ Z 1 ∥ H S = o P ( 1 ) . \sqrt{n}\left\|\widehat{\Sigma}_{\ddot{X}_jX_{j+}\mid Z}-\widetilde{\Sigma}^{1}_{\ddot{X}_jX_{j+}\mid Z}\right\|_{\mathrm{HS}}=o_P(1). n Σ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z 1 HS = o P ( 1 ) .
Recall that the influence function is
ψ ( X ¨ j , X j + ; Z ) = ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) − Σ X ¨ j X j + ∣ Z . (55) \psi(\ddot{X}_j,X_{j+};Z)=\xi_{\ddot{X}_j\mid Z}(\cdot,\ddot{X}_j;Z)\otimes\xi_{X_{j+}\mid Z}(\cdot,X_{j+};Z)-\Sigma_{\ddot{X}_jX_{j+}\mid Z}. \tag*{(55)} ψ ( X ¨ j , X j + ; Z ) = ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) − Σ X ¨ j X j + ∣ Z . (55)
By the definition of Σ ~ X ¨ j X j + ∣ Z 1 \widetilde{\Sigma}^{1}_{\ddot{X}_jX_{j+}\mid Z} Σ X ¨ j X j + ∣ Z 1 ,
n ( Σ ^ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) = n ( Σ ~ X ¨ j X j + ∣ Z 1 − Σ X ¨ j X j + ∣ Z ) + o P ( 1 ) = 1 n ∑ i = 1 n ψ ( X ¨ j , i , X j + , i ; Z i ) + o P ( 1 ) , \begin{aligned}
\sqrt{n}\left(\widehat{\Sigma}_{\ddot{X}_jX_{j+}\mid Z}-\Sigma_{\ddot{X}_jX_{j+}\mid Z}\right)
=\sqrt{n}\left(\widetilde{\Sigma}^{1}_{\ddot{X}_jX_{j+}\mid Z}-\Sigma_{\ddot{X}_jX_{j+}\mid Z}\right)+o_P(1) \\
&=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi(\ddot{X}_{j,i},X_{j+,i};Z_i)+o_P(1),
\end{aligned} n ( Σ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) = n ( Σ X ¨ j X j + ∣ Z 1 − Σ X ¨ j X j + ∣ Z ) + o P ( 1 ) = n 1 i = 1 ∑ n ψ ( X ¨ j , i , X j + , i ; Z i ) + o P ( 1 ) ,
where the remainders are measured in Hilbert–Schmidt norm. Since the kernels and regression operators are bounded,
E ∥ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ∥ H S 2 = E [ ∥ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ∥ H X j 2 ∥ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) ∥ H X j + 2 ] < ∞ . \mathbb{E}\left\|\xi_{\ddot{X}_j\mid Z}(\cdot,\ddot{X}_j;Z)\otimes\xi_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right\|_{\mathrm{HS}}^2
=\mathbb{E}\left[\left\|\xi_{\ddot{X}_j\mid Z}(\cdot,\ddot{X}_j;Z)\right\|_{\mathcal{H}_{X_j}}^2\left\|\xi_{X_{j+}\mid Z}(\cdot,X_{j+};Z)\right\|_{\mathcal{H}_{X_{j+}}}^2\right]<\infty. E ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + ; Z ) HS 2 = E [ ξ X ¨ j ∣ Z ( ⋅ , X ¨ j ; Z ) H X j 2 ξ X j + ∣ Z ( ⋅ , X j + ; Z ) H X j + 2 ] < ∞.
By the residual representation of Σ X ¨ j X j + ∣ Z \Sigma_{\ddot{X}_jX_{j+}\mid Z} Σ X ¨ j X j + ∣ Z , E [ ψ ( X ¨ j , X j + ; Z ) ] = 0 \mathbb{E}[\psi(\ddot{X}_j,X_{j+};Z)]=0 E [ ψ ( X ¨ j , X j + ; Z )] = 0 . The preceding bound also gives E ∥ ψ ( X ¨ j , X j + ; Z ) ∥ H S 2 < ∞ \mathbb{E}\|\psi(\ddot{X}_j,X_{j+};Z)\|_{\mathrm{HS}}^2<\infty E ∥ ψ ( X ¨ j , X j + ; Z ) ∥ HS 2 < ∞ . We can apply the central limit theorem in Hilbert space and Slutsky’s theorem to get
n ( Σ ^ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) → D N ( 0 , Γ X ¨ j X j + ∣ Z ) . \sqrt{n}\left(\widehat{\Sigma}_{\ddot{X}_jX_{j+}\mid Z}-\Sigma_{\ddot{X}_jX_{j+}\mid Z}\right)\xrightarrow{\mathcal{D}}N\left(0,\Gamma_{\ddot{X}_jX_{j+}\mid Z}\right). n ( Σ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) D N ( 0 , Γ X ¨ j X j + ∣ Z ) .
Proof of Theorem 4. For each j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 , the proof of Lemma 2 gives
n ( Σ ^ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) = 1 n ∑ i = 1 n ψ j , i + o P ( 1 ) , \sqrt{n}\left(\widehat{\Sigma}_{\ddot{X}_jX_{j+}\mid Z}-\Sigma_{\ddot{X}_jX_{j+}\mid Z}\right)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi_{j,i}+o_P(1), n ( Σ X ¨ j X j + ∣ Z − Σ X ¨ j X j + ∣ Z ) = n 1 i = 1 ∑ n ψ j , i + o P ( 1 ) ,
where
ψ j , i = ξ X ¨ j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + , i ; Z i ) − Σ X ¨ j X j + ∣ Z . \psi_{j,i}=\xi_{\ddot{X}_j\mid Z}(\cdot,\ddot{X}_{j,i};Z_i)\otimes\xi_{X_{j+}\mid Z}(\cdot,X_{j+,i};Z_i)-\Sigma_{\ddot{X}_jX_{j+}\mid Z}. ψ j , i = ξ X ¨ j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) ⊗ ξ X j + ∣ Z ( ⋅ , X j + , i ; Z i ) − Σ X ¨ j X j + ∣ Z .
Let ψ i = ( ψ 1 , i , … , ψ d − 1 , i ) T \psi_i=(\psi_{1,i},\ldots,\psi_{d-1,i})^{\mathsf{T}} ψ i = ( ψ 1 , i , … , ψ d − 1 , i ) T . Then ψ i \psi_i ψ i , as a random element of H S \mathbb{H}_S H S , is the joint influence function of S ^ \hat{S} S ^ . That is,
n ( S ^ − S ) = 1 n ∑ i = 1 n ψ i + o P ( 1 ) . (56) \sqrt{n}(\hat{S}-S)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi_i+o_P(1). \tag*{(56)} n ( S ^ − S ) = n 1 i = 1 ∑ n ψ i + o P ( 1 ) . (56)
By the residual representation of the conditional cross-covariance operators, E [ ψ i ] = 0 \mathbb{E}[\psi_i]=0 E [ ψ i ] = 0 . Moreover, Γ S ∣ Z = E [ ψ i ⊗ ψ i ] \Gamma_{S\mid Z}=\mathbb{E}[\psi_i\otimes\psi_i] Γ S ∣ Z = E [ ψ i ⊗ ψ i ] is trace-class. Therefore, the Hilbert space CLT gives
1 n ∑ i = 1 n ψ i → D N ( 0 , Γ S ∣ Z ) , \frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi_i\xrightarrow{\mathcal{D}}N(0,\Gamma_{S\mid Z}), n 1 i = 1 ∑ n ψ i D N ( 0 , Γ S ∣ Z ) ,
For any j , k ∈ { 1 , … , d − 1 } j,k\in\{1,\ldots,d-1\} j , k ∈ { 1 , … , d − 1 } , the ( j , k ) (j,k) ( j , k ) block of this covariance operator is
( Γ S ∣ Z ) ( j , k ) = E [ ψ j , i ⊗ ψ k , i ] . (\Gamma_{S\mid Z})_{(j,k)}=\mathbb{E}\left[\psi_{j,i}\otimes\psi_{k,i}\right]. ( Γ S ∣ Z ) ( j , k ) = E [ ψ j , i ⊗ ψ k , i ] .
Combining this Hilbert-space central limit theorem with (56) and Slutsky’s theorem proves (11). □
OF THEOREM 5. We first verify that the asymptotic linear representation established in the proof of Lemma 2 remains valid along { P n } \{P_n\} { P n } . Assumption 5 makes every bound used there uniform in n n n . Therefore, uniformly over all relevant subsets J J J ,
∥ μ ^ X J , n − μ X J , n ∥ H X J = O P n ( n − 1 / 2 ) , ∥ μ ^ X ~ J , n − μ X ~ J , n ∥ H X ~ J = O P n ( n − 1 / 2 ) , ∥ μ ^ Z , n − μ Z , n ∥ H Z = O P n ( n − 1 / 2 ) . (57) \begin{aligned}
\left\|\hat{\mu}_{X_J,n}-\mu_{X_J,n}\right\|_{\mathcal{H}_{X_J}}&=O_{P_n}(n^{-1/2}),\qquad
\left\|\hat{\mu}_{\widetilde{X}_J,n}-\mu_{\widetilde{X}_J,n}\right\|_{\mathcal{H}_{\widetilde{X}_J}}=O_{P_n}(n^{-1/2}),\\
\left\|\hat{\mu}_{Z,n}-\mu_{Z,n}\right\|_{\mathcal{H}_Z}&=O_{P_n}(n^{-1/2}). \tag*{(57)}
\end{aligned} ∥ μ ^ X J , n − μ X J , n ∥ H X J ∥ μ ^ Z , n − μ Z , n ∥ H Z = O P n ( n − 1/2 ) , μ ^ X J , n − μ X J , n H X J = O P n ( n − 1/2 ) , = O P n ( n − 1/2 ) . (57)
and
∥ Σ ^ X J Z , n − Σ X J Z , n ∥ H S = O P n ( n − 1 / 2 ) , ∥ Σ ^ X ~ J Z , n − Σ X ~ J Z , n ∥ H S = O P n ( n − 1 / 2 ) , ∥ Σ ^ Z Z , n − Σ Z Z , n ∥ H S = O P n ( n − 1 / 2 ) . (58) \begin{aligned}
\left\|\hat{\Sigma}_{X_JZ,n}-\Sigma_{X_JZ,n}\right\|_{\mathrm{HS}}&=O_{P_n}(n^{-1/2}),\qquad
\left\|\hat{\Sigma}_{\widetilde{X}_JZ,n}-\Sigma_{\widetilde{X}_JZ,n}\right\|_{\mathrm{HS}}=O_{P_n}(n^{-1/2}),\\
\left\|\hat{\Sigma}_{ZZ,n}-\Sigma_{ZZ,n}\right\|_{\mathrm{HS}}&=O_{P_n}(n^{-1/2}). \tag*{(58)}
\end{aligned} Σ ^ X J Z , n − Σ X J Z , n HS Σ ^ Z Z , n − Σ Z Z , n HS = O P n ( n − 1/2 ) , Σ ^ X J Z , n − Σ X J Z , n HS = O P n ( n − 1/2 ) , = O P n ( n − 1/2 ) . (58)
The uniform source bounds and the uniform decay rate of eigenvalues imply, by the proofs of Theorem 3, Corollary 1, and Lemma 1, that their conclusions hold their conclusions hold with O P O_P O P and o P o_P o P replaced by O P n O_{P_n} O P n and o P n o_{P_n} o P n , respectively. Here R n R_n R n is defined as in (5), with the population quantities replaced by their P n P_n P n versions; its dependence on the subset J J J is again suppressed. In particular,
∥ B ^ X J ∣ Z , n − B X J ∣ Z , n ∥ o p = O P n ( n − α ( β ∧ 1 ) 2 α ( β ∧ 1 ) + α + 1 ) = O P n ( n − 1 / 4 ) , (59) \left\|\hat{B}_{X_J\mid Z,n}-B_{X_J\mid Z,n}\right\|_{\mathrm{op}}
=O_{P_n}\left(n^{-\frac{\alpha(\beta\wedge1)}{2\alpha(\beta\wedge1)+\alpha+1}}\right)
=O_{P_n}(n^{-1/4}), \tag*{(59)} B ^ X J ∣ Z , n − B X J ∣ Z , n op = O P n ( n − 2 α ( β ∧ 1 ) + α + 1 α ( β ∧ 1 ) ) = O P n ( n − 1/4 ) , (59)
∥ B ^ X ~ J ∣ Z , n − B X ~ J ∣ Z , n − R n ∥ o p = o P n ( n − 1 / 4 ) , (60) \left\|\hat{B}_{\widetilde{X}_J\mid Z,n}-B_{\widetilde{X}_J\mid Z,n}-R_n\right\|_{\mathrm{op}}
=o_{P_n}(n^{-1/4}), \tag*{(60)} B ^ X J ∣ Z , n − B X J ∣ Z , n − R n op = o P n ( n − 1/4 ) , (60)
∥ R n Σ Z Z , n θ ∥ o p = O P n ( n − 1 / 2 ϵ n θ ∧ 1 − 1 + ϵ n θ ∧ 1 ) . (61) \left\|R_n\Sigma_{ZZ,n}^{\theta}\right\|_{\mathrm{op}}
=O_{P_n}\left(n^{-1/2}\epsilon_n^{\theta\wedge1-1}+\epsilon_n^{\theta\wedge1}\right). \tag*{(61)} R n Σ Z Z , n θ op = O P n ( n − 1/2 ϵ n θ ∧ 1 − 1 + ϵ n θ ∧ 1 ) . (61)
The uniform eigenvalue bounds imply
sup n ≥ 1 ∑ k = 1 ∞ γ k , n 1 − 2 θ < ∞ if 0 < θ < α − 1 2 α , (62) \sup_{n\ge1}\sum_{k=1}^{\infty}\gamma_{k,n}^{1-2\theta}<\infty
\quad\text{if }0<\theta<\frac{\alpha-1}{2\alpha}, \tag*{(62)} n ≥ 1 sup k = 1 ∑ ∞ γ k , n 1 − 2 θ < ∞ if 0 < θ < 2 α α − 1 , (62)
sup n ≥ 1 ∑ k = 1 ∞ γ k , n 2 ( 1 − θ ) < ∞ if θ < 1 − 1 2 α . (63) \sup_{n\ge1}\sum_{k=1}^{\infty}\gamma_{k,n}^{2(1-\theta)}<\infty\quad\text{if }\theta<1-\frac{1}{2\alpha}. \tag*{(63)} n ≥ 1 sup k = 1 ∑ ∞ γ k , n 2 ( 1 − θ ) < ∞ if θ < 1 − 2 α 1 . (63)
We now verify explicitly that the remainder terms in the asymptotic expansion are uniformly negligible. Fix j ∈ { 1 , … , d − 1 } j\in\{1,\ldots,d-1\} j ∈ { 1 , … , d − 1 } . In the remainder calculations below, R n R_n R n denotes the operator in (5) with J = { j } J=\{j\} J = { j } and population quantities evaluated under P n P_n P n . First, fix 0 < θ < α − 1 2 α 0<\theta<\frac{\alpha-1}{2\alpha} 0 < θ < 2 α α − 1 . By (62),
sup n ≥ 1 E P n ∥ ( Σ Z Z , n † ) θ κ ~ Z ( ⋅ , Z ) ∥ H Z 2 = sup n ≥ 1 ∑ k = 1 ∞ γ k , n 1 − 2 θ < ∞ . \sup_{n\ge1}\mathbb{E}_{P_n}\left\|\left(\Sigma_{ZZ,n}^{\dagger}\right)^\theta\widetilde{\kappa}_Z(\cdot,Z)\right\|_{\mathcal{H}_Z}^{2}
=\sup_{n\ge1}\sum_{k=1}^{\infty}\gamma_{k,n}^{1-2\theta}<\infty. n ≥ 1 sup E P n ( Σ Z Z , n † ) θ κ Z ( ⋅ , Z ) H Z 2 = n ≥ 1 sup k = 1 ∑ ∞ γ k , n 1 − 2 θ < ∞.
The conditional mean-zero property and the uniform boundedness of the residuals therefore give
n ∥ E P n [ ( Σ Z Z , n † ) θ κ ~ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z , n ( ⋅ , X j + ; Z ) ] ∥ H S = O P n ( 1 ) . (64) \sqrt{n}\left\|\mathbb{E}_{P_n}\left[\left(\Sigma_{ZZ,n}^{\dagger}\right)^\theta\widetilde{\kappa}_Z(\cdot,Z)\otimes\xi_{X_{j+}|Z,n}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}=O_{P_n}(1). \tag*{(64)} n E P n [ ( Σ Z Z , n † ) θ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z , n ( ⋅ , X j + ; Z ) ] HS = O P n ( 1 ) . (64)
Moreover, (61) gives
∥ R n Σ Z Z , n θ ∥ o p = O P n ( n − 1 / 2 ϵ n θ − 1 + ϵ n θ ) = O P n ( 1 ) . \left\|R_n\Sigma_{ZZ,n}^{\theta}\right\|_{\mathrm{op}}
=O_{P_n}\left(n^{-1/2}\epsilon_n^{\theta-1}+\epsilon_n^\theta\right)=O_{P_n}(1). R n Σ Z Z , n θ op = O P n ( n − 1/2 ϵ n θ − 1 + ϵ n θ ) = O P n ( 1 ) .
Consequently,
n ∥ E P n [ R n κ ~ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z , n ( ⋅ , X j + ; Z ) ] ∥ H S ≤ ∥ R n Σ Z Z , n θ ∥ o p n ∥ E P n [ ( Σ Z Z , n † ) θ κ ~ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z , n ( ⋅ , X j + ; Z ) ] ∥ H S = o P n ( 1 ) . (65) \begin{aligned}
\sqrt{n}\left\|\mathbb{E}_{P_n}\left[R_n\widetilde{\kappa}_Z(\cdot,Z)\otimes\xi_{X_{j+}|Z,n}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}
&\leq\left\|R_n\Sigma_{ZZ,n}^{\theta}\right\|_{\mathrm{op}}\sqrt{n}\left\|\mathbb{E}_{P_n}\left[\left(\Sigma_{ZZ,n}^{\dagger}\right)^\theta\widetilde{\kappa}_Z(\cdot,Z)\otimes\xi_{X_{j+}|Z,n}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}\\
&=o_{P_n}(1). \tag*{(65)}
\end{aligned} n E P n [ R n κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z , n ( ⋅ , X j + ; Z ) ] HS ≤ R n Σ Z Z , n θ op n E P n [ ( Σ Z Z , n † ) θ κ Z ( ⋅ , Z ) ⊗ ξ X j + ∣ Z , n ( ⋅ , X j + ; Z ) ] HS = o P n ( 1 ) . (65)
For the “quadratic” remainder, choose a possibly different value of θ \theta θ such that
α + 1 2 α < θ < 1 − 1 2 α . \frac{\alpha+1}{2\alpha}<\theta<1-\frac{1}{2\alpha}. 2 α α + 1 < θ < 1 − 2 α 1 .
Such a choice is possible because α > 2 \alpha>2 α > 2 . By (63),
sup n ≥ 1 ∥ Σ Z Z , n 1 − θ ∥ H S 2 = sup n ≥ 1 ∑ k = 1 ∞ γ k , n 2 ( 1 − θ ) < ∞ . \sup_{n\ge1}\left\|\Sigma_{ZZ,n}^{1-\theta}\right\|_{\mathrm{HS}}^2
=\sup_{n\ge1}\sum_{k=1}^{\infty}\gamma_{k,n}^{2(1-\theta)}<\infty. n ≥ 1 sup Σ Z Z , n 1 − θ HS 2 = n ≥ 1 sup k = 1 ∑ ∞ γ k , n 2 ( 1 − θ ) < ∞.
The argument used in Proposition 2, together with (59) and (61), then gives
n ∥ R n Σ Z Z , n ( B ^ X j + ∣ Z , n ∗ − B X j + ∣ Z , n ) ∥ H S = O P n ( 1 ) . (66) \sqrt{n}\left\|R_n\Sigma_{ZZ,n}\left(\widehat{B}_{X_{j+}|Z,n}^{*}-B_{X_{j+}|Z,n}\right)\right\|_{\mathrm{HS}}=O_{P_n}(1). \tag*{(66)} n R n Σ Z Z , n ( B X j + ∣ Z , n ∗ − B X j + ∣ Z , n ) HS = O P n ( 1 ) . (66)
We now list the remaining terms in the algebraic expansion used in the proof of Theorem 2. The sample-centering terms satisfy
n ∥ Σ ^ X j X j + ∣ Z , n − Σ ~ X j X j + ∣ Z , n 2 ∥ H S = O P n ( 1 ) . (67) \sqrt{n}\left\|\widehat{\Sigma}_{X_jX_{j+|Z,n}}-\widetilde{\Sigma}_{X_jX_{j+|Z,n}}^{2}\right\|_{\mathrm{HS}}=O_{P_n}(1). \tag*{(67)} n Σ X j X j + ∣ Z , n − Σ X j X j + ∣ Z , n 2 HS = O P n ( 1 ) . (67)
The linear terms involving the corrected regression-operator error satisfy
n ∥ ( B ^ X ⃗ j ∣ Z , n − B X ⃗ j ∣ Z , n − R n ) E n [ κ ~ ( Z , ⋅ ) ⊗ ξ ~ X j + ∣ Z , n ( ⋅ , X j + ; Z ) ] ∥ H S = o P n ( 1 ) , (68) \sqrt{n}\left\|\left(\widehat{B}_{\vec{X}_j|Z,n}-B_{\vec{X}_j|Z,n}-R_n\right)\mathbb{E}_n\left[\widetilde{\kappa}(Z,\cdot)\otimes\widetilde{\xi}_{X_{j+}|Z,n}(\cdot,X_{j+};Z)\right]\right\|_{\mathrm{HS}}=o_{P_n}(1), \tag*{(68)} n ( B X j ∣ Z , n − B X j ∣ Z , n − R n ) E n [ κ ( Z , ⋅ ) ⊗ ξ X j + ∣ Z , n ( ⋅ , X j + ; Z ) ] HS = o P n ( 1 ) , (68)
because the operator error is o P n ( n − 1 / 4 ) o_{P_n}(n^{-1/4}) o P n ( n − 1/4 ) , while all the mean-zero empirical average is O P n ( n − 1 / 2 ) O_{P_n}(n^{-1/2}) O P n ( n − 1/2 ) by Hilbert space CLT. Similarly,
n ∥ E n [ ξ ~ X ⃗ j ∣ Z , n ( ⋅ , X ~ j ; Z ) ⊗ κ ~ ( Z , ⋅ ) ] ( B ^ X j + ∣ Z , n − B X j + ∣ Z , n ) ∗ ∥ H S = o P n ( 1 ) . (69) \sqrt{n}\left\|\mathbb{E}_n\left[\widetilde{\xi}_{\vec{X}_j|Z,n}(\cdot,\widetilde{X}_j;Z)\otimes\widetilde{\kappa}(Z,\cdot)\right]\left(\widehat{B}_{X_{j+}|Z,n}-B_{X_{j+}|Z,n}\right)^*\right\|_{\mathrm{HS}}=o_{P_n}(1). \tag*{(69)} n E n [ ξ X j ∣ Z , n ( ⋅ , X j ; Z ) ⊗ κ ( Z , ⋅ ) ] ( B X j + ∣ Z , n − B X j + ∣ Z , n ) ∗ HS = o P n ( 1 ) . (69)
Finally, the quadratic operator-estimation term satisfies
n ∥ ( B ^ X ⃗ j ∣ Z , n − B X ⃗ j ∣ Z , n ) E n [ κ ~ ( Z , ⋅ ) ⊗ κ ~ ( Z , ⋅ ) ] ( B ^ X j + ∣ Z , n − B X j + ∣ Z , n ) ∗ ∥ H S = o P n ( 1 ) . (70) \sqrt{n}\left\|\left(\widehat{B}_{\vec{X}_j|Z,n}-B_{\vec{X}_j|Z,n}\right)\mathbb{E}_n\left[\widetilde{\kappa}(Z,\cdot)\otimes\widetilde{\kappa}(Z,\cdot)\right]\left(\widehat{B}_{X_{j+}|Z,n}-B_{X_{j+}|Z,n}\right)^*\right\|_{\mathrm{HS}}=o_{P_n}(1). \tag*{(70)} n ( B X j ∣ Z , n − B X j ∣ Z , n ) E n [ κ ( Z , ⋅ ) ⊗ κ ( Z , ⋅ ) ] ( B X j + ∣ Z , n − B X j + ∣ Z , n ) ∗ HS = o P n ( 1 ) . (70)
Indeed, after writing
B ^ X ⃗ j ∣ Z , n − B X ⃗ j ∣ Z , n = ( B ^ X ⃗ j ∣ Z , n − B X ⃗ j ∣ Z , n − R n ) + R n \widehat{B}_{\vec{X}_j|Z,n}-B_{\vec{X}_j|Z,n}=\left(\widehat{B}_{\vec{X}_j|Z,n}-B_{\vec{X}_j|Z,n}-R_n\right)+R_n B X j ∣ Z , n − B X j ∣ Z , n = ( B X j ∣ Z , n − B X j ∣ Z , n − R n ) + R n
and
E n [ κ ~ Z ⊗ κ ~ Z ] = Σ Z Z , n + { E n [ κ ~ Z ⊗ κ ~ Z ] − Σ Z Z , n } , \mathbb{E}_n[\widetilde{\kappa}_Z\otimes\widetilde{\kappa}_Z]=\Sigma_{ZZ,n}+\left\{\mathbb{E}_n[\widetilde{\kappa}_Z\otimes\widetilde{\kappa}_Z]-\Sigma_{ZZ,n}\right\}, E n [ κ Z ⊗ κ Z ] = Σ Z Z , n + { E n [ κ Z ⊗ κ Z ] − Σ Z Z , n } ,
the terms not containing R n Σ Z Z , n R_n\Sigma_{ZZ,n} R n Σ Z Z , n are o P n ( 1 ) o_{P_n}(1) o P n ( 1 ) by (59) –(60) and (58) , whereas the remaining term is o P n ( 1 ) o_{P_n}(1) o P n ( 1 ) by (66) . Combining (65) and (67) –(70) , we obtain, for every j = 1 , … , d − 1 j=1,\ldots,d-1 j = 1 , … , d − 1 ,
n ( Σ ^ X j X j + ∣ Z , n − Σ ^ X j X j + ∣ Z , n ) = 1 n ∑ i = 1 n ψ j , n , i + o P n ( 1 ) . \sqrt{n}\left(\widehat{\Sigma}_{X_jX_{j+}|Z,n}-\widehat{\Sigma}_{X_jX_{j+}|Z,n}\right)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi_{j,n,i}+o_{P_n}(1). n ( Σ X j X j + ∣ Z , n − Σ X j X j + ∣ Z , n ) = n 1 i = 1 ∑ n ψ j , n , i + o P n ( 1 ) .
Here ψ j , n , i \psi_{j,n,i} ψ j , n , i denotes the influence function ψ j , n \psi_{j,n} ψ j , n evaluated at the i i i th observation, and ψ n , i = ( ψ 1 , n , i , … , ψ d − 1 , n , i ) T \psi_{n,i}=(\psi_{1,n,i},\ldots,\psi_{d-1,n,i})^{\mathsf{T}} ψ n , i = ( ψ 1 , n , i , … , ψ d − 1 , n , i ) T . Combining the d − 1 d-1 d − 1 components gives the vector-form influence function
n ( S ^ n − S n ) = 1 n ∑ i = 1 n ψ n , i + o P n ( 1 ) . (71) \sqrt{n}(\widehat{S}_n-S_n)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi_{n,i}+o_{P_n}(1). \tag*{(71)} n ( S n − S n ) = n 1 i = 1 ∑ n ψ n , i + o P n ( 1 ) . (71)
The uniform kernel and source bounds imply for any n ≥ 1 n\ge1 n ≥ 1
P n ( ∥ ψ n ∥ H S ≤ C ) ≥ P n ( ∑ j = 1 d − 1 ∥ ψ j , n ∥ H S 2 ≤ C 2 ) ≥ P n ( ∑ j = 1 d − 1 ( ∥ ξ ~ X j ∣ Z , n ∥ H X j ∥ ξ ~ X j + ∣ Z , n ∥ H X j + + ∥ Σ ~ X j X j + ∣ Z , n ∥ H S ) 2 ≤ C 2 ) = 1 \begin{aligned}
P_n\left(\|\psi_n\|_{\mathcal{H}_S}\le C\right) \\
&\ge P_n\left(\sum_{j=1}^{d-1}\|\psi_{j,n}\|_{\mathrm{HS}}^2\le C^2\right) \\
&\ge P_n\left(\sum_{j=1}^{d-1}\left(\|\widetilde{\xi}_{X_j|Z,n}\|_{\mathcal{H}_{X_j}}\|\widetilde{\xi}_{X_{j+}|Z,n}\|_{\mathcal{H}_{X_{j+}}}+\|\widetilde{\Sigma}_{X_jX_{j+}|Z,n}\|_{\mathrm{HS}}\right)^2\le C^2\right) \\
&=1
\end{aligned} P n ( ∥ ψ n ∥ H S ≤ C ) ≥ P n ( j = 1 ∑ d − 1 ∥ ψ j , n ∥ HS 2 ≤ C 2 ) ≥ P n ( j = 1 ∑ d − 1 ( ∥ ξ X j ∣ Z , n ∥ H X j ∥ ξ X j + ∣ Z , n ∥ H X j + + ∥ Σ X j X j + ∣ Z , n ∥ HS ) 2 ≤ C 2 ) = 1
for some constant C < ∞ C < \infty C < ∞ independent of n n n . Hence almost surely
max 1 ≤ i ≤ n ∥ ψ n , i n ∥ H S ≤ C n ⟶ 0. \max_{1\leq i\leq n}\left\|\frac{\psi_{n,i}}{\sqrt{n}}\right\|_{\mathcal{H}_S}\leq\frac{C}{\sqrt{n}}\longrightarrow0. 1 ≤ i ≤ n max n ψ n , i H S ≤ n C ⟶ 0.
Then the Lindeberg condition holds. Together with (14) , the Hilbert-space CLT for triangular arrays gives
1 n ∑ i = 1 n ψ n , i → D G , where G ∼ N ( 0 , Γ S ∣ Z , 0 ) . \frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi_{n,i}\xrightarrow{\mathcal{D}}G,\qquad\text{where }G\sim N(0,\Gamma_{S\mid Z,0}). n 1 i = 1 ∑ n ψ n , i D G , where G ∼ N ( 0 , Γ S ∣ Z , 0 ) .
Combining this with (71) proves (16) .
Under (15) ,
n S ^ n = n ( S ^ n − S n ) + S h → D G + S h . \sqrt{n}\widehat{S}_n=\sqrt{n}(\widehat{S}_n-S_n)+S_h\xrightarrow{\mathcal{D}}G+S_h. n S n = n ( S n − S n ) + S h D G + S h .
The continuous mapping theorem proves (17) . Finally, the Karhunen–Loève expansion gives
G = ∑ k = 1 ∞ λ k Z k v k . G=\sum_{k=1}^{\infty}\sqrt{\lambda_k}Z_kv_k. G = k = 1 ∑ ∞ λ k Z k v k .
Since S h ∈ ker ( Γ S ∣ Z , 0 ) ⊥ S_h\in\ker(\Gamma_{S\mid Z,0})^\perp S h ∈ ker ( Γ S ∣ Z , 0 ) ⊥ , Parseval’s identity therefore yields
∥ G + S h ∥ H S 2 = ∑ k = 1 ∞ ( λ k Z k + σ k ) 2 , \|G+S_h\|_{\mathcal{H}_S}^{2}=\sum_{k=1}^{\infty}\left(\sqrt{\lambda_k}Z_k+\sigma_k\right)^2, ∥ G + S h ∥ H S 2 = k = 1 ∑ ∞ ( λ k Z k + σ k ) 2 ,
which proves (18) . □
Proofs for Section 5
Proof of Theorem 6. Assertion 1 follows immediately from Theorem 1. Now we prove assertion 2. We need to show for any j ∈ { 1 , … , d − 1 } j\in\{1,\ldots,d-1\} j ∈ { 1 , … , d − 1 } ,
n ( Σ ^ X j X j + − Σ X j X j + ) → D N ( 0 , Γ X j X j + ) (72) \sqrt{n}\left(\widehat{\Sigma}_{X_jX_{j+}}-\Sigma_{X_jX_{j+}}\right)\xrightarrow{\mathcal{D}}N\left(0,\Gamma_{X_jX_{j+}}\right) \tag*{(72)} n ( Σ X j X j + − Σ X j X j + ) D N ( 0 , Γ X j X j + ) (72)
where Γ X j X j + = E [ ψ j u c ⊗ ψ j u c ] \Gamma_{X_jX_{j+}}=\mathbb{E}\left[\psi_j^{\mathrm{uc}}\otimes\psi_j^{\mathrm{uc}}\right] Γ X j X j + = E [ ψ j uc ⊗ ψ j uc ] . Recall that
Σ ^ X j X j + = E n [ κ ~ X j ⊗ κ ~ X j + ] − E n [ κ ~ X j ] ⊗ E n [ κ ~ X j + ] \widehat{\Sigma}_{X_jX_{j+}}=\mathbb{E}_n\left[\widetilde{\kappa}_{X_j}\otimes\widetilde{\kappa}_{X_{j+}}\right]-\mathbb{E}_n\left[\widetilde{\kappa}_{X_j}\right]\otimes\mathbb{E}_n\left[\widetilde{\kappa}_{X_{j+}}\right] Σ X j X j + = E n [ κ X j ⊗ κ X j + ] − E n [ κ X j ] ⊗ E n [ κ X j + ]
Thus
n ( Σ ^ X j X j + − Σ X j X j + ) = n ( E n [ κ ~ X j ⊗ κ ~ X j + ] − E [ κ ~ X j ⊗ κ ~ X j + ] ) − n E n [ κ ~ X j ] ⊗ E n [ κ ~ X j + ] . (73) \begin{aligned}
\sqrt{n}\left(\widehat{\Sigma}_{X_jX_{j+}}-\Sigma_{X_jX_{j+}}\right) \\
&= \sqrt{n}\left(\mathbb{E}_n\left[\tilde{\kappa}_{X_j}\otimes\tilde{\kappa}_{X_{j+}}\right]-\mathbb{E}\left[\tilde{\kappa}_{X_j}\otimes\tilde{\kappa}_{X_{j+}}\right]\right)-\sqrt{n}\mathbb{E}_n\left[\tilde{\kappa}_{X_j}\right]\otimes\mathbb{E}_n\left[\tilde{\kappa}_{X_{j+}}\right]. \tag*{(73)}
\end{aligned} n ( Σ X j X j + − Σ X j X j + ) = n ( E n [ κ ~ X j ⊗ κ ~ X j + ] − E [ κ ~ X j ⊗ κ ~ X j + ] ) − n E n [ κ ~ X j ] ⊗ E n [ κ ~ X j + ] . (73)
Since all the kernels are bounded, by Hilbert space CLT,
n ( E n [ κ ~ X j ⊗ κ ~ X j + ] − E [ κ ~ X j ⊗ κ ~ X j + ] ) → D N ( 0 , Γ X j X j + ) . (74) \sqrt{n}\left(\mathbb{E}_n\left[\tilde{\kappa}_{X_j}\otimes\tilde{\kappa}_{X_{j+}}\right]-\mathbb{E}\left[\tilde{\kappa}_{X_j}\otimes\tilde{\kappa}_{X_{j+}}\right]\right)\xrightarrow{\mathcal{D}}N\left(0,\Gamma_{X_jX_{j+}}\right). \tag*{(74)} n ( E n [ κ ~ X j ⊗ κ ~ X j + ] − E [ κ ~ X j ⊗ κ ~ X j + ] ) D N ( 0 , Γ X j X j + ) . (74)
The centered kernels have mean zero, then again by Hilbert space CLT (or Chebyshev inequality)
n E n [ κ ~ X j ] ⊗ E n [ κ ~ X j + ] = n 1 / 2 O P ( n − 1 / 2 ) O P ( n − 1 / 2 ) = o P ( 1 ) . (75) \sqrt{n}\mathbb{E}_n\left[\tilde{\kappa}_{X_j}\right]\otimes\mathbb{E}_n\left[\tilde{\kappa}_{X_{j+}}\right]=n^{1/2}O_P(n^{-1/2})O_P(n^{-1/2})=o_P(1). \tag*{(75)} n E n [ κ ~ X j ] ⊗ E n [ κ ~ X j + ] = n 1/2 O P ( n − 1/2 ) O P ( n − 1/2 ) = o P ( 1 ) . (75)
Combining (73)–(75) gives the desired results (72) . For assertion 2, we can use the fact obtained above
n ( Σ ^ X j X j + − Σ X j X j + ) = 1 n ∑ i = 1 n ψ j , i u c + o P ( 1 ) . \sqrt{n}\left(\widehat{\Sigma}_{X_jX_{j+}}-\Sigma_{X_jX_{j+}}\right)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi^{\mathrm{uc}}_{j,i}+o_P(1). n ( Σ X j X j + − Σ X j X j + ) = n 1 i = 1 ∑ n ψ j , i uc + o P ( 1 ) .
Let ψ i u c = ( ψ 1 , i u c , … , ψ d − 1 , i u c ) T \psi^{\mathrm{uc}}_i=(\psi^{\mathrm{uc}}_{1,i},\ldots,\psi^{\mathrm{uc}}_{d-1,i})^\mathsf{T} ψ i uc = ( ψ 1 , i uc , … , ψ d − 1 , i uc ) T . Combining d − 1 d-1 d − 1 components, we get
n ( S ^ u c − S u c ) = 1 n ∑ i = 1 n ψ i u c + o P ( 1 ) . \sqrt{n}\left(\widehat{S}^{\mathrm{uc}}-S^{\mathrm{uc}}\right)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi^{\mathrm{uc}}_i+o_P(1). n ( S uc − S uc ) = n 1 i = 1 ∑ n ψ i uc + o P ( 1 ) .
The random elements ψ i u c \psi^{\mathrm{uc}}_i ψ i uc are i.i.d., with E [ ψ i u c ] = 0 \mathbb{E}[\psi^{\mathrm{uc}}_i]=0 E [ ψ i uc ] = 0 . Boundedness of kernels implies E ∥ ψ i u c ∥ H S 2 < ∞ \mathbb{E}\lVert\psi^{\mathrm{uc}}_i\rVert_{\mathcal{H}_S}^2<\infty E ∥ ψ i uc ∥ H S 2 < ∞ . Therefore, the Hilbert space CLT and Slutsky’s theorem give
n ( S ^ u c − S u c ) → D N ( 0 , Γ S u c ) , where Γ S u c = E [ ψ i u c ⊗ ψ i u c ] , \sqrt{n}\left(\widehat{S}^{\mathrm{uc}}-S^{\mathrm{uc}}\right)\xrightarrow{\mathcal{D}}N\left(0,\Gamma^{\mathrm{uc}}_S\right),\qquad\text{where }\Gamma^{\mathrm{uc}}_S=\mathbb{E}\left[\psi^{\mathrm{uc}}_i\otimes\psi^{\mathrm{uc}}_i\right], n ( S uc − S uc ) D N ( 0 , Γ S uc ) , where Γ S uc = E [ ψ i uc ⊗ ψ i uc ] ,
which proves assertion 2. Assertions 3 just follow as corollaries. □
Proof of Theorem 7. For each relevant subset J J J , boundedness of kernels and independence imply
E P n ∥ μ ^ X J , n − μ X J , n ∥ H X J 2 = 1 n E P n ∥ κ ~ X J , n ( ⋅ , X J ) ∥ H X J 2 = 1 n ( E P n [ κ X J ( X J , X J ) ] − ∥ μ X J , n ∥ H X J 2 ) ≤ 1 n sup x J κ X J ( x J , x J ) . \mathbb{E}_{P_n}\left\lVert\widehat{\mu}_{X_J,n}-\mu_{X_J,n}\right\rVert_{\mathcal{H}_{X_J}}^2
=\frac{1}{n}\mathbb{E}_{P_n}\left\lVert\tilde{\kappa}_{X_J,n}(\cdot,X_J)\right\rVert_{\mathcal{H}_{X_J}}^2
=\frac{1}{n}\left(\mathbb{E}_{P_n}\left[\kappa_{X_J}(X_J,X_J)\right]-\left\lVert\mu_{X_J,n}\right\rVert_{\mathcal{H}_{X_J}}^2\right)
\leq\frac{1}{n}\sup_{x_J}\kappa_{X_J}(x_J,x_J). E P n ∥ μ X J , n − μ X J , n ∥ H X J 2 = n 1 E P n ∥ κ ~ X J , n ( ⋅ , X J ) ∥ H X J 2 = n 1 ( E P n [ κ X J ( X J , X J ) ] − ∥ μ X J , n ∥ H X J 2 ) ≤ n 1 x J sup κ X J ( x J , x J ) .
Thus
∥ μ ^ X J , n − μ X J , n ∥ H X J = O P n ( n − 1 / 2 ) . \left\lVert\widehat{\mu}_{X_J,n}-\mu_{X_J,n}\right\rVert_{\mathcal{H}_{X_J}}=O_{P_n}(n^{-1/2}). ∥ μ X J , n − μ X J , n ∥ H X J = O P n ( n − 1/2 ) .
Fix j ∈ { 1 , … , d − 1 } j\in\{1,\ldots,d-1\} j ∈ { 1 , … , d − 1 } . By simple calculation,
Σ ^ X j X j + , n = E n [ κ ~ X j , n ( ⋅ , X j ) ⊗ κ ~ X j + , n ( ⋅ , X j + ) ] − ( μ ^ X j , n − μ X j , n ) ⊗ ( μ ^ X j + , n − μ X j + , n ) . \widehat{\Sigma}_{X_jX_{j+},n}=\mathbb{E}_n\left[\tilde{\kappa}_{X_j,n}(\cdot,X_j)\otimes\tilde{\kappa}_{X_{j+},n}(\cdot,X_{j+})\right]-\left(\widehat{\mu}_{X_j,n}-\mu_{X_j,n}\right)\otimes\left(\widehat{\mu}_{X_{j+},n}-\mu_{X_{j+},n}\right). Σ X j X j + , n = E n [ κ ~ X j , n ( ⋅ , X j ) ⊗ κ ~ X j + , n ( ⋅ , X j + ) ] − ( μ X j , n − μ X j , n ) ⊗ ( μ X j + , n − μ X j + , n ) .
Let ψ j , n , i u c \psi^{\mathrm{uc}}_{j,n,i} ψ j , n , i uc denote ψ j , n u c \psi^{\mathrm{uc}}_{j,n} ψ j , n uc evaluated at observation i i i . Then
n ( Σ ^ X j X j + , n − Σ X j X j + , n ) = 1 n ∑ i = 1 n ψ j , n , i u c − n ( μ ^ X j , n − μ X j , n ) ⊗ ( μ ^ X j + , n − μ X j + , n ) . \sqrt{n}\left(\widehat{\Sigma}_{X_jX_{j+},n}-\Sigma_{X_jX_{j+},n}\right)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi^{\mathrm{uc}}_{j,n,i}-\sqrt{n}\left(\widehat{\mu}_{X_j,n}-\mu_{X_j,n}\right)\otimes\left(\widehat{\mu}_{X_{j+},n}-\mu_{X_{j+},n}\right). n ( Σ X j X j + , n − Σ X j X j + , n ) = n 1 i = 1 ∑ n ψ j , n , i uc − n ( μ X j , n − μ X j , n ) ⊗ ( μ X j + , n − μ X j + , n ) .
It follows that
n ∥ ( μ ^ X j , n − μ X j , n ) ⊗ ( μ ^ X j + , n − μ X j + , n ) ∥ H S = n ∥ μ ^ X j , n − μ X j , n ∥ H X j ∥ μ ^ X j + , n − μ X j + , n ∥ H X j + = O P n ( n − 1 / 2 ) = O P n ( 1 ) . \begin{aligned}
\sqrt{n}\left\|\left(\widehat{\mu}_{X_j,n}-\mu_{X_j,n}\right)\otimes\left(\widehat{\mu}_{X_{j+},n}-\mu_{X_{j+},n}\right)\right\|_{\mathrm{HS}} \\
&=\sqrt{n}\left\|\widehat{\mu}_{X_j,n}-\mu_{X_j,n}\right\|_{\mathcal{H}_{X_j}}\left\|\widehat{\mu}_{X_{j+},n}-\mu_{X_{j+},n}\right\|_{\mathcal{H}_{X_{j+}}} \\
&=O_{\mathbb{P}_n}\left(n^{-1/2}\right)=O_{\mathbb{P}_n}(1).
\end{aligned} n ( μ X j , n − μ X j , n ) ⊗ ( μ X j + , n − μ X j + , n ) HS = n μ X j , n − μ X j , n H X j μ X j + , n − μ X j + , n H X j + = O P n ( n − 1/2 ) = O P n ( 1 ) .
Combining the d − 1 d-1 d − 1 components gives the vector-form influence function
n ( S ^ n u c − S n u c ) = 1 n ∑ i = 1 n ψ n , i u c + o P n ( 1 ) . (76) \sqrt{n}\left(\widehat{S}^{\mathrm{uc}}_n-S^{\mathrm{uc}}_n\right)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi^{\mathrm{uc}}_{n,i}+o_{\mathbb{P}_n}(1). \tag*{(76)} n ( S n uc − S n uc ) = n 1 i = 1 ∑ n ψ n , i uc + o P n ( 1 ) . (76)
where ψ n u c = ( ψ 1 , n , 1 u c , … , ψ d − 1 , n , 1 u c ) T \psi^{\mathrm{uc}}_n=\left(\psi^{\mathrm{uc}}_{1,n,1},\ldots,\psi^{\mathrm{uc}}_{d-1,n,1}\right)^{\mathsf{T}} ψ n uc = ( ψ 1 , n , 1 uc , … , ψ d − 1 , n , 1 uc ) T .
Recall that the residual term has mean-zero property, i.e., E P n [ ψ n u c ] = 0 \mathbb{E}_{\mathbb{P}_n}[\psi^{\mathrm{uc}}_n]=0 E P n [ ψ n uc ] = 0 . The boundedness of kernels then implies for every n ≥ 1 n\ge1 n ≥ 1 ,
P n ( ∥ ψ n u c ∥ H S ≤ C ) = P n ( ∑ j = 1 d − 1 ∥ ψ j , n u c ∥ H S 2 ≤ C 2 ) ≥ P n ( ∑ j = 1 d − 1 ( ∥ K ~ X j , n ( ⋅ , X j ) ∥ H X j ∥ K ~ X j + , n ( ⋅ , X j + ) ∥ H X j + + ∥ Σ X j X j + , n ∥ H S ) 2 ≤ C 2 ) = 1 \begin{aligned}
\mathbb{P}_n\left(\left\|\psi^{\mathrm{uc}}_n\right\|_{\mathrm{HS}}\le C\right) \\
&=\mathbb{P}_n\left(\sum_{j=1}^{d-1}\left\|\psi^{\mathrm{uc}}_{j,n}\right\|_{\mathrm{HS}}^2\le C^2\right) \\
&\ge\mathbb{P}_n\left(\sum_{j=1}^{d-1}\left(\left\|\widetilde{K}_{X_j,n}(\cdot,X_j)\right\|_{\mathcal{H}_{X_j}}\left\|\widetilde{K}_{X_{j+},n}(\cdot,X_{j+})\right\|_{\mathcal{H}_{X_{j+}}}+\left\|\Sigma_{X_jX_{j+},n}\right\|_{\mathrm{HS}}\right)^2\le C^2\right) \\
&=1
\end{aligned} P n ( ∥ ψ n uc ∥ HS ≤ C ) = P n ( j = 1 ∑ d − 1 ψ j , n uc HS 2 ≤ C 2 ) ≥ P n ( j = 1 ∑ d − 1 ( K X j , n ( ⋅ , X j ) H X j K X j + , n ( ⋅ , X j + ) H X j + + Σ X j X j + , n HS ) 2 ≤ C 2 ) = 1
for some constant C < ∞ C<\infty C < ∞ independent of n n n . Hence almost surely,
max 1 ≤ i ≤ n ∥ ψ n , i u c n ∥ H S ≤ C n ⟶ 0. \max_{1\le i\le n}\left\|\frac{\psi^{\mathrm{uc}}_{n,i}}{\sqrt{n}}\right\|_{\mathrm{HS}}\le\frac{C}{\sqrt{n}}\longrightarrow0. 1 ≤ i ≤ n max n ψ n , i uc HS ≤ n C ⟶ 0.
Thus the Lindeberg condition holds. Together with (21), the Hilbert-space CLT for triangular arrays gives
1 n ∑ i = 1 n ψ n , i u c → D G , where G ∼ N ( 0 , Γ S , 0 u c ) . \frac{1}{\sqrt{n}}\sum_{i=1}^{n}\psi^{\mathrm{uc}}_{n,i}\xrightarrow{\mathcal{D}}G,\quad\text{where }G\sim N\left(0,\Gamma^{\mathrm{uc}}_{S,0}\right). n 1 i = 1 ∑ n ψ n , i uc D G , where G ∼ N ( 0 , Γ S , 0 uc ) .
Combining this with (76) proves (23). Under (22),
n S ^ n u c = n ( S ^ n u c − S n u c ) + S h u c → D G + S h u c . \sqrt{n}\widehat{S}^{\mathrm{uc}}_n=\sqrt{n}\left(\widehat{S}^{\mathrm{uc}}_n-S^{\mathrm{uc}}_n\right)+S^{\mathrm{uc}}_h\xrightarrow{\mathcal{D}}G+S^{\mathrm{uc}}_h. n S n uc = n ( S n uc − S n uc ) + S h uc D G + S h uc .
The continuous mapping theorem proves (24) . Finally, the Karhunen–Loève expansion gives
G = ∑ k = 1 ∞ λ k Z k v k . G = \sum_{k=1}^{\infty} \sqrt{\lambda_k}Z_kv_k. G = k = 1 ∑ ∞ λ k Z k v k .
Since S h u c ∈ ker ( Γ S , 0 u c ) ⊥ S_h^{\mathrm{uc}} \in\ker(\Gamma_{S,0}^{\mathrm{uc}})^\perp S h uc ∈ ker ( Γ S , 0 uc ) ⊥ , Parseval’s identity therefore yields
∥ G + S h u c ∥ H S 2 = ∑ k = 1 ∞ ( λ k Z k + σ k ) 2 , \left\|G+S_h^{\mathrm{uc}}\right\|_{\mathcal{H}_S}^{2}
= \sum_{k=1}^{\infty}\left(\sqrt{\lambda_k}Z_k+\sigma_k\right)^2, ∥ G + S h uc ∥ H S 2 = k = 1 ∑ ∞ ( λ k Z k + σ k ) 2 ,
which proves (25). □
Proofs for Section 6
Proof of Lemma 3. Let ψ ^ i \hat{\psi}_i ψ ^ i and ψ i \psi_i ψ i be the estimated and population influence functions. We first show that E n ∥ R n κ ^ Z ∥ H X j = o P ( 1 ) \mathbb{E}_n\left\|R_n\hat{\kappa}Z\right\|_{\mathcal{H}_{X_j}} = o_P(1) E n ∥ R n κ ^ Z ∥ H X j = o P ( 1 ) . From the boundedness of kernels, it’s easy to derive that ∥ R n ∥ \left\|R_n\right\| ∥ R n ∥ is bounded uniformly over n n n . Also by Lemma 1 with θ = 1 \theta= 1 θ = 1 ,
∥ R n Σ Z Z ∥ o p = O P ( n − 1 / 2 + ϵ n ) = o P ( 1 ) . \left\|R_n\Sigma_{ZZ}\right\|_{\mathrm{op}} = O_P\left(n^{-1/2}+\epsilon_n\right)=o_P(1). ∥ R n Σ Z Z ∥ op = O P ( n − 1/2 + ϵ n ) = o P ( 1 ) .
Let { λ ℓ } \{\lambda_\ell\} { λ ℓ } and { e ℓ } \{e_\ell\} { e ℓ } be positive eigenvalues and corresponding orthonormal eigenvectors of Σ Z Z \Sigma_{ZZ} Σ Z Z . Since Σ Z Z \Sigma_{ZZ} Σ Z Z is a trace-class operator, ∑ ℓ λ ℓ < ∞ \sum_\ell\lambda_\ell< \infty ∑ ℓ λ ℓ < ∞ . Then, for some constant C > 0 C > 0 C > 0 , we have
∥ R n Σ Z Z 1 / 2 ∥ H S 2 = ∑ ℓ λ ℓ ∥ R n e ℓ ∥ H X j 2 = ∑ ℓ = 1 m λ ℓ − 1 ∥ R n Σ Z Z e ℓ ∥ H X j 2 + ∑ ℓ > m λ ℓ ∥ R n e ℓ ∥ H X j 2 ≤ ∥ R n Σ Z Z ∥ o p 2 ∑ ℓ = 1 m λ ℓ − 1 + C ∑ ℓ > m λ ℓ . \begin{aligned}
\left\|R_n\Sigma_{ZZ}^{1/2}\right\|_{\mathrm{HS}}^2
= \sum_\ell\lambda_\ell\left\|R_ne_\ell\right\|_{\mathcal{H}_{X_j}}^2 \\
&= \sum_{\ell=1}^{m}\lambda_\ell^{-1}\left\|R_n\Sigma_{ZZ}e_\ell\right\|_{\mathcal{H}_{X_j}}^2
+\sum_{\ell>m}\lambda_\ell\left\|R_ne_\ell\right\|_{\mathcal{H}_{X_j}}^2 \\
&\leq\left\|R_n\Sigma_{ZZ}\right\|_{\mathrm{op}}^2\sum_{\ell=1}^{m}\lambda_\ell^{-1}
+C\sum_{\ell>m}\lambda_\ell.
\end{aligned} R n Σ Z Z 1/2 HS 2 = ℓ ∑ λ ℓ ∥ R n e ℓ ∥ H X j 2 = ℓ = 1 ∑ m λ ℓ − 1 ∥ R n Σ Z Z e ℓ ∥ H X j 2 + ℓ > m ∑ λ ℓ ∥ R n e ℓ ∥ H X j 2 ≤ ∥ R n Σ Z Z ∥ op 2 ℓ = 1 ∑ m λ ℓ − 1 + C ℓ > m ∑ λ ℓ .
Note that the above argument holds for all positive integers m m m . Thus, given any η > 0 \eta> 0 η > 0 , we fix a sufficiently large m m m such that C ∑ ℓ > m λ ℓ < η / 2 C\sum_{\ell>m}\lambda_\ell< \eta/2 C ∑ ℓ > m λ ℓ < η /2 , which implies
Pr ( ∥ R n Σ Z Z 1 / 2 ∥ H S 2 > η ) ≤ Pr ( ∥ R n Σ Z Z ∥ o p 2 ∑ ℓ = 1 m λ ℓ − 1 > η 2 ) → 0 , \Pr\left(\left\|R_n\Sigma_{ZZ}^{1/2}\right\|_{\mathrm{HS}}^2>\eta\right)
\leq
\Pr\left(\left\|R_n\Sigma_{ZZ}\right\|_{\mathrm{op}}^2
\sum_{\ell=1}^{m}\lambda_\ell^{-1}>\frac{\eta}{2}\right)
\to0, Pr ( R n Σ Z Z 1/2 HS 2 > η ) ≤ Pr ( ∥ R n Σ Z Z ∥ op 2 ℓ = 1 ∑ m λ ℓ − 1 > 2 η ) → 0 ,
where the limit holds because ∥ R n Σ Z Z ∥ o p = o P ( 1 ) \left\|R_n\Sigma_{ZZ}\right\|_{\mathrm{op}}=o_P(1) ∥ R n Σ Z Z ∥ op = o P ( 1 ) . Thus, ∥ R n Σ Z Z 1 / 2 ∥ H S 2 = o P ( 1 ) \left\|R_n\Sigma_{ZZ}^{1/2}\right\|_{\mathrm{HS}}^2=o_P(1) R n Σ Z Z 1/2 HS 2 = o P ( 1 ) . By the Banach-space strong law, we get
∥ Σ ^ Z Z − Σ Z Z ∥ t r ≤ ∥ 1 n ∑ i = 1 n ( k Z ( ⋅ , Z i ) − μ Z ) ⊗ ( k Z ( ⋅ , Z i ) − μ Z ) − Σ Z Z ∥ t r + ∥ 1 n ∑ i = 1 n k Z ( ⋅ , Z i ) − μ Z ∥ H Z 2 = o P ( 1 ) . \begin{aligned}
\left\|\hat{\Sigma}_{ZZ}-\Sigma_{ZZ}\right\|_{\mathrm{tr}} \\
&\le\left\lVert\frac{1}{n}\sum_{i=1}^{n}\left(k_Z(\cdot,Z_i)-\mu_Z\right)\otimes\left(k_Z(\cdot,Z_i)-\mu_Z\right)-\Sigma_{ZZ}\right\rVert_{\mathrm{tr}}+\left\lVert\frac{1}{n}\sum_{i=1}^{n}k_Z(\cdot,Z_i)-\mu_Z\right\rVert_{\mathcal{H}_Z}^{2} \\
&=o_P(1).
\end{aligned} Σ ^ Z Z − Σ Z Z tr ≤ n 1 i = 1 ∑ n ( k Z ( ⋅ , Z i ) − μ Z ) ⊗ ( k Z ( ⋅ , Z i ) − μ Z ) − Σ Z Z tr + n 1 i = 1 ∑ n k Z ( ⋅ , Z i ) − μ Z H Z 2 = o P ( 1 ) .
Then
E n ∥ R n κ ^ Z ∥ H X j 2 = tr ( R n Σ ^ Z Z R n ∗ ) = tr ( R n ( Σ ^ Z Z − Σ Z Z ) R n ∗ ) + tr ( R n Σ Z Z R n ∗ ) ≤ ∥ R n ( Σ ^ Z Z − Σ Z Z ) R n ∗ ∥ t r + ∥ R n Σ ^ Z Z 1 2 ∥ H S 2 ≤ ∥ R n ∥ o p 2 ∥ Σ ^ Z Z − Σ Z Z ∥ t r + ∥ R n Σ ^ Z Z 1 2 ∥ H S 2 = o P ( 1 ) . (77) \begin{aligned}
\mathbb{E}_n\left\lVert R_n\hat{\kappa}_Z\right\rVert_{\mathcal{H}_{X_j}}^2
&=\operatorname{tr}\left(R_n\hat{\Sigma}_{ZZ}R_n^*\right)\\
&=\operatorname{tr}\left(R_n(\hat{\Sigma}_{ZZ}-\Sigma_{ZZ})R_n^*\right)+\operatorname{tr}\left(R_n\Sigma_{ZZ}R_n^*\right)\\
&\le\left\lVert R_n(\hat{\Sigma}_{ZZ}-\Sigma_{ZZ})R_n^*\right\rVert_{\mathrm{tr}}+\left\lVert R_n\hat{\Sigma}_{ZZ}^{\frac{1}{2}}\right\rVert_{\mathrm{HS}}^2\\
&\le\left\lVert R_n\right\rVert_{\mathrm{op}}^2\left\lVert\hat{\Sigma}_{ZZ}-\Sigma_{ZZ}\right\rVert_{\mathrm{tr}}+\left\lVert R_n\hat{\Sigma}_{ZZ}^{\frac{1}{2}}\right\rVert_{\mathrm{HS}}^2\\
&=o_P(1). \tag*{(77)}
\end{aligned} E n ∥ R n κ ^ Z ∥ H X j 2 = tr ( R n Σ ^ Z Z R n ∗ ) = tr ( R n ( Σ ^ Z Z − Σ Z Z ) R n ∗ ) + tr ( R n Σ Z Z R n ∗ ) ≤ R n ( Σ ^ Z Z − Σ Z Z ) R n ∗ tr + R n Σ ^ Z Z 2 1 HS 2 ≤ ∥ R n ∥ op 2 Σ ^ Z Z − Σ Z Z tr + R n Σ ^ Z Z 2 1 HS 2 = o P ( 1 ) . (77)
The definition of residuals gives the decomposition
ξ ^ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) − ξ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) = − ( μ ^ X j , i − μ X j ) − ( B ^ X ¨ j ∣ Z − B X ¨ j ∣ Z ) κ ^ Z ( ⋅ , Z i ) + B X ¨ j ∣ Z ( μ ^ Z − μ Z ) . \hat{\xi}_{X_j|Z}(\cdot,\ddot{X}_{j,i};Z_i)-\xi_{X_j|Z}(\cdot,\ddot{X}_{j,i};Z_i)=-(\hat{\mu}_{X_j,i}-\mu_{X_j})-(\hat{B}_{\ddot{X}_j|Z}-B_{\ddot{X}_j|Z})\hat{\kappa}_Z(\cdot,Z_i)+B_{\ddot{X}_j|Z}(\hat{\mu}_Z-\mu_Z). ξ ^ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) − ξ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) = − ( μ ^ X j , i − μ X j ) − ( B ^ X ¨ j ∣ Z − B X ¨ j ∣ Z ) κ ^ Z ( ⋅ , Z i ) + B X ¨ j ∣ Z ( μ ^ Z − μ Z ) .
Applying Cauchy–Schwartz inequality, we have
E n ∥ ξ ^ X j ∣ Z − ξ X j ∣ Z ∥ H X j 2 ≤ 4 ∥ μ ^ X j − μ X j ∥ H X j 2 + 4 ∥ B ^ X ¨ j ∣ Z − B X ¨ j ∣ Z − R n ∥ o p 2 E n ∥ κ ^ Z ∥ H Z 2 + 4 E n ∥ R n κ ^ Z ∥ H X j 2 + 4 ∥ B X ¨ j ∣ Z ∥ o p 2 ∥ μ ^ Z − μ Z ∥ H Z 2 = o P ( 1 ) . (78) \begin{aligned}
\mathbb{E}_n\left\lVert\hat{\xi}_{X_j|Z}-\xi_{X_j|Z}\right\rVert_{\mathcal{H}_{X_j}}^2
&\le4\left\lVert\hat{\mu}_{X_j}-\mu_{X_j}\right\rVert_{\mathcal{H}_{X_j}}^2+4\left\lVert\hat{B}_{\ddot{X}_j|Z}-B_{\ddot{X}_j|Z}-R_n\right\rVert_{\mathrm{op}}^2\mathbb{E}_n\left\lVert\hat{\kappa}_Z\right\rVert_{\mathcal{H}_Z}^2\\
&\quad+4\mathbb{E}_n\left\lVert R_n\hat{\kappa}_Z\right\rVert_{\mathcal{H}_{X_j}}^2+4\left\lVert B_{\ddot{X}_j|Z}\right\rVert_{\mathrm{op}}^2\left\lVert\hat{\mu}_Z-\mu_Z\right\rVert_{\mathcal{H}_Z}^2\\
&=o_P(1). \tag*{(78)}
\end{aligned} E n ξ ^ X j ∣ Z − ξ X j ∣ Z H X j 2 ≤ 4 μ ^ X j − μ X j H X j 2 + 4 B ^ X ¨ j ∣ Z − B X ¨ j ∣ Z − R n op 2 E n ∥ κ ^ Z ∥ H Z 2 + 4 E n ∥ R n κ ^ Z ∥ H X j 2 + 4 B X ¨ j ∣ Z op 2 ∥ μ ^ Z − μ Z ∥ H Z 2 = o P ( 1 ) . (78)
Similarly, the residual decomposition for X j + ∣ Z X_{j+}\mid Z X j + ∣ Z gives
E n ∥ ξ ^ X j + ∣ Z − ξ X j + ∣ Z ∥ H X j + 2 ≤ 3 ∥ μ ^ X j + − μ X j + ∥ H X j + 2 + 3 ∥ B ^ X j + ∣ Z − B X j + ∣ Z ∥ o p 2 E n ∥ κ ^ Z ∥ H Z 2 + 3 ∥ B X j + ∣ Z ∥ o p 2 ∥ μ ^ Z − μ Z ∥ H Z 2 = o P ( 1 ) . (79) \begin{aligned}
\mathbb{E}_n\left\lVert\hat{\xi}_{X_{j+}|Z}-\xi_{X_{j+}|Z}\right\rVert_{\mathcal{H}_{X_{j+}}}^2
&\le3\left\lVert\hat{\mu}_{X_{j+}}-\mu_{X_{j+}}\right\rVert_{\mathcal{H}_{X_{j+}}}^2+3\left\lVert\hat{B}_{X_{j+}|Z}-B_{X_{j+}|Z}\right\rVert_{\mathrm{op}}^2\mathbb{E}_n\left\lVert\hat{\kappa}_Z\right\rVert_{\mathcal{H}_Z}^2\\
&\quad+3\left\lVert B_{X_{j+}|Z}\right\rVert_{\mathrm{op}}^2\left\lVert\hat{\mu}_Z-\mu_Z\right\rVert_{\mathcal{H}_Z}^2\\
&=o_P(1). \tag*{(79)}
\end{aligned} E n ξ ^ X j + ∣ Z − ξ X j + ∣ Z H X j + 2 ≤ 3 μ ^ X j + − μ X j + H X j + 2 + 3 B ^ X j + ∣ Z − B X j + ∣ Z op 2 E n ∥ κ ^ Z ∥ H Z 2 + 3 B X j + ∣ Z op 2 ∥ μ ^ Z − μ Z ∥ H Z 2 = o P ( 1 ) . (79)
We also have universal boundedness of the residual terms, both the estimated and the population ones, which is an immediate result from the boundedness of kernels and regression operators.
max 1 ≤ i ≤ n ∥ ξ ^ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) ∥ H X j = O P ( 1 ) , max 1 ≤ i ≤ n ∥ ξ ^ X j + ∣ Z ( ⋅ , X j + , i ; Z i ) ∥ H X j + = O P ( 1 ) . (80) \max_{1\le i\le n}\left\lVert\hat{\xi}_{X_j|Z}(\cdot,\ddot{X}_{j,i};Z_i)\right\rVert_{\mathcal{H}_{X_j}}=O_P(1),\qquad
\max_{1\le i\le n}\left\lVert\hat{\xi}_{X_{j+}|Z}(\cdot,X_{j+,i};Z_i)\right\rVert_{\mathcal{H}_{X_{j+}}}=O_P(1). \tag*{(80)} 1 ≤ i ≤ n max ξ ^ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) H X j = O P ( 1 ) , 1 ≤ i ≤ n max ξ ^ X j + ∣ Z ( ⋅ , X j + , i ; Z i ) H X j + = O P ( 1 ) . (80)
Combining (77)–(80) we get
E n ∥ ξ ^ X j ∣ Z ⊗ ξ ^ X j + ∣ Z − ξ X j ∣ Z ⊗ ξ X j + ∣ Z ∥ H S 2 \mathbb{E}_n\left\lVert\hat{\xi}_{X_j|Z}\otimes\hat{\xi}_{X_{j+}|Z}-\xi_{X_j|Z}\otimes\xi_{X_{j+}|Z}\right\rVert_{\mathrm{HS}}^2 E n ξ ^ X j ∣ Z ⊗ ξ ^ X j + ∣ Z − ξ X j ∣ Z ⊗ ξ X j + ∣ Z HS 2
≤ 2 max 1 ≤ i ≤ n ∥ ξ ¨ X j + ( ⋅ , X j + , i ; Z i ) ∥ H X j + 2 E n ∥ ξ ^ X j + ∣ Z − ξ X j + ∣ Z ∥ H X j + 2 + 2 max 1 ≤ i ≤ n ∥ ξ ¨ X j ( ⋅ , X ¨ j , i ; Z i ) ∥ H X j 2 E n ∥ ξ ^ X j + ∣ Z − ξ X j + ∣ Z ∥ H X j + 2 = o P ( 1 ) . \begin{aligned}
\leq2\max_{1\leq i\leq n}\left\|\ddot{\xi}_{X_{j+}}(\cdot,X_{j+,i};Z_i)\right\|_{\mathcal{H}_{X_{j+}}}^{2}\mathbb{E}_{n}\left\|\widehat{\xi}_{X_{j+}|Z}-\xi_{X_{j+}|Z}\right\|_{\mathcal{H}_{X_{j+}}}^{2} \\
&+2\max_{1\leq i\leq n}\left\|\ddot{\xi}_{X_j}(\cdot,\ddot{X}_{j,i};Z_i)\right\|_{\mathcal{H}_{X_j}}^{2}\mathbb{E}_{n}\left\|\widehat{\xi}_{X_{j+}|Z}-\xi_{X_{j+}|Z}\right\|_{\mathcal{H}_{X_{j+}}}^{2}
=o_P(1).
\end{aligned} ≤ 2 1 ≤ i ≤ n max ξ ¨ X j + ( ⋅ , X j + , i ; Z i ) H X j + 2 E n ξ X j + ∣ Z − ξ X j + ∣ Z H X j + 2 + 2 1 ≤ i ≤ n max ξ ¨ X j ( ⋅ , X ¨ j , i ; Z i ) H X j 2 E n ξ X j + ∣ Z − ξ X j + ∣ Z H X j + 2 = o P ( 1 ) .
Combining the results above yields
E n ∥ ψ ^ j − ψ j ∥ H S 2 ≤ 2 E n ∥ ξ ^ X j + ∣ Z ⊗ ξ ^ X j + ∣ Z − ξ ~ X j ∣ Z ⊗ ξ ~ X j + ∣ Z ∥ H S 2 + 2 ∥ Σ ^ X j + X j + ∣ Z − Σ X j X j + ∣ Z ∥ H S 2 = o P ( 1 ) . \mathbb{E}_{n}\left\|\widehat{\psi}_j-\psi_j\right\|_{\mathrm{HS}}^{2}\leq2\mathbb{E}_{n}\left\|\widehat{\xi}_{X_{j+}|Z}\otimes\widehat{\xi}_{X_{j+}|Z}-\widetilde{\xi}_{X_j|Z}\otimes\widetilde{\xi}_{X_{j+}|Z}\right\|_{\mathrm{HS}}^{2}+2\left\|\widehat{\Sigma}_{X_{j+}X_{j+}|Z}-\Sigma_{X_jX_{j+}|Z}\right\|_{\mathrm{HS}}^{2}
=o_P(1). E n ψ j − ψ j HS 2 ≤ 2 E n ξ X j + ∣ Z ⊗ ξ X j + ∣ Z − ξ X j ∣ Z ⊗ ξ X j + ∣ Z HS 2 + 2 Σ X j + X j + ∣ Z − Σ X j X j + ∣ Z HS 2 = o P ( 1 ) .
Thus
E n ∥ ψ ^ − ψ ∥ H S 2 = ∑ j = 1 d − 1 E n ∥ ψ ^ j − ψ j ∥ H S 2 = o P ( 1 ) . \mathbb{E}_{n}\left\|\widehat{\psi}-\psi\right\|_{\mathrm{HS}}^{2}=\sum_{j=1}^{d-1}\mathbb{E}_{n}\left\|\widehat{\psi}_j-\psi_j\right\|_{\mathrm{HS}}^{2}=o_P(1). E n ψ − ψ HS 2 = j = 1 ∑ d − 1 E n ψ j − ψ j HS 2 = o P ( 1 ) .
We next establish consistency of the covariance estimator. By Cauchy–Schwarz inequality,
∥ Γ ^ S ∣ Z − E n [ ψ ⊗ ψ ] ∥ t r = ∥ 1 n ∑ i = 1 n ( ψ ^ i ⊗ ψ ^ i − ψ i ⊗ ψ i ) ∥ t r ≤ 1 n ∑ i = 1 n ∥ ψ ^ i ⊗ ψ ^ i − ψ i ⊗ ψ i ∥ t r ≤ 1 n ∑ i = 1 n ( 2 ∥ ψ ^ i − ψ i ∥ H S ∥ ψ i ∥ H S + ∥ ψ ^ i − ψ i ∥ H S 2 ) \begin{aligned}
\left\|\widehat{\Gamma}_{S|Z}-\mathbb{E}_{n}[\psi\otimes\psi]\right\|_{\mathrm{tr}}=\left\|\frac{1}{n}\sum_{i=1}^{n}\left(\widehat{\psi}_i\otimes\widehat{\psi}_i-\psi_i\otimes\psi_i\right)\right\|_{\mathrm{tr}} \\
&\leq\frac{1}{n}\sum_{i=1}^{n}\left\|\widehat{\psi}_i\otimes\widehat{\psi}_i-\psi_i\otimes\psi_i\right\|_{\mathrm{tr}} \\
&\leq\frac{1}{n}\sum_{i=1}^{n}\left(2\left\|\widehat{\psi}_i-\psi_i\right\|_{\mathrm{HS}}\left\|\psi_i\right\|_{\mathrm{HS}}+\left\|\widehat{\psi}_i-\psi_i\right\|_{\mathrm{HS}}^{2}\right)
\end{aligned} Γ S ∣ Z − E n [ ψ ⊗ ψ ] tr = n 1 i = 1 ∑ n ( ψ i ⊗ ψ i − ψ i ⊗ ψ i ) tr ≤ n 1 i = 1 ∑ n ψ i ⊗ ψ i − ψ i ⊗ ψ i tr ≤ n 1 i = 1 ∑ n ( 2 ψ i − ψ i HS ∥ ψ i ∥ HS + ψ i − ψ i HS 2 )
≤ 2 ( E n ∥ ψ ^ − ψ ∥ H S 2 ) 1 / 2 ( E n ∥ ψ ∥ H S 2 ) 1 / 2 + E n ∥ ψ ^ − ψ ∥ H S 2 = o P ( 1 ) . \begin{aligned}
\leq2\left(\mathbb{E}_{n}\left\|\widehat{\psi}-\psi\right\|_{\mathrm{HS}}^{2}\right)^{1/2}\left(\mathbb{E}_{n}\left\|\psi\right\|_{\mathrm{HS}}^{2}\right)^{1/2}+\mathbb{E}_{n}\left\|\widehat{\psi}-\psi\right\|_{\mathrm{HS}}^{2} \\
&=o_P(1).
\end{aligned} ≤ 2 ( E n ψ − ψ HS 2 ) 1/2 ( E n ∥ ψ ∥ HS 2 ) 1/2 + E n ψ − ψ HS 2 = o P ( 1 ) .
Combining with the fact that Γ S ∣ Z \Gamma_{S|Z} Γ S ∣ Z is a trace-class operator, the law of large numbers in the separable Banach space of trace-class operators gives
∥ E n [ ψ ⊗ ψ ] − Γ S ∣ Z ∥ t r = o P ( 1 ) . \left\|\mathbb{E}_{n}[\psi\otimes\psi]-\Gamma_{S|Z}\right\|_{\mathrm{tr}}=o_P(1). E n [ ψ ⊗ ψ ] − Γ S ∣ Z tr = o P ( 1 ) .
Thus
∥ Γ ^ S ∣ Z − Γ S ∣ Z ∥ t r ≤ ∥ Γ ^ S ∣ Z − E n [ ψ ⊗ ψ ] ∥ t r + ∥ E n [ ψ ⊗ ψ ] − Γ S ∣ Z ∥ t r = o P ( 1 ) , \left\|\widehat{\Gamma}_{S|Z}-\Gamma_{S|Z}\right\|_{\mathrm{tr}}\leq\left\|\widehat{\Gamma}_{S|Z}-\mathbb{E}_{n}[\psi\otimes\psi]\right\|_{\mathrm{tr}}+\left\|\mathbb{E}_{n}[\psi\otimes\psi]-\Gamma_{S|Z}\right\|_{\mathrm{tr}}=o_P(1), Γ S ∣ Z − Γ S ∣ Z tr ≤ Γ S ∣ Z − E n [ ψ ⊗ ψ ] tr + E n [ ψ ⊗ ψ ] − Γ S ∣ Z tr = o P ( 1 ) ,
which completes the proof. □
Proof of Proposition 3. We first derive the coordinate mapping for the numerical influence function.
B X J [ B ^ X J ] B Z = K ^ Z ( K ^ Z + n ϵ n I ) − 1 \mathcal{B}_{X_J}\left[\widehat{B}_{X_J}\right]_{\mathcal{B}_Z}=\widehat{K}_Z\left(\widehat{K}_Z+n\epsilon_n I\right)^{-1} B X J [ B X J ] B Z = K Z ( K Z + n ϵ n I ) − 1
[ κ ^ X J ( ⋅ , X J , i ) ] B X J = e i [ κ ^ Z ( ⋅ , Z i ) ] B Z = e i \left[\widehat{\kappa}_{X_J}(\cdot,X_{J,i})\right]_{\mathcal{B}_{X_J}}=e_i
\qquad
\left[\widehat{\kappa}_Z(\cdot,Z_i)\right]_{\mathcal{B}_Z}=e_i [ κ X J ( ⋅ , X J , i ) ] B X J = e i [ κ Z ( ⋅ , Z i ) ] B Z = e i
[ ξ ^ X j ∣ Z ( ⋅ , X j , i ; Z i ) ] B X j = [ κ ^ X j ( ⋅ , X j , i ) − B ^ X j K ^ Z ( ⋅ , Z i ) ] B X j = ( I − K ^ Z ( K ^ Z + n ϵ n I ) − 1 ) e i = R Z e i \left[\hat{\xi}_{X_j\mid Z}(\cdot,X_{j,i};Z_i)\right]_{\mathcal{B}_{X_j}}=\left[\hat{\kappa}_{X_j}(\cdot,X_{j,i})-\hat{B}_{X_j}\hat{K}_Z(\cdot,Z_i)\right]_{\mathcal{B}_{X_j}}=\left(I-\hat{K}_Z\left(\hat{K}_Z+n\epsilon_n I\right)^{-1}\right)e_i=R_Ze_i [ ξ ^ X j ∣ Z ( ⋅ , X j , i ; Z i ) ] B X j = [ κ ^ X j ( ⋅ , X j , i ) − B ^ X j K ^ Z ( ⋅ , Z i ) ] B X j = ( I − K ^ Z ( K ^ Z + n ϵ n I ) − 1 ) e i = R Z e i
For the Gamma approximation under the null hypothesis, we use Σ ^ X ~ j X ~ j + ∣ Z = 0 \widehat{\Sigma}_{\tilde{X}_j\tilde{X}_{j+}\mid Z}=0 Σ X ~ j X ~ j + ∣ Z = 0 when constructing the empirical influence function. Accordingly, throughout this proof, ψ ^ j , i \hat{\psi}_{j,i} ψ ^ j , i denotes the tensor product of the estimated residual feature maps, without subtracting Σ ^ X ~ j X ~ j + ∣ Z \widehat{\Sigma}_{\tilde{X}_j\tilde{X}_{j+}\mid Z} Σ X ~ j X ~ j + ∣ Z . Its coordinate representation is
[ ψ ^ ( X ¨ j , i , X j + ; i ; Z i ) ] B X j ⊗ B X j + = [ ξ ^ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) ⊗ ξ ^ X j + ∣ Z ( ⋅ , X j + ; i ; Z i ) ] B X j ⊗ B X j + = [ ξ ^ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) ] B X j [ ξ ^ X j + ∣ Z ( ⋅ , X j + ; i ; Z i ) ] B X j + T K ^ X j + = ( R Z e i ) ( R Z e i ) T K ^ X j + \begin{aligned}
\left[\hat{\psi}(\ddot{X}_{j,i},X_{j+;i};Z_i)\right]_{\mathcal{B}_{X_j}\otimes\mathcal{B}_{X_{j+}}}
=
\left[\hat{\xi}_{X_j\mid Z}(\cdot,\ddot{X}_{j,i};Z_i)\otimes\hat{\xi}_{X_{j+}\mid Z}(\cdot,X_{j+;i};Z_i)\right]_{\mathcal{B}_{X_j}\otimes\mathcal{B}_{X_{j+}}} \\
&=
\left[\hat{\xi}_{X_j\mid Z}(\cdot,\ddot{X}_{j,i};Z_i)\right]_{\mathcal{B}_{X_j}}
\left[\hat{\xi}_{X_{j+}\mid Z}(\cdot,X_{j+;i};Z_i)\right]_{\mathcal{B}_{X_{j+}}}^{\mathsf{T}}\hat{K}_{X_{j+}} \\
&=(R_Ze_i)(R_Ze_i)^{\mathsf{T}}\hat{K}_{X_{j+}}
\end{aligned} [ ψ ^ ( X ¨ j , i , X j + ; i ; Z i ) ] B X j ⊗ B X j + = [ ξ ^ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) ⊗ ξ ^ X j + ∣ Z ( ⋅ , X j + ; i ; Z i ) ] B X j ⊗ B X j + = [ ξ ^ X j ∣ Z ( ⋅ , X ¨ j , i ; Z i ) ] B X j [ ξ ^ X j + ∣ Z ( ⋅ , X j + ; i ; Z i ) ] B X j + T K ^ X j + = ( R Z e i ) ( R Z e i ) T K ^ X j +
Consider the Gram matrix of the tensor product space H X j ⊗ H X j + \mathcal{H}_{X_j}\otimes\mathcal{H}_{X_{j+}} H X j ⊗ H X j + , the spanning system is B X j X j + = { κ ^ X j X j + ( ⋅ , X ¨ j , i , ⋆ , X j + ; ℓ ) = κ ^ X j ( ⋅ , X ¨ j , i ) κ ^ X j + ( ⋆ , X j + ; ℓ ) , i , ℓ = 1 , … , n } = B X j ⊗ B X j + \mathcal{B}_{X_jX_{j+}}=\{\hat{\kappa}_{X_jX_{j+}}(\cdot,\ddot{X}_{j,i},\star,X_{j+;\ell})=\hat{\kappa}_{X_j}(\cdot,\ddot{X}_{j,i})\hat{\kappa}_{X_{j+}}(\star,X_{j+;\ell}),i,\ell=1,\ldots,n\}=\mathcal{B}_{X_j}\otimes\mathcal{B}_{X_{j+}} B X j X j + = { κ ^ X j X j + ( ⋅ , X ¨ j , i , ⋆ , X j + ; ℓ ) = κ ^ X j ( ⋅ , X ¨ j , i ) κ ^ X j + ( ⋆ , X j + ; ℓ ) , i , ℓ = 1 , … , n } = B X j ⊗ B X j + . If we reconsider the influence function under the tensor product space, we have
[ ψ ^ j , i ] B X j X j + = [ ψ ^ ( X ¨ j , i , X j + ; i ; Z i ) ] B X j X j + = R Z e i ⊗ R Z e i \left[\hat{\psi}_{j,i}\right]_{\mathcal{B}_{X_jX_{j+}}}
=
\left[\hat{\psi}(\ddot{X}_{j,i},X_{j+;i};Z_i)\right]_{\mathcal{B}_{X_jX_{j+}}}
=R_Ze_i\otimes R_Ze_i [ ψ ^ j , i ] B X j X j + = [ ψ ^ ( X ¨ j , i , X j + ; i ; Z i ) ] B X j X j + = R Z e i ⊗ R Z e i
Thus
[ ψ ^ j , i ⊗ ψ ^ k , i ] B X j X j + = [ ψ ^ j , i ] B X j X j + [ ψ ^ k , i ] B X k X k + T K ^ X k X k + = ( R Z e i ⊗ R Z e i ) ( R Z e i ⊗ R Z e i ) T K ^ X k X k + \begin{aligned}
\left[\hat{\psi}_{j,i}\otimes\hat{\psi}_{k,i}\right]_{\mathcal{B}_{X_jX_{j+}}}
=
\left[\hat{\psi}_{j,i}\right]_{\mathcal{B}_{X_jX_{j+}}}
\left[\hat{\psi}_{k,i}\right]_{\mathcal{B}_{X_kX_{k+}}}^{\mathsf{T}}\hat{K}_{X_kX_{k+}} \\
&=(R_Ze_i\otimes R_Ze_i)(R_Ze_i\otimes R_Ze_i)^{\mathsf{T}}\hat{K}_{X_kX_{k+}}
\end{aligned} [ ψ ^ j , i ⊗ ψ ^ k , i ] B X j X j + = [ ψ ^ j , i ] B X j X j + [ ψ ^ k , i ] B X k X k + T K ^ X k X k + = ( R Z e i ⊗ R Z e i ) ( R Z e i ⊗ R Z e i ) T K ^ X k X k +
We next compute the trace of Γ ^ S ∣ Z \hat{\Gamma}_{S\mid Z} Γ ^ S ∣ Z . By its coordinate representation,
∑ j = 1 d − 1 tr ( ( Γ ^ S ∣ Z ) ( j , j ) ) = ∑ j = 1 d − 1 tr ( [ 1 n ∑ i = 1 n ψ ^ j , i ⊗ ψ ^ j , i ] B X j X j + ) = ∑ j = 1 d − 1 tr ( ( 1 n ∑ i = 1 n ( R Z e i ⊗ R Z e i ) ( R Z e i ⊗ R Z e i ) T ) ( K ^ X j X j + ) ) . \begin{aligned}
\sum_{j=1}^{d-1}\operatorname{tr}\left((\hat{\Gamma}_{S\mid Z})_{(j,j)}\right)
=
\sum_{j=1}^{d-1}\operatorname{tr}\left(
\left[
\frac{1}{n}\sum_{i=1}^{n}\hat{\psi}_{j,i}\otimes\hat{\psi}_{j,i}
\right]_{\mathcal{B}_{X_jX_{j+}}}
\right) \\
&=
\sum_{j=1}^{d-1}\operatorname{tr}\left(
\left(
\frac{1}{n}\sum_{i=1}^{n}(R_Ze_i\otimes R_Ze_i)(R_Ze_i\otimes R_Ze_i)^{\mathsf{T}}
\right)(\hat{K}_{X_jX_{j+}})
\right).
\end{aligned} j = 1 ∑ d − 1 tr ( ( Γ ^ S ∣ Z ) ( j , j ) ) = j = 1 ∑ d − 1 tr [ n 1 i = 1 ∑ n ψ ^ j , i ⊗ ψ ^ j , i ] B X j X j + = j = 1 ∑ d − 1 tr ( ( n 1 i = 1 ∑ n ( R Z e i ⊗ R Z e i ) ( R Z e i ⊗ R Z e i ) T ) ( K ^ X j X j + ) ) .
Using the Kronecker product identities and the symmetry of R Z R_Z R Z , it becomes
∑ j = 1 d − 1 tr ( ( 1 n ∑ i = 1 n ( R Z e i ) ( R Z e i ) T ⊗ ( ( R Z e i ) ( R Z e i ) T ) ) ( K ^ X j ⊗ K ^ X j + ) ) = ∑ j = 1 d − 1 1 n ∑ i = 1 n tr ( ( ( R Z e i ) ( R Z e i ) T K ^ X j ) ⊗ ( ( R Z e i ) ( R Z e i ) T K ^ X j + ) ) = ∑ j = 1 d − 1 1 n ∑ i = 1 n tr ( ( R Z e i ) ( R Z e i ) T K ^ X j ) tr ( ( R Z e i ) ( R Z e i ) T K ^ X j + ) = ∑ j = 1 d − 1 1 n ∑ i = 1 n ( e i T R Z K ^ X j R Z e i ) ( e i T R Z K ^ X j + 1 R Z e i ) . \begin{aligned}
\sum_{j=1}^{d-1}\operatorname{tr}\left(
\left(
\frac{1}{n}\sum_{i=1}^{n}(R_Ze_i)(R_Ze_i)^{\mathsf{T}}\otimes((R_Ze_i)(R_Ze_i)^{\mathsf{T}})
\right)(\hat{K}_{X_j}\otimes\hat{K}_{X_{j+}})
\right) \\
&=
\sum_{j=1}^{d-1}\frac{1}{n}\sum_{i=1}^{n}\operatorname{tr}\left(
\left((R_Ze_i)(R_Ze_i)^{\mathsf{T}}\hat{K}_{X_j}\right)\otimes
\left((R_Ze_i)(R_Ze_i)^{\mathsf{T}}\hat{K}_{X_{j+}}\right)
\right) \\
&=
\sum_{j=1}^{d-1}\frac{1}{n}\sum_{i=1}^{n}
\operatorname{tr}\left((R_Ze_i)(R_Ze_i)^{\mathsf{T}}\hat{K}_{X_j}\right)
\operatorname{tr}\left((R_Ze_i)(R_Ze_i)^{\mathsf{T}}\hat{K}_{X_{j+}}\right) \\
&= \sum_{j=1}^{d-1} \frac{1}{n} \sum_{i=1}^{n} \left(e_i^{\mathsf{T}} R_Z \widehat{K}_{X_j} R_Z e_i\right)\left(e_i^{\mathsf{T}} R_Z \widehat{K}_{X_{j+1}} R_Z e_i\right).
\end{aligned} j = 1 ∑ d − 1 tr ( ( n 1 i = 1 ∑ n ( R Z e i ) ( R Z e i ) T ⊗ (( R Z e i ) ( R Z e i ) T ) ) ( K ^ X j ⊗ K ^ X j + ) ) = j = 1 ∑ d − 1 n 1 i = 1 ∑ n tr ( ( ( R Z e i ) ( R Z e i ) T K ^ X j ) ⊗ ( ( R Z e i ) ( R Z e i ) T K ^ X j + ) ) = j = 1 ∑ d − 1 n 1 i = 1 ∑ n tr ( ( R Z e i ) ( R Z e i ) T K ^ X j ) tr ( ( R Z e i ) ( R Z e i ) T K ^ X j + ) = j = 1 ∑ d − 1 n 1 i = 1 ∑ n ( e i T R Z K X j R Z e i ) ( e i T R Z K X j + 1 R Z e i ) .
The two quadratic forms are the i i i th diagonal entries of K ^ X j ∣ Z \widehat{K}_{X_j\mid Z} K X j ∣ Z and K ^ X j + 1 ∣ Z \widehat{K}_{X_{j+1}\mid Z} K X j + 1 ∣ Z , respectively. Therefore,
∑ j = 1 d − 1 tr ( Γ ^ S ∣ Z ) ( j , j ) = ∑ j = 1 d − 1 1 n tr ( K ^ X j ∣ Z ∘ K ^ X j + 1 ∣ Z ) . \sum_{j=1}^{d-1} \operatorname{tr}\left(\widehat{\Gamma}_{S\mid Z}\right)_{(j,j)} = \sum_{j=1}^{d-1} \frac{1}{n}\operatorname{tr}\left(\widehat{K}_{X_j\mid Z} \circ\widehat{K}_{X_{j+1}\mid Z}\right). j = 1 ∑ d − 1 tr ( Γ S ∣ Z ) ( j , j ) = j = 1 ∑ d − 1 n 1 tr ( K X j ∣ Z ∘ K X j + 1 ∣ Z ) .
We next compute the trace of the second moment Γ ^ S ∣ Z 2 \widehat{\Gamma}_{S\mid Z}^{2} Γ S ∣ Z 2 . For fixed j , k = 1 , … , d − 1 j,k=1,\ldots,d-1 j , k = 1 , … , d − 1 , the coordinate representation gives
tr ( ( Γ ^ S ∣ Z ) ( j , k ) ( Γ ^ S ∣ Z ) ( k , j ) ) = tr ( [ 1 n ∑ i = 1 n ψ ^ j , i ⊗ ψ ^ k , i ] B X j X j + 1 [ 1 n ∑ i ′ = 1 n ψ ^ k , i ′ ⊗ ψ ^ j , i ′ ] B X k X k + 1 ) . \begin{aligned}
\operatorname{tr}\left(\left(\widehat{\Gamma}_{S\mid Z}\right)_{(j,k)}\left(\widehat{\Gamma}_{S\mid Z}\right)_{(k,j)}\right) \\
&= \operatorname{tr}\left(\left[\frac{1}{n}\sum_{i=1}^{n}\widehat{\psi}_{j,i}\otimes\widehat{\psi}_{k,i}\right]_{\mathcal{B}_{X_jX_{j+1}}}\left[\frac{1}{n}\sum_{i'=1}^{n}\widehat{\psi}_{k,i'}\otimes\widehat{\psi}_{j,i'}\right]_{\mathcal{B}_{X_kX_{k+1}}}\right).
\end{aligned} tr ( ( Γ S ∣ Z ) ( j , k ) ( Γ S ∣ Z ) ( k , j ) ) = tr [ n 1 i = 1 ∑ n ψ j , i ⊗ ψ k , i ] B X j X j + 1 [ n 1 i ′ = 1 ∑ n ψ k , i ′ ⊗ ψ j , i ′ ] B X k X k + 1 .
Substituting the matrix representations and using the cyclic property of the trace, we obtain
tr ( ( Γ ^ S ∣ Z ) ( j , k ) ( Γ ^ S ∣ Z ) ( k , j ) ) = tr ( ( 1 n ∑ i = 1 n ( R Z e i ⊗ R Z e i ) ( R Z e i ⊗ R Z e i ) T ) ( K ^ X j X j + 1 ) \begin{aligned}
\operatorname{tr}\left(\left(\widehat{\Gamma}_{S\mid Z}\right)_{(j,k)}\left(\widehat{\Gamma}_{S\mid Z}\right)_{(k,j)}\right) \\
&= \operatorname{tr}\left(\left(\frac{1}{n}\sum_{i=1}^{n}(R_Ze_i\otimes R_Ze_i)(R_Ze_i\otimes R_Ze_i)^{\mathsf{T}}\right)\left(\widehat{K}_{X_jX_{j+1}}\right)
\right.
\end{aligned} tr ( ( Γ S ∣ Z ) ( j , k ) ( Γ S ∣ Z ) ( k , j ) ) = tr ( ( n 1 i = 1 ∑ n ( R Z e i ⊗ R Z e i ) ( R Z e i ⊗ R Z e i ) T ) ( K X j X j + 1 )
× ( 1 n ∑ i ′ = 1 n ( R Z e i ′ ⊗ R Z e i ′ ) ( R Z e i ′ ⊗ R Z e i ′ ) T ) ( K ^ X k X k + 1 ) ) . \left.
\times\left(\frac{1}{n}\sum_{i'=1}^{n}(R_Ze_{i'}\otimes R_Ze_{i'})(R_Ze_{i'}\otimes R_Ze_{i'})^{\mathsf{T}}\right)\left(\widehat{K}_{X_kX_{k+1}}\right)\right). × ( n 1 i ′ = 1 ∑ n ( R Z e i ′ ⊗ R Z e i ′ ) ( R Z e i ′ ⊗ R Z e i ′ ) T ) ( K X k X k + 1 ) ) .
Expanding the two sums and applying the Kronecker product identities, it becomes
1 n 2 ∑ i = 1 n ∑ i ′ = 1 n tr ( ( R Z e i ) ( R Z e i ) T K ^ X j ( R Z e i ′ ) ( R Z e i ′ ) T K ^ X k ) × tr ( ( R Z e i ) ( R Z e i ) T K ^ X j + 1 ( R Z e i ′ ) ( R Z e i ′ ) T K ^ X k + 1 ) . = 1 n 2 ∑ i = 1 n ∑ i ′ = 1 n ( e i T R Z K ^ X j R Z e i ′ ⋅ e i ′ T R Z K ^ X j + 1 R Z e i ) × ( e i ′ T R Z K ^ X k R Z e i ⋅ e i T R Z K ^ X k + 1 R Z e i ′ ) = 1 n 2 tr ( ( K ^ X j ∣ Z ∘ K ^ X j + 1 ∣ Z ) ⋅ ( K ^ X k ∣ Z ∘ K ^ X k + 1 ∣ Z ) ) . \begin{aligned}
\frac{1}{n^2}\sum_{i=1}^{n}\sum_{i'=1}^{n}\operatorname{tr}\left((R_Ze_i)(R_Ze_i)^{\mathsf{T}}\widehat{K}_{X_j}(R_Ze_{i'})(R_Ze_{i'})^{\mathsf{T}}\widehat{K}_{X_k}\right) \\
&\times\operatorname{tr}\left((R_Ze_i)(R_Ze_i)^{\mathsf{T}}\widehat{K}_{X_{j+1}}(R_Ze_{i'})(R_Ze_{i'})^{\mathsf{T}}\widehat{K}_{X_{k+1}}\right). \\
&= \frac{1}{n^2}\sum_{i=1}^{n}\sum_{i'=1}^{n}\left(e_i^{\mathsf{T}}R_Z\widehat{K}_{X_j}R_Ze_{i'}\cdot e_{i'}^{\mathsf{T}}R_Z\widehat{K}_{X_j+1}R_Ze_i\right) \\
&\times\left(e_{i'}^{\mathsf{T}}R_Z\widehat{K}_{X_k}R_Ze_i\cdot e_i^{\mathsf{T}}R_Z\widehat{K}_{X_{k+1}}R_Ze_{i'}\right) \\
&= \frac{1}{n^2}\operatorname{tr}\left(\left(\widehat{K}_{X_j\mid Z}\circ\widehat{K}_{X_{j+1}\mid Z}\right)\cdot\left(\widehat{K}_{X_k\mid Z}\circ\widehat{K}_{X_{k+1}\mid Z}\right)\right).
\end{aligned} n 2 1 i = 1 ∑ n i ′ = 1 ∑ n tr ( ( R Z e i ) ( R Z e i ) T K X j ( R Z e i ′ ) ( R Z e i ′ ) T K X k ) × tr ( ( R Z e i ) ( R Z e i ) T K X j + 1 ( R Z e i ′ ) ( R Z e i ′ ) T K X k + 1 ) . = n 2 1 i = 1 ∑ n i ′ = 1 ∑ n ( e i T R Z K X j R Z e i ′ ⋅ e i ′ T R Z K X j + 1 R Z e i ) × ( e i ′ T R Z K X k R Z e i ⋅ e i T R Z K X k + 1 R Z e i ′ ) = n 2 1 tr ( ( K X j ∣ Z ∘ K X j + 1 ∣ Z ) ⋅ ( K X k ∣ Z ∘ K X k + 1 ∣ Z ) ) .
Summing over j j j and k k k , we conclude that
∑ j = 1 d − 1 ∑ k = 1 d − 1 tr ( ( Γ ^ S ∣ Z ) ( j , k ) ( Γ ^ S ∣ Z ) ( k , j ) ) = 1 n 2 ∑ j = 1 d − 1 ∑ k = 1 d − 1 tr ( ( K ^ X j ∣ Z ∘ K ^ X j + 1 ∣ Z ) ⋅ ( K ^ X k ∣ Z ∘ K ^ X k + 1 ∣ Z ) ) , \begin{aligned}
\sum_{j=1}^{d-1}\sum_{k=1}^{d-1}\operatorname{tr}\left(\left(\widehat{\Gamma}_{S\mid Z}\right)_{(j,k)}\left(\widehat{\Gamma}_{S\mid Z}\right)_{(k,j)}\right) \\
&= \frac{1}{n^2}\sum_{j=1}^{d-1}\sum_{k=1}^{d-1}\operatorname{tr}\left((\widehat{K}_{X_j\mid Z} \circ\widehat{K}_{X_{j+1}\mid Z})\cdot(\widehat{K}_{X_k\mid Z} \circ\widehat{K}_{X_{k+1}\mid Z})\right),
\end{aligned} j = 1 ∑ d − 1 k = 1 ∑ d − 1 tr ( ( Γ S ∣ Z ) ( j , k ) ( Γ S ∣ Z ) ( k , j ) ) = n 2 1 j = 1 ∑ d − 1 k = 1 ∑ d − 1 tr ( ( K X j ∣ Z ∘ K X j + 1 ∣ Z ) ⋅ ( K X k ∣ Z ∘ K X k + 1 ∣ Z ) ) ,
which gives the desired formula. □