Introduction
Across many scientific disciplines, the characteristics distinguishing subpopulations are often missing, unmeasured, or unobservable. This lack of observability complicates statistical analysis and motivates methods that explicitly account for heterogeneity in covariate–response relationships. Such frameworks are known as finite mixtures of regressions in statistics [19 ] , mixtures of experts in machine learning [10 ] , switching regression in econometrics [25 ] , and clusterwise regression in marketing [4 , 33 ] . Compared with existing parametric models in the literature, semiparametric class-specific index regression provides a flexible alternative, reducing the risk of misspecifying the parametric response or distribution function and avoiding the need to predefine class-specific central subspaces.
Analyzing 2018 transaction-level data from a North American supermarket chain, we observed distinct subgroups in purchase amounts in the second half of the year (H2) and an association with covariates measured in the first half (H1) that is invariant across customer segments. In another application, data from the AIDS Clinical Trials Group (ACTG) Study 175 reveal heterogeneity in treatment effects across patient subgroups. In particular, CD4 cell counts measured at 20 ± 5 20 \pm5 20 ± 5 weeks exhibit substantial heterogeneity suggestive of latent subgroups, and treatment effects—zidovudine monotherapy versus alternative therapies—on CD4 cell counts vary across these subgroups. The estimated response function from the class-specific index-mean model exhibits a non-monotonic pattern. In contrast to conventional class-specific linear, generalized linear, and quasi-likelihood models, our semiparametric class-specific index regression flexibly captures complex functional relationships without imposing restrictive parametric structural or distributional assumptions, providing a robust framework for analyzing data with heterogeneous effects.
Building on the subject-specific formulation of the proposed semiparametric class-specific index mean model, we develop a unified estimation and latent class recovery framework for uncovering latent structure without prespecified class labels. The procedure employs a separation-penalty estimation strategy, implemented via a computationally efficient alternating direction method of multipliers (ADMM; Boyd, Parikh, Chu, Peleato, and Eckstein, [2 ] ), with a well-designed initialization scheme that ensures rapid and stable convergence, to obtain estimators of subject index coefficients and latent class memberships. Furthermore, we propose a semiparametric information criterion that consistently estimates the number of classes without relying on the response distribution, complementing the estimation and latent class recovery procedure. Under a general characterization of the response distribution, we derive a Bayes classification rule, incorporating a sufficient dimension reduction (SDR; Li [15 ] ) representation of latent class membership probabilities, to achieve asymptotic optimality by maximizing the probability of correct assignment. We also propose a pseudo-maximum likelihood criterion to improve the efficiency of the separation-penalty estimators of the model parameters. The asymptotic variance matrix of the resulting pseudo-maximum likelihood estimators attains the semiparametric efficiency bound. As noted by Teng and Chiang [31 ] , their pseudo-maximum likelihood method, developed under a semiparametric latent class index distribution model, is more sensitive to extreme values in the estimated density and requires more restrictive bandwidth conditions for continuous responses. Our method avoids these limitations under the proposed model formulation. Finally, by modifying the distribution estimator underlying the information criteria of Teng and Chiang [31 ] , we construct alternative criteria for selecting the number of classes.
In estimating class-specific covariate–response relationships, subject-specific formulations have emerged as a standard approach, evolving from the early combinatorial optimization technique (Späth [29 ] ) to contemporary regularization methods. The latter imposes concave pairwise fusion penalties ((Ma and Huang, 2017; Ma, Huang, Zhang, and Liu, 2020)) or adaptive ℓ 1 \ell_1 ℓ 1 penalties ([3 ] ). Recent innovations, including multidirectional separation penalties ([30 ] ) and center-augmented regularization ((He, Zhou, Xia, and Lin, 2023)), aim to reduce computational complexity. Nonetheless, these approaches remain computationally burdensome and susceptible to inaccurate latent class recovery. Recent work by Teng and Chiang [31 ] partially addresses these challenges within a general semiparametric modeling framework. Building on their computational strategy, we further improve computational efficiency by reducing the number of ADMM loops required from two to one.
Alternatively, mixture regression models explicitly parameterize latent class membership. Early approaches assumed covariate-independent class membership probabilities ([13 , 25 ] ), but were later extended to accommodate covariate-dependent structures, including hierarchical chical mixture-of-experts models ([10 , 11 ] ) and conditional mixture models ([4 ] ). Although robust procedures for testing the presence of latent subpopulations have been developed ([23 , 27 ] ), this class of models remains highly sensitive to misspecification of latent class membership models.
Our target remains a latent-class model. Rather than estimating the model through a marginal mixture likelihood, we introduce the subject-specific indices as a complete-data representation of the realized latent labels. The separation-penalty step estimates this realized-label representation, while the SDR-based class membership model is used in the distributional refinement step. In the proposed class-specific index density model with SDR-based class membership probabilities, direct marginal mixture-likelihood estimation is difficult because the unknown functions in these semiparametric settings create severe identifiability and computational complications.
The remainder of the article is organized as follows. § 2 formalizes the semiparametric class-specific index regression and its subject-specific formulation. § 3 introduces a unified estimation and latent class recovery procedure and a novel criterion for selecting the number of classes. § 4 refines the separation-penalty parameter estimator under a class-specific index density model. § 5 assesses the method’s finite-sample performance through simulation studies. § 6 illustrates its practical utility in personalized marketing and patient stratification. Finally, § 7 concludes with key findings and directions for future research. Appendix § 8 summarizes the notation used in § 2 –§ 4 .
Semiparametric Class-Specific Index Regression
Let Y ∈ Y Y \in\mathcal{Y} Y ∈ Y denote a quantitative response variable, X ∈ X X \in\mathcal{X} X ∈ X a p × 1 p \times1 p × 1 covariate vector, k k k the number of latent classes, and C ∈ C = { 1 , … , k } C \in\mathcal{C} = \{1,\ldots,k\} C ∈ C = { 1 , … , k } a latent class variable. We consider a semiparametric class-specific index model (CSIM) for the conditional mean:
E [ Y ∣ X = x , C = c ] = m ( γ c T z ) , ( x , c ) ∈ X × C , (1) \mathbb{E}[Y \mid X=x,C=c] = m(\gamma_c^{\mathsf{T}}z),\quad(x,c) \in\mathcal{X} \times\mathcal{C}, \tag*{(1)} E [ Y ∣ X = x , C = c ] = m ( γ c T z ) , ( x , c ) ∈ X × C , (1)
where γ c = ( γ c 0 , γ c 1 , … , γ c p ) T \gamma_c = (\gamma_{c0}, \gamma_{c1}, \ldots, \gamma_{cp})^{\mathsf{T}} γ c = ( γ c 0 , γ c 1 , … , γ c p ) T is the index coefficient vector for Class c c c , z z z is a realization of the augmented covariate vector Z = ( 1 , X T ) T Z = (1, X^{\mathsf{T}})^{\mathsf{T}} Z = ( 1 , X T ) T , and m ( ⋅ ) m(\cdot) m ( ⋅ ) is an unknown response function common across classes. This latent class model is identifiable under the following conditions. First, γ 11 ≠ 0 \gamma_{11} \ne0 γ 11 = 0 , corresponding to a continuous covariate that satisfies the continuity condition of Ichimura [9 ] , and ( γ 10 , γ 11 ) (\gamma_{10}, \gamma_{11}) ( γ 10 , γ 11 ) is normalized to ( 0 , 1 ) (0,1) ( 0 , 1 ) . Second, the separation condition m ( γ c T z ) ≠ m ( γ ℓ T z ) m(\gamma_c^{\mathsf{T}}z) \ne m(\gamma_\ell^{\mathsf{T}}z) m ( γ c T z ) = m ( γ ℓ T z ) holds for all c ≠ ℓ c \ne\ell c = ℓ , c , ℓ ∈ C c,\ell\in\mathcal{C} c , ℓ ∈ C , and almost every x ∈ X x \in\mathcal{X} x ∈ X . Under these assumptions, the CSIM (1) is identifiable up to a permutation of the class labels. Notably, the separation condition rules out periodic response functions over the covariate space in the CSIM. Since our approach is based on a subject-specific formulation of the CSIM rather than the mixture representation E [ Y ∣ X = x ] = ∑ c = 1 k P ( C = c ∣ X = x ) m ( γ c T z ) \operatorname{E}[Y \mid X = x] = \sum_{c=1}^{k} P(C = c \mid X = x)m(\gamma_c^{\mathsf{T}}z) E [ Y ∣ X = x ] = ∑ c = 1 k P ( C = c ∣ X = x ) m ( γ c T z ) , it avoids the associated identifiability issues.
The proposed model encompasses, as special cases, parametric mean structures arising in generalized linear models (Nelder and Wedderburn [20 ] ) and quasi-likelihood models (Wedderburn [32 ] ). We decompose γ c \gamma_c γ c into a class-specific index coefficient vector γ ( 1 ) c \gamma_{(1)c} γ ( 1 ) c and a class-invariant index coefficient vector γ ( 2 ) \gamma_{(2)} γ ( 2 ) , aligned with the covariate subsets Z ( 1 ) Z_{(1)} Z ( 1 ) and Z ( 2 ) Z_{(2)} Z ( 2 ) , so that the class-specific index can be written as γ c T Z = γ ( 1 ) c T Z ( 1 ) + γ ( 2 ) T Z ( 2 ) \gamma_c^{\mathsf{T}} Z = \gamma_{(1)c}^{\mathsf{T}} Z_{(1)} + \gamma_{(2)}^{\mathsf{T}} Z_{(2)} γ c T Z = γ ( 1 ) c T Z ( 1 ) + γ ( 2 ) T Z ( 2 ) for each c ∈ C c \in\mathcal{C} c ∈ C . Under this parameterization, the CSIM (1) is given by
E [ Y ∣ X = x , C = c ] = m ( γ ( 1 ) c T z ( 1 ) + γ ( 2 ) T z ( 2 ) ) , ( x , c ) ∈ X × C . (2) \operatorname{E}[Y \mid X = x, C = c] = m\left(\gamma_{(1)c}^{\mathsf{T}} z_{(1)} + \gamma_{(2)}^{\mathsf{T}} z_{(2)}\right), \quad(x,c) \in\mathcal{X} \times\mathcal{C}. \tag*{(2)} E [ Y ∣ X = x , C = c ] = m ( γ ( 1 ) c T z ( 1 ) + γ ( 2 ) T z ( 2 ) ) , ( x , c ) ∈ X × C . (2)
Given observations { ( X i , Y i ) } i = 1 n \{(X_i,Y_i)\}_{i=1}^{n} {( X i , Y i ) } i = 1 n , we adopt a subject-specific formulation of the CSIM (1) within a pseudo-estimation framework:
E [ Y i ∣ X i , β i ] = m ( β i T Z i ) , i = 1 , … , n , (3) \operatorname{E}[Y_i \mid X_i,\beta_i] = m\left(\beta_i^{\mathsf{T}}Z_i\right), \quad i = 1,\ldots,n, \tag*{(3)} E [ Y i ∣ X i , β i ] = m ( β i T Z i ) , i = 1 , … , n , (3)
where β i ∈ { γ 1 , … , γ k } \beta_i \in\{\gamma_1,\ldots,\gamma_k\} β i ∈ { γ 1 , … , γ k } and β i = γ c \beta_i = \gamma_c β i = γ c if and only if C i = c C_i = c C i = c . Although the population model is a latent class model, the subject index coefficients β i = γ C i \beta_i = \gamma_{C_i} β i = γ C i , i = 1 , … , n i = 1,\ldots,n i = 1 , … , n , are used as a complete-data representation of the realized latent class labels. This subject-specific index model (SSIM) enables direct estimation of the response function m ( ⋅ ) m(\cdot) m ( ⋅ ) via the subject index coefficient vectors { β i } i = 1 n \{\beta_i\}_{i=1}^{n} { β i } i = 1 n , without requiring the latent class memberships to be known in advance. For each observation i i i , the subject index coefficient vector is partitioned as β i = ( β ( 1 ) i ⊤ , β ( 2 ) ⊤ ) ⊤ \beta_i = \left(\beta_{(1)i}^{\top}, \beta_{(2)}^{\top}\right)^{\top} β i = ( β ( 1 ) i ⊤ , β ( 2 ) ⊤ ) ⊤ , where β ( 1 ) i ∈ { γ ( 1 ) 1 , … , γ ( 1 ) k } \beta_{(1)i} \in\{\gamma_{(1)1}, \ldots, \gamma_{(1)k}\} β ( 1 ) i ∈ { γ ( 1 ) 1 , … , γ ( 1 ) k } and β ( 2 ) = γ ( 2 ) \beta_{(2)} = \gamma_{(2)} β ( 2 ) = γ ( 2 ) .
Specifying the distribution for Y Y Y —for example, within an exponential or location family—enables the estimation procedure to fully exploit this information, thereby improving the efficiency of parameter estimators. In particular, consider the SSIM
Y i = m ( β i ⊤ Z i ) + U i , i = 1 , … , n , (4) Y_i = m\left(\beta_i^{\top} Z_i\right) + U_i,\quad i = 1,\ldots,n, \tag*{(4)} Y i = m ( β i ⊤ Z i ) + U i , i = 1 , … , n , (4)
where U i U_i U i is independent of ( X i , β i ) (X_i,\beta_i) ( X i , β i ) and has an unspecified probability density function f ( ⋅ ) f(\cdot) f ( ⋅ ) . Its corresponding CSIM representation is
Y = m ( γ C ⊤ Z ) + U , C ∈ C . (5) Y = m\left(\gamma_C^{\top} Z\right) + U,\quad C \in\mathcal{C}. \tag*{(5)} Y = m ( γ C ⊤ Z ) + U , C ∈ C . (5)
Therefore, the conditional probability density function f ( y ∣ x , c ) f(y \mid x,c) f ( y ∣ x , c ) of Y Y Y given ( X , C ) (X,C) ( X , C ) is
f ( y ∣ x , c ) = f ( y − m ( γ c ⊤ z ) ) , ( y , x , c ) ∈ Y × X × C . (6) f(y \mid x,c) = f\left(y - m\left(\gamma_c^{\top}z\right)\right),\quad(y,x,c) \in\mathcal{Y} \times\mathcal{X} \times\mathcal{C}. \tag*{(6)} f ( y ∣ x , c ) = f ( y − m ( γ c ⊤ z ) ) , ( y , x , c ) ∈ Y × X × C . (6)
In many applications, including bioinformatics and marketing, responses are nonnegative (e.g., biomarker concentrations or purchase amounts) and are often analyzed on the log scale. The CSIM (5) accommodates such log-linear specifications as special cases.
Remark 1. When Y Y Y follows a distribution in the exponential family, the conditional probability density function of Y Y Y given ( X , C ) (X,C) ( X , C ) can be written as
f ( y ∣ x , c ) = exp ( y θ ( m ( γ c ⊤ z ) ) − b ( θ ( m ( γ c ⊤ z ) ) ) ϕ + c ( y , ϕ ) ) , ( y , x , c ) ∈ Y × X × C , (7) f(y \mid x,c) = \exp\left(\frac{y\theta\left(m\left(\gamma_c^{\top}z\right)\right)-b\left(\theta\left(m\left(\gamma_c^{\top}z\right)\right)\right)}{\phi}+c(y,\phi)\right),\quad(y,x,c) \in\mathcal{Y} \times\mathcal{X} \times\mathcal{C}, \tag*{(7)} f ( y ∣ x , c ) = exp ( ϕ y θ ( m ( γ c ⊤ z ) ) − b ( θ ( m ( γ c ⊤ z ) ) ) + c ( y , ϕ ) ) , ( y , x , c ) ∈ Y × X × C , (7)
where θ ( ⋅ ) \theta(\cdot) θ ( ⋅ ) , b ( ⋅ ) b(\cdot) b ( ⋅ ) , and c ( ⋅ , ⋅ ) c(\cdot,\cdot) c ( ⋅ , ⋅ ) are known functions determined with the chosen exponential family, and ϕ \phi ϕ is a scale parameter. An estimation procedure can be constructed in close analogy to that under the class-specific index probability density model (6) , while the corresponding theoretical analysis proceeds similarly but is simplified by the absence of any need to estimate the error density f ( ⋅ ) f(\cdot) f ( ⋅ ) .
Estimation and Class Number Selection
Let θ ∗ \theta_* θ ∗ denote the unknown components of a generic parameter vector θ \theta θ . Corresponding to the alternative expression (2) of the CSIM (1) and the SSIM (3) , let γ = ( γ ( 1 ) ⊤ , γ ( 2 ) ⊤ ) \gamma= \left(\gamma_{(1)}^{\top},\gamma_{(2)}^{\top}\right) γ = ( γ ( 1 ) ⊤ , γ ( 2 ) ⊤ ) and β = \beta= β = ( β ( 1 ) T , β ( 2 ) T ) T (\beta_{(1)}^{\mathsf T},\beta_{(2)}^{\mathsf T})^{\mathsf T} ( β ( 1 ) T , β ( 2 ) T ) T , where γ ( 1 ) = ( γ ( 1 ) 1 T , … , γ ( 1 ) k T ) T \gamma_{(1)}=(\gamma_{(1)1}^{\mathsf T},\ldots,\gamma_{(1)k}^{\mathsf T})^{\mathsf T} γ ( 1 ) = ( γ ( 1 ) 1 T , … , γ ( 1 ) k T ) T and β ( 1 ) = ( β ( 1 ) 1 T , … , β ( 1 ) n T ) T \beta_{(1)}=(\beta_{(1)1}^{\mathsf T},\ldots,\beta_{(1)n}^{\mathsf T})^{\mathsf T} β ( 1 ) = ( β ( 1 ) 1 T , … , β ( 1 ) n T ) T . The true parameters are denoted by γ o \gamma^{\mathrm{o}} γ o and β o \beta^{\mathrm{o}} β o . For the estimators defined below, we adopt the convention that a ratio of the form 0 / 0 0/0 0/0 is assigned the value zero.
Pseudo-Least-Squares Estimation with a Separation Penalty
Given a coefficient vector β \beta β , we estimate the response function m ( v ) m(v) m ( v ) using the kernel estimator
m ^ h ( v ; β ) = ∑ i = 1 n Y i K h ( β i T Z i − v ) ∑ i = 1 n K h ( β i T Z i − v ) (8) \widehat{m}_h(v;\beta)=\frac{\sum_{i=1}^{n}Y_iK_h(\beta_i^{\mathsf T}Z_i-v)}{\sum_{i=1}^{n}K_h(\beta_i^{\mathsf T}Z_i-v)} \tag*{(8)} m h ( v ; β ) = ∑ i = 1 n K h ( β i T Z i − v ) ∑ i = 1 n Y i K h ( β i T Z i − v ) (8)
where K h ( v ) = K ( v / h ) / h K_h(v)=K(v/h)/h K h ( v ) = K ( v / h ) / h is a scaled kernel with bandwidth h > 0 h>0 h > 0 . The kernel K ( ⋅ ) K(\cdot) K ( ⋅ ) is symmetric and compactly supported, has a constant fourth derivative, and satisfies ∫ K ( 2 ) ( u ) d u = 0 \int K^{(2)}(u)du=0 ∫ K ( 2 ) ( u ) d u = 0 . The biweight kernel K ( u ) = 15 / 16 ( 1 − u 2 ) 2 I ( ∣ u ∣ ≤ 1 ) K(u)=15/16(1-u^2)^2I(|u|\leq1) K ( u ) = 15/16 ( 1 − u 2 ) 2 I ( ∣ u ∣ ≤ 1 ) is used in the numerical studies. For fixed h h h and λ \lambda λ , the separation-penalty (SP) estimator ( β ^ , γ ^ ( 1 ) ) (\widehat{\beta},\widehat{\gamma}_{(1)}) ( β , γ ( 1 ) ) of ( β o , γ ( 1 ) o ) (\beta^{\mathrm{o}},\gamma^{\mathrm{o}}_{(1)}) ( β o , γ ( 1 ) o ) is defined as the minimizer of a pseudo-least-squares criterion augmented with an ℓ 1 \ell_1 ℓ 1 separation penalty:
PSS S P ( β , γ ( 1 ) ) = 1 2 ∑ i = 1 n ( Y i − m ^ h − i ( β i T Z i ; β ) ) 2 + λ ∑ i = 1 n min 1 ≤ c ≤ k ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 (9) \operatorname{PSS}_{\mathrm{SP}}(\beta,\gamma_{(1)})=\frac{1}{2}\sum_{i=1}^{n}\left(Y_i-\widehat{m}_h^{-i}(\beta_i^{\mathsf T}Z_i;\beta)\right)^2+\lambda\sum_{i=1}^{n}\min_{1\leq c\leq k}\lVert\beta_{(1)i}-\gamma_{(1)c}\rVert_1 \tag*{(9)} PSS SP ( β , γ ( 1 ) ) = 2 1 i = 1 ∑ n ( Y i − m h − i ( β i T Z i ; β ) ) 2 + λ i = 1 ∑ n 1 ≤ c ≤ k min ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 (9)
where m ^ h − i ( v ; β ) \widehat{m}_h^{-i}(v;\beta) m h − i ( v ; β ) denotes the leave-one-out version of m ^ h ( v ; β ) \widehat{m}_h(v;\beta) m h ( v ; β ) , i = 1 , … , n i=1,\ldots,n i = 1 , … , n , λ > 0 \lambda>0 λ > 0 controls class separation, and ∥ ⋅ ∥ 1 \lVert\cdot\rVert_1 ∥ ⋅ ∥ 1 denotes the ℓ 1 \ell_1 ℓ 1 -norm. Larger values of λ \lambda λ yield more homogeneous classes, whereas smaller values allow for greater within-class variability. The estimator of γ o \gamma^{\mathrm{o}} γ o is given by γ ^ = ( γ ^ ( 1 ) T , β ^ ( 2 ) T ) T \widehat{\gamma}=(\widehat{\gamma}_{(1)}^{\mathsf T},\widehat{\beta}_{(2)}^{\mathsf T})^{\mathsf T} γ = ( γ ( 1 ) T , β ( 2 ) T ) T , since γ ( 2 ) o = β ( 2 ) o \gamma^{\mathrm{o}}_{(2)}=\beta^{\mathrm{o}}_{(2)} γ ( 2 ) o = β ( 2 ) o . If β i \beta_i β i contains no subject-invariant component, the penalty is applied to ( β i , γ c ) (\beta_i,\gamma_c) ( β i , γ c ) rather than ( β ( 1 ) i , γ ( 1 ) c ) (\beta_{(1)i},\gamma_{(1)c}) ( β ( 1 ) i , γ ( 1 ) c ) for all i = 1 , … , n i=1,\ldots,n i = 1 , … , n and c ∈ C c\in\mathcal{C} c ∈ C . The local theoretical analysis restricts the criterion to the truth-centered parameter region specified in assumption A4. We do not, however, assume that the initial values generated by the initialization system are known to belong to this region. The unrestricted SP estimator is further evaluated in § 5 .
Let G o = { G 1 o , … , G k o } \mathcal{G}^{\mathrm{o}}=\{\mathcal{G}^{\mathrm{o}}_1,\ldots,\mathcal{G}^{\mathrm{o}}_k\} G o = { G 1 o , … , G k o } , where G c o = { i : C i = c } \mathcal{G}^{\mathrm{o}}_c=\{i:C_i=c\} G c o = { i : C i = c } , denote the realized latent partition. Based on the SP estimator ( β ^ , γ ^ ( 1 ) ) (\widehat{\beta},\widehat{\gamma}_{(1)}) ( β , γ ( 1 ) ) , define the estimated latent class partition G ^ = ( G ^ 1 , … , G ^ k ) \widehat{\mathcal{G}}=(\widehat{\mathcal{G}}_1,\ldots,\widehat{\mathcal{G}}_k) G = ( G 1 , … , G k ) of G o \mathcal{G}^{\mathrm{o}} G o by assigning observation i i i to Class c c c whenever
c ∈ arg min 1 ≤ ℓ ≤ k ∥ β ^ ( 1 ) i − γ ^ ( 1 ) ℓ ∥ 1 . c\in\underset{1\leq\ell\leq k}{\arg\min}\lVert\widehat{\beta}_{(1)i}-\widehat{\gamma}_{(1)\ell}\rVert_1. c ∈ 1 ≤ ℓ ≤ k arg min ∥ β ( 1 ) i − γ ( 1 ) ℓ ∥ 1 .
Conditional on the realized labels, β i o = γ c o \beta_i^{\mathrm{o}}=\gamma_c^{\mathrm{o}} β i o = γ c o for i ∈ G c o i\in\mathcal{G}_c^{\mathrm{o}} i ∈ G c o and c ∈ C c\in\mathcal{C} c ∈ C . We define the objective function
PSS ( γ , G ) = 1 2 ∑ i = 1 n ( Y i − ∑ c = 1 k I ( i ∈ G c ) m ^ h − i ( γ c T Z i ; G , γ ) ) 2 , (10) \operatorname{PSS}(\gamma,\mathcal{G})=\frac{1}{2}\sum_{i=1}^{n}\left(Y_i-\sum_{c=1}^{k}I(i\in\mathcal{G}_c)\widehat{m}_{h}^{-i}(\gamma_c^{\mathsf{T}}Z_i;\mathcal{G},\gamma)\right)^2, \tag*{(10)} PSS ( γ , G ) = 2 1 i = 1 ∑ n ( Y i − c = 1 ∑ k I ( i ∈ G c ) m h − i ( γ c T Z i ; G , γ ) ) 2 , (10)
where G = { G 1 , … , G k } \mathcal{G}=\{\mathcal{G}_1,\ldots,\mathcal{G}_k\} G = { G 1 , … , G k } denotes a partition of the individual index set { 1 , … , n } \{1,\ldots,n\} { 1 , … , n } , I ( ⋅ ) I(\cdot) I ( ⋅ ) is the indicator function, and m ^ h ( v ; G , γ ) \widehat{m}_h(v;\mathcal{G},\gamma) m h ( v ; G , γ ) is an alternative representation of m ^ h ( v ; β ) \widehat{m}_h(v;\beta) m h ( v ; β ) in (8) , given by
m ^ h ( v ; G , γ ) = ∑ i = 1 n Y i ∑ c = 1 k I ( i ∈ G c ) K h ( γ c T Z i − v ) ∑ i = 1 n ∑ c = 1 k I ( i ∈ G c ) K h ( γ c T Z i − v ) . (11) \widehat{m}_h(v;\mathcal{G},\gamma)=\frac{\sum_{i=1}^{n}Y_i\sum_{c=1}^{k}I(i\in\mathcal{G}_c)K_h(\gamma_c^{\mathsf{T}}Z_i-v)}{\sum_{i=1}^{n}\sum_{c=1}^{k}I(i\in\mathcal{G}_c)K_h(\gamma_c^{\mathsf{T}}Z_i-v)}. \tag*{(11)} m h ( v ; G , γ ) = ∑ i = 1 n ∑ c = 1 k I ( i ∈ G c ) K h ( γ c T Z i − v ) ∑ i = 1 n Y i ∑ c = 1 k I ( i ∈ G c ) K h ( γ c T Z i − v ) . (11)
The bandwidth is selected as h ^ = n 3 / 80 h ~ \widehat{h}=n^{3/80}\widetilde{h} h = n 3/80 h to guarantee the n \sqrt{n} n -consistency of the SP estimator γ ^ \widehat{\gamma} γ , where h ~ = O p ( n − 1 / 5 ) \widetilde{h}=O_p(n^{-1/5}) h = O p ( n − 1/5 ) is the converged value of h ~ ( s ) ∈ arg min h PSS ( γ ^ ( s ) , G ^ ( s ) ) \widetilde{h}^{(s)}\in\arg\min_h\operatorname{PSS}(\widehat{\gamma}^{(s)},\widehat{\mathcal{G}}^{(s)}) h ( s ) ∈ arg min h PSS ( γ ( s ) , G ( s ) ) for a fixed λ \lambda λ , and γ ^ ( s ) \widehat{\gamma}^{(s)} γ ( s ) and G ^ ( s ) \widehat{\mathcal{G}}^{(s)} G ( s ) are obtained from the ADMM iterates at iteration s ≥ 0 s\geq0 s ≥ 0 . The tuning parameter λ \lambda λ is chosen to minimize PSS ( γ ^ , G ^ ) \operatorname{PSS}(\widehat{\gamma},\widehat{\mathcal{G}}) PSS ( γ , G ) over a pre-specified shrinkage grid in Appendix § 9 .§ 2 . In our exploratory comparison, the SP-stage results obtained using the selected rule are broadly competitive with those obtained by fixing the position in the data-dependent sequence, although they do not uniformly yield the best performance.
Let B B B and Γ \Gamma Γ denote the compact parameter spaces of β ∗ \beta_* β ∗ and γ ∗ \gamma_* γ ∗ , respectively, and suppose that β ∗ o \beta_*^{\mathrm{o}} β ∗ o and γ ∗ o \gamma_*^{\mathrm{o}} γ ∗ o lie in their interiors. Define the oracle estimator of β o \beta^{\mathrm{o}} β o as β ^ o r = ∑ c = 1 k ( I ( 1 ∈ G c o ) , … , I ( n ∈ G c o ) ) T ⊗ γ ^ c o r \widehat{\beta}^{\mathrm{or}}=\sum_{c=1}^{k}(I(1\in\mathcal{G}_c^{\mathrm{o}}),\ldots,I(n\in\mathcal{G}_c^{\mathrm{o}}))^{\mathsf{T}}\otimes\widehat{\gamma}_c^{\mathrm{or}} β or = ∑ c = 1 k ( I ( 1 ∈ G c o ) , … , I ( n ∈ G c o ) ) T ⊗ γ c or , where γ ^ o r \widehat{\gamma}^{\mathrm{or}} γ or minimizes the criterion (10) with G = G o \mathcal{G}=\mathcal{G}^{\mathrm{o}} G = G o , and ⊗ \otimes ⊗ denotes the Kronecker product. The oracle estimator is the infeasible estimator that knows G o \mathcal{G}^{\mathrm{o}} G o . The following theorem establishes the asymptotic properties of the oracle estimator γ ^ ∗ o r \widehat{\gamma}_*^{\mathrm{or}} γ ∗ or .
Theorem 1. Under assumptions A1–A3, A5, and A6,
γ ^ ∗ o r → p γ ∗ o and n ( γ ^ ∗ o r − γ ∗ o ) → d N ( 0 , V 2 − 1 V 1 V 2 − 1 ) as n ⟶ ∞ , \widehat{\gamma}_*^{\mathrm{or}}\xrightarrow{p}\gamma_*^{\mathrm{o}}\quad\text{and}\quad\sqrt{n}\left(\widehat{\gamma}_*^{\mathrm{or}}-\gamma_*^{\mathrm{o}}\right)\xrightarrow{d}N\left(0,V_2^{-1}V_1V_2^{-1}\right)\text{ as }n\longrightarrow\infty, γ ∗ or p γ ∗ o and n ( γ ∗ or − γ ∗ o ) d N ( 0 , V 2 − 1 V 1 V 2 − 1 ) as n ⟶ ∞ , where V 1 = E [ ( ( Y − m ( γ o T W ) ) m [ 1 ] ( W , γ o T W ) ) ⊗ 2 ] V_1=E[((Y-m(\gamma^{\mathrm{o}\mathsf{T}}W))m^{[1]}(W,\gamma^{\mathrm{o}\mathsf{T}}W))^{\otimes2}] V 1 = E [(( Y − m ( γ o T W )) m [ 1 ] ( W , γ o T W ) ) ⊗ 2 ] and V 2 = E [ ( m [ 1 ] ( W , γ o T W ) ) ⊗ 2 ] V_2=E[(m^{[1]}(W,\gamma^{\mathrm{o}\mathsf{T}}W))^{\otimes2}] V 2 = E [( m [ 1 ] ( W , γ o T W ) ) ⊗ 2 ] , with W = ( I ( C = 1 ) , … , I ( C = k ) ) T ⊗ Z ∈ W W=(I(C=1),\ldots,I(C=k))^{\mathsf{T}}\otimes Z\in\mathcal{W} W = ( I ( C = 1 ) , … , I ( C = k ) ) T ⊗ Z ∈ W , m [ t ] ( w , γ T w ) m^{[t]}(w,\gamma^{\mathsf{T}}w) m [ t ] ( w , γ T w ) , t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 , defined in Lemma S2, and ( ⋅ ) ⊗ (\cdot)^{\otimes} ( ⋅ ) ⊗ denoting the Kronecker power.
Proof. See Appendix E.
The next theorem establishes the local oracle property of the proposed SP estimator ( β ^ , γ ^ ( 1 ) ) (\widehat{\beta},\widehat{\gamma}^{(1)}) ( β , γ ( 1 ) ) .
Theorem 2. Under assumptions A1–A7, there exists a local minimizer ( β ^ , γ ^ ( 1 ) ) (\widehat{\beta},\widehat{\gamma}(1)) ( β , γ ( 1 )) of the criterion (9) satisfying
P ( ( β ^ , γ ^ ) = ( β ^ o r , γ ^ o r ) ) ⟶ 1 as n ⟶ ∞ . P\left((\widehat{\beta},\widehat{\gamma})=(\widehat{\beta}^{or},\widehat{\gamma}^{or})\right)\longrightarrow1 \text{ as } n\longrightarrow\infty. P ( ( β , γ ) = ( β or , γ or ) ) ⟶ 1 as n ⟶ ∞.
Proof. See Appendix § 13 .
Because ( β ^ , γ ^ ) = ( β ^ o r , γ ^ o r ) (\widehat{\beta},\widehat{\gamma})=(\widehat{\beta}^{or},\widehat{\gamma}^{or}) ( β , γ ) = ( β or , γ or ) entails G ^ = G o \widehat{G}=G^{o} G = G o under the label alignment used in assumption A4, it follows that
P ( G ^ = G o ) ⟶ 1 as n ⟶ ∞ . (12) P(\widehat{G}=G^{o})\longrightarrow1 \text{ as } n\longrightarrow\infty. \tag*{(12)} P ( G = G o ) ⟶ 1 as n ⟶ ∞. (12)
The exact latent-partition recovery should not be interpreted as a general perfect-classification result for mixtures with fixed nonvanishing overlap. Under such overlap, a positive per-observation Bayes classification error generally precludes recovery of every realized latent label with probability converging to one. Within the localized problem, assumption A4 constrains each candidate subject parameter to a neighborhood of its corresponding true center, while assumption A7 imposes a sufficiently strong separation penalty to ensure the dominance of the oracle configuration over competing local alternatives. Combining Theorems 1 and 2 yields
n ( γ ^ ∗ − γ ∗ o ) → d N ( 0 , V 2 − 1 V 1 V 2 − 1 ) as n ⟶ ∞ . (13) \sqrt{n}(\widehat{\gamma}_{*}-\gamma_{*}^{o}) \xrightarrow{d} N(0,V_{2}^{-1}V_{1}V_{2}^{-1}) \text{ as } n\longrightarrow\infty. \tag*{(13)} n ( γ ∗ − γ ∗ o ) d N ( 0 , V 2 − 1 V 1 V 2 − 1 ) as n ⟶ ∞. (13)
The asymptotic variance matrix of γ ^ ∗ \widehat{\gamma}_{*} γ ∗ can be consistently estimated by V ^ 2 − 1 V ^ 1 V ^ 2 − 1 \widehat{V}_{2}^{-1}\widehat{V}_{1}\widehat{V}_{2}^{-1} V 2 − 1 V 1 V 2 − 1 , where V ^ 1 \widehat{V}_{1} V 1 and V ^ 2 \widehat{V}_{2} V 2 are constructed using the optimal bandwidth h ~ \widetilde{h} h for the response function estimator, with
V ^ 1 = 1 n ∑ i = 1 n ∑ c = 1 k I ( i ∈ G ^ c ) [ ( Y i − m ^ h ~ ( γ ^ c T Z i ; G ^ , γ ^ ) ) ∂ γ c m ^ h ~ ( γ ^ c T Z i ; G ^ , γ ^ ) ] ⊗ 2 \widehat{V}_{1}=\frac{1}{n}\sum_{i=1}^{n}\sum_{c=1}^{k}I(i\in\widehat{G}_{c})\left[\left(Y_{i}-\widehat{m}_{\widetilde{h}}(\widehat{\gamma}_{c}^{\mathsf{T}}Z_{i};\widehat{G},\widehat{\gamma})\right)\partial_{\gamma_{c}}\widehat{m}_{\widetilde{h}}(\widehat{\gamma}_{c}^{\mathsf{T}}Z_{i};\widehat{G},\widehat{\gamma})\right]^{\otimes2} V 1 = n 1 i = 1 ∑ n c = 1 ∑ k I ( i ∈ G c ) [ ( Y i − m h ( γ c T Z i ; G , γ ) ) ∂ γ c m h ( γ c T Z i ; G , γ ) ] ⊗ 2
and
V ^ 2 = 1 n ∑ i = 1 n ∑ c = 1 k I ( i ∈ G ^ c ) ( ∂ γ c m ^ h ~ ( γ ^ c T Z i ; G ^ , γ ^ ) ) ⊗ 2 . \widehat{V}_{2}=\frac{1}{n}\sum_{i=1}^{n}\sum_{c=1}^{k}I(i\in\widehat{G}_{c})\left(\partial_{\gamma_{c}}\widehat{m}_{\widetilde{h}}(\widehat{\gamma}_{c}^{\mathsf{T}}Z_{i};\widehat{G},\widehat{\gamma})\right)^{\otimes2}. V 2 = n 1 i = 1 ∑ n c = 1 ∑ k I ( i ∈ G c ) ( ∂ γ c m h ( γ c T Z i ; G , γ ) ) ⊗ 2 .
Remark 2. In the spirit of quasi-likelihood methods that avoid full distributional specification, the proposed SSIM (3) extends naturally to accommodate a class-specific working variance matrix of the form σ 2 ( m ( γ c T z ) ) \sigma^{2}(m(\gamma_{c}^{\mathsf{T}}z)) σ 2 ( m ( γ c T z )) for each c ∈ C c\in\mathcal{C} c ∈ C , where σ 2 ( ⋅ ) \sigma^{2}(\cdot) σ 2 ( ⋅ ) may be specified a priori or left unspecified. For estimation, the SP criterion in (9) is modified by replacing the pseudo sum of squares with ∑ i = 1 n ω i ( Y i − m ^ h − i ( β i T Z i ; β ) ) 2 \sum_{i=1}^{n}\omega_i(Y_i-\widehat{m}^{-i}_h(\beta_i^{\mathsf{T}}Z_i;\beta))^2 ∑ i = 1 n ω i ( Y i − m h − i ( β i T Z i ; β ) ) 2 . The weights ω 1 , … , ω n \omega_1,\ldots,\omega_n ω 1 , … , ω n are initialized at one and subsequently updated to consistent estimators of σ − 2 ( m ( β 1 T Z 1 ) ) , … , σ − 2 ( m ( β n T Z n ) ) \sigma^{-2}(m(\beta_1^{\mathsf{T}}Z_1)),\ldots,\sigma^{-2}(m(\beta_n^{\mathsf{T}}Z_n)) σ − 2 ( m ( β 1 T Z 1 )) , … , σ − 2 ( m ( β n T Z n )) until a convergence criterion is met.
Selecting the Number of Classes
We select the number of classes, k ^ \widehat{k} k , by minimizing the semiparametric information criterion (SPIC)
SPIC 0 ( k ) = 2 PSS ( γ ^ , G ^ ) ∑ i = 1 n ( Y i − Y ‾ ) 2 + k log n n 4 / 5 , k ≥ 1 , (14) \operatorname{SPIC}_0(k)=\frac{2\operatorname{PSS}(\widehat{\gamma},\widehat{G})}{\sum_{i=1}^{n}(Y_i-\overline{Y})^2}+\frac{k\log n}{n^{4/5}},\quad k\geq1, \tag*{(14)} SPIC 0 ( k ) = ∑ i = 1 n ( Y i − Y ) 2 2 PSS ( γ , G ) + n 4/5 k log n , k ≥ 1 , (14)
where Y ‾ = ∑ i = 1 n Y i / n \overline{Y}=\sum_{i=1}^{n}Y_i/n Y = ∑ i = 1 n Y i / n . For large samples (e.g., n ≳ 600 n\gtrsim600 n ≳ 600 in our numerical studies), we augment the penalty in (14) to k ( log n / n 4 / 5 + log n / n 9 / 10 ) k(\log n/n^{4/5}+\log n/n^{9/10}) k ( log n / n 4/5 + log n / n 9/10 ) to mitigate overestimation of the number of classes. This adjustment accounts for the interaction between the parametric and nonparametric convergence rates, namely sup v ∣ m ^ h ( v ; G ^ , γ ^ ) − m ( v ) ∣ = o p ( log n / n 4 / 5 ) \sup_v|\widehat{m}_h(v;\widehat{G},\widehat{\gamma})-m(v)|=o_p(\sqrt{\log n/n^{4/5}}) sup v ∣ m h ( v ; G , γ ) − m ( v ) ∣ = o p ( log n / n 4/5 ) and γ ^ ∗ − γ ∗ ∗ = O p ( n − 1 / 2 ) \widehat{\gamma}_*-\gamma^*_*=O_p(n^{-1/2}) γ ∗ − γ ∗ ∗ = O p ( n − 1/2 ) .
Let k ∗ , C ∗ ∈ C ∗ = { 1 , … , k ∗ } k^*,\ C^*\in\mathcal{C}^*=\{1,\ldots,k^*\} k ∗ , C ∗ ∈ C ∗ = { 1 , … , k ∗ } , and γ 1 ∗ o , … , γ k ∗ ∗ o \gamma_1^{*o},\ldots,\gamma_{k^*}^{*o} γ 1 ∗ o , … , γ k ∗ ∗ o denote the true values of the number of classes, the latent class variable, and the class index coefficients, respectively, in the CSIM (1) . The realized latent partition is denoted by G ∗ o = { G 1 ∗ o , … , G k ∗ ∗ o } \mathcal{G}^{*o}=\{\mathcal{G}_1^{*o},\ldots,\mathcal{G}_{k^*}^{*o}\} G ∗ o = { G 1 ∗ o , … , G k ∗ ∗ o } , where G c ∗ o = { i : C i ∗ = c } \mathcal{G}_c^{*o}=\{i:C_i^*=c\} G c ∗ o = { i : C i ∗ = c } . We denote U ∗ U^* U ∗ as Y − m ( γ ∗ o T W ∗ ) Y-m(\gamma^{*o\mathsf{T}}W^*) Y − m ( γ ∗ o T W ∗ ) , where W ∗ = ( I ( C ∗ = 1 ) , … , I ( C ∗ = k ∗ ) ) T ⊗ Z W^*=(I(C^*=1),\ldots,I(C^*=k^*))^{\mathsf{T}}\otimes Z W ∗ = ( I ( C ∗ = 1 ) , … , I ( C ∗ = k ∗ ) ) T ⊗ Z . Assume that lim n → ∞ P ( G ^ = G o ) = 1 \lim_{n\to\infty}P(\widehat{G}=G^o)=1 lim n → ∞ P ( G = G o ) = 1 for all candidate values k ≥ 1 k\geq1 k ≥ 1 under consideration. This condition is not necessarily guaranteed by assumptions A4 and A7 when k ≠ k ∗ k\ne k^* k = k ∗ . Under this condition, the following theorem establishes the consistency of k ^ \widehat{k} k to k ∗ k^* k ∗ .
Theorem 3. Under the condition that lim n → ∞ P ( G ^ = G o ) = 1 \lim_{n\to\infty}P(\widehat{G}=G^o)=1 lim n → ∞ P ( G = G o ) = 1 for all k ≥ 1 k\geq1 k ≥ 1 and assumptions A1–A3, A5, and A6,
k ^ → p k ∗ as n ⟶ ∞ . \widehat{k}\xrightarrow{p}k^*\text{ as }n\longrightarrow\infty. k p k ∗ as n ⟶ ∞.
Proof. See Appendix G.
Optimal Classification and Semiparametric Efficient Estimation
Under the class-specific index density model (6) , we improve the SP estimation by leveraging SDR-based class membership probabilities.
Estimation of SDR-based Class Membership Probabilities
We characterize the conditional distribution of C C C given X X X through SDR of the covariate space,
P ( C = c ∣ X = x ) = π c ( A d T x ) , ( x , c ) ∈ X × C , (15) P(C=c\mid X=x)=\pi_c(A_d^{\mathsf{T}}x),\qquad(x,c)\in\mathcal{X}\times\mathcal{C}, \tag*{(15)} P ( C = c ∣ X = x ) = π c ( A d T x ) , ( x , c ) ∈ X × C , (15)
where A d A_d A d is a p × d p\times d p × d full rank matrix, and { π c ( ν ) } c = 1 k \{\pi_c(\nu)\}_{c=1}^{k} { π c ( ν ) } c = 1 k are the unknown functions satisfying 0 ≤ π c ( ν ) ≤ 1 0\le\pi_c(\nu)\le1 0 ≤ π c ( ν ) ≤ 1 and ∑ c = 1 k π c ( ν ) = 1 \sum_{c=1}^{k}\pi_c(\nu)=1 ∑ c = 1 k π c ( ν ) = 1 for all ν ∈ R d \nu\in\mathbb{R}^{d} ν ∈ R d . Under the consistent latent class recovery condition in (12) and the identity I ( C i = c ) = I ( i ∈ G c o ) I(C_i=c)=I(i\in\mathcal{G}^{o}_{c}) I ( C i = c ) = I ( i ∈ G c o ) for all i = 1 , … , n i=1,\ldots,n i = 1 , … , n and c ∈ C c\in\mathcal{C} c ∈ C , we estimate π c ( ν ) \pi_c(\nu) π c ( ν ) , given a basis matrix A d A_d A d , by
π ^ c , h d ( ν ; A d ) = ∑ i = 1 n I ( i ∈ G ^ c ) K r , h d ( A d T X i − ν ) ∑ i = 1 n K r , h d ( A d T X i − ν ) , c ∈ C , (16) \widehat{\pi}_{c,h_d}(\nu;A_d)=\frac{\sum_{i=1}^{n}I(i\in\widehat{\mathcal{G}}_{c})K_{r,h_d}(A_d^{\mathsf{T}}X_i-\nu)}{\sum_{i=1}^{n}K_{r,h_d}(A_d^{\mathsf{T}}X_i-\nu)},\qquad c\in\mathcal{C}, \tag*{(16)} π c , h d ( ν ; A d ) = ∑ i = 1 n K r , h d ( A d T X i − ν ) ∑ i = 1 n I ( i ∈ G c ) K r , h d ( A d T X i − ν ) , c ∈ C , (16)
where K r , h d ( ν ) = ∏ j = 1 d K r , h d ( ν j ) K_{r,h_d}(\nu)=\prod_{j=1}^{d}K_{r,h_d}(\nu_j) K r , h d ( ν ) = ∏ j = 1 d K r , h d ( ν j ) and K r , h d ( v ) = K r ( v / h d ) / h d K_{r,h_d}(v)=K_r(v/h_d)/h_d K r , h d ( v ) = K r ( v / h d ) / h d is a kernel function with bandwidth h d > 0 h_d>0 h d > 0 . The function K r ( ⋅ ) K_r(\cdot) K r ( ⋅ ) is an r r r th-order kernel that is a symmetric, compactly supported, and twice continuously differentiable, satisfying ∫ K r ( 2 ) ( u ) d u = 0 \int K_r^{(2)}(u)du=0 ∫ K r ( 2 ) ( u ) d u = 0 , with r > max { ( 4 + d ) / 4 , ( 1 + d ) / 2 } r>\max\{(4+d)/4,(1+d)/2\} r > max {( 4 + d ) /4 , ( 1 + d ) /2 } .
Because an r r r th-order kernel need not be nonnegative, the estimator π ^ c , h d ( ν ; A d ) \widehat{\pi}_{c,h_d}(\nu;A_d) π c , h d ( ν ; A d ) may take values outside [ 0 , 1 ] [0,1] [ 0 , 1 ] . We therefore truncate the estimator to [ 0 , 1 ] [0,1] [ 0 , 1 ] and set π ^ c , h d ( ν ; A d ) = 1 / k \widehat{\pi}_{c,h_d}(\nu;A_d)=1/k π c , h d ( ν ; A d ) = 1/ k for all c ∈ C c\in\mathcal{C} c ∈ C whenever all truncated estimates are zero at a given ν \nu ν . For fixed d d d and h d h_d h d , the pseudo-least squares estimator A ^ d \widehat{A}_d A d is defined as the minimizer of the pseudo sum of squares
PSS C M ( A d , h d ) = 1 2 ∑ i = 1 n ∑ c = 1 k − 1 ( I ( i ∈ G ^ c ) − π ^ c , h d − i ( A d T X i ; A d ) ) 2 , (17) \operatorname{PSS}_{\mathrm{CM}}(A_d,h_d)=\frac{1}{2}\sum_{i=1}^{n}\sum_{c=1}^{k-1}\left(I(i\in\widehat{\mathcal{G}}_c)-\widehat{\pi}^{-i}_{c,h_d}(A_d^{\mathsf{T}}X_i;A_d)\right)^2, \tag*{(17)} PSS CM ( A d , h d ) = 2 1 i = 1 ∑ n c = 1 ∑ k − 1 ( I ( i ∈ G c ) − π c , h d − i ( A d T X i ; A d ) ) 2 , (17)
where π ^ c , h d − i ( ν ; A d ) \widehat{\pi}^{-i}_{c,h_d}(\nu;A_d) π c , h d − i ( ν ; A d ) denotes the leave-one-out version of π ^ c , h d ( ν ; A d ) \widehat{\pi}_{c,h_d}(\nu;A_d) π c , h d ( ν ; A d ) , i = 1 , … , n i=1,\ldots,n i = 1 , … , n . Using A ^ d \widehat{A}_d A d and a bandwidth selector h ~ d ∈ arg min h d PSS C M ( A ^ d , h d ) \widetilde{h}_d\in\arg\min_{h_d}\operatorname{PSS}_{\mathrm{CM}}(\widehat{A}_d,h_d) h d ∈ arg min h d PSS CM ( A d , h d ) , which attains the rate O p ( n − 1 / ( 2 r + d ) ) O_p(n^{-1/(2r+d)}) O p ( n − 1/ ( 2 r + d ) ) , the structural dimension d o d^o d o is estimated by a minimizer d ^ \widehat{d} d of the criterion:
PSS C M ( d ) = 2 n PSS C M ( A ^ d , h ~ d ) + log n 2 n 2 r / ( 2 r + d ) for d ≥ 1. (18) \operatorname{PSS}_{\mathrm{CM}}(d)=\frac{2}{n}\operatorname{PSS}_{\mathrm{CM}}(\widehat{A}_d,\widetilde{h}_d)+\frac{\log n}{2n^{2r/(2r+d)}}\qquad\text{for }d\ge1. \tag*{(18)} PSS CM ( d ) = n 2 PSS CM ( A d , h d ) + 2 n 2 r / ( 2 r + d ) log n for d ≥ 1. (18)
Based on the selected structural dimension, the resulting estimator of the central subspace (CS) matrix A o A^o A o is denoted by A ^ \widehat{A} A . In practice, covariate independence of class membership probabilities can be assessed by comparing PSS C M ( 1 ) \operatorname{PSS}_{\mathrm{CM}}(1) PSS CM ( 1 ) with
1 2 n ∑ i = 1 n ∑ c = 1 k − 1 ( I ( i ∈ G ^ c ) − 1 n ∑ j = 1 n I ( j ∈ G ^ c ) ) 2 + log n 2 n . \frac{1}{2n}\sum_{i=1}^{n}\sum_{c=1}^{k-1}\left(I(i\in\widehat{\mathcal{G}}_c)-\frac{1}{n}\sum_{j=1}^{n}I(j\in\widehat{\mathcal{G}}_c)\right)^2+\frac{\log n}{2n}. 2 n 1 i = 1 ∑ n c = 1 ∑ k − 1 ( I ( i ∈ G c ) − n 1 j = 1 ∑ n I ( j ∈ G c ) ) 2 + 2 n log n .
Let vec ( ⋅ ) \operatorname{vec}(\cdot) vec ( ⋅ ) denote the column-stacking operator, P A P_A P A the projection matrix onto the column space of a generic matrix A A A , and π ^ c [ t ] ( x , A d ⊤ x ) \widehat{\pi}_c^{[t]}(x,A_d^\top x) π c [ t ] ( x , A d ⊤ x ) the uniformly convergent function of ∂ vec ( A d ∗ ) t π ^ c , h d ( A d ⊤ x ; A d ) \partial^t_{\operatorname{vec}(A_{d*})}\widehat{\pi}_{c,h_d}(A_d^\top x;A_d) ∂ vec ( A d ∗ ) t π c , h d ( A d ⊤ x ; A d ) for t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 and c ∈ C c\in\mathcal{C} c ∈ C , with π ^ c [ 0 ] ( x , A o ⊤ x ) = π c ( A o ⊤ x ) \widehat{\pi}_c^{[0]}(x,A^{o\top}x)=\pi_c(A^{o\top}x) π c [ 0 ] ( x , A o ⊤ x ) = π c ( A o ⊤ x ) . The following theorem establishes the consistency of d ^ \widehat d d to d o d^o d o and the asymptotic normality of A ^ ∗ . \widehat A_*. A ∗ .
Theorem 4. Under assumptions A1–A7 and B1–B4,
d ^ → p d o and n vec ( P A ^ ∗ − P A ∗ o ) → d N ( 0 , Σ ) as n → ∞ , \widehat d \xrightarrow{p} d^o \text{ and } \sqrt{n}\operatorname{vec}(P_{\widehat A_*}-P_{A_*^o}) \xrightarrow{d} N(0,\Sigma) \text{ as } n\to\infty, d p d o and n vec ( P A ∗ − P A ∗ o ) d N ( 0 , Σ ) as n → ∞ , where
Σ = E [ ( vec ( ( I p − P A ∗ o ) ( I d o , V 0 ⊤ ) ⊤ ) ( A o ⊤ A o ) − 1 A o ⊤ + A o ( A o ⊤ A o ) − 1 ( I d o , V 0 ⊤ ) ( I p − P A ∗ o ) ) ⊗ 2 ] , \Sigma=\mathbb{E}\left[\left(\operatorname{vec}\left((I_p-P_{A_*^o})(I_{d^o},V_0^\top)^\top\right)(A^{o\top}A^o)^{-1}A^{o\top}+A^o(A^{o\top}A^o)^{-1}(I_{d^o},V_0^\top)(I_p-P_{A_*^o})\right)^{\otimes2}\right], Σ = E [ ( vec ( ( I p − P A ∗ o ) ( I d o , V 0 ⊤ ) ⊤ ) ( A o ⊤ A o ) − 1 A o ⊤ + A o ( A o ⊤ A o ) − 1 ( I d o , V 0 ⊤ ) ( I p − P A ∗ o ) ) ⊗ 2 ] , with
vec ( V 0 ) = ( ∑ c = 1 k − 1 E [ ( π c [ 1 ] ( X , A o ⊤ X ) ) ⊗ 2 ] ) − 1 ( ∑ c = 1 k − 1 E [ ( I ( C = c ) − π c ( X , A o ⊤ X ) ) π c [ 1 ] ( X , A o ⊤ X ) ] ) . \operatorname{vec}(V_0)=\left(\sum_{c=1}^{k-1}\mathbb{E}\left[\left(\pi_c^{[1]}(X,A^{o\top}X)\right)^{\otimes2}\right]\right)^{-1}\left(\sum_{c=1}^{k-1}\mathbb{E}\left[(I(C=c)-\pi_c(X,A^{o\top}X))\pi_c^{[1]}(X,A^{o\top}X)\right]\right). vec ( V 0 ) = ( c = 1 ∑ k − 1 E [ ( π c [ 1 ] ( X , A o ⊤ X ) ) ⊗ 2 ] ) − 1 ( c = 1 ∑ k − 1 E [ ( I ( C = c ) − π c ( X , A o ⊤ X )) π c [ 1 ] ( X , A o ⊤ X ) ] ) .
Proof. See Theorem 2 in Teng and Chiang [31 ] .
The proof strategy of Theorem 1 can be extended to the present setting and used to strengthen existing theoretical developments by relaxing the compact-support condition on X X X and the lower-bound condition on the probability density of the CS directions. In particular, the SDR estimator A ^ ∗ \widehat A_* A ∗ is asymptotically equivalent to the infeasible oracle estimator constructed from the true class indicators I ( C i = c ) I(C_i=c) I ( C i = c ) for i = 1 , … , n i=1,\ldots,n i = 1 , … , n and c ∈ C c\in\mathcal{C} c ∈ C .
Bayes Classification Rule and Pseudo-Maximum Likelihood Estimation
We estimate the probability density function f ( u ) f(u) f ( u ) in the class-specific index density model (6) using the SP-estimated latent class partition G ^ \widehat{\mathcal{G}} G and the SP estimator γ ^ \widehat{\gamma} γ :
f ^ ξ ( u ) = 1 n ∑ i = 1 n K ξ ( Y i − ∑ c = 1 k I ( i ∈ G ^ c ) m ^ h ^ − i ( γ ^ c ⊤ Z i ; G ^ , γ ^ ) − u ) . (19) \widehat f_{\xi}(u)=\frac{1}{n}\sum_{i=1}^{n}K_{\xi}\left(Y_i-\sum_{c=1}^{k}I(i\in\widehat{\mathcal{G}}_c)\widehat m_{\widehat h}^{-i}(\widehat\gamma_c^\top Z_i;\widehat{\mathcal{G}},\widehat\gamma)-u\right). \tag*{(19)} f ξ ( u ) = n 1 i = 1 ∑ n K ξ ( Y i − c = 1 ∑ k I ( i ∈ G c ) m h − i ( γ c ⊤ Z i ; G , γ ) − u ) . (19)
The bandwidth ξ \xi ξ is selected as the minimizer of the cross-validation criterion of Bowman [1 ] ,
CV ( h ) = 1 n ∑ i = 1 n [ ∫ ( f ^ ξ − i ( u ) ) 2 d u − 2 f ^ ξ − i ( Y i − ∑ c = 1 k I ( i ∈ G ^ c ) m ^ h ^ − i ( γ ^ c ⊤ Z i ; G ^ , γ ^ ) ) ] , (20) \operatorname{CV}(h)=\frac{1}{n}\sum_{i=1}^{n}\left[\int\left(\widehat f_{\xi}^{-i}(u)\right)^2du-2\widehat f_{\xi}^{-i}\left(Y_i-\sum_{c=1}^{k}I(i\in\widehat{\mathcal{G}}_c)\widehat m_{\widehat h}^{-i}(\widehat\gamma_c^\top Z_i;\widehat{\mathcal{G}},\widehat\gamma)\right)\right], \tag*{(20)} CV ( h ) = n 1 i = 1 ∑ n [ ∫ ( f ξ − i ( u ) ) 2 d u − 2 f ξ − i ( Y i − c = 1 ∑ k I ( i ∈ G c ) m h − i ( γ c ⊤ Z i ; G , γ ) ) ] , (20)
where f ^ ξ − i ( u ) \widehat f_{\xi}^{-i}(u) f ξ − i ( u ) denotes the leave-one out version of f ^ ξ ( u ) \widehat f_{\xi}(u) f ξ ( u ) . Using the estimators f ^ ξ ~ ( y − m ^ h ~ ( γ ^ ⊤ z ; G ^ , γ ^ ) ) \widehat f_{\widetilde{\xi}}(y-\widehat m_{\widetilde h}(\widehat\gamma^\top z;\widehat{\mathcal{G}},\widehat\gamma)) f ξ ( y − m h ( γ ⊤ z ; G , γ )) and π ^ c , h ^ d ( A ^ ⊤ X i ; A ^ ) \widehat\pi_{c,\widehat h_d}(\widehat A^\top X_i;\widehat A) π c , h d ( A ⊤ X i ; A ) for ( y , x , c ) ∈ Y × X × C (y,x,c)\in\mathcal{Y}\times\mathcal{X}\times\mathcal{C} ( y , x , c ) ∈ Y × X × C , we construct the Bayes classification rule by assigning a new observation ( X 0 , Y 0 ) (X_0,Y_0) ( X 0 , Y 0 ) to Class c c c whenever
f ~ ξ ~ ( Y 0 − m ^ h ( γ ^ c T Z 0 ; G ^ , γ ^ ) ) π ~ c , h ^ ( A ^ T X 0 ; A ^ ) = max 1 ≤ ℓ ≤ k f ~ ξ ~ ( Y 0 − m ^ h ( γ ^ ℓ T Z 0 ; G ^ , γ ^ ) ) π ~ ℓ , h ^ ( A ^ T X 0 ; A ^ ) . (21) \widetilde{f}_{\widetilde{\xi}}\left(Y_0-\widehat{m}_{h}\left(\widehat{\gamma}_{c}^{\mathsf{T}}Z_0;\widehat{G},\widehat{\gamma}\right)\right)\widetilde{\pi}_{c,\widehat{h}}\left(\widehat{A}^{\mathsf{T}}X_0;\widehat{A}\right)=\max_{1\leq\ell\leq k}\widetilde{f}_{\widetilde{\xi}}\left(Y_0-\widehat{m}_{h}\left(\widehat{\gamma}_{\ell}^{\mathsf{T}}Z_0;\widehat{G},\widehat{\gamma}\right)\right)\widetilde{\pi}_{\ell,\widehat{h}}\left(\widehat{A}^{\mathsf{T}}X_0;\widehat{A}\right). \tag*{(21)} f ξ ( Y 0 − m h ( γ c T Z 0 ; G , γ ) ) π c , h ( A T X 0 ; A ) = 1 ≤ ℓ ≤ k max f ξ ( Y 0 − m h ( γ ℓ T Z 0 ; G , γ ) ) π ℓ , h ( A T X 0 ; A ) . (21)
The optimality result for Bayes assignment in Teng and Chiang [31 ] implies that this classifier achieves the maximal asymptotic probability of correct assignment. In practice, the same classification rule can be used to construct the estimated latent class partition G ~ = ( G ~ 1 , … , G ~ k ) \widetilde{G}=(\widetilde{G}_1,\ldots,\widetilde{G}_k) G = ( G 1 , … , G k ) of G o G^o G o by assigning observation i i i to G ~ c \widetilde{G}_c G c whenever condition (21) holds after replacing ( X 0 , Y 0 ) (X_0,Y_0) ( X 0 , Y 0 ) with ( X i , Y i ) (X_i,Y_i) ( X i , Y i ) .
The Bayes-rule-based (BR) estimated latent class partition is not guaranteed to achieve consistent latent class recovery, unlike the SP-estimated latent class partition in (12) . Nevertheless, it exhibits comparable—and in some cases superior—finite-sample performance when the component probability densities exhibit moderate overlap.
Given a parameter vector γ \gamma γ , we estimate f ( u ) f(u) f ( u ) using the SP-estimated latent class partition G ^ \widehat{G} G and the bandwidth selectors h ^ \widehat{h} h and ξ ^ = n 3 / 80 ξ ~ \widehat{\xi}=n^{3/80}\widetilde{\xi} ξ = n 3/80 ξ , where ξ ~ = O p ( n − 1 / 5 ) \widetilde{\xi}=O_p(n^{-1/5}) ξ = O p ( n − 1/5 ) . The bandwidth ξ ^ \widehat{\xi} ξ is selected to guarantee the n \sqrt{n} n -consistency of the resulting estimator, defined as
f ~ ξ ^ ( u ; γ ) = 1 n ∑ i = 1 n K ξ ^ ( Y i − ∑ c = 1 k I ( i ∈ G ^ c ) m ^ h ^ ( γ ^ c T Z i ; G ^ , γ ) − u ) . (22) \widetilde{f}_{\widehat{\xi}}(u;\gamma)=\frac{1}{n}\sum_{i=1}^{n}K_{\widehat{\xi}}\left(Y_i-\sum_{c=1}^{k}I(i\in\widehat{G}_c)\widehat{m}_{\widehat{h}}\left(\widehat{\gamma}_c^{\mathsf{T}}Z_i;\widehat{G},\gamma\right)-u\right). \tag*{(22)} f ξ ( u ; γ ) = n 1 i = 1 ∑ n K ξ ( Y i − c = 1 ∑ k I ( i ∈ G c ) m h ( γ c T Z i ; G , γ ) − u ) . (22)
The pseudo-maximum likelihood (PML) estimator γ ~ \widetilde{\gamma} γ of γ o \gamma^o γ o is obtained by maximizing the log-pseudo-likelihood
p ℓ ( γ ) = ∑ i = 1 n log ( f ~ ξ ^ ( Y i − ∑ c = 1 k I ( i ∈ G ^ c ) m ^ h ^ ( γ ^ c T Z i ; G ^ , γ ) ; γ ) ) . (23) p\ell(\gamma)=\sum_{i=1}^{n}\log\left(\widetilde{f}_{\widehat{\xi}}\left(Y_i-\sum_{c=1}^{k}I(i\in\widehat{G}_c)\widehat{m}_{\widehat{h}}\left(\widehat{\gamma}_c^{\mathsf{T}}Z_i;\widehat{G},\gamma\right);\gamma\right)\right). \tag*{(23)} p ℓ ( γ ) = i = 1 ∑ n log ( f ξ ( Y i − c = 1 ∑ k I ( i ∈ G c ) m h ( γ c T Z i ; G , γ ) ; γ ) ) . (23)
This optimization is initialized at the SP estimator γ ^ \widehat{\gamma} γ .
The proposed pseudo-maximum likelihood estimator γ ~ \widetilde{\gamma} γ is consistent and asymptotically normal.
Theorem 5. Under assumptions A1–A7 and C1–C4,
γ ~ ∗ ⟶ p γ ∗ o and n ( γ ~ ∗ − γ ∗ o ) ⟶ d N ( 0 , V − 1 ) as n ⟶ ∞ , \widetilde{\gamma}_{*}\mathrel{\mathop{\longrightarrow}^{p}}\gamma_{*}^{o}\ \text{and}\ \sqrt{n}\left(\widetilde{\gamma}_{*}-\gamma_{*}^{o}\right)\mathrel{\mathop{\longrightarrow}^{d}}N(0,V^{-1})\ \text{as }n\longrightarrow\infty, γ ∗ ⟶ p γ ∗ o and n ( γ ∗ − γ ∗ o ) ⟶ d N ( 0 , V − 1 ) as n ⟶ ∞ , where V = var ( f [ 1 ] ( W , Y − m ( γ o T W ) ; γ o ) / f [ 0 ] ( W , Y − m ( γ o T W ) ; γ o ) ) V=\operatorname{var}\left(f^{[1]}(W,Y-m(\gamma^{o\mathsf{T}}W);\gamma^o)/f^{[0]}(W,Y-m(\gamma^{o\mathsf{T}}W);\gamma^o)\right) V = var ( f [ 1 ] ( W , Y − m ( γ o T W ) ; γ o ) / f [ 0 ] ( W , Y − m ( γ o T W ) ; γ o ) ) , with f [ t ] ( w , u ; γ ) f^{[t]}(w,u;\gamma) f [ t ] ( w , u ; γ ) , t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 , defined in Lemma S3.
Proof. See Appendix H.
Under the assumptions of Theorem 5, the consistent latent class recovery condition in (12) establishes that the PML estimator based on G ^ \widehat{\mathcal{G}} G is asymptotically equivalent to its infeasible oracle counterpart constructed from G o \mathcal{G}^{\mathrm{o}} G o . The asymptotic variance attains the semiparametric efficiency bound; see the proof of Theorem 2 in Ma and Zhu (2013) for a related argument. Moreover, the asymptotic variance matrix of γ ~ ∗ \widetilde{\gamma}_* γ ∗ is estimated by V ~ − 1 \widetilde{V}^{-1} V − 1 , with
V ~ = 1 n ∑ i = 1 n ( ∂ γ ∗ f ~ ξ ( Y i − ∑ c = 1 k I ( i ∈ G ^ c ) m ~ h ( γ ~ c T Z i ; G ^ , γ ~ ) ; γ ~ ) f ~ ξ ( Y i − ∑ c = 1 k I ( i ∈ G ^ c ) m ~ h ( γ ~ c T Z i ; G ^ , γ ~ ) ; γ ~ ) ) ⊗ 2 \widetilde{V}=\frac{1}{n}\sum_{i=1}^{n}\left(\frac{\partial_{\gamma_*}\widetilde{f}_{\xi}\left(Y_i-\sum_{c=1}^{k}I\left(i\in\widehat{\mathcal{G}}_c\right)\widetilde{m}_{h}\left(\widetilde{\gamma}_c^{\mathsf{T}}Z_i;\widehat{\mathcal{G}},\widetilde{\gamma}\right);\widetilde{\gamma}\right)}{\widetilde{f}_{\xi}\left(Y_i-\sum_{c=1}^{k}I\left(i\in\widehat{\mathcal{G}}_c\right)\widetilde{m}_{h}\left(\widetilde{\gamma}_c^{\mathsf{T}}Z_i;\widehat{\mathcal{G}},\widetilde{\gamma}\right);\widetilde{\gamma}\right)}\right)^{\otimes2} V = n 1 i = 1 ∑ n f ξ ( Y i − ∑ c = 1 k I ( i ∈ G c ) m h ( γ c T Z i ; G , γ ) ; γ ) ∂ γ ∗ f ξ ( Y i − ∑ c = 1 k I ( i ∈ G c ) m h ( γ c T Z i ; G , γ ) ; γ ) ⊗ 2
Remark 3. Within the semiparametric generalized linear model with an unspecified response function in (7) , the parameters ( γ , ϕ ) (\gamma,\phi) ( γ , ϕ ) can be estimated by the maximizer ( γ ^ , ϕ ^ ) (\widehat{\gamma},\widehat{\phi}) ( γ , ϕ ) of
p ℓ 0 ( γ , ϕ ) = ∑ i = 1 n ( ∑ c = 1 k I ( i ∈ G ^ c ) Y i θ ( m ^ h ( γ ^ c T Z i ; G ^ , γ ) ) − b ( θ ( m ^ h ( γ ^ c T Z i ; G ^ , γ ) ) ) ϕ ) + c ( Y i , ϕ ) . p\ell_0(\gamma,\phi)=\sum_{i=1}^{n}\left(\sum_{c=1}^{k}I\left(i\in\widehat{\mathcal{G}}_c\right)\frac{Y_i\theta\left(\widehat{m}_{h}\left(\widehat{\gamma}_c^{\mathsf{T}}Z_i;\widehat{\mathcal{G}},\gamma\right)\right)-b\left(\theta\left(\widehat{m}_{h}\left(\widehat{\gamma}_c^{\mathsf{T}}Z_i;\widehat{\mathcal{G}},\gamma\right)\right)\right)}{\phi}\right)+c(Y_i,\phi). p ℓ 0 ( γ , ϕ ) = i = 1 ∑ n c = 1 ∑ k I ( i ∈ G c ) ϕ Y i θ ( m h ( γ c T Z i ; G , γ ) ) − b ( θ ( m h ( γ c T Z i ; G , γ ) ) ) + c ( Y i , ϕ ) .
Modified Criteria for Selecting the Number of Classes
As alternatives to SPIC0 _0 0 in (14) , we consider two criteria from Teng and Chiang [31 ] , replacing their bivariate function estimator with the estimator F ~ ξ ~ ( u ) = ∫ − ∞ u f ~ ξ ~ ( v ; γ ~ ) d v \widetilde{F}_{\widetilde{\xi}}(u)=\int_{-\infty}^{u}\widetilde{f}_{\widetilde{\xi}}(v;\widetilde{\gamma})\,dv F ξ ( u ) = ∫ − ∞ u f ξ ( v ; γ ) d v of the distribution function F ( u ) = ∫ − ∞ u f ( v ) d v F(u)=\int_{-\infty}^{u}f(v)\,dv F ( u ) = ∫ − ∞ u f ( v ) d v , to select the number of classes:
SPIC 1 ( k ) = 1 n ∑ i = 1 n ∫ ( I ( Y i ≤ y ) − F ~ ξ ~ − i ( e i y ) ) 2 d F ~ Y ( y ) + k log n 2 n 4 5 (24) \operatorname{SPIC}_1(k)=\frac{1}{n}\sum_{i=1}^{n}\int\left(I(Y_i\le y)-\widetilde{F}_{\widetilde{\xi}}^{-i}(e_{iy})\right)^2d\widetilde{F}_Y(y)+\frac{k\log n}{2n^{\frac{4}{5}}} \tag*{(24)} SPIC 1 ( k ) = n 1 i = 1 ∑ n ∫ ( I ( Y i ≤ y ) − F ξ − i ( e i y ) ) 2 d F Y ( y ) + 2 n 5 4 k log n (24)
and
SPIC 2 ( k ) = − 1 n ∑ i = 1 n ∫ [ I ( Y i ≤ y ) log ( F ~ ξ ~ − i ( e i y ) ) + I ( Y i > y ) log ( 1 − F ~ ξ ~ − i ( e i y ) ) ] d F ~ Y ( y ) + k log n n 4 5 , k ≥ 1 (25) \operatorname{SPIC}_2(k)=-\frac{1}{n}\sum_{i=1}^{n}\int\left[I(Y_i\le y)\log\left(\widetilde{F}_{\widetilde{\xi}}^{-i}(e_{iy})\right)+I(Y_i>y)\log\left(1-\widetilde{F}_{\widetilde{\xi}}^{-i}(e_{iy})\right)\right]d\widetilde{F}_Y(y)+\frac{k\log n}{n^{\frac{4}{5}}},\quad k\ge1 \tag*{(25)} SPIC 2 ( k ) = − n 1 i = 1 ∑ n ∫ [ I ( Y i ≤ y ) log ( F ξ − i ( e i y ) ) + I ( Y i > y ) log ( 1 − F ξ − i ( e i y ) ) ] d F Y ( y ) + n 5 4 k log n , k ≥ 1 (25)
where F ~ Y ( y ) = ∑ i = 1 n I ( Y i ≤ y ) / n \widetilde{F}_Y(y)=\sum_{i=1}^{n}I(Y_i\le y)/n F Y ( y ) = ∑ i = 1 n I ( Y i ≤ y ) / n , e i y = y − ∑ c = 1 k I ( i ∈ G ^ c ) m ^ h ( γ ~ c T Z i ; γ ~ ) e_{iy}=y-\sum_{c=1}^{k}I(i\in\widehat{\mathcal{G}}_c)\widehat{m}_h(\widetilde{\gamma}_c^{\mathsf{T}}Z_i;\widetilde{\gamma}) e i y = y − ∑ c = 1 k I ( i ∈ G c ) m h ( γ c T Z i ; γ ) , and F ~ ξ ~ − i ( e i y ) = F ~ ξ ~ − i ( e i y ) + ( − 1 ) s I ( F ~ ξ ~ − i ( e i y ) = s ) \widetilde{F}_{\widetilde{\xi}}^{-i}(e_{iy})=\widetilde{F}_{\widetilde{\xi}}^{-i}(e_{iy})+(-1)^sI\left(\widetilde{F}_{\widetilde{\xi}}^{-i}(e_{iy})=s\right) F ξ − i ( e i y ) = F ξ − i ( e i y ) + ( − 1 ) s I ( F ξ − i ( e i y ) = s ) for i = 1 , … , n i=1,\ldots,n i = 1 , … , n and s = 0 , 1 s=0,1 s = 0 , 1 . The penalty term in (24) and (25) corresponds to the uniform convergence rate sup u ∣ F ~ ξ ~ ( u ) − F ( u ) ∣ = O p ( n − 2 / 5 ) \sup_u\left|\widetilde{F}_{\widetilde{\xi}}(u)-F(u)\right|=O_p(n^{-2/5}) sup u F ξ ( u ) − F ( u ) = O p ( n − 2/5 ) . Under the conditions of Theorem 3 and the uniform convergence rate of F ~ ξ ~ ( u ) \widetilde{F}_{\widetilde{\xi}}(u) F ξ ( u ) to F ( u ) F(u) F ( u ) , we establish the consistency of the minimizers k ^ 1 \widehat{k}_1 k 1 of SPIC 1 \operatorname{SPIC}_1 SPIC 1 and k ^ 2 \widehat{k}_2 k 2 of SPIC 2 \operatorname{SPIC}_2 SPIC 2 for estimating k ∗ k^* k ∗ , using arguments similar to those in the proof of Theorem 3 in Teng and Chiang [31 ] .
Remark 4. Alternatives to the goodness-of-fit terms in (24) and (25) include the log pseudo-likelihood and the log pseudo-marginal likelihood. Numerical experiments show that the criterion based on the log pseudo-likelihood overestimates the number of classes, even for large sample sizes. Furthermore, the log pseudo-marginal likelihood induces complex dependencies through SDR-based class membership probabilities, precluding a tractable theoretical adjustment for overfitting.
Monte Carlo Simulations
The simulation settings were motivated by transaction-level data from a North American supermarket chain and clinical data from ACTG 175. We conducted simulation studies to assess the finite-sample performance of the proposed methods for the two models (1) and (5) , along with benchmark methods for class-specific linear regression, using sample sizes n ∈ { 200 , 400 , 600 } n \in\{200, 400, 600\} n ∈ { 200 , 400 , 600 } and 500 replications per scenario. The comparisons evaluate the performance of the implemented procedures under linear versus nonlinear regression specifications and normal versus nonnormal error distributions; they are not intended as a comprehensive comparison of latent class regression methods.
Simulation Design
Data were generated from the CSIM (5) with the response function m ( v ) = ( v + 5 ) 3 / 5 m(v) = (v + 5)^{3/5} m ( v ) = ( v + 5 ) 3/5 and the number of classes set to k = 2 k = 2 k = 2 or k = 3 k = 3 k = 3 . The error term U U U was independently drawn from one of three distributions: N ( 0 , σ 2 ) N(0, \sigma^{2}) N ( 0 , σ 2 ) , Logistic( 0 , 3 σ / π ) (0, \sqrt{3}\sigma/\pi) ( 0 , 3 σ / π ) , or Gamma( 4 , σ / 2 ) − 2 σ (4, \sigma/2) - 2\sigma ( 4 , σ /2 ) − 2 σ , with σ = 0.15 \sigma= 0.15 σ = 0.15 for k = 2 k = 2 k = 2 and σ = 0.10 \sigma= 0.10 σ = 0.10 for k = 3 k = 3 k = 3 .
The class-invariant index coefficient vector was fixed at γ ( 2 ) = ( 1 , − 1 , 1 , − 1 , 1 ) T / 5 \gamma_{(2)} = (1, -1, 1, -1, 1)^{\mathsf{T}}/\sqrt{5} γ ( 2 ) = ( 1 , − 1 , 1 , − 1 , 1 ) T / 5 across all scenarios. The first four components of Z ( 2 ) Z_{(2)} Z ( 2 ) were sampled from a multivariate normal distribution N 4 ( 0 4 , 0.8 I 4 + 0.2141 4 1 4 T ) N_{4}(0_{4}, 0.8I_{4} + 0.2141_{4}1_{4}^{\mathsf{T}}) N 4 ( 0 4 , 0.8 I 4 + 0.214 1 4 1 4 T ) , while the fifth component was independently sampled from { − 0.5 , 0.5 } \{-0.5, 0.5\} { − 0.5 , 0.5 } with equal probability. The class-specific index coefficient vector was set to γ ( 1 ) = ( 0 , 2.5 ) T \gamma_{(1)} = (0, 2.5)^{\mathsf{T}} γ ( 1 ) = ( 0 , 2.5 ) T with Z ( 1 ) = 1 Z_{(1)} = 1 Z ( 1 ) = 1 for k = 2 k = 2 k = 2 , and to γ ( 1 ) = ( 0 , 0.1 , 0.1 , 1.5 , 0.3 , 1 , 3 , 0.3 , 1 ) T \gamma_{(1)} = (0, 0.1, 0.1, 1.5, 0.3, 1, 3, 0.3, 1)^{\mathsf{T}} γ ( 1 ) = ( 0 , 0.1 , 0.1 , 1.5 , 0.3 , 1 , 3 , 0.3 , 1 ) T , with Z ( 1 ) Z_{(1)} Z ( 1 ) drawn uniformly from { ( 1 , 0 , 0 ) T , ( 1 , 1 , 0 ) T , ( 1 , 0 , 1 ) T } \{(1, 0, 0)^{\mathsf{T}}, (1, 1, 0)^{\mathsf{T}}, (1, 0, 1)^{\mathsf{T}}\} {( 1 , 0 , 0 ) T , ( 1 , 1 , 0 ) T , ( 1 , 0 , 1 ) T } , for k = 3 k = 3 k = 3 . The latent class membership probabilities were specified as
π ( 1 ∣ x ) = 0.25 + 2.5 ϕ ( 2.8 α 1 T x ) 1 + ϕ ( 2.8 α 1 T x ) , with α 1 = ( 1 , 0 , 0 , 0 , 0.29 ) T , for k = 2 , and \pi(1 \mid x) = 0.25 + \frac{2.5\phi(2.8\alpha_1^{\mathsf{T}}x)}{1 + \phi(2.8\alpha_1^{\mathsf{T}}x)}, \text{ with } \alpha_1 = (1, 0, 0, 0, 0.29)^{\mathsf{T}}, \text{ for } k = 2, \text{ and} π ( 1 ∣ x ) = 0.25 + 1 + ϕ ( 2.8 α 1 T x ) 2.5 ϕ ( 2.8 α 1 T x ) , with α 1 = ( 1 , 0 , 0 , 0 , 0.29 ) T , for k = 2 , and
π ( 1 ∣ x ) = 0.15 + 1.5 ϕ ( 0.1 α 1 T x ) 1 + ϕ ( 0.1 α 1 T x ) and π ( 2 ∣ x ) = 0.3 , with α 1 = ( 1 , 1 , 35 , 0 , 0 , 0 , 10 ) T , for k = 3 , \pi(1 \mid x) = 0.15 + \frac{1.5\phi(0.1\alpha_1^{\mathsf{T}}x)}{1 + \phi(0.1\alpha_1^{\mathsf{T}}x)} \text{ and } \pi(2 \mid x) = 0.3, \text{ with } \alpha_1 = (1, 1, 35, 0, 0, 0, 10)^{\mathsf{T}}, \text{ for } k = 3, π ( 1 ∣ x ) = 0.15 + 1 + ϕ ( 0.1 α 1 T x ) 1.5 ϕ ( 0.1 α 1 T x ) and π ( 2 ∣ x ) = 0.3 , with α 1 = ( 1 , 1 , 35 , 0 , 0 , 0 , 10 ) T , for k = 3 ,
where ϕ ( u ) \phi(u) ϕ ( u ) denotes the standard normal density.
Performance of Estimation, Class Recovery, and Class Number Selection
Agreement between the realized latent class partition and each estimated latent class partition was quantified using the Rand index [26 ] . Table 1 reports the RI values for the SP- and BR-estimated latent class partitions when the number of classes is known. Both estimators perform well across the examined settings; however, when the distributional structure is correctly specified, the BR-estimated latent class partition more accurately recovers the realized latent class partition. Accuracy improves for both estimated latent class partitions as the sample size increases. However, distinct patterns emerge: the performance gap narrows for k = 2 k = 2 k = 2 , whereas the advantage of the BR-estimated latent class partition becomes apparent for k = 3 k = 3 k = 3 . These patterns persist across the three error distributions considered.
Table 1. Means (standard deviations) of 500 RI values for the SP- and BR-estimated latent class partitions under various model settings.
Estimation performance for a generic parameter vector θ ∘ \theta^{\circ} θ ∘ was evaluated using the normalized root squared error (RSE): ( θ ˇ − θ ∘ ) T ( θ ˇ − θ ∘ ) / ∣ θ ∘ ∣ \sqrt{(\check{\theta} - \theta^{\circ})^{\mathsf{T}}(\check{\theta} - \theta^{\circ})/|\theta^{\circ}|} ( θ ˇ − θ ∘ ) T ( θ ˇ − θ ∘ ) /∣ θ ∘ ∣ , where θ ˇ \check{\theta} θ ˇ denotes the corresponding estimator. Table 2 compares the RSEs of the SP and PML estimators with those of their respective oracle counterparts when the number of classes is known. Across all simulation settings, the PML estimator and its corresponding oracle estimator achieve lower RSEs than their SP counterparts, respectively, reflecting efficiency gains from leveraging the underlying distributional structure. As the sample size increases, the RSEs of both SP and PML estimators decrease and converge to those of their oracle counterparts. This pattern confirms that the proposed estimators achieve near-oracle efficiency in large samples. Notably, the deviations from the oracle estimators are smaller for k = 2 k = 2 k = 2 than for k = 3 k = 3 k = 3 , reflecting the superior latent class recovery accuracy reported in Table 1 .
Table 2. Means of 500 RSEs for SP and PML estimates of the class index coefficients, along with the corresponding oracles (ORCL1 _1 1 and ORCL2 _2 2 ), under various model settings.
Table S2 and Table S3 report the performance of the proposed methods relative to the SP method for class-specific linear regression (SPL _{\mathrm{L}} L method), which circumvents the computational complexity of the pairwise-fusion-penalty method (Ma and Huang, 2017; Ma et al., 2020), and the mixture maximum-likelihood method for class-specific linear regression (MML method) of Jordan and Jacobs [11 ] , with normally distributed errors and logistic class membership probabilities. Estimation performance for the subject-specific conditional means was evaluated using the normalized RSE: ∑ i = 1 n ( m ~ ( β ~ i T Z i ) − m ( β i ∘ T Z i ) ) 2 / n \sqrt{\sum_{i=1}^{n}\left(\widetilde{m}(\widetilde{\beta}_{i}^{\mathsf{T}} Z_i)-m(\beta_{i}^{\circ\mathsf{T}} Z_i)\right)^2/n} ∑ i = 1 n ( m ( β i T Z i ) − m ( β i ∘ T Z i ) ) 2 / n , where m ~ ( v ) \widetilde{m}(v) m ( v ) denotes either a prespecified or an estimated response function, and β ~ 1 , … , β ~ n \widetilde{\beta}_1,\ldots,\widetilde{\beta}_n β 1 , … , β n are estimators of subject index coefficient vectors. For k = 2 k = 2 k = 2 , the SPL _{\mathrm{L}} L method demonstrates competitive performance in both RI and RSE, suggesting that a linear approximation adequately captures the underlying nonlinear response. However, its limitations become evident when k = 3 k = 3 k = 3 : performance of latent class recovery is substantially inferior for n ≥ 400 n \ge400 n ≥ 400 , and estimation of subject-specific conditional means exhibits larger RSE across the sample sizes considered. In contrast, the MML method generally produces lower RI values and higher RSEs, reflecting reduced accuracy in both latent class recovery and estimation.
Table S2. Means (standard deviations) of 500 RI values for the SPL- and MML-estimated latent class partitions under various model settings.
Table S3. Means of 500 RSEs of conditional mean function estimates from various methods under various model settings.
Table 3 summarizes the performance of three semiparametric information criteria in selecting the number of classes. For k = 2 k = 2 k = 2 , all criteria correctly select the true number of classes in almost all replications across all experimental settings. In the more challenging scenario with k = 3 k = 3 k = 3 , underestimation occurs at n = 200 n = 200 n = 200 , most notably for SPIC1 _1 1 . Selection accuracy increases with sample size, with all criteria exhibiting comparable performance by n = 600 n = 600 n = 600 . Overall, the proposed criterion SPIC0 _0 0 , which does not rely on the underlying distributional structure, performs at least as well as the modified criteria SPIC1 _1 1 and SPIC2 _2 2 .
Table 3. Means (proportions of correct selection) of 500 estimated numbers of classes from various selection criteria under various model settings.
To evaluate the performance of the criteria associated with the SPL and MML methods for selecting the number of classes, Table S4 reports results for the best-performing modified Bayesian information criterion (BIC1 _1 1 ) among those proposed by Ma and Huang (2017) and Ma et al. (2020), as well as the Bayesian-type information criterion (BIC2 _2 2 ) of Khalili [12 ] . While the SPL method demonstrates robustness to model misspecification, BIC1 _1 1 systematically selects the maximum allowable number of classes, indicating that its penalty term inadequately accounts for the structural complexity of the nonlinear response. In contrast, BIC2 _2 2 avoids this severe overestimation but remains biased. For k = 2 k = 2 k = 2 , this criterion exhibits a modest upward bias in the selected number of classes, whereas for k = 3 k = 3 k = 3 , it underestimates the number of classes for n ≤ 400 n \le400 n ≤ 400 and exhibits instability at n = 600 n = 600 n = 600 . The inferior performance of these parametric information criteria underscores the necessity of the proposed semiparametric information criteria when structural or distributional assumptions are violated.
Table S4. Means (proportions of correct selection) of 500 estimated number of classes across selection criteria under various model settings.
Applications
We demonstrate the proposed methodology through two illustrative applications in personalized marketing and patient stratification. To ensure comparability across methods, all continuous covariates were standardized to have mean zero and variance one prior to model fitting. Statistical significance was evaluated at α = 0.05 \alpha= 0.05 α = 0.05 .
Application in Customer Segmentation
Customer segmentation is essential for targeted marketing, allowing firms to identify latent groups with homogeneous purchasing behavior. We apply the proposed methodology to high-frequency transaction data from a major North American supermarket chain collected between 2017 and 2020. The dataset comprises transaction records linked to detailed product information. To ensure temporal stability and avoid COVID-19 distortions, we restrict the analysis to loyal 2018 customers completing at least 13 transactions in both H1 and H2. Following industry practice, multiple transactions occurring within the same week are aggregated into a single purchasing occasion to capture routine shopping behavior better. After this consolidation, the analysis sample comprises 1,906 customers, with 28,376 and 29,111 transactions in H1 and H2, respectively.
We consider an empirical application that estimates the log-transformed purchase amount in H2 (p a h 2 pah2 p ah 2 ) using behavioral features measured in H1. The covariates include the log-transformed purchase amount (p a h 1 pah1 p ah 1 ), log-transformed average basket size (a b s abs ab s ), the number of distinct product categories purchased among the top 30 categories (c a t e cate c a t e ), an indicator for high-intensity natural food consumption (n f nf n f ; 1 if the purchase amount is at or above the median and 0 otherwise), and an indicator for baby-product purchases (b a b y baby bab y ; 1 if at least one such purchase occurred and 0 otherwise).
The CSIM was specified with class-specific intercepts and class-invariant index coefficients for the covariates, identified during the initialization stage, to account for latent heterogeneity in customer spending. The number of classes was selected using the information criteria SPIC0 _0 0 , SPIC1 _1 1 , and SPIC2 _2 2 , all of which favored a four-class solution. The BR-estimated latent class partition coincided with the SP-estimated latent class partition; Classes 1–4 contained 702, 301, 582, and 321 customers, respectively. The corresponding class-specific mean purchase amounts in H2 (491.20, 714.27, 359.60, and 254.96) indicate that the identified classes capture well-separated levels of customer spending.
Using the PML method, we obtain the estimates of the CSIM parameters, with standard errors in parentheses. The estimated class-specific intercepts for Classes 1–4 are 0, 1.26 (0.032), -0.97 (0.022), and -1.98 (0.032), respectively. The coefficient of p a h 1 pah1 p ah 1 is fixed at 1 for all four classes. The remaining coefficients are common across classes, with estimates of 0.19 (0.013) for a b s abs ab s , -0.03 (0.008) for c a t e cate c a t e , -0.05 (0.015) for n f nf n f , and 0.04 (0.025) for b a b y baby bab y . Standard errors were computed from the asymptotic variance matrix estimator V ~ − 1 \widetilde{V}^{-1} V − 1 described below Theorem 5. The observed significant differences in the class-specific intercept estimates underscore the necessity of the multi-class structure and support the validity of our class number selection criteria. Among the class-invariant index coefficient estimates, p a h 1 pah1 p ah 1 and a b s abs ab s exhibit significant associations with future spending, whereas c a t e cate c a t e and n f nf n f are adversely associated. Figure 1 presents the estimated conditional mean functions given the class-invariant index, which exhibit an approximately increasing relationship.
Figure 1. Scatter plot of log-transformed purchase amounts in H2, overlaid with the estimated conditional mean functions given the class-invariant index.
Based on the estimated number of classes and the corresponding estimated latent class partitions, the structural dimension of the SDR-based class membership probabilities was determined to be one, with Class 4 designated as the reference group. The coefficient estimates (standard errors) of the central subspace direction are 1 for p a h 1 pah1 p ah 1 , 0.59 (0.051) for a b s abs ab s , -1.66 (0.077) for c a t e cate c a t e , 3.32 (0.137) for n f nf n f , and 3.74 (0.131) for b a b y baby bab y . The effects of p a h 1 pah1 p ah 1 , a b s abs ab s , n f nf n f , and b a b y baby bab y are all significant in one direction, while the effect of c a t e cate c a t e is significant in the opposite direction, supporting covariatedependent class membership probabilities. Figure S1 further shows that the logistic model fails to capture the structure of the latent class membership probabilities.
Figure S1. Estimated log-odds of membership in Classes 1–3 relative to Class 4 based on the SDR-based and logistic class membership probabilities.
Based on the near-linear trends observed in the estimated conditional mean functions (Figure 1 ) and the limited flexibility of the logistic class membership probabilities (Figure S1 ), we fitted a class-specific linear model using the S P L SP_{L} S P L method for comparison. Although B I C 1 BIC_{1} B I C 1 suggested 15 classes, simulation results indicated that fewer classes were supported. Accordingly, we restricted the model to four classes, aligning it with the fitted CSIM and facilitating an assessment of class agreement. The estimated class-specific intercepts for Classes 1–4 are 0.38 (0.013), 1.08 (0.017), -0.23 (0.012), and -1.05 (0.017), respectively. The coefficient of p a h 1 pah1 p ah 1 is fixed at 1 for all four classes. The coefficients are common across classes, with estimates (standard errors) of 0.64 (0.013) for p a h 1 pah1 p ah 1 , 0.11 (0.009) for a b s abs ab s , -0.01 (0.008) for c a t e cate c a t e , -0.05 (0.013) for n f nf n f , and 0.04 (0.020) for b a b y baby bab y . The S P L SP_{L} S P L estimates have the same interpretations as the PML estimates, and the corresponding linear mean functions (Figure S2 ) closely match those in Figure 1 . For this dataset, the high RI (0.864) between the S P L SP_{L} S P L - and BR-estimated latent class partitions provides, at most, evidence of the robustness of class-specific linear regression in the presence of mild model misspecification, highlighting the need for a rigorous evaluation of model goodness-of-fit.
Figure S2. Scatter plot of log-transformed purchase amounts in H2, overlaid with estimated linear mean functions of the class-invariant index.
Application in ACTG Study 175
The ACTG Study 175 was a landmark randomized clinical trial that had a lasting impact on the development of antiretroviral treatment strategies. The trial evaluated the comparative efficacy of four antiretroviral regimens in adults infected with HIV-1, all of whom had baseline CD4 T-cell counts between 200 and 500 cells/m m 3 \mathrm{mm}^{3} mm 3 . Our analysis used the ACTG175 dataset available in R, which comprises 2,139 adults from the trial. A detailed description of the study design and primary findings is provided in Hammer, Katzenstein, Hughes, Gundacker, Schooley, Haubrich, Henry, Lederman, Phair, Niu, Hirsch, and Merigan [6 ] .
Our second application of the proposed methodology analyzes log-transformed CD4 cell counts measured at 20 ± 5 20 \pm5 20 ± 5 weeks (c d 4 cd4 c d 4 ) as the response variable. The covariates include treatment indicator (t r tr t r ; 0 = zidovudine monotherapy, 1 = all other regimens combined), log-transformed baseline CD8 cell counts (c d 8 cd8 c d 8 ), age (a g e age a g e ), weight (w e i g h t weight w e i g h t ), Karnofsky performance score (k s ks k s ), and gender (g e n d e r gender g e n d er ; 0 = female, 1 = male). To enhance numerical stability, individuals with extreme values in either the response variable or covariates were excluded, yielding a final sample of 2,054 adult participants.
The CSIM was specified with class-specific intercepts and class-specific non-zidovudine treatment effects, identified during the initialization stage, to account for heterogeneity in treatment effects across latent classes. In this analysis, S P I C 0 \mathrm{SPIC}_{0} SPIC 0 selected a five-class solution, whereas S P I C 1 \mathrm{SPIC}_{1} SPIC 1 and S P I C 2 \mathrm{SPIC}_{2} SPIC 2 favored a four-class configuration. The five-class model exhibited statistically significant differences in the estimated intercepts and treatment coefficients across classes, supporting k = 5 k = 5 k = 5 as a reliable and plausible representation of latent treatment–response heterogeneity. The BR-estimated latent class partition coincided with the SP-estimated latent class partition. Under both estimated latent class partitions, Classes 1–5 contained 663, 295, 482, 185, and 429 adults, respectively.
Using the PML method, we obtain estimates of the CSIM parameters, with standard errors in parentheses. The estimated class-specific intercepts for Classes 1–5 are 0, -29.37 (0.637), -10.80 (0.171), -53.53 (0.125), and 12.52 (0.590), respectively. The coefficient of c d 8 cd8 c d 8 is fixed at 1 across all five classes. The coefficient of t r tr t r varies across classes, with estimates of 3.41 (0.227), -10.59 (0.502), -8.52 (0.264), -25.37 (0.272), and 63.03 (0.219) for Classes 1–5, respectively. The remaining coefficients are shared across classes, with estimates of -0.85 (0.056) for a g e age a g e , 0.16 (0.053) for w e i g h t weight w e i g h t , 0.89 (0.081) for k s ks k s , and -1.04 (0.161) for g e n d e r gender g e n d er . The class-specific index coefficient estimates indicate that Class 5 has the highest estimated CD4 cell counts across all treatment conditions, followed in descending order by Classes 1, 3, 2, and 4. The non-zidovudine treatment effects exhibit significant heterogeneity across subgroups, with effects in one direction for Classes 2, 3, and 4 and in the opposite direction for Classes 1 and 5. Among the class-invariant index coefficient estimates, c d 8 cd8 c d 8 , w e i g h t weight w e i g h t , and k s ks k s show significant associations with c d 4 cd4 c d 4 , while a g e age a g e and g e n d e r gender g e n d er exhibit significant adverse associations. Figure 2 presents the estimated conditional mean functions given the class-invariant index, which exhibit a distinct non-monotonic pattern, indicating that class-specific linear regression is inadequate. Additionally, the unreported S P L \mathrm{SP}_{\mathrm{L}} SP L estimates from a class-specific linear model fail to capture the complex treatment–response heterogeneity across subgroups revealed by the fitted CSIM.
Figure 2. Scatter plot of log-transformed CD4 cell counts, overlaid with the estimated conditional mean functions given the class-invariant index (solid line: zidovudine monotherapy; dashed line: non-zidovudine therapy).
With Class 5 designated as the reference group, the structural dimension in the SDR-based class membership probabilities was identified as one. The coefficient estimates (standard errors) of the central subspace direction are 1 for a g e age a g e , 0.66 (0.164) for t r tr t r , -0.19 (0.075) for c d 8 cd8 c d 8 , 0.32 (0.082) for w e i g h t weight w e i g h t , 0.50 (0.097) for k s ks k s , and 0.00 (0.135) for g e n d e r gender g e n d er . The coefficient estimates indicate that t r tr t r , a g e age a g e , and k s ks k s have significant contributions to the class membership probabilities, whereas c d 8 cd8 c d 8 contributes in the opposite direction. These findings provide compelling evidence of covariate-dependent class membership probabilities. Figure S3 further highlights the limitations of the logistic model in capturing the underlying structure of these probabilities.
Figure S3. Estimated log-odds of membership in Classes 1–4 relative to Class 5 based on the SDR-based and logistic class membership probabilities.
Concluding Remarks and Future Challenges
We introduce semiparametric class-specific index regression that characterizes the conditional mean of a response variable and accommodates a general distributional specification. The Bayes classification rule asymptotically maximizes the probability of correct assignment, and the proposed PML estimator achieves semiparametric efficiency. Building on the subject-specific formulation of the CSIM, we develop a SP estimation procedure implemented via a computationally efficient algorithm that avoids high-dimensional matrix operations and updates auxiliary parameters in explicit closed-form at each ADMM iteration. In our initialization procedure, the index coefficients of the CSIM and the latent class partition of the individual-level index set are estimated by minimizing the pseudo-least-squares criterion in (10) for each candidate number of classes. This strategy promotes rapid and stable convergence of the SP estimation procedure. Simulation studies and empirical data analyses demonstrate that our method outperforms existing alternatives in estimation precision and latent class recovery accuracy.
A central challenge lies in the joint identification and characterization of both the response function and the underlying distributional form within the proposed framework. To evaluate the adequacy of the assumed parametric specification for the response function, our estimation procedure—together with its parametric analogue—provides a theoretically grounded and rigorous framework for model assessment. Although the class-specific index distribution model of Teng and Chiang [31 ] encompasses ours as a special case when the induced response function is nondegenerate, simulation evidence demonstrates that our method achieves superior performance. In particular, the pseudo-maximum likelihood estimators exhibit poor finite-sample properties, largely due to the sparsity inherent in estimating the bivariate probability density function. Nevertheless, the pseudo-least-integrated-squares procedure, combined with our distributional estimator, remains a valuable instrument for evaluating the adequacy of the assumed distributional specification.
Weekly purchase amounts and longitudinally measured CD4 cell counts were analyzed for customer segmentation and HIV/AIDS patient stratification, respectively. The CSIM can be extended to accommodate longitudinal outcomes with class-specific, time-varying covariate effects. A modified pseudo-least-squares criterion, augmented with an ℓ 1 \ell_{1} ℓ 1 separation penalty, naturally accounts for correlations among repeated measurements within individuals. By remaining distribution-free, the proposed semiparametric information criterion can be adapted to consistently select the number of classes in longitudinal settings. Collectively, the methodology introduced in this work establishes a rigorous framework for inference in complex clustered and longitudinal data structures.
Summary of Notation
Table A.
SP Estimation Procedure
Initialization Strategy
To implement the SP estimation procedure, we first fit a semiparametric single-index mean model, m ( γ 1 ⊤ z ) m(\gamma_1^{\top}z) m ( γ 1 ⊤ z ) , and a linear mean model, γ 1 ⊤ z \gamma_1^{\top}z γ 1 ⊤ z . Based on these fits, we compute the residuals { Y i − m ^ h − i ( γ ^ 1 ⊤ Z i ; γ ^ 1 ) } i = 1 n \{Y_i-\widehat{m}_h^{-i}(\widehat{\gamma}_1^{\top}Z_i;\widehat{\gamma}_1)\}_{i=1}^n { Y i − m h − i ( γ 1 ⊤ Z i ; γ 1 ) } i = 1 n , where m ^ h ( γ ^ 1 ⊤ z ; γ ^ 1 ) \widehat{m}_h(\widehat{\gamma}_1^{\top}z;\widehat{\gamma}_1) m h ( γ 1 ⊤ z ; γ 1 ) is obtained by minimizing the pseudo-least squares criterion in (10) with k = 1 k=1 k = 1 , and { Y i − γ ^ ⊤ Z i } i = 1 n \{Y_i-\widehat{\gamma}^{\top}Z_i\}_{i=1}^n { Y i − γ ⊤ Z i } i = 1 n , where γ ^ \widehat{\gamma} γ is the ordinary least squares estimator. We then apply a distribution-free clustering procedure, such as k k k -means or hierarchical clustering, to each set of residuals, yielding initial estimated latent class partitions for a given number of classes c = 2 , … , k ‾ c=2,\ldots,\overline{k} c = 2 , … , k , where k ‾ \overline{k} k may be chosen, for instance, as ⌊ n ⌋ \lfloor\sqrt{n}\rfloor ⌊ n ⌋ . Substituting these estimators into (10) yields two pseudo-least squares estimators; we define γ ‾ c \overline{\gamma}^{c} γ c as the one achieving the smaller objective value. The corresponding estimated latent class partition and subject coefficient estimators are denoted by G ‾ c = { G ‾ 1 c , … , G ‾ c c } \overline{\mathcal{G}}^{c}=\{\overline{G}^{c}_1,\ldots,\overline{G}^{c}_c\} G c = { G 1 c , … , G c c } and β ‾ c \overline{\beta}^{c} β c , respectively.
A new estimated latent class partition G ~ c \widetilde{\mathcal{G}}^{c} G c is obtained by assigning each observation i i i to G ~ P S ~ i ( c ) c = min { 1 ≤ ℓ ≤ c } P S i ( ℓ ) \widetilde{G}^{c}_{\widetilde{\mathrm{PS}}_i(c)}=\min_{\{1\leq\ell\leq c\}}\mathrm{PS}_i(\ell) G PS i ( c ) c = min { 1 ≤ ℓ ≤ c } PS i ( ℓ ) , where P S i ( ℓ ) = ( Y i − m ^ h − i ( γ ‾ ℓ c ⊤ Z i ; G ‾ c , γ ‾ c ) ) 2 \mathrm{PS}_i(\ell)=\left(Y_i-\widehat{m}^{-i}_h(\overline{\gamma}^{c\top}_{\ell}Z_i;\overline{\mathcal{G}}^{c},\overline{\gamma}^{c})\right)^2 PS i ( ℓ ) = ( Y i − m h − i ( γ ℓ c ⊤ Z i ; G c , γ c ) ) 2 , i = 1 , … , n i=1,\ldots,n i = 1 , … , n , ℓ = 1 , … , c \ell=1,\ldots,c ℓ = 1 , … , c . The updated pair ( G ~ c , γ ~ c ) (\widetilde{\mathcal{G}}^{c},\widetilde{\gamma}^{c}) ( G c , γ c ) replaces ( G ‾ c , γ ‾ c ) (\overline{\mathcal{G}}^{c},\overline{\gamma}^{c}) ( G c , γ c ) if P S S ( γ ~ c , G ~ c ) < P S S ( γ ‾ c , G ‾ c ) \mathrm{PSS}(\widetilde{\gamma}^{c},\widetilde{\mathcal{G}}^{c})<\mathrm{PSS}(\overline{\gamma}^{c},\overline{\mathcal{G}}^{c}) PSS ( γ c , G c ) < PSS ( γ c , G c ) ; otherwise, ( G ~ c , γ ~ c ) (\widetilde{\mathcal{G}}^{c},\widetilde{\gamma}^{c}) ( G c , γ c ) is set to ( G ‾ c , γ ‾ c ) (\overline{\mathcal{G}}^{c},\overline{\gamma}^{c}) ( G c , γ c ) , where γ ~ c \widetilde{\gamma}^{c} γ c is obtained by minimizing P S S ( γ c , G ~ c ) \mathrm{PSS}(\gamma^{c},\widetilde{\mathcal{G}}^{c}) PSS ( γ c , G c ) with respect to γ c \gamma^{c} γ c . This procedure is iterated until the objective function attains its minimum or a specified convergence threshold is met.
For each c = 1 , … , k ˉ c=1,\ldots,\bar{k} c = 1 , … , k ˉ , let β ~ c \tilde{\beta}^{c} β ~ c denote the corresponding subject coefficient estimator obtained from initializer ( G ~ c , γ ~ c ) (\tilde{\mathcal{G}}^{c},\tilde{\gamma}^{c}) ( G ~ c , γ ~ c ) . The reported experiments employ the initialization strategy described above; they do not establish whether alternative initial partitions would lead to equivalent final estimates or partitions.
Computational Algorithm
The optimization problem in (9) can be rewritten as
Minimize 1 2 ∑ i = 1 n ( Y i − m ^ h − i ( β i T Z i ; β ) ) 2 + λ ∑ i = 1 n min 1 ≤ c ≤ k ∥ η i c ∥ 1 subject to β ( 1 ) i − γ ( 1 ) c = η i c , i = 1 , … , n , c = 1 , … , k . (26) \begin{aligned}
\operatorname{Minimize}\quad&\frac{1}{2}\sum_{i=1}^{n}\left(Y_i-\widehat{m}_{h}^{-i}(\beta_i^{\mathsf{T}}Z_i;\beta)\right)^2+\lambda\sum_{i=1}^{n}\min_{1\leq c\leq k}\|\eta_{ic}\|_1\\
\text{subject to}\quad&\beta_{(1)i}-\gamma_{(1)c}=\eta_{ic},\quad i=1,\ldots,n,\ c=1,\ldots,k. \tag*{(26)}
\end{aligned} Minimize subject to 2 1 i = 1 ∑ n ( Y i − m h − i ( β i T Z i ; β ) ) 2 + λ i = 1 ∑ n 1 ≤ c ≤ k min ∥ η i c ∥ 1 β ( 1 ) i − γ ( 1 ) c = η i c , i = 1 , … , n , c = 1 , … , k . (26)
Because the objective function is separable in ( β , γ ) (\beta,\gamma) ( β , γ ) and { η i c } \{\eta_{ic}\} { η i c } , we minimize the augmented Lagrangian via ADMM:
1 2 ∑ i = 1 n ( Y i − m ^ h − i ( β i T Z i ; β ) ) 2 + λ ∑ i = 1 n min 1 ≤ c ≤ k ∥ η i c ∥ 1 + ∑ i = 1 n ∑ c = 1 k ν i c T ( β ( 1 ) i − γ ( 1 ) c − η i c ) + 1 2 ∑ i = 1 n ∑ c = 1 k ∥ β ( 1 ) i − γ ( 1 ) c − η i c ∥ 2 , (27) \begin{aligned}
&\frac{1}{2}\sum_{i=1}^{n}\left(Y_i-\widehat{m}_{h}^{-i}(\beta_i^{\mathsf{T}}Z_i;\beta)\right)^2+\lambda\sum_{i=1}^{n}\min_{1\leq c\leq k}\|\eta_{ic}\|_1\\
&\quad+\sum_{i=1}^{n}\sum_{c=1}^{k}\nu_{ic}^{\mathsf{T}}(\beta_{(1)i}-\gamma_{(1)c}-\eta_{ic})+\frac{1}{2}\sum_{i=1}^{n}\sum_{c=1}^{k}\|\beta_{(1)i}-\gamma_{(1)c}-\eta_{ic}\|^2, \tag*{(27)}
\end{aligned} 2 1 i = 1 ∑ n ( Y i − m h − i ( β i T Z i ; β ) ) 2 + λ i = 1 ∑ n 1 ≤ c ≤ k min ∥ η i c ∥ 1 + i = 1 ∑ n c = 1 ∑ k ν i c T ( β ( 1 ) i − γ ( 1 ) c − η i c ) + 2 1 i = 1 ∑ n c = 1 ∑ k ∥ β ( 1 ) i − γ ( 1 ) c − η i c ∥ 2 , (27)
where { ν i c } \{\nu_{ic}\} { ν i c } are the Lagrange multipliers and ∥ ⋅ ∥ \|\cdot\| ∥ ⋅ ∥ denotes the Frobenius norm.
Denote by γ ^ ( 1 ) ( t ) \widehat{\gamma}_{(1)}^{(t)} γ ( 1 ) ( t ) , β ^ ( t ) \widehat{\beta}^{(t)} β ( t ) , { η ^ i c ( t ) } \{\widehat{\eta}_{ic}^{(t)}\} { η i c ( t ) } , and { ν ^ i c ( t ) } \{\widehat{\nu}_{ic}^{(t)}\} { ν i c ( t ) } the ADMM iterates at iteration t ≥ 0 t\geq0 t ≥ 0 for a fixed λ \lambda λ , with initial values η ^ i c ( 0 ) = β ~ ( 1 ) i − γ ~ ( 1 ) c \widehat{\eta}_{ic}^{(0)}=\widetilde{\beta}_{(1)i}-\widetilde{\gamma}_{(1)c} η i c ( 0 ) = β ( 1 ) i − γ ( 1 ) c and ν ^ i c ( 0 ) = 0 \widehat{\nu}_{ic}^{(0)}=0 ν i c ( 0 ) = 0 for i = 1 , … , n i=1,\ldots,n i = 1 , … , n and c = 1 , … , k c=1,\ldots,k c = 1 , … , k . Here, β ~ \widetilde{\beta} β is chosen from { β ~ c : c = k , … , ⌊ n / log n ⌋ } \{\widetilde{\beta}^{c}:c=k,\ldots,\lfloor\sqrt{n/\log n}\rfloor\} { β c : c = k , … , ⌊ n / log n ⌋} and γ ~ \widetilde{\gamma} γ is set to γ ~ k \widetilde{\gamma}^{k} γ k . The shrinkage sequence λ 1 < ⋯ < λ J \lambda_1<\cdots<\lambda_J λ 1 < ⋯ < λ J is selected over the interval [ λ 1 , λ J ] [\lambda_1,\lambda_J] [ λ 1 , λ J ] , where λ 1 = n α min { ( i , j , c ) : ( β ~ i − γ ~ c ) j ≠ 0 } ∣ ( β ~ i − γ ~ c ) j ∣ \lambda_1=n^\alpha\min_{\{(i,j,c):(\widetilde{\beta}_i-\widetilde{\gamma}_c)_j\ne0\}}|(\widetilde{\beta}_i-\widetilde{\gamma}_c)_j| λ 1 = n α min {( i , j , c ) : ( β i − γ c ) j = 0 } ∣ ( β i − γ c ) j ∣ and λ J = n α max ( i , j , c ) ∣ ( β ~ i − γ ~ c ) j ∣ \lambda_J=n^\alpha\max_{(i,j,c)}|(\widetilde{\beta}_i-\widetilde{\gamma}_c)_j| λ J = n α max ( i , j , c ) ∣ ( β i − γ c ) j ∣ . In our implementation, we set J = 10 J=10 J = 10 and specify α \alpha α so that n α n^\alpha n α is close to 1. We compute the SP estimator via the following iterative scheme for t > 0 t>0 t > 0 :
((β , γ ( 1 ) \beta,\gamma_{(1)} β , γ ( 1 ) )-minimization) The updates β ^ ( t + 1 ) \widehat{\beta}^{(t+1)} β ( t + 1 ) and γ ^ ( t + 1 ) \widehat{\gamma}^{(t+1)} γ ( t + 1 ) are obtained via the iterations for s ≥ 0 s\geq0 s ≥ 0 :
β ^ ( 1 ) i ( s + 1 ) = β ^ ( 1 ) i ( s ) − ( H 1 i ( β ( 1 ) [ i ] ( s ) , β ( 2 ) ( s ) ) + k I q ) − 1 [ S 1 i ( β ( 1 ) [ i ] ( s ) , β ( 2 ) ( s ) ) + ( ∑ c = 1 k β ^ ( 1 ) i ( s ) − γ ^ ( 1 ) c ( s ) − η ^ i c ( t ) + ν ^ i c ( t ) ) ] , \widehat{\beta}_{(1)i}^{(s+1)}=\widehat{\beta}_{(1)i}^{(s)}-\left(H_{1i}\left(\beta_{(1)}^{[i](s)},\beta_{(2)}^{(s)}\right)+kI_q\right)^{-1}\left[S_{1i}\left(\beta_{(1)}^{[i](s)},\beta_{(2)}^{(s)}\right)+\left(\sum_{c=1}^{k}\widehat{\beta}_{(1)i}^{(s)}-\widehat{\gamma}_{(1)c}^{(s)}-\widehat{\eta}_{ic}^{(t)}+\widehat{\nu}_{ic}^{(t)}\right)\right], β ( 1 ) i ( s + 1 ) = β ( 1 ) i ( s ) − ( H 1 i ( β ( 1 ) [ i ] ( s ) , β ( 2 ) ( s ) ) + k I q ) − 1 [ S 1 i ( β ( 1 ) [ i ] ( s ) , β ( 2 ) ( s ) ) + ( c = 1 ∑ k β ( 1 ) i ( s ) − γ ( 1 ) c ( s ) − η i c ( t ) + ν i c ( t ) ) ] ,
β ^ ( 2 ) ( s + 1 ) = β ^ ( 2 ) ( s ) − ( H 2 ( β ^ ( 1 ) ( s + 1 ) , β ^ ( 2 ) ( s ) ) ) − 1 S 2 ( β ^ ( 1 ) ( s + 1 ) , β ^ ( 2 ) ( s ) ) , γ ^ ( 1 ) c ( s + 1 ) = 1 n ∑ i = 1 n ( β ^ ( 1 ) i ( s + 1 ) − η ^ i c ( t ) + ν ^ i c ( t ) ) , \widehat{\beta}_{(2)}^{(s+1)}=\widehat{\beta}_{(2)}^{(s)}-\left(H_2\left(\widehat{\beta}_{(1)}^{(s+1)},\widehat{\beta}_{(2)}^{(s)}\right)\right)^{-1}S_2\left(\widehat{\beta}_{(1)}^{(s+1)},\widehat{\beta}_{(2)}^{(s)}\right),\qquad\widehat{\gamma}_{(1)c}^{(s+1)}=\frac{1}{n}\sum_{i=1}^{n}\left(\widehat{\beta}_{(1)i}^{(s+1)}-\widehat{\eta}_{ic}^{(t)}+\widehat{\nu}_{ic}^{(t)}\right), β ( 2 ) ( s + 1 ) = β ( 2 ) ( s ) − ( H 2 ( β ( 1 ) ( s + 1 ) , β ( 2 ) ( s ) ) ) − 1 S 2 ( β ( 1 ) ( s + 1 ) , β ( 2 ) ( s ) ) , γ ( 1 ) c ( s + 1 ) = n 1 i = 1 ∑ n ( β ( 1 ) i ( s + 1 ) − η i c ( t ) + ν i c ( t ) ) ,
i = 1 , … , n , c = 1 , … , k , (28) i=1,\ldots,n,\ c=1,\ldots,k, \tag*{(28)} i = 1 , … , n , c = 1 , … , k , (28)
where β ( 0 ) = β ^ ( t ) \beta^{(0)}=\widehat{\beta}^{(t)} β ( 0 ) = β ( t ) , β ( 1 ) [ i ] ( s ) = ( β ( 1 ) 1 ( s + 1 ) T , … , β ( 1 ) i − 1 ( s + 1 ) T , β ( 1 ) i ( s ) T , … , β ( 1 ) n ( s ) T ) T \beta_{(1)}^{[i](s)}=\left(\beta_{(1)1}^{(s+1)\mathsf{T}},\ldots,\beta_{(1)i-1}^{(s+1)\mathsf{T}},\beta_{(1)i}^{(s)\mathsf{T}},\ldots,\beta_{(1)n}^{(s)\mathsf{T}}\right)^{\mathsf{T}} β ( 1 ) [ i ] ( s ) = ( β ( 1 ) 1 ( s + 1 ) T , … , β ( 1 ) i − 1 ( s + 1 ) T , β ( 1 ) i ( s ) T , … , β ( 1 ) n ( s ) T ) T with β ( 1 ) [ 1 ] ( s ) = β ( 1 ) ( s ) \beta_{(1)}^{[1](s)}=\beta_{(1)}^{(s)} β ( 1 ) [ 1 ] ( s ) = β ( 1 ) ( s ) , I q I_q I q denotes the q × q q\times q q × q identity matrix,
S 1 i ( β ) = − ∑ j = 1 n ( Y j − m ^ h − j ( β j T Z j ; β ) ) ∂ β ( 1 ) i m ^ h − j ( β j T Z j ; β ) , H 1 i ( β ) = ∑ j = 1 n ( ∂ β ( 1 ) i m ^ h − j ( β j T Z j ; β ) ) ⊗ 2 , S_{1i}(\beta)=-\sum_{j=1}^{n}\left(Y_j-\widehat{m}_{h}^{-j}(\beta_j^{\mathsf{T}}Z_j;\beta)\right)\partial_{\beta_{(1)i}}\widehat{m}_{h}^{-j}(\beta_j^{\mathsf{T}}Z_j;\beta),\qquad H_{1i}(\beta)=\sum_{j=1}^{n}\left(\partial_{\beta_{(1)i}}\widehat{m}_{h}^{-j}(\beta_j^{\mathsf{T}}Z_j;\beta)\right)^{\otimes2}, S 1 i ( β ) = − j = 1 ∑ n ( Y j − m h − j ( β j T Z j ; β ) ) ∂ β ( 1 ) i m h − j ( β j T Z j ; β ) , H 1 i ( β ) = j = 1 ∑ n ( ∂ β ( 1 ) i m h − j ( β j T Z j ; β ) ) ⊗ 2 ,
S 2 ( β ) = − ∑ i = 1 n ( Y i − m ^ h − i ( β i T Z i ; β ) ) ∂ β ( 2 ) m ^ h − i ( β i T Z i ; β ) , H 2 ( β ) = ∑ i = 1 n ( ∂ β ( 2 ) m ^ h − i ( β i T Z i ; β ) ) ⊗ 2 . S_2(\beta)=-\sum_{i=1}^{n}\left(Y_i-\widehat{m}_{h}^{-i}(\beta_i^{\mathsf{T}}Z_i;\beta)\right)\partial_{\beta_{(2)}}\widehat{m}_{h}^{-i}(\beta_i^{\mathsf{T}}Z_i;\beta),\qquad H_2(\beta)=\sum_{i=1}^{n}\left(\partial_{\beta_{(2)}}\widehat{m}_{h}^{-i}(\beta_i^{\mathsf{T}}Z_i;\beta)\right)^{\otimes2}. S 2 ( β ) = − i = 1 ∑ n ( Y i − m h − i ( β i T Z i ; β ) ) ∂ β ( 2 ) m h − i ( β i T Z i ; β ) , H 2 ( β ) = i = 1 ∑ n ( ∂ β ( 2 ) m h − i ( β i T Z i ; β ) ) ⊗ 2 .
η ^ i c ( t + 1 ) = { S λ ( ( β ^ ( 1 ) i ( t + 1 ) − γ ^ ( 1 ) c ( t + 1 ) + ν ^ i c ( t ) ) j ) , c = c 0 , β ^ ( 1 ) i ( t + 1 ) − γ ^ ( 1 ) c ( t + 1 ) + ν ^ i c ( t ) , c ≠ c 0 , (29) \widehat{\eta}_{ic}^{(t+1)}=
\begin{cases}
\mathcal{S}_{\lambda}\left(\left(\widehat{\beta}_{(1)i}^{(t+1)}-\widehat{\gamma}_{(1)c}^{(t+1)}+\widehat{\nu}_{ic}^{(t)}\right)_j\right), & c=c_0,\\
\widehat{\beta}_{(1)i}^{(t+1)}-\widehat{\gamma}_{(1)c}^{(t+1)}+\widehat{\nu}_{ic}^{(t)}, & c\ne c_0,
\end{cases} \tag*{(29)} η i c ( t + 1 ) = ⎩ ⎨ ⎧ S λ ( ( β ( 1 ) i ( t + 1 ) − γ ( 1 ) c ( t + 1 ) + ν i c ( t ) ) j ) , β ( 1 ) i ( t + 1 ) − γ ( 1 ) c ( t + 1 ) + ν i c ( t ) , c = c 0 , c = c 0 , (29)
where S λ ( z ) = sgn ( z ) max ( ∣ z ∣ − λ , 0 ) S_\lambda(z)=\operatorname{sgn}(z)\max(|z|-\lambda,0) S λ ( z ) = sgn ( z ) max ( ∣ z ∣ − λ , 0 ) and c 0 c_0 c 0 is a minimizer of
λ ∥ ( S λ ( ( β ^ ( 1 ) i ( t + 1 ) − γ ^ ( 1 ) c ( t + 1 ) + ν ^ i c ( t ) ) j ) ) j ∥ 1 + 1 2 ∥ ( S λ ( ( β ^ ( 1 ) i ( t + 1 ) − γ ^ ( 1 ) c ( t + 1 ) + ν ^ i c ( t ) ) j ) ) j − ( β ^ ( 1 ) i ( t + 1 ) − γ ^ ( 1 ) c ( t + 1 ) + ν ^ i c ( t ) ) ∥ 2 . \lambda\left\|\left(S_\lambda\left(\left(\widehat{\beta}_{(1)i}^{(t+1)}-\widehat{\gamma}_{(1)c}^{(t+1)}+\widehat{\nu}_{ic}^{(t)}\right)_j\right)\right)_j\right\|_1+\frac{1}{2}\left\|\left(S_\lambda\left(\left(\widehat{\beta}_{(1)i}^{(t+1)}-\widehat{\gamma}_{(1)c}^{(t+1)}+\widehat{\nu}_{ic}^{(t)}\right)_j\right)\right)_j-\left(\widehat{\beta}_{(1)i}^{(t+1)}-\widehat{\gamma}_{(1)c}^{(t+1)}+\widehat{\nu}_{ic}^{(t)}\right)\right\|^2. λ ( S λ ( ( β ( 1 ) i ( t + 1 ) − γ ( 1 ) c ( t + 1 ) + ν i c ( t ) ) j ) ) j 1 + 2 1 ( S λ ( ( β ( 1 ) i ( t + 1 ) − γ ( 1 ) c ( t + 1 ) + ν i c ( t ) ) j ) ) j − ( β ( 1 ) i ( t + 1 ) − γ ( 1 ) c ( t + 1 ) + ν i c ( t ) ) 2 .
ν ^ i c ( t + 1 ) = ν ^ i c ( t ) + β ^ ( 1 ) i ( t + 1 ) − γ ^ ( 1 ) c ( t + 1 ) − η ^ i c ( t + 1 ) . (30) \widehat{\nu}_{ic}^{(t+1)}=\widehat{\nu}_{ic}^{(t)}+\widehat{\beta}_{(1)i}^{(t+1)}-\widehat{\gamma}_{(1)c}^{(t+1)}-\widehat{\eta}_{ic}^{(t+1)}. \tag*{(30)} ν i c ( t + 1 ) = ν i c ( t ) + β ( 1 ) i ( t + 1 ) − γ ( 1 ) c ( t + 1 ) − η i c ( t + 1 ) . (30)
Following Section 3.3 in Boyd et al. [2 ] , we terminate the algorithm once the primal residual ( ∑ i = 1 n ∑ c = 1 k ∥ β ^ ( 1 ) i ( t ) − γ ^ ( 1 ) c ( t ) − η ^ i c ( t ) ∥ 2 ) 1 / 2 \left(\sum_{i=1}^{n}\sum_{c=1}^{k}\left\|\widehat{\beta}_{(1)i}^{(t)}-\widehat{\gamma}_{(1)c}^{(t)}-\widehat{\eta}_{ic}^{(t)}\right\|^2\right)^{1/2} ( ∑ i = 1 n ∑ c = 1 k β ( 1 ) i ( t ) − γ ( 1 ) c ( t ) − η i c ( t ) 2 ) 1/2 and the dual residual ( ∑ i = 1 n ∥ ∑ c = 1 k ( η ^ i c ( t ) − η ^ i c ( t − 1 ) ) ∥ 2 + ∑ c = 1 k ∥ ∑ i = 1 n ( η ^ i c ( t ) − η ^ i c ( t − 1 ) ) ∥ 2 ) 1 / 2 \left(\sum_{i=1}^{n}\left\|\sum_{c=1}^{k}\left(\widehat{\eta}_{ic}^{(t)}-\widehat{\eta}_{ic}^{(t-1)}\right)\right\|^2+\sum_{c=1}^{k}\left\|\sum_{i=1}^{n}\left(\widehat{\eta}_{ic}^{(t)}-\widehat{\eta}_{ic}^{(t-1)}\right)\right\|^2\right)^{1/2} ( ∑ i = 1 n ∑ c = 1 k ( η i c ( t ) − η i c ( t − 1 ) ) 2 + ∑ c = 1 k ∑ i = 1 n ( η i c ( t ) − η i c ( t − 1 ) ) 2 ) 1/2 , fall below prescribed tolerances. Since the η \eta η -update is the exact minimizer of its subproblem and the ( β , γ ) (\beta,\gamma) ( β , γ ) -update satisfies the first-order optimality condition of its subproblem, the output then satisfies the KKT conditions of (B.1) up to these tolerances. This result establishes convergence of the numerical iterates to a stationary point, but does not guarantee that the limiting point is a global minimizer of (9) .
Assumptions
Assumptions for the Oracle Property of the SP Estimator
Let W n \mathcal{W}_n W n denote the collection { W 1 , … , W n } \{W_1,\ldots,W_n\} { W 1 , … , W n } , f γ ( v ) f_\gamma(v) f γ ( v ) the probability density function of γ ⊤ W \gamma^\top W γ ⊤ W , with support V \mathcal{V} V , W ∗ W_* W ∗ the subvector of W W W corresponding γ ∗ \gamma^* γ ∗ , U γ U_\gamma U γ the random quantity Y − m [ 0 ] ( W , γ ⊤ W ) Y-m^{[0]}(W,\gamma^\top W) Y − m [ 0 ] ( W , γ ⊤ W ) , with U ∘ = Y − m ( γ ∘ ⊤ W ) U^\circ=Y-m(\gamma^{\circ\top}W) U ∘ = Y − m ( γ ∘ ⊤ W ) , and U γ n U_{\gamma n} U γ n the collection { U γ 1 , … , U γ n } \{U_{\gamma_1},\ldots,U_{\gamma_n}\} { U γ 1 , … , U γ n } . Define
D γ = min c ≠ ℓ ∥ γ c ∘ − γ ℓ ∘ ∥ , ϕ n = ϕ 0 log n ⋅ n 2 / 5 , N s h ( v ; G ∘ , γ ) = 1 n ∑ i = 1 n Y i ∗ K h ( γ ⊤ W i − v ) , D_\gamma=\min_{c\ne\ell}\|\gamma_c^\circ-\gamma_\ell^\circ\|,\qquad\phi_n=\frac{\phi_0}{\log n\cdot n^{2/5}},\qquad N_{\mathrm{sh}}(v;\mathcal{G}^\circ,\gamma)=\frac{1}{n}\sum_{i=1}^{n}Y_i^*K_h(\gamma^\top W_i-v), D γ = c = ℓ min ∥ γ c ∘ − γ ℓ ∘ ∥ , ϕ n = log n ⋅ n 2/5 ϕ 0 , N sh ( v ; G ∘ , γ ) = n 1 i = 1 ∑ n Y i ∗ K h ( γ ⊤ W i − v ) ,
and
M s , t ( w , v ) = ( − 1 ) t E [ ( m ( γ ⊤ W ) ) s ( W ∗ − w ∗ ) ⊗ t ∣ γ ⊤ W = v ] , s = 0 , 1 , 2 , t = 0 , 1 , 2 , 3 , 4 , M_{s,t}(w,v)=(-1)^t\mathbb{E}\left[\left(m(\gamma^\top W)\right)^s(W_*-w_*)^{\otimes t}\mid\gamma^\top W=v\right],\qquad s=0,1,2,\quad t=0,1,2,3,4, M s , t ( w , v ) = ( − 1 ) t E [ ( m ( γ ⊤ W ) ) s ( W ∗ − w ∗ ) ⊗ t ∣ γ ⊤ W = v ] , s = 0 , 1 , 2 , t = 0 , 1 , 2 , 3 , 4 ,
where ϕ 0 > 0 \phi_0>0 ϕ 0 > 0 is a constant.
The following assumptions are imposed to establish the oracle property of the SP estimator:
A1. h = h 0 n − δ h=h_0n^{-\delta} h = h 0 n − δ for δ ∈ ( 1 / 8 , 1 / 5 ) \delta\in(1/8,1/5) δ ∈ ( 1/8 , 1/5 ) and some constant h 0 > 0 h_0>0 h 0 > 0 .
A2. E [ Y 4 ] < ∞ \mathbb{E}[Y^4]<\infty E [ Y 4 ] < ∞ and each component of X X X is sub-Gaussian.
A3. d v 4 f γ ( v ) d_v^4f_\gamma(v) d v 4 f γ ( v ) and ∂ v t + 2 M s , t ( w , v ) \partial_v^{t+2}M_{s,t}(w,v) ∂ v t + 2 M s , t ( w , v ) are bounded for w ∈ W n w\in\mathcal{W}_n w ∈ W n , and Lipschitz continuous in ( v , γ ∗ ) (v,\gamma_*) ( v , γ ∗ ) for s = 0 , 1 s=0,1 s = 0 , 1 and t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 , with Lipschitz constants independent of w w w .
A4. B n = { β : max 1 ≤ i ≤ n ∥ β i − β i ∘ ∥ 1 ≤ ϕ n } B_n=\{\beta:\max_{1\le i\le n}\|\beta_i-\beta_i^\circ\|_1\le\phi_n\} B n = { β : max 1 ≤ i ≤ n ∥ β i − β i ∘ ∥ 1 ≤ ϕ n } and Γ n = { γ : max c ∈ C ∥ γ c − γ c ∘ ∥ 1 ≤ ϕ n } \Gamma_n=\{\gamma:\max_{c\in\mathcal{C}}\|\gamma_c-\gamma_c^\circ\|_1\le\phi_n\} Γ n = { γ : max c ∈ C ∥ γ c − γ c ∘ ∥ 1 ≤ ϕ n } .
A5. inf γ ∗ ∈ Γ f γ ( γ ⊤ w ) \inf_{\gamma_*\in\Gamma}f_\gamma(\gamma^\top w) inf γ ∗ ∈ Γ f γ ( γ ⊤ w ) is far away from zero for w ∈ W n w\in\mathcal{W}_n w ∈ W n .
A6. V 1 V_1 V 1 and V 2 V_2 V 2 are nonsingular.
A7. D γ > 4 ϕ n D_\gamma>4\phi_n D γ > 4 ϕ n and λ > n α \lambda>n^\alpha λ > n α for some sufficiently small fixed α > 0 \alpha>0 α > 0 .
Assumption A1 imposes a restriction on the bandwidth to ensure the n \sqrt{n} n -consistency of the SP estimator. Assumption A2 imposes a moment condition on the response variable and a sub-Gaussian tail condition on the covariates for the oracle analysis of a subject-specific formulation of the CSIM. The sub-Gaussian condition can be weakened to a finite-moment condition, E [ ∥ X ∥ ι ] < ∞ \mathbb{E}[\|X\|^\iota]<\infty E [ ∥ X ∥ ι ] < ∞ with ι = 21 \iota=21 ι = 21 , provided that ϕ n = O ( n − 2 ( ι + 5 ) / 5 ι ) \phi_n=O(n^{-2(\iota+5)/5\iota}) ϕ n = O ( n − 2 ( ι + 5 ) /5 ι ) and n ϕ n → ∞ \sqrt{n}\phi_n\to\infty n ϕ n → ∞ . Assumption A3 imposes smoothness conditions to guarantee the uniform convergence of the estimated functions to their targets. Assumptions A4 and A7 ensure that the oracle estimator is the unique local minimizer with probability converging to one, while assumptions A5 and A6 guarantee the asymptotic normality of the oracle estimator.
Assumptions for the Asymptotic Properties of the CS Estimator
Let A d \mathcal{A}_d A d denote the compact parameter space of A d ∗ A_{d*} A d ∗ , X n \mathcal{X}_n X n the collection { X 1 , … , X n } \{X_1,\ldots,X_n\} { X 1 , … , X n } , and f A d ( ν ) f_{A_d}(\nu) f A d ( ν ) the probability density function of A d T X A_d^{\mathsf{T}}X A d T X . For each fixed d d d , define
A d ∘ ∈ arg min A d ∈ A d ∑ c = 1 k − 1 E [ ( I ( C = c ) − π c [ 0 ] ( X , A d T X ) ) 2 ] . A_d^{\circ} \in\mathop{\arg\min}_{A_d \in\mathcal{A}_d} \sum_{c=1}^{k-1} \mathbb{E}\left[\left(I(C=c)-\pi_c^{[0]}(X,A_d^{\mathsf{T}}X)\right)^2\right]. A d ∘ ∈ arg min A d ∈ A d c = 1 ∑ k − 1 E [ ( I ( C = c ) − π c [ 0 ] ( X , A d T X ) ) 2 ] .
S A d = ∑ c = 1 k − 1 ( I ( C = c ) − π c [ 0 ] ( X , A d T X ) ) π c [ 1 ] ( X , A d T X ) , S_{A_d}=\sum_{c=1}^{k-1}\left(I(C=c)-\pi_c^{[0]}(X,A_d^{\mathsf{T}}X)\right)\pi_c^{[1]}(X,A_d^{\mathsf{T}}X), S A d = c = 1 ∑ k − 1 ( I ( C = c ) − π c [ 0 ] ( X , A d T X ) ) π c [ 1 ] ( X , A d T X ) ,
and
V A d = ∑ c = 1 k − 1 E [ ( π c [ 1 ] ( X , A d T X ) ) ⊗ 2 − ( I ( C = c ) − π c [ 0 ] ( X , A d T X ) ) π c [ 2 ] ( X , A d T X ) ] . V_{A_d}=\sum_{c=1}^{k-1}\mathbb{E}\left[\left(\pi_c^{[1]}(X,A_d^{\mathsf{T}}X)\right)^{\otimes2}-\left(I(C=c)-\pi_c^{[0]}(X,A_d^{\mathsf{T}}X)\right)\pi_c^{[2]}(X,A_d^{\mathsf{T}}X)\right]. V A d = c = 1 ∑ k − 1 E [ ( π c [ 1 ] ( X , A d T X ) ) ⊗ 2 − ( I ( C = c ) − π c [ 0 ] ( X , A d T X ) ) π c [ 2 ] ( X , A d T X ) ] .
By construction, it follows that, for d = d ∘ , … , p d=d^{\circ},\ldots,p d = d ∘ , … , p , we have E [ S A d ∘ ] = 0 \mathbb{E}[S_{A_d^{\circ}}]=0 E [ S A d ∘ ] = 0 and V A d ∘ = ∑ c = 1 k − 1 E [ ( π c [ 1 ] ( X , A d ∘ T X ) ) ⊗ 2 ] V_{A_d^{\circ}}=\sum_{c=1}^{k-1}\mathbb{E}[(\pi_c^{[1]}(X,A_d^{\circ\mathsf{T}}X))^{\otimes2}] V A d ∘ = ∑ c = 1 k − 1 E [( π c [ 1 ] ( X , A d ∘ T X ) ) ⊗ 2 ] .
The following assumptions are imposed for the asymptotic properties of d ^ \hat d d ^ and A ^ k ^ \widehat{A}_{\hat k} A k ^ :
B1. h d = h 0 n − δ h_d=h_0n^{-\delta} h d = h 0 n − δ for δ ∈ ( 1 / 4 r , 1 / max { 2 d + 2 , d + 4 } ) \delta\in(1/4r,1/\max\{2d+2,d+4\}) δ ∈ ( 1/4 r , 1/ max { 2 d + 2 , d + 4 }) and some constant h 0 > 0 h_0>0 h 0 > 0 .
B2. ∂ ν r + 2 f A d ( ν ) \partial_{\nu}^{r+2}f_{A_d}(\nu) ∂ ν r + 2 f A d ( ν ) and ∂ ν t + r E [ π c ( A T X ) ( X d ∗ − x d ∗ ) ⊗ t ∣ A d T X = ν ] \partial_{\nu}^{t+r}\mathbb{E}[\pi_c(A^{\mathsf{T}}X)(X_{d*}-x_{d*})^{\otimes t}\mid A_d^{\mathsf{T}}X=\nu] ∂ ν t + r E [ π c ( A T X ) ( X d ∗ − x d ∗ ) ⊗ t ∣ A d T X = ν ] are bounded for x ∈ X n x\in\mathcal{X}_n x ∈ X n , and Lipschitz continuous in ( ν , A d ∗ ) (\nu,A_{d*}) ( ν , A d ∗ ) for t = 1 , 2 t=1,2 t = 1 , 2 and c = 1 , … , k − 1 c=1,\ldots,k-1 c = 1 , … , k − 1 , with Lipschitz constants independent of x x x .
B3. inf { A d ∗ ∈ A d } f A d ( A d T x ) \inf_{\{A_{d*}\in\mathcal{A}_d\}}f_{A_d}(A_d^{\mathsf{T}}x) inf { A d ∗ ∈ A d } f A d ( A d T x ) is far away from zero for x ∈ X n x\in\mathcal{X}_n x ∈ X n .
B4. E [ S A d ∘ ⊗ 2 ] \mathbb{E}[S_{A_d^{\circ}}^{\otimes2}] E [ S A d ∘ ⊗ 2 ] and V A d ∘ V_{A_d^{\circ}} V A d ∘ are nonsingular.
Assumption B1 imposes a restriction on the bandwidth to ensure the n \sqrt{n} n -consistency of the CS estimator. Assumption B2 imposes the smoothness conditions to guarantee the uniform convergence of the estimated functions to their targets. Assumptions B3 and B4 ensure the asymptotic normality of the CS estimator.
Assumptions for the Asymptotic Properties of the PML Estimator
Define
Q t 1 , t 2 ( w , u ) = E [ ( − m [ t 1 ] ( W , γ T W ) + m [ t 1 ] ( w , γ T w ) ) ⊗ t 2 ∣ U = u ] , t 1 = 1 , 2 , t 2 = 1 , 2 , 4. Q_{t_1,t_2}(w,u)=\mathbb{E}\left[\left(-m^{[t_1]}(W,\gamma^{\mathsf{T}}W)+m^{[t_1]}(w,\gamma^{\mathsf{T}}w)\right)^{\otimes t_2}\mid U=u\right],\quad t_1=1,2,\ t_2=1,2,4. Q t 1 , t 2 ( w , u ) = E [ ( − m [ t 1 ] ( W , γ T W ) + m [ t 1 ] ( w , γ T w ) ) ⊗ t 2 ∣ U = u ] , t 1 = 1 , 2 , t 2 = 1 , 2 , 4.
Let f 0 ( u ; γ ) f_0(u;\gamma) f 0 ( u ; γ ) denote the probability density function of U γ U_\gamma U γ , and f ( w , u ; γ ) f(w,u;\gamma) f ( w , u ; γ ) the joint probability density function of ( W , U γ ) (W,U_\gamma) ( W , U γ ) .
The following assumptions are imposed to establish the asymptotic properties of the PML estimator:
C1. ξ / h ⟶ ξ 0 \xi/h\longrightarrow\xi_0 ξ / h ⟶ ξ 0 as n ⟶ ∞ n\longrightarrow\infty n ⟶ ∞ for some constant ξ 0 > 0 \xi_0>0 ξ 0 > 0 .
C2. ∂ u 4 f 0 ( u ; γ ) \partial_u^4f_0(u;\gamma) ∂ u 4 f 0 ( u ; γ ) , ∂ u 4 f ( w , u ; γ ) \partial_u^4f(w,u;\gamma) ∂ u 4 f ( w , u ; γ ) , and ∂ u t 2 + 2 Q t 1 , t 2 ( w , u ) \partial_u^{t_2+2}Q_{t_1,t_2}(w,u) ∂ u t 2 + 2 Q t 1 , t 2 ( w , u ) are bounded for u ∈ U γ n u\in\mathcal{U}_{\gamma n} u ∈ U γ n and w ∈ W n w\in\mathcal{W}_n w ∈ W n , and are Lipschitz continuous in ( u , γ ∗ ) (u,\gamma_*) ( u , γ ∗ ) for each ( t 1 , t 2 ) ∈ { ( 1 , 1 ) , ( 1 , 2 ) , ( 2 , 1 ) } (t_1,t_2)\in\{(1,1),(1,2),(2,1)\} ( t 1 , t 2 ) ∈ {( 1 , 1 ) , ( 1 , 2 ) , ( 2 , 1 )} , with a Lipschitz constant independent of u u u . C3. inf { γ ∗ ∈ Γ } f 0 ( u ; γ ) \inf_{\{\gamma^*\in\Gamma\}} f_0(u;\gamma) inf { γ ∗ ∈ Γ } f 0 ( u ; γ ) is far away from zero for u ∈ U γ n u\in\mathcal{U}_{\gamma n} u ∈ U γ n .
C4. V V V is nonsingular.
Assumption C1 restricts the bandwidth rates to ensure the n \sqrt{n} n -consistency of the PML estimator. Assumption C2 imposes the smoothness conditions to guarantee the uniform convergence of the estimated functions to their targets. Assumptions C3 and C4 ensure the asymptotic normality of the PML estimator.
Technical Lemmas
Lemma S1. Under assumptions A2 and A3, as n ⟶ ∞ n\longrightarrow\infty n ⟶ ∞ and h ⟶ 0 h\longrightarrow0 h ⟶ 0 , with n h 2 t + 1 ⟶ ∞ nh^{2t+1}\longrightarrow\infty n h 2 t + 1 ⟶ ∞ for t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 ,
sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ ∂ γ ∗ t N s h ( γ ⊤ w ; G o , γ ) − ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) ∥ = sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ 1 n ∑ i = 1 n ξ i s [ t ] ( w , γ ) ∥ = o p ( log n n h 2 t + 1 ) + O ( h 2 ) , s = 0 , 1 , \begin{aligned}
\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left\|\partial_{\gamma_*}^{t}N_{sh}(\gamma^\top w;\mathcal{G}^o,\gamma)-\partial_v^t(M_{s,t}(w,\gamma^\top w)f_\gamma(\gamma^\top w))\right\| \\
&=\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left\|\frac{1}{n}\sum_{i=1}^{n}\xi_{is}^{[t]}(w,\gamma)\right\|=o_p\left(\sqrt{\frac{\log n}{nh^{2t+1}}}\right)+O(h^2),\qquad s=0,1,
\end{aligned} {( w , γ ∗ ) ∈ W n × Γ } sup ∂ γ ∗ t N s h ( γ ⊤ w ; G o , γ ) − ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w )) = {( w , γ ∗ ) ∈ W n × Γ } sup n 1 i = 1 ∑ n ξ i s [ t ] ( w , γ ) = o p ( n h 2 t + 1 log n ) + O ( h 2 ) , s = 0 , 1 ,
where ξ i s [ t ] ( w , γ ) = ∂ γ ∗ t ( Y i s K h ( γ ⊤ W i − γ ⊤ w ) ) − ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) \xi_{is}^{[t]}(w,\gamma)=\partial_{\gamma_*}^{t}(Y_i^sK_h(\gamma^\top W_i-\gamma^\top w))-\partial_v^t(M_{s,t}(w,\gamma^\top w)f_\gamma(\gamma^\top w)) ξ i s [ t ] ( w , γ ) = ∂ γ ∗ t ( Y i s K h ( γ ⊤ W i − γ ⊤ w )) − ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w )) , i = 1 , … , n i=1,\ldots,n i = 1 , … , n .
Proof. By Lemma 9.6 in Kosorok [14 ] , for each t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 , the classes { ( W ∗ − w ∗ ) ⊗ t : w ∈ W n } \{(W_*-w_*)^{\otimes t}:w\in\mathcal{W}_n\} {( W ∗ − w ∗ ) ⊗ t : w ∈ W n } and { c 1 K ( t ) ( γ ⊤ W + c 2 ) : c 1 , c 2 ∈ R , γ ∗ ∈ Γ } \{c_1K^{(t)}(\gamma^\top W+c_2):c_1,c_2\in\mathbb{R},\gamma_*\in\Gamma\} { c 1 K ( t ) ( γ ⊤ W + c 2 ) : c 1 , c 2 ∈ R , γ ∗ ∈ Γ } have finite VC indices. It follows from Lemma 2.12 in Pakes and Pollard [22 ] that both classes are Euclidean. By Lemma 2.14 therein, their product class
F t = { c 1 K ( t ) ( γ ⊤ W + c 2 ) ( W ∗ − w ∗ ) ⊗ t : w ∈ W n , γ ∗ ∈ Γ , c 1 , c 2 ∈ R } \mathcal{F}_t=\{c_1K^{(t)}(\gamma^\top W+c_2)(W_*-w_*)^{\otimes t}:w\in\mathcal{W}_n,\gamma_*\in\Gamma,c_1,c_2\in\mathbb{R}\} F t = { c 1 K ( t ) ( γ ⊤ W + c 2 ) ( W ∗ − w ∗ ) ⊗ t : w ∈ W n , γ ∗ ∈ Γ , c 1 , c 2 ∈ R }
is also Euclidean. Under assumptions A2 and A3, an application of Hölder’s inequality gives
sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ E [ ( 1 h t + 1 Y s K ( t ) ( γ ⊤ ( W − w ) h ) ) 2 ( W ∗ − w ∗ ) ⊗ 2 t ] ∥ = O ( 1 h 2 t + 1 ) , s = 0 , 1 , t = 0 , 1 , 2. \sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left\|\mathbb{E}\left[\left(\frac{1}{h^{t+1}}Y^sK^{(t)}\left(\frac{\gamma^\top(W-w)}{h}\right)\right)^2(W_*-w_*)^{\otimes2t}\right]\right\|=O\left(\frac{1}{h^{2t+1}}\right),\qquad s=0,1,\ t=0,1,2. {( w , γ ∗ ) ∈ W n × Γ } sup E [ ( h t + 1 1 Y s K ( t ) ( h γ ⊤ ( W − w ) ) ) 2 ( W ∗ − w ∗ ) ⊗ 2 t ] = O ( h 2 t + 1 1 ) , s = 0 , 1 , t = 0 , 1 , 2.
Since F t \mathcal{F}_t F t is Euclidean class, Theorem II.37 in Pollard [24 ] implies that, for s = 0 , 1 s=0,1 s = 0 , 1 and t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 ,
sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ ∂ γ ∗ t N s h ( γ ⊤ w ; G o , γ ) − E [ ∂ γ ∗ t ( Y s K h ( γ ⊤ W − γ ⊤ w ) ) ] ∥ = o p ( log n n h 2 t + 1 ) . (31) \sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left\|\partial_{\gamma_*}^{t}N_{sh}(\gamma^\top w;\mathcal{G}^o,\gamma)-\mathbb{E}\left[\partial_{\gamma_*}^{t}(Y^sK_h(\gamma^\top W-\gamma^\top w))\right]\right\|=o_p\left(\sqrt{\frac{\log n}{nh^{2t+1}}}\right). \tag*{(31)} {( w , γ ∗ ) ∈ W n × Γ } sup ∂ γ ∗ t N s h ( γ ⊤ w ; G o , γ ) − E [ ∂ γ ∗ t ( Y s K h ( γ ⊤ W − γ ⊤ w )) ] = o p ( n h 2 t + 1 log n ) . (31)
By assumption A3, the iterated expectation, and Taylor’s theorem,
E [ ∂ γ ∗ t ( Y s K h ( γ ⊤ W − γ ⊤ w ) ) ] = 1 h t + 1 E [ K ( t ) ( γ ⊤ ( W − w ) h ) M s , t ( w , γ ⊤ W ) ] = 1 h t ∫ K ( t ) ( v ) M s , t ( w , γ ⊤ w + v h ) f γ ( γ ⊤ w + v h ) d v = ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) + r s [ t ] ( w , γ ) , s = 0 , 1 , t = 0 , 1 , 2 , w ∈ W n , (32) \begin{aligned}
\mathbb{E}\left[\partial_{\gamma_*}^{t}(Y^sK_h(\gamma^\top W-\gamma^\top w))\right]=\frac{1}{h^{t+1}}\mathbb{E}\left[K^{(t)}\left(\frac{\gamma^\top(W-w)}{h}\right)M_{s,t}(w,\gamma^\top W)\right] \\
&=\frac{1}{h^t}\int K^{(t)}(v)M_{s,t}(w,\gamma^\top w+vh)f_\gamma(\gamma^\top w+vh)\,dv \\
&=\partial_v^t(M_{s,t}(w,\gamma^\top w)f_\gamma(\gamma^\top w))+r_s^{[t]}(w,\gamma),\qquad s=0,1,\ t=0,1,2,\ w\in\mathcal{W}_n, \tag*{(32)}
\end{aligned} E [ ∂ γ ∗ t ( Y s K h ( γ ⊤ W − γ ⊤ w )) ] = h t + 1 1 E [ K ( t ) ( h γ ⊤ ( W − w ) ) M s , t ( w , γ ⊤ W ) ] = h t 1 ∫ K ( t ) ( v ) M s , t ( w , γ ⊤ w + v h ) f γ ( γ ⊤ w + v h ) d v = ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w )) + r s [ t ] ( w , γ ) , s = 0 , 1 , t = 0 , 1 , 2 , w ∈ W n , (32)
with sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ r s [ t ] ( w , γ ) ∥ = O ( h 2 ) \sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\|r_s^{[t]}(w,\gamma)\|=O(h^2) sup {( w , γ ∗ ) ∈ W n × Γ } ∥ r s [ t ] ( w , γ ) ∥ = O ( h 2 ) . Substituting (D.2) into (D.1) completes the proof. □
Lemma S2. Under assumptions A2, A3 and A5, as n ⟶ ∞ n\longrightarrow\infty n ⟶ ∞ and h ⟶ 0 h\longrightarrow0 h ⟶ 0 , with n h 2 t + 1 ⟶ ∞ nh^{2t+1}\longrightarrow\infty n h 2 t + 1 ⟶ ∞ for t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 ,
sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ ∂ γ ∗ t m ^ h ( γ ⊤ w ; G o , γ ) − m [ t ] ( w , γ ⊤ w ) − 1 n ∑ i = 1 n ξ i [ t ] ( w , γ ) ∥ = o p ( log n n h 2 t + 1 ) + O ( h 4 ) , \sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left\|\partial_{\gamma_*}^{t}\widehat{m}_h(\gamma^\top w;\mathcal{G}^o,\gamma)-m^{[t]}(w,\gamma^\top w)-\frac{1}{n}\sum_{i=1}^{n}\xi_i^{[t]}(w,\gamma)\right\|=o_p\left(\sqrt{\frac{\log n}{nh^{2t+1}}}\right)+O(h^4), {( w , γ ∗ ) ∈ W n × Γ } sup ∂ γ ∗ t m h ( γ ⊤ w ; G o , γ ) − m [ t ] ( w , γ ⊤ w ) − n 1 i = 1 ∑ n ξ i [ t ] ( w , γ ) = o p ( n h 2 t + 1 log n ) + O ( h 4 ) ,
with
sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ 1 n ∑ i = 1 n ξ i [ t ] ( w , γ ) ∥ = o p ( log n n h 2 t + 1 ) + O ( h 2 ) . \sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left\|\frac{1}{n}\sum_{i=1}^{n}\xi_i^{[t]}(w,\gamma)\right\|=o_p\left(\sqrt{\frac{\log n}{nh^{2t+1}}}\right)+O(h^2). {( w , γ ∗ ) ∈ W n × Γ } sup n 1 i = 1 ∑ n ξ i [ t ] ( w , γ ) = o p ( n h 2 t + 1 log n ) + O ( h 2 ) .
where m [ t ] ( w , v ) m^{[t]}(w,v) m [ t ] ( w , v ) and ξ s i [ t ] ( w , γ ) \xi_{si}^{[t]}(w,\gamma) ξ s i [ t ] ( w , γ ) , t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 , i = 1 , … , n i=1,\ldots,n i = 1 , … , n , are defined in (D.4)–(D.6).
Proof. By Taylor’s theorem, Lemma S1, and assumption A5, it follows that, for t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 ,
∂ γ ∗ t N s h ( γ ⊤ w ; G o , γ ) N 0 h ( γ ⊤ w ; G o , γ ) = ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) f γ ( γ ⊤ w ) + 1 n ∑ i = 1 n ξ s i ∗ [ t ] ( w , γ ) + r s ∗ [ t ] ( w , γ ) , (33) \frac{\partial_{\gamma^*}^{t}N_{sh}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}{N_{0h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}
=
\frac{\partial_{v}^{t}\left(M_{s,t}(w,\gamma^{\top}w)f_{\gamma}(\gamma^{\top}w)\right)}{f_{\gamma}(\gamma^{\top}w)}
+\frac{1}{n}\sum_{i=1}^{n}\xi_{si}^{*[t]}(w,\gamma)+r_{s}^{*[t]}(w,\gamma), \tag*{(33)} N 0 h ( γ ⊤ w ; G o , γ ) ∂ γ ∗ t N s h ( γ ⊤ w ; G o , γ ) = f γ ( γ ⊤ w ) ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) + n 1 i = 1 ∑ n ξ s i ∗ [ t ] ( w , γ ) + r s ∗ [ t ] ( w , γ ) , (33)
s = 0 , 1 s=0,1 s = 0 , 1 , w ∈ W n w\in\mathcal{W}_{n} w ∈ W n , with
ξ s i s ∗ [ t ] ( w , γ ) = ξ s i s [ t ] ( w , γ ) f γ ( γ ⊤ w ) − ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) ξ s i [ 0 ] ( w , γ ) ( f γ ( γ ⊤ w ) ) 2 , i = 1 , … , n , \xi_{sis}^{*[t]}(w,\gamma)
=
\frac{\xi_{sis}^{[t]}(w,\gamma)}{f_{\gamma}(\gamma^{\top}w)}
-
\frac{\partial_{v}^{t}\left(M_{s,t}(w,\gamma^{\top}w)f_{\gamma}(\gamma^{\top}w)\right)\xi_{si}^{[0]}(w,\gamma)}
{\left(f_{\gamma}(\gamma^{\top}w)\right)^{2}},
\qquad i=1,\ldots,n, ξ s i s ∗ [ t ] ( w , γ ) = f γ ( γ ⊤ w ) ξ s i s [ t ] ( w , γ ) − ( f γ ( γ ⊤ w ) ) 2 ∂ v t ( M s , t ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) ξ s i [ 0 ] ( w , γ ) , i = 1 , … , n ,
sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ 1 n ∑ i = 1 n ξ s i ∗ [ t ] ( w , γ ) ∥ = o p ( log n n h 2 t + 1 ) + O ( h 2 ) , \sup_{\{(w,\gamma^*)\in\mathcal{W}_{n}\times\Gamma\}}
\left\|
\frac{1}{n}\sum_{i=1}^{n}\xi_{si}^{*[t]}(w,\gamma)
\right\|
=
o_{p}\left(\sqrt{\frac{\log n}{nh^{2t+1}}}\right)+O(h^{2}), {( w , γ ∗ ) ∈ W n × Γ } sup n 1 i = 1 ∑ n ξ s i ∗ [ t ] ( w , γ ) = o p ( n h 2 t + 1 log n ) + O ( h 2 ) ,
and
sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ r s ∗ [ t ] ( w , γ ) ∥ = o p ( log n n h 2 t + 1 ) + O ( h 4 ) . \sup_{\{(w,\gamma^*)\in\mathcal{W}_{n}\times\Gamma\}}
\left\|r_{s}^{*[t]}(w,\gamma)\right\|
=
o_{p}\left(\frac{\log n}{nh^{2t+1}}\right)+O(h^{4}). {( w , γ ∗ ) ∈ W n × Γ } sup r s ∗ [ t ] ( w , γ ) = o p ( n h 2 t + 1 log n ) + O ( h 4 ) .
For a square matrix A A A , let sym ( A ) : = ( A + A ⊤ ) / 2 \operatorname{sym}(A):=(A+A^{\top})/2 sym ( A ) := ( A + A ⊤ ) /2 denote its symmetric part. Combining (D.3) with the identities
m ^ h ( γ ⊤ w ; G o , γ ) = N 1 h ( γ ⊤ w ; G o , γ ) N 0 h ( γ ⊤ w ; G o , γ ) , ∂ γ ∗ m ^ h ( γ ⊤ w ; G o , γ ) = ∑ s = 0 1 ( − m ^ h ( γ ⊤ w ; γ ) ) 1 − s ∂ γ ∗ N s h ( γ ⊤ w ; G o , γ ) N 0 h ( γ ⊤ w ; G o , γ ) , \widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)
=
\frac{N_{1h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}
{N_{0h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)},
\qquad
\partial_{\gamma^*}\widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)
=
\sum_{s=0}^{1}
\left(-\widehat{m}_{h}(\gamma^{\top}w;\gamma)\right)^{1-s}
\frac{\partial_{\gamma^*}N_{sh}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}
{N_{0h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}, m h ( γ ⊤ w ; G o , γ ) = N 0 h ( γ ⊤ w ; G o , γ ) N 1 h ( γ ⊤ w ; G o , γ ) , ∂ γ ∗ m h ( γ ⊤ w ; G o , γ ) = s = 0 ∑ 1 ( − m h ( γ ⊤ w ; γ ) ) 1 − s N 0 h ( γ ⊤ w ; G o , γ ) ∂ γ ∗ N s h ( γ ⊤ w ; G o , γ ) ,
∂ γ ∗ 2 m ^ h ( γ ⊤ w ; G o , γ ) = ∑ s = 0 1 ( − m ^ h ( γ ⊤ w ; G o , γ ) ) 1 − s ∂ γ ∗ 2 N s h ( γ ⊤ w ; G o , γ ) N 0 h ( γ ⊤ w ; G o , γ ) − 2 sym ( ∂ γ ∗ N 0 h ( γ ⊤ w ; G o , γ ) N 0 h ( γ ⊤ w ; G o , γ ) ( ∂ γ ∗ m ^ h ( γ ⊤ w ; G o , γ ) ) ⊤ ) , \partial_{\gamma^*}^{2}\widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)
=
\sum_{s=0}^{1}
\left(-\widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)\right)^{1-s}
\frac{\partial_{\gamma^*}^{2}N_{sh}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}
{N_{0h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}
-
2\operatorname{sym}\left(
\frac{\partial_{\gamma^*}N_{0h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}
{N_{0h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}
\left(\partial_{\gamma^*}\widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)\right)^{\top}
\right), ∂ γ ∗ 2 m h ( γ ⊤ w ; G o , γ ) = s = 0 ∑ 1 ( − m h ( γ ⊤ w ; G o , γ ) ) 1 − s N 0 h ( γ ⊤ w ; G o , γ ) ∂ γ ∗ 2 N s h ( γ ⊤ w ; G o , γ ) − 2 sym ( N 0 h ( γ ⊤ w ; G o , γ ) ∂ γ ∗ N 0 h ( γ ⊤ w ; G o , γ ) ( ∂ γ ∗ m h ( γ ⊤ w ; G o , γ ) ) ⊤ ) ,
yields the following expressions:
m ^ h ( γ ⊤ w ; G o , γ ) = M 1 , 0 ( w , γ ⊤ w ) f γ ( γ ⊤ w ) f γ ( γ ⊤ w ) + 1 n ∑ i = 1 n ξ i ∗ [ 0 ] ( w , γ ) + r 1 ∗ [ 0 ] ( w , γ ) \widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)
=
\frac{M_{1,0}(w,\gamma^{\top}w)f_{\gamma}(\gamma^{\top}w)}
{f_{\gamma}(\gamma^{\top}w)}
+\frac{1}{n}\sum_{i=1}^{n}\xi_{i}^{*[0]}(w,\gamma)+r_{1}^{*[0]}(w,\gamma) m h ( γ ⊤ w ; G o , γ ) = f γ ( γ ⊤ w ) M 1 , 0 ( w , γ ⊤ w ) f γ ( γ ⊤ w ) + n 1 i = 1 ∑ n ξ i ∗ [ 0 ] ( w , γ ) + r 1 ∗ [ 0 ] ( w , γ )
m ^ h ( γ ⊤ w ; G o , γ ) = ^ m [ 0 ] ( w , γ ⊤ w ) + 1 n ∑ i = 1 n ξ i [ 0 ] ( w , γ ) + r [ 0 ] ( w , γ ) , (34) \mathrel{\phantom{\widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}}
\mathrel{\widehat{=}}
m^{[0]}(w,\gamma^{\top}w)+\frac{1}{n}\sum_{i=1}^{n}\xi_{i}^{[0]}(w,\gamma)+r^{[0]}(w,\gamma), \tag*{(34)} m h ( γ ⊤ w ; G o , γ ) = m [ 0 ] ( w , γ ⊤ w ) + n 1 i = 1 ∑ n ξ i [ 0 ] ( w , γ ) + r [ 0 ] ( w , γ ) , (34)
∂ γ ∗ m ^ h ( γ ⊤ w ; G o , γ ) = ∑ s = 0 1 [ − ( m [ 0 ] ( w , γ ⊤ w ) + 1 n ∑ i = 1 n ξ i [ 0 ] ( w , γ ) + r [ 0 ] ( w , γ ) ) ] 1 − s ( ∂ v ( M s , 1 ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) f γ ( γ ⊤ w ) + 1 n ∑ i = 1 n ξ s i ∗ [ 1 ] ( w , γ ) + r s ∗ [ 1 ] ( w , γ ) ) \partial_{\gamma^*}\widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)
=
\sum_{s=0}^{1}
\left[
-\left(
m^{[0]}(w,\gamma^{\top}w)+\frac{1}{n}\sum_{i=1}^{n}\xi_{i}^{[0]}(w,\gamma)+r^{[0]}(w,\gamma)
\right)
\right]^{1-s}
\left(
\frac{\partial_{v}\left(M_{s,1}(w,\gamma^{\top}w)f_{\gamma}(\gamma^{\top}w)\right)}
{f_{\gamma}(\gamma^{\top}w)}
+\frac{1}{n}\sum_{i=1}^{n}\xi_{si}^{*[1]}(w,\gamma)+r_{s}^{*[1]}(w,\gamma)
\right) ∂ γ ∗ m h ( γ ⊤ w ; G o , γ ) = s = 0 ∑ 1 [ − ( m [ 0 ] ( w , γ ⊤ w ) + n 1 i = 1 ∑ n ξ i [ 0 ] ( w , γ ) + r [ 0 ] ( w , γ ) ) ] 1 − s ( f γ ( γ ⊤ w ) ∂ v ( M s , 1 ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) + n 1 i = 1 ∑ n ξ s i ∗ [ 1 ] ( w , γ ) + r s ∗ [ 1 ] ( w , γ ) )
∂ γ ∗ m ^ h ( γ ⊤ w ; G o , γ ) = ^ m [ 1 ] ( w , γ ⊤ w ) + 1 n ∑ i = 1 n ξ i [ 1 ] ( w , γ ) + r [ 1 ] ( w , γ ) , (35) \mathrel{\phantom{\partial_{\gamma^*}\widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}}
\mathrel{\widehat{=}}
m^{[1]}(w,\gamma^{\top}w)+\frac{1}{n}\sum_{i=1}^{n}\xi_{i}^{[1]}(w,\gamma)+r^{[1]}(w,\gamma), \tag*{(35)} ∂ γ ∗ m h ( γ ⊤ w ; G o , γ ) = m [ 1 ] ( w , γ ⊤ w ) + n 1 i = 1 ∑ n ξ i [ 1 ] ( w , γ ) + r [ 1 ] ( w , γ ) , (35)
∂ γ ∗ 2 m ^ h ( γ ⊤ w ; G o , γ ) = { ∑ s = 0 1 [ − ( m [ 0 ] ( w , γ ⊤ w ) + 1 n ∑ i = 1 n ξ i [ 0 ] ( w , γ ) + r [ 0 ] ( w , γ ) ) ] 1 − s ( ∂ v 2 ( M s , 2 ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) f γ ( γ ⊤ w ) + 1 n ∑ i = 1 n ξ s i ∗ [ 2 ] ( w , γ ) + r s ∗ [ 2 ] ( w , γ ) ) } \partial_{\gamma^*}^{2}\widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)
=
\left\{
\sum_{s=0}^{1}
\left[
-\left(
m^{[0]}(w,\gamma^{\top}w)+\frac{1}{n}\sum_{i=1}^{n}\xi_{i}^{[0]}(w,\gamma)+r^{[0]}(w,\gamma)
\right)
\right]^{1-s}
\left(
\frac{\partial_{v}^{2}\left(M_{s,2}(w,\gamma^{\top}w)f_{\gamma}(\gamma^{\top}w)\right)}
{f_{\gamma}(\gamma^{\top}w)}
+\frac{1}{n}\sum_{i=1}^{n}\xi_{si}^{*[2]}(w,\gamma)+r_{s}^{*[2]}(w,\gamma)
\right)
\right\} ∂ γ ∗ 2 m h ( γ ⊤ w ; G o , γ ) = ⎩ ⎨ ⎧ s = 0 ∑ 1 [ − ( m [ 0 ] ( w , γ ⊤ w ) + n 1 i = 1 ∑ n ξ i [ 0 ] ( w , γ ) + r [ 0 ] ( w , γ ) ) ] 1 − s ( f γ ( γ ⊤ w ) ∂ v 2 ( M s , 2 ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) + n 1 i = 1 ∑ n ξ s i ∗ [ 2 ] ( w , γ ) + r s ∗ [ 2 ] ( w , γ ) ) ⎭ ⎬ ⎫
∂ γ ∗ 2 m ^ h ( γ ⊤ w ; G o , γ ) − 2 sym ( [ ∂ v ( M 0 , 1 ( w , γ ⊤ w ) f γ ( γ ⊤ w ) ) f γ ( γ ⊤ w ) + 1 n ∑ i = 1 n ξ 0 i ∗ [ 1 ] ( w , γ ) + r 0 ∗ [ 1 ] ( w , γ ) ] \mathrel{\phantom{\partial_{\gamma^*}^{2}\widehat{m}_{h}(\gamma^{\top}w;\mathcal{G}^{o},\gamma)}}
-
2\operatorname{sym}(
[
\frac{\partial_{v}(M_{0,1}(w,\gamma^{\top}w)f_{\gamma}(\gamma^{\top}w))}
{f_{\gamma}(\gamma^{\top}w)}
+\frac{1}{n}\sum_{i=1}^{n}\xi_{0i}^{*[1]}(w,\gamma)+r_{0}^{*[1]}(w,\gamma)
] ∂ γ ∗ 2 m h ( γ ⊤ w ; G o , γ ) − 2 sym ([ f γ ( γ ⊤ w ) ∂ v ( M 0 , 1 ( w , γ ⊤ w ) f γ ( γ ⊤ w )) + n 1 i = 1 ∑ n ξ 0 i ∗ [ 1 ] ( w , γ ) + r 0 ∗ [ 1 ] ( w , γ )]
( m [ 1 ] ( w , γ ⊤ w ) + 1 n ∑ i = 1 n ξ i [ 1 ] ( w , γ ) + r [ 1 ] ( w , γ ) ) ⊤ = ^ m [ 2 ] ( w , γ ⊤ w ) + 1 n ∑ i = 1 n ξ i [ 2 ] ( w , γ ) + r [ 2 ] ( w , γ ) , (36) \left(m^{[1]}(w,\gamma^\top w)+\frac{1}{n}\sum_{i=1}^{n}\xi_i^{[1]}(w,\gamma)+r^{[1]}(w,\gamma)\right)^\top
\widehat{=}\ m^{[2]}(w,\gamma^\top w)+\frac{1}{n}\sum_{i=1}^{n}\xi_i^{[2]}(w,\gamma)+r^{[2]}(w,\gamma), \tag*{(36)} ( m [ 1 ] ( w , γ ⊤ w ) + n 1 i = 1 ∑ n ξ i [ 1 ] ( w , γ ) + r [ 1 ] ( w , γ ) ) ⊤ = m [ 2 ] ( w , γ ⊤ w ) + n 1 i = 1 ∑ n ξ i [ 2 ] ( w , γ ) + r [ 2 ] ( w , γ ) , (36)
where
m [ 0 ] ( w , v ) = M 1 , 0 ( w , v ) , m [ 1 ] ( w , v ) = ∑ s = 0 1 ( − m [ 0 ] ( w , v ) ) 1 − s ∂ v ( M s , 1 ( w , v ) f γ ( v ) ) f γ ( v ) , m^{[0]}(w,v)=M_{1,0}(w,v),\quad m^{[1]}(w,v)=\sum_{s=0}^{1}\left(-m^{[0]}(w,v)\right)^{1-s}\frac{\partial_v\left(M_{s,1}(w,v)f_\gamma(v)\right)}{f_\gamma(v)}, m [ 0 ] ( w , v ) = M 1 , 0 ( w , v ) , m [ 1 ] ( w , v ) = s = 0 ∑ 1 ( − m [ 0 ] ( w , v ) ) 1 − s f γ ( v ) ∂ v ( M s , 1 ( w , v ) f γ ( v ) ) ,
and
m [ 2 ] ( w , v ) = ∑ s = 0 1 ( − m [ 0 ] ( w , v ) ) 1 − s ∂ v ( M s , 2 ( w , v ) f γ ( v ) ) f γ ( v ) − 2 sym ( ∂ v ( M 0 , 1 ( w , v ) f γ ( v ) ) f γ ( v ) ( m [ 1 ] ( w , v ) ) ⊤ ) , m^{[2]}(w,v)=\sum_{s=0}^{1}\left(-m^{[0]}(w,v)\right)^{1-s}\frac{\partial_v\left(M_{s,2}(w,v)f_\gamma(v)\right)}{f_\gamma(v)}-2\operatorname{sym}\left(\frac{\partial_v\left(M_{0,1}(w,v)f_\gamma(v)\right)}{f_\gamma(v)}\left(m^{[1]}(w,v)\right)^\top\right), m [ 2 ] ( w , v ) = s = 0 ∑ 1 ( − m [ 0 ] ( w , v ) ) 1 − s f γ ( v ) ∂ v ( M s , 2 ( w , v ) f γ ( v ) ) − 2 sym ( f γ ( v ) ∂ v ( M 0 , 1 ( w , v ) f γ ( v ) ) ( m [ 1 ] ( w , v ) ) ⊤ ) ,
with
sup { ( w , γ ∗ ) ∈ W n × Γ } ∥ r [ t ] ( W i , γ ) ∥ = O p ( log n n h 2 t + 1 ) + O ( h 4 ) , t = 0 , 1 , 2. \sup_{\{(w,\gamma^*)\in\mathcal{W}_n\times\Gamma\}}\left\|r^{[t]}(W_i,\gamma)\right\|=O_p\left(\frac{\log n}{nh^{2t+1}}\right)+O(h^4),\quad t=0,1,2. {( w , γ ∗ ) ∈ W n × Γ } sup r [ t ] ( W i , γ ) = O p ( n h 2 t + 1 log n ) + O ( h 4 ) , t = 0 , 1 , 2.
The lemma follows.
Lemma S3. Under assumptions A 2 A2 A 2 , A 3 A3 A 3 , A 5 A5 A 5 , and C 2 C2 C 2 , as n ⟶ ∞ n\longrightarrow\infty n ⟶ ∞ and ξ ⟶ 0 \xi\longrightarrow0 ξ ⟶ 0 , with n ξ 2 t + 1 ⟶ ∞ n\xi^{2t+1}\longrightarrow\infty n ξ 2 t + 1 ⟶ ∞ for t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 ,
sup { ( w , u , γ ∗ ) ∈ W n × U γ n × Γ } ∥ ∂ γ ∗ t f ˉ ξ ( u ; γ ) − f [ t ] ( w , u ; γ ) ∥ = sup { ( w , u , γ ∗ ) ∈ W n × U γ n × Γ } ∥ 1 n ∑ i = 1 n ζ i [ t ] ( w , u , γ ) ∥ = O p ( log n n ξ 2 t + 1 ) + O ( ξ 2 ) \sup_{\{(w,u,\gamma^*)\in\mathcal{W}_n\times\mathcal{U}_{\gamma n}\times\Gamma\}}\left\|\partial_{\gamma^*}^{t}\bar f_\xi(u;\gamma)-f^{[t]}(w,u;\gamma)\right\|
=\sup_{\{(w,u,\gamma^*)\in\mathcal{W}_n\times\mathcal{U}_{\gamma n}\times\Gamma\}}\left\|\frac{1}{n}\sum_{i=1}^{n}\zeta_i^{[t]}(w,u,\gamma)\right\|
=O_p\left(\sqrt{\frac{\log n}{n\xi^{2t+1}}}\right)+O(\xi^2) {( w , u , γ ∗ ) ∈ W n × U γ n × Γ } sup ∂ γ ∗ t f ˉ ξ ( u ; γ ) − f [ t ] ( w , u ; γ ) = {( w , u , γ ∗ ) ∈ W n × U γ n × Γ } sup n 1 i = 1 ∑ n ζ i [ t ] ( w , u , γ ) = O p ( n ξ 2 t + 1 log n ) + O ( ξ 2 ) and
sup { ( u , γ ∗ ) ∈ U γ n × Γ } ∥ ∂ u t f ˉ ξ ( u ; γ ) − ∂ u t f 0 ( u ; γ ) ∥ = sup { ( u , γ ∗ ) ∈ U γ n × Γ } ∥ 1 n ∑ i = 1 n ζ i [ t ] ( u , γ ) ∥ = O p ( log n n ξ 2 t + 1 ) + O ( ξ 2 ) , \sup_{\{(u,\gamma^*)\in\mathcal{U}_{\gamma n}\times\Gamma\}}\left\|\partial_u^t\bar f_\xi(u;\gamma)-\partial_u^t f_0(u;\gamma)\right\|
=\sup_{\{(u,\gamma^*)\in\mathcal{U}_{\gamma n}\times\Gamma\}}\left\|\frac{1}{n}\sum_{i=1}^{n}\zeta_i^{[t]}(u,\gamma)\right\|
=O_p\left(\sqrt{\frac{\log n}{n\xi^{2t+1}}}\right)+O(\xi^2), {( u , γ ∗ ) ∈ U γ n × Γ } sup ∂ u t f ˉ ξ ( u ; γ ) − ∂ u t f 0 ( u ; γ ) = {( u , γ ∗ ) ∈ U γ n × Γ } sup n 1 i = 1 ∑ n ζ i [ t ] ( u , γ ) = O p ( n ξ 2 t + 1 log n ) + O ( ξ 2 ) , where f ˉ ξ ( u ; γ ) = ∑ i = 1 n K ξ ( U γ i − u ) / n \bar f_\xi(u;\gamma)=\sum_{i=1}^{n}K_\xi(U_{\gamma_i}-u)/n f ˉ ξ ( u ; γ ) = ∑ i = 1 n K ξ ( U γ i − u ) / n , f [ t ] ( w , u ; γ ) f^{[t]}(w,u;\gamma) f [ t ] ( w , u ; γ ) , t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 , are defined in (D.10)–(D.12), ζ i [ t ] ( w , u , γ ) = ( ∂ γ ∗ t K ξ ( U γ i − u ) ) − f [ t ] ( w , u ; γ ) \zeta_i^{[t]}(w,u,\gamma)=\left(\partial_{\gamma^*}^{t}K_\xi(U_{\gamma_i}-u)\right)-f^{[t]}(w,u;\gamma) ζ i [ t ] ( w , u , γ ) = ( ∂ γ ∗ t K ξ ( U γ i − u ) ) − f [ t ] ( w , u ; γ ) , ζ i [ t ] ( u , γ ) = ( ∂ u t K ξ ( U γ i − u ) ) − ∂ u t f 0 ( u ; γ ) \zeta_i^{[t]}(u,\gamma)=\left(\partial_u^tK_\xi(U_{\gamma_i}-u)\right)-\partial_u^t f_0(u;\gamma) ζ i [ t ] ( u , γ ) = ( ∂ u t K ξ ( U γ i − u ) ) − ∂ u t f 0 ( u ; γ ) , with f [ 0 ] ( w , u ; γ ) = f 0 ( u ; γ ) f^{[0]}(w,u;\gamma)=f_0(u;\gamma) f [ 0 ] ( w , u ; γ ) = f 0 ( u ; γ ) and ζ i [ 0 ] ( w , u , γ ) = ζ i [ 0 ] ( u , γ ) \zeta_i^{[0]}(w,u,\gamma)=\zeta_i^{[0]}(u,\gamma) ζ i [ 0 ] ( w , u , γ ) = ζ i [ 0 ] ( u , γ ) , i = 1 , … , n i=1,\ldots,n i = 1 , … , n .
Proof. It suffices to prove the first assertion, as the second assertion follows from Giné and Guillou [5 ] . Applying the chain rule, the summands appearing in ∂ γ ∗ f ˉ ξ ( u ; γ ) \partial_{\gamma^*}\bar f_\xi(u;\gamma) ∂ γ ∗ f ˉ ξ ( u ; γ ) and ∂ γ ∗ 2 f ˉ ξ ( u ; γ ) \partial_{\gamma^*}^2\bar f_\xi(u;\gamma) ∂ γ ∗ 2 f ˉ ξ ( u ; γ ) admit the following representations:
∂ γ ∗ ( 1 ξ K ( U γ i − u ξ ) ) = 1 ξ 2 K ( 1 ) ( U γ i − u ξ ) ( − m [ 1 ] ( W i , γ ⊤ W i ) + m [ 1 ] ( w , γ ⊤ w ) ) (37) \partial_{\gamma^*}\left(\frac{1}{\xi}K\left(\frac{U_{\gamma_i}-u}{\xi}\right)\right)=\frac{1}{\xi^2}K^{(1)}\left(\frac{U_{\gamma_i}-u}{\xi}\right)\left(-m^{[1]}(W_i,\gamma^\top W_i)+m^{[1]}(w,\gamma^\top w)\right) \tag*{(37)} ∂ γ ∗ ( ξ 1 K ( ξ U γ i − u ) ) = ξ 2 1 K ( 1 ) ( ξ U γ i − u ) ( − m [ 1 ] ( W i , γ ⊤ W i ) + m [ 1 ] ( w , γ ⊤ w ) ) (37)
and
∂ γ ∗ 2 ( 1 ξ K ( U γ i − u ξ ) ) = 1 ξ 3 K ( 2 ) ( U γ i − u ξ ) ( m [ 1 ] ( W i , γ ⊤ W i ) − m [ 1 ] ( w , γ ⊤ w ) ) ⊗ 2 + 1 ξ 2 K ( 1 ) ( U γ i − u ξ ) ( − m [ 2 ] ( W i , γ ⊤ W i ) + m [ 2 ] ( w , γ ⊤ w ) ) , (38) \begin{aligned}
\partial_{\gamma^*}^2\left(\frac{1}{\xi}K\left(\frac{U_{\gamma_i}-u}{\xi}\right)\right)
={}&\frac{1}{\xi^3}K^{(2)}\left(\frac{U_{\gamma_i}-u}{\xi}\right)\left(m^{[1]}(W_i,\gamma^\top W_i)-m^{[1]}(w,\gamma^\top w)\right)^{\otimes2}\\
&+\frac{1}{\xi^2}K^{(1)}\left(\frac{U_{\gamma_i}-u}{\xi}\right)\left(-m^{[2]}(W_i,\gamma^\top W_i)+m^{[2]}(w,\gamma^\top w)\right), \tag*{(38)}
\end{aligned} ∂ γ ∗ 2 ( ξ 1 K ( ξ U γ i − u ) ) = ξ 3 1 K ( 2 ) ( ξ U γ i − u ) ( m [ 1 ] ( W i , γ ⊤ W i ) − m [ 1 ] ( w , γ ⊤ w ) ) ⊗ 2 + ξ 2 1 K ( 1 ) ( ξ U γ i − u ) ( − m [ 2 ] ( W i , γ ⊤ W i ) + m [ 2 ] ( w , γ ⊤ w ) ) , (38)
where m [ 1 ] ( w , γ ⊤ w ) m^{[1]}(w,\gamma^\top w) m [ 1 ] ( w , γ ⊤ w ) and m [ 2 ] ( w , γ ⊤ w ) m^{[2]}(w,\gamma^\top w) m [ 2 ] ( w , γ ⊤ w ) are defined in (D.5) and (D.6), respectively. By Lemma 9.6 in Kosorok [14 ] , the class
{ m [ t ] ( W , γ ⊤ W ) − m [ t ] ( w , γ ⊤ w ) : w ∈ W n , γ ∗ ∈ Γ } \left\{m^{[t]}(W,\gamma^\top W)-m^{[t]}(w,\gamma^\top w):w\in\mathcal{W}_n,\gamma^*\in\Gamma\right\} { m [ t ] ( W , γ ⊤ W ) − m [ t ] ( w , γ ⊤ w ) : w ∈ W n , γ ∗ ∈ Γ }
is of finite VC index for each t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 . It follows from Lemma 2.12 in Pakes and Pollard [22 ] that these classes are Euclidean. By Lemma 2.14 therein, the class
F t 1 t 2 = { c 1 K ( t 1 ) ( U γ + c 2 ) ( m [ t 2 ] ( W , γ ⊤ W ) − m [ t 2 ] ( w , γ ⊤ w ) ) ⊗ t 1 : w ∈ W n , γ ∗ ∈ Γ , c 1 , c 2 ∈ R } \mathcal{F}_{t_1t_2}=\left\{c_1K^{(t_1)}(U_\gamma+c_2)\left(m^{[t_2]}(W,\gamma^\top W)-m^{[t_2]}(w,\gamma^\top w)\right)^{\otimes t_1}:w\in\mathcal{W}_n,\gamma^*\in\Gamma,c_1,c_2\in\mathbb{R}\right\} F t 1 t 2 = { c 1 K ( t 1 ) ( U γ + c 2 ) ( m [ t 2 ] ( W , γ ⊤ W ) − m [ t 2 ] ( w , γ ⊤ w ) ) ⊗ t 1 : w ∈ W n , γ ∗ ∈ Γ , c 1 , c 2 ∈ R }
is also Euclidean for each t 1 , t 2 = 0 , 1 , 2 t_1,t_2=0,1,2 t 1 , t 2 = 0 , 1 , 2 .
Under assumptions A2, A3, A5, and C2, and using the identities in (D.7) and (D.8), we obtain that, for t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 ,
sup { ( w , u , γ ∗ ) ∈ W n × U γ n × Γ } ∥ E [ ( ∂ γ ∗ t ( 1 ξ K ( U γ − u ξ ) ) ) ⊗ 2 ] ∥ = O ( 1 ξ 2 t + 1 ) . \sup_{\{(w,u,\gamma_*)\in\mathcal{W}_n\times\mathcal{U}_{\gamma n}\times\Gamma\}}\left\lVert\mathbb{E}\left[\left(\partial_{\gamma_*}^{t}\left(\frac{1}{\xi}K\left(\frac{U_\gamma-u}{\xi}\right)\right)\right)^{\otimes2}\right]\right\rVert=O\left(\frac{1}{\xi^{2t+1}}\right). {( w , u , γ ∗ ) ∈ W n × U γ n × Γ } sup E [ ( ∂ γ ∗ t ( ξ 1 K ( ξ U γ − u ) ) ) ⊗ 2 ] = O ( ξ 2 t + 1 1 ) .
Since F 00 \mathcal{F}_{00} F 00 , F 11 \mathcal{F}_{11} F 11 , and F 12 \mathcal{F}_{12} F 12 are Euclidean, Theorem II.37 in Pollard [24 ] implies that, for t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 ,
sup { ( w , u , γ ∗ ) ∈ W n × U γ n × Γ } ∥ ∂ γ ∗ t f ˉ ξ ( u ; γ ) − E [ ∂ γ ∗ t f ˉ ξ ( u ; γ ) ] ∥ = o p ( log n n ξ 2 t + 1 ) . (39) \sup_{\{(w,u,\gamma_*)\in\mathcal{W}_n\times\mathcal{U}_{\gamma n}\times\Gamma\}}\left\lVert\partial_{\gamma_*}^{t}\bar{f}_{\xi}(u;\gamma)-\mathbb{E}\left[\partial_{\gamma_*}^{t}\bar{f}_{\xi}(u;\gamma)\right]\right\rVert=o_p\left(\sqrt{\frac{\log n}{n\xi^{2t+1}}}\right). \tag*{(39)} {( w , u , γ ∗ ) ∈ W n × U γ n × Γ } sup ∂ γ ∗ t f ˉ ξ ( u ; γ ) − E [ ∂ γ ∗ t f ˉ ξ ( u ; γ ) ] = o p ( n ξ 2 t + 1 log n ) . (39)
Under assumption C2, by the iterated expectation and Taylor’s theorem,
E [ K ξ ( U γ − u ) ] = ∫ K ( v ) f 0 ( u + v ξ ; γ ) d v = f 0 ( u ; γ ) + r 0 ( u ; γ ) ≜ f 0 [ 0 ] ( w , u ; γ ) + r 0 ( w , u ; γ ) . (40) \mathbb{E}\left[K_{\xi}(U_{\gamma}-u)\right]=\int K(v)f_0(u+v\xi;\gamma)\,dv=f_0(u;\gamma)+r_0(u;\gamma)\mathrel{\triangleq}f^{[0]}_0(w,u;\gamma)+r_0(w,u;\gamma). \tag*{(40)} E [ K ξ ( U γ − u ) ] = ∫ K ( v ) f 0 ( u + v ξ ; γ ) d v = f 0 ( u ; γ ) + r 0 ( u ; γ ) ≜ f 0 [ 0 ] ( w , u ; γ ) + r 0 ( w , u ; γ ) . (40)
E [ ∂ γ ∗ K ξ ( U γ − u ) ] = 1 ξ 2 E [ K ( 1 ) ( U γ − u ξ ) Q 1 , 1 ( w , U γ ) ] = ∫ 1 ξ K ( 1 ) ( v ) Q 1 , 1 ( w , u + v ξ ) f 0 ( u + v ξ ; γ ) d v = − ∂ u ( Q 1 , 1 ( w , u ) f 0 ( u ; γ ) ) + r 1 ( w , u ; γ ) ≜ f 0 [ 1 ] ( w , u ; γ ) + r 1 ( w , u ; γ ) , (41) \begin{aligned}
\mathbb{E}\left[\partial_{\gamma_*}K_{\xi}(U_{\gamma}-u)\right]
&=\frac{1}{\xi^2}\mathbb{E}\left[K^{(1)}\left(\frac{U_{\gamma}-u}{\xi}\right)Q_{1,1}(w,U_{\gamma})\right] \\
&=\int\frac{1}{\xi}K^{(1)}(v)Q_{1,1}(w,u+v\xi)f_0(u+v\xi;\gamma)\,dv \\
&=-\partial_u\left(Q_{1,1}(w,u)f_0(u;\gamma)\right)+r_1(w,u;\gamma) \\
&\mathrel{\triangleq}f^{[1]}_0(w,u;\gamma)+r_1(w,u;\gamma), \tag*{(41)}
\end{aligned} E [ ∂ γ ∗ K ξ ( U γ − u ) ] = ξ 2 1 E [ K ( 1 ) ( ξ U γ − u ) Q 1 , 1 ( w , U γ ) ] = ∫ ξ 1 K ( 1 ) ( v ) Q 1 , 1 ( w , u + v ξ ) f 0 ( u + v ξ ; γ ) d v = − ∂ u ( Q 1 , 1 ( w , u ) f 0 ( u ; γ ) ) + r 1 ( w , u ; γ ) ≜ f 0 [ 1 ] ( w , u ; γ ) + r 1 ( w , u ; γ ) , (41)
E [ ∂ γ ∗ 2 K ξ ( U γ − u ) ] = 1 ξ 3 E [ K ( 2 ) ( U γ − u ξ ) Q 1 , 2 ( w , U γ ) ] + 1 ξ 2 E [ K ( 1 ) ( U γ − u ξ ) Q 2 , 1 ( w , U γ ) ] = ∫ 1 ξ 2 K ( 2 ) ( v ) Q 1 , 2 ( w , u + v ξ ) f 0 ( u + v ξ ; γ ) d v + ∫ 1 ξ K ( 1 ) ( v ) Q 2 , 1 ( w , u + v ξ ) f 0 ( u + v ξ ; γ ) d v = ∂ u 2 ( Q 1 , 2 ( w , u ) f 0 ( u ; γ ) ) − ∂ u ( Q 2 , 1 ( w , u ) f 0 ( u ; γ ) ) + r 2 ( w , u ; γ ) ≜ f 0 [ 2 ] ( w , u ; γ ) + r 2 ( w , u ; γ ) , (42) \begin{aligned}
\mathbb{E}\left[\partial_{\gamma_*}^{2}K_{\xi}(U_{\gamma}-u)\right]
&=\frac{1}{\xi^3}\mathbb{E}\left[K^{(2)}\left(\frac{U_{\gamma}-u}{\xi}\right)Q_{1,2}(w,U_{\gamma})\right] \\
&\quad+\frac{1}{\xi^2}\mathbb{E}\left[K^{(1)}\left(\frac{U_{\gamma}-u}{\xi}\right)Q_{2,1}(w,U_{\gamma})\right] \\
&=\int\frac{1}{\xi^2}K^{(2)}(v)Q_{1,2}(w,u+v\xi)f_0(u+v\xi;\gamma)\,dv \\
&\quad+\int\frac{1}{\xi}K^{(1)}(v)Q_{2,1}(w,u+v\xi)f_0(u+v\xi;\gamma)\,dv \\
&=\partial_u^2\left(Q_{1,2}(w,u)f_0(u;\gamma)\right)-\partial_u\left(Q_{2,1}(w,u)f_0(u;\gamma)\right)+r_2(w,u;\gamma) \\
&\mathrel{\triangleq}f^{[2]}_0(w,u;\gamma)+r_2(w,u;\gamma), \tag*{(42)}
\end{aligned} E [ ∂ γ ∗ 2 K ξ ( U γ − u ) ] = ξ 3 1 E [ K ( 2 ) ( ξ U γ − u ) Q 1 , 2 ( w , U γ ) ] + ξ 2 1 E [ K ( 1 ) ( ξ U γ − u ) Q 2 , 1 ( w , U γ ) ] = ∫ ξ 2 1 K ( 2 ) ( v ) Q 1 , 2 ( w , u + v ξ ) f 0 ( u + v ξ ; γ ) d v + ∫ ξ 1 K ( 1 ) ( v ) Q 2 , 1 ( w , u + v ξ ) f 0 ( u + v ξ ; γ ) d v = ∂ u 2 ( Q 1 , 2 ( w , u ) f 0 ( u ; γ ) ) − ∂ u ( Q 2 , 1 ( w , u ) f 0 ( u ; γ ) ) + r 2 ( w , u ; γ ) ≜ f 0 [ 2 ] ( w , u ; γ ) + r 2 ( w , u ; γ ) , (42)
with sup { ( w , u , γ ∗ ) ∈ W n × U γ n × Γ } ∣ r t ( w , u ; γ ) ∣ = O ( ξ 2 ) \sup_{\{(w,u,\gamma_*)\in\mathcal{W}_n\times\mathcal{U}_{\gamma n}\times\Gamma\}}|r_t(w,u;\gamma)|=O(\xi^2) sup {( w , u , γ ∗ ) ∈ W n × U γ n × Γ } ∣ r t ( w , u ; γ ) ∣ = O ( ξ 2 ) . Using E [ ∂ γ ∗ t f ˉ ξ ( u ; γ ) ] = E [ ∂ γ ∗ t K ξ ( U γ − u ) ] \mathbb{E}[\partial_{\gamma_*}^{t}\bar{f}_{\xi}(u;\gamma)]=\mathbb{E}[\partial_{\gamma_*}^{t}K_{\xi}(U_{\gamma}-u)] E [ ∂ γ ∗ t f ˉ ξ ( u ; γ )] = E [ ∂ γ ∗ t K ξ ( U γ − u )] for each t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 , and substituting (D.10)–(D.12) into (D.9), we complete the proof of the first assertion.
Lemma S4. Under assumptions A2, A3, A5, and C2, as n → ∞ n\to\infty n → ∞ , h → 0 h\to0 h → 0 , and ξ → 0 \xi\to0 ξ → 0 , with n h 2 t + 1 → ∞ nh^{2t+1}\to\infty n h 2 t + 1 → ∞ and n ξ 2 t + 1 → ∞ n\xi^{2t+1}\to\infty n ξ 2 t + 1 → ∞ for t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 ,
sup { i , γ ∗ } ∥ ∂ γ ∗ t f ~ ξ − i ( U ^ γ i ; γ ) − f [ t ] ( W i , U γ i ; γ ) ∥ = sup { i , γ ∗ } ∥ 1 n − 1 ∑ { j : j ≠ i } φ j [ t ] ( W i , U γ i ; γ ) ∥ ( 1 + o p ( 1 ) ) \sup_{\{i,\gamma_*\}}\left\lVert\partial_{\gamma_*}^{t}\tilde{f}_{\xi}^{-i}(\widehat{U}_{\gamma i};\gamma)-f^{[t]}(W_i,U_{\gamma i};\gamma)\right\rVert
=\sup_{\{i,\gamma_*\}}\left\lVert\frac{1}{n-1}\sum_{\{j:j\ne i\}}\varphi_j^{[t]}(W_i,U_{\gamma i};\gamma)\right\rVert(1+o_p(1)) { i , γ ∗ } sup ∂ γ ∗ t f ~ ξ − i ( U γ i ; γ ) − f [ t ] ( W i , U γ i ; γ ) = { i , γ ∗ } sup n − 1 1 { j : j = i } ∑ φ j [ t ] ( W i , U γ i ; γ ) ( 1 + o p ( 1 )) = o p ( log n n ξ m 2 t + 1 ) + O ( ξ M 2 ) , =o_p\left(\sqrt{\frac{\log n}{n\xi_m^{2t+1}}}\right)+O(\xi_M^2), = o p ( n ξ m 2 t + 1 log n ) + O ( ξ M 2 ) , where ξ m = min { h , ξ } \xi_m=\min\{h,\xi\} ξ m = min { h , ξ } , ξ M = max { h , ξ } \xi_M=\max\{h,\xi\} ξ M = max { h , ξ } , f ~ ξ − i ( u ; γ ) = ∑ { j : j ≠ i } K ξ ( U ^ γ j − u ) / ( n − 1 ) \tilde{f}_{\xi}^{-i}(u;\gamma)=\sum_{\{j:j\ne i\}}K_{\xi}(\widehat{U}_{\gamma j}-u)/(n-1) f ~ ξ − i ( u ; γ ) = ∑ { j : j = i } K ξ ( U γ j − u ) / ( n − 1 ) , U ^ γ i = Y i − m ^ h − i ( γ T W i ; γ ) \widehat{U}_{\gamma i}=Y_i-\widehat{m}_h^{-i}(\gamma^\mathsf{T}W_i;\gamma) U γ i = Y i − m h − i ( γ T W i ; γ ) , and φ i [ t ] ( w , u ; γ ) \varphi_i^{[t]}(w,u;\gamma) φ i [ t ] ( w , u ; γ ) is defined in (D.28), i = 1 , … , n i=1,\ldots,n i = 1 , … , n .
Proof. For each t = 0 , 1 , 2 t=0,1,2 t = 0 , 1 , 2 , a direct decomposition gives
∂ γ ∗ t f ~ ξ − i ( U ^ γ i ; γ ) − f [ t ] ( W i , U γ i ; γ ) = ( ∂ γ ∗ t f ~ ξ − i ( U ^ γ i ; γ ) − ∂ γ ∗ t f ~ ξ − i ( U γ i ; γ ) ) + ( ∂ γ ∗ t f ~ ξ − i ( U γ i ; γ ) − f [ t ] ( W i , U γ i ; γ ) ) . (43) \partial_{\gamma_*}^{t}\tilde{f}_{\xi}^{-i}(\widehat{U}_{\gamma i};\gamma)-f^{[t]}(W_i,U_{\gamma i};\gamma)
=\left(\partial_{\gamma_*}^{t}\tilde{f}_{\xi}^{-i}(\widehat{U}_{\gamma i};\gamma)-\partial_{\gamma_*}^{t}\tilde{f}_{\xi}^{-i}(U_{\gamma i};\gamma)\right)+\left(\partial_{\gamma_*}^{t}\tilde{f}_{\xi}^{-i}(U_{\gamma i};\gamma)-f^{[t]}(W_i,U_{\gamma i};\gamma)\right). \tag*{(43)} ∂ γ ∗ t f ~ ξ − i ( U γ i ; γ ) − f [ t ] ( W i , U γ i ; γ ) = ( ∂ γ ∗ t f ~ ξ − i ( U γ i ; γ ) − ∂ γ ∗ t f ~ ξ − i ( U γ i ; γ ) ) + ( ∂ γ ∗ t f ~ ξ − i ( U γ i ; γ ) − f [ t ] ( W i , U γ i ; γ ) ) . (43)
Applying a third-order Taylor expansion of K ( ( U ^ γ j − U ^ γ i ) / ξ ) K((\widehat{U}_{\gamma j}-\widehat{U}_{\gamma i})/\xi) K (( U γ j − U γ i ) / ξ ) around U ^ γ j − U ^ γ i = U γ j − U γ i \widehat{U}_{\gamma j}-\widehat{U}_{\gamma i}=U_{\gamma j}-U_{\gamma i} U γ j − U γ i = U γ j − U γ i , the first term on the right-hand side of (D.13) with t = 0 t=0 t = 0 can be written as
f ~ ξ − i ( U ^ γ i ; γ ) − f ~ ξ − i ( U γ i ; γ ) = ∑ { j : j ≠ i } ∑ s = 1 3 1 s ! ( n − 1 ) ξ s + 1 K ( s ) ( U γ j − U γ i ξ ) Δ i j s + ∑ { j : j ≠ i } 1 6 ( n − 1 ) ξ 4 ( K ( 3 ) ( U ^ γ j − U ^ γ i ξ ) − K ( 3 ) ( U γ j − U γ i ξ ) ) Δ i j 3 , (44) \begin{aligned}
\tilde{f}_{\xi}^{-i}(\widehat{U}_{\gamma i};\gamma)-\tilde{f}_{\xi}^{-i}(U_{\gamma i};\gamma)
&=\sum_{\{j:j\ne i\}}\sum_{s=1}^{3}\frac{1}{s!(n-1)\xi^{s+1}}K^{(s)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)\Delta_{ij}^{s} \\
&\quad+\sum_{\{j:j\ne i\}}\frac{1}{6(n-1)\xi^4}\left(K^{(3)}\left(\frac{\widehat{U}_{\gamma j}-\widehat{U}_{\gamma i}}{\xi}\right)-K^{(3)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)\right)\Delta_{ij}^{3}, \tag*{(44)}
\end{aligned} f ~ ξ − i ( U γ i ; γ ) − f ~ ξ − i ( U γ i ; γ ) = { j : j = i } ∑ s = 1 ∑ 3 s ! ( n − 1 ) ξ s + 1 1 K ( s ) ( ξ U γ j − U γ i ) Δ ij s + { j : j = i } ∑ 6 ( n − 1 ) ξ 4 1 ( K ( 3 ) ( ξ U γ j − U γ i ) − K ( 3 ) ( ξ U γ j − U γ i ) ) Δ ij 3 , (44)
where
Δ i j = ( m ^ h − i ( γ T W i ; γ ) − m ^ h − j ( γ T W j ; γ ) ) − ( m [ 0 ] ( W i , γ T W i ) − m [ 0 ] ( W j , γ T W j ) ) \Delta_{ij}=\left(\widehat{m}^{-i}_{h}(\gamma^{\mathsf T}W_i;\gamma)-\widehat{m}^{-j}_{h}(\gamma^{\mathsf T}W_j;\gamma)\right)-\left(m^{[0]}(W_i,\gamma^{\mathsf T}W_i)-m^{[0]}(W_j,\gamma^{\mathsf T}W_j)\right) Δ ij = ( m h − i ( γ T W i ; γ ) − m h − j ( γ T W j ; γ ) ) − ( m [ 0 ] ( W i , γ T W i ) − m [ 0 ] ( W j , γ T W j ) )
and U ‾ γ i = Y i − m ‾ ( γ T W i ; γ ) \overline{U}_{\gamma i}=Y_i-\overline{m}(\gamma^{\mathsf T}W_i;\gamma) U γ i = Y i − m ( γ T W i ; γ ) , with m ‾ ( γ T W i ; γ ) \overline{m}(\gamma^{\mathsf T}W_i;\gamma) m ( γ T W i ; γ ) lying on the line segment between m ^ h − i ( γ T W i ; γ ) \widehat{m}^{-i}_{h}(\gamma^{\mathsf T}W_i;\gamma) m h − i ( γ T W i ; γ ) and m [ 0 ] ( W i , γ T W i ) m^{[0]}(W_i,\gamma^{\mathsf T}W_i) m [ 0 ] ( W i , γ T W i ) for each i , j = 1 , … , n i,j=1,\ldots,n i , j = 1 , … , n . By Lemma S2 with t = 0 t=0 t = 0 , the s = 1 s=1 s = 1 term in the first sum on the right-hand side of (D.14) can be expressed as
∑ { j : j ≠ i } 1 ( n − 1 ) ξ 2 K ( 1 ) ( U γ j − U γ i ξ ) Δ i j = − 1 ( n − 1 ) 2 ξ 2 ∑ { ℓ , j : ℓ ≠ j , ℓ ≠ i } K ( 1 ) ( U γ j − U γ i ξ ) ξ ~ ℓ [ 0 ] ( W j , γ ) − 1 ( n − 1 ) ξ 2 ∑ { j : j ≠ i } K ( 1 ) ( U γ j − U γ i ξ ) × E [ ξ ~ ℓ [ 0 ] ( W j , γ ) ∣ W j ] − 1 ( n − 1 ) ξ 2 ∑ { j : j ≠ i } K ( 1 ) ( U γ j − U γ i ξ ) r [ 0 ] ( W j , γ ) − ∂ u f ~ ξ − i ( U i ; γ ) ( 1 n − 1 ∑ { ℓ : ℓ ≠ i } ( ξ ~ ℓ [ 0 ] ( W i , γ ) + r [ 0 ] ( W i , γ ) ) ) , (45) \begin{aligned}
&\sum_{\{j:j\ne i\}}\frac{1}{(n-1)\xi^{2}}K^{(1)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)\Delta_{ij}\\
&=-\frac{1}{(n-1)^{2}\xi^{2}}\sum_{\{\ell,j:\ell\ne j,\ell\ne i\}}K^{(1)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)\widetilde{\xi}^{[0]}_{\ell}(W_j,\gamma)-\frac{1}{(n-1)\xi^{2}}\sum_{\{j:j\ne i\}}K^{(1)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)\\
&\qquad{}\times\mathbb{E}\left[\widetilde{\xi}^{[0]}_{\ell}(W_j,\gamma)\mid W_j\right]-\frac{1}{(n-1)\xi^{2}}\sum_{\{j:j\ne i\}}K^{(1)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)r^{[0]}(W_j,\gamma)\\
&\qquad{}-\partial_u\widetilde{f}_{\xi}^{-i}(U_i;\gamma)\left(\frac{1}{n-1}\sum_{\{\ell:\ell\ne i\}}\left(\widetilde{\xi}^{[0]}_{\ell}(W_i,\gamma)+r^{[0]}(W_i,\gamma)\right)\right), \tag*{(45)}
\end{aligned} { j : j = i } ∑ ( n − 1 ) ξ 2 1 K ( 1 ) ( ξ U γ j − U γ i ) Δ ij = − ( n − 1 ) 2 ξ 2 1 { ℓ , j : ℓ = j , ℓ = i } ∑ K ( 1 ) ( ξ U γ j − U γ i ) ξ ℓ [ 0 ] ( W j , γ ) − ( n − 1 ) ξ 2 1 { j : j = i } ∑ K ( 1 ) ( ξ U γ j − U γ i ) × E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] − ( n − 1 ) ξ 2 1 { j : j = i } ∑ K ( 1 ) ( ξ U γ j − U γ i ) r [ 0 ] ( W j , γ ) − ∂ u f ξ − i ( U i ; γ ) n − 1 1 { ℓ : ℓ = i } ∑ ( ξ ℓ [ 0 ] ( W i , γ ) + r [ 0 ] ( W i , γ ) ) , (45)
where ξ ~ ℓ [ 0 ] ( W j , γ ) = ξ ℓ [ 0 ] ( W j , γ ) − E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] \widetilde{\xi}^{[0]}_{\ell}(W_j,\gamma)=\xi^{[0]}_{\ell}(W_j,\gamma)-\mathbb{E}[\xi^{[0]}_{\ell}(W_j,\gamma)\mid W_j] ξ ℓ [ 0 ] ( W j , γ ) = ξ ℓ [ 0 ] ( W j , γ ) − E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] , 1 ≤ ℓ ≠ j ≤ n 1\le\ell\ne j\le n 1 ≤ ℓ = j ≤ n . The first term in (D.15) is a degenerate second-order U U U -process, as ensured by the identities
E [ 1 ξ 2 K ( 1 ) ( U γ j − u ξ ) ξ ~ ℓ [ 0 ] ( W j , γ ) ∣ W j , U γ j ] = 0 \mathbb{E}\left[\frac{1}{\xi^{2}}K^{(1)}\left(\frac{U_{\gamma j}-u}{\xi}\right)\widetilde{\xi}^{[0]}_{\ell}(W_j,\gamma)\mid W_j,U_{\gamma j}\right]=0 E [ ξ 2 1 K ( 1 ) ( ξ U γ j − u ) ξ ℓ [ 0 ] ( W j , γ ) ∣ W j , U γ j ] = 0
and
E [ 1 ξ 2 K ( 1 ) ( U γ j − u ξ ) ξ ~ ℓ [ 0 ] ( W j , γ ) ∣ W ℓ , U γ ℓ ] = E [ 1 ξ 2 K ( 1 ) ( U γ j − u ξ ) E [ ξ ~ ℓ [ 0 ] ( W j , γ ) ∣ U γ j , W ℓ , U γ ℓ ] ∣ W ℓ , U γ ℓ ] = 0. \begin{aligned}
\mathbb{E}\left[\frac{1}{\xi^{2}}K^{(1)}\left(\frac{U_{\gamma j}-u}{\xi}\right)\widetilde{\xi}^{[0]}_{\ell}(W_j,\gamma)\mid W_\ell,U_{\gamma\ell}\right]\\
=\mathbb{E}\left[\frac{1}{\xi^{2}}K^{(1)}\left(\frac{U_{\gamma j}-u}{\xi}\right)\mathbb{E}\left[\widetilde{\xi}^{[0]}_{\ell}(W_j,\gamma)\mid U_{\gamma j},W_\ell,U_{\gamma\ell}\right]\mid W_\ell,U_{\gamma\ell}\right]=0.
\end{aligned} E [ ξ 2 1 K ( 1 ) ( ξ U γ j − u ) ξ ℓ [ 0 ] ( W j , γ ) ∣ W ℓ , U γ ℓ ] = E [ ξ 2 1 K ( 1 ) ( ξ U γ j − u ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ U γ j , W ℓ , U γ ℓ ] ∣ W ℓ , U γ ℓ ] = 0.
Since the class
{ K ( 1 ) ( U γ j − u ξ ) ξ ~ ℓ [ 0 ] ( W j , γ ) : u ∈ U γ n , γ ∗ ∈ Γ } \left\{K^{(1)}\left(\frac{U_{\gamma j}-u}{\xi}\right)\widetilde{\xi}^{[0]}_{\ell}(W_j,\gamma):u\in\mathcal{U}_{\gamma n},\gamma_*\in\Gamma\right\} { K ( 1 ) ( ξ U γ j − u ) ξ ℓ [ 0 ] ( W j , γ ) : u ∈ U γ n , γ ∗ ∈ Γ }
is Euclidean by Lemma 22 in Nolan and Pollard [21 ] , and
sup { ( u , γ ∗ ) ∈ U γ n × Γ } E [ ( 1 ξ 2 K ( 1 ) ( U γ j − u ξ ) ξ ~ ℓ [ 0 ] ( W j , γ ) ) 2 ] = O ( 1 h ξ 3 ) , \sup_{\{(u,\gamma_*)\in\mathcal{U}_{\gamma n}\times\Gamma\}}\mathbb{E}\left[\left(\frac{1}{\xi^{2}}K^{(1)}\left(\frac{U_{\gamma j}-u}{\xi}\right)\widetilde{\xi}^{[0]}_{\ell}(W_j,\gamma)\right)^{2}\right]=O\left(\frac{1}{h\xi^{3}}\right), {( u , γ ∗ ) ∈ U γ n × Γ } sup E [ ( ξ 2 1 K ( 1 ) ( ξ U γ j − u ) ξ ℓ [ 0 ] ( W j , γ ) ) 2 ] = O ( h ξ 3 1 ) ,
Corollary 4 in Sherman [28 ] yields
sup { ( u , γ ∗ ) ∈ U γ n × Γ } ∣ 1 ( n − 1 ) ξ 2 ∑ { ℓ , j : ℓ ≠ j } K ( 1 ) ( U γ j − u ξ ) ξ ~ ℓ [ 0 ] ( W j , γ ) ∣ = O p ( 1 n h ξ 3 ) . (46) \sup_{\{(u,\gamma_*)\in\mathcal{U}_{\gamma n}\times\Gamma\}}\left|\frac{1}{(n-1)\xi^{2}}\sum_{\{\ell,j:\ell\ne j\}}K^{(1)}\left(\frac{U_{\gamma j}-u}{\xi}\right)\widetilde{\xi}^{[0]}_{\ell}(W_j,\gamma)\right|=O_p\left(\frac{1}{n\sqrt{h\xi^{3}}}\right). \tag*{(46)} {( u , γ ∗ ) ∈ U γ n × Γ } sup ( n − 1 ) ξ 2 1 { ℓ , j : ℓ = j } ∑ K ( 1 ) ( ξ U γ j − u ) ξ ℓ [ 0 ] ( W j , γ ) = O p ( n h ξ 3 1 ) . (46)
By the definition of ξ ℓ [ 0 ] ( w , γ ) \xi^{[0]}_{\ell}(w,\gamma) ξ ℓ [ 0 ] ( w , γ ) in Lemma S2, we have
E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j = w ] = ( ∂ v 2 M 1 , 0 ( w , γ T w ) + ∂ v f γ ( γ T w ) f γ ( γ T w ) ∂ v M 1 , 0 ( w , γ T w ) ) h 2 ( 1 + o ( 1 ) ) = O ( h 2 ) . \mathbb{E}\left[\xi^{[0]}_{\ell}(W_j,\gamma)\mid W_j=w\right]=\left(\partial_v^2M_{1,0}(w,\gamma^{\mathsf T}w)+\frac{\partial_v f_\gamma(\gamma^{\mathsf T}w)}{f_\gamma(\gamma^{\mathsf T}w)}\partial_vM_{1,0}(w,\gamma^{\mathsf T}w)\right)h^2(1+o(1))=O(h^2). E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j = w ] = ( ∂ v 2 M 1 , 0 ( w , γ T w ) + f γ ( γ T w ) ∂ v f γ ( γ T w ) ∂ v M 1 , 0 ( w , γ T w ) ) h 2 ( 1 + o ( 1 )) = O ( h 2 ) .
Together with assumptions A2, A3 and C2, the first and second moments of the second term on the right-hand side of (D.15) are given by
E [ 1 ξ 2 K ( 1 ) ( U γ j − u ξ ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j = w ] ] = h 2 ∫ ( ∂ v 2 M 1 , 0 ( w , v ) + ( ∂ v log f γ ( γ T w ) ) ∂ v M 1 , 0 ( w , v ) ) ∂ u f ( w , u ; γ ) d w ( 1 + o ( 1 ) ) = O ( h 2 ) \begin{aligned}
\mathbb{E}\left[\frac{1}{\xi^{2}}K^{(1)}\left(\frac{U_{\gamma j}-u}{\xi}\right)\mathbb{E}\left[\xi^{[0]}_{\ell}(W_j,\gamma)\mid W_j=w\right]\right] \\
&=h^2\int\left(\partial_v^2M_{1,0}(w,v)+(\partial_v\log f_\gamma(\gamma^{\mathsf T}w))\partial_vM_{1,0}(w,v)\right)\partial_uf(w,u;\gamma)\,dw(1+o(1))=O(h^2)
\end{aligned} E [ ξ 2 1 K ( 1 ) ( ξ U γ j − u ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j = w ] ] = h 2 ∫ ( ∂ v 2 M 1 , 0 ( w , v ) + ( ∂ v log f γ ( γ T w )) ∂ v M 1 , 0 ( w , v ) ) ∂ u f ( w , u ; γ ) d w ( 1 + o ( 1 )) = O ( h 2 )
and
E [ ( 1 ξ 2 K ( 1 ) ( U γ j − u ξ ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j = w ] ) 2 ] = h 4 ξ 3 ∫ ( ∂ v 2 M 1 , 0 ( w , v ) + ( ∂ v log f γ ( γ T w ) ) ∂ v M 1 , 0 ( w , v ) ) 2 f ( w , u ; γ ) d w ( 1 + o ( 1 ) ) = O ( h 4 ξ 3 ) . \begin{aligned}
\mathbb{E}\left[\left(\frac{1}{\xi^{2}}K^{(1)}\left(\frac{U_{\gamma j}-u}{\xi}\right)\mathbb{E}\left[\xi^{[0]}_{\ell}(W_j,\gamma)\mid W_j=w\right]\right)^2\right] \\
&= \frac{h^{4}}{\xi^{3}} \int\left(\partial_{v}^{2}M_{1,0}(w,v) + \left(\partial_{v}\log f_{\gamma}(\gamma^{\mathsf{T}}w)\right)\partial_{v}M_{1,0}(w,v)\right)^{2} f(w,u;\gamma)\,dw(1+o(1)) = O\left(\frac{h^{4}}{\xi^{3}}\right).
\end{aligned} E [ ( ξ 2 1 K ( 1 ) ( ξ U γ j − u ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j = w ] ) 2 ] = ξ 3 h 4 ∫ ( ∂ v 2 M 1 , 0 ( w , v ) + ( ∂ v log f γ ( γ T w ) ) ∂ v M 1 , 0 ( w , v ) ) 2 f ( w , u ; γ ) d w ( 1 + o ( 1 )) = O ( ξ 3 h 4 ) .
It follows from Corollary 4 in Sherman [28 ] that
sup { i , γ ∗ } ∣ 1 ( n − 1 ) ξ 2 ∑ { j : j ≠ i } K ( 1 ) ( U γ j − U γ i ξ ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j = w ] ∣ = o p ( h 2 n ξ 3 ) + O ( h 2 ) . (47) \sup_{\{i,\gamma_{*}\}}\left|\frac{1}{(n-1)\xi^{2}}\sum_{\{j:j\ne i\}}K^{(1)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)\mathbb{E}\left[\xi_{\ell}^{[0]}(W_{j},\gamma)\mid W_{j}=w\right]\right|=o_{p}\left(\frac{h^{2}}{\sqrt{n\xi^{3}}}\right)+O(h^{2}). \tag*{(47)} { i , γ ∗ } sup ( n − 1 ) ξ 2 1 { j : j = i } ∑ K ( 1 ) ( ξ U γ j − U γ i ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j = w ] = o p ( n ξ 3 h 2 ) + O ( h 2 ) . (47)
By assumption C2 and Lemmas S2 and S3 with t = 0 t=0 t = 0 , the remaining terms on the right-hand side of (D.15) satisfy
sup { i , γ ∗ } ∣ 1 ( n − 1 ) ξ 2 ∑ { j : j ≠ i } K ( 1 ) ( U γ j − U γ i ξ ) r [ 0 ] ( W j , γ ) ∣ = o p ( log n n h ) + O ( h 4 ) (48) \sup_{\{i,\gamma_{*}\}}\left|\frac{1}{(n-1)\xi^{2}}\sum_{\{j:j\ne i\}}K^{(1)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)r^{[0]}(W_{j},\gamma)\right|=o_{p}\left(\frac{\log n}{nh}\right)+O(h^{4}) \tag*{(48)} { i , γ ∗ } sup ( n − 1 ) ξ 2 1 { j : j = i } ∑ K ( 1 ) ( ξ U γ j − U γ i ) r [ 0 ] ( W j , γ ) = o p ( nh log n ) + O ( h 4 ) (48)
( ∂ u f ˉ ξ − i ( U γ i ; γ ) ) ( 1 n − 1 ∑ { ℓ : ℓ ≠ i } ξ ℓ [ 0 ] ( W i , γ ) + r [ 0 ] ( W i , γ ) ) = ( ∂ u f 0 ( U γ i ; γ ) + 1 n − 1 ∑ { j : j ≠ i } ζ j [ 1 ] ( U γ i ; γ ) ) ( 1 n − 1 ∑ { ℓ : ℓ ≠ i } ξ ℓ [ 0 ] ( W i , γ ) + r [ 0 ] ( W i , γ ) ) = 1 n − 1 ∑ { ℓ : ℓ ≠ i } ( ∂ u f 0 ( U γ i ; γ ) ) ξ ℓ [ 0 ] ( W i , γ ) + ( o p ( log n n ξ m 4 ) + O ( ξ M 4 ) ) , (49) \begin{aligned}
\left(\partial_{u}\bar{f}_{\xi}^{-i}(U_{\gamma i};\gamma)\right)\left(\frac{1}{n-1}\sum_{\{\ell:\ell\ne i\}}\xi_{\ell}^{[0]}(W_{i},\gamma)+r^{[0]}(W_{i},\gamma)\right) \\
&= \left(\partial_{u}f_{0}(U_{\gamma i};\gamma)+\frac{1}{n-1}\sum_{\{j:j\ne i\}}\zeta_{j}^{[1]}(U_{\gamma i};\gamma)\right)\left(\frac{1}{n-1}\sum_{\{\ell:\ell\ne i\}}\xi_{\ell}^{[0]}(W_{i},\gamma)+r^{[0]}(W_{i},\gamma)\right) \\
&= \frac{1}{n-1}\sum_{\{\ell:\ell\ne i\}}\left(\partial_{u}f_{0}(U_{\gamma i};\gamma)\right)\xi_{\ell}^{[0]}(W_{i},\gamma)+\left(o_{p}\left(\frac{\log n}{n\xi_{m}^{4}}\right)+O(\xi_{M}^{4})\right), \tag*{(49)}
\end{aligned} ( ∂ u f ˉ ξ − i ( U γ i ; γ ) ) n − 1 1 { ℓ : ℓ = i } ∑ ξ ℓ [ 0 ] ( W i , γ ) + r [ 0 ] ( W i , γ ) = ∂ u f 0 ( U γ i ; γ ) + n − 1 1 { j : j = i } ∑ ζ j [ 1 ] ( U γ i ; γ ) n − 1 1 { ℓ : ℓ = i } ∑ ξ ℓ [ 0 ] ( W i , γ ) + r [ 0 ] ( W i , γ ) = n − 1 1 { ℓ : ℓ = i } ∑ ( ∂ u f 0 ( U γ i ; γ ) ) ξ ℓ [ 0 ] ( W i , γ ) + ( o p ( n ξ m 4 log n ) + O ( ξ M 4 ) ) , (49)
sup { i , γ ∗ } ∣ 1 n − 1 ∑ { j : j ≠ i } ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 0 ] ( W i , γ ) ∣ = o p ( log n n h ) + O ( h 2 ) . \sup_{\{i,\gamma_{*}\}}\left|\frac{1}{n-1}\sum_{\{j:j\ne i\}}\left(\partial_{u}f_{0}(U_{\gamma i};\gamma)\right)\xi_{j}^{[0]}(W_{i},\gamma)\right|=o_{p}\left(\sqrt{\frac{\log n}{nh}}\right)+O(h^{2}). { i , γ ∗ } sup n − 1 1 { j : j = i } ∑ ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 0 ] ( W i , γ ) = o p ( nh log n ) + O ( h 2 ) .
Substituting (D.16)–(D.19) into (D.15), the s = 1 s=1 s = 1 term in the first sum on the right-hand side of (D.14) satisfies
sup { i , γ ∗ } ∣ ∑ { j : j ≠ i } 1 ( n − 1 ) ξ 2 K ( 1 ) ( U γ j − U γ i ξ ) Δ i j ∣ = sup { i , γ ∗ } ∣ ∑ { j : j ≠ i } 1 n − 1 ( ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 0 ] ( W i , γ ) + 1 ξ 2 K ( 1 ) ( U γ j + U γ i ξ ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] ) ∣ = o p ( log n n h ) + O ( h 2 ) . (50) \begin{aligned}
\sup_{\{i,\gamma_{*}\}}\left|\sum_{\{j:j\ne i\}}\frac{1}{(n-1)\xi^{2}}K^{(1)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)\Delta_{ij}\right| \\
&= \sup_{\{i,\gamma_{*}\}}\left|\sum_{\{j:j\ne i\}}\frac{1}{n-1}\left(\left(\partial_{u}f_{0}(U_{\gamma i};\gamma)\right)\xi_{j}^{[0]}(W_{i},\gamma)+\frac{1}{\xi^{2}}K^{(1)}\left(\frac{U_{\gamma j}+U_{\gamma i}}{\xi}\right)\mathbb{E}\left[\xi_{\ell}^{[0]}(W_{j},\gamma)\mid W_{j}\right]\right)\right| \\
&=o_{p}\left(\sqrt{\frac{\log n}{nh}}\right)+O(h^{2}). \tag*{(50)}
\end{aligned} { i , γ ∗ } sup { j : j = i } ∑ ( n − 1 ) ξ 2 1 K ( 1 ) ( ξ U γ j − U γ i ) Δ ij = { i , γ ∗ } sup { j : j = i } ∑ n − 1 1 ( ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 0 ] ( W i , γ ) + ξ 2 1 K ( 1 ) ( ξ U γ j + U γ i ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] ) = o p ( nh log n ) + O ( h 2 ) . (50)
Analogous arguments applied to the s = 2 , 3 s=2,3 s = 2 , 3 terms in the first sum on the right-hand side of (D.14) yield
sup { i , γ ∗ } ∣ ∑ { j : j ≠ i } 1 s ! ( n − 1 ) ξ s + 1 K ( s ) ( U γ j − U γ i ξ ) Δ i j s ∣ = ( o p ( log n n h ) + O ( h 2 ) ) s . (51) \sup_{\{i,\gamma_{*}\}}\left|\sum_{\{j:j\ne i\}}\frac{1}{s!(n-1)\xi^{s+1}}K^{(s)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)\Delta_{ij}^{s}\right|=\left(o_{p}\left(\sqrt{\frac{\log n}{nh}}\right)+O(h^{2})\right)^{s}. \tag*{(51)} { i , γ ∗ } sup { j : j = i } ∑ s ! ( n − 1 ) ξ s + 1 1 K ( s ) ( ξ U γ j − U γ i ) Δ ij s = ( o p ( nh log n ) + O ( h 2 ) ) s . (51)
The uniform boundedness of K ( 3 ) K^{(3)} K ( 3 ) implies that the last term on the right-hand side of (D.14) satisfies
sup { i , γ ∗ } ∣ ∑ { j : j ≠ i } 1 6 ( n − 1 ) ξ 4 ( K ( 3 ) ( U ‾ γ j − U ‾ γ i ξ ) − K ( 3 ) ( U γ j − U γ i ξ ) ) Δ i j 3 ∣ ≤ O ( 1 ξ 4 ( sup { i , j , γ ∗ } ∣ Δ i j ∣ ) 3 ) = o p ( 1 ξ 4 ( log n n h ) 3 2 ) + O ( h 6 ξ 4 ) . (52) \begin{aligned}
\sup_{\{i,\gamma_{*}\}}\left|\sum_{\{j:j\ne i\}}\frac{1}{6(n-1)\xi^{4}}\left(K^{(3)}\left(\frac{\overline{U}_{\gamma j}-\overline{U}_{\gamma i}}{\xi}\right)-K^{(3)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)\right)\Delta_{ij}^{3}\right| \\
&\le O\left(\frac{1}{\xi^{4}}\left(\sup_{\{i,j,\gamma_{*}\}}|\Delta_{ij}|\right)^{3}\right)=o_{p}\left(\frac{1}{\xi^{4}}\left(\frac{\log n}{nh}\right)^{\frac{3}{2}}\right)+O\left(\frac{h^{6}}{\xi^{4}}\right). \tag*{(52)}
\end{aligned} { i , γ ∗ } sup { j : j = i } ∑ 6 ( n − 1 ) ξ 4 1 ( K ( 3 ) ( ξ U γ j − U γ i ) − K ( 3 ) ( ξ U γ j − U γ i ) ) Δ ij 3 ≤ O ξ 4 1 ( { i , j , γ ∗ } sup ∣ Δ ij ∣ ) 3 = o p ( ξ 4 1 ( nh log n ) 2 3 ) + O ( ξ 4 h 6 ) . (52)
Substituting (D.20)–(D.22) into (D.14) gives
sup { i , γ ∗ } ∣ f ˉ ξ − i ( U ^ γ i ; γ ) − f ˉ ξ − i ( U γ i ; γ ) ∣ = sup { i , γ ∗ } ∥ 1 n − 1 ∑ { j : j ≠ i } ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 0 ] ( W i , γ ) − E [ 1 ξ 2 K ( 1 ) ( U γ j − U γ i ξ ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] ∣ U γ i ] ∥ \begin{aligned}
\sup_{\{i,\gamma_{*}\}}\left|\bar{f}_{\xi}^{-i}(\widehat{U}_{\gamma i};\gamma)-\bar{f}_{\xi}^{-i}(U_{\gamma i};\gamma)\right| \\
&= \sup_{\{i,\gamma^*\}} \left\| \frac{1}{n-1}\sum_{\{j:j\ne i\}} \left(\partial_u f_0(U_{\gamma_i};\gamma)\right)\xi_j^{[0]}(W_i,\gamma)-\mathbb{E}\left[\frac{1}{\xi^2}K^{(1)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\mathbb{E}\left[\xi_\ell^{[0]}(W_j,\gamma)\mid W_j\right]\mid U_{\gamma_i}\right]\right\|
\end{aligned} { i , γ ∗ } sup f ˉ ξ − i ( U γ i ; γ ) − f ˉ ξ − i ( U γ i ; γ ) = { i , γ ∗ } sup n − 1 1 { j : j = i } ∑ ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 0 ] ( W i , γ ) − E [ ξ 2 1 K ( 1 ) ( ξ U γ j − U γ i ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] ∣ U γ i ]
≜ sup { i , γ ∗ } ∣ 1 n − 1 ∑ { j : j ≠ i } ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 0 ] ( W i , γ ) − ψ [ 0 ] ( W i , U γ i ; γ ) ∣ = O p ( log n n h ) + O ( h 2 ) . (53) \triangleq\sup_{\{i,\gamma^*\}} \left|\frac{1}{n-1}\sum_{\{j:j\ne i\}} \left(\partial_u f_0(U_{\gamma_i};\gamma)\right)\xi_j^{[0]}(W_i,\gamma)-\psi^{[0]}(W_i,U_{\gamma_i};\gamma)\right|=O_p\left(\sqrt{\frac{\log n}{nh}}\right)+O(h^2). \tag*{(53)} ≜ { i , γ ∗ } sup n − 1 1 { j : j = i } ∑ ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 0 ] ( W i , γ ) − ψ [ 0 ] ( W i , U γ i ; γ ) = O p ( nh log n ) + O ( h 2 ) . (53)
Applying a third-order Taylor expansion of ∂ γ ∗ t K ( ( U ^ γ j − U ^ γ i ) / ξ ) \partial_{\gamma^*}^t K((\widehat{U}_{\gamma_j}-\widehat{U}_{\gamma_i})/\xi) ∂ γ ∗ t K (( U γ j − U γ i ) / ξ ) around U ^ γ j − U ^ γ i = U γ j − U γ i \widehat{U}_{\gamma_j}-\widehat{U}_{\gamma_i}=U_{\gamma_j}-U_{\gamma_i} U γ j − U γ i = U γ j − U γ i , the first term on the right-hand side of (D.13) with t = 1 , 2 t=1,2 t = 1 , 2 can be written as
∂ γ ∗ f ~ ξ − i ( U ^ γ i ; γ ) − ∂ γ ∗ f ‾ ξ − i ( U γ i ; γ ) = 1 n − 1 ∑ { j : j ≠ i } 1 ξ 2 K ( 1 ) ( U γ j − U γ i ξ ) ( 1 n − 1 ∑ { ℓ : ℓ ≠ i } ξ ℓ [ 1 ] ( W i , γ ) + r [ 1 ] ( W i , γ ) − 1 n − 1 ∑ { ℓ : ℓ ≠ j } ξ ℓ [ 1 ] ( W j , γ ) − r [ 1 ] ( W j , γ ) \begin{aligned}
\partial_{\gamma^*}\widetilde{f}^{-i}_{\xi}(\widehat{U}_{\gamma_i};\gamma)-\partial_{\gamma^*}\overline{f}^{-i}_{\xi}(U_{\gamma_i};\gamma) \\
&=\frac{1}{n-1}\sum_{\{j:j\ne i\}}\frac{1}{\xi^2}K^{(1)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\left(\frac{1}{n-1}\sum_{\{\ell:\ell\ne i\}}\xi_{\ell}^{[1]}(W_i,\gamma)+r^{[1]}(W_i,\gamma)-\frac{1}{n-1}\sum_{\{\ell:\ell\ne j\}}\xi_{\ell}^{[1]}(W_j,\gamma)-r^{[1]}(W_j,\gamma)\right.
\end{aligned} ∂ γ ∗ f ξ − i ( U γ i ; γ ) − ∂ γ ∗ f ξ − i ( U γ i ; γ ) = n − 1 1 { j : j = i } ∑ ξ 2 1 K ( 1 ) ( ξ U γ j − U γ i ) n − 1 1 { ℓ : ℓ = i } ∑ ξ ℓ [ 1 ] ( W i , γ ) + r [ 1 ] ( W i , γ ) − n − 1 1 { ℓ : ℓ = j } ∑ ξ ℓ [ 1 ] ( W j , γ ) − r [ 1 ] ( W j , γ )
+ 1 n − 1 ∑ { j : j ≠ i } ∑ s = 1 3 1 s ! ξ s + 2 K ( s + 1 ) ( U γ j − U γ i ξ ) Δ i j s ( ∂ γ ∗ m ^ h − i ( γ ⊤ W i ; γ ) − ∂ γ ∗ m ^ h − j ( γ ⊤ W j ; γ ) ) ) \left.+\frac{1}{n-1}\sum_{\{j:j\ne i\}}\sum_{s=1}^{3}\frac{1}{s!\xi^{s+2}}K^{(s+1)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\Delta_{ij}^{s}\left(\partial_{\gamma^*}\widehat{m}_h^{-i}(\gamma^\top W_i;\gamma)-\partial_{\gamma^*}\widehat{m}_h^{-j}(\gamma^\top W_j;\gamma)\right)\right) + n − 1 1 { j : j = i } ∑ s = 1 ∑ 3 s ! ξ s + 2 1 K ( s + 1 ) ( ξ U γ j − U γ i ) Δ ij s ( ∂ γ ∗ m h − i ( γ ⊤ W i ; γ ) − ∂ γ ∗ m h − j ( γ ⊤ W j ; γ ) )
− ∂ γ ∗ m ^ h − j ( γ ⊤ W j ; γ ) ) . (54) \qquad\qquad-\partial_{\gamma^*}\widehat{m}_h^{-j}(\gamma^\top W_j;\gamma)). \tag*{(54)} − ∂ γ ∗ m h − j ( γ ⊤ W j ; γ )) . (54)
∂ γ ∗ 2 f ~ ξ − i ( U ^ γ i ; γ ) − ∂ γ ∗ 2 f ‾ ξ − i ( U γ i ; γ ) = 1 n − 1 ∑ { j : j ≠ i } 1 ξ 3 K ( 2 ) ( U γ j − U γ i ξ ) ( 1 n − 1 ∑ { ℓ : ℓ ≠ i } ξ ℓ [ 1 ] ( W i , γ ) + r [ 1 ] ( W i , γ ) − 1 n − 1 ∑ { ℓ : ℓ ≠ j } ξ ℓ [ 1 ] ( W j , γ ) − r [ 1 ] ( W j , γ ) ) ⊗ 2 + 2 sym ( 1 n − 1 ∑ { j : j ≠ i } 1 ξ 3 K ( 2 ) ( U γ j − U γ i ξ ) ( 1 n − 1 ∑ { ℓ : ℓ ≠ i } ξ ℓ [ 1 ] ( W i , γ ) + r [ 1 ] ( W i , γ ) − 1 n − 1 ∑ { ℓ : ℓ ≠ j } ξ ℓ [ 1 ] ( W j , γ ) − r [ 1 ] ( W j , γ ) ) ( m [ 1 ] ( W i , γ ⊤ W i ) − m [ 1 ] ( W j , γ ⊤ W j ) ) ⊤ ) + 1 n − 1 ∑ { j : j ≠ i } 1 ξ 2 K ( 1 ) ( U γ j − U γ i ξ ) ( 1 n − 1 ∑ { ℓ : ℓ ≠ i } ξ ℓ [ 2 ] ( W i , γ ) + r [ 2 ] ( W i , γ ) − 1 n − 1 ∑ { ℓ : ℓ ≠ j } ξ ℓ [ 2 ] ( W j , γ ) − r [ 2 ] ( W j , γ ) ) + 1 n − 1 ∑ { j : j ≠ i } ∑ s = 1 3 1 s ! ξ s + 3 K ( s + 2 ) ( U γ j − U γ i ξ ) Δ i j s ( ∂ γ ∗ m ^ h − i ( γ ⊤ W i ; γ ) − ∂ γ ∗ m ^ h − j ( γ ⊤ W j ; γ ) ) ⊗ 2 + 1 n − 1 ∑ { j : j ≠ i } ∑ s = 1 3 1 s ! ξ s + 2 K ( s + 1 ) ( U γ j − U γ i ξ ) Δ i j s ( ∂ γ ∗ 2 m ^ h − i ( γ ⊤ W i ; γ ) − ∂ γ ∗ 2 m ^ h − j ( γ ⊤ W j ; γ ) ) . (55) \begin{aligned}
\partial_{\gamma^*}^2\widetilde{f}^{-i}_{\xi}(\widehat{U}_{\gamma_i};\gamma)-\partial_{\gamma^*}^2\overline{f}^{-i}_{\xi}(U_{\gamma_i};\gamma) \\
&=\frac{1}{n-1}\sum_{\{j:j\ne i\}}\frac{1}{\xi^3}K^{(2)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\left(\frac{1}{n-1}\sum_{\{\ell:\ell\ne i\}}\xi_{\ell}^{[1]}(W_i,\gamma)+r^{[1]}(W_i,\gamma)-\frac{1}{n-1}\sum_{\{\ell:\ell\ne j\}}\xi_{\ell}^{[1]}(W_j,\gamma)-r^{[1]}(W_j,\gamma)\right)^{\otimes2} \\
&+2\operatorname{sym}\left(\frac{1}{n-1}\sum_{\{j:j\ne i\}}\frac{1}{\xi^3}K^{(2)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\left(\frac{1}{n-1}\sum_{\{\ell:\ell\ne i\}}\xi_{\ell}^{[1]}(W_i,\gamma)+r^{[1]}(W_i,\gamma)-\frac{1}{n-1}\sum_{\{\ell:\ell\ne j\}}\xi_{\ell}^{[1]}(W_j,\gamma)-r^{[1]}(W_j,\gamma)\right)\left(m^{[1]}(W_i,\gamma^\top W_i)-m^{[1]}(W_j,\gamma^\top W_j)\right)^\top\right) \\
&+\frac{1}{n-1}\sum_{\{j:j\ne i\}}\frac{1}{\xi^2}K^{(1)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\left(\frac{1}{n-1}\sum_{\{\ell:\ell\ne i\}}\xi_{\ell}^{[2]}(W_i,\gamma)+r^{[2]}(W_i,\gamma)-\frac{1}{n-1}\sum_{\{\ell:\ell\ne j\}}\xi_{\ell}^{[2]}(W_j,\gamma)-r^{[2]}(W_j,\gamma)\right) \\
&+\frac{1}{n-1}\sum_{\{j:j\ne i\}}\sum_{s=1}^{3}\frac{1}{s!\xi^{s+3}}K^{(s+2)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\Delta_{ij}^{s}\left(\partial_{\gamma^*} \widehat{m}_h^{-i}(\gamma^\top W_i;\gamma)-\partial_{\gamma^*}\widehat{m}_h^{-j}(\gamma^\top W_j;\gamma)\right)^{\otimes2} \\
&+\frac{1}{n-1}\sum_{\{j:j\ne i\}}\sum_{s=1}^{3}\frac{1}{s!\xi^{s+2}}K^{(s+1)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\Delta_{ij}^{s}\left(\partial_{\gamma^*}^2\widehat{m}_h^{-i}(\gamma^\top W_i;\gamma)-\partial_{\gamma^*}^2\widehat{m}_h^{-j}(\gamma^\top W_j;\gamma)\right). \tag*{(55)}
\end{aligned} ∂ γ ∗ 2 f ξ − i ( U γ i ; γ ) − ∂ γ ∗ 2 f ξ − i ( U γ i ; γ ) = n − 1 1 { j : j = i } ∑ ξ 3 1 K ( 2 ) ( ξ U γ j − U γ i ) n − 1 1 { ℓ : ℓ = i } ∑ ξ ℓ [ 1 ] ( W i , γ ) + r [ 1 ] ( W i , γ ) − n − 1 1 { ℓ : ℓ = j } ∑ ξ ℓ [ 1 ] ( W j , γ ) − r [ 1 ] ( W j , γ ) ⊗ 2 + 2 sym n − 1 1 { j : j = i } ∑ ξ 3 1 K ( 2 ) ( ξ U γ j − U γ i ) n − 1 1 { ℓ : ℓ = i } ∑ ξ ℓ [ 1 ] ( W i , γ ) + r [ 1 ] ( W i , γ ) − n − 1 1 { ℓ : ℓ = j } ∑ ξ ℓ [ 1 ] ( W j , γ ) − r [ 1 ] ( W j , γ ) ( m [ 1 ] ( W i , γ ⊤ W i ) − m [ 1 ] ( W j , γ ⊤ W j ) ) ⊤ + n − 1 1 { j : j = i } ∑ ξ 2 1 K ( 1 ) ( ξ U γ j − U γ i ) n − 1 1 { ℓ : ℓ = i } ∑ ξ ℓ [ 2 ] ( W i , γ ) + r [ 2 ] ( W i , γ ) − n − 1 1 { ℓ : ℓ = j } ∑ ξ ℓ [ 2 ] ( W j , γ ) − r [ 2 ] ( W j , γ ) + n − 1 1 { j : j = i } ∑ s = 1 ∑ 3 s ! ξ s + 3 1 K ( s + 2 ) ( ξ U γ j − U γ i ) Δ ij s ( ∂ γ ∗ m h − i ( γ ⊤ W i ; γ ) − ∂ γ ∗ m h − j ( γ ⊤ W j ; γ ) ) ⊗ 2 + n − 1 1 { j : j = i } ∑ s = 1 ∑ 3 s ! ξ s + 2 1 K ( s + 1 ) ( ξ U γ j − U γ i ) Δ ij s ( ∂ γ ∗ 2 m h − i ( γ ⊤ W i ; γ ) − ∂ γ ∗ 2 m h − j ( γ ⊤ W j ; γ ) ) . (55)
Paralleling to the arguments for (D.23), the uniform convergence rates of ∂ γ ∗ t f ~ ξ − i ( U ^ γ i ; γ ) − ∂ γ ∗ t f ‾ ξ − i ( U γ i ; γ ) \partial_{\gamma^*}^t\widetilde{f}^{-i}_{\xi}(\widehat{U}_{\gamma_i};\gamma)-\partial_{\gamma^*}^t\overline{f}^{-i}_{\xi}(U_{\gamma_i};\gamma) ∂ γ ∗ t f ξ − i ( U γ i ; γ ) − ∂ γ ∗ t f ξ − i ( U γ i ; γ ) for t = 1 , 2 t=1,2 t = 1 , 2 follow. Specifically,
sup { i , γ ∗ } ∥ ∂ γ ∗ f ~ ξ − i ( U ^ γ i ; γ ) − ∂ γ ∗ f ‾ ξ − i ( U γ i ; γ ) ∥ = sup { i , γ ∗ } ∥ 1 n − 1 ∑ { j : j ≠ i } ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 1 ] ( W i , γ ) − E [ 1 ξ 2 K ( 1 ) ( U γ j − U γ i ξ ) E [ ξ ℓ [ 1 ] ( W j , γ ) ∣ W j ] \begin{aligned}
\sup_{\{i,\gamma^*\}}\left\|\partial_{\gamma^*}\widetilde{f}^{-i}_{\xi}(\widehat{U}_{\gamma_i};\gamma)-\partial_{\gamma^*}\overline{f}^{-i}_{\xi}(U_{\gamma_i};\gamma)\right\| \\
&=\sup_{\{i,\gamma^*\}}\left\|\frac{1}{n-1}\sum_{\{j:j\ne i\}}\left(\partial_u f_0(U_{\gamma_i};\gamma)\right)\xi_j^{[1]}(W_i,\gamma)-\mathbb{E}\left[\frac{1}{\xi^2}K^{(1)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\mathbb{E}\left[\xi_\ell^{[1]}(W_j,\gamma)\mid W_j\right]\right.\right.
\end{aligned} { i , γ ∗ } sup ∂ γ ∗ f ξ − i ( U γ i ; γ ) − ∂ γ ∗ f ξ − i ( U γ i ; γ ) = { i , γ ∗ } sup n − 1 1 { j : j = i } ∑ ( ∂ u f 0 ( U γ i ; γ ) ) ξ j [ 1 ] ( W i , γ ) − E [ ξ 2 1 K ( 1 ) ( ξ U γ j − U γ i ) E [ ξ ℓ [ 1 ] ( W j , γ ) ∣ W j ]
+ ( ∂ u 2 f 0 ( U γ i ; γ ) ) E [ ξ j [ 0 ] ( W i , γ ) ∣ W i ] m [ 1 ] ( W i , γ ⊤ W i ) ∣ U γ i ] \left.\left.+\left(\partial_u^2f_0(U_{\gamma_i};\gamma)\right)\mathbb{E}\left[\xi_j^{[0]}(W_i,\gamma)\mid W_i\right]m^{[1]}(W_i,\gamma^\top W_i)\mid U_{\gamma_i}\right]\right. + ( ∂ u 2 f 0 ( U γ i ; γ ) ) E [ ξ j [ 0 ] ( W i , γ ) ∣ W i ] m [ 1 ] ( W i , γ ⊤ W i ) ∣ U γ i ]
− E [ 1 ξ 3 K ( 2 ) ( U γ j − U γ i ξ ) E [ ξ ℓ [ 0 ] ( W i , γ ) ∣ W i ] m [ 1 ] ( W j , γ ⊤ W j ) ∣ W i , U γ i ] ∥ + E [ 1 ξ 3 K ( 2 ) ( U γ j − U γ i ξ ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] ( m [ 1 ] ( W j , γ T W j ) − m [ 1 ] ( W i , γ T W i ) ) ∣ W i , U γ i ] ≜ sup { i , γ ∗ } ∥ 1 n − 1 ∑ { j : j ≠ i } ( ∂ u f 0 ( U γ i ; γ ) ξ j [ 1 ] ( W i , γ ) − ψ [ 1 ] ( W i , U γ i ; γ ) ) ∥ = o p ( log n n h 3 ) + O ( h 2 ) (56) \begin{aligned}
\left.\left.-\mathbb{E}\left[\frac{1}{\xi^3}K^{(2)}\left(\frac{U_{\gamma_j}-U_{\gamma_i}}{\xi}\right)\mathbb{E}\left[\xi_\ell^{[0]}(W_i,\gamma)\mid W_i\right]m^{[1]}(W_j,\gamma^\top W_j)\mid W_i,U_{\gamma_i}\right]\right\|\right. \\
&+ E\left[\frac{1}{\xi^{3}}K^{(2)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)E\left[\xi_{\ell}^{[0]}(W_j,\gamma)\mid W_j\right]\left(m^{[1]}(W_j,\gamma^\mathsf{T}W_j)-m^{[1]}(W_i,\gamma^\mathsf{T}W_i)\right)\mid W_i,U_{\gamma i}\right] \\
\mathrel{\triangleq} \sup_{\{i,\gamma_*\}}\left\|\frac{1}{n-1}\sum_{\{j:j\ne i\}}\left(\partial_u f_0(U_{\gamma i};\gamma)\xi_j^{[1]}(W_i,\gamma)-\psi^{[1]}(W_i,U_{\gamma i};\gamma)\right)\right\|=o_p\left(\sqrt{\frac{\log n}{nh^3}}\right)+O(h^2) \tag*{(56)}
\end{aligned} − E [ ξ 3 1 K ( 2 ) ( ξ U γ j − U γ i ) E [ ξ ℓ [ 0 ] ( W i , γ ) ∣ W i ] m [ 1 ] ( W j , γ ⊤ W j ) ∣ W i , U γ i ] ≜ { i , γ ∗ } sup n − 1 1 { j : j = i } ∑ ( ∂ u f 0 ( U γ i ; γ ) ξ j [ 1 ] ( W i , γ ) − ψ [ 1 ] ( W i , U γ i ; γ ) ) = o p ( n h 3 log n ) + O ( h 2 ) + E [ ξ 3 1 K ( 2 ) ( ξ U γ j − U γ i ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] ( m [ 1 ] ( W j , γ T W j ) − m [ 1 ] ( W i , γ T W i ) ) ∣ W i , U γ i ] (56)
and
sup { i , γ ∗ } ∣ ∂ γ ∗ 2 f ~ ξ − i ( U γ i ; γ ) − ∂ γ ∗ 2 f ξ − i ( U γ i ; γ ) ∣ = sup { i , γ ∗ } ∣ 1 n − 1 ∑ { j : j ≠ i } ( ∂ u f 0 ( U γ i ; γ ) ξ j [ 2 ] ( W i , γ ) + ∂ u 2 f 0 ( U γ i ; γ ) ( E [ ξ j [ 1 ] ( W i , γ ) ∣ W i ] ( m [ 1 ] ( W i , γ T W i ) ) T + m [ 1 ] ( W i , γ T W i ) ( E [ ξ j [ 1 ] ( W i , γ ) ∣ W i ] ) T \begin{aligned}
\sup_{\{i,\gamma_*\}}\left|\partial_{\gamma_*}^2\tilde f_{\xi^{-i}}(U_{\gamma i};\gamma)-\partial_{\gamma_*}^2f_{\xi^{-i}}(U_{\gamma i};\gamma)\right| \\
&= \sup_{\{i,\gamma_*\}}\left|\frac{1}{n-1}\sum_{\{j:j\ne i\}}\left(\partial_u f_0(U_{\gamma i};\gamma)\xi_j^{[2]}(W_i,\gamma)+\partial_u^2f_0(U_{\gamma i};\gamma)\left(E[\xi_j^{[1]}(W_i,\gamma)\mid W_i]\left(m^{[1]}(W_i,\gamma^\mathsf{T}W_i)\right)^\mathsf{T}+m^{[1]}(W_i,\gamma^\mathsf{T}W_i)\left(E[\xi_j^{[1]}(W_i,\gamma)\mid W_i]\right)^\mathsf{T}\right.\right.\right.
\end{aligned} { i , γ ∗ } sup ∂ γ ∗ 2 f ~ ξ − i ( U γ i ; γ ) − ∂ γ ∗ 2 f ξ − i ( U γ i ; γ ) = { i , γ ∗ } sup n − 1 1 { j : j = i } ∑ ( ∂ u f 0 ( U γ i ; γ ) ξ j [ 2 ] ( W i , γ ) + ∂ u 2 f 0 ( U γ i ; γ ) ( E [ ξ j [ 1 ] ( W i , γ ) ∣ W i ] ( m [ 1 ] ( W i , γ T W i ) ) T + m [ 1 ] ( W i , γ T W i ) ( E [ ξ j [ 1 ] ( W i , γ ) ∣ W i ] ) T
+ E [ ξ j [ 0 ] ( W i , γ ) ∣ W i ] m [ 2 ] ( W i , γ T W i ) ) + ∂ u 3 f 0 ( U γ i ; γ ) E [ ξ [ 0 ] ( W i , γ ) ∣ W i ] ( m [ 1 ] ( W i , γ T W i ) ) ⊗ 2 \left.\left.\left.\qquad+E[\xi_j^{[0]}(W_i,\gamma)\mid W_i]m^{[2]}(W_i,\gamma^\mathsf{T}W_i)\right)+\partial_u^3f_0(U_{\gamma i};\gamma)E[\xi^{[0]}(W_i,\gamma)\mid W_i]\left(m^{[1]}(W_i,\gamma^\mathsf{T}W_i)\right)^{\otimes2}\right.\right. + E [ ξ j [ 0 ] ( W i , γ ) ∣ W i ] m [ 2 ] ( W i , γ T W i ) ) + ∂ u 3 f 0 ( U γ i ; γ ) E [ ξ [ 0 ] ( W i , γ ) ∣ W i ] ( m [ 1 ] ( W i , γ T W i ) ) ⊗ 2
− E [ 1 ξ 3 K ( 2 ) ( U γ j − U γ i ξ ) ( ξ ℓ [ 0 ] ( W i , γ ) m [ 2 ] ( W j , γ T W j ) + 2 sym ( m [ 1 ] ( W j , γ T W j ) ( ξ ℓ [ 1 ] ( W i , γ ) ) T ) ) ∣ W i , U γ i ] E [ 1 ξ 3 K ( 2 ) ( U γ j − U γ i ξ ) \qquad-E[\frac{1}{\xi^3}K^{(2)}(\frac{U_{\gamma j}-U_{\gamma i}}{\xi})(\xi_\ell^{[0]}(W_i,\gamma)m^{[2]}(W_j,\gamma^\mathsf{T}W_j)+2\operatorname{sym}(m^{[1]}(W_j,\gamma^\mathsf{T}W_j)(\xi_\ell^{[1]}(W_i,\gamma))^\mathsf{T}))\mid W_i,U_{\gamma i}]E[\frac{1}{\xi^3}K^{(2)}(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}) − E [ ξ 3 1 K ( 2 ) ( ξ U γ j − U γ i ) ( ξ ℓ [ 0 ] ( W i , γ ) m [ 2 ] ( W j , γ T W j ) + 2 sym ( m [ 1 ] ( W j , γ T W j ) ( ξ ℓ [ 1 ] ( W i , γ ) ) T )) ∣ W i , U γ i ] E [ ξ 3 1 K ( 2 ) ( ξ U γ j − U γ i )
( m [ 1 ] ( W j , γ T W j ) − m [ 1 ] ( W i , γ T W i ) ) ( E [ ξ j [ 1 ] ( W j , γ ) ∣ W j ] ) T + ( E [ ξ j [ 0 ] ( W j , γ ) ∣ W j ] ) \left.\left.\qquad\left(m^{[1]}(W_j,\gamma^\mathsf{T}W_j)-m^{[1]}(W_i,\gamma^\mathsf{T}W_i)\right)\left(E[\xi_j^{[1]}(W_j,\gamma)\mid W_j]\right)^\mathsf{T}+\left(E[\xi_j^{[0]}(W_j,\gamma)\mid W_j]\right)\right.\right. ( m [ 1 ] ( W j , γ T W j ) − m [ 1 ] ( W i , γ T W i ) ) ( E [ ξ j [ 1 ] ( W j , γ ) ∣ W j ] ) T + ( E [ ξ j [ 0 ] ( W j , γ ) ∣ W j ] )
( m [ 2 ] ( W j , γ T W j ) − m [ 2 ] ( W i , γ T W i ) ) ∣ W i , U γ i ] − E [ 1 ξ 2 K ( 1 ) ( U γ j − U γ i ξ ) E [ ξ ℓ [ 2 ] ( W j , γ ) ∣ W j ] ∣ W i , U γ i \left.\left.\qquad\left(m^{[2]}(W_j,\gamma^\mathsf{T}W_j)-m^{[2]}(W_i,\gamma^\mathsf{T}W_i)\right)\mid W_i,U_{\gamma i}\right]-E\left[\frac{1}{\xi^2}K^{(1)}\left(\frac{U_{\gamma j}-U_{\gamma i}}{\xi}\right)E[\xi_\ell^{[2]}(W_j,\gamma)\mid W_j]\mid W_i,U_{\gamma i}\right.\right. ( m [ 2 ] ( W j , γ T W j ) − m [ 2 ] ( W i , γ T W i ) ) ∣ W i , U γ i ] − E [ ξ 2 1 K ( 1 ) ( ξ U γ j − U γ i ) E [ ξ ℓ [ 2 ] ( W j , γ ) ∣ W j ] ∣ W i , U γ i
U γ i − E [ 1 ξ 4 K ( 3 ) ( U γ j − U γ i ξ ) ( m [ 1 ] ( W i , γ T W i ) − m [ 1 ] ( W j , γ T W j ) ) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] ∣ W i , U γ i ] \qquad U_{\gamma i}-E[\frac{1}{\xi^4}K^{(3)}(\frac{U_{\gamma j}-U_{\gamma i}}{\xi})(m^{[1]}(W_i,\gamma^\mathsf{T}W_i)-m^{[1]}(W_j,\gamma^\mathsf{T}W_j))E[\xi_\ell^{[0]}(W_j,\gamma)\mid W_j]\mid W_i,U_{\gamma i}] U γ i − E [ ξ 4 1 K ( 3 ) ( ξ U γ j − U γ i ) ( m [ 1 ] ( W i , γ T W i ) − m [ 1 ] ( W j , γ T W j )) E [ ξ ℓ [ 0 ] ( W j , γ ) ∣ W j ] ∣ W i , U γ i ]
− E [ 1 ξ 4 K ( 3 ) ( U γ j − U γ i ξ ) ( ( m [ 1 ] ( W j , γ T W j ) ) ⊗ 2 − 2 sym ( m [ 1 ] ( W i , γ T W i ) ( m [ 1 ] ( W j , γ T W j ) ) T ) ) \qquad-E[\frac{1}{\xi^4}K^{(3)}(\frac{U_{\gamma j}-U_{\gamma i}}{\xi})((m^{[1]}(W_j,\gamma^\mathsf{T}W_j))^{\otimes2}-2\operatorname{sym}(m^{[1]}(W_i,\gamma^\mathsf{T}W_i)(m^{[1]}(W_j,\gamma^\mathsf{T}W_j))^\mathsf{T})) − E [ ξ 4 1 K ( 3 ) ( ξ U γ j − U γ i ) (( m [ 1 ] ( W j , γ T W j ) ) ⊗ 2 − 2 sym ( m [ 1 ] ( W i , γ T W i ) ( m [ 1 ] ( W j , γ T W j ) ) T ))
ξ ℓ [ 0 ] ( W i , γ ) ∣ W i , U γ i ] ∥ ≜ sup { i , γ ∗ } ∥ 1 n − 1 ∑ { j : j ≠ i } ( ∂ u f 0 ( U γ i ; γ ) ξ j [ 2 ] ( W i , γ ) − ψ [ 2 ] ( W i , U γ i ; γ ) ) ∥ = o p ( log n n h 5 ) + O ( h 2 ) . (57) \left.\left.\qquad\left.\left.\xi_\ell^{[0]}(W_i,\gamma)\mid W_i,U_{\gamma i}\right]\right\|\right. \right.
\mathrel{\triangleq}\sup_{\{i,\gamma_*\}}\left\|\frac{1}{n-1}\sum_{\{j:j\ne i\}}\left(\partial_u f_0(U_{\gamma i};\gamma)\xi_j^{[2]}(W_i,\gamma)-\psi^{[2]}(W_i,U_{\gamma i};\gamma)\right)\right\|=o_p\left(\sqrt{\frac{\log n}{nh^5}}\right)+O(h^2). \tag*{(57)} ξ ℓ [ 0 ] ( W i , γ ) ∣ W i , U γ i ] ≜ { i , γ ∗ } sup n − 1 1 { j : j = i } ∑ ( ∂ u f 0 ( U γ i ; γ ) ξ j [ 2 ] ( W i , γ ) − ψ [ 2 ] ( W i , U γ i ; γ ) ) = o p ( n h 5 log n ) + O ( h 2 ) . (57)
Combining (D.23), (D.26), and (D.27) with Lemma S3 establishes the assertions of the lemma,
φ i [ t ] ( w , u ; γ ) = ( ∂ u f 0 ( u ; γ ) ) ξ i [ t ] ( w , γ ) − ψ [ t ] ( w , u ; γ ) + ζ i [ t ] ( w , u ; γ ) , t = 0 , 1 , 2 , i = 1 , … , n . (58) \varphi_i^{[t]}(w,u;\gamma)=(\partial_u f_0(u;\gamma))\xi_i^{[t]}(w,\gamma)-\psi^{[t]}(w,u;\gamma)+\zeta_i^{[t]}(w,u;\gamma),\quad t=0,1,2,\ i=1,\ldots,n. \tag*{(58)} φ i [ t ] ( w , u ; γ ) = ( ∂ u f 0 ( u ; γ )) ξ i [ t ] ( w , γ ) − ψ [ t ] ( w , u ; γ ) + ζ i [ t ] ( w , u ; γ ) , t = 0 , 1 , 2 , i = 1 , … , n . (58)
Proof of Theorem 1
A direct decomposition yields
2 n PSS ( γ , G ∘ ) = 1 n ∑ i = 1 n U γ i 2 + 1 n ∑ i = 1 n ( m [ 0 ] ( W i , γ T W i ) − m ^ h − i ( γ T W i ; G ∘ , γ ) ) 2 \frac{2}{n}\operatorname{PSS}(\gamma,\mathcal{G}^\circ)=\frac{1}{n}\sum_{i=1}^{n}U_{\gamma i}^{2}+\frac{1}{n}\sum_{i=1}^{n}\left(m^{[0]}(W_i,\gamma^\mathsf{T}W_i)-\widehat m_h^{-i}(\gamma^\mathsf{T}W_i;\mathcal{G}^\circ,\gamma)\right)^2 n 2 PSS ( γ , G ∘ ) = n 1 i = 1 ∑ n U γ i 2 + n 1 i = 1 ∑ n ( m [ 0 ] ( W i , γ T W i ) − m h − i ( γ T W i ; G ∘ , γ ) ) 2
+ 2 n ∑ i = 1 n U γ i ( m [ 0 ] ( W i , γ T W i ) − m ^ h − i ( γ T W i ; G ∘ , γ ) ) ≜ I 1 n ( γ ) + I 2 n ( γ ) + I 3 n ( γ ) . (59) \qquad+\frac{2}{n}\sum_{i=1}^{n}U_{\gamma i}\left(m^{[0]}(W_i,\gamma^\mathsf{T}W_i)-\widehat m_h^{-i}(\gamma^\mathsf{T}W_i;\mathcal{G}^\circ,\gamma)\right)\mathrel{\triangleq}I_{1n}(\gamma)+I_{2n}(\gamma)+I_{3n}(\gamma). \tag*{(59)} + n 2 i = 1 ∑ n U γ i ( m [ 0 ] ( W i , γ T W i ) − m h − i ( γ T W i ; G ∘ , γ ) ) ≜ I 1 n ( γ ) + I 2 n ( γ ) + I 3 n ( γ ) . (59)
Under assumptions A2 and A3, the representation U γ = U ∘ + ( m ( γ ∘ T W ) − m [ 0 ] ( W , γ T W ) ) U_\gamma=U^\circ+(m(\gamma^{\mathord\circ\mathsf{T}}W)-m^{[0]}(W,\gamma^\mathsf{T}W)) U γ = U ∘ + ( m ( γ ∘ T W ) − m [ 0 ] ( W , γ T W )) , together with ∣ ( γ ∘ − γ ) T W ∣ ≤ ∥ ( γ ∘ − γ ) ∥ ∥ W ∥ \left|(\gamma^\circ-\gamma)^\mathsf{T}W\right|\le\|(\gamma^\circ-\gamma)\|\|W\| ( γ ∘ − γ ) T W ≤ ∥ ( γ ∘ − γ ) ∥∥ W ∥ and the compactness of Γ \Gamma Γ , implies that E [ U γ 2 ] < ∞ E[U_\gamma^2]<\infty E [ U γ 2 ] < ∞ . Since { U γ : γ ∗ ∈ Γ } \{U_\gamma:\gamma_*\in\Gamma\} { U γ : γ ∗ ∈ Γ } is a Euclidean class, which is ensured by Lemma 22 in Nolan and Pollard [21 ] , an application of Theorem II.24 in Pollard [24 ] gives
sup { γ ∗ ∈ Γ } ∣ I 1 n ( γ ) − E [ U γ 2 ] ∣ = o p ( 1 ) . (60) \sup_{\{\gamma^*\in\Gamma\}}\left|I_{1n}(\gamma)-\mathbb{E}[U_\gamma^2]\right|=o_p(1). \tag*{(60)} { γ ∗ ∈ Γ } sup I 1 n ( γ ) − E [ U γ 2 ] = o p ( 1 ) . (60)
Under assumptions A1 and A2, and upon setting t = 0 t=0 t = 0 in Lemma S2, we obtain
sup { γ ∗ ∈ Γ } ∣ I 2 n ( γ ) ∣ = o p ( 1 ) and sup { γ ∗ ∈ Γ } ∣ I 3 n ( γ ) ∣ = o p ( 1 ) . (61) \sup_{\{\gamma^*\in\Gamma\}}|I_{2n}(\gamma)|=o_p(1)\quad\text{and}\quad\sup_{\{\gamma^*\in\Gamma\}}|I_{3n}(\gamma)|=o_p(1). \tag*{(61)} { γ ∗ ∈ Γ } sup ∣ I 2 n ( γ ) ∣ = o p ( 1 ) and { γ ∗ ∈ Γ } sup ∣ I 3 n ( γ ) ∣ = o p ( 1 ) . (61)
Substituting (E.2) and (E.3) into (E.1) and applying the triangle inequality yield
sup { γ ∗ ∈ Γ } ∣ 2 n PSS ( γ , G o ) − E [ U γ 2 ] ∣ = o p ( 1 ) . (62) \sup_{\{\gamma^*\in\Gamma\}}\left|\frac{2}{n}\operatorname{PSS}(\gamma,\mathcal{G}^{o})-\mathbb{E}[U_\gamma^2]\right|=o_p(1). \tag*{(62)} { γ ∗ ∈ Γ } sup n 2 PSS ( γ , G o ) − E [ U γ 2 ] = o p ( 1 ) . (62)
Since E [ U γ 2 ] ≥ E [ ( U o ) 2 ] \mathbb{E}[U_\gamma^2]\geq\mathbb{E}[(U^{o})^2] E [ U γ 2 ] ≥ E [( U o ) 2 ] , with strict inequality whenever γ ≠ γ o \gamma\ne\gamma^{o} γ = γ o , it follows that under assumption A6, γ o \gamma^{o} γ o is the unique minimizer of E [ U γ 2 ] \mathbb{E}[U_\gamma^2] E [ U γ 2 ] over Γ \Gamma Γ . Consequently, by (E.4) and the law of large numbers, 2 PSS ( γ o , G o ) / n − 2 PSS ( γ ^ o r , G o ) / n = E [ U γ ^ o r 2 ] − E [ ( U o ) 2 ] + o p ( 1 ) 2\operatorname{PSS}(\gamma^{o},\mathcal{G}^{o})/n-2\operatorname{PSS}(\widehat{\gamma}^{or},\mathcal{G}^{o})/n=\mathbb{E}[U_{\widehat{\gamma}^{or}}^2]-\mathbb{E}[(U^{o})^2]+o_p(1) 2 PSS ( γ o , G o ) / n − 2 PSS ( γ or , G o ) / n = E [ U γ or 2 ] − E [( U o ) 2 ] + o p ( 1 ) , where E [ U γ ^ o r 2 ] − E [ ( U o ) 2 ] > 0 \mathbb{E}[U_{\widehat{\gamma}^{or}}^2]-\mathbb{E}[(U^{o})^2]>0 E [ U γ or 2 ] − E [( U o ) 2 ] > 0 . On the other hand, by the definition of γ ^ o r \widehat{\gamma}^{or} γ or as a minimizer of PSS ( γ , G o ) \operatorname{PSS}(\gamma,\mathcal{G}^{o}) PSS ( γ , G o ) , we have PSS ( γ o , G o ) ≥ PSS ( γ ^ o r , G o ) \operatorname{PSS}(\gamma^{o},\mathcal{G}^{o})\geq\operatorname{PSS}(\widehat{\gamma}^{or},\mathcal{G}^{o}) PSS ( γ o , G o ) ≥ PSS ( γ or , G o ) . Therefore, E [ U γ ^ o r 2 ] − E [ ( U o ) 2 ] = o p ( 1 ) \mathbb{E}[U_{\widehat{\gamma}^{or}}^2]-\mathbb{E}[(U^{o})^2]=o_p(1) E [ U γ or 2 ] − E [( U o ) 2 ] = o p ( 1 ) . The consistency of γ ^ o r \widehat{\gamma}^{or} γ or to γ o \gamma^{o} γ o then follows from the uniqueness of γ o \gamma^{o} γ o as the minimizer of E [ U γ 2 ] \mathbb{E}[U_\gamma^2] E [ U γ 2 ] over Γ \Gamma Γ and the continuity of E [ U γ 2 ] \mathbb{E}[U_\gamma^2] E [ U γ 2 ] on Γ \Gamma Γ .
Define PS ( γ ) = ∂ γ ∗ PSS ( γ , G o ) / n \operatorname{PS}(\gamma)=\partial_{\gamma^*}\operatorname{PSS}(\gamma,\mathcal{G}^{o})/n PS ( γ ) = ∂ γ ∗ PSS ( γ , G o ) / n and PH ( γ ) = ∂ γ ∗ 2 PSS ( γ , G o ) / n \operatorname{PH}(\gamma)=\partial_{\gamma^*}^2\operatorname{PSS}(\gamma,\mathcal{G}^{o})/n PH ( γ ) = ∂ γ ∗ 2 PSS ( γ , G o ) / n . For an arbitrary constant vector ν ∈ R k ( p + 1 ) − 2 \nu\in\mathbb{R}^{k(p+1)-2} ν ∈ R k ( p + 1 ) − 2 , a first-order Taylor expansion of ν T PS ( γ ^ o r ) \nu^{\mathsf{T}}\operatorname{PS}(\widehat{\gamma}^{or}) ν T PS ( γ or ) around γ ∗ o \gamma_*^{o} γ ∗ o yields
ν T ( n PS ( γ ∗ o ) ) + ν T { PH ( γ ˉ ) [ n ( γ ^ o r − γ ∗ o ) ] } = 0 , (63) \nu^{\mathsf{T}}\left(\sqrt{n}\operatorname{PS}(\gamma_*^{o})\right)+\nu^{\mathsf{T}}\left\{\operatorname{PH}(\bar{\gamma})\left[\sqrt{n}(\widehat{\gamma}^{or}-\gamma_*^{o})\right]\right\}=0, \tag*{(63)} ν T ( n PS ( γ ∗ o ) ) + ν T { PH ( γ ˉ ) [ n ( γ or − γ ∗ o ) ] } = 0 , (63)
where γ ˉ \bar{\gamma} γ ˉ lies on the line segment between γ ^ o r \widehat{\gamma}^{or} γ or and γ ∗ o \gamma_*^{o} γ ∗ o . The pseudo score PS ( γ o ) \operatorname{PS}(\gamma^{o}) PS ( γ o ) in (E.5) admits the decomposition
PS ( γ o ) = ∑ ℓ 1 = 0 1 ∑ ℓ 2 = 0 1 1 n ∑ i = 1 n ( − 1 ) ℓ 1 ( U i o ) 1 − ℓ 1 ( m ^ h − i ( γ o T W i ; G o , γ o ) − m ( γ o T W i ) ) ℓ 1 ( m [ 1 ] ( W i , γ o T W i ) ) 1 − ℓ 2 \operatorname{PS}(\gamma^{o})=\sum_{\ell_1=0}^{1}\sum_{\ell_2=0}^{1}\frac{1}{n}\sum_{i=1}^{n}(-1)^{\ell_1}(U_i^{o})^{1-\ell_1}\left(\widehat{m}_{h}^{-i}(\gamma^{o\mathsf{T}}W_i;\mathcal{G}^{o},\gamma^{o})-m(\gamma^{o\mathsf{T}}W_i)\right)^{\ell_1}\left(m^{[1]}(W_i,\gamma^{o\mathsf{T}}W_i)\right)^{1-\ell_2} PS ( γ o ) = ℓ 1 = 0 ∑ 1 ℓ 2 = 0 ∑ 1 n 1 i = 1 ∑ n ( − 1 ) ℓ 1 ( U i o ) 1 − ℓ 1 ( m h − i ( γ o T W i ; G o , γ o ) − m ( γ o T W i ) ) ℓ 1 ( m [ 1 ] ( W i , γ o T W i ) ) 1 − ℓ 2
( ∂ γ ∗ o m ^ h − i ( γ o T W i ; G o , γ o ) − m [ 1 ] ( W i , γ o T W i ) ) ℓ 2 ≜ ∑ ℓ 1 = 0 1 ∑ ℓ 2 = 0 1 PS ℓ 1 ℓ 2 ( γ o ) . (64) \left(\partial_{\gamma_*^{o}}\widehat{m}_{h}^{-i}(\gamma^{o\mathsf{T}}W_i;\mathcal{G}^{o},\gamma^{o})-m^{[1]}(W_i,\gamma^{o\mathsf{T}}W_i)\right)^{\ell_2}\triangleq\sum_{\ell_1=0}^{1}\sum_{\ell_2=0}^{1}\operatorname{PS}_{\ell_1\ell_2}(\gamma^{o}). \tag*{(64)} ( ∂ γ ∗ o m h − i ( γ o T W i ; G o , γ o ) − m [ 1 ] ( W i , γ o T W i ) ) ℓ 2 ≜ ℓ 1 = 0 ∑ 1 ℓ 2 = 0 ∑ 1 PS ℓ 1 ℓ 2 ( γ o ) . (64)
Under assumptions A1 and A2, and upon setting t = 0 , 1 t=0,1 t = 0 , 1 in Lemma S2, it follows that
n PS 01 ( γ o ) = n 1 n ∑ i = 1 n U i o ( 1 n − 1 ∑ { j : j ≠ i } ξ j [ 1 ] ( W i , γ o ) + r [ 1 ] ( W i , γ o ) ) = n n ( n − 1 ) ∑ i ≠ j U i o E [ ξ j [ 1 ] ( W i , γ o ) ∣ W i ] + n n ( n − 1 ) ∑ i ≠ j U i o ξ ˉ j [ 1 ] ( W i , γ o ) + o p ( 1 ) , (65) \begin{aligned}
\sqrt{n}\operatorname{PS}_{01}(\gamma^{o})=\sqrt{n}\frac{1}{n}\sum_{i=1}^{n}U_i^{o}\left(\frac{1}{n-1}\sum_{\{j:j\ne i\}}\xi_j^{[1]}(W_i,\gamma^{o})+r^{[1]}(W_i,\gamma^{o})\right) \\
&=\frac{\sqrt{n}}{n(n-1)}\sum_{i\ne j}U_i^{o}\mathbb{E}[\xi_j^{[1]}(W_i,\gamma^{o})\mid W_i]+\frac{\sqrt{n}}{n(n-1)}\sum_{i\ne j}U_i^{o}\bar{\xi}_j^{[1]}(W_i,\gamma^{o})+o_p(1), \tag*{(65)}
\end{aligned} n PS 01 ( γ o ) = n n 1 i = 1 ∑ n U i o n − 1 1 { j : j = i } ∑ ξ j [ 1 ] ( W i , γ o ) + r [ 1 ] ( W i , γ o ) = n ( n − 1 ) n i = j ∑ U i o E [ ξ j [ 1 ] ( W i , γ o ) ∣ W i ] + n ( n − 1 ) n i = j ∑ U i o ξ ˉ j [ 1 ] ( W i , γ o ) + o p ( 1 ) , (65)
n PS 10 ( γ o ) = − n 1 n ∑ i = 1 n m [ 1 ] ( W i , γ o T W i ) ( 1 n − 1 ∑ { j : j ≠ i } ξ j [ 0 ] ( W i , γ o ) + r [ 0 ] ( W i , γ o ) ) = − n n ( n − 1 ) ∑ i ≠ j m [ 1 ] ( W i , γ o T W i ) ξ ˉ j [ 0 ] ( W i , γ o ) \begin{aligned}
\sqrt{n}\operatorname{PS}_{10}(\gamma^{o})=-\sqrt{n}\frac{1}{n}\sum_{i=1}^{n}m^{[1]}(W_i,\gamma^{o\mathsf{T}}W_i)\left(\frac{1}{n-1}\sum_{\{j:j\ne i\}}\xi_j^{[0]}(W_i,\gamma^{o})+r^{[0]}(W_i,\gamma^{o})\right) \\
&=-\frac{\sqrt{n}}{n(n-1)}\sum_{i\ne j}m^{[1]}(W_i,\gamma^{o\mathsf{T}}W_i)\bar{\xi}_j^{[0]}(W_i,\gamma^{o})
\end{aligned} n PS 10 ( γ o ) = − n n 1 i = 1 ∑ n m [ 1 ] ( W i , γ o T W i ) n − 1 1 { j : j = i } ∑ ξ j [ 0 ] ( W i , γ o ) + r [ 0 ] ( W i , γ o ) = − n ( n − 1 ) n i = j ∑ m [ 1 ] ( W i , γ o T W i ) ξ ˉ j [ 0 ] ( W i , γ o )
− n n ( n − 1 ) ∑ i ≠ j m [ 1 ] ( W i , γ o T W i ) E [ ξ j [ 0 ] ( W i , γ o ) ∣ W i ] + o p ( 1 ) , (66) -\frac{\sqrt{n}}{n(n-1)}\sum_{i\ne j}m^{[1]}(W_i,\gamma^{o\mathsf{T}}W_i)\mathbb{E}[\xi_j^{[0]}(W_i,\gamma^{o})\mid W_i]+o_p(1), \tag*{(66)} − n ( n − 1 ) n i = j ∑ m [ 1 ] ( W i , γ o T W i ) E [ ξ j [ 0 ] ( W i , γ o ) ∣ W i ] + o p ( 1 ) , (66)
n ∥ PS 11 ( γ o ) ∥ = n ∥ 1 n ∑ i = 1 n ∏ t = 0 1 ( ∂ γ ∗ o m ^ h − i ( γ o T W i ; γ o ) − m [ t ] ( W i , γ o T W i ) ) ∥ ≤ n ∏ t = 0 1 sup i ∥ ∂ γ ∗ o t m ^ h − i ( γ o T W i ; γ o ) − m [ t ] ( W i , γ o T W i ) ∥ = n ( o p ( log n n h ) + O ( h 2 ) ) ( o p ( log n n h 3 ) + O ( h 2 ) ) = o p ( 1 ) . (67) \begin{aligned}
\sqrt{n}\|\operatorname{PS}_{11}(\gamma^{o})\|=\sqrt{n}\left\|\frac{1}{n}\sum_{i=1}^{n}\prod_{t=0}^{1}\left(\partial_{\gamma_*^{o}}\widehat{m}_{h}^{-i}(\gamma^{o\mathsf{T}}W_i;\gamma^{o})-m^{[t]}(W_i,\gamma^{o\mathsf{T}}W_i)\right)\right\| \\
&\leq\sqrt{n}\prod_{t=0}^{1}\sup_i\left\|\partial_{\gamma_*^{o}}^{t}\widehat{m}_{h}^{-i}(\gamma^{o\mathsf{T}}W_i;\gamma^{o})-m^{[t]}(W_i,\gamma^{o\mathsf{T}}W_i)\right\| \\
&=\sqrt{n}\left(o_p\left(\sqrt{\frac{\log n}{nh}}\right)+O(h^2)\right)\left(o_p\left(\sqrt{\frac{\log n}{nh^3}}\right)+O(h^2)\right)=o_p(1). \tag*{(67)}
\end{aligned} n ∥ PS 11 ( γ o ) ∥ = n n 1 i = 1 ∑ n t = 0 ∏ 1 ( ∂ γ ∗ o m h − i ( γ o T W i ; γ o ) − m [ t ] ( W i , γ o T W i ) ) ≤ n t = 0 ∏ 1 i sup ∂ γ ∗ o t m h − i ( γ o T W i ; γ o ) − m [ t ] ( W i , γ o T W i ) = n ( o p ( nh log n ) + O ( h 2 ) ) ( o p ( n h 3 log n ) + O ( h 2 ) ) = o p ( 1 ) . (67)
where ξ ˉ j [ t ] ( W i , γ ∘ ) = ξ j [ t ] ( W i , γ ∘ ) − E [ ξ j [ t ] ( W i , γ ∘ ) ∣ W i ] \bar{\xi}_{j}^{[t]}(W_i,\gamma^\circ)=\xi_{j}^{[t]}(W_i,\gamma^\circ)-\mathbb{E}[\xi_{j}^{[t]}(W_i,\gamma^\circ)\mid W_i] ξ ˉ j [ t ] ( W i , γ ∘ ) = ξ j [ t ] ( W i , γ ∘ ) − E [ ξ j [ t ] ( W i , γ ∘ ) ∣ W i ] , i , j = 1 , … , n i,j=1,\ldots,n i , j = 1 , … , n , t = 0 , 1 t=0,1 t = 0 , 1 . The identities
ξ j [ 0 ] ( W i , γ ∘ ) = ( Y j − m ( γ ∘ T W i ) ) K h ( γ ∘ T ( W j − W i ) ) f γ ∘ ( γ ∘ T W i ) , \xi_{j}^{[0]}(W_i,\gamma^\circ)=\frac{(Y_j-m(\gamma^{\circ\mathsf{T}}W_i))K_h(\gamma^{\circ\mathsf{T}}(W_j-W_i))}{f_{\gamma^\circ}(\gamma^{\circ\mathsf{T}}W_i)}, ξ j [ 0 ] ( W i , γ ∘ ) = f γ ∘ ( γ ∘ T W i ) ( Y j − m ( γ ∘ T W i )) K h ( γ ∘ T ( W j − W i )) ,
m [ 1 ] ( W i , γ ∘ T W i ) = ( ∂ v m ( γ ∘ T W i ) ) ( W ∗ i − H ( γ ∘ T W i ) ) , m^{[1]}(W_i,\gamma^{\circ\mathsf{T}}W_i)=(\partial_v m(\gamma^{\circ\mathsf{T}}W_i))(W_{*i}-H(\gamma^{\circ\mathsf{T}}W_i)), m [ 1 ] ( W i , γ ∘ T W i ) = ( ∂ v m ( γ ∘ T W i )) ( W ∗ i − H ( γ ∘ T W i )) ,
with H ( v ) = E [ W ∗ ∣ γ ∘ T W = v ] H(v)=\mathbb{E}[W_*\mid\gamma^{\circ\mathsf{T}}W=v] H ( v ) = E [ W ∗ ∣ γ ∘ T W = v ] ,
E [ U i ∘ ξ ˉ j [ 1 ] ( W i , γ ∘ ) ∣ W i , Y i ] = 0 , E [ m [ 1 ] ( W i , γ ∘ T W i ) ξ ˉ j [ 0 ] ( W i , γ ∘ ) ∣ W i , Y i ] = 0 , \mathbb{E}[U_i^\circ\bar{\xi}_{j}^{[1]}(W_i,\gamma^\circ)\mid W_i,Y_i]=0,\quad\mathbb{E}[m^{[1]}(W_i,\gamma^{\circ\mathsf{T}}W_i)\bar{\xi}_{j}^{[0]}(W_i,\gamma^\circ)\mid W_i,Y_i]=0, E [ U i ∘ ξ ˉ j [ 1 ] ( W i , γ ∘ ) ∣ W i , Y i ] = 0 , E [ m [ 1 ] ( W i , γ ∘ T W i ) ξ ˉ j [ 0 ] ( W i , γ ∘ ) ∣ W i , Y i ] = 0 ,
E [ U i ∘ ξ ˉ j [ 1 ] ( W i , γ ∘ ) ∣ W j , Y j ] = E [ E [ U i ∘ ∣ W i ] ξ ˉ j [ 1 ] ( W i , γ ∘ ) ∣ W j , Y j ] = 0 , \mathbb{E}[U_i^\circ\bar{\xi}_{j}^{[1]}(W_i,\gamma^\circ)\mid W_j,Y_j]=\mathbb{E}[\mathbb{E}[U_i^\circ\mid W_i]\bar{\xi}_{j}^{[1]}(W_i,\gamma^\circ)\mid W_j,Y_j]=0, E [ U i ∘ ξ ˉ j [ 1 ] ( W i , γ ∘ ) ∣ W j , Y j ] = E [ E [ U i ∘ ∣ W i ] ξ ˉ j [ 1 ] ( W i , γ ∘ ) ∣ W j , Y j ] = 0 ,
and
E [ m [ 1 ] ( W i , γ ∘ T W i ) ξ ˉ j [ 0 ] ( W i , γ ∘ ) ∣ W j , Y j ] = E [ ( ∂ v m ( γ ∘ T W i ) ) ( H ( γ ∘ T W i ) − E [ W ∗ i ∣ γ ∘ T W i ] ) ξ ˉ j [ 0 ] ( W i , γ ∘ ) ∣ W j , Y j ] = 0 , j = 1 , … , n , \begin{aligned}
\mathbb{E}[m^{[1]}(W_i,\gamma^{\circ\mathsf{T}}W_i)\bar{\xi}_{j}^{[0]}(W_i,\gamma^\circ)\mid W_j,Y_j]
&=\mathbb{E}[(\partial_v m(\gamma^{\circ\mathsf{T}}W_i))(H(\gamma^{\circ\mathsf{T}}W_i)-\mathbb{E}[W_{*i}\mid\gamma^{\circ\mathsf{T}}W_i])\bar{\xi}_{j}^{[0]}(W_i,\gamma^\circ)\mid W_j,Y_j]\\
&=0,\quad j=1,\ldots,n,
\end{aligned} E [ m [ 1 ] ( W i , γ ∘ T W i ) ξ ˉ j [ 0 ] ( W i , γ ∘ ) ∣ W j , Y j ] = E [( ∂ v m ( γ ∘ T W i )) ( H ( γ ∘ T W i ) − E [ W ∗ i ∣ γ ∘ T W i ]) ξ ˉ j [ 0 ] ( W i , γ ∘ ) ∣ W j , Y j ] = 0 , j = 1 , … , n ,
further imply that the second-order U U U -statistics in (E.7) and (E.8) are degenerate. Since E [ ( U i ∘ ξ ˉ j [ 1 ] ( W i , γ ∘ ) ) 2 ] = O ( h − 3 ) \mathbb{E}[(U_i^\circ\bar{\xi}_{j}^{[1]}(W_i,\gamma^\circ))^2]=O(h^{-3}) E [( U i ∘ ξ ˉ j [ 1 ] ( W i , γ ∘ ) ) 2 ] = O ( h − 3 ) and E [ ( m [ 1 ] ( W i , γ ∘ T W i ) ξ ˉ j [ 0 ] ( W i , γ ∘ ) ) 2 ] = O ( h − 1 ) \mathbb{E}[(m^{[1]}(W_i,\gamma^{\circ\mathsf{T}}W_i)\bar{\xi}_{j}^{[0]}(W_i,\gamma^\circ))^2]=O(h^{-1}) E [( m [ 1 ] ( W i , γ ∘ T W i ) ξ ˉ j [ 0 ] ( W i , γ ∘ ) ) 2 ] = O ( h − 1 ) , an application of Theorem 8.1 in Hoefding [8 ] yields
∥ 1 n ( n − 1 ) ∑ i ≠ j U i ∘ ξ ˉ j [ 1 ] ( W i , γ ∘ ) ∥ = O p ( 1 n h 3 ) (68) \left\|\frac{1}{n(n-1)}\sum_{i\ne j}U_i^\circ\bar{\xi}_{j}^{[1]}(W_i,\gamma^\circ)\right\|=O_p\left(\frac{1}{n\sqrt{h^3}}\right) \tag*{(68)} n ( n − 1 ) 1 i = j ∑ U i ∘ ξ ˉ j [ 1 ] ( W i , γ ∘ ) = O p ( n h 3 1 ) (68)
and
∥ 1 n ( n − 1 ) ∑ i ≠ j m [ 1 ] ( W i , γ ∘ T W i ) ξ ˉ j [ 0 ] ( W i , γ ∘ ) ∥ = O p ( 1 n h ) . (69) \left\|\frac{1}{n(n-1)}\sum_{i\ne j}m^{[1]}(W_i,\gamma^{\circ\mathsf{T}}W_i)\bar{\xi}_{j}^{[0]}(W_i,\gamma^\circ)\right\|=O_p\left(\frac{1}{n\sqrt{h}}\right). \tag*{(69)} n ( n − 1 ) 1 i = j ∑ m [ 1 ] ( W i , γ ∘ T W i ) ξ ˉ j [ 0 ] ( W i , γ ∘ ) = O p ( n h 1 ) . (69)
Moreover, Lemma S2 with t = 0 , 1 t=0,1 t = 0 , 1 , together with the central limit theorem, implies
∥ 1 n ∑ i = 1 n U i ∘ E [ ξ j [ 1 ] ( W i , γ ∘ ) ∣ W i ] ∥ = O ( h 2 n ) (70) \left\|\frac{1}{n}\sum_{i=1}^{n}U_i^\circ\mathbb{E}[\xi_{j}^{[1]}(W_i,\gamma^\circ)\mid W_i]\right\|=O\left(\frac{h^2}{\sqrt{n}}\right) \tag*{(70)} n 1 i = 1 ∑ n U i ∘ E [ ξ j [ 1 ] ( W i , γ ∘ ) ∣ W i ] = O ( n h 2 ) (70)
and
∥ 1 n ∑ i = 1 n m [ 1 ] ( W i , γ ∘ T W i ) E [ ξ j [ 0 ] ( W i , γ ∘ ) ∣ W i ] ∥ = O ( h 2 n ) . (71) \left\|\frac{1}{n}\sum_{i=1}^{n}m^{[1]}(W_i,\gamma^{\circ\mathsf{T}}W_i)\mathbb{E}[\xi_{j}^{[0]}(W_i,\gamma^\circ)\mid W_i]\right\|=O\left(\frac{h^2}{\sqrt{n}}\right). \tag*{(71)} n 1 i = 1 ∑ n m [ 1 ] ( W i , γ ∘ T W i ) E [ ξ j [ 0 ] ( W i , γ ∘ ) ∣ W i ] = O ( n h 2 ) . (71)
Substituting (E.10) and (E.12) into (E.7), and (E.11) and (E.13) into (E.8), we obtain
n P S 01 ( γ ∘ ) = o p ( 1 ) and n P S 10 ( γ ∘ ) = o p ( 1 ) . (72) \sqrt{n}\mathrm{PS}_{01}(\gamma^\circ)=o_p(1)\quad\text{and}\quad\sqrt{n}\mathrm{PS}_{10}(\gamma^\circ)=o_p(1). \tag*{(72)} n PS 01 ( γ ∘ ) = o p ( 1 ) and n PS 10 ( γ ∘ ) = o p ( 1 ) . (72)
Combining (E.9) and (E.14) with the central limit theorem result n ν T P S 00 ( γ ∘ ) → d N ( 0 , ν T V 1 ν ) \sqrt{n}\nu^{\mathsf{T}}\mathrm{PS}_{00}(\gamma^\circ)\xrightarrow{d}N(0,\nu^{\mathsf{T}}V_1\nu) n ν T PS 00 ( γ ∘ ) d N ( 0 , ν T V 1 ν ) , and using (E.6), we obtain
n P S ( γ ∘ ) → d N ( 0 , V 1 ) . (73) \sqrt{n}\mathrm{PS}(\gamma^\circ)\xrightarrow{d}N(0,V_1). \tag*{(73)} n PS ( γ ∘ ) d N ( 0 , V 1 ) . (73)
By Lemma S2 with t = 1 , 2 t=1,2 t = 1 , 2 , it follows that
P H ( γ ) = 1 n ∑ i = 1 n ( m [ 1 ] ( W i , γ T W i ) ) ⊗ 2 − 1 n ∑ i = 1 n U i m [ 2 ] ( W i , γ T W i ) + O p ( log n n h 5 ) + O ( h 2 ) . (74) \mathrm{PH}(\gamma)=\frac{1}{n}\sum_{i=1}^{n}(m^{[1]}(W_i,\gamma^{\mathsf{T}}W_i))^{\otimes2}-\frac{1}{n}\sum_{i=1}^{n}U_i m^{[2]}(W_i,\gamma^{\mathsf{T}}W_i)+O_p\left(\sqrt{\frac{\log n}{nh^5}}\right)+O(h^2). \tag*{(74)} PH ( γ ) = n 1 i = 1 ∑ n ( m [ 1 ] ( W i , γ T W i ) ) ⊗ 2 − n 1 i = 1 ∑ n U i m [ 2 ] ( W i , γ T W i ) + O p ( n h 5 log n ) + O ( h 2 ) . (74)
Under assumption A1, applying the law of large numbers to (E.16) gives
P H ( γ ) → p E [ ( m [ 1 ] ( W , γ T W ) ) ⊗ 2 − U m [ 2 ] ( W , γ T W ) ] . (75) \mathrm{PH}(\gamma)\xrightarrow{p}\mathbb{E}[(m^{[1]}(W,\gamma^{\mathsf{T}}W))^{\otimes2}-U m^{[2]}(W,\gamma^{\mathsf{T}}W)]. \tag*{(75)} PH ( γ ) p E [( m [ 1 ] ( W , γ T W ) ) ⊗ 2 − U m [ 2 ] ( W , γ T W )] . (75)
The continuity of P H ( γ ) \mathrm{PH}(\gamma) PH ( γ ) in γ \gamma γ , the consistency of γ ^ ∗ \hat{\gamma}_* γ ^ ∗ to γ ∗ ∘ \gamma_*^\circ γ ∗ ∘ , and E [ U m [ 2 ] ( W , γ T W ) ] = 0 \mathbb{E}[U m^{[2]}(W,\gamma^{\mathsf{T}}W)]=0 E [ U m [ 2 ] ( W , γ T W )] = 0 for γ = γ ∘ \gamma=\gamma^\circ γ = γ ∘ , yield
P H ( γ ˉ ) → p V 2 . (76) \mathrm{PH}(\bar{\gamma})\xrightarrow{p}V_2. \tag*{(76)} PH ( γ ˉ ) p V 2 . (76)
Substituting (E.15) and (E.18) into (E.5) and applying Slutsky’s theorem, we establish the asymptotic normality of γ ^ ∗ o r \hat{\gamma}_*^{\mathsf{or}} γ ^ ∗ or under assumption A6.
Proof of Theorem 2
The SP estimator ( β ^ , γ ^ ) (\widehat{\beta},\widehat{\gamma}) ( β , γ ) possesses the oracle property if, with probability converging to one,
PSS S P ( β , γ ) > PSS S P ( β ^ o r , γ ^ o r ) \operatorname{PSS}_{\mathrm{SP}}(\beta,\gamma)>\operatorname{PSS}_{\mathrm{SP}}(\widehat{\beta}^{\mathrm{or}},\widehat{\gamma}^{\mathrm{or}}) PSS SP ( β , γ ) > PSS SP ( β or , γ or )
for all ( β , γ ) ∈ B n × Γ n (\beta,\gamma)\in B_n\times\Gamma_n ( β , γ ) ∈ B n × Γ n such that ( β , γ ) ≠ ( β ^ o r , γ ^ o r ) (\beta,\gamma)\ne(\widehat{\beta}^{\mathrm{or}},\widehat{\gamma}^{\mathrm{or}}) ( β , γ ) = ( β or , γ or ) . It suffices to show that, with probability converging to one,
PSS S P ( β ∗ , γ ∗ ) > PSS S P ( β ^ o r , γ ^ o r ) and PSS S P ( β , γ ) ≥ PSS S P ( β ∗ , γ ∗ ) , \operatorname{PSS}_{\mathrm{SP}}(\beta^{*},\gamma^{*})>\operatorname{PSS}_{\mathrm{SP}}(\widehat{\beta}^{\mathrm{or}},\widehat{\gamma}^{\mathrm{or}})\ \text{and}\ \operatorname{PSS}_{\mathrm{SP}}(\beta,\gamma)\ge\operatorname{PSS}_{\mathrm{SP}}(\beta^{*},\gamma^{*}), PSS SP ( β ∗ , γ ∗ ) > PSS SP ( β or , γ or ) and PSS SP ( β , γ ) ≥ PSS SP ( β ∗ , γ ∗ ) ,
where
γ c ∗ = 1 ∣ G c o ∣ ∑ { i ∈ G c o } β i and β i ∗ = ∑ c = 1 k γ c ∗ I ( i ∈ G c o ) , c ∈ C , i = 1 , … , n . (77) \gamma_c^{*}=\frac{1}{|\mathcal{G}_c^{o}|}\sum_{\{i\in\mathcal{G}_c^{o}\}}\beta_i\ \text{and}\ \beta_i^{*}=\sum_{c=1}^{k}\gamma_c^{*}I(i\in\mathcal{G}_c^{o}),\ c\in\mathcal{C},\ i=1,\ldots,n. \tag*{(77)} γ c ∗ = ∣ G c o ∣ 1 { i ∈ G c o } ∑ β i and β i ∗ = c = 1 ∑ k γ c ∗ I ( i ∈ G c o ) , c ∈ C , i = 1 , … , n . (77)
By the definitions of ( β ∗ , γ ∗ ) (\beta^{*},\gamma^{*}) ( β ∗ , γ ∗ ) in (F.1) and PSS( γ , G ) (\gamma,\mathcal{G}) ( γ , G ) in (10) , the penalty term satisfies ∑ i = 1 n min c ∥ β ( 1 ) i ∗ − γ ( 1 ) c ∗ ∥ 1 = 0 \sum_{i=1}^{n}\min_c\lVert\beta_{(1)i}^{*}-\gamma_{(1)c}^{*}\rVert_1=0 ∑ i = 1 n min c ∥ β ( 1 ) i ∗ − γ ( 1 ) c ∗ ∥ 1 = 0 , which implies that
PSS S P ( β ∗ , γ ∗ ) = PSS ( γ ∗ , G o ) . (78) \operatorname{PSS}_{\mathrm{SP}}(\beta^{*},\gamma^{*})=\operatorname{PSS}(\gamma^{*},\mathcal{G}^{o}). \tag*{(78)} PSS SP ( β ∗ , γ ∗ ) = PSS ( γ ∗ , G o ) . (78)
A second-order Taylor expansion of PSS( γ ∗ , G o ) (\gamma^{*},\mathcal{G}^{o}) ( γ ∗ , G o ) around γ ∗ = γ ^ o r \gamma^{*}=\widehat{\gamma}^{\mathrm{or}} γ ∗ = γ or , combined with γ ∗ = γ ∗ o + o ( 1 ) \gamma_{*}=\gamma_{*}^{o}+o(1) γ ∗ = γ ∗ o + o ( 1 ) , the consistency of γ ^ o r \widehat{\gamma}^{\mathrm{or}} γ or to γ ∗ o \gamma_{*}^{o} γ ∗ o in Theorem 1, and (E.18), yields
2 n PSS ( γ ∗ , G o ) − 2 n PSS ( γ ^ o r , G o ) = 1 2 ( γ ∗ − γ ^ o r ) T V 2 ( γ ∗ − γ ^ o r ) ( 1 + o p ( 1 ) ) . (79) \frac{2}{n}\operatorname{PSS}(\gamma^{*},\mathcal{G}^{o})-\frac{2}{n}\operatorname{PSS}(\widehat{\gamma}^{\mathrm{or}},\mathcal{G}^{o})=\frac{1}{2}(\gamma^{*}-\widehat{\gamma}^{\mathrm{or}})^{\mathsf{T}}V_2(\gamma^{*}-\widehat{\gamma}^{\mathrm{or}})(1+o_p(1)). \tag*{(79)} n 2 PSS ( γ ∗ , G o ) − n 2 PSS ( γ or , G o ) = 2 1 ( γ ∗ − γ or ) T V 2 ( γ ∗ − γ or ) ( 1 + o p ( 1 )) . (79)
It follows from (F.2) and (F.3) that under assumption A6,
1 n PSS S P ( β ∗ , γ ∗ ) > 1 n PSS S P ( β ^ o r , γ ^ o r ) + o p ( 1 ) for β ∗ ∈ B with β ∗ ≠ β ^ o r . (80) \frac{1}{n}\operatorname{PSS}_{\mathrm{SP}}(\beta^{*},\gamma^{*})>\frac{1}{n}\operatorname{PSS}_{\mathrm{SP}}(\widehat{\beta}^{\mathrm{or}},\widehat{\gamma}^{\mathrm{or}})+o_p(1)\ \text{for }\beta^{*}\in B\text{ with }\beta^{*}\ne\widehat{\beta}^{\mathrm{or}}. \tag*{(80)} n 1 PSS SP ( β ∗ , γ ∗ ) > n 1 PSS SP ( β or , γ or ) + o p ( 1 ) for β ∗ ∈ B with β ∗ = β or . (80)
A first-order Taylor expansion of ∑ i = 1 n ( Y i − m ^ h − i ( β i T Z i ; β ) ) 2 \sum_{i=1}^{n}(Y_i-\widehat{m}_h^{-i}(\beta_i^{\mathsf{T}}Z_i;\beta))^2 ∑ i = 1 n ( Y i − m h − i ( β i T Z i ; β ) ) 2 around β = β ∗ \beta=\beta^{*} β = β ∗ yields
PSS S P ( β , γ ) − PSS S P ( β ∗ , γ ∗ ) = − ∑ i = 1 n ( Y i − m ^ h − i ( β i T Z i ; β ‾ ) ) [ ∑ j = 1 n ( ∂ β j m ^ h − i ( β i T Z i ; β ‾ ) ) T ( β j − β j ∗ ) ] + λ ∑ i = 1 n min c ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 ≜ I 1 + I 2 , (81) \begin{aligned}
\operatorname{PSS}_{\mathrm{SP}}(\beta,\gamma)-\operatorname{PSS}_{\mathrm{SP}}(\beta^{*},\gamma^{*})
={}&-\sum_{i=1}^{n}\left(Y_i-\widehat{m}_h^{-i}(\beta_i^{\mathsf{T}}Z_i;\overline{\beta})\right)
\left[\sum_{j=1}^{n}\left(\partial_{\beta_j}\widehat{m}_h^{-i}(\beta_i^{\mathsf{T}}Z_i;\overline{\beta})\right)^{\mathsf{T}}(\beta_j-\beta_j^{*})\right]\\
&+\lambda\sum_{i=1}^{n}\min_c\lVert\beta_{(1)i}-\gamma_{(1)c}\rVert_1\\
&\triangleq I_1+I_2, \tag*{(81)}
\end{aligned} PSS SP ( β , γ ) − PSS SP ( β ∗ , γ ∗ ) = − i = 1 ∑ n ( Y i − m h − i ( β i T Z i ; β ) ) [ j = 1 ∑ n ( ∂ β j m h − i ( β i T Z i ; β ) ) T ( β j − β j ∗ ) ] + λ i = 1 ∑ n c min ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 ≜ I 1 + I 2 , (81)
where β ‾ = α 0 β + ( 1 − α 0 ) β ∗ \overline{\beta}=\alpha_0\beta+(1-\alpha_0)\beta^{*} β = α 0 β + ( 1 − α 0 ) β ∗ for some α 0 ∈ [ 0 , 1 ] \alpha_0\in[0,1] α 0 ∈ [ 0 , 1 ] .
A direct calculation shows that
I 1 = − ∑ i = 1 n ( Y i − m ^ h − i ( β i T Z i ; β ‾ ) ) ( ∂ β i m ^ h − i ( β i T Z i ; β ‾ ) ) T ( β i − β i ∗ ) − ∑ j = 1 n ∑ { i : i ≠ j } ( Y i − m ^ h − i ( β i T Z i ; β ‾ ) ) ( ∂ β j m ^ h − i ( β i T Z i ; β ‾ ) ) T ( β j − β j ∗ ) = ∑ c = 1 k ∑ { i ∈ G c o } ( PS i i ( β ) ) T ∑ { j ∈ G c o } β i − β j ∣ G c o ∣ + ∑ j = 1 n ∑ { i : i ≠ j } ( PS i j ( β ‾ ) ) T ( β j − β j ∗ ) = ∑ c = 1 k ∑ { i ∈ G c o } ( s i ( β o ) ) T ∑ { j ∈ G c o } β i − β j ∣ G c o ∣ + ∑ c = 1 k ∑ { i ∈ G c o } ( PS i i ( β ‾ ) − s i ( β o ) ) T ∑ { j ∈ G c o } β i − β j ∣ G c o ∣ + ∑ j = 1 n ∑ { i : i ≠ j } ( PS i j ( β ‾ ) ) T ( β j − β j ∗ ) ≜ I 11 + I 12 + I 13 , (82) \begin{aligned}
I_1={}&-\sum_{i=1}^{n}\left(Y_i-\widehat{m}_h^{-i}(\beta_i^{\mathsf{T}}Z_i;\overline{\beta})\right)\left(\partial_{\beta_i}\widehat{m}_h^{-i}(\beta_i^{\mathsf{T}}Z_i;\overline{\beta})\right)^{\mathsf{T}}(\beta_i-\beta_i^{*})\\
&-\sum_{j=1}^{n}\sum_{\{i:i\ne j\}}\left(Y_i-\widehat{m}_h^{-i}(\beta_i^{\mathsf{T}}Z_i;\overline{\beta})\right)\left(\partial_{\beta_j}\widehat{m}_h^{-i}(\beta_i^{\mathsf{T}}Z_i;\overline{\beta})\right)^{\mathsf{T}}(\beta_j-\beta_j^{*})\\
={}&\sum_{c=1}^{k}\sum_{\{i\in\mathcal{G}_c^{o}\}}(\operatorname{PS}_{ii}(\beta))^{\mathsf{T}}\sum_{\{j\in\mathcal{G}_c^{o}\}}\frac{\beta_i-\beta_j}{|\mathcal{G}_c^{o}|}
+\sum_{j=1}^{n}\sum_{\{i:i\ne j\}}(\operatorname{PS}_{ij}(\overline{\beta}))^{\mathsf{T}}(\beta_j-\beta_j^{*})\\
={}&\sum_{c=1}^{k}\sum_{\{i\in\mathcal{G}_c^{o}\}}(s_i(\beta^{o}))^{\mathsf{T}}\sum_{\{j\in\mathcal{G}_c^{o}\}}\frac{\beta_i-\beta_j}{|\mathcal{G}_c^{o}|}
+\sum_{c=1}^{k}\sum_{\{i\in\mathcal{G}_c^{o}\}}(\operatorname{PS}_{ii}(\overline{\beta})-s_i(\beta^{o}))^{\mathsf{T}}\sum_{\{j\in\mathcal{G}_c^{o}\}}\frac{\beta_i-\beta_j}{|\mathcal{G}_c^{o}|}\\
&+\sum_{j=1}^{n}\sum_{\{i:i\ne j\}}(\operatorname{PS}_{ij}(\overline{\beta}))^{\mathsf{T}}(\beta_j-\beta_j^{*})\\
&\triangleq I_{11}+I_{12}+I_{13}, \tag*{(82)}
\end{aligned} I 1 = = = − i = 1 ∑ n ( Y i − m h − i ( β i T Z i ; β ) ) ( ∂ β i m h − i ( β i T Z i ; β ) ) T ( β i − β i ∗ ) − j = 1 ∑ n { i : i = j } ∑ ( Y i − m h − i ( β i T Z i ; β ) ) ( ∂ β j m h − i ( β i T Z i ; β ) ) T ( β j − β j ∗ ) c = 1 ∑ k { i ∈ G c o } ∑ ( PS ii ( β ) ) T { j ∈ G c o } ∑ ∣ G c o ∣ β i − β j + j = 1 ∑ n { i : i = j } ∑ ( PS ij ( β ) ) T ( β j − β j ∗ ) c = 1 ∑ k { i ∈ G c o } ∑ ( s i ( β o ) ) T { j ∈ G c o } ∑ ∣ G c o ∣ β i − β j + c = 1 ∑ k { i ∈ G c o } ∑ ( PS ii ( β ) − s i ( β o ) ) T { j ∈ G c o } ∑ ∣ G c o ∣ β i − β j + j = 1 ∑ n { i : i = j } ∑ ( PS ij ( β ) ) T ( β j − β j ∗ ) ≜ I 11 + I 12 + I 13 , (82)
where PS i j ( β ) = − ( Y i − m ^ h − i ( β i T Z i ; β ) ) ∂ β j m ^ h − i ( β i T Z i ; β ) \operatorname{PS}_{ij}(\beta)=-(Y_i-\widehat{m}_h^{-i}(\beta_i^{\mathsf{T}}Z_i;\beta))\partial_{\beta_j}\widehat{m}_h^{-i}(\beta_i^{\mathsf{T}}Z_i;\beta) PS ij ( β ) = − ( Y i − m h − i ( β i T Z i ; β )) ∂ β j m h − i ( β i T Z i ; β ) , s i ( β o ) = − ∑ c = 1 k I ( i ∈ G c o ) ( Y i − m c [ 0 ] ( Z i , s_i(\beta^{o})=-\sum_{c=1}^{k}I(i\in\mathcal{G}_c^{o})(Y_i-m_c^{[0]}(Z_i, s i ( β o ) = − ∑ c = 1 k I ( i ∈ G c o ) ( Y i − m c [ 0 ] ( Z i , γ c T Z i ) ) m c [ 1 ] ( Z i , γ c T Z i ) \gamma_c^\mathsf{T} Z_i))m_c^{[1]}(Z_i,\gamma_c^\mathsf{T} Z_i) γ c T Z i )) m c [ 1 ] ( Z i , γ c T Z i ) , i , j = 1 , … , n i,j=1,\ldots,n i , j = 1 , … , n , and m c [ t ] ( z , v ) m_c^{[t]}(z,v) m c [ t ] ( z , v ) is defined as m [ t ] ( w , v ) m^{[t]}(w,v) m [ t ] ( w , v ) with M s , t ( w , v ) M_{s,t}(w,v) M s , t ( w , v ) replaced by M c , s , t ( z , v ) = E [ ( m ( γ c T Z ) ) ∣ γ c T Z = v ] ( − z ∗ ) ⊗ t M_{c,s,t}(z,v)=\mathbb{E}[(m(\gamma_c^\mathsf{T} Z))\mid\gamma_c^\mathsf{T} Z=v](-z_*)^{\otimes t} M c , s , t ( z , v ) = E [( m ( γ c T Z )) ∣ γ c T Z = v ] ( − z ∗ ) ⊗ t , s , t = 0 , 1 s,t=0,1 s , t = 0 , 1 . A second-order Taylor expansion of K ( t ) ( ⋅ ) K^{(t)}(\cdot) K ( t ) ( ⋅ ) around ( β j o T Z j − β i o T Z i ) / h (\beta_j^{\mathsf{o}\mathsf{T}}Z_j-\beta_i^{\mathsf{o}\mathsf{T}}Z_i)/h ( β j oT Z j − β i oT Z i ) / h yields
1 ( n − 1 ) h t + 1 ∑ { j : j ≠ i } Y j s K ( t ) ( β j T Z j − β i T Z i h ) Z ∗ i ⊗ t = 1 ( n − 1 ) h t + 1 ∑ { j : j ≠ i } Y j s K ( t ) ( β j o T Z j − β i o T Z i h ) Z ∗ i ⊗ t + 1 ( n − 1 ) h t + 2 ∑ { j : j ≠ i } Y j s K ( t + 1 ) ( β j o T Z j − β i o T Z i h ) ( β j o T Z j − β i o T Z i h ) \begin{aligned}
\frac{1}{(n-1)h^{t+1}}\sum_{\{j:j\ne i\}}Y_j^sK^{(t)}\left(\frac{\beta_j^{\mathsf{T}}Z_j-\beta_i^{\mathsf{T}}Z_i}{h}\right)Z_{*i}^{\otimes t} \\
&=\frac{1}{(n-1)h^{t+1}}\sum_{\{j:j\ne i\}}Y_j^sK^{(t)}\left(\frac{\beta_j^{\mathsf{o}\mathsf{T}}Z_j-\beta_i^{\mathsf{o}\mathsf{T}}Z_i}{h}\right)Z_{*i}^{\otimes t}+\frac{1}{(n-1)h^{t+2}}\sum_{\{j:j\ne i\}}Y_j^sK^{(t+1)}\left(\frac{\beta_j^{\mathsf{o}\mathsf{T}}Z_j-\beta_i^{\mathsf{o}\mathsf{T}}Z_i}{h}\right)\left(\frac{\beta_j^{\mathsf{o}\mathsf{T}}Z_j-\beta_i^{\mathsf{o}\mathsf{T}}Z_i}{h}\right)
\end{aligned} ( n − 1 ) h t + 1 1 { j : j = i } ∑ Y j s K ( t ) ( h β j T Z j − β i T Z i ) Z ∗ i ⊗ t = ( n − 1 ) h t + 1 1 { j : j = i } ∑ Y j s K ( t ) ( h β j oT Z j − β i oT Z i ) Z ∗ i ⊗ t + ( n − 1 ) h t + 2 1 { j : j = i } ∑ Y j s K ( t + 1 ) ( h β j oT Z j − β i oT Z i ) ( h β j oT Z j − β i oT Z i )
[ Z j T ( β ˉ j − β j o ) − Z i T ( β ˉ i − β i o ) ] Z ∗ i ⊗ t + 1 2 ( n − 1 ) h t + 3 ∑ { j : j ≠ i } Y j s K ( t + 2 ) ( β ˉ j ∗ T Z j − β ˉ i ∗ T Z i h ) \left[Z_j^\mathsf{T}(\bar{\beta}_j-\beta_j^{\mathsf{o}})-Z_i^\mathsf{T}(\bar{\beta}_i-\beta_i^{\mathsf{o}})\right]Z_{*i}^{\otimes t}+\frac{1}{2(n-1)h^{t+3}}\sum_{\{j:j\ne i\}}Y_j^sK^{(t+2)}\left(\frac{\bar{\beta}_j^{* \mathsf{T}}Z_j-\bar{\beta}_i^{* \mathsf{T}}Z_i}{h}\right) [ Z j T ( β ˉ j − β j o ) − Z i T ( β ˉ i − β i o ) ] Z ∗ i ⊗ t + 2 ( n − 1 ) h t + 3 1 { j : j = i } ∑ Y j s K ( t + 2 ) ( h β ˉ j ∗ T Z j − β ˉ i ∗ T Z i )
[ ( Z j T ( β ˉ j − β j o ) ) 2 − 2 ( β ˉ j − β j o ) T Z j Z i T ( β ˉ i − β i o ) + ( Z i T ( β ˉ i − β i o ) ) 2 ] Z ∗ i ⊗ t , (83) \left[\left(Z_j^\mathsf{T}(\bar{\beta}_j-\beta_j^{\mathsf{o}})\right)^2-2(\bar{\beta}_j-\beta_j^{\mathsf{o}})^\mathsf{T}Z_jZ_i^\mathsf{T}(\bar{\beta}_i-\beta_i^{\mathsf{o}})+\left(Z_i^\mathsf{T}(\bar{\beta}_i-\beta_i^{\mathsf{o}})\right)^2\right]Z_{*i}^{\otimes t}, \tag*{(83)} [ ( Z j T ( β ˉ j − β j o ) ) 2 − 2 ( β ˉ j − β j o ) T Z j Z i T ( β ˉ i − β i o ) + ( Z i T ( β ˉ i − β i o ) ) 2 ] Z ∗ i ⊗ t , (83)
where β ˉ i ∗ \bar{\beta}_i^* β ˉ i ∗ lies on the line segment between β ˉ i \bar{\beta}_i β ˉ i and β i o \beta_i^{\mathsf{o}} β i o . Under assumptions A2 and A3, arguments parallel to those used in the proof of Lemma S1 establish
sup { i , c : i ∈ G c } ∥ 1 ( n − 1 ) h t + 1 ∑ { j : j ≠ i } Y j s K ( t ) ( β j o T Z j − β i o T Z i h ) Z ∗ i ⊗ t − ∂ v t ( M c , s , t ( Z i , γ c T Z i ) f c ( γ c T Z i ) ) ∥ = O p ( log n n h 2 t + 1 ) + O ( h 2 ) and (84) \begin{aligned}
\sup_{\{i,c:i\in\mathcal{G}_c\}}\left\|\frac{1}{(n-1)h^{t+1}}\sum_{\{j:j\ne i\}}Y_j^sK^{(t)}\left(\frac{\beta_j^{\mathsf{o}\mathsf{T}}Z_j-\beta_i^{\mathsf{o}\mathsf{T}}Z_i}{h}\right)Z_{*i}^{\otimes t}-\partial_v^t\left(M_{c,s,t}(Z_i,\gamma_c^\mathsf{T}Z_i)f_c(\gamma_c^\mathsf{T}Z_i)\right)\right\| \\
&=O_p\left(\sqrt{\frac{\log n}{nh^{2t+1}}}\right)+O(h^2)\ \text{and} \tag*{(84)}
\end{aligned} { i , c : i ∈ G c } sup ( n − 1 ) h t + 1 1 { j : j = i } ∑ Y j s K ( t ) ( h β j oT Z j − β i oT Z i ) Z ∗ i ⊗ t − ∂ v t ( M c , s , t ( Z i , γ c T Z i ) f c ( γ c T Z i ) ) = O p ( n h 2 t + 1 log n ) + O ( h 2 ) and (84)
sup { i , c : i ∈ G c } 1 ( n − 1 ) h t + 2 ∑ { j : j ≠ i } ∣ Y j s K ( t + 1 ) ( β j o T Z j − β i o T Z i h ) ∣ = O ( 1 h t + 1 ) + o p ( log n n h 2 t + 3 ) , (85) \sup_{\{i,c:i\in\mathcal{G}_c\}}\frac{1}{(n-1)h^{t+2}}\sum_{\{j:j\ne i\}}\left|Y_j^sK^{(t+1)}\left(\frac{\beta_j^{\mathsf{o}\mathsf{T}}Z_j-\beta_i^{\mathsf{o}\mathsf{T}}Z_i}{h}\right)\right|=O\left(\frac{1}{h^{t+1}}\right)+o_p\left(\sqrt{\frac{\log n}{nh^{2t+3}}}\right), \tag*{(85)} { i , c : i ∈ G c } sup ( n − 1 ) h t + 2 1 { j : j = i } ∑ Y j s K ( t + 1 ) ( h β j oT Z j − β i oT Z i ) = O ( h t + 1 1 ) + o p ( n h 2 t + 3 log n ) , (85)
where f c ( v ) f_c(v) f c ( v ) denotes the probability density function of γ c T Z i \gamma_c^\mathsf{T}Z_i γ c T Z i , c ∈ C c\in\mathcal{C} c ∈ C . Under assumption A2, there exists a constant c 1 > 0 c_1>0 c 1 > 0 such that
P ( sup i ∥ X i ∥ ≥ c 1 2 p log n ) ≤ ∑ i = 1 n ∑ j = 1 p P ( ∣ X i j ∣ ≥ c 1 2 log n ) ≤ 2 p n ⟶ 0 as n ⟶ ∞ . (86) \mathbb{P}\left(\sup_i\|X_i\|\ge c_1\sqrt{2p\log n}\right)\le\sum_{i=1}^{n}\sum_{j=1}^{p}\mathbb{P}\left(|X_{ij}|\ge c_1\sqrt{2\log n}\right)\le\frac{2p}{n}\longrightarrow0\ \text{as }n\longrightarrow\infty. \tag*{(86)} P ( i sup ∥ X i ∥ ≥ c 1 2 p log n ) ≤ i = 1 ∑ n j = 1 ∑ p P ( ∣ X ij ∣ ≥ c 1 2 log n ) ≤ n 2 p ⟶ 0 as n ⟶ ∞. (86)
By assumption A4, the triangle inequality, and the identity max i ∥ β ( 1 ) i ∗ − β ( 1 ) i o ∥ 2 = max c ∥ γ ( 1 ) c ∗ − γ ( 1 ) c o ∥ 2 \max_i\|\beta_{(1)i}^*-\beta_{(1)i}^{\mathsf{o}}\|^2=\max_c\|\gamma_{(1)c}^*-\gamma_{(1)c}^{\mathsf{o}}\|^2 max i ∥ β ( 1 ) i ∗ − β ( 1 ) i o ∥ 2 = max c ∥ γ ( 1 ) c ∗ − γ ( 1 ) c o ∥ 2 , we obtain
max i ∥ β ˉ ( 1 ) i − β ( 1 ) i o ∥ ≤ α 1 max i ∥ β ( 1 ) i − β ( 1 ) i o ∥ + ( 1 − α 1 ) max i ∥ β ( 1 ) i ∗ − β ( 1 ) i o ∥ ≤ Φ n . (87) \max_i\|\bar{\beta}_{(1)i}-\beta_{(1)i}^{\mathsf{o}}\|\le\alpha_1\max_i\|\beta_{(1)i}-\beta_{(1)i}^{\mathsf{o}}\|+(1-\alpha_1)\max_i\|\beta_{(1)i}^*-\beta_{(1)i}^{\mathsf{o}}\|\le\Phi_n. \tag*{(87)} i max ∥ β ˉ ( 1 ) i − β ( 1 ) i o ∥ ≤ α 1 i max ∥ β ( 1 ) i − β ( 1 ) i o ∥ + ( 1 − α 1 ) i max ∥ β ( 1 ) i ∗ − β ( 1 ) i o ∥ ≤ Φ n . (87)
It follows from (F.9)–(F.11) and the uniform boundedness of K ( t + 2 ) ( u ) K^{(t+2)}(u) K ( t + 2 ) ( u ) that
sup { i , c : i ∈ G c } ∥ 1 ( n − 1 ) h t + 2 ∑ { j : j ≠ i } Y j s K ( t + 1 ) ( β j o T Z j − β i o T Z i h ) [ Z j T ( β ˉ j − β j o ) − Z i T ( β ˉ i − β i o ) ] Z ∗ i ⊗ t ∥ ≤ ( sup { i , c : i ∈ G c } 1 ( n − 1 ) h t + 2 ∑ { j : j ≠ i } ∣ Y j s K ( t + 1 ) ( β j o T Z j − β i o T Z i h ) ∣ ) ( 2 Φ n sup i ∥ Z i ∥ t + 1 ) = O p ( ( log n ) t + 1 2 Φ n h t + 1 ) and (88) \begin{aligned}
\sup_{\{i,c:i\in\mathcal{G}_c\}}\left\|\frac{1}{(n-1)h^{t+2}}\sum_{\{j:j\ne i\}}Y_j^sK^{(t+1)}\left(\frac{\beta_j^{\mathsf{o}\mathsf{T}}Z_j-\beta_i^{\mathsf{o}\mathsf{T}}Z_i}{h}\right)\left[Z_j^\mathsf{T}(\bar{\beta}_j-\beta_j^{\mathsf{o}})-Z_i^\mathsf{T}(\bar{\beta}_i-\beta_i^{\mathsf{o}})\right]Z_{*i}^{\otimes t}\right\| \\
&\le\left(\sup_{\{i,c:i\in\mathcal{G}_c\}}\frac{1}{(n-1)h^{t+2}}\sum_{\{j:j\ne i\}}\left|Y_j^sK^{(t+1)}\left(\frac{\beta_j^{\mathsf{o}\mathsf{T}}Z_j-\beta_i^{\mathsf{o}\mathsf{T}}Z_i}{h}\right)\right|\right)\left(2\Phi_n\sup_i\|Z_i\|^{t+1}\right) \\
&=O_p\left(\frac{(\log n)^{\frac{t+1}{2}}\Phi_n}{h^{t+1}}\right)\ \text{and} \tag*{(88)}
\end{aligned} { i , c : i ∈ G c } sup ( n − 1 ) h t + 2 1 { j : j = i } ∑ Y j s K ( t + 1 ) ( h β j oT Z j − β i oT Z i ) [ Z j T ( β ˉ j − β j o ) − Z i T ( β ˉ i − β i o ) ] Z ∗ i ⊗ t ≤ { i , c : i ∈ G c } sup ( n − 1 ) h t + 2 1 { j : j = i } ∑ Y j s K ( t + 1 ) ( h β j oT Z j − β i oT Z i ) ( 2 Φ n i sup ∥ Z i ∥ t + 1 ) = O p ( h t + 1 ( log n ) 2 t + 1 Φ n ) and (88)
sup { i , c : i ∈ G c } ∥ 1 2 ( n − 1 ) h t + 3 ∑ { j : j ≠ i } Y j s K ( t + 2 ) ( β ˉ j ∗ T Z j − β ˉ i ∗ T Z i h ) [ ( Z j T ( β ˉ j − β j o ) ) 2 − 2 ( β ˉ j − β j o ) T Z j Z i T ≤ 1 2 ( n − 1 ) h t + 3 ∑ { j : j ≠ i } ∣ Y j s ∣ ∣ K ( t + 2 ) ( β ˉ j T Z j − β ˉ i T Z i h ) ∣ ( 4 ϕ n 2 sup i ∥ Z i ∥ t + 2 ) = o p ( ( log n ) t + 2 2 ϕ n 2 h t + 3 ) . (89) \begin{aligned}
\sup_{\{i,c:i\in\mathcal{G}_c\}}\|\frac{1}{2(n-1)h^{t+3}}\sum_{\{j:j\ne i\}}Y_j^sK^{(t+2)}(\frac{\bar{\beta}_j^{*\mathsf{T}}Z_j-\bar{\beta}_i^{*\mathsf{T}}Z_i}{h})[(Z_j^\mathsf{T}(\bar{\beta}_j-\beta_j^{\mathsf{o}}))^2-2(\bar{\beta}_j-\beta_j^{\mathsf{o}})^\mathsf{T}Z_jZ_i^\mathsf{T} \\
&\le\frac{1}{2(n-1)h^{t+3}}\sum_{\{j:j\ne i\}} |Y_j^s| \left|K^{(t+2)}\left(\frac{\bar{\beta}_j^{\mathsf{T}}Z_j-\bar{\beta}_i^{\mathsf{T}}Z_i}{h}\right)\right|\left(4\phi_n^2\sup_i\|Z_i\|^{t+2}\right) \\
&= o_p\left(\frac{(\log n)^{\frac{t+2}{2}}\phi_n^2}{h^{t+3}}\right). \tag*{(89)}
\end{aligned} { i , c : i ∈ G c } sup ∥ 2 ( n − 1 ) h t + 3 1 { j : j = i } ∑ Y j s K ( t + 2 ) ( h β ˉ j ∗ T Z j − β ˉ i ∗ T Z i ) [( Z j T ( β ˉ j − β j o ) ) 2 − 2 ( β ˉ j − β j o ) T Z j Z i T ≤ 2 ( n − 1 ) h t + 3 1 { j : j = i } ∑ ∣ Y j s ∣ K ( t + 2 ) ( h β ˉ j T Z j − β ˉ i T Z i ) ( 4 ϕ n 2 i sup ∥ Z i ∥ t + 2 ) = o p ( h t + 3 ( log n ) 2 t + 2 ϕ n 2 ) . (89)
Applying the triangle inequality to (F.7) and using (F.8), (F.12), and (F.13), we obtain
sup { i , c : i ∈ G c } ∥ 1 ( n − 1 ) h t + 1 ∑ { j : j ≠ i } Y j s K ( t ) ( β ˉ j T Z j − β ˉ i T Z i h ) Z ∗ i ⊗ t − ∂ β t ( M c , s , t ( Z i , γ c O T Z i ) f c ( γ c O T Z i ) ) ∥ ≤ o p ( log n n h 2 t + 1 ) + O ( h 2 ) + o p ( ( log n ) t + 1 2 ϕ n h t + 1 ) + o p ( ( log n ) t + 2 2 ϕ n 2 h t + 3 ) = o p ( 1 ) , t , s = 0 , 1. (90) \begin{aligned}
\sup_{\{i,c:i\in\mathcal{G}_c\}}\left\|\frac{1}{(n-1)h^{t+1}}\sum_{\{j:j\ne i\}}Y_j^sK^{(t)}\left(\frac{\bar{\beta}_j^{\mathsf{T}}Z_j-\bar{\beta}_i^{\mathsf{T}}Z_i}{h}\right)Z_{*i}^{\otimes t}-\partial_{\beta}^{t}(M_{c,s,t}(Z_i,\gamma_c^{\mathsf{O}\mathsf{T}}Z_i)f_c(\gamma_c^{\mathsf{O}\mathsf{T}}Z_i))\right\| \\
&\le o_p\left(\sqrt{\frac{\log n}{nh^{2t+1}}}\right)+O(h^2)+o_p\left(\frac{(\log n)^{\frac{t+1}{2}}\phi_n}{h^{t+1}}\right)+o_p\left(\frac{(\log n)^{\frac{t+2}{2}}\phi_n^2}{h^{t+3}}\right)=o_p(1),\quad t,s=0,1. \tag*{(90)}
\end{aligned} { i , c : i ∈ G c } sup ( n − 1 ) h t + 1 1 { j : j = i } ∑ Y j s K ( t ) ( h β ˉ j T Z j − β ˉ i T Z i ) Z ∗ i ⊗ t − ∂ β t ( M c , s , t ( Z i , γ c OT Z i ) f c ( γ c OT Z i )) ≤ o p ( n h 2 t + 1 log n ) + O ( h 2 ) + o p ( h t + 1 ( log n ) 2 t + 1 ϕ n ) + o p ( h t + 3 ( log n ) 2 t + 2 ϕ n 2 ) = o p ( 1 ) , t , s = 0 , 1. (90)
Combining (F.14) with assumption A1 and proceeding as in the proof of Lemma S2 with t = 0 , 1 t=0,1 t = 0 , 1 yields
sup { i , c : i ∈ G c } ∥ ∂ β i t m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m c [ t ] ( Z i , γ c O T Z i ) ∥ = o p ( ( log n ) t + 1 2 ϕ n h t + 1 ) = o p ( 1 ) , t = 0 , 1. (91) \sup_{\{i,c:i\in\mathcal{G}_c\}}\left\|\partial_{\beta_i}^{t}\hat{m}_h^{-i}(\bar{\beta}_i^{\mathsf{T}}Z_i;\bar{\beta})-m_c^{[t]}(Z_i,\gamma_c^{\mathsf{O}\mathsf{T}}Z_i)\right\|=o_p\left(\frac{(\log n)^{\frac{t+1}{2}}\phi_n}{h^{t+1}}\right)=o_p(1),\quad t=0,1. \tag*{(91)} { i , c : i ∈ G c } sup ∂ β i t m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m c [ t ] ( Z i , γ c OT Z i ) = o p ( h t + 1 ( log n ) 2 t + 1 ϕ n ) = o p ( 1 ) , t = 0 , 1. (91)
Consequently, the triangle inequality, (F.15), and assumptions A2–A4 imply
∣ I 12 ∣ ≤ 2 ϕ n ∑ c = 1 k ∑ { i ∈ G c } ∥ P S i i ( β ˉ ) − s i ( β O ) ∥ |I_{12}|\le2\phi_n\sum_{c=1}^{k}\sum_{\{i\in\mathcal{G}_c\}}\left\|\mathbb{P}S_{ii}(\bar{\beta})-s_i(\beta^{\mathsf{O}})\right\| ∣ I 12 ∣ ≤ 2 ϕ n c = 1 ∑ k { i ∈ G c } ∑ P S ii ( β ˉ ) − s i ( β O )
≤ 2 ϕ n ∑ c = 1 k ∑ { i ∈ G c } [ ∥ U i O ( ∂ β i m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m c [ 1 ] ( Z i , γ c O T Z i ) ) ∥ + ∣ m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m c [ 0 ] ( Z i , γ c O T Z i ) ∣ \le2\phi_n\sum_{c=1}^{k}\sum_{\{i\in\mathcal{G}_c\}}[\|U_i^{\mathsf{O}}(\partial_{\beta_i}\hat{m}_h^{-i}(\bar{\beta}_i^{\mathsf{T}}Z_i;\bar{\beta})-m_c^{[1]}(Z_i,\gamma_c^{\mathsf{O}\mathsf{T}}Z_i))\|+|\hat{m}_h^{-i}(\bar{\beta}_i^{\mathsf{T}}Z_i;\bar{\beta})-m_c^{[0]}(Z_i,\gamma_c^{\mathsf{O}\mathsf{T}}Z_i)| ≤ 2 ϕ n c = 1 ∑ k { i ∈ G c } ∑ [ ∥ U i O ( ∂ β i m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m c [ 1 ] ( Z i , γ c OT Z i )) ∥ + ∣ m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m c [ 0 ] ( Z i , γ c OT Z i ) ∣
( ∥ m c [ 1 ] ( Z i , γ c O T Z i ) ∥ + ∥ ∂ β i m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m c [ 1 ] ( Z i , γ c O T Z i ) ∥ ) ] = o p ( n ϕ n ) . (92) (\|m_c^{[1]}(Z_i,\gamma_c^{\mathsf{O}\mathsf{T}}Z_i)\|+\|\partial_{\beta_i}\hat{m}_h^{-i}(\bar{\beta}_i^{\mathsf{T}}Z_i;\bar{\beta})-m_c^{[1]}(Z_i,\gamma_c^{\mathsf{O}\mathsf{T}}Z_i)\|)]=o_p(n\phi_n). \tag*{(92)} ( ∥ m c [ 1 ] ( Z i , γ c OT Z i ) ∥ + ∥ ∂ β i m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m c [ 1 ] ( Z i , γ c OT Z i ) ∥ )] = o p ( n ϕ n ) . (92)
Furthermore, using the alternative representation of I 11 I_{11} I 11 in (F.6), I 11 = − ∑ c = 1 k ∑ { i , j ∈ G c , i < j } ( s i ( β O ) − s j ( β O ) ) T ( β j − β i ) / ∣ G c ∣ I_{11}=-\sum_{c=1}^{k}\sum_{\{i,j\in\mathcal{G}_c,i<j\}}(s_i(\beta^{\mathsf{O}})-s_j(\beta^{\mathsf{O}}))^{\mathsf{T}}(\beta_j-\beta_i)/|\mathcal{G}_c| I 11 = − ∑ c = 1 k ∑ { i , j ∈ G c , i < j } ( s i ( β O ) − s j ( β O ) ) T ( β j − β i ) /∣ G c ∣ , together with assumptions A2 and A3, we obtain
P ( ∣ I 11 ∣ > n α ∑ c = 1 k ∑ { i , j ∈ G c , i < j } ∥ β j − β i ∥ ∣ G c ∣ ) ≤ P ( ∑ c = 1 k ∑ { i , j ∈ G c , i < j } ∥ s i ( β O ) − s j ( β O ) ∥ ∥ β j − β i ∥ ∣ G c ∣ > n α ∑ c = 1 k ∑ { i , j ∈ G c , i < j } ∥ β j − β i ∥ ∣ G c ∣ ) \begin{aligned}
P\left(|I_{11}|>n^\alpha\sum_{c=1}^{k}\sum_{\{i,j\in\mathcal{G}_c,i<j\}}\frac{\|\beta_j-\beta_i\|}{|\mathcal{G}_c|}\right) \\
&\le P\left(\sum_{c=1}^{k}\sum_{\{i,j\in\mathcal{G}_c,i<j\}}\|s_i(\beta^{\mathsf{O}})-s_j(\beta^{\mathsf{O}})\|\frac{\|\beta_j-\beta_i\|}{|\mathcal{G}_c|}>n^\alpha\sum_{c=1}^{k}\sum_{\{i,j\in\mathcal{G}_c,i<j\}}\frac{\|\beta_j-\beta_i\|}{|\mathcal{G}_c|}\right)
\end{aligned} P ∣ I 11 ∣ > n α c = 1 ∑ k { i , j ∈ G c , i < j } ∑ ∣ G c ∣ ∥ β j − β i ∥ ≤ P c = 1 ∑ k { i , j ∈ G c , i < j } ∑ ∥ s i ( β O ) − s j ( β O ) ∥ ∣ G c ∣ ∥ β j − β i ∥ > n α c = 1 ∑ k { i , j ∈ G c , i < j } ∑ ∣ G c ∣ ∥ β j − β i ∥
≤ 2 n − α E [ ∥ s 1 ( β O ) ∥ ] ⟶ 0 as n ⟶ ∞ for some sufficiently small fixed α > 0. \le2n^{-\alpha}\mathbb{E}[\|s_1(\beta^{\mathsf{O}})\|]\longrightarrow0\text{ as }n\longrightarrow\infty\text{ for some sufficiently small fixed }\alpha>0. ≤ 2 n − α E [ ∥ s 1 ( β O ) ∥ ] ⟶ 0 as n ⟶ ∞ for some sufficiently small fixed α > 0.
It follows that, with probability converging to one,
∣ I 11 ∣ ≤ n α ( ∑ c = 1 k 1 ∣ G c ∣ ∑ { i , j ∈ G c , i < j } ∥ β ( 1 ) i − β ( 1 ) j ∥ ) . (93) |I_{11}|\le n^\alpha\left(\sum_{c=1}^{k}\frac{1}{|\mathcal{G}_c|}\sum_{\{i,j\in\mathcal{G}_c,i<j\}}\|\beta_{(1)i}-\beta_{(1)j}\|\right). \tag*{(93)} ∣ I 11 ∣ ≤ n α c = 1 ∑ k ∣ G c ∣ 1 { i , j ∈ G c , i < j } ∑ ∥ β ( 1 ) i − β ( 1 ) j ∥ . (93)
By the uniform convergence result in (F.14) with s = t = 0 s=t=0 s = t = 0 , I 13 I_{13} I 13 in (F.6) has the representation:
I 13 = [ − 1 ( n − 1 ) h 2 ∑ j = 1 n ( ∑ c = 1 k ∑ { i : i ∈ G c , i ≠ j } E i i E j i f c ( γ c O T Z i ) K ( 1 ) ( β j O T Z j − β i O T Z i h ) Z j T ( β j − β j ∗ ) ) ] ( 1 + o p ( 1 ) ) , (94) I_{13}=\left[-\frac{1}{(n-1)h^2}\sum_{j=1}^{n}\left(\sum_{c=1}^{k}\sum_{\{i:i\in\mathcal{G}_c,i\ne j\}}\frac{E_{ii}E_{ji}}{f_c(\gamma_c^{\mathsf{O}\mathsf{T}}Z_i)}K^{(1)}\left(\frac{\beta_j^{\mathsf{O}\mathsf{T}}Z_j-\beta_i^{\mathsf{O}\mathsf{T}}Z_i}{h}\right)Z_j^{\mathsf{T}}(\beta_j-\beta_j^*)\right)\right](1+o_p(1)), \tag*{(94)} I 13 = − ( n − 1 ) h 2 1 j = 1 ∑ n c = 1 ∑ k { i : i ∈ G c , i = j } ∑ f c ( γ c OT Z i ) E ii E j i K ( 1 ) ( h β j OT Z j − β i OT Z i ) Z j T ( β j − β j ∗ ) ( 1 + o p ( 1 )) , (94)
where E j i = Y j − m ^ h − i ( β ˉ i T Z i ; β ˉ ) = U j O + ( m ( β j O T Z j ) − m ( β i O T Z i ) ) − ( m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m ( β i O T Z i ) ) E_{ji}=Y_j-\hat{m}_h^{-i}(\bar{\beta}_i^{\mathsf{T}}Z_i;\bar{\beta})=U_j^{\mathsf{O}}+(m(\beta_j^{\mathsf{O}\mathsf{T}}Z_j)-m(\beta_i^{\mathsf{O}\mathsf{T}}Z_i))-(\hat{m}_h^{-i}(\bar{\beta}_i^{\mathsf{T}}Z_i;\bar{\beta})-m(\beta_i^{\mathsf{O}\mathsf{T}}Z_i)) E j i = Y j − m ^ h − i ( β ˉ i T Z i ; β ˉ ) = U j O + ( m ( β j OT Z j ) − m ( β i OT Z i )) − ( m ^ h − i ( β ˉ i T Z i ; β ˉ ) − m ( β i OT Z i )) , i , j = 1 , … , n i,j=1,\ldots,n i , j = 1 , … , n . Using the argument for F 1 t 2 \mathcal{F}_{1t_2} F 1 t 2 in the proof of Lemma S3, we conclude that under assumptions A1–A3 and A5, the following class is Euclidean:
{ U o f c 2 ( γ c 2 o T Z ) ( m ( γ c 1 o T z ) − m ( γ c 2 o T Z ) ) s K ( 1 ) ( γ c 1 o T z − γ c 2 o T Z h ) : x ∈ X n , c 1 , c 2 ∈ C } , s = 0 , 1. \left\{\frac{U^{o}}{f_{c_2}(\gamma_{c_2}^{o\mathsf{T}}Z)}\left(m(\gamma_{c_1}^{o\mathsf{T}}z)-m(\gamma_{c_2}^{o\mathsf{T}}Z)\right)^s K^{(1)}\left(\frac{\gamma_{c_1}^{o\mathsf{T}}z-\gamma_{c_2}^{o\mathsf{T}}Z}{h}\right):x\in\mathcal{X}_n,c_1,c_2\in\mathcal{C}\right\},\quad s=0,1. { f c 2 ( γ c 2 o T Z ) U o ( m ( γ c 1 o T z ) − m ( γ c 2 o T Z ) ) s K ( 1 ) ( h γ c 1 o T z − γ c 2 o T Z ) : x ∈ X n , c 1 , c 2 ∈ C } , s = 0 , 1.
By E [ U o ∣ W ] = 0 \mathbb{E}[U^o\mid W]=0 E [ U o ∣ W ] = 0 and
sup { x ∈ X n , c 1 , c 2 ∈ C } E [ ( 1 h 2 U o ( m ( γ c 1 o T z ) − m ( γ c 2 o T Z ) ) s K ( 1 ) ( γ c 1 o T z − γ c 2 o T Z h ) ) 2 ] = O ( 1 h 3 ) , s = 0 , 1 , \sup_{\{x\in\mathcal{X}_n,c_1,c_2\in\mathcal{C}\}}\mathbb{E}\left[\left(\frac{1}{h^2}U^o\left(m(\gamma_{c_1}^{o\mathsf{T}}z)-m(\gamma_{c_2}^{o\mathsf{T}}Z)\right)^sK^{(1)}\left(\frac{\gamma_{c_1}^{o\mathsf{T}}z-\gamma_{c_2}^{o\mathsf{T}}Z}{h}\right)\right)^2\right]=O\left(\frac{1}{h^3}\right),\quad s=0,1, { x ∈ X n , c 1 , c 2 ∈ C } sup E ( h 2 1 U o ( m ( γ c 1 o T z ) − m ( γ c 2 o T Z ) ) s K ( 1 ) ( h γ c 1 o T z − γ c 2 o T Z ) ) 2 = O ( h 3 1 ) , s = 0 , 1 ,
Theorem II.37 in Pollard [24 ] implies
sup { x ∈ X n , c 1 , c 2 ∈ C } 1 ( n − 1 ) h 2 ∣ ∑ { i : i ∈ G c 2 o , i ≠ j } U i o f c 2 ( γ c 2 o T Z i ) ( m ( γ c 1 o T z ) − m ( γ c 2 o T Z i ) ) s K ( 1 ) ( γ c 1 o T z − γ c 2 o T Z i h ) ∣ = o p ( log n n h 3 ) = o p ( 1 ) , s = 0 , 1. (95) \begin{aligned}
\sup_{\{x\in\mathcal{X}_n,c_1,c_2\in\mathcal{C}\}}\frac{1}{(n-1)h^2}\left|\sum_{\{i:i\in\mathcal{G}_{c_2}^{o},i\ne j\}}\frac{U_i^o}{f_{c_2}(\gamma_{c_2}^{o\mathsf{T}}Z_i)}\left(m(\gamma_{c_1}^{o\mathsf{T}}z)-m(\gamma_{c_2}^{o\mathsf{T}}Z_i)\right)^sK^{(1)}\left(\frac{\gamma_{c_1}^{o\mathsf{T}}z-\gamma_{c_2}^{o\mathsf{T}}Z_i}{h}\right)\right| \\
&= o_p\left(\sqrt{\frac{\log n}{nh^3}}\right)=o_p(1),\quad s=0,1. \tag*{(95)}
\end{aligned} { x ∈ X n , c 1 , c 2 ∈ C } sup ( n − 1 ) h 2 1 { i : i ∈ G c 2 o , i = j } ∑ f c 2 ( γ c 2 o T Z i ) U i o ( m ( γ c 1 o T z ) − m ( γ c 2 o T Z i ) ) s K ( 1 ) ( h γ c 1 o T z − γ c 2 o T Z i ) = o p ( n h 3 log n ) = o p ( 1 ) , s = 0 , 1. (95)
It follows from (F.15) with t = 0 t=0 t = 0 that under assumptions A1–A3 and A5,
sup { j , c } ∣ 1 ( n − 1 ) h 2 ∑ { i : i ∈ G c o , i ≠ j } ( U i o ) s 1 f c ( γ c o T Z i ) ( m ( β j o T Z j ) − m ( β i o T Z i ) ) s 2 ( m ^ h − i ( β ˉ T Z i ; β ˉ ) − m ( β j o T Z i ) ) s 3 K ( 1 ) ( β j o T Z j − β i o T Z i h ) ∣ ≤ ( sup { j , c } ∣ 1 ( n − 1 ) h 2 ∑ { i : i ∈ G c o , i ≠ j } ∣ U i o ∣ s 1 f c ( γ c o T Z i ) ∣ m ( β j o T Z j ) − m ( β i o T Z i ) ∣ s 2 K ( 1 ) ( β j o T Z j − β i o T Z i h ) ∣ ) \begin{aligned}
\sup_{\{j,c\}}\left|\frac{1}{(n-1)h^2}\sum_{\{i:i\in\mathcal{G}_{c}^{o},i\ne j\}}\frac{(U_i^o)^{s_1}}{f_c(\gamma_c^{o\mathsf{T}}Z_i)}\left(m(\beta_j^{o\mathsf{T}}Z_j)-m(\beta_i^{o\mathsf{T}}Z_i)\right)^{s_2}\left(\widehat{m}_h^{-i}(\bar{\beta}_{T}Z_i;\bar{\beta})-m(\beta_j^{o\mathsf{T}}Z_i)\right)^{s_3}K^{(1)}\left(\frac{\beta_j^{o\mathsf{T}}Z_j-\beta_i^{o\mathsf{T}}Z_i}{h}\right)\right| \\
&\leq\left(\sup_{\{j,c\}}\left|\frac{1}{(n-1)h^2}\sum_{\{i:i\in\mathcal{G}_{c}^{o},i\ne j\}}\frac{|U_i^o|^{s_1}}{f_c(\gamma_c^{o\mathsf{T}}Z_i)}\left|m(\beta_j^{o\mathsf{T}}Z_j)-m(\beta_i^{o\mathsf{T}}Z_i)\right|^{s_2}K^{(1)}\left(\frac{\beta_j^{o\mathsf{T}}Z_j-\beta_i^{o\mathsf{T}}Z_i}{h}\right)\right|\right)
\end{aligned} { j , c } sup ( n − 1 ) h 2 1 { i : i ∈ G c o , i = j } ∑ f c ( γ c o T Z i ) ( U i o ) s 1 ( m ( β j o T Z j ) − m ( β i o T Z i ) ) s 2 ( m h − i ( β ˉ T Z i ; β ˉ ) − m ( β j o T Z i ) ) s 3 K ( 1 ) ( h β j o T Z j − β i o T Z i ) ≤ { j , c } sup ( n − 1 ) h 2 1 { i : i ∈ G c o , i = j } ∑ f c ( γ c o T Z i ) ∣ U i o ∣ s 1 m ( β j o T Z j ) − m ( β i o T Z i ) s 2 K ( 1 ) ( h β j o T Z j − β i o T Z i )
× ( sup { i , c : i ∈ G c o } ∣ m ^ h − i ( β ˉ T Z i ; β ˉ ) − m ( β i o T Z i ) ∣ ) s 3 = ( O ( 1 h ) + o p ( log n n h 3 ) ) ( o p ( ( log n ) 1 / 2 ϕ n h ) ) s 3 = o p ( 1 ) , s 1 , s 2 = 0 , 1 , s 3 = 1 , 2. (96) \begin{aligned}
\qquad\times\left(\sup_{\{i,c:i\in\mathcal{G}_{c}^{o}\}}\left|\widehat{m}_h^{-i}(\bar{\beta}_{T}Z_i;\bar{\beta})-m(\beta_i^{o\mathsf{T}}Z_i)\right|\right)^{s_3} \\
&=\left(O\left(\frac{1}{h}\right)+o_p\left(\sqrt{\frac{\log n}{nh^3}}\right)\right)\left(o_p\left(\frac{(\log n)^{1/2}\phi_n}{h}\right)\right)^{s_3}=o_p(1),\quad s_1,s_2=0,1,\ s_3=1,2. \tag*{(96)}
\end{aligned} × ( { i , c : i ∈ G c o } sup m h − i ( β ˉ T Z i ; β ˉ ) − m ( β i o T Z i ) ) s 3 = ( O ( h 1 ) + o p ( n h 3 log n ) ) ( o p ( h ( log n ) 1/2 ϕ n ) ) s 3 = o p ( 1 ) , s 1 , s 2 = 0 , 1 , s 3 = 1 , 2. (96)
Applying the triangular inequality to (F.18) and invoking (F.19), (F.20), and assumption A4, we obtain
∣ I 13 ∣ ≤ ( sup j ∣ 1 ( n − 1 ) h 2 ∑ c = 1 k ∑ { i : i ∈ G c o , i ≠ j } E i i E j i f c ( γ c o T Z i ) K ( 1 ) ( β j o T Z j − β i o T Z i h ) ∣ ) ( ∑ j = 1 n ∣ U j o ∣ ∥ Z j ∥ ) |I_{13}|\leq\left(\sup_j\left|\frac{1}{(n-1)h^2}\sum_{c=1}^{k}\sum_{\{i:i\in\mathcal{G}_{c}^{o},i\ne j\}}\frac{E_{ii}E_{ji}}{f_c(\gamma_c^{o\mathsf{T}}Z_i)}K^{(1)}\left(\frac{\beta_j^{o\mathsf{T}}Z_j-\beta_i^{o\mathsf{T}}Z_i}{h}\right)\right|\right)\left(\sum_{j=1}^{n}|U_j^o|\|Z_j\|\right) ∣ I 13 ∣ ≤ j sup ( n − 1 ) h 2 1 c = 1 ∑ k { i : i ∈ G c o , i = j } ∑ f c ( γ c o T Z i ) E ii E j i K ( 1 ) ( h β j o T Z j − β i o T Z i ) ( j = 1 ∑ n ∣ U j o ∣∥ Z j ∥ )
× ( max j ∥ β j − β j ∗ ∥ ) ( 1 + o p ( 1 ) ) = o p ( n ϕ n ) . (97) \qquad\times\left(\max_j\|\beta_j-\beta_j^*\|\right)(1+o_p(1))=o_p(n\phi_n). \tag*{(97)} × ( j max ∥ β j − β j ∗ ∥ ) ( 1 + o p ( 1 )) = o p ( n ϕ n ) . (97)
Substituting (F.16), (F.17), and (F.21) into I 1 I_1 I 1 in (F.6) yields
I 1 ≥ − n α ( ∑ c = 1 k 1 ∣ G c o ∣ ∑ { i , j ∈ G c o , i < j } ∥ β ( 1 ) i − β ( 1 ) j ∥ ) ( 1 + o p ( 1 ) ) . (98) I_1\geq-n^\alpha\left(\sum_{c=1}^{k}\frac{1}{|\mathcal{G}_{c}^{o}|}\sum_{\{i,j\in\mathcal{G}_{c}^{o},i<j\}}\|\beta_{(1)i}-\beta_{(1)j}\|\right)(1+o_p(1)). \tag*{(98)} I 1 ≥ − n α c = 1 ∑ k ∣ G c o ∣ 1 { i , j ∈ G c o , i < j } ∑ ∥ β ( 1 ) i − β ( 1 ) j ∥ ( 1 + o p ( 1 )) . (98)
Given ( β , γ ) ∈ B n × Γ n (\beta,\gamma)\in B_n\times\Gamma_n ( β , γ ) ∈ B n × Γ n , it follows from the triangle inequality that under assumption A4, i ∈ G c 1 o i\in\mathcal{G}_{c_1}^{o} i ∈ G c 1 o , and c 1 ≠ c 2 c_1\ne c_2 c 1 = c 2 ,
∥ β ( 1 ) i − γ ( 1 ) c 1 ∥ 1 ≤ ∥ β ( 1 ) i − β ( 1 ) i o ∥ 1 + ∥ γ ( 1 ) c 1 − γ ( 1 ) c 1 o ∥ 1 ≤ 2 ϕ n \|\beta_{(1)i}-\gamma_{(1)c_1}\|_1\leq\|\beta_{(1)i}-\beta_{(1)i}^{o}\|_1+\|\gamma_{(1)c_1}-\gamma_{(1)c_1}^{o}\|_1\leq2\phi_n ∥ β ( 1 ) i − γ ( 1 ) c 1 ∥ 1 ≤ ∥ β ( 1 ) i − β ( 1 ) i o ∥ 1 + ∥ γ ( 1 ) c 1 − γ ( 1 ) c 1 o ∥ 1 ≤ 2 ϕ n
and
∥ β ( 1 ) i − γ ( 1 ) c 2 ∥ 1 = ∥ β ( 1 ) i − β ( 1 ) i o + γ ( 1 ) c 1 o − γ ( 1 ) c 2 o + γ ( 1 ) c 2 o − γ ( 1 ) c 2 ∥ 1 ≥ ∥ γ ( 1 ) c 1 o − γ ( 1 ) c 2 o ∥ 1 − ∥ β ( 1 ) i − β ( 1 ) i o ∥ 1 − ∥ γ ( 1 ) c 2 o − γ ( 1 ) c 2 ∥ 1 ≥ D γ − 2 ϕ n > 2 ϕ n . \begin{aligned}
\|\beta_{(1)i}-\gamma_{(1)c_2}\|_1=\|\beta_{(1)i}-\beta_{(1)i}^{o}+\gamma_{(1)c_1}^{o}-\gamma_{(1)c_2}^{o}+\gamma_{(1)c_2}^{o}-\gamma_{(1)c_2}\|_1 \\
&\geq\|\gamma_{(1)c_1}^{o}-\gamma_{(1)c_2}^{o}\|_1-\|\beta_{(1)i}-\beta_{(1)i}^{o}\|_1-\|\gamma_{(1)c_2}^{o}-\gamma_{(1)c_2}\|_1\geq D_\gamma-2\phi_n>2\phi_n.
\end{aligned} ∥ β ( 1 ) i − γ ( 1 ) c 2 ∥ 1 = ∥ β ( 1 ) i − β ( 1 ) i o + γ ( 1 ) c 1 o − γ ( 1 ) c 2 o + γ ( 1 ) c 2 o − γ ( 1 ) c 2 ∥ 1 ≥ ∥ γ ( 1 ) c 1 o − γ ( 1 ) c 2 o ∥ 1 − ∥ β ( 1 ) i − β ( 1 ) i o ∥ 1 − ∥ γ ( 1 ) c 2 o − γ ( 1 ) c 2 ∥ 1 ≥ D γ − 2 ϕ n > 2 ϕ n .
Consequently, min c ∈ C ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 = ∥ β ( 1 ) i − γ ( 1 ) c 1 ∥ 1 \min_{c\in\mathcal{C}}\|\beta_{(1)i}-\gamma_{(1)c}\|_1=\|\beta_{(1)i}-\gamma_{(1)c_1}\|_1 min c ∈ C ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 = ∥ β ( 1 ) i − γ ( 1 ) c 1 ∥ 1 . Using this fact and the norm inequality ∥ ⋅ ∥ 1 ≥ ∥ ⋅ ∥ \|\cdot\|_1\geq\|\cdot\| ∥ ⋅ ∥ 1 ≥ ∥ ⋅ ∥ , we obtain
I 2 = λ ∑ c = 1 k ∑ { i ∈ G c } ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 = λ ∑ c = 1 k 1 2 ∣ G c o ∣ ∑ { i , j ∈ G c o } ( ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 + ∥ β ( 1 ) j − γ ( 1 ) c ∥ 1 ) ≥ λ ∑ c = 1 k 1 ∣ G c o ∣ ∑ { i , j ∈ G c o , i < j } ∥ β ( 1 ) i − β ( 1 ) j ∥ 1 ≥ λ ∑ c = 1 k 1 ∣ G c o ∣ ∑ { i , j ∈ G c o , i < j } ∥ β ( 1 ) i − β ( 1 ) j ∥ . (99) \begin{aligned}
I_2=\lambda\sum_{c=1}^{k}\sum_{\{i\in\mathcal{G}_c\}}\lVert\beta_{(1)i}-\gamma_{(1)c}\rVert_1=\lambda\sum_{c=1}^{k}\frac{1}{2|\mathcal{G}_c^o|}\sum_{\{i,j\in\mathcal{G}_c^o\}}\left(\lVert\beta_{(1)i}-\gamma_{(1)c}\rVert_1+\lVert\beta_{(1)j}-\gamma_{(1)c}\rVert_1\right) \\
&\geq\lambda\sum_{c=1}^{k}\frac{1}{|\mathcal{G}_c^o|}\sum_{\{i,j\in\mathcal{G}_c^o,i<j\}}\lVert\beta_{(1)i}-\beta_{(1)j}\rVert_1\geq\lambda\sum_{c=1}^{k}\frac{1}{|\mathcal{G}_c^o|}\sum_{\{i,j\in\mathcal{G}_c^o,i<j\}}\lVert\beta_{(1)i}-\beta_{(1)j}\rVert. \tag*{(99)}
\end{aligned} I 2 = λ c = 1 ∑ k { i ∈ G c } ∑ ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 = λ c = 1 ∑ k 2∣ G c o ∣ 1 { i , j ∈ G c o } ∑ ( ∥ β ( 1 ) i − γ ( 1 ) c ∥ 1 + ∥ β ( 1 ) j − γ ( 1 ) c ∥ 1 ) ≥ λ c = 1 ∑ k ∣ G c o ∣ 1 { i , j ∈ G c o , i < j } ∑ ∥ β ( 1 ) i − β ( 1 ) j ∥ 1 ≥ λ c = 1 ∑ k ∣ G c o ∣ 1 { i , j ∈ G c o , i < j } ∑ ∥ β ( 1 ) i − β ( 1 ) j ∥ . (99)
Substituting (F.22) and (F.23) into (F.5) yields
PSS S P ( β , γ ; λ ) − PSS S P ( β ∗ , γ ∗ ; λ ) ≥ ( λ − n α ) ( ∑ c = 1 k 1 ∣ G c o ∣ ∑ { i , j ∈ G c o , i < j } ∥ β ( 1 ) i − β ( 1 ) j ∥ ) ( 1 + o p ( 1 ) ) . (100) \begin{aligned}
\operatorname{PSS}_{\mathrm{SP}}(\beta,\gamma;\lambda)-\operatorname{PSS}_{\mathrm{SP}}(\beta^*,\gamma^*;\lambda) \\
&\geq(\lambda-n^\alpha)\left(\sum_{c=1}^{k}\frac{1}{|\mathcal{G}_c^o|}\sum_{\{i,j\in\mathcal{G}_c^o,i<j\}}\lVert\beta_{(1)i}-\beta_{(1)j}\rVert\right)(1+o_p(1)). \tag*{(100)}
\end{aligned} PSS SP ( β , γ ; λ ) − PSS SP ( β ∗ , γ ∗ ; λ ) ≥ ( λ − n α ) c = 1 ∑ k ∣ G c o ∣ 1 { i , j ∈ G c o , i < j } ∑ ∥ β ( 1 ) i − β ( 1 ) j ∥ ( 1 + o p ( 1 )) . (100)
The local oracle property in Theorem 2 follows directly from (F.4) and (F.24).
Proof of Theorem 3
Define
N s h ( v ; G , γ ) = 1 n ∑ i = 1 n Y i s ∑ c = 1 k I ( i ∈ G c ) K h ( γ c T Z i − v ) , s = 0 , 1. N_{\mathrm{sh}}(v;\mathcal{G},\gamma)=\frac{1}{n}\sum_{i=1}^{n}Y_i^s\sum_{c=1}^{k}I(i\in\mathcal{G}_c)K_h(\gamma_c^{\mathsf T}Z_i-v),\quad s=0,1. N sh ( v ; G , γ ) = n 1 i = 1 ∑ n Y i s c = 1 ∑ k I ( i ∈ G c ) K h ( γ c T Z i − v ) , s = 0 , 1.
Since P ( G ^ = G o ) ⟶ 1 P(\widehat{\mathcal{G}}=\mathcal{G}^o)\longrightarrow1 P ( G = G o ) ⟶ 1 as n ⟶ ∞ n\longrightarrow\infty n ⟶ ∞ , we have
n r sup { i , c } ∣ I ( i ∈ G ^ c ) − I ( i ∈ G c o ) ∣ = o p ( 1 ) (101) n^r\sup_{\{i,c\}}\left|I(i\in\widehat{\mathcal{G}}_c)-I(i\in\mathcal{G}_c^o)\right|=o_p(1) \tag*{(101)} n r { i , c } sup I ( i ∈ G c ) − I ( i ∈ G c o ) = o p ( 1 ) (101)
for each positive integer r r r . By (G.1), h ~ = O p ( n − 1 / 5 ) \widetilde h=O_p(n^{-1/5}) h = O p ( n − 1/5 ) , the uniform boundedness of K ( u ) K(u) K ( u ) , and the triangle inequality, it follows that under assumptions A2 and A3,
sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ N h ~ ( γ T w ; G ^ , γ ) − N h ~ ( γ T w ; G o , γ ) ∣ = sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ 1 n ∑ i = 1 n Y i s ∑ c = 1 k ( I ( i ∈ G ^ c ) − I ( i ∈ G c o ) ) K h ~ ( γ c T Z i − γ T w ) ∣ ≤ sup { i , c } ∣ I ( i ∈ G ^ c ) − I ( i ∈ G c o ) ∣ sup { ( w , γ ∗ ) ∈ W n × Γ } ( 1 n ∑ i = 1 n ( ∣ m ( γ ∗ T Z i ) ∣ + ∣ U i o ∣ ) s ∑ c = 1 k K h ~ ( γ c T Z i − γ T w ) ) ≤ o p ( log n n 4 / 5 ) , s = 0 , 1. (102) \begin{aligned}
\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left|N_{\widetilde h}(\gamma^{\mathsf T}w;\widehat{\mathcal{G}},\gamma)-N_{\widetilde h}(\gamma^{\mathsf T}w;\mathcal{G}^o,\gamma)\right| \\
&=\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left|\frac{1}{n}\sum_{i=1}^{n}Y_i^s\sum_{c=1}^{k}\left(I(i\in\widehat{\mathcal{G}}_c)-I(i\in\mathcal{G}_c^o)\right)K_{\widetilde h}(\gamma_c^{\mathsf T}Z_i-\gamma^{\mathsf T}w)\right| \\
&\leq\sup_{\{i,c\}}\left|I(i\in\widehat{\mathcal{G}}_c)-I(i\in\mathcal{G}_c^o)\right|\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left(\frac{1}{n}\sum_{i=1}^{n}\left(|m(\gamma_*^{\mathsf T}Z_i)|+|U_i^o|\right)^s\sum_{c=1}^{k}K_{\widetilde h}(\gamma_c^{\mathsf T}Z_i-\gamma^{\mathsf T}w)\right) \\
&\leq o_p\left(\sqrt{\frac{\log n}{n^{4/5}}}\right),\quad s=0,1. \tag*{(102)}
\end{aligned} {( w , γ ∗ ) ∈ W n × Γ } sup N h ( γ T w ; G , γ ) − N h ( γ T w ; G o , γ ) = {( w , γ ∗ ) ∈ W n × Γ } sup n 1 i = 1 ∑ n Y i s c = 1 ∑ k ( I ( i ∈ G c ) − I ( i ∈ G c o ) ) K h ( γ c T Z i − γ T w ) ≤ { i , c } sup I ( i ∈ G c ) − I ( i ∈ G c o ) {( w , γ ∗ ) ∈ W n × Γ } sup ( n 1 i = 1 ∑ n ( ∣ m ( γ ∗ T Z i ) ∣ + ∣ U i o ∣ ) s c = 1 ∑ k K h ( γ c T Z i − γ T w ) ) ≤ o p ( n 4/5 log n ) , s = 0 , 1. (102)
Under assumptions A2, A3, and A5, combining (G.2) with Lemma S1 (with t = 0 t=0 t = 0 ) yields
sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ m ~ h ~ ( γ T w ; G ^ , γ ) − m ~ h ~ ( γ T w ; G o , γ ) ∣ ≤ sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ N 0 h ~ ( γ T w ; G ^ , γ ) − N 0 h ~ ( γ T w ; G o , γ ) ∣ sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ N 1 h ~ ( γ T w ; G o , γ ) ∣ ( inf { ( w , γ ∗ ) ∈ W n × Γ } f γ ( γ T w ) + o p ( log n n 4 / 5 ) ) 2 + sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ N 1 h ~ ( γ T w ; G ^ , γ ) − N 1 h ~ ( γ T w ; G o , γ ) ∣ sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ N 0 h ~ ( γ T w ; G o , γ ) ∣ ( inf { ( w , γ ∗ ) ∈ W n × Γ } f γ ( γ T w ) + o p ( log n n 4 / 5 ) ) 2 = o p ( log n n 4 / 5 ) . (103) \begin{aligned}
\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}\left|\widetilde m_{\widetilde h}(\gamma^{\mathsf T}w;\widehat{\mathcal{G}},\gamma)-\widetilde m_{\widetilde h}(\gamma^{\mathsf T}w;\mathcal{G}^o,\gamma)\right| \\
&\leq\frac{\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}|N_{0\widetilde h}(\gamma^{\mathsf T}w;\widehat{\mathcal{G}},\gamma)-N_{0\widetilde h}(\gamma^{\mathsf T}w;\mathcal{G}^o,\gamma)|\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}|N_{1\widetilde h}(\gamma^{\mathsf T}w;\mathcal{G}^o,\gamma)|}{\left(\inf_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}f_\gamma(\gamma^{\mathsf T}w)+o_p\left(\sqrt{\frac{\log n}{n^{4/5}}}\right)\right)^2} \\
&+\frac{\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}|N_{1\widetilde h}(\gamma^{\mathsf T}w;\widehat{\mathcal{G}},\gamma)-N_{1\widetilde h}(\gamma^{\mathsf T}w;\mathcal{G}^o,\gamma)|\sup_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}|N_{0\widetilde h}(\gamma^{\mathsf T}w;\mathcal{G}^o,\gamma)|}{\left(\inf_{\{(w,\gamma_*)\in\mathcal{W}_n\times\Gamma\}}f_\gamma(\gamma^{\mathsf T}w)+o_p\left(\sqrt{\frac{\log n}{n^{4/5}}}\right)\right)^2} \\
&=o_p\left(\sqrt{\frac{\log n}{n^{4/5}}}\right). \tag*{(103)}
\end{aligned} {( w , γ ∗ ) ∈ W n × Γ } sup m h ( γ T w ; G , γ ) − m h ( γ T w ; G o , γ ) ≤ ( inf {( w , γ ∗ ) ∈ W n × Γ } f γ ( γ T w ) + o p ( n 4/5 l o g n ) ) 2 sup {( w , γ ∗ ) ∈ W n × Γ } ∣ N 0 h ( γ T w ; G , γ ) − N 0 h ( γ T w ; G o , γ ) ∣ sup {( w , γ ∗ ) ∈ W n × Γ } ∣ N 1 h ( γ T w ; G o , γ ) ∣ + ( inf {( w , γ ∗ ) ∈ W n × Γ } f γ ( γ T w ) + o p ( n 4/5 l o g n ) ) 2 sup {( w , γ ∗ ) ∈ W n × Γ } ∣ N 1 h ( γ T w ; G , γ ) − N 1 h ( γ T w ; G o , γ ) ∣ sup {( w , γ ∗ ) ∈ W n × Γ } ∣ N 0 h ( γ T w ; G o , γ ) ∣ = o p ( n 4/5 log n ) . (103)
Arguments parallel to those used in the proof of Theorem 1 establish
∥ γ ^ − γ o ∥ = { O p ( 1 n ) G o ∈ C G , o p ( 1 ) otherwise , (104) \lVert\widehat{\gamma}-\gamma^{o}\rVert=
\begin{cases}
O_{p}\left(\frac{1}{\sqrt{n}}\right) & \mathcal{G}^{o}\in\mathcal{C}_{\mathcal{G}},\\
o_{p}(1) & \text{otherwise},
\end{cases} \tag*{(104)} ∥ γ − γ o ∥ = { O p ( n 1 ) o p ( 1 ) G o ∈ C G , otherwise , (104)
where
C G = { G o : G c ∗ o = ⋃ { ℓ : G ℓ o ⊆ G c ∗ o , γ ℓ o = γ c ∗ ∗ } G ℓ o ∀ c ∈ C ∗ } . \mathcal{C}_{\mathcal{G}}=\left\{\mathcal{G}^{o}:\mathcal{G}^{o}_{c^{*}}=
\bigcup_{\{\ell:\mathcal{G}^{o}_{\ell}\subseteq\mathcal{G}^{o}_{c^{*}},\gamma^{o}_{\ell}=\gamma^{*}_{c^{*}}\}}
\mathcal{G}^{o}_{\ell}\ \forall c\in\mathcal{C}^{*}\right\}. C G = ⎩ ⎨ ⎧ G o : G c ∗ o = { ℓ : G ℓ o ⊆ G c ∗ o , γ ℓ o = γ c ∗ ∗ } ⋃ G ℓ o ∀ c ∈ C ∗ ⎭ ⎬ ⎫ .
The decomposition of PSS ( γ ^ , G ^ ) \operatorname{PSS}(\widehat{\gamma},\widehat{\mathcal{G}}) PSS ( γ , G ) admits the following representation:
2 n PSS ( γ ^ , G ^ ) = 1 n ∑ i = 1 n ( U i o ) 2 + 1 n ∑ i = 1 n ( m ^ h − i ( γ ^ T W ^ i ; G ^ , γ ^ ) − m [ 0 ] ( W i , γ o T W i ) ) 2 − 2 n ∑ i = 1 n U i o ( m ^ h − i ( γ ^ T W ^ i ; G ^ , γ ^ ) − m [ 0 ] ( W i , γ o T W i ) ) ≔ I 1 ( k ) + I 2 ( k ) + I 3 ( k ) , (105) \frac{2}{n}\operatorname{PSS}(\widehat{\gamma},\widehat{\mathcal{G}})
=\frac{1}{n}\sum_{i=1}^{n}(U_{i}^{o})^{2}
+\frac{1}{n}\sum_{i=1}^{n}\left(\widehat{m}_{h}^{-i}(\widehat{\gamma}^{\mathsf{T}}\widehat{W}_{i};\widehat{\mathcal{G}},\widehat{\gamma})-m^{[0]}(W_{i},\gamma^{o\mathsf{T}}W_{i})\right)^{2}
-\frac{2}{n}\sum_{i=1}^{n}U_{i}^{o}\left(\widehat{m}_{h}^{-i}(\widehat{\gamma}^{\mathsf{T}}\widehat{W}_{i};\widehat{\mathcal{G}},\widehat{\gamma})-m^{[0]}(W_{i},\gamma^{o\mathsf{T}}W_{i})\right)
\coloneqq I_{1}(k)+I_{2}(k)+I_{3}(k), \tag*{(105)} n 2 PSS ( γ , G ) = n 1 i = 1 ∑ n ( U i o ) 2 + n 1 i = 1 ∑ n ( m h − i ( γ T W i ; G , γ ) − m [ 0 ] ( W i , γ o T W i ) ) 2 − n 2 i = 1 ∑ n U i o ( m h − i ( γ T W i ; G , γ ) − m [ 0 ] ( W i , γ o T W i ) ) : = I 1 ( k ) + I 2 ( k ) + I 3 ( k ) , (105)
where W i = ( I ( i ∈ G 1 o ) , … , I ( i ∈ G k o ) ) T ⊗ Z i W_{i}=(I(i\in\mathcal{G}_{1}^{o}),\ldots,I(i\in\mathcal{G}_{k}^{o}))^{\mathsf{T}}\otimes Z_{i} W i = ( I ( i ∈ G 1 o ) , … , I ( i ∈ G k o ) ) T ⊗ Z i denotes the alternative representation of W i W_{i} W i introduced in § 3.1 , and W ^ i = ( I ( i ∈ G ^ 1 ) , … , I ( i ∈ G ^ k ) ) T ⊗ Z i \widehat{W}_{i}=(I(i\in\widehat{\mathcal{G}}_{1}),\ldots,I(i\in\widehat{\mathcal{G}}_{k}))^{\mathsf{T}}\otimes Z_{i} W i = ( I ( i ∈ G 1 ) , … , I ( i ∈ G k ) ) T ⊗ Z i for i = 1 , … , n i=1,\ldots,n i = 1 , … , n . By U i o = U i ∗ U_{i}^{o}=U_{i}^{*} U i o = U i ∗ when G o ∈ C G \mathcal{G}^{o}\in\mathcal{C}_{\mathcal{G}} G o ∈ C G , and by applying the law of large numbers to I 1 ( k ) I_{1}(k) I 1 ( k ) otherwise, we obtain
I 1 ( k ) − I 1 ( k ∗ ) = { 0 G o ∈ C G , b 2 ( k ) ( 1 + o p ( 1 ) ) otherwise , (106) I_{1}(k)-I_{1}(k^{*})=
\begin{cases}
0 & \mathcal{G}^{o}\in\mathcal{C}_{\mathcal{G}},\\
b^{2}(k)(1+o_{p}(1)) & \text{otherwise},
\end{cases} \tag*{(106)} I 1 ( k ) − I 1 ( k ∗ ) = { 0 b 2 ( k ) ( 1 + o p ( 1 )) G o ∈ C G , otherwise , (106)
where b 2 ( k ) = E [ ( m [ 0 ] ( W , γ o T W ) − m ( γ ∗ T W ∗ ) ) 2 ] > 0 b^{2}(k)=\mathbb{E}\left[(m^{[0]}(W,\gamma^{o\mathsf{T}}W)-m(\gamma^{* \mathsf{T}}W^{*}))^{2}\right]>0 b 2 ( k ) = E [ ( m [ 0 ] ( W , γ o T W ) − m ( γ ∗ T W ∗ ) ) 2 ] > 0 when G o ∉ C G \mathcal{G}^{o}\notin\mathcal{C}_{\mathcal{G}} G o ∈ / C G . Straightforward algebra yields
m ^ h − i ( γ ^ T W ^ i ; G ^ , γ ^ ) − m [ 0 ] ( W i , γ o T W i ) = ( m ^ h − i ( γ ^ T W ^ i ; G ^ , γ ^ ) − m ^ h − i ( γ ^ T W ^ i ; G o , γ ^ ) ) + ( m ^ h − i ( γ ^ T W ^ i ; G o , γ ^ ) − m [ 0 ] ( W i , γ ^ T W i ) ) + ( m [ 0 ] ( W i , γ ^ T W i ) − m [ 0 ] ( W i , γ o T W i ) ) ≔ I I 1 i + I I 2 i + I I 3 i , i = 1 , … , n . (107) \begin{aligned}
\widehat{m}_{h}^{-i}(\widehat{\gamma}^{\mathsf{T}}\widehat{W}_{i};\widehat{\mathcal{G}},\widehat{\gamma})-m^{[0]}(W_{i},\gamma^{o\mathsf{T}}W_{i}) \\
&=\left(\widehat{m}_{h}^{-i}(\widehat{\gamma}^{\mathsf{T}}\widehat{W}_{i};\widehat{\mathcal{G}},\widehat{\gamma})-\widehat{m}_{h}^{-i}(\widehat{\gamma}^{\mathsf{T}}\widehat{W}_{i};\mathcal{G}^{o},\widehat{\gamma})\right)
+\left(\widehat{m}_{h}^{-i}(\widehat{\gamma}^{\mathsf{T}}\widehat{W}_{i};\mathcal{G}^{o},\widehat{\gamma})-m^{[0]}(W_{i},\widehat{\gamma}^{\mathsf{T}}W_{i})\right) \\
&+\left(m^{[0]}(W_{i},\widehat{\gamma}^{\mathsf{T}}W_{i})-m^{[0]}(W_{i},\gamma^{o\mathsf{T}}W_{i})\right)
\coloneqq II_{1i}+II_{2i}+II_{3i},\quad i=1,\ldots,n. \tag*{(107)}
\end{aligned} m h − i ( γ T W i ; G , γ ) − m [ 0 ] ( W i , γ o T W i ) = ( m h − i ( γ T W i ; G , γ ) − m h − i ( γ T W i ; G o , γ ) ) + ( m h − i ( γ T W i ; G o , γ ) − m [ 0 ] ( W i , γ T W i ) ) + ( m [ 0 ] ( W i , γ T W i ) − m [ 0 ] ( W i , γ o T W i ) ) : = I I 1 i + I I 2 i + I I 3 i , i = 1 , … , n . (107)
By (G.1) and Lemma S2 with t = 0 t=0 t = 0 , it follows that
sup i ∣ I I 1 i ∣ ≤ k ( sup { i , c } ∣ I ( i ∈ G ^ c ) − I ( i ∈ G c o ) ∣ ) ( sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ m ^ h ( γ T w ; G o , γ ) − m [ 0 ] ( w , γ T w ) ∣ + sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ m [ 0 ] ( w , γ T w ) ∣ ) = o p ( 1 n ) \sup_{i}|II_{1i}|\leq k\left(\sup_{\{i,c\}}|I(i\in\widehat{\mathcal{G}}_{c})-I(i\in\mathcal{G}_{c}^{o})|\right)
\left(\sup_{\{(w,\gamma^{*})\in\mathcal{W}_{n}\times\Gamma\}}\left|\widehat{m}_{h}(\gamma^{\mathsf{T}}w;\mathcal{G}^{o},\gamma)-m^{[0]}(w,\gamma^{\mathsf{T}}w)\right|
+\sup_{\{(w,\gamma^{*})\in\mathcal{W}_{n}\times\Gamma\}}\left|m^{[0]}(w,\gamma^{\mathsf{T}}w)\right|\right)
=o_{p}\left(\frac{1}{n}\right) i sup ∣ I I 1 i ∣ ≤ k ( { i , c } sup ∣ I ( i ∈ G c ) − I ( i ∈ G c o ) ∣ ) ( {( w , γ ∗ ) ∈ W n × Γ } sup m h ( γ T w ; G o , γ ) − m [ 0 ] ( w , γ T w ) + {( w , γ ∗ ) ∈ W n × Γ } sup m [ 0 ] ( w , γ T w ) ) = o p ( n 1 )
and
sup i ∣ I I 2 i ∣ ≤ sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ m ^ h ( γ T w ; G o , γ ) − m [ 0 ] ( w , γ T w ) ∣ = o p ( log n n 4 / 5 ) . (108) \sup_{i}|II_{2i}|\leq\sup_{\{(w,\gamma^{*})\in\mathcal{W}_{n}\times\Gamma\}}\left|\widehat{m}_{h}(\gamma^{\mathsf{T}}w;\mathcal{G}^{o},\gamma)-m^{[0]}(w,\gamma^{\mathsf{T}}w)\right|
=o_{p}\left(\sqrt{\frac{\log n}{n^{4/5}}}\right). \tag*{(108)} i sup ∣ I I 2 i ∣ ≤ {( w , γ ∗ ) ∈ W n × Γ } sup m h ( γ T w ; G o , γ ) − m [ 0 ] ( w , γ T w ) = o p ( n 4/5 log n ) . (108)
For each i = 1 , … , n i=1,\ldots,n i = 1 , … , n , first- and second-order Taylor expansions of m [ 0 ] ( w , γ ^ T w ) m^{[0]}(w,\widehat{\gamma}^{\mathsf{T}}w) m [ 0 ] ( w , γ T w ) around γ ^ = γ o \widehat{\gamma}=\gamma^{o} γ = γ o yield
I I 3 i = ( ∂ γ m [ 0 ] ( W i , γ ‾ 1 T W i ) ) ( γ ^ − γ o ) T W i = ∂ γ m [ 0 ] ( W i , γ o T W i ) ( γ ^ − γ o ) T W i + 1 2 ( ∂ γ 2 m [ 0 ] ( W i , γ ‾ T W i ) ) ( ( γ ^ − γ o ) T W i ) 2 , (109) \begin{aligned}
II_{3i}=(\partial_{\gamma}m^{[0]}(W_{i},\overline{\gamma}_{1}^{\mathsf{T}}W_{i}))(\widehat{\gamma}-\gamma^{o})^{\mathsf{T}}W_{i} \\
&=\partial_{\gamma}m^{[0]}(W_{i},\gamma^{o\mathsf{T}}W_{i})(\widehat{\gamma}-\gamma^{o})^{\mathsf{T}}W_{i}
+\frac{1}{2}(\partial_{\gamma}^{2}m^{[0]}(W_{i},\overline{\gamma}^{\mathsf{T}}W_{i}))\left((\widehat{\gamma}-\gamma^{o})^{\mathsf{T}}W_{i}\right)^{2}, \tag*{(109)}
\end{aligned} I I 3 i = ( ∂ γ m [ 0 ] ( W i , γ 1 T W i )) ( γ − γ o ) T W i = ∂ γ m [ 0 ] ( W i , γ o T W i ) ( γ − γ o ) T W i + 2 1 ( ∂ γ 2 m [ 0 ] ( W i , γ T W i )) ( ( γ − γ o ) T W i ) 2 , (109)
where γ ‾ t \overline{\gamma}_{t} γ t lies on the line segment between γ ^ \widehat{\gamma} γ and γ o \gamma^{o} γ o , t = 0 , 1 t=0,1 t = 0 , 1 . When G o ∈ C G \mathcal{G}^{o}\in\mathcal{C}_{\mathcal{G}} G o ∈ C G , we have E [ U o W ∂ γ m [ 0 ] ( W , γ o T W ) ] = 0 \mathbb{E}[U^{o}W\partial_{\gamma}m^{[0]}(W,\gamma^{o\mathsf{T}}W)]=0 E [ U o W ∂ γ m [ 0 ] ( W , γ o T W )] = 0 . Moreover, under assumptions A2 and A3, E [ ∣ ∣ ( U o ∂ γ m [ 0 ] ( W , γ o T W ) 2 W ⊗ W T ) ∣ ∣ < ∞ \mathbb{E}[||(U^{o}\partial_{\gamma}m^{[0]}(W,\gamma^{o\mathsf{T}}W)^{2}W\otimes W^{\mathsf{T}})||<\infty E [ ∣∣ ( U o ∂ γ m [ 0 ] ( W , γ o T W ) 2 W ⊗ W T ) ∣∣ < ∞ and E [ ∣ U o ∣ s ∣ ∣ W ⊗ t ∣ ∣ ] < ∞ \mathbb{E}[|U^{o}|^{s}||W\otimes t||]<\infty E [ ∣ U o ∣ s ∣∣ W ⊗ t ∣∣ ] < ∞ , s = 0 , 1 s=0,1 s = 0 , 1 , t = 1 , 2 t=1,2 t = 1 , 2 . Consequently, by the central limit theorem and the law of large numbers,
1 n ∑ i = 1 n U i o ( ∂ γ m [ 0 ] ( W i , γ o T W i ) ) W i = O p ( 1 n ) and 1 n ∑ i = 1 n ∣ U i o ∣ s ∣ ∣ W i ⊗ t ∣ ∣ = O p ( 1 ) , s = 0 , 1 , t = 1 , 2. (110) \frac{1}{n}\sum_{i=1}^{n}U_{i}^{o}\left(\partial_{\gamma}m^{[0]}(W_{i},\gamma^{o\mathsf{T}}W_{i})\right)W_{i}=O_{p}\left(\frac{1}{\sqrt{n}}\right)
\quad\text{and}\quad
\frac{1}{n}\sum_{i=1}^{n}|U_{i}^{o}|^{s}||W_{i}^{\otimes t}||=O_{p}(1),\quad s=0,1,\ t=1,2. \tag*{(110)} n 1 i = 1 ∑ n U i o ( ∂ γ m [ 0 ] ( W i , γ o T W i ) ) W i = O p ( n 1 ) and n 1 i = 1 ∑ n ∣ U i o ∣ s ∣∣ W i ⊗ t ∣∣ = O p ( 1 ) , s = 0 , 1 , t = 1 , 2. (110)
It follows from (G.4), the second-order Taylor expansion in (G.10), and (G.11) that
∣ 1 n ∑ i = 1 n U i ∘ I I 3 i ∣ ≤ ∥ γ ^ − γ ∘ ∥ ∥ 1 n ∑ i = 1 n U i ∘ ( ∂ v m [ 0 ] ( W i , γ ∘ T W i ) ) W i ∥ + 1 2 ( sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ ∂ v 2 m [ 0 ] ( w , γ T w ) ∣ ) ∥ γ ^ − γ ∘ ∥ 2 ( 1 n ∑ i = 1 n ∣ U i ∘ ∣ ∥ W i ∥ 2 ) = O p ( 1 n ) O p ( 1 n ) + O p ( 1 n ) = O p ( 1 n ) . \begin{aligned}
\left|\frac{1}{n}\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{3i}\right|\leq\left\|\widehat{\gamma}-\gamma^\circ\right\|\left\|\frac{1}{n}\sum_{i=1}^{n}U_i^\circ\left(\partial_v m^{[0]}(W_i,\gamma^{\circ\mathrm{T}}W_i)\right)W_i\right\| \\
&+\frac{1}{2}\left(\sup_{\{(w,\gamma^\ast)\in\mathcal{W}_n\times\Gamma\}}\left|\partial_v^2m^{[0]}(w,\gamma^{\mathrm{T}}w)\right|\right)\left\|\widehat{\gamma}-\gamma^\circ\right\|^2\left(\frac{1}{n}\sum_{i=1}^{n}|U_i^\circ|\|W_i\|^2\right) \\
&=O_p\left(\frac{1}{\sqrt{n}}\right)O_p\left(\frac{1}{\sqrt{n}}\right)+O_p\left(\frac{1}{n}\right)=O_p\left(\frac{1}{n}\right).
\end{aligned} n 1 i = 1 ∑ n U i ∘ II 3 i ≤ ∥ γ − γ ∘ ∥ n 1 i = 1 ∑ n U i ∘ ( ∂ v m [ 0 ] ( W i , γ ∘ T W i ) ) W i + 2 1 ( {( w , γ ∗ ) ∈ W n × Γ } sup ∂ v 2 m [ 0 ] ( w , γ T w ) ) ∥ γ − γ ∘ ∥ 2 ( n 1 i = 1 ∑ n ∣ U i ∘ ∣∥ W i ∥ 2 ) = O p ( n 1 ) O p ( n 1 ) + O p ( n 1 ) = O p ( n 1 ) .
When G ∘ ∉ C G \mathcal{G}^\circ\notin\mathcal{C}_{\mathcal{G}} G ∘ ∈ / C G , the first-order Taylor expansion in (G.10), (G.11), and assumption A3 ensure that
∣ 1 n ∑ i = 1 n U i ∘ I I 3 i ∣ ≤ ( sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ ∂ v m [ 0 ] ( w , γ T w ) ∣ ) ∥ γ ^ − γ ∘ ∥ ( 1 n ∑ i = 1 n ∣ U i ∘ ∣ ∥ W i ∥ ) = O p ( ∥ γ ^ − γ ∘ ∥ ) . \left|\frac{1}{n}\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{3i}\right|\leq\left(\sup_{\{(w,\gamma^\ast)\in\mathcal{W}_n\times\Gamma\}}\left|\partial_vm^{[0]}(w,\gamma^{\mathrm{T}}w)\right|\right)\left\|\widehat{\gamma}-\gamma^\circ\right\|\left(\frac{1}{n}\sum_{i=1}^{n}|U_i^\circ|\|W_i\|\right)=O_p\left(\|\widehat{\gamma}-\gamma^\circ\|\right). n 1 i = 1 ∑ n U i ∘ II 3 i ≤ ( {( w , γ ∗ ) ∈ W n × Γ } sup ∂ v m [ 0 ] ( w , γ T w ) ) ∥ γ − γ ∘ ∥ ( n 1 i = 1 ∑ n ∣ U i ∘ ∣∥ W i ∥ ) = O p ( ∥ γ − γ ∘ ∥ ) .
Hence,
∣ 1 n ∑ i = 1 n U i ∘ I I 3 i ∣ = { O p ( 1 n ) G ∘ ∈ C G , O p ( ∥ γ ^ − γ ∘ ∥ ) otherwise . (111) \left|\frac{1}{n}\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{3i}\right|=
\begin{cases}
O_p\left(\frac{1}{n}\right)&\mathcal{G}^\circ\in\mathcal{C}_{\mathcal{G}},\\
O_p\left(\|\widehat{\gamma}-\gamma^\circ\|\right)&\text{otherwise}.
\end{cases} \tag*{(111)} n 1 i = 1 ∑ n U i ∘ II 3 i = { O p ( n 1 ) O p ( ∥ γ − γ ∘ ∥ ) G ∘ ∈ C G , otherwise . (111)
Moreover, by (G.4), the first-order Taylor expansion in (G.10), and (G.11),
1 n ∑ i = 1 n I I 3 i 2 ≤ ( sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ ∂ v m [ 0 ] ( w , γ T w ) ∣ ) ( γ ^ − γ ∘ ) T ( 1 n ∑ i = 1 n W i ⊗ 2 ) ( γ ^ − γ ∘ ) = { O p ( 1 n ) G ∘ ∈ C G , O p ( ∥ γ ^ − γ ∘ ∥ 2 ) otherwise . (112) \begin{aligned}
\frac{1}{n}\sum_{i=1}^{n}\mathrm{II}_{3i}^{2}\leq\left(\sup_{\{(w,\gamma^\ast)\in\mathcal{W}_n\times\Gamma\}}\left|\partial_vm^{[0]}(w,\gamma^{\mathrm{T}}w)\right|\right)(\widehat{\gamma}-\gamma^\circ)^{\mathrm{T}}\left(\frac{1}{n}\sum_{i=1}^{n}W_i^{\otimes2}\right)(\widehat{\gamma}-\gamma^\circ) \\
&=
\begin{cases}
O_p\left(\frac{1}{n}\right)&\mathcal{G}^\circ\in\mathcal{C}_{\mathcal{G}},\\
O_p\left(\|\widehat{\gamma}-\gamma^\circ\|^2\right)&\text{otherwise}.
\end{cases} \tag*{(112)}
\end{aligned} n 1 i = 1 ∑ n II 3 i 2 ≤ ( {( w , γ ∗ ) ∈ W n × Γ } sup ∂ v m [ 0 ] ( w , γ T w ) ) ( γ − γ ∘ ) T ( n 1 i = 1 ∑ n W i ⊗ 2 ) ( γ − γ ∘ ) = { O p ( n 1 ) O p ( ∥ γ − γ ∘ ∥ 2 ) G ∘ ∈ C G , otherwise . (112)
Combining I 2 ( k ) ≤ O ( ( sup i ∣ I 1 i ∣ ) 2 + ( sup i ∣ I 2 i ∣ ) 2 + ∑ i = 1 n I 3 i 2 / n ) I_2(k)\leq O((\sup_i|I_{1i}|)^2+(\sup_i|I_{2i}|)^2+\sum_{i=1}^{n}I_{3i}^2/n) I 2 ( k ) ≤ O (( sup i ∣ I 1 i ∣ ) 2 + ( sup i ∣ I 2 i ∣ ) 2 + ∑ i = 1 n I 3 i 2 / n ) with (G.8), (G.9), and (G.13) yields
I 2 ( k ) ≤ { O p ( log n n 4 5 ) G ∘ ∈ C G , o p ( log n n 4 5 ) + O p ( ∥ γ ^ − γ ∘ ∥ 2 ) otherwise . (113) I_2(k)\leq
\begin{cases}
O_p\left(\frac{\log n}{n^{\frac{4}{5}}}\right)&\mathcal{G}^\circ\in\mathcal{C}_{\mathcal{G}},\\
o_p\left(\frac{\log n}{n^{\frac{4}{5}}}\right)+O_p\left(\|\widehat{\gamma}-\gamma^\circ\|^2\right)&\text{otherwise}.
\end{cases} \tag*{(113)} I 2 ( k ) ≤ ⎩ ⎨ ⎧ O p ( n 5 4 l o g n ) o p ( n 5 4 l o g n ) + O p ( ∥ γ − γ ∘ ∥ 2 ) G ∘ ∈ C G , otherwise . (113)
By ∑ i = 1 n ∣ U i ∘ ∣ / n → p E [ ∣ U ∘ ∣ ] \sum_{i=1}^{n}|U_i^\circ|/n\xrightarrow{p}\mathbb{E}[|U^\circ|] ∑ i = 1 n ∣ U i ∘ ∣/ n p E [ ∣ U ∘ ∣ ] , (G.8), and assumption A2,
∣ 1 n ∑ i = 1 n U i ∘ I I 1 i ∣ ≤ ( 1 n ∑ i = 1 n ∣ U i ∘ ∣ ) sup i ∣ I I 1 i ∣ = o p ( 1 n ) . (114) \left|\frac{1}{n}\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{1i}\right|\leq\left(\frac{1}{n}\sum_{i=1}^{n}|U_i^\circ|\right)\sup_i|\mathrm{II}_{1i}|=o_p\left(\frac{1}{n}\right). \tag*{(114)} n 1 i = 1 ∑ n U i ∘ II 1 i ≤ ( n 1 i = 1 ∑ n ∣ U i ∘ ∣ ) i sup ∣ II 1 i ∣ = o p ( n 1 ) . (114)
It follows from Lemma § 2 with t = 0 t=0 t = 0 that
1 n ∑ i = 1 n U i ∘ I I 2 i = 1 n ( n − 1 ) ∑ i ≠ j U i ∘ ξ j [ 0 ] ( W i , γ ^ ) + 1 n ∑ i = 1 n U i ∘ r [ 0 ] ( W i , γ ^ ) , \frac{1}{n}\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{2i}=\frac{1}{n(n-1)}\sum_{i\ne j}U_i^\circ\xi_j^{[0]}(W_i,\widehat{\gamma})+\frac{1}{n}\sum_{i=1}^{n}U_i^\circ r^{[0]}(W_i,\widehat{\gamma}), n 1 i = 1 ∑ n U i ∘ II 2 i = n ( n − 1 ) 1 i = j ∑ U i ∘ ξ j [ 0 ] ( W i , γ ) + n 1 i = 1 ∑ n U i ∘ r [ 0 ] ( W i , γ ) ,
with the second term satisfying
∣ 1 n ∑ i = 1 n U i ∘ r [ 0 ] ( W i , γ ^ ) ∣ ≤ ( 1 n ∑ i = 1 n ∣ U i ∘ ∣ ) sup { ( w , γ ∗ ) ∈ W n × Γ } ∣ r [ 0 ] ( w , γ ) ∣ = o p ( log n n 4 5 ) . \left|\frac{1}{n}\sum_{i=1}^{n}U_i^\circ r^{[0]}(W_i,\widehat{\gamma})\right|\leq\left(\frac{1}{n}\sum_{i=1}^{n}|U_i^\circ|\right)\sup_{\{(w,\gamma^\ast)\in\mathcal{W}_n\times\Gamma\}}|r^{[0]}(w,\gamma)|=o_p\left(\frac{\log n}{n^{\frac{4}{5}}}\right). n 1 i = 1 ∑ n U i ∘ r [ 0 ] ( W i , γ ) ≤ ( n 1 i = 1 ∑ n ∣ U i ∘ ∣ ) {( w , γ ∗ ) ∈ W n × Γ } sup ∣ r [ 0 ] ( w , γ ) ∣ = o p ( n 5 4 log n ) .
Hence,
∣ 1 n ∑ i = 1 n U i ∘ I I 2 i ∣ ≤ ∣ 1 n ( n − 1 ) ∑ i ≠ j U i ∘ ξ j [ 0 ] ( W i , γ ^ ) ∣ + o p ( log n n 4 5 ) . (115) \left|\frac{1}{n}\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{2i}\right|\leq\left|\frac{1}{n(n-1)}\sum_{i\ne j}U_i^\circ\xi_j^{[0]}(W_i,\widehat{\gamma})\right|+o_p\left(\frac{\log n}{n^{\frac{4}{5}}}\right). \tag*{(115)} n 1 i = 1 ∑ n U i ∘ II 2 i ≤ n ( n − 1 ) 1 i = j ∑ U i ∘ ξ j [ 0 ] ( W i , γ ) + o p ( n 5 4 log n ) . (115)
Using the n \sqrt{n} n -consistency of γ ^ ∗ \widehat{\gamma}_\ast γ ∗ to γ ∘ \gamma^\circ γ ∘ in (G.4), and applying the arguments in (E.11) and (E.13) to the first term on the right-hand side of (G.16) when G ∘ ∈ C G \mathcal{G}^\circ\in\mathcal{C}_{\mathcal{G}} G ∘ ∈ C G , we obtain
∣ 1 n ∑ i = 1 n U i ∘ I I 2 i ∣ ≤ O p ( 1 n 9 10 ) + o p ( log n n 4 5 ) = o p ( log n n 4 5 ) . (116) \left|\frac{1}{n}\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{2i}\right|\leq O_p\left(\frac{1}{n^{\frac{9}{10}}}\right)+o_p\left(\frac{\log n}{n^{\frac{4}{5}}}\right)=o_p\left(\frac{\log n}{n^{\frac{4}{5}}}\right). \tag*{(116)} n 1 i = 1 ∑ n U i ∘ II 2 i ≤ O p ( n 10 9 1 ) + o p ( n 5 4 log n ) = o p ( n 5 4 log n ) . (116)
Combining ∣ I 3 ( k ) ∣ ≤ O ( ∣ ∑ i = 1 n U i ∘ I I 1 i / n ∣ + ∣ ∑ i = 1 n U i ∘ I I 2 i / n ∣ + ∣ ∑ i = 1 n U i ∘ I I 3 i / n ∣ ) |I_3(k)|\leq O(|\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{1i}/n|+|\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{2i}/n|+|\sum_{i=1}^{n}U_i^\circ\mathrm{II}_{3i}/n|) ∣ I 3 ( k ) ∣ ≤ O ( ∣ ∑ i = 1 n U i ∘ II 1 i / n ∣ + ∣ ∑ i = 1 n U i ∘ II 2 i / n ∣ + ∣ ∑ i = 1 n U i ∘ II 3 i / n ∣ ) with (G.12), (G.15), and (G.17) yields
I 3 ( k ) = o p ( log n n 4 / 5 ) if G o ∈ C G . (117) I_3(k)=o_p\left(\frac{\log n}{n^{4/5}}\right)\ \text{if } \mathcal{G}^{o}\in\mathcal{C}_{\mathcal{G}}. \tag*{(117)} I 3 ( k ) = o p ( n 4/5 log n ) if G o ∈ C G . (117)
When G o ∉ C G \mathcal{G}^{o}\notin\mathcal{C}_{\mathcal{G}} G o ∈ / C G , U o = U ∗ − ( m ^ h ( γ ^ ⊤ W ; G o , γ ^ ) − m [ 0 ] ( W , γ o ⊤ W ) ) U^{o}=U^{*}-\left(\widehat{m}_{h}(\widehat{\gamma}^{\top}W;\mathcal{G}^{o},\widehat{\gamma})-m^{[0]}(W,\gamma^{o\top}W)\right) U o = U ∗ − ( m h ( γ ⊤ W ; G o , γ ) − m [ 0 ] ( W , γ o ⊤ W ) ) . Under assumption A2, the law of large number implies ∑ i = 1 n ∣ U i ∗ ∣ / n → p E [ ∣ U ∗ ∣ ] \sum_{i=1}^{n}|U_{i}^{*}|/n\xrightarrow{p}\mathbb{E}[|U^{*}|] ∑ i = 1 n ∣ U i ∗ ∣/ n p E [ ∣ U ∗ ∣ ] . Combining this result with (G.12) yields
∣ I 3 ( k ) ∣ = o p ( log n n 4 / 5 ) + O p ( ∥ γ ^ − γ o ∥ ) if G o ∉ C G . (118) |I_3(k)|=o_p\left(\frac{\log n}{n^{4/5}}\right)+O_p(\|\widehat{\gamma}-\gamma^{o}\|)\ \text{if } \mathcal{G}^{o}\notin\mathcal{C}_{\mathcal{G}}. \tag*{(118)} ∣ I 3 ( k ) ∣ = o p ( n 4/5 log n ) + O p ( ∥ γ − γ o ∥ ) if G o ∈ / C G . (118)
Using ∑ i = 1 n ( Y − Y ‾ ) 2 / n → p var ( Y ) \sum_{i=1}^{n}(Y-\overline{Y})^{2}/n\xrightarrow{p}\operatorname{var}(Y) ∑ i = 1 n ( Y − Y ) 2 / n p var ( Y ) and substituting (G.6), (G.14), (G.18), and (G.19) into (G.5), we obtain
SPIC 0 ( k ) − SPIC 0 ( k ∗ ) = { 1 var ( Y ) max { b 2 ( k ) , ( k − k ∗ ) log n n 4 / 5 } ( 1 + o p ( 1 ) ) k > k ∗ , b 2 ( k ) var ( Y ) ( 1 + o p ( 1 ) ) k < k ∗ . (119) \operatorname{SPIC}_{0}(k)-\operatorname{SPIC}_{0}(k^{*})=
\begin{cases}
\frac{1}{\operatorname{var}(Y)}\max\left\{b^{2}(k),(k-k^{*})\frac{\log n}{n^{4/5}}\right\}(1+o_p(1)) & k>k^{*},\\[6pt]
\frac{b^{2}(k)}{\operatorname{var}(Y)}(1+o_p(1)) & k<k^{*}.
\end{cases} \tag*{(119)} SPIC 0 ( k ) − SPIC 0 ( k ∗ ) = ⎩ ⎨ ⎧ var ( Y ) 1 max { b 2 ( k ) , ( k − k ∗ ) n 4/5 l o g n } ( 1 + o p ( 1 )) var ( Y ) b 2 ( k ) ( 1 + o p ( 1 )) k > k ∗ , k < k ∗ . (119)
The consistency of k ^ ∗ \widehat{k}^{*} k ∗ to k ∗ k^{*} k ∗ follows directly from (G.20).
Proof of Theorem 5
By assumption C3, the triangle inequality, Taylor’s theorem, and Lemma S4 with t = 0 t=0 t = 0 ,
sup γ ∗ ∈ Γ ∣ 1 n p ℓ ( γ ) − 1 n ∑ i = 1 n log ( f 0 ( U γ i ; γ ) ) ∣ ≤ sup γ ∗ ∈ Γ ∣ 1 n p ℓ ( γ ) − 1 n ∑ i = 1 n log ( f ~ ξ ( U ^ γ i ; γ ) ) ∣ + sup γ ∗ ∈ Γ ∣ 1 n ∑ i = 1 n [ log ( f ~ ξ ( U ^ γ i ; γ ) ) − log ( f 0 ( U γ i ; γ ) ) ] ∣ ≤ sup γ ∗ ∈ Γ ∣ 1 n p ℓ ( γ ) − 1 n ∑ i = 1 n log ( f ~ ξ ( U ^ γ i ; γ ) ) ∣ + o p ( 1 ) f 3 + o p ( 1 ) . (120) \begin{aligned}
\sup_{\gamma^{*}\in\Gamma}\left|\frac{1}{n}p\ell(\gamma)-\frac{1}{n}\sum_{i=1}^{n}\log\left(f_{0}(U_{\gamma i};\gamma)\right)\right| \\
&\leq\sup_{\gamma^{*}\in\Gamma}\left|\frac{1}{n}p\ell(\gamma)-\frac{1}{n}\sum_{i=1}^{n}\log\left(\widetilde{f}_{\xi}(\widehat{U}_{\gamma i};\gamma)\right)\right|+\sup_{\gamma^{*}\in\Gamma}\left|\frac{1}{n}\sum_{i=1}^{n}\left[\log\left(\widetilde{f}_{\xi}(\widehat{U}_{\gamma i};\gamma)\right)-\log\left(f_{0}(U_{\gamma i};\gamma)\right)\right]\right| \\
&\leq\sup_{\gamma^{*}\in\Gamma}\left|\frac{1}{n}p\ell(\gamma)-\frac{1}{n}\sum_{i=1}^{n}\log\left(\widetilde{f}_{\xi}(\widehat{U}_{\gamma i};\gamma)\right)\right|+\frac{o_p(1)}{f_{3}+o_p(1)}. \tag*{(120)}
\end{aligned} γ ∗ ∈ Γ sup n 1 p ℓ ( γ ) − n 1 i = 1 ∑ n log ( f 0 ( U γ i ; γ ) ) ≤ γ ∗ ∈ Γ sup n 1 p ℓ ( γ ) − n 1 i = 1 ∑ n log ( f ξ ( U γ i ; γ ) ) + γ ∗ ∈ Γ sup n 1 i = 1 ∑ n [ log ( f ξ ( U γ i ; γ ) ) − log ( f 0 ( U γ i ; γ ) ) ] ≤ γ ∗ ∈ Γ sup n 1 p ℓ ( γ ) − n 1 i = 1 ∑ n log ( f ξ ( U γ i ; γ ) ) + f 3 + o p ( 1 ) o p ( 1 ) . (120)
Using arguments analogous to those in (G.2) and (G.3), it follows that under assumptions A1–A3 and A5,
sup { i , γ ∗ } ∣ ∑ c = 1 k I ( i ∈ G ^ c ) ∂ γ ∗ t m ^ h ( γ c ⊤ Z i ; G ^ , γ ) − ∑ c = 1 k I ( i ∈ G c o ) ∂ γ ∗ t m ^ h ( γ c ⊤ Z i ; G o , γ ) ∣ = o p ( 1 n 2 ) , t = 0 , 1 , 2. (121) \sup_{\{i,\gamma^{*}\}}\left|\sum_{c=1}^{k}I(i\in\widehat{\mathcal{G}}_{c})\partial_{\gamma^{*}}^{t}\widehat{m}_{h}(\gamma_{c}^{\top}Z_{i};\widehat{\mathcal{G}},\gamma)-\sum_{c=1}^{k}I(i\in\mathcal{G}_{c}^{o})\partial_{\gamma^{*}}^{t}\widehat{m}_{h}(\gamma_{c}^{\top}Z_{i};\mathcal{G}^{o},\gamma)\right|=o_p\left(\frac{1}{n^{2}}\right),\quad t=0,1,2. \tag*{(121)} { i , γ ∗ } sup c = 1 ∑ k I ( i ∈ G c ) ∂ γ ∗ t m h ( γ c ⊤ Z i ; G , γ ) − c = 1 ∑ k I ( i ∈ G c o ) ∂ γ ∗ t m h ( γ c ⊤ Z i ; G o , γ ) = o p ( n 2 1 ) , t = 0 , 1 , 2. (121)
Combining (H.2) with the Lipschitz continuity of the kernel function and assumptions A1 and C1, we obtain
sup { i , γ ∗ } ∣ ∂ γ ∗ f ~ ξ ( U ~ γ i ; γ ) − ∂ γ ∗ f ~ ξ ( U ^ γ i ; γ ) ∣ ≤ 1 ξ sup { i , j , γ ∗ } ∣ ∂ γ ∗ K ( U ~ γ j − U ~ γ i ξ ) − ∂ γ ∗ K ( U ^ γ j − U ^ γ i ξ ) ∣ = o p ( 1 n 2 ξ t + 2 ) = o p ( 1 ) , t = 0 , 1 , 2. (122) \begin{aligned}
\sup_{\{i,\gamma^{*}\}}\left|\partial_{\gamma^{*}}\widetilde{f}_{\xi}(\widetilde{U}_{\gamma i};\gamma)-\partial_{\gamma^{*}}\widetilde{f}_{\xi}(\widehat{U}_{\gamma i};\gamma)\right|
\leq\frac{1}{\xi}\sup_{\{i,j,\gamma^{*}\}}\left|\partial_{\gamma^{*}}K\left(\frac{\widetilde{U}_{\gamma j}-\widetilde{U}_{\gamma i}}{\xi}\right)-\partial_{\gamma^{*}}K\left(\frac{\widehat{U}_{\gamma j}-\widehat{U}_{\gamma i}}{\xi}\right)\right| \\
&= o_p\left(\frac{1}{n^{2}\xi^{t+2}}\right)=o_p(1),\quad t=0,1,2. \tag*{(122)}
\end{aligned} { i , γ ∗ } sup ∂ γ ∗ f ξ ( U γ i ; γ ) − ∂ γ ∗ f ξ ( U γ i ; γ ) ≤ ξ 1 { i , j , γ ∗ } sup ∂ γ ∗ K ( ξ U γ j − U γ i ) − ∂ γ ∗ K ( ξ U γ j − U γ i ) = o p ( n 2 ξ t + 2 1 ) = o p ( 1 ) , t = 0 , 1 , 2. (122)
where U ~ γ i = Y i − ∑ c = 1 k I ( i ∈ G ^ c ) m ^ h ( γ c ⊤ Z i ; G ^ , γ ) \widetilde{U}_{\gamma i}=Y_i-\sum_{c=1}^{k}I(i\in\widehat{\mathcal{G}}_{c})\widehat{m}_{h}(\gamma_{c}^{\top}Z_i;\widehat{\mathcal{G}},\gamma) U γ i = Y i − ∑ c = 1 k I ( i ∈ G c ) m h ( γ c ⊤ Z i ; G , γ ) , i = 1 , … , n i=1,\ldots,n i = 1 , … , n . Hence,
sup { γ ∗ ∈ Γ } ∣ 1 n p ℓ ( γ ) − 1 n ∑ i = 1 n log ( f ~ ξ ( U ^ γ i ; γ ) ) ∣ = o p ( 1 ) . (123) \sup_{\{\gamma^{*}\in\Gamma\}}\left|\frac{1}{n}p\ell(\gamma)-\frac{1}{n}\sum_{i=1}^{n}\log\left(\widetilde{f}_{\xi}(\widehat{U}_{\gamma i};\gamma)\right)\right|=o_p(1). \tag*{(123)} { γ ∗ ∈ Γ } sup n 1 p ℓ ( γ ) − n 1 i = 1 ∑ n log ( f ξ ( U γ i ; γ ) ) = o p ( 1 ) . (123)
Substituting (H.4) into (H.1) yields
sup { γ ∗ ∈ Γ } ∣ 1 n p ℓ ( γ ) − 1 n ∑ i = 1 n log ( f 0 ( U γ i ; γ ) ) ∣ = o p ( 1 ) . (124) \sup_{\{\gamma^{*}\in\Gamma\}}\left|\frac{1}{n}p\ell(\gamma)-\frac{1}{n}\sum_{i=1}^{n}\log\left(f_{0}(U_{\gamma i};\gamma)\right)\right|=o_p(1). \tag*{(124)} { γ ∗ ∈ Γ } sup n 1 p ℓ ( γ ) − n 1 i = 1 ∑ n log ( f 0 ( U γ i ; γ ) ) = o p ( 1 ) . (124)
Since f 0 ( u ; γ ) f_{0}(u;\gamma) f 0 ( u ; γ ) is of bounded variation on U n × Γ \mathcal{U}_{n}\times\Gamma U n × Γ by assumption C2, Lemma 22 in Nolan and Pollard [21 ] implies that the class
{ f 0 ( u ; γ ) : u ∈ U γ n , γ ∗ ∈ Γ } \left\{f_0(u;\gamma):u\in\mathcal{U}_{\gamma_n},\gamma^*\in\Gamma\right\} { f 0 ( u ; γ ) : u ∈ U γ n , γ ∗ ∈ Γ }
is Euclidean. It follows from Corollary 4 in Sherman [28 ] that
sup { γ ∗ ∈ Γ } ∣ 1 n ∑ i = 1 n log ( f 0 ( U γ i ; γ ) ) − E [ log ( f 0 ( U γ i ; γ ) ) ] ∣ = O p ( 1 n ) . (125) \sup_{\{\gamma^*\in\Gamma\}}\left|\frac{1}{n}\sum_{i=1}^{n}\log\left(f_0(U_{\gamma_i};\gamma)\right)-\mathbb{E}\left[\log\left(f_0(U_{\gamma_i};\gamma)\right)\right]\right|=O_p\left(\frac{1}{\sqrt{n}}\right). \tag*{(125)} { γ ∗ ∈ Γ } sup n 1 i = 1 ∑ n log ( f 0 ( U γ i ; γ ) ) − E [ log ( f 0 ( U γ i ; γ ) ) ] = O p ( n 1 ) . (125)
Combining (H.5) and (H.6) yields
sup { γ ∗ ∈ Γ } ∣ 1 n p ℓ ( γ ) − E [ log ( f 0 ( U γ ; γ ) ) ] ∣ = o p ( 1 ) . (126) \sup_{\{\gamma^*\in\Gamma\}}\left|\frac{1}{n}p\ell(\gamma)-\mathbb{E}\left[\log\left(f_0(U_{\gamma};\gamma)\right)\right]\right|=o_p(1). \tag*{(126)} { γ ∗ ∈ Γ } sup n 1 p ℓ ( γ ) − E [ log ( f 0 ( U γ ; γ ) ) ] = o p ( 1 ) . (126)
Let f W ( w ) f_W(w) f W ( w ) denote the probability density function of W W W . By Jensen’s inequality and a change of variables,
E [ log ( f 0 ( U γ ; γ ) ) ] − E [ log ( f 0 ( U ∘ ; γ ∘ ) ) ] ≤ log E [ f 0 ( U γ ; γ ) f 0 ( U ∘ ; γ ∘ ) ] = log ∫ Y ∫ W f 0 ( y − m [ 0 ] ( w , γ ⊤ w ) ; γ ) f W ( w ) d w d y ≤ log ∫ − ∞ ∞ f 0 ( w ; γ ) d u ∫ W f W ( w ) d w = log 1 = 0 , (127) \begin{aligned}
\mathbb{E}\left[\log\left(f_0(U_{\gamma};\gamma)\right)\right]-\mathbb{E}\left[\log\left(f_0(U^{\circ};\gamma^\circ)\right)\right] \\
&\leq\log\mathbb{E}\left[\frac{f_0(U_{\gamma};\gamma)}{f_0(U^{\circ};\gamma^\circ)}\right]=\log\int_{\mathcal{Y}}\int_{\mathcal{W}}f_0\left(y-m^{[0]}(w,\gamma^\top w);\gamma\right)f_W(w)\,dw\,dy \\
&\leq\log\int_{-\infty}^{\infty}f_0(w;\gamma)\,du\int_{\mathcal{W}}f_W(w)\,dw=\log1=0, \tag*{(127)}
\end{aligned} E [ log ( f 0 ( U γ ; γ ) ) ] − E [ log ( f 0 ( U ∘ ; γ ∘ ) ) ] ≤ log E [ f 0 ( U ∘ ; γ ∘ ) f 0 ( U γ ; γ ) ] = log ∫ Y ∫ W f 0 ( y − m [ 0 ] ( w , γ ⊤ w ) ; γ ) f W ( w ) d w d y ≤ log ∫ − ∞ ∞ f 0 ( w ; γ ) d u ∫ W f W ( w ) d w = log 1 = 0 , (127)
with strict inequality for γ ≠ γ ∘ \gamma\ne\gamma^\circ γ = γ ∘ . Hence, under assumption C4, γ ∘ \gamma^\circ γ ∘ uniquely maximizes E [ log ( f 0 ( U γ ; γ ) ) ] \mathbb{E}[\log(f_0(U_{\gamma};\gamma))] E [ log ( f 0 ( U γ ; γ ))] . Combining (H.7) and (H.8) yields p ℓ ( γ ~ ) / n − p ℓ ( γ ∘ ) / n = E [ log ( f 0 ( U γ ~ ; γ ~ ) ) ] − E [ log ( f 0 ( U ∘ ; γ ∘ ) ) ] + o p ( 1 ) p\ell(\widetilde{\gamma})/n-p\ell(\gamma^\circ)/n=\mathbb{E}[\log(f_0(U_{\widetilde{\gamma}};\widetilde{\gamma}))]-\mathbb{E}[\log(f_0(U^\circ;\gamma^\circ))]+o_p(1) p ℓ ( γ ) / n − p ℓ ( γ ∘ ) / n = E [ log ( f 0 ( U γ ; γ ))] − E [ log ( f 0 ( U ∘ ; γ ∘ ))] + o p ( 1 ) , where E [ log ( f 0 ( U γ ~ ; γ ~ ) ) ] − E [ log ( f 0 ( U ∘ ; γ ∘ ) ) ] < 0 \mathbb{E}[\log(f_0(U_{\widetilde{\gamma}};\widetilde{\gamma}))]-\mathbb{E}[\log(f_0(U^\circ;\gamma^\circ))]<0 E [ log ( f 0 ( U γ ; γ ))] − E [ log ( f 0 ( U ∘ ; γ ∘ ))] < 0 . On the other hand, by the definition of γ ~ \widetilde{\gamma} γ , we have p ℓ ( γ ~ ) ≥ p ℓ ( γ ∘ ) p\ell(\widetilde{\gamma})\geq p\ell(\gamma^\circ) p ℓ ( γ ) ≥ p ℓ ( γ ∘ ) . Therefore, E [ log ( f 0 ( U γ ~ ; γ ~ ) ) ] − E [ log ( f 0 ( U ∘ ; γ ∘ ) ) ] = o p ( 1 ) \mathbb{E}[\log(f_0(U_{\widetilde{\gamma}};\widetilde{\gamma}))]-\mathbb{E}[\log(f_0(U^\circ;\gamma^\circ))]=o_p(1) E [ log ( f 0 ( U γ ; γ ))] − E [ log ( f 0 ( U ∘ ; γ ∘ ))] = o p ( 1 ) . The consistency of γ ~ \widetilde{\gamma} γ to γ ∘ \gamma^\circ γ ∘ then follows from the uniqueness of γ ∘ \gamma^\circ γ ∘ as the maximizer of E [ log ( f 0 ( U γ ; γ ) ) ] \mathbb{E}[\log(f_0(U_{\gamma};\gamma))] E [ log ( f 0 ( U γ ; γ ))] over Γ \Gamma Γ and the continuity of E [ log ( f 0 ( U γ ; γ ) ) ] \mathbb{E}[\log(f_0(U_{\gamma};\gamma))] E [ log ( f 0 ( U γ ; γ ))] on Γ \Gamma Γ .
Define p S ( γ ) = ∂ γ ∗ p ℓ ( γ ) pS(\gamma)=\partial_{\gamma^*}p\ell(\gamma) pS ( γ ) = ∂ γ ∗ p ℓ ( γ ) and p I ( γ ) = − ∂ γ ∗ 2 p ℓ ( γ ) pI(\gamma)=-\partial_{\gamma^*}^2p\ell(\gamma) p I ( γ ) = − ∂ γ ∗ 2 p ℓ ( γ ) . For an arbitrary fixed vector v ∈ R k ( p + 1 ) − 2 v\in\mathbb{R}^{k(p+1)-2} v ∈ R k ( p + 1 ) − 2 , a first-order Taylor expansion of v ⊤ p S ( γ ~ ) / n v^\top pS(\widetilde{\gamma})/\sqrt{n} v ⊤ pS ( γ ) / n around γ ∘ \gamma^\circ γ ∘ yields
1 n v ⊤ p S ( γ ~ ) = 1 n v ⊤ p S ( γ ∘ ) − 1 n v ⊤ p I ( γ ~ ) ( γ ~ − γ ∘ ) , (128) \frac{1}{\sqrt{n}}v^\top pS(\widetilde{\gamma})=\frac{1}{\sqrt{n}}v^\top pS(\gamma^\circ)-\frac{1}{\sqrt{n}}v^\top pI(\widetilde{\gamma})(\widetilde{\gamma}-\gamma^\circ), \tag*{(128)} n 1 v ⊤ pS ( γ ) = n 1 v ⊤ pS ( γ ∘ ) − n 1 v ⊤ p I ( γ ) ( γ − γ ∘ ) , (128)
where γ ~ \widetilde{\gamma} γ lies on the line segment between γ ~ \widetilde{\gamma} γ and γ ∘ \gamma^\circ γ ∘ . By straightforward algebra, p S ( γ ∘ ) / n pS(\gamma^\circ)/\sqrt{n} pS ( γ ∘ ) / n in (H.9) is rewritten as
1 n p S ( γ ∘ ) = 1 n ∑ i = 1 n f [ 1 ] ( W i , U i ∘ ; γ ∘ ) f 0 ( U i ∘ ; γ ∘ ) + ( 1 n p S ( γ ∘ ) − 1 n ∑ i = 1 n ∂ γ ∗ f ~ ξ ( U ~ γ ∘ i ; γ ∘ ) f ~ ξ ( U ~ γ ∘ i ; γ ∘ ) ) + 1 n ∑ i = 1 n ∑ t = 0 1 ( − 1 ) 1 − t f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) ( ∂ γ ∗ t f ~ ξ ( U ^ γ ∘ i ; γ ∘ ) − f [ t ] ( W i , U i ∘ ; γ ∘ ) ) f 0 ( U i ∘ ; γ ∘ ) f ~ ξ ( U ^ γ ∘ i ; γ ∘ ) \begin{aligned}
\frac{1}{\sqrt{n}}pS(\gamma^\circ)=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\frac{f^{[1]}(W_i,U_i^\circ;\gamma^\circ)}{f_0(U_i^\circ;\gamma^\circ)}+\left(\frac{1}{\sqrt{n}}pS(\gamma^\circ)-\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\frac{\partial_{\gamma^*}\widetilde{f}_\xi(\widetilde{U}_{\gamma^\circ i};\gamma^\circ)}{\widetilde{f}_\xi(\widetilde{U}_{\gamma^\circ i};\gamma^\circ)}\right) \\
&+\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\sum_{t=0}^{1}\frac{(-1)^{1-t}f^{[1-t]}(W_i,U_i^\circ;\gamma^\circ)\left(\partial_{\gamma^*}^{t}\widetilde{f}_\xi(\widehat{U}_{\gamma^\circ i};\gamma^\circ)-f^{[t]}(W_i,U_i^\circ;\gamma^\circ)\right)}{f_0(U_i^\circ;\gamma^\circ)\widetilde{f}_\xi(\widehat{U}_{\gamma^\circ i};\gamma^\circ)}
\end{aligned} n 1 pS ( γ ∘ ) = n 1 i = 1 ∑ n f 0 ( U i ∘ ; γ ∘ ) f [ 1 ] ( W i , U i ∘ ; γ ∘ ) + ( n 1 pS ( γ ∘ ) − n 1 i = 1 ∑ n f ξ ( U γ ∘ i ; γ ∘ ) ∂ γ ∗ f ξ ( U γ ∘ i ; γ ∘ ) ) + n 1 i = 1 ∑ n t = 0 ∑ 1 f 0 ( U i ∘ ; γ ∘ ) f ξ ( U γ ∘ i ; γ ∘ ) ( − 1 ) 1 − t f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) ( ∂ γ ∗ t f ξ ( U γ ∘ i ; γ ∘ ) − f [ t ] ( W i , U i ∘ ; γ ∘ ) )
≜ 1 n S ( γ ∘ ) + I I + I I I . (129) \triangleq\frac{1}{\sqrt{n}}S(\gamma^\circ)+II+III. \tag*{(129)} ≜ n 1 S ( γ ∘ ) + I I + I I I . (129)
By the triangle inequality, (H.3), and Lemma S4 with t = 0 , 1 t=0,1 t = 0 , 1 , it follows that under assumptions A1, C1, and C3,
∣ I I ∣ ≤ ∑ t = 0 1 sup i ∣ ∂ γ ∗ t f ~ ξ ( U ^ γ ∘ i ; γ ∘ ) ∣ sup i ∣ ∂ γ ∗ 1 − t f ~ ξ ( U ~ γ ∘ i ; γ ∘ ) − ∂ γ ∗ 1 − t f ~ ξ ( U ^ γ ∘ i ; γ ∘ ) ∣ inf i ∣ f ~ ξ ( U ~ γ ∘ i ; γ ∘ ) ∣ inf i ∣ f ~ ξ ( U ~ γ ∘ i ; γ ∘ ) ∣ ≤ ∑ t = 0 1 ( sup i ∣ f [ t ] ( W i , U i ∘ ; γ ∘ ) ∣ + o p ( 1 ) ) o p ( 1 ) ( f 3 + o p ( 1 ) ) 2 = o p ( 1 ) and (130) \begin{aligned}
|II|\leq\frac{\sum_{t=0}^{1}\sup_i\left|\partial_{\gamma^*}^{t}\widetilde{f}_\xi(\widehat{U}_{\gamma^\circ i};\gamma^\circ)\right|\sup_i\left|\partial_{\gamma^*}^{1-t}\widetilde{f}_\xi(\widetilde{U}_{\gamma^\circ i};\gamma^\circ)-\partial_{\gamma^*}^{1-t}\widetilde{f}_\xi(\widehat{U}_{\gamma^\circ i};\gamma^\circ)\right|}{\inf_i\left|\widetilde{f}_\xi(\widetilde{U}_{\gamma^\circ i};\gamma^\circ)\right|\inf_i\left|\widetilde{f}_\xi(\widetilde{U}_{\gamma^\circ i};\gamma^\circ)\right|} \\
&\leq\frac{\sum_{t=0}^{1}\left(\sup_i\left|f^{[t]}(W_i,U_i^\circ;\gamma^\circ)\right|+o_p(1)\right)o_p(1)}{(f_3+o_p(1))^2}=o_p(1)\text{ and} \tag*{(130)}
\end{aligned} ∣ I I ∣ ≤ inf i f ξ ( U γ ∘ i ; γ ∘ ) inf i f ξ ( U γ ∘ i ; γ ∘ ) ∑ t = 0 1 sup i ∂ γ ∗ t f ξ ( U γ ∘ i ; γ ∘ ) sup i ∂ γ ∗ 1 − t f ξ ( U γ ∘ i ; γ ∘ ) − ∂ γ ∗ 1 − t f ξ ( U γ ∘ i ; γ ∘ ) ≤ ( f 3 + o p ( 1 ) ) 2 ∑ t = 0 1 ( sup i f [ t ] ( W i , U i ∘ ; γ ∘ ) + o p ( 1 ) ) o p ( 1 ) = o p ( 1 ) and (130)
I I I = 1 n ∑ i = 1 n ∑ t = 0 1 ( − 1 ) 1 − t f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) ( ∂ γ ∗ t f ~ ξ ( U ^ γ ∘ i ; γ ∘ ) − f [ t ] ( W i , U i ∘ ; γ ∘ ) ) f 0 ( U i ∘ ; γ ∘ ) f ~ ξ ( U ^ γ ∘ i ; γ ∘ ) . (131) III=\frac{1}{\sqrt{n}}\sum_{i=1}^{n}\sum_{t=0}^{1}\frac{(-1)^{1-t}f^{[1-t]}(W_i,U_i^\circ;\gamma^\circ)\left(\partial_{\gamma^*}^{t}\widetilde{f}_\xi(\widehat{U}_{\gamma^\circ i};\gamma^\circ)-f^{[t]}(W_i,U_i^\circ;\gamma^\circ)\right)}{f_0(U_i^\circ;\gamma^\circ)\widetilde{f}_\xi(\widehat{U}_{\gamma^\circ i};\gamma^\circ)}. \tag*{(131)} I I I = n 1 i = 1 ∑ n t = 0 ∑ 1 f 0 ( U i ∘ ; γ ∘ ) f ξ ( U γ ∘ i ; γ ∘ ) ( − 1 ) 1 − t f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) ( ∂ γ ∗ t f ξ ( U γ ∘ i ; γ ∘ ) − f [ t ] ( W i , U i ∘ ; γ ∘ ) ) . (131)
Moreover, under assumptions A1 and C1,
I I I = n ∑ t = 0 1 ( − 1 ) 1 − t ( 1 n ( n − 1 ) ∑ i ≠ j f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) ( f 0 ( U i ∘ ; γ ∘ ) ) 2 φ j [ t ] ( W i , U i ∘ ; γ ∘ ) ) = n ∑ t = 0 1 ( − 1 ) 1 − t ( 1 n ( n − 1 ) ∑ i ≠ j U i j [ t ] + 1 n ∑ i = 1 n U ˉ i [ t ] ) , \begin{aligned}
III=\sqrt{n}\sum_{t=0}^{1}(-1)^{1-t}\left(\frac{1}{n(n-1)}\sum_{i\ne j}\frac{f^{[1-t]}(W_i,U_i^{\circ};\gamma^{\circ})}{(f_0(U_i^{\circ};\gamma^{\circ}))^2}\varphi_j^{[t]}(W_i,U_i^{\circ};\gamma^{\circ})\right) \\
&=\sqrt{n}\sum_{t=0}^{1}(-1)^{1-t}\left(\frac{1}{n(n-1)}\sum_{i\ne j}U_{ij}^{[t]}+\frac{1}{n}\sum_{i=1}^{n}\bar{U}_i^{[t]}\right),
\end{aligned} I I I = n t = 0 ∑ 1 ( − 1 ) 1 − t n ( n − 1 ) 1 i = j ∑ ( f 0 ( U i ∘ ; γ ∘ ) ) 2 f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) φ j [ t ] ( W i , U i ∘ ; γ ∘ ) = n t = 0 ∑ 1 ( − 1 ) 1 − t n ( n − 1 ) 1 i = j ∑ U ij [ t ] + n 1 i = 1 ∑ n U ˉ i [ t ] ,
where
U i j [ t ] = ( f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) ( f 0 ( U i ∘ ; γ ∘ ) ) 2 φ j [ t ] ( W i , U i ∘ ; γ ∘ ) ) − U ˉ i j [ t ] , U ˉ i j [ t ] = f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) ( f 0 ( U i ∘ ; γ ∘ ) ) 2 E [ φ j [ t ] ( W i , U i ∘ ; γ ∘ ) ∣ i ] . U_{ij}^{[t]}=\left(\frac{f^{[1-t]}(W_i,U_i^{\circ};\gamma^{\circ})}{(f_0(U_i^{\circ};\gamma^{\circ}))^2}\varphi_j^{[t]}(W_i,U_i^{\circ};\gamma^{\circ})\right)-\bar{U}_{ij}^{[t]},\quad\bar{U}_{ij}^{[t]}=\frac{f^{[1-t]}(W_i,U_i^{\circ};\gamma^{\circ})}{(f_0(U_i^{\circ};\gamma^{\circ}))^2}\mathbb{E}[\varphi_j^{[t]}(W_i,U_i^{\circ};\gamma^{\circ})\mid i]. U ij [ t ] = ( ( f 0 ( U i ∘ ; γ ∘ ) ) 2 f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) φ j [ t ] ( W i , U i ∘ ; γ ∘ ) ) − U ˉ ij [ t ] , U ˉ ij [ t ] = ( f 0 ( U i ∘ ; γ ∘ ) ) 2 f [ 1 − t ] ( W i , U i ∘ ; γ ∘ ) E [ φ j [ t ] ( W i , U i ∘ ; γ ∘ ) ∣ i ] .
The definitions of f [ 1 ] ( w , u ; γ ) f^{[1]}(w,u;\gamma) f [ 1 ] ( w , u ; γ ) in (D.11), ξ j [ 1 ] ( w , γ ) \xi_j^{[1]}(w,\gamma) ξ j [ 1 ] ( w , γ ) in (D.5), and φ j [ 1 ] ( w , u ; γ ) \varphi_j^{[1]}(w,u;\gamma) φ j [ 1 ] ( w , u ; γ ) in (D.28) yield
E [ f [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ U i ∘ ] = E [ − ∂ u ( Q 1 , 1 ( W i , U i ∘ ) f ( U i ∘ ; γ ∘ ) ) ∣ U i ∘ ] = E [ ( − E [ m [ 1 ] ( W , γ ∘ ⊤ W ) ] + m [ 1 ] ( W i , γ ∘ ⊤ W i ) ) ( − ∂ u f ( U i ∘ ; γ ∘ ) ) ∣ U i ∘ ] = ( − E [ m [ 1 ] ( W , γ ∘ ⊤ W ) ] + E [ m [ 1 ] ( W , γ ∘ ⊤ W ) ] ) ( − ∂ u f ( U i ∘ ; γ ∘ ) ) = 0 , \begin{aligned}
\mathbb{E}[f^{[1]}(W_i,U_i^{\circ};\gamma^{\circ})\mid U_i^{\circ}]=\mathbb{E}\left[-\partial_u(Q_{1,1}(W_i,U_i^{\circ})f(U_i^{\circ};\gamma^{\circ}))\mid U_i^{\circ}\right] \\
&=\mathbb{E}\left[\left(-\mathbb{E}[m^{[1]}(W,\gamma^{\circ\top}W)]+m^{[1]}(W_i,\gamma^{\circ\top}W_i)\right)(-\partial_uf(U_i^{\circ};\gamma^{\circ}))\mid U_i^{\circ}\right] \\
&=\left(-\mathbb{E}[m^{[1]}(W,\gamma^{\circ\top}W)]+\mathbb{E}[m^{[1]}(W,\gamma^{\circ\top}W)]\right)(-\partial_uf(U_i^{\circ};\gamma^{\circ}))=0,
\end{aligned} E [ f [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ U i ∘ ] = E [ − ∂ u ( Q 1 , 1 ( W i , U i ∘ ) f ( U i ∘ ; γ ∘ )) ∣ U i ∘ ] = E [ ( − E [ m [ 1 ] ( W , γ ∘ ⊤ W )] + m [ 1 ] ( W i , γ ∘ ⊤ W i ) ) ( − ∂ u f ( U i ∘ ; γ ∘ )) ∣ U i ∘ ] = ( − E [ m [ 1 ] ( W , γ ∘ ⊤ W )] + E [ m [ 1 ] ( W , γ ∘ ⊤ W )] ) ( − ∂ u f ( U i ∘ ; γ ∘ )) = 0 ,
ξ j [ 1 ] ( W i , γ ) − E [ ξ j [ 1 ] ( W i , γ ) ∣ i ] = a ( W j , W i ) W i , and \xi_j^{[1]}(W_i,\gamma)-\mathbb{E}[\xi_j^{[1]}(W_i,\gamma)\mid i]=a(W_j,W_i)W_i,\quad\text{and} ξ j [ 1 ] ( W i , γ ) − E [ ξ j [ 1 ] ( W i , γ ) ∣ i ] = a ( W j , W i ) W i , and
E [ ξ j [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ W j , U j ∘ , U i ∘ ] = E [ ( ∂ γ ∗ K ξ ( U j ∘ − U i ∘ ) − f [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ W j , U j ∘ , U i ∘ ] = 1 ξ 2 E [ K ( 1 ) ( U j ∘ − U i ∘ ξ ) Q 1 , 1 ( W i , U j ∘ ) ∣ W j , U j ∘ , U i ∘ ] − E [ f [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ U i ∘ ] = 1 ξ 2 K ( 1 ) ( U j ∘ − U i ∘ ξ ) ( − E [ m [ 1 ] ( W , γ ∘ ⊤ W ) ] + E [ m [ 1 ] ( W , γ ∘ ⊤ W ) ] ) = 0 , \begin{aligned}
\mathbb{E}[\xi_j^{[1]}(W_i,U_i^{\circ};\gamma^{\circ})\mid W_j,U_j^{\circ},U_i^{\circ}]
=\mathbb{E}\left[(\partial_{\gamma_*}K_{\xi}(U_j^{\circ}-U_i^{\circ})-f^{[1]}(W_i,U_i^{\circ};\gamma^{\circ})\mid W_j,U_j^{\circ},U_i^{\circ}\right] \\
&=\frac{1}{\xi^2}\mathbb{E}\left[K^{(1)}\left(\frac{U_j^{\circ}-U_i^{\circ}}{\xi}\right)Q_{1,1}(W_i,U_j^{\circ})\mid W_j,U_j^{\circ},U_i^{\circ}\right]-\mathbb{E}[f^{[1]}(W_i,U_i^{\circ};\gamma^{\circ})\mid U_i^{\circ}] \\
&=\frac{1}{\xi^2}K^{(1)}\left(\frac{U_j^{\circ}-U_i^{\circ}}{\xi}\right)\left(-\mathbb{E}[m^{[1]}(W,\gamma^{\circ\top}W)]+\mathbb{E}[m^{[1]}(W,\gamma^{\circ\top}W)]\right)=0,
\end{aligned} E [ ξ j [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ W j , U j ∘ , U i ∘ ] = E [ ( ∂ γ ∗ K ξ ( U j ∘ − U i ∘ ) − f [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ W j , U j ∘ , U i ∘ ] = ξ 2 1 E [ K ( 1 ) ( ξ U j ∘ − U i ∘ ) Q 1 , 1 ( W i , U j ∘ ) ∣ W j , U j ∘ , U i ∘ ] − E [ f [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ U i ∘ ] = ξ 2 1 K ( 1 ) ( ξ U j ∘ − U i ∘ ) ( − E [ m [ 1 ] ( W , γ ∘ ⊤ W )] + E [ m [ 1 ] ( W , γ ∘ ⊤ W )] ) = 0 ,
where a ( W j , W i ) a(W_j,W_i) a ( W j , W i ) is a function of W j W_j W j and W i W_i W i . Hence,
E [ U i j [ 0 ] ∣ W j , U j ∘ ] = E [ φ j [ 0 ] ( W i , U i ∘ ; γ ∘ ) − E [ φ j [ 0 ] ( W i , U i ∘ ; γ ∘ ) ∣ i ] ( f 0 ( U i ∘ ; γ ∘ ) ) 2 E [ f [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ U i ∘ ] ∣ W j , U j ∘ ] = 0 \mathbb{E}[U_{ij}^{[0]}\mid W_j,U_j^{\circ}]=\mathbb{E}\left[\frac{\varphi_j^{[0]}(W_i,U_i^{\circ};\gamma^{\circ})-\mathbb{E}[\varphi_j^{[0]}(W_i,U_i^{\circ};\gamma^{\circ})\mid i]}{(f_0(U_i^{\circ};\gamma^{\circ}))^2}\mathbb{E}[f^{[1]}(W_i,U_i^{\circ};\gamma^{\circ})\mid U_i^{\circ}]\mid W_j,U_j^{\circ}\right]=0 E [ U ij [ 0 ] ∣ W j , U j ∘ ] = E [ ( f 0 ( U i ∘ ; γ ∘ ) ) 2 φ j [ 0 ] ( W i , U i ∘ ; γ ∘ ) − E [ φ j [ 0 ] ( W i , U i ∘ ; γ ∘ ) ∣ i ] E [ f [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ U i ∘ ] ∣ W j , U j ∘ ] = 0
and
E [ U i j [ 1 ] ∣ W j , U j ∘ ] = E [ φ j [ 0 ] ( W i , U i ∘ ; γ ∘ ) − E [ φ j [ 0 ] ( W i , U i ∘ ; γ ∘ ) ∣ i ] f 0 ( U i ∘ ; γ ∘ ) ∣ W j , U j ∘ ] = E [ ( ∂ u f 0 ( U i ∘ ; γ ∘ ) ) ( ξ j [ 1 ] ( W i , γ ) − E [ ξ j [ 1 ] ( W i , γ ) ∣ i ] ) f 0 ( U i ∘ ; γ ∘ ) ∣ W j , U j ∘ ] + E [ 1 f 0 ( U i ∘ ; γ ∘ ) ( E [ ξ j [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ W j , U j ∘ , U i ∘ ] − E [ ξ j [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ i ] ) ∣ W j , U j ∘ ] = ( ∫ ∂ γ ∗ f 0 ( u ; γ ∘ ) d u ) E [ a ( W j , w ) ∣ W j ] = ( ∂ γ ∗ ∫ f 0 ( u ; γ ∘ ) d u ) E [ a ( W j , w ) ∣ W j ] = 0. \begin{aligned}
\mathbb{E}[U_{ij}^{[1]}\mid W_j,U_j^{\circ}]=\mathbb{E}\left[\frac{\varphi_j^{[0]}(W_i,U_i^{\circ};\gamma^{\circ})-\mathbb{E}[\varphi_j^{[0]}(W_i,U_i^{\circ};\gamma^{\circ})\mid i]}{f_0(U_i^{\circ};\gamma^{\circ})}\mid W_j,U_j^{\circ}\right] \\
&=\mathbb{E}\left[\frac{(\partial_uf_0(U_i^{\circ};\gamma^{\circ}))(\xi_j^{[1]}(W_i,\gamma)-\mathbb{E}[\xi_j^{[1]}(W_i,\gamma)\mid i])}{f_0(U_i^{\circ};\gamma^{\circ})}\mid W_j,U_j^{\circ}\right] \\
&+\mathbb{E}\left[\frac{1}{f_0(U_i^{\circ};\gamma^{\circ})}\left(\mathbb{E}[\xi_j^{[1]}(W_i,U_i^{\circ};\gamma^{\circ})\mid W_j,U_j^{\circ},U_i^{\circ}]-\mathbb{E}[\xi_j^{[1]}(W_i,U_i^{\circ};\gamma^{\circ})\mid i]\right)\mid W_j,U_j^{\circ}\right] \\
&=\left(\int\partial_{\gamma_*}f_0(u;\gamma^{\circ})\,du\right)\mathbb{E}[a(W_j,w)\mid W_j]=\left(\partial_{\gamma_*}\int f_0(u;\gamma^{\circ})\,du\right)\mathbb{E}[a(W_j,w)\mid W_j]=0.
\end{aligned} E [ U ij [ 1 ] ∣ W j , U j ∘ ] = E [ f 0 ( U i ∘ ; γ ∘ ) φ j [ 0 ] ( W i , U i ∘ ; γ ∘ ) − E [ φ j [ 0 ] ( W i , U i ∘ ; γ ∘ ) ∣ i ] ∣ W j , U j ∘ ] = E [ f 0 ( U i ∘ ; γ ∘ ) ( ∂ u f 0 ( U i ∘ ; γ ∘ )) ( ξ j [ 1 ] ( W i , γ ) − E [ ξ j [ 1 ] ( W i , γ ) ∣ i ]) ∣ W j , U j ∘ ] + E [ f 0 ( U i ∘ ; γ ∘ ) 1 ( E [ ξ j [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ W j , U j ∘ , U i ∘ ] − E [ ξ j [ 1 ] ( W i , U i ∘ ; γ ∘ ) ∣ i ] ) ∣ W j , U j ∘ ] = ( ∫ ∂ γ ∗ f 0 ( u ; γ ∘ ) d u ) E [ a ( W j , w ) ∣ W j ] = ( ∂ γ ∗ ∫ f 0 ( u ; γ ∘ ) d u ) E [ a ( W j , w ) ∣ W j ] = 0.
Consequently, E [ U i j [ t ] ∣ j ] = 0 \mathbb{E}[U_{ij}^{[t]}\mid j]=0 E [ U ij [ t ] ∣ j ] = 0 . Combining with E [ U i j [ t ] ∣ i ] = 0 \mathbb{E}[U_{ij}^{[t]}\mid i]=0 E [ U ij [ t ] ∣ i ] = 0 , U i j [ t ] U_{ij}^{[t]} U ij [ t ] is a degenerate second-order U U U -statistic with variance of order O ( ξ m − 2 t − 1 ) O(\xi_m^{-2t-1}) O ( ξ m − 2 t − 1 ) for t = 0 , 1 t=0,1 t = 0 , 1 . By Theorem 8.1 in Hoeffding [8 ] ,
1 n ( n − 1 ) ∑ i ≠ j U i j [ t ] = O p ( 1 n ξ m 2 t + 1 ) , t = 0 , 1. (132) \frac{1}{n(n-1)}\sum_{i\ne j}U_{ij}^{[t]}=O_p\left(\frac{1}{n\sqrt{\xi_m^{2t+1}}}\right),\quad t=0,1. \tag*{(132)} n ( n − 1 ) 1 i = j ∑ U ij [ t ] = O p ( n ξ m 2 t + 1 1 ) , t = 0 , 1. (132)
The central limit theorem implies
1 n ∑ i = 1 n U ˉ i [ t ] = O p ( ξ M 2 n ) , t = 0 , 1. (133) \frac{1}{n}\sum_{i=1}^{n}\bar{U}_i^{[t]}=O_p\left(\frac{\xi_M^2}{\sqrt{n}}\right),\quad t=0,1. \tag*{(133)} n 1 i = 1 ∑ n U ˉ i [ t ] = O p ( n ξ M 2 ) , t = 0 , 1. (133)
Combining (H.13) and (H.14) yields
I I I = o p ( 1 ) . (134) III=o_p(1). \tag*{(134)} I I I = o p ( 1 ) . (134)
Substituting (H.11) and (H.15) into (H.10) and applying the central limit theorem to S ( γ ∘ ) / n S(\gamma^{\circ})/\sqrt{n} S ( γ ∘ ) / n yield
1 n p S ( γ o ) = 1 n S ( γ o ) + o p ( 1 ) → d N ( 0 , V ) . (135) \frac{1}{\sqrt{n}}pS(\gamma^{o})=\frac{1}{\sqrt{n}}S(\gamma^{o})+o_p(1)\xrightarrow{d}N(0,V). \tag*{(135)} n 1 pS ( γ o ) = n 1 S ( γ o ) + o p ( 1 ) d N ( 0 , V ) . (135)
Using arguments analogous to those for (H.11), we obtain
1 n p I ( γ ) = 1 n ∑ i = 1 n [ ( ∂ γ ∗ f ˉ ξ ( U ^ γ i ; γ ) f ˉ ξ ( U ^ γ i ; γ ) ) ⊗ 2 − ∂ γ ∗ 2 f ˉ ξ ( U ^ γ i ; γ ) f ˉ ξ ( U ^ γ i ; γ ) ] + o p ( 1 ) = 1 n ∑ i = 1 n [ ( f [ 1 ] ( W i , U γ i ; γ ) f 0 ( U γ i ; γ ) ) ⊗ 2 − f [ 2 ] ( W i , U γ i ; γ ) f 0 ( U γ i ; γ ) ] + o p ( 1 ) . (136) \begin{aligned}
\frac{1}{n}pI(\gamma)=\frac{1}{n}\sum_{i=1}^{n}\left[\left(\frac{\partial_{\gamma^*}\bar f_{\xi}(\widehat U_{\gamma i};\gamma)}{\bar f_{\xi}(\widehat U_{\gamma i};\gamma)}\right)^{\otimes2}-\frac{\partial_{\gamma^*}^{2}\bar f_{\xi}(\widehat U_{\gamma i};\gamma)}{\bar f_{\xi}(\widehat U_{\gamma i};\gamma)}\right]+o_p(1) \\
&=\frac{1}{n}\sum_{i=1}^{n}\left[\left(\frac{f^{[1]}(W_i,U_{\gamma i};\gamma)}{f_0(U_{\gamma i};\gamma)}\right)^{\otimes2}-\frac{f^{[2]}(W_i,U_{\gamma i};\gamma)}{f_0(U_{\gamma i};\gamma)}\right]+o_p(1). \tag*{(136)}
\end{aligned} n 1 p I ( γ ) = n 1 i = 1 ∑ n ( f ˉ ξ ( U γ i ; γ ) ∂ γ ∗ f ˉ ξ ( U γ i ; γ ) ) ⊗ 2 − f ˉ ξ ( U γ i ; γ ) ∂ γ ∗ 2 f ˉ ξ ( U γ i ; γ ) + o p ( 1 ) = n 1 i = 1 ∑ n [ ( f 0 ( U γ i ; γ ) f [ 1 ] ( W i , U γ i ; γ ) ) ⊗ 2 − f 0 ( U γ i ; γ ) f [ 2 ] ( W i , U γ i ; γ ) ] + o p ( 1 ) . (136)
Applying the law of large numbers to the i.i.d. representation in (H.17) yields
1 n p I ( γ ) → p E [ ( f [ 1 ] ( W , U γ ; γ ) f 0 ( U γ ; γ ) ) ⊗ 2 − f [ 2 ] ( W , U γ ; γ ) f 0 ( U γ ; γ ) ] . (137) \frac{1}{n}pI(\gamma)\xrightarrow{p}\mathbb{E}\left[\left(\frac{f^{[1]}(W,U_{\gamma};\gamma)}{f_0(U_{\gamma};\gamma)}\right)^{\otimes2}-\frac{f^{[2]}(W,U_{\gamma};\gamma)}{f_0(U_{\gamma};\gamma)}\right]. \tag*{(137)} n 1 p I ( γ ) p E [ ( f 0 ( U γ ; γ ) f [ 1 ] ( W , U γ ; γ ) ) ⊗ 2 − f 0 ( U γ ; γ ) f [ 2 ] ( W , U γ ; γ ) ] . (137)
By Lemma S3 with t = 2 t=2 t = 2 , the uniform integrability of ∂ γ ∗ 2 f ˉ ξ ( U o ; γ o ) / f ( U o ; γ o ) \partial_{\gamma^*}^{2}\bar f_{\xi}(U^{o};\gamma^{o})/f(U^{o};\gamma^{o}) ∂ γ ∗ 2 f ˉ ξ ( U o ; γ o ) / f ( U o ; γ o ) , and
E [ ∂ γ ∗ 2 f ˉ ξ ( U o ; γ o ) f 0 ( U o ; γ o ) ] = ∂ γ ∗ 2 ∫ f ˉ ξ ( u ; γ ) d u ∣ γ = γ o = 0 , \mathbb{E}\left[\frac{\partial_{\gamma^*}^{2}\bar f_{\xi}(U^{o};\gamma^{o})}{f_0(U^{o};\gamma^{o})}\right]=\left.\partial_{\gamma^*}^{2}\int\bar f_{\xi}(u;\gamma)\,du\right|_{\gamma=\gamma^{o}}=0, E [ f 0 ( U o ; γ o ) ∂ γ ∗ 2 f ˉ ξ ( U o ; γ o ) ] = ∂ γ ∗ 2 ∫ f ˉ ξ ( u ; γ ) d u γ = γ o = 0 ,
it follows that
E [ f ξ [ 2 ] ( W i , U o ; γ o ) f 0 ( U o ; γ o ) ] = 0. \mathbb{E}\left[\frac{f_{\xi}^{[2]}(W_i,U^{o};\gamma^{o})}{f_0(U^{o};\gamma^{o})}\right]=0. E [ f 0 ( U o ; γ o ) f ξ [ 2 ] ( W i , U o ; γ o ) ] = 0.
The continuity of p I ( γ ) pI(\gamma) p I ( γ ) on Γ \Gamma Γ and the consistency of γ ~ ∗ \widetilde{\gamma}_* γ ∗ to γ ∗ o \gamma_*^{o} γ ∗ o imply
1 n p I ( γ ~ ) → p V . (138) \frac{1}{n}pI(\widetilde{\gamma})\xrightarrow{p}V. \tag*{(138)} n 1 p I ( γ ) p V . (138)
Substituting (H.16) and (H.19) into (H.9) and applying Slutsky’s theorem, the asymptotic normality of γ ~ \widetilde{\gamma} γ is established under assumption C4.
Supplementary Tables and Figures